September, 2024
+ +2024-09-01
+-
+
- Upgrade CGSpace to DSpace 7.6.2 +
2024-09-05
+-
+
- Finalize work on migrating DSpace Angular from Yarn to NPM +
2024-09-06
+-
+
- This morning Tomcat crashed due to an OOM kill: +
Sep 06 00:00:24 server systemd[1]: tomcat9.service: A process of this unit has been killed by the OOM killer.
+Sep 06 00:00:25 server systemd[1]: tomcat9.service: Main process exited, code=killed, status=9/KILL
+Sep 06 00:00:25 server systemd[1]: tomcat9.service: Failed with result 'oom-kill'.
+
-
+
- According to the system journal, it was a Node.js dspace-angular process that tried to allocate memory and failed, thus invoking the OOM killer +
- Currently I see high memory usage in those processes: +
$ pm2 status
+┌────┬──────────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
+│ id │ name │ namespace │ version │ mode │ pid │ uptime │ ↺ │ status │ cpu │ mem │ user │ watching │
+├────┼──────────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
+│ 0 │ dspace-ui │ default │ 7.6.3-… │ cluster │ 994 │ 4D │ 0 │ online │ 0% │ 3.4gb │ dspace │ disabled │
+│ 1 │ dspace-ui │ default │ 7.6.3-… │ cluster │ 1015 │ 4D │ 0 │ online │ 0% │ 3.4gb │ dspace │ disabled │
+│ 2 │ dspace-ui │ default │ 7.6.3-… │ cluster │ 1029 │ 4D │ 0 │ online │ 0% │ 3.4gb │ dspace │ disabled │
+│ 3 │ dspace-ui │ default │ 7.6.3-… │ cluster │ 1042 │ 4D │ 0 │ online │ 0% │ 3.4gb │ dspace │ disabled │
+└────┴──────────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘
+
-
+
- I bet if I look in the logs I’d find some kind of heavy traffic on the frontend, causing high caching for Angular SSR +
2024-09-08
+-
+
- Analyzing memory use in our DSpace hosts, which have 32GB of memory
+
-
+
- Effective cache of PostgreSQL is estimated at 11GB, which seems way high since the database is only 2GB +
- Realistically this should be how we adjust, with PostgreSQL using ~8GB (or less) and each dspace-angular process pinned at 2GB… +
+
++Total - Solr - Tomcat Postgres - Nginx - Angular +31366 − (1024×4.4) − 7168 − (8×1024) − 512 - (4x2048) = 2796.4 left…
+
-
+
- I put some of these changes in on DSpace Test and will monitor this week +