Add notes for 2024-09

content/posts/2024-09.md

@@ -0,0 +1,56 @@
+---
+title: "September, 2024"
+date: 2024-09-01T21:16:00-07:00
+author: "Alan Orth"
+categories: ["Notes"]
+---
+
+## 2024-09-01
+
+- Upgrade CGSpace to DSpace 7.6.2
+
+<!--more-->
+
+## 2024-09-05
+
+- Finalize work on migrating DSpace Angular from Yarn to NPM
+
+## 2024-09-06
+
+- This morning Tomcat crashed due to an OOM kill:
+
+```
+Sep 06 00:00:24 server systemd[1]: tomcat9.service: A process of this unit has been killed by the OOM killer.
+Sep 06 00:00:25 server systemd[1]: tomcat9.service: Main process exited, code=killed, status=9/KILL
+Sep 06 00:00:25 server systemd[1]: tomcat9.service: Failed with result 'oom-kill'.
+```
+
+- According to the system journal, it was a Node.js dspace-angular process that tried to allocate memory and failed, thus invoking the OOM killer
+- Currently I see high memory usage in those processes:
+
+```console
+$ pm2 status
+┌────┬──────────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
+│ id │ name         │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
+├────┼──────────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
+│ 0  │ dspace-ui    │ default     │ 7.6.3-… │ cluster │ 994      │ 4D     │ 0    │ online    │ 0%       │ 3.4gb    │ dspace   │ disabled │
+│ 1  │ dspace-ui    │ default     │ 7.6.3-… │ cluster │ 1015     │ 4D     │ 0    │ online    │ 0%       │ 3.4gb    │ dspace   │ disabled │
+│ 2  │ dspace-ui    │ default     │ 7.6.3-… │ cluster │ 1029     │ 4D     │ 0    │ online    │ 0%       │ 3.4gb    │ dspace   │ disabled │
+│ 3  │ dspace-ui    │ default     │ 7.6.3-… │ cluster │ 1042     │ 4D     │ 0    │ online    │ 0%       │ 3.4gb    │ dspace   │ disabled │
+└────┴──────────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘
+```
+
+- I bet if I look in the logs I'd find some kind of heavy traffic on the frontend, causing high caching for Angular SSR
+
+## 2024-09-08
+
+- Analyzing memory use in our DSpace hosts, which have 32GB of memory
+  - Effective cache of PostgreSQL is estimated at 11GB, which seems way high since the database is only 2GB
+  - Realistically this should be how we adjust, with PostgreSQL using ~8GB (or less) and each dspace-angular process pinned at 2GB...
+
+> Total - Solr       - Tomcat   Postgres  - Nginx - Angular
+> 31366 − (1024×4.4) − 7168   − (8×1024)  − 512   - (4x2048) = 2796.4 left...
+
+- I put some of these changes in on DSpace Test and will monitor this week
+
+<!-- vim: set sw=2 ts=2: -->