Update notes for 2017-11-29

content/post/2017-11.md

@@ -893,3 +893,24 @@ $ grep 70.32.83.92 dspace.log.2017-11-23 | grep -o -E 'session_id=[A-Z0-9]{32}'
 
 - PostgreSQL activity shows 69 connections
 - I don't have time to troubleshoot more as I'm in Nairobi working on the HPC so I just restarted Tomcat for now
+- A few hours later Uptime Robot says the server is down again
+- I don't see much activity in the logs but there are 87 PostgreSQL connections
+- But shit, there were 10,000 unique Tomcat sessions today:
+
+```
+$ cat dspace.log.2017-11-29 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+10037
+```
+
+- Although maybe that's not much, as the previous two days had more:
+
+```
+$ cat dspace.log.2017-11-27 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+12377
+$ cat dspace.log.2017-11-28 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+16984
+```
+
+- I think we just need start increasing the number of allowed PostgreSQL connections instead of fighting this, as it's the most common source of crashes we have
+- I will bump DSpace's `db.maxconnections` from 60 to 90, and PostgreSQL's `max_connections` from 183 to 273 (which is using my loose formula of 90 * webapps + 3)
+- I really need to figure out how to get DSpace to use a PostgreSQL connection pool