Add notes for 2018-01-04

content/post/2018-01.md

@@ -185,3 +185,42 @@ $ grep 134.155.96.78 dspace.log.2018-01-03 | grep -o -E 'session_id=[A-Z0-9]{32}
 - I have no idea what these are but they seem to be coming from Amazon...
 - I guess for now I just have to increase the database connection pool's max active
 - It's currently 75 and normally I'd just bump it by 25 but let me be a bit daring and push it by 50 to 125, because I used to see at least 121 connections in pg_stat_activity before when we were using the shitty default pooling
+
+## 2018-01-04
+
+- CGSpace went down and up a bunch of times last night and ILRI staff were complaining a lot last night
+- The XMLUI logs show this activity:
+
+```
+# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -E "4/Jan/2018" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+    968 197.211.63.81
+    981 213.55.99.121
+   1039 66.249.64.93
+   1258 157.55.39.175
+   1273 207.46.13.182
+   1311 157.55.39.191
+   1319 157.55.39.197
+   1775 66.249.64.78
+   2216 104.196.152.243
+   3366 66.249.64.91
+```
+
+- Again we ran out of PostgreSQL database connections, even after bumping the pool max active limit from 50 to 75 to 125 yesterday!
+
+```
+2018-01-04 07:36:08,089 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error -
+org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-256] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:125; busy:125; idle:0; lastwait:5000].
+```
+
+- So for this week that is the number one problem!
+
+```
+$ grep -c "Timeout: Pool empty." dspace.log.2018-01-*
+dspace.log.2018-01-01:0
+dspace.log.2018-01-02:1972
+dspace.log.2018-01-03:1909
+dspace.log.2018-01-04:1559
+```
+
+- I will just bump the connection limit to 300 because I'm fucking fed up with this shit
+- Once I get back to Amman I will have to try to create different database pools for different web applications, like recently discussed on the dspace-tech mailing list