Add notes for 2017-11-29

content/post/2017-11.md

@@ -870,3 +870,26 @@ $ grep 70.32.83.92 dspace.log.2017-11-23 | grep -o -E 'session_id=[A-Z0-9]{32}'
     379 66.249.66.70
    1855 66.249.66.90
 ```
+
+## 2017-11-29
+
+- Linode alerted that CGSpace was using 279% CPU from 6 to 8 AM this morning
+- About an hour later Uptime Robot said that the server was down
+- Here are all the top XMLUI and REST users from today:
+
+```
+# cat /var/log/nginx/rest.log  /var/log/nginx/rest.log.1  /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -E "29/Nov/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+    540 66.249.66.83
+    659 40.77.167.36
+    663 157.55.39.214
+    681 157.55.39.206
+    733 157.55.39.158
+    850 66.249.66.70
+   1311 66.249.66.90
+   1340 104.196.152.243
+   4008 70.32.83.92
+   6053 45.5.184.196
+```
+
+- PostgreSQL activity shows 69 connections
+- I don't have time to troubleshoot more as I'm in Nairobi working on the HPC so I just restarted Tomcat for now