Add notes for 2017-12-07

content/post/2017-12.md

@@ -106,3 +106,61 @@ $ grep 2.86.122.76 /home/cgspace.cgiar.org/log/dspace.log.2017-12-01 | grep -o -
 ## 2017-12-06
 
 - Linode alerted again that the CPU usage on CGSpace was high this morning from 6 to 8 AM
+- Uptime Robot alerted that the server went down and up around 8:53 this morning
+- Uptime Robot alerted that CGSpace was down and up again a few minutes later
+- I don't see any errors in the DSpace logs but I see in nginx's access.log that UptimeRobot was returned with HTTP 499 status (Client Closed Request)
+- Looking at the REST API logs I see some new client IP I haven't noticed before:
+
+```
+# cat /var/log/nginx/rest.log /var/log/nginx/rest.log.1 | grep -E "6/Dec/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+     18 95.108.181.88
+     19 68.180.229.254
+     30 207.46.13.151
+     33 207.46.13.110
+     38 40.77.167.20
+     41 157.55.39.223
+     82 104.196.152.243
+   1529 50.116.102.77
+   4005 70.32.83.92
+   6045 45.5.184.196
+```
+
+- 50.116.102.77 is apparently in the US on websitewelcome.com
+
+## 2017-12-07
+
+- Uptime Robot reported a few times today that CGSpace was down and then up
+- At one point Tsega restarted Tomcat
+- I never got any alerts about high load from Linode though...
+- I looked just now and see that there are 121 PostgreSQL connections!
+- The top users right now are:
+
+```
+# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -E "7/Dec/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail 
+    838 40.77.167.11
+    939 66.249.66.223
+   1149 66.249.66.206
+   1316 207.46.13.110
+   1322 207.46.13.151
+   1323 2001:da8:203:2224:c912:1106:d94f:9189
+   1414 157.55.39.223
+   2378 104.196.152.243
+   2662 66.249.66.219
+   5110 124.17.34.60
+```
+
+- We've never seen 124.17.34.60 yet, but it's really hammering us!
+- Apparently it is from China, and here is one of its user agents:
+
+```
+Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.2; Win64; x64; Trident/7.0; LCTE)
+```
+
+- It is responsible for 4,500 Tomcat sessions today alone:
+
+```
+$ grep 124.17.34.60 /home/cgspace.cgiar.org/log/dspace.log.2017-12-07 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+4574
+```
+
+- I've adjusted the nginx IP mapping that I set up last month to account for 124.17.34.60 and 124.17.34.59 using a regex, as it's the same bot on the same subnet