Add notes for 2019-02-10

content/posts/2019-02.md

@@ -439,4 +439,54 @@ Error sending email:
 
 - 151.80.203.180 is on OVH so I sent a message to their abuse email...
 
+## 2019-02-10
+
+- Linode sent another alert about CGSpace (linode18) CPU load this morning, here are the top IPs in the web server XMLUI and API logs before, during, and after that time:
+
+```
+# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -E "10/Feb/2019:0(5|6|7|8|9)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    232 18.195.78.144
+    238 35.237.175.180
+    281 66.249.66.221
+    314 151.80.203.180
+    319 34.218.226.147
+    326 40.77.167.178
+    352 157.55.39.149
+    444 2a01:4f8:140:3192::2
+   1171 5.9.6.51
+   1196 66.249.66.219
+# zcat --force /var/log/nginx/{oai,rest,statistics}.log /var/log/nginx/{oai,rest,statistics}.log.1 | grep -E "10/Feb/2019:0(5|6|7|8|9)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+      6 112.203.241.69
+      7 157.55.39.149
+      9 40.77.167.178
+     15 66.249.66.219
+    368 45.5.184.72
+    432 50.116.102.77
+    971 34.218.226.147
+   4403 45.5.186.2
+   4668 205.186.128.185
+   4668 70.32.83.92
+```
+
+- Another interesting thing might be the total number of requests for web and API services during that time:
+
+```
+# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -cE "10/Feb/2019:0(5|6|7|8|9)"
+16333
+# zcat --force /var/log/nginx/{oai,rest,statistics}.log /var/log/nginx/{oai,rest,statistics}.log.1 | grep -cE "10/Feb/2019:0(5|6|7|8|9)"
+15964
+```
+
+- Also, the number of unique IPs served during that time:
+
+```
+# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -E "10/Feb/2019:0(5|6|7|8|9)" | awk '{print $1}' | sort | uniq | wc -l
+1622
+# zcat --force /var/log/nginx/{oai,rest,statistics}.log /var/log/nginx/{oai,rest,statistics}.log.1 | grep -E "10/Feb/2019:0(5|6|7|8|9)" | awk '{print $1}' | sort | uniq | wc -l
+95
+```
+
+- It's very clear to me now that the API requests are the heaviest!
+- I think I need to increase the Linode alert threshold from 300 to 350% now so I stop getting some of these alerts—it's becoming a bit of the boy who cried wolf because it alerts like clockwork twice per day!
+
 <!-- vim: set sw=2 ts=2: -->