Add notes for 2020-05-07

content/posts/2020-05.md

@@ -33,4 +33,29 @@ categories: ["Notes"]
 - Atmire responded asking for more information about the Solr statistics processing bug in CUA so I sent them some full logs
   - Also I asked again about the Maven variable interpolation issue for `cua.version.number`, and if they would be willing to upgrade CUA to use Font Awesome 5 instead of 4.
 
+## 2020-05-07
+
+- Linode sent an alert that there was high CPU usage on CGSpace (linode18) early this morning
+  - I looked at the nginx logs using goaccess and I found a few IPs making lots of requests around then:
+
+```
+# cat /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "07/May/2020:(01|03|04)" | goaccess --log-format=COMBINED -
+```
+
+- The two main IPs making requests around then are 188.134.31.88 and 212.34.8.188
+  - The first is in Russia and it is hitting mostly XMLUI Discover links using _dozens_ of different user agents, a total of 20,000 requests this week
+  - The second IP is CodeObia testing AReS, a total of 171,000 hits this month
+  - I will purge both of those IPs from the Solr stats using my `check-spider-ip-hits.sh` script:
+
+```
+$ ./check-spider-ip-hits.sh -f /tmp/ips -s statistics -p
+Purging 171641 hits from 212.34.8.188 in statistics
+Purging 20691 hits from 188.134.31.88 in statistics
+
+Total number of bot hits purged: 192332
+```
+
+- And then I will add 188.134.31.88 to the nginx bad bot list and tell CodeObia to please use a "bot" user agent
+- I also changed the nginx config to block requests with blank user agents
+
 <!-- vim: set sw=2 ts=2: -->