Add notes for 2021-07-20

content/posts/2021-07.md

@@ -466,4 +466,51 @@ Total number of bot hits purged: 197116
 
 - I started a harvest on AReS and it finished in a few hours now that the load on CGSpace is back to a normal level
 
-<!-- vim: set sw=2 ts=2: -->
+## 2021-07-20
+
+- Looking again at the IPs making connections to CGSpace over the last few days from these seven ASNs, it's much higher than I noticed yesterday:
+
+```console
+$ csvgrep -c asn -r '^(49453|46844|206485|62282|36352|35913|35624)$' /tmp/out.csv | csvcut -c ip | sed 1d | sort | uniq | wc -l
+5643
+```
+
+- I purged 27,000 more hits from the Solr stats using this new list of IPs with my `check-spider-ip-hits.sh` script
+- Surprise surprise, I checked the nginx logs from 2021-06-23 when we last had issues with thousands of XMLUI sessions and PostgreSQL connections and I see IPs from the same ASNs!
+
+```console
+$ sudo zcat --force /var/log/nginx/access.log.27.gz /var/log/nginx/access.log.28.gz | grep -E " (200|499) " | grep -v -E "(mahider|Googlebot|Turnitin|Grammarly|Unpaywall|UptimeRobot|bot)" | awk '{print $1}' | sort | uniq > /tmp/all-ips-june-23.txt
+$ ./ilri/resolve-addresses-geoip2.py -i /tmp/all-ips-june-23.txt -o /tmp/out.csv
+$ csvcut -c 2,4 /tmp/out.csv | sed 1d | sort | uniq -c | sort -n | tail -n 15
+    265 GOOGLE,15169
+    277 Silverstar Invest Limited,35624
+    280 FACEBOOK,32934
+    288 SAFARICOM-LIMITED,33771
+    399 AMAZON-AES,14618
+    427 MICROSOFT-CORP-MSN-AS-BLOCK,8075
+    455 Opera Software AS,39832
+    481 MTN NIGERIA Communication limited,29465
+    502 DEDIPATH-LLC,35913
+    506 AS-COLOCROSSING,36352
+    602 UAB Rakrejus,62282
+    822 ST-BGP,46844
+    874 Ethiopian Telecommunication Corporation,24757
+    912 UGB Hosting OU,206485
+   1607 Global Layer B.V.,49453
+```
+
+- Again it was over 5,000 IPs:
+
+```console
+$ csvgrep -c asn -r '^(49453|46844|206485|62282|36352|35913|35624)$' /tmp/out.csv | csvcut -c ip | sed 1d | sort | uniq | wc -l         
+5228
+```
+
+- Interestingly, it seems these are five thousand *different* IP addresses than the attack from last weekend, as there are over 10,000 unique ones if I combine them!
+
+```console
+$ cat /tmp/ips-june23.txt /tmp/ips-jul16.txt | sort | uniq | wc -l
+10458
+```
+
+- I purged all the (26,000) hits from these new IP addresses from Solr as well