CGSpace Notes

Documenting day-to-day work on the CGSpace repository.

June, 2022

2022-06-06

  • Look at the Solr statistics on CGSpace
    • I see 167,000 hits from a bunch of Microsoft IPs with reverse DNS “msnbot-” using the Solr query dns:*msnbot* AND dns:*.msn.com
    • I purged these first so I could see the other “real” IPs in the Solr facets
  • I see 47,500 hits from 80.248.237.167 on a data center ISP in Sweden, using a normal user agent
  • I see 13,000 hits from 163.237.216.11 on a data center ISP in Australia, using a normal user agent
  • I see 7,300 hits from 208.185.238.57 from Britanica, using a normal user agent
    • There seem to be many more of these:
# zcat --force /var/log/nginx/access.log* | grep 208.185.238. | awk '{print $1}' | sort | uniq -c | sort -h
      2 208.185.238.1
    166 208.185.238.54
   1293 208.185.238.51
   2587 208.185.238.59
   4692 208.185.238.56
   5480 208.185.238.53
   6277 208.185.238.52
   6400 208.185.238.58
   8261 208.185.238.55
  17549 208.185.238.57
  • I see 3,000 hits from 178.208.75.33 by a Russian-owned IP in the Netherlands that is making a GET to / every one minute, using a normal user agent
  • I see 3,000 hits from 134.122.124.196 on Digital Ocean to the REST API with a normal user agent
  • I purged all these hits from IPs for a total of about 265,000
  • Then I faceted by user agent and found
    • 1,000 hits by insomnia/2022.2.1, which I also saw last month and submitted to COUNTER-Robots
    • 265 hits by omgili/0.5 +http://omgili.com
    • 150 hits by Vizzit
    • 132 hits by MetaInspector/5.7.0 (+https://github.com/jaimeiniesta/metainspector)
    • 73 hits by Scoop.it
    • 62 hits by bitdiscovery
    • 59 hits by Asana/1.4.0 WebsiteMetadataRetriever
    • 32 hits by Sprout Social (Link Attachment)
    • 29 hits by CyotekWebCopy/1.9 CyotekHTTP/6.2
    • 20 hits by Hootsuite-Authoring/1.0
  • I purged about 4,100 hits from these user agents