CGSpace Notes

Documenting day-to-day work on the CGSpace repository.

May, 2021

in  Notes

2021-05-01

  • I looked at the top user agents and IPs in the Solr statistics for last month and I see these user agents:
    • “RI/1.0”, 1337
    • “Microsoft Office Word 2014”, 941
  • I will add the RI/1.0 pattern to our DSpace agents overload and purge them from Solr (we had previously seen this agent with 9,000 hits or so in 2020-09), but I think I will leave the Microsoft Word one… as that’s an actual user…
  • I should probably add the RI/1.0 pattern to COUNTER-Robots project
  • As well as these IPs:
    • 193.169.254.178, 21648
    • 181.62.166.177, 20323
    • 45.146.166.180, 19376
  • The first IP seems to be in Estonia and their requests to the REST API change user agents from curl to Mac OS X to Windows and more
    • Also, they seem to be trying to exploit something:
193.169.254.178 - - [21/Apr/2021:01:59:01 +0200] "GET /rest/collections/1179/items?limit=812&expand=metadata\x22%20and%20\x2221\x22=\x2221 HTTP/1.1" 400 5 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
193.169.254.178 - - [21/Apr/2021:02:00:36 +0200] "GET /rest/collections/1179/items?limit=812&expand=metadata-21%2B21*01 HTTP/1.1" 200 458201 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
193.169.254.178 - - [21/Apr/2021:02:00:36 +0200] "GET /rest/collections/1179/items?limit=812&expand=metadata'||lower('')||' HTTP/1.1" 400 5 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
193.169.254.178 - - [21/Apr/2021:02:02:10 +0200] "GET /rest/collections/1179/items?limit=812&expand=metadata'%2Brtrim('')%2B' HTTP/1.1" 200 458209 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
  • I will report the IP on abuseipdb.com and purge their hits from Solr
  • The second IP is in Colombia and is making thousands of requests for what looks like some test site:
181.62.166.177 - - [20/Apr/2021:22:48:42 +0200] "GET /rest/collections/d1e11546-c62a-4aee-af91-fd482b3e7653/items?expand=metadata HTTP/2.0" 200 123613 "http://cassavalighthousetest.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36"
181.62.166.177 - - [20/Apr/2021:22:55:39 +0200] "GET /rest/collections/d1e11546-c62a-4aee-af91-fd482b3e7653/items?expand=metadata HTTP/2.0" 200 123613 "http://cassavalighthousetest.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36"
  • But this site does not exist (yet?)
    • I will purge them from Solr
  • The third IP is in Russia apparently, and the user agent has the pl-PL locale with thousands of requests like this:
45.146.166.180 - - [18/Apr/2021:16:28:44 +0200] "GET /bitstream/handle/10947/4153/.AAS%202014%20Annual%20Report.pdf?sequence=1%22%29%29%20AND%201691%3DUTL_INADDR.GET_HOST_ADDRESS%28CHR%28113%29%7C%7CCHR%28118%29%7C%7CCHR%28113%29%7C%7CCHR%28106%29%7C%7CCHR%28113%29%7C%7C%28SELECT%20%28CASE%20WHEN%20%281691%3D1691%29%20THEN%201%20ELSE%200%20END%29%20FROM%20DUAL%29%7C%7CCHR%28113%29%7C%7CCHR%2898%29%7C%7CCHR%28122%29%7C%7CCHR%28120%29%7C%7CCHR%28113%29%29%20AND%20%28%28%22RKbp%22%3D%22RKbp&isAllowed=y HTTP/1.1" 200 918998 "http://cgspace.cgiar.org:80/bitstream/handle/10947/4153/.AAS 2014 Annual Report.pdf" "Mozilla/5.0 (Windows; U; Windows NT 5.1; pl-PL) AppleWebKit/523.15 (KHTML, like Gecko) Version/3.0 Safari/523.15"
  • I will purge these all with my check-spider-ip-hits.sh script:
$ ./ilri/check-spider-ip-hits.sh -f /tmp/ips.txt -p
Purging 21648 hits from 193.169.254.178 in statistics
Purging 20323 hits from 181.62.166.177 in statistics
Purging 19376 hits from 45.146.166.180 in statistics

Total number of bot hits purged: 61347

2021-05-02

  • Check the AReS Harvester indexes:
$ curl -s http://localhost:9200/_cat/indices | grep openrxv-items
yellow open openrxv-items-temp       H-CGsyyLTaqAj6-nKXZ-7w 1 1      0 0    283b    283b
yellow open openrxv-items-final      ul3SKsa7Q9Cd_K7qokBY_w 1 1 103951 0   254mb   254mb
$ curl -s 'http://localhost:9200/_alias/' | python -m json.tool
...
    "openrxv-items-temp": {
        "aliases": {}
    },
    "openrxv-items-final": {
        "aliases": {
            "openrxv-items": {}
        }
    },
  • I think they look OK (openrxv-items is an alias of openrxv-items-final), but I took a backup just in case:
$ elasticdump --input=http://localhost:9200/openrxv-items --output=/home/aorth/openrxv-items_mapping.json --type=mapping
$ elasticdump --input=http://localhost:9200/openrxv-items --output=/home/aorth/openrxv-items_data.json --type=data --limit=1000
  • Then I started an indexing in the AReS Explorer admin dashboard
  • The indexing finished, but it looks like the aliases are messed up again:
$ curl -s http://localhost:9200/_cat/indices | grep openrxv-items
yellow open openrxv-items-temp       H-CGsyyLTaqAj6-nKXZ-7w 1 1 104165 105024 487.7mb 487.7mb
yellow open openrxv-items-final      d0tbMM_SRWimirxr_gm9YA 1 1    937      0   2.2mb   2.2mb

2021-05-05

  • Peter noticed that we no longer display cg.link.reference on the item view
    • It seems that this got dropped accidentally when we migrated to dcterms.relation in CG Core v2
    • I fixed it in the 6_x-prod branch and told him it will be live soon

2021-05-09

  • I set up a clean DSpace 6.4 instance locally to test some things against, for example to be able to rule out whether some issues are due to Atmire modules or are fixed in the as-of-yet-unreleased DSpace 6.4
    • I had to delete all the Atmire schemas, then it worked fine on Tomcat 8.5 with Mirage (I didn’t want to bother with npm and ruby for Mirage 2)
    • Then I tried to see if I could reproduce the mapping issue that Marianne raised last month
      • I tried unmapping and remapping to the CGIAR Gender grants collection and the collection appears in the item view’s list of mapped collections, but not on the collection browse itself
      • Then I tried mapping to a new collection and it was the same as above
      • So this issue is really just a DSpace bug, and nothing to do with Atmire and not fixed in the unreleased DSpace 6.4
      • I will try one more time after updating the Discovery index (I’m also curious how fast it is on vanilla DSpace 6.4, though I think I tried that when I did the flame graphs in 2019 and it was miserable)
$ time ~/dspace64/bin/dspace index-discovery -b
~/dspace64/bin/dspace index-discovery -b  4053.24s user 53.17s system 38% cpu 2:58:53.83 total
  • Nope! Still slow, and still no mapped item…
    • I even tried unmapping it from all collections, and adding it to a single new owning collection…
  • Ah hah! Actually, I was inspecting the item’s authorization policies when I noticed that someone had made the item private!
    • After making it public again I was able to see it in the target collection
  • The indexes on AReS Explorer are messed up after last week’s harvesting:
$ curl -s http://localhost:9200/_cat/indices | grep openrxv-items
yellow open openrxv-items-temp       H-CGsyyLTaqAj6-nKXZ-7w 1 1 104165 105024 487.7mb 487.7mb
yellow open openrxv-items-final      d0tbMM_SRWimirxr_gm9YA 1 1    937      0   2.2mb   2.2mb

$ curl -s 'http://localhost:9200/_alias/' | python -m json.tool
...
    "openrxv-items-final": {
        "aliases": {}
    },
    "openrxv-items-temp": {
        "aliases": {
            "openrxv-items": {}
        }
    }
  • openrxv-items should be an alias of openrxv-items-final…
  • I made a backup of the temp index and then started indexing on the AReS Explorer admin dashboard:
$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items-temp-backup
$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": false}}'

2021-05-10

  • Amazing, the harvesting on AReS finished but it messed up all the indexes and now there are no items in any index!
$ curl -s http://localhost:9200/_cat/indices | grep openrxv-items
yellow open openrxv-items-temp        8thRX0WVRUeAzmd2hkG6TA 1 1      0     0    283b    283b
yellow open openrxv-items-temp-backup _0tyvctBTg2pjOlcoVP1LA 1 1 104165 20134 305.5mb 305.5mb
yellow open openrxv-items-final       BtvV9kwVQ3yBYCZvJS1QyQ 1 1      0     0    283b    283b
  • I fixed the indexes manually by re-creating them and cloning from the backup:
$ curl -XDELETE 'http://localhost:9200/openrxv-items-final'
$ curl -X PUT "localhost:9200/openrxv-items-temp-backup/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
$ curl -s -X POST http://localhost:9200/openrxv-items-temp-backup/_clone/openrxv-items-final
$ curl -s -X POST 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -d'{"actions" : [{"add" : { "index" : "openrxv-items-final", "alias" : "openrxv-items"}}]}'
$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp-backup'
  • Also I ran all updated on the server and updated all Docker images, then rebooted the server (linode20):
$ docker images | grep -v ^REPO | sed 's/ \+/:/g' | cut -d: -f1,2 | xargs -L1 docker pull
  • I backed up the AReS Elasticsearch data using elasticdump, then started a new harvest:
$ elasticdump --input=http://localhost:9200/openrxv-items --output=/home/aorth/openrxv-items_mapping.json --type=mapping
$ elasticdump --input=http://localhost:9200/openrxv-items --output=/home/aorth/openrxv-items_data.json --type=data --limit=1000
  • Discuss CGSpace statistics with the CIP team
    • They were wondering why their numbers for 2020 were so low
    • I checked their community using the DSpace Statistics API and found very accurate numbers for 2020 and 2019 for them
    • I think they had been using AReS, which actually doesn’t even give stats for a time period…

2021-05-11

  • The AReS harvesting from yesterday finished, but the indexes are messed up again so I will have to fix them again before I harvest next time
  • I also spent some time looking at IWMI’s reports again
    • On AReS we don’t have a way to group by peer reviewed or item type other than doing “if type is Journal Article”
    • Also, we don’t have a way to check the IWMI Strategic Priorities because those are communities, not metadata…
    • We can get the collections an item is in from the parentCollectionList metadata, but it is saved in Elasticsearch as a string instead of a list…
    • I told them it won’t be possible to replicate their reports exactly
  • I decided to look at the CLARISA controlled vocabularies again
    • They now have 6,200 institutions (was around 3,400 when I last looked in 2020-07)
    • They have updated their Swagger interface but it still requires an API key if you want to use it from curl
    • They have ISO 3166 countries and UN M.49 regions, but I notice they have some weird names like “Russian Federation (the)”, which is not in ISO 3166 as far as I can see
    • I exported a list of the institutions to look closer
      • I found twelve items with whitespace issues
      • There are some weird entries like Research Institute for Aquaculture No1 and Research Institute for Aquaculture No2
      • A few items have weird Unicode characters like U+00AD, U+200B, and U+00A0
      • I found 100+ items with multiple languages in there name like Ministère de l’Agriculture, de la pêche et des ressources hydrauliques / Ministry of Agriculture, Hydraulic Resources and Fisheries
      • Over 600 institutions have the country in their name like Ministry of Coordination of Environmental Affairs (Mozambique)
      • For URLs they have null in some places… which is weird… why not just leave it blank?
  • I checked the CLARISA list against ROR’s April, 2020 release (“Version 9”, on figshare, though it is version 8 in the dump):
$ ./ilri/ror-lookup.py -i /tmp/clarisa-institutions.txt -r ror-data-2021-04-06.json -o /tmp/clarisa-ror-matches.csv
$ csvgrep -c matched -m 'true' /tmp/clarisa-ror-matches.csv | sed '1d' | wc -l
1770
  • With 1770 out of 6230 matched, that’s 28.5%…
  • I sent an email to Hector Tobon to point out the issues in CLARISA again and ask him to chat
  • Meeting with GARDIAN developers about CG Core and how GARDIAN works

2021-05-13

  • Fix a few thousand IWMI URLs that are using HTTP instead of HTTPS on CGSpace:
localhost/dspace63= > UPDATE metadatavalue SET text_value = REGEXP_REPLACE(text_value, 'http://www.iwmi.cgiar.org','https://www.iwmi.cgiar.org', 'g') WHERE text_value LIKE 'http://www.iwmi.cgiar.org%' AND metadata_field_id=219;
UPDATE 1132
localhost/dspace63= > UPDATE metadatavalue SET text_value = REGEXP_REPLACE(text_value, 'http://publications.iwmi.org','https://publications.iwmi.org', 'g') WHERE text_value LIKE 'http://publications.iwmi.org%' AND metadata_field_id=219;
UPDATE 1803
  • In the case of the latter, the HTTP links don’t even work! The web server returns HTTP 404 unless the request is HTTPS
  • IWMI also says that their subjects are a subset of AGROVOC so they no longer want to use cg.subject.iwmi for their subjects
    • They asked if I can move them to dcterms.subject
  • Delete two items for Udana because he was getting the “Authorization denied for action OBSOLETE (DELETE) …” error when trying to delete them (DSpace 6 bug I found a few months ago)