CGSpace Notes

Documenting day-to-day work on the CGSpace repository.

March, 2021

in  Notes

2021-03-01

  • Discuss some OpenRXV issues with Abdullah from CodeObia
    • He’s trying to work on the DSpace 6+ metadata schema autoimport using the DSpace 6+ REST API
    • Also, we found some issues building and running OpenRXV currently due to ecosystem shift in the Node.js dependencies

2021-03-02

2021-03-03

2021-03-04

  • Peter is having issues with the workflow since yesterday
    • I looked at the Munin stats and see a high number of database locks since yesterday

PostgreSQL locks week PostgreSQL connections week

  • I looked at the number of connections in PostgreSQL and it’s definitely high again:
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
1020
  • I reported it to Atmire to take a look, on the same issue we had been tracking this before
  • Abenet asked me to add a new ORCID for ILRI staff member Zoe Campbell
  • I added it to the controlled vocabulary and then tagged her existing items on CGSpace using my add-orcid-identifier.py script:
$ cat 2021-03-04-add-zoe-campbell-orcid.csv 
dc.contributor.author,cg.creator.identifier
"Campbell, Zoë","Zoe Campbell: 0000-0002-4759-9976"
"Campbell, Zoe A.","Zoe Campbell: 0000-0002-4759-9976"
$ ./ilri/add-orcid-identifiers-csv.py -i 2021-03-04-add-zoe-campbell-orcid.csv -db dspace -u dspace -p 'fuuu'
  • I still need to do cleanup on the journal articles metadata
    • Peter sent me some cleanups but I can’t use them in the search/replace format he gave
    • I think it’s better to export the metadata values with IDs and import cleaned up ones as CSV
localhost/dspace63= > \COPY (SELECT dspace_object_id AS id, text_value as "cg.journal" FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=251) to /tmp/2021-02-24-journals.csv WITH CSV HEADER;
COPY 32087
  • I used OpenRefine to remove all journal values that didn’t have one of these values: ; ( )
    • Then I cloned the cg.journal field to cg.volume and cg.issue
    • I used some GREL expressions like these to extract the journal name, volume, and issue:
value.partition(';')[0].trim() # to get journal names
value.partition(/[0-9]+\([0-9]+\)/)[1].replace(/^(\d+)\(\d+\)/,"$1") # to get journal volumes
value.partition(/[0-9]+\([0-9]+\)/)[1].replace(/^\d+\((\d+)\)/,"$1") # to get journal issues
  • Then I uploaded the changes to CGSpace using dspace metadata-import
  • Margarita from CCAFS was asking about an error deleting some items that were showing up in Google and should have been private
  • Yesterday Abenet added me to a WLE collection approver/editer steps so we can try to figure out why Niroshini is having issues adding metadata to Udana’s submissions
    • I edited Udana’s submission to CGSpace:
      • corrected the title
      • added language English
      • changed the link to the external item page instead of PDF
      • added SDGs from the external item page
      • added AGROVOC subjects from the external item page
      • added pagination (extent)
      • changed the license to “other” because CC-BY-NC-ND is not printed anywhere in the PDF or external item page

2021-03-05

  • I migrated the Docker bind mount for the AReS Elasticsearch container to a Docker volume:
$ docker-compose -f docker/docker-compose.yml down
$ docker volume create docker_esData_7
$ docker container create --name es_dummy -v docker_esData_7:/usr/share/elasticsearch/data:rw elasticsearch:7.6.2
$ docker cp docker/esData_7/nodes es_dummy:/usr/share/elasticsearch/data
$ docker rm es_dummy
# edit docker/docker-compose.yml to switch from bind mount to volume
$ docker-compose -f docker/docker-compose.yml up -d
  • The trick is that when you create a volume like “myvolume” from a docker-compose.yml file, Docker will create it with the name “docker_myvolume”
    • If you create it manually on the command line with docker volume create myvolume then the name is literally “myvolume”
  • I still need to make the changes to git master and add these notes to the pull request so Moayad and others can benefit
  • Delete the openrxv-items-temp index to test a fresh harvesting:
$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'

2021-03-05

  • Check the results of the AReS harvesting from last night:
$ curl -s 'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'
{
  "count" : 101761,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  }
}
  • Set the current items index to read only and make a backup:
$ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d' {"settings": {"index.blocks.write":true}}'
$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-2021-03-05
  • Delete the current items index and clone the temp one to it:
$ curl -XDELETE 'http://localhost:9200/openrxv-items'
$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items
  • Then delete the temp and backup:
$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
{"acknowledged":true}%
$ curl -XDELETE 'http://localhost:9200/openrxv-items-2021-03-05'

2021-03-07

  • I realized there is something wrong with the Elasticsearch indexes on AReS
    • On a new test environment I see openrxv-items is correctly an alias of openrxv-items-final:
$ curl -s 'http://localhost:9200/_alias/' | python -m json.tool | less
...
    "openrxv-items-final": {
        "aliases": {
            "openrxv-items": {}
        }
    },
  • But on AReS production openrxv-items has somehow become an index:
$ curl -s 'http://localhost:9200/_alias/' | python -m json.tool | less
...
    "openrxv-items": {
        "aliases": {}
    },
    "openrxv-items-final": {
        "aliases": {}
    },
    "openrxv-items-temp": {
        "aliases": {}
    },
  • I fixed the issue on production by cloning the openrxv-items index to openrxv-items-final, deleting openrxv-items, and then re-creating it as an alias:
$ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-2021-03-07
$ curl -XDELETE 'http://localhost:9200/openrxv-items-final'
$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-final
$ curl -XDELETE 'http://localhost:9200/openrxv-items'
$ curl -s -X POST 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -d'{"actions" : [{"add" : { "index" : "openrxv-items-final", "alias" : "openrxv-items"}}]}'
  • Delete backups and remove read-only mode on openrxv-items:
$ curl -XDELETE 'http://localhost:9200/openrxv-items-2021-03-07'
$ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": false}}'
  • Linode sent alerts about the CPU usage on CGSpace yesterday and the day before
    • Looking in the logs I see a few IPs making heavy usage on the REST API and XMLUI:
# zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/access.log.2.gz /var/log/nginx/access.log.3.gz | grep -E '0[56]/Mar/2021' | goaccess --log-format=COMBINED -
  • I see the usual IPs for CCAFS and ILRI importer bots, but also 143.233.242.132 which appears to be for GARDIAN:
# zgrep '143.233.242.132' /var/log/nginx/access.log.1 | grep -c Delphi
6237
# zgrep '143.233.242.132' /var/log/nginx/access.log.1 | grep -c -v Delphi
6418