CGSpace Notes

Documenting day-to-day work on the CGSpace repository.

September, 2021

in  Notes

2021-09-02

  • Troubleshooting the missing Altmetric scores on AReS
    • Turns out that I didn’t actually fix them last month because the check for content.altmetric still exists, and I can’t access the DOIs using _h.source.DOI for some reason
    • I can access all other kinds of item metadata using the Elasticsearch label, but not DOI!!!
    • I will change DOI to tomato in the repository setup and start a re-harvest… I need to see if this is some kind of reserved word or something…
    • Even as tomato I can’t access that field as _h.source.tomato in Angular, but it does work as a filter source… sigh
  • I’m having problems using the OpenRXV API
    • The syntax Moayad showed me last month doesn’t seem to honor the search query properly…

2021-09-05

  • Update Docker images on AReS server (linode20) and rebuild OpenRXV:
$ docker images | grep -v ^REPO | sed 's/ \+/:/g' | cut -d: -f1,2 | xargs -L1 docker pull
$ docker-compose build
  • Then run system updates and reboot the server
    • After the system came back up I started a fresh re-harvesting

2021-09-07

  • Checking last month’s Solr statistics to see if there are any new bots that I need to purge and add to the list
    • 78.203.225.68 made 50,000 requests on one day in August, and it is using this user agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36
    • It’s a fixed line ISP in Montpellier according to AbuseIPDB.com, and has not been flagged as abusive, so it must be some CGIAR SMO person doing some web application harvesting from the browser
    • 130.255.162.154 is in Sweden and made 46,000 requests in August and it is using this user agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11.1; rv:84.0) Gecko/20100101 Firefox/84.0
    • 35.174.144.154 is on Amazon and made 28,000 requests with this user agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36
    • 192.121.135.6 is in Sweden and made 9,000 requests with this user agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11.1; rv:84.0) Gecko/20100101 Firefox/84.0
    • 185.38.40.66 is in Germany and made 6,000 requests with this user agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:89.0) Gecko/20100101 Firefox/89.0 BoldBrains SC/1.10.2.4
    • 3.225.28.105 is in Amazon and made 3,000 requests with this user agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36
    • I also noticed that we still have tons (25,000) of requests by MSNbot using this normal-looking user agent: Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko
    • I can identify them by their reverse DNS: msnbot-40-77-167-105.search.msn.com.
    • I had already purged a bunch of these by their IPs in 2021-06, so it looks like I have to do that again
    • While looking at the MSN requests I noticed tons of requests from another strange host using reverse IP DNS: malta2095.startdedicated.com., astra5139.startdedicated.com., and many others
    • They must be related, because I see them all using the exact same user agent: Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko
    • So this startdedicated.com DNS is some Bing bot also…
  • I extracted all the IPs and purged them using my check-spider-ip-hits.sh script
    • In total I purged 225,000 hits…

2021-09-12

  • Start a harvest on AReS

2021-09-13

  • Mishell Portilla asked me about thumbnails on CGSpace being small
    • For example, 10568/114576 has a lot of white space on the left side
    • I created a new thumbnail with vipsthumbnail:
$ vipsthumbnail ARRTB2020ST.pdf -s x600 -o '%s.jpg[Q=85,optimize_coding,strip]'
  • Looking at the PDF’s metadata I see:
    • Producer: iLovePDF
    • Creator: Adobe InDesign 15.0 (Windows)
    • Format: PDF-1.7
  • Eventually I should do more tests on this and perhaps file a bug with DSpace…
  • Some Alliance people contacted me about getting access to the CGSpace API to deposit with their TIP tool
    • I told them I can give them access to DSpace Test and that we should have a meeting soon
    • We need to figure out what controlled vocabularies they should use

2021-09-14

  • Some people from the Alliance contacted me last week about AICCRA metadata
    • They have internal things called Components and Clusters, so they were asking how to store these in CGSpace
    • I suggested adding new metadata values: cg.subject.aiccraComponent and cg.subject.aiccraCluster
    • On second thought, these are identifiers so perhaps this is better: cg.identifier.aiccraComponent and cg.identifier.aiccraCluster

2021-09-15

  • Add ORCID identifier for new ILRI staff to our controlled vocabualary
    • Also tag their twenty-five existing items on CGSpace:
$ cat 2021-09-15-add-orcids.csv                                                                                  
dc.contributor.author,cg.creator.identifier
"Kotchofa, Pacem","Pacem Kotchofa: 0000-0002-1640-8807"
$ ./ilri/add-orcid-identifiers-csv.py -i 2021-09-15-add-orcids.csv -db dspace -u dspace -p 'fuuuu'
  • Meeting with Leroy Mwanzia and some other Alliance people about depositing to CGSpace via API
    • I gave them some technical information about the CGSpace API and links to the controlled vocabularies and metadata registries we are using
    • I also told them that I would create some documentation listing the metadata fields, which are mandatory, and the respective controlled vocabularies

2021-09-16

  • Start writing a Python script to parse input-forms.xml to create documentation for submissions
  • I decided to update all the metadata field descriptions in our registry so I can use that instead of the “hint” for each field in the input form
    • I will include examples as well so that it becomes a better resource

2021-09-17

$ psql -c 'SELECT * FROM pg_stat_activity' | wc -l
63
  • Load on the server is under 1.0, and there are only about 1,000 XMLUI sessions, which seems to be normal for this time of day according to Munin
  • But the DSpace log file shows tons of database issues:
$ grep -c "Timeout waiting for idle object" dspace.log.2021-09-17 
14779
  • The earliest one I see is around midnight (now is 2PM):
2021-09-17 00:01:49,572 WARN  org.hibernate.engine.jdbc.spi.SqlExceptionHelper @ SQL Error: 0, SQLState: null
2021-09-17 00:01:49,572 ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper @ Cannot get a connection, pool error Timeout waiting for idle object
  • But I was definitely logged into the site this morning so there were no issues then…
  • It seems that a few errors are normal, but there’s obviously something wrong today:
$ grep -c "Timeout waiting for idle object" dspace.log.2021-09-*
dspace.log.2021-09-01:116
dspace.log.2021-09-02:163
dspace.log.2021-09-03:77
dspace.log.2021-09-04:13
dspace.log.2021-09-05:310
dspace.log.2021-09-06:0
dspace.log.2021-09-07:29
dspace.log.2021-09-08:86
dspace.log.2021-09-09:24
dspace.log.2021-09-10:26
dspace.log.2021-09-11:12
dspace.log.2021-09-12:5
dspace.log.2021-09-13:10
dspace.log.2021-09-14:102
dspace.log.2021-09-15:542
dspace.log.2021-09-16:368
dspace.log.2021-09-17:15235
  • I restarted the server and DSpace came up fine… so it must have been some kind of fluke
  • Continue working on cleaning up and annotating the metadata registry on CGSpace
    • I removed two old metadata fields that we stopped using earlier this year with the CG Core v2 migration: cg.targetaudience and cg.title.journal

2021-09-18

2021-09-19

  • Improve CGSpace Submission Guidelines metadata parsing and documentation
  • Start a full harvest on AReS
    • The harvest completed successfully, but for some reason there were only 92,000 items…
    • I updated all Docker images, rebuilt the application, then ran all system updates and rebooted the system:
$ docker images | grep -v ^REPO | sed 's/ \+/:/g' | cut -d: -f1,2 | xargs -L1 docker pull
$ docker-compose build

2021-09-20

  • I synchronized the production CGSpace PostreSQL, Solr, and Assetstore data with DSpace Test
  • Over the weekend a few users reported that they could not log into CGSpace
    • I checked LDAP and it seems there is something wrong:
$ ldapsearch -x -H ldaps://AZCGNEROOT2.CGIARAD.ORG:636/ -b "dc=cgiarad,dc=org" -D "cgspace-ldap-account@cgiarad.org" -W "(sAMAccountName=someaccountnametocheck)"
Enter LDAP Password: 
ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)
  • I sent a message to CGNET to ask about the server settings and see if our IP is still whitelisted
    • It turns out that CGNET created a new Active Directory server (AZCGNEROOT3.cgiarad.org) and decomissioned the old one last week
    • I updated the configuration on CGSpace and confirmed that it is working
  • Create another test account for Rafael from Bioversity-CIAT to submit some items to DSpace Test:
$ dspace user -a -m tip-submit@cgiar.org -g CIAT -s Submit -p 'fuuuuuuuu'
  • I added the account to the Alliance Admins account, which is should allow him to submit to any Alliance collection
    • According to my notes from 2020-10 the account must be in the admin group in order to submit via the REST API
  • Run dspace cleanup -v process on CGSpace to clean up old bitstreams
  • Export lists of authors, donors, and affiliations for Peter Ballantyne to clean up:
localhost/dspace63= > \COPY (SELECT DISTINCT text_value as "dc.contributor.author", count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 3 GROUP BY text_value ORDER BY count DESC) to /tmp/2021-09-20-authors.csv WITH CSV HEADER;
COPY 80901
localhost/dspace63= > \COPY (SELECT DISTINCT text_value as "cg.contributor.donor", count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 248 GROUP BY text_value ORDER BY count DESC) to /tmp/2021-09-20-donors.csv WITH CSV HEADER;
COPY 1274
localhost/dspace63= > \COPY (SELECT DISTINCT text_value as "cg.contributor.affiliation", count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC) to /tmp/2021-09-20-affiliations.csv WITH CSV HEADER;
COPY 8091

2021-09-23

  • Peter sent me back the corrections for the affiliations
    • It is about 1,280 corrections and fourteen deletions
    • I cleaned them up in csv-metadata-quality and then extracted the deletes and fixes to separate files to run with fix-metadata-values.py and delete-metadata-values.py:
$ csv-metadata-quality -i ~/Downloads/2021-09-20-affiliations.csv -o /tmp/affiliations.csv -x cg.contributor.affiliation
$ csvgrep -c 'correct' -m 'DELETE' /tmp/affiliations.csv > /tmp/affiliations-delete.csv
$ csvgrep -c 'correct' -r '^.+$' /tmp/affiliations.csv | csvgrep -i -c 'correct' -m 'DELETE' > /tmp/affiliations-fix.csv
$ ./ilri/fix-metadata-values.py -i /tmp/affiliations-fix.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.affiliation -t 'correct' -m 211
$ ./ilri/delete-metadata-values.py -i /tmp/affiliations-fix.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.affiliation -m 211
  • Then I updated the controlled vocabulary for affiliations by exporting the top 1,000 used terms:
localhost/dspace63= > \COPY (SELECT DISTINCT text_value as "cg.contributor.affiliation", count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC LIMIT 1000) to /tmp/2021-09-23-affiliations.csv WITH CSV HEADER;
$ csvcut -c 1 /tmp/2021-09-23-affiliations.csv | sed 1d > /tmp/affiliations.txt
  • Peter also sent me 310 corrections and 234 deletions for donors so I applied those and updated the controlled vocabularies too
  • Move some One CGIAR-related collections around the CGSpace hierarchy for Peter Ballantyne
  • Mohammed Salem asked me for an ID to UUID mapping for CGSpace collections, so I generated one similar to the ID one I sent him in 2020-11:
localhost/dspace63= > \COPY (SELECT collection_id,uuid FROM collection WHERE collection_id IS NOT NULL) TO /tmp/2021-09-23-collection-id2uuid.csv WITH CSV HEADER;
COPY 1139

2021-09-24

  • Peter and Abenet agreed that we should consider converting more of our UPPER CASE metadata values to Title Case
    • It seems that these fields are all still using UPPER CASE:
      • cg.subject.alliancebiovciat
      • cg.species.breed
      • cg.subject.bioversity
      • cg.subject.ccafs
      • cg.subject.ciat
      • cg.subject.cip
      • cg.identifier.iitatheme
      • cg.subject.iita
      • cg.subject.ilri
      • cg.subject.pabra
      • cg.river.basin
      • cg.coverage.subregion (done)
      • dcterms.audience (done)
      • cg.subject.wle
    • We can do some of these without even asking anyone, for example cg.coverage.subregion, cg.river.basin, and dcterms.audience
  • First, I will look at cg.coverage.subregion
    • These should ideally come from ISO 3166-2 subdivisions
    • I will sentence case them and then create a controlled vocabulary from those that are matching (and worry about cleaning the rest up later)
localhost/dspace63= > UPDATE metadatavalue SET text_value=INITCAP(text_value) WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=231;
UPDATE 2903
localhost/dspace63= > \COPY (SELECT DISTINCT text_value as "cg.coverage.subregion" FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 231) to /tmp/2021-09-24-subregions.txt;
COPY 1200
  • Then I process the list for matches with my subdivision-lookup.py script, and extract only the values that matched:
$ ./ilri/subdivision-lookup.py -i /tmp/2021-09-24-subregions.txt -o /tmp/subregions.csv
$ csvgrep -c matched -m 'true' /tmp/subregions.csv | csvcut -c 1 | sed 1d > /tmp/subregions-matched.txt
$ wc -l /tmp/subregions-matched.txt 
81 /tmp/subregions-matched.txt
  • Then I updated the controlled vocabulary in the submission forms
  • I did the same for dcterms.audience, taking special care to a few all-caps values:
localhost/dspace63= > UPDATE metadatavalue SET text_value=INITCAP(text_value) WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=144 AND text_value != 'NGOS' AND text_value != 'CGIAR';
localhost/dspace63= > UPDATE metadatavalue SET text_value='NGOs' WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=144 AND text_value = 'NGOS';
  • Update submission form comment for DOIs because it was still recommending people use the “dx.doi.org” format even though I batch updated all DOIs to the “doi.org” format a few times in the last year
    • Then I updated all existing metadata to the new format again:
dspace=# UPDATE metadatavalue SET text_value = regexp_replace(text_value, 'https://dx.doi.org', 'https://doi.org') WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 220 AND text_value LIKE 'https://dx.doi.org%';
UPDATE 49

2021-09-26

  • Mohammed Salem told me last week that MELSpace and WorldFish have been upgraded to DSpace 6 so I updated the repository setup in AReS to use the UUID field instead of IDs
    • This could explain how I had problems harvesting last week, when I only had 90,000 items…
  • I started a fresh harvest on AReS
    • I realized that the sitemap on MELSpace is missing so AReS skips it, which means we cannot harvest right now… ouch
    • I sent a message to Salem and he fixed it quickly
    • I added WorldFish’s DSpace Statistics API instance to AReS before starting the plugins and now our numbers are much higher, nice!

2021-09-27

  • Add CGIAR Action Area (cg.subject.actionArea) to CGSpace as Peter had asked me a few days ago

2021-09-28

  • Francesca from the Alliance asked for help moving a bunch of reports from one collections to another on CGSpace
    • She is having problems with the “move” dialog taking minutes for each item
    • I exported the collection and sent her a copy with just the few fields she would need in order to mark the ones that need to move, then I can do the rest:
$ csvcut -c 'id,collection,dc.title[en_US]' ~/Downloads/10568-106990.csv > /tmp/2021-09-28-alliance-reports.csv
  • She sent it back fairly quickly with a new column marked “Move” so I extracted those items that matched and set them to the new owning collection:
$ csvgrep -c Move -m 'Yes' ~/Downloads/2021_28_09_alliance_reports_csv.csv | csvcut -c 1,2 | sed 's_10568/106990_10568/111506_' > /tmp/alliance-move.csv
  • Maria from the Alliance emailed us to say that approving submissions was slow on CGSpace
    • I looked at the PostgreSQL activity and it seems low:
postgres@linode18:~$ psql -c 'SELECT * FROM pg_stat_activity' | wc -l
59
  • Locks look high though:
postgres@linode18:~$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | sort | uniq -c | wc -l
1154
  • Indeed it seems something started causing locks to increase yesterday:

PostgreSQL locks week

  • And query length increasing since yesterday:

PostgreSQL query length week

  • The number of DSpace sessions is normal, hovering around 1,000…
  • Looking closer at the PostgreSQL activity log, I see the locks are all held by the dspaceCli user… which seem weird:
postgres@linode18:~$ psql -c "SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid WHERE application_name='dspaceCli';" | wc -l
1096
  • Now I’m wondering why there are no connections from dspaceApi or dspaceWeb. Could it be that our Tomcat JDBC pooling via JNDI isn’t working?
    • I see the same thing on DSpace Test hmmmm
    • The configuration in server.xml is correct, but it could be that when I changed to using the updated JDBC driver from pom.xml instead of dropping it in the Tomcat lib directory that something broke…
    • I downloaded the latest JDBC jar and put it in Tomcat’s lib directory on DSpace Test and after restarting Tomcat I can see connections from dspaceWeb and dspaceApi again
    • I will do the same on CGSpace and then revert the JDBC change in Ansible and DSpace pom.xml

2021-09-29

  • Export a list of ILRI subjects from CGSpace to validate against AGROVOC for Peter and Abenet:
localhost/dspace63= > \COPY (SELECT DISTINCT text_value FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 203) to /tmp/2021-09-29-ilri-subject.txt;
COPY 149
  • Then validate and format the matches:
$ ./ilri/agrovoc-lookup.py -i /tmp/2021-09-29-ilri-subject.txt -o /tmp/2021-09-29-ilri-subjects.csv -d
$ csvcut -c subject,'match type' /tmp/2021-09-29-ilri-subjects.csv | sed -e 's/match type/matched/' -e 's/\(alt\|pref\)Label/yes/' > /tmp/2021-09-29-ilri-subjects2.csv
  • I talked to Salem about depositing from MEL to CGSpace
    • He mentioned that the one issue is that when you deposit to a workflow you don’t get a Handle or any kind of identifier back!
    • We might have to come to some kind of agreement that they deposit items without going into the workflow but that we have some kind of edit role in MEL
    • He also said that they are looking into using the Research Organization Registry (RoR) in MEL, at least adding the ror_id and storing it
    • I need to propose this to Peter again and perhaps start aligning our affiliations closer (I could even do something like the country codes with a process that scans every day)
  • Talk to Moayad about OpenRXV
    • We decided that we’d keep harvesting all the Handles from the Altmetric prefix API, but then have a plugin to retrive DOI scores that we can run manually

2021-09-30

  • Look over 292 non-IWMI publications from Udana for inclusion into the Virtual library on water management collection on CGSpace
    • I did some minor cleanup to remove blank columns and run it through the csv-metadata-quality tool
    • I told him to add licenses and journal volume/issue and asked Abenet for input as well