cgspace-notes/content/posts/2021-01.md

title	date	author	categories
January, 2021	2021-01-03T10:13:54+02:00	Alan Orth	Notes

2021-01-03

• Peter notified me that some filters on AReS were broken again
  • It's the same issue with the field names getting .keyword appended to the end that I already filed an issue on OpenRXV about last month
  • I fixed the broken filters (careful to not edit any others, lest they break too!)
• Fix an issue with start page number for the DSpace REST API and statistics API in OpenRXV
  • The start page had been "1" in the UI, but in the backend they were doing some gymnastics to adjust to the zero-based offset/limit/page of the DSpace REST API and the statistics API
  • I adjusted it to default to 0 and added a note to the admin screen
  • I realized that this issue was actually causing the first page of 100 statistics to be missing...
  • For example, this item has 51 views on CGSpace, but 0 on AReS

• Start a re-index on AReS
  • First delete the old Elasticsearch temp index:

$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
# start indexing in AReS

• Then, the next morning when it's done, check the results of the harvesting, backup the current openrxv-items index, and clone the openrxv-items-temp index to openrxv-items:

$ curl -s 'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'
{
  "count" : 100278,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  }
}
$ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-2021-01-04
$ curl -XDELETE 'http://localhost:9200/openrxv-items'
$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items
$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
$ curl -XDELETE 'http://localhost:9200/openrxv-items-2021-01-04'

2021-01-04

• There is one item that appears twice in AReS: 10568/66839
  • If I use the Handle filter I see it twice... whereas other items don't appear twice
  • I filed a bug on OpenRXV: https://github.com/ilri/OpenRXV/issues/67
• Help Peter troubleshoot an issue with Altmetric badges on AReS
  • He generated a report of our repository from Altmetric and noticed that many were missing scores despite having scores on CGSpace item pages
  • AReS harvest Altmetric scores using the Handle prefix (10568) in batch, while CGSpace uses the DOI if it is found, and falls back to using the Handle
  • I think it's due to the fact that some items were never tweeted, so Altmetric never made the link between the DOI and the Handle
  • I did some tweets of five items and within an hour or so the DOI API link registers the associated Handle, and within an hour or so the Handle API link is live with the same score

2021-01-05

• A user sent me feedback about the dspace-statistics-api
  • He noticed that the indexer fails if there are unmigrated legacy records in Solr
  • I added a UUID filter to the queries in the indexer
• I generated a CSV of titles and Handles for 2019 and 2020 items for Peter to Tweet
  • We need to make sure that Altmetric has linked them all with their DOIs
  • I wrote a quick and dirty script called doi-to-handle.py to read the DOIs from a text file, query the database, and save the handles and titles to a CSV

$ ./doi-to-handle.py -db dspace -u dspace -p 'fuuu' -i /tmp/dois.txt -o /tmp/out.csv

• Help Udana export IWMI records from AReS
  • He wanted me to give him CSV export permissions on CGSpace, but I told him that this requires super admin so I'm not comfortable with it
• Import one item to CGSpace for Peter

2021-01-07

• Import twenty CABI book chapters for Abenet
• Udana and some editors from IWMI are still having problems editing metadata during the workflow step
  • It is the same issue Peter reported last month, that values he edits are not saved when the item gets archived
  • I added myself the the edit and approval steps of the collection on DSpace Test and asked Udana to submit an item there for me to test
• Atmire got back to me about the duplicate data in Solr
  • They want to arrange a time for us to do the stats processing so they can monitor it
  • I proposed that I set everything up with a fresh Solr snapshot from CGSpace and then let them start the stats process

2021-01-10

• Dominique from IWMI asked about API access to the IWMI collections
  • A partner of theirs called AMCOW is interested in harvesting their publications
  • I told her that they can use the REST API or OAI to get them from the IWMI Journal Articles collection:
    • CGSpace REST API: https://cgspace.cgiar.org/rest/collections/c2618391-184e-4091-8a93-280fdf01238b/items
    • CGSpace OAI API: https://cgspace.cgiar.org/oai/request?verb=ListRecords&metadataPrefix=oai_dc&set=col_10568_36185
• Udana submitted an item to the collection on DSpace Test that I discussed last week
  • I was able to take the task, add a new AGROVOC subject, approve the task, and commit it to archive
  • The final item had my new AGROVOC subject, so I don't see the issue
  • Perhaps the issue only occurs when we replace an existing field? Or only on IWMI fields? I don't know...
  • Also there is this warning that occurs in the DSpace log during editing (and many other operations):

2021-01-10 10:03:27,692 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY, SubjectType=ITEM, SubjectID=1e8fb96c-b994-4fe2-8f0c-0a98ab138be0, ObjectType=(Unknown), ObjectID=null, TimeStamp=1610269383279, dispatcher=1544803905, detail=[null], transactionID="TX35636856957739531161091194485578658698")

• I filed a bug on Atmire's issue tracker
• Peter asked me to move the CGIAR Gender Platform community to the top level of CGSpace, but I get an error when I use the community-filiator command:

$ dspace community-filiator --remove --parent=10568/66598 --child=10568/106605
Loading @mire database changes for module MQM
Changes have been processed
Exception: null
java.lang.UnsupportedOperationException
        at java.util.AbstractList.remove(AbstractList.java:161)
        at java.util.AbstractList$Itr.remove(AbstractList.java:374)
        at java.util.AbstractCollection.remove(AbstractCollection.java:293)
        at org.dspace.administer.CommunityFiliator.defiliate(CommunityFiliator.java:264)
        at org.dspace.administer.CommunityFiliator.main(CommunityFiliator.java:164)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)

• There is apparently a bug in DSpace 6.x that makes community-filiator not work
  • There is a patch for the as-of-yet unreleased DSpace 6.4 so I will try that
  • I tested the patch on DSpace Test and it worked, so I will do the same on CGSpace tomorrow
• Udana had asked about exporting IWMI's community on CGSpace, but we don't want to give him super admin permissions to do that
  • I suggested that he use AReS, but there are some fields missing because we don't harvest them all
  • I added a few more fields to the configuration and will start a fresh harvest.
• Start a re-index on AReS
  • First delete the old Elasticsearch temp index:

$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
# start indexing in AReS
... after ten hours
$ curl -s 'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'
{
  "count" : 100411,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  }
}
$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings?pretty" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
$ curl -XDELETE 'http://localhost:9200/openrxv-items'
$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items
$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'

• Looking over the last month of Solr stats I see a familiar bot that should have been marked as a bot months ago:

Mozilla/5.0 (compatible; +centuryb.o.t9[at]gmail.com)

• There are 51,961 hits from this bot on 64.62.202.71 and 64.62.202.73
  • Ah! Actually I added the bot pattern to the Tomcat Crawler Session Manager Valve, which mitigated the abuse of Tomcat sessions:

$ cat log/dspace.log.2020-12-2* | grep -E 'session_id=[A-Z0-9]{32}:ip_addr=64.62.202.71' | sort | uniq | wc -l
0

• So now I should really add it to the DSpace spider agent list so it doesn't create Solr hits
  • I added it to the "ilri" lists of spider agent patterns
• I purged the existing hits using my check-spider-ip-hits.sh script:

$ ./check-spider-ip-hits.sh -d -f /tmp/ips -s http://localhost:8081/solr -s statistics -p

2021-01-11

• The AReS indexing finished this morning and I moved the openrxv-items-temp core to openrxv-items (see above)
  • I sorted the explorer results by Altmetric attention score and I see a few new ones on the top so I think the recent tweeting of Handles by Peter and myself worked
• I deployed the community-filiator fix on CGSpace and moved the Gender Platform community to the top level of CGSpace:

$ dspace community-filiator --remove --parent=10568/66598 --child=10568/106605

2021-01-12

• IWMI is really pressuring us to have a periodic CSV export of their community
  • I decided to write a systemd timer to use dspace metadata-export every week, and made an nginx alias to make it available publicly
  • It is part of the Ansible infrastructure scripts that I use to provision the servers
• I wrote to Atmire to tell them to try their CUA duplicates processor on DSpace Test whenever they get a chance this week
  • I verified that there were indeed duplicate metadata values in the userAgent_ngram and userAgent_search fields, even in the first few results I saw in Solr
  • For reference, the UID of the record I saw with duplicate metadata was: 50e52a06-ffb7-4597-8d92-1c608cc71c98

2021-01-13

• I filed an issue on cg-core asking about how to handle series name / number
  • Currently the values are in format "series name; series number" in the dc.relation.ispartofseries field, but Peter wants to be able to separate them
• Start working on CG Core v2 migration for DSpace 6, using my work from last year on DSpace 5

2021-01-14

• More work on the CG Core v2 migration for DSpace 6
• Publish v1.4.1 of the DSpace Statistics API based on feedback from the community
  • This includes the fix for limiting the Solr query to UUIDs