CGSpace Notes

2023-07-01

• Export CGSpace to check for missing Initiative collection mappings
• Start harvesting on AReS

2023-07-02

• Minor edits to the crossref_doi_lookup.py script while running some checks from 22,000 CGSpace DOIs

2023-07-03

• I analyzed the licenses declared by Crossref and found with high confidence that ~400 of ours were incorrect
  • I took the more accurate ones from Crossref and updated the items on CGSpace
  • I took a few hundred ISBNs as well for where we were missing them
  • I also tagged ~4,700 items with missing licenses as “Copyrighted; all rights reserved” based on their Crossref license status being TDM, mostly from Elsevier, Wiley, and Springer
  • Checking a dozen or so manually, I confirmed that if Crossref only has a TDM license then it’s usually copyrighted (could still be open access, but we can’t tell via Crossref)
  • I would be curious to write a script to check the Unpaywall API for open access status…
  • In the past I found that their license status was not very accurate, but the open access status might be more reliable
• More minor work on the DSpace 7 item views
  • I learned some new Angular template syntax
  • I created a custom component to show Creative Commons licenses on the simple item page
  • I also decided that I don’t like the Impact Area icons as a component because they don’t have any visual meaning

2023-07-04

• Focus group meeting with CGSpace partners about DSpace 7
• I added a themed file selection component to the CGSpace theme
  • It displays the bistream description instead of the file name, just like we did in DSpace 6 XMLUI
• I added a custom component to show share icons

2023-07-05

• I spent some time trying to update OpenRXV from Angular 9 to 10 to 11 to 12 to 13
  • Most things work but there are some minor bugs it seems
• Mishell from CIP emailed me to say she was having problems approving an item on CGSpace
  • Looking at PostgreSQL I saw there were a dozen or so locks that were several hours and even over one day old so I killed those processes and told her to try again

2023-07-06

• Types meeting
• I wrote a Python script to check Unpaywall for some information about DOIs

2023-07-7

• Continue exploring Unpaywall data for some of our DOIs
  • In the past I’ve found their licensing information to not be very reliable (preferring Crossref), but I think their open access status is more reliable, especially when the provider is listed as being the publisher
  • Even so, sometimes the version can be “acceptedVersion”, which is presumably the author’s version, as opposed to the “publishedVersion”, which means it’s available as open access on the publisher’s website
  • I did some quality assurance and found ~100 that were marked as Limited Access, but should have been Open Access, and fixed a handful of licenses
• Delete duplicate metadata as describe in my DSpace issue from last year: https://github.com/DSpace/DSpace/issues/8253
• Start working on some statistics on AGROVOC usage for my presenation next week
  • I used the following SQL query to dump values from all subject fields and lower case them:

localhost/dspacetest= ☘ \COPY (SELECT DISTINCT(lower(text_value)) AS "subject" FROM metadatavalue WHERE dspace_object_id in (SELECT dspace_object_id FROM item) AND metadata_field_id IN (187, 120, 210, 122, 215, 127, 208, 124, 128, 123, 125, 135, 203, 236, 238, 119)) to /tmp/2023-07-07-cgspace-subjects.csv WITH CSV HEADER;
COPY 26443
Time: 2564.851 ms (00:02.565)

• Then I extracted the subjects and looked them up against AGROVOC:

$ csvcut -c subject /tmp/2023-07-07-cgspace-subjects.csv | sed '1d' > /tmp/2023-07-07-cgspace-subjects.txt
$ ./ilri/agrovoc_lookup.py -i /tmp/2023-07-07-cgspace-subjects.txt -o /tmp/2023-07-07-cgspace-subjects-results.csv

• I did some more tests with Angular 13 on OpenRXV and found out why the repository type dropdown wasn’t working
  • It was because of a missing 1-line JSON file in the data directory, which is runtime data, not code
  • I copied the data directory from the production serve and rebuild and the site is working well now
  • I did a full harvest with plugins and it worked!
  • So it seems Angular 13.4.0 will work, yay

2023-07-08

• Export CGSpace to check for missing Initiative collection mappings
  • Start a harvest on AReS
• The AGROVOC lookup finished, so I checked the number of matches:

$ csvgrep -c 'match type' -r '^.+$' ~/Downloads/2023-07-07-cgspace-subjects-resolved.csv | sed 1d | wc -l
12528

• So that’s 12,528 out of 26,443 unique terms (47.3%)
• I did a LOT of work on the OpenRXV frontend build dependencies to bring more in line with Angular 13