CGSpace Notes

Documenting day-to-day work on the CGSpace repository.

September, 2023

in  Notes

2023-09-02

  • Export CGSpace to check for missing Initiative collection mappings
  • Start a harvest on AReS

2023-09-03

  • I figured out how to use Altmetric and Dimensions badges in the DSpace Angular frontend
    • It still feels hacky, but using AfterViewInit, and importing the Altmetric embed.js in the component works
    • The style on mobile also needs work…

2023-09-06

  • Discussion with Marie about finalizing the output types list on GitHub
    • I did some review and cleanup in preparation for publishing the new list

2023-09-07

  • Export CGSpace to start doing a review of the metadata
  • First I will start by extracting all items with DOIs, along with some fields I can compare against Crossref:
$ csvgrep -c 'cg.identifier.doi[en_US]' -r 'doi.org' ~/Downloads/2023-09-07-cgspace.csv \
    | csvcut -c 'id,dc.title[en_US],dcterms.issued[en_US],dcterms.available[en_US],cg.issn[en_US],cg.isbn[en_US],cg.volume[en_US],cg.issue[en_US],cg.number[en_US],dcterms.extent[en_US],cg.identifier.doi[en_US],cg.reviewStatus[en_US],cg.isijournal[en_US],dcterms.license[en_US],dcterms.accessRights[en_US],dcterms.type[en_US],dc.identifier.uri[en_US]' \
    > /tmp/2023-09-07-cgspace-dois.csv
$ csvgrep -c 'cg.identifier.doi[en_US]' -r 'doi.org' ~/Downloads/2023-09-07-cgspace.csv | csvcut -c 'cg.identifier.doi[en_US]' | sed 1d > /tmp/2023-09-07-cgspace-dois.txt
  • Then I resolved the DOIs from Crossref:
$ ./ilri/crossref_doi_lookup.py -i /tmp/2023-09-07-cgspace-dois.txt -o /tmp/2023-09-07-cgspace-dois-results.csv -e a.orth@cgiar.org
  • A user emailed to ask about uploading a 180MB PDF to CGSpace
    • I used GhostScript to try reducing it using the screen, ebook and prepress presets:
$ gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/screen -dNOPAUSE -dQUIET -dBATCH -sOutputFile=primer-screen.pdf Primer\ \(digital\)_Climate-\ smart\ and\ regenerative\ agriculture\ in\ climate\ change\ adaptation.pdf 
$ gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/ebook -dNOPAUSE -dQUIET -dBATCH -sOutputFile=primer-ebook.pdf Primer\ \(digital\)_Climate-\ smart\ and\ regenerative\ agriculture\ in\ climate\ change\ adaptation.pdf
$ gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/prepress -dNOPAUSE -dQUIET -dBATCH -sOutputFile=primer-prepress.pdf Primer\ \(digital\)_Climate-\ smart\ and\ regenerative\ agriculture\ in\ climate\ change\ adaptation.pdf
  • The prepress one is 300DPI and looks visually identical to the original, so I proposed that we use that one

2023-09-08

  • I did a review of the metadata for our items with DOIs, comparing with data from Crossref
    • I spot checked a handful of issue / online dates and licenses, and saw that Crossref’s dates are always more accurate than ours when they differ
    • I also filled in some missing volumes, issues, ISSNs, and extents
    • This results in 14,000 changes to existing items, which will take several days to import unfortunately
    • After eight hours the first file is only about 2/3 finished… sigh
  • Meet with Peter to discuss changes to the DSpace 7 test
    • Minor updates to submission forms and some new ideas for the home page and item page
  • I figured out how to use a themed home page component and add a cards UI to our CGSpace theme

2023-09-09

  • I can’t believe that almost 18 hours later the first CSV import with 5,000 changes is not done…
  • Run all system updates on CGSpace and reboot it, as it had been two months since the last time

2023-09-10

  • Minor work on the DSpace 7 home page

2023-09-11

  • Export CGSpace to check for missing Initiative collection mappings
  • Start a harvest on AReS

2023-09-12

  • Minor work on DSpace 7 home page
  • Minor work on CG Core types
    • I published a new HTML version of the updated IPtypes and archived the current version as v2.0.0 so we can still reference it

2023-09-13

  • Stefano reminded me about the updated OAI MODS mappings on CGSpace so I re-applied them on DSpace Test and updated the OAI index so he could confirm
    • Now I’m ready to put it on CGSpace if he confirms
  • I created a basic theme for CIP on DSpace 7
    • While doing that I noticed that a bunch of CIP bitstreams didn’t have the latest 500px thumbnails so I re-ran filter-media on a handful of their collections
  • I had two occurrences of an OOM kill of the Tomcat 9 java process on DSpace 7 test tonight
    • Once while doing a Discovery index, the other while doing filter media

2023-09-15

  • Discuss issues with the Altmetric API with the Altmetric support team
    • Apparently we can use a different API, the Explorer API, since we already have access to the Explorer dashboard
  • I reduced the Solr heap size on DSpace 7 from 3GB to 2GB
    • Apparentlty I already did this from 4GB to 3GB a few months ago
    • The Solr admin interface was showing Solr taking ~1GB of RAM so I think this should be safe
  • Mark on DSpace Slack said he uses PM2’s --max-memory-restart so the processes restart when they hit the limit
    • Also, he said he had to reduce cache:serverSide:botCache:max from 1000 to 500 to cache less SSR pages in memory
  • I decided to try deploying DSpace 7 Test on a Hetzner server with 64GB RAM, 6 CPUs, and 2x512GB NVMe SSD

2023-09-16

  • Export CGSpace to check for missing Initiative collection mappings
  • Start a harvest on AReS
  • Configure the privacy policy page on DSpace 7 using a themed component with the text from our DSpace 6 site