CGSpace Notes

Documenting day-to-day work on the CGSpace repository.

November, 2023

in  Notes

2023-11-01

  • Work a bit on the ETL pipeline for the CGIAR Climate Change Synthesis
    • I improved the filtering and wrote some Python using pandas to merge my sources more reliably

2023-11-02

  • Export CGSpace to check missing Initiative collection mappings
  • Start a harvest on AReS
  • IFPRI contacted us about importing their Slideshare presentations to CGSpace
    • There are ~1,700 of them and date back to as early as 2008
    • I did a quick cleanup of the metadata export from Slideshare (including tagging with some AGROVOC in OpenRefine) and uploaded to DSpace Test

2023-11-03

  • A little bit of work on the CGIAR Climate Change Synthesis
  • Discuss some CGSpace migration plans with Leigh from IFPRI
    • For their Slideshare content we agreed:
      • Exclude private
      • Exclude deleted
      • Exclude non presentation types
      • Exclude duplicates within the collection for now until we can sort them out
    • That leaves about 1,500 items out of the 1,700
  • I did a duplicate check against CGSpace and found 44 items with 1.0 similarity so I removed those

2023-11-04

  • Export CGSpace to check for missing Initiative collection mappings
  • I ran through the list of potential duplicates on the IFPRI Slideshare presentations

2023-11-05

  • Work with Salem to migrate AReS to the new version

2023-11-07

  • DSpace 7 Test went down and there is very high load on the server
    • I saw very high load from Java but didn’t have time to check exactly what was wrong so I just rebooted the host
    • A few hours after restarting the system went down again, with very high load from Java again
    • I see lots of messages like this in the Tomcat log:
tomcat9[732]: [9955.662s][info   ][gc] GC(6291) Pause Full (G1 Compaction Pause) 4085M->4080M(4096M) 677.251ms
tomcat9[732]: [9955.662s][info   ][gc] GC(6290) Concurrent Mark Cycle 677.558ms
tomcat9[732]: [9955.666s][info   ][gc] GC(6292) To-space exhausted
  • I see some messages in dspace.log about heap space:
Caused by: java.lang.OutOfMemoryError: Java heap space
  • I will increase Tomcat’s heap from 4096m to 5120m
    • A few hours later it happened again, so I increased the heap from 5120m to 6144m
    • Not sure what’s going on today…
  • I tested moving the CGIAR Fund Council community to the CGIAR historic archive on DSpace Test:
$ dspace community-filiator -r -p 10568/83389 -c 10947/2516
$ dspace community-filiator -s -p 10947/2515 -c 10947/2516
$ dspace index-discovery -r 10947/2516
$ dspace index-discovery -r 10947/2515
$ dspace index-discovery -r 10568/83389
$ dspace index-discovery
  • I think this is the minimal we can do to avoid a full Discovery reindex which is very expensive
  • I helped Maria resize some massive PDFs for upload to CGSpace using GhostScript prepress mode as I had done before in September, 2023,

2023-11-08

  • DSpace 7 Test has very high load again and I see more Java heap space errors in the log
# grep -c 'Caused by: java.lang.OutOfMemoryError: Java heap space' /home/dspace7/log/dspace.log-2023-11-07 
35
# grep -c 'Caused by: java.lang.OutOfMemoryError: Java heap space' /home/dspace7/log/dspace.log
7
  • I don’t know what is happening… I will increase the heap size from 6144m to 7168m again…
  • I did some work on the value mappings in AReS
    • I wanted to test the import/export feature, and found that I could get a JSON and convert it to CSV for manipulation in OpenRefine
    • Importing duplicates records, so I deleted and re-created the index in Elasticsearch first
    • Then I started a new harvest on AReS to make sure the mappings are applied

2023-11-09

  • Ryan asked me for help uploading a large PDF to CGSpace
    • I tried my usual GhostScript preprint invocation and found the size decrease significantly, but some minor artifacts appeared in the images
    • Interestingly, the GhostScript docs mention that prepress doesn’t give the best results:

Please be aware that the /prepress setting does not indicate the highest quality conversion. Using any of these presets will involve altering the input, and as such may result in a PDF of poorer quality (compared to the input) than simply using the defaults. The ‘best’ quality (where best means closest to the original input) is obtained by not setting this parameter at all (or by using /default).

$ gs -sOutputFile=137166-default-dct.pdf -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dNOPAUSE -dBATCH -dPDFSETTINGS=/default -c "<< /ColorACSImageDict << /VSamples [ 1 1 1 1 ] /HSam
ples [ 1 1 1 1 ] /QFactor 0.08 /Blend 1 >> /ColorImageDownsampleType /Bicubic /ColorConversionStrategy /LeaveColorUnchanged >> setdistillerparams" -f 137166.pdf
  • This looks much better, and is still much smaller than the original
    • Also, I used pdfimages to extract all the images from the original and the one above and found:
$ du -sh images-*
886M    images-default-dct
1012M   images-original

2023-11-10

  • I finished checking the IFPRI Slideshare records and added some tagging of countries, regions, and CRPs and then uploaded them to CGSpace

2023-11-11

  • Salem fixed a bug on OpenRXV that was splitting country values by “,” before matching them with ISO countries
  • I exported CGSpace to check for missing Initiative collection mappings
  • Start a fresh harvest on AReS

2023-11-16

  • Discuss mapping ICARDA outputs from Initiatives to ICARDA collections on CGSpace
    • I added MEL’s CGSpace user to the administrator group of a handful of collections
    • I also did a batch mapping of 274 existing Initiative outputs from ICARDA to the relevant collections

2023-11-18

  • Export CGSpace to check for missing Initiative collection mappings
  • Start a harvest on AReS