--- title: "May, 2023" date: 2023-05-03T08:53:36+03:00 author: "Alan Orth" categories: ["Notes"] --- ## 2023-05-03 - Alliance's TIP team emailed me to ask about issues authenticating on CGSpace - It seems their password expired, which is annoying - I continued looking at the CGSpace subjects for the FAO / AGROVOC exercise that I started last week - There are many of our subjects that would match if they added a "-" like "high yielding varieties" or used singular... - Also I found at least two spelling mistakes, for example "decison support systems", which would match if it was spelled correctly - Work on cleaning, proofing, and uploading twenty-seven records for IFPRI to CGSpace - I notice there are a few dozen locks from the `dspaceWeb` pool that are five days old on CGSpace so I killed them ```console $ psql < locks-age.sql | grep " days " | awk -F"|" '{print $10}' | sort -u | xargs kill ``` ## 2023-05-04 - Sync DSpace Test with CGSpace - I replaced one item's thumbnail with a WebP version and XMLUI displays it fine - I spent some time checking the CMYK issue with Arch's ImageMagick 7 and the Docker container and I think ImageMagick 7 just handles CMYK wrong... - libvips does it correctly automatically and looks closer to the PDF - Meeting about CG Core types ## 2023-05-10 - Write a script to find the `metadata_field_id` values associated with the non-AGROVOC subjects I am working on for Sara - This is useful because we want to know who to contact for a definition - The script was: ```bash while read -r subject; do metadata_field_id=$(psql -h localhost -U postgres -d dspacetest -qtAX < /tmp/dois.txt $ ./ilri/crossref_doi_lookup.py -i /tmp/dois.txt -e fuuu@example.com -o /tmp/dois-resolved.csv $ csvgrep -c license -m 'creativecommons' /tmp/dois-resolved.csv \ | csvgrep -c license -m 'by-nc-nd' --invert-match \ | csvcut -c doi \ | sed '2,$s_^\(.*\)$_https://doi.org/\1_' \ | sed 1d > /tmp/dois-for-cc-items-with-old-bitstreams.txt ``` - This results in 262 items that have DOIs that are CC-BY (but not ND) - This is a good starting point, but misses some that had low-quality thumbnails uploaded after they were added (ie, there's no record of a bitstream in the provenance field) - I ran the list through my Sci-Hub download script and filtered out a few that downloaded invalid PDFs (manually), then generated thumbnails for all of them: ```console $ ~/src/git/DSpace/ilri/get_scihub_pdfs.py -i /tmp/dois-for-cc-items-with-old-bitstreams.txt -o bitstreams.csv $ chrt -b 0 vipsthumbnail *.pdf --export-profile srgb -s 600x600 -o './%s.pdf.jpg[Q=02,optimize_coding,strip]' ``` - Then I joined the CSVs on the DOI column, filtered out any that we didn't find PDFs for, and formatted the resulting CSV with an id, filename, and bundle column: ```console $ csvjoin -c doi bitstreams.csv /tmp/items-with-old-bitstreams.csv \ | csvgrep -c filename --invert-match -r '^$' \ | sed '1s/dspace_object_id/id/' \ | csvcut -c id,filename \ | sed -e '1s/^\(.*\)$/\1,bundle/' -e '2,$s/^\(.*\)$/\1.jpg__description:libvips thumbnail,THUMBNAIL/' > new-thumbnails.csv ``` - I did a dry run with `ilri/post_bitstreams.py` and it seems that most (all?) already have thumbnails from the last time I did a massive Sci-Hub check - So relying on the provenance field is not very reliable it seems, and that was a waste of two hours... - I did discover, while originally posting WebP thumbnails, that the format doesn't seem to be set correctly when uploading WebP via the REST API, but it does work when uploading via XMLUI—the format is set to Unknown - POSTing a JPG to the THUMBNAIL bundle sets the format to JPEG... - I am guessing that is a bug that I won't bother troubleshooting since the DSpace 6.x REST API is deprecated ## 2023-05-27 - Export CGSpace to check for missing Initiative collection mappings - Then I also ran the csv-metadata-quality tool on the Initiatives to do some easy fixes like country/region mapping and whitespace fixes - Start a havest on AReS ## 2023-05-29 - Re-create my local PostgreSQL 14 container: ```console $ podman rm dspacedb14 $ podman pull docker.io/postgres:14-alpine $ podman run --name dspacedb14 -v dspacedb14_data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d docker.io/postgres:14-alpine -c shared_buffers=1024MB -c random_page_cost=1.1 ``` - Export CGSpace again to do some major cleanups in OpenRefine - I found a few countries that are in the ISO 3166-1 and UN M.49 lists, but not in ours so I added them to the list in `input-forms.xml` and regenerated the controlled vocabularies for the CGSpace Submission Guidelines - There were a handful of issues with ISSNs, ISBNs, DOIs, access status, licenses, and missing CGIAR Trust Fund donors for Initiatives outputs - This was about 455 items - Helping the Alliance web team understand the DSpace REST API for determining which collection an item belongs to