From 416d2bc7a70f079ff685181405cbae4096da3a0c Mon Sep 17 00:00:00 2001 From: Alan Orth Date: Fri, 26 May 2023 17:04:18 +0300 Subject: [PATCH] Add notes for 2023-05-26 --- content/posts/2023-05.md | 76 +++++++++++++++++++++++++++++++++++++ docs/2023-05/index.html | 82 +++++++++++++++++++++++++++++++++++++++- 2 files changed, 157 insertions(+), 1 deletion(-) diff --git a/content/posts/2023-05.md b/content/posts/2023-05.md index fd66f844c..52697c83a 100644 --- a/content/posts/2023-05.md +++ b/content/posts/2023-05.md @@ -96,4 +96,80 @@ done < <(csvcut -c 1 ~/Downloads/2023-04-26\ CGIAR\ non-AGROVOC\ subjects.csv | - I exported CGSpace to check for missing Initiative mappings - Then I started a harvest on AReS +## 2023-05-23 + +- Help Francesca with an import of a journal article with a few hundred authors + - I used the DSpace 7 live import from PubMed +- I also noticed a bug in the CrossRef live import if you change the DOI field, so I [filed an issue](https://github.com/DSpace/DSpace/issues/8865) + +## 2023-05-25 + +- Meeting on output types +- Make a [pull request on DSpace to capture publisher during live import from Crossref](https://github.com/DSpace/DSpace/pull/8866) + +## 2023-05-26 + +- Make a [pull request on DSpace to update checkstyle](https://github.com/DSpace/DSpace/pull/8868) +- Make a [pull request on DSpace-angular to fix an incorrect i18n UI string](https://github.com/DSpace/dspace-angular/pull/2274) +- I'm experimenting with replacing old thumbnails + - In the past we used to upload thumbnails for journal covers, but those were low quality and look horrible now + - Using the provenance field I want to identify items with 1 bitstream of type gif or jpg, then extract the item IDs along with DOIs: + +```sql +\COPY (SELECT + text_value, + dspace_object_id +FROM + metadatavalue +WHERE + dspace_object_id IN ( + SELECT + dspace_object_id + FROM + metadatavalue + WHERE + metadata_field_id = 28 + AND place = 0 + AND (text_value LIKE '%No. of bitstreams: 1%' + AND text_value SIMILAR TO '%.(gif|jpg|jpeg)%')) + AND metadata_field_id = 220) TO /tmp/items-with-old-bitstreams.csv WITH CSV HEADER; +``` + +- I extract the DOIs and look them up on CrossRef to see which are CC-BY, then extract those: + +```console +$ csvcut -c text_value /tmp/items-with-old-bitstreams.csv | sed 1d > /tmp/dois.txt +$ ./ilri/crossref_doi_lookup.py -i /tmp/dois.txt -e fuuu@example.com -o /tmp/dois-resolved.csv +$ csvgrep -c license -m 'creativecommons' /tmp/dois-resolved.csv \ + | csvgrep -c license -m 'by-nc-nd' --invert-match \ + | csvcut -c doi \ + | sed '2,$s_^\(.*\)$_https://doi.org/\1_' \ + | sed 1d > /tmp/dois-for-cc-items-with-old-bitstreams.txt +``` + +- This results in 262 items that have DOIs that are CC-BY (but not ND) + - This is a good starting point, but misses some that had low-quality thumbnails uploaded after they were added (ie, there's no record of a bitstream in the provenance field) +- I ran the list through my Sci-Hub download script and filtered out a few that downloaded invalid PDFs (manually), then generated thumbnails for all of them: + +```console +$ ~/src/git/DSpace/ilri/get_scihub_pdfs.py -i /tmp/dois-for-cc-items-with-old-bitstreams.txt -o bitstreams.csv +$ chrt -b 0 vipsthumbnail *.pdf --export-profile srgb -s 600x600 -o './%s.pdf.jpg[Q=02,optimize_coding,strip]' +``` + +- Then I joined the CSVs on the DOI column, filtered out any that we didn't find PDFs for, and formatted the resulting CSV with an id, filename, and bundle column: + +```console +$ csvjoin -c doi bitstreams.csv /tmp/items-with-old-bitstreams.csv \ + | csvgrep -c filename --invert-match -r '^$' \ + | sed '1s/dspace_object_id/id/' \ + | csvcut -c id,filename \ + | sed -e '1s/^\(.*\)$/\1,bundle/' -e '2,$s/^\(.*\)$/\1.jpg__description:libvips thumbnail,THUMBNAIL/' > new-thumbnails.csv +``` + +- I did a dry run with `ilri/post_bitstreams.py` and it seems that most (all?) already have thumbnails from the last time I did a massive Sci-Hub check + - So relying on the provenance field is not very reliable it seems, and that was a waste of two hours... + - I did discover, while originally posting WebP thumbnails, that the format doesn't seem to be set correctly when uploading WebP via the REST API, but it does work when uploading via XMLUI—the format is set to Unknown + - POSTing a JPG to the THUMBNAIL bundle sets the format to JPEG... + - I am guessing that is a bug that I won't bother troubleshooting since the DSpace 6.x REST API is deprecated + diff --git a/docs/2023-05/index.html b/docs/2023-05/index.html index ec5a07bd9..5ff9d7253 100644 --- a/docs/2023-05/index.html +++ b/docs/2023-05/index.html @@ -56,7 +56,7 @@ Work on cleaning, proofing, and uploading twenty-seven records for IFPRI to CGSp "@type": "BlogPosting", "headline": "May, 2023", "url": "https://alanorth.github.io/cgspace-notes/2023-05/", - "wordCount": "609", + "wordCount": "1121", "datePublished": "2023-05-03T08:53:36+03:00", "dateModified": "2023-05-20T11:10:05+03:00", "author": { @@ -247,6 +247,86 @@ Work on cleaning, proofing, and uploading twenty-seven records for IFPRI to CGSp
  • I exported CGSpace to check for missing Initiative mappings
  • Then I started a harvest on AReS
    • +

    2023-05-23

    + +

    2023-05-25

    + +

    2023-05-26

    + +
    \COPY (SELECT
+    text_value,
+    dspace_object_id
+FROM
+    metadatavalue
+WHERE
+    dspace_object_id IN (
+        SELECT
+            dspace_object_id
+        FROM
+            metadatavalue
+        WHERE
+            metadata_field_id = 28
+            AND place = 0
+            AND (text_value LIKE '%No. of bitstreams: 1%'
+                AND text_value SIMILAR TO '%.(gif|jpg|jpeg)%'))
+    AND metadata_field_id = 220) TO /tmp/items-with-old-bitstreams.csv WITH CSV HEADER;
+
    +
    $ csvcut -c text_value /tmp/items-with-old-bitstreams.csv | sed 1d > /tmp/dois.txt
+$ ./ilri/crossref_doi_lookup.py -i /tmp/dois.txt -e fuuu@example.com -o /tmp/dois-resolved.csv
+$ csvgrep -c license -m 'creativecommons' /tmp/dois-resolved.csv \
+    | csvgrep -c license -m 'by-nc-nd' --invert-match \
+    | csvcut -c doi \
+    | sed '2,$s_^\(.*\)$_https://doi.org/\1_' \
+    | sed 1d > /tmp/dois-for-cc-items-with-old-bitstreams.txt
+
    +
    $ ~/src/git/DSpace/ilri/get_scihub_pdfs.py -i /tmp/dois-for-cc-items-with-old-bitstreams.txt -o bitstreams.csv
+$ chrt -b 0 vipsthumbnail *.pdf --export-profile srgb -s 600x600 -o './%s.pdf.jpg[Q=02,optimize_coding,strip]'
+
    +
    $ csvjoin -c doi bitstreams.csv /tmp/items-with-old-bitstreams.csv \
+    | csvgrep -c filename --invert-match -r '^$' \
+    | sed '1s/dspace_object_id/id/' \
+    | csvcut -c id,filename \
+    | sed -e '1s/^\(.*\)$/\1,bundle/' -e '2,$s/^\(.*\)$/\1.jpg__description:libvips thumbnail,THUMBNAIL/' > new-thumbnails.csv
+