<li>Alliance’s TIP team emailed me to ask about issues authenticating on CGSpace
<ul>
<li>It seems their password expired, which is annoying</li>
</ul>
</li>
<li>I continued looking at the CGSpace subjects for the FAO / AGROVOC exercise that I started last week
<ul>
<li>There are many of our subjects that would match if they added a “-” like “high yielding varieties” or used singular…</li>
<li>Also I found at least two spelling mistakes, for example “decison support systems”, which would match if it was spelled correctly</li>
</ul>
</li>
<li>Work on cleaning, proofing, and uploading twenty-seven records for IFPRI to CGSpace</li>
</ul>
<ul>
<li>I notice there are a few dozen locks from the <code>dspaceWeb</code> pool that are five days old on CGSpace so I killed them</li>
<li>I replaced one item’s thumbnail with a WebP version and XMLUI displays it fine</li>
<li>I spent some time checking the CMYK issue with Arch’s ImageMagick 7 and the Docker container and I think ImageMagick 7 just handles CMYK wrong…
<ul>
<li>libvips does it correctly automatically and looks closer to the PDF</li>
</span></span></span><spanstyle="display:flex;"><span><spanstyle="color:#e6db74"> SELECT DISTINCT(metadata_field_id) FROM metadatavalue WHERE LOWER(text_value)='$subject'
<li>I finally cleaned up and published my latest evaluation of <ahref="https://alanorth.github.io/improved-dspace-thumbnails/evaluating-jpeg-webp-avif.html">JPEG, WebP, and AVIF</a>
<ul>
<li>I <ahref="https://github.com/DSpace/DSpace/issues/8849">filed an issue on DSpace</a> to track this</li>
</ul>
</li>
</ul>
<h2id="2023-05-17">2023-05-17</h2>
<ul>
<li>Re-sync CGSpace to DSpace 7 Test</li>
<li>I came up with a naive patch to use WebP instead of JPEG in the DSpace ImageMagick filter, and it works, but doesn’t replace existing JPEGs… hmmm
<ul>
<li>Also, it does PDF to WebP to WebP haha</li>
</ul>
</li>
</ul>
<h2id="2023-05-18">2023-05-18</h2>
<ul>
<li>I created a <ahref="https://github.com/DSpace/DSpace/pull/8850">pull request</a> to improve some minor documentation, typo, and logic issues in the DSpace ImageMagick thumbnail filters</li>
<li>I realized that there is a quick win to the generation loss issue with ImageMagickThumbnailFilter
<ul>
<li>We can use ImageMagick’s internal MIFF instead of JPEG when writing the intermediate image</li>
<li>According to the <ahref="https://github.com/libvips/libvips/issues/571">libvips author PNG is very slow</a>!</li>
<li>I re-ran my <code>generation-loss.sh</code> script using MIFF and found that it had essentially the same results as PNG, which is about 1.1 points higher on the ssimulacra2 (v2.1) scoring scale</li>
<li>Also, according to my tests with the cosmo rusage.com utility, I see that MIFF is indeed much faster than PNG</li>
<li>I updated my pull request to add this quick win</li>
</ul>
</li>
<li>Weekly CG Core types meeting
<ul>
<li>Low attendance so I just kept working on the spreadsheet</li>
<li>We are at the stage of voting on definitions</li>
<li>Help Francesca with an import of a journal article with a few hundred authors
<ul>
<li>I used the DSpace 7 live import from PubMed</li>
</ul>
</li>
<li>I also noticed a bug in the CrossRef live import if you change the DOI field, so I <ahref="https://github.com/DSpace/DSpace/issues/8865">filed an issue</a></li>
</ul>
<h2id="2023-05-25">2023-05-25</h2>
<ul>
<li>Meeting on output types</li>
<li>Make a <ahref="https://github.com/DSpace/DSpace/pull/8866">pull request on DSpace to capture publisher during live import from Crossref</a></li>
</ul>
<h2id="2023-05-26">2023-05-26</h2>
<ul>
<li>Make a <ahref="https://github.com/DSpace/DSpace/pull/8868">pull request on DSpace to update checkstyle</a></li>
<li>Make a <ahref="https://github.com/DSpace/dspace-angular/pull/2274">pull request on DSpace-angular to fix an incorrect i18n UI string</a></li>
<li>I’m experimenting with replacing old thumbnails
<ul>
<li>In the past we used to upload thumbnails for journal covers, but those were low quality and look horrible now</li>
<li>Using the provenance field I want to identify items with 1 bitstream of type gif or jpg, then extract the item IDs along with DOIs:</li>
</span></span><spanstyle="display:flex;"><span><spanstyle="color:#66d9ef">AND</span> place <spanstyle="color:#f92672">=</span><spanstyle="color:#ae81ff">0</span>
</span></span><spanstyle="display:flex;"><span><spanstyle="color:#66d9ef">AND</span> (text_value <spanstyle="color:#66d9ef">LIKE</span><spanstyle="color:#e6db74">'%No. of bitstreams: 1%'</span>
</span></span><spanstyle="display:flex;"><span> | csvcut -c doi \
</span></span><spanstyle="display:flex;"><span> | sed '2,$s_^\(.*\)$_https://doi.org/\1_' \
</span></span><spanstyle="display:flex;"><span> | sed 1d > /tmp/dois-for-cc-items-with-old-bitstreams.txt
</span></span></code></pre></div><ul>
<li>This results in 262 items that have DOIs that are CC-BY (but not ND)
<ul>
<li>This is a good starting point, but misses some that had low-quality thumbnails uploaded after they were added (ie, there’s no record of a bitstream in the provenance field)</li>
</ul>
</li>
<li>I ran the list through my Sci-Hub download script and filtered out a few that downloaded invalid PDFs (manually), then generated thumbnails for all of them:</li>
<li>Then I joined the CSVs on the DOI column, filtered out any that we didn’t find PDFs for, and formatted the resulting CSV with an id, filename, and bundle column:</li>
</ul>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span>$ csvjoin -c doi bitstreams.csv /tmp/items-with-old-bitstreams.csv <spanstyle="color:#ae81ff">\
</span></span><spanstyle="display:flex;"><span> | sed -e '1s/^\(.*\)$/\1,bundle/' -e '2,$s/^\(.*\)$/\1.jpg__description:libvips thumbnail,THUMBNAIL/'> new-thumbnails.csv
</span></span></code></pre></div><ul>
<li>I did a dry run with <code>ilri/post_bitstreams.py</code> and it seems that most (all?) already have thumbnails from the last time I did a massive Sci-Hub check
<ul>
<li>So relying on the provenance field is not very reliable it seems, and that was a waste of two hours…</li>
<li>I did discover, while originally posting WebP thumbnails, that the format doesn’t seem to be set correctly when uploading WebP via the REST API, but it does work when uploading via XMLUI—the format is set to Unknown</li>
<li>POSTing a JPG to the THUMBNAIL bundle sets the format to JPEG…</li>
<li>I am guessing that is a bug that I won’t bother troubleshooting since the DSpace 6.x REST API is deprecated</li>
<li>Export CGSpace again to do some major cleanups in OpenRefine
<ul>
<li>I found a few countries that are in the ISO 3166-1 and UN M.49 lists, but not in ours so I added them to the list in <code>input-forms.xml</code> and regenerated the controlled vocabularies for the CGSpace Submission Guidelines</li>
<li>There were a handful of issues with ISSNs, ISBNs, DOIs, access status, licenses, and missing CGIAR Trust Fund donors for Initiatives outputs</li>
<li>This was about 455 items</li>
</ul>
</li>
<li>Helping the Alliance web team understand the DSpace REST API for determining which collection an item belongs to</li>
