cgspace-notes/docs/2023-05/index.html
2023-10-04 09:24:33 +03:00

429 lines
22 KiB
HTML

<!DOCTYPE html>
<html lang="en" >
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<meta property="og:title" content="May, 2023" />
<meta property="og:description" content="2023-05-03
Alliance&rsquo;s TIP team emailed me to ask about issues authenticating on CGSpace
It seems their password expired, which is annoying
I continued looking at the CGSpace subjects for the FAO / AGROVOC exercise that I started last week
There are many of our subjects that would match if they added a &ldquo;-&rdquo; like &ldquo;high yielding varieties&rdquo; or used singular&hellip;
Also I found at least two spelling mistakes, for example &ldquo;decison support systems&rdquo;, which would match if it was spelled correctly
Work on cleaning, proofing, and uploading twenty-seven records for IFPRI to CGSpace
" />
<meta property="og:type" content="article" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2023-05/" />
<meta property="article:published_time" content="2023-05-03T08:53:36+03:00" />
<meta property="article:modified_time" content="2023-05-30T20:19:17+03:00" />
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="May, 2023"/>
<meta name="twitter:description" content="2023-05-03
Alliance&rsquo;s TIP team emailed me to ask about issues authenticating on CGSpace
It seems their password expired, which is annoying
I continued looking at the CGSpace subjects for the FAO / AGROVOC exercise that I started last week
There are many of our subjects that would match if they added a &ldquo;-&rdquo; like &ldquo;high yielding varieties&rdquo; or used singular&hellip;
Also I found at least two spelling mistakes, for example &ldquo;decison support systems&rdquo;, which would match if it was spelled correctly
Work on cleaning, proofing, and uploading twenty-seven records for IFPRI to CGSpace
"/>
<meta name="generator" content="Hugo 0.119.0">
<script type="application/ld+json">
{
"@context": "http://schema.org",
"@type": "BlogPosting",
"headline": "May, 2023",
"url": "https://alanorth.github.io/cgspace-notes/2023-05/",
"wordCount": "1282",
"datePublished": "2023-05-03T08:53:36+03:00",
"dateModified": "2023-05-30T20:19:17+03:00",
"author": {
"@type": "Person",
"name": "Alan Orth"
},
"keywords": "Notes"
}
</script>
<link rel="canonical" href="https://alanorth.github.io/cgspace-notes/2023-05/">
<title>May, 2023 | CGSpace Notes</title>
<!-- combined, minified CSS -->
<link href="https://alanorth.github.io/cgspace-notes/css/style.c6ba80bc50669557645abe05f86b73cc5af84408ed20f1551a267bc19ece8228.css" rel="stylesheet" integrity="sha256-xrqAvFBmlVdkWr4F&#43;GtzzFr4RAjtIPFVGiZ7wZ7Ogig=" crossorigin="anonymous">
<!-- minified Font Awesome for SVG icons -->
<script defer src="https://alanorth.github.io/cgspace-notes/js/fontawesome.min.f5072c55a0721857184db93a50561d7dc13975b4de2e19db7f81eb5f3fa57270.js" integrity="sha256-9QcsVaByGFcYTbk6UFYdfcE5dbTeLhnbf4HrXz&#43;lcnA=" crossorigin="anonymous"></script>
<!-- RSS 2.0 feed -->
</head>
<body>
<div class="blog-masthead">
<div class="container">
<nav class="nav blog-nav">
<a class="nav-link " href="https://alanorth.github.io/cgspace-notes/">Home</a>
</nav>
</div>
</div>
<header class="blog-header">
<div class="container">
<h1 class="blog-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/" rel="home">CGSpace Notes</a></h1>
<p class="lead blog-description" dir="auto">Documenting day-to-day work on the <a href="https://cgspace.cgiar.org">CGSpace</a> repository.</p>
</div>
</header>
<div class="container">
<div class="row">
<div class="col-sm-8 blog-main">
<article class="blog-post">
<header>
<h2 class="blog-post-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/2023-05/">May, 2023</a></h2>
<p class="blog-post-meta">
<time datetime="2023-05-03T08:53:36+03:00">Wed May 03, 2023</time>
in
<span class="fas fa-folder" aria-hidden="true"></span>&nbsp;<a href="/categories/notes/" rel="category tag">Notes</a>
</p>
</header>
<h2 id="2023-05-03">2023-05-03</h2>
<ul>
<li>Alliance&rsquo;s TIP team emailed me to ask about issues authenticating on CGSpace
<ul>
<li>It seems their password expired, which is annoying</li>
</ul>
</li>
<li>I continued looking at the CGSpace subjects for the FAO / AGROVOC exercise that I started last week
<ul>
<li>There are many of our subjects that would match if they added a &ldquo;-&rdquo; like &ldquo;high yielding varieties&rdquo; or used singular&hellip;</li>
<li>Also I found at least two spelling mistakes, for example &ldquo;decison support systems&rdquo;, which would match if it was spelled correctly</li>
</ul>
</li>
<li>Work on cleaning, proofing, and uploading twenty-seven records for IFPRI to CGSpace</li>
</ul>
<ul>
<li>I notice there are a few dozen locks from the <code>dspaceWeb</code> pool that are five days old on CGSpace so I killed them</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ psql &lt; locks-age.sql | grep <span style="color:#e6db74">&#34; days &#34;</span> | awk -F<span style="color:#e6db74">&#34;|&#34;</span> <span style="color:#e6db74">&#39;{print $10}&#39;</span> | sort -u | xargs kill
</span></span></code></pre></div><h2 id="2023-05-04">2023-05-04</h2>
<ul>
<li>Sync DSpace Test with CGSpace</li>
<li>I replaced one item&rsquo;s thumbnail with a WebP version and XMLUI displays it fine</li>
<li>I spent some time checking the CMYK issue with Arch&rsquo;s ImageMagick 7 and the Docker container and I think ImageMagick 7 just handles CMYK wrong&hellip;
<ul>
<li>libvips does it correctly automatically and looks closer to the PDF</li>
</ul>
</li>
<li>Meeting about CG Core types</li>
</ul>
<h2 id="2023-05-10">2023-05-10</h2>
<ul>
<li>Write a script to find the <code>metadata_field_id</code> values associated with the non-AGROVOC subjects I am working on for Sara
<ul>
<li>This is useful because we want to know who to contact for a definition</li>
<li>The script was:</li>
</ul>
</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span><span style="color:#66d9ef">while</span> read -r subject; <span style="color:#66d9ef">do</span>
</span></span><span style="display:flex;"><span> metadata_field_id<span style="color:#f92672">=</span><span style="color:#66d9ef">$(</span>psql -h localhost -U postgres -d dspacetest -qtAX <span style="color:#e6db74">&lt;&lt;SQL
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74"> SELECT DISTINCT(metadata_field_id) FROM metadatavalue WHERE LOWER(text_value)=&#39;$subject&#39;
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">SQL</span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">)</span>
</span></span><span style="display:flex;"><span> metadata_field_id<span style="color:#f92672">=</span><span style="color:#66d9ef">$(</span>echo $metadata_field_id | sed <span style="color:#e6db74">&#39;s/[[:space:]]/||/g&#39;</span><span style="color:#66d9ef">)</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> echo <span style="color:#e6db74">&#34;</span>$subject<span style="color:#e6db74">,</span>$metadata_field_id<span style="color:#e6db74">&#34;</span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">done</span> &lt; &lt;<span style="color:#f92672">(</span>csvcut -c <span style="color:#ae81ff">1</span> ~/Downloads/2023-04-26<span style="color:#ae81ff">\ </span>CGIAR<span style="color:#ae81ff">\ </span>non-AGROVOC<span style="color:#ae81ff">\ </span>subjects.csv | sed 1d<span style="color:#f92672">)</span>
</span></span></code></pre></div><ul>
<li>I also realized that Bernard Bett didn&rsquo;t have any items on CGSpace tagged with his ORCID identifier, so I tagged 230!</li>
</ul>
<h2 id="2023-05-11">2023-05-11</h2>
<ul>
<li>CG Core meeting</li>
<li>Finalize looking at the CGSpace non-AGROVOC subjects for FAO</li>
</ul>
<h2 id="2023-05-12">2023-05-12</h2>
<ul>
<li>Export the Alliance community to do some country/region fixes
<ul>
<li>I also sent Maria and Francesca the export because they want to add more regions and subregions</li>
</ul>
</li>
<li>Export the entire CGSpace to check for missing Initiative collection mappings
<ul>
<li>I also adding missing regions</li>
</ul>
</li>
</ul>
<h2 id="2023-05-16">2023-05-16</h2>
<ul>
<li>I finally cleaned up and published my latest evaluation of <a href="https://alanorth.github.io/improved-dspace-thumbnails/evaluating-jpeg-webp-avif.html">JPEG, WebP, and AVIF</a>
<ul>
<li>I <a href="https://github.com/DSpace/DSpace/issues/8849">filed an issue on DSpace</a> to track this</li>
</ul>
</li>
</ul>
<h2 id="2023-05-17">2023-05-17</h2>
<ul>
<li>Re-sync CGSpace to DSpace 7 Test</li>
<li>I came up with a naive patch to use WebP instead of JPEG in the DSpace ImageMagick filter, and it works, but doesn&rsquo;t replace existing JPEGs&hellip; hmmm
<ul>
<li>Also, it does PDF to WebP to WebP haha</li>
</ul>
</li>
</ul>
<h2 id="2023-05-18">2023-05-18</h2>
<ul>
<li>I created a <a href="https://github.com/DSpace/DSpace/pull/8850">pull request</a> to improve some minor documentation, typo, and logic issues in the DSpace ImageMagick thumbnail filters</li>
<li>I realized that there is a quick win to the generation loss issue with ImageMagickThumbnailFilter
<ul>
<li>We can use ImageMagick&rsquo;s internal MIFF instead of JPEG when writing the intermediate image</li>
<li>According to the <a href="https://github.com/libvips/libvips/issues/571">libvips author PNG is very slow</a>!</li>
<li>I re-ran my <code>generation-loss.sh</code> script using MIFF and found that it had essentially the same results as PNG, which is about 1.1 points higher on the ssimulacra2 (v2.1) scoring scale</li>
<li>Also, according to my tests with the cosmo rusage.com utility, I see that MIFF is indeed much faster than PNG</li>
<li>I updated my pull request to add this quick win</li>
</ul>
</li>
<li>Weekly CG Core types meeting
<ul>
<li>Low attendance so I just kept working on the spreadsheet</li>
<li>We are at the stage of voting on definitions</li>
</ul>
</li>
</ul>
<h2 id="2023-05-19">2023-05-19</h2>
<ul>
<li>I ported a few of the minor ImageMagick Thumbnail Filter improvements to our <code>6_x-prod</code> branch</li>
</ul>
<h2 id="2023-05-20">2023-05-20</h2>
<ul>
<li>I deployed the latest thumbnail changes on CGSpace, ran all updates, and rebooted it</li>
<li>I exported CGSpace to check for missing Initiative mappings</li>
<li>Then I started a harvest on AReS</li>
</ul>
<h2 id="2023-05-23">2023-05-23</h2>
<ul>
<li>Help Francesca with an import of a journal article with a few hundred authors
<ul>
<li>I used the DSpace 7 live import from PubMed</li>
</ul>
</li>
<li>I also noticed a bug in the CrossRef live import if you change the DOI field, so I <a href="https://github.com/DSpace/DSpace/issues/8865">filed an issue</a></li>
</ul>
<h2 id="2023-05-25">2023-05-25</h2>
<ul>
<li>Meeting on output types</li>
<li>Make a <a href="https://github.com/DSpace/DSpace/pull/8866">pull request on DSpace to capture publisher during live import from Crossref</a></li>
</ul>
<h2 id="2023-05-26">2023-05-26</h2>
<ul>
<li>Make a <a href="https://github.com/DSpace/DSpace/pull/8868">pull request on DSpace to update checkstyle</a></li>
<li>Make a <a href="https://github.com/DSpace/dspace-angular/pull/2274">pull request on DSpace-angular to fix an incorrect i18n UI string</a></li>
<li>I&rsquo;m experimenting with replacing old thumbnails
<ul>
<li>In the past we used to upload thumbnails for journal covers, but those were low quality and look horrible now</li>
<li>Using the provenance field I want to identify items with 1 bitstream of type gif or jpg, then extract the item IDs along with DOIs:</li>
</ul>
</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010">\</span><span style="color:#66d9ef">COPY</span> (<span style="color:#66d9ef">SELECT</span>
</span></span><span style="display:flex;"><span> text_value,
</span></span><span style="display:flex;"><span> dspace_object_id
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">FROM</span>
</span></span><span style="display:flex;"><span> metadatavalue
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">WHERE</span>
</span></span><span style="display:flex;"><span> dspace_object_id <span style="color:#66d9ef">IN</span> (
</span></span><span style="display:flex;"><span> <span style="color:#66d9ef">SELECT</span>
</span></span><span style="display:flex;"><span> dspace_object_id
</span></span><span style="display:flex;"><span> <span style="color:#66d9ef">FROM</span>
</span></span><span style="display:flex;"><span> metadatavalue
</span></span><span style="display:flex;"><span> <span style="color:#66d9ef">WHERE</span>
</span></span><span style="display:flex;"><span> metadata_field_id <span style="color:#f92672">=</span> <span style="color:#ae81ff">28</span>
</span></span><span style="display:flex;"><span> <span style="color:#66d9ef">AND</span> place <span style="color:#f92672">=</span> <span style="color:#ae81ff">0</span>
</span></span><span style="display:flex;"><span> <span style="color:#66d9ef">AND</span> (text_value <span style="color:#66d9ef">LIKE</span> <span style="color:#e6db74">&#39;%No. of bitstreams: 1%&#39;</span>
</span></span><span style="display:flex;"><span> <span style="color:#66d9ef">AND</span> text_value <span style="color:#66d9ef">SIMILAR</span> <span style="color:#66d9ef">TO</span> <span style="color:#e6db74">&#39;%.(gif|jpg|jpeg)%&#39;</span>))
</span></span><span style="display:flex;"><span> <span style="color:#66d9ef">AND</span> metadata_field_id <span style="color:#f92672">=</span> <span style="color:#ae81ff">220</span>) <span style="color:#66d9ef">TO</span> <span style="color:#f92672">/</span>tmp<span style="color:#f92672">/</span>items<span style="color:#f92672">-</span><span style="color:#66d9ef">with</span><span style="color:#f92672">-</span><span style="color:#66d9ef">old</span><span style="color:#f92672">-</span>bitstreams.csv <span style="color:#66d9ef">WITH</span> CSV HEADER;
</span></span></code></pre></div><ul>
<li>I extract the DOIs and look them up on CrossRef to see which are CC-BY, then extract those:</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ csvcut -c text_value /tmp/items-with-old-bitstreams.csv | sed 1d &gt; /tmp/dois.txt
</span></span><span style="display:flex;"><span>$ ./ilri/crossref_doi_lookup.py -i /tmp/dois.txt -e fuuu@example.com -o /tmp/dois-resolved.csv
</span></span><span style="display:flex;"><span>$ csvgrep -c license -m <span style="color:#e6db74">&#39;creativecommons&#39;</span> /tmp/dois-resolved.csv <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> | csvgrep -c license -m &#39;by-nc-nd&#39; --invert-match \
</span></span><span style="display:flex;"><span> | csvcut -c doi \
</span></span><span style="display:flex;"><span> | sed &#39;2,$s_^\(.*\)$_https://doi.org/\1_&#39; \
</span></span><span style="display:flex;"><span> | sed 1d &gt; /tmp/dois-for-cc-items-with-old-bitstreams.txt
</span></span></code></pre></div><ul>
<li>This results in 262 items that have DOIs that are CC-BY (but not ND)
<ul>
<li>This is a good starting point, but misses some that had low-quality thumbnails uploaded after they were added (ie, there&rsquo;s no record of a bitstream in the provenance field)</li>
</ul>
</li>
<li>I ran the list through my Sci-Hub download script and filtered out a few that downloaded invalid PDFs (manually), then generated thumbnails for all of them:</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ ~/src/git/DSpace/ilri/get_scihub_pdfs.py -i /tmp/dois-for-cc-items-with-old-bitstreams.txt -o bitstreams.csv
</span></span><span style="display:flex;"><span>$ chrt -b <span style="color:#ae81ff">0</span> vipsthumbnail *.pdf --export-profile srgb -s 600x600 -o <span style="color:#e6db74">&#39;./%s.pdf.jpg[Q=02,optimize_coding,strip]&#39;</span>
</span></span></code></pre></div><ul>
<li>Then I joined the CSVs on the DOI column, filtered out any that we didn&rsquo;t find PDFs for, and formatted the resulting CSV with an id, filename, and bundle column:</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ csvjoin -c doi bitstreams.csv /tmp/items-with-old-bitstreams.csv <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> | csvgrep -c filename --invert-match -r &#39;^$&#39; \
</span></span><span style="display:flex;"><span> | sed &#39;1s/dspace_object_id/id/&#39; \
</span></span><span style="display:flex;"><span> | csvcut -c id,filename \
</span></span><span style="display:flex;"><span> | sed -e &#39;1s/^\(.*\)$/\1,bundle/&#39; -e &#39;2,$s/^\(.*\)$/\1.jpg__description:libvips thumbnail,THUMBNAIL/&#39; &gt; new-thumbnails.csv
</span></span></code></pre></div><ul>
<li>I did a dry run with <code>ilri/post_bitstreams.py</code> and it seems that most (all?) already have thumbnails from the last time I did a massive Sci-Hub check
<ul>
<li>So relying on the provenance field is not very reliable it seems, and that was a waste of two hours&hellip;</li>
<li>I did discover, while originally posting WebP thumbnails, that the format doesn&rsquo;t seem to be set correctly when uploading WebP via the REST API, but it does work when uploading via XMLUI—the format is set to Unknown</li>
<li>POSTing a JPG to the THUMBNAIL bundle sets the format to JPEG&hellip;</li>
<li>I am guessing that is a bug that I won&rsquo;t bother troubleshooting since the DSpace 6.x REST API is deprecated</li>
</ul>
</li>
</ul>
<h2 id="2023-05-27">2023-05-27</h2>
<ul>
<li>Export CGSpace to check for missing Initiative collection mappings
<ul>
<li>Then I also ran the csv-metadata-quality tool on the Initiatives to do some easy fixes like country/region mapping and whitespace fixes</li>
</ul>
</li>
<li>Start a havest on AReS</li>
</ul>
<h2 id="2023-05-29">2023-05-29</h2>
<ul>
<li>Re-create my local PostgreSQL 14 container:</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ podman rm dspacedb14
</span></span><span style="display:flex;"><span>$ podman pull docker.io/postgres:14-alpine
</span></span><span style="display:flex;"><span>$ podman run --name dspacedb14 -v dspacedb14_data:/var/lib/postgresql/data -e POSTGRES_PASSWORD<span style="color:#f92672">=</span>postgres -p 5432:5432 -d docker.io/postgres:14-alpine -c shared_buffers<span style="color:#f92672">=</span>1024MB -c random_page_cost<span style="color:#f92672">=</span>1.1
</span></span></code></pre></div><ul>
<li>Export CGSpace again to do some major cleanups in OpenRefine
<ul>
<li>I found a few countries that are in the ISO 3166-1 and UN M.49 lists, but not in ours so I added them to the list in <code>input-forms.xml</code> and regenerated the controlled vocabularies for the CGSpace Submission Guidelines</li>
<li>There were a handful of issues with ISSNs, ISBNs, DOIs, access status, licenses, and missing CGIAR Trust Fund donors for Initiatives outputs</li>
<li>This was about 455 items</li>
</ul>
</li>
<li>Helping the Alliance web team understand the DSpace REST API for determining which collection an item belongs to</li>
</ul>
<!-- raw HTML omitted -->
</article>
</div> <!-- /.blog-main -->
<aside class="col-sm-3 ml-auto blog-sidebar">
<section class="sidebar-module">
<h4>Recent Posts</h4>
<ol class="list-unstyled">
<li><a href="/cgspace-notes/2023-10/">October, 2023</a></li>
<li><a href="/cgspace-notes/2023-09/">September, 2023</a></li>
<li><a href="/cgspace-notes/2023-08/">August, 2023</a></li>
<li><a href="/cgspace-notes/2023-07/">July, 2023</a></li>
<li><a href="/cgspace-notes/2023-06/">June, 2023</a></li>
</ol>
</section>
<section class="sidebar-module">
<h4>Links</h4>
<ol class="list-unstyled">
<li><a href="https://cgspace.cgiar.org">CGSpace</a></li>
<li><a href="https://dspacetest.cgiar.org">DSpace Test</a></li>
<li><a href="https://github.com/ilri/DSpace">CGSpace @ GitHub</a></li>
</ol>
</section>
</aside>
</div> <!-- /.row -->
</div> <!-- /.container -->
<footer class="blog-footer">
<p dir="auto">
Blog template created by <a href="https://twitter.com/mdo">@mdo</a>, ported to Hugo by <a href='https://twitter.com/mralanorth'>@mralanorth</a>.
</p>
<p>
<a href="#">Back to top</a>
</p>
</footer>
</body>
</html>