alanorth/cgspace-notes
1
0
Fork 0
mirror of https://github.com/alanorth/cgspace-notes.git synced 2024-11-27 00:48:19 +01:00
Code Issues Projects Releases Wiki Activity
2569fa215b
cgspace-notes/docs/2022-01/index.html

435 lines
21 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE html>
<html lang="en" >
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<meta property="og:title" content="January, 2022" />
<meta property="og:description" content="2022-01-01
Start a full harvest on AReS
" />
<meta property="og:type" content="article" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2022-01/" />
<meta property="article:published_time" content="2022-01-01T15:20:54+02:00" />
<meta property="article:modified_time" content="2022-02-07T09:49:34+03:00" />
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="January, 2022"/>
<meta name="twitter:description" content="2022-01-01
Start a full harvest on AReS
"/>
<meta name="generator" content="Hugo 0.93.1" />
<script type="application/ld+json">
{
"@context": "http://schema.org",
"@type": "BlogPosting",
"headline": "January, 2022",
"url": "https://alanorth.github.io/cgspace-notes/2022-01/",
"wordCount": "1224",
"datePublished": "2022-01-01T15:20:54+02:00",
"dateModified": "2022-02-07T09:49:34+03:00",
"author": {
"@type": "Person",
"name": "Alan Orth"
},
"keywords": "Notes"
}
</script>
<link rel="canonical" href="https://alanorth.github.io/cgspace-notes/2022-01/">
<title>January, 2022 | CGSpace Notes</title>
<!-- combined, minified CSS -->
<link href="https://alanorth.github.io/cgspace-notes/css/style.beb8012edc08ba10be012f079d618dc243812267efe62e11f22fe49618f976a4.css" rel="stylesheet" integrity="sha256-vrgBLtwIuhC&#43;AS8HnWGNwkOBImfv5i4R8i/klhj5dqQ=" crossorigin="anonymous">
<!-- minified Font Awesome for SVG icons -->
<script defer src="https://alanorth.github.io/cgspace-notes/js/fontawesome.min.f5072c55a0721857184db93a50561d7dc13975b4de2e19db7f81eb5f3fa57270.js" integrity="sha256-9QcsVaByGFcYTbk6UFYdfcE5dbTeLhnbf4HrXz&#43;lcnA=" crossorigin="anonymous"></script>
<!-- RSS 2.0 feed -->
</head>
<body>
<div class="blog-masthead">
<div class="container">
<nav class="nav blog-nav">
<a class="nav-link " href="https://alanorth.github.io/cgspace-notes/">Home</a>
</nav>
</div>
</div>
<header class="blog-header">
<div class="container">
<h1 class="blog-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/" rel="home">CGSpace Notes</a></h1>
<p class="lead blog-description" dir="auto">Documenting day-to-day work on the <a href="https://cgspace.cgiar.org">CGSpace</a> repository.</p>
</div>
</header>
<div class="container">
<div class="row">
<div class="col-sm-8 blog-main">
<article class="blog-post">
<header>
<h2 class="blog-post-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/2022-01/">January, 2022</a></h2>
<p class="blog-post-meta">
<time datetime="2022-01-01T15:20:54+02:00">Sat Jan 01, 2022</time>
in
<span class="fas fa-folder" aria-hidden="true"></span>&nbsp;<a href="/cgspace-notes/categories/notes/" rel="category tag">Notes</a>
</p>
</header>
<h2 id="2022-01-01">2022-01-01</h2>
<ul>
<li>Start a full harvest on AReS</li>
</ul>
<h2 id="2022-01-06">2022-01-06</h2>
<ul>
<li>Add ORCID identifier for Chris Jones to CGSpace
<ul>
<li>Also tag eighty-eight of his items in CGSpace:</li>
</ul>
</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ cat 2022-01-06-add-orcids.csv
</span></span><span style="display:flex;"><span>dc.contributor.author,cg.creator.identifier
</span></span><span style="display:flex;"><span>&#34;Jones, Chris&#34;,&#34;Chris Jones: 0000-0001-9096-9728&#34;
</span></span><span style="display:flex;"><span>&#34;Jones, Christopher S.&#34;,&#34;Chris Jones: 0000-0001-9096-9728&#34;
</span></span><span style="display:flex;"><span>$ ./ilri/add-orcid-identifiers-csv.py -i 2022-01-06-add-orcids.csv -db dspace63 -u dspacetest -p <span style="color:#e6db74">&#39;dom@in34sniper&#39;</span>
</span></span></code></pre></div><h2 id="2022-01-09">2022-01-09</h2>
<ul>
<li>Validate and register CGSpace on <a href="https://www.openarchives.org/Register/ValidateSite?log=Z2V7WCT7">OpenArchives</a>
<ul>
<li>Last month IWMI colleagues were asking me to look into this, and after checking the OpenArchives mailing list it seems there was a problem on the server side</li>
<li>Now it has worked and the message is &ldquo;Successfully updated OAI registration database to status COMPLIANT.&rdquo;</li>
<li>I received an email (as the Admin contact on our OAI) that says:</li>
</ul>
</li>
</ul>
<blockquote>
<p>Your repository has been registered in the OAI database of conforming repositories.</p>
</blockquote>
<ul>
<li>Now I&rsquo;m taking a screenshot of the validation page for posterity, because the logs seem to go away after some time</li>
</ul>
<p><img src="/cgspace-notes/2022/01/openarchives-registration.png" alt="OpenArchives.org registration"></p>
<ul>
<li>I tried to re-build the Docker image for OpenRXV and got an error in the backend:</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>...
</span></span><span style="display:flex;"><span>&gt; openrxv-backend@0.0.1 build
</span></span><span style="display:flex;"><span>&gt; nest build
</span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010">
</span></span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010"></span>node_modules/@elastic/elasticsearch/api/types.d.ts:2454:13 - error TS2456: Type alias &#39;AggregationsAggregate&#39; circularly references itself.
</span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010">
</span></span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010"></span>2454 export type AggregationsAggregate = AggregationsSingleBucketAggregate | AggregationsAutoDateHistogramAggregate | AggregationsFiltersAggregate | AggregationsSignificantTermsAggregate&lt;any&gt; | AggregationsTermsAggregate&lt;any&gt; | AggregationsBucketAggregate | AggregationsCompositeBucketAggregate | AggregationsMultiBucketAggregate&lt;AggregationsBucket&gt; | AggregationsMatrixStatsAggregate | AggregationsKeyedValueAggregate | AggregationsMetricAggregate
</span></span><span style="display:flex;"><span> ~~~~~~~~~~~~~~~~~~~~~
</span></span><span style="display:flex;"><span>node_modules/@elastic/elasticsearch/api/types.d.ts:3209:13 - error TS2456: Type alias &#39;AggregationsSingleBucketAggregate&#39; circularly references itself.
</span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010">
</span></span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010"></span>3209 export type AggregationsSingleBucketAggregate = AggregationsSingleBucketAggregateKeys
</span></span><span style="display:flex;"><span> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
</span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010">
</span></span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010"></span>Found 2 error(s).
</span></span></code></pre></div><ul>
<li>Ah, it seems the code on the server was slightly out of date
<ul>
<li>I checked out the latest master branch and it built</li>
</ul>
</li>
</ul>
<h2 id="2022-01-12">2022-01-12</h2>
<ul>
<li>Fix some citation formatting issues in Gaia&rsquo;s <a href="https://dspacetest.cgiar.org/handle/10568/115230">eighteen CAS Green Cover publications on DSpace Test</a></li>
</ul>
<h2 id="2022-01-19">2022-01-19</h2>
<ul>
<li>Francesca was having issues with a submission on CGSpace this week
<ul>
<li>I checked and see a lot of locks in PostgreSQL:</li>
</ul>
</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ psql -c <span style="color:#e6db74">&#34;SELECT application_name FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid&#34;</span> | sort | uniq -c | sort -n
</span></span><span style="display:flex;"><span> 1
</span></span><span style="display:flex;"><span> 1 ------------------
</span></span><span style="display:flex;"><span> 1 (3506 rows)
</span></span><span style="display:flex;"><span> 1 application_name
</span></span><span style="display:flex;"><span> 9 psql
</span></span><span style="display:flex;"><span> 10
</span></span><span style="display:flex;"><span> 3487 dspaceWeb
</span></span></code></pre></div><ul>
<li>As before, I see messages from PostgreSQL about processes waiting for locks since I enabled the <code>log_lock_waits</code> setting last month:</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ grep -E <span style="color:#e6db74">&#39;^2022-01*&#39;</span> /var/log/postgresql/postgresql-10-main.log | grep -c <span style="color:#e6db74">&#39;still waiting for&#39;</span>
</span></span><span style="display:flex;"><span>12
</span></span></code></pre></div><ul>
<li>I set a system alert on DSpace and then restarted the server</li>
</ul>
<h2 id="2022-01-20">2022-01-20</h2>
<ul>
<li>Abenet gave me a thumbs up for Gaia&rsquo;s eighteen CAS Green Cover items from last month
<ul>
<li>I created a SimpleArchiveFormat bundle with SAFBuilder and then imported them on CGSpace:</li>
</ul>
</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ JAVA_OPTS<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;-Xmx1024m -Dfile.encoding=UTF-8&#34;</span> dspace import --add --eperson<span style="color:#f92672">=</span>aorth@mjanja.ch --source /tmp/SimpleArchiveFormat --mapfile<span style="color:#f92672">=</span>./2022-01-20-green-covers.map
</span></span></code></pre></div><h2 id="2022-01-21">2022-01-21</h2>
<ul>
<li>Start working on the rest of the ~980 CGIAR TAC and ICW documents from Gaia
<ul>
<li>I did some cleanups and standardization of author names</li>
<li>I also noticed that a few dozen items had no dates at all, so I checked the PDFs and found dates for them in the text</li>
<li>Otherwise all items have only a year, which is not great&hellip;</li>
</ul>
</li>
<li>Proof of concept upgrade of OpenRXV from Angular 9 to Angular 10
<ul>
<li>I did some basic tests and created a <a href="https://github.com/ilri/OpenRXV/pull/128">pull request</a></li>
</ul>
</li>
</ul>
<h2 id="2022-01-22">2022-01-22</h2>
<ul>
<li>Spend some time adding months to the CGIAR TAC and IWC records from Gaia
<ul>
<li>Most of the PDFs have only YYYY, so this is annoying&hellip;</li>
</ul>
</li>
</ul>
<h2 id="2022-01-23">2022-01-23</h2>
<ul>
<li>Finalize cleaning up the dates on the CGIAR TAC and IWC records from Gaia</li>
<li>Rebuild AReS and start a fresh harvest</li>
</ul>
<h2 id="2022-01-25">2022-01-25</h2>
<ul>
<li>Help Udana from IWMI answer some questions about licenses on their journal articles
<ul>
<li>I was surprised to see they have 921 total, but only about 200 have a <code>dcterms.license</code> field</li>
<li>I updated about thirty manually, but really Udana should do more&hellip;</li>
</ul>
</li>
<li>Normalize the metadata <code>text_lang</code> attributes on CGSpace database:</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>dspace=# SELECT DISTINCT text_lang, count(text_lang) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) GROUP BY text_lang ORDER BY count DESC;
</span></span><span style="display:flex;"><span> text_lang | count
</span></span><span style="display:flex;"><span>-----------+---------
</span></span><span style="display:flex;"><span> en_US | 2803350
</span></span><span style="display:flex;"><span> en | 6232
</span></span><span style="display:flex;"><span> | 3200
</span></span><span style="display:flex;"><span> fr | 2
</span></span><span style="display:flex;"><span> vn | 2
</span></span><span style="display:flex;"><span> 92 | 1
</span></span><span style="display:flex;"><span> sp | 1
</span></span><span style="display:flex;"><span> | 0
</span></span><span style="display:flex;"><span>(8 rows)
</span></span><span style="display:flex;"><span>dspace=# UPDATE metadatavalue SET text_lang=&#39;en_US&#39; WHERE dspace_object_id IN (SELECT uuid FROM item) AND text_lang IN (&#39;en&#39;, &#39;92&#39;, &#39;&#39;);
</span></span><span style="display:flex;"><span>UPDATE 9433
</span></span></code></pre></div><ul>
<li>Then export the WLE Journal Articles collection again so there are fewer columns to mess with</li>
</ul>
<h2 id="2022-01-26">2022-01-26</h2>
<ul>
<li>Send Gaia an example of the duplicate report for the first 200 TAC items to see what she thinks</li>
</ul>
<h2 id="2022-01-27">2022-01-27</h2>
<ul>
<li>Work on WLE&rsquo;s Journal Articles a bit more
<ul>
<li>I realized that ~130 items have DOIs in their citation, but no <code>cg.identifier.doi</code> field</li>
<li>I used this OpenRefine GREL to copy them:</li>
</ul>
</li>
</ul>
<pre tabindex="0"><code>cells[&#39;dcterms.bibliographicCitation[en_US]&#39;].value.split(&#34;doi: &#34;)[1]
</code></pre><ul>
<li>I also spent a bit of time cleaning up ILRI Journal Articles, but I notice that we don&rsquo;t put DOIs in the citation so it&rsquo;s not possible to fix items that are missing DOIs that way
<ul>
<li>And I cleaned up and normalized some licenses</li>
</ul>
</li>
<li>Francesca from Bioversity was having issues with a submission on CGSpace again
<ul>
<li>I looked at PostgreSQL and see an increasing number of locks:</li>
</ul>
</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ psql -c <span style="color:#e6db74">&#34;SELECT application_name FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid&#34;</span> | sort | uniq -c | sort -n
</span></span><span style="display:flex;"><span> 1
</span></span><span style="display:flex;"><span> 1 ------------------
</span></span><span style="display:flex;"><span> 1 (537 rows)
</span></span><span style="display:flex;"><span> 1 application_name
</span></span><span style="display:flex;"><span> 9 psql
</span></span><span style="display:flex;"><span> 51 dspaceApi
</span></span><span style="display:flex;"><span> 477 dspaceWeb
</span></span><span style="display:flex;"><span>$ grep -E <span style="color:#e6db74">&#39;^2022-01*&#39;</span> /var/log/postgresql/postgresql-10-main.log | grep -c <span style="color:#e6db74">&#39;still waiting for&#39;</span>
</span></span><span style="display:flex;"><span>3
</span></span></code></pre></div><ul>
<li>I set a system alert on CGSpace and then restarted Tomcat and PostgreSQL
<ul>
<li>The issue in Francesca&rsquo;s case was actually that someone had taken the task, not that PostgreSQL transactions were locked!</li>
</ul>
</li>
</ul>
<h2 id="2022-01-28">2022-01-28</h2>
<ul>
<li>Finalize the last ~100 WLE Journal Article items without licensese and DOIs
<ul>
<li>I did as many as I could, also updating http links to https for many journal links</li>
</ul>
</li>
<li>Federica Bottamedi contacted us from the system office to say that she took over for Vini (Abhilasha Vaid)
<ul>
<li>She created an account on CGSpace and now we need to see which workflows she should belong to</li>
</ul>
</li>
<li>Start a fresh harvesting on AReS</li>
<li>I adjusted the <code>check-duplicates.py</code> script to write the output to a CSV file including the id, both titles, both dates, and the handle link
<ul>
<li>I included the id because I will need a unique field to join the resulting list of non-duplicates with the original CSV where the rest of the metadata and filenames are</li>
<li>Since these items are not in DSpace yet, I generated simple numeric IDs in OpenRefine using this GREL transform: <code>row.index + 1</code></li>
<li>Then I ran <code>check-duplicates.py</code> on items 1–200 and sent the resulting CSV to Gaia</li>
</ul>
</li>
<li>Delete one duplicate item I saw in IITA&rsquo;s Journal Articles that was uploaded earlier in WLE
<ul>
<li>Also do some general cleanup on IITA&rsquo;s Journal Articles collection in OpenRefine</li>
</ul>
</li>
<li>Delete one duplicate item I saw in ILRI&rsquo;s Journal Articles collection
<ul>
<li>Also do some general cleanup on ILRI&rsquo;s Journal Articles collection in OpenRefine and csv-metadata-quality</li>
</ul>
</li>
</ul>
<h2 id="2022-01-29">2022-01-29</h2>
<ul>
<li>I did some more cleanup on the ILRI Journal Articles
<ul>
<li>I added missing journal titles for items that had ISSNs</li>
<li>Then I added pages for items that had them in the citation</li>
<li>First, I faceted the citation field based on whether or not the item had something like &ldquo;: 232-234&rdquo; present:</li>
</ul>
</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>value.contains(/:\s?\d+(-|–)\d+/)
</span></span></code></pre></div><ul>
<li>Then I faceted by blank on <code>dcterms.extent</code> and did a transform to extract the page information for over 1,000 items!</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>&#39;p. &#39; +
</span></span><span style="display:flex;"><span>cells[&#39;dcterms.bibliographicCitation[en_US]&#39;].value.match(/.*:\s?(\d+)(-|–)(\d+).*/)[0] +
</span></span><span style="display:flex;"><span>&#39;-&#39; +
</span></span><span style="display:flex;"><span>cells[&#39;dcterms.bibliographicCitation[en_US]&#39;].value.match(/.*:\s?(\d+)(-|–)(\d+).*/)[2]
</span></span></code></pre></div><ul>
<li>Then I did similar for <code>cg.volume</code> and <code>cg.issue</code>, also based on the citation, for example to extract the &ldquo;16&rdquo; from &ldquo;Journal of Blah 16(1)&rdquo;, where &ldquo;16&rdquo; is the second capture group in a zero-based match:</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>cells[&#39;dcterms.bibliographicCitation[en_US]&#39;].value.match(/.*( |;)(\d+)\((\d+)\).*/)[1]
</span></span></code></pre></div><ul>
<li>This was 3,000 items so I imported the changes on CGSpace 1,000 at a time&hellip;</li>
</ul>
<!-- raw HTML omitted -->
</article>
</div> <!-- /.blog-main -->
<aside class="col-sm-3 ml-auto blog-sidebar">
<section class="sidebar-module">
<h4>Recent Posts</h4>
<ol class="list-unstyled">
<li><a href="/cgspace-notes/2022-03/">March, 2022</a></li>
<li><a href="/cgspace-notes/2022-02/">February, 2022</a></li>
<li><a href="/cgspace-notes/2022-01/">January, 2022</a></li>
<li><a href="/cgspace-notes/2021-12/">December, 2021</a></li>
<li><a href="/cgspace-notes/2021-11/">November, 2021</a></li>
</ol>
</section>
<section class="sidebar-module">
<h4>Links</h4>
<ol class="list-unstyled">
<li><a href="https://cgspace.cgiar.org">CGSpace</a></li>
<li><a href="https://dspacetest.cgiar.org">DSpace Test</a></li>
<li><a href="https://github.com/ilri/DSpace">CGSpace @ GitHub</a></li>
</ol>
</section>
</aside>
</div> <!-- /.row -->
</div> <!-- /.container -->
<footer class="blog-footer">
<p dir="auto">
Blog template created by <a href="https://twitter.com/mdo">@mdo</a>, ported to Hugo by <a href='https://twitter.com/mralanorth'>@mralanorth</a>.
</p>
<p>
<a href="#">Back to top</a>
</p>
</footer>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink