alanorth/cgspace-notes
1
0
Fork 0
mirror of https://github.com/alanorth/cgspace-notes.git synced 2025-01-27 05:49:12 +01:00
Code Issues Projects Releases Wiki Activity

Add notes for 2023-02-21

Browse Source
This commit is contained in:
Alan Orth
2023-02-21 20:46:53 +03:00
parent a667e6986e
commit 47f2c6c17f
31 changed files with 331 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files

176
docs/2023-02/index.html
View File

@ -18,7 +18,7 @@ I want to try to expand my use of their data to journals, publishers, volumes, i
<meta property="og:type" content="article" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2023-02/" />
<meta property="article:published_time" content="2023-02-01T10:57:36+03:00" />
<meta property="article:modified_time" content="2023-02-14T23:13:35+03:00" />
<meta property="article:modified_time" content="2023-02-15T19:47:13+03:00" />
@ -42,9 +42,9 @@ I want to try to expand my use of their data to journals, publishers, volumes, i
"@type": "BlogPosting",
"headline": "February, 2023",
"url": "https://alanorth.github.io/cgspace-notes/2023-02/",
"wordCount": "1245",
"wordCount": "2333",
"datePublished": "2023-02-01T10:57:36+03:00",
"dateModified": "2023-02-14T23:13:35+03:00",
"dateModified": "2023-02-15T19:47:13+03:00",
"author": {
"@type": "Person",
"name": "Alan Orth"
@ -340,7 +340,175 @@ I want to try to expand my use of their data to journals, publishers, volumes, i
</span></span></code></pre></div><ul>
<li>I think that particular error is because I applied the <a href="https://github.com/DSpace/DSpace/pull/1792">indexes in this unmerged DSpace 6 patch</a>, so I don&rsquo;t need to report this as an error in DSpace 7</li>
</ul>
<!-- raw HTML omitted -->
<h2 id="2023-02-16">2023-02-16</h2>
<ul>
<li>I found a suspicious number of PostgreSQL locks on CGSpace and decided to investigate:</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ psql -c <span style="color:#e6db74">&#39;SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;&#39;</span> | grep -o -E <span style="color:#e6db74">&#39;(dspaceWeb|dspaceApi|dspaceCli)&#39;</span> | sort | uniq -c
</span></span><span style="display:flex;"><span> 44 dspaceApi
</span></span><span style="display:flex;"><span> 372 dspaceCli
</span></span><span style="display:flex;"><span> 446 dspaceWeb
</span></span></code></pre></div><ul>
<li>This started happening yesterday and I killed a few locks that were several hours old after inspecting the <code>locks-age.sql</code> output</li>
<li>I also checked the <code>locks.sql</code> output, which helpfully lists the blocked PID and the blocking PID, to find one blocking PID that was idle in transaction
<ul>
<li>I killed that process and then all other locks were instantly processed</li>
</ul>
</li>
<li>I filed <a href="https://github.com/DSpace/dspace-angular/issues/2103">a GitHub issue</a> on dspace-angular requesting the item view to use the bitstream description instead of the file name if present</li>
<li>Weekly CG Core types meeting
<ul>
<li>I need to go through the actions and remove those items that are only for CGSpace internal use, ie:
<ul>
<li>CD-ROM</li>
<li>Manuscript-unpublished</li>
<li>Photo Report</li>
<li>Questionnaire</li>
<li>Wiki</li>
</ul>
</li>
</ul>
</li>
<li>Weekly CGIAR Repository Working Group meeting</li>
<li>I did some experiments with Crossref dates for about 20,000 DOIs in CGSpace using my <code>crossref-doi-lookup.py</code> script</li>
<li>Some things I noted from reading the <a href="https://github.com/CrossRef/rest-api-doc/blob/master/api_format.md">Crossref API docs</a> and inspecting the records for a few dozen DOIs manually:
<ul>
<li><code>[&quot;created&quot;][&quot;date-parts&quot;]</code> → Date on which the DOI was first registered (not useful for us)</li>
<li><code>[&quot;published-print&quot;][&quot;date-parts&quot;]</code> → Date on which the work was published in print</li>
<li><code>[&quot;journal-issue&quot;][&quot;published-print&quot;][&quot;date-parts&quot;]</code> → When present, is 99% the same as the above</li>
<li><code>[&quot;published-online&quot;][&quot;date-parts&quot;]</code> → Date on which the work was published online</li>
<li><code>[&quot;journal-issue&quot;][&quot;published-online&quot;][&quot;date-parts&quot;]</code> → Much more rare, and only 50% the same as the above, so unreliable</li>
<li><code>[&quot;issued&quot;][&quot;date-parts&quot;]</code> → Earliest of published-print and published-online (not useful to us)</li>
</ul>
</li>
<li>After checking the DOIs manully I decided that when the <code>published-print</code> date exists, it is usually more accurate than our issued dates
<ul>
<li>I set 12,300 issue dates to those from Crossref</li>
</ul>
</li>
<li>I also decided that, when <code>published-online</code> exists, it is usually accurate when I check the publisher page (we don&rsquo;t have many online dates to compare)
<ul>
<li>I set the available date for ~7,000 items to the published-online date as long as:
<ul>
<li>There was no <code>dcterms.available</code> date already</li>
<li>It was different than the issued date, because for now I only want online dates that are different, in case this is an online only journal in which case that can be the issue date&hellip; maybe I&rsquo;ll re-visit that later</li>
</ul>
</li>
</ul>
</li>
</ul>
<h2 id="2023-02-17">2023-02-17</h2>
<ul>
<li>It seems some (all?) of the changes I applied to dates last night didn&rsquo;t get saved&hellip;
<ul>
<li>I don&rsquo;t know what happened, so I will run them again after some investigation</li>
<li>I submitted the first batch of ~7,600 changes and it took twelve hours!</li>
<li>I almost cancelled it because after applying the changes there was a lock blocking everything for two hours, and it seemed to be stuck, but I kept checking it and saw that the <code>query_start</code> and <code>state_change</code> were being updated despite it being state &ldquo;idle in transaction&rdquo;:</li>
</ul>
</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ psql -c <span style="color:#e6db74">&#39;SELECT * FROM pg_stat_activity WHERE pid=1025176&#39;</span> | less -S
</span></span></code></pre></div><ul>
<li>I will apply the other changes in smaller batches&hellip;</li>
<li>Lately I&rsquo;ve noticed a lot of activity from the country code tagger curation task
<ul>
<li>Looking in the logs I see items being tagged that are very old and should have already been tagged years ago</li>
<li>Also, I see a ton of these errors whenever the task is updating an item:</li>
</ul>
</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>2023-02-17 08:01:00,252 INFO org.dspace.curate.Curator @ Curation task: countrycodetagger performed on: 10568/89020 with status: 0. Result: &#39;10568/89020: added 1 alpha2 country code(s)&#39;
</span></span><span style="display:flex;"><span>2023-02-17 08:01:00,467 ERROR com.atmire.versioning.ModificationLogger @ Error while writing item to versioning index: a0fe9d9a-6ac1-4b6a-8fcb-dae07a6bbf58 message:missing required field: epersonID
</span></span><span style="display:flex;"><span>org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: missing required field: epersonID
</span></span><span style="display:flex;"><span> at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
</span></span><span style="display:flex;"><span> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
</span></span><span style="display:flex;"><span> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
</span></span><span style="display:flex;"><span> at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
</span></span><span style="display:flex;"><span> at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
</span></span><span style="display:flex;"><span> at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
</span></span><span style="display:flex;"><span> at com.atmire.versioning.ModificationLogger.indexItem(ModificationLogger.java:263)
</span></span><span style="display:flex;"><span> at com.atmire.versioning.ModificationConsumer.end(ModificationConsumer.java:134)
</span></span><span style="display:flex;"><span> at org.dspace.event.BasicDispatcher.dispatch(BasicDispatcher.java:157)
</span></span><span style="display:flex;"><span> at org.dspace.core.Context.dispatchEvents(Context.java:455)
</span></span><span style="display:flex;"><span> at org.dspace.curate.Curator.visit(Curator.java:541)
</span></span><span style="display:flex;"><span> at org.dspace.curate.Curator$TaskRunner.run(Curator.java:568)
</span></span><span style="display:flex;"><span> at org.dspace.curate.Curator.doCollection(Curator.java:515)
</span></span><span style="display:flex;"><span> at org.dspace.curate.Curator.doCommunity(Curator.java:487)
</span></span><span style="display:flex;"><span> at org.dspace.curate.Curator.doSite(Curator.java:451)
</span></span><span style="display:flex;"><span> at org.dspace.curate.Curator.curate(Curator.java:269)
</span></span><span style="display:flex;"><span> at org.dspace.curate.Curator.curate(Curator.java:203)
</span></span><span style="display:flex;"><span> at org.dspace.curate.CurationCli.main(CurationCli.java:220)
</span></span><span style="display:flex;"><span> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
</span></span><span style="display:flex;"><span> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
</span></span><span style="display:flex;"><span> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
</span></span><span style="display:flex;"><span> at java.lang.reflect.Method.invoke(Method.java:498)
</span></span><span style="display:flex;"><span> at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
</span></span><span style="display:flex;"><span> at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
</span></span></code></pre></div><ul>
<li>This must be related&hellip;</li>
</ul>
<h2 id="2023-02-18">2023-02-18</h2>
<ul>
<li>I realized why the country-code-tagger was tagging everything: I had overridden the <code>force</code> parameter last week!</li>
<li>Start a harvest on AReS</li>
</ul>
<h2 id="2023-02-20">2023-02-20</h2>
<ul>
<li>IWMI is concerned that some of their items with top Altmetric attention scores don&rsquo;t show up in the AReS Explorer
<ul>
<li>I looked into it for one and found that AReS is using the Handle, but Altmetric hasn&rsquo;t associated the Handle with the DOI</li>
</ul>
</li>
<li>Looking into country and region issues for the PRMS team
<ul>
<li>Last week they had some questions about some invalid countries that ended up being typos</li>
<li>I realized my cgspace-java-helpers country-code-tagger curation task is not using the latest version, so it was missing Türkiye</li>
<li>I compiled the new version and ran it manually, but I have to upload a new version to Maven Central and then update the dependency in <code>dspace/modules/additions/pom.xml</code> ughhhhhh</li>
<li>I tagged version 6.2 with the change for Türkiye and uploaded to to Maven Central with <code>mvn clean deploy</code></li>
</ul>
</li>
<li>I&rsquo;m having second thoughts about switching to UN M.49 for countries because there are just too many tradeoffs
<ul>
<li>I want to find a way to keep our existing list, and codify some rules for it</li>
<li>There are several discussions related to the shortcomings of ISO themselves and the iso-codes project, for example:
<ul>
<li><a href="https://salsa.debian.org/iso-codes-team/iso-codes/-/issues/33">Inconsistency with articles in ISO-3166-1 English short names</a> (this one was filed by me two years ago!)</li>
<li><a href="https://salsa.debian.org/iso-codes-team/iso-codes/-/issues/44">ISO 3166-1: What&rsquo;s the policy for <code>common_name</code>?</a></li>
</ul>
</li>
<li>I almost want to say fuck it, let&rsquo;s just use iso-codes and tell everyone to deal with it, but make sure we handle ISO 3166-1 Alpha2 or probably Alpha3 in the future</li>
<li>Something like:
<ul>
<li>Prefer <code>common_name</code> if it exists</li>
<li>Prefer the shorter of <code>name</code> and <code>official name</code></li>
</ul>
</li>
</ul>
</li>
</ul>
<h2 id="2023-02-21">2023-02-21</h2>
<ul>
<li>Continue working on my <code>parse-iso-codes.py</code> script to parse the iso-codes JSON for ISO 3166-1
<ul>
<li>I also started a spreadsheet to track current CGSpace country names, proposed new names using the compromise above, and UN M.49 names</li>
<li>I proposed this to Peter but he wasn&rsquo;t happy because there are still some stupidly long and political names there</li>
</ul>
</li>
<li>I bumped the version of cgspace-java-helpers to 6.2-SNAPSHOT and pushed it to Maven Central because I can&rsquo;t figure out how to get non-snapshot releases to go there</li>
<li>Ouch, grunt 1.6.0 was released a few weeks ago, which relies on Node.js v16, thus breaking the Mirage 2 build in DSpace 6
<ul>
<li>I filed <a href="https://github.com/DSpace/DSpace/issues/8676">an issue in DSpace</a></li>
</ul>
</li>
<li>Help Moises from CIP troubleshoot harvesting issues on their WordPress site
<ul>
<li>I see 2,000 requests with the user agent &ldquo;RTB website BOT&rdquo; today and they are all HTTP 200</li>
</ul>
</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span># grep <span style="color:#e6db74">&#39;RTB website BOT&#39;</span> /var/log/nginx/rest.log | awk <span style="color:#e6db74">&#39;{print $9}&#39;</span> | sort | uniq -c | sort -h
</span></span><span style="display:flex;"><span> 2023 200
</span></span></code></pre></div><!-- raw HTML omitted -->