mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2024-12-18 19:22:18 +01:00
342 lines
21 KiB
HTML
342 lines
21 KiB
HTML
<!DOCTYPE html>
|
|
<html lang="en" >
|
|
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
|
|
|
|
|
|
<meta property="og:title" content="July, 2023" />
|
|
<meta property="og:description" content="2023-07-01 Export CGSpace to check for missing Initiative collection mappings Start harvesting on AReS 2023-07-02 Minor edits to the crossref_doi_lookup.py script while running some checks from 22,000 CGSpace DOIs 2023-07-03 I analyzed the licenses declared by Crossref and found with high confidence that ~400 of ours were incorrect I took the more accurate ones from Crossref and updated the items on CGSpace I took a few hundred ISBNs as well for where we were missing them I also tagged ~4,700 items with missing licenses as “Copyrighted; all rights reserved” based on their Crossref license status being TDM, mostly from Elsevier, Wiley, and Springer Checking a dozen or so manually, I confirmed that if Crossref only has a TDM license then it’s usually copyrighted (could still be open access, but we can’t tell via Crossref) I would be curious to write a script to check the Unpaywall API for open access status… In the past I found that their license status was not very accurate, but the open access status might be more reliable More minor work on the DSpace 7 item views I learned some new Angular template syntax I created a custom component to show Creative Commons licenses on the simple item page I also decided that I don’t like the Impact Area icons as a component because they don’t have any visual meaning 2023-07-04 Focus group meeting with CGSpace partners about DSpace 7 I added a themed file selection component to the CGSpace theme It displays the bistream description instead of the file name, just like we did in DSpace 6 XMLUI I added a custom component to show share icons 2023-07-05 I spent some time trying to update OpenRXV from Angular 9 to 10 to 11 to 12 to 13 Most things work but there are some minor bugs it seems Mishell from CIP emailed me to say she was having problems approving an item on CGSpace Looking at PostgreSQL I saw there were a dozen or so locks that were several hours and even over one day old so I killed those processes and told her to try again 2023-07-06 Types meeting I wrote a Python script to check Unpaywall for some information about DOIs 2023-07-7 Continue exploring Unpaywall data for some of our DOIs In the past I’ve found their licensing information to not be very reliable (preferring Crossref), but I think their open access status is more reliable, especially when the provider is listed as being the publisher Even so, sometimes the version can be “acceptedVersion”, which is presumably the author’s version, as opposed to the “publishedVersion”, which means it’s available as open access on the publisher’s website I did some quality assurance and found ~100 that were marked as Limited Access, but should have been Open Access, and fixed a handful of licenses Delete duplicate metadata as describe in my DSpace issue from last year: https://github." />
|
|
<meta property="og:type" content="article" />
|
|
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2023-07/" />
|
|
<meta property="article:published_time" content="2023-07-01T17:14:36+03:00" />
|
|
<meta property="article:modified_time" content="2023-07-08T23:20:53+03:00" />
|
|
|
|
|
|
|
|
<meta name="twitter:card" content="summary"/>
|
|
<meta name="twitter:title" content="July, 2023"/>
|
|
<meta name="twitter:description" content="2023-07-01 Export CGSpace to check for missing Initiative collection mappings Start harvesting on AReS 2023-07-02 Minor edits to the crossref_doi_lookup.py script while running some checks from 22,000 CGSpace DOIs 2023-07-03 I analyzed the licenses declared by Crossref and found with high confidence that ~400 of ours were incorrect I took the more accurate ones from Crossref and updated the items on CGSpace I took a few hundred ISBNs as well for where we were missing them I also tagged ~4,700 items with missing licenses as “Copyrighted; all rights reserved” based on their Crossref license status being TDM, mostly from Elsevier, Wiley, and Springer Checking a dozen or so manually, I confirmed that if Crossref only has a TDM license then it’s usually copyrighted (could still be open access, but we can’t tell via Crossref) I would be curious to write a script to check the Unpaywall API for open access status… In the past I found that their license status was not very accurate, but the open access status might be more reliable More minor work on the DSpace 7 item views I learned some new Angular template syntax I created a custom component to show Creative Commons licenses on the simple item page I also decided that I don’t like the Impact Area icons as a component because they don’t have any visual meaning 2023-07-04 Focus group meeting with CGSpace partners about DSpace 7 I added a themed file selection component to the CGSpace theme It displays the bistream description instead of the file name, just like we did in DSpace 6 XMLUI I added a custom component to show share icons 2023-07-05 I spent some time trying to update OpenRXV from Angular 9 to 10 to 11 to 12 to 13 Most things work but there are some minor bugs it seems Mishell from CIP emailed me to say she was having problems approving an item on CGSpace Looking at PostgreSQL I saw there were a dozen or so locks that were several hours and even over one day old so I killed those processes and told her to try again 2023-07-06 Types meeting I wrote a Python script to check Unpaywall for some information about DOIs 2023-07-7 Continue exploring Unpaywall data for some of our DOIs In the past I’ve found their licensing information to not be very reliable (preferring Crossref), but I think their open access status is more reliable, especially when the provider is listed as being the publisher Even so, sometimes the version can be “acceptedVersion”, which is presumably the author’s version, as opposed to the “publishedVersion”, which means it’s available as open access on the publisher’s website I did some quality assurance and found ~100 that were marked as Limited Access, but should have been Open Access, and fixed a handful of licenses Delete duplicate metadata as describe in my DSpace issue from last year: https://github."/>
|
|
<meta name="generator" content="Hugo 0.112.3">
|
|
|
|
|
|
|
|
<script type="application/ld+json">
|
|
{
|
|
"@context": "http://schema.org",
|
|
"@type": "BlogPosting",
|
|
"headline": "July, 2023",
|
|
"url": "https://alanorth.github.io/cgspace-notes/2023-07/",
|
|
"wordCount": "1114",
|
|
"datePublished": "2023-07-01T17:14:36+03:00",
|
|
"dateModified": "2023-07-08T23:20:53+03:00",
|
|
"author": {
|
|
"@type": "Person",
|
|
"name": "Alan Orth"
|
|
},
|
|
"keywords": "Notes"
|
|
}
|
|
</script>
|
|
|
|
|
|
|
|
<link rel="canonical" href="https://alanorth.github.io/cgspace-notes/2023-07/">
|
|
|
|
<title>July, 2023 | CGSpace Notes</title>
|
|
|
|
|
|
<!-- combined, minified CSS -->
|
|
|
|
<link href="https://alanorth.github.io/cgspace-notes/css/style.c6ba80bc50669557645abe05f86b73cc5af84408ed20f1551a267bc19ece8228.css" rel="stylesheet" integrity="sha256-xrqAvFBmlVdkWr4F+GtzzFr4RAjtIPFVGiZ7wZ7Ogig=" crossorigin="anonymous">
|
|
|
|
|
|
<!-- minified Font Awesome for SVG icons -->
|
|
|
|
<script defer src="https://alanorth.github.io/cgspace-notes/js/fontawesome.min.f5072c55a0721857184db93a50561d7dc13975b4de2e19db7f81eb5f3fa57270.js" integrity="sha256-9QcsVaByGFcYTbk6UFYdfcE5dbTeLhnbf4HrXz+lcnA=" crossorigin="anonymous"></script>
|
|
|
|
<!-- RSS 2.0 feed -->
|
|
|
|
|
|
|
|
|
|
</head>
|
|
|
|
<body>
|
|
|
|
|
|
<div class="blog-masthead">
|
|
<div class="container">
|
|
<nav class="nav blog-nav">
|
|
<a class="nav-link " href="https://alanorth.github.io/cgspace-notes/">Home</a>
|
|
</nav>
|
|
</div>
|
|
</div>
|
|
|
|
|
|
|
|
|
|
<header class="blog-header">
|
|
<div class="container">
|
|
<h1 class="blog-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/" rel="home">CGSpace Notes</a></h1>
|
|
<p class="lead blog-description" dir="auto">Documenting day-to-day work on the <a href="https://cgspace.cgiar.org">CGSpace</a> repository.</p>
|
|
</div>
|
|
</header>
|
|
|
|
|
|
|
|
|
|
<div class="container">
|
|
<div class="row">
|
|
<div class="col-sm-8 blog-main">
|
|
|
|
|
|
|
|
|
|
<article class="blog-post">
|
|
<header>
|
|
<h2 class="blog-post-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/2023-07/">July, 2023</a></h2>
|
|
<p class="blog-post-meta">
|
|
<time datetime="2023-07-01T17:14:36+03:00">Sat Jul 01, 2023</time>
|
|
in
|
|
<span class="fas fa-folder" aria-hidden="true"></span> <a href="/categories/notes/" rel="category tag">Notes</a>
|
|
|
|
|
|
</p>
|
|
</header>
|
|
<h2 id="2023-07-01">2023-07-01</h2>
|
|
<ul>
|
|
<li>Export CGSpace to check for missing Initiative collection mappings</li>
|
|
<li>Start harvesting on AReS</li>
|
|
</ul>
|
|
<h2 id="2023-07-02">2023-07-02</h2>
|
|
<ul>
|
|
<li>Minor edits to the <code>crossref_doi_lookup.py</code> script while running some checks from 22,000 CGSpace DOIs</li>
|
|
</ul>
|
|
<h2 id="2023-07-03">2023-07-03</h2>
|
|
<ul>
|
|
<li>I analyzed the licenses declared by Crossref and found with high confidence that ~400 of ours were incorrect
|
|
<ul>
|
|
<li>I took the more accurate ones from Crossref and updated the items on CGSpace</li>
|
|
<li>I took a few hundred ISBNs as well for where we were missing them</li>
|
|
<li>I also tagged ~4,700 items with missing licenses as “Copyrighted; all rights reserved” based on their Crossref license status being TDM, mostly from Elsevier, Wiley, and Springer</li>
|
|
<li>Checking a dozen or so manually, I confirmed that if Crossref only has a TDM license then it’s usually copyrighted (could still be open access, but we can’t tell via Crossref)</li>
|
|
<li>I would be curious to write a script to check the Unpaywall API for open access status…</li>
|
|
<li>In the past I found that their <em>license</em> status was not very accurate, but the open access status might be more reliable</li>
|
|
</ul>
|
|
</li>
|
|
<li>More minor work on the DSpace 7 item views
|
|
<ul>
|
|
<li>I learned some new Angular template syntax</li>
|
|
<li>I created a custom component to show Creative Commons licenses on the simple item page</li>
|
|
<li>I also decided that I don’t like the Impact Area icons as a component because they don’t have any visual meaning</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<h2 id="2023-07-04">2023-07-04</h2>
|
|
<ul>
|
|
<li>Focus group meeting with CGSpace partners about DSpace 7</li>
|
|
<li>I added a themed file selection component to the CGSpace theme
|
|
<ul>
|
|
<li>It displays the bistream description instead of the file name, just like we did in DSpace 6 XMLUI</li>
|
|
</ul>
|
|
</li>
|
|
<li>I added a custom component to show share icons</li>
|
|
</ul>
|
|
<h2 id="2023-07-05">2023-07-05</h2>
|
|
<ul>
|
|
<li>I spent some time trying to update OpenRXV from Angular 9 to 10 to 11 to 12 to 13
|
|
<ul>
|
|
<li>Most things work but there are some minor bugs it seems</li>
|
|
</ul>
|
|
</li>
|
|
<li>Mishell from CIP emailed me to say she was having problems approving an item on CGSpace
|
|
<ul>
|
|
<li>Looking at PostgreSQL I saw there were a dozen or so locks that were several hours and even over one day old so I killed those processes and told her to try again</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<h2 id="2023-07-06">2023-07-06</h2>
|
|
<ul>
|
|
<li>Types meeting</li>
|
|
<li>I wrote a Python script to check Unpaywall for some information about DOIs</li>
|
|
</ul>
|
|
<h2 id="2023-07-7">2023-07-7</h2>
|
|
<ul>
|
|
<li>Continue exploring Unpaywall data for some of our DOIs
|
|
<ul>
|
|
<li>In the past I’ve found their <em>licensing</em> information to not be very reliable (preferring Crossref), but I think their <em>open access status</em> is more reliable, especially when the provider is listed as being the publisher</li>
|
|
<li>Even so, sometimes the version can be “acceptedVersion”, which is presumably the author’s version, as opposed to the “publishedVersion”, which means it’s available as open access on the publisher’s website</li>
|
|
<li>I did some quality assurance and found ~100 that were marked as Limited Access, but should have been Open Access, and fixed a handful of licenses</li>
|
|
</ul>
|
|
</li>
|
|
<li>Delete duplicate metadata as describe in my DSpace issue from last year: <a href="https://github.com/DSpace/DSpace/issues/8253">https://github.com/DSpace/DSpace/issues/8253</a></li>
|
|
<li>Start working on some statistics on AGROVOC usage for my presenation next week
|
|
<ul>
|
|
<li>I used the following SQL query to dump values from all subject fields and lower case them:</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>localhost/dspacetest= ☘ \COPY (SELECT DISTINCT(lower(text_value)) AS "subject" FROM metadatavalue WHERE dspace_object_id in (SELECT dspace_object_id FROM item) AND metadata_field_id IN (187, 120, 210, 122, 215, 127, 208, 124, 128, 123, 125, 135, 203, 236, 238, 119)) to /tmp/2023-07-07-cgspace-subjects.csv WITH CSV HEADER;
|
|
</span></span><span style="display:flex;"><span>COPY 26443
|
|
</span></span><span style="display:flex;"><span>Time: 2564.851 ms (00:02.565)
|
|
</span></span></code></pre></div><ul>
|
|
<li>Then I extracted the subjects and looked them up against AGROVOC:</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ csvcut -c subject /tmp/2023-07-07-cgspace-subjects.csv | sed <span style="color:#e6db74">'1d'</span> > /tmp/2023-07-07-cgspace-subjects.txt
|
|
</span></span><span style="display:flex;"><span>$ ./ilri/agrovoc_lookup.py -i /tmp/2023-07-07-cgspace-subjects.txt -o /tmp/2023-07-07-cgspace-subjects-results.csv
|
|
</span></span></code></pre></div><ul>
|
|
<li>I did some more tests with Angular 13 on OpenRXV and found out why the repository type dropdown wasn’t working
|
|
<ul>
|
|
<li>It was because of a missing 1-line JSON file in the data directory, which is runtime data, not code</li>
|
|
<li>I copied the data directory from the production serve and rebuild and the site is working well now</li>
|
|
<li>I did a full harvest with plugins and it worked!</li>
|
|
<li>So it seems Angular 13.4.0 will work, yay</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<h2 id="2023-07-08">2023-07-08</h2>
|
|
<ul>
|
|
<li>Export CGSpace to check for missing Initiative collection mappings
|
|
<ul>
|
|
<li>Start a harvest on AReS</li>
|
|
</ul>
|
|
</li>
|
|
<li>The AGROVOC lookup finished, so I checked the number of matches:</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ csvgrep -c <span style="color:#e6db74">'match type'</span> -r <span style="color:#e6db74">'^.+$'</span> ~/Downloads/2023-07-07-cgspace-subjects-resolved.csv | sed 1d | wc -l
|
|
</span></span><span style="display:flex;"><span>12528
|
|
</span></span></code></pre></div><ul>
|
|
<li>So that’s 12,528 out of 26,443 unique terms (47.3%)</li>
|
|
<li>I did a LOT of work on the OpenRXV frontend build dependencies to bring more in line with Angular 13</li>
|
|
</ul>
|
|
<h2 id="2023-07-10">2023-07-10</h2>
|
|
<ul>
|
|
<li>I did a lot more work on OpenRXV to test and update dependencies</li>
|
|
<li>I deployed the latest version on the production server</li>
|
|
</ul>
|
|
<h2 id="2023-07-12">2023-07-12</h2>
|
|
<ul>
|
|
<li>CGSpace upgrade meeting with Americas and Africa group</li>
|
|
</ul>
|
|
<h2 id="2023-07-13">2023-07-13</h2>
|
|
<ul>
|
|
<li>Michael Victor asked me to help Aditi extract some information from CGSpace
|
|
<ul>
|
|
<li>She was interested in journal articles published between 2018 and 2023 with a range of subjects related to drought, flooding, resilience, etc</li>
|
|
<li>I used an advanced query with some AGROVOC terms:</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>dcterms.issued:[2018 TO 2023] AND dcterms.type:"Journal Article" AND (dcterms.subject:flooding OR dcterms.subject:flood OR dcterms.subject:"extreme weather events" OR dcterms.subject:drought OR dcterms.subject:"drought resistance" OR dcterms.subject:"drought tolerance" OR dcterms.subject:"soil salinity" OR dcterms.subject:"pests of plants" OR dcterms.subject:pests OR dcterms.subject:heat OR dcterms.subject:fertilizers OR dcterms.subject:"fertilizer technology" OR dcterms.subject:"rice fields" OR dcterms.subject:"landscape conservation" OR dcterms.subject:"landscape restoration" OR dcterms.subject:livestock)
|
|
</span></span></code></pre></div><ul>
|
|
<li>Interestingly, some variations of this same exact query produce no search results, and I see this error in the DSpace log:</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>org.dspace.discovery.SearchServiceException: org.apache.solr.search.SyntaxError: Cannot parse 'dcterms.issued:[2018 TO 2023] AND dcterms.type:"Journal Article" AND (dcterms.subject:flooding OR dcterms.subject:flood OR dcterms.subject:"extreme weather events" OR dcterms.subject:drought OR dcterms.subject:"drought resistance" OR dcterms.subject:"drought tolerance" OR dcterms.subject:"soil salinity" OR dcterms.subject:"pests of plants" OR dcterms.subject:pests OR dcterms.subject:heat OR dcterms.subject:fertilizers OR dcterms.subject:"fertilizer technology" OR dcterms.subject:"rice fields" OR dcterms.subject:livestock OR dcterms.subject:"landscape conservation" OR dcterms.subject:"landscape restoration\"\)': Lexical error at line 1, column 617. Encountered: <EOF> after : "\"landscape restoration\\\"\\)"
|
|
</span></span></code></pre></div><ul>
|
|
<li>It seems to be when there is a quoted search term at the end of the parenthesized group
|
|
<ul>
|
|
<li>For what it’s worth this same query worked fine on DSpace 7.6</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<h2 id="2023-07-15">2023-07-15</h2>
|
|
<ul>
|
|
<li>Export CGSpace to fix missing Initiative collection mappings</li>
|
|
<li>Start a harvest on AReS</li>
|
|
</ul>
|
|
<h2 id="2023-07-17">2023-07-17</h2>
|
|
<ul>
|
|
<li>Rasika had sent me a list of new ORCID identifiers for new IWMI staff so I combined them with our existing list and ran <code>resolve_orcids.py</code> to refresh the names in our database
|
|
<ul>
|
|
<li>I updated the list, updated names in the database, and tagged new authors with missing identifiers in existing items</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<h2 id="2023-07-18">2023-07-18</h2>
|
|
<ul>
|
|
<li>Meeting with IWMI, IRRI, and IITA colleagues about CGSpace upgrade plans</li>
|
|
<li>Maria from the Alliance mentioned having some submissions stuck on CGSpace
|
|
<ul>
|
|
<li>I looked and found a number of locks stuck for many nineteen, eighteen, and more hours…</li>
|
|
<li>I killed them and told her to try again</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ psql < locks-age.sql | less -S
|
|
</span></span><span style="display:flex;"><span>$ psql < locks-age.sql | grep -E <span style="color:#e6db74">" (19|18|17|16|12):"</span> | awk -F<span style="color:#e6db74">"|"</span> <span style="color:#e6db74">'{print $10}'</span> | sort -u | xargs kill
|
|
</span></span></code></pre></div><!-- raw HTML omitted -->
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</article>
|
|
|
|
|
|
|
|
</div> <!-- /.blog-main -->
|
|
|
|
<aside class="col-sm-3 ml-auto blog-sidebar">
|
|
|
|
|
|
|
|
<section class="sidebar-module">
|
|
<h4>Recent Posts</h4>
|
|
<ol class="list-unstyled">
|
|
|
|
|
|
<li><a href="/cgspace-notes/2023-07/">July, 2023</a></li>
|
|
|
|
<li><a href="/cgspace-notes/2023-06/">June, 2023</a></li>
|
|
|
|
<li><a href="/cgspace-notes/2023-05/">May, 2023</a></li>
|
|
|
|
<li><a href="/cgspace-notes/2023-04/">April, 2023</a></li>
|
|
|
|
<li><a href="/cgspace-notes/2023-03/">March, 2023</a></li>
|
|
|
|
</ol>
|
|
</section>
|
|
|
|
|
|
|
|
|
|
<section class="sidebar-module">
|
|
<h4>Links</h4>
|
|
<ol class="list-unstyled">
|
|
|
|
<li><a href="https://cgspace.cgiar.org">CGSpace</a></li>
|
|
|
|
<li><a href="https://dspacetest.cgiar.org">DSpace Test</a></li>
|
|
|
|
<li><a href="https://github.com/ilri/DSpace">CGSpace @ GitHub</a></li>
|
|
|
|
</ol>
|
|
</section>
|
|
|
|
</aside>
|
|
|
|
|
|
</div> <!-- /.row -->
|
|
</div> <!-- /.container -->
|
|
|
|
|
|
|
|
<footer class="blog-footer">
|
|
<p dir="auto">
|
|
|
|
Blog template created by <a href="https://twitter.com/mdo">@mdo</a>, ported to Hugo by <a href='https://twitter.com/mralanorth'>@mralanorth</a>.
|
|
|
|
</p>
|
|
<p>
|
|
<a href="#">Back to top</a>
|
|
</p>
|
|
</footer>
|
|
|
|
|
|
</body>
|
|
|
|
</html>
|