Create an “AfricaRice books and book chapters” collection on CGSpace for AfricaRice
Last month Sisay asked why the following “most popular” statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace:
DSpace Test
CGSpace
Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community
Create an “AfricaRice books and book chapters” collection on CGSpace for AfricaRice
Last month Sisay asked why the following “most popular” statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace:
DSpace Test
CGSpace
Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community
<li>Create an “AfricaRice books and book chapters” collection on CGSpace for AfricaRice</li>
<li>Last month Sisay asked why the following “most popular” statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace:
<li>Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community</li>
</ul>
<ul>
<li>If I change the parameters to 2019 I see stats, so I’m really thinking it has something to do with the sharded yearly Solr statistics cores
<ul>
<li>I checked the Solr admin UI and I see all Solr cores loaded, so I don’t know what it could be</li>
<li>When I check the Atmire content and usage module it seems obvious that there is a problem with the old cores because I dont have anything before 2019-01</li>
<li>I don’t see anyone logged in right now so I’m going to try to restart Tomcat and see if the stats are accessible after Solr comes back up</li>
<li>I decided to run all system updates on the server (linode18) and reboot it
<ul>
<li>After rebooting Tomcat came back up, but the the Solr statistics cores were not all loaded</li>
<li>The error is always (with a different core):</li>
<li>I stopped Tomcat, deleted the old locks, and will try to use the “simple” lock file type in <code>solr/statistics/conf/solrconfig.xml</code>:</li>
<li>Now the cores seem to load, but I still see an error in the Solr Admin UI and I still can’t access any stats before 2018</li>
<li>I filed an <ahref="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=685">issue with Atmire</a>, so let’s see if they can help</li>
<li>And since I’m annoyed and it’s been a few months, I’m going to move the JVM heap settings that I’ve been testing on DSpace Test to CGSpace</li>
<li>And the new ones come from Solr 4.10.x’s startup scripts:</li>
</ul>
<pretabindex="0"><code> -Djava.awt.headless=true
-Xms8192m -Xmx8192m
-Dfile.encoding=UTF-8
-XX:NewRatio=3
-XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90
-XX:MaxTenuringThreshold=8
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-XX:ConcGCThreads=4 -XX:ParallelGCThreads=4
-XX:+CMSScavengeBeforeRemark
-XX:PretenureSizeThreshold=64m
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
-XX:CMSMaxAbortablePrecleanTime=6000
-XX:+CMSParallelRemarkEnabled
-XX:+ParallelRefProcEnabled
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=1337
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
</code></pre><h2id="2019-07-02">2019-07-02</h2>
<ul>
<li>Help upload twenty-seven posters from the 2019-05 Sharefair to CGSpace
<ul>
<li>Sisay had already done the SAFBundle so I did some minor corrections to and uploaded them to a temporary collection so I could check them in OpenRefine:</li>
</ul>
</li>
</ul>
<pretabindex="0"><code>$ sed -i 's/CC-BY 4.0/CC-BY-4.0/' item_*/dublin_core.xml
$ dspace import -a -e me@cgiar.org -m 2019-07-02-Sharefair.map -s /tmp/Sharefair_mapped
</code></pre><ul>
<li>I noticed that all twenty-seven items had double dates like “2019-05||2019-05” so I fixed those, but the rest of the metadata looked good so I unmapped them from the temporary collection</li>
<li>Finish looking at the fifty-six AfricaRice items and upload them to CGSpace:</li>
</ul>
<pretabindex="0"><code>$ dspace import -a -e me@cgiar.org -m 2019-07-02-AfricaRice-11to73.map -s /tmp/SimpleArchiveFormat
</code></pre><ul>
<li>Peter pointed out that the Sharefair dates I fixed were not actually fixed
<ul>
<li>It seems there is a bug that causes DSpace to not detect changes if the values are the same like “2019-05||2019-05” and you try to remove one</li>
<li>To get it to work I had to change some of them to 2019-01, then remove them</li>
</ul>
</li>
</ul>
<h2id="2019-07-03">2019-07-03</h2>
<ul>
<li>Atmire responded about the <ahref="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=685">Solr issue</a> and said they would be willing to help</li>
</ul>
<h2id="2019-07-04">2019-07-04</h2>
<ul>
<li>Maria Garruccio sent me some new ORCID identifiers for Bioversity authors
<ul>
<li>I combined them with our existing list and then used my <code>resolve-orcids.py</code> script to update the names from ORCID.org:</li>
<li>Send a reminder to Marie about my notes on the <ahref="https://github.com/AgriculturalSemantics/cg-core/issues/2">CG Core v2 issue I created two weeks ago</a></li>
</ul>
<h2id="2019-07-08">2019-07-08</h2>
<ul>
<li>Communicate with Atmire about the Solr statistics cores issue
<ul>
<li>I suspect we might need to get more disk space on DSpace Test so we can try to replicate the production environment more closely</li>
</ul>
</li>
<li>Meeting with AgroKnow and CTA about their new ICT Update story telling thing
<ul>
<li>AgroKnow has developed a React application to display tag clouds based on harvesting metadata and full text from CGSpace items</li>
<li>We discussed how to host it technically, perhaps we purchase a server to run it on and just give AgroKnow guys access</li>
</ul>
</li>
<li>Playing with the idea of using <ahref="https://github.com/BurntSushi/xsv">xsv</a> to do some basic batch quality checks on CSVs, for example to find items that might be duplicates if they have the same DOI or title:</li>
<li>Or perhaps items with invalid ISSNs (according to the <ahref="https://en.wikipedia.org/wiki/International_Standard_Serial_Number#Code_format">ISSN code format</a>):</li>
<li>Skype call with Marie Angelique about CG Core v2
<ul>
<li>We discussed my comments and suggestions from last week</li>
<li>One comment she had was that we should try to move our center-specific subjects into <code>DCTERMS.subject</code> and normalize them against AGROVOC</li>
<li>I updated my <ahref="https://gist.github.com/alanorth/2db39e91f48d116e00a4edffd6ba6409">gist about CGSpace metadata changes</a></li>
</ul>
</li>
<li>Skype call with Jane Poole to discuss OpenRXV/AReS Phase II TORs
<ul>
<li>I need to follow up with Moayad about the reporting functionality</li>
<li>Also, I need to email Harrison my notes on the CG Core v2 stuff</li>
<li>Also, Jane asked me to check the Data Portal to see which email address requests for confidential data are going</li>
<li>Yesterday Thierry from CTA asked me about an error he was getting while submitting an item on CGSpace: “Unable to load Submission Information, since WorkspaceID (ID:S106658) is not a valid in-process submission.”</li>
<li>I looked in the DSpace logs and found this right around the time of the screenshot he sent me:</li>
</ul>
<pretabindex="0"><code>2019-07-10 11:50:27,433 INFO org.dspace.submit.step.CompleteStep @ lewyllie@cta.int:session_id=A920730003BCAECE8A3B31DCDE11A97E:submission_complete:Completed submission with id=106658
</code></pre><ul>
<li>I’m assuming something happened in his browser (like a refresh) after the item was submitted…</li>
</ul>
<h2id="2019-07-12">2019-07-12</h2>
<ul>
<li>Atmire responded with some initial feedback about our Tomcat configuration related to the <ahref="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=685">Solr issue I raised recently</a>
<ul>
<li>Unfortunately there is no concrete feedback yet</li>
<li>I think we need to upgrade our DSpace Test server so we can fit all the Solr cores…</li>
<li>Actually, I looked and there were over 40 GB free on DSpace Test so I copied the Solr statistics cores for the years 2017 to 2010 from CGSpace to DSpace Test because they weren’t actually very large</li>
<li>I re-deployed DSpace for good measure, and I think all Solr cores are loading… I will do more tests later</li>
</ul>
</li>
<li>Run all system updates on DSpace Test (linode19) and reboot it</li>
<li>Try to run <code>dspace cleanup -v</code> on CGSpace and ran into an error:</li>
</ul>
<pretabindex="0"><code>Error: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
Detail: Key (bitstream_id)=(167394) is still referenced from table "bundle".
</code></pre><ul>
<li>The solution is, as always:</li>
</ul>
<pretabindex="0"><code># su - postgres
$ psql dspace -c 'update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (167394);'
UPDATE 1
</code></pre><h2id="2019-07-16">2019-07-16</h2>
<ul>
<li>Completely reset the Podman configuration on my laptop because there were some layers that I couldn’t delete and it had been some time since I did a cleanup:</li>
</ul>
<pretabindex="0"><code>$ podman system prune -a -f --volumes
$ sudo rm -rf ~/.local/share/containers
</code></pre><ul>
<li>Then pull a new PostgreSQL 9.6 image and load a CGSpace database dump into a new local test container:</li>
<li>Start working on implementing the <ahref="https://gist.github.com/alanorth/2db39e91f48d116e00a4edffd6ba6409">CG Core v2 changes</a> on my local DSpace test environment</li>
<li>Make a pull request to CG Core v2 with some fixes for typos in the specification (<ahref="https://github.com/AgriculturalSemantics/cg-core/pull/5">#5</a>)</li>
</ul>
<h2id="2019-07-18">2019-07-18</h2>
<ul>
<li>Talk to Moayad about the remaining issues for OpenRXV / AReS
<ul>
<li>He sent a pull request with some changes for the bar chart and documentation about configuration, and said he’d finish the export feature next week</li>
</ul>
</li>
<li>Sisay said a user was having problems registering on CGSpace and it looks like the email account expired again:</li>
</ul>
<pretabindex="0"><code>$ dspace test-email
About to send test email:
- To: blahh@cgiar.org
- Subject: DSpace test email
- Server: smtp.office365.com
Error sending email:
- Error: javax.mail.AuthenticationFailedException
Please see the DSpace documentation for assistance.
</code></pre><ul>
<li>I emailed ICT to ask them to reset it and make the expiration period longer if possible</li>
</ul>
<h2id="2019-07-19">2019-07-19</h2>
<ul>
<li>ICT reset the password for the CGSpace support account and apparently removed the expiry requirement
<ul>
<li>I tested the account and it’s working</li>
</ul>
</li>
</ul>
<h2id="2019-07-20">2019-07-20</h2>
<ul>
<li>Create an account for Lionelle Samnick on CGSpace because the registration isn’t working for some reason:</li>
<li>I left a note saying that DSpace is technically limited to a flat schema so we use <code>cg.coverage.country: Kenya</code></li>
<li>Do a little more work on CG Core v2 in the input forms</li>
</ul>
<h2id="2019-07-25">2019-07-25</h2>
<ul>
<li>
<p>Generate a list of the ORCID identifiers that we added to CGSpace in 2019 for Sara Jani at ICARDA</p>
</li>
<li>
<p>Bioversity sent a new file for their migration to CGSpace</p>
<ul>
<li>There is always a blank row and blank column at the end</li>
<li>One invalid type (Brie)</li>
<li>824 items with leading/trailing spaces in dc.identifier.citation</li>
<li>175 items with a trailing comma in dc.identifier.citation (using custom text facet with GREL <code>value.endsWith(',').toString()</code>)</li>
<li>Fix them with GREL transform: <code>value.replace(/,$/, '')</code></li>
<li>A few strange publishers after splitting multi-value cells, like “(Belgium)”</li>
<li>Deleted four ISSNs that are actually ISBNs and are already present in the ISBN field</li>
<li>Eight invalid ISBNs</li>
<li>Convert all DOIs to “<ahref="https://doi.org">https://doi.org</a>” format and fix one invalid DOI</li>
<li>Fix a handful of incorrect CRPs that seem to have been split on comma “,”</li>
<li>Lots of strange values in cg.link.reference, and I normalized all DOIs to <ahref="https://doi.org">https://doi.org</a> format
<ul>
<li>There are lots of invalid links here, like “36” and “recordlink:publications:2606” and “t3://record?identifier=publications&uid=2606”</li>
<li>Also there are hundreds of items that use the same value for cg.link.reference AND cg.link.dataurl</li>
</ul>
</li>
<li>Use https:// for all Bioversity links (reference, data url, permalink)</li>
</ul>
</li>
<li>
<p>I might be able to use <ahref="https://pypi.org/project/isbnlib/">isbnlib</a> to validate ISBNs in Python:</p>
</li>
</ul>
<pretabindex="0"><code>if isbnlib.is_isbn10('9966-955-07-0') or isbnlib.is_isbn13('9966-955-07-0'):
print("Yes")
else:
print("No")
</code></pre><ul>
<li>Or with <ahref="https://github.com/arthurdejong/python-stdnum">python-stdnum</a>:</li>
</ul>
<pretabindex="0"><code>from stdnum import isbn
from stdnum import issn
isbn.validate('978-92-9043-389-7')
issn.validate('1020-3362')
</code></pre><h2id="2019-07-26">2019-07-26</h2>
<ul>
<li>
<p>Bioversity sent me an updated CSV file that fixes some of the issues I pointed out yesterday</p>
<ul>
<li>There are still 1429 records</li>
<li>There are still one extra row and one extra column</li>
<li>There are still eight invalid ISBNs (according to my <code>validate.py</code> script)</li>
</ul>
</li>
<li>
<p>I figured out a GREL to trim spaces in multi-value cells without splitting them:</p>
<li>I whipped up a quick script using Python Pandas to do whitespace cleanup</li>
</ul>
<h2id="2019-07-29">2019-07-29</h2>
<ul>
<li>I turned the Pandas script into a proper Python package called <ahref="https://git.sr.ht/~alanorth/csv-metadata-quality">csv-metadata-quality</a>
<ul>
<li>It supports CSV and Excel files</li>
<li>It fixes whitespace errors and erroneous multi-value separators ("|") and validates ISSN, ISBNs, and dates</li>
<li>Also I added a bunch of other checks/fixes for unnecessary and “suspicious” Unicode characters</li>
<li>I added fixes to drop duplicate metadata values</li>
<li>And lastly, I added validation of ISO 639-2 and ISO 639-3 languages</li>
<li>And lastly lastly, I added AGROVOC validation of subject terms</li>
</ul>
</li>
<li>Inform Bioversity that there is an error in their CSV, seemingly caused by quotes in the citation field</li>
</ul>
<h2id="2019-07-30">2019-07-30</h2>
<ul>
<li>Add support for removing newlines (line feeds) to <ahref="https://git.sr.ht/~alanorth/csv-metadata-quality">csv-metadata-quality</a></li>
<li>On the subject of validating some of our fields like countries and regions, Abenet pointed out that these should all be valid AGROVOC terms, so we can actually try to validate against that!</li>
