mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2024-12-18 19:22:18 +01:00
656 lines
26 KiB
HTML
656 lines
26 KiB
HTML
<!DOCTYPE html>
|
|
<html lang="en">
|
|
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
|
|
|
|
<meta property="og:title" content="July, 2019" />
|
|
<meta property="og:description" content="2019-07-01
|
|
|
|
|
|
Create an “AfricaRice books and book chapters” collection on CGSpace for AfricaRice
|
|
Last month Sisay asked why the following “most popular” statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace:
|
|
|
|
|
|
DSpace Test
|
|
CGSpace
|
|
|
|
Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community
|
|
" />
|
|
<meta property="og:type" content="article" />
|
|
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2019-07/" />
|
|
<meta property="article:published_time" content="2019-07-01T12:13:51+03:00"/>
|
|
<meta property="article:modified_time" content="2019-07-25T16:58:23+03:00"/>
|
|
|
|
<meta name="twitter:card" content="summary"/>
|
|
<meta name="twitter:title" content="July, 2019"/>
|
|
<meta name="twitter:description" content="2019-07-01
|
|
|
|
|
|
Create an “AfricaRice books and book chapters” collection on CGSpace for AfricaRice
|
|
Last month Sisay asked why the following “most popular” statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace:
|
|
|
|
|
|
DSpace Test
|
|
CGSpace
|
|
|
|
Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community
|
|
"/>
|
|
<meta name="generator" content="Hugo 0.55.6" />
|
|
|
|
|
|
|
|
<script type="application/ld+json">
|
|
{
|
|
"@context": "http://schema.org",
|
|
"@type": "BlogPosting",
|
|
"headline": "July, 2019",
|
|
"url": "https:\/\/alanorth.github.io\/cgspace-notes\/2019-07\/",
|
|
"wordCount": "2191",
|
|
"datePublished": "2019-07-01T12:13:51\x2b03:00",
|
|
"dateModified": "2019-07-25T16:58:23\x2b03:00",
|
|
"author": {
|
|
"@type": "Person",
|
|
"name": "Alan Orth"
|
|
},
|
|
"keywords": "Notes"
|
|
}
|
|
</script>
|
|
|
|
|
|
|
|
<link rel="canonical" href="https://alanorth.github.io/cgspace-notes/2019-07/">
|
|
|
|
<title>July, 2019 | CGSpace Notes</title>
|
|
|
|
<!-- combined, minified CSS -->
|
|
<link href="https://alanorth.github.io/cgspace-notes/css/style.css" rel="stylesheet" integrity="sha384-G5B34w7DFTumWTswxYzTX7NWfbvQEg1HbFFEg6ItN03uTAAoS2qkPS/fu3LhuuSA" crossorigin="anonymous">
|
|
|
|
<!-- RSS 2.0 feed -->
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</head>
|
|
|
|
<body>
|
|
|
|
|
|
<div class="blog-masthead">
|
|
<div class="container">
|
|
<nav class="nav blog-nav">
|
|
<a class="nav-link " href="https://alanorth.github.io/cgspace-notes/">Home</a>
|
|
</nav>
|
|
</div>
|
|
</div>
|
|
|
|
|
|
|
|
|
|
<header class="blog-header">
|
|
<div class="container">
|
|
<h1 class="blog-title"><a href="https://alanorth.github.io/cgspace-notes/" rel="home">CGSpace Notes</a></h1>
|
|
<p class="lead blog-description">Documenting day-to-day work on the <a href="https://cgspace.cgiar.org">CGSpace</a> repository.</p>
|
|
</div>
|
|
</header>
|
|
|
|
|
|
|
|
|
|
<div class="container">
|
|
<div class="row">
|
|
<div class="col-sm-8 blog-main">
|
|
|
|
|
|
|
|
|
|
<article class="blog-post">
|
|
<header>
|
|
<h2 class="blog-post-title"><a href="https://alanorth.github.io/cgspace-notes/2019-07/">July, 2019</a></h2>
|
|
<p class="blog-post-meta"><time datetime="2019-07-01T12:13:51+03:00">Mon Jul 01, 2019</time> by Alan Orth in
|
|
|
|
<i class="fa fa-tag" aria-hidden="true"></i> <a href="/cgspace-notes/tags/notes" rel="tag">Notes</a>
|
|
|
|
</p>
|
|
</header>
|
|
<h2 id="2019-07-01">2019-07-01</h2>
|
|
|
|
<ul>
|
|
<li>Create an “AfricaRice books and book chapters” collection on CGSpace for AfricaRice</li>
|
|
<li>Last month Sisay asked why the following “most popular” statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace:
|
|
|
|
<ul>
|
|
<li><a href="https://dspacetest.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&time_filter_end_date=01%2F12%2F2018">DSpace Test</a></li>
|
|
<li><a href="https://cgspace.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&time_filter_end_date=01%2F12%2F2018">CGSpace</a></li>
|
|
</ul></li>
|
|
<li>Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community</li>
|
|
</ul>
|
|
|
|
<ul>
|
|
<li>If I change the parameters to 2019 I see stats, so I’m really thinking it has something to do with the sharded yearly Solr statistics cores
|
|
|
|
<ul>
|
|
<li>I checked the Solr admin UI and I see all Solr cores loaded, so I don’t know what it could be</li>
|
|
<li>When I check the Atmire content and usage module it seems obvious that there is a problem with the old cores because I dont have anything before 2019-01</li>
|
|
</ul></li>
|
|
</ul>
|
|
|
|
<p><img src="/cgspace-notes/2019/07/atmire-cua-2018-missing.png" alt="Atmire CUA 2018 stats missing" /></p>
|
|
|
|
<ul>
|
|
<li>I don’t see anyone logged in right now so I’m going to try to restart Tomcat and see if the stats are accessible after Solr comes back up</li>
|
|
|
|
<li><p>I decided to run all system updates on the server (linode18) and reboot it</p>
|
|
|
|
<ul>
|
|
<li>After rebooting Tomcat came back up, but the the Solr statistics cores were not all loaded</li>
|
|
|
|
<li><p>The error is always (with a different core):</p>
|
|
|
|
<pre><code>org.apache.solr.common.SolrException: Error CREATEing SolrCore 'statistics-2010': Unable to create core [statistics-2010] Caused by: Lock obtain timed out: NativeFSLock@/home/cgspace.cgiar.org/solr/statistics-2010/data/index/write.lock
|
|
</code></pre></li>
|
|
</ul></li>
|
|
|
|
<li><p>I restarted Tomcat <em>ten times</em> and it never worked…</p></li>
|
|
|
|
<li><p>I tried to stop Tomcat and delete the write locks:</p>
|
|
|
|
<pre><code># systemctl stop tomcat7
|
|
# find /dspace/solr/statistics* -iname "*.lock" -print -delete
|
|
/dspace/solr/statistics/data/index/write.lock
|
|
/dspace/solr/statistics-2010/data/index/write.lock
|
|
/dspace/solr/statistics-2011/data/index/write.lock
|
|
/dspace/solr/statistics-2012/data/index/write.lock
|
|
/dspace/solr/statistics-2013/data/index/write.lock
|
|
/dspace/solr/statistics-2014/data/index/write.lock
|
|
/dspace/solr/statistics-2015/data/index/write.lock
|
|
/dspace/solr/statistics-2016/data/index/write.lock
|
|
/dspace/solr/statistics-2017/data/index/write.lock
|
|
/dspace/solr/statistics-2018/data/index/write.lock
|
|
# find /dspace/solr/statistics* -iname "*.lock" -print -delete
|
|
# systemctl start tomcat7
|
|
</code></pre></li>
|
|
|
|
<li><p>But it still didn’t work!</p></li>
|
|
|
|
<li><p>I stopped Tomcat, deleted the old locks, and will try to use the “simple” lock file type in <code>solr/statistics/conf/solrconfig.xml</code>:</p>
|
|
|
|
<pre><code><lockType>${solr.lock.type:simple}</lockType>
|
|
</code></pre></li>
|
|
|
|
<li><p>And after restarting Tomcat it still doesn’t work</p></li>
|
|
|
|
<li><p>Now I’ll try going back to “native” locking with <code>unlockAtStartup</code>:</p>
|
|
|
|
<pre><code><unlockOnStartup>true</unlockOnStartup>
|
|
</code></pre></li>
|
|
|
|
<li><p>Now the cores seem to load, but I still see an error in the Solr Admin UI and I still can’t access any stats before 2018</p></li>
|
|
|
|
<li><p>I filed an <a href="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=685">issue with Atmire</a>, so let’s see if they can help</p></li>
|
|
|
|
<li><p>And since I’m annoyed and it’s been a few months, I’m going to move the JVM heap settings that I’ve been testing on DSpace Test to CGSpace</p></li>
|
|
|
|
<li><p>The old ones were:</p>
|
|
|
|
<pre><code>-Djava.awt.headless=true -Xms8192m -Xmx8192m -XX:+UseConcMarkSweepGC -Dfile.encoding=UTF-8 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=5400 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
|
|
</code></pre></li>
|
|
|
|
<li><p>And the new ones come from Solr 4.10.x’s startup scripts:</p>
|
|
|
|
<pre><code>-Djava.awt.headless=true
|
|
-Xms8192m -Xmx8192m
|
|
-Dfile.encoding=UTF-8
|
|
-XX:NewRatio=3
|
|
-XX:SurvivorRatio=4
|
|
-XX:TargetSurvivorRatio=90
|
|
-XX:MaxTenuringThreshold=8
|
|
-XX:+UseConcMarkSweepGC
|
|
-XX:+UseParNewGC
|
|
-XX:ConcGCThreads=4 -XX:ParallelGCThreads=4
|
|
-XX:+CMSScavengeBeforeRemark
|
|
-XX:PretenureSizeThreshold=64m
|
|
-XX:+UseCMSInitiatingOccupancyOnly
|
|
-XX:CMSInitiatingOccupancyFraction=50
|
|
-XX:CMSMaxAbortablePrecleanTime=6000
|
|
-XX:+CMSParallelRemarkEnabled
|
|
-XX:+ParallelRefProcEnabled
|
|
-Dcom.sun.management.jmxremote
|
|
-Dcom.sun.management.jmxremote.port=1337
|
|
-Dcom.sun.management.jmxremote.ssl=false
|
|
-Dcom.sun.management.jmxremote.authenticate=false
|
|
</code></pre></li>
|
|
</ul>
|
|
|
|
<h2 id="2019-07-02">2019-07-02</h2>
|
|
|
|
<ul>
|
|
<li><p>Help upload twenty-seven posters from the 2019-05 Sharefair to CGSpace</p>
|
|
|
|
<ul>
|
|
<li><p>Sisay had already done the SAFBundle so I did some minor corrections to and uploaded them to a temporary collection so I could check them in OpenRefine:</p>
|
|
|
|
<pre><code>$ sed -i 's/CC-BY 4.0/CC-BY-4.0/' item_*/dublin_core.xml
|
|
$ echo "10568/101992" >> item_*/collections
|
|
$ dspace import -a -e me@cgiar.org -m 2019-07-02-Sharefair.map -s /tmp/Sharefair_mapped
|
|
</code></pre></li>
|
|
</ul></li>
|
|
|
|
<li><p>I noticed that all twenty-seven items had double dates like “2019-05||2019-05” so I fixed those, but the rest of the metadata looked good so I unmapped them from the temporary collection</p></li>
|
|
|
|
<li><p>Finish looking at the fifty-six AfricaRice items and upload them to CGSpace:</p>
|
|
|
|
<pre><code>$ dspace import -a -e me@cgiar.org -m 2019-07-02-AfricaRice-11to73.map -s /tmp/SimpleArchiveFormat
|
|
</code></pre></li>
|
|
|
|
<li><p>Peter pointed out that the Sharefair dates I fixed were not actually fixed</p>
|
|
|
|
<ul>
|
|
<li>It seems there is a bug that causes DSpace to not detect changes if the values are the same like “2019-05||2019-05” and you try to remove one</li>
|
|
<li>To get it to work I had to change some of them to 2019-01, then remove them</li>
|
|
</ul></li>
|
|
</ul>
|
|
|
|
<h2 id="2019-07-03">2019-07-03</h2>
|
|
|
|
<ul>
|
|
<li>Atmire responded about the <a href="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=685">Solr issue</a> and said they would be willing to help</li>
|
|
</ul>
|
|
|
|
<h2 id="2019-07-04">2019-07-04</h2>
|
|
|
|
<ul>
|
|
<li><p>Maria Garruccio sent me some new ORCID identifiers for Bioversity authors</p>
|
|
|
|
<ul>
|
|
<li><p>I combined them with our existing list and then used my <code>resolve-orcids.py</code> script to update the names from ORCID.org:</p>
|
|
|
|
<pre><code>$ cat dspace/config/controlled-vocabularies/cg-creator-id.xml /tmp/new-bioversity-orcids.txt | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort -u > /tmp/2019-07-04-orcid-ids.txt
|
|
$ ./resolve-orcids.py -i /tmp/2019-07-04-orcid-ids.txt -o 2019-07-04-orcid-names.txt -d
|
|
</code></pre></li>
|
|
</ul></li>
|
|
|
|
<li><p>Send and merge a pull request for the new ORCID identifiers (<a href="https://github.com/ilri/DSpace/pull/428">#428</a>)</p></li>
|
|
|
|
<li><p>I created a CSV with some ORCID identifiers that I had seen change so I could update any existing ones in the databse:</p>
|
|
|
|
<pre><code>cg.creator.id,correct
|
|
"Marius Ekué: 0000-0002-5829-6321","Marius R.M. Ekué: 0000-0002-5829-6321"
|
|
"Mwungu: 0000-0001-6181-8445","Chris Miyinzi Mwungu: 0000-0001-6181-8445"
|
|
"Mwungu: 0000-0003-1658-287X","Chris Miyinzi Mwungu: 0000-0003-1658-287X"
|
|
</code></pre></li>
|
|
|
|
<li><p>But when I ran <code>fix-metadata-values.py</code> I didn’t see any changes:</p>
|
|
|
|
<pre><code>$ ./fix-metadata-values.py -i 2019-07-04-update-orcids.csv -db dspace -u dspace -p 'fuuu' -f cg.creator.id -m 240 -t correct -d
|
|
</code></pre></li>
|
|
</ul>
|
|
|
|
<h2 id="2019-07-06">2019-07-06</h2>
|
|
|
|
<ul>
|
|
<li>Send a reminder to Marie about my notes on the <a href="https://github.com/AgriculturalSemantics/cg-core/issues/2">CG Core v2 issue I created two weeks ago</a></li>
|
|
</ul>
|
|
|
|
<h2 id="2019-07-08">2019-07-08</h2>
|
|
|
|
<ul>
|
|
<li>Communicate with Atmire about the Solr statistics cores issue
|
|
|
|
<ul>
|
|
<li>I suspect we might need to get more disk space on DSpace Test so we can try to replicate the production environment more closely</li>
|
|
</ul></li>
|
|
<li>Meeting with AgroKnow and CTA about their new ICT Update story telling thing
|
|
|
|
<ul>
|
|
<li>AgroKnow has developed a React application to display tag clouds based on harvesting metadata and full text from CGSpace items</li>
|
|
<li>We discussed how to host it technically, perhaps we purchase a server to run it on and just give AgroKnow guys access</li>
|
|
</ul></li>
|
|
|
|
<li><p>Playing with the idea of using <a href="https://github.com/BurntSushi/xsv">xsv</a> to do some basic batch quality checks on CSVs, for example to find items that might be duplicates if they have the same DOI or title:</p>
|
|
|
|
<pre><code>$ xsv frequency --select cg.identifier.doi --no-nulls cgspace_metadata_africaRice-11to73_ay_id.csv | grep -v -E ',1'
|
|
field,value,count
|
|
cg.identifier.doi,https://doi.org/10.1016/j.agwat.2018.06.018,2
|
|
$ xsv frequency --select dc.title --no-nulls cgspace_metadata_africaRice-11to73_ay_id.csv | grep -v -E ',1'
|
|
field,value,count
|
|
dc.title,Reference evapotranspiration prediction using hybridized fuzzy model with firefly algorithm: Regional case study in Burkina Faso,2
|
|
</code></pre></li>
|
|
|
|
<li><p>Or perhaps if DOIs are valid or not (having doi.org in the URL):</p>
|
|
|
|
<pre><code>$ xsv frequency --select cg.identifier.doi --no-nulls cgspace_metadata_africaRice-11to73_ay_id.csv | grep -v -E 'doi.org'
|
|
field,value,count
|
|
cg.identifier.doi,https://hdl.handle.net/10520/EJC-1236ac700f,1
|
|
</code></pre></li>
|
|
|
|
<li><p>Or perhaps items with invalid ISSNs (according to the <a href="https://en.wikipedia.org/wiki/International_Standard_Serial_Number#Code_format">ISSN code format</a>):</p>
|
|
|
|
<pre><code>$ xsv select dc.identifier.issn cgspace_metadata_africaRice-11to73_ay_id.csv | grep -v '"' | grep -v -E '^[0-9]{4}-[0-9]{3}[0-9xX]$'
|
|
dc.identifier.issn
|
|
978-3-319-71997-9
|
|
978-3-319-71997-9
|
|
978-3-319-71997-9
|
|
978-3-319-58789-9
|
|
2320-7035
|
|
2593-9173
|
|
</code></pre></li>
|
|
</ul>
|
|
|
|
<h2 id="2019-07-09">2019-07-09</h2>
|
|
|
|
<ul>
|
|
<li>Thinking about data cleaning automation again and found some resources about Python and Pandas:
|
|
|
|
<ul>
|
|
<li><a href="https://realpython.com/python-data-cleaning-numpy-pandas/">https://realpython.com/python-data-cleaning-numpy-pandas/</a></li>
|
|
<li><a href="https://mode.com/blog/python-data-cleaning-libraries">https://mode.com/blog/python-data-cleaning-libraries</a></li>
|
|
</ul></li>
|
|
</ul>
|
|
|
|
<h2 id="2019-07-11">2019-07-11</h2>
|
|
|
|
<ul>
|
|
<li>Skype call with Marie Angelique about CG Core v2
|
|
|
|
<ul>
|
|
<li>We discussed my comments and suggestions from last week</li>
|
|
<li>One comment she had was that we should try to move our center-specific subjects into <code>DCTERMS.subject</code> and normalize them against AGROVOC</li>
|
|
<li>I updated my <a href="https://gist.github.com/alanorth/2db39e91f48d116e00a4edffd6ba6409">gist about CGSpace metadata changes</a></li>
|
|
</ul></li>
|
|
<li>Skype call with Jane Poole to discuss OpenRXV/AReS Phase II TORs
|
|
|
|
<ul>
|
|
<li>I need to follow up with Moayad about the reporting functionality</li>
|
|
<li>Also, I need to email Harrison my notes on the CG Core v2 stuff</li>
|
|
<li>Also, Jane asked me to check the Data Portal to see which email address requests for confidential data are going</li>
|
|
</ul></li>
|
|
<li>Yesterday Theirry from CTA asked me about an error he was getting while submitting an item on CGSpace: “Unable to load Submission Information, since WorkspaceID (ID:S106658) is not a valid in-process submission.”</li>
|
|
|
|
<li><p>I looked in the DSpace logs and found this right around the time of the screenshot he sent me:</p>
|
|
|
|
<pre><code>2019-07-10 11:50:27,433 INFO org.dspace.submit.step.CompleteStep @ lewyllie@cta.int:session_id=A920730003BCAECE8A3B31DCDE11A97E:submission_complete:Completed submission with id=106658
|
|
</code></pre></li>
|
|
|
|
<li><p>I’m assuming something happened in his browser (like a refresh) after the item was submitted…</p></li>
|
|
</ul>
|
|
|
|
<h2 id="2019-07-12">2019-07-12</h2>
|
|
|
|
<ul>
|
|
<li>Atmire responded with some initial feedback about our Tomcat configuration related to the <a href="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=685">Solr issue I raised recently</a>
|
|
|
|
<ul>
|
|
<li>Unfortunately there is no concrete feedback yet</li>
|
|
<li>I think we need to upgrade our DSpace Test server so we can fit all the Solr cores…</li>
|
|
<li>Actually, I looked and there were over 40 GB free on DSpace Test so I copied the Solr statistics cores for the years 2017 to 2010 from CGSpace to DSpace Test because they weren’t actually very large</li>
|
|
<li>I re-deployed DSpace for good measure, and I think all Solr cores are loading… I will do more tests later</li>
|
|
</ul></li>
|
|
<li>Run all system updates on DSpace Test (linode19) and reboot it</li>
|
|
|
|
<li><p>Try to run <code>dspace cleanup -v</code> on CGSpace and ran into an error:</p>
|
|
|
|
<pre><code>Error: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
|
|
Detail: Key (bitstream_id)=(167394) is still referenced from table "bundle".
|
|
</code></pre></li>
|
|
|
|
<li><p>The solution is, as always:</p>
|
|
|
|
<pre><code># su - postgres
|
|
$ psql dspace -c 'update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (167394);'
|
|
UPDATE 1
|
|
</code></pre></li>
|
|
</ul>
|
|
|
|
<h2 id="2019-07-16">2019-07-16</h2>
|
|
|
|
<ul>
|
|
<li><p>Completely reset the Podman configuration on my laptop because there were some layers that I couldn’t delete and it had been some time since I did a cleanup:</p>
|
|
|
|
<pre><code>$ podman system prune -a -f --volumes
|
|
$ sudo rm -rf ~/.local/share/containers
|
|
</code></pre></li>
|
|
|
|
<li><p>Then pull a new PostgreSQL 9.6 image and load a CGSpace database dump into a new local test container:</p>
|
|
|
|
<pre><code>$ podman pull postgres:9.6-alpine
|
|
$ podman run --name dspacedb -v dspacedb_data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:9.6-alpine
|
|
$ createuser -h localhost -U postgres --pwprompt dspacetest
|
|
$ createdb -h localhost -U postgres -O dspacetest --encoding=UNICODE dspacetest
|
|
$ psql -h localhost -U postgres dspacetest -c 'alter user dspacetest superuser;'
|
|
$ pg_restore -h localhost -U postgres -d dspacetest -O --role=dspacetest -h localhost ~/Downloads/cgspace_2019-07-16.backup
|
|
$ psql -h localhost -U postgres dspacetest -c 'alter user dspacetest nosuperuser;'
|
|
$ psql -h localhost -U postgres -f ~/src/git/DSpace/dspace/etc/postgres/update-sequences.sql dspacetest
|
|
</code></pre></li>
|
|
|
|
<li><p>Start working on implementing the <a href="https://gist.github.com/alanorth/2db39e91f48d116e00a4edffd6ba6409">CG Core v2 changes</a> on my local DSpace test environment</p></li>
|
|
|
|
<li><p>Make a pull request to CG Core v2 with some fixes for typos in the specification (<a href="https://github.com/AgriculturalSemantics/cg-core/pull/5">#5</a>)</p></li>
|
|
</ul>
|
|
|
|
<h2 id="2019-07-18">2019-07-18</h2>
|
|
|
|
<ul>
|
|
<li>Talk to Moayad about the remaining issues for OpenRXV / AReS
|
|
|
|
<ul>
|
|
<li>He sent a pull request with some changes for the bar chart and documentation about configuration, and said he’d finish the export feature next week</li>
|
|
</ul></li>
|
|
|
|
<li><p>Sisay said a user was having problems registering on CGSpace and it looks like the email account expired again:</p>
|
|
|
|
<pre><code>$ dspace test-email
|
|
|
|
About to send test email:
|
|
- To: blahh@cgiar.org
|
|
- Subject: DSpace test email
|
|
- Server: smtp.office365.com
|
|
|
|
Error sending email:
|
|
- Error: javax.mail.AuthenticationFailedException
|
|
|
|
Please see the DSpace documentation for assistance.
|
|
</code></pre></li>
|
|
|
|
<li><p>I emailed ICT to ask them to reset it and make the expiration period longer if possible</p></li>
|
|
</ul>
|
|
|
|
<h2 id="2019-07-19">2019-07-19</h2>
|
|
|
|
<ul>
|
|
<li>ICT reset the password for the CGSpace support account and apparently removed the expiry requirement
|
|
|
|
<ul>
|
|
<li>I tested the account and it’s working</li>
|
|
</ul></li>
|
|
</ul>
|
|
|
|
<h2 id="2019-07-20">2019-07-20</h2>
|
|
|
|
<ul>
|
|
<li><p>Create an account for Lionelle Samnick on CGSpace because the registration isn’t working for some reason:</p>
|
|
|
|
<pre><code>$ dspace user --add --givenname Lionelle --surname Samnick --email blah@blah.com --password 'blah'
|
|
</code></pre></li>
|
|
|
|
<li><p>I added her as a submitter to <a href="https://cgspace.cgiar.org/handle/10568/74536">CTA ISF Pro-Agro series</a></p></li>
|
|
|
|
<li><p>Start looking at 1429 records for the Bioversity batch import</p>
|
|
|
|
<ul>
|
|
<li>Multiple authors should be specified with multi-value separatator (||) instead of ;</li>
|
|
<li>We don’t use “(eds)” as an author</li>
|
|
<li>Same issue with dc.publisher using “;” for multiple values</li>
|
|
<li>Some invalid ISSNs in dc.identifier.issn (they look like ISBNs)</li>
|
|
<li>I see some ISSNs in the dc.identifier.isbn field</li>
|
|
<li>I see some invalid ISBNs that look like Excel errors (9,78E+12)</li>
|
|
<li>For DOI we just use the URL, not “DOI: <a href="https://doi.org..."">https://doi.org..."</a></li>
|
|
<li>I see an invalid “LEAVE BLANK” in the cg.contributor.crp field</li>
|
|
<li>Country field is using “,” for multiple values instead of “||”</li>
|
|
<li>Region field is using “,” for multiple values instead of “||”</li>
|
|
<li>Language field should be lowercase like “en”, and it is using the wrong multiple value separator, and has some invalid values</li>
|
|
<li>What is the cg.identifier.url2 field? You should probably add those as cg.link.reference</li>
|
|
</ul></li>
|
|
</ul>
|
|
|
|
<h2 id="2019-07-22">2019-07-22</h2>
|
|
|
|
<ul>
|
|
<li><p>Raise an <a href="https://github.com/AgriculturalSemantics/cg-core/issues/8">issue on CG Core v2 spec regarding country and region coverage</a></p>
|
|
|
|
<ul>
|
|
<li><p>The current standard has them implemented as a class like this:</p>
|
|
|
|
<pre><code><dct:coverage>
|
|
<dct:spatial>
|
|
<type>Country</type>
|
|
<dct:identifier>http://sws.geonames.org/192950</dct:identifier>
|
|
<rdfs:label>Kenya</rdfs:label>
|
|
</dct:spatial>
|
|
</dct:coverage>
|
|
</code></pre></li>
|
|
</ul></li>
|
|
|
|
<li><p>I left a note saying that DSpace is technically limited to a flat schema so we use <code>cg.coverage.country: Kenya</code></p></li>
|
|
|
|
<li><p>Do a little more work on CG Core v2 in the input forms</p></li>
|
|
</ul>
|
|
|
|
<h2 id="2019-07-25">2019-07-25</h2>
|
|
|
|
<ul>
|
|
<li>Generate a list of the ORCID identifiers that we added to CGSpace in 2019 for Sara Jani at ICARDA</li>
|
|
|
|
<li><p>Bioversity sent a new file for their migration to CGSpace</p>
|
|
|
|
<ul>
|
|
<li>There is always a blank row and blank column at the end</li>
|
|
<li>One invalid type (Brie)</li>
|
|
<li>824 items with leading/trailing spaces in dc.identifier.citation</li>
|
|
<li>175 items with a trailing comma in dc.identifier.citation (using custom text facet with GREL <code>value.endsWith(',').toString()</code>)</li>
|
|
<li>Fix them with GREL transform: <code>value.replace(/,$/, '')</code></li>
|
|
<li>A few strange publishers after splitting multi-value cells, like “(Belgium)”</li>
|
|
<li>Deleted four ISSNs that are actually ISBNs and are already present in the ISBN field</li>
|
|
<li>Eight invalid ISBNs</li>
|
|
<li>Convert all DOIs to “<a href="https://doi.org"">https://doi.org"</a> format and fix one invalid DOI</li>
|
|
<li>Fix a handful of incorrect CRPs that seem to have been split on comma “,”</li>
|
|
<li>Lots of strange values in cg.link.reference, and I normalized all DOIs to <a href="https://doi.org">https://doi.org</a> format</li>
|
|
<li>There are lots of invalid links here, like “36” and “recordlink:publications:2606” and “t3://record?identifier=publications&uid=2606”</li>
|
|
<li>Also there are hundreds of items that use the same value for cg.link.reference AND cg.link.dataurl</li>
|
|
<li>Use https:// for all Bioversity links (reference, data url, permalink)</li>
|
|
</ul></li>
|
|
|
|
<li><p>I might be able to use <a href="https://pypi.org/project/isbnlib/">isbnlib</a> to validate ISBNs in Python:</p>
|
|
|
|
<pre><code>if isbnlib.is_isbn10('9966-955-07-0') or isbnlib.is_isbn13('9966-955-07-0'):
|
|
print("Yes")
|
|
else:
|
|
print("No")
|
|
</code></pre></li>
|
|
|
|
<li><p>Or with <a href="https://github.com/arthurdejong/python-stdnum">python-stdnum</a>:</p>
|
|
|
|
<pre><code>from stdnum import isbn
|
|
from stdnum import issn
|
|
|
|
isbn.validate('978-92-9043-389-7')
|
|
issn.validate('1020-3362')
|
|
</code></pre></li>
|
|
</ul>
|
|
|
|
<h2 id="2019-07-26">2019-07-26</h2>
|
|
|
|
<ul>
|
|
<li><p>Bioversity sent me an updated CSV file that fixes some of the issues I pointed out yesterday</p>
|
|
|
|
<ul>
|
|
<li>There are still 1429 records</li>
|
|
<li>There are still one extra row and one extra column</li>
|
|
<li>There are still eight invalid ISBNs (according to my <code>validate.py</code> script)</li>
|
|
</ul></li>
|
|
|
|
<li><p>I figured out a GREL to trim spaces in multi-value cells without splitting them:</p>
|
|
|
|
<pre><code>value.replace(/\s+\|\|/,"||").replace(/\|\|\s+/,"||")
|
|
</code></pre></li>
|
|
|
|
<li><p>I whipped up a quick script using Python Pandas to do whitespace cleanup</p></li>
|
|
</ul>
|
|
|
|
<!-- vim: set sw=2 ts=2: -->
|
|
|
|
|
|
|
|
|
|
|
|
</article>
|
|
|
|
|
|
|
|
</div> <!-- /.blog-main -->
|
|
|
|
<aside class="col-sm-3 ml-auto blog-sidebar">
|
|
|
|
|
|
|
|
<section class="sidebar-module">
|
|
<h4>Recent Posts</h4>
|
|
<ol class="list-unstyled">
|
|
|
|
|
|
<li><a href="/cgspace-notes/2019-07/">July, 2019</a></li>
|
|
|
|
<li><a href="/cgspace-notes/posts/">Posts</a></li>
|
|
|
|
<li><a href="/cgspace-notes/2019-06/">June, 2019</a></li>
|
|
|
|
<li><a href="/cgspace-notes/2019-05/">May, 2019</a></li>
|
|
|
|
<li><a href="/cgspace-notes/2019-04/">April, 2019</a></li>
|
|
|
|
</ol>
|
|
</section>
|
|
|
|
|
|
|
|
|
|
<section class="sidebar-module">
|
|
<h4>Links</h4>
|
|
<ol class="list-unstyled">
|
|
|
|
<li><a href="https://cgspace.cgiar.org">CGSpace</a></li>
|
|
|
|
<li><a href="https://dspacetest.cgiar.org">DSpace Test</a></li>
|
|
|
|
<li><a href="https://github.com/ilri/DSpace">CGSpace @ GitHub</a></li>
|
|
|
|
</ol>
|
|
</section>
|
|
|
|
</aside>
|
|
|
|
|
|
</div> <!-- /.row -->
|
|
</div> <!-- /.container -->
|
|
|
|
|
|
|
|
<footer class="blog-footer">
|
|
<p>
|
|
|
|
Blog template created by <a href="https://twitter.com/mdo">@mdo</a>, ported to Hugo by <a href='https://twitter.com/mralanorth'>@mralanorth</a>.
|
|
|
|
</p>
|
|
<p>
|
|
<a href="#">Back to top</a>
|
|
</p>
|
|
</footer>
|
|
|
|
|
|
</body>
|
|
|
|
</html>
|