cgspace-notes/docs/2018-06/index.html

458 lines
20 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<meta property="og:title" content="June, 2018" />
<meta property="og:description" content="2018-06-04
Test the DSpace 5.8 module upgrades from Atmire (#378)
There seems to be a problem with the CUA and L&amp;R versions in pom.xml because they are using SNAPSHOT and it doesn&rsquo;t build
I added the new CCAFS Phase II Project Tag PII-FP1_PACCA2 and merged it into the 5_x-prod branch (#379)
I proofed and tested the ILRI author corrections that Peter sent back to me this week:
$ ./fix-metadata-values.py -i /tmp/2018-05-30-Correct-660-authors.csv -db dspace -u dspace -p &#39;fuuu&#39; -f dc.contributor.author -t correct -m 3 -n
I think a sane proofing workflow in OpenRefine is to apply the custom text facets for check/delete/remove and illegal characters that I developed in March, 2018
Time to index ~70,000 items on CGSpace:
$ time schedtool -D -e ionice -c2 -n7 nice -n19 [dspace]/bin/dspace index-discovery -b
real 74m42.646s
user 8m5.056s
sys 2m7.289s
" />
<meta property="og:type" content="article" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2018-06/" />
<meta property="article:published_time" content="2018-06-04T19:49:54-07:00"/>
<meta property="article:modified_time" content="2018-06-11T15:21:14&#43;03:00"/>
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="June, 2018"/>
<meta name="twitter:description" content="2018-06-04
Test the DSpace 5.8 module upgrades from Atmire (#378)
There seems to be a problem with the CUA and L&amp;R versions in pom.xml because they are using SNAPSHOT and it doesn&rsquo;t build
I added the new CCAFS Phase II Project Tag PII-FP1_PACCA2 and merged it into the 5_x-prod branch (#379)
I proofed and tested the ILRI author corrections that Peter sent back to me this week:
$ ./fix-metadata-values.py -i /tmp/2018-05-30-Correct-660-authors.csv -db dspace -u dspace -p &#39;fuuu&#39; -f dc.contributor.author -t correct -m 3 -n
I think a sane proofing workflow in OpenRefine is to apply the custom text facets for check/delete/remove and illegal characters that I developed in March, 2018
Time to index ~70,000 items on CGSpace:
$ time schedtool -D -e ionice -c2 -n7 nice -n19 [dspace]/bin/dspace index-discovery -b
real 74m42.646s
user 8m5.056s
sys 2m7.289s
"/>
<meta name="generator" content="Hugo 0.40.3" />
<script type="application/ld+json">
{
"@context": "http://schema.org",
"@type": "BlogPosting",
"headline": "June, 2018",
"url": "https://alanorth.github.io/cgspace-notes/2018-06/",
"wordCount": "1462",
"datePublished": "2018-06-04T19:49:54-07:00",
"dateModified": "2018-06-11T15:21:14&#43;03:00",
"author": {
"@type": "Person",
"name": "Alan Orth"
},
"keywords": "Notes"
}
</script>
<link rel="canonical" href="https://alanorth.github.io/cgspace-notes/2018-06/">
<title>June, 2018 | CGSpace Notes</title>
<!-- combined, minified CSS -->
<link href="https://alanorth.github.io/cgspace-notes/css/style.css" rel="stylesheet" integrity="sha384-ZwlQQbzhEPf3PrXZ3h/XKT/4UHafQ/TYI72AL&#43;7WOJ8D6JmpGC8JMMse6xX7cyeI" crossorigin="anonymous">
</head>
<body>
<div class="blog-masthead">
<div class="container">
<nav class="nav blog-nav">
<a class="nav-link " href="https://alanorth.github.io/cgspace-notes/">Home</a>
</nav>
</div>
</div>
<header class="blog-header">
<div class="container">
<h1 class="blog-title"><a href="https://alanorth.github.io/cgspace-notes/" rel="home">CGSpace Notes</a></h1>
<p class="lead blog-description">Documenting day-to-day work on the <a href="https://cgspace.cgiar.org">CGSpace</a> repository.</p>
</div>
</header>
<div class="container">
<div class="row">
<div class="col-sm-8 blog-main">
<article class="blog-post">
<header>
<h2 class="blog-post-title"><a href="https://alanorth.github.io/cgspace-notes/2018-06/">June, 2018</a></h2>
<p class="blog-post-meta"><time datetime="2018-06-04T19:49:54-07:00">Mon Jun 04, 2018</time> by Alan Orth in
<i class="fa fa-tag" aria-hidden="true"></i>&nbsp;<a href="/cgspace-notes/tags/notes" rel="tag">Notes</a>
</p>
</header>
<h2 id="2018-06-04">2018-06-04</h2>
<ul>
<li>Test the <a href="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=560">DSpace 5.8 module upgrades from Atmire</a> (<a href="https://github.com/ilri/DSpace/pull/378">#378</a>)
<ul>
<li>There seems to be a problem with the CUA and L&amp;R versions in <code>pom.xml</code> because they are using SNAPSHOT and it doesn&rsquo;t build</li>
</ul></li>
<li>I added the new CCAFS Phase II Project Tag <code>PII-FP1_PACCA2</code> and merged it into the <code>5_x-prod</code> branch (<a href="https://github.com/ilri/DSpace/pull/379">#379</a>)</li>
<li>I proofed and tested the ILRI author corrections that Peter sent back to me this week:</li>
</ul>
<pre><code>$ ./fix-metadata-values.py -i /tmp/2018-05-30-Correct-660-authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t correct -m 3 -n
</code></pre>
<ul>
<li>I think a sane proofing workflow in OpenRefine is to apply the custom text facets for check/delete/remove and illegal characters that I developed in <a href="/cgspace-notes/2018-03/">March, 2018</a></li>
<li>Time to index ~70,000 items on CGSpace:</li>
</ul>
<pre><code>$ time schedtool -D -e ionice -c2 -n7 nice -n19 [dspace]/bin/dspace index-discovery -b
real 74m42.646s
user 8m5.056s
sys 2m7.289s
</code></pre>
<p></p>
<h2 id="2018-06-06">2018-06-06</h2>
<ul>
<li>It turns out that I needed to add a server block for <code>atmire.com-snapshots</code> to my Maven settings, so now the Atmire code builds</li>
<li>Now Maven and Ant run properly, but I&rsquo;m getting SQL migration errors in <code>dspace.log</code> after starting Tomcat</li>
<li>I&rsquo;ve updated my ticket on Atmire&rsquo;s bug tracker: <a href="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=560">https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=560</a></li>
</ul>
<h2 id="2018-06-07">2018-06-07</h2>
<ul>
<li>Proofing 200 IITA records on DSpace Test for Sisay: <a href="https://dspacetest.cgiar.org/handle/10568/95391">IITA_Junel_06 (<sup>10568</sup>&frasl;<sub>95391</sub>)</a>
<ul>
<li>Mispelled authorship type: CGAIR single center should be: CGIAR single centre</li>
<li>I see some encoding errors in author affiliations, for example:</li>
<li>Universidade de SÆo Paulo</li>
<li>Institut National des Recherches Agricoles du B nin</li>
<li>Centre de Coop ration Internationale en Recherche Agronomique pour le D veloppement</li>
<li>Institut des Recherches Agricoles du B nin</li>
<li>Institut des Savannes, C te d&rsquo; Ivoire</li>
<li>Institut f r Pflanzenpathologie und Pflanzenschutz der Universit t, Germany</li>
<li>Projet de Gestion des Ressources Naturelles, B nin</li>
<li>Universit t Hannover</li>
<li>Universit F lix Houphouet-Boigny</li>
</ul></li>
<li>I uploaded fixes for all those now, but I will continue with the rest of the data later</li>
<li>Regarding the SQL migration errors, Atmire told me I need to run some migrations manually in PostgreSQL:</li>
</ul>
<pre><code>delete from schema_version where version = '5.6.2015.12.03.2';
update schema_version set version = '5.6.2015.12.03.2' where version = '5.5.2015.12.03.2';
update schema_version set version = '5.8.2015.12.03.3' where version = '5.5.2015.12.03.3';
</code></pre>
<ul>
<li>And then I need to ignore the ignored ones:</li>
</ul>
<pre><code>$ ~/dspace/bin/dspace database migrate ignored
</code></pre>
<ul>
<li>Now DSpace starts up properly!</li>
<li>Gabriela from CIP got back to me about the author names we were correcting on CGSpace</li>
<li>I did a quick sanity check on them and then did a test import with my <a href="https://gist.github.com/alanorth/df92cbfb54d762ba21b28f7cd83b6897"><code>fix-metadata-value.py</code></a> script:</li>
</ul>
<pre><code>$ ./fix-metadata-values.py -i /tmp/2018-06-08-CIP-Authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t correct -m 3
</code></pre>
<ul>
<li>I will apply them on CGSpace tomorrow I think&hellip;</li>
</ul>
<h2 id="2018-06-09">2018-06-09</h2>
<ul>
<li>It&rsquo;s pretty annoying, but the JVM monitoring for Munin was never set up when I migrated DSpace Test to its new server a few months ago</li>
<li>I ran the tomcat and munin-node tags in Ansible again and now the stuff is all wired up and recording stats properly</li>
<li>I applied the CIP author corrections on CGSpace and DSpace Test and re-ran the Discovery indexing</li>
</ul>
<h2 id="2018-06-10">2018-06-10</h2>
<ul>
<li>I spent some time removing the Atmire Metadata Quality Module (MQM) from the proposed DSpace 5.8 changes</li>
<li>After removing all code mentioning MQM, mqm, metadata-quality, batchedit, duplicatechecker, etc, I think I got most of it removed, but there is a Spring error during Tomcat startup:</li>
</ul>
<pre><code> INFO [org.dspace.servicemanager.DSpaceServiceManager] Shutdown DSpace core service manager
Failed to startup the DSpace Service Manager: failure starting up spring service manager: Error creating bean with name 'org.dspace.servicemanager.spring.DSpaceBeanPostProcessor#0' defined in class path resource [spring/spring-dspace-applicationContext.xml]: Unsatisfied dependency expressed through constructor argument with index 0 of type [org.dspace.servicemanager.config.DSpaceConfigurationService]: : Cannot find class [com.atmire.dspace.discovery.ItemCollectionPlugin] for bean with name 'itemCollectionPlugin' defined in file [/home/aorth/dspace/config/spring/api/discovery.xml];
</code></pre>
<ul>
<li>I can fix this by commenting out the <code>ItemCollectionPlugin</code> line of <code>discovery.xml</code>, but from looking at the git log I&rsquo;m not actually sure if that is related to MQM or not</li>
<li>I will have to ask Atmire</li>
<li>I continued to look at Sisay&rsquo;s IITA records from last week
<ul>
<li>I normalized all DOIs to use HTTPS and &ldquo;doi.org&rdquo; instead of &ldquo;dx.doi.org&rdquo;</li>
<li>I cleaned up white space in <code>cg.subject.iita</code> and <code>dc.subject</code></li>
<li>Even a bunch of IITA and AGROVOC subjects are missing accents, ie &ldquo;FERTILIT DU SOL&rdquo;</li>
<li>More organization names in <code>dc.description.sponsorship</code> are incorrect (ie, missing accents) or inconsistent (ie, CGIAR centers should be spelled in English or multiple spellings of the same one, like &ldquo;Rockefeller Foundation&rdquo; and &ldquo;Rockefeller foundation&rdquo;)</li>
<li>A few dozen items have abstracts with character encoding errors, ie:</li>
<li>33.7øC</li>
<li>MgSO4ú7H2O</li>
<li>ha??1&amp;/sup;</li>
<li>En gen6ral</li>
<li>dÕpassÕ</li>
<li>Also the abstracts have missing accents, ie &ldquo;recherche sur le d veloppement&rdquo;</li>
</ul></li>
<li>I will have to tell IITA people to redo these entirely I think&hellip;</li>
</ul>
<h2 id="2018-06-11">2018-06-11</h2>
<ul>
<li>Sisay sent a new version of the last IITA records that he created from the original CSV from IITA</li>
<li>The 200 records are in the <a href="https://dspacetest.cgiar.org/handle/10568/95870">IITA_Junel_11 (<sup>10568</sup>&frasl;<sub>95870</sub>)</a> collection</li>
<li>Many errors:
<ul>
<li>Authorship types: &ldquo;CGIAR ans advanced research institute&rdquo;, &ldquo;CGAIR and advanced research institute&rdquo;, &ldquo;CGIAR and advanced research institutes&rdquo;, &ldquo;CGAIR single center&rdquo;</li>
<li>Lots of inconsistencies and mispellings in author affiliations:</li>
<li>&ldquo;Institut des Recherches Agricoles du Bénin&rdquo; and &ldquo;Institut National des Recherche Agricoles du Benin&rdquo; and &ldquo;National Agricultural Research Institute, Benin&rdquo;</li>
<li>International Insitute of Tropical Agriculture</li>
<li>Centro Internacional de Agricultura Tropical</li>
<li>&ldquo;Rivers State University of Science and Technology&rdquo; and &ldquo;Rivers State University&rdquo;</li>
<li>&ldquo;Institut de la Recherche Agronomique, Cameroon&rdquo; and &ldquo;Institut de Recherche Agronomique, Cameroon&rdquo;</li>
<li>Inconsistency in countries: &ldquo;COTE DIVOIRE&rdquo; and &ldquo;COTE D&rsquo;IVOIRE&rdquo;</li>
<li>A few DOIs with spaces or invalid characters</li>
<li>Inconsistency in IITA subjects, for example &ldquo;PRODUCTION VEGETALE&rdquo; and &ldquo;PRODUCTION VÉGÉTALE&rdquo; and several others</li>
<li>I ran <code>value.unescape('javascript')</code> on the abstract and citation fields because it looks like this data came from a SQL database and some stuff was escaped</li>
</ul></li>
<li>It turns out that Abenet actually did a lot of small corrections on this data so when Sisay uses Bosede&rsquo;s original file it doesn&rsquo;t have all those corrections</li>
<li>So I told Sisay to re-create the collection using Abenet&rsquo;s XLS from last week (<code>Mercy1805_AY.xls</code>)</li>
<li>I was curious to see if I could create a GREL for use with a custom text facet in Open Refine to find cells with two or more consecutive spaces</li>
<li>I always use the built-in trim and collapse transformations anyways, but this seems to work to find the offending cells: <code>isNotNull(value.match(/.*?\s{2,}.*?/))</code></li>
<li>I wonder if I should start checking for &ldquo;smart&rdquo; quotes like (hex 2019)</li>
</ul>
<h2 id="2018-06-12">2018-06-12</h2>
<ul>
<li>Udana from IWMI asked about the OAI base URL for their community on CGSpace</li>
<li>I think it should be this: <a href="https://cgspace.cgiar.org/oai/request?verb=ListRecords&amp;metadataPrefix=oai_dc&amp;set=com_10568_16814">https://cgspace.cgiar.org/oai/request?verb=ListRecords&amp;metadataPrefix=oai_dc&amp;set=com_10568_16814</a></li>
<li>The style sheet obfuscates the data, but if you look at the source it is all there, including information about pagination of results</li>
<li>Regarding Udana&rsquo;s Book Chapters and Reports on DSpace Test last week, Abenet told him to fix some character encoding and CRP issues, then I told him I&rsquo;d check them after that</li>
<li>The latest batch of IITA&rsquo;s 200 records (based on Abenet&rsquo;s version <code>Mercy1805_AY.xls</code>) are now in the <a href="https://dspacetest.cgiar.org/handle/10568/96071">IITA_Jan_9_II_Ab</a> collection</li>
<li>So here are some corrections:
<ul>
<li>use of Unicode smart quote (hex 2019) in countries and affiliations, for example &ldquo;COTE DIVOIRE&rdquo; and &ldquo;Institut dEconomic Rurale, Mali&rdquo;</li>
<li>inconsistencies in <code>cg.contributor.affiliation</code>:</li>
<li>&ldquo;Centro Internacional de Agricultura Tropical&rdquo; and &ldquo;Centro International de Agricultura Tropical&rdquo; should use the English name of CIAT (International Center for Tropical Agriculture)</li>
<li>&ldquo;Institut International d&rsquo;Agriculture Tropicale&rdquo; should use the English name of IITA (International Institute of Tropical Agriculture)</li>
<li>&ldquo;East and Southern Africa Regional Center&rdquo; and &ldquo;Eastern and Southern Africa Regional Centre&rdquo;</li>
<li>&ldquo;Institut de la Recherche Agronomique, Cameroon&rdquo; and &ldquo;Institut de Recherche Agronomique, Cameroon&rdquo;</li>
<li>&ldquo;Institut des Recherches Agricoles du Bénin&rdquo; and &ldquo;Institut National des Recherche Agricoles du Benin&rdquo; and &ldquo;National Agricultural Research Institute, Benin&rdquo;</li>
<li>&ldquo;Institute of Agronomic Research, Cameroon&rdquo; and &ldquo;Institute of Agronomy Research, Cameroon&rdquo;</li>
<li>&ldquo;Rivers State University&rdquo; and &ldquo;Rivers State University of Science and Technology&rdquo;</li>
<li>&ldquo;Universität Hannover&rdquo; and &ldquo;University of Hannover&rdquo;</li>
<li>inconsistencies in <code>cg.subject.iita</code>:</li>
<li>&ldquo;AMELIORATION DES PLANTES&rdquo; and &ldquo;AMÉLIORATION DES PLANTES&rdquo;</li>
<li>&ldquo;PRODUCTION VEGETALE&rdquo; and &ldquo;PRODUCTION VÉGÉTALE&rdquo;</li>
<li>&ldquo;CONTRÔLE DE MALADIES&rdquo; and &ldquo;CONTROLE DES MALADIES&rdquo;</li>
<li>&ldquo;HANDLING, TRANSPORT, STORAGE AND PROTECTION OF AGRICULTURAL PRODUCT&rdquo; and &ldquo;HANDLING, TRANSPORT, STORAGE AND PROTECTION OF AGRICULTURAL PRODUCTS&rdquo;</li>
<li>&ldquo;RAVAGEURS DE PLANTES&rdquo; and &ldquo;RAVAGEURS DES PLANTES&rdquo;</li>
<li>&ldquo;SANTE DES PLANTES&rdquo; and &ldquo;SANTÉ DES PLANTES&rdquo;</li>
<li>&ldquo;SOCIOECONOMIE&rdquo; and &ldquo;SOCIOECONOMY&rdquo;</li>
<li>inconsistencies in <code>dc.description.sponsorship</code>:</li>
<li>&ldquo;Belgian Corporation&rdquo; and &ldquo;Belgium Corporation&rdquo;</li>
<li>inconsistencies in <code>dc.subject</code>:</li>
<li>&ldquo;AFRICAN CASSAVA MOSAIC&rdquo; and &ldquo;AFRICAN CASSAVA MOSAIC DISEASE&rdquo;</li>
<li>&ldquo;ASPERGILLU FLAVUS&rdquo; and &ldquo;ASPERGILLUS FLAVUS&rdquo;</li>
<li>&ldquo;BIOTECHNOLOGIES&rdquo; and &ldquo;BIOTECHNOLOGY&rdquo;</li>
<li>&ldquo;CASSAVA MOSAIC DISEASE&rdquo; and &ldquo;CASSAVA MOSAIC DISEASES&rdquo; and &ldquo;CASSAVA MOSAIC VIRUS&rdquo;</li>
<li>&ldquo;CASSAVA PROCESSING&rdquo; and &ldquo;CASSAVA PROCESSING TECHNOLOGY&rdquo;</li>
<li>&ldquo;CROPPING SYSTEM&rdquo; and &ldquo;CROPPING SYSTEMS&rdquo;</li>
<li>&ldquo;DRY SEASON&rdquo; and &ldquo;DRY-SEASON&rdquo;</li>
<li>&ldquo;FERTILIZER&rdquo; and &ldquo;FERTILIZERS&rdquo;</li>
<li>&ldquo;LEGUME&rdquo; and &ldquo;LEGUMES&rdquo;</li>
<li>&ldquo;LEGUMINOSAE&rdquo; and &ldquo;LEGUMINOUS&rdquo;</li>
<li>&ldquo;LEGUMINOUS COVER CROP&rdquo; and &ldquo;LEGUMINOUS COVER CROPS&rdquo;</li>
<li>&ldquo;MATÉRIEL DE PLANTATION&rdquo; and &ldquo;MATÉRIELS DE PLANTATION&rdquo;</li>
<li>I noticed that some records do have encoding errors in the <code>dc.description.abstract</code> field, but only four of them so probably not from Abenet&rsquo;s handling of the XLS file</li>
<li>Based on manually eyeballing the text I used a custom text facet with this GREL to identify the records:</li>
</ul></li>
</ul>
<pre><code>or(
value.contains('€'),
value.contains('6g'),
value.contains('6m'),
value.contains('6d'),
value.contains('6e')
)
</code></pre>
<ul>
<li>So IITA should double check the abstracts for these:
<ul>
<li><a href="https://dspacetest.cgiar.org/10568/96184">https://dspacetest.cgiar.org/10568/96184</a></li>
<li><a href="https://dspacetest.cgiar.org/10568/96141">https://dspacetest.cgiar.org/10568/96141</a></li>
<li><a href="https://dspacetest.cgiar.org/10568/96118">https://dspacetest.cgiar.org/10568/96118</a></li>
<li><a href="https://dspacetest.cgiar.org/10568/96113">https://dspacetest.cgiar.org/10568/96113</a>
<br /></li>
</ul></li>
</ul>
<h1 id="vim-set-sw-2-ts-2">vim: set sw=2 ts=2:</h1>
</article>
</div> <!-- /.blog-main -->
<aside class="col-sm-3 ml-auto blog-sidebar">
<section class="sidebar-module">
<h4>Recent Posts</h4>
<ol class="list-unstyled">
<li><a href="/cgspace-notes/2018-06/">June, 2018</a></li>
<li><a href="/cgspace-notes/2018-05/">May, 2018</a></li>
<li><a href="/cgspace-notes/2018-04/">April, 2018</a></li>
<li><a href="/cgspace-notes/2018-03/">March, 2018</a></li>
<li><a href="/cgspace-notes/2018-02/">February, 2018</a></li>
</ol>
</section>
<section class="sidebar-module">
<h4>Links</h4>
<ol class="list-unstyled">
<li><a href="https://cgspace.cgiar.org">CGSpace</a></li>
<li><a href="https://dspacetest.cgiar.org">DSpace Test</a></li>
<li><a href="https://github.com/ilri/DSpace">CGSpace @ GitHub</a></li>
</ol>
</section>
</aside>
</div> <!-- /.row -->
</div> <!-- /.container -->
<footer class="blog-footer">
<p>
Blog template created by <a href="https://twitter.com/mdo">@mdo</a>, ported to Hugo by <a href='https://twitter.com/mralanorth'>@mralanorth</a>.
</p>
<p>
<a href="#">Back to top</a>
</p>
</footer>
</body>
</html>