mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2024-11-23 07:00:20 +01:00
448 lines
17 KiB
HTML
448 lines
17 KiB
HTML
<!DOCTYPE html>
|
||
<html lang="en" >
|
||
|
||
<head>
|
||
<meta charset="utf-8">
|
||
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
|
||
|
||
<meta property="og:title" content="October, 2019" />
|
||
<meta property="og:description" content="2019-10-01 Udana from IWMI asked me for a CSV export of their community on CGSpace
|
||
I exported it, but a quick run through the csv-metadata-quality tool shows that there are some low-hanging fruits we can fix before I send him the data I will limit the scope to the titles, regions, subregions, and river basins for now to manually fix some non-breaking spaces (U+00A0) there that would otherwise be removed by the csv-metadata-quality script’s “unneccesary Unicode” fix:" />
|
||
<meta property="og:type" content="article" />
|
||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2019-10/" />
|
||
<meta property="article:published_time" content="2019-10-01T13:20:51+03:00" />
|
||
<meta property="article:modified_time" content="2019-10-15T18:58:36+03:00" />
|
||
|
||
<meta name="twitter:card" content="summary"/>
|
||
<meta name="twitter:title" content="October, 2019"/>
|
||
<meta name="twitter:description" content="2019-10-01 Udana from IWMI asked me for a CSV export of their community on CGSpace
|
||
I exported it, but a quick run through the csv-metadata-quality tool shows that there are some low-hanging fruits we can fix before I send him the data I will limit the scope to the titles, regions, subregions, and river basins for now to manually fix some non-breaking spaces (U+00A0) there that would otherwise be removed by the csv-metadata-quality script’s “unneccesary Unicode” fix:"/>
|
||
<meta name="generator" content="Hugo 0.58.3" />
|
||
|
||
|
||
|
||
<script type="application/ld+json">
|
||
{
|
||
"@context": "http://schema.org",
|
||
"@type": "BlogPosting",
|
||
"headline": "October, 2019",
|
||
"url": "https:\/\/alanorth.github.io\/cgspace-notes\/2019-10\/",
|
||
"wordCount": "1478",
|
||
"datePublished": "2019-10-01T13:20:51+03:00",
|
||
"dateModified": "2019-10-15T18:58:36+03:00",
|
||
"author": {
|
||
"@type": "Person",
|
||
"name": "Alan Orth"
|
||
},
|
||
"keywords": "Notes"
|
||
}
|
||
</script>
|
||
|
||
|
||
|
||
<link rel="canonical" href="https://alanorth.github.io/cgspace-notes/2019-10/">
|
||
|
||
<title>October, 2019 | CGSpace Notes</title>
|
||
|
||
|
||
<!-- combined, minified CSS -->
|
||
<link href="https://alanorth.github.io/cgspace-notes/css/style.css" rel="stylesheet" integrity="sha384-G5B34w7DFTumWTswxYzTX7NWfbvQEg1HbFFEg6ItN03uTAAoS2qkPS/fu3LhuuSA" crossorigin="anonymous">
|
||
|
||
|
||
<!-- RSS 2.0 feed -->
|
||
|
||
|
||
|
||
|
||
|
||
|
||
</head>
|
||
|
||
<body>
|
||
|
||
|
||
<div class="blog-masthead">
|
||
<div class="container">
|
||
<nav class="nav blog-nav">
|
||
<a class="nav-link " href="https://alanorth.github.io/cgspace-notes/">Home</a>
|
||
</nav>
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
<header class="blog-header">
|
||
<div class="container">
|
||
<h1 class="blog-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/" rel="home">CGSpace Notes</a></h1>
|
||
<p class="lead blog-description" dir="auto">Documenting day-to-day work on the <a href="https://cgspace.cgiar.org">CGSpace</a> repository.</p>
|
||
</div>
|
||
</header>
|
||
|
||
|
||
|
||
|
||
<div class="container">
|
||
<div class="row">
|
||
<div class="col-sm-8 blog-main">
|
||
|
||
|
||
|
||
|
||
<article class="blog-post">
|
||
<header>
|
||
<h2 class="blog-post-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/2019-10/">October, 2019</a></h2>
|
||
<p class="blog-post-meta"><time datetime="2019-10-01T13:20:51+03:00">Tue Oct 01, 2019</time> by Alan Orth in
|
||
|
||
<i class="fa fa-tag" aria-hidden="true"></i> <a href="/cgspace-notes/tags/notes" rel="tag">Notes</a>
|
||
|
||
</p>
|
||
</header>
|
||
|
||
|
||
<h2 id="2019-10-01">2019-10-01</h2>
|
||
|
||
<ul>
|
||
<li><p>Udana from IWMI asked me for a CSV export of their community on CGSpace</p>
|
||
|
||
<ul>
|
||
<li>I exported it, but a quick run through the <code>csv-metadata-quality</code> tool shows that there are some low-hanging fruits we can fix before I send him the data</li>
|
||
|
||
<li><p>I will limit the scope to the titles, regions, subregions, and river basins for now to manually fix some non-breaking spaces (U+00A0) there that would otherwise be removed by the csv-metadata-quality script’s “unneccesary Unicode” fix:</p>
|
||
|
||
<pre><code>$ csvcut -c 'id,dc.title[en_US],cg.coverage.region[en_US],cg.coverage.subregion[en_US],cg.river.basin[en_US]' ~/Downloads/10568-16814.csv > /tmp/iwmi-title-region-subregion-river.csv
|
||
</code></pre></li>
|
||
|
||
<li><p>Then I replace them in vim with <code>:% s/\%u00a0/ /g</code> because I can’t figure out the correct sed syntax to do it directly from the pipe above</p></li>
|
||
|
||
<li><p>I uploaded those to CGSpace and then re-exported the metadata</p></li>
|
||
|
||
<li><p>Now that I think about it, I shouldn’t be removing non-breaking spaces (U+00A0), I should be replacing them with normal spaces!</p></li>
|
||
|
||
<li><p>I modified the script so it replaces the non-breaking spaces instead of removing them</p></li>
|
||
|
||
<li><p>Then I ran the csv-metadata-quality script to do some general cleanups (though I temporarily commented out the whitespace fixes because it was too many thousands of rows):</p>
|
||
|
||
<pre><code>$ csv-metadata-quality -i ~/Downloads/10568-16814.csv -o /tmp/iwmi.csv -x 'dc.date.issued,dc.date.issued[],dc.date.issued[en_US]' -u
|
||
</code></pre></li>
|
||
|
||
<li><p>That fixed 153 items (unnecessary Unicode, duplicates, comma–space fixes, etc)</p></li>
|
||
</ul></li>
|
||
|
||
<li><p>Release <a href="https://github.com/ilri/csv-metadata-quality/releases/tag/v0.3.1">version 0.3.1 of the csv-metadata-quality script</a> with the non-breaking spaces change</p></li>
|
||
</ul>
|
||
|
||
<h2 id="2019-10-03">2019-10-03</h2>
|
||
|
||
<ul>
|
||
<li>Upload the 117 IITA records that we had been working on last month (aka 20196th.xls aka Sept 6) to CGSpace</li>
|
||
</ul>
|
||
|
||
<h2 id="2019-10-04">2019-10-04</h2>
|
||
|
||
<ul>
|
||
<li><p>Create an account for Bioversity’s ICT consultant Francesco on DSpace Test:</p>
|
||
|
||
<pre><code>$ dspace user -a -m blah@mail.it -g Francesco -s Vernocchi -p 'fffff'
|
||
</code></pre></li>
|
||
|
||
<li><p>Email Francesca and Carol to ask for follow up about the test upload I did on 2019-09-21</p>
|
||
|
||
<ul>
|
||
<li>I suggested that if they still want to do value addition of those records (like adding countries, regions, etc) that they could maybe do it after we migrate the records to CGSpace</li>
|
||
<li>Carol responded to tell me where to map the items with type Brochure, Journal Item, and Thesis, so I applied them to the <a href="https://dspacetest.cgiar.org/handle/10568/103688">collection on DSpace Test</a></li>
|
||
</ul></li>
|
||
</ul>
|
||
|
||
<h2 id="2019-10-06">2019-10-06</h2>
|
||
|
||
<ul>
|
||
<li>Hector from CCAFS responded about my feedback of their CLARISA API
|
||
|
||
<ul>
|
||
<li>He made some fixes to the metadata values they are using based on my feedback and said they are happy if we would use it</li>
|
||
</ul></li>
|
||
<li>Gabriela from CIP asked me if it was possible to generate an RSS feed of items that have the CIP subject “POTATO AGRI-FOOD SYSTEMS”
|
||
|
||
<ul>
|
||
<li>I notice that there is a similar term “SWEETPOTATO AGRI-FOOD SYSTEMS” so I had to come up with a way to exclude that using the boolean “AND NOT” in the <a href="https://cgspace.cgiar.org/open-search/discover?query=cipsubject:POTATO%20AGRI%E2%80%90FOOD%20SYSTEMS%20AND%20NOT%20cipsubject:SWEETPOTATO%20AGRI%E2%80%90FOOD%20SYSTEMS&scope=10568/51671&sort_by=3&order=DESC">OpenSearch query</a></li>
|
||
<li>Again, the <code>sort_by=3</code> parameter is the accession date, as configured in <code>dspace.cfg</code></li>
|
||
</ul></li>
|
||
</ul>
|
||
|
||
<h2 id="2019-10-08">2019-10-08</h2>
|
||
|
||
<ul>
|
||
<li>Fix 108 more issues with authors in the ongoing Bioversity migration on DSpace Test, for example:
|
||
|
||
<ul>
|
||
<li>Europeanooperative Programme for Plant Genetic Resources</li>
|
||
<li>Bioversity International. Capacity Development Unit</li>
|
||
<li>W.M. van der Heide, W.M., Tripp, R.</li>
|
||
<li>Internationallant Genetic Resources Institute</li>
|
||
</ul></li>
|
||
<li>Start looking at duplicates in the Bioversity migration data on DSpace Test
|
||
|
||
<ul>
|
||
<li>I’m keeping track of the originals and duplicates in a Google Docs spreadsheet that I will share with Bioversity</li>
|
||
</ul></li>
|
||
</ul>
|
||
|
||
<h2 id="2019-10-09">2019-10-09</h2>
|
||
|
||
<ul>
|
||
<li>Continue working on identifying duplicates in the Bioversity migration
|
||
|
||
<ul>
|
||
<li>I have been recording the originals and duplicates in a spreadsheet so I can map them later</li>
|
||
<li>For now I am just reconciling any incorrect or missing metadata in the original items on CGSpace, deleting the duplicate from DSpace Test, and mapping the original to the correct place on CGSpace</li>
|
||
<li>So far I have deleted thirty duplicates and mapped fourteen</li>
|
||
</ul></li>
|
||
<li>Run all system updates on DSpace Test (linode19) and reboot the server</li>
|
||
</ul>
|
||
|
||
<h2 id="2019-10-10">2019-10-10</h2>
|
||
|
||
<ul>
|
||
<li><p>Felix Shaw from Earlham emailed me to ask about his admin account on DSpace Test</p>
|
||
|
||
<ul>
|
||
<li>His old one got lost when I re-sync’d DSpace Test with CGSpace a few weeks ago</li>
|
||
|
||
<li><p>I added a new account for him and added it to the Administrators group:</p>
|
||
|
||
<pre><code>$ dspace user -a -m wow@me.com -g Felix -s Shaw -p 'fuananaaa'
|
||
</code></pre></li>
|
||
</ul></li>
|
||
</ul>
|
||
|
||
<h2 id="2019-10-11">2019-10-11</h2>
|
||
|
||
<ul>
|
||
<li><p>I ran the DSpace cleanup function on CGSpace and it found some errors:</p>
|
||
|
||
<pre><code>$ dspace cleanup -v
|
||
...
|
||
Error: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
|
||
Detail: Key (bitstream_id)=(171221) is still referenced from table "bundle".
|
||
</code></pre></li>
|
||
|
||
<li><p>The solution, as always, is (repeat as many times as needed):</p>
|
||
|
||
<pre><code># su - postgres
|
||
$ psql dspace -c 'update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (171221);'
|
||
UPDATE 1
|
||
</code></pre></li>
|
||
</ul>
|
||
|
||
<h2 id="2019-10-12">2019-10-12</h2>
|
||
|
||
<ul>
|
||
<li>More work on identifying duplicates in the Bioversity migration data on DSpace Test
|
||
|
||
<ul>
|
||
<li>I mapped twenty-five more items on CGSpace and deleted them from the migration test collection on DSpace Test</li>
|
||
<li>After a few hours I think I finished all the duplicates that were identified by Atmire’s Duplicate Checker module</li>
|
||
<li>According to my spreadsheet there were fifty-two in total</li>
|
||
</ul></li>
|
||
|
||
<li><p>I was preparing to check the affiliations on the Bioversity records when I noticed that the last list of top affiliations I generated has some anomalies</p>
|
||
|
||
<ul>
|
||
<li><p>I made some corrections in a CSV:</p>
|
||
|
||
<pre><code>from,to
|
||
CIAT,International Center for Tropical Agriculture
|
||
International Centre for Tropical Agriculture,International Center for Tropical Agriculture
|
||
International Maize and Wheat Improvement Center (CIMMYT),International Maize and Wheat Improvement Center
|
||
International Centre for Agricultural Research in the Dry Areas,International Center for Agricultural Research in the Dry Areas
|
||
International Maize and Wheat Improvement Centre,International Maize and Wheat Improvement Center
|
||
"Agricultural Information Resource Centre, Kenya.","Agricultural Information Resource Centre, Kenya"
|
||
"Centre for Livestock and Agricultural Development, Cambodia","Centre for Livestock and Agriculture Development, Cambodia"
|
||
</code></pre></li>
|
||
</ul></li>
|
||
|
||
<li><p>Then I applied it with my <code>fix-metadata-values.py</code> script on CGSpace:</p>
|
||
|
||
<pre><code>$ ./fix-metadata-values.py -i /tmp/affiliations.csv -db dspace -u dspace -p 'fuuu' -f from -m 211 -t to
|
||
</code></pre></li>
|
||
|
||
<li><p>I did some manual curation of about 300 authors in OpenRefine in preparation for telling Peter and Abenet that the migration is almost ready</p>
|
||
|
||
<ul>
|
||
<li>I would still like to perhaps (re)move institutional authors from <code>dc.contributor.author</code> to <code>cg.contributor.affiliation</code>, but I will have to run that by Francesca, Carol, and Abenet</li>
|
||
<li>I could use a custom text facet like this in OpenRefine to find authors that likely match the “Last, F.” pattern: <code>isNotNull(value.match(/^.*, \p{Lu}\.?.*$/))</code></li>
|
||
<li>The <code>\p{Lu}</code> is a cool <a href="https://www.regular-expressions.info/unicode.html">regex character class</a> to make sure this works for letters with accents</li>
|
||
<li>As cool as that is, it’s actually more effective to just search for authors that have “.” in them!</li>
|
||
<li>I’ve decided to add a <code>cg.contributor.affiliation</code> column to 1,025 items based on the logic above where the author name is not an actual person</li>
|
||
</ul></li>
|
||
</ul>
|
||
|
||
<h2 id="2019-10-13">2019-10-13</h2>
|
||
|
||
<ul>
|
||
<li>More cleanup work on the authors in the Bioversity migration
|
||
|
||
<ul>
|
||
<li>Now I sent the final feedback to Francesca, Carol, and Abenet</li>
|
||
</ul></li>
|
||
|
||
<li><p>Peter is still seeing some authors listed with “|” in the “Top Authors” statistics for some collections</p>
|
||
|
||
<ul>
|
||
<li>I looked in some of the items that are listed and the author field does not contain those invalid separators</li>
|
||
|
||
<li><p>I decided to try doing a full Discovery re-indexing on CGSpace (linode18):</p>
|
||
|
||
<pre><code>$ time schedtool -B -e ionice -c2 -n7 nice -n19 dspace index-discovery -b
|
||
|
||
real 82m35.993s
|
||
</code></pre></li>
|
||
</ul></li>
|
||
|
||
<li><p>After the re-indexing the top authors still list the following:</p>
|
||
|
||
<pre><code>Jagwe, J.|Ouma, E.A.|Brandes-van Dorresteijn, D.|Kawuma, Brian|Smith, J.
|
||
</code></pre></li>
|
||
|
||
<li><p>I looked in the database to find authors that had “|” in them:</p>
|
||
|
||
<pre><code>dspace=# SELECT text_value, resource_id FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=3 AND text_value LIKE '%|%';
|
||
text_value | resource_id
|
||
----------------------------------+-------------
|
||
Anandajayasekeram, P.|Puskur, R. | 157
|
||
Morales, J.|Renner, I. | 22779
|
||
Zahid, A.|Haque, M.A. | 25492
|
||
(3 rows)
|
||
</code></pre></li>
|
||
|
||
<li><p>Then I found their handles and corrected them, for example:</p>
|
||
|
||
<pre><code>dspacetest=# select handle from item, handle where handle.resource_id = item.item_id AND item.item_id = '157' and handle.resource_type_id=2;
|
||
handle
|
||
-----------
|
||
10568/129
|
||
(1 row)
|
||
</code></pre></li>
|
||
|
||
<li><p>So I’m still not sure where these weird authors in the “Top Author” stats are coming from</p></li>
|
||
</ul>
|
||
|
||
<h2 id="2019-10-14">2019-10-14</h2>
|
||
|
||
<ul>
|
||
<li>I talked to Peter about the Bioversity items and he said that we should add the institutional authors back to <code>dc.contributor.author</code>, because I had moved them to <code>cg.contributor.affiliation</code>
|
||
|
||
<ul>
|
||
<li>Otherwise he said the data looks good</li>
|
||
</ul></li>
|
||
</ul>
|
||
|
||
<h2 id="2019-10-15">2019-10-15</h2>
|
||
|
||
<ul>
|
||
<li><p>I did a test export / import of the Bioversity migration items on DSpace Test</p>
|
||
|
||
<ul>
|
||
<li><p>First export them:</p>
|
||
|
||
<pre><code>$ export JAVA_OPTS='-Dfile.encoding=UTF-8 -Xmx512m'
|
||
$ mkdir 2019-10-15-Bioversity
|
||
$ dspace export -i 10568/108684 -t COLLECTION -m -n 0 -d 2019-10-15-Bioversity
|
||
$ sed -i '/<dcvalue element="identifier" qualifier="uri">/d' 2019-10-15-Bioversity/*/dublin_core.xml
|
||
</code></pre></li>
|
||
</ul></li>
|
||
|
||
<li><p>It’s really stupid, but for some reason the handles are included even though I specified the <code>-m</code> option, so after the export I removed the <code>dc.identifier.uri</code> metadata values from the items</p></li>
|
||
|
||
<li><p>Then I imported a test subset of them in my local test environment:</p>
|
||
|
||
<pre><code>$ ~/dspace/bin/dspace import -a -c 10568/104049 -e fuu@cgiar.org -m 2019-10-15-Bioversity.map -s /tmp/2019-10-15-Bioversity
|
||
</code></pre></li>
|
||
|
||
<li><p>I had forgotten (again) that the <code>dspace export</code> command doesn’t preserve collection ownership or mappings, so I will have to create a temporary collection on CGSpace to import these to, then do the mappings again after import…</p></li>
|
||
|
||
<li><p>On CGSpace I will increase the RAM of the command line Java process for good luck before import…</p>
|
||
|
||
<pre><code>$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx1024m"
|
||
$ dspace import -a -c 10568/104057 -e fuu@cgiar.org -m 2019-10-15-Bioversity.map -s 2019-10-15-Bioversity
|
||
</code></pre></li>
|
||
|
||
<li><p>After importing them I re-exported the metadata, changed the owning collections to those based on their type, then re-imported them</p></li>
|
||
</ul>
|
||
|
||
<!-- vim: set sw=2 ts=2: -->
|
||
|
||
|
||
|
||
|
||
|
||
|
||
</article>
|
||
|
||
|
||
|
||
</div> <!-- /.blog-main -->
|
||
|
||
<aside class="col-sm-3 ml-auto blog-sidebar">
|
||
|
||
|
||
|
||
<section class="sidebar-module">
|
||
<h4>Recent Posts</h4>
|
||
<ol class="list-unstyled">
|
||
|
||
|
||
<li><a href="/cgspace-notes/2019-10/">October, 2019</a></li>
|
||
|
||
<li><a href="/cgspace-notes/2019-09/">September, 2019</a></li>
|
||
|
||
<li><a href="/cgspace-notes/2019-08/">August, 2019</a></li>
|
||
|
||
<li><a href="/cgspace-notes/2019-07/">July, 2019</a></li>
|
||
|
||
<li><a href="/cgspace-notes/2019-06/">June, 2019</a></li>
|
||
|
||
</ol>
|
||
</section>
|
||
|
||
|
||
|
||
|
||
<section class="sidebar-module">
|
||
<h4>Links</h4>
|
||
<ol class="list-unstyled">
|
||
|
||
<li><a href="https://cgspace.cgiar.org">CGSpace</a></li>
|
||
|
||
<li><a href="https://dspacetest.cgiar.org">DSpace Test</a></li>
|
||
|
||
<li><a href="https://github.com/ilri/DSpace">CGSpace @ GitHub</a></li>
|
||
|
||
</ol>
|
||
</section>
|
||
|
||
</aside>
|
||
|
||
|
||
</div> <!-- /.row -->
|
||
</div> <!-- /.container -->
|
||
|
||
|
||
|
||
<footer class="blog-footer">
|
||
<p dir="auto">
|
||
|
||
Blog template created by <a href="https://twitter.com/mdo">@mdo</a>, ported to Hugo by <a href='https://twitter.com/mralanorth'>@mralanorth</a>.
|
||
|
||
</p>
|
||
<p>
|
||
<a href="#">Back to top</a>
|
||
</p>
|
||
</footer>
|
||
|
||
|
||
</body>
|
||
|
||
</html>
|