mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2024-12-23 13:34:32 +01:00
349 lines
15 KiB
HTML
349 lines
15 KiB
HTML
<!DOCTYPE html>
|
|
<html lang="en-us">
|
|
<head prefix="og: http://ogp.me/ns#">
|
|
<meta charset="utf-8" />
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1" />
|
|
<meta property="og:title" content=" June, 2016 · CGSpace Notes" />
|
|
|
|
<meta property="og:site_name" content="CGSpace Notes" />
|
|
<meta property="og:url" content="/cgspace-notes/2016-06/" />
|
|
|
|
|
|
<meta property="og:type" content="article" />
|
|
|
|
<meta property="og:article:published_time" content="2016-06-01T10:53:00+03:00" />
|
|
|
|
<meta property="og:article:tag" content="notes" />
|
|
|
|
|
|
|
|
<title>
|
|
June, 2016 · CGSpace Notes
|
|
</title>
|
|
|
|
<link rel="stylesheet" href="/cgspace-notes/css/bootstrap.min.css" />
|
|
<link rel="stylesheet" href="/cgspace-notes/css/main.css" />
|
|
<link rel="stylesheet" href="/cgspace-notes/css/font-awesome.min.css" />
|
|
<link rel="stylesheet" href="/cgspace-notes/css/github.css" />
|
|
<link rel="stylesheet" href="//fonts.googleapis.com/css?family=Source+Sans+Pro:200,300,400" type="text/css">
|
|
<link rel="shortcut icon" href="/cgspace-notes/images/favicon.ico" />
|
|
<link rel="apple-touch-icon" href="/cgspace-notes/images/apple-touch-icon.png" />
|
|
|
|
</head>
|
|
<body>
|
|
<header class="global-header" style="background-image:url(../images/bg.jpg )">
|
|
<section class="header-text">
|
|
<h1><a href="/cgspace-notes/">CGSpace Notes</a></h1>
|
|
|
|
<div class="sns-links hidden-print">
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</div>
|
|
|
|
|
|
<a href="/cgspace-notes/" class="btn-header btn-back hidden-xs">
|
|
<i class="fa fa-angle-left" aria-hidden="true"></i>
|
|
Home
|
|
</a>
|
|
|
|
|
|
</section>
|
|
</header>
|
|
<main class="container">
|
|
|
|
|
|
<article>
|
|
<header>
|
|
<h1 class="text-primary">June, 2016</h1>
|
|
<div class="post-meta clearfix">
|
|
<div class="post-date pull-left">
|
|
Posted on
|
|
<time datetime="2016-06-01T10:53:00+03:00">
|
|
Jun 1, 2016
|
|
</time>
|
|
</div>
|
|
<div class="pull-right">
|
|
|
|
<span class="post-tag small"><a href="/cgspace-notes//tags/notes">#notes</a></span>
|
|
|
|
</div>
|
|
</div>
|
|
</header>
|
|
<section>
|
|
|
|
|
|
<h2 id="2016-06-01:6783872e82b68b1517e00f494e6b6504">2016-06-01</h2>
|
|
|
|
<ul>
|
|
<li>Experimenting with IFPRI OAI (we want to harvest their publications)</li>
|
|
<li>After reading the <a href="https://www.oclc.org/support/services/contentdm/help/server-admin-help/oai-support.en.html">ContentDM documentation</a> I found IFPRI’s OAI endpoint: <a href="http://ebrary.ifpri.org/oai/oai.php">http://ebrary.ifpri.org/oai/oai.php</a></li>
|
|
<li>After reading the <a href="https://www.openarchives.org/OAI/openarchivesprotocol.html">OAI documentation</a> and testing with an <a href="http://validator.oaipmh.com/">OAI validator</a> I found out how to get their publications</li>
|
|
<li>This is their publications set: <a href="http://ebrary.ifpri.org/oai/oai.php?verb=ListRecords&from=2016-01-01&set=p15738coll2&metadataPrefix=oai_dc">http://ebrary.ifpri.org/oai/oai.php?verb=ListRecords&from=2016-01-01&set=p15738coll2&metadataPrefix=oai_dc</a></li>
|
|
<li>You can see the others by using the OAI <code>ListSets</code> verb: <a href="http://ebrary.ifpri.org/oai/oai.php?verb=ListSets">http://ebrary.ifpri.org/oai/oai.php?verb=ListSets</a></li>
|
|
<li>Working on second phase of metadata migration, looks like this will work for moving CPWF-specific data in <code>dc.identifier.fund</code> to <code>cg.identifier.cpwfproject</code> and then the rest to <code>dc.description.sponsorship</code></li>
|
|
</ul>
|
|
|
|
<pre><code>dspacetest=# update metadatavalue set metadata_field_id=130 where metadata_field_id=75 and (text_value like 'PN%' or text_value like 'PHASE%' or text_value = 'CBA' or text_value = 'IA');
|
|
UPDATE 497
|
|
dspacetest=# update metadatavalue set metadata_field_id=29 where metadata_field_id=75;
|
|
UPDATE 14
|
|
</code></pre>
|
|
|
|
<ul>
|
|
<li>Fix a few minor miscellaneous issues in <code>dspace.cfg</code> (<a href="https://github.com/ilri/DSpace/pull/227">#227</a>)</li>
|
|
</ul>
|
|
|
|
<h2 id="2016-06-02:6783872e82b68b1517e00f494e6b6504">2016-06-02</h2>
|
|
|
|
<ul>
|
|
<li>Testing the configuration and theme changes for the upcoming metadata migration and I found some issues with <code>cg.coverage.admin-unit</code></li>
|
|
<li>Seems that the Browse configuration in <code>dspace.cfg</code> can’t handle the ‘-’ in the field name:</li>
|
|
</ul>
|
|
|
|
<pre><code>webui.browse.index.12 = subregion:metadata:cg.coverage.admin-unit:text
|
|
</code></pre>
|
|
|
|
<ul>
|
|
<li>But actually, I think since DSpace 4 or 5 (we are 5.1) the Browse indexes come from Discovery (defined in discovery.xml) so this is really just a parsing error</li>
|
|
<li>I’ve sent a message to the DSpace mailing list to ask about the Browse index definition</li>
|
|
<li>A user was having problems with submission and from the stacktrace it looks like a Sherpa/Romeo issue</li>
|
|
<li>I found a thread on the mailing list talking about it and there is bug report and a patch: <a href="https://jira.duraspace.org/browse/DS-2740">https://jira.duraspace.org/browse/DS-2740</a></li>
|
|
<li>The patch applies successfully on DSpace 5.1 so I will try it later</li>
|
|
</ul>
|
|
|
|
<h2 id="2016-06-03:6783872e82b68b1517e00f494e6b6504">2016-06-03</h2>
|
|
|
|
<ul>
|
|
<li>Investigating the CCAFS authority issue, I exported the metadata for the Videos collection</li>
|
|
<li>The top two authors are:</li>
|
|
</ul>
|
|
|
|
<pre><code>CGIAR Research Program on Climate Change, Agriculture and Food Security::acd00765-02f1-4b5b-92fa-bfa3877229ce::500
|
|
CGIAR Research Program on Climate Change, Agriculture and Food Security::acd00765-02f1-4b5b-92fa-bfa3877229ce::600
|
|
</code></pre>
|
|
|
|
<ul>
|
|
<li>So the only difference is the “confidence”</li>
|
|
<li>Ok, well THAT is interesting:</li>
|
|
</ul>
|
|
|
|
<pre><code>dspacetest=# select text_value, authority, confidence from metadatavalue where metadata_field_id=3 and text_value like '%Orth, %';
|
|
text_value | authority | confidence
|
|
------------+--------------------------------------+------------
|
|
Orth, A. | ab606e3a-2b04-4c7d-9423-14beccf54257 | -1
|
|
Orth, A. | ab606e3a-2b04-4c7d-9423-14beccf54257 | -1
|
|
Orth, A. | ab606e3a-2b04-4c7d-9423-14beccf54257 | -1
|
|
Orth, Alan | | -1
|
|
Orth, Alan | | -1
|
|
Orth, Alan | | -1
|
|
Orth, Alan | | -1
|
|
Orth, A. | 05c2c622-d252-4efb-b9ed-95a07d3adf11 | -1
|
|
Orth, A. | 05c2c622-d252-4efb-b9ed-95a07d3adf11 | -1
|
|
Orth, A. | ab606e3a-2b04-4c7d-9423-14beccf54257 | -1
|
|
Orth, A. | ab606e3a-2b04-4c7d-9423-14beccf54257 | -1
|
|
Orth, Alan | ad281dbf-ef81-4007-96c3-a7f5d2eaa6d9 | 600
|
|
Orth, Alan | ad281dbf-ef81-4007-96c3-a7f5d2eaa6d9 | 600
|
|
(13 rows)
|
|
</code></pre>
|
|
|
|
<ul>
|
|
<li>And now an actually relevent example:</li>
|
|
</ul>
|
|
|
|
<pre><code>dspacetest=# select count(*) from metadatavalue where metadata_field_id=3 and text_value like 'CGIAR Research Program on Climate Change, Agriculture and Food Security' and confidence = 500;
|
|
count
|
|
-------
|
|
707
|
|
(1 row)
|
|
|
|
dspacetest=# select count(*) from metadatavalue where metadata_field_id=3 and text_value like 'CGIAR Research Program on Climate Change, Agriculture and Food Security' and confidence != 500;
|
|
count
|
|
-------
|
|
253
|
|
(1 row)
|
|
</code></pre>
|
|
|
|
<ul>
|
|
<li>Trying something experimental:</li>
|
|
</ul>
|
|
|
|
<pre><code>dspacetest=# update metadatavalue set confidence=500 where metadata_field_id=3 and text_value like 'CGIAR Research Program on Climate Change, Agriculture and Food Security';
|
|
UPDATE 960
|
|
</code></pre>
|
|
|
|
<ul>
|
|
<li>And then re-indexing authority and Discovery…?</li>
|
|
<li>After Discovery reindex the CCAFS authors are all together in the Authors sidebar facet</li>
|
|
<li>The docs for the ORCiD and Authority stuff for DSpace 5 mention changing the browse indexes to use the Authority as well:</li>
|
|
</ul>
|
|
|
|
<pre><code>webui.browse.index.2 = author:metadataAuthority:dc.contributor.author:authority
|
|
</code></pre>
|
|
|
|
<ul>
|
|
<li>That would only be for the “Browse by” function… so we’ll have to see what effect that has later</li>
|
|
</ul>
|
|
|
|
<h2 id="2016-06-04:6783872e82b68b1517e00f494e6b6504">2016-06-04</h2>
|
|
|
|
<ul>
|
|
<li>Re-sync DSpace Test with CGSpace and perform test of metadata migration again</li>
|
|
<li>Run phase two of metadata migrations on CGSpace (see the <a href="https://gist.github.com/alanorth/1a730bec5ac9457a8fb0e3e72c98d09c">migration notes</a>)</li>
|
|
<li>Run all system updates and reboot CGSpace server</li>
|
|
</ul>
|
|
|
|
<h2 id="2016-06-07:6783872e82b68b1517e00f494e6b6504">2016-06-07</h2>
|
|
|
|
<ul>
|
|
<li>Figured out how to export a list of the unique values from a metadata field ordered by count:</li>
|
|
</ul>
|
|
|
|
<pre><code>dspacetest=# \copy (select text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=29 group by text_value order by count desc) to /tmp/sponsorship.csv with csv;
|
|
</code></pre>
|
|
|
|
<ul>
|
|
<li><p>Identified the next round of fields to migrate:</p>
|
|
|
|
<ul>
|
|
<li>dc.title.jtitle → dc.source</li>
|
|
<li>dc.crsubject.crpsubject → cg.contributor.crp</li>
|
|
<li>dc.contributor.affiliation → cg.contributor.affiliation</li>
|
|
<li>dc.Species → cg.species</li>
|
|
<li>dc.contributor.corporate → dc.contributor</li>
|
|
<li>dc.identifier.url → cg.identifier.url</li>
|
|
<li>dc.identifier.doi → cg.identifier.doi</li>
|
|
<li>dc.identifier.googleurl → cg.identifier.googleurl</li>
|
|
<li>dc.identifier.dataurl → cg.identifier.dataurl</li>
|
|
</ul></li>
|
|
|
|
<li><p>Discuss pulling data from IFPRI’s ContentDM with Ryan Miller</p></li>
|
|
|
|
<li><p>Looks like OAI is kinda obtuse for this, and if we use ContentDM’s API we’ll be able to access their internal field names (rather than trying to figure out how they stuffed them into various, repeated Dublin Core fields)</p></li>
|
|
</ul>
|
|
|
|
<h2 id="2016-06-08:6783872e82b68b1517e00f494e6b6504">2016-06-08</h2>
|
|
|
|
<ul>
|
|
<li>Discuss controlled vocabularies for ~28 fields</li>
|
|
<li>Looks like this is all we need: <a href="https://wiki.duraspace.org/display/DSDOC5x/Submission+User+Interface#SubmissionUserInterface-ConfiguringControlledVocabularies">https://wiki.duraspace.org/display/DSDOC5x/Submission+User+Interface#SubmissionUserInterface-ConfiguringControlledVocabularies</a></li>
|
|
<li>I wrote an XPath expression to extract the ILRI subjects from <code>input-forms.xml</code> (uses xmlstartlet):</li>
|
|
</ul>
|
|
|
|
<pre><code>$ xml sel -t -m '//value-pairs[@value-pairs-name="ilrisubject"]/pair/displayed-value/text()' -c '.' -n dspace/config/input-forms.xml
|
|
</code></pre>
|
|
|
|
<ul>
|
|
<li>Write to Atmire about the use of <code>atmire.orcid.id</code> to see if we can change it</li>
|
|
<li>Seems to be a virtual field that is queried from the authority cache… hmm</li>
|
|
<li>In other news, I found out that the About page that we haven’t been using lives in <code>dspace/config/about.xml</code>, so now we can update the text</li>
|
|
<li>File bug about <code>closed="true"</code> attribute of controlled vocabularies not working: <a href="https://jira.duraspace.org/browse/DS-3238">https://jira.duraspace.org/browse/DS-3238</a></li>
|
|
</ul>
|
|
|
|
<h2 id="2016-06-09:6783872e82b68b1517e00f494e6b6504">2016-06-09</h2>
|
|
|
|
<ul>
|
|
<li>Atmire explained that the <code>atmire.orcid.id</code> field doesn’t exist in the schema, as it actually comes from the authority cache during XMLUI run time</li>
|
|
<li>This means we don’t see it when harvesting via OAI or REST, for example</li>
|
|
<li>They opened a feature ticket on the DSpace tracker to ask for support of this: <a href="https://jira.duraspace.org/browse/DS-3239">https://jira.duraspace.org/browse/DS-3239</a></li>
|
|
</ul>
|
|
|
|
<h2 id="2016-06-10:6783872e82b68b1517e00f494e6b6504">2016-06-10</h2>
|
|
|
|
<ul>
|
|
<li>Investigating authority confidences</li>
|
|
<li>It looks like the values are documented in <code>Choices.java</code></li>
|
|
<li>Experiment with setting all 960 CCAFS author values to be 500:</li>
|
|
</ul>
|
|
|
|
<pre><code>dspacetest=# SELECT authority, confidence FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=3 AND text_value = 'CGIAR Research Program on Climate Change, Agriculture and Food Security';
|
|
|
|
dspacetest=# UPDATE metadatavalue set confidence = 500 where resource_type_id=2 AND metadata_field_id=3 AND text_value = 'CGIAR Research Program on Climate Change, Agriculture and Food Security';
|
|
UPDATE 960
|
|
</code></pre>
|
|
|
|
<ul>
|
|
<li>After the database edit, I did a full Discovery re-index</li>
|
|
<li>And now there are exactly 960 items in the authors facet for ‘CGIAR Research Program on Climate Change, Agriculture and Food Security’</li>
|
|
<li>Now I ran the same on CGSpace</li>
|
|
<li>Merge controlled vocabulary functionality for animal breeds to <code>5_x-prod</code> (<a href="https://github.com/ilri/DSpace/pull/236">#236</a>)</li>
|
|
<li>Write python script to update metadata values in batch via PostgreSQL: <a href="https://gist.github.com/alanorth/df92cbfb54d762ba21b28f7cd83b6897">fix-metadata-values.py</a></li>
|
|
<li>We need to use this to correct some pretty ugly values in fields like <code>dc.description.sponsorship</code></li>
|
|
<li>Merge item display tweaks from earlier this week (<a href="https://github.com/ilri/DSpace/pull/231">#231</a>)</li>
|
|
<li>Merge controlled vocabulary functionality for subregions (<a href="https://github.com/ilri/DSpace/pull/238">#238</a>)</li>
|
|
</ul>
|
|
|
|
<h2 id="2016-06-11:6783872e82b68b1517e00f494e6b6504">2016-06-11</h2>
|
|
|
|
<ul>
|
|
<li>Merge controlled vocabulary for sponsorship field (<a href="https://github.com/ilri/DSpace/pull/239">#239</a>)</li>
|
|
<li>Fix character encoding issues for animal breed lookup that I merged yesterday</li>
|
|
</ul>
|
|
|
|
<h2 id="2016-06-17:6783872e82b68b1517e00f494e6b6504">2016-06-17</h2>
|
|
|
|
<ul>
|
|
<li>Linode has free RAM upgrades for their 13th birthday so I migrated DSpace Test (4→8GB of RAM)</li>
|
|
</ul>
|
|
|
|
</section>
|
|
<footer>
|
|
|
|
<section class="author-info row">
|
|
<div class="author-avatar col-md-2">
|
|
|
|
</div>
|
|
<div class="author-meta col-md-6">
|
|
|
|
<h1 class="author-name text-primary">Alan Orth</h1>
|
|
|
|
|
|
</div>
|
|
|
|
</section>
|
|
<ul class="pager">
|
|
|
|
<li class="previous"><a href="/cgspace-notes/2016-05/"><span aria-hidden="true">←</span> Older</a></li>
|
|
|
|
|
|
<li class="next disabled"><a href="#">Newer <span aria-hidden="true">→</span></a></li>
|
|
|
|
</ul>
|
|
</footer>
|
|
</article>
|
|
|
|
</main>
|
|
<footer class="container global-footer">
|
|
<div class="copyright-note pull-left">
|
|
|
|
</div>
|
|
<div class="sns-links hidden-print">
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</div>
|
|
|
|
</footer>
|
|
|
|
<script src="/cgspace-notes/js/highlight.pack.js"></script>
|
|
<script>
|
|
hljs.initHighlightingOnLoad();
|
|
</script>
|
|
|
|
|
|
</body>
|
|
</html>
|
|
|