mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2024-11-22 22:55:04 +01:00
431 lines
17 KiB
HTML
431 lines
17 KiB
HTML
<!DOCTYPE html>
|
|
<html lang="en">
|
|
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
|
|
|
|
<meta property="og:title" content="March, 2017" />
|
|
<meta property="og:description" content="2017-03-01
|
|
|
|
|
|
Run the 279 CIAT author corrections on CGSpace
|
|
|
|
|
|
2017-03-02
|
|
|
|
|
|
Skype with Michael and Peter, discussing moving the CGIAR Library to CGSpace
|
|
CGIAR people possibly open to moving content, redirecting library.cgiar.org to CGSpace and letting CGSpace resolve their handles
|
|
They might come in at the top level in one “CGIAR System” community, or with several communities
|
|
I need to spend a bit of time looking at the multiple handle support in DSpace and see if new content can be minted in both handles, or just one?
|
|
Need to send Peter and Michael some notes about this in a few days
|
|
Also, need to consider talking to Atmire about hiring them to bring ORCiD metadata to REST / OAI
|
|
Filed an issue on DSpace issue tracker for the filter-media bug that causes it to process JPGs even when limiting to the PDF thumbnail plugin: DS-3516
|
|
Discovered that the ImageMagic filter-media plugin creates JPG thumbnails with the CMYK colorspace when the source PDF is using CMYK
|
|
Interestingly, it seems DSpace 4.x’s thumbnails were sRGB, but forcing regeneration using DSpace 5.x’s ImageMagick plugin creates CMYK JPGs if the source PDF was CMYK (see 10568⁄51999):
|
|
|
|
|
|
$ identify ~/Desktop/alc_contrastes_desafios.jpg
|
|
/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000
|
|
|
|
|
|
" />
|
|
<meta property="og:type" content="article" />
|
|
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2017-03/" />
|
|
|
|
|
|
|
|
<meta property="article:published_time" content="2017-03-01T17:08:52+02:00"/>
|
|
<meta property="article:modified_time" content="2017-03-09T14:29:50+02:00"/>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<meta name="twitter:card" content="summary"/>
|
|
|
|
|
|
|
|
<meta name="twitter:text:title" content="March, 2017"/>
|
|
<meta name="twitter:title" content="March, 2017"/>
|
|
<meta name="twitter:description" content="2017-03-01
|
|
|
|
|
|
Run the 279 CIAT author corrections on CGSpace
|
|
|
|
|
|
2017-03-02
|
|
|
|
|
|
Skype with Michael and Peter, discussing moving the CGIAR Library to CGSpace
|
|
CGIAR people possibly open to moving content, redirecting library.cgiar.org to CGSpace and letting CGSpace resolve their handles
|
|
They might come in at the top level in one “CGIAR System” community, or with several communities
|
|
I need to spend a bit of time looking at the multiple handle support in DSpace and see if new content can be minted in both handles, or just one?
|
|
Need to send Peter and Michael some notes about this in a few days
|
|
Also, need to consider talking to Atmire about hiring them to bring ORCiD metadata to REST / OAI
|
|
Filed an issue on DSpace issue tracker for the filter-media bug that causes it to process JPGs even when limiting to the PDF thumbnail plugin: DS-3516
|
|
Discovered that the ImageMagic filter-media plugin creates JPG thumbnails with the CMYK colorspace when the source PDF is using CMYK
|
|
Interestingly, it seems DSpace 4.x’s thumbnails were sRGB, but forcing regeneration using DSpace 5.x’s ImageMagick plugin creates CMYK JPGs if the source PDF was CMYK (see 10568⁄51999):
|
|
|
|
|
|
$ identify ~/Desktop/alc_contrastes_desafios.jpg
|
|
/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000
|
|
|
|
|
|
"/>
|
|
|
|
<meta name="generator" content="Hugo 0.19" />
|
|
|
|
|
|
|
|
<script type="application/ld+json">
|
|
{
|
|
"@context": "http://schema.org",
|
|
"@type": "BlogPosting",
|
|
|
|
"headline": "March, 2017",
|
|
"url": "https://alanorth.github.io/cgspace-notes/2017-03/",
|
|
"wordCount": "1059",
|
|
|
|
|
|
"datePublished": "2017-03-01T17:08:52+02:00",
|
|
|
|
|
|
"dateModified": "2017-03-09T14:29:50+02:00",
|
|
|
|
|
|
"author": {
|
|
"@type": "Person",
|
|
"name": "Alan Orth"
|
|
}
|
|
|
|
,
|
|
"keywords": "Notes"
|
|
|
|
}
|
|
</script>
|
|
|
|
|
|
|
|
<link rel="canonical" href="https://alanorth.github.io/cgspace-notes/2017-03/">
|
|
|
|
<title>March, 2017 | CGSpace Notes</title>
|
|
|
|
<!-- combined, minified CSS -->
|
|
<link href="https://alanorth.github.io/cgspace-notes/css/style.css" rel="stylesheet" integrity="sha384-Cdt2yG10w21pA8DMpImJIvsLOME686p75OYD9jGCZVWvOol9zkEsaF3ctGEuBXK6" crossorigin="anonymous">
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</head>
|
|
|
|
<body>
|
|
|
|
<div class="blog-masthead">
|
|
<div class="container">
|
|
<nav class="nav blog-nav">
|
|
<a class="nav-link " href="https://alanorth.github.io/cgspace-notes/">Home</a>
|
|
|
|
|
|
|
|
</nav>
|
|
</div>
|
|
</div>
|
|
|
|
<header class="blog-header">
|
|
<div class="container">
|
|
<h1 class="blog-title"><a href="https://alanorth.github.io/cgspace-notes/" rel="home">CGSpace Notes</a></h1>
|
|
<p class="lead blog-description">Documenting day-to-day work on the <a href="https://cgspace.cgiar.org">CGSpace</a> repository.</p>
|
|
</div>
|
|
</header>
|
|
|
|
<div class="container">
|
|
<div class="row">
|
|
<div class="col-sm-8 blog-main">
|
|
|
|
|
|
|
|
|
|
<article class="blog-post">
|
|
<header>
|
|
<h2 class="blog-post-title"><a href="https://alanorth.github.io/cgspace-notes/2017-03/">March, 2017</a></h2>
|
|
<p class="blog-post-meta"><time datetime="2017-03-01T17:08:52+02:00">Wed Mar 01, 2017</time> by Alan Orth in
|
|
|
|
<i class="fa fa-tag" aria-hidden="true"></i> <a href="/cgspace-notes/tags/notes" rel="tag">Notes</a>
|
|
|
|
</p>
|
|
</header>
|
|
<h2 id="2017-03-01">2017-03-01</h2>
|
|
|
|
<ul>
|
|
<li>Run the 279 CIAT author corrections on CGSpace</li>
|
|
</ul>
|
|
|
|
<h2 id="2017-03-02">2017-03-02</h2>
|
|
|
|
<ul>
|
|
<li>Skype with Michael and Peter, discussing moving the CGIAR Library to CGSpace</li>
|
|
<li>CGIAR people possibly open to moving content, redirecting library.cgiar.org to CGSpace and letting CGSpace resolve their handles</li>
|
|
<li>They might come in at the top level in one “CGIAR System” community, or with several communities</li>
|
|
<li>I need to spend a bit of time looking at the multiple handle support in DSpace and see if new content can be minted in both handles, or just one?</li>
|
|
<li>Need to send Peter and Michael some notes about this in a few days</li>
|
|
<li>Also, need to consider talking to Atmire about hiring them to bring ORCiD metadata to REST / OAI</li>
|
|
<li>Filed an issue on DSpace issue tracker for the <code>filter-media</code> bug that causes it to process JPGs even when limiting to the PDF thumbnail plugin: <a href="https://jira.duraspace.org/browse/DS-3516">DS-3516</a></li>
|
|
<li>Discovered that the ImageMagic <code>filter-media</code> plugin creates JPG thumbnails with the CMYK colorspace when the source PDF is using CMYK</li>
|
|
<li>Interestingly, it seems DSpace 4.x’s thumbnails were sRGB, but forcing regeneration using DSpace 5.x’s ImageMagick plugin creates CMYK JPGs if the source PDF was CMYK (see <a href="https://cgspace.cgiar.org/handle/10568/51999"><sup>10568</sup>⁄<sub>51999</sub></a>):</li>
|
|
</ul>
|
|
|
|
<pre><code>$ identify ~/Desktop/alc_contrastes_desafios.jpg
|
|
/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000
|
|
</code></pre>
|
|
|
|
<p></p>
|
|
|
|
<ul>
|
|
<li>This results in discolored thumbnails when compared to the original PDF, for example sRGB and CMYK:</li>
|
|
</ul>
|
|
|
|
<p><img src="/cgspace-notes/2017/03/thumbnail-srgb.jpg" alt="Thumbnail in sRGB colorspace" /></p>
|
|
|
|
<p><img src="/cgspace-notes/2017/03/thumbnail-cmyk.jpg" alt="Thumbnial in CMYK colorspace" /></p>
|
|
|
|
<ul>
|
|
<li>I filed an issue for the color space thing: <a href="https://jira.duraspace.org/browse/DS-3517">DS-3517</a></li>
|
|
</ul>
|
|
|
|
<h2 id="2017-03-03">2017-03-03</h2>
|
|
|
|
<ul>
|
|
<li>I created a patch for DS-3517 and made a pull request against upstream <code>dspace-5_x</code>: <a href="https://github.com/DSpace/DSpace/pull/1669">https://github.com/DSpace/DSpace/pull/1669</a></li>
|
|
<li>Looks like <code>-colorspace sRGB</code> alone isn’t enough, we need to use profiles:</li>
|
|
</ul>
|
|
|
|
<pre><code>$ convert alc_contrastes_desafios.pdf\[0\] -profile /opt/brew/Cellar/ghostscript/9.20/share/ghostscript/9.20/iccprofiles/default_cmyk.icc -thumbnail 300x300 -flatten -profile /opt/brew/Cellar/ghostscript/9.20/share/ghostscript/9.20/iccprofiles/default_rgb.icc alc_contrastes_desafios.pdf.jpg
|
|
</code></pre>
|
|
|
|
<ul>
|
|
<li>This reads the input file, applies the CMYK profile, applies the RGB profile, then writes the file</li>
|
|
<li>Note that you should set the first profile immediately after the input file</li>
|
|
<li>Also, it is better to use profiles than setting <code>-colorspace</code></li>
|
|
<li>This is a great resource describing the color stuff: <a href="http://www.imagemagick.org/Usage/formats/#profiles">http://www.imagemagick.org/Usage/formats/#profiles</a></li>
|
|
<li>Somehow we need to detect the color system being used by the input file and handle each case differently (with profiles)</li>
|
|
<li>This is trivial with <code>identify</code> (even by the <a href="http://im4java.sourceforge.net/api/org/im4java/core/IMOps.html#identify">Java ImageMagick API</a>):</li>
|
|
</ul>
|
|
|
|
<pre><code>$ identify -format '%r\n' alc_contrastes_desafios.pdf\[0\]
|
|
DirectClass CMYK
|
|
$ identify -format '%r\n' Africa\ group\ of\ negotiators.pdf\[0\]
|
|
DirectClass sRGB Alpha
|
|
</code></pre>
|
|
|
|
<h2 id="2017-03-04">2017-03-04</h2>
|
|
|
|
<ul>
|
|
<li>Spent more time looking at the ImageMagick CMYK issue</li>
|
|
<li>The <code>default_cmyk.icc</code> and <code>default_rgb.icc</code> files are both part of the Ghostscript GPL distribution, but according to DSpace’s <code>LICENSES_THIRD_PARTY</code> file, DSpace doesn’t allow distribution of dependencies that are licensed solely under the GPL</li>
|
|
<li>So this issue is kinda pointless now, as the ICC profiles are absolutely necessary to make a meaningful CMYK→sRGB conversion</li>
|
|
</ul>
|
|
|
|
<h2 id="2017-03-05">2017-03-05</h2>
|
|
|
|
<ul>
|
|
<li>Look into helping developers from landportal.info with a query for items related to LAND on the REST API</li>
|
|
<li>They want something like the items that are returned by the general “LAND” query in the search interface, but we cannot do that</li>
|
|
<li>We can only return specific results for metadata fields, like:</li>
|
|
</ul>
|
|
|
|
<pre><code>$ curl -s -H "accept: application/json" -H "Content-Type: application/json" -X POST "https://dspacetest.cgiar.org/rest/items/find-by-metadata-field" -d '{"key": "cg.subject.ilri","value": "LAND REFORM", "language": null}' | json_pp
|
|
</code></pre>
|
|
|
|
<ul>
|
|
<li>But there are hundreds of combinations of fields and values (like <code>dc.subject</code> and all the center subjects), and we can’t use wildcards in REST!</li>
|
|
<li>Reading about enabling multiple handle prefixes in DSpace</li>
|
|
<li>There is a mailing list thread from 2011 about it: <a href="http://dspace.2283337.n4.nabble.com/Multiple-handle-prefixes-merged-DSpace-instances-td3427192.html">http://dspace.2283337.n4.nabble.com/Multiple-handle-prefixes-merged-DSpace-instances-td3427192.html</a></li>
|
|
<li>And a comment from Atmire’s Bram about it on the DSpace wiki: <a href="https://wiki.duraspace.org/display/DSDOC5x/Installing+DSpace?focusedCommentId=78163296#comment-78163296">https://wiki.duraspace.org/display/DSDOC5x/Installing+DSpace?focusedCommentId=78163296#comment-78163296</a></li>
|
|
<li>Bram mentions an undocumented configuration option <code>handle.plugin.checknameauthority</code>, but I noticed another one in <code>dspace.cfg</code>:</li>
|
|
</ul>
|
|
|
|
<pre><code># List any additional prefixes that need to be managed by this handle server
|
|
# (as for examle handle prefix coming from old dspace repository merged in
|
|
# that repository)
|
|
# handle.additional.prefixes = prefix1[, prefix2]
|
|
</code></pre>
|
|
|
|
<ul>
|
|
<li>Because of this I noticed that our Handle server’s <code>config.dct</code> was potentially misconfigured!</li>
|
|
<li>We had some default values still present:</li>
|
|
</ul>
|
|
|
|
<pre><code>"300:0.NA/YOUR_NAMING_AUTHORITY"
|
|
</code></pre>
|
|
|
|
<ul>
|
|
<li>I’ve changed them to the following and restarted the handle server:</li>
|
|
</ul>
|
|
|
|
<pre><code>"300:0.NA/10568"
|
|
</code></pre>
|
|
|
|
<ul>
|
|
<li>In looking at all the configs I just noticed that we are not providing a DOI in the Google-specific metadata crosswalk</li>
|
|
<li>From <code>dspace/config/crosswalks/google-metadata.properties</code>:</li>
|
|
</ul>
|
|
|
|
<pre><code>google.citation_doi = cg.identifier.doi
|
|
</code></pre>
|
|
|
|
<ul>
|
|
<li>This works, and makes DSpace output the following metadata on the item view page:</li>
|
|
</ul>
|
|
|
|
<pre><code><meta content="https://dx.doi.org/10.1186/s13059-017-1153-y" name="citation_doi">
|
|
</code></pre>
|
|
|
|
<ul>
|
|
<li>Submitted and merged pull request for this: <a href="https://github.com/ilri/DSpace/pull/305">https://github.com/ilri/DSpace/pull/305</a></li>
|
|
<li>Submit pull request to set the author separator for XMLUI item lists to a semicolon instead of “,”: <a href="https://github.com/ilri/DSpace/pull/306">https://github.com/ilri/DSpace/pull/306</a></li>
|
|
<li>I want to show it briefly to Abenet and Peter to get feedback</li>
|
|
</ul>
|
|
|
|
<h2 id="2017-03-06">2017-03-06</h2>
|
|
|
|
<ul>
|
|
<li>Someone on the mailing list said that <code>handle.plugin.checknameauthority</code> should be false if we’re using multiple handle prefixes</li>
|
|
</ul>
|
|
|
|
<h2 id="2017-03-07">2017-03-07</h2>
|
|
|
|
<ul>
|
|
<li>I set up a top-level community as a test for the CGIAR Library and imported one item with the the 10947 handle prefix</li>
|
|
<li>When testing the Handle resolver locally it shows the item to be on the local repository</li>
|
|
<li>So this seems to work, with the following caveats:
|
|
|
|
<ul>
|
|
<li>New items will have the default handle</li>
|
|
<li>Communities and collections will have the default handle</li>
|
|
<li>Only items imported manually can have the other handles</li>
|
|
</ul></li>
|
|
<li>I need to talk to Michael and Peter to share the news, and discuss the structure of their community(s) and try some actual test data</li>
|
|
<li>We’ll need to do some data cleaning to make sure they are using the same fields we are, like <code>dc.type</code> and <code>cg.identifier.status</code></li>
|
|
<li>Another thing is that the import process creates new <code>dc.date.accessioned</code> and <code>dc.date.available</code> fields, so we end up with duplicates (is it important to preserve the originals for these?)</li>
|
|
<li>Report DS-3520 issue to Atmire</li>
|
|
</ul>
|
|
|
|
<h2 id="2017-03-08">2017-03-08</h2>
|
|
|
|
<ul>
|
|
<li>Merge the author separator changes to <code>5_x-prod</code>, as everyone has responded positively about it, and it’s the default in Mirage2 afterall!</li>
|
|
<li>Cherry pick the <code>commons-collections</code> patch from DSpace’s <code>dspace-5_x</code> branch to address DS-3520: <a href="https://jira.duraspace.org/browse/DS-3520">https://jira.duraspace.org/browse/DS-3520</a></li>
|
|
</ul>
|
|
|
|
<h2 id="2017-03-09">2017-03-09</h2>
|
|
|
|
<ul>
|
|
<li>Export list of sponsors so Peter can clean it up:</li>
|
|
</ul>
|
|
|
|
<pre><code>dspace=# \copy (select text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'description' and qualifier = 'sponsorship') group by text_value order by count desc) to /tmp/sponsorship.csv with csv;
|
|
COPY 285
|
|
</code></pre>
|
|
|
|
<h2 id="2017-03-12">2017-03-12</h2>
|
|
|
|
<ul>
|
|
<li>Test the sponsorship fixes and deletes from Peter:</li>
|
|
</ul>
|
|
|
|
<pre><code>$ ./fix-metadata-values.py -i Investors-Fix-51.csv -f dc.description.sponsorship -t Action -m 29 -d dspace -u dspace -p fuuuu
|
|
$ ./delete-metadata-values.py -i Investors-Delete-121.csv -f dc.description.sponsorship -m 29 -d dspace -u dspace -p fuuu
|
|
</code></pre>
|
|
|
|
<ul>
|
|
<li>Generate a new list of unique sponsors so we can update the controlled vocabulary:</li>
|
|
</ul>
|
|
|
|
<pre><code>dspace=# \copy (select distinct text_value from metadatavalue where resource_type_id=2 and metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'description' and qualifier = 'sponsorship')) to /tmp/sponsorship.csv with csv;
|
|
</code></pre>
|
|
|
|
|
|
|
|
|
|
|
|
</article>
|
|
|
|
|
|
|
|
</div> <!-- /.blog-main -->
|
|
|
|
<aside class="col-sm-3 offset-sm-1 blog-sidebar">
|
|
|
|
|
|
|
|
|
|
|
|
<section class="sidebar-module">
|
|
<h4>Recent Posts</h4>
|
|
<ol class="list-unstyled">
|
|
|
|
<li><a href="/cgspace-notes/2017-03/">March, 2017</a></li>
|
|
|
|
<li><a href="/cgspace-notes/2017-02/">February, 2017</a></li>
|
|
|
|
<li><a href="/cgspace-notes/2017-01/">January, 2017</a></li>
|
|
|
|
<li><a href="/cgspace-notes/2016-12/">December, 2016</a></li>
|
|
|
|
<li><a href="/cgspace-notes/2016-11/">November, 2016</a></li>
|
|
|
|
</ol>
|
|
</section>
|
|
|
|
|
|
|
|
|
|
<section class="sidebar-module">
|
|
<h4>Links</h4>
|
|
<ol class="list-unstyled">
|
|
|
|
<li><a href="https://cgspace.cgiar.org">CGSpace</a></li>
|
|
|
|
<li><a href="https://dspacetest.cgiar.org">DSpace Test</a></li>
|
|
|
|
<li><a href="https://github.com/ilri/DSpace">CGSpace @ GitHub</a></li>
|
|
|
|
</ol>
|
|
</section>
|
|
|
|
</aside>
|
|
|
|
|
|
</div> <!-- /.row -->
|
|
</div> <!-- /.container -->
|
|
|
|
<footer class="blog-footer">
|
|
<p>
|
|
|
|
Blog template created by <a href="https://twitter.com/mdo">@mdo</a>, ported to Hugo by <a href='https://twitter.com/mralanorth'>@mralanorth</a>.
|
|
|
|
</p>
|
|
<p>
|
|
<a href="#">Back to top</a>
|
|
</p>
|
|
</footer>
|
|
|
|
</body>
|
|
|
|
</html>
|