mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2025-01-27 05:49:12 +01:00
Add notes for 2020-01-27
This commit is contained in:
@ -53,7 +53,7 @@ I tweeted the CGSpace repository link
|
||||
|
||||
|
||||
"/>
|
||||
<meta name="generator" content="Hugo 0.62.2" />
|
||||
<meta name="generator" content="Hugo 0.63.1" />
|
||||
|
||||
|
||||
|
||||
@ -63,7 +63,7 @@ I tweeted the CGSpace repository link
|
||||
"@type": "BlogPosting",
|
||||
"headline": "January, 2020",
|
||||
"url": "https:\/\/alanorth.github.io\/cgspace-notes\/2020-01\/",
|
||||
"wordCount": "2117",
|
||||
"wordCount": "2754",
|
||||
"datePublished": "2020-01-06T10:48:30+02:00",
|
||||
"dateModified": "2020-01-23T15:56:46+02:00",
|
||||
"author": {
|
||||
@ -83,7 +83,7 @@ I tweeted the CGSpace repository link
|
||||
|
||||
<!-- combined, minified CSS -->
|
||||
|
||||
<link href="https://alanorth.github.io/cgspace-notes/css/style.a20c1a4367639632cdb341d23c27ca44fedcc75b0f8b3cbea6203010da153d3c.css" rel="stylesheet" integrity="sha256-ogwaQ2djljLNs0HSPCfKRP7cx1sPizy+piAwENoVPTw=" crossorigin="anonymous">
|
||||
<link href="https://alanorth.github.io/cgspace-notes/css/style.23e2c3298bcc8c1136c19aba330c211ec94c36f7c4454ea15cf4d3548370042a.css" rel="stylesheet" integrity="sha256-I+LDKYvMjBE2wZq6MwwhHslMNvfERU6hXPTTVINwBCo=" crossorigin="anonymous">
|
||||
|
||||
|
||||
<!-- RSS 2.0 feed -->
|
||||
@ -130,7 +130,7 @@ I tweeted the CGSpace repository link
|
||||
<header>
|
||||
<h2 class="blog-post-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/2020-01/">January, 2020</a></h2>
|
||||
<p class="blog-post-meta"><time datetime="2020-01-06T10:48:30+02:00">Mon Jan 06, 2020</time> by Alan Orth in
|
||||
<i class="fa fa-folder" aria-hidden="true"></i> <a href="/cgspace-notes/categories/notes" rel="category tag">Notes</a>
|
||||
<span class="fas fa-folder" aria-hidden="true"></span> <a href="/cgspace-notes/categories/notes" rel="category tag">Notes</a>
|
||||
|
||||
|
||||
</p>
|
||||
@ -185,7 +185,7 @@ $ sed -n '5227p' /tmp/2020-01-08-authors.csv | xxd -c1
|
||||
</ul>
|
||||
<pre><code><e> 101, Hex 65, Octal 145 < ́> 769, Hex 0301, Octal 1401
|
||||
</code></pre><ul>
|
||||
<li>If I understand the situation correctly it sounds like this means that the character is not actually encoded as UTF-8, so it's stored incorrectly in the database…</li>
|
||||
<li>If I understand the situation correctly it sounds like this means that the character is not actually encoded as UTF-8, so it’s stored incorrectly in the database…</li>
|
||||
<li>Other encodings like <code>windows-1251</code> and <code>windows-1257</code> also fail on different characters like “ž” and “é” that <em>are</em> legitimate UTF-8 characters</li>
|
||||
<li>Then there is the issue of Russian, Chinese, etc characters, which are simply not representable in any of those encodings</li>
|
||||
<li>I think the solution is to upload it to Google Docs, or just send it to him and deal with each case manually in the corrections he sends me</li>
|
||||
@ -206,8 +206,8 @@ java.net.SocketTimeoutException: Read timed out
|
||||
<li>I am not sure how I will fix that shard…</li>
|
||||
<li>I discovered a very interesting tool called <a href="https://github.com/LuminosoInsight/python-ftfy">ftfy</a> that attempts to fix errors in UTF-8
|
||||
<ul>
|
||||
<li>I'm curious to start checking input files with this to see what it highlights</li>
|
||||
<li>I ran it on the authors file from last week and it converted characters like those with Spanish accents from multi-byte sequences (I don't know what it's called?) to digraphs (é→é), which vim identifies as:</li>
|
||||
<li>I’m curious to start checking input files with this to see what it highlights</li>
|
||||
<li>I ran it on the authors file from last week and it converted characters like those with Spanish accents from multi-byte sequences (I don’t know what it’s called?) to digraphs (é→é), which vim identifies as:</li>
|
||||
<li><code><e> 101, Hex 65, Octal 145 < ́> 769, Hex 0301, Octal 1401</code></li>
|
||||
<li><code><é> 233, Hex 00e9, Oct 351, Digr e'</code></li>
|
||||
</ul>
|
||||
@ -283,10 +283,10 @@ COPY 35
|
||||
<li>I opened a new <a href="https://github.com/AgriculturalSemantics/cg-core/pull/24">pull request</a> on the cg-core repository validate and fix the formatting of the HTML files</li>
|
||||
<li>Create more issues for OpenRXV:
|
||||
<ul>
|
||||
<li>Based on Peter's feedback on the <a href="https://github.com/ilri/OpenRXV/issues/33">text for labels and tooltips</a></li>
|
||||
<li>Based on Peter's feedback for the <a href="https://github.com/ilri/OpenRXV/issues/35">export icon</a></li>
|
||||
<li>Based on Peter's feedback for the <a href="https://github.com/ilri/OpenRXV/issues/31">sort options</a></li>
|
||||
<li>Based on Abenet's feedback that <a href="https://github.com/ilri/OpenRXV/issues/34">PDF and Word exports are not working</a></li>
|
||||
<li>Based on Peter’s feedback on the <a href="https://github.com/ilri/OpenRXV/issues/33">text for labels and tooltips</a></li>
|
||||
<li>Based on Peter’s feedback for the <a href="https://github.com/ilri/OpenRXV/issues/35">export icon</a></li>
|
||||
<li>Based on Peter’s feedback for the <a href="https://github.com/ilri/OpenRXV/issues/31">sort options</a></li>
|
||||
<li>Based on Abenet’s feedback that <a href="https://github.com/ilri/OpenRXV/issues/34">PDF and Word exports are not working</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
@ -352,7 +352,7 @@ $ wc -l hung-nguyen-a*handles.txt
|
||||
56 hung-nguyen-atmire-handles.txt
|
||||
102 total
|
||||
</code></pre><ul>
|
||||
<li>Comparing the lists of items, I see that nine of the ten missing items were added less than twenty-four hours ago, and the other was added last week, so they apparently just haven't been indexed yet
|
||||
<li>Comparing the lists of items, I see that nine of the ten missing items were added less than twenty-four hours ago, and the other was added last week, so they apparently just haven’t been indexed yet
|
||||
<ul>
|
||||
<li>I am curious to check tomorrow to see if they are there</li>
|
||||
</ul>
|
||||
@ -383,7 +383,7 @@ $ wc -l hung-nguyen-a*handles.txt
|
||||
</ul>
|
||||
<pre><code>$ convert -density 288 -filter lagrange -thumbnail 25% -background white -alpha remove -sampling-factor 1:1 -colorspace sRGB 10568-97925.pdf\[0\] 10568-97925.jpg
|
||||
</code></pre><ul>
|
||||
<li>Here I'm also explicitly setting the background to white and removing any alpha layers, but I could probably also just keep using <code>-flatten</code> like DSpace already does</li>
|
||||
<li>Here I’m also explicitly setting the background to white and removing any alpha layers, but I could probably also just keep using <code>-flatten</code> like DSpace already does</li>
|
||||
<li>I did some tests with a modified version of above that uses uses <code>-flatten</code> and drops the sampling-factor and colorspace, but bumps up the image size to 600px (default on CGSpace is currently 300):</li>
|
||||
</ul>
|
||||
<pre><code>$ convert -density 288 -filter lagrange -resize 25% -flatten 10568-97925.pdf\[0\] 10568-97925-d288-lagrange.pdf.jpg
|
||||
@ -391,16 +391,58 @@ $ convert -flatten 10568-97925.pdf\[0\] 10568-97925.pdf.jpg
|
||||
$ convert -thumbnail x600 10568-97925-d288-lagrange.pdf.jpg 10568-97925-d288-lagrange-thumbnail.pdf.jpg
|
||||
$ convert -thumbnail x600 10568-97925.pdf.jpg 10568-97925-thumbnail.pdf.jpg
|
||||
</code></pre><ul>
|
||||
<li>This emulate's DSpace's method of generating a high-quality image from the PDF and then creating a thumbnail</li>
|
||||
<li>I put together a proof of concept of this by adding the extra options to dspace-api's <code>ImageMagickThumbnailFilter.java</code> and it works</li>
|
||||
<li>This emulate’s DSpace’s method of generating a high-quality image from the PDF and then creating a thumbnail</li>
|
||||
<li>I put together a proof of concept of this by adding the extra options to dspace-api’s <code>ImageMagickThumbnailFilter.java</code> and it works</li>
|
||||
<li>I need to run tests on a handful of PDFs to see if there are any side effects</li>
|
||||
<li>The file size is about double the old ones, but the quality is very good and the file size is nowhere near ilri.org's 400KiB PNG!</li>
|
||||
<li>The file size is about double the old ones, but the quality is very good and the file size is nowhere near ilri.org’s 400KiB PNG!</li>
|
||||
<li>Peter sent me the corrections and deletions for affiliations last night so I imported them into OpenRefine to work around the normal UTF-8 issue, ran them through csv-metadata-quality to make sure all Unicode values were normalized (NFC), then applied them on DSpace Test and CGSpace:</li>
|
||||
</ul>
|
||||
<pre><code>$ csv-metadata-quality -i ~/Downloads/2020-01-22-fix-1113-affiliations.csv -o /tmp/2020-01-22-fix-1113-affiliations.csv -u --exclude-fields 'dc.date.issued,dc.date.issued[],cg.contributor.affiliation'
|
||||
$ ./fix-metadata-values.py -i /tmp/2020-01-22-fix-1113-affiliations.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.affiliation -m 211 -t correct
|
||||
$ ./delete-metadata-values.py -i /tmp/2020-01-22-delete-36-affiliations.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.affiliation -m 211
|
||||
</code></pre><!-- raw HTML omitted -->
|
||||
</code></pre><h2 id="2020-01-26">2020-01-26</h2>
|
||||
<ul>
|
||||
<li>Add “Gender” to controlled vocabulary for CRPs (<a href="https://github.com/ilri/DSpace/pull/442">#442</a>)</li>
|
||||
<li>Deploy the changes on CGSpace and run all updates on the server and reboot it
|
||||
<ul>
|
||||
<li>I had to restart the <code>tomcat7</code> service several times until all Solr statistics cores came up OK</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>I spent a few hours writing a script (<a href="https://gist.github.com/alanorth/1c7c8b2131a19559e273fbc1e58d6a71">create-thumbnails</a>) to compare the default DSpace thumbnails with the improved parameters above and actually when comparing them at size 600px I don’t really notice much difference, other than the new ones have slightly crisper text
|
||||
<ul>
|
||||
<li>So that was a waste of time, though I think our 300px thumbnails are a bit small now</li>
|
||||
<li><a href="https://www.imagemagick.org/discourse-server/viewtopic.php?t=14561">Another thread on the ImageMagick forum</a> mentions that you need to set the density, then read the image, then set the density again:</li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<pre><code>$ convert -density 288 10568-97925.pdf\[0\] -density 72 -filter lagrange -flatten 10568-97925-density.jpg
|
||||
</code></pre><ul>
|
||||
<li>One thing worth mentioning was this syntax for extracting bits from JSON in bash using <code>jq</code>:</li>
|
||||
</ul>
|
||||
<pre><code>$ RESPONSE=$(curl -s 'https://dspacetest.cgiar.org/rest/handle/10568/103447?expand=bitstreams')
|
||||
$ echo $RESPONSE | jq '.bitstreams[] | select(.bundleName=="ORIGINAL") | .retrieveLink'
|
||||
"/bitstreams/172559/retrieve"
|
||||
</code></pre><h2 id="2020-01-27">2020-01-27</h2>
|
||||
<ul>
|
||||
<li>Bizu has been having problems when she logs into CGSpace, she can’t see the community list on the front page
|
||||
<ul>
|
||||
<li>This last happened for another user in <a href="https://alanorth.github.io/cgspace-notes/2016-11/">2016-11</a>, and it was related to the Tomcat <code>maxHttpHeaderSize</code> being too small because the user was in too many groups</li>
|
||||
<li>I see that it is similar, with this message appearing in the DSpace log just after she logs in:</li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<pre><code>2020-01-27 06:02:23,681 ERROR org.dspace.app.xmlui.aspect.discovery.AbstractRecentSubmissionTransformer @ Caught SearchServiceException while retrieving recent submission for: home page
|
||||
org.dspace.discovery.SearchServiceException: org.apache.solr.search.SyntaxError: Cannot parse 'read:(g0 OR e610 OR g0 OR g3 OR g5 OR g4102 OR g9 OR g4105 OR g10 OR g4107 OR g4108 OR g13 OR g4109 OR g14 OR g15 OR g16 OR g18 OR g20 OR g23 OR g24 OR g2072 OR g2074 OR g28 OR g2076 OR g29 OR g2078 OR g2080 OR g34 OR g2082 OR g2084 OR g38 OR g2086 OR g2088 OR g43 OR g2093 OR g2095 OR g2097 OR g50 OR g51 OR g2101 OR g2103 OR g62 OR g65 OR g77 OR g78 OR g2127 OR g2142 OR g2151 OR g2152 OR g2153 OR g2154 OR g2156 OR g2165 OR g2171 OR g2174 OR g2175 OR g129 OR g2178 OR g2182 OR g2186 OR g153 OR g155 OR g158 OR g166 OR g167 OR g168 OR g169 OR g2225 OR g179 OR g2227 OR g2229 OR g183 OR g2231 OR g184 OR g2233 OR g186 OR g2235 OR g2237 OR g191 OR g192 OR g193 OR g2242 OR g2244 OR g2246 OR g2250 OR g204 OR g205 OR g207 OR g208 OR g2262 OR g2265 OR g218 OR g2268 OR g222 OR g223 OR g2271 OR g2274 OR g2277 OR g230 OR g231 OR g2280 OR g2283 OR g238 OR g2286 OR g241 OR g2289 OR g244 OR g2292 OR g2295 OR g2298 OR g2301 OR g254 OR g255 OR g2305 OR g2308 OR g262 OR g2311 OR g265 OR g268 OR g269 OR g273 OR g276 OR g277 OR g279 OR g282 OR g292 OR g293 OR g296 OR g297 OR g301 OR g303 OR g305 OR g2353 OR g310 OR g311 OR g313 OR g321 OR g325 OR g328 OR g333 OR g334 OR g342 OR g343 OR g345 OR g348 OR g2409 [...] ': too many boolean clauses
|
||||
</code></pre><ul>
|
||||
<li>Now this appears to be a Solr limit of some kind (“too many boolean clauses”)
|
||||
<ul>
|
||||
<li>I changed the <code>maxBooleanClauses</code> for all Solr cores on DSpace Test from 1024 to 2048 and then she was able to see her communities…</li>
|
||||
<li>I made a <a href="https://github.com/ilri/DSpace/pull/443">pull request</a> and merged it to the <code>5_x-prod</code> branch and will deploy on CGSpace later tonight</li>
|
||||
<li>I am curious if anyone on the dspace-tech mailing list has run into this, so I will try to send a message about this there when I get a chance</li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<!-- raw HTML omitted -->
|
||||
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user