mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2024-11-05 14:53:02 +01:00
740 lines
50 KiB
HTML
740 lines
50 KiB
HTML
<!DOCTYPE html>
|
||
<html lang="en" >
|
||
|
||
<head>
|
||
<meta charset="utf-8">
|
||
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
|
||
|
||
|
||
<meta property="og:title" content="May, 2021" />
|
||
<meta property="og:description" content="2021-05-01
|
||
|
||
I looked at the top user agents and IPs in the Solr statistics for last month and I see these user agents:
|
||
|
||
“RI/1.0”, 1337
|
||
“Microsoft Office Word 2014”, 941
|
||
|
||
|
||
I will add the RI/1.0 pattern to our DSpace agents overload and purge them from Solr (we had previously seen this agent with 9,000 hits or so in 2020-09), but I think I will leave the Microsoft Word one… as that’s an actual user…
|
||
" />
|
||
<meta property="og:type" content="article" />
|
||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2021-05/" />
|
||
<meta property="article:published_time" content="2021-05-02T09:50:54+03:00" />
|
||
<meta property="article:modified_time" content="2021-07-06T17:03:55+03:00" />
|
||
|
||
|
||
|
||
<meta name="twitter:card" content="summary"/>
|
||
<meta name="twitter:title" content="May, 2021"/>
|
||
<meta name="twitter:description" content="2021-05-01
|
||
|
||
I looked at the top user agents and IPs in the Solr statistics for last month and I see these user agents:
|
||
|
||
“RI/1.0”, 1337
|
||
“Microsoft Office Word 2014”, 941
|
||
|
||
|
||
I will add the RI/1.0 pattern to our DSpace agents overload and purge them from Solr (we had previously seen this agent with 9,000 hits or so in 2020-09), but I think I will leave the Microsoft Word one… as that’s an actual user…
|
||
"/>
|
||
<meta name="generator" content="Hugo 0.92.0" />
|
||
|
||
|
||
|
||
<script type="application/ld+json">
|
||
{
|
||
"@context": "http://schema.org",
|
||
"@type": "BlogPosting",
|
||
"headline": "May, 2021",
|
||
"url": "https://alanorth.github.io/cgspace-notes/2021-05/",
|
||
"wordCount": "3605",
|
||
"datePublished": "2021-05-02T09:50:54+03:00",
|
||
"dateModified": "2021-07-06T17:03:55+03:00",
|
||
"author": {
|
||
"@type": "Person",
|
||
"name": "Alan Orth"
|
||
},
|
||
"keywords": "Notes"
|
||
}
|
||
</script>
|
||
|
||
|
||
|
||
<link rel="canonical" href="https://alanorth.github.io/cgspace-notes/2021-05/">
|
||
|
||
<title>May, 2021 | CGSpace Notes</title>
|
||
|
||
|
||
<!-- combined, minified CSS -->
|
||
|
||
<link href="https://alanorth.github.io/cgspace-notes/css/style.beb8012edc08ba10be012f079d618dc243812267efe62e11f22fe49618f976a4.css" rel="stylesheet" integrity="sha256-vrgBLtwIuhC+AS8HnWGNwkOBImfv5i4R8i/klhj5dqQ=" crossorigin="anonymous">
|
||
|
||
|
||
<!-- minified Font Awesome for SVG icons -->
|
||
|
||
<script defer src="https://alanorth.github.io/cgspace-notes/js/fontawesome.min.f5072c55a0721857184db93a50561d7dc13975b4de2e19db7f81eb5f3fa57270.js" integrity="sha256-9QcsVaByGFcYTbk6UFYdfcE5dbTeLhnbf4HrXz+lcnA=" crossorigin="anonymous"></script>
|
||
|
||
<!-- RSS 2.0 feed -->
|
||
|
||
|
||
|
||
|
||
</head>
|
||
|
||
<body>
|
||
|
||
|
||
<div class="blog-masthead">
|
||
<div class="container">
|
||
<nav class="nav blog-nav">
|
||
<a class="nav-link " href="https://alanorth.github.io/cgspace-notes/">Home</a>
|
||
</nav>
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
<header class="blog-header">
|
||
<div class="container">
|
||
<h1 class="blog-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/" rel="home">CGSpace Notes</a></h1>
|
||
<p class="lead blog-description" dir="auto">Documenting day-to-day work on the <a href="https://cgspace.cgiar.org">CGSpace</a> repository.</p>
|
||
</div>
|
||
</header>
|
||
|
||
|
||
|
||
|
||
<div class="container">
|
||
<div class="row">
|
||
<div class="col-sm-8 blog-main">
|
||
|
||
|
||
|
||
|
||
<article class="blog-post">
|
||
<header>
|
||
<h2 class="blog-post-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/2021-05/">May, 2021</a></h2>
|
||
<p class="blog-post-meta">
|
||
<time datetime="2021-05-02T09:50:54+03:00">Sun May 02, 2021</time>
|
||
in
|
||
<span class="fas fa-folder" aria-hidden="true"></span> <a href="/cgspace-notes/categories/notes/" rel="category tag">Notes</a>
|
||
|
||
|
||
</p>
|
||
</header>
|
||
<h2 id="2021-05-01">2021-05-01</h2>
|
||
<ul>
|
||
<li>I looked at the top user agents and IPs in the Solr statistics for last month and I see these user agents:
|
||
<ul>
|
||
<li>“RI/1.0”, 1337</li>
|
||
<li>“Microsoft Office Word 2014”, 941</li>
|
||
</ul>
|
||
</li>
|
||
<li>I will add the RI/1.0 pattern to our DSpace agents overload and purge them from Solr (we had previously seen this agent with 9,000 hits or so in 2020-09), but I think I will leave the Microsoft Word one… as that’s an actual user…</li>
|
||
</ul>
|
||
<ul>
|
||
<li>I should probably add the <code>RI/1.0</code> pattern to COUNTER-Robots project</li>
|
||
<li>As well as these IPs:
|
||
<ul>
|
||
<li>193.169.254.178, 21648</li>
|
||
<li>181.62.166.177, 20323</li>
|
||
<li>45.146.166.180, 19376</li>
|
||
</ul>
|
||
</li>
|
||
<li>The first IP seems to be in Estonia and their requests to the REST API change user agents from curl to Mac OS X to Windows and more
|
||
<ul>
|
||
<li>Also, they seem to be trying to exploit something:</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">193.169.254.178 - - [21/Apr/2021:01:59:01 +0200] "GET /rest/collections/1179/items?limit=812&expand=metadata\x22%20and%20\x2221\x22=\x2221 HTTP/1.1" 400 5 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
|
||
193.169.254.178 - - [21/Apr/2021:02:00:36 +0200] "GET /rest/collections/1179/items?limit=812&expand=metadata-21%2B21*01 HTTP/1.1" 200 458201 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
|
||
193.169.254.178 - - [21/Apr/2021:02:00:36 +0200] "GET /rest/collections/1179/items?limit=812&expand=metadata'||lower('')||' HTTP/1.1" 400 5 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
|
||
193.169.254.178 - - [21/Apr/2021:02:02:10 +0200] "GET /rest/collections/1179/items?limit=812&expand=metadata'%2Brtrim('')%2B' HTTP/1.1" 200 458209 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
|
||
</code></pre></div><ul>
|
||
<li>I will report the IP on abuseipdb.com and purge their hits from Solr</li>
|
||
<li>The second IP is in Colombia and is making thousands of requests for what looks like some test site:</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">181.62.166.177 - - [20/Apr/2021:22:48:42 +0200] "GET /rest/collections/d1e11546-c62a-4aee-af91-fd482b3e7653/items?expand=metadata HTTP/2.0" 200 123613 "http://cassavalighthousetest.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36"
|
||
181.62.166.177 - - [20/Apr/2021:22:55:39 +0200] "GET /rest/collections/d1e11546-c62a-4aee-af91-fd482b3e7653/items?expand=metadata HTTP/2.0" 200 123613 "http://cassavalighthousetest.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36"
|
||
</code></pre></div><ul>
|
||
<li>But this site does not exist (yet?)
|
||
<ul>
|
||
<li>I will purge them from Solr</li>
|
||
</ul>
|
||
</li>
|
||
<li>The third IP is in Russia apparently, and the user agent has the <code>pl-PL</code> locale with thousands of requests like this:</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">45.146.166.180 - - [18/Apr/2021:16:28:44 +0200] "GET /bitstream/handle/10947/4153/.AAS%202014%20Annual%20Report.pdf?sequence=1%22%29%29%20AND%201691%3DUTL_INADDR.GET_HOST_ADDRESS%28CHR%28113%29%7C%7CCHR%28118%29%7C%7CCHR%28113%29%7C%7CCHR%28106%29%7C%7CCHR%28113%29%7C%7C%28SELECT%20%28CASE%20WHEN%20%281691%3D1691%29%20THEN%201%20ELSE%200%20END%29%20FROM%20DUAL%29%7C%7CCHR%28113%29%7C%7CCHR%2898%29%7C%7CCHR%28122%29%7C%7CCHR%28120%29%7C%7CCHR%28113%29%29%20AND%20%28%28%22RKbp%22%3D%22RKbp&isAllowed=y HTTP/1.1" 200 918998 "http://cgspace.cgiar.org:80/bitstream/handle/10947/4153/.AAS 2014 Annual Report.pdf" "Mozilla/5.0 (Windows; U; Windows NT 5.1; pl-PL) AppleWebKit/523.15 (KHTML, like Gecko) Version/3.0 Safari/523.15"
|
||
</code></pre></div><ul>
|
||
<li>I will purge these all with my <code>check-spider-ip-hits.sh</code> script:</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ ./ilri/check-spider-ip-hits.sh -f /tmp/ips.txt -p
|
||
Purging 21648 hits from 193.169.254.178 in statistics
|
||
Purging 20323 hits from 181.62.166.177 in statistics
|
||
Purging 19376 hits from 45.146.166.180 in statistics
|
||
<span style="color:#960050;background-color:#1e0010">
|
||
</span><span style="color:#960050;background-color:#1e0010"></span>Total number of bot hits purged: 61347
|
||
</code></pre></div><h2 id="2021-05-02">2021-05-02</h2>
|
||
<ul>
|
||
<li>Check the AReS Harvester indexes:</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ curl -s http://localhost:9200/_cat/indices | grep openrxv-items
|
||
yellow open openrxv-items-temp H-CGsyyLTaqAj6-nKXZ-7w 1 1 0 0 283b 283b
|
||
yellow open openrxv-items-final ul3SKsa7Q9Cd_K7qokBY_w 1 1 103951 0 254mb 254mb
|
||
$ curl -s <span style="color:#e6db74">'http://localhost:9200/_alias/'</span> | python -m json.tool
|
||
...
|
||
"openrxv-items-temp": {
|
||
"aliases": {}
|
||
},
|
||
"openrxv-items-final": {
|
||
"aliases": {
|
||
"openrxv-items": {}
|
||
}
|
||
},
|
||
</code></pre></div><ul>
|
||
<li>I think they look OK (<code>openrxv-items</code> is an alias of <code>openrxv-items-final</code>), but I took a backup just in case:</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ elasticdump --input<span style="color:#f92672">=</span>http://localhost:9200/openrxv-items --output<span style="color:#f92672">=</span>/home/aorth/openrxv-items_mapping.json --type<span style="color:#f92672">=</span>mapping
|
||
$ elasticdump --input<span style="color:#f92672">=</span>http://localhost:9200/openrxv-items --output<span style="color:#f92672">=</span>/home/aorth/openrxv-items_data.json --type<span style="color:#f92672">=</span>data --limit<span style="color:#f92672">=</span><span style="color:#ae81ff">1000</span>
|
||
</code></pre></div><ul>
|
||
<li>Then I started an indexing in the AReS Explorer admin dashboard</li>
|
||
<li>The indexing finished, but it looks like the aliases are messed up again:</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ curl -s http://localhost:9200/_cat/indices | grep openrxv-items
|
||
yellow open openrxv-items-temp H-CGsyyLTaqAj6-nKXZ-7w 1 1 104165 105024 487.7mb 487.7mb
|
||
yellow open openrxv-items-final d0tbMM_SRWimirxr_gm9YA 1 1 937 0 2.2mb 2.2mb
|
||
</code></pre></div><h2 id="2021-05-05">2021-05-05</h2>
|
||
<ul>
|
||
<li>Peter noticed that we no longer display <code>cg.link.reference</code> on the item view
|
||
<ul>
|
||
<li>It seems that this got dropped accidentally when we migrated to <code>dcterms.relation</code> in CG Core v2</li>
|
||
<li>I fixed it in the <code>6_x-prod</code> branch and told him it will be live soon</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<h2 id="2021-05-09">2021-05-09</h2>
|
||
<ul>
|
||
<li>I set up a clean DSpace 6.4 instance locally to test some things against, for example to be able to rule out whether some issues are due to Atmire modules or are fixed in the as-of-yet-unreleased DSpace 6.4
|
||
<ul>
|
||
<li>I had to delete all the Atmire schemas, then it worked fine on Tomcat 8.5 with Mirage (I didn’t want to bother with npm and ruby for Mirage 2)</li>
|
||
<li>Then I tried to see if I could reproduce the mapping issue that Marianne raised last month
|
||
<ul>
|
||
<li>I tried unmapping and remapping to the CGIAR Gender grants collection and the collection appears in the item view’s list of mapped collections, but not on the collection browse itself</li>
|
||
<li>Then I tried mapping to a new collection and it was the same as above</li>
|
||
<li>So this issue is really just a DSpace bug, and nothing to do with Atmire and not fixed in the unreleased DSpace 6.4</li>
|
||
<li>I will try one more time after updating the Discovery index (I’m also curious how fast it is on vanilla DSpace 6.4, though I think I tried that when I did the flame graphs in 2019 and it was miserable)</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ time ~/dspace64/bin/dspace index-discovery -b
|
||
~/dspace64/bin/dspace index-discovery -b 4053.24s user 53.17s system 38% cpu 2:58:53.83 total
|
||
</code></pre></div><ul>
|
||
<li>Nope! Still slow, and still no mapped item…
|
||
<ul>
|
||
<li>I even tried unmapping it from all collections, and adding it to a single new owning collection…</li>
|
||
</ul>
|
||
</li>
|
||
<li>Ah hah! Actually, I was inspecting the item’s authorization policies when I noticed that someone had made the item private!
|
||
<ul>
|
||
<li>After making it public again I was able to see it in the target collection</li>
|
||
</ul>
|
||
</li>
|
||
<li>The indexes on AReS Explorer are messed up after last week’s harvesting:</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ curl -s http://localhost:9200/_cat/indices | grep openrxv-items
|
||
yellow open openrxv-items-temp H-CGsyyLTaqAj6-nKXZ-7w 1 1 104165 105024 487.7mb 487.7mb
|
||
yellow open openrxv-items-final d0tbMM_SRWimirxr_gm9YA 1 1 937 0 2.2mb 2.2mb
|
||
<span style="color:#960050;background-color:#1e0010">
|
||
</span><span style="color:#960050;background-color:#1e0010"></span>$ curl -s <span style="color:#e6db74">'http://localhost:9200/_alias/'</span> | python -m json.tool
|
||
...
|
||
"openrxv-items-final": {
|
||
"aliases": {}
|
||
},
|
||
"openrxv-items-temp": {
|
||
"aliases": {
|
||
"openrxv-items": {}
|
||
}
|
||
}
|
||
</code></pre></div><ul>
|
||
<li><code>openrxv-items</code> should be an alias of <code>openrxv-items-final</code>…</li>
|
||
<li>I made a backup of the temp index and then started indexing on the AReS Explorer admin dashboard:</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ curl -X PUT <span style="color:#e6db74">"localhost:9200/openrxv-items-temp/_settings"</span> -H <span style="color:#e6db74">'Content-Type: application/json'</span> -d<span style="color:#e6db74">'{"settings": {"index.blocks.write": true}}'</span>
|
||
$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items-temp-backup
|
||
$ curl -X PUT <span style="color:#e6db74">"localhost:9200/openrxv-items-temp/_settings"</span> -H <span style="color:#e6db74">'Content-Type: application/json'</span> -d<span style="color:#e6db74">'{"settings": {"index.blocks.write": false}}'</span>
|
||
</code></pre></div><h2 id="2021-05-10">2021-05-10</h2>
|
||
<ul>
|
||
<li>Amazing, the harvesting on AReS finished but it messed up all the indexes and now there are no items in any index!</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ curl -s http://localhost:9200/_cat/indices | grep openrxv-items
|
||
yellow open openrxv-items-temp 8thRX0WVRUeAzmd2hkG6TA 1 1 0 0 283b 283b
|
||
yellow open openrxv-items-temp-backup _0tyvctBTg2pjOlcoVP1LA 1 1 104165 20134 305.5mb 305.5mb
|
||
yellow open openrxv-items-final BtvV9kwVQ3yBYCZvJS1QyQ 1 1 0 0 283b 283b
|
||
</code></pre></div><ul>
|
||
<li>I fixed the indexes manually by re-creating them and cloning from the backup:</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-final'</span>
|
||
$ curl -X PUT <span style="color:#e6db74">"localhost:9200/openrxv-items-temp-backup/_settings"</span> -H <span style="color:#e6db74">'Content-Type: application/json'</span> -d<span style="color:#e6db74">'{"settings": {"index.blocks.write": true}}'</span>
|
||
$ curl -s -X POST http://localhost:9200/openrxv-items-temp-backup/_clone/openrxv-items-final
|
||
$ curl -s -X POST <span style="color:#e6db74">'http://localhost:9200/_aliases'</span> -H <span style="color:#e6db74">'Content-Type: application/json'</span> -d<span style="color:#e6db74">'{"actions" : [{"add" : { "index" : "openrxv-items-final", "alias" : "openrxv-items"}}]}'</span>
|
||
$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp-backup'</span>
|
||
</code></pre></div><ul>
|
||
<li>Also I ran all updated on the server and updated all Docker images, then rebooted the server (linode20):</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ docker images | grep -v ^REPO | sed <span style="color:#e6db74">'s/ \+/:/g'</span> | cut -d: -f1,2 | xargs -L1 docker pull
|
||
</code></pre></div><ul>
|
||
<li>I backed up the AReS Elasticsearch data using elasticdump, then started a new harvest:</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ elasticdump --input<span style="color:#f92672">=</span>http://localhost:9200/openrxv-items --output<span style="color:#f92672">=</span>/home/aorth/openrxv-items_mapping.json --type<span style="color:#f92672">=</span>mapping
|
||
$ elasticdump --input<span style="color:#f92672">=</span>http://localhost:9200/openrxv-items --output<span style="color:#f92672">=</span>/home/aorth/openrxv-items_data.json --type<span style="color:#f92672">=</span>data --limit<span style="color:#f92672">=</span><span style="color:#ae81ff">1000</span>
|
||
</code></pre></div><ul>
|
||
<li>Discuss CGSpace statistics with the CIP team
|
||
<ul>
|
||
<li>They were wondering why their numbers for 2020 were so low</li>
|
||
<li>I checked their community using the DSpace Statistics API and found very accurate numbers for 2020 and 2019 for them</li>
|
||
<li>I think they had been using AReS, which actually doesn’t even give stats for a time period…</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<h2 id="2021-05-11">2021-05-11</h2>
|
||
<ul>
|
||
<li>The AReS harvesting from yesterday finished, but the indexes are messed up again so I will have to fix them again before I harvest next time</li>
|
||
<li>I also spent some time looking at IWMI’s reports again
|
||
<ul>
|
||
<li>On AReS we don’t have a way to group by peer reviewed or item type other than doing “if type is Journal Article”</li>
|
||
<li>Also, we don’t have a way to check the IWMI Strategic Priorities because those are communities, not metadata…</li>
|
||
<li>We can get the collections an item is in from the <code>parentCollectionList</code> metadata, but it is saved in Elasticsearch as a string instead of a list…</li>
|
||
<li>I told them it won’t be possible to replicate their reports exactly</li>
|
||
</ul>
|
||
</li>
|
||
<li>I decided to look at the CLARISA controlled vocabularies again
|
||
<ul>
|
||
<li>They now have 6,200 institutions (was around 3,400 when I last looked in 2020-07)</li>
|
||
<li>They have updated their Swagger interface but it still requires an API key if you want to use it from curl</li>
|
||
<li>They have ISO 3166 countries and UN M.49 regions, but I notice they have some weird names like “Russian Federation (the)”, which is not in ISO 3166 as far as I can see</li>
|
||
<li>I exported a list of the institutions to look closer
|
||
<ul>
|
||
<li>I found twelve items with whitespace issues</li>
|
||
<li>There are some weird entries like <code>Research Institute for Aquaculture No1</code> and <code>Research Institute for Aquaculture No2</code></li>
|
||
<li>A few items have weird Unicode characters like U+00AD, U+200B, and U+00A0</li>
|
||
<li>I found 100+ items with multiple languages in there name like <code>Ministère de l’Agriculture, de la pêche et des ressources hydrauliques / Ministry of Agriculture, Hydraulic Resources and Fisheries</code></li>
|
||
<li>Over 600 institutions have the country in their name like <code>Ministry of Coordination of Environmental Affairs (Mozambique)</code></li>
|
||
<li>For URLs they have <code>null</code> in some places… which is weird… why not just leave it blank?</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
</li>
|
||
<li>I checked the CLARISA list against ROR’s April, 2020 release (“Version 9”, on figshare, though it is version 8 in the dump):</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ ./ilri/ror-lookup.py -i /tmp/clarisa-institutions.txt -r ror-data-2021-04-06.json -o /tmp/clarisa-ror-matches.csv
|
||
$ csvgrep -c matched -m <span style="color:#e6db74">'true'</span> /tmp/clarisa-ror-matches.csv | sed <span style="color:#e6db74">'1d'</span> | wc -l
|
||
1770
|
||
</code></pre></div><ul>
|
||
<li>With 1770 out of 6230 matched, that’s 28.5%…</li>
|
||
<li>I sent an email to Hector Tobon to point out the issues in CLARISA again and ask him to chat</li>
|
||
<li>Meeting with GARDIAN developers about CG Core and how GARDIAN works</li>
|
||
</ul>
|
||
<h2 id="2021-05-13">2021-05-13</h2>
|
||
<ul>
|
||
<li>Fix a few thousand IWMI URLs that are using HTTP instead of HTTPS on CGSpace:</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">localhost/dspace63= > UPDATE metadatavalue SET text_value = REGEXP_REPLACE(text_value, 'http://www.iwmi.cgiar.org','https://www.iwmi.cgiar.org', 'g') WHERE text_value LIKE 'http://www.iwmi.cgiar.org%' AND metadata_field_id=219;
|
||
UPDATE 1132
|
||
localhost/dspace63= > UPDATE metadatavalue SET text_value = REGEXP_REPLACE(text_value, 'http://publications.iwmi.org','https://publications.iwmi.org', 'g') WHERE text_value LIKE 'http://publications.iwmi.org%' AND metadata_field_id=219;
|
||
UPDATE 1803
|
||
</code></pre></div><ul>
|
||
<li>In the case of the latter, the HTTP links don’t even work! The web server returns HTTP 404 unless the request is HTTPS</li>
|
||
<li>IWMI also says that their subjects are a subset of AGROVOC so they no longer want to use <code>cg.subject.iwmi</code> for their subjects
|
||
<ul>
|
||
<li>They asked if I can move them to <code>dcterms.subject</code></li>
|
||
</ul>
|
||
</li>
|
||
<li>Delete two items for Udana because he was getting the “Authorization denied for action OBSOLETE (DELETE) …” error when trying to delete them (DSpace 6 bug I found a few months ago)
|
||
<ul>
|
||
<li><a href="https://cgspace.cgiar.org/handle/10568/34536">https://cgspace.cgiar.org/handle/10568/34536</a></li>
|
||
<li><a href="https://cgspace.cgiar.org/handle/10568/34570">https://cgspace.cgiar.org/handle/10568/34570</a></li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<h2 id="2021-05-14">2021-05-14</h2>
|
||
<ul>
|
||
<li>I updated the PostgreSQL JDBC driver in the Ansible playbooks to version 42.2.20 and deployed it on DSpace Test (linode26)</li>
|
||
</ul>
|
||
<h2 id="2021-05-15">2021-05-15</h2>
|
||
<ul>
|
||
<li>I have to fix the Elasticsearch indexes on AReS after last week’s harvesting because, as always, the <code>openrxv-items</code> index should be an alias of <code>openrxv-items-final</code> instead of <code>openrxv-items-temp</code>:</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ curl -s <span style="color:#e6db74">'http://localhost:9200/_alias/'</span> | python -m json.tool
|
||
"openrxv-items-final": {
|
||
"aliases": {}
|
||
},
|
||
"openrxv-items-temp": {
|
||
"aliases": {
|
||
"openrxv-items": {}
|
||
}
|
||
},
|
||
...
|
||
</code></pre></div><ul>
|
||
<li>I took a backup of the <code>openrxv-items</code> index with elasticdump so I can re-create them manually before starting a new harvest tomorrow:</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ elasticdump --input<span style="color:#f92672">=</span>http://localhost:9200/openrxv-items --output<span style="color:#f92672">=</span>/home/aorth/openrxv-items_mapping.json --type<span style="color:#f92672">=</span>mapping
|
||
$ elasticdump --input<span style="color:#f92672">=</span>http://localhost:9200/openrxv-items --output<span style="color:#f92672">=</span>/home/aorth/openrxv-items_data.json --type<span style="color:#f92672">=</span>data --limit<span style="color:#f92672">=</span><span style="color:#ae81ff">1000</span>
|
||
</code></pre></div><h2 id="2021-05-16">2021-05-16</h2>
|
||
<ul>
|
||
<li>I deleted and re-created the Elasticsearch indexes on AReS:</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-final'</span>
|
||
$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp'</span>
|
||
$ curl -XPUT <span style="color:#e6db74">'http://localhost:9200/openrxv-items-final'</span>
|
||
$ curl -XPUT <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp'</span>
|
||
$ curl -s -X POST <span style="color:#e6db74">'http://localhost:9200/_aliases'</span> -H <span style="color:#e6db74">'Content-Type: application/json'</span> -d<span style="color:#e6db74">'{"actions" : [{"add" : { "index" : "openrxv-items-final", "alias" : "openrxv-items"}}]}'</span>
|
||
</code></pre></div><ul>
|
||
<li>Then I re-imported the backup that I created with elasticdump yesterday:</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ elasticdump --input<span style="color:#f92672">=</span>/home/aorth/openrxv-items_mapping.json --output<span style="color:#f92672">=</span>http://localhost:9200/openrxv-items-final --type<span style="color:#f92672">=</span>mapping
|
||
$ elasticdump --input<span style="color:#f92672">=</span>/home/aorth/openrxv-items_data.json --output<span style="color:#f92672">=</span>http://localhost:9200/openrxv-items-final --type<span style="color:#f92672">=</span>data --limit<span style="color:#f92672">=</span><span style="color:#ae81ff">1000</span>
|
||
</code></pre></div><ul>
|
||
<li>Then I started a new harvest on AReS</li>
|
||
</ul>
|
||
<h2 id="2021-05-17">2021-05-17</h2>
|
||
<ul>
|
||
<li>The AReS harvest finished and the Elasticsearch indexes seem OK so I shouldn’t have to fix them next time…</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ curl -s http://localhost:9200/_cat/indices | grep openrxv-items
|
||
yellow open openrxv-items-temp o3ijJLcyTtGMOPeWpAJiVA 1 1 0 0 283b 283b
|
||
yellow open openrxv-items-final TrJ1Ict3QZ-vFkj-4VcAzw 1 1 104317 0 259.4mb 259.4mb
|
||
$ curl -s <span style="color:#e6db74">'http://localhost:9200/_alias/'</span> | python -m json.tool
|
||
"openrxv-items-temp": {
|
||
"aliases": {}
|
||
},
|
||
"openrxv-items-final": {
|
||
"aliases": {
|
||
"openrxv-items": {}
|
||
}
|
||
},
|
||
...
|
||
</code></pre></div><ul>
|
||
<li>Abenet said she and some others can’t log into CGSpace
|
||
<ul>
|
||
<li>I tried to check the CGSpace LDAP account and it does seem to be not working:</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ ldapsearch -x -H ldaps://AZCGNEROOT2.CGIARAD.ORG:636/ -b <span style="color:#e6db74">"dc=cgiarad,dc=org"</span> -D <span style="color:#e6db74">"cgspace-ldap@cgiarad.org"</span> -W <span style="color:#e6db74">"(sAMAccountName=aorth)"</span>
|
||
Enter LDAP Password:
|
||
ldap_bind: Invalid credentials (49)
|
||
additional info: 80090308: LdapErr: DSID-0C090453, comment: AcceptSecurityContext error, data 532, v3839
|
||
</code></pre></div><ul>
|
||
<li>I sent a message to Biruk so he can check the LDAP account</li>
|
||
<li>IWMI confirmed that they do indeed want to move all their subjects to AGROVOC, so I made the changes in the XMLUI and config (<a href="https://github.com/ilri/DSpace/pull/467">#467</a>)
|
||
<ul>
|
||
<li>Then I used the <code>migrate-fields.sh</code> script to move them (46,000 metadata entries!)</li>
|
||
</ul>
|
||
</li>
|
||
<li>I tested Abdullah’s latest pull request to add clickable filters to AReS and I think it works
|
||
<ul>
|
||
<li>I had some issues with my local AReS installation so I was only able to do limited testing…</li>
|
||
<li>I will have to wait until I have more time to test before I can merge that</li>
|
||
</ul>
|
||
</li>
|
||
<li>CCAFS asked me to add eight more phase II project tags to the CGSpace input form
|
||
<ul>
|
||
<li>I extracted the existing ones using xmllint, added the new ones, sorted them, and then replaced them in <code>input-forms.xml</code>:</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ xmllint --xpath <span style="color:#e6db74">'//value-pairs[@value-pairs-name="ccafsprojectpii"]/pair/stored-value/node()'</span> dspace/config/input-forms.xml
|
||
</code></pre></div><ul>
|
||
<li>I formatted the input file with tidy, especially because one of the new project tags has an ampersand character… grrr:</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ tidy -xml -utf8 -m -iq -w <span style="color:#ae81ff">0</span> dspace/config/input-forms.xml
|
||
line 3658 column 26 - Warning: unescaped & or unknown entity "&WA_EU-IFAD"
|
||
line 3659 column 23 - Warning: unescaped & or unknown entity "&WA_EU-IFAD"
|
||
</code></pre></div><ul>
|
||
<li>After testing whether this escaped value worked during submission, I created and merged a pull request to <code>6_x-prod</code> (<a href="https://github.com/ilri/DSpace/pull/468">#468</a>)</li>
|
||
</ul>
|
||
<h2 id="2021-05-18">2021-05-18</h2>
|
||
<ul>
|
||
<li>Paola from the Alliance emailed me some new ORCID identifiers to add to CGSpace</li>
|
||
<li>I saved the new ones to a text file, combined them with the others, extracted the ORCID iDs themselves, and updated the names using <code>resolve-orcids.py</code>:</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ cat ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-identifier.xml /tmp/new | grep -oE <span style="color:#e6db74">'[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}'</span> | sort | uniq > /tmp/2021-05-18-combined.txt
|
||
$ ./ilri/resolve-orcids.py -i /tmp/2021-05-18-combined.txt -o /tmp/2021-05-18-combined-names.txt
|
||
</code></pre></div><ul>
|
||
<li>I sorted the names and added the XML formatting in vim, then ran it through tidy:</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ tidy -xml -utf8 -m -iq -w <span style="color:#ae81ff">0</span> dspace/config/controlled-vocabularies/cg-creator-identifier.xml
|
||
</code></pre></div><ul>
|
||
<li>Tag fifty-five items from the Alliance’s new authors with ORCID iDs using <code>add-orcid-identifiers-csv.py</code>:</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ cat 2021-05-18-add-orcids.csv
|
||
dc.contributor.author,cg.creator.identifier
|
||
"Urioste Daza, Sergio",Sergio Alejandro Urioste Daza: 0000-0002-3208-032X
|
||
"Urioste, Sergio",Sergio Alejandro Urioste Daza: 0000-0002-3208-032X
|
||
"Villegas, Daniel",Daniel M. Villegas: 0000-0001-6801-3332
|
||
"Villegas, Daniel M.",Daniel M. Villegas: 0000-0001-6801-3332
|
||
"Giles, James",James Giles: 0000-0003-1899-9206
|
||
"Simbare, Alice",Alice Simbare: 0000-0003-2389-0969
|
||
"Simbare, Alice",Alice Simbare: 0000-0003-2389-0969
|
||
"Simbare, A.",Alice Simbare: 0000-0003-2389-0969
|
||
"Dita Rodriguez, Miguel",Miguel Angel Dita Rodriguez: 0000-0002-0496-4267
|
||
"Templer, Noel",Noel Templer: 0000-0002-3201-9043
|
||
"Jalonen, R.",Riina Jalonen: 0000-0003-1669-9138
|
||
"Jalonen, Riina",Riina Jalonen: 0000-0003-1669-9138
|
||
"Izquierdo, Paulo",Paulo Izquierdo: 0000-0002-2153-0655
|
||
"Reyes, Byron",Byron Reyes: 0000-0003-2672-9636
|
||
"Reyes, Byron A.",Byron Reyes: 0000-0003-2672-9636
|
||
$ ./ilri/add-orcid-identifiers-csv.py -i /tmp/2021-05-18-add-orcids.csv -db dspace -u dspace -p <span style="color:#e6db74">'fuuu'</span> -d
|
||
</code></pre></div><ul>
|
||
<li>I deployed the latest <code>6_x-prod</code> branch on CGSpace, ran all system updates, and rebooted the server
|
||
<ul>
|
||
<li>This included the IWMI changes, so I also migrated the <code>cg.subject.iwmi</code> metadata to <code>dcterms.subject</code> and deleted the subject term</li>
|
||
<li>Then I started a full Discovery reindex</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<h2 id="2021-05-19">2021-05-19</h2>
|
||
<ul>
|
||
<li>I realized that I need to lower case the IWMI subjects that I just moved to AGROVOC because they were probably mostly uppercase
|
||
<ul>
|
||
<li>To my surprise I checked <code>dcterms.subject</code> has 47,000 metadata fields that are upper or mixed case!</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">dspace=# UPDATE metadatavalue SET text_value=LOWER(text_value) WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=187 AND text_value ~ '[[:upper:]]';
|
||
UPDATE 47405
|
||
</code></pre></div><ul>
|
||
<li>That’s interesting because we lowercased them all a few months ago, so these must all be new… wow
|
||
<ul>
|
||
<li>We have 405,000 total AGROVOC terms, with 20,600 of them being unique</li>
|
||
<li>I will have to start another Discovery re-indexing to pick up these new changes</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<h2 id="2021-05-20">2021-05-20</h2>
|
||
<ul>
|
||
<li>Export the top 5,000 AGROVOC terms to validate them:</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">localhost/dspace63= > \COPY (SELECT DISTINCT text_value, count(*) FROM metadatavalue WHERE dspace_object_id in (SELECT dspace_object_id FROM item) AND metadata_field_id = 187 GROUP BY text_value ORDER BY count DESC LIMIT 5000) to /tmp/2021-05-20-agrovoc.csv WITH CSV HEADER;
|
||
COPY 5000
|
||
$ csvcut -c <span style="color:#ae81ff">1</span> /tmp/2021-05-20-agrovoc.csv| sed 1d > /tmp/2021-05-20-agrovoc.txt
|
||
$ ./ilri/agrovoc-lookup.py -i /tmp/2021-05-20-agrovoc.txt -o /tmp/2021-05-20-agrovoc-results.csv
|
||
$ csvgrep -c <span style="color:#e6db74">"number of matches"</span> -r <span style="color:#e6db74">'^0$'</span> /tmp/2021-05-20-agrovoc-results.csv > /tmp/2021-05-20-agrovoc-rejected.csv
|
||
</code></pre></div><ul>
|
||
<li>Meeting with Medha and Pythagoras about the FAIR Workflow tool
|
||
<ul>
|
||
<li>Discussed the need for such a tool, other tools being developed, etc</li>
|
||
<li>I stressed the important of controlled vocabularies</li>
|
||
<li>No real outcome, except to keep us posted and let us know if they need help testing on DSpace</li>
|
||
</ul>
|
||
</li>
|
||
<li>Meeting with Hector Tobon to discuss issues with CLARISA
|
||
<ul>
|
||
<li>They pushed back a bit, saying they were more focused on the needs of the CG</li>
|
||
<li>They are not against the idea of aligning closer to ROR, but lack the man power</li>
|
||
<li>They pointed out that their countries come directly from the <a href="https://www.iso.org/iso-3166-country-codes.html">ISO 3166 online browsing platform on the ISO website</a></li>
|
||
<li>Indeed the text value for Russia is “Russian Federation (the)” there… I find that strange</li>
|
||
<li>I filed <a href="https://salsa.debian.org/iso-codes-team/iso-codes/-/issues/33">an issue</a> on the iso-codes GitLab repository</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<h2 id="2021-05-24">2021-05-24</h2>
|
||
<ul>
|
||
<li>Add ORCID identifiers for missing ILRI authors and tag 550 others based on a few authors I noticed that were missing them:</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ cat 2021-05-24-add-orcids.csv
|
||
dc.contributor.author,cg.creator.identifier
|
||
"Patel, Ekta","Ekta Patel: 0000-0001-9400-6988"
|
||
"Dessie, Tadelle","Tadelle Dessie: 0000-0002-1630-0417"
|
||
"Tadelle, D.","Tadelle Dessie: 0000-0002-1630-0417"
|
||
"Dione, Michel M.","Michel Dione: 0000-0001-7812-5776"
|
||
"Kiara, Henry K.","Henry Kiara: 0000-0001-9578-1636"
|
||
"Naessens, Jan","Jan Naessens: 0000-0002-7075-9915"
|
||
"Steinaa, Lucilla","Lucilla Steinaa: 0000-0003-3691-3971"
|
||
"Wieland, Barbara","Barbara Wieland: 0000-0003-4020-9186"
|
||
"Grace, Delia","Delia Grace: 0000-0002-0195-9489"
|
||
"Rao, Idupulapati M.","Idupulapati M. Rao: 0000-0002-8381-9358"
|
||
"Cardoso Arango, Juan Andrés","Juan Andrés Cardoso Arango: 0000-0002-0252-4655"
|
||
$ ./ilri/add-orcid-identifiers-csv.py -i 2021-05-24-add-orcids.csv -db dspace -u dspace -p <span style="color:#e6db74">'fuuu'</span>
|
||
</code></pre></div><ul>
|
||
<li>A few days ago I took a backup of the Elasticsearch indexes on AReS using elasticdump:</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ elasticdump --input<span style="color:#f92672">=</span>http://localhost:9200/openrxv-items --output<span style="color:#f92672">=</span>/home/aorth/openrxv-items_data.json --type<span style="color:#f92672">=</span>data --limit<span style="color:#f92672">=</span><span style="color:#ae81ff">1000</span>
|
||
$ elasticdump --input<span style="color:#f92672">=</span>http://localhost:9200/openrxv-items --output<span style="color:#f92672">=</span>/home/aorth/openrxv-items_mapping.json --type<span style="color:#f92672">=</span>mapping
|
||
</code></pre></div><ul>
|
||
<li>The indexes look OK so I started a harvesting on AReS</li>
|
||
</ul>
|
||
<h2 id="2021-05-25">2021-05-25</h2>
|
||
<ul>
|
||
<li>The AReS harvest got messed up somehow, as I see the number of items in the indexes are the same as before the harvesting:</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ curl -s http://localhost:9200/_cat/indices | grep openrxv-items
|
||
yellow open openrxv-items-temp o3ijJLcyTtGMOPeWpAJiVA 1 1 104373 106455 491.5mb 491.5mb
|
||
yellow open openrxv-items-final soEzAnp3TDClIGZbmVyEIw 1 1 953 0 2.3mb 2.3mb
|
||
</code></pre></div><ul>
|
||
<li>Update all docker images on the AReS server (linode20):</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ docker images | grep -v ^REPO | sed <span style="color:#e6db74">'s/ \+/:/g'</span> | cut -d: -f1,2 | xargs -L1 docker pull
|
||
$ docker-compose -f docker/docker-compose.yml down
|
||
$ docker-compose -f docker/docker-compose.yml build
|
||
</code></pre></div><ul>
|
||
<li>Then run all system updates on the server and reboot it</li>
|
||
<li>Oh crap, I deleted everything on AReS and restored the backup and the total items are now 104317… so it was actually correct before!</li>
|
||
<li>For reference, this is how I re-created everything:</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">curl -XDELETE 'http://localhost:9200/openrxv-items-final'
|
||
curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
|
||
curl -XPUT 'http://localhost:9200/openrxv-items-final'
|
||
curl -XPUT 'http://localhost:9200/openrxv-items-temp'
|
||
curl -s -X POST 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -d'{"actions" : [{"add" : { "index" : "openrxv-items-final", "alias" : "openrxv-items"}}]}'
|
||
elasticdump --input=/home/aorth/openrxv-items_mapping.json --output=http://localhost:9200/openrxv-items-final --type=mapping
|
||
elasticdump --input=/home/aorth/openrxv-items_data.json --output=http://localhost:9200/openrxv-items-final --type=data --limit=1000
|
||
</code></pre></div><ul>
|
||
<li>I will just start a new harvest… sigh</li>
|
||
</ul>
|
||
<h2 id="2021-05-26">2021-05-26</h2>
|
||
<ul>
|
||
<li>The AReS harvest last night got stuck at 3:20AM (UTC+3) at 752 pages for some reason…
|
||
<ul>
|
||
<li>Something seems to have happened on CGSpace this morning around then as I have an alert from UptimeRobot as well</li>
|
||
<li>I re-created everything as above (again) and restarted the harvest</li>
|
||
</ul>
|
||
</li>
|
||
<li>Looking in the DSpace log for this morning I see a big hole in the logs at that time (UTC+2 server time):</li>
|
||
</ul>
|
||
<pre tabindex="0"><code>2021-05-26 02:17:52,808 INFO org.dspace.curate.Curator @ Curation task: countrycodetagger performed on: 10568/70659 with status: 2. Result: '10568/70659: item has country codes, skipping'
|
||
2021-05-26 02:17:52,853 INFO org.dspace.curate.Curator @ Curation task: countrycodetagger performed on: 10568/66761 with status: 2. Result: '10568/66761: item has country codes, skipping'
|
||
2021-05-26 03:00:05,772 INFO org.dspace.statistics.SolrLoggerServiceImpl @ solr-statistics.spidersfile:null
|
||
2021-05-26 03:00:05,773 INFO org.dspace.statistics.SolrLoggerServiceImpl @ solr-statistics.server:http://localhost:8081/solr/statistics
|
||
</code></pre><ul>
|
||
<li>There are no logs between 02:17 and 03:00… hmmm.</li>
|
||
<li>I see a similar gap in the Solr log, though it starts at 02:15:</li>
|
||
</ul>
|
||
<pre tabindex="0"><code>2021-05-26 02:15:07,968 INFO org.apache.solr.core.SolrCore @ [search] webapp=/solr path=/select params={f.location.coll.facet.sort=count&facet.field=location.comm&facet.field=location.coll&fl=handle,search.resourcetype,search.resourceid,search.uniqueid&start=0&fq=NOT(withdrawn:true)&fq=NOT(discoverable:false)&fq=search.resourcetype:2&fq=NOT(discoverable:false)&rows=0&version=2&q=*:*&f.location.coll.facet.limit=-1&facet.mincount=1&facet=true&f.location.comm.facet.sort=count&wt=javabin&facet.offset=0&f.location.comm.facet.limit=-1} hits=90792 status=0 QTime=6
|
||
2021-05-26 02:15:09,446 INFO org.apache.solr.core.SolrCore @ [statistics] webapp=/solr path=/update params={wt=javabin&version=2} status=0 QTime=1
|
||
2021-05-26 02:28:03,602 INFO org.apache.solr.update.UpdateHandler @ start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
|
||
2021-05-26 02:28:03,630 INFO org.apache.solr.core.SolrCore @ SolrDeletionPolicy.onCommit: commits: num=2
|
||
commit{dir=NRTCachingDirectory(MMapDirectory@/home/cgspace.cgiar.org/solr/statistics/data/index lockFactory=NativeFSLockFactory@/home/cgspace.cgiar.org/solr/statistics/data/index; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_n6ns,generation=1081720}
|
||
commit{dir=NRTCachingDirectory(MMapDirectory@/home/cgspace.cgiar.org/solr/statistics/data/index lockFactory=NativeFSLockFactory@/home/cgspace.cgiar.org/solr/statistics/data/index; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_n6nt,generation=1081721}
|
||
2021-05-26 02:28:03,630 INFO org.apache.solr.core.SolrCore @ newest commit generation = 1081721
|
||
2021-05-26 02:28:03,632 INFO org.apache.solr.search.SolrIndexSearcher @ Opening Searcher@34f2c871[statistics] main
|
||
2021-05-26 02:28:03,633 INFO org.apache.solr.core.SolrCore @ QuerySenderListener sending requests to Searcher@34f2c871[statistics] main{StandardDirectoryReader(segments_n5xy:4540675:nrt _1befl(4.10.4):C42054400/925069:delGen=8891 _1bksq(4.10.4):C685090/92211:delGen=1227 _1bx0c(4.10.4):C1069897/49988:delGen=966 _1bxr2(4.10.4):C197387/5860:delGen=485 _1cc1x(4.10.4):C353338/40887:delGen=626 _1ck6k(4.10.4):C1009357/39041:delGen=166 _1celj(4.10.4):C268907/18097:delGen=340 _1clq9(4.10.4):C147453/25003:delGen=68 _1cn3t(4.10.4):C260311/1802:delGen=82 _1cl3c(4.10.4):C47408/2610:delGen=39 _1cnmh(4.10.4):C32851/237:delGen=41 _1cod4(4.10.4):C85915/281:delGen=35 _1coy4(4.10.4):C178367/483:delGen=27 _1cpgs(4.10.4):C25465/81:delGen=13 _1cppf(4.10.4):C101411/154:delGen=15 _1cqc4(4.10.4):C26003/39:delGen=8 _1cpvl(4.10.4):C24160/91:delGen=8 _1cq3n(4.10.4):C18167/39:delGen=4 _1cq15(4.10.4):C9983/13:delGen=2 _1cq79(4.10.4):C13077/19:delGen=4 _1cqhz(4.10.4):C21251/2:delGen=1 _1cqka(4.10.4):C3531 _1cqku(4.10.4):C2597 _1cqkk(4.10.4):C2951 _1cqjq(4.10.4):C2675 _1cql5(4.10.4):C993 _1cql6(4.10.4):C161 _1cql7(4.10.4):C106 _1cql8(4.10.4):C19 _1cql9(4.10.4):C147 _1cqla(4.10.4):C2 _1cqlb(4.10.4):C15)}
|
||
2021-05-26 02:28:03,633 INFO org.apache.solr.core.SolrCore @ QuerySenderListener done.
|
||
</code></pre><ul>
|
||
<li>Ah, it seems to have been a <a href="https://status.linode.com/incidents/byqmt6nss9l0">Linode network issue in the Frankfurt region</a>:</li>
|
||
</ul>
|
||
<pre tabindex="0"><code>May 26, 2021
|
||
Connectivity Issue - Frankfurt
|
||
Resolved - We haven’t observed any additional connectivity issues in our Frankfurt data center, and will now consider this incident resolved. If you continue to experience problems, please open a Support ticket for assistance.
|
||
May 26, 02:57 UTC
|
||
</code></pre><ul>
|
||
<li>While looking in the logs I noticed an error about SMTP:</li>
|
||
</ul>
|
||
<pre tabindex="0"><code>2021-05-26 02:00:18,015 ERROR org.dspace.eperson.SubscribeCLITool @ Failed to send subscription to eperson_id=934cb92f-2e77-4881-89e2-6f13ad4b1378
|
||
2021-05-26 02:00:18,015 ERROR org.dspace.eperson.SubscribeCLITool @ javax.mail.SendFailedException: Send failure (javax.mail.MessagingException: Could not convert socket to TLS (javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)))
|
||
</code></pre><ul>
|
||
<li>And indeed the email seems to be broken:</li>
|
||
</ul>
|
||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ dspace test-email
|
||
<span style="color:#960050;background-color:#1e0010">
|
||
</span><span style="color:#960050;background-color:#1e0010"></span>About to send test email:
|
||
- To: fuuuuuu
|
||
- Subject: DSpace test email
|
||
- Server: smtp.office365.com
|
||
<span style="color:#960050;background-color:#1e0010">
|
||
</span><span style="color:#960050;background-color:#1e0010"></span>Error sending email:
|
||
- Error: javax.mail.SendFailedException: Send failure (javax.mail.MessagingException: Could not convert socket to TLS (javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)))
|
||
<span style="color:#960050;background-color:#1e0010">
|
||
</span><span style="color:#960050;background-color:#1e0010"></span>Please see the DSpace documentation for assistance.
|
||
</code></pre></div><ul>
|
||
<li>I saw a recent thread on the dspace-tech mailing list about this that makes me wonder if Microsoft changed something on Office 365
|
||
<ul>
|
||
<li>I added <code>mail.smtp.ssl.protocols=TLSv1.2</code> to the <code>mail.extraproperties</code> in dspace.cfg and the test email sent successfully</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<h2 id="2021-05-30">2021-05-30</h2>
|
||
<ul>
|
||
<li>Reset the Elasticsearch indexes on AReS as above and start a fresh harvest
|
||
<ul>
|
||
<li>The indexing finished and the total number of items is now 104504, but I’m sure the indexes are messed up so I will just start by taking a backup and re-creating them manually before every indexing</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<!-- raw HTML omitted -->
|
||
|
||
|
||
|
||
|
||
|
||
</article>
|
||
|
||
|
||
|
||
</div> <!-- /.blog-main -->
|
||
|
||
<aside class="col-sm-3 ml-auto blog-sidebar">
|
||
|
||
|
||
|
||
<section class="sidebar-module">
|
||
<h4>Recent Posts</h4>
|
||
<ol class="list-unstyled">
|
||
|
||
|
||
<li><a href="/cgspace-notes/2022-01/">January, 2022</a></li>
|
||
|
||
<li><a href="/cgspace-notes/2021-12/">December, 2021</a></li>
|
||
|
||
<li><a href="/cgspace-notes/2021-11/">November, 2021</a></li>
|
||
|
||
<li><a href="/cgspace-notes/2021-10/">October, 2021</a></li>
|
||
|
||
<li><a href="/cgspace-notes/2021-09/">September, 2021</a></li>
|
||
|
||
</ol>
|
||
</section>
|
||
|
||
|
||
|
||
|
||
<section class="sidebar-module">
|
||
<h4>Links</h4>
|
||
<ol class="list-unstyled">
|
||
|
||
<li><a href="https://cgspace.cgiar.org">CGSpace</a></li>
|
||
|
||
<li><a href="https://dspacetest.cgiar.org">DSpace Test</a></li>
|
||
|
||
<li><a href="https://github.com/ilri/DSpace">CGSpace @ GitHub</a></li>
|
||
|
||
</ol>
|
||
</section>
|
||
|
||
</aside>
|
||
|
||
|
||
</div> <!-- /.row -->
|
||
</div> <!-- /.container -->
|
||
|
||
|
||
|
||
<footer class="blog-footer">
|
||
<p dir="auto">
|
||
|
||
Blog template created by <a href="https://twitter.com/mdo">@mdo</a>, ported to Hugo by <a href='https://twitter.com/mralanorth'>@mralanorth</a>.
|
||
|
||
</p>
|
||
<p>
|
||
<a href="#">Back to top</a>
|
||
</p>
|
||
</footer>
|
||
|
||
|
||
</body>
|
||
|
||
</html>
|