mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2024-12-22 21:22:19 +01:00
523 lines
17 KiB
HTML
523 lines
17 KiB
HTML
<!DOCTYPE html>
|
||
<html lang="en">
|
||
|
||
<head>
|
||
<meta charset="utf-8">
|
||
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
|
||
|
||
<meta property="og:title" content="CGSpace Notes" />
|
||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||
<meta property="og:type" content="website" />
|
||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
|
||
|
||
<meta property="og:updated_time" content="2019-08-03T12:39:51+03:00" />
|
||
|
||
<meta name="twitter:card" content="summary"/>
|
||
<meta name="twitter:title" content="CGSpace Notes"/>
|
||
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
|
||
<meta name="generator" content="Hugo 0.57.2" />
|
||
|
||
|
||
|
||
<script type="application/ld+json">
|
||
{
|
||
"@context": "http://schema.org",
|
||
"@type": "Blog",
|
||
"headline": "CGSpace Notes",
|
||
"url" : "https:\/\/alanorth.github.io\/cgspace-notes\/",
|
||
"author": {
|
||
"@type": "Person",
|
||
"name": "Alan Orth"
|
||
},
|
||
"dateModified": "2019-08-03T12:39:51\x2b03:00",
|
||
"keywords": "notes,notes,",
|
||
"description": "Documenting day-to-day work on the [CGSpace](https:\/\/cgspace.cgiar.org) repository."
|
||
}
|
||
</script>
|
||
|
||
|
||
<link rel="canonical" href="https://alanorth.github.io/cgspace-notes/">
|
||
|
||
<title>CGSpace Notes</title>
|
||
|
||
<!-- combined, minified CSS -->
|
||
<link href="https://alanorth.github.io/cgspace-notes/css/style.css" rel="stylesheet" integrity="sha384-G5B34w7DFTumWTswxYzTX7NWfbvQEg1HbFFEg6ItN03uTAAoS2qkPS/fu3LhuuSA" crossorigin="anonymous">
|
||
|
||
<!-- RSS 2.0 feed -->
|
||
<link rel="alternate" type="application/rss+xml" href="https://alanorth.github.io/cgspace-notes/index.xml" title="CGSpace Notes" />
|
||
|
||
|
||
|
||
|
||
|
||
|
||
</head>
|
||
|
||
<body>
|
||
|
||
|
||
<div class="blog-masthead">
|
||
<div class="container">
|
||
<nav class="nav blog-nav">
|
||
<a class="nav-link active" href="https://alanorth.github.io/cgspace-notes/">Home</a>
|
||
</nav>
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
<header class="blog-header">
|
||
<div class="container">
|
||
<h1 class="blog-title"><a href="https://alanorth.github.io/cgspace-notes/" rel="home">CGSpace Notes</a></h1>
|
||
<p class="lead blog-description">Documenting day-to-day work on the <a href="https://cgspace.cgiar.org">CGSpace</a> repository.</p>
|
||
</div>
|
||
</header>
|
||
|
||
|
||
|
||
|
||
<div class="container">
|
||
<div class="row">
|
||
<div class="col-sm-8 blog-main">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<article class="blog-post">
|
||
<header>
|
||
<h2 class="blog-post-title"><a href="https://alanorth.github.io/cgspace-notes/2019-08/">August, 2019</a></h2>
|
||
<p class="blog-post-meta"><time datetime="2019-08-03T12:39:51+03:00">Sat Aug 03, 2019</time> by Alan Orth in
|
||
|
||
<i class="fa fa-tag" aria-hidden="true"></i> <a href="/cgspace-notes/tags/notes" rel="tag">Notes</a>
|
||
|
||
</p>
|
||
</header>
|
||
<h2 id="2019-08-03">2019-08-03</h2>
|
||
|
||
<ul>
|
||
<li>Look at Bioversity’s latest migration CSV and now I see that Francesco has cleaned up the extra columns and the newline at the end of the file, but many of the column headers have an extra space in the name…</li>
|
||
</ul>
|
||
|
||
<h2 id="2019-08-04">2019-08-04</h2>
|
||
|
||
<ul>
|
||
<li>Deploy ORCID identifier updates requested by Bioversity to CGSpace</li>
|
||
<li>Run system updates on CGSpace (linode18) and reboot it
|
||
|
||
<ul>
|
||
<li>Before updating it I checked Solr and verified that all statistics cores were loaded properly…</li>
|
||
<li>After rebooting, all statistics cores were loaded… wow, that’s lucky.</li>
|
||
</ul></li>
|
||
<li>Run system updates on DSpace Test (linode19) and reboot it</li>
|
||
</ul>
|
||
<a href='https://alanorth.github.io/cgspace-notes/2019-08/'>Read more →</a>
|
||
</article>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<article class="blog-post">
|
||
<header>
|
||
<h2 class="blog-post-title"><a href="https://alanorth.github.io/cgspace-notes/2019-07/">July, 2019</a></h2>
|
||
<p class="blog-post-meta"><time datetime="2019-07-01T12:13:51+03:00">Mon Jul 01, 2019</time> by Alan Orth in
|
||
|
||
<i class="fa fa-tag" aria-hidden="true"></i> <a href="/cgspace-notes/tags/notes" rel="tag">Notes</a>
|
||
|
||
</p>
|
||
</header>
|
||
<h2 id="2019-07-01">2019-07-01</h2>
|
||
|
||
<ul>
|
||
<li>Create an “AfricaRice books and book chapters” collection on CGSpace for AfricaRice</li>
|
||
<li>Last month Sisay asked why the following “most popular” statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace:
|
||
|
||
<ul>
|
||
<li><a href="https://dspacetest.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&time_filter_end_date=01%2F12%2F2018">DSpace Test</a></li>
|
||
<li><a href="https://cgspace.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&time_filter_end_date=01%2F12%2F2018">CGSpace</a></li>
|
||
</ul></li>
|
||
<li>Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community</li>
|
||
</ul>
|
||
<a href='https://alanorth.github.io/cgspace-notes/2019-07/'>Read more →</a>
|
||
</article>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<article class="blog-post">
|
||
<header>
|
||
<h2 class="blog-post-title"><a href="https://alanorth.github.io/cgspace-notes/2019-06/">June, 2019</a></h2>
|
||
<p class="blog-post-meta"><time datetime="2019-06-02T10:57:51+03:00">Sun Jun 02, 2019</time> by Alan Orth in
|
||
|
||
<i class="fa fa-tag" aria-hidden="true"></i> <a href="/cgspace-notes/tags/notes" rel="tag">Notes</a>
|
||
|
||
</p>
|
||
</header>
|
||
<h2 id="2019-06-02">2019-06-02</h2>
|
||
|
||
<ul>
|
||
<li>Merge the <a href="https://github.com/ilri/DSpace/pull/425">Solr filterCache</a> and <a href="https://github.com/ilri/DSpace/pull/426">XMLUI ISI journal</a> changes to the <code>5_x-prod</code> branch and deploy on CGSpace</li>
|
||
<li>Run system updates on CGSpace (linode18) and reboot it</li>
|
||
</ul>
|
||
|
||
<h2 id="2019-06-03">2019-06-03</h2>
|
||
|
||
<ul>
|
||
<li>Skype with Marie-Angélique and Abenet about <a href="https://agriculturalsemantics.github.io/cg-core/cgcore.html">CG Core v2</a></li>
|
||
</ul>
|
||
<a href='https://alanorth.github.io/cgspace-notes/2019-06/'>Read more →</a>
|
||
</article>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<article class="blog-post">
|
||
<header>
|
||
<h2 class="blog-post-title"><a href="https://alanorth.github.io/cgspace-notes/2019-05/">May, 2019</a></h2>
|
||
<p class="blog-post-meta"><time datetime="2019-05-01T07:37:43+03:00">Wed May 01, 2019</time> by Alan Orth in
|
||
|
||
<i class="fa fa-tag" aria-hidden="true"></i> <a href="/cgspace-notes/tags/notes" rel="tag">Notes</a>
|
||
|
||
</p>
|
||
</header>
|
||
<h2 id="2019-05-01">2019-05-01</h2>
|
||
|
||
<ul>
|
||
<li>Help CCAFS with regenerating some item thumbnails after they uploaded new PDFs to some items on CGSpace</li>
|
||
<li>A user on the dspace-tech mailing list offered some suggestions for troubleshooting the problem with the inability to delete certain items
|
||
|
||
<ul>
|
||
<li>Apparently if the item is in the <code>workflowitem</code> table it is submitted to a workflow</li>
|
||
<li>And if it is in the <code>workspaceitem</code> table it is in the pre-submitted state</li>
|
||
</ul></li>
|
||
|
||
<li><p>The item seems to be in a pre-submitted state, so I tried to delete it from there:</p>
|
||
|
||
<pre><code>dspace=# DELETE FROM workspaceitem WHERE item_id=74648;
|
||
DELETE 1
|
||
</code></pre></li>
|
||
|
||
<li><p>But after this I tried to delete the item from the XMLUI and it is <em>still</em> present…</p></li>
|
||
</ul>
|
||
<a href='https://alanorth.github.io/cgspace-notes/2019-05/'>Read more →</a>
|
||
</article>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<article class="blog-post">
|
||
<header>
|
||
<h2 class="blog-post-title"><a href="https://alanorth.github.io/cgspace-notes/2019-04/">April, 2019</a></h2>
|
||
<p class="blog-post-meta"><time datetime="2019-04-01T09:00:43+03:00">Mon Apr 01, 2019</time> by Alan Orth in
|
||
|
||
<i class="fa fa-tag" aria-hidden="true"></i> <a href="/cgspace-notes/tags/notes" rel="tag">Notes</a>
|
||
|
||
</p>
|
||
</header>
|
||
<h2 id="2019-04-01">2019-04-01</h2>
|
||
|
||
<ul>
|
||
<li>Meeting with AgroKnow to discuss CGSpace, ILRI data, AReS, GARDIAN, etc
|
||
|
||
<ul>
|
||
<li>They asked if we had plans to enable RDF support in CGSpace</li>
|
||
</ul></li>
|
||
|
||
<li><p>There have been 4,400 more downloads of the CTA Spore publication from those strange Amazon IP addresses today</p>
|
||
|
||
<ul>
|
||
<li><p>I suspected that some might not be successful, because the stats show less, but today they were all HTTP 200!</p>
|
||
|
||
<pre><code># cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep 'Spore-192-EN-web.pdf' | grep -E '(18.196.196.108|18.195.78.144|18.195.218.6)' | awk '{print $9}' | sort | uniq -c | sort -n | tail -n 5
|
||
4432 200
|
||
</code></pre></li>
|
||
</ul></li>
|
||
|
||
<li><p>In the last two weeks there have been 47,000 downloads of this <em>same exact PDF</em> by these three IP addresses</p></li>
|
||
|
||
<li><p>Apply country and region corrections and deletions on DSpace Test and CGSpace:</p>
|
||
|
||
<pre><code>$ ./fix-metadata-values.py -i /tmp/2019-02-21-fix-9-countries.csv -db dspace -u dspace -p 'fuuu' -f cg.coverage.country -m 228 -t ACTION -d
|
||
$ ./fix-metadata-values.py -i /tmp/2019-02-21-fix-4-regions.csv -db dspace -u dspace -p 'fuuu' -f cg.coverage.region -m 231 -t action -d
|
||
$ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-2-countries.csv -db dspace -u dspace -p 'fuuu' -m 228 -f cg.coverage.country -d
|
||
$ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace -u dspace -p 'fuuu' -m 231 -f cg.coverage.region -d
|
||
</code></pre></li>
|
||
</ul>
|
||
<a href='https://alanorth.github.io/cgspace-notes/2019-04/'>Read more →</a>
|
||
</article>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<article class="blog-post">
|
||
<header>
|
||
<h2 class="blog-post-title"><a href="https://alanorth.github.io/cgspace-notes/2019-03/">March, 2019</a></h2>
|
||
<p class="blog-post-meta"><time datetime="2019-03-01T12:16:30+01:00">Fri Mar 01, 2019</time> by Alan Orth in
|
||
|
||
<i class="fa fa-tag" aria-hidden="true"></i> <a href="/cgspace-notes/tags/notes" rel="tag">Notes</a>
|
||
|
||
</p>
|
||
</header>
|
||
<h2 id="2019-03-01">2019-03-01</h2>
|
||
|
||
<ul>
|
||
<li>I checked IITA’s 259 Feb 14 records from last month for duplicates using Atmire’s Duplicate Checker on a fresh snapshot of CGSpace on my local machine and everything looks good</li>
|
||
<li>I am now only waiting to hear from her about where the items should go, though I assume Journal Articles go to IITA Journal Articles collection, etc…</li>
|
||
<li>Looking at the other half of Udana’s WLE records from 2018-11
|
||
|
||
<ul>
|
||
<li>I finished the ones for Restoring Degraded Landscapes (RDL), but these are for Variability, Risks and Competing Uses (VRC)</li>
|
||
<li>I did the usual cleanups for whitespace, added regions where they made sense for certain countries, cleaned up the DOI link formats, added rights information based on the publications page for a few items</li>
|
||
<li>Most worryingly, there are encoding errors in the abstracts for eleven items, for example:</li>
|
||
<li>68.15% <20> 9.45 instead of 68.15% ± 9.45</li>
|
||
<li>2003<EFBFBD>2013 instead of 2003–2013</li>
|
||
</ul></li>
|
||
<li>I think I will need to ask Udana to re-copy and paste the abstracts with more care using Google Docs</li>
|
||
</ul>
|
||
<a href='https://alanorth.github.io/cgspace-notes/2019-03/'>Read more →</a>
|
||
</article>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<article class="blog-post">
|
||
<header>
|
||
<h2 class="blog-post-title"><a href="https://alanorth.github.io/cgspace-notes/2019-02/">February, 2019</a></h2>
|
||
<p class="blog-post-meta"><time datetime="2019-02-01T21:37:30+02:00">Fri Feb 01, 2019</time> by Alan Orth in
|
||
|
||
<i class="fa fa-tag" aria-hidden="true"></i> <a href="/cgspace-notes/tags/notes" rel="tag">Notes</a>
|
||
|
||
</p>
|
||
</header>
|
||
<h2 id="2019-02-01">2019-02-01</h2>
|
||
|
||
<ul>
|
||
<li>Linode has alerted a few times since last night that the CPU usage on CGSpace (linode18) was high despite me increasing the alert threshold last week from 250% to 275%—I might need to increase it again!</li>
|
||
|
||
<li><p>The top IPs before, during, and after this latest alert tonight were:</p>
|
||
|
||
<pre><code># zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "01/Feb/2019:(17|18|19|20|21)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
|
||
245 207.46.13.5
|
||
332 54.70.40.11
|
||
385 5.143.231.38
|
||
405 207.46.13.173
|
||
405 207.46.13.75
|
||
1117 66.249.66.219
|
||
1121 35.237.175.180
|
||
1546 5.9.6.51
|
||
2474 45.5.186.2
|
||
5490 85.25.237.71
|
||
</code></pre></li>
|
||
|
||
<li><p><code>85.25.237.71</code> is the “Linguee Bot” that I first saw last month</p></li>
|
||
|
||
<li><p>The Solr statistics the past few months have been very high and I was wondering if the web server logs also showed an increase</p></li>
|
||
|
||
<li><p>There were just over 3 million accesses in the nginx logs last month:</p>
|
||
|
||
<pre><code># time zcat --force /var/log/nginx/* | grep -cE "[0-9]{1,2}/Jan/2019"
|
||
3018243
|
||
|
||
real 0m19.873s
|
||
user 0m22.203s
|
||
sys 0m1.979s
|
||
</code></pre></li>
|
||
</ul>
|
||
<a href='https://alanorth.github.io/cgspace-notes/2019-02/'>Read more →</a>
|
||
</article>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<article class="blog-post">
|
||
<header>
|
||
<h2 class="blog-post-title"><a href="https://alanorth.github.io/cgspace-notes/2019-01/">January, 2019</a></h2>
|
||
<p class="blog-post-meta"><time datetime="2019-01-02T09:48:30+02:00">Wed Jan 02, 2019</time> by Alan Orth in
|
||
|
||
<i class="fa fa-tag" aria-hidden="true"></i> <a href="/cgspace-notes/tags/notes" rel="tag">Notes</a>
|
||
|
||
</p>
|
||
</header>
|
||
<h2 id="2019-01-02">2019-01-02</h2>
|
||
|
||
<ul>
|
||
<li>Linode alerted that CGSpace (linode18) had a higher outbound traffic rate than normal early this morning</li>
|
||
|
||
<li><p>I don’t see anything interesting in the web server logs around that time though:</p>
|
||
|
||
<pre><code># zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "02/Jan/2019:0(1|2|3)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
|
||
92 40.77.167.4
|
||
99 210.7.29.100
|
||
120 38.126.157.45
|
||
177 35.237.175.180
|
||
177 40.77.167.32
|
||
216 66.249.75.219
|
||
225 18.203.76.93
|
||
261 46.101.86.248
|
||
357 207.46.13.1
|
||
903 54.70.40.11
|
||
</code></pre></li>
|
||
</ul>
|
||
<a href='https://alanorth.github.io/cgspace-notes/2019-01/'>Read more →</a>
|
||
</article>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<article class="blog-post">
|
||
<header>
|
||
<h2 class="blog-post-title"><a href="https://alanorth.github.io/cgspace-notes/2018-12/">December, 2018</a></h2>
|
||
<p class="blog-post-meta"><time datetime="2018-12-02T02:09:30+02:00">Sun Dec 02, 2018</time> by Alan Orth in
|
||
|
||
<i class="fa fa-tag" aria-hidden="true"></i> <a href="/cgspace-notes/tags/notes" rel="tag">Notes</a>
|
||
|
||
</p>
|
||
</header>
|
||
<h2 id="2018-12-01">2018-12-01</h2>
|
||
|
||
<ul>
|
||
<li>Switch CGSpace (linode18) to use OpenJDK instead of Oracle JDK</li>
|
||
<li>I manually installed OpenJDK, then removed Oracle JDK, then re-ran the <a href="http://github.com/ilri/rmg-ansible-public">Ansible playbook</a> to update all configuration files, etc</li>
|
||
<li>Then I ran all system updates and restarted the server</li>
|
||
</ul>
|
||
|
||
<h2 id="2018-12-02">2018-12-02</h2>
|
||
|
||
<ul>
|
||
<li>I noticed that there is another issue with PDF thumbnails on CGSpace, and I see there was another <a href="https://usn.ubuntu.com/3831-1/">Ghostscript vulnerability last week</a></li>
|
||
</ul>
|
||
<a href='https://alanorth.github.io/cgspace-notes/2018-12/'>Read more →</a>
|
||
</article>
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<article class="blog-post">
|
||
<header>
|
||
<h2 class="blog-post-title"><a href="https://alanorth.github.io/cgspace-notes/2018-11/">November, 2018</a></h2>
|
||
<p class="blog-post-meta"><time datetime="2018-11-01T16:41:30+02:00">Thu Nov 01, 2018</time> by Alan Orth in
|
||
|
||
<i class="fa fa-tag" aria-hidden="true"></i> <a href="/cgspace-notes/tags/notes" rel="tag">Notes</a>
|
||
|
||
</p>
|
||
</header>
|
||
<h2 id="2018-11-01">2018-11-01</h2>
|
||
|
||
<ul>
|
||
<li>Finalize AReS Phase I and Phase II ToRs</li>
|
||
<li>Send a note about my <a href="https://github.com/ilri/dspace-statistics-api">dspace-statistics-api</a> to the dspace-tech mailing list</li>
|
||
</ul>
|
||
|
||
<h2 id="2018-11-03">2018-11-03</h2>
|
||
|
||
<ul>
|
||
<li>Linode has been sending mails a few times a day recently that CGSpace (linode18) has had high CPU usage</li>
|
||
<li>Today these are the top 10 IPs:</li>
|
||
</ul>
|
||
<a href='https://alanorth.github.io/cgspace-notes/2018-11/'>Read more →</a>
|
||
</article>
|
||
|
||
|
||
|
||
|
||
|
||
<nav class="blog-pagination">
|
||
|
||
|
||
|
||
<a class="btn btn-outline-primary disabled" href="#" role="button" aria-disabled="true">Previous page</a>
|
||
<a class="btn btn-outline-primary" href="/cgspace-notes/page/2/" rel="next" role="button">Next page</a>
|
||
|
||
</nav>
|
||
|
||
|
||
|
||
|
||
|
||
</div> <!-- /.blog-main -->
|
||
|
||
<aside class="col-sm-3 ml-auto blog-sidebar">
|
||
|
||
|
||
|
||
<section class="sidebar-module">
|
||
<h4>Recent Posts</h4>
|
||
<ol class="list-unstyled">
|
||
|
||
|
||
<li><a href="/cgspace-notes/2019-08/">August, 2019</a></li>
|
||
|
||
<li><a href="/cgspace-notes/posts/">Posts</a></li>
|
||
|
||
<li><a href="/cgspace-notes/2019-07/">July, 2019</a></li>
|
||
|
||
<li><a href="/cgspace-notes/2019-06/">June, 2019</a></li>
|
||
|
||
<li><a href="/cgspace-notes/2019-05/">May, 2019</a></li>
|
||
|
||
</ol>
|
||
</section>
|
||
|
||
|
||
|
||
|
||
<section class="sidebar-module">
|
||
<h4>Links</h4>
|
||
<ol class="list-unstyled">
|
||
|
||
<li><a href="https://cgspace.cgiar.org">CGSpace</a></li>
|
||
|
||
<li><a href="https://dspacetest.cgiar.org">DSpace Test</a></li>
|
||
|
||
<li><a href="https://github.com/ilri/DSpace">CGSpace @ GitHub</a></li>
|
||
|
||
</ol>
|
||
</section>
|
||
|
||
</aside>
|
||
|
||
|
||
</div> <!-- /.row -->
|
||
</div> <!-- /.container -->
|
||
|
||
|
||
|
||
<footer class="blog-footer">
|
||
<p>
|
||
|
||
Blog template created by <a href="https://twitter.com/mdo">@mdo</a>, ported to Hugo by <a href='https://twitter.com/mralanorth'>@mralanorth</a>.
|
||
|
||
</p>
|
||
<p>
|
||
<a href="#">Back to top</a>
|
||
</p>
|
||
</footer>
|
||
|
||
|
||
</body>
|
||
|
||
</html>
|