alanorth/cgspace-notes
1
0
Fork 0
mirror of https://github.com/alanorth/cgspace-notes.git synced 2024-11-26 00:18:21 +01:00
Code Issues Projects Releases Wiki Activity
2bd3c89f18
cgspace-notes/docs/2020-05/index.html

320 lines
12 KiB
HTML
Raw Blame History

<!DOCTYPE html>
<html lang="en" >
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<meta property="og:title" content="May, 2020" />
<meta property="og:description" content="2020-05-02
Peter said that CTA is having problems submitting an item to CGSpace
Looking at the PostgreSQL stats it seems to be the same issue that Tezira was having last week, as I see the number of connections in &lsquo;idle in transaction&rsquo; and &lsquo;waiting for lock&rsquo; state are increasing again
I see that CGSpace (linode18) is still using PostgreSQL JDBC driver version 42.2.11, and there were some bugs related to transactions fixed in 42.2.12 (which I had updated in the Ansible playbooks, but not deployed yet)
" />
<meta property="og:type" content="article" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2020-05/" />
<meta property="article:published_time" content="2020-05-02T09:52:04+03:00" />
<meta property="article:modified_time" content="2020-05-11T17:30:31+03:00" />
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="May, 2020"/>
<meta name="twitter:description" content="2020-05-02
Peter said that CTA is having problems submitting an item to CGSpace
Looking at the PostgreSQL stats it seems to be the same issue that Tezira was having last week, as I see the number of connections in &lsquo;idle in transaction&rsquo; and &lsquo;waiting for lock&rsquo; state are increasing again
I see that CGSpace (linode18) is still using PostgreSQL JDBC driver version 42.2.11, and there were some bugs related to transactions fixed in 42.2.12 (which I had updated in the Ansible playbooks, but not deployed yet)
"/>
<meta name="generator" content="Hugo 0.70.0" />
<script type="application/ld+json">
{
"@context": "http://schema.org",
"@type": "BlogPosting",
"headline": "May, 2020",
"url": "https://alanorth.github.io/cgspace-notes/2020-05/",
"wordCount": "808",
"datePublished": "2020-05-02T09:52:04+03:00",
"dateModified": "2020-05-11T17:30:31+03:00",
"author": {
"@type": "Person",
"name": "Alan Orth"
},
"keywords": "Notes"
}
</script>
<link rel="canonical" href="https://alanorth.github.io/cgspace-notes/2020-05/">
<title>May, 2020 | CGSpace Notes</title>
<!-- combined, minified CSS -->
<link href="https://alanorth.github.io/cgspace-notes/css/style.6da5c906cc7a8fbb93f31cd2316c5dbe3f19ac4aa6bfb066f1243045b8f6061e.css" rel="stylesheet" integrity="sha256-baXJBsx6j7uT8xzSMWxdvj8ZrEqmv7Bm8SQwRbj2Bh4=" crossorigin="anonymous">
<!-- minified Font Awesome for SVG icons -->
<script defer src="https://alanorth.github.io/cgspace-notes/js/fontawesome.min.f3d2a1f5980bab30ddd0d8cadbd496475309fc48e2b1d052c5c09e6facffcb0f.js" integrity="sha256-89Kh9ZgLqzDd0NjK29SWR1MJ/EjisdBSxcCeb6z/yw8=" crossorigin="anonymous"></script>
<!-- RSS 2.0 feed -->
</head>
<body>
<div class="blog-masthead">
<div class="container">
<nav class="nav blog-nav">
<a class="nav-link " href="https://alanorth.github.io/cgspace-notes/">Home</a>
</nav>
</div>
</div>
<header class="blog-header">
<div class="container">
<h1 class="blog-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/" rel="home">CGSpace Notes</a></h1>
<p class="lead blog-description" dir="auto">Documenting day-to-day work on the <a href="https://cgspace.cgiar.org">CGSpace</a> repository.</p>
</div>
</header>
<div class="container">
<div class="row">
<div class="col-sm-8 blog-main">
<article class="blog-post">
<header>
<h2 class="blog-post-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/2020-05/">May, 2020</a></h2>
<p class="blog-post-meta"><time datetime="2020-05-02T09:52:04+03:00">Sat May 02, 2020</time> by Alan Orth in
<span class="fas fa-folder" aria-hidden="true"></span>&nbsp;<a href="/cgspace-notes/categories/notes/" rel="category tag">Notes</a>
</p>
</header>
<h2 id="2020-05-02">2020-05-02</h2>
<ul>
<li>Peter said that CTA is having problems submitting an item to CGSpace
<ul>
<li>Looking at the PostgreSQL stats it seems to be the same issue that Tezira was having last week, as I see the number of connections in &lsquo;idle in transaction&rsquo; and &lsquo;waiting for lock&rsquo; state are increasing again</li>
<li>I see that CGSpace (linode18) is still using PostgreSQL JDBC driver version 42.2.11, and there were some bugs related to transactions fixed in 42.2.12 (which I had updated in the Ansible playbooks, but not deployed yet)</li>
</ul>
</li>
</ul>
<h2 id="2020-05-03">2020-05-03</h2>
<ul>
<li>Purge a few remaining bots from CGSpace Solr statistics that I had identified a few months ago
<ul>
<li><code>lua-resty-http/0.10 (Lua) ngx_lua/10000</code></li>
<li><code>omgili/0.5 +http://omgili.com</code></li>
<li><code>IZaBEE/IZaBEE-1.01 (Buzzing Abound The Web; https://izabee.com; info at izabee dot com)</code></li>
<li><code>Twurly v1.1 (https://twurly.org)</code></li>
<li><code>Pattern/2.6 +http://www.clips.ua.ac.be/pattern</code></li>
<li><code>CyotekWebCopy/1.7 CyotekHTTP/2.0</code></li>
</ul>
</li>
<li>This is only about 2,500 hits total from the last ten years, and half of these bots no longer seem to exist, so I won&rsquo;t bother submitting them to the COUNTER-Robots project</li>
<li>I noticed that our custom themes were incorrectly linking to the OpenSearch XML file
<ul>
<li>The bug <a href="https://jira.lyrasis.org/browse/DS-2592">was fixed</a> for Mirage2 in 2015</li>
<li>Note that this did not prevent OpenSearch itself from working</li>
<li>I will patch this on our DSpace 5.x and 6.x branches</li>
</ul>
</li>
</ul>
<h2 id="2020-05-06">2020-05-06</h2>
<ul>
<li>Atmire responded asking for more information about the Solr statistics processing bug in CUA so I sent them some full logs
<ul>
<li>Also I asked again about the Maven variable interpolation issue for <code>cua.version.number</code>, and if they would be willing to upgrade CUA to use Font Awesome 5 instead of 4.</li>
</ul>
</li>
</ul>
<h2 id="2020-05-07">2020-05-07</h2>
<ul>
<li>Linode sent an alert that there was high CPU usage on CGSpace (linode18) early this morning
<ul>
<li>I looked at the nginx logs using goaccess and I found a few IPs making lots of requests around then:</li>
</ul>
</li>
</ul>
<pre><code># cat /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E &quot;07/May/2020:(01|03|04)&quot; | goaccess --log-format=COMBINED -
</code></pre><ul>
<li>The two main IPs making requests around then are 188.134.31.88 and 212.34.8.188
<ul>
<li>The first is in Russia and it is hitting mostly XMLUI Discover links using <em>dozens</em> of different user agents, a total of 20,000 requests this week</li>
<li>The second IP is CodeObia testing AReS, a total of 171,000 hits this month</li>
<li>I will purge both of those IPs from the Solr stats using my <code>check-spider-ip-hits.sh</code> script:</li>
</ul>
</li>
</ul>
<pre><code>$ ./check-spider-ip-hits.sh -f /tmp/ips -s statistics -p
Purging 171641 hits from 212.34.8.188 in statistics
Purging 20691 hits from 188.134.31.88 in statistics
Total number of bot hits purged: 192332
</code></pre><ul>
<li>And then I will add 188.134.31.88 to the nginx bad bot list and tell CodeObia to please use a &ldquo;bot&rdquo; user agent</li>
<li>I also changed the nginx config to block requests with blank user agents</li>
</ul>
<h2 id="2020-05-11">2020-05-11</h2>
<ul>
<li>Bizu said she was having issues submitting to CGSpace last week
<ul>
<li>The issue sounds like the one Tezira and CTA were having in the last few weeks</li>
<li>I looked at the PostgreSQL graphs and see there are a lot of connections in &ldquo;idle in transaction&rdquo; and &ldquo;waiting for lock&rdquo; state:</li>
</ul>
</li>
</ul>
<p><img src="/cgspace-notes/2020/05/postgres_connections_cgspace-week.png" alt="PostgreSQL connections"></p>
<ul>
<li>I think I&rsquo;ll downgrade the PostgreSQL JDBC driver from 42.2.12 to 42.2.10, which was the version we were using before these issues started happening</li>
<li>Atmire sent some feedback about my ongoing issues with their CUA module, but none of it was conclusive yet
<ul>
<li>Regarding Font Awesome 5 they will check how much work it will take and give me a quote</li>
</ul>
</li>
<li>Abenet said some users are questioning why the statistics dropped so much lately, so I made a <a href="https://www.yammer.com/dspacedevelopers/#/Threads/show?threadId=674923030216704">post to Yammer</a> to explain about the robots</li>
<li>Last week Peter had asked me to add a new ILRI author&rsquo;s ORCID iD
<ul>
<li>I added it to the controlled vocabulary and tagged the user&rsquo;s existing ~11 items in CGSpace using this CSV file with my <code>add-orcid-identifiers-csv.py</code> script:</li>
</ul>
</li>
</ul>
<pre><code>$ cat 2020-05-11-add-orcids.csv
dc.contributor.author,cg.creator.id
&quot;Lutakome, P.&quot;,&quot;Pius Lutakome: 0000-0002-0804-2649&quot;
&quot;Lutakome, Pius&quot;,&quot;Pius Lutakome: 0000-0002-0804-2649&quot;
$ ./add-orcid-identifiers-csv.py -i 2020-05-11-add-orcids.csv -db dspace -u dspace -p 'fuuu' -d
</code></pre><ul>
<li>Run system updates on CGSpace (linode18) and reboot it
<ul>
<li>I had to restart Tomcat five times before all Solr statistics cores came up OK, ugh.</li>
</ul>
</li>
</ul>
<h2 id="2020-05-12">2020-05-12</h2>
<ul>
<li>Peter noticed that CGSpace is no longer on AReS, because I blocked all requests that don&rsquo;t specify a user agent
<ul>
<li>I&rsquo;ve temporarily disabled that restriction and asked Moayad to look into how he can specify a user agent in the AReS harvester</li>
</ul>
</li>
</ul>
<h2 id="2020-05-13">2020-05-13</h2>
<ul>
<li>Atmire responded about Font Awesome and said they can switch to version 5 for 16 credits
<ul>
<li>I told them to go ahead</li>
</ul>
</li>
<li>Also, Atmire gave me a small workaround for the <code>cua.version.number</code> interpolation issue and said they would look into the crash that happens when processing our Solr stats</li>
<li>Run system updates and reboot AReS server (linode20) for the first time in almost 100 days
<ul>
<li>I notice that AReS now has some of CGSpace&rsquo;s data in it (but not all) since I dropped the user-agent restriction on the REST API yesterday</li>
</ul>
</li>
</ul>
<h2 id="2020-05-17">2020-05-17</h2>
<ul>
<li>Create an issue in the OpenRXV project for Moayad to change the default harvester user agent (<a href="https://github.com/ilri/OpenRXV/issues/36">#36</a>)</li>
</ul>
<!-- raw HTML omitted -->
</article>
</div> <!-- /.blog-main -->
<aside class="col-sm-3 ml-auto blog-sidebar">
<section class="sidebar-module">
<h4>Recent Posts</h4>
<ol class="list-unstyled">
<li><a href="/cgspace-notes/2020-05/">May, 2020</a></li>
<li><a href="/cgspace-notes/2020-04/">April, 2020</a></li>
<li><a href="/cgspace-notes/2020-03/">March, 2020</a></li>
<li><a href="/cgspace-notes/2020-02/">February, 2020</a></li>
<li><a href="/cgspace-notes/2020-01/">January, 2020</a></li>
</ol>
</section>
<section class="sidebar-module">
<h4>Links</h4>
<ol class="list-unstyled">
<li><a href="https://cgspace.cgiar.org">CGSpace</a></li>
<li><a href="https://dspacetest.cgiar.org">DSpace Test</a></li>
<li><a href="https://github.com/ilri/DSpace">CGSpace @ GitHub</a></li>
</ol>
</section>
</aside>
</div> <!-- /.row -->
</div> <!-- /.container -->
<footer class="blog-footer">
<p dir="auto">
Blog template created by <a href="https://twitter.com/mdo">@mdo</a>, ported to Hugo by <a href='https://twitter.com/mralanorth'>@mralanorth</a>.
</p>
<p>
<a href="#">Back to top</a>
</p>
</footer>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink