Peter said that CTA is having problems submitting an item to CGSpace
Looking at the PostgreSQL stats it seems to be the same issue that Tezira was having last week, as I see the number of connections in ‘idle in transaction’ and ‘waiting for lock’ state are increasing again
I see that CGSpace (linode18) is still using PostgreSQL JDBC driver version 42.2.11, and there were some bugs related to transactions fixed in 42.2.12 (which I had updated in the Ansible playbooks, but not deployed yet)
Peter said that CTA is having problems submitting an item to CGSpace
Looking at the PostgreSQL stats it seems to be the same issue that Tezira was having last week, as I see the number of connections in ‘idle in transaction’ and ‘waiting for lock’ state are increasing again
I see that CGSpace (linode18) is still using PostgreSQL JDBC driver version 42.2.11, and there were some bugs related to transactions fixed in 42.2.12 (which I had updated in the Ansible playbooks, but not deployed yet)
<li>Peter said that CTA is having problems submitting an item to CGSpace
<ul>
<li>Looking at the PostgreSQL stats it seems to be the same issue that Tezira was having last week, as I see the number of connections in ‘idle in transaction’ and ‘waiting for lock’ state are increasing again</li>
<li>I see that CGSpace (linode18) is still using PostgreSQL JDBC driver version 42.2.11, and there were some bugs related to transactions fixed in 42.2.12 (which I had updated in the Ansible playbooks, but not deployed yet)</li>
<li>This is only about 2,500 hits total from the last ten years, and half of these bots no longer seem to exist, so I won’t bother submitting them to the COUNTER-Robots project</li>
<li>I noticed that our custom themes were incorrectly linking to the OpenSearch XML file
<ul>
<li>The bug <ahref="https://jira.lyrasis.org/browse/DS-2592">was fixed</a> for Mirage2 in 2015</li>
<li>Note that this did not prevent OpenSearch itself from working</li>
<li>I will patch this on our DSpace 5.x and 6.x branches</li>
<li>Atmire responded asking for more information about the Solr statistics processing bug in CUA so I sent them some full logs
<ul>
<li>Also I asked again about the Maven variable interpolation issue for <code>cua.version.number</code>, and if they would be willing to upgrade CUA to use Font Awesome 5 instead of 4.</li>
<li>The two main IPs making requests around then are 188.134.31.88 and 212.34.8.188
<ul>
<li>The first is in Russia and it is hitting mostly XMLUI Discover links using <em>dozens</em> of different user agents, a total of 20,000 requests this week</li>
<li>The second IP is CodeObia testing AReS, a total of 171,000 hits this month</li>
<li>I will purge both of those IPs from the Solr stats using my <code>check-spider-ip-hits.sh</code> script:</li>
<li>Bizu said she was having issues submitting to CGSpace last week
<ul>
<li>The issue sounds like the one Tezira and CTA were having in the last few weeks</li>
<li>I looked at the PostgreSQL graphs and see there are a lot of connections in “idle in transaction” and “waiting for lock” state:</li>
<li>I think I’ll downgrade the PostgreSQL JDBC driver from 42.2.12 to 42.2.10, which was the version we were using before these issues started happening</li>
<li>Atmire sent some feedback about my ongoing issues with their CUA module, but none of it was conclusive yet
<ul>
<li>Regarding Font Awesome 5 they will check how much work it will take and give me a quote</li>
</ul>
</li>
<li>Abenet said some users are questioning why the statistics dropped so much lately, so I made a <ahref="https://www.yammer.com/dspacedevelopers/#/Threads/show?threadId=674923030216704">post to Yammer</a> to explain about the robots</li>
<li>Last week Peter had asked me to add a new ILRI author’s ORCID iD
<ul>
<li>I added it to the controlled vocabulary and tagged the user’s existing ~11 items in CGSpace using this CSV file with my <code>add-orcid-identifiers-csv.py</code> script:</li>
<li>Peter noticed that CGSpace is no longer on AReS, because I blocked all requests that don’t specify a user agent
<ul>
<li>I’ve temporarily disabled that restriction and asked Moayad to look into how he can specify a user agent in the AReS harvester</li>
</ul>
</li>
</ul>
<h2id="2020-05-13">2020-05-13</h2>
<ul>
<li>Atmire responded about Font Awesome and said they can switch to version 5 for 16 credits
<ul>
<li>I told them to go ahead</li>
</ul>
</li>
<li>Also, Atmire gave me a small workaround for the <code>cua.version.number</code> interpolation issue and said they would look into the crash that happens when processing our Solr stats</li>
<li>Run system updates and reboot AReS server (linode20) for the first time in almost 100 days
<ul>
<li>I notice that AReS now has some of CGSpace’s data in it (but not all) since I dropped the user-agent restriction on the REST API yesterday</li>
</ul>
</li>
</ul>
<h2id="2020-05-17">2020-05-17</h2>
<ul>
<li>Create an issue in the OpenRXV project for Moayad to change the default harvester user agent (<ahref="https://github.com/ilri/OpenRXV/issues/36">#36</a>)</li>
<li>Atmire responded and said they still can’t figure out the CUA statistics issue, though they seem to only be trying to understand what’s going on using static analysis
<ul>
<li>I told them that they should try to run the code with the Solr statistics that I shared with them a few weeks ago</li>
</ul>
</li>
</ul>
<h2id="2020-05-19">2020-05-19</h2>
<ul>
<li>Add ORCID identifier for Sirak Bahta
<ul>
<li>I added it to the controlled vocabulary and tagged the user’s existing ~40 items in CGSpace using this CSV file with my <code>add-orcid-identifiers-csv.py</code> script:</li>
<li>An IITA user is having issues submitting to CGSpace and I see there are a rising number of PostgreSQL connections waiting in transaction and in lock:</li>
<li>This is the same issue Tezira, Bizu, and CTA were having in the last few weeks and it I already downgraded the PostgreSQL JDBC driver version to the last version I was using before this started (42.2.10)
<ul>
<li>I will downgrade it to version 42.2.9 for now…</li>
<li>The only other thing I can think of is that I upgraded Tomcat to 7.0.103 in March</li>
</ul>
</li>
<li>Run system updates on DSpace Test (linode26) and reboot it</li>
<li>Run system updates on CGSpace (linode18) and reboot it
<ul>
<li>After the system came back up I had to restart Tomcat 7 three times before all the Solr statistics cores came up OK</li>
</ul>
</li>
<li>Send Atmire a snapshot of the CGSpace database for them to possibly troubleshoot the CUA issue with DSpace 6</li>
</ul>
<h2id="2020-05-20">2020-05-20</h2>
<ul>
<li>Send CodeObia some logos and footer text for the next phase of OpenRXV development (<ahref="https://github.com/ilri/OpenRXV/issues/18">#18</a>)</li>
