Add notes for 2018-02-27

content/post/2018-02.md

@@ -903,3 +903,45 @@ Nor Azwadi: 0000-0001-9634-1958
 - I don't know why it would take so long, but this logic kinda makes sense
 - I think I should increase the `removeAbandonedTimeout` from 90 to something like 180 and continue observing
 - I also reduced the timeout for the API pool back to 60 because those interfaces are only used by bots
+
+## 2018-02-27
+
+- Peter is still having problems with "Socket closed" on his submissions page
+- I have disabled `removeAbandoned` for now because that's the only thing I changed in the last few weeks since he started having issues
+- I think the real line of logic to follow here is why the submissions page is so slow for him (presumably because of loading all his submissions?)
+- I need to see which SQL queries are run during that time
+- And only a few hours after I disabled the `removeAbandoned` thing CGSpace went down and lo and behold, there were 264 connections, most of which were idle:
+
+```
+$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+      5 dspaceApi
+    279 dspaceWeb
+$ psql -c 'select * from pg_stat_activity' | grep dspaceWeb | grep -c "idle in transaction"
+218
+```
+
+- So I'm re-enabling the `removeAbandoned` setting
+- I grabbed a snapshot of the active connections in `pg_stat_activity` for all queries running longer than 2 minutes:
+
+```
+dspace=# \copy (SELECT now() - query_start as "runtime", application_name, usename, datname, waiting, state, query
+  FROM  pg_stat_activity
+  WHERE now() - query_start > '2 minutes'::interval
+ ORDER BY runtime DESC) to /tmp/2018-02-27-postgresql.txt
+COPY 263
+```
+
+- 100 of these idle in transaction connections are the following query:
+
+```
+SELECT * FROM resourcepolicy WHERE resource_type_id= $1 AND resource_id= $2 AND action_id= $3
+```
+
+- ... but according to the [pg_locks documentation](https://www.postgresql.org/docs/9.5/static/view-pg-locks.html) I should have done this to correlate the locks with the activity:
+
+```
+SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;
+```
+
+- Finally finished the [orcid-authority-to-item.py](https://gist.github.com/alanorth/6d7489b50f06a6a1f04ae1c8b899cb6e) script!
+- It successfully mapped 2600 ORCID identifiers to items!

docs/2015-11/index.html

@@ -53,7 +53,7 @@ $ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspac
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2015-12/index.html

@@ -55,7 +55,7 @@ Replace lzop with xz in log compression cron jobs on DSpace Test—it uses less
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2016-01/index.html

@@ -45,7 +45,7 @@ Update GitHub wiki for documentation of maintenance tasks.
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2016-02/index.html

@@ -59,7 +59,7 @@ Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE&r
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2016-03/index.html

@@ -45,7 +45,7 @@ Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Ja
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2016-04/index.html

@@ -49,7 +49,7 @@ Also, I noticed the checker log has some errors we should pay attention to:
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2016-05/index.html

@@ -53,7 +53,7 @@ There are 3,000 IPs accessing the REST API in a 24-hour period!
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2016-06/index.html

@@ -51,7 +51,7 @@ Working on second phase of metadata migration, looks like this will work for mov
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2016-07/index.html

@@ -67,7 +67,7 @@ In this case the select query was showing 95 results before the update
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2016-08/index.html

@@ -61,7 +61,7 @@ $ git rebase -i dspace-5.5
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2016-09/index.html

@@ -53,7 +53,7 @@ $ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=or
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2016-10/index.html

@@ -61,7 +61,7 @@ I exported a random item’s metadata as CSV, deleted all columns except id
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2016-11/index.html

@@ -45,7 +45,7 @@ Add dc.type to the output options for Atmire’s Listings and Reports module
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2016-12/index.html

@@ -69,7 +69,7 @@ Another worrying error from dspace.log is:
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2017-01/index.html

@@ -45,7 +45,7 @@ I asked on the dspace-tech mailing list because it seems to be broken, and actua
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2017-02/index.html

@@ -73,7 +73,7 @@ Looks like we’ll be using cg.identifier.ccafsprojectpii as the field name
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2017-03/index.html

@@ -77,7 +77,7 @@ $ identify ~/Desktop/alc_contrastes_desafios.jpg
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2017-04/index.html

@@ -63,7 +63,7 @@ $ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Th
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2017-05/index.html

@@ -29,7 +29,7 @@
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="May, 2017"/>
 <meta name="twitter:description" content="2017-05-01  ICARDA apparently started working on CG Core on their MEL repository They have done a few cg.* fields, but not very consistent and even copy some of CGSpace items:  https://mel.cgiar.org/xmlui/handle/20.500.11766/6911?show=full https://cgspace.cgiar.org/handle/10568/73683   2017-05-02  Atmire got back about the Workflow Statistics issue, and apparently it&rsquo;s a bug in the CUA module so they will send us a pull request  2017-05-04  Sync DSpace Test with database and assetstore from CGSpace Re-deploy DSpace Test with Atmire&rsquo;s CUA patch for workflow statistics, run system updates, and restart the server Now I can see the workflow statistics and am able to select users, but everything returns 0 items Megan says there are still some mapped items are not appearing since last week, so I forced a full index-discovery -b Need to remember to check if the collection has more items (currently 39 on CGSpace, but 118 on the freshly reindexed DSPace Test) tomorrow: https://cgspace."/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2017-06/index.html

@@ -29,7 +29,7 @@
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="June, 2017"/>
 <meta name="twitter:description" content="2017-06-01  After discussion with WLE and CGSpace content people, we decided to just add one metadata field for the WLE Research Themes The cg.identifier.wletheme field will be used for both Phase I and Phase II Research Themes Then we&rsquo;ll create a new sub-community for Phase II and create collections for the research themes there The current &ldquo;Research Themes&rdquo; community will be renamed to &ldquo;WLE Phase I Research Themes&rdquo; Tagged all items in the current Phase I collections with their appropriate themes Create pull request to add Phase II research themes to the submission form: #328 Add cg."/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2017-07/index.html

@@ -57,7 +57,7 @@ We can use PostgreSQL’s extended output format (-x) plus sed to format the
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2017-08/index.html

@@ -77,7 +77,7 @@ Then I cleaned up the author authorities and HTML characters in OpenRefine and s
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2017-09/index.html

@@ -53,7 +53,7 @@ Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2017-10/index.html

@@ -57,7 +57,7 @@ Add Katherine Lutz to the groups for content submission and edit steps of the CG
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2017-11/index.html

@@ -77,7 +77,7 @@ COPY 54701
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2017-12/index.html

@@ -47,7 +47,7 @@ The list of connections to XMLUI and REST API for today:
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2018-01/index.html

@@ -185,7 +185,7 @@ Danny wrote to ask for help renewing the wildcard ilri.org certificate and I adv
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/2018-02/index.html

@@ -23,7 +23,7 @@ I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu’s munin-pl
 
 <meta property="article:published_time" content="2018-02-01T16:28:54&#43;02:00"/>
 
-<meta property="article:modified_time" content="2018-02-26T16:41:28&#43;02:00"/>
+<meta property="article:modified_time" content="2018-02-26T18:12:27&#43;02:00"/>
 
 
 
@@ -47,7 +47,7 @@ I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu&rsquo;s munin-pl
 
 
 "/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     
@@ -57,9 +57,9 @@ I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu&rsquo;s munin-pl
   "@type": "BlogPosting",
   "headline": "February, 2018",
   "url": "https://alanorth.github.io/cgspace-notes/2018-02/",
-  "wordCount": "5609",
+  "wordCount": "5873",
   "datePublished": "2018-02-01T16:28:54&#43;02:00",
-  "dateModified": "2018-02-26T16:41:28&#43;02:00",
+  "dateModified": "2018-02-26T18:12:27&#43;02:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"
@@ -1136,6 +1136,54 @@ Nor Azwadi: 0000-0001-9634-1958
 <li>I also reduced the timeout for the API pool back to 60 because those interfaces are only used by bots</li>
 </ul>
 
+<h2 id="2018-02-27">2018-02-27</h2>
+
+<ul>
+<li>Peter is still having problems with &ldquo;Socket closed&rdquo; on his submissions page</li>
+<li>I have disabled <code>removeAbandoned</code> for now because that&rsquo;s the only thing I changed in the last few weeks since he started having issues</li>
+<li>I think the real line of logic to follow here is why the submissions page is so slow for him (presumably because of loading all his submissions?)</li>
+<li>I need to see which SQL queries are run during that time</li>
+<li>And only a few hours after I disabled the <code>removeAbandoned</code> thing CGSpace went down and lo and behold, there were 264 connections, most of which were idle:</li>
+</ul>
+
+<pre><code>$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+      5 dspaceApi
+    279 dspaceWeb
+$ psql -c 'select * from pg_stat_activity' | grep dspaceWeb | grep -c &quot;idle in transaction&quot;
+218
+</code></pre>
+
+<ul>
+<li>So I&rsquo;m re-enabling the <code>removeAbandoned</code> setting</li>
+<li>I grabbed a snapshot of the active connections in <code>pg_stat_activity</code> for all queries running longer than 2 minutes:</li>
+</ul>
+
+<pre><code>dspace=# \copy (SELECT now() - query_start as &quot;runtime&quot;, application_name, usename, datname, waiting, state, query
+  FROM  pg_stat_activity
+  WHERE now() - query_start &gt; '2 minutes'::interval
+ ORDER BY runtime DESC) to /tmp/2018-02-27-postgresql.txt
+COPY 263
+</code></pre>
+
+<ul>
+<li>100 of these idle in transaction connections are the following query:</li>
+</ul>
+
+<pre><code>SELECT * FROM resourcepolicy WHERE resource_type_id= $1 AND resource_id= $2 AND action_id= $3
+</code></pre>
+
+<ul>
+<li>&hellip; but according to the <a href="https://www.postgresql.org/docs/9.5/static/view-pg-locks.html">pg_locks documentation</a> I should have done this to correlate the locks with the activity:</li>
+</ul>
+
+<pre><code>SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;
+</code></pre>
+
+<ul>
+<li>Finally finished the <a href="https://gist.github.com/alanorth/6d7489b50f06a6a1f04ae1c8b899cb6e">orcid-authority-to-item.py</a> script!</li>
+<li>It successfully mapped 2600 ORCID identifiers to items!</li>
+</ul>
+
   
 
   

docs/404.html

@@ -26,7 +26,7 @@
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="404 Page not found"/>
 <meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/categories/index.html

@@ -26,7 +26,7 @@
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Categories"/>
 <meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/categories/notes/index.html

@@ -26,7 +26,7 @@
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Notes"/>
 <meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/cgiar-library-migration/index.html

@@ -29,7 +29,7 @@
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="CGIAR Library Migration"/>
 <meta name="twitter:description" content="Notes on the migration of the CGIAR Library to CGSpace"/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/index.html

@@ -26,7 +26,7 @@
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="CGSpace Notes"/>
 <meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/page/2/index.html

@@ -26,7 +26,7 @@
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="CGSpace Notes"/>
 <meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/page/3/index.html

@@ -26,7 +26,7 @@
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="CGSpace Notes"/>
 <meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/post/index.html

@@ -26,7 +26,7 @@
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Posts"/>
 <meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/post/page/2/index.html

@@ -26,7 +26,7 @@
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Posts"/>
 <meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/post/page/3/index.html

@@ -26,7 +26,7 @@
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Posts"/>
 <meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/sitemap.xml

@@ -4,7 +4,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2018-02/</loc>
-    <lastmod>2018-02-26T16:41:28+02:00</lastmod>
+    <lastmod>2018-02-26T18:12:27+02:00</lastmod>
   </url>
   
   <url>
@@ -149,7 +149,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2018-02-26T16:41:28+02:00</lastmod>
+    <lastmod>2018-02-26T18:12:27+02:00</lastmod>
     <priority>0</priority>
   </url>
   
@@ -160,7 +160,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
-    <lastmod>2018-02-26T16:41:28+02:00</lastmod>
+    <lastmod>2018-02-26T18:12:27+02:00</lastmod>
     <priority>0</priority>
   </url>
   
@@ -172,13 +172,13 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/post/</loc>
-    <lastmod>2018-02-26T16:41:28+02:00</lastmod>
+    <lastmod>2018-02-26T18:12:27+02:00</lastmod>
     <priority>0</priority>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/tags/</loc>
-    <lastmod>2018-02-26T16:41:28+02:00</lastmod>
+    <lastmod>2018-02-26T18:12:27+02:00</lastmod>
     <priority>0</priority>
   </url>
   

docs/tags/index.html

@@ -26,7 +26,7 @@
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Tags"/>
 <meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/tags/notes/index.html

@@ -26,7 +26,7 @@
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Notes"/>
 <meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/tags/notes/page/2/index.html

@@ -26,7 +26,7 @@
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Notes"/>
 <meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />
 
 
     

docs/tags/notes/page/3/index.html

@@ -26,7 +26,7 @@
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Notes"/>
 <meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
-<meta name="generator" content="Hugo 0.36.1" />
+<meta name="generator" content="Hugo 0.37" />