diff --git a/content/post/2018-02.md b/content/post/2018-02.md index e027d1a08..985f8a45b 100644 --- a/content/post/2018-02.md +++ b/content/post/2018-02.md @@ -903,3 +903,45 @@ Nor Azwadi: 0000-0001-9634-1958 - I don't know why it would take so long, but this logic kinda makes sense - I think I should increase the `removeAbandonedTimeout` from 90 to something like 180 and continue observing - I also reduced the timeout for the API pool back to 60 because those interfaces are only used by bots + +## 2018-02-27 + +- Peter is still having problems with "Socket closed" on his submissions page +- I have disabled `removeAbandoned` for now because that's the only thing I changed in the last few weeks since he started having issues +- I think the real line of logic to follow here is why the submissions page is so slow for him (presumably because of loading all his submissions?) +- I need to see which SQL queries are run during that time +- And only a few hours after I disabled the `removeAbandoned` thing CGSpace went down and lo and behold, there were 264 connections, most of which were idle: + +``` +$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c + 5 dspaceApi + 279 dspaceWeb +$ psql -c 'select * from pg_stat_activity' | grep dspaceWeb | grep -c "idle in transaction" +218 +``` + +- So I'm re-enabling the `removeAbandoned` setting +- I grabbed a snapshot of the active connections in `pg_stat_activity` for all queries running longer than 2 minutes: + +``` +dspace=# \copy (SELECT now() - query_start as "runtime", application_name, usename, datname, waiting, state, query + FROM pg_stat_activity + WHERE now() - query_start > '2 minutes'::interval + ORDER BY runtime DESC) to /tmp/2018-02-27-postgresql.txt +COPY 263 +``` + +- 100 of these idle in transaction connections are the following query: + +``` +SELECT * FROM resourcepolicy WHERE resource_type_id= $1 AND resource_id= $2 AND action_id= $3 +``` + +- ... but according to the [pg_locks documentation](https://www.postgresql.org/docs/9.5/static/view-pg-locks.html) I should have done this to correlate the locks with the activity: + +``` +SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid; +``` + +- Finally finished the [orcid-authority-to-item.py](https://gist.github.com/alanorth/6d7489b50f06a6a1f04ae1c8b899cb6e) script! +- It successfully mapped 2600 ORCID identifiers to items! diff --git a/docs/2015-11/index.html b/docs/2015-11/index.html index c76b6031e..a79831da9 100644 --- a/docs/2015-11/index.html +++ b/docs/2015-11/index.html @@ -53,7 +53,7 @@ $ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspac "/> - + diff --git a/docs/2015-12/index.html b/docs/2015-12/index.html index 80779a192..520460d5e 100644 --- a/docs/2015-12/index.html +++ b/docs/2015-12/index.html @@ -55,7 +55,7 @@ Replace lzop with xz in log compression cron jobs on DSpace Test—it uses less "/> - + diff --git a/docs/2016-01/index.html b/docs/2016-01/index.html index dcc174ed0..ff6279d9a 100644 --- a/docs/2016-01/index.html +++ b/docs/2016-01/index.html @@ -45,7 +45,7 @@ Update GitHub wiki for documentation of maintenance tasks. "/> - + diff --git a/docs/2016-02/index.html b/docs/2016-02/index.html index 79f229108..d849ff9ff 100644 --- a/docs/2016-02/index.html +++ b/docs/2016-02/index.html @@ -59,7 +59,7 @@ Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE&r "/> - + diff --git a/docs/2016-03/index.html b/docs/2016-03/index.html index 59d2f70bd..4f764ac1f 100644 --- a/docs/2016-03/index.html +++ b/docs/2016-03/index.html @@ -45,7 +45,7 @@ Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Ja "/> - + diff --git a/docs/2016-04/index.html b/docs/2016-04/index.html index 730c1f26f..b98962244 100644 --- a/docs/2016-04/index.html +++ b/docs/2016-04/index.html @@ -49,7 +49,7 @@ Also, I noticed the checker log has some errors we should pay attention to: "/> - + diff --git a/docs/2016-05/index.html b/docs/2016-05/index.html index bac22c722..d0b02ff97 100644 --- a/docs/2016-05/index.html +++ b/docs/2016-05/index.html @@ -53,7 +53,7 @@ There are 3,000 IPs accessing the REST API in a 24-hour period! "/> - + diff --git a/docs/2016-06/index.html b/docs/2016-06/index.html index eed0c5f70..61ff906bb 100644 --- a/docs/2016-06/index.html +++ b/docs/2016-06/index.html @@ -51,7 +51,7 @@ Working on second phase of metadata migration, looks like this will work for mov "/> - + diff --git a/docs/2016-07/index.html b/docs/2016-07/index.html index 3e330d995..2b81496bf 100644 --- a/docs/2016-07/index.html +++ b/docs/2016-07/index.html @@ -67,7 +67,7 @@ In this case the select query was showing 95 results before the update "/> - + diff --git a/docs/2016-08/index.html b/docs/2016-08/index.html index bf022b5f0..17c42602d 100644 --- a/docs/2016-08/index.html +++ b/docs/2016-08/index.html @@ -61,7 +61,7 @@ $ git rebase -i dspace-5.5 "/> - + diff --git a/docs/2016-09/index.html b/docs/2016-09/index.html index af0dfded5..0bba981a1 100644 --- a/docs/2016-09/index.html +++ b/docs/2016-09/index.html @@ -53,7 +53,7 @@ $ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=or "/> - + diff --git a/docs/2016-10/index.html b/docs/2016-10/index.html index f0b20832c..c29e4b81c 100644 --- a/docs/2016-10/index.html +++ b/docs/2016-10/index.html @@ -61,7 +61,7 @@ I exported a random item’s metadata as CSV, deleted all columns except id "/> - + diff --git a/docs/2016-11/index.html b/docs/2016-11/index.html index 677a2b761..0b8f327e4 100644 --- a/docs/2016-11/index.html +++ b/docs/2016-11/index.html @@ -45,7 +45,7 @@ Add dc.type to the output options for Atmire’s Listings and Reports module "/> - + diff --git a/docs/2016-12/index.html b/docs/2016-12/index.html index faeca4989..fa582465e 100644 --- a/docs/2016-12/index.html +++ b/docs/2016-12/index.html @@ -69,7 +69,7 @@ Another worrying error from dspace.log is: "/> - + diff --git a/docs/2017-01/index.html b/docs/2017-01/index.html index f86b33ed9..693588552 100644 --- a/docs/2017-01/index.html +++ b/docs/2017-01/index.html @@ -45,7 +45,7 @@ I asked on the dspace-tech mailing list because it seems to be broken, and actua "/> - + diff --git a/docs/2017-02/index.html b/docs/2017-02/index.html index 192393422..fcb292937 100644 --- a/docs/2017-02/index.html +++ b/docs/2017-02/index.html @@ -73,7 +73,7 @@ Looks like we’ll be using cg.identifier.ccafsprojectpii as the field name "/> - + diff --git a/docs/2017-03/index.html b/docs/2017-03/index.html index 4ba82cb2e..02992d488 100644 --- a/docs/2017-03/index.html +++ b/docs/2017-03/index.html @@ -77,7 +77,7 @@ $ identify ~/Desktop/alc_contrastes_desafios.jpg "/> - + diff --git a/docs/2017-04/index.html b/docs/2017-04/index.html index a4878800c..2ea8453d3 100644 --- a/docs/2017-04/index.html +++ b/docs/2017-04/index.html @@ -63,7 +63,7 @@ $ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Th "/> - + diff --git a/docs/2017-05/index.html b/docs/2017-05/index.html index 980d9c364..5025774e1 100644 --- a/docs/2017-05/index.html +++ b/docs/2017-05/index.html @@ -29,7 +29,7 @@ - + diff --git a/docs/2017-06/index.html b/docs/2017-06/index.html index a1f8ae421..e3e6e8598 100644 --- a/docs/2017-06/index.html +++ b/docs/2017-06/index.html @@ -29,7 +29,7 @@ - + diff --git a/docs/2017-07/index.html b/docs/2017-07/index.html index 32863e200..e417f6bf3 100644 --- a/docs/2017-07/index.html +++ b/docs/2017-07/index.html @@ -57,7 +57,7 @@ We can use PostgreSQL’s extended output format (-x) plus sed to format the "/> - + diff --git a/docs/2017-08/index.html b/docs/2017-08/index.html index 345524666..f56711d29 100644 --- a/docs/2017-08/index.html +++ b/docs/2017-08/index.html @@ -77,7 +77,7 @@ Then I cleaned up the author authorities and HTML characters in OpenRefine and s "/> - + diff --git a/docs/2017-09/index.html b/docs/2017-09/index.html index 7460f24d4..644bf1a80 100644 --- a/docs/2017-09/index.html +++ b/docs/2017-09/index.html @@ -53,7 +53,7 @@ Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account "/> - + diff --git a/docs/2017-10/index.html b/docs/2017-10/index.html index f6d0a2cae..ab4e418ed 100644 --- a/docs/2017-10/index.html +++ b/docs/2017-10/index.html @@ -57,7 +57,7 @@ Add Katherine Lutz to the groups for content submission and edit steps of the CG "/> - + diff --git a/docs/2017-11/index.html b/docs/2017-11/index.html index 05d5c0438..50e163524 100644 --- a/docs/2017-11/index.html +++ b/docs/2017-11/index.html @@ -77,7 +77,7 @@ COPY 54701 "/> - + diff --git a/docs/2017-12/index.html b/docs/2017-12/index.html index 52154e512..711c11285 100644 --- a/docs/2017-12/index.html +++ b/docs/2017-12/index.html @@ -47,7 +47,7 @@ The list of connections to XMLUI and REST API for today: "/> - + diff --git a/docs/2018-01/index.html b/docs/2018-01/index.html index 4d510c565..7e63ccc21 100644 --- a/docs/2018-01/index.html +++ b/docs/2018-01/index.html @@ -185,7 +185,7 @@ Danny wrote to ask for help renewing the wildcard ilri.org certificate and I adv "/> - + diff --git a/docs/2018-02/index.html b/docs/2018-02/index.html index 45dec803f..6d58b6bef 100644 --- a/docs/2018-02/index.html +++ b/docs/2018-02/index.html @@ -23,7 +23,7 @@ I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu’s munin-pl - + @@ -47,7 +47,7 @@ I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu’s munin-pl "/> - + @@ -57,9 +57,9 @@ I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu’s munin-pl "@type": "BlogPosting", "headline": "February, 2018", "url": "https://alanorth.github.io/cgspace-notes/2018-02/", - "wordCount": "5609", + "wordCount": "5873", "datePublished": "2018-02-01T16:28:54+02:00", - "dateModified": "2018-02-26T16:41:28+02:00", + "dateModified": "2018-02-26T18:12:27+02:00", "author": { "@type": "Person", "name": "Alan Orth" @@ -1136,6 +1136,54 @@ Nor Azwadi: 0000-0001-9634-1958
  • I also reduced the timeout for the API pool back to 60 because those interfaces are only used by bots
    • +

    2018-02-27

    + + + +
    $ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+      5 dspaceApi
+    279 dspaceWeb
+$ psql -c 'select * from pg_stat_activity' | grep dspaceWeb | grep -c "idle in transaction"
+218
+
    + + + +
    dspace=# \copy (SELECT now() - query_start as "runtime", application_name, usename, datname, waiting, state, query
+  FROM  pg_stat_activity
+  WHERE now() - query_start > '2 minutes'::interval
+ ORDER BY runtime DESC) to /tmp/2018-02-27-postgresql.txt
+COPY 263
+
    + + + +
    SELECT * FROM resourcepolicy WHERE resource_type_id= $1 AND resource_id= $2 AND action_id= $3
+
    + + + +
    SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;
+
    + + + diff --git a/docs/404.html b/docs/404.html index f9d664875..c50c4c122 100644 --- a/docs/404.html +++ b/docs/404.html @@ -26,7 +26,7 @@ - + diff --git a/docs/categories/index.html b/docs/categories/index.html index 5a8617f7c..91e12d1c5 100644 --- a/docs/categories/index.html +++ b/docs/categories/index.html @@ -26,7 +26,7 @@ - + diff --git a/docs/categories/notes/index.html b/docs/categories/notes/index.html index 750242b3e..ba9a5d8ef 100644 --- a/docs/categories/notes/index.html +++ b/docs/categories/notes/index.html @@ -26,7 +26,7 @@ - + diff --git a/docs/cgiar-library-migration/index.html b/docs/cgiar-library-migration/index.html index a30097020..d3d6c9e5f 100644 --- a/docs/cgiar-library-migration/index.html +++ b/docs/cgiar-library-migration/index.html @@ -29,7 +29,7 @@ - + diff --git a/docs/index.html b/docs/index.html index 330cae28d..7e45cee23 100644 --- a/docs/index.html +++ b/docs/index.html @@ -26,7 +26,7 @@ - + diff --git a/docs/page/2/index.html b/docs/page/2/index.html index 4f3b3bbd6..35c2fb74a 100644 --- a/docs/page/2/index.html +++ b/docs/page/2/index.html @@ -26,7 +26,7 @@ - + diff --git a/docs/page/3/index.html b/docs/page/3/index.html index b3721f2fd..407668558 100644 --- a/docs/page/3/index.html +++ b/docs/page/3/index.html @@ -26,7 +26,7 @@ - + diff --git a/docs/post/index.html b/docs/post/index.html index 82957aee1..6806b41ee 100644 --- a/docs/post/index.html +++ b/docs/post/index.html @@ -26,7 +26,7 @@ - + diff --git a/docs/post/page/2/index.html b/docs/post/page/2/index.html index 4be82b93c..f6114adf4 100644 --- a/docs/post/page/2/index.html +++ b/docs/post/page/2/index.html @@ -26,7 +26,7 @@ - + diff --git a/docs/post/page/3/index.html b/docs/post/page/3/index.html index 8472f7e83..37ebc8b04 100644 --- a/docs/post/page/3/index.html +++ b/docs/post/page/3/index.html @@ -26,7 +26,7 @@ - + diff --git a/docs/sitemap.xml b/docs/sitemap.xml index 193869079..d17efb6e6 100644 --- a/docs/sitemap.xml +++ b/docs/sitemap.xml @@ -4,7 +4,7 @@ https://alanorth.github.io/cgspace-notes/2018-02/ - 2018-02-26T16:41:28+02:00 + 2018-02-26T18:12:27+02:00 @@ -149,7 +149,7 @@ https://alanorth.github.io/cgspace-notes/ - 2018-02-26T16:41:28+02:00 + 2018-02-26T18:12:27+02:00 0 @@ -160,7 +160,7 @@ https://alanorth.github.io/cgspace-notes/tags/notes/ - 2018-02-26T16:41:28+02:00 + 2018-02-26T18:12:27+02:00 0 @@ -172,13 +172,13 @@ https://alanorth.github.io/cgspace-notes/post/ - 2018-02-26T16:41:28+02:00 + 2018-02-26T18:12:27+02:00 0 https://alanorth.github.io/cgspace-notes/tags/ - 2018-02-26T16:41:28+02:00 + 2018-02-26T18:12:27+02:00 0 diff --git a/docs/tags/index.html b/docs/tags/index.html index 264e6be03..88442557a 100644 --- a/docs/tags/index.html +++ b/docs/tags/index.html @@ -26,7 +26,7 @@ - + diff --git a/docs/tags/notes/index.html b/docs/tags/notes/index.html index a4888b1ac..ac6c6994e 100644 --- a/docs/tags/notes/index.html +++ b/docs/tags/notes/index.html @@ -26,7 +26,7 @@ - + diff --git a/docs/tags/notes/page/2/index.html b/docs/tags/notes/page/2/index.html index c23d9c0ca..861a35185 100644 --- a/docs/tags/notes/page/2/index.html +++ b/docs/tags/notes/page/2/index.html @@ -26,7 +26,7 @@ - + diff --git a/docs/tags/notes/page/3/index.html b/docs/tags/notes/page/3/index.html index 2fbee55af..b419705e2 100644 --- a/docs/tags/notes/page/3/index.html +++ b/docs/tags/notes/page/3/index.html @@ -26,7 +26,7 @@ - +