Add notes for 2018-02-27

content/post/2018-02.md

@@ -903,3 +903,45 @@ Nor Azwadi: 0000-0001-9634-1958
 - I don't know why it would take so long, but this logic kinda makes sense
 - I think I should increase the `removeAbandonedTimeout` from 90 to something like 180 and continue observing
 - I also reduced the timeout for the API pool back to 60 because those interfaces are only used by bots
+
+## 2018-02-27
+
+- Peter is still having problems with "Socket closed" on his submissions page
+- I have disabled `removeAbandoned` for now because that's the only thing I changed in the last few weeks since he started having issues
+- I think the real line of logic to follow here is why the submissions page is so slow for him (presumably because of loading all his submissions?)
+- I need to see which SQL queries are run during that time
+- And only a few hours after I disabled the `removeAbandoned` thing CGSpace went down and lo and behold, there were 264 connections, most of which were idle:
+
+```
+$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+      5 dspaceApi
+    279 dspaceWeb
+$ psql -c 'select * from pg_stat_activity' | grep dspaceWeb | grep -c "idle in transaction"
+218
+```
+
+- So I'm re-enabling the `removeAbandoned` setting
+- I grabbed a snapshot of the active connections in `pg_stat_activity` for all queries running longer than 2 minutes:
+
+```
+dspace=# \copy (SELECT now() - query_start as "runtime", application_name, usename, datname, waiting, state, query
+  FROM  pg_stat_activity
+  WHERE now() - query_start > '2 minutes'::interval
+ ORDER BY runtime DESC) to /tmp/2018-02-27-postgresql.txt
+COPY 263
+```
+
+- 100 of these idle in transaction connections are the following query:
+
+```
+SELECT * FROM resourcepolicy WHERE resource_type_id= $1 AND resource_id= $2 AND action_id= $3
+```
+
+- ... but according to the [pg_locks documentation](https://www.postgresql.org/docs/9.5/static/view-pg-locks.html) I should have done this to correlate the locks with the activity:
+
+```
+SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;
+```
+
+- Finally finished the [orcid-authority-to-item.py](https://gist.github.com/alanorth/6d7489b50f06a6a1f04ae1c8b899cb6e) script!
+- It successfully mapped 2600 ORCID identifiers to items!