Add notes for 2021-09-28

content/posts/2021-09.md

@@ -309,4 +309,57 @@ UPDATE 49
 
 - Add CGIAR Action Area (cg.subject.actionArea) to CGSpace as Peter had asked me a few days ago
 
+## 2021-09-28
+
+- Francesca from the Alliance asked for help moving a bunch of reports from one collections to another on CGSpace
+  - She is having problems with the "move" dialog taking minutes for each item
+  - I exported the collection and sent her a copy with just the few fields she would need in order to mark the ones that need to move, then I can do the rest:
+
+```console
+$ csvcut -c 'id,collection,dc.title[en_US]' ~/Downloads/10568-106990.csv > /tmp/2021-09-28-alliance-reports.csv
+```
+
+- She sent it back fairly quickly with a new column marked "Move" so I extracted those items that matched and set them to the new owning collection:
+
+```console
+$ csvgrep -c Move -m 'Yes' ~/Downloads/2021_28_09_alliance_reports_csv.csv | csvcut -c 1,2 | sed 's_10568/106990_10568/111506_' > /tmp/alliance-move.csv
+```
+
+- Maria from the Alliance emailed us to say that approving submissions was slow on CGSpace
+  - I looked at the PostgreSQL activity and it seems low:
+
+```console
+postgres@linode18:~$ psql -c 'SELECT * FROM pg_stat_activity' | wc -l
+59
+```
+
+- Locks look high though:
+
+```console
+postgres@linode18:~$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | sort | uniq -c | wc -l
+1154
+```
+
+- Indeed it seems something started causing locks to increase yesterday:
+
+![PostgreSQL locks week](/cgspace-notes/2021/09/postgres_locks_ALL-week.png)
+
+- And query length increasing since yesterday:
+
+![PostgreSQL query length week](/cgspace-notes/2021/09/postgres_querylength_ALL-week.png)
+
+- The number of DSpace sessions is normal, hovering around 1,000...
+- Looking closer at the PostgreSQL activity log, I see the locks are all held by the `dspaceCli` user... which seem weird:
+
+```console
+postgres@linode18:~$ psql -c "SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid WHERE application_name='dspaceCli';" | wc -l
+1096
+```
+
+- Now I'm wondering why there are no connections from `dspaceApi` or `dspaceWeb`. Could it be that our Tomcat JDBC pooling via JNDI isn't working?
+  - I see the same thing on DSpace Test hmmmm
+  - The configuration in `server.xml` is correct, but it could be that when I changed to using the updated JDBC driver from `pom.xml` instead of dropping it in the Tomcat lib directory that something broke...
+  - I downloaded the latest JDBC jar and put it in Tomcat's lib directory on DSpace Test and after restarting Tomcat I can see connections from `dspaceWeb` and `dspaceApi` again
+  - I will do the same on CGSpace and then revert the JDBC change in Ansible and DSpace `pom.xml`
+
 <!-- vim: set sw=2 ts=2: -->