Add notes for 2017-11-17

content/post/2017-11.md

@@ -647,3 +647,49 @@ dspace6=# CREATE EXTENSION pgcrypto;
 - Send Adam Hunt an invite to the DSpace Developers network on Yammer
 - He is the new head of communications at WLE, since Michael left
 - Merge changes to item view's wording of link metadata ([#348](https://github.com/ilri/DSpace/pull/348))
+
+## 2017-11-17
+
+- Uptime Robot said that CGSpace went down today and I see lots of `Timeout waiting for idle object` errors in the DSpace logs
+- I looked in PostgreSQL using `SELECT * FROM pg_stat_activity;` and saw that there were 73 active connections
+- After a few minutes the connecitons went down to 44 and CGSpace was kinda back up, it seems like Tsega restarted Tomcat
+- Looking at the REST and XMLUI log files, I don't see anything too crazy:
+
+```
+# cat /var/log/nginx/rest.log /var/log/nginx/rest.log.1 | grep "17/Nov/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+     13 66.249.66.223
+     14 207.46.13.36
+     17 207.46.13.137
+     22 207.46.13.23
+     23 66.249.66.221
+     92 66.249.66.219
+    187 104.196.152.243
+   1400 70.32.83.92
+   1503 50.116.102.77
+   6037 45.5.184.196
+# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep "17/Nov/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+    325 139.162.247.24
+    354 66.249.66.223
+    422 207.46.13.36
+    434 207.46.13.23
+    501 207.46.13.137
+    647 66.249.66.221
+    662 34.192.116.178
+    762 213.55.99.121
+   1867 104.196.152.243
+   2020 66.249.66.219
+```
+
+- I need to look into using JMX to analyze active sessions I think, rather than looking at log files
+- After adding appropriate [JMX listener options to Tomcat's JAVA_OPTS](https://geekflare.com/enable-jmx-tomcat-to-monitor-administer/) and restarting Tomcat, I can connect remotely using an SSH dynamic port forward (SOCKS) on port 7777 for example, and then start jconsole locally like:
+
+```
+$ jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=7777 service:jmx:rmi:///jndi/rmi://localhost:9000/jmxrmi -J-DsocksNonProxyHosts=
+```
+
+- Looking at the MBeans you can drill down in Catalina→Manager→<webapp>→localhost→Attributes and see active sessions, etc
+- I want to enable JMX listener on CGSpace but I need to do some more testing on DSpace Test and see if it causes any performance impact, for example
+- If I hit the server with some requests as a normal user I see the session counter increase, but if I specify a bot user agent then the sessions seem to be reused (meaning the Crawler Session Manager is working)
+- Here is the Jconsole screen after looping `http --print Hh https://dspacetest.cgiar.org/handle/10568/1` for a few minutes:
+
+![Jconsole sessions for XMLUI](/cgspace-notes/2017/11/jconsole-sessions.png)