Add notes for 2017-11-17

content/post/2017-11.md

@@ -647,3 +647,49 @@ dspace6=# CREATE EXTENSION pgcrypto;
 - Send Adam Hunt an invite to the DSpace Developers network on Yammer
 - He is the new head of communications at WLE, since Michael left
 - Merge changes to item view's wording of link metadata ([#348](https://github.com/ilri/DSpace/pull/348))
+
+## 2017-11-17
+
+- Uptime Robot said that CGSpace went down today and I see lots of `Timeout waiting for idle object` errors in the DSpace logs
+- I looked in PostgreSQL using `SELECT * FROM pg_stat_activity;` and saw that there were 73 active connections
+- After a few minutes the connecitons went down to 44 and CGSpace was kinda back up, it seems like Tsega restarted Tomcat
+- Looking at the REST and XMLUI log files, I don't see anything too crazy:
+
+```
+# cat /var/log/nginx/rest.log /var/log/nginx/rest.log.1 | grep "17/Nov/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+     13 66.249.66.223
+     14 207.46.13.36
+     17 207.46.13.137
+     22 207.46.13.23
+     23 66.249.66.221
+     92 66.249.66.219
+    187 104.196.152.243
+   1400 70.32.83.92
+   1503 50.116.102.77
+   6037 45.5.184.196
+# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep "17/Nov/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+    325 139.162.247.24
+    354 66.249.66.223
+    422 207.46.13.36
+    434 207.46.13.23
+    501 207.46.13.137
+    647 66.249.66.221
+    662 34.192.116.178
+    762 213.55.99.121
+   1867 104.196.152.243
+   2020 66.249.66.219
+```
+
+- I need to look into using JMX to analyze active sessions I think, rather than looking at log files
+- After adding appropriate [JMX listener options to Tomcat's JAVA_OPTS](https://geekflare.com/enable-jmx-tomcat-to-monitor-administer/) and restarting Tomcat, I can connect remotely using an SSH dynamic port forward (SOCKS) on port 7777 for example, and then start jconsole locally like:
+
+```
+$ jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=7777 service:jmx:rmi:///jndi/rmi://localhost:9000/jmxrmi -J-DsocksNonProxyHosts=
+```
+
+- Looking at the MBeans you can drill down in Catalina→Manager→<webapp>→localhost→Attributes and see active sessions, etc
+- I want to enable JMX listener on CGSpace but I need to do some more testing on DSpace Test and see if it causes any performance impact, for example
+- If I hit the server with some requests as a normal user I see the session counter increase, but if I specify a bot user agent then the sessions seem to be reused (meaning the Crawler Session Manager is working)
+- Here is the Jconsole screen after looping `http --print Hh https://dspacetest.cgiar.org/handle/10568/1` for a few minutes:
+
+![Jconsole sessions for XMLUI](/cgspace-notes/2017/11/jconsole-sessions.png)

public/2017-11/index.html

@@ -38,7 +38,7 @@ COPY 54701
 
 <meta property="article:published_time" content="2017-11-02T09:37:54&#43;02:00"/>
 
-<meta property="article:modified_time" content="2017-11-15T15:12:56&#43;02:00"/>
+<meta property="article:modified_time" content="2017-11-16T10:15:33&#43;02:00"/>
 
 
 
@@ -86,9 +86,9 @@ COPY 54701
   "@type": "BlogPosting",
   "headline": "November, 2017",
   "url": "https://alanorth.github.io/cgspace-notes/2017-11/",
-  "wordCount": "3743",
+  "wordCount": "4062",
   "datePublished": "2017-11-02T09:37:54&#43;02:00",
-  "dateModified": "2017-11-15T15:12:56&#43;02:00",
+  "dateModified": "2017-11-16T10:15:33&#43;02:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"
@@ -874,6 +874,56 @@ dspace6=# CREATE EXTENSION pgcrypto;
 <li>Merge changes to item view&rsquo;s wording of link metadata (<a href="https://github.com/ilri/DSpace/pull/348">#348</a>)</li>
 </ul>
 
+<h2 id="2017-11-17">2017-11-17</h2>
+
+<ul>
+<li>Uptime Robot said that CGSpace went down today and I see lots of <code>Timeout waiting for idle object</code> errors in the DSpace logs</li>
+<li>I looked in PostgreSQL using <code>SELECT * FROM pg_stat_activity;</code> and saw that there were 73 active connections</li>
+<li>After a few minutes the connecitons went down to 44 and CGSpace was kinda back up, it seems like Tsega restarted Tomcat</li>
+<li>Looking at the REST and XMLUI log files, I don&rsquo;t see anything too crazy:</li>
+</ul>
+
+<pre><code># cat /var/log/nginx/rest.log /var/log/nginx/rest.log.1 | grep &quot;17/Nov/2017&quot; | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+     13 66.249.66.223
+     14 207.46.13.36
+     17 207.46.13.137
+     22 207.46.13.23
+     23 66.249.66.221
+     92 66.249.66.219
+    187 104.196.152.243
+   1400 70.32.83.92
+   1503 50.116.102.77
+   6037 45.5.184.196
+# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep &quot;17/Nov/2017&quot; | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+    325 139.162.247.24
+    354 66.249.66.223
+    422 207.46.13.36
+    434 207.46.13.23
+    501 207.46.13.137
+    647 66.249.66.221
+    662 34.192.116.178
+    762 213.55.99.121
+   1867 104.196.152.243
+   2020 66.249.66.219
+</code></pre>
+
+<ul>
+<li>I need to look into using JMX to analyze active sessions I think, rather than looking at log files</li>
+<li>After adding appropriate <a href="https://geekflare.com/enable-jmx-tomcat-to-monitor-administer/">JMX listener options to Tomcat&rsquo;s JAVA_OPTS</a> and restarting Tomcat, I can connect remotely using an SSH dynamic port forward (SOCKS) on port 7777 for example, and then start jconsole locally like:</li>
+</ul>
+
+<pre><code>$ jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=7777 service:jmx:rmi:///jndi/rmi://localhost:9000/jmxrmi -J-DsocksNonProxyHosts=
+</code></pre>
+
+<ul>
+<li>Looking at the MBeans you can drill down in Catalina→Manager→<webapp>→localhost→Attributes and see active sessions, etc</li>
+<li>I want to enable JMX listener on CGSpace but I need to do some more testing on DSpace Test and see if it causes any performance impact, for example</li>
+<li>If I hit the server with some requests as a normal user I see the session counter increase, but if I specify a bot user agent then the sessions seem to be reused (meaning the Crawler Session Manager is working)</li>
+<li>Here is the Jconsole screen after looping <code>http --print Hh https://dspacetest.cgiar.org/handle/10568/1</code> for a few minutes:</li>
+</ul>
+
+<p><img src="/cgspace-notes/2017/11/jconsole-sessions.png" alt="Jconsole sessions for XMLUI" /></p>
+
   
 
   

public/sitemap.xml

@@ -4,7 +4,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2017-11/</loc>
-    <lastmod>2017-11-15T15:12:56+02:00</lastmod>
+    <lastmod>2017-11-16T10:15:33+02:00</lastmod>
   </url>
   
   <url>
@@ -134,7 +134,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2017-11-15T15:12:56+02:00</lastmod>
+    <lastmod>2017-11-16T10:15:33+02:00</lastmod>
     <priority>0</priority>
   </url>
   
@@ -145,7 +145,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
-    <lastmod>2017-11-15T15:12:56+02:00</lastmod>
+    <lastmod>2017-11-16T10:15:33+02:00</lastmod>
     <priority>0</priority>
   </url>
   
@@ -157,13 +157,13 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/post/</loc>
-    <lastmod>2017-11-15T15:12:56+02:00</lastmod>
+    <lastmod>2017-11-16T10:15:33+02:00</lastmod>
     <priority>0</priority>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/tags/</loc>
-    <lastmod>2017-11-15T15:12:56+02:00</lastmod>
+    <lastmod>2017-11-16T10:15:33+02:00</lastmod>
     <priority>0</priority>
   </url>