diff --git a/content/post/2018-01.md b/content/post/2018-01.md index 564400069..7fae0a865 100644 --- a/content/post/2018-01.md +++ b/content/post/2018-01.md @@ -1239,3 +1239,52 @@ dspace.log.2018-01-29:0 - `processorCache` from 200 (default) to 400, [recommended to be the same as `maxThreads`](https://tomcat.apache.org/tomcat-7.0-doc/config/http.html) - `minSpareThreads` from 10 (default) to 20 - `acceptorThreadCount` from 1 (default) to 2, [recommended to be 2 on multi-CPU systems](https://tomcat.apache.org/tomcat-7.0-doc/config/http.html) +- Looks like I only enabled the new thread stuff on the connector used internally by Solr, so I probably need to match that by increasing them on the other connector that nginx proxies to +- Jesus Christ I need to fucking fix the Munin monitoring so that I can tell how many fucking threads I have running +- Wow, so apparently you need to specify which connector to check if you want any of the Munin Tomcat plugins besides "tomcat_jvm" to work (the connector name can be seen in the Catalina logs) +- I modified _/etc/munin/plugin-conf.d/tomcat_ to add the connector (with surrounding quotes!) and now the other plugins work (obviously the credentials are incorrect): + +``` +[tomcat_*] + env.host 127.0.0.1 + env.port 8081 + env.connector "http-bio-127.0.0.1-8443" + env.user munin + env.password munin +``` + +- For example, I can see the threads: + +``` +# munin-run tomcat_threads +busy.value 0 +idle.value 20 +max.value 400 +``` + +- Apparently you can't monitor more than one connector, so I guess the most important to monitor would be the one that nginx is sending stuff to +- So for now I think I'll just monitor these and skip trying to configure the jmx plugins +- Although following the logic of _/usr/share/munin/plugins/jmx_tomcat_dbpools_ could be useful for getting the active Tomcat sessions +- From debugging the `jmx_tomcat_db_pools` script from the `munin-plugins-java` package, I see that this is how you call arbitrary mbeans: + +``` +# port=5400 ip="127.0.0.1" /usr/bin/java -cp /usr/share/munin/munin-jmx-plugins.jar org.munin.plugin.jmx.Beans Catalina:type=DataSource,class=javax.sql.DataSource,name=* maxActive +Catalina:type=DataSource,class=javax.sql.DataSource,name="jdbc/dspace" maxActive 300 +``` + +- More notes here: https://github.com/munin-monitoring/contrib/tree/master/plugins/jmx +- Looking at the Munin graphs, I that the load is 200% every morning from 03:00 to almost 08:00 +- Tomcat's catalina.out log file is full of spam from this thing too, with lines like this + +``` +[===================> ]38% time remaining: 5 hour(s) 21 minute(s) 47 seconds. timestamp: 2018-01-29 06:25:16 +``` + +- There are millions of these status lines, for example in just this one log file: + +``` +# zgrep -c "time remaining" /var/log/tomcat7/catalina.out.1.gz +1084741 +``` + +- I filed a ticket with Atmire: https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=566 diff --git a/public/2018-01/index.html b/public/2018-01/index.html index a1d46b901..a41cc2347 100644 --- a/public/2018-01/index.html +++ b/public/2018-01/index.html @@ -92,7 +92,7 @@ Danny wrote to ask for help renewing the wildcard ilri.org certificate and I adv - + @@ -194,9 +194,9 @@ Danny wrote to ask for help renewing the wildcard ilri.org certificate and I adv "@type": "BlogPosting", "headline": "January, 2018", "url": "https://alanorth.github.io/cgspace-notes/2018-01/", - "wordCount": "7230", + "wordCount": "7537", "datePublished": "2018-01-02T08:35:54-08:00", - "dateModified": "2018-01-29T09:46:48+02:00", + "dateModified": "2018-01-29T09:47:55+02:00", "author": { "@type": "Person", "name": "Alan Orth" @@ -1632,6 +1632,60 @@ dspace.log.2018-01-29:0
minSpareThreads
from 10 (default) to 20acceptorThreadCount
from 1 (default) to 2, recommended to be 2 on multi-CPU systems[tomcat_*]
+ env.host 127.0.0.1
+ env.port 8081
+ env.connector "http-bio-127.0.0.1-8443"
+ env.user munin
+ env.password munin
+
+
+# munin-run tomcat_threads
+busy.value 0
+idle.value 20
+max.value 400
+
+
+jmx_tomcat_db_pools
script from the munin-plugins-java
package, I see that this is how you call arbitrary mbeans:# port=5400 ip="127.0.0.1" /usr/bin/java -cp /usr/share/munin/munin-jmx-plugins.jar org.munin.plugin.jmx.Beans Catalina:type=DataSource,class=javax.sql.DataSource,name=* maxActive
+Catalina:type=DataSource,class=javax.sql.DataSource,name="jdbc/dspace" maxActive 300
+
+
+[===================> ]38% time remaining: 5 hour(s) 21 minute(s) 47 seconds. timestamp: 2018-01-29 06:25:16
+
+
+# zgrep -c "time remaining" /var/log/tomcat7/catalina.out.1.gz
+1084741
+
+
+