diff --git a/content/post/2018-01.md b/content/post/2018-01.md index 564400069..7fae0a865 100644 --- a/content/post/2018-01.md +++ b/content/post/2018-01.md @@ -1239,3 +1239,52 @@ dspace.log.2018-01-29:0 - `processorCache` from 200 (default) to 400, [recommended to be the same as `maxThreads`](https://tomcat.apache.org/tomcat-7.0-doc/config/http.html) - `minSpareThreads` from 10 (default) to 20 - `acceptorThreadCount` from 1 (default) to 2, [recommended to be 2 on multi-CPU systems](https://tomcat.apache.org/tomcat-7.0-doc/config/http.html) +- Looks like I only enabled the new thread stuff on the connector used internally by Solr, so I probably need to match that by increasing them on the other connector that nginx proxies to +- Jesus Christ I need to fucking fix the Munin monitoring so that I can tell how many fucking threads I have running +- Wow, so apparently you need to specify which connector to check if you want any of the Munin Tomcat plugins besides "tomcat_jvm" to work (the connector name can be seen in the Catalina logs) +- I modified _/etc/munin/plugin-conf.d/tomcat_ to add the connector (with surrounding quotes!) and now the other plugins work (obviously the credentials are incorrect): + +``` +[tomcat_*] + env.host 127.0.0.1 + env.port 8081 + env.connector "http-bio-127.0.0.1-8443" + env.user munin + env.password munin +``` + +- For example, I can see the threads: + +``` +# munin-run tomcat_threads +busy.value 0 +idle.value 20 +max.value 400 +``` + +- Apparently you can't monitor more than one connector, so I guess the most important to monitor would be the one that nginx is sending stuff to +- So for now I think I'll just monitor these and skip trying to configure the jmx plugins +- Although following the logic of _/usr/share/munin/plugins/jmx_tomcat_dbpools_ could be useful for getting the active Tomcat sessions +- From debugging the `jmx_tomcat_db_pools` script from the `munin-plugins-java` package, I see that this is how you call arbitrary mbeans: + +``` +# port=5400 ip="127.0.0.1" /usr/bin/java -cp /usr/share/munin/munin-jmx-plugins.jar org.munin.plugin.jmx.Beans Catalina:type=DataSource,class=javax.sql.DataSource,name=* maxActive +Catalina:type=DataSource,class=javax.sql.DataSource,name="jdbc/dspace" maxActive 300 +``` + +- More notes here: https://github.com/munin-monitoring/contrib/tree/master/plugins/jmx +- Looking at the Munin graphs, I that the load is 200% every morning from 03:00 to almost 08:00 +- Tomcat's catalina.out log file is full of spam from this thing too, with lines like this + +``` +[===================> ]38% time remaining: 5 hour(s) 21 minute(s) 47 seconds. timestamp: 2018-01-29 06:25:16 +``` + +- There are millions of these status lines, for example in just this one log file: + +``` +# zgrep -c "time remaining" /var/log/tomcat7/catalina.out.1.gz +1084741 +``` + +- I filed a ticket with Atmire: https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=566 diff --git a/public/2018-01/index.html b/public/2018-01/index.html index a1d46b901..a41cc2347 100644 --- a/public/2018-01/index.html +++ b/public/2018-01/index.html @@ -92,7 +92,7 @@ Danny wrote to ask for help renewing the wildcard ilri.org certificate and I adv - + @@ -194,9 +194,9 @@ Danny wrote to ask for help renewing the wildcard ilri.org certificate and I adv "@type": "BlogPosting", "headline": "January, 2018", "url": "https://alanorth.github.io/cgspace-notes/2018-01/", - "wordCount": "7230", + "wordCount": "7537", "datePublished": "2018-01-02T08:35:54-08:00", - "dateModified": "2018-01-29T09:46:48+02:00", + "dateModified": "2018-01-29T09:47:55+02:00", "author": { "@type": "Person", "name": "Alan Orth" @@ -1632,6 +1632,60 @@ dspace.log.2018-01-29:0
  • minSpareThreads from 10 (default) to 20
  • acceptorThreadCount from 1 (default) to 2, recommended to be 2 on multi-CPU systems
  • +
  • Looks like I only enabled the new thread stuff on the connector used internally by Solr, so I probably need to match that by increasing them on the other connector that nginx proxies to
  • +
  • Jesus Christ I need to fucking fix the Munin monitoring so that I can tell how many fucking threads I have running
  • +
  • Wow, so apparently you need to specify which connector to check if you want any of the Munin Tomcat plugins besides “tomcat_jvm” to work (the connector name can be seen in the Catalina logs)
  • +
  • I modified /etc/munin/plugin-conf.d/tomcat to add the connector (with surrounding quotes!) and now the other plugins work (obviously the credentials are incorrect):
  • + + +
    [tomcat_*]
    +    env.host 127.0.0.1
    +    env.port 8081
    +    env.connector "http-bio-127.0.0.1-8443"
    +    env.user munin
    +    env.password munin
    +
    + + + +
    # munin-run tomcat_threads
    +busy.value 0
    +idle.value 20
    +max.value 400
    +
    + + + +
    # port=5400 ip="127.0.0.1" /usr/bin/java -cp /usr/share/munin/munin-jmx-plugins.jar org.munin.plugin.jmx.Beans Catalina:type=DataSource,class=javax.sql.DataSource,name=* maxActive
    +Catalina:type=DataSource,class=javax.sql.DataSource,name="jdbc/dspace"  maxActive       300
    +
    + + + +
    [===================>                               ]38% time remaining: 5 hour(s) 21 minute(s) 47 seconds. timestamp: 2018-01-29 06:25:16
    +
    + + + +
    # zgrep -c "time remaining" /var/log/tomcat7/catalina.out.1.gz
    +1084741
    +
    + + diff --git a/public/sitemap.xml b/public/sitemap.xml index 07a3d1410..ef9d1ff7f 100644 --- a/public/sitemap.xml +++ b/public/sitemap.xml @@ -4,7 +4,7 @@ https://alanorth.github.io/cgspace-notes/2018-01/ - 2018-01-29T09:46:48+02:00 + 2018-01-29T09:47:55+02:00 @@ -144,7 +144,7 @@ https://alanorth.github.io/cgspace-notes/ - 2018-01-29T09:46:48+02:00 + 2018-01-29T09:47:55+02:00 0 @@ -155,7 +155,7 @@ https://alanorth.github.io/cgspace-notes/tags/notes/ - 2018-01-29T09:46:48+02:00 + 2018-01-29T09:47:55+02:00 0 @@ -167,13 +167,13 @@ https://alanorth.github.io/cgspace-notes/post/ - 2018-01-29T09:46:48+02:00 + 2018-01-29T09:47:55+02:00 0 https://alanorth.github.io/cgspace-notes/tags/ - 2018-01-29T09:46:48+02:00 + 2018-01-29T09:47:55+02:00 0