From d1acea0709b79ecef0b957d4f83eab8f6d1e734f Mon Sep 17 00:00:00 2001 From: Alan Orth Date: Wed, 14 Sep 2016 22:07:16 +0300 Subject: [PATCH] Update notes for 2016-09-16 --- content/2016-09.md | 59 ++++++++++++++++++ public/2016-09/index.html | 67 +++++++++++++++++++++ public/images/2016/09/tomcat_jvm-day.png | Bin 0 -> 39996 bytes public/images/2016/09/tomcat_jvm-month.png | Bin 0 -> 21856 bytes public/images/2016/09/tomcat_jvm-week.png | Bin 0 -> 26773 bytes public/index.xml | 67 +++++++++++++++++++++ public/tags/notes/index.xml | 67 +++++++++++++++++++++ static/images/2016/09/tomcat_jvm-day.png | Bin 0 -> 39996 bytes static/images/2016/09/tomcat_jvm-month.png | Bin 0 -> 21856 bytes static/images/2016/09/tomcat_jvm-week.png | Bin 0 -> 26773 bytes 10 files changed, 260 insertions(+) create mode 100644 public/images/2016/09/tomcat_jvm-day.png create mode 100644 public/images/2016/09/tomcat_jvm-month.png create mode 100644 public/images/2016/09/tomcat_jvm-week.png create mode 100644 static/images/2016/09/tomcat_jvm-day.png create mode 100644 static/images/2016/09/tomcat_jvm-month.png create mode 100644 static/images/2016/09/tomcat_jvm-week.png diff --git a/content/2016-09.md b/content/2016-09.md index 541425461..928baf9a3 100644 --- a/content/2016-09.md +++ b/content/2016-09.md @@ -280,3 +280,62 @@ org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error dspacetest=# \copy (select text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=211 group by text_value order by count desc) to /tmp/affiliations.csv with csv; ``` + +- Looking into the Catalina logs again around the time of the first crash, I see: + +``` +Wed Sep 14 09:47:27 UTC 2016 | Query:id: 78581 AND type:2 +Wed Sep 14 09:47:28 UTC 2016 | Updating : 6/6 docs. +Commit +Commit done +dn:CN=Haman\, Magdalena (CIAT-CCAFS),OU=Standard,OU=Users,OU=HQ,OU=CIATHUB,dc=cgiarad,dc=org +Exception in thread "http-bio-127.0.0.1-8081-exec-193" java.lang.OutOfMemoryError: Java heap space +``` + +- And after that I see a bunch of "pool error Timeout waiting for idle object" +- Later, near the time of the next crash I see: + +``` +dn:CN=Haman\, Magdalena (CIAT-CCAFS),OU=Standard,OU=Users,OU=HQ,OU=CIATHUB,dc=cgiarad,dc=org +Wed Sep 14 11:29:55 UTC 2016 | Query:id: 79078 AND type:2 +Wed Sep 14 11:30:20 UTC 2016 | Updating : 6/6 docs. +Commit +Commit done +Sep 14, 2016 11:32:22 AM com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator buildModelAndSchemas +SEVERE: Failed to generate the schema for the JAX-B elements +com.sun.xml.bind.v2.runtime.IllegalAnnotationsException: 2 counts of IllegalAnnotationExceptions +java.util.Map is an interface, and JAXB can't handle interfaces. + this problem is related to the following location: + at java.util.Map + at public java.util.Map com.atmire.dspace.rest.common.Statlet.getRender() + at com.atmire.dspace.rest.common.Statlet +java.util.Map does not have a no-arg default constructor. + this problem is related to the following location: + at java.util.Map + at public java.util.Map com.atmire.dspace.rest.common.Statlet.getRender() + at com.atmire.dspace.rest.common.Statlet +``` + +- Then 20 minutes later another outOfMemoryError: + +``` +Exception in thread "http-bio-127.0.0.1-8081-exec-25" java.lang.OutOfMemoryError: Java heap space + at java.lang.StringCoding.decode(StringCoding.java:215) +``` + +- Perhaps these particular issues *are* memory issues, the munin graphs definitely show some weird purging/allocating behavior starting this week + +![Tomcat JVM usage day](../images/2016/09/tomcat_jvm-day.png) +![Tomcat JVM usage week](../images/2016/09/tomcat_jvm-week.png) +![Tomcat JVM usage month](../images/2016/09/tomcat_jvm-month.png) + +- And really, we did reduce the memory of CGSpace in late 2015, so maybe we should just increase it again, now that our usage is higher and we are having memory errors in the logs +- Oh great, the configuration on the actual server is different than in configuration management! +- Seems we added a bunch of settings to the `/etc/default/tomcat7` in December, 2015 and never updated our ansible repository: + +``` +JAVA_OPTS="-Djava.awt.headless=true -Xms3584m -Xmx3584m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -Dfile.encoding=UTF-8 -XX:-UseGCOverheadLimit -XX:MaxGCPauseMillis=250 -XX:GCTimeRatio=9 -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled -XX:G1HeapRegionSize=8m -XX:InitiatingHeapOccupancyPercent=75 -XX:+UseLargePages -XX:+AggressiveOpts" +``` + +- So I'm going to bump the heap +512m and remove all the other experimental shit (and update ansible!) +- Increased JVM heap to 4096m on CGSpace (linode01) diff --git a/public/2016-09/index.html b/public/2016-09/index.html index 062ebacff..7c9e66b6e 100644 --- a/public/2016-09/index.html +++ b/public/2016-09/index.html @@ -401,6 +401,73 @@ org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error to /tmp/affiliations.csv with csv; + + +
Wed Sep 14 09:47:27 UTC 2016 | Query:id: 78581 AND type:2
+Wed Sep 14 09:47:28 UTC 2016 | Updating : 6/6 docs.
+Commit
+Commit done
+dn:CN=Haman\, Magdalena  (CIAT-CCAFS),OU=Standard,OU=Users,OU=HQ,OU=CIATHUB,dc=cgiarad,dc=org
+Exception in thread "http-bio-127.0.0.1-8081-exec-193" java.lang.OutOfMemoryError: Java heap space
+
+ + + +
dn:CN=Haman\, Magdalena  (CIAT-CCAFS),OU=Standard,OU=Users,OU=HQ,OU=CIATHUB,dc=cgiarad,dc=org
+Wed Sep 14 11:29:55 UTC 2016 | Query:id: 79078 AND type:2
+Wed Sep 14 11:30:20 UTC 2016 | Updating : 6/6 docs.
+Commit
+Commit done
+Sep 14, 2016 11:32:22 AM com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator buildModelAndSchemas
+SEVERE: Failed to generate the schema for the JAX-B elements
+com.sun.xml.bind.v2.runtime.IllegalAnnotationsException: 2 counts of IllegalAnnotationExceptions
+java.util.Map is an interface, and JAXB can't handle interfaces.
+        this problem is related to the following location:
+                at java.util.Map
+                at public java.util.Map com.atmire.dspace.rest.common.Statlet.getRender()
+                at com.atmire.dspace.rest.common.Statlet
+java.util.Map does not have a no-arg default constructor.
+        this problem is related to the following location:
+                at java.util.Map
+                at public java.util.Map com.atmire.dspace.rest.common.Statlet.getRender()
+                at com.atmire.dspace.rest.common.Statlet
+
+ + + +
Exception in thread "http-bio-127.0.0.1-8081-exec-25" java.lang.OutOfMemoryError: Java heap space
+        at java.lang.StringCoding.decode(StringCoding.java:215)
+
+ + + +

Tomcat JVM usage day +Tomcat JVM usage week +Tomcat JVM usage month

+ + + +
JAVA_OPTS="-Djava.awt.headless=true -Xms3584m -Xmx3584m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -Dfile.encoding=UTF-8 -XX:-UseGCOverheadLimit -XX:MaxGCPauseMillis=250 -XX:GCTimeRatio=9 -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled -XX:G1HeapRegionSize=8m -XX:InitiatingHeapOccupancyPercent=75 -XX:+UseLargePages -XX:+AggressiveOpts"
+
+ + +