diff --git a/content/2016-09.md b/content/2016-09.md index f50343885..a6f867c0b 100644 --- a/content/2016-09.md +++ b/content/2016-09.md @@ -191,3 +191,36 @@ org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error ``` org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114) ``` +- I enabled logging of requests to `/rest` again + +## 2016-09-14 + +- CGSpace crashed again, errors from `catalina.out`: + +``` +org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object + at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114) +``` + +- I restarted Tomcat and it was ok again +- CGSpace crashed a few hours later, errors from `catalina.out`: + +``` +Exception in thread "http-bio-127.0.0.1-8081-exec-25" java.lang.OutOfMemoryError: Java heap space + at java.lang.StringCoding.decode(StringCoding.java:215) +``` + +- We haven't seen that in quite a while... +- Indeed, in a month of logs it only occurs 15 times: + +``` +# grep -rsI "OutOfMemoryError" /var/log/tomcat7/catalina.* | wc -l +15 +``` + +- I also see a bunch of errors from dspace.log: + +``` +2016-09-14 12:23:07,981 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error - +org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object +``` diff --git a/public/2016-09/index.html b/public/2016-09/index.html index 75a071ff7..fbda90187 100644 --- a/public/2016-09/index.html +++ b/public/2016-09/index.html @@ -297,6 +297,46 @@ org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error
org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
+
+
+/rest
againcatalina.out
:org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
+ at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
+
+
+catalina.out
:Exception in thread "http-bio-127.0.0.1-8081-exec-25" java.lang.OutOfMemoryError: Java heap space
+ at java.lang.StringCoding.decode(StringCoding.java:215)
+
+
+# grep -rsI "OutOfMemoryError" /var/log/tomcat7/catalina.* | wc -l
+15
+
+
+2016-09-14 12:23:07,981 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error -
+org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
diff --git a/public/index.xml b/public/index.xml
index 35e4fbd10..cf9530d45 100644
--- a/public/index.xml
+++ b/public/index.xml
@@ -236,6 +236,46 @@ org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error
<pre><code>org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
</code></pre>
+
+<ul>
+<li>I enabled logging of requests to <code>/rest</code> again</li>
+</ul>
+
+<h2 id="2016-09-14">2016-09-14</h2>
+
+<ul>
+<li>CGSpace crashed again, errors from <code>catalina.out</code>:</li>
+</ul>
+
+<pre><code>org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
+ at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
+</code></pre>
+
+<ul>
+<li>I restarted Tomcat and it was ok again</li>
+<li>CGSpace crashed a few hours later, errors from <code>catalina.out</code>:</li>
+</ul>
+
+<pre><code>Exception in thread "http-bio-127.0.0.1-8081-exec-25" java.lang.OutOfMemoryError: Java heap space
+ at java.lang.StringCoding.decode(StringCoding.java:215)
+</code></pre>
+
+<ul>
+<li>We haven’t seen that in quite a while…</li>
+<li>Indeed, in a month of logs it only occurs 15 times:</li>
+</ul>
+
+<pre><code># grep -rsI "OutOfMemoryError" /var/log/tomcat7/catalina.* | wc -l
+15
+</code></pre>
+
+<ul>
+<li>I also see a bunch of errors from dspace.log:</li>
+</ul>
+
+<pre><code>2016-09-14 12:23:07,981 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error -
+org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
+</code></pre>
diff --git a/public/tags/notes/index.xml b/public/tags/notes/index.xml
index f8f9d170f..b7f5392e7 100644
--- a/public/tags/notes/index.xml
+++ b/public/tags/notes/index.xml
@@ -236,6 +236,46 @@ org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error
<pre><code>org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
</code></pre>
+
+<ul>
+<li>I enabled logging of requests to <code>/rest</code> again</li>
+</ul>
+
+<h2 id="2016-09-14">2016-09-14</h2>
+
+<ul>
+<li>CGSpace crashed again, errors from <code>catalina.out</code>:</li>
+</ul>
+
+<pre><code>org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
+ at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
+</code></pre>
+
+<ul>
+<li>I restarted Tomcat and it was ok again</li>
+<li>CGSpace crashed a few hours later, errors from <code>catalina.out</code>:</li>
+</ul>
+
+<pre><code>Exception in thread "http-bio-127.0.0.1-8081-exec-25" java.lang.OutOfMemoryError: Java heap space
+ at java.lang.StringCoding.decode(StringCoding.java:215)
+</code></pre>
+
+<ul>
+<li>We haven’t seen that in quite a while…</li>
+<li>Indeed, in a month of logs it only occurs 15 times:</li>
+</ul>
+
+<pre><code># grep -rsI "OutOfMemoryError" /var/log/tomcat7/catalina.* | wc -l
+15
+</code></pre>
+
+<ul>
+<li>I also see a bunch of errors from dspace.log:</li>
+</ul>
+
+<pre><code>2016-09-14 12:23:07,981 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error -
+org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
+</code></pre>