Update notes for 2018-03-21

content/posts/2018-03.md

@@ -370,3 +370,48 @@ COPY 56156
 ```
 
 - Afterwards we'll want to do some batch tagging of ORCID identifiers to these names
+
+## 2018-03-20
+
+- CGSpace crashed again this afternoon, I'm not sure of the cause but there are a lot of SQL errors in the DSpace log:
+
+```
+2018-03-21 15:11:08,166 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL QueryTable Error - 
+java.sql.SQLException: Connection has already been closed.
+```
+
+- I have no idea why so many connections were abandoned this afternoon:
+
+```
+# grep 'Mar 21, 2018' /var/log/tomcat7/catalina.out | grep -c 'org.apache.tomcat.jdbc.pool.ConnectionPool abandon'
+268
+```
+
+- DSpace Test crashed again due to Java heap space, this is from the DSpace log:
+
+```
+2018-03-21 15:18:48,149 ERROR org.dspace.app.xmlui.cocoon.DSpaceCocoonServletFilter @ Serious Error Occurred Processing Request!
+org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space
+```
+
+- And this is from the Tomcat Catalina log:
+
+```
+Mar 21, 2018 11:20:00 AM org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor run
+SEVERE: Unexpected death of background thread ContainerBackgroundProcessor[StandardEngine[Catalina]]
+java.lang.OutOfMemoryError: Java heap space
+```
+
+- But there are tons of heap space errors on DSpace Test actually:
+
+```
+# grep -c 'java.lang.OutOfMemoryError: Java heap space' /var/log/tomcat7/catalina.out
+319
+```
+
+- I guess we need to give it more RAM because it now has CGSpace's large Solr core
+- I will increase the memory from 3072m to 4096m
+- Update [Ansible playbooks](https://github.com/ilri/rmg-ansible-public) to use [PostgreSQL JBDC driver](https://jdbc.postgresql.org/) 42.2.2
+- Deploy the new JDBC driver on DSpace Test
+- I'm also curious to see how long the `dspace index-discovery -b` takes on DSpace Test where the DSpace installation directory is on one of Linode's new block storage volumes
+- I should also check the raw read speed with `hdparm -tT /dev/sdc`

docs/2018-03/index.html

@@ -20,7 +20,7 @@ Export a CSV of the IITA community metadata for Martin Mueller
 
 <meta property="article:published_time" content="2018-03-02T16:07:54&#43;02:00"/>
 
-<meta property="article:modified_time" content="2018-03-20T21:04:10&#43;02:00"/>
+<meta property="article:modified_time" content="2018-03-21T11:44:06&#43;02:00"/>
 
 
 
@@ -51,9 +51,9 @@ Export a CSV of the IITA community metadata for Martin Mueller
   "@type": "BlogPosting",
   "headline": "March, 2018",
   "url": "https://alanorth.github.io/cgspace-notes/2018-03/",
-  "wordCount": "2113",
+  "wordCount": "2343",
   "datePublished": "2018-03-02T16:07:54&#43;02:00",
-  "dateModified": "2018-03-20T21:04:10&#43;02:00",
+  "dateModified": "2018-03-21T11:44:06&#43;02:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"
@@ -538,6 +538,58 @@ COPY 56156
 <li>Afterwards we&rsquo;ll want to do some batch tagging of ORCID identifiers to these names</li>
 </ul>
 
+<h2 id="2018-03-20-1">2018-03-20</h2>
+
+<ul>
+<li>CGSpace crashed again this afternoon, I&rsquo;m not sure of the cause but there are a lot of SQL errors in the DSpace log:</li>
+</ul>
+
+<pre><code>2018-03-21 15:11:08,166 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL QueryTable Error - 
+java.sql.SQLException: Connection has already been closed.
+</code></pre>
+
+<ul>
+<li>I have no idea why so many connections were abandoned this afternoon:</li>
+</ul>
+
+<pre><code># grep 'Mar 21, 2018' /var/log/tomcat7/catalina.out | grep -c 'org.apache.tomcat.jdbc.pool.ConnectionPool abandon'
+268
+</code></pre>
+
+<ul>
+<li>DSpace Test crashed again due to Java heap space, this is from the DSpace log:</li>
+</ul>
+
+<pre><code>2018-03-21 15:18:48,149 ERROR org.dspace.app.xmlui.cocoon.DSpaceCocoonServletFilter @ Serious Error Occurred Processing Request!
+org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space
+</code></pre>
+
+<ul>
+<li>And this is from the Tomcat Catalina log:</li>
+</ul>
+
+<pre><code>Mar 21, 2018 11:20:00 AM org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor run
+SEVERE: Unexpected death of background thread ContainerBackgroundProcessor[StandardEngine[Catalina]]
+java.lang.OutOfMemoryError: Java heap space
+</code></pre>
+
+<ul>
+<li>But there are tons of heap space errors on DSpace Test actually:</li>
+</ul>
+
+<pre><code># grep -c 'java.lang.OutOfMemoryError: Java heap space' /var/log/tomcat7/catalina.out
+319
+</code></pre>
+
+<ul>
+<li>I guess we need to give it more RAM because it now has CGSpace&rsquo;s large Solr core</li>
+<li>I will increase the memory from 3072m to 4096m</li>
+<li>Update <a href="https://github.com/ilri/rmg-ansible-public">Ansible playbooks</a> to use <a href="https://jdbc.postgresql.org/">PostgreSQL JBDC driver</a> 42.2.2</li>
+<li>Deploy the new JDBC driver on DSpace Test</li>
+<li>I&rsquo;m also curious to see how long the <code>dspace index-discovery -b</code> takes on DSpace Test where the DSpace installation directory is on one of Linode&rsquo;s new block storage volumes</li>
+<li>I should also check the raw read speed with <code>hdparm -tT /dev/sdc</code></li>
+</ul>
+
   
 
   

docs/sitemap.xml

@@ -4,7 +4,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2018-03/</loc>
-    <lastmod>2018-03-20T21:04:10+02:00</lastmod>
+    <lastmod>2018-03-21T11:44:06+02:00</lastmod>
   </url>
   
   <url>
@@ -154,7 +154,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2018-03-20T21:04:10+02:00</lastmod>
+    <lastmod>2018-03-21T11:44:06+02:00</lastmod>
     <priority>0</priority>
   </url>
   
@@ -165,7 +165,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
-    <lastmod>2018-03-20T21:04:10+02:00</lastmod>
+    <lastmod>2018-03-21T11:44:06+02:00</lastmod>
     <priority>0</priority>
   </url>
   
@@ -177,13 +177,13 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
-    <lastmod>2018-03-20T21:04:10+02:00</lastmod>
+    <lastmod>2018-03-21T11:44:06+02:00</lastmod>
     <priority>0</priority>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/tags/</loc>
-    <lastmod>2018-03-20T21:04:10+02:00</lastmod>
+    <lastmod>2018-03-21T11:44:06+02:00</lastmod>
     <priority>0</priority>
   </url>