Add notes for 2018-02-15

content/post/2018-02.md

@@ -532,3 +532,29 @@ UPDATE 1
 - And this item doesn't even exist on CGSpace!
 - Start working on XMLUI item display code for ORCIDs
 - Send emails to Macaroni Bros and Usman at CIFOR about ORCID metadata
+- CGSpace crashed while I was driving to Tel Aviv, and was down for four hours!
+- I only looked quickly in the logs but saw a bunch of database errors
+- PostgreSQL connections are currently:
+
+```
+$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | uniq -c
+      2 dspaceApi
+      1 dspaceWeb
+      3 dspaceApi
+```
+
+- I see shitloads of memory errors in Tomcat's logs:
+    
+```
+# grep -c "Java heap space" /var/log/tomcat7/catalina.out
+56
+```
+
+- And shit tons of database connections abandoned:
+
+```
+# grep -c 'org.apache.tomcat.jdbc.pool.ConnectionPool abandon' /var/log/tomcat7/catalina.out
+612
+```
+
+- I have no fucking idea why it crashed

docs/2018-02/index.html

@@ -23,7 +23,7 @@ I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu’s munin-pl
 
 <meta property="article:published_time" content="2018-02-01T16:28:54&#43;02:00"/>
 
-<meta property="article:modified_time" content="2018-02-14T16:45:03&#43;02:00"/>
+<meta property="article:modified_time" content="2018-02-15T14:00:34&#43;02:00"/>
 
 
 
@@ -57,9 +57,9 @@ I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu&rsquo;s munin-pl
   "@type": "BlogPosting",
   "headline": "February, 2018",
   "url": "https://alanorth.github.io/cgspace-notes/2018-02/",
-  "wordCount": "3581",
+  "wordCount": "3674",
   "datePublished": "2018-02-01T16:28:54&#43;02:00",
-  "dateModified": "2018-02-14T16:45:03&#43;02:00",
+  "dateModified": "2018-02-15T14:00:34&#43;02:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"
@@ -722,6 +722,36 @@ UPDATE 1
 <li>And this item doesn&rsquo;t even exist on CGSpace!</li>
 <li>Start working on XMLUI item display code for ORCIDs</li>
 <li>Send emails to Macaroni Bros and Usman at CIFOR about ORCID metadata</li>
+<li>CGSpace crashed while I was driving to Tel Aviv, and was down for four hours!</li>
+<li>I only looked quickly in the logs but saw a bunch of database errors</li>
+<li>PostgreSQL connections are currently:</li>
+</ul>
+
+<pre><code>$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | uniq -c
+      2 dspaceApi
+      1 dspaceWeb
+      3 dspaceApi
+</code></pre>
+
+<ul>
+<li>I see shitloads of memory errors in Tomcat&rsquo;s logs:
+<br /></li>
+</ul>
+
+<pre><code># grep -c &quot;Java heap space&quot; /var/log/tomcat7/catalina.out
+56
+</code></pre>
+
+<ul>
+<li>And shit tons of database connections abandoned:</li>
+</ul>
+
+<pre><code># grep -c 'org.apache.tomcat.jdbc.pool.ConnectionPool abandon' /var/log/tomcat7/catalina.out
+612
+</code></pre>
+
+<ul>
+<li>I have no fucking idea why it crashed</li>
 </ul>
 
   

docs/sitemap.xml

@@ -4,7 +4,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2018-02/</loc>
-    <lastmod>2018-02-14T16:45:03+02:00</lastmod>
+    <lastmod>2018-02-15T14:00:34+02:00</lastmod>
   </url>
   
   <url>
@@ -149,7 +149,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2018-02-14T16:45:03+02:00</lastmod>
+    <lastmod>2018-02-15T14:00:34+02:00</lastmod>
     <priority>0</priority>
   </url>
   
@@ -160,7 +160,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
-    <lastmod>2018-02-14T16:45:03+02:00</lastmod>
+    <lastmod>2018-02-15T14:00:34+02:00</lastmod>
     <priority>0</priority>
   </url>
   
@@ -172,13 +172,13 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/post/</loc>
-    <lastmod>2018-02-14T16:45:03+02:00</lastmod>
+    <lastmod>2018-02-15T14:00:34+02:00</lastmod>
     <priority>0</priority>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/tags/</loc>
-    <lastmod>2018-02-14T16:45:03+02:00</lastmod>
+    <lastmod>2018-02-15T14:00:34+02:00</lastmod>
     <priority>0</priority>
   </url>