Add notes for 2020-04-29

content/posts/2020-04.md

@@ -359,5 +359,41 @@ Caused by: java.lang.NullPointerException
 - Also, now I can actually see the CUA statlets and usage statistics
   - Unfortunately it seems they are using Font Awesome 4 in their CUA module and this means that some icons are broken because the names have changed, but also some of their code is using Unicode characters instead of classes in spans!
   - I've reverted my sweet SVG work from yesterday and adjusted the Font Awesome 5 SCSS to add a few more icons that they are using
+- Tezira said she was having issues submitting items on CGSpace today
+  - I looked at all the errors in the DSpace log and see a few SQL pooling errors around mid day:
+
+```
+$ grep ERROR dspace.log.2020-04-29 | cut -f 3- -d' ' | sort | uniq -c | sort -n
+      1 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL findByUnique Error -
+      1 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL find Error -
+      1 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL query singleTable Error -
+      1 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL QueryTable Error -
+```
+
+- I looked in Munin and I see that there is a strange spike in the database pool usage this afternoon:
+
+![Tomcat Database Pool usage day](/cgspace-notes/2020/04/jmx_tomcat_dbpools-day.png)
+
+- Looking at the past month it does seem to be something strange:
+
+![Tomcat Database Pool usage month](/cgspace-notes/2020/04/jmx_tomcat_dbpools-month.png)
+
+- Database connections do seem high:
+
+```
+$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+      5 dspaceApi
+      6 dspaceCli
+     88 dspaceWeb
+```
+
+- Most of those are idle in transaction:
+
+```
+$ psql -c 'select * from pg_stat_activity' | grep 'dspaceWeb' | grep -c "idle in transaction"
+67
+```
+
+- I don't see anything in the PostgreSQL or Tomcat logs suggesting anything is wrong... I think the solution to clear these idle connections is probably to just restart Tomcat
 
 <!-- vim: set sw=2 ts=2: -->

docs/2020-04/index.html

@@ -25,7 +25,7 @@ On the same note, the one item Abenet pointed out last week now has a donut with
 <meta property="og:type" content="article" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2020-04/" />
 <meta property="article:published_time" content="2020-04-02T10:53:24+03:00" />
-<meta property="article:modified_time" content="2020-04-28T18:18:34+03:00" />
+<meta property="article:modified_time" content="2020-04-29T14:20:28+03:00" />
 
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="April, 2020"/>
@@ -55,9 +55,9 @@ On the same note, the one item Abenet pointed out last week now has a donut with
   "@type": "BlogPosting",
   "headline": "April, 2020",
   "url": "https://alanorth.github.io/cgspace-notes/2020-04/",
-  "wordCount": "2592",
+  "wordCount": "2787",
   "datePublished": "2020-04-02T10:53:24+03:00",
-  "dateModified": "2020-04-28T18:18:34+03:00",
+  "dateModified": "2020-04-29T14:20:28+03:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"
@@ -499,6 +499,39 @@ Caused by: java.lang.NullPointerException
 <li>I&rsquo;ve reverted my sweet SVG work from yesterday and adjusted the Font Awesome 5 SCSS to add a few more icons that they are using</li>
 </ul>
 </li>
+<li>Tezira said she was having issues submitting items on CGSpace today
+<ul>
+<li>I looked at all the errors in the DSpace log and see a few SQL pooling errors around mid day:</li>
+</ul>
+</li>
+</ul>
+<pre><code>$ grep ERROR dspace.log.2020-04-29 | cut -f 3- -d' ' | sort | uniq -c | sort -n
+      1 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL findByUnique Error -
+      1 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL find Error -
+      1 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL query singleTable Error -
+      1 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL QueryTable Error -
+</code></pre><ul>
+<li>I looked in Munin and I see that there is a strange spike in the database pool usage this afternoon:</li>
+</ul>
+<p><img src="/cgspace-notes/2020/04/jmx_tomcat_dbpools-day.png" alt="Tomcat Database Pool usage day"></p>
+<ul>
+<li>Looking at the past month it does seem to be something strange:</li>
+</ul>
+<p><img src="/cgspace-notes/2020/04/jmx_tomcat_dbpools-month.png" alt="Tomcat Database Pool usage month"></p>
+<ul>
+<li>Database connections do seem high:</li>
+</ul>
+<pre><code>$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+      5 dspaceApi
+      6 dspaceCli
+     88 dspaceWeb
+</code></pre><ul>
+<li>Most of those are idle in transaction:</li>
+</ul>
+<pre><code>$ psql -c 'select * from pg_stat_activity' | grep 'dspaceWeb' | grep -c &quot;idle in transaction&quot;
+67
+</code></pre><ul>
+<li>I don&rsquo;t see anything in the PostgreSQL or Tomcat logs suggesting anything is wrong&hellip; I think the solution to clear these idle connections is probably to just restart Tomcat</li>
 </ul>
 <!-- raw HTML omitted -->
 

docs/sitemap.xml

@@ -4,27 +4,27 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2020-04/</loc>
-    <lastmod>2020-04-28T18:18:34+03:00</lastmod>
+    <lastmod>2020-04-29T14:20:28+03:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/</loc>
-    <lastmod>2020-04-28T18:18:34+03:00</lastmod>
+    <lastmod>2020-04-29T14:20:28+03:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2020-04-28T18:18:34+03:00</lastmod>
+    <lastmod>2020-04-29T14:20:28+03:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
-    <lastmod>2020-04-28T18:18:34+03:00</lastmod>
+    <lastmod>2020-04-29T14:20:28+03:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
-    <lastmod>2020-04-28T18:18:34+03:00</lastmod>
+    <lastmod>2020-04-29T14:20:28+03:00</lastmod>
   </url>
   
   <url>