Update notes for 2018-01-02

content/post/2018-01.md

@@ -9,7 +9,17 @@ tags: ["Notes"]
 
 - Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time
 - I didn't get any load alerts from Linode and the REST and XMLUI logs don't show anything out of the ordinary
-- So I don't know WHY Uptime Robot thought it was down so many times
+- The nginx logs show HTTP 200s until `02/Jan/2018:11:27:17 +0000` when Uptime Robot got an HTTP 500
+- In dspace.log around that time I see many errors like "Client closed the connection before file download was complete"
+- And just before that I see this:
+
+```
+Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000].
+```
+
+- Ah hah! So the pool was actually empty!
+- I need to increase that, let's try to bump it up from 50 to 75
+- After that one client got an HTTP 499 but then the rest were HTTP 200, so I don't know what the hell Uptime Robot saw
 - I notice this error quite a few times in dspace.log:
 
 ```

public/2015-11/index.html

@@ -26,7 +26,7 @@ $ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspac
 
 <meta property="article:published_time" content="2015-11-23T17:00:57&#43;03:00"/>
 
-<meta property="article:modified_time" content="2015-11-23T17:00:57&#43;03:00"/>
+<meta property="article:modified_time" content="2016-09-28T17:02:30&#43;03:00"/>
 
 
 
@@ -64,7 +64,7 @@ $ psql -c &#39;SELECT * from pg_stat_activity;&#39; | grep idle | grep -c cgspac
   "url": "https://alanorth.github.io/cgspace-notes/2015-11/",
   "wordCount": "798",
   "datePublished": "2015-11-23T17:00:57&#43;03:00",
-  "dateModified": "2015-11-23T17:00:57&#43;03:00",
+  "dateModified": "2016-09-28T17:02:30&#43;03:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2015-12/index.html

@@ -27,7 +27,7 @@ Replace lzop with xz in log compression cron jobs on DSpace Test—it uses less
 
 <meta property="article:published_time" content="2015-12-02T13:18:00&#43;03:00"/>
 
-<meta property="article:modified_time" content="2015-12-02T13:18:00&#43;03:00"/>
+<meta property="article:modified_time" content="2017-01-09T16:18:07&#43;02:00"/>
 
 
 
@@ -66,7 +66,7 @@ Replace lzop with xz in log compression cron jobs on DSpace Test—it uses less
   "url": "https://alanorth.github.io/cgspace-notes/2015-12/",
   "wordCount": "753",
   "datePublished": "2015-12-02T13:18:00&#43;03:00",
-  "dateModified": "2015-12-02T13:18:00&#43;03:00",
+  "dateModified": "2017-01-09T16:18:07&#43;02:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2016-01/index.html

@@ -22,7 +22,7 @@ Update GitHub wiki for documentation of maintenance tasks.
 
 <meta property="article:published_time" content="2016-01-13T13:18:00&#43;03:00"/>
 
-<meta property="article:modified_time" content="2016-01-13T13:18:00&#43;03:00"/>
+<meta property="article:modified_time" content="2017-01-09T16:18:07&#43;02:00"/>
 
 
 
@@ -56,7 +56,7 @@ Update GitHub wiki for documentation of maintenance tasks.
   "url": "https://alanorth.github.io/cgspace-notes/2016-01/",
   "wordCount": "466",
   "datePublished": "2016-01-13T13:18:00&#43;03:00",
-  "dateModified": "2016-01-13T13:18:00&#43;03:00",
+  "dateModified": "2017-01-09T16:18:07&#43;02:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2016-02/index.html

@@ -29,7 +29,7 @@ Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE&r
 
 <meta property="article:published_time" content="2016-02-05T13:18:00&#43;03:00"/>
 
-<meta property="article:modified_time" content="2016-02-05T13:18:00&#43;03:00"/>
+<meta property="article:modified_time" content="2017-01-09T16:18:07&#43;02:00"/>
 
 
 
@@ -70,7 +70,7 @@ Also, lots of things like &ldquo;COTE D`LVOIRE&rdquo; and &ldquo;COTE D IVOIRE&r
   "url": "https://alanorth.github.io/cgspace-notes/2016-02/",
   "wordCount": "1657",
   "datePublished": "2016-02-05T13:18:00&#43;03:00",
-  "dateModified": "2016-02-05T13:18:00&#43;03:00",
+  "dateModified": "2017-01-09T16:18:07&#43;02:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2016-03/index.html

@@ -22,7 +22,7 @@ Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Ja
 
 <meta property="article:published_time" content="2016-03-02T16:50:00&#43;03:00"/>
 
-<meta property="article:modified_time" content="2016-03-02T16:50:00&#43;03:00"/>
+<meta property="article:modified_time" content="2017-01-09T16:18:07&#43;02:00"/>
 
 
 
@@ -56,7 +56,7 @@ Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Ja
   "url": "https://alanorth.github.io/cgspace-notes/2016-03/",
   "wordCount": "1581",
   "datePublished": "2016-03-02T16:50:00&#43;03:00",
-  "dateModified": "2016-03-02T16:50:00&#43;03:00",
+  "dateModified": "2017-01-09T16:18:07&#43;02:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2016-04/index.html

@@ -24,7 +24,7 @@ Also, I noticed the checker log has some errors we should pay attention to:
 
 <meta property="article:published_time" content="2016-04-04T11:06:00&#43;03:00"/>
 
-<meta property="article:modified_time" content="2016-04-04T11:06:00&#43;03:00"/>
+<meta property="article:modified_time" content="2016-09-28T17:02:30&#43;03:00"/>
 
 
 
@@ -60,7 +60,7 @@ Also, I noticed the checker log has some errors we should pay attention to:
   "url": "https://alanorth.github.io/cgspace-notes/2016-04/",
   "wordCount": "2006",
   "datePublished": "2016-04-04T11:06:00&#43;03:00",
-  "dateModified": "2016-04-04T11:06:00&#43;03:00",
+  "dateModified": "2016-09-28T17:02:30&#43;03:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2016-05/index.html

@@ -26,7 +26,7 @@ There are 3,000 IPs accessing the REST API in a 24-hour period!
 
 <meta property="article:published_time" content="2016-05-01T23:06:00&#43;03:00"/>
 
-<meta property="article:modified_time" content="2016-05-01T23:06:00&#43;03:00"/>
+<meta property="article:modified_time" content="2017-01-09T16:18:07&#43;02:00"/>
 
 
 
@@ -64,7 +64,7 @@ There are 3,000 IPs accessing the REST API in a 24-hour period!
   "url": "https://alanorth.github.io/cgspace-notes/2016-05/",
   "wordCount": "1349",
   "datePublished": "2016-05-01T23:06:00&#43;03:00",
-  "dateModified": "2016-05-01T23:06:00&#43;03:00",
+  "dateModified": "2017-01-09T16:18:07&#43;02:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2016-06/index.html

@@ -25,7 +25,7 @@ Working on second phase of metadata migration, looks like this will work for mov
 
 <meta property="article:published_time" content="2016-06-01T10:53:00&#43;03:00"/>
 
-<meta property="article:modified_time" content="2016-06-01T10:53:00&#43;03:00"/>
+<meta property="article:modified_time" content="2017-01-09T16:18:07&#43;02:00"/>
 
 
 
@@ -62,7 +62,7 @@ Working on second phase of metadata migration, looks like this will work for mov
   "url": "https://alanorth.github.io/cgspace-notes/2016-06/",
   "wordCount": "1549",
   "datePublished": "2016-06-01T10:53:00&#43;03:00",
-  "dateModified": "2016-06-01T10:53:00&#43;03:00",
+  "dateModified": "2017-01-09T16:18:07&#43;02:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2016-07/index.html

@@ -33,7 +33,7 @@ In this case the select query was showing 95 results before the update
 
 <meta property="article:published_time" content="2016-07-01T10:53:00&#43;03:00"/>
 
-<meta property="article:modified_time" content="2016-07-01T10:53:00&#43;03:00"/>
+<meta property="article:modified_time" content="2017-01-09T16:18:07&#43;02:00"/>
 
 
 
@@ -78,7 +78,7 @@ In this case the select query was showing 95 results before the update
   "url": "https://alanorth.github.io/cgspace-notes/2016-07/",
   "wordCount": "866",
   "datePublished": "2016-07-01T10:53:00&#43;03:00",
-  "dateModified": "2016-07-01T10:53:00&#43;03:00",
+  "dateModified": "2017-01-09T16:18:07&#43;02:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2016-08/index.html

@@ -30,7 +30,7 @@ $ git rebase -i dspace-5.5
 
 <meta property="article:published_time" content="2016-08-01T15:53:00&#43;03:00"/>
 
-<meta property="article:modified_time" content="2016-08-01T15:53:00&#43;03:00"/>
+<meta property="article:modified_time" content="2017-01-09T16:18:07&#43;02:00"/>
 
 
 
@@ -72,7 +72,7 @@ $ git rebase -i dspace-5.5
   "url": "https://alanorth.github.io/cgspace-notes/2016-08/",
   "wordCount": "1514",
   "datePublished": "2016-08-01T15:53:00&#43;03:00",
-  "dateModified": "2016-08-01T15:53:00&#43;03:00",
+  "dateModified": "2017-01-09T16:18:07&#43;02:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2016-09/index.html

@@ -26,7 +26,7 @@ $ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=or
 
 <meta property="article:published_time" content="2016-09-01T15:53:00&#43;03:00"/>
 
-<meta property="article:modified_time" content="2016-09-01T15:53:00&#43;03:00"/>
+<meta property="article:modified_time" content="2017-01-09T16:18:07&#43;02:00"/>
 
 
 
@@ -64,7 +64,7 @@ $ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b &quot;dc=cgiarad,dc=or
   "url": "https://alanorth.github.io/cgspace-notes/2016-09/",
   "wordCount": "3298",
   "datePublished": "2016-09-01T15:53:00&#43;03:00",
-  "dateModified": "2016-09-01T15:53:00&#43;03:00",
+  "dateModified": "2017-01-09T16:18:07&#43;02:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2016-10/index.html

@@ -30,7 +30,7 @@ I exported a random item’s metadata as CSV, deleted all columns except id
 
 <meta property="article:published_time" content="2016-10-03T15:53:00&#43;03:00"/>
 
-<meta property="article:modified_time" content="2016-10-03T15:53:00&#43;03:00"/>
+<meta property="article:modified_time" content="2017-01-10T16:21:47&#43;02:00"/>
 
 
 
@@ -72,7 +72,7 @@ I exported a random item&rsquo;s metadata as CSV, deleted all columns except id
   "url": "https://alanorth.github.io/cgspace-notes/2016-10/",
   "wordCount": "1828",
   "datePublished": "2016-10-03T15:53:00&#43;03:00",
-  "dateModified": "2016-10-03T15:53:00&#43;03:00",
+  "dateModified": "2017-01-10T16:21:47&#43;02:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2016-11/index.html

@@ -22,7 +22,7 @@ Add dc.type to the output options for Atmire’s Listings and Reports module
 
 <meta property="article:published_time" content="2016-11-01T09:21:00&#43;03:00"/>
 
-<meta property="article:modified_time" content="2016-11-01T09:21:00&#43;03:00"/>
+<meta property="article:modified_time" content="2017-01-10T16:21:47&#43;02:00"/>
 
 
 
@@ -56,7 +56,7 @@ Add dc.type to the output options for Atmire&rsquo;s Listings and Reports module
   "url": "https://alanorth.github.io/cgspace-notes/2016-11/",
   "wordCount": "2825",
   "datePublished": "2016-11-01T09:21:00&#43;03:00",
-  "dateModified": "2016-11-01T09:21:00&#43;03:00",
+  "dateModified": "2017-01-10T16:21:47&#43;02:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2016-12/index.html

@@ -34,7 +34,7 @@ Another worrying error from dspace.log is:
 
 <meta property="article:published_time" content="2016-12-02T10:43:00&#43;03:00"/>
 
-<meta property="article:modified_time" content="2016-12-02T10:43:00&#43;03:00"/>
+<meta property="article:modified_time" content="2017-09-19T16:07:20&#43;03:00"/>
 
 
 
@@ -80,7 +80,7 @@ Another worrying error from dspace.log is:
   "url": "https://alanorth.github.io/cgspace-notes/2016-12/",
   "wordCount": "4078",
   "datePublished": "2016-12-02T10:43:00&#43;03:00",
-  "dateModified": "2016-12-02T10:43:00&#43;03:00",
+  "dateModified": "2017-09-19T16:07:20&#43;03:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2017-01/index.html

@@ -22,7 +22,7 @@ I asked on the dspace-tech mailing list because it seems to be broken, and actua
 
 <meta property="article:published_time" content="2017-01-02T10:43:00&#43;03:00"/>
 
-<meta property="article:modified_time" content="2017-01-02T10:43:00&#43;03:00"/>
+<meta property="article:modified_time" content="2017-01-29T13:18:32&#43;02:00"/>
 
 
 
@@ -56,7 +56,7 @@ I asked on the dspace-tech mailing list because it seems to be broken, and actua
   "url": "https://alanorth.github.io/cgspace-notes/2017-01/",
   "wordCount": "1594",
   "datePublished": "2017-01-02T10:43:00&#43;03:00",
-  "dateModified": "2017-01-02T10:43:00&#43;03:00",
+  "dateModified": "2017-01-29T13:18:32&#43;02:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2017-02/index.html

@@ -36,7 +36,7 @@ Looks like we’ll be using cg.identifier.ccafsprojectpii as the field name
 
 <meta property="article:published_time" content="2017-02-07T07:04:52-08:00"/>
 
-<meta property="article:modified_time" content="2017-02-07T07:04:52-08:00"/>
+<meta property="article:modified_time" content="2017-02-28T22:58:29&#43;02:00"/>
 
 
 
@@ -84,7 +84,7 @@ Looks like we&rsquo;ll be using cg.identifier.ccafsprojectpii as the field name
   "url": "https://alanorth.github.io/cgspace-notes/2017-02/",
   "wordCount": "2028",
   "datePublished": "2017-02-07T07:04:52-08:00",
-  "dateModified": "2017-02-07T07:04:52-08:00",
+  "dateModified": "2017-02-28T22:58:29&#43;02:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2017-03/index.html

@@ -38,7 +38,7 @@ $ identify ~/Desktop/alc_contrastes_desafios.jpg
 
 <meta property="article:published_time" content="2017-03-01T17:08:52&#43;02:00"/>
 
-<meta property="article:modified_time" content="2017-03-01T17:08:52&#43;02:00"/>
+<meta property="article:modified_time" content="2017-03-31T05:36:10&#43;03:00"/>
 
 
 
@@ -88,7 +88,7 @@ $ identify ~/Desktop/alc_contrastes_desafios.jpg
   "url": "https://alanorth.github.io/cgspace-notes/2017-03/",
   "wordCount": "1538",
   "datePublished": "2017-03-01T17:08:52&#43;02:00",
-  "dateModified": "2017-03-01T17:08:52&#43;02:00",
+  "dateModified": "2017-03-31T05:36:10&#43;03:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2017-04/index.html

@@ -31,7 +31,7 @@ $ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Th
 
 <meta property="article:published_time" content="2017-04-02T17:08:52&#43;02:00"/>
 
-<meta property="article:modified_time" content="2017-04-02T17:08:52&#43;02:00"/>
+<meta property="article:modified_time" content="2017-04-26T13:35:10&#43;03:00"/>
 
 
 
@@ -74,7 +74,7 @@ $ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p &quot;ImageMagick PDF Th
   "url": "https://alanorth.github.io/cgspace-notes/2017-04/",
   "wordCount": "2917",
   "datePublished": "2017-04-02T17:08:52&#43;02:00",
-  "dateModified": "2017-04-02T17:08:52&#43;02:00",
+  "dateModified": "2017-04-26T13:35:10&#43;03:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2017-05/index.html

@@ -14,7 +14,7 @@
 
 <meta property="article:published_time" content="2017-05-01T16:21:52&#43;02:00"/>
 
-<meta property="article:modified_time" content="2017-05-01T16:21:52&#43;02:00"/>
+<meta property="article:modified_time" content="2017-09-10T17:46:54&#43;03:00"/>
 
 
 
@@ -40,7 +40,7 @@
   "url": "https://alanorth.github.io/cgspace-notes/2017-05/",
   "wordCount": "2398",
   "datePublished": "2017-05-01T16:21:52&#43;02:00",
-  "dateModified": "2017-05-01T16:21:52&#43;02:00",
+  "dateModified": "2017-09-10T17:46:54&#43;03:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2017-06/index.html

@@ -14,7 +14,7 @@
 
 <meta property="article:published_time" content="2017-06-01T10:14:52&#43;03:00"/>
 
-<meta property="article:modified_time" content="2017-06-01T10:14:52&#43;03:00"/>
+<meta property="article:modified_time" content="2017-06-30T18:34:51&#43;03:00"/>
 
 
 
@@ -40,7 +40,7 @@
   "url": "https://alanorth.github.io/cgspace-notes/2017-06/",
   "wordCount": "1261",
   "datePublished": "2017-06-01T10:14:52&#43;03:00",
-  "dateModified": "2017-06-01T10:14:52&#43;03:00",
+  "dateModified": "2017-06-30T18:34:51&#43;03:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2017-07/index.html

@@ -28,7 +28,7 @@ We can use PostgreSQL’s extended output format (-x) plus sed to format the
 
 <meta property="article:published_time" content="2017-07-01T18:03:52&#43;03:00"/>
 
-<meta property="article:modified_time" content="2017-07-01T18:03:52&#43;03:00"/>
+<meta property="article:modified_time" content="2017-08-01T08:55:37&#43;03:00"/>
 
 
 
@@ -68,7 +68,7 @@ We can use PostgreSQL&rsquo;s extended output format (-x) plus sed to format the
   "url": "https://alanorth.github.io/cgspace-notes/2017-07/",
   "wordCount": "1151",
   "datePublished": "2017-07-01T18:03:52&#43;03:00",
-  "dateModified": "2017-07-01T18:03:52&#43;03:00",
+  "dateModified": "2017-08-01T08:55:37&#43;03:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2017-08/index.html

@@ -38,7 +38,7 @@ Then I cleaned up the author authorities and HTML characters in OpenRefine and s
 
 <meta property="article:published_time" content="2017-08-01T11:51:52&#43;03:00"/>
 
-<meta property="article:modified_time" content="2017-08-01T11:51:52&#43;03:00"/>
+<meta property="article:modified_time" content="2017-09-10T19:18:52&#43;03:00"/>
 
 
 
@@ -88,7 +88,7 @@ Then I cleaned up the author authorities and HTML characters in OpenRefine and s
   "url": "https://alanorth.github.io/cgspace-notes/2017-08/",
   "wordCount": "3542",
   "datePublished": "2017-08-01T11:51:52&#43;03:00",
-  "dateModified": "2017-08-01T11:51:52&#43;03:00",
+  "dateModified": "2017-09-10T19:18:52&#43;03:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2017-09/index.html

@@ -26,7 +26,7 @@ Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account
 
 <meta property="article:published_time" content="2017-09-07T16:54:52&#43;07:00"/>
 
-<meta property="article:modified_time" content="2017-09-07T16:54:52&#43;07:00"/>
+<meta property="article:modified_time" content="2017-09-28T07:56:11&#43;03:00"/>
 
 
 
@@ -64,7 +64,7 @@ Ask Sisay to clean up the WLE approvers a bit, as Marianne&rsquo;s user account
   "url": "https://alanorth.github.io/cgspace-notes/2017-09/",
   "wordCount": "4199",
   "datePublished": "2017-09-07T16:54:52&#43;07:00",
-  "dateModified": "2017-09-07T16:54:52&#43;07:00",
+  "dateModified": "2017-09-28T07:56:11&#43;03:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2017-10/index.html

@@ -28,7 +28,7 @@ Add Katherine Lutz to the groups for content submission and edit steps of the CG
 
 <meta property="article:published_time" content="2017-10-01T08:07:54&#43;03:00"/>
 
-<meta property="article:modified_time" content="2017-10-01T08:07:54&#43;03:00"/>
+<meta property="article:modified_time" content="2017-11-02T16:13:10&#43;02:00"/>
 
 
 
@@ -68,7 +68,7 @@ Add Katherine Lutz to the groups for content submission and edit steps of the CG
   "url": "https://alanorth.github.io/cgspace-notes/2017-10/",
   "wordCount": "2613",
   "datePublished": "2017-10-01T08:07:54&#43;03:00",
-  "dateModified": "2017-10-01T08:07:54&#43;03:00",
+  "dateModified": "2017-11-02T16:13:10&#43;02:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2017-11/index.html

@@ -38,7 +38,7 @@ COPY 54701
 
 <meta property="article:published_time" content="2017-11-02T09:37:54&#43;02:00"/>
 
-<meta property="article:modified_time" content="2017-11-02T09:37:54&#43;02:00"/>
+<meta property="article:modified_time" content="2017-12-17T10:43:56&#43;02:00"/>
 
 
 
@@ -88,7 +88,7 @@ COPY 54701
   "url": "https://alanorth.github.io/cgspace-notes/2017-11/",
   "wordCount": "5428",
   "datePublished": "2017-11-02T09:37:54&#43;02:00",
-  "dateModified": "2017-11-02T09:37:54&#43;02:00",
+  "dateModified": "2017-12-17T10:43:56&#43;02:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2017-12/index.html

@@ -23,7 +23,7 @@ The list of connections to XMLUI and REST API for today:
 
 <meta property="article:published_time" content="2017-12-01T13:53:54&#43;03:00"/>
 
-<meta property="article:modified_time" content="2017-12-01T13:53:54&#43;03:00"/>
+<meta property="article:modified_time" content="2017-12-31T10:42:16-08:00"/>
 
 
 
@@ -58,7 +58,7 @@ The list of connections to XMLUI and REST API for today:
   "url": "https://alanorth.github.io/cgspace-notes/2017-12/",
   "wordCount": "4088",
   "datePublished": "2017-12-01T13:53:54&#43;03:00",
-  "dateModified": "2017-12-01T13:53:54&#43;03:00",
+  "dateModified": "2017-12-31T10:42:16-08:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/2018-01/index.html

@@ -11,7 +11,18 @@
 
 Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time
 I didn’t get any load alerts from Linode and the REST and XMLUI logs don’t show anything out of the ordinary
-So I don’t know WHY Uptime Robot thought it was down so many times
+The nginx logs show HTTP 200s until 02/Jan/2018:11:27:17 +0000 when Uptime Robot got an HTTP 500
+In dspace.log around that time I see many errors like “Client closed the connection before file download was complete”
+And just before that I see this:
+
+
+Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000].
+
+
+
+Ah hah! So the pool was actually empty!
+I need to increase that, let’s try to bump it up from 50 to 75
+After that one client got an HTTP 499 but then the rest were HTTP 200, so I don’t know what the hell Uptime Robot saw
 I notice this error quite a few times in dspace.log:
 
 
@@ -81,7 +92,7 @@ Danny wrote to ask for help renewing the wildcard ilri.org certificate and I adv
 
 <meta property="article:published_time" content="2018-01-02T08:35:54-08:00"/>
 
-<meta property="article:modified_time" content="2018-01-02T08:35:54-08:00"/>
+<meta property="article:modified_time" content="2018-01-02T08:52:14-08:00"/>
 
 
 
@@ -99,7 +110,18 @@ Danny wrote to ask for help renewing the wildcard ilri.org certificate and I adv
 
 Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time
 I didn&rsquo;t get any load alerts from Linode and the REST and XMLUI logs don&rsquo;t show anything out of the ordinary
-So I don&rsquo;t know WHY Uptime Robot thought it was down so many times
+The nginx logs show HTTP 200s until 02/Jan/2018:11:27:17 &#43;0000 when Uptime Robot got an HTTP 500
+In dspace.log around that time I see many errors like &ldquo;Client closed the connection before file download was complete&rdquo;
+And just before that I see this:
+
+
+Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000].
+
+
+
+Ah hah! So the pool was actually empty!
+I need to increase that, let&rsquo;s try to bump it up from 50 to 75
+After that one client got an HTTP 499 but then the rest were HTTP 200, so I don&rsquo;t know what the hell Uptime Robot saw
 I notice this error quite a few times in dspace.log:
 
 
@@ -172,9 +194,9 @@ Danny wrote to ask for help renewing the wildcard ilri.org certificate and I adv
   "@type": "BlogPosting",
   "headline": "January, 2018",
   "url": "https://alanorth.github.io/cgspace-notes/2018-01/",
-  "wordCount": "186",
+  "wordCount": "282",
   "datePublished": "2018-01-02T08:35:54-08:00",
-  "dateModified": "2018-01-02T08:35:54-08:00",
+  "dateModified": "2018-01-02T08:52:14-08:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"
@@ -242,7 +264,18 @@ Danny wrote to ask for help renewing the wildcard ilri.org certificate and I adv
 <ul>
 <li>Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time</li>
 <li>I didn&rsquo;t get any load alerts from Linode and the REST and XMLUI logs don&rsquo;t show anything out of the ordinary</li>
-<li>So I don&rsquo;t know WHY Uptime Robot thought it was down so many times</li>
+<li>The nginx logs show HTTP 200s until <code>02/Jan/2018:11:27:17 +0000</code> when Uptime Robot got an HTTP 500</li>
+<li>In dspace.log around that time I see many errors like &ldquo;Client closed the connection before file download was complete&rdquo;</li>
+<li>And just before that I see this:</li>
+</ul>
+
+<pre><code>Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000].
+</code></pre>
+
+<ul>
+<li>Ah hah! So the pool was actually empty!</li>
+<li>I need to increase that, let&rsquo;s try to bump it up from 50 to 75</li>
+<li>After that one client got an HTTP 499 but then the rest were HTTP 200, so I don&rsquo;t know what the hell Uptime Robot saw</li>
 <li>I notice this error quite a few times in dspace.log:</li>
 </ul>
 

public/cgiar-library-migration/index.html

@@ -14,7 +14,7 @@
 
 <meta property="article:published_time" content="2017-09-18T16:38:35&#43;03:00"/>
 
-<meta property="article:modified_time" content="2017-09-18T16:38:35&#43;03:00"/>
+<meta property="article:modified_time" content="2017-09-28T12:00:49&#43;03:00"/>
 
 
 
@@ -40,7 +40,7 @@
   "url": "https://alanorth.github.io/cgspace-notes/cgiar-library-migration/",
   "wordCount": "1278",
   "datePublished": "2017-09-18T16:38:35&#43;03:00",
-  "dateModified": "2017-09-18T16:38:35&#43;03:00",
+  "dateModified": "2017-09-28T12:00:49&#43;03:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"

public/index.html

@@ -111,7 +111,18 @@
 <ul>
 <li>Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time</li>
 <li>I didn&rsquo;t get any load alerts from Linode and the REST and XMLUI logs don&rsquo;t show anything out of the ordinary</li>
-<li>So I don&rsquo;t know WHY Uptime Robot thought it was down so many times</li>
+<li>The nginx logs show HTTP 200s until <code>02/Jan/2018:11:27:17 +0000</code> when Uptime Robot got an HTTP 500</li>
+<li>In dspace.log around that time I see many errors like &ldquo;Client closed the connection before file download was complete&rdquo;</li>
+<li>And just before that I see this:</li>
+</ul>
+
+<pre><code>Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000].
+</code></pre>
+
+<ul>
+<li>Ah hah! So the pool was actually empty!</li>
+<li>I need to increase that, let&rsquo;s try to bump it up from 50 to 75</li>
+<li>After that one client got an HTTP 499 but then the rest were HTTP 200, so I don&rsquo;t know what the hell Uptime Robot saw</li>
 <li>I notice this error quite a few times in dspace.log:</li>
 </ul>
 

public/index.xml

@@ -22,7 +22,18 @@
 <ul>
 <li>Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time</li>
 <li>I didn’t get any load alerts from Linode and the REST and XMLUI logs don’t show anything out of the ordinary</li>
-<li>So I don’t know WHY Uptime Robot thought it was down so many times</li>
+<li>The nginx logs show HTTP 200s until <code>02/Jan/2018:11:27:17 +0000</code> when Uptime Robot got an HTTP 500</li>
+<li>In dspace.log around that time I see many errors like “Client closed the connection before file download was complete”</li>
+<li>And just before that I see this:</li>
+</ul>
+
+<pre><code>Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000].
+</code></pre>
+
+<ul>
+<li>Ah hah! So the pool was actually empty!</li>
+<li>I need to increase that, let’s try to bump it up from 50 to 75</li>
+<li>After that one client got an HTTP 499 but then the rest were HTTP 200, so I don’t know what the hell Uptime Robot saw</li>
 <li>I notice this error quite a few times in dspace.log:</li>
 </ul>
 

public/post/index.html

@@ -111,7 +111,18 @@
 <ul>
 <li>Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time</li>
 <li>I didn&rsquo;t get any load alerts from Linode and the REST and XMLUI logs don&rsquo;t show anything out of the ordinary</li>
-<li>So I don&rsquo;t know WHY Uptime Robot thought it was down so many times</li>
+<li>The nginx logs show HTTP 200s until <code>02/Jan/2018:11:27:17 +0000</code> when Uptime Robot got an HTTP 500</li>
+<li>In dspace.log around that time I see many errors like &ldquo;Client closed the connection before file download was complete&rdquo;</li>
+<li>And just before that I see this:</li>
+</ul>
+
+<pre><code>Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000].
+</code></pre>
+
+<ul>
+<li>Ah hah! So the pool was actually empty!</li>
+<li>I need to increase that, let&rsquo;s try to bump it up from 50 to 75</li>
+<li>After that one client got an HTTP 499 but then the rest were HTTP 200, so I don&rsquo;t know what the hell Uptime Robot saw</li>
 <li>I notice this error quite a few times in dspace.log:</li>
 </ul>
 

public/post/index.xml

@@ -22,7 +22,18 @@
 <ul>
 <li>Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time</li>
 <li>I didn’t get any load alerts from Linode and the REST and XMLUI logs don’t show anything out of the ordinary</li>
-<li>So I don’t know WHY Uptime Robot thought it was down so many times</li>
+<li>The nginx logs show HTTP 200s until <code>02/Jan/2018:11:27:17 +0000</code> when Uptime Robot got an HTTP 500</li>
+<li>In dspace.log around that time I see many errors like “Client closed the connection before file download was complete”</li>
+<li>And just before that I see this:</li>
+</ul>
+
+<pre><code>Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000].
+</code></pre>
+
+<ul>
+<li>Ah hah! So the pool was actually empty!</li>
+<li>I need to increase that, let’s try to bump it up from 50 to 75</li>
+<li>After that one client got an HTTP 499 but then the rest were HTTP 200, so I don’t know what the hell Uptime Robot saw</li>
 <li>I notice this error quite a few times in dspace.log:</li>
 </ul>
 

public/sitemap.xml

@@ -4,147 +4,147 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2018-01/</loc>
-    <lastmod>2018-01-02T08:35:54-08:00</lastmod>
+    <lastmod>2018-01-02T08:52:14-08:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2017-12/</loc>
-    <lastmod>2017-12-01T13:53:54+03:00</lastmod>
+    <lastmod>2017-12-31T10:42:16-08:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2017-11/</loc>
-    <lastmod>2017-11-02T09:37:54+02:00</lastmod>
+    <lastmod>2017-12-17T10:43:56+02:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2017-10/</loc>
-    <lastmod>2017-10-01T08:07:54+03:00</lastmod>
+    <lastmod>2017-11-02T16:13:10+02:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/cgiar-library-migration/</loc>
-    <lastmod>2017-09-18T16:38:35+03:00</lastmod>
+    <lastmod>2017-09-28T12:00:49+03:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2017-09/</loc>
-    <lastmod>2017-09-07T16:54:52+07:00</lastmod>
+    <lastmod>2017-09-28T07:56:11+03:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2017-08/</loc>
-    <lastmod>2017-08-01T11:51:52+03:00</lastmod>
+    <lastmod>2017-09-10T19:18:52+03:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2017-07/</loc>
-    <lastmod>2017-07-01T18:03:52+03:00</lastmod>
+    <lastmod>2017-08-01T08:55:37+03:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2017-06/</loc>
-    <lastmod>2017-06-01T10:14:52+03:00</lastmod>
+    <lastmod>2017-06-30T18:34:51+03:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2017-05/</loc>
-    <lastmod>2017-05-01T16:21:52+02:00</lastmod>
+    <lastmod>2017-09-10T17:46:54+03:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2017-04/</loc>
-    <lastmod>2017-04-02T17:08:52+02:00</lastmod>
+    <lastmod>2017-04-26T13:35:10+03:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2017-03/</loc>
-    <lastmod>2017-03-01T17:08:52+02:00</lastmod>
+    <lastmod>2017-03-31T05:36:10+03:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2017-02/</loc>
-    <lastmod>2017-02-07T07:04:52-08:00</lastmod>
+    <lastmod>2017-02-28T22:58:29+02:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2017-01/</loc>
-    <lastmod>2017-01-02T10:43:00+03:00</lastmod>
+    <lastmod>2017-01-29T13:18:32+02:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2016-12/</loc>
-    <lastmod>2016-12-02T10:43:00+03:00</lastmod>
+    <lastmod>2017-09-19T16:07:20+03:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2016-11/</loc>
-    <lastmod>2016-11-01T09:21:00+03:00</lastmod>
+    <lastmod>2017-01-10T16:21:47+02:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2016-10/</loc>
-    <lastmod>2016-10-03T15:53:00+03:00</lastmod>
+    <lastmod>2017-01-10T16:21:47+02:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2016-09/</loc>
-    <lastmod>2016-09-01T15:53:00+03:00</lastmod>
+    <lastmod>2017-01-09T16:18:07+02:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2016-08/</loc>
-    <lastmod>2016-08-01T15:53:00+03:00</lastmod>
+    <lastmod>2017-01-09T16:18:07+02:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2016-07/</loc>
-    <lastmod>2016-07-01T10:53:00+03:00</lastmod>
+    <lastmod>2017-01-09T16:18:07+02:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2016-06/</loc>
-    <lastmod>2016-06-01T10:53:00+03:00</lastmod>
+    <lastmod>2017-01-09T16:18:07+02:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2016-05/</loc>
-    <lastmod>2016-05-01T23:06:00+03:00</lastmod>
+    <lastmod>2017-01-09T16:18:07+02:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2016-04/</loc>
-    <lastmod>2016-04-04T11:06:00+03:00</lastmod>
+    <lastmod>2016-09-28T17:02:30+03:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2016-03/</loc>
-    <lastmod>2016-03-02T16:50:00+03:00</lastmod>
+    <lastmod>2017-01-09T16:18:07+02:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2016-02/</loc>
-    <lastmod>2016-02-05T13:18:00+03:00</lastmod>
+    <lastmod>2017-01-09T16:18:07+02:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2016-01/</loc>
-    <lastmod>2016-01-13T13:18:00+03:00</lastmod>
+    <lastmod>2017-01-09T16:18:07+02:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2015-12/</loc>
-    <lastmod>2015-12-02T13:18:00+03:00</lastmod>
+    <lastmod>2017-01-09T16:18:07+02:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2015-11/</loc>
-    <lastmod>2015-11-23T17:00:57+03:00</lastmod>
+    <lastmod>2016-09-28T17:02:30+03:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2018-01-02T08:35:54-08:00</lastmod>
+    <lastmod>2018-01-02T08:52:14-08:00</lastmod>
     <priority>0</priority>
   </url>
   
@@ -155,25 +155,25 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
-    <lastmod>2018-01-02T08:35:54-08:00</lastmod>
+    <lastmod>2018-01-02T08:52:14-08:00</lastmod>
     <priority>0</priority>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
-    <lastmod>2017-09-18T16:38:35+03:00</lastmod>
+    <lastmod>2017-09-28T12:00:49+03:00</lastmod>
     <priority>0</priority>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/post/</loc>
-    <lastmod>2018-01-02T08:35:54-08:00</lastmod>
+    <lastmod>2018-01-02T08:52:14-08:00</lastmod>
     <priority>0</priority>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/tags/</loc>
-    <lastmod>2018-01-02T08:35:54-08:00</lastmod>
+    <lastmod>2018-01-02T08:52:14-08:00</lastmod>
     <priority>0</priority>
   </url>
   

public/tags/notes/index.html

@@ -111,7 +111,18 @@
 <ul>
 <li>Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time</li>
 <li>I didn&rsquo;t get any load alerts from Linode and the REST and XMLUI logs don&rsquo;t show anything out of the ordinary</li>
-<li>So I don&rsquo;t know WHY Uptime Robot thought it was down so many times</li>
+<li>The nginx logs show HTTP 200s until <code>02/Jan/2018:11:27:17 +0000</code> when Uptime Robot got an HTTP 500</li>
+<li>In dspace.log around that time I see many errors like &ldquo;Client closed the connection before file download was complete&rdquo;</li>
+<li>And just before that I see this:</li>
+</ul>
+
+<pre><code>Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000].
+</code></pre>
+
+<ul>
+<li>Ah hah! So the pool was actually empty!</li>
+<li>I need to increase that, let&rsquo;s try to bump it up from 50 to 75</li>
+<li>After that one client got an HTTP 499 but then the rest were HTTP 200, so I don&rsquo;t know what the hell Uptime Robot saw</li>
 <li>I notice this error quite a few times in dspace.log:</li>
 </ul>
 

public/tags/notes/index.xml

@@ -22,7 +22,18 @@
 <ul>
 <li>Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time</li>
 <li>I didn’t get any load alerts from Linode and the REST and XMLUI logs don’t show anything out of the ordinary</li>
-<li>So I don’t know WHY Uptime Robot thought it was down so many times</li>
+<li>The nginx logs show HTTP 200s until <code>02/Jan/2018:11:27:17 +0000</code> when Uptime Robot got an HTTP 500</li>
+<li>In dspace.log around that time I see many errors like “Client closed the connection before file download was complete”</li>
+<li>And just before that I see this:</li>
+</ul>
+
+<pre><code>Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000].
+</code></pre>
+
+<ul>
+<li>Ah hah! So the pool was actually empty!</li>
+<li>I need to increase that, let’s try to bump it up from 50 to 75</li>
+<li>After that one client got an HTTP 499 but then the rest were HTTP 200, so I don’t know what the hell Uptime Robot saw</li>
 <li>I notice this error quite a few times in dspace.log:</li>
 </ul>