diff --git a/content/post/2018-01.md b/content/post/2018-01.md index 354241341..e6fe21567 100644 --- a/content/post/2018-01.md +++ b/content/post/2018-01.md @@ -9,7 +9,17 @@ tags: ["Notes"] - Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time - I didn't get any load alerts from Linode and the REST and XMLUI logs don't show anything out of the ordinary -- So I don't know WHY Uptime Robot thought it was down so many times +- The nginx logs show HTTP 200s until `02/Jan/2018:11:27:17 +0000` when Uptime Robot got an HTTP 500 +- In dspace.log around that time I see many errors like "Client closed the connection before file download was complete" +- And just before that I see this: + +``` +Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000]. +``` + +- Ah hah! So the pool was actually empty! +- I need to increase that, let's try to bump it up from 50 to 75 +- After that one client got an HTTP 499 but then the rest were HTTP 200, so I don't know what the hell Uptime Robot saw - I notice this error quite a few times in dspace.log: ``` diff --git a/public/2015-11/index.html b/public/2015-11/index.html index ef456c9e8..169e51070 100644 --- a/public/2015-11/index.html +++ b/public/2015-11/index.html @@ -26,7 +26,7 @@ $ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspac - + @@ -64,7 +64,7 @@ $ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspac "url": "https://alanorth.github.io/cgspace-notes/2015-11/", "wordCount": "798", "datePublished": "2015-11-23T17:00:57+03:00", - "dateModified": "2015-11-23T17:00:57+03:00", + "dateModified": "2016-09-28T17:02:30+03:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2015-12/index.html b/public/2015-12/index.html index d4561f244..cc076202f 100644 --- a/public/2015-12/index.html +++ b/public/2015-12/index.html @@ -27,7 +27,7 @@ Replace lzop with xz in log compression cron jobs on DSpace Test—it uses less - + @@ -66,7 +66,7 @@ Replace lzop with xz in log compression cron jobs on DSpace Test—it uses less "url": "https://alanorth.github.io/cgspace-notes/2015-12/", "wordCount": "753", "datePublished": "2015-12-02T13:18:00+03:00", - "dateModified": "2015-12-02T13:18:00+03:00", + "dateModified": "2017-01-09T16:18:07+02:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2016-01/index.html b/public/2016-01/index.html index ff2798d70..858e1a6d9 100644 --- a/public/2016-01/index.html +++ b/public/2016-01/index.html @@ -22,7 +22,7 @@ Update GitHub wiki for documentation of maintenance tasks. - + @@ -56,7 +56,7 @@ Update GitHub wiki for documentation of maintenance tasks. "url": "https://alanorth.github.io/cgspace-notes/2016-01/", "wordCount": "466", "datePublished": "2016-01-13T13:18:00+03:00", - "dateModified": "2016-01-13T13:18:00+03:00", + "dateModified": "2017-01-09T16:18:07+02:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2016-02/index.html b/public/2016-02/index.html index e3252f793..ca9515203 100644 --- a/public/2016-02/index.html +++ b/public/2016-02/index.html @@ -29,7 +29,7 @@ Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE&r - + @@ -70,7 +70,7 @@ Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE&r "url": "https://alanorth.github.io/cgspace-notes/2016-02/", "wordCount": "1657", "datePublished": "2016-02-05T13:18:00+03:00", - "dateModified": "2016-02-05T13:18:00+03:00", + "dateModified": "2017-01-09T16:18:07+02:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2016-03/index.html b/public/2016-03/index.html index ac83c70e4..70e484888 100644 --- a/public/2016-03/index.html +++ b/public/2016-03/index.html @@ -22,7 +22,7 @@ Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Ja - + @@ -56,7 +56,7 @@ Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Ja "url": "https://alanorth.github.io/cgspace-notes/2016-03/", "wordCount": "1581", "datePublished": "2016-03-02T16:50:00+03:00", - "dateModified": "2016-03-02T16:50:00+03:00", + "dateModified": "2017-01-09T16:18:07+02:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2016-04/index.html b/public/2016-04/index.html index 0104e1ca1..a752d0c18 100644 --- a/public/2016-04/index.html +++ b/public/2016-04/index.html @@ -24,7 +24,7 @@ Also, I noticed the checker log has some errors we should pay attention to: - + @@ -60,7 +60,7 @@ Also, I noticed the checker log has some errors we should pay attention to: "url": "https://alanorth.github.io/cgspace-notes/2016-04/", "wordCount": "2006", "datePublished": "2016-04-04T11:06:00+03:00", - "dateModified": "2016-04-04T11:06:00+03:00", + "dateModified": "2016-09-28T17:02:30+03:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2016-05/index.html b/public/2016-05/index.html index 4e4c40d44..49535e2c1 100644 --- a/public/2016-05/index.html +++ b/public/2016-05/index.html @@ -26,7 +26,7 @@ There are 3,000 IPs accessing the REST API in a 24-hour period! - + @@ -64,7 +64,7 @@ There are 3,000 IPs accessing the REST API in a 24-hour period! "url": "https://alanorth.github.io/cgspace-notes/2016-05/", "wordCount": "1349", "datePublished": "2016-05-01T23:06:00+03:00", - "dateModified": "2016-05-01T23:06:00+03:00", + "dateModified": "2017-01-09T16:18:07+02:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2016-06/index.html b/public/2016-06/index.html index 68976909d..d8818f23e 100644 --- a/public/2016-06/index.html +++ b/public/2016-06/index.html @@ -25,7 +25,7 @@ Working on second phase of metadata migration, looks like this will work for mov - + @@ -62,7 +62,7 @@ Working on second phase of metadata migration, looks like this will work for mov "url": "https://alanorth.github.io/cgspace-notes/2016-06/", "wordCount": "1549", "datePublished": "2016-06-01T10:53:00+03:00", - "dateModified": "2016-06-01T10:53:00+03:00", + "dateModified": "2017-01-09T16:18:07+02:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2016-07/index.html b/public/2016-07/index.html index 33dca5084..28612193b 100644 --- a/public/2016-07/index.html +++ b/public/2016-07/index.html @@ -33,7 +33,7 @@ In this case the select query was showing 95 results before the update - + @@ -78,7 +78,7 @@ In this case the select query was showing 95 results before the update "url": "https://alanorth.github.io/cgspace-notes/2016-07/", "wordCount": "866", "datePublished": "2016-07-01T10:53:00+03:00", - "dateModified": "2016-07-01T10:53:00+03:00", + "dateModified": "2017-01-09T16:18:07+02:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2016-08/index.html b/public/2016-08/index.html index f8d3354d8..53c34058c 100644 --- a/public/2016-08/index.html +++ b/public/2016-08/index.html @@ -30,7 +30,7 @@ $ git rebase -i dspace-5.5 - + @@ -72,7 +72,7 @@ $ git rebase -i dspace-5.5 "url": "https://alanorth.github.io/cgspace-notes/2016-08/", "wordCount": "1514", "datePublished": "2016-08-01T15:53:00+03:00", - "dateModified": "2016-08-01T15:53:00+03:00", + "dateModified": "2017-01-09T16:18:07+02:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2016-09/index.html b/public/2016-09/index.html index 681cd114d..26e66450f 100644 --- a/public/2016-09/index.html +++ b/public/2016-09/index.html @@ -26,7 +26,7 @@ $ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=or - + @@ -64,7 +64,7 @@ $ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=or "url": "https://alanorth.github.io/cgspace-notes/2016-09/", "wordCount": "3298", "datePublished": "2016-09-01T15:53:00+03:00", - "dateModified": "2016-09-01T15:53:00+03:00", + "dateModified": "2017-01-09T16:18:07+02:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2016-10/index.html b/public/2016-10/index.html index b43d4ebd9..719e0f933 100644 --- a/public/2016-10/index.html +++ b/public/2016-10/index.html @@ -30,7 +30,7 @@ I exported a random item’s metadata as CSV, deleted all columns except id - + @@ -72,7 +72,7 @@ I exported a random item’s metadata as CSV, deleted all columns except id "url": "https://alanorth.github.io/cgspace-notes/2016-10/", "wordCount": "1828", "datePublished": "2016-10-03T15:53:00+03:00", - "dateModified": "2016-10-03T15:53:00+03:00", + "dateModified": "2017-01-10T16:21:47+02:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2016-11/index.html b/public/2016-11/index.html index d350aa0ab..982737408 100644 --- a/public/2016-11/index.html +++ b/public/2016-11/index.html @@ -22,7 +22,7 @@ Add dc.type to the output options for Atmire’s Listings and Reports module - + @@ -56,7 +56,7 @@ Add dc.type to the output options for Atmire’s Listings and Reports module "url": "https://alanorth.github.io/cgspace-notes/2016-11/", "wordCount": "2825", "datePublished": "2016-11-01T09:21:00+03:00", - "dateModified": "2016-11-01T09:21:00+03:00", + "dateModified": "2017-01-10T16:21:47+02:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2016-12/index.html b/public/2016-12/index.html index 84761ef12..86298ba7e 100644 --- a/public/2016-12/index.html +++ b/public/2016-12/index.html @@ -34,7 +34,7 @@ Another worrying error from dspace.log is: - + @@ -80,7 +80,7 @@ Another worrying error from dspace.log is: "url": "https://alanorth.github.io/cgspace-notes/2016-12/", "wordCount": "4078", "datePublished": "2016-12-02T10:43:00+03:00", - "dateModified": "2016-12-02T10:43:00+03:00", + "dateModified": "2017-09-19T16:07:20+03:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2017-01/index.html b/public/2017-01/index.html index 8a68388d1..1305a11a0 100644 --- a/public/2017-01/index.html +++ b/public/2017-01/index.html @@ -22,7 +22,7 @@ I asked on the dspace-tech mailing list because it seems to be broken, and actua - + @@ -56,7 +56,7 @@ I asked on the dspace-tech mailing list because it seems to be broken, and actua "url": "https://alanorth.github.io/cgspace-notes/2017-01/", "wordCount": "1594", "datePublished": "2017-01-02T10:43:00+03:00", - "dateModified": "2017-01-02T10:43:00+03:00", + "dateModified": "2017-01-29T13:18:32+02:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2017-02/index.html b/public/2017-02/index.html index bd5a4ced1..3b2911cc4 100644 --- a/public/2017-02/index.html +++ b/public/2017-02/index.html @@ -36,7 +36,7 @@ Looks like we’ll be using cg.identifier.ccafsprojectpii as the field name - + @@ -84,7 +84,7 @@ Looks like we’ll be using cg.identifier.ccafsprojectpii as the field name "url": "https://alanorth.github.io/cgspace-notes/2017-02/", "wordCount": "2028", "datePublished": "2017-02-07T07:04:52-08:00", - "dateModified": "2017-02-07T07:04:52-08:00", + "dateModified": "2017-02-28T22:58:29+02:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2017-03/index.html b/public/2017-03/index.html index 94b9722bc..9ade1839e 100644 --- a/public/2017-03/index.html +++ b/public/2017-03/index.html @@ -38,7 +38,7 @@ $ identify ~/Desktop/alc_contrastes_desafios.jpg - + @@ -88,7 +88,7 @@ $ identify ~/Desktop/alc_contrastes_desafios.jpg "url": "https://alanorth.github.io/cgspace-notes/2017-03/", "wordCount": "1538", "datePublished": "2017-03-01T17:08:52+02:00", - "dateModified": "2017-03-01T17:08:52+02:00", + "dateModified": "2017-03-31T05:36:10+03:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2017-04/index.html b/public/2017-04/index.html index 3f2377879..8c2d7f472 100644 --- a/public/2017-04/index.html +++ b/public/2017-04/index.html @@ -31,7 +31,7 @@ $ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Th - + @@ -74,7 +74,7 @@ $ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Th "url": "https://alanorth.github.io/cgspace-notes/2017-04/", "wordCount": "2917", "datePublished": "2017-04-02T17:08:52+02:00", - "dateModified": "2017-04-02T17:08:52+02:00", + "dateModified": "2017-04-26T13:35:10+03:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2017-05/index.html b/public/2017-05/index.html index 7ab1cc7c5..210793cb0 100644 --- a/public/2017-05/index.html +++ b/public/2017-05/index.html @@ -14,7 +14,7 @@ - + @@ -40,7 +40,7 @@ "url": "https://alanorth.github.io/cgspace-notes/2017-05/", "wordCount": "2398", "datePublished": "2017-05-01T16:21:52+02:00", - "dateModified": "2017-05-01T16:21:52+02:00", + "dateModified": "2017-09-10T17:46:54+03:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2017-06/index.html b/public/2017-06/index.html index 22dda8179..ef5b29adb 100644 --- a/public/2017-06/index.html +++ b/public/2017-06/index.html @@ -14,7 +14,7 @@ - + @@ -40,7 +40,7 @@ "url": "https://alanorth.github.io/cgspace-notes/2017-06/", "wordCount": "1261", "datePublished": "2017-06-01T10:14:52+03:00", - "dateModified": "2017-06-01T10:14:52+03:00", + "dateModified": "2017-06-30T18:34:51+03:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2017-07/index.html b/public/2017-07/index.html index e04825d16..8f48c81be 100644 --- a/public/2017-07/index.html +++ b/public/2017-07/index.html @@ -28,7 +28,7 @@ We can use PostgreSQL’s extended output format (-x) plus sed to format the - + @@ -68,7 +68,7 @@ We can use PostgreSQL’s extended output format (-x) plus sed to format the "url": "https://alanorth.github.io/cgspace-notes/2017-07/", "wordCount": "1151", "datePublished": "2017-07-01T18:03:52+03:00", - "dateModified": "2017-07-01T18:03:52+03:00", + "dateModified": "2017-08-01T08:55:37+03:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2017-08/index.html b/public/2017-08/index.html index 9b5fd1e3e..9ca36b772 100644 --- a/public/2017-08/index.html +++ b/public/2017-08/index.html @@ -38,7 +38,7 @@ Then I cleaned up the author authorities and HTML characters in OpenRefine and s - + @@ -88,7 +88,7 @@ Then I cleaned up the author authorities and HTML characters in OpenRefine and s "url": "https://alanorth.github.io/cgspace-notes/2017-08/", "wordCount": "3542", "datePublished": "2017-08-01T11:51:52+03:00", - "dateModified": "2017-08-01T11:51:52+03:00", + "dateModified": "2017-09-10T19:18:52+03:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2017-09/index.html b/public/2017-09/index.html index 8f21a1b82..3dd2e1810 100644 --- a/public/2017-09/index.html +++ b/public/2017-09/index.html @@ -26,7 +26,7 @@ Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account - + @@ -64,7 +64,7 @@ Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account "url": "https://alanorth.github.io/cgspace-notes/2017-09/", "wordCount": "4199", "datePublished": "2017-09-07T16:54:52+07:00", - "dateModified": "2017-09-07T16:54:52+07:00", + "dateModified": "2017-09-28T07:56:11+03:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2017-10/index.html b/public/2017-10/index.html index 409fa16e2..d53f3fd25 100644 --- a/public/2017-10/index.html +++ b/public/2017-10/index.html @@ -28,7 +28,7 @@ Add Katherine Lutz to the groups for content submission and edit steps of the CG - + @@ -68,7 +68,7 @@ Add Katherine Lutz to the groups for content submission and edit steps of the CG "url": "https://alanorth.github.io/cgspace-notes/2017-10/", "wordCount": "2613", "datePublished": "2017-10-01T08:07:54+03:00", - "dateModified": "2017-10-01T08:07:54+03:00", + "dateModified": "2017-11-02T16:13:10+02:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2017-11/index.html b/public/2017-11/index.html index 3cd6de491..292a8635e 100644 --- a/public/2017-11/index.html +++ b/public/2017-11/index.html @@ -38,7 +38,7 @@ COPY 54701 - + @@ -88,7 +88,7 @@ COPY 54701 "url": "https://alanorth.github.io/cgspace-notes/2017-11/", "wordCount": "5428", "datePublished": "2017-11-02T09:37:54+02:00", - "dateModified": "2017-11-02T09:37:54+02:00", + "dateModified": "2017-12-17T10:43:56+02:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2017-12/index.html b/public/2017-12/index.html index e7d7ae7da..e202d3097 100644 --- a/public/2017-12/index.html +++ b/public/2017-12/index.html @@ -23,7 +23,7 @@ The list of connections to XMLUI and REST API for today: - + @@ -58,7 +58,7 @@ The list of connections to XMLUI and REST API for today: "url": "https://alanorth.github.io/cgspace-notes/2017-12/", "wordCount": "4088", "datePublished": "2017-12-01T13:53:54+03:00", - "dateModified": "2017-12-01T13:53:54+03:00", + "dateModified": "2017-12-31T10:42:16-08:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/2018-01/index.html b/public/2018-01/index.html index b53af2ce5..7c22b794e 100644 --- a/public/2018-01/index.html +++ b/public/2018-01/index.html @@ -11,7 +11,18 @@ Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time I didn’t get any load alerts from Linode and the REST and XMLUI logs don’t show anything out of the ordinary -So I don’t know WHY Uptime Robot thought it was down so many times +The nginx logs show HTTP 200s until 02/Jan/2018:11:27:17 +0000 when Uptime Robot got an HTTP 500 +In dspace.log around that time I see many errors like “Client closed the connection before file download was complete” +And just before that I see this: + + +Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000]. + + + +Ah hah! So the pool was actually empty! +I need to increase that, let’s try to bump it up from 50 to 75 +After that one client got an HTTP 499 but then the rest were HTTP 200, so I don’t know what the hell Uptime Robot saw I notice this error quite a few times in dspace.log: @@ -81,7 +92,7 @@ Danny wrote to ask for help renewing the wildcard ilri.org certificate and I adv - + @@ -99,7 +110,18 @@ Danny wrote to ask for help renewing the wildcard ilri.org certificate and I adv Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time I didn’t get any load alerts from Linode and the REST and XMLUI logs don’t show anything out of the ordinary -So I don’t know WHY Uptime Robot thought it was down so many times +The nginx logs show HTTP 200s until 02/Jan/2018:11:27:17 +0000 when Uptime Robot got an HTTP 500 +In dspace.log around that time I see many errors like “Client closed the connection before file download was complete” +And just before that I see this: + + +Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000]. + + + +Ah hah! So the pool was actually empty! +I need to increase that, let’s try to bump it up from 50 to 75 +After that one client got an HTTP 499 but then the rest were HTTP 200, so I don’t know what the hell Uptime Robot saw I notice this error quite a few times in dspace.log: @@ -172,9 +194,9 @@ Danny wrote to ask for help renewing the wildcard ilri.org certificate and I adv "@type": "BlogPosting", "headline": "January, 2018", "url": "https://alanorth.github.io/cgspace-notes/2018-01/", - "wordCount": "186", + "wordCount": "282", "datePublished": "2018-01-02T08:35:54-08:00", - "dateModified": "2018-01-02T08:35:54-08:00", + "dateModified": "2018-01-02T08:52:14-08:00", "author": { "@type": "Person", "name": "Alan Orth" @@ -242,7 +264,18 @@ Danny wrote to ask for help renewing the wildcard ilri.org certificate and I adv + +
Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000].
+
+ + diff --git a/public/cgiar-library-migration/index.html b/public/cgiar-library-migration/index.html index dab71b0d7..3ff130f57 100644 --- a/public/cgiar-library-migration/index.html +++ b/public/cgiar-library-migration/index.html @@ -14,7 +14,7 @@ - + @@ -40,7 +40,7 @@ "url": "https://alanorth.github.io/cgspace-notes/cgiar-library-migration/", "wordCount": "1278", "datePublished": "2017-09-18T16:38:35+03:00", - "dateModified": "2017-09-18T16:38:35+03:00", + "dateModified": "2017-09-28T12:00:49+03:00", "author": { "@type": "Person", "name": "Alan Orth" diff --git a/public/index.html b/public/index.html index 862323eb9..678b74453 100644 --- a/public/index.html +++ b/public/index.html @@ -111,7 +111,18 @@ + +
Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000].
+
+ + diff --git a/public/index.xml b/public/index.xml index 4a75fd916..d81fb3a8f 100644 --- a/public/index.xml +++ b/public/index.xml @@ -22,7 +22,18 @@ <ul> <li>Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time</li> <li>I didn&rsquo;t get any load alerts from Linode and the REST and XMLUI logs don&rsquo;t show anything out of the ordinary</li> -<li>So I don&rsquo;t know WHY Uptime Robot thought it was down so many times</li> +<li>The nginx logs show HTTP 200s until <code>02/Jan/2018:11:27:17 +0000</code> when Uptime Robot got an HTTP 500</li> +<li>In dspace.log around that time I see many errors like &ldquo;Client closed the connection before file download was complete&rdquo;</li> +<li>And just before that I see this:</li> +</ul> + +<pre><code>Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000]. +</code></pre> + +<ul> +<li>Ah hah! So the pool was actually empty!</li> +<li>I need to increase that, let&rsquo;s try to bump it up from 50 to 75</li> +<li>After that one client got an HTTP 499 but then the rest were HTTP 200, so I don&rsquo;t know what the hell Uptime Robot saw</li> <li>I notice this error quite a few times in dspace.log:</li> </ul> diff --git a/public/post/index.html b/public/post/index.html index 063abeca8..74aa25081 100644 --- a/public/post/index.html +++ b/public/post/index.html @@ -111,7 +111,18 @@ + +
Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000].
+
+ + diff --git a/public/post/index.xml b/public/post/index.xml index fc6dbddfc..f709a18c5 100644 --- a/public/post/index.xml +++ b/public/post/index.xml @@ -22,7 +22,18 @@ <ul> <li>Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time</li> <li>I didn&rsquo;t get any load alerts from Linode and the REST and XMLUI logs don&rsquo;t show anything out of the ordinary</li> -<li>So I don&rsquo;t know WHY Uptime Robot thought it was down so many times</li> +<li>The nginx logs show HTTP 200s until <code>02/Jan/2018:11:27:17 +0000</code> when Uptime Robot got an HTTP 500</li> +<li>In dspace.log around that time I see many errors like &ldquo;Client closed the connection before file download was complete&rdquo;</li> +<li>And just before that I see this:</li> +</ul> + +<pre><code>Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000]. +</code></pre> + +<ul> +<li>Ah hah! So the pool was actually empty!</li> +<li>I need to increase that, let&rsquo;s try to bump it up from 50 to 75</li> +<li>After that one client got an HTTP 499 but then the rest were HTTP 200, so I don&rsquo;t know what the hell Uptime Robot saw</li> <li>I notice this error quite a few times in dspace.log:</li> </ul> diff --git a/public/sitemap.xml b/public/sitemap.xml index 5366134e5..8f00b632f 100644 --- a/public/sitemap.xml +++ b/public/sitemap.xml @@ -4,147 +4,147 @@ https://alanorth.github.io/cgspace-notes/2018-01/ - 2018-01-02T08:35:54-08:00 + 2018-01-02T08:52:14-08:00 https://alanorth.github.io/cgspace-notes/2017-12/ - 2017-12-01T13:53:54+03:00 + 2017-12-31T10:42:16-08:00 https://alanorth.github.io/cgspace-notes/2017-11/ - 2017-11-02T09:37:54+02:00 + 2017-12-17T10:43:56+02:00 https://alanorth.github.io/cgspace-notes/2017-10/ - 2017-10-01T08:07:54+03:00 + 2017-11-02T16:13:10+02:00 https://alanorth.github.io/cgspace-notes/cgiar-library-migration/ - 2017-09-18T16:38:35+03:00 + 2017-09-28T12:00:49+03:00 https://alanorth.github.io/cgspace-notes/2017-09/ - 2017-09-07T16:54:52+07:00 + 2017-09-28T07:56:11+03:00 https://alanorth.github.io/cgspace-notes/2017-08/ - 2017-08-01T11:51:52+03:00 + 2017-09-10T19:18:52+03:00 https://alanorth.github.io/cgspace-notes/2017-07/ - 2017-07-01T18:03:52+03:00 + 2017-08-01T08:55:37+03:00 https://alanorth.github.io/cgspace-notes/2017-06/ - 2017-06-01T10:14:52+03:00 + 2017-06-30T18:34:51+03:00 https://alanorth.github.io/cgspace-notes/2017-05/ - 2017-05-01T16:21:52+02:00 + 2017-09-10T17:46:54+03:00 https://alanorth.github.io/cgspace-notes/2017-04/ - 2017-04-02T17:08:52+02:00 + 2017-04-26T13:35:10+03:00 https://alanorth.github.io/cgspace-notes/2017-03/ - 2017-03-01T17:08:52+02:00 + 2017-03-31T05:36:10+03:00 https://alanorth.github.io/cgspace-notes/2017-02/ - 2017-02-07T07:04:52-08:00 + 2017-02-28T22:58:29+02:00 https://alanorth.github.io/cgspace-notes/2017-01/ - 2017-01-02T10:43:00+03:00 + 2017-01-29T13:18:32+02:00 https://alanorth.github.io/cgspace-notes/2016-12/ - 2016-12-02T10:43:00+03:00 + 2017-09-19T16:07:20+03:00 https://alanorth.github.io/cgspace-notes/2016-11/ - 2016-11-01T09:21:00+03:00 + 2017-01-10T16:21:47+02:00 https://alanorth.github.io/cgspace-notes/2016-10/ - 2016-10-03T15:53:00+03:00 + 2017-01-10T16:21:47+02:00 https://alanorth.github.io/cgspace-notes/2016-09/ - 2016-09-01T15:53:00+03:00 + 2017-01-09T16:18:07+02:00 https://alanorth.github.io/cgspace-notes/2016-08/ - 2016-08-01T15:53:00+03:00 + 2017-01-09T16:18:07+02:00 https://alanorth.github.io/cgspace-notes/2016-07/ - 2016-07-01T10:53:00+03:00 + 2017-01-09T16:18:07+02:00 https://alanorth.github.io/cgspace-notes/2016-06/ - 2016-06-01T10:53:00+03:00 + 2017-01-09T16:18:07+02:00 https://alanorth.github.io/cgspace-notes/2016-05/ - 2016-05-01T23:06:00+03:00 + 2017-01-09T16:18:07+02:00 https://alanorth.github.io/cgspace-notes/2016-04/ - 2016-04-04T11:06:00+03:00 + 2016-09-28T17:02:30+03:00 https://alanorth.github.io/cgspace-notes/2016-03/ - 2016-03-02T16:50:00+03:00 + 2017-01-09T16:18:07+02:00 https://alanorth.github.io/cgspace-notes/2016-02/ - 2016-02-05T13:18:00+03:00 + 2017-01-09T16:18:07+02:00 https://alanorth.github.io/cgspace-notes/2016-01/ - 2016-01-13T13:18:00+03:00 + 2017-01-09T16:18:07+02:00 https://alanorth.github.io/cgspace-notes/2015-12/ - 2015-12-02T13:18:00+03:00 + 2017-01-09T16:18:07+02:00 https://alanorth.github.io/cgspace-notes/2015-11/ - 2015-11-23T17:00:57+03:00 + 2016-09-28T17:02:30+03:00 https://alanorth.github.io/cgspace-notes/ - 2018-01-02T08:35:54-08:00 + 2018-01-02T08:52:14-08:00 0 @@ -155,25 +155,25 @@ https://alanorth.github.io/cgspace-notes/tags/notes/ - 2018-01-02T08:35:54-08:00 + 2018-01-02T08:52:14-08:00 0 https://alanorth.github.io/cgspace-notes/categories/notes/ - 2017-09-18T16:38:35+03:00 + 2017-09-28T12:00:49+03:00 0 https://alanorth.github.io/cgspace-notes/post/ - 2018-01-02T08:35:54-08:00 + 2018-01-02T08:52:14-08:00 0 https://alanorth.github.io/cgspace-notes/tags/ - 2018-01-02T08:35:54-08:00 + 2018-01-02T08:52:14-08:00 0 diff --git a/public/tags/notes/index.html b/public/tags/notes/index.html index 2d96bf237..31767f35f 100644 --- a/public/tags/notes/index.html +++ b/public/tags/notes/index.html @@ -111,7 +111,18 @@ + +
Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000].
+
+ + diff --git a/public/tags/notes/index.xml b/public/tags/notes/index.xml index 3d2144465..b0b852ac5 100644 --- a/public/tags/notes/index.xml +++ b/public/tags/notes/index.xml @@ -22,7 +22,18 @@ <ul> <li>Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time</li> <li>I didn&rsquo;t get any load alerts from Linode and the REST and XMLUI logs don&rsquo;t show anything out of the ordinary</li> -<li>So I don&rsquo;t know WHY Uptime Robot thought it was down so many times</li> +<li>The nginx logs show HTTP 200s until <code>02/Jan/2018:11:27:17 +0000</code> when Uptime Robot got an HTTP 500</li> +<li>In dspace.log around that time I see many errors like &ldquo;Client closed the connection before file download was complete&rdquo;</li> +<li>And just before that I see this:</li> +</ul> + +<pre><code>Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000]. +</code></pre> + +<ul> +<li>Ah hah! So the pool was actually empty!</li> +<li>I need to increase that, let&rsquo;s try to bump it up from 50 to 75</li> +<li>After that one client got an HTTP 499 but then the rest were HTTP 200, so I don&rsquo;t know what the hell Uptime Robot saw</li> <li>I notice this error quite a few times in dspace.log:</li> </ul>