2015-11-30 11:51:37 +01:00
<!DOCTYPE html>
2016-09-21 14:24:28 +02:00
< html lang = "en" >
2016-11-24 14:17:06 +01:00
< head >
< meta charset = "utf-8" >
< meta name = "viewport" content = "width=device-width, initial-scale=1, shrink-to-fit=no" >
2016-09-21 14:24:28 +02:00
2016-11-24 14:17:06 +01:00
< meta property = "og:title" content = "November, 2015" / >
2016-11-14 08:27:03 +01:00
< meta property = "og:description" content = "2015-11-22
CGSpace went down
Looks like DSpace exhausted its PostgreSQL connection pool
Last week I had increased the limit from 30 to 60, which seemed to help, but now there are many more idle connections:
$ psql -c ' SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
78
" />
< meta property = "og:type" content = "article" / >
< meta property = "og:url" content = "https://alanorth.github.io/cgspace-notes/2015-11/" / >
2017-01-27 12:03:13 +01:00
< meta property = "article:published_time" content = "2015-11-23T17:00:57+03:00" / >
2017-10-01 07:13:31 +02:00
< meta property = "article:modified_time" content = "2015-11-23T17:00:57+03:00" / >
2017-01-27 12:03:13 +01:00
2016-11-14 08:27:03 +01:00
2016-09-21 14:24:28 +02:00
2017-08-08 11:04:22 +02:00
< meta name = "twitter:card" content = "summary" / > < meta name = "twitter:title" content = "November, 2015" / >
2016-11-14 08:27:03 +01:00
< meta name = "twitter:description" content = "2015-11-22
CGSpace went down
Looks like DSpace exhausted its PostgreSQL connection pool
Last week I had increased the limit from 30 to 60, which seemed to help, but now there are many more idle connections:
$ psql -c ' SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
78
"/>
2017-09-27 11:22:52 +02:00
< meta name = "generator" content = "Hugo 0.29" / >
2016-11-14 08:27:03 +01:00
2016-09-21 14:24:28 +02:00
2017-01-22 10:11:46 +01:00
< script type = "application/ld+json" >
{
"@context": "http://schema.org",
"@type": "BlogPosting",
"headline": "November, 2015",
"url": "https://alanorth.github.io/cgspace-notes/2015-11/",
"wordCount": "798",
2017-02-27 20:30:13 +01:00
"datePublished": "2015-11-23T17:00:57+ 03:00",
2017-10-01 07:13:31 +02:00
"dateModified": "2015-11-23T17:00:57+ 03:00",
2017-01-22 10:11:46 +01:00
"author": {
"@type": "Person",
"name": "Alan Orth"
2017-03-30 13:51:57 +02:00
},
2017-01-22 10:11:46 +01:00
"keywords": "Notes"
}
< / script >
2016-10-14 23:13:52 +02:00
2016-09-21 14:24:28 +02:00
< link rel = "canonical" href = "https://alanorth.github.io/cgspace-notes/2015-11/" >
< title > November, 2015 | CGSpace Notes< / title >
<!-- combined, minified CSS -->
2017-08-14 11:41:06 +02:00
< link href = "https://alanorth.github.io/cgspace-notes/css/style.css" rel = "stylesheet" integrity = "sha384-zYRhIy0/Yl1e5lW9cimY1AugfdkHChXyCbs2NKFaLTgeQjVfj/CMPIUdjXm/JPWV" crossorigin = "anonymous" >
2016-09-21 14:24:28 +02:00
2016-11-14 08:27:03 +01:00
2016-11-24 14:17:06 +01:00
2016-09-21 14:24:28 +02:00
< / head >
2016-11-24 14:17:06 +01:00
< body >
2016-09-21 14:24:28 +02:00
2016-11-24 14:17:06 +01:00
< div class = "blog-masthead" >
< div class = "container" >
< nav class = "nav blog-nav" >
< a class = "nav-link " href = "https://alanorth.github.io/cgspace-notes/" > Home< / a >
2016-11-17 14:59:59 +01:00
2016-11-24 14:17:06 +01:00
< / nav >
< / div >
2016-09-21 14:24:28 +02:00
< / div >
2016-11-24 14:17:06 +01:00
< header class = "blog-header" >
< div class = "container" >
< h1 class = "blog-title" > < a href = "https://alanorth.github.io/cgspace-notes/" rel = "home" > CGSpace Notes< / a > < / h1 >
2017-01-22 10:11:46 +01:00
< p class = "lead blog-description" > Documenting day-to-day work on the < a href = "https://cgspace.cgiar.org" > CGSpace< / a > repository.< / p >
2016-11-24 14:17:06 +01:00
< / div >
< / header >
2016-09-21 14:24:28 +02:00
2016-11-24 14:17:06 +01:00
< div class = "container" >
< div class = "row" >
< div class = "col-sm-8 blog-main" >
2016-09-21 14:24:28 +02:00
2016-11-24 14:17:06 +01:00
2016-11-14 08:27:03 +01:00
2016-11-24 14:17:06 +01:00
< article class = "blog-post" >
< header >
< h2 class = "blog-post-title" > < a href = "https://alanorth.github.io/cgspace-notes/2015-11/" > November, 2015< / a > < / h2 >
< p class = "blog-post-meta" > < time datetime = "2015-11-23T17:00:57+03:00" > Mon Nov 23, 2015< / time > by Alan Orth in
2016-09-27 22:54:30 +02:00
2016-11-24 14:17:06 +01:00
< i class = "fa fa-tag" aria-hidden = "true" > < / i > < a href = "/cgspace-notes/tags/notes" rel = "tag" > Notes< / a >
2016-09-27 22:54:30 +02:00
< / p >
2016-11-24 14:17:06 +01:00
< / header >
< h2 id = "2015-11-22" > 2015-11-22< / h2 >
2015-11-30 11:51:37 +01:00
< ul >
< li > CGSpace went down< / li >
< li > Looks like DSpace exhausted its PostgreSQL connection pool< / li >
< li > Last week I had increased the limit from 30 to 60, which seemed to help, but now there are many more idle connections:< / li >
< / ul >
< pre > < code > $ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
78
< / code > < / pre >
2016-10-03 17:28:33 +02:00
< p > < / p >
2015-11-30 11:51:37 +01:00
< ul >
< li > For now I have increased the limit from 60 to 90, run updates, and rebooted the server< / li >
< / ul >
2016-08-03 09:09:36 +02:00
< h2 id = "2015-11-24" > 2015-11-24< / h2 >
2015-11-30 11:51:37 +01:00
< ul >
< li > CGSpace went down again< / li >
< li > Getting emails from uptimeRobot and uptimeButler that it’ s down, and Google Webmaster Tools is sending emails that there is an increase in crawl errors< / li >
< li > Looks like there are still a bunch of idle PostgreSQL connections:< / li >
< / ul >
< pre > < code > $ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
96
< / code > < / pre >
< ul >
< li > For some reason the number of idle connections is very high since we upgraded to DSpace 5< / li >
< / ul >
2016-08-03 09:09:36 +02:00
< h2 id = "2015-11-25" > 2015-11-25< / h2 >
2015-11-30 11:51:37 +01:00
< ul >
< li > Troubleshoot the DSpace 5 OAI breakage caused by nginx routing config< / li >
< li > The OAI application requests stylesheets and javascript files with the path < code > /oai/static/css< / code > , which gets matched here:< / li >
< / ul >
< pre > < code > # static assets we can load from the file system directly with nginx
location ~ /(themes|static|aspects/ReportingSuite) {
try_files $uri @tomcat;
...
< / code > < / pre >
< ul >
< li > The document root is relative to the xmlui app, so this gets a 404—I’ m not sure why it doesn’ t pass to < code > @tomcat< / code > < / li >
< li > Anyways, I can’ t find any URIs with path < code > /static< / code > , and the more important point is to handle all the static theme assets, so we can just remove < code > static< / code > from the regex for now (who cares if we can’ t use nginx to send Etags for OAI CSS!)< / li >
< li > Also, I noticed we aren’ t setting CSP headers on the static assets, because in nginx headers are inherited in child blocks, but if you use < code > add_header< / code > in a child block it doesn’ t inherit the others< / li >
< li > We simply need to add < code > include extra-security.conf;< / code > to the above location block (but research and test first)< / li >
< li > We should add WOFF assets to the list of things to set expires for:< / li >
< / ul >
< pre > < code > location ~* \.(?:ico|css|js|gif|jpe?g|png|woff)$ {
< / code > < / pre >
< ul >
< li > We should also add < code > aspects/Statistics< / code > to the location block for static assets (minus < code > static< / code > from above):< / li >
< / ul >
< pre > < code > location ~ /(themes|aspects/ReportingSuite|aspects/Statistics) {
< / code > < / pre >
< ul >
< li > Need to check < code > /about< / code > on CGSpace, as it’ s blank on my local test server and we might need to add something there< / li >
< li > CGSpace has been up and down all day due to PostgreSQL idle connections (current DSpace pool is 90):< / li >
< / ul >
< pre > < code > $ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
93
< / code > < / pre >
< ul >
< li > I looked closer at the idle connections and saw that many have been idle for hours (current time on server is < code > 2015-11-25T20:20:42+0000< / code > ):< / li >
< / ul >
< pre > < code > $ psql -c 'SELECT * from pg_stat_activity;' | less -S
datid | datname | pid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | xact_start |
-------+----------+-------+----------+----------+------------------+-------------+-----------------+-------------+-------------------------------+-------------------------------+---
20951 | cgspace | 10966 | 18205 | cgspace | | 127.0.0.1 | | 37731 | 2015-11-25 13:13:02.837624+00 | | 20
20951 | cgspace | 10967 | 18205 | cgspace | | 127.0.0.1 | | 37737 | 2015-11-25 13:13:03.069421+00 | | 20
...
< / code > < / pre >
< ul >
< li > There is a relevant Jira issue about this: < a href = "https://jira.duraspace.org/browse/DS-1458" > https://jira.duraspace.org/browse/DS-1458< / a > < / li >
< li > It seems there is some sense changing DSpace’ s default < code > db.maxidle< / code > from unlimited (-1) to something like 8 (Tomcat default) or 10 (Confluence default)< / li >
< li > Change < code > db.maxidle< / code > from -1 to 10, reduce < code > db.maxconnections< / code > from 90 to 50, and restart postgres and tomcat7< / li >
< li > Also redeploy DSpace Test with a clean sync of CGSpace and mirror these database settings there as well< / li >
< li > Also deploy the nginx fixes for the < code > try_files< / code > location block as well as the expires block< / li >
< / ul >
2016-08-03 09:09:36 +02:00
< h2 id = "2015-11-26" > 2015-11-26< / h2 >
2015-11-30 11:51:37 +01:00
< ul >
< li > CGSpace behaving much better since changing < code > db.maxidle< / code > yesterday, but still two up/down notices from monitoring this morning (better than 50!)< / li >
< li > CCAFS colleagues mentioned that the REST API is very slow, 24 seconds for one item< / li >
< li > Not as bad for me, but still unsustainable if you have to get many:< / li >
< / ul >
< pre > < code > $ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
8.415
< / code > < / pre >
< ul >
< li > Monitoring e-mailed in the evening to say CGSpace was down< / li >
< li > Idle connections in PostgreSQL again:< / li >
< / ul >
< pre > < code > $ psql -c 'SELECT * from pg_stat_activity;' | grep cgspace | grep -c idle
66
< / code > < / pre >
< ul >
< li > At the time, the current DSpace pool size was 50… < / li >
< li > I reduced the pool back to the default of 30, and reduced the < code > db.maxidle< / code > settings from 10 to 8< / li >
< / ul >
2016-08-03 09:09:36 +02:00
< h2 id = "2015-11-29" > 2015-11-29< / h2 >
2015-11-30 11:51:37 +01:00
< ul >
< li > Still more alerts that CGSpace has been up and down all day< / li >
< li > Current database settings for DSpace:< / li >
< / ul >
< pre > < code > db.maxconnections = 30
db.maxwait = 5000
db.maxidle = 8
db.statementpool = true
< / code > < / pre >
< ul >
< li > And idle connections:< / li >
< / ul >
< pre > < code > $ psql -c 'SELECT * from pg_stat_activity;' | grep cgspace | grep -c idle
49
< / code > < / pre >
< ul >
< li > Perhaps I need to start drastically increasing the connection limits—like to 300—to see if DSpace’ s thirst can ever be quenched< / li >
< li > On another note, SUNScholar’ s notes suggest adjusting some other postgres variables: < a href = "http://wiki.lib.sun.ac.za/index.php/SUNScholar/Optimisations/Database" > http://wiki.lib.sun.ac.za/index.php/SUNScholar/Optimisations/Database< / a > < / li >
< li > This might help with REST API speed (which I mentioned above and still need to do real tests)< / li >
< / ul >
2016-11-14 08:27:03 +01:00
2016-11-24 14:17:06 +01:00
2017-01-08 16:08:08 +01:00
2016-11-24 14:17:06 +01:00
2017-01-08 16:08:08 +01:00
< / article >
2016-11-14 08:27:03 +01:00
2016-09-21 14:24:28 +02:00
2016-11-24 14:17:06 +01:00
< / div > <!-- /.blog - main -->
2016-09-21 14:24:28 +02:00
2017-08-14 11:41:06 +02:00
< aside class = "col-sm-3 ml-auto blog-sidebar" >
2016-09-21 14:24:28 +02:00
2017-03-12 12:41:42 +01:00
< section class = "sidebar-module" >
2016-09-21 14:24:28 +02:00
< h4 > Recent Posts< / h4 >
< ol class = "list-unstyled" >
2017-03-12 12:41:42 +01:00
2017-10-01 07:13:31 +02:00
< li > < a href = "/cgspace-notes/2017-10/" > October, 2017< / a > < / li >
2017-09-28 09:24:40 +02:00
< li > < a href = "/cgspace-notes/cgiar-library-migration/" > CGIAR Library Migration< / a > < / li >
2017-09-07 12:02:38 +02:00
< li > < a href = "/cgspace-notes/2017-09/" > September, 2017< / a > < / li >
2017-08-01 10:57:37 +02:00
< li > < a href = "/cgspace-notes/2017-08/" > August, 2017< / a > < / li >
2017-07-01 17:03:19 +02:00
< li > < a href = "/cgspace-notes/2017-07/" > July, 2017< / a > < / li >
2016-09-21 14:24:28 +02:00
< / ol >
< / section >
2016-02-08 07:59:05 +01:00
2017-01-09 15:20:52 +01:00
2016-09-21 14:24:28 +02:00
< section class = "sidebar-module" >
< h4 > Links< / h4 >
< ol class = "list-unstyled" >
< li > < a href = "https://cgspace.cgiar.org" > CGSpace< / a > < / li >
< li > < a href = "https://dspacetest.cgiar.org" > DSpace Test< / a > < / li >
< li > < a href = "https://github.com/ilri/DSpace" > CGSpace @ GitHub< / a > < / li >
< / ol >
< / section >
2016-02-08 07:59:05 +01:00
2016-09-21 14:24:28 +02:00
< / aside >
2015-11-30 11:51:37 +01:00
2016-11-24 14:17:06 +01:00
< / div > <!-- /.row -->
< / div > <!-- /.container -->
2016-09-21 14:24:28 +02:00
2016-11-24 14:17:06 +01:00
< footer class = "blog-footer" >
< p >
2016-10-14 23:13:52 +02:00
Blog template created by < a href = "https://twitter.com/mdo" > @mdo< / a > , ported to Hugo by < a href = 'https://twitter.com/mralanorth' > @mralanorth< / a > .
2016-11-24 14:17:06 +01:00
< / p >
< p >
2017-01-05 14:44:45 +01:00
< a href = "#" > Back to top< / a >
2016-11-24 14:17:06 +01:00
< / p >
< / footer >
2015-11-30 11:51:37 +01:00
2016-11-24 14:17:06 +01:00
< / body >
2015-11-30 11:51:37 +01:00
2016-09-21 14:24:28 +02:00
< / html >
