2015-12-02 12:25:34 +01:00
<!DOCTYPE html>
< html lang = "en-us" >
2016-02-07 20:33:55 +01:00
< head >
< meta charset = "utf-8" >
< meta name = "description" content = "" >
< meta name = "keywords" content = "" >
< meta name = "author" content = "Alan Orth" >
< meta name = "generator" content = "Hugo 0.16-DEV" / >
< meta name = "viewport" content = "width=device-width, initial-scale=1" >
< link rel = "stylesheet" href = "/css/style.css" type = "text/css" >
< link rel = "stylesheet" href = "https://fonts.googleapis.com/css?family=Source+Code+Pro:400,700" type = "text/css" >
< link rel = "alternate" href = "/index.xml" type = "application/rss+xml" title = "CGSpace Notes" >
< title > December, 2015 - CGSpace Notes< / title >
2015-12-02 12:25:34 +01:00
< / head >
< body >
2016-02-07 20:33:55 +01:00
< header >
< div class = "container" >
< a class = "path" href = "/cgspace-notes/" > [CGSpace Notes]< / a >
< span class = "caret" > # _< / span >
< / div >
< / header >
< div class = "container" >
< main role = "main" class = "article" >
2015-12-02 12:25:34 +01:00
2016-02-07 20:33:55 +01:00
< article class = "single" itemscope itemtype = "http://schema.org/BlogPosting" >
< div class = "meta" >
< span class = "key" > published on< / span >
< span class = "val" > < time itemprop = "datePublished" datetime = "2015-12-02" > December 02, 2015< / time > < / span >
< br >
< span class = "key" > tags:< / span >
< span class = "val" >
< a href = "/cgspace-notes/tags/notes" > notes< / a >
2015-12-02 12:25:34 +01:00
2016-02-07 20:33:55 +01:00
< / span >
< / div >
< h1 class = "headline" itemprop = "headline" > December, 2015< / h1 >
< section class = "body" itemprop = "articleBody" >
2015-12-02 12:25:34 +01:00
< h2 id = "2015-12-02:012a628feed6d64ae1151cbd6151ccd6" > 2015-12-02< / h2 >
< ul >
< li > Replace < code > lzop< / code > with < code > xz< / code > in log compression cron jobs on DSpace Test—it uses less space:< / li >
< / ul >
< pre > < code > # cd /home/dspacetest.cgiar.org/log
# ls -lh dspace.log.2015-11-18*
-rw-rw-r-- 1 tomcat7 tomcat7 2.0M Nov 18 23:59 dspace.log.2015-11-18
-rw-rw-r-- 1 tomcat7 tomcat7 387K Nov 18 23:59 dspace.log.2015-11-18.lzo
-rw-rw-r-- 1 tomcat7 tomcat7 169K Nov 18 23:59 dspace.log.2015-11-18.xz
< / code > < / pre >
< ul >
< li > I had used lrzip once, but it needs more memory and is harder to use as it requires the lrztar wrapper< / li >
< li > Need to remember to go check if everything is ok in a few days and then change CGSpace< / li >
2015-12-02 18:16:44 +01:00
< li > CGSpace went down again (due to PostgreSQL idle connections of course)< / li >
< li > Current database settings for DSpace are < code > db.maxconnections = 30< / code > and < code > db.maxidle = 8< / code > , yet idle connections are exceeding this:< / li >
< / ul >
< pre > < code > $ psql -c 'SELECT * from pg_stat_activity;' | grep cgspace | grep -c idle
39
< / code > < / pre >
< ul >
< li > I restarted PostgreSQL and Tomcat and it’ s back< / li >
< li > On a related note of why CGSpace is so slow, I decided to finally try the < code > pgtune< / code > script to tune the postgres settings:< / li >
< / ul >
< pre > < code > # apt-get install pgtune
# pgtune -i /etc/postgresql/9.3/main/postgresql.conf -o postgresql.conf-pgtune
# mv /etc/postgresql/9.3/main/postgresql.conf /etc/postgresql/9.3/main/postgresql.conf.orig
# mv postgresql.conf-pgtune /etc/postgresql/9.3/main/postgresql.conf
< / code > < / pre >
< ul >
< li > It introduced the following new settings:< / li >
< / ul >
< pre > < code > default_statistics_target = 50
maintenance_work_mem = 480MB
constraint_exclusion = on
checkpoint_completion_target = 0.9
effective_cache_size = 5632MB
work_mem = 48MB
wal_buffers = 8MB
checkpoint_segments = 16
shared_buffers = 1920MB
max_connections = 80
< / code > < / pre >
< ul >
< li > Now I need to go read PostgreSQL docs about these options, and watch memory settings in munin etc< / li >
2015-12-02 20:11:28 +01:00
< li > For what it’ s worth, now the REST API should be faster (because of these PostgreSQL tweaks):< / li >
< / ul >
< pre > < code > $ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
1.474
$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
2.141
$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
1.685
$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
1.995
$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
1.786
< / code > < / pre >
< ul >
< li > Last week it was an average of 8 seconds… now this is < sup > 1< / sup > ⁄ < sub > 4< / sub > of that< / li >
2015-12-02 21:19:56 +01:00
< li > CCAFS noticed that one of their items displays only the Atmire statlets: < a href = "https://cgspace.cgiar.org/handle/10568/42445" > https://cgspace.cgiar.org/handle/10568/42445< / a > < / li >
< / ul >
2015-12-05 16:44:11 +01:00
< p > < img src = "../images/2015/12/ccafs-item-no-metadata.png" alt = "CCAFS item" / > < / p >
2015-12-02 21:19:56 +01:00
< ul >
< li > The authorizations for the item are all public READ, and I don’ t see any errors in dspace.log when browsing that item< / li >
< li > I filed a ticket on Atmire’ s issue tracker< / li >
< li > I also filed a ticket on Atmire’ s issue tracker for the PostgreSQL stuff< / li >
2015-12-03 10:08:14 +01:00
< / ul >
< h2 id = "2015-12-03:012a628feed6d64ae1151cbd6151ccd6" > 2015-12-03< / h2 >
< ul >
< li > CGSpace very slow, and monitoring emailing me to say its down, even though I can load the page (very slowly)< / li >
< li > Idle postgres connections look like this (with no change in DSpace db settings lately):< / li >
< / ul >
< pre > < code > $ psql -c 'SELECT * from pg_stat_activity;' | grep cgspace | grep -c idle
29
< / code > < / pre >
< ul >
< li > I restarted Tomcat and postgres… < / li >
2015-12-03 23:08:49 +01:00
< li > Atmire commented that we should raise the JVM heap size by ~500M, so it is now < code > -Xms3584m -Xmx3584m< / code > < / li >
< li > We weren’ t out of heap yet, but it’ s probably fair enough that the DSpace 5 upgrade (and new Atmire modules) requires more memory so it’ s ok< / li >
2015-12-03 23:09:41 +01:00
< li > A possible side effect is that I see that the REST API is twice as fast for the request above now:< / li >
2015-12-02 12:25:34 +01:00
< / ul >
2015-12-03 23:09:41 +01:00
< pre > < code > $ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
1.368
$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
0.968
$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
1.006
$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
0.849
$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
0.806
$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
0.854
< / code > < / pre >
2015-12-05 16:42:56 +01:00
< h2 id = "2015-12-05:012a628feed6d64ae1151cbd6151ccd6" > 2015-12-05< / h2 >
< ul >
< li > CGSpace has been up and down all day and REST API is completely unresponsive< / li >
< li > PostgreSQL idle connections are currently:< / li >
< / ul >
< pre > < code > postgres@linode01:~$ psql -c 'SELECT * from pg_stat_activity;' | grep cgspace | grep -c idle
28
< / code > < / pre >
< ul >
< li > I have reverted all the pgtune tweaks from the other day, as they didn’ t fix the stability issues, so I’ d rather not have them introducing more variables into the equation< / li >
< li > The PostgreSQL stats from Munin all point to something database-related with the DSpace 5 upgrade around mid– < / li >
< / ul >
< p > < img src = "../images/2015/12/postgres_bgwriter-year.png" alt = "PostgreSQL bgwriter (year)" / >
< img src = "../images/2015/12/postgres_cache_cgspace-year.png" alt = "PostgreSQL cache (year)" / >
< img src = "../images/2015/12/postgres_locks_cgspace-year.png" alt = "PostgreSQL locks (year)" / >
2015-12-05 16:44:36 +01:00
< img src = "../images/2015/12/postgres_scans_cgspace-year.png" alt = "PostgreSQL scans (year)" / > < / p >
2015-12-05 16:42:56 +01:00
2015-12-07 18:10:54 +01:00
< h2 id = "2015-12-07:012a628feed6d64ae1151cbd6151ccd6" > 2015-12-07< / h2 >
< ul >
< li > Atmire sent < a href = "https://github.com/ilri/DSpace/pull/161" > some fixes< / a > to DSpace’ s REST API code that was leaving contexts open (causing the slow performance and database issues)< / li >
< li > After deploying the fix to CGSpace the REST API is consistently faster:< / li >
< / ul >
< pre > < code > $ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
0.675
$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
0.599
$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
0.588
$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
0.566
$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
0.497
< / code > < / pre >
2015-12-08 11:01:25 +01:00
< h2 id = "2015-12-08:012a628feed6d64ae1151cbd6151ccd6" > 2015-12-08< / h2 >
< ul >
< li > Switch CGSpace log compression cron jobs from using lzop to xz—the compression isn’ t as good, but it’ s much faster and causes less IO/CPU load< / li >
2015-12-08 20:49:06 +01:00
< li > Since we figured out (and fixed) the cause of the performance issue, I reverted Google Bot’ s crawl rate to the “ Let Google optimize” setting< / li >
2015-12-08 11:01:25 +01:00
< / ul >
2015-12-02 12:25:34 +01:00
< / section >
< / article >
2016-02-07 20:33:55 +01:00
< / main >
2015-12-02 12:25:34 +01:00
< / div >
2016-02-07 20:33:55 +01:00
< footer >
< div class = "container" >
< span class = "copyright" > © 2015 CGSpace Notes - < a rel = "license" href = "http://creativecommons.org/licenses/by/4.0/" > CC BY 4.0< / a > < / span >
< / div >
< / footer >
2015-12-02 12:25:34 +01:00
< / body >
< / html >
