mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2025-01-27 05:49:12 +01:00
Add notes for 2017-02-07
This commit is contained in:
@ -13,7 +13,7 @@
|
||||
|
||||
|
||||
|
||||
<meta property="og:updated_time" content="2017-01-02T10:43:00+03:00"/>
|
||||
<meta property="og:updated_time" content="2017-02-07T07:04:52-08:00"/>
|
||||
|
||||
|
||||
|
||||
@ -43,7 +43,7 @@
|
||||
},
|
||||
|
||||
|
||||
"dateModified": "2017-01-02T10:43:00+03:00",
|
||||
"dateModified": "2017-02-07T07:04:52-08:00",
|
||||
|
||||
|
||||
|
||||
@ -105,6 +105,39 @@
|
||||
|
||||
|
||||
|
||||
<article class="blog-post">
|
||||
<header>
|
||||
<h2 class="blog-post-title"><a href="https://alanorth.github.io/cgspace-notes/2017-02/">February, 2017</a></h2>
|
||||
<p class="blog-post-meta"><time datetime="2017-02-07T07:04:52-08:00">Tue Feb 07, 2017</time> by Alan Orth in
|
||||
|
||||
<i class="fa fa-tag" aria-hidden="true"></i> <a href="/cgspace-notes/tags/notes" rel="tag">Notes</a>
|
||||
|
||||
</p>
|
||||
</header>
|
||||
<h2 id="2017-02-07">2017-02-07</h2>
|
||||
|
||||
<ul>
|
||||
<li>An item was mapped twice erroneously again, so I had to remove one of the mappings manually:</li>
|
||||
</ul>
|
||||
|
||||
<pre><code>dspace=# select * from collection2item where item_id = '80278';
|
||||
id | collection_id | item_id
|
||||
-------+---------------+---------
|
||||
92551 | 313 | 80278
|
||||
92550 | 313 | 80278
|
||||
90774 | 1051 | 80278
|
||||
(3 rows)
|
||||
dspace=# delete from collection2item where id = 92551 and item_id = 80278;
|
||||
DELETE 1
|
||||
</code></pre>
|
||||
|
||||
<p></p>
|
||||
<a href='https://alanorth.github.io/cgspace-notes/2017-02/'>Read more →</a>
|
||||
</article>
|
||||
|
||||
|
||||
|
||||
|
||||
<article class="blog-post">
|
||||
<header>
|
||||
<h2 class="blog-post-title"><a href="https://alanorth.github.io/cgspace-notes/2017-01/">January, 2017</a></h2>
|
||||
@ -371,32 +404,6 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
|
||||
|
||||
|
||||
|
||||
<article class="blog-post">
|
||||
<header>
|
||||
<h2 class="blog-post-title"><a href="https://alanorth.github.io/cgspace-notes/2016-04/">April, 2016</a></h2>
|
||||
<p class="blog-post-meta"><time datetime="2016-04-04T11:06:00+03:00">Mon Apr 04, 2016</time> by Alan Orth in
|
||||
|
||||
<i class="fa fa-tag" aria-hidden="true"></i> <a href="/cgspace-notes/tags/notes" rel="tag">Notes</a>
|
||||
|
||||
</p>
|
||||
</header>
|
||||
<h2 id="2016-04-04">2016-04-04</h2>
|
||||
|
||||
<ul>
|
||||
<li>Looking at log file use on CGSpace and notice that we need to work on our cron setup a bit</li>
|
||||
<li>We are backing up all logs in the log folder, including useless stuff like solr, cocoon, handle-plugin, etc</li>
|
||||
<li>After running DSpace for over five years I’ve never needed to look in any other log file than dspace.log, leave alone one from last year!</li>
|
||||
<li>This will save us a few gigs of backup space we’re paying for on S3</li>
|
||||
<li>Also, I noticed the <code>checker</code> log has some errors we should pay attention to:</li>
|
||||
</ul>
|
||||
|
||||
<p></p>
|
||||
<a href='https://alanorth.github.io/cgspace-notes/2016-04/'>Read more →</a>
|
||||
</article>
|
||||
|
||||
|
||||
|
||||
|
||||
<nav class="blog-pagination">
|
||||
|
||||
|
||||
@ -421,6 +428,8 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
|
||||
<h4>Recent Posts</h4>
|
||||
<ol class="list-unstyled">
|
||||
|
||||
<li><a href="/cgspace-notes/2017-02/">February, 2017</a></li>
|
||||
|
||||
<li><a href="/cgspace-notes/2017-01/">January, 2017</a></li>
|
||||
|
||||
<li><a href="/cgspace-notes/2016-12/">December, 2016</a></li>
|
||||
@ -429,8 +438,6 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
|
||||
|
||||
<li><a href="/cgspace-notes/2016-10/">October, 2016</a></li>
|
||||
|
||||
<li><a href="/cgspace-notes/2016-09/">September, 2016</a></li>
|
||||
|
||||
</ol>
|
||||
</section>
|
||||
|
||||
|
||||
@ -6,9 +6,35 @@
|
||||
<description>Recent content in Posts on CGSpace Notes</description>
|
||||
<generator>Hugo -- gohugo.io</generator>
|
||||
<language>en-us</language>
|
||||
<lastBuildDate>Mon, 02 Jan 2017 10:43:00 +0300</lastBuildDate>
|
||||
<lastBuildDate>Tue, 07 Feb 2017 07:04:52 -0800</lastBuildDate>
|
||||
<atom:link href="https://alanorth.github.io/cgspace-notes/post/index.xml" rel="self" type="application/rss+xml" />
|
||||
|
||||
<item>
|
||||
<title>February, 2017</title>
|
||||
<link>https://alanorth.github.io/cgspace-notes/2017-02/</link>
|
||||
<pubDate>Tue, 07 Feb 2017 07:04:52 -0800</pubDate>
|
||||
|
||||
<guid>https://alanorth.github.io/cgspace-notes/2017-02/</guid>
|
||||
<description><h2 id="2017-02-07">2017-02-07</h2>
|
||||
|
||||
<ul>
|
||||
<li>An item was mapped twice erroneously again, so I had to remove one of the mappings manually:</li>
|
||||
</ul>
|
||||
|
||||
<pre><code>dspace=# select * from collection2item where item_id = '80278';
|
||||
id | collection_id | item_id
|
||||
-------+---------------+---------
|
||||
92551 | 313 | 80278
|
||||
92550 | 313 | 80278
|
||||
90774 | 1051 | 80278
|
||||
(3 rows)
|
||||
dspace=# delete from collection2item where id = 92551 and item_id = 80278;
|
||||
DELETE 1
|
||||
</code></pre>
|
||||
|
||||
<p></p></description>
|
||||
</item>
|
||||
|
||||
<item>
|
||||
<title>January, 2017</title>
|
||||
<link>https://alanorth.github.io/cgspace-notes/2017-01/</link>
|
||||
@ -5168,160 +5194,6 @@ $ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle
|
||||
<ul>
|
||||
<li>Switch CGSpace log compression cron jobs from using lzop to xz—the compression isn&rsquo;t as good, but it&rsquo;s much faster and causes less IO/CPU load</li>
|
||||
<li>Since we figured out (and fixed) the cause of the performance issue, I reverted Google Bot&rsquo;s crawl rate to the &ldquo;Let Google optimize&rdquo; setting</li>
|
||||
</ul></description>
|
||||
</item>
|
||||
|
||||
<item>
|
||||
<title>November, 2015</title>
|
||||
<link>https://alanorth.github.io/cgspace-notes/2015-11/</link>
|
||||
<pubDate>Mon, 23 Nov 2015 17:00:57 +0300</pubDate>
|
||||
|
||||
<guid>https://alanorth.github.io/cgspace-notes/2015-11/</guid>
|
||||
<description><h2 id="2015-11-22">2015-11-22</h2>
|
||||
|
||||
<ul>
|
||||
<li>CGSpace went down</li>
|
||||
<li>Looks like DSpace exhausted its PostgreSQL connection pool</li>
|
||||
<li>Last week I had increased the limit from 30 to 60, which seemed to help, but now there are many more idle connections:</li>
|
||||
</ul>
|
||||
|
||||
<pre><code>$ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
|
||||
78
|
||||
</code></pre>
|
||||
|
||||
<p></p>
|
||||
|
||||
<ul>
|
||||
<li>For now I have increased the limit from 60 to 90, run updates, and rebooted the server</li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2015-11-24">2015-11-24</h2>
|
||||
|
||||
<ul>
|
||||
<li>CGSpace went down again</li>
|
||||
<li>Getting emails from uptimeRobot and uptimeButler that it&rsquo;s down, and Google Webmaster Tools is sending emails that there is an increase in crawl errors</li>
|
||||
<li>Looks like there are still a bunch of idle PostgreSQL connections:</li>
|
||||
</ul>
|
||||
|
||||
<pre><code>$ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
|
||||
96
|
||||
</code></pre>
|
||||
|
||||
<ul>
|
||||
<li>For some reason the number of idle connections is very high since we upgraded to DSpace 5</li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2015-11-25">2015-11-25</h2>
|
||||
|
||||
<ul>
|
||||
<li>Troubleshoot the DSpace 5 OAI breakage caused by nginx routing config</li>
|
||||
<li>The OAI application requests stylesheets and javascript files with the path <code>/oai/static/css</code>, which gets matched here:</li>
|
||||
</ul>
|
||||
|
||||
<pre><code># static assets we can load from the file system directly with nginx
|
||||
location ~ /(themes|static|aspects/ReportingSuite) {
|
||||
try_files $uri @tomcat;
|
||||
...
|
||||
</code></pre>
|
||||
|
||||
<ul>
|
||||
<li>The document root is relative to the xmlui app, so this gets a 404—I&rsquo;m not sure why it doesn&rsquo;t pass to <code>@tomcat</code></li>
|
||||
<li>Anyways, I can&rsquo;t find any URIs with path <code>/static</code>, and the more important point is to handle all the static theme assets, so we can just remove <code>static</code> from the regex for now (who cares if we can&rsquo;t use nginx to send Etags for OAI CSS!)</li>
|
||||
<li>Also, I noticed we aren&rsquo;t setting CSP headers on the static assets, because in nginx headers are inherited in child blocks, but if you use <code>add_header</code> in a child block it doesn&rsquo;t inherit the others</li>
|
||||
<li>We simply need to add <code>include extra-security.conf;</code> to the above location block (but research and test first)</li>
|
||||
<li>We should add WOFF assets to the list of things to set expires for:</li>
|
||||
</ul>
|
||||
|
||||
<pre><code>location ~* \.(?:ico|css|js|gif|jpe?g|png|woff)$ {
|
||||
</code></pre>
|
||||
|
||||
<ul>
|
||||
<li>We should also add <code>aspects/Statistics</code> to the location block for static assets (minus <code>static</code> from above):</li>
|
||||
</ul>
|
||||
|
||||
<pre><code>location ~ /(themes|aspects/ReportingSuite|aspects/Statistics) {
|
||||
</code></pre>
|
||||
|
||||
<ul>
|
||||
<li>Need to check <code>/about</code> on CGSpace, as it&rsquo;s blank on my local test server and we might need to add something there</li>
|
||||
<li>CGSpace has been up and down all day due to PostgreSQL idle connections (current DSpace pool is 90):</li>
|
||||
</ul>
|
||||
|
||||
<pre><code>$ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
|
||||
93
|
||||
</code></pre>
|
||||
|
||||
<ul>
|
||||
<li>I looked closer at the idle connections and saw that many have been idle for hours (current time on server is <code>2015-11-25T20:20:42+0000</code>):</li>
|
||||
</ul>
|
||||
|
||||
<pre><code>$ psql -c 'SELECT * from pg_stat_activity;' | less -S
|
||||
datid | datname | pid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | xact_start |
|
||||
-------+----------+-------+----------+----------+------------------+-------------+-----------------+-------------+-------------------------------+-------------------------------+---
|
||||
20951 | cgspace | 10966 | 18205 | cgspace | | 127.0.0.1 | | 37731 | 2015-11-25 13:13:02.837624+00 | | 20
|
||||
20951 | cgspace | 10967 | 18205 | cgspace | | 127.0.0.1 | | 37737 | 2015-11-25 13:13:03.069421+00 | | 20
|
||||
...
|
||||
</code></pre>
|
||||
|
||||
<ul>
|
||||
<li>There is a relevant Jira issue about this: <a href="https://jira.duraspace.org/browse/DS-1458">https://jira.duraspace.org/browse/DS-1458</a></li>
|
||||
<li>It seems there is some sense changing DSpace&rsquo;s default <code>db.maxidle</code> from unlimited (-1) to something like 8 (Tomcat default) or 10 (Confluence default)</li>
|
||||
<li>Change <code>db.maxidle</code> from -1 to 10, reduce <code>db.maxconnections</code> from 90 to 50, and restart postgres and tomcat7</li>
|
||||
<li>Also redeploy DSpace Test with a clean sync of CGSpace and mirror these database settings there as well</li>
|
||||
<li>Also deploy the nginx fixes for the <code>try_files</code> location block as well as the expires block</li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2015-11-26">2015-11-26</h2>
|
||||
|
||||
<ul>
|
||||
<li>CGSpace behaving much better since changing <code>db.maxidle</code> yesterday, but still two up/down notices from monitoring this morning (better than 50!)</li>
|
||||
<li>CCAFS colleagues mentioned that the REST API is very slow, 24 seconds for one item</li>
|
||||
<li>Not as bad for me, but still unsustainable if you have to get many:</li>
|
||||
</ul>
|
||||
|
||||
<pre><code>$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
|
||||
8.415
|
||||
</code></pre>
|
||||
|
||||
<ul>
|
||||
<li>Monitoring e-mailed in the evening to say CGSpace was down</li>
|
||||
<li>Idle connections in PostgreSQL again:</li>
|
||||
</ul>
|
||||
|
||||
<pre><code>$ psql -c 'SELECT * from pg_stat_activity;' | grep cgspace | grep -c idle
|
||||
66
|
||||
</code></pre>
|
||||
|
||||
<ul>
|
||||
<li>At the time, the current DSpace pool size was 50&hellip;</li>
|
||||
<li>I reduced the pool back to the default of 30, and reduced the <code>db.maxidle</code> settings from 10 to 8</li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2015-11-29">2015-11-29</h2>
|
||||
|
||||
<ul>
|
||||
<li>Still more alerts that CGSpace has been up and down all day</li>
|
||||
<li>Current database settings for DSpace:</li>
|
||||
</ul>
|
||||
|
||||
<pre><code>db.maxconnections = 30
|
||||
db.maxwait = 5000
|
||||
db.maxidle = 8
|
||||
db.statementpool = true
|
||||
</code></pre>
|
||||
|
||||
<ul>
|
||||
<li>And idle connections:</li>
|
||||
</ul>
|
||||
|
||||
<pre><code>$ psql -c 'SELECT * from pg_stat_activity;' | grep cgspace | grep -c idle
|
||||
49
|
||||
</code></pre>
|
||||
|
||||
<ul>
|
||||
<li>Perhaps I need to start drastically increasing the connection limits—like to 300—to see if DSpace&rsquo;s thirst can ever be quenched</li>
|
||||
<li>On another note, SUNScholar&rsquo;s notes suggest adjusting some other postgres variables: <a href="http://wiki.lib.sun.ac.za/index.php/SUNScholar/Optimisations/Database">http://wiki.lib.sun.ac.za/index.php/SUNScholar/Optimisations/Database</a></li>
|
||||
<li>This might help with REST API speed (which I mentioned above and still need to do real tests)</li>
|
||||
</ul></description>
|
||||
</item>
|
||||
|
||||
|
||||
@ -13,7 +13,7 @@
|
||||
|
||||
|
||||
|
||||
<meta property="og:updated_time" content="2016-03-02T16:50:00+03:00"/>
|
||||
<meta property="og:updated_time" content="2016-04-04T11:06:00+03:00"/>
|
||||
|
||||
|
||||
|
||||
@ -43,7 +43,7 @@
|
||||
},
|
||||
|
||||
|
||||
"dateModified": "2016-03-02T16:50:00+03:00",
|
||||
"dateModified": "2016-04-04T11:06:00+03:00",
|
||||
|
||||
|
||||
|
||||
@ -105,6 +105,32 @@
|
||||
|
||||
|
||||
|
||||
<article class="blog-post">
|
||||
<header>
|
||||
<h2 class="blog-post-title"><a href="https://alanorth.github.io/cgspace-notes/2016-04/">April, 2016</a></h2>
|
||||
<p class="blog-post-meta"><time datetime="2016-04-04T11:06:00+03:00">Mon Apr 04, 2016</time> by Alan Orth in
|
||||
|
||||
<i class="fa fa-tag" aria-hidden="true"></i> <a href="/cgspace-notes/tags/notes" rel="tag">Notes</a>
|
||||
|
||||
</p>
|
||||
</header>
|
||||
<h2 id="2016-04-04">2016-04-04</h2>
|
||||
|
||||
<ul>
|
||||
<li>Looking at log file use on CGSpace and notice that we need to work on our cron setup a bit</li>
|
||||
<li>We are backing up all logs in the log folder, including useless stuff like solr, cocoon, handle-plugin, etc</li>
|
||||
<li>After running DSpace for over five years I’ve never needed to look in any other log file than dspace.log, leave alone one from last year!</li>
|
||||
<li>This will save us a few gigs of backup space we’re paying for on S3</li>
|
||||
<li>Also, I noticed the <code>checker</code> log has some errors we should pay attention to:</li>
|
||||
</ul>
|
||||
|
||||
<p></p>
|
||||
<a href='https://alanorth.github.io/cgspace-notes/2016-04/'>Read more →</a>
|
||||
</article>
|
||||
|
||||
|
||||
|
||||
|
||||
<article class="blog-post">
|
||||
<header>
|
||||
<h2 class="blog-post-title"><a href="https://alanorth.github.io/cgspace-notes/2016-03/">March, 2016</a></h2>
|
||||
@ -265,6 +291,8 @@
|
||||
<h4>Recent Posts</h4>
|
||||
<ol class="list-unstyled">
|
||||
|
||||
<li><a href="/cgspace-notes/2017-02/">February, 2017</a></li>
|
||||
|
||||
<li><a href="/cgspace-notes/2017-01/">January, 2017</a></li>
|
||||
|
||||
<li><a href="/cgspace-notes/2016-12/">December, 2016</a></li>
|
||||
@ -273,8 +301,6 @@
|
||||
|
||||
<li><a href="/cgspace-notes/2016-10/">October, 2016</a></li>
|
||||
|
||||
<li><a href="/cgspace-notes/2016-09/">September, 2016</a></li>
|
||||
|
||||
</ol>
|
||||
</section>
|
||||
|
||||
|
||||
Reference in New Issue
Block a user