mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2025-01-27 05:49:12 +01:00
Add notes for 2020-01-27
This commit is contained in:
@ -33,7 +33,7 @@ Replace lzop with xz in log compression cron jobs on DSpace Test—it uses less
|
||||
-rw-rw-r-- 1 tomcat7 tomcat7 387K Nov 18 23:59 dspace.log.2015-11-18.lzo
|
||||
-rw-rw-r-- 1 tomcat7 tomcat7 169K Nov 18 23:59 dspace.log.2015-11-18.xz
|
||||
"/>
|
||||
<meta name="generator" content="Hugo 0.62.2" />
|
||||
<meta name="generator" content="Hugo 0.63.1" />
|
||||
|
||||
|
||||
|
||||
@ -63,7 +63,7 @@ Replace lzop with xz in log compression cron jobs on DSpace Test—it uses less
|
||||
|
||||
<!-- combined, minified CSS -->
|
||||
|
||||
<link href="https://alanorth.github.io/cgspace-notes/css/style.a20c1a4367639632cdb341d23c27ca44fedcc75b0f8b3cbea6203010da153d3c.css" rel="stylesheet" integrity="sha256-ogwaQ2djljLNs0HSPCfKRP7cx1sPizy+piAwENoVPTw=" crossorigin="anonymous">
|
||||
<link href="https://alanorth.github.io/cgspace-notes/css/style.23e2c3298bcc8c1136c19aba330c211ec94c36f7c4454ea15cf4d3548370042a.css" rel="stylesheet" integrity="sha256-I+LDKYvMjBE2wZq6MwwhHslMNvfERU6hXPTTVINwBCo=" crossorigin="anonymous">
|
||||
|
||||
|
||||
<!-- RSS 2.0 feed -->
|
||||
@ -111,7 +111,7 @@ Replace lzop with xz in log compression cron jobs on DSpace Test—it uses less
|
||||
<h2 class="blog-post-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/2015-12/">December, 2015</a></h2>
|
||||
<p class="blog-post-meta"><time datetime="2015-12-02T13:18:00+03:00">Wed Dec 02, 2015</time> by Alan Orth in
|
||||
|
||||
<i class="fa fa-tag" aria-hidden="true"></i> <a href="/cgspace-notes/tags/notes" rel="tag">Notes</a>
|
||||
<span class="fas fa-tag" aria-hidden="true"></span> <a href="/cgspace-notes/tags/notes" rel="tag">Notes</a>
|
||||
|
||||
</p>
|
||||
</header>
|
||||
@ -133,7 +133,7 @@ Replace lzop with xz in log compression cron jobs on DSpace Test—it uses less
|
||||
<pre><code>$ psql -c 'SELECT * from pg_stat_activity;' | grep cgspace | grep -c idle
|
||||
39
|
||||
</code></pre><ul>
|
||||
<li>I restarted PostgreSQL and Tomcat and it's back</li>
|
||||
<li>I restarted PostgreSQL and Tomcat and it’s back</li>
|
||||
<li>On a related note of why CGSpace is so slow, I decided to finally try the <code>pgtune</code> script to tune the postgres settings:</li>
|
||||
</ul>
|
||||
<pre><code># apt-get install pgtune
|
||||
@ -155,7 +155,7 @@ shared_buffers = 1920MB
|
||||
max_connections = 80
|
||||
</code></pre><ul>
|
||||
<li>Now I need to go read PostgreSQL docs about these options, and watch memory settings in munin etc</li>
|
||||
<li>For what it's worth, now the REST API should be faster (because of these PostgreSQL tweaks):</li>
|
||||
<li>For what it’s worth, now the REST API should be faster (because of these PostgreSQL tweaks):</li>
|
||||
</ul>
|
||||
<pre><code>$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
|
||||
1.474
|
||||
@ -173,9 +173,9 @@ $ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle
|
||||
</ul>
|
||||
<p><img src="/cgspace-notes/2015/12/ccafs-item-no-metadata.png" alt="CCAFS item"></p>
|
||||
<ul>
|
||||
<li>The authorizations for the item are all public READ, and I don't see any errors in dspace.log when browsing that item</li>
|
||||
<li>I filed a ticket on Atmire's issue tracker</li>
|
||||
<li>I also filed a ticket on Atmire's issue tracker for the PostgreSQL stuff</li>
|
||||
<li>The authorizations for the item are all public READ, and I don’t see any errors in dspace.log when browsing that item</li>
|
||||
<li>I filed a ticket on Atmire’s issue tracker</li>
|
||||
<li>I also filed a ticket on Atmire’s issue tracker for the PostgreSQL stuff</li>
|
||||
</ul>
|
||||
<h2 id="2015-12-03">2015-12-03</h2>
|
||||
<ul>
|
||||
@ -187,7 +187,7 @@ $ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle
|
||||
</code></pre><ul>
|
||||
<li>I restarted Tomcat and postgres…</li>
|
||||
<li>Atmire commented that we should raise the JVM heap size by ~500M, so it is now <code>-Xms3584m -Xmx3584m</code></li>
|
||||
<li>We weren't out of heap yet, but it's probably fair enough that the DSpace 5 upgrade (and new Atmire modules) requires more memory so it's ok</li>
|
||||
<li>We weren’t out of heap yet, but it’s probably fair enough that the DSpace 5 upgrade (and new Atmire modules) requires more memory so it’s ok</li>
|
||||
<li>A possible side effect is that I see that the REST API is twice as fast for the request above now:</li>
|
||||
</ul>
|
||||
<pre><code>$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
|
||||
@ -210,7 +210,7 @@ $ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle
|
||||
<pre><code>postgres@linode01:~$ psql -c 'SELECT * from pg_stat_activity;' | grep cgspace | grep -c idle
|
||||
28
|
||||
</code></pre><ul>
|
||||
<li>I have reverted all the pgtune tweaks from the other day, as they didn't fix the stability issues, so I'd rather not have them introducing more variables into the equation</li>
|
||||
<li>I have reverted all the pgtune tweaks from the other day, as they didn’t fix the stability issues, so I’d rather not have them introducing more variables into the equation</li>
|
||||
<li>The PostgreSQL stats from Munin all point to something database-related with the DSpace 5 upgrade around mid–late November</li>
|
||||
</ul>
|
||||
<p><img src="/cgspace-notes/2015/12/postgres_bgwriter-year.png" alt="PostgreSQL bgwriter (year)">
|
||||
@ -219,7 +219,7 @@ $ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle
|
||||
<img src="/cgspace-notes/2015/12/postgres_scans_cgspace-year.png" alt="PostgreSQL scans (year)"></p>
|
||||
<h2 id="2015-12-07">2015-12-07</h2>
|
||||
<ul>
|
||||
<li>Atmire sent <a href="https://github.com/ilri/DSpace/pull/161">some fixes</a> to DSpace's REST API code that was leaving contexts open (causing the slow performance and database issues)</li>
|
||||
<li>Atmire sent <a href="https://github.com/ilri/DSpace/pull/161">some fixes</a> to DSpace’s REST API code that was leaving contexts open (causing the slow performance and database issues)</li>
|
||||
<li>After deploying the fix to CGSpace the REST API is consistently faster:</li>
|
||||
</ul>
|
||||
<pre><code>$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
|
||||
@ -234,8 +234,8 @@ $ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle
|
||||
0.497
|
||||
</code></pre><h2 id="2015-12-08">2015-12-08</h2>
|
||||
<ul>
|
||||
<li>Switch CGSpace log compression cron jobs from using lzop to xz—the compression isn't as good, but it's much faster and causes less IO/CPU load</li>
|
||||
<li>Since we figured out (and fixed) the cause of the performance issue, I reverted Google Bot's crawl rate to the “Let Google optimize” setting</li>
|
||||
<li>Switch CGSpace log compression cron jobs from using lzop to xz—the compression isn’t as good, but it’s much faster and causes less IO/CPU load</li>
|
||||
<li>Since we figured out (and fixed) the cause of the performance issue, I reverted Google Bot’s crawl rate to the “Let Google optimize” setting</li>
|
||||
</ul>
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user