alanorth/cgspace-notes
1
0
Fork 0
mirror of https://github.com/alanorth/cgspace-notes.git synced 2025-01-27 05:49:12 +01:00
Code Issues Projects Releases Wiki Activity

Add notes for 2019-12-17

Browse Source
This commit is contained in:
Alan Orth
2019-12-17 14:49:24 +02:00
parent d83c951532
commit d54e5b69f1
90 changed files with 1420 additions and 1377 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
docs/2017-12/index.html
View File

@ -27,7 +27,7 @@ The logs say “Timeout waiting for idle object”
PostgreSQL activity says there are 115 connections currently
The list of connections to XMLUI and REST API for today:
"/>
<meta name="generator" content="Hugo 0.60.1" />
<meta name="generator" content="Hugo 0.61.0" />
@ -108,7 +108,7 @@ The list of connections to XMLUI and REST API for today:
</p>
</header>
<h2 id="20171201">2017-12-01</h2>
<h2 id="2017-12-01">2017-12-01</h2>
<ul>
<li>Uptime Robot noticed that CGSpace went down</li>
<li>The logs say &ldquo;Timeout waiting for idle object&rdquo;</li>
@ -166,11 +166,11 @@ The list of connections to XMLUI and REST API for today:
14 2a01:7e00::f03c:91ff:fe18:7396
46 2001:4b99:1:1:216:3eff:fe2c:dc6c
319 2001:4b99:1:1:216:3eff:fe76:205b
</code></pre><h2 id="20171203">2017-12-03</h2>
</code></pre><h2 id="2017-12-03">2017-12-03</h2>
<ul>
<li>Linode alerted that CGSpace's load was 327.5% from 6 to 8 AM again</li>
</ul>
<h2 id="20171204">2017-12-04</h2>
<h2 id="2017-12-04">2017-12-04</h2>
<ul>
<li>Linode alerted that CGSpace's load was 255.5% from 8 to 10 AM again</li>
<li>I looked at the Munin stats on DSpace Test (linode02) again to see how the PostgreSQL tweaks from a few weeks ago were holding up:</li>
@ -184,13 +184,13 @@ The list of connections to XMLUI and REST API for today:
<li>For reference, here is the past month's connections:</li>
</ul>
<p><img src="/cgspace-notes/2017/12/postgres-connections-month-cgspace.png" alt="CGSpace PostgreSQL connections month"></p>
<h2 id="20171205">2017-12-05</h2>
<h2 id="2017-12-05">2017-12-05</h2>
<ul>
<li>Linode alerted again that the CPU usage on CGSpace was high this morning from 8 to 10 AM</li>
<li>CORE updated the entry for CGSpace on their index: <a href="https://core.ac.uk/search?q=repositories.id:(1016)&amp;fullTextOnly=false">https://core.ac.uk/search?q=repositories.id:(1016)&amp;fullTextOnly=false</a></li>
<li>Linode alerted again that the CPU usage on CGSpace was high this evening from 8 to 10 PM</li>
</ul>
<h2 id="20171206">2017-12-06</h2>
<h2 id="2017-12-06">2017-12-06</h2>
<ul>
<li>Linode alerted again that the CPU usage on CGSpace was high this morning from 6 to 8 AM</li>
<li>Uptime Robot alerted that the server went down and up around 8:53 this morning</li>
@ -212,7 +212,7 @@ The list of connections to XMLUI and REST API for today:
</code></pre><ul>
<li>50.116.102.77 is apparently in the US on websitewelcome.com</li>
</ul>
<h2 id="20171207">2017-12-07</h2>
<h2 id="2017-12-07">2017-12-07</h2>
<ul>
<li>Uptime Robot reported a few times today that CGSpace was down and then up</li>
<li>At one point Tsega restarted Tomcat</li>
@ -254,17 +254,17 @@ Error: ERROR: update or delete on table &quot;bitstream&quot; violates foreign k
</ul>
<pre><code>dspace=# update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (144666);
UPDATE 1
</code></pre><h2 id="20171213">2017-12-13</h2>
</code></pre><h2 id="2017-12-13">2017-12-13</h2>
<ul>
<li>Linode alerted that CGSpace was using high CPU from 10:13 to 12:13 this morning</li>
</ul>
<h2 id="20171216">2017-12-16</h2>
<h2 id="2017-12-16">2017-12-16</h2>
<ul>
<li>Re-work the XMLUI base theme to allow child themes to override the header logo's image and link destination: <a href="https://github.com/ilri/DSpace/pull/349">#349</a></li>
<li>This required a little bit of work to restructure the XSL templates</li>
<li>Optimize PNG and SVG image assets in the CGIAR base theme using pngquant and svgo: <a href="https://github.com/ilri/DSpace/pull/350">#350</a></li>
</ul>
<h2 id="20171217">2017-12-17</h2>
<h2 id="2017-12-17">2017-12-17</h2>
<ul>
<li>Reboot DSpace Test to get new Linode Linux kernel</li>
<li>Looking at CCAFS bulk import for Magdalena Haman (she originally sent them in November but some of the thumbnails were missing and dates were messed up so she resent them now)</li>
@ -358,7 +358,7 @@ Elapsed time: 2 secs (2559 msecs)
<li>I will apply it on our branch but I need to make a note to NOT cherry-pick it when I rebase on to the latest 5.x upstream later</li>
<li>Pull request: <a href="https://github.com/ilri/DSpace/pull/351">#351</a></li>
</ul>
<h2 id="20171218">2017-12-18</h2>
<h2 id="2017-12-18">2017-12-18</h2>
<ul>
<li>Linode alerted this morning that there was high outbound traffic from 6 to 8 AM</li>
<li>The XMLUI logs show that the CORE bot from last night (137.108.70.7) is very active still:</li>
@ -453,7 +453,7 @@ $ schedtool -D -e ionice -c2 -n7 nice -n19 dspace index-discovery
</code></pre><ul>
<li>The PostgreSQL issues are getting out of control, I need to figure out how to enable connection pools in Tomcat!</li>
</ul>
<h2 id="20171219">2017-12-19</h2>
<h2 id="2017-12-19">2017-12-19</h2>
<ul>
<li>Briefly had PostgreSQL connection issues on CGSpace for the millionth time</li>
<li>I'm fucking sick of this!</li>
@ -651,7 +651,7 @@ javax.naming.NoInitialContextException: Need to specify class name in environmen
<li>If you monitor the <code>pg_stat_activity</code> while you run <code>dspace database info</code> you can see that it doesn't use the JNDI and creates ~9 extra PostgreSQL connections!</li>
<li>And in the middle of all of this Linode sends an alert that CGSpace has high CPU usage from 2 to 4 PM</li>
</ul>
<h2 id="20171220">2017-12-20</h2>
<h2 id="2017-12-20">2017-12-20</h2>
<ul>
<li>The database connection pooling is definitely better!</li>
</ul>
@ -674,7 +674,7 @@ $ schedtool -D -e ionice -c2 -n7 nice -n19 dspace filter-media -i 10568/89287
</code></pre><ul>
<li>The final code for the JNDI work in the Ansible infrastructure scripts is here: <a href="https://github.com/ilri/rmg-ansible-public/commit/1959d9cb7a0e7a7318c77f769253e5e029bdfa3b">https://github.com/ilri/rmg-ansible-public/commit/1959d9cb7a0e7a7318c77f769253e5e029bdfa3b</a></li>
</ul>
<h2 id="20171224">2017-12-24</h2>
<h2 id="2017-12-24">2017-12-24</h2>
<ul>
<li>Linode alerted that CGSpace was using high CPU this morning around 6 AM</li>
<li>I'm playing with reading all of a month's nginx logs into goaccess:</li>
@ -690,13 +690,13 @@ $ schedtool -D -e ionice -c2 -n7 nice -n19 dspace filter-media -i 10568/89287
</ul>
</li>
</ul>
<h2 id="20171225">2017-12-25</h2>
<h2 id="2017-12-25">2017-12-25</h2>
<ul>
<li>The PostgreSQL connection pooling is much better when using the Tomcat JNDI pool</li>
<li>Here are the Munin stats for the past week on CGSpace:</li>
</ul>
<p><img src="/cgspace-notes/2017/12/postgres-connections-cgspace.png" alt="CGSpace PostgreSQL connections week"></p>
<h2 id="20171229">2017-12-29</h2>
<h2 id="2017-12-29">2017-12-29</h2>
<ul>
<li>Looking at some old notes for metadata to clean up, I found a few hundred corrections in <code>cg.fulltextstatus</code> and <code>dc.language.iso</code>:</li>
</ul>
@ -721,7 +721,7 @@ DELETE 20
</code></pre><ul>
<li>I need to figure out why we have records with language <code>in</code> because that's not a language!</li>
</ul>
<h2 id="20171230">2017-12-30</h2>
<h2 id="2017-12-30">2017-12-30</h2>
<ul>
<li>Linode alerted that CGSpace was using 259% CPU from 4 to 6 AM</li>
<li>Uptime Robot noticed that the server went down for 1 minute a few hours later, around 9AM</li>
@ -748,7 +748,7 @@ DELETE 20
</code></pre><ul>
<li>216.244.66.245 seems to be moz.com's DotBot</li>
</ul>
<h2 id="20171231">2017-12-31</h2>
<h2 id="2017-12-31">2017-12-31</h2>
<ul>
<li>I finished working on the 42 records for CCAFS after Magdalena sent the remaining corrections</li>
<li>After that I uploaded them to CGSpace:</li>