Add notes for 2019-04-05

This commit is contained in:
Alan Orth 2019-04-05 22:22:41 +03:00
parent 3fa0fd4458
commit 8fa47fea50
Signed by: alanorth
GPG Key ID: 0FB860CC9C45B1B9
5 changed files with 64 additions and 8 deletions

View File

@ -77,4 +77,29 @@ $ grep 'org.dspace.statistics.SolrLogger @ Updating' /home/cgspace.cgiar.org/log
- I will have to keep an eye on it because nothing should be updating 2018 stats in 2019... - I will have to keep an eye on it because nothing should be updating 2018 stats in 2019...
## 2019-04-05
- Uptime Robot reported that CGSpace (linode18) went down tonight
- I see there are lots of PostgreSQL connections:
```
$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
5 dspaceApi
10 dspaceCli
250 dspaceWeb
```
- I still see those weird messages about updating the statistics-2018 Solr core:
```
2019-04-05 21:06:53,770 INFO org.dspace.statistics.SolrLogger @ Updating : 2444600/21697 docs in http://localhost:8081/solr//statistics-2018
```
- Looking at `iostat 1 10` I also see some CPU steal has come back, and I can confirm it by looking at the Munin graphs:
![CPU usage week](/cgspace-notes/2019/04/cpu-week.png)
- The other thing visible there is that the past few days the load has spiked to 500% and I don't think it's a coincidence that the Solr updating thing is happening...
- I ran all system updates and rebooted the server
<!-- vim: set sw=2 ts=2: --> <!-- vim: set sw=2 ts=2: -->

View File

@ -38,7 +38,7 @@ $ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace
<meta property="og:type" content="article" /> <meta property="og:type" content="article" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2019-04/" /> <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2019-04/" />
<meta property="article:published_time" content="2019-04-01T09:00:43&#43;03:00"/> <meta property="article:published_time" content="2019-04-01T09:00:43&#43;03:00"/>
<meta property="article:modified_time" content="2019-04-03T17:01:31&#43;03:00"/> <meta property="article:modified_time" content="2019-04-03T17:40:05&#43;03:00"/>
<meta name="twitter:card" content="summary"/> <meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="April, 2019"/> <meta name="twitter:title" content="April, 2019"/>
@ -81,9 +81,9 @@ $ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace
"@type": "BlogPosting", "@type": "BlogPosting",
"headline": "April, 2019", "headline": "April, 2019",
"url": "https://alanorth.github.io/cgspace-notes/2019-04/", "url": "https://alanorth.github.io/cgspace-notes/2019-04/",
"wordCount": "492", "wordCount": "621",
"datePublished": "2019-04-01T09:00:43&#43;03:00", "datePublished": "2019-04-01T09:00:43&#43;03:00",
"dateModified": "2019-04-03T17:01:31&#43;03:00", "dateModified": "2019-04-03T17:40:05&#43;03:00",
"author": { "author": {
"@type": "Person", "@type": "Person",
"name": "Alan Orth" "name": "Alan Orth"
@ -244,6 +244,37 @@ $ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace
<li>I will have to keep an eye on it because nothing should be updating 2018 stats in 2019&hellip;</li> <li>I will have to keep an eye on it because nothing should be updating 2018 stats in 2019&hellip;</li>
</ul> </ul>
<h2 id="2019-04-05">2019-04-05</h2>
<ul>
<li>Uptime Robot reported that CGSpace (linode18) went down tonight</li>
<li>I see there are lots of PostgreSQL connections:</li>
</ul>
<pre><code>$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
5 dspaceApi
10 dspaceCli
250 dspaceWeb
</code></pre>
<ul>
<li>I still see those weird messages about updating the statistics-2018 Solr core:</li>
</ul>
<pre><code>2019-04-05 21:06:53,770 INFO org.dspace.statistics.SolrLogger @ Updating : 2444600/21697 docs in http://localhost:8081/solr//statistics-2018
</code></pre>
<ul>
<li>Looking at <code>iostat 1 10</code> I also see some CPU steal has come back, and I can confirm it by looking at the Munin graphs:</li>
</ul>
<p><img src="/cgspace-notes/2019/04/cpu-week.png" alt="CPU usage week" /></p>
<ul>
<li>The other thing visible there is that the past few days the load has spiked to 500% and I don&rsquo;t think it&rsquo;s a coincidence that the Solr updating thing is happening&hellip;</li>
<li>I ran all system updates and rebooted the server</li>
</ul>
<!-- vim: set sw=2 ts=2: --> <!-- vim: set sw=2 ts=2: -->

BIN
docs/2019/04/cpu-week.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 16 KiB

View File

@ -4,7 +4,7 @@
<url> <url>
<loc>https://alanorth.github.io/cgspace-notes/2019-04/</loc> <loc>https://alanorth.github.io/cgspace-notes/2019-04/</loc>
<lastmod>2019-04-03T17:01:31+03:00</lastmod> <lastmod>2019-04-03T17:40:05+03:00</lastmod>
</url> </url>
<url> <url>
@ -219,7 +219,7 @@
<url> <url>
<loc>https://alanorth.github.io/cgspace-notes/</loc> <loc>https://alanorth.github.io/cgspace-notes/</loc>
<lastmod>2019-04-03T17:01:31+03:00</lastmod> <lastmod>2019-04-03T17:40:05+03:00</lastmod>
<priority>0</priority> <priority>0</priority>
</url> </url>
@ -230,7 +230,7 @@
<url> <url>
<loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc> <loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
<lastmod>2019-04-03T17:01:31+03:00</lastmod> <lastmod>2019-04-03T17:40:05+03:00</lastmod>
<priority>0</priority> <priority>0</priority>
</url> </url>
@ -242,13 +242,13 @@
<url> <url>
<loc>https://alanorth.github.io/cgspace-notes/posts/</loc> <loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
<lastmod>2019-04-03T17:01:31+03:00</lastmod> <lastmod>2019-04-03T17:40:05+03:00</lastmod>
<priority>0</priority> <priority>0</priority>
</url> </url>
<url> <url>
<loc>https://alanorth.github.io/cgspace-notes/tags/</loc> <loc>https://alanorth.github.io/cgspace-notes/tags/</loc>
<lastmod>2019-04-03T17:01:31+03:00</lastmod> <lastmod>2019-04-03T17:40:05+03:00</lastmod>
<priority>0</priority> <priority>0</priority>
</url> </url>

BIN
static/2019/04/cpu-week.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 16 KiB