diff --git a/content/posts/2019-04.md b/content/posts/2019-04.md index a69b6bcc5..71e6557d2 100644 --- a/content/posts/2019-04.md +++ b/content/posts/2019-04.md @@ -77,4 +77,29 @@ $ grep 'org.dspace.statistics.SolrLogger @ Updating' /home/cgspace.cgiar.org/log - I will have to keep an eye on it because nothing should be updating 2018 stats in 2019... +## 2019-04-05 + +- Uptime Robot reported that CGSpace (linode18) went down tonight +- I see there are lots of PostgreSQL connections: + +``` +$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c + 5 dspaceApi + 10 dspaceCli + 250 dspaceWeb +``` + +- I still see those weird messages about updating the statistics-2018 Solr core: + +``` +2019-04-05 21:06:53,770 INFO org.dspace.statistics.SolrLogger @ Updating : 2444600/21697 docs in http://localhost:8081/solr//statistics-2018 +``` + +- Looking at `iostat 1 10` I also see some CPU steal has come back, and I can confirm it by looking at the Munin graphs: + +![CPU usage week](/cgspace-notes/2019/04/cpu-week.png) + +- The other thing visible there is that the past few days the load has spiked to 500% and I don't think it's a coincidence that the Solr updating thing is happening... +- I ran all system updates and rebooted the server + diff --git a/docs/2019-04/index.html b/docs/2019-04/index.html index 0cbfd2edd..4cd71f98f 100644 --- a/docs/2019-04/index.html +++ b/docs/2019-04/index.html @@ -38,7 +38,7 @@ $ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace - + @@ -81,9 +81,9 @@ $ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace "@type": "BlogPosting", "headline": "April, 2019", "url": "https://alanorth.github.io/cgspace-notes/2019-04/", - "wordCount": "492", + "wordCount": "621", "datePublished": "2019-04-01T09:00:43+03:00", - "dateModified": "2019-04-03T17:01:31+03:00", + "dateModified": "2019-04-03T17:40:05+03:00", "author": { "@type": "Person", "name": "Alan Orth" @@ -244,6 +244,37 @@ $ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace
  • I will have to keep an eye on it because nothing should be updating 2018 stats in 2019…
  • +

    2019-04-05

    + + + +
    $ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
    +      5 dspaceApi
    +     10 dspaceCli
    +    250 dspaceWeb
    +
    + + + +
    2019-04-05 21:06:53,770 INFO  org.dspace.statistics.SolrLogger @ Updating : 2444600/21697 docs in http://localhost:8081/solr//statistics-2018
    +
    + + + +

    CPU usage week

    + + + diff --git a/docs/2019/04/cpu-week.png b/docs/2019/04/cpu-week.png new file mode 100644 index 000000000..06b95d262 Binary files /dev/null and b/docs/2019/04/cpu-week.png differ diff --git a/docs/sitemap.xml b/docs/sitemap.xml index 03dc066df..9094afe68 100644 --- a/docs/sitemap.xml +++ b/docs/sitemap.xml @@ -4,7 +4,7 @@ https://alanorth.github.io/cgspace-notes/2019-04/ - 2019-04-03T17:01:31+03:00 + 2019-04-03T17:40:05+03:00 @@ -219,7 +219,7 @@ https://alanorth.github.io/cgspace-notes/ - 2019-04-03T17:01:31+03:00 + 2019-04-03T17:40:05+03:00 0 @@ -230,7 +230,7 @@ https://alanorth.github.io/cgspace-notes/tags/notes/ - 2019-04-03T17:01:31+03:00 + 2019-04-03T17:40:05+03:00 0 @@ -242,13 +242,13 @@ https://alanorth.github.io/cgspace-notes/posts/ - 2019-04-03T17:01:31+03:00 + 2019-04-03T17:40:05+03:00 0 https://alanorth.github.io/cgspace-notes/tags/ - 2019-04-03T17:01:31+03:00 + 2019-04-03T17:40:05+03:00 0 diff --git a/static/2019/04/cpu-week.png b/static/2019/04/cpu-week.png new file mode 100644 index 000000000..06b95d262 Binary files /dev/null and b/static/2019/04/cpu-week.png differ