Add notes for 2019-04-05

content/posts/2019-04.md

@@ -77,4 +77,29 @@ $ grep 'org.dspace.statistics.SolrLogger @ Updating' /home/cgspace.cgiar.org/log
 
 - I will have to keep an eye on it because nothing should be updating 2018 stats in 2019...
 
+## 2019-04-05
+
+- Uptime Robot reported that CGSpace (linode18) went down tonight
+- I see there are lots of PostgreSQL connections:
+
+```
+$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+      5 dspaceApi
+     10 dspaceCli
+    250 dspaceWeb
+```
+
+- I still see those weird messages about updating the statistics-2018 Solr core:
+
+```
+2019-04-05 21:06:53,770 INFO  org.dspace.statistics.SolrLogger @ Updating : 2444600/21697 docs in http://localhost:8081/solr//statistics-2018
+```
+
+- Looking at `iostat 1 10` I also see some CPU steal has come back, and I can confirm it by looking at the Munin graphs:
+
+![CPU usage week](/cgspace-notes/2019/04/cpu-week.png)
+
+- The other thing visible there is that the past few days the load has spiked to 500% and I don't think it's a coincidence that the Solr updating thing is happening...
+- I ran all system updates and rebooted the server
+
 <!-- vim: set sw=2 ts=2: -->