Add notes for 2019-03-21

content/posts/2019-03.md

@@ -561,4 +561,39 @@ Max realtime timeout      unlimited            unlimited            us
 - Create a branch for Solr 4.10.4 changes so I can test on DSpace Test (linode19)
   - Deployed Solr 4.10.4 on DSpace Test and will leave it there for a few weeks, as well as on my local environment
 
+## 2019-03-21
+
+- It's been two days since we had the blank page issue on CGSpace, and looking in the Cocoon logs I see very low numbers of the errors that we were seeing the last time the issue occurred:
+
+```
+$ grep 'Can not load requested doc' cocoon.log.2019-03-20 | grep -oE '2019-03-20 [0-9]{2}:' | sort | uniq -c
+      3 2019-03-20 00:
+     12 2019-03-20 02:
+$ grep 'Can not load requested doc' cocoon.log.2019-03-21 | grep -oE '2019-03-21 [0-9]{2}:' | sort | uniq -c
+      4 2019-03-21 00:
+      1 2019-03-21 02:
+      4 2019-03-21 03:
+      1 2019-03-21 05:
+      4 2019-03-21 06:
+     11 2019-03-21 07:
+     14 2019-03-21 08:
+      3 2019-03-21 09:
+      4 2019-03-21 10:
+      5 2019-03-21 11:
+      4 2019-03-21 12:
+      3 2019-03-21 13:
+      6 2019-03-21 14:
+      2 2019-03-21 15:
+      3 2019-03-21 16:
+      3 2019-03-21 18:
+      1 2019-03-21 19:
+      6 2019-03-21 20:
+```
+
+- To investigate the Solr lock issue I added a `find` command to the Tomcat 7 service with `ExecStartPre` and `ExecStopPost` and noticed that the lock files are always there...
+  - Perhaps the lock files are less of an issue than I thought?
+  - I will share my thoughts with the dspace-tech community
+- In other news, I notice that that systemd always thinks that Tomcat has failed when it stops because the JVM exits with code 143, which is apparently normal
+  - We can add `SuccessExitStatus=143` to the systemd service so that it knows this is a successful exit
+
 <!-- vim: set sw=2 ts=2: -->