Add notes for 2018-02-15

content/post/2018-02.md

@@ -532,3 +532,29 @@ UPDATE 1
 - And this item doesn't even exist on CGSpace!
 - Start working on XMLUI item display code for ORCIDs
 - Send emails to Macaroni Bros and Usman at CIFOR about ORCID metadata
+- CGSpace crashed while I was driving to Tel Aviv, and was down for four hours!
+- I only looked quickly in the logs but saw a bunch of database errors
+- PostgreSQL connections are currently:
+
+```
+$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | uniq -c
+      2 dspaceApi
+      1 dspaceWeb
+      3 dspaceApi
+```
+
+- I see shitloads of memory errors in Tomcat's logs:
+    
+```
+# grep -c "Java heap space" /var/log/tomcat7/catalina.out
+56
+```
+
+- And shit tons of database connections abandoned:
+
+```
+# grep -c 'org.apache.tomcat.jdbc.pool.ConnectionPool abandon' /var/log/tomcat7/catalina.out
+612
+```
+
+- I have no fucking idea why it crashed