Add notes for 2018-02-22

content/post/2018-02.md

@@ -756,3 +756,56 @@ $ cat orcid-test-values.txt
 - I told them that it's their JavaScript that is fucked up
 - Remove CPWF project number and Humidtropics subject from submission form ([#3](https://github.com/alanorth/DSpace/pull/3))
 - I accidentally merged it into my own repository, oops
+
+## 2018-02-22
+
+- CGSpace was apparently down today around 13:00 server time and I didn't get any emails on my phone, but saw them later on the computer
+- It looks like Sisay restarted Tomcat because I was offline
+- There was absolutely nothing interesting going on at 13:00 on the server, WTF?
+
+```
+# cat /var/log/nginx/*.log | grep -E "22/Feb/2018:13" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+     55 192.99.39.235
+     60 207.46.13.26
+     62 40.77.167.38
+     65 207.46.13.23
+    103 41.57.108.208
+    120 104.196.152.243
+    133 104.154.216.0
+    145 68.180.228.117
+    159 54.92.197.82
+    231 5.9.6.51
+```
+
+- Otherwise there was pretty normal traffic the rest of the day:
+
+```
+# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "22/Feb/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    839 216.244.66.245
+   1074 68.180.228.117
+   1114 157.55.39.100
+   1162 207.46.13.26
+   1178 207.46.13.23
+   2749 104.196.152.243
+   3109 50.116.102.77
+   4199 70.32.83.92
+   5208 5.9.6.51
+   8686 45.5.184.196
+```
+
+- While looking at the logs I see some new bot:
+
+```
+Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.4.2661.102 Safari/537.36; 360Spider
+```
+
+- It seems to re-use its user agent but makes tons of useless requests and I wonder if I should add ".*spider.*" to the Tomcat Crawler Session Manager valve?
+- This is definitely not good:
+
+```
+# grep -c 'org.apache.tomcat.jdbc.pool.ConnectionPool abandon' /var/log/tomcat7/catalina.out
+729
+```
+
+- I think the `removeAbandonedTimeout` might still be too low (I increased it from 60 to 90 seconds last week)
+- Perhaps something more like a few minutes is better?