Update notes

2025-01-27 05:49:12 +01:00 · 2018-01-10 13:05:02 +02:00
parent 727c761af4
commit a421ac8227
3 changed files with 172 additions and 8 deletions
--- a/content/post/2018-01.md
+++ b/content/post/2018-01.md
@ -370,6 +370,84 @@ $ http 'http://localhost:3000/solr/statistics/select?q=-owningColl%3A*&wt=json&i
  "response":{"numFound":34879872,"start":0,"docs":[
 ```

+- I tested the `dspace stats-util -s` process on my local machine and it failed the same way
+- It doesn't seem to be helpful, but the dspace log shows this:
+
+```
+2018-01-10 10:51:19,301 INFO  org.dspace.statistics.SolrLogger @ Created core with name: statistics-2016
+2018-01-10 10:51:19,301 INFO  org.dspace.statistics.SolrLogger @ Moving: 3821 records into core statistics-2016
+```
+
+- Terry Brady has written some notes on the DSpace Wiki about Solr sharing issues: https://wiki.duraspace.org/display/%7Eterrywbrady/Statistics+Import+Export+Issues
+- Uptime Robot said that CGSpace went down at around 9:43 AM
+- I looked at PostgreSQL's `pg_stat_activity` table and saw 161 active connections, but no pool errors in the DSpace logs:
+
+```
+$ grep -c "Timeout: Pool empty." dspace.log.2018-01-10 
+0
+```
+
+- The XMLUI logs show quite a bit of activity today:
+
+```
+# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep "10/Jan/2018" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+    951 207.46.13.159
+    954 157.55.39.123
+   1217 95.108.181.88
+   1503 104.196.152.243
+   6455 70.36.107.50
+  11412 70.36.107.190
+  16730 70.36.107.49
+  17386 2607:fa98:40:9:26b6:fdff:feff:1c96
+  21566 2607:fa98:40:9:26b6:fdff:feff:195d
+  45384 2607:fa98:40:9:26b6:fdff:feff:1888
+```
+
+- The user agent for the top six or so IPs are all the same:
+
+```
+"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36"
+```
+
+- `whois` says they come from [Perfect IP](http://www.perfectip.net/)
+- I've never seen those top IPs before, but they have created 50,000 Tomcat sessions today:
+
+```
+$ grep -E '(2607:fa98:40:9:26b6:fdff:feff:1888|2607:fa98:40:9:26b6:fdff:feff:195d|2607:fa98:40:9:26b6:fdff:feff:1c96|70.36.107.49|70.36.107.190|70.36.107.50)' /home/cgspace.cgiar.org/log/dspace.log.2018-01-10 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l                                                                                                                                                                                                  
+49096
+```
+
+- Rather than blocking their IPs, I think I might just add their user agent to the "badbots" zone with Baidu, because they seem to be the only ones using that user agent:
+
+```
+# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari
+/537.36" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+   6796 70.36.107.50
+  11870 70.36.107.190
+  17323 70.36.107.49
+  19204 2607:fa98:40:9:26b6:fdff:feff:1c96
+  23401 2607:fa98:40:9:26b6:fdff:feff:195d 
+  47875 2607:fa98:40:9:26b6:fdff:feff:1888
+```
+
+- I added the user agent to nginx's badbots limit req zone but upon testing the config I got an error:
+
+```
+# nginx -t
+nginx: [emerg] could not build map_hash, you should increase map_hash_bucket_size: 64
+nginx: configuration file /etc/nginx/nginx.conf test failed
+```
+
+- According to nginx docs the [bucket size should be a multiple of the CPU's cache alignment](https://nginx.org/en/docs/hash.html), which is 64 for us:
+
+```
+# cat /proc/cpuinfo | grep cache_alignment | head -n1
+cache_alignment : 64
+```
+
+- On our servers that is 64, so I increased this parameter to 128 and deployed the changes to nginx
+- Almost immediately the PostgreSQL connections dropped back down to 40 or so, and UptimeRobot said the site was back up
+- So that's interesting that we're not out of PostgreSQL connections (current pool maxActive is 300!) but the system is "down" to UptimeRobot and very slow to use
 - Linode continues to test mitigations for Meltdown and Spectre: https://blog.linode.com/2018/01/03/cpu-vulnerabilities-meltdown-spectre/
 - I rebooted DSpace Test to see if the kernel will be updated (currently Linux 4.14.12-x86_64-linode92)... nope.
 - It looks like Linode will reboot the KVM hosts later this week, though