Update notes for 2017-10-31

content/post/2017-10.md

@@ -338,3 +338,18 @@ WARNING: [SetPropertiesRule]{Server/Service/Engine/Host/Valve} Setting property
 ```
 # goaccess /var/log/nginx/access.log --log-format=COMBINED
 ```
+
+- According to Uptime Robot CGSpace went down and up a few times
+- I had a look at goaccess and I saw that CORE was actively indexing
+- Also, PostgreSQL connections were at 91 (with the max being 60 per web app, hmmm)
+- I'm really starting to get annoyed with these guys, and thinking about blocking their IP address for a few days to see if CGSpace becomes more stable
+- Actually, come to think of it, they aren't even obeying `robots.txt`, because we actually disallow `/discover` and `/search-filter` URLs but they are hitting those massively:
+
+```
+# grep "CORE/0.6" /var/log/nginx/access.log | grep -o -E "GET /(discover|search-filter)" | sort -n | uniq -c | sort -rn 
+ 158058 GET /discover
+  14260 GET /search-filter
+```
+
+- I tested a URL of pattern `/discover` in Google's webmaster tools and it was indeed identified as blocked
+- I will send feedback to the CORE bot team