Update notes for 2017-10-31

content/post/2017-10.md

@@ -322,3 +322,19 @@ WARNING: [SetPropertiesRule]{Server/Service/Engine/Host/Valve} Setting property
 
 - Very nice, Linode alerted that CGSpace had high CPU usage at 2AM again
 - Ask on the dspace-tech mailing list if it's possible to use an existing item as a template for a new item
+- To follow up on the CORE bot traffic, there were almost 300,000 request yesterday:
+
+```
+# grep "CORE/0.6" /var/log/nginx/access.log.1 | awk '{print $1}' | sort -n | uniq -c | sort -h
+ 139109 137.108.70.6
+ 139253 137.108.70.7
+```
+
+- I've emailed the CORE people to ask if they can update the repository information from CGIAR Library to CGSpace
+- Also, I asked if they could perhaps use the `sitemap.xml`, OAI-PMH, or REST APIs to index us more efficiently, because they mostly seem to be crawling the nearly endless Discovery facets
+- I added [GoAccess](https://goaccess.io/) to the list of package to install in the DSpace role of the [Ansible infrastructure scripts](https://github.com/ilri/rmg-ansible-public)
+- It makes it very easy to analyze nginx logs from the command line, to see where traffic is coming from:
+
+```
+# goaccess /var/log/nginx/access.log --log-format=COMBINED
+```