Update notes for 2016-09-14

content/2016-09.md

@@ -224,3 +224,59 @@ Exception in thread "http-bio-127.0.0.1-8081-exec-25" java.lang.OutOfMemoryError
 2016-09-14 12:23:07,981 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error -
 org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
 ```
+
+- Looking at REST requests, it seems there is one IP hitting us nonstop:
+
+```
+# awk '{print $1}' /var/log/nginx/rest.log  | sort -n | uniq -c | sort -h | tail -n 3
+    820 50.87.54.15
+  12872 70.32.99.142
+  25744 70.32.83.92
+# awk '{print $1}' /var/log/nginx/rest.log.1  | sort -n | uniq -c | sort -h | tail -n 3
+   7966 181.118.144.29
+  54706 70.32.99.142
+ 109412 70.32.83.92
+```
+
+- Those are the same IPs that were hitting us heavily in July, 2016 as well...
+- I think the stability issues are definitely from REST
+- Crashed AGAIN, errors from dspace.log:
+
+```
+2016-09-14 14:31:43,069 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error -
+org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
+```
+
+- And more heap space errors:
+
+```
+# grep -rsI "OutOfMemoryError" /var/log/tomcat7/catalina.* | wc -l
+19
+```
+
+- There are no more rest requests since the last crash, so maybe there are other things causing this.
+- Hmm, I noticed a shitload of IPs from 180.76.0.0/16 are connecting to both CGSpace and DSpace Test (58 unique IPs concurrently!)
+- They seem to be coming from Baidu, and so far during today alone account for 1/6 of every connection:
+
+```
+# grep -c ip_addr= /home/cgspace.cgiar.org/log/dspace.log.2016-09-14
+29084
+# grep -c ip_addr=180.76.15 /home/cgspace.cgiar.org/log/dspace.log.2016-09-14
+5192
+```
+
+- Other recent days are the same... hmmm.
+- From the activity control panel I can see 58 unique IPs hitting the site *concurrently*, which has GOT to hurt our stability
+- A list of all 2000 unique IPs from CGSpace logs today:
+
+```
+# grep ip_addr= /home/cgspace.cgiar.org/log/dspace.log.2016-09-11 | awk -F: '{print $5}' | sort -n | uniq -c | sort -h | tail -n 100
+```
+
+- Looking at the top 20 IPs or so, most are Yahoo, MSN, Google, Baidu, TurnitIn (iParadigm), etc... do we have any real users?
+- Generate a list of all Affiliations for Peter Ballantyne to go through, make corrections, and create a lookup list from:
+
+```
+dspacetest=# \copy (select text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=211 group by text_value order by count desc)
+ to /tmp/affiliations.csv with csv;
+```