diff --git a/content/2016-09.md b/content/2016-09.md index a6f867c0b..541425461 100644 --- a/content/2016-09.md +++ b/content/2016-09.md @@ -224,3 +224,59 @@ Exception in thread "http-bio-127.0.0.1-8081-exec-25" java.lang.OutOfMemoryError 2016-09-14 12:23:07,981 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error - org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object ``` + +- Looking at REST requests, it seems there is one IP hitting us nonstop: + +``` +# awk '{print $1}' /var/log/nginx/rest.log | sort -n | uniq -c | sort -h | tail -n 3 + 820 50.87.54.15 + 12872 70.32.99.142 + 25744 70.32.83.92 +# awk '{print $1}' /var/log/nginx/rest.log.1 | sort -n | uniq -c | sort -h | tail -n 3 + 7966 181.118.144.29 + 54706 70.32.99.142 + 109412 70.32.83.92 +``` + +- Those are the same IPs that were hitting us heavily in July, 2016 as well... +- I think the stability issues are definitely from REST +- Crashed AGAIN, errors from dspace.log: + +``` +2016-09-14 14:31:43,069 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error - +org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object +``` + +- And more heap space errors: + +``` +# grep -rsI "OutOfMemoryError" /var/log/tomcat7/catalina.* | wc -l +19 +``` + +- There are no more rest requests since the last crash, so maybe there are other things causing this. +- Hmm, I noticed a shitload of IPs from 180.76.0.0/16 are connecting to both CGSpace and DSpace Test (58 unique IPs concurrently!) +- They seem to be coming from Baidu, and so far during today alone account for 1/6 of every connection: + +``` +# grep -c ip_addr= /home/cgspace.cgiar.org/log/dspace.log.2016-09-14 +29084 +# grep -c ip_addr=180.76.15 /home/cgspace.cgiar.org/log/dspace.log.2016-09-14 +5192 +``` + +- Other recent days are the same... hmmm. +- From the activity control panel I can see 58 unique IPs hitting the site *concurrently*, which has GOT to hurt our stability +- A list of all 2000 unique IPs from CGSpace logs today: + +``` +# grep ip_addr= /home/cgspace.cgiar.org/log/dspace.log.2016-09-11 | awk -F: '{print $5}' | sort -n | uniq -c | sort -h | tail -n 100 +``` + +- Looking at the top 20 IPs or so, most are Yahoo, MSN, Google, Baidu, TurnitIn (iParadigm), etc... do we have any real users? +- Generate a list of all Affiliations for Peter Ballantyne to go through, make corrections, and create a lookup list from: + +``` +dspacetest=# \copy (select text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=211 group by text_value order by count desc) + to /tmp/affiliations.csv with csv; +``` diff --git a/public/2016-09/index.html b/public/2016-09/index.html index fbda90187..062ebacff 100644 --- a/public/2016-09/index.html +++ b/public/2016-09/index.html @@ -339,6 +339,68 @@ org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object + + +
# awk '{print $1}' /var/log/nginx/rest.log  | sort -n | uniq -c | sort -h | tail -n 3
+    820 50.87.54.15
+  12872 70.32.99.142
+  25744 70.32.83.92
+# awk '{print $1}' /var/log/nginx/rest.log.1  | sort -n | uniq -c | sort -h | tail -n 3
+   7966 181.118.144.29
+  54706 70.32.99.142
+ 109412 70.32.83.92
+
+ + + +
2016-09-14 14:31:43,069 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error -
+org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
+
+ + + +
# grep -rsI "OutOfMemoryError" /var/log/tomcat7/catalina.* | wc -l
+19
+
+ + + +
# grep -c ip_addr= /home/cgspace.cgiar.org/log/dspace.log.2016-09-14
+29084
+# grep -c ip_addr=180.76.15 /home/cgspace.cgiar.org/log/dspace.log.2016-09-14
+5192
+
+ + + +
# grep ip_addr= /home/cgspace.cgiar.org/log/dspace.log.2016-09-11 | awk -F: '{print $5}' | sort -n | uniq -c | sort -h | tail -n 100
+
+ + + +
dspacetest=# \copy (select text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=211 group by text_value order by count desc)
+ to /tmp/affiliations.csv with csv;
+
+