diff --git a/content/posts/2019-04.md b/content/posts/2019-04.md index 1e509fee2..67aa7a729 100644 --- a/content/posts/2019-04.md +++ b/content/posts/2019-04.md @@ -351,5 +351,44 @@ $ http --print b 'http://localhost:8080/solr/statistics/select?q=type%3A0+AND+ti - See: [DS-3832](https://jira.duraspace.org/browse/DS-3832) - DSpace 5.10 upgraded to use GeoIP2, but we are on 5.8 so I just copied the missing database file from another server because it has been *removed* from MaxMind's server as of 2018-04-01 - Now I made 100 requests and I see them in the Solr statistics... fuck my life for wasting five hours debugging this +- UptimeRobot said CGSpace went down and up a few times tonight, and my first instict was to check `iostat 1 10` and I saw that CPU steal is around 10–30 percent right now... +- The load average is super high right now, as I've noticed the last few times UptimeRobot said that CGSpace went down: + +``` +$ cat /proc/loadavg +10.70 9.17 8.85 18/633 4198 +``` + +- According to the server logs there is actually not much going on right now: + +``` +# zcat --force /var/log/nginx/{access,library-access}.log /var/log/nginx/{access,library-access}.log.1 | grep -E "07/Apr/2019:(18|19)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10 + 116 40.77.167.128 + 128 207.46.13.219 + 129 167.114.64.100 + 159 207.46.13.129 + 179 207.46.13.33 + 188 2408:8214:7a00:868f:7c1e:e0f3:20c6:c142 + 195 66.249.79.59 + 363 40.77.167.21 + 740 2a01:4f8:140:3192::2 + 3711 45.5.184.72 +# zcat --force /var/log/nginx/{rest,oai}.log /var/log/nginx/{rest,oai}.log.1 | grep -E "07/Apr/2019:(18|19)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10 + 3 2a02:1206:4576:bae0:95ff:8bbd:a274:6223 + 3 66.249.79.62 + 3 66.249.83.196 + 5 82.145.222.150 + 6 2a01:4f9:2b:1263::2 + 6 41.204.190.40 + 7 35.174.176.49 + 10 40.77.167.21 + 11 194.246.119.6 + 11 66.249.79.59 +``` + +- `45.5.184.72` is CIAT, who I already blocked and am waiting to hear from +- `2a01:4f8:140:3192::2` is BLEXbot, which should be handled by the Tomcat Crawler Session Manager Valve +- `2408:8214:7a00:868f:7c1e:e0f3:20c6:c142` is some stupid Chinese bot making malicious POST requests +- Anyways, it seems that the issue with CGSpace being "down" is actually because of CPU steal again!!! diff --git a/docs/2019-04/index.html b/docs/2019-04/index.html index 0fc45940e..26f69d2d2 100644 --- a/docs/2019-04/index.html +++ b/docs/2019-04/index.html @@ -38,7 +38,7 @@ $ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace - + @@ -81,9 +81,9 @@ $ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace "@type": "BlogPosting", "headline": "April, 2019", "url": "https://alanorth.github.io/cgspace-notes/2019-04/", - "wordCount": "1954", + "wordCount": "2174", "datePublished": "2019-04-01T09:00:43+03:00", - "dateModified": "2019-04-07T11:45:34+03:00", + "dateModified": "2019-04-07T18:08:38+03:00", "author": { "@type": "Person", "name": "Alan Orth" @@ -569,6 +569,47 @@ X-XSS-Protection: 1; mode=block
  • DSpace 5.10 upgraded to use GeoIP2, but we are on 5.8 so I just copied the missing database file from another server because it has been removed from MaxMind’s server as of 2018-04-01
  • Now I made 100 requests and I see them in the Solr statistics… fuck my life for wasting five hours debugging this
    • +
  • UptimeRobot said CGSpace went down and up a few times tonight, and my first instict was to check iostat 1 10 and I saw that CPU steal is around 10–30 percent right now…
    • +
  • The load average is super high right now, as I’ve noticed the last few times UptimeRobot said that CGSpace went down:
    • + + +
    $ cat /proc/loadavg 
+10.70 9.17 8.85 18/633 4198
+
    + + + +
    # zcat --force /var/log/nginx/{access,library-access}.log /var/log/nginx/{access,library-access}.log.1 | grep -E "07/Apr/2019:(18|19)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    116 40.77.167.128
+    128 207.46.13.219
+    129 167.114.64.100
+    159 207.46.13.129
+    179 207.46.13.33
+    188 2408:8214:7a00:868f:7c1e:e0f3:20c6:c142
+    195 66.249.79.59
+    363 40.77.167.21
+    740 2a01:4f8:140:3192::2
+   3711 45.5.184.72
+# zcat --force /var/log/nginx/{rest,oai}.log /var/log/nginx/{rest,oai}.log.1 | grep -E "07/Apr/2019:(18|19)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+      3 2a02:1206:4576:bae0:95ff:8bbd:a274:6223
+      3 66.249.79.62
+      3 66.249.83.196
+      5 82.145.222.150
+      6 2a01:4f9:2b:1263::2
+      6 41.204.190.40
+      7 35.174.176.49
+     10 40.77.167.21
+     11 194.246.119.6
+     11 66.249.79.59
+
    + + diff --git a/docs/robots.txt b/docs/robots.txt index fc9a8bbb4..9ea3f6b9d 100644 --- a/docs/robots.txt +++ b/docs/robots.txt @@ -46,7 +46,7 @@ Disallow: /cgspace-notes/2015-12/ Disallow: /cgspace-notes/2015-11/ Disallow: /cgspace-notes/ Disallow: /cgspace-notes/categories/ -Disallow: /cgspace-notes/categories/notes/ Disallow: /cgspace-notes/tags/notes/ +Disallow: /cgspace-notes/categories/notes/ Disallow: /cgspace-notes/posts/ Disallow: /cgspace-notes/tags/ diff --git a/docs/sitemap.xml b/docs/sitemap.xml index 990e9dc2b..afad8e25d 100644 --- a/docs/sitemap.xml +++ b/docs/sitemap.xml @@ -4,7 +4,7 @@ https://alanorth.github.io/cgspace-notes/2019-04/ - 2019-04-07T11:45:34+03:00 + 2019-04-07T18:08:38+03:00 @@ -219,7 +219,7 @@ https://alanorth.github.io/cgspace-notes/ - 2019-04-07T11:45:34+03:00 + 2019-04-07T18:08:38+03:00 0 @@ -228,27 +228,27 @@ 0 + + https://alanorth.github.io/cgspace-notes/tags/notes/ + 2019-04-07T18:08:38+03:00 + 0 + + https://alanorth.github.io/cgspace-notes/categories/notes/ 2018-03-09T22:10:33+02:00 0 - - https://alanorth.github.io/cgspace-notes/tags/notes/ - 2019-04-07T11:45:34+03:00 - 0 - - https://alanorth.github.io/cgspace-notes/posts/ - 2019-04-07T11:45:34+03:00 + 2019-04-07T18:08:38+03:00 0 https://alanorth.github.io/cgspace-notes/tags/ - 2019-04-07T11:45:34+03:00 + 2019-04-07T18:08:38+03:00 0