Update notes for 2019-04-07

content/posts/2019-04.md

@@ -351,5 +351,44 @@ $ http --print b 'http://localhost:8080/solr/statistics/select?q=type%3A0+AND+ti
   - See: [DS-3832](https://jira.duraspace.org/browse/DS-3832)
   - DSpace 5.10 upgraded to use GeoIP2, but we are on 5.8 so I just copied the missing database file from another server because it has been *removed* from MaxMind's server as of 2018-04-01
   - Now I made 100 requests and I see them in the Solr statistics... fuck my life for wasting five hours debugging this
+- UptimeRobot said CGSpace went down and up a few times tonight, and my first instict was to check `iostat 1 10` and I saw that CPU steal is around 10–30 percent right now...
+- The load average is super high right now, as I've noticed the last few times UptimeRobot said that CGSpace went down:
+
+```
+$ cat /proc/loadavg 
+10.70 9.17 8.85 18/633 4198
+```
+
+- According to the server logs there is actually not much going on right now:
+
+```
+# zcat --force /var/log/nginx/{access,library-access}.log /var/log/nginx/{access,library-access}.log.1 | grep -E "07/Apr/2019:(18|19)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    116 40.77.167.128
+    128 207.46.13.219
+    129 167.114.64.100
+    159 207.46.13.129
+    179 207.46.13.33
+    188 2408:8214:7a00:868f:7c1e:e0f3:20c6:c142
+    195 66.249.79.59
+    363 40.77.167.21
+    740 2a01:4f8:140:3192::2
+   3711 45.5.184.72
+# zcat --force /var/log/nginx/{rest,oai}.log /var/log/nginx/{rest,oai}.log.1 | grep -E "07/Apr/2019:(18|19)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+      3 2a02:1206:4576:bae0:95ff:8bbd:a274:6223
+      3 66.249.79.62
+      3 66.249.83.196
+      5 82.145.222.150
+      6 2a01:4f9:2b:1263::2
+      6 41.204.190.40
+      7 35.174.176.49
+     10 40.77.167.21
+     11 194.246.119.6
+     11 66.249.79.59
+```
+
+- `45.5.184.72` is CIAT, who I already blocked and am waiting to hear from
+- `2a01:4f8:140:3192::2` is BLEXbot, which should be handled by the Tomcat Crawler Session Manager Valve
+- `2408:8214:7a00:868f:7c1e:e0f3:20c6:c142` is some stupid Chinese bot making malicious POST requests
+- Anyways, it seems that the issue with CGSpace being "down" is actually because of CPU steal again!!!
 
 <!-- vim: set sw=2 ts=2: -->

docs/2019-04/index.html

@@ -38,7 +38,7 @@ $ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace
 <meta property="og:type" content="article" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2019-04/" />
 <meta property="article:published_time" content="2019-04-01T09:00:43&#43;03:00"/>
-<meta property="article:modified_time" content="2019-04-07T11:45:34&#43;03:00"/>
+<meta property="article:modified_time" content="2019-04-07T18:08:38&#43;03:00"/>
 
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="April, 2019"/>
@@ -81,9 +81,9 @@ $ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace
   "@type": "BlogPosting",
   "headline": "April, 2019",
   "url": "https://alanorth.github.io/cgspace-notes/2019-04/",
-  "wordCount": "1954",
+  "wordCount": "2174",
   "datePublished": "2019-04-01T09:00:43&#43;03:00",
-  "dateModified": "2019-04-07T11:45:34&#43;03:00",
+  "dateModified": "2019-04-07T18:08:38&#43;03:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"
@@ -569,6 +569,47 @@ X-XSS-Protection: 1; mode=block
 <li>DSpace 5.10 upgraded to use GeoIP2, but we are on 5.8 so I just copied the missing database file from another server because it has been <em>removed</em> from MaxMind&rsquo;s server as of 2018-04-01</li>
 <li>Now I made 100 requests and I see them in the Solr statistics&hellip; fuck my life for wasting five hours debugging this</li>
 </ul></li>
+<li>UptimeRobot said CGSpace went down and up a few times tonight, and my first instict was to check <code>iostat 1 10</code> and I saw that CPU steal is around 10–30 percent right now&hellip;</li>
+<li>The load average is super high right now, as I&rsquo;ve noticed the last few times UptimeRobot said that CGSpace went down:</li>
+</ul>
+
+<pre><code>$ cat /proc/loadavg 
+10.70 9.17 8.85 18/633 4198
+</code></pre>
+
+<ul>
+<li>According to the server logs there is actually not much going on right now:</li>
+</ul>
+
+<pre><code># zcat --force /var/log/nginx/{access,library-access}.log /var/log/nginx/{access,library-access}.log.1 | grep -E &quot;07/Apr/2019:(18|19)&quot; | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    116 40.77.167.128
+    128 207.46.13.219
+    129 167.114.64.100
+    159 207.46.13.129
+    179 207.46.13.33
+    188 2408:8214:7a00:868f:7c1e:e0f3:20c6:c142
+    195 66.249.79.59
+    363 40.77.167.21
+    740 2a01:4f8:140:3192::2
+   3711 45.5.184.72
+# zcat --force /var/log/nginx/{rest,oai}.log /var/log/nginx/{rest,oai}.log.1 | grep -E &quot;07/Apr/2019:(18|19)&quot; | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+      3 2a02:1206:4576:bae0:95ff:8bbd:a274:6223
+      3 66.249.79.62
+      3 66.249.83.196
+      5 82.145.222.150
+      6 2a01:4f9:2b:1263::2
+      6 41.204.190.40
+      7 35.174.176.49
+     10 40.77.167.21
+     11 194.246.119.6
+     11 66.249.79.59
+</code></pre>
+
+<ul>
+<li><code>45.5.184.72</code> is CIAT, who I already blocked and am waiting to hear from</li>
+<li><code>2a01:4f8:140:3192::2</code> is BLEXbot, which should be handled by the Tomcat Crawler Session Manager Valve</li>
+<li><code>2408:8214:7a00:868f:7c1e:e0f3:20c6:c142</code> is some stupid Chinese bot making malicious POST requests</li>
+<li>Anyways, it seems that the issue with CGSpace being &ldquo;down&rdquo; is actually because of CPU steal again!!!</li>
 </ul>
 
 <!-- vim: set sw=2 ts=2: -->

docs/robots.txt

@@ -46,7 +46,7 @@ Disallow: /cgspace-notes/2015-12/
 Disallow: /cgspace-notes/2015-11/
 Disallow: /cgspace-notes/
 Disallow: /cgspace-notes/categories/
-Disallow: /cgspace-notes/categories/notes/
 Disallow: /cgspace-notes/tags/notes/
+Disallow: /cgspace-notes/categories/notes/
 Disallow: /cgspace-notes/posts/
 Disallow: /cgspace-notes/tags/

docs/sitemap.xml

@@ -4,7 +4,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2019-04/</loc>
-    <lastmod>2019-04-07T11:45:34+03:00</lastmod>
+    <lastmod>2019-04-07T18:08:38+03:00</lastmod>
   </url>
   
   <url>
@@ -219,7 +219,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2019-04-07T11:45:34+03:00</lastmod>
+    <lastmod>2019-04-07T18:08:38+03:00</lastmod>
     <priority>0</priority>
   </url>
   
@@ -228,27 +228,27 @@
     <priority>0</priority>
   </url>
   
+  <url>
+    <loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
+    <lastmod>2019-04-07T18:08:38+03:00</lastmod>
+    <priority>0</priority>
+  </url>
+  
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
     <lastmod>2018-03-09T22:10:33+02:00</lastmod>
     <priority>0</priority>
   </url>
   
-  <url>
-    <loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
-    <lastmod>2019-04-07T11:45:34+03:00</lastmod>
-    <priority>0</priority>
-  </url>
-  
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
-    <lastmod>2019-04-07T11:45:34+03:00</lastmod>
+    <lastmod>2019-04-07T18:08:38+03:00</lastmod>
     <priority>0</priority>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/tags/</loc>
-    <lastmod>2019-04-07T11:45:34+03:00</lastmod>
+    <lastmod>2019-04-07T18:08:38+03:00</lastmod>
     <priority>0</priority>
   </url>