Add notes for 2017-12-07

content/post/2017-12.md

@@ -106,3 +106,61 @@ $ grep 2.86.122.76 /home/cgspace.cgiar.org/log/dspace.log.2017-12-01 | grep -o -
 ## 2017-12-06
 
 - Linode alerted again that the CPU usage on CGSpace was high this morning from 6 to 8 AM
+- Uptime Robot alerted that the server went down and up around 8:53 this morning
+- Uptime Robot alerted that CGSpace was down and up again a few minutes later
+- I don't see any errors in the DSpace logs but I see in nginx's access.log that UptimeRobot was returned with HTTP 499 status (Client Closed Request)
+- Looking at the REST API logs I see some new client IP I haven't noticed before:
+
+```
+# cat /var/log/nginx/rest.log /var/log/nginx/rest.log.1 | grep -E "6/Dec/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+     18 95.108.181.88
+     19 68.180.229.254
+     30 207.46.13.151
+     33 207.46.13.110
+     38 40.77.167.20
+     41 157.55.39.223
+     82 104.196.152.243
+   1529 50.116.102.77
+   4005 70.32.83.92
+   6045 45.5.184.196
+```
+
+- 50.116.102.77 is apparently in the US on websitewelcome.com
+
+## 2017-12-07
+
+- Uptime Robot reported a few times today that CGSpace was down and then up
+- At one point Tsega restarted Tomcat
+- I never got any alerts about high load from Linode though...
+- I looked just now and see that there are 121 PostgreSQL connections!
+- The top users right now are:
+
+```
+# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -E "7/Dec/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail 
+    838 40.77.167.11
+    939 66.249.66.223
+   1149 66.249.66.206
+   1316 207.46.13.110
+   1322 207.46.13.151
+   1323 2001:da8:203:2224:c912:1106:d94f:9189
+   1414 157.55.39.223
+   2378 104.196.152.243
+   2662 66.249.66.219
+   5110 124.17.34.60
+```
+
+- We've never seen 124.17.34.60 yet, but it's really hammering us!
+- Apparently it is from China, and here is one of its user agents:
+
+```
+Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.2; Win64; x64; Trident/7.0; LCTE)
+```
+
+- It is responsible for 4,500 Tomcat sessions today alone:
+
+```
+$ grep 124.17.34.60 /home/cgspace.cgiar.org/log/dspace.log.2017-12-07 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+4574
+```
+
+- I've adjusted the nginx IP mapping that I set up last month to account for 124.17.34.60 and 124.17.34.59 using a regex, as it's the same bot on the same subnet

public/2017-12/index.html

@@ -23,7 +23,7 @@ The list of connections to XMLUI and REST API for today:
 
 <meta property="article:published_time" content="2017-12-01T13:53:54&#43;03:00"/>
 
-<meta property="article:modified_time" content="2017-12-05T17:57:02&#43;03:00"/>
+<meta property="article:modified_time" content="2017-12-06T09:51:05&#43;03:00"/>
 
 
 
@@ -56,9 +56,9 @@ The list of connections to XMLUI and REST API for today:
   "@type": "BlogPosting",
   "headline": "December, 2017",
   "url": "https://alanorth.github.io/cgspace-notes/2017-12/",
-  "wordCount": "516",
+  "wordCount": "823",
   "datePublished": "2017-12-01T13:53:54&#43;03:00",
-  "dateModified": "2017-12-05T17:57:02&#43;03:00",
+  "dateModified": "2017-12-06T09:51:05&#43;03:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"
@@ -237,6 +237,70 @@ The list of connections to XMLUI and REST API for today:
 
 <ul>
 <li>Linode alerted again that the CPU usage on CGSpace was high this morning from 6 to 8 AM</li>
+<li>Uptime Robot alerted that the server went down and up around 8:53 this morning</li>
+<li>Uptime Robot alerted that CGSpace was down and up again a few minutes later</li>
+<li>I don&rsquo;t see any errors in the DSpace logs but I see in nginx&rsquo;s access.log that UptimeRobot was returned with HTTP 499 status (Client Closed Request)</li>
+<li>Looking at the REST API logs I see some new client IP I haven&rsquo;t noticed before:</li>
+</ul>
+
+<pre><code># cat /var/log/nginx/rest.log /var/log/nginx/rest.log.1 | grep -E &quot;6/Dec/2017&quot; | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+     18 95.108.181.88
+     19 68.180.229.254
+     30 207.46.13.151
+     33 207.46.13.110
+     38 40.77.167.20
+     41 157.55.39.223
+     82 104.196.152.243
+   1529 50.116.102.77
+   4005 70.32.83.92
+   6045 45.5.184.196
+</code></pre>
+
+<ul>
+<li>50.116.102.77 is apparently in the US on websitewelcome.com</li>
+</ul>
+
+<h2 id="2017-12-07">2017-12-07</h2>
+
+<ul>
+<li>Uptime Robot reported a few times today that CGSpace was down and then up</li>
+<li>At one point Tsega restarted Tomcat</li>
+<li>I never got any alerts about high load from Linode though&hellip;</li>
+<li>I looked just now and see that there are 121 PostgreSQL connections!</li>
+<li>The top users right now are:</li>
+</ul>
+
+<pre><code># cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -E &quot;7/Dec/2017&quot; | awk '{print $1}' | sort -n | uniq -c | sort -h | tail 
+    838 40.77.167.11
+    939 66.249.66.223
+   1149 66.249.66.206
+   1316 207.46.13.110
+   1322 207.46.13.151
+   1323 2001:da8:203:2224:c912:1106:d94f:9189
+   1414 157.55.39.223
+   2378 104.196.152.243
+   2662 66.249.66.219
+   5110 124.17.34.60
+</code></pre>
+
+<ul>
+<li>We&rsquo;ve never seen 124.17.34.60 yet, but it&rsquo;s really hammering us!</li>
+<li>Apparently it is from China, and here is one of its user agents:</li>
+</ul>
+
+<pre><code>Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.2; Win64; x64; Trident/7.0; LCTE)
+</code></pre>
+
+<ul>
+<li>It is responsible for 4,500 Tomcat sessions today alone:</li>
+</ul>
+
+<pre><code>$ grep 124.17.34.60 /home/cgspace.cgiar.org/log/dspace.log.2017-12-07 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+4574
+</code></pre>
+
+<ul>
+<li>I&rsquo;ve adjusted the nginx IP mapping that I set up last month to account for 124.17.34.60 and 124.17.34.59 using a regex, as it&rsquo;s the same bot on the same subnet</li>
 </ul>
 
   

public/robots.txt

@@ -30,7 +30,7 @@ Disallow: /cgspace-notes/2015-12/
 Disallow: /cgspace-notes/2015-11/
 Disallow: /cgspace-notes/
 Disallow: /cgspace-notes/categories/
-Disallow: /cgspace-notes/tags/notes/
 Disallow: /cgspace-notes/categories/notes/
+Disallow: /cgspace-notes/tags/notes/
 Disallow: /cgspace-notes/post/
 Disallow: /cgspace-notes/tags/

public/sitemap.xml

@@ -4,7 +4,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2017-12/</loc>
-    <lastmod>2017-12-05T17:57:02+03:00</lastmod>
+    <lastmod>2017-12-06T09:51:05+03:00</lastmod>
   </url>
   
   <url>
@@ -139,7 +139,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2017-12-05T17:57:02+03:00</lastmod>
+    <lastmod>2017-12-06T09:51:05+03:00</lastmod>
     <priority>0</priority>
   </url>
   
@@ -148,27 +148,27 @@
     <priority>0</priority>
   </url>
   
-  <url>
-    <loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
-    <lastmod>2017-12-05T17:57:02+03:00</lastmod>
-    <priority>0</priority>
-  </url>
-  
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
     <lastmod>2017-09-28T12:00:49+03:00</lastmod>
     <priority>0</priority>
   </url>
   
+  <url>
+    <loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
+    <lastmod>2017-12-06T09:51:05+03:00</lastmod>
+    <priority>0</priority>
+  </url>
+  
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/post/</loc>
-    <lastmod>2017-12-05T17:57:02+03:00</lastmod>
+    <lastmod>2017-12-06T09:51:05+03:00</lastmod>
     <priority>0</priority>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/tags/</loc>
-    <lastmod>2017-12-05T17:57:02+03:00</lastmod>
+    <lastmod>2017-12-06T09:51:05+03:00</lastmod>
     <priority>0</priority>
   </url>