Add notes for 2017-11-09

content/post/2017-11.md

@@ -476,3 +476,29 @@ proxy_set_header User-Agent $ua;
 - I will whip up something in nginx later
 - Run system updates on CGSpace and reboot the server
 - Re-deploy latest `5_x-prod` branch on CGSpace and DSpace Test (includes the clickable thumbnails, CCAFS phase II project tags, and updated news text)
+
+## 2017-11-09
+
+- Awesome, it seems my bot mapping stuff in nginx actually reduced the number of Tomcat sessions used by the CIAT scraper today, total requests and unique sessions:
+
+```
+# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep '09/Nov/2017' | grep -c 104.196.152.243 
+5769
+$ grep 104.196.152.243 dspace.log.2017-11-09 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+223
+```
+
+- Versus the same stats for yesterday and the day before:
+
+```
+# zcat -f -- /var/log/nginx/access.log.1 /var/log/nginx/access.log.2.gz | grep '08/Nov/2017' | grep -c 104.196.152.243 
+10216
+$ grep 104.196.152.243 dspace.log.2017-11-08 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+2592
+# zcat -f -- /var/log/nginx/access.log.2.gz /var/log/nginx/access.log.3.gz | grep '07/Nov/2017' | grep -c 104.196.152.243                                          
+8120
+$ grep 104.196.152.243 dspace.log.2017-11-07 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l         
+3506
+```
+
+- The number of total requests vary by a few thousand, but the number of sessions is over *ten times less*!

public/2017-11/index.html

@@ -38,7 +38,7 @@ COPY 54701
 
 <meta property="article:published_time" content="2017-11-02T09:37:54&#43;02:00"/>
 
-<meta property="article:modified_time" content="2017-11-08T22:26:37&#43;02:00"/>
+<meta property="article:modified_time" content="2017-11-08T22:36:15&#43;02:00"/>
 
 
 
@@ -86,9 +86,9 @@ COPY 54701
   "@type": "BlogPosting",
   "headline": "November, 2017",
   "url": "https://alanorth.github.io/cgspace-notes/2017-11/",
-  "wordCount": "2725",
+  "wordCount": "2877",
   "datePublished": "2017-11-02T09:37:54&#43;02:00",
-  "dateModified": "2017-11-08T22:26:37&#43;02:00",
+  "dateModified": "2017-11-08T22:36:15&#43;02:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"
@@ -678,6 +678,36 @@ proxy_set_header User-Agent $ua;
 <li>Re-deploy latest <code>5_x-prod</code> branch on CGSpace and DSpace Test (includes the clickable thumbnails, CCAFS phase II project tags, and updated news text)</li>
 </ul>
 
+<h2 id="2017-11-09">2017-11-09</h2>
+
+<ul>
+<li>Awesome, it seems my bot mapping stuff in nginx actually reduced the number of Tomcat sessions used by the CIAT scraper today, total requests and unique sessions:</li>
+</ul>
+
+<pre><code># cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep '09/Nov/2017' | grep -c 104.196.152.243 
+5769
+$ grep 104.196.152.243 dspace.log.2017-11-09 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+223
+</code></pre>
+
+<ul>
+<li>Versus the same stats for yesterday and the day before:</li>
+</ul>
+
+<pre><code># zcat -f -- /var/log/nginx/access.log.1 /var/log/nginx/access.log.2.gz | grep '08/Nov/2017' | grep -c 104.196.152.243 
+10216
+$ grep 104.196.152.243 dspace.log.2017-11-08 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+2592
+# zcat -f -- /var/log/nginx/access.log.2.gz /var/log/nginx/access.log.3.gz | grep '07/Nov/2017' | grep -c 104.196.152.243                                          
+8120
+$ grep 104.196.152.243 dspace.log.2017-11-07 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l         
+3506
+</code></pre>
+
+<ul>
+<li>The number of total requests vary by a few thousand, but the number of sessions is over <em>ten times less</em>!</li>
+</ul>
+
   
 
   

public/sitemap.xml

@@ -4,7 +4,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2017-11/</loc>
-    <lastmod>2017-11-08T22:26:37+02:00</lastmod>
+    <lastmod>2017-11-08T22:36:15+02:00</lastmod>
   </url>
   
   <url>
@@ -134,7 +134,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2017-11-08T22:26:37+02:00</lastmod>
+    <lastmod>2017-11-08T22:36:15+02:00</lastmod>
     <priority>0</priority>
   </url>
   
@@ -145,7 +145,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
-    <lastmod>2017-11-08T22:26:37+02:00</lastmod>
+    <lastmod>2017-11-08T22:36:15+02:00</lastmod>
     <priority>0</priority>
   </url>
   
@@ -157,13 +157,13 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/post/</loc>
-    <lastmod>2017-11-08T22:26:37+02:00</lastmod>
+    <lastmod>2017-11-08T22:36:15+02:00</lastmod>
     <priority>0</priority>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/tags/</loc>
-    <lastmod>2017-11-08T22:26:37+02:00</lastmod>
+    <lastmod>2017-11-08T22:36:15+02:00</lastmod>
     <priority>0</priority>
   </url>