Add notes for 2017-10-26

content/post/2017-10.md

@@ -159,3 +159,30 @@ http://library.cgiar.org/browse?value=Intellectual%20Assets%20Reports&type=subje
 - Re-deploy CGSpace from latest `5_x-prod` (adds ISI Journal to search filters and adds Discovery index for CGIAR Library `systemsubject`)
 - Deploy nginx redirect fixes to catch CGIAR Library browse links (redirect to their community and translate subject→systemsubject)
 - Run migration of CGSpace server (linode18) for Linode security alert, which took 42 minutes of downtime
+
+## 2017-10-26
+
+- In the last 24 hours we've gotten a few alerts from Linode that there was high CPU and outgoing traffic on CGSpace
+- Uptime Robot even noticed CGSpace go "down" for a few minutes
+- In other news, I was trying to look at a question about stats raised by Magdalena and then CGSpace went down due to SQL connection pool
+- Looking at the PostgreSQL activity I see there are 93 connections, but after a minute or two they went down and CGSpace came back up
+- Annnd I reloaded the Atmire Usage Stats module and the connections shot back up and CGSpace went down again
+- Still not sure where the load is coming from right now, but it's clear why there were so many alerts yesterday on the 25th!
+
+```
+# grep -o -E 'session_id=[A-Z0-9]{32}' dspace.log.2017-10-25 | sort -n | uniq | wc -l
+18022
+```
+
+- Compared to other days there were two or three times the number of requests yesterday!
+
+```
+# grep -o -E 'session_id=[A-Z0-9]{32}' dspace.log.2017-10-23 | sort -n | uniq | wc -l
+3141
+# grep -o -E 'session_id=[A-Z0-9]{32}' dspace.log.2017-10-26 | sort -n | uniq | wc -l
+7851
+```
+
+- I still have no idea what was causing the load to go up today
+- I finally investigated Magdalena's issue with the item download stats and now I can't reproduce it: I get the same number of downloads reported in the stats widget on the item page, the "Most Popular Items" page, and in Usage Stats
+- I think it might have been an issue with the statistics not being fresh

public/2017-10/index.html

@@ -28,7 +28,7 @@ Add Katherine Lutz to the groups for content sumission and edit steps of the CGI
 
 <meta property="article:published_time" content="2017-10-01T08:07:54&#43;03:00"/>
 
-<meta property="article:modified_time" content="2017-10-22T23:51:48&#43;03:00"/>
+<meta property="article:modified_time" content="2017-10-23T09:58:33&#43;03:00"/>
 
 
 
@@ -66,9 +66,9 @@ Add Katherine Lutz to the groups for content sumission and edit steps of the CGI
   "@type": "BlogPosting",
   "headline": "October, 2017",
   "url": "https://alanorth.github.io/cgspace-notes/2017-10/",
-  "wordCount": "1194",
+  "wordCount": "1451",
   "datePublished": "2017-10-01T08:07:54&#43;03:00",
-  "dateModified": "2017-10-22T23:51:48&#43;03:00",
+  "dateModified": "2017-10-23T09:58:33&#43;03:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"
@@ -318,6 +318,37 @@ Add Katherine Lutz to the groups for content sumission and edit steps of the CGI
 <li>Run migration of CGSpace server (linode18) for Linode security alert, which took 42 minutes of downtime</li>
 </ul>
 
+<h2 id="2017-10-26">2017-10-26</h2>
+
+<ul>
+<li>In the last 24 hours we&rsquo;ve gotten a few alerts from Linode that there was high CPU and outgoing traffic on CGSpace</li>
+<li>Uptime Robot even noticed CGSpace go &ldquo;down&rdquo; for a few minutes</li>
+<li>In other news, I was trying to look at a question about stats raised by Magdalena and then CGSpace went down due to SQL connection pool</li>
+<li>Looking at the PostgreSQL activity I see there are 93 connections, but after a minute or two they went down and CGSpace came back up</li>
+<li>Annnd I reloaded the Atmire Usage Stats module and the connections shot back up and CGSpace went down again</li>
+<li>Still not sure where the load is coming from right now, but it&rsquo;s clear why there were so many alerts yesterday on the 25th!</li>
+</ul>
+
+<pre><code># grep -o -E 'session_id=[A-Z0-9]{32}' dspace.log.2017-10-25 | sort -n | uniq | wc -l
+18022
+</code></pre>
+
+<ul>
+<li>Compared to other days there were two or three times the number of requests yesterday!</li>
+</ul>
+
+<pre><code># grep -o -E 'session_id=[A-Z0-9]{32}' dspace.log.2017-10-23 | sort -n | uniq | wc -l
+3141
+# grep -o -E 'session_id=[A-Z0-9]{32}' dspace.log.2017-10-26 | sort -n | uniq | wc -l
+7851
+</code></pre>
+
+<ul>
+<li>I still have no idea what was causing the load to go up today</li>
+<li>I finally investigated Magdalena&rsquo;s issue with the item download stats and now I can&rsquo;t reproduce it: I get the same number of downloads reported in the stats widget on the item page, the &ldquo;Most Popular Items&rdquo; page, and in Usage Stats</li>
+<li>I think it might have been an issue with the statistics not being fresh</li>
+</ul>
+
   
 
   

public/robots.txt

@@ -28,7 +28,7 @@ Disallow: /cgspace-notes/2015-12/
 Disallow: /cgspace-notes/2015-11/
 Disallow: /cgspace-notes/
 Disallow: /cgspace-notes/categories/
-Disallow: /cgspace-notes/tags/notes/
 Disallow: /cgspace-notes/categories/notes/
+Disallow: /cgspace-notes/tags/notes/
 Disallow: /cgspace-notes/post/
 Disallow: /cgspace-notes/tags/

public/sitemap.xml

@@ -4,7 +4,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2017-10/</loc>
-    <lastmod>2017-10-22T23:51:48+03:00</lastmod>
+    <lastmod>2017-10-23T09:58:33+03:00</lastmod>
   </url>
   
   <url>
@@ -129,7 +129,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2017-10-22T23:51:48+03:00</lastmod>
+    <lastmod>2017-10-23T09:58:33+03:00</lastmod>
     <priority>0</priority>
   </url>
   
@@ -138,27 +138,27 @@
     <priority>0</priority>
   </url>
   
-  <url>
-    <loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
-    <lastmod>2017-10-22T23:51:48+03:00</lastmod>
-    <priority>0</priority>
-  </url>
-  
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
     <lastmod>2017-09-28T12:00:49+03:00</lastmod>
     <priority>0</priority>
   </url>
   
+  <url>
+    <loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
+    <lastmod>2017-10-23T09:58:33+03:00</lastmod>
+    <priority>0</priority>
+  </url>
+  
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/post/</loc>
-    <lastmod>2017-10-22T23:51:48+03:00</lastmod>
+    <lastmod>2017-10-23T09:58:33+03:00</lastmod>
     <priority>0</priority>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/tags/</loc>
-    <lastmod>2017-10-22T23:51:48+03:00</lastmod>
+    <lastmod>2017-10-23T09:58:33+03:00</lastmod>
     <priority>0</priority>
   </url>