Add notes for 2022-10-17

content/posts/2022-10.md

@@ -500,4 +500,34 @@ $ convert 10568-103447-dspace-step1.pdf.jpg -thumbnail 600x600 -unsharp 1.5x1+0.
 - I'm going to turn the REST API cache on CGSpace back on to see if the ilri.org publications thing gets broken again
 - Start a harvest on AReS
 
+## 2022-10-16
+
+- The harvest on AReS finished but somehow there are 1,100 less items than the previous indexing... hmmm
+  - I don't see any hits from MELSpace there so I will start another harvest...
+  - After starting the harvesting the load on the server went up to 20 and UptimeRobot said CGSpace was down for three hours, sigh
+  - I stopped the harvesting and the load went down immediately
+  - I am trying to find a pattern with the load on Sundays
+- I see this in the AReS backend logs:
+
+```console
+[Nest] 1   - 10/16/2022, 6:42:04 PM   [HarvesterService] Starting Harvest =>0
+[Nest] 1   - 10/16/2022, 6:42:07 PM   [HarvesterService] Starting Harvest =>101555
+[Nest] 1   - 10/16/2022, 6:42:10 PM   [HarvesterService] Starting Harvest =>4936
+```
+
+- Which means MELSpace is having some issue
+- I'm not sure what was going on on CGSpace yesterday, but the load was indeed very high according to Munin:
+
+![CGSpace CPU load day](/cgspace-notes/2022/10/cpu-day.png)
+
+- The pattern is clear on Sundays if you look at the past month:
+
+![CGSpace CPU load month](/cgspace-notes/2022/10/cpu-month.png)
+
+- I have yet to find an increased nginx request pattern correlating with the increased load, but looking back on the last year it seems something started happening around March, 2022, and also I start seeing CPU steal in July (red coming from the top of the graph):
+
+![CGSpace CPU load year](/cgspace-notes/2022/10/cpu-year.png)
+
+- The amount of CPU steal is very low if I look at it now, around 1 or 2 percent, but what's happening now reminds me of the mysterious load problems I had in 2019-03 that were due to CPU steal
+
 <!-- vim: set sw=2 ts=2: -->

docs/2022-10/index.html

@@ -20,7 +20,7 @@ I filed an issue to ask about Java 11+ support
 <meta property="og:type" content="article" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2022-10/" />
 <meta property="article:published_time" content="2022-10-01T19:45:36+03:00" />
-<meta property="article:modified_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="article:modified_time" content="2022-10-15T17:38:47+03:00" />
 
 
 
@@ -46,9 +46,9 @@ I filed an issue to ask about Java 11&#43; support
   "@type": "BlogPosting",
   "headline": "October, 2022",
   "url": "https://alanorth.github.io/cgspace-notes/2022-10/",
-  "wordCount": "2090",
+  "wordCount": "2332",
   "datePublished": "2022-10-01T19:45:36+03:00",
-  "dateModified": "2022-10-13T07:10:59+03:00",
+  "dateModified": "2022-10-15T17:38:47+03:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"
@@ -655,6 +655,37 @@ I filed an issue to ask about Java 11&#43; support
 <li>I&rsquo;m going to turn the REST API cache on CGSpace back on to see if the ilri.org publications thing gets broken again</li>
 <li>Start a harvest on AReS</li>
 </ul>
+<h2 id="2022-10-16">2022-10-16</h2>
+<ul>
+<li>The harvest on AReS finished but somehow there are 1,100 less items than the previous indexing&hellip; hmmm
+<ul>
+<li>I don&rsquo;t see any hits from MELSpace there so I will start another harvest&hellip;</li>
+<li>After starting the harvesting the load on the server went up to 20 and UptimeRobot said CGSpace was down for three hours, sigh</li>
+<li>I stopped the harvesting and the load went down immediately</li>
+<li>I am trying to find a pattern with the load on Sundays</li>
+</ul>
+</li>
+<li>I see this in the AReS backend logs:</li>
+</ul>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>[Nest] 1   - 10/16/2022, 6:42:04 PM   [HarvesterService] Starting Harvest =&gt;0
+</span></span><span style="display:flex;"><span>[Nest] 1   - 10/16/2022, 6:42:07 PM   [HarvesterService] Starting Harvest =&gt;101555
+</span></span><span style="display:flex;"><span>[Nest] 1   - 10/16/2022, 6:42:10 PM   [HarvesterService] Starting Harvest =&gt;4936
+</span></span></code></pre></div><ul>
+<li>Which means MELSpace is having some issue</li>
+<li>I&rsquo;m not sure what was going on on CGSpace yesterday, but the load was indeed very high according to Munin:</li>
+</ul>
+<p><img src="/cgspace-notes/2022/10/cpu-day.png" alt="CGSpace CPU load day"></p>
+<ul>
+<li>The pattern is clear on Sundays if you look at the past month:</li>
+</ul>
+<p><img src="/cgspace-notes/2022/10/cpu-month.png" alt="CGSpace CPU load month"></p>
+<ul>
+<li>I have yet to find an increased nginx request pattern correlating with the increased load, but looking back on the last year it seems something started happening around March, 2022, and also I start seeing CPU steal in July (red coming from the top of the graph):</li>
+</ul>
+<p><img src="/cgspace-notes/2022/10/cpu-year.png" alt="CGSpace CPU load year"></p>
+<ul>
+<li>The amount of CPU steal is very low if I look at it now, around 1 or 2 percent, but what&rsquo;s happening now reminds me of the mysterious load problems I had in 2019-03 that were due to CPU steal</li>
+</ul>
 <!-- raw HTML omitted -->
 
   

docs/categories/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/categories/notes/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/categories/notes/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/categories/notes/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/categories/notes/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/categories/notes/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/categories/notes/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/categories/notes/page/7/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/page/7/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/page/8/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/page/9/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/posts/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/posts/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/posts/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/posts/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/posts/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/posts/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/posts/page/7/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/posts/page/8/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/posts/page/9/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-10-13T07:10:59+03:00" />
+<meta property="og:updated_time" content="2022-10-15T17:38:47+03:00" />
 
 
 

docs/sitemap.xml

@@ -3,19 +3,19 @@
   xmlns:xhtml="http://www.w3.org/1999/xhtml">
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/</loc>
-    <lastmod>2022-10-13T07:10:59+03:00</lastmod>
+    <lastmod>2022-10-15T17:38:47+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2022-10-13T07:10:59+03:00</lastmod>
+    <lastmod>2022-10-15T17:38:47+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
-    <lastmod>2022-10-13T07:10:59+03:00</lastmod>
+    <lastmod>2022-10-15T17:38:47+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/2022-10/</loc>
-    <lastmod>2022-10-13T07:10:59+03:00</lastmod>
+    <lastmod>2022-10-15T17:38:47+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
-    <lastmod>2022-10-13T07:10:59+03:00</lastmod>
+    <lastmod>2022-10-15T17:38:47+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/2022-09/</loc>
     <lastmod>2022-09-30T17:29:50+03:00</lastmod>