Add notes for 2022-12-21

content/posts/2022-12.md

@@ -234,4 +234,23 @@ $ ./ilri/move-metadata-values.py -i /tmp/data.txt -db dspace -u dspace -p 'dom@i
 - RStudio is already in the ILRI bot overrides for DSpace so it shouldn't be causing any extra hits, but I'll put an HTTP 403 in the nginx config to tell the user to use the REST API
 - Start a harvest on AReS
 
+## 2022-12-21
+
+- I saw that load on CGSpace was over 20.0 for several hours
+  - I saw there were some stuck locks in PostgreSQL:
+
+```console
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+    948 dspaceApi
+     30 dspaceCli
+   1237 dspaceWeb
+```
+
+- Ah, it's likely there is something stuck because I see the load high since yesterday at 6AM, which is 24 hours now:
+
+![CPU load day](/cgspace-notes/2022/12/cpu-day.png)
+![PostgreSQL locks week](/cgspace-notes/2022/12/postgres_locks_ALL-week.png)
+
+- I ran all updates and restarted the server
+
 <!-- vim: set sw=2 ts=2: -->

docs/2022-12/index.html

@@ -20,7 +20,7 @@ Replace “East Asia” with “Eastern Asia” region on CGSpac
 <meta property="og:type" content="article" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2022-12/" />
 <meta property="article:published_time" content="2022-12-01T08:52:36+03:00" />
-<meta property="article:modified_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="article:modified_time" content="2022-12-19T07:03:13+02:00" />
 
 
 
@@ -46,9 +46,9 @@ Replace &ldquo;East Asia&rdquo; with &ldquo;Eastern Asia&rdquo; region on CGSpac
   "@type": "BlogPosting",
   "headline": "December, 2022",
   "url": "https://alanorth.github.io/cgspace-notes/2022-12/",
-  "wordCount": "1588",
+  "wordCount": "1672",
   "datePublished": "2022-12-01T08:52:36+03:00",
-  "dateModified": "2022-12-15T16:41:04+03:00",
+  "dateModified": "2022-12-19T07:03:13+02:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"
@@ -384,6 +384,26 @@ Replace &ldquo;East Asia&rdquo; with &ldquo;Eastern Asia&rdquo; region on CGSpac
 <li>RStudio is already in the ILRI bot overrides for DSpace so it shouldn&rsquo;t be causing any extra hits, but I&rsquo;ll put an HTTP 403 in the nginx config to tell the user to use the REST API</li>
 <li>Start a harvest on AReS</li>
 </ul>
+<h2 id="2022-12-21">2022-12-21</h2>
+<ul>
+<li>I saw that load on CGSpace was over 20.0 for several hours
+<ul>
+<li>I saw there were some stuck locks in PostgreSQL:</li>
+</ul>
+</li>
+</ul>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ psql -c <span style="color:#e6db74">&#39;SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;&#39;</span> | grep -o -E <span style="color:#e6db74">&#39;(dspaceWeb|dspaceApi|dspaceCli)&#39;</span> | sort | uniq -c
+</span></span><span style="display:flex;"><span>    948 dspaceApi
+</span></span><span style="display:flex;"><span>     30 dspaceCli
+</span></span><span style="display:flex;"><span>   1237 dspaceWeb
+</span></span></code></pre></div><ul>
+<li>Ah, it&rsquo;s likely there is something stuck because I see the load high since yesterday at 6AM, which is 24 hours now:</li>
+</ul>
+<p><img src="/cgspace-notes/2022/12/cpu-day.png" alt="CPU load day">
+<img src="/cgspace-notes/2022/12/postgres_locks_ALL-week.png" alt="PostgreSQL locks week"></p>
+<ul>
+<li>I ran all updates and restarted the server</li>
+</ul>
 <!-- raw HTML omitted -->
 
   

docs/categories/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/categories/notes/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/categories/notes/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/categories/notes/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/categories/notes/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/categories/notes/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/categories/notes/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/categories/notes/page/7/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/page/7/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/page/8/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/page/9/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/posts/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/posts/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/posts/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/posts/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/posts/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/posts/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/posts/page/7/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/posts/page/8/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/posts/page/9/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-12-15T16:41:04+03:00" />
+<meta property="og:updated_time" content="2022-12-19T07:03:13+02:00" />
 
 
 

docs/sitemap.xml

@@ -3,19 +3,19 @@
   xmlns:xhtml="http://www.w3.org/1999/xhtml">
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/</loc>
-    <lastmod>2022-12-15T16:41:04+03:00</lastmod>
+    <lastmod>2022-12-19T07:03:13+02:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2022-12-15T16:41:04+03:00</lastmod>
+    <lastmod>2022-12-19T07:03:13+02:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/2022-12/</loc>
-    <lastmod>2022-12-15T16:41:04+03:00</lastmod>
+    <lastmod>2022-12-19T07:03:13+02:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
-    <lastmod>2022-12-15T16:41:04+03:00</lastmod>
+    <lastmod>2022-12-19T07:03:13+02:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
-    <lastmod>2022-12-15T16:41:04+03:00</lastmod>
+    <lastmod>2022-12-19T07:03:13+02:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/2022-11/</loc>
     <lastmod>2022-12-03T10:46:29+03:00</lastmod>