Add notes for 2021-10-11

content/posts/2021-10.md

@@ -330,8 +330,42 @@ $ psql -h localhost -p 5433 -U postgres dspace7 -c "DELETE FROM schema_version W
 ```
 
 - Now DSpace 7 starts with my CGSpace data... nice
+  - The Discovery indexing still takes seven hours... fuck
 - I tested the `metadata-export` on DSpace 7.1-SNAPSHOT and it still has the duplicate items issue introduced by DS-4211
   - I filed a GitHub issue and notified nwoodward: https://github.com/DSpace/DSpace/issues/7988
 - Start a full reindex on AReS
 
+## 2021-10-11
+
+- Start a full Discovery reindex on my local DSpace 6.3 instance:
+
+```console
+$ /usr/bin/time -f %M:%e chrt -b 0 ~/dspace63/bin/dspace index-discovery -b
+Loading @mire database changes for module MQM
+Changes have been processed
+836140:6543.6
+```
+
+- So that's 1.8 hours versus 7 on DSpace 7, with the same database!
+- Several users wrote to me that CGSpace was slow recently
+  - Looking at the PostgreSQL database I see connections look normal, but locks for `dspaceWeb` are high:
+
+```console
+$ psql -c 'SELECT * FROM pg_stat_activity' | wc -l
+53
+$ psql -c "SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid" | wc -l
+1697
+$ psql -c "SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid WHERE application_name='dspaceWeb'" | wc -l
+1681
+```
+
+- Looking at Munin, I see there are indeed a higher number of locks starting on the morning of 2021-10-07:
+
+![PostgreSQL locks week](/cgspace-notes/2021/10/postgres_locks_ALL-week.png)
+
+- The only thing I did on 2021-10-07 was import a few thousand metadata corrections...
+- I restarted PostgreSQL (instead of restarting Tomcat), so let's see if that helps
+- I filed [a bug for the DSpace 6/7 duplicate values metadata import issue](https://github.com/DSpace/DSpace/issues/7989)
+- I tested the two patches for removing abandoned submissions from the workflow but unfortunately it seems that they are for the configurable aka XML workflow, and we are using the basic workflow
+
 <!-- vim: set sw=2 ts=2: -->

docs/2021-10/index.html

@@ -25,7 +25,7 @@ So we have 1879/7100 (26.46%) matching already
 <meta property="og:type" content="article" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2021-10/" />
 <meta property="article:published_time" content="2021-10-01T11:14:07+03:00" />
-<meta property="article:modified_time" content="2021-10-09T22:00:59+03:00" />
+<meta property="article:modified_time" content="2021-10-10T16:01:27+03:00" />
 
 
 
@@ -56,9 +56,9 @@ So we have 1879/7100 (26.46%) matching already
   "@type": "BlogPosting",
   "headline": "October, 2021",
   "url": "https://alanorth.github.io/cgspace-notes/2021-10/",
-  "wordCount": "2199",
+  "wordCount": "2424",
   "datePublished": "2021-10-01T11:14:07+03:00",
-  "dateModified": "2021-10-09T22:00:59+03:00",
+  "dateModified": "2021-10-10T16:01:27+03:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"
@@ -453,7 +453,11 @@ $ psql -h localhost -p 5433 -U postgres -c 'alter user dspacetest nosuperuser;'
 <pre tabindex="0"><code class="language-console" data-lang="console">$ psql -h localhost -p 5433 -U postgres dspace7 -c &quot;DELETE FROM schema_version WHERE description LIKE '%Atmire%' OR description LIKE '%CUA%' OR description LIKE '%cua%';&quot;
 $ psql -h localhost -p 5433 -U postgres dspace7 -c &quot;DELETE FROM schema_version WHERE version IN ('5.0.2017.09.25', '6.0.2017.01.30', '6.0.2017.09.25');&quot;
 </code></pre><ul>
-<li>Now DSpace 7 starts with my CGSpace data&hellip; nice</li>
+<li>Now DSpace 7 starts with my CGSpace data&hellip; nice
+<ul>
+<li>The Discovery indexing still takes seven hours&hellip; fuck</li>
+</ul>
+</li>
 <li>I tested the <code>metadata-export</code> on DSpace 7.1-SNAPSHOT and it still has the duplicate items issue introduced by DS-4211
 <ul>
 <li>I filed a GitHub issue and notified nwoodward: <a href="https://github.com/DSpace/DSpace/issues/7988">https://github.com/DSpace/DSpace/issues/7988</a></li>
@@ -461,6 +465,38 @@ $ psql -h localhost -p 5433 -U postgres dspace7 -c &quot;DELETE FROM schema_vers
 </li>
 <li>Start a full reindex on AReS</li>
 </ul>
+<h2 id="2021-10-11">2021-10-11</h2>
+<ul>
+<li>Start a full Discovery reindex on my local DSpace 6.3 instance:</li>
+</ul>
+<pre tabindex="0"><code class="language-console" data-lang="console">$ /usr/bin/time -f %M:%e chrt -b 0 ~/dspace63/bin/dspace index-discovery -b
+Loading @mire database changes for module MQM
+Changes have been processed
+836140:6543.6
+</code></pre><ul>
+<li>So that&rsquo;s 1.8 hours versus 7 on DSpace 7, with the same database!</li>
+<li>Several users wrote to me that CGSpace was slow recently
+<ul>
+<li>Looking at the PostgreSQL database I see connections look normal, but locks for <code>dspaceWeb</code> are high:</li>
+</ul>
+</li>
+</ul>
+<pre tabindex="0"><code class="language-console" data-lang="console">$ psql -c 'SELECT * FROM pg_stat_activity' | wc -l
+53
+$ psql -c &quot;SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid&quot; | wc -l
+1697
+$ psql -c &quot;SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid WHERE application_name='dspaceWeb'&quot; | wc -l
+1681
+</code></pre><ul>
+<li>Looking at Munin, I see there are indeed a higher number of locks starting on the morning of 2021-10-07:</li>
+</ul>
+<p><img src="/cgspace-notes/2021/10/postgres_locks_ALL-week.png" alt="PostgreSQL locks week"></p>
+<ul>
+<li>The only thing I did on 2021-10-07 was import a few thousand metadata corrections&hellip;</li>
+<li>I restarted PostgreSQL (instead of restarting Tomcat), so let&rsquo;s see if that helps</li>
+<li>I filed <a href="https://github.com/DSpace/DSpace/issues/7989">a bug for the DSpace 6/7 duplicate values metadata import issue</a></li>
+<li>I tested the two patches for removing abandoned submissions from the workflow but unfortunately it seems that they are for the configurable aka XML workflow, and we are using the basic workflow</li>
+</ul>
 <!-- raw HTML omitted -->
 
   

docs/categories/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/" />
-<meta property="og:updated_time" content="2021-10-09T22:00:59+03:00" />
+<meta property="og:updated_time" content="2021-10-10T16:01:27+03:00" />
 
 
 

docs/categories/notes/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-10-09T22:00:59+03:00" />
+<meta property="og:updated_time" content="2021-10-10T16:01:27+03:00" />
 
 
 

docs/categories/notes/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-10-09T22:00:59+03:00" />
+<meta property="og:updated_time" content="2021-10-10T16:01:27+03:00" />
 
 
 

docs/categories/notes/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-10-09T22:00:59+03:00" />
+<meta property="og:updated_time" content="2021-10-10T16:01:27+03:00" />
 
 
 

docs/categories/notes/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-10-09T22:00:59+03:00" />
+<meta property="og:updated_time" content="2021-10-10T16:01:27+03:00" />
 
 
 

docs/categories/notes/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-10-09T22:00:59+03:00" />
+<meta property="og:updated_time" content="2021-10-10T16:01:27+03:00" />
 
 
 

docs/categories/notes/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-10-09T22:00:59+03:00" />
+<meta property="og:updated_time" content="2021-10-10T16:01:27+03:00" />
 
 
 

docs/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-10-09T22:00:59+03:00" />
+<meta property="og:updated_time" content="2021-10-10T16:01:27+03:00" />
 
 
 

docs/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-10-09T22:00:59+03:00" />
+<meta property="og:updated_time" content="2021-10-10T16:01:27+03:00" />
 
 
 

docs/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-10-09T22:00:59+03:00" />
+<meta property="og:updated_time" content="2021-10-10T16:01:27+03:00" />
 
 
 

docs/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-10-09T22:00:59+03:00" />
+<meta property="og:updated_time" content="2021-10-10T16:01:27+03:00" />
 
 
 

docs/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-10-09T22:00:59+03:00" />
+<meta property="og:updated_time" content="2021-10-10T16:01:27+03:00" />
 
 
 

docs/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-10-09T22:00:59+03:00" />
+<meta property="og:updated_time" content="2021-10-10T16:01:27+03:00" />
 
 
 

docs/page/7/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-10-09T22:00:59+03:00" />
+<meta property="og:updated_time" content="2021-10-10T16:01:27+03:00" />
 
 
 

docs/page/8/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-10-09T22:00:59+03:00" />
+<meta property="og:updated_time" content="2021-10-10T16:01:27+03:00" />
 
 
 

docs/posts/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-10-09T22:00:59+03:00" />
+<meta property="og:updated_time" content="2021-10-10T16:01:27+03:00" />
 
 
 

docs/posts/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-10-09T22:00:59+03:00" />
+<meta property="og:updated_time" content="2021-10-10T16:01:27+03:00" />
 
 
 

docs/posts/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-10-09T22:00:59+03:00" />
+<meta property="og:updated_time" content="2021-10-10T16:01:27+03:00" />
 
 
 

docs/posts/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-10-09T22:00:59+03:00" />
+<meta property="og:updated_time" content="2021-10-10T16:01:27+03:00" />
 
 
 

docs/posts/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-10-09T22:00:59+03:00" />
+<meta property="og:updated_time" content="2021-10-10T16:01:27+03:00" />
 
 
 

docs/posts/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-10-09T22:00:59+03:00" />
+<meta property="og:updated_time" content="2021-10-10T16:01:27+03:00" />
 
 
 

docs/posts/page/7/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-10-09T22:00:59+03:00" />
+<meta property="og:updated_time" content="2021-10-10T16:01:27+03:00" />
 
 
 

docs/posts/page/8/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-10-09T22:00:59+03:00" />
+<meta property="og:updated_time" content="2021-10-10T16:01:27+03:00" />
 
 
 

docs/sitemap.xml

@@ -3,19 +3,19 @@
   xmlns:xhtml="http://www.w3.org/1999/xhtml">
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/</loc>
-    <lastmod>2021-10-09T22:00:59+03:00</lastmod>
+    <lastmod>2021-10-10T16:01:27+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2021-10-09T22:00:59+03:00</lastmod>
+    <lastmod>2021-10-10T16:01:27+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
-    <lastmod>2021-10-09T22:00:59+03:00</lastmod>
+    <lastmod>2021-10-10T16:01:27+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/2021-10/</loc>
-    <lastmod>2021-10-09T22:00:59+03:00</lastmod>
+    <lastmod>2021-10-10T16:01:27+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
-    <lastmod>2021-10-09T22:00:59+03:00</lastmod>
+    <lastmod>2021-10-10T16:01:27+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/2021-09/</loc>
     <lastmod>2021-10-04T11:10:54+03:00</lastmod>