Add notes for 2021-03-08

content/posts/2021-03.md

@@ -234,4 +234,46 @@ $ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: applica
   - [Migrated away from links to use networks](https://github.com/ilri/OpenRXV/issues/89)
   - [Converted the backend container to use a custom image that includes `unoconv`](https://github.com/ilri/OpenRXV/issues/68) so we don't have to manually install it anymore
 
+## 2021-03-08
+
+- I approved the WLE item that I edited last week, and all the metadata is there: https://hdl.handle.net/10568/111810
+  - So I'm not sure what Niroshini's issue with metadata is...
+- Peter sent a message yesterday saying that his item finally got committed
+  - I looked at the Munin graphs and there was a MASSIVE spike in database activity two days ago, and now database locks are back down to normal levels (from 1000+):
+
+```console
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+13
+```
+
+- On 2021-03-03 the PostgreSQL transactions started rising:
+
+![PostgreSQL query length week](/cgspace-notes/2021/03/postgres_querylength_ALL-week.png)
+
+- After that the connections and locks started going up, peaking on 2021-03-06:
+
+![PostgreSQL locks week](/cgspace-notes/2021/03/postgres_locks_ALL-week.png)
+![PostgreSQL connections week](/cgspace-notes/2021/03/postgres_connections_ALL-week.png)
+
+- I sent another message to Atmire to ask if they have time to look into this
+- CIFOR is pressuring me to upload the batch items from last week
+  - Vika sent me a final file with some duplicates that Peter identified removed
+  - I extracted and re-applied my basic corrections from last week in OpenRefine, then ran the items through `csv-metadata-quality` checker and uploaded them to CGSpace
+  - In total there are 1,088 items
+- Udana from IWMI emailed to ask about CGSpace thumbnails
+- Udana from IWMI emailed to ask about an item uploaded recently that does not appear in AReS
+  - [The item](https://hdl.handle.net/10568/111794) was added to the archive on 2021-03-05, and I last harvested on 2021-03-06, so this might be an issue of a missing item
+- Abenet got a quote from Atmire to buy 125 credits for 3750€
+- Maria at Bioversity sent some feedback about duplicate items on AReS
+- I'm wondering if the issue of the `openrxv-items-final` index not getting cleared after a successful harvest (which results in having 200,000, then 300,000, etc items) has to do with the alias issue I fixed yesterday
+  - I will start a fresh harvest on AReS without now to check, but first back up the current index just in case:
+
+```console
+$ curl -X PUT "localhost:9200/openrxv-items-final/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items-final/_clone/openrxv-items-final-2021-03-08
+# start harvesting on AReS
+```
+
+- As I saw on my local test instance, even when you cancel a harvesting, it replaces the `openrxv-items-final` index with whatever is in `openrxv-items-temp` automatically, so I assume it will do the same now
+
 <!-- vim: set sw=2 ts=2: -->

docs/2021-03/index.html

@@ -19,7 +19,7 @@ Also, we found some issues building and running OpenRXV currently due to ecosyst
 <meta property="og:type" content="article" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2021-03/" />
 <meta property="article:published_time" content="2021-03-01T10:13:54+02:00" />
-<meta property="article:modified_time" content="2021-03-06T13:35:20+02:00" />
+<meta property="article:modified_time" content="2021-03-07T15:51:12+02:00" />
 
 
 
@@ -44,9 +44,9 @@ Also, we found some issues building and running OpenRXV currently due to ecosyst
   "@type": "BlogPosting",
   "headline": "March, 2021",
   "url": "https://alanorth.github.io/cgspace-notes/2021-03/",
-  "wordCount": "1306",
+  "wordCount": "1673",
   "datePublished": "2021-03-01T10:13:54+02:00",
-  "dateModified": "2021-03-06T13:35:20+02:00",
+  "dateModified": "2021-03-07T15:51:12+02:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"
@@ -362,6 +362,59 @@ $ curl -X PUT &quot;localhost:9200/openrxv-items/_settings&quot; -H 'Content-Typ
 </ul>
 </li>
 </ul>
+<h2 id="2021-03-08">2021-03-08</h2>
+<ul>
+<li>I approved the WLE item that I edited last week, and all the metadata is there: <a href="https://hdl.handle.net/10568/111810">https://hdl.handle.net/10568/111810</a>
+<ul>
+<li>So I&rsquo;m not sure what Niroshini&rsquo;s issue with metadata is&hellip;</li>
+</ul>
+</li>
+<li>Peter sent a message yesterday saying that his item finally got committed
+<ul>
+<li>I looked at the Munin graphs and there was a MASSIVE spike in database activity two days ago, and now database locks are back down to normal levels (from 1000+):</li>
+</ul>
+</li>
+</ul>
+<pre><code class="language-console" data-lang="console">$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+13
+</code></pre><ul>
+<li>On 2021-03-03 the PostgreSQL transactions started rising:</li>
+</ul>
+<p><img src="/cgspace-notes/2021/03/postgres_querylength_ALL-week.png" alt="PostgreSQL query length week"></p>
+<ul>
+<li>After that the connections and locks started going up, peaking on 2021-03-06:</li>
+</ul>
+<p><img src="/cgspace-notes/2021/03/postgres_locks_ALL-week.png" alt="PostgreSQL locks week">
+<img src="/cgspace-notes/2021/03/postgres_connections_ALL-week.png" alt="PostgreSQL connections week"></p>
+<ul>
+<li>I sent another message to Atmire to ask if they have time to look into this</li>
+<li>CIFOR is pressuring me to upload the batch items from last week
+<ul>
+<li>Vika sent me a final file with some duplicates that Peter identified removed</li>
+<li>I extracted and re-applied my basic corrections from last week in OpenRefine, then ran the items through <code>csv-metadata-quality</code> checker and uploaded them to CGSpace</li>
+<li>In total there are 1,088 items</li>
+</ul>
+</li>
+<li>Udana from IWMI emailed to ask about CGSpace thumbnails</li>
+<li>Udana from IWMI emailed to ask about an item uploaded recently that does not appear in AReS
+<ul>
+<li><a href="https://hdl.handle.net/10568/111794">The item</a> was added to the archive on 2021-03-05, and I last harvested on 2021-03-06, so this might be an issue of a missing item</li>
+</ul>
+</li>
+<li>Abenet got a quote from Atmire to buy 125 credits for 3750€</li>
+<li>Maria at Bioversity sent some feedback about duplicate items on AReS</li>
+<li>I&rsquo;m wondering if the issue of the <code>openrxv-items-final</code> index not getting cleared after a successful harvest (which results in having 200,000, then 300,000, etc items) has to do with the alias issue I fixed yesterday
+<ul>
+<li>I will start a fresh harvest on AReS without now to check, but first back up the current index just in case:</li>
+</ul>
+</li>
+</ul>
+<pre><code class="language-console" data-lang="console">$ curl -X PUT &quot;localhost:9200/openrxv-items-final/_settings&quot; -H 'Content-Type: application/json' -d'{&quot;settings&quot;: {&quot;index.blocks.write&quot;: true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items-final/_clone/openrxv-items-final-2021-03-08
+# start harvesting on AReS
+</code></pre><ul>
+<li>As I saw on my local test instance, even when you cancel a harvesting, it replaces the <code>openrxv-items-final</code> index with whatever is in <code>openrxv-items-temp</code> automatically, so I assume it will do the same now</li>
+</ul>
 <!-- raw HTML omitted -->
 
   

docs/categories/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/" />
-<meta property="og:updated_time" content="2021-03-06T13:35:20+02:00" />
+<meta property="og:updated_time" content="2021-03-07T15:51:12+02:00" />
 
 
 

docs/categories/notes/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-03-06T13:35:20+02:00" />
+<meta property="og:updated_time" content="2021-03-07T15:51:12+02:00" />
 
 
 

docs/categories/notes/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-03-06T13:35:20+02:00" />
+<meta property="og:updated_time" content="2021-03-07T15:51:12+02:00" />
 
 
 

docs/categories/notes/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-03-06T13:35:20+02:00" />
+<meta property="og:updated_time" content="2021-03-07T15:51:12+02:00" />
 
 
 

docs/categories/notes/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-03-06T13:35:20+02:00" />
+<meta property="og:updated_time" content="2021-03-07T15:51:12+02:00" />
 
 
 

docs/categories/notes/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-03-06T13:35:20+02:00" />
+<meta property="og:updated_time" content="2021-03-07T15:51:12+02:00" />
 
 
 

docs/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-03-06T13:35:20+02:00" />
+<meta property="og:updated_time" content="2021-03-07T15:51:12+02:00" />
 
 
 

docs/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-03-06T13:35:20+02:00" />
+<meta property="og:updated_time" content="2021-03-07T15:51:12+02:00" />
 
 
 

docs/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-03-06T13:35:20+02:00" />
+<meta property="og:updated_time" content="2021-03-07T15:51:12+02:00" />
 
 
 

docs/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-03-06T13:35:20+02:00" />
+<meta property="og:updated_time" content="2021-03-07T15:51:12+02:00" />
 
 
 

docs/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-03-06T13:35:20+02:00" />
+<meta property="og:updated_time" content="2021-03-07T15:51:12+02:00" />
 
 
 

docs/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-03-06T13:35:20+02:00" />
+<meta property="og:updated_time" content="2021-03-07T15:51:12+02:00" />
 
 
 

docs/page/7/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-03-06T13:35:20+02:00" />
+<meta property="og:updated_time" content="2021-03-07T15:51:12+02:00" />
 
 
 

docs/posts/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-03-06T13:35:20+02:00" />
+<meta property="og:updated_time" content="2021-03-07T15:51:12+02:00" />
 
 
 

docs/posts/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-03-06T13:35:20+02:00" />
+<meta property="og:updated_time" content="2021-03-07T15:51:12+02:00" />
 
 
 

docs/posts/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-03-06T13:35:20+02:00" />
+<meta property="og:updated_time" content="2021-03-07T15:51:12+02:00" />
 
 
 

docs/posts/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-03-06T13:35:20+02:00" />
+<meta property="og:updated_time" content="2021-03-07T15:51:12+02:00" />
 
 
 

docs/posts/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-03-06T13:35:20+02:00" />
+<meta property="og:updated_time" content="2021-03-07T15:51:12+02:00" />
 
 
 

docs/posts/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-03-06T13:35:20+02:00" />
+<meta property="og:updated_time" content="2021-03-07T15:51:12+02:00" />
 
 
 

docs/posts/page/7/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-03-06T13:35:20+02:00" />
+<meta property="og:updated_time" content="2021-03-07T15:51:12+02:00" />
 
 
 

docs/sitemap.xml

@@ -3,19 +3,19 @@
   xmlns:xhtml="http://www.w3.org/1999/xhtml">
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/</loc>
-    <lastmod>2021-03-06T13:35:20+02:00</lastmod>
+    <lastmod>2021-03-07T15:51:12+02:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2021-03-06T13:35:20+02:00</lastmod>
+    <lastmod>2021-03-07T15:51:12+02:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/2021-03/</loc>
-    <lastmod>2021-03-06T13:35:20+02:00</lastmod>
+    <lastmod>2021-03-07T15:51:12+02:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
-    <lastmod>2021-03-06T13:35:20+02:00</lastmod>
+    <lastmod>2021-03-07T15:51:12+02:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
-    <lastmod>2021-03-06T13:35:20+02:00</lastmod>
+    <lastmod>2021-03-07T15:51:12+02:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/2021-02/</loc>
     <lastmod>2021-03-04T22:46:05+02:00</lastmod>