Add notes for 2020-08-05

content/posts/2020-08.md

@@ -34,4 +34,91 @@ $ chrt -b 0 dspace dsrun com.atmire.statistics.util.update.atomic.AtomicStatisti
 - Andrea from Macaroni Bros emailed me a few days ago to say he's having issues with the CGSpace REST API
   - He said he noticed the issues when they were developing the WordPress plugin to harvest CGSpace for the RTB website: https://www.rtb.cgiar.org/publications/
 
+## 2020-08-04
+
+- Look into the REST API issues that Macaroni Bros raised last week:
+  - The first one was about the `collections` endpoint returning empty items:
+    - https://cgspace.cgiar.org/rest/collections/1445/items?limit=1&offset=2 (offset=2 is correct)
+    - https://cgspace.cgiar.org/rest/collections/1445/items?limit=1&offset=3 (offset=3 is empty)
+    - https://cgspace.cgiar.org/rest/collections/1445/items?limit=1&offset=4 (offset=4 is correct again)
+  - I confirm that the second link returns zero items on CGSpace...
+    - I tested on my local development instance and it returns one item correctly...
+    - I tested on DSpace Test (currently DSpace 6 with UUIDs) and it returns one item correctly...
+    - Perhaps an indexing issue?
+  - The second issue is the `collections` endpoint returning the wrong number of items:
+    - https://cgspace.cgiar.org/rest/collections/1445 (numberItems: 63)
+    - https://cgspace.cgiar.org/rest/collections/1445/items (real number of items: 61)
+  - I confirm that it is indeed happening on CGSpace...
+    - And actually I can replicate the same issue on my local CGSpace 5.8 instance:
+
+```
+$ http 'http://localhost:8080/rest/collections/1445' | json_pp | grep numberItems
+   "numberItems" : 63,
+$ http 'http://localhost:8080/rest/collections/1445/items' jq '. | length'
+61
+```
+
+    - Also on DSpace Test (which is running DSpace 6!), though the issue is slightly different there:
+
+```
+$ http 'https://dspacetest.cgiar.org/rest/collections/5471c3aa-202e-42f0-96c2-497a18e3b708' | json_pp | grep numberItems
+   "numberItems" : 61,
+$ http 'https://dspacetest.cgiar.org/rest/collections/5471c3aa-202e-42f0-96c2-497a18e3b708/items' | jq '. | length'
+59
+```
+
+- Ah! I exported that collection's metadata and checked it in OpenRefine, where I noticed that two items are mapped twice
+  - I dealt with this problem in 2017-01 and the solution is to check the `collection2item` table:
+
+```
+dspace=# SELECT * FROM collection2item WHERE item_id = '107687';
+   id   | collection_id | item_id
+--------+---------------+---------
+ 133698 |           966 |  107687
+ 134685 |          1445 |  107687
+ 134686 |          1445 |  107687
+(3 rows)
+```
+
+- So for each id you can delete one duplicate mapping:
+
+```
+dspace=# DELETE FROM collection2item WHERE id='134686';
+dspace=# DELETE FROM collection2item WHERE id='128819';
+```
+
+- Update countries on CGSpace to be closer to ISO 3166-1 with some minor differences based on Peter's preferred display names
+
+```
+$ cat 2020-08-04-PB-new-countries.csv
+cg.coverage.country,correct
+CAPE VERDE,CABO VERDE
+COCOS ISLANDS,COCOS (KEELING) ISLANDS
+"CONGO, DR","CONGO, DEMOCRATIC REPUBLIC OF"
+COTE D'IVOIRE,CÔTE D'IVOIRE
+"KOREA, REPUBLIC","KOREA, REPUBLIC OF"
+PALESTINE,"PALESTINE, STATE OF"
+$ ./fix-metadata-values.py -i 2020-08-04-PB-new-countries.csv -db dspace -u dspace -p 'fuuu' -f cg.coverage.country -t 'correct' -m 228
+```
+
+- I had to restart Tomcat 7 three times before all the Solr statistics cores came up properly
+  - I started a full Discovery re-indexing
+
+## 2020-08-05
+
+- Port my [dspace-curation-tasks](https://github.com/ilri/dspace-curation-tasks) to DSpace 6 and tag version `6.0-SNAPSHOT`
+- I downloaded the [UN M.49](https://unstats.un.org/unsd/methodology/m49/overview/) CSV file to start working on updating the CGSpace regions
+  - First issue is they don't version the file so you have no idea when it was released
+  - Second issue is that three rows have errors due to not using quotes around "China, Macao Special Administrative Region"
+- Bizu said she was having problems approving tasks on CGSpace
+  - I looked at the PostgreSQL locks and they have skyrocketed since yesterday:
+
+![PostgreSQL locks day](/cgspace-notes/2020/08/postgres_locks_ALL-day.png)
+
+![PostgreSQL query length day](/cgspace-notes/2020/08/postgres_querylength_ALL-day.png)
+
+- Seems that something happened yesterday afternoon at around 5PM...
+  - For now I will just run all updates on the server and reboot it, as I have no idea what causes this issue
+  - I had to restart Tomcat 7 three times after the server came back up before all Solr statistics cores came up properly
+
 <!-- vim: set sw=2 ts=2: -->

docs/categories/index.html

@@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/" />
-<meta property="og:updated_time" content="2020-08-02T23:55:04+03:00" />
+<meta property="og:updated_time" content="2020-08-03T16:27:51+03:00" />
 
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Categories"/>

docs/categories/notes/index.html

@@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2020-08-02T23:55:04+03:00" />
+<meta property="og:updated_time" content="2020-08-03T16:27:51+03:00" />
 
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Notes"/>

docs/categories/notes/page/2/index.html

@@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2020-08-02T23:55:04+03:00" />
+<meta property="og:updated_time" content="2020-08-03T16:27:51+03:00" />
 
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Notes"/>

docs/categories/notes/page/3/index.html

@@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2020-08-02T23:55:04+03:00" />
+<meta property="og:updated_time" content="2020-08-03T16:27:51+03:00" />
 
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Notes"/>

docs/categories/notes/page/4/index.html

@@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2020-08-02T23:55:04+03:00" />
+<meta property="og:updated_time" content="2020-08-03T16:27:51+03:00" />
 
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Notes"/>

docs/index.html

@@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2020-08-02T23:55:04+03:00" />
+<meta property="og:updated_time" content="2020-08-03T16:27:51+03:00" />
 
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="CGSpace Notes"/>

docs/page/2/index.html

@@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2020-08-02T23:55:04+03:00" />
+<meta property="og:updated_time" content="2020-08-03T16:27:51+03:00" />
 
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="CGSpace Notes"/>

docs/page/3/index.html

@@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2020-08-02T23:55:04+03:00" />
+<meta property="og:updated_time" content="2020-08-03T16:27:51+03:00" />
 
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="CGSpace Notes"/>

docs/page/4/index.html

@@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2020-08-02T23:55:04+03:00" />
+<meta property="og:updated_time" content="2020-08-03T16:27:51+03:00" />
 
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="CGSpace Notes"/>

docs/page/5/index.html

@@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2020-08-02T23:55:04+03:00" />
+<meta property="og:updated_time" content="2020-08-03T16:27:51+03:00" />
 
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="CGSpace Notes"/>

docs/page/6/index.html

@@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2020-08-02T23:55:04+03:00" />
+<meta property="og:updated_time" content="2020-08-03T16:27:51+03:00" />
 
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="CGSpace Notes"/>

docs/posts/index.html

@@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2020-08-02T23:55:04+03:00" />
+<meta property="og:updated_time" content="2020-08-03T16:27:51+03:00" />
 
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Posts"/>

docs/posts/page/2/index.html

@@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2020-08-02T23:55:04+03:00" />
+<meta property="og:updated_time" content="2020-08-03T16:27:51+03:00" />
 
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Posts"/>

docs/posts/page/3/index.html

@@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2020-08-02T23:55:04+03:00" />
+<meta property="og:updated_time" content="2020-08-03T16:27:51+03:00" />
 
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Posts"/>

docs/posts/page/4/index.html

@@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2020-08-02T23:55:04+03:00" />
+<meta property="og:updated_time" content="2020-08-03T16:27:51+03:00" />
 
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Posts"/>

docs/posts/page/5/index.html

@@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2020-08-02T23:55:04+03:00" />
+<meta property="og:updated_time" content="2020-08-03T16:27:51+03:00" />
 
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Posts"/>

docs/posts/page/6/index.html

@@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2020-08-02T23:55:04+03:00" />
+<meta property="og:updated_time" content="2020-08-03T16:27:51+03:00" />
 
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Posts"/>

docs/sitemap.xml

@@ -4,27 +4,27 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2020-07/</loc>
-    <lastmod>2020-08-02T23:55:04+03:00</lastmod>
+    <lastmod>2020-08-03T16:27:51+03:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/</loc>
-    <lastmod>2020-08-02T23:55:04+03:00</lastmod>
+    <lastmod>2020-08-03T16:27:51+03:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2020-08-02T23:55:04+03:00</lastmod>
+    <lastmod>2020-08-03T16:27:51+03:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
-    <lastmod>2020-08-02T23:55:04+03:00</lastmod>
+    <lastmod>2020-08-03T16:27:51+03:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
-    <lastmod>2020-08-02T23:55:04+03:00</lastmod>
+    <lastmod>2020-08-03T16:27:51+03:00</lastmod>
   </url>
   
   <url>