Update notes for 2021-09-23

content/posts/2021-09.md

@@ -211,4 +211,34 @@ localhost/dspace63= > \COPY (SELECT DISTINCT text_value as "cg.contributor.affil
 COPY 8091
 ```
 
+## 2021-09-23
+
+- Peter sent me back the corrections for the affiliations
+  - It is about 1,280 corrections and fourteen deletions
+  - I cleaned them up in csv-metadata-quality and then extracted the deletes and fixes to separate files to run with `fix-metadata-values.py` and `delete-metadata-values.py`:
+
+```console
+$ csv-metadata-quality -i ~/Downloads/2021-09-20-affiliations.csv -o /tmp/affiliations.csv -x cg.contributor.affiliation
+$ csvgrep -c 'correct' -m 'DELETE' /tmp/affiliations.csv > /tmp/affiliations-delete.csv
+$ csvgrep -c 'correct' -r '^.+$' /tmp/affiliations.csv | csvgrep -i -c 'correct' -m 'DELETE' > /tmp/affiliations-fix.csv
+$ ./ilri/fix-metadata-values.py -i /tmp/affiliations-fix.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.affiliation -t 'correct' -m 211
+$ ./ilri/delete-metadata-values.py -i /tmp/affiliations-fix.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.affiliation -m 211
+```
+
+- Then I updated the controlled vocabulary for affiliations by exporting the top 1,000 used terms:
+
+```console
+localhost/dspace63= > \COPY (SELECT DISTINCT text_value as "cg.contributor.affiliation", count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC LIMIT 1000) to /tmp/2021-09-23-affiliations.csv WITH CSV HEADER;
+$ csvcut -c 1 /tmp/2021-09-23-affiliations.csv | sed 1d > /tmp/affiliations.txt
+```
+
+- Peter also sent me 310 corrections and 234 deletions for donors so I applied those and updated the controlled vocabularies too
+- Move some One CGIAR-related collections around the CGSpace hierarchy for Peter Ballantyne
+- Mohammed Salem asked me for an ID to UUID mapping for CGSpace collections, so I generated one similar to the ID one I sent him in 2020-11:
+
+```console
+localhost/dspace63= > \COPY (SELECT collection_id,uuid FROM collection WHERE collection_id IS NOT NULL) TO /tmp/2021-09-23-collection-id2uuid.csv WITH CSV HEADER;
+COPY 1139
+```
+
 <!-- vim: set sw=2 ts=2: -->

docs/2021-09/index.html

@@ -26,7 +26,7 @@ The syntax Moayad showed me last month doesn’t seem to honor the search qu
 <meta property="og:type" content="article" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2021-09/" />
 <meta property="article:published_time" content="2021-09-01T09:14:07+03:00" />
-<meta property="article:modified_time" content="2021-09-20T17:31:45+03:00" />
+<meta property="article:modified_time" content="2021-09-23T18:19:11+03:00" />
 
 
 
@@ -58,9 +58,9 @@ The syntax Moayad showed me last month doesn&rsquo;t seem to honor the search qu
   "@type": "BlogPosting",
   "headline": "September, 2021",
   "url": "https://alanorth.github.io/cgspace-notes/2021-09/",
-  "wordCount": "1532",
+  "wordCount": "1775",
   "datePublished": "2021-09-01T09:14:07+03:00",
-  "dateModified": "2021-09-20T17:31:45+03:00",
+  "dateModified": "2021-09-23T18:19:11+03:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"
@@ -377,6 +377,32 @@ localhost/dspace63= &gt; \COPY (SELECT DISTINCT text_value as &quot;cg.contribut
 COPY 1274
 localhost/dspace63= &gt; \COPY (SELECT DISTINCT text_value as &quot;cg.contributor.affiliation&quot;, count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC) to /tmp/2021-09-20-affiliations.csv WITH CSV HEADER;
 COPY 8091
+</code></pre><h2 id="2021-09-23">2021-09-23</h2>
+<ul>
+<li>Peter sent me back the corrections for the affiliations
+<ul>
+<li>It is about 1,280 corrections and fourteen deletions</li>
+<li>I cleaned them up in csv-metadata-quality and then extracted the deletes and fixes to separate files to run with <code>fix-metadata-values.py</code> and <code>delete-metadata-values.py</code>:</li>
+</ul>
+</li>
+</ul>
+<pre tabindex="0"><code class="language-console" data-lang="console">$ csv-metadata-quality -i ~/Downloads/2021-09-20-affiliations.csv -o /tmp/affiliations.csv -x cg.contributor.affiliation
+$ csvgrep -c 'correct' -m 'DELETE' /tmp/affiliations.csv &gt; /tmp/affiliations-delete.csv
+$ csvgrep -c 'correct' -r '^.+$' /tmp/affiliations.csv | csvgrep -i -c 'correct' -m 'DELETE' &gt; /tmp/affiliations-fix.csv
+$ ./ilri/fix-metadata-values.py -i /tmp/affiliations-fix.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.affiliation -t 'correct' -m 211
+$ ./ilri/delete-metadata-values.py -i /tmp/affiliations-fix.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.affiliation -m 211
+</code></pre><ul>
+<li>Then I updated the controlled vocabulary for affiliations by exporting the top 1,000 used terms:</li>
+</ul>
+<pre tabindex="0"><code class="language-console" data-lang="console">localhost/dspace63= &gt; \COPY (SELECT DISTINCT text_value as &quot;cg.contributor.affiliation&quot;, count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC LIMIT 1000) to /tmp/2021-09-23-affiliations.csv WITH CSV HEADER;
+$ csvcut -c 1 /tmp/2021-09-23-affiliations.csv | sed 1d &gt; /tmp/affiliations.txt
+</code></pre><ul>
+<li>Peter also sent me 310 corrections and 234 deletions for donors so I applied those and updated the controlled vocabularies too</li>
+<li>Move some One CGIAR-related collections around the CGSpace hierarchy for Peter Ballantyne</li>
+<li>Mohammed Salem asked me for an ID to UUID mapping for CGSpace collections, so I generated one similar to the ID one I sent him in 2020-11:</li>
+</ul>
+<pre tabindex="0"><code class="language-console" data-lang="console">localhost/dspace63= &gt; \COPY (SELECT collection_id,uuid FROM collection WHERE collection_id IS NOT NULL) TO /tmp/2021-09-23-collection-id2uuid.csv WITH CSV HEADER;
+COPY 1139
 </code></pre><!-- raw HTML omitted -->
 
   

docs/categories/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/" />
-<meta property="og:updated_time" content="2021-09-21T12:46:34+03:00" />
+<meta property="og:updated_time" content="2021-09-23T18:19:11+03:00" />
 
 
 

docs/categories/notes/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-09-21T12:46:34+03:00" />
+<meta property="og:updated_time" content="2021-09-23T18:19:11+03:00" />
 
 
 

docs/categories/notes/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-09-21T12:46:34+03:00" />
+<meta property="og:updated_time" content="2021-09-23T18:19:11+03:00" />
 
 
 

docs/categories/notes/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-09-21T12:46:34+03:00" />
+<meta property="og:updated_time" content="2021-09-23T18:19:11+03:00" />
 
 
 

docs/categories/notes/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-09-21T12:46:34+03:00" />
+<meta property="og:updated_time" content="2021-09-23T18:19:11+03:00" />
 
 
 

docs/categories/notes/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-09-21T12:46:34+03:00" />
+<meta property="og:updated_time" content="2021-09-23T18:19:11+03:00" />
 
 
 

docs/categories/notes/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-09-21T12:46:34+03:00" />
+<meta property="og:updated_time" content="2021-09-23T18:19:11+03:00" />
 
 
 

docs/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-09-21T12:46:34+03:00" />
+<meta property="og:updated_time" content="2021-09-23T18:19:11+03:00" />
 
 
 

docs/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-09-21T12:46:34+03:00" />
+<meta property="og:updated_time" content="2021-09-23T18:19:11+03:00" />
 
 
 

docs/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-09-21T12:46:34+03:00" />
+<meta property="og:updated_time" content="2021-09-23T18:19:11+03:00" />
 
 
 

docs/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-09-21T12:46:34+03:00" />
+<meta property="og:updated_time" content="2021-09-23T18:19:11+03:00" />
 
 
 

docs/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-09-21T12:46:34+03:00" />
+<meta property="og:updated_time" content="2021-09-23T18:19:11+03:00" />
 
 
 

docs/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-09-21T12:46:34+03:00" />
+<meta property="og:updated_time" content="2021-09-23T18:19:11+03:00" />
 
 
 

docs/page/7/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-09-21T12:46:34+03:00" />
+<meta property="og:updated_time" content="2021-09-23T18:19:11+03:00" />
 
 
 

docs/page/8/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-09-21T12:46:34+03:00" />
+<meta property="og:updated_time" content="2021-09-23T18:19:11+03:00" />
 
 
 

docs/posts/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-09-21T12:46:34+03:00" />
+<meta property="og:updated_time" content="2021-09-23T18:19:11+03:00" />
 
 
 

docs/posts/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-09-21T12:46:34+03:00" />
+<meta property="og:updated_time" content="2021-09-23T18:19:11+03:00" />
 
 
 

docs/posts/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-09-21T12:46:34+03:00" />
+<meta property="og:updated_time" content="2021-09-23T18:19:11+03:00" />
 
 
 

docs/posts/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-09-21T12:46:34+03:00" />
+<meta property="og:updated_time" content="2021-09-23T18:19:11+03:00" />
 
 
 

docs/posts/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-09-21T12:46:34+03:00" />
+<meta property="og:updated_time" content="2021-09-23T18:19:11+03:00" />
 
 
 

docs/posts/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-09-21T12:46:34+03:00" />
+<meta property="og:updated_time" content="2021-09-23T18:19:11+03:00" />
 
 
 

docs/posts/page/7/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-09-21T12:46:34+03:00" />
+<meta property="og:updated_time" content="2021-09-23T18:19:11+03:00" />
 
 
 

docs/posts/page/8/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-09-21T12:46:34+03:00" />
+<meta property="og:updated_time" content="2021-09-23T18:19:11+03:00" />
 
 
 

docs/sitemap.xml

@@ -3,19 +3,19 @@
   xmlns:xhtml="http://www.w3.org/1999/xhtml">
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/</loc>
-    <lastmod>2021-09-21T12:46:34+03:00</lastmod>
+    <lastmod>2021-09-23T18:19:11+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2021-09-21T12:46:34+03:00</lastmod>
+    <lastmod>2021-09-23T18:19:11+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
-    <lastmod>2021-09-21T12:46:34+03:00</lastmod>
+    <lastmod>2021-09-23T18:19:11+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
-    <lastmod>2021-09-21T12:46:34+03:00</lastmod>
+    <lastmod>2021-09-23T18:19:11+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/2021-09/</loc>
-    <lastmod>2021-09-20T17:31:45+03:00</lastmod>
+    <lastmod>2021-09-23T18:19:11+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/2021-08/</loc>
     <lastmod>2021-09-02T17:06:28+03:00</lastmod>