Update notes for 2020-10-22

2024-12-22 21:22:19 +01:00 · 2020-10-22 20:07:42 +03:00 · 2020-10-22 20:07:42 +03:00 · f60bb8b10f
commit f60bb8b10f
parent 8614add1c9
22 changed files with 67 additions and 27 deletions
--- a/content/posts/2020-10.md
+++ b/content/posts/2020-10.md
@ -640,5 +640,20 @@ Authorization denied for action WORKFLOW_STEP_1 on COLLECTION:1072 by user 1759
  - The submit step is defined, but has no users or groups
  - I added the IITA submitters there and told Bosede to try again
 - Add two new blocks to list the top communities and collections on AReS
+- I want to extract all CRPs and affiliations from AReS to do some text processing and create some mappings...
+  - First extract 10,000 affiliations from Elasticsearch by only including the `affiliation` source:
+
+```
+$ http 'http://localhost:9200/openrxv-items-final/_search?_source_includes=affiliation&size=10000&q=*:*' > /tmp/affiliations.json
+```
+
+- Then I decided to try a different approach and I adjusted my `convert-mapping.py` script to re-consider some replacement patterns with acronyms from the original AReS `mapping.json` file to hopefully address some MEL to CGSpace mappings
+  - For example, to changes this:
+    - find: International Livestock Research Institute
+    - replace: International Livestock Research Institute - ILRI
+  - ... into this:
+    - find: International Livestock Research Institute - ILRI
+    - replace: International Livestock Research Institute
+- I re-uploaded the mappings to Elasticsearch like I did yesterday and restarted the harvesting

 <!-- vim: set sw=2 ts=2: -->
--- a/docs/2020-10/index.html
+++ b/docs/2020-10/index.html
@ -23,7 +23,7 @@ During the FlywayDB migration I got an error:
 <meta property="og:type" content="article" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2020-10/" />
 <meta property="article:published_time" content="2020-10-06T16:55:54+03:00" />
-<meta property="article:modified_time" content="2020-10-21T15:36:31+03:00" />
+<meta property="article:modified_time" content="2020-10-22T11:58:26+03:00" />

 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="October, 2020"/>
@ -51,9 +51,9 @@ During the FlywayDB migration I got an error:
  "@type": "BlogPosting",
  "headline": "October, 2020",
  "url": "https://alanorth.github.io/cgspace-notes/2020-10/",
-  "wordCount": "4233",
+  "wordCount": "4350",
  "datePublished": "2020-10-06T16:55:54+03:00",
-  "dateModified": "2020-10-21T15:36:31+03:00",
+  "dateModified": "2020-10-22T11:58:26+03:00",
  "author": {
    "@type": "Person",
    "name": "Alan Orth"
@ -839,6 +839,31 @@ $ csvcut -c 'id,dc.subject[],dc.subject[en_US],cg.subject.ilri[],cg.subject.ilri
 </ul>
 </li>
 <li>Add two new blocks to list the top communities and collections on AReS</li>
+<li>I want to extract all CRPs and affiliations from AReS to do some text processing and create some mappings&hellip;
+<ul>
+<li>First extract 10,000 affiliations from Elasticsearch by only including the <code>affiliation</code> source:</li>
+</ul>
+</li>
+</ul>
+<pre><code>$ http 'http://localhost:9200/openrxv-items-final/_search?_source_includes=affiliation&amp;size=10000&amp;q=*:*' &gt; /tmp/affiliations.json
+</code></pre><ul>
+<li>Then I decided to try a different approach and I adjusted my <code>convert-mapping.py</code> script to re-consider some replacement patterns with acronyms from the original AReS <code>mapping.json</code> file to hopefully address some MEL to CGSpace mappings
+<ul>
+<li>For example, to changes this:
+<ul>
+<li>find: International Livestock Research Institute</li>
+<li>replace: International Livestock Research Institute - ILRI</li>
+</ul>
+</li>
+<li>&hellip; into this:
+<ul>
+<li>find: International Livestock Research Institute - ILRI</li>
+<li>replace: International Livestock Research Institute</li>
+</ul>
+</li>
+</ul>
+</li>
+<li>I re-uploaded the mappings to Elasticsearch like I did yesterday and restarted the harvesting</li>
 </ul>
 <!-- raw HTML omitted -->

--- a/docs/categories/index.html
+++ b/docs/categories/index.html
@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/" />
-<meta property="og:updated_time" content="2020-10-21T15:36:31+03:00" />
+<meta property="og:updated_time" content="2020-10-22T11:58:26+03:00" />

 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Categories"/>
--- a/docs/categories/notes/index.html
+++ b/docs/categories/notes/index.html
@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2020-10-21T15:36:31+03:00" />
+<meta property="og:updated_time" content="2020-10-22T11:58:26+03:00" />

 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Notes"/>
--- a/docs/categories/notes/page/2/index.html
+++ b/docs/categories/notes/page/2/index.html
@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2020-10-21T15:36:31+03:00" />
+<meta property="og:updated_time" content="2020-10-22T11:58:26+03:00" />

 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Notes"/>
--- a/docs/categories/notes/page/3/index.html
+++ b/docs/categories/notes/page/3/index.html
@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2020-10-21T15:36:31+03:00" />
+<meta property="og:updated_time" content="2020-10-22T11:58:26+03:00" />

 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Notes"/>
--- a/docs/categories/notes/page/4/index.html
+++ b/docs/categories/notes/page/4/index.html
@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2020-10-21T15:36:31+03:00" />
+<meta property="og:updated_time" content="2020-10-22T11:58:26+03:00" />

 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Notes"/>
--- a/docs/index.html
+++ b/docs/index.html
@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2020-10-21T15:36:31+03:00" />
+<meta property="og:updated_time" content="2020-10-22T11:58:26+03:00" />

 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="CGSpace Notes"/>
--- a/docs/page/2/index.html
+++ b/docs/page/2/index.html
@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2020-10-21T15:36:31+03:00" />
+<meta property="og:updated_time" content="2020-10-22T11:58:26+03:00" />

 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="CGSpace Notes"/>
--- a/docs/page/3/index.html
+++ b/docs/page/3/index.html
@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2020-10-21T15:36:31+03:00" />
+<meta property="og:updated_time" content="2020-10-22T11:58:26+03:00" />

 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="CGSpace Notes"/>
--- a/docs/page/4/index.html
+++ b/docs/page/4/index.html
@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2020-10-21T15:36:31+03:00" />
+<meta property="og:updated_time" content="2020-10-22T11:58:26+03:00" />

 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="CGSpace Notes"/>
--- a/docs/page/5/index.html
+++ b/docs/page/5/index.html
@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2020-10-21T15:36:31+03:00" />
+<meta property="og:updated_time" content="2020-10-22T11:58:26+03:00" />

 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="CGSpace Notes"/>
--- a/docs/page/6/index.html
+++ b/docs/page/6/index.html
@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2020-10-21T15:36:31+03:00" />
+<meta property="og:updated_time" content="2020-10-22T11:58:26+03:00" />

 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="CGSpace Notes"/>
--- a/docs/page/7/index.html
+++ b/docs/page/7/index.html
@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2020-10-21T15:36:31+03:00" />
+<meta property="og:updated_time" content="2020-10-22T11:58:26+03:00" />

 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="CGSpace Notes"/>
--- a/docs/posts/index.html
+++ b/docs/posts/index.html
@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2020-10-21T15:36:31+03:00" />
+<meta property="og:updated_time" content="2020-10-22T11:58:26+03:00" />

 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Posts"/>
--- a/docs/posts/page/2/index.html
+++ b/docs/posts/page/2/index.html
@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2020-10-21T15:36:31+03:00" />
+<meta property="og:updated_time" content="2020-10-22T11:58:26+03:00" />

 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Posts"/>
--- a/docs/posts/page/3/index.html
+++ b/docs/posts/page/3/index.html
@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2020-10-21T15:36:31+03:00" />
+<meta property="og:updated_time" content="2020-10-22T11:58:26+03:00" />

 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Posts"/>
--- a/docs/posts/page/4/index.html
+++ b/docs/posts/page/4/index.html
@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2020-10-21T15:36:31+03:00" />
+<meta property="og:updated_time" content="2020-10-22T11:58:26+03:00" />

 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Posts"/>
--- a/docs/posts/page/5/index.html
+++ b/docs/posts/page/5/index.html
@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2020-10-21T15:36:31+03:00" />
+<meta property="og:updated_time" content="2020-10-22T11:58:26+03:00" />

 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Posts"/>
--- a/docs/posts/page/6/index.html
+++ b/docs/posts/page/6/index.html
@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2020-10-21T15:36:31+03:00" />
+<meta property="og:updated_time" content="2020-10-22T11:58:26+03:00" />

 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Posts"/>
--- a/docs/posts/page/7/index.html
+++ b/docs/posts/page/7/index.html
@ -9,7 +9,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2020-10-21T15:36:31+03:00" />
+<meta property="og:updated_time" content="2020-10-22T11:58:26+03:00" />

 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="Posts"/>
--- a/docs/sitemap.xml
+++ b/docs/sitemap.xml
@ -4,27 +4,27 @@
  
  <url>
    <loc>https://alanorth.github.io/cgspace-notes/categories/</loc>
-    <lastmod>2020-10-21T15:36:31+03:00</lastmod>
+    <lastmod>2020-10-22T11:58:26+03:00</lastmod>
  </url>
  
  <url>
    <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2020-10-21T15:36:31+03:00</lastmod>
+    <lastmod>2020-10-22T11:58:26+03:00</lastmod>
  </url>
  
  <url>
    <loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
-    <lastmod>2020-10-21T15:36:31+03:00</lastmod>
+    <lastmod>2020-10-22T11:58:26+03:00</lastmod>
  </url>
  
  <url>
    <loc>https://alanorth.github.io/cgspace-notes/2020-10/</loc>
-    <lastmod>2020-10-21T15:36:31+03:00</lastmod>
+    <lastmod>2020-10-22T11:58:26+03:00</lastmod>
  </url>
  
  <url>
    <loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
-    <lastmod>2020-10-21T15:36:31+03:00</lastmod>
+    <lastmod>2020-10-22T11:58:26+03:00</lastmod>
  </url>
  
  <url>