Add notes for 2021-08-17

content/posts/2021-08.md

@@ -294,5 +294,54 @@ $ dspace community-filiator --set --parent=10568/114644 --child=10568/76451
 
 - I made an initial attempt on the policy statements page on DSpace Test
   - It is modeled on Sherpa Romeo's OpenDOAR policy statements advice
+- Sit with Moayad and discuss the future of AReS
+  - We specifically discussed formalizing the API and documenting its use to allow as an alternative to harvesting directly from CGSpace
+  - We also discussed allowing linking to search results to enable something like "Explore this collection" links on CGSpace collection pages
+- Lower case all AGROVOC metadata, as I had noticed a few in sentence case:
+
+```console
+dspace=# UPDATE metadatavalue SET text_value=LOWER(text_value) WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=187 AND text_value ~ '[[:upper:]]';
+UPDATE 484
+```
+
+- Also update some DOIs using the `dx.doi.org` format, just to keep things uniform:
+
+```console
+dspace=# UPDATE metadatavalue SET text_value = regexp_replace(text_value, 'https://dx.doi.org', 'https://doi.org') WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 220 AND text_value LIKE 'https://dx.doi.org%';
+UPDATE 469
+```
+
+- Then start a full Discovery re-indexing to update the Feed the Future community item counts that have been stuck at 0 since we moved the three projects to be a subcommunity a few days ago:
+
+```console
+$ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    322m16.917s
+user    226m43.121s
+sys     3m17.469s
+```
+
+- I learned how to use the OpenRXV API, which is just a thin wrapper around Elasticsearch:
+
+```console
+$ curl -X POST 'https://cgspace.cgiar.org/explorer/api/search?scroll=1d' \
+    -H 'Content-Type: application/json' \
+    -d '{
+    "size": 10,
+    "query": {
+        "bool": {
+            "filter": {
+                "term": {
+                    "repo.keyword": "CGSpace"
+                }
+            }
+        }
+    }
+}'
+$ curl -X POST 'https://cgspace.cgiar.org/explorer/api/search/scroll/DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAASekWMTRwZ3lEMkVRYUtKZjgyMno4dV9CUQ==
+```
+
+- This uses the Elasticsearch scroll ID to page through results
+  - The second query doesn't need the request body because it is saved for 1 day as part of the first request
 
 <!-- v[im: set sw=2 ts=2: -->

docs/2021-08/index.html

@@ -18,7 +18,7 @@ I decided to upgrade linode20 from Ubuntu 18.04 to 20.04
 <meta property="og:type" content="article" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2021-08/" />
 <meta property="article:published_time" content="2021-08-01T09:01:07+03:00" />
-<meta property="article:modified_time" content="2021-08-16T21:35:44+03:00" />
+<meta property="article:modified_time" content="2021-08-17T10:59:14+03:00" />
 
 
 
@@ -42,9 +42,9 @@ I decided to upgrade linode20 from Ubuntu 18.04 to 20.04
   "@type": "BlogPosting",
   "headline": "August, 2021",
   "url": "https://alanorth.github.io/cgspace-notes/2021-08/",
-  "wordCount": "2255",
+  "wordCount": "2512",
   "datePublished": "2021-08-01T09:01:07+03:00",
-  "dateModified": "2021-08-16T21:35:44+03:00",
+  "dateModified": "2021-08-17T10:59:14+03:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"
@@ -438,6 +438,53 @@ $ dspace community-filiator --set --parent=10568/114644 --child=10568/76451
 <li>It is modeled on Sherpa Romeo&rsquo;s OpenDOAR policy statements advice</li>
 </ul>
 </li>
+<li>Sit with Moayad and discuss the future of AReS
+<ul>
+<li>We specifically discussed formalizing the API and documenting its use to allow as an alternative to harvesting directly from CGSpace</li>
+<li>We also discussed allowing linking to search results to enable something like &ldquo;Explore this collection&rdquo; links on CGSpace collection pages</li>
+</ul>
+</li>
+<li>Lower case all AGROVOC metadata, as I had noticed a few in sentence case:</li>
+</ul>
+<pre><code class="language-console" data-lang="console">dspace=# UPDATE metadatavalue SET text_value=LOWER(text_value) WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=187 AND text_value ~ '[[:upper:]]';
+UPDATE 484
+</code></pre><ul>
+<li>Also update some DOIs using the <code>dx.doi.org</code> format, just to keep things uniform:</li>
+</ul>
+<pre><code class="language-console" data-lang="console">dspace=# UPDATE metadatavalue SET text_value = regexp_replace(text_value, 'https://dx.doi.org', 'https://doi.org') WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 220 AND text_value LIKE 'https://dx.doi.org%';
+UPDATE 469
+</code></pre><ul>
+<li>Then start a full Discovery re-indexing to update the Feed the Future community item counts that have been stuck at 0 since we moved the three projects to be a subcommunity a few days ago:</li>
+</ul>
+<pre><code class="language-console" data-lang="console">$ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    322m16.917s
+user    226m43.121s
+sys     3m17.469s
+</code></pre><ul>
+<li>I learned how to use the OpenRXV API, which is just a thin wrapper around Elasticsearch:</li>
+</ul>
+<pre><code class="language-console" data-lang="console">$ curl -X POST 'https://cgspace.cgiar.org/explorer/api/search?scroll=1d' \
+    -H 'Content-Type: application/json' \
+    -d '{
+    &quot;size&quot;: 10,
+    &quot;query&quot;: {
+        &quot;bool&quot;: {
+            &quot;filter&quot;: {
+                &quot;term&quot;: {
+                    &quot;repo.keyword&quot;: &quot;CGSpace&quot;
+                }
+            }
+        }
+    }
+}'
+$ curl -X POST 'https://cgspace.cgiar.org/explorer/api/search/scroll/DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAASekWMTRwZ3lEMkVRYUtKZjgyMno4dV9CUQ==
+</code></pre><ul>
+<li>This uses the Elasticsearch scroll ID to page through results
+<ul>
+<li>The second query doesn&rsquo;t need the request body because it is saved for 1 day as part of the first request</li>
+</ul>
+</li>
 </ul>
 <!-- raw HTML omitted -->
 

docs/categories/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/" />
-<meta property="og:updated_time" content="2021-08-16T21:35:44+03:00" />
+<meta property="og:updated_time" content="2021-08-17T10:59:14+03:00" />
 
 
 

docs/categories/notes/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-08-16T21:35:44+03:00" />
+<meta property="og:updated_time" content="2021-08-17T10:59:14+03:00" />
 
 
 

docs/categories/notes/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-08-16T21:35:44+03:00" />
+<meta property="og:updated_time" content="2021-08-17T10:59:14+03:00" />
 
 
 

docs/categories/notes/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-08-16T21:35:44+03:00" />
+<meta property="og:updated_time" content="2021-08-17T10:59:14+03:00" />
 
 
 

docs/categories/notes/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-08-16T21:35:44+03:00" />
+<meta property="og:updated_time" content="2021-08-17T10:59:14+03:00" />
 
 
 

docs/categories/notes/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-08-16T21:35:44+03:00" />
+<meta property="og:updated_time" content="2021-08-17T10:59:14+03:00" />
 
 
 

docs/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-08-16T21:35:44+03:00" />
+<meta property="og:updated_time" content="2021-08-17T10:59:14+03:00" />
 
 
 

docs/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-08-16T21:35:44+03:00" />
+<meta property="og:updated_time" content="2021-08-17T10:59:14+03:00" />
 
 
 

docs/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-08-16T21:35:44+03:00" />
+<meta property="og:updated_time" content="2021-08-17T10:59:14+03:00" />
 
 
 

docs/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-08-16T21:35:44+03:00" />
+<meta property="og:updated_time" content="2021-08-17T10:59:14+03:00" />
 
 
 

docs/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-08-16T21:35:44+03:00" />
+<meta property="og:updated_time" content="2021-08-17T10:59:14+03:00" />
 
 
 

docs/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-08-16T21:35:44+03:00" />
+<meta property="og:updated_time" content="2021-08-17T10:59:14+03:00" />
 
 
 

docs/page/7/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-08-16T21:35:44+03:00" />
+<meta property="og:updated_time" content="2021-08-17T10:59:14+03:00" />
 
 
 

docs/page/8/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-08-16T21:35:44+03:00" />
+<meta property="og:updated_time" content="2021-08-17T10:59:14+03:00" />
 
 
 

docs/posts/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-08-16T21:35:44+03:00" />
+<meta property="og:updated_time" content="2021-08-17T10:59:14+03:00" />
 
 
 

docs/posts/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-08-16T21:35:44+03:00" />
+<meta property="og:updated_time" content="2021-08-17T10:59:14+03:00" />
 
 
 

docs/posts/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-08-16T21:35:44+03:00" />
+<meta property="og:updated_time" content="2021-08-17T10:59:14+03:00" />
 
 
 

docs/posts/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-08-16T21:35:44+03:00" />
+<meta property="og:updated_time" content="2021-08-17T10:59:14+03:00" />
 
 
 

docs/posts/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-08-16T21:35:44+03:00" />
+<meta property="og:updated_time" content="2021-08-17T10:59:14+03:00" />
 
 
 

docs/posts/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-08-16T21:35:44+03:00" />
+<meta property="og:updated_time" content="2021-08-17T10:59:14+03:00" />
 
 
 

docs/posts/page/7/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-08-16T21:35:44+03:00" />
+<meta property="og:updated_time" content="2021-08-17T10:59:14+03:00" />
 
 
 

docs/posts/page/8/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-08-16T21:35:44+03:00" />
+<meta property="og:updated_time" content="2021-08-17T10:59:14+03:00" />
 
 
 

docs/sitemap.xml

@@ -3,19 +3,19 @@
   xmlns:xhtml="http://www.w3.org/1999/xhtml">
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2021-08/</loc>
-    <lastmod>2021-08-16T21:35:44+03:00</lastmod>
+    <lastmod>2021-08-17T10:59:14+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/</loc>
-    <lastmod>2021-08-16T21:35:44+03:00</lastmod>
+    <lastmod>2021-08-17T10:59:14+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2021-08-16T21:35:44+03:00</lastmod>
+    <lastmod>2021-08-17T10:59:14+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
-    <lastmod>2021-08-16T21:35:44+03:00</lastmod>
+    <lastmod>2021-08-17T10:59:14+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
-    <lastmod>2021-08-16T21:35:44+03:00</lastmod>
+    <lastmod>2021-08-17T10:59:14+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/2021-07/</loc>
     <lastmod>2021-08-01T16:19:05+03:00</lastmod>