From ff3660d7360e38690affcd5fcf011b227c9ea753 Mon Sep 17 00:00:00 2001 From: Alan Orth Date: Sun, 18 Apr 2021 10:07:54 +0300 Subject: [PATCH] Add notes for 2021-04-18 --- content/posts/2021-04.md | 77 ++++++++++++++++++++++++- docs/2021-04/index.html | 69 ++++++++++++++++++++-- docs/categories/index.html | 2 +- docs/categories/notes/index.html | 2 +- docs/categories/notes/page/2/index.html | 2 +- docs/categories/notes/page/3/index.html | 2 +- docs/categories/notes/page/4/index.html | 2 +- docs/categories/notes/page/5/index.html | 2 +- docs/index.html | 2 +- docs/page/2/index.html | 2 +- docs/page/3/index.html | 2 +- docs/page/4/index.html | 2 +- docs/page/5/index.html | 2 +- docs/page/6/index.html | 2 +- docs/page/7/index.html | 2 +- docs/posts/index.html | 2 +- docs/posts/page/2/index.html | 2 +- docs/posts/page/3/index.html | 2 +- docs/posts/page/4/index.html | 2 +- docs/posts/page/5/index.html | 2 +- docs/posts/page/6/index.html | 2 +- docs/posts/page/7/index.html | 2 +- docs/sitemap.xml | 10 ++-- 23 files changed, 166 insertions(+), 30 deletions(-) diff --git a/content/posts/2021-04.md b/content/posts/2021-04.md index d0fca0747..f1d37ca08 100644 --- a/content/posts/2021-04.md +++ b/content/posts/2021-04.md @@ -492,7 +492,6 @@ $ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items- $ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": false}}' $ curl -XDELETE 'http://localhost:9200/openrxv-items-temp' $ curl -XDELETE 'http://localhost:9200/openrxv-items-final' -$ curl -XDELETE 'http://localhost:9200/openrxv-items-final' ``` - Then I updated all Docker containers and rebooted the server (linode20) so that the correct indexes would be created again: @@ -611,4 +610,80 @@ $ dspace user -a -m tip-submit@cgiar.org -g CIAT -s Submit -p 'fuuuuuuuu' - I added the account to the Alliance Admins account, which is should allow him to submit to any Alliance collection - According to my notes from [2020-10]({{< relref "2020-10.md" >}}) the account must be in the admin group in order to submit via the REST API +## 2021-04-18 + +- Update all containers on AReS (linode20): + +```console +$ docker images | grep -v ^REPO | sed 's/ \+/:/g' | cut -d: -f1,2 | xargs -L1 docker pull +``` + +- Then run all system updates and reboot the server +- I learned a new command for Elasticsearch: + +```console +$ curl http://localhost:9200/_cat/indices +yellow open openrxv-values ChyhGwMDQpevJtlNWO1vcw 1 1 1579 0 537.6kb 537.6kb +yellow open openrxv-items-temp PhV5ieuxQsyftByvCxzSIw 1 1 103585 104372 482.7mb 482.7mb +yellow open openrxv-shared J_8cxIz6QL6XTRZct7UBBQ 1 1 127 0 115.7kb 115.7kb +yellow open openrxv-values-00001 jAoXTLR0R9mzivlDVbQaqA 1 1 3903 0 696.2kb 696.2kb +green open .kibana_task_manager_1 O1zgJ0YlQhKCFAwJZaNSIA 1 0 2 2 20.6kb 20.6kb +yellow open openrxv-users 1hWGXh9kS_S6YPxAaBN8ew 1 1 5 0 28.6kb 28.6kb +green open .apm-agent-configuration f3RAkSEBRGaxJZs3ePVxsA 1 0 0 0 283b 283b +yellow open openrxv-items-final sgk-s8O-RZKdcLRoWt3G8A 1 1 970 0 2.3mb 2.3mb +green open .kibana_1 HHPN7RD_T7qe0zDj4rauQw 1 0 25 7 36.8kb 36.8kb +yellow open users M0t2LaZhSm2NrF5xb64dnw 1 1 2 0 11.6kb 11.6kb +``` + +- Somehow the `openrxv-items-final` index only has a few items and the majority are in `openrxv-items-temp`, via the `openrxv-items` alias (which is in the temp index): + +```console +$ curl -s 'http://localhost:9200/openrxv-items/_count?q=*&pretty' +{ + "count" : 103585, + "_shards" : { + "total" : 1, + "successful" : 1, + "skipped" : 0, + "failed" : 0 + } +} +``` + +- I found a cool tool to help with exporting and restoring Elasticsearch indexes: + +```console +$ elasticdump --input=http://localhost:9200/openrxv-items --output=/home/aorth/openrxv-items_mapping.json --type=mapping +$ elasticdump --input=http://localhost:9200/openrxv-items --output=/home/aorth/openrxv-items_data.json --limit=1000 --type=data +... +Sun, 18 Apr 2021 06:27:07 GMT | Total Writes: 103585 +Sun, 18 Apr 2021 06:27:07 GMT | dump complete +``` + +- It took only two or three minutes to export everything... +- I did a test to restore the index: + +```console +$ elasticdump --input=/home/aorth/openrxv-items_mapping.json --output=http://localhost:9200/openrxv-items-test --type=mapping +$ elasticdump --input=/home/aorth/openrxv-items_data.json --output=http://localhost:9200/openrxv-items-test --limit 1000 --type=data +``` + +- So that's pretty cool! +- I deleted the `openrxv-items-final` index and `openrxv-items-temp` indexes and then restored the mappings to `openrxv-items-final`, added the `openrxv-items` alias, and started restoring the data to `openrxv-items` with elasticdump: + +```console +$ curl -XDELETE 'http://localhost:9200/openrxv-items-final' +$ elasticdump --input=/home/aorth/openrxv-items_mapping.json --output=http://localhost:9200/openrxv-items-final --type=mapping +$ curl -s -X POST 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -d'{"actions" : [{"add" : { "index" : "openrxv-items-final", "alias" : "openrxv-items"}}]}' +$ elasticdump --input=/home/aorth/openrxv-items_data.json --output=http://localhost:9200/openrxv-items --limit 1000 --type=data +``` + +- AReS seems to be working fine аfter that, so I created the `openrxv-items-temp` index and then started a fresh harvest on AReS Explorer: + +```console +$ curl -X PUT "localhost:9200/openrxv-items-temp" +``` + +- Run system updates on CGSpace (linode18) and run the latest Ansible infrastructure playbook to update the DSpace Statistics API, PostgreSQL JDBC driver, etc, and then reboot the system + diff --git a/docs/2021-04/index.html b/docs/2021-04/index.html index 76f40df91..1a13a4563 100644 --- a/docs/2021-04/index.html +++ b/docs/2021-04/index.html @@ -24,7 +24,7 @@ Perhaps one of the containers crashed, I should have looked closer but I was in - + @@ -54,9 +54,9 @@ Perhaps one of the containers crashed, I should have looked closer but I was in "@type": "BlogPosting", "headline": "April, 2021", "url": "https://alanorth.github.io/cgspace-notes/2021-04/", - "wordCount": "3137", + "wordCount": "3527", "datePublished": "2021-04-01T09:50:54+03:00", - "dateModified": "2021-04-13T21:13:08+03:00", + "dateModified": "2021-04-15T16:27:06+03:00", "author": { "@type": "Person", "name": "Alan Orth" @@ -613,7 +613,6 @@ $ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items- $ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": false}}' $ curl -XDELETE 'http://localhost:9200/openrxv-items-temp' $ curl -XDELETE 'http://localhost:9200/openrxv-items-final' -$ curl -XDELETE 'http://localhost:9200/openrxv-items-final' @@ -732,6 +731,68 @@ Total number of bot hits purged: 13159 +

2021-04-18

+ +
$ docker images | grep -v ^REPO | sed 's/ \+/:/g' | cut -d: -f1,2 | xargs -L1 docker pull
+
+
$ curl http://localhost:9200/_cat/indices
+yellow open openrxv-values           ChyhGwMDQpevJtlNWO1vcw 1 1   1579      0 537.6kb 537.6kb
+yellow open openrxv-items-temp       PhV5ieuxQsyftByvCxzSIw 1 1 103585 104372 482.7mb 482.7mb
+yellow open openrxv-shared           J_8cxIz6QL6XTRZct7UBBQ 1 1    127      0 115.7kb 115.7kb
+yellow open openrxv-values-00001     jAoXTLR0R9mzivlDVbQaqA 1 1   3903      0 696.2kb 696.2kb
+green  open .kibana_task_manager_1   O1zgJ0YlQhKCFAwJZaNSIA 1 0      2      2  20.6kb  20.6kb
+yellow open openrxv-users            1hWGXh9kS_S6YPxAaBN8ew 1 1      5      0  28.6kb  28.6kb
+green  open .apm-agent-configuration f3RAkSEBRGaxJZs3ePVxsA 1 0      0      0    283b    283b
+yellow open openrxv-items-final      sgk-s8O-RZKdcLRoWt3G8A 1 1    970      0   2.3mb   2.3mb
+green  open .kibana_1                HHPN7RD_T7qe0zDj4rauQw 1 0     25      7  36.8kb  36.8kb
+yellow open users                    M0t2LaZhSm2NrF5xb64dnw 1 1      2      0  11.6kb  11.6kb
+
+
$ curl -s 'http://localhost:9200/openrxv-items/_count?q=*&pretty' 
+{
+  "count" : 103585,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+
+
$ elasticdump --input=http://localhost:9200/openrxv-items --output=/home/aorth/openrxv-items_mapping.json --type=mapping
+$ elasticdump --input=http://localhost:9200/openrxv-items --output=/home/aorth/openrxv-items_data.json --limit=1000 --type=data
+...
+Sun, 18 Apr 2021 06:27:07 GMT | Total Writes: 103585
+Sun, 18 Apr 2021 06:27:07 GMT | dump complete
+
+
$ elasticdump --input=/home/aorth/openrxv-items_mapping.json --output=http://localhost:9200/openrxv-items-test --type=mapping
+$ elasticdump --input=/home/aorth/openrxv-items_data.json --output=http://localhost:9200/openrxv-items-test --limit 1000 --type=data
+
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items-final'
+$ elasticdump --input=/home/aorth/openrxv-items_mapping.json --output=http://localhost:9200/openrxv-items-final --type=mapping
+$ curl -s -X POST 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -d'{"actions" : [{"add" : { "index" : "openrxv-items-final", "alias" : "openrxv-items"}}]}'
+$ elasticdump --input=/home/aorth/openrxv-items_data.json --output=http://localhost:9200/openrxv-items --limit 1000 --type=data
+
+
$ curl -X PUT "localhost:9200/openrxv-items-temp"
+
diff --git a/docs/categories/index.html b/docs/categories/index.html index 061e88240..8c8fa617c 100644 --- a/docs/categories/index.html +++ b/docs/categories/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/index.html b/docs/categories/notes/index.html index a1b9131c5..ac460575d 100644 --- a/docs/categories/notes/index.html +++ b/docs/categories/notes/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/2/index.html b/docs/categories/notes/page/2/index.html index 654724bfe..02a849b41 100644 --- a/docs/categories/notes/page/2/index.html +++ b/docs/categories/notes/page/2/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/3/index.html b/docs/categories/notes/page/3/index.html index 317ce7a71..d24a530af 100644 --- a/docs/categories/notes/page/3/index.html +++ b/docs/categories/notes/page/3/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/4/index.html b/docs/categories/notes/page/4/index.html index 1ce250858..abefd62c7 100644 --- a/docs/categories/notes/page/4/index.html +++ b/docs/categories/notes/page/4/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/5/index.html b/docs/categories/notes/page/5/index.html index 47533069e..2cc0656df 100644 --- a/docs/categories/notes/page/5/index.html +++ b/docs/categories/notes/page/5/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/index.html b/docs/index.html index 3a0121179..ba052455c 100644 --- a/docs/index.html +++ b/docs/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/2/index.html b/docs/page/2/index.html index 01d75b770..a905b15d7 100644 --- a/docs/page/2/index.html +++ b/docs/page/2/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/3/index.html b/docs/page/3/index.html index e5c28e84a..5df86e5ea 100644 --- a/docs/page/3/index.html +++ b/docs/page/3/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/4/index.html b/docs/page/4/index.html index a3f4c7343..163ab79f1 100644 --- a/docs/page/4/index.html +++ b/docs/page/4/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/5/index.html b/docs/page/5/index.html index 848fbac91..50a847bb9 100644 --- a/docs/page/5/index.html +++ b/docs/page/5/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/6/index.html b/docs/page/6/index.html index 9311dfe67..b93a186ac 100644 --- a/docs/page/6/index.html +++ b/docs/page/6/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/7/index.html b/docs/page/7/index.html index 9dfd513ba..92e7c964f 100644 --- a/docs/page/7/index.html +++ b/docs/page/7/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/index.html b/docs/posts/index.html index 50606e5bb..ac4960bca 100644 --- a/docs/posts/index.html +++ b/docs/posts/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/2/index.html b/docs/posts/page/2/index.html index 50f401e7e..04361daa1 100644 --- a/docs/posts/page/2/index.html +++ b/docs/posts/page/2/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/3/index.html b/docs/posts/page/3/index.html index 681f33496..7101b4190 100644 --- a/docs/posts/page/3/index.html +++ b/docs/posts/page/3/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/4/index.html b/docs/posts/page/4/index.html index e1b20f30d..8cf6c2ac6 100644 --- a/docs/posts/page/4/index.html +++ b/docs/posts/page/4/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/5/index.html b/docs/posts/page/5/index.html index 92481b6fc..409899b3e 100644 --- a/docs/posts/page/5/index.html +++ b/docs/posts/page/5/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/6/index.html b/docs/posts/page/6/index.html index 3ff850050..9b6b7e936 100644 --- a/docs/posts/page/6/index.html +++ b/docs/posts/page/6/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/7/index.html b/docs/posts/page/7/index.html index d93baca4b..34d76304f 100644 --- a/docs/posts/page/7/index.html +++ b/docs/posts/page/7/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/sitemap.xml b/docs/sitemap.xml index 43bbc76d1..59d4ae09d 100644 --- a/docs/sitemap.xml +++ b/docs/sitemap.xml @@ -3,19 +3,19 @@ xmlns:xhtml="http://www.w3.org/1999/xhtml"> https://alanorth.github.io/cgspace-notes/2021-04/ - 2021-04-13T21:13:08+03:00 + 2021-04-15T16:27:06+03:00 https://alanorth.github.io/cgspace-notes/categories/ - 2021-04-13T21:13:08+03:00 + 2021-04-15T16:27:06+03:00 https://alanorth.github.io/cgspace-notes/ - 2021-04-13T21:13:08+03:00 + 2021-04-15T16:27:06+03:00 https://alanorth.github.io/cgspace-notes/categories/notes/ - 2021-04-13T21:13:08+03:00 + 2021-04-15T16:27:06+03:00 https://alanorth.github.io/cgspace-notes/posts/ - 2021-04-13T21:13:08+03:00 + 2021-04-15T16:27:06+03:00 https://alanorth.github.io/cgspace-notes/2021-03/ 2021-04-13T21:13:08+03:00