Add notes for 2021-04-18

2025-01-27 05:49:12 +01:00 · 2021-04-18 10:07:54 +03:00
parent e96067374e
commit ff3660d736
23 changed files with 166 additions and 30 deletions
--- a/content/posts/2021-04.md
+++ b/content/posts/2021-04.md
@ -492,7 +492,6 @@ $ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items-
 $ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": false}}'
 $ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
 $ curl -XDELETE 'http://localhost:9200/openrxv-items-final'
-$ curl -XDELETE 'http://localhost:9200/openrxv-items-final'
 ```

 - Then I updated all Docker containers and rebooted the server (linode20) so that the correct indexes would be created again:
@ -611,4 +610,80 @@ $ dspace user -a -m tip-submit@cgiar.org -g CIAT -s Submit -p 'fuuuuuuuu'
 - I added the account to the Alliance Admins account, which is should allow him to submit to any Alliance collection
  - According to my notes from [2020-10]({{< relref "2020-10.md" >}}) the account must be in the admin group in order to submit via the REST API

+## 2021-04-18
+
+- Update all containers on AReS (linode20):
+
+```console
+$ docker images | grep -v ^REPO | sed 's/ \+/:/g' | cut -d: -f1,2 | xargs -L1 docker pull
+```
+
+- Then run all system updates and reboot the server
+- I learned a new command for Elasticsearch:
+
+```console
+$ curl http://localhost:9200/_cat/indices
+yellow open openrxv-values           ChyhGwMDQpevJtlNWO1vcw 1 1   1579      0 537.6kb 537.6kb
+yellow open openrxv-items-temp       PhV5ieuxQsyftByvCxzSIw 1 1 103585 104372 482.7mb 482.7mb
+yellow open openrxv-shared           J_8cxIz6QL6XTRZct7UBBQ 1 1    127      0 115.7kb 115.7kb
+yellow open openrxv-values-00001     jAoXTLR0R9mzivlDVbQaqA 1 1   3903      0 696.2kb 696.2kb
+green  open .kibana_task_manager_1   O1zgJ0YlQhKCFAwJZaNSIA 1 0      2      2  20.6kb  20.6kb
+yellow open openrxv-users            1hWGXh9kS_S6YPxAaBN8ew 1 1      5      0  28.6kb  28.6kb
+green  open .apm-agent-configuration f3RAkSEBRGaxJZs3ePVxsA 1 0      0      0    283b    283b
+yellow open openrxv-items-final      sgk-s8O-RZKdcLRoWt3G8A 1 1    970      0   2.3mb   2.3mb
+green  open .kibana_1                HHPN7RD_T7qe0zDj4rauQw 1 0     25      7  36.8kb  36.8kb
+yellow open users                    M0t2LaZhSm2NrF5xb64dnw 1 1      2      0  11.6kb  11.6kb
+```
+
+- Somehow the `openrxv-items-final` index only has a few items and the majority are in `openrxv-items-temp`, via the `openrxv-items` alias (which is in the temp index):
+
+```console
+$ curl -s 'http://localhost:9200/openrxv-items/_count?q=*&pretty' 
+{
+  "count" : 103585,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+```
+
+- I found a cool tool to help with exporting and restoring Elasticsearch indexes:
+
+```console
+$ elasticdump --input=http://localhost:9200/openrxv-items --output=/home/aorth/openrxv-items_mapping.json --type=mapping
+$ elasticdump --input=http://localhost:9200/openrxv-items --output=/home/aorth/openrxv-items_data.json --limit=1000 --type=data
+...
+Sun, 18 Apr 2021 06:27:07 GMT | Total Writes: 103585
+Sun, 18 Apr 2021 06:27:07 GMT | dump complete
+```
+
+- It took only two or three minutes to export everything...
+- I did a test to restore the index:
+
+```console
+$ elasticdump --input=/home/aorth/openrxv-items_mapping.json --output=http://localhost:9200/openrxv-items-test --type=mapping
+$ elasticdump --input=/home/aorth/openrxv-items_data.json --output=http://localhost:9200/openrxv-items-test --limit 1000 --type=data
+```
+
+- So that's pretty cool!
+- I deleted the `openrxv-items-final` index and `openrxv-items-temp` indexes and then restored the mappings to `openrxv-items-final`, added the `openrxv-items` alias, and started restoring the data to `openrxv-items` with elasticdump:
+
+```console
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-final'
+$ elasticdump --input=/home/aorth/openrxv-items_mapping.json --output=http://localhost:9200/openrxv-items-final --type=mapping
+$ curl -s -X POST 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -d'{"actions" : [{"add" : { "index" : "openrxv-items-final", "alias" : "openrxv-items"}}]}'
+$ elasticdump --input=/home/aorth/openrxv-items_data.json --output=http://localhost:9200/openrxv-items --limit 1000 --type=data
+```
+
+- AReS seems to be working fine аfter that, so I created the `openrxv-items-temp` index and then started a fresh harvest on AReS Explorer:
+
+```console
+$ curl -X PUT "localhost:9200/openrxv-items-temp"
+```
+
+- Run system updates on CGSpace (linode18) and run the latest Ansible infrastructure playbook to update the DSpace Statistics API, PostgreSQL JDBC driver, etc, and then reboot the system
+
 <!-- vim: set sw=2 ts=2: -->