From 75596fd52460e68e78b5e9617e55cc4df37e1a86 Mon Sep 17 00:00:00 2001 From: Alan Orth Date: Sun, 7 Mar 2021 15:51:12 +0200 Subject: [PATCH] Add notes for 2021-03-07 --- content/posts/2021-03.md | 72 +++++++++++++++++++++++ docs/2021-03/index.html | 78 ++++++++++++++++++++++++- docs/categories/index.html | 2 +- docs/categories/notes/index.html | 2 +- docs/categories/notes/page/2/index.html | 2 +- docs/categories/notes/page/3/index.html | 2 +- docs/categories/notes/page/4/index.html | 2 +- docs/categories/notes/page/5/index.html | 2 +- docs/index.html | 2 +- docs/page/2/index.html | 2 +- docs/page/3/index.html | 2 +- docs/page/4/index.html | 2 +- docs/page/5/index.html | 2 +- docs/page/6/index.html | 2 +- docs/page/7/index.html | 2 +- docs/posts/index.html | 2 +- docs/posts/page/2/index.html | 2 +- docs/posts/page/3/index.html | 2 +- docs/posts/page/4/index.html | 2 +- docs/posts/page/5/index.html | 2 +- docs/posts/page/6/index.html | 2 +- docs/posts/page/7/index.html | 2 +- docs/sitemap.xml | 10 ++-- 23 files changed, 172 insertions(+), 28 deletions(-) diff --git a/content/posts/2021-03.md b/content/posts/2021-03.md index 4f0660435..03b0b9998 100644 --- a/content/posts/2021-03.md +++ b/content/posts/2021-03.md @@ -162,4 +162,76 @@ $ curl -XDELETE 'http://localhost:9200/openrxv-items-2021-03-05' - [docker/docker-compose.yml: Pin Redis to version 5](https://github.com/ilri/OpenRXV/pull/87) - I deployed the latest changes from the last few days on AReS production +## 2021-03-07 + +- I realized there is something wrong with the Elasticsearch indexes on AReS + - On a new test environment I see `openrxv-items` is correctly an alias of `openrxv-items-final`: + +```console +$ curl -s 'http://localhost:9200/_alias/' | python -m json.tool | less +... + "openrxv-items-final": { + "aliases": { + "openrxv-items": {} + } + }, +``` + +- But on AReS production `openrxv-items` has somehow become an index: + +```console +$ curl -s 'http://localhost:9200/_alias/' | python -m json.tool | less +... + "openrxv-items": { + "aliases": {} + }, + "openrxv-items-final": { + "aliases": {} + }, + "openrxv-items-temp": { + "aliases": {} + }, +``` + +- I fixed the issue on production by cloning the `openrxv-items` index to `openrxv-items-final`, deleting `openrxv-items`, and then re-creating it as an alias: + +```console +$ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}' +$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-2021-03-07 +$ curl -XDELETE 'http://localhost:9200/openrxv-items-final' +$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-final +$ curl -XDELETE 'http://localhost:9200/openrxv-items' +$ curl -s -X POST 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -d'{"actions" : [{"add" : { "index" : "openrxv-items-final", "alias" : "openrxv-items"}}]}' +``` + +- Delete backups and remove read-only mode on `openrxv-items`: + +```console +$ curl -XDELETE 'http://localhost:9200/openrxv-items-2021-03-07' +$ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": false}}' +``` + +- Linode sent alerts about the CPU usage on CGSpace yesterday and the day before + - Looking in the logs I see a few IPs making heavy usage on the REST API and XMLUI: + +```console +# zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/access.log.2.gz /var/log/nginx/access.log.3.gz | grep -E '0[56]/Mar/2021' | goaccess --log-format=COMBINED - +``` + +- I see the usual IPs for CCAFS and ILRI importer bots, but also `143.233.242.132` which appears to be for GARDIAN: + +```console +# zgrep '143.233.242.132' /var/log/nginx/access.log.1 | grep -c Delphi +6237 +# zgrep '143.233.242.132' /var/log/nginx/access.log.1 | grep -c -v Delphi +6418 +``` + +- They seem to make requests twice, once with the Delphi user agent that we know and already mark as a bot, and once with a "normal" user agent + - Looking in Solr I see they have been using this IP for awhile, as they have 100,000 hits going back into 2020 + - I will add this IP to the list of bots in nginx and purge it from Solr with my `check-spider-ip-hits.sh` script +- I made a few changes to OpenRXV: + - [Migrated away from links to use networks](https://github.com/ilri/OpenRXV/issues/89) + - [Converted the backend container to use a custom image that includes `unoconv`](https://github.com/ilri/OpenRXV/issues/68) so we don't have to manually install it anymore + diff --git a/docs/2021-03/index.html b/docs/2021-03/index.html index 2a777317c..7de9c358f 100644 --- a/docs/2021-03/index.html +++ b/docs/2021-03/index.html @@ -19,7 +19,7 @@ Also, we found some issues building and running OpenRXV currently due to ecosyst - + @@ -44,9 +44,9 @@ Also, we found some issues building and running OpenRXV currently due to ecosyst "@type": "BlogPosting", "headline": "March, 2021", "url": "https://alanorth.github.io/cgspace-notes/2021-03/", - "wordCount": "936", + "wordCount": "1306", "datePublished": "2021-03-01T10:13:54+02:00", - "dateModified": "2021-03-05T20:52:36+02:00", + "dateModified": "2021-03-06T13:35:20+02:00", "author": { "@type": "Person", "name": "Alan Orth" @@ -290,6 +290,78 @@ $ curl -XDELETE 'http://localhost:9200/openrxv-items-2021-03-05'
  • I deployed the latest changes from the last few days on AReS production
    • +

    2021-03-07

    + +
    $ curl -s 'http://localhost:9200/_alias/' | python -m json.tool | less
+...
+    "openrxv-items-final": {
+        "aliases": {
+            "openrxv-items": {}
+        }
+    },
+
    +
    $ curl -s 'http://localhost:9200/_alias/' | python -m json.tool | less
+...
+    "openrxv-items": {
+        "aliases": {}
+    },
+    "openrxv-items-final": {
+        "aliases": {}
+    },
+    "openrxv-items-temp": {
+        "aliases": {}
+    },
+
    +
    $ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-2021-03-07
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-final'
+$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-final
+$ curl -XDELETE 'http://localhost:9200/openrxv-items'
+$ curl -s -X POST 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -d'{"actions" : [{"add" : { "index" : "openrxv-items-final", "alias" : "openrxv-items"}}]}'
+
    +
    $ curl -XDELETE 'http://localhost:9200/openrxv-items-2021-03-07'
+$ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": false}}'
+
    +
    # zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/access.log.2.gz /var/log/nginx/access.log.3.gz | grep -E '0[56]/Mar/2021' | goaccess --log-format=COMBINED -
+
    +
    # zgrep '143.233.242.132' /var/log/nginx/access.log.1 | grep -c Delphi
+6237
+# zgrep '143.233.242.132' /var/log/nginx/access.log.1 | grep -c -v Delphi
+6418
+
    diff --git a/docs/categories/index.html b/docs/categories/index.html index 9bb066de5..107fc3d7a 100644 --- a/docs/categories/index.html +++ b/docs/categories/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/index.html b/docs/categories/notes/index.html index ec2c3f468..aa4edb700 100644 --- a/docs/categories/notes/index.html +++ b/docs/categories/notes/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/2/index.html b/docs/categories/notes/page/2/index.html index 052e38bc7..bf96dc8ad 100644 --- a/docs/categories/notes/page/2/index.html +++ b/docs/categories/notes/page/2/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/3/index.html b/docs/categories/notes/page/3/index.html index 1a303084b..795dd6f18 100644 --- a/docs/categories/notes/page/3/index.html +++ b/docs/categories/notes/page/3/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/4/index.html b/docs/categories/notes/page/4/index.html index cc3cf52b5..164b1b117 100644 --- a/docs/categories/notes/page/4/index.html +++ b/docs/categories/notes/page/4/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/5/index.html b/docs/categories/notes/page/5/index.html index 6f08a0684..4fa11784b 100644 --- a/docs/categories/notes/page/5/index.html +++ b/docs/categories/notes/page/5/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/index.html b/docs/index.html index 927db07d8..66a2b97da 100644 --- a/docs/index.html +++ b/docs/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/2/index.html b/docs/page/2/index.html index c7081f263..55992e57c 100644 --- a/docs/page/2/index.html +++ b/docs/page/2/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/3/index.html b/docs/page/3/index.html index 8149c44b5..391c6b952 100644 --- a/docs/page/3/index.html +++ b/docs/page/3/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/4/index.html b/docs/page/4/index.html index 993bc9189..f35214fbe 100644 --- a/docs/page/4/index.html +++ b/docs/page/4/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/5/index.html b/docs/page/5/index.html index d424e87b5..2fc28a652 100644 --- a/docs/page/5/index.html +++ b/docs/page/5/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/6/index.html b/docs/page/6/index.html index 11f129fac..d7159fc06 100644 --- a/docs/page/6/index.html +++ b/docs/page/6/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/7/index.html b/docs/page/7/index.html index edd2c6bcc..bc03612b4 100644 --- a/docs/page/7/index.html +++ b/docs/page/7/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/index.html b/docs/posts/index.html index de4152b96..27deae005 100644 --- a/docs/posts/index.html +++ b/docs/posts/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/2/index.html b/docs/posts/page/2/index.html index 65ea78878..98539984f 100644 --- a/docs/posts/page/2/index.html +++ b/docs/posts/page/2/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/3/index.html b/docs/posts/page/3/index.html index 5bf016d37..a5994d525 100644 --- a/docs/posts/page/3/index.html +++ b/docs/posts/page/3/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/4/index.html b/docs/posts/page/4/index.html index a86f16766..65b951983 100644 --- a/docs/posts/page/4/index.html +++ b/docs/posts/page/4/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/5/index.html b/docs/posts/page/5/index.html index 91f83e7b1..8ce21adfe 100644 --- a/docs/posts/page/5/index.html +++ b/docs/posts/page/5/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/6/index.html b/docs/posts/page/6/index.html index 9e5a49370..02edebdeb 100644 --- a/docs/posts/page/6/index.html +++ b/docs/posts/page/6/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/7/index.html b/docs/posts/page/7/index.html index 556246998..ab8dd038f 100644 --- a/docs/posts/page/7/index.html +++ b/docs/posts/page/7/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/sitemap.xml b/docs/sitemap.xml index 966893480..da55fb7bc 100644 --- a/docs/sitemap.xml +++ b/docs/sitemap.xml @@ -3,19 +3,19 @@ xmlns:xhtml="http://www.w3.org/1999/xhtml"> https://alanorth.github.io/cgspace-notes/categories/ - 2021-03-05T20:52:36+02:00 + 2021-03-06T13:35:20+02:00 https://alanorth.github.io/cgspace-notes/ - 2021-03-05T20:52:36+02:00 + 2021-03-06T13:35:20+02:00 https://alanorth.github.io/cgspace-notes/2021-03/ - 2021-03-05T20:52:36+02:00 + 2021-03-06T13:35:20+02:00 https://alanorth.github.io/cgspace-notes/categories/notes/ - 2021-03-05T20:52:36+02:00 + 2021-03-06T13:35:20+02:00 https://alanorth.github.io/cgspace-notes/posts/ - 2021-03-05T20:52:36+02:00 + 2021-03-06T13:35:20+02:00 https://alanorth.github.io/cgspace-notes/2021-02/ 2021-03-04T22:46:05+02:00