diff --git a/content/posts/2021-03.md b/content/posts/2021-03.md index 4f0660435..03b0b9998 100644 --- a/content/posts/2021-03.md +++ b/content/posts/2021-03.md @@ -162,4 +162,76 @@ $ curl -XDELETE 'http://localhost:9200/openrxv-items-2021-03-05' - [docker/docker-compose.yml: Pin Redis to version 5](https://github.com/ilri/OpenRXV/pull/87) - I deployed the latest changes from the last few days on AReS production +## 2021-03-07 + +- I realized there is something wrong with the Elasticsearch indexes on AReS + - On a new test environment I see `openrxv-items` is correctly an alias of `openrxv-items-final`: + +```console +$ curl -s 'http://localhost:9200/_alias/' | python -m json.tool | less +... + "openrxv-items-final": { + "aliases": { + "openrxv-items": {} + } + }, +``` + +- But on AReS production `openrxv-items` has somehow become an index: + +```console +$ curl -s 'http://localhost:9200/_alias/' | python -m json.tool | less +... + "openrxv-items": { + "aliases": {} + }, + "openrxv-items-final": { + "aliases": {} + }, + "openrxv-items-temp": { + "aliases": {} + }, +``` + +- I fixed the issue on production by cloning the `openrxv-items` index to `openrxv-items-final`, deleting `openrxv-items`, and then re-creating it as an alias: + +```console +$ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}' +$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-2021-03-07 +$ curl -XDELETE 'http://localhost:9200/openrxv-items-final' +$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-final +$ curl -XDELETE 'http://localhost:9200/openrxv-items' +$ curl -s -X POST 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -d'{"actions" : [{"add" : { "index" : "openrxv-items-final", "alias" : "openrxv-items"}}]}' +``` + +- Delete backups and remove read-only mode on `openrxv-items`: + +```console +$ curl -XDELETE 'http://localhost:9200/openrxv-items-2021-03-07' +$ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": false}}' +``` + +- Linode sent alerts about the CPU usage on CGSpace yesterday and the day before + - Looking in the logs I see a few IPs making heavy usage on the REST API and XMLUI: + +```console +# zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/access.log.2.gz /var/log/nginx/access.log.3.gz | grep -E '0[56]/Mar/2021' | goaccess --log-format=COMBINED - +``` + +- I see the usual IPs for CCAFS and ILRI importer bots, but also `143.233.242.132` which appears to be for GARDIAN: + +```console +# zgrep '143.233.242.132' /var/log/nginx/access.log.1 | grep -c Delphi +6237 +# zgrep '143.233.242.132' /var/log/nginx/access.log.1 | grep -c -v Delphi +6418 +``` + +- They seem to make requests twice, once with the Delphi user agent that we know and already mark as a bot, and once with a "normal" user agent + - Looking in Solr I see they have been using this IP for awhile, as they have 100,000 hits going back into 2020 + - I will add this IP to the list of bots in nginx and purge it from Solr with my `check-spider-ip-hits.sh` script +- I made a few changes to OpenRXV: + - [Migrated away from links to use networks](https://github.com/ilri/OpenRXV/issues/89) + - [Converted the backend container to use a custom image that includes `unoconv`](https://github.com/ilri/OpenRXV/issues/68) so we don't have to manually install it anymore + diff --git a/docs/2021-03/index.html b/docs/2021-03/index.html index 2a777317c..7de9c358f 100644 --- a/docs/2021-03/index.html +++ b/docs/2021-03/index.html @@ -19,7 +19,7 @@ Also, we found some issues building and running OpenRXV currently due to ecosyst - + @@ -44,9 +44,9 @@ Also, we found some issues building and running OpenRXV currently due to ecosyst "@type": "BlogPosting", "headline": "March, 2021", "url": "https://alanorth.github.io/cgspace-notes/2021-03/", - "wordCount": "936", + "wordCount": "1306", "datePublished": "2021-03-01T10:13:54+02:00", - "dateModified": "2021-03-05T20:52:36+02:00", + "dateModified": "2021-03-06T13:35:20+02:00", "author": { "@type": "Person", "name": "Alan Orth" @@ -290,6 +290,78 @@ $ curl -XDELETE 'http://localhost:9200/openrxv-items-2021-03-05'
openrxv-items
is correctly an alias of openrxv-items-final
:$ curl -s 'http://localhost:9200/_alias/' | python -m json.tool | less
+...
+ "openrxv-items-final": {
+ "aliases": {
+ "openrxv-items": {}
+ }
+ },
+
openrxv-items
has somehow become an index:$ curl -s 'http://localhost:9200/_alias/' | python -m json.tool | less
+...
+ "openrxv-items": {
+ "aliases": {}
+ },
+ "openrxv-items-final": {
+ "aliases": {}
+ },
+ "openrxv-items-temp": {
+ "aliases": {}
+ },
+
openrxv-items
index to openrxv-items-final
, deleting openrxv-items
, and then re-creating it as an alias:$ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-2021-03-07
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-final'
+$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-final
+$ curl -XDELETE 'http://localhost:9200/openrxv-items'
+$ curl -s -X POST 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -d'{"actions" : [{"add" : { "index" : "openrxv-items-final", "alias" : "openrxv-items"}}]}'
+
openrxv-items
:$ curl -XDELETE 'http://localhost:9200/openrxv-items-2021-03-07'
+$ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": false}}'
+
# zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/access.log.2.gz /var/log/nginx/access.log.3.gz | grep -E '0[56]/Mar/2021' | goaccess --log-format=COMBINED -
+
143.233.242.132
which appears to be for GARDIAN:# zgrep '143.233.242.132' /var/log/nginx/access.log.1 | grep -c Delphi
+6237
+# zgrep '143.233.242.132' /var/log/nginx/access.log.1 | grep -c -v Delphi
+6418
+
check-spider-ip-hits.sh
scriptunoconv
so we don’t have to manually install it anymore