diff --git a/content/posts/2021-03.md b/content/posts/2021-03.md index 03b0b9998..e1c7d00da 100644 --- a/content/posts/2021-03.md +++ b/content/posts/2021-03.md @@ -234,4 +234,46 @@ $ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: applica - [Migrated away from links to use networks](https://github.com/ilri/OpenRXV/issues/89) - [Converted the backend container to use a custom image that includes `unoconv`](https://github.com/ilri/OpenRXV/issues/68) so we don't have to manually install it anymore +## 2021-03-08 + +- I approved the WLE item that I edited last week, and all the metadata is there: https://hdl.handle.net/10568/111810 + - So I'm not sure what Niroshini's issue with metadata is... +- Peter sent a message yesterday saying that his item finally got committed + - I looked at the Munin graphs and there was a MASSIVE spike in database activity two days ago, and now database locks are back down to normal levels (from 1000+): + +```console +$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l +13 +``` + +- On 2021-03-03 the PostgreSQL transactions started rising: + +![PostgreSQL query length week](/cgspace-notes/2021/03/postgres_querylength_ALL-week.png) + +- After that the connections and locks started going up, peaking on 2021-03-06: + +![PostgreSQL locks week](/cgspace-notes/2021/03/postgres_locks_ALL-week.png) +![PostgreSQL connections week](/cgspace-notes/2021/03/postgres_connections_ALL-week.png) + +- I sent another message to Atmire to ask if they have time to look into this +- CIFOR is pressuring me to upload the batch items from last week + - Vika sent me a final file with some duplicates that Peter identified removed + - I extracted and re-applied my basic corrections from last week in OpenRefine, then ran the items through `csv-metadata-quality` checker and uploaded them to CGSpace + - In total there are 1,088 items +- Udana from IWMI emailed to ask about CGSpace thumbnails +- Udana from IWMI emailed to ask about an item uploaded recently that does not appear in AReS + - [The item](https://hdl.handle.net/10568/111794) was added to the archive on 2021-03-05, and I last harvested on 2021-03-06, so this might be an issue of a missing item +- Abenet got a quote from Atmire to buy 125 credits for 3750€ +- Maria at Bioversity sent some feedback about duplicate items on AReS +- I'm wondering if the issue of the `openrxv-items-final` index not getting cleared after a successful harvest (which results in having 200,000, then 300,000, etc items) has to do with the alias issue I fixed yesterday + - I will start a fresh harvest on AReS without now to check, but first back up the current index just in case: + +```console +$ curl -X PUT "localhost:9200/openrxv-items-final/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}' +$ curl -s -X POST http://localhost:9200/openrxv-items-final/_clone/openrxv-items-final-2021-03-08 +# start harvesting on AReS +``` + +- As I saw on my local test instance, even when you cancel a harvesting, it replaces the `openrxv-items-final` index with whatever is in `openrxv-items-temp` automatically, so I assume it will do the same now + diff --git a/docs/2021-03/index.html b/docs/2021-03/index.html index 7de9c358f..769c18cd9 100644 --- a/docs/2021-03/index.html +++ b/docs/2021-03/index.html @@ -19,7 +19,7 @@ Also, we found some issues building and running OpenRXV currently due to ecosyst - + @@ -44,9 +44,9 @@ Also, we found some issues building and running OpenRXV currently due to ecosyst "@type": "BlogPosting", "headline": "March, 2021", "url": "https://alanorth.github.io/cgspace-notes/2021-03/", - "wordCount": "1306", + "wordCount": "1673", "datePublished": "2021-03-01T10:13:54+02:00", - "dateModified": "2021-03-06T13:35:20+02:00", + "dateModified": "2021-03-07T15:51:12+02:00", "author": { "@type": "Person", "name": "Alan Orth" @@ -362,6 +362,59 @@ $ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Typ +

2021-03-08

+ +
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+13
+
+

PostgreSQL query length week

+ +

PostgreSQL locks week +PostgreSQL connections week

+ +
$ curl -X PUT "localhost:9200/openrxv-items-final/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items-final/_clone/openrxv-items-final-2021-03-08
+# start harvesting on AReS
+
diff --git a/docs/2021/03/postgres_connections_ALL-week.png b/docs/2021/03/postgres_connections_ALL-week.png new file mode 100644 index 000000000..3b9aa9ee9 Binary files /dev/null and b/docs/2021/03/postgres_connections_ALL-week.png differ diff --git a/docs/2021/03/postgres_locks_ALL-week.png b/docs/2021/03/postgres_locks_ALL-week.png index 59eafded7..aa44644c4 100644 Binary files a/docs/2021/03/postgres_locks_ALL-week.png and b/docs/2021/03/postgres_locks_ALL-week.png differ diff --git a/docs/2021/03/postgres_querylength_ALL-week.png b/docs/2021/03/postgres_querylength_ALL-week.png new file mode 100644 index 000000000..5072ac336 Binary files /dev/null and b/docs/2021/03/postgres_querylength_ALL-week.png differ diff --git a/docs/categories/index.html b/docs/categories/index.html index 107fc3d7a..c02a6922b 100644 --- a/docs/categories/index.html +++ b/docs/categories/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/index.html b/docs/categories/notes/index.html index aa4edb700..62d1ac873 100644 --- a/docs/categories/notes/index.html +++ b/docs/categories/notes/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/2/index.html b/docs/categories/notes/page/2/index.html index bf96dc8ad..155830d06 100644 --- a/docs/categories/notes/page/2/index.html +++ b/docs/categories/notes/page/2/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/3/index.html b/docs/categories/notes/page/3/index.html index 795dd6f18..2c20c5751 100644 --- a/docs/categories/notes/page/3/index.html +++ b/docs/categories/notes/page/3/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/4/index.html b/docs/categories/notes/page/4/index.html index 164b1b117..683e63c5f 100644 --- a/docs/categories/notes/page/4/index.html +++ b/docs/categories/notes/page/4/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/5/index.html b/docs/categories/notes/page/5/index.html index 4fa11784b..6a2bba58e 100644 --- a/docs/categories/notes/page/5/index.html +++ b/docs/categories/notes/page/5/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/index.html b/docs/index.html index 66a2b97da..07c203a24 100644 --- a/docs/index.html +++ b/docs/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/2/index.html b/docs/page/2/index.html index 55992e57c..72d21109b 100644 --- a/docs/page/2/index.html +++ b/docs/page/2/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/3/index.html b/docs/page/3/index.html index 391c6b952..6e103075a 100644 --- a/docs/page/3/index.html +++ b/docs/page/3/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/4/index.html b/docs/page/4/index.html index f35214fbe..a46cd85a3 100644 --- a/docs/page/4/index.html +++ b/docs/page/4/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/5/index.html b/docs/page/5/index.html index 2fc28a652..702d73494 100644 --- a/docs/page/5/index.html +++ b/docs/page/5/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/6/index.html b/docs/page/6/index.html index d7159fc06..abc53ea61 100644 --- a/docs/page/6/index.html +++ b/docs/page/6/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/7/index.html b/docs/page/7/index.html index bc03612b4..3c7671654 100644 --- a/docs/page/7/index.html +++ b/docs/page/7/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/index.html b/docs/posts/index.html index 27deae005..66cf517b4 100644 --- a/docs/posts/index.html +++ b/docs/posts/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/2/index.html b/docs/posts/page/2/index.html index 98539984f..b168e0aba 100644 --- a/docs/posts/page/2/index.html +++ b/docs/posts/page/2/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/3/index.html b/docs/posts/page/3/index.html index a5994d525..c826f7f34 100644 --- a/docs/posts/page/3/index.html +++ b/docs/posts/page/3/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/4/index.html b/docs/posts/page/4/index.html index 65b951983..d9d5ab77a 100644 --- a/docs/posts/page/4/index.html +++ b/docs/posts/page/4/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/5/index.html b/docs/posts/page/5/index.html index 8ce21adfe..49bb617fd 100644 --- a/docs/posts/page/5/index.html +++ b/docs/posts/page/5/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/6/index.html b/docs/posts/page/6/index.html index 02edebdeb..bff1e0770 100644 --- a/docs/posts/page/6/index.html +++ b/docs/posts/page/6/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/7/index.html b/docs/posts/page/7/index.html index ab8dd038f..5fe8ec1cd 100644 --- a/docs/posts/page/7/index.html +++ b/docs/posts/page/7/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/sitemap.xml b/docs/sitemap.xml index da55fb7bc..371991f13 100644 --- a/docs/sitemap.xml +++ b/docs/sitemap.xml @@ -3,19 +3,19 @@ xmlns:xhtml="http://www.w3.org/1999/xhtml"> https://alanorth.github.io/cgspace-notes/categories/ - 2021-03-06T13:35:20+02:00 + 2021-03-07T15:51:12+02:00 https://alanorth.github.io/cgspace-notes/ - 2021-03-06T13:35:20+02:00 + 2021-03-07T15:51:12+02:00 https://alanorth.github.io/cgspace-notes/2021-03/ - 2021-03-06T13:35:20+02:00 + 2021-03-07T15:51:12+02:00 https://alanorth.github.io/cgspace-notes/categories/notes/ - 2021-03-06T13:35:20+02:00 + 2021-03-07T15:51:12+02:00 https://alanorth.github.io/cgspace-notes/posts/ - 2021-03-06T13:35:20+02:00 + 2021-03-07T15:51:12+02:00 https://alanorth.github.io/cgspace-notes/2021-02/ 2021-03-04T22:46:05+02:00 diff --git a/static/2021/03/postgres_connections_ALL-week.png b/static/2021/03/postgres_connections_ALL-week.png new file mode 100644 index 000000000..3b9aa9ee9 Binary files /dev/null and b/static/2021/03/postgres_connections_ALL-week.png differ diff --git a/static/2021/03/postgres_locks_ALL-week.png b/static/2021/03/postgres_locks_ALL-week.png index 59eafded7..aa44644c4 100644 Binary files a/static/2021/03/postgres_locks_ALL-week.png and b/static/2021/03/postgres_locks_ALL-week.png differ diff --git a/static/2021/03/postgres_querylength_ALL-week.png b/static/2021/03/postgres_querylength_ALL-week.png new file mode 100644 index 000000000..5072ac336 Binary files /dev/null and b/static/2021/03/postgres_querylength_ALL-week.png differ