From 0f2fe01a4278a9322301f0749a451d3501852132 Mon Sep 17 00:00:00 2001 From: Alan Orth Date: Fri, 25 Jun 2021 21:32:18 +0300 Subject: [PATCH] Add notes for 2021-06-25 --- content/posts/2021-06.md | 45 ++++++++++++++++++++++ docs/2021-06/index.html | 51 +++++++++++++++++++++++-- docs/categories/index.html | 2 +- docs/categories/notes/index.html | 2 +- docs/categories/notes/page/2/index.html | 2 +- docs/categories/notes/page/3/index.html | 2 +- docs/categories/notes/page/4/index.html | 2 +- docs/categories/notes/page/5/index.html | 2 +- docs/index.html | 2 +- docs/page/2/index.html | 2 +- docs/page/3/index.html | 2 +- docs/page/4/index.html | 2 +- docs/page/5/index.html | 2 +- docs/page/6/index.html | 2 +- docs/page/7/index.html | 2 +- docs/page/8/index.html | 2 +- docs/posts/index.html | 2 +- docs/posts/page/2/index.html | 2 +- docs/posts/page/3/index.html | 2 +- docs/posts/page/4/index.html | 2 +- docs/posts/page/5/index.html | 2 +- docs/posts/page/6/index.html | 2 +- docs/posts/page/7/index.html | 2 +- docs/posts/page/8/index.html | 2 +- docs/sitemap.xml | 10 ++--- 25 files changed, 120 insertions(+), 30 deletions(-) diff --git a/content/posts/2021-06.md b/content/posts/2021-06.md index 5c84b3e8d..f2ba33f27 100644 --- a/content/posts/2021-06.md +++ b/content/posts/2021-06.md @@ -324,4 +324,49 @@ $ docker logs api 2>/dev/null | grep dspace_add_missing_items | sort | uniq | wc - Spent a few hours with Moayad troubleshooting and improving OpenRXV - We found a bug in the harvesting code that can occur when you are harvesting DSpace 5 and DSpace 6 instances, as DSpace 5 uses numeric (long) IDs, and DSpace 6 uses UUIDs +## 2021-06-25 + +- The new OpenRXV code creates almost 200,000 jobs when the plugins start + - I figured out how to use [bee-queue/arena](https://github.com/bee-queue/arena/tree/master/example) to view our Bull job queue + - Also, we can see the jobs directly using redis-cli: + +```console +$ redis-cli +127.0.0.1:6379> SCAN 0 COUNT 5 +1) "49152" +2) 1) "bull:plugins:476595" + 2) "bull:plugins:367382" + 3) "bull:plugins:369228" + 4) "bull:plugins:438986" + 5) "bull:plugins:366215" +``` + +- We can apparently get the names of the jobs in each hash using `hget`: + +```console +127.0.0.1:6379> TYPE bull:plugins:401827 +hash +127.0.0.1:6379> HGET bull:plugins:401827 name +"dspace_add_missing_items" +``` + +- I whipped up a one liner to get the keys for all plugin jobs, convert to redis `HGET` commands to extract the value of the name field, and then sort them by their counts: + +```console +$ redis-cli KEYS "bull:plugins:*" \ + | sed -e 's/^bull/HGET bull/' -e 's/\([[:digit:]]\)$/\1 name/' \ + | ncat -w 3 localhost 6379 \ + | grep -v -E '^\$' | sort | uniq -c | sort -h + 3 dspace_health_check + 4 -ERR wrong number of arguments for 'hget' command + 12 mel_downloads_and_views + 129 dspace_altmetrics + 932 dspace_downloads_and_views + 186428 dspace_add_missing_items +``` + +- Note that this uses `ncat` to send commands directly to redis all at once instead of one at a time (`netcat` didn't work here, as it doesn't know when our input is finished and never quits) + - I thought of using `redis-cli --pipe` but then you have to construct the commands in the redis protocol format with the number of args and length of each command +- There is clearly something wrong with the new DSpace health check plugin, as it creates WAY too many jobs every time we run the plugins + diff --git a/docs/2021-06/index.html b/docs/2021-06/index.html index 50d47d782..de2d35fdf 100644 --- a/docs/2021-06/index.html +++ b/docs/2021-06/index.html @@ -20,7 +20,7 @@ I simply started it and AReS was running again: - + @@ -46,9 +46,9 @@ I simply started it and AReS was running again: "@type": "BlogPosting", "headline": "June, 2021", "url": "https://alanorth.github.io/cgspace-notes/2021-06/", - "wordCount": "2396", + "wordCount": "2651", "datePublished": "2021-06-01T10:51:07+03:00", - "dateModified": "2021-06-22T15:22:15+03:00", + "dateModified": "2021-06-25T09:34:29+03:00", "author": { "@type": "Person", "name": "Alan Orth" @@ -487,6 +487,51 @@ $ grep -oE '"handle":"([[:digit:]]|\.)+/[[:digit:]]+"' cgspa +

2021-06-25

+ +
$ redis-cli
+127.0.0.1:6379> SCAN 0 COUNT 5
+1) "49152"
+2) 1) "bull:plugins:476595"
+   2) "bull:plugins:367382"
+   3) "bull:plugins:369228"
+   4) "bull:plugins:438986"
+   5) "bull:plugins:366215"
+
+
127.0.0.1:6379> TYPE bull:plugins:401827
+hash
+127.0.0.1:6379> HGET bull:plugins:401827 name
+"dspace_add_missing_items"
+
+
$ redis-cli KEYS "bull:plugins:*" \
+  | sed -e 's/^bull/HGET bull/' -e 's/\([[:digit:]]\)$/\1 name/' \
+  | ncat -w 3 localhost 6379 \
+  | grep -v -E '^\$' | sort | uniq -c | sort -h
+      3 dspace_health_check
+      4 -ERR wrong number of arguments for 'hget' command
+     12 mel_downloads_and_views
+    129 dspace_altmetrics
+    932 dspace_downloads_and_views
+ 186428 dspace_add_missing_items
+
diff --git a/docs/categories/index.html b/docs/categories/index.html index 855e30a6e..cb1942966 100644 --- a/docs/categories/index.html +++ b/docs/categories/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/index.html b/docs/categories/notes/index.html index 5672a7dfd..fb7fe4d7a 100644 --- a/docs/categories/notes/index.html +++ b/docs/categories/notes/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/2/index.html b/docs/categories/notes/page/2/index.html index 8caa3602c..696ef8c2d 100644 --- a/docs/categories/notes/page/2/index.html +++ b/docs/categories/notes/page/2/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/3/index.html b/docs/categories/notes/page/3/index.html index e606bdbee..9c5e03155 100644 --- a/docs/categories/notes/page/3/index.html +++ b/docs/categories/notes/page/3/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/4/index.html b/docs/categories/notes/page/4/index.html index 66269fbfc..314b2cc78 100644 --- a/docs/categories/notes/page/4/index.html +++ b/docs/categories/notes/page/4/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/5/index.html b/docs/categories/notes/page/5/index.html index 34bdc5d34..5ad0bf4a0 100644 --- a/docs/categories/notes/page/5/index.html +++ b/docs/categories/notes/page/5/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/index.html b/docs/index.html index 80eac58f3..d46b63d80 100644 --- a/docs/index.html +++ b/docs/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/2/index.html b/docs/page/2/index.html index e92bf2f4a..6e814e654 100644 --- a/docs/page/2/index.html +++ b/docs/page/2/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/3/index.html b/docs/page/3/index.html index 335dc0cc1..a885ed817 100644 --- a/docs/page/3/index.html +++ b/docs/page/3/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/4/index.html b/docs/page/4/index.html index ffd1edf7c..43336ba91 100644 --- a/docs/page/4/index.html +++ b/docs/page/4/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/5/index.html b/docs/page/5/index.html index 1170fa787..0663fc271 100644 --- a/docs/page/5/index.html +++ b/docs/page/5/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/6/index.html b/docs/page/6/index.html index 8cca8be5c..1cbfa98e2 100644 --- a/docs/page/6/index.html +++ b/docs/page/6/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/7/index.html b/docs/page/7/index.html index c7116b328..14251d6d5 100644 --- a/docs/page/7/index.html +++ b/docs/page/7/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/8/index.html b/docs/page/8/index.html index bcf6386d6..1fa0c8374 100644 --- a/docs/page/8/index.html +++ b/docs/page/8/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/index.html b/docs/posts/index.html index 414ae7b22..2bc37f157 100644 --- a/docs/posts/index.html +++ b/docs/posts/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/2/index.html b/docs/posts/page/2/index.html index c67142db9..4e02eb292 100644 --- a/docs/posts/page/2/index.html +++ b/docs/posts/page/2/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/3/index.html b/docs/posts/page/3/index.html index b381a0145..129693125 100644 --- a/docs/posts/page/3/index.html +++ b/docs/posts/page/3/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/4/index.html b/docs/posts/page/4/index.html index fdc89eb83..8489a09d7 100644 --- a/docs/posts/page/4/index.html +++ b/docs/posts/page/4/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/5/index.html b/docs/posts/page/5/index.html index 0f9c83448..e9a7fa48c 100644 --- a/docs/posts/page/5/index.html +++ b/docs/posts/page/5/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/6/index.html b/docs/posts/page/6/index.html index c29899106..d479918ea 100644 --- a/docs/posts/page/6/index.html +++ b/docs/posts/page/6/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/7/index.html b/docs/posts/page/7/index.html index b7a9fac64..c66429598 100644 --- a/docs/posts/page/7/index.html +++ b/docs/posts/page/7/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/8/index.html b/docs/posts/page/8/index.html index 5e6fc6722..84ddc22cf 100644 --- a/docs/posts/page/8/index.html +++ b/docs/posts/page/8/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/sitemap.xml b/docs/sitemap.xml index 8f5721606..c58100387 100644 --- a/docs/sitemap.xml +++ b/docs/sitemap.xml @@ -3,19 +3,19 @@ xmlns:xhtml="http://www.w3.org/1999/xhtml"> https://alanorth.github.io/cgspace-notes/categories/ - 2021-06-22T15:22:15+03:00 + 2021-06-25T09:34:29+03:00 https://alanorth.github.io/cgspace-notes/ - 2021-06-22T15:22:15+03:00 + 2021-06-25T09:34:29+03:00 https://alanorth.github.io/cgspace-notes/2021-06/ - 2021-06-22T15:22:15+03:00 + 2021-06-25T09:34:29+03:00 https://alanorth.github.io/cgspace-notes/categories/notes/ - 2021-06-22T15:22:15+03:00 + 2021-06-25T09:34:29+03:00 https://alanorth.github.io/cgspace-notes/posts/ - 2021-06-22T15:22:15+03:00 + 2021-06-25T09:34:29+03:00 https://alanorth.github.io/cgspace-notes/2021-05/ 2021-05-30T22:09:06+03:00