diff --git a/content/posts/2021-06.md b/content/posts/2021-06.md
index 5c84b3e8d..f2ba33f27 100644
--- a/content/posts/2021-06.md
+++ b/content/posts/2021-06.md
@@ -324,4 +324,49 @@ $ docker logs api 2>/dev/null | grep dspace_add_missing_items | sort | uniq | wc
- Spent a few hours with Moayad troubleshooting and improving OpenRXV
- We found a bug in the harvesting code that can occur when you are harvesting DSpace 5 and DSpace 6 instances, as DSpace 5 uses numeric (long) IDs, and DSpace 6 uses UUIDs
+## 2021-06-25
+
+- The new OpenRXV code creates almost 200,000 jobs when the plugins start
+ - I figured out how to use [bee-queue/arena](https://github.com/bee-queue/arena/tree/master/example) to view our Bull job queue
+ - Also, we can see the jobs directly using redis-cli:
+
+```console
+$ redis-cli
+127.0.0.1:6379> SCAN 0 COUNT 5
+1) "49152"
+2) 1) "bull:plugins:476595"
+ 2) "bull:plugins:367382"
+ 3) "bull:plugins:369228"
+ 4) "bull:plugins:438986"
+ 5) "bull:plugins:366215"
+```
+
+- We can apparently get the names of the jobs in each hash using `hget`:
+
+```console
+127.0.0.1:6379> TYPE bull:plugins:401827
+hash
+127.0.0.1:6379> HGET bull:plugins:401827 name
+"dspace_add_missing_items"
+```
+
+- I whipped up a one liner to get the keys for all plugin jobs, convert to redis `HGET` commands to extract the value of the name field, and then sort them by their counts:
+
+```console
+$ redis-cli KEYS "bull:plugins:*" \
+ | sed -e 's/^bull/HGET bull/' -e 's/\([[:digit:]]\)$/\1 name/' \
+ | ncat -w 3 localhost 6379 \
+ | grep -v -E '^\$' | sort | uniq -c | sort -h
+ 3 dspace_health_check
+ 4 -ERR wrong number of arguments for 'hget' command
+ 12 mel_downloads_and_views
+ 129 dspace_altmetrics
+ 932 dspace_downloads_and_views
+ 186428 dspace_add_missing_items
+```
+
+- Note that this uses `ncat` to send commands directly to redis all at once instead of one at a time (`netcat` didn't work here, as it doesn't know when our input is finished and never quits)
+ - I thought of using `redis-cli --pipe` but then you have to construct the commands in the redis protocol format with the number of args and length of each command
+- There is clearly something wrong with the new DSpace health check plugin, as it creates WAY too many jobs every time we run the plugins
+
diff --git a/docs/2021-06/index.html b/docs/2021-06/index.html
index 50d47d782..de2d35fdf 100644
--- a/docs/2021-06/index.html
+++ b/docs/2021-06/index.html
@@ -20,7 +20,7 @@ I simply started it and AReS was running again:
-
+
@@ -46,9 +46,9 @@ I simply started it and AReS was running again:
"@type": "BlogPosting",
"headline": "June, 2021",
"url": "https://alanorth.github.io/cgspace-notes/2021-06/",
- "wordCount": "2396",
+ "wordCount": "2651",
"datePublished": "2021-06-01T10:51:07+03:00",
- "dateModified": "2021-06-22T15:22:15+03:00",
+ "dateModified": "2021-06-25T09:34:29+03:00",
"author": {
"@type": "Person",
"name": "Alan Orth"
@@ -487,6 +487,51 @@ $ grep -oE '"handle":"([[:digit:]]|\.)+/[[:digit:]]+"' cgspa
+
2021-06-25
+
+- The new OpenRXV code creates almost 200,000 jobs when the plugins start
+
+- I figured out how to use bee-queue/arena to view our Bull job queue
+- Also, we can see the jobs directly using redis-cli:
+
+
+
+$ redis-cli
+127.0.0.1:6379> SCAN 0 COUNT 5
+1) "49152"
+2) 1) "bull:plugins:476595"
+ 2) "bull:plugins:367382"
+ 3) "bull:plugins:369228"
+ 4) "bull:plugins:438986"
+ 5) "bull:plugins:366215"
+
+- We can apparently get the names of the jobs in each hash using
hget
:
+
+127.0.0.1:6379> TYPE bull:plugins:401827
+hash
+127.0.0.1:6379> HGET bull:plugins:401827 name
+"dspace_add_missing_items"
+
+- I whipped up a one liner to get the keys for all plugin jobs, convert to redis
HGET
commands to extract the value of the name field, and then sort them by their counts:
+
+$ redis-cli KEYS "bull:plugins:*" \
+ | sed -e 's/^bull/HGET bull/' -e 's/\([[:digit:]]\)$/\1 name/' \
+ | ncat -w 3 localhost 6379 \
+ | grep -v -E '^\$' | sort | uniq -c | sort -h
+ 3 dspace_health_check
+ 4 -ERR wrong number of arguments for 'hget' command
+ 12 mel_downloads_and_views
+ 129 dspace_altmetrics
+ 932 dspace_downloads_and_views
+ 186428 dspace_add_missing_items
+
+- Note that this uses
ncat
to send commands directly to redis all at once instead of one at a time (netcat
didn’t work here, as it doesn’t know when our input is finished and never quits)
+
+- I thought of using
redis-cli --pipe
but then you have to construct the commands in the redis protocol format with the number of args and length of each command
+
+
+- There is clearly something wrong with the new DSpace health check plugin, as it creates WAY too many jobs every time we run the plugins
+
diff --git a/docs/categories/index.html b/docs/categories/index.html
index 855e30a6e..cb1942966 100644
--- a/docs/categories/index.html
+++ b/docs/categories/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/categories/notes/index.html b/docs/categories/notes/index.html
index 5672a7dfd..fb7fe4d7a 100644
--- a/docs/categories/notes/index.html
+++ b/docs/categories/notes/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/categories/notes/page/2/index.html b/docs/categories/notes/page/2/index.html
index 8caa3602c..696ef8c2d 100644
--- a/docs/categories/notes/page/2/index.html
+++ b/docs/categories/notes/page/2/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/categories/notes/page/3/index.html b/docs/categories/notes/page/3/index.html
index e606bdbee..9c5e03155 100644
--- a/docs/categories/notes/page/3/index.html
+++ b/docs/categories/notes/page/3/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/categories/notes/page/4/index.html b/docs/categories/notes/page/4/index.html
index 66269fbfc..314b2cc78 100644
--- a/docs/categories/notes/page/4/index.html
+++ b/docs/categories/notes/page/4/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/categories/notes/page/5/index.html b/docs/categories/notes/page/5/index.html
index 34bdc5d34..5ad0bf4a0 100644
--- a/docs/categories/notes/page/5/index.html
+++ b/docs/categories/notes/page/5/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/index.html b/docs/index.html
index 80eac58f3..d46b63d80 100644
--- a/docs/index.html
+++ b/docs/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/page/2/index.html b/docs/page/2/index.html
index e92bf2f4a..6e814e654 100644
--- a/docs/page/2/index.html
+++ b/docs/page/2/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/page/3/index.html b/docs/page/3/index.html
index 335dc0cc1..a885ed817 100644
--- a/docs/page/3/index.html
+++ b/docs/page/3/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/page/4/index.html b/docs/page/4/index.html
index ffd1edf7c..43336ba91 100644
--- a/docs/page/4/index.html
+++ b/docs/page/4/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/page/5/index.html b/docs/page/5/index.html
index 1170fa787..0663fc271 100644
--- a/docs/page/5/index.html
+++ b/docs/page/5/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/page/6/index.html b/docs/page/6/index.html
index 8cca8be5c..1cbfa98e2 100644
--- a/docs/page/6/index.html
+++ b/docs/page/6/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/page/7/index.html b/docs/page/7/index.html
index c7116b328..14251d6d5 100644
--- a/docs/page/7/index.html
+++ b/docs/page/7/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/page/8/index.html b/docs/page/8/index.html
index bcf6386d6..1fa0c8374 100644
--- a/docs/page/8/index.html
+++ b/docs/page/8/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/index.html b/docs/posts/index.html
index 414ae7b22..2bc37f157 100644
--- a/docs/posts/index.html
+++ b/docs/posts/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/page/2/index.html b/docs/posts/page/2/index.html
index c67142db9..4e02eb292 100644
--- a/docs/posts/page/2/index.html
+++ b/docs/posts/page/2/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/page/3/index.html b/docs/posts/page/3/index.html
index b381a0145..129693125 100644
--- a/docs/posts/page/3/index.html
+++ b/docs/posts/page/3/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/page/4/index.html b/docs/posts/page/4/index.html
index fdc89eb83..8489a09d7 100644
--- a/docs/posts/page/4/index.html
+++ b/docs/posts/page/4/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/page/5/index.html b/docs/posts/page/5/index.html
index 0f9c83448..e9a7fa48c 100644
--- a/docs/posts/page/5/index.html
+++ b/docs/posts/page/5/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/page/6/index.html b/docs/posts/page/6/index.html
index c29899106..d479918ea 100644
--- a/docs/posts/page/6/index.html
+++ b/docs/posts/page/6/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/page/7/index.html b/docs/posts/page/7/index.html
index b7a9fac64..c66429598 100644
--- a/docs/posts/page/7/index.html
+++ b/docs/posts/page/7/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/page/8/index.html b/docs/posts/page/8/index.html
index 5e6fc6722..84ddc22cf 100644
--- a/docs/posts/page/8/index.html
+++ b/docs/posts/page/8/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/sitemap.xml b/docs/sitemap.xml
index 8f5721606..c58100387 100644
--- a/docs/sitemap.xml
+++ b/docs/sitemap.xml
@@ -3,19 +3,19 @@
xmlns:xhtml="http://www.w3.org/1999/xhtml">
https://alanorth.github.io/cgspace-notes/categories/
- 2021-06-22T15:22:15+03:00
+ 2021-06-25T09:34:29+03:00
https://alanorth.github.io/cgspace-notes/
- 2021-06-22T15:22:15+03:00
+ 2021-06-25T09:34:29+03:00
https://alanorth.github.io/cgspace-notes/2021-06/
- 2021-06-22T15:22:15+03:00
+ 2021-06-25T09:34:29+03:00
https://alanorth.github.io/cgspace-notes/categories/notes/
- 2021-06-22T15:22:15+03:00
+ 2021-06-25T09:34:29+03:00
https://alanorth.github.io/cgspace-notes/posts/
- 2021-06-22T15:22:15+03:00
+ 2021-06-25T09:34:29+03:00
https://alanorth.github.io/cgspace-notes/2021-05/
2021-05-30T22:09:06+03:00