diff --git a/content/posts/2024-10.md b/content/posts/2024-10.md
index b960c4371..aa2a30d09 100644
--- a/content/posts/2024-10.md
+++ b/content/posts/2024-10.md
@@ -21,4 +21,31 @@ $ csvcut -c 'id,dc.title[en_US],dcterms.abstract[en_US],cg.identifier.doi[en_US]
- Then wrote a script to get them from OpenAlex
- After inspecting and cleaning a few dozen up in OpenRefine (removing "Keywords:" and copyright, and HTML entities, etc) I managed to get about 440
+## 2024-10-06
+
+- Since I increase Solr's heap from 2 to 3G a few weeks ago it seems like Solr is always using 100% CPU
+ - I don't understand this because it was running well before, and I only increased it in anticipation of running the dspace-statistics-api-js, though never got around to it
+ - I just realized that this may be related to the JMX monitoring, as I've seen gaps in the Grafana dashboards and remember that it took surprisingly long to scrape the metrics
+ - Maybe I need to change the scrape interval
+
+## 2024-10-08
+
+- I checked the VictoriaMetrics vmagent dashboard and saw that there were thousands of errors scraping the `jvm_solr` target from Solr
+ - So it seems like I do need to change the scrape interval
+ - I will increase it from 15s (global) to 20s for that job
+ - Reading some documentation I found [this reference from Brian Brazil that discusses this very problem](https://www.robustperception.io/keep-it-simple-scrape_interval-id/)
+ - He recommends keeping a single scrape interval for all targets, but also checking the slow exporter (`jmx_exporter` in this case) and seeing if we can limit the data we scrape
+ - To keep things simple for now I will increase the global scrape interval to 20s
+ - Long term I should limit the metrics...
+ - Oh wow, I found out that [Solr ships with a Prometheus exporter!](https://solr.apache.org/guide/8_11/monitoring-solr-with-prometheus-and-grafana.html) and even includes a Grafana dashboard
+- I'm trying to run the Solr prometheus-exporter as a one-off systemd unit to test it:
+
+```console
+# cd /opt/solr-8.11.3/contrib/prometheus-exporter
+# systemd-run --uid=victoriametrics --gid=victoriametrics --working-directory=/opt/solr-8.11.3/contrib/prometheus-exporter ./bin/solr-exporter -p 9854 -b http://localhost:8983/solr -f ./conf/solr-exporter-config.xml -s 20
+```
+
+- The default scrape interval is 60 seconds, so if we scrape it more than that the metrics will be stale
+ - From what I've seen this returns in less than one second so it should be safe to reduce the scrape interval
+
diff --git a/docs/2024-10/index.html b/docs/2024-10/index.html
index 0fc23d911..8122d840d 100644
--- a/docs/2024-10/index.html
+++ b/docs/2024-10/index.html
@@ -18,7 +18,7 @@ For copyright reasons they don’t include plain abstracts, but the pyalex l
-
+
@@ -42,9 +42,9 @@ For copyright reasons they don’t include plain abstracts, but the pyalex l
"@type": "BlogPosting",
"headline": "October, 2024",
"url": "https://alanorth.github.io/cgspace-notes/2024-10/",
- "wordCount": "125",
+ "wordCount": "417",
"datePublished": "2024-10-03T11:01:00+03:00",
- "dateModified": "2024-10-03T11:01:00+03:00",
+ "dateModified": "2024-10-03T11:51:44+03:00",
"author": {
"@type": "Person",
"name": "Alan Orth"
@@ -137,6 +137,40 @@ For copyright reasons they don’t include plain abstracts, but the pyalex l
+
2024-10-06
+
+- Since I increase Solr’s heap from 2 to 3G a few weeks ago it seems like Solr is always using 100% CPU
+
+- I don’t understand this because it was running well before, and I only increased it in anticipation of running the dspace-statistics-api-js, though never got around to it
+- I just realized that this may be related to the JMX monitoring, as I’ve seen gaps in the Grafana dashboards and remember that it took surprisingly long to scrape the metrics
+- Maybe I need to change the scrape interval
+
+
+
+2024-10-08
+
+- I checked the VictoriaMetrics vmagent dashboard and saw that there were thousands of errors scraping the
jvm_solr
target from Solr
+
+- So it seems like I do need to change the scrape interval
+- I will increase it from 15s (global) to 20s for that job
+- Reading some documentation I found this reference from Brian Brazil that discusses this very problem
+- He recommends keeping a single scrape interval for all targets, but also checking the slow exporter (
jmx_exporter
in this case) and seeing if we can limit the data we scrape
+- To keep things simple for now I will increase the global scrape interval to 20s
+- Long term I should limit the metrics…
+- Oh wow, I found out that Solr ships with a Prometheus exporter! and even includes a Grafana dashboard
+
+
+- I’m trying to run the Solr prometheus-exporter as a one-off systemd unit to test it:
+
+# cd /opt/solr-8.11.3/contrib/prometheus-exporter
+# systemd-run --uid=victoriametrics --gid=victoriametrics --working-directory=/opt/solr-8.11.3/contrib/prometheus-exporter ./bin/solr-exporter -p 9854 -b http://localhost:8983/solr -f ./conf/solr-exporter-config.xml -s 20
+
+- The default scrape interval is 60 seconds, so if we scrape it more than that the metrics will be stale
+
+- From what I’ve seen this returns in less than one second so it should be safe to reduce the scrape interval
+
+
+
diff --git a/docs/categories/index.html b/docs/categories/index.html
index fbb4e131b..e6f687014 100644
--- a/docs/categories/index.html
+++ b/docs/categories/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/categories/index.xml b/docs/categories/index.xml
index 4ad78df13..01ddea953 100644
--- a/docs/categories/index.xml
+++ b/docs/categories/index.xml
@@ -6,7 +6,7 @@
Recent content in Categories on CGSpace Notes
Hugo
en-us
- Thu, 03 Oct 2024 11:01:00 +0300
+ Thu, 03 Oct 2024 11:51:44 +0300
-
Notes
diff --git a/docs/categories/notes/index.html b/docs/categories/notes/index.html
index d5bb45aac..509ccba3a 100644
--- a/docs/categories/notes/index.html
+++ b/docs/categories/notes/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/categories/notes/index.xml b/docs/categories/notes/index.xml
index 09f25f0db..2b98e1e16 100644
--- a/docs/categories/notes/index.xml
+++ b/docs/categories/notes/index.xml
@@ -6,7 +6,7 @@
Recent content in Notes on CGSpace Notes
Hugo
en-us
- Thu, 03 Oct 2024 11:01:00 +0300
+ Thu, 03 Oct 2024 11:51:44 +0300
-
October, 2024
diff --git a/docs/categories/notes/page/2/index.html b/docs/categories/notes/page/2/index.html
index d37efeb94..99afe2b4b 100644
--- a/docs/categories/notes/page/2/index.html
+++ b/docs/categories/notes/page/2/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/categories/notes/page/3/index.html b/docs/categories/notes/page/3/index.html
index b397a883d..d8fdedbf1 100644
--- a/docs/categories/notes/page/3/index.html
+++ b/docs/categories/notes/page/3/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/categories/notes/page/4/index.html b/docs/categories/notes/page/4/index.html
index b3d87ef09..3a2c2e843 100644
--- a/docs/categories/notes/page/4/index.html
+++ b/docs/categories/notes/page/4/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/categories/notes/page/5/index.html b/docs/categories/notes/page/5/index.html
index 49257e9c8..396081cc1 100644
--- a/docs/categories/notes/page/5/index.html
+++ b/docs/categories/notes/page/5/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/categories/notes/page/6/index.html b/docs/categories/notes/page/6/index.html
index dd7262b64..243a8746f 100644
--- a/docs/categories/notes/page/6/index.html
+++ b/docs/categories/notes/page/6/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/categories/notes/page/7/index.html b/docs/categories/notes/page/7/index.html
index a5bcfd75d..d1a5c4a64 100644
--- a/docs/categories/notes/page/7/index.html
+++ b/docs/categories/notes/page/7/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/categories/notes/page/8/index.html b/docs/categories/notes/page/8/index.html
index b7570e4ec..9bfdc9506 100644
--- a/docs/categories/notes/page/8/index.html
+++ b/docs/categories/notes/page/8/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/categories/notes/page/9/index.html b/docs/categories/notes/page/9/index.html
index ea727439a..c15d5a185 100644
--- a/docs/categories/notes/page/9/index.html
+++ b/docs/categories/notes/page/9/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/index.html b/docs/index.html
index 09bc0a54d..8842c798e 100644
--- a/docs/index.html
+++ b/docs/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/index.xml b/docs/index.xml
index d8e398289..6c2397263 100644
--- a/docs/index.xml
+++ b/docs/index.xml
@@ -6,7 +6,7 @@
Recent content on CGSpace Notes
Hugo
en-us
- Thu, 03 Oct 2024 11:01:00 +0300
+ Thu, 03 Oct 2024 11:51:44 +0300
-
October, 2024
diff --git a/docs/page/10/index.html b/docs/page/10/index.html
index 2809ffffb..2e52067b9 100644
--- a/docs/page/10/index.html
+++ b/docs/page/10/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/page/11/index.html b/docs/page/11/index.html
index 1c4190658..6f44f80b1 100644
--- a/docs/page/11/index.html
+++ b/docs/page/11/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/page/12/index.html b/docs/page/12/index.html
index 4c57f043a..049853d55 100644
--- a/docs/page/12/index.html
+++ b/docs/page/12/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/page/2/index.html b/docs/page/2/index.html
index f4ef639da..3caa4b17e 100644
--- a/docs/page/2/index.html
+++ b/docs/page/2/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/page/3/index.html b/docs/page/3/index.html
index 2744ac2f2..57a354d38 100644
--- a/docs/page/3/index.html
+++ b/docs/page/3/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/page/4/index.html b/docs/page/4/index.html
index 397627e6e..f1cf0364c 100644
--- a/docs/page/4/index.html
+++ b/docs/page/4/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/page/5/index.html b/docs/page/5/index.html
index a466aa5eb..cdaabc5c4 100644
--- a/docs/page/5/index.html
+++ b/docs/page/5/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/page/6/index.html b/docs/page/6/index.html
index b4e075819..c1fe79949 100644
--- a/docs/page/6/index.html
+++ b/docs/page/6/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/page/7/index.html b/docs/page/7/index.html
index 49e0bb622..506ecaf1b 100644
--- a/docs/page/7/index.html
+++ b/docs/page/7/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/page/8/index.html b/docs/page/8/index.html
index 9a35941f3..3e09a4031 100644
--- a/docs/page/8/index.html
+++ b/docs/page/8/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/page/9/index.html b/docs/page/9/index.html
index 1d7544b16..07be4326e 100644
--- a/docs/page/9/index.html
+++ b/docs/page/9/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/index.html b/docs/posts/index.html
index 0fd368762..039387b1c 100644
--- a/docs/posts/index.html
+++ b/docs/posts/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/index.xml b/docs/posts/index.xml
index 3df481b1b..7193e882b 100644
--- a/docs/posts/index.xml
+++ b/docs/posts/index.xml
@@ -6,7 +6,7 @@
Recent content in Posts on CGSpace Notes
Hugo
en-us
- Thu, 03 Oct 2024 11:01:00 +0300
+ Thu, 03 Oct 2024 11:51:44 +0300
-
October, 2024
diff --git a/docs/posts/page/10/index.html b/docs/posts/page/10/index.html
index 94a936716..b1c0dc6f2 100644
--- a/docs/posts/page/10/index.html
+++ b/docs/posts/page/10/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/page/11/index.html b/docs/posts/page/11/index.html
index 9c56c8649..9416ea9c5 100644
--- a/docs/posts/page/11/index.html
+++ b/docs/posts/page/11/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/page/12/index.html b/docs/posts/page/12/index.html
index 40452847d..8ebe366bb 100644
--- a/docs/posts/page/12/index.html
+++ b/docs/posts/page/12/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/page/2/index.html b/docs/posts/page/2/index.html
index fea6f6dce..1ec18649c 100644
--- a/docs/posts/page/2/index.html
+++ b/docs/posts/page/2/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/page/3/index.html b/docs/posts/page/3/index.html
index 375d647eb..a5adee8b9 100644
--- a/docs/posts/page/3/index.html
+++ b/docs/posts/page/3/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/page/4/index.html b/docs/posts/page/4/index.html
index 4cedf451b..ecd0d7ade 100644
--- a/docs/posts/page/4/index.html
+++ b/docs/posts/page/4/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/page/5/index.html b/docs/posts/page/5/index.html
index 8129ac80e..cfa3df99a 100644
--- a/docs/posts/page/5/index.html
+++ b/docs/posts/page/5/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/page/6/index.html b/docs/posts/page/6/index.html
index 329e5a161..d70fba6e6 100644
--- a/docs/posts/page/6/index.html
+++ b/docs/posts/page/6/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/page/7/index.html b/docs/posts/page/7/index.html
index c73034d3a..d0ae793d3 100644
--- a/docs/posts/page/7/index.html
+++ b/docs/posts/page/7/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/page/8/index.html b/docs/posts/page/8/index.html
index c3a9b7d01..8c4ebab02 100644
--- a/docs/posts/page/8/index.html
+++ b/docs/posts/page/8/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/page/9/index.html b/docs/posts/page/9/index.html
index d85b8a213..b67ce1fce 100644
--- a/docs/posts/page/9/index.html
+++ b/docs/posts/page/9/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/sitemap.xml b/docs/sitemap.xml
index 766993400..dca806ab2 100644
--- a/docs/sitemap.xml
+++ b/docs/sitemap.xml
@@ -3,19 +3,19 @@
xmlns:xhtml="http://www.w3.org/1999/xhtml">
https://alanorth.github.io/cgspace-notes/categories/
- 2024-10-03T11:01:00+03:00
+ 2024-10-03T11:51:44+03:00
https://alanorth.github.io/cgspace-notes/
- 2024-10-03T11:01:00+03:00
+ 2024-10-03T11:51:44+03:00
https://alanorth.github.io/cgspace-notes/categories/notes/
- 2024-10-03T11:01:00+03:00
+ 2024-10-03T11:51:44+03:00
https://alanorth.github.io/cgspace-notes/2024-10/
- 2024-10-03T11:01:00+03:00
+ 2024-10-03T11:51:44+03:00
https://alanorth.github.io/cgspace-notes/posts/
- 2024-10-03T11:01:00+03:00
+ 2024-10-03T11:51:44+03:00
https://alanorth.github.io/cgspace-notes/2024-09/
2024-09-30T07:56:53+03:00