diff --git a/content/posts/2022-10.md b/content/posts/2022-10.md index d555be75e..8edd86adc 100644 --- a/content/posts/2022-10.md +++ b/content/posts/2022-10.md @@ -500,4 +500,34 @@ $ convert 10568-103447-dspace-step1.pdf.jpg -thumbnail 600x600 -unsharp 1.5x1+0. - I'm going to turn the REST API cache on CGSpace back on to see if the ilri.org publications thing gets broken again - Start a harvest on AReS +## 2022-10-16 + +- The harvest on AReS finished but somehow there are 1,100 less items than the previous indexing... hmmm + - I don't see any hits from MELSpace there so I will start another harvest... + - After starting the harvesting the load on the server went up to 20 and UptimeRobot said CGSpace was down for three hours, sigh + - I stopped the harvesting and the load went down immediately + - I am trying to find a pattern with the load on Sundays +- I see this in the AReS backend logs: + +```console +[Nest] 1 - 10/16/2022, 6:42:04 PM [HarvesterService] Starting Harvest =>0 +[Nest] 1 - 10/16/2022, 6:42:07 PM [HarvesterService] Starting Harvest =>101555 +[Nest] 1 - 10/16/2022, 6:42:10 PM [HarvesterService] Starting Harvest =>4936 +``` + +- Which means MELSpace is having some issue +- I'm not sure what was going on on CGSpace yesterday, but the load was indeed very high according to Munin: + +![CGSpace CPU load day](/cgspace-notes/2022/10/cpu-day.png) + +- The pattern is clear on Sundays if you look at the past month: + +![CGSpace CPU load month](/cgspace-notes/2022/10/cpu-month.png) + +- I have yet to find an increased nginx request pattern correlating with the increased load, but looking back on the last year it seems something started happening around March, 2022, and also I start seeing CPU steal in July (red coming from the top of the graph): + +![CGSpace CPU load year](/cgspace-notes/2022/10/cpu-year.png) + +- The amount of CPU steal is very low if I look at it now, around 1 or 2 percent, but what's happening now reminds me of the mysterious load problems I had in 2019-03 that were due to CPU steal + diff --git a/docs/2022-10/index.html b/docs/2022-10/index.html index c0d990cac..bfbb8ef6f 100644 --- a/docs/2022-10/index.html +++ b/docs/2022-10/index.html @@ -20,7 +20,7 @@ I filed an issue to ask about Java 11+ support - + @@ -46,9 +46,9 @@ I filed an issue to ask about Java 11+ support "@type": "BlogPosting", "headline": "October, 2022", "url": "https://alanorth.github.io/cgspace-notes/2022-10/", - "wordCount": "2090", + "wordCount": "2332", "datePublished": "2022-10-01T19:45:36+03:00", - "dateModified": "2022-10-13T07:10:59+03:00", + "dateModified": "2022-10-15T17:38:47+03:00", "author": { "@type": "Person", "name": "Alan Orth" @@ -655,6 +655,37 @@ I filed an issue to ask about Java 11+ support
  • I’m going to turn the REST API cache on CGSpace back on to see if the ilri.org publications thing gets broken again
  • Start a harvest on AReS
  • +

    2022-10-16

    + +
    [Nest] 1   - 10/16/2022, 6:42:04 PM   [HarvesterService] Starting Harvest =>0
    +[Nest] 1   - 10/16/2022, 6:42:07 PM   [HarvesterService] Starting Harvest =>101555
    +[Nest] 1   - 10/16/2022, 6:42:10 PM   [HarvesterService] Starting Harvest =>4936
    +
    +

    CGSpace CPU load day

    + +

    CGSpace CPU load month

    + +

    CGSpace CPU load year

    + diff --git a/docs/2022/10/cpu-day.png b/docs/2022/10/cpu-day.png new file mode 100644 index 000000000..d23fe7a1f Binary files /dev/null and b/docs/2022/10/cpu-day.png differ diff --git a/docs/2022/10/cpu-month.png b/docs/2022/10/cpu-month.png new file mode 100644 index 000000000..5333f6e11 Binary files /dev/null and b/docs/2022/10/cpu-month.png differ diff --git a/docs/2022/10/cpu-year.png b/docs/2022/10/cpu-year.png new file mode 100644 index 000000000..2ed1fbdab Binary files /dev/null and b/docs/2022/10/cpu-year.png differ diff --git a/docs/categories/index.html b/docs/categories/index.html index 091ef9db3..71dc98ec3 100644 --- a/docs/categories/index.html +++ b/docs/categories/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/index.html b/docs/categories/notes/index.html index 2279b44d8..0a1768422 100644 --- a/docs/categories/notes/index.html +++ b/docs/categories/notes/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/2/index.html b/docs/categories/notes/page/2/index.html index 4f55609a3..7db1fc939 100644 --- a/docs/categories/notes/page/2/index.html +++ b/docs/categories/notes/page/2/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/3/index.html b/docs/categories/notes/page/3/index.html index 55bf1da54..a3e9dc33c 100644 --- a/docs/categories/notes/page/3/index.html +++ b/docs/categories/notes/page/3/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/4/index.html b/docs/categories/notes/page/4/index.html index f87322357..8c916c854 100644 --- a/docs/categories/notes/page/4/index.html +++ b/docs/categories/notes/page/4/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/5/index.html b/docs/categories/notes/page/5/index.html index 3a9eca6c5..b34f9db28 100644 --- a/docs/categories/notes/page/5/index.html +++ b/docs/categories/notes/page/5/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/6/index.html b/docs/categories/notes/page/6/index.html index 616c7217e..139685bfb 100644 --- a/docs/categories/notes/page/6/index.html +++ b/docs/categories/notes/page/6/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/7/index.html b/docs/categories/notes/page/7/index.html index c1c493b44..ad2cc5c10 100644 --- a/docs/categories/notes/page/7/index.html +++ b/docs/categories/notes/page/7/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/index.html b/docs/index.html index 680e8e91f..4d8319bfb 100644 --- a/docs/index.html +++ b/docs/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/2/index.html b/docs/page/2/index.html index cf949b3c7..e886dec39 100644 --- a/docs/page/2/index.html +++ b/docs/page/2/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/3/index.html b/docs/page/3/index.html index bd8572935..08959e207 100644 --- a/docs/page/3/index.html +++ b/docs/page/3/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/4/index.html b/docs/page/4/index.html index fa948e272..bc6aa03fd 100644 --- a/docs/page/4/index.html +++ b/docs/page/4/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/5/index.html b/docs/page/5/index.html index 578cded54..a555bff0a 100644 --- a/docs/page/5/index.html +++ b/docs/page/5/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/6/index.html b/docs/page/6/index.html index 6dededd79..9c9b87483 100644 --- a/docs/page/6/index.html +++ b/docs/page/6/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/7/index.html b/docs/page/7/index.html index 4229691d2..93720ea75 100644 --- a/docs/page/7/index.html +++ b/docs/page/7/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/8/index.html b/docs/page/8/index.html index 4e9e3f3a5..fe81a7772 100644 --- a/docs/page/8/index.html +++ b/docs/page/8/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/9/index.html b/docs/page/9/index.html index 57e029006..8ef6b62f5 100644 --- a/docs/page/9/index.html +++ b/docs/page/9/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/index.html b/docs/posts/index.html index 40c3685a5..a60e036e7 100644 --- a/docs/posts/index.html +++ b/docs/posts/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/2/index.html b/docs/posts/page/2/index.html index 9428a0530..9bec24397 100644 --- a/docs/posts/page/2/index.html +++ b/docs/posts/page/2/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/3/index.html b/docs/posts/page/3/index.html index e401a3102..6830862b1 100644 --- a/docs/posts/page/3/index.html +++ b/docs/posts/page/3/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/4/index.html b/docs/posts/page/4/index.html index 84cdf4097..a7789d17f 100644 --- a/docs/posts/page/4/index.html +++ b/docs/posts/page/4/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/5/index.html b/docs/posts/page/5/index.html index 3032ada2e..0e7d482c6 100644 --- a/docs/posts/page/5/index.html +++ b/docs/posts/page/5/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/6/index.html b/docs/posts/page/6/index.html index d42861ebc..225e29caa 100644 --- a/docs/posts/page/6/index.html +++ b/docs/posts/page/6/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/7/index.html b/docs/posts/page/7/index.html index 400e33439..f846fc9ae 100644 --- a/docs/posts/page/7/index.html +++ b/docs/posts/page/7/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/8/index.html b/docs/posts/page/8/index.html index 49431c688..d056b173d 100644 --- a/docs/posts/page/8/index.html +++ b/docs/posts/page/8/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/9/index.html b/docs/posts/page/9/index.html index 6e229a98b..f46f0ff04 100644 --- a/docs/posts/page/9/index.html +++ b/docs/posts/page/9/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/sitemap.xml b/docs/sitemap.xml index 010d57ead..e29ee2e79 100644 --- a/docs/sitemap.xml +++ b/docs/sitemap.xml @@ -3,19 +3,19 @@ xmlns:xhtml="http://www.w3.org/1999/xhtml"> https://alanorth.github.io/cgspace-notes/categories/ - 2022-10-13T07:10:59+03:00 + 2022-10-15T17:38:47+03:00 https://alanorth.github.io/cgspace-notes/ - 2022-10-13T07:10:59+03:00 + 2022-10-15T17:38:47+03:00 https://alanorth.github.io/cgspace-notes/categories/notes/ - 2022-10-13T07:10:59+03:00 + 2022-10-15T17:38:47+03:00 https://alanorth.github.io/cgspace-notes/2022-10/ - 2022-10-13T07:10:59+03:00 + 2022-10-15T17:38:47+03:00 https://alanorth.github.io/cgspace-notes/posts/ - 2022-10-13T07:10:59+03:00 + 2022-10-15T17:38:47+03:00 https://alanorth.github.io/cgspace-notes/2022-09/ 2022-09-30T17:29:50+03:00 diff --git a/static/2022/10/cpu-day.png b/static/2022/10/cpu-day.png new file mode 100644 index 000000000..d23fe7a1f Binary files /dev/null and b/static/2022/10/cpu-day.png differ diff --git a/static/2022/10/cpu-month.png b/static/2022/10/cpu-month.png new file mode 100644 index 000000000..5333f6e11 Binary files /dev/null and b/static/2022/10/cpu-month.png differ diff --git a/static/2022/10/cpu-year.png b/static/2022/10/cpu-year.png new file mode 100644 index 000000000..2ed1fbdab Binary files /dev/null and b/static/2022/10/cpu-year.png differ