From b99114d8e4d3d76af715f020d52fab39aed4164e Mon Sep 17 00:00:00 2001 From: Alan Orth Date: Tue, 10 Nov 2020 17:00:02 +0200 Subject: [PATCH] Add notes for 2020-11-10 --- content/posts/2020-11.md | 43 ++++++++++++++++++++ docs/2020-11/index.html | 54 +++++++++++++++++++++++-- docs/categories/index.html | 2 +- docs/categories/notes/index.html | 2 +- docs/categories/notes/page/2/index.html | 2 +- docs/categories/notes/page/3/index.html | 2 +- docs/categories/notes/page/4/index.html | 2 +- docs/index.html | 2 +- docs/page/2/index.html | 2 +- docs/page/3/index.html | 2 +- docs/page/4/index.html | 2 +- docs/page/5/index.html | 2 +- docs/page/6/index.html | 2 +- docs/page/7/index.html | 2 +- docs/posts/index.html | 2 +- docs/posts/page/2/index.html | 2 +- docs/posts/page/3/index.html | 2 +- docs/posts/page/4/index.html | 2 +- docs/posts/page/5/index.html | 2 +- docs/posts/page/6/index.html | 2 +- docs/posts/page/7/index.html | 2 +- docs/sitemap.xml | 10 ++--- 22 files changed, 118 insertions(+), 27 deletions(-) diff --git a/content/posts/2020-11.md b/content/posts/2020-11.md index 117335b70..f7305f21c 100644 --- a/content/posts/2020-11.md +++ b/content/posts/2020-11.md @@ -75,5 +75,48 @@ dspace=# COMMIT; - Run system updates on CGSpace (linode18) and reboot it - I had to restart Tomcat once after the machine started up to get all Solr statistics cores to load properly +- After about ten more hours the rest of the Solr statistics cores finished processing on DSpace Test and I started optimizing them in Solr admin UI + +## 2020-11-10 + +- I am noticing that CGSpace doesn't have any statistics showing for years before 2020, but all cores are loaded successfully in Solr Admin UI... strange + - I restarted Tomcat and I see in Solr Admin UI that the statistics-2015 core failed to load + - Looking in the DSpace log I see: + +``` +2020-11-10 08:43:59,634 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2015 +2020-11-10 08:43:59,687 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2018 +2020-11-10 08:43:59,707 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2015 +2020-11-10 08:44:00,004 WARN org.dspace.core.ConfigurationManager @ Requested configuration module: atmire-datatables not found +2020-11-10 08:44:00,005 WARN org.dspace.core.ConfigurationManager @ Requested configuration module: atmire-datatables not found +2020-11-10 08:44:00,005 WARN org.dspace.core.ConfigurationManager @ Requested configuration module: atmire-datatables not found +2020-11-10 08:44:00,325 INFO org.dspace.statistics.SolrLogger @ Created core with name: statistics-2015 +``` + +- Seems that the core gets probed twice... perhaps a threading issue? + - The only thing I can think of is the `acceptorThreadCount` parameter in Tomcat's server.xml, which has been set to 2 since 2018-01 (we started sharding the Solr statistics cores in 2019-01 and that's when this problem arose) + - I will try reducing that to 1 + - Wow, now it's even worse: + +``` +2020-11-10 08:51:03,007 INFO org.dspace.statistics.SolrLogger @ Created core with name: statistics-2018 +2020-11-10 08:51:03,008 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2015 +2020-11-10 08:51:03,137 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2018 +2020-11-10 08:51:03,153 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2015 +2020-11-10 08:51:03,289 INFO org.dspace.statistics.SolrLogger @ Created core with name: statistics-2015 +2020-11-10 08:51:03,289 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2010 +2020-11-10 08:51:03,475 INFO org.dspace.statistics.SolrLogger @ Created core with name: statistics-2010 +2020-11-10 08:51:03,475 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2016 +2020-11-10 08:51:03,730 INFO org.dspace.statistics.SolrLogger @ Created core with name: statistics-2016 +2020-11-10 08:51:03,731 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2017 +2020-11-10 08:51:03,992 INFO org.dspace.statistics.SolrLogger @ Created core with name: statistics-2017 +2020-11-10 08:51:03,992 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2011 +2020-11-10 08:51:04,178 INFO org.dspace.statistics.SolrLogger @ Created core with name: statistics-2011 +2020-11-10 08:51:04,178 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2012 +``` + +- Could it be because we have two Tomcat connectors? + - I restarted Tomcat a few more times before all cores loaded, and still there are no stats before 2020-01... hmmmmm +- I added a [lowercase formatter to OpenRXV](https://github.com/ilri/OpenRXV/commit/3816b9b3f3d9182d2ba1a899c1017c5895a59dee) so that we can lowercase AGROVOC subjects during harvesting diff --git a/docs/2020-11/index.html b/docs/2020-11/index.html index 65bad2a3b..74206555b 100644 --- a/docs/2020-11/index.html +++ b/docs/2020-11/index.html @@ -17,7 +17,7 @@ So far we’ve spent at least fifty hours to process the statistics and stat - + @@ -39,9 +39,9 @@ So far we’ve spent at least fifty hours to process the statistics and stat "@type": "BlogPosting", "headline": "November, 2020", "url": "https://alanorth.github.io/cgspace-notes/2020-11/", - "wordCount": "423", + "wordCount": "817", "datePublished": "2020-11-01T13:11:54+02:00", - "dateModified": "2020-11-08T14:43:00+02:00", + "dateModified": "2020-11-08T15:03:02+02:00", "author": { "@type": "Person", "name": "Alan Orth" @@ -197,6 +197,54 @@ dspace=# COMMIT;
  • I had to restart Tomcat once after the machine started up to get all Solr statistics cores to load properly
    • +
  • After about ten more hours the rest of the Solr statistics cores finished processing on DSpace Test and I started optimizing them in Solr admin UI
    • + +

    2020-11-10

    + +
    2020-11-10 08:43:59,634 INFO  org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2015
+2020-11-10 08:43:59,687 INFO  org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2018
+2020-11-10 08:43:59,707 INFO  org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2015
+2020-11-10 08:44:00,004 WARN  org.dspace.core.ConfigurationManager @ Requested configuration module: atmire-datatables not found
+2020-11-10 08:44:00,005 WARN  org.dspace.core.ConfigurationManager @ Requested configuration module: atmire-datatables not found
+2020-11-10 08:44:00,005 WARN  org.dspace.core.ConfigurationManager @ Requested configuration module: atmire-datatables not found
+2020-11-10 08:44:00,325 INFO  org.dspace.statistics.SolrLogger @ Created core with name: statistics-2015
+
    +
    2020-11-10 08:51:03,007 INFO  org.dspace.statistics.SolrLogger @ Created core with name: statistics-2018
+2020-11-10 08:51:03,008 INFO  org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2015
+2020-11-10 08:51:03,137 INFO  org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2018
+2020-11-10 08:51:03,153 INFO  org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2015
+2020-11-10 08:51:03,289 INFO  org.dspace.statistics.SolrLogger @ Created core with name: statistics-2015
+2020-11-10 08:51:03,289 INFO  org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2010
+2020-11-10 08:51:03,475 INFO  org.dspace.statistics.SolrLogger @ Created core with name: statistics-2010
+2020-11-10 08:51:03,475 INFO  org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2016
+2020-11-10 08:51:03,730 INFO  org.dspace.statistics.SolrLogger @ Created core with name: statistics-2016
+2020-11-10 08:51:03,731 INFO  org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2017
+2020-11-10 08:51:03,992 INFO  org.dspace.statistics.SolrLogger @ Created core with name: statistics-2017
+2020-11-10 08:51:03,992 INFO  org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2011
+2020-11-10 08:51:04,178 INFO  org.dspace.statistics.SolrLogger @ Created core with name: statistics-2011
+2020-11-10 08:51:04,178 INFO  org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2012
+
    diff --git a/docs/categories/index.html b/docs/categories/index.html index 5afadef1c..466a72626 100644 --- a/docs/categories/index.html +++ b/docs/categories/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/categories/notes/index.html b/docs/categories/notes/index.html index bc85feb83..64ccf5837 100644 --- a/docs/categories/notes/index.html +++ b/docs/categories/notes/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/categories/notes/page/2/index.html b/docs/categories/notes/page/2/index.html index 83fefdda0..b7fb5b53d 100644 --- a/docs/categories/notes/page/2/index.html +++ b/docs/categories/notes/page/2/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/categories/notes/page/3/index.html b/docs/categories/notes/page/3/index.html index 0be8e76d9..d3f7cf7f7 100644 --- a/docs/categories/notes/page/3/index.html +++ b/docs/categories/notes/page/3/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/categories/notes/page/4/index.html b/docs/categories/notes/page/4/index.html index ebfab7947..f3ef37359 100644 --- a/docs/categories/notes/page/4/index.html +++ b/docs/categories/notes/page/4/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/index.html b/docs/index.html index efb040b78..aa3753b03 100644 --- a/docs/index.html +++ b/docs/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/page/2/index.html b/docs/page/2/index.html index dccfbcebe..5b73c1fd2 100644 --- a/docs/page/2/index.html +++ b/docs/page/2/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/page/3/index.html b/docs/page/3/index.html index 7c12524bb..51c3488af 100644 --- a/docs/page/3/index.html +++ b/docs/page/3/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/page/4/index.html b/docs/page/4/index.html index 174c4590f..8fa58cf87 100644 --- a/docs/page/4/index.html +++ b/docs/page/4/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/page/5/index.html b/docs/page/5/index.html index 68353fcbc..f700d26ed 100644 --- a/docs/page/5/index.html +++ b/docs/page/5/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/page/6/index.html b/docs/page/6/index.html index 6ca4edc8c..573a339f9 100644 --- a/docs/page/6/index.html +++ b/docs/page/6/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/page/7/index.html b/docs/page/7/index.html index 5cdf58d45..dcfaf493a 100644 --- a/docs/page/7/index.html +++ b/docs/page/7/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/posts/index.html b/docs/posts/index.html index 93e1d2153..f2f3037c8 100644 --- a/docs/posts/index.html +++ b/docs/posts/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/posts/page/2/index.html b/docs/posts/page/2/index.html index 58ae71908..f0d77b85c 100644 --- a/docs/posts/page/2/index.html +++ b/docs/posts/page/2/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/posts/page/3/index.html b/docs/posts/page/3/index.html index c6058b07a..332c3cab1 100644 --- a/docs/posts/page/3/index.html +++ b/docs/posts/page/3/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/posts/page/4/index.html b/docs/posts/page/4/index.html index 3ed834ba8..a0ad29bab 100644 --- a/docs/posts/page/4/index.html +++ b/docs/posts/page/4/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/posts/page/5/index.html b/docs/posts/page/5/index.html index b7efedf4e..d53a1987d 100644 --- a/docs/posts/page/5/index.html +++ b/docs/posts/page/5/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/posts/page/6/index.html b/docs/posts/page/6/index.html index ba4a0633b..2d8ad7560 100644 --- a/docs/posts/page/6/index.html +++ b/docs/posts/page/6/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/posts/page/7/index.html b/docs/posts/page/7/index.html index b507df8e1..75ff0d4f8 100644 --- a/docs/posts/page/7/index.html +++ b/docs/posts/page/7/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/sitemap.xml b/docs/sitemap.xml index e7fae30cd..d367b8a5f 100644 --- a/docs/sitemap.xml +++ b/docs/sitemap.xml @@ -4,27 +4,27 @@ https://alanorth.github.io/cgspace-notes/categories/ - 2020-11-08T14:43:00+02:00 + 2020-11-08T15:03:02+02:00 https://alanorth.github.io/cgspace-notes/ - 2020-11-08T14:43:00+02:00 + 2020-11-08T15:03:02+02:00 https://alanorth.github.io/cgspace-notes/categories/notes/ - 2020-11-08T14:43:00+02:00 + 2020-11-08T15:03:02+02:00 https://alanorth.github.io/cgspace-notes/2020-11/ - 2020-11-08T14:43:00+02:00 + 2020-11-08T15:03:02+02:00 https://alanorth.github.io/cgspace-notes/posts/ - 2020-11-08T14:43:00+02:00 + 2020-11-08T15:03:02+02:00