<li>Continue with processing the statistics-2019 Solr core with the AtomicStatisticsUpdateCLI tool on DSpace Test
<ul>
<li>So far we’ve spent at least fifty hours to process the statistics and statistics-2019 core… wow.</li>
</ul>
</li>
</ul>
<h2id="2020-11-02">2020-11-02</h2>
<ul>
<li>Talk to Moayad and fix a few issues on OpenRXV:
<ul>
<li>Incorrect views and downloads (caused by Elasticsearch’s default result set size of 10)</li>
<li>Invalid share link</li>
<li>Missing “https://” for Handles in the Simple Excel report (caused by using the <code>handle</code> instead of the <code>uri</code>)</li>
<li>Sorting the list of items by views</li>
</ul>
</li>
<li>I resumed the processing of the statistics-2018 Solr core after it spent 20 hours to get to 60%</li>
</ul>
<h2id="2020-11-04">2020-11-04</h2>
<ul>
<li>After 29 hours the statistics-2017 core finished processing so I started the statistics-2016 core on DSpace Test</li>
<li>The statistics-2014 core finished processing after five hours, so I started processing the statistics-2013 core on DSpace Test</li>
<li>Since I was going to restart CGSpace and update the Discovery indexes anyways I decided to check for any straggling upper case AGROVOC entries and lower case them:</li>
</ul>
<pre><code>dspace=# BEGIN;
dspace=# UPDATE metadatavalue SET text_value=LOWER(text_value) WHERE resource_type_id=2 AND metadata_field_id=57 AND text_value ~ '[[:upper:]]';
UPDATE 164
dspace=# COMMIT;
</code></pre><ul>
<li>Run system updates on CGSpace (linode18) and reboot it
<ul>
<li>I had to restart Tomcat once after the machine started up to get all Solr statistics cores to load properly</li>
<li>After about ten more hours the rest of the Solr statistics cores finished processing on DSpace Test and I started optimizing them in Solr admin UI</li>
</ul>
<h2id="2020-11-10">2020-11-10</h2>
<ul>
<li>I am noticing that CGSpace doesn’t have any statistics showing for years before 2020, but all cores are loaded successfully in Solr Admin UI… strange
<ul>
<li>I restarted Tomcat and I see in Solr Admin UI that the statistics-2015 core failed to load</li>
<li>Looking in the DSpace log I see:</li>
</ul>
</li>
</ul>
<pre><code>2020-11-10 08:43:59,634 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2015
2020-11-10 08:43:59,687 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2018
2020-11-10 08:43:59,707 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2015
2020-11-10 08:44:00,004 WARN org.dspace.core.ConfigurationManager @ Requested configuration module: atmire-datatables not found
2020-11-10 08:44:00,005 WARN org.dspace.core.ConfigurationManager @ Requested configuration module: atmire-datatables not found
2020-11-10 08:44:00,005 WARN org.dspace.core.ConfigurationManager @ Requested configuration module: atmire-datatables not found
2020-11-10 08:44:00,325 INFO org.dspace.statistics.SolrLogger @ Created core with name: statistics-2015
</code></pre><ul>
<li>Seems that the core gets probed twice… perhaps a threading issue?
<ul>
<li>The only thing I can think of is the <code>acceptorThreadCount</code> parameter in Tomcat’s server.xml, which has been set to 2 since 2018-01 (we started sharding the Solr statistics cores in 2019-01 and that’s when this problem arose)</li>
<li>I will try reducing that to 1</li>
<li>Wow, now it’s even worse:</li>
</ul>
</li>
</ul>
<pre><code>2020-11-10 08:51:03,007 INFO org.dspace.statistics.SolrLogger @ Created core with name: statistics-2018
2020-11-10 08:51:03,008 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2015
2020-11-10 08:51:03,137 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2018
2020-11-10 08:51:03,153 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2015
2020-11-10 08:51:03,289 INFO org.dspace.statistics.SolrLogger @ Created core with name: statistics-2015
2020-11-10 08:51:03,289 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2010
2020-11-10 08:51:03,475 INFO org.dspace.statistics.SolrLogger @ Created core with name: statistics-2010
2020-11-10 08:51:03,475 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2016
2020-11-10 08:51:03,730 INFO org.dspace.statistics.SolrLogger @ Created core with name: statistics-2016
2020-11-10 08:51:03,731 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2017
2020-11-10 08:51:03,992 INFO org.dspace.statistics.SolrLogger @ Created core with name: statistics-2017
2020-11-10 08:51:03,992 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2011
2020-11-10 08:51:04,178 INFO org.dspace.statistics.SolrLogger @ Created core with name: statistics-2011
2020-11-10 08:51:04,178 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2012
</code></pre><ul>
<li>Could it be because we have two Tomcat connectors?
<ul>
<li>I restarted Tomcat a few more times before all cores loaded, and still there are no stats before 2020-01… hmmmmm</li>
</ul>
</li>
<li>I added a <ahref="https://github.com/ilri/OpenRXV/commit/3816b9b3f3d9182d2ba1a899c1017c5895a59dee">lowercase formatter to OpenRXV</a> so that we can lowercase AGROVOC subjects during harvesting</li>