<!DOCTYPE html> <html lang="en" > <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> <meta property="og:title" content="November, 2020" /> <meta property="og:description" content="2020-11-01 Continue with processing the statistics-2019 Solr core with the AtomicStatisticsUpdateCLI tool on DSpace Test So far we’ve spent at least fifty hours to process the statistics and statistics-2019 core… wow. " /> <meta property="og:type" content="article" /> <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2020-11/" /> <meta property="article:published_time" content="2020-11-01T13:11:54+02:00" /> <meta property="article:modified_time" content="2020-11-10T17:00:02+02:00" /> <meta name="twitter:card" content="summary"/> <meta name="twitter:title" content="November, 2020"/> <meta name="twitter:description" content="2020-11-01 Continue with processing the statistics-2019 Solr core with the AtomicStatisticsUpdateCLI tool on DSpace Test So far we’ve spent at least fifty hours to process the statistics and statistics-2019 core… wow. "/> <meta name="generator" content="Hugo 0.78.1" /> <script type="application/ld+json"> { "@context": "http://schema.org", "@type": "BlogPosting", "headline": "November, 2020", "url": "https://alanorth.github.io/cgspace-notes/2020-11/", "wordCount": "1266", "datePublished": "2020-11-01T13:11:54+02:00", "dateModified": "2020-11-10T17:00:02+02:00", "author": { "@type": "Person", "name": "Alan Orth" }, "keywords": "Notes" } </script> <link rel="canonical" href="https://alanorth.github.io/cgspace-notes/2020-11/"> <title>November, 2020 | CGSpace Notes</title> <!-- combined, minified CSS --> <link href="https://alanorth.github.io/cgspace-notes/css/style.6da5c906cc7a8fbb93f31cd2316c5dbe3f19ac4aa6bfb066f1243045b8f6061e.css" rel="stylesheet" integrity="sha256-baXJBsx6j7uT8xzSMWxdvj8ZrEqmv7Bm8SQwRbj2Bh4=" crossorigin="anonymous"> <!-- minified Font Awesome for SVG icons --> <script defer src="https://alanorth.github.io/cgspace-notes/js/fontawesome.min.f3d2a1f5980bab30ddd0d8cadbd496475309fc48e2b1d052c5c09e6facffcb0f.js" integrity="sha256-89Kh9ZgLqzDd0NjK29SWR1MJ/EjisdBSxcCeb6z/yw8=" crossorigin="anonymous"></script> <!-- RSS 2.0 feed --> </head> <body> <div class="blog-masthead"> <div class="container"> <nav class="nav blog-nav"> <a class="nav-link " href="https://alanorth.github.io/cgspace-notes/">Home</a> </nav> </div> </div> <header class="blog-header"> <div class="container"> <h1 class="blog-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/" rel="home">CGSpace Notes</a></h1> <p class="lead blog-description" dir="auto">Documenting day-to-day work on the <a href="https://cgspace.cgiar.org">CGSpace</a> repository.</p> </div> </header> <div class="container"> <div class="row"> <div class="col-sm-8 blog-main"> <article class="blog-post"> <header> <h2 class="blog-post-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/2020-11/">November, 2020</a></h2> <p class="blog-post-meta"><time datetime="2020-11-01T13:11:54+02:00">Sun Nov 01, 2020</time> by Alan Orth in <span class="fas fa-folder" aria-hidden="true"></span> <a href="/cgspace-notes/categories/notes/" rel="category tag">Notes</a> </p> </header> <h2 id="2020-11-01">2020-11-01</h2> <ul> <li>Continue with processing the statistics-2019 Solr core with the AtomicStatisticsUpdateCLI tool on DSpace Test <ul> <li>So far we’ve spent at least fifty hours to process the statistics and statistics-2019 core… wow.</li> </ul> </li> </ul> <h2 id="2020-11-02">2020-11-02</h2> <ul> <li>Talk to Moayad and fix a few issues on OpenRXV: <ul> <li>Incorrect views and downloads (caused by Elasticsearch’s default result set size of 10)</li> <li>Invalid share link</li> <li>Missing “https://” for Handles in the Simple Excel report (caused by using the <code>handle</code> instead of the <code>uri</code>)</li> <li>Sorting the list of items by views</li> </ul> </li> <li>I resumed the processing of the statistics-2018 Solr core after it spent 20 hours to get to 60%</li> </ul> <h2 id="2020-11-04">2020-11-04</h2> <ul> <li>After 29 hours the statistics-2017 core finished processing so I started the statistics-2016 core on DSpace Test</li> </ul> <h2 id="2020-11-05">2020-11-05</h2> <ul> <li>Peter sent me corrections and deletions for the author affiliations <ul> <li>I quickly proofed them for UTF-8 issues in OpenRefine and csv-metadata-quality and then tested them locally and then applied them on CGSpace:</li> </ul> </li> </ul> <pre><code>$ ./fix-metadata-values.py -i 2020-11-05-fix-862-affiliations.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.affiliation -t 'correct' -m 211 $ ./delete-metadata-values.py -i 2020-11-05-delete-29-affiliations.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.affiliation -m 211 </code></pre><ul> <li>Then I started a Discovery re-index on CGSpace:</li> </ul> <pre><code>$ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b real 92m24.993s user 8m11.858s sys 2m26.931s </code></pre><h2 id="2020-11-06">2020-11-06</h2> <ul> <li>Restart the AtomicStatisticsUpdateCLI processing of the statistics-2016 core on DSpace Test after 20 hours… <ul> <li>This phase finished after five hours so I started it on the statistics-2015 core</li> </ul> </li> </ul> <h2 id="2020-11-07">2020-11-07</h2> <ul> <li>Atmire responded about the issue with duplicate values in owningComm and containerCommunity etc <ul> <li>I told them to please look into it and use some of our credits if need be</li> </ul> </li> <li>The statistics-2015 core finished after 20 hours so I started the statistics-2014 core</li> </ul> <h2 id="2020-11-08">2020-11-08</h2> <ul> <li>Add “Data Paper” to types on CGSpace</li> <li>Add “SCALING CLIMATE-SMART AGRICULTURE” to CCAFS subjects on CGSpace</li> <li>Add “ANDEAN ROOTS AND TUBERS” to CIP subjects on CGSpace</li> <li>Add CGIAR System subjects to Discovery sidebar facets on CGSpace <ul> <li>Also add the System subject to item view on CGSpace</li> </ul> </li> <li>The statistics-2014 core finished processing after five hours, so I started processing the statistics-2013 core on DSpace Test</li> <li>Since I was going to restart CGSpace and update the Discovery indexes anyways I decided to check for any straggling upper case AGROVOC entries and lower case them:</li> </ul> <pre><code>dspace=# BEGIN; dspace=# UPDATE metadatavalue SET text_value=LOWER(text_value) WHERE resource_type_id=2 AND metadata_field_id=57 AND text_value ~ '[[:upper:]]'; UPDATE 164 dspace=# COMMIT; </code></pre><ul> <li>Run system updates on CGSpace (linode18) and reboot it <ul> <li>I had to restart Tomcat once after the machine started up to get all Solr statistics cores to load properly</li> </ul> </li> <li>After about ten more hours the rest of the Solr statistics cores finished processing on DSpace Test and I started optimizing them in Solr admin UI</li> </ul> <h2 id="2020-11-10">2020-11-10</h2> <ul> <li>I am noticing that CGSpace doesn’t have any statistics showing for years before 2020, but all cores are loaded successfully in Solr Admin UI… strange <ul> <li>I restarted Tomcat and I see in Solr Admin UI that the statistics-2015 core failed to load</li> <li>Looking in the DSpace log I see:</li> </ul> </li> </ul> <pre><code>2020-11-10 08:43:59,634 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2015 2020-11-10 08:43:59,687 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2018 2020-11-10 08:43:59,707 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2015 2020-11-10 08:44:00,004 WARN org.dspace.core.ConfigurationManager @ Requested configuration module: atmire-datatables not found 2020-11-10 08:44:00,005 WARN org.dspace.core.ConfigurationManager @ Requested configuration module: atmire-datatables not found 2020-11-10 08:44:00,005 WARN org.dspace.core.ConfigurationManager @ Requested configuration module: atmire-datatables not found 2020-11-10 08:44:00,325 INFO org.dspace.statistics.SolrLogger @ Created core with name: statistics-2015 </code></pre><ul> <li>Seems that the core gets probed twice… perhaps a threading issue? <ul> <li>The only thing I can think of is the <code>acceptorThreadCount</code> parameter in Tomcat’s server.xml, which has been set to 2 since 2018-01 (we started sharding the Solr statistics cores in 2019-01 and that’s when this problem arose)</li> <li>I will try reducing that to 1</li> <li>Wow, now it’s even worse:</li> </ul> </li> </ul> <pre><code>2020-11-10 08:51:03,007 INFO org.dspace.statistics.SolrLogger @ Created core with name: statistics-2018 2020-11-10 08:51:03,008 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2015 2020-11-10 08:51:03,137 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2018 2020-11-10 08:51:03,153 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2015 2020-11-10 08:51:03,289 INFO org.dspace.statistics.SolrLogger @ Created core with name: statistics-2015 2020-11-10 08:51:03,289 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2010 2020-11-10 08:51:03,475 INFO org.dspace.statistics.SolrLogger @ Created core with name: statistics-2010 2020-11-10 08:51:03,475 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2016 2020-11-10 08:51:03,730 INFO org.dspace.statistics.SolrLogger @ Created core with name: statistics-2016 2020-11-10 08:51:03,731 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2017 2020-11-10 08:51:03,992 INFO org.dspace.statistics.SolrLogger @ Created core with name: statistics-2017 2020-11-10 08:51:03,992 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2011 2020-11-10 08:51:04,178 INFO org.dspace.statistics.SolrLogger @ Created core with name: statistics-2011 2020-11-10 08:51:04,178 INFO org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2012 </code></pre><ul> <li>Could it be because we have two Tomcat connectors? <ul> <li>I restarted Tomcat a few more times before all cores loaded, and still there are no stats before 2020-01… hmmmmm</li> </ul> </li> <li>I added a <a href="https://github.com/ilri/OpenRXV/commit/3816b9b3f3d9182d2ba1a899c1017c5895a59dee">lowercase formatter to OpenRXV</a> so that we can lowercase AGROVOC subjects during harvesting</li> </ul> <h2 id="2020-11-11">2020-11-11</h2> <ul> <li>Atmire responded with a quote for the work to fix the duplicate owningComm, etc in our Solr data <ul> <li>I told them to proceed, as it’s within our budget of credits</li> <li>They will write a processor for DSpace 6 to remove the duplicates</li> </ul> </li> <li>I did some tests to add a usage statistics chart to the item views on DSpace Test <ul> <li>It is inspired by Salem’s work on WorldFish’s repository, and it hits the dspace-statistics-api for the current item and displays a graph</li> <li>I got it working very easily for all-time statistics with Chart.js, but I think I will need to use Highcharts or something else because Chart.js is HTML5 canvas and doesn’t allow theming via CSS (so our Bootstrap brand colors for each theme won’t work)</li> <li>I think I’ll pursue this after the DSpace 6 upgrade…</li> </ul> </li> </ul> <h2 id="2020-11-12">2020-11-12</h2> <ul> <li>I was looking at Solr again trying to find a way to get community and collection stats by faceting on <code>owningComm</code> and <code>owningColl</code> and it seems to work actually <ul> <li>The duplicated values in the multi-value fields don’t seem to affect the counts, as I had thought previously (though we should still get rid of them)</li> <li>One major difference between the raw numbers I was looking at and Atmire’s numbers is that Atmire’s code filters “Internal” IP addresses…</li> <li>Also, instead of doing <code>isBot:false</code> I think I should do <code>-isBot:true</code> because it’s not a given that all documents will have this field and have it false, but we can definitely exclude the ones that have it as true</li> </ul> </li> <li>First we get the total number of communities with stats (using calcdistinct):</li> </ul> <pre><code>facet=true&facet.field=owningComm&facet.mincount=1&facet.limit=1&facet.offset=0&stats=true&stats.field=owningComm&stats.calcdistinct=true&shards=http://localhost:8081/solr/statistics,http://localhost:8081/solr/statistics-2019,http://localhost:8081/solr/statistics-2018,http://localhost:8081/solr/statistics-2017,http://localhost:8081/solr/statistics-2016,http://localhost:8081/solr/statistics-2015,http://localhost:8081/solr/statistics-2014,http://localhost:8081/solr/statistics-2013,http://localhost:8081/solr/statistics-2012,http://localhost:8081/solr/statistics-2011,http://localhost:8081/solr/statistics-2010 </code></pre><ul> <li>Then get stats themselves, iterating 100 items at a time with limit and offset:</li> </ul> <pre><code>facet=true&facet.field=owningComm&facet.mincount=1&facet.limit=100&facet.offset=0&shards=http://localhost:8081/solr/statistics,http://localhost:8081/solr/statistics-2019,http://localhost:8081/solr/statistics-2018,http://localhost:8081/solr/statistics-2017,http://localhost:8081/solr/statistics-2016,http://localhost:8081/solr/statistics-2015,http://localhost:8081/solr/statistics-2014,http://localhost:8081/solr/statistics-2013,http://localhost:8081/solr/statistics-2012,http://localhost:8081/solr/statistics-2011,http://localhost:8081/solr/statistics-2010 </code></pre><ul> <li>I was surprised to see 10,000,000 docs with <code>isBot:true</code> when I was testing on DSpace Test… <ul> <li>This has got to be a mistake of some kind, as I see 4 million in 2014 that are from <code>dns:localhost.</code>, perhaps that’s when we didn’t have useProxies set up correctly?</li> <li>I don’t see the same thing on CGSpace… I wonder what happened?</li> <li>Perhaps they got re-tagged during the DSpace 6 upgrade, somehow during the Solr migration? Hmmmmm. Definitely have to be careful with <code>isBot:true</code> in the future and not automatically purge these!!!</li> </ul> </li> <li>I noticed 120,000+ hits from monit, FeedBurner, and Blackboard Safeassign in 2014, 2015, 2016, 2017, etc… <ul> <li>I hadn’t seen monit before, but the others are already in DSpace’s spider agents lists for some time so probably only appear in older stats cores</li> <li>The issue with purging these using <code>check-spider-hits.sh</code> is that it can’t do case-insensitive regexes and some metacharacters like <code>\s</code> don’t work so I added case-sensitive patterns to a local agents file and purged them with the script</li> </ul> </li> </ul> <!-- raw HTML omitted --> </article> </div> <!-- /.blog-main --> <aside class="col-sm-3 ml-auto blog-sidebar"> <section class="sidebar-module"> <h4>Recent Posts</h4> <ol class="list-unstyled"> <li><a href="/cgspace-notes/2020-11/">November, 2020</a></li> <li><a href="/cgspace-notes/2020-10/">October, 2020</a></li> <li><a href="/cgspace-notes/2020-09/">September, 2020</a></li> <li><a href="/cgspace-notes/2020-08/">August, 2020</a></li> <li><a href="/cgspace-notes/2020-07/">July, 2020</a></li> </ol> </section> <section class="sidebar-module"> <h4>Links</h4> <ol class="list-unstyled"> <li><a href="https://cgspace.cgiar.org">CGSpace</a></li> <li><a href="https://dspacetest.cgiar.org">DSpace Test</a></li> <li><a href="https://github.com/ilri/DSpace">CGSpace @ GitHub</a></li> </ol> </section> </aside> </div> <!-- /.row --> </div> <!-- /.container --> <footer class="blog-footer"> <p dir="auto"> Blog template created by <a href="https://twitter.com/mdo">@mdo</a>, ported to Hugo by <a href='https://twitter.com/mralanorth'>@mralanorth</a>. </p> <p> <a href="#">Back to top</a> </p> </footer> </body> </html>