Update notes for 2019-07-01

content/posts/2019-07.md

@@ -8,7 +8,63 @@ tags: ["Notes"]
 ## 2019-07-01
 
 - Create an "AfricaRice books and book chapters" collection on CGSpace for AfricaRice
+- Last month Sisay asked why the following "most popular" statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace:
+  - [DSpace Test](https://dspacetest.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&time_filter_end_date=01%2F12%2F2018)
+  - [CGSpace](https://cgspace.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&time_filter_end_date=01%2F12%2F2018)
+- Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community
 
 <!--more-->
 
+- If I change the parameters to 2019 I see stats, so I'm really thinking it has something to do with the sharded yearly Solr statistics cores
+  - I checked the Solr admin UI and I see all Solr cores loaded, so I don't know what it could be
+  - When I check the Atmire content and usage module it seems obvious that there is a problem with the old cores because I dont have anything before 2019-01
+
+![Atmire CUA 2018 stats missing](/cgspace-notes/2019/07/atmire-cua-2018-missing.png)
+
+- I don't see anyone logged in right now so I'm going to try to restart Tomcat and see if the stats are accessible after Solr comes back up
+- I decided to run all system updates on the server (linode18) and reboot it
+  - After rebooting Tomcat came back up, but the the Solr statistics cores were not all loaded
+  - The error is always (with a different core):
+
+```
+org.apache.solr.common.SolrException: Error CREATEing SolrCore 'statistics-2010': Unable to create core [statistics-2010] Caused by: Lock obtain timed out: NativeFSLock@/home/cgspace.cgiar.org/solr/statistics-2010/data/index/write.lock
+```
+
+- I restarted Tomcat *ten times* and it never worked...
+- I tried to stop Tomcat and delete the write locks:
+
+```
+# systemctl stop tomcat7
+# find /dspace/solr/statistics* -iname "*.lock" -print -delete
+/dspace/solr/statistics/data/index/write.lock
+/dspace/solr/statistics-2010/data/index/write.lock
+/dspace/solr/statistics-2011/data/index/write.lock
+/dspace/solr/statistics-2012/data/index/write.lock
+/dspace/solr/statistics-2013/data/index/write.lock
+/dspace/solr/statistics-2014/data/index/write.lock
+/dspace/solr/statistics-2015/data/index/write.lock
+/dspace/solr/statistics-2016/data/index/write.lock
+/dspace/solr/statistics-2017/data/index/write.lock
+/dspace/solr/statistics-2018/data/index/write.lock
+# find /dspace/solr/statistics* -iname "*.lock" -print -delete
+# systemctl start tomcat7
+```
+
+- But it still didn't work!
+- I stopped Tomcat, deleted the old locks, and will try to use the "simple" lock file type in `solr/statistics/conf/solrconfig.xml`:
+
+```
+<lockType>${solr.lock.type:simple}</lockType>
+```
+
+- And after restarting Tomcat it still doesn't work
+- Now I'll try going back to "native" locking with `unlockAtStartup`:
+
+```
+<unlockOnStartup>true</unlockOnStartup>
+```
+
+- Now the cores seem to load, but I still see an error in the Solr Admin UI and I still can't access any stats before 2018
+- I filed an [issue with Atmire](https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=685), so let's see if they can help
+
 <!-- vim: set sw=2 ts=2: -->

docs/2019-07/index.html

@@ -10,11 +10,18 @@
 
 
 Create an “AfricaRice books and book chapters” collection on CGSpace for AfricaRice
+Last month Sisay asked why the following “most popular” statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace:
+
+
+DSpace Test
+CGSpace
+
+Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community
 " />
 <meta property="og:type" content="article" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2019-07/" />
 <meta property="article:published_time" content="2019-07-01T12:13:51&#43;03:00"/>
-<meta property="article:modified_time" content="2019-07-01T12:13:51&#43;03:00"/>
+<meta property="article:modified_time" content="2019-07-01T12:22:43&#43;03:00"/>
 
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="July, 2019"/>
@@ -22,6 +29,13 @@ Create an &ldquo;AfricaRice books and book chapters&rdquo; collection on CGSpace
 
 
 Create an &ldquo;AfricaRice books and book chapters&rdquo; collection on CGSpace for AfricaRice
+Last month Sisay asked why the following &ldquo;most popular&rdquo; statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace:
+
+
+DSpace Test
+CGSpace
+
+Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community
 "/>
 <meta name="generator" content="Hugo 0.55.6" />
 
@@ -33,9 +47,9 @@ Create an &ldquo;AfricaRice books and book chapters&rdquo; collection on CGSpace
   "@type": "BlogPosting",
   "headline": "July, 2019",
   "url": "https:\/\/alanorth.github.io\/cgspace-notes\/2019-07\/",
-  "wordCount": "13",
+  "wordCount": "359",
   "datePublished": "2019-07-01T12:13:51\x2b03:00",
-  "dateModified": "2019-07-01T12:13:51\x2b03:00",
+  "dateModified": "2019-07-01T12:22:43\x2b03:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"
@@ -106,6 +120,77 @@ Create an &ldquo;AfricaRice books and book chapters&rdquo; collection on CGSpace
 
 <ul>
 <li>Create an &ldquo;AfricaRice books and book chapters&rdquo; collection on CGSpace for AfricaRice</li>
+<li>Last month Sisay asked why the following &ldquo;most popular&rdquo; statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace:
+
+<ul>
+<li><a href="https://dspacetest.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&amp;time_filter_end_date=01%2F12%2F2018">DSpace Test</a></li>
+<li><a href="https://cgspace.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&amp;time_filter_end_date=01%2F12%2F2018">CGSpace</a></li>
+</ul></li>
+<li>Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community</li>
+</ul>
+
+<ul>
+<li>If I change the parameters to 2019 I see stats, so I&rsquo;m really thinking it has something to do with the sharded yearly Solr statistics cores
+
+<ul>
+<li>I checked the Solr admin UI and I see all Solr cores loaded, so I don&rsquo;t know what it could be</li>
+<li>When I check the Atmire content and usage module it seems obvious that there is a problem with the old cores because I dont have anything before 2019-01</li>
+</ul></li>
+</ul>
+
+<p><img src="/cgspace-notes/2019/07/atmire-cua-2018-missing.png" alt="Atmire CUA 2018 stats missing" /></p>
+
+<ul>
+<li>I don&rsquo;t see anyone logged in right now so I&rsquo;m going to try to restart Tomcat and see if the stats are accessible after Solr comes back up</li>
+
+<li><p>I decided to run all system updates on the server (linode18) and reboot it</p>
+
+<ul>
+<li>After rebooting Tomcat came back up, but the the Solr statistics cores were not all loaded</li>
+
+<li><p>The error is always (with a different core):</p>
+
+<pre><code>org.apache.solr.common.SolrException: Error CREATEing SolrCore 'statistics-2010': Unable to create core [statistics-2010] Caused by: Lock obtain timed out: NativeFSLock@/home/cgspace.cgiar.org/solr/statistics-2010/data/index/write.lock
+</code></pre></li>
+</ul></li>
+
+<li><p>I restarted Tomcat <em>ten times</em> and it never worked&hellip;</p></li>
+
+<li><p>I tried to stop Tomcat and delete the write locks:</p>
+
+<pre><code># systemctl stop tomcat7
+# find /dspace/solr/statistics* -iname &quot;*.lock&quot; -print -delete
+/dspace/solr/statistics/data/index/write.lock
+/dspace/solr/statistics-2010/data/index/write.lock
+/dspace/solr/statistics-2011/data/index/write.lock
+/dspace/solr/statistics-2012/data/index/write.lock
+/dspace/solr/statistics-2013/data/index/write.lock
+/dspace/solr/statistics-2014/data/index/write.lock
+/dspace/solr/statistics-2015/data/index/write.lock
+/dspace/solr/statistics-2016/data/index/write.lock
+/dspace/solr/statistics-2017/data/index/write.lock
+/dspace/solr/statistics-2018/data/index/write.lock
+# find /dspace/solr/statistics* -iname &quot;*.lock&quot; -print -delete
+# systemctl start tomcat7
+</code></pre></li>
+
+<li><p>But it still didn&rsquo;t work!</p></li>
+
+<li><p>I stopped Tomcat, deleted the old locks, and will try to use the &ldquo;simple&rdquo; lock file type in <code>solr/statistics/conf/solrconfig.xml</code>:</p>
+
+<pre><code>&lt;lockType&gt;${solr.lock.type:simple}&lt;/lockType&gt;
+</code></pre></li>
+
+<li><p>And after restarting Tomcat it still doesn&rsquo;t work</p></li>
+
+<li><p>Now I&rsquo;ll try going back to &ldquo;native&rdquo; locking with <code>unlockAtStartup</code>:</p>
+
+<pre><code>&lt;unlockOnStartup&gt;true&lt;/unlockOnStartup&gt;
+</code></pre></li>
+
+<li><p>Now the cores seem to load, but I still see an error in the Solr Admin UI and I still can&rsquo;t access any stats before 2018</p></li>
+
+<li><p>I filed an <a href="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=685">issue with Atmire</a>, so let&rsquo;s see if they can help</p></li>
 </ul>
 
 <!-- vim: set sw=2 ts=2: -->

docs/categories/index.html

@@ -102,6 +102,13 @@
 
 <ul>
 <li>Create an &ldquo;AfricaRice books and book chapters&rdquo; collection on CGSpace for AfricaRice</li>
+<li>Last month Sisay asked why the following &ldquo;most popular&rdquo; statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace:
+
+<ul>
+<li><a href="https://dspacetest.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&amp;time_filter_end_date=01%2F12%2F2018">DSpace Test</a></li>
+<li><a href="https://cgspace.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&amp;time_filter_end_date=01%2F12%2F2018">CGSpace</a></li>
+</ul></li>
+<li>Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community</li>
 </ul>
   <a href='https://alanorth.github.io/cgspace-notes/2019-07/'>Read more →</a>
 </article> 

docs/index.html

@@ -102,6 +102,13 @@
 
 <ul>
 <li>Create an &ldquo;AfricaRice books and book chapters&rdquo; collection on CGSpace for AfricaRice</li>
+<li>Last month Sisay asked why the following &ldquo;most popular&rdquo; statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace:
+
+<ul>
+<li><a href="https://dspacetest.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&amp;time_filter_end_date=01%2F12%2F2018">DSpace Test</a></li>
+<li><a href="https://cgspace.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&amp;time_filter_end_date=01%2F12%2F2018">CGSpace</a></li>
+</ul></li>
+<li>Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community</li>
 </ul>
   <a href='https://alanorth.github.io/cgspace-notes/2019-07/'>Read more →</a>
 </article> 

docs/index.xml

@@ -21,6 +21,13 @@
 
 <ul>
 <li>Create an “AfricaRice books and book chapters” collection on CGSpace for AfricaRice</li>
+<li>Last month Sisay asked why the following “most popular” statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace:
+
+<ul>
+<li><a href="https://dspacetest.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&time_filter_end_date=01%2F12%2F2018">DSpace Test</a></li>
+<li><a href="https://cgspace.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&time_filter_end_date=01%2F12%2F2018">CGSpace</a></li>
+</ul></li>
+<li>Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community</li>
 &lt;/ul&gt;</description>
     </item>
     

docs/posts/index.html

@@ -102,6 +102,13 @@
 
 <ul>
 <li>Create an &ldquo;AfricaRice books and book chapters&rdquo; collection on CGSpace for AfricaRice</li>
+<li>Last month Sisay asked why the following &ldquo;most popular&rdquo; statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace:
+
+<ul>
+<li><a href="https://dspacetest.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&amp;time_filter_end_date=01%2F12%2F2018">DSpace Test</a></li>
+<li><a href="https://cgspace.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&amp;time_filter_end_date=01%2F12%2F2018">CGSpace</a></li>
+</ul></li>
+<li>Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community</li>
 </ul>
   <a href='https://alanorth.github.io/cgspace-notes/2019-07/'>Read more →</a>
 </article> 

docs/posts/index.xml

@@ -21,6 +21,13 @@
 
 <ul>
 <li>Create an “AfricaRice books and book chapters” collection on CGSpace for AfricaRice</li>
+<li>Last month Sisay asked why the following “most popular” statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace:
+
+<ul>
+<li><a href="https://dspacetest.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&time_filter_end_date=01%2F12%2F2018">DSpace Test</a></li>
+<li><a href="https://cgspace.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&time_filter_end_date=01%2F12%2F2018">CGSpace</a></li>
+</ul></li>
+<li>Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community</li>
 &lt;/ul&gt;</description>
     </item>
     

docs/sitemap.xml

@@ -4,30 +4,30 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2019-07-01T12:14:35+03:00</lastmod>
+    <lastmod>2019-07-01T12:22:43+03:00</lastmod>
     <priority>0</priority>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2019-07/</loc>
-    <lastmod>2019-07-01T12:13:51+03:00</lastmod>
+    <lastmod>2019-07-01T12:22:43+03:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
-    <lastmod>2019-07-01T12:14:35+03:00</lastmod>
+    <lastmod>2019-07-01T12:22:43+03:00</lastmod>
     <priority>0</priority>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
-    <lastmod>2019-07-01T12:14:35+03:00</lastmod>
+    <lastmod>2019-07-01T12:22:43+03:00</lastmod>
     <priority>0</priority>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/tags/</loc>
-    <lastmod>2019-07-01T12:14:35+03:00</lastmod>
+    <lastmod>2019-07-01T12:22:43+03:00</lastmod>
     <priority>0</priority>
   </url>
   

docs/tags/index.html

@@ -102,6 +102,13 @@
 
 <ul>
 <li>Create an &ldquo;AfricaRice books and book chapters&rdquo; collection on CGSpace for AfricaRice</li>
+<li>Last month Sisay asked why the following &ldquo;most popular&rdquo; statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace:
+
+<ul>
+<li><a href="https://dspacetest.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&amp;time_filter_end_date=01%2F12%2F2018">DSpace Test</a></li>
+<li><a href="https://cgspace.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&amp;time_filter_end_date=01%2F12%2F2018">CGSpace</a></li>
+</ul></li>
+<li>Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community</li>
 </ul>
   <a href='https://alanorth.github.io/cgspace-notes/2019-07/'>Read more →</a>
 </article> 

docs/tags/notes/index.html

@@ -87,6 +87,13 @@
 
 <ul>
 <li>Create an &ldquo;AfricaRice books and book chapters&rdquo; collection on CGSpace for AfricaRice</li>
+<li>Last month Sisay asked why the following &ldquo;most popular&rdquo; statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace:
+
+<ul>
+<li><a href="https://dspacetest.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&amp;time_filter_end_date=01%2F12%2F2018">DSpace Test</a></li>
+<li><a href="https://cgspace.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&amp;time_filter_end_date=01%2F12%2F2018">CGSpace</a></li>
+</ul></li>
+<li>Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community</li>
 </ul>
   <a href='https://alanorth.github.io/cgspace-notes/2019-07/'>Read more →</a>
 </article> 

docs/tags/notes/index.xml

@@ -21,6 +21,13 @@
 
 <ul>
 <li>Create an “AfricaRice books and book chapters” collection on CGSpace for AfricaRice</li>
+<li>Last month Sisay asked why the following “most popular” statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace:
+
+<ul>
+<li><a href="https://dspacetest.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&time_filter_end_date=01%2F12%2F2018">DSpace Test</a></li>
+<li><a href="https://cgspace.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&time_filter_end_date=01%2F12%2F2018">CGSpace</a></li>
+</ul></li>
+<li>Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community</li>
 &lt;/ul&gt;</description>
     </item>