July, 2019

Mon Jul 01, 2019 by Alan Orth in Notes

2019-07-01

Create an “AfricaRice books and book chapters” collection on CGSpace for AfricaRice
Last month Sisay asked why the following “most popular” statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace:
- DSpace Test
- CGSpace
Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community

If I change the parameters to 2019 I see stats, so I’m really thinking it has something to do with the sharded yearly Solr statistics cores
- I checked the Solr admin UI and I see all Solr cores loaded, so I don’t know what it could be
- When I check the Atmire content and usage module it seems obvious that there is a problem with the old cores because I dont have anything before 2019-01

Atmire CUA 2018 stats missing

I don’t see anyone logged in right now so I’m going to try to restart Tomcat and see if the stats are accessible after Solr comes back up

I decided to run all system updates on the server (linode18) and reboot it

After rebooting Tomcat came back up, but the the Solr statistics cores were not all loaded

The error is always (with a different core):

org.apache.solr.common.SolrException: Error CREATEing SolrCore 'statistics-2010': Unable to create core [statistics-2010] Caused by: Lock obtain timed out: NativeFSLock@/home/cgspace.cgiar.org/solr/statistics-2010/data/index/write.lock

I restarted Tomcat ten times and it never worked…

I tried to stop Tomcat and delete the write locks:

# systemctl stop tomcat7
# find /dspace/solr/statistics* -iname "*.lock" -print -delete
/dspace/solr/statistics/data/index/write.lock
/dspace/solr/statistics-2010/data/index/write.lock
/dspace/solr/statistics-2011/data/index/write.lock
/dspace/solr/statistics-2012/data/index/write.lock
/dspace/solr/statistics-2013/data/index/write.lock
/dspace/solr/statistics-2014/data/index/write.lock
/dspace/solr/statistics-2015/data/index/write.lock
/dspace/solr/statistics-2016/data/index/write.lock
/dspace/solr/statistics-2017/data/index/write.lock
/dspace/solr/statistics-2018/data/index/write.lock
# find /dspace/solr/statistics* -iname "*.lock" -print -delete
# systemctl start tomcat7

But it still didn’t work!
I stopped Tomcat, deleted the old locks, and will try to use the “simple” lock file type in solr/statistics/conf/solrconfig.xml:
```
<lockType>${solr.lock.type:simple}</lockType>
```
And after restarting Tomcat it still doesn’t work
Now I’ll try going back to “native” locking with unlockAtStartup:
```
<unlockOnStartup>true</unlockOnStartup>
```
Now the cores seem to load, but I still see an error in the Solr Admin UI and I still can’t access any stats before 2018
I filed an issue with Atmire, so let’s see if they can help