---
title: "July, 2019"
date: 2019-07-01T12:13:51+03:00
author: "Alan Orth"
tags: ["Notes"]
---
## 2019-07-01
- Create an "AfricaRice books and book chapters" collection on CGSpace for AfricaRice
- Last month Sisay asked why the following "most popular" statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace:
- [DSpace Test](https://dspacetest.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&time_filter_end_date=01%2F12%2F2018)
- [CGSpace](https://cgspace.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&time_filter_end_date=01%2F12%2F2018)
- Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community
- If I change the parameters to 2019 I see stats, so I'm really thinking it has something to do with the sharded yearly Solr statistics cores
- I checked the Solr admin UI and I see all Solr cores loaded, so I don't know what it could be
- When I check the Atmire content and usage module it seems obvious that there is a problem with the old cores because I dont have anything before 2019-01
- I don't see anyone logged in right now so I'm going to try to restart Tomcat and see if the stats are accessible after Solr comes back up
- I decided to run all system updates on the server (linode18) and reboot it
- After rebooting Tomcat came back up, but the the Solr statistics cores were not all loaded
- The error is always (with a different core):
```
org.apache.solr.common.SolrException: Error CREATEing SolrCore 'statistics-2010': Unable to create core [statistics-2010] Caused by: Lock obtain timed out: NativeFSLock@/home/cgspace.cgiar.org/solr/statistics-2010/data/index/write.lock
```
- I restarted Tomcat *ten times* and it never worked...
- I tried to stop Tomcat and delete the write locks:
```
# systemctl stop tomcat7
# find /dspace/solr/statistics* -iname "*.lock" -print -delete
/dspace/solr/statistics/data/index/write.lock
/dspace/solr/statistics-2010/data/index/write.lock
/dspace/solr/statistics-2011/data/index/write.lock
/dspace/solr/statistics-2012/data/index/write.lock
/dspace/solr/statistics-2013/data/index/write.lock
/dspace/solr/statistics-2014/data/index/write.lock
/dspace/solr/statistics-2015/data/index/write.lock
/dspace/solr/statistics-2016/data/index/write.lock
/dspace/solr/statistics-2017/data/index/write.lock
/dspace/solr/statistics-2018/data/index/write.lock
# find /dspace/solr/statistics* -iname "*.lock" -print -delete
# systemctl start tomcat7
```
- But it still didn't work!
- I stopped Tomcat, deleted the old locks, and will try to use the "simple" lock file type in `solr/statistics/conf/solrconfig.xml`:
```
${solr.lock.type:simple}
```
- And after restarting Tomcat it still doesn't work
- Now I'll try going back to "native" locking with `unlockAtStartup`:
```
true
```
- Now the cores seem to load, but I still see an error in the Solr Admin UI and I still can't access any stats before 2018
- I filed an [issue with Atmire](https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=685), so let's see if they can help
- And since I'm annoyed and it's been a few months, I'm going to move the JVM heap settings that I've been testing on DSpace Test to CGSpace
- The old ones were:
```
-Djava.awt.headless=true -Xms8192m -Xmx8192m -XX:+UseConcMarkSweepGC -Dfile.encoding=UTF-8 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=5400 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
```
- And the new ones come from Solr 4.10.x's startup scripts:
```
-Djava.awt.headless=true
-Xms8192m -Xmx8192m
-Dfile.encoding=UTF-8
-XX:NewRatio=3
-XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90
-XX:MaxTenuringThreshold=8
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-XX:ConcGCThreads=4 -XX:ParallelGCThreads=4
-XX:+CMSScavengeBeforeRemark
-XX:PretenureSizeThreshold=64m
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
-XX:CMSMaxAbortablePrecleanTime=6000
-XX:+CMSParallelRemarkEnabled
-XX:+ParallelRefProcEnabled
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=1337
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
```
## 2019-07-02
- Help upload twenty-seven posters from the 2019-05 Sharefair to CGSpace
- Sisay had already done the SAFBundle so I did some minor corrections to and uploaded them to a temporary collection so I could check them in OpenRefine:
```
$ sed -i 's/CC-BY 4.0/CC-BY-4.0/' item_*/dublin_core.xml
$ echo "10568/101992" >> item_*/collections
$ dspace import -a -e me@cgiar.org -m 2019-07-02-Sharefair.map -s /tmp/Sharefair_mapped
```
- I noticed that all twenty-seven items had double dates like "2019-05||2019-05" so I fixed those, but the rest of the metadata looked good so I unmapped them from the temporary collection
- Finish looking at the fifty-six AfricaRice items and upload them to CGSpace:
```
$ dspace import -a -e me@cgiar.org -m 2019-07-02-AfricaRice-11to73.map -s /tmp/SimpleArchiveFormat
```
- Peter pointed out that the Sharefair dates I fixed were not actually fixed
- It seems there is a bug that causes DSpace to not detect changes if the values are the same like "2019-05||2019-05" and you try to remove one
- To get it to work I had to change some of them to 2019-01, then remove them