CGSpace Notes

Documenting day-to-day work on the CGSpace repository.

July, 2019

2019-07-01

  • Create an “AfricaRice books and book chapters” collection on CGSpace for AfricaRice
  • Last month Sisay asked why the following “most popular” statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace:
  • Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community
  • If I change the parameters to 2019 I see stats, so I’m really thinking it has something to do with the sharded yearly Solr statistics cores
    • I checked the Solr admin UI and I see all Solr cores loaded, so I don’t know what it could be
    • When I check the Atmire content and usage module it seems obvious that there is a problem with the old cores because I dont have anything before 2019-01

Atmire CUA 2018 stats missing

  • I don’t see anyone logged in right now so I’m going to try to restart Tomcat and see if the stats are accessible after Solr comes back up
  • I decided to run all system updates on the server (linode18) and reboot it

    • After rebooting Tomcat came back up, but the the Solr statistics cores were not all loaded
    • The error is always (with a different core):

      org.apache.solr.common.SolrException: Error CREATEing SolrCore 'statistics-2010': Unable to create core [statistics-2010] Caused by: Lock obtain timed out: NativeFSLock@/home/cgspace.cgiar.org/solr/statistics-2010/data/index/write.lock
      
  • I restarted Tomcat ten times and it never worked…

  • I tried to stop Tomcat and delete the write locks:

    # systemctl stop tomcat7
    # find /dspace/solr/statistics* -iname "*.lock" -print -delete
    /dspace/solr/statistics/data/index/write.lock
    /dspace/solr/statistics-2010/data/index/write.lock
    /dspace/solr/statistics-2011/data/index/write.lock
    /dspace/solr/statistics-2012/data/index/write.lock
    /dspace/solr/statistics-2013/data/index/write.lock
    /dspace/solr/statistics-2014/data/index/write.lock
    /dspace/solr/statistics-2015/data/index/write.lock
    /dspace/solr/statistics-2016/data/index/write.lock
    /dspace/solr/statistics-2017/data/index/write.lock
    /dspace/solr/statistics-2018/data/index/write.lock
    # find /dspace/solr/statistics* -iname "*.lock" -print -delete
    # systemctl start tomcat7
    
  • But it still didn’t work!

  • I stopped Tomcat, deleted the old locks, and will try to use the “simple” lock file type in solr/statistics/conf/solrconfig.xml:

    <lockType>${solr.lock.type:simple}</lockType>
    
  • And after restarting Tomcat it still doesn’t work

  • Now I’ll try going back to “native” locking with unlockAtStartup:

    <unlockOnStartup>true</unlockOnStartup>
    
  • Now the cores seem to load, but I still see an error in the Solr Admin UI and I still can’t access any stats before 2018

  • I filed an issue with Atmire, so let’s see if they can help

  • And since I’m annoyed and it’s been a few months, I’m going to move the JVM heap settings that I’ve been testing on DSpace Test to CGSpace

  • The old ones were:

    -Djava.awt.headless=true -Xms8192m -Xmx8192m -XX:+UseConcMarkSweepGC -Dfile.encoding=UTF-8 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=5400 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
    
  • And the new ones come from Solr 4.10.x’s startup scripts:

    -Djava.awt.headless=true
    -Xms8192m -Xmx8192m
    -Dfile.encoding=UTF-8
    -XX:NewRatio=3
    -XX:SurvivorRatio=4
    -XX:TargetSurvivorRatio=90
    -XX:MaxTenuringThreshold=8
    -XX:+UseConcMarkSweepGC
    -XX:+UseParNewGC
    -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4
    -XX:+CMSScavengeBeforeRemark
    -XX:PretenureSizeThreshold=64m
    -XX:+UseCMSInitiatingOccupancyOnly
    -XX:CMSInitiatingOccupancyFraction=50
    -XX:CMSMaxAbortablePrecleanTime=6000
    -XX:+CMSParallelRemarkEnabled
    -XX:+ParallelRefProcEnabled
    -Dcom.sun.management.jmxremote
    -Dcom.sun.management.jmxremote.port=1337
    -Dcom.sun.management.jmxremote.ssl=false
    -Dcom.sun.management.jmxremote.authenticate=false