From 6c0563f5cebfb12e47f64f3f3bae34b877384935 Mon Sep 17 00:00:00 2001 From: Alan Orth Date: Tue, 19 Mar 2019 14:30:38 +0200 Subject: [PATCH] Update notes for 2019-03-19 --- content/posts/2019-03.md | 51 +++++++++++++++++++++++++++++++ docs/2019-03/index.html | 66 ++++++++++++++++++++++++++++++++++++++-- docs/robots.txt | 2 +- docs/sitemap.xml | 20 ++++++------ 4 files changed, 125 insertions(+), 14 deletions(-) diff --git a/content/posts/2019-03.md b/content/posts/2019-03.md index 2af39a221..03adcad01 100644 --- a/content/posts/2019-03.md +++ b/content/posts/2019-03.md @@ -499,5 +499,56 @@ dspace=# SELECT count(text_value) FROM metadatavalue WHERE resource_type_id=2 AN ``` - Perhaps my `agrovoc-lookup.py` script could notify if it finds these because they potentially give false negatives +- CGSpace (linode18) is having problems with Solr again, I'm seeing "Error opening new searcher" in the Solr logs and there are no stats for previous years +- Apparently the Solr statistics shards didn't load properly when we restarted Tomcat *yesterday*: + +``` +2019-03-18 12:32:39,799 ERROR org.apache.solr.core.CoreContainer @ Error creating core [statistics-2018]: Error opening new searcher +... +Caused by: org.apache.solr.common.SolrException: Error opening new searcher + at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1565) + at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1677) + at org.apache.solr.core.SolrCore.(SolrCore.java:845) + ... 31 more +Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/home/cgspace.cgiar.org/solr/statistics-2018/data/index/write.lock +``` + +- For reference, I don't see the `ulimit -v unlimited` in the `catalina.sh` script, though the `tomcat7` systemd service has `LimitAS=infinity` +- The limits of the current Tomcat java process are: + +``` +# cat /proc/27182/limits +Limit Soft Limit Hard Limit Units +Max cpu time unlimited unlimited seconds +Max file size unlimited unlimited bytes +Max data size unlimited unlimited bytes +Max stack size 8388608 unlimited bytes +Max core file size 0 unlimited bytes +Max resident set unlimited unlimited bytes +Max processes 128589 128589 processes +Max open files 16384 16384 files +Max locked memory 65536 65536 bytes +Max address space unlimited unlimited bytes +Max file locks unlimited unlimited locks +Max pending signals 128589 128589 signals +Max msgqueue size 819200 819200 bytes +Max nice priority 0 0 +Max realtime priority 0 0 +Max realtime timeout unlimited unlimited us +``` + +- I will try to add `ulimit -v unlimited` to the Catalina startup script and check the output of the limits to see if it's different in practice, as some wisdom on Stack Overflow says this solves the Solr core issues and I've superstitiously tried it various times in the past + - The result is the same before and after, so *adding the ulimit directly is unneccessary* (whether or not unlimited address space is useful or not is another question) +- For now I will just stop Tomcat, delete Solr locks, then start Tomcat again: + +``` +# systemctl stop tomcat7 +# find /home/cgspace.cgiar.org/solr/ -iname "*.lock" -delete +# systemctl start tomcat7 +``` + +- After restarting I confirmed that all Solr statistics cores were loaded successfully... +- Another avenue might be to look at point releases in Solr 4.10.x, as we're running 4.10.2 and they released 4.10.3 and 4.10.4 back in 2014 or 2015 + - I see several issues regarding locks and IndexWriter that were fixed in Solr and Lucene 4.10.3 and 4.10.4... diff --git a/docs/2019-03/index.html b/docs/2019-03/index.html index 4050cfc52..2dca45d86 100644 --- a/docs/2019-03/index.html +++ b/docs/2019-03/index.html @@ -25,7 +25,7 @@ I think I will need to ask Udana to re-copy and paste the abstracts with more ca - + @@ -55,9 +55,9 @@ I think I will need to ask Udana to re-copy and paste the abstracts with more ca "@type": "BlogPosting", "headline": "March, 2019", "url": "https://alanorth.github.io/cgspace-notes/2019-03/", - "wordCount": "3049", + "wordCount": "3421", "datePublished": "2019-03-01T12:16:30+01:00", - "dateModified": "2019-03-18T21:55:08+02:00", + "dateModified": "2019-03-19T11:33:51+02:00", "author": { "@type": "Person", "name": "Alan Orth" @@ -702,6 +702,66 @@ $ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|ds + +
2019-03-18 12:32:39,799 ERROR org.apache.solr.core.CoreContainer @ Error creating core [statistics-2018]: Error opening new searcher
+...
+Caused by: org.apache.solr.common.SolrException: Error opening new searcher
+        at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1565)
+        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1677)
+        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:845)
+        ... 31 more
+Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/home/cgspace.cgiar.org/solr/statistics-2018/data/index/write.lock
+
+ + + +
# cat /proc/27182/limits 
+Limit                     Soft Limit           Hard Limit           Units     
+Max cpu time              unlimited            unlimited            seconds   
+Max file size             unlimited            unlimited            bytes     
+Max data size             unlimited            unlimited            bytes     
+Max stack size            8388608              unlimited            bytes     
+Max core file size        0                    unlimited            bytes     
+Max resident set          unlimited            unlimited            bytes     
+Max processes             128589               128589               processes 
+Max open files            16384                16384                files     
+Max locked memory         65536                65536                bytes     
+Max address space         unlimited            unlimited            bytes     
+Max file locks            unlimited            unlimited            locks     
+Max pending signals       128589               128589               signals   
+Max msgqueue size         819200               819200               bytes     
+Max nice priority         0                    0                    
+Max realtime priority     0                    0                    
+Max realtime timeout      unlimited            unlimited            us
+
+ + + +
# systemctl stop tomcat7
+# find /home/cgspace.cgiar.org/solr/ -iname "*.lock" -delete
+# systemctl start tomcat7
+
+ + diff --git a/docs/robots.txt b/docs/robots.txt index 2b88ae317..9c8f41429 100644 --- a/docs/robots.txt +++ b/docs/robots.txt @@ -45,7 +45,7 @@ Disallow: /cgspace-notes/2015-12/ Disallow: /cgspace-notes/2015-11/ Disallow: /cgspace-notes/ Disallow: /cgspace-notes/categories/ -Disallow: /cgspace-notes/categories/notes/ Disallow: /cgspace-notes/tags/notes/ +Disallow: /cgspace-notes/categories/notes/ Disallow: /cgspace-notes/posts/ Disallow: /cgspace-notes/tags/ diff --git a/docs/sitemap.xml b/docs/sitemap.xml index 6af4f5374..c7d99816d 100644 --- a/docs/sitemap.xml +++ b/docs/sitemap.xml @@ -4,7 +4,7 @@ https://alanorth.github.io/cgspace-notes/2019-03/ - 2019-03-18T21:55:08+02:00 + 2019-03-19T11:33:51+02:00 @@ -214,7 +214,7 @@ https://alanorth.github.io/cgspace-notes/ - 2019-03-18T21:55:08+02:00 + 2019-03-19T11:33:51+02:00 0 @@ -223,27 +223,27 @@ 0 + + https://alanorth.github.io/cgspace-notes/tags/notes/ + 2019-03-19T11:33:51+02:00 + 0 + + https://alanorth.github.io/cgspace-notes/categories/notes/ 2018-03-09T22:10:33+02:00 0 - - https://alanorth.github.io/cgspace-notes/tags/notes/ - 2019-03-18T21:55:08+02:00 - 0 - - https://alanorth.github.io/cgspace-notes/posts/ - 2019-03-18T21:55:08+02:00 + 2019-03-19T11:33:51+02:00 0 https://alanorth.github.io/cgspace-notes/tags/ - 2019-03-18T21:55:08+02:00 + 2019-03-19T11:33:51+02:00 0