alanorth/cgspace-notes
1
0
Fork 0
mirror of https://github.com/alanorth/cgspace-notes.git synced 2024-11-29 18:08:20 +01:00
Code Issues Projects Releases Wiki Activity

Update notes for 2019-03-19

Browse Source
This commit is contained in:
Alan Orth 2019-03-19 14:30:38 +02:00
parent fa99c86dd2
commit 6c0563f5ce
Signed by: alanorth
GPG Key ID: 0FB860CC9C45B1B9
4 changed files with 125 additions and 14 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

51
content/posts/2019-03.md
View File

@ -499,5 +499,56 @@ dspace=# SELECT count(text_value) FROM metadatavalue WHERE resource_type_id=2 AN
``` ```
- Perhaps my `agrovoc-lookup.py` script could notify if it finds these because they potentially give false negatives - Perhaps my `agrovoc-lookup.py` script could notify if it finds these because they potentially give false negatives
- CGSpace (linode18) is having problems with Solr again, I'm seeing "Error opening new searcher" in the Solr logs and there are no stats for previous years
- Apparently the Solr statistics shards didn't load properly when we restarted Tomcat *yesterday*:
```
2019-03-18 12:32:39,799 ERROR org.apache.solr.core.CoreContainer @ Error creating core [statistics-2018]: Error opening new searcher
...
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1565)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1677)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:845)
... 31 more
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/home/cgspace.cgiar.org/solr/statistics-2018/data/index/write.lock
```
- For reference, I don't see the `ulimit -v unlimited` in the `catalina.sh` script, though the `tomcat7` systemd service has `LimitAS=infinity`
- The limits of the current Tomcat java process are:
```
# cat /proc/27182/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 128589 128589 processes
Max open files 16384 16384 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 128589 128589 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
```
- I will try to add `ulimit -v unlimited` to the Catalina startup script and check the output of the limits to see if it's different in practice, as some wisdom on Stack Overflow says this solves the Solr core issues and I've superstitiously tried it various times in the past
- The result is the same before and after, so *adding the ulimit directly is unneccessary* (whether or not unlimited address space is useful or not is another question)
- For now I will just stop Tomcat, delete Solr locks, then start Tomcat again:
```
# systemctl stop tomcat7
# find /home/cgspace.cgiar.org/solr/ -iname "*.lock" -delete
# systemctl start tomcat7
```
- After restarting I confirmed that all Solr statistics cores were loaded successfully...
- Another avenue might be to look at point releases in Solr 4.10.x, as we're running 4.10.2 and they released 4.10.3 and 4.10.4 back in 2014 or 2015
- I see several issues regarding locks and IndexWriter that were fixed in Solr and Lucene 4.10.3 and 4.10.4...
<!-- vim: set sw=2 ts=2: --> <!-- vim: set sw=2 ts=2: -->

66
docs/2019-03/index.html
View File

@ -25,7 +25,7 @@ I think I will need to ask Udana to re-copy and paste the abstracts with more ca
<meta property="og:type" content="article" /> <meta property="og:type" content="article" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2019-03/" /> <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2019-03/" />
<meta property="article:published_time" content="2019-03-01T12:16:30&#43;01:00"/> <meta property="article:published_time" content="2019-03-01T12:16:30&#43;01:00"/>
<meta property="article:modified_time" content="2019-03-18T21:55:08&#43;02:00"/> <meta property="article:modified_time" content="2019-03-19T11:33:51&#43;02:00"/>
<meta name="twitter:card" content="summary"/> <meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="March, 2019"/> <meta name="twitter:title" content="March, 2019"/>
@ -55,9 +55,9 @@ I think I will need to ask Udana to re-copy and paste the abstracts with more ca
"@type": "BlogPosting", "@type": "BlogPosting",
"headline": "March, 2019", "headline": "March, 2019",
"url": "https://alanorth.github.io/cgspace-notes/2019-03/", "url": "https://alanorth.github.io/cgspace-notes/2019-03/",
"wordCount": "3049", "wordCount": "3421",
"datePublished": "2019-03-01T12:16:30&#43;01:00", "datePublished": "2019-03-01T12:16:30&#43;01:00",
"dateModified": "2019-03-18T21:55:08&#43;02:00", "dateModified": "2019-03-19T11:33:51&#43;02:00",
"author": { "author": {
"@type": "Person", "@type": "Person",
"name": "Alan Orth" "name": "Alan Orth"
@ -702,6 +702,66 @@ $ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|ds
<ul> <ul>
<li>Perhaps my <code>agrovoc-lookup.py</code> script could notify if it finds these because they potentially give false negatives</li> <li>Perhaps my <code>agrovoc-lookup.py</code> script could notify if it finds these because they potentially give false negatives</li>
<li>CGSpace (linode18) is having problems with Solr again, I&rsquo;m seeing &ldquo;Error opening new searcher&rdquo; in the Solr logs and there are no stats for previous years</li>
<li>Apparently the Solr statistics shards didn&rsquo;t load properly when we restarted Tomcat <em>yesterday</em>:</li>
</ul>
<pre><code>2019-03-18 12:32:39,799 ERROR org.apache.solr.core.CoreContainer @ Error creating core [statistics-2018]: Error opening new searcher
...
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1565)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1677)
at org.apache.solr.core.SolrCore.&lt;init&gt;(SolrCore.java:845)
... 31 more
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/home/cgspace.cgiar.org/solr/statistics-2018/data/index/write.lock
</code></pre>
<ul>
<li>For reference, I don&rsquo;t see the <code>ulimit -v unlimited</code> in the <code>catalina.sh</code> script, though the <code>tomcat7</code> systemd service has <code>LimitAS=infinity</code></li>
<li>The limits of the current Tomcat java process are:</li>
</ul>
<pre><code># cat /proc/27182/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 128589 128589 processes
Max open files 16384 16384 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 128589 128589 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
</code></pre>
<ul>
<li>I will try to add <code>ulimit -v unlimited</code> to the Catalina startup script and check the output of the limits to see if it&rsquo;s different in practice, as some wisdom on Stack Overflow says this solves the Solr core issues and I&rsquo;ve superstitiously tried it various times in the past
<ul>
<li>The result is the same before and after, so <em>adding the ulimit directly is unneccessary</em> (whether or not unlimited address space is useful or not is another question)</li>
</ul></li>
<li>For now I will just stop Tomcat, delete Solr locks, then start Tomcat again:</li>
</ul>
<pre><code># systemctl stop tomcat7
# find /home/cgspace.cgiar.org/solr/ -iname &quot;*.lock&quot; -delete
# systemctl start tomcat7
</code></pre>
<ul>
<li>After restarting I confirmed that all Solr statistics cores were loaded successfully&hellip;</li>
<li>Another avenue might be to look at point releases in Solr 4.10.x, as we&rsquo;re running 4.10.2 and they released 4.10.3 and 4.10.4 back in 2014 or 2015
<ul>
<li>I see several issues regarding locks and IndexWriter that were fixed in Solr and Lucene 4.10.3 and 4.10.4&hellip;</li>
</ul></li>
</ul> </ul>
<!-- vim: set sw=2 ts=2: --> <!-- vim: set sw=2 ts=2: -->

2
docs/robots.txt
View File

@ -45,7 +45,7 @@ Disallow: /cgspace-notes/2015-12/
Disallow: /cgspace-notes/2015-11/ Disallow: /cgspace-notes/2015-11/
Disallow: /cgspace-notes/ Disallow: /cgspace-notes/
Disallow: /cgspace-notes/categories/ Disallow: /cgspace-notes/categories/
Disallow: /cgspace-notes/categories/notes/
Disallow: /cgspace-notes/tags/notes/ Disallow: /cgspace-notes/tags/notes/
Disallow: /cgspace-notes/categories/notes/
Disallow: /cgspace-notes/posts/ Disallow: /cgspace-notes/posts/
Disallow: /cgspace-notes/tags/ Disallow: /cgspace-notes/tags/

20
docs/sitemap.xml
View File

@ -4,7 +4,7 @@
<url> <url>
<loc>https://alanorth.github.io/cgspace-notes/2019-03/</loc> <loc>https://alanorth.github.io/cgspace-notes/2019-03/</loc>
<lastmod>2019-03-18T21:55:08+02:00</lastmod> <lastmod>2019-03-19T11:33:51+02:00</lastmod>
</url> </url>
<url> <url>
@ -214,7 +214,7 @@
<url> <url>
<loc>https://alanorth.github.io/cgspace-notes/</loc> <loc>https://alanorth.github.io/cgspace-notes/</loc>
<lastmod>2019-03-18T21:55:08+02:00</lastmod> <lastmod>2019-03-19T11:33:51+02:00</lastmod>
<priority>0</priority> <priority>0</priority>
</url> </url>
@ -223,27 +223,27 @@
<priority>0</priority> <priority>0</priority>
</url> </url>
<url>
<loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
<lastmod>2019-03-19T11:33:51+02:00</lastmod>
<priority>0</priority>
</url>
<url> <url>
<loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc> <loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
<lastmod>2018-03-09T22:10:33+02:00</lastmod> <lastmod>2018-03-09T22:10:33+02:00</lastmod>
<priority>0</priority> <priority>0</priority>
</url> </url>
<url>
<loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
<lastmod>2019-03-18T21:55:08+02:00</lastmod>
<priority>0</priority>
</url>
<url> <url>
<loc>https://alanorth.github.io/cgspace-notes/posts/</loc> <loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
<lastmod>2019-03-18T21:55:08+02:00</lastmod> <lastmod>2019-03-19T11:33:51+02:00</lastmod>
<priority>0</priority> <priority>0</priority>
</url> </url>
<url> <url>
<loc>https://alanorth.github.io/cgspace-notes/tags/</loc> <loc>https://alanorth.github.io/cgspace-notes/tags/</loc>
<lastmod>2019-03-18T21:55:08+02:00</lastmod> <lastmod>2019-03-19T11:33:51+02:00</lastmod>
<priority>0</priority> <priority>0</priority>
</url> </url>