mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2025-01-27 05:49:12 +01:00
Add notes for 2020-01-27
This commit is contained in:
@ -8,7 +8,7 @@
|
||||
<meta property="og:title" content="April, 2018" />
|
||||
<meta property="og:description" content="2018-04-01
|
||||
|
||||
I tried to test something on DSpace Test but noticed that it's down since god knows when
|
||||
I tried to test something on DSpace Test but noticed that it’s down since god knows when
|
||||
Catalina logs at least show some memory errors yesterday:
|
||||
" />
|
||||
<meta property="og:type" content="article" />
|
||||
@ -20,10 +20,10 @@ Catalina logs at least show some memory errors yesterday:
|
||||
<meta name="twitter:title" content="April, 2018"/>
|
||||
<meta name="twitter:description" content="2018-04-01
|
||||
|
||||
I tried to test something on DSpace Test but noticed that it's down since god knows when
|
||||
I tried to test something on DSpace Test but noticed that it’s down since god knows when
|
||||
Catalina logs at least show some memory errors yesterday:
|
||||
"/>
|
||||
<meta name="generator" content="Hugo 0.62.2" />
|
||||
<meta name="generator" content="Hugo 0.63.1" />
|
||||
|
||||
|
||||
|
||||
@ -53,7 +53,7 @@ Catalina logs at least show some memory errors yesterday:
|
||||
|
||||
<!-- combined, minified CSS -->
|
||||
|
||||
<link href="https://alanorth.github.io/cgspace-notes/css/style.a20c1a4367639632cdb341d23c27ca44fedcc75b0f8b3cbea6203010da153d3c.css" rel="stylesheet" integrity="sha256-ogwaQ2djljLNs0HSPCfKRP7cx1sPizy+piAwENoVPTw=" crossorigin="anonymous">
|
||||
<link href="https://alanorth.github.io/cgspace-notes/css/style.23e2c3298bcc8c1136c19aba330c211ec94c36f7c4454ea15cf4d3548370042a.css" rel="stylesheet" integrity="sha256-I+LDKYvMjBE2wZq6MwwhHslMNvfERU6hXPTTVINwBCo=" crossorigin="anonymous">
|
||||
|
||||
|
||||
<!-- RSS 2.0 feed -->
|
||||
@ -100,14 +100,14 @@ Catalina logs at least show some memory errors yesterday:
|
||||
<header>
|
||||
<h2 class="blog-post-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/2018-04/">April, 2018</a></h2>
|
||||
<p class="blog-post-meta"><time datetime="2018-04-01T16:13:54+02:00">Sun Apr 01, 2018</time> by Alan Orth in
|
||||
<i class="fa fa-folder" aria-hidden="true"></i> <a href="/cgspace-notes/categories/notes" rel="category tag">Notes</a>
|
||||
<span class="fas fa-folder" aria-hidden="true"></span> <a href="/cgspace-notes/categories/notes" rel="category tag">Notes</a>
|
||||
|
||||
|
||||
</p>
|
||||
</header>
|
||||
<h2 id="2018-04-01">2018-04-01</h2>
|
||||
<ul>
|
||||
<li>I tried to test something on DSpace Test but noticed that it's down since god knows when</li>
|
||||
<li>I tried to test something on DSpace Test but noticed that it’s down since god knows when</li>
|
||||
<li>Catalina logs at least show some memory errors yesterday:</li>
|
||||
</ul>
|
||||
<pre><code>Mar 31, 2018 10:26:42 PM org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor run
|
||||
@ -124,7 +124,7 @@ Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]
|
||||
</ul>
|
||||
<h2 id="2018-04-04">2018-04-04</h2>
|
||||
<ul>
|
||||
<li>Peter noticed that there were still some old CRP names on CGSpace, because I hadn't forced the Discovery index to be updated after I fixed the others last week</li>
|
||||
<li>Peter noticed that there were still some old CRP names on CGSpace, because I hadn’t forced the Discovery index to be updated after I fixed the others last week</li>
|
||||
<li>For completeness I re-ran the CRP corrections on CGSpace:</li>
|
||||
</ul>
|
||||
<pre><code>$ ./fix-metadata-values.py -i /tmp/Correct-21-CRPs-2018-03-16.csv -f cg.contributor.crp -t correct -m 230 -db dspace -u dspace -p 'fuuu'
|
||||
@ -139,7 +139,7 @@ real 76m13.841s
|
||||
user 8m22.960s
|
||||
sys 2m2.498s
|
||||
</code></pre><ul>
|
||||
<li>Elizabeth from CIAT emailed to ask if I could help her by adding ORCID identifiers to all of Joseph Tohme's items</li>
|
||||
<li>Elizabeth from CIAT emailed to ask if I could help her by adding ORCID identifiers to all of Joseph Tohme’s items</li>
|
||||
<li>I used my <a href="https://gist.githubusercontent.com/alanorth/a49d85cd9c5dea89cddbe809813a7050/raw/f67b6e45a9a940732882ae4bb26897a9b245ef31/add-orcid-identifiers-csv.py">add-orcid-identifiers-csv.py</a> script:</li>
|
||||
</ul>
|
||||
<pre><code>$ ./add-orcid-identifiers-csv.py -i /tmp/jtohme-2018-04-04.csv -db dspace -u dspace -p 'fuuu'
|
||||
@ -165,13 +165,13 @@ $ git rebase -i dspace-5.8
|
||||
<li>DS-3583 Usage of correct Collection Array (#1731) (upstream commit on dspace-5_x: c8f62e6f496fa86846bfa6bcf2d16811087d9761)</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>… but somehow git knew, and didn't include them in my interactive rebase!</li>
|
||||
<li>… but somehow git knew, and didn’t include them in my interactive rebase!</li>
|
||||
<li>I need to send this branch to Atmire and also arrange payment (see <a href="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=560">ticket #560</a> in their tracker)</li>
|
||||
<li>Fix Sisay's SSH access to the new DSpace Test server (linode19)</li>
|
||||
<li>Fix Sisay’s SSH access to the new DSpace Test server (linode19)</li>
|
||||
</ul>
|
||||
<h2 id="2018-04-05">2018-04-05</h2>
|
||||
<ul>
|
||||
<li>Fix Sisay's sudo access on the new DSpace Test server (linode19)</li>
|
||||
<li>Fix Sisay’s sudo access on the new DSpace Test server (linode19)</li>
|
||||
<li>The reindexing process on DSpace Test took <em>forever</em> yesterday:</li>
|
||||
</ul>
|
||||
<pre><code>$ time schedtool -D -e ionice -c2 -n7 nice -n19 dspace index-discovery -b
|
||||
@ -220,15 +220,15 @@ sys 2m52.585s
|
||||
<pre><code>$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=95.108.181.88' dspace.log.2018-04-10
|
||||
4363
|
||||
</code></pre><ul>
|
||||
<li>70.32.83.92 appears to be some harvester we've seen before, but on a new IP</li>
|
||||
<li>70.32.83.92 appears to be some harvester we’ve seen before, but on a new IP</li>
|
||||
<li>They are not creating new Tomcat sessions so there is no problem there</li>
|
||||
<li>178.154.200.38 also appears to be Yandex, and is also creating many Tomcat sessions:</li>
|
||||
</ul>
|
||||
<pre><code>$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=178.154.200.38' dspace.log.2018-04-10
|
||||
3982
|
||||
</code></pre><ul>
|
||||
<li>I'm not sure why Yandex creates so many Tomcat sessions, as its user agent should match the Crawler Session Manager valve</li>
|
||||
<li>Let's try a manual request with and without their user agent:</li>
|
||||
<li>I’m not sure why Yandex creates so many Tomcat sessions, as its user agent should match the Crawler Session Manager valve</li>
|
||||
<li>Let’s try a manual request with and without their user agent:</li>
|
||||
</ul>
|
||||
<pre><code>$ http --print Hh https://cgspace.cgiar.org/bitstream/handle/10568/21794/ILRI_logo_usage.jpg.jpg 'User-Agent:Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)'
|
||||
GET /bitstream/handle/10568/21794/ILRI_logo_usage.jpg.jpg HTTP/1.1
|
||||
@ -312,7 +312,7 @@ UPDATE 1
|
||||
2115
|
||||
</code></pre><ul>
|
||||
<li>Apparently from these stacktraces we should be able to see which code is not closing connections properly</li>
|
||||
<li>Here's a pretty good overview of days where we had database issues recently:</li>
|
||||
<li>Here’s a pretty good overview of days where we had database issues recently:</li>
|
||||
</ul>
|
||||
<pre><code># zcat /var/log/tomcat7/catalina.out.[1-9].gz | grep 'org.apache.tomcat.jdbc.pool.ConnectionPool abandon' | awk '{print $1,$2, $3}' | sort | uniq -c | sort -n
|
||||
1 Feb 18, 2018
|
||||
@ -337,9 +337,9 @@ UPDATE 1
|
||||
<li>In Tomcat 8.5 the <code>removeAbandoned</code> property has been split into two: <code>removeAbandonedOnBorrow</code> and <code>removeAbandonedOnMaintenance</code></li>
|
||||
<li>See: <a href="https://tomcat.apache.org/tomcat-8.5-doc/jndi-datasource-examples-howto.html#Database_Connection_Pool_(DBCP_2)_Configurations">https://tomcat.apache.org/tomcat-8.5-doc/jndi-datasource-examples-howto.html#Database_Connection_Pool_(DBCP_2)_Configurations</a></li>
|
||||
<li>I assume we want <code>removeAbandonedOnBorrow</code> and make updates to the Tomcat 8 templates in Ansible</li>
|
||||
<li>After reading more documentation I see that Tomcat 8.5's default DBCP seems to now be Commons DBCP2 instead of Tomcat DBCP</li>
|
||||
<li>It can be overridden in Tomcat's <em>server.xml</em> by setting <code>factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"</code> in the <code><Resource></code></li>
|
||||
<li>I think we should use this default, so we'll need to remove some other settings that are specific to Tomcat's DBCP like <code>jdbcInterceptors</code> and <code>abandonWhenPercentageFull</code></li>
|
||||
<li>After reading more documentation I see that Tomcat 8.5’s default DBCP seems to now be Commons DBCP2 instead of Tomcat DBCP</li>
|
||||
<li>It can be overridden in Tomcat’s <em>server.xml</em> by setting <code>factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"</code> in the <code><Resource></code></li>
|
||||
<li>I think we should use this default, so we’ll need to remove some other settings that are specific to Tomcat’s DBCP like <code>jdbcInterceptors</code> and <code>abandonWhenPercentageFull</code></li>
|
||||
<li>Merge the changes adding ORCID identifier to advanced search and Atmire Listings and Reports (<a href="https://github.com/ilri/DSpace/pull/371">#371</a>)</li>
|
||||
<li>Fix one more issue of missing XMLUI strings (for CRP subject when clicking “view more” in the Discovery sidebar)</li>
|
||||
<li>I told Udana to fix the citation and abstract of the one item, and to correct the <code>dc.language.iso</code> for the five Spanish items in his Book Chapters collection</li>
|
||||
@ -377,7 +377,7 @@ java.lang.NullPointerException
|
||||
<li>I see the same error on DSpace Test so this is definitely a problem</li>
|
||||
<li>After disabling the authority consumer I no longer see the error</li>
|
||||
<li>I merged a pull request to the <code>5_x-prod</code> branch to clean that up (<a href="https://github.com/ilri/DSpace/pull/372">#372</a>)</li>
|
||||
<li>File a ticket on DSpace's Jira for the <code>target="_blank"</code> security and performance issue (<a href="https://jira.duraspace.org/browse/DS-3891">DS-3891</a>)</li>
|
||||
<li>File a ticket on DSpace’s Jira for the <code>target="_blank"</code> security and performance issue (<a href="https://jira.duraspace.org/browse/DS-3891">DS-3891</a>)</li>
|
||||
<li>I re-deployed DSpace Test (linode19) and was surprised by how long it took the ant update to complete:</li>
|
||||
</ul>
|
||||
<pre><code>BUILD SUCCESSFUL
|
||||
@ -394,7 +394,7 @@ Total time: 4 minutes 12 seconds
|
||||
<ul>
|
||||
<li>IWMI people are asking about building a search query that outputs RSS for their reports</li>
|
||||
<li>They want the same results as this Discovery query: <a href="https://cgspace.cgiar.org/discover?filtertype_1=dateAccessioned&filter_relational_operator_1=contains&filter_1=2018&submit_apply_filter=&query=&scope=10568%2F16814&rpp=100&sort_by=dc.date.issued_dt&order=desc">https://cgspace.cgiar.org/discover?filtertype_1=dateAccessioned&filter_relational_operator_1=contains&filter_1=2018&submit_apply_filter=&query=&scope=10568%2F16814&rpp=100&sort_by=dc.date.issued_dt&order=desc</a></li>
|
||||
<li>They will need to use OpenSearch, but I can't remember all the parameters</li>
|
||||
<li>They will need to use OpenSearch, but I can’t remember all the parameters</li>
|
||||
<li>Apparently search sort options for OpenSearch are in <code>dspace.cfg</code>:</li>
|
||||
</ul>
|
||||
<pre><code>webui.itemlist.sort-option.1 = title:dc.title:title
|
||||
@ -410,15 +410,15 @@ webui.itemlist.sort-option.4 = type:dc.type:text
|
||||
<li>For example, set <code>rpp=1</code> and then check the results for <code>start</code> values of 0, 1, and 2 and they are all the same!</li>
|
||||
<li>If I have time I will check if this behavior persists on DSpace 6.x on the official DSpace demo and file a bug</li>
|
||||
<li>Also, the DSpace Manual as of 5.x has very poor documentation for OpenSearch</li>
|
||||
<li>They don't tell you to use Discovery search filters in the <code>query</code> (with format <code>query=dateIssued:2018</code>)</li>
|
||||
<li>They don't tell you that the sort options are actually defined in <code>dspace.cfg</code> (ie, you need to use <code>2</code> instead of <code>dc.date.issued_dt</code>)</li>
|
||||
<li>They don’t tell you to use Discovery search filters in the <code>query</code> (with format <code>query=dateIssued:2018</code>)</li>
|
||||
<li>They don’t tell you that the sort options are actually defined in <code>dspace.cfg</code> (ie, you need to use <code>2</code> instead of <code>dc.date.issued_dt</code>)</li>
|
||||
<li>They are missing the <code>order</code> parameter (ASC vs DESC)</li>
|
||||
<li>I notice that DSpace Test has crashed again, due to memory:</li>
|
||||
</ul>
|
||||
<pre><code># grep -c 'java.lang.OutOfMemoryError: Java heap space' /var/log/tomcat7/catalina.out
|
||||
178
|
||||
</code></pre><ul>
|
||||
<li>I will increase the JVM heap size from 5120M to 6144M, though we don't have much room left to grow as DSpace Test (linode19) is using a smaller instance size than CGSpace</li>
|
||||
<li>I will increase the JVM heap size from 5120M to 6144M, though we don’t have much room left to grow as DSpace Test (linode19) is using a smaller instance size than CGSpace</li>
|
||||
<li>Gabriela from CIP asked if I could send her a list of all CIP authors so she can do some replacements on the name formats</li>
|
||||
<li>I got a list of all the CIP collections manually and use the same query that I used in <a href="/cgspace-notes/2017-08">August, 2017</a>:</li>
|
||||
</ul>
|
||||
@ -445,8 +445,8 @@ sys 2m2.687s
|
||||
</ul>
|
||||
<h2 id="2018-04-20">2018-04-20</h2>
|
||||
<ul>
|
||||
<li>Gabriela from CIP emailed to say that CGSpace was returning a white page, but I haven't seen any emails from UptimeRobot</li>
|
||||
<li>I confirm that it's just giving a white page around 4:16</li>
|
||||
<li>Gabriela from CIP emailed to say that CGSpace was returning a white page, but I haven’t seen any emails from UptimeRobot</li>
|
||||
<li>I confirm that it’s just giving a white page around 4:16</li>
|
||||
<li>The DSpace logs show that there are no database connections:</li>
|
||||
</ul>
|
||||
<pre><code>org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-715] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:250; busy:18; idle:0; lastwait:5000].
|
||||
@ -456,7 +456,7 @@ sys 2m2.687s
|
||||
<pre><code># grep -c 'org.apache.tomcat.jdbc.pool.PoolExhaustedException' /home/cgspace.cgiar.org/log/dspace.log.2018-04-20
|
||||
32147
|
||||
</code></pre><ul>
|
||||
<li>I can't even log into PostgreSQL as the <code>postgres</code> user, WTF?</li>
|
||||
<li>I can’t even log into PostgreSQL as the <code>postgres</code> user, WTF?</li>
|
||||
</ul>
|
||||
<pre><code>$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
|
||||
^C
|
||||
@ -475,7 +475,7 @@ sys 2m2.687s
|
||||
4325 70.32.83.92
|
||||
10718 45.5.184.2
|
||||
</code></pre><ul>
|
||||
<li>It doesn't even seem like there is a lot of traffic compared to the previous days:</li>
|
||||
<li>It doesn’t even seem like there is a lot of traffic compared to the previous days:</li>
|
||||
</ul>
|
||||
<pre><code># zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "20/Apr/2018" | wc -l
|
||||
74931
|
||||
@ -485,9 +485,9 @@ sys 2m2.687s
|
||||
93459
|
||||
</code></pre><ul>
|
||||
<li>I tried to restart Tomcat but <code>systemctl</code> hangs</li>
|
||||
<li>I tried to reboot the server from the command line but after a few minutes it didn't come back up</li>
|
||||
<li>I tried to reboot the server from the command line but after a few minutes it didn’t come back up</li>
|
||||
<li>Looking at the Linode console I see that it is stuck trying to shut down</li>
|
||||
<li>Even “Reboot” via Linode console doesn't work!</li>
|
||||
<li>Even “Reboot” via Linode console doesn’t work!</li>
|
||||
<li>After shutting it down a few times via the Linode console it finally rebooted</li>
|
||||
<li>Everything is back but I have no idea what caused this—I suspect something with the hosting provider</li>
|
||||
<li>Also super weird, the last entry in the DSpace log file is from <code>2018-04-20 16:35:09</code>, and then immediately it goes to <code>2018-04-20 19:15:04</code> (three hours later!):</li>
|
||||
@ -518,13 +518,13 @@ org.apache.tomcat.jdbc.pool.PoolExhaustedException: [localhost-startStop-2] Time
|
||||
</ul>
|
||||
<h2 id="2018-04-24">2018-04-24</h2>
|
||||
<ul>
|
||||
<li>Testing my Ansible playbooks with a clean and updated installation of Ubuntu 18.04 and I fixed some issues that I hadn't run into a few weeks ago</li>
|
||||
<li>Testing my Ansible playbooks with a clean and updated installation of Ubuntu 18.04 and I fixed some issues that I hadn’t run into a few weeks ago</li>
|
||||
<li>There seems to be a new issue with Java dependencies, though</li>
|
||||
<li>The <code>default-jre</code> package is going to be Java 10 on Ubuntu 18.04, but I want to use <code>openjdk-8-jre-headless</code> (well, the JDK actually, but it uses this JRE)</li>
|
||||
<li>Tomcat and Ant are fine with Java 8, but the <code>maven</code> package wants to pull in Java 10 for some reason</li>
|
||||
<li>Looking closer, I see that <code>maven</code> depends on <code>java7-runtime-headless</code>, which is indeed provided by <code>openjdk-8-jre-headless</code></li>
|
||||
<li>So it must be one of Maven's dependencies…</li>
|
||||
<li>I will watch it for a few days because it could be an issue that will be resolved before Ubuntu 18.04's release</li>
|
||||
<li>So it must be one of Maven’s dependencies…</li>
|
||||
<li>I will watch it for a few days because it could be an issue that will be resolved before Ubuntu 18.04’s release</li>
|
||||
<li>Otherwise I will post a bug to the ubuntu-release mailing list</li>
|
||||
<li>Looks like the only way to fix this is to install <code>openjdk-8-jdk-headless</code> before (so it pulls in the JRE) in a separate transaction, or to manually install <code>openjdk-8-jre-headless</code> in the same apt transaction as <code>maven</code></li>
|
||||
<li>Also, I started porting PostgreSQL 9.6 into the Ansible infrastructure scripts</li>
|
||||
@ -534,12 +534,12 @@ org.apache.tomcat.jdbc.pool.PoolExhaustedException: [localhost-startStop-2] Time
|
||||
<ul>
|
||||
<li>Still testing the <a href="https://github.com/ilri/rmg-ansible-public">Ansible infrastructure playbooks</a> for Ubuntu 18.04, Tomcat 8.5, and PostgreSQL 9.6</li>
|
||||
<li>One other new thing I notice is that PostgreSQL 9.6 no longer uses <code>createuser</code> and <code>nocreateuser</code>, as those have actually meant <code>superuser</code> and <code>nosuperuser</code> and have been deprecated for <em>ten years</em></li>
|
||||
<li>So for my notes, when I'm importing a CGSpace database dump I need to amend my notes to give super user permission to a user, rather than create user:</li>
|
||||
<li>So for my notes, when I’m importing a CGSpace database dump I need to amend my notes to give super user permission to a user, rather than create user:</li>
|
||||
</ul>
|
||||
<pre><code>$ psql dspacetest -c 'alter user dspacetest superuser;'
|
||||
$ pg_restore -O -U dspacetest -d dspacetest -W -h localhost /tmp/dspace_2018-04-18.backup
|
||||
</code></pre><ul>
|
||||
<li>There's another issue with Tomcat in Ubuntu 18.04:</li>
|
||||
<li>There’s another issue with Tomcat in Ubuntu 18.04:</li>
|
||||
</ul>
|
||||
<pre><code>25-Apr-2018 13:26:21.493 SEVERE [http-nio-127.0.0.1-8443-exec-1] org.apache.coyote.AbstractProtocol$ConnectionHandler.process Error reading request, ignored
|
||||
java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
|
||||
@ -554,13 +554,13 @@ $ pg_restore -O -U dspacetest -d dspacetest -W -h localhost /tmp/dspace_2018-04-
|
||||
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
|
||||
at java.lang.Thread.run(Thread.java:748)
|
||||
</code></pre><ul>
|
||||
<li>There's a <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=895866">Debian bug about this from a few weeks ago</a></li>
|
||||
<li>Apparently Tomcat was compiled with Java 9, so doesn't work with Java 8</li>
|
||||
<li>There’s a <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=895866">Debian bug about this from a few weeks ago</a></li>
|
||||
<li>Apparently Tomcat was compiled with Java 9, so doesn’t work with Java 8</li>
|
||||
</ul>
|
||||
<h2 id="2018-04-29">2018-04-29</h2>
|
||||
<ul>
|
||||
<li>DSpace Test crashed again, looks like memory issues again</li>
|
||||
<li>JVM heap size was last increased to 6144m but the system only has 8GB total so there's not much we can do here other than get a bigger Linode instance or remove the massive Solr Statistics data</li>
|
||||
<li>JVM heap size was last increased to 6144m but the system only has 8GB total so there’s not much we can do here other than get a bigger Linode instance or remove the massive Solr Statistics data</li>
|
||||
</ul>
|
||||
<h2 id="2018-04-30">2018-04-30</h2>
|
||||
<ul>
|
||||
|
||||
Reference in New Issue
Block a user