<li>I tried to test something on DSpace Test but noticed that it’s down since god knows when</li>
<li>Catalina logs at least show some memory errors yesterday:</li>
</ul>
<pretabindex="0"><code>Mar 31, 2018 10:26:42 PM org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor run
SEVERE: Unexpected death of background thread ContainerBackgroundProcessor[StandardEngine[Catalina]]
java.lang.OutOfMemoryError: Java heap space
Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" java.lang.OutOfMemoryError: Java heap space
</code></pre><ul>
<li>So this is getting super annoying</li>
<li>I ran all system updates on DSpace Test and rebooted it</li>
<li>For some reason Listings and Reports is not giving any results for any queries now…</li>
<li>I posted a message on Yammer to ask if people are using the Duplicate Check step from the Metadata Quality Module</li>
<li>Help Lili Szilagyi with a question about statistics on some CCAFS items</li>
</ul>
<h2id="2018-04-04">2018-04-04</h2>
<ul>
<li>Peter noticed that there were still some old CRP names on CGSpace, because I hadn’t forced the Discovery index to be updated after I fixed the others last week</li>
<li>For completeness I re-ran the CRP corrections on CGSpace:</li>
<li>Elizabeth from CIAT emailed to ask if I could help her by adding ORCID identifiers to all of Joseph Tohme’s items</li>
<li>I used my <ahref="https://gist.githubusercontent.com/alanorth/a49d85cd9c5dea89cddbe809813a7050/raw/f67b6e45a9a940732882ae4bb26897a9b245ef31/add-orcid-identifiers-csv.py">add-orcid-identifiers-csv.py</a> script:</li>
<li>I was prepared to skip some commits that I had cherry picked from the upstream <code>dspace-5_x</code> branch when we did the DSpace 5.5 upgrade (see notes on 2016-10-19 and 2017-12-17):
<ul>
<li>[DS-3246] Improve cleanup in recyclable components (upstream commit on dspace-5_x: 9f0f5940e7921765c6a22e85337331656b18a403)</li>
<li>[DS-3250] applying patch provided by Atmire (upstream commit on dspace-5_x: c6fda557f731dbc200d7d58b8b61563f86fe6d06)</li>
<li>bump up to latest minor pdfbox version (upstream commit on dspace-5_x: b5330b78153b2052ed3dc2fd65917ccdbfcc0439)</li>
<li>DS-3583 Usage of correct Collection Array (#1731) (upstream commit on dspace-5_x: c8f62e6f496fa86846bfa6bcf2d16811087d9761)</li>
</ul>
</li>
<li>… but somehow git knew, and didn’t include them in my interactive rebase!</li>
<li>I need to send this branch to Atmire and also arrange payment (see <ahref="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=560">ticket #560</a> in their tracker)</li>
<li>Fix Sisay’s SSH access to the new DSpace Test server (linode19)</li>
</ul>
<h2id="2018-04-05">2018-04-05</h2>
<ul>
<li>Fix Sisay’s sudo access on the new DSpace Test server (linode19)</li>
<li>The reindexing process on DSpace Test took <em>forever</em> yesterday:</li>
<li>I ran all system updates on DSpace Test and rebooted it</li>
<li>Proof some records on DSpace Test for Udana from IWMI</li>
<li>He has done better with the small syntax and consistency issues but then there are larger concerns with not linking to DOIs, copying titles incorrectly, etc</li>
</ul>
<h2id="2018-04-10">2018-04-10</h2>
<ul>
<li>I got a notice that CGSpace CPU usage was very high this morning</li>
<li>Looking at the nginx logs, here are the top users today so far:</li>
<li>In Tomcat 8.5 the <code>removeAbandoned</code> property has been split into two: <code>removeAbandonedOnBorrow</code> and <code>removeAbandonedOnMaintenance</code></li>
<li>I assume we want <code>removeAbandonedOnBorrow</code> and make updates to the Tomcat 8 templates in Ansible</li>
<li>After reading more documentation I see that Tomcat 8.5’s default DBCP seems to now be Commons DBCP2 instead of Tomcat DBCP</li>
<li>It can be overridden in Tomcat’s <em>server.xml</em> by setting <code>factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"</code> in the <code><Resource></code></li>
<li>I think we should use this default, so we’ll need to remove some other settings that are specific to Tomcat’s DBCP like <code>jdbcInterceptors</code> and <code>abandonWhenPercentageFull</code></li>
<li>Merge the changes adding ORCID identifier to advanced search and Atmire Listings and Reports (<ahref="https://github.com/ilri/DSpace/pull/371">#371</a>)</li>
<li>Fix one more issue of missing XMLUI strings (for CRP subject when clicking “view more” in the Discovery sidebar)</li>
<li>I told Udana to fix the citation and abstract of the one item, and to correct the <code>dc.language.iso</code> for the five Spanish items in his Book Chapters collection</li>
<li>Then we can import the records to CGSpace</li>
</ul>
<h2id="2018-04-11">2018-04-11</h2>
<ul>
<li>DSpace Test (linode19) crashed again some time since yesterday:</li>
<li>I ran all system updates and rebooted the server</li>
</ul>
<h2id="2018-04-12">2018-04-12</h2>
<ul>
<li>I caught wind of an interesting XMLUI performance optimization coming in DSpace 6.3: <ahref="https://jira.duraspace.org/browse/DS-3883">https://jira.duraspace.org/browse/DS-3883</a></li>
<li>I asked for it to be ported to DSpace 5.x</li>
</ul>
<h2id="2018-04-13">2018-04-13</h2>
<ul>
<li>Add <code>PII-LAM_CSAGender</code> to CCAFS Phase II project tags in <code>input-forms.xml</code></li>
</ul>
<h2id="2018-04-15">2018-04-15</h2>
<ul>
<li>While testing an XMLUI patch for <ahref="https://jira.duraspace.org/browse/DS-3883">DS-3883</a> I noticed that there is still some remaining Authority / Solr configuration left that we need to remove:</li>
</ul>
<pretabindex="0"><code>2018-04-14 18:55:25,841 ERROR org.dspace.authority.AuthoritySolrServiceImpl @ Authority solr is not correctly configured, check "solr.authority.server" property in the dspace.cfg
java.lang.NullPointerException
</code></pre><ul>
<li>I assume we need to remove <code>authority</code> from the consumers in <code>dspace/config/dspace.cfg</code>:</li>
<li>I see the same error on DSpace Test so this is definitely a problem</li>
<li>After disabling the authority consumer I no longer see the error</li>
<li>I merged a pull request to the <code>5_x-prod</code> branch to clean that up (<ahref="https://github.com/ilri/DSpace/pull/372">#372</a>)</li>
<li>File a ticket on DSpace’s Jira for the <code>target="_blank"</code> security and performance issue (<ahref="https://jira.duraspace.org/browse/DS-3891">DS-3891</a>)</li>
<li>I re-deployed DSpace Test (linode19) and was surprised by how long it took the ant update to complete:</li>
</ul>
<pretabindex="0"><code>BUILD SUCCESSFUL
Total time: 4 minutes 12 seconds
</code></pre><ul>
<li>The Linode block storage is much slower than the instance storage</li>
<li>I ran all system updates and rebooted DSpace Test (linode19)</li>
</ul>
<h2id="2018-04-16">2018-04-16</h2>
<ul>
<li>Communicate with Bioversity about their project to migrate their e-Library (Typo3) and Sci-lit databases to CGSpace</li>
</ul>
<h2id="2018-04-18">2018-04-18</h2>
<ul>
<li>IWMI people are asking about building a search query that outputs RSS for their reports</li>
<li>They want the same results as this Discovery query: <ahref="https://cgspace.cgiar.org/discover?filtertype_1=dateAccessioned&filter_relational_operator_1=contains&filter_1=2018&submit_apply_filter=&query=&scope=10568%2F16814&rpp=100&sort_by=dc.date.issued_dt&order=desc">https://cgspace.cgiar.org/discover?filtertype_1=dateAccessioned&filter_relational_operator_1=contains&filter_1=2018&submit_apply_filter=&query=&scope=10568%2F16814&rpp=100&sort_by=dc.date.issued_dt&order=desc</a></li>
<li>They will need to use OpenSearch, but I can’t remember all the parameters</li>
<li>Apparently search sort options for OpenSearch are in <code>dspace.cfg</code>:</li>
<li>They want items by issue date, so we need to use sort option 2</li>
<li>According to the DSpace Manual there are only the following parameters to OpenSearch: format, scope, rpp, start, and sort_by</li>
<li>The OpenSearch <code>query</code> parameter expects a Discovery search filter that is defined in <code>dspace/config/spring/api/discovery.xml</code></li>
<li>So for IWMI they should be able to use something like this: <ahref="https://cgspace.cgiar.org/open-search/discover?query=dateIssued:2018&scope=10568/16814&sort_by=2&order=DESC&format=rss">https://cgspace.cgiar.org/open-search/discover?query=dateIssued:2018&scope=10568/16814&sort_by=2&order=DESC&format=rss</a></li>
<li>There are also <code>rpp</code> (results per page) and <code>start</code> parameters but in my testing now on DSpace 5.5 they behave very strangely</li>
<li>For example, set <code>rpp=1</code> and then check the results for <code>start</code> values of 0, 1, and 2 and they are all the same!</li>
<li>If I have time I will check if this behavior persists on DSpace 6.x on the official DSpace demo and file a bug</li>
<li>Also, the DSpace Manual as of 5.x has very poor documentation for OpenSearch</li>
<li>They don’t tell you to use Discovery search filters in the <code>query</code> (with format <code>query=dateIssued:2018</code>)</li>
<li>They don’t tell you that the sort options are actually defined in <code>dspace.cfg</code> (ie, you need to use <code>2</code> instead of <code>dc.date.issued_dt</code>)</li>
<li>They are missing the <code>order</code> parameter (ASC vs DESC)</li>
<li>I notice that DSpace Test has crashed again, due to memory:</li>
<li>I will increase the JVM heap size from 5120M to 6144M, though we don’t have much room left to grow as DSpace Test (linode19) is using a smaller instance size than CGSpace</li>
<li>Gabriela from CIP asked if I could send her a list of all CIP authors so she can do some replacements on the name formats</li>
<li>I got a list of all the CIP collections manually and use the same query that I used in <ahref="/cgspace-notes/2017-08">August, 2017</a>:</li>
</ul>
<pretabindex="0"><code>dspace#= \copy (select distinct text_value, count(*) from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 AND resource_id IN (select item_id from collection2item where collection_id IN (select resource_id from handle where handle in ('10568/89347', '10568/88229', '10568/53086', '10568/53085', '10568/69069', '10568/53087', '10568/53088', '10568/53089', '10568/53090', '10568/53091', '10568/53092', '10568/70150', '10568/53093', '10568/64874', '10568/53094'))) group by text_value order by count desc) to /tmp/cip-authors.csv with csv;
</code></pre><h2id="2018-04-19">2018-04-19</h2>
<ul>
<li>Run updates on DSpace Test (linode19) and reboot the server</li>
<li>Also try deploying updated GeoLite database during ant update while re-deploying code:</li>
</ul>
<pretabindex="0"><code>$ ant update update_geolite clean_backups
</code></pre><ul>
<li>I also re-deployed CGSpace (linode18) to make the ORCID search, authority cleanup, CCAFS project tag <code>PII-LAM_CSAGender</code> live</li>
<li>When re-deploying I also updated the GeoLite databases so I hope the country stats become more accurate…</li>
<li>After re-deployment I ran all system updates on the server and rebooted it</li>
<li>After the reboot I forced a reïndexing of the Discovery to populate the new ORCID index:</li>
<li>This time is with about 70,000 items in the repository</li>
</ul>
<h2id="2018-04-20">2018-04-20</h2>
<ul>
<li>Gabriela from CIP emailed to say that CGSpace was returning a white page, but I haven’t seen any emails from UptimeRobot</li>
<li>I confirm that it’s just giving a white page around 4:16</li>
<li>The DSpace logs show that there are no database connections:</li>
</ul>
<pretabindex="0"><code>org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-715] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:250; busy:18; idle:0; lastwait:5000].
</code></pre><ul>
<li>And there have been shit tons of errors in the last (starting only 20 minutes ago luckily):</li>
<li>I tried to restart Tomcat but <code>systemctl</code> hangs</li>
<li>I tried to reboot the server from the command line but after a few minutes it didn’t come back up</li>
<li>Looking at the Linode console I see that it is stuck trying to shut down</li>
<li>Even “Reboot” via Linode console doesn’t work!</li>
<li>After shutting it down a few times via the Linode console it finally rebooted</li>
<li>Everything is back but I have no idea what caused this—I suspect something with the hosting provider</li>
<li>Also super weird, the last entry in the DSpace log file is from <code>2018-04-20 16:35:09</code>, and then immediately it goes to <code>2018-04-20 19:15:04</code> (three hours later!):</li>
</ul>
<pretabindex="0"><code>2018-04-20 16:35:09,144 ERROR org.dspace.app.util.AbstractDSpaceWebapp @ Failed to record shutdown in Webapp table.
org.apache.tomcat.jdbc.pool.PoolExhaustedException: [localhost-startStop-2] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:250; busy:18; idle
:0; lastwait:5000].
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:685)
at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:187)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)
at org.dspace.storage.rdbms.DatabaseManager.getConnection(DatabaseManager.java:632)
at org.dspace.core.Context.init(Context.java:121)
at org.dspace.core.Context.<init>(Context.java:95)
at org.dspace.app.util.AbstractDSpaceWebapp.deregister(AbstractDSpaceWebapp.java:97)
at org.dspace.app.util.DSpaceContextListener.contextDestroyed(DSpaceContextListener.java:146)
at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:5115)
at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5779)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:224)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1588)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1577)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-04-20 19:15:04,006 INFO org.dspace.core.ConfigurationManager @ Loading from classloader: file:/home/cgspace.cgiar.org/config/dspace.cfg
</code></pre><ul>
<li>Very suspect!</li>
</ul>
<h2id="2018-04-24">2018-04-24</h2>
<ul>
<li>Testing my Ansible playbooks with a clean and updated installation of Ubuntu 18.04 and I fixed some issues that I hadn’t run into a few weeks ago</li>
<li>There seems to be a new issue with Java dependencies, though</li>
<li>The <code>default-jre</code> package is going to be Java 10 on Ubuntu 18.04, but I want to use <code>openjdk-8-jre-headless</code> (well, the JDK actually, but it uses this JRE)</li>
<li>Tomcat and Ant are fine with Java 8, but the <code>maven</code> package wants to pull in Java 10 for some reason</li>
<li>Looking closer, I see that <code>maven</code> depends on <code>java7-runtime-headless</code>, which is indeed provided by <code>openjdk-8-jre-headless</code></li>
<li>So it must be one of Maven’s dependencies…</li>
<li>I will watch it for a few days because it could be an issue that will be resolved before Ubuntu 18.04’s release</li>
<li>Otherwise I will post a bug to the ubuntu-release mailing list</li>
<li>Looks like the only way to fix this is to install <code>openjdk-8-jdk-headless</code> before (so it pulls in the JRE) in a separate transaction, or to manually install <code>openjdk-8-jre-headless</code> in the same apt transaction as <code>maven</code></li>
<li>Also, I started porting PostgreSQL 9.6 into the Ansible infrastructure scripts</li>
<li>This should be a drop in I believe, though I will definitely test it more locally as well as on DSpace Test once we move to DSpace 5.8 and Ubuntu 18.04 in the coming months</li>
</ul>
<h2id="2018-04-25">2018-04-25</h2>
<ul>
<li>Still testing the <ahref="https://github.com/ilri/rmg-ansible-public">Ansible infrastructure playbooks</a> for Ubuntu 18.04, Tomcat 8.5, and PostgreSQL 9.6</li>
<li>One other new thing I notice is that PostgreSQL 9.6 no longer uses <code>createuser</code> and <code>nocreateuser</code>, as those have actually meant <code>superuser</code> and <code>nosuperuser</code> and have been deprecated for <em>ten years</em></li>
<li>So for my notes, when I’m importing a CGSpace database dump I need to amend my notes to give super user permission to a user, rather than create user:</li>
</ul>
<pretabindex="0"><code>$ psql dspacetest -c 'alter user dspacetest superuser;'
at org.apache.coyote.http11.Http11InputBuffer.init(Http11InputBuffer.java:688)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:672)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
</code></pre><ul>
<li>There’s a <ahref="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=895866">Debian bug about this from a few weeks ago</a></li>
<li>Apparently Tomcat was compiled with Java 9, so doesn’t work with Java 8</li>
</ul>
<h2id="2018-04-29">2018-04-29</h2>
<ul>
<li>DSpace Test crashed again, looks like memory issues again</li>
<li>JVM heap size was last increased to 6144m but the system only has 8GB total so there’s not much we can do here other than get a bigger Linode instance or remove the massive Solr Statistics data</li>
</ul>
<h2id="2018-04-30">2018-04-30</h2>
<ul>
<li>DSpace Test crashed again</li>
<li>I will email the CGSpace team to ask them whether or not we want to commit to having a public test server that accurately mirrors CGSpace (ie, to upgrade to the next largest Linode)</li>
