Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000].
2018-01-02 01:21:19,137 ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ Error while searching for sidebar facets
org.dspace.discovery.SearchServiceException: org.apache.solr.search.SyntaxError: Cannot parse 'dateIssued_keyword:[1976+TO+1979]': Encountered ""]""] "" at line 1, column 32.
And there are many of these errors every day for the past month:
$ grep -c "Error while searching for sidebar facets" dspace.log.*
dspace.log.2017-11-21:4
dspace.log.2017-11-22:1
dspace.log.2017-11-23:4
dspace.log.2017-11-24:11
dspace.log.2017-11-25:0
dspace.log.2017-11-26:1
dspace.log.2017-11-27:7
dspace.log.2017-11-28:21
dspace.log.2017-11-29:31
dspace.log.2017-11-30:15
dspace.log.2017-12-01:15
dspace.log.2017-12-02:20
dspace.log.2017-12-03:38
dspace.log.2017-12-04:65
dspace.log.2017-12-05:43
dspace.log.2017-12-06:72
dspace.log.2017-12-07:27
dspace.log.2017-12-08:15
dspace.log.2017-12-09:29
dspace.log.2017-12-10:35
dspace.log.2017-12-11:20
dspace.log.2017-12-12:44
dspace.log.2017-12-13:36
dspace.log.2017-12-14:59
dspace.log.2017-12-15:104
dspace.log.2017-12-16:53
dspace.log.2017-12-17:66
dspace.log.2017-12-18:83
dspace.log.2017-12-19:101
dspace.log.2017-12-20:74
dspace.log.2017-12-21:55
dspace.log.2017-12-22:66
dspace.log.2017-12-23:50
dspace.log.2017-12-24:85
dspace.log.2017-12-25:62
dspace.log.2017-12-26:49
dspace.log.2017-12-27:30
dspace.log.2017-12-28:54
dspace.log.2017-12-29:68
dspace.log.2017-12-30:89
dspace.log.2017-12-31:53
dspace.log.2018-01-01:45
dspace.log.2018-01-02:34
Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let’s Encrypt if it’s just a handful of domains
Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000].
2018-01-02 01:21:19,137 ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ Error while searching for sidebar facets
org.dspace.discovery.SearchServiceException: org.apache.solr.search.SyntaxError: Cannot parse 'dateIssued_keyword:[1976+TO+1979]': Encountered ""]""] "" at line 1, column 32.
And there are many of these errors every day for the past month:
$ grep -c "Error while searching for sidebar facets" dspace.log.*
dspace.log.2017-11-21:4
dspace.log.2017-11-22:1
dspace.log.2017-11-23:4
dspace.log.2017-11-24:11
dspace.log.2017-11-25:0
dspace.log.2017-11-26:1
dspace.log.2017-11-27:7
dspace.log.2017-11-28:21
dspace.log.2017-11-29:31
dspace.log.2017-11-30:15
dspace.log.2017-12-01:15
dspace.log.2017-12-02:20
dspace.log.2017-12-03:38
dspace.log.2017-12-04:65
dspace.log.2017-12-05:43
dspace.log.2017-12-06:72
dspace.log.2017-12-07:27
dspace.log.2017-12-08:15
dspace.log.2017-12-09:29
dspace.log.2017-12-10:35
dspace.log.2017-12-11:20
dspace.log.2017-12-12:44
dspace.log.2017-12-13:36
dspace.log.2017-12-14:59
dspace.log.2017-12-15:104
dspace.log.2017-12-16:53
dspace.log.2017-12-17:66
dspace.log.2017-12-18:83
dspace.log.2017-12-19:101
dspace.log.2017-12-20:74
dspace.log.2017-12-21:55
dspace.log.2017-12-22:66
dspace.log.2017-12-23:50
dspace.log.2017-12-24:85
dspace.log.2017-12-25:62
dspace.log.2017-12-26:49
dspace.log.2017-12-27:30
dspace.log.2017-12-28:54
dspace.log.2017-12-29:68
dspace.log.2017-12-30:89
dspace.log.2017-12-31:53
dspace.log.2018-01-01:45
dspace.log.2018-01-02:34
Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let’s Encrypt if it’s just a handful of domains
<pre><code>Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000].
<li><p>Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let’s Encrypt if it’s just a handful of domains</p></li>
<li>I woke up to more up and down of CGSpace, this time UptimeRobot noticed a few rounds of up and down of a few minutes each and Linode also notified of high CPU load from 12 to 2 PM</li>
<li><p>I have no idea what these are but they seem to be coming from Amazon…</p></li>
<li><p>I guess for now I just have to increase the database connection pool’s max active</p></li>
<li><p>It’s currently 75 and normally I’d just bump it by 25 but let me be a bit daring and push it by 50 to 125, because I used to see at least 121 connections in pg_stat_activity before when we were using the shitty default pooling</p></li>
org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-256] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:125; busy:125; idle:0; lastwait:5000].
<li><p>I will just bump the connection limit to 300 because I’m fucking fed up with this shit</p></li>
<li><p>Once I get back to Amman I will have to try to create different database pools for different web applications, like recently discussed on the dspace-tech mailing list</p></li>
<li><p>Create accounts on CGSpace for two CTA staff km4ard@cta.int and bheenick@cta.int</p></li>
<pre><code>[Fri Jan 05 09:31:22.965398 2018] [:error] [pid 9340] [client 213.55.99.121:64476] WARNING: Unable to find a match for "9-16-1-RV.doc" in "/home/files/journals/6//articles/9/". Skipping this file., referer: http://dagris.info/reviewtool/index.php/index/install/upgrade
<li><p>I will delete the log file for now and tell Danny</p></li>
<li><p>Also, I’m still seeing a hundred or so of the “ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer” errors in dspace logs, I need to search the dspace-tech mailing list to see what the cause is</p></li>
<li><p>I will run a full Discovery reindex in the mean time to see if it’s something wrong with the Discovery Solr core</p>
<li><p>Reboot CGSpace and DSpace Test for new kernels (4.14.12-x86_64-linode92) that partially mitigate the <ahref="https://blog.linode.com/2018/01/03/cpu-vulnerabilities-meltdown-spectre/">Spectre and Meltdown CPU vulnerabilities</a></p></li>
<pre><code>dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'affiliation') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/affiliations.csv with csv;
<li><p>I looked to see what happened to this year’s Solr statistics sharding task that should have run on 2018-01-01 and of course it failed:</p>
Caused by: org.apache.http.client.NonRepeatableRequestException: Cannot retry request with a non-repeatable request entity. The cause lists the reason the original request failed.
<li><p>There is interesting documentation about this on the DSpace Wiki: <ahref="https://wiki.duraspace.org/display/DSDOC5x/SOLR+Statistics+Maintenance#SOLRStatisticsMaintenance-SolrShardingByYear">https://wiki.duraspace.org/display/DSDOC5x/SOLR+Statistics+Maintenance#SOLRStatisticsMaintenance-SolrShardingByYear</a></p></li>
<li><p>I’m looking to see maybe if we’re hitting the issues mentioned in <ahref="https://jira.duraspace.org/browse/DS-2212">DS-2212</a> that were apparently fixed in DSpace 5.2</p></li>
<li><p>I can apparently search for records in the Solr stats core that have an empty <code>owningColl</code> field using this in the Solr admin query: <code>-owningColl:*</code></p></li>
<li><p>On CGSpace I see 48,000,000 records that have an <code>owningColl</code> field and 34,000,000 that don’t:</p>
<li><p>Terry Brady has written some notes on the DSpace Wiki about Solr sharing issues: <ahref="https://wiki.duraspace.org/display/%7Eterrywbrady/Statistics+Import+Export+Issues">https://wiki.duraspace.org/display/%7Eterrywbrady/Statistics+Import+Export+Issues</a></p></li>
<li><p>Uptime Robot said that CGSpace went down at around 9:43 AM</p></li>
<li><p>I looked at PostgreSQL’s <code>pg_stat_activity</code> table and saw 161 active connections, but no pool errors in the DSpace logs:</p>
<li><p>Rather than blocking their IPs, I think I might just add their user agent to the “badbots” zone with Baidu, because they seem to be the only ones using that user agent:</p>
<li><p>According to nginx docs the <ahref="https://nginx.org/en/docs/hash.html">bucket size should be a multiple of the CPU’s cache alignment</a>, which is 64 for us:</p>
<li><p>On our servers that is 64, so I increased this parameter to 128 and deployed the changes to nginx</p></li>
<li><p>Almost immediately the PostgreSQL connections dropped back down to 40 or so, and UptimeRobot said the site was back up</p></li>
<li><p>So that’s interesting that we’re not out of PostgreSQL connections (current pool maxActive is 300!) but the system is “down” to UptimeRobot and very slow to use</p></li>
<li><p>Linode continues to test mitigations for Meltdown and Spectre: <ahref="https://blog.linode.com/2018/01/03/cpu-vulnerabilities-meltdown-spectre/">https://blog.linode.com/2018/01/03/cpu-vulnerabilities-meltdown-spectre/</a></p></li>
<li><p>I rebooted DSpace Test to see if the kernel will be updated (currently Linux 4.14.12-x86_64-linode92)… nope.</p></li>
<li><p>It looks like Linode will reboot the KVM hosts later this week, though</p></li>
<li><p>Udana from WLE asked if we could give him permission to upload CSVs to CGSpace (which would require super admin access)</p></li>
<li><p>Citing concerns with metadata quality, I suggested adding him on DSpace Test first</p></li>
<li><p>I opened a ticket with Atmire to ask them about DSpace 5.8 compatibility: <ahref="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=560">https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=560</a></p></li>
<li><p>Following up with the Solr sharding issue on the dspace-tech mailing list, I noticed this interesting snippet in the Tomcat <code>localhost_access_log</code> at the time of my sharding attempt on my test machine:</p>
<li><p>The new core is created but when DSpace attempts to POST to it there is an HTTP 409 error</p></li>
<li><p>This is apparently a common Solr error code that means “version conflict”: <ahref="http://yonik.com/solr/optimistic-concurrency/">http://yonik.com/solr/optimistic-concurrency/</a></p></li>
<li><p>Looks like that bot from the PerfectIP.net host ended up making about 450,000 requests to XMLUI alone yesterday:</p>
<li><p>So theoretically I could name each connection “xmlui” or “dspaceWeb” or something meaningful and it would show up in PostgreSQL’s <code>pg_stat_activity</code> table!</p></li>
<li><p>This would be super helpful for figuring out where load was coming from (now I wonder if I could figure out how to graph this)</p></li>
<li><p>Also, I realized that the <code>db.jndi</code> parameter in dspace.cfg needs to match the <code>name</code> value in your applicaiton’s context—not the <code>global</code> one</p></li>
<li><p>Ah hah! Also, I can name the default DSpace connection pool in dspace.cfg as well, like:</p>
<li><p>I’m looking at the <ahref="https://wiki.duraspace.org/display/DSDOC6x/Installing+DSpace#InstallingDSpace-ServletEngine(ApacheTomcat7orlater,Jetty,CauchoResinorequivalent)">DSpace 6.0 Install docs</a> and notice they tweak the number of threads in their Tomcat connector:</p>
<li><p>In Tomcat 8.5 the <code>maxThreads</code> defaults to 200 which is probably fine, but tweaking <code>minSpareThreads</code> could be good</p></li>
<li><p>I don’t see a setting for <code>maxSpareThreads</code> in the docs so that might be an error</p></li>
<li><p>Looks like in Tomcat 8.5 the default URIEncoding for Connectors is UTF-8, so we don’t need to specify that manually anymore: <ahref="https://tomcat.apache.org/tomcat-8.5-doc/config/http.html">https://tomcat.apache.org/tomcat-8.5-doc/config/http.html</a></p></li>
<li><p>Ooh, I just saw the <code>acceptorThreadCount</code> setting (in Tomcat 7 and 8.5):</p>
<pre><code>The number of threads to be used to accept connections. Increase this value on a multi CPU machine, although you would never really need more than 2. Also, with a lot of non keep alive connections, you might want to increase this value as well. Default value is 1.
<pre><code>13-Jan-2018 13:59:05.245 WARNING [main] org.apache.tomcat.dbcp.dbcp2.BasicDataSourceFactory.getObjectInstance Name = dspace6 Property maxActive is not used in DBCP2, use maxTotal instead. maxTotal default value is 8. You have set value of "35" for "maxActive" property, which is being ignored.
13-Jan-2018 13:59:05.245 WARNING [main] org.apache.tomcat.dbcp.dbcp2.BasicDataSourceFactory.getObjectInstance Name = dspace6 Property maxWait is not used in DBCP2 , use maxWaitMillis instead. maxWaitMillis default value is -1. You have set value of "5000" for "maxWait" property, which is being ignored.
<li><p>I looked in my Tomcat 7.0.82 logs and I don’t see anything about DBCP2 errors, so I guess this a Tomcat 8.0.x or 8.5.x thing</p></li>
<li><p>DBCP2 appears to be Tomcat 8.0.x and up according to the <ahref="https://tomcat.apache.org/migration-8.html">Tomcat 8.0 migration guide</a></p></li>
<li><p>I have updated our <ahref="https://github.com/ilri/rmg-ansible-public/commit/246f9d7b06d53794f189f0cc57ad5ddd80f0b014">Ansible infrastructure scripts</a> so that it will be ready whenever we switch to Tomcat 8 (probably with Ubuntu 18.04 later this year)</p></li>
<li><p>When I enable the ResourceLink in the ROOT.xml context I get the following error in the Tomcat localhost log:</p>
<pre><code>13-Jan-2018 14:14:36.017 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [org.dspace.app.util.DSpaceWebappListener]
<li><p>Interesting blog post benchmarking Tomcat JDBC vs Apache Commons DBCP2, with configuration snippets: <ahref="http://www.tugay.biz/2016/07/tomcat-connection-pool-vs-apache.html">http://www.tugay.biz/2016/07/tomcat-connection-pool-vs-apache.html</a></p></li>
<li><p>The Tomcat vs Apache pool thing is confusing, but apparently we’re using Apache Commons DBCP2 because we don’t specify <code>factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"</code> in our global resource</p></li>
<li><p>So at least I know that I’m not looking for documentation or troubleshooting on the Tomcat JDBC pool!</p></li>
<li><p>I looked at <code>pg_stat_activity</code> during Tomcat’s startup and I see that the pool created in server.xml is indeed connecting, just that nothing uses it</p></li>
<li><p>Also, the fallback connection parameters specified in local.cfg (not dspace.cfg) are used</p></li>
<li><p>Shit, this might actually be a DSpace error: <ahref="https://jira.duraspace.org/browse/DS-3434">https://jira.duraspace.org/browse/DS-3434</a></p></li>
<li>Looking at the authors Peter had corrected</li>
<li>Some had multiple and he’s corrected them by adding <code>||</code> in the correction column, but I can’t process those this way so I will just have to flag them and do those manually later</li>
<li>Also, I can flag the values that have “DELETE”</li>
<li>Then I need to facet the correction column on isBlank(value) and not flagged</li>
</ul>
<h2id="2018-01-15">2018-01-15</h2>
<ul>
<li>Help Udana from IWMI export a CSV from DSpace Test so he can start trying a batch upload</li>
<pre><code>update metadatavalue set text_value='Formally Published' where resource_type_id=2 and metadata_field_id=214 and text_value like 'Formally published';
delete from metadatavalue where resource_type_id=2 and metadata_field_id=214 and text_value like 'NO';
update metadatavalue set text_value='en' where resource_type_id=2 and metadata_field_id=38 and text_value ~ '(En|English)';
update metadatavalue set text_value='fr' where resource_type_id=2 and metadata_field_id=38 and text_value ~ '(fre|frn|French)';
update metadatavalue set text_value='es' where resource_type_id=2 and metadata_field_id=38 and text_value ~ '(Spanish|spa)';
update metadatavalue set text_value='vi' where resource_type_id=2 and metadata_field_id=38 and text_value='Vietnamese';
update metadatavalue set text_value='ru' where resource_type_id=2 and metadata_field_id=38 and text_value='Ru';
update metadatavalue set text_value='in' where resource_type_id=2 and metadata_field_id=38 and text_value ~ '(IN|In)';
delete from metadatavalue where resource_type_id=2 and metadata_field_id=38 and text_value ~ '(dc.language.iso|CGIAR Challenge Program on Water and Food)';
<li><p>Continue proofing Peter’s author corrections that I started yesterday, faceting on non blank, non flagged, and briefly scrolling through the values of the corrections to find encoding errors for French and Spanish names</p></li>
<li><p>Even searching in the DSpace advanced search for author equals “Tarawali” produces nothing…</p></li>
<li><p>Otherwise, the <ahref="https://wiki.duraspace.org/display/DSPACE/Helper+SQL+functions+for+DSpace+5">DSpace 5 SQL Helper Functions</a> provide <code>ds5_item2itemhandle()</code>, which is much easier than my long query above that I always have to go search for</p></li>
<li><p>For example, to find the Handle for an item that has the author “Erni”:</p>
<pre><code>dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where metadata_schema_id = 2 and element = 'contributor' and qualifier = 'affiliation') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/affiliations.csv with csv;
<li><p>Looking over the affiliations again I see dozens of CIAT ones with their affiliation formatted like: International Center for Tropical Agriculture (CIAT)</p></li>
<li><p>For example, this one is from just last month: <ahref="https://cgspace.cgiar.org/handle/10568/89930">https://cgspace.cgiar.org/handle/10568/89930</a></p></li>
<li><p>Our controlled vocabulary has this in the format without the abbreviation: International Center for Tropical Agriculture</p></li>
<li><p>So some submitters don’t know to use the controlled vocabulary lookup</p></li>
<li><p>Help Sisay with some thumbnails for book chapters in Open Refine and SAFBuilder</p></li>
<li><p>CGSpace users were having problems logging in, I think something’s wrong with LDAP because I see this in the logs:</p>
<li>Meeting with CGSpace team, a few action items:
<ul>
<li>Discuss standardized names for CRPs and centers with ICARDA (don’t wait for CG Core)</li>
<li>Re-send DC rights implementation and forward to everyone so we can move forward with it (without the URI field for now)</li>
<li>Start looking at where I was with the AGROVOC API</li>
<li>Have a controlled vocabulary for CGIAR authors’ names and ORCIDs? Perhaps values like: Orth, Alan S. (0000-0002-1735-7458)</li>
<li>Need to find the metadata field name that ICARDA is using for their ORCIDs</li>
<li>Update text for DSpace version plan on wiki</li>
<li>Come up with an SLA, something like: <em>In return for your contribution we will, to the best of our ability, ensure 99.5% (“two and a half nines”) uptime of CGSpace, ensure data is stored in open formats and safely backed up, follow CG Core metadata standards, …</em></li>
<li>Add Sisay and Danny to Uptime Robot and allow them to restart Tomcat on CGSpace ✔</li>
</ul></li>
<li>I removed Tsega’s SSH access to the web and DSpace servers, and asked Danny to check whether there is anything he needs from Tsega’s home directories so we can delete the accounts completely</li>
<li>I removed Tsega’s access to Linode dashboard as well</li>
<li>I ended up creating a Jira issue for my <code>db.jndi</code> documentation fix: <ahref="https://jira.duraspace.org/browse/DS-3803">DS-3803</a></li>
<li>The DSpace developers said they wanted each pull request to be associated with a Jira issue</li>
</ul>
<h2id="2018-01-17">2018-01-17</h2>
<ul>
<li>Abenet asked me to proof and upload 54 records for LIVES</li>
<li>A few records were missing countries (even though they’re all from Ethiopia)</li>
<li>Also, there are whitespace issues in many columns, and the items are mapped to the LIVES and ILRI articles collections, not Theses</li>
<pre><code>2018-01-17 07:59:25,856 INFO org.apache.http.impl.client.SystemDefaultHttpClient @ I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://localhost:8081: The target server failed to respond
2018-01-17 07:59:25,856 INFO org.apache.http.impl.client.SystemDefaultHttpClient @ Retrying request to {}->http://localhost:8081
<li><p>Looking at the JVM graphs from Munin it does look like the heap ran out of memory (see the blue dip just before the green spike when I restarted Tomcat):</p></li>
<li><p>I’m playing with maven repository caching using Artifactory in a Docker instance: <ahref="https://www.jfrog.com/confluence/display/RTF/Installing+with+Docker">https://www.jfrog.com/confluence/display/RTF/Installing+with+Docker</a></p>
<li><p>Then configure the local maven to use it in settings.xml with the settings from “Set Me Up”: <ahref="https://www.jfrog.com/confluence/display/RTF/Using+Artifactory">https://www.jfrog.com/confluence/display/RTF/Using+Artifactory</a></p></li>
<li><p>This could be a game changer for testing and running the Docker DSpace image</p></li>
<li><p>Wow, I even managed to add the Atmire repository as a remote and map it into the <code>libs-release</code> virtual repository, then tell maven to use it for <code>atmire.com-releases</code> in settings.xml!</p></li>
<li><p>Hmm, some maven dependencies for the SWORDv2 web application in DSpace 5.5 are broken:</p>
<pre><code>[ERROR] Failed to execute goal on project dspace-swordv2: Could not resolve dependencies for project org.dspace:dspace-swordv2:war:5.5: Failed to collect dependencies at org.swordapp:sword2-server:jar:classes:1.0 -> org.apache.abdera:abdera-client:jar:1.1.1 -> org.apache.abdera:abdera-core:jar:1.1.1 -> org.apache.abdera:abdera-i18n:jar:1.1.1 -> org.apache.geronimo.specs:geronimo-activation_1.0.2_spec:jar:1.1: Failed to read artifact descriptor for org.apache.geronimo.specs:geronimo-activation_1.0.2_spec:jar:1.1: Could not find artifact org.apache.geronimo.specs:specs:pom:1.1 in central (http://localhost:8081/artifactory/libs-release) -> [Help 1]
<li><p>UptimeRobot said CGSpace went down for a few minutes</p></li>
<li><p>I didn’t do anything but it came back up on its own</p></li>
<li><p>I don’t see anything unusual in the XMLUI or REST/OAI logs</p></li>
<li><p>Now Linode alert says the CPU load is high, <em>sigh</em></p></li>
<li><p>Regarding the heap space error earlier today, it looks like it does happen a few times a week or month (I’m not sure how far these logs go back, as they are not strictly daily):</p>
<li><p>Overall the heap space usage in the munin graph seems ok, though I usually increase it by 512MB over the average a few times per year as usage grows</p></li>
<li><p>But maybe I should increase it by more, like 1024MB, to give a bit more head room</p></li>
<li>UptimeRobot said CGSpace was down for 1 minute last night</li>
<li>I don’t see any errors in the nginx or catalina logs, so I guess UptimeRobot just got impatient and closed the request, which caused nginx to send an HTTP 499</li>
<li><p>Maria from Bioversity asked if I could remove the abstracts from all of their Limited Access items in the <ahref="https://cgspace.cgiar.org/handle/10568/35501">Bioversity Journal Articles</a> collection</p></li>
<li><p>It’s easy enough to do in OpenRefine, but you have to be careful to only get those items that are uploaded into Bioversity’s collection, not the ones that are mapped from others!</p></li>
<li><p>Use this GREL in OpenRefine after isolating all the Limited Access items: <code>value.startsWith("10568/35501")</code></p></li>
<li><p>UptimeRobot said CGSpace went down AGAIN and both Sisay and Danny immediately logged in and restarted Tomcat without talking to me <em>or</em> each other!</p>
<li><p>I had to cancel the Discovery indexing and I’ll have to re-try it another time when the server isn’t so busy (it had already taken two hours and wasn’t even close to being done)</p></li>
<li><p>For now I’ve increased the Tomcat JVM heap from 5632 to 6144m, to give ~1GB of free memory over the average usage to hopefully account for spikes caused by load or background jobs</p></li>
<li><p>Linode alerted again and said that CGSpace was using 301% CPU</p></li>
<li><p>Peter emailed to ask why <ahref="https://cgspace.cgiar.org/handle/10568/88090">this item</a> doesn’t have an Altmetric badge on CGSpace but does have one on the <ahref="https://www.altmetric.com/details/26709041">Altmetric dashboard</a></p></li>
<li><p>Looks like our badge code calls the <code>handle</code> endpoint which doesn’t exist:</p>
<li><p>I tested the abstract cleanups on Bioversity’s Journal Articles collection again that I had started a few days ago</p></li>
<li><p>In the end there were 324 items in the collection that were Limited Access, but only 199 had abstracts</p></li>
<li><p>I want to document the workflow of adding a production PostgreSQL database to a development instance of <ahref="https://github.com/alanorth/docker-dspace">DSpace in Docker</a>:</p>
<li>Look over Udana’s CSV of 25 WLE records from last week</li>
<li>I sent him some corrections:
<ul>
<li>The file encoding is Windows-1252</li>
<li>There were whitespace issues in the dc.identifier.citation field (spaces at the beginning and end, and multiple spaces in between some words)</li>
<li>Also, the authors listed in the citation need to be in normal format, separated by commas or colons (however you prefer), not with ||</li>
<li>There were spaces in the beginning and end of some cg.identifier.doi fields</li>
<li>Make sure that the cg.coverage.countries field is just countries: ie, no “SOUTH ETHIOPIA” or “EAST AFRICA” (the first should just be ETHIOPIA, the second should be in cg.coverage.region instead)</li>
<li>The current list of regions we use is here: <ahref="https://github.com/ilri/DSpace/blob/5_x-prod/dspace/config/input-forms.xml#L5162">https://github.com/ilri/DSpace/blob/5_x-prod/dspace/config/input-forms.xml#L5162</a></li>
<li>You have a syntax error in your cg.coverage.regions (extra ||)</li>
<li>The value of dc.identifier.issn should just be the ISSN but you have: eISSN: 1479-487X</li>
</ul></li>
<li>I wrote a quick Python script to use the DSpace REST API to find all collections under a given community</li>
<li>The source code is here: <ahref="https://gist.github.com/alanorth/ddd7f555f0e487fe0e9d3eb4ff26ce50">rest-find-collections.py</a></li>
<li><p>Looking at the <ahref="https://tomcat.apache.org/tomcat-7.0-doc/config/http.html">Tomcat connector docs</a> I think we really need to increase <code>maxThreads</code></p></li>
<li><p>The default is 200, which can easily be taken up by bots considering that Google and Bing each browse with fifty (50) connections each sometimes!</p></li>
<li><p>Before I increase this I want to see if I can measure and graph this, and then benchmark</p></li>
<li><p>I’ll probably also increase <code>minSpareThreads</code> to 20 (its default is 10)</p></li>
<li><p>I still want to bump up <code>acceptorThreadCount</code> from 1 to 2 as well, as the documentation says this should be increased on multi-core systems</p></li>
<li><p>I spent quite a bit of time looking at <code>jvisualvm</code> and <code>jconsole</code> today</p></li>
<li><p>Run system updates on DSpace Test and reboot it</p></li>
<li><p>I see I can monitor the number of Tomcat threads and some detailed JVM memory stuff if I install <code>munin-plugins-java</code></p></li>
<li><p>I’d still like to get arbitrary mbeans like activeSessions etc, though</p></li>
<li><p>I can’t remember if I had to configure the jmx settings in <code>/etc/munin/plugin-conf.d/munin-node</code> or not—I think all I did was re-run the <code>munin-node-configure</code> script and of course enable JMX in Tomcat’s JVM options</p></li>
<li>Thinking about generating a jmeter test plan for DSpace, along the lines of <ahref="https://github.com/Georgetown-University-Libraries/dspace-performance-test">Georgetown’s dspace-performance-test</a></li>
<li><p>I got a list of all the GET requests on CGSpace for January 21st (the last time Linode complained the load was high), excluding admin calls:</p>
<li><p>Atmire responded to <ahref="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=560">my issue from two weeks ago</a> and said they will start looking into DSpace 5.8 compatibility for CGSpace</p></li>
<li><p>I set up a new Arch Linux Linode instance with 8192 MB of RAM and ran the test plan a few times to get a baseline:</p>
<li><p>Run another round of tests on DSpace Test with jmeter after changing Tomcat’s <code>minSpareThreads</code> to 20 (default is 10) and <code>acceptorThreadCount</code> to 2 (default is 1):</p>
<li><p>I changed the parameters back to the baseline ones and switched the Tomcat JVM garbage collector to G1GC and re-ran the tests</p></li>
<li><p>JVM options for Tomcat changed from <code>-Xms3072m -Xmx3072m -XX:+UseConcMarkSweepGC</code> to <code>-Xms3072m -Xmx3072m -XX:+UseG1GC -XX:+PerfDisableSharedMem</code></p>
<li>Peter followed up about some of the points from the Skype meeting last week</li>
<li>Regarding the ORCID field issue, I see <ahref="http://repo.mel.cgiar.org/handle/20.500.11766/7668?show=full">ICARDA’s MELSpace is using <code>cg.creator.ID</code></a>: 0000-0001-9156-7691</li>
<li>I had floated the idea of using a controlled vocabulary with values formatted something like: Orth, Alan S. (0000-0002-1735-7458)</li>
<li>Update PostgreSQL JDBC driver version from 42.1.4 to 42.2.1 on DSpace Test, see: <ahref="https://jdbc.postgresql.org/">https://jdbc.postgresql.org/</a></li>
<li>Reboot DSpace Test to get new Linode kernel (Linux 4.14.14-x86_64-linode94)</li>
<li>I am testing my old work on the <code>dc.rights</code> field, I had added a branch for it a few months ago</li>
<li>I added a list of Creative Commons and other licenses in <code>input-forms.xml</code></li>
<li>The problem is that Peter wanted to use two questions, one for CG centers and one for other, but using the same metadata value, which isn’t possible (?)</li>
<li><p>I was worried that if a user selected this field for some reason that DSpace would store an empty value, but it simply doesn’t register that as a valid option:</p></li>
<li>I submitted a test item with ORCiDs and dc.rights from a controlled vocabulary on DSpace Test: <ahref="https://dspacetest.cgiar.org/handle/10568/97703">https://dspacetest.cgiar.org/handle/10568/97703</a></li>
<li>I will send it to Peter to check and give feedback (ie, about the ORCiD field name as well as allowing users to add ORCiDs manually or not)</li>
</ul>
<h2id="2018-01-28">2018-01-28</h2>
<ul>
<li>Assist Udana from WLE again to proof his 25 records and upload them to DSpace Test</li>
<li>I am playing with the <code>startStopThreads="0"</code> parameter in Tomcat <code><Engine></code> and <code><Host></code> configuration</li>
<li>It reduces the start up time of Catalina by using multiple threads to start web applications in parallel</li>
<li>On my local test machine the startup time went from 70 to 30 seconds</li>
<li><p>So is this an error caused by this particular client (which happens to be <code>Yahoo! Slurp</code>)?</p></li>
<li><p>I see a few dozen HTTP 499 errors in the nginx access log for a few minutes before this happened, but HTTP 499 is just when nginx says that the client closed the request early</p></li>
<li><p>Perhaps this from the nginx error log is relevant?</p>
<li><p>I guess responses that don’t fit in RAM get saved to disk (a default of 1024M), so this is definitely not the issue here, and that warning is totally unrelated</p></li>
<li><p>My best guess is that the Solr search error is related somehow but I can’t figure it out</p></li>
<li><p>We definitely have enough database connections, as I haven’t seen a pool error in weeks:</p>
<li><p>Adam Hunt from WLE complained that pages take “1-2 minutes” to load each, from France and Sri Lanka</p></li>
<li><p>I asked him which particular pages, as right now pages load in 2 or 3 seconds for me</p></li>
<li><p>UptimeRobot said CGSpace went down again, and I looked at PostgreSQL and saw 211 active database connections</p></li>
<li><p>If it’s not memory and it’s not database, it’s gotta be Tomcat threads, seeing as the default <code>maxThreads</code> is 200 anyways, it actually makes sense</p></li>
<li><p>I decided to change the Tomcat thread settings on CGSpace:</p>
<li><code>maxThreads</code> from 200 (default) to 400</li>
<li><code>processorCache</code> from 200 (default) to 400, <ahref="https://tomcat.apache.org/tomcat-7.0-doc/config/http.html">recommended to be the same as <code>maxThreads</code></a></li>
<li><code>minSpareThreads</code> from 10 (default) to 20</li>
<li><code>acceptorThreadCount</code> from 1 (default) to 2, <ahref="https://tomcat.apache.org/tomcat-7.0-doc/config/http.html">recommended to be 2 on multi-CPU systems</a></li>
<li><p>Looks like I only enabled the new thread stuff on the connector used internally by Solr, so I probably need to match that by increasing them on the other connector that nginx proxies to</p></li>
<li><p>Jesus Christ I need to fucking fix the Munin monitoring so that I can tell how many fucking threads I have running</p></li>
<li><p>Wow, so apparently you need to specify which connector to check if you want any of the Munin Tomcat plugins besides “tomcat_jvm” to work (the connector name can be seen in the Catalina logs)</p></li>
<li><p>I modified <em>/etc/munin/plugin-conf.d/tomcat</em> to add the connector (with surrounding quotes!) and now the other plugins work (obviously the credentials are incorrect):</p>
<li><p>Apparently you can’t monitor more than one connector, so I guess the most important to monitor would be the one that nginx is sending stuff to</p></li>
<li><p>So for now I think I’ll just monitor these and skip trying to configure the jmx plugins</p></li>
<li><p>Although following the logic of _/usr/share/munin/plugins/jmx_tomcat<em>dbpools</em> could be useful for getting the active Tomcat sessions</p></li>
<li><p>From debugging the <code>jmx_tomcat_db_pools</code> script from the <code>munin-plugins-java</code> package, I see that this is how you call arbitrary mbeans:</p>
<li><p>I filed a ticket with Atmire: <ahref="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=566">https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=566</a></p></li>
org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-451] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:300; busy:300; idle:0; lastwait:5000].
<li><p>Ok, so this is interesting: I figured out how to get the MBean path to query Tomcat’s activeSessions from JMX (using <code>munin-plugins-java</code>):</p>
