<li>CTA says the Amazon IPs are AWS gateways for real user traffic</li>
<li>I was trying to add Felix Shaw’s account back to the Administrators group on DSpace Test, but I couldn’t find his name in the user search of the groups page
<ul>
<li>If I searched for “Felix” or “Shaw” I saw other matches, included one for his personal email address!</li>
<li>I ended up finding him via searching for his email address</li>
</ul>
</li>
</ul>
<h2id="2019-04-03">2019-04-03</h2>
<ul>
<li>Maria from Bioversity emailed me a list of new ORCID identifiers for their researchers so I will add them to our controlled vocabulary
<ul>
<li>First I need to extract the ones that are unique from their list compared to our existing one:</li>
<li>I created a pull request and merged the changes to the <code>5_x-prod</code> branch (<ahref="https://github.com/ilri/DSpace/pull/417">#417</a>)</li>
<li>A few days ago I noticed some weird update process for the statistics-2018 Solr core and I see it’s still going:</li>
</ul>
<pretabindex="0"><code>2019-04-03 16:34:02,262 INFO org.dspace.statistics.SolrLogger @ Updating : 1754500/21701 docs in http://localhost:8081/solr//statistics-2018
</code></pre><ul>
<li>Interestingly, there are 5666 occurences, and they are mostly for the 2018 core:</li>
<li>The other thing visible there is that the past few days the load has spiked to 500% and I don’t think it’s a coincidence that the Solr updating thing is happening…</li>
<li>I ran all system updates and rebooted the server
<ul>
<li>The load was lower on the server after reboot, but Solr didn’t come back up properly according to the Solr Admin UI:</li>
</ul>
</li>
</ul>
<pretabindex="0"><code>statistics-2017: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Error opening new searcher
</code></pre><ul>
<li>I restarted it again and all the Solr cores came up properly…</li>
</ul>
<h2id="2019-04-06">2019-04-06</h2>
<ul>
<li>Udana asked why item <ahref="https://cgspace.cgiar.org/handle/10568/91278">10568/91278</a> didn’t have an Altmetric badge on CGSpace, but on the <ahref="https://wle.cgiar.org/food-and-agricultural-innovation-pathways-prosperity">WLE website</a> it does
<ul>
<li>I looked and saw that the WLE website is using the Altmetric score associated with the DOI, and that the Handle has no score at all</li>
<li>I tweeted the item and I assume this will link the Handle with the DOI in the system</li>
<li>Twenty minutes later I see the same Altmetric score (9) on CGSpace</li>
</ul>
</li>
<li>Linode sent an alert that there was high CPU usage this morning on CGSpace (linode18) and these were the top IPs in the webserver access logs around the time:</li>
<li><code>45.5.184.72</code> is in Colombia so it’s probably CIAT, and I see they are indeed trying to get crawl the Discover pages on CIAT’s datasets collection:</li>
<li>Their user agent is the one I added to the badbots list in nginx last week: “GuzzleHttp/6.3.3 curl/7.47.0 PHP/7.0.30-0ubuntu0.16.04.1”</li>
<li>They made 22,000 requests to Discover on this collection today alone (and it’s only 11AM):</li>
<li>I need to find a contact at CIAT to tell them to use the REST API rather than crawling Discover</li>
<li>Maria from Bioversity recommended that we use the phrase “AGROVOC subject” instead of “Subject” in Listings and Reports
<ul>
<li>I made a pull request to update this and merged it to the <code>5_x-prod</code> branch (<ahref="https://github.com/ilri/DSpace/pull/418">#418</a>)</li>
</ul>
</li>
</ul>
<h2id="2019-04-07">2019-04-07</h2>
<ul>
<li>Looking into the impact of harvesters like <code>45.5.184.72</code>, I see in Solr that this user is not categorized as a bot so it definitely impacts the usage stats by some tens of thousands <em>per day</em></li>
<li>Last week CTA switched their frontend code to use HEAD requests instead of GET requests for bitstreams
<ul>
<li>I am trying to see if these are registered as downloads in Solr or not</li>
<li>I see 96,925 downloads from their AWS gateway IPs in 2019-03:</li>
</ul>
</li>
</ul>
<pretabindex="0"><code>$ http --print b 'http://localhost:8081/solr/statistics/select?q=type%3A0+AND+(ip%3A18.196.196.108+OR+ip%3A18.195.78.144+OR+ip%3A18.195.218.6)&fq=statistics_type%3Aview&fq=bundleName%3AORIGINAL&fq=dateYearMonth%3A2019-03&rows=0&wt=json&indent=true'
{
"response": {
"docs": [],
"numFound": 96925,
"start": 0
},
"responseHeader": {
"QTime": 1,
"params": {
"fq": [
"statistics_type:view",
"bundleName:ORIGINAL",
"dateYearMonth:2019-03"
],
"indent": "true",
"q": "type:0 AND (ip:18.196.196.108 OR ip:18.195.78.144 OR ip:18.195.218.6)",
"rows": "0",
"wt": "json"
},
"status": 0
}
}
</code></pre><ul>
<li>Strangely I don’t see many hits in 2019-04:</li>
</ul>
<pretabindex="0"><code>$ http --print b 'http://localhost:8081/solr/statistics/select?q=type%3A0+AND+(ip%3A18.196.196.108+OR+ip%3A18.195.78.144+OR+ip%3A18.195.218.6)&fq=statistics_type%3Aview&fq=bundleName%3AORIGINAL&fq=dateYearMonth%3A2019-04&rows=0&wt=json&indent=true'
{
"response": {
"docs": [],
"numFound": 38,
"start": 0
},
"responseHeader": {
"QTime": 1,
"params": {
"fq": [
"statistics_type:view",
"bundleName:ORIGINAL",
"dateYearMonth:2019-04"
],
"indent": "true",
"q": "type:0 AND (ip:18.196.196.108 OR ip:18.195.78.144 OR ip:18.195.218.6)",
"rows": "0",
"wt": "json"
},
"status": 0
}
}
</code></pre><ul>
<li>Making some tests on GET vs HEAD requests on the <ahref="https://dspacetest.cgiar.org/handle/10568/100289">CTA Spore 192 item</a> on DSpace Test:</li>
</ul>
<pretabindex="0"><code>$ http --print Hh GET https://dspacetest.cgiar.org/bitstream/handle/10568/100289/Spore-192-EN-web.pdf
GET /bitstream/handle/10568/100289/Spore-192-EN-web.pdf HTTP/1.1
<li>So definitely the <em>size</em> of the transfer is more efficient with a HEAD, but I need to wait to see if these requests show up in Solr
<ul>
<li>After twenty minutes of waiting I still don’t see any new requests in the statistics core, but when I try the requests from the command line again I see the following in the DSpace log:</li>
</ul>
</li>
</ul>
<pretabindex="0"><code>2019-04-07 02:05:30,966 INFO org.dspace.usage.LoggerUsageEventListener @ anonymous:session_id=EF2DB6E4F69926C5555B3492BB0071A8:ip_addr=78.x.x.x:view_bitstream:bitstream_id=165818
2019-04-07 02:05:39,265 INFO org.dspace.usage.LoggerUsageEventListener @ anonymous:session_id=B6381FC590A5160D84930102D068C7A3:ip_addr=78.x.x.x:view_bitstream:bitstream_id=165818
</code></pre><ul>
<li>So my inclination is that both HEAD and GET requests are registered as views as far as Solr and DSpace are concerned
<ul>
<li>Strangely, the statistics Solr core says it hasn’t been modified in 24 hours, so I tried to start the “optimize” process from the Admin UI and I see this in the Solr log:</li>
</ul>
</li>
</ul>
<pretabindex="0"><code>2019-04-07 02:08:44,186 INFO org.apache.solr.update.UpdateHandler @ start commit{,optimize=true,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
</code></pre><ul>
<li>Ugh, even after optimizing there are no Solr results for requests from my IP, and actually I only see 18 results from 2019-04 so far and none of them are <code>statistics_type:view</code>… very weird
<ul>
<li>I don’t even see many hits for days after 2019-03-17, when I migrated the server to Ubuntu 18.04 and copied the statistics core from CGSpace (linode18)</li>
<li>I will try to re-deploy the <code>5_x-dev</code> branch and test again</li>
</ul>
</li>
<li>According to the <ahref="https://wiki.lyrasis.org/display/DSDOC5x/SOLR+Statistics">DSpace 5.x Solr documentation</a> the default commit time is after 15 minutes or 10,000 documents (see <code>solrconfig.xml</code>)</li>
<li>I looped some GET and HEAD requests to a bitstream on my local instance and after some time I see that they <em>do</em> register as downloads (even though they are internal):</li>
</ul>
<pretabindex="0"><code>$ http --print b 'http://localhost:8080/solr/statistics/select?q=type%3A0+AND+time%3A2019-04-07*&fq=statistics_type%3Aview&fq=isInternal%3Atrue&rows=0&wt=json&indent=true'
{
"response": {
"docs": [],
"numFound": 909,
"start": 0
},
"responseHeader": {
"QTime": 0,
"params": {
"fq": [
"statistics_type:view",
"isInternal:true"
],
"indent": "true",
"q": "type:0 AND time:2019-04-07*",
"rows": "0",
"wt": "json"
},
"status": 0
}
}
</code></pre><ul>
<li>I confirmed the same on CGSpace itself after making one HEAD request</li>
<li>So I’m pretty sure it’s something about DSpace Test using the CGSpace statistics core, and not that I deployed Solr 4.10.4 there last week
<ul>
<li>I deployed Solr 4.10.4 locally and ran a bunch of requests for bitstreams and they do show up in the Solr statistics log, so the issue must be with re-using the existing Solr core from CGSpace</li>
</ul>
</li>
<li>Now this gets more frustrating: I did the same GET and HEAD tests on a local Ubuntu 16.04 VM with Solr 4.10.2 and 4.10.4 and the statistics are recorded
<ul>
<li>This leads me to believe there is something specifically wrong with DSpace Test (linode19)</li>
<li>The only thing I can think of is that the JVM is using G1GC instead of ConcMarkSweepGC</li>
</ul>
</li>
<li>Holy shit, all this is actually because of the GeoIP1 deprecation and a missing <code>GeoLiteCity.dat</code>
<ul>
<li>For some reason the missing GeoIP data causes stats to not be recorded whatsoever and there is no error!</li>
<li>DSpace 5.10 upgraded to use GeoIP2, but we are on 5.8 so I just copied the missing database file from another server because it has been <em>removed</em> from MaxMind’s server as of 2018-04-01</li>
<li>Now I made 100 requests and I see them in the Solr statistics… fuck my life for wasting five hours debugging this</li>
</ul>
</li>
<li>UptimeRobot said CGSpace went down and up a few times tonight, and my first instict was to check <code>iostat 1 10</code> and I saw that CPU steal is around 10–30 percent right now…</li>
<li>The load average is super high right now, as I’ve noticed the last few times UptimeRobot said that CGSpace went down:</li>
</ul>
<pretabindex="0"><code>$ cat /proc/loadavg
10.70 9.17 8.85 18/633 4198
</code></pre><ul>
<li>According to the server logs there is actually not much going on right now:</li>
<li>It seems that the issue with CGSpace being “down” is actually because of CPU steal again!!!</li>
<li>I opened a ticket with support and asked them to migrate the VM to a less busy host</li>
</ul>
<h2id="2019-04-08">2019-04-08</h2>
<ul>
<li>Start checking IITA’s last round of batch uploads from <ahref="https://dspacetest.cgiar.org/handle/10568/100333">March on DSpace Test</a> (20193rd.xls)
<ul>
<li>Lots of problems with affiliations, I had to correct about sixty of them</li>
<li>I used lein to host the latest CSV of our affiliations for OpenRefine to reconcile against:</li>
</ul>
</li>
</ul>
<pretabindex="0"><code>$ lein run ~/src/git/DSpace/2019-02-22-affiliations.csv name id
</code></pre><ul>
<li>After matching the values and creating some new matches I had trouble remembering how to copy the reconciled values to a new column
<ul>
<li>The matched values can be accessed with <code>cell.recon.match.name</code>, but some of the new values don’t appear, perhaps because I edited the original cell values?</li>
<li>I ended up using this GREL expression to copy all values to a new column:</li>
<li>See the <ahref="https://github.com/OpenRefine/OpenRefine/wiki/Variables#recon">OpenRefine variables documentation</a> for more notes about the <code>recon</code> object</li>
<li>I also noticed a handful of errors in our current list of affiliations so I corrected them:</li>
<li>We should create a new list of affiliations to update our controlled vocabulary again</li>
<li>I dumped a list of the top 1500 affiliations:</li>
</ul>
<pretabindex="0"><code>dspace=# \COPY (SELECT DISTINCT text_value, count(*) FROM metadatavalue WHERE metadata_field_id = 211 AND resource_type_id = 2 GROUP BY text_value ORDER BY count DESC LIMIT 1500) to /tmp/2019-04-08-top-1500-affiliations.csv WITH CSV HEADER;
COPY 1500
</code></pre><ul>
<li>Fix a few more messed up affiliations that have return characters in them (use Ctrl-V Ctrl-M to re-create control character):</li>
</ul>
<pretabindex="0"><code>dspace=# UPDATE metadatavalue SET text_value='International Institute for Environment and Development' WHERE resource_type_id = 2 AND metadata_field_id = 211 AND text_value LIKE 'International Institute^M%';
dspace=# UPDATE metadatavalue SET text_value='Kenya Agriculture and Livestock Research Organization' WHERE resource_type_id = 2 AND metadata_field_id = 211 AND text_value LIKE 'Kenya Agricultural and Livestock Research^M%';
</code></pre><ul>
<li>I noticed a bunch of subjects and affiliations that use stylized apostrophes so I will export those and then batch update them:</li>
</ul>
<pretabindex="0"><code>dspace=# \COPY (SELECT DISTINCT text_value FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 211 AND text_value LIKE '%’%') to /tmp/2019-04-08-affiliations-apostrophes.csv WITH CSV HEADER;
COPY 60
dspace=# \COPY (SELECT DISTINCT text_value FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 57 AND text_value LIKE '%’%') to /tmp/2019-04-08-subject-apostrophes.csv WITH CSV HEADER;
COPY 20
</code></pre><ul>
<li>I cleaned them up in OpenRefine and then applied the fixes on CGSpace and DSpace Test:</li>
org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-319] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:250; busy:250; idle:0; lastwait:5000].
</code></pre><ul>
<li>But still I see 10 to 30% CPU steal in <code>iostat</code> that is also reflected in the Munin graphs:</li>
<li>Linode Support still didn’t respond to my ticket from yesterday, so I attached a new output of <code>iostat 1 10</code> and asked them to move the VM to a less busy host</li>
<li><code>45.5.186.2</code> is at CIAT in Colombia and I see they are mostly making requests to the REST API, but also to XMLUI with the following user agent:</li>
</ul>
<pretabindex="0"><code>Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36
<li>Ironically I do still see some 2 to 10% of CPU steal in <code>iostat 1 10</code></li>
<li>Leroy from CIAT contacted me to say he knows the team who is making all those requests to CGSpace
<ul>
<li>I told them how to use the REST API to get the CIAT Datasets collection and enumerate its items</li>
</ul>
</li>
<li>In other news, Linode staff identified a noisy neighbor sharing our host and migrated it elsewhere last night</li>
</ul>
<h2id="2019-04-10">2019-04-10</h2>
<ul>
<li>Abenet pointed out a possibility of validating funders against the <ahref="https://support.crossref.org/hc/en-us/articles/215788143-Funder-data-via-the-API">CrossRef API</a></li>
<li>Note that if you use HTTPS and specify a contact address in the API request you have less likelihood of being blocked</li>
<li>Otherwise, they provide the funder data in <ahref="https://www.crossref.org/services/funder-registry/">CSV and RDF format</a></li>
<li>I did a quick test with the recent IITA records against reconcile-csv in OpenRefine and it matched a few, but the ones that didn’t match will need a human to go and do some manual checking and informed decision making…</li>
<li>If I want to write a script for this I could use the Python <ahref="https://habanero.readthedocs.io/en/latest/modules/crossref.html">habanero library</a>:</li>
<li>Continue proofing IITA’s last round of batch uploads from <ahref="https://dspacetest.cgiar.org/handle/10568/100333">March on DSpace Test</a> (20193rd.xls)
<ul>
<li>One misspelled country</li>
<li>Three incorrect regions</li>
<li>Potential duplicates (same DOI, similar title, same authors):
<ul>
<li><ahref="https://dspacetest.cgiar.org/handle/10568/100580">10568/100580</a> and <ahref="https://dspacetest.cgiar.org/handle/10568/100579">10568/100579</a></li>
<li><ahref="https://dspacetest.cgiar.org/handle/10568/100444">10568/100444</a> and <ahref="https://dspacetest.cgiar.org/handle/10568/100423">10568/100423</a></li>
</ul>
</li>
<li>Two DOIs with incorrect URL formatting</li>
<li>Two misspelled IITA subjects</li>
<li>Two authors with smart quotes</li>
<li>Lots of issues with sponsors</li>
<li>One misspelled “Peer review”</li>
<li>One invalid ISBN that I fixed by Googling the title</li>
<li>Lots of issues with sponsors (German Aid Agency, Swiss Aid Agency, Italian Aid Agency, Dutch Aid Agency, etc)</li>
<li>I validated all the AGROVOC subjects against our latest list with reconcile-csv
<ul>
<li>About 720 of the 900 terms were matched, then I checked and fixed or deleted the rest manually</li>
</ul>
</li>
</ul>
</li>
<li>I captured a few general corrections and deletions for AGROVOC subjects while looking at IITA’s records, so I applied them to DSpace Test and CGSpace:</li>
<li>Answer more questions about DOIs and Altmetric scores from WLE</li>
<li>Answer more questions about DOIs and Altmetric scores from IWMI
<ul>
<li>They can’t seem to understand the Altmetric + Twitter flow for associating Handles and DOIs</li>
<li>To make things worse, many of their items DON’T have DOIs, so when Altmetric harvests them of course there is no link! - Then, a bunch of their items don’t have scores because they never tweeted them!</li>
<li>They added a DOI to this old item <ahref="https://cgspace.cgiar.org/handle/10568/97087">10567/97087</a> this morning and wonder why Altmetric’s score hasn’t linked with the DOI magically</li>
<li>We should check in a week to see if Altmetric will make the association after one week when they harvest again</li>
</ul>
</li>
</ul>
<h2id="2019-04-13">2019-04-13</h2>
<ul>
<li>I copied the <code>statistics</code> and <code>statistics-2018</code> Solr cores from CGSpace to my local machine and watched the Java process in VisualVM while indexing item views and downloads with my <ahref="https://github.com/ilri/dspace-statistics-api">dspace-statistics-api</a>:</li>
</ul>
<p><imgsrc="/cgspace-notes/2019/04/visualvm-solr-indexing.png"alt="Java GC during Solr indexing with CMS"></p>
<ul>
<li>It took about eight minutes to index 784 pages of item views and 268 of downloads, and you can see a clear “sawtooth” pattern in the garbage collection</li>
<li>I am curious if the GC pattern would be different if I switched from the <code>-XX:+UseConcMarkSweepGC</code> to G1GC</li>
<li>I switched to G1GC and restarted Tomcat but for some reason I couldn’t see the Tomcat PID in VisualVM…
<ul>
<li>Anyways, the indexing process took much longer, perhaps twice as long!</li>
</ul>
</li>
<li>I tried again with the GC tuning settings from the Solr 4.10.4 release:</li>
</ul>
<p><imgsrc="/cgspace-notes/2019/04/visualvm-solr-indexing-solr-settings.png"alt="Java GC during Solr indexing Solr 4.10.4 settings"></p>
<h2id="2019-04-14">2019-04-14</h2>
<ul>
<li>Change DSpace Test (linode19) to use the Java GC tuning from the Solr 4.10.4 startup script:</li>
<li>I need to remember to check the Munin JVM graphs in a few days</li>
<li>It might be placebo, but the site <em>does</em> feel snappier…</li>
</ul>
<h2id="2019-04-15">2019-04-15</h2>
<ul>
<li>Rework the dspace-statistics-api to use the vanilla Python requests library instead of Solr client
<ul>
<li><ahref="https://github.com/ilri/dspace-statistics-api/releases/tag/v1.0.0">Tag version 1.0.0</a> and deploy it on DSpace Test</li>
</ul>
</li>
<li>Pretty annoying to see CGSpace (linode18) with 20–50% CPU steal according to <code>iostat 1 10</code>, though I haven’t had any Linode alerts in a few days</li>
<li>Abenet sent me a list of ILRI items that don’t have CRPs added to them
<ul>
<li>The spreadsheet only had Handles (no IDs), so I’m experimenting with using Python in OpenRefine to get the IDs</li>
<li>I cloned the handle column and then did a transform to get the IDs from the CGSpace REST API:</li>
<li>Export IITA’s community from CGSpace because they want to experiment with importing it into their internal DSpace for some testing or something</li>
</ul>
<h2id="2019-04-17">2019-04-17</h2>
<ul>
<li>Reading an interesting <ahref="https://teaspoon-consulting.com/articles/solr-cache-tuning.html">blog post about Solr caching</a></li>
<li>Did some tests of the dspace-statistics-api on my local DSpace instance with 28 million documents in a sharded statistics core (<code>statistics</code> and <code>statistics-2018</code>) and monitored the memory usage of Tomcat in VisualVM</li>
<li>4GB heap, CMS GC, 512 filter cache, 512 query cache, with 28 million documents in two shards
<ul>
<li>Run 1:
<ul>
<li>Time: 3.11s user 0.44s system 0% cpu 13:45.07 total</li>
<li>Tomcat (not Solr) max JVM heap usage: 2.04 GiB</li>
</ul>
</li>
<li>Run 2:
<ul>
<li>Time: 3.23s user 0.43s system 0% cpu 13:46.10 total</li>
<li>Tomcat (not Solr) max JVM heap usage: 2.06 GiB</li>
</ul>
</li>
<li>Run 3:
<ul>
<li>Time: 3.23s user 0.42s system 0% cpu 13:14.70 total</li>
<li>Tomcat (not Solr) max JVM heap usage: 2.13 GiB</li>
<li>The biggest takeaway I have is that this workload benefits from a larger <code>filterCache</code> (for Solr fq parameter), but barely uses the <code>queryResultCache</code> (for Solr q parameter) at all
<ul>
<li>The number of hits goes up and the time taken decreases when we increase the <code>filterCache</code>, and total JVM heap memory doesn’t seem to increase much at all</li>
<li>I guess the <code>queryResultCache</code> size is always 2 because I’m only doing two queries: <code>type:0</code> and <code>type:2</code> (downloads and views, respectively)</li>
</ul>
</li>
<li>Here is the general pattern of running three sequential indexing runs as seen in VisualVM while monitoring the Tomcat process:</li>
<li>I ran one test with a <code>filterCache</code> of 16384 to try to see if I could make the Tomcat JVM memory balloon, but actually it <em>drastically</em> increased the performance and memory usage of the dspace-statistics-api indexer</li>
<li>4GB heap, CMS GC, 16384 filter cache, 512 query cache, with 28 million documents in two shards
<ul>
<li>Run 1:
<ul>
<li>Time: 2.85s user 0.42s system 2% cpu 2:28.92 total</li>
<li>I’ve been trying to copy the <code>statistics-2018</code> Solr core from CGSpace to DSpace Test since yesterday, but the network speed is like 20KiB/sec
<ul>
<li>I opened a support ticket to ask Linode to investigate</li>
<li>They asked me to send an <code>mtr</code> report from Fremont to Frankfurt and vice versa</li>
</ul>
</li>
<li>Deploy Tomcat 7.0.94 on DSpace Test (linode19)
<ul>
<li>Also, I realized that the CMS GC changes I deployed a few days ago were ignored by Tomcat because of something with how Ansible formatted the options string</li>
<li>I needed to use the “folded” YAML variable format <code>>-</code> (with the dash so it doesn’t add a return at the end)</li>
</ul>
</li>
<li>UptimeRobot says that CGSpace went “down” this afternoon, but I looked at the CPU steal with <code>iostat 1 10</code> and it’s in the 50s and 60s
<ul>
<li>The munin graph shows a lot of CPU steal (red) currently (and over all during the week):</li>
<li>I opened a ticket with Linode to migrate us somewhere
<ul>
<li>They agreed to migrate us to a quieter host</li>
</ul>
</li>
</ul>
<h2id="2019-04-20">2019-04-20</h2>
<ul>
<li>Linode agreed to move CGSpace (linode18) to a new machine shortly after I filed my ticket about CPU steal two days ago and now the load is much more sane:</li>
[ 5] local 45.79.x.x port 36440 connected with 139.162.x.x port 5001
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.2 sec 172 MBytes 142 Mbits/sec
[ 4] 0.0-10.5 sec 202 MBytes 162 Mbits/sec
</code></pre><ul>
<li>Even with the software firewalls disabled the rsync speed was low, so it’s not a rate limiting issue</li>
<li>I also tried to download a file over HTTPS from CGSpace to DSpace Test, but it was capped at 20KiB/sec
<ul>
<li>I updated the Linode issue with this information</li>
</ul>
</li>
<li>I’m going to try to switch the kernel to the latest upstream (5.0.8) instead of Linode’s latest x86_64
<ul>
<li>Nope, still 20KiB/sec</li>
</ul>
</li>
</ul>
<h2id="2019-04-21">2019-04-21</h2>
<ul>
<li>Deploy Solr 4.10.4 on CGSpace (linode18)</li>
<li>Deploy Tomcat 7.0.94 on CGSpace</li>
<li>Deploy dspace-statistics-api v1.0.0 on CGSpace</li>
<li>Linode support replicated the results I had from the network speed testing and said they don’t know why it’s so slow
<ul>
<li>They offered to live migrate the instance to another host to see if that helps</li>
</ul>
</li>
</ul>
<h2id="2019-04-22">2019-04-22</h2>
<ul>
<li>Abenet pointed out <ahref="https://hdl.handle.net/10568/97912">an item</a> that doesn’t have an Altmetric score on CGSpace, but has a score of 343 in the CGSpace Altmetric dashboard
<ul>
<li>I tweeted the Handle to see if it will pick it up…</li>
<li>Like clockwork, after fifteen minutes there was a donut showing on CGSpace</li>
</ul>
</li>
<li>I want to get rid of this annoying warning that is constantly in our DSpace logs:</li>
</ul>
<pretabindex="0"><code>2019-04-08 19:02:31,770 WARN org.dspace.xoai.services.impl.xoai.DSpaceRepositoryConfiguration @ { OAI 2.0 :: DSpace } Not able to retrieve the dspace.oai.url property from oai.cfg. Falling back to request address
</code></pre><ul>
<li>Apparently it happens once per request, which can be at least 1,500 times per day according to the DSpace logs on CGSpace (linode18):</li>
</ul>
<pretabindex="0"><code>$ grep -c 'Falling back to request address' dspace.log.2019-04-20
dspace.log.2019-04-20:1515
</code></pre><ul>
<li>I will fix it in <code>dspace/config/modules/oai.cfg</code></li>
<li>Linode says that it is likely that the host CGSpace (linode18) is on is showing signs of hardware failure and they recommended that I migrate the VM to a new host
<ul>
<li>I told them to migrate it at 04:00:00AM Frankfurt time, when nobody in East Africa, Europe, or South America should be using the server</li>
</ul>
</li>
</ul>
<h2id="2019-04-23">2019-04-23</h2>
<ul>
<li>One blog post says that there is <ahref="https://kvaes.wordpress.com/2017/07/01/what-azure-virtual-machine-size-should-i-pick/">no overprovisioning in Azure</a>:</li>
</ul>
<!-- raw HTML omitted -->
<ul>
<li>Perhaps that’s why the <ahref="https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/">Azure pricing</a> is so expensive!</li>
<li>Add a privacy page to CGSpace
<ul>
<li>The work was mostly similar to the About page at <code>/page/about</code>, but in addition to adding i18n strings etc, I had to add the logic for the trail to <code>dspace-xmlui-mirage2/src/main/webapp/xsl/preprocess/general.xsl</code></li>
</ul>
</li>
</ul>
<h2id="2019-04-24">2019-04-24</h2>
<ul>
<li>Linode migrated CGSpace (linode18) to a new host, but I am still getting poor performance when copying data to DSpace Test (linode19)
<ul>
<li>I asked them if we can migrate DSpace Test to a new host</li>
<li>They migrated DSpace Test to a new host and the rsync speed from Frankfurt was still capped at 20KiB/sec…</li>
<li>I booted DSpace Test to a rescue CD and tried the rsync from CGSpace there too, but it was still capped at 20KiB/sec…</li>
<li>I copied the 18GB <code>statistics-2018</code> Solr core from Frankfurt to a Linode in London at 15MiB/sec, then from the London one to DSpace Test in Fremont at 15MiB/sec… so WTF us up with Frankfurt→Fremont?!</li>
</ul>
</li>
<li>Finally upload the 218 IITA items from March to CGSpace
<ul>
<li>Abenet and I had to do a little bit more work to correct the metadata of one item that appeared to be a duplicate, but really just had the wrong DOI</li>
</ul>
</li>
<li>While I was uploading the IITA records I noticed that twenty of the records Sisay uploaded in 2018-09 had double Handles (<code>dc.identifier.uri</code>)
<ul>
<li>According to my notes in 2018-09 I had noticed this when he uploaded the records and told him to remove them, but he didn’t…</li>
<li>I exported the IITA community as a CSV then used <code>csvcut</code> to extract the two URI columns and identify and fix the records:</li>
<li>Carlos Tejo from the Land Portal had been emailing me this week to ask about the old REST API that Tsega was building in 2017
<ul>
<li>I told him we never finished it, and that he should try to use the <code>/items/find-by-metadata-field</code> endpoint, with the caveat that you need to match the language attribute exactly (ie “en”, “en_US”, null, etc)</li>
<li>I asked him how many terms they are interested in, as we could probably make it easier by normalizing the language attributes of these fields (it would help us anyways)</li>
<li>He says he’s getting HTTP 401 errors when trying to search for CPWF subject terms, which I can reproduce:</li>
<li>Note that curl only shows the HTTP 401 error if you use <code>-f</code> (fail), and only then if you <em>don’t</em> include <code>-s</code>
<ul>
<li>I see there are about 1,000 items using CPWF subject “WATER MANAGEMENT” in the database, so there should definitely be results</li>
<li>The breakdown of <code>text_lang</code> fields used in those items is 942:</li>
</ul>
</li>
</ul>
<pretabindex="0"><code>dspace=# SELECT COUNT(text_value) FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=208 AND text_value='WATER MANAGEMENT' AND text_lang='en_US';
count
-------
376
(1 row)
dspace=# SELECT COUNT(text_value) FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=208 AND text_value='WATER MANAGEMENT' AND text_lang='';
count
-------
149
(1 row)
dspace=# SELECT COUNT(text_value) FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=208 AND text_value='WATER MANAGEMENT' AND text_lang IS NULL;
count
-------
417
(1 row)
</code></pre><ul>
<li>I see that the HTTP 401 issue seems to be a bug due to an item that the user doesn’t have permission to access… from the DSpace log:</li>
</ul>
<pretabindex="0"><code>2019-04-24 08:11:51,129 INFO org.dspace.rest.ItemsResource @ Looking for item with metadata(key=cg.subject.cpwf,value=WATER MANAGEMENT, language=en_US).
2019-04-24 08:11:51,231 INFO org.dspace.usage.LoggerUsageEventListener @ anonymous::view_item:handle=10568/72448
2019-04-24 08:11:51,238 INFO org.dspace.usage.LoggerUsageEventListener @ anonymous::view_item:handle=10568/72491
2019-04-24 08:11:51,243 INFO org.dspace.usage.LoggerUsageEventListener @ anonymous::view_item:handle=10568/75703
2019-04-24 08:11:51,252 ERROR org.dspace.rest.ItemsResource @ User(anonymous) has not permission to read item!
</code></pre><ul>
<li>Nevertheless, if I request using the <code>null</code> language I get 1020 results, plus 179 for a blank language attribute:</li>
<li>This is weird because I see 942–1156 items with “WATER MANAGEMENT” (depending on wildcard matching for errors in subject spelling):</li>
</ul>
<pretabindex="0"><code>dspace=# SELECT COUNT(text_value) FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=208 AND text_value='WATER MANAGEMENT';
count
-------
942
(1 row)
dspace=# SELECT COUNT(text_value) FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=208 AND text_value LIKE '%WATER MANAGEMENT%';
count
-------
1156
(1 row)
</code></pre><ul>
<li>I sent a message to the dspace-tech mailing list to ask for help</li>
</ul>
<h2id="2019-04-25">2019-04-25</h2>
<ul>
<li>Peter pointed out that we need to remove Delicious and Google+ from our social sharing links
<ul>
<li>Also, it would be nice if we could include the item title in the shared link</li>
<li>I created an issue on GitHub to track this (<ahref="https://github.com/ilri/DSpace/issues/419">#419</a>)</li>
</ul>
</li>
<li>I tested the REST API after logging in with my super admin account and I was able to get results for the problematic query:</li>
74648 | 113 | f | f | 2016-03-30 09:00:52.131+00 | | t
(1 row)
</code></pre><ul>
<li>I tried to delete the item in the web interface, and it seems successful, but I can still access the item in the admin interface, and nothing changes in PostgreSQL</li>
<li>Meet with CodeObia to see progress on AReS version 2</li>
<li>Marissa Van Epp asked me to add a few new metadata values to their Phase II Project Tags field (cg.identifier.ccafsprojectpii)
<ul>
<li>I created a <ahref="https://github.com/ilri/DSpace/pull/420">pull request</a> for it and will do it the next time I run updates on CGSpace</li>
</ul>
</li>
<li>Communicate with Carlos Tejo from the Land Portal about the <code>/items/find-by-metadata-value</code> endpoint</li>
<li>Run all system updates on DSpace Test (linode19) and reboot it</li>
</ul>
<h2id="2019-04-26">2019-04-26</h2>
<ul>
<li>Export a list of authors for Peter to look through:</li>
</ul>
<pretabindex="0"><code>dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/2019-04-26-all-authors.csv with csv header;
COPY 65752
</code></pre><h2id="2019-04-28">2019-04-28</h2>
<ul>
<li>Still trying to figure out the issue with the items that cause the REST API’s <code>/items/find-by-metadata-value</code> endpoint to throw an exception
<ul>
<li>I made the item private in the UI and then I see in the UI and PostgreSQL that it is no longer discoverable:</li>
</ul>
</li>
</ul>
<pretabindex="0"><code>dspace=# SELECT * FROM item WHERE item_id=74648;
74648 | 113 | f | f | 2019-04-28 08:48:52.114-07 | | f
(1 row)
</code></pre><ul>
<li>And I tried the <code>curl</code> command from above again, but I still get the HTTP 401 and and the same error in the DSpace log:</li>
</ul>
<pretabindex="0"><code>2019-04-28 08:53:07,170 ERROR org.dspace.rest.ItemsResource @ User(anonymous) has not permission to read item(id=74648)!
</code></pre><ul>
<li>I even tried to “expunge” the item using an <ahref="https://wiki.lyrasis.org/display/DSDOC5x/Batch+Metadata+Editing#BatchMetadataEditing-Performing'actions'onitems">action in CSV</a>, and it said “EXPUNGED!” but the item is still there…</li>
</ul>
<h2id="2019-04-30">2019-04-30</h2>
<ul>
<li>Send mail to the dspace-tech mailing list to ask about the item expunge issue</li>
<li>Delete and re-create Podman container for dspacedb after pulling a new PostgreSQL container:</li>
<li>Carlos from LandPortal asked if I could export CGSpace in a machine-readable format so I think I’ll try to do a CSV
<ul>
<li>In order to make it easier for him to understand the CSV I will normalize the text languages (minus the provenance field) on my local development instance before exporting:</li>
</ul>
</li>
</ul>
<pretabindex="0"><code>dspace=# SELECT DISTINCT text_lang, count(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id != 28 GROUP BY text_lang;
text_lang | count
-----------+---------
| 358647
* | 11
E. | 1
en | 1635
en_US | 602312
es | 12
es_ES | 2
ethnob | 1
fr | 2
spa | 2
| 1074345
(11 rows)
dspace=# UPDATE metadatavalue SET text_lang='en_US' WHERE resource_type_id=2 AND metadata_field_id != 28 AND text_lang IN ('ethnob', 'en', '*', 'E.', '');
UPDATE 360295
dspace=# UPDATE metadatavalue SET text_lang='en_US' WHERE resource_type_id=2 AND metadata_field_id != 28 AND text_lang IS NULL;
UPDATE 1074345
dspace=# UPDATE metadatavalue SET text_lang='es_ES' WHERE resource_type_id=2 AND metadata_field_id != 28 AND text_lang IN ('es', 'spa');
UPDATE 14
</code></pre><ul>
<li>Then I exported the whole repository as CSV, imported it into OpenRefine, removed a few unneeded columns, exported it, zipped it down to 36MB, and emailed a link to Carlos</li>
<li>In other news, while I was looking through the CSV in OpenRefine I saw lots of weird values in some fields… we should check, for example:
