Maria asked me to update Charles Staver’s ORCID iD in the submission template and on CGSpace, as his name was lower case before, and now he has corrected it
Maria asked me to update Charles Staver’s ORCID iD in the submission template and on CGSpace, as his name was lower case before, and now he has corrected it
<li>Maria asked me to update Charles Staver’s ORCID iD in the submission template and on CGSpace, as his name was lower case before, and now he has corrected it
<ul>
<li>I updated the fifty-eight existing items on CGSpace</li>
</ul>
</li>
<li>Looking into the items Udana had asked about last week that were missing Altmetric donuts:
<li><ahref="https://hdl.handle.net/10568/103225">The first</a> is still missing its DOI, so I added it and <ahref="https://twitter.com/mralanorth/status/1245632619661766657">tweeted its handle</a> (after a few hours there was a donut with score 222)</li>
<li><ahref="https://hdl.handle.net/10568/106899">The second item</a> now has a donut with score 2 since I <ahref="https://twitter.com/mralanorth/status/1243158045540134913">tweeted its handle</a> last week</li>
<li><ahref="https://hdl.handle.net/10568/107258">The third item</a> now has a donut with score 1 since I <ahref="https://twitter.com/mralanorth/status/1243158786392625153">tweeted it</a> last week</li>
</ul>
</li>
<li>On the same note, the <ahref="https://hdl.handle.net/10568/106573">one item</a> Abenet pointed out last week now has a donut with score of 104 after I <ahref="https://twitter.com/mralanorth/status/1243163710241345536">tweeted it</a> last week</li>
<li>Altmetric responded about <ahref="https://hdl.handle.net/10568/101286">one item</a> that had no donut since at least 2019-12 and said they fixed some problems with their bot’s user agent
<ul>
<li>I decided to <ahref="https://twitter.com/mralanorth/status/1245703049445851140">tweet the item</a>, as I can’t remember if I ever did it before</li>
<li>Update PostgreSQL JDBC driver to version 42.2.12</li>
</ul>
<h2id="2020-04-07">2020-04-07</h2>
<ul>
<li>Yesterday Atmire sent me their <ahref="https://github.com/ilri/DSpace/pull/445">pull request for DSpace 6 modules</a></li>
<li>Peter pointed out that some items have his ORCID identifier (<code>cg.creator.id</code>) twice
<ul>
<li>I think this is because my early <code>add-orcid-identifiers.py</code> script was adding identifiers to existing records without properly checking if there was already one present (at first it only checked if there was one with the exact <code>place</code> value)</li>
<li>As a test I dropped all his ORCID identifiers and added them back with the <code>add-orcid-identifiers.py</code> script:</li>
</ul>
</li>
</ul>
<pre><code>$ psql -h localhost -U postgres dspace -c "DELETE FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=240 AND text_value LIKE '%Ballantyne%';"
<li>I used this CSV with the script (all records with his name have the name standardized like this):</li>
</ul>
<pre><code>dc.contributor.author,cg.creator.id
"Ballantyne, Peter G.","Peter G. Ballantyne: 0000-0001-9346-2893"
</code></pre><ul>
<li>Then I tried another way, to identify all duplicate ORCID identifiers for a given resource ID and group them so I can see if count is greater than 1:</li>
</ul>
<pre><code>dspace=# \COPY (SELECT DISTINCT(resource_id, text_value) as distinct_orcid, COUNT(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 240 GROUP BY distinct_orcid ORDER BY count DESC) TO /tmp/2020-04-07-duplicate-orcids.csv WITH CSV HEADER;
COPY 15209
</code></pre><ul>
<li>Of those, about nine authors had duplicate ORCID identifiers over about thirty records, so I created a CSV with all their name variations and ORCID identifiers:</li>
</ul>
<pre><code>dc.contributor.author,cg.creator.id
"Ballantyne, Peter G.","Peter G. Ballantyne: 0000-0001-9346-2893"
<li>Then I deleted <em>all</em> their existing ORCID identifier records:</li>
</ul>
<pre><code>dspace=# DELETE FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=240 AND text_value SIMILAR TO '%(0000-0001-6543-0798|0000-0001-9346-2893|0000-0002-6950-4018|0000-0002-7583-3811|0000-0002-8044-583X|0000-0002-8599-7895|0000-0003-0934-1218|0000-0003-2765-7101)%';
DELETE 994
</code></pre><ul>
<li>And then I added them again using the <code>add-orcid-identifiers</code> records:</li>
<li>I started testing the <ahref="https://github.com/ilri/DSpace/pull/445">pull request</a> sent by Atmire yesterday
<ul>
<li>I notice that we now need <code>yarn</code> to build, and I need to bump the Node.js <code>engine</code> version in our Mirage 2 theme in order to get it to build on Node.js 10.x</li>
<li>Font Awesome icons for GitHub etc weren’t loading, and after a bit of troubleshooting I replaced version 4.5.0 with 5.13.0 and to my surprise they now include Mendeley and ORCID so we can get rid of the Academicons dependency</li>
</ul>
</li>
</ul>
<h2id="2020-04-12">2020-04-12</h2>
<ul>
<li>Testing the Atmire DSpace 6.3 code with a clean CGSpace DSpace 5.8 database snapshot
<ul>
<li>One Flyway migration failed so I had to manually remove it (and of course create the pgcrypto extension):</li>
</ul>
</li>
</ul>
<pre><code>dspace63=# DELETE FROM schema_version WHERE version IN ('5.8.2015.12.03.3');
dspace63=# CREATE EXTENSION pgcrypto;
</code></pre><ul>
<li>Then DSpace 6.3 started up OK and I was able to see some statistics in the Content and Usage Analysis (CUA) module, but not on community, collection, or item pages
<ul>
<li>I also noticed at least one of these errors in the DSpace log:</li>
<li>Run system updates on DSpace Test (linode26) and reboot it</li>
<li>More work on the DSpace 6.3 stuff, improving the GDPR consent logic to use <ahref="https://github.com/chiiya/haven">haven</a> instead of cookieconsent
<ul>
<li>It works better by injecting the Google Analytics script after the user clicks agree, and it also has a preferences section that gets automatically injected on the privacy page!</li>
<li>I realized that <code>solr-upgrade-statistics-6x</code> only processes 100,000 records by default so I think we actually need to finish running it for all legacy Solr records before asking Atmire why CUA statlets and detailed statistics aren’t working</li>
<li>For now I am just doing 250,000 records at a time on my local environment:</li>
<li>Despite running the migration for all of my local 1.5 million Solr records, I still see a few hundred thousand like <code>-1</code> and <code>0-unmigrated</code>
<ul>
<li>I will purge them all and try to import only a subset…</li>
<li>After importing again I see there are indeed tens of thousands of these documents with IDs “-1” and “0”</li>
<li>They are all <code>type: 5</code>, which is “SITE” according to <code>Constants.java</code>:</li>
</ul>
</li>
</ul>
<pre><code>/** DSpace site type */
public static final int SITE = 5;
</code></pre><ul>
<li>Even after deleting those documents and re-running <code>solr-upgrade-statistics-6x</code> I still get the UUID errors when using CUA and the statlets</li>
<li>I have sent some feedback and questions to Atmire (including about the  issue with glypicons in the header trail)</li>
<li>In other news, my local Artifactory container stopped working for some reason so I re-created it and it seems some things have changed upstream (port 8082 for web UI?):</li>
<li>A few days ago Peter asked me to update an author’s name on CGSpace and in the controlled vocabularies:</li>
</ul>
<pre><code>dspace=# UPDATE metadatavalue SET text_value='Knight-Jones, Theodore J.D.' WHERE resource_type_id=2 AND metadata_field_id=3 AND text_value='Knight-Jones, T.J.D.';
</code></pre><ul>
<li>I updated his existing records on CGSpace, changed the controlled lists, added his ORCID identifier to the controlled list, and tagged his thirty-nine items with the ORCID iD</li>
<li>The new DSpace 6 stuff that Atmire sent modifies the Mirage 2’s <code>pom.xml</code> to copy the each theme’s resulting <code>node_modules</code> to each theme after building and installing with <code>ant update</code> because they moved some packages from bower to npm and now reference them in <code>page-structure.xsl</code>
<ul>
<li>This is a good idea, because bower is no longer supported, and npm has gotten a lot better, but it causes an extra 200,000 files to get copied!</li>
<li>Most scripts are concatenated into <code>theme.js</code> during build, so we don’t need the <code>node_modules</code> after that, but there are three scripts in <code>page-structure.xsl</code> that are not included there</li>
<li>The scripts are a very old version of modernizr which is not even available on npm, html5shiv, and respond.js</li>
<li>For modernizr I can simply download a static copy and put it in <code>0_CGIAR/scripts</code> and concatenate it into <code>theme.js</code></li>
<li>For the others, I can revert to using them from bower’s <code>vendor</code> directory, which is installed by the parent XMLUI Mirage 2 theme</li>
<li>During this process I also realized that <code>mvn clean</code> doesn’t actually clean everything, and <code>dspace/modules/xmlui-mirage2/target</code> is remaining from previous builds and contains a bunch of shit from previous builds (including all the themes which I was trying to build without!)
<ul>
<li>This must be a DSpace bug, but I should theoretically check on vanilla DSpace and then file a bug…</li>
<li>Atmire responded to some of the issues I raised earlier this week about the DSpace 6 pull request
<ul>
<li>They said they don’t think the glyphicon encoding issue is due to their changes, but I built a new clean version of the vanilla <code>6_x-dev</code> branch from before their pull request and it <em>does not</em> have the encoding issue in the Mirage 2 header trails</li>
<li>Also, they said we need to use something called <code>AtomicStatisticsUpdateCLI</code> to do the Solr legacy integer ID to UUID conversion so I asked for more information about that workflow</li>
(DEBUG) Checking for hits from spider IP: 91.241.19.70
Purging 8909 hits from 91.241.19.70 in statistics
Total number of bot hits purged: 8909
</code></pre><ul>
<li>While investigating that I noticed ORCID identifiers missing from a few authors names, so I added them with my <code>add-orcid-identifiers.py</code> script:</li>
<li>I researched a bit more about the  issue with glypicons and realized it’s due to <ahref="https://github.com/twbs/bootstrap-sass/issues/919">a bug in libsass</a>
<ul>
<li>I have Ruby sass version 3.4.25 installed in my local environment, but DSpace Mirage 2 is supposed to build with 3.3.14</li>
<li>Downgrading the version fixes it, though I wonder: why did I not have this issue on the <code>6.x-dev</code> branch before Atmire’s pull request, and also on <code>5_x-prod</code> where I’ve been building for a few months here…</li>
</ul>
</li>
<li>I deployed the latest <code>5_x-prod</code> branch on CGSpace (linode18), ran all updates, and rebooted the server
<ul>
<li>This includes the “COVID19” ILRI subject, the new CCAFS Phase II project tags, and the changes to an ILRI author name</li>
<li>After restarting the server I had to restart Tomcat three times before all Solr statistics cores loaded properly.</li>
<li>After that I started a full Discovery reindexing to pick up some author changes I made last week:</li>
<li>I spent some time working on the XMLUI themes in DSpace 6
<ul>
<li>Atmire’s pull request modifies <code>pom.xml</code> to <em>not</em> exclude <code>node_modules</code>, which means an extra ~260,000 files get copied to our installation folder because of all the themes</li>
<li>I worked on the <code>Gruntfile.js</code> to copy Font Awesome and Bootstrap glyphicon fonts out of <code>node_modules</code> and into <code>fonts</code> at build time, but still <code>jquery-ui.min.css</code> was being referenced as a <code>url()</code> in CSS</li>
<li>SASS can include imported CSS in your compiled CSS—instead of including an <code>@import url(..)</code> if you import it without the “.css”, but our version of Ruby SASS doesn’t support that</li>
<li>I hacked <code>Gruntfile.js</code> to use <code>dart-sass</code> instead of Ruby <code>compass</code> (including installing compass’s mixins via npm!) but then <ahref="https://github.com/sass/dart-sass/issues/345">dart-sass converts all the glyphicon ASCII escape codes to Unicode literals</a> and they show up garbled in Firefox</li>
<li>I tried to use <code>node-sass</code> instead of dart-sass and it doesn’t replace the ASCII escapes with literals, but then I get the the  issue with the glyphicon in the header trail again! Back to square one!</li>
<li>So that was a waste of five hours…</li>
<li>I might just leave this <ahref="https://github.com/twbs/bootstrap-sass/issues/919">tiny hack</a> in <code>0_CGIAR/styles/_style.scss</code> to override this and be done with it:</li>
<li>File an issue on DSpace Jira about the <ahref="https://jira.lyrasis.org/browse/DS-4492"><code>mvn clean</code> task not removing the Mirage 2 target directory</a></li>
<li>My changes to DSpace XMLUI Mirage 2 build process mean that we don’t need Ruby gems at all anymore! We can completely build without them!</li>
<li>I manually edited the CUA version variable and was then able to run the <code>com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI</code> script
<li>On the third run it took one hour to process 150,000 records</li>
</ul>
</li>
</ul>
<h2id="2020-04-29">2020-04-29</h2>
<ul>
<li>I found out that running the Atmire CUA script with more memory and a larger number of records (-r 20000) makes it run faster
<ul>
<li>Now the process finishes, but there are errors on some records:</li>
</ul>
</li>
</ul>
<pre><code>Record uid: ee085cc0-0110-42c5-80b9-0fad4015ed9f couldn't be processed
com.atmire.statistics.util.update.atomic.ProcessingException: something went wrong while processing record uid: ee085cc0-0110-42c5-80b9-0fad4015ed9f, an error occured in the com.atmire.statistics.util.update.atomic.processor.ContainerOwnerDBProcessor
at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.applyProcessors(AtomicStatisticsUpdater.java:304)
at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.processRecords(AtomicStatisticsUpdater.java:176)
at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.performRun(AtomicStatisticsUpdater.java:161)
at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.update(AtomicStatisticsUpdater.java:128)
at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI.main(AtomicStatisticsUpdateCLI.java:78)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
Caused by: java.lang.NullPointerException
at com.atmire.dspace.cua.CUADSOServiceImpl.findByLegacyID(CUADSOServiceImpl.java:40)
at com.atmire.statistics.util.update.atomic.processor.AtomicUpdateProcessor.getDSpaceObject(AtomicUpdateProcessor.java:49)
at com.atmire.statistics.util.update.atomic.processor.ContainerOwnerDBProcessor.process(ContainerOwnerDBProcessor.java:45)
at com.atmire.statistics.util.update.atomic.processor.ContainerOwnerDBProcessor.visit(ContainerOwnerDBProcessor.java:38)
at com.atmire.statistics.util.update.atomic.record.UsageRecord.accept(UsageRecord.java:23)
at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.applyProcessors(AtomicStatisticsUpdater.java:301)
... 10 more
</code></pre><ul>
<li>I’ve sent a message to Atmire to ask for advice</li>
<li>Also, now I can actually see the CUA statlets and usage statistics
<ul>
<li>Unfortunately it seems they are using Font Awesome 4 in their CUA module and this means that some icons are broken because the names have changed, but also some of their code is using Unicode characters instead of classes in spans!</li>
<li>I’ve reverted my sweet SVG work from yesterday and adjusted the Font Awesome 5 SCSS to add a few more icons that they are using</li>
<pre><code>$ psql -c 'select * from pg_stat_activity' | grep 'dspaceWeb' | grep -c "idle in transaction"
67
</code></pre><ul>
<li>I don’t see anything in the PostgreSQL or Tomcat logs suggesting anything is wrong… I think the solution to clear these idle connections is probably to just restart Tomcat</li>
<li>Delphi is only used by IP addresses in Greece, so that’s obviously the GARDIAN people harvesting us…</li>
<li>I have no idea what OgScrper is, but it’s not a user!</li>
<li>Then there are 276,000 hits from <code>MEL-API</code> from Jordanian IPs in 2018, so that’s obviously CodeObia guys…</li>
<li>Other user agents:
<ul>
<li>GigablastOpenSource/1</li>
<li>Owlin Domain Resolver V1</li>
<li>API scraper</li>
<li>MetaURI</li>
</ul>
</li>
<li>I don’t know why, but my <code>check-spider-hits.sh</code> script doesn’t seem to be handling the user agents with spaces properly so I will delete those manually after</li>
<li>First delete the ones without spaces, creating a temp file in <code>/tmp/agents</code> containing the patterns:</li>
</ul>
<pre><code>$ for year in {2010..2019}; do ./check-spider-hits.sh -f /tmp/agents -s statistics-$year -p; done
$ for year in {2010..2019}; do curl -s "http://localhost:8081/solr/statistics-$year/update?softCommit=true" -H "Content-Type: text/xml" --data-binary '<delete><query>userAgent:"Delphi 2009"</query></delete>'; done
