localhost/dspace63= > \COPY (SELECT DISTINCT text_value as "cg.contributor.affiliation", count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC) to /tmp/2021-10-01-affiliations.csv WITH CSV HEADER;
localhost/dspace63= > \COPY (SELECT DISTINCT text_value as "cg.contributor.affiliation", count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC) to /tmp/2021-10-01-affiliations.csv WITH CSV HEADER;
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span>localhost/dspace63= > \COPY (SELECT DISTINCT text_value as "cg.contributor.affiliation", count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC) to /tmp/2021-10-01-affiliations.csv WITH CSV HEADER;
</span></span><spanstyle="display:flex;"><span>$ csvcut -c <spanstyle="color:#ae81ff">1</span> /tmp/2021-10-01-affiliations.csv | sed 1d > /tmp/2021-10-01-affiliations.txt
<li>Start looking at the last month of Solr statistics on CGSpace
<ul>
<li>I see a number of IPs with “normal” user agents who clearly behave like bots
<ul>
<li>198.15.130.18: 21,000 requests to /discover with a normal-looking user agent, from ASN 11282 (SERVERYOU, US)</li>
<li>93.158.90.107: 8,500 requests to handle and browse links with a Firefox 84.0 user agent, from ASN 12552 (IPO-EU, SE)</li>
<li>193.235.141.162: 4,800 requests to handle, browse, and discovery links with a Firefox 84.0 user agent, from ASN 51747 (INTERNETBOLAGET, SE)</li>
<li>3.225.28.105: 2,900 requests to REST API for the CIAT Story Maps collection with a normal user agent, from ASN 14618 (AMAZON-AES, US)</li>
<li>34.228.236.6: 2,800 requests to discovery for the CGIAR System community with user agent <code>Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)</code>, from ASN 14618 (AMAZON-AES, US)</li>
<li>18.212.137.2: 2,800 requests to discovery for the CGIAR System community with user agent <code>Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)</code>, from ASN 14618 (AMAZON-AES, US)</li>
<li>3.81.123.72: 2,800 requests to discovery and handles for the CGIAR System community with user agent <code>Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)</code>, from ASN 14618 (AMAZON-AES, US)</li>
<li>3.227.16.188: 2,800 requests to discovery and handles for the CGIAR System community with user agent <code>Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)</code>, from ASN 14618 (AMAZON-AES, US)</li>
</ul>
</li>
<li>Looking closer into the requests with this Mozilla/4.0 user agent, I see 500+ IPs using it:</li>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span> 1592 GET /handle/10947/2526
</span></span><spanstyle="display:flex;"><span> 1592 GET /handle/10947/2527
</span></span><spanstyle="display:flex;"><span> 1592 GET /handle/10947/34
</span></span><spanstyle="display:flex;"><span> 1593 GET /handle/10947/6
</span></span><spanstyle="display:flex;"><span> 1594 GET /handle/10947/1
</span></span><spanstyle="display:flex;"><span> 1598 GET /handle/10947/2515
</span></span><spanstyle="display:flex;"><span> 1598 GET /handle/10947/2516
</span></span><spanstyle="display:flex;"><span> 1599 GET /handle/10568/101335
</span></span><spanstyle="display:flex;"><span> 1599 GET /handle/10568/91688
</span></span><spanstyle="display:flex;"><span> 1599 GET /handle/10947/2517
</span></span><spanstyle="display:flex;"><span> 1599 GET /handle/10947/2518
</span></span><spanstyle="display:flex;"><span> 1599 GET /handle/10947/2519
</span></span><spanstyle="display:flex;"><span> 1599 GET /handle/10947/2708
</span></span><spanstyle="display:flex;"><span> 1599 GET /handle/10947/2871
</span></span><spanstyle="display:flex;"><span> 1600 GET /handle/10568/89342
</span></span><spanstyle="display:flex;"><span> 1600 GET /handle/10947/4467
</span></span><spanstyle="display:flex;"><span> 1607 GET /handle/10568/103816
</span></span><spanstyle="display:flex;"><span> 290382 GET /handle/10568/83389
</span></span></span><spanstyle="display:flex;"><span><spanstyle="color:#960050;background-color:#1e0010"></span>Total number of bot hits purged: 465119
<li>Thinking about how we could check for duplicates before importing
<ul>
<li>I found out that <ahref="https://www.freecodecamp.org/news/fuzzy-string-matching-with-postgresql/">PostgreSQL has a built-in similarity function</a>:</li>
</span></span><spanstyle="display:flex;"><span>localhost/dspace63= > SELECT metadata_value_id, text_value, dspace_object_id FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=64 AND SIMILARITY(text_value,'Molecular marker based genetic diversity assessment of Striga resistant maize inbred lines') > 0.5;
<li>I was able to find an exact duplicate for an IITA item by searching for its title (I already knew that these existed)</li>
<li>I started working on a basic Python script to do this and managed to find an actual duplicate in the recent IWMI items
<ul>
<li>I think I will check for similar titles, and if I find them I will print out the handles for verification</li>
<li>I could also proceed to check other metadata like type because those shouldn’t vary too much</li>
</ul>
</li>
<li>I ran my new <code>check-duplicates.py</code> script on the 292 non-IWMI publications from Udana and found twelve potential duplicates
<ul>
<li>Upon checking them manually, I found that 7/12 were indeed already present on CGSpace!</li>
<li>This is with the similarity threshold at 0.5. I wonder if tweaking that higher will make the script run faster and eliminate some false positives</li>
<li>I re-ran it with higher thresholds this eliminated all false positives, but it still took 24 minutes to run for 292 items!
<ul>
<li>0.6: ./ilri/check-duplicates.py -i ~/Downloads/2021-10-03-non-IWMI-publications.cs 0.09s user 0.03s system 0% cpu 24:40.42 total</li>
<li>0.7: ./ilri/check-duplicates.py -i ~/Downloads/2021-10-03-non-IWMI-publications.cs 0.12s user 0.03s system 0% cpu 24:29.15 total</li>
<li>0.8: ./ilri/check-duplicates.py -i ~/Downloads/2021-10-03-non-IWMI-publications.cs 0.09s user 0.03s system 0% cpu 25:44.13 total</li>
</ul>
</li>
</ul>
</li>
<li>Some minor updates to csv-metadata-quality
<ul>
<li>Fix two issues with regular expressions in the duplicate items and experimental language checks</li>
<li>Add a check for items that have a DOI listed in their citation, but are missing a standalone DOI field</li>
</ul>
</li>
<li>Then I ran this new version of csv-metadata-quality on an export of IWMI’s community, minus some fields I don’t want to check:</li>
<li>I noticed each CSV only had 10 or 20 corrections, mostly that none of the duplicate metadata values were removed in the CSVs…
<ul>
<li>I cut a subset of the fields from the main CSV and tried again, but DSpace said “no changes detected”</li>
<li>The duplicates are definitely removed from the CSV, but DSpace doesn’t detect them</li>
<li>I realized this is an issue I’ve had before, but forgot because I usually use csv-metadata-quality for new items, not ones already inside DSpace!</li>
<li>I found a comment on thread on the dspace-tech mailing list from helix84 in 2015 (“No changes were detected” when importing metadata via XMLUI") where he says:</li>
</ul>
</li>
</ul>
<blockquote>
<p>It’s very likely that multiple values in a single field are being compared as an unordered set rather than an ordered list.
Try doing it in two imports. In first import, remove all authors. In second import, add them in the new order.</p>
</blockquote>
<ul>
<li>Shit, so that’s worth looking into…</li>
<li>I decided to upload the cleaned IWMI community by moving the cleaned metadata field from <code>dcterms.subject[en_US]</code> to <code>dcterms.subject[en_Fu]</code> temporarily, uploading them, then moving them back, and uploading again
<ul>
<li>I started by copying just a handful of fields from the iwmi.csv community export:</li>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span>cgspace=# SELECT DISTINCT text_lang, count(text_lang) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) GROUP BY text_lang ORDER BY count DESC;
</span></span><spanstyle="display:flex;"><span>cgspace=# UPDATE metadatavalue SET text_lang='en_US' WHERE dspace_object_id IN (SELECT uuid FROM item) AND text_lang IN ('en_Fu', 'en', '');
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span>$ psql -h localhost -p <spanstyle="color:#ae81ff">5433</span> -U postgres dspace7 -c <spanstyle="color:#e6db74">"DELETE FROM schema_version WHERE description LIKE '%Atmire%' OR description LIKE '%CUA%' OR description LIKE '%cua%';"</span>
</span></span><spanstyle="display:flex;"><span>$ psql -h localhost -p <spanstyle="color:#ae81ff">5433</span> -U postgres dspace7 -c <spanstyle="color:#e6db74">"DELETE FROM schema_version WHERE version IN ('5.0.2017.09.25', '6.0.2017.01.30', '6.0.2017.09.25');"</span>
<li>I tested the <code>metadata-export</code> on DSpace 7.1-SNAPSHOT and it still has the duplicate items issue introduced by DS-4211
<ul>
<li>I filed a GitHub issue and notified nwoodward: <ahref="https://github.com/DSpace/DSpace/issues/7988">https://github.com/DSpace/DSpace/issues/7988</a></li>
<li>The only thing I did on 2021-10-07 was import a few thousand metadata corrections…</li>
<li>I restarted PostgreSQL (instead of restarting Tomcat), so let’s see if that helps</li>
<li>I filed <ahref="https://github.com/DSpace/DSpace/issues/7989">a bug for the DSpace 6/7 duplicate values metadata import issue</a></li>
<li>I tested the two patches for removing abandoned submissions from the workflow but unfortunately it seems that they are for the configurable aka XML workflow, and we are using the basic workflow</li>
<li>I discussed PostgreSQL issues with some people on the DSpace Slack
<ul>
<li>Looking at postgresqltuner.pl and <ahref="https://pgtune.leopard.in.ua">https://pgtune.leopard.in.ua</a> I realized that there were some settings that I hadn’t changed in a few years that I probably need to re-evaluate</li>
<li>For example, <code>random_page_cost</code> is recommended to be 1.1 in the PostgreSQL 10 docs (default is 4.0, but we use 1 since 2017 when it came up in Hacker News)</li>
<li>Also, <code>effective_io_concurrency</code> is recommended to be “hundreds” if you are using an SSD (default is 1)</li>
</ul>
</li>
<li>I also enabled the <code>pg_stat_statements</code> extension to try to understand what queries are being run the most often, and how long they take</li>
</ul>
<h2id="2021-10-12">2021-10-12</h2>
<ul>
<li>I looked again at the duplicate items query I was doing with trigrams recently and found a few new things
<ul>
<li>Looking at the <code>EXPLAIN ANALYZE</code> plan for the query I noticed it wasn’t using any indexes</li>
<li>I <ahref="https://dba.stackexchange.com/questions/103821/best-index-for-similarity-function/103823">read on StackExchange</a> that, if we want to make use of indexes, we need to use the similarity operator (<code>%</code>), not the function <code>similarity()</code> because “index support is bound to operators in Postgres, not to functions”</li>
<li>A note about the query plan output is that we need to read it from the bottom up!</li>
<li>So with the similary operator we need to set the threshold like this now:</li>
<li>Next I experimented with using GIN or GiST indexes on <code>metadatavalue</code>, but they were slower than the existing DSpace indexes
<ul>
<li>I tested a few variations of the query I had been using and found it’s <em>much</em> faster if I use the similarity operator and keep the condition that object IDs are in the item table…</li>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span>localhost/dspace= > SELECT text_value, dspace_object_id FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=64 AND text_value % 'Traditional knowledge affects soil management ability of smallholder farmers in marginal areas';
</span></span><spanstyle="display:flex;"><span> Traditional knowledge affects soil management ability of smallholder farmers in marginal areas │ 7af059af-9cd7-431b-8a79-7514896ca7dc
<li>Now this script runs in four minutes (versus twenty-four!) and it still finds the same seven duplicates! Amazing!</li>
<li>I still don’t understand the differences in the query plan well enough, but I see it is using the DSpace default indexes and the results are accurate</li>
<li>So to summarize, the best to the worst query, all returning the same result:</li>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span>localhost/dspace= > SET pg_trgm.similarity_threshold = 0.6;
</span></span><spanstyle="display:flex;"><span>localhost/dspace= > SELECT text_value, dspace_object_id FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=64 AND text_value % 'Traditional knowledge affects soil management ability of smallholder farmers in marginal areas';
</span></span><spanstyle="display:flex;"><span> Traditional knowledge affects soil management ability of smallholder farmers in marginal areas │ 7af059af-9cd7-431b-8a79-7514896ca7dc
</span></span><spanstyle="display:flex;"><span>localhost/dspace= > SET pg_trgm.similarity_threshold = 0.6;
</span></span><spanstyle="display:flex;"><span>localhost/dspace= > SELECT text_value, dspace_object_id FROM metadatavalue WHERE metadata_field_id=64 AND text_value % 'Traditional knowledge affects soil management ability of smallholder farmers in marginal areas';
</span></span><spanstyle="display:flex;"><span> Traditional knowledge affects soil management ability of smallholder farmers in marginal areas │ 7af059af-9cd7-431b-8a79-7514896ca7dc
</span></span><spanstyle="display:flex;"><span>localhost/dspace= > SELECT text_value, dspace_object_id FROM metadatavalue WHERE metadata_field_id=64 AND SIMILARITY(text_value,'Traditional knowledge affects soil management ability of smallholder farmers in marginal areas') > 0.6;
</span></span><spanstyle="display:flex;"><span> Traditional knowledge affects soil management ability of smallholder farmers in marginal areas │ 7af059af-9cd7-431b-8a79-7514896ca7dc
</span></span><spanstyle="display:flex;"><span>localhost/dspace= > SELECT text_value, dspace_object_id FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=64 AND SIMILARITY(text_value,'Traditional knowledge affects soil management ability of smallholder farmers in marginal areas') > 0.6;
</span></span><spanstyle="display:flex;"><span> Traditional knowledge affects soil management ability of smallholder farmers in marginal areas │ 7af059af-9cd7-431b-8a79-7514896ca7dc
<li>I looked into the <ahref="https://github.com/DSpace/DSpace/issues/7946">REST API issue where fields without qualifiers throw an HTTP 500</a>
<ul>
<li>The fix is to check if the qualifier is not null AND not empty in dspace-api</li>
<li>I submitted a fix: <ahref="https://github.com/DSpace/DSpace/pull/7993">https://github.com/DSpace/DSpace/pull/7993</a></li>
</ul>
</li>
</ul>
<h2id="2021-10-14">2021-10-14</h2>
<ul>
<li>Someone in the DSpace community already posted a fix for the DSpace 6/7 duplicate items export bug!
<ul>
<li>I tested it and it works so I left feedback: <ahref="https://github.com/DSpace/DSpace/pull/7995">https://github.com/DSpace/DSpace/pull/7995</a></li>
</ul>
</li>
<li>Altmetric support got back to us about the missing DOI–Handle link and said it was due to the TLS certificate chain on CGSpace
<ul>
<li>I checked and everything is actually working fine, so it could be their backend servers are old and don’t support the new Let’s Encrypt trust path</li>
<li>I asked them to put me in touch with their backend developers directly</li>
</ul>
</li>
</ul>
<h2id="2021-10-17">2021-10-17</h2>
<ul>
<li>Revert the ssl-cert change on the Ansible infrastructure scripts so that nginx uses a manually generated “snakeoil” TLS certificate
<ul>
<li>The ssl-cert one is easier because it’s automatic, but they include the hostname in the bogus cert so it’s an unecessary leak of information</li>
<li>I started doing some tests to upgrade Elasticsearch from 7.6.2 to 7.7, 7.8, 7.9, and eventually 7.10 on OpenRXV
<ul>
<li>I tested harvesting, reporting, filtering, and various admin actions with each version and they all worked fine, with no errors in any logs as far as I can see</li>
<li>This fixes bunches of issues, updates Java from 13 to 15, and the base image from CentOS 7 to 8, so it’s a decent amount of technical debt!</li>
<li>I even tried Elasticsearch 7.13.2, which has Java 16, and it works fine…</li>
<li>I submitted a pull request: <ahref="https://github.com/ilri/OpenRXV/pull/126">https://github.com/ilri/OpenRXV/pull/126</a></li>
</ul>
</li>
</ul>
<h2id="2021-10-20">2021-10-20</h2>
<ul>
<li>Meeting with Big Data and CGIAR repository players about the feasibility of moving to a single repository
<ul>
<li>We discussed several options, for example moving all DSpaces to CGSpace along with their permanent identifiers</li>
<li>The issue would be for centers like IFPRI who don’t use DSpace and have integrations with their website etc with their current repository</li>
</span></span><spanstyle="display:flex;"><span># Ghetto way to extract the IPs using jq, but I can<spanstyle="color:#960050;background-color:#1e0010">'</span>t figure out how only print them and not the facet counts, so I just use sed
<li>This IP has made 36,000 requests to CGSpace…</li>
<li>The IP is owned by <ahref="internetvikings.com">Internet Vikings</a> in Sweden</li>
<li>I purged their statistics and set up a temporary HTTP 403 telling them to use a real user agent</li>
<li>I see another one in Sweden a few days ago (192.36.109.131), also using the same exact user agent as above, but belonging to <ahref="http://webb.resilans.se/">Resilans AB</a>
<ul>
<li>I purged another 74,619 hits from this bot</li>
</ul>
</li>
<li>I added these two IPs to the nginx IP bot identifier</li>
<li>Jesus I found a few Russian IPs attempting SQL injection and path traversal, ie:</li>
</span></span></span><spanstyle="display:flex;"><span><spanstyle="color:#960050;background-color:#1e0010"></span>Total number of bot hits purged: 17786
<li>Update Docker containers for AReS on linode20 and run a fresh harvest</li>
<li>Found some strange IP (94.71.3.44) making 51,000 requests today with the user agent “Microsoft Internet Explorer”
<ul>
<li>It is in Greece, and it seems to be requesting each item’s XMLUI full metadata view, so I suspect it’s Gardian actually</li>
<li>I found it making another 25,000 requests yesterday…</li>
<li>I purged them from Solr</li>
</ul>
</li>
<li>Found 20,000 hits from Qualys (according to AbuseIPDB.com) using normal user agents… ugh, must be some ILRI ICT scan</li>
<li>Found more request from a Swedish IP (93.158.90.34) using that weird Firefox user agent that I noticed a few weeks ago:</li>
</ul>
<pretabindex="0"><code>Mozilla/5.0 (Macintosh; Intel Mac OS X 11.1; rv:84.0) Gecko/20100101 Firefox/84.0
</code></pre><ul>
<li>That’s from ASN 12552 (IPO-EU, SE), which is operated by Internet Vikings, though AbuseIPDB.com says it’s <ahref="availo.se">Availo Networks AB</a></li>
<li>There’s another IP (3.225.28.105) that made a few thousand requests to the REST API from Amazon, though it’s using a normal user agent</li>
