2021-10-04 18:40:13 +02:00
<!DOCTYPE html>
< html lang = "en" >
< head >
< meta charset = "utf-8" >
< meta name = "viewport" content = "width=device-width, initial-scale=1, shrink-to-fit=no" >
< meta property = "og:title" content = "October, 2021" / >
< meta property = "og:description" content = "2021-10-01
Export all affiliations on CGSpace and run them against the latest RoR data dump:
localhost/dspace63= > \COPY (SELECT DISTINCT text_value as " cg.contributor.affiliation" , count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC) to /tmp/2021-10-01-affiliations.csv WITH CSV HEADER;
$ csvcut -c 1 /tmp/2021-10-01-affiliations.csv | sed 1d > /tmp/2021-10-01-affiliations.txt
$ ./ilri/ror-lookup.py -i /tmp/2021-10-01-affiliations.txt -r 2021-09-23-ror-data.json -o /tmp/2021-10-01-affili
ations-matching.csv
$ csvgrep -c matched -m true /tmp/2021-10-01-affiliations-matching.csv | sed 1d | wc -l
1879
$ wc -l /tmp/2021-10-01-affiliations.txt
7100 /tmp/2021-10-01-affiliations.txt
So we have 1879/7100 (26.46%) matching already
" />
< meta property = "og:type" content = "article" / >
< meta property = "og:url" content = "https://alanorth.github.io/cgspace-notes/2021-10/" / >
< meta property = "article:published_time" content = "2021-10-01T11:14:07+03:00" / >
2021-10-05 17:54:39 +02:00
< meta property = "article:modified_time" content = "2021-10-04T19:40:13+03:00" / >
2021-10-04 18:40:13 +02:00
< meta name = "twitter:card" content = "summary" / >
< meta name = "twitter:title" content = "October, 2021" / >
< meta name = "twitter:description" content = "2021-10-01
Export all affiliations on CGSpace and run them against the latest RoR data dump:
localhost/dspace63= > \COPY (SELECT DISTINCT text_value as " cg.contributor.affiliation" , count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC) to /tmp/2021-10-01-affiliations.csv WITH CSV HEADER;
$ csvcut -c 1 /tmp/2021-10-01-affiliations.csv | sed 1d > /tmp/2021-10-01-affiliations.txt
$ ./ilri/ror-lookup.py -i /tmp/2021-10-01-affiliations.txt -r 2021-09-23-ror-data.json -o /tmp/2021-10-01-affili
ations-matching.csv
$ csvgrep -c matched -m true /tmp/2021-10-01-affiliations-matching.csv | sed 1d | wc -l
1879
$ wc -l /tmp/2021-10-01-affiliations.txt
7100 /tmp/2021-10-01-affiliations.txt
So we have 1879/7100 (26.46%) matching already
"/>
< meta name = "generator" content = "Hugo 0.88.1" / >
< script type = "application/ld+json" >
{
"@context": "http://schema.org",
"@type": "BlogPosting",
"headline": "October, 2021",
"url": "https://alanorth.github.io/cgspace-notes/2021-10/",
2021-10-05 17:54:39 +02:00
"wordCount": "771",
2021-10-04 18:40:13 +02:00
"datePublished": "2021-10-01T11:14:07+03:00",
2021-10-05 17:54:39 +02:00
"dateModified": "2021-10-04T19:40:13+03:00",
2021-10-04 18:40:13 +02:00
"author": {
"@type": "Person",
"name": "Alan Orth"
},
"keywords": "Notes"
}
< / script >
< link rel = "canonical" href = "https://alanorth.github.io/cgspace-notes/2021-10/" >
< title > October, 2021 | CGSpace Notes< / title >
<!-- combined, minified CSS -->
< link href = "https://alanorth.github.io/cgspace-notes/css/style.beb8012edc08ba10be012f079d618dc243812267efe62e11f22fe49618f976a4.css" rel = "stylesheet" integrity = "sha256-vrgBLtwIuhC+AS8HnWGNwkOBImfv5i4R8i/klhj5dqQ=" crossorigin = "anonymous" >
<!-- minified Font Awesome for SVG icons -->
< script defer src = "https://alanorth.github.io/cgspace-notes/js/fontawesome.min.f5072c55a0721857184db93a50561d7dc13975b4de2e19db7f81eb5f3fa57270.js" integrity = "sha256-9QcsVaByGFcYTbk6UFYdfcE5dbTeLhnbf4HrXz+lcnA=" crossorigin = "anonymous" > < / script >
<!-- RSS 2.0 feed -->
< / head >
< body >
< div class = "blog-masthead" >
< div class = "container" >
< nav class = "nav blog-nav" >
< a class = "nav-link " href = "https://alanorth.github.io/cgspace-notes/" > Home< / a >
< / nav >
< / div >
< / div >
< header class = "blog-header" >
< div class = "container" >
< h1 class = "blog-title" dir = "auto" > < a href = "https://alanorth.github.io/cgspace-notes/" rel = "home" > CGSpace Notes< / a > < / h1 >
< p class = "lead blog-description" dir = "auto" > Documenting day-to-day work on the < a href = "https://cgspace.cgiar.org" > CGSpace< / a > repository.< / p >
< / div >
< / header >
< div class = "container" >
< div class = "row" >
< div class = "col-sm-8 blog-main" >
< article class = "blog-post" >
< header >
< h2 class = "blog-post-title" dir = "auto" > < a href = "https://alanorth.github.io/cgspace-notes/2021-10/" > October, 2021< / a > < / h2 >
< p class = "blog-post-meta" >
< time datetime = "2021-10-01T11:14:07+03:00" > Fri Oct 01, 2021< / time >
in
< span class = "fas fa-folder" aria-hidden = "true" > < / span > < a href = "/cgspace-notes/categories/notes/" rel = "category tag" > Notes< / a >
< / p >
< / header >
< h2 id = "2021-10-01" > 2021-10-01< / h2 >
< ul >
< li > Export all affiliations on CGSpace and run them against the latest RoR data dump:< / li >
< / ul >
< pre tabindex = "0" > < code class = "language-console" data-lang = "console" > localhost/dspace63= > \COPY (SELECT DISTINCT text_value as " cg.contributor.affiliation" , count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC) to /tmp/2021-10-01-affiliations.csv WITH CSV HEADER;
$ csvcut -c 1 /tmp/2021-10-01-affiliations.csv | sed 1d > /tmp/2021-10-01-affiliations.txt
$ ./ilri/ror-lookup.py -i /tmp/2021-10-01-affiliations.txt -r 2021-09-23-ror-data.json -o /tmp/2021-10-01-affili
ations-matching.csv
$ csvgrep -c matched -m true /tmp/2021-10-01-affiliations-matching.csv | sed 1d | wc -l
1879
$ wc -l /tmp/2021-10-01-affiliations.txt
7100 /tmp/2021-10-01-affiliations.txt
< / code > < / pre > < ul >
< li > So we have 1879/7100 (26.46%) matching already< / li >
< / ul >
< h2 id = "2021-10-03" > 2021-10-03< / h2 >
< ul >
< li > Dominique from IWMI asked me for information about how CGSpace partners are using CGSpace APIs to feed their websites< / li >
< li > Start a fresh indexing on AReS< / li >
< li > Udana sent me his file of 292 non-IWMI publications for the Virtual library on water management
< ul >
< li > He added licenses< / li >
< li > I want to clean up the < code > dcterms.extent< / code > field though because it has volume, issue, and pages there< / li >
< li > I cloned the column several times and extracted values based on their positions, for example:
< ul >
< li > Volume: < code > value.partition(" :" )[0]< / code > < / li >
< li > Issue: < code > value.partition(" (" )[2].partition(" )" )[0]< / code > < / li >
< li > Page: < code > " p. " + value.replace(" ." , " " )< / code > < / li >
< / ul >
< / li >
< / ul >
< / li >
< / ul >
< h2 id = "2021-10-04" > 2021-10-04< / h2 >
< ul >
< li > Start looking at the last month of Solr statistics on CGSpace
< ul >
< li > I see a number of IPs with “ normal” user agents who clearly behave like bots
< ul >
< li > 198.15.130.18: 21,000 requests to /discover with a normal-looking user agent, from ASN 11282 (SERVERYOU, US)< / li >
< li > 93.158.90.107: 8,500 requests to handle and browse links with a Firefox 84.0 user agent, from ASN 12552 (IPO-EU, SE)< / li >
< li > 193.235.141.162: 4,800 requests to handle, browse, and discovery links with a Firefox 84.0 user agent, from ASN 51747 (INTERNETBOLAGET, SE)< / li >
< li > 3.225.28.105: 2,900 requests to REST API for the CIAT Story Maps collection with a normal user agent, from ASN 14618 (AMAZON-AES, US)< / li >
< li > 34.228.236.6: 2,800 requests to discovery for the CGIAR System community with user agent < code > Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)< / code > , from ASN 14618 (AMAZON-AES, US)< / li >
< li > 18.212.137.2: 2,800 requests to discovery for the CGIAR System community with user agent < code > Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)< / code > , from ASN 14618 (AMAZON-AES, US)< / li >
< li > 3.81.123.72: 2,800 requests to discovery and handles for the CGIAR System community with user agent < code > Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)< / code > , from ASN 14618 (AMAZON-AES, US)< / li >
< li > 3.227.16.188: 2,800 requests to discovery and handles for the CGIAR System community with user agent < code > Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)< / code > , from ASN 14618 (AMAZON-AES, US)< / li >
< / ul >
< / li >
< li > Looking closer into the requests with this Mozilla/4.0 user agent, I see 500+ IPs using it:< / li >
< / ul >
< / li >
< / ul >
< pre tabindex = "0" > < code class = "language-console" data-lang = "console" > # zcat --force /var/log/nginx/*.log* | grep 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)' | awk '{print $1}' | sort | uniq > /tmp/mozilla-4.0-ips.txt
# wc -l /tmp/mozilla-4.0-ips.txt
543 /tmp/mozilla-4.0-ips.txt
< / code > < / pre > < ul >
< li > Then I resolved the IPs and extracted the ones belonging to Amazon:< / li >
< / ul >
< pre tabindex = "0" > < code class = "language-console" data-lang = "console" > $ ./ilri/resolve-addresses-geoip2.py -i /tmp/mozilla-4.0-ips.txt -k " $ABUSEIPDB_API_KEY" -o /tmp/mozilla-4.0-ips.csv
$ csvgrep -c asn -m 14618 /tmp/mozilla-4.0-ips.csv | csvcut -c ip | sed 1d | tee /tmp/amazon-ips.txt | wc -l
< / code > < / pre > < ul >
< li > I am thinking I will purge them all, as I have several indicators that they are bots: mysterious user agent, IP owned by Amazon< / li >
< li > Even more interesting, these requests are weighted VERY heavily on the CGIAR System community:< / li >
< / ul >
< pre tabindex = "0" > < code class = "language-console" data-lang = "console" > 1592 GET /handle/10947/2526
1592 GET /handle/10947/2527
1592 GET /handle/10947/34
1593 GET /handle/10947/6
1594 GET /handle/10947/1
1598 GET /handle/10947/2515
1598 GET /handle/10947/2516
1599 GET /handle/10568/101335
1599 GET /handle/10568/91688
1599 GET /handle/10947/2517
1599 GET /handle/10947/2518
1599 GET /handle/10947/2519
1599 GET /handle/10947/2708
1599 GET /handle/10947/2871
1600 GET /handle/10568/89342
1600 GET /handle/10947/4467
1607 GET /handle/10568/103816
290382 GET /handle/10568/83389
< / code > < / pre > < ul >
2021-10-05 17:54:39 +02:00
< li > Before I purge all those I will ask someone Samuel Stacey from the System Office to hopefully get an insight… < / li >
2021-10-04 18:40:13 +02:00
< li > Meeting with Michael Victor, Peter, Jane, and Abenet about the future of repositories in the One CGIAR< / li >
< li > Meeting with Michelle from Altmetric about their new CSV upload system
< ul >
< li > I sent her some examples of Handles that have DOIs, but no linked score (yet) to see if an association will be created when she uploads them< / li >
< / ul >
< / li >
< / ul >
< pre tabindex = "0" > < code class = "language-csv" data-lang = "csv" > doi,handle
10.1016/j.agsy.2021.103263,10568/115288
10.3389/fgene.2021.723360,10568/115287
10.3389/fpls.2021.720670,10568/115285
< / code > < / pre > < ul >
< li > Extract the AGROVOC subjects from IWMI’ s 292 publications to validate them against AGROVOC:< / li >
< / ul >
< pre tabindex = "0" > < code class = "language-console" data-lang = "console" > $ csvcut -c 'dcterms.subject[en_US]' ~/Downloads/2021-10-03-non-IWMI-publications.csv | sed -e 1d -e 's/||/\n/g' -e 's/" //g' | sort -u > /tmp/agrovoc.txt
$ ./ilri/agrovoc-lookup.py -i /tmp/agrovoc-sorted.txt -o /tmp/agrovoc-matches.csv
$ csvgrep -c 'number of matches' -m '0' /tmp/agrovoc-matches.csv | csvcut -c 1 > /tmp/invalid-agrovoc.csv
2021-10-05 17:54:39 +02:00
< / code > < / pre > < h2 id = "2021-10-05" > 2021-10-05< / h2 >
< ul >
< li > Sam put me in touch with Dodi from the System Office web team and he confirmed that the Amazon requests are not theirs
< ul >
< li > I added < code > Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)< / code > to the list of bad bots in nginx< / li >
< li > I purged all the Amazon IPs using this user agent, as well as the few other IPs I identified yesterday< / li >
< / ul >
< / li >
< / ul >
< pre tabindex = "0" > < code class = "language-console" data-lang = "console" > $ ./ilri/check-spider-ip-hits.sh -f /tmp/robot-ips.txt -p
...
Total number of bot hits purged: 465119
2021-10-04 18:40:13 +02:00
< / code > < / pre > <!-- raw HTML omitted -->
< / article >
< / div > <!-- /.blog - main -->
< aside class = "col-sm-3 ml-auto blog-sidebar" >
< section class = "sidebar-module" >
< h4 > Recent Posts< / h4 >
< ol class = "list-unstyled" >
< li > < a href = "/cgspace-notes/2021-10/" > October, 2021< / a > < / li >
< li > < a href = "/cgspace-notes/2021-09/" > September, 2021< / a > < / li >
< li > < a href = "/cgspace-notes/2021-08/" > August, 2021< / a > < / li >
< li > < a href = "/cgspace-notes/2021-07/" > July, 2021< / a > < / li >
< li > < a href = "/cgspace-notes/2021-06/" > June, 2021< / a > < / li >
< / ol >
< / section >
< section class = "sidebar-module" >
< h4 > Links< / h4 >
< ol class = "list-unstyled" >
< li > < a href = "https://cgspace.cgiar.org" > CGSpace< / a > < / li >
< li > < a href = "https://dspacetest.cgiar.org" > DSpace Test< / a > < / li >
< li > < a href = "https://github.com/ilri/DSpace" > CGSpace @ GitHub< / a > < / li >
< / ol >
< / section >
< / aside >
< / div > <!-- /.row -->
< / div > <!-- /.container -->
< footer class = "blog-footer" >
< p dir = "auto" >
Blog template created by < a href = "https://twitter.com/mdo" > @mdo< / a > , ported to Hugo by < a href = 'https://twitter.com/mralanorth' > @mralanorth< / a > .
< / p >
< p >
< a href = "#" > Back to top< / a >
< / p >
< / footer >
< / body >
< / html >
