mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2024-12-27 23:44:30 +01:00
859 lines
43 KiB
HTML
859 lines
43 KiB
HTML
<!DOCTYPE html>
|
||
<html lang="en" >
|
||
|
||
<head>
|
||
<meta charset="utf-8">
|
||
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
|
||
|
||
<meta property="og:title" content="July, 2020" />
|
||
<meta property="og:description" content="2020-07-01
|
||
|
||
A few users noticed that CGSpace wasn’t loading items today, item pages seem blank
|
||
|
||
I looked at the PostgreSQL locks but they don’t seem unusual
|
||
I guess this is the same “blank item page” issue that we had a few times in 2019 that we never solved
|
||
I restarted Tomcat and PostgreSQL and the issue was gone
|
||
|
||
|
||
Since I was restarting Tomcat anyways I decided to redeploy the latest changes from the 5_x-prod branch and I added a note about COVID-19 items to the CGSpace frontpage at Peter’s request
|
||
" />
|
||
<meta property="og:type" content="article" />
|
||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2020-07/" />
|
||
<meta property="article:published_time" content="2020-07-01T10:53:54+03:00" />
|
||
<meta property="article:modified_time" content="2020-07-21T13:44:52+03:00" />
|
||
|
||
<meta name="twitter:card" content="summary"/>
|
||
<meta name="twitter:title" content="July, 2020"/>
|
||
<meta name="twitter:description" content="2020-07-01
|
||
|
||
A few users noticed that CGSpace wasn’t loading items today, item pages seem blank
|
||
|
||
I looked at the PostgreSQL locks but they don’t seem unusual
|
||
I guess this is the same “blank item page” issue that we had a few times in 2019 that we never solved
|
||
I restarted Tomcat and PostgreSQL and the issue was gone
|
||
|
||
|
||
Since I was restarting Tomcat anyways I decided to redeploy the latest changes from the 5_x-prod branch and I added a note about COVID-19 items to the CGSpace frontpage at Peter’s request
|
||
"/>
|
||
<meta name="generator" content="Hugo 0.73.0" />
|
||
|
||
|
||
|
||
<script type="application/ld+json">
|
||
{
|
||
"@context": "http://schema.org",
|
||
"@type": "BlogPosting",
|
||
"headline": "July, 2020",
|
||
"url": "https://alanorth.github.io/cgspace-notes/2020-07/",
|
||
"wordCount": "4195",
|
||
"datePublished": "2020-07-01T10:53:54+03:00",
|
||
"dateModified": "2020-07-21T13:44:52+03:00",
|
||
"author": {
|
||
"@type": "Person",
|
||
"name": "Alan Orth"
|
||
},
|
||
"keywords": "Notes"
|
||
}
|
||
</script>
|
||
|
||
|
||
|
||
<link rel="canonical" href="https://alanorth.github.io/cgspace-notes/2020-07/">
|
||
|
||
<title>July, 2020 | CGSpace Notes</title>
|
||
|
||
|
||
<!-- combined, minified CSS -->
|
||
|
||
<link href="https://alanorth.github.io/cgspace-notes/css/style.6da5c906cc7a8fbb93f31cd2316c5dbe3f19ac4aa6bfb066f1243045b8f6061e.css" rel="stylesheet" integrity="sha256-baXJBsx6j7uT8xzSMWxdvj8ZrEqmv7Bm8SQwRbj2Bh4=" crossorigin="anonymous">
|
||
|
||
|
||
<!-- minified Font Awesome for SVG icons -->
|
||
|
||
<script defer src="https://alanorth.github.io/cgspace-notes/js/fontawesome.min.f3d2a1f5980bab30ddd0d8cadbd496475309fc48e2b1d052c5c09e6facffcb0f.js" integrity="sha256-89Kh9ZgLqzDd0NjK29SWR1MJ/EjisdBSxcCeb6z/yw8=" crossorigin="anonymous"></script>
|
||
|
||
<!-- RSS 2.0 feed -->
|
||
|
||
|
||
|
||
|
||
|
||
|
||
</head>
|
||
|
||
<body>
|
||
|
||
|
||
<div class="blog-masthead">
|
||
<div class="container">
|
||
<nav class="nav blog-nav">
|
||
<a class="nav-link " href="https://alanorth.github.io/cgspace-notes/">Home</a>
|
||
</nav>
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
|
||
<header class="blog-header">
|
||
<div class="container">
|
||
<h1 class="blog-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/" rel="home">CGSpace Notes</a></h1>
|
||
<p class="lead blog-description" dir="auto">Documenting day-to-day work on the <a href="https://cgspace.cgiar.org">CGSpace</a> repository.</p>
|
||
</div>
|
||
</header>
|
||
|
||
|
||
|
||
|
||
<div class="container">
|
||
<div class="row">
|
||
<div class="col-sm-8 blog-main">
|
||
|
||
|
||
|
||
|
||
<article class="blog-post">
|
||
<header>
|
||
<h2 class="blog-post-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/2020-07/">July, 2020</a></h2>
|
||
<p class="blog-post-meta"><time datetime="2020-07-01T10:53:54+03:00">Wed Jul 01, 2020</time> by Alan Orth in
|
||
<span class="fas fa-folder" aria-hidden="true"></span> <a href="/cgspace-notes/categories/notes/" rel="category tag">Notes</a>
|
||
|
||
|
||
</p>
|
||
</header>
|
||
<h2 id="2020-07-01">2020-07-01</h2>
|
||
<ul>
|
||
<li>A few users noticed that CGSpace wasn’t loading items today, item pages seem blank
|
||
<ul>
|
||
<li>I looked at the PostgreSQL locks but they don’t seem unusual</li>
|
||
<li>I guess this is the same “blank item page” issue that we had a few times in 2019 that we never solved</li>
|
||
<li>I restarted Tomcat and PostgreSQL and the issue was gone</li>
|
||
</ul>
|
||
</li>
|
||
<li>Since I was restarting Tomcat anyways I decided to redeploy the latest changes from the <code>5_x-prod</code> branch and I added a note about COVID-19 items to the CGSpace frontpage at Peter’s request</li>
|
||
</ul>
|
||
<ul>
|
||
<li>Also, Linode is alerting that we had high outbound traffic rate early this morning around midnight AND high CPU load later in the morning</li>
|
||
<li>First looking at the traffic in the morning:</li>
|
||
</ul>
|
||
<pre><code># cat /var/log/nginx/*.log.1 /var/log/nginx/*.log | grep -E "01/Jul/2020:(00|01|02|03|04)" | goaccess --log-format=COMBINED -
|
||
...
|
||
9659 33.56% 1 0.08% 340.94 MiB 64.39.99.13
|
||
3317 11.53% 1 0.08% 871.71 MiB 199.47.87.140
|
||
2986 10.38% 1 0.08% 17.39 MiB 199.47.87.144
|
||
2286 7.94% 1 0.08% 13.04 MiB 199.47.87.142
|
||
</code></pre><ul>
|
||
<li>64.39.99.13 belongs to Qualys, but I see they are using a normal desktop user agent:</li>
|
||
</ul>
|
||
<pre><code>Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15
|
||
</code></pre><ul>
|
||
<li>I will purge hits from that IP from Solr</li>
|
||
<li>The 199.47.87.x IPs belong to Turnitin, and apparently they are NOT marked as bots and we have 40,000 hits from them in 2020 statistics alone:</li>
|
||
</ul>
|
||
<pre><code>$ curl -s "http://localhost:8081/solr/statistics/select" -d "q=userAgent:/Turnitin.*/&rows=0" | grep -oE 'numFound="[0-9]+"'
|
||
numFound="41694"
|
||
</code></pre><ul>
|
||
<li>They used to be “TurnitinBot”… hhmmmm, seems they use both: <a href="https://turnitin.com/robot/crawlerinfo.html">https://turnitin.com/robot/crawlerinfo.html</a></li>
|
||
<li>I will add Turnitin to the DSpace bot user agent list, but I see they are reqesting <code>robots.txt</code> and only requesting item pages, so that’s impressive! I don’t need to add them to the “bad bot” rate limit list in nginx</li>
|
||
<li>While looking at the logs I noticed eighty-one IPs in the range 185.152.250.x making little requests this user agent:</li>
|
||
</ul>
|
||
<pre><code>Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:76.0) Gecko/20100101 Firefox/76.0
|
||
</code></pre><ul>
|
||
<li>The IPs all belong to HostRoyale:</li>
|
||
</ul>
|
||
<pre><code># cat /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep '01/Jul/2020' | awk '{print $1}' | grep 185.152.250. | sort | uniq | wc -l
|
||
81
|
||
# cat /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep '01/Jul/2020' | awk '{print $1}' | grep 185.152.250. | sort | uniq | sort -h
|
||
185.152.250.1
|
||
185.152.250.101
|
||
185.152.250.103
|
||
185.152.250.105
|
||
185.152.250.107
|
||
185.152.250.111
|
||
185.152.250.115
|
||
185.152.250.119
|
||
185.152.250.121
|
||
185.152.250.123
|
||
185.152.250.125
|
||
185.152.250.129
|
||
185.152.250.13
|
||
185.152.250.131
|
||
185.152.250.133
|
||
185.152.250.135
|
||
185.152.250.137
|
||
185.152.250.141
|
||
185.152.250.145
|
||
185.152.250.149
|
||
185.152.250.153
|
||
185.152.250.155
|
||
185.152.250.157
|
||
185.152.250.159
|
||
185.152.250.161
|
||
185.152.250.163
|
||
185.152.250.165
|
||
185.152.250.167
|
||
185.152.250.17
|
||
185.152.250.171
|
||
185.152.250.183
|
||
185.152.250.189
|
||
185.152.250.191
|
||
185.152.250.197
|
||
185.152.250.201
|
||
185.152.250.205
|
||
185.152.250.209
|
||
185.152.250.21
|
||
185.152.250.213
|
||
185.152.250.217
|
||
185.152.250.219
|
||
185.152.250.221
|
||
185.152.250.223
|
||
185.152.250.225
|
||
185.152.250.227
|
||
185.152.250.229
|
||
185.152.250.231
|
||
185.152.250.233
|
||
185.152.250.235
|
||
185.152.250.239
|
||
185.152.250.243
|
||
185.152.250.247
|
||
185.152.250.249
|
||
185.152.250.25
|
||
185.152.250.251
|
||
185.152.250.253
|
||
185.152.250.255
|
||
185.152.250.27
|
||
185.152.250.29
|
||
185.152.250.3
|
||
185.152.250.31
|
||
185.152.250.39
|
||
185.152.250.41
|
||
185.152.250.47
|
||
185.152.250.5
|
||
185.152.250.59
|
||
185.152.250.63
|
||
185.152.250.65
|
||
185.152.250.67
|
||
185.152.250.7
|
||
185.152.250.71
|
||
185.152.250.73
|
||
185.152.250.77
|
||
185.152.250.81
|
||
185.152.250.85
|
||
185.152.250.89
|
||
185.152.250.9
|
||
185.152.250.93
|
||
185.152.250.95
|
||
185.152.250.97
|
||
185.152.250.99
|
||
</code></pre><ul>
|
||
<li>It’s only a few hundred requests each, but I am very suspicious so I will record it here and purge their IPs from Solr</li>
|
||
<li>Then I see 185.187.30.14 and 185.187.30.13 making requests also, with several different “normal” user agents
|
||
<ul>
|
||
<li>They are both apparently in France, belonging to Scalair FR hosting</li>
|
||
<li>I will purge their requests from Solr too</li>
|
||
</ul>
|
||
</li>
|
||
<li>Now I see some other new bots I hadn’t noticed before:
|
||
<ul>
|
||
<li><code>Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) LinkCheck by Siteimprove.com</code></li>
|
||
<li><code>Consilio (WebHare Platform 4.28.2-dev); LinkChecker)</code>, which appears to be a <a href="https://www.utwente.nl/en/websites/webhare/">university CMS</a></li>
|
||
<li>I will add <code>LinkCheck</code>, <code>Consilio</code>, and <code>WebHare</code> to the list of DSpace bot agents and purge them from Solr stats</li>
|
||
<li>COUNTER-Robots list already has <code>link.?check</code> but for some reason DSpace didn’t match that and I see hits for some of these…</li>
|
||
<li>Maybe I should add <code>[Ll]ink.?[Cc]heck.?</code> to a custom list for now?</li>
|
||
<li>For now I added <code>Turnitin</code> to the <a href="https://github.com/atmire/COUNTER-Robots/pull/34">new bots pull request on COUNTER-Robots</a></li>
|
||
</ul>
|
||
</li>
|
||
<li>I purged 20,000 hits from IPs and 45,000 hits from user agents</li>
|
||
<li>I will revert the default “example” agents file back to the upstream master branch of COUNTER-Robots, and then add all my custom ones that are pending in pull requests they haven’t merged yet:</li>
|
||
</ul>
|
||
<pre><code>$ diff --unchanged-line-format= --old-line-format= --new-line-format='%L' dspace/config/spiders/agents/example ~/src/git/COUNTER-Robots/COUNTER_Robots_list.txt
|
||
Citoid
|
||
ecointernet
|
||
GigablastOpenSource
|
||
Jersey\/\d
|
||
MarcEdit
|
||
OgScrper
|
||
okhttp
|
||
^Pattern\/\d
|
||
ReactorNetty\/\d
|
||
sqlmap
|
||
Typhoeus
|
||
7siters
|
||
</code></pre><ul>
|
||
<li>Just a note that I <em>still</em> can’t deploy the <code>6_x-dev-atmire-modules</code> branch as it fails at ant update:</li>
|
||
</ul>
|
||
<pre><code> [java] java.lang.RuntimeException: Failed to startup the DSpace Service Manager: failure starting up spring service manager: Error creating bean with name 'DefaultStorageUpdateConfig': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire method: public void com.atmire.statistics.util.StorageReportsUpdater.setStorageReportServices(java.util.List); nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cuaEPersonStorageReportService': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.atmire.dspace.cua.dao.storage.CUAEPersonStorageReportDAO com.atmire.dspace.cua.CUAStorageReportServiceImpl$CUAEPersonStorageReportServiceImpl.CUAEPersonStorageReportDAO; nested exception is org.springframework.beans.factory.NoUniqueBeanDefinitionException: No qualifying bean of type [com.atmire.dspace.cua.dao.storage.CUAEPersonStorageReportDAO] is defined: expected single matching bean but found 2: com.atmire.dspace.cua.dao.impl.CUAStorageReportDAOImpl$CUAEPersonStorageReportDAOImpl#0,com.atmire.dspace.cua.dao.impl.CUAStorageReportDAOImpl$CUAEPersonStorageReportDAOImpl#1
|
||
</code></pre><ul>
|
||
<li>I had told Atmire about this several weeks ago… but I reminded them again in the ticket
|
||
<ul>
|
||
<li>Atmire says they are able to build fine, so I tried again and noticed that I had been building with <code>-Denv=dspacetest.cgiar.org</code>, which is not necessary for DSpace 6 of course</li>
|
||
<li>Once I removed that it builds fine</li>
|
||
</ul>
|
||
</li>
|
||
<li>I quickly re-applied the Font Awesome 5 changes to use SVG+JS instead of web fonts (from 2020-04) and things are looking good!</li>
|
||
<li>Run all system updates on DSpace Test (linode26), deploy latest <code>6_x-dev-atmire-modules</code> branch, and reboot it</li>
|
||
</ul>
|
||
<h2 id="2020-07-02">2020-07-02</h2>
|
||
<ul>
|
||
<li>I need to export some Solr statistics data from CGSpace to test Salem’s modifications to the dspace-statistics-api
|
||
<ul>
|
||
<li>He modified it to query Solr on the fly instead of indexing it, which will be heavier and slower, but allows us to get more granular stats and countries/cities</li>
|
||
<li>Because have so many records I want to use solr-import-export-json to get several months at a time with a date range, but it seems there are first issues with curl (need to disable globbing with <code>-g</code> and URL encode the range)</li>
|
||
<li>For reference, the <a href="https://lucene.apache.org/solr/4_10_2/solr-core/org/apache/solr/schema/DateField.html">Solr 4.10.x DateField docs</a></li>
|
||
<li>This range works in Solr UI: <code>[2019-01-01T00:00:00Z TO 2019-06-30T23:59:59Z]</code></li>
|
||
<li>As well in curl:</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<pre><code>$ curl -g -s 'http://localhost:8081/solr/statistics-2019/select?q=*:*&fq=time:%5B2019-01-01T00%3A00%3A00Z%20TO%202019-06-30T23%3A59%3A59Z%5D&rows=0&wt=json&indent=true'
|
||
{
|
||
"responseHeader":{
|
||
"status":0,
|
||
"QTime":0,
|
||
"params":{
|
||
"q":"*:*",
|
||
"indent":"true",
|
||
"fq":"time:[2019-01-01T00:00:00Z TO 2019-06-30T23:59:59Z]",
|
||
"rows":"0",
|
||
"wt":"json"}},
|
||
"response":{"numFound":7784285,"start":0,"docs":[]
|
||
}}
|
||
</code></pre><ul>
|
||
<li>But not in solr-import-export-json… hmmm… seems we need to URL encode <em>only</em> the date range itself, but not the brackets:</li>
|
||
</ul>
|
||
<pre><code>$ ./run.sh -s http://localhost:8081/solr/statistics-2019 -a export -o /tmp/statistics-2019-1.json -f 'time:%5B2019-01-01T00%3A00%3A00Z%20TO%202019-06-30T23%3A59%3A59Z]' -k uid
|
||
$ zstd /tmp/statistics-2019-1.json
|
||
</code></pre><ul>
|
||
<li>Then import it on my local dev environment:</li>
|
||
</ul>
|
||
<pre><code>$ zstd -d statistics-2019-1.json.zst
|
||
$ ./run.sh -s http://localhost:8080/solr/statistics -a import -o ~/Downloads/statistics-2019-1.json -k uid
|
||
</code></pre><h2 id="2020-07-05">2020-07-05</h2>
|
||
<ul>
|
||
<li>Import twelve items into the <a href="https://hdl.handle.net/10568/97076">CRP Livestock multimedia</a> collection for Peter Ballantyne
|
||
<ul>
|
||
<li>I ran the data through csv-metadata-quality first to validate and fix some common mistakes</li>
|
||
<li>Interesting to check the data with <code>csvstat</code> to see if there are any duplicates</li>
|
||
</ul>
|
||
</li>
|
||
<li>Peter recently asked me to add Target audience (<code>cg.targetaudience</code>) to the CGSpace sidebar facets and AReS filters
|
||
<ul>
|
||
<li>I added it on my local DSpace test instance, but I’m waiting for him to tell me what he wants the header to be “Audiences” or “Target audience” etc…</li>
|
||
</ul>
|
||
</li>
|
||
<li>Peter also asked me to increase the size of links in the CGSpace “Welcome” text
|
||
<ul>
|
||
<li>I suggested using the CSS <code>font-size: larger</code> property to just bump it up one relative to what it already is</li>
|
||
<li>He said it looks good, but that actually now the links seem OK (I told him to refresh, as I had made them bold a few days ago) so we don’t need to adjust it actually</li>
|
||
</ul>
|
||
</li>
|
||
<li>Mohammed Salem modified my <a href="https://github.com/ilri/dspace-statistics-api">dspace-statistics-api</a> to query Solr directly so I started writing a script to benchmark it today
|
||
<ul>
|
||
<li>I will monitor the JVM memory and CPU usage in visualvm, just like I did in 2019-04</li>
|
||
<li>I noticed an issue with his limit parameter so I sent him some feedback on that in the meantime</li>
|
||
</ul>
|
||
</li>
|
||
<li>I noticed that we have 20,000 distinct values for <code>dc.subject</code>, but there are at least 500 that are lower or mixed case that we should simply uppercase without further thought:</li>
|
||
</ul>
|
||
<pre><code>dspace=# UPDATE metadatavalue SET text_value=UPPER(text_value) WHERE resource_type_id=2 AND metadata_field_id=57 AND text_value ~ '[[:lower:]]';
|
||
</code></pre><ul>
|
||
<li>DSpace Test needs a different query because it is running DSpace 6 with UUIDs for everything:</li>
|
||
</ul>
|
||
<pre><code>dspace63=# UPDATE metadatavalue SET text_value=UPPER(text_value) WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=57 AND text_value ~ '[[:lower:]]';
|
||
</code></pre><ul>
|
||
<li>Note the use of the POSIX character class :)</li>
|
||
<li>I suggest that we generate a list of the top 5,000 values that don’t match AGROVOC so that Sisay can correct them
|
||
<ul>
|
||
<li>Start by getting the top 6,500 subjects (assuming that the top ~1,500 are valid from our previous work):</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<pre><code>dspace=# \COPY (SELECT DISTINCT text_value, count(text_value) FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=57 GROUP BY text_value ORDER BY count DESC) TO /tmp/2020-07-05-subjects.csv WITH CSV;
|
||
COPY 19640
|
||
dspace=# \q
|
||
$ csvcut -c1 /tmp/2020-07-05-subjects-upper.csv | head -n 6500 > 2020-07-05-cgspace-subjects.txt
|
||
</code></pre><ul>
|
||
<li>Then start looking them up using <code>agrovoc-lookup.py</code>:</li>
|
||
</ul>
|
||
<pre><code>$ ./agrovoc-lookup.py -i 2020-07-05-cgspace-subjects.txt -om 2020-07-05-cgspace-subjects-matched.txt -or 2020-07-05-cgspace-subjects-rejected.txt -d
|
||
</code></pre><h2 id="2020-07-06">2020-07-06</h2>
|
||
<ul>
|
||
<li>I made some optimizations to the suite of Python utility scripts in our DSpace directory as well as the <a href="https://github.com/ilri/csv-metadata-quality">csv-metadata-quality</a> script
|
||
<ul>
|
||
<li>Mostly to make more efficient usage of the requests cache and to use parameterized requests instead of building the request URL by concatenating the URL with query parameters</li>
|
||
</ul>
|
||
</li>
|
||
<li>I modified the <code>agrovoc-lookup.py</code> script to save its results as a CSV, with the subject, language, type of match (preferred, alternate, and total number of matches) rather than save two separate files
|
||
<ul>
|
||
<li>Note that I see <code>prefLabel</code>, <code>matchedPrefLabel</code>, and <code>altLabel</code> in the REST API responses and I’m not sure what the second one means</li>
|
||
<li>I emailed FAO’s AGROVOC contact to ask them</li>
|
||
<li>They responded to say that <code>matchedPrefLabel</code> is not a property in SKOS/SKOSXL vocabulary, but their SKOSMOS system seems to use it to hint that the search terms matched a <code>prefLabel</code> in another language</li>
|
||
<li>I will treat the <code>matchedPrefLabel</code> values as if they were <code>prefLabel</code> values for the indicated language then</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<h2 id="2020-07-07">2020-07-07</h2>
|
||
<ul>
|
||
<li>Peter asked me to send him a list of sponsors on CGSpace</li>
|
||
</ul>
|
||
<pre><code>dspace=# \COPY (SELECT DISTINCT text_value as "dc.description.sponsorship", count(text_value) FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=29 GROUP BY "dc.description.sponsorship" ORDER BY count DESC) TO /tmp/2020-07-07-sponsors.csv WITH CSV HEADER;
|
||
COPY 707
|
||
</code></pre><ul>
|
||
<li>I ran it quickly through my <code>csv-metadata-quality</code> tool and found two issues that I will correct with <code>fix-metadata-values.py</code> on CGSpace immediately:</li>
|
||
</ul>
|
||
<pre><code>$ cat 2020-07-07-fix-sponsors.csv
|
||
dc.description.sponsorship,correct
|
||
"Ministe`re des Affaires Etrange`res et Européennes, France","Ministère des Affaires Étrangères et Européennes, France"
|
||
"Global Food Security Programme, United Kingdom","Global Food Security Programme, United Kingdom"
|
||
$ ./fix-metadata-values.py -i 2020-07-07-fix-sponsors.csv -db dspace -u dspace -p 'fuuu' -f dc.description.sponsorship -t correct -m 29
|
||
</code></pre><ul>
|
||
<li>Upload the Capacity Development July newsletter to CGSpace for Ben Hack because Abenet and Bizu usually do it, but they are currently offline due to the Internet being turned off in Ethiopia
|
||
<ul>
|
||
<li>Here: <a href="https://hdl.handle.net/10568/108708">https://hdl.handle.net/10568/108708</a></li>
|
||
</ul>
|
||
</li>
|
||
<li>I implemented the Dimensions.ai badge on DSpace Test for Peter to see, as he’s been asking me for awhile:</li>
|
||
</ul>
|
||
<p><img src="/cgspace-notes/2020/07/dimensions-badge.png" alt="Dimensions.ai badge"></p>
|
||
<ul>
|
||
<li>It was easy once I figured out how to do the XSLT in the DSpace theme (need to get the DOI link and remove the “<a href="https://doi.org/%22">https://doi.org/"</a> from the string)
|
||
<ul>
|
||
<li>Actually this raised an issue that the Altmetric badges weren’t enabled in our DSpace 6 branch yet because I had forgotten to copy the config</li>
|
||
<li>Also, I noticed a big issue in both our DSpace 5 and DSpace 6 branches related to the <code>$identifier_doi</code> variable being defined incorrectly and thus never getting set (has to do with DRI)</li>
|
||
<li>I fixed both and now the Altmetric badge and the Dimensions badge both appear… nice</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<p><img src="/cgspace-notes/2020/07/dimensions-badge2.png" alt="Altmetric and Dimensions.ai badge"></p>
|
||
<h2 id="2020-07-08">2020-07-08</h2>
|
||
<ul>
|
||
<li>Generate a CSV of all the AGROVOC subjects that didn’t match from the top 6500 I exported earlier this week:</li>
|
||
</ul>
|
||
<pre><code>$ csvgrep -c 'number of matches' -r "^0$" 2020-07-05-cgspace-subjects.csv | csvcut -c 1 > 2020-07-05-cgspace-invalid-subjects.csv
|
||
</code></pre><ul>
|
||
<li>Yesterday Gabriela from CIP emailed to say that she was removing the accents from her authors’ names because of “funny character” issues with reports generated from CGSpace
|
||
<ul>
|
||
<li>I told her that it’s probably her Windows / Excel that is messing up the data, and she figured out how to open them correctly!</li>
|
||
<li>Now she says she doesn’t want to remove the accents after all and she sent me a new list of corrections</li>
|
||
<li>I used csvgrep and found a few where she is still removing accents:</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<pre><code>$ csvgrep -c 2 -r "^.+$" ~/Downloads/cip-authors-GH-20200706.csv | csvgrep -c 1 -r "^.*[À-ú].*$" | csvgrep -c 2 -r "^.*[À-ú].*$" -i | csvcut -c 1,2
|
||
dc.contributor.author,correction
|
||
"López, G.","Lopez, G."
|
||
"Gómez, R.","Gomez, R."
|
||
"García, M.","Garcia, M."
|
||
"Mejía, A.","Mejia, A."
|
||
"Quiróz, Roberto A.","Quiroz, R."
|
||
</code></pre><ul>
|
||
<li>
|
||
<p>csvgrep from the csvkit suite is <em>so cool</em>:</p>
|
||
<ul>
|
||
<li>Select lines with column two (the correction) having a value</li>
|
||
<li>Select lines with column one (the original author name) having an accent / diacritic</li>
|
||
<li>Select lines with column two (the correction) NOT having an accent (ie, she’s not removing an accent)</li>
|
||
<li>Select columns one and two</li>
|
||
</ul>
|
||
</li>
|
||
<li>
|
||
<p>Peter said he liked the work I didn on the badges yesterday so I put some finishing touches on it to detect more DOI URI styles and pushed it to the <code>5_x-prod</code> branch</p>
|
||
<ul>
|
||
<li>I will port it to DSpace 6 soon</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<p><img src="/cgspace-notes/2020/07/altmetrics-dimensions-badges.png" alt="Altmetric and Dimensions badges"></p>
|
||
<ul>
|
||
<li>I wrote a quick script to lookup organizations (affiliations) in the Research Organization Repository (ROR) JSON data release v5
|
||
<ul>
|
||
<li>I want to use this to evaluate ROR as a controlled vocabulary for CGSpace and MELSpace</li>
|
||
<li>I exported a list of affiliations from CGSpace:</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<pre><code>dspace=# \COPY (SELECT DISTINCT text_value as "cg.contributor.affiliation", count(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC) to /tmp/2020-07-08-affiliations.csv WITH CSV HEADER;
|
||
</code></pre><ul>
|
||
<li>Then I stripped the CSV header and quotes to make it a plain text file and ran <code>ror-lookup.py</code>:</li>
|
||
</ul>
|
||
<pre><code>$ ./ror-lookup.py -i /tmp/2020-07-08-affiliations.txt -r ror.json -o 2020-07-08-affiliations-ror.csv -d
|
||
$ wc -l /tmp/2020-07-08-affiliations.txt
|
||
5866 /tmp/2020-07-08-affiliations.txt
|
||
$ csvgrep -c matched -m true 2020-07-08-affiliations-ror.csv | wc -l
|
||
1406
|
||
$ csvgrep -c matched -m false 2020-07-08-affiliations-ror.csv | wc -l
|
||
4462
|
||
</code></pre><ul>
|
||
<li>So, minus the CSV header, we have 1405 case-insensitive matches out of 5866 (23.9%)</li>
|
||
</ul>
|
||
<h2 id="2020-07-09">2020-07-09</h2>
|
||
<ul>
|
||
<li>Atmire responded to the ticket about DSpace 6 and Solr yesterday
|
||
<ul>
|
||
<li>They said that the CUA issue is due to the “unmigrated” Solr records and that we should delete them</li>
|
||
<li>I told them that <a href="https://wiki.lyrasis.org/display/DSDOC6x/SOLR+Statistics+Maintenance">the “unmigrated” IDs are a known issue in DSpace 6</a> and we should rather figure out why they are unmigrated</li>
|
||
<li>I didn’t see any discussion on the dspace-tech mailing list or on DSpace Jira about unmigrated IDs, so I sent a mail to the mailing list to ask</li>
|
||
</ul>
|
||
</li>
|
||
<li>I updated <code>ror-lookup.py</code> to check aliases and acronyms as well and now the results are better for CGSpace’s affiliation list:</li>
|
||
</ul>
|
||
<pre><code>$ wc -l /tmp/2020-07-08-affiliations.txt
|
||
5866 /tmp/2020-07-08-affiliations.txt
|
||
$ csvgrep -c matched -m true 2020-07-08-affiliations-ror.csv | wc -l
|
||
1516
|
||
$ csvgrep -c matched -m false 2020-07-08-affiliations-ror.csv | wc -l
|
||
4352
|
||
</code></pre><ul>
|
||
<li>So now our matching improves to 1515 out of 5866 (25.8%)</li>
|
||
<li>Gabriela from CIP said that I should run the author corrections minus those that remove accent characters so I will run it on CGSpace:</li>
|
||
</ul>
|
||
<pre><code>$ ./fix-metadata-values.py -i /tmp/2020-07-09-fix-90-cip-authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t correction -m 3
|
||
</code></pre><ul>
|
||
<li>Apply 110 fixes and 90 deletions to sponsorships that Peter sent me a few days ago:</li>
|
||
</ul>
|
||
<pre><code>$ ./fix-metadata-values.py -i /tmp/2020-07-07-fix-110-sponsors.csv -db dspace -u dspace -p 'fuuu' -f dc.description.sponsorship -t 'correct/action' -m 29
|
||
$ ./delete-metadata-values.py -i /tmp/2020-07-07-delete-90-sponsors.csv -db dspace -u dspace -p 'fuuu' -f dc.description.sponsorship -m 29
|
||
</code></pre><ul>
|
||
<li>Start a full Discovery re-index on CGSpace:</li>
|
||
</ul>
|
||
<pre><code>$ time chrt -b 0 dspace index-discovery -b
|
||
|
||
real 94m21.413s
|
||
user 9m40.364s
|
||
sys 2m37.246s
|
||
</code></pre><ul>
|
||
<li>I modified <code>crossref-funders-lookup.py</code> to be case insensitive and now CGSpace’s sponsors match 173 out of 534 (32.4%):</li>
|
||
</ul>
|
||
<pre><code>$ ./crossref-funders-lookup.py -i 2020-07-09-cgspace-sponsors.txt -o 2020-07-09-cgspace-sponsors-crossref.csv -d -e a.orth@cgiar.org
|
||
$ wc -l 2020-07-09-cgspace-sponsors.txt
|
||
534 2020-07-09-cgspace-sponsors.txt
|
||
$ csvgrep -c matched -m true 2020-07-09-cgspace-sponsors-crossref.csv | wc -l
|
||
174
|
||
</code></pre><h2 id="2020-07-12">2020-07-12</h2>
|
||
<ul>
|
||
<li>On 2020-07-10 Macaroni Bros emailed to ask if there are issues with CGSpace because they are getting HTTP 504 on the REST API
|
||
<ul>
|
||
<li>First, I looked in Munin and I see high number of DSpace sessions and threads on Friday evening around midnight, though that was much later than his email:</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<p><img src="/cgspace-notes/2020/07/jmx_dspace_sessions-day.png" alt="DSpace sessions">
|
||
<img src="/cgspace-notes/2020/07/threads-day.png" alt="Threads">
|
||
<img src="/cgspace-notes/2020/07/postgres_locks_ALL-day.png" alt="PostgreSQL locks">
|
||
<img src="/cgspace-notes/2020/07/postgres_transactions_ALL-day.png" alt="PostgreSQL transactions"></p>
|
||
<ul>
|
||
<li>CPU load and memory were not high then, but there was some load on the database and firewall…
|
||
<ul>
|
||
<li>Looking in the nginx logs I see a few IPs we’ve seen recently, like those 199.47.x.x IPs from Turnitin (which I need to remember to purge from Solr again because I didn’t update the spider agents on CGSpace yet) and some new one 186.112.8.167</li>
|
||
<li>Also, the Turnitin bot doesn’t re-use its Tomcat JSESSIONID, I see this from today:</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<pre><code># grep 199.47.87 dspace.log.2020-07-12 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort | uniq | wc -l
|
||
2815
|
||
</code></pre><ul>
|
||
<li>So I need to add this alternative user-agent to the Tomcat Crawler Session Manager valve to force it to re-use a common bot session</li>
|
||
<li>There are around 9,000 requests from <code>186.112.8.167</code> in Colombia and has the user agent <code>Java/1.8.0_241</code>, but those were mostly to REST API and I don’t see any hits in Solr</li>
|
||
<li>Earlier in the day Linode had alerted that there was high outgoing bandwidth
|
||
<ul>
|
||
<li>I see some new bot from 134.155.96.78 made ~10,000 requests with the user agent… but it appears to already be in our DSpace user agent list via COUNTER-Robots:</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<pre><code>Mozilla/5.0 (compatible; heritrix/3.4.0-SNAPSHOT-2019-02-07T13:53:20Z +http://ifm.uni-mannheim.de)
|
||
</code></pre><ul>
|
||
<li>Generate a list of sponsors to update our controlled vocabulary:</li>
|
||
</ul>
|
||
<pre><code>dspace=# \COPY (SELECT DISTINCT text_value as "dc.description.sponsorship", count(text_value) FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=29 GROUP BY "dc.description.sponsorship" ORDER BY count DESC LIMIT 125) TO /tmp/2020-07-12-sponsors.csv;
|
||
COPY 125
|
||
dspace=# \q
|
||
$ csvcut -c 1 --tabs /tmp/2020-07-12-sponsors.csv > dspace/config/controlled-vocabularies/dc-description-sponsorship.xml
|
||
# add XML formatting
|
||
$ dspace/config/controlled-vocabularies/dc-description-sponsorship.xml
|
||
$ tidy -xml -utf8 -m -iq -w 0 dspace/config/controlled-vocabularies/dc-description-sponsorship.xml
|
||
</code></pre><ul>
|
||
<li>Deploy latest <code>5_x-prod</code> branch on CGSpace (linode18), run all system updates, and reboot the server
|
||
<ul>
|
||
<li>After rebooting it I had to restart Tomcat 7 once to get all Solr statistics cores to come up properly</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<h2 id="2020-07-13">2020-07-13</h2>
|
||
<ul>
|
||
<li>I recommended to Marie–Angelique that we use ROR for CG Core V2 (<a href="https://github.com/AgriculturalSemantics/cg-core/issues/27">#27</a>)</li>
|
||
<li>Purge 2,700 hits from CodeObia IP addresses in CGSpace statistics… I wonder when they will figure out how to use a bot user agent</li>
|
||
</ul>
|
||
<h2 id="2020-07-14">2020-07-14</h2>
|
||
<ul>
|
||
<li>I ran the <code>dspace cleanup -v</code> process on CGSpace and got an error:</li>
|
||
</ul>
|
||
<pre><code>Error: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
|
||
Detail: Key (bitstream_id)=(189618) is still referenced from table "bundle".
|
||
</code></pre><ul>
|
||
<li>The solution is, as always:</li>
|
||
</ul>
|
||
<pre><code>$ psql -d dspace -U dspace -c 'update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (189618, 188837);'
|
||
UPDATE 1
|
||
</code></pre><ul>
|
||
<li>Udana from WLE asked me about some items that didn’t show Altmetric donuts
|
||
<ul>
|
||
<li>I checked his list and at least three of them actually <em>did</em> show donuts, and for four others I tweeted them manually to see if they would get a donut in a few hours:
|
||
<ul>
|
||
<li><a href="https://hdl.handle.net/10568/108477">https://hdl.handle.net/10568/108477</a></li>
|
||
<li><a href="https://hdl.handle.net/10568/108475">https://hdl.handle.net/10568/108475</a></li>
|
||
<li><a href="https://hdl.handle.net/10568/108361">https://hdl.handle.net/10568/108361</a></li>
|
||
<li><a href="https://hdl.handle.net/10568/108360">https://hdl.handle.net/10568/108360</a></li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<h2 id="2020-07-15">2020-07-15</h2>
|
||
<ul>
|
||
<li>All four IWMI items that I tweeted yesterday have Altmetric donuts with a score of 1 now…</li>
|
||
<li>Export CGSpace countries to check them against ISO 3166-1 and ISO 3166-3 (historic countries):</li>
|
||
</ul>
|
||
<pre><code>dspace=# \COPY (SELECT DISTINCT text_value FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=228) TO /tmp/2020-07-15-countries.csv;
|
||
COPY 194
|
||
</code></pre><ul>
|
||
<li>I wrote a script <code>iso3166-lookup.py</code> to check them:</li>
|
||
</ul>
|
||
<pre><code>$ ./iso3166-1-lookup.py -i /tmp/2020-07-15-countries.csv -o /tmp/2020-07-15-countries-resolved.csv
|
||
$ csvgrep -c matched -m false /tmp/2020-07-15-countries-resolved.csv
|
||
country,match type,matched
|
||
CAPE VERDE,,false
|
||
"KOREA, REPUBLIC",,false
|
||
PALESTINE,,false
|
||
"CONGO, DR",,false
|
||
COTE D'IVOIRE,,false
|
||
RUSSIA,,false
|
||
SYRIA,,false
|
||
"KOREA, DPR",,false
|
||
SWAZILAND,,false
|
||
MICRONESIA,,false
|
||
TIBET,,false
|
||
ZAIRE,,false
|
||
COCOS ISLANDS,,false
|
||
LAOS,,false
|
||
IRAN,,false
|
||
</code></pre><ul>
|
||
<li>Check the database for DOIs that are not in the preferred “<a href="https://doi.org/%22">https://doi.org/"</a> format:</li>
|
||
</ul>
|
||
<pre><code>dspace=# \COPY (SELECT text_value as "cg.identifier.doi" FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=220 AND text_value NOT LIKE 'https://doi.org/%') TO /tmp/2020-07-15-doi.csv WITH CSV HEADER;
|
||
COPY 186
|
||
</code></pre><ul>
|
||
<li>Then I imported them into OpenRefine and replaced them in a new “correct” column using this GREL transform:</li>
|
||
</ul>
|
||
<pre><code>value.replace("dx.doi.org", "doi.org").replace("http://", "https://").replace("https://dx,doi,org", "https://doi.org").replace("https://doi.dx.org", "https://doi.org").replace("https://dx.doi:", "https://doi.org").replace("DOI: ", "https://doi.org/").replace("doi: ", "https://doi.org/").replace("http://dx.doi.org", "https://doi.org").replace("https://dx. doi.org. ", "https://doi.org").replace("https://dx.doi", "https://doi.org").replace("https://dx.doi:", "https://doi.org/").replace("hdl.handle.net", "doi.org")
|
||
</code></pre><ul>
|
||
<li>Then I fixed the DOIs on CGSpace:</li>
|
||
</ul>
|
||
<pre><code>$ ./fix-metadata-values.py -i /tmp/2020-07-15-fix-164-DOIs.csv -db dspace -u dspace -p 'fuuu' -f cg.identifier.doi -t 'correct' -m 220
|
||
</code></pre><ul>
|
||
<li>I filed <a href="https://salsa.debian.org/iso-codes-team/iso-codes/-/issues/10">an issue on Debian’s iso-codes</a> project to ask why “Swaziland” does not appear in the ISO 3166-3 list of historical country names despite it being changed to “Eswatini” in 2018.</li>
|
||
<li>Atmire responded about the Solr issue
|
||
<ul>
|
||
<li>They said that it seems like a DSpace issue so that it’s not their responsibility, and nobody responded to my question on the dspace-tech mailing list…</li>
|
||
<li>I said I would try to do a migration on DSpace Test with more of CGSpace’s Solr data to try and approximate how much of our data be affected</li>
|
||
<li>I also asked them about the Tomcat 8.5 issue with CUA as well as the CUA group name issue that I had asked originally in April</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<h2 id="2020-07-20">2020-07-20</h2>
|
||
<ul>
|
||
<li>Looking at the nginx logs on CGSpace (linode18) last night I see that the Macaroni Bros have started using a unique identifier for at least one of their harvesters:</li>
|
||
</ul>
|
||
<pre><code>217.64.192.138 - - [20/Jul/2020:01:01:39 +0200] "GET /rest/rest/bitstreams/114779/retrieve HTTP/1.0" 302 138 "-" "ILRI Livestock Website Publications importer BOT"
|
||
</code></pre><ul>
|
||
<li>I still see 12,000 records in Solr from this user agent, though.
|
||
<ul>
|
||
<li>I wonder why the DSpace bot list didn’t get those… because it has “bot” which should cause Solr to not log the hit</li>
|
||
</ul>
|
||
</li>
|
||
<li>I purged ~30,000 hits from Solr statistics based on the IPs above, but also for some agents like Drupal (which isn’t in the list yet) and OgScrper (which is as of 2020-03)</li>
|
||
<li>Some of my user agent patterns had been incorporated into COUNTER-Robots in 2020-07, but not all
|
||
<ul>
|
||
<li>I closed the <a href="https://github.com/atmire/COUNTER-Robots/pull/34">old pull request</a> and created a <a href="https://github.com/atmire/COUNTER-Robots/pull/36">new one</a></li>
|
||
<li>Then I updated the lists in the <code>5_x-prod</code> and 6.x branches</li>
|
||
</ul>
|
||
</li>
|
||
<li>I re-ran the <code>check-spider-hits.sh</code> script with the new lists and purged another 14,000 more stats hits for several years each (2020, 2019, 2018, 2017, 2016), around 70,000 total</li>
|
||
<li>I looked at the <a href="https://clarisa.cgiar.org/">CLARISA</a> institutions list again, since I hadn’t looked at it in over six months:</li>
|
||
</ul>
|
||
<pre><code>$ cat ~/Downloads/response_1595270924560.json | jq '.[] | {name: .name}' | grep name | awk -F: '{print $2}' | sed -e 's/"//g' -e 's/^ //' -e '1iname' | csvcut -l | sed '1s/line_number/id/' > /tmp/clarisa-institutions.csv
|
||
</code></pre><ul>
|
||
<li>The API still needs a key unless you query from Swagger web interface
|
||
<ul>
|
||
<li>They currently have 3,469 institutions…</li>
|
||
<li>Also, they still combine multiple text names into one string along with acronyms and countries:
|
||
<ul>
|
||
<li>Bundesministerium für wirtschaftliche Zusammenarbeit und Entwicklung / Federal Ministry of Economic Cooperation and Development (Germany)</li>
|
||
<li>Ministerio del Ambiente / Ministry of Environment (Peru)</li>
|
||
<li>Carthage University / Université de Carthage</li>
|
||
<li>Sweet Potato Research Institute (SPRI) of Chinese Academy of Agricultural Sciences (CAAS)</li>
|
||
</ul>
|
||
</li>
|
||
<li>And I checked the list with my csv-metadata-quality tool and found it still has whitespace and unnecessary Unicode characters in several records:</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<pre><code>$ csv-metadata-quality -i /tmp/clarisa-institutions.csv -o /tmp/clarisa-institutions-cleaned.csv
|
||
Removing excessive whitespace (name): Comitato Internazionale per lo Sviluppo dei Popoli / International Committee for the Development of Peoples
|
||
Removing excessive whitespace (name): Deutsche Landwirtschaftsgesellschaft / German agriculture society
|
||
Removing excessive whitespace (name): Institute of Arid Regions of Medenine
|
||
Replacing unnecessary Unicode (U+00AD): Bundesministerium für wirtschaftliche Zusammenarbeit und Entwicklung / Federal Ministry of Economic Cooperation and Development (Germany)
|
||
Removing unnecessary Unicode (U+200B): Agencia de Servicios a la Comercialización y Desarrollo de Mercados Agropecuarios
|
||
</code></pre><ul>
|
||
<li>I think the ROR is much better in every possible way</li>
|
||
<li>Re-enabled all the yearly Solr statistics cores on DSpace Test (linode26) because they had been disabled by Atmire when they were testing on the server
|
||
<ul>
|
||
<li>Run system updates on the server and reboot it</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<h2 id="2020-07-21">2020-07-21</h2>
|
||
<ul>
|
||
<li>I built the latest 6.x branch on DSpace Test (linode26) and I noticed a few Font Awesome icons are missing in the Atmire CUA statlets
|
||
<ul>
|
||
<li>One was simple to fix by adding it to our font library in <code>fontawesome.js</code>, but there are two more that are printing hex values instead of using HTML elements:</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<p><img src="/cgspace-notes/2020/07/cua-font-awesome.png" alt="Atmire CUA missing icons"></p>
|
||
<ul>
|
||
<li>I had previously thought these were fixed by setting the <code>font-family</code> on the elements, but it doesn’t appear to be working now
|
||
<ul>
|
||
<li>I filed a ticket with Atmire to ask them to use the HTML elements instead, as their code already uses those elsewhere</li>
|
||
<li>I don’t want to go back to using the large webfonts with CSS because the SVG + JS method saves us ~140KiB and causes at least three fewer network requests</li>
|
||
</ul>
|
||
</li>
|
||
<li>I started processing the 2019 stats in a batch of 1 million on DSpace Test:</li>
|
||
</ul>
|
||
<pre><code>$ export JAVA_OPTS='-Dfile.encoding=UTF-8 -Xmx2048m'
|
||
$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 1000000 -i statistics-2019
|
||
...
|
||
*** Statistics Records with Legacy Id ***
|
||
|
||
6,359,966 Bistream View
|
||
2,204,775 Item View
|
||
139,266 Community View
|
||
131,234 Collection View
|
||
948,529 Community Search
|
||
593,974 Collection Search
|
||
1,682,818 Unexpected Type & Full Site
|
||
--------------------------------------
|
||
12,060,562 TOTAL
|
||
</code></pre><ul>
|
||
<li>The statistics-2019 finished processing after about 9 hours so I started the 2018 ones:</li>
|
||
</ul>
|
||
<pre><code>$ export JAVA_OPTS='-Dfile.encoding=UTF-8 -Xmx2048m'
|
||
$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 1000000 -i statistics-2018
|
||
*** Statistics Records with Legacy Id ***
|
||
|
||
3,684,394 Bistream View
|
||
2,183,032 Item View
|
||
131,222 Community View
|
||
79,348 Collection View
|
||
345,529 Collection Search
|
||
322,223 Community Search
|
||
874,107 Unexpected Type & Full Site
|
||
--------------------------------------
|
||
7,619,855 TOTAL
|
||
</code></pre><ul>
|
||
<li>Moayad finally made OpenRXV use a unique user agent:</li>
|
||
</ul>
|
||
<pre><code>OpenRXV harvesting bot; https://github.com/ilri/OpenRXV
|
||
</code></pre><ul>
|
||
<li>I see nearly 200,000 hits in Solr from the IP address, though, so I need to make sure those are old ones from before today
|
||
<ul>
|
||
<li>I purged the hits for 178.62.93.141 as well as any from the old <code>axios/0.19.2</code> user agent</li>
|
||
<li>I made some requests with and without the new user agent and only the ones without showed up in Solr</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<h2 id="2020-07-22">2020-07-22</h2>
|
||
<ul>
|
||
<li>Atmire merged my latest bot suggestions to the COUNTER-Robots project:
|
||
<ul>
|
||
<li><a href="https://github.com/atmire/COUNTER-Robots/pull/36">Add new bots</a></li>
|
||
<li><a href="https://github.com/atmire/COUNTER-Robots/pull/35">COUNTER_Robots_list.json: Escape literal dots</a></li>
|
||
<li><a href="https://github.com/atmire/COUNTER-Robots/pull/33">COUNTER_Robots_list.json: Remove anchors from okhttp</a></li>
|
||
</ul>
|
||
</li>
|
||
<li>I will update the agent patterns on the CGSpace <code>5_x-prod</code> and 6.x branches</li>
|
||
<li>Make some changes to the Bootstrap CSS and HTML configuration to improve readability and style on the CG Core v2 metadata reference guide and send a pull request to Marie (<a href="https://github.com/AgriculturalSemantics/cg-core/pull/29">#29</a>)</li>
|
||
</ul>
|
||
<!-- raw HTML omitted -->
|
||
|
||
|
||
|
||
|
||
|
||
</article>
|
||
|
||
|
||
|
||
</div> <!-- /.blog-main -->
|
||
|
||
<aside class="col-sm-3 ml-auto blog-sidebar">
|
||
|
||
|
||
|
||
<section class="sidebar-module">
|
||
<h4>Recent Posts</h4>
|
||
<ol class="list-unstyled">
|
||
|
||
|
||
<li><a href="/cgspace-notes/2020-07/">July, 2020</a></li>
|
||
|
||
<li><a href="/cgspace-notes/2020-06/">June, 2020</a></li>
|
||
|
||
<li><a href="/cgspace-notes/2020-05/">May, 2020</a></li>
|
||
|
||
<li><a href="/cgspace-notes/2020-04/">April, 2020</a></li>
|
||
|
||
<li><a href="/cgspace-notes/2020-03/">March, 2020</a></li>
|
||
|
||
</ol>
|
||
</section>
|
||
|
||
|
||
|
||
|
||
<section class="sidebar-module">
|
||
<h4>Links</h4>
|
||
<ol class="list-unstyled">
|
||
|
||
<li><a href="https://cgspace.cgiar.org">CGSpace</a></li>
|
||
|
||
<li><a href="https://dspacetest.cgiar.org">DSpace Test</a></li>
|
||
|
||
<li><a href="https://github.com/ilri/DSpace">CGSpace @ GitHub</a></li>
|
||
|
||
</ol>
|
||
</section>
|
||
|
||
</aside>
|
||
|
||
|
||
</div> <!-- /.row -->
|
||
</div> <!-- /.container -->
|
||
|
||
|
||
|
||
<footer class="blog-footer">
|
||
<p dir="auto">
|
||
|
||
Blog template created by <a href="https://twitter.com/mdo">@mdo</a>, ported to Hugo by <a href='https://twitter.com/mralanorth'>@mralanorth</a>.
|
||
|
||
</p>
|
||
<p>
|
||
<a href="#">Back to top</a>
|
||
</p>
|
||
</footer>
|
||
|
||
|
||
</body>
|
||
|
||
</html>
|