2021-11-01 09:49:21 +01:00
<!DOCTYPE html>
< html lang = "en" >
< head >
< meta charset = "utf-8" >
< meta name = "viewport" content = "width=device-width, initial-scale=1, shrink-to-fit=no" >
< meta property = "og:title" content = "November, 2021" / >
2021-11-03 14:56:15 +01:00
< meta property = "og:description" content = "2021-11-02
I experimented with manually sharding the Solr statistics on DSpace Test
First I exported all the 2019 stats from CGSpace:
$ ./run.sh -s http://localhost:8081/solr/statistics -f ' time:2019-*' -a export -o statistics-2019.json -k uid
$ zstd statistics-2019.json
" />
2021-11-01 09:49:21 +01:00
< meta property = "og:type" content = "article" / >
< meta property = "og:url" content = "https://alanorth.github.io/cgspace-notes/2021-11/" / >
2021-11-03 14:56:15 +01:00
< meta property = "article:published_time" content = "2021-11-02T22:27:07+02:00" / >
2021-11-07 10:26:32 +01:00
< meta property = "article:modified_time" content = "2021-11-03T15:56:15+02:00" / >
2021-11-01 09:49:21 +01:00
< meta name = "twitter:card" content = "summary" / >
< meta name = "twitter:title" content = "November, 2021" / >
2021-11-03 14:56:15 +01:00
< meta name = "twitter:description" content = "2021-11-02
I experimented with manually sharding the Solr statistics on DSpace Test
First I exported all the 2019 stats from CGSpace:
$ ./run.sh -s http://localhost:8081/solr/statistics -f ' time:2019-*' -a export -o statistics-2019.json -k uid
$ zstd statistics-2019.json
"/>
2021-11-01 09:49:21 +01:00
< meta name = "generator" content = "Hugo 0.88.1" / >
< script type = "application/ld+json" >
{
"@context": "http://schema.org",
"@type": "BlogPosting",
"headline": "November, 2021",
"url": "https://alanorth.github.io/cgspace-notes/2021-11/",
2021-11-07 10:26:32 +01:00
"wordCount": "468",
2021-11-03 14:56:15 +01:00
"datePublished": "2021-11-02T22:27:07+02:00",
2021-11-07 10:26:32 +01:00
"dateModified": "2021-11-03T15:56:15+02:00",
2021-11-01 09:49:21 +01:00
"author": {
"@type": "Person",
"name": "Alan Orth"
},
"keywords": "Notes"
}
< / script >
< link rel = "canonical" href = "https://alanorth.github.io/cgspace-notes/2021-11/" >
< title > November, 2021 | CGSpace Notes< / title >
<!-- combined, minified CSS -->
< link href = "https://alanorth.github.io/cgspace-notes/css/style.beb8012edc08ba10be012f079d618dc243812267efe62e11f22fe49618f976a4.css" rel = "stylesheet" integrity = "sha256-vrgBLtwIuhC+AS8HnWGNwkOBImfv5i4R8i/klhj5dqQ=" crossorigin = "anonymous" >
<!-- minified Font Awesome for SVG icons -->
< script defer src = "https://alanorth.github.io/cgspace-notes/js/fontawesome.min.f5072c55a0721857184db93a50561d7dc13975b4de2e19db7f81eb5f3fa57270.js" integrity = "sha256-9QcsVaByGFcYTbk6UFYdfcE5dbTeLhnbf4HrXz+lcnA=" crossorigin = "anonymous" > < / script >
<!-- RSS 2.0 feed -->
< / head >
< body >
< div class = "blog-masthead" >
< div class = "container" >
< nav class = "nav blog-nav" >
< a class = "nav-link " href = "https://alanorth.github.io/cgspace-notes/" > Home< / a >
< / nav >
< / div >
< / div >
< header class = "blog-header" >
< div class = "container" >
< h1 class = "blog-title" dir = "auto" > < a href = "https://alanorth.github.io/cgspace-notes/" rel = "home" > CGSpace Notes< / a > < / h1 >
< p class = "lead blog-description" dir = "auto" > Documenting day-to-day work on the < a href = "https://cgspace.cgiar.org" > CGSpace< / a > repository.< / p >
< / div >
< / header >
< div class = "container" >
< div class = "row" >
< div class = "col-sm-8 blog-main" >
< article class = "blog-post" >
< header >
< h2 class = "blog-post-title" dir = "auto" > < a href = "https://alanorth.github.io/cgspace-notes/2021-11/" > November, 2021< / a > < / h2 >
< p class = "blog-post-meta" >
2021-11-03 14:56:15 +01:00
< time datetime = "2021-11-02T22:27:07+02:00" > Tue Nov 02, 2021< / time >
2021-11-01 09:49:21 +01:00
in
< span class = "fas fa-folder" aria-hidden = "true" > < / span > < a href = "/cgspace-notes/categories/notes/" rel = "category tag" > Notes< / a >
< / p >
< / header >
2021-11-03 14:56:15 +01:00
< h2 id = "2021-11-02" > 2021-11-02< / h2 >
< ul >
< li > I experimented with manually sharding the Solr statistics on DSpace Test< / li >
< li > First I exported all the 2019 stats from CGSpace:< / li >
< / ul >
< pre tabindex = "0" > < code class = "language-console" data-lang = "console" > $ ./run.sh -s http://localhost:8081/solr/statistics -f 'time:2019-*' -a export -o statistics-2019.json -k uid
$ zstd statistics-2019.json
< / code > < / pre > < ul >
< li > Then on DSpace Test I created a < code > statistics-2019< / code > core with the same instance dir as the main < code > statistics< / code > core (as < a href = "https://wiki.lyrasis.org/display/DSDOC6x/Testing+Solr+Shards" > illustrated in the DSpace docs< / a > )< / li >
< / ul >
< pre tabindex = "0" > < code class = "language-console" data-lang = "console" > $ mkdir -p /home/dspacetest.cgiar.org/solr/statistics-2019/data
# create core in Solr admin
$ curl -s " http://localhost:8081/solr/statistics/update?softCommit=true" -H " Content-Type: text/xml" --data-binary " < delete> < query> time:2019-*< /query> < /delete> "
$ ./run.sh -s http://localhost:8081/solr/statistics-2019 -a import -o statistics-2019.json -k uid
< / code > < / pre > < ul >
< li > The key thing above is that you create the core in the Solr admin UI, but the data directory must already exist so you have to do that first in the file system< / li >
< li > I restarted the server after the import was done to see if the cores would come back up OK
< ul >
< li > I remember last time I tried this the manually created statistics cores didn’ t come back up after I rebooted, but this time they did< / li >
< / ul >
< / li >
< / ul >
< h2 id = "2021-11-03" > 2021-11-03< / h2 >
< ul >
< li > While inspecting the stats for the new statistics-2019 shard on DSpace Test I noticed that I can’ t find any stats via the DSpace Statistics API for an item that < em > should< / em > have some
< ul >
< li > I checked on CGSpace’ s and I can’ t find them there either, but I see them in Solr when I query in the admin UI< / li >
< li > I need to debug that, but it doesn’ t seem to be related to the sharding… < / li >
< / ul >
< / li >
< / ul >
2021-11-07 10:26:32 +01:00
< h2 id = "2021-11-04" > 2021-11-04< / h2 >
< ul >
< li > I spent a little bit of time debugging the Solr bug with the statistics-2019 shard but couldn’ t reproduce it for the few items I tested
< ul >
< li > So that’ s good, it seems the sharding worked< / li >
< / ul >
< / li >
< li > Linode alerted me to high CPU usage on CGSpace (linode18) yesterday
< ul >
< li > Looking at the Solr hits from yesterday I see 91.213.50.11 making 2,300 requests< / li >
< li > According to AbuseIPDB.com this is owned by Registrarus LLC (registrarus.ru) and it has been reported for malicious activity by several users< / li >
< li > The ASN is 50340 (SELECTEL-MSK, RU)< / li >
< li > They are attempting SQL injection:< / li >
< / ul >
< / li >
< / ul >
< pre tabindex = "0" > < code class = "language-console" data-lang = "console" > 91.213.50.11 - - [03/Nov/2021:06:47:20 +0100] " HEAD /bitstream/handle/10568/106239/U19ArtSimonikovaChromosomeInthomNodev.pdf?sequence=1%60%20WHERE%206158%3D6158%20AND%204894%3D4741--%20kIlq& isAllowed=y HTTP/1.1" 200 0 " https://cgspace.cgiar.org:443/bitstream/handle/10568/106239/U19ArtSimonikovaChromosomeInthomNodev.pdf" " Mozilla/5.0 (X11; U; Linux i686; en-CA; rv:1.8.0.10) Gecko/20070223 Fedora/1.5.0.10-1.fc5 Firefox/1.5.0.10"
< / code > < / pre > < ul >
< li > Another is in China, and they grabbed 1,200 PDFs from the REST API in under an hour:< / li >
< / ul >
< pre tabindex = "0" > < code class = "language-console" data-lang = "console" > # zgrep 222.129.53.160 /var/log/nginx/rest.log.2.gz | wc -l
1178
< / code > < / pre > < ul >
< li > I will continue to split the Solr statistics back into year-shards on DSpace Test (linode26)
< ul >
< li > Today I did all 2018 stats… < / li >
< li > I want to see if there is a noticeable change in JVM memory, Solr response time, etc< / li >
< / ul >
< / li >
< / ul >
< h2 id = "2021-11-07" > 2021-11-07< / h2 >
< ul >
< li > Update all Docker containers on AReS and rebuild OpenRXV:< / li >
< / ul >
< pre tabindex = "0" > < code class = "language-console" data-lang = "console" > $ docker images | grep -v ^REPO | sed 's/ \+/:/g' | cut -d: -f1,2 | xargs -L1 docker pull
$ docker-compose build
< / code > < / pre > < ul >
< li > Then restart the server and start a fresh harvest< / li >
< li > Continue splitting the Solr statistics into yearly shards on DSpace Test (doing 2017 today)< / li >
< / ul >
2021-11-01 09:49:21 +01:00
<!-- raw HTML omitted -->
< / article >
< / div > <!-- /.blog - main -->
< aside class = "col-sm-3 ml-auto blog-sidebar" >
< section class = "sidebar-module" >
< h4 > Recent Posts< / h4 >
< ol class = "list-unstyled" >
< li > < a href = "/cgspace-notes/2021-11/" > November, 2021< / a > < / li >
< li > < a href = "/cgspace-notes/2021-10/" > October, 2021< / a > < / li >
< li > < a href = "/cgspace-notes/2021-09/" > September, 2021< / a > < / li >
< li > < a href = "/cgspace-notes/2021-08/" > August, 2021< / a > < / li >
< li > < a href = "/cgspace-notes/2021-07/" > July, 2021< / a > < / li >
< / ol >
< / section >
< section class = "sidebar-module" >
< h4 > Links< / h4 >
< ol class = "list-unstyled" >
< li > < a href = "https://cgspace.cgiar.org" > CGSpace< / a > < / li >
< li > < a href = "https://dspacetest.cgiar.org" > DSpace Test< / a > < / li >
< li > < a href = "https://github.com/ilri/DSpace" > CGSpace @ GitHub< / a > < / li >
< / ol >
< / section >
< / aside >
< / div > <!-- /.row -->
< / div > <!-- /.container -->
< footer class = "blog-footer" >
< p dir = "auto" >
Blog template created by < a href = "https://twitter.com/mdo" > @mdo< / a > , ported to Hugo by < a href = 'https://twitter.com/mralanorth' > @mralanorth< / a > .
< / p >
< p >
< a href = "#" > Back to top< / a >
< / p >
< / footer >
< / body >
< / html >
