280 lines
11 KiB
HTML
280 lines
11 KiB
HTML
|
<!DOCTYPE html>
|
||
|
|
<html lang="en" >
|
||
|
|
|
||
|
|
<head>
|
||
|
|
<meta charset="utf-8">
|
||
|
|
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
|
||
|
|
|
||
|
|
|
||
|
|
<meta property="og:title" content="May, 2021" />
|
||
|
|
<meta property="og:description" content="2021-05-01
|
||
|
|
|
||
|
|
I looked at the top user agents and IPs in the Solr statistics for last month and I see these user agents:
|
||
|
|
|
||
|
|
“RI/1.0”, 1337
|
||
|
|
“Microsoft Office Word 2014”, 941
|
||
|
|
|
||
|
|
|
||
|
|
I will add the RI/1.0 pattern to our DSpace agents overload and purge them from Solr (we had previously seen this agent with 9,000 hits or so in 2020-09), but I think I will leave the Microsoft Word one… as that’s an actual user…
|
||
|
|
" />
|
||
|
|
<meta property="og:type" content="article" />
|
||
|
|
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2021-05/" />
|
||
|
|
<meta property="article:published_time" content="2021-05-02T09:50:54+03:00" />
|
||
|
|
<meta property="article:modified_time" content="2021-05-02T09:50:54+03:00" />
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
<meta name="twitter:card" content="summary"/>
|
||
|
|
<meta name="twitter:title" content="May, 2021"/>
|
||
|
|
<meta name="twitter:description" content="2021-05-01
|
||
|
|
|
||
|
|
I looked at the top user agents and IPs in the Solr statistics for last month and I see these user agents:
|
||
|
|
|
||
|
|
“RI/1.0”, 1337
|
||
|
|
“Microsoft Office Word 2014”, 941
|
||
|
|
|
||
|
|
|
||
|
|
I will add the RI/1.0 pattern to our DSpace agents overload and purge them from Solr (we had previously seen this agent with 9,000 hits or so in 2020-09), but I think I will leave the Microsoft Word one… as that’s an actual user…
|
||
|
|
"/>
|
||
|
|
<meta name="generator" content="Hugo 0.82.0" />
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
<script type="application/ld+json">
|
||
|
|
{
|
||
|
|
"@context": "http://schema.org",
|
||
|
|
"@type": "BlogPosting",
|
||
|
|
"headline": "May, 2021",
|
||
|
|
"url": "https://alanorth.github.io/cgspace-notes/2021-05/",
|
||
|
|
"wordCount": "537",
|
||
|
|
"datePublished": "2021-05-02T09:50:54+03:00",
|
||
|
|
"dateModified": "2021-05-02T09:50:54+03:00",
|
||
|
|
"author": {
|
||
|
|
"@type": "Person",
|
||
|
|
"name": "Alan Orth"
|
||
|
|
},
|
||
|
|
"keywords": "Notes"
|
||
|
|
}
|
||
|
|
</script>
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
<link rel="canonical" href="https://alanorth.github.io/cgspace-notes/2021-05/">
|
||
|
|
|
||
|
|
<title>May, 2021 | CGSpace Notes</title>
|
||
|
|
|
||
|
|
|
||
|
|
<!-- combined, minified CSS -->
|
||
|
|
|
||
|
|
<link href="https://alanorth.github.io/cgspace-notes/css/style.beb8012edc08ba10be012f079d618dc243812267efe62e11f22fe49618f976a4.css" rel="stylesheet" integrity="sha256-vrgBLtwIuhC+AS8HnWGNwkOBImfv5i4R8i/klhj5dqQ=" crossorigin="anonymous">
|
||
|
|
|
||
|
|
|
||
|
|
<!-- minified Font Awesome for SVG icons -->
|
||
|
|
|
||
|
|
<script defer src="https://alanorth.github.io/cgspace-notes/js/fontawesome.min.ffbfea088a9a1666ec65c3a8cb4906e2a0e4f92dc70dbbf400a125ad2422123a.js" integrity="sha256-/7/qCIqaFmbsZcOoy0kG4qDk+S3HDbv0AKElrSQiEjo=" crossorigin="anonymous"></script>
|
||
|
|
|
||
|
|
<!-- RSS 2.0 feed -->
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
</head>
|
||
|
|
|
||
|
|
<body>
|
||
|
|
|
||
|
|
|
||
|
|
<div class="blog-masthead">
|
||
|
|
<div class="container">
|
||
|
|
<nav class="nav blog-nav">
|
||
|
|
<a class="nav-link " href="https://alanorth.github.io/cgspace-notes/">Home</a>
|
||
|
|
</nav>
|
||
|
|
</div>
|
||
|
|
</div>
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
<header class="blog-header">
|
||
|
|
<div class="container">
|
||
|
|
<h1 class="blog-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/" rel="home">CGSpace Notes</a></h1>
|
||
|
|
<p class="lead blog-description" dir="auto">Documenting day-to-day work on the <a href="https://cgspace.cgiar.org">CGSpace</a> repository.</p>
|
||
|
|
</div>
|
||
|
|
</header>
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
<div class="container">
|
||
|
|
<div class="row">
|
||
|
|
<div class="col-sm-8 blog-main">
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
<article class="blog-post">
|
||
|
|
<header>
|
||
|
|
<h2 class="blog-post-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/2021-05/">May, 2021</a></h2>
|
||
|
|
<p class="blog-post-meta">
|
||
|
|
<time datetime="2021-05-02T09:50:54+03:00">Sun May 02, 2021</time>
|
||
|
|
in
|
||
|
|
<span class="fas fa-folder" aria-hidden="true"></span> <a href="/cgspace-notes/categories/notes/" rel="category tag">Notes</a>
|
||
|
|
|
||
|
|
|
||
|
|
</p>
|
||
|
|
</header>
|
||
|
|
<h2 id="2021-05-01">2021-05-01</h2>
|
||
|
|
<ul>
|
||
|
|
<li>I looked at the top user agents and IPs in the Solr statistics for last month and I see these user agents:
|
||
|
|
<ul>
|
||
|
|
<li>“RI/1.0”, 1337</li>
|
||
|
|
<li>“Microsoft Office Word 2014”, 941</li>
|
||
|
|
</ul>
|
||
|
|
</li>
|
||
|
|
<li>I will add the RI/1.0 pattern to our DSpace agents overload and purge them from Solr (we had previously seen this agent with 9,000 hits or so in 2020-09), but I think I will leave the Microsoft Word one… as that’s an actual user…</li>
|
||
|
|
</ul>
|
||
|
|
<ul>
|
||
|
|
<li>I should probably add the <code>RI/1.0</code> pattern to COUNTER-Robots project</li>
|
||
|
|
<li>As well as these IPs:
|
||
|
|
<ul>
|
||
|
|
<li>193.169.254.178, 21648</li>
|
||
|
|
<li>181.62.166.177, 20323</li>
|
||
|
|
<li>45.146.166.180, 19376</li>
|
||
|
|
</ul>
|
||
|
|
</li>
|
||
|
|
<li>The first IP seems to be in Estonia and their requests to the REST API change user agents from curl to Mac OS X to Windows and more
|
||
|
|
<ul>
|
||
|
|
<li>Also, they seem to be trying to exploit something:</li>
|
||
|
|
</ul>
|
||
|
|
</li>
|
||
|
|
</ul>
|
||
|
|
<pre><code class="language-console" data-lang="console">193.169.254.178 - - [21/Apr/2021:01:59:01 +0200] "GET /rest/collections/1179/items?limit=812&expand=metadata\x22%20and%20\x2221\x22=\x2221 HTTP/1.1" 400 5 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
|
||
|
|
193.169.254.178 - - [21/Apr/2021:02:00:36 +0200] "GET /rest/collections/1179/items?limit=812&expand=metadata-21%2B21*01 HTTP/1.1" 200 458201 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
|
||
|
|
193.169.254.178 - - [21/Apr/2021:02:00:36 +0200] "GET /rest/collections/1179/items?limit=812&expand=metadata'||lower('')||' HTTP/1.1" 400 5 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
|
||
|
|
193.169.254.178 - - [21/Apr/2021:02:02:10 +0200] "GET /rest/collections/1179/items?limit=812&expand=metadata'%2Brtrim('')%2B' HTTP/1.1" 200 458209 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
|
||
|
|
</code></pre><ul>
|
||
|
|
<li>I will report the IP on abuseipdb.com and purge their hits from Solr</li>
|
||
|
|
<li>The second IP is in Colombia and is making thousands of requests for what looks like some test site:</li>
|
||
|
|
</ul>
|
||
|
|
<pre><code class="language-console" data-lang="console">181.62.166.177 - - [20/Apr/2021:22:48:42 +0200] "GET /rest/collections/d1e11546-c62a-4aee-af91-fd482b3e7653/items?expand=metadata HTTP/2.0" 200 123613 "http://cassavalighthousetest.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36"
|
||
|
|
181.62.166.177 - - [20/Apr/2021:22:55:39 +0200] "GET /rest/collections/d1e11546-c62a-4aee-af91-fd482b3e7653/items?expand=metadata HTTP/2.0" 200 123613 "http://cassavalighthousetest.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36"
|
||
|
|
</code></pre><ul>
|
||
|
|
<li>But this site does not exist (yet?)
|
||
|
|
<ul>
|
||
|
|
<li>I will purge them from Solr</li>
|
||
|
|
</ul>
|
||
|
|
</li>
|
||
|
|
<li>The third IP is in Russia apparently, and the user agent has the <code>pl-PL</code> locale with thousands of requests like this:</li>
|
||
|
|
</ul>
|
||
|
|
<pre><code class="language-console" data-lang="console">45.146.166.180 - - [18/Apr/2021:16:28:44 +0200] "GET /bitstream/handle/10947/4153/.AAS%202014%20Annual%20Report.pdf?sequence=1%22%29%29%20AND%201691%3DUTL_INADDR.GET_HOST_ADDRESS%28CHR%28113%29%7C%7CCHR%28118%29%7C%7CCHR%28113%29%7C%7CCHR%28106%29%7C%7CCHR%28113%29%7C%7C%28SELECT%20%28CASE%20WHEN%20%281691%3D1691%29%20THEN%201%20ELSE%200%20END%29%20FROM%20DUAL%29%7C%7CCHR%28113%29%7C%7CCHR%2898%29%7C%7CCHR%28122%29%7C%7CCHR%28120%29%7C%7CCHR%28113%29%29%20AND%20%28%28%22RKbp%22%3D%22RKbp&isAllowed=y HTTP/1.1" 200 918998 "http://cgspace.cgiar.org:80/bitstream/handle/10947/4153/.AAS 2014 Annual Report.pdf" "Mozilla/5.0 (Windows; U; Windows NT 5.1; pl-PL) AppleWebKit/523.15 (KHTML, like Gecko) Version/3.0 Safari/523.15"
|
||
|
|
</code></pre><ul>
|
||
|
|
<li>I will purge these all with my <code>check-spider-ip-hits.sh</code> script:</li>
|
||
|
|
</ul>
|
||
|
|
<pre><code class="language-console" data-lang="console">$ ./ilri/check-spider-ip-hits.sh -f /tmp/ips.txt -p
|
||
|
|
Purging 21648 hits from 193.169.254.178 in statistics
|
||
|
|
Purging 20323 hits from 181.62.166.177 in statistics
|
||
|
|
Purging 19376 hits from 45.146.166.180 in statistics
|
||
|
|
|
||
|
|
Total number of bot hits purged: 61347
|
||
|
|
</code></pre><h2 id="2021-05-02">2021-05-02</h2>
|
||
|
|
<ul>
|
||
|
|
<li>Check the AReS Harvester indexes:</li>
|
||
|
|
</ul>
|
||
|
|
<pre><code class="language-console" data-lang="console">$ curl -s http://localhost:9200/_cat/indices | grep openrxv-items
|
||
|
|
yellow open openrxv-items-temp H-CGsyyLTaqAj6-nKXZ-7w 1 1 0 0 283b 283b
|
||
|
|
yellow open openrxv-items-final ul3SKsa7Q9Cd_K7qokBY_w 1 1 103951 0 254mb 254mb
|
||
|
|
$ curl -s 'http://localhost:9200/_alias/' | python -m json.tool
|
||
|
|
...
|
||
|
|
"openrxv-items-temp": {
|
||
|
|
"aliases": {}
|
||
|
|
},
|
||
|
|
"openrxv-items-final": {
|
||
|
|
"aliases": {
|
||
|
|
"openrxv-items": {}
|
||
|
|
}
|
||
|
|
},
|
||
|
|
</code></pre><ul>
|
||
|
|
<li>I think they look OK (<code>openrxv-items</code> is an alias of <code>openrxv-items-final</code>), but I took a backup just in case:</li>
|
||
|
|
</ul>
|
||
|
|
<pre><code class="language-console" data-lang="console">$ elasticdump --input=http://localhost:9200/openrxv-items --output=/home/aorth/openrxv-items_mapping.json --type=mapping
|
||
|
|
$ elasticdump --input=http://localhost:9200/openrxv-items --output=/home/aorth/openrxv-items_data.json --type=data --limit=1000
|
||
|
|
</code></pre><ul>
|
||
|
|
<li>Then I started an indexing in the AReS Explorer admin dashboard</li>
|
||
|
|
<li>The indexing finished, but it looks like the aliases are messed up again:</li>
|
||
|
|
</ul>
|
||
|
|
<pre><code class="language-console" data-lang="console">$ curl -s http://localhost:9200/_cat/indices | grep openrxv-items
|
||
|
|
yellow open openrxv-items-temp H-CGsyyLTaqAj6-nKXZ-7w 1 1 104165 105024 487.7mb 487.7mb
|
||
|
|
yellow open openrxv-items-final d0tbMM_SRWimirxr_gm9YA 1 1 937 0 2.2mb 2.2mb
|
||
|
|
</code></pre><!-- raw HTML omitted -->
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
</article>
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
</div> <!-- /.blog-main -->
|
||
|
|
|
||
|
|
<aside class="col-sm-3 ml-auto blog-sidebar">
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
<section class="sidebar-module">
|
||
|
|
<h4>Recent Posts</h4>
|
||
|
|
<ol class="list-unstyled">
|
||
|
|
|
||
|
|
|
||
|
|
<li><a href="/cgspace-notes/2021-05/">May, 2021</a></li>
|
||
|
|
|
||
|
|
<li><a href="/cgspace-notes/2021-04/">April, 2021</a></li>
|
||
|
|
|
||
|
|
<li><a href="/cgspace-notes/2021-03/">March, 2021</a></li>
|
||
|
|
|
||
|
|
<li><a href="/cgspace-notes/cgspace-cgcorev2-migration/">CGSpace CG Core v2 Migration</a></li>
|
||
|
|
|
||
|
|
<li><a href="/cgspace-notes/2021-02/">February, 2021</a></li>
|
||
|
|
|
||
|
|
</ol>
|
||
|
|
</section>
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
<section class="sidebar-module">
|
||
|
|
<h4>Links</h4>
|
||
|
|
<ol class="list-unstyled">
|
||
|
|
|
||
|
|
<li><a href="https://cgspace.cgiar.org">CGSpace</a></li>
|
||
|
|
|
||
|
|
<li><a href="https://dspacetest.cgiar.org">DSpace Test</a></li>
|
||
|
|
|
||
|
|
<li><a href="https://github.com/ilri/DSpace">CGSpace @ GitHub</a></li>
|
||
|
|
|
||
|
|
</ol>
|
||
|
|
</section>
|
||
|
|
|
||
|
|
</aside>
|
||
|
|
|
||
|
|
|
||
|
|
</div> <!-- /.row -->
|
||
|
|
</div> <!-- /.container -->
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
<footer class="blog-footer">
|
||
|
|
<p dir="auto">
|
||
|
|
|
||
|
|
Blog template created by <a href="https://twitter.com/mdo">@mdo</a>, ported to Hugo by <a href='https://twitter.com/mralanorth'>@mralanorth</a>.
|
||
|
|
|
||
|
|
</p>
|
||
|
|
<p>
|
||
|
|
<a href="#">Back to top</a>
|
||
|
|
</p>
|
||
|
|
</footer>
|
||
|
|
|
||
|
|
|
||
|
|
</body>
|
||
|
|
|
||
|
|
</html>
|