mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2024-11-25 16:08:19 +01:00
632 lines
37 KiB
HTML
632 lines
37 KiB
HTML
<!DOCTYPE html>
|
|
<html lang="en" >
|
|
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
|
|
|
|
|
|
<meta property="og:title" content="December, 2021" />
|
|
<meta property="og:description" content="2021-12-01
|
|
|
|
Atmire merged some changes I had submitted to the COUNTER-Robots project
|
|
I updated our local spider user agents and then re-ran the list with my check-spider-hits.sh script on CGSpace:
|
|
|
|
$ ./ilri/check-spider-hits.sh -f /tmp/agents -p
|
|
Purging 1989 hits from The Knowledge AI in statistics
|
|
Purging 1235 hits from MaCoCu in statistics
|
|
Purging 455 hits from WhatsApp in statistics
|
|
|
|
Total number of bot hits purged: 3679
|
|
" />
|
|
<meta property="og:type" content="article" />
|
|
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2021-12/" />
|
|
<meta property="article:published_time" content="2021-12-01T16:07:07+02:00" />
|
|
<meta property="article:modified_time" content="2022-01-09T10:39:51+02:00" />
|
|
|
|
|
|
|
|
<meta name="twitter:card" content="summary"/>
|
|
<meta name="twitter:title" content="December, 2021"/>
|
|
<meta name="twitter:description" content="2021-12-01
|
|
|
|
Atmire merged some changes I had submitted to the COUNTER-Robots project
|
|
I updated our local spider user agents and then re-ran the list with my check-spider-hits.sh script on CGSpace:
|
|
|
|
$ ./ilri/check-spider-hits.sh -f /tmp/agents -p
|
|
Purging 1989 hits from The Knowledge AI in statistics
|
|
Purging 1235 hits from MaCoCu in statistics
|
|
Purging 455 hits from WhatsApp in statistics
|
|
|
|
Total number of bot hits purged: 3679
|
|
"/>
|
|
<meta name="generator" content="Hugo 0.125.0">
|
|
|
|
|
|
|
|
<script type="application/ld+json">
|
|
{
|
|
"@context": "http://schema.org",
|
|
"@type": "BlogPosting",
|
|
"headline": "December, 2021",
|
|
"url": "https://alanorth.github.io/cgspace-notes/2021-12/",
|
|
"wordCount": "2686",
|
|
"datePublished": "2021-12-01T16:07:07+02:00",
|
|
"dateModified": "2022-01-09T10:39:51+02:00",
|
|
"author": {
|
|
"@type": "Person",
|
|
"name": "Alan Orth"
|
|
},
|
|
"keywords": "Notes"
|
|
}
|
|
</script>
|
|
|
|
|
|
|
|
<link rel="canonical" href="https://alanorth.github.io/cgspace-notes/2021-12/">
|
|
|
|
<title>December, 2021 | CGSpace Notes</title>
|
|
|
|
|
|
<!-- combined, minified CSS -->
|
|
|
|
<link href="https://alanorth.github.io/cgspace-notes/css/style.c6ba80bc50669557645abe05f86b73cc5af84408ed20f1551a267bc19ece8228.css" rel="stylesheet" integrity="sha256-xrqAvFBmlVdkWr4F+GtzzFr4RAjtIPFVGiZ7wZ7Ogig=" crossorigin="anonymous">
|
|
|
|
|
|
<!-- minified Font Awesome for SVG icons -->
|
|
|
|
<script defer src="https://alanorth.github.io/cgspace-notes/js/fontawesome.min.f5072c55a0721857184db93a50561d7dc13975b4de2e19db7f81eb5f3fa57270.js" integrity="sha256-9QcsVaByGFcYTbk6UFYdfcE5dbTeLhnbf4HrXz+lcnA=" crossorigin="anonymous"></script>
|
|
|
|
<!-- RSS 2.0 feed -->
|
|
|
|
|
|
|
|
|
|
</head>
|
|
|
|
<body>
|
|
|
|
|
|
<div class="blog-masthead">
|
|
<div class="container">
|
|
<nav class="nav blog-nav">
|
|
<a class="nav-link " href="https://alanorth.github.io/cgspace-notes/">Home</a>
|
|
</nav>
|
|
</div>
|
|
</div>
|
|
|
|
|
|
|
|
|
|
<header class="blog-header">
|
|
<div class="container">
|
|
<h1 class="blog-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/" rel="home">CGSpace Notes</a></h1>
|
|
<p class="lead blog-description" dir="auto">Documenting day-to-day work on the <a href="https://cgspace.cgiar.org">CGSpace</a> repository.</p>
|
|
</div>
|
|
</header>
|
|
|
|
|
|
|
|
|
|
<div class="container">
|
|
<div class="row">
|
|
<div class="col-sm-8 blog-main">
|
|
|
|
|
|
|
|
|
|
<article class="blog-post">
|
|
<header>
|
|
<h2 class="blog-post-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/2021-12/">December, 2021</a></h2>
|
|
<p class="blog-post-meta">
|
|
<time datetime="2021-12-01T16:07:07+02:00">Wed Dec 01, 2021</time>
|
|
in
|
|
<span class="fas fa-folder" aria-hidden="true"></span> <a href="/categories/notes/" rel="category tag">Notes</a>
|
|
|
|
|
|
</p>
|
|
</header>
|
|
<h2 id="2021-12-01">2021-12-01</h2>
|
|
<ul>
|
|
<li>Atmire merged some changes I had submitted to the COUNTER-Robots project</li>
|
|
<li>I updated our local spider user agents and then re-ran the list with my <code>check-spider-hits.sh</code> script on CGSpace:</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ ./ilri/check-spider-hits.sh -f /tmp/agents -p
|
|
</span></span><span style="display:flex;"><span>Purging 1989 hits from The Knowledge AI in statistics
|
|
</span></span><span style="display:flex;"><span>Purging 1235 hits from MaCoCu in statistics
|
|
</span></span><span style="display:flex;"><span>Purging 455 hits from WhatsApp in statistics
|
|
</span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010">
|
|
</span></span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010"></span>Total number of bot hits purged: 3679
|
|
</span></span></code></pre></div><h2 id="2021-12-02">2021-12-02</h2>
|
|
<ul>
|
|
<li>Francesca from Alliance asked me for help with approving a submission that gets stuck
|
|
<ul>
|
|
<li>I looked at the PostgreSQL activity and the locks are back up like they were earlier this week</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ psql -c <span style="color:#e6db74">"SELECT application_name FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid"</span> | sort | uniq -c | sort -n
|
|
</span></span><span style="display:flex;"><span> 1
|
|
</span></span><span style="display:flex;"><span> 1 ------------------
|
|
</span></span><span style="display:flex;"><span> 1 (1437 rows)
|
|
</span></span><span style="display:flex;"><span> 1 application_name
|
|
</span></span><span style="display:flex;"><span> 9 psql
|
|
</span></span><span style="display:flex;"><span> 1428 dspaceWeb
|
|
</span></span></code></pre></div><ul>
|
|
<li>Munin shows the same:</li>
|
|
</ul>
|
|
<p><img src="/cgspace-notes/2021/12/postgres_locks_ALL-week.png" alt="PostgreSQL locks week"></p>
|
|
<ul>
|
|
<li>Last month I enabled the <code>log_lock_waits</code> in PostgreSQL so I checked the log and was surprised to find only a few since I restarted PostgreSQL three days ago:</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span># grep -E <span style="color:#e6db74">'^2021-(11-29|11-30|12-01|12-02)'</span> /var/log/postgresql/postgresql-10-main.log | grep -c <span style="color:#e6db74">'still waiting for'</span>
|
|
</span></span><span style="display:flex;"><span>15
|
|
</span></span></code></pre></div><ul>
|
|
<li>I think you could analyze the locks for the <code>dspaceWeb</code> user (XMLUI) and find out what queries were locking… but it’s so much information and I don’t know where to start
|
|
<ul>
|
|
<li>For now I just restarted PostgreSQL…</li>
|
|
<li>Francesca was able to do her submission immediately…</li>
|
|
</ul>
|
|
</li>
|
|
<li>On a related note, I want to enable the <code>pg_stat_statement</code> feature to see which queries get run the most, so I created the extension on the CGSpace database</li>
|
|
<li>I was doing some research on PostgreSQL locks and found some interesting things to consider
|
|
<ul>
|
|
<li>The default <code>lock_timeout</code> is 0, aka disabled</li>
|
|
<li>The default <code>statement_timeout</code> is 0, aka disabled</li>
|
|
<li>It seems to be recommended to start by setting <code>statement_timeout</code> first, rule of thumb <a href="https://github.com/jberkus/annotated.conf/blob/master/postgresql.10.simple.conf#L211">ten times longer than your longest query</a></li>
|
|
</ul>
|
|
</li>
|
|
<li>Mark Wood mentioned the <code>checker</code> cron job that apparently runs in one transaction and might be an issue
|
|
<ul>
|
|
<li>I definitely saw it holding a bunch of locks for ~30 minutes during the first part of its execution, then it dropped them and did some other less-intensive things without locks</li>
|
|
</ul>
|
|
</li>
|
|
<li>Bizuwork was still not receiving emails even after we fixed the SMTP access on CGSpace
|
|
<ul>
|
|
<li>After some troubleshooting it turns out that the emails from CGSpace were going in her Junk!</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<h2 id="2021-12-03">2021-12-03</h2>
|
|
<ul>
|
|
<li>I see GARDIAN is now using a “GARDIAN” user agent finally
|
|
<ul>
|
|
<li>I will add them to our local spider agent override in DSpace so that the hits don’t get counted in Solr</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<h2 id="2021-12-05">2021-12-05</h2>
|
|
<ul>
|
|
<li>Proof fifty records Abenet sent me from Africa Rice Center (“AfricaRice 1st batch Import”)
|
|
<ul>
|
|
<li>Fixed forty-six incorrect collections</li>
|
|
<li>Cleaned up and normalize affiliations</li>
|
|
<li>Cleaned up dates (extra <code>*</code> character in all?)</li>
|
|
<li>Cleaned up citation format</li>
|
|
<li>Fixed some encoding issues in abstracts</li>
|
|
<li>Removed empty columns</li>
|
|
<li>Removed one duplicate: Enhancing Rice Productivity and Soil Nitrogen Using Dual-Purpose Cowpea-NERICA® Rice Sequence in Degraded Savanna</li>
|
|
<li>Added volume and issue metadata by extracting it from the citations</li>
|
|
<li>All PDFs hosted on davidpublishing.com are dead…</li>
|
|
<li>All DOIs linking to African Journal of Agricultural Research are dead…</li>
|
|
<li>Fixed a handful of items marked as “Open Access” that are actually closed</li>
|
|
<li>Added many missing ISSNs</li>
|
|
<li>Added many missing countries/regions</li>
|
|
<li>Fixed invalid AGROVOC terms and added some more based on article subjects</li>
|
|
</ul>
|
|
</li>
|
|
<li>I also made some minor changes to the <a href="https://github.com/ilri/csv-metadata-quality">CSV Metadata Quality Checker</a>
|
|
<ul>
|
|
<li>Added the ability to check if the item’s title exists in the citation</li>
|
|
<li>Updated to only run the mojibake check if we’re not running in unsafe mode (so we don’t print the same warning during both the check and fix steps)</li>
|
|
</ul>
|
|
</li>
|
|
<li>I ran the re-harvesting on AReS</li>
|
|
</ul>
|
|
<h2 id="2021-12-06">2021-12-06</h2>
|
|
<ul>
|
|
<li>Some minor work on the <code>check-duplicates.py</code> script I wrote last month
|
|
<ul>
|
|
<li>I found some corner cases where there were items that matched in the database, but they were <code>in_archive=f</code> and or <code>withdrawn=t</code> so I check that before trying to resolve the handles of potential duplicates</li>
|
|
</ul>
|
|
</li>
|
|
<li>More work on the Africa Rice Center 1st batch import
|
|
<ul>
|
|
<li>I merged the metadata for three duplicates in Africa Rice’s items and mapped them on CGSpace</li>
|
|
<li>I did a bit more work to add missing AGROVOC subjects, countries, regions, extents, etc and then uploaded the forty-six items to CGSpace</li>
|
|
</ul>
|
|
</li>
|
|
<li>I started looking at the seventy CAS records that Abenet has been working on for the past few months</li>
|
|
</ul>
|
|
<h2 id="2021-12-07">2021-12-07</h2>
|
|
<ul>
|
|
<li>I sent Vini from CGIAR CAS some questions about the seventy records I was working on yesterday
|
|
<ul>
|
|
<li>Also, I ran the <code>check-duplicates.py</code> script on them and found that they might ALL be duplicates!!!</li>
|
|
<li>I tweaked the script a bit more to use the issue dates as a third criteria and now there are less duplicates, but it’s still at least twenty or so…</li>
|
|
<li>The script now checks if the issue date of the item in the CSV and the issue date of the item in the database are less than 365 days apart (by default)</li>
|
|
<li>For example, many items like “Annual Report 2020” can have similar title and type to previous annual reports, but are not duplicates</li>
|
|
</ul>
|
|
</li>
|
|
<li>I noticed a strange user agent in the XMLUI logs on CGSpace:</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>20.84.225.129 - - [07/Dec/2021:11:51:24 +0100] "GET /handle/10568/33203 HTTP/1.1" 200 6328 "-" "python-requests/2.25.1"
|
|
</span></span><span style="display:flex;"><span>20.84.225.129 - - [07/Dec/2021:11:51:27 +0100] "GET /handle/10568/33203 HTTP/2.0" 200 6315 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36"
|
|
</span></span></code></pre></div><ul>
|
|
<li>I looked into it more and I see a dozen other IPs using that user agent, and they are all owned by Microsoft
|
|
<ul>
|
|
<li>It could be someone on Azure?</li>
|
|
<li>I opened <a href="https://github.com/atmire/COUNTER-Robots/pull/49">a pull request to COUNTER-Robots</a> and I’ll add this user agent to our local override until they decide to include it or not</li>
|
|
</ul>
|
|
</li>
|
|
<li>I purged 34,000 hits from this user agent in our Solr statistics:</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ ./ilri/check-spider-hits.sh -f dspace/config/spiders/agents/ilri -p
|
|
</span></span><span style="display:flex;"><span>Purging 34458 hits from HeadlessChrome in statistics
|
|
</span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010">
|
|
</span></span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010"></span>Total number of bot hits purged: 34458
|
|
</span></span></code></pre></div><ul>
|
|
<li>Meeting with partners about repositories in the One CGIAR</li>
|
|
</ul>
|
|
<h2 id="2021-12-08">2021-12-08</h2>
|
|
<ul>
|
|
<li>Finalize country/region changes in csv-metadata-quality checker and release v0.5.0: <a href="https://github.com/ilri/csv-metadata-quality/releases/tag/v0.5.0">https://github.com/ilri/csv-metadata-quality/releases/tag/v0.5.0</a>
|
|
<ul>
|
|
<li>This also includes the mojibake fixes and title/citation checks and some bug fixes</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<h2 id="2021-12-09">2021-12-09</h2>
|
|
<ul>
|
|
<li>Help Francesca upload the dataset for one CIAT publication (it has like 100 authors so we did it via CSV)</li>
|
|
</ul>
|
|
<h2 id="2021-12-12">2021-12-12</h2>
|
|
<ul>
|
|
<li>Patch OpenRXV’s Elasticsearch for the CVE-2021-44228 log4j vulnerability and re-deploy AReS
|
|
<ul>
|
|
<li>I added <code>-Dlog4j2.formatMsgNoLookups=true</code> to the Elasticsearch Java environment</li>
|
|
</ul>
|
|
</li>
|
|
<li>Run AReS harvesting</li>
|
|
</ul>
|
|
<h2 id="2021-12-13">2021-12-13</h2>
|
|
<ul>
|
|
<li>I ran the <code>check-duplicates.py</code> script on the 1,000 items from the CGIAR System Office TAC/ICW/Green Cover archives and found hundreds or thousands of potential duplicates
|
|
<ul>
|
|
<li>I sent feedback to Gaia</li>
|
|
</ul>
|
|
</li>
|
|
<li>Help Jacquie from WorldFish try to find all outputs for the Fish CRP because there are a few different formats for that name</li>
|
|
<li>Create a temporary account for Rafael Rodriguez on DSpace Test so he can investigate the submission workflow
|
|
<ul>
|
|
<li>I added him to the admin group on the Alliance community…</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<h2 id="2021-12-14">2021-12-14</h2>
|
|
<ul>
|
|
<li>I finally caught some stuck locks on CGSpace after checking several times per day for the last week:</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ psql -c <span style="color:#e6db74">"SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid"</span> | wc -l
|
|
</span></span><span style="display:flex;"><span>1508
|
|
</span></span></code></pre></div><ul>
|
|
<li>Now looking at the locks query sorting by age of locks:</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ cat locks-age.sql
|
|
</span></span><span style="display:flex;"><span>SELECT a.datname,
|
|
</span></span><span style="display:flex;"><span> l.relation::regclass,
|
|
</span></span><span style="display:flex;"><span> l.transactionid,
|
|
</span></span><span style="display:flex;"><span> l.mode,
|
|
</span></span><span style="display:flex;"><span> l.GRANTED,
|
|
</span></span><span style="display:flex;"><span> a.usename,
|
|
</span></span><span style="display:flex;"><span> a.query,
|
|
</span></span><span style="display:flex;"><span> a.query_start,
|
|
</span></span><span style="display:flex;"><span> age(now(), a.query_start) AS "age",
|
|
</span></span><span style="display:flex;"><span> a.pid
|
|
</span></span><span style="display:flex;"><span>FROM pg_stat_activity a
|
|
</span></span><span style="display:flex;"><span>JOIN pg_locks l ON l.pid = a.pid
|
|
</span></span><span style="display:flex;"><span>ORDER BY a.query_start;
|
|
</span></span></code></pre></div><ul>
|
|
<li>The oldest locks are 9 hours and 26 minutes old and the time on the server is <code>Tue Dec 14 18:41:58 CET 2021</code>, so it seems something happened around 9:15 this morning
|
|
<ul>
|
|
<li>I looked at the maintenance tasks and there is nothing running around then (only the sitemap update that runs at 8AM, and should be quick)</li>
|
|
<li>I looked at the DSpace log, but didn’t see anything interesting there: only editors making edits…</li>
|
|
<li>I looked at the nginx REST API logs and saw lots of GET action there from Drupal sites harvesting us…</li>
|
|
<li>So I’m not sure what it causing this… perhaps something in the XMLUI submission / task workflow</li>
|
|
<li>For now I just ran all system updates and rebooted the server</li>
|
|
<li>I also enabled Atmire’s <code>log-db-activity.sh</code> script to run every four hours (in the DSpace user’s crontab) so perhaps that will be better than me checking manually</li>
|
|
</ul>
|
|
</li>
|
|
<li>Regarding Gaia’s 1,000 items to upload to CGSpace, I checked the eighteen Green Cover records and there are no duplicates, so that’s at least a starting point!
|
|
<ul>
|
|
<li>I sent her a spreadsheet with the eighteen items with a new collection column to indicate where they should go</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<h2 id="2021-12-16">2021-12-16</h2>
|
|
<ul>
|
|
<li>Working on the CGIAR CAS Green Cover records for Gaia
|
|
<ul>
|
|
<li>Add months to dcterms.issued from PDFs</li>
|
|
<li>Add languages</li>
|
|
<li>Format and fix several authors</li>
|
|
</ul>
|
|
</li>
|
|
<li>I created a SAF archive with SAFBuilder and then imported it to DSpace Test:</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ JAVA_OPTS<span style="color:#f92672">=</span><span style="color:#e6db74">"-Xmx1024m -Dfile.encoding=UTF-8"</span> dspace import --add --eperson<span style="color:#f92672">=</span>fuuu@fuuu.com --source /tmp/SimpleArchiveFormat --mapfile<span style="color:#f92672">=</span>./2021-12-16-green-covers.map
|
|
</span></span></code></pre></div><h2 id="2021-12-19">2021-12-19</h2>
|
|
<ul>
|
|
<li>I tried to update all Docker containers on AReS and then run a build, but I got an error in the backend:</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>> openrxv-backend@0.0.1 build
|
|
</span></span><span style="display:flex;"><span>> nest build
|
|
</span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010">
|
|
</span></span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010"></span>node_modules/@elastic/elasticsearch/api/types.d.ts:2454:13 - error TS2456: Type alias 'AggregationsAggregate' circularly references itself.
|
|
</span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010">
|
|
</span></span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010"></span>2454 export type AggregationsAggregate = AggregationsSingleBucketAggregate | AggregationsAutoDateHistogramAggregate | AggregationsFiltersAggregate | AggregationsSignificantTermsAggregate<any> | AggregationsTermsAggregate<any> | AggregationsBucketAggregate | AggregationsCompositeBucketAggregate | AggregationsMultiBucketAggregate<AggregationsBucket> | AggregationsMatrixStatsAggregate | AggregationsKeyedValueAggregate | AggregationsMetricAggregate
|
|
</span></span><span style="display:flex;"><span> ~~~~~~~~~~~~~~~~~~~~~
|
|
</span></span><span style="display:flex;"><span>node_modules/@elastic/elasticsearch/api/types.d.ts:3209:13 - error TS2456: Type alias 'AggregationsSingleBucketAggregate' circularly references itself.
|
|
</span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010">
|
|
</span></span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010"></span>3209 export type AggregationsSingleBucketAggregate = AggregationsSingleBucketAggregateKeys
|
|
</span></span><span style="display:flex;"><span> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
</span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010">
|
|
</span></span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010"></span>Found 2 error(s).
|
|
</span></span></code></pre></div><ul>
|
|
<li>I’m not sure why because I build the backend successfully on my local machine…
|
|
<ul>
|
|
<li>For now I just ran all the system updates and rebooted the machine (linode20)</li>
|
|
<li>Then I started a fresh harvest</li>
|
|
</ul>
|
|
</li>
|
|
<li>Now I cleared all images on my local machine and I get the same error when building the backend
|
|
<ul>
|
|
<li>It seems to be related to <code>@elastic/elasticsearch-js</code>](<a href="https://github.com/elastic/elasticsearch-js)">https://github.com/elastic/elasticsearch-js)</a>, which our <code>package.json</code> pins with version <code>^7.13.0</code></li>
|
|
<li>I see that AReS is currently using 7.15.0 in its <code>package-lock.json</code>, and 7.16.0 was released four days ago so perhaps it’s that…</li>
|
|
<li>Pinning <code>~7.15.0</code> allows nest to build fine…</li>
|
|
<li>I made a pull request</li>
|
|
</ul>
|
|
</li>
|
|
<li>But since software sucks, now I get an error in the frontend while starting nginx:</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>nginx: [emerg] host not found in upstream "backend:3000" in /etc/nginx/conf.d/default.conf:2
|
|
</span></span></code></pre></div><ul>
|
|
<li>In other news, looking at updating our Redis from version 5 to 6 (which is slightly less old, but still old!) and I’m happy to see that the <a href="https://raw.githubusercontent.com/redis/redis/6.0/00-RELEASENOTES">release notes for version 6</a> say that it is compatible with 5 except for one minor thing that we don’t seem to be using (SPOP?)</li>
|
|
<li>For reference I see that our Redis 5 container is based on Debian 11, which I didn’t expect… but I still want to try to upgrade to Redis 6 eventually:</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ docker exec -it redis bash
|
|
</span></span><span style="display:flex;"><span>root@23692d6b51c5:/data# cat /etc/os-release
|
|
</span></span><span style="display:flex;"><span>PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
|
|
</span></span><span style="display:flex;"><span>NAME="Debian GNU/Linux"
|
|
</span></span><span style="display:flex;"><span>VERSION_ID="11"
|
|
</span></span><span style="display:flex;"><span>VERSION="11 (bullseye)"
|
|
</span></span><span style="display:flex;"><span>VERSION_CODENAME=bullseye
|
|
</span></span><span style="display:flex;"><span>ID=debian
|
|
</span></span><span style="display:flex;"><span>HOME_URL="https://www.debian.org/"
|
|
</span></span><span style="display:flex;"><span>SUPPORT_URL="https://www.debian.org/support"
|
|
</span></span><span style="display:flex;"><span>BUG_REPORT_URL="https://bugs.debian.org/"
|
|
</span></span></code></pre></div><ul>
|
|
<li>I bumped the version to 6 on my local test machine and the logs look good:</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ docker logs redis
|
|
</span></span><span style="display:flex;"><span>1:C 19 Dec 2021 19:27:15.583 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
|
|
</span></span><span style="display:flex;"><span>1:C 19 Dec 2021 19:27:15.583 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=1, just started
|
|
</span></span><span style="display:flex;"><span>1:C 19 Dec 2021 19:27:15.583 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
|
|
</span></span><span style="display:flex;"><span>1:M 19 Dec 2021 19:27:15.584 * monotonic clock: POSIX clock_gettime
|
|
</span></span><span style="display:flex;"><span>1:M 19 Dec 2021 19:27:15.584 * Running mode=standalone, port=6379.
|
|
</span></span><span style="display:flex;"><span>1:M 19 Dec 2021 19:27:15.584 # Server initialized
|
|
</span></span><span style="display:flex;"><span>1:M 19 Dec 2021 19:27:15.585 * Loading RDB produced by version 5.0.14
|
|
</span></span><span style="display:flex;"><span>1:M 19 Dec 2021 19:27:15.585 * RDB age 33 seconds
|
|
</span></span><span style="display:flex;"><span>1:M 19 Dec 2021 19:27:15.585 * RDB memory usage when created 3.17 Mb
|
|
</span></span><span style="display:flex;"><span>1:M 19 Dec 2021 19:27:15.595 # Done loading RDB, keys loaded: 932, keys expired: 1.
|
|
</span></span><span style="display:flex;"><span>1:M 19 Dec 2021 19:27:15.595 * DB loaded from disk: 0.011 seconds
|
|
</span></span><span style="display:flex;"><span>1:M 19 Dec 2021 19:27:15.595 * Ready to accept connections
|
|
</span></span></code></pre></div><ul>
|
|
<li>The interface and harvesting all work as expected…
|
|
<ul>
|
|
<li>I pushed the update to OpenRXV</li>
|
|
</ul>
|
|
</li>
|
|
<li>I also fixed the weird “unsafe” issue in the links on AReS that Abenet told me about last week
|
|
<ul>
|
|
<li>When testing my local instance I realized that the <code>thumbnail</code> field was missing on the production AReS, and that somehow breaks the links</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<h2 id="2021-12-22">2021-12-22</h2>
|
|
<ul>
|
|
<li>Fix apt error on DSpace servers due to updated <code>/etc/java-8-openjdk/security/java.security</code> file</li>
|
|
</ul>
|
|
<h2 id="2021-12-23">2021-12-23</h2>
|
|
<ul>
|
|
<li>Add support for dropping invalid AGROVOC subjects to csv-metadata-quality</li>
|
|
<li>Move invalid AGROVOC subjects in Gaia’s eighteen green cover items on DSpace Test to <code>cg.subject.system</code></li>
|
|
<li>I created an “approve” user for Rafael from CIAT to do tests on DSpace Test:</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ dspace user -a -m rafael-approve@cgiar.org -g Rafael -s Rodriguez -p <span style="color:#e6db74">'fuuuuuu'</span>
|
|
</span></span></code></pre></div><h2 id="2021-12-27">2021-12-27</h2>
|
|
<ul>
|
|
<li>Start a fresh harvest on AReS</li>
|
|
</ul>
|
|
<h2 id="2021-12-29">2021-12-29</h2>
|
|
<ul>
|
|
<li>Looking at the top IPs and user agents on CGSpace’s Solr statistics I see a strange user agent:</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.{random.randint(0, 9999)} Safari/537.{random.randint(0, 99)}
|
|
</span></span></code></pre></div><ul>
|
|
<li>I found two IPs using user agents with the “randint” bug:
|
|
<ul>
|
|
<li>47.252.80.214 (AliCloud in the US)</li>
|
|
<li>61.143.40.50 (ChinaNet in China)</li>
|
|
</ul>
|
|
</li>
|
|
<li>I wonder what other requests have been made from those hosts where the randint spoofer was working… ugh.</li>
|
|
<li>I found some IPs from the Russian SELECTEL network making thousands of requests with SQL injection attempts…
|
|
<ul>
|
|
<li>45.134.26.171</li>
|
|
<li>45.146.166.173</li>
|
|
</ul>
|
|
</li>
|
|
<li>3.225.28.105 is on Amazon and making thousands of requests for the same URL:</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>/rest/collections/1118/items?expand=all&limit=1
|
|
</span></span></code></pre></div><ul>
|
|
<li>Most of the time it has a real-looking user agent, but sometimes it uses <code>Apache-HttpClient/4.3.4 (java 1.5)</code></li>
|
|
<li>Another 82.65.26.228 is doing SQL injection attempts from France</li>
|
|
<li>216.213.28.138 is some scrape-as-a-service bot from Sprious</li>
|
|
<li>I used my <code>resolve-addresses-geoip2.py</code> script to get the ASNs for all the IPs in Solr stats this month, then extracted the ASNs that were responsible for more than one IP:</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ ./ilri/resolve-addresses-geoip2.py -i /tmp/ips.txt -o /tmp/2021-12-29-ips.csv
|
|
</span></span><span style="display:flex;"><span>$ csvcut -c asn /tmp/2021-12-29-ips.csv | sed 1d | sort | uniq -c | sort -h | awk <span style="color:#e6db74">'$1 > 1'</span>
|
|
</span></span><span style="display:flex;"><span> 2 10620
|
|
</span></span><span style="display:flex;"><span> 2 265696
|
|
</span></span><span style="display:flex;"><span> 2 6147
|
|
</span></span><span style="display:flex;"><span> 2 9299
|
|
</span></span><span style="display:flex;"><span> 3 3269
|
|
</span></span><span style="display:flex;"><span> 5 16509
|
|
</span></span><span style="display:flex;"><span> 5 49505
|
|
</span></span><span style="display:flex;"><span> 9 24757
|
|
</span></span><span style="display:flex;"><span> 9 24940
|
|
</span></span><span style="display:flex;"><span> 9 64267
|
|
</span></span></code></pre></div><ul>
|
|
<li>AS 64267 is Sprious, and it has used these IPs this month:
|
|
<ul>
|
|
<li>216.213.28.136</li>
|
|
<li>207.182.27.191</li>
|
|
<li>216.41.235.187</li>
|
|
<li>216.41.232.169</li>
|
|
<li>216.41.235.186</li>
|
|
<li>52.124.19.190</li>
|
|
<li>216.213.28.138</li>
|
|
<li>216.41.234.163</li>
|
|
</ul>
|
|
</li>
|
|
<li>To be honest I want to ban all their networks but I’m afraid it’s too many IPs… hmmm</li>
|
|
<li>AS 24940 is Hetzner, but I don’t feel like going through all the IPs to see… they always pretend to be normal users and make semi-sane requests so it might be a proxy or something</li>
|
|
<li>AS 24757 is Ethiopian Telecom</li>
|
|
<li>I’m going to purge all these for sure, as they are a scraping-as-a-service company and don’t use proper user agents or request robots.txt</li>
|
|
<li>AS 49505 is the Russian Selectel, and it has used these IPs this month:
|
|
<ul>
|
|
<li>45.146.166.173</li>
|
|
<li>45.134.26.171</li>
|
|
<li>45.146.164.123</li>
|
|
<li>45.155.205.231</li>
|
|
<li>195.54.167.122</li>
|
|
</ul>
|
|
</li>
|
|
<li>I will purge them all too because they are up to no good, as I already saw earlier today (SQL injections)</li>
|
|
<li>AS 16509 is Amazon, and it has used these IPs this month:
|
|
<ul>
|
|
<li>18.135.23.223 (made requests using the <code>Mozilla/5.0 (compatible; U; Koha checkurl)</code> user agent, so I will purge it and add it to our DSpace user agent override and <a href="https://github.com/atmire/COUNTER-Robots/pull/51">submit to COUNTER-Robots</a>)</li>
|
|
<li>54.76.137.83 (made hundreds of requests to “/” with a normal user agent)</li>
|
|
<li>34.253.119.85 (made hundreds of requests to “/” with a normal user agent)</li>
|
|
<li>34.216.201.131 (made hundreds of requests to “/” with a normal user agent)</li>
|
|
<li>54.203.193.46 (made hundreds of requests to “/” with a normal user agent)</li>
|
|
</ul>
|
|
</li>
|
|
<li>I ran the script to purge spider agents with the latest updates:</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ ./ilri/check-spider-hits.sh -f dspace/config/spiders/agents/ilri -p
|
|
</span></span><span style="display:flex;"><span>Purging 2530 hits from HeadlessChrome in statistics
|
|
</span></span><span style="display:flex;"><span>Purging 10676 hits from randint in statistics
|
|
</span></span><span style="display:flex;"><span>Purging 3579 hits from Koha in statistics
|
|
</span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010">
|
|
</span></span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010"></span>Total number of bot hits purged: 16785
|
|
</span></span></code></pre></div><ul>
|
|
<li>Then the IPs:</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ ./ilri/check-spider-ip-hits.sh -f /tmp/ips-to-purge.txt -p
|
|
</span></span><span style="display:flex;"><span>Purging 1190 hits from 216.213.28.136 in statistics
|
|
</span></span><span style="display:flex;"><span>Purging 1128 hits from 207.182.27.191 in statistics
|
|
</span></span><span style="display:flex;"><span>Purging 1095 hits from 216.41.235.187 in statistics
|
|
</span></span><span style="display:flex;"><span>Purging 1087 hits from 216.41.232.169 in statistics
|
|
</span></span><span style="display:flex;"><span>Purging 1011 hits from 216.41.235.186 in statistics
|
|
</span></span><span style="display:flex;"><span>Purging 945 hits from 52.124.19.190 in statistics
|
|
</span></span><span style="display:flex;"><span>Purging 933 hits from 216.213.28.138 in statistics
|
|
</span></span><span style="display:flex;"><span>Purging 930 hits from 216.41.234.163 in statistics
|
|
</span></span><span style="display:flex;"><span>Purging 4410 hits from 45.146.166.173 in statistics
|
|
</span></span><span style="display:flex;"><span>Purging 2688 hits from 45.134.26.171 in statistics
|
|
</span></span><span style="display:flex;"><span>Purging 1130 hits from 45.146.164.123 in statistics
|
|
</span></span><span style="display:flex;"><span>Purging 536 hits from 45.155.205.231 in statistics
|
|
</span></span><span style="display:flex;"><span>Purging 10676 hits from 195.54.167.122 in statistics
|
|
</span></span><span style="display:flex;"><span>Purging 1350 hits from 54.76.137.83 in statistics
|
|
</span></span><span style="display:flex;"><span>Purging 1240 hits from 34.253.119.85 in statistics
|
|
</span></span><span style="display:flex;"><span>Purging 2879 hits from 34.216.201.131 in statistics
|
|
</span></span><span style="display:flex;"><span>Purging 2909 hits from 54.203.193.46 in statistics
|
|
</span></span><span style="display:flex;"><span>Purging 1822 hits from 2605\:b100\:316\:7f74\:8d67\:5860\:a9f3\:d87c in statistics
|
|
</span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010">
|
|
</span></span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010"></span>Total number of bot hits purged: 37959
|
|
</span></span></code></pre></div><!-- raw HTML omitted -->
|
|
|
|
|
|
|
|
|
|
|
|
</article>
|
|
|
|
|
|
|
|
</div> <!-- /.blog-main -->
|
|
|
|
<aside class="col-sm-3 ml-auto blog-sidebar">
|
|
|
|
|
|
|
|
<section class="sidebar-module">
|
|
<h4>Recent Posts</h4>
|
|
<ol class="list-unstyled">
|
|
|
|
|
|
<li><a href="/cgspace-notes/2024-04/">April, 2024</a></li>
|
|
|
|
<li><a href="/cgspace-notes/2024-03/">March, 2024</a></li>
|
|
|
|
<li><a href="/cgspace-notes/2024-02/">February, 2024</a></li>
|
|
|
|
<li><a href="/cgspace-notes/2024-01/">January, 2024</a></li>
|
|
|
|
<li><a href="/cgspace-notes/2023-12/">December, 2023</a></li>
|
|
|
|
</ol>
|
|
</section>
|
|
|
|
|
|
|
|
|
|
<section class="sidebar-module">
|
|
<h4>Links</h4>
|
|
<ol class="list-unstyled">
|
|
|
|
<li><a href="https://cgspace.cgiar.org">CGSpace</a></li>
|
|
|
|
<li><a href="https://dspacetest.cgiar.org">DSpace Test</a></li>
|
|
|
|
<li><a href="https://github.com/ilri/DSpace">CGSpace @ GitHub</a></li>
|
|
|
|
</ol>
|
|
</section>
|
|
|
|
</aside>
|
|
|
|
|
|
</div> <!-- /.row -->
|
|
</div> <!-- /.container -->
|
|
|
|
|
|
|
|
<footer class="blog-footer">
|
|
<p dir="auto">
|
|
|
|
Blog template created by <a href="https://twitter.com/mdo">@mdo</a>, ported to Hugo by <a href='https://twitter.com/mralanorth'>@mralanorth</a>.
|
|
|
|
</p>
|
|
<p>
|
|
<a href="#">Back to top</a>
|
|
</p>
|
|
</footer>
|
|
|
|
|
|
</body>
|
|
|
|
</html>
|