Add notes for 2021-10-24

content/posts/2021-10.md

@@ -479,4 +479,47 @@ Time: 4417.909 ms (00:04.418)
   - We discussed several options, for example moving all DSpaces to CGSpace along with their permanent identifiers
   - The issue would be for centers like IFPRI who don't use DSpace and have integrations with their website etc with their current repository
 
+## 2021-10-21
+
+- Udana from IWMI contacted me to ask if I could do a one-off AReS harvest because they have some new items they need to report on
+
+## 2021-10-22
+
+- Abenet and others contacted me to say that the LDAP login was not working on CGSpace
+  - I checked with `ldapsearch` and it is indeed not working:
+
+```console
+$ ldapsearch -x -H ldaps://AZCGNEROOT3.CGIARAD.ORG:636/ -b "dc=cgiarad,dc=org" -D "booo" -W "(sAMAccountName=fuuu)"
+Enter LDAP Password:
+ldap_bind: Invalid credentials (49)
+        additional info: 80090308: LdapErr: DSID-0C090447, comment: AcceptSecurityContext error, data 52e, v3839
+```
+
+- I sent a message to ILRI ICT to ask them to check the account
+  - They reset the password so I ran all system updates and rebooted the server since users weren't able to log in anyways
+
+## 2021-10-24
+
+- CIP was asking about CGSpace stats again
+  - The last time I helped them with this was in 2021-04, when I extracted stats for their community from the DSpace Statistics API
+- In looking at the CIP stats request I got curious if there were any hits from all those Russian IPs before 2021-07 that I could purge
+  - Sure enough there were a few hundred IPs belonging to those ASNs:
+
+```console
+$ http 'localhost:8081/solr/statistics/select?q=time%3A2021-04*&fl=ip&wt=json&indent=true&facet=true&facet.field=ip&facet.limit=200000&facet.mincount=1' > /tmp/2021-04-ips.json
+# Ghetto way to extract the IPs using jq, but I can't figure out how only print them and not the facet counts, so I just use sed
+$ jq '.facet_counts.facet_fields.ip[]' /tmp/2021-04-ips.json | grep -E '^"' | sed -e 's/"//g' > /tmp/ips.txt
+$ ./ilri/resolve-addresses-geoip2.py -i /tmp/ips.txt -o /tmp/2021-04-ips.csv
+$ csvgrep -c asn -r '^(49453|46844|206485|62282|36352|35913|35624|8100)$' /tmp/2021-04-ips.csv | csvcut -c network | sed 1d | sort -u > /tmp/networks-to-block.txt
+$ wc -l /tmp/networks-to-block.txt 
+125 /tmp/networks-to-block.txt
+$ grepcidr -f /tmp/networks-to-block.txt /tmp/ips.txt > /tmp/ips-to-purge.txt
+$ wc -l /tmp/ips-to-purge.txt
+202
+```
+
+- Attempting to purge those only shows about 3,500 hits, but I will do it anyways
+  - Adding 64.39.108.48 from Qualys I get a total of 22631 hits purged
+- I also purged another 5306 hits after checking the IPv4 list from AbuseIPDB.com
+
 <!-- vim: set sw=2 ts=2: -->

docs/2021-10/index.html

@@ -25,7 +25,7 @@ So we have 1879/7100 (26.46%) matching already
 <meta property="og:type" content="article" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2021-10/" />
 <meta property="article:published_time" content="2021-10-01T11:14:07+03:00" />
-<meta property="article:modified_time" content="2021-10-17T20:47:01+03:00" />
+<meta property="article:modified_time" content="2021-10-20T22:21:55+03:00" />
 
 
 
@@ -56,9 +56,9 @@ So we have 1879/7100 (26.46%) matching already
   "@type": "BlogPosting",
   "headline": "October, 2021",
   "url": "https://alanorth.github.io/cgspace-notes/2021-10/",
-  "wordCount": "3392",
+  "wordCount": "3710",
   "datePublished": "2021-10-01T11:14:07+03:00",
-  "dateModified": "2021-10-17T20:47:01+03:00",
+  "dateModified": "2021-10-20T22:21:55+03:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"
@@ -628,6 +628,60 @@ Time: 4417.909 ms (00:04.418)
 </ul>
 </li>
 </ul>
+<h2 id="2021-10-21">2021-10-21</h2>
+<ul>
+<li>Udana from IWMI contacted me to ask if I could do a one-off AReS harvest because they have some new items they need to report on</li>
+</ul>
+<h2 id="2021-10-22">2021-10-22</h2>
+<ul>
+<li>Abenet and others contacted me to say that the LDAP login was not working on CGSpace
+<ul>
+<li>I checked with <code>ldapsearch</code> and it is indeed not working:</li>
+</ul>
+</li>
+</ul>
+<pre tabindex="0"><code class="language-console" data-lang="console">$ ldapsearch -x -H ldaps://AZCGNEROOT3.CGIARAD.ORG:636/ -b &quot;dc=cgiarad,dc=org&quot; -D &quot;booo&quot; -W &quot;(sAMAccountName=fuuu)&quot;
+Enter LDAP Password:
+ldap_bind: Invalid credentials (49)
+        additional info: 80090308: LdapErr: DSID-0C090447, comment: AcceptSecurityContext error, data 52e, v3839
+</code></pre><ul>
+<li>I sent a message to ILRI ICT to ask them to check the account
+<ul>
+<li>They reset the password so I ran all system updates and rebooted the server since users weren&rsquo;t able to log in anyways</li>
+</ul>
+</li>
+</ul>
+<h2 id="2021-10-24">2021-10-24</h2>
+<ul>
+<li>CIP was asking about CGSpace stats again
+<ul>
+<li>The last time I helped them with this was in 2021-04, when I extracted stats for their community from the DSpace Statistics API</li>
+</ul>
+</li>
+<li>In looking at the CIP stats request I got curious if there were any hits from all those Russian IPs before 2021-07 that I could purge
+<ul>
+<li>Sure enough there were a few hundred IPs belonging to those ASNs:</li>
+</ul>
+</li>
+</ul>
+<pre tabindex="0"><code class="language-console" data-lang="console">$ http 'localhost:8081/solr/statistics/select?q=time%3A2021-04*&amp;fl=ip&amp;wt=json&amp;indent=true&amp;facet=true&amp;facet.field=ip&amp;facet.limit=200000&amp;facet.mincount=1' &gt; /tmp/2021-04-ips.json
+# Ghetto way to extract the IPs using jq, but I can't figure out how only print them and not the facet counts, so I just use sed
+$ jq '.facet_counts.facet_fields.ip[]' /tmp/2021-04-ips.json | grep -E '^&quot;' | sed -e 's/&quot;//g' &gt; /tmp/ips.txt
+$ ./ilri/resolve-addresses-geoip2.py -i /tmp/ips.txt -o /tmp/2021-04-ips.csv
+$ csvgrep -c asn -r '^(49453|46844|206485|62282|36352|35913|35624|8100)$' /tmp/2021-04-ips.csv | csvcut -c network | sed 1d | sort -u &gt; /tmp/networks-to-block.txt
+$ wc -l /tmp/networks-to-block.txt 
+125 /tmp/networks-to-block.txt
+$ grepcidr -f /tmp/networks-to-block.txt /tmp/ips.txt &gt; /tmp/ips-to-purge.txt
+$ wc -l /tmp/ips-to-purge.txt
+202
+</code></pre><ul>
+<li>Attempting to purge those only shows about 3,500 hits, but I will do it anyways
+<ul>
+<li>Adding 64.39.108.48 from Qualys I get a total of 22631 hits purged</li>
+</ul>
+</li>
+<li>I also purged another 5306 hits after checking the IPv4 list from AbuseIPDB.com</li>
+</ul>
 <!-- raw HTML omitted -->
 
   

docs/categories/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/" />
-<meta property="og:updated_time" content="2021-10-17T20:47:01+03:00" />
+<meta property="og:updated_time" content="2021-10-20T22:21:55+03:00" />
 
 
 

docs/categories/notes/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-10-17T20:47:01+03:00" />
+<meta property="og:updated_time" content="2021-10-20T22:21:55+03:00" />
 
 
 

docs/categories/notes/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-10-17T20:47:01+03:00" />
+<meta property="og:updated_time" content="2021-10-20T22:21:55+03:00" />
 
 
 

docs/categories/notes/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-10-17T20:47:01+03:00" />
+<meta property="og:updated_time" content="2021-10-20T22:21:55+03:00" />
 
 
 

docs/categories/notes/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-10-17T20:47:01+03:00" />
+<meta property="og:updated_time" content="2021-10-20T22:21:55+03:00" />
 
 
 

docs/categories/notes/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-10-17T20:47:01+03:00" />
+<meta property="og:updated_time" content="2021-10-20T22:21:55+03:00" />
 
 
 

docs/categories/notes/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-10-17T20:47:01+03:00" />
+<meta property="og:updated_time" content="2021-10-20T22:21:55+03:00" />
 
 
 

docs/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-10-17T20:47:01+03:00" />
+<meta property="og:updated_time" content="2021-10-20T22:21:55+03:00" />
 
 
 

docs/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-10-17T20:47:01+03:00" />
+<meta property="og:updated_time" content="2021-10-20T22:21:55+03:00" />
 
 
 

docs/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-10-17T20:47:01+03:00" />
+<meta property="og:updated_time" content="2021-10-20T22:21:55+03:00" />
 
 
 

docs/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-10-17T20:47:01+03:00" />
+<meta property="og:updated_time" content="2021-10-20T22:21:55+03:00" />
 
 
 

docs/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-10-17T20:47:01+03:00" />
+<meta property="og:updated_time" content="2021-10-20T22:21:55+03:00" />
 
 
 

docs/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-10-17T20:47:01+03:00" />
+<meta property="og:updated_time" content="2021-10-20T22:21:55+03:00" />
 
 
 

docs/page/7/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-10-17T20:47:01+03:00" />
+<meta property="og:updated_time" content="2021-10-20T22:21:55+03:00" />
 
 
 

docs/page/8/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-10-17T20:47:01+03:00" />
+<meta property="og:updated_time" content="2021-10-20T22:21:55+03:00" />
 
 
 

docs/posts/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-10-17T20:47:01+03:00" />
+<meta property="og:updated_time" content="2021-10-20T22:21:55+03:00" />
 
 
 

docs/posts/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-10-17T20:47:01+03:00" />
+<meta property="og:updated_time" content="2021-10-20T22:21:55+03:00" />
 
 
 

docs/posts/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-10-17T20:47:01+03:00" />
+<meta property="og:updated_time" content="2021-10-20T22:21:55+03:00" />
 
 
 

docs/posts/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-10-17T20:47:01+03:00" />
+<meta property="og:updated_time" content="2021-10-20T22:21:55+03:00" />
 
 
 

docs/posts/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-10-17T20:47:01+03:00" />
+<meta property="og:updated_time" content="2021-10-20T22:21:55+03:00" />
 
 
 

docs/posts/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-10-17T20:47:01+03:00" />
+<meta property="og:updated_time" content="2021-10-20T22:21:55+03:00" />
 
 
 

docs/posts/page/7/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-10-17T20:47:01+03:00" />
+<meta property="og:updated_time" content="2021-10-20T22:21:55+03:00" />
 
 
 

docs/posts/page/8/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-10-17T20:47:01+03:00" />
+<meta property="og:updated_time" content="2021-10-20T22:21:55+03:00" />
 
 
 

docs/sitemap.xml

@@ -3,19 +3,19 @@
   xmlns:xhtml="http://www.w3.org/1999/xhtml">
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/</loc>
-    <lastmod>2021-10-17T20:47:01+03:00</lastmod>
+    <lastmod>2021-10-20T22:21:55+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2021-10-17T20:47:01+03:00</lastmod>
+    <lastmod>2021-10-20T22:21:55+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
-    <lastmod>2021-10-17T20:47:01+03:00</lastmod>
+    <lastmod>2021-10-20T22:21:55+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/2021-10/</loc>
-    <lastmod>2021-10-17T20:47:01+03:00</lastmod>
+    <lastmod>2021-10-20T22:21:55+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
-    <lastmod>2021-10-17T20:47:01+03:00</lastmod>
+    <lastmod>2021-10-20T22:21:55+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/2021-09/</loc>
     <lastmod>2021-10-04T11:10:54+03:00</lastmod>