Add notes for 2022-04-18

content/posts/2022-04.md

@@ -59,4 +59,115 @@ $ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid =
 
 - Start harvest on AReS
 
+## 2022-04-18
+
+- I woke up to several notices from UptimeRobot that CGSpace had gone down and up in the night (of course I'm on holiday out of the country for Easter)
+  - I see there are many locks in use from the XMLUI:
+
+```console
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | grep -o -E '(dspaceWeb|dspaceApi)' | sort | uniq -c
+   8932 dspaceWeb
+```
+
+- Looking at the top IPs making requests it seems they are Yandex, bingbot, and Googlebot:
+
+```console
+# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | awk '{print $1}' | sort | uniq -c | sort -h
+    752 69.162.124.231
+    759 66.249.64.213
+    864 66.249.66.222
+    905 2a01:4f8:221:f::2
+   1013 84.33.2.97
+   1201 157.55.39.159
+   1204 157.55.39.144
+   1209 157.55.39.102
+   1217 157.55.39.161
+   1252 207.46.13.177
+   1274 157.55.39.162
+   2553 66.249.66.221
+   2941 95.108.213.28
+```
+
+- One IP is using a stange user agent though:
+
+```console
+84.33.2.97 - - [18/Apr/2022:00:20:38 +0200] "GET /bitstream/handle/10568/109581/Banana_Blomme%20_2020.pdf.jpg HTTP/1.1" 404 10890 "-" "SomeRandomText"
+```
+
+- Overall, it seems we had 17,000 unique IPs connecting in the last nine hours (currently 9:14AM and log file rolled over at 00:00):
+
+```console
+# cat /var/log/nginx/access.log | awk '{print $1}' | sort | uniq | wc -l
+17314
+```
+
+- That's a lot of unique IPs, and I see some patterns of IPs in China making ten to twenty requests each
+  - The ISPs I've seen so far are ChinaNet and China Unicom
+- I extracted all the IPs from today and resolved them:
+
+```console
+# cat /var/log/nginx/access.log | awk '{print $1}' | sort | uniq > /tmp/2022-04-18-ips.txt
+$ ./ilri/resolve-addresses-geoip2.py -i /tmp/2022-04-18-ips.txt -o /tmp/2022-04-18-ips.csv
+```
+
+- The top ASNs by IP are:
+
+```console
+$ csvcut -c 2 /tmp/2022-04-18-ips.csv | sed 1d | sort | uniq -c | sort -n | tail -n 10 
+    102 GOOGLE
+    139 Maxihost LTDA
+    165 AMAZON-02
+    393 "China Mobile Communications Group Co., Ltd."
+    473 AMAZON-AES
+    616 China Mobile communications corporation
+    642 M247 Ltd
+   2336 HostRoyale Technologies Pvt Ltd
+   4556 Chinanet
+   5527 CHINA UNICOM China169 Backbone
+$ csvcut -c 4 /tmp/2022-04-18-ips.csv | sed 1d | sort | uniq -c | sort -n | tail -n 10
+    139 262287
+    165 16509
+    180 204287
+    393 9808
+    473 14618
+    615 56041
+    642 9009
+   2156 203020
+   4556 4134
+   5527 4837
+```
+
+- I spot checked a few IPs from each of these and they are definitely just making bullshit requests to Discovery and HTML sitemap etc
+- I will download the IP blocks for each ASN except Google and Amazon and ban them
+
+```console
+$ wget https://asn.ipinfo.app/api/text/nginx/AS4837 https://asn.ipinfo.app/api/text/nginx/AS4134 https://asn.ipinfo.app/api/text/nginx/AS203020 https://asn.ipinfo.app/api/text/nginx/AS9009 https://asn.ipinfo.app/api/text/nginx/AS56041 https://asn.ipinfo.app/api/text/nginx/AS9808
+$ cat AS* | sed -e '/^$/d' -e '/^#/d' -e '/^{/d' -e 's/deny //' -e 's/;//' | sort | uniq | wc -l
+20296
+```
+
+- I extracted the IPv4 and IPv6 networks:
+
+```console
+$ cat AS* | sed -e '/^$/d' -e '/^#/d' -e '/^{/d' -e 's/deny //' -e 's/;//' | grep ":" | sort > /tmp/ipv6-networks.txt
+$ cat AS* | sed -e '/^$/d' -e '/^#/d' -e '/^{/d' -e 's/deny //' -e 's/;//' | grep -v ":" | sort > /tmp/ipv4-networks.txt
+```
+
+- I suspect we need to aggregate these networks since they are so many and nftables doesn't like it when they overlap:
+
+```console
+$ wc -l /tmp/ipv4-networks.txt
+15464 /tmp/ipv4-networks.txt
+$ aggregate6 /tmp/ipv4-networks.txt | wc -l
+2781
+$ wc -l /tmp/ipv6-networks.txt             
+4833 /tmp/ipv6-networks.txt
+$ aggregate6 /tmp/ipv6-networks.txt | wc -l
+338
+```
+
+- I deployed these lists on CGSpace, ran all updates, and rebooted the server
+  - This list is SURELY too broad because we will block legitimate users in China... but right now how can I discern?
+  - Also, I need to purge the hits from these 14,000 IPs in Solr when I get time
+
 <!-- vim: set sw=2 ts=2: -->

docs/categories/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/" />
-<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
+<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
 
 
 

docs/categories/notes/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
+<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
 
 
 

docs/categories/notes/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
+<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
 
 
 

docs/categories/notes/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
+<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
 
 
 

docs/categories/notes/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
+<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
 
 
 

docs/categories/notes/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
+<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
 
 
 

docs/categories/notes/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
+<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
 
 
 

docs/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
+<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
 
 
 

docs/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
+<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
 
 
 

docs/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
+<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
 
 
 

docs/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
+<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
 
 
 

docs/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
+<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
 
 
 

docs/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
+<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
 
 
 

docs/page/7/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
+<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
 
 
 

docs/page/8/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
+<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
 
 
 

docs/page/9/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
+<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
 
 
 

docs/posts/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
+<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
 
 
 

docs/posts/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
+<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
 
 
 

docs/posts/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
+<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
 
 
 

docs/posts/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
+<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
 
 
 

docs/posts/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
+<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
 
 
 

docs/posts/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
+<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
 
 
 

docs/posts/page/7/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
+<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
 
 
 

docs/posts/page/8/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
+<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
 
 
 

docs/posts/page/9/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
+<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
 
 
 

docs/sitemap.xml

@@ -3,22 +3,22 @@
   xmlns:xhtml="http://www.w3.org/1999/xhtml">
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/</loc>
-    <lastmod>2022-04-13T16:52:34+03:00</lastmod>
+    <lastmod>2022-04-16T22:41:45+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2022-04-13T16:52:34+03:00</lastmod>
+    <lastmod>2022-04-16T22:41:45+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/2022-03/</loc>
     <lastmod>2022-04-04T19:15:58+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
-    <lastmod>2022-04-13T16:52:34+03:00</lastmod>
+    <lastmod>2022-04-16T22:41:45+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
-    <lastmod>2022-04-13T16:52:34+03:00</lastmod>
+    <lastmod>2022-04-16T22:41:45+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/2022-03/</loc>
-    <lastmod>2022-04-13T16:52:34+03:00</lastmod>
+    <lastmod>2022-04-16T22:41:45+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/2022-02/</loc>
     <lastmod>2022-03-01T17:17:27+03:00</lastmod>