mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2024-12-22 13:12:19 +01:00
Add notes for 2022-04-18
This commit is contained in:
parent
9b88b678c1
commit
9d880998ad
GPG Key ID:
0FB860CC9C45B1B9
@ -59,4 +59,115 @@ $ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid =
|
||||
|
||||
- Start harvest on AReS
|
||||
|
||||
## 2022-04-18
|
||||
|
||||
- I woke up to several notices from UptimeRobot that CGSpace had gone down and up in the night (of course I'm on holiday out of the country for Easter)
|
||||
- I see there are many locks in use from the XMLUI:
|
||||
|
||||
```console
|
||||
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | grep -o -E '(dspaceWeb|dspaceApi)' | sort | uniq -c
|
||||
8932 dspaceWeb
|
||||
```
|
||||
|
||||
- Looking at the top IPs making requests it seems they are Yandex, bingbot, and Googlebot:
|
||||
|
||||
```console
|
||||
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | awk '{print $1}' | sort | uniq -c | sort -h
|
||||
752 69.162.124.231
|
||||
759 66.249.64.213
|
||||
864 66.249.66.222
|
||||
905 2a01:4f8:221:f::2
|
||||
1013 84.33.2.97
|
||||
1201 157.55.39.159
|
||||
1204 157.55.39.144
|
||||
1209 157.55.39.102
|
||||
1217 157.55.39.161
|
||||
1252 207.46.13.177
|
||||
1274 157.55.39.162
|
||||
2553 66.249.66.221
|
||||
2941 95.108.213.28
|
||||
```
|
||||
|
||||
- One IP is using a stange user agent though:
|
||||
|
||||
```console
|
||||
84.33.2.97 - - [18/Apr/2022:00:20:38 +0200] "GET /bitstream/handle/10568/109581/Banana_Blomme%20_2020.pdf.jpg HTTP/1.1" 404 10890 "-" "SomeRandomText"
|
||||
```
|
||||
|
||||
- Overall, it seems we had 17,000 unique IPs connecting in the last nine hours (currently 9:14AM and log file rolled over at 00:00):
|
||||
|
||||
```console
|
||||
# cat /var/log/nginx/access.log | awk '{print $1}' | sort | uniq | wc -l
|
||||
17314
|
||||
```
|
||||
|
||||
- That's a lot of unique IPs, and I see some patterns of IPs in China making ten to twenty requests each
|
||||
- The ISPs I've seen so far are ChinaNet and China Unicom
|
||||
- I extracted all the IPs from today and resolved them:
|
||||
|
||||
```console
|
||||
# cat /var/log/nginx/access.log | awk '{print $1}' | sort | uniq > /tmp/2022-04-18-ips.txt
|
||||
$ ./ilri/resolve-addresses-geoip2.py -i /tmp/2022-04-18-ips.txt -o /tmp/2022-04-18-ips.csv
|
||||
```
|
||||
|
||||
- The top ASNs by IP are:
|
||||
|
||||
```console
|
||||
$ csvcut -c 2 /tmp/2022-04-18-ips.csv | sed 1d | sort | uniq -c | sort -n | tail -n 10
|
||||
102 GOOGLE
|
||||
139 Maxihost LTDA
|
||||
165 AMAZON-02
|
||||
393 "China Mobile Communications Group Co., Ltd."
|
||||
473 AMAZON-AES
|
||||
616 China Mobile communications corporation
|
||||
642 M247 Ltd
|
||||
2336 HostRoyale Technologies Pvt Ltd
|
||||
4556 Chinanet
|
||||
5527 CHINA UNICOM China169 Backbone
|
||||
$ csvcut -c 4 /tmp/2022-04-18-ips.csv | sed 1d | sort | uniq -c | sort -n | tail -n 10
|
||||
139 262287
|
||||
165 16509
|
||||
180 204287
|
||||
393 9808
|
||||
473 14618
|
||||
615 56041
|
||||
642 9009
|
||||
2156 203020
|
||||
4556 4134
|
||||
5527 4837
|
||||
```
|
||||
|
||||
- I spot checked a few IPs from each of these and they are definitely just making bullshit requests to Discovery and HTML sitemap etc
|
||||
- I will download the IP blocks for each ASN except Google and Amazon and ban them
|
||||
|
||||
```console
|
||||
$ wget https://asn.ipinfo.app/api/text/nginx/AS4837 https://asn.ipinfo.app/api/text/nginx/AS4134 https://asn.ipinfo.app/api/text/nginx/AS203020 https://asn.ipinfo.app/api/text/nginx/AS9009 https://asn.ipinfo.app/api/text/nginx/AS56041 https://asn.ipinfo.app/api/text/nginx/AS9808
|
||||
$ cat AS* | sed -e '/^$/d' -e '/^#/d' -e '/^{/d' -e 's/deny //' -e 's/;//' | sort | uniq | wc -l
|
||||
20296
|
||||
```
|
||||
|
||||
- I extracted the IPv4 and IPv6 networks:
|
||||
|
||||
```console
|
||||
$ cat AS* | sed -e '/^$/d' -e '/^#/d' -e '/^{/d' -e 's/deny //' -e 's/;//' | grep ":" | sort > /tmp/ipv6-networks.txt
|
||||
$ cat AS* | sed -e '/^$/d' -e '/^#/d' -e '/^{/d' -e 's/deny //' -e 's/;//' | grep -v ":" | sort > /tmp/ipv4-networks.txt
|
||||
```
|
||||
|
||||
- I suspect we need to aggregate these networks since they are so many and nftables doesn't like it when they overlap:
|
||||
|
||||
```console
|
||||
$ wc -l /tmp/ipv4-networks.txt
|
||||
15464 /tmp/ipv4-networks.txt
|
||||
$ aggregate6 /tmp/ipv4-networks.txt | wc -l
|
||||
2781
|
||||
$ wc -l /tmp/ipv6-networks.txt
|
||||
4833 /tmp/ipv6-networks.txt
|
||||
$ aggregate6 /tmp/ipv6-networks.txt | wc -l
|
||||
338
|
||||
```
|
||||
|
||||
- I deployed these lists on CGSpace, ran all updates, and rebooted the server
|
||||
- This list is SURELY too broad because we will block legitimate users in China... but right now how can I discern?
|
||||
- Also, I need to purge the hits from these 14,000 IPs in Solr when I get time
|
||||
|
||||
<!-- vim: set sw=2 ts=2: -->
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/" />
|
||||
<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
|
||||
<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
|
||||
|
||||
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
|
||||
<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
|
||||
<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
|
||||
|
||||
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
|
||||
<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
|
||||
<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
|
||||
|
||||
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
|
||||
<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
|
||||
<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
|
||||
|
||||
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
|
||||
<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
|
||||
<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
|
||||
|
||||
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
|
||||
<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
|
||||
<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
|
||||
|
||||
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
|
||||
<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
|
||||
<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
|
||||
|
||||
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
|
||||
<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
|
||||
<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
|
||||
|
||||
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
|
||||
<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
|
||||
<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
|
||||
|
||||
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
|
||||
<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
|
||||
<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
|
||||
|
||||
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
|
||||
<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
|
||||
<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
|
||||
|
||||
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
|
||||
<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
|
||||
<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
|
||||
|
||||
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
|
||||
<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
|
||||
<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
|
||||
|
||||
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
|
||||
<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
|
||||
<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
|
||||
|
||||
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
|
||||
<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
|
||||
<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
|
||||
|
||||
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
|
||||
<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
|
||||
<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
|
||||
|
||||
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
|
||||
<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
|
||||
<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
|
||||
|
||||
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
|
||||
<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
|
||||
<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
|
||||
|
||||
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
|
||||
<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
|
||||
<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
|
||||
|
||||
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
|
||||
<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
|
||||
<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
|
||||
|
||||
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
|
||||
<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
|
||||
<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
|
||||
|
||||
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
|
||||
<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
|
||||
<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
|
||||
|
||||
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
|
||||
<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
|
||||
<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
|
||||
|
||||
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
|
||||
<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
|
||||
<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
|
||||
|
||||
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
|
||||
<meta property="og:updated_time" content="2022-04-13T16:52:34+03:00" />
|
||||
<meta property="og:updated_time" content="2022-04-16T22:41:45+03:00" />
|
||||
|
||||
|
||||
|
||||
|
||||
@ -3,22 +3,22 @@
|
||||
xmlns:xhtml="http://www.w3.org/1999/xhtml">
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/categories/</loc>
|
||||
<lastmod>2022-04-13T16:52:34+03:00</lastmod>
|
||||
<lastmod>2022-04-16T22:41:45+03:00</lastmod>
|
||||
</url><url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/</loc>
|
||||
<lastmod>2022-04-13T16:52:34+03:00</lastmod>
|
||||
<lastmod>2022-04-16T22:41:45+03:00</lastmod>
|
||||
</url><url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/2022-03/</loc>
|
||||
<lastmod>2022-04-04T19:15:58+03:00</lastmod>
|
||||
</url><url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
|
||||
<lastmod>2022-04-13T16:52:34+03:00</lastmod>
|
||||
<lastmod>2022-04-16T22:41:45+03:00</lastmod>
|
||||
</url><url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
|
||||
<lastmod>2022-04-13T16:52:34+03:00</lastmod>
|
||||
<lastmod>2022-04-16T22:41:45+03:00</lastmod>
|
||||
</url><url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/2022-03/</loc>
|
||||
<lastmod>2022-04-13T16:52:34+03:00</lastmod>
|
||||
<lastmod>2022-04-16T22:41:45+03:00</lastmod>
|
||||
</url><url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/2022-02/</loc>
|
||||
<lastmod>2022-03-01T17:17:27+03:00</lastmod>
|
||||
|
||||
Loading…
Reference in New Issue
Block a user