alanorth/cgspace-notes
1
0
Fork 0
mirror of https://github.com/alanorth/cgspace-notes.git synced 2025-01-27 05:49:12 +01:00
Code Issues Projects Releases Wiki Activity

Add notes for 2022-04-27

Browse Source
This commit is contained in:
Alan Orth
2022-04-27 09:58:45 +03:00
parent 4ddc675e55
commit 5cfe954a53
123 changed files with 1461 additions and 655 deletions
Download Patch File Download Diff File Expand all files Collapse all files

145
content/posts/2022-04.md
View File

@ -228,4 +228,149 @@ $ ./ilri/fix-metadata-values.py -i /tmp/regions.csv -db dspace -u dspace -p 'fuu
- Then I started a new harvest on AReS
## 2022-04-27
- I woke up to many up down notices for CGSpace from UptimeRobot
- The server has load 111.0... sigh.
- According to Grafana it seems to have started at 4:00 AM
![Grafana load](/cgspace-notes/2022/04/cgspace-load.png)
- There are a metric fuck ton of database locks from the XMLUI:
```console
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | grep -o -E '(dspaceWeb|dspaceApi)' | sort | uniq -c
128 dspaceApi
16890 dspaceWeb
```
- As for the server logs, I don't see many IPs connecting today:
```console
# cat /var/log/nginx/access.log | awk '{print $1}' | sort | uniq | wc -l
2924
```
- But there appear to be some IPs making many requests:
```console
# cat /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -h
...
345 207.46.13.53
646 66.249.66.222
678 54.90.79.112
1529 136.243.148.249
1797 54.175.8.110
2304 174.129.118.171
2523 66.249.66.221
2632 52.73.204.196
2667 54.174.240.122
5206 35.172.193.232
5646 35.153.131.101
6373 3.85.92.145
7383 34.227.10.4
8330 100.24.63.172
8342 34.236.36.176
8369 44.200.190.111
8371 3.238.116.153
8391 18.232.101.158
8631 3.239.81.247
8634 54.82.125.225
```
- 54.82.125.225, 3.239.81.247, 18.232.101.158, 3.238.116.153, 44.200.190.111, 34.236.36.176, 100.24.63.172, 3.85.92.145, 35.153.131.101, 35.172.193.232, 54.174.240.122, 52.73.204.196, 174.129.118.171, 54.175.8.110, and 54.90.79.112 are all on Amazon and using this normal-looking user agent:
```
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.125 Safari/537.3
```
- None of these hosts are re-using their DSpace session ID so they are definitely not normal browsers as they are claiming:
```console
$ grep 54.82.125.225 dspace.log.2022-04-27 | grep -oE 'session_id=[A-Z0-9]{32}:ip_addr=' | sort | uniq | wc -l
5760
$ grep 3.239.81.247 dspace.log.2022-04-27 | grep -oE 'session_id=[A-Z0-9]{32}:ip_addr=' | sort | uniq | wc -l
6053
$ grep 18.232.101.158 dspace.log.2022-04-27 | grep -oE 'session_id=[A-Z0-9]{32}:ip_addr=' | sort | uniq | wc -l
5841
$ grep 3.238.116.153 dspace.log.2022-04-27 | grep -oE 'session_id=[A-Z0-9]{32}:ip_addr=' | sort | uniq | wc -l
5887
$ grep 44.200.190.111 dspace.log.2022-04-27 | grep -oE 'session_id=[A-Z0-9]{32}:ip_addr=' | sort | uniq | wc -l
5899
...
```
- And we can see a massive spike in sessions in Munin:
![Grafana load](/cgspace-notes/2022/04/jmx_dspace_sessions-day2.png)
- I see the following IPs using that user agent today:
```console
# grep 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.125 Safari/537.36' /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -h
678 54.90.79.112
1797 54.175.8.110
2697 174.129.118.171
2765 52.73.204.196
3072 54.174.240.122
5206 35.172.193.232
5646 35.153.131.101
6783 3.85.92.145
7763 34.227.10.4
8738 100.24.63.172
8748 34.236.36.176
8787 3.238.116.153
8794 18.232.101.158
8806 44.200.190.111
9021 54.82.125.225
9027 3.239.81.247
```
- I added those IPs to the firewall and then purged their hits from Solr:
```console
$ ./ilri/check-spider-ip-hits.sh -f /tmp/ips.txt -p
Purging 6024 hits from 100.24.63.172 in statistics
Purging 1719 hits from 174.129.118.171 in statistics
Purging 5972 hits from 18.232.101.158 in statistics
Purging 6053 hits from 3.238.116.153 in statistics
Purging 6228 hits from 3.239.81.247 in statistics
Purging 5305 hits from 34.227.10.4 in statistics
Purging 6002 hits from 34.236.36.176 in statistics
Purging 3908 hits from 35.153.131.101 in statistics
Purging 3692 hits from 35.172.193.232 in statistics
Purging 4525 hits from 3.85.92.145 in statistics
Purging 6048 hits from 44.200.190.111 in statistics
Purging 1942 hits from 52.73.204.196 in statistics
Purging 1944 hits from 54.174.240.122 in statistics
Purging 1264 hits from 54.175.8.110 in statistics
Purging 6117 hits from 54.82.125.225 in statistics
Purging 486 hits from 54.90.79.112 in statistics
Total number of bot hits purged: 67229
```
- Then I created a CSV with these IPs and reported them to AbuseIPDB.com:
```console
$ cat /tmp/ips.csv
IP,Categories,ReportDate,Comment
100.24.63.172,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
174.129.118.171,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
18.232.101.158,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
3.238.116.153,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
3.239.81.247,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
34.227.10.4,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
34.236.36.176,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
35.153.131.101,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
35.172.193.232,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
3.85.92.145,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
44.200.190.111,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
52.73.204.196,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
54.174.240.122,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
54.175.8.110,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
54.82.125.225,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
54.90.79.112,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
```
<!-- vim: set sw=2 ts=2: -->