alanorth/cgspace-notes
1
0
Fork 0
mirror of https://github.com/alanorth/cgspace-notes.git synced 2025-01-27 05:49:12 +01:00
Code Issues Projects Releases Wiki Activity

Update notes for 2018-11-03

Browse Source
This commit is contained in:
Alan Orth
2018-11-04 01:02:29 +02:00
parent ee189c2ebf
commit 6bf2bdae09
11 changed files with 139 additions and 530 deletions
Download Patch File Download Diff File Expand all files Collapse all files

65
content/posts/2018-11.md
View File

@ -15,6 +15,9 @@ tags: ["Notes"]
- Linode has been sending mails a few times a day recently that CGSpace (linode18) has had high CPU usage
- Today these are the top 10 IPs:
<!--more-->
```
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "03/Nov/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
1300 66.249.64.63
@ -61,7 +64,67 @@ $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=138.201.52.218' dspace.log.2018-11
- Ah, we've apparently seen this server exactly a year ago in 2017-11, making 40,000 requests in one day...
- I wonder if it's worth adding them to the list of bots in the nginx config?
- Linode sent a mail that CGSpace (linode18) is using high outgoing bandwidth
- Looking at the nginx logs again I see the following top ten IPs:
<!--more-->
```
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "03/Nov/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
1979 50.116.102.77
1980 35.237.175.180
2186 207.46.13.156
2208 40.77.167.175
2843 66.249.64.63
4220 84.38.130.177
4537 70.32.83.92
5593 66.249.64.61
12557 78.46.89.18
32152 66.249.64.59
```
- `78.46.89.18` is new since I last checked a few hours ago, and it's from Hetzner with the following user agent:
```
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:62.0) Gecko/20100101 Firefox/62.0
```
- It's making lots of requests and using quite a number of Tomcat sessions:
```
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' /home/cgspace.cgiar.org/log/dspace.log.2018-11-03 | sort | uniq
8449
```
- I could add this IP to the list of bot IPs in nginx, but it seems like a futile effort when some new IP could come along and do the same thing
- Perhaps I should think about adding rate limits to dynamic pages like `/discover` and `/browse`
- I think it's reasonable for a human to click one of those links five or ten times a minute...
- To contrast, `78.46.89.18` made about 300 requests per minute for a few hours today:
```
# grep 78.46.89.18 /var/log/nginx/access.log | grep -o -E '03/Nov/2018:[0-9][0-9]:[0-9][0-9]' | sort | uniq -c | sort -n | tail -n 20
286 03/Nov/2018:18:02
287 03/Nov/2018:18:21
289 03/Nov/2018:18:23
291 03/Nov/2018:18:27
293 03/Nov/2018:18:34
300 03/Nov/2018:17:58
300 03/Nov/2018:18:22
300 03/Nov/2018:18:32
304 03/Nov/2018:18:12
305 03/Nov/2018:18:13
305 03/Nov/2018:18:24
312 03/Nov/2018:18:39
322 03/Nov/2018:18:17
326 03/Nov/2018:18:38
327 03/Nov/2018:18:16
330 03/Nov/2018:17:57
332 03/Nov/2018:18:19
336 03/Nov/2018:17:56
340 03/Nov/2018:18:14
341 03/Nov/2018:18:18
```
- If they want to download all our metadata and PDFs they should use an API rather than scraping the XMLUI
- I will add them to the list of bot IPs in nginx for now and think about enforcing rate limits in XMLUI later
- Also, this is the third (?) time a mysterious IP in Hetzner has done this... who is this?
<!-- vim: set sw=2 ts=2: -->