mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2024-11-22 22:55:04 +01:00
Update some inaccurate data for 2018-11
My grep commands were incorrect.
This commit is contained in:
parent
ab5f69694e
commit
5f051ca9ee
@ -17,7 +17,6 @@ tags: ["Notes"]
|
|||||||
|
|
||||||
<!--more-->
|
<!--more-->
|
||||||
|
|
||||||
|
|
||||||
```
|
```
|
||||||
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "03/Nov/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
|
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "03/Nov/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
|
||||||
1300 66.249.64.63
|
1300 66.249.64.63
|
||||||
@ -43,7 +42,7 @@ Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.7
|
|||||||
- They at least seem to be re-using their Tomcat sessions:
|
- They at least seem to be re-using their Tomcat sessions:
|
||||||
|
|
||||||
```
|
```
|
||||||
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=84.38.130.177' dspace.log.2018-11-03 | sort | uniq
|
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=84.38.130.177' dspace.log.2018-11-03
|
||||||
342
|
342
|
||||||
```
|
```
|
||||||
|
|
||||||
@ -58,7 +57,7 @@ Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:62.0) Gecko/20100101 Firefox/62
|
|||||||
- And it doesn't seem they are re-using their Tomcat sessions:
|
- And it doesn't seem they are re-using their Tomcat sessions:
|
||||||
|
|
||||||
```
|
```
|
||||||
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=138.201.52.218' dspace.log.2018-11-03 | sort | uniq
|
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=138.201.52.218' dspace.log.2018-11-03
|
||||||
1243
|
1243
|
||||||
```
|
```
|
||||||
|
|
||||||
@ -87,13 +86,16 @@ $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=138.201.52.218' dspace.log.2018-11
|
|||||||
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:62.0) Gecko/20100101 Firefox/62.0
|
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:62.0) Gecko/20100101 Firefox/62.0
|
||||||
```
|
```
|
||||||
|
|
||||||
- It's making lots of requests and using quite a number of Tomcat sessions:
|
- It's making lots of requests, though actually it does seem to be re-using its Tomcat sessions:
|
||||||
|
|
||||||
```
|
```
|
||||||
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' /home/cgspace.cgiar.org/log/dspace.log.2018-11-03 | sort | uniq
|
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-03
|
||||||
8449
|
8449
|
||||||
|
$ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-03 | sort | uniq | wc -l
|
||||||
|
1
|
||||||
```
|
```
|
||||||
|
|
||||||
|
- *Updated on 2018-12-04 to correct the grep command above, as it was inaccurate and it seems the bot was actually already re-using its Tomcat sessions*
|
||||||
- I could add this IP to the list of bot IPs in nginx, but it seems like a futile effort when some new IP could come along and do the same thing
|
- I could add this IP to the list of bot IPs in nginx, but it seems like a futile effort when some new IP could come along and do the same thing
|
||||||
- Perhaps I should think about adding rate limits to dynamic pages like `/discover` and `/browse`
|
- Perhaps I should think about adding rate limits to dynamic pages like `/discover` and `/browse`
|
||||||
- I think it's reasonable for a human to click one of those links five or ten times a minute...
|
- I think it's reasonable for a human to click one of those links five or ten times a minute...
|
||||||
@ -147,13 +149,16 @@ $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' /home/cgspace.cgiar.o
|
|||||||
20407 66.249.64.59
|
20407 66.249.64.59
|
||||||
```
|
```
|
||||||
|
|
||||||
- `78.46.89.18` is back... and still making tons of Tomcat sessions:
|
- `78.46.89.18` is back... and it is still actually re-using its Tomcat sessions:
|
||||||
|
|
||||||
```
|
```
|
||||||
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-04 | sort | uniq
|
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-04
|
||||||
8765
|
8765
|
||||||
|
$ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-04 | sort | uniq | wc -l
|
||||||
|
1
|
||||||
```
|
```
|
||||||
|
|
||||||
|
- *Updated on 2018-12-04 to correct the grep command and point out that the bot was actually re-using its Tomcat sessions properly*
|
||||||
- Also, now we have a ton of Facebook crawlers:
|
- Also, now we have a ton of Facebook crawlers:
|
||||||
|
|
||||||
```
|
```
|
||||||
@ -177,13 +182,14 @@ $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-04
|
|||||||
1116 2a03:2880:11ff:f::face:b00c
|
1116 2a03:2880:11ff:f::face:b00c
|
||||||
```
|
```
|
||||||
|
|
||||||
- They are really making shit tons of Tomcat sessions:
|
- They are really making shit tons of requests:
|
||||||
|
|
||||||
```
|
```
|
||||||
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-04 | sort | uniq
|
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-04
|
||||||
14368
|
37721
|
||||||
```
|
```
|
||||||
|
|
||||||
|
- *Updated on 2018-12-04 to correct the grep command to accurately show the number of requests*
|
||||||
- Their user agent is:
|
- Their user agent is:
|
||||||
|
|
||||||
```
|
```
|
||||||
@ -214,15 +220,16 @@ facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
|
|||||||
2178 2a03:2880:11ff:f::face:b00c
|
2178 2a03:2880:11ff:f::face:b00c
|
||||||
```
|
```
|
||||||
|
|
||||||
- And still making shit tons of Tomcat sessions:
|
- Now at least the Tomcat Crawler Session Manager Valve seems to be forcing it to re-use some Tomcat sessions:
|
||||||
|
|
||||||
```
|
```
|
||||||
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-04 | sort | uniq
|
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-04
|
||||||
28470
|
37721
|
||||||
|
$ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-04 | sort | uniq | wc -l
|
||||||
|
15206
|
||||||
```
|
```
|
||||||
|
|
||||||
- And that's even using the Tomcat Crawler Session Manager valve!
|
- I think we still need to limit more of the dynamic pages, like the "most popular" country, item, and author pages
|
||||||
- Maybe we need to limit more dynamic pages, like the "most popular" country, item, and author pages
|
|
||||||
- It seems these are popular too, and there is no fucking way Facebook needs that information, yet they are requesting thousands of them!
|
- It seems these are popular too, and there is no fucking way Facebook needs that information, yet they are requesting thousands of them!
|
||||||
|
|
||||||
```
|
```
|
||||||
@ -244,18 +251,21 @@ $ ./add-dc-rights.py -i /tmp/marlo.csv -db dspace -u dspace -p 'fuuu'
|
|||||||
- The file `marlo.csv` was cleaned up and formatted in Open Refine
|
- The file `marlo.csv` was cleaned up and formatted in Open Refine
|
||||||
- 165 of the items in their 2017 data are from CGSpace!
|
- 165 of the items in their 2017 data are from CGSpace!
|
||||||
- I will add the data to CGSpace this week (done!)
|
- I will add the data to CGSpace this week (done!)
|
||||||
- Jesus, is Facebook *trying* to be annoying?
|
- Jesus, is Facebook *trying* to be annoying? At least the Tomcat Crawler Session Manager Valve is working to force the bot to re-use its Tomcat sessions:
|
||||||
|
|
||||||
```
|
```
|
||||||
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "05/Nov/2018" | grep -c "2a03:2880:11ff:"
|
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "05/Nov/2018" | grep -c "2a03:2880:11ff:"
|
||||||
29889
|
29889
|
||||||
# grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-05 | sort | uniq
|
# grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-05
|
||||||
29156
|
29763
|
||||||
|
# grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-05 | sort | uniq | wc -l
|
||||||
|
1057
|
||||||
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "05/Nov/2018" | grep "2a03:2880:11ff:" | grep -c -E "(handle|bitstream)"
|
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "05/Nov/2018" | grep "2a03:2880:11ff:" | grep -c -E "(handle|bitstream)"
|
||||||
29896
|
29896
|
||||||
```
|
```
|
||||||
|
|
||||||
- 29,000 requests from Facebook, 29,000 Tomcat sessions, and none of the requests are to the dynamic pages I rate limited yesterday!
|
- 29,000 requests from Facebook and none of the requests are to the dynamic pages I rate limited yesterday!
|
||||||
|
- At least the Tomcat Crawler Session Manager Valve is working now...
|
||||||
|
|
||||||
## 2018-11-06
|
## 2018-11-06
|
||||||
|
|
||||||
|
Loading…
Reference in New Issue
Block a user