alanorth/cgspace-notes
1
0
Fork 0
mirror of https://github.com/alanorth/cgspace-notes.git synced 2024-11-22 14:45:03 +01:00
Code Issues Projects Releases Wiki Activity

Update some inaccurate data for 2018-11

Browse Source 
My grep commands were incorrect.
This commit is contained in:
Alan Orth 2018-12-04 09:50:36 +02:00
parent ab5f69694e
commit 5f051ca9ee
Signed by: alanorth
GPG Key ID: 0FB860CC9C45B1B9
1 changed files with 29 additions and 19 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

48
content/posts/2018-11.md
View File

@ -17,7 +17,6 @@ tags: ["Notes"]
<!--more--> <!--more-->
``` ```
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "03/Nov/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10 # zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "03/Nov/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
1300 66.249.64.63 1300 66.249.64.63
@ -43,7 +42,7 @@ Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.7
- They at least seem to be re-using their Tomcat sessions: - They at least seem to be re-using their Tomcat sessions:
``` ```
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=84.38.130.177' dspace.log.2018-11-03 | sort | uniq $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=84.38.130.177' dspace.log.2018-11-03
342 342
``` ```
@ -58,7 +57,7 @@ Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:62.0) Gecko/20100101 Firefox/62
- And it doesn't seem they are re-using their Tomcat sessions: - And it doesn't seem they are re-using their Tomcat sessions:
``` ```
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=138.201.52.218' dspace.log.2018-11-03 | sort | uniq $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=138.201.52.218' dspace.log.2018-11-03
1243 1243
``` ```
@ -87,13 +86,16 @@ $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=138.201.52.218' dspace.log.2018-11
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:62.0) Gecko/20100101 Firefox/62.0 Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:62.0) Gecko/20100101 Firefox/62.0
``` ```
- It's making lots of requests and using quite a number of Tomcat sessions: - It's making lots of requests, though actually it does seem to be re-using its Tomcat sessions:
``` ```
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' /home/cgspace.cgiar.org/log/dspace.log.2018-11-03 | sort | uniq $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-03
8449 8449
$ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-03 | sort | uniq | wc -l
1
``` ```
- *Updated on 2018-12-04 to correct the grep command above, as it was inaccurate and it seems the bot was actually already re-using its Tomcat sessions*
- I could add this IP to the list of bot IPs in nginx, but it seems like a futile effort when some new IP could come along and do the same thing - I could add this IP to the list of bot IPs in nginx, but it seems like a futile effort when some new IP could come along and do the same thing
- Perhaps I should think about adding rate limits to dynamic pages like `/discover` and `/browse` - Perhaps I should think about adding rate limits to dynamic pages like `/discover` and `/browse`
- I think it's reasonable for a human to click one of those links five or ten times a minute... - I think it's reasonable for a human to click one of those links five or ten times a minute...
@ -147,13 +149,16 @@ $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' /home/cgspace.cgiar.o
20407 66.249.64.59 20407 66.249.64.59
``` ```
- `78.46.89.18` is back... and still making tons of Tomcat sessions: - `78.46.89.18` is back... and it is still actually re-using its Tomcat sessions:
``` ```
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-04 | sort | uniq $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-04
8765 8765
$ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-04 | sort | uniq | wc -l
1
``` ```
- *Updated on 2018-12-04 to correct the grep command and point out that the bot was actually re-using its Tomcat sessions properly*
- Also, now we have a ton of Facebook crawlers: - Also, now we have a ton of Facebook crawlers:
``` ```
@ -177,13 +182,14 @@ $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-04
1116 2a03:2880:11ff:f::face:b00c 1116 2a03:2880:11ff:f::face:b00c
``` ```
- They are really making shit tons of Tomcat sessions: - They are really making shit tons of requests:
``` ```
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-04 | sort | uniq $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-04
14368 37721
``` ```
- *Updated on 2018-12-04 to correct the grep command to accurately show the number of requests*
- Their user agent is: - Their user agent is:
``` ```
@ -214,15 +220,16 @@ facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
2178 2a03:2880:11ff:f::face:b00c 2178 2a03:2880:11ff:f::face:b00c
``` ```
- And still making shit tons of Tomcat sessions: - Now at least the Tomcat Crawler Session Manager Valve seems to be forcing it to re-use some Tomcat sessions:
``` ```
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-04 | sort | uniq $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-04
28470 37721
$ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-04 | sort | uniq | wc -l
15206
``` ```
- And that's even using the Tomcat Crawler Session Manager valve! - I think we still need to limit more of the dynamic pages, like the "most popular" country, item, and author pages
- Maybe we need to limit more dynamic pages, like the "most popular" country, item, and author pages
- It seems these are popular too, and there is no fucking way Facebook needs that information, yet they are requesting thousands of them! - It seems these are popular too, and there is no fucking way Facebook needs that information, yet they are requesting thousands of them!
``` ```
@ -244,18 +251,21 @@ $ ./add-dc-rights.py -i /tmp/marlo.csv -db dspace -u dspace -p 'fuuu'
- The file `marlo.csv` was cleaned up and formatted in Open Refine - The file `marlo.csv` was cleaned up and formatted in Open Refine
- 165 of the items in their 2017 data are from CGSpace! - 165 of the items in their 2017 data are from CGSpace!
- I will add the data to CGSpace this week (done!) - I will add the data to CGSpace this week (done!)
- Jesus, is Facebook *trying* to be annoying? - Jesus, is Facebook *trying* to be annoying? At least the Tomcat Crawler Session Manager Valve is working to force the bot to re-use its Tomcat sessions:
``` ```
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "05/Nov/2018" | grep -c "2a03:2880:11ff:" # zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "05/Nov/2018" | grep -c "2a03:2880:11ff:"
29889 29889
# grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-05 | sort | uniq # grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-05
29156 29763
# grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-05 | sort | uniq | wc -l
1057
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "05/Nov/2018" | grep "2a03:2880:11ff:" | grep -c -E "(handle|bitstream)" # zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "05/Nov/2018" | grep "2a03:2880:11ff:" | grep -c -E "(handle|bitstream)"
29896 29896
``` ```
- 29,000 requests from Facebook, 29,000 Tomcat sessions, and none of the requests are to the dynamic pages I rate limited yesterday! - 29,000 requests from Facebook and none of the requests are to the dynamic pages I rate limited yesterday!
- At least the Tomcat Crawler Session Manager Valve is working now...
## 2018-11-06 ## 2018-11-06