From 5f051ca9eee0d8ee6a73c64f5a06a66c68b578b3 Mon Sep 17 00:00:00 2001 From: Alan Orth Date: Tue, 4 Dec 2018 09:50:36 +0200 Subject: [PATCH] Update some inaccurate data for 2018-11 My grep commands were incorrect. --- content/posts/2018-11.md | 48 ++++++++++++++++++++++++---------------- 1 file changed, 29 insertions(+), 19 deletions(-) diff --git a/content/posts/2018-11.md b/content/posts/2018-11.md index f20815281..20ae0842a 100644 --- a/content/posts/2018-11.md +++ b/content/posts/2018-11.md @@ -17,7 +17,6 @@ tags: ["Notes"] - ``` # zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "03/Nov/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10 1300 66.249.64.63 @@ -43,7 +42,7 @@ Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.7 - They at least seem to be re-using their Tomcat sessions: ``` -$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=84.38.130.177' dspace.log.2018-11-03 | sort | uniq +$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=84.38.130.177' dspace.log.2018-11-03 342 ``` @@ -58,7 +57,7 @@ Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:62.0) Gecko/20100101 Firefox/62 - And it doesn't seem they are re-using their Tomcat sessions: ``` -$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=138.201.52.218' dspace.log.2018-11-03 | sort | uniq +$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=138.201.52.218' dspace.log.2018-11-03 1243 ``` @@ -87,13 +86,16 @@ $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=138.201.52.218' dspace.log.2018-11 Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:62.0) Gecko/20100101 Firefox/62.0 ``` -- It's making lots of requests and using quite a number of Tomcat sessions: +- It's making lots of requests, though actually it does seem to be re-using its Tomcat sessions: ``` -$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' /home/cgspace.cgiar.org/log/dspace.log.2018-11-03 | sort | uniq +$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-03 8449 +$ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-03 | sort | uniq | wc -l +1 ``` +- *Updated on 2018-12-04 to correct the grep command above, as it was inaccurate and it seems the bot was actually already re-using its Tomcat sessions* - I could add this IP to the list of bot IPs in nginx, but it seems like a futile effort when some new IP could come along and do the same thing - Perhaps I should think about adding rate limits to dynamic pages like `/discover` and `/browse` - I think it's reasonable for a human to click one of those links five or ten times a minute... @@ -147,13 +149,16 @@ $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' /home/cgspace.cgiar.o 20407 66.249.64.59 ``` -- `78.46.89.18` is back... and still making tons of Tomcat sessions: +- `78.46.89.18` is back... and it is still actually re-using its Tomcat sessions: ``` -$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-04 | sort | uniq +$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-04 8765 +$ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-04 | sort | uniq | wc -l +1 ``` +- *Updated on 2018-12-04 to correct the grep command and point out that the bot was actually re-using its Tomcat sessions properly* - Also, now we have a ton of Facebook crawlers: ``` @@ -177,13 +182,14 @@ $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-04 1116 2a03:2880:11ff:f::face:b00c ``` -- They are really making shit tons of Tomcat sessions: +- They are really making shit tons of requests: ``` -$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-04 | sort | uniq -14368 +$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-04 +37721 ``` +- *Updated on 2018-12-04 to correct the grep command to accurately show the number of requests* - Their user agent is: ``` @@ -214,15 +220,16 @@ facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) 2178 2a03:2880:11ff:f::face:b00c ``` -- And still making shit tons of Tomcat sessions: +- Now at least the Tomcat Crawler Session Manager Valve seems to be forcing it to re-use some Tomcat sessions: ``` -$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-04 | sort | uniq -28470 +$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-04 +37721 +$ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-04 | sort | uniq | wc -l +15206 ``` -- And that's even using the Tomcat Crawler Session Manager valve! -- Maybe we need to limit more dynamic pages, like the "most popular" country, item, and author pages +- I think we still need to limit more of the dynamic pages, like the "most popular" country, item, and author pages - It seems these are popular too, and there is no fucking way Facebook needs that information, yet they are requesting thousands of them! ``` @@ -244,18 +251,21 @@ $ ./add-dc-rights.py -i /tmp/marlo.csv -db dspace -u dspace -p 'fuuu' - The file `marlo.csv` was cleaned up and formatted in Open Refine - 165 of the items in their 2017 data are from CGSpace! - I will add the data to CGSpace this week (done!) -- Jesus, is Facebook *trying* to be annoying? +- Jesus, is Facebook *trying* to be annoying? At least the Tomcat Crawler Session Manager Valve is working to force the bot to re-use its Tomcat sessions: ``` # zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "05/Nov/2018" | grep -c "2a03:2880:11ff:" 29889 -# grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-05 | sort | uniq -29156 +# grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-05 +29763 +# grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-05 | sort | uniq | wc -l +1057 # zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "05/Nov/2018" | grep "2a03:2880:11ff:" | grep -c -E "(handle|bitstream)" 29896 ``` -- 29,000 requests from Facebook, 29,000 Tomcat sessions, and none of the requests are to the dynamic pages I rate limited yesterday! +- 29,000 requests from Facebook and none of the requests are to the dynamic pages I rate limited yesterday! +- At least the Tomcat Crawler Session Manager Valve is working now... ## 2018-11-06