Update some inaccurate data for 2018-11

My grep commands were incorrect.

content/posts/2018-11.md

@@ -17,7 +17,6 @@ tags: ["Notes"]
 
 <!--more-->
 
-
 ```
 # zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "03/Nov/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
    1300 66.249.64.63
@@ -43,7 +42,7 @@ Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.7
 - They at least seem to be re-using their Tomcat sessions:
 
 ```
-$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=84.38.130.177' dspace.log.2018-11-03 | sort | uniq
+$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=84.38.130.177' dspace.log.2018-11-03
 342
 ```
 
@@ -58,7 +57,7 @@ Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:62.0) Gecko/20100101 Firefox/62
 - And it doesn't seem they are re-using their Tomcat sessions:
 
 ```
-$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=138.201.52.218' dspace.log.2018-11-03 | sort | uniq
+$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=138.201.52.218' dspace.log.2018-11-03
 1243
 ```
 
@@ -87,13 +86,16 @@ $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=138.201.52.218' dspace.log.2018-11
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:62.0) Gecko/20100101 Firefox/62.0
 ```
 
-- It's making lots of requests and using quite a number of Tomcat sessions:
+- It's making lots of requests, though actually it does seem to be re-using its Tomcat sessions:
 
 ```
-$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' /home/cgspace.cgiar.org/log/dspace.log.2018-11-03 | sort | uniq
+$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-03
 8449
+$ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-03 | sort | uniq | wc -l
+1
 ```
 
+- *Updated on 2018-12-04 to correct the grep command above, as it was inaccurate and it seems the bot was actually already re-using its Tomcat sessions*
 - I could add this IP to the list of bot IPs in nginx, but it seems like a futile effort when some new IP could come along and do the same thing
 - Perhaps I should think about adding rate limits to dynamic pages like `/discover` and `/browse`
 - I think it's reasonable for a human to click one of those links five or ten times a minute...
@@ -147,13 +149,16 @@ $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' /home/cgspace.cgiar.o
   20407 66.249.64.59
 ```
 
-- `78.46.89.18` is back... and still making tons of Tomcat sessions:
+- `78.46.89.18` is back... and it is still actually re-using its Tomcat sessions:
 
 ```
-$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-04 | sort | uniq
+$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-04
 8765
+$ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-04 | sort | uniq | wc -l
+1
 ```
 
+- *Updated on 2018-12-04 to correct the grep command and point out that the bot was actually re-using its Tomcat sessions properly*
 - Also, now we have a ton of Facebook crawlers:
 
 ```
@@ -177,13 +182,14 @@ $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-04
    1116 2a03:2880:11ff:f::face:b00c
 ```
 
-- They are really making shit tons of Tomcat sessions:
+- They are really making shit tons of requests:
 
 ```
-$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-04 | sort | uniq
-14368
+$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-04
+37721
 ```
 
+- *Updated on 2018-12-04 to correct the grep command to accurately show the number of requests*
 - Their user agent is:
 
 ```
@@ -214,15 +220,16 @@ facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
    2178 2a03:2880:11ff:f::face:b00c
 ```
 
-- And still making shit tons of Tomcat sessions:
+- Now at least the Tomcat Crawler Session Manager Valve seems to be forcing it to re-use some Tomcat sessions:
 
 ```
-$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-04 | sort | uniq
-28470
+$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-04
+37721
+$ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-04 | sort | uniq | wc -l
+15206
 ```
 
-- And that's even using the Tomcat Crawler Session Manager valve!
-- Maybe we need to limit more dynamic pages, like the "most popular" country, item, and author pages
+- I think we still need to limit more of the dynamic pages, like the "most popular" country, item, and author pages
 - It seems these are popular too, and there is no fucking way Facebook needs that information, yet they are requesting thousands of them!
 
 ```
@@ -244,18 +251,21 @@ $ ./add-dc-rights.py -i /tmp/marlo.csv -db dspace -u dspace -p 'fuuu'
 - The file `marlo.csv` was cleaned up and formatted in Open Refine
 - 165 of the items in their 2017 data are from CGSpace!
 - I will add the data to CGSpace this week (done!)
-- Jesus, is Facebook *trying* to be annoying?
+- Jesus, is Facebook *trying* to be annoying? At least the Tomcat Crawler Session Manager Valve is working to force the bot to re-use its Tomcat sessions:
 
 ```
 # zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "05/Nov/2018" | grep -c "2a03:2880:11ff:"
 29889
-# grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-05 | sort | uniq
-29156
+# grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-05
+29763
+# grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-05 | sort | uniq | wc -l
+1057
 # zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "05/Nov/2018" | grep "2a03:2880:11ff:" | grep -c -E "(handle|bitstream)"
 29896
 ```
 
-- 29,000 requests from Facebook, 29,000 Tomcat sessions, and none of the requests are to the dynamic pages I rate limited yesterday!
+- 29,000 requests from Facebook and none of the requests are to the dynamic pages I rate limited yesterday!
+- At least the Tomcat Crawler Session Manager Valve is working now...
 
 ## 2018-11-06