From 72996d3c7c119e4acfa8707f2ad6567456ecebd0 Mon Sep 17 00:00:00 2001 From: Alan Orth Date: Fri, 15 Nov 2019 19:17:08 +0200 Subject: [PATCH] Add notes for 2019-11-15 --- content/posts/2019-11.md | 40 +++++++++++++++++++++++++ docs/2019-11/index.html | 65 ++++++++++++++++++++++++++++++---------- docs/sitemap.xml | 10 +++---- 3 files changed, 95 insertions(+), 20 deletions(-) diff --git a/content/posts/2019-11.md b/content/posts/2019-11.md index 50ce9c842..7b4d26474 100644 --- a/content/posts/2019-11.md +++ b/content/posts/2019-11.md @@ -321,4 +321,44 @@ $ tidy -xml -utf8 -iq -m -w 0 dspace/config/controlled-vocabularies/cg-creator-i - This allows me to detect and purge many more hits from the Solr statistics core - I've tested it quite a bit on DSpace Test, but I need to do a little more before I feel comfortable running the new code on CGSpace's Solr cores +## 2019-11-15 + +- Run the new version of `check-spider-hits.sh` on CGSpace's Solr statistics cores one by one, starting from the oldest just in case something goes wrong +- But then I noticed that some (all?) of the hits weren't actually getting purged, all of which were using regular expressions like: + - `MetaURI[\+\s]API\/[0-9]\.[0-9]` + - `FDM(\s|\+)[0-9]` + - `Goldfire(\s|\+)Server` + - `^Mozilla\/4\.0\+\(compatible;\)$` + - `^Mozilla\/4\.0\+\(compatible;\+ICS\)$` + - `^Mozilla\/4\.5\+\[en]\+\(Win98;\+I\)$` +- Upon closer inspection, the plus signs seem to be getting misinterpreted somehow in the delete, but not in the select! +- Plus signs are special in regular expressions, URLs, and Solr's Lucene query parser, so I'm actually not sure where the issue is + - I tried to do URL encoding of the +, double escaping, etc... but nothing worked + - I'm going to ignore regular expressions that have pluses for now +- I think I might also have to ignore patterns that have percent signs, like `^\%?default\%?$` +- After I added the ignores and did some more testing I finally ran the `check-spider-hits.sh` on all CGSpace Solr statistics cores and these are the number of hits purged from each core: + - statistics-2010: 113 + - statistics-2011: 7235 + - statistics-2012: 0 + - statistics-2013: 0 + - statistics-2014: 316 + - statistics-2015: 16809 + - statistics-2016: 41732 + - statistics-2017: 39207 + - statistics-2018: 295546 + - statistics: 1043373 +- That's 1.4 million hits in addition to the 2 million I purged earlier this week... +- For posterity, the major contributors to the hits on the statistics core were: + - Purging 812429 hits from curl\/ in statistics + - Purging 48206 hits from facebookexternalhit\/ in statistics + - Purging 72004 hits from PHP\/ in statistics + - Purging 76072 hits from Yeti\/[0-9] in statistics +- Most of the curl hits were from CIAT in mid-2019, where they were using [GuzzleHttp](https://guzzle3.readthedocs.io/http-client/client.html) from PHP, which uses something like this for its user agent: + +``` +Guzzle/ curl/ PHP/ +``` + +- Run system updates on DSpace Test and reboot the server + diff --git a/docs/2019-11/index.html b/docs/2019-11/index.html index 66e5dad60..c796d4390 100644 --- a/docs/2019-11/index.html +++ b/docs/2019-11/index.html @@ -34,7 +34,7 @@ Let’s see how many of the REST API requests were for bitstreams (because t - + @@ -73,9 +73,9 @@ Let’s see how many of the REST API requests were for bitstreams (because t "@type": "BlogPosting", "headline": "November, 2019", "url": "https:\/\/alanorth.github.io\/cgspace-notes\/2019-11\/", - "wordCount": "2115", + "wordCount": "2435", "datePublished": "2019-11-04T12:20:30+02:00", - "dateModified": "2019-11-14T09:47:49+02:00", + "dateModified": "2019-11-15T00:54:13+02:00", "author": { "@type": "Person", "name": "Alan Orth" @@ -479,21 +479,56 @@ $ ./resolve-orcids.py -i /tmp/2019-11-14-combined-orcids.txt -o /tmp/2019-11-14-

sort names, copy to cg-creator-id.xml, add XML formatting, and then format with tidy (preserving accents)

-

$ tidy -xml -utf8 -iq -m -w 0 dspace/config/controlled-vocabularies/cg-creator-id.xml +

$ tidy -xml -utf8 -iq -m -w 0 dspace/config/controlled-vocabularies/cg-creator-id.xml

+ +

+- I created a [pull request](https://github.com/ilri/DSpace/pull/437) and merged them into the `5_x-prod` branch
+  - I will deploy them to CGSpace in the next few days
+- Greatly improve my `check-spider-hits.sh` script to handle regular expressions in the spider agents patterns file
+  - This allows me to detect and purge many more hits from the Solr statistics core
+  - I've tested it quite a bit on DSpace Test, but I need to do a little more before I feel comfortable running the new code on CGSpace's Solr cores
+
+## 2019-11-15
+
+- Run the new version of `check-spider-hits.sh` on CGSpace's Solr statistics cores one by one, starting from the oldest just in case something goes wrong
+- But then I noticed that some (all?) of the hits weren't actually getting purged, all of which were using regular expressions like:
+  - `MetaURI[\+\s]API\/[0-9]\.[0-9]`
+  - `FDM(\s|\+)[0-9]`
+  - `Goldfire(\s|\+)Server`
+  - `^Mozilla\/4\.0\+\(compatible;\)$`
+  - `^Mozilla\/4\.0\+\(compatible;\+ICS\)$`
+  - `^Mozilla\/4\.5\+\[en]\+\(Win98;\+I\)$`
+- Upon closer inspection, the plus signs seem to be getting misinterpreted somehow in the delete, but not in the select!
+- Plus signs are special in regular expressions, URLs, and Solr's Lucene query parser, so I'm actually not sure where the issue is
+  - I tried to do URL encoding of the +, double escaping, etc... but nothing worked 
+  - I'm going to ignore regular expressions that have pluses for now
+- I think I might also have to ignore patterns that have percent signs, like `^\%?default\%?$`
+- After I added the ignores and did some more testing I finally ran the `check-spider-hits.sh` on all CGSpace Solr statistics cores and these are the number of hits purged from each core:
+  - statistics-2010: 113
+  - statistics-2011: 7235
+  - statistics-2012: 0
+  - statistics-2013: 0
+  - statistics-2014: 316
+  - statistics-2015: 16809
+  - statistics-2016: 41732
+  - statistics-2017: 39207
+  - statistics-2018: 295546
+  - statistics: 1043373
+- That's 1.4 million hits in addition to the 2 million I purged earlier this week...
+- For posterity, the major contributors to the hits on the statistics core were:
+  - Purging 812429 hits from curl\/ in statistics
+  - Purging 48206 hits from facebookexternalhit\/ in statistics
+  - Purging 72004 hits from PHP\/ in statistics
+  - Purging 76072 hits from Yeti\/[0-9] in statistics
+- Most of the curl hits were from CIAT in mid-2019, where they were using [GuzzleHttp](https://guzzle3.readthedocs.io/http-client/client.html) from PHP, which uses something like this for its user agent:
+
+
+ +

Guzzle/ curl/ PHP/ ```

    -
  • I created a pull request and merged them into the 5_x-prod branch - -
      -
    • I will deploy them to CGSpace in the next few days
      • -
    • -
  • Greatly improve my check-spider-hits.sh script to handle regular expressions in the spider agents patterns file - -
      -
    • This allows me to detect and purge many more hits from the Solr statistics core
      • -
    • I’ve tested it quite a bit on DSpace Test, but I need to do a little more before I feel comfortable running the new code on CGSpace’s Solr cores
      • -
    • +
  • Run system updates on DSpace Test and reboot the server
diff --git a/docs/sitemap.xml b/docs/sitemap.xml index adb2797b2..526e0b000 100644 --- a/docs/sitemap.xml +++ b/docs/sitemap.xml @@ -4,27 +4,27 @@ https://alanorth.github.io/cgspace-notes/categories/ - 2019-11-14T09:47:49+02:00 + 2019-11-15T00:54:13+02:00 https://alanorth.github.io/cgspace-notes/ - 2019-11-14T09:47:49+02:00 + 2019-11-15T00:54:13+02:00 https://alanorth.github.io/cgspace-notes/categories/notes/ - 2019-11-14T09:47:49+02:00 + 2019-11-15T00:54:13+02:00 https://alanorth.github.io/cgspace-notes/2019-11/ - 2019-11-14T09:47:49+02:00 + 2019-11-15T00:54:13+02:00 https://alanorth.github.io/cgspace-notes/posts/ - 2019-11-14T09:47:49+02:00 + 2019-11-15T00:54:13+02:00