mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2024-11-22 06:35:03 +01:00
Add notes for 2019-11-15
This commit is contained in:
parent
b59a6defe5
commit
72996d3c7c
GPG Key ID:
0FB860CC9C45B1B9
@ -321,4 +321,44 @@ $ tidy -xml -utf8 -iq -m -w 0 dspace/config/controlled-vocabularies/cg-creator-i
|
||||
- This allows me to detect and purge many more hits from the Solr statistics core
|
||||
- I've tested it quite a bit on DSpace Test, but I need to do a little more before I feel comfortable running the new code on CGSpace's Solr cores
|
||||
|
||||
## 2019-11-15
|
||||
|
||||
- Run the new version of `check-spider-hits.sh` on CGSpace's Solr statistics cores one by one, starting from the oldest just in case something goes wrong
|
||||
- But then I noticed that some (all?) of the hits weren't actually getting purged, all of which were using regular expressions like:
|
||||
- `MetaURI[\+\s]API\/[0-9]\.[0-9]`
|
||||
- `FDM(\s|\+)[0-9]`
|
||||
- `Goldfire(\s|\+)Server`
|
||||
- `^Mozilla\/4\.0\+\(compatible;\)$`
|
||||
- `^Mozilla\/4\.0\+\(compatible;\+ICS\)$`
|
||||
- `^Mozilla\/4\.5\+\[en]\+\(Win98;\+I\)$`
|
||||
- Upon closer inspection, the plus signs seem to be getting misinterpreted somehow in the delete, but not in the select!
|
||||
- Plus signs are special in regular expressions, URLs, and Solr's Lucene query parser, so I'm actually not sure where the issue is
|
||||
- I tried to do URL encoding of the +, double escaping, etc... but nothing worked
|
||||
- I'm going to ignore regular expressions that have pluses for now
|
||||
- I think I might also have to ignore patterns that have percent signs, like `^\%?default\%?$`
|
||||
- After I added the ignores and did some more testing I finally ran the `check-spider-hits.sh` on all CGSpace Solr statistics cores and these are the number of hits purged from each core:
|
||||
- statistics-2010: 113
|
||||
- statistics-2011: 7235
|
||||
- statistics-2012: 0
|
||||
- statistics-2013: 0
|
||||
- statistics-2014: 316
|
||||
- statistics-2015: 16809
|
||||
- statistics-2016: 41732
|
||||
- statistics-2017: 39207
|
||||
- statistics-2018: 295546
|
||||
- statistics: 1043373
|
||||
- That's 1.4 million hits in addition to the 2 million I purged earlier this week...
|
||||
- For posterity, the major contributors to the hits on the statistics core were:
|
||||
- Purging 812429 hits from curl\/ in statistics
|
||||
- Purging 48206 hits from facebookexternalhit\/ in statistics
|
||||
- Purging 72004 hits from PHP\/ in statistics
|
||||
- Purging 76072 hits from Yeti\/[0-9] in statistics
|
||||
- Most of the curl hits were from CIAT in mid-2019, where they were using [GuzzleHttp](https://guzzle3.readthedocs.io/http-client/client.html) from PHP, which uses something like this for its user agent:
|
||||
|
||||
```
|
||||
Guzzle/<Guzzle_Version> curl/<curl_version> PHP/<PHP_VERSION>
|
||||
```
|
||||
|
||||
- Run system updates on DSpace Test and reboot the server
|
||||
|
||||
<!-- vim: set sw=2 ts=2: -->
|
||||
|
||||
@ -34,7 +34,7 @@ Let’s see how many of the REST API requests were for bitstreams (because t
|
||||
<meta property="og:type" content="article" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2019-11/" />
|
||||
<meta property="article:published_time" content="2019-11-04T12:20:30+02:00" />
|
||||
<meta property="article:modified_time" content="2019-11-14T09:47:49+02:00" />
|
||||
<meta property="article:modified_time" content="2019-11-15T00:54:13+02:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="November, 2019"/>
|
||||
@ -73,9 +73,9 @@ Let’s see how many of the REST API requests were for bitstreams (because t
|
||||
"@type": "BlogPosting",
|
||||
"headline": "November, 2019",
|
||||
"url": "https:\/\/alanorth.github.io\/cgspace-notes\/2019-11\/",
|
||||
"wordCount": "2115",
|
||||
"wordCount": "2435",
|
||||
"datePublished": "2019-11-04T12:20:30+02:00",
|
||||
"dateModified": "2019-11-14T09:47:49+02:00",
|
||||
"dateModified": "2019-11-15T00:54:13+02:00",
|
||||
"author": {
|
||||
"@type": "Person",
|
||||
"name": "Alan Orth"
|
||||
@ -479,21 +479,56 @@ $ ./resolve-orcids.py -i /tmp/2019-11-14-combined-orcids.txt -o /tmp/2019-11-14-
|
||||
|
||||
<h1 id="sort-names-copy-to-cg-creator-id-xml-add-xml-formatting-and-then-format-with-tidy-preserving-accents">sort names, copy to cg-creator-id.xml, add XML formatting, and then format with tidy (preserving accents)</h1>
|
||||
|
||||
<p>$ tidy -xml -utf8 -iq -m -w 0 dspace/config/controlled-vocabularies/cg-creator-id.xml
|
||||
<p>$ tidy -xml -utf8 -iq -m -w 0 dspace/config/controlled-vocabularies/cg-creator-id.xml</p>
|
||||
|
||||
<pre><code>
|
||||
- I created a [pull request](https://github.com/ilri/DSpace/pull/437) and merged them into the `5_x-prod` branch
|
||||
- I will deploy them to CGSpace in the next few days
|
||||
- Greatly improve my `check-spider-hits.sh` script to handle regular expressions in the spider agents patterns file
|
||||
- This allows me to detect and purge many more hits from the Solr statistics core
|
||||
- I've tested it quite a bit on DSpace Test, but I need to do a little more before I feel comfortable running the new code on CGSpace's Solr cores
|
||||
|
||||
## 2019-11-15
|
||||
|
||||
- Run the new version of `check-spider-hits.sh` on CGSpace's Solr statistics cores one by one, starting from the oldest just in case something goes wrong
|
||||
- But then I noticed that some (all?) of the hits weren't actually getting purged, all of which were using regular expressions like:
|
||||
- `MetaURI[\+\s]API\/[0-9]\.[0-9]`
|
||||
- `FDM(\s|\+)[0-9]`
|
||||
- `Goldfire(\s|\+)Server`
|
||||
- `^Mozilla\/4\.0\+\(compatible;\)$`
|
||||
- `^Mozilla\/4\.0\+\(compatible;\+ICS\)$`
|
||||
- `^Mozilla\/4\.5\+\[en]\+\(Win98;\+I\)$`
|
||||
- Upon closer inspection, the plus signs seem to be getting misinterpreted somehow in the delete, but not in the select!
|
||||
- Plus signs are special in regular expressions, URLs, and Solr's Lucene query parser, so I'm actually not sure where the issue is
|
||||
- I tried to do URL encoding of the +, double escaping, etc... but nothing worked
|
||||
- I'm going to ignore regular expressions that have pluses for now
|
||||
- I think I might also have to ignore patterns that have percent signs, like `^\%?default\%?$`
|
||||
- After I added the ignores and did some more testing I finally ran the `check-spider-hits.sh` on all CGSpace Solr statistics cores and these are the number of hits purged from each core:
|
||||
- statistics-2010: 113
|
||||
- statistics-2011: 7235
|
||||
- statistics-2012: 0
|
||||
- statistics-2013: 0
|
||||
- statistics-2014: 316
|
||||
- statistics-2015: 16809
|
||||
- statistics-2016: 41732
|
||||
- statistics-2017: 39207
|
||||
- statistics-2018: 295546
|
||||
- statistics: 1043373
|
||||
- That's 1.4 million hits in addition to the 2 million I purged earlier this week...
|
||||
- For posterity, the major contributors to the hits on the statistics core were:
|
||||
- Purging 812429 hits from curl\/ in statistics
|
||||
- Purging 48206 hits from facebookexternalhit\/ in statistics
|
||||
- Purging 72004 hits from PHP\/ in statistics
|
||||
- Purging 76072 hits from Yeti\/[0-9] in statistics
|
||||
- Most of the curl hits were from CIAT in mid-2019, where they were using [GuzzleHttp](https://guzzle3.readthedocs.io/http-client/client.html) from PHP, which uses something like this for its user agent:
|
||||
|
||||
</code></pre>
|
||||
|
||||
<p>Guzzle/<Guzzle_Version> curl/<curl_version> PHP/<PHP_VERSION>
|
||||
```</p>
|
||||
|
||||
<ul>
|
||||
<li>I created a <a href="https://github.com/ilri/DSpace/pull/437">pull request</a> and merged them into the <code>5_x-prod</code> branch
|
||||
|
||||
<ul>
|
||||
<li>I will deploy them to CGSpace in the next few days</li>
|
||||
</ul></li>
|
||||
<li>Greatly improve my <code>check-spider-hits.sh</code> script to handle regular expressions in the spider agents patterns file
|
||||
|
||||
<ul>
|
||||
<li>This allows me to detect and purge many more hits from the Solr statistics core</li>
|
||||
<li>I’ve tested it quite a bit on DSpace Test, but I need to do a little more before I feel comfortable running the new code on CGSpace’s Solr cores</li>
|
||||
</ul></li>
|
||||
<li>Run system updates on DSpace Test and reboot the server</li>
|
||||
</ul>
|
||||
|
||||
<!-- vim: set sw=2 ts=2: -->
|
||||
|
||||
@ -4,27 +4,27 @@
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/categories/</loc>
|
||||
<lastmod>2019-11-14T09:47:49+02:00</lastmod>
|
||||
<lastmod>2019-11-15T00:54:13+02:00</lastmod>
|
||||
</url>
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/</loc>
|
||||
<lastmod>2019-11-14T09:47:49+02:00</lastmod>
|
||||
<lastmod>2019-11-15T00:54:13+02:00</lastmod>
|
||||
</url>
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
|
||||
<lastmod>2019-11-14T09:47:49+02:00</lastmod>
|
||||
<lastmod>2019-11-15T00:54:13+02:00</lastmod>
|
||||
</url>
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/2019-11/</loc>
|
||||
<lastmod>2019-11-14T09:47:49+02:00</lastmod>
|
||||
<lastmod>2019-11-15T00:54:13+02:00</lastmod>
|
||||
</url>
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
|
||||
<lastmod>2019-11-14T09:47:49+02:00</lastmod>
|
||||
<lastmod>2019-11-15T00:54:13+02:00</lastmod>
|
||||
</url>
|
||||
|
||||
<url>
|
||||
|
||||
Loading…
Reference in New Issue
Block a user