alanorth/cgspace-notes
1
0
Fork 0
mirror of https://github.com/alanorth/cgspace-notes.git synced 2025-01-27 05:49:12 +01:00
Code Issues Projects Releases Wiki Activity

Update notes for 2019-11-09

Browse Source
This commit is contained in:
Alan Orth
2019-11-09 21:41:15 +02:00
parent b7dbd8497d
commit 9798063ec8
3 changed files with 53 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
content/posts/2019-11.md
View File

@ -246,4 +246,19 @@ $ curl -s 'http://localhost:8081/solr/statistics/select?facet=true&facet.field=d
- `User-Agent:Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) IKU/7.0.5.9226;IKUCID/IKU;`
- I filed [an issue](https://github.com/atmire/COUNTER-Robots/issues/27) on the COUNTER-Robots project to see if they agree to add `User-Agent:` to the list of robot user agents
## 2019-11-09
- Deploy the latest `5_x-prod` branch on CGSpace (linode19)
- This includes the updated CCAFS phase II project tags and the updated spider user agents
- Run all system updates on CGSpace and reboot the server
- After rebooting it seems that all Solr statistics cores came back up fine...
- I did some work to clean up my bot processing script and removed about 2 million hits from the statistics cores on CGSpace
- The script is called `check-spider-hits.sh`
- After a bunch of tests and checks I ran it for each statistics shard like so:
```
$ for shard in statistics statistics-2018 statistics-2017 statistics-2016 statistics-2015 stat
istics-2014 statistics-2013 statistics-2012 statistics-2011 statistics-2010; do ./check-spider-hits.sh -s $shard -p yes; done
```
<!-- vim: set sw=2 ts=2: -->