alanorth/cgspace-notes
1
0
Fork 0
mirror of https://github.com/alanorth/cgspace-notes.git synced 2024-06-29 09:33:48 +02:00
Code Issues Projects Releases Wiki Activity
c05c7213c2
cgspace-notes/content/posts/2021-09.md

4.4 KiB
Raw Blame History

title date author categories
September, 2021 2021-09-01T09:14:07+03:00 Alan Orth
Notes

2021-09-02

  • Troubleshooting the missing Altmetric scores on AReS
    • Turns out that I didn't actually fix them last month because the check for content.altmetric still exists, and I can't access the DOIs using _h.source.DOI for some reason
    • I can access all other kinds of item metadata using the Elasticsearch label, but not DOI!!!
    • I will change DOI to tomato in the repository setup and start a re-harvest... I need to see if this is some kind of reserved word or something...
    • Even as tomato I can't access that field as _h.source.tomato in Angular, but it does work as a filter source... sigh
  • I'm having problems using the OpenRXV API
    • The syntax Moayad showed me last month doesn't seem to honor the search query properly...

2021-09-05

  • Update Docker images on AReS server (linode20) and rebuild OpenRXV:
$ docker images | grep -v ^REPO | sed 's/ \+/:/g' | cut -d: -f1,2 | xargs -L1 docker pull
$ docker-compose build
  • Then run system updates and reboot the server
    • After the system came back up I started a fresh re-harvesting

2021-09-07

  • Checking last month's Solr statistics to see if there are any new bots that I need to purge and add to the list
    • 78.203.225.68 made 50,000 requests on one day in August, and it is using this user agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36
    • It's a fixed line ISP in Montpellier according to AbuseIPDB.com, and has not been flagged as abusive, so it must be some CGIAR SMO person doing some web application harvesting from the browser
    • 130.255.162.154 is in Sweden and made 46,000 requests in August and it is using this user agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11.1; rv:84.0) Gecko/20100101 Firefox/84.0
    • 35.174.144.154 is on Amazon and made 28,000 requests with this user agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36
    • 192.121.135.6 is in Sweden and made 9,000 requests with this user agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11.1; rv:84.0) Gecko/20100101 Firefox/84.0
    • 185.38.40.66 is in Germany and made 6,000 requests with this user agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:89.0) Gecko/20100101 Firefox/89.0 BoldBrains SC/1.10.2.4
    • 3.225.28.105 is in Amazon and made 3,000 requests with this user agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36
    • I also noticed that we still have tons (25,000) of requests by MSNbot using this normal-looking user agent: Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko
    • I can identify them by their reverse DNS: msnbot-40-77-167-105.search.msn.com.
    • I had already purged a bunch of these by their IPs in 2021-06, so it looks like I have to do that again
    • While looking at the MSN requests I noticed tons of requests from another strange host using reverse IP DNS: malta2095.startdedicated.com., astra5139.startdedicated.com., and many others
    • They must be related, because I see them all using the exact same user agent: Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko
    • So this startdedicated.com DNS is some Bing bot also...
  • I extracted all the IPs and purged them using my check-spider-ip-hits.sh script
    • In total I purged 225,000 hits...

2021-09-12

  • Start a harvest on AReS

2021-09-13

  • Mishell Portilla asked me about thumbnails on CGSpace being small
    • For example, 10568/114576 has a lot of white space on the left side
    • I created a new thumbnail with vipsthumbnail:
$ vipsthumbnail ARRTB2020ST.pdf -s x600 -o '%s.jpg[Q=85,optimize_coding,strip]'
  • Looking at the PDF's metadata I see:
    • Producer: iLovePDF
    • Creator: Adobe InDesign 15.0 (Windows)
    • Format: PDF-1.7
  • Eventually I should do more tests on this and perhaps file a bug with DSpace...
  • Some Alliance people contacted me about getting access to the CGSpace API to deposit with their TIP tool
    • I told them I can give them access to DSpace Test and that we should have a meeting soon
    • We need to figure out what controlled vocabularies they should use