CGSpace Notes

Documenting day-to-day work on the CGSpace repository.

November, 2017

2017-11-01

  • The CORE developers responded to say they are looking into their bot not respecting our robots.txt

2017-11-02

  • Today there have been no hits by CORE and no alerts from Linode (coincidence?)
# grep -c "CORE" /var/log/nginx/access.log
0
  • Generate list of authors on CGSpace for Peter to go through and correct:
dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors.csv with csv;
COPY 54701

  • Abenet asked if it would be possible to generate a report of items in Listing and Reports that had “International Fund for Agricultural Development” as the only investor
  • I opened a ticket with Atmire to ask if this was possible: https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=540
  • Work on making the thumbnails in the item view clickable
  • Basically, once you read the METS XML for an item it becomes easy to trace the structure to find the bitstream link
//mets:fileSec/mets:fileGrp[@USE='CONTENT']/mets:file/mets:FLocat[@LOCTYPE='URL']/@xlink:href
  • METS XML is available for all items with this pattern: /metadata/handle/10568/95947/mets.xml
  • I whipped up a quick hack to print a clickable link with this URL on the thumbnail but it needs to check a few corner cases, like when there is a thumbnail but no content bitstream!
  • Help proof fifty-three CIAT records for Sisay: https://dspacetest.cgiar.org/handle/10568/95895
  • A handful of issues with cg.place using format like “Lima, PE” instead of “Lima, Peru”
  • Also, some dates like with completely invalid format like “2010- 06” and “2011-3-28”
  • I also collapsed some consecutive whitespace on a handful of fields

2017-11-03

  • Atmire got back to us to say that they estimate it will take two days of labor to implement the change to Listings and Reports
  • I said I’d ask Abenet if she wants that feature

2017-11-04

  • I finished looking through Sisay’s CIAT records for the “Alianzas de Aprendizaje” data
  • I corrected about half of the authors to standardize them
  • Linode emailed this morning to say that the CPU usage was high again, this time at 6:14AM
  • It’s the first time in a few days that this has happened
  • I had a look to see what was going on, but it isn’t the CORE bot:
# awk '{print $1}' /var/log/nginx/access.log | sort -n | uniq -c | sort -h | tail
    306 68.180.229.31
    323 61.148.244.116
    414 66.249.66.91
    507 40.77.167.16
    618 157.55.39.161
    652 207.46.13.103
    666 157.55.39.254
   1173 104.196.152.243
   1737 66.249.66.90
  23101 138.201.52.218
  • 138.201.52.218 is from some Hetzner server, and I see it making 40,000 requests yesterday too, but none before that:
# zgrep -c 138.201.52.218 /var/log/nginx/access.log*
/var/log/nginx/access.log:24403
/var/log/nginx/access.log.1:45958
/var/log/nginx/access.log.2.gz:0
/var/log/nginx/access.log.3.gz:0
/var/log/nginx/access.log.4.gz:0
/var/log/nginx/access.log.5.gz:0
/var/log/nginx/access.log.6.gz:0
  • It’s clearly a bot as it’s making tens of thousands of requests, but it’s using a “normal” user agent:
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36
  • For now I don’t know what this user is!