May, 2016

2016-05-01

  • Since yesterday there have been 10,000 REST errors and the site has been unstable again
  • I have blocked access to the API now
  • There are 3,000 IPs accessing the REST API in a 24-hour period!
# awk '{print $1}' /var/log/nginx/rest.log  | uniq | wc -l
3168
  • The two most often requesters are in Ethiopia and Colombia: 213.55.99.121 and 181.118.144.29
  • 100% of the requests coming from Ethiopia are like this and result in an HTTP 500:
GET /rest/handle/10568/NaN?expand=parentCommunityList,metadata HTTP/1.1
  • For now I’ll block just the Ethiopian IP
  • The owner of that application has said that the NaN (not a number) is an error in his code and he’ll fix it

2016-05-03

  • Update nginx to 1.10.x branch on CGSpace
  • Fix a reference to dc.type.output in Discovery that I had missed when we migrated to dc.type last month (#223)

Item type in Discovery results

2016-05-06

  • DSpace Test is down, catalina.out has lots of messages about heap space from some time yesterday (!)
  • It looks like Sisay was doing some batch imports
  • Hmm, also disk space is full
  • I decided to blow away the solr indexes, since they are 50GB and we don’t really need all the Atmire stuff there right now
  • I will re-generate the Discovery indexes after re-deploying