CGSpace Notes

Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called CGIAR System Organization.

2017-09-06

• Linode sent an alert that CGSpace (linode18) was using 261% CPU for the past two hours

2017-09-07

• Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account is both in the approvers step as well as the group

2017-08-01

• Linode sent an alert that CGSpace (linode18) was using 350% CPU for the past two hours
• I looked in the Activity pane of the Admin Control Panel and it seems that Google, Baidu, Yahoo, and Bing are all crawling with massive numbers of bots concurrently (~100 total, mostly Baidu and Google)
• The good thing is that, according to dspace.log.2017-08-01, they are all using the same Tomcat session
• This means our Tomcat Crawler Session Valve is working
• But many of the bots are browsing dynamic URLs like:
  • /handle/10568/3353/discover
  • /handle/10568/16510/browse
• The robots.txt only blocks the top-level /discover and /browse URLs… we will need to find a way to forbid them from accessing these!
• Relevant issue from DSpace Jira (semi resolved in DSpace 6.0): https://jira.duraspace.org/browse/DS-2962
• It turns out that we’re already adding the X-Robots-Tag "none" HTTP header, but this only forbids the search engine from indexing the page, not crawling it!
• Also, the bot has to successfully browse the page first so it can receive the HTTP header…
• We might actually have to block these requests with HTTP 403 depending on the user agent
• Abenet pointed out that the CGIAR Library Historical Archive collection I sent July 20th only had ~100 entries, instead of 2415
• This was due to newline characters in the dc.description.abstract column, which caused OpenRefine to choke when exporting the CSV
• I exported a new CSV from the collection on DSpace Test and then manually removed the characters in vim using g/^$/d
• Then I cleaned up the author authorities and HTML characters in OpenRefine and sent the file back to Abenet

2017-07-01

• Run system updates and reboot DSpace Test

2017-07-04

• Merge changes for WLE Phase II theme rename (#329)
• Looking at extracting the metadata registries from ICARDA’s MEL DSpace database so we can compare fields with CGSpace
• We can use PostgreSQL’s extended output format (-x) plus sed to format the output into quasi XML:

2017-04-02

• Merge one change to CCAFS flagships that I had forgotten to remove last month (“MANAGING CLIMATE RISK”): https://github.com/ilri/DSpace/pull/317
• Quick proof-of-concept hack to add dc.rights to the input form, including some inline instructions/hints:

• Remove redundant/duplicate text in the DSpace submission license

• Testing the CMYK patch on a collection with 650 items:

  $ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Thumbnail" -v >& /tmp/filter-media-cmyk.txt
  

2017-03-01

• Run the 279 CIAT author corrections on CGSpace

2017-03-02

• Skype with Michael and Peter, discussing moving the CGIAR Library to CGSpace
• CGIAR people possibly open to moving content, redirecting library.cgiar.org to CGSpace and letting CGSpace resolve their handles
• They might come in at the top level in one “CGIAR System” community, or with several communities
• I need to spend a bit of time looking at the multiple handle support in DSpace and see if new content can be minted in both handles, or just one?
• Need to send Peter and Michael some notes about this in a few days
• Also, need to consider talking to Atmire about hiring them to bring ORCiD metadata to REST / OAI
• Filed an issue on DSpace issue tracker for the filter-media bug that causes it to process JPGs even when limiting to the PDF thumbnail plugin: DS-3516
• Discovered that the ImageMagic filter-media plugin creates JPG thumbnails with the CMYK colorspace when the source PDF is using CMYK

• Interestingly, it seems DSpace 4.x’s thumbnails were sRGB, but forcing regeneration using DSpace 5.x’s ImageMagick plugin creates CMYK JPGs if the source PDF was CMYK (see ¹⁰⁵⁶⁸⁄₅₁₉₉₉):

  $ identify ~/Desktop/alc_contrastes_desafios.jpg
  /Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000
  

2017-02-07

• An item was mapped twice erroneously again, so I had to remove one of the mappings manually:

  dspace=# select * from collection2item where item_id = '80278';
  id   | collection_id | item_id
  -------+---------------+---------
  92551 |           313 |   80278
  92550 |           313 |   80278
  90774 |          1051 |   80278
  (3 rows)
  dspace=# delete from collection2item where id = 92551 and item_id = 80278;
  DELETE 1
  

• Create issue on GitHub to track the addition of CCAFS Phase II project tags (#301)

• Looks like we’ll be using cg.identifier.ccafsprojectpii as the field name

2017-01-02

• I checked to see if the Solr sharding task that is supposed to run on January 1st had run and saw there was an error
• I tested on DSpace Test as well and it doesn’t work there either
• I asked on the dspace-tech mailing list because it seems to be broken, and actually now I’m not sure if we’ve ever had the sharding task run successfully over all these years