## September, 2017

Thu Sep 07, 2017 by Alan Orth in Notes

## 2017-09-06

- Linode sent an alert that CGSpace (linode18) was using 261% CPU for the past two hours

## 2017-09-07

- Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account is both in the approvers step as well as the group

Read more →

## August, 2017

Tue Aug 01, 2017 by Alan Orth in Notes

## 2017-08-01

- Linode sent an alert that CGSpace (linode18) was using 350% CPU for the past two hours
- I looked in the Activity pane of the Admin Control Panel and it seems that Google, Baidu, Yahoo, and Bing are all crawling with massive numbers of bots concurrently (~100 total, mostly Baidu and Google)
- The good thing is that, according to `dspace.log.2017-08-01`, they are all using the same Tomcat session
- This means our Tomcat Crawler Session Valve is working
- But many of the bots are browsing dynamic URLs like:
  - /handle/10568/3353/discover
  - /handle/10568/16510/browse
- The `robots.txt` only blocks the top-level `/discover` and `/browse` URLs… we will need to find a way to forbid them from accessing these!
- Relevant issue from DSpace Jira (semi resolved in DSpace 6.0): https://jira.duraspace.org/browse/DS-2962
- It turns out that we’re already adding the `X-Robots-Tag "none"` HTTP header, but this only forbids the search engine from indexing the page, not crawling it!
- Also, the bot has to successfully browse the page first so it can receive the HTTP header…
- We might actually have to block these requests with HTTP 403 depending on the user agent
- Abenet pointed out that the CGIAR Library Historical Archive collection I sent July 20th only had ~100 entries, instead of 2415
- This was due to newline characters in the `dc.description.abstract` column, which caused OpenRefine to choke when exporting the CSV
- I exported a new CSV from the collection on DSpace Test and then manually removed the characters in vim using `g/^$/d`
- Then I cleaned up the author authorities and HTML characters in OpenRefine and sent the file back to Abenet

Read more →

## July, 2017

Sat Jul 01, 2017 by Alan Orth in Notes

## 2017-07-01

- Run system updates and reboot DSpace Test

## 2017-07-04

- Merge changes for WLE Phase II theme rename (#329)
- Looking at extracting the metadata registries from ICARDA’s MEL DSpace database so we can compare fields with CGSpace
- We can use PostgreSQL’s extended output format (`-x`) plus `sed` to format the output into quasi XML:

Read more →

## June, 2017

Thu Jun 01, 2017 by Alan Orth in Notes

2017-06-01 After discussion with WLE and CGSpace content people, we decided to just add one metadata field for the WLE Research Themes The cg.identifier.wletheme field will be used for both Phase I and Phase II Research Themes Then we’ll create a new sub-community for Phase II and create collections for the research themes there The current “Research Themes” community will be renamed to “WLE Phase I Research Themes” Tagged all items in the current Phase I collections with their appropriate themes Create pull request to add Phase II research themes to the submission form: #328 Add cg. Read more →

## May, 2017

Mon May 01, 2017 by Alan Orth in Notes

2017-05-01 ICARDA apparently started working on CG Core on their MEL repository They have done a few cg.* fields, but not very consistent and even copy some of CGSpace items: https://mel.cgiar.org/xmlui/handle/20.500.11766/6911?show=full https://cgspace.cgiar.org/handle/10568/73683 2017-05-02 Atmire got back about the Workflow Statistics issue, and apparently it’s a bug in the CUA module so they will send us a pull request 2017-05-04 Sync DSpace Test with database and assetstore from CGSpace Re-deploy DSpace Test with Atmire’s CUA patch for workflow statistics, run system updates, and restart the server Now I can see the workflow statistics and am able to select users, but everything returns 0 items Megan says there are still some mapped items are not appearing since last week, so I forced a full index-discovery -b Need to remember to check if the collection has more items (currently 39 on CGSpace, but 118 on the freshly reindexed DSPace Test) tomorrow: https://cgspace. Read more →

## April, 2017

Sun Apr 02, 2017 by Alan Orth in Notes

## 2017-04-02

- Merge one change to CCAFS flagships that I had forgotten to remove last month (“MANAGING CLIMATE RISK”): https://github.com/ilri/DSpace/pull/317
- Quick proof-of-concept hack to add `dc.rights` to the input form, including some inline instructions/hints:

- Remove redundant/duplicate text in the DSpace submission license
- Testing the CMYK patch on a collection with 650 items:

    $ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Thumbnail" -v >& /tmp/filter-media-cmyk.txt

Read more →

## March, 2017

Wed Mar 01, 2017 by Alan Orth in Notes

## 2017-03-01

- Run the 279 CIAT author corrections on CGSpace

## 2017-03-02

- Skype with Michael and Peter, discussing moving the CGIAR Library to CGSpace
- CGIAR people possibly open to moving content, redirecting library.cgiar.org to CGSpace and letting CGSpace resolve their handles
- They might come in at the top level in one “CGIAR System” community, or with several communities
- I need to spend a bit of time looking at the multiple handle support in DSpace and see if new content can be minted in both handles, or just one?
- Need to send Peter and Michael some notes about this in a few days
- Also, need to consider talking to Atmire about hiring them to bring ORCiD metadata to REST / OAI
- Filed an issue on DSpace issue tracker for the `filter-media` bug that causes it to process JPGs even when limiting to the PDF thumbnail plugin: DS-3516
- Discovered that the ImageMagic `filter-media` plugin creates JPG thumbnails with the CMYK colorspace when the source PDF is using CMYK
- Interestingly, it seems DSpace 4.x’s thumbnails were sRGB, but forcing regeneration using DSpace 5.x’s ImageMagick plugin creates CMYK JPGs if the source PDF was CMYK (see 10568/51999):

    $ identify ~/Desktop/alc_contrastes_desafios.jpg
    /Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000

Read more →

## February, 2017

Tue Feb 07, 2017 by Alan Orth in Notes

## 2017-02-07

- An item was mapped twice erroneously again, so I had to remove one of the mappings manually:

    dspace=# select * from collection2item where item_id = '80278';
      id   | collection_id | item_id
    -------+---------------+---------
     92551 |           313 |   80278
     92550 |           313 |   80278
     90774 |          1051 |   80278
    (3 rows)
    dspace=# delete from collection2item where id = 92551 and item_id = 80278;
    DELETE 1

- Create issue on GitHub to track the addition of CCAFS Phase II project tags (#301)
- Looks like we’ll be using `cg.identifier.ccafsprojectpii` as the field name

Read more →

## January, 2017

Mon Jan 02, 2017 by Alan Orth in Notes

## 2017-01-02

- I checked to see if the Solr sharding task that is supposed to run on January 1st had run and saw there was an error
- I tested on DSpace Test as well and it doesn’t work there either
- I asked on the dspace-tech mailing list because it seems to be broken, and actually now I’m not sure if we’ve ever had the sharding task run successfully over all these years

Read more →

## December, 2016

Fri Dec 02, 2016 by Alan Orth in Notes

## 2016-12-02

- CGSpace was down for five hours in the morning while I was sleeping
- While looking in the logs for errors, I see tons of warnings about Atmire MQM:

    2016-12-02 03:00:32,352 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=CREATE, SubjectType=BUNDLE, SubjectID=70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632305, dispatcher=1544803905, detail=[null], transactionID="TX157907838689377964651674089851855413607")
    2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY_METADATA, SubjectType=BUNDLE, SubjectID =70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632309, dispatcher=1544803905, detail="dc.title", transactionID="TX157907838689377964651674089851855413607")
    2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=ITEM, SubjectID=80044, Object Type=BUNDLE, ObjectID=70316, TimeStamp=1480647632311, dispatcher=1544803905, detail="THUMBNAIL", transactionID="TX157907838689377964651674089851855413607")
    2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=BUNDLE, SubjectID=70316, Obje ctType=BITSTREAM, ObjectID=86715, TimeStamp=1480647632318, dispatcher=1544803905, detail="-1", transactionID="TX157907838689377964651674089851855413607")
    2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY, SubjectType=ITEM, SubjectID=80044, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632351, dispatcher=1544803905, detail=[null], transactionID="TX157907838689377964651674089851855413607")

- I see thousands of them in the logs for the last few months, so it’s not related to the DSpace 5.5 upgrade
- I’ve raised a ticket with Atmire to ask
- Another worrying error from dspace.log is:

Read more →