cgspace-notes/content/post/2016-10.md

+++
date = "2016-10-03T15:53:00+03:00"
author = "Alan Orth"
title = "October, 2016"
tags = ["Notes"]

+++
## 2016-10-03

- Testing adding [ORCIDs to a CSV](https://wiki.duraspace.org/display/DSDOC5x/ORCID+Integration#ORCIDIntegration-EditingexistingitemsusingBatchCSVEditing) file for a single item to see if the author orders get messed up
- Need to test the following scenarios to see how author order is affected:
  - ORCIDs only
  - ORCIDs plus normal authors
- I exported a random item's metadata as CSV, deleted *all columns* except id and collection, and made a new coloum called `ORCID:dc.contributor.author` with the following random ORCIDs from the ORCID registry:

```
0000-0002-6115-0956||0000-0002-3812-8793||0000-0001-7462-405X
```

- Hmm, with the `dc.contributor.author` column removed, DSpace doesn't detect any changes
- With a blank `dc.contributor.author` column, DSpace wants to remove all non-ORCID authors and add the new ORCID authors
- I added the [disclaimer text](https://github.com/ilri/DSpace/issues/234) to the About page, then added a footer link to the disclaimer's ID, but there is a Bootstrap issue that causes the page content to disappear when using in-page anchors: https://github.com/twbs/bootstrap/issues/1768

![Bootstrap issue with in-page anchors](2016/10/bootstrap-issue.png)

- Looks like we'll just have to add the text to the About page (without a link) or add a separate page

## 2016-10-04

- Start testing cleanups of authors that Peter sent last week
- Out of 40,000+ rows, Peter had indicated corrections for ~3,200 of them—too many to look through carefully, so I did some basic quality checking:
  - Trim leading/trailing whitespace
  - Find invalid characters
  - Cluster values to merge obvious authors
- That left us with 3,180 valid corrections and 3 deletions:

```
$ ./fix-metadata-values.py -i authors-fix-3180.csv -f dc.contributor.author -t correct -m 3 -d dspacetest -u dspacetest -p fuuu
$ ./delete-metadata-values.py -i authors-delete-3.csv -f dc.contributor.author -m 3 -d dspacetest -u dspacetest -p fuuu
```

- Remove old about page ([#284](https://github.com/ilri/DSpace/pull/284))
- CGSpace crashed a few times today
- Generate list of unique authors in CCAFS collections:

```
dspacetest=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 AND resource_id IN (select item_id from collection2item where collection_id IN (select resource_id from handle where handle in ('10568/32729', '10568/5472', '10568/5473', '10568/10288', '10568/70974', '10568/3547', '10568/3549', '10568/3531','10568/16890','10568/5470','10568/3546', '10568/36024', '10568/66581', '10568/21789', '10568/5469', '10568/5468', '10568/3548', '10568/71053', '10568/25167'))) group by text_value order by count desc) to /tmp/ccafs-authors.csv with csv;
```

## 2016-10-05

- Work on more infrastructure cleanups for Ansible DSpace role
- Clean up Let's Encrypt plumbing and submit pull request for rmg-ansible-public ([#60](https://github.com/ilri/rmg-ansible-public/pull/60))

## 2016-10-06

- Nice! DSpace Test (linode02) is now having `java.lang.OutOfMemoryError: Java heap space` errors...
- Heap space is 2048m, and we have 5GB of RAM being used for OS cache (Solr!) so let's just bump the memory to 3072m 
- Magdalena from CCAFS asked why the colors in the thumbnails for these [two](https://cgspace.cgiar.org/handle/10568/71249) [items](https://cgspace.cgiar.org/handle/10568/71259) look different, even though they are the same in the PDF itself

![CMYK vs sRGB colors](2016/10/cmyk-vs-srgb.jpg)

- Turns out the first PDF was exported from InDesign using CMYK and the second one was using sRGB
- Run all system updates on DSpace Test and reboot it
Add notes for 2016-10 2016-10-03 17:28:33 +02:00			`+++`
			`date = "2016-10-03T15:53:00+03:00"`
			`author = "Alan Orth"`
			`title = "October, 2016"`
			`tags = ["Notes"]`

			`+++`
			`## 2016-10-03`

			`- Testing adding [ORCIDs to a CSV](https://wiki.duraspace.org/display/DSDOC5x/ORCID+Integration#ORCIDIntegration-EditingexistingitemsusingBatchCSVEditing) file for a single item to see if the author orders get messed up`
			`- Need to test the following scenarios to see how author order is affected:`
			`- ORCIDs only`
			`- ORCIDs plus normal authors`
			- I exported a random item's metadata as CSV, deleted all columns except id and collection, and made a new coloum called `ORCID:dc.contributor.author` with the following random ORCIDs from the ORCID registry:

			```
			`0000-0002-6115-0956\|\|0000-0002-3812-8793\|\|0000-0001-7462-405X`
			```

			- Hmm, with the `dc.contributor.author` column removed, DSpace doesn't detect any changes
			- With a blank `dc.contributor.author` column, DSpace wants to remove all non-ORCID authors and add the new ORCID authors
			`- I added the [disclaimer text](https://github.com/ilri/DSpace/issues/234) to the About page, then added a footer link to the disclaimer's ID, but there is a Bootstrap issue that causes the page content to disappear when using in-page anchors: https://github.com/twbs/bootstrap/issues/1768`

			`![Bootstrap issue with in-page anchors](2016/10/bootstrap-issue.png)`

			`- Looks like we'll just have to add the text to the About page (without a link) or add a separate page`
Add notes for 2016-10-04 2016-10-04 10:34:57 +02:00
			`## 2016-10-04`

			`- Start testing cleanups of authors that Peter sent last week`
			`- Out of 40,000+ rows, Peter had indicated corrections for ~3,200 of them—too many to look through carefully, so I did some basic quality checking:`
			`- Trim leading/trailing whitespace`
			`- Find invalid characters`
			`- Cluster values to merge obvious authors`
			`- That left us with 3,180 valid corrections and 3 deletions:`

			```
			`$ ./fix-metadata-values.py -i authors-fix-3180.csv -f dc.contributor.author -t correct -m 3 -d dspacetest -u dspacetest -p fuuu`
			`$ ./delete-metadata-values.py -i authors-delete-3.csv -f dc.contributor.author -m 3 -d dspacetest -u dspacetest -p fuuu`
			```

			`- Remove old about page ([#284](https://github.com/ilri/DSpace/pull/284))`
Add notes for 2016-10-05 2016-10-06 07:06:32 +02:00			`- CGSpace crashed a few times today`
			`- Generate list of unique authors in CCAFS collections:`

			```
			dspacetest=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 AND resource_id IN (select item_id from collection2item where collection_id IN (select resource_id from handle where handle in ('10568/32729', '10568/5472', '10568/5473', '10568/10288', '10568/70974', '10568/3547', '10568/3549', '10568/3531','10568/16890','10568/5470','10568/3546', '10568/36024', '10568/66581', '10568/21789', '10568/5469', '10568/5468', '10568/3548', '10568/71053', '10568/25167'))) group by text_value order by count desc) to /tmp/ccafs-authors.csv with csv;
			```

			`## 2016-10-05`

			`- Work on more infrastructure cleanups for Ansible DSpace role`
Add notes for 2016-10-06 2016-10-06 08:41:36 +02:00			`- Clean up Let's Encrypt plumbing and submit pull request for rmg-ansible-public ([#60](https://github.com/ilri/rmg-ansible-public/pull/60))`

			`## 2016-10-06`

			- Nice! DSpace Test (linode02) is now having `java.lang.OutOfMemoryError: Java heap space` errors...
			`- Heap space is 2048m, and we have 5GB of RAM being used for OS cache (Solr!) so let's just bump the memory to 3072m`
			`- Magdalena from CCAFS asked why the colors in the thumbnails for these [two](https://cgspace.cgiar.org/handle/10568/71249) [items](https://cgspace.cgiar.org/handle/10568/71259) look different, even though they are the same in the PDF itself`

			`![CMYK vs sRGB colors](2016/10/cmyk-vs-srgb.jpg)`

			`- Turns out the first PDF was exported from InDesign using CMYK and the second one was using sRGB`
			`- Run all system updates on DSpace Test and reboot it`