cgspace-notes/content/post/2017-02.md

+++
date = "2017-02-07T07:04:52-08:00"
author = "Alan Orth"
title = "February, 2017"
tags = ["Notes"]

+++
## 2017-02-07

- An item was mapped twice erroneously again, so I had to remove one of the mappings manually:

```
dspace=# select * from collection2item where item_id = '80278';
  id   | collection_id | item_id
-------+---------------+---------
 92551 |           313 |   80278
 92550 |           313 |   80278
 90774 |          1051 |   80278
(3 rows)
dspace=# delete from collection2item where id = 92551 and item_id = 80278;
DELETE 1
```

- Create issue on GitHub to track the addition of CCAFS Phase II project tags ([#301](https://github.com/ilri/DSpace/issues/301))
- Looks like we'll be using `cg.identifier.ccafsprojectpii` as the field name

<!--more-->

## 2017-02-08

- We also need to rename some of the CCAFS Phase I flagships:
  - CLIMATE-SMART AGRICULTURAL PRACTICES → CLIMATE-SMART TECHNOLOGIES AND PRACTICES
  - CLIMATE RISK MANAGEMENT → CLIMATE SERVICES AND SAFETY NETS
  - LOW EMISSIONS AGRICULTURE → LOW EMISSIONS DEVELOPMENT
  - POLICIES AND INSTITUTIONS → PRIORITIES AND POLICIES FOR CSA
- The climate risk management one doesn't exist, so I will have to ask Magdalena if they want me to add it to the input forms
- Start testing some nearly 500 author corrections that CCAFS sent me:

```
$ ./fix-metadata-values.py -i /tmp/CCAFS-Authors-Feb-7.csv -f dc.contributor.author -t 'correct name' -m 3 -d dspace -u dspace -p fuuu
```

## 2017-02-09

- More work on CCAFS Phase II stuff
- Looks like simply adding a new metadata field to `dspace/config/registries/cgiar-types.xml` and restarting DSpace causes the field to get added to the rregistry
- It requires a restart but at least it allows you to manage the registry programmatically
- It's not a very good way to manage the registry, though, as removing one there doesn't cause it to be removed from the registry, and we always restore from database backups so there would never be a scenario when we needed these to be created
- Testing some corrections on CCAFS Phase II flagships (`cg.subject.ccafs`):

```
$ ./fix-metadata-values.py -i ccafs-flagships-feb7.csv -f cg.subject.ccafs -t correct -m 210 -d dspace -u dspace -p fuuu
```

## 2017-02-10

- CCAFS said they want to wait on the flagship updates (`cg.subject.ccafs`) on CGSpace, perhaps for a month or so
- Help Marianne Gadeberg (WLE) with some user permissions as it seems she had previously been using a personal email account, and is now on a CGIAR one
- I manually added her new account to ~25 authorizations that her hold user was on

## 2017-02-14

- Add `SCALING` to ILRI subjects ([#304](https://github.com/ilri/DSpace/pull/304)), as Sisay's attempts were all sloppy
- Cherry pick some patches from the DSpace 5.7 branch:
  - DS-3363 CSV import error says "row", means "column": f7b6c83e991db099003ee4e28ca33d3c7bab48c0
  - DS-3479 avoid adding empty metadata values during import: 329f3b48a6de7fad074d825fd12118f7e181e151
  - [DS-3456] 5x Clarify command line options for statisics import/export tools (#1623): 567ec083c8a94eb2bcc1189816eb4f767745b278
  - [DS-3458]5x Allow Shard Process to Append to an existing repo: 3c8ecb5d1fd69a1dcfee01feed259e80abbb7749
- I still need to test these, especially as the last two which change some stuff with Solr maintenance

## 2017-02-15

- Update rvm on DSpace Test and CGSpace as there was a [security disclosure about versions less than 1.28.0](https://github.com/justinsteven/advisories/blob/master/2017_rvm_cd_command_execution.md)

## 2017-02-16

- Looking at memory info from munin on CGSpace:

![CGSpace meminfo](/cgspace-notes/2017/02/meminfo_phisical-week.png)

- We are using only ~8GB of RAM for applications, and 16GB for caches!
- The Linode machine we're on has 24GB of RAM but only because that's the only instance that had enough disk space for us (384GB)...
- We should probably look into Google Compute Engine or Digital Ocean where we can get more storage without having to follow a linear increase in instance pricing for CPU/memory as well
- Especially because we only use 2 out of 8 CPUs basically:

![CGSpace CPU](/cgspace-notes/2017/02/cpu-week.png)

- Fix issue with duplicate declaration of in atmire-dspace-xmlui `pom.xml` (causing non-fatal warnings during the maven build)
- Experiment with making DSpace generate HTTPS handle links, first a change in dspace.cfg or the site's properties file:

```
handle.canonical.prefix = https://hdl.handle.net/
```

- And then a SQL command to update existing records:

```
dspace=# update metadatavalue set text_value = regexp_replace(text_value, 'http://hdl.handle.net', 'https://hdl.handle.net') where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'uri');
UPDATE 58193
```

- Seems to work fine!
- I noticed a few items that have incorrect DOI links (`dc.identifier.doi`), and after looking in the database I see there are over 100 that are missing the scheme or are just plain wrong:

```
dspace=# select distinct text_value from metadatavalue where resource_type_id=2 and metadata_field_id=220 and text_value not like 'http%://%';
```

- This will replace any that begin with `10.`:

```
dspace=# update metadatavalue set text_value = regexp_replace(text_value, '(^10\..+$)', 'https://dx.doi.org/\1') where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value like '10.%';
```
-												Add notes for 2017-02-07

											
										
										
											2017-02-07 16:07:44 +01:00
+								+++
 								date = "2017-02-07T07:04:52-08:00"
 								author = "Alan Orth"
 								title = "February, 2017"
 								tags = ["Notes"]
 								+++
 								## 2017-02-07
 								- An item was mapped twice erroneously again, so I had to remove one of the mappings manually:
 								```
 								dspace=# select * from collection2item where item_id = '80278';
 								  id   | collection_id | item_id
 								-------+---------------+---------
 |           313 |   80278
 |           313 |   80278
 |          1051 |   80278
 								(3 rows)
 								dspace=# delete from collection2item where id = 92551 and item_id = 80278;
 								DELETE 1
 								```
-												Update notes for 2017-02-07

											
										
										
											2017-02-08 07:11:37 +01:00
+								- Create issue on GitHub to track the addition of CCAFS Phase II project tags ([#301](https://github.com/ilri/DSpace/issues/301))
 								- Looks like we'll be using `cg.identifier.ccafsprojectpii` as the field name
-												Add notes for 2017-02-07

											
										
										
											2017-02-07 16:07:44 +01:00
+								<!--more-->
-												Add notes for 2017-02-08

											
										
										
											2017-02-08 18:31:52 +01:00
 								## 2017-02-08
 								- We also need to rename some of the CCAFS Phase I flagships:
 								  - CLIMATE-SMART AGRICULTURAL PRACTICES → CLIMATE-SMART TECHNOLOGIES AND PRACTICES
 								  - CLIMATE RISK MANAGEMENT → CLIMATE SERVICES AND SAFETY NETS
 								  - LOW EMISSIONS AGRICULTURE → LOW EMISSIONS DEVELOPMENT
 								  - POLICIES AND INSTITUTIONS → PRIORITIES AND POLICIES FOR CSA
 								- The climate risk management one doesn't exist, so I will have to ask Magdalena if they want me to add it to the input forms
-												Update notes for 2017-02-08

											
										
										
											2017-02-09 07:27:37 +01:00
+								- Start testing some nearly 500 author corrections that CCAFS sent me:
 								```
 								$ ./fix-metadata-values.py -i /tmp/CCAFS-Authors-Feb-7.csv -f dc.contributor.author -t 'correct name' -m 3 -d dspace -u dspace -p fuuu
 								```
-												Update notes for 2017-02-09

											
										
										
											2017-02-10 07:36:23 +01:00
 								## 2017-02-09
 								- More work on CCAFS Phase II stuff
 								- Looks like simply adding a new metadata field to `dspace/config/registries/cgiar-types.xml` and restarting DSpace causes the field to get added to the rregistry
 								- It requires a restart but at least it allows you to manage the registry programmatically
 								- It's not a very good way to manage the registry, though, as removing one there doesn't cause it to be removed from the registry, and we always restore from database backups so there would never be a scenario when we needed these to be created
 								- Testing some corrections on CCAFS Phase II flagships (`cg.subject.ccafs`):
 								```
 								$ ./fix-metadata-values.py -i ccafs-flagships-feb7.csv -f cg.subject.ccafs -t correct -m 210 -d dspace -u dspace -p fuuu
 								```
-												Update notes for 2017-02-10

											
										
										
											2017-02-11 00:28:56 +01:00
 								## 2017-02-10
 								- CCAFS said they want to wait on the flagship updates (`cg.subject.ccafs`) on CGSpace, perhaps for a month or so
 								- Help Marianne Gadeberg (WLE) with some user permissions as it seems she had previously been using a personal email account, and is now on a CGIAR one
 								- I manually added her new account to ~25 authorizations that her hold user was on
-												Add notes for 2017-02-14

											
										
										
											2017-02-14 12:03:09 +01:00
 								## 2017-02-14
 								- Add `SCALING` to ILRI subjects ([#304](https://github.com/ilri/DSpace/pull/304)), as Sisay's attempts were all sloppy
 								- Cherry pick some patches from the DSpace 5.7 branch:
 								  - DS-3363 CSV import error says "row", means "column": f7b6c83e991db099003ee4e28ca33d3c7bab48c0
 								  - DS-3479 avoid adding empty metadata values during import: 329f3b48a6de7fad074d825fd12118f7e181e151
 								  - [DS-3456] 5x Clarify command line options for statisics import/export tools (#1623): 567ec083c8a94eb2bcc1189816eb4f767745b278
 								  - [DS-3458]5x Allow Shard Process to Append to an existing repo: 3c8ecb5d1fd69a1dcfee01feed259e80abbb7749
 								- I still need to test these, especially as the last two which change some stuff with Solr maintenance
-												Add notes for 2017-02-15

											
										
										
											2017-02-15 15:04:55 +01:00
 								## 2017-02-15
-												Add notes for 2017-02-16

											
										
										
											2017-02-16 10:46:05 +01:00
+								- Update rvm on DSpace Test and CGSpace as there was a [security disclosure about versions less than 1.28.0](https://github.com/justinsteven/advisories/blob/master/2017_rvm_cd_command_execution.md)
 								## 2017-02-16
 								- Looking at memory info from munin on CGSpace:
 								![CGSpace meminfo](/cgspace-notes/2017/02/meminfo_phisical-week.png)
 								- We are using only ~8GB of RAM for applications, and 16GB for caches!
 								- The Linode machine we're on has 24GB of RAM but only because that's the only instance that had enough disk space for us (384GB)...
 								- We should probably look into Google Compute Engine or Digital Ocean where we can get more storage without having to follow a linear increase in instance pricing for CPU/memory as well
 								- Especially because we only use 2 out of 8 CPUs basically:
 								![CGSpace CPU](/cgspace-notes/2017/02/cpu-week.png)
-												Update notes for 2017-02-16

											
										
										
											2017-02-16 13:20:22 +01:00
 								- Fix issue with duplicate declaration of in atmire-dspace-xmlui `pom.xml` (causing non-fatal warnings during the maven build)
 								- Experiment with making DSpace generate HTTPS handle links, first a change in dspace.cfg or the site's properties file:
 								```
 								handle.canonical.prefix = https://hdl.handle.net/
 								```
 								- And then a SQL command to update existing records:
 								```
 								dspace=# update metadatavalue set text_value = regexp_replace(text_value, 'http://hdl.handle.net', 'https://hdl.handle.net') where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'uri');
 								UPDATE 58193
 								```
 								- Seems to work fine!
-												Update notes for 2017-02-16

											
										
										
											2017-02-16 18:31:01 +01:00
+								- I noticed a few items that have incorrect DOI links (`dc.identifier.doi`), and after looking in the database I see there are over 100 that are missing the scheme or are just plain wrong:
 								```
 								dspace=# select distinct text_value from metadatavalue where resource_type_id=2 and metadata_field_id=220 and text_value not like 'http%://%';
 								```
 								- This will replace any that begin with `10.`:
 								```
 								dspace=# update metadatavalue set text_value = regexp_replace(text_value, '(^10\..+$)', 'https://dx.doi.org/\1') where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value like '10.%';
 								```