Posts on CGSpace Notes

March, 2017

Wed, 01 Mar 2017 17:08:52 +0200

2017-03-01

Run the 279 CIAT author corrections on CGSpace

2017-03-02

Skype with Michael and Peter, discussing moving the CGIAR Library to CGSpace
CGIAR people possibly open to moving content, redirecting library.cgiar.org to CGSpace and letting CGSpace resolve their handles
They might come in at the top level in one “CGIAR System” community, or with several communities
I need to spend a bit of time looking at the multiple handle support in DSpace and see if new content can be minted in both handles, or just one?
Need to send Peter and Michael some notes about this in a few days
Also, need to consider talking to Atmire about hiring them to bring ORCiD metadata to REST / OAI
Filed an issue on DSpace issue tracker for the filter-media bug that causes it to process JPGs even when limiting to the PDF thumbnail plugin: DS-3516
Discovered that the ImageMagic filter-media plugin creates JPG thumbnails with the CMYK colorspace when the source PDF is using CMYK
Interestingly, it seems DSpace 4.x’s thumbnails were sRGB, but forcing regeneration using DSpace 5.x’s ImageMagick plugin creates CMYK JPGs if the source PDF was CMYK (see ¹⁰⁵⁶⁸⁄₅₁₉₉₉):

$ identify ~/Desktop/alc_contrastes_desafios.jpg
/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000

This results in discolored thumbnails when compared to the original PDF, for example sRGB and CMYK:

I filed an issue for the color space thing: DS-3517

2017-03-03

I created a patch for DS-3517 and made a pull request against upstream dspace-5_x: https://github.com/DSpace/DSpace/pull/1669
Looks like -colorspace sRGB alone isn’t enough, we need to use profiles:

$ convert alc_contrastes_desafios.pdf\[0\] -profile /opt/brew/Cellar/ghostscript/9.20/share/ghostscript/9.20/iccprofiles/default_cmyk.icc -thumbnail 300x300 -flatten -profile /opt/brew/Cellar/ghostscript/9.20/share/ghostscript/9.20/iccprofiles/default_rgb.icc alc_contrastes_desafios.pdf.jpg

This reads the input file, applies the CMYK profile, applies the RGB profile, then writes the file
Note that you should set the first profile immediately after the input file
Also, it is better to use profiles than setting -colorspace
This is a great resource describing the color stuff: http://www.imagemagick.org/Usage/formats/#profiles
Somehow we need to detect the color system being used by the input file and handle each case differently (with profiles)
This is trivial with identify (even by the Java ImageMagick API):

$ identify -format '%r\n' alc_contrastes_desafios.pdf\[0\]
DirectClass CMYK
$ identify -format '%r\n' Africa\ group\ of\ negotiators.pdf\[0\]
DirectClass sRGB Alpha

2017-03-04

Spent more time looking at the ImageMagick CMYK issue
The default_cmyk.icc and default_rgb.icc files are both part of the Ghostscript GPL distribution, but according to DSpace’s LICENSES_THIRD_PARTY file, DSpace doesn’t allow distribution of dependencies that are licensed solely under the GPL
So this issue is kinda pointless now, as the ICC profiles are absolutely necessary to make a meaningful CMYK→sRGB conversion

2017-03-05

Look into helping developers from landportal.info with a query for items related to LAND on the REST API
They want something like the items that are returned by the general “LAND” query in the search interface, but we cannot do that
We can only return specific results for metadata fields, like:

$ curl -s -H "accept: application/json" -H "Content-Type: application/json" -X POST "https://dspacetest.cgiar.org/rest/items/find-by-metadata-field" -d '{"key": "cg.subject.ilri","value": "LAND REFORM", "language": null}' | json_pp

But there are hundreds of combinations of fields and values (like dc.subject and all the center subjects), and we can’t use wildcards in REST!
Reading about enabling multiple handle prefixes in DSpace
There is a mailing list thread from 2011 about it: http://dspace.2283337.n4.nabble.com/Multiple-handle-prefixes-merged-DSpace-instances-td3427192.html
And a comment from Atmire’s Bram about it on the DSpace wiki: https://wiki.duraspace.org/display/DSDOC5x/Installing+DSpace?focusedCommentId=78163296#comment-78163296
Bram mentions an undocumented configuration option handle.plugin.checknameauthority, but I noticed another one in dspace.cfg:

# List any additional prefixes that need to be managed by this handle server
# (as for examle handle prefix coming from old dspace repository merged in
# that repository)
# handle.additional.prefixes = prefix1[, prefix2]

Because of this I noticed that our Handle server’s config.dct was potentially misconfigured!
We had some default values still present:

"300:0.NA/YOUR_NAMING_AUTHORITY"

I’ve changed them to the following and restarted the handle server:

"300:0.NA/10568"

In looking at all the configs I just noticed that we are not providing a DOI in the Google-specific metadata crosswalk
From dspace/config/crosswalks/google-metadata.properties:

google.citation_doi = cg.identifier.doi

This works, and makes DSpace output the following metadata on the item view page:

<meta content="https://dx.doi.org/10.1186/s13059-017-1153-y" name="citation_doi">

Submitted and merged pull request for this: https://github.com/ilri/DSpace/pull/305
Submit pull request to set the author separator for XMLUI item lists to a semicolon instead of “,”: https://github.com/ilri/DSpace/pull/306
I want to show it briefly to Abenet and Peter to get feedback

2017-03-06

Someone on the mailing list said that handle.plugin.checknameauthority should be false if we’re using multiple handle prefixes

2017-03-07

I set up a top-level community as a test for the CGIAR Library and imported one item with the the 10947 handle prefix
When testing the Handle resolver locally it shows the item to be on the local repository
So this seems to work, with the following caveats:
- New items will have the default handle
- Communities and collections will have the default handle
- Only items imported manually can have the other handles
I need to talk to Michael and Peter to share the news, and discuss the structure of their community(s) and try some actual test data
We’ll need to do some data cleaning to make sure they are using the same fields we are, like dc.type and cg.identifier.status
Another thing is that the import process creates new dc.date.accessioned and dc.date.available fields, so we end up with duplicates (is it important to preserve the originals for these?)
Report DS-3520 issue to Atmire

2017-03-08

Merge the author separator changes to 5_x-prod, as everyone has responded positively about it, and it’s the default in Mirage2 afterall!
Cherry pick the commons-collections patch from DSpace’s dspace-5_x branch to address DS-3520: https://jira.duraspace.org/browse/DS-3520

2017-03-09

Export list of sponsors so Peter can clean it up:

dspace=# \copy (select text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'description' and qualifier = 'sponsorship') group by text_value order by count desc) to /tmp/sponsorship.csv with csv;
COPY 285

2017-03-12

Test the sponsorship fixes and deletes from Peter:

$ ./fix-metadata-values.py -i Investors-Fix-51.csv -f dc.description.sponsorship -t Action -m 29 -d dspace -u dspace -p fuuuu
$ ./delete-metadata-values.py -i Investors-Delete-121.csv -f dc.description.sponsorship -m 29 -d dspace -u dspace -p fuuu

Generate a new list of unique sponsors so we can update the controlled vocabulary:

dspace=# \copy (select distinct text_value from metadatavalue where resource_type_id=2 and metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'description' and qualifier = 'sponsorship')) to /tmp/sponsorship.csv with csv;

Pull request for controlled vocabulary if Peter approves: https://github.com/ilri/DSpace/pull/308
Review Sisay’s roots, tubers, and bananas (RTB) theme, which still needs some fixes to work properly: https://github.com/ilri/DSpace/pull/307
Created an issue to track the progress on the Livestock CRP theme: https://github.com/ilri/DSpace/issues/309
Created a basic theme for the Livestock CRP community

2017-03-15

Merge pull request for controlled vocabulary updates for sponsor: https://github.com/ilri/DSpace/pull/308
Merge pull request for Livestock CRP theme: https://github.com/ilri/DSpace/issues/309
Create pull request for PABRA subjects (waiting for confirmation from Abenet before merging): https://github.com/ilri/DSpace/pull/310
Create pull request for CCAFS Phase II migrations (waiting for confirmation from CCAFS people): https://github.com/ilri/DSpace/pull/311
I also need to ask if either of these new fields need to be added to Discovery facets, search, and Atmire modules
Run all system updates on DSpace Test and re-deploy CGSpace

2017-03-16

Merge pull request for PABRA subjects: https://github.com/ilri/DSpace/pull/310
Abenet and Peter say we can add them to Discovery, Atmire modules, etc, but I might not have time to do it now
Help Sisay with RTB theme again
Remove ICARDA subject from Discovery sidebar facets: https://github.com/ilri/DSpace/pull/312
Remove ICARDA subject from Browse and item submission form: https://github.com/ilri/DSpace/pull/313
Merge the CCAFS Phase II changes but hold off on doing the flagship metadata updates until Macaroni Bros gets their importer updated
Deploy latest changes and investor fixes/deletions on CGSpace
Run system updates on CGSpace and reboot server

2017-03-20

Create basic XMLUI theme for PABRA community: https://github.com/ilri/DSpace/pull/315

February, 2017

Tue, 07 Feb 2017 07:04:52 -0800

2017-02-07

An item was mapped twice erroneously again, so I had to remove one of the mappings manually:

dspace=# select * from collection2item where item_id = '80278';
  id   | collection_id | item_id
-------+---------------+---------
 92551 |           313 |   80278
 92550 |           313 |   80278
 90774 |          1051 |   80278
(3 rows)
dspace=# delete from collection2item where id = 92551 and item_id = 80278;
DELETE 1

Create issue on GitHub to track the addition of CCAFS Phase II project tags (#301)
Looks like we’ll be using cg.identifier.ccafsprojectpii as the field name

2017-02-08

We also need to rename some of the CCAFS Phase I flagships:
- CLIMATE-SMART AGRICULTURAL PRACTICES → CLIMATE-SMART TECHNOLOGIES AND PRACTICES
- CLIMATE RISK MANAGEMENT → CLIMATE SERVICES AND SAFETY NETS
- LOW EMISSIONS AGRICULTURE → LOW EMISSIONS DEVELOPMENT
- POLICIES AND INSTITUTIONS → PRIORITIES AND POLICIES FOR CSA
The climate risk management one doesn’t exist, so I will have to ask Magdalena if they want me to add it to the input forms
Start testing some nearly 500 author corrections that CCAFS sent me:

$ ./fix-metadata-values.py -i /tmp/CCAFS-Authors-Feb-7.csv -f dc.contributor.author -t 'correct name' -m 3 -d dspace -u dspace -p fuuu

2017-02-09

More work on CCAFS Phase II stuff
Looks like simply adding a new metadata field to dspace/config/registries/cgiar-types.xml and restarting DSpace causes the field to get added to the rregistry
It requires a restart but at least it allows you to manage the registry programmatically
It’s not a very good way to manage the registry, though, as removing one there doesn’t cause it to be removed from the registry, and we always restore from database backups so there would never be a scenario when we needed these to be created
Testing some corrections on CCAFS Phase II flagships (cg.subject.ccafs):

$ ./fix-metadata-values.py -i ccafs-flagships-feb7.csv -f cg.subject.ccafs -t correct -m 210 -d dspace -u dspace -p fuuu

2017-02-10

CCAFS said they want to wait on the flagship updates (cg.subject.ccafs) on CGSpace, perhaps for a month or so
Help Marianne Gadeberg (WLE) with some user permissions as it seems she had previously been using a personal email account, and is now on a CGIAR one
I manually added her new account to ~25 authorizations that her hold user was on

2017-02-14

Add SCALING to ILRI subjects (#304), as Sisay’s attempts were all sloppy
Cherry pick some patches from the DSpace 5.7 branch:
- DS-3363 CSV import error says “row”, means “column”: f7b6c83e991db099003ee4e28ca33d3c7bab48c0
- DS-3479 avoid adding empty metadata values during import: 329f3b48a6de7fad074d825fd12118f7e181e151
- [DS-3456] 5x Clarify command line options for statisics import/export tools (#1623): 567ec083c8a94eb2bcc1189816eb4f767745b278
- [DS-3458]5x Allow Shard Process to Append to an existing repo: 3c8ecb5d1fd69a1dcfee01feed259e80abbb7749
I still need to test these, especially as the last two which change some stuff with Solr maintenance

2017-02-15

Update rvm on DSpace Test and CGSpace as there was a security disclosure about versions less than 1.28.0

2017-02-16

Looking at memory info from munin on CGSpace:

We are using only ~8GB of RAM for applications, and 16GB for caches!
The Linode machine we’re on has 24GB of RAM but only because that’s the only instance that had enough disk space for us (384GB)…
We should probably look into Google Compute Engine or Digital Ocean where we can get more storage without having to follow a linear increase in instance pricing for CPU/memory as well
Especially because we only use 2 out of 8 CPUs basically:

Fix issue with duplicate declaration of in atmire-dspace-xmlui pom.xml (causing non-fatal warnings during the maven build)
Experiment with making DSpace generate HTTPS handle links, first a change in dspace.cfg or the site’s properties file:

handle.canonical.prefix = https://hdl.handle.net/

And then a SQL command to update existing records:

dspace=# update metadatavalue set text_value = regexp_replace(text_value, 'http://hdl.handle.net', 'https://hdl.handle.net') where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'uri');
UPDATE 58193

Seems to work fine!
I noticed a few items that have incorrect DOI links (dc.identifier.doi), and after looking in the database I see there are over 100 that are missing the scheme or are just plain wrong:

dspace=# select distinct text_value from metadatavalue where resource_type_id=2 and metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value not like 'http%://%';

This will replace any that begin with 10. and change them to https://dx.doi.org/10.:

dspace=# update metadatavalue set text_value = regexp_replace(text_value, '(^10\..+$)', 'https://dx.doi.org/\1') where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value like '10.%';

This will get any that begin with doi:10. and change them to https://dx.doi.org/10.x:

dspace=# update metadatavalue set text_value = regexp_replace(text_value, '^doi:(10\..+$)', 'https://dx.doi.org/\1') where resource_type_id=2 and metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value like 'doi:10%';

Fix DOIs like dx.doi.org/10. to be https://dx.doi.org/10.:

dspace=# update metadatavalue set text_value = regexp_replace(text_value, '(^dx.doi.org/.+$)', 'https://dx.doi.org/\1') where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value like 'dx.doi.org/%';

Fix DOIs like http//:

dspace=# update metadatavalue set text_value = regexp_replace(text_value, '^http//(dx.doi.org/.+$)', 'https://dx.doi.org/\1') where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value like 'http//%';

Fix DOIs like dx.doi.org./:

dspace=# update metadatavalue set text_value = regexp_replace(text_value, '(^dx.doi.org\./.+$)', 'https://dx.doi.org/\1') where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value like 'dx.doi.org./%'

Delete some invalid DOIs:

dspace=# delete from metadatavalue where resource_type_id=2 and metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value in ('DOI','CPWF Mekong','Bulawayo, Zimbabwe','bb');

Fix some other random outliers:

dspace=# update metadatavalue set text_value = 'https://dx.doi.org/10.1016/j.aquaculture.2015.09.003' where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value = 'http:/dx.doi.org/10.1016/j.aquaculture.2015.09.003';
dspace=# update metadatavalue set text_value = 'https://dx.doi.org/10.5337/2016.200' where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value = 'doi: https://dx.doi.org/10.5337/2016.200';
dspace=# update metadatavalue set text_value = 'https://dx.doi.org/doi:10.1371/journal.pone.0062898' where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value = 'Http://dx.doi.org/doi:10.1371/journal.pone.0062898';
dspace=# update metadatavalue set text_value = 'https://dx.doi.10.1016/j.cosust.2013.11.012' where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value = 'http:dx.doi.10.1016/j.cosust.2013.11.012';
dspace=# update metadatavalue set text_value = 'https://dx.doi.org/10.1080/03632415.2014.883570' where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value = 'org/10.1080/03632415.2014.883570';
dspace=# update metadatavalue set text_value = 'https://dx.doi.org/10.15446/agron.colomb.v32n3.46052' where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value = 'Doi: 10.15446/agron.colomb.v32n3.46052';

And do another round of http:// → https:// cleanups:

dspace=# update metadatavalue set text_value = regexp_replace(text_value, 'http://dx.doi.org', 'https://dx.doi.org') where resource_type_id=2 and metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value like 'http://dx.doi.org%';

Run all DOI corrections on CGSpace
Something to think about here is to write a Curation Task in Java to do these sanity checks / corrections every night
Then we could add a cron job for them and run them from the command line like:

[dspace]/bin/dspace curate -t noop -i 10568/79891

2017-02-20

Run all system updates on DSpace Test and reboot the server
Run CCAFS author corrections on DSpace Test and CGSpace and force a full discovery reindex
Fix label of CCAFS subjects in Atmire Listings and Reports module
Help Sisay with SQL commands
Help Paola from CCAFS with the Atmire Listings and Reports module
Testing the fix-metadata-values.py script on macOS and it seems like we don’t need to use .encode('utf-8') anymore when printing strings to the screen
It seems this might have only been a temporary problem, as both Python 3.5.2 and 3.6.0 are able to print the problematic string “Entwicklung & Ländlicher Raum” without the encode() call, but print it as a bytes when it is used:

$ python
Python 3.6.0 (default, Dec 25 2016, 17:30:53)
>>> print('Entwicklung & Ländlicher Raum')
Entwicklung & Ländlicher Raum
>>> print('Entwicklung & Ländlicher Raum'.encode())
b'Entwicklung & L\xc3\xa4ndlicher Raum'

So for now I will remove the encode call from the script (though it was never used on the versions on the Linux hosts), leading me to believe it really was a temporary problem, perhaps due to macOS or the Python build I was using.

2017-02-21

Testing regenerating PDF thumbnails, like I started in 2016-11
It seems there is a bug in filter-media that causes it to process formats that aren’t part of its configuration:

$ [dspace]/bin/dspace filter-media -f -i 10568/16856 -p "ImageMagick PDF Thumbnail"
File: earlywinproposal_esa_postharvest.pdf.jpg
FILTERED: bitstream 13787 (item: 10568/16881) and created 'earlywinproposal_esa_postharvest.pdf.jpg'
File: postHarvest.jpg.jpg
FILTERED: bitstream 16524 (item: 10568/24655) and created 'postHarvest.jpg.jpg'

According to dspace.cfg the ImageMagick PDF Thumbnail plugin should only process PDFs:

filter.org.dspace.app.mediafilter.ImageMagickImageThumbnailFilter.inputFormats = BMP, GIF, image/png, JPG, TIFF, JPEG, JPEG 2000
filter.org.dspace.app.mediafilter.ImageMagickPdfThumbnailFilter.inputFormats = Adobe PDF

I’ve sent a message to the mailing list and might file a Jira issue
Ask Atmire about the failed interpolation of the dspace.internalUrl variable in atmire-cua.cfg

2017-02-22

Atmire said I can add dspace.internalUrl to my build properties and the error will go away
It should be the local URL for accessing Tomcat from the server’s own perspective, ie: http://localhost:8080

2017-02-26

Find all fields with “http://hdl.handle.net" values (most are in dc.identifier.uri, but some are in other URL-related fields like cg.link.reference, cg.identifier.dataurl, and cg.identifier.url):

dspace=# select distinct metadata_field_id from metadatavalue where resource_type_id=2 and text_value like 'http://hdl.handle.net%';
dspace=# update metadatavalue set text_value = regexp_replace(text_value, 'http://hdl.handle.net', 'https://hdl.handle.net') where resource_type_id=2 and metadata_field_id IN (25, 113, 179, 219, 220, 223) and text_value like 'http://hdl.handle.net%';
UPDATE 58633

This works but I’m thinking I’ll wait on the replacement as there are perhaps some other places that rely on http://hdl.handle.net (grep the code, it’s scary how many things are hard coded)
Send message to dspace-tech mailing list with concerns about this

2017-02-27

LDAP users cannot log in today, looks to be an issue with CGIAR’s LDAP server:

$ openssl s_client -connect svcgroot2.cgiarad.org:3269
CONNECTED(00000003)
depth=0 CN = SVCGROOT2.CGIARAD.ORG
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = SVCGROOT2.CGIARAD.ORG
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/CN=SVCGROOT2.CGIARAD.ORG
   i:/CN=CGIARAD-RDWA-CA
---

For some reason it is now signed by a private certificate authority
This error seems to have started on 2017-02-25:

$ grep -c "unable to find valid certification path" [dspace]/log/dspace.log.2017-02-*
[dspace]/log/dspace.log.2017-02-01:0
[dspace]/log/dspace.log.2017-02-02:0
[dspace]/log/dspace.log.2017-02-03:0
[dspace]/log/dspace.log.2017-02-04:0
[dspace]/log/dspace.log.2017-02-05:0
[dspace]/log/dspace.log.2017-02-06:0
[dspace]/log/dspace.log.2017-02-07:0
[dspace]/log/dspace.log.2017-02-08:0
[dspace]/log/dspace.log.2017-02-09:0
[dspace]/log/dspace.log.2017-02-10:0
[dspace]/log/dspace.log.2017-02-11:0
[dspace]/log/dspace.log.2017-02-12:0
[dspace]/log/dspace.log.2017-02-13:0
[dspace]/log/dspace.log.2017-02-14:0
[dspace]/log/dspace.log.2017-02-15:0
[dspace]/log/dspace.log.2017-02-16:0
[dspace]/log/dspace.log.2017-02-17:0
[dspace]/log/dspace.log.2017-02-18:0
[dspace]/log/dspace.log.2017-02-19:0
[dspace]/log/dspace.log.2017-02-20:0
[dspace]/log/dspace.log.2017-02-21:0
[dspace]/log/dspace.log.2017-02-22:0
[dspace]/log/dspace.log.2017-02-23:0
[dspace]/log/dspace.log.2017-02-24:0
[dspace]/log/dspace.log.2017-02-25:7
[dspace]/log/dspace.log.2017-02-26:8
[dspace]/log/dspace.log.2017-02-27:90

Also, it seems that we need to use a different user for LDAP binds, as we’re still using the temporary one from the root migration, so maybe we can go back to the previous user we were using
So it looks like the certificate is invalid AND the bind users we had been using were deleted
Biruk Debebe recreated the bind user and now we are just waiting for CGNET to update their certificates
Regarding the filter-media issue I found earlier, it seems that the ImageMagick PDF plugin will also process JPGs if they are in the “Content Files” (aka ORIGINAL) bundle
The problem likely lies in the logic of ImageMagickThumbnailFilter.java, as ImageMagickPdfThumbnailFilter.java extends it
Run CIAT corrections on CGSpace

dspace=# update metadatavalue set authority='3026b1de-9302-4f3e-85ab-ef48da024eb2', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value = 'International Center for Tropical Agriculture';

CGNET has fixed the certificate chain on their LDAP server
Redeploy CGSpace and DSpace Test to on latest 5_x-prod branch with fixes for LDAP bind user
Run all system updates on CGSpace server and reboot

2017-02-28

After running the CIAT corrections and updating the Discovery and authority indexes, there is still no change in the number of items listed for CIAT in Discovery
Ah, this is probably because some items have the International Center for Tropical Agriculture author twice, which I first noticed in 2016-12 but couldn’t figure out how to fix
I think I can do it by first exporting all metadatavalues that have the author International Center for Tropical Agriculture

dspace=# \copy (select resource_id, metadata_value_id from metadatavalue where resource_type_id=2 and metadata_field_id=3 and text_value='International Center for Tropical Agriculture') to /tmp/ciat.csv with csv;
COPY 1968

And then use awk to print the duplicate lines to a separate file:

$ awk -F',' 'seen[$1]++' /tmp/ciat.csv > /tmp/ciat-dupes.csv

From that file I can create a list of 279 deletes and put them in a batch script like:

delete from metadatavalue where resource_type_id=2 and metadata_field_id=3 and metadata_value_id=2742061;

January, 2017

Mon, 02 Jan 2017 10:43:00 +0300

2017-01-02

I checked to see if the Solr sharding task that is supposed to run on January 1st had run and saw there was an error
I tested on DSpace Test as well and it doesn’t work there either
I asked on the dspace-tech mailing list because it seems to be broken, and actually now I’m not sure if we’ve ever had the sharding task run successfully over all these years

2017-01-04

I tried to shard my local dev instance and it fails the same way:

$ JAVA_OPTS="-Xms768m -Xmx768m -Dfile.encoding=UTF-8" ~/dspace/bin/dspace stats-util -s
Moving: 9318 into core statistics-2016
Exception: IOException occured when talking to server at: http://localhost:8081/solr//statistics-2016
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://localhost:8081/solr//statistics-2016
        at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:566)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
        at org.dspace.statistics.SolrLogger.shardSolrIndex(SourceFile:2291)
        at org.dspace.statistics.util.StatisticsClient.main(StatisticsClient.java:106)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:226)
        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:78)
Caused by: org.apache.http.client.ClientProtocolException
        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:867)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:448)
        ... 10 more
Caused by: org.apache.http.client.NonRepeatableRequestException: Cannot retry request with a non-repeatable request entity.  The cause lists the reason the original request failed.
        at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:659)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
        ... 14 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
        at org.apache.http.impl.io.AbstractSessionOutputBuffer.write(AbstractSessionOutputBuffer.java:181)
        at org.apache.http.impl.io.ChunkedOutputStream.flushCacheWithAppend(ChunkedOutputStream.java:124)
        at org.apache.http.impl.io.ChunkedOutputStream.write(ChunkedOutputStream.java:181)
        at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:132)
        at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:89)
        at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108)
        at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:117)
        at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:265)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestEntity(ManagedClientConnectionImpl.java:203)
        at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:236)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:121)
        at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
        ... 16 more

And the DSpace log shows:

2017-01-04 22:39:05,412 INFO  org.dspace.statistics.SolrLogger @ Created core with name: statistics-2016
2017-01-04 22:39:05,412 INFO  org.dspace.statistics.SolrLogger @ Moving: 9318 records into core statistics-2016
2017-01-04 22:39:07,310 INFO  org.apache.http.impl.client.SystemDefaultHttpClient @ I/O exception (java.net.SocketException) caught when processing request to {}->http://localhost:8081: Broken pipe (Write failed)
2017-01-04 22:39:07,310 INFO  org.apache.http.impl.client.SystemDefaultHttpClient @ Retrying request to {}->http://localhost:8081

Despite failing instantly, a statistics-2016 directory was created, but it only has a data dir (no conf)
The Tomcat access logs show more:

127.0.0.1 - - [04/Jan/2017:22:39:05 +0200] "GET /solr/statistics/select?q=type%3A2+AND+id%3A1&wt=javabin&version=2 HTTP/1.1" 200 107
127.0.0.1 - - [04/Jan/2017:22:39:05 +0200] "GET /solr/statistics/select?q=*%3A*&rows=0&facet=true&facet.range=time&facet.range.start=NOW%2FYEAR-17YEARS&facet.range.end=NOW%2FYEAR%2B0YEARS&facet.range.gap=%2B1YEAR&facet.mincount=1&wt=javabin&version=2 HTTP/1.1" 200 423
127.0.0.1 - - [04/Jan/2017:22:39:05 +0200] "GET /solr/admin/cores?action=STATUS&core=statistics-2016&indexInfo=true&wt=javabin&version=2 HTTP/1.1" 200 77
127.0.0.1 - - [04/Jan/2017:22:39:05 +0200] "GET /solr/admin/cores?action=CREATE&name=statistics-2016&instanceDir=statistics&dataDir=%2FUsers%2Faorth%2Fdspace%2Fsolr%2Fstatistics-2016%2Fdata&wt=javabin&version=2 HTTP/1.1" 200 63
127.0.0.1 - - [04/Jan/2017:22:39:07 +0200] "GET /solr/statistics/select?csv.mv.separator=%7C&q=*%3A*&fq=time%3A%28%5B2016%5C-01%5C-01T00%5C%3A00%5C%3A00Z+TO+2017%5C-01%5C-01T00%5C%3A00%5C%3A00Z%5D+NOT+2017%5C-01%5C-01T00%5C%3A00%5C%3A00Z%29&rows=10000&wt=csv HTTP/1.1" 200 4359517
127.0.0.1 - - [04/Jan/2017:22:39:07 +0200] "GET /solr/statistics/admin/luke?show=schema&wt=javabin&version=2 HTTP/1.1" 200 16248
127.0.0.1 - - [04/Jan/2017:22:39:07 +0200] "POST /solr//statistics-2016/update/csv?commit=true&softCommit=false&waitSearcher=true&f.previousWorkflowStep.split=true&f.previousWorkflowStep.separator=%7C&f.previousWorkflowStep.encapsulator=%22&f.actingGroupId.split=true&f.actingGroupId.separator=%7C&f.actingGroupId.encapsulator=%22&f.containerCommunity.split=true&f.containerCommunity.separator=%7C&f.containerCommunity.encapsulator=%22&f.range.split=true&f.range.separator=%7C&f.range.encapsulator=%22&f.containerItem.split=true&f.containerItem.separator=%7C&f.containerItem.encapsulator=%22&f.p_communities_map.split=true&f.p_communities_map.separator=%7C&f.p_communities_map.encapsulator=%22&f.ngram_query_search.split=true&f.ngram_query_search.separator=%7C&f.ngram_query_search.encapsulator=%22&f.containerBitstream.split=true&f.containerBitstream.separator=%7C&f.containerBitstream.encapsulator=%22&f.owningItem.split=true&f.owningItem.separator=%7C&f.owningItem.encapsulator=%22&f.actingGroupParentId.split=true&f.actingGroupParentId.separator=%7C&f.actingGroupParentId.encapsulator=%22&f.text.split=true&f.text.separator=%7C&f.text.encapsulator=%22&f.simple_query_search.split=true&f.simple_query_search.separator=%7C&f.simple_query_search.encapsulator=%22&f.owningComm.split=true&f.owningComm.separator=%7C&f.owningComm.encapsulator=%22&f.owner.split=true&f.owner.separator=%7C&f.owner.encapsulator=%22&f.filterquery.split=true&f.filterquery.separator=%7C&f.filterquery.encapsulator=%22&f.p_group_map.split=true&f.p_group_map.separator=%7C&f.p_group_map.encapsulator=%22&f.actorMemberGroupId.split=true&f.actorMemberGroupId.separator=%7C&f.actorMemberGroupId.encapsulator=%22&f.bitstreamId.split=true&f.bitstreamId.separator=%7C&f.bitstreamId.encapsulator=%22&f.group_name.split=true&f.group_name.separator=%7C&f.group_name.encapsulator=%22&f.p_communities_name.split=true&f.p_communities_name.separator=%7C&f.p_communities_name.encapsulator=%22&f.query.split=true&f.query.separator=%7C&f.query.encapsulator=%22&f.workflowStep.split=true&f.workflowStep.separator=%7C&f.workflowStep.encapsulator=%22&f.containerCollection.split=true&f.containerCollection.separator=%7C&f.containerCollection.encapsulator=%22&f.complete_query_search.split=true&f.complete_query_search.separator=%7C&f.complete_query_search.encapsulator=%22&f.p_communities_id.split=true&f.p_communities_id.separator=%7C&f.p_communities_id.encapsulator=%22&f.rangeDescription.split=true&f.rangeDescription.separator=%7C&f.rangeDescription.encapsulator=%22&f.group_id.split=true&f.group_id.separator=%7C&f.group_id.encapsulator=%22&f.bundleName.split=true&f.bundleName.separator=%7C&f.bundleName.encapsulator=%22&f.ngram_simplequery_search.split=true&f.ngram_simplequery_search.separator=%7C&f.ngram_simplequery_search.encapsulator=%22&f.group_map.split=true&f.group_map.separator=%7C&f.group_map.encapsulator=%22&f.owningColl.split=true&f.owningColl.separator=%7C&f.owningColl.encapsulator=%22&f.p_group_id.split=true&f.p_group_id.separator=%7C&f.p_group_id.encapsulator=%22&f.p_group_name.split=true&f.p_group_name.separator=%7C&f.p_group_name.encapsulator=%22&wt=javabin&version=2 HTTP/1.1" 409 156
127.0.0.1 - - [04/Jan/2017:22:44:00 +0200] "POST /solr/datatables/update?wt=javabin&version=2 HTTP/1.1" 200 41
127.0.0.1 - - [04/Jan/2017:22:44:00 +0200] "POST /solr/datatables/update HTTP/1.1" 200 40

Very interesting… it creates the core and then fails somehow

2017-01-08

Put Sisay’s item-view.xsl code to show mapped collections on CGSpace (#295)

2017-01-09

A user wrote to tell me that the new display of an item’s mappings had a crazy bug for at least one item: https://cgspace.cgiar.org/handle/10568/78596
She said she only mapped it once, but it appears to be mapped 184 times

2017-01-10

I tried to clean up the duplicate mappings by exporting the item’s metadata to CSV, editing, and re-importing, but DSpace said “no changes were detected”
I’ve asked on the dspace-tech mailing list to see if anyone can help
I found an old post on the mailing list discussing a similar issue, and listing some SQL commands that might help
For example, this shows 186 mappings for the item, the first three of which are real:

dspace=#  select * from collection2item where item_id = '80596';

Then I deleted the others:

dspace=# delete from collection2item where item_id = '80596' and id not in (90792, 90806, 90807);

And in the item view it now shows the correct mappings
I will have to ask the DSpace people if this is a valid approach
Finish looking at the Journal Title corrections of the top 500 Journal Titles so we can make a controlled vocabulary from it

2017-01-11

Maria found another item with duplicate mappings: https://cgspace.cgiar.org/handle/10568/78658
Error in fix-metadata-values.py when it tries to print the value for Entwicklung & Ländlicher Raum:

Traceback (most recent call last):
  File "./fix-metadata-values.py", line 80, in <module>
    print("Fixing {} occurences of: {}".format(records_to_fix, record[0]))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 15: ordinal not in range(128)

Seems we need to encode as UTF-8 before printing to screen, ie:

print("Fixing {} occurences of: {}".format(records_to_fix, record[0].encode('utf-8')))

See: http://stackoverflow.com/a/36427358/487333
I’m actually not sure if we need to encode() the strings to UTF-8 before writing them to the database… I’ve never had this issue before
Now back to cleaning up some journal titles so we can make the controlled vocabulary:

$ ./fix-metadata-values.py -i /tmp/fix-27-journal-titles.csv -f dc.source -t correct -m 55 -d dspace -u dspace -p 'fuuu'

Now get the top 500 journal titles:

dspace-# \copy (select distinct text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=55 group by text_value order by count desc limit 500) to /tmp/journal-titles.csv with csv;

The values are a bit dirty and outdated, since the file I had given to Abenet and Peter was from November
I will have to go through these and fix some more before making the controlled vocabulary
Added 30 more corrections or so, now there are 49 total and I’ll have to get the top 500 after applying them

2017-01-13

Add FOOD SYSTEMS to CIAT subjects, waiting to merge: https://github.com/ilri/DSpace/pull/296

2017-01-16

Fix the two items Maria found with duplicate mappings with this script:

/* 184 in correct mappings: https://cgspace.cgiar.org/handle/10568/78596 */
delete from collection2item where item_id = '80596' and id not in (90792, 90806, 90807);
/* 1 incorrect mapping: https://cgspace.cgiar.org/handle/10568/78658 */
delete from collection2item where id = '91082';

2017-01-17

Helping clean up some file names in the 232 CIAT records that Sisay worked on last week
There are about 30 files with %20 (space) and Spanish accents in the file name
At first I thought we should fix these, but actually it is prescribed by the W3 working group to convert these to UTF8 and URL encode them!
And the file names don’t really matter either, as long as the SAF Builder tool can read them—after that DSpace renames them with a hash in the assetstore
Seems like the only ones I should replace are the ' apostrophe characters, as %27:

value.replace("'",'%27')

Add the item’s Type to the filename column as a hint to SAF Builder so it can set a more useful description field:

value + "__description:" + cells["dc.type"].value

Test importing of the new CIAT records (actually there are 232, not 234):

$ JAVA_OPTS="-Xmx512m -Dfile.encoding=UTF-8" /home/dspacetest.cgiar.org/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568/79042 --source /home/aorth/CIAT_234/SimpleArchiveFormat/ --mapfile=/tmp/ciat.map &> /tmp/ciat.log

Many of the PDFs are 20, 30, 40, 50+ MB, which makes a total of 4GB
These are scanned from paper and likely have no compression, so we should try to test if these compression techniques help without comprimising the quality too much:

$ convert -compress Zip -density 150x150 input.pdf output.pdf
$ gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/ebook -dNOPAUSE -dQUIET -dBATCH -sOutputFile=output.pdf input.pdf

Somewhere on the Internet suggested using a DPI of 144

2017-01-19

In testing a random sample of CIAT’s PDFs for compressability, it looks like all of these methods generally increase the file size so we will just import them as they are
Import 232 CIAT records into CGSpace:

$ JAVA_OPTS="-Xmx512m -Dfile.encoding=UTF-8" /home/cgspace.cgiar.org/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568/68704 --source /home/aorth/CIAT_232/SimpleArchiveFormat/ --mapfile=/tmp/ciat.map &> /tmp/ciat.log

2017-01-22

Looking at some records that Sisay is having problems importing into DSpace Test (seems to be because of copious whitespace return characters from Excel’s CSV exporter)
There were also some issues with an invalid dc.date.issued field, and I trimmed leading / trailing whitespace and cleaned up some URLs with unneeded parameters like ?show=full

2017-01-23

I merged Atmire’s pull request into the development branch so they can deploy it on DSpace Test
Move some old ILRI Program communities to a new subcommunity for former programs (¹⁰⁵⁶⁸⁄₇₉₁₆₄):

$ for community in 10568/171 10568/27868 10568/231 10568/27869 10568/150 10568/230 10568/32724 10568/172; do /home/cgspace.cgiar.org/bin/dspace community-filiator --remove --parent=10568/27866 --child="$community" && /home/cgspace.cgiar.org/bin/dspace community-filiator --set --parent=10568/79164 --child="$community"; done

Move some collections with move-collections.sh using the following config:

10568/42161 10568/171 10568/79341
10568/41914 10568/171 10568/79340

2017-01-24

Run all updates on DSpace Test and reboot the server
Run fixes for Journal titles on CGSpace:

$ ./fix-metadata-values.py -i /tmp/fix-49-journal-titles.csv -f dc.source -t correct -m 55 -d dspace -u dspace -p 'password'

Create a new list of the top 500 journal titles from the database:

dspace-# \copy (select distinct text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=55 group by text_value order by count desc limit 500) to /tmp/journal-titles.csv with csv;

Then sort them in OpenRefine and create a controlled vocabulary by manually adding the XML markup, pull request (#298)
This would be the last issue remaining to close the meta issue about switching to controlled vocabularies (#69)

2017-01-25

Atmire says the com.atmire.statistics.util.UpdateSolrStorageReports and com.atmire.utils.ReportSender are no longer necessary because they are using a Spring scheduler for these tasks now
Pull request to remove them from the Ansible templates: https://github.com/ilri/rmg-ansible-public/pull/80
Still testing the Atmire modules on DSpace Test, and it looks like a few issues we had reported are now fixed:
- XLS Export from Content statistics
- Most popular items
- Show statistics on collection pages
But now we have a new issue with the “Types” in Content statistics not being respected—we only get the defaults, despite having custom settings in dspace/config/modules/atmire-cua.cfg

2017-01-27

Magdalena pointed out that somehow the Anonymous group had been added to the Administrators group on CGSpace (!)
Discuss plans to update CCAFS metadata and communities for their new flagships and phase II project identifiers
The flagships are in cg.subject.ccafs, and we need to probably make a new field for the phase II project identifiers

2017-01-28

Merge controlled vocabulary for journal titles (dc.source) into CGSpace (#298)
Merge new CIAT subject into CGSpace (#296)

2017-01-29

Run all system updates on DSpace Test, redeploy DSpace code, and reboot the server
Run all system updates on CGSpace, redeploy DSpace code, and reboot the server

December, 2016

Fri, 02 Dec 2016 10:43:00 +0300

2016-12-02

CGSpace was down for five hours in the morning while I was sleeping
While looking in the logs for errors, I see tons of warnings about Atmire MQM:

2016-12-02 03:00:32,352 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=CREATE, SubjectType=BUNDLE, SubjectID=70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632305, dispatcher=1544803905, detail=[null], transactionID="TX157907838689377964651674089851855413607")
2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY_METADATA, SubjectType=BUNDLE, SubjectID =70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632309, dispatcher=1544803905, detail="dc.title", transactionID="TX157907838689377964651674089851855413607")
2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=ITEM, SubjectID=80044, Object Type=BUNDLE, ObjectID=70316, TimeStamp=1480647632311, dispatcher=1544803905, detail="THUMBNAIL", transactionID="TX157907838689377964651674089851855413607")
2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=BUNDLE, SubjectID=70316, Obje ctType=BITSTREAM, ObjectID=86715, TimeStamp=1480647632318, dispatcher=1544803905, detail="-1", transactionID="TX157907838689377964651674089851855413607")
2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY, SubjectType=ITEM, SubjectID=80044, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632351, dispatcher=1544803905, detail=[null], transactionID="TX157907838689377964651674089851855413607")

I see thousands of them in the logs for the last few months, so it’s not related to the DSpace 5.5 upgrade
I’ve raised a ticket with Atmire to ask
Another worrying error from dspace.log is:

org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.NoSuchMethodError: com.atmire.statistics.generator.DSpaceObjectDatasetGenerator.toDatasetQuery(Lorg/dspace/core/Context;)Lcom/atmire/statistics/content/DatasetQuery;
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:972)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.dspace.app.xmlui.cocoon.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.dspace.app.xmlui.cocoon.DSpaceCocoonServletFilter.doFilter(DSpaceCocoonServletFilter.java:274)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.dspace.app.xmlui.cocoon.servlet.multipart.DSpaceMultipartFilter.doFilter(DSpaceMultipartFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.dspace.utils.servlet.DSpaceWebappServletFilter.doFilter(DSpaceWebappServletFilter.java:78)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
        at com.googlecode.psiprobe.Tomcat70AgentValve.invoke(Tomcat70AgentValve.java:44)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
        at org.apache.catalina.valves.CrawlerSessionManagerValve.invoke(CrawlerSessionManagerValve.java:180)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodError: com.atmire.statistics.generator.DSpaceObjectDatasetGenerator.toDatasetQuery(Lorg/dspace/core/Context;)Lcom/atmire/statistics/content/DatasetQuery;
        at com.atmire.statistics.generator.TopNDSODatasetGenerator.toDatasetQuery(SourceFile:39)
        at com.atmire.statistics.display.StatisticsDataVisitsMultidata.createDataset(SourceFile:108)
        at org.dspace.statistics.content.StatisticsDisplay.createDataset(SourceFile:384)
        at org.dspace.statistics.content.StatisticsDisplay.getDataset(SourceFile:404)
        at com.atmire.statistics.mostpopular.JSONStatsMostPopularGenerator.generateJsonData(SourceFile:170)
        at com.atmire.statistics.mostpopular.JSONStatsMostPopularGenerator.generate(SourceFile:246)
        at com.atmire.app.xmlui.aspect.statistics.JSONStatsMostPopular.generate(JSONStatsMostPopular.java:145)
        at sun.reflect.GeneratedMethodAccessor296.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
        at com.sun.proxy.$Proxy96.process(Unknown Source)
        at org.apache.cocoon.components.treeprocessor.sitemap.ReadNode.invoke(ReadNode.java:94)
        at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:55)
        at org.apache.cocoon.components.treeprocessor.sitemap.MatchNode.invoke(MatchNode.java:87)
        at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:55)
        at org.apache.cocoon.components.treeprocessor.sitemap.MatchNode.invoke(MatchNode.java:87)
        at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
        at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:143)
        at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
        at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:81)
        at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:239)
        at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:171)
        at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:247)
        at org.apache.cocoon.components.treeprocessor.sitemap.MountNode.invoke(MountNode.java:117)
        at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:55)
        at org.apache.cocoon.components.treeprocessor.sitemap.MatchNode.invoke(MatchNode.java:87)
        at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
        at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:143)
        at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
        at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:81)
        at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:239)
        at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:171)
        at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:247)
        at org.apache.cocoon.servlet.RequestProcessor.process(RequestProcessor.java:351)
        at org.apache.cocoon.servlet.RequestProcessor.service(RequestProcessor.java:169)
        at org.apache.cocoon.sitemap.SitemapServlet.service(SitemapServlet.java:84)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:468)
        at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:443)
        at org.apache.cocoon.servletservice.spring.ServletFactoryBean$ServiceInterceptor.invoke(ServletFactoryBean.java:264)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
        at com.sun.proxy.$Proxy89.service(Unknown Source)
        at org.dspace.springmvc.CocoonView.render(CocoonView.java:113)
        at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1180)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:950)
        ... 35 more

The first error I see in dspace.log this morning is:

2016-12-02 03:00:46,656 ERROR org.dspace.authority.AuthorityValueFinder @ anonymous::Error while retrieving AuthorityValue from solr:query\colon; id\colon;"b0b541c1-ec15-48bf-9209-6dbe8e338cdc"
org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:8081/solr/authority

Looking through DSpace’s solr log I see that about 20 seconds before this, there were a few 30+ KiB solr queries
The last logs here right before Solr became unresponsive (and right after I restarted it five hours later) were:

2016-12-02 03:00:42,606 INFO  org.apache.solr.core.SolrCore @ [statistics] webapp=/solr path=/select params={q=containerItem:72828+AND+type:0&shards=localhost:8081/solr/statistics-2010,localhost:8081/solr/statistics&fq=-isInternal:true&fq=-(author_mtdt:"CGIAR\+Institutional\+Learning\+and\+Change\+Initiative"++AND+subject_mtdt:"PARTNERSHIPS"+AND+subject_mtdt:"RESEARCH"+AND+subject_mtdt:"AGRICULTURE"+AND+subject_mtdt:"DEVELOPMENT"++AND+iso_mtdt:"en"+)&rows=0&wt=javabin&version=2} hits=0 status=0 QTime=19
2016-12-02 08:28:23,908 INFO  org.apache.solr.servlet.SolrDispatchFilter @ SolrDispatchFilter.init()

DSpace’s own Solr logs don’t give IP addresses, so I will have to enable Nginx’s logging of /solr so I can see where this request came from
I enabled logging of /rest/ and I think I’ll leave it on for good
Also, the disk is nearly full because of log file issues, so I’m running some compression on DSpace logs
Normally these stay uncompressed for a month just in case we need to look at them, so now I’ve just compressed anything older than 2 weeks so we can get some disk space back

2016-12-04

I got a weird report from the CGSpace checksum checker this morning
It says 732 bitstreams have potential issues, for example:

------------------------------------------------ 
Bitstream Id = 6
Process Start Date = Dec 4, 2016
Process End Date = Dec 4, 2016
Checksum Expected = a1d9eef5e2d85f50f67ce04d0329e96a
Checksum Calculated = a1d9eef5e2d85f50f67ce04d0329e96a
Result = Bitstream marked deleted in bitstream table
----------------------------------------------- 
...
------------------------------------------------ 
Bitstream Id = 77581
Process Start Date = Dec 4, 2016
Process End Date = Dec 4, 2016
Checksum Expected = 9959301aa4ca808d00957dff88214e38
Checksum Calculated = 
Result = The bitstream could not be found
-----------------------------------------------

The first one seems ok, but I don’t know what to make of the second one…
I had a look and there is indeed no file with the second checksum in the assetstore (ie, looking in [dspace-dir]/assetstore/99/59/30/...)
For what it’s worth, there is no item on DSpace Test or S3 backups with that checksum either…
In other news, I’m looking at JVM settings from the Solr 4.10.2 release, from bin/solr.in.sh:

# These GC settings have shown to work well for a number of common Solr workloads
GC_TUNE="-XX:-UseSuperWord \
-XX:NewRatio=3 \
-XX:SurvivorRatio=4 \
-XX:TargetSurvivorRatio=90 \
-XX:MaxTenuringThreshold=8 \
-XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC \
-XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
-XX:+CMSScavengeBeforeRemark \
-XX:PretenureSizeThreshold=64m \
-XX:CMSFullGCsBeforeCompaction=1 \
-XX:+UseCMSInitiatingOccupancyOnly \
-XX:CMSInitiatingOccupancyFraction=50 \
-XX:CMSTriggerPermRatio=80 \
-XX:CMSMaxAbortablePrecleanTime=6000 \
-XX:+CMSParallelRemarkEnabled \
-XX:+ParallelRefProcEnabled \
-XX:+AggressiveOpts"

I need to try these because they are recommended by the Solr project itself
Also, as always, I need to read Shawn Heisey’s wiki page on Solr

2016-12-05

I did some basic benchmarking on a local DSpace before and after the JVM settings above, but there wasn’t anything amazingly obvious
I want to make the changes on DSpace Test and monitor the JVM heap graphs for a few days to see if they change the JVM GC patterns or anything (munin graphs)
Spin up new CGSpace server on Linode
I did a few traceroutes from Jordan and Kenya and it seems that Linode’s Frankfurt datacenter is a few less hops and perhaps less packet loss than the London one, so I put the new server in Frankfurt
Do initial provisioning
Atmire responded about the MQM warnings in the DSpace logs
Apparently we need to change the batch edit consumers in dspace/config/dspace.cfg:

event.consumer.batchedit.filters = Community|Collection+Create

I haven’t tested it yet, but I created a pull request: #289

2016-12-06

Some author authority corrections and name standardizations for Peter:

dspace=# update metadatavalue set authority='b041f2f4-19e7-4113-b774-0439baabd197', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Mora Benard%';
UPDATE 11
dspace=# update metadatavalue set text_value = 'Hoek, Rein van der', authority='4d6cbce2-6fd5-4b43-9363-58d18e7952c9', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like 'Hoek, R%';
UPDATE 36
dspace=# update metadatavalue set text_value = 'Hoek, Rein van der', authority='4d6cbce2-6fd5-4b43-9363-58d18e7952c9', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like '%an der Hoek%' and text_value !~ '^.*W\.?$';
UPDATE 14
dspace=# update metadatavalue set authority='18349f29-61b1-44d7-ac60-89e55546e812', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like 'Thorne, P%';
UPDATE 42
dspace=# update metadatavalue set authority='0d8369bb-57f7-4b2f-92aa-af820b183aca', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like 'Thornton, P%';
UPDATE 360
dspace=# update metadatavalue set text_value='Grace, Delia', authority='0b4fcbc1-d930-4319-9b4d-ea1553cca70b', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like 'Grace, D%';
UPDATE 561

Pay attention to the regex to prevent false positives in tricky cases with Dutch names!
I will run these updates on DSpace Test and then force a Discovery reindex, and then run them on CGSpace next week
More work on the KM4Dev Journal article
In other news, it seems the batch edit patch is working, there are no more WARN errors in the logs and the batch edit seems to work
I need to check the CGSpace logs to see if there are still errors there, and then deploy/monitor it there
Paola from CCAFS mentioned she also has the “take task” bug on CGSpace
Reading about shared_buffers in PostgreSQL configuration (default is 128MB)
Looks like we have ~5GB of memory used by caches on the test server (after OS and JVM heap!), so we might as well bump up the buffers for Postgres
The docs say a good starting point for a dedicated server is 25% of the system RAM, and our server isn’t dedicated (also runs Solr, which can benefit from OS cache) so let’s try 1024MB
In other news, the authority reindexing keeps crashing (I was manually running it after the author updates above):

$ time JAVA_OPTS="-Xms768m -Xmx768m -Dfile.encoding=UTF-8" /home/dspacetest.cgiar.org/bin/dspace index-authority
Retrieving all data
Initialize org.dspace.authority.indexer.DSpaceAuthorityIndexer
Exception: null
java.lang.NullPointerException
        at org.dspace.authority.AuthorityValueGenerator.generateRaw(AuthorityValueGenerator.java:82)
        at org.dspace.authority.AuthorityValueGenerator.generate(AuthorityValueGenerator.java:39)
        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.prepareNextValue(DSpaceAuthorityIndexer.java:201)
        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:132)
        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:144)
        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:144)
        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:159)
        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:144)
        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:144)
        at org.dspace.authority.indexer.AuthorityIndexClient.main(AuthorityIndexClient.java:61)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:226)
        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:78)

real    8m39.913s
user    1m54.190s
sys     0m22.647s

2016-12-07

For what it’s worth, after running the same SQL updates on my local test server, index-authority runs and completes just fine
I will have to test more
Anyways, I noticed that some of the authority values I set actually have versions of author names we don’t want, ie “Grace, D.”
For example, do a Solr query for “first_name:Grace” and look at the results
Querying that ID shows the fields that need to be changed:

{
  "responseHeader": {
    "status": 0,
    "QTime": 1,
    "params": {
      "q": "id:0b4fcbc1-d930-4319-9b4d-ea1553cca70b",
      "indent": "true",
      "wt": "json",
      "_": "1481102189244"
    }
  },
  "response": {
    "numFound": 1,
    "start": 0,
    "docs": [
      {
        "id": "0b4fcbc1-d930-4319-9b4d-ea1553cca70b",
        "field": "dc_contributor_author",
        "value": "Grace, D.",
        "deleted": false,
        "creation_date": "2016-11-10T15:13:40.318Z",
        "last_modified_date": "2016-11-10T15:13:40.318Z",
        "authority_type": "person",
        "first_name": "D.",
        "last_name": "Grace"
      }
    ]
  }
}

I think I can just update the value, first_name, and last_name fields…
The update syntax should be something like this, but I’m getting errors from Solr:

$ curl 'localhost:8081/solr/authority/update?commit=true&wt=json&indent=true' -H 'Content-type:application/json' -d '[{"id":"1","price":{"set":100}}]'
{
  "responseHeader":{
    "status":400,
    "QTime":0},
  "error":{
    "msg":"Unexpected character '[' (code 91) in prolog; expected '<'\n at [row,col {unknown-source}]: [1,1]",
    "code":400}}

When I try using the XML format I get an error that the updateLog needs to be configured for that core
Maybe I can just remove the authority UUID from the records, run the indexing again so it creates a new one for each name variant, then match them correctly?

dspace=# update metadatavalue set authority=null, confidence=-1 where resource_type_id=2 and metadata_field_id=3 and text_value like 'Grace, D%';
UPDATE 561

Then I’ll reindex discovery and authority and see how the authority Solr core looks
After this, now there are authorities for some of the “Grace, D.” and “Grace, Delia” text_values in the database (the first version is actually the same authority that already exists in the core, so it was just added back to some text_values, but the second one is new):

$ curl 'localhost:8081/solr/authority/select?q=id%3A18ea1525-2513-430a-8817-a834cd733fbc&wt=json&indent=true'
{
  "responseHeader":{
    "status":0,
    "QTime":0,
    "params":{
      "q":"id:18ea1525-2513-430a-8817-a834cd733fbc",
      "indent":"true",
      "wt":"json"}},
  "response":{"numFound":1,"start":0,"docs":[
      {
        "id":"18ea1525-2513-430a-8817-a834cd733fbc",
        "field":"dc_contributor_author",
        "value":"Grace, Delia",
        "deleted":false,
        "creation_date":"2016-12-07T10:54:34.356Z",
        "last_modified_date":"2016-12-07T10:54:34.356Z",
        "authority_type":"person",
        "first_name":"Delia",
        "last_name":"Grace"}]
  }}

So now I could set them all to this ID and the name would be ok, but there has to be a better way!
In this case it seems that since there were also two different IDs in the original database, I just picked the wrong one!
Better to use:

dspace#= update metadatavalue set text_value='Grace, Delia', authority='bfa61d7c-7583-4175-991c-2e7315000f0c', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like 'Grace, D%';

This proves that unifying author name varieties in authorities is easy, but fixing the name in the authority is tricky!
Perhaps another way is to just add our own UUID to the authority field for the text_value we like, then re-index authority so they get synced from PostgreSQL to Solr, then set the other text_values to use that authority ID
Deploy MQM WARN fix on CGSpace (#289)
Deploy “take task” hack/fix on CGSpace (#290)
I ran the following author corrections and then reindexed discovery:

update metadatavalue set authority='b041f2f4-19e7-4113-b774-0439baabd197', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Mora Benard%';
update metadatavalue set text_value = 'Hoek, Rein van der', authority='4d6cbce2-6fd5-4b43-9363-58d18e7952c9', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like 'Hoek, R%';
update metadatavalue set text_value = 'Hoek, Rein van der', authority='4d6cbce2-6fd5-4b43-9363-58d18e7952c9', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like '%an der Hoek%' and text_value !~ '^.*W\.?$';
update metadatavalue set authority='18349f29-61b1-44d7-ac60-89e55546e812', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like 'Thorne, P%';
update metadatavalue set authority='0d8369bb-57f7-4b2f-92aa-af820b183aca', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like 'Thornton, P%';
update metadatavalue set text_value='Grace, Delia', authority='bfa61d7c-7583-4175-991c-2e7315000f0c', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like 'Grace, D%';

2016-12-08

Something weird happened and Peter Thorne’s names all ended up as “Thorne”, I guess because the original authority had that as its name value:

dspace=# select distinct text_value, authority, confidence from metadatavalue where resource_type_id=2 and metadata_field_id=3 and text_value like 'Thorne%';
    text_value    |              authority               | confidence
------------------+--------------------------------------+------------
 Thorne, P.J.     | 18349f29-61b1-44d7-ac60-89e55546e812 |        600
 Thorne           | 18349f29-61b1-44d7-ac60-89e55546e812 |        600
 Thorne-Lyman, A. | 0781e13a-1dc8-4e3f-82e8-5c422b44a344 |         -1
 Thorne, M. D.    | 54c52649-cefd-438d-893f-3bcef3702f07 |         -1
 Thorne, P.J      | 18349f29-61b1-44d7-ac60-89e55546e812 |        600
 Thorne, P.       | 18349f29-61b1-44d7-ac60-89e55546e812 |        600
(6 rows)

I generated a new UUID using uuidgen | tr [A-Z] [a-z] and set it along with correct name variation for all records:

dspace=# update metadatavalue set authority='b2f7603d-2fb5-4018-923a-c4ec8d85b3bb', text_value='Thorne, P.J.' where resource_type_id=2 and metadata_field_id=3 and authority='18349f29-61b1-44d7-ac60-89e55546e812';
UPDATE 43

Apparently we also need to normalize Phil Thornton’s names to Thornton, Philip K.:

dspace=# select distinct text_value, authority, confidence from metadatavalue where resource_type_id=2 and metadata_field_id=3 and text_value ~ '^Thornton[,\.]? P.*';
     text_value      |              authority               | confidence
---------------------+--------------------------------------+------------
 Thornton, P         | 0d8369bb-57f7-4b2f-92aa-af820b183aca |        600
 Thornton, P K.      | 0d8369bb-57f7-4b2f-92aa-af820b183aca |        600
 Thornton, P K       | 0d8369bb-57f7-4b2f-92aa-af820b183aca |        600
 Thornton. P.K.      | 3e1e6639-d4fb-449e-9fce-ce06b5b0f702 |         -1
 Thornton, P K .     | 0d8369bb-57f7-4b2f-92aa-af820b183aca |        600
 Thornton, P.K.      | 0d8369bb-57f7-4b2f-92aa-af820b183aca |        600
 Thornton, P.K       | 0d8369bb-57f7-4b2f-92aa-af820b183aca |        600
 Thornton, Philip K  | 0d8369bb-57f7-4b2f-92aa-af820b183aca |        600
 Thornton, Philip K. | 0d8369bb-57f7-4b2f-92aa-af820b183aca |        600
 Thornton, P. K.     | 0d8369bb-57f7-4b2f-92aa-af820b183aca |        600
(10 rows)

Seems his original authorities are using an incorrect version of the name so I need to generate another UUID and tie it to the correct name, then reindex:

dspace=# update metadatavalue set authority='2df8136e-d8f4-4142-b58c-562337cab764', text_value='Thornton, Philip K.', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value ~ '^Thornton[,\.]? P.*';
UPDATE 362

It seems that, when you are messing with authority and author text values in the database, it is better to run authority reindex first (postgres→solr authority core) and then Discovery reindex (postgres→solr Discovery core)
Everything looks ok after authority and discovery reindex
In other news, I think we should really be using more RAM for PostgreSQL’s shared_buffers
The PostgreSQL documentation recommends using 25% of the system’s RAM on dedicated systems, but we should use a bit less since we also have a massive JVM heap and also benefit from some RAM being used by the OS cache

2016-12-09

More work on finishing rough draft of KM4Dev article
Set PostgreSQL’s shared_buffers on CGSpace to 10% of system RAM (1200MB)
Run the following author corrections on CGSpace:

dspace=# update metadatavalue set authority='34df639a-42d8-4867-a3f2-1892075fcb3f', text_value='Thorne, P.J.' where resource_type_id=2 and metadata_field_id=3 and authority='18349f29-61b1-44d7-ac60-89e55546e812' or authority='021cd183-946b-42bb-964e-522ebff02993';
dspace=# update metadatavalue set authority='2df8136e-d8f4-4142-b58c-562337cab764', text_value='Thornton, Philip K.', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value ~ '^Thornton[,\.]? P.*';

The authority IDs were different now than when I was looking a few days ago so I had to adjust them here

2016-12-11

After enabling a sizable shared_buffers for CGSpace’s PostgreSQL configuration the number of connections to the database dropped significantly

Looking at CIAT records from last week again, they have a lot of double authors like:

International Center for Tropical Agriculture::3026b1de-9302-4f3e-85ab-ef48da024eb2::600
International Center for Tropical Agriculture::3026b1de-9302-4f3e-85ab-ef48da024eb2::500
International Center for Tropical Agriculture::3026b1de-9302-4f3e-85ab-ef48da024eb2::0

Some in the same dc.contributor.author field, and some in others like dc.contributor.author[en_US] etc
Removing the duplicates in OpenRefine and uploading a CSV to DSpace says “no changes detected”
Seems like the only way to sortof clean these up would be to start in SQL:

dspace=# select distinct text_value, authority, confidence from metadatavalue where resource_type_id=2 and metadata_field_id=3 and text_value like 'International Center for Tropical Agriculture';
                  text_value                   |              authority               | confidence
-----------------------------------------------+--------------------------------------+------------
 International Center for Tropical Agriculture | cc726b78-a2f4-4ee9-af98-855c2ea31c36 |         -1
 International Center for Tropical Agriculture |                                      |        600
 International Center for Tropical Agriculture | 3026b1de-9302-4f3e-85ab-ef48da024eb2 |        500
 International Center for Tropical Agriculture | cc726b78-a2f4-4ee9-af98-855c2ea31c36 |        600
 International Center for Tropical Agriculture |                                      |         -1
 International Center for Tropical Agriculture | cc726b78-a2f4-4ee9-af98-855c2ea31c36 |        500
 International Center for Tropical Agriculture | 3026b1de-9302-4f3e-85ab-ef48da024eb2 |        600
 International Center for Tropical Agriculture | 3026b1de-9302-4f3e-85ab-ef48da024eb2 |         -1
 International Center for Tropical Agriculture | 3026b1de-9302-4f3e-85ab-ef48da024eb2 |          0
dspace=# update metadatavalue set authority='3026b1de-9302-4f3e-85ab-ef48da024eb2', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value = 'International Center for Tropical Agriculture';
UPDATE 1693
dspace=# update metadatavalue set authority='3026b1de-9302-4f3e-85ab-ef48da024eb2', text_value='International Center for Tropical Agriculture', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like '%CIAT%';
UPDATE 35

Work on article for KM4Dev journal

2016-12-13

Checking in on CGSpace postgres stats again, looks like the shared_buffers change from a few days ago really made a big impact:

Looking at logs, it seems we need to evaluate which logs we keep and for how long
Basically the only ones we need are dspace.log because those are used for legacy statistics (need to keep for 1 month)
Other logs will be an issue because they don’t have date stamps
I will add date stamps to the logs we’re storing from the tomcat7 user’s cron jobs at least, using: $(date --iso-8601)
Would probably be better to make custom logrotate files for them in the future
Clean up some unneeded log files from 2014 (they weren’t large, just don’t need them)
So basically, new cron jobs for logs should look something like this:
Find any file named *.log* that isn’t dspace.log*, isn’t already zipped, and is older than one day, and zip it:

# find /home/dspacetest.cgiar.org/log -regextype posix-extended -iregex ".*\.log.*" ! -iregex ".*dspace\.log.*" ! -iregex ".*\.(gz|lrz|lzo|xz)" ! -newermt "Yesterday" -exec schedtool -B -e ionice -c2 -n7 xz {} \;

Since there is xzgrep and xzless we can actually just zip them after one day, why not?!
We can keep the zipped ones for two weeks just in case we need to look for errors, etc, and delete them after that
I use schedtool -B and ionice -c2 -n7 to set the CPU scheduling to SCHED_BATCH and the IO to best effort which should, in theory, impact important system processes like Tomcat and PostgreSQL less
When the tasks are running you can see that the policies do apply:

$ schedtool $(ps aux | grep "xz /home" | grep -v grep | awk '{print $2}') && ionice -p $(ps aux | grep "xz /home" | grep -v grep | awk '{print $2}')
PID 17049: PRIO   0, POLICY B: SCHED_BATCH   , NICE   0, AFFINITY 0xf
best-effort: prio 7

All in all this should free up a few gigs (we were at 9.3GB free when I started)
Next thing to look at is whether we need Tomcat’s access logs
I just looked and it seems that we saved 10GB by zipping these logs
Some users pointed out issues with the “most popular” stats on a community or collection
This error appears in the logs when you try to view them:

2016-12-13 21:17:37,486 ERROR org.dspace.app.xmlui.cocoon.DSpaceCocoonServletFilter @ Serious Error Occurred Processing Request!
org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.NoSuchMethodError: com.atmire.statistics.generator.DSpaceObjectDatasetGenerator.toDatasetQuery(Lorg/dspace/core/Context;)Lcom/atmire/statistics/content/DatasetQuery;
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:972)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.dspace.app.xmlui.cocoon.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.dspace.app.xmlui.cocoon.DSpaceCocoonServletFilter.doFilter(DSpaceCocoonServletFilter.java:274)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.dspace.app.xmlui.cocoon.servlet.multipart.DSpaceMultipartFilter.doFilter(DSpaceMultipartFilter.java:119)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.dspace.utils.servlet.DSpaceWebappServletFilter.doFilter(DSpaceWebappServletFilter.java:78)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:221)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
	at org.apache.catalina.valves.CrawlerSessionManagerValve.invoke(CrawlerSessionManagerValve.java:180)
	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodError: com.atmire.statistics.generator.DSpaceObjectDatasetGenerator.toDatasetQuery(Lorg/dspace/core/Context;)Lcom/atmire/statistics/content/DatasetQuery;
	at com.atmire.statistics.generator.TopNDSODatasetGenerator.toDatasetQuery(SourceFile:39)
	at com.atmire.statistics.display.StatisticsDataVisitsMultidata.createDataset(SourceFile:108)
	at org.dspace.statistics.content.StatisticsDisplay.createDataset(SourceFile:384)
	at org.dspace.statistics.content.StatisticsDisplay.getDataset(SourceFile:404)
	at com.atmire.statistics.mostpopular.JSONStatsMostPopularGenerator.generateJsonData(SourceFile:170)
	at com.atmire.statistics.mostpopular.JSONStatsMostPopularGenerator.generate(SourceFile:246)
	at com.atmire.app.xmlui.aspect.statistics.JSONStatsMostPopular.generate(JSONStatsMostPopular.java:145)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

It happens on development and production, so I will have to ask Atmire
Most likely an issue with installation/configuration

2016-12-14

Atmire sent a quick fix for the last-update.txt file not found error
After applying pull request #291 on DSpace Test I no longer see the error in the logs after the UpdateSolrStorageReports task runs
Also, I’m toying with the idea of moving the tomcat7 user’s cron jobs to /etc/cron.d so we can manage them in Ansible
Made a pull request with a template for the cron jobs (#75)
Testing SMTP from the new CGSpace server and it’s not working, I’ll have to tell James

2016-12-15

Start planning for server migration this weekend, letting users know
I am trying to figure out what the process is to update the server’s IP in the Handle system, and emailing the hdladmin account bounces(!)
I will contact the Jane Euler directly as I know I’ve corresponded with her in the past
She said that I should indeed just re-run the [dspace]/bin/dspace make-handle-server command and submit the new sitebndl.zip file to the CNRI website
Also I was troubleshooting some workflow issues from Bizuwork
I re-created the same scenario by adding a non-admin account and submitting an item, but I was able to successfully approve and commit it
So it turns out it’s not a bug, it’s just that Peter was added as a reviewer/admin AFTER the items were submitted
This is how DSpace works, and I need to ask if there is a way to override someone’s submission, as the other reviewer seems to not be paying attention, or has perhaps taken the item from the task pool?
Run a batch edit to add “RANGELANDS” ILRI subject to all items containing the word “RANGELANDS” in their metadata for Peter Ballantyne

2016-12-18

Add four new CRP subjects for 2017 and sort the input forms alphabetically (#294)
Test the SMTP on the new server and it’s working
Last week, when we asked CGNET to update the DNS records this weekend, they misunderstood and did it immediately
We quickly told them to undo it, but I just realized they didn’t undo the IPv6 AAAA record!
None of our users in African institutes will have IPv6, but some Europeans might, so I need to check if any submissions have been added since then
Update some names and authorities in the database:

dspace=# update metadatavalue set authority='5ff35043-942e-4d0a-b377-4daed6e3c1a3', confidence=600, text_value='Duncan, Alan' where resource_type_id=2 and metadata_field_id=3 and text_value ~ '^.*Duncan,? A.*';
UPDATE 204
dspace=# update metadatavalue set authority='46804b53-ea30-4a85-9ccf-b79a35816fa9', confidence=600, text_value='Mekonnen, Kindu' where resource_type_id=2 and metadata_field_id=3 and text_value like '%Mekonnen, K%';
UPDATE 89
dspace=# update metadatavalue set authority='f840da02-26e7-4a74-b7ba-3e2b723f3684', confidence=600, text_value='Lukuyu, Ben A.' where resource_type_id=2 and metadata_field_id=3 and text_value like '%Lukuyu, B%';
UPDATE 140

Generated a new UUID for Ben using uuidgen | tr [A-Z] [a-z] as the one in Solr had his ORCID but the name format was incorrect
In theory DSpace should be able to check names from ORCID and update the records in the database, but I find that this doesn’t work (see Jira bug DS-3302)
I need to run these updates along with the other one for CIAT that I found last week
Enable OCSP stapling for hosts >= Ubuntu 16.04 in our Ansible playbooks (#76)
Working for DSpace Test on the second response:

$ openssl s_client -connect dspacetest.cgiar.org:443 -servername dspacetest.cgiar.org -tls1_2 -tlsextdebug -status
...
OCSP response: no response sent
$ openssl s_client -connect dspacetest.cgiar.org:443 -servername dspacetest.cgiar.org -tls1_2 -tlsextdebug -status
...
OCSP Response Data:
...
    Cert Status: good

Migrate CGSpace to new server, roughly following these steps:
On old server:

# service tomcat7 stop
# /home/backup/scripts/postgres_backup.sh

On new server:

# systemctl stop tomcat7
# rsync -4 -av --delete 178.79.187.182:/home/cgspace.cgiar.org/assetstore/ /home/cgspace.cgiar.org/assetstore/
# rsync -4 -av --delete 178.79.187.182:/home/backup/ /home/backup/
# rsync -4 -av --delete 178.79.187.182:/home/cgspace.cgiar.org/solr/ /home/cgspace.cgiar.org/solr
# su - postgres
$ dropdb cgspace
$ createdb -O cgspace --encoding=UNICODE cgspace
$ psql cgspace -c 'alter user cgspace createuser;'
$ pg_restore -O -U cgspace -d cgspace -W -h localhost /home/backup/postgres/cgspace_2016-12-18.backup
$ psql cgspace -c 'alter user cgspace nocreateuser;'
$ psql -U cgspace -f ~tomcat7/src/git/DSpace/dspace/etc/postgres/update-sequences.sql cgspace -h localhost
$ vacuumdb cgspace
$ psql cgspace
postgres=# \i /tmp/author-authority-updates-2016-12-11.sql
postgres=# \q
$ exit
# chown -R tomcat7:tomcat7 /home/cgspace.cgiar.org
# rsync -4 -av 178.79.187.182:/home/cgspace.cgiar.org/log/*.dat /home/cgspace.cgiar.org/log/
# rsync -4 -av 178.79.187.182:/home/cgspace.cgiar.org/log/dspace.log.2016-1[12]* /home/cgspace.cgiar.org/log/
# su - tomcat7
$ cd src/git/DSpace/dspace/target/dspace-installer
$ ant update clean_backups
$ exit
# systemctl start tomcat7

It took about twenty minutes and afterwards I had to check a few things, like:
- check and enable systemd timer for let’s encrypt
- enable root cron jobs
- disable root cron jobs on old server after!
- enable tomcat7 cron jobs
- disable tomcat7 cron jobs on old server after!
- regenerate sitebndl.zip with new IP for handle server and submit it to Handle.net

2016-12-22

Abenet wanted a CSV of the IITA community, but the web export doesn’t include the dc.date.accessioned field
I had to export it from the command line using the -a flag:

$ [dspace]/bin/dspace metadata-export -a -f /tmp/iita.csv -i 10568/68616

2016-12-28

We’ve been getting two alerts per day about CPU usage on the new server from Linode
These are caused by the batch jobs for Solr etc that run in the early morning hours
The Linode default is to alert at 90% CPU usage for two hours, but I see the old server was at 150%, so maybe we just need to adjust it
Speaking of the old server (linode01), I think we can decommission it now
I checked the S3 logs on the new server (linode18) to make sure the backups have been running and everything looks good
In other news, I was looking at the Munin graphs for PostgreSQL on the new server and it looks slightly worrying:

I will have to check later why the size keeps increasing

November, 2016

Tue, 01 Nov 2016 09:21:00 +0300

2016-11-01

Add dc.type to the output options for Atmire’s Listings and Reports module (#286)

2016-11-02

Migrate DSpace Test to DSpace 5.5 (notes)
Run all updates on DSpace Test and reboot the server
Looks like the OAI bug from DSpace 5.1 that caused validation at Base Search to fail is now fixed and DSpace Test passes validation! (#63)
Indexing Discovery on DSpace Test took 332 minutes, which is like five times as long as it usually takes
At the end it appeared to finish correctly but there were lots of errors right after it finished:

2016-11-02 15:09:48,578 INFO  com.atmire.dspace.discovery.AtmireSolrService @ Wrote Collection: 10568/76454 to Index
2016-11-02 15:09:48,584 INFO  com.atmire.dspace.discovery.AtmireSolrService @ Wrote Community: 10568/3202 to Index
2016-11-02 15:09:48,589 INFO  com.atmire.dspace.discovery.AtmireSolrService @ Wrote Collection: 10568/76455 to Index
2016-11-02 15:09:48,590 INFO  com.atmire.dspace.discovery.AtmireSolrService @ Wrote Community: 10568/51693 to Index
2016-11-02 15:09:48,590 INFO  org.dspace.discovery.IndexClient @ Done with indexing
2016-11-02 15:09:48,600 INFO  com.atmire.dspace.discovery.AtmireSolrService @ Wrote Collection: 10568/76456 to Index
2016-11-02 15:09:48,613 INFO  org.dspace.discovery.SolrServiceImpl @ Wrote Item: 10568/55536 to Index
2016-11-02 15:09:48,616 INFO  com.atmire.dspace.discovery.AtmireSolrService @ Wrote Collection: 10568/76457 to Index
2016-11-02 15:09:48,634 ERROR com.atmire.dspace.discovery.AtmireSolrService @
java.lang.NullPointerException
        at org.dspace.discovery.SearchUtils.getDiscoveryConfiguration(SourceFile:57)
        at org.dspace.discovery.SolrServiceImpl.buildDocument(SolrServiceImpl.java:824)
        at com.atmire.dspace.discovery.AtmireSolrService.indexContent(AtmireSolrService.java:821)
        at com.atmire.dspace.discovery.AtmireSolrService.updateIndex(AtmireSolrService.java:898)
        at org.dspace.discovery.SolrServiceImpl.createIndex(SolrServiceImpl.java:370)
        at org.dspace.storage.rdbms.DatabaseUtils$ReindexerThread.run(DatabaseUtils.java:945)

DSpace is still up, and a few minutes later I see the default DSpace indexer is still running
Sure enough, looking back before the first one finished, I see output from both indexers interleaved in the log:

2016-11-02 15:09:28,545 INFO  org.dspace.discovery.SolrServiceImpl @ Wrote Item: 10568/47242 to Index
2016-11-02 15:09:28,633 INFO  org.dspace.discovery.SolrServiceImpl @ Wrote Item: 10568/60785 to Index
2016-11-02 15:09:28,678 INFO  com.atmire.dspace.discovery.AtmireSolrService @ Processing (55695 of 55722): 43557
2016-11-02 15:09:28,688 INFO  com.atmire.dspace.discovery.AtmireSolrService @ Processing (55703 of 55722): 34476

I will raise a ticket with Atmire to ask them

2016-11-06

After re-deploying and re-indexing I didn’t see the same issue, and the indexing completed in 85 minutes, which is about how long it is supposed to take

2016-11-07

Horrible one liner to get Linode ID from certain Ansible host vars:

$ grep -A 3 contact_info * | grep -E "(Orth|Sisay|Peter|Daniel|Tsega)" | awk -F'-' '{print $1}' | grep linode | uniq | xargs grep linode_id

I noticed some weird CRPs in the database, and they don’t show up in Discovery for some reason, perhaps the :
I’ll export these and fix them in batch:

dspace=# \copy (select distinct text_value, count(*) from metadatavalue where metadata_field_id=230 group by text_value order by count desc) to /tmp/crp.csv with csv;
COPY 22

Test running the replacements:

$ ./fix-metadata-values.py -i /tmp/CRPs.csv -f cg.contributor.crp -t correct -m 230 -d dspace -u dspace -p 'fuuu'

Add AMR to ILRI subjects and remove one duplicate instance of IITA in author affiliations controlled vocabulary (#288)

2016-11-08

Atmire’s Listings and Reports module seems to be broken on DSpace 5.5

I’ve filed a ticket with Atmire
Thinking about batch updates for ORCIDs and authors
Playing with SolrClient in Python to query Solr
All records in the authority core are either authority_type:orcid or authority_type:person
There is a deleted field and all items seem to be false, but might be important sanity check to remember
The way to go is probably to have a CSV of author names and authority IDs, then to batch update them in PostgreSQL
Dump of the top ~200 authors in CGSpace:

dspace=# \copy (select distinct text_value, count(*) from metadatavalue where metadata_field_id=3 group by text_value order by count desc limit 210) to /tmp/210-authors.csv with csv;

2016-11-09

CGSpace crashed so I quickly ran system updates, applied one or two of the waiting changes from the 5_x-prod branch, and rebooted the server
The error was Timeout waiting for idle object but I haven’t looked into the Tomcat logs to see what happened
Also, I ran the corrections for CRPs from earlier this week

2016-11-10

Helping Megan Zandstra and CIAT with some questions about the REST API
Playing with find-by-metadata-field, this works:

$ curl -s -H "accept: application/json" -H "Content-Type: application/json" -X POST "http://localhost:8080/rest/items/find-by-metadata-field" -d '{"key": "cg.subject.ilri","value": "SEEDS"}'

But the results are deceiving because metadata fields can have text languages and your query must match exactly!

dspace=# select distinct text_value, text_lang from metadatavalue where resource_type_id=2 and metadata_field_id=203 and text_value='SEEDS';
 text_value | text_lang
------------+-----------
 SEEDS      |
 SEEDS      |
 SEEDS      | en_US
(3 rows)

So basically, the text language here could be null, blank, or en_US
To query metadata with these properties, you can do:

$ curl -s -H "accept: application/json" -H "Content-Type: application/json" -X POST "http://localhost:8080/rest/items/find-by-metadata-field" -d '{"key": "cg.subject.ilri","value": "SEEDS"}' | jq length
55
$ curl -s -H "accept: application/json" -H "Content-Type: application/json" -X POST "http://localhost:8080/rest/items/find-by-metadata-field" -d '{"key": "cg.subject.ilri","value": "SEEDS", "language":""}' | jq length
34
$ curl -s -H "accept: application/json" -H "Content-Type: application/json" -X POST "http://localhost:8080/rest/items/find-by-metadata-field" -d '{"key": "cg.subject.ilri","value": "SEEDS", "language":"en_US"}' | jq length

The results (55+34=89) don’t seem to match those from the database:

dspace=# select count(text_value) from metadatavalue where resource_type_id=2 and metadata_field_id=203 and text_value='SEEDS' and text_lang is null;
 count
-------
    15
dspace=# select count(text_value) from metadatavalue where resource_type_id=2 and metadata_field_id=203 and text_value='SEEDS' and text_lang='';
 count
-------
     4
dspace=# select count(text_value) from metadatavalue where resource_type_id=2 and metadata_field_id=203 and text_value='SEEDS' and text_lang='en_US';
 count
-------
    66

So, querying from the API I get 55 + 34 = 89 results, but the database actually only has 85…
And the find-by-metadata-field endpoint doesn’t seem to have a way to get all items with the field, or a wildcard value
I’ll ask a question on the dspace-tech mailing list
And speaking of text_lang, this is interesting:

dspacetest=# select distinct text_lang from metadatavalue where resource_type_id=2;
 text_lang
-----------

 ethnob
 en
 spa
 EN
 es
 frn
 en_
 en_US

 EN_US
 eng
 en_U
 fr
(14 rows)

Generate a list of all these so I can maybe fix them in batch:

dspace=# \copy (select distinct text_lang, count(*) from metadatavalue where resource_type_id=2 group by text_lang order by count desc) to /tmp/text-langs.csv with csv;
COPY 14

Perhaps we need to fix them all in batch, or experiment with fixing only certain metadatavalues:

dspace=# update metadatavalue set text_lang='en_US' where resource_type_id=2 and metadata_field_id=203 and text_value='SEEDS';
UPDATE 85

The fix-metadata.py script I have is meant for specific metadata values, so if I want to update some text_lang values I should just do it directly in the database
For example, on a limited set:

dspace=# update metadatavalue set text_lang=NULL where resource_type_id=2 and metadata_field_id=203 and text_value='LIVESTOCK' and text_lang='';
UPDATE 420

And assuming I want to do it for all fields:

dspacetest=# update metadatavalue set text_lang=NULL where resource_type_id=2 and text_lang='';
UPDATE 183726

After that restarted Tomcat and PostgreSQL (because I’m superstitious about caches) and now I see the following in REST API query:

$ curl -s -H "accept: application/json" -H "Content-Type: application/json" -X POST "http://localhost:8080/rest/items/find-by-metadata-field" -d '{"key": "cg.subject.ilri","value": "SEEDS"}' | jq length
71
$ curl -s -H "accept: application/json" -H "Content-Type: application/json" -X POST "http://localhost:8080/rest/items/find-by-metadata-field" -d '{"key": "cg.subject.ilri","value": "SEEDS", "language":""}' | jq length
0
$ curl -s -H "accept: application/json" -H "Content-Type: application/json" -X POST "http://localhost:8080/rest/items/find-by-metadata-field" -d '{"key": "cg.subject.ilri","value": "SEEDS", "language":"en_US"}' | jq length

Not sure what’s going on, but Discovery shows 83 values, and database shows 85, so I’m going to reindex Discovery just in case

2016-11-14

I applied Atmire’s suggestions to fix Listings and Reports for DSpace 5.5 and now it works
There were some issues with the dspace/modules/jspui/pom.xml, which is annoying because all I did was rebase our working 5.1 code on top of 5.5, meaning Atmire’s installation procedure must have changed
So there is apparently this Tomcat native way to limit web crawlers to one session: Crawler Session Manager
After adding that to server.xml bots matching the pattern in the configuration will all use ONE session, just like normal users:

$ http --print h https://dspacetest.cgiar.org 'User-Agent:Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
HTTP/1.1 200 OK
Connection: keep-alive
Content-Encoding: gzip
Content-Language: en-US
Content-Type: text/html;charset=utf-8
Date: Mon, 14 Nov 2016 19:47:29 GMT
Server: nginx
Set-Cookie: JSESSIONID=323694E079A53D5D024F839290EDD7E8; Path=/; Secure; HttpOnly
Transfer-Encoding: chunked
Vary: Accept-Encoding
X-Cocoon-Version: 2.2.0
X-Robots-Tag: none

$ http --print h https://dspacetest.cgiar.org 'User-Agent:Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
HTTP/1.1 200 OK
Connection: keep-alive
Content-Encoding: gzip
Content-Language: en-US
Content-Type: text/html;charset=utf-8
Date: Mon, 14 Nov 2016 19:47:35 GMT
Server: nginx
Transfer-Encoding: chunked
Vary: Accept-Encoding
X-Cocoon-Version: 2.2.0

The first one gets a session, and any after that — within 60 seconds — will be internally mapped to the same session by Tomcat
This means that when Google or Baidu slam you with tens of concurrent connections they will all map to ONE internal session, which saves RAM!

2016-11-15

The Tomcat JVM heap looks really good after applying the Crawler Session Manager fix on DSpace Test last night:

Seems the default regex doesn’t catch Baidu, though:

$ http --print h https://dspacetest.cgiar.org 'User-Agent:Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)'
HTTP/1.1 200 OK
Connection: keep-alive
Content-Encoding: gzip
Content-Language: en-US
Content-Type: text/html;charset=utf-8
Date: Tue, 15 Nov 2016 08:49:54 GMT
Server: nginx
Set-Cookie: JSESSIONID=131409D143E8C01DE145C50FC748256E; Path=/; Secure; HttpOnly
Transfer-Encoding: chunked
Vary: Accept-Encoding
X-Cocoon-Version: 2.2.0

$ http --print h https://dspacetest.cgiar.org 'User-Agent:Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)'
HTTP/1.1 200 OK
Connection: keep-alive
Content-Encoding: gzip
Content-Language: en-US
Content-Type: text/html;charset=utf-8
Date: Tue, 15 Nov 2016 08:49:59 GMT
Server: nginx
Set-Cookie: JSESSIONID=F6403C084480F765ED787E41D2521903; Path=/; Secure; HttpOnly
Transfer-Encoding: chunked
Vary: Accept-Encoding
X-Cocoon-Version: 2.2.0

Adding Baiduspider to the list of user agents seems to work, and the final configuration should be:

<!-- Crawler Session Manager Valve helps mitigate damage done by web crawlers -->
<Valve className="org.apache.catalina.valves.CrawlerSessionManagerValve"
       crawlerUserAgents=".*[bB]ot.*|.*Yahoo! Slurp.*|.*Feedfetcher-Google.*|.*Baiduspider.*" />

Looking at the bots that were active yesterday it seems the above regex should be sufficient:

$ grep -o -E 'Mozilla/5\.0 \(compatible;.*\"' /var/log/nginx/access.log | sort | uniq
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" "-"
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "-"
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" "-"
Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)" "-"

Neat maven trick to exclude some modules from being built:

$ mvn -U -Dmirage2.on=true -Dmirage2.deps.included=false -Denv=localhost -P \!dspace-lni,\!dspace-rdf,\!dspace-sword,\!dspace-swordv2 clean package

We absolutely don’t use those modules, so we shouldn’t build them in the first place

2016-11-17

Generate a list of journal titles for Peter and Abenet to look through so we can make a controlled vocabulary out of them:

dspace=# \copy (select distinct text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=55 group by text_value order by count desc) to /tmp/journal-titles.csv with csv;
COPY 2515

Send a message to users of the CGSpace REST API to notify them of upcoming upgrade so they can test their apps against DSpace Test
Test an update old, non-HTTPS links to the CCAFS website in CGSpace metadata:

dspacetest=# update metadatavalue set text_value = regexp_replace(text_value, 'http://ccafs.cgiar.org','https://ccafs.cgiar.org') where resource_type_id=2 and text_value like '%http://ccafs.cgiar.org%';
UPDATE 164
dspace=# update metadatavalue set text_value = regexp_replace(text_value, 'http://ccafs.cgiar.org','https://ccafs.cgiar.org') where resource_type_id=2 and text_value like '%http://ccafs.cgiar.org%';
UPDATE 7

Had to run it twice to get all (not sure about “global” regex in PostgreSQL)
Run the updates on CGSpace as well
Run through some collections and manually regenerate some PDF thumbnails for items from before 2016 on DSpace Test to compare with CGSpace
I’m debating forcing the re-generation of ALL thumbnails, since some come from DSpace 3 and 4 when the thumbnailing wasn’t as good
The results were very good, I think that after we upgrade to 5.5 I will do it, perhaps one community / collection at a time:

$ [dspace]/bin/dspace filter-media -f -i 10568/67156 -p "ImageMagick PDF Thumbnail"

In related news, I’m looking at thumbnails of thumbnails (the ones we uploaded manually before, and now DSpace’s media filter has made thumbnails of THEM):

dspace=# select text_value from metadatavalue where text_value like '%.jpg.jpg';

I’m not sure if there’s anything we can do, actually, because we would have to remove those from the thumbnail bundles, and replace them with the regular JPGs from the content bundle, and then remove them from the assetstore…

2016-11-18

Enable Tomcat Crawler Session Manager on CGSpace

2016-11-21

More work on Ansible playbooks for PostgreSQL 9.3→9.5 and Java 7→8 work
CGSpace virtual managers meeting
I need to look into making the item thumbnail clickable
Macaroni Bros said they tested the DSpace Test (DSpace 5.5) REST API for CCAFS and WLE sites and it works as expected

2016-11-23

Upgrade Java from 7 to 8 on CGSpace
I had started planning the inplace PostgreSQL 9.3→9.5 upgrade but decided that I will have to pg_dump and pg_restore when I move to the new server soon anyways, so there’s no need to upgrade the database right now
Chat with Carlos about CGCore and the CGSpace metadata registry
Dump CGSpace metadata field registry for Carlos: https://gist.github.com/alanorth/8cbd0bb2704d4bbec78025b4742f8e70
Send some feedback to Carlos on CG Core so they can better understand how DSpace/CGSpace uses metadata
Notes about PostgreSQL tuning from James: https://paste.fedoraproject.org/488776/14798952/
Play with Creative Commons stuff in DSpace submission step
It seems to work but it doesn’t let you choose a version of CC (like 4.0), and we would need to customize the XMLUI item display so it doesn’t display the gross CC badges

2016-11-24

Bizuwork was testing DSpace Test on DSPace 5.5 and noticed that the Listings and Reports module seems to be case sensitive, whereas CGSpace’s Listings and Reports isn’t (ie, a search for “orth, alan” vs “Orth, Alan” returns the same results on CGSpace, but different on DSpace Test)
I have raised a ticket with Atmire
Looks like this issue is actually the new Listings and Reports module honoring the Solr search queries more correctly

2016-11-27

Run system updates on DSpace Test and reboot the server
Deploy DSpace 5.5 on CGSpace:
- maven package
- stop tomcat
- backup postgresql
- run Atmire 5.5 schema deletions
- delete the deployed spring folder
- ant update
- run system updates
- reboot server
Need to do updates for ansible infrastructure role defaults, and switch the GitHub branch to the new 5.5 one
Testing DSpace 5.5 on CGSpace, it seems CUA’s export as XLS works for Usage statistics, but not Content statistics
I will raise a bug with Atmire

2016-11-28

One user says they are still getting a blank page when he logs in (just CGSpace header, but no community list)
Looking at the Catlina logs I see there is some super long-running indexing process going on:

INFO: FrameworkServlet 'oai': initialization completed in 2600 ms
[>                                                  ] 0% time remaining: Calculating... timestamp: 2016-11-28 03:00:18
[>                                                  ] 0% time remaining: 11 hour(s) 57 minute(s) 46 seconds. timestamp: 2016-11-28 03:00:19
[>                                                  ] 0% time remaining: 23 hour(s) 4 minute(s) 28 seconds. timestamp: 2016-11-28 03:00:19
[>                                                  ] 0% time remaining: 15 hour(s) 35 minute(s) 23 seconds. timestamp: 2016-11-28 03:00:19
[>                                                  ] 0% time remaining: 14 hour(s) 5 minute(s) 56 seconds. timestamp: 2016-11-28 03:00:19
[>                                                  ] 0% time remaining: 11 hour(s) 23 minute(s) 49 seconds. timestamp: 2016-11-28 03:00:19
[>                                                  ] 0% time remaining: 11 hour(s) 21 minute(s) 57 seconds. timestamp: 2016-11-28 03:00:20

It says OAI, and seems to start at 3:00 AM, but I only see the filter-media cron job set to start then
Double checking the DSpace 5.x upgrade notes for anything I missed, or troubleshooting tips
Running some manual processes just in case:

$ /home/dspacetest.cgiar.org/bin/dspace registry-loader -metadata /home/dspacetest.cgiar.org/config/registries/dcterms-types.xml
$ /home/dspacetest.cgiar.org/bin/dspace registry-loader -metadata /home/dspacetest.cgiar.org/config/registries/dublin-core-types.xml
$ /home/dspacetest.cgiar.org/bin/dspace registry-loader -metadata /home/dspacetest.cgiar.org/config/registries/eperson-types.xml
$ /home/dspacetest.cgiar.org/bin/dspace registry-loader -metadata /home/dspacetest.cgiar.org/config/registries/workflow-types.xml

Start working on paper for KM4Dev journal
Wow, Bram from Atmire pointed out this solution for using multiple handles with one DSpace instance: https://wiki.duraspace.org/display/DSDOC5x/Installing+DSpace?focusedCommentId=78163296#comment-78163296
We might be able to migrate the CGIAR Library now, as they had wanted to keep their handles

2016-11-29

Sisay tried deleting and re-creating Goshu’s account but he still can’t see any communities on the homepage after he logs in
Around the time of his login I see this in the DSpace logs:

2016-11-29 07:56:36,350 INFO  org.dspace.authenticate.LDAPAuthentication @ g.cherinet@cgiar.org:session_id=F628E13AB4EF2BA949198A99EFD8EBE4:ip_addr=213.55.99.121:failed_login:no DN found for user g.cherinet@cgiar.org
2016-11-29 07:56:36,350 INFO  org.dspace.authenticate.PasswordAuthentication @ g.cherinet@cgiar.org:session_id=F628E13AB4EF2BA949198A99EFD8EBE4:ip_addr=213.55.99.121:authenticate:attempting password auth of user=g.cherinet@cgiar.org
2016-11-29 07:56:36,352 INFO  org.dspace.app.xmlui.utils.AuthenticationUtil @ g.cherinet@cgiar.org:session_id=F628E13AB4EF2BA949198A99EFD8EBE4:ip_addr=213.55.99.121:failed_login:email=g.cherinet@cgiar.org, realm=null, result=2
2016-11-29 07:56:36,545 INFO  com.atmire.utils.UpdateSolrStatsMetadata @ Start processing item 10568/50391 id:51744
2016-11-29 07:56:36,545 INFO  com.atmire.utils.UpdateSolrStatsMetadata @ Processing item stats
2016-11-29 07:56:36,583 INFO  com.atmire.utils.UpdateSolrStatsMetadata @ Solr metadata up-to-date
2016-11-29 07:56:36,583 INFO  com.atmire.utils.UpdateSolrStatsMetadata @ Processing item's bitstream stats
2016-11-29 07:56:36,608 INFO  com.atmire.utils.UpdateSolrStatsMetadata @ Solr metadata up-to-date
2016-11-29 07:56:36,701 INFO  org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ facets for scope, null: 23
2016-11-29 07:56:36,747 ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ Error while searching for sidebar facets
org.dspace.discovery.SearchServiceException: Error executing query
        at org.dspace.discovery.SolrServiceImpl.search(SolrServiceImpl.java:1618)
        at org.dspace.discovery.SolrServiceImpl.search(SolrServiceImpl.java:1600)
        at org.dspace.discovery.SolrServiceImpl.search(SolrServiceImpl.java:1583)
        at org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer.performSearch(SidebarFacetsTransformer.java:165)
        at org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer.addOptions(SidebarFacetsTransformer.java:174)
        at org.dspace.app.xmlui.wing.AbstractWingTransformer.startElement(AbstractWingTransformer.java:228)
        at sun.reflect.GeneratedMethodAccessor277.invoke(Unknown Source)
...

At about the same time in the solr log I see a super long query:

2016-11-29 07:56:36,734 INFO  org.apache.solr.core.SolrCore @ [search] webapp=/solr path=/select params={q=*:*&fl=dateIssued.year,handle,search.resourcetype,search.resourceid,search.uniqueid&start=0&fq=NOT(withdrawn:true)&fq=NOT(discoverable:false)&fq=dateIssued.year:[*+TO+*]&fq=read:(g0+OR+e574+OR+g0+OR+g3+OR+g9+OR+g10+OR+g14+OR+g16+OR+g18+OR+g20+OR+g23+OR+g24+OR+g2072+OR+g2074+OR+g28+OR+g2076+OR+g29+OR+g2078+OR+g2080+OR+g34+OR+g2082+OR+g2084+OR+g38+OR+g2086+OR+g2088+OR+g2091+OR+g43+OR+g2092+OR+g2093+OR+g2095+OR+g2097+OR+g50+OR+g2099+OR+g51+OR+g2103+OR+g62+OR+g65+OR+g2115+OR+g2117+OR+g2119+OR+g2121+OR+g2123+OR+g2125+OR+g77+OR+g78+OR+g79+OR+g2127+OR+g80+OR+g2129+OR+g2131+OR+g2133+OR+g2134+OR+g2135+OR+g2136+OR+g2137+OR+g2138+OR+g2139+OR+g2140+OR+g2141+OR+g2142+OR+g2148+OR+g2149+OR+g2150+OR+g2151+OR+g2152+OR+g2153+OR+g2154+OR+g2156+OR+g2165+OR+g2167+OR+g2171+OR+g2174+OR+g2175+OR+g129+OR+g2182+OR+g2186+OR+g2189+OR+g153+OR+g158+OR+g166+OR+g167+OR+g168+OR+g169+OR+g2225+OR+g179+OR+g2227+OR+g2229+OR+g183+OR+g2231+OR+g184+OR+g2233+OR+g186+OR+g2235+OR+g2237+OR+g191+OR+g192+OR+g193+OR+g202+OR+g203+OR+g204+OR+g205+OR+g207+OR+g208+OR+g218+OR+g219+OR+g222+OR+g223+OR+g230+OR+g231+OR+g238+OR+g241+OR+g244+OR+g254+OR+g255+OR+g262+OR+g265+OR+g268+OR+g269+OR+g273+OR+g276+OR+g277+OR+g279+OR+g282+OR+g2332+OR+g2335+OR+g2338+OR+g292+OR+g293+OR+g2341+OR+g296+OR+g2344+OR+g297+OR+g2347+OR+g301+OR+g2350+OR+g303+OR+g305+OR+g2356+OR+g310+OR+g311+OR+g2359+OR+g313+OR+g2362+OR+g2365+OR+g2368+OR+g321+OR+g2371+OR+g325+OR+g2374+OR+g328+OR+g2377+OR+g2380+OR+g333+OR+g2383+OR+g2386+OR+g2389+OR+g342+OR+g343+OR+g2392+OR+g345+OR+g2395+OR+g348+OR+g2398+OR+g2401+OR+g2404+OR+g2407+OR+g364+OR+g366+OR+g2425+OR+g2427+OR+g385+OR+g387+OR+g388+OR+g389+OR+g2442+OR+g395+OR+g2443+OR+g2444+OR+g401+OR+g403+OR+g405+OR+g408+OR+g2457+OR+g2458+OR+g411+OR+g2459+OR+g414+OR+g2463+OR+g417+OR+g2465+OR+g2467+OR+g421+OR+g2469+OR+g2471+OR+g424+OR+g2473+OR+g2475+OR+g2476+OR+g429+OR+g433+OR+g2481+OR+g2482+OR+g2483+OR+g443+OR+g444+OR+g445+OR+g446+OR+g448+OR+g453+OR+g455+OR+g456+OR+g457+OR+g458+OR+g459+OR+g461+OR+g462+OR+g463+OR+g464+OR+g465+OR+g467+OR+g468+OR+g469+OR+g474+OR+g476+OR+g477+OR+g480+OR+g483+OR+g484+OR+g493+OR+g496+OR+g497+OR+g498+OR+g500+OR+g502+OR+g504+OR+g505+OR+g2559+OR+g2560+OR+g513+OR+g2561+OR+g515+OR+g516+OR+g518+OR+g519+OR+g2567+OR+g520+OR+g521+OR+g522+OR+g2570+OR+g523+OR+g2571+OR+g524+OR+g525+OR+g2573+OR+g526+OR+g2574+OR+g527+OR+g528+OR+g2576+OR+g529+OR+g531+OR+g2579+OR+g533+OR+g534+OR+g2582+OR+g535+OR+g2584+OR+g538+OR+g2586+OR+g540+OR+g2588+OR+g541+OR+g543+OR+g544+OR+g545+OR+g546+OR+g548+OR+g2596+OR+g549+OR+g551+OR+g555+OR+g556+OR+g558+OR+g561+OR+g569+OR+g570+OR+g571+OR+g2619+OR+g572+OR+g2620+OR+g573+OR+g2621+OR+g2622+OR+g575+OR+g578+OR+g581+OR+g582+OR+g584+OR+g585+OR+g586+OR+g587+OR+g588+OR+g590+OR+g591+OR+g593+OR+g595+OR+g596+OR+g598+OR+g599+OR+g601+OR+g602+OR+g603+OR+g604+OR+g605+OR+g606+OR+g608+OR+g609+OR+g610+OR+g612+OR+g614+OR+g616+OR+g620+OR+g621+OR+g623+OR+g630+OR+g635+OR+g636+OR+g646+OR+g649+OR+g683+OR+g684+OR+g687+OR+g689+OR+g691+OR+g695+OR+g697+OR+g698+OR+g699+OR+g700+OR+g701+OR+g707+OR+g708+OR+g709+OR+g710+OR+g711+OR+g712+OR+g713+OR+g714+OR+g715+OR+g716+OR+g717+OR+g719+OR+g720+OR+g729+OR+g732+OR+g733+OR+g734+OR+g736+OR+g737+OR+g738+OR+g2786+OR+g752+OR+g754+OR+g2804+OR+g757+OR+g2805+OR+g2806+OR+g760+OR+g761+OR+g2810+OR+g2815+OR+g769+OR+g771+OR+g773+OR+g776+OR+g786+OR+g787+OR+g788+OR+g789+OR+g791+OR+g792+OR+g793+OR+g794+OR+g795+OR+g796+OR+g798+OR+g800+OR+g802+OR+g803+OR+g806+OR+g808+OR+g810+OR+g814+OR+g815+OR+g817+OR+g829+OR+g830+OR+g849+OR+g893+OR+g895+OR+g898+OR+g902+OR+g903+OR+g917+OR+g919+OR+g921+OR+g922+OR+g923+OR+g924+OR+g925+OR+g926+OR+g927+OR+g928+OR+g929+OR+g930+OR+g932+OR+g933+OR+g934+OR+g938+OR+g939+OR+g944+OR+g945+OR+g946+OR+g947+OR+g948+OR+g949+OR+g950+OR+g951+OR+g953+OR+g954+OR+g955+OR+g956+OR+g958+OR+g959+OR+g960+OR+g963+OR+g964+OR+g965+OR+g968+OR+g969+OR+g970+OR+g971+OR+g972+OR+g973+OR+g974+OR+g976+OR+g978+OR+g979+OR+g984+OR+g985+OR+g987+OR+g988+OR+g991+OR+g993+OR+g994+OR+g999+OR+g1000+OR+g1003+OR+g1005+OR+g1006+OR+g1007+OR+g1012+OR+g1013+OR+g1015+OR+g1016+OR+g1018+OR+g1023+OR+g1024+OR+g1026+OR+g1028+OR+g1030+OR+g1032+OR+g1033+OR+g1035+OR+g1036+OR+g1038+OR+g1039+OR+g1041+OR+g1042+OR+g1044+OR+g1045+OR+g1047+OR+g1048+OR+g1050+OR+g1051+OR+g1053+OR+g1054+OR+g1056+OR+g1057+OR+g1058+OR+g1059+OR+g1060+OR+g1061+OR+g1062+OR+g1063+OR+g1064+OR+g1065+OR+g1066+OR+g1068+OR+g1071+OR+g1072+OR+g1074+OR+g1075+OR+g1076+OR+g1077+OR+g1078+OR+g1080+OR+g1081+OR+g1082+OR+g1084+OR+g1085+OR+g1087+OR+g1088+OR+g1089+OR+g1090+OR+g1091+OR+g1092+OR+g1093+OR+g1094+OR+g1095+OR+g1096+OR+g1097+OR+g1106+OR+g1108+OR+g1110+OR+g1112+OR+g1114+OR+g1117+OR+g1120+OR+g1121+OR+g1126+OR+g1128+OR+g1129+OR+g1131+OR+g1136+OR+g1138+OR+g1140+OR+g1141+OR+g1143+OR+g1145+OR+g1146+OR+g1148+OR+g1152+OR+g1154+OR+g1156+OR+g1158+OR+g1159+OR+g1160+OR+g1162+OR+g1163+OR+g1165+OR+g1166+OR+g1168+OR+g1170+OR+g1172+OR+g1175+OR+g1177+OR+g1179+OR+g1181+OR+g1185+OR+g1191+OR+g1193+OR+g1197+OR+g1199+OR+g1201+OR+g1203+OR+g1204+OR+g1215+OR+g1217+OR+g1219+OR+g1221+OR+g1224+OR+g1226+OR+g1227+OR+g1228+OR+g1230+OR+g1231+OR+g1232+OR+g1233+OR+g1234+OR+g1235+OR+g1236+OR+g1237+OR+g1238+OR+g1240+OR+g1241+OR+g1242+OR+g1243+OR+g1244+OR+g1246+OR+g1248+OR+g1250+OR+g1252+OR+g1254+OR+g1256+OR+g1257+OR+g1259+OR+g1261+OR+g1263+OR+g1275+OR+g1276+OR+g1277+OR+g1278+OR+g1279+OR+g1282+OR+g1284+OR+g1288+OR+g1290+OR+g1293+OR+g1296+OR+g1297+OR+g1299+OR+g1303+OR+g1304+OR+g1306+OR+g1309+OR+g1310+OR+g1311+OR+g1312+OR+g1313+OR+g1316+OR+g1318+OR+g1320+OR+g1322+OR+g1323+OR+g1324+OR+g1325+OR+g1326+OR+g1329+OR+g1331+OR+g1347+OR+g1348+OR+g1361+OR+g1362+OR+g1363+OR+g1364+OR+g1367+OR+g1368+OR+g1369+OR+g1370+OR+g1371+OR+g1374+OR+g1376+OR+g1377+OR+g1378+OR+g1380+OR+g1381+OR+g1386+OR+g1389+OR+g1391+OR+g1392+OR+g1393+OR+g1395+OR+g1396+OR+g1397+OR+g1400+OR+g1402+OR+g1406+OR+g1408+OR+g1415+OR+g1417+OR+g1433+OR+g1435+OR+g1441+OR+g1442+OR+g1443+OR+g1444+OR+g1446+OR+g1448+OR+g1450+OR+g1451+OR+g1452+OR+g1453+OR+g1454+OR+g1456+OR+g1458+OR+g1460+OR+g1462+OR+g1464+OR+g1466+OR+g1468+OR+g1470+OR+g1471+OR+g1475+OR+g1476+OR+g1477+OR+g1478+OR+g1479+OR+g1481+OR+g1482+OR+g1483+OR+g1484+OR+g1485+OR+g1486+OR+g1487+OR+g1488+OR+g1489+OR+g1490+OR+g1491+OR+g1492+OR+g1493+OR+g1495+OR+g1497+OR+g1499+OR+g1501+OR+g1503+OR+g1504+OR+g1506+OR+g1508+OR+g1511+OR+g1512+OR+g1513+OR+g1516+OR+g1522+OR+g1535+OR+g1536+OR+g1537+OR+g1539+OR+g1540+OR+g1541+OR+g1542+OR+g1547+OR+g1549+OR+g1551+OR+g1553+OR+g1555+OR+g1557+OR+g1559+OR+g1561+OR+g1563+OR+g1565+OR+g1567+OR+g1569+OR+g1571+OR+g1573+OR+g1580+OR+g1583+OR+g1588+OR+g1590+OR+g1592+OR+g1594+OR+g1595+OR+g1596+OR+g1598+OR+g1599+OR+g1600+OR+g1601+OR+g1602+OR+g1604+OR+g1606+OR+g1610+OR+g1611+OR+g1612+OR+g1613+OR+g1616+OR+g1619+OR+g1622+OR+g1624+OR+g1625+OR+g1626+OR+g1628+OR+g1629+OR+g1631+OR+g1632+OR+g1692+OR+g1694+OR+g1695+OR+g1697+OR+g1705+OR+g1706+OR+g1707+OR+g1708+OR+g1711+OR+g1715+OR+g1717+OR+g1719+OR+g1721+OR+g1722+OR+g1723+OR+g1724+OR+g1725+OR+g1726+OR+g1727+OR+g1731+OR+g1732+OR+g1736+OR+g1737+OR+g1738+OR+g1740+OR+g1742+OR+g1743+OR+g1753+OR+g1755+OR+g1758+OR+g1759+OR+g1764+OR+g1766+OR+g1769+OR+g1774+OR+g1782+OR+g1794+OR+g1796+OR+g1797+OR+g1814+OR+g1818+OR+g1826+OR+g1853+OR+g1855+OR+g1857+OR+g1858+OR+g1859+OR+g1860+OR+g1861+OR+g1863+OR+g1864+OR+g1865+OR+g1867+OR+g1869+OR+g1871+OR+g1873+OR+g1875+OR+g1877+OR+g1879+OR+g1881+OR+g1883+OR+g1884+OR+g1885+OR+g1887+OR+g1889+OR+g1891+OR+g1892+OR+g1894+OR+g1896+OR+g1898+OR+g1900+OR+g1902+OR+g1907+OR+g1910+OR+g1915+OR+g1916+OR+g1917+OR+g1918+OR+g1929+OR+g1931+OR+g1932+OR+g1933+OR+g1934+OR+g1936+OR+g1937+OR+g1938+OR+g1939+OR+g1940+OR+g1942+OR+g1944+OR+g1945+OR+g1948+OR+g1950+OR+g1955+OR+g1961+OR+g1962+OR+g1964+OR+g1966+OR+g1968+OR+g1970+OR+g1972+OR+g1974+OR+g1976+OR+g1979+OR+g1982+OR+g1984+OR+g1985+OR+g1986+OR+g1987+OR+g1989+OR+g1991+OR+g1996+OR+g2003+OR+g2007+OR+g2011+OR+g2019+OR+g2020+OR+g2046)&sort=dateIssued.year_sort+desc&rows=1&wt=javabin&version=2} hits=56080 status=0 QTime=3

Which, according to some old threads on DSpace Tech, means that the user has a lot of permissions (from groups or on the individual eperson) which increases the Solr query size / query URL
It might be fixed by increasing the Tomcat maxHttpHeaderSize, which is 8192 (or 8KB) by default
I’ve increased the maxHttpHeaderSize to 16384 on DSpace Test and the user said he is now able to see the communities on the homepage
I will make the changes on CGSpace soon
A few users are reporting having issues with their workflows, they get the following message: “You are not allowed to perform this task”
Might be the same as DS-2920 on the bug tracker

2016-11-30

The maxHttpHeaderSize fix worked on CGSpace (user is able to see the community list on the homepage)
The “take task” cache fix worked on DSpace Test but it’s not an official patch, so I’ll have to report the bug to DSpace people and try to get advice
More work on the KM4Dev Journal article

October, 2016

Mon, 03 Oct 2016 15:53:00 +0300

2016-10-03

Testing adding ORCIDs to a CSV file for a single item to see if the author orders get messed up
Need to test the following scenarios to see how author order is affected:
- ORCIDs only
- ORCIDs plus normal authors
I exported a random item’s metadata as CSV, deleted all columns except id and collection, and made a new coloum called ORCID:dc.contributor.author with the following random ORCIDs from the ORCID registry:

0000-0002-6115-0956||0000-0002-3812-8793||0000-0001-7462-405X

Hmm, with the dc.contributor.author column removed, DSpace doesn’t detect any changes
With a blank dc.contributor.author column, DSpace wants to remove all non-ORCID authors and add the new ORCID authors
I added the disclaimer text to the About page, then added a footer link to the disclaimer’s ID, but there is a Bootstrap issue that causes the page content to disappear when using in-page anchors: https://github.com/twbs/bootstrap/issues/1768

Looks like we’ll just have to add the text to the About page (without a link) or add a separate page

2016-10-04

Start testing cleanups of authors that Peter sent last week
Out of 40,000+ rows, Peter had indicated corrections for ~3,200 of them—too many to look through carefully, so I did some basic quality checking:
- Trim leading/trailing whitespace
- Find invalid characters
- Cluster values to merge obvious authors
That left us with 3,180 valid corrections and 3 deletions:

$ ./fix-metadata-values.py -i authors-fix-3180.csv -f dc.contributor.author -t correct -m 3 -d dspacetest -u dspacetest -p fuuu
$ ./delete-metadata-values.py -i authors-delete-3.csv -f dc.contributor.author -m 3 -d dspacetest -u dspacetest -p fuuu

Remove old about page (#284)
CGSpace crashed a few times today
Generate list of unique authors in CCAFS collections:

dspacetest=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 AND resource_id IN (select item_id from collection2item where collection_id IN (select resource_id from handle where handle in ('10568/32729', '10568/5472', '10568/5473', '10568/10288', '10568/70974', '10568/3547', '10568/3549', '10568/3531','10568/16890','10568/5470','10568/3546', '10568/36024', '10568/66581', '10568/21789', '10568/5469', '10568/5468', '10568/3548', '10568/71053', '10568/25167'))) group by text_value order by count desc) to /tmp/ccafs-authors.csv with csv;

2016-10-05

Work on more infrastructure cleanups for Ansible DSpace role
Clean up Let’s Encrypt plumbing and submit pull request for rmg-ansible-public (#60)

2016-10-06

Nice! DSpace Test (linode02) is now having java.lang.OutOfMemoryError: Java heap space errors…
Heap space is 2048m, and we have 5GB of RAM being used for OS cache (Solr!) so let’s just bump the memory to 3072m
Magdalena from CCAFS asked why the colors in the thumbnails for these two items look different, even though they are the same in the PDF itself

Turns out the first PDF was exported from InDesign using CMYK and the second one was using sRGB
Run all system updates on DSpace Test and reboot it

2016-10-08

Re-deploy CGSpace with latest changes from late September and early October
Run fixes for ILRI subjects and delete blank metadata values:

dspace=# delete from metadatavalue where resource_type_id=2 and text_value='';
DELETE 11

Run all system updates and reboot CGSpace
Delete ten gigs of old 2015 Tomcat logs that never got rotated (WTF?):

root@linode01:~# ls -lh /var/log/tomcat7/localhost_access_log.2015* | wc -l
47

Delete 2GB cron-filter-media.log file, as it is just a log from a cron job and it doesn’t get rotated like normal log files (almost a year now maybe)

2016-10-14

Run all system updates on DSpace Test and reboot server
Looking into some issues with Discovery filters in Atmire’s content and usage analysis module after adjusting the filter class
Looks like changing the filters from configuration.DiscoverySearchFilterFacet to configuration.DiscoverySearchFilter breaks them in Atmire CUA module

2016-10-17

A bit more cleanup on the CCAFS authors, and run the corrections on DSpace Test:

$ ./fix-metadata-values.py -i ccafs-authors-oct-16.csv -f dc.contributor.author -t 'correct name' -m 3 -d dspace -u dspace -p fuuu

One observation is that there are still some old versions of names in the author lookup because authors appear in other communities (as we only corrected authors from CCAFS for this round)

2016-10-18

Start working on DSpace 5.5 porting work again:

$ git checkout -b 5_x-55 5_x-prod $ git rebase -i dspace-5.5
Have to fix about ten merge conflicts, mostly in the SCSS for the CGIAR theme
Skip 1e34751b8cf17021f45d4cf2b9a5800c93fb4cb2 in lieu of upstream’s 55e623d1c2b8b7b1fa45db6728e172e06bfa8598 (fixes X-Forwarded-For header) because I had made the same fix myself and it’s better to use the upstream one
I notice this rebase gets rid of GitHub merge commits… which actually might be fine because merges are fucking annoying to deal with when remote people merge without pulling and rebasing their branch first
Finished up applying the 5.5 sitemap changes to all themes
Merge the discovery.xml cleanups (#278)
Merge some minor edits to the distribution license (#285)

2016-10-19

When we move to DSpace 5.5 we should also cherry pick some patches from 5.6 branch:
- memory cleanup: 9f0f5940e7921765c6a22e85337331656b18a403
- sql injection: c6fda557f731dbc200d7d58b8b61563f86fe6d06
- pdfbox security issue: b5330b78153b2052ed3dc2fd65917ccdbfcc0439

2016-10-20

Run CCAFS author corrections on CGSpace
Discovery reindexing took forever and kinda caused CGSpace to crash, so I ran all system updates and rebooted the server

2016-10-25

Move the LIVES community from the top level to the ILRI projects community

$ /home/cgspace.cgiar.org/bin/dspace community-filiator --set --parent=10568/27629 --child=10568/25101

Start testing some things for DSpace 5.5, like command line metadata import, PDF media filter, and Atmire CUA
Start looking at batch fixing of “old” ILRI website links without www or https, for example:

dspace=# select * from metadatavalue where resource_type_id=2 and text_value like 'http://ilri.org%';

Also CCAFS has HTTPS and their links should use it where possible:

dspace=# select * from metadatavalue where resource_type_id=2 and text_value like 'http://ccafs.cgiar.org%';

And this will find community and collection HTML text that is using the old style PNG/JPG icons for RSS and email (we should be using Font Awesome icons instead):

dspace=# select text_value from metadatavalue where resource_type_id in (3,4) and text_value like '%Iconrss2.png%';

Turns out there are shit tons of varieties of this, like with http, https, www, separate </img> tags, alignments, etc
Had to find all variations and replace them individually:

dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img align="left" src="https://www.ilri.org/images/Iconrss2.png"/>','<span class="fa fa-rss fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img align="left" src="https://www.ilri.org/images/Iconrss2.png"/>%';
dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img align="left" src="https://www.ilri.org/images/email.jpg"/>', '<span class="fa fa-at fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img align="left" src="https://www.ilri.org/images/email.jpg"/>%';
dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img align="left" src="http://www.ilri.org/images/Iconrss2.png"/>', '<span class="fa fa-rss fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img align="left" src="http://www.ilri.org/images/Iconrss2.png"/>%';
dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img align="left" src="http://www.ilri.org/images/email.jpg"/>', '<span class="fa fa-at fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img align="left" src="http://www.ilri.org/images/email.jpg"/>%';
dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img align="left" src="http://www.ilri.org/images/Iconrss2.png"></img>', '<span class="fa fa-rss fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img align="left" src="http://www.ilri.org/images/Iconrss2.png"></img>%';
dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img align="left" src="http://www.ilri.org/images/email.jpg"></img>', '<span class="fa fa-at fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img align="left" src="http://www.ilri.org/images/email.jpg"></img>%';
dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img align="left" src="https://ilri.org/images/Iconrss2.png"></img>', '<span class="fa fa-rss fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img align="left" src="https://ilri.org/images/Iconrss2.png"></img>%';
dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img align="left" src="https://ilri.org/images/email.jpg"></img>', '<span class="fa fa-at fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img align="left" src="https://ilri.org/images/email.jpg"></img>%';
dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img align="left" src="https://www.ilri.org/images/Iconrss2.png"></img>', '<span class="fa fa-rss fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img align="left" src="https://www.ilri.org/images/Iconrss2.png"></img>%';
dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img align="left" src="https://www.ilri.org/images/email.jpg"></img>', '<span class="fa fa-at fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img align="left" src="https://www.ilri.org/images/email.jpg"></img>%';
dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img align="left" src="https://ilri.org/images/Iconrss2.png"/>', '<span class="fa fa-rss fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img align="left" src="https://ilri.org/images/Iconrss2.png"/>%';
dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img align="left" src="https://ilri.org/images/email.jpg"/>', '<span class="fa fa-at fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img align="left" src="https://ilri.org/images/email.jpg"/>%';
dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img valign="center" align="left" src="https://www.ilri.org/images/Iconrss2.png"/>', '<span class="fa fa-rss fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img valign="center" align="left" src="https://www.ilri.org/images/Iconrss2.png"/>%';
dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img valign="center" align="left" src="https://www.ilri.org/images/email.jpg"/>', '<span class="fa fa-at fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img valign="center" align="left" src="https://www.ilri.org/images/email.jpg"/>%';
dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img valign="center" align="left" src="http://www.ilri.org/images/Iconrss2.png"/>', '<span class="fa fa-rss fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img valign="center" align="left" src="http://www.ilri.org/images/Iconrss2.png"/>%';
dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img valign="center" align="left" src="http://www.ilri.org/images/email.jpg"/>', '<span class="fa fa-at fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img valign="center" align="left" src="http://www.ilri.org/images/email.jpg"/>%';

Getting rid of these reduces the number of network requests each client makes on community/collection pages, and makes use of Font Awesome icons (which they are already loading anyways!)
And now that I start looking, I want to fix a bunch of links to popular sites that should be using HTTPS, like Twitter, Facebook, Google, Feed Burner, DOI, etc
I should look to see if any of those domains is sending an HTTP 301 or setting HSTS headers to their HTTPS domains, then just replace them

2016-10-27

Run Font Awesome fixes on DSpace Test:

dspace=# \i /tmp/font-awesome-text-replace.sql
UPDATE 17
UPDATE 17
UPDATE 3
UPDATE 3
UPDATE 30
UPDATE 30
UPDATE 1
UPDATE 1
UPDATE 7
UPDATE 7
UPDATE 1
UPDATE 1
UPDATE 1
UPDATE 1
UPDATE 1
UPDATE 1
UPDATE 0

Looks much better now:

Run the same replacements on CGSpace

2016-10-30

Fix some messed up authors on CGSpace:

dspace=# update metadatavalue set authority='799da1d8-22f3-43f5-8233-3d2ef5ebf8a8', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Charleston, B.%';
UPDATE 10
dspace=# update metadatavalue set authority='e936f5c5-343d-4c46-aa91-7a1fff6277ed', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Knight-Jones%';
UPDATE 36

I updated the authority index but nothing seemed to change, so I’ll wait and do it again after I update Discovery below
Skype chat with Tsega about the IFPRI contentdm bridge
We tested harvesting OAI in an example collection to see how it works
Talk to Carlos Quiros about CG Core metadata in CGSpace
Get a list of countries from CGSpace so I can do some batch corrections:

dspace=# \copy (select distinct text_value, count(*) from metadatavalue where metadata_field_id=228 group by text_value order by count desc) to /tmp/countries.csv with csv;

Fix a bunch of countries in Open Refine and run the corrections on CGSpace:

$ ./fix-metadata-values.py -i countries-fix-18.csv -f dc.coverage.country -t 'correct' -m 228 -d dspace -u dspace -p fuuu
$ ./delete-metadata-values.py -i countries-delete-2.csv -f dc.coverage.country -m 228 -d dspace -u dspace -p fuuu

Run a shit ton of author fixes from Peter Ballantyne that we’ve been cleaning up for two months:

$ ./fix-metadata-values.py -i /tmp/authors-fix-pb2.csv -f dc.contributor.author -t correct -m 3 -u dspace -d dspace -p fuuu

Run a few URL corrections for ilri.org and doi.org, etc:

dspace=# update metadatavalue set text_value = regexp_replace(text_value, 'http://www.ilri.org','https://www.ilri.org') where resource_type_id=2 and text_value like '%http://www.ilri.org%';
dspace=# update metadatavalue set text_value = regexp_replace(text_value, 'http://mahider.ilri.org', 'https://cgspace.cgiar.org') where resource_type_id=2 and text_value like '%http://mahider.%.org%' and metadata_field_id not in (28);
dspace=# update metadatavalue set text_value = regexp_replace(text_value, 'http://dx.doi.org', 'https://dx.doi.org') where resource_type_id=2 and text_value like '%http://dx.doi.org%' and metadata_field_id not in (18,26,28,111);
dspace=# update metadatavalue set text_value = regexp_replace(text_value, 'http://doi.org', 'https://dx.doi.org') where resource_type_id=2 and text_value like '%http://doi.org%' and metadata_field_id not in (18,26,28,111);

I skipped metadata fields like citation and description

September, 2016

Thu, 01 Sep 2016 15:53:00 +0300

2016-09-01

Discuss helping CCAFS with some batch tagging of ORCID IDs for their authors
Discuss how the migration of CGIAR’s Active Directory to a flat structure will break our LDAP groups in DSpace
We had been using DC=ILRI to determine whether a user was ILRI or not
It looks like we might be able to use OUs now, instead of DCs:

$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=org" -D "admigration1@cgiarad.org" -W "(sAMAccountName=admigration1)"

User who has been migrated to the root vs user still in the hierarchical structure:

distinguishedName: CN=Last\, First (ILRI),OU=ILRI Kenya Employees,OU=ILRI Kenya,OU=ILRIHUB,DC=CGIARAD,DC=ORG
distinguishedName: CN=Last\, First (ILRI),OU=ILRI Ethiopia Employees,OU=ILRI Ethiopia,DC=ILRI,DC=CGIARAD,DC=ORG

Changing the DSpace LDAP config to use OU=ILRIHUB seems to work:

Notes for local PostgreSQL database recreation from production snapshot:

$ dropdb dspacetest
$ createdb -O dspacetest --encoding=UNICODE dspacetest
$ psql dspacetest -c 'alter user dspacetest createuser;'
$ pg_restore -O -U dspacetest -d dspacetest ~/Downloads/cgspace_2016-09-01.backup
$ psql dspacetest -c 'alter user dspacetest nocreateuser;'
$ psql -U dspacetest -f ~/src/git/DSpace/dspace/etc/postgres/update-sequences.sql dspacetest -h localhost
$ vacuumdb dspacetest

Some names that I thought I fixed in July seem not to be:

dspacetest=# select distinct text_value, authority, confidence from metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value like 'Poole, %';
      text_value       |              authority               | confidence
-----------------------+--------------------------------------+------------
 Poole, Elizabeth Jane | b6efa27f-8829-4b92-80fe-bc63e03e3ccb |        600
 Poole, Elizabeth Jane | 41628f42-fc38-4b38-b473-93aec9196326 |        600
 Poole, Elizabeth Jane | 83b82da0-f652-4ebc-babc-591af1697919 |        600
 Poole, Elizabeth Jane | c3a22456-8d6a-41f9-bba0-de51ef564d45 |        600
 Poole, E.J.           | c3a22456-8d6a-41f9-bba0-de51ef564d45 |        600
 Poole, E.J.           | 0fbd91b9-1b71-4504-8828-e26885bf8b84 |        600
(6 rows)

At least a few of these actually have the correct ORCID, but I will unify the authority to be c3a22456-8d6a-41f9-bba0-de51ef564d45

dspacetest=# update metadatavalue set authority='c3a22456-8d6a-41f9-bba0-de51ef564d45', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Poole, %';
UPDATE 69

And for Peter Ballantyne:

dspacetest=# select distinct text_value, authority, confidence from metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value like 'Ballantyne, %';
    text_value     |              authority               | confidence
-------------------+--------------------------------------+------------
 Ballantyne, Peter | 2dcbcc7b-47b0-4fd7-bef9-39d554494081 |        600
 Ballantyne, Peter | 4f04ca06-9a76-4206-bd9c-917ca75d278e |        600
 Ballantyne, P.G.  | 4f04ca06-9a76-4206-bd9c-917ca75d278e |        600
 Ballantyne, Peter | ba5f205b-b78b-43e5-8e80-0c9a1e1ad2ca |        600
 Ballantyne, Peter | 20f21160-414c-4ecf-89ca-5f2cb64e75c1 |        600
(5 rows)

Again, a few have the correct ORCID, but there should only be one authority…

dspacetest=# update metadatavalue set authority='4f04ca06-9a76-4206-bd9c-917ca75d278e', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Ballantyne, %';
UPDATE 58

And for me:

dspacetest=# select distinct text_value, authority, confidence from metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value like 'Orth, A%';
 text_value |              authority               | confidence
------------+--------------------------------------+------------
 Orth, Alan | 4884def0-4d7e-4256-9dd4-018cd60a5871 |        600
 Orth, A.   | 4884def0-4d7e-4256-9dd4-018cd60a5871 |        600
 Orth, A.   | 1a1943a0-3f87-402f-9afe-e52fb46a513e |        600
(3 rows)
dspacetest=# update metadatavalue set authority='1a1943a0-3f87-402f-9afe-e52fb46a513e', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Orth, %';
UPDATE 11

And for CCAFS author Bruce Campbell that I had discussed with CCAFS earlier this week:

dspacetest=# update metadatavalue set authority='0e414b4c-4671-4a23-b570-6077aca647d8', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Campbell, B%';
UPDATE 166
dspacetest=# select distinct text_value, authority, confidence from metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value like 'Campbell, B%';
       text_value       |              authority               | confidence
------------------------+--------------------------------------+------------
 Campbell, Bruce        | 0e414b4c-4671-4a23-b570-6077aca647d8 |        600
 Campbell, Bruce Morgan | 0e414b4c-4671-4a23-b570-6077aca647d8 |        600
 Campbell, B.           | 0e414b4c-4671-4a23-b570-6077aca647d8 |        600
 Campbell, B.M.         | 0e414b4c-4671-4a23-b570-6077aca647d8 |        600
(4 rows)

After updating the Authority indexes (bin/dspace index-authority) everything looks good
Run authority updates on CGSpace

2016-09-05

After one week of logging TLS connections on CGSpace:

# zgrep "DES-CBC3" /var/log/nginx/cgspace.cgiar.org-access-ssl.log* | wc -l
217
# zcat -f -- /var/log/nginx/cgspace.cgiar.org-access-ssl.log* | wc -l
1164376
# zgrep "DES-CBC3" /var/log/nginx/cgspace.cgiar.org-access-ssl.log* | awk '{print $6}' | sort | uniq
TLSv1/DES-CBC3-SHA
TLSv1/EDH-RSA-DES-CBC3-SHA

So this represents 0.02% of 1.16M connections over a one-week period
Transforming some filenames in OpenRefine so they can have a useful description for SAFBuilder:

value + "__description:" + cells["dc.type"].value

This gives you, for example: Mainstreaming gender in agricultural R&D.pdf__description:Brief

2016-09-06

Trying to import the records for CIAT from yesterday, but having filename encoding issues from their zip file
Create a zip on Mac OS X from a SAF bundle containing only one record with one PDF:
- Filename: Complementing Farmers Genetic Knowledge Farmer Breeding Workshop in Turipaná, Colombia.pdf
- Imports fine on DSpace running on Mac OS X
- Fails to import on DSpace running on Linux with error No such file or directory
Change diacritic in file name from á to a and re-create SAF bundle and zip
- Success on both Mac OS X and Linux…
Looks like on the Mac OS X file system the file names represent á as: a (U+0061) + ́ (U+0301)
See: http://www.fileformat.info/info/unicode/char/e1/index.htm
See: http://demo.icu-project.org/icu-bin/nbrowser?t=%C3%A1&s=&uv=0
If I unzip the original zip from CIAT on Windows, re-zip it with 7zip on Windows, and then unzip it on Linux directly, the file names seem to be proper UTF-8
We should definitely clean filenames so they don’t use characters that are tricky to process in CSV and shell scripts, like: ,, ', and "

value.replace("'","").replace(",","").replace('"','')

I need to write a Python script to match that for renaming files in the file system
When importing SAF bundles it seems you can specify the target collection on the command line using -c 10568/4003 or in the collections file inside each item in the bundle
Seems that the latter method causes a null pointer exception, so I will just have to use the former method
In the end I was able to import the files after unzipping them ONLY on Linux
- The CSV file was giving file names in UTF-8, and unzipping the zip on Mac OS X and transferring it was converting the file names to Unicode equivalence like I saw above
Import CIAT Gender Network records to CGSpace, first creating the SAF bundles as my user, then importing as the tomcat7 user, and deleting the bundle, for each collection’s items:

$ ./safbuilder.sh -c /home/aorth/ciat-gender-2016-09-06/66601.csv
$ JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx512m" /home/cgspace.cgiar.org/bin/dspace import -a -e aorth@mjanja.ch -c 10568/66601 -s /home/aorth/ciat-gender-2016-09-06/SimpleArchiveFormat -m 66601.map
$ rm -rf ~/ciat-gender-2016-09-06/SimpleArchiveFormat/

2016-09-07

Erase and rebuild DSpace Test based on latest Ubuntu 16.04, PostgreSQL 9.5, and Java 8 stuff
Reading about PostgreSQL maintenance and it seems manual vacuuming is only for certain workloads, such as heavy update/write loads
I suggest we disable our nightly manual vacuum task, as we’re a mostly read workload, and I’d rather stick as close to the documentation as possible since we haven’t done any testing/observation of PostgreSQL
See: https://www.postgresql.org/docs/9.3/static/routine-vacuuming.html
CGSpace went down and the error seems to be the same as always (lately):

2016-09-07 11:39:23,162 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error -
org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
...

Since CGSpace had crashed I quickly deployed the new LDAP settings before restarting Tomcat

2016-09-13

CGSpace crashed twice today, errors from catalina.out:

org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)

I enabled logging of requests to /rest again

2016-09-14

CGSpace crashed again, errors from catalina.out:

org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)

I restarted Tomcat and it was ok again
CGSpace crashed a few hours later, errors from catalina.out:

Exception in thread "http-bio-127.0.0.1-8081-exec-25" java.lang.OutOfMemoryError: Java heap space
        at java.lang.StringCoding.decode(StringCoding.java:215)

We haven’t seen that in quite a while…
Indeed, in a month of logs it only occurs 15 times:

# grep -rsI "OutOfMemoryError" /var/log/tomcat7/catalina.* | wc -l
15

I also see a bunch of errors from dspace.log:

2016-09-14 12:23:07,981 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error -
org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object

Looking at REST requests, it seems there is one IP hitting us nonstop:

# awk '{print $1}' /var/log/nginx/rest.log  | sort -n | uniq -c | sort -h | tail -n 3
    820 50.87.54.15
  12872 70.32.99.142
  25744 70.32.83.92
# awk '{print $1}' /var/log/nginx/rest.log.1  | sort -n | uniq -c | sort -h | tail -n 3
   7966 181.118.144.29
  54706 70.32.99.142
 109412 70.32.83.92

Those are the same IPs that were hitting us heavily in July, 2016 as well…
I think the stability issues are definitely from REST
Crashed AGAIN, errors from dspace.log:

2016-09-14 14:31:43,069 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error -
org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object

And more heap space errors:

# grep -rsI "OutOfMemoryError" /var/log/tomcat7/catalina.* | wc -l
19

There are no more rest requests since the last crash, so maybe there are other things causing this.
Hmm, I noticed a shitload of IPs from 180.76.0.0/16 are connecting to both CGSpace and DSpace Test (58 unique IPs concurrently!)
They seem to be coming from Baidu, and so far during today alone account for ¹⁄₆ of every connection:

# grep -c ip_addr= /home/cgspace.cgiar.org/log/dspace.log.2016-09-14
29084
# grep -c ip_addr=180.76.15 /home/cgspace.cgiar.org/log/dspace.log.2016-09-14
5192

Other recent days are the same… hmmm.
From the activity control panel I can see 58 unique IPs hitting the site concurrently, which has GOT to hurt our stability
A list of all 2000 unique IPs from CGSpace logs today:

# grep ip_addr= /home/cgspace.cgiar.org/log/dspace.log.2016-09-11 | awk -F: '{print $5}' | sort -n | uniq -c | sort -h | tail -n 100

Looking at the top 20 IPs or so, most are Yahoo, MSN, Google, Baidu, TurnitIn (iParadigm), etc… do we have any real users?
Generate a list of all author affiliations for Peter Ballantyne to go through, make corrections, and create a lookup list from:

dspacetest=# \copy (select text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=211 group by text_value order by count desc) to /tmp/affiliations.csv with csv;

Looking into the Catalina logs again around the time of the first crash, I see:

Wed Sep 14 09:47:27 UTC 2016 | Query:id: 78581 AND type:2
Wed Sep 14 09:47:28 UTC 2016 | Updating : 6/6 docs.
Commit
Commit done
dn:CN=Haman\, Magdalena  (CIAT-CCAFS),OU=Standard,OU=Users,OU=HQ,OU=CIATHUB,dc=cgiarad,dc=org
Exception in thread "http-bio-127.0.0.1-8081-exec-193" java.lang.OutOfMemoryError: Java heap space

And after that I see a bunch of “pool error Timeout waiting for idle object”
Later, near the time of the next crash I see:

dn:CN=Haman\, Magdalena  (CIAT-CCAFS),OU=Standard,OU=Users,OU=HQ,OU=CIATHUB,dc=cgiarad,dc=org
Wed Sep 14 11:29:55 UTC 2016 | Query:id: 79078 AND type:2
Wed Sep 14 11:30:20 UTC 2016 | Updating : 6/6 docs.
Commit
Commit done
Sep 14, 2016 11:32:22 AM com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator buildModelAndSchemas
SEVERE: Failed to generate the schema for the JAX-B elements
com.sun.xml.bind.v2.runtime.IllegalAnnotationsException: 2 counts of IllegalAnnotationExceptions
java.util.Map is an interface, and JAXB can't handle interfaces.
        this problem is related to the following location:
                at java.util.Map
                at public java.util.Map com.atmire.dspace.rest.common.Statlet.getRender()
                at com.atmire.dspace.rest.common.Statlet
java.util.Map does not have a no-arg default constructor.
        this problem is related to the following location:
                at java.util.Map
                at public java.util.Map com.atmire.dspace.rest.common.Statlet.getRender()
                at com.atmire.dspace.rest.common.Statlet

Then 20 minutes later another outOfMemoryError:

Exception in thread "http-bio-127.0.0.1-8081-exec-25" java.lang.OutOfMemoryError: Java heap space
        at java.lang.StringCoding.decode(StringCoding.java:215)

Perhaps these particular issues are memory issues, the munin graphs definitely show some weird purging/allocating behavior starting this week

And really, we did reduce the memory of CGSpace in late 2015, so maybe we should just increase it again, now that our usage is higher and we are having memory errors in the logs
Oh great, the configuration on the actual server is different than in configuration management!
Seems we added a bunch of settings to the /etc/default/tomcat7 in December, 2015 and never updated our ansible repository:

JAVA_OPTS="-Djava.awt.headless=true -Xms3584m -Xmx3584m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -Dfile.encoding=UTF-8 -XX:-UseGCOverheadLimit -XX:MaxGCPauseMillis=250 -XX:GCTimeRatio=9 -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled -XX:G1HeapRegionSize=8m -XX:InitiatingHeapOccupancyPercent=75 -XX:+UseLargePages -XX:+AggressiveOpts"

So I’m going to bump the heap +512m and remove all the other experimental shit (and update ansible!)
Increased JVM heap to 4096m on CGSpace (linode01)

2016-09-15

Looking at Google Webmaster Tools again, it seems the work I did on URL query parameters and blocking via the X-Robots-Tag HTTP header in March, 2016 seem to have had a positive effect on Google’s index for CGSpace

2016-09-16

CGSpace crashed again, and there are TONS of heap space errors but the datestamps aren’t on those lines so I’m not sure if they were yesterday:

dn:CN=Orentlicher\, Natalie (CIAT),OU=Standard,OU=Users,OU=HQ,OU=CIATHUB,dc=cgiarad,dc=org
Thu Sep 15 18:45:25 UTC 2016 | Query:id: 55785 AND type:2
Thu Sep 15 18:45:26 UTC 2016 | Updating : 100/218 docs.
Thu Sep 15 18:45:26 UTC 2016 | Updating : 200/218 docs.
Thu Sep 15 18:45:27 UTC 2016 | Updating : 218/218 docs.
Commit
Commit done
Exception in thread "http-bio-127.0.0.1-8081-exec-247" java.lang.OutOfMemoryError: Java heap space
Exception in thread "http-bio-127.0.0.1-8081-exec-241" java.lang.OutOfMemoryError: Java heap space
Exception in thread "http-bio-127.0.0.1-8081-exec-243" java.lang.OutOfMemoryError: Java heap space
Exception in thread "http-bio-127.0.0.1-8081-exec-258" java.lang.OutOfMemoryError: Java heap space
Exception in thread "http-bio-127.0.0.1-8081-exec-268" java.lang.OutOfMemoryError: Java heap space
Exception in thread "http-bio-127.0.0.1-8081-exec-263" java.lang.OutOfMemoryError: Java heap space
Exception in thread "http-bio-127.0.0.1-8081-exec-280" java.lang.OutOfMemoryError: Java heap space
Exception in thread "Thread-54216" org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Exception writing document id 7feaa95d-8e1f-4f45-80bb
-e14ef82ee224 to the index; possible analysis error.
        at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
        at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
        at com.atmire.statistics.SolrLogThread.run(SourceFile:25)

I bumped the heap space from 4096m to 5120m to see if this is really about heap speace or not.
Looking into some of these errors that I’ve seen this week but haven’t noticed before:

# zcat -f -- /var/log/tomcat7/catalina.* | grep -c 'Failed to generate the schema for the JAX-B elements'
113

I’ve sent a message to Atmire about the Solr error to see if it’s related to their batch update module

2016-09-19

Work on cleanups for author affiliations after Peter sent me his list of corrections/deletions:

$ ./fix-metadata-values.py -i affiliations_pb-322-corrections.csv -f cg.contributor.affiliation -t correct -m 211 -d dspace -u dspace -p fuuu
$ ./delete-metadata-values.py -f cg.contributor.affiliation -i affiliations_pb-2-deletions.csv -m 211 -u dspace -d dspace -p fuuu

After that we need to take the top ~300 and make a controlled vocabulary for it
I dumped a list of the top 300 affiliations from the database, sorted it alphabetically in OpenRefine, and created a controlled vocabulary for it (#267)

2016-09-20

Run all system updates on DSpace Test and reboot the server
Merge changes for sponsorship and affiliation controlled vocabularies (#267, #268)
Merge minor changes to messages.xml to reconcile it with the stock DSpace 5.1 one (#269)
Peter asked about adding title search to Discovery
The index was already defined, so I just added it to the search filters
It works but CGSpace apparently uses OR for search terms, which makes the search results basically useless
I need to read the docs and ask on the mailing list to see if we can tweak that
Generate a new list of sponsors from the database for Peter Ballantyne so we can clean them up and update the controlled vocabulary

2016-09-21

Turns out the Solr search logic switched from OR to AND in DSpace 6.0 and the change is easy to backport: https://jira.duraspace.org/browse/DS-2809
We just need to set this in dspace/solr/search/conf/schema.xml:

<solrQueryParser defaultOperator="AND"/>

It actually works really well, and search results return much less hits now (before, after):

Found a way to improve the configuration of Atmire’s Content and Usage Analysis (CUA) module for date fields

-content.analysis.dataset.option.8=metadata:dateAccessioned:discovery
+content.analysis.dataset.option.8=metadata:dc.date.accessioned:date(month)

This allows the module to treat the field as a date rather than a text string, so we can interrogate it more intelligently
Add dc.date.accessioned to XMLUI Discovery search filters
Major CGSpace crash because ILRI forgot to pay the Linode bill
45 minutes of downtime!
Start processing the fixes to dc.description.sponsorship from Peter Ballantyne:

$ ./fix-metadata-values.py -i sponsors-fix-23.csv -f dc.description.sponsorship -t correct -m 29 -d dspace -u dspace -p fuuu
$ ./delete-metadata-values.py -i sponsors-delete-8.csv -f dc.description.sponsorship -m 29 -d dspace -u dspace -p fuuu

I need to run these and the others from a few days ago on CGSpace the next time we run updates
Also, I need to update the controlled vocab for sponsors based on these

2016-09-22

Update controlled vocabulary for sponsorship based on the latest corrected values from the database

2016-09-25

Merge accession date improvements for CUA module (#275)
Merge addition of accession date to Discovery search filters (#276)
Merge updates to sponsorship controlled vocabulary (#277)
I’ve been trying to add a search filter for dc.description so the IITA people can search for some tags they use there, but for some reason the filter never shows up in Atmire’s CUA
Not sure if it’s something like we already have too many filters there (30), or the filter name is reserved, etc…
Generate a list of ILRI subjects for Peter and Abenet to look through/fix:

dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where resource_type_id=2 and metadata_field_id=203 group by text_value order by count desc) to /tmp/ilrisubjects.csv with csv;

Regenerate Discovery indexes a few times after playing with discovery.xml index definitions (syntax, parameters, etc).
Merge changes to boolean logic in Solr search (#274)
Run all sponsorship and affiliation fixes on CGSpace, deploy latest 5_x-prod branch, and re-index Discovery on CGSpace
Tested OCSP stapling on DSpace Test’s nginx and it works:

$ openssl s_client -connect dspacetest.cgiar.org:443 -servername dspacetest.cgiar.org -tls1_2 -tlsextdebug -status
...
OCSP response:
======================================
OCSP Response Data:
...
    Cert Status: good

I’ve been monitoring this for almost two years in this GitHub issue: https://github.com/ilri/DSpace/issues/38

2016-09-27

Discuss fixing some ORCIDs for CCAFS author Sonja Vermeulen with Magdalena Haman
This author has a few variations:

dspacetest=# select distinct text_value, authority, confidence from metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value like 'Vermeu
len, S%';

And it looks like fe4b719f-6cc4-4d65-8504-7a83130b9f83 is the authority with the correct ORCID linked

dspacetest=# update metadatavalue set authority='fe4b719f-6cc4-4d65-8504-7a83130b9f83w', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Vermeulen, S%';
UPDATE 101

Hmm, now her name is missing from the authors facet and only shows the authority ID
On the production server there is an item with her ORCID but it is using a different authority: f01f7b7b-be3f-4df7-a61d-b73c067de88d
Maybe I used the wrong one… I need to look again at the production database
On a clean snapshot of the database I see the correct authority should be f01f7b7b-be3f-4df7-a61d-b73c067de88d, not fe4b719f-6cc4-4d65-8504-7a83130b9f83
Updating her authorities again and reindexing:

dspacetest=# update metadatavalue set authority='f01f7b7b-be3f-4df7-a61d-b73c067de88d', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Vermeulen, S%';
UPDATE 101

Use GitHub icon from Font Awesome instead of a PNG to save one extra network request
We can also replace the RSS and mail icons in community text!
Fix reference to dc.type.* in Atmire CUA module, as we now only index dc.type for “Output type”

2016-09-28

Make a placeholder pull request for discovery.xml changes (#278), as I still need to test their effect on Atmire content analysis module
Make a placeholder pull request for Font Awesome changes (#279), which replaces the GitHub image in the footer with an icon, and add style for RSS and @ icons that I will start replacing in community/collection HTML intros
Had some issues with local test server after messing with Solr too much, had to blow everything away and re-install from CGSpace
Going to try to update Sonja Vermeulen’s authority to 2b4166b7-6e4d-4f66-9d8b-ddfbec9a6ae0, as that seems to be one of her authorities that has an ORCID
Merge Font Awesome changes (#279)
Minor fix to a string in Atmire’s CUA module (#280)
This seems to be what I’ll need to do for Sonja Vermeulen (but with 2b4166b7-6e4d-4f66-9d8b-ddfbec9a6ae0 instead on the live site):

dspacetest=# update metadatavalue set authority='09e4da69-33a3-45ca-b110-7d3f82d2d6d2', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Vermeulen, S%';
dspacetest=# update metadatavalue set authority='09e4da69-33a3-45ca-b110-7d3f82d2d6d2', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Vermeulen SJ%';

And then update Discovery and Authority indexes
Minor fix for “Subject” string in Discovery search and Atmire modules (#281)
Start testing batch fixes for ILRI subject from Peter:

$ ./fix-metadata-values.py -i ilrisubjects-fix-32.csv -f cg.subject.ilri -t correct -m 203 -d dspace -u dspace -p fuuuu
$ ./delete-metadata-values.py -i ilrisubjects-delete-13.csv -f cg.subject.ilri -m 203 -d dspace -u dspace -p fuuu

2016-09-29

Add cg.identifier.ciatproject to metadata registry in preparation for CIAT project tag
Merge changes for CIAT project tag (#282)
DSpace Test (linode02) became unresponsive for some reason, I had to hard reboot it from the Linode console
People on DSpace mailing list gave me a query to get authors from certain collections:

dspacetest=# select distinct text_value from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 AND resource_id IN (select item_id from collection2item where collection_id IN (select resource_id from handle where handle in ('10568/5472', '10568/5473')));

2016-09-30

Deny access to REST API’s find-by-metadata-field endpoint to protect against an upstream security issue (DS-3250)
There is a patch but it is only for 5.5 and doesn’t apply cleanly to 5.1

August, 2016

Mon, 01 Aug 2016 15:53:00 +0300

2016-08-01

Add updated distribution license from Sisay (#259)
Play with upgrading Mirage 2 dependencies in bower.json because most are several versions of out date
Bootstrap is at 3.3.0 but upstream is at 3.3.7, and upgrading to anything beyond 3.3.1 breaks glyphicons and probably more
bower stuff is a dead end, waste of time, too many issues
Anything after Bootstrap 3.3.1 makes glyphicons disappear (HTTP 404 trying to access from incorrect path of fonts)
Start working on DSpace 5.1 → 5.5 port:

$ git checkout -b 55new 5_x-prod
$ git reset --hard ilri/5_x-prod
$ git rebase -i dspace-5.5

Lots of conflicts that don’t make sense (ie, shouldn’t conflict!)
This file in particular conflicts almost 10 times: dspace/modules/xmlui-mirage2/src/main/webapp/themes/CGIAR/styles/_style.scss
Checking out a clean branch at 5.5 and cherry-picking our commits works where that file would normally have a conflict
Seems to be related to merge commits
git rebase --preserve-merges doesn’t seem to help
Eventually I just turned on git rerere and solved the conflicts and completed the 403 commit rebase
The 5.5 code now builds but doesn’t run (white page in Tomcat)

2016-08-02

Ask Atmire for help with DSpace 5.5 issue
Vanilla DSpace 5.5 deploys and runs fine
Playing with DSpace in Ubuntu 16.04 and Tomcat 7
Everything is still fucked up, even vanilla DSpace 5.5

2016-08-04

Ask on DSpace mailing list about duplicate authors, Discovery and author text values
Atmire responded with some new DSpace 5.5 ready versions to try for their modules

2016-08-05

Fix item display incorrectly displaying Species when Breeds were present (#260)
Experiment with fixing more authors, like Delia Grace:

dspacetest=# update metadatavalue set authority='0b4fcbc1-d930-4319-9b4d-ea1553cca70b', confidence=600 where metadata_field_id=3 and text_value='Grace, D.';

2016-08-06

Finally figured out how to remove “View/Open” and “Bitstreams” from the item view

2016-08-07

Start working on Ubuntu 16.04 Ansible playbook for Tomcat 8, PostgreSQL 9.5, Oracle 8, etc

2016-08-08

Still troubleshooting Atmire modules on DSpace 5.5
Vanilla DSpace 5.5 works on Tomcat 7…
Ooh, and vanilla DSpace 5.5 works on Tomcat 8 with Java 8!
Some notes about setting up Tomcat 8, since it’s new on this machine…
Install latest Oracle Java 8 JDK
Create setenv.sh in Tomcat 8 libexec/bin directory:

CATALINA_OPTS="-Djava.awt.headless=true -Xms3072m -Xmx3072m -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -Dfile.encoding=UTF-8"
CATALINA_OPTS="$CATALINA_OPTS -Djava.library.path=/opt/brew/Cellar/tomcat-native/1.2.8/lib"

JRE_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_102.jdk/Contents/Home

Edit Tomcat 8 server.xml to add regular HTTP listener for solr
Symlink webapps:

$ rm -rf /opt/brew/Cellar/tomcat/8.5.4/libexec/webapps/ROOT
$ ln -sv ~/dspace/webapps/xmlui /opt/brew/Cellar/tomcat/8.5.4/libexec/webapps/ROOT
$ ln -sv ~/dspace/webapps/oai /opt/brew/Cellar/tomcat/8.5.4/libexec/webapps/oai
$ ln -sv ~/dspace/webapps/jspui /opt/brew/Cellar/tomcat/8.5.4/libexec/webapps/jspui
$ ln -sv ~/dspace/webapps/rest /opt/brew/Cellar/tomcat/8.5.4/libexec/webapps/rest
$ ln -sv ~/dspace/webapps/solr /opt/brew/Cellar/tomcat/8.5.4/libexec/webapps/solr

2016-08-09

More tests of Atmire’s 5.5 modules on a clean, working instance of 5_x-prod
Still fails, though perhaps differently than before (Flyway): https://gist.github.com/alanorth/5d49c45a16efd7c6bc1e6642e66118b2
More work on Tomcat 8 and Java 8 stuff for Ansible playbooks

2016-08-10

Turns out DSpace 5.x isn’t ready for Tomcat 8: https://jira.duraspace.org/browse/DS-3092
So we’ll need to use Tomcat 7 + Java 8 on Ubuntu 16.04
More work on the Ansible stuff for this, allowing Tomcat 7 to use Java 8
Merge pull request for fixing the type Discovery index to use dc.type (#262)
Merge pull request for removing “Bitstream” text from item display, as it confuses users and isn’t necessary (#263)

2016-08-11

Finally got DSpace (5.5) running on Ubuntu 16.04, Tomcat 7, Java 8, PostgreSQL 9.5 via the updated Ansible stuff

2016-08-14

Update Mirage 2 build notes for Ubuntu 16.04: https://gist.github.com/alanorth/2cf9c15834dc68a514262fcb04004cb0

2016-08-15

Notes on NodeJS + nginx + systemd: https://gist.github.com/alanorth/51acd476891c67dfe27725848cf5ace1

2016-08-16

Troubleshoot Paramiko connection issues with Ansible on ILRI servers: #37
Turns out we need to add some MACs to our sshd_config: hmac-sha2-512,hmac-sha2-256
Update DSpace Test’s Java to version 8 to start testing this configuration (seeing as Solr recommends it)

2016-08-17

More work on Let’s Encrypt stuff for Ansible roles
Yesterday Atmire responded about DSpace 5.5 issues and asked me to try the dspace database repair command to fix Flyway issues
The dspace database command doesn’t even run: https://gist.github.com/alanorth/c43c8d89e8df346d32c0ee938be90cd5
Oops, it looks like the missing classes causing dspace database to fail were coming from the old ~/dspace/config/spring folder
After removing the spring folder and running ant install again, dspace database works
I see there are missing and pending Flyway migrations, but running dspace database repair and dspace database migrate does nothing: https://gist.github.com/alanorth/41ed5abf2ff32d8ac9eedd1c3d015d70

2016-08-18

Fix “CONGO,DR” country name in input-forms.xml (#264)
Also need to fix existing records using the incorrect form in the database:

dspace=# update metadatavalue set text_value='CONGO, DR' where resource_type_id=2 and metadata_field_id=228 and text_value='CONGO,DR';

I asked a question on the DSpace mailing list about updating “preferred” forms of author names from ORCID

2016-08-21

A few days ago someone on the DSpace mailing list suggested I try dspace dsrun org.dspace.authority.UpdateAuthorities to update preferred author names from ORCID
If you set auto-update-items=true in dspace/config/modules/solrauthority.cfg it is supposed to update records it finds automatically
I updated my name format on ORCID and I’ve been running that script a few times per day since then but nothing has changed
Still troubleshooting Atmire modules on DSpace 5.5
I sent them some new verbose logs: https://gist.github.com/alanorth/700748995649688148ceba89d760253e

2016-08-22

Database migrations are fine on DSpace 5.1:

$ ~/dspace/bin/dspace database info

Database URL: jdbc:postgresql://localhost:5432/dspacetest
Database Schema: public
Database Software: PostgreSQL version 9.3.14
Database Driver: PostgreSQL Native Driver version PostgreSQL 9.1 JDBC4 (build 901)

+----------------+----------------------------+---------------------+---------+
| Version        | Description                | Installed on        | State   |
+----------------+----------------------------+---------------------+---------+
| 1.1            | Initial DSpace 1.1 databas |                     | PreInit |
| 1.2            | Upgrade to DSpace 1.2 sche |                     | PreInit |
| 1.3            | Upgrade to DSpace 1.3 sche |                     | PreInit |
| 1.3.9          | Drop constraint for DSpace |                     | PreInit |
| 1.4            | Upgrade to DSpace 1.4 sche |                     | PreInit |
| 1.5            | Upgrade to DSpace 1.5 sche |                     | PreInit |
| 1.5.9          | Drop constraint for DSpace |                     | PreInit |
| 1.6            | Upgrade to DSpace 1.6 sche |                     | PreInit |
| 1.7            | Upgrade to DSpace 1.7 sche |                     | PreInit |
| 1.8            | Upgrade to DSpace 1.8 sche |                     | PreInit |
| 3.0            | Upgrade to DSpace 3.x sche |                     | PreInit |
| 4.0            | Initializing from DSpace 4 | 2015-11-20 12:42:52 | Success |
| 5.0.2014.08.08 | DS-1945 Helpdesk Request a | 2015-11-20 12:42:53 | Success |
| 5.0.2014.09.25 | DS 1582 Metadata For All O | 2015-11-20 12:42:55 | Success |
| 5.0.2014.09.26 | DS-1582 Metadata For All O | 2015-11-20 12:42:55 | Success |
| 5.0.2015.01.27 | MigrateAtmireExtraMetadata | 2015-11-20 12:43:29 | Success |
| 5.1.2015.12.03 | Atmire CUA 4 migration     | 2016-03-21 17:10:41 | Success |
| 5.1.2015.12.03 | Atmire MQM migration       | 2016-03-21 17:10:42 | Success |
+----------------+----------------------------+---------------------+---------+

So I’m not sure why they have problems when we move to DSpace 5.5 (even the 5.1 migrations themselves show as “Missing”)

2016-08-23

Help Paola from CCAFS with her thumbnails again
Talk to Atmire about the DSpace 5.5 issue, and it seems to be caused by a bug in FlywayDB
They said I should delete the Atmire migrations

dspacetest=# delete from schema_version where description =  'Atmire CUA 4 migration' and version='5.1.2015.12.03.2';
dspacetest=# delete from schema_version where description =  'Atmire MQM migration' and version='5.1.2015.12.03.3';

After that DSpace starts up by XMLUI now has unrelated issues that I need to solve!

org.apache.avalon.framework.configuration.ConfigurationException: Type 'ThemeResourceReader' does not exist for 'map:read' at jndi:/localhost/themes/0_CGIAR/sitemap.xmap:136:77
context:/jndi:/localhost/themes/0_CGIAR/sitemap.xmap - 136:77

Looks like we’re missing some stuff in the XMLUI module’s sitemap.xmap, as well as in each of our XMLUI themes
Diff them with these to get the ThemeResourceReader changes:
- dspace-xmlui/src/main/webapp/sitemap.xmap
- dspace-xmlui-mirage2/src/main/webapp/sitemap.xmap
Then we had some NullPointerException from the SolrLogger class, which is apparently part of Atmire’s CUA module
I tried with a small version bump to CUA but it didn’t work (version 5.5-4.1.1-0)
Also, I started looking into huge pages to prepare for PostgreSQL 9.5, but it seems Linode’s kernels don’t enable them

2016-08-24

Clean up and import 48 CCAFS records into DSpace Test
SQL to get all journal titles from dc.source (55), since it’s apparently used for internal DSpace filename shit, but we moved all our journal titles there a few months ago:

dspacetest=# select distinct text_value from metadatavalue where metadata_field_id=55 and text_value !~ '.*(\.pdf|\.png|\.PDF|\.Pdf|\.JPEG|\.jpg|\.JPG|\.jpeg|\.xls|\.rtf|\.docx?|\.potx|\.dotx|\.eqa|\.tiff|\.mp4|\.mp3|\.gif|\.zip|\.txt|\.pptx|\.indd|\.PNG|\.bmp|\.exe|org\.dspace\.app\.mediafilter).*';

2016-08-25

Atmire suggested adding a missing bean to dspace/config/spring/api/atmire-cua.xml but it doesn’t help:

...
Error creating bean with name 'MetadataStorageInfoService'
...

Atmire sent an updated version of dspace/config/spring/api/atmire-cua.xml and now XMLUI starts but gives a null pointer exception:

Java stacktrace: java.lang.NullPointerException
    at org.dspace.app.xmlui.aspect.statistics.Navigation.addOptions(Navigation.java:129)
    at org.dspace.app.xmlui.wing.AbstractWingTransformer.startElement(AbstractWingTransformer.java:228)
    at sun.reflect.GeneratedMethodAccessor126.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
    at com.sun.proxy.$Proxy103.startElement(Unknown Source)
    at org.apache.cocoon.environment.internal.EnvironmentChanger.startElement(EnvironmentStack.java:140)
    at org.apache.cocoon.environment.internal.EnvironmentChanger.startElement(EnvironmentStack.java:140)
    at org.apache.cocoon.xml.AbstractXMLPipe.startElement(AbstractXMLPipe.java:94)
...

Import the 47 CCAFS records to CGSpace, creating the SimpleArchiveFormat bundles and importing like:

$ ./safbuilder.sh -c /tmp/Thumbnails\ to\ Upload\ to\ CGSpace/3546.csv
$ JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx512m" /home/cgspace.cgiar.org/bin/dspace import -a -e aorth@mjanja.ch -c 10568/3546 -s /tmp/Thumbnails\ to\ Upload\ to\ CGSpace/SimpleArchiveFormat -m 3546.map

Finally got DSpace 5.5 working with the Atmire modules after a few rounds of back and forth with Atmire devs

2016-08-26

CGSpace had issues tonight, not entirely crashing, but becoming unresponsive
The dspace log had this:

2016-08-26 20:48:05,040 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error -                                                               org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object

Related to /rest no doubt

2016-08-27

Run corrections for Delia Grace and CONGO, DR, and deploy August changes to CGSpace
Run all system updates and reboot the server

July, 2016

Fri, 01 Jul 2016 10:53:00 +0300

2016-07-01

Add dc.description.sponsorship to Discovery sidebar facets and make investors clickable in item view (#232)
I think this query should find and replace all authors that have “,” at the end of their names:

dspacetest=# update metadatavalue set text_value = regexp_replace(text_value, '(^.+?),$', '\1') where metadata_field_id=3 and resource_type_id=2 and text_value ~ '^.+?,$';
UPDATE 95
dspacetest=# select text_value from  metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value ~ '^.+?,$';
 text_value
------------
(0 rows)

In this case the select query was showing 95 results before the update

2016-07-02

Comment on DSpace Jira ticket about author lookup search text (DS-2329)

2016-07-04

Seems the database’s author authority values mean nothing without the authority Solr core from the host where they were created!

2016-07-05

Amend backup-solr.sh script so it backs up the entire Solr folder
We really only need statistics and authority but meh
Fix metadata for species on DSpace Test:

$ ./fix-metadata-values.py -i /tmp/Species-Peter-Fix.csv -f dc.Species -t CORRECT -m 94 -d dspacetest -u dspacetest -p 'fuuu'

Will run later on CGSpace
A user is still having problems with Sherpa/Romeo causing crashes during the submission process when the journal is “ungraded”
I tested the patch for DS-2740 that I had found last month and it seems to work
I will merge it to 5_x-prod

2016-07-06

Delete 23 blank metadata values from CGSpace:

cgspace=# delete from metadatavalue where resource_type_id=2 and text_value='';
DELETE 23

Complete phase three of metadata migration, for the following fields:
- dc.title.jtitle → dc.source
- dc.crsubject.crpsubject → cg.contributor.crp
- dc.contributor.affiliation → cg.contributor.affiliation
- dc.Species → cg.species
- dc.srplace.subregion → cg.coverage.subregion
- dc.contributor.corporate → dc.contributor.author
- dc.identifier.url → cg.identifier.url
- dc.identifier.doi → cg.identifier.doi
- dc.identifier.googleurl → cg.identifier.googleurl
- dc.identifier.dataurl → cg.identifier.dataurl
Also, run fixes and deletes for species and author affiliations (over 1000 corrections!)

$ ./fix-metadata-values.py -i Species-Peter-Fix.csv -f dc.Species -t CORRECT -m 212 -d dspace -u dspace -p 'fuuu'
$ ./fix-metadata-values.py -i Affiliations-Fix-1045-Peter-Abenet.csv -f dc.contributor.affiliation -t Correct -m 211 -d dspace -u dspace -p 'fuuu'
$ ./delete-metadata-values.py -f dc.contributor.affiliation -i Affiliations-Delete-Peter-Abenet.csv -m 211 -u dspace -d dspace -p 'fuuu'

I then ran all server updates and rebooted the server

2016-07-11

Doing some author cleanups from Peter and Abenet:

$ ./fix-metadata-values.py -i /tmp/Authors-Fix-205-UTF8.csv -f dc.contributor.author -t correct -m 3 -d dspacetest -u dspacetest -p fuuu
$ ./delete-metadata-values.py -f dc.contributor.author -i /tmp/Authors-Delete-UTF8.csv -m 3 -u dspacetest -d dspacetest -p fuuu

2016-07-13

Run the author cleanups on CGSpace and start a full Discovery re-index

2016-07-14

Test LDAP settings for new root LDAP
Seems to work when binding as a top-level user

2016-07-18

Adjust identifiers in XMLUI item display to be more prominent
Add species and breed to the XMLUI item display
CGSpace crashed late at night and the DSpace logs were showing:

2016-07-18 20:26:30,941 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error - 
org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
...

I suspect it’s someone hitting REST too much:

# awk '{print $1}' /var/log/nginx/rest.log  | sort -n | uniq -c | sort -h | tail -n 3
    710 66.249.78.38
   1781 181.118.144.29
  24904 70.32.99.142

I just blocked access to /rest for that last IP for now:

     # log rest requests
     location /rest {
         access_log /var/log/nginx/rest.log;
         proxy_pass http://127.0.0.1:8443;
         deny 70.32.99.142;
     }

2016-07-21

Mitigate the HTTPoxy vulnerability for Tomcat etc in nginx: https://github.com/ilri/rmg-ansible-public/pull/38
Unblock 70.32.99.142 from /rest as it has been blocked for a few days

2016-07-22

Help Paola from CCAFS with thumbnails for batch uploads
She has been struggling to get the dimensions right, and manually enlarging smaller thumbnails, renaming PNGs to JPG, etc
Altmetric reports having an issue with some of our authors being doubled…
This is related to authority and confidence!
We might need to use index.authority.ignore-prefered=true to tell the Discovery index to prefer the variation that exists in the metadatavalue rather than what it finds in the authority cache.
Trying these on DSpace Test after a discussion by Daniel Scharon on the dspace-tech mailing list:

index.authority.ignore-prefered.dc.contributor.author=true
index.authority.ignore-variants.dc.contributor.author=false

After reindexing I don’t see any change in Discovery’s display of authors, and still have entries like:

Grace, D. (464)
Grace, D. (62)

I asked for clarification of the following options on the DSpace mailing list:

index.authority.ignore
index.authority.ignore-prefered
index.authority.ignore-variants

In the mean time, I will try these on DSpace Test (plus a reindex):

index.authority.ignore=true
index.authority.ignore-prefered=true
index.authority.ignore-variants=true

Enabled usage of X-Forwarded-For in DSpace admin control panel (#255
It was misconfigured and disabled, but already working for some reason sigh
… no luck. Trying with just:

index.authority.ignore=true

After re-indexing and clearing the XMLUI cache nothing has changed

2016-07-25

Trying a few more settings (plus reindex) for Discovery on DSpace Test:

index.authority.ignore-prefered.dc.contributor.author=true
index.authority.ignore-variants=true

Run all OS updates and reboot DSpace Test server
No changes to Discovery after reindexing… hmm.
Integrate and massively clean up About page (#256)

The DSpace source code mentions the configuration key discovery.index.authority.ignore-prefered.* (with prefix of discovery, despite the docs saying otherwise), so I’m trying the following on DSpace Test:

discovery.index.authority.ignore-prefered.dc.contributor.author=true
discovery.index.authority.ignore-variants=true

Still no change!
Deploy species, breed, and identifier changes to CGSpace, as well as About page
Run Linode RAM upgrade (8→12GB)
Re-sync DSpace Test with CGSpace
I noticed that our backup scripts don’t send Solr cores to S3 so I amended the script

2016-07-31

Work on removing Dryland Systems and Humidtropics subjects from Discovery sidebar and Browse by
Also change “Subjects” to “AGROVOC keywords” in Discovery sidebar/search and Browse by (#257)

June, 2016

Wed, 01 Jun 2016 10:53:00 +0300

2016-06-01

Experimenting with IFPRI OAI (we want to harvest their publications)
After reading the ContentDM documentation I found IFPRI’s OAI endpoint: http://ebrary.ifpri.org/oai/oai.php
After reading the OAI documentation and testing with an OAI validator I found out how to get their publications
This is their publications set: http://ebrary.ifpri.org/oai/oai.php?verb=ListRecords&from=2016-01-01&set=p15738coll2&metadataPrefix=oai_dc
You can see the others by using the OAI ListSets verb: http://ebrary.ifpri.org/oai/oai.php?verb=ListSets
Working on second phase of metadata migration, looks like this will work for moving CPWF-specific data in dc.identifier.fund to cg.identifier.cpwfproject and then the rest to dc.description.sponsorship

dspacetest=# update metadatavalue set metadata_field_id=130 where metadata_field_id=75 and (text_value like 'PN%' or text_value like 'PHASE%' or text_value = 'CBA' or text_value = 'IA');
UPDATE 497
dspacetest=# update metadatavalue set metadata_field_id=29 where metadata_field_id=75;
UPDATE 14

Fix a few minor miscellaneous issues in dspace.cfg (#227)

2016-06-02

Testing the configuration and theme changes for the upcoming metadata migration and I found some issues with cg.coverage.admin-unit
Seems that the Browse configuration in dspace.cfg can’t handle the ‘-’ in the field name:

webui.browse.index.12 = subregion:metadata:cg.coverage.admin-unit:text

But actually, I think since DSpace 4 or 5 (we are 5.1) the Browse indexes come from Discovery (defined in discovery.xml) so this is really just a parsing error
I’ve sent a message to the DSpace mailing list to ask about the Browse index definition
A user was having problems with submission and from the stacktrace it looks like a Sherpa/Romeo issue
I found a thread on the mailing list talking about it and there is bug report and a patch: https://jira.duraspace.org/browse/DS-2740
The patch applies successfully on DSpace 5.1 so I will try it later

2016-06-03

Investigating the CCAFS authority issue, I exported the metadata for the Videos collection
The top two authors are:

CGIAR Research Program on Climate Change, Agriculture and Food Security::acd00765-02f1-4b5b-92fa-bfa3877229ce::500
CGIAR Research Program on Climate Change, Agriculture and Food Security::acd00765-02f1-4b5b-92fa-bfa3877229ce::600

So the only difference is the “confidence”
Ok, well THAT is interesting:

dspacetest=# select text_value, authority, confidence from metadatavalue where metadata_field_id=3 and text_value like '%Orth, %';
 text_value |              authority               | confidence
------------+--------------------------------------+------------
 Orth, A.   | ab606e3a-2b04-4c7d-9423-14beccf54257 |         -1
 Orth, A.   | ab606e3a-2b04-4c7d-9423-14beccf54257 |         -1
 Orth, A.   | ab606e3a-2b04-4c7d-9423-14beccf54257 |         -1
 Orth, Alan |                                      |         -1
 Orth, Alan |                                      |         -1
 Orth, Alan |                                      |         -1
 Orth, Alan |                                      |         -1
 Orth, A.   | 05c2c622-d252-4efb-b9ed-95a07d3adf11 |         -1
 Orth, A.   | 05c2c622-d252-4efb-b9ed-95a07d3adf11 |         -1
 Orth, A.   | ab606e3a-2b04-4c7d-9423-14beccf54257 |         -1
 Orth, A.   | ab606e3a-2b04-4c7d-9423-14beccf54257 |         -1
 Orth, Alan | ad281dbf-ef81-4007-96c3-a7f5d2eaa6d9 |        600
 Orth, Alan | ad281dbf-ef81-4007-96c3-a7f5d2eaa6d9 |        600
(13 rows)

And now an actually relevent example:

dspacetest=# select count(*) from metadatavalue where metadata_field_id=3 and text_value like 'CGIAR Research Program on Climate Change, Agriculture and Food Security' and confidence = 500;
 count
-------
   707
(1 row)

dspacetest=# select count(*) from metadatavalue where metadata_field_id=3 and text_value like 'CGIAR Research Program on Climate Change, Agriculture and Food Security' and confidence != 500;
 count
-------
   253
(1 row)

Trying something experimental:

dspacetest=# update metadatavalue set confidence=500 where metadata_field_id=3 and text_value like 'CGIAR Research Program on Climate Change, Agriculture and Food Security';
UPDATE 960

And then re-indexing authority and Discovery…?
After Discovery reindex the CCAFS authors are all together in the Authors sidebar facet
The docs for the ORCiD and Authority stuff for DSpace 5 mention changing the browse indexes to use the Authority as well:

webui.browse.index.2 = author:metadataAuthority:dc.contributor.author:authority

That would only be for the “Browse by” function… so we’ll have to see what effect that has later

2016-06-04

Re-sync DSpace Test with CGSpace and perform test of metadata migration again
Run phase two of metadata migrations on CGSpace (see the migration notes)
Run all system updates and reboot CGSpace server

2016-06-07

Figured out how to export a list of the unique values from a metadata field ordered by count:

dspacetest=# \copy (select text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=29 group by text_value order by count desc) to /tmp/sponsorship.csv with csv;

Identified the next round of fields to migrate:
- dc.title.jtitle → dc.source
- dc.crsubject.crpsubject → cg.contributor.crp
- dc.contributor.affiliation → cg.contributor.affiliation
- dc.Species → cg.species
- dc.contributor.corporate → dc.contributor
- dc.identifier.url → cg.identifier.url
- dc.identifier.doi → cg.identifier.doi
- dc.identifier.googleurl → cg.identifier.googleurl
- dc.identifier.dataurl → cg.identifier.dataurl
Discuss pulling data from IFPRI’s ContentDM with Ryan Miller
Looks like OAI is kinda obtuse for this, and if we use ContentDM’s API we’ll be able to access their internal field names (rather than trying to figure out how they stuffed them into various, repeated Dublin Core fields)

2016-06-08

Discuss controlled vocabularies for ~28 fields
Looks like this is all we need: https://wiki.duraspace.org/display/DSDOC5x/Submission+User+Interface#SubmissionUserInterface-ConfiguringControlledVocabularies
I wrote an XPath expression to extract the ILRI subjects from input-forms.xml (uses xmlstartlet):

$ xml sel -t -m '//value-pairs[@value-pairs-name="ilrisubject"]/pair/displayed-value/text()' -c '.' -n dspace/config/input-forms.xml

Write to Atmire about the use of atmire.orcid.id to see if we can change it
Seems to be a virtual field that is queried from the authority cache… hmm
In other news, I found out that the About page that we haven’t been using lives in dspace/config/about.xml, so now we can update the text
File bug about closed="true" attribute of controlled vocabularies not working: https://jira.duraspace.org/browse/DS-3238

2016-06-09

Atmire explained that the atmire.orcid.id field doesn’t exist in the schema, as it actually comes from the authority cache during XMLUI run time
This means we don’t see it when harvesting via OAI or REST, for example
They opened a feature ticket on the DSpace tracker to ask for support of this: https://jira.duraspace.org/browse/DS-3239

2016-06-10

Investigating authority confidences
It looks like the values are documented in Choices.java
Experiment with setting all 960 CCAFS author values to be 500:

dspacetest=# SELECT authority, confidence FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=3 AND text_value = 'CGIAR Research Program on Climate Change, Agriculture and Food Security';

dspacetest=# UPDATE metadatavalue set confidence = 500 where resource_type_id=2 AND metadata_field_id=3 AND text_value = 'CGIAR Research Program on Climate Change, Agriculture and Food Security';
UPDATE 960

After the database edit, I did a full Discovery re-index
And now there are exactly 960 items in the authors facet for ‘CGIAR Research Program on Climate Change, Agriculture and Food Security’
Now I ran the same on CGSpace
Merge controlled vocabulary functionality for animal breeds to 5_x-prod (#236)
Write python script to update metadata values in batch via PostgreSQL: fix-metadata-values.py
We need to use this to correct some pretty ugly values in fields like dc.description.sponsorship
Merge item display tweaks from earlier this week (#231)
Merge controlled vocabulary functionality for subregions (#238)

2016-06-11

Merge controlled vocabulary for sponsorship field (#239)
Fix character encoding issues for animal breed lookup that I merged yesterday

2016-06-17

Linode has free RAM upgrades for their 13th birthday so I migrated DSpace Test (4→8GB of RAM)

2016-06-18

Clean up titles and hints in input-forms.xml to use title/sentence case and a few more consistency things (#241)
The final list of fields to migrate in the third phase of metadata migrations is:
- dc.title.jtitle → dc.source
- dc.crsubject.crpsubject → cg.contributor.crp
- dc.contributor.affiliation → cg.contributor.affiliation
- dc.srplace.subregion → cg.coverage.subregion
- dc.Species → cg.species
- dc.contributor.corporate → dc.contributor
- dc.identifier.url → cg.identifier.url
- dc.identifier.doi → cg.identifier.doi
- dc.identifier.googleurl → cg.identifier.googleurl
- dc.identifier.dataurl → cg.identifier.dataurl
Interesting “Sunburst” visualization on a Digital Commons page: http://www.repository.law.indiana.edu/sunburst.html
Final testing on metadata fix/delete for dc.description.sponsorship cleanup
Need to run fix-metadata-values.py and then fix-metadata-values.py

2016-06-20

CGSpace’s HTTPS certificate expired last night and I didn’t notice, had to renew:

# /opt/letsencrypt/letsencrypt-auto renew --standalone --pre-hook "/usr/bin/service nginx stop" --post-hook "/usr/bin/service nginx start"

I really need to fix that cron job…

2016-06-24

Run the replacements/deletes for dc.description.sponsorship (investors) on CGSpace:

$ ./fix-metadata-values.py -i investors-not-blank-not-delete-85.csv -f dc.description.sponsorship -t 'correct investor' -m 29 -d cgspace -p 'fuuu' -u cgspace
$ ./delete-metadata-values.py -i investors-delete-82.csv -f dc.description.sponsorship -m 29 -d cgspace -p 'fuuu' -u cgspace

The scripts for this are here:
- fix-metadata-values.py
- delete-metadata-values.py
Add new sponsors to controlled vocabulary (#244)
Refine submission form labels and hints

2016-06-28

Testing the cleanup of dc.contributor.corporate with 13 deletions and 121 replacements
There are still ~97 fields that weren’t indicated to do anything
After the above deletions and replacements I regenerated a CSV and sent it to Peter et al to have a look

dspacetest=# \copy (select text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=126 group by text_value order by count desc) to /tmp/contributors-june28.csv with csv;

Re-evaluate dc.contributor.corporate and it seems we will move it to dc.contributor.author as this is more in line with how editors are actually using it

2016-06-29

Test run of migrate-fields.sh with the following re-mappings:

72  55  #dc.source
86  230 #cg.contributor.crp
91  211 #cg.contributor.affiliation
94  212 #cg.species
107 231 #cg.coverage.subregion
126 3   #dc.contributor.author
73  219 #cg.identifier.url
74  220 #cg.identifier.doi
79  222 #cg.identifier.googleurl
89  223 #cg.identifier.dataurl

Run all cleanups and deletions of dc.contributor.corporate on CGSpace:

$ ./fix-metadata-values.py -i Corporate-Authors-Fix-121.csv -f dc.contributor.corporate -t 'Correct style' -m 126 -d cgspace -u cgspace -p 'fuuu'
$ ./fix-metadata-values.py -i Corporate-Authors-Fix-PB.csv -f dc.contributor.corporate -t 'should be' -m 126 -d cgspace -u cgspace -p 'fuuu'
$ ./delete-metadata-values.py -f dc.contributor.corporate -i Corporate-Authors-Delete-13.csv -m 126 -u cgspace -d cgspace -p 'fuuu'

Re-deploy CGSpace and DSpace Test with latest June changes
Now the sharing and Altmetric bits are more prominent:

Run all system updates on the servers and reboot
Start working on config changes for phase three of the metadata migrations

2016-06-30

Wow, there are 95 authors in the database who have ‘,’ at the end of their name:

# select text_value from  metadatavalue where metadata_field_id=3 and text_value like '%,';

We need to use something like this to fix them, need to write a proper regex later:

# update metadatavalue set text_value = regexp_replace(text_value, '(Poole, J),', '\1') where metadata_field_id=3 and text_value = 'Poole, J,';

May, 2016

Sun, 01 May 2016 23:06:00 +0300

2016-05-01

Since yesterday there have been 10,000 REST errors and the site has been unstable again
I have blocked access to the API now
There are 3,000 IPs accessing the REST API in a 24-hour period!

# awk '{print $1}' /var/log/nginx/rest.log  | uniq | wc -l
3168

The two most often requesters are in Ethiopia and Colombia: 213.55.99.121 and 181.118.144.29
100% of the requests coming from Ethiopia are like this and result in an HTTP 500:

GET /rest/handle/10568/NaN?expand=parentCommunityList,metadata HTTP/1.1

For now I’ll block just the Ethiopian IP
The owner of that application has said that the NaN (not a number) is an error in his code and he’ll fix it

2016-05-03

Update nginx to 1.10.x branch on CGSpace
Fix a reference to dc.type.output in Discovery that I had missed when we migrated to dc.type last month (#223)

2016-05-06

DSpace Test is down, catalina.out has lots of messages about heap space from some time yesterday (!)
It looks like Sisay was doing some batch imports
Hmm, also disk space is full
I decided to blow away the solr indexes, since they are 50GB and we don’t really need all the Atmire stuff there right now
I will re-generate the Discovery indexes after re-deploying
Testing renew-letsencrypt.sh script for nginx

#!/usr/bin/env bash

readonly SERVICE_BIN=/usr/sbin/service
readonly LETSENCRYPT_BIN=/opt/letsencrypt/letsencrypt-auto

# stop nginx so LE can listen on port 443
$SERVICE_BIN nginx stop

$LETSENCRYPT_BIN renew -nvv --standalone --standalone-supported-challenges tls-sni-01 > /var/log/letsencrypt/renew.log 2>&1

LE_RESULT=$?

$SERVICE_BIN nginx start

if [[ "$LE_RESULT" != 0 ]]; then
    echo 'Automated renewal failed:'

    cat /var/log/letsencrypt/renew.log

    exit 1
fi

Seems to work well

2016-05-10

Start looking at more metadata migrations
There are lots of fields in dcterms namespace that look interesting, like:
- dcterms.type
- dcterms.spatial
Not sure what dcterms is…
Looks like these were added in DSpace 4 to allow for future work to make DSpace more flexible
CGSpace’s dc registry has 96 items, and the default DSpace one has 73.

2016-05-11

Identify and propose the next phase of CGSpace fields to migrate:
- dc.title.jtitle → cg.title.journal
- dc.identifier.status → cg.identifier.status
- dc.river.basin → cg.river.basin
- dc.Species → cg.species
- dc.targetaudience → cg.targetaudience
- dc.fulltextstatus → cg.fulltextstatus
- dc.editon → cg.edition
- dc.isijournal → cg.isijournal
Start a test rebase of the 5_x-prod branch on top of the dspace-5.5 tag
There were a handful of conflicts that I didn’t understand
After completing the rebase I tried to build with the module versions Atmire had indicated as being 5.5 ready but I got this error:

[ERROR] Failed to execute goal on project additions: Could not resolve dependencies for project org.dspace.modules:additions:jar:5.5: Could not find artifact com.atmire:atmire-metadata-quality-api:jar:5.5-2.10.1-0 in sonatype-releases (https://oss.sonatype.org/content/repositories/releases/) -> [Help 1]

I’ve sent them a question about it
A user mentioned having problems with uploading a 33 MB PDF
I told her I would increase the limit temporarily tomorrow morning
Turns out she was able to decrease the size of the PDF so we didn’t have to do anything

2016-05-12

Looks like the issue that Abenet was having a few days ago with “Connection Reset” in Firefox might be due to a Firefox 46 issue: https://bugzilla.mozilla.org/show_bug.cgi?id=1268775
I finally found a copy of the latest CG Core metadata guidelines and it looks like we can add a few more fields to our next migration:
- dc.rplace.region → cg.coverage.region
- dc.cplace.country → cg.coverage.country
Questions for CG people:
- Our dc.place and dc.srplace.subregion could both map to cg.coverage.admin-unit?
- Should we use dc.contributor.crp or cg.contributor.crp for the CRP (ours is dc.crsubject.crpsubject)?
- Our dc.contributor.affiliation and dc.contributor.corporate could both map to dc.contributor and possibly dc.contributor.center depending on if it’s a CG center or not
- dc.title.jtitle could either map to dc.publisher or dc.source depending on how you read things
Found ~200 messed up CIAT values in dc.publisher:

# select text_value from metadatavalue where resource_type_id=2 and metadata_field_id=39 and text_value similar to "%  %";

2016-05-13

More theorizing about CGcore
Add two new fields:
- dc.srplace.subregion → cg.coverage.admin-unit
- dc.place → cg.place
dc.place is our own field, so it’s easy to move
I’ve removed dc.title.jtitle from the list for now because there’s no use moving it out of DC until we know where it will go (see discussion yesterday)

2016-05-18

Work on 707 CCAFS records
They have thumbnails on Flickr and elsewhere
In OpenRefine I created a new filename column based on the thumbnail column with the following GREL:

if(cells['thumbnails'].value.contains('hqdefault'), cells['thumbnails'].value.split('/')[-2] + '.jpg', cells['thumbnails'].value.split('/')[-1])

Because ~400 records had the same filename on Flickr (hqdefault.jpg) but different UUIDs in the URL
So for the hqdefault.jpg ones I just take the UUID (-2) and use it as the filename
Before importing with SAFBuilder I tested adding “__bundle:THUMBNAIL” to the filename column and it works fine

2016-05-19

More quality control on filename field of CCAFS records to make processing in shell and SAFBuilder more reliable:

value.replace('_','').replace('-','')

We need to hold off on moving dc.Species to cg.species because it is only used for plants, and might be better to move it to something like cg.species.plant
And dc.identifier.fund is MOSTLY used for CPWF project identifier but has some other sponsorship things
- We should move PN, SG, CBA, IA, and PHASE* values to cg.identifier.cpwfproject
- The rest, like BMGF and USAID etc, might have to go to either dc.description.sponsorship or cg.identifier.fund (not sure yet)
- There are also some mistakes in CPWF’s things, like “PN 47”
- This ought to catch all the CPWF values (there don’t appear to be and SG* values):

# select text_value from metadatavalue where resource_type_id=2 and metadata_field_id=75 and (text_value like 'PN%' or text_value like 'PHASE%' or text_value = 'CBA' or text_value = 'IA');

2016-05-20

More work on CCAFS Video and Images records
For SAFBuilder we need to modify filename column to have the thumbnail bundle:

value + "__bundle:THUMBNAIL"

Also, I fixed some weird characters using OpenRefine’s transform with the following GREL:

value.replace(/\u0081/,'')

Write shell script to resize thumbnails with height larger than 400: https://gist.github.com/alanorth/131401dcd39d00e0ce12e1be3ed13256
Upload 707 CCAFS records to DSpace Test
A few miscellaneous fixes for XMLUI display niggles (spaces in item lists and link target _black): #224
Work on configuration changes for Phase 2 metadata migrations

2016-05-23

Try to import the CCAFS Images and Videos to CGSpace but had some issues with LibreOffice and OpenRefine
LibreOffice excludes empty cells when it exports and all the fields shift over to the left and cause URLs to go to Subjects, etc.
Google Docs does this better, but somehow reorders the rows and when I paste the thumbnail/filename row in they don’t match!
I will have to try later

2016-05-30

Export CCAFS video and image records from DSpace Test using the migrate option (-m):

$ mkdir ~/ccafs-images
$ /home/dspacetest.cgiar.org/bin/dspace export -t COLLECTION -i 10568/79355 -d ~/ccafs-images -n 0 -m

And then import to CGSpace:

$ JAVA_OPTS="-Xmx512m -Dfile.encoding=UTF-8" /home/cgspace.cgiar.org/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568/70974 --source /tmp/ccafs-images --mapfile=/tmp/ccafs-images-may30.map &> /tmp/ccafs-images-may30.log

But now we have double authors for “CGIAR Research Program on Climate Change, Agriculture and Food Security” in the authority
I’m trying to do a Discovery index before messing with the authority index
Looks like we are missing the index-authority cron job, so who knows what’s up with our authority index
Run system updates on DSpace Test, re-deploy code, and reboot the server
Clean up and import ~200 CTA records to CGSpace via CSV like:

$ export JAVA_OPTS="-Xmx512m -Dfile.encoding=UTF-8"
$ /home/cgspace.cgiar.org/bin/dspace metadata-import -e aorth@mjanja.ch -f ~/CTA-May30/CTA-42229.csv &> ~/CTA-May30/CTA-42229.log

Discovery indexing took a few hours for some reason, and after that I started the index-authority script

$ JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8" /home/cgspace.cgiar.org/bin/dspace index-authority

2016-05-31

The index-authority script ran over night and was finished in the morning
Hopefully this was because we haven’t been running it regularly and it will speed up next time
I am running it again with a timer to see:

$ time /home/cgspace.cgiar.org/bin/dspace index-authority
Retrieving all data
Initialize org.dspace.authority.indexer.DSpaceAuthorityIndexer
Cleaning the old index
Writing new data
All done !

real    37m26.538s
user    2m24.627s
sys     0m20.540s

Update tomcat7 crontab on CGSpace and DSpace Test to have the index-authority script that we were missing
Add new ILRI subject and CCAFS project tags to input-forms.xml (#226, #225)
Manually mapped the authors of a few old CCAFS records to the new CCAFS authority UUID and re-indexed authority indexes to see if it helps correct those items.
Re-sync DSpace Test data with CGSpace
Clean up and import ~65 more CTA items into CGSpace

April, 2016

Mon, 04 Apr 2016 11:06:00 +0300

2016-04-04

Looking at log file use on CGSpace and notice that we need to work on our cron setup a bit
We are backing up all logs in the log folder, including useless stuff like solr, cocoon, handle-plugin, etc
After running DSpace for over five years I’ve never needed to look in any other log file than dspace.log, leave alone one from last year!
This will save us a few gigs of backup space we’re paying for on S3
Also, I noticed the checker log has some errors we should pay attention to:

Run start time: 03/06/2016 04:00:22
Error retrieving bitstream ID 71274 from asset store.
java.io.FileNotFoundException: /home/cgspace.cgiar.org/assetstore/64/29/06/64290601546459645925328536011917633626 (Too many open files)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:146)
        at edu.sdsc.grid.io.local.LocalFileInputStream.open(LocalFileInputStream.java:171)
        at edu.sdsc.grid.io.GeneralFileInputStream.<init>(GeneralFileInputStream.java:145)
        at edu.sdsc.grid.io.local.LocalFileInputStream.<init>(LocalFileInputStream.java:139)
        at edu.sdsc.grid.io.FileFactory.newFileInputStream(FileFactory.java:630)
        at org.dspace.storage.bitstore.BitstreamStorageManager.retrieve(BitstreamStorageManager.java:525)
        at org.dspace.checker.BitstreamDAO.getBitstream(BitstreamDAO.java:60)
        at org.dspace.checker.CheckerCommand.processBitstream(CheckerCommand.java:303)
        at org.dspace.checker.CheckerCommand.checkBitstream(CheckerCommand.java:171)
        at org.dspace.checker.CheckerCommand.process(CheckerCommand.java:120)
        at org.dspace.app.checker.ChecksumChecker.main(ChecksumChecker.java:236)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:225)
        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:77)
******************************************************

So this would be the tomcat7 Unix user, who seems to have a default limit of 1024 files in its shell
For what it’s worth, we have been setting the actual Tomcat 7 process’ limit to 16384 for a few years (in /etc/default/tomcat7)
Looks like cron will read limits from /etc/security/limits.* so we can do something for the tomcat7 user there
Submit pull request for Tomcat 7 limits in Ansible dspace role (#30)

2016-04-05

Reduce Amazon S3 storage used for logs from 46 GB to 6GB by deleting a bunch of logs we don’t need!

# s3cmd ls s3://cgspace.cgiar.org/log/ > /tmp/s3-logs.txt
# grep checker.log /tmp/s3-logs.txt | awk '{print $4}' | xargs s3cmd del
# grep cocoon.log /tmp/s3-logs.txt | awk '{print $4}' | xargs s3cmd del
# grep handle-plugin.log /tmp/s3-logs.txt | awk '{print $4}' | xargs s3cmd del
# grep solr.log /tmp/s3-logs.txt | awk '{print $4}' | xargs s3cmd del

Also, adjust the cron jobs for backups so they only backup dspace.log and some stats files (.dat)
Try to do some metadata field migrations using the Atmire batch UI (dc.Species → cg.species) but it took several hours and even missed a few records

2016-04-06

A better way to move metadata on this scale is via SQL, for example dc.type.output → dc.type (their IDs in the metadatafieldregistry are 66 and 109, respectively):

dspacetest=# update metadatavalue set metadata_field_id=109 where metadata_field_id=66;
UPDATE 40852

After that an index-discovery -bf is required
Start working on metadata migrations, add 25 or so new metadata fields to CGSpace

2016-04-07

Write shell script to do the migration of fields: https://gist.github.com/alanorth/72a70aca856d76f24c127a6e67b3342b
Testing with a few fields it seems to work well:

$ ./migrate-fields.sh
UPDATE metadatavalue SET metadata_field_id=109 WHERE metadata_field_id=66
UPDATE 40883
UPDATE metadatavalue SET metadata_field_id=202 WHERE metadata_field_id=72
UPDATE 21420
UPDATE metadatavalue SET metadata_field_id=203 WHERE metadata_field_id=76
UPDATE 51258

2016-04-08

Discuss metadata renaming with Abenet, we decided it’s better to start with the center-specific subjects like ILRI, CIFOR, CCAFS, IWMI, and CPWF
I’ve e-mailed CCAFS and CPWF people to ask them how much time it will take for them to update their systems to cope with this change

2016-04-10

Looking at the DOI issue reported by Leroy from CIAT a few weeks ago
It seems the dx.doi.org URLs are much more proper in our repository!

dspacetest=# select count(*) from metadatavalue where metadata_field_id=74 and text_value like 'http://dx.doi.org%';
 count
-------
  5638
(1 row)

dspacetest=# select count(*) from metadatavalue where metadata_field_id=74 and text_value like 'http://doi.org%';
 count
-------
     3

I will manually edit the dc.identifier.doi in ¹⁰⁵⁶⁸⁄₇₂₅₀₉ and tweet the link, then check back in a week to see if the donut gets updated

2016-04-11

The donut is already updated and shows the correct number now
CCAFS people say it will only take them an hour to update their code for the metadata renames, so I proposed we’d do it tentatively on Monday the 18th.

2016-04-12

Looking at quality of WLE data (cg.subject.iwmi) in SQL:

dspacetest=# select text_value, count(*) from metadatavalue where metadata_field_id=217 group by text_value order by count(*) desc;

Listings and Reports is still not returning reliable data for dc.type
I think we need to ask Atmire, as their documentation isn’t too clear on the format of the filter configs
Alternatively, I want to see if I move all the data from dc.type.output to dc.type and then re-index, if it behaves better
Looking at our input-forms.xml I see we have two sets of ILRI subjects, but one has a few extra subjects
Remove one set of ILRI subjects and remove duplicate VALUE CHAINS from existing list (#216)
I decided to keep the set of subjects that had FMD and RANGELANDS added, as it appears to have been requested to have been added, and might be the newer list
I found 226 blank metadatavalues:

dspacetest# select * from metadatavalue where resource_type_id=2 and text_value='';

I think we should delete them and do a full re-index:

dspacetest=# delete from metadatavalue where resource_type_id=2 and text_value='';
DELETE 226

I deleted them on CGSpace but I’ll wait to do the re-index as we’re going to be doing one in a few days for the metadata changes anyways
In other news, moving the dc.type.output to dc.type and re-indexing seems to have fixed the Listings and Reports issue from above
Unfortunately this isn’t a very good solution, because Listings and Reports config should allow us to filter on dc.type.* but the documentation isn’t very clear and I couldn’t reach Atmire today
We want to do the dc.type.output move on CGSpace anyways, but we should wait as it might affect other external people!

2016-04-14

Communicate with Macaroni Bros again about dc.type
Help Sisay with some rsync and Linux stuff
Notify CIAT people of metadata changes (I had forgotten them last week)

2016-04-15

DSpace Test had crashed, so I ran all system updates, rebooted, and re-deployed DSpace code

2016-04-18

Talk to CIAT people about their portal again
Start looking more at the fields we want to delete
The following metadata fields have 0 items using them, so we can just remove them from the registry and any references in XMLUI, input forms, etc:
- dc.description.abstractother
- dc.whatwasknown
- dc.whatisnew
- dc.description.nationalpartners
- dc.peerreviewprocess
- cg.species.animal
Deleted!
The following fields have some items using them and I have to decide what to do with them (delete or move):
- dc.icsubject.icrafsubject: 6 items, mostly in CPWF collections
- dc.type.journal: 11 items, mostly in ILRI collections
- dc.publicationcategory: 1 item, in CPWF
- dc.GRP: 2 items, CPWF
- dc.Species.animal: 6 items, in ILRI and AnGR
- cg.livestock.agegroup: 9 items, in ILRI collections
- cg.livestock.function: 20 items, mostly in EADD
Test metadata migration on local instance again:

$ ./migrate-fields.sh
UPDATE metadatavalue SET metadata_field_id=109 WHERE metadata_field_id=66
UPDATE 40885
UPDATE metadatavalue SET metadata_field_id=203 WHERE metadata_field_id=76
UPDATE 51330
UPDATE metadatavalue SET metadata_field_id=208 WHERE metadata_field_id=82
UPDATE 5986
UPDATE metadatavalue SET metadata_field_id=210 WHERE metadata_field_id=88
UPDATE 2456
UPDATE metadatavalue SET metadata_field_id=215 WHERE metadata_field_id=106
UPDATE 3872
UPDATE metadatavalue SET metadata_field_id=217 WHERE metadata_field_id=108
UPDATE 46075
$ JAVA_OPTS="-Xms512m -Xmx512m -Dfile.encoding=UTF-8" ~/dspace/bin/dspace index-discovery -bf

CGSpace was down but I’m not sure why, this was in catalina.out:

Apr 18, 2016 7:32:26 PM com.sun.jersey.spi.container.ContainerResponse logException
SEVERE: Mapped exception to response: 500 (Internal Server Error)
javax.ws.rs.WebApplicationException
        at org.dspace.rest.Resource.processFinally(Resource.java:163)
        at org.dspace.rest.HandleResource.getObject(HandleResource.java:81)
        at sun.reflect.GeneratedMethodAccessor198.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1511)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1442)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1391)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1381)
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
...

Everything else in the system looked normal (50GB disk space available, nothing weird in dmesg, etc)
After restarting Tomcat a few more of these errors were logged but the application was up

2016-04-19

Get handles for items that are using a given metadata field, ie dc.Species.animal (105):

# select handle from item, handle where handle.resource_id = item.item_id AND item.item_id in (select resource_id from metadatavalue where resource_type_id=2 and metadata_field_id=105);
   handle
-------------
 10568/10298
 10568/16413
 10568/16774
 10568/34487

Delete metadata values for dc.GRP and dc.icsubject.icrafsubject:

# delete from metadatavalue where resource_type_id=2 and metadata_field_id=96;
# delete from metadatavalue where resource_type_id=2 and metadata_field_id=83;

They are old ICRAF fields and we haven’t used them since 2011 or so
Also delete them from the metadata registry
CGSpace went down again, dspace.log had this:

2016-04-19 15:02:17,025 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error -
org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object

I restarted Tomcat and PostgreSQL and now it’s back up
I bet this is the same crash as yesterday, but I only saw the errors in catalina.out
Looks to be related to this, from dspace.log:

2016-04-19 15:16:34,670 ERROR org.dspace.rest.Resource @ Something get wrong. Aborting context in finally statement.

We have 18,000 of these errors right now…
Delete a few more old metadata values: dc.Species.animal, dc.type.journal, and dc.publicationcategory:

# delete from metadatavalue where resource_type_id=2 and metadata_field_id=105;
# delete from metadatavalue where resource_type_id=2 and metadata_field_id=85;
# delete from metadatavalue where resource_type_id=2 and metadata_field_id=95;

And then remove them from the metadata registry

2016-04-20

Re-deploy DSpace Test with the new subject and type fields, run all system updates, and reboot the server
Migrate fields and re-deploy CGSpace with the new subject and type fields, run all system updates, and reboot the server
Field migration went well:

$ ./migrate-fields.sh
UPDATE metadatavalue SET metadata_field_id=109 WHERE metadata_field_id=66
UPDATE 40909
UPDATE metadatavalue SET metadata_field_id=203 WHERE metadata_field_id=76
UPDATE 51419
UPDATE metadatavalue SET metadata_field_id=208 WHERE metadata_field_id=82
UPDATE 5986
UPDATE metadatavalue SET metadata_field_id=210 WHERE metadata_field_id=88
UPDATE 2458
UPDATE metadatavalue SET metadata_field_id=215 WHERE metadata_field_id=106
UPDATE 3872
UPDATE metadatavalue SET metadata_field_id=217 WHERE metadata_field_id=108
UPDATE 46075

Also, I migrated CGSpace to using the PGDG PostgreSQL repo as the infrastructure playbooks had been using it for a while and it seemed to be working well
Basically, this gives us the ability to use the latest upstream stable 9.3.x release (currently 9.3.12)
Looking into the REST API errors again, it looks like these started appearing a few days ago in the tens of thousands:

$ grep -c "Aborting context in finally statement" dspace.log.2016-04-20
21252

I found a recent discussion on the DSpace mailing list and I’ve asked for advice there
Looks like this issue was noted and fixed in DSpace 5.5 (we’re on 5.1): https://jira.duraspace.org/browse/DS-2936
I’ve sent a message to Atmire asking about compatibility with DSpace 5.5

2016-04-21

Fix a bunch of metadata consistency issues with IITA Journal Articles (Peer review, Formally published, messed up DOIs, etc)
Atmire responded with DSpace 5.5 compatible versions for their modules, so I’ll start testing those in a few weeks

2016-04-22

Import 95 records into CTA’s Agrodok collection

2016-04-26

Test embargo during item upload
Seems to be working but the help text is misleading as to the date format
It turns out the robots.txt issue we thought we solved last month isn’t solved because you can’t use wildcards in URL patterns: https://jira.duraspace.org/browse/DS-2962
Write some nginx rules to add X-Robots-Tag HTTP headers to the dynamic requests from robots.txt instead
A few URLs to test with:

2016-04-27

I woke up to ten or fifteen “up” and “down” emails from the monitoring website
Looks like the last one was “down” from about four hours ago
I think there must be something with this REST stuff:

# grep -c "Aborting context in finally statement" dspace.log.2016-04-*
dspace.log.2016-04-01:0
dspace.log.2016-04-02:0
dspace.log.2016-04-03:0
dspace.log.2016-04-04:0
dspace.log.2016-04-05:0
dspace.log.2016-04-06:0
dspace.log.2016-04-07:0
dspace.log.2016-04-08:0
dspace.log.2016-04-09:0
dspace.log.2016-04-10:0
dspace.log.2016-04-11:0
dspace.log.2016-04-12:235
dspace.log.2016-04-13:44
dspace.log.2016-04-14:0
dspace.log.2016-04-15:35
dspace.log.2016-04-16:0
dspace.log.2016-04-17:0
dspace.log.2016-04-18:11942
dspace.log.2016-04-19:28496
dspace.log.2016-04-20:28474
dspace.log.2016-04-21:28654
dspace.log.2016-04-22:28763
dspace.log.2016-04-23:28773
dspace.log.2016-04-24:28775
dspace.log.2016-04-25:28626
dspace.log.2016-04-26:28655
dspace.log.2016-04-27:7271

I restarted tomcat and it is back up
Add Spanish XMLUI strings so those users see “CGSpace” instead of “DSpace” in the user interface (#222)
Submit patch to upstream DSpace for the misleading help text in the embargo step of the item submission: https://jira.duraspace.org/browse/DS-3172
Update infrastructure playbooks for nginx 1.10.x (stable) release: https://github.com/ilri/rmg-ansible-public/issues/32
Currently running on DSpace Test, we’ll give it a few days before we adjust CGSpace
CGSpace down, restarted tomcat and it’s back up

2016-04-28

Problems with stability again. I’ve blocked access to /rest for now to see if the number of errors in the log files drop
Later we could maybe start logging access to /rest and perhaps whitelist some IPs…

2016-04-30

Logs for today and yesterday have zero references to this REST error, so I’m going to open back up the REST API but log all requests

location /rest {
	access_log /var/log/nginx/rest.log;
	proxy_pass http://127.0.0.1:8443;
}

I will check the logs again in a few days to look for patterns, see who is accessing it, etc

March, 2016

Wed, 02 Mar 2016 16:50:00 +0300

2016-03-02

Looking at issues with author authorities on CGSpace
For some reason we still have the index-lucene-update cron job active on CGSpace, but I’m pretty sure we don’t need it as of the latest few versions of Atmire’s Listings and Reports module
Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Java JDK 1.7 to match environment on CGSpace server

2016-03-07

Troubleshooting the issues with the slew of commits for Atmire modules in #182
Their changes on 5_x-dev branch work, but it is messy as hell with merge commits and old branch base
When I rebase their branch on the latest 5_x-prod I get blank white pages
I identified one commit that causes the issue and let them know
Restart DSpace Test, as it seems to have crashed after Sisay tried to import some CSV or zip or something:

Exception in thread "Lucene Merge Thread #19" org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: No space left on device

2016-03-08

Add a few new filters to Atmire’s Listings and Reports module (#180)
We had also wanted to add a few to the Content and Usage module but I have to ask the editors which ones they were

2016-03-10

Disable the lucene cron job on CGSpace as it shouldn’t be needed anymore
Discuss ORCiD and duplicate authors on Yammer
Request new documentation for Atmire CUA and L&R modules, as ours are from 2013
Walk Sisay through some data cleaning workflows in OpenRefine
Start cleaning up the configuration for Atmire’s CUA module (#184)
It is very messed up because some labels are incorrect, fields are missing, etc

Update documentation for Atmire modules

2016-03-11

As I was looking at the CUA config I realized our Discovery config is all messed up and confusing
I’ve opened an issue to track some of that work (#186)
I did some major cleanup work on Discovery and XMLUI stuff related to the dc.type indexes (#187)
We had been confusing dc.type (a Dublin Core value) with dc.type.output (a value we invented) for a few years and it had permeated all aspects of our data, indexes, item displays, etc.
There is still some more work to be done to remove references to old outputtype and output

2016-03-14

Fix some items that had invalid dates (I noticed them in the log during a re-indexing)
Reset search.index.* to the default, as it is only used by Lucene (deprecated by Discovery in DSpace 5.x): #188
Make titles in Discovery and Browse by more consistent (singular, sentence case, etc) (#186)
Also four or so center-specific subject strings were missing for Discovery

2016-03-15

Create simple theme for new AVCD community just for a unique Google Tracking ID (#191)

2016-03-16

Still having problems deploying Atmire’s CUA updates and fixes from January!
More discussion on the GitHub issue here: https://github.com/ilri/DSpace/pull/182
Clean up Atmire CUA config (#193)
Help Sisay with some PostgreSQL queries to clean up the incorrect dc.contributor.corporateauthor field
I noticed that we have some weird values in dc.language:

# select * from metadatavalue where metadata_field_id=37;
 metadata_value_id | resource_id | metadata_field_id | text_value | text_lang | place | authority | confidence | resource_type_id
-------------------+-------------+-------------------+------------+-----------+-------+-----------+------------+------------------
           1942571 |       35342 |                37 | hi         |           |     1 |           |         -1 |                2
           1942468 |       35345 |                37 | hi         |           |     1 |           |         -1 |                2
           1942479 |       35337 |                37 | hi         |           |     1 |           |         -1 |                2
           1942505 |       35336 |                37 | hi         |           |     1 |           |         -1 |                2
           1942519 |       35338 |                37 | hi         |           |     1 |           |         -1 |                2
           1942535 |       35340 |                37 | hi         |           |     1 |           |         -1 |                2
           1942555 |       35341 |                37 | hi         |           |     1 |           |         -1 |                2
           1942588 |       35343 |                37 | hi         |           |     1 |           |         -1 |                2
           1942610 |       35346 |                37 | hi         |           |     1 |           |         -1 |                2
           1942624 |       35347 |                37 | hi         |           |     1 |           |         -1 |                2
           1942639 |       35339 |                37 | hi         |           |     1 |           |         -1 |                2

It seems this dc.language field isn’t really used, but we should delete these values
Also, dc.language.iso has some weird values, like “En” and “English”

2016-03-17

It turns out hi is the ISO 639 language code for Hindi, but these should be in dc.language.iso instead of dc.language
I fixed the eleven items with hi as well as some using the incorrect vn for Vietnamese
Start discussing CG core with Abenet and Sisay
Re-sync CGSpace database to DSpace Test for Atmire to do some tests about the problematic CUA patches
The patches work fine with a clean database, so the error was caused by some mismatch in CUA versions and the database during my testing

2016-03-18

Merge Atmire fixes into 5_x-prod
Discuss thumbnails with Francesca from Bioversity
Some of their items end up with thumbnails that have a big white border around them:

Turns out we can add -trim to the GraphicsMagick options to trim the whitespace

Command used:

$ gm convert -trim -quality 82 -thumbnail x300 -flatten Descriptor\ for\ Butia_EN-2015_2021.pdf\[0\] cover.jpg

Also, it looks like adding -sharpen 0x1.0 really improves the quality of the image for only a few KB

2016-03-21

Fix 66 site errors in Google’s webmaster tools
I looked at a bunch of them and they were old URLs, weird things linked from non-existent items, etc, so I just marked them all as fixed
We also have 1,300 “soft 404” errors for URLs like: https://cgspace.cgiar.org/handle/10568/440/browse?type=bioversity
I’ve marked them as fixed as well since the ones I tested were working fine
This raises another question, as many of these pages are linked from Discovery search results and might create a duplicate content problem…
Results pages like this give items that Google already knows from the sitemap: https://cgspace.cgiar.org/discover?filtertype=author&filter_relational_operator=equals&filter=Orth%2C+A.
There are some access denied errors on JSPUI links (of course! we forbid them!), but I’m not sure why Google is trying to index them…
For example:
- This: https://cgspace.cgiar.org/jspui/bitstream/10568/809/1/main-page.pdf
- Linked from: https://cgspace.cgiar.org/jspui/handle/10568/809
I will mark these errors as resolved because they are returning HTTP 403 on purpose, for a long time!
Google says the first time it saw this particular error was September 29, 2015… so maybe it accidentally saw it somehow…
On a related note, we have 51,000 items indexed from the sitemap, but 500,000 items in the Google index, so we DEFINITELY have a problem with duplicate content

Turns out this is a problem with DSpace’s robots.txt, and there’s a Jira ticket since December, 2015: https://jira.duraspace.org/browse/DS-2962
I am not sure if I want to apply it yet
For now I’ve just set a bunch of these dynamic pages to not appear in search results by using the URL Parameters tool in Webmaster Tools

Move AVCD collection to new community and update move_collection.sh script: https://gist.github.com/alanorth/392c4660e8b022d99dfa
It seems Feedburner can do HTTPS now, so we might be able to update our feeds and simplify the nginx configs
De-deploy CGSpace with latest 5_x-prod branch
Run updates on CGSpace and reboot server (new kernel, 4.5.0)
Deploy Let’s Encrypt certificate for cgspace.cgiar.org, but still need to work it into the ansible playbooks

2016-03-22

Merge robots.txt patch and disallow indexing of browse pages as our sitemap is consumed correctly (#198)

2016-03-23

Abenet is having problems saving group memberships, and she gets this error: https://gist.github.com/alanorth/87281c061c2de57b773e

Can't find method org.dspace.app.xmlui.aspect.administrative.FlowGroupUtils.processSaveGroup(org.dspace.core.Context,number,string,[Ljava.lang.String;,[Ljava.lang.String;,org.apache.cocoon.environment.wrapper.RequestWrapper). (resource://aspects/Administrative/administrative.js#967)

I can reproduce the same error on DSpace Test and on my Mac
Looks to be an issue with the Atmire modules, I’ve submitted a ticket to their tracker.

2016-03-24

Atmire sent a patch for the group saving issue: https://github.com/ilri/DSpace/pull/201
I tested it locally and it works, so I merged it to 5_x-prod and will deploy on CGSpace this week

2016-03-25

Having problems with Listings and Reports, seems to be caused by a rogue reference to dc.type.output
This is the error we get when we proceed to the second page of Listings and Reports: https://gist.github.com/alanorth/b2d7fb5b82f94898caaf
Commenting out the line works, but I haven’t figured out the proper syntax for referring to dc.type.*

2016-03-28

Look into enabling the embargo during item submission, see: https://wiki.duraspace.org/display/DSDOC5x/Embargo#Embargo-SubmissionProcess
Seems we only want AccessStep because UploadWithEmbargoStep disables the ability to edit embargos at the item level
This pull request enables the ability to set an item-level embargo during submission: https://github.com/ilri/DSpace/pull/203
I figured out that the problem with Listings and Reports was because I disabled the search.index.* last week, and they are still used by JSPUI apparently
This pull request re-enables them: https://github.com/ilri/DSpace/pull/202
Re-deploy DSpace Test, run all system updates, and restart the server
Looks like the Listings and Reports fix was NOT due to the search indexes (which are actually not used), and rather due to the filter configuration in the Listings and Reports config
This pull request simply updates the config for the dc.type.output → dc.type change that was made last week: https://github.com/ilri/DSpace/pull/204
Deploy robots.txt fix, embargo for item submissions, and listings and reports fix on CGSpace

2016-03-29

Skype meeting with Peter and Addis team to discuss metadata changes for Dublin Core, CGcore, and CGSpace-specific fields
We decided to proceed with some deletes first, then identify CGSpace-specific fields to clean/move to cg.*, and then worry about broader changes to DC
Before we move or rename and fields we need to circulate a list of fields we intend to change to CCAFS, CWPF, etc who might be harvesting the fields
After all of this we need to start implementing controlled vocabularies for fields, either with the Javascript lookup or like existing ILRI subjects

February, 2016

Fri, 05 Feb 2016 13:18:00 +0300

2016-02-05

Looking at some DAGRIS data for Abenet Yabowork
Lots of issues with spaces, newlines, etc causing the import to fail
I noticed we have a very interesting list of countries on CGSpace:

Not only are there 49,000 countries, we have some blanks (25)…
Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE”

2016-02-06

Found a way to get items with null/empty metadata values from SQL
First, find the metadata_field_id for the field you want from the metadatafieldregistry table:

dspacetest=# select * from metadatafieldregistry;

In this case our country field is 78
Now find all resources with type 2 (item) that have null/empty values for that field:

dspacetest=# select resource_id from metadatavalue where resource_type_id=2 and metadata_field_id=78 and (text_value='' OR text_value IS NULL);

Then you can find the handle that owns it from its resource_id:

dspacetest=# select handle from item, handle where handle.resource_id = item.item_id AND item.item_id = '22678';

It’s 25 items so editing in the web UI is annoying, let’s try SQL!

dspacetest=# delete from metadatavalue where metadata_field_id=78 and text_value='';
DELETE 25

After that perhaps a regular dspace index-discovery (no -b) should suffice…
Hmm, I indexed, cleared the Cocoon cache, and restarted Tomcat but the 25 “|||” countries are still there
Maybe I need to do a full re-index…
Yep! The full re-index seems to work.
Process the empty countries on CGSpace

2016-02-07

Working on cleaning up Abenet’s DAGRIS data with OpenRefine
I discovered two really nice functions in OpenRefine: value.trim() and value.escape("javascript") which shows whitespace characters like \r\n!
For some reason when you import an Excel file into OpenRefine it exports dates like 1949 to 1949.0 in the CSV
I re-import the resulting CSV and run a GREL on the date issued column: value.replace("\.0", "")
I need to start running DSpace in Mac OS X instead of a Linux VM
Install PostgreSQL from homebrew, then configure and import CGSpace database dump:

$ postgres -D /opt/brew/var/postgres
$ createuser --superuser postgres
$ createuser --pwprompt dspacetest
$ createdb -O dspacetest --encoding=UNICODE dspacetest
$ psql postgres
postgres=# alter user dspacetest createuser;
postgres=# \q
$ pg_restore -O -U dspacetest -d dspacetest ~/Downloads/cgspace_2016-02-07.backup 
$ psql postgres
postgres=# alter user dspacetest nocreateuser;
postgres=# \q
$ vacuumdb dspacetest
$ psql -U dspacetest -f ~/src/git/DSpace/dspace/etc/postgres/update-sequences.sql dspacetest -h localhost

After building and running a fresh_install I symlinked the webapps into Tomcat’s webapps folder:

$ mv /opt/brew/Cellar/tomcat/8.0.30/libexec/webapps/ROOT /opt/brew/Cellar/tomcat/8.0.30/libexec/webapps/ROOT.orig
$ ln -sfv ~/dspace/webapps/xmlui /opt/brew/Cellar/tomcat/8.0.30/libexec/webapps/ROOT
$ ln -sfv ~/dspace/webapps/rest /opt/brew/Cellar/tomcat/8.0.30/libexec/webapps/rest
$ ln -sfv ~/dspace/webapps/jspui /opt/brew/Cellar/tomcat/8.0.30/libexec/webapps/jspui
$ ln -sfv ~/dspace/webapps/oai /opt/brew/Cellar/tomcat/8.0.30/libexec/webapps/oai
$ ln -sfv ~/dspace/webapps/solr /opt/brew/Cellar/tomcat/8.0.30/libexec/webapps/solr
$ /opt/brew/Cellar/tomcat/8.0.30/bin/catalina start

Add CATALINA_OPTS in /opt/brew/Cellar/tomcat/8.0.30/libexec/bin/setenv.sh, as this script is sourced by the catalina startup script
For example:

CATALINA_OPTS="-Djava.awt.headless=true -Xms2048m -Xmx2048m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -Dfile.encoding=UTF-8"

After verifying that the site is working, start a full index:

$ ~/dspace/bin/dspace index-discovery -b

2016-02-08

Finish cleaning up and importing ~400 DAGRIS items into CGSpace
Whip up some quick CSS to make the button in the submission workflow use the XMLUI theme’s brand colors (#154)

2016-02-09

Re-sync DSpace Test with CGSpace
Help Sisay with OpenRefine
Enable HTTPS on DSpace Test using Let’s Encrypt:

$ cd ~/src/git
$ git clone https://github.com/letsencrypt/letsencrypt
$ cd letsencrypt
$ sudo service nginx stop
# add port 443 to firewall rules
$ ./letsencrypt-auto certonly --standalone -d dspacetest.cgiar.org
$ sudo service nginx start
$ ansible-playbook dspace.yml -l linode02 -t nginx,firewall -u aorth --ask-become-pass

We should install it in /opt/letsencrypt and then script the renewal script, but first we have to wire up some variables and template stuff based on the script here: https://letsencrypt.org/howitworks/
I had to export some CIAT items that were being cleaned up on the test server and I noticed their dc.contributor.author fields have DSpace 5 authority index UUIDs…
To clean those up in OpenRefine I used this GREL expression: value.replace(/::\w{8}-\w{4}-\w{4}-\w{4}-\w{12}::600/,"")
Getting more and more hangs on DSpace Test, seemingly random but also during CSV import
Logs don’t always show anything right when it fails, but eventually one of these appears:

org.dspace.discovery.SearchServiceException: Error while processing facet fields: java.lang.OutOfMemoryError: Java heap space

Caused by: java.util.NoSuchElementException: Timeout waiting for idle object

Right now DSpace Test’s Tomcat heap is set to 1536m and we have quite a bit of free RAM:

# free -m
             total       used       free     shared    buffers     cached
Mem:          3950       3902         48          9         37       1311
-/+ buffers/cache:       2552       1397
Swap:          255         57        198

So I’ll bump up the Tomcat heap to 2048 (CGSpace production server is using 3GB)

2016-02-11

Massaging some CIAT data in OpenRefine
There are 1200 records that have PDFs, and will need to be imported into CGSpace
I created a filename column based on the dc.identifier.url column using the following transform:

value.split('/')[-1]

Then I wrote a tool called generate-thumbnails.py to download the PDFs and generate thumbnails for them, for example:

$ ./generate-thumbnails.py ciat-reports.csv
Processing 64661.pdf
> Downloading 64661.pdf
> Creating thumbnail for 64661.pdf
Processing 64195.pdf
> Downloading 64195.pdf
> Creating thumbnail for 64195.pdf

2016-02-12

Looking at CIAT’s records again, there are some problems with a dozen or so files (out of 1200)
A few items are using the same exact PDF
A few items are using HTM or DOC files
A few items link to PDFs on IFPRI’s e-Library or Research Gate
A few items have no item
Also, I’m not sure if we import these items, will be remove the dc.identifier.url field from the records?

2016-02-12

Looking at CIAT’s records again, there are some files linking to PDFs on Slide Share, Embrapa, UEA UK, and Condesan, so I’m not sure if we can use those
265 items have dirty, URL-encoded filenames:

$ ls | grep -c -E "%"
265

I suggest that we import ~850 or so of the clean ones first, then do the rest after I can find a clean/reliable way to decode the filenames
This python2 snippet seems to work in the CLI, but not so well in OpenRefine:

$ python -c "import urllib, sys; print urllib.unquote(sys.argv[1])" CIAT_COLOMBIA_000169_T%C3%A9cnicas_para_el_aislamiento_y_cultivo_de_protoplastos_de_yuca.pdf
CIAT_COLOMBIA_000169_Técnicas_para_el_aislamiento_y_cultivo_de_protoplastos_de_yuca.pdf

Merge pull requests for submission form theming (#178) and missing center subjects in XMLUI item views (#176)
They will be deployed on CGSpace the next time I re-deploy

2016-02-16

Turns out OpenRefine has an unescape function!

value.unescape("url")

This turns the URLs into human-readable versions that we can use as proper filenames
Run web server and system updates on DSpace Test and reboot
To merge dc.identifier.url and dc.identifier.url[], rename the second column so it doesn’t have the brackets, like dc.identifier.url2
Then you create a facet for blank values on each column, show the rows that have values for one and not the other, then transform each independently to have the contents of the other, with “||” in between
Work on Python script for parsing and downloading PDF records from dc.identifier.url
To get filenames from dc.identifier.url, create a new column based on this transform: forEach(value.split('||'), v, v.split('/')[-1]).join('||')
This also works for records that have multiple URLs (separated by “||”)

2016-02-17

Re-deploy CGSpace, run all system updates, and reboot
More work on CIAT data, cleaning and doing a last metadata-only import into DSpace Test
SAFBuilder has a bug preventing it from processing filenames containing more than one underscore
Need to re-process the filename column to replace multiple underscores with one: value.replace(/_{2,}/, "_")

2016-02-20

Turns out the “bug” in SAFBuilder isn’t a bug, it’s a feature that allows you to encode extra information like the destintion bundle in the filename
Also, it seems DSpace’s SAF import tool doesn’t like importing filenames that have accents in them:

java.io.FileNotFoundException: /usr/share/tomcat7/SimpleArchiveFormat/item_1021/CIAT_COLOMBIA_000075_Medición_de_palatabilidad_en_forrajes.pdf (No such file or directory)

Need to rename files to have no accents or umlauts, etc…
Useful custom text facet for URLs ending with “.pdf”: value.endsWith(".pdf")

2016-02-22

To change Spanish accents to ASCII in OpenRefine:

value.replace('ó','o').replace('í','i').replace('á','a').replace('é','e').replace('ñ','n')

But actually, the accents might not be an issue, as I can successfully import files containing Spanish accents on my Mac
On closer inspection, I can import files with the following names on Linux (DSpace Test):

Bitstream: tést.pdf
Bitstream: tést señora.pdf
Bitstream: tést señora alimentación.pdf

Seems it could be something with the HFS+ filesystem actually, as it’s not UTF-8 (it’s something like UCS-2)
HFS+ stores filenames as a string, and filenames with accents get stored as character+accent whereas Linux’s ext4 stores them as an array of bytes
Running the SAFBuilder on Mac OS X works if you’re going to import the resulting bundle on Mac OS X, but if your DSpace is running on Linux you need to run the SAFBuilder there where the filesystem’s encoding matches

2016-02-29

Got notified by some CIFOR colleagues that the Google Scholar team had contacted them about CGSpace’s incorrect ordering of authors in Google Scholar metadata
Turns out there is a patch, and it was merged in DSpace 5.4: https://jira.duraspace.org/browse/DS-2679
I’ve merged it into our 5_x-prod branch that is currently based on DSpace 5.1
We found a bug when a user searches from the homepage, sorts the results, and then tries to click “View More” in a sidebar facet
I am not sure what causes it yet, but I opened an issue for it: https://github.com/ilri/DSpace/issues/179
Have more problems with SAFBuilder on Mac OS X
Now it doesn’t recognize description hints in the filename column, like: test.pdf__description:Blah
But on Linux it works fine
Trying to test Atmire’s series of stats and CUA fixes from January and February, but their branch history is really messy and it’s hard to see what’s going on
Rebasing their branch on top of our production branch results in a broken Tomcat, so I’m going to tell them to fix their history and make a proper pull request
Looking at the filenames for the CIAT Reports, some have some really ugly characters, like: ' or , or = or [ or ] or ( or ) or _.pdf or ._ etc
It’s tricky to parse those things in some programming languages so I’d rather just get rid of the weird stuff now in OpenRefine:

value.replace("'",'').replace('_=_','_').replace(',','').replace('[','').replace(']','').replace('(','').replace(')','').replace('_.pdf','.pdf').replace('._','_')

Finally import the 1127 CIAT items into CGSpace: https://cgspace.cgiar.org/handle/10568/35710
Re-deploy CGSpace with the Google Scholar fix, but I’m waiting on the Atmire fixes for now, as the branch history is ugly

January, 2016

Wed, 13 Jan 2016 13:18:00 +0300

2016-01-13

Move ILRI collection 10568/12503 from 10568/27869 to 10568/27629 using the move_collections.sh script I wrote last year.
I realized it is only necessary to clear the Cocoon cache after moving collections—rather than reindexing—as no metadata has changed, and therefore no search or browse indexes need to be updated.
Update GitHub wiki for documentation of maintenance tasks.

2016-01-14

Update CCAFS project identifiers in input-forms.xml
Run system updates and restart the server

2016-01-18

Change “Extension material” to “Extension Material” in input-forms.xml (a mistake that fell through the cracks when we fixed the others in DSpace 4 era)

2016-01-19

Work on tweaks and updates for the social sharing icons on item pages: add Delicious and Mendeley (from Academicons), make links open in new windows, and set the icon color to the theme’s primary color (#157)
Tweak date-based facets to show more values in drill-down ranges (#162)
Need to remember to clear the Cocoon cache after deployment or else you don’t see the new ranges immediately
Set up recipe on IFTTT to tweet new items from the CGSpace Atom feed to my twitter account
Altmetrics’ support for Handles is kinda weak, so they can’t associate our items with DOIs until they are tweeted or blogged, etc first.

2016-01-21

Still waiting for my IFTTT recipe to fire, two days later
It looks like the Atom feed on CGSpace hasn’t changed in two days, but there have definitely been new items
The RSS feed is nearly as old, but has different old items there
On a hunch I cleared the Cocoon cache and now the feeds are fresh
Looks like there is configuration option related to this, webui.feed.cache.age, which defaults to 48 hours, though I’m not sure what relation it has to the Cocoon cache
In any case, we should change this cache to be something more like 6 hours, as we publish new items several times per day.
Work around a CSS issue with long URLs in the item view (#172)

2016-01-25

Re-deploy CGSpace and DSpace Test with latest 5_x-prod branch
This included the social icon fixes/updates, date-based facet tweaks, reducing the feed cache age, and fixing a layout issue in XMLUI item view when an item had long URLs

2016-01-26

Run nginx updates on CGSpace and DSpace Test (1.8.1 and 1.9.10, respectively)
Run updates on DSpace Test and reboot for new Linode kernel Linux 4.4.0-x86_64-linode63 (first update in months)

2016-01-28

Start looking at importing some Bioversity data that had been prepared earlier this week
While checking the data I noticed something strange, there are 79 items but only 8 unique PDFs:

$ ls SimpleArchiveForBio/ | wc -l 79 $ find SimpleArchiveForBio/ -iname “*.pdf” -exec basename {} \; | sort -u | wc -l 8

2016-01-29

Add five missing center-specific subjects to XMLUI item view (#174)
This CCAFS item Before:

After: