2018-06-04
- Test the DSpace 5.8 module upgrades from Atmire (#378)
- There seems to be a problem with the CUA and L&R versions in
pom.xml
because they are using SNAPSHOT and it doesn’t build
- I added the new CCAFS Phase II Project Tag
PII-FP1_PACCA2
and merged it into the 5_x-prod
branch (#379)
- I proofed and tested the ILRI author corrections that Peter sent back to me this week:
$ ./fix-metadata-values.py -i /tmp/2018-05-30-Correct-660-authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t correct -m 3 -n
- I think a sane proofing workflow in OpenRefine is to apply the custom text facets for check/delete/remove and illegal characters that I developed in March, 2018
- Time to index ~70,000 items on CGSpace:
$ time schedtool -D -e ionice -c2 -n7 nice -n19 [dspace]/bin/dspace index-discovery -b
real 74m42.646s
user 8m5.056s
sys 2m7.289s
2018-06-06
- It turns out that I needed to add a server block for
atmire.com-snapshots
to my Maven settings, so now the Atmire code builds
- Now Maven and Ant run properly, but I’m getting SQL migration errors in
dspace.log
after starting Tomcat
- I’ve updated my ticket on Atmire’s bug tracker: https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=560
2018-06-07
- Proofing 200 IITA records on DSpace Test for Sisay: IITA_Junel_06 (10568⁄95391)
- Mispelled authorship type: CGAIR single center should be: CGIAR single centre
- I see some encoding errors in author affiliations, for example:
- Universidade de SÆo Paulo
- Institut National des Recherches Agricoles du B nin
- Centre de Coop ration Internationale en Recherche Agronomique pour le D veloppement
- Institut des Recherches Agricoles du B nin
- Institut des Savannes, C te d’ Ivoire
- Institut f r Pflanzenpathologie und Pflanzenschutz der Universit t, Germany
- Projet de Gestion des Ressources Naturelles, B nin
- Universit t Hannover
- Universit F lix Houphouet-Boigny
- I uploaded fixes for all those now, but I will continue with the rest of the data later
- Regarding the SQL migration errors, Atmire told me I need to run some migrations manually in PostgreSQL:
delete from schema_version where version = '5.6.2015.12.03.2';
update schema_version set version = '5.6.2015.12.03.2' where version = '5.5.2015.12.03.2';
update schema_version set version = '5.8.2015.12.03.3' where version = '5.5.2015.12.03.3';
- And then I need to ignore the ignored ones:
$ ~/dspace/bin/dspace database migrate ignored
- Now DSpace starts up properly!
- Gabriela from CIP got back to me about the author names we were correcting on CGSpace
- I did a quick sanity check on them and then did a test import with my
fix-metadata-value.py
script:
$ ./fix-metadata-values.py -i /tmp/2018-06-08-CIP-Authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t correct -m 3
- I will apply them on CGSpace tomorrow I think…
2018-06-09
- It’s pretty annoying, but the JVM monitoring for Munin was never set up when I migrated DSpace Test to its new server a few months ago
- I ran the tomcat and munin-node tags in Ansible again and now the stuff is all wired up and recording stats properly
- I applied the CIP author corrections on CGSpace and DSpace Test and re-ran the Discovery indexing
2018-06-10
- I spent some time removing the Atmire Metadata Quality Module (MQM) from the proposed DSpace 5.8 changes
- After removing all code mentioning MQM, mqm, metadata-quality, batchedit, duplicatechecker, etc, I think I got most of it removed, but there is a Spring error during Tomcat startup:
INFO [org.dspace.servicemanager.DSpaceServiceManager] Shutdown DSpace core service manager
Failed to startup the DSpace Service Manager: failure starting up spring service manager: Error creating bean with name 'org.dspace.servicemanager.spring.DSpaceBeanPostProcessor#0' defined in class path resource [spring/spring-dspace-applicationContext.xml]: Unsatisfied dependency expressed through constructor argument with index 0 of type [org.dspace.servicemanager.config.DSpaceConfigurationService]: : Cannot find class [com.atmire.dspace.discovery.ItemCollectionPlugin] for bean with name 'itemCollectionPlugin' defined in file [/home/aorth/dspace/config/spring/api/discovery.xml];
- I can fix this by commenting out the
ItemCollectionPlugin
line of discovery.xml
, but from looking at the git log I’m not actually sure if that is related to MQM or not
- I will have to ask Atmire
- I continued to look at Sisay’s IITA records from last week
- I normalized all DOIs to use HTTPS and “doi.org” instead of “dx.doi.org”
- I cleaned up white space in
cg.subject.iita
and dc.subject
- Even a bunch of IITA and AGROVOC subjects are missing accents, ie “FERTILIT DU SOL”
- More organization names in
dc.description.sponsorship
are incorrect (ie, missing accents) or inconsistent (ie, CGIAR centers should be spelled in English or multiple spellings of the same one, like “Rockefeller Foundation” and “Rockefeller foundation”)
- A few dozen items have abstracts with character encoding errors, ie:
- 33.7øC
- MgSO4ú7H2O
- ha??1&/sup;
- En gen6ral
- dÕpassÕ
- Also the abstracts have missing accents, ie “recherche sur le d veloppement”
- I will have to tell IITA people to redo these entirely I think…