CGSpace Notes

Documenting day-to-day work on the CGSpace repository.

May, 2018

by Alan Orth in  

2018-05-01

2018-05-02

  • Advise Fabio Fidanza about integrating CGSpace content in the new CGIAR corporate website
  • I think they can mostly rely on using the cg.contributor.crp field
  • Looking over some IITA records for Sisay
    • Other than trimming and collapsing consecutive whitespace, I made some other corrections
    • I need to check the correct formatting of COTE D’IVOIRE vs COTE D’IVOIRE
    • I replaced all DOIs with HTTPS
    • I checked a few DOIs and found at least one that was missing, so I Googled the title of the paper and found the correct DOI
    • Also, I found an FAQ for DOI that says the dx.doi.org syntax is older, so I will replace all the DOIs with doi.org instead
    • I found five records with “ISI Jounal” instead of “ISI Journal”
    • I found one item with IITA subject “.”
    • Need to remember to check the facets for things like this in sponsorship:
    • Deutsche Gesellschaft für Internationale Zusammenarbeit
    • Deutsche Gesellschaft fur Internationale Zusammenarbeit
    • Eight records with language “fn” instead of “fr”
    • One incorrect type (lowercase “proceedings”): Conference proceedings
    • Found some capitalized CRPs in cg.contributor.crp
    • Found some incorrect author affiliations, ie “Institut de Recherche pour le Developpement Agricolc” should be “Institut de Recherche pour le Developpement Agricole“
    • Wow, and for sponsors there are the following:
    • Incorrect: Flemish Agency for Development Cooperation and Technical Assistance
    • Incorrect: Flemish Organization for Development Cooperation and Technical Assistance
    • Correct: Flemish Association for Development Cooperation and Technical Assistance
    • One item had region “WEST” (I corrected it to “WEST AFRICA”)

2018-05-03

  • It turns out that the IITA records that I was helping Sisay with in March were imported in 2018-04 without a final check by Abenet or I
  • There are lots of errors on language, CRP, and even some encoding errors on abstract fields
  • I export them and include the hidden metadata fields like dc.date.accessioned so I can filter the ones from 2018-04 and correct them in Open Refine:
$ dspace metadata-export -a -f /tmp/iita.csv -i 10568/68616
  • Abenet sent a list of 46 ORCID identifiers for ILRI authors so I need to get their names using my resolve-orcids.py script and merge them into our controlled vocabulary
  • On the messed up IITA records from 2018-04 I see sixty DOIs in incorrect format (cg.identifier.doi)

2018-05-06

  • Fixing the IITA records from Sisay, sixty DOIs have completely invalid format like http:dx.doi.org10.1016j.cropro.2008.07.003
  • I corrected all the DOIs and then checked them for validity with a quick bash loop:
$ for line in $(< /tmp/links.txt); do echo $line; http --print h $line; done
  • Most of the links are good, though one is duplicate and one seems to even be incorrect in the publisher’s site so…
  • Also, there are some duplicates:
    • 10568/92241 and 10568/92230 (same DOI)
    • 10568/92151 and 10568/92150 (same ISBN)
    • 10568/92291 and 10568/92286 (same citation, title, authors, year)
  • Messed up abstracts:
    • 10568/92309
  • Fixed some issues in regions, countries, sponsors, ISSN, and cleaned whitespace errors from citation, abstract, author, and titles
  • Fixed all issues with CRPs
  • A few more interesting Unicode characters to look for in text fields like author, abstracts, and citations might be: ’ (0x2019), · (0x00b7), and € (0x20ac)
  • A custom text facit in OpenRefine with this GREL expression could be a good for finding invalid characters or encoding errors in authors, abstracts, etc:
or(
  isNotNull(value.match(/.*[(|)].*/)),
  isNotNull(value.match(/.*\uFFFD.*/)),
  isNotNull(value.match(/.*\u00A0.*/)),
  isNotNull(value.match(/.*\u200A.*/)),
  isNotNull(value.match(/.*\u2019.*/)),
  isNotNull(value.match(/.*\u00b7.*/)),
  isNotNull(value.match(/.*\u20ac.*/))
)
  • I found some more IITA records that Sisay imported on 2018-03-23 that have invalid CRP names, so now I kinda want to check those ones!
  • Combine the ORCID identifiers Abenet sent with our existing list and resolve their names using the resolve-orcids.py script:
$ cat ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-id.xml /tmp/ilri-orcids.txt | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort | uniq > /tmp/2018-05-06-combined.txt
$ ./resolve-orcids.py -i /tmp/2018-05-06-combined.txt -o /tmp/2018-05-06-combined-names.txt -d
# sort names, copy to cg-creator-id.xml, add XML formatting, and then format with tidy (preserving accents)
$ tidy -xml -utf8 -iq -m -w 0 dspace/config/controlled-vocabularies/cg-creator-id.xml
  • I made a pull request (#373) for this that I’ll merge some time next week (I’m expecting Atmire to get back to us about DSpace 5.8 soon)
  • After testing quickly I just decided to merge it, and I noticed that I don’t even need to restart Tomcat for the changes to get loaded

2018-05-07

  • I spent a bit of time playing with conciliator and Solr, trying to figure out how to reconcile columns in OpenRefine with data in our existing Solr cores (like CRP subjects)
  • The documentation regarding the Solr stuff is limited, and I cannot figure out what all the fields in conciliator.properties are supposed to be
  • But then I found reconcile-csv, which allows you to reconcile against values in a CSV file!
  • That, combined with splitting our multi-value fields on “||” in OpenRefine is amaaaaazing, because after reconciliation you can just join them again
  • Oh wow, you can also facet on the individual values once you’ve split them! That’s going to be amazing for proofing CRPs, subjects, etc.

2018-05-09

  • Udana asked about the Book Chapters we had been proofing on DSpace Test in 2018-04
  • I told him that there were still some TODO items for him on that data, for example to update the dc.language.iso field for the Spanish items
  • I was trying to remember how I parsed the input-forms.xml using xmllint to extract subjects neatly
  • I could use it with reconcile-csv or to populate a Solr instance for reconciliation
  • This XPath expression gets close, but outputs all items on one line:
$ xmllint --xpath '//value-pairs[@value-pairs-name="crpsubject"]/pair/stored-value/node()' dspace/config/input-forms.xml        
Agriculture for Nutrition and HealthBig DataClimate Change, Agriculture and Food SecurityExcellence in BreedingFishForests, Trees and AgroforestryGenebanksGrain Legumes and Dryland CerealsLivestockMaizePolicies, Institutions and MarketsRiceRoots, Tubers and BananasWater, Land and EcosystemsWheatAquatic Agricultural SystemsDryland CerealsDryland SystemsGrain LegumesIntegrated Systems for the Humid TropicsLivestock and Fish
  • Maybe xmlstarlet is better:
$ xmlstarlet sel -t -v '//value-pairs[@value-pairs-name="crpsubject"]/pair/stored-value/text()' dspace/config/input-forms.xml
Agriculture for Nutrition and Health
Big Data
Climate Change, Agriculture and Food Security
Excellence in Breeding
Fish
Forests, Trees and Agroforestry
Genebanks
Grain Legumes and Dryland Cereals
Livestock
Maize
Policies, Institutions and Markets
Rice
Roots, Tubers and Bananas
Water, Land and Ecosystems
Wheat
Aquatic Agricultural Systems
Dryland Cereals
Dryland Systems
Grain Legumes
Integrated Systems for the Humid Tropics
Livestock and Fish
  • Discuss Colombian BNARS harvesting the CIAT data from CGSpace
  • They are using a system called Primo and the only options for data harvesting in that system are via FTP and OAI
  • I told them to get all CIAT records via OAI
  • Just a note to myself, I figured out how to get reconcile-csv to run from source rather than running the old pre-compiled JAR file:
$ lein run /tmp/crps.csv id
  • I tried to reconcile against a CSV of our countries but reconcile-csv crashes

2018-05-13

  • It turns out there was a space in my “country” header that was causing reconcile-csv to crash
  • After removing that it works fine!
  • Looking at Sisay’s 2,000 CIFOR records on DSpace Test (10568⁄92904)
    • Trimmed all leading / trailing white space and condensed multiple spaces into one
    • Corrected DOIs to use HTTPS and “doi.org” instead of “dx.doi.org”
    • There are eight items in cg.identifier.doi that are not DOIs)
    • Corrected cg.identifier.url links to cifor.org to use HTTPS
    • Corrected dc.language.iso from vt to vi (Vietnamese)
    • Corrected affiliations to not use acronyms
    • Reconcile countries against our countries list (removing terms like LATIN AMERICA, CENTRAL AFRICA, etc that are not countries)
    • Reconcile regions against our list of regions