June, 2019
+ +2019-06-02
+ +-
+
- Merge the Solr filterCache and XMLUI ISI journal changes to the
5_x-prod
branch and deploy on CGSpace
+ - Run system updates on CGSpace (linode18) and reboot it +
2019-06-03
+ +-
+
- Skype with Marie-Angélique and Abenet about CG Core v2 +
-
+
- Here is a list of proposed metadata migrations for CGSpace
+
+
-
+
- dc.language.iso→DCTERMS.language (and switch to ISO 639-2 Alpha 3) +
- dc.description.abstract→DCTERMS.abstract +
- dc.identifier.citation→DCTERMS.bibliographicCitation +
- dc.contributor.author→DCTERMS.creator (for people) +
- dc.description.sponsorship→cg.contributor.donor (values from CrossRef or Grid.ac if possible) +
- dc.rights→DCTERMS.license +
- cg.identifier.status→DCTERMS.accessRights (values “open” or “restricted”) +
- cg.creator.id→cg.creator.identifier? +
- dc.relation.ispartofseries→DCTERMS.isPartOf +
- cg.link.relation→DCTERMS.relation +
+ - Marie agreed that we need to adopt some controlled lists for our values, and pointed out that the MARLO team maintains a list of CRPs and Centers at CLARISA
+
+
-
+
- There is an API there but it needs a password for access… +
+
2019-06-04
+ +-
+
- The MARLO team responded and said they will give us access to the CLARISA API +
- Marie-Angélique proposed to integrate
dcterms.isPartOf
,dcterms.abstract
, anddcterms.bibliographicCitation
into the CG Core v2 schema + +-
+
- I told her I would attempt to integrate those and the others above into DSpace Test soon and report back +
- We also need to discuss with the ILRI Data Portal, MEL/MELSpace, and users who consume the CGSpace API +
+ - Add Arabic language to input-forms.xml (#427), as Bioversity is adding some Arabic items and noticed it missing +
2019-06-05
+ +-
+
- Send mail to CGSpace and MELSpace people to let them know about the proposed metadata field migrations after the discussion with Marie-Angélique +
2019-06-07
+ +-
+
Thierry noticed that the CUA statistics were missing previous years again, and I see that the Solr admin UI has the following message:
+ +statistics-2018: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Error opening new searcher +
+
+I had to restart Tomcat a few times for all the stats cores to get loaded with no issue
+
2019-06-10
+ +-
+
- Rename the AReS repository on GitHub to OpenRXV: https://github.com/ilri/OpenRXV +
- Create a new AReS repository: https://github.com/ilri/AReS +
- Start looking at the 203 IITA records on DSpace Test from last month (IITA_May_16 aka “20194th.xls”) using OpenRefine
+
+
-
+
- Trim leading, trailing, and consecutive whitespace on all columns, but I didn’t notice very many issues +
- Validate affiliations against latest list of top 1500 terms using reconcile-csv, correcting and standardizing about twenty-seven +
- Validate countries against latest list of countries using reconcile-csv, correcting three +
- Convert all DOIs to “https://dx.doi.org" format +
- Normalize all
cg.identifier.url
Google book fields to “books.google.com”
+ - Correct some inconsistencies in IITA subjects +
- Correct two incorrect “Peer Review” in
dc.description.version
+ - About fifteen items have incorrect ISBNs (looks like an Excel error because the values look like scientific numbers) +
- Delete one blank item +
- I managed to get to subjects, so I’ll continue from there when I start working next +
+
+ Generate a new list of countries from the database for use with reconcile-csv
+ +-
+
After dumping, use csvcut to add line numbers, then change the csv header to match those you use in reconcile-csv, for example
+ +id
andname
:dspace=# \COPY (SELECT DISTINCT text_value, count(*) FROM metadatavalue WHERE metadata_field_id = 228 AND resource_type_id = 2 GROUP BY text_value ORDER BY count DESC) to /tmp/countries.csv WITH CSV HEADER +COPY 192 +$ csvcut -l -c 0 /tmp/countries.csv > 2019-06-10-countries.csv +
+
+
+Get a list of all the unique AGROVOC subject terms in IITA’s data and export it to a text file so I can validate them with my
+ +agrovoc-lookup.py
script:$ csvcut -c dc.subject ~/Downloads/2019-06-10-IITA-20194th-Round-2.csv| sed 's/||/\n/g' | grep -v dc.subject | sort -u > iita-agrovoc.txt +$ ./agrovoc-lookup.py -i iita-agrovoc.txt -om iita-agrovoc-matches.txt -or iita-agrovoc-rejects.txt +$ wc -l iita-agrovoc* +402 iita-agrovoc-matches.txt +29 iita-agrovoc-rejects.txt +431 iita-agrovoc.txt +
+
+Combine these IITA matches with the subjects I matched a few months ago:
+ +$ csvcut -c name 2019-03-18-subjects-matched.csv | grep -v name | cat - iita-agrovoc-matches.txt | sort -u > 2019-06-10-subjects-matched.txt +
+
+Then make a new list to use with reconcile-csv by adding line numbers with csvcut and changing the line number header to
+ +id
:$ csvcut -c name -l 2019-06-10-subjects-matched.txt | sed 's/line_number/id/' > 2019-06-10-subjects-matched.csv +
+
2019-06-20
+ +-
+
- Share some feedback about AReS v2 with the colleagues and encourage them to do the same +
2019-06-23
+ +-
+
- Continue work on reviewing CG Core v2 standard and its implications to CGSpace an DSpace platforms in general
+
+
-
+
- Update my list of fields to migrate +
- Submit an issue with my feedback to the CG Core project +
+
+ Update my local PostgreSQL container:
+ +$ podman pull docker.io/library/postgres:9.6-alpine +$ podman rm dspacedb +$ podman run --name dspacedb -v dspacedb_data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:9.6-alpine +
+
2019-06-25
+ +-
+
Normalize
+ +text_lang
values for metadata on DSpace Test and CGSpace:dspace=# UPDATE metadatavalue SET text_lang='en_US' WHERE resource_type_id=2 AND metadata_field_id != 28 AND text_lang IN ('ethnob', 'en', '*', 'E.', ''); +UPDATE 1551 +dspace=# UPDATE metadatavalue SET text_lang='en_US' WHERE resource_type_id=2 AND metadata_field_id != 28 AND text_lang IS NULL; +UPDATE 2070 +dspace=# UPDATE metadatavalue SET text_lang='es_ES' WHERE resource_type_id=2 AND metadata_field_id != 28 AND text_lang IN ('es', 'spa'); +UPDATE 2 +
+
+Upload 202 IITA records from earlier this month (20194th.xls) to CGSpace
+
+Communicate with Bioversity contractor in charge of their migration from Typo3 to CGSpace
+
2019-06-28
+ +-
+
- Start looking at the fifty-seven AfricaRice records sent by Ibnou earlier this month
+
+
-
+
- First, I see there are several items with type “Book” and “Book Chapter” should go in an “AfricaRice books and book chapters” collection, but none exists in the AfricaRice community +
- Trim and collapse consecutive whitespace on author, affiliation, authorship types, title, subjects, doi, issn, source, citation, country, sponsors +
- Standardize and correct affiliations like “Africa Rice Cente” and “Africa Rice Centre”, including syntax errors with multi-value separators +
- Lots of variation in affiliations, for example: +
- Université Abomey-Calavi +
- Université d’Abomey +
- Université d’Abomey Calavi +
- Université d’Abomey-Calavi +
- University of Abomey-Calavi +
- Validate and normalize affiliations against our 2019-04 list using reconcile-csv and OpenRefine: +
$ lein run ~/src/git/DSpace/2019-04-08-affiliations.csv name id
+- I always forget how to copy the reconciled values in OpenRefine, but you need to make a new colume and populate it using this GREL:
if(cell.recon.matched, cell.recon.match.name, value)
+ - Replace smart quotes with standard ASCII ones +
- Fix typos in authoriship types +
- Validate and normalize subjects against our 2019-06 list using reconcile-csv and OpenRefine: +
$ lein run ~/src/git/DSpace/2019-06-10-subjects-matched.csv name id
+- Also add about 30 new AGROVOC subjects to our list that I verified manually +
- There is one duplicate, both have the same DOI: https://doi.org/10.1016/j.agwat.2018.06.018 +
- Fix four ISBNs that were in the ISSN field +
+