--- title: "May, 2019" date: 2019-05-06T10:30:43+03:00 author: "Alan Orth" tags: ["Notes"] --- ## 2019-06-02 - Merge the [Solr filterCache](https://github.com/ilri/DSpace/pull/425) and [XMLUI ISI journal](https://github.com/ilri/DSpace/pull/426) changes to the `5_x-prod` branch and deploy on CGSpace - Run system updates on CGSpace (linode18) and reboot it ## 2019-06-03 - Skype with Marie-Angélique and Abenet about [CG Core v2](https://agriculturalsemantics.github.io/cg-core/cgcore.html) - Here is a list of proposed metadata migrations for CGSpace - dc.language.iso→DCTERMS.language (and switch to ISO 639-2 Alpha 3) - dc.description.abstract→DCTERMS.abstract - dc.identifier.citation→DCTERMS.bibliographicCitation - dc.contributor.author→DCTERMS.creator (for people) - dc.description.sponsorship→cg.contributor.donor (values from CrossRef or Grid.ac if possible) - dc.rights→DCTERMS.license - cg.identifier.status→DCTERMS.accessRights (values "open" or "restricted") - cg.creator.id→cg.creator.identifier? - dc.relation.ispartofseries→DCTERMS.isPartOf - cg.link.relation→DCTERMS.relation - Marie agreed that we need to adopt some controlled lists for our values, and pointed out that the MARLO team maintains a list of CRPs and Centers at [CLARISA](https://clarisa.cgiar.org/) - There is an API there but it needs a password for access... ## 2019-06-04 - The MARLO team responded and said they will give us access to the CLARISA API - Marie-Angélique [proposed](https://github.com/AgriculturalSemantics/cg-core/pull/1) to integrate `dcterms.isPartOf`, `dcterms.abstract`, and `dcterms.bibliographicCitation` into the CG Core v2 schema - I told her I would attempt to integrate those and the others above into DSpace Test soon and report back - We also need to discuss with the ILRI Data Portal, MEL/MELSpace, and users who consume the CGSpace API - Add Arabic language to input-forms.xml ([#427](https://github.com/ilri/DSpace/pull/427)), as Bioversity is adding some Arabic items and noticed it missing ## 2019-06-05 - Send mail to CGSpace and MELSpace people to let them know about the proposed metadata field migrations after the discussion with Marie-Angélique ## 2019-06-07 - Thierry noticed that the CUA statistics were missing previous years again, and I see that the Solr admin UI has the following message: ``` statistics-2018: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Error opening new searcher ``` - I had to restart Tomcat a few times for all the stats cores to get loaded with no issue ## 2019-06-10 - Rename the AReS repository on GitHub to OpenRXV: https://github.com/ilri/OpenRXV - Create a new AReS repository: https://github.com/ilri/AReS - Start looking at the 203 IITA records on DSpace Test from last month ([IITA_May_16](https://dspacetest.cgiar.org/handle/10568/102032) aka "20194th.xls") using OpenRefine - Trim leading, trailing, and consecutive whitespace on all columns, but I didn't notice very many issues - Validate affiliations against latest list of top 1500 terms using reconcile-csv, correcting and standardizing about twenty-seven - Validate countries against latest list of countries using reconcile-csv, correcting three - Convert all DOIs to "https://dx.doi.org" format - Normalize all `cg.identifier.url` Google book fields to "books.google.com" - Correct some inconsistencies in IITA subjects - Correct two incorrect "Peer Review" in `dc.description.version` - About fifteen items have incorrect ISBNs (looks like an Excel error because the values look like scientific numbers) - Delete one blank item - I managed to get to subjects, so I'll continue from there when I start working next - Generate a new list of countries from the database for use with reconcile-csv - After dumping, use csvcut to add line numbers, then change the csv header to match those you use in reconcile-csv, for example `id` and `name`: ``` dspace=# \COPY (SELECT DISTINCT text_value, count(*) FROM metadatavalue WHERE metadata_field_id = 228 AND resource_type_id = 2 GROUP BY text_value ORDER BY count DESC) to /tmp/countries.csv WITH CSV HEADER COPY 192 $ csvcut -l -c 0 /tmp/countries.csv > 2019-06-10-countries.csv ``` - Get a list of all the unique AGROVOC subject terms in IITA's data and export it to a text file so I can validate them with my `agrovoc-lookup.py` script: ``` $ csvcut -c dc.subject ~/Downloads/2019-06-10-IITA-20194th-Round-2.csv| sed 's/||/\n/g' | grep -v dc.subject | sort -u > iita-agrovoc.txt $ ./agrovoc-lookup.py -i iita-agrovoc.txt -om iita-agrovoc-matches.txt -or iita-agrovoc-rejects.txt $ wc -l iita-agrovoc* 402 iita-agrovoc-matches.txt 29 iita-agrovoc-rejects.txt 431 iita-agrovoc.txt ``` - Combine these IITA matches with the subjects I matched a few months ago: ``` $ csvcut -c name 2019-03-18-subjects-matched.csv | grep -v name | cat - iita-agrovoc-matches.txt | sort -u > 2019-06-10-subjects-matched.txt ``` - Then make a new list to use with reconcile-csv by adding line numbers with csvcut and changing the line number header to `id`: ``` $ csvcut -c name -l 2019-06-10-subjects-matched.txt | sed 's/line_number/id/' > 2019-06-10-subjects-matched.csv ``` ## 2019-06-20 - Share some feedback about AReS v2 with the colleagues and encourage them to do the same ## 2019-06-23 - Continue work on reviewing CG Core v2 standard and its implications to CGSpace an DSpace platforms in general - Update my [list of fields to migrate](https://gist.github.com/alanorth/2db39e91f48d116e00a4edffd6ba6409) - Submit an [issue with my feedback to the CG Core project](https://github.com/AgriculturalSemantics/cg-core/issues/2) - Update my local PostgreSQL container: ``` $ podman pull docker.io/library/postgres:9.6-alpine $ podman rm dspacedb $ podman run --name dspacedb -v dspacedb_data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:9.6-alpine ``` ## 2019-06-25 - Normalize `text_lang` values for metadata on DSpace Test and CGSpace: ``` dspace=# UPDATE metadatavalue SET text_lang='en_US' WHERE resource_type_id=2 AND metadata_field_id != 28 AND text_lang IN ('ethnob', 'en', '*', 'E.', ''); UPDATE 1551 dspace=# UPDATE metadatavalue SET text_lang='en_US' WHERE resource_type_id=2 AND metadata_field_id != 28 AND text_lang IS NULL; UPDATE 2070 dspace=# UPDATE metadatavalue SET text_lang='es_ES' WHERE resource_type_id=2 AND metadata_field_id != 28 AND text_lang IN ('es', 'spa'); UPDATE 2 ``` - Upload 202 IITA records from earlier this month (20194th.xls) to CGSpace