From 910bdc3abf8f11706fcc687f5d4e494ede2951b1 Mon Sep 17 00:00:00 2001 From: Alan Orth Date: Sun, 30 Jun 2019 14:38:00 +0300 Subject: [PATCH] Regenerate public --- docs/2019-06/index.html | 376 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 376 insertions(+) create mode 100644 docs/2019-06/index.html diff --git a/docs/2019-06/index.html b/docs/2019-06/index.html new file mode 100644 index 000000000..0060127b3 --- /dev/null +++ b/docs/2019-06/index.html @@ -0,0 +1,376 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + June, 2019 | CGSpace Notes + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

June, 2019

+

by Alan Orth in + +  + +

+
+

2019-06-02

+ + + +

2019-06-03

+ +
    +
  • Skype with Marie-Angélique and Abenet about CG Core v2
    • +
+ +
    +
  • Here is a list of proposed metadata migrations for CGSpace + +
      +
    • dc.language.iso→DCTERMS.language (and switch to ISO 639-2 Alpha 3)
      • +
    • dc.description.abstract→DCTERMS.abstract
      • +
    • dc.identifier.citation→DCTERMS.bibliographicCitation
      • +
    • dc.contributor.author→DCTERMS.creator (for people)
      • +
    • dc.description.sponsorship→cg.contributor.donor (values from CrossRef or Grid.ac if possible)
      • +
    • dc.rights→DCTERMS.license
      • +
    • cg.identifier.status→DCTERMS.accessRights (values “open” or “restricted”)
      • +
    • cg.creator.id→cg.creator.identifier?
      • +
    • dc.relation.ispartofseries→DCTERMS.isPartOf
      • +
    • cg.link.relation→DCTERMS.relation
      • +
    • +
  • Marie agreed that we need to adopt some controlled lists for our values, and pointed out that the MARLO team maintains a list of CRPs and Centers at CLARISA + +
      +
    • There is an API there but it needs a password for access…
      • +
    • +
+ +

2019-06-04

+ +
    +
  • The MARLO team responded and said they will give us access to the CLARISA API
    • +
  • Marie-Angélique proposed to integrate dcterms.isPartOf, dcterms.abstract, and dcterms.bibliographicCitation into the CG Core v2 schema + +
      +
    • I told her I would attempt to integrate those and the others above into DSpace Test soon and report back
      • +
    • We also need to discuss with the ILRI Data Portal, MEL/MELSpace, and users who consume the CGSpace API
      • +
    • +
  • Add Arabic language to input-forms.xml (#427), as Bioversity is adding some Arabic items and noticed it missing
    • +
+ +

2019-06-05

+ +
    +
  • Send mail to CGSpace and MELSpace people to let them know about the proposed metadata field migrations after the discussion with Marie-Angélique
    • +
+ +

2019-06-07

+ +
    +

  • Thierry noticed that the CUA statistics were missing previous years again, and I see that the Solr admin UI has the following message:

    + +
    statistics-2018: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Error opening new searcher 
+
    • + +

  • I had to restart Tomcat a few times for all the stats cores to get loaded with no issue

    • +
+ +

2019-06-10

+ +
    +
  • Rename the AReS repository on GitHub to OpenRXV: https://github.com/ilri/OpenRXV
    • +
  • Create a new AReS repository: https://github.com/ilri/AReS
    • +
  • Start looking at the 203 IITA records on DSpace Test from last month (IITA_May_16 aka “20194th.xls”) using OpenRefine + +
      +
    • Trim leading, trailing, and consecutive whitespace on all columns, but I didn’t notice very many issues
      • +
    • Validate affiliations against latest list of top 1500 terms using reconcile-csv, correcting and standardizing about twenty-seven
      • +
    • Validate countries against latest list of countries using reconcile-csv, correcting three
      • +
    • Convert all DOIs to “https://dx.doi.org" format
      • +
    • Normalize all cg.identifier.url Google book fields to “books.google.com”
      • +
    • Correct some inconsistencies in IITA subjects
      • +
    • Correct two incorrect “Peer Review” in dc.description.version
      • +
    • About fifteen items have incorrect ISBNs (looks like an Excel error because the values look like scientific numbers)
      • +
    • Delete one blank item
      • +
    • I managed to get to subjects, so I’ll continue from there when I start working next
      • +
    • + +

  • Generate a new list of countries from the database for use with reconcile-csv

    + +
      +

    • After dumping, use csvcut to add line numbers, then change the csv header to match those you use in reconcile-csv, for example id and name:

      + +
      dspace=# \COPY (SELECT DISTINCT text_value, count(*) FROM metadatavalue WHERE metadata_field_id = 228 AND resource_type_id = 2 GROUP BY text_value ORDER BY count DESC) to /tmp/countries.csv WITH CSV HEADER
+COPY 192
+$ csvcut -l -c 0 /tmp/countries.csv > 2019-06-10-countries.csv
+
      • +
    • + +

  • Get a list of all the unique AGROVOC subject terms in IITA’s data and export it to a text file so I can validate them with my agrovoc-lookup.py script:

    + +
    $ csvcut -c dc.subject ~/Downloads/2019-06-10-IITA-20194th-Round-2.csv| sed 's/||/\n/g' | grep -v dc.subject | sort -u > iita-agrovoc.txt
+$ ./agrovoc-lookup.py -i iita-agrovoc.txt -om iita-agrovoc-matches.txt -or iita-agrovoc-rejects.txt
+$ wc -l iita-agrovoc*
+402 iita-agrovoc-matches.txt
+29 iita-agrovoc-rejects.txt
+431 iita-agrovoc.txt
+
    • + +

  • Combine these IITA matches with the subjects I matched a few months ago:

    + +
    $ csvcut -c name 2019-03-18-subjects-matched.csv | grep -v name | cat - iita-agrovoc-matches.txt | sort -u > 2019-06-10-subjects-matched.txt
+
    • + +

  • Then make a new list to use with reconcile-csv by adding line numbers with csvcut and changing the line number header to id:

    + +
    $ csvcut -c name -l 2019-06-10-subjects-matched.txt | sed 's/line_number/id/' > 2019-06-10-subjects-matched.csv
+
    • +
+ +

2019-06-20

+ +
    +
  • Share some feedback about AReS v2 with the colleagues and encourage them to do the same
    • +
+ +

2019-06-23

+ +
    +
  • Continue work on reviewing CG Core v2 standard and its implications to CGSpace an DSpace platforms in general + +
    • + +

  • Update my local PostgreSQL container:

    + +
    $ podman pull docker.io/library/postgres:9.6-alpine
+$ podman rm dspacedb
+$ podman run --name dspacedb -v dspacedb_data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:9.6-alpine
+
    • +
+ +

2019-06-25

+ +
    +

  • Normalize text_lang values for metadata on DSpace Test and CGSpace:

    + +
    dspace=# UPDATE metadatavalue SET text_lang='en_US' WHERE resource_type_id=2 AND metadata_field_id != 28 AND text_lang IN ('ethnob', 'en', '*', 'E.', '');
+UPDATE 1551
+dspace=# UPDATE metadatavalue SET text_lang='en_US' WHERE resource_type_id=2 AND metadata_field_id != 28 AND text_lang IS NULL;
+UPDATE 2070
+dspace=# UPDATE metadatavalue SET text_lang='es_ES' WHERE resource_type_id=2 AND metadata_field_id != 28 AND text_lang IN ('es', 'spa');
+UPDATE 2
+
    • + +

  • Upload 202 IITA records from earlier this month (20194th.xls) to CGSpace

    • + +

  • Communicate with Bioversity contractor in charge of their migration from Typo3 to CGSpace

    • +
+ +

2019-06-28

+ +
    +
  • Start looking at the fifty-seven AfricaRice records sent by Ibnou earlier this month + +
      +
    • First, I see there are several items with type “Book” and “Book Chapter” should go in an “AfricaRice books and book chapters” collection, but none exists in the AfricaRice community
      • +
    • Trim and collapse consecutive whitespace on author, affiliation, authorship types, title, subjects, doi, issn, source, citation, country, sponsors
      • +
    • Standardize and correct affiliations like “Africa Rice Cente” and “Africa Rice Centre”, including syntax errors with multi-value separators
      • +
    • Lots of variation in affiliations, for example:
      • +
    • Université Abomey-Calavi
      • +
    • Université d’Abomey
      • +
    • Université d’Abomey Calavi
      • +
    • Université d’Abomey-Calavi
      • +
    • University of Abomey-Calavi
      • +
    • Validate and normalize affiliations against our 2019-04 list using reconcile-csv and OpenRefine:
      • +
    • $ lein run ~/src/git/DSpace/2019-04-08-affiliations.csv name id
      • +
    • I always forget how to copy the reconciled values in OpenRefine, but you need to make a new colume and populate it using this GREL: if(cell.recon.matched, cell.recon.match.name, value)
      • +
    • Replace smart quotes with standard ASCII ones
      • +
    • Fix typos in authoriship types
      • +
    • Validate and normalize subjects against our 2019-06 list using reconcile-csv and OpenRefine:
      • +
    • $ lein run ~/src/git/DSpace/2019-06-10-subjects-matched.csv name id
      • +
    • Also add about 30 new AGROVOC subjects to our list that I verified manually
      • +
    • There is one duplicate, both have the same DOI: https://doi.org/10.1016/j.agwat.2018.06.018
      • +
    • Fix four ISBNs that were in the ISSN field
      • +
    • +
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + +