February, 2023

Wed Feb 01, 2023 in Notes

2023-02-01

Export CGSpace to cross check the DOI metadata with Crossref
- I want to try to expand my use of their data to journals, publishers, volumes, issues, etc…

First, extract a list of DOIs for use with crossref-doi-lookup.py:

$ csvcut -c 'cg.identifier.doi[en_US]' ~/Downloads/2023-02-01-cgspace.csv \
  | csvgrep -c 1 -m 'doi.org' \
  | csvgrep -c 1 -m ' ' -i \
  | csvgrep -c 1 -r '.*cifor.*' -i \
  | sed 1d > /tmp/2023-02-01-dois.txt
$ ./ilri/crossref-doi-lookup.py -e a.orth@cgiar.org -i /tmp/2023-02-01-dois.txt -o ~/Downloads/2023-01-31-crossref-results.csv -d

Then extract the ID, DOI, journal, volume, issue, publisher, etc from the CGSpace dump and rename the cg.identifier.doi[en_US] to doi so we can join on it with the Crossref results file:

$ csvcut -c 'id,cg.identifier.doi[en_US],cg.journal[en_US],cg.volume[en_US],cg.issue[en_US],dcterms.publisher[en_US],cg.number[en_US],dcterms.license[en_US]' ~/Downloads/2023-02-01-cgspace.csv \
  | csvgrep -c 'cg.identifier.doi[en_US]' -r '.*cifor.*' -i \
  | sed -e '1s/cg.identifier.doi\[en_US\]/doi/' \
    -e 's_https://doi.org/__g' \
    -e 's_https://dx.doi.org/__g' \
  > /tmp/2023-02-01-cgspace-doi-metadata.csv
$ csvjoin -c doi /tmp/2023-02-01-cgspace-doi-metadata.csv ~/Downloads/2023-02-01-crossref-results.csv > /tmp/2023-02-01-cgspace-crossref-check.csv

And import into OpenRefine for analysis and cleaning
I just noticed that Crossref also has types, so we could use that in the future too!
I got a few corrections after examining manually, but I didn’t manage to identify any patterns that I could use to do any automatic matching or cleaning

2023-02-05

Normalize text lang attributes in PostgreSQL, run a quick Discovery index, and then export CGSpace to check Initiative mappings and countries/regions
Run all system updates on CGSpace (linode18) and reboot it

2023-02-06

Peter said that a new Initiative was approved last month so we need to add it to CGSpace: Fragility, Conflict, and Migration
There is lots of discussion about the “issue date” versus “available date” with Enrico and IFPRI, after lots of feedback from the PRMS QA
- I filed an issue on CG Core to propose using dcterms.available as an optional field to indicate the online date

2023-02-07

IFPRI’s web developer Tony managed to get his Drupal harvester to have a useful user agent:

54.x.x.x - - [06/Feb/2023:10:10:32 +0100] "POST /rest/items/find-by-metadata-field?limit=%22100&offset=0 HTTP/1.1" 200 58855 "-" "IFPRI drupal POST harvester"

He also noticed that there is no pagination on POST requests to /rest/items/find-by-metadata-field, and that he needs to increase his timeout for requests that return 100+ results, ie:

curl -f -H "Content-Type: application/json" -X POST "https://dspacetest.cgiar.org/rest/items/find-by-metadata-field" -d '{"key":"cg.subject.actionArea", "value":"Systems Transformation", "language": "en_US"}'

I need to ask on the DSpace Slack about this POST pagination
Abenet and Udana noticed that the Handle server was not running
- Looking in the error.log file I see that the service is complaining about a lock file being present
- This is because Linode had to do emergency maintenance on the VM host this morning and the Handle server didn’t shut down properly
I’m having an issue with poetry update so I spent some time debugging and filed an issue
Proof and import nine items for the Digital Innovation Inititive for IFPRI
- There were only some minor issues in the metadata
- I also did a duplicate check with check-duplicates.py just in case
I did some minor updates on csv-metadata-quality
- First, to reduce warnings on non-SPDX licenses like “Copyrighted; all rights reserved” and “Other” since they are very common for us and I’m sick of seeing the warnings
- Second, to skip whitespace and newline fixes on the abstract field since so many times they are intended

2023-02-08

Make some edits to IFPRI records requested by Jawoo and Leigh
Help Alessandra upload a last minute report for SAPLING
Proof and upload twenty-seven IFPRI records to CGSpace
- It’s a good thing I did a duplicate check because I found three duplicates!
Export CGSpace to update Initiative mappings and country/region mappings
- Then start a harvest on AReS

2023-02-09

Do some minor work on the CSS on the DSpace 7 test

2023-02-10

I noticed a large number of PostgreSQL locks from dspaceWeb on CGSpace:

$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
   2033 dspaceWeb

Looking at the lock age, I see some already 1 day old, including this curious query:

select nextval ('public.registrationdata_seq')

I killed all locks that were more than a few hours old
Export CGSpace to update Initiative collection mappings
Discuss adding dcterms.available to the submission form
- I also looked in the dcterms.description field on CGSpace and found ~1,500 items where the is an indication of an online published date
- Using some facets in OpenRefine I narrowed down the ones mentioning “online” and then extracted the dates to a new column:

cells['dcterms.description[en_US]'].value.replace(/.*?(\d+{2}) ([a-zA-Z]+) (\d+{2}).*/,"$3-$2-$1")

Then to handle formats like “2022-April-26” and “2021-Nov-11” I used some replacement GRELs (note the order so we don’t replace short patterns in longer strings prematurely):

value.replace("January","01").replace("February","02").replace("March","03").replace("April","04").replace("May","05").replace("June","06").replace("July","07").replace("August","08").replace("September","09").replace("October","10").replace("November","11").replace("December","12")
value.replace("Jan","01").replace("Feb","02").replace("Mar","03").replace("Apr","04").replace("May","05").replace("Jun","06").replace("Jul","07").replace("Aug","08").replace("Sep","09").replace("Oct","10").replace("Nov","11").replace("Dec","12")

This covered about 1,300 items, then I did about 100 more messier ones with some more regex wranling
- I removed the dcterms.description[en_US] field from items where I updated the dates
Then I added dcterms.available to the submission form and the item view
- We need to announce this to the editors

2023-02-13

Export CGSpace to do some metadata quality checks
- I added CGIAR Trust Fund as a donor to some new Initiative outputs
- I moved some abstracts from the description field
- I moved some version information to the cg.edition field

2023-02-14

The PRMS team in Colombia sent some questions about countries on CGSpace
- I had to fix some, that were clearly wrong, but there is also a difference between CGSpace and MEL because we use mostly iso-codes, and MEL uses the UN M.49 list
- Then I re-ran the country code tagger from cgspace-java-helpers, forcing the update on all items in the Initiatives community
Remove Alliance research levers from cg.contributor.crp field after discussing with Daniel and Maria
- This was a mistake on TIP’s part, and there is no direct mapping between research levers and CRPs
I exported CGSpace to check Initiative collection mappings, regions, and licenses
- Peter told me that all CGIAR blog posts for the Initiatives should be CC-BY-4.0, and I see the logo at the bottom in light gray!
- I had previously missed that and removed some licenses for blog posts
- I checked cgiar.org, ifpri.org, icarda.org, iwmi.cgiar.org, irri.org, etc and corrected a handful
Start a harvest on AReS

2023-02-15

Work on rebasing my local DSpace 7 dev branches on top of the latest 7.5-SNAPSHOT
- It seems the issues I had with the dspace submission-forms-migrate tool in August, 2022 were fixed
I imported a fresh PostgreSQL snapshot from CGSpace and then removed the Atmire migrations and ran the new migrations as I originally noted in March, 2022, and is pointed out in the DSpace 7 upgrade notes
- Now I get a new error:

localhost/dspace7= ☘ DELETE FROM schema_version WHERE version IN ('5.0.2017.09.25', '6.0.2017.01.30', '6.0.2017.09.25');
localhost/dspace7= ☘ DELETE FROM schema_version WHERE description LIKE '%Atmire%' OR description LIKE '%CUA%' OR description LIKE '%cua%';
localhost/dspace7= \q
$ ./bin/dspace database migrate ignored
...

CREATE INDEX resourcepolicy_action_idx ON resourcepolicy(action_id)

        at org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor.handleException(DefaultSqlScriptExecutor.java:275)
        at org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor.executeStatement(DefaultSqlScriptExecutor.java:222)
        at org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor.execute(DefaultSqlScriptExecutor.java:126)
        at org.flywaydb.core.internal.resolver.sql.SqlMigrationExecutor.executeOnce(SqlMigrationExecutor.java:69)
        at org.flywaydb.core.internal.resolver.sql.SqlMigrationExecutor.lambda$execute$0(SqlMigrationExecutor.java:58)
        at org.flywaydb.core.internal.database.DefaultExecutionStrategy.execute(DefaultExecutionStrategy.java:27)
        at org.flywaydb.core.internal.resolver.sql.SqlMigrationExecutor.execute(SqlMigrationExecutor.java:57)
        at org.flywaydb.core.internal.command.DbMigrate.doMigrateGroup(DbMigrate.java:377)
        ... 24 more
Caused by: org.postgresql.util.PSQLException: ERROR: relation "resourcepolicy_action_idx" already exists
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2676)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2366)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356)
        at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:496)
        at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:413)
        at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:333)
        at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:319)
        at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:295)
        at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:290)
        at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:193)
        at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:193)
        at org.flywaydb.core.internal.jdbc.JdbcTemplate.executeStatement(JdbcTemplate.java:201)
        at org.flywaydb.core.internal.sqlscript.ParsedSqlStatement.execute(ParsedSqlStatement.java:95)
        at org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor.executeStatement(DefaultSqlScriptExecutor.java:210)
        ... 30 more

I dropped that index and then the migration succeeded:

localhost/dspace7= ☘ DROP INDEX resourcepolicy_action_idx;
localhost/dspace7= ☘ \q
$ ./bin/dspace database migrate ignored
Done.

I think that particular error is because I applied the indexes in this unmerged DSpace 6 patch, so I don’t need to report this as an error in DSpace 7