cgspace-notes/2018-03.md at fca6df69e6f74b2579c17da7903e9830e53f4889

alanorth/cgspace-notes

Fork 0

mirror of https://github.com/alanorth/cgspace-notes.git synced 2024-11-18 12:47:04 +01:00

Alan Orth fca6df69e6

Update notes for 2018-03-16

2018-03-16 18:44:32 +02:00

10 KiB

Raw Blame History

title

date

author

2018-03-02

Export a CSV of the IITA community metadata for Martin Mueller

2018-03-06

Add three new CCAFS project tags to input-forms.xml (#357)
Andrea from Macaroni Bros had sent me an email that CCAFS needs them
Give Udana more feedback on his WLE records from last month
There were some records using a non-breaking space in their AGROVOC subject field
I checked and tested some author corrections from Peter from last week, and then applied them on CGSpace

$ ./fix-metadata-values.py -i Correct-309-authors-2018-03-06.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t correct -m 3      
$ ./delete-metadata-values.py -i Delete-3-Authors-2018-03-06.csv -db dspace -u dspace-p 'fuuu' -f dc.contributor.author -m 3

This time there were no errors in whitespace but I did have to correct one incorrectly encoded accent character
Add new CRP subject "GRAIN LEGUMES AND DRYLAND CEREALS" to input-forms.xml (#358)
Merge the ORCID integration stuff in to 5_x-prod for deployment on CGSpace soon (#359)
Deploy ORCID changes on CGSpace (linode18), run all system updates, and reboot the server
Run all system updates on DSpace Test and reboot server
I ran the orcid-authority-to-item.py script on CGSpace and mapped 2,864 ORCID identifiers from Solr to item metadata

$ ./orcid-authority-to-item.py -db dspace -u dspace -p 'fuuu' -s http://localhost:8081/solr -d

I ran the DSpace cleanup script on CGSpace and it threw an error (as always):

Error: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
  Detail: Key (bitstream_id)=(150659) is still referenced from table "bundle".

The solution is, as always:

$ psql dspace -c 'update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (150659);'
UPDATE 1

Apply the proposed PostgreSQL indexes from DS-3636 (pull request #1791 on CGSpace (linode18)

2018-03-07

Add CIAT author Mauricio Efren Sotelo Cabrera to controlled vocabulary for ORCID identifiers (#360)
Help Sisay proof 200 IITA records on DSpace Test
Finally import Udana's 24 items to IWMI Journal Articles on CGSpace
Skype with James Stapleton to discuss CGSpace, ILRI website, CKM staff issues, etc

2018-03-08

Looking at a CSV dump of the CIAT community I see there are tons of stupid text languages people add for their metadata
This makes the CSV have tons of columns, for example dc.title, dc.title[], dc.title[en], dc.title[eng], dc.title[en_US] and so on!
I think I can fix — or at least normalize — them in the database:

dspace=# select distinct text_lang from metadatavalue where resource_type_id=2;
 text_lang 
-----------
 
 ethnob
 en
 spa
 EN
 En
 en_
 en_US
 E.
 
 EN_US
 en_U
 eng
 fr
 es_ES
 es
(16 rows)

dspace=# update metadatavalue set text_lang='en_US' where resource_type_id=2 and text_lang in ('en','EN','En','en_','EN_US','en_U','eng');
UPDATE 122227
dspacetest=# select distinct text_lang from metadatavalue where resource_type_id=2;
 text_lang
-----------

 ethnob
 en_US
 spa
 E.

 fr
 es_ES
 es
(9 rows)

On second inspection it looks like dc.description.provenance fields use the text_lang "en" so that's probably why there are over 100,000 fields changed...
If I skip that, there are about 2,000, which seems more reasonably like the amount of fields users have edited manually, or fucked up during CSV import, etc:

dspace=# update metadatavalue set text_lang='en_US' where resource_type_id=2 and text_lang in ('EN','En','en_','EN_US','en_U','eng');
UPDATE 2309

I will apply this on CGSpace right now
In other news, I was playing with adding ORCID identifiers to a dump of CIAT's community via CSV in OpenRefine
Using a series of filters, flags, and GREL expressions to isolate items for a certain author, I figured out how to add ORCID identifiers to the cg.creator.id field
For example, a GREL expression in a custom text facet to get all items with dc.contributor.author[en_US] of a certain author with several name variations (this is how you use a logical OR in OpenRefine):

or(value.contains('Ceballos, Hern'), value.contains('Hernández Ceballos'))

Then you can flag or star matching items and then use a conditional to either set the value directly or add it to an existing value:

if(isBlank(value), "Hernan Ceballos: 0000-0002-8744-7918", value + "||Hernan Ceballos: 0000-0002-8744-7918")

One thing that bothers me is that this won't honor author order
It might be better to do batches of these in PostgreSQL with a script that takes the place column of an author into account when setting the cg.creator.id
I wrote a Python script to read the author names and ORCID identifiers from CSV and create matching cg.creator.id fieldsa: add-orcid-identifiers-csv.py
The CSV should have two columns: author name and ORCID identifier:

dc.contributor.author,cg.creator.id
"Orth, Alan",Alan S. Orth: 0000-0002-1735-7458
"Orth, A.",Alan S. Orth: 0000-0002-1735-7458

I didn't integrate the ORCID API lookup for author names in this script for now because I was only interested in "tagging" old items for a few given authors
I added ORCID identifers for 187 items by CIAT's Hernan Ceballos, because that is what Elizabeth was trying to do manually!
Also, I decided to add ORCID identifiers for all records from Peter, Abenet, and Sisay as well

2018-03-09

Give James Stapleton input on Sisay's KRAs
Create a pull request to disable ORCID authority integration for dc.contributor.author in the submission forms and XMLUI display (#363)

2018-03-11

Peter also wrote to say he is having issues with the Atmire Listings and Reports module
When I logged in to try it I get a blank white page after continuing and I see this in dspace.log.2018-03-11:

2018-03-11 11:38:15,592 WARN  org.dspace.app.webui.servlet.InternalErrorServlet @ :session_id=91C2C0C59669B33A7683570F6010603A:internal_error:-- URL Was: https://cgspace.cgiar.or
g/jspui/listings-and-reports
-- Method: POST
-- Parameters were:
-- selected_admin_preset: "ilri authors2"
-- load: "normal"
-- next: "NEXT STEP >>"
-- step: "1"

org.apache.jasper.JasperException: java.lang.NullPointerException

Looks like I needed to remove the Humidtropics subject from Listings and Reports because it was looking for the terms and couldn't find them
I made a quick fix and it's working now (#364)

2018-03-12

Increase upload size on CGSpace's nginx config to 85MB so Sisay can upload some data

2018-03-13

I created a new Linode server for DSpace Test (linode6623840) so I could try the block storage stuff, but when I went to add a 300GB volume it said that block storage capacity was exceeded in that datacenter (Newark, NJ)
I deleted the Linode and created another one (linode6624164) in the Fremont, CA region
After that I deployed the Ubuntu 16.04 image and attached a 300GB block storage volume to the image
Magdalena wrote to ask why there was no Altmetric donut for an item on CGSpace, but there was one on the related CCAFS publication page
It looks the the CCAFS publications page fetches the donut using its DOI, whereas CGSpace queries via Handle
I will write to Altmetric support and ask them, as perhaps its part of a larger issue
CGSpace item: https://cgspace.cgiar.org/handle/10568/89643
CCAFS publication page: https://ccafs.cgiar.org/publications/can-scenario-planning-catalyse-transformational-change-evaluating-climate-change-policy
Peter tweeted the Handle link and now Altmetric shows the donut for both the DOI and the Handle

2018-03-14

Help Abenet with a troublesome Listings and Report question for CIAT author Steve Beebe
Continue migrating DSpace Test to the new server (linode6624164)
I emailed ILRI service desk to update the DNS records for dspacetest.cgiar.org
Abenet was having problems saving Listings and Reports configurations or layouts but I tested it and it works

2018-03-15

Help Abenet troubleshoot the Listings and Reports issue again
It looks like it's an issue with the layouts, if you create a new layout that only has one type (dc.identifier.citation):

The error in the DSpace log is:

org.apache.jasper.JasperException: java.lang.ArrayIndexOutOfBoundsException: -1

The full error is here: https://gist.github.com/alanorth/ea47c092725960e39610db9b0c13f6ca
If I do a report for "Orth, Alan" with the same custom layout it works!
I submitted a ticket to Atmire: https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=589
Small fix to the example citation text in Listings and Reports (#365)

2018-03-16

ICT made the DNS updates for dspacetest.cgiar.org late last night
I have removed the old server (linode02 aka linode578611) in favor of linode19 aka linode6624164
Looking at the CRP subjects on CGSpace I see there is one blank one so I'll just fix it:

dspace=# delete from metadatavalue where resource_type_id=2 and metadata_field_id=230 and text_value='';

Copy all CRP subjects to a CSV to do the mass updates:

dspace=# \copy (select distinct text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=230 group by text_value order by count desc) to /tmp/crps.csv with csv header;
COPY 21

Once I prepare the new input forms (#362) I will need to do the batch corrections:

$ ./fix-metadata-values.py -i Correct-21-CRPs-2018-03-16.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.crp -t correct -m 230 -n -d

10 KiB Raw Blame History Unescape Escape

2018-03-02

2018-03-06

2018-03-07

2018-03-08

2018-03-09

2018-03-11

2018-03-12

2018-03-13

2018-03-14

2018-03-15

2018-03-16

10 KiB

Raw Blame History