mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2024-06-26 16:13:48 +02:00
158 lines
8.3 KiB
Markdown
158 lines
8.3 KiB
Markdown
---
|
|
title: "July, 2023"
|
|
date: 2023-07-01T17:14:36+03:00
|
|
author: "Alan Orth"
|
|
categories: ["Notes"]
|
|
---
|
|
|
|
## 2023-07-01
|
|
|
|
- Export CGSpace to check for missing Initiative collection mappings
|
|
- Start harvesting on AReS
|
|
|
|
## 2023-07-02
|
|
|
|
- Minor edits to the `crossref_doi_lookup.py` script while running some checks from 22,000 CGSpace DOIs
|
|
|
|
## 2023-07-03
|
|
|
|
- I analyzed the licenses declared by Crossref and found with high confidence that ~400 of ours were incorrect
|
|
- I took the more accurate ones from Crossref and updated the items on CGSpace
|
|
- I took a few hundred ISBNs as well for where we were missing them
|
|
- I also tagged ~4,700 items with missing licenses as "Copyrighted; all rights reserved" based on their Crossref license status being TDM, mostly from Elsevier, Wiley, and Springer
|
|
- Checking a dozen or so manually, I confirmed that if Crossref only has a TDM license then it's usually copyrighted (could still be open access, but we can't tell via Crossref)
|
|
- I would be curious to write a script to check the Unpaywall API for open access status...
|
|
- In the past I found that their *license* status was not very accurate, but the open access status might be more reliable
|
|
- More minor work on the DSpace 7 item views
|
|
- I learned some new Angular template syntax
|
|
- I created a custom component to show Creative Commons licenses on the simple item page
|
|
- I also decided that I don't like the Impact Area icons as a component because they don't have any visual meaning
|
|
|
|
## 2023-07-04
|
|
|
|
- Focus group meeting with CGSpace partners about DSpace 7
|
|
- I added a themed file selection component to the CGSpace theme
|
|
- It displays the bistream description instead of the file name, just like we did in DSpace 6 XMLUI
|
|
- I added a custom component to show share icons
|
|
|
|
## 2023-07-05
|
|
|
|
- I spent some time trying to update OpenRXV from Angular 9 to 10 to 11 to 12 to 13
|
|
- Most things work but there are some minor bugs it seems
|
|
- Mishell from CIP emailed me to say she was having problems approving an item on CGSpace
|
|
- Looking at PostgreSQL I saw there were a dozen or so locks that were several hours and even over one day old so I killed those processes and told her to try again
|
|
|
|
## 2023-07-06
|
|
|
|
- Types meeting
|
|
- I wrote a Python script to check Unpaywall for some information about DOIs
|
|
|
|
## 2023-07-7
|
|
|
|
- Continue exploring Unpaywall data for some of our DOIs
|
|
- In the past I've found their _licensing_ information to not be very reliable (preferring Crossref), but I think their _open access status_ is more reliable, especially when the provider is listed as being the publisher
|
|
- Even so, sometimes the version can be "acceptedVersion", which is presumably the author's version, as opposed to the "publishedVersion", which means it's available as open access on the publisher's website
|
|
- I did some quality assurance and found ~100 that were marked as Limited Access, but should have been Open Access, and fixed a handful of licenses
|
|
- Delete duplicate metadata as describe in my DSpace issue from last year: https://github.com/DSpace/DSpace/issues/8253
|
|
- Start working on some statistics on AGROVOC usage for my presenation next week
|
|
- I used the following SQL query to dump values from all subject fields and lower case them:
|
|
|
|
```console
|
|
localhost/dspacetest= ☘ \COPY (SELECT DISTINCT(lower(text_value)) AS "subject" FROM metadatavalue WHERE dspace_object_id in (SELECT dspace_object_id FROM item) AND metadata_field_id IN (187, 120, 210, 122, 215, 127, 208, 124, 128, 123, 125, 135, 203, 236, 238, 119)) to /tmp/2023-07-07-cgspace-subjects.csv WITH CSV HEADER;
|
|
COPY 26443
|
|
Time: 2564.851 ms (00:02.565)
|
|
```
|
|
|
|
- Then I extracted the subjects and looked them up against AGROVOC:
|
|
|
|
```console
|
|
$ csvcut -c subject /tmp/2023-07-07-cgspace-subjects.csv | sed '1d' > /tmp/2023-07-07-cgspace-subjects.txt
|
|
$ ./ilri/agrovoc_lookup.py -i /tmp/2023-07-07-cgspace-subjects.txt -o /tmp/2023-07-07-cgspace-subjects-results.csv
|
|
```
|
|
|
|
- I did some more tests with Angular 13 on OpenRXV and found out why the repository type dropdown wasn't working
|
|
- It was because of a missing 1-line JSON file in the data directory, which is runtime data, not code
|
|
- I copied the data directory from the production serve and rebuild and the site is working well now
|
|
- I did a full harvest with plugins and it worked!
|
|
- So it seems Angular 13.4.0 will work, yay
|
|
|
|
## 2023-07-08
|
|
|
|
- Export CGSpace to check for missing Initiative collection mappings
|
|
- Start a harvest on AReS
|
|
- The AGROVOC lookup finished, so I checked the number of matches:
|
|
|
|
```console
|
|
$ csvgrep -c 'match type' -r '^.+$' ~/Downloads/2023-07-07-cgspace-subjects-resolved.csv | sed 1d | wc -l
|
|
12528
|
|
```
|
|
|
|
- So that's 12,528 out of 26,443 unique terms (47.3%)
|
|
- I did a LOT of work on the OpenRXV frontend build dependencies to bring more in line with Angular 13
|
|
|
|
## 2023-07-10
|
|
|
|
- I did a lot more work on OpenRXV to test and update dependencies
|
|
- I deployed the latest version on the production server
|
|
|
|
## 2023-07-12
|
|
|
|
- CGSpace upgrade meeting with Americas and Africa group
|
|
|
|
## 2023-07-13
|
|
|
|
- Michael Victor asked me to help Aditi extract some information from CGSpace
|
|
- She was interested in journal articles published between 2018 and 2023 with a range of subjects related to drought, flooding, resilience, etc
|
|
- I used an advanced query with some AGROVOC terms:
|
|
|
|
```console
|
|
dcterms.issued:[2018 TO 2023] AND dcterms.type:"Journal Article" AND (dcterms.subject:flooding OR dcterms.subject:flood OR dcterms.subject:"extreme weather events" OR dcterms.subject:drought OR dcterms.subject:"drought resistance" OR dcterms.subject:"drought tolerance" OR dcterms.subject:"soil salinity" OR dcterms.subject:"pests of plants" OR dcterms.subject:pests OR dcterms.subject:heat OR dcterms.subject:fertilizers OR dcterms.subject:"fertilizer technology" OR dcterms.subject:"rice fields" OR dcterms.subject:"landscape conservation" OR dcterms.subject:"landscape restoration" OR dcterms.subject:livestock)
|
|
```
|
|
|
|
- Interestingly, some variations of this same exact query produce no search results, and I see this error in the DSpace log:
|
|
|
|
```console
|
|
org.dspace.discovery.SearchServiceException: org.apache.solr.search.SyntaxError: Cannot parse 'dcterms.issued:[2018 TO 2023] AND dcterms.type:"Journal Article" AND (dcterms.subject:flooding OR dcterms.subject:flood OR dcterms.subject:"extreme weather events" OR dcterms.subject:drought OR dcterms.subject:"drought resistance" OR dcterms.subject:"drought tolerance" OR dcterms.subject:"soil salinity" OR dcterms.subject:"pests of plants" OR dcterms.subject:pests OR dcterms.subject:heat OR dcterms.subject:fertilizers OR dcterms.subject:"fertilizer technology" OR dcterms.subject:"rice fields" OR dcterms.subject:livestock OR dcterms.subject:"landscape conservation" OR dcterms.subject:"landscape restoration\"\)': Lexical error at line 1, column 617. Encountered: <EOF> after : "\"landscape restoration\\\"\\)"
|
|
```
|
|
|
|
- It seems to be when there is a quoted search term at the end of the parenthesized group
|
|
- For what it's worth this same query worked fine on DSpace 7.6
|
|
|
|
## 2023-07-15
|
|
|
|
- Export CGSpace to fix missing Initiative collection mappings
|
|
- Start a harvest on AReS
|
|
|
|
## 2023-07-17
|
|
|
|
- Rasika had sent me a list of new ORCID identifiers for new IWMI staff so I combined them with our existing list and ran `resolve_orcids.py` to refresh the names in our database
|
|
- I updated the list, updated names in the database, and tagged new authors with missing identifiers in existing items
|
|
|
|
## 2023-07-18
|
|
|
|
- Meeting with IWMI, IRRI, and IITA colleagues about CGSpace upgrade plans
|
|
- Maria from the Alliance mentioned having some submissions stuck on CGSpace
|
|
- I looked and found a number of locks stuck for many nineteen, eighteen, and more hours...
|
|
- I killed them and told her to try again
|
|
|
|
```console
|
|
$ psql < locks-age.sql | less -S
|
|
$ psql < locks-age.sql | grep -E " (19|18|17|16|12):" | awk -F"|" '{print $10}' | sort -u | xargs kill
|
|
```
|
|
|
|
## 2023-07-19
|
|
|
|
- I had to kill a bunch more locked processes in PostgreSQL, I'm not sure what's going on
|
|
- After some discussion about an advanced search bug with Tim on Slack, I filed [an issue on GitHub](https://github.com/DSpace/DSpace/issues/8962)
|
|
|
|
## 2023-07-20
|
|
|
|
- I added a new metadata field for CGIAR Impact Platforms (`cg.subject.impactPlatform`) to CGSpace
|
|
|
|
## 2023-07-22
|
|
|
|
- Export CGSpace tp fix missing Initiative collections
|
|
- Start a harvest on AReS
|
|
|
|
<!-- vim: set sw=2 ts=2: -->
|