mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2025-01-27 05:49:12 +01:00
Add notes for 2020-06-30
This commit is contained in:
@ -19,7 +19,7 @@ I tried to build the OAI registry on the freshly migrated DSpace 6 on DSpace Tes
|
||||
<meta property="og:type" content="article" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2020-06/" />
|
||||
<meta property="article:published_time" content="2020-06-01T13:55:39+03:00" />
|
||||
<meta property="article:modified_time" content="2020-06-23T16:13:27+03:00" />
|
||||
<meta property="article:modified_time" content="2020-06-28T18:13:44+03:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="June, 2020"/>
|
||||
@ -33,7 +33,7 @@ I sent Atmire the dspace.log from today and told them to log into the server to
|
||||
In other news, I checked the statistics API on DSpace 6 and it’s working
|
||||
I tried to build the OAI registry on the freshly migrated DSpace 6 on DSpace Test and I get an error:
|
||||
"/>
|
||||
<meta name="generator" content="Hugo 0.72.0" />
|
||||
<meta name="generator" content="Hugo 0.73.0" />
|
||||
|
||||
|
||||
|
||||
@ -43,9 +43,9 @@ I tried to build the OAI registry on the freshly migrated DSpace 6 on DSpace Tes
|
||||
"@type": "BlogPosting",
|
||||
"headline": "June, 2020",
|
||||
"url": "https://alanorth.github.io/cgspace-notes/2020-06/",
|
||||
"wordCount": "3319",
|
||||
"wordCount": "4764",
|
||||
"datePublished": "2020-06-01T13:55:39+03:00",
|
||||
"dateModified": "2020-06-23T16:13:27+03:00",
|
||||
"dateModified": "2020-06-28T18:13:44+03:00",
|
||||
"author": {
|
||||
"@type": "Person",
|
||||
"name": "Alan Orth"
|
||||
@ -594,6 +594,191 @@ COPY 3917
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<h2 id="2020-06-28">2020-06-28</h2>
|
||||
<ul>
|
||||
<li>Email GRID.ac to ask them about where old names for institutes are stores, as I see them in the “Disambiguate” search function online, but not in the standalone data
|
||||
<ul>
|
||||
<li>For example, both “International Laboratory for Research on Animal Diseases” (ILRAD) and “International Livestock Centre for Africa” (ILCA) correctly return a hit for “International Livestock Research Institute”, but it’s nowhere in the data</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>I discovered two interesting OpenRefine reconciliation services:
|
||||
<ul>
|
||||
<li><a href="https://github.com/ror-community/ror-reconciler">OpenRefine reconciler for the Research Organization Registry</a></li>
|
||||
<li><a href="https://www.getty.edu/research/tools/vocabularies/obtain/openrefine.html">Getty Vocabularies OpenRefine Reconciliation</a> (see the Getty Thesaurus of Geographic Names ® (TGN))</li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<h2 id="2020-06-29">2020-06-29</h2>
|
||||
<ul>
|
||||
<li>I stumbled upon a sort of <a href="https://rightsstatements.org/page/1.0/">standard for rights statements</a> that we might want to use for <code>dc.rights</code> eventually</li>
|
||||
<li>I’m trying to understand the difference between <code>dcterms.coverage</code>, <code>dcterms.spatial</code>, and <code>dcterms.temporal</code>
|
||||
<ul>
|
||||
<li>According to the <a href="https://www.dublincore.org/specifications/dublin-core/dcmi-terms/terms/coverage/">Dublin Core specification for coverage</a> the more specific spatial and temporal subproperties:</li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<blockquote>
|
||||
<p>Because coverage is so broadly defined, it is preferable to use the more specific subproperties Temporal Coverage and Spatial Coverage.</p>
|
||||
</blockquote>
|
||||
<ul>
|
||||
<li>So I guess we should be using this for countries… but then all regions, countries, etc get merged together into this when you use DCTERMS
|
||||
<ul>
|
||||
<li>Perhaps better to use <code>cg.coverage.country</code> and crosswalk to <code>dcterms.spatial</code></li>
|
||||
<li>Another thing is that these values are not literals—you are supposed to embed classes…</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>I also notice that there is a <a href="https://www.crossref.org/services/funder-registry/">CrossRef funders registry</a> with 23,000+ funders that you can <a href="https://gitlab.com/crossref/open_funder_registry">download as RDF</a> or <a href="https://www.crossref.org/education/funder-registry/accessing-the-funder-registry/">access via an API</a></li>
|
||||
</ul>
|
||||
<pre><code>$ http 'https://api.crossref.org/funders?query=Bill+and+Melinda+Gates&mailto=a.orth@cgiar.org'
|
||||
</code></pre><ul>
|
||||
<li>Searching for “Bill and Melinda Gates” we can see the <code>name</code> literal and a list of <code>alt-names</code> literals
|
||||
<ul>
|
||||
<li>This could be good for checking our funders</li>
|
||||
<li>The API currently returns pages for each funder in the vocabulary, but they are giving HTTP 404 right now: <a href="https://data.crossref.org/fundingdata/vocabulary/Label-599174">https://data.crossref.org/fundingdata/vocabulary/Label-599174</a></li>
|
||||
<li>I sent an email to the CrossRef Funders Registry team</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>See the <a href="https://github.com/CrossRef/rest-api-doc">CrossRef API docs</a> (specifically the parameters and filters)</li>
|
||||
<li>I made a pull request on CG Core v2 to recommend using persistent identifiers for DOIs and ORCID iDs (<a href="https://github.com/AgriculturalSemantics/cg-core/pull/26">#26</a>)</li>
|
||||
<li>I exported sponsors/funders from CGSpace and wrote a script to query the CrossRef API for matches:</li>
|
||||
</ul>
|
||||
<pre><code>dspace=# \COPY (SELECT DISTINCT text_value FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=29) TO /tmp/2020-06-29-sponsors.csv;
|
||||
COPY 682
|
||||
</code></pre><ul>
|
||||
<li>The script is <code>crossref-funders-lookup.py</code> and it is based on <code>agrovoc-lookup.py</code>
|
||||
<ul>
|
||||
<li>On that note, I realized I need to URL encode the funder before making the search request with requests because, while the requests library <em>does</em> do URL encoding, it seems that it interprets characters like <code>&</code> as indicative query parameters and this causes searches for funders like <code>Bill & Melinda Gates Foundation</code> to get misinterpreted</li>
|
||||
<li>So then I noticed that I had worked around this in <code>agrovoc-lookup.py</code> a few years ago by just ignoring subjects with special characters like apostrophes and accents!</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>I tested the script on our funders:</li>
|
||||
</ul>
|
||||
<pre><code>$ ./crossref-funders-lookup.py -i /tmp/2020-06-29-sponsors.csv -om /tmp/sponsors-matched.txt -or /tmp/sponsors-rejected.txt -d -e blah@blah.com
|
||||
$ wc -l /tmp/2020-06-29-sponsors.csv
|
||||
682 /tmp/2020-06-29-sponsors.csv
|
||||
$ wc -l /tmp/sponsors-*
|
||||
180 /tmp/sponsors-matched.txt
|
||||
502 /tmp/sponsors-rejected.txt
|
||||
682 total
|
||||
</code></pre><ul>
|
||||
<li>It seems that 35% of our funders already match… I bet a few more will match if I check for simple errors
|
||||
<ul>
|
||||
<li>Interesting, I found a few funders that we have correct, but can’t figure out how to match them in the API:
|
||||
<ul>
|
||||
<li><code>Claussen-Simon-Stiftung</code></li>
|
||||
<li><code>H2020 Marie Skłodowska-Curie Actions</code></li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<h2 id="2020-06-30">2020-06-30</h2>
|
||||
<ul>
|
||||
<li>GRID responded to my question about historical names
|
||||
<ul>
|
||||
<li>They said the information is not part of the public GRID or ROR lists, but you can access it with a license to the Dimensions API</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>Gabriela from CIP sent me a list of erroneously added CIP subjects to remove from CGSpace:</li>
|
||||
</ul>
|
||||
<pre><code>$ cat /tmp/2020-06-30-remove-cip-subjects.csv
|
||||
cg.subject.cip
|
||||
INTEGRATED PEST MANAGEMENT
|
||||
ORANGE FLESH SWEET POTATOES
|
||||
AEROPONICS
|
||||
FOOD SUPPLY
|
||||
SASHA
|
||||
SPHI
|
||||
INSECT LIFE CYCLE MODELLING
|
||||
SUSTAIN
|
||||
AGRICULTURAL INNOVATIONS
|
||||
NATIVE VARIETIES
|
||||
PHYTOPHTHORA INFESTANS
|
||||
$ ./delete-metadata-values.py -i /tmp/2020-06-30-remove-cip-subjects.csv -db dspace -u dspace -p 'fuuu' -f cg.subject.cip -m 127 -d
|
||||
</code></pre><ul>
|
||||
<li>She also wants to change their <code>SWEET POTATOES</code> term to <code>SWEETPOTATOES</code>, both in the CIP subject list and existing items so I updated those too:</li>
|
||||
</ul>
|
||||
<pre><code>$ cat /tmp/2020-06-30-fix-cip-subjects.csv
|
||||
cg.subject.cip,correct
|
||||
SWEET POTATOES,SWEETPOTATOES
|
||||
$ ./fix-metadata-values.py -i /tmp/2020-06-30-fix-cip-subjects.csv -db dspace -u dspace -p 'fuuu' -f cg.subject.cip -t correct -m 127 -d
|
||||
</code></pre><ul>
|
||||
<li>She also finished doing all the corrections to authors that I had sent her last week, but many of the changes are removing Spanish accents from authors names so I asked if she’s really should she wants to do that</li>
|
||||
<li>I ran the fixes and deletes on CGSpace, but not on DSpace Test yet because those scripts need updating for DSpace 6 UUIDs</li>
|
||||
<li>I spent about two hours manually checking our sponsors that were rejected from CrossRef and found about fifty-five corrections that I ran on CGSpace:</li>
|
||||
</ul>
|
||||
<pre><code>$ cat 2020-06-29-fix-sponsors.csv
|
||||
dc.description.sponsorship,correct
|
||||
"Conselho Nacional de Desenvolvimento Científico e Tecnológico, Brazil","Conselho Nacional de Desenvolvimento Científico e Tecnológico"
|
||||
"Claussen Simon Stiftung","Claussen-Simon-Stiftung"
|
||||
"Fonds pour la formation á la Recherche dans l'Industrie et dans l'Agriculture, Belgium","Fonds pour la Formation à la Recherche dans l’Industrie et dans l’Agriculture"
|
||||
"Fundação de Amparo à Pesquisa do Estado de São Paulo, Brazil","Fundação de Amparo à Pesquisa do Estado de São Paulo"
|
||||
"Schlumberger Foundation Faculty for the Future","Schlumberger Foundation"
|
||||
"Wildlife Conservation Society, United States","Wildlife Conservation Society"
|
||||
"Portuguese Foundation for Science and Technology","Portuguese Science and Technology Foundation"
|
||||
"Wageningen University and Research","Wageningen University and Research Centre"
|
||||
"Leverhulme Centre for Integrative Research in Agriculture and Health","Leverhulme Centre for Integrative Research on Agriculture and Health"
|
||||
"Natural Science and Engineering Research Council of Canada","Natural Sciences and Engineering Research Council of Canada"
|
||||
"Biotechnology and Biological Sciences Research Council, United Kingdom","Biotechnology and Biological Sciences Research Council"
|
||||
"Home Grown Ceraels Authority United Kingdom","Home-Grown Cereals Authority"
|
||||
"Fiat Panis Foundation","Foundation fiat panis"
|
||||
"Defence Science and Technology Laboratory, United Kingdom","Defence Science and Technology Laboratory"
|
||||
"African Development Bank","African Development Bank Group"
|
||||
"Ministry of Health, Labour, and Welfare, Japan","Ministry of Health, Labour and Welfare"
|
||||
"World Academy of Sciences","The World Academy of Sciences"
|
||||
"Agricultural Research Council, South Africa","Agricultural Research Council"
|
||||
"Department of Homeland Security, USA","U.S. Department of Homeland Security"
|
||||
"Quadram Institute","Quadram Institute Bioscience"
|
||||
"Google.org","Google"
|
||||
"Department for Environment, Food and Rural Affairs, United Kingdom","Department for Environment, Food and Rural Affairs, UK Government"
|
||||
"National Commission for Science, Technology and Innovation, Kenya","National Commission for Science, Technology and Innovation"
|
||||
"Hainan Province Natural Science Foundation of China","Natural Science Foundation of Hainan Province"
|
||||
"German Society for International Cooperation (GIZ)","GIZ"
|
||||
"German Federal Ministry of Food and Agriculture","Federal Ministry of Food and Agriculture"
|
||||
"State Key Laboratory of Environmental Geochemistry, China","State Key Laboratory of Environmental Geochemistry"
|
||||
"QUT student scholarship","Queensland University of Technology"
|
||||
"Australia Centre for International Agricultural Research","Australian Centre for International Agricultural Research"
|
||||
"Belgian Science Policy","Belgian Federal Science Policy Office"
|
||||
"U.S. Department of Agriculture USDA","U.S. Department of Agriculture"
|
||||
"U.S.. Department of Agriculture (USDA)","U.S. Department of Agriculture"
|
||||
"Fundação de Amparo à Pesquisa do Estado de São Paulo ( FAPESP)","Fundação de Amparo à Pesquisa do Estado de São Paulo"
|
||||
"Fundação de Amparo à Pesquisa do Estado do Rio Grande do Sul, Brazil","Fundação de Amparo à Pesquisa do Estado do Rio Grande do Sul"
|
||||
"Fundação de Amparo à Pesquisa do Estado do Rio de Janeiro, Brazil","Fundação Carlos Chagas Filho de Amparo à Pesquisa do Estado do Rio de Janeiro"
|
||||
"Swedish University of Agricultural Sciences (SLU)","Swedish University of Agricultural Sciences"
|
||||
"U.S. Department of Agriculture (USDA)","U.S. Department of Agriculture"
|
||||
"Swedish International Development Cooperation Agency (Sida)","Sida"
|
||||
"Swedish International Development Agency","Sida"
|
||||
"Federal Ministry for Economic Cooperation and Development, Germany","Federal Ministry for Economic Cooperation and Development"
|
||||
"Natural Environment Research Council, United Kingdom","Natural Environment Research Council"
|
||||
"Economic and Social Research Council, United Kingdom","Economic and Social Research Council"
|
||||
"Medical Research Council, United Kingdom","Medical Research Council"
|
||||
"Federal Ministry for Education and Research, Germany","Federal Ministry for Education, Science, Research and Technology"
|
||||
"UK Government’s Department for International Development","Department for International Development, UK Government"
|
||||
"Department for International Development, United Kingdom","Department for International Development, UK Government"
|
||||
"United Nations Children's Fund","United Nations Children's Emergency Fund"
|
||||
"Swedish Research Council for Environment, Agricultural Science and Spatial Planning","Swedish Research Council for Environment, Agricultural Sciences and Spatial Planning"
|
||||
"Agence Nationale de la Recherche, France","French National Research Agency"
|
||||
"Fondation pour la recherche sur la biodiversité","Foundation for Research on Biodiversity"
|
||||
"Programa Nacional de Innovacion Agraria, Peru","Programa Nacional de Innovación Agraria, Peru"
|
||||
"United States Agency for International Development (USAID)","United States Agency for International Development"
|
||||
"West Africa Agricultural Productivity Programme","West Africa Agricultural Productivity Program"
|
||||
"West African Agricultural Productivity Project","West Africa Agricultural Productivity Program"
|
||||
"Rural Development Administration, Republic of Korea","Rural Development Administration"
|
||||
"UK’s Biotechnology and Biological Sciences Research Council (BBSRC)","Biotechnology and Biological Sciences Research Council"
|
||||
$ ./fix-metadata-values.py -i /tmp/2020-06-29-fix-sponsors.csv -db dspace -u dspace -p 'fuuu' -f dc.description.sponsorship -t correct -m 29
|
||||
</code></pre><ul>
|
||||
<li>Peter wants me to add “CORONAVIRUS DISEASE” to all ILRI items that have ILRI subject “COVID19”
|
||||
<ul>
|
||||
<li>I exported the ILRI community and cut the columns I needed, then opened the file in OpenRefine:</li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<pre><code>$ export JAVA_OPTS="-Xmx512m -Dfile.encoding=UTF-8"
|
||||
$ dspace metadata-export -i 10568/1 -f /tmp/ilri.cs
|
||||
$ csvcut -c 'id,cg.subject.ilri[],cg.subject.ilri[en_US],dc.subject[en_US]' /tmp/ilri.csv > /tmp/ilri-covid19.csv
|
||||
</code></pre><ul>
|
||||
<li>I see that all items with “COVID19” already have “CORONAVIRUS DISEASE” so I don’t need to do anything</li>
|
||||
</ul>
|
||||
<!-- raw HTML omitted -->
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user