alanorth/cgspace-notes
1
0
Fork 0
mirror of https://github.com/alanorth/cgspace-notes.git synced 2024-12-22 13:12:19 +01:00
Code Issues Projects Releases Wiki Activity

Add notes for 2020-01-16

Browse Source
This commit is contained in:
Alan Orth 2020-01-16 12:49:21 +02:00
parent 3736370c6e
commit a386dda3f7
Signed by: alanorth
GPG Key ID: 0FB860CC9C45B1B9
3 changed files with 49 additions and 9 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

18
content/posts/2020-01.md
View File

@ -112,4 +112,22 @@ COPY 1325
$ ./fix-metadata-values.py -i 2020-01-15-fix-8-ilri-subjects.csv -db dspace -u dspace -p 'fuuu' -f cg.subject.ilri -m 203 -t correct -d
```
## 2020-01-16
- Extract a list of CIAT subjects from CGSpace for Elizabeth Arnaud from Bioversity:
```
dspace=# \COPY (SELECT DISTINCT text_value as "cg.subject.ciat", count(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 122 GROUP BY text_value ORDER BY count DESC) to /tmp/2020-01-16-ciat-subjects.csv WITH CSV HEADER;
COPY 35
```
- Start examining the 175 IITA records that Bosede originally sent in October, 2019 (201907.xls)
- We had delayed processing them because DSpace Test (linode19) was testing CG Core v2 implementation for the last few months
- Sisay uploaded the records to DSpace Test as [IITA_201907_Jan13](https://dspacetest.cgiar.org/handle/10568/106567)
- I started first with basic sanity checks using my csv-metadata-quality tool and found twenty-two items with extra whitespace, invalid multi-value separators, and duplicates, which means Sisay did not do any quality checking on the data
- I corrected one invalid AGROVOC subject
- Validate and normalize affiliations against our 2019-04 list using reconcile-csv and OpenRefine:
- `$ lein run ~/src/git/DSpace/2019-04-08-affiliations.csv name id`
- I always forget how to copy the reconciled values in OpenRefine, but you need to make a new colum and populate it using this GREL: `if(cell.recon.matched, cell.recon.match.name, value)`
<!-- vim: set sw=2 ts=2: -->

30
docs/2020-01/index.html
View File

@ -29,7 +29,7 @@ I tweeted the CGSpace repository link
<meta property="og:type" content="article" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2020-01/" />
<meta property="article:published_time" content="2020-01-06T10:48:30+02:00" />
<meta property="article:modified_time" content="2020-01-14T20:40:41+02:00" />
<meta property="article:modified_time" content="2020-01-15T13:51:35+02:00" />
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="January, 2020"/>
@ -63,9 +63,9 @@ I tweeted the CGSpace repository link
"@type": "BlogPosting",
"headline": "January, 2020",
"url": "https:\/\/alanorth.github.io\/cgspace-notes\/2020-01\/",
"wordCount": "765",
"wordCount": "941",
"datePublished": "2020-01-06T10:48:30+02:00",
"dateModified": "2020-01-14T20:40:41+02:00",
"dateModified": "2020-01-15T13:51:35+02:00",
"author": {
"@type": "Person",
"name": "Alan Orth"
@ -236,7 +236,29 @@ COPY 1325
<li>I noticed a few errors in the ILRI subjects so I fixed them locally and on CGSpace (linode18) using my <code>fix-metadata.py</code> script:</li>
</ul>
<pre><code>$ ./fix-metadata-values.py -i 2020-01-15-fix-8-ilri-subjects.csv -db dspace -u dspace -p 'fuuu' -f cg.subject.ilri -m 203 -t correct -d
</code></pre><!-- raw HTML omitted -->
</code></pre><h2 id="2020-01-16">2020-01-16</h2>
<ul>
<li>Extract a list of CIAT subjects from CGSpace for Elizabeth Arnaud from Bioversity:</li>
</ul>
<pre><code>dspace=# \COPY (SELECT DISTINCT text_value as &quot;cg.subject.ciat&quot;, count(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 122 GROUP BY text_value ORDER BY count DESC) to /tmp/2020-01-16-ciat-subjects.csv WITH CSV HEADER;
COPY 35
</code></pre><ul>
<li>Start examining the 175 IITA records that Bosede originally sent in October, 2019 (201907.xls)
<ul>
<li>We had delayed processing them because DSpace Test (linode19) was testing CG Core v2 implementation for the last few months</li>
<li>Sisay uploaded the records to DSpace Test as <a href="https://dspacetest.cgiar.org/handle/10568/106567">IITA_201907_Jan13</a></li>
<li>I started first with basic sanity checks using my csv-metadata-quality tool and found twenty-two items with extra whitespace, invalid multi-value separators, and duplicates, which means Sisay did not do any quality checking on the data</li>
<li>I corrected one invalid AGROVOC subject</li>
<li>Validate and normalize affiliations against our 2019-04 list using reconcile-csv and OpenRefine:
<ul>
<li><code>$ lein run ~/src/git/DSpace/2019-04-08-affiliations.csv name id</code></li>
<li>I always forget how to copy the reconciled values in OpenRefine, but you need to make a new colum and populate it using this GREL: <code>if(cell.recon.matched, cell.recon.match.name, value)</code></li>
</ul>
</li>
</ul>
</li>
</ul>
<!-- raw HTML omitted -->

10
docs/sitemap.xml
View File

@ -4,27 +4,27 @@
<url>
<loc>https://alanorth.github.io/cgspace-notes/categories/</loc>
<lastmod>2020-01-14T20:40:41+02:00</lastmod>
<lastmod>2020-01-15T13:51:35+02:00</lastmod>
</url>
<url>
<loc>https://alanorth.github.io/cgspace-notes/</loc>
<lastmod>2020-01-14T20:40:41+02:00</lastmod>
<lastmod>2020-01-15T13:51:35+02:00</lastmod>
</url>
<url>
<loc>https://alanorth.github.io/cgspace-notes/2020-01/</loc>
<lastmod>2020-01-14T20:40:41+02:00</lastmod>
<lastmod>2020-01-15T13:51:35+02:00</lastmod>
</url>
<url>
<loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
<lastmod>2020-01-14T20:40:41+02:00</lastmod>
<lastmod>2020-01-15T13:51:35+02:00</lastmod>
</url>
<url>
<loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
<lastmod>2020-01-14T20:40:41+02:00</lastmod>
<lastmod>2020-01-15T13:51:35+02:00</lastmod>
</url>
<url>