mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2024-06-26 16:13:48 +02:00
32 lines
1.8 KiB
Markdown
32 lines
1.8 KiB
Markdown
|
---
|
|||
|
|
title: "October, 2019"
|
|||
|
|
date: 2019-10-01T13:20:51+03:00
|
|||
|
|
author: "Alan Orth"
|
|||
|
|
tags: ["Notes"]
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 2019-10-01
|
|||
|
|
|
|||
|
|
- Udana from IWMI asked me for a CSV export of their community on CGSpace
|
|||
|
|
- I exported it, but a quick run through the `csv-metadata-quality` tool shows that there are some low-hanging fruits we can fix before I send him the data
|
|||
|
|
- I will limit the scope to the titles, regions, subregions, and river basins for now to manually fix some non-breaking spaces (U+00A0) there that would otherwise be removed by the csv-metadata-quality script's "unneccesary Unicode" fix:
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
$ csvcut -c 'id,dc.title[en_US],cg.coverage.region[en_US],cg.coverage.subregion[en_US],cg.river.basin[en_US]' ~/Downloads/10568-16814.csv > /tmp/iwmi-title-region-subregion-river.csv
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- Then I replace them in vim with `:% s/\%u00a0/ /g` because I can't figure out the correct sed syntax to do it directly from the pipe above
|
|||
|
|
- I uploaded those to CGSpace and then re-exported the metadata
|
|||
|
|
- Now that I think about it, I shouldn't be removing non-breaking spaces (U+00A0), I should be replacing them with normal spaces!
|
|||
|
|
- I modified the script so it replaces the non-breaking spaces instead of removing them
|
|||
|
|
- Then I ran the csv-metadata-quality script to do some general cleanups (though I temporarily commented out the whitespace fixes because it was too many thousands of rows):
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
$ csv-metadata-quality -i ~/Downloads/10568-16814.csv -o /tmp/iwmi.csv -x 'dc.date.issued,dc.date.issued[],dc.date.issued[en_US]' -u
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- That fixed 153 items (unnecessary Unicode, duplicates, comma–space fixes, etc)
|
|||
|
|
- Release [version 0.3.1 of the csv-metadata-quality script](https://github.com/ilri/csv-metadata-quality/releases/tag/v0.3.1) with the non-breaking spaces change
|
|||
|
|
|
|||
|
|
<!-- vim: set sw=2 ts=2: -->
|