Alan Orth
8047a57cc5
These are things like non-breaking spaces, "replacement" characters, etc that add nothing to the metadata and often cause errors during parsing or displaying in a UI. |
||
|---|---|---|
| csv_metadata_quality | ||
| data | ||
| tests | ||
| .build.yml | ||
| .flake8 | ||
| .travis.yml | ||
| LICENSE.txt | ||
| Pipfile | ||
| Pipfile.lock | ||
| README.md | ||
| requirements-dev.txt | ||
| requirements.txt | ||
CSV Metadata Quality 
A simple but opinionated metadata quality checker and fixer designed to work with CSVs in the DSpace ecosystem. Supports multi-value fields using the standard DSpace value separator ("||"). Despite the name it does support reading Excel files.
Requires Python 3.6 or greater. CSV and Excel support comes from the Pandas library.
Functionality
- Read/write CSV files ✓
- Read Excel files ✓
- Validate dates, ISSNs, ISBNs, and multi-value separators ("||") ✓
- Fix leading, trailing, and excessive whitespace ✓
- Fix invalid multi-value separators ("|") using
--unsafe-fixes✓ - Remove unnecessary Unicode like non-breaking spaces, replacement characters, etc ✓
Installation
The easiest way to install CSV Metadata Quality is with pipenv:
$ git clone https://git.sr.ht/~alanorth/csv-metadata-quality
$ cd csv-metadata-quality
$ pipenv install
$ pipenv shell
Otherwise, if you don't have pipenv, you can use a vanilla Python virtual environment:
$ git clone https://git.sr.ht/~alanorth/csv-metadata-quality
$ cd csv-metadata-quality
$ python3 -m venv venv
$ source venv/bin/activate
$ pip install -r requirements.txt
Usage
Run CSV Metadata Quality with the --help flag to see available options:
$ python -m csv_metadata_quality --help
To validate and clean a CSV file you must specify input and output files using the -i and -o options. For example, using the included test file:
$ python -m csv_metadata_quality -i data/test.csv -o /tmp/test.csv
You can enable "unsafe fixes" with the --unsafe-fixes option. This will attempt
Todo
- Reporting / summary
- Real logging
- Detect and fix duplicate values like "Alan||Alan"
License
This work is licensed under the GPLv3.
The license allows you to use and modify the work for personal and commercial purposes, but if you distribute the work you must provide users with a means to access the source code for the version you are distributing. Read more about the GPLv3 at TL;DR Legal.