alanorth
/
csv-metadata-quality
mirror of https://github.com/ilri/csv-metadata-quality.git
1
0
Fork 0
Code Releases Activity
154 Commits 7 Branches 17 Tags 5.4 MiB
Commit Graph

154 Commits

Author SHA1 Message Date
Alan Orth 0388145b81
Add configuration for isort 
See: https://sourcery.ai/blog/python-best-practices/
 2019-08-29 01:14:31 +03:00
Alan Orth d97dcd19db
Format with black 2019-08-29 01:10:39 +03:00
Alan Orth b375f0e895
Add black and isort to pipenv dev dependencies 
These do a very opinionated automatic formatting and validation of
code.

See: https://sourcery.ai/blog/python-best-practices/
 2019-08-29 01:08:38 +03:00
Alan Orth 865c61d316
Add note about updated python dependencies 2019-08-28 21:02:21 +03:00
Alan Orth 3b2ba57b75
Update python requirements 
Generated using pipenv:

  $ pipenv lock -r > requirements.txt
  $ pipenv lock -r -d > requirements-dev.txt
 2019-08-28 21:01:48 +03:00
Alan Orth 2805c556a9
Pipfile.lock: Run pipenv update 
Brings numpy 1.17.1, pandas 0.25.1, requests-cache 0.5.2, and pandas
0.25.1.
 2019-08-28 20:58:35 +03:00
Alan Orth c354a3687c
Release version 0.2.2 2019-08-28 00:10:17 +03:00
Alan Orth 07f80cb37f
tests/test_fix.py: Add test for missing space after comma 2019-08-28 00:08:56 +03:00
Alan Orth 89d72540f1
data/test.csv: Add sample for missing space after comma 2019-08-28 00:08:26 +03:00
Alan Orth 81190d56bb
Add fix for missing space after commas 
This happens in names very often, for example in the contributor
and citation fields. I will limit this to those fields for now and
hide this fix behind the "unsafe fixes" option until I test it more.
 2019-08-28 00:05:52 +03:00
Alan Orth 2af714fb05
README.md: Add a handful of TODOs 2019-08-27 00:12:41 +03:00
Alan Orth cc863a6bdd
CHANGELOG.md: Add note about excluding fields 2019-08-27 00:11:22 +03:00
Alan Orth 113e7cd8b6
csv_metadata_quality/app.py: Add ability to skip fields 
The user may want to skip the checking and fixing of certain fields
in the input file.
 2019-08-27 00:10:07 +03:00
Alan Orth bd984f3db5
README.md: Update TravisCI badge 2019-08-22 15:07:03 +03:00
Alan Orth 3f4e84a638
README.md: Use ILRI GitHub remote 2019-08-22 14:54:12 +03:00
Alan Orth c52b3ed131
CHANGELOG.md: Add note about AGROVOC 2019-08-21 16:37:49 +03:00
Alan Orth 884e8f970d
csv_metadata_quality/check.py: Simplify AGROVOC check 
I recycled this code from a separate agrovoc-lookup.py script that
checks lines in a text file to see if they are valid AGROVOC terms
or not. There I was concerned about skipping comments or something
I think, but we don't need to check that here. We simply check the
term that is in the field and inform the user if it's valid or not.
 2019-08-21 16:35:29 +03:00
Alan Orth 6d02f5026a
CHANGELOG.md: Add note about date checks 2019-08-21 15:35:46 +03:00
Alan Orth e7cb8920db
tests/test_check.py: Update date tests 2019-08-21 15:34:52 +03:00
Alan Orth ed5612fbcf
Add column name to output in date checks 
This makes it easier to understand where the error is in case a CSV
has multiple date fields, for example:

    Missing date (dc.date.issued).
    Missing date (dc.date.issued[]).

If you have 126 items and you get 126 "Missing date" messages then
it's likely that 100 of the items have dates in one field, and the
others have dates in other field.
 2019-08-21 15:31:12 +03:00
Alan Orth 3247495cee
CHANGELOG.md: Remove extra space 2019-08-11 10:43:27 +03:00
Alan Orth 7255bf4707
Version 0.2.1 2019-08-11 10:39:39 +03:00
Alan Orth 3aaf18c290
CHANGELOG.md: Move unreleased changes to 0.2.1 2019-08-11 10:39:18 +03:00
Alan Orth 745306edd7
CHANGELOG.md: Add note about replacement of unnccesary Unicode 2019-08-11 00:09:35 +03:00
Alan Orth e324e321a2
data/test.csv: Add test for replacement of unneccessary Unicode 2019-08-11 00:08:44 +03:00
Alan Orth 232ff99898
csv_metadata_quality/fix.py: Add more unneccessary Unicode fixes 
Add a check for soft hyphens (U+00AD). In one sample CSV I have a
normal hyphen followed by a soft hyphen in an ISBN. This causes the
ISBN validation to fail.
 2019-08-11 00:07:21 +03:00
Alan Orth 13d5221378
csv_metadata_quality/check.py: Fix test for False 2019-08-10 23:52:53 +03:00
Alan Orth 3c7a9eb75b
CHANGELOG.md: Add check for uncommon filename extensions 2019-08-10 23:47:46 +03:00
Alan Orth a99fbd8a51
data/test.csv: Add test case for uncommon filename extension 2019-08-10 23:46:56 +03:00
Alan Orth e801042340
tests/test_check.py: Fix unused result 
We don't need to capture the function's return value here because
pytest will capture stdout from the function.
 2019-08-10 23:45:41 +03:00
Alan Orth 62ef2a4489
tests/test_check.py: Add tests for file extensions 2019-08-10 23:44:13 +03:00
Alan Orth 9ce7dc6716
Add check for uncommon filenames 
Generally we want people to upload documents in accessible formats
like PDF, Word, Excel, and PowerPoint. This check warns if a file
is using an uncommon extension.
 2019-08-10 23:41:16 +03:00
Alan Orth 5ff584a8d7
Version 0.2.0 2019-08-09 01:39:51 +03:00
Alan Orth 4cf7bc182b
Update requirements-dev.txt 
Generated with:

  $ pipenv lock -r -d > requirements-dev.txt
 2019-08-09 01:34:54 +03:00
Alan Orth 7d3f5aae66
CHANGELOG.md: Add pytest-clarity 2019-08-09 01:33:34 +03:00
Alan Orth c77c065e25
Update Pipfile.lock 2019-08-09 01:32:53 +03:00
Alan Orth 8fb40d96b1
Pipfile: Add pytest-clarity to dev packages 
This helps you understand the cryptic assertion error output from
pytest. For some reason pytest-clarity is a pre-release package so
we need to install it in pipenv with --pre.
 2019-08-09 01:30:37 +03:00
Alan Orth 5f2e3ff4bd
CHANGELOG.md: Add improved suspicious character check 2019-08-09 01:28:07 +03:00
Alan Orth d93c2aae13
tests/test_check.py: Update suspicious character check 
The suspicious character check was updated to include the name of
the field where the metadata value with the  suspicious character
exists.
 2019-08-09 01:26:38 +03:00
Alan Orth 62fea95087
Improve suspicious character detection 
Now it will print just the part of the metadata value that contains
the suspicious character (up to 80 characters, so we don't make the
line break on terminals that use 80 character width by default).

Also, print the name of the field in which the metadata value is so
that it is easier for the user to locate.
 2019-08-09 01:25:40 +03:00
Alan Orth 8772bdec51
csv_metadata_quality/app.py: Explicitly exit with success 2019-08-04 09:10:37 +03:00
Alan Orth 6d4ecd75aa
csv_metadata_quality/app.py: Close files before exit 2019-08-04 09:10:19 +03:00
Alan Orth 264ce1d1df
CHANGELOG.md: Add new item for Ctrl-C handling 2019-08-03 22:18:44 +03:00
Alan Orth f4e7fd73f5
csv_metadata_quality/app.py: Handle Ctrl-C 
Instead of printing an ugly two-page stack trace.
 2019-08-03 21:11:57 +03:00
Alan Orth a00d3d7ea5
README.md: Simplify installation instructions 
Pipenv has captured the local dependency with `-e .` so now it gets
installed by the Pipfile or requirements.txt.
 2019-08-02 11:02:50 +03:00
Alan Orth f772a3be41
Update python requirements 
Generated using pipenv:

  $ pipenv lock -r > requirements.txt
 2019-08-02 11:02:25 +03:00
Alan Orth d1b3e9e375
pipenv install -e . 2019-08-02 10:58:21 +03:00
Alan Orth 85ae7bdc5a
Increment version to 0.1.0 2019-08-02 00:12:47 +03:00
Alan Orth 980ef0d98d
Add initial changelog 2019-08-02 00:10:28 +03:00
Alan Orth 0561300ebe
Add option to print version with `--version` or `-V` 
I guess `-v` is more commonly used for "verbose" so I will use the
short option of `-V` for version.
 2019-08-02 00:09:54 +03:00