alanorth/csv-metadata-quality
1
0
Fork 0
mirror of https://github.com/ilri/csv-metadata-quality.git synced 2025-08-19 19:25:51 +02:00
Code Releases Activity
586 Commits 4 Branches 18 Tags
fa7bde6fc0a11298e447ca79d305460469fffe7d
Commit Graph

586 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Alan Orth
e7cb8920db tests/test_check.py: Update date tests 2019-08-21 15:34:52 +03:00
Alan Orth
ed5612fbcf Add column name to output in date checks 
This makes it easier to understand where the error is in case a CSV
has multiple date fields, for example:

    Missing date (dc.date.issued).
    Missing date (dc.date.issued[]).

If you have 126 items and you get 126 "Missing date" messages then
it's likely that 100 of the items have dates in one field, and the
others have dates in other field.
 2019-08-21 15:31:12 +03:00
Alan Orth
3247495cee CHANGELOG.md: Remove extra space 2019-08-11 10:43:27 +03:00
Alan Orth
7255bf4707 Version 0.2.1 v0.2.1 2019-08-11 10:39:39 +03:00
Alan Orth
3aaf18c290 CHANGELOG.md: Move unreleased changes to 0.2.1 2019-08-11 10:39:18 +03:00
Alan Orth
745306edd7 CHANGELOG.md: Add note about replacement of unnccesary Unicode 2019-08-11 00:09:35 +03:00
Alan Orth
e324e321a2 data/test.csv: Add test for replacement of unneccessary Unicode 2019-08-11 00:08:44 +03:00
Alan Orth
232ff99898 csv_metadata_quality/fix.py: Add more unneccessary Unicode fixes 
Add a check for soft hyphens (U+00AD). In one sample CSV I have a
normal hyphen followed by a soft hyphen in an ISBN. This causes the
ISBN validation to fail.
 2019-08-11 00:07:21 +03:00
Alan Orth
13d5221378 csv_metadata_quality/check.py: Fix test for False 2019-08-10 23:52:53 +03:00
Alan Orth
3c7a9eb75b CHANGELOG.md: Add check for uncommon filename extensions 2019-08-10 23:47:46 +03:00
Alan Orth
a99fbd8a51 data/test.csv: Add test case for uncommon filename extension 2019-08-10 23:46:56 +03:00
Alan Orth
e801042340 tests/test_check.py: Fix unused result 
We don't need to capture the function's return value here because
pytest will capture stdout from the function.
 2019-08-10 23:45:41 +03:00
Alan Orth
62ef2a4489 tests/test_check.py: Add tests for file extensions 2019-08-10 23:44:13 +03:00
Alan Orth
9ce7dc6716 Add check for uncommon filenames 
Generally we want people to upload documents in accessible formats
like PDF, Word, Excel, and PowerPoint. This check warns if a file
is using an uncommon extension.
 2019-08-10 23:41:16 +03:00
Alan Orth
5ff584a8d7 Version 0.2.0 v0.2.0 2019-08-09 01:39:51 +03:00
Alan Orth
4cf7bc182b Update requirements-dev.txt 
Generated with:

  $ pipenv lock -r -d > requirements-dev.txt
 2019-08-09 01:34:54 +03:00
Alan Orth
7d3f5aae66 CHANGELOG.md: Add pytest-clarity 2019-08-09 01:33:34 +03:00
Alan Orth
c77c065e25 Update Pipfile.lock 2019-08-09 01:32:53 +03:00
Alan Orth
8fb40d96b1 Pipfile: Add pytest-clarity to dev packages 
This helps you understand the cryptic assertion error output from
pytest. For some reason pytest-clarity is a pre-release package so
we need to install it in pipenv with --pre.
 2019-08-09 01:30:37 +03:00
Alan Orth
5f2e3ff4bd CHANGELOG.md: Add improved suspicious character check 2019-08-09 01:28:07 +03:00
Alan Orth
d93c2aae13 tests/test_check.py: Update suspicious character check 
The suspicious character check was updated to include the name of
the field where the metadata value with the  suspicious character
exists.
 2019-08-09 01:26:38 +03:00
Alan Orth
62fea95087 Improve suspicious character detection 
Now it will print just the part of the metadata value that contains
the suspicious character (up to 80 characters, so we don't make the
line break on terminals that use 80 character width by default).

Also, print the name of the field in which the metadata value is so
that it is easier for the user to locate.
 2019-08-09 01:25:40 +03:00
Alan Orth
8772bdec51 csv_metadata_quality/app.py: Explicitly exit with success 2019-08-04 09:10:37 +03:00
Alan Orth
6d4ecd75aa csv_metadata_quality/app.py: Close files before exit 2019-08-04 09:10:19 +03:00
Alan Orth
264ce1d1df CHANGELOG.md: Add new item for Ctrl-C handling 2019-08-03 22:18:44 +03:00
Alan Orth
f4e7fd73f5 csv_metadata_quality/app.py: Handle Ctrl-C 
Instead of printing an ugly two-page stack trace.
 2019-08-03 21:11:57 +03:00
Alan Orth
a00d3d7ea5 README.md: Simplify installation instructions 
Pipenv has captured the local dependency with `-e .` so now it gets
installed by the Pipfile or requirements.txt.
 2019-08-02 11:02:50 +03:00
Alan Orth
f772a3be41 Update python requirements 
Generated using pipenv:

  $ pipenv lock -r > requirements.txt
 2019-08-02 11:02:25 +03:00
Alan Orth
d1b3e9e375 pipenv install -e . 2019-08-02 10:58:21 +03:00
Alan Orth
85ae7bdc5a Increment version to 0.1.0 v0.1.0 2019-08-02 00:12:47 +03:00
Alan Orth
980ef0d98d Add initial changelog 2019-08-02 00:10:28 +03:00
Alan Orth
0561300ebe Add option to print version with --version or -V 
I guess `-v` is more commonly used for "verbose" so I will use the
short option of `-V` for version.
 2019-08-02 00:09:54 +03:00
Alan Orth
456b8a2f26 Update tests 2019-08-01 23:59:11 +03:00
Alan Orth
0ed390dbd5 README.md: Update AGROVOC information 
Now details the new `--agrovoc-fields` option.
 2019-08-01 23:54:40 +03:00
Alan Orth
bf876a046a Rework AGROVOC validation 
AGROVOC validation is now disabled by default, but can be enabled
on a field-by-field basis. For example, countries and regions are
also present in AGROVOC. Fields with these values can be enabled
using the new `--agrovoc-fields` option.

I reworked the script output to show the field name when printing
an invalid term so that the user knows in which field the term is.
 2019-08-01 23:51:58 +03:00
Alan Orth
576b3a3638 csv_metadata_quality/__main__.py: Fix spacing 
Identified by flake8.
 2019-08-01 23:28:16 +03:00
Alan Orth
a6eba0fc1a .build.yml: Use pipenv run instead of pipenv shell 
The latter creates a shell and of course it doesn't ever exit!
 2019-07-31 17:58:21 +03:00
Alan Orth
857492cd93 .build.yml: Try fix CLI test 
I was writing the CLI output to /dev/null because I was lazy, but
actually this might break the SourceHut build.
 2019-07-31 17:54:49 +03:00
Alan Orth
db42fbfea9 .build.yml: Try to run the script itself 2019-07-31 17:48:17 +03:00
Alan Orth
734ae7f011 setup.py: Add "DSpace" to description 
I suppose someone searching PyPi (eventually) would be happy to see
that this was written specifically with DSpace in mind.
 2019-07-31 17:42:27 +03:00
Alan Orth
fd3861e7cd README.md: Update installation and usage instructions 
It is much easier now that I have created a proper package.
 2019-07-31 17:41:18 +03:00
Alan Orth
cec1a34dfe .gitignore: Ignore egg cache from distutils 2019-07-31 17:38:54 +03:00
Alan Orth
9100efdf50 Re-work as a proper standalone Python package 
Add a setup.py so that installation is easier and a standalone CLI
script called csv-metadata-quality is provided. Now the user only
needs to run this from a virtual environment inside the project
directory:

    $ pip install .

Eventually I could publish this on PyPi when I settle on a more
appropriate package name.

See: https://packaging.python.org/tutorials/packaging-projects/
See: https://chriswarrick.com/blog/2014/09/15/python-apps-the-right-way-entry_points-and-scripts/
 2019-07-31 17:34:36 +03:00
Alan Orth
4c4f4a3ba2 README.md: Update todos 2019-07-31 16:33:49 +03:00
Alan Orth
22cc7bc793 README.md: Improve section on unsafe fixes 2019-07-31 16:00:05 +03:00
Alan Orth
915327539a pytest.ini: Ignore deprecation warnings 
These come from third-party libraries I have no control over.

See: https://docs.pytest.org/en/latest/warnings.html#deprecationwarning-and-pendingdeprecationwarning
 2019-07-31 13:33:20 +03:00
Alan Orth
63ffd77723 data/test.csv: Clarify that newline is a line feed 2019-07-31 13:03:43 +03:00
Alan Orth
b9d041927e Update python requirements 
Generated using pipenv:

  $ pipenv lock -r > requirements.txt
  $ pipenv lock -r -d > requirements-dev.txt
 2019-07-30 23:06:26 +03:00
Alan Orth
020c87768e data/test.csv: Add item with missing date 2019-07-30 21:02:51 +03:00
Alan Orth
abf74909ee data/test.csv: Add missing date 
I was meaning to test for an invalid multi-value separator here!
 2019-07-30 21:01:42 +03:00