alanorth/csv-metadata-quality
1
0
Fork 0
mirror of https://github.com/ilri/csv-metadata-quality.git synced 2025-05-08 06:06:00 +02:00
Code Releases Activity
477 Commits 5 Branches 18 Tags
15f52f8be83e47bf142c23f13c65a687de8be3c1
Commit Graph

477 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Alan Orth
e15c98cccb Move unreleased changes to v0.3.0 v0.3.0 2019-09-26 14:06:31 +03:00
Alan Orth
93c4e1a993 Update python requirements 
Generated using pipenv:

  $ pipenv lock -r > requirements.txt
  $ pipenv lock -r -d > requirements-dev.txt
 2019-09-26 14:05:37 +03:00
Alan Orth
9963b2bb64 Pipfile.lock: Run pipenv update 2019-09-26 14:04:50 +03:00
Alan Orth
76291c1876 CHANGELOG.md: Add note about language validation 2019-09-26 14:03:18 +03:00
Alan Orth
604bd5bda6 Reformat tests with black 2019-09-26 14:02:51 +03:00
Alan Orth
e7c220039b README.md: Add note about experimental language validation 2019-09-26 13:59:50 +03:00
Alan Orth
d7b5e378bc setup.py: Add langid 2019-09-26 13:49:32 +03:00
Alan Orth
8435ee242d Experimental language detection using langid 
Works decenty well assuming the title, abstract, and citation fields
are an accurate representation of the language as identified by the
language field. Handles ISO 639-1 (alpha 2) and ISO 639-3 (alpha 3)
values seamlessly.

This includes updated pipenv environment, test data, pytest tests
for both correct and incorrect ISO 639-1 and ISO 639-3 languages,
and a new command line option "-e".
 2019-09-26 13:46:32 +03:00
Alan Orth
7ac1c6f554 README.md: Update comment about ISO 639-3 
The pycountry library is actually using ISO 639-3 apparently.

See: https://pypi.org/project/pycountry/
 2019-09-26 07:51:41 +03:00
Alan Orth
86d4623fd3 More ISO 639-1 and ISO 639-3 fixes 
ISO 639-1 uses two-letter codes and ISO 639-3 uses three-letter codes.
Technically there ISO 639-2/T and ISO 639-2/B, which also uses three
letter codes, but those are not supported by the pycountry library
so I won't even worry about them.

See: https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes
 2019-09-26 07:44:39 +03:00
Alan Orth
ddbe970342 data/test.csv: Update titles of language tests 
ISO 639-1 is alpha 2 and ISO 639-3 is alpha 3.

See: https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes
 2019-09-26 07:40:27 +03:00
Alan Orth
31c78ca6f3 data/test.csv: Rename contributor column to title 
This makes more sense as a description of each test and the titles
are obviously not authors.
 2019-09-26 05:50:40 +03:00
Alan Orth
154d05b5e2 CHANGELOG.md: Update notes 2019-09-24 18:55:05 +03:00
Alan Orth
186f146edb Pipfile.lock: Run pipenv update 
Synchronizes state with the Pipfile and brings some new deps.
 2019-09-24 18:54:49 +03:00
Alan Orth
a4cb301943 CHANGELOG.md: Add note about csvkit 2019-09-24 18:49:20 +03:00
Alan Orth
219e37526d Pipfile: Add csvkit to dev requirements 
Used to inspect CSV files during testing and development.
 2019-09-24 18:48:01 +03:00
Alan Orth
f304ca6a33 csv_metadata_quality/app.py: Use simpler column iteration 
I don't know where I got the other one...
 2019-09-21 17:19:39 +03:00
Alan Orth
3d5c8bdf5d CHANGELOG.md: Add notes about updated python packages 2019-09-11 16:45:39 +03:00
Alan Orth
480956d54d Pipfile.lock: Run pipenv update 2019-09-11 16:45:16 +03:00
Alan Orth
d9fc09f121 Fix references to ISO 639 
It turns out that ISO 639-1 is the two-letter codes, and ISO 639-2
is the three-letter codes, aka alpha2 and alpha3.

See: https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes
 2019-09-11 16:36:53 +03:00
Alan Orth
b5899001b7 CHANGELOG.md: Add note about black and isort 2019-08-29 01:26:11 +03:00
Alan Orth
c92977d1ca Update requirements-dev.txt 
Generated with:

  $ pipenv lock -r -d > requirements-dev.txt
 2019-08-29 01:25:14 +03:00
Alan Orth
280a99c8a8 Sort imports with isort 
See: https://sourcery.ai/blog/python-best-practices/
 2019-08-29 01:15:04 +03:00
Alan Orth
0388145b81 Add configuration for isort 
See: https://sourcery.ai/blog/python-best-practices/
 2019-08-29 01:14:31 +03:00
Alan Orth
d97dcd19db Format with black 2019-08-29 01:10:39 +03:00
Alan Orth
b375f0e895 Add black and isort to pipenv dev dependencies 
These do a very opinionated automatic formatting and validation of
code.

See: https://sourcery.ai/blog/python-best-practices/
 2019-08-29 01:08:38 +03:00
Alan Orth
865c61d316 Add note about updated python dependencies 2019-08-28 21:02:21 +03:00
Alan Orth
3b2ba57b75 Update python requirements 
Generated using pipenv:

  $ pipenv lock -r > requirements.txt
  $ pipenv lock -r -d > requirements-dev.txt
 2019-08-28 21:01:48 +03:00
Alan Orth
2805c556a9 Pipfile.lock: Run pipenv update 
Brings numpy 1.17.1, pandas 0.25.1, requests-cache 0.5.2, and pandas
0.25.1.
 2019-08-28 20:58:35 +03:00
Alan Orth
c354a3687c Release version 0.2.2 v0.2.2 2019-08-28 00:10:17 +03:00
Alan Orth
07f80cb37f tests/test_fix.py: Add test for missing space after comma 2019-08-28 00:08:56 +03:00
Alan Orth
89d72540f1 data/test.csv: Add sample for missing space after comma 2019-08-28 00:08:26 +03:00
Alan Orth
81190d56bb Add fix for missing space after commas 
This happens in names very often, for example in the contributor
and citation fields. I will limit this to those fields for now and
hide this fix behind the "unsafe fixes" option until I test it more.
 2019-08-28 00:05:52 +03:00
Alan Orth
2af714fb05 README.md: Add a handful of TODOs 2019-08-27 00:12:41 +03:00
Alan Orth
cc863a6bdd CHANGELOG.md: Add note about excluding fields 2019-08-27 00:11:22 +03:00
Alan Orth
113e7cd8b6 csv_metadata_quality/app.py: Add ability to skip fields 
The user may want to skip the checking and fixing of certain fields
in the input file.
 2019-08-27 00:10:07 +03:00
Alan Orth
bd984f3db5 README.md: Update TravisCI badge 2019-08-22 15:07:03 +03:00
Alan Orth
3f4e84a638 README.md: Use ILRI GitHub remote 2019-08-22 14:54:12 +03:00
Alan Orth
c52b3ed131 CHANGELOG.md: Add note about AGROVOC 2019-08-21 16:37:49 +03:00
Alan Orth
884e8f970d csv_metadata_quality/check.py: Simplify AGROVOC check 
I recycled this code from a separate agrovoc-lookup.py script that
checks lines in a text file to see if they are valid AGROVOC terms
or not. There I was concerned about skipping comments or something
I think, but we don't need to check that here. We simply check the
term that is in the field and inform the user if it's valid or not.
 2019-08-21 16:35:29 +03:00
Alan Orth
6d02f5026a CHANGELOG.md: Add note about date checks 2019-08-21 15:35:46 +03:00
Alan Orth
e7cb8920db tests/test_check.py: Update date tests 2019-08-21 15:34:52 +03:00
Alan Orth
ed5612fbcf Add column name to output in date checks 
This makes it easier to understand where the error is in case a CSV
has multiple date fields, for example:

    Missing date (dc.date.issued).
    Missing date (dc.date.issued[]).

If you have 126 items and you get 126 "Missing date" messages then
it's likely that 100 of the items have dates in one field, and the
others have dates in other field.
 2019-08-21 15:31:12 +03:00
Alan Orth
3247495cee CHANGELOG.md: Remove extra space 2019-08-11 10:43:27 +03:00
Alan Orth
7255bf4707 Version 0.2.1 v0.2.1 2019-08-11 10:39:39 +03:00
Alan Orth
3aaf18c290 CHANGELOG.md: Move unreleased changes to 0.2.1 2019-08-11 10:39:18 +03:00
Alan Orth
745306edd7 CHANGELOG.md: Add note about replacement of unnccesary Unicode 2019-08-11 00:09:35 +03:00
Alan Orth
e324e321a2 data/test.csv: Add test for replacement of unneccessary Unicode 2019-08-11 00:08:44 +03:00
Alan Orth
232ff99898 csv_metadata_quality/fix.py: Add more unneccessary Unicode fixes 
Add a check for soft hyphens (U+00AD). In one sample CSV I have a
normal hyphen followed by a soft hyphen in an ISBN. This causes the
ISBN validation to fail.
 2019-08-11 00:07:21 +03:00
Alan Orth
13d5221378 csv_metadata_quality/check.py: Fix test for False 2019-08-10 23:52:53 +03:00