alanorth/csv-metadata-quality
1
0
Fork 0
mirror of https://github.com/ilri/csv-metadata-quality.git synced 2024-12-22 20:22:18 +01:00
Code Releases Activity
337 Commits 4 Branches 17 Tags 2 MiB
9f2dc0a0f5
Commit Graph

337 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Alan Orth
abf74909ee
data/test.csv: Add missing date 
I was meaning to test for an invalid multi-value separator here!
 2019-07-30 21:01:42 +03:00
Alan Orth
40d5f7d81b
Add support for removing newlines 
This was tricky because of the nature of newlines. In actuality we
are removing Unix line feeds here (U+000A) because Windows carriage
returns are actually already removed by the string stripping in the
whitespace fix.

Creating the test case in Vim was difficult because I couldn't fig-
ure out how to manually enter a line feed character. In the end I
used a search and replace on a known pattern like "ALAN", replacing
it with \r. Neither entering the Unicode code point (U+000A) direc-
tly or typing an "Enter" character after ^V worked. Grrr.
 2019-07-30 20:05:12 +03:00
Alan Orth
346e66ca98
README.md: Add more information to introduction 2019-07-30 17:44:30 +03:00
Alan Orth
bad2fce124
pytest.ini: Don't print captured output 
It makes the summary of passes and fails more annoying to read due
to the lines being long and including newlines. I am actually not
sure why this fixes it, though...

See: https://pytest.readthedocs.io/en/latest/capture.html
 2019-07-30 16:43:31 +03:00
Alan Orth
3c798fb504
Use pycountry instead of iso-639 for languages 
The latter is a fork that hasn't been updated since 2016 and the
original still seems to be well maintained, with recent database
updates as well as tests for Python 3.7.

Also, pycountry supports ISO 3166-2 (administrative zones), which
we could eventually use for sub regions.
 2019-07-30 16:39:26 +03:00
Alan Orth
a85b410ab9
README.md: Improve introduction and functionality 2019-07-30 16:09:15 +03:00
Alan Orth
4e3511cd55
csv_metadata_quality/check.py: Fix AGROVOC lookup 
We actually only need to see if there are more than zero matches
because a term like "Nigeria" will match in English, Spanish, etc,
whereas terms that *really* don't match will have zero results.
 2019-07-30 14:51:44 +03:00
Alan Orth
1fd3d8bc2f
Add pytest.ini from responder 
This seems to be produce a more informative output, though I'm not
sure how to filter the annoying deprecation warnings pytest throws
about things that are usually done by modules I'm using, not by me.

From: https://github.com/taoufik07/responder/blob/master/pytest.ini
 2019-07-30 00:45:18 +03:00
Alan Orth
b4aaffd6f1
Update python requirements 
Generated using pipenv:

  $ pipenv lock -r > requirements.txt
 2019-07-30 00:34:32 +03:00
Alan Orth
5ea2e856e4
data/test.csv: Add dc.subject column for AGROVOC tests 2019-07-30 00:33:31 +03:00
Alan Orth
1f65a28307
Add support for validating subjects against AGROVOC 
Checks values in the dc.subject or dcterms.subject field against the
AGROVOC REST API hosted by FAO. Code borrowed from agrovoc-lookup.py.

See: http://agrovoc.uniroma2.it/agrovoc/agrovoc/en/
See: https://github.com/ilri/DSpace/blob/5_x-prod/agrovoc-lookup.py
 2019-07-30 00:30:31 +03:00
Alan Orth
bb882315f1
Update python requirements 
Generated using pipenv:

  $ pipenv lock -r > requirements.txt
  $ pipenv lock -r -d > requirements-dev.txt
 2019-07-29 19:10:49 +03:00
Alan Orth
a36454a3ac
Add support for validating languages 
Will validate against ISO 639-2 or ISO 639-3 depending on how long
the language field is. Otherwise will return that the language is
invalid.

Does not currently have any support for generic values like "Other".
 2019-07-29 18:59:42 +03:00
Alan Orth
1978fa7b48
Add iso-639 library to validate languages 2019-07-29 18:58:50 +03:00
Alan Orth
e49b4e8f22
README.md: Try to simplify list of functionality 2019-07-29 18:25:38 +03:00
Alan Orth
0eb852a65b
README.md: Improve note about unsafe options 2019-07-29 18:14:50 +03:00
Alan Orth
8d92498bbf
Add .gitignore 2019-07-29 18:10:29 +03:00
Alan Orth
8c34c2d6e6
README.md: Add note about removing duplicate values 2019-07-29 18:09:48 +03:00
Alan Orth
1e444cf040
Add fix for duplicate metadata values 2019-07-29 18:05:03 +03:00
Alan Orth
d7888d59a8
csv_metadata_quality/check.py: Return date even if it is invalid 
Otherwise it is missing from the final CSV and then we can't even
fix it. :)
 2019-07-29 17:40:14 +03:00
Alan Orth
8509006165
data/test.csv: Use more descriptive tests 
To make it obvious what each item is testing.
 2019-07-29 17:38:46 +03:00
Alan Orth
e33551776c
README.md: Update note about unsafe options 2019-07-29 17:25:42 +03:00
Alan Orth
7f781d7077
README.md: Finish writing usage section 2019-07-29 17:21:34 +03:00
Alan Orth
50ae4e17f2
csv_metadata_quality/fix.py: Fix indent 2019-07-29 17:14:48 +03:00
Alan Orth
fa4fa3491b
Add check for "suspicious" characters 
These standalone characters often indicate issues with encoding or
copy/paste in languages with accents like French and Spanish. For
example: foreˆt should be forêt.

It is not possible to fix these issues automatically, but this will
print a warning so you can notify the owner of the data.
 2019-07-29 17:08:49 +03:00
Alan Orth
8047a57cc5
Add support for fixing "unnecessary" Unicode 
These are things like non-breaking spaces, "replacement" characters,
etc that add nothing to the metadata and often cause errors during
parsing or displaying in a UI.
 2019-07-29 16:38:10 +03:00
Alan Orth
ae66382046
README.md: Add note about unnecessary Unicode fixes 2019-07-29 16:34:39 +03:00
Alan Orth
d73f7b54b1
csv_metadata_quality/app.py: Improve comments 2019-07-29 16:24:35 +03:00
Alan Orth
42920e9c7c
Test Python regular expression matches directly 
Match objects always have a boolean value of True.

See: https://docs.python.org/3.7/library/re.html
 2019-07-29 16:16:30 +03:00
Alan Orth
7b5db1f5d9
csv_metadata_quality/app.py: Remove erroneous comment 2019-07-29 16:15:25 +03:00
Alan Orth
9ac9474c69
README.md: Add todo about duplicates 2019-07-29 12:40:53 +03:00
Alan Orth
e000bd1f88
README.md: Add Travis CI badge 2019-07-29 12:19:10 +03:00
Alan Orth
3554c2991f
README.md: Add note about Python version 2019-07-29 12:15:09 +03:00
Alan Orth
8e3e7a3573
.travis.yml: Only test Python 3.6 and 3.7 
I'm using f-strings, which are only supported in Python 3.6+.
 2019-07-29 12:13:55 +03:00
Alan Orth
913500cfff
.travis.yml: Remove env 
This seems to make Travis run one job for each variable.
 2019-07-29 11:51:12 +03:00
Alan Orth
17f4a26ac0
Add Travis CI support 
I'm happy with SourceHut's build support, but Travis allows us to
tests other Python versions.
 2019-07-29 11:45:58 +03:00
Alan Orth
ac127e7f8a
README.md: Add usage section 2019-07-29 11:30:06 +03:00
Alan Orth
aabb57321c
README.md: Improve 
Reorganize functionality section and add installation section.
 2019-07-29 11:15:51 +03:00
Alan Orth
a8a41d60b6
README.md: Add note about Pandas 2019-07-29 10:56:02 +03:00
Alan Orth
cf6c01caaf
README.md: Add notes about unsafe fixes 2019-07-28 23:01:00 +03:00
Alan Orth
40e77db713
Add "unsafe fixes" runtime option 
In this case it fixes occurences of invalid multi-value separators.
DSpace uses "||" to separate multiple values in one field, but our
editors sometimes give us files with mistakes like "|". We can fix
these to be correct multi-value separators if we are sure that the
metadata is not actually using "|" for some legitimate purpose.
 2019-07-28 22:53:39 +03:00
Alan Orth
a93b5b31c5
Add support for command line arguments 
Currently only supports specifying input and output files with -i
and -o. Eventually I'll add more options like dry run, debug, and
maybe things like forcing unsafe fixes.
 2019-07-28 20:31:57 +03:00
Alan Orth
4e6225c0a9
README.md: Add note about Excel files 2019-07-28 18:38:36 +03:00
Alan Orth
d293214d3a
README.md: Add note about checking dates 2019-07-28 18:37:46 +03:00
Alan Orth
87b1997051
Fix whitespace errors found by flake8 2019-07-28 17:47:28 +03:00
Alan Orth
c6e7d6d9b5
Add flake8 to pipenv dev environment 
To help check PEP8 formatting/style compliance.
 2019-07-28 17:46:30 +03:00
Alan Orth
aadb3117eb
csv_metadata_quality/app.py: Remove unused test input files 2019-07-28 17:45:05 +03:00
Alan Orth
ce8f140c66
tests: Remove unused pytest import 2019-07-28 17:42:54 +03:00
Alan Orth
e88d35ace3
csv_metadata_quality/app.py: Use regex in column match 
Check for a column that has "issn" or "isbn" in the name rather
than by its explicit name, as the column is dc.identifier.issn now,
but will be cg.issn in the future if CG Core v2 happens.
 2019-07-28 17:27:20 +03:00
Alan Orth
4687e2f5fa
README.md: Remove todo for date validation 2019-07-28 17:26:39 +03:00