Commit Graph

486 Commits

Author SHA1 Message Date
Alan Orth 141b2e1da3
csv_metadata_quality/check.py: update region output
Add the country to the message about missing regions. This makes it
easier to see which country is triggering the missing region error,
and helps in case of debugging possible mistakes in the data coming
from the country_converter library.
2022-11-28 17:40:27 +03:00
Alan Orth 7097136b7e
Use my fork of country_converter again
There is an issue with the UN M.49 region for Myanmar.
2022-11-28 17:38:45 +03:00
Alan Orth d134c93663
Update requirements
Generated with poetry export:

    $ poetry export --without-hashes -f requirements.txt > requirements.txt
    $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt

I am trying `--without-hashes` to work around an error on pip install
when running in CI:

    ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==
2022-11-28 17:16:09 +03:00
Alan Orth 9858406894
poetry.lock: run poetry update 2022-11-28 17:15:19 +03:00
Alan Orth b02f1f65ee
pyproject.toml: use upstream country_converter
Version 0.8.0 has the country and UN M.49 region fixes.

See: https://github.com/konstantinstadler/country_converter/releases/tag/v0.8.0
2022-11-28 17:14:16 +03:00
Alan Orth 4d5ef38dde
pyproject.toml: add ipython to dev dependencies 2022-11-28 17:11:18 +03:00
Alan Orth eaa8f31faf
Update requirements
continuous-integration/drone/push Build is failing Details
Generated with poetry export:

    $ poetry export --without-hashes -f requirements.txt > requirements.txt
    $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt

I am trying `--without-hashes` to work around an error on pip install
when running in CI:

    ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==
2022-11-08 10:22:39 +03:00
Alan Orth df57988e5a
Use my fork of pycountry
Until they update to iso-codes 4.12.0.

See: https://github.com/flyingcircusio/pycountry/pull/149
2022-11-08 10:21:28 +03:00
Alan Orth bddf4da559
Update requirements
Generated with poetry export:

    $ poetry export --without-hashes -f requirements.txt > requirements.txt
    $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt

I am trying `--without-hashes` to work around an error on pip install
when running in CI:

    ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==
2022-11-08 10:06:26 +03:00
Alan Orth 15f52f8be8
Switch to my fork of country-converter
Until a few issues are resolved regarding new countries and regions.

See: https://github.com/konstantinstadler/country_converter/pull/122
See: https://github.com/konstantinstadler/country_converter/pull/123
2022-11-08 10:04:31 +03:00
Alan Orth bc909464c7
Update requirements
continuous-integration/drone/push Build is passing Details
Generated with poetry export:

    $ poetry export --without-hashes -f requirements.txt > requirements.txt
    $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt

I am trying `--without-hashes` to work around an error on pip install
when running in CI:

    ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==
2022-11-07 12:14:46 +03:00
Alan Orth 2d46259dfe
poetry.lock: run poetry update 2022-11-07 12:13:44 +03:00
Alan Orth ca82820a8e
pyproject.toml: update dependencies to latest 2022-11-07 12:13:28 +03:00
Alan Orth 86b4e5e182
Update requirements
continuous-integration/drone/push Build is passing Details
Generated with poetry export:

    $ poetry export --without-hashes -f requirements.txt > requirements.txt
    $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt

I am trying `--without-hashes` to work around an error on pip install
when running in CI:

    ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==
2022-11-01 12:21:41 +03:00
Alan Orth e5d5ae7e5d
poetry.lock: run poetry update 2022-11-01 12:20:43 +03:00
Alan Orth 8f3db86a36
CHANGELOG.md: fix header
continuous-integration/drone/push Build is passing Details
2022-10-31 11:43:14 +03:00
Alan Orth b0721b0a7a
.github: use ubuntu-22.04 for actions
continuous-integration/drone/push Build is passing Details
Apparently 'ubuntu-latest' is still 20.04 and today is 2022-10-03,
which seems a bit old!

See: https://github.com/actions/runner-images
2022-10-03 19:49:24 +03:00
Alan Orth 4e5faf51bd .github/workflows: use pip caching
See: https://github.com/actions/setup-python/blob/main/docs/advanced-usage.md#caching-packages
2022-10-03 19:39:52 +03:00
Alan Orth 5ea38d65bd .github/workflows: update actions
Update actions to latest versions:

- actions/checkout@v3
- actions/setup-python@v4
2022-10-03 19:39:52 +03:00
Alan Orth 58b7b6e9d8
Version 0.6.0
continuous-integration/drone/push Build is passing Details
2022-09-02 16:35:58 +03:00
Alan Orth ffdf1eca7b
setup.py: remove Python 3.7 support
I had already set the minimum to Python 3.8 elsewhere, but forgot
to do it here. I am not sure if Python 3.7 will still work here or
not so let's just keep it in sync with the other docs.
2022-09-02 16:34:16 +03:00
Alan Orth 59742e47f1
Update requirements
Generated with poetry export:

    $ poetry export --without-hashes -f requirements.txt > requirements.txt
    $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt

I am trying `--without-hashes` to work around an error on pip install
when running in CI:

    ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==
2022-09-02 16:32:04 +03:00
Alan Orth 9c741b1d49
poetry.lock: sync latest deps 2022-09-02 16:31:19 +03:00
Alan Orth 21e9948a75
pyproject.toml: manually updated all deps
Update all deps to their latest versions on pypi.org and remove the
explicit dependency on SQLAlchemy.
2022-09-02 16:30:40 +03:00
Alan Orth f64435fc9d
tests/test_check.py: add missing excludes 2022-09-02 16:24:33 +03:00
Alan Orth 566c2b45cf
Remove Excel support
I never used this and it seems xlrd doesn't even support .xlsx any-
more anyways. If this was needed I could theoretically use openpyxl
but I'd rather just stick to CSV.
2022-09-02 16:14:24 +03:00
Alan Orth 41b813be6e
CHANGELOG.md: add not about exclude logic 2022-09-02 16:03:51 +03:00
Alan Orth 040e56fc76
Improve exclude function
When a user explicitly requests that a field be excluded with -x we
skip that field in most checks. Up until now that did not include
the item-based checks using a transposed dataframe because we don't
know the metadata field names (labels) until we iterate over them.

Now the excludes are respected for item-based checks.
2022-09-02 15:59:22 +03:00
Alan Orth 1f76247353
csv_metadata_quality/app.py: rework exclude/skip
Instead of processing the excludes inside the for column loop we do
it once before and then only need to check if the current column is
in the list.
2022-09-02 10:35:04 +03:00
Alan Orth 2e489fc921
Add new data/test-geography.csv test file
continuous-integration/drone/push Build is passing Details
This file has metadata to test different scenarios related to chec-
king and fixing missing regions.
2022-09-01 16:57:29 +03:00
Alan Orth 117c6ca85d
csv_metadata_quality/check.py: missing region fixes
Port over the recent fixes and logic improvements to regions from
fix.py.
2022-09-01 16:38:35 +03:00
Alan Orth f49214fa2e
csv_metadata_quality/fix.py: fix bug in regions
We need to make sure we're only manipulating the regions if we have
any missing. The previous code was always manipulating the existing
row, even when there were no missing regions, which resulted in new
values like "Eastern Africa||".
2022-09-01 16:15:32 +03:00
Alan Orth 7ce20726d0
csv_metadata_quality/fix.py: minor change
Print missing regions when we know they are missing, instead of do-
ing another check later and looping over them again.
2022-09-01 16:03:49 +03:00
Alan Orth 473be5ac2f
csv_metadata_quality/fix.py: don't add "not found" region
country_converter returns the literal "not found" string if a coun-
try cannot be found. In that case we do not want to consider that as
a region!
2022-09-01 15:46:21 +03:00
Alan Orth 7c61cae417 csv_metadata_quality/fix.py: silence warning
By default country_converter prints "not found in regex" if a coun-
try is not found. We can silence this by switching the logging lev-
el to something above WARNING.
2022-09-01 15:44:50 +03:00
Alan Orth ae16289637
csv_metadata_quality/fix.py: Minor change
The country_converter documentation says we should instantiate the
CountryConverter() class once instead of calling coco.convert() in
each iteration of the loop so we don't end up loading the data file
more than once.
2022-09-01 15:40:45 +03:00
Alan Orth fdb7900cd0
Update requirements
continuous-integration/drone/push Build is passing Details
Generated with poetry export:

    $ poetry export --without-hashes -f requirements.txt > requirements.txt
    $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt

I am trying `--without-hashes` to work around an error on pip install
when running in CI:

    ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==
2022-09-01 11:21:10 +03:00
Alan Orth 9c65569c43
poetry.lock: run poetry update 2022-09-01 08:44:12 +03:00
Alan Orth 0cf0bc97f0
csv_metadata_quality/fix.py: fix logic error again
continuous-integration/drone/push Build is passing Details
It seems there was another logic error raised by the test in pytest.
With my real data, it was enough to check if the region column was
None, but with my test I was explicitly setting the region to "" (an
empty string). So to be really sure we should check if the string
is not None *and* if its length is greater than 0.
2022-08-03 20:51:14 +03:00
Alan Orth 40c3585bab
csv_metadata_quality/fix.py: fix logic error
Fix string concatenation with existing regions.
2022-08-03 18:26:08 +03:00
Alan Orth b9c44aed7d
csv_metadata_quality/fix.py: fix logic issue
continuous-integration/drone/push Build is passing Details
Forgot to return the row as-is if we don't find any countries.
2022-08-02 10:17:30 +03:00
Alan Orth 032a1db392
README.md: Add note about missing regions
continuous-integration/drone/push Build is passing Details
2022-07-28 16:58:01 +03:00
Alan Orth da87531779
CHANGELOG.md: Add note about adding missing regions 2022-07-28 16:54:05 +03:00
Alan Orth 689ee184f7
Add unsafe check to add missing regions 2022-07-28 16:52:43 +03:00
Alan Orth 344993370c
Update requirements
continuous-integration/drone/push Build is passing Details
Generated with poetry export:

    $ poetry export --without-hashes -f requirements.txt > requirements.txt
    $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt

I am trying `--without-hashes` to work around an error on pip install
when running in CI:

    ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==.
2022-07-08 15:50:42 +03:00
Alan Orth 00b4dca185
poetry.lock: run poetry update 2022-07-08 15:50:03 +03:00
Alan Orth 5a87bf4317
Update requirements
continuous-integration/drone/push Build is passing Details
Generated with poetry export:

    $ poetry export --without-hashes -f requirements.txt > requirements.txt
    $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt

I am trying `--without-hashes` to work around an error on pip install
when running in CI:

    ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==.
2022-03-21 14:37:38 +03:00
Alan Orth c706719d8b
poetry.lock: run poetry update 2022-03-21 14:37:03 +03:00
Alan Orth e7ea8ef9f0
README.md: add note about spdx-license-list
continuous-integration/drone/push Build is passing Details
This Python module was deprecated in favor of using the SPDX license
data directly.

See: https://github.com/spdx/license-list-data
2022-01-30 13:27:20 +03:00
Alan Orth ea050376fc
Update requirements
Generated with poetry export:

    $ poetry export --without-hashes -f requirements.txt > requirements.txt
    $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt

I am trying `--without-hashes` to work around an error on pip install
when running in CI:

    ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==.
2022-01-30 13:26:37 +03:00