alanorth
/
csv-metadata-quality
mirror of https://github.com/ilri/csv-metadata-quality.git
1
0
Fork 0
Code Releases Activity
619 Commits 7 Branches 17 Tags 5.4 MiB
Commit Graph

619 Commits

Author SHA1 Message Date
Alan Orth 4ed2786703
pyproject.toml: update pycountry 
Use the latest branch in my fork that has iso-codes 4.15.0.
 2023-10-15 21:53:09 +03:00
renovate[bot] 8728789183
Update actions/checkout action to v4
continuous-integration/drone/push Build is passing Details
2023-09-04 14:26:25 +00:00
Alan Orth bf90464809
poetry.lock: run poetry update
continuous-integration/drone/push Build is failing Details
continuous-integration/drone Build is passing Details
2023-08-08 09:55:41 +02:00
Alan Orth 1878002391 poetry.lock: run poetry update
continuous-integration/drone/push Build is passing Details
2023-06-12 10:42:50 +03:00
Alan Orth d21d2621e3 csv_metadata_quality/app.py: read fields as strings 
I suspect this undermines the PyArrow backend performance gains in
recent Pandas 2.0.0, but we are dealing with messy data sometimes
and we must rely on data being strings.
 2023-06-12 10:42:50 +03:00
Alan Orth f3fb1ff7fb Don't crash when title is missing 
We shouldn't crash the country/region checker/fixer when the title
field is missing, since we only use it to show status to the user.
 2023-06-12 10:42:50 +03:00
Alan Orth 1fa81f7558
Merge pull request #13 from ilri/renovate/ipython-8.x-lockfile
continuous-integration/drone/push Build is passing Details 
Update dependency ipython to v8.14.0
 2023-06-03 17:09:21 +03:00
renovate[bot] 7409193b6b
Update dependency ipython to v8.14.0
continuous-integration/drone/push Build is passing Details
2023-06-02 15:58:34 +00:00
Alan Orth a84fcf0b7b
.drone.yml: try to use poetry instead of pip
continuous-integration/drone/push Build is passing Details
2023-05-30 11:39:08 +03:00
Alan Orth 25ac290df4
.github: update Python actions
continuous-integration/drone/push Build is failing Details 
We don't need to use `python setup.py install` anymore. We can use
poetry directly in CI.

See: https://github.com/actions/setup-python/blob/main/docs/advanced-usage.md
 2023-05-29 22:58:01 +03:00
Alan Orth 3f52bad1e3
Remove setup.py 
As far as I understand this is deprecated.
 2023-05-29 22:41:37 +03:00
Alan Orth 0208ad0ade
Merge pull request #12 from ilri/renovate/requests-cache-1.x 
Update dependency requests-cache to v1
 2023-05-29 22:37:23 +03:00
renovate[bot] 3632ae0fc9
Update dependency requests-cache to v1
continuous-integration/drone/push Build is passing Details
2023-05-29 19:25:58 +00:00
Alan Orth 17d089cc6e
poetry.lock: run poetry update
continuous-integration/drone/push Build is passing Details
2023-05-29 22:24:22 +03:00
Alan Orth bc470a4343
pyproject.toml: rework pandas and pyarrow 
We don't explicitly depend on PyArrow. It should come as a pandas
extra. I installed it like this:

    $ poetry add pandas=="^2.0.2[feather,performance]"

See: https://pandas.pydata.org/docs/getting_started/install.html#other-data-sources
 2023-05-29 22:24:04 +03:00
Alan Orth be609a809d
setup.py: add Python 3.11 classifier 2023-05-29 21:32:59 +03:00
Alan Orth de3387ded7
Use Python 3.11 in Drone CI and GitHub Actions 2023-05-29 21:31:03 +03:00
Alan Orth f343e87f0c
renovate.json: fix json 2023-05-29 21:26:03 +03:00
Alan Orth 7d3524fbd5
renovate.json: disable requirements.txt support 
Poetry is used to manage dependencies. The requirements.txt files
are generated manually by exporting from Poetry.
 2023-05-29 21:11:48 +03:00
Alan Orth c614b71a52
Merge pull request #5 from ilri/renovate/configure 
Configure Renovate
 2023-05-29 21:02:16 +03:00
renovate[bot] d159a839f3
Add renovate.json 2023-05-29 17:40:33 +00:00
Alan Orth 36e2ebe5f4
poetry.lock: run poetry update
continuous-integration/drone/push Build is passing Details
2023-05-10 15:06:41 +03:00
Alan Orth 33f67b7a7c
Update requirements
continuous-integration/drone/push Build is passing Details 
Generated with poetry export:

    $ poetry export --without-hashes -f requirements.txt > requirements.txt
    $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt

I am trying `--without-hashes` to work around an error on pip install
when running in CI:

    ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==
 2023-05-03 14:29:12 +03:00
Alan Orth c0e1448439
poetry.lock: run poetry update 2023-05-03 14:28:47 +03:00
Alan Orth 5d0804a08f
Update requirements
continuous-integration/drone/push Build is failing Details 
Generated with poetry export:

    $ poetry export --without-hashes -f requirements.txt > requirements.txt
    $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt

I am trying `--without-hashes` to work around an error on pip install
when running in CI:

    ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==
 2023-04-22 12:44:54 -07:00
Alan Orth f01c9edf17
poetry.lock: run poetry update 2023-04-22 12:44:16 -07:00
Alan Orth 8d4295b2b3
CHANGELOG.md: add note about description field 2023-04-22 12:17:44 -07:00
Alan Orth e2d46e9495
csv_metadata_quality/app.py: skip newline fix on description 
The description field often has free-form text like the abstract and
there are too many legitimate newlines here to be correcting them
automatically.
 2023-04-22 12:16:13 -07:00
Alan Orth 1491e1edb0
Fix path to data/licenses.json
continuous-integration/drone/push Build is passing Details 
When we install and run this from CI, this file needs to exist in
the package's folder inside site-packages. Then we can use __file__
to get the path relative to the package.

See: https://python-packaging.readthedocs.io/en/latest/non-code-files.html
 2023-04-05 15:28:21 +03:00
Alan Orth 34142c3e6b
Update requirements
continuous-integration/drone/push Build is failing Details 
Generated with poetry export:

    $ poetry export --without-hashes -f requirements.txt > requirements.txt
    $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt

I am trying `--without-hashes` to work around an error on pip install
when running in CI:

    ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==
 2023-04-05 12:51:56 +03:00
Alan Orth 0c88b96e8d
poetry.lock: run poetry update 2023-04-05 12:51:19 +03:00
Alan Orth 2e55b4d6e3
pyproject.toml: add pyarrow explicitly 
CI was failing because pyarrow is not an extra provided by pandas.
Indeed, according to the docs the named extras installing pyarrow
are actually feather and parquet, so we need to install pyarrow
explicitly.

See: https://pandas.pydata.org/pandas-docs/version/2.0/getting_started/install.html#install-dependencies
 2023-04-05 12:49:40 +03:00
Alan Orth c90aad29f0
Use poetry dev group 
This is the new syntax since Poetry 1.2.0.

See: https://python-poetry.org/docs/managing-dependencies/#installing-group-dependencies
 2023-04-05 12:37:03 +03:00
Alan Orth 6fd1e1377f
Add pyarrow extra to Python Pandas deps 2023-04-05 11:40:22 +03:00
Alan Orth c64b7eb1f1
CHANGELOG.md: add note about Pandas 2.0.0 2023-04-05 11:17:48 +03:00
Alan Orth 29cbc4f3a3
Update requirements 
Generated with poetry export:

    $ poetry export --without-hashes -f requirements.txt > requirements.txt
    $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt

I am trying `--without-hashes` to work around an error on pip install
when running in CI:

    ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==
 2023-04-05 11:17:06 +03:00
Alan Orth 307af1acfc
poetry.lock: run poetry update 2023-04-05 11:15:55 +03:00
Alan Orth b5106de9df
pyproject.toml: Pandas 2.0.0 2023-04-05 11:15:40 +03:00
Alan Orth 9eeadfc44e
poetry.lock: after adding pandas 2.0.0rc1
continuous-integration/drone/push Build is failing Details 
This is going to be an issue on the master branch if I update any
dependencies in the mean time...
 2023-03-22 12:17:26 +03:00
Alan Orth d4aed378cf
Switch to pandas 2.0.0rc1 
Seems to work fine with the new PyArrow datatypes.
 2023-03-22 12:16:56 +03:00
Alan Orth 20a2cce34b
CHANGELOG.md: add fixes
continuous-integration/drone/push Build is failing Details
2023-03-10 16:17:20 +03:00
Alan Orth d661ffe439
Check comma space on bibliographicCitation too 
The regex was only matching `dc.identifier.citation`, but we need
to match `dcterms.bibliographicCitation` too.
 2023-03-10 16:13:16 +03:00
Alan Orth 45a310387a
Don't fix multi-value separators on citations 2023-03-10 16:12:30 +03:00
Alan Orth 47b03c49ba
README.md: Update TODOs
continuous-integration/drone/push Build is failing Details
2023-03-07 10:45:04 +03:00
Alan Orth 986b81cbf4
Update requirements
continuous-integration/drone/push Build is failing Details 
Generated with poetry export:

    $ poetry export --without-hashes -f requirements.txt > requirements.txt
    $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt

I am trying `--without-hashes` to work around an error on pip install
when running in CI:

    ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==
 2023-03-04 07:35:36 +03:00
Alan Orth d43a47ae32
poetry.lock: run poetry update 2023-03-04 07:34:50 +03:00
Alan Orth ede37569f1
pyproject.toml: use pycountry with iso-codes 4.13.0 2023-03-04 07:33:48 +03:00
Alan Orth 0c53efe60a
Update requirements 
Generated with poetry export:

    $ poetry export --without-hashes -f requirements.txt > requirements.txt
    $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt

I am trying `--without-hashes` to work around an error on pip install
when running in CI:

    ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==
 2023-03-04 06:54:34 +03:00
Alan Orth 5f0e25b818
poetry.lock: run poetry update 2023-03-04 06:53:55 +03:00
Alan Orth 4776154d6c
pyproject.toml: switch back to upstream country_converter 
Version 1.0.0 incorporates my change to Myanmar.

See: https://github.com/IndEcol/country_converter/releases/tag/v1.0.0
 2023-03-04 06:52:56 +03:00