59 Commits

Author	SHA1	Message	Date
renovate[bot]	df040b70c7	chore(deps): update dependency flake8 to v7	2024-01-05 00:58:28 +00:00
Alan Orth	a21ffb0fa8	Use py3langid instead of langid ... Faster and more modern code for Python 3 as a drop-in replacement. See: https://adrien.barbaresi.eu/blog/language-detection-langid-py-faster.html	2023-12-28 14:11:21 +03:00
Alan Orth	59b3b307c9	pyproject.toml: use official pycountry ... The project is moving again and has all the latest data from the iso-codes project.	2023-12-09 12:04:14 +03:00
Alan Orth	80c3f5b45a	Add fixit to dev dependencies	2023-11-22 21:54:09 +03:00
renovate[bot]	58d4de973e	Update dependency country-converter to ~1.1.0	2023-11-20 18:37:44 +00:00
Alan Orth	4ed2786703	pyproject.toml: update pycountry ... Use the latest branch in my fork that has iso-codes 4.15.0.	2023-10-15 21:53:09 +03:00
renovate[bot]	3632ae0fc9	Update dependency requests-cache to v1	2023-05-29 19:25:58 +00:00
Alan Orth	bc470a4343	pyproject.toml: rework pandas and pyarrow ... We don't explicitly depend on PyArrow. It should come as a pandas extra. I installed it like this: $ poetry add pandas=="^2.0.2[feather,performance]" See: https://pandas.pydata.org/docs/getting_started/install.html#other-data-sources	2023-05-29 22:24:04 +03:00
Alan Orth	2e55b4d6e3	pyproject.toml: add pyarrow explicitly ... CI was failing because pyarrow is not an extra provided by pandas. Indeed, according to the docs the named extras installing pyarrow are actually feather and parquet, so we need to install pyarrow explicitly. See: https://pandas.pydata.org/pandas-docs/version/2.0/getting_started/install.html#install-dependencies	2023-04-05 12:49:40 +03:00
Alan Orth	c90aad29f0	Use poetry dev group ... This is the new syntax since Poetry 1.2.0. See: https://python-poetry.org/docs/managing-dependencies/#installing-group-dependencies	2023-04-05 12:37:03 +03:00
Alan Orth	6fd1e1377f	Add pyarrow extra to Python Pandas deps	2023-04-05 11:40:22 +03:00
Alan Orth	b5106de9df	pyproject.toml: Pandas 2.0.0	2023-04-05 11:15:40 +03:00
Alan Orth	d4aed378cf	Switch to pandas 2.0.0rc1 ... Seems to work fine with the new PyArrow datatypes.	2023-03-22 12:16:56 +03:00
Alan Orth	ede37569f1	pyproject.toml: use pycountry with iso-codes 4.13.0	2023-03-04 07:33:48 +03:00
Alan Orth	4776154d6c	pyproject.toml: switch back to upstream country_converter ... Version 1.0.0 incorporates my change to Myanmar. See: https://github.com/IndEcol/country_converter/releases/tag/v1.0.0	2023-03-04 06:52:56 +03:00
Alan Orth	fdccdf7318	Version 0.6.1	2023-02-23 13:46:56 +03:00
Alan Orth	03d824b78e	pyproject.toml: update some dependencies	2023-02-18 22:09:05 +03:00
Alan Orth	66893753ba	Move isort config to pyproject.toml ... See: https://pycqa.github.io/isort/docs/configuration/black_compatibility.html	2022-12-20 15:03:10 +02:00
Alan Orth	2dfb073b6b	Update minimum Python version to 3.9 ... Due to importlib.resources.files. It's a very minor thing and there are ways to use back-ported third-party modules with this function- ality, but I'm the only one use this so... See: https://docs.python.org/3/library/importlib.resources.html#importlib.resources.files	2022-12-13 10:41:32 +03:00
Alan Orth	7cc49b500d	Use licenses.json from SPDX instead of spdx-license-list ... spdx-license-list has been deprecated[1] and already has outdated information compared to recent SPDX data releases. Now I use the JSON license data directly from SPDX[2] (currently version 3.19). The JSON file is loaded from the package's data directory using Python 3's stdlib functions from importlib[3], though we now need Python 3.9 as a minimum for importlib.resources.files[4]. Also note that the data directory is not properly packaged via setuptools, so this only works for local installs, and not via versions published to pypi, for example (I'm currently not doing this anyways). If I want to publish this in the future I will need to modify setup.py/pyproject.toml to include the data files. [1] https://gitlab.com/uniqx/spdx-license-list [2] https://github.com/spdx/license-list-data/blob/main/json/licenses.json [3] https://copdips.com/2022/09/adding-data-files-to-python-package-with-setup-py.html [4] https://docs.python.org/3/library/importlib.resources.html#importlib.resources.files	2022-12-13 10:39:17 +03:00
Alan Orth	7097136b7e	Use my fork of country_converter again ... There is an issue with the UN M.49 region for Myanmar.	2022-11-28 17:38:45 +03:00
Alan Orth	b02f1f65ee	pyproject.toml: use upstream country_converter ... Version 0.8.0 has the country and UN M.49 region fixes. See: https://github.com/konstantinstadler/country_converter/releases/tag/v0.8.0	2022-11-28 17:14:16 +03:00
Alan Orth	4d5ef38dde	pyproject.toml: add ipython to dev dependencies	2022-11-28 17:11:18 +03:00
Alan Orth	df57988e5a	Use my fork of pycountry ... Until they update to iso-codes 4.12.0. See: https://github.com/flyingcircusio/pycountry/pull/149	2022-11-08 10:21:28 +03:00
Alan Orth	15f52f8be8	Switch to my fork of country-converter ... Until a few issues are resolved regarding new countries and regions. See: https://github.com/konstantinstadler/country_converter/pull/122 See: https://github.com/konstantinstadler/country_converter/pull/123	2022-11-08 10:04:31 +03:00
Alan Orth	ca82820a8e	pyproject.toml: update dependencies to latest	2022-11-07 12:13:28 +03:00
Alan Orth	58b7b6e9d8	Version 0.6.0	2022-09-02 16:35:58 +03:00
Alan Orth	21e9948a75	pyproject.toml: manually updated all deps ... Update all deps to their latest versions on pypi.org and remove the explicit dependency on SQLAlchemy.	2022-09-02 16:30:40 +03:00
Alan Orth	566c2b45cf	Remove Excel support ... I never used this and it seems xlrd doesn't even support .xlsx any- more anyways. If this was needed I could theoretically use openpyxl but I'd rather just stick to CSV.	2022-09-02 16:14:24 +03:00
Alan Orth	b0d46cd864	pyproject.toml: update black ... It's no longer in beta!	2022-01-30 13:22:47 +03:00
Alan Orth	3ee9319d84	pyproject.toml: bump flake8	2022-01-30 13:21:09 +03:00
Alan Orth	4d5f4b5abb	pyproject.toml: update pycountry ... Seems to be a few major versions from 19.x.x to 21.x.x. All tests passing in pytest so it's probably fine.	2022-01-30 13:15:38 +03:00
Alan Orth	98d38801fa	pyproject.toml: update requests and requests-cache	2022-01-30 13:11:01 +03:00
Alan Orth	e94a4539bf	pyproject.toml: bump Pandas to v1.4.0 ... As of Pandas v1.4.0 the minimum Python version is 3.8. See: https://pandas.pydata.org/docs/whatsnew/v1.4.0.html	2022-01-30 13:03:56 +03:00
Alan Orth	d9e427a80e	pyproject.toml: don't install ipython ... It always complains about running in a virtual environment anyways, and I can use the one from the OS instead.	2022-01-29 16:25:58 +03:00
Alan Orth	8b15154285	pyproject.toml: use ftfy 6.0 ... Lots of improvements here! Improvements to heuristics and a new way to configure which fixes get applied. See: https://github.com/rspeer/python-ftfy/blob/master/CHANGELOG.md#version-60-april-2-2021	2021-12-15 21:48:56 +02:00
Alan Orth	9905e183ea	Bump version to 0.6.0-dev	2021-12-09 23:21:30 +02:00
Alan Orth	cc34db7ff8	Version 0.5.0	2021-12-08 15:29:46 +02:00
Alan Orth	ccc2a73456	Add check for countries without matching regions ... If we have country "Kenya" we should have region "Eastern Africa" according to the UN M.49 geolocation scheme.	2021-12-08 15:02:20 +02:00
Alan Orth	215d61c188	pyproject.toml: limit SQLAlchemy to < 1.4.23 ... SQLAlchemy gets pulled in by csvkit's agate-sql dependency and there is currently an issue with Poetry's parsing of the SQLAlchemy 1.4.23 constraints. Temporarily explicitly install a version of SQLAlchemy that works (can remove later once Poetry fixes this). Anyways, I am not using any SQLAlchemy features that I know of. See: https://github.com/python-poetry/poetry/issues/4402	2021-09-06 21:01:09 +03:00
Alan Orth	b8f4be9ebb	pyproject.toml: Update pytest-clarity and black ... These seem to have much newer versions that didn't get updated in this project due to the version pinning selector I was using with poetry. In the case of pytest-clarity the previous version was 0.3.1 and the version selector was a caret (^), which will never update the left-most (major) number. Now they seem to be on 1.x.x so it will be OK in the future. In the case of black, they use weird numbering so it's anyone's guess how this will work! Luckily it's only used for linting and formatting.	2021-07-06 15:30:41 +03:00
Alan Orth	4e2eab68b0	Update requests-cache ... Apparently we were stuck on an older version of requests-cache due to the fact that we were using the caret, which will never update the left-most (major) version. Upstream requests-cache is currently version 0.6.4, and there seems to have been some changes to the API.	2021-07-06 15:24:39 +03:00
Alan Orth	8eddb76aab	Bump version to 0.4.8-dev	2021-03-19 11:53:56 +02:00
Alan Orth	898bb412c3	Add checks and unsafe fixes for mojibake ... This detects whether text has likely been encoded in one encoding and decoded in another, perhaps multiple times. This often results in display of "mojibake" characters. For example, a file encoded in UTF-8 is opened as CP-1252 (Windows Latin codepage) in Microsoft Excel, and saved again as UTF-8. You will see strings like this in the resulting file: - CIAT Publicaçao - CIAT Publicación The correct version of these in UTF-8 would be: - CIAT Publicaçao - CIAT Publicación I use a code snippet from Martijn Pieters on StackOverflow to de- tect whether a string is "weird" as determined by the excellent "fixes text for you" (ftfy) Python library, then check if a weird string encodes as CP-1252 or not. If so, I can try to fix it. See: https://stackoverflow.com/questions/29071995/identify-garbage-unicode-string-using-python	2021-03-19 10:22:21 +02:00
Alan Orth	f816e17fe7	Version 0.4.7	2021-03-17 10:00:34 +02:00
Alan Orth	fa84cfa440	Bump version to 0.4.6-dev	2021-03-11 22:44:36 +02:00
Alan Orth	6cc1401f88	pyproject.toml: Minimum Python is technically 3.7.1 ... See: https://pandas.pydata.org/pandas-docs/stable/whatsnew/v1.2.0.html	2021-03-11 13:41:58 +02:00
Alan Orth	1554cfd5c9	Version 0.4.6	2021-03-11 12:14:54 +02:00
Alan Orth	6e4b0e5c1b	Add validation of SPDX license identifiers ... Currently this only checks the dcterms.license field and the result will only be a warning.	2021-03-11 10:33:16 +02:00
Alan Orth	b16fa9121f	pyproject.toml: Add csv-metadata-quality as a script ... For some reason I stopped having csv-metadata-quality available in my poetry environment after install. It seems I need to add it as a poetry tool script? I had already done this in setup.py years ago, which works for regular python setup.py installs, but hadn't needed to do it in poetry for a year or more that I've been using it, until now.	2021-03-08 09:50:05 +02:00