535 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Alan Orth	c64b7eb1f1	CHANGELOG.md: add note about Pandas 2.0.0	2023-04-05 11:17:48 +03:00
Alan Orth	29cbc4f3a3	Update requirements ... Generated with poetry export: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt I am trying `--without-hashes` to work around an error on pip install when running in CI: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==	2023-04-05 11:17:06 +03:00
Alan Orth	307af1acfc	poetry.lock: run poetry update	2023-04-05 11:15:55 +03:00
Alan Orth	b5106de9df	pyproject.toml: Pandas 2.0.0	2023-04-05 11:15:40 +03:00
Alan Orth	9eeadfc44e	poetry.lock: after adding pandas 2.0.0rc1 ... This is going to be an issue on the master branch if I update any dependencies in the mean time...	2023-03-22 12:17:26 +03:00
Alan Orth	d4aed378cf	Switch to pandas 2.0.0rc1 ... Seems to work fine with the new PyArrow datatypes.	2023-03-22 12:16:56 +03:00
Alan Orth	20a2cce34b	CHANGELOG.md: add fixes	2023-03-10 16:17:20 +03:00
Alan Orth	d661ffe439	Check comma space on bibliographicCitation too ... The regex was only matching `dc.identifier.citation`, but we need to match `dcterms.bibliographicCitation` too.	2023-03-10 16:13:16 +03:00
Alan Orth	45a310387a	Don't fix multi-value separators on citations	2023-03-10 16:12:30 +03:00
Alan Orth	47b03c49ba	README.md: Update TODOs	2023-03-07 10:45:04 +03:00
Alan Orth	986b81cbf4	Update requirements ... Generated with poetry export: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt I am trying `--without-hashes` to work around an error on pip install when running in CI: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==	2023-03-04 07:35:36 +03:00
Alan Orth	d43a47ae32	poetry.lock: run poetry update	2023-03-04 07:34:50 +03:00
Alan Orth	ede37569f1	pyproject.toml: use pycountry with iso-codes 4.13.0	2023-03-04 07:33:48 +03:00
Alan Orth	0c53efe60a	Update requirements ... Generated with poetry export: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt I am trying `--without-hashes` to work around an error on pip install when running in CI: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==	2023-03-04 06:54:34 +03:00
Alan Orth	5f0e25b818	poetry.lock: run poetry update	2023-03-04 06:53:55 +03:00
Alan Orth	4776154d6c	pyproject.toml: switch back to upstream country_converter ... Version 1.0.0 incorporates my change to Myanmar. See: https://github.com/IndEcol/country_converter/releases/tag/v1.0.0	2023-03-04 06:52:56 +03:00
Alan Orth	fdccdf7318	Version 0.6.1	2023-02-23 13:46:56 +03:00
Alan Orth	ff2c986eec	setup.py: minimum python 3.9	2023-02-23 11:47:40 +03:00
Alan Orth	547574866e	Update requirements ... Generated with poetry export: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt I am trying `--without-hashes` to work around an error on pip install when running in CI: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==	2023-02-23 11:46:24 +03:00
Alan Orth	8aa7b93d87	poetry.lock: run poetry update	2023-02-23 11:45:53 +03:00
Alan Orth	53fdb50906	csv_metadata_quality/check.py: run black	2023-02-18 22:10:04 +03:00
Alan Orth	3e0e9a7f8b	poetry.lock: run poetry update	2023-02-18 22:09:33 +03:00
Alan Orth	03d824b78e	pyproject.toml: update some dependencies	2023-02-18 22:09:05 +03:00
Alan Orth	8bc4cd419c	Strip filename descriptions before checking ... When checking for uncommon file extensions in the filename field we should strip descriptions that are meant for SAF Bundler, for example: Annual_Report_2020.pdf__description:Report. This ends up as a false positive that spams the output with warnings.	2023-02-13 11:00:57 +03:00
Alan Orth	bde38e9ed4	CHANGELOG.md: add notes about abstracts	2023-02-13 10:39:03 +03:00
Alan Orth	8db1e36a6d	csv_metadata_quality/app.py: skip abstract in separator check ... Also skip abstract in the separator check, since it's rare to have any "|" here, but more likely that if one is present then it's for a reason.	2023-02-13 10:37:33 +03:00
Alan Orth	fbb625be5c	Ignore common non-SPDX licenses ... This is meant to catch licenses that are supposed to be SPDX but aren't, not licenses that *aren't* supposed to be SPDX. We have so many free-text license descriptions like "Copyrighted" and "Other" that I'm sick of seeing warnings for them!	2023-02-07 17:01:56 +03:00
Alan Orth	084b970798	CHANGELOG.md: add note about abstract field	2023-02-07 16:52:34 +03:00
Alan Orth	171b35b015	Add data/abstract-check.csv ... A test file with several whitespace and newline scenarios in the abstract. I am currently disabling whitespace/newline fixes in the abstract because they are too agressive.	2023-02-07 16:50:47 +03:00
Alan Orth	545bb8cd0c	csv_metadata_quality/app.py: disable whitespace on abstracts ... It's too aggressive on abstracts. If people paste in text from a PDF there are often newlines, and most of the time this is what they want.	2023-02-07 16:48:40 +03:00
Alan Orth	d5afbad788	Update requirements ... Generated with poetry export: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt I am trying `--without-hashes` to work around an error on pip install when running in CI: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==	2023-01-24 14:18:19 +03:00
Alan Orth	d40c9ed97a	poetry.lock: run poetry update	2023-01-24 14:17:44 +03:00
Alan Orth	c4a2ee8563	CHANGELOG.md: add note about fix.separators()	2023-01-24 14:16:23 +03:00
Alan Orth	3596381d03	csv_metadata_quality/app.py: separators fix ... Don't run the invalid separators fix on title fields because some items use "|" in the title to indicate something like a subtitle. For example: Progress Review and Work Planning Meeting | Day 1	2023-01-24 14:13:55 +03:00
Alan Orth	5abd32a41f	CHANGELOG.md: run poetry update	2022-12-20 15:09:58 +02:00
Alan Orth	0ed0fabe21	tests/test_check.py: remove local variables ... This was raised by ruff. > F841 Local variable `result` is assigned to but never used We don't actually need the output of the function since these tests capture the stdout.	2022-12-20 15:09:20 +02:00
Alan Orth	d5cfec65bd	tests/test_check.py: fix logic in assert ... This was raised by ruff. > E711 Comparison to `None` should be `cond is None`	2022-12-20 15:07:41 +02:00
Alan Orth	66893753ba	Move isort config to pyproject.toml ... See: https://pycqa.github.io/isort/docs/configuration/black_compatibility.html	2022-12-20 15:03:10 +02:00
Alan Orth	57be05ebb6	poetry.lock: run poetry update	2022-12-20 14:59:35 +02:00
Alan Orth	8c23382b22	Update requirements ... Generated with poetry export: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt I am trying `--without-hashes` to work around an error on pip install when running in CI: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==	2022-12-13 10:47:16 +03:00
Alan Orth	f640161d87	CHANGELOG.md: add notes about SPDX and Python	2022-12-13 10:45:36 +03:00
Alan Orth	e81ae93bf0	poetry.lock: run poetry update	2022-12-13 10:44:06 +03:00
Alan Orth	50ea5863dd	.drone.yml: only test on Python 3.9+	2022-12-13 10:43:18 +03:00
Alan Orth	2dfb073b6b	Update minimum Python version to 3.9 ... Due to importlib.resources.files. It's a very minor thing and there are ways to use back-ported third-party modules with this function- ality, but I'm the only one use this so... See: https://docs.python.org/3/library/importlib.resources.html#importlib.resources.files	2022-12-13 10:41:32 +03:00
Alan Orth	7cc49b500d	Use licenses.json from SPDX instead of spdx-license-list ... spdx-license-list has been deprecated[1] and already has outdated information compared to recent SPDX data releases. Now I use the JSON license data directly from SPDX[2] (currently version 3.19). The JSON file is loaded from the package's data directory using Python 3's stdlib functions from importlib[3], though we now need Python 3.9 as a minimum for importlib.resources.files[4]. Also note that the data directory is not properly packaged via setuptools, so this only works for local installs, and not via versions published to pypi, for example (I'm currently not doing this anyways). If I want to publish this in the future I will need to modify setup.py/pyproject.toml to include the data files. [1] https://gitlab.com/uniqx/spdx-license-list [2] https://github.com/spdx/license-list-data/blob/main/json/licenses.json [3] https://copdips.com/2022/09/adding-data-files-to-python-package-with-setup-py.html [4] https://docs.python.org/3/library/importlib.resources.html#importlib.resources.files	2022-12-13 10:39:17 +03:00
Alan Orth	051777bcec	Ignore subregion field for missing region checks ... Due to a sloppy regex I was sometimes matching the subregion field when checking for missing UN M.49 regions in the region field.	2022-12-07 23:18:47 +01:00
Alan Orth	58e956360a	Add tests/test_check.py: fix test	2022-11-28 22:12:17 +03:00
Alan Orth	3532175748	.drone.yml: install git ... Apparently the slim images don't come with git, which we need for cloning some dependencies.	2022-11-28 22:05:34 +03:00
Alan Orth	a7bc929af8	Update requirements ... Generated with poetry export: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt I am trying `--without-hashes` to work around an error on pip install when running in CI: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==	2022-11-28 17:42:26 +03:00
Alan Orth	141b2e1da3	csv_metadata_quality/check.py: update region output ... Add the country to the message about missing regions. This makes it easier to see which country is triggering the missing region error, and helps in case of debugging possible mistakes in the data coming from the country_converter library.	2022-11-28 17:40:27 +03:00