542 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Alan Orth	e2d46e9495	csv_metadata_quality/app.py: skip newline fix on description ... The description field often has free-form text like the abstract and there are too many legitimate newlines here to be correcting them automatically.	2023-04-22 12:16:13 -07:00
Alan Orth	1491e1edb0	Fix path to data/licenses.json ... When we install and run this from CI, this file needs to exist in the package's folder inside site-packages. Then we can use __file__ to get the path relative to the package. See: https://python-packaging.readthedocs.io/en/latest/non-code-files.html	2023-04-05 15:28:21 +03:00
Alan Orth	34142c3e6b	Update requirements ... Generated with poetry export: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt I am trying `--without-hashes` to work around an error on pip install when running in CI: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==	2023-04-05 12:51:56 +03:00
Alan Orth	0c88b96e8d	poetry.lock: run poetry update	2023-04-05 12:51:19 +03:00
Alan Orth	2e55b4d6e3	pyproject.toml: add pyarrow explicitly ... CI was failing because pyarrow is not an extra provided by pandas. Indeed, according to the docs the named extras installing pyarrow are actually feather and parquet, so we need to install pyarrow explicitly. See: https://pandas.pydata.org/pandas-docs/version/2.0/getting_started/install.html#install-dependencies	2023-04-05 12:49:40 +03:00
Alan Orth	c90aad29f0	Use poetry dev group ... This is the new syntax since Poetry 1.2.0. See: https://python-poetry.org/docs/managing-dependencies/#installing-group-dependencies	2023-04-05 12:37:03 +03:00
Alan Orth	6fd1e1377f	Add pyarrow extra to Python Pandas deps	2023-04-05 11:40:22 +03:00
Alan Orth	c64b7eb1f1	CHANGELOG.md: add note about Pandas 2.0.0	2023-04-05 11:17:48 +03:00
Alan Orth	29cbc4f3a3	Update requirements ... Generated with poetry export: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt I am trying `--without-hashes` to work around an error on pip install when running in CI: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==	2023-04-05 11:17:06 +03:00
Alan Orth	307af1acfc	poetry.lock: run poetry update	2023-04-05 11:15:55 +03:00
Alan Orth	b5106de9df	pyproject.toml: Pandas 2.0.0	2023-04-05 11:15:40 +03:00
Alan Orth	9eeadfc44e	poetry.lock: after adding pandas 2.0.0rc1 ... This is going to be an issue on the master branch if I update any dependencies in the mean time...	2023-03-22 12:17:26 +03:00
Alan Orth	d4aed378cf	Switch to pandas 2.0.0rc1 ... Seems to work fine with the new PyArrow datatypes.	2023-03-22 12:16:56 +03:00
Alan Orth	20a2cce34b	CHANGELOG.md: add fixes	2023-03-10 16:17:20 +03:00
Alan Orth	d661ffe439	Check comma space on bibliographicCitation too ... The regex was only matching `dc.identifier.citation`, but we need to match `dcterms.bibliographicCitation` too.	2023-03-10 16:13:16 +03:00
Alan Orth	45a310387a	Don't fix multi-value separators on citations	2023-03-10 16:12:30 +03:00
Alan Orth	47b03c49ba	README.md: Update TODOs	2023-03-07 10:45:04 +03:00
Alan Orth	986b81cbf4	Update requirements ... Generated with poetry export: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt I am trying `--without-hashes` to work around an error on pip install when running in CI: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==	2023-03-04 07:35:36 +03:00
Alan Orth	d43a47ae32	poetry.lock: run poetry update	2023-03-04 07:34:50 +03:00
Alan Orth	ede37569f1	pyproject.toml: use pycountry with iso-codes 4.13.0	2023-03-04 07:33:48 +03:00
Alan Orth	0c53efe60a	Update requirements ... Generated with poetry export: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt I am trying `--without-hashes` to work around an error on pip install when running in CI: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==	2023-03-04 06:54:34 +03:00
Alan Orth	5f0e25b818	poetry.lock: run poetry update	2023-03-04 06:53:55 +03:00
Alan Orth	4776154d6c	pyproject.toml: switch back to upstream country_converter ... Version 1.0.0 incorporates my change to Myanmar. See: https://github.com/IndEcol/country_converter/releases/tag/v1.0.0	2023-03-04 06:52:56 +03:00
Alan Orth	fdccdf7318	Version 0.6.1	2023-02-23 13:46:56 +03:00
Alan Orth	ff2c986eec	setup.py: minimum python 3.9	2023-02-23 11:47:40 +03:00
Alan Orth	547574866e	Update requirements ... Generated with poetry export: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt I am trying `--without-hashes` to work around an error on pip install when running in CI: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==	2023-02-23 11:46:24 +03:00
Alan Orth	8aa7b93d87	poetry.lock: run poetry update	2023-02-23 11:45:53 +03:00
Alan Orth	53fdb50906	csv_metadata_quality/check.py: run black	2023-02-18 22:10:04 +03:00
Alan Orth	3e0e9a7f8b	poetry.lock: run poetry update	2023-02-18 22:09:33 +03:00
Alan Orth	03d824b78e	pyproject.toml: update some dependencies	2023-02-18 22:09:05 +03:00
Alan Orth	8bc4cd419c	Strip filename descriptions before checking ... When checking for uncommon file extensions in the filename field we should strip descriptions that are meant for SAF Bundler, for example: Annual_Report_2020.pdf__description:Report. This ends up as a false positive that spams the output with warnings.	2023-02-13 11:00:57 +03:00
Alan Orth	bde38e9ed4	CHANGELOG.md: add notes about abstracts	2023-02-13 10:39:03 +03:00
Alan Orth	8db1e36a6d	csv_metadata_quality/app.py: skip abstract in separator check ... Also skip abstract in the separator check, since it's rare to have any "|" here, but more likely that if one is present then it's for a reason.	2023-02-13 10:37:33 +03:00
Alan Orth	fbb625be5c	Ignore common non-SPDX licenses ... This is meant to catch licenses that are supposed to be SPDX but aren't, not licenses that *aren't* supposed to be SPDX. We have so many free-text license descriptions like "Copyrighted" and "Other" that I'm sick of seeing warnings for them!	2023-02-07 17:01:56 +03:00
Alan Orth	084b970798	CHANGELOG.md: add note about abstract field	2023-02-07 16:52:34 +03:00
Alan Orth	171b35b015	Add data/abstract-check.csv ... A test file with several whitespace and newline scenarios in the abstract. I am currently disabling whitespace/newline fixes in the abstract because they are too agressive.	2023-02-07 16:50:47 +03:00
Alan Orth	545bb8cd0c	csv_metadata_quality/app.py: disable whitespace on abstracts ... It's too aggressive on abstracts. If people paste in text from a PDF there are often newlines, and most of the time this is what they want.	2023-02-07 16:48:40 +03:00
Alan Orth	d5afbad788	Update requirements ... Generated with poetry export: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt I am trying `--without-hashes` to work around an error on pip install when running in CI: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==	2023-01-24 14:18:19 +03:00
Alan Orth	d40c9ed97a	poetry.lock: run poetry update	2023-01-24 14:17:44 +03:00
Alan Orth	c4a2ee8563	CHANGELOG.md: add note about fix.separators()	2023-01-24 14:16:23 +03:00
Alan Orth	3596381d03	csv_metadata_quality/app.py: separators fix ... Don't run the invalid separators fix on title fields because some items use "|" in the title to indicate something like a subtitle. For example: Progress Review and Work Planning Meeting | Day 1	2023-01-24 14:13:55 +03:00
Alan Orth	5abd32a41f	CHANGELOG.md: run poetry update	2022-12-20 15:09:58 +02:00
Alan Orth	0ed0fabe21	tests/test_check.py: remove local variables ... This was raised by ruff. > F841 Local variable `result` is assigned to but never used We don't actually need the output of the function since these tests capture the stdout.	2022-12-20 15:09:20 +02:00
Alan Orth	d5cfec65bd	tests/test_check.py: fix logic in assert ... This was raised by ruff. > E711 Comparison to `None` should be `cond is None`	2022-12-20 15:07:41 +02:00
Alan Orth	66893753ba	Move isort config to pyproject.toml ... See: https://pycqa.github.io/isort/docs/configuration/black_compatibility.html	2022-12-20 15:03:10 +02:00
Alan Orth	57be05ebb6	poetry.lock: run poetry update	2022-12-20 14:59:35 +02:00
Alan Orth	8c23382b22	Update requirements ... Generated with poetry export: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --with dev -f requirements.txt > requirements-dev.txt I am trying `--without-hashes` to work around an error on pip install when running in CI: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==	2022-12-13 10:47:16 +03:00
Alan Orth	f640161d87	CHANGELOG.md: add notes about SPDX and Python	2022-12-13 10:45:36 +03:00
Alan Orth	e81ae93bf0	poetry.lock: run poetry update	2022-12-13 10:44:06 +03:00
Alan Orth	50ea5863dd	.drone.yml: only test on Python 3.9+	2022-12-13 10:43:18 +03:00