620 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Alan Orth	a7727b8431	Add support for dropping invalid AGROVOC terms ... Requires --agrovoc-fields <field.name> to do the actual validation, and -d to drop invalid ones.	2021-12-23 12:43:55 +02:00
Alan Orth	7763a021c5	csv_metadata_quality/fix.py: sort imports with isort	2021-12-15 23:15:02 +02:00
Alan Orth	3c12ef3f66	Update requirements ... Generated with poetry export: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt I am trying `--without-hashes` to work around an error on pip install when running in CI: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==.	2021-12-15 23:11:44 +02:00
Alan Orth	aee2438e94	poetry.lock: run poetry update	2021-12-15 23:10:27 +02:00
Alan Orth	a351ba9706	CHANGELOG.md: add notes about ftfy	2021-12-15 22:09:01 +02:00
Alan Orth	e4faf114dc	csv_metadata_quality/util.py: update for ftfy 6.0 ... The sequence_weirdness() heuristic is deprecated. Now we should use is_bad(). See: https://ftfy.readthedocs.io/en/v6.0/heuristic.html See: https://github.com/rspeer/python-ftfy/blob/master/CHANGELOG.md#version-60-april-2-2021	2021-12-15 21:58:07 +02:00
Alan Orth	ff49a80432	csv_metadata_quality/fix.py: configure ftfy ... Don't replace smart quotes in ftfy. If our text has them we should keep them.	2021-12-15 21:51:51 +02:00
Alan Orth	8b15154285	pyproject.toml: use ftfy 6.0 ... Lots of improvements here! Improvements to heuristics and a new way to configure which fixes get applied. See: https://github.com/rspeer/python-ftfy/blob/master/CHANGELOG.md#version-60-april-2-2021	2021-12-15 21:48:56 +02:00
Alan Orth	5854f8e865	CHANGELOG.md: add note about unnecessary Unicode	2021-12-15 13:56:31 +02:00
Alan Orth	e7322efadd	csv_metadata_quality/app.py: move unnecessary Unicode fix ... We actually want to do this after we try to fix mojibake with ftfy. These "unnecessary" Unicode characters could actually help ftfy in some cases because often times they indicate that some character from another encoding was there before (like an accent, dash, or smart quote).	2021-12-15 13:53:25 +02:00
Alan Orth	95015febbd	csv_metadata_quality/fix.py: fix thin spaces ... Replace thin spaces with normal spaces. Sometimes I see these get mis handled on Windows machines and they end up as "?" or so.	2021-12-09 23:22:53 +02:00
Alan Orth	cef6c66b30	CHANGELOG.md: start next changes	2021-12-09 23:21:58 +02:00
Alan Orth	9905e183ea	Bump version to 0.6.0-dev	2021-12-09 23:21:30 +02:00
Alan Orth	cc34db7ff8	Version 0.5.0	2021-12-08 15:29:46 +02:00
Alan Orth	b79e07b814	CHANGELOG.md: Add note about countries without regions	2021-12-08 15:21:45 +02:00
Alan Orth	865b950c33	Update requirements ... Generated with poetry export: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt I am trying `--without-hashes` to work around an error on pip install when running in CI: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==.	2021-12-08 15:20:22 +02:00
Alan Orth	6f269ca6b1	poetry.lock: run poetry update	2021-12-08 15:19:49 +02:00
Alan Orth	120e8cf09f	tests/test_check.py: add checks for countries without regions	2021-12-08 15:18:50 +02:00
Alan Orth	a4eb79f625	data/test.csv: add data for countries without regions check	2021-12-08 15:17:55 +02:00
Alan Orth	ccc2a73456	Add check for countries without matching regions ... If we have country "Kenya" we should have region "Eastern Africa" according to the UN M.49 geolocation scheme.	2021-12-08 15:02:20 +02:00
Alan Orth	ad33195ba3	README.md: adjust intro ... Makes the badges not wrap and looks better in my opinion.	2021-12-08 11:36:34 +02:00
Alan Orth	72fe38972e	Update requirements ... Generated with poetry export: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt I am trying `--without-hashes` to work around an error on pip install when running in CI: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==.	2021-12-05 16:29:37 +02:00
Alan Orth	04232d0ede	poetry.lock: run poetry update	2021-12-05 16:29:09 +02:00
Alan Orth	f5fa33bbc6	CHANGELOG.md: add title in citation note	2021-12-05 16:23:39 +02:00
Alan Orth	1b978159c1	data/text.csv: Add data for title in citation test	2021-12-05 16:23:06 +02:00
Alan Orth	4d5696c4cb	csv_metadata_quality/check.py: update title in citation check ... Initialize the titles and citations before the for loop so we can access them later. This makes it easier to check if the item actua- lly has a citation.	2021-12-05 16:21:44 +02:00
Alan Orth	e02678cd7c	tests/test_check.py: add tests for title in citation	2021-12-05 16:01:11 +02:00
Alan Orth	01b4354a14	tests/test_check.py: fix comment	2021-12-05 15:58:25 +02:00
Alan Orth	3b40a68279	Add check for title in citation ... This checks if the item title exists in the citation. If it is not present it could just be missing, or could have minor differences in the whitespace, accents, etc.	2021-12-05 15:52:42 +02:00
Alan Orth	999cc65097	csv_metadata_quality/app.py: adjust mojibake check ... If unsafe fixes (-u) are enabled then we don't need to do the check first before actually fixing them. Doing the check first creates e- tra output that needs to be reviewed by the user.	2021-12-05 15:18:35 +02:00
Alan Orth	a7c3be280d	Update requirements ... Generated with poetry export: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt I am trying `--without-hashes` to work around an error on pip install when running in CI: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==.	2021-11-27 12:26:21 +02:00
Alan Orth	69f68e0a72	poetry.lock: Run poetry update	2021-11-27 12:25:40 +02:00
Alan Orth	c941a90944	.drone.yml: Test on Python 3.10	2021-10-11 20:09:32 +03:00
Alan Orth	c95261f522	CHANGELOG.md: Add note about fix.newlines	2021-10-08 14:37:12 +03:00
Alan Orth	787fa9e8d9	Add field name to fix.newlines output	2021-10-08 14:36:43 +03:00
Alan Orth	82261f7fe0	tests/test_check.py: Run black	2021-10-06 22:10:26 +03:00
Alan Orth	8a27fb2589	Add check for missing DOIs ... Sometimes an editor includes a DOI in the citation field, but does not add a standalone DOI field.	2021-10-06 21:25:39 +03:00
Alan Orth	831ce979c3	CHANGELOG.md: Clarify regex fixes	2021-10-06 21:23:35 +03:00
Alan Orth	58ef62fbcd	Update requirements ... Generated with poetry export: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt I am trying `--without-hashes` to work around an error on pip install when running in CI: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==.	2021-10-06 21:20:35 +03:00
Alan Orth	8c59f57e76	poetry.lock: Run poetry update	2021-10-06 21:19:54 +03:00
Alan Orth	72dd3e7272	CHANGELOG.md: Add notes about regexes	2021-10-06 19:35:59 +03:00
Alan Orth	6ba16d5d4c	csv_metadata_quality/check.py: Fix duplicate checker ... Fix the incorrect type field regex, and improve the title regex to consider dcterms.title and dc.title (along with the DSpace language variants like dc.title[en_US]), but ignore dc.title.alternative. See: https://regex101.com/r/I4m06F/1	2021-10-06 19:32:40 +03:00
Alan Orth	81069259ba	CHANGELOG.md: Add note about bibliographicCitation	2021-10-06 16:16:51 +03:00
Alan Orth	54ab869297	csv_metadata_quality/experimental.py: Adjust citation match ... We need to match both of these citation fields: - dc.identifier.citation - dcterms.bibliographicCitation	2021-10-06 16:13:10 +03:00
Alan Orth	22b359c8a8	Update requirements ... Generated with poetry export: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt I am trying `--without-hashes` to work around an error on pip install when running in CI: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==.	2021-09-27 14:15:01 +03:00
Alan Orth	3e06788d88	poetry.lock: Run poetry update	2021-09-27 14:11:21 +03:00
Alan Orth	3c41cc283f	Update requirements ... Generated with poetry export: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt I am trying `--without-hashes` to work around an error on pip install when running in CI: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==.	2021-09-06 21:04:05 +03:00
Alan Orth	5741e94571	poetry.lock: Run poetry update	2021-09-06 21:03:30 +03:00
Alan Orth	215d61c188	pyproject.toml: limit SQLAlchemy to < 1.4.23 ... SQLAlchemy gets pulled in by csvkit's agate-sql dependency and there is currently an issue with Poetry's parsing of the SQLAlchemy 1.4.23 constraints. Temporarily explicitly install a version of SQLAlchemy that works (can remove later once Poetry fixes this). Anyways, I am not using any SQLAlchemy features that I know of. See: https://github.com/python-poetry/poetry/issues/4402	2021-09-06 21:01:09 +03:00
Alan Orth	11ddde3327	data/test.csv: Update mojibake example ... I was trying to find where I got this one and it seems to have been the other way around. Doesn't matter here only that I was curious.	2021-08-19 15:48:41 +03:00