646 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Alan Orth	89d72540f1	data/test.csv: Add sample for missing space after comma	2019-08-28 00:08:26 +03:00
Alan Orth	81190d56bb	Add fix for missing space after commas ... This happens in names very often, for example in the contributor and citation fields. I will limit this to those fields for now and hide this fix behind the "unsafe fixes" option until I test it more.	2019-08-28 00:05:52 +03:00
Alan Orth	2af714fb05	README.md: Add a handful of TODOs	2019-08-27 00:12:41 +03:00
Alan Orth	cc863a6bdd	CHANGELOG.md: Add note about excluding fields	2019-08-27 00:11:22 +03:00
Alan Orth	113e7cd8b6	csv_metadata_quality/app.py: Add ability to skip fields ... The user may want to skip the checking and fixing of certain fields in the input file.	2019-08-27 00:10:07 +03:00
Alan Orth	bd984f3db5	README.md: Update TravisCI badge	2019-08-22 15:07:03 +03:00
Alan Orth	3f4e84a638	README.md: Use ILRI GitHub remote	2019-08-22 14:54:12 +03:00
Alan Orth	c52b3ed131	CHANGELOG.md: Add note about AGROVOC	2019-08-21 16:37:49 +03:00
Alan Orth	884e8f970d	csv_metadata_quality/check.py: Simplify AGROVOC check ... I recycled this code from a separate agrovoc-lookup.py script that checks lines in a text file to see if they are valid AGROVOC terms or not. There I was concerned about skipping comments or something I think, but we don't need to check that here. We simply check the term that is in the field and inform the user if it's valid or not.	2019-08-21 16:35:29 +03:00
Alan Orth	6d02f5026a	CHANGELOG.md: Add note about date checks	2019-08-21 15:35:46 +03:00
Alan Orth	e7cb8920db	tests/test_check.py: Update date tests	2019-08-21 15:34:52 +03:00
Alan Orth	ed5612fbcf	Add column name to output in date checks ... This makes it easier to understand where the error is in case a CSV has multiple date fields, for example: Missing date (dc.date.issued). Missing date (dc.date.issued[]). If you have 126 items and you get 126 "Missing date" messages then it's likely that 100 of the items have dates in one field, and the others have dates in other field.	2019-08-21 15:31:12 +03:00
Alan Orth	3247495cee	CHANGELOG.md: Remove extra space	2019-08-11 10:43:27 +03:00
Alan Orth	7255bf4707	Version 0.2.1	2019-08-11 10:39:39 +03:00
Alan Orth	3aaf18c290	CHANGELOG.md: Move unreleased changes to 0.2.1	2019-08-11 10:39:18 +03:00
Alan Orth	745306edd7	CHANGELOG.md: Add note about replacement of unnccesary Unicode	2019-08-11 00:09:35 +03:00
Alan Orth	e324e321a2	data/test.csv: Add test for replacement of unneccessary Unicode	2019-08-11 00:08:44 +03:00
Alan Orth	232ff99898	csv_metadata_quality/fix.py: Add more unneccessary Unicode fixes ... Add a check for soft hyphens (U+00AD). In one sample CSV I have a normal hyphen followed by a soft hyphen in an ISBN. This causes the ISBN validation to fail.	2019-08-11 00:07:21 +03:00
Alan Orth	13d5221378	csv_metadata_quality/check.py: Fix test for False	2019-08-10 23:52:53 +03:00
Alan Orth	3c7a9eb75b	CHANGELOG.md: Add check for uncommon filename extensions	2019-08-10 23:47:46 +03:00
Alan Orth	a99fbd8a51	data/test.csv: Add test case for uncommon filename extension	2019-08-10 23:46:56 +03:00
Alan Orth	e801042340	tests/test_check.py: Fix unused result ... We don't need to capture the function's return value here because pytest will capture stdout from the function.	2019-08-10 23:45:41 +03:00
Alan Orth	62ef2a4489	tests/test_check.py: Add tests for file extensions	2019-08-10 23:44:13 +03:00
Alan Orth	9ce7dc6716	Add check for uncommon filenames ... Generally we want people to upload documents in accessible formats like PDF, Word, Excel, and PowerPoint. This check warns if a file is using an uncommon extension.	2019-08-10 23:41:16 +03:00
Alan Orth	5ff584a8d7	Version 0.2.0	2019-08-09 01:39:51 +03:00
Alan Orth	4cf7bc182b	Update requirements-dev.txt ... Generated with: $ pipenv lock -r -d > requirements-dev.txt	2019-08-09 01:34:54 +03:00
Alan Orth	7d3f5aae66	CHANGELOG.md: Add pytest-clarity	2019-08-09 01:33:34 +03:00
Alan Orth	c77c065e25	Update Pipfile.lock	2019-08-09 01:32:53 +03:00
Alan Orth	8fb40d96b1	Pipfile: Add pytest-clarity to dev packages ... This helps you understand the cryptic assertion error output from pytest. For some reason pytest-clarity is a pre-release package so we need to install it in pipenv with --pre.	2019-08-09 01:30:37 +03:00
Alan Orth	5f2e3ff4bd	CHANGELOG.md: Add improved suspicious character check	2019-08-09 01:28:07 +03:00
Alan Orth	d93c2aae13	tests/test_check.py: Update suspicious character check ... The suspicious character check was updated to include the name of the field where the metadata value with the suspicious character exists.	2019-08-09 01:26:38 +03:00
Alan Orth	62fea95087	Improve suspicious character detection ... Now it will print just the part of the metadata value that contains the suspicious character (up to 80 characters, so we don't make the line break on terminals that use 80 character width by default). Also, print the name of the field in which the metadata value is so that it is easier for the user to locate.	2019-08-09 01:25:40 +03:00
Alan Orth	8772bdec51	csv_metadata_quality/app.py: Explicitly exit with success	2019-08-04 09:10:37 +03:00
Alan Orth	6d4ecd75aa	csv_metadata_quality/app.py: Close files before exit	2019-08-04 09:10:19 +03:00
Alan Orth	264ce1d1df	CHANGELOG.md: Add new item for Ctrl-C handling	2019-08-03 22:18:44 +03:00
Alan Orth	f4e7fd73f5	csv_metadata_quality/app.py: Handle Ctrl-C ... Instead of printing an ugly two-page stack trace.	2019-08-03 21:11:57 +03:00
Alan Orth	a00d3d7ea5	README.md: Simplify installation instructions ... Pipenv has captured the local dependency with `-e .` so now it gets installed by the Pipfile or requirements.txt.	2019-08-02 11:02:50 +03:00
Alan Orth	f772a3be41	Update python requirements ... Generated using pipenv: $ pipenv lock -r > requirements.txt	2019-08-02 11:02:25 +03:00
Alan Orth	d1b3e9e375	pipenv install -e .	2019-08-02 10:58:21 +03:00
Alan Orth	85ae7bdc5a	Increment version to 0.1.0	2019-08-02 00:12:47 +03:00
Alan Orth	980ef0d98d	Add initial changelog	2019-08-02 00:10:28 +03:00
Alan Orth	0561300ebe	Add option to print version with --version or -V ... I guess `-v` is more commonly used for "verbose" so I will use the short option of `-V` for version.	2019-08-02 00:09:54 +03:00
Alan Orth	456b8a2f26	Update tests	2019-08-01 23:59:11 +03:00
Alan Orth	0ed390dbd5	README.md: Update AGROVOC information ... Now details the new `--agrovoc-fields` option.	2019-08-01 23:54:40 +03:00
Alan Orth	bf876a046a	Rework AGROVOC validation ... AGROVOC validation is now disabled by default, but can be enabled on a field-by-field basis. For example, countries and regions are also present in AGROVOC. Fields with these values can be enabled using the new `--agrovoc-fields` option. I reworked the script output to show the field name when printing an invalid term so that the user knows in which field the term is.	2019-08-01 23:51:58 +03:00
Alan Orth	576b3a3638	csv_metadata_quality/__main__.py: Fix spacing ... Identified by flake8.	2019-08-01 23:28:16 +03:00
Alan Orth	a6eba0fc1a	.build.yml: Use pipenv run instead of pipenv shell ... The latter creates a shell and of course it doesn't ever exit!	2019-07-31 17:58:21 +03:00
Alan Orth	857492cd93	.build.yml: Try fix CLI test ... I was writing the CLI output to /dev/null because I was lazy, but actually this might break the SourceHut build.	2019-07-31 17:54:49 +03:00
Alan Orth	db42fbfea9	.build.yml: Try to run the script itself	2019-07-31 17:48:17 +03:00
Alan Orth	734ae7f011	setup.py: Add "DSpace" to description ... I suppose someone searching PyPi (eventually) would be happy to see that this was written specifically with DSpace in mind.	2019-07-31 17:42:27 +03:00