437 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Alan Orth	abf74909ee	data/test.csv: Add missing date ... I was meaning to test for an invalid multi-value separator here!	2019-07-30 21:01:42 +03:00
Alan Orth	40d5f7d81b	Add support for removing newlines ... This was tricky because of the nature of newlines. In actuality we are removing Unix line feeds here (U+000A) because Windows carriage returns are actually already removed by the string stripping in the whitespace fix. Creating the test case in Vim was difficult because I couldn't fig- ure out how to manually enter a line feed character. In the end I used a search and replace on a known pattern like "ALAN", replacing it with \r. Neither entering the Unicode code point (U+000A) direc- tly or typing an "Enter" character after ^V worked. Grrr.	2019-07-30 20:05:12 +03:00
Alan Orth	346e66ca98	README.md: Add more information to introduction	2019-07-30 17:44:30 +03:00
Alan Orth	bad2fce124	pytest.ini: Don't print captured output ... It makes the summary of passes and fails more annoying to read due to the lines being long and including newlines. I am actually not sure why this fixes it, though... See: https://pytest.readthedocs.io/en/latest/capture.html	2019-07-30 16:43:31 +03:00
Alan Orth	3c798fb504	Use pycountry instead of iso-639 for languages ... The latter is a fork that hasn't been updated since 2016 and the original still seems to be well maintained, with recent database updates as well as tests for Python 3.7. Also, pycountry supports ISO 3166-2 (administrative zones), which we could eventually use for sub regions.	2019-07-30 16:39:26 +03:00
Alan Orth	a85b410ab9	README.md: Improve introduction and functionality	2019-07-30 16:09:15 +03:00
Alan Orth	4e3511cd55	csv_metadata_quality/check.py: Fix AGROVOC lookup ... We actually only need to see if there are more than zero matches because a term like "Nigeria" will match in English, Spanish, etc, whereas terms that *really* don't match will have zero results.	2019-07-30 14:51:44 +03:00
Alan Orth	1fd3d8bc2f	Add pytest.ini from responder ... This seems to be produce a more informative output, though I'm not sure how to filter the annoying deprecation warnings pytest throws about things that are usually done by modules I'm using, not by me. From: https://github.com/taoufik07/responder/blob/master/pytest.ini	2019-07-30 00:45:18 +03:00
Alan Orth	b4aaffd6f1	Update python requirements ... Generated using pipenv: $ pipenv lock -r > requirements.txt	2019-07-30 00:34:32 +03:00
Alan Orth	5ea2e856e4	data/test.csv: Add dc.subject column for AGROVOC tests	2019-07-30 00:33:31 +03:00
Alan Orth	1f65a28307	Add support for validating subjects against AGROVOC ... Checks values in the dc.subject or dcterms.subject field against the AGROVOC REST API hosted by FAO. Code borrowed from agrovoc-lookup.py. See: http://agrovoc.uniroma2.it/agrovoc/agrovoc/en/ See: https://github.com/ilri/DSpace/blob/5_x-prod/agrovoc-lookup.py	2019-07-30 00:30:31 +03:00
Alan Orth	bb882315f1	Update python requirements ... Generated using pipenv: $ pipenv lock -r > requirements.txt $ pipenv lock -r -d > requirements-dev.txt	2019-07-29 19:10:49 +03:00
Alan Orth	a36454a3ac	Add support for validating languages ... Will validate against ISO 639-2 or ISO 639-3 depending on how long the language field is. Otherwise will return that the language is invalid. Does not currently have any support for generic values like "Other".	2019-07-29 18:59:42 +03:00
Alan Orth	1978fa7b48	Add iso-639 library to validate languages	2019-07-29 18:58:50 +03:00
Alan Orth	e49b4e8f22	README.md: Try to simplify list of functionality	2019-07-29 18:25:38 +03:00
Alan Orth	0eb852a65b	README.md: Improve note about unsafe options	2019-07-29 18:14:50 +03:00
Alan Orth	8d92498bbf	Add .gitignore	2019-07-29 18:10:29 +03:00
Alan Orth	8c34c2d6e6	README.md: Add note about removing duplicate values	2019-07-29 18:09:48 +03:00
Alan Orth	1e444cf040	Add fix for duplicate metadata values	2019-07-29 18:05:03 +03:00
Alan Orth	d7888d59a8	csv_metadata_quality/check.py: Return date even if it is invalid ... Otherwise it is missing from the final CSV and then we can't even fix it. :)	2019-07-29 17:40:14 +03:00
Alan Orth	8509006165	data/test.csv: Use more descriptive tests ... To make it obvious what each item is testing.	2019-07-29 17:38:46 +03:00
Alan Orth	e33551776c	README.md: Update note about unsafe options	2019-07-29 17:25:42 +03:00
Alan Orth	7f781d7077	README.md: Finish writing usage section	2019-07-29 17:21:34 +03:00
Alan Orth	50ae4e17f2	csv_metadata_quality/fix.py: Fix indent	2019-07-29 17:14:48 +03:00
Alan Orth	fa4fa3491b	Add check for "suspicious" characters ... These standalone characters often indicate issues with encoding or copy/paste in languages with accents like French and Spanish. For example: foreˆt should be forêt. It is not possible to fix these issues automatically, but this will print a warning so you can notify the owner of the data.	2019-07-29 17:08:49 +03:00
Alan Orth	8047a57cc5	Add support for fixing "unnecessary" Unicode ... These are things like non-breaking spaces, "replacement" characters, etc that add nothing to the metadata and often cause errors during parsing or displaying in a UI.	2019-07-29 16:38:10 +03:00
Alan Orth	ae66382046	README.md: Add note about unnecessary Unicode fixes	2019-07-29 16:34:39 +03:00
Alan Orth	d73f7b54b1	csv_metadata_quality/app.py: Improve comments	2019-07-29 16:24:35 +03:00
Alan Orth	42920e9c7c	Test Python regular expression matches directly ... Match objects always have a boolean value of True. See: https://docs.python.org/3.7/library/re.html	2019-07-29 16:16:30 +03:00
Alan Orth	7b5db1f5d9	csv_metadata_quality/app.py: Remove erroneous comment	2019-07-29 16:15:25 +03:00
Alan Orth	9ac9474c69	README.md: Add todo about duplicates	2019-07-29 12:40:53 +03:00
Alan Orth	e000bd1f88	README.md: Add Travis CI badge	2019-07-29 12:19:10 +03:00
Alan Orth	3554c2991f	README.md: Add note about Python version	2019-07-29 12:15:09 +03:00
Alan Orth	8e3e7a3573	.travis.yml: Only test Python 3.6 and 3.7 ... I'm using f-strings, which are only supported in Python 3.6+.	2019-07-29 12:13:55 +03:00
Alan Orth	913500cfff	.travis.yml: Remove env ... This seems to make Travis run one job for each variable.	2019-07-29 11:51:12 +03:00
Alan Orth	17f4a26ac0	Add Travis CI support ... I'm happy with SourceHut's build support, but Travis allows us to tests other Python versions.	2019-07-29 11:45:58 +03:00
Alan Orth	ac127e7f8a	README.md: Add usage section	2019-07-29 11:30:06 +03:00
Alan Orth	aabb57321c	README.md: Improve ... Reorganize functionality section and add installation section.	2019-07-29 11:15:51 +03:00
Alan Orth	a8a41d60b6	README.md: Add note about Pandas	2019-07-29 10:56:02 +03:00
Alan Orth	cf6c01caaf	README.md: Add notes about unsafe fixes	2019-07-28 23:01:00 +03:00
Alan Orth	40e77db713	Add "unsafe fixes" runtime option ... In this case it fixes occurences of invalid multi-value separators. DSpace uses "||" to separate multiple values in one field, but our editors sometimes give us files with mistakes like "|". We can fix these to be correct multi-value separators if we are sure that the metadata is not actually using "|" for some legitimate purpose.	2019-07-28 22:53:39 +03:00
Alan Orth	a93b5b31c5	Add support for command line arguments ... Currently only supports specifying input and output files with -i and -o. Eventually I'll add more options like dry run, debug, and maybe things like forcing unsafe fixes.	2019-07-28 20:31:57 +03:00
Alan Orth	4e6225c0a9	README.md: Add note about Excel files	2019-07-28 18:38:36 +03:00
Alan Orth	d293214d3a	README.md: Add note about checking dates	2019-07-28 18:37:46 +03:00
Alan Orth	87b1997051	Fix whitespace errors found by flake8	2019-07-28 17:47:28 +03:00
Alan Orth	c6e7d6d9b5	Add flake8 to pipenv dev environment ... To help check PEP8 formatting/style compliance.	2019-07-28 17:46:30 +03:00
Alan Orth	aadb3117eb	csv_metadata_quality/app.py: Remove unused test input files	2019-07-28 17:45:05 +03:00
Alan Orth	ce8f140c66	tests: Remove unused pytest import	2019-07-28 17:42:54 +03:00
Alan Orth	e88d35ace3	csv_metadata_quality/app.py: Use regex in column match ... Check for a column that has "issn" or "isbn" in the name rather than by its explicit name, as the column is dc.identifier.issn now, but will be cg.issn in the future if CG Core v2 happens.	2019-07-28 17:27:20 +03:00
Alan Orth	4687e2f5fa	README.md: Remove todo for date validation	2019-07-28 17:26:39 +03:00