Version 0.4.2

CHANGELOG.md: Update changes
Remove pipenv
2025-05-09 22:56:01 +02:00 · 2020-07-06 14:04:34 +03:00 · 2020-07-06 14:00:21 +03:00 · 2020-07-06 13:59:49 +03:00 · 2020-07-06 13:59:11 +03:00 · 2020-07-06 13:57:07 +03:00
22 changed files with 2158 additions and 676 deletions
--- a/.build.yml
+++ b/.build.yml
@ -1,19 +1,15 @@
 image: archlinux
 packages:
-  - python-pipenv
+  - python-poetry
 sources:
  - https://git.sr.ht/~alanorth/csv-metadata-quality
 tasks:
  - setup: |
      cd csv-metadata-quality
-      pipenv install --dev
+      poetry install
  - pytest: |
      cd csv-metadata-quality
-      pipenv run pytest
+      poetry run pytest
  - testcli: |
      cd csv-metadata-quality
-      pipenv run pip install .
-      pipenv run csv-metadata-quality -i data/test.csv -o /tmp/test.csv -u --agrovoc-fields dc.subject,cg.coverage.country
-environment:
-  PIPENV_NOSPIN: 'True'
-  PIPENV_HIDE_EMOJIS: 'True'
+      poetry run csv-metadata-quality -i data/test.csv -o /tmp/test.csv -e -u --agrovoc-fields dc.subject,cg.coverage.country
--- a/.travis.yml
+++ b/.travis.yml
@ -1,11 +1,16 @@
-dist: xenial
+dist: bionic
 language: python
 python:
  - "3.6"
  - "3.7"
+  - "3.8"
+  - "3.8-dev"  # 3.8 development branch
+jobs:
+  allow_failures:
+    - python: "3.8-dev"
 install:
-  - "pip install pipenv --upgrade-strategy=only-if-needed"
-  - "pipenv install --dev"
+  - "pip install -r requirements.txt"
+  - "pip install -r requirements-dev.txt"
 script: pytest

 # vim: ts=2 sw=2 et
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@ -4,6 +4,52 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

+## [0.4.2] - 2020-07-06
+### Changed
+- Add field name to the output for more fixes and checks to help identify where
+the error is
+- Minor optimizations to AGROVOC subject lookup
+- Use Poetry instead of Pipenv
+
+### Updated
+- Update python dependencies to latest versions
+
+## [0.4.1] - 2020-01-15
+### Changed
+- Reduce minimum Python version to 3.6 by working around the `is_normalized()`
+that only works in Python >= 3.8
+
+## [0.4.0] - 2020-01-15
+### Added
+- Unicode normalization (enable with `--unsafe-fixes`, see README.md)
+
+### Updated
+- Update python dependencies to latest versions, including numpy 1.18.1, pandas
+1.0.0rc0, flake8 3.7.9, pytest 5.3.2, and black 19.10b0
+- Regenerate requirements.txt and requirements-dev.txt
+
+### Changed
+- Use Python 3.8.0 for pipenv
+- Use Ubuntu 18.04 "Bionic" for TravisCI builds
+- Test Python 3.8 in TravisCI builds
+
+## [0.3.1] - 2019-10-01
+## Changed
+- Replace non-breaking spaces (U+00A0) with space instead of removing them
+- Harmonize language of script output when fixing various issues
+
+## [0.3.0] - 2019-09-26
+### Updated
+- Update python dependencies to latest versions, including numpy 1.17.2, pandas
+0.25.1, pytest 5.1.3, and requests-cache 0.5.2
+
+### Added
+- csvkit to dev requirements (csvcut etc are useful during development)
+- Experimental language validation using the Python `langid` library (enable with `-e`, see README.md)
+
+### Changed
+- Re-formatted code with black and isort
+
 ## [0.2.2] - 2019-08-27
 ### Changed
 - Output of date checks to include column names (helps debugging in case there are multiple date fields)
--- a/25
+++ b/25
@ -1,25 +0,0 @@
-[[source]]
-name = "pypi"
-url = "https://pypi.org/simple"
-verify_ssl = true
-
-[dev-packages]
-pytest = "*"
-ipython = "*"
-flake8 = "*"
-pytest-clarity = "*"
-
-[packages]
-pandas = "*"
-python-stdnum = "*"
-xlrd = "*"
-requests = "*"
-requests-cache = "*"
-pycountry = "*"
-csv-metadata-quality = {editable = true,path = "."}
-
-[requires]
-python_version = "3.7"
-
-[pipenv]
-allow_prereleases = true
--- a/Pipfile.lock
+++ b/Pipfile.lock
@ -1,376 +0,0 @@
-{
-    "_meta": {
-        "hash": {
-            "sha256": "f8f0a9f208ec41f4d8183ecfc68356b40674b083b2f126c37468b3c9533ba5df"
-        },
-        "pipfile-spec": 6,
-        "requires": {
-            "python_version": "3.7"
-        },
-        "sources": [
-            {
-                "name": "pypi",
-                "url": "https://pypi.org/simple",
-                "verify_ssl": true
-            }
-        ]
-    },
-    "default": {
-        "certifi": {
-            "hashes": [
-                "sha256:046832c04d4e752f37383b628bc601a7ea7211496b4638f6514d0e5b9acc4939",
-                "sha256:945e3ba63a0b9f577b1395204e13c3a231f9bc0223888be653286534e5873695"
-            ],
-            "version": "==2019.6.16"
-        },
-        "chardet": {
-            "hashes": [
-                "sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae",
-                "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691"
-            ],
-            "version": "==3.0.4"
-        },
-        "csv-metadata-quality": {
-            "editable": true,
-            "path": "."
-        },
-        "idna": {
-            "hashes": [
-                "sha256:c357b3f628cf53ae2c4c05627ecc484553142ca23264e593d327bcde5e9c3407",
-                "sha256:ea8b7f6188e6fa117537c3df7da9fc686d485087abf6ac197f9c46432f7e4a3c"
-            ],
-            "version": "==2.8"
-        },
-        "numpy": {
-            "hashes": [
-                "sha256:03e311b0a4c9f5755da7d52161280c6a78406c7be5c5cc7facfbcebb641efb7e",
-                "sha256:0cdd229a53d2720d21175012ab0599665f8c9588b3b8ffa6095dd7b90f0691dd",
-                "sha256:312bb18e95218bedc3563f26fcc9c1c6bfaaf9d453d15942c0839acdd7e4c473",
-                "sha256:464b1c48baf49e8505b1bb754c47a013d2c305c5b14269b5c85ea0625b6a988a",
-                "sha256:5adfde7bd3ee4864536e230bcab1c673f866736698724d5d28c11a4d63672658",
-                "sha256:7724e9e31ee72389d522b88c0d4201f24edc34277999701ccd4a5392e7d8af61",
-                "sha256:8d36f7c53ae741e23f54793ffefb2912340b800476eb0a831c6eb602e204c5c4",
-                "sha256:910d2272403c2ea8a52d9159827dc9f7c27fb4b263749dca884e2e4a8af3b302",
-                "sha256:951fefe2fb73f84c620bec4e001e80a80ddaa1b84dce244ded7f1e0cbe0ed34a",
-                "sha256:9588c6b4157f493edeb9378788dcd02cb9e6a6aeaa518b511a1c79d06cbd8094",
-                "sha256:9ce8300950f2f1d29d0e49c28ebfff0d2f1e2a7444830fbb0b913c7c08f31511",
-                "sha256:be39cca66cc6806652da97103605c7b65ee4442c638f04ff064a7efd9a81d50a",
-                "sha256:c3ab2d835b95ccb59d11dfcd56eb0480daea57cdf95d686d22eff35584bc4554",
-                "sha256:eb0fc4a492cb896346c9e2c7a22eae3e766d407df3eb20f4ce027f23f76e4c54",
-                "sha256:ec0c56eae6cee6299f41e780a0280318a93db519bbb2906103c43f3e2be1206c",
-                "sha256:f4e4612de60a4f1c4d06c8c2857cdcb2b8b5289189a12053f37d3f41f06c60d0"
-            ],
-            "version": "==1.17.0"
-        },
-        "pandas": {
-            "hashes": [
-                "sha256:074a032f99bb55d178b93bd98999c971542f19317829af08c99504febd9e9b8b",
-                "sha256:20f1728182b49575c2f6f681b3e2af5fac9e84abdf29488e76d569a7969b362e",
-                "sha256:2745ba6e16c34d13d765c3657bb64fa20a0e2daf503e6216a36ed61770066179",
-                "sha256:32c44e5b628c48ba17703f734d59f369d4cdcb4239ef26047d6c8a8bfda29a6b",
-                "sha256:3b9f7dcee6744d9dcdd53bce19b91d20b4311bf904303fa00ef58e7df398e901",
-                "sha256:544f2033250980fb6f069ce4a960e5f64d99b8165d01dc39afd0b244eeeef7d7",
-                "sha256:58f9ef68975b9f00ba96755d5702afdf039dea9acef6a0cfd8ddcde32918a79c",
-                "sha256:9023972a92073a495eba1380824b197ad1737550fe1c4ef8322e65fe58662888",
-                "sha256:914341ad2d5b1ea522798efa4016430b66107d05781dbfe7cf05eba8f37df995",
-                "sha256:9d151bfb0e751e2c987f931c57792871c8d7ff292bcdfcaa7233012c367940ee",
-                "sha256:b932b127da810fef57d427260dde1ad54542c136c44b227a1e367551bb1a684b",
-                "sha256:cfb862aa37f4dd5be0730731fdb8185ac935aba8b51bf3bd035658111c9ee1c9",
-                "sha256:de7ecb4b120e98b91e8a2a21f186571266a8d1faa31d92421e979c7ca67d8e5c",
-                "sha256:df7e1933a0b83920769611c5d6b9a1bf301e3fa6a544641c6678c67621fe9843"
-            ],
-            "index": "pypi",
-            "version": "==0.25.0"
-        },
-        "pycountry": {
-            "hashes": [
-                "sha256:68e58bfd3bedeea49ba9d4b38f2bd5e042f9753628eba9a819fb03f551d89096"
-            ],
-            "index": "pypi",
-            "version": "==19.7.15"
-        },
-        "python-dateutil": {
-            "hashes": [
-                "sha256:7e6584c74aeed623791615e26efd690f29817a27c73085b78e4bad02493df2fb",
-                "sha256:c89805f6f4d64db21ed966fda138f8a5ed7a4fdbc1a8ee329ce1b74e3c74da9e"
-            ],
-            "version": "==2.8.0"
-        },
-        "python-stdnum": {
-            "hashes": [
-                "sha256:d5f0af1bee9ddd9a20b398b46ce062dbd4d41fcc9646940f2667256a44df3854",
-                "sha256:f445ec32bf5246c90389204cabba465f494545371c29a83fa2d30e6c872a6763"
-            ],
-            "index": "pypi",
-            "version": "==1.11"
-        },
-        "pytz": {
-            "hashes": [
-                "sha256:26c0b32e437e54a18161324a2fca3c4b9846b74a8dccddd843113109e1116b32",
-                "sha256:c894d57500a4cd2d5c71114aaab77dbab5eabd9022308ce5ac9bb93a60a6f0c7"
-            ],
-            "version": "==2019.2"
-        },
-        "requests": {
-            "hashes": [
-                "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4",
-                "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31"
-            ],
-            "index": "pypi",
-            "version": "==2.22.0"
-        },
-        "requests-cache": {
-            "hashes": [
-                "sha256:6822f788c5ee248995c4bfbd725de2002ad710182ba26a666e85b64981866060",
-                "sha256:73a7211870f7d67af5fd81cad2f67cfe1cd3eb4ee6a85155e07613968cc72dfc"
-            ],
-            "index": "pypi",
-            "version": "==0.5.0"
-        },
-        "six": {
-            "hashes": [
-                "sha256:3350809f0555b11f552448330d0b52d5f24c91a322ea4a15ef22629740f3761c",
-                "sha256:d16a0141ec1a18405cd4ce8b4613101da75da0e9a7aec5bdd4fa804d0e0eba73"
-            ],
-            "version": "==1.12.0"
-        },
-        "urllib3": {
-            "hashes": [
-                "sha256:b246607a25ac80bedac05c6f282e3cdaf3afb65420fd024ac94435cabe6e18d1",
-                "sha256:dbe59173209418ae49d485b87d1681aefa36252ee85884c31346debd19463232"
-            ],
-            "version": "==1.25.3"
-        },
-        "xlrd": {
-            "hashes": [
-                "sha256:546eb36cee8db40c3eaa46c351e67ffee6eeb5fa2650b71bc4c758a29a1b29b2",
-                "sha256:e551fb498759fa3a5384a94ccd4c3c02eb7c00ea424426e212ac0c57be9dfbde"
-            ],
-            "index": "pypi",
-            "version": "==1.2.0"
-        }
-    },
-    "develop": {
-        "atomicwrites": {
-            "hashes": [
-                "sha256:03472c30eb2c5d1ba9227e4c2ca66ab8287fbfbbda3888aa93dc2e28fc6811b4",
-                "sha256:75a9445bac02d8d058d5e1fe689654ba5a6556a1dfd8ce6ec55a0ed79866cfa6"
-            ],
-            "version": "==1.3.0"
-        },
-        "attrs": {
-            "hashes": [
-                "sha256:69c0dbf2ed392de1cb5ec704444b08a5ef81680a61cb899dc08127123af36a79",
-                "sha256:f0b870f674851ecbfbbbd364d6b5cbdff9dcedbc7f3f5e18a6891057f21fe399"
-            ],
-            "version": "==19.1.0"
-        },
-        "backcall": {
-            "hashes": [
-                "sha256:38ecd85be2c1e78f77fd91700c76e14667dc21e2713b63876c0eb901196e01e4",
-                "sha256:bbbf4b1e5cd2bdb08f915895b51081c041bac22394fdfcfdfbe9f14b77c08bf2"
-            ],
-            "version": "==0.1.0"
-        },
-        "decorator": {
-            "hashes": [
-                "sha256:86156361c50488b84a3f148056ea716ca587df2f0de1d34750d35c21312725de",
-                "sha256:f069f3a01830ca754ba5258fde2278454a0b5b79e0d7f5c13b3b97e57d4acff6"
-            ],
-            "version": "==4.4.0"
-        },
-        "entrypoints": {
-            "hashes": [
-                "sha256:589f874b313739ad35be6e0cd7efde2a4e9b6fea91edcc34e58ecbb8dbe56d19",
-                "sha256:c70dd71abe5a8c85e55e12c19bd91ccfeec11a6e99044204511f9ed547d48451"
-            ],
-            "version": "==0.3"
-        },
-        "flake8": {
-            "hashes": [
-                "sha256:19241c1cbc971b9962473e4438a2ca19749a7dd002dd1a946eaba171b4114548",
-                "sha256:8e9dfa3cecb2400b3738a42c54c3043e821682b9c840b0448c0503f781130696"
-            ],
-            "index": "pypi",
-            "version": "==3.7.8"
-        },
-        "importlib-metadata": {
-            "hashes": [
-                "sha256:23d3d873e008a513952355379d93cbcab874c58f4f034ff657c7a87422fa64e8",
-                "sha256:80d2de76188eabfbfcf27e6a37342c2827801e59c4cc14b0371c56fed43820e3"
-            ],
-            "version": "==0.19"
-        },
-        "ipython": {
-            "hashes": [
-                "sha256:1d3a1692921e932751bc1a1f7bb96dc38671eeefdc66ed33ee4cbc57e92a410e",
-                "sha256:537cd0176ff6abd06ef3e23f2d0c4c2c8a4d9277b7451544c6cbf56d1c79a83d"
-            ],
-            "index": "pypi",
-            "version": "==7.7.0"
-        },
-        "ipython-genutils": {
-            "hashes": [
-                "sha256:72dd37233799e619666c9f639a9da83c34013a73e8bbc79a7a6348d93c61fab8",
-                "sha256:eb2e116e75ecef9d4d228fdc66af54269afa26ab4463042e33785b887c628ba8"
-            ],
-            "version": "==0.2.0"
-        },
-        "jedi": {
-            "hashes": [
-                "sha256:53c850f1a7d3cfcd306cc513e2450a54bdf5cacd7604b74e42dd1f0758eaaf36",
-                "sha256:e07457174ef7cb2342ff94fa56484fe41cec7ef69b0059f01d3f812379cb6f7c"
-            ],
-            "version": "==0.14.1"
-        },
-        "mccabe": {
-            "hashes": [
-                "sha256:ab8a6258860da4b6677da4bd2fe5dc2c659cff31b3ee4f7f5d64e79735b80d42",
-                "sha256:dd8d182285a0fe56bace7f45b5e7d1a6ebcbf524e8f3bd87eb0f125271b8831f"
-            ],
-            "version": "==0.6.1"
-        },
-        "more-itertools": {
-            "hashes": [
-                "sha256:409cd48d4db7052af495b09dec721011634af3753ae1ef92d2b32f73a745f832",
-                "sha256:92b8c4b06dac4f0611c0729b2f2ede52b2e1bac1ab48f089c7ddc12e26bb60c4"
-            ],
-            "version": "==7.2.0"
-        },
-        "packaging": {
-            "hashes": [
-                "sha256:a7ac867b97fdc07ee80a8058fe4435ccd274ecc3b0ed61d852d7d53055528cf9",
-                "sha256:c491ca87294da7cc01902edbe30a5bc6c4c28172b5138ab4e4aa1b9d7bfaeafe"
-            ],
-            "version": "==19.1"
-        },
-        "parso": {
-            "hashes": [
-                "sha256:63854233e1fadb5da97f2744b6b24346d2750b85965e7e399bec1620232797dc",
-                "sha256:666b0ee4a7a1220f65d367617f2cd3ffddff3e205f3f16a0284df30e774c2a9c"
-            ],
-            "version": "==0.5.1"
-        },
-        "pexpect": {
-            "hashes": [
-                "sha256:2094eefdfcf37a1fdbfb9aa090862c1a4878e5c7e0e7e7088bdb511c558e5cd1",
-                "sha256:9e2c1fd0e6ee3a49b28f95d4b33bc389c89b20af6a1255906e90ff1262ce62eb"
-            ],
-            "markers": "sys_platform != 'win32'",
-            "version": "==4.7.0"
-        },
-        "pickleshare": {
-            "hashes": [
-                "sha256:87683d47965c1da65cdacaf31c8441d12b8044cdec9aca500cd78fc2c683afca",
-                "sha256:9649af414d74d4df115d5d718f82acb59c9d418196b7b4290ed47a12ce62df56"
-            ],
-            "version": "==0.7.5"
-        },
-        "pluggy": {
-            "hashes": [
-                "sha256:0825a152ac059776623854c1543d65a4ad408eb3d33ee114dff91e57ec6ae6fc",
-                "sha256:b9817417e95936bf75d85d3f8767f7df6cdde751fc40aed3bb3074cbcb77757c"
-            ],
-            "version": "==0.12.0"
-        },
-        "prompt-toolkit": {
-            "hashes": [
-                "sha256:11adf3389a996a6d45cc277580d0d53e8a5afd281d0c9ec71b28e6f121463780",
-                "sha256:2519ad1d8038fd5fc8e770362237ad0364d16a7650fb5724af6997ed5515e3c1",
-                "sha256:977c6583ae813a37dc1c2e1b715892461fcbdaa57f6fc62f33a528c4886c8f55"
-            ],
-            "version": "==2.0.9"
-        },
-        "ptyprocess": {
-            "hashes": [
-                "sha256:923f299cc5ad920c68f2bc0bc98b75b9f838b93b599941a6b63ddbc2476394c0",
-                "sha256:d7cc528d76e76342423ca640335bd3633420dc1366f258cb31d05e865ef5ca1f"
-            ],
-            "version": "==0.6.0"
-        },
-        "py": {
-            "hashes": [
-                "sha256:64f65755aee5b381cea27766a3a147c3f15b9b6b9ac88676de66ba2ae36793fa",
-                "sha256:dc639b046a6e2cff5bbe40194ad65936d6ba360b52b3c3fe1d08a82dd50b5e53"
-            ],
-            "version": "==1.8.0"
-        },
-        "pycodestyle": {
-            "hashes": [
-                "sha256:95a2219d12372f05704562a14ec30bc76b05a5b297b21a5dfe3f6fac3491ae56",
-                "sha256:e40a936c9a450ad81df37f549d676d127b1b66000a6c500caa2b085bc0ca976c"
-            ],
-            "version": "==2.5.0"
-        },
-        "pyflakes": {
-            "hashes": [
-                "sha256:17dbeb2e3f4d772725c777fabc446d5634d1038f234e77343108ce445ea69ce0",
-                "sha256:d976835886f8c5b31d47970ed689944a0262b5f3afa00a5a7b4dc81e5449f8a2"
-            ],
-            "version": "==2.1.1"
-        },
-        "pygments": {
-            "hashes": [
-                "sha256:71e430bc85c88a430f000ac1d9b331d2407f681d6f6aec95e8bcfbc3df5b0127",
-                "sha256:881c4c157e45f30af185c1ffe8d549d48ac9127433f2c380c24b84572ad66297"
-            ],
-            "version": "==2.4.2"
-        },
-        "pyparsing": {
-            "hashes": [
-                "sha256:6f98a7b9397e206d78cc01df10131398f1c8b8510a2f4d97d9abd82e1aacdd80",
-                "sha256:d9338df12903bbf5d65a0e4e87c2161968b10d2e489652bb47001d82a9b028b4"
-            ],
-            "version": "==2.4.2"
-        },
-        "pytest": {
-            "hashes": [
-                "sha256:6ef6d06de77ce2961156013e9dff62f1b2688aa04d0dc244299fe7d67e09370d",
-                "sha256:a736fed91c12681a7b34617c8fcefe39ea04599ca72c608751c31d89579a3f77"
-            ],
-            "index": "pypi",
-            "version": "==5.0.1"
-        },
-        "pytest-clarity": {
-            "hashes": [
-                "sha256:3f40d5ae7cb21cc95e622fc4f50d9466f80ae0f91460225b8c95c07afbf93e20"
-            ],
-            "index": "pypi",
-            "version": "==0.2.0a1"
-        },
-        "six": {
-            "hashes": [
-                "sha256:3350809f0555b11f552448330d0b52d5f24c91a322ea4a15ef22629740f3761c",
-                "sha256:d16a0141ec1a18405cd4ce8b4613101da75da0e9a7aec5bdd4fa804d0e0eba73"
-            ],
-            "version": "==1.12.0"
-        },
-        "termcolor": {
-            "hashes": [
-                "sha256:1d6d69ce66211143803fbc56652b41d73b4a400a2891d7bf7a1cdf4c02de613b"
-            ],
-            "version": "==1.1.0"
-        },
-        "traitlets": {
-            "hashes": [
-                "sha256:9c4bd2d267b7153df9152698efb1050a5d84982d3384a37b2c1f7723ba3e7835",
-                "sha256:c6cb5e6f57c5a9bdaa40fa71ce7b4af30298fbab9ece9815b5d995ab6217c7d9"
-            ],
-            "version": "==4.3.2"
-        },
-        "wcwidth": {
-            "hashes": [
-                "sha256:3df37372226d6e63e1b1e1eda15c594bca98a22d33a23832a90998faa96bc65e",
-                "sha256:f4ebe71925af7b40a864553f761ed559b43544f8f71746c2d756c7fe788ade7c"
-            ],
-            "version": "==0.1.7"
-        },
-        "zipp": {
-            "hashes": [
-                "sha256:4970c3758f4e89a7857a973b1e2a5d75bcdc47794442f2e2dd4fe8e0466e809a",
-                "sha256:8a5712cfd3bb4248015eb3b0b3c54a5f6ee3f2425963ef2a0125b8bc40aafaec"
-            ],
-            "version": "==0.5.2"
-        }
-    }
-}
--- a/README.md
+++ b/README.md
@ -1,12 +1,13 @@
 # CSV Metadata Quality [![Build Status](https://travis-ci.org/ilri/csv-metadata-quality.svg?branch=master)](https://travis-ci.org/ilri/csv-metadata-quality) [![builds.sr.ht status](https://builds.sr.ht/~alanorth/csv-metadata-quality.svg)](https://builds.sr.ht/~alanorth/csv-metadata-quality?)
-A simple, but opinionated metadata quality checker and fixer designed to work with CSVs in the DSpace ecosystem. The implementation is essentially a pipeline of checks and fixes that begins with splitting multi-value fields on the standard DSpace "||" separator, trimming leading/trailing whitespace, and then proceeding to more specialized cases like ISSNs, ISBNs, languages, etc.
+A simple, but opinionated metadata quality checker and fixer designed to work with CSVs in the DSpace ecosystem (though it could theoretically work on any CSV that uses Dublin Core fields as columns). The implementation is essentially a pipeline of checks and fixes that begins with splitting multi-value fields on the standard DSpace "||" separator, trimming leading/trailing whitespace, and then proceeding to more specialized cases like ISSNs, ISBNs, languages, etc.

-Requires Python 3.6 or greater. CSV and Excel support comes from the [Pandas](https://pandas.pydata.org/) library, though your mileage may vary with Excel because this is much less tested.
+Requires Python 3.8 or greater. CSV and Excel support comes from the [Pandas](https://pandas.pydata.org/) library, though your mileage may vary with Excel because this is much less tested.

 ## Functionality

 - Validate dates, ISSNs, ISBNs, and multi-value separators ("||")
- Validate languages against ISO 639-2 and ISO 639-3
+- Validate languages against ISO 639-1 (alpha2) and ISO 639-3 (alpha3)
+- Experimental validation of titles and abstracts against item's Dublin Core language field
 - Validate subjects against the AGROVOC REST API (see the `--agrovoc-fields` option)
 - Fix leading, trailing, and excessive (ie, more than one) whitespace
 - Fix invalid multi-value separators (`|`) using `--unsafe-fixes`
@ -14,18 +15,19 @@ Requires Python 3.6 or greater. CSV and Excel support comes from the [Pandas](ht
 - Remove unnecessary Unicode like [non-breaking spaces](https://en.wikipedia.org/wiki/Non-breaking_space), [replacement characters](https://en.wikipedia.org/wiki/Specials_(Unicode_block)#Replacement_character), etc
 - Check for "suspicious" characters that indicate encoding or copy/paste issues, for example "foreˆt" should be "forêt"
 - Remove duplicate metadata values
+- Perform [Unicode normalization](https://withblue.ink/2019/03/11/why-you-need-to-normalize-unicode-strings.html) on strings using `--unsafe-fixes`

 ## Installation
-The easiest way to install CSV Metadata Quality is with [pipenv](https://github.com/pypa/pipenv):
+The easiest way to install CSV Metadata Quality is with [poetry](https://python-poetry.org):

 ```
 $ git clone https://github.com/ilri/csv-metadata-quality.git
 $ cd csv-metadata-quality
-$ pipenv install
-$ pipenv shell
+$ poetry install
+$ poetry shell
 ```

-Otherwise, if you don't have pipenv, you can use a vanilla Python virtual environment:
+Otherwise, if you don't have poetry, you can use a vanilla Python virtual environment:

 ```
 $ git clone https://github.com/ilri/csv-metadata-quality.git
@ -57,6 +59,14 @@ This is considered "unsafe" because it is *theoretically* possible for a single
 ### Newlines
 This is considered "unsafe" because some systems give special importance to vertical space and render it properly. DSpace does not support rendering newlines in its XMLUI and has, at times, suffered from parsing errors that cause the import process to fail if an input file had newlines. The `--unsafe-fixes` option strips Unix line feeds (U+000A).

+### Unicode Normalization
+[Unicode](https://en.wikipedia.org/wiki/Unicode) is a standard for encoding text. As the standard aims to support most of the world's languages, characters can often be represented in different ways and still be valid Unicode. This leads to interesting problems that can be confusing unless you know what's going on behind the scenes. For example, the characters `é` and `é` *look* the same, but are not — technically they refer to different code points in the Unicode standard:
+
+- `é` is the Unicode code point `U+00E9`
+- `é` is the Unicode code points `U+0065` + `U+0301`
+
+Read more about [Unicode normalization](https://withblue.ink/2019/03/11/why-you-need-to-normalize-unicode-strings.html).
+
 ## AGROVOC Validation
 You can enable validation of metadata values in certain fields against the AGROVOC REST API with the `--agrovoc-fields` option. For example, in addition to agricultural subjects, many countries and regions are also present AGROVOC. Enable this validation by specifying a comma-separated list of fields:

@ -69,6 +79,18 @@ Invalid AGROVOC (cg.coverage.country): KENYAA

 *Note: Requests to the AGROVOC REST API are cached using [requests_cache](https://pypi.org/project/requests-cache/) to speed up subsequent runs with the same data and to be kind to the system's administrators.*

+## Experimental Checks
+You can enable experimental support for validating whether the value of an item's `dc.language.iso` or `dcterms.language` field matches the actual language used in its title, abstract, and citation.
+
+```
+$ csv-metadata-quality -i data/test.csv -o /tmp/test.csv -e
+...
+Possibly incorrect language es (detected en): Incorrect ISO 639-1 language
+Possibly incorrect language spa (detected eng): Incorrect ISO 639-3 language
+```
+
+This currently uses the [Python langid](https://github.com/saffsd/langid.py) library. In the future I would like to move to the fastText library, but there is currently an [issue with their Python bindings](https://github.com/facebookresearch/fastText/issues/909) that makes this unfeasible.
+
 ## Todo

 - Reporting / summary
@ -79,7 +101,6 @@ Invalid AGROVOC (cg.coverage.country): KENYAA
 - Validate DOIs? Normalize to https://doi.org format? Or use just the DOI part: 10.1016/j.worlddev.2010.06.006
 - Warn if two items use the same file in `filename` column
 - Add an option to drop invalid AGROVOC subjects?
- Add check for author names with incorrect spacing after commas, ie "Orth,Alan S."
 - Add tests for application invocation, ie `tests/test_app.py`?

 ## License
--- a/csv_metadata_quality/main.py
+++ b/csv_metadata_quality/main.py
@ -1,10 +1,11 @@
-from csv_metadata_quality import app
 from sys import argv

+from csv_metadata_quality import app
+

 def main():
    app.run(argv)


-if __name__ == '__main__':
+if __name__ == "__main__":
    main()
--- a/csv_metadata_quality/app.py
+++ b/csv_metadata_quality/app.py
@ -1,21 +1,54 @@
-from csv_metadata_quality.version import VERSION
 import argparse
-import csv_metadata_quality.check as check
-import csv_metadata_quality.fix as fix
-import pandas as pd
 import re
 import signal
 import sys

+import pandas as pd
+
+import csv_metadata_quality.check as check
+import csv_metadata_quality.experimental as experimental
+import csv_metadata_quality.fix as fix
+from csv_metadata_quality.version import VERSION
+

 def parse_args(argv):
-    parser = argparse.ArgumentParser(description='Metadata quality checker and fixer.')
-    parser.add_argument('--agrovoc-fields', '-a', help='Comma-separated list of fields to validate against AGROVOC, for example: dc.subject,cg.coverage.country')
-    parser.add_argument('--input-file', '-i', help='Path to input file. Can be UTF-8 CSV or Excel XLSX.', required=True, type=argparse.FileType('r', encoding='UTF-8'))
-    parser.add_argument('--output-file', '-o', help='Path to output file (always CSV).', required=True, type=argparse.FileType('w', encoding='UTF-8'))
-    parser.add_argument('--unsafe-fixes', '-u', help='Perform unsafe fixes.', action='store_true')
-    parser.add_argument('--version', '-V', action='version', version=f'CSV Metadata Quality v{VERSION}')
-    parser.add_argument('--exclude-fields', '-x', help='Comma-separated list of fields to skip, for example: dc.contributor.author,dc.identifier.citation')
+    parser = argparse.ArgumentParser(description="Metadata quality checker and fixer.")
+    parser.add_argument(
+        "--agrovoc-fields",
+        "-a",
+        help="Comma-separated list of fields to validate against AGROVOC, for example: dc.subject,cg.coverage.country",
+    )
+    parser.add_argument(
+        "--experimental-checks",
+        "-e",
+        help="Enable experimental checks like language detection",
+        action="store_true",
+    )
+    parser.add_argument(
+        "--input-file",
+        "-i",
+        help="Path to input file. Can be UTF-8 CSV or Excel XLSX.",
+        required=True,
+        type=argparse.FileType("r", encoding="UTF-8"),
+    )
+    parser.add_argument(
+        "--output-file",
+        "-o",
+        help="Path to output file (always CSV).",
+        required=True,
+        type=argparse.FileType("w", encoding="UTF-8"),
+    )
+    parser.add_argument(
+        "--unsafe-fixes", "-u", help="Perform unsafe fixes.", action="store_true"
+    )
+    parser.add_argument(
+        "--version", "-V", action="version", version=f"CSV Metadata Quality v{VERSION}"
+    )
+    parser.add_argument(
+        "--exclude-fields",
+        "-x",
+        help="Comma-separated list of fields to skip, for example: dc.contributor.author,dc.identifier.citation",
+    )
    args = parser.parse_args()

    return args
@ -34,22 +67,22 @@ def run(argv):
    # Read all fields as strings so dates don't get converted from 1998 to 1998.0
    df = pd.read_csv(args.input_file, dtype=str)

-    for column in df.columns.values.tolist():
+    for column in df.columns:
        # Check if the user requested to skip any fields
        if args.exclude_fields:
            skip = False
            # Split the list of excludes on ',' so we can test exact matches
            # rather than fuzzy matches with regexes or "if word in string"
-            for exclude in args.exclude_fields.split(','):
+            for exclude in args.exclude_fields.split(","):
                if column == exclude and skip is False:
                    skip = True
            if skip:
-                print(f'Skipping {column}')
+                print(f"Skipping {column}")

                continue

        # Fix: whitespace
-        df[column] = df[column].apply(fix.whitespace)
+        df[column] = df[column].apply(fix.whitespace, field_name=column)

        # Fix: newlines
        if args.unsafe_fixes:
@ -58,59 +91,82 @@ def run(argv):
        # Fix: missing space after comma. Only run on author and citation
        # fields for now, as this problem is mostly an issue in names.
        if args.unsafe_fixes:
-            match = re.match(r'^.*?(author|citation).*$', column)
+            match = re.match(r"^.*?(author|citation).*$", column)
            if match is not None:
                df[column] = df[column].apply(fix.comma_space, field_name=column)

+        # Fix: perform Unicode normalization (NFC) to convert decomposed
+        # characters into their canonical forms.
+        if args.unsafe_fixes:
+            df[column] = df[column].apply(fix.normalize_unicode, field_name=column)
+
        # Fix: unnecessary Unicode
        df[column] = df[column].apply(fix.unnecessary_unicode)

        # Check: invalid multi-value separator
-        df[column] = df[column].apply(check.separators)
+        df[column] = df[column].apply(check.separators, field_name=column)

        # Check: suspicious characters
        df[column] = df[column].apply(check.suspicious_characters, field_name=column)

        # Fix: invalid multi-value separator
        if args.unsafe_fixes:
-            df[column] = df[column].apply(fix.separators)
+            df[column] = df[column].apply(fix.separators, field_name=column)
            # Run whitespace fix again after fixing invalid separators
-            df[column] = df[column].apply(fix.whitespace)
+            df[column] = df[column].apply(fix.whitespace, field_name=column)

        # Fix: duplicate metadata values
-        df[column] = df[column].apply(fix.duplicates)
+        df[column] = df[column].apply(fix.duplicates, field_name=column)

        # Check: invalid AGROVOC subject
        if args.agrovoc_fields:
            # Identify fields the user wants to validate against AGROVOC
-            for field in args.agrovoc_fields.split(','):
+            for field in args.agrovoc_fields.split(","):
                if column == field:
                    df[column] = df[column].apply(check.agrovoc, field_name=column)

        # Check: invalid language
-        match = re.match(r'^.*?language.*$', column)
+        match = re.match(r"^.*?language.*$", column)
        if match is not None:
            df[column] = df[column].apply(check.language)

        # Check: invalid ISSN
-        match = re.match(r'^.*?issn.*$', column)
+        match = re.match(r"^.*?issn.*$", column)
        if match is not None:
            df[column] = df[column].apply(check.issn)

        # Check: invalid ISBN
-        match = re.match(r'^.*?isbn.*$', column)
+        match = re.match(r"^.*?isbn.*$", column)
        if match is not None:
            df[column] = df[column].apply(check.isbn)

        # Check: invalid date
-        match = re.match(r'^.*?date.*$', column)
+        match = re.match(r"^.*?date.*$", column)
        if match is not None:
            df[column] = df[column].apply(check.date, field_name=column)

        # Check: filename extension
-        if column == 'filename':
+        if column == "filename":
            df[column] = df[column].apply(check.filename_extension)

+    ##
+    # Perform some checks on rows so we can consider items as a whole rather
+    # than simple on a field-by-field basis. This allows us to check whether
+    # the language used in the title and abstract matches the language indi-
+    # cated in the language field, for example.
+    #
+    # This is slower and apparently frowned upon in the Pandas community be-
+    # cause it requires iterating over rows rather than using apply over a
+    # column. For now it will have to do.
+    ##
+
+    if args.experimental_checks:
+        # Transpose the DataFrame so we can consider each row as a column
+        df_transposed = df.T
+
+        for column in df_transposed.columns:
+            experimental.correct_language(df_transposed[column])
+
    # Write
    df.to_csv(args.output_file, index=False)

--- a/csv_metadata_quality/check.py
+++ b/csv_metadata_quality/check.py
@ -18,10 +18,10 @@ def issn(field):
        return

    # Try to split multi-value field on "||" separator
-    for value in field.split('||'):
+    for value in field.split("||"):

        if not issn.is_valid(value):
-            print(f'Invalid ISSN: {value}')
+            print(f"Invalid ISSN: {value}")

    return field

@ -43,15 +43,15 @@ def isbn(field):
        return

    # Try to split multi-value field on "||" separator
-    for value in field.split('||'):
+    for value in field.split("||"):

        if not isbn.is_valid(value):
-            print(f'Invalid ISBN: {value}')
+            print(f"Invalid ISBN: {value}")

    return field


-def separators(field):
+def separators(field, field_name):
    """Check for invalid multi-value separators (ie "|" or "|||").

    Prints the field with the invalid multi-value separator.
@ -64,13 +64,13 @@ def separators(field):
        return

    # Try to split multi-value field on "||" separator
-    for value in field.split('||'):
+    for value in field.split("||"):

        # After splitting, see if there are any remaining "|" characters
-        match = re.findall(r'^.*?\|.*$', value)
+        match = re.findall(r"^.*?\|.*$", value)

        if match:
-            print(f'Invalid multi-value separator: {field}')
+            print(f"Invalid multi-value separator ({field_name}): {field}")

    return field

@ -88,22 +88,22 @@ def date(field, field_name):
    from datetime import datetime

    if pd.isna(field):
-        print(f'Missing date ({field_name}).')
+        print(f"Missing date ({field_name}).")

        return

    # Try to split multi-value field on "||" separator
-    multiple_dates = field.split('||')
+    multiple_dates = field.split("||")

    # We don't allow multi-value date fields
    if len(multiple_dates) > 1:
-        print(f'Multiple dates not allowed ({field_name}): {field}')
+        print(f"Multiple dates not allowed ({field_name}): {field}")

        return field

    try:
        # Check if date is valid YYYY format
-        datetime.strptime(field, '%Y')
+        datetime.strptime(field, "%Y")

        return field
    except ValueError:
@ -111,7 +111,7 @@ def date(field, field_name):

    try:
        # Check if date is valid YYYY-MM format
-        datetime.strptime(field, '%Y-%m')
+        datetime.strptime(field, "%Y-%m")

        return field
    except ValueError:
@ -119,11 +119,11 @@ def date(field, field_name):

    try:
        # Check if date is valid YYYY-MM-DD format
-        datetime.strptime(field, '%Y-%m-%d')
+        datetime.strptime(field, "%Y-%m-%d")

        return field
    except ValueError:
-        print(f'Invalid date ({field_name}): {field}')
+        print(f"Invalid date ({field_name}): {field}")

        return field

@ -140,7 +140,7 @@ def suspicious_characters(field, field_name):
        return

    # List of suspicious characters, for example:  ́ˆ~`
-    suspicious_characters = ['\u00B4', '\u02C6', '\u007E', '\u0060']
+    suspicious_characters = ["\u00B4", "\u02C6", "\u007E", "\u0060"]

    for character in suspicious_characters:
        # Find the position of the suspicious character in the string
@ -156,14 +156,16 @@ def suspicious_characters(field, field_name):
            # character and spanning enough of the rest to give a preview,
            # but not too much to cause the line to break in terminals with
            # a default of 80 characters width.
-            suspicious_character_msg = f'Suspicious character ({field_name}): {field_subset}'
-            print(f'{suspicious_character_msg:1.80}')
+            suspicious_character_msg = (
+                f"Suspicious character ({field_name}): {field_subset}"
+            )
+            print(f"{suspicious_character_msg:1.80}")

    return field


 def language(field):
-    """Check if a language is valid ISO 639-2 or ISO 639-3.
+    """Check if a language is valid ISO 639-1 (alpha 2) or ISO 639-3 (alpha 3).

    Prints the value if it is invalid.
    """
@ -177,22 +179,22 @@ def language(field):
    # need to handle "Other" values here...

    # Try to split multi-value field on "||" separator
-    for value in field.split('||'):
+    for value in field.split("||"):

        # After splitting, check if language value is 2 or 3 characters so we
-        # can check it against ISO 639-2 or ISO 639-3 accordingly.
+        # can check it against ISO 639-1 or ISO 639-3 accordingly.
        if len(value) == 2:
            if not languages.get(alpha_2=value):
-                print(f'Invalid ISO 639-2 language: {value}')
+                print(f"Invalid ISO 639-1 language: {value}")

                pass
        elif len(value) == 3:
            if not languages.get(alpha_3=value):
-                print(f'Invalid ISO 639-3 language: {value}')
+                print(f"Invalid ISO 639-3 language: {value}")

                pass
        else:
-            print(f'Invalid language: {value}')
+            print(f"Invalid language: {value}")

    return field

@ -219,25 +221,28 @@ def agrovoc(field, field_name):
    if pd.isna(field):
        return

+    # enable transparent request cache with thirty days expiry
+    expire_after = timedelta(days=30)
+    requests_cache.install_cache(
+        "agrovoc-response-cache", expire_after=expire_after
+    )
+
+    # prune old cache entries
+    requests_cache.core.remove_expired_responses()
+
    # Try to split multi-value field on "||" separator
-    for value in field.split('||'):
-        request_url = f'http://agrovoc.uniroma2.it/agrovoc/rest/v1/agrovoc/search?query={value}'
+    for value in field.split("||"):
+        request_url = "http://agrovoc.uniroma2.it/agrovoc/rest/v1/agrovoc/search"
+        request_params = {"query": value}

-        # enable transparent request cache with thirty days expiry
-        expire_after = timedelta(days=30)
-        requests_cache.install_cache('agrovoc-response-cache', expire_after=expire_after)
-
-        request = requests.get(request_url)
-
-        # prune old cache entries
-        requests_cache.core.remove_expired_responses()
+        request = requests.get(request_url, params=request_params)

        if request.status_code == requests.codes.ok:
            data = request.json()

            # check if there are any results
-            if len(data['results']) == 0:
-                print(f'Invalid AGROVOC ({field_name}): {value}')
+            if len(data["results"]) == 0:
+                print(f"Invalid AGROVOC ({field_name}): {value}")

    return field

@ -260,10 +265,18 @@ def filename_extension(field):
        return

    # Try to split multi-value field on "||" separator
-    values = field.split('||')
+    values = field.split("||")

    # List of common filename extentions
-    common_filename_extensions = ['.pdf', '.doc', '.docx', '.ppt', '.pptx', '.xls', '.xlsx']
+    common_filename_extensions = [
+        ".pdf",
+        ".doc",
+        ".docx",
+        ".ppt",
+        ".pptx",
+        ".xls",
+        ".xlsx",
+    ]

    # Iterate over all values
    for value in values:
@ -272,7 +285,7 @@ def filename_extension(field):

        for filename_extension in common_filename_extensions:
            # Check for extension at the end of the filename
-            pattern = re.escape(filename_extension) + r'$'
+            pattern = re.escape(filename_extension) + r"$"
            match = re.search(pattern, value, re.IGNORECASE)

            if match is not None:
@ -282,6 +295,6 @@ def filename_extension(field):
                break

        if filename_extension_match is False:
-            print(f'Filename with uncommon extension: {value}')
+            print(f"Filename with uncommon extension: {value}")

    return field
--- a/csv_metadata_quality/experimental.py
+++ b/csv_metadata_quality/experimental.py
@ -0,0 +1,95 @@
+import pandas as pd
+
+
+def correct_language(row):
+    """Analyze the text used in the title, abstract, and citation fields to pre-
+    dict the language being used and compare it with the item's dc.language.iso
+    field.
+
+    Function prints an error if the language field does not match the detected
+    language and returns the value in the language field if it does match.
+    """
+
+    from pycountry import languages
+    import langid
+    import re
+
+    # Initialize some variables at global scope so that we can set them in the
+    # loop scope below and still be able to access them afterwards.
+    language = ""
+    sample_strings = list()
+    title = None
+
+    # Iterate over the labels of the current row's values. Before we transposed
+    # the DataFrame these were the columns in the CSV, ie dc.title and dc.type.
+    for label in row.axes[0]:
+        # Skip fields with missing values
+        if pd.isna(row[label]):
+            continue
+
+        # Check if current row has multiple language values (separated by "||")
+        match = re.match(r"^.*?language.*$", label)
+        if match is not None:
+            # Skip fields with multiple language values
+            if "||" in row[label]:
+                return
+
+            language = row[label]
+
+        # Extract title if it is present
+        match = re.match(r"^.*?title.*$", label)
+        if match is not None:
+            title = row[label]
+            # Append title to sample strings
+            sample_strings.append(row[label])
+
+        # Extract abstract if it is present
+        match = re.match(r"^.*?abstract.*$", label)
+        if match is not None:
+            sample_strings.append(row[label])
+
+        # Extract citation if it is present
+        match = re.match(r"^.*?citation.*$", label)
+        if match is not None:
+            sample_strings.append(row[label])
+
+    # Make sure language is not blank and is valid ISO 639-1/639-3 before proceeding with language prediction
+    if language != "":
+        # Check language value like "es"
+        if len(language) == 2:
+            if not languages.get(alpha_2=language):
+                return
+        # Check language value like "spa"
+        elif len(language) == 3:
+            if not languages.get(alpha_3=language):
+                return
+        # Language value is something else like "Span", do not proceed
+        else:
+            return
+    # Language is blank, do not proceed
+    else:
+        return
+
+    # Concatenate all sample strings into one string
+    sample_text = " ".join(sample_strings)
+
+    # Restrict the langid detection space to reduce false positives
+    langid.set_languages(
+        ["ar", "de", "en", "es", "fr", "hi", "it", "ja", "ko", "pt", "ru", "vi", "zh"]
+    )
+    langid_classification = langid.classify(sample_text)
+
+    # langid returns an ISO 639-1 (alpha 2) representation of the detected language, but the current item's language field might be ISO 639-3 (alpha 3) so we should use a pycountry Language object to compare both represenations and give appropriate error messages that match the format used by in the input file.
+    detected_language = languages.get(alpha_2=langid_classification[0])
+    if len(language) == 2 and language != detected_language.alpha_2:
+        print(
+            f"Possibly incorrect language {language} (detected {detected_language.alpha_2}): {title}"
+        )
+
+    elif len(language) == 3 and language != detected_language.alpha_3:
+        print(
+            f"Possibly incorrect language {language} (detected {detected_language.alpha_3}): {title}"
+        )
+
+    else:
+        return language
--- a/csv_metadata_quality/fix.py
+++ b/csv_metadata_quality/fix.py
@ -1,8 +1,9 @@
-import pandas as pd
 import re

+import pandas as pd

-def whitespace(field):
+
+def whitespace(field, field_name):
    """Fix whitespace issues.

    Return string with leading, trailing, and consecutive whitespace trimmed.
@ -16,28 +17,28 @@ def whitespace(field):
    values = list()

    # Try to split multi-value field on "||" separator
-    for value in field.split('||'):
+    for value in field.split("||"):
        # Strip leading and trailing whitespace
        value = value.strip()

        # Replace excessive whitespace (>2) with one space
-        pattern = re.compile(r'\s{2,}')
+        pattern = re.compile(r"\s{2,}")
        match = re.findall(pattern, value)

        if match:
-            print(f'Excessive whitespace: {value}')
-            value = re.sub(pattern, ' ', value)
+            print(f"Removing excessive whitespace ({field_name}): {value}")
+            value = re.sub(pattern, " ", value)

        # Save cleaned value
        values.append(value)

    # Create a new field consisting of all values joined with "||"
-    new_field = '||'.join(values)
+    new_field = "||".join(values)

    return new_field


-def separators(field):
+def separators(field, field_name):
    """Fix for invalid multi-value separators (ie "|")."""

    # Skip fields with missing values
@ -48,21 +49,21 @@ def separators(field):
    values = list()

    # Try to split multi-value field on "||" separator
-    for value in field.split('||'):
+    for value in field.split("||"):
        # After splitting, see if there are any remaining "|" characters
-        pattern = re.compile(r'\|')
+        pattern = re.compile(r"\|")
        match = re.findall(pattern, value)

        if match:
-            print(f'Fixing invalid multi-value separator: {value}')
+            print(f"Fixing invalid multi-value separator ({field_name}): {value}")

-            value = re.sub(pattern, '||', value)
+            value = re.sub(pattern, "||", value)

        # Save cleaned value
        values.append(value)

    # Create a new field consisting of all values joined with "||"
-    new_field = '||'.join(values)
+    new_field = "||".join(values)

    return new_field

@ -73,10 +74,10 @@ def unnecessary_unicode(field):
    Removes unnecessary Unicode characters like:
        - Zero-width space (U+200B)
        - Replacement character (U+FFFD)
-        - No-break space (U+00A0)

    Replaces unnecessary Unicode characters like:
        - Soft hyphen (U+00AD) → hyphen
+        - No-break space (U+00A0) → space

    Return string with characters removed or replaced.
    """
@ -86,41 +87,41 @@ def unnecessary_unicode(field):
        return

    # Check for zero-width space characters (U+200B)
-    pattern = re.compile(r'\u200B')
+    pattern = re.compile(r"\u200B")
    match = re.findall(pattern, field)

    if match:
-        print(f'Removing unnecessary Unicode (U+200B): {field}')
-        field = re.sub(pattern, '', field)
+        print(f"Removing unnecessary Unicode (U+200B): {field}")
+        field = re.sub(pattern, "", field)

    # Check for replacement characters (U+FFFD)
-    pattern = re.compile(r'\uFFFD')
+    pattern = re.compile(r"\uFFFD")
    match = re.findall(pattern, field)

    if match:
-        print(f'Removing unnecessary Unicode (U+FFFD): {field}')
-        field = re.sub(pattern, '', field)
+        print(f"Removing unnecessary Unicode (U+FFFD): {field}")
+        field = re.sub(pattern, "", field)

    # Check for no-break spaces (U+00A0)
-    pattern = re.compile(r'\u00A0')
+    pattern = re.compile(r"\u00A0")
    match = re.findall(pattern, field)

    if match:
-        print(f'Removing unnecessary Unicode (U+00A0): {field}')
-        field = re.sub(pattern, '', field)
+        print(f"Replacing unnecessary Unicode (U+00A0): {field}")
+        field = re.sub(pattern, " ", field)

    # Check for soft hyphens (U+00AD), sometimes preceeded with a normal hyphen
-    pattern = re.compile(r'\u002D*?\u00AD')
+    pattern = re.compile(r"\u002D*?\u00AD")
    match = re.findall(pattern, field)

    if match:
-        print(f'Replacing unnecessary Unicode (U+00AD): {field}')
-        field = re.sub(pattern, '-', field)
+        print(f"Replacing unnecessary Unicode (U+00AD): {field}")
+        field = re.sub(pattern, "-", field)

    return field


-def duplicates(field):
+def duplicates(field, field_name):
    """Remove duplicate metadata values."""

    # Skip fields with missing values
@ -128,7 +129,7 @@ def duplicates(field):
        return

    # Try to split multi-value field on "||" separator
-    values = field.split('||')
+    values = field.split("||")

    # Initialize an empty list to hold the de-duplicated values
    new_values = list()
@ -139,10 +140,10 @@ def duplicates(field):
        if value not in new_values:
            new_values.append(value)
        else:
-            print(f'Dropping duplicate value: {value}')
+            print(f"Removing duplicate value ({field_name}): {value}")

    # Create a new field consisting of all values joined with "||"
-    new_field = '||'.join(new_values)
+    new_field = "||".join(new_values)

    return new_field

@ -169,11 +170,11 @@ def newlines(field):
        return

    # Check for Unix line feed (LF)
-    match = re.findall(r'\n', field)
+    match = re.findall(r"\n", field)

    if match:
-        print(f'Removing newline: {field}')
-        field = field.replace('\n', '')
+        print(f"Removing newline: {field}")
+        field = field.replace("\n", "")

    return field

@ -193,10 +194,34 @@ def comma_space(field, field_name):
        return

    # Check for comma followed by a word character
-    match = re.findall(r',\w', field)
+    match = re.findall(r",\w", field)

    if match:
-        print(f'Adding space after comma ({field_name}): {field}')
-        field = re.sub(r',(\w)', r', \1', field)
+        print(f"Adding space after comma ({field_name}): {field}")
+        field = re.sub(r",(\w)", r", \1", field)
+
+    return field
+
+
+def normalize_unicode(field, field_name):
+    """Fix occurrences of decomposed Unicode characters by normalizing them
+    with NFC to their canonical forms, for example:
+
+    Ouédraogo, Mathieu → Ouédraogo, Mathieu
+
+    Return normalized string.
+    """
+
+    from csv_metadata_quality.util import is_nfc
+    from unicodedata import normalize
+
+    # Skip fields with missing values
+    if pd.isna(field):
+        return
+
+    # Check if the current string is using normalized Unicode (NFC)
+    if not is_nfc(field):
+        print(f"Normalizing Unicode ({field_name}): {field}")
+        field = normalize("NFC", field)

    return field
--- a/csv_metadata_quality/util.py
+++ b/csv_metadata_quality/util.py
@ -0,0 +1,14 @@
+def is_nfc(field):
+    """Utility function to check whether a string is using normalized Unicode.
+    Python's built-in unicodedata library has the is_normalized() function, but
+    it was only introduced in Python 3.8. By using a simple utility function we
+    are able to run on Python >= 3.6 again.
+
+    See: https://docs.python.org/3/library/unicodedata.html
+
+    Return boolean.
+    """
+
+    from unicodedata import normalize
+
+    return field == normalize("NFC", field)
--- a/csv_metadata_quality/version.py
+++ b/csv_metadata_quality/version.py
@ -1 +1 @@
-VERSION = '0.2.2'
+VERSION = "0.4.2"
--- a/data/test.csv
+++ b/data/test.csv
@ -1,4 +1,4 @@
-dc.contributor.author,birthdate,dc.identifier.issn,dc.identifier.isbn,dc.language.iso,dc.subject,cg.coverage.country,filename
+dc.title,dc.date.issued,dc.identifier.issn,dc.identifier.isbn,dc.language.iso,dc.subject,cg.coverage.country,filename
 Leading space,2019-07-29,,,,,,
 Trailing space ,2019-07-29,,,,,,
 Excessive  space,2019-07-29,,,,,,
@ -13,8 +13,8 @@ Multiple dates,2019-07-26||2019-01-10,,,,,,
 Invalid multi-value separator,2019-07-29,0378-5955|0024-9319,,,,,
 Unnecessary Unicode,2019-07-29,,,,,,
 Suspicious character||foreˆt,2019-07-29,,,,,,
-Invalid ISO 639-2 language,2019-07-29,,,jp,,,
-Invalid ISO 639-3 language,2019-07-29,,,chi,,,
+Invalid ISO 639-1 (alpha 2) language,2019-07-29,,,jp,,,
+Invalid ISO 639-3 (alpha 3) language,2019-07-29,,,chi,,,
 Invalid language,2019-07-29,,,Span,,,
 Invalid AGROVOC subject,2019-07-29,,,,FOREST,,
 Newline (LF),2019-07-30,,,,"TANZA
@ -24,3 +24,7 @@ Invalid country,2019-08-01,,,,,KENYAA,
 Uncommon filename extension,2019-08-10,,,,,,file.pdf.lck
 Unneccesary unicode (U+002D + U+00AD),2019-08-10,,978-92-9043-823-6,,,,
 "Missing space,after comma",2019-08-27,,,,,,
+Incorrect ISO 639-1 language,2019-09-26,,,es,,,
+Incorrect ISO 639-3 language,2019-09-26,,,spa,,,
+Composéd Unicode,2020-01-14,,,,,,
+Decomposéd Unicode,2020-01-14,,,,,,
--- a/poetry.lock
+++ b/poetry.lock
--- a/pyproject.toml
+++ b/pyproject.toml
@ -0,0 +1,31 @@
+[tool.poetry]
+name = "csv-metadata-quality"
+version = "0.4.2"
+description="A simple, but opinionated CSV quality checking and fixing pipeline for CSVs in the DSpace ecosystem."
+authors = ["Alan Orth <alan.orth@gmail.com>"]
+license="GPL-3.0-only"
+repository = "https://github.com/ilri/csv-metadata-quality"
+homepage = "https://github.com/ilri/csv-metadata-quality"
+
+[tool.poetry.dependencies]
+python = "^3.8"
+pandas = "^1.0.4"
+python-stdnum = "^1.13"
+xlrd = "^1.2.0"
+requests = "^2.23.0"
+requests-cache = "^0.5.2"
+pycountry = "^19.8.18"
+langid = "^1.1.6"
+
+[tool.poetry.dev-dependencies]
+pytest = "^5.4.2"
+ipython = "^7.15.0"
+flake8 = "^3.8.2"
+pytest-clarity = "^0.3.0-alpha.0"
+black = "^19.10b0"
+isort = "^4.3.21"
+csvkit = "^1.0.5"
+
+[build-system]
+requires = ["poetry>=0.12"]
+build-backend = "poetry.masonry.api"
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@ -1,32 +1,300 @@
-i https://pypi.org/simple
-atomicwrites==1.3.0
-attrs==19.1.0
-backcall==0.1.0
-decorator==4.4.0
-entrypoints==0.3
-flake8==3.7.8
-importlib-metadata==0.19
-ipython-genutils==0.2.0
-ipython==7.7.0
-jedi==0.14.1
-mccabe==0.6.1
-more-itertools==7.2.0
-packaging==19.1
-parso==0.5.1
-pexpect==4.7.0 ; sys_platform != 'win32'
-pickleshare==0.7.5
-pluggy==0.12.0
-prompt-toolkit==2.0.9
-ptyprocess==0.6.0
-py==1.8.0
-pycodestyle==2.5.0
-pyflakes==2.1.1
-pygments==2.4.2
-pyparsing==2.4.2
-pytest-clarity==0.2.0a1
-pytest==5.0.1
-six==1.12.0
-termcolor==1.1.0
-traitlets==4.3.2
-wcwidth==0.1.7
-zipp==0.5.2
+agate==1.6.1 \
+    --hash=sha256:48d6f80b35611c1ba25a642cbc5b90fcbdeeb2a54711c4a8d062ee2809334d1c \
+    --hash=sha256:c93aaa500b439d71e4a5cf088d0006d2ce2c76f1950960c8843114e5f361dfd3
+agate-dbf==0.2.1 \
+    --hash=sha256:00c93c498ec9a04cc587bf63dd7340e67e2541f0df4c9a7259d7cb3dd4ce372f \
+    --hash=sha256:f618fadb413d41468c90d72fca945681d82d9e4d1b3d89f9bda52e607b828c0b
+agate-excel==0.2.3 \
+    --hash=sha256:8f255ef2c87c436b7132049e1dd86c8e08bf82d8c773aea86f3069b461a17d52
+agate-sql==0.5.4 \
+    --hash=sha256:9277490ba8b8e7c747a9ae3671f52fe486784b48d4a14e78ca197fb0e36f281b
+appdirs==1.4.4 \
+    --hash=sha256:a841dacd6b99318a741b166adb07e19ee71a274450e68237b4650ca1055ab128 \
+    --hash=sha256:7d5d0167b2b1ba821647616af46a749d1c653740dd0d2415100fe26e27afdf41
+appnope==0.1.0; sys_platform == "darwin" \
+    --hash=sha256:5b26757dc6f79a3b7dc9fab95359328d5747fcb2409d331ea66d0272b90ab2a0 \
+    --hash=sha256:8b995ffe925347a2138d7ac0fe77155e4311a0ea6d6da4f5128fe4b3cbe5ed71
+atomicwrites==1.4.0; sys_platform == "win32" \
+    --hash=sha256:6d1784dea7c0c8d4a5172b6c620f40b6e4cbfdf96d783691f2e1302a7b88e197 \
+    --hash=sha256:ae70396ad1a434f9c7046fd2dd196fc04b12f9e91ffb859164193be8b6168a7a
+attrs==19.3.0 \
+    --hash=sha256:08a96c641c3a74e44eb59afb61a24f2cb9f4d7188748e76ba4bb5edfa3cb7d1c \
+    --hash=sha256:f7b7ce16570fe9965acd6d30101a28f62fb4a7f9e926b3bbc9b61f8b04247e72
+babel==2.8.0 \
+    --hash=sha256:d670ea0b10f8b723672d3a6abeb87b565b244da220d76b4dba1b66269ec152d4 \
+    --hash=sha256:1aac2ae2d0d8ea368fa90906567f5c08463d98ade155c0c4bfedd6a0f7160e38
+backcall==0.2.0 \
+    --hash=sha256:fbbce6a29f263178a1f7915c1940bde0ec2b2a967566fe1c65c1dfb7422bd255 \
+    --hash=sha256:5cbdbf27be5e7cfadb448baf0aa95508f91f2bbc6c6437cd9cd06e2a4c215e1e
+black==19.10b0 \
+    --hash=sha256:1b30e59be925fafc1ee4565e5e08abef6b03fe455102883820fe5ee2e4734e0b \
+    --hash=sha256:c2edb73a08e9e0e6f65a0e6af18b059b8b1cdd5bef997d7a0b181df93dc81539
+certifi==2020.6.20 \
+    --hash=sha256:8fc0819f1f30ba15bdb34cceffb9ef04d99f420f68eb75d901e9560b8749fc41 \
+    --hash=sha256:5930595817496dd21bb8dc35dad090f1c2cd0adfaf21204bf6732ca5d8ee34d3
+chardet==3.0.4 \
+    --hash=sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691 \
+    --hash=sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae
+click==7.1.2 \
+    --hash=sha256:dacca89f4bfadd5de3d7489b7c8a566eee0d3676333fbb50030263894c38c0dc \
+    --hash=sha256:d2b5255c7c6349bc1bd1e59e08cd12acbbd63ce649f2588755783aa94dfb6b1a
+colorama==0.4.3; sys_platform == "win32" \
+    --hash=sha256:7d73d2a99753107a36ac6b455ee49046802e59d9d076ef8e47b61499fa29afff \
+    --hash=sha256:e96da0d330793e2cb9485e9ddfd918d456036c7149416295932478192f4436a1
+csvkit==1.0.5 \
+    --hash=sha256:7bd390f4d300e45dc9ed67a32af762a916bae7d9a85087a10fd4f64ce65fd5b9
+dbfread==2.0.7 \
+    --hash=sha256:f604def58c59694fa0160d7be5d0b8d594467278d2bb6a47d46daf7162c84cec \
+    --hash=sha256:07c8a9af06ffad3f6f03e8fe91ad7d2733e31a26d2b72c4dd4cfbae07ee3b73d
+decorator==4.4.2 \
+    --hash=sha256:41fa54c2a0cc4ba648be4fd43cff00aedf5b9465c9bf18d64325bc225f08f760 \
+    --hash=sha256:e3a62f0520172440ca0dcc823749319382e377f37f140a0b99ef45fecb84bfe7
+et-xmlfile==1.0.1 \
+    --hash=sha256:614d9722d572f6246302c4491846d2c393c199cfa4edc9af593437691683335b
+flake8==3.8.3 \
+    --hash=sha256:15e351d19611c887e482fb960eae4d44845013cc142d42896e9862f775d8cf5c \
+    --hash=sha256:f04b9fcbac03b0a3e58c0ab3a0ecc462e023a9faf046d57794184028123aa208
+idna==2.10 \
+    --hash=sha256:b97d804b1e9b523befed77c48dacec60e6dcb0b5391d57af6a65a312a90648c0 \
+    --hash=sha256:b307872f855b18632ce0c21c5e45be78c0ea7ae4c15c828c20788b26921eb3f6
+ipython==7.16.1 \
+    --hash=sha256:2dbcc8c27ca7d3cfe4fcdff7f45b27f9a8d3edfa70ff8024a71c7a8eb5f09d64 \
+    --hash=sha256:9f4fcb31d3b2c533333893b9172264e4821c1ac91839500f31bd43f2c59b3ccf
+ipython-genutils==0.2.0 \
+    --hash=sha256:72dd37233799e619666c9f639a9da83c34013a73e8bbc79a7a6348d93c61fab8 \
+    --hash=sha256:eb2e116e75ecef9d4d228fdc66af54269afa26ab4463042e33785b887c628ba8
+isodate==0.6.0 \
+    --hash=sha256:aa4d33c06640f5352aca96e4b81afd8ab3b47337cc12089822d6f322ac772c81 \
+    --hash=sha256:2e364a3d5759479cdb2d37cce6b9376ea504db2ff90252a2e5b7cc89cc9ff2d8
+isort==4.3.21 \
+    --hash=sha256:6e811fcb295968434526407adb8796944f1988c5b65e8139058f2014cbe100fd \
+    --hash=sha256:54da7e92468955c4fceacd0c86bd0ec997b0e1ee80d97f67c35a78b719dccab1
+jdcal==1.4.1 \
+    --hash=sha256:1abf1305fce18b4e8aa248cf8fe0c56ce2032392bc64bbd61b5dff2a19ec8bba \
+    --hash=sha256:472872e096eb8df219c23f2689fc336668bdb43d194094b5cc1707e1640acfc8
+jedi==0.17.1 \
+    --hash=sha256:1ddb0ec78059e8e27ec9eb5098360b4ea0a3dd840bedf21415ea820c21b40a22 \
+    --hash=sha256:807d5d4f96711a2bcfdd5dfa3b1ae6d09aa53832b182090b222b5efb81f52f63
+langid==1.1.6 \
+    --hash=sha256:044bcae1912dab85c33d8e98f2811b8f4ff1213e5e9a9e9510137b84da2cb293
+leather==0.3.3 \
+    --hash=sha256:e0bb36a6d5f59fbf3c1a6e75e7c8bee29e67f06f5b48c0134407dde612eba5e2 \
+    --hash=sha256:076d1603b5281488285718ce1a5ce78cf1027fe1e76adf9c548caf83c519b988
+mccabe==0.6.1 \
+    --hash=sha256:ab8a6258860da4b6677da4bd2fe5dc2c659cff31b3ee4f7f5d64e79735b80d42 \
+    --hash=sha256:dd8d182285a0fe56bace7f45b5e7d1a6ebcbf524e8f3bd87eb0f125271b8831f
+more-itertools==8.4.0 \
+    --hash=sha256:68c70cc7167bdf5c7c9d8f6954a7837089c6a36bf565383919bb595efb8a17e5 \
+    --hash=sha256:b78134b2063dd214000685165d81c154522c3ee0a1c0d4d113c80361c234c5a2
+numpy==1.19.0 \
+    --hash=sha256:63d971bb211ad3ca37b2adecdd5365f40f3b741a455beecba70fd0dde8b2a4cb \
+    --hash=sha256:b6aaeadf1e4866ca0fdf7bb4eed25e521ae21a7947c59f78154b24fc7abbe1dd \
+    --hash=sha256:13af0184177469192d80db9bd02619f6fa8b922f9f327e077d6f2a6acb1ce1c0 \
+    --hash=sha256:356f96c9fbec59974a592452ab6a036cd6f180822a60b529a975c9467fcd5f23 \
+    --hash=sha256:fa1fe75b4a9e18b66ae7f0b122543c42debcf800aaafa0212aaff3ad273c2596 \
+    --hash=sha256:cbe326f6d364375a8e5a8ccb7e9cd73f4b2f6dc3b2ed205633a0db8243e2a96a \
+    --hash=sha256:a2e3a39f43f0ce95204beb8fe0831199542ccab1e0c6e486a0b4947256215632 \
+    --hash=sha256:7b852817800eb02e109ae4a9cef2beda8dd50d98b76b6cfb7b5c0099d27b52d4 \
+    --hash=sha256:d97a86937cf9970453c3b62abb55a6475f173347b4cde7f8dcdb48c8e1b9952d \
+    --hash=sha256:a86c962e211f37edd61d6e11bb4df7eddc4a519a38a856e20a6498c319efa6b0 \
+    --hash=sha256:d34fbb98ad0d6b563b95de852a284074514331e6b9da0a9fc894fb1cdae7a79e \
+    --hash=sha256:658624a11f6e1c252b2cd170d94bf28c8f9410acab9f2fd4369e11e1cd4e1aaf \
+    --hash=sha256:4d054f013a1983551254e2379385e359884e5af105e3efe00418977d02f634a7 \
+    --hash=sha256:26a45798ca2a4e168d00de75d4a524abf5907949231512f372b217ede3429e98 \
+    --hash=sha256:3c40c827d36c6d1c3cf413694d7dc843d50997ebffbc7c87d888a203ed6403a7 \
+    --hash=sha256:be62aeff8f2f054eff7725f502f6228298891fd648dc2630e03e44bf63e8cee0 \
+    --hash=sha256:dd53d7c4a69e766e4900f29db5872f5824a06827d594427cf1a4aa542818b796 \
+    --hash=sha256:30a59fb41bb6b8c465ab50d60a1b298d1cd7b85274e71f38af5a75d6c475d2d2 \
+    --hash=sha256:df1889701e2dfd8ba4dc9b1a010f0a60950077fb5242bb92c8b5c7f1a6f2668a \
+    --hash=sha256:33c623ef9ca5e19e05991f127c1be5aeb1ab5cdf30cb1c5cf3960752e58b599b \
+    --hash=sha256:26f509450db547e4dfa3ec739419b31edad646d21fb8d0ed0734188b35ff6b27 \
+    --hash=sha256:7b57f26e5e6ee2f14f960db46bd58ffdca25ca06dd997729b1b179fddd35f5a3 \
+    --hash=sha256:a8705c5073fe3fcc297fb8e0b31aa794e05af6a329e81b7ca4ffecab7f2b95ef \
+    --hash=sha256:c2edbb783c841e36ca0fa159f0ae97a88ce8137fb3a6cd82eae77349ba4b607b \
+    --hash=sha256:8cde829f14bd38f6da7b2954be0f2837043e8b8d7a9110ec5e318ae6bf706610 \
+    --hash=sha256:76766cc80d6128750075378d3bb7812cf146415bd29b588616f72c943c00d598
+openpyxl==3.0.4 \
+    --hash=sha256:6e62f058d19b09b95d20ebfbfb04857ad08d0833190516c1660675f699c6186f \
+    --hash=sha256:d88dd1480668019684c66cfff3e52a5de4ed41e9df5dd52e008cbf27af0dbf87
+packaging==20.4 \
+    --hash=sha256:998416ba6962ae7fbd6596850b80e17859a5753ba17c32284f67bfff33784181 \
+    --hash=sha256:4357f74f47b9c12db93624a82154e9b120fa8293699949152b22065d556079f8
+pandas==1.0.5 \
+    --hash=sha256:faa42a78d1350b02a7d2f0dbe3c80791cf785663d6997891549d0f86dc49125e \
+    --hash=sha256:9c31d52f1a7dd2bb4681d9f62646c7aa554f19e8e9addc17e8b1b20011d7522d \
+    --hash=sha256:8778a5cc5a8437a561e3276b85367412e10ae9fff07db1eed986e427d9a674f8 \
+    --hash=sha256:9871ef5ee17f388f1cb35f76dc6106d40cb8165c562d573470672f4cdefa59ef \
+    --hash=sha256:35b670b0abcfed7cad76f2834041dcf7ae47fd9b22b63622d67cdc933d79f453 \
+    --hash=sha256:c9410ce8a3dee77653bc0684cfa1535a7f9c291663bd7ad79e39f5ab58f67ab3 \
+    --hash=sha256:02f1e8f71cd994ed7fcb9a35b6ddddeb4314822a0e09a9c5b2d278f8cb5d4096 \
+    --hash=sha256:b3c4f93fcb6e97d993bf87cdd917883b7dab7d20c627699f360a8fb49e9e0b91 \
+    --hash=sha256:5759edf0b686b6f25a5d4a447ea588983a33afc8a0081a0954184a4a87fd0dd7 \
+    --hash=sha256:ab8173a8efe5418bbe50e43f321994ac6673afc5c7c4839014cf6401bbdd0705 \
+    --hash=sha256:13f75fb18486759da3ff40f5345d9dd20e7d78f2a39c5884d013456cec9876f0 \
+    --hash=sha256:5a7cf6044467c1356b2b49ef69e50bf4d231e773c3ca0558807cdba56b76820b \
+    --hash=sha256:ae961f1f0e270f1e4e2273f6a539b2ea33248e0e3a11ffb479d757918a5e03a9 \
+    --hash=sha256:f69e0f7b7c09f1f612b1f8f59e2df72faa8a6b41c5a436dde5b615aaf948f107 \
+    --hash=sha256:4c73f373b0800eb3062ffd13d4a7a2a6d522792fa6eb204d67a4fad0a40f03dc \
+    --hash=sha256:69c5d920a0b2a9838e677f78f4dde506b95ea8e4d30da25859db6469ded84fa8
+parsedatetime==2.6 \
+    --hash=sha256:cb96edd7016872f58479e35879294258c71437195760746faffedb692aef000b \
+    --hash=sha256:4cb368fbb18a0b7231f4d76119165451c8d2e35951455dfee97c62a87b04d455
+parso==0.7.0 \
+    --hash=sha256:158c140fc04112dc45bca311633ae5033c2c2a7b732fa33d0955bad8152a8dd0 \
+    --hash=sha256:908e9fae2144a076d72ae4e25539143d40b8e3eafbaeae03c1bfe226f4cdf12c
+pathspec==0.8.0 \
+    --hash=sha256:7d91249d21749788d07a2d0f94147accd8f845507400749ea19c1ec9054a12b0 \
+    --hash=sha256:da45173eb3a6f2a5a487efba21f050af2b41948be6ab52b6a1e3ff22bb8b7061
+pexpect==4.8.0; sys_platform != "win32" \
+    --hash=sha256:0b48a55dcb3c05f3329815901ea4fc1537514d6ba867a152b581d69ae3710937 \
+    --hash=sha256:fc65a43959d153d0114afe13997d439c22823a27cefceb5ff35c2178c6784c0c
+pickleshare==0.7.5 \
+    --hash=sha256:9649af414d74d4df115d5d718f82acb59c9d418196b7b4290ed47a12ce62df56 \
+    --hash=sha256:87683d47965c1da65cdacaf31c8441d12b8044cdec9aca500cd78fc2c683afca
+pluggy==0.13.1 \
+    --hash=sha256:966c145cd83c96502c3c3868f50408687b38434af77734af1e9ca461a4081d2d \
+    --hash=sha256:15b2acde666561e1298d71b523007ed7364de07029219b604cf808bfa1c765b0
+prompt-toolkit==3.0.5 \
+    --hash=sha256:df7e9e63aea609b1da3a65641ceaf5bc7d05e0a04de5bd45d05dbeffbabf9e04 \
+    --hash=sha256:563d1a4140b63ff9dd587bda9557cffb2fe73650205ab6f4383092fb882e7dc8
+ptyprocess==0.6.0; sys_platform != "win32" \
+    --hash=sha256:d7cc528d76e76342423ca640335bd3633420dc1366f258cb31d05e865ef5ca1f \
+    --hash=sha256:923f299cc5ad920c68f2bc0bc98b75b9f838b93b599941a6b63ddbc2476394c0
+py==1.9.0 \
+    --hash=sha256:366389d1db726cd2fcfc79732e75410e5fe4d31db13692115529d34069a043c2 \
+    --hash=sha256:9ca6883ce56b4e8da7e79ac18787889fa5206c79dcc67fb065376cd2fe03f342
+pycodestyle==2.6.0 \
+    --hash=sha256:2295e7b2f6b5bd100585ebcb1f616591b652db8a741695b3d8f5d28bdc934367 \
+    --hash=sha256:c58a7d2815e0e8d7972bf1803331fb0152f867bd89adf8a01dfd55085434192e
+pycountry==19.8.18 \
+    --hash=sha256:3c57aa40adcf293d59bebaffbe60d8c39976fba78d846a018dc0c2ec9c6cb3cb
+pyflakes==2.2.0 \
+    --hash=sha256:0d94e0e05a19e57a99444b6ddcf9a6eb2e5c68d3ca1e98e90707af8152c90a92 \
+    --hash=sha256:35b2d75ee967ea93b55750aa9edbbf72813e06a66ba54438df2cfac9e3c27fc8
+pygments==2.6.1 \
+    --hash=sha256:ff7a40b4860b727ab48fad6360eb351cc1b33cbf9b15a0f689ca5353e9463324 \
+    --hash=sha256:647344a061c249a3b74e230c739f434d7ea4d8b1d5f3721bc0f3558049b38f44
+pyparsing==2.4.7 \
+    --hash=sha256:ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b \
+    --hash=sha256:c203ec8783bf771a155b207279b9bccb8dea02d8f0c9e5f8ead507bc3246ecc1
+pytest==5.4.3 \
+    --hash=sha256:5c0db86b698e8f170ba4582a492248919255fcd4c79b1ee64ace34301fb589a1 \
+    --hash=sha256:7979331bfcba207414f5e1263b5a0f8f521d0f457318836a7355531ed1a4c7d8
+pytest-clarity==0.3.0a0 \
+    --hash=sha256:5cc99e3d9b7969dfe17e5f6072d45a917c59d363b679686d3c958a1ded2e4dcf
+python-dateutil==2.8.1 \
+    --hash=sha256:73ebfe9dbf22e832286dafa60473e4cd239f8592f699aa5adaf10050e6e1823c \
+    --hash=sha256:75bb3f31ea686f1197762692a9ee6a7550b59fc6ca3a1f4b5d7e32fb98e2da2a
+python-slugify==4.0.1 \
+    --hash=sha256:69a517766e00c1268e5bbfc0d010a0a8508de0b18d30ad5a1ff357f8ae724270
+python-stdnum==1.13 \
+    --hash=sha256:120f83d33fb8b8be1b282f20dd755a892d5facf84f54fa21f75bbd2633128160 \
+    --hash=sha256:3d5d4430579cba88211d3ba4855a16faff235352a25a01d6ab70024686a75823
+pytimeparse==1.1.8 \
+    --hash=sha256:04b7be6cc8bd9f5647a6325444926c3ac34ee6bc7e69da4367ba282f076036bd \
+    --hash=sha256:e86136477be924d7e670646a98561957e8ca7308d44841e21f5ddea757556a0a
+pytz==2020.1 \
+    --hash=sha256:a494d53b6d39c3c6e44c3bec237336e14305e4f29bbf800b599253057fbb79ed \
+    --hash=sha256:c35965d010ce31b23eeb663ed3cc8c906275d6be1a34393a1d73a41febf4a048
+regex==2020.6.8 \
+    --hash=sha256:fbff901c54c22425a5b809b914a3bfaf4b9570eee0e5ce8186ac71eb2025191c \
+    --hash=sha256:112e34adf95e45158c597feea65d06a8124898bdeac975c9087fe71b572bd938 \
+    --hash=sha256:92d8a043a4241a710c1cf7593f5577fbb832cf6c3a00ff3fc1ff2052aff5dd89 \
+    --hash=sha256:bae83f2a56ab30d5353b47f9b2a33e4aac4de9401fb582b55c42b132a8ac3868 \
+    --hash=sha256:b2ba0f78b3ef375114856cbdaa30559914d081c416b431f2437f83ce4f8b7f2f \
+    --hash=sha256:95fa7726d073c87141f7bbfb04c284901f8328e2d430eeb71b8ffdd5742a5ded \
+    --hash=sha256:e3cdc9423808f7e1bb9c2e0bdb1c9dc37b0607b30d646ff6faf0d4e41ee8fee3 \
+    --hash=sha256:c78e66a922de1c95a208e4ec02e2e5cf0bb83a36ceececc10a72841e53fbf2bd \
+    --hash=sha256:08997a37b221a3e27d68ffb601e45abfb0093d39ee770e4257bd2f5115e8cb0a \
+    --hash=sha256:2f6f211633ee8d3f7706953e9d3edc7ce63a1d6aad0be5dcee1ece127eea13ae \
+    --hash=sha256:55b4c25cbb3b29f8d5e63aeed27b49fa0f8476b0d4e1b3171d85db891938cc3a \
+    --hash=sha256:89cda1a5d3e33ec9e231ece7307afc101b5217523d55ef4dc7fb2abd6de71ba3 \
+    --hash=sha256:690f858d9a94d903cf5cada62ce069b5d93b313d7d05456dbcd99420856562d9 \
+    --hash=sha256:1700419d8a18c26ff396b3b06ace315b5f2a6e780dad387e4c48717a12a22c29 \
+    --hash=sha256:654cb773b2792e50151f0e22be0f2b6e1c3a04c5328ff1d9d59c0398d37ef610 \
+    --hash=sha256:52e1b4bef02f4040b2fd547357a170fc1146e60ab310cdbdd098db86e929b387 \
+    --hash=sha256:cf59bbf282b627130f5ba68b7fa3abdb96372b24b66bdf72a4920e8153fc7910 \
+    --hash=sha256:5aaa5928b039ae440d775acea11d01e42ff26e1561c0ffcd3d805750973c6baf \
+    --hash=sha256:97712e0d0af05febd8ab63d2ef0ab2d0cd9deddf4476f7aa153f76feef4b2754 \
+    --hash=sha256:6ad8663c17db4c5ef438141f99e291c4d4edfeaacc0ce28b5bba2b0bf273d9b5 \
+    --hash=sha256:e9b64e609d37438f7d6e68c2546d2cb8062f3adb27e6336bc129b51be20773ac
+requests==2.24.0 \
+    --hash=sha256:fe75cc94a9443b9246fc7049224f75604b113c36acb93f87b80ed42c44cbb898 \
+    --hash=sha256:b3559a131db72c33ee969480840fff4bb6dd111de7dd27c8ee1f820f4f00231b
+requests-cache==0.5.2 \
+    --hash=sha256:813023269686045f8e01e2289cc1e7e9ae5ab22ddd1e2849a9093ab3ab7270eb \
+    --hash=sha256:81e13559baee64677a7d73b85498a5a8f0639e204517b5d05ff378e44a57831a
+six==1.15.0 \
+    --hash=sha256:8b74bedcbbbaca38ff6d7491d76f2b06b3592611af620f8426e82dddb04a5ced \
+    --hash=sha256:30639c035cdb23534cd4aa2dd52c3bf48f06e5f4a941509c8bafd8ce11080259
+sqlalchemy==1.3.18 \
+    --hash=sha256:f11c2437fb5f812d020932119ba02d9e2bc29a6eca01a055233a8b449e3e1e7d \
+    --hash=sha256:0ec575db1b54909750332c2e335c2bb11257883914a03bc5a3306a4488ecc772 \
+    --hash=sha256:f57be5673e12763dd400fea568608700a63ce1c6bd5bdbc3cc3a2c5fdb045274 \
+    --hash=sha256:8cac7bb373a5f1423e28de3fd5fc8063b9c8ffe8957dc1b1a59cb90453db6da1 \
+    --hash=sha256:adad60eea2c4c2a1875eb6305a0b6e61a83163f8e233586a4d6a55221ef984fe \
+    --hash=sha256:57aa843b783179ab72e863512e14bdcba186641daf69e4e3a5761d705dcc35b1 \
+    --hash=sha256:621f58cd921cd71ba6215c42954ffaa8a918eecd8c535d97befa1a8acad986dd \
+    --hash=sha256:fc728ece3d5c772c196fd338a99798e7efac7a04f9cb6416299a3638ee9a94cd \
+    --hash=sha256:736d41cfebedecc6f159fc4ac0769dc89528a989471dc1d378ba07d29a60ba1c \
+    --hash=sha256:427273b08efc16a85aa2b39892817e78e3ed074fcb89b2a51c4979bae7e7ba98 \
+    --hash=sha256:cbe1324ef52ff26ccde2cb84b8593c8bf930069dfc06c1e616f1bfd4e47f48a3 \
+    --hash=sha256:8fd452dc3d49b3cc54483e033de6c006c304432e6f84b74d7b2c68afa2569ae5 \
+    --hash=sha256:e89e0d9e106f8a9180a4ca92a6adde60c58b1b0299e1b43bd5e0312f535fbf33 \
+    --hash=sha256:6ac2558631a81b85e7fb7a44e5035347938b0a73f5fdc27a8566777d0792a6a4 \
+    --hash=sha256:87fad64529cde4f1914a5b9c383628e1a8f9e3930304c09cf22c2ae118a1280e \
+    --hash=sha256:e4624d7edb2576cd72bb83636cd71c8ce544d8e272f308bd80885056972ca299 \
+    --hash=sha256:89494df7f93b1836cae210c42864b292f9b31eeabca4810193761990dc689cce \
+    --hash=sha256:716754d0b5490bdcf68e1e4925edc02ac07209883314ad01a137642ddb2056f1 \
+    --hash=sha256:50c4ee32f0e1581828843267d8de35c3298e86ceecd5e9017dc45788be70a864 \
+    --hash=sha256:d98bc827a1293ae767c8f2f18be3bb5151fd37ddcd7da2a5f9581baeeb7a3fa1 \
+    --hash=sha256:0942a3a0df3f6131580eddd26d99071b48cfe5aaf3eab2783076fbc5a1c1882e \
+    --hash=sha256:16593fd748944726540cd20f7e83afec816c2ac96b082e26ae226e8f7e9688cf \
+    --hash=sha256:c26f95e7609b821b5f08a72dab929baa0d685406b953efd7c89423a511d5c413 \
+    --hash=sha256:512a85c3c8c3995cc91af3e90f38f460da5d3cade8dc3a229c8e0879037547c9 \
+    --hash=sha256:d05c4adae06bd0c7f696ae3ec8d993ed8ffcc4e11a76b1b35a5af8a099bd2284 \
+    --hash=sha256:109581ccc8915001e8037b73c29590e78ce74be49ca0a3630a23831f9e3ed6c7 \
+    --hash=sha256:8619b86cb68b185a778635be5b3e6018623c0761dde4df2f112896424aa27bd8 \
+    --hash=sha256:da2fb75f64792c1fc64c82313a00c728a7c301efe6a60b7a9fe35b16b4368ce7
+termcolor==1.1.0 \
+    --hash=sha256:1d6d69ce66211143803fbc56652b41d73b4a400a2891d7bf7a1cdf4c02de613b
+text-unidecode==1.3 \
+    --hash=sha256:bad6603bb14d279193107714b288be206cac565dfa49aa5b105294dd5c4aab93 \
+    --hash=sha256:1311f10e8b895935241623731c2ba64f4c455287888b18189350b67134a822e8
+toml==0.10.1 \
+    --hash=sha256:bda89d5935c2eac546d648028b9901107a595863cb36bae0c73ac804a9b4ce88 \
+    --hash=sha256:926b612be1e5ce0634a2ca03470f95169cf16f939018233a670519cb4ac58b0f
+traitlets==4.3.3 \
+    --hash=sha256:70b4c6a1d9019d7b4f6846832288f86998aa3b9207c6821f3578a6a6a467fe44 \
+    --hash=sha256:d023ee369ddd2763310e4c3eae1ff649689440d4ae59d7485eb4cfbbe3e359f7
+typed-ast==1.4.1 \
+    --hash=sha256:73d785a950fc82dd2a25897d525d003f6378d1cb23ab305578394694202a58c3 \
+    --hash=sha256:aaee9905aee35ba5905cfb3c62f3e83b3bec7b39413f0a7f19be4e547ea01ebb \
+    --hash=sha256:0c2c07682d61a629b68433afb159376e24e5b2fd4641d35424e462169c0a7919 \
+    --hash=sha256:4083861b0aa07990b619bd7ddc365eb7fa4b817e99cf5f8d9cf21a42780f6e01 \
+    --hash=sha256:269151951236b0f9a6f04015a9004084a5ab0d5f19b57de779f908621e7d8b75 \
+    --hash=sha256:24995c843eb0ad11a4527b026b4dde3da70e1f2d8806c99b7b4a7cf491612652 \
+    --hash=sha256:fe460b922ec15dd205595c9b5b99e2f056fd98ae8f9f56b888e7a17dc2b757e7 \
+    --hash=sha256:4e3e5da80ccbebfff202a67bf900d081906c358ccc3d5e3c8aea42fdfdfd51c1 \
+    --hash=sha256:249862707802d40f7f29f6e1aad8d84b5aa9e44552d2cc17384b209f091276aa \
+    --hash=sha256:8ce678dbaf790dbdb3eba24056d5364fb45944f33553dd5869b7580cdbb83614 \
+    --hash=sha256:c9e348e02e4d2b4a8b2eedb48210430658df6951fa484e59de33ff773fbd4b41 \
+    --hash=sha256:bcd3b13b56ea479b3650b82cabd6b5343a625b0ced5429e4ccad28a8973f301b \
+    --hash=sha256:d5d33e9e7af3b34a40dc05f498939f0ebf187f07c385fd58d591c533ad8562fe \
+    --hash=sha256:0666aa36131496aed8f7be0410ff974562ab7eeac11ef351def9ea6fa28f6355 \
+    --hash=sha256:d205b1b46085271b4e15f670058ce182bd1199e56b317bf2ec004b6a44f911f6 \
+    --hash=sha256:6daac9731f172c2a22ade6ed0c00197ee7cc1221aa84cfdf9c31defeb059a907 \
+    --hash=sha256:498b0f36cc7054c1fead3d7fc59d2150f4d5c6c56ba7fb150c013fbc683a8d2d \
+    --hash=sha256:715ff2f2df46121071622063fc7543d9b1fd19ebfc4f5c8895af64a77a8c852c \
+    --hash=sha256:fc0fea399acb12edbf8a628ba8d2312f583bdbdb3335635db062fa98cf71fca4 \
+    --hash=sha256:d43943ef777f9a1c42bf4e552ba23ac77a6351de620aa9acf64ad54933ad4d34 \
+    --hash=sha256:8c8aaad94455178e3187ab22c8b01a3837f8ee50e09cf31f1ba129eb293ec30b
+urllib3==1.25.9 \
+    --hash=sha256:88206b0eb87e6d677d424843ac5209e3fb9d0190d0ee169599165ec25e9d9115 \
+    --hash=sha256:3018294ebefce6572a474f0604c2021e33b3fd8006ecd11d62107a5d2a963527
+wcwidth==0.2.5 \
+    --hash=sha256:beb4802a9cebb9144e99086eff703a642a13d6a0052920003a230f3294bbe784 \
+    --hash=sha256:c4d647b99872929fdb7bdcaa4fbe7f01413ed3d98077df798530e5b04f116c83
+xlrd==1.2.0 \
+    --hash=sha256:e551fb498759fa3a5384a94ccd4c3c02eb7c00ea424426e212ac0c57be9dfbde \
+    --hash=sha256:546eb36cee8db40c3eaa46c351e67ffee6eeb5fa2650b71bc4c758a29a1b29b2
--- a/requirements.txt
+++ b/requirements.txt
@ -1,16 +1,81 @@
-i https://pypi.org/simple
-e .
-certifi==2019.6.16
-chardet==3.0.4
-idna==2.8
-numpy==1.17.0
-pandas==0.25.0
-pycountry==19.7.15
-python-dateutil==2.8.0
-python-stdnum==1.11
-pytz==2019.2
-requests-cache==0.5.0
-requests==2.22.0
-six==1.12.0
-urllib3==1.25.3
-xlrd==1.2.0
+certifi==2020.6.20 \
+    --hash=sha256:8fc0819f1f30ba15bdb34cceffb9ef04d99f420f68eb75d901e9560b8749fc41 \
+    --hash=sha256:5930595817496dd21bb8dc35dad090f1c2cd0adfaf21204bf6732ca5d8ee34d3
+chardet==3.0.4 \
+    --hash=sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691 \
+    --hash=sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae
+idna==2.10 \
+    --hash=sha256:b97d804b1e9b523befed77c48dacec60e6dcb0b5391d57af6a65a312a90648c0 \
+    --hash=sha256:b307872f855b18632ce0c21c5e45be78c0ea7ae4c15c828c20788b26921eb3f6
+langid==1.1.6 \
+    --hash=sha256:044bcae1912dab85c33d8e98f2811b8f4ff1213e5e9a9e9510137b84da2cb293
+numpy==1.19.0 \
+    --hash=sha256:63d971bb211ad3ca37b2adecdd5365f40f3b741a455beecba70fd0dde8b2a4cb \
+    --hash=sha256:b6aaeadf1e4866ca0fdf7bb4eed25e521ae21a7947c59f78154b24fc7abbe1dd \
+    --hash=sha256:13af0184177469192d80db9bd02619f6fa8b922f9f327e077d6f2a6acb1ce1c0 \
+    --hash=sha256:356f96c9fbec59974a592452ab6a036cd6f180822a60b529a975c9467fcd5f23 \
+    --hash=sha256:fa1fe75b4a9e18b66ae7f0b122543c42debcf800aaafa0212aaff3ad273c2596 \
+    --hash=sha256:cbe326f6d364375a8e5a8ccb7e9cd73f4b2f6dc3b2ed205633a0db8243e2a96a \
+    --hash=sha256:a2e3a39f43f0ce95204beb8fe0831199542ccab1e0c6e486a0b4947256215632 \
+    --hash=sha256:7b852817800eb02e109ae4a9cef2beda8dd50d98b76b6cfb7b5c0099d27b52d4 \
+    --hash=sha256:d97a86937cf9970453c3b62abb55a6475f173347b4cde7f8dcdb48c8e1b9952d \
+    --hash=sha256:a86c962e211f37edd61d6e11bb4df7eddc4a519a38a856e20a6498c319efa6b0 \
+    --hash=sha256:d34fbb98ad0d6b563b95de852a284074514331e6b9da0a9fc894fb1cdae7a79e \
+    --hash=sha256:658624a11f6e1c252b2cd170d94bf28c8f9410acab9f2fd4369e11e1cd4e1aaf \
+    --hash=sha256:4d054f013a1983551254e2379385e359884e5af105e3efe00418977d02f634a7 \
+    --hash=sha256:26a45798ca2a4e168d00de75d4a524abf5907949231512f372b217ede3429e98 \
+    --hash=sha256:3c40c827d36c6d1c3cf413694d7dc843d50997ebffbc7c87d888a203ed6403a7 \
+    --hash=sha256:be62aeff8f2f054eff7725f502f6228298891fd648dc2630e03e44bf63e8cee0 \
+    --hash=sha256:dd53d7c4a69e766e4900f29db5872f5824a06827d594427cf1a4aa542818b796 \
+    --hash=sha256:30a59fb41bb6b8c465ab50d60a1b298d1cd7b85274e71f38af5a75d6c475d2d2 \
+    --hash=sha256:df1889701e2dfd8ba4dc9b1a010f0a60950077fb5242bb92c8b5c7f1a6f2668a \
+    --hash=sha256:33c623ef9ca5e19e05991f127c1be5aeb1ab5cdf30cb1c5cf3960752e58b599b \
+    --hash=sha256:26f509450db547e4dfa3ec739419b31edad646d21fb8d0ed0734188b35ff6b27 \
+    --hash=sha256:7b57f26e5e6ee2f14f960db46bd58ffdca25ca06dd997729b1b179fddd35f5a3 \
+    --hash=sha256:a8705c5073fe3fcc297fb8e0b31aa794e05af6a329e81b7ca4ffecab7f2b95ef \
+    --hash=sha256:c2edbb783c841e36ca0fa159f0ae97a88ce8137fb3a6cd82eae77349ba4b607b \
+    --hash=sha256:8cde829f14bd38f6da7b2954be0f2837043e8b8d7a9110ec5e318ae6bf706610 \
+    --hash=sha256:76766cc80d6128750075378d3bb7812cf146415bd29b588616f72c943c00d598
+pandas==1.0.5 \
+    --hash=sha256:faa42a78d1350b02a7d2f0dbe3c80791cf785663d6997891549d0f86dc49125e \
+    --hash=sha256:9c31d52f1a7dd2bb4681d9f62646c7aa554f19e8e9addc17e8b1b20011d7522d \
+    --hash=sha256:8778a5cc5a8437a561e3276b85367412e10ae9fff07db1eed986e427d9a674f8 \
+    --hash=sha256:9871ef5ee17f388f1cb35f76dc6106d40cb8165c562d573470672f4cdefa59ef \
+    --hash=sha256:35b670b0abcfed7cad76f2834041dcf7ae47fd9b22b63622d67cdc933d79f453 \
+    --hash=sha256:c9410ce8a3dee77653bc0684cfa1535a7f9c291663bd7ad79e39f5ab58f67ab3 \
+    --hash=sha256:02f1e8f71cd994ed7fcb9a35b6ddddeb4314822a0e09a9c5b2d278f8cb5d4096 \
+    --hash=sha256:b3c4f93fcb6e97d993bf87cdd917883b7dab7d20c627699f360a8fb49e9e0b91 \
+    --hash=sha256:5759edf0b686b6f25a5d4a447ea588983a33afc8a0081a0954184a4a87fd0dd7 \
+    --hash=sha256:ab8173a8efe5418bbe50e43f321994ac6673afc5c7c4839014cf6401bbdd0705 \
+    --hash=sha256:13f75fb18486759da3ff40f5345d9dd20e7d78f2a39c5884d013456cec9876f0 \
+    --hash=sha256:5a7cf6044467c1356b2b49ef69e50bf4d231e773c3ca0558807cdba56b76820b \
+    --hash=sha256:ae961f1f0e270f1e4e2273f6a539b2ea33248e0e3a11ffb479d757918a5e03a9 \
+    --hash=sha256:f69e0f7b7c09f1f612b1f8f59e2df72faa8a6b41c5a436dde5b615aaf948f107 \
+    --hash=sha256:4c73f373b0800eb3062ffd13d4a7a2a6d522792fa6eb204d67a4fad0a40f03dc \
+    --hash=sha256:69c5d920a0b2a9838e677f78f4dde506b95ea8e4d30da25859db6469ded84fa8
+pycountry==19.8.18 \
+    --hash=sha256:3c57aa40adcf293d59bebaffbe60d8c39976fba78d846a018dc0c2ec9c6cb3cb
+python-dateutil==2.8.1 \
+    --hash=sha256:73ebfe9dbf22e832286dafa60473e4cd239f8592f699aa5adaf10050e6e1823c \
+    --hash=sha256:75bb3f31ea686f1197762692a9ee6a7550b59fc6ca3a1f4b5d7e32fb98e2da2a
+python-stdnum==1.13 \
+    --hash=sha256:120f83d33fb8b8be1b282f20dd755a892d5facf84f54fa21f75bbd2633128160 \
+    --hash=sha256:3d5d4430579cba88211d3ba4855a16faff235352a25a01d6ab70024686a75823
+pytz==2020.1 \
+    --hash=sha256:a494d53b6d39c3c6e44c3bec237336e14305e4f29bbf800b599253057fbb79ed \
+    --hash=sha256:c35965d010ce31b23eeb663ed3cc8c906275d6be1a34393a1d73a41febf4a048
+requests==2.24.0 \
+    --hash=sha256:fe75cc94a9443b9246fc7049224f75604b113c36acb93f87b80ed42c44cbb898 \
+    --hash=sha256:b3559a131db72c33ee969480840fff4bb6dd111de7dd27c8ee1f820f4f00231b
+requests-cache==0.5.2 \
+    --hash=sha256:813023269686045f8e01e2289cc1e7e9ae5ab22ddd1e2849a9093ab3ab7270eb \
+    --hash=sha256:81e13559baee64677a7d73b85498a5a8f0639e204517b5d05ff378e44a57831a
+six==1.15.0 \
+    --hash=sha256:8b74bedcbbbaca38ff6d7491d76f2b06b3592611af620f8426e82dddb04a5ced \
+    --hash=sha256:30639c035cdb23534cd4aa2dd52c3bf48f06e5f4a941509c8bafd8ce11080259
+urllib3==1.25.9 \
+    --hash=sha256:88206b0eb87e6d677d424843ac5209e3fb9d0190d0ee169599165ec25e9d9115 \
+    --hash=sha256:3018294ebefce6572a474f0604c2021e33b3fd8006ecd11d62107a5d2a963527
+xlrd==1.2.0 \
+    --hash=sha256:e551fb498759fa3a5384a94ccd4c3c02eb7c00ea424426e212ac0c57be9dfbde \
+    --hash=sha256:546eb36cee8db40c3eaa46c351e67ffee6eeb5fa2650b71bc4c758a29a1b29b2
--- a/setup.cfg
+++ b/setup.cfg
@ -0,0 +1,6 @@
+[isort]
+multi_line_output=3
+include_trailing_comma=True
+force_grid_wrap=0
+use_parentheses=True
+line_length=88
--- a/setup.py
+++ b/setup.py
@ -4,16 +4,17 @@ with open("README.md", "r") as fh:
    long_description = fh.read()

 install_requires = [
-    'pandas',
-    'python-stdnum',
-    'requests',
-    'requests-cache',
-    'pycountry'
+    "pandas",
+    "python-stdnum",
+    "requests",
+    "requests-cache",
+    "pycountry",
+    "langid",
 ]

 setuptools.setup(
    name="csv-metadata-quality",
-    version="0.2.2",
+    version="0.4.2",
    author="Alan Orth",
    author_email="aorth@mjanja.ch",
    description="A simple, but opinionated CSV quality checking and fixing pipeline for CSVs in the DSpace ecosystem.",
@ -24,15 +25,14 @@ setuptools.setup(
    classifiers=[
        "Programming Language :: Python :: 3.6",
        "Programming Language :: Python :: 3.7",
+        "Programming Language :: Python :: 3.8",
        "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
        "Operating System :: OS Independent",
-        "Development Status :: 4 - Beta"
+        "Development Status :: 4 - Beta",
    ],
-    packages=['csv_metadata_quality'],
+    packages=["csv_metadata_quality"],
    entry_points={
-        'console_scripts': [
-            'csv-metadata-quality = csv_metadata_quality.__main__:main'
-        ]
+        "console_scripts": ["csv-metadata-quality = csv_metadata_quality.__main__:main"]
    },
-    install_requires=install_requires
+    install_requires=install_requires,
 )
--- a/tests/test_check.py
+++ b/tests/test_check.py
@ -1,21 +1,24 @@
+import pandas as pd
+
 import csv_metadata_quality.check as check
+import csv_metadata_quality.experimental as experimental


 def test_check_invalid_issn(capsys):
-    '''Test checking invalid ISSN.'''
+    """Test checking invalid ISSN."""

-    value = '2321-2302'
+    value = "2321-2302"

    check.issn(value)

    captured = capsys.readouterr()
-    assert captured.out == f'Invalid ISSN: {value}\n'
+    assert captured.out == f"Invalid ISSN: {value}\n"


 def test_check_valid_issn():
-    '''Test checking valid ISSN.'''
+    """Test checking valid ISSN."""

-    value = '0024-9319'
+    value = "0024-9319"

    result = check.issn(value)

@ -23,20 +26,20 @@ def test_check_valid_issn():


 def test_check_invalid_isbn(capsys):
-    '''Test checking invalid ISBN.'''
+    """Test checking invalid ISBN."""

-    value = '99921-58-10-6'
+    value = "99921-58-10-6"

    check.isbn(value)

    captured = capsys.readouterr()
-    assert captured.out == f'Invalid ISBN: {value}\n'
+    assert captured.out == f"Invalid ISBN: {value}\n"


 def test_check_valid_isbn():
-    '''Test checking valid ISBN.'''
+    """Test checking valid ISBN."""

-    value = '99921-58-10-7'
+    value = "99921-58-10-7"

    result = check.isbn(value)

@ -44,71 +47,75 @@ def test_check_valid_isbn():


 def test_check_invalid_separators(capsys):
-    '''Test checking invalid multi-value separators.'''
+    """Test checking invalid multi-value separators."""

-    value = 'Alan|Orth'
+    value = "Alan|Orth"

-    check.separators(value)
+    field_name = "dc.contributor.author"
+
+    check.separators(value, field_name)

    captured = capsys.readouterr()
-    assert captured.out == f'Invalid multi-value separator: {value}\n'
+    assert captured.out == f"Invalid multi-value separator ({field_name}): {value}\n"


 def test_check_valid_separators():
-    '''Test checking valid multi-value separators.'''
+    """Test checking valid multi-value separators."""

-    value = 'Alan||Orth'
+    value = "Alan||Orth"

-    result = check.separators(value)
+    field_name = "dc.contributor.author"
+
+    result = check.separators(value, field_name)

    assert result == value


 def test_check_missing_date(capsys):
-    '''Test checking missing date.'''
+    """Test checking missing date."""

    value = None

-    field_name = 'dc.date.issued'
+    field_name = "dc.date.issued"

    check.date(value, field_name)

    captured = capsys.readouterr()
-    assert captured.out == f'Missing date ({field_name}).\n'
+    assert captured.out == f"Missing date ({field_name}).\n"


 def test_check_multiple_dates(capsys):
-    '''Test checking multiple dates.'''
+    """Test checking multiple dates."""

-    value = '1990||1991'
+    value = "1990||1991"

-    field_name = 'dc.date.issued'
+    field_name = "dc.date.issued"

    check.date(value, field_name)

    captured = capsys.readouterr()
-    assert captured.out == f'Multiple dates not allowed ({field_name}): {value}\n'
+    assert captured.out == f"Multiple dates not allowed ({field_name}): {value}\n"


 def test_check_invalid_date(capsys):
-    '''Test checking invalid ISO8601 date.'''
+    """Test checking invalid ISO8601 date."""

-    value = '1990-0'
+    value = "1990-0"

-    field_name = 'dc.date.issued'
+    field_name = "dc.date.issued"

    check.date(value, field_name)

    captured = capsys.readouterr()
-    assert captured.out == f'Invalid date ({field_name}): {value}\n'
+    assert captured.out == f"Invalid date ({field_name}): {value}\n"


 def test_check_valid_date():
-    '''Test checking valid ISO8601 date.'''
+    """Test checking valid ISO8601 date."""

-    value = '1990'
+    value = "1990"

-    field_name = 'dc.date.issued'
+    field_name = "dc.date.issued"

    result = check.date(value, field_name)

@ -116,22 +123,22 @@ def test_check_valid_date():


 def test_check_suspicious_characters(capsys):
-    '''Test checking for suspicious characters.'''
+    """Test checking for suspicious characters."""

-    value = 'foreˆt'
+    value = "foreˆt"

-    field_name = 'dc.contributor.author'
+    field_name = "dc.contributor.author"

    check.suspicious_characters(value, field_name)

    captured = capsys.readouterr()
-    assert captured.out == f'Suspicious character ({field_name}): ˆt\n'
+    assert captured.out == f"Suspicious character ({field_name}): ˆt\n"


-def test_check_valid_iso639_2_language():
-    '''Test valid ISO 639-2 language.'''
+def test_check_valid_iso639_1_language():
+    """Test valid ISO 639-1 (alpha 2) language."""

-    value = 'ja'
+    value = "ja"

    result = check.language(value)

@ -139,65 +146,65 @@ def test_check_valid_iso639_2_language():


 def test_check_valid_iso639_3_language():
-    '''Test invalid ISO 639-3 language.'''
+    """Test valid ISO 639-3 (alpha 3) language."""

-    value = 'eng'
+    value = "eng"

    result = check.language(value)

    assert result == value


-def test_check_invalid_iso639_2_language(capsys):
-    '''Test invalid ISO 639-2 language.'''
+def test_check_invalid_iso639_1_language(capsys):
+    """Test invalid ISO 639-1 (alpha 2) language."""

-    value = 'jp'
+    value = "jp"

    check.language(value)

    captured = capsys.readouterr()
-    assert captured.out == f'Invalid ISO 639-2 language: {value}\n'
+    assert captured.out == f"Invalid ISO 639-1 language: {value}\n"


 def test_check_invalid_iso639_3_language(capsys):
-    '''Test invalid ISO 639-3 language.'''
+    """Test invalid ISO 639-3 (alpha 3) language."""

-    value = 'chi'
+    value = "chi"

    check.language(value)

    captured = capsys.readouterr()
-    assert captured.out == f'Invalid ISO 639-3 language: {value}\n'
+    assert captured.out == f"Invalid ISO 639-3 language: {value}\n"


 def test_check_invalid_language(capsys):
-    '''Test invalid language.'''
+    """Test invalid language."""

-    value = 'Span'
+    value = "Span"

    check.language(value)

    captured = capsys.readouterr()
-    assert captured.out == f'Invalid language: {value}\n'
+    assert captured.out == f"Invalid language: {value}\n"


 def test_check_invalid_agrovoc(capsys):
-    '''Test invalid AGROVOC subject.'''
+    """Test invalid AGROVOC subject."""

-    value = 'FOREST'
-    field_name = 'dc.subject'
+    value = "FOREST"
+    field_name = "dc.subject"

    check.agrovoc(value, field_name)

    captured = capsys.readouterr()
-    assert captured.out == f'Invalid AGROVOC ({field_name}): {value}\n'
+    assert captured.out == f"Invalid AGROVOC ({field_name}): {value}\n"


 def test_check_valid_agrovoc():
-    '''Test valid AGROVOC subject.'''
+    """Test valid AGROVOC subject."""

-    value = 'FORESTS'
-    field_name = 'dc.subject'
+    value = "FORESTS"
+    field_name = "dc.subject"

    result = check.agrovoc(value, field_name)

@ -205,21 +212,89 @@ def test_check_valid_agrovoc():


 def test_check_uncommon_filename_extension(capsys):
-    '''Test uncommon filename extension.'''
+    """Test uncommon filename extension."""

-    value = 'file.pdf.lck'
+    value = "file.pdf.lck"

    check.filename_extension(value)

    captured = capsys.readouterr()
-    assert captured.out == f'Filename with uncommon extension: {value}\n'
+    assert captured.out == f"Filename with uncommon extension: {value}\n"


 def test_check_common_filename_extension():
-    '''Test common filename extension.'''
+    """Test common filename extension."""

-    value = 'file.pdf'
+    value = "file.pdf"

    result = check.filename_extension(value)

    assert result == value
+
+
+def test_check_incorrect_iso_639_1_language(capsys):
+    """Test incorrect ISO 639-1 language, as determined by comparing the item's language field with the actual language predicted in the item's title."""
+
+    title = "A randomised vaccine field trial in Kenya demonstrates protection against wildebeest-associated malignant catarrhal fever in cattle"
+    language = "es"
+
+    # Create a dictionary to mimic Pandas series
+    row = {"dc.title": title, "dc.language.iso": language}
+    series = pd.Series(row)
+
+    experimental.correct_language(series)
+
+    captured = capsys.readouterr()
+    assert (
+        captured.out
+        == f"Possibly incorrect language {language} (detected en): {title}\n"
+    )
+
+
+def test_check_incorrect_iso_639_3_language(capsys):
+    """Test incorrect ISO 639-3 language, as determined by comparing the item's language field with the actual language predicted in the item's title."""
+
+    title = "A randomised vaccine field trial in Kenya demonstrates protection against wildebeest-associated malignant catarrhal fever in cattle"
+    language = "spa"
+
+    # Create a dictionary to mimic Pandas series
+    row = {"dc.title": title, "dc.language.iso": language}
+    series = pd.Series(row)
+
+    experimental.correct_language(series)
+
+    captured = capsys.readouterr()
+    assert (
+        captured.out
+        == f"Possibly incorrect language {language} (detected eng): {title}\n"
+    )
+
+
+def test_check_correct_iso_639_1_language():
+    """Test correct ISO 639-1 language, as determined by comparing the item's language field with the actual language predicted in the item's title."""
+
+    title = "A randomised vaccine field trial in Kenya demonstrates protection against wildebeest-associated malignant catarrhal fever in cattle"
+    language = "en"
+
+    # Create a dictionary to mimic Pandas series
+    row = {"dc.title": title, "dc.language.iso": language}
+    series = pd.Series(row)
+
+    result = experimental.correct_language(series)
+
+    assert result == language
+
+
+def test_check_correct_iso_639_3_language():
+    """Test correct ISO 639-3 language, as determined by comparing the item's language field with the actual language predicted in the item's title."""
+
+    title = "A randomised vaccine field trial in Kenya demonstrates protection against wildebeest-associated malignant catarrhal fever in cattle"
+    language = "eng"
+
+    # Create a dictionary to mimic Pandas series
+    row = {"dc.title": title, "dc.language.iso": language}
+    series = pd.Series(row)
+
+    result = experimental.correct_language(series)
+
+    assert result == language
--- a/tests/test_fix.py
+++ b/tests/test_fix.py
@ -2,67 +2,99 @@ import csv_metadata_quality.fix as fix


 def test_fix_leading_whitespace():
-    '''Test fixing leading whitespace.'''
+    """Test fixing leading whitespace."""

-    value = ' Alan'
+    value = " Alan"

-    assert fix.whitespace(value) == 'Alan'
+    field_name = "dc.contributor.author"
+
+    assert fix.whitespace(value, field_name) == "Alan"


 def test_fix_trailing_whitespace():
-    '''Test fixing trailing whitespace.'''
+    """Test fixing trailing whitespace."""

-    value = 'Alan '
+    value = "Alan "

-    assert fix.whitespace(value) == 'Alan'
+    field_name = "dc.contributor.author"
+
+    assert fix.whitespace(value, field_name) == "Alan"


 def test_fix_excessive_whitespace():
-    '''Test fixing excessive whitespace.'''
+    """Test fixing excessive whitespace."""

-    value = 'Alan  Orth'
+    value = "Alan  Orth"

-    assert fix.whitespace(value) == 'Alan Orth'
+    field_name = "dc.contributor.author"
+
+    assert fix.whitespace(value, field_name) == "Alan Orth"


 def test_fix_invalid_separators():
-    '''Test fixing invalid multi-value separators.'''
+    """Test fixing invalid multi-value separators."""

-    value = 'Alan|Orth'
+    value = "Alan|Orth"

-    assert fix.separators(value) == 'Alan||Orth'
+    field_name = "dc.contributor.author"
+
+    assert fix.separators(value, field_name) == "Alan||Orth"


 def test_fix_unnecessary_unicode():
-    '''Test fixing unnecessary Unicode.'''
+    """Test fixing unnecessary Unicode."""

-    value = 'Alan Orth'
+    value = "Alan Orth"

-    assert fix.unnecessary_unicode(value) == 'Alan Orth'
+    assert fix.unnecessary_unicode(value) == "Alan Orth"


 def test_fix_duplicates():
-    '''Test fixing duplicate metadata values.'''
+    """Test fixing duplicate metadata values."""

-    value = 'Kenya||Kenya'
+    value = "Kenya||Kenya"

-    assert fix.duplicates(value) == 'Kenya'
+    field_name = "dc.contributor.author"
+
+    assert fix.duplicates(value, field_name) == "Kenya"


 def test_fix_newlines():
-    '''Test fixing newlines.'''
+    """Test fixing newlines."""

-    value = '''Ken
-ya'''
+    value = """Ken
+ya"""

-    assert fix.newlines(value) == 'Kenya'
+    assert fix.newlines(value) == "Kenya"


 def test_fix_comma_space():
-    '''Test adding space after comma.'''
+    """Test adding space after comma."""

-    value = 'Orth,Alan S.'
+    value = "Orth,Alan S."

-    field_name = 'dc.contributor.author'
+    field_name = "dc.contributor.author"

-    assert fix.comma_space(value, field_name) == 'Orth, Alan S.'
+    assert fix.comma_space(value, field_name) == "Orth, Alan S."
+
+
+def test_fix_normalized_unicode():
+    """Test fixing a string that is already in its normalized (NFC) Unicode form."""
+
+    # string using the normalized canonical form of é
+    value = "Ouédraogo, Mathieu"
+
+    field_name = "dc.contributor.author"
+
+    assert fix.normalize_unicode(value, field_name) == "Ouédraogo, Mathieu"
+
+
+def test_fix_decomposed_unicode():
+    """Test fixing a string that contains Unicode string."""
+
+    # string using the decomposed form of é
+    value = "Ouédraogo, Mathieu"
+
+    field_name = "dc.contributor.author"
+
+    assert fix.normalize_unicode(value, field_name) == "Ouédraogo, Mathieu"
Author	SHA1	Message	Date
Alan Orth	cb07d357d4	Version 0.4.2	2020-07-06 14:04:34 +03:00
Alan Orth	65cd48a26f	CHANGELOG.md: Update changes	2020-07-06 14:00:21 +03:00
Alan Orth	0f883f640c	Remove pipenv	2020-07-06 13:59:49 +03:00
Alan Orth	f4c5c5781e	README.md: Switch to poetry	2020-07-06 13:59:11 +03:00
Alan Orth	6aa784ad8c	Update requirements Generated with poetry export: $ poetry export -f requirements.txt > requirements.txt $ poetry export --dev -f requirements.txt > requirements-dev.txt	2020-07-06 13:57:07 +03:00
Alan Orth	7b8da94f41	poetry.lock: Update Python dependencies	2020-07-06 13:56:31 +03:00
Alan Orth	2a1566af62	csv_metadata_quality/check.py: Parameterize AGROVOC request	2020-07-06 13:44:46 +03:00
Alan Orth	5fcaa63bd5	csv_metadata_quality/check.py: Prune requests cache once We only need to prune the requests cache once before using it, not for every value we check.	2020-07-06 13:42:19 +03:00
Alan Orth	aa9e23b46c	pyproject.toml: Update license specifier We need to use valid SPDX license identifiers.	2020-06-09 14:22:53 +03:00
Alan Orth	73acb1661f	Update requirements Generated with poetry export: $ poetry export -f requirements.txt > requirements.txt $ poetry export --dev -f requirements.txt > requirements-dev.txt	2020-05-31 17:51:16 +03:00
Alan Orth	2a068fddc4	.build.yml: Fix test	2020-05-31 17:44:37 +03:00
Alan Orth	c6c2f13e88	.build.yml: Fix poetry install invocation Poetry apparently installs dev dependencies by default.	2020-05-31 17:37:09 +03:00
Alan Orth	56f16e37ed	.build.yml: Use poetry in SourceHut CI	2020-05-31 17:35:04 +03:00
Alan Orth	0c44b967b6	Add poetry project file and lock I want to try to use poetry instead of pipenv because pipenv takes forever to do dependency resolution sometimes. Also, I have had a few issues with Python modules like black that don't have releases other than pre-releases, and even including the project itself in the dependencies (pip install -e . ...?). My initial experience is that poetry handles this better.	2020-05-31 17:33:40 +03:00
Alan Orth	8a267bb40b	.travis.yml: Try to build with Python 3.8-dev But allow failures.	2020-03-29 16:40:11 +03:00
Alan Orth	8fda8f1ef1	Pipfile.lock: Run pipenv update All tests still passing.	2020-03-20 16:22:04 +02:00
Alan Orth	5e471813e8	CHANGELOG.md: Add note about python dependencies	2020-01-29 12:41:43 +02:00
Alan Orth	79244b9ac3	Pipfile.lock: Run pipenv update	2020-01-29 12:39:12 +02:00
Alan Orth	5e81a33482	CHANGELOG.md: Add note about field names	2020-01-16 12:37:11 +02:00
Alan Orth	28b5996aa6	Output field name for more fixes and checks This helps identify which field has the error.	2020-01-16 12:35:11 +02:00
Alan Orth	40ba9bae6c	README.md: Adjust heading size	2020-01-15 12:26:11 +02:00
Alan Orth	0b2d211455	Version 0.4.1	2020-01-15 12:19:42 +02:00
Alan Orth	7f1df0b47c	Support Python 3.6 and 3.7 again	2020-01-15 12:19:17 +02:00
Alan Orth	365ecda324	Add utility function to check normalization Python's built-in unicodedata library includes the is_normalized() function starting with Python 3.8. This utility function allows us to do the same thing with earlier Python versions. See: https://docs.python.org/3/library/unicodedata.html	2020-01-15 12:17:52 +02:00
Alan Orth	550ce7fb7e	.travis.yml: Only test Python 3.8 The Unicode normalization feature requires Python 3.8 because the unicodedata.is_normalized() function only appears there. If I find another way to check if a string is normalized without normalizing it first I will drop the requirements back down to Python 3.6. See: https://docs.python.org/3/library/unicodedata.html	2020-01-15 11:57:21 +02:00
Alan Orth	705127fd28	Version 0.4.0	2020-01-15 11:44:56 +02:00
Alan Orth	894e0a196d	setup.py: Change Python requirements The `unicodedata.is_normalized()` function requires Python 3.8. See: https://docs.python.org/3/library/unicodedata.html	2020-01-15 11:43:25 +02:00
Alan Orth	87181bc7b8	Run black, isort, and flake8.	2020-01-15 11:41:31 +02:00
Alan Orth	8de5d862b6	CHANGELOG.md: Add note about Unicode normalization	2020-01-15 11:40:40 +02:00
Alan Orth	49e3543878	Add Unicode normalization This will check all strings for un-normalized Unicode characters. Normalization is done using NFC. This includes tests and updated sample data (data/test.csv). See: https://withblue.ink/2019/03/11/why-you-need-to-normalize-unicode-strings.html	2020-01-15 11:37:54 +02:00
Alan Orth	403b253762	CHANGELOG.md: Update python library versions	2020-01-15 10:58:44 +02:00
Alan Orth	c5fbaf407a	Update python requirements Generated using pipenv: $ pipenv lock -r > requirements.txt $ pipenv lock -r -d > requirements-dev.txt	2020-01-15 10:51:58 +02:00
Alan Orth	4f81f6c83c	Pipfile.lock: Run pipenv update	2020-01-15 10:51:19 +02:00
Alan Orth	4b9d1e060f	setup.py: Add Python 3.8 classifier	2019-12-14 12:56:11 +02:00
Alan Orth	c8a71e3143	Pipfile.lock: Run pipenv update	2019-12-14 12:53:39 +02:00
Alan Orth	7964d98ca5	Pipfile: Specify exact version of black Black only releases pre-release versions, which causes issues with pipenv. Instead of always running pipenv with "--pre" and potenti- ally letting in some other pre-release versions for other depende- ncies, I would rather specify the latest black version explicitly. See: https://github.com/psf/black/issues/517 See: https://github.com/microsoft/vscode-python/issues/5171	2019-12-14 12:41:28 +02:00
Alan Orth	64ffc2f1da	.travis.yml: Install packages from requirements.txt too	2019-11-14 23:42:28 +02:00
Alan Orth	7b1bc29a92	.travis.yml: Try using pip instead of pipenv The Pipfile knows it was created with Python 3.8, yet we're running with multiple Python versions on Travis. I'm curious if would work better to use pip to install dependencies instead of pipenv in this case.	2019-11-14 23:37:25 +02:00
Alan Orth	f0110d8e74	CHANGELOG.md: Add note about requirements	2019-11-14 23:30:26 +02:00
Alan Orth	86498deee8	Update python requirements Generated using pipenv: $ pipenv lock -r > requirements.txt $ pipenv lock -r -d > requirements-dev.txt	2019-11-14 23:28:42 +02:00
Alan Orth	251647a15f	CHANGELOG.md: Add TravisCI changes	2019-11-14 23:24:08 +02:00
Alan Orth	0bd28e22ec	.travis.yml: Test Python 3.8	2019-11-14 23:22:37 +02:00
Alan Orth	63fdce7d13	.travis.yml: Use Ubuntu 18.04 "Bionic"	2019-11-14 23:22:19 +02:00
Alan Orth	f068c0e16a	CHANGELOG.md: Use Python 3.8.0 for pipenv	2019-11-14 23:11:43 +02:00
Alan Orth	79b8f62a85	Use Python 3.8 for pipenv Python 3.8.0 entered Arch Linux core repositories now and all tests pass with Python 3.8.0 so it's time...	2019-11-14 23:10:20 +02:00
Alan Orth	6c1e132531	CHANGELOG.md: Add unreleased changes	2019-11-14 09:19:19 +02:00
Alan Orth	c0f3c866bd	Pipfile.lock: Run pipenv update Updates the following dependencies: - numpy 1.17.2→1.17.4 - pandas 0.25.1→0.25.3 - flake8 3.7.8→3.7.9 - pytest 5.1.3→5.2.2 - black 19.3b0→19.10b0	2019-11-14 09:17:31 +02:00
Alan Orth	36d0474b95	CHANGELOG.md: Move unreleased changes to v0.3.1	2019-10-01 17:11:52 +03:00
Alan Orth	efdc3a841a	Version 0.3.1	2019-10-01 17:11:13 +03:00
Alan Orth	fd2ba6845d	CHANGELOG.md: Update unreleased notes	2019-10-01 17:10:23 +03:00
Alan Orth	e55380b4d5	csv_metadata_quality/fix.py: Harmonize language in fix output We should always say if we're removing or replacing something.	2019-10-01 17:09:49 +03:00
Alan Orth	85ae16d9b7	CHANGELOG.md: Add note about non-breaking spaces	2019-10-01 16:56:37 +03:00
Alan Orth	c42f8b4812	csv_metadata_quality/fix.py: Replace non-breaking spaces We should be replacing non-breaking spaces (U+00A0) with normal sp- aces instead of removing them.	2019-10-01 16:55:04 +03:00
Alan Orth	1c75608d54	README.md: Update introduction text We should mention that this is not DSpace specific. Rather, it is much more realistically Dublin Core specific.	2019-09-26 14:19:13 +03:00
Alan Orth	0b15a8ed3b	README.md: Remove TODO about lack of space after comma This was added as an automatic global fix a few weeks ago.	2019-09-26 14:16:33 +03:00
Alan Orth	9ca266f5f0	data/test.csv: Change birthdate column to dc.date.issued More accurately reflects actual data we will be validating.	2019-09-26 14:15:48 +03:00
Alan Orth	0d3f948708	CHANGELOG.md: Update comment about language validation	2019-09-26 14:14:57 +03:00
Alan Orth	c04207fcfc	CHANGELOG.md: Fix header formatting	2019-09-26 14:13:50 +03:00
Alan Orth	9d4eceddc7	.build.yml: Enable experimental CLI checks on SourceHut	2019-09-26 14:11:35 +03:00
Alan Orth	e15c98cccb	Move unreleased changes to v0.3.0	2019-09-26 14:06:31 +03:00
Alan Orth	93c4e1a993	Update python requirements Generated using pipenv: $ pipenv lock -r > requirements.txt $ pipenv lock -r -d > requirements-dev.txt	2019-09-26 14:05:37 +03:00
Alan Orth	9963b2bb64	Pipfile.lock: Run pipenv update	2019-09-26 14:04:50 +03:00
Alan Orth	76291c1876	CHANGELOG.md: Add note about language validation	2019-09-26 14:03:18 +03:00
Alan Orth	604bd5bda6	Reformat tests with black	2019-09-26 14:02:51 +03:00
Alan Orth	e7c220039b	README.md: Add note about experimental language validation	2019-09-26 13:59:50 +03:00
Alan Orth	d7b5e378bc	setup.py: Add langid	2019-09-26 13:49:32 +03:00
Alan Orth	8435ee242d	Experimental language detection using langid Works decenty well assuming the title, abstract, and citation fields are an accurate representation of the language as identified by the language field. Handles ISO 639-1 (alpha 2) and ISO 639-3 (alpha 3) values seamlessly. This includes updated pipenv environment, test data, pytest tests for both correct and incorrect ISO 639-1 and ISO 639-3 languages, and a new command line option "-e".	2019-09-26 13:46:32 +03:00
Alan Orth	7ac1c6f554	README.md: Update comment about ISO 639-3 The pycountry library is actually using ISO 639-3 apparently. See: https://pypi.org/project/pycountry/	2019-09-26 07:51:41 +03:00
Alan Orth	86d4623fd3	More ISO 639-1 and ISO 639-3 fixes ISO 639-1 uses two-letter codes and ISO 639-3 uses three-letter codes. Technically there ISO 639-2/T and ISO 639-2/B, which also uses three letter codes, but those are not supported by the pycountry library so I won't even worry about them. See: https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes	2019-09-26 07:44:39 +03:00
Alan Orth	ddbe970342	data/test.csv: Update titles of language tests ISO 639-1 is alpha 2 and ISO 639-3 is alpha 3. See: https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes	2019-09-26 07:40:27 +03:00
Alan Orth	31c78ca6f3	data/test.csv: Rename contributor column to title This makes more sense as a description of each test and the titles are obviously not authors.	2019-09-26 05:50:40 +03:00
Alan Orth	154d05b5e2	CHANGELOG.md: Update notes	2019-09-24 18:55:05 +03:00
Alan Orth	186f146edb	Pipfile.lock: Run pipenv update Synchronizes state with the Pipfile and brings some new deps.	2019-09-24 18:54:49 +03:00
Alan Orth	a4cb301943	CHANGELOG.md: Add note about csvkit	2019-09-24 18:49:20 +03:00
Alan Orth	219e37526d	Pipfile: Add csvkit to dev requirements Used to inspect CSV files during testing and development.	2019-09-24 18:48:01 +03:00
Alan Orth	f304ca6a33	csv_metadata_quality/app.py: Use simpler column iteration I don't know where I got the other one...	2019-09-21 17:19:39 +03:00
Alan Orth	3d5c8bdf5d	CHANGELOG.md: Add notes about updated python packages	2019-09-11 16:45:39 +03:00
Alan Orth	480956d54d	Pipfile.lock: Run pipenv update	2019-09-11 16:45:16 +03:00
Alan Orth	d9fc09f121	Fix references to ISO 639 It turns out that ISO 639-1 is the two-letter codes, and ISO 639-2 is the three-letter codes, aka alpha2 and alpha3. See: https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes	2019-09-11 16:36:53 +03:00
Alan Orth	b5899001b7	CHANGELOG.md: Add note about black and isort	2019-08-29 01:26:11 +03:00
Alan Orth	c92977d1ca	Update requirements-dev.txt Generated with: $ pipenv lock -r -d > requirements-dev.txt	2019-08-29 01:25:14 +03:00
Alan Orth	280a99c8a8	Sort imports with isort See: https://sourcery.ai/blog/python-best-practices/	2019-08-29 01:15:04 +03:00
Alan Orth	0388145b81	Add configuration for isort See: https://sourcery.ai/blog/python-best-practices/	2019-08-29 01:14:31 +03:00
Alan Orth	d97dcd19db	Format with black	2019-08-29 01:10:39 +03:00
Alan Orth	b375f0e895	Add black and isort to pipenv dev dependencies These do a very opinionated automatic formatting and validation of code. See: https://sourcery.ai/blog/python-best-practices/	2019-08-29 01:08:38 +03:00
Alan Orth	865c61d316	Add note about updated python dependencies	2019-08-28 21:02:21 +03:00
Alan Orth	3b2ba57b75	Update python requirements Generated using pipenv: $ pipenv lock -r > requirements.txt $ pipenv lock -r -d > requirements-dev.txt	2019-08-28 21:01:48 +03:00
Alan Orth	2805c556a9	Pipfile.lock: Run pipenv update Brings numpy 1.17.1, pandas 0.25.1, requests-cache 0.5.2, and pandas 0.25.1.	2019-08-28 20:58:35 +03:00