Version 0.4.0

setup.py: Change Python requirements
The `unicodedata.is_normalized()` function requires Python 3.8. See: https://docs.python.org/3/library/unicodedata.html
2025-05-09 22:56:01 +02:00 · 2020-01-15 11:44:56 +02:00 · 2020-01-15 11:43:25 +02:00 · 2020-01-15 11:41:31 +02:00 · 2020-01-15 11:40:40 +02:00 · 2020-01-15 11:37:54 +02:00
15 changed files with 335 additions and 222 deletions
--- a/.build.yml
+++ b/.build.yml
@ -13,7 +13,7 @@ tasks:
  - testcli: |
      cd csv-metadata-quality
      pipenv run pip install .
-      pipenv run csv-metadata-quality -i data/test.csv -o /tmp/test.csv -u --agrovoc-fields dc.subject,cg.coverage.country
+      pipenv run csv-metadata-quality -i data/test.csv -o /tmp/test.csv -e -u --agrovoc-fields dc.subject,cg.coverage.country
 environment:
  PIPENV_NOSPIN: 'True'
  PIPENV_HIDE_EMOJIS: 'True'
--- a/.travis.yml
+++ b/.travis.yml
@ -1,11 +1,12 @@
-dist: xenial
+dist: bionic
 language: python
 python:
  - "3.6"
  - "3.7"
+  - "3.8"
 install:
-  - "pip install pipenv --upgrade-strategy=only-if-needed"
-  - "pipenv install --dev"
+  - "pip install -r requirements.txt"
+  - "pip install -r requirements-dev.txt"
 script: pytest

 # vim: ts=2 sw=2 et
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@ -4,14 +4,33 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

+## [0.4.0] - 2020-01-15
+### Added
+- Unicode normalization (enable with `--unsafe-fixes`, see README.md)
+
+### Updated
+- Update python dependencies to latest versions, including numpy 1.18.1, pandas
+1.0.0rc0, flake8 3.7.9, pytest 5.3.2, and black 19.10b0
+- Regenerate requirements.txt and requirements-dev.txt
+
+### Changed
+- Use Python 3.8.0 for pipenv
+- Use Ubuntu 18.04 "Bionic" for TravisCI builds
+- Test Python 3.8 in TravisCI builds
+
+## [0.3.1] - 2019-10-01
+## Changed
+- Replace non-breaking spaces (U+00A0) with space instead of removing them
+- Harmonize language of script output when fixing various issues
+
 ## [0.3.0] - 2019-09-26
 ### Updated
 - Update python dependencies to latest versions, including numpy 1.17.2, pandas
 0.25.1, pytest 5.1.3, and requests-cache 0.5.2

-## Added
+### Added
 - csvkit to dev requirements (csvcut etc are useful during development)
- Experimental language validation using `-e` (see README.md)
+- Experimental language validation using the Python `langid` library (enable with `-e`, see README.md)

 ### Changed
 - Re-formatted code with black and isort
--- a/4
+++ b/4
@ -8,7 +8,7 @@ pytest = "*"
 ipython = "*"
 flake8 = "*"
 pytest-clarity = "*"
-black = "*"
+black = "==19.10b0"
 isort = "*"
 csvkit = "*"

@ -23,7 +23,7 @@ csv-metadata-quality = {editable = true,path = "."}
 langid = "*"

 [requires]
-python_version = "3.7"
+python_version = "3.8"

 [pipenv]
 allow_prereleases = true
--- a/Pipfile.lock
+++ b/Pipfile.lock
@ -1,11 +1,11 @@
 {
    "_meta": {
        "hash": {
-            "sha256": "59562d8c59eb09e23b49475d6901687edbf605f5b84e283e90cc8e2de518641f"
+            "sha256": "bc933a2deb26ed095c46d6ccddf0f305f84157bdd95548c6b6a4356537951890"
        },
        "pipfile-spec": 6,
        "requires": {
-            "python_version": "3.7"
+            "python_version": "3.8"
        },
        "sources": [
            {
@ -18,10 +18,10 @@
    "default": {
        "certifi": {
            "hashes": [
-                "sha256:e4f3620cfea4f83eedc95b24abd9cd56f3c4b146dd0177e83a21b4eb49e21e50",
-                "sha256:fd7c7c74727ddcf00e9acd26bba8da604ffec95bf1c2144e67aff7a8b50e6cef"
+                "sha256:017c25db2a153ce562900032d5bc68e9f191e44e9a0f762f373977de9df1fbb3",
+                "sha256:25b64c7da4cd7479594d035c08c2d809eb4aab3a26e5a990ea98cc450c320f1f"
            ],
-            "version": "==2019.9.11"
+            "version": "==2019.11.28"
        },
        "chardet": {
            "hashes": [
@ -50,45 +50,49 @@
        },
        "numpy": {
            "hashes": [
-                "sha256:05dbfe72684cc14b92568de1bc1f41e5f62b00f714afc9adee42f6311738091f",
-                "sha256:0d82cb7271a577529d07bbb05cb58675f2deb09772175fab96dc8de025d8ac05",
-                "sha256:10132aa1fef99adc85a905d82e8497a580f83739837d7cbd234649f2e9b9dc58",
-                "sha256:12322df2e21f033a60c80319c25011194cd2a21294cc66fee0908aeae2c27832",
-                "sha256:16f19b3aa775dddc9814e02a46b8e6ae6a54ed8cf143962b4e53f0471dbd7b16",
-                "sha256:3d0b0989dd2d066db006158de7220802899a1e5c8cf622abe2d0bd158fd01c2c",
-                "sha256:438a3f0e7b681642898fd7993d38e2bf140a2d1eafaf3e89bb626db7f50db355",
-                "sha256:5fd214f482ab53f2cea57414c5fb3e58895b17df6e6f5bca5be6a0bb6aea23bb",
-                "sha256:73615d3edc84dd7c4aeb212fa3748fb83217e00d201875a47327f55363cef2df",
-                "sha256:7bd355ad7496f4ce1d235e9814ec81ee3d28308d591c067ce92e49f745ba2c2f",
-                "sha256:7d077f2976b8f3de08a0dcf5d72083f4af5411e8fddacd662aae27baa2601196",
-                "sha256:a4092682778dc48093e8bda8d26ee8360153e2047826f95a3f5eae09f0ae3abf",
-                "sha256:b458de8624c9f6034af492372eb2fee41a8e605f03f4732f43fc099e227858b2",
-                "sha256:e70fc8ff03a961f13363c2c95ef8285e0cf6a720f8271836f852cc0fa64e97c8",
-                "sha256:ee8e9d7cad5fe6dde50ede0d2e978d81eafeaa6233fb0b8719f60214cf226578",
-                "sha256:f4a4f6aba148858a5a5d546a99280f71f5ee6ec8182a7d195af1a914195b21a2"
+                "sha256:1786a08236f2c92ae0e70423c45e1e62788ed33028f94ca99c4df03f5be6b3c6",
+                "sha256:17aa7a81fe7599a10f2b7d95856dc5cf84a4eefa45bc96123cbbc3ebc568994e",
+                "sha256:20b26aaa5b3da029942cdcce719b363dbe58696ad182aff0e5dcb1687ec946dc",
+                "sha256:2d75908ab3ced4223ccba595b48e538afa5ecc37405923d1fea6906d7c3a50bc",
+                "sha256:39d2c685af15d3ce682c99ce5925cc66efc824652e10990d2462dfe9b8918c6a",
+                "sha256:56bc8ded6fcd9adea90f65377438f9fea8c05fcf7c5ba766bef258d0da1554aa",
+                "sha256:590355aeade1a2eaba17617c19edccb7db8d78760175256e3cf94590a1a964f3",
+                "sha256:70a840a26f4e61defa7bdf811d7498a284ced303dfbc35acb7be12a39b2aa121",
+                "sha256:77c3bfe65d8560487052ad55c6998a04b654c2fbc36d546aef2b2e511e760971",
+                "sha256:9537eecf179f566fd1c160a2e912ca0b8e02d773af0a7a1120ad4f7507cd0d26",
+                "sha256:9acdf933c1fd263c513a2df3dceecea6f3ff4419d80bf238510976bf9bcb26cd",
+                "sha256:ae0975f42ab1f28364dcda3dde3cf6c1ddab3e1d4b2909da0cb0191fa9ca0480",
+                "sha256:b3af02ecc999c8003e538e60c89a2b37646b39b688d4e44d7373e11c2debabec",
+                "sha256:b6ff59cee96b454516e47e7721098e6ceebef435e3e21ac2d6c3b8b02628eb77",
+                "sha256:b765ed3930b92812aa698a455847141869ef755a87e099fddd4ccf9d81fffb57",
+                "sha256:c98c5ffd7d41611407a1103ae11c8b634ad6a43606eca3e2a5a269e5d6e8eb07",
+                "sha256:cf7eb6b1025d3e169989416b1adcd676624c2dbed9e3bcb7137f51bfc8cc2572",
+                "sha256:d92350c22b150c1cae7ebb0ee8b5670cc84848f6359cf6b5d8f86617098a9b73",
+                "sha256:e422c3152921cece8b6a2fb6b0b4d73b6579bd20ae075e7d15143e711f3ca2ca",
+                "sha256:e840f552a509e3380b0f0ec977e8124d0dc34dc0e68289ca28f4d7c1d0d79474",
+                "sha256:f3d0a94ad151870978fb93538e95411c83899c9dc63e6fb65542f769568ecfa5"
            ],
-            "version": "==1.17.2"
+            "version": "==1.18.1"
        },
        "pandas": {
            "hashes": [
-                "sha256:18d91a9199d1dfaa01ad645f7540370ba630bdcef09daaf9edf45b4b1bca0232",
-                "sha256:3f26e5da310a0c0b83ea50da1fd397de2640b02b424aa69be7e0784228f656c9",
-                "sha256:4182e32f4456d2c64619e97c58571fa5ca0993d1e8c2d9ca44916185e1726e15",
-                "sha256:426e590e2eb0e60f765271d668a30cf38b582eaae5ec9b31229c8c3c10c5bc21",
-                "sha256:5eb934a8f0dc358f0e0cdf314072286bbac74e4c124b64371395e94644d5d919",
-                "sha256:717928808043d3ea55b9bcde636d4a52d2236c246f6df464163a66ff59980ad8",
-                "sha256:8145f97c5ed71827a6ec98ceaef35afed1377e2d19c4078f324d209ff253ecb5",
-                "sha256:8744c84c914dcc59cbbb2943b32b7664df1039d99e834e1034a3372acb89ea4d",
-                "sha256:c1ac1d9590d0c9314ebf01591bd40d4c03d710bfc84a3889e5263c97d7891dee",
-                "sha256:cb2e197b7b0687becb026b84d3c242482f20cbb29a9981e43604eb67576da9f6",
-                "sha256:d4001b71ad2c9b84ff18b182cea22b7b6cbf624216da3ea06fb7af28d1f93165",
-                "sha256:d8930772adccb2882989ab1493fa74bd87d47c8ac7417f5dd3dd834ba8c24dc9",
-                "sha256:dfbb0173ee2399bc4ed3caf2d236e5c0092f948aafd0a15fbe4a0e77ee61a958",
-                "sha256:eebfbba048f4fa8ac711b22c78516e16ff8117d05a580e7eeef6b0c2be554c18",
-                "sha256:f1b21bc5cf3dbea53d33615d1ead892dfdae9d7052fa8898083bec88be20dcd2"
+                "sha256:0f52d8a2358de840eca388f50bcab137d9d2f161f55c9c32e888387ac2e4505b",
+                "sha256:111d77cac6c0e2d8bb76bdad75b3a416729f5f31f705276becbf8035b26ac5e0",
+                "sha256:223f97e52a4d82cf918da5dcbdc92c69ab00686e2b6adeb3012326ace3dc1aee",
+                "sha256:3b09cae3d39e71187fcc6817c3f60a8c9bad5f503e6aa8d72e4cbb2e1cd7a585",
+                "sha256:4a37ab58d7c3017d71650a7d9b44d056005c1d0d9be931d8af9c8b2ca2c8a8b8",
+                "sha256:57628cd142f09165bca3ce0b2f82f14568ae14a6c2c125a29d167c9b9df6f76e",
+                "sha256:5c42b463d25780d5d5addc79b1cfb1b8d8db44d4184186da8e2a25f2c794ad43",
+                "sha256:656443bf914f5e9307fcc694d5f400d19e616d7aafa4faf57711e0449093272f",
+                "sha256:7e5dc9137b9fc2e3ccd00df092fa3af6e01430dcba747f5f063b33ea1ed0999c",
+                "sha256:8305fb7b2817e3da6071f0032b6ca1402cbe303094ab5594f552d7052782b8de",
+                "sha256:a98b46eec0e245fd3dc0d11012109f41aa37c96066aa642d65f4a4c332d193c1",
+                "sha256:b254f0c4308ff0c8c896a9de980642a55b716dff4d1fc8a730657e6d4711e35d",
+                "sha256:cce070caeb357ef89267482c7dd1a9adaa57444be5663ea294675ab0cdb5f033",
+                "sha256:f4e74a38cc48453bceda51c0d13122c38f0a49dd4c737f8091b8cdc88f47eb8c"
            ],
            "index": "pypi",
-            "version": "==0.25.1"
+            "version": "==1.0.0rc0"
        },
        "pycountry": {
            "hashes": [
@ -99,25 +103,25 @@
        },
        "python-dateutil": {
            "hashes": [
-                "sha256:7e6584c74aeed623791615e26efd690f29817a27c73085b78e4bad02493df2fb",
-                "sha256:c89805f6f4d64db21ed966fda138f8a5ed7a4fdbc1a8ee329ce1b74e3c74da9e"
+                "sha256:73ebfe9dbf22e832286dafa60473e4cd239f8592f699aa5adaf10050e6e1823c",
+                "sha256:75bb3f31ea686f1197762692a9ee6a7550b59fc6ca3a1f4b5d7e32fb98e2da2a"
            ],
-            "version": "==2.8.0"
+            "version": "==2.8.1"
        },
        "python-stdnum": {
            "hashes": [
-                "sha256:d5f0af1bee9ddd9a20b398b46ce062dbd4d41fcc9646940f2667256a44df3854",
-                "sha256:f445ec32bf5246c90389204cabba465f494545371c29a83fa2d30e6c872a6763"
+                "sha256:4c1347c414d7bdffb454924998f62c04d907a5c01faff0e35df659b0b52acba5",
+                "sha256:bb58877dafc2e590dbfddc63fa04876ab2005c3f35c8356a2dd01f62a9bdc4d6"
            ],
            "index": "pypi",
-            "version": "==1.11"
+            "version": "==1.12"
        },
        "pytz": {
            "hashes": [
-                "sha256:26c0b32e437e54a18161324a2fca3c4b9846b74a8dccddd843113109e1116b32",
-                "sha256:c894d57500a4cd2d5c71114aaab77dbab5eabd9022308ce5ac9bb93a60a6f0c7"
+                "sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d",
+                "sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be"
            ],
-            "version": "==2019.2"
+            "version": "==2019.3"
        },
        "requests": {
            "hashes": [
@ -137,17 +141,17 @@
        },
        "six": {
            "hashes": [
-                "sha256:3350809f0555b11f552448330d0b52d5f24c91a322ea4a15ef22629740f3761c",
-                "sha256:d16a0141ec1a18405cd4ce8b4613101da75da0e9a7aec5bdd4fa804d0e0eba73"
+                "sha256:1f1b7d42e254082a9db6279deae68afb421ceba6158efa6131de7b3003ee93fd",
+                "sha256:30f610279e8b2578cab6db20741130331735c781b56053c59c4076da27f06b66"
            ],
-            "version": "==1.12.0"
+            "version": "==1.13.0"
        },
        "urllib3": {
            "hashes": [
-                "sha256:3de946ffbed6e6746608990594d08faac602528ac7015ac28d33cee6a45b7398",
-                "sha256:9a107b99a5393caf59c7aa3c1249c16e6879447533d0887f4336dde834c7be86"
+                "sha256:a8a318824cc77d1fd4b2bec2ded92646630d7fe8619497b142c84a9e6f5a7293",
+                "sha256:f3c5fd51747d450d4dcf6f923c81f78f811aab8205fda64b0aba34a4e48b0745"
            ],
-            "version": "==1.25.6"
+            "version": "==1.25.7"
        },
        "xlrd": {
            "hashes": [
@ -191,26 +195,19 @@
            ],
            "version": "==1.4.3"
        },
-        "atomicwrites": {
-            "hashes": [
-                "sha256:03472c30eb2c5d1ba9227e4c2ca66ab8287fbfbbda3888aa93dc2e28fc6811b4",
-                "sha256:75a9445bac02d8d058d5e1fe689654ba5a6556a1dfd8ce6ec55a0ed79866cfa6"
-            ],
-            "version": "==1.3.0"
-        },
        "attrs": {
            "hashes": [
-                "sha256:69c0dbf2ed392de1cb5ec704444b08a5ef81680a61cb899dc08127123af36a79",
-                "sha256:f0b870f674851ecbfbbbd364d6b5cbdff9dcedbc7f3f5e18a6891057f21fe399"
+                "sha256:08a96c641c3a74e44eb59afb61a24f2cb9f4d7188748e76ba4bb5edfa3cb7d1c",
+                "sha256:f7b7ce16570fe9965acd6d30101a28f62fb4a7f9e926b3bbc9b61f8b04247e72"
            ],
-            "version": "==19.1.0"
+            "version": "==19.3.0"
        },
        "babel": {
            "hashes": [
-                "sha256:af92e6106cb7c55286b25b38ad7695f8b4efb36a90ba483d7f7a6628c46158ab",
-                "sha256:e86135ae101e31e2c8ec20a4e0c5220f4eed12487d5cf3f78be7e98d3a57fc28"
+                "sha256:1aac2ae2d0d8ea368fa90906567f5c08463d98ade155c0c4bfedd6a0f7160e38",
+                "sha256:d670ea0b10f8b723672d3a6abeb87b565b244da220d76b4dba1b66269ec152d4"
            ],
-            "version": "==2.7.0"
+            "version": "==2.8.0"
        },
        "backcall": {
            "hashes": [
@ -221,11 +218,11 @@
        },
        "black": {
            "hashes": [
-                "sha256:09a9dcb7c46ed496a9850b76e4e825d6049ecd38b611f1224857a79bd985a8cf",
-                "sha256:68950ffd4d9169716bcb8719a56c07a2f4485354fec061cdd5910aa07369731c"
+                "sha256:1b30e59be925fafc1ee4565e5e08abef6b03fe455102883820fe5ee2e4734e0b",
+                "sha256:c2edb73a08e9e0e6f65a0e6af18b059b8b1cdd5bef997d7a0b181df93dc81539"
            ],
            "index": "pypi",
-            "version": "==19.3b0"
+            "version": "==19.10b0"
        },
        "click": {
            "hashes": [
@ -250,10 +247,10 @@
        },
        "decorator": {
            "hashes": [
-                "sha256:86156361c50488b84a3f148056ea716ca587df2f0de1d34750d35c21312725de",
-                "sha256:f069f3a01830ca754ba5258fde2278454a0b5b79e0d7f5c13b3b97e57d4acff6"
+                "sha256:54c38050039232e1db4ad7375cfce6748d7b41c29e95a081c8a6d2c30364a2ce",
+                "sha256:5d19b92a3c8f7f101c8dd86afd86b0f061a8ce4540ab8cd401fa2542756bce6d"
            ],
-            "version": "==4.4.0"
+            "version": "==4.4.1"
        },
        "entrypoints": {
            "hashes": [
@ -270,33 +267,19 @@
        },
        "flake8": {
            "hashes": [
-                "sha256:19241c1cbc971b9962473e4438a2ca19749a7dd002dd1a946eaba171b4114548",
-                "sha256:8e9dfa3cecb2400b3738a42c54c3043e821682b9c840b0448c0503f781130696"
+                "sha256:45681a117ecc81e870cbf1262835ae4af5e7a8b08e40b944a8a6e6b895914cfb",
+                "sha256:49356e766643ad15072a789a20915d3c91dc89fd313ccd71802303fd67e4deca"
            ],
            "index": "pypi",
-            "version": "==3.7.8"
-        },
-        "future": {
-            "hashes": [
-                "sha256:67045236dcfd6816dc439556d009594abf643e5eb48992e36beac09c2ca659b8"
-            ],
-            "version": "==0.17.1"
-        },
-        "importlib-metadata": {
-            "hashes": [
-                "sha256:aa18d7378b00b40847790e7c27e11673d7fed219354109d0e7b9e5b25dc3ad26",
-                "sha256:d5f18a79777f3aa179c145737780282e27b508fc8fd688cb17c7a813e8bd39af"
-            ],
-            "markers": "python_version < '3.8'",
-            "version": "==0.23"
+            "version": "==3.7.9"
        },
        "ipython": {
            "hashes": [
-                "sha256:c4ab005921641e40a68e405e286e7a1fcc464497e14d81b6914b4fd95e5dee9b",
-                "sha256:dd76831f065f17bddd7eaa5c781f5ea32de5ef217592cf019e34043b56895aa1"
+                "sha256:0f4bcf18293fb666df8511feec0403bdb7e061a5842ea6e88a3177b0ceb34ead",
+                "sha256:387686dd7fc9caf29d2fddcf3116c4b07a11d9025701d220c589a430b0171d8a"
            ],
            "index": "pypi",
-            "version": "==7.8.0"
+            "version": "==7.11.1"
        },
        "ipython-genutils": {
            "hashes": [
@ -329,10 +312,10 @@
        },
        "jedi": {
            "hashes": [
-                "sha256:786b6c3d80e2f06fd77162a07fed81b8baa22dde5d62896a790a331d6ac21a27",
-                "sha256:ba859c74fa3c966a22f2aeebe1b74ee27e2a462f56d3f5f7ca4a59af61bfe42e"
+                "sha256:1349c1e8c107095a55386628bb3b2a79422f3a2cab8381e34ce19909e0cf5064",
+                "sha256:e909527104a903606dd63bea6e8e888833f0ef087057829b89a18364a856f807"
            ],
-            "version": "==0.15.1"
+            "version": "==0.15.2"
        },
        "leather": {
            "hashes": [
@ -350,37 +333,44 @@
        },
        "more-itertools": {
            "hashes": [
-                "sha256:409cd48d4db7052af495b09dec721011634af3753ae1ef92d2b32f73a745f832",
-                "sha256:92b8c4b06dac4f0611c0729b2f2ede52b2e1bac1ab48f089c7ddc12e26bb60c4"
+                "sha256:1a2a32c72400d365000412fe08eb4a24ebee89997c18d3d147544f70f5403b39",
+                "sha256:c468adec578380b6281a114cb8a5db34eb1116277da92d7c46f904f0b52d3288"
            ],
-            "version": "==7.2.0"
+            "version": "==8.1.0"
        },
        "openpyxl": {
            "hashes": [
-                "sha256:340a1ab2069764559b9d58027a43a24db18db0e25deb80f81ecb8ca7ee5253db"
+                "sha256:547a9fc6aafcf44abe358b89ed4438d077e9d92e4f182c87e2dc294186dc4b64"
            ],
-            "version": "==3.0.0"
+            "version": "==3.0.3"
        },
        "packaging": {
            "hashes": [
-                "sha256:28b924174df7a2fa32c1953825ff29c61e2f5e082343165438812f00d3a7fc47",
-                "sha256:d9551545c6d761f3def1677baf08ab2a3ca17c56879e70fecba2fc4dde4ed108"
+                "sha256:aec3fdbb8bc9e4bb65f0634b9f551ced63983a529d6a8931817d52fdd0816ddb",
+                "sha256:fe1d8331dfa7cc0a883b49d75fc76380b2ab2734b220fbb87d774e4fd4b851f8"
            ],
-            "version": "==19.2"
+            "version": "==20.0"
        },
        "parsedatetime": {
            "hashes": [
-                "sha256:3d817c58fb9570d1eec1dd46fa9448cd644eeed4fb612684b02dfda3a79cb84b",
-                "sha256:9ee3529454bf35c40a77115f5a596771e59e1aee8c53306f346c461b8e913094"
+                "sha256:3b835fc54e472c17ef447be37458b400e3fefdf14bb1ffdedb5d2c853acf4ba1",
+                "sha256:d2e9ddb1e463de871d32088a3f3cea3dc8282b1b2800e081bd0ef86900451667"
            ],
-            "version": "==2.4"
+            "version": "==2.5"
        },
        "parso": {
            "hashes": [
-                "sha256:63854233e1fadb5da97f2744b6b24346d2750b85965e7e399bec1620232797dc",
-                "sha256:666b0ee4a7a1220f65d367617f2cd3ffddff3e205f3f16a0284df30e774c2a9c"
+                "sha256:55cf25df1a35fd88b878715874d2c4dc1ad3f0eebd1e0266a67e1f55efccfbe1",
+                "sha256:5c1f7791de6bd5dbbeac8db0ef5594b36799de198b3f7f7014643b0c5536b9d3"
            ],
-            "version": "==0.5.1"
+            "version": "==0.5.2"
+        },
+        "pathspec": {
+            "hashes": [
+                "sha256:163b0632d4e31cef212976cf57b43d9fd6b0bac6e67c26015d611a647d5e7424",
+                "sha256:562aa70af2e0d434367d9790ad37aed893de47f1693e4201fd1d3dca15d19b96"
+            ],
+            "version": "==0.7.0"
        },
        "pexpect": {
            "hashes": [
@ -399,18 +389,17 @@
        },
        "pluggy": {
            "hashes": [
-                "sha256:0db4b7601aae1d35b4a033282da476845aa19185c1e6964b25cf324b5e4ec3e6",
-                "sha256:fa5fa1622fa6dd5c030e9cad086fa19ef6a0cf6d7a2d12318e10cb49d6d68f34"
+                "sha256:15b2acde666561e1298d71b523007ed7364de07029219b604cf808bfa1c765b0",
+                "sha256:966c145cd83c96502c3c3868f50408687b38434af77734af1e9ca461a4081d2d"
            ],
-            "version": "==0.13.0"
+            "version": "==0.13.1"
        },
        "prompt-toolkit": {
            "hashes": [
-                "sha256:11adf3389a996a6d45cc277580d0d53e8a5afd281d0c9ec71b28e6f121463780",
-                "sha256:2519ad1d8038fd5fc8e770362237ad0364d16a7650fb5724af6997ed5515e3c1",
-                "sha256:977c6583ae813a37dc1c2e1b715892461fcbdaa57f6fc62f33a528c4886c8f55"
+                "sha256:0278d2f51b5ceba6ea8da39f76d15684e84c996b325475f6e5720edc584326a7",
+                "sha256:63daee79aa8366c8f1c637f1a4876b890da5fc92a19ebd2f7080ebacb901e990"
            ],
-            "version": "==2.0.9"
+            "version": "==3.0.2"
        },
        "ptyprocess": {
            "hashes": [
@ -421,10 +410,10 @@
        },
        "py": {
            "hashes": [
-                "sha256:64f65755aee5b381cea27766a3a147c3f15b9b6b9ac88676de66ba2ae36793fa",
-                "sha256:dc639b046a6e2cff5bbe40194ad65936d6ba360b52b3c3fe1d08a82dd50b5e53"
+                "sha256:5e27081401262157467ad6e7f851b7aa402c5852dbcb3dae06768434de5752aa",
+                "sha256:c20fdd83a5dbc0af9efd622bee9a5564e278f6380fffcacc43ba6f43db2813b0"
            ],
-            "version": "==1.8.0"
+            "version": "==1.8.1"
        },
        "pycodestyle": {
            "hashes": [
@ -442,25 +431,25 @@
        },
        "pygments": {
            "hashes": [
-                "sha256:71e430bc85c88a430f000ac1d9b331d2407f681d6f6aec95e8bcfbc3df5b0127",
-                "sha256:881c4c157e45f30af185c1ffe8d549d48ac9127433f2c380c24b84572ad66297"
+                "sha256:2a3fe295e54a20164a9df49c75fa58526d3be48e14aceba6d6b1e8ac0bfd6f1b",
+                "sha256:98c8aa5a9f778fcd1026a17361ddaf7330d1b7c62ae97c3bb0ae73e0b9b6b0fe"
            ],
-            "version": "==2.4.2"
+            "version": "==2.5.2"
        },
        "pyparsing": {
            "hashes": [
-                "sha256:6f98a7b9397e206d78cc01df10131398f1c8b8510a2f4d97d9abd82e1aacdd80",
-                "sha256:d9338df12903bbf5d65a0e4e87c2161968b10d2e489652bb47001d82a9b028b4"
+                "sha256:4c830582a84fb022400b85429791bc551f1f4871c33f23e44f353119e92f969f",
+                "sha256:c342dccb5250c08d45fd6f8b4a559613ca603b57498511740e65cd11a2e7dcec"
            ],
-            "version": "==2.4.2"
+            "version": "==2.4.6"
        },
        "pytest": {
            "hashes": [
-                "sha256:813b99704b22c7d377bbd756ebe56c35252bb710937b46f207100e843440b3c2",
-                "sha256:cc6620b96bc667a0c8d4fa592a8c9c94178a1bd6cc799dbb057dfd9286d31a31"
+                "sha256:6b571215b5a790f9b41f19f3531c53a45cf6bb8ef2988bc1ff9afb38270b25fa",
+                "sha256:e41d489ff43948babd0fad7ad5e49b8735d5d55e26628a58673c39ff61d95de4"
            ],
            "index": "pypi",
-            "version": "==5.1.3"
+            "version": "==5.3.2"
        },
        "pytest-clarity": {
            "hashes": [
@ -471,9 +460,9 @@
        },
        "python-slugify": {
            "hashes": [
-                "sha256:575d03256a132fc1efb4c52966c6eb11c57a13b071618f0b26076057a23f6937"
+                "sha256:a8fc3433821140e8f409a9831d13ae5deccd0b033d4744d94b31fea141bdd84c"
            ],
-            "version": "==3.0.4"
+            "version": "==4.0.0"
        },
        "pytimeparse": {
            "hashes": [
@ -484,23 +473,49 @@
        },
        "pytz": {
            "hashes": [
-                "sha256:26c0b32e437e54a18161324a2fca3c4b9846b74a8dccddd843113109e1116b32",
-                "sha256:c894d57500a4cd2d5c71114aaab77dbab5eabd9022308ce5ac9bb93a60a6f0c7"
+                "sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d",
+                "sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be"
            ],
-            "version": "==2019.2"
+            "version": "==2019.3"
+        },
+        "regex": {
+            "hashes": [
+                "sha256:07b39bf943d3d2fe63d46281d8504f8df0ff3fe4c57e13d1656737950e53e525",
+                "sha256:0932941cdfb3afcbc26cc3bcf7c3f3d73d5a9b9c56955d432dbf8bbc147d4c5b",
+                "sha256:0e182d2f097ea8549a249040922fa2b92ae28be4be4895933e369a525ba36576",
+                "sha256:10671601ee06cf4dc1bc0b4805309040bb34c9af423c12c379c83d7895622bb5",
+                "sha256:23e2c2c0ff50f44877f64780b815b8fd2e003cda9ce817a7fd00dea5600c84a0",
+                "sha256:26ff99c980f53b3191d8931b199b29d6787c059f2e029b2b0c694343b1708c35",
+                "sha256:27429b8d74ba683484a06b260b7bb00f312e7c757792628ea251afdbf1434003",
+                "sha256:3e77409b678b21a056415da3a56abfd7c3ad03da71f3051bbcdb68cf44d3c34d",
+                "sha256:4e8f02d3d72ca94efc8396f8036c0d3bcc812aefc28ec70f35bb888c74a25161",
+                "sha256:4eae742636aec40cf7ab98171ab9400393360b97e8f9da67b1867a9ee0889b26",
+                "sha256:6a6ae17bf8f2d82d1e8858a47757ce389b880083c4ff2498dba17c56e6c103b9",
+                "sha256:6a6ba91b94427cd49cd27764679024b14a96874e0dc638ae6bdd4b1a3ce97be1",
+                "sha256:7bcd322935377abcc79bfe5b63c44abd0b29387f267791d566bbb566edfdd146",
+                "sha256:98b8ed7bb2155e2cbb8b76f627b2fd12cf4b22ab6e14873e8641f266e0fb6d8f",
+                "sha256:bd25bb7980917e4e70ccccd7e3b5740614f1c408a642c245019cff9d7d1b6149",
+                "sha256:d0f424328f9822b0323b3b6f2e4b9c90960b24743d220763c7f07071e0778351",
+                "sha256:d58e4606da2a41659c84baeb3cfa2e4c87a74cec89a1e7c56bee4b956f9d7461",
+                "sha256:e3cd21cc2840ca67de0bbe4071f79f031c81418deb544ceda93ad75ca1ee9f7b",
+                "sha256:e6c02171d62ed6972ca8631f6f34fa3281d51db8b326ee397b9c83093a6b7242",
+                "sha256:e7c7661f7276507bce416eaae22040fd91ca471b5b33c13f8ff21137ed6f248c",
+                "sha256:ecc6de77df3ef68fee966bb8cb4e067e84d4d1f397d0ef6fce46913663540d77"
+            ],
+            "version": "==2020.1.8"
        },
        "six": {
            "hashes": [
-                "sha256:3350809f0555b11f552448330d0b52d5f24c91a322ea4a15ef22629740f3761c",
-                "sha256:d16a0141ec1a18405cd4ce8b4613101da75da0e9a7aec5bdd4fa804d0e0eba73"
+                "sha256:1f1b7d42e254082a9db6279deae68afb421ceba6158efa6131de7b3003ee93fd",
+                "sha256:30f610279e8b2578cab6db20741130331735c781b56053c59c4076da27f06b66"
            ],
-            "version": "==1.12.0"
+            "version": "==1.13.0"
        },
        "sqlalchemy": {
            "hashes": [
-                "sha256:2f8ff566a4d3a92246d367f2e9cd6ed3edeef670dcd6dda6dfdc9efed88bcd80"
+                "sha256:bfb8f464a5000b567ac1d350b9090cf081180ec1ab4aa87e7bca12dab25320ec"
            ],
-            "version": "==1.3.8"
+            "version": "==1.3.12"
        },
        "termcolor": {
            "hashes": [
@ -524,17 +539,43 @@
        },
        "traitlets": {
            "hashes": [
-                "sha256:262089114405f22f4833be96b31e143ab906d7764a22c04c71fee0bbda4787ba",
-                "sha256:6ad5b30dacd5e2424c46cc94a0aeab990d98ae17d181acea2cc4272ac3409fca"
+                "sha256:70b4c6a1d9019d7b4f6846832288f86998aa3b9207c6821f3578a6a6a467fe44",
+                "sha256:d023ee369ddd2763310e4c3eae1ff649689440d4ae59d7485eb4cfbbe3e359f7"
            ],
-            "version": "==4.3.3.dev0"
+            "version": "==4.3.3"
+        },
+        "typed-ast": {
+            "hashes": [
+                "sha256:0666aa36131496aed8f7be0410ff974562ab7eeac11ef351def9ea6fa28f6355",
+                "sha256:0c2c07682d61a629b68433afb159376e24e5b2fd4641d35424e462169c0a7919",
+                "sha256:249862707802d40f7f29f6e1aad8d84b5aa9e44552d2cc17384b209f091276aa",
+                "sha256:24995c843eb0ad11a4527b026b4dde3da70e1f2d8806c99b7b4a7cf491612652",
+                "sha256:269151951236b0f9a6f04015a9004084a5ab0d5f19b57de779f908621e7d8b75",
+                "sha256:4083861b0aa07990b619bd7ddc365eb7fa4b817e99cf5f8d9cf21a42780f6e01",
+                "sha256:498b0f36cc7054c1fead3d7fc59d2150f4d5c6c56ba7fb150c013fbc683a8d2d",
+                "sha256:4e3e5da80ccbebfff202a67bf900d081906c358ccc3d5e3c8aea42fdfdfd51c1",
+                "sha256:6daac9731f172c2a22ade6ed0c00197ee7cc1221aa84cfdf9c31defeb059a907",
+                "sha256:715ff2f2df46121071622063fc7543d9b1fd19ebfc4f5c8895af64a77a8c852c",
+                "sha256:73d785a950fc82dd2a25897d525d003f6378d1cb23ab305578394694202a58c3",
+                "sha256:8c8aaad94455178e3187ab22c8b01a3837f8ee50e09cf31f1ba129eb293ec30b",
+                "sha256:8ce678dbaf790dbdb3eba24056d5364fb45944f33553dd5869b7580cdbb83614",
+                "sha256:aaee9905aee35ba5905cfb3c62f3e83b3bec7b39413f0a7f19be4e547ea01ebb",
+                "sha256:bcd3b13b56ea479b3650b82cabd6b5343a625b0ced5429e4ccad28a8973f301b",
+                "sha256:c9e348e02e4d2b4a8b2eedb48210430658df6951fa484e59de33ff773fbd4b41",
+                "sha256:d205b1b46085271b4e15f670058ce182bd1199e56b317bf2ec004b6a44f911f6",
+                "sha256:d43943ef777f9a1c42bf4e552ba23ac77a6351de620aa9acf64ad54933ad4d34",
+                "sha256:d5d33e9e7af3b34a40dc05f498939f0ebf187f07c385fd58d591c533ad8562fe",
+                "sha256:fc0fea399acb12edbf8a628ba8d2312f583bdbdb3335635db062fa98cf71fca4",
+                "sha256:fe460b922ec15dd205595c9b5b99e2f056fd98ae8f9f56b888e7a17dc2b757e7"
+            ],
+            "version": "==1.4.1"
        },
        "wcwidth": {
            "hashes": [
-                "sha256:3df37372226d6e63e1b1e1eda15c594bca98a22d33a23832a90998faa96bc65e",
-                "sha256:f4ebe71925af7b40a864553f761ed559b43544f8f71746c2d756c7fe788ade7c"
+                "sha256:8fd29383f539be45b20bd4df0dc29c20ba48654a41e661925e612311e9f3c603",
+                "sha256:f28b3e8a6483e5d49e7f8949ac1a78314e740333ae305b4ba5defd3e74fb37a8"
            ],
-            "version": "==0.1.7"
+            "version": "==0.1.8"
        },
        "xlrd": {
            "hashes": [
@ -543,13 +584,6 @@
            ],
            "index": "pypi",
            "version": "==1.2.0"
-        },
-        "zipp": {
-            "hashes": [
-                "sha256:3718b1cbcd963c7d4c5511a8240812904164b7f381b647143a89d3b98f9bcd8e",
-                "sha256:f06903e9f1f43b12d371004b4ac7b06ab39a44adc747266928ae6debfa7b3335"
-            ],
-            "version": "==0.6.0"
        }
    }
 }
--- a/README.md
+++ b/README.md
@ -1,7 +1,7 @@
 # CSV Metadata Quality [![Build Status](https://travis-ci.org/ilri/csv-metadata-quality.svg?branch=master)](https://travis-ci.org/ilri/csv-metadata-quality) [![builds.sr.ht status](https://builds.sr.ht/~alanorth/csv-metadata-quality.svg)](https://builds.sr.ht/~alanorth/csv-metadata-quality?)
-A simple, but opinionated metadata quality checker and fixer designed to work with CSVs in the DSpace ecosystem. The implementation is essentially a pipeline of checks and fixes that begins with splitting multi-value fields on the standard DSpace "||" separator, trimming leading/trailing whitespace, and then proceeding to more specialized cases like ISSNs, ISBNs, languages, etc.
+A simple, but opinionated metadata quality checker and fixer designed to work with CSVs in the DSpace ecosystem (though it could theoretically work on any CSV that uses Dublin Core fields as columns). The implementation is essentially a pipeline of checks and fixes that begins with splitting multi-value fields on the standard DSpace "||" separator, trimming leading/trailing whitespace, and then proceeding to more specialized cases like ISSNs, ISBNs, languages, etc.

-Requires Python 3.6 or greater. CSV and Excel support comes from the [Pandas](https://pandas.pydata.org/) library, though your mileage may vary with Excel because this is much less tested.
+Requires Python 3.8 or greater. CSV and Excel support comes from the [Pandas](https://pandas.pydata.org/) library, though your mileage may vary with Excel because this is much less tested.

 ## Functionality

@ -15,6 +15,7 @@ Requires Python 3.6 or greater. CSV and Excel support comes from the [Pandas](ht
 - Remove unnecessary Unicode like [non-breaking spaces](https://en.wikipedia.org/wiki/Non-breaking_space), [replacement characters](https://en.wikipedia.org/wiki/Specials_(Unicode_block)#Replacement_character), etc
 - Check for "suspicious" characters that indicate encoding or copy/paste issues, for example "foreˆt" should be "forêt"
 - Remove duplicate metadata values
+- Perform [Unicode normalization](https://withblue.ink/2019/03/11/why-you-need-to-normalize-unicode-strings.html) on strings using `--unsafe-fixes`

 ## Installation
 The easiest way to install CSV Metadata Quality is with [pipenv](https://github.com/pypa/pipenv):
@ -58,6 +59,14 @@ This is considered "unsafe" because it is *theoretically* possible for a single
 ### Newlines
 This is considered "unsafe" because some systems give special importance to vertical space and render it properly. DSpace does not support rendering newlines in its XMLUI and has, at times, suffered from parsing errors that cause the import process to fail if an input file had newlines. The `--unsafe-fixes` option strips Unix line feeds (U+000A).

+## Unicode Normalization
+[Unicode](https://en.wikipedia.org/wiki/Unicode) is a standard for encoding text. As the standard aims to support most of the world's languages, characters can often be represented in different ways and still be valid Unicode. This leads to interesting problems that can be confusing unless you know what's going on behind the scenes. For example, the characters `é` and `é` *look* the same, but are not — technically they refer to different code points in the Unicode standard:
+
+- `é` is the Unicode code point `U+00E9`
+- `é` is the Unicode code points `U+0065` + `U+0301`
+
+Read more about [Unicode normalization](https://withblue.ink/2019/03/11/why-you-need-to-normalize-unicode-strings.html).
+
 ## AGROVOC Validation
 You can enable validation of metadata values in certain fields against the AGROVOC REST API with the `--agrovoc-fields` option. For example, in addition to agricultural subjects, many countries and regions are also present AGROVOC. Enable this validation by specifying a comma-separated list of fields:

@ -92,7 +101,6 @@ This currently uses the [Python langid](https://github.com/saffsd/langid.py) lib
 - Validate DOIs? Normalize to https://doi.org format? Or use just the DOI part: 10.1016/j.worlddev.2010.06.006
 - Warn if two items use the same file in `filename` column
 - Add an option to drop invalid AGROVOC subjects?
- Add check for author names with incorrect spacing after commas, ie "Orth,Alan S."
 - Add tests for application invocation, ie `tests/test_app.py`?

 ## License
--- a/csv_metadata_quality/app.py
+++ b/csv_metadata_quality/app.py
@ -21,7 +21,8 @@ def parse_args(argv):
    parser.add_argument(
        "--experimental-checks",
        "-e",
-        help="Enable experimental checks like language detection", action="store_true"
+        help="Enable experimental checks like language detection",
+        action="store_true",
    )
    parser.add_argument(
        "--input-file",
@ -94,6 +95,11 @@ def run(argv):
            if match is not None:
                df[column] = df[column].apply(fix.comma_space, field_name=column)

+        # Fix: perform Unicode normalization (NFC) to convert decomposed
+        # characters into their canonical forms.
+        if args.unsafe_fixes:
+            df[column] = df[column].apply(fix.normalize_unicode, field_name=column)
+
        # Fix: unnecessary Unicode
        df[column] = df[column].apply(fix.unnecessary_unicode)

--- a/csv_metadata_quality/fix.py
+++ b/csv_metadata_quality/fix.py
@ -26,7 +26,7 @@ def whitespace(field):
        match = re.findall(pattern, value)

        if match:
-            print(f"Excessive whitespace: {value}")
+            print(f"Removing excessive whitespace: {value}")
            value = re.sub(pattern, " ", value)

        # Save cleaned value
@ -74,10 +74,10 @@ def unnecessary_unicode(field):
    Removes unnecessary Unicode characters like:
        - Zero-width space (U+200B)
        - Replacement character (U+FFFD)
-        - No-break space (U+00A0)

    Replaces unnecessary Unicode characters like:
        - Soft hyphen (U+00AD) → hyphen
+        - No-break space (U+00A0) → space

    Return string with characters removed or replaced.
    """
@ -107,8 +107,8 @@ def unnecessary_unicode(field):
    match = re.findall(pattern, field)

    if match:
-        print(f"Removing unnecessary Unicode (U+00A0): {field}")
-        field = re.sub(pattern, "", field)
+        print(f"Replacing unnecessary Unicode (U+00A0): {field}")
+        field = re.sub(pattern, " ", field)

    # Check for soft hyphens (U+00AD), sometimes preceeded with a normal hyphen
    pattern = re.compile(r"\u002D*?\u00AD")
@ -140,7 +140,7 @@ def duplicates(field):
        if value not in new_values:
            new_values.append(value)
        else:
-            print(f"Dropping duplicate value: {value}")
+            print(f"Removing duplicate value: {value}")

    # Create a new field consisting of all values joined with "||"
    new_field = "||".join(new_values)
@ -201,3 +201,27 @@ def comma_space(field, field_name):
        field = re.sub(r",(\w)", r", \1", field)

    return field
+
+
+def normalize_unicode(field, field_name):
+    """Fix occurrences of decomposed Unicode characters by normalizing them
+    with NFC to their canonical forms, for example:
+
+    Ouédraogo, Mathieu → Ouédraogo, Mathieu
+
+    Return normalized string.
+    """
+
+    from unicodedata import is_normalized
+    from unicodedata import normalize
+
+    # Skip fields with missing values
+    if pd.isna(field):
+        return
+
+    # Check if the current string is using normalized Unicode (NFC)
+    if not is_normalized("NFC", field):
+        print(f"Normalizing Unicode ({field_name}): {field}")
+        field = normalize("NFC", field)
+
+    return field
--- a/csv_metadata_quality/version.py
+++ b/csv_metadata_quality/version.py
@ -1 +1 @@
-VERSION = "0.3.0"
+VERSION = "0.4.0"
--- a/data/test.csv
+++ b/data/test.csv
@ -1,4 +1,4 @@
-dc.title,birthdate,dc.identifier.issn,dc.identifier.isbn,dc.language.iso,dc.subject,cg.coverage.country,filename
+dc.title,dc.date.issued,dc.identifier.issn,dc.identifier.isbn,dc.language.iso,dc.subject,cg.coverage.country,filename
 Leading space,2019-07-29,,,,,,
 Trailing space ,2019-07-29,,,,,,
 Excessive  space,2019-07-29,,,,,,
@ -26,3 +26,5 @@ Unneccesary unicode (U+002D + U+00AD),2019-08-10,,978-92-9043-823-6,,,,
 "Missing space,after comma",2019-08-27,,,,,,
 Incorrect ISO 639-1 language,2019-09-26,,,es,,,
 Incorrect ISO 639-3 language,2019-09-26,,,spa,,,
+Composéd Unicode,2020-01-14,,,,,,
+Decomposéd Unicode,2020-01-14,,,,,,
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@ -4,54 +4,53 @@ agate-excel==0.2.3
 agate-sql==0.5.4
 agate==1.6.1
 appdirs==1.4.3
-atomicwrites==1.3.0
-attrs==19.1.0
-babel==2.7.0
+attrs==19.3.0
+babel==2.8.0
 backcall==0.1.0
-black==19.3b0
+black==19.10b0
 click==7.0
 csvkit==1.0.4
 dbfread==2.0.7
-decorator==4.4.0
+decorator==4.4.1
 entrypoints==0.3
 et-xmlfile==1.0.1
-flake8==3.7.8
-future==0.17.1
-importlib-metadata==0.23 ; python_version < '3.8'
+flake8==3.7.9
 ipython-genutils==0.2.0
-ipython==7.8.0
+ipython==7.11.1
 isodate==0.6.0
 isort==4.3.21
 jdcal==1.4.1
-jedi==0.15.1
+jedi==0.15.2
 leather==0.3.3
 mccabe==0.6.1
-more-itertools==7.2.0
-openpyxl==3.0.0
-packaging==19.2
-parsedatetime==2.4
-parso==0.5.1
+more-itertools==8.1.0
+openpyxl==3.0.3
+packaging==20.0
+parsedatetime==2.5
+parso==0.5.2
+pathspec==0.7.0
 pexpect==4.7.0 ; sys_platform != 'win32'
 pickleshare==0.7.5
-pluggy==0.13.0
-prompt-toolkit==2.0.9
+pluggy==0.13.1
+prompt-toolkit==3.0.2
 ptyprocess==0.6.0
-py==1.8.0
+py==1.8.1
 pycodestyle==2.5.0
 pyflakes==2.1.1
-pygments==2.4.2
-pyparsing==2.4.2
+pygments==2.5.2
+pyparsing==2.4.6
 pytest-clarity==0.2.0a1
-pytest==5.1.3
-python-slugify==3.0.4
+pytest==5.3.2
+python-slugify==4.0.0
 pytimeparse==1.1.8
-pytz==2019.2
-six==1.12.0
-sqlalchemy==1.3.8
+pytz==2019.3
+regex==2020.1.8
+six==1.13.0
+sqlalchemy==1.3.12
 termcolor==1.1.0
 text-unidecode==1.3
 toml==0.10.0
-traitlets==4.3.3.dev0
-wcwidth==0.1.7
+traitlets==4.3.3
+typed-ast==1.4.1
+wcwidth==0.1.8
 xlrd==1.2.0
-zipp==0.6.0
--- a/requirements.txt
+++ b/requirements.txt
@ -1,17 +1,17 @@
 -i https://pypi.org/simple
 -e .
-certifi==2019.9.11
+certifi==2019.11.28
 chardet==3.0.4
 idna==2.8
 langid==1.1.6
-numpy==1.17.2
-pandas==0.25.1
+numpy==1.18.1
+pandas==1.0.0rc0
 pycountry==19.8.18
-python-dateutil==2.8.0
-python-stdnum==1.11
-pytz==2019.2
+python-dateutil==2.8.1
+python-stdnum==1.12
+pytz==2019.3
 requests-cache==0.5.2
 requests==2.22.0
-six==1.12.0
-urllib3==1.25.6
+six==1.13.0
+urllib3==1.25.7
 xlrd==1.2.0
--- a/setup.py
+++ b/setup.py
@ -4,17 +4,17 @@ with open("README.md", "r") as fh:
    long_description = fh.read()

 install_requires = [
-    'pandas',
-    'python-stdnum',
-    'requests',
-    'requests-cache',
-    'pycountry',
-    'langid'
+    "pandas",
+    "python-stdnum",
+    "requests",
+    "requests-cache",
+    "pycountry",
+    "langid",
 ]

 setuptools.setup(
    name="csv-metadata-quality",
-    version="0.3.0",
+    version="0.4.0",
    author="Alan Orth",
    author_email="aorth@mjanja.ch",
    description="A simple, but opinionated CSV quality checking and fixing pipeline for CSVs in the DSpace ecosystem.",
@ -23,17 +23,14 @@ setuptools.setup(
    long_description_content_type="text/markdown",
    url="https://github.com/alanorth/csv-metadata-quality",
    classifiers=[
-        "Programming Language :: Python :: 3.6",
-        "Programming Language :: Python :: 3.7",
+        "Programming Language :: Python :: 3.8",
        "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
        "Operating System :: OS Independent",
-        "Development Status :: 4 - Beta"
+        "Development Status :: 4 - Beta",
    ],
-    packages=['csv_metadata_quality'],
+    packages=["csv_metadata_quality"],
    entry_points={
-        'console_scripts': [
-            'csv-metadata-quality = csv_metadata_quality.__main__:main'
-        ]
+        "console_scripts": ["csv-metadata-quality = csv_metadata_quality.__main__:main"]
    },
-    install_requires=install_requires
+    install_requires=install_requires,
 )
--- a/tests/test_check.py
+++ b/tests/test_check.py
@ -1,6 +1,7 @@
+import pandas as pd
+
 import csv_metadata_quality.check as check
 import csv_metadata_quality.experimental as experimental
-import pandas as pd


 def test_check_invalid_issn(capsys):
--- a/tests/test_fix.py
+++ b/tests/test_fix.py
@ -66,3 +66,25 @@ def test_fix_comma_space():
    field_name = "dc.contributor.author"

    assert fix.comma_space(value, field_name) == "Orth, Alan S."
+
+
+def test_fix_normalized_unicode():
+    """Test fixing a string that is already in its normalized (NFC) Unicode form."""
+
+    # string using the normalized canonical form of é
+    value = "Ouédraogo, Mathieu"
+
+    field_name = "dc.contributor.author"
+
+    assert fix.normalize_unicode(value, field_name) == "Ouédraogo, Mathieu"
+
+
+def test_fix_decomposed_unicode():
+    """Test fixing a string that contains Unicode string."""
+
+    # string using the decomposed form of é
+    value = "Ouédraogo, Mathieu"
+
+    field_name = "dc.contributor.author"
+
+    assert fix.normalize_unicode(value, field_name) == "Ouédraogo, Mathieu"
Author	SHA1	Message	Date
Alan Orth	705127fd28	Version 0.4.0	2020-01-15 11:44:56 +02:00
Alan Orth	894e0a196d	setup.py: Change Python requirements The `unicodedata.is_normalized()` function requires Python 3.8. See: https://docs.python.org/3/library/unicodedata.html	2020-01-15 11:43:25 +02:00
Alan Orth	87181bc7b8	Run black, isort, and flake8.	2020-01-15 11:41:31 +02:00
Alan Orth	8de5d862b6	CHANGELOG.md: Add note about Unicode normalization	2020-01-15 11:40:40 +02:00
Alan Orth	49e3543878	Add Unicode normalization This will check all strings for un-normalized Unicode characters. Normalization is done using NFC. This includes tests and updated sample data (data/test.csv). See: https://withblue.ink/2019/03/11/why-you-need-to-normalize-unicode-strings.html	2020-01-15 11:37:54 +02:00
Alan Orth	403b253762	CHANGELOG.md: Update python library versions	2020-01-15 10:58:44 +02:00
Alan Orth	c5fbaf407a	Update python requirements Generated using pipenv: $ pipenv lock -r > requirements.txt $ pipenv lock -r -d > requirements-dev.txt	2020-01-15 10:51:58 +02:00
Alan Orth	4f81f6c83c	Pipfile.lock: Run pipenv update	2020-01-15 10:51:19 +02:00
Alan Orth	4b9d1e060f	setup.py: Add Python 3.8 classifier	2019-12-14 12:56:11 +02:00
Alan Orth	c8a71e3143	Pipfile.lock: Run pipenv update	2019-12-14 12:53:39 +02:00
Alan Orth	7964d98ca5	Pipfile: Specify exact version of black Black only releases pre-release versions, which causes issues with pipenv. Instead of always running pipenv with "--pre" and potenti- ally letting in some other pre-release versions for other depende- ncies, I would rather specify the latest black version explicitly. See: https://github.com/psf/black/issues/517 See: https://github.com/microsoft/vscode-python/issues/5171	2019-12-14 12:41:28 +02:00
Alan Orth	64ffc2f1da	.travis.yml: Install packages from requirements.txt too	2019-11-14 23:42:28 +02:00
Alan Orth	7b1bc29a92	.travis.yml: Try using pip instead of pipenv The Pipfile knows it was created with Python 3.8, yet we're running with multiple Python versions on Travis. I'm curious if would work better to use pip to install dependencies instead of pipenv in this case.	2019-11-14 23:37:25 +02:00
Alan Orth	f0110d8e74	CHANGELOG.md: Add note about requirements	2019-11-14 23:30:26 +02:00
Alan Orth	86498deee8	Update python requirements Generated using pipenv: $ pipenv lock -r > requirements.txt $ pipenv lock -r -d > requirements-dev.txt	2019-11-14 23:28:42 +02:00
Alan Orth	251647a15f	CHANGELOG.md: Add TravisCI changes	2019-11-14 23:24:08 +02:00
Alan Orth	0bd28e22ec	.travis.yml: Test Python 3.8	2019-11-14 23:22:37 +02:00
Alan Orth	63fdce7d13	.travis.yml: Use Ubuntu 18.04 "Bionic"	2019-11-14 23:22:19 +02:00
Alan Orth	f068c0e16a	CHANGELOG.md: Use Python 3.8.0 for pipenv	2019-11-14 23:11:43 +02:00
Alan Orth	79b8f62a85	Use Python 3.8 for pipenv Python 3.8.0 entered Arch Linux core repositories now and all tests pass with Python 3.8.0 so it's time...	2019-11-14 23:10:20 +02:00
Alan Orth	6c1e132531	CHANGELOG.md: Add unreleased changes	2019-11-14 09:19:19 +02:00
Alan Orth	c0f3c866bd	Pipfile.lock: Run pipenv update Updates the following dependencies: - numpy 1.17.2→1.17.4 - pandas 0.25.1→0.25.3 - flake8 3.7.8→3.7.9 - pytest 5.1.3→5.2.2 - black 19.3b0→19.10b0	2019-11-14 09:17:31 +02:00
Alan Orth	36d0474b95	CHANGELOG.md: Move unreleased changes to v0.3.1	2019-10-01 17:11:52 +03:00
Alan Orth	efdc3a841a	Version 0.3.1	2019-10-01 17:11:13 +03:00
Alan Orth	fd2ba6845d	CHANGELOG.md: Update unreleased notes	2019-10-01 17:10:23 +03:00
Alan Orth	e55380b4d5	csv_metadata_quality/fix.py: Harmonize language in fix output We should always say if we're removing or replacing something.	2019-10-01 17:09:49 +03:00
Alan Orth	85ae16d9b7	CHANGELOG.md: Add note about non-breaking spaces	2019-10-01 16:56:37 +03:00
Alan Orth	c42f8b4812	csv_metadata_quality/fix.py: Replace non-breaking spaces We should be replacing non-breaking spaces (U+00A0) with normal sp- aces instead of removing them.	2019-10-01 16:55:04 +03:00
Alan Orth	1c75608d54	README.md: Update introduction text We should mention that this is not DSpace specific. Rather, it is much more realistically Dublin Core specific.	2019-09-26 14:19:13 +03:00
Alan Orth	0b15a8ed3b	README.md: Remove TODO about lack of space after comma This was added as an automatic global fix a few weeks ago.	2019-09-26 14:16:33 +03:00
Alan Orth	9ca266f5f0	data/test.csv: Change birthdate column to dc.date.issued More accurately reflects actual data we will be validating.	2019-09-26 14:15:48 +03:00
Alan Orth	0d3f948708	CHANGELOG.md: Update comment about language validation	2019-09-26 14:14:57 +03:00
Alan Orth	c04207fcfc	CHANGELOG.md: Fix header formatting	2019-09-26 14:13:50 +03:00
Alan Orth	9d4eceddc7	.build.yml: Enable experimental CLI checks on SourceHut	2019-09-26 14:11:35 +03:00