Add check for missing DOIs

Sometimes an editor includes a DOI in the citation field, but does
not add a standalone DOI field.

CHANGELOG.md

@@ -4,6 +4,19 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## Unreleased
+### Added
+- Ability to check for, and fix, "mojibake" characters using [ftfy](https://github.com/LuminosoInsight/python-ftfy)
+
+### Updated
+- Python dependencies
+
+### Fixed
+- Regular expression to match all citation fields (dc.identifier.citation as
+well as dcterms.bibliographicCitation) in `experimental.correct_language()`
+- Regular expression to match dc.title and dcterms.title, but
+ignore dc.title.alternative `check.duplicate_items()`
+
 ## [0.4.7] - 2021-03-17
 ### Changed
 - Fixing invalid multi-value separators like `|` and `|||` is no longer class-

CITATION.cff

@@ -0,0 +1,19 @@
+cff-version: "1.1.0"
+abstract: "A simple but opinionated metadata quality checker and fixer designed to work with CSVs in the DSpace ecosystem."
+authors: 
+  -
+    affiliation: "International Livestock Research Institute"
+    family-names: Orth
+    given-names: "Alan S."
+    orcid: "https://orcid.org/0000-0002-1735-7458"
+date-released: 2019-07-26
+doi: "10568/110997"
+keywords: 
+  - dspace
+  - "dublin-core"
+  - csv
+  - metadata
+license: "GPL-3.0-only"
+message: "If you use this software, please cite it using these metadata."
+repository-code: "https://github.com/ilri/csv-metadata-quality"
+title: "DSpace CSV Metadata Quality Checker"

README.md

@@ -20,7 +20,9 @@ If you use the DSpace CSV metadata quality checker please cite:
 - Perform [Unicode normalization](https://withblue.ink/2019/03/11/why-you-need-to-normalize-unicode-strings.html) on strings using `--unsafe-fixes`
 - Remove unnecessary Unicode like [non-breaking spaces](https://en.wikipedia.org/wiki/Non-breaking_space), [replacement characters](https://en.wikipedia.org/wiki/Specials_(Unicode_block)#Replacement_character), etc
 - Check for "suspicious" characters that indicate encoding or copy/paste issues, for example "foreˆt" should be "forêt"
+- Check for "mojibake" characters (and attempt to fix with `--unsafe-fixes`)
 - Remove duplicate metadata values
+- Check for duplicate items, using the title, type, and date issued as an indicator
 
 ## Installation
 The easiest way to install CSV Metadata Quality is with [poetry](https://python-poetry.org):
@@ -61,7 +63,7 @@ While it is *theoretically* possible for a single `|` character to be used legit
 This will also remove unnecessary trailing multi-value separators, for example `Kenya||Tanzania||`.
 
 ## Unsafe Fixes
-You can enable several "unsafe" fixes with the `--unsafe-fixes` option. Currently this will remove newlines and perform Unicode normalization.
+You can enable several "unsafe" fixes with the `--unsafe-fixes` option. Currently this will remove newlines, perform Unicode normalization, and attempt to fix "mojibake" characters.
 
 ### Newlines
 This is considered "unsafe" because some systems give special importance to vertical space and render it properly. DSpace does not support rendering newlines in its XMLUI and has, at times, suffered from parsing errors that cause the import process to fail if an input file had newlines. The `--unsafe-fixes` option strips Unix line feeds (U+000A).
@@ -74,6 +76,14 @@ This is considered "unsafe" because some systems give special importance to vert
 
 Read more about [Unicode normalization](https://withblue.ink/2019/03/11/why-you-need-to-normalize-unicode-strings.html).
 
+### Encoding Issues aka "Mojibake"
+[Mojibake](https://en.wikipedia.org/wiki/Mojibake) is a phenomenon that occurs when text is decoded using an unintended character encoding. This usually presents itself in the form of strange, garbled characters in the text. Enabling "unsafe" fixes will attempt to correct these, for example:
+
+- CIAT Publicaçao → CIAT Publicaçao
+- CIAT Publicación → CIAT Publicación
+
+Pay special attention to the output of the script as well as the resulting file to make sure no new issues have been introduced. The ideal way to solve these issues is to avoid it in the first place. See [this guide about opening CSVs in UTF-8 format in Excel](https://www.itg.ias.edu/content/how-import-csv-file-uses-utf-8-character-encoding-0).
+
 ## AGROVOC Validation
 You can enable validation of metadata values in certain fields against the AGROVOC REST API with the `--agrovoc-fields` option. For example, in addition to agricultural subjects, many countries and regions are also present AGROVOC. Enable this validation by specifying a comma-separated list of fields:
 
@@ -116,10 +126,6 @@ This currently uses the [Python langid](https://github.com/saffsd/langid.py) lib
   - Warn if item is Open Access, but missing a license
   - Warn if item has an ISSN but no journal title
   - Update journal titles from ISSN
-- Check for duplicates
-  - If I check titles only, then I might miss if one is a Report and another is a Presentation
-  - I could just check each item against each other item, but that sounds slow...
-  - Perhaps I could check for the number of unique values in a few rows, like title and doi, and see if it is the same as the total number of items
 
 ## License
 This work is licensed under the [GPLv3](https://www.gnu.org/licenses/gpl-3.0.en.html).

csv_metadata_quality/__main__.py

@@ -1,3 +1,5 @@
+# SPDX-License-Identifier: GPL-3.0-only
+
 from sys import argv
 
 from csv_metadata_quality import app

csv_metadata_quality/app.py

@@ -1,3 +1,5 @@
+# SPDX-License-Identifier: GPL-3.0-only
+
 import argparse
 import re
 import signal
@@ -107,6 +109,13 @@ def run(argv):
         # Check: suspicious characters
         df[column].apply(check.suspicious_characters, field_name=column)
 
+        # Check: mojibake
+        df[column].apply(check.mojibake, field_name=column)
+
+        # Fix: mojibake
+        if args.unsafe_fixes:
+            df[column] = df[column].apply(fix.mojibake, field_name=column)
+
         # Fix: invalid and unnecessary multi-value separators
         df[column] = df[column].apply(fix.separators, field_name=column)
         # Run whitespace fix again after fixing invalid separators
@@ -155,13 +164,16 @@ def run(argv):
 
     # Check: duplicate items
     # We extract just the title, type, and date issued columns to analyze
-    duplicates_df = df.filter(
-        regex=r"dcterms\.title|dc\.title|dcterms\.type|dc\.type|dcterms\.issued|dc\.date\.issued"
-    )
-    check.duplicate_items(duplicates_df)
+    try:
+        duplicates_df = df.filter(
+            regex=r"dcterms\.title|dc\.title|dcterms\.type|dc\.type|dcterms\.issued|dc\.date\.issued"
+        )
+        check.duplicate_items(duplicates_df)
 
-    # Delete the temporary duplicates DataFrame
-    del duplicates_df
+        # Delete the temporary duplicates DataFrame
+        del duplicates_df
+    except IndexError:
+        pass
 
     ##
     # Perform some checks on rows so we can consider items as a whole rather
@@ -174,11 +186,16 @@ def run(argv):
     # column. For now it will have to do.
     ##
 
-    if args.experimental_checks:
-        # Transpose the DataFrame so we can consider each row as a column
-        df_transposed = df.T
+    # Transpose the DataFrame so we can consider each row as a column
+    df_transposed = df.T
 
-        for column in df_transposed.columns:
+    # Remember, here a "column" is an item (previously row). Perhaps I
+    # should rename column in this for loop...
+    for column in df_transposed.columns:
+        # Check: citation DOI
+        check.citation_doi(df_transposed[column])
+
+        if args.experimental_checks:
             experimental.correct_language(df_transposed[column])
 
     # Write

csv_metadata_quality/check.py

@@ -1,3 +1,5 @@
+# SPDX-License-Identifier: GPL-3.0-only
+
 import os
 import re
 from datetime import datetime, timedelta
@@ -11,6 +13,8 @@ from pycountry import languages
 from stdnum import isbn as stdnum_isbn
 from stdnum import issn as stdnum_issn
 
+from csv_metadata_quality.util import is_mojibake
+
 
 def issn(field):
     """Check if an ISSN is valid.
@@ -174,13 +178,9 @@ def language(field):
         if len(value) == 2:
             if not languages.get(alpha_2=value):
                 print(f"{Fore.RED}Invalid ISO 639-1 language: {Fore.RESET}{value}")
-
-                pass
         elif len(value) == 3:
             if not languages.get(alpha_3=value):
                 print(f"{Fore.RED}Invalid ISO 639-3 language: {Fore.RESET}{value}")
-
-                pass
         else:
             print(f"{Fore.RED}Invalid language: {Fore.RESET}{value}")
 
@@ -216,7 +216,7 @@ def agrovoc(field, field_name):
     )
 
     # prune old cache entries
-    requests_cache.core.remove_expired_responses()
+    requests_cache.remove_expired_responses()
 
     # Try to split multi-value field on "||" separator
     for value in field.split("||"):
@@ -301,8 +301,6 @@ def spdx_license_identifier(field):
         if value not in spdx_license_list.LICENSES:
             print(f"{Fore.YELLOW}Non-SPDX license identifier: {Fore.RESET}{value}")
 
-            pass
-
     return
 
 
@@ -323,10 +321,16 @@ def duplicate_items(df):
     #
     #   Index(['dcterms.title[en_US]'], dtype='object')
     #
-    title_column_name = df.filter(regex=r"dcterms\.title|dc\.title").columns[0]
-    type_column_name = df.filter(regex=r"dcterms\.title|dc\.title").columns[0]
+    # But, we need to consider that dc.title.alternative might come before the
+    # main title in the CSV, so use a negative lookahead to eliminate that.
+    #
+    # See: https://regex101.com/r/elyXkW/1
+    title_column_name = df.filter(
+        regex=r"^(dc|dcterms)\.title(?!\.alternative).*$"
+    ).columns[0]
+    type_column_name = df.filter(regex=r"^(dcterms\.type|dc\.type).*$").columns[0]
     date_column_name = df.filter(
-        regex=r"dcterms\.issued|dc\.date\.accessioned"
+        regex=r"^(dcterms\.issued|dc\.date\.accessioned).*$"
     ).columns[0]
 
     items_count_total = df[title_column_name].count()
@@ -345,3 +349,64 @@ def duplicate_items(df):
                 )
             else:
                 items.append(item_title_type_date)
+
+
+def mojibake(field, field_name):
+    """Check for mojibake (text that was encoded in one encoding and decoded in
+    in another, perhaps multiple times). See util.py.
+
+    Prints the string if it contains suspected mojibake.
+    """
+
+    # Skip fields with missing values
+    if pd.isna(field):
+        return
+
+    if is_mojibake(field):
+        print(
+            f"{Fore.YELLOW}Possible encoding issue ({field_name}): {Fore.RESET}{field}"
+        )
+
+    return
+
+
+def citation_doi(row):
+    """Check for the scenario where an item has a DOI listed in its citation,
+    but does not have a cg.identifier.doi field.
+
+    Function prints a warning if the DOI field is missing, but there is a DOI
+    in the citation.
+    """
+    # Initialize some variables at global scope so that we can set them in the
+    # loop scope below and still be able to access them afterwards.
+    citation = ""
+
+    # Iterate over the labels of the current row's values to check if a DOI
+    # exists. If not, then we extract the citation to see if there is a DOI
+    # listed there.
+    for label in row.axes[0]:
+        # Skip fields with missing values
+        if pd.isna(row[label]):
+            continue
+
+        # If a DOI field exists we don't need to check the citation
+        match = re.match(r"^.*?doi.*$", label)
+        if match is not None:
+            return
+
+        # Get the name of the citation field
+        match = re.match(r"^.*?[cC]itation.*$", label)
+        if match is not None:
+            citation = row[label]
+
+    if citation != "":
+        # Check the citation for "doi: 10.1186/1743-422X-9-218"
+        doi_match1 = re.match(r"^.*?doi:\s.*$", citation)
+        # Check the citation for a DOI URL (doi.org, dx.doi.org, etc)
+        doi_match2 = re.match(r"^.*?doi\.org.*$", citation)
+        if doi_match1 is not None or doi_match2 is not None:
+            print(
+                f"{Fore.YELLOW}DOI in citation, but missing a DOI field: {Fore.RESET}{citation}"
+            )
+
+    return

csv_metadata_quality/experimental.py

@@ -1,3 +1,5 @@
+# SPDX-License-Identifier: GPL-3.0-only
+
 import re
 
 import langid
@@ -50,7 +52,7 @@ def correct_language(row):
             sample_strings.append(row[label])
 
         # Extract citation if it is present
-        match = re.match(r"^.*?citation.*$", label)
+        match = re.match(r"^.*?[cC]itation.*$", label)
         if match is not None:
             sample_strings.append(row[label])
 

csv_metadata_quality/fix.py

@@ -1,10 +1,13 @@
+# SPDX-License-Identifier: GPL-3.0-only
+
 import re
 from unicodedata import normalize
 
 import pandas as pd
 from colorama import Fore
+from ftfy import fix_text
 
-from csv_metadata_quality.util import is_nfc
+from csv_metadata_quality.util import is_mojibake, is_nfc
 
 
 def whitespace(field, field_name):
@@ -253,3 +256,22 @@ def normalize_unicode(field, field_name):
         field = normalize("NFC", field)
 
     return field
+
+
+def mojibake(field, field_name):
+    """Attempts to fix mojibake (text that was encoded in one encoding and deco-
+    ded in another, perhaps multiple times). See util.py.
+
+    Return fixed string.
+    """
+
+    # Skip fields with missing values
+    if pd.isna(field):
+        return field
+
+    if is_mojibake(field):
+        print(f"{Fore.GREEN}Fixing encoding issue ({field_name}): {Fore.RESET}{field}")
+
+        return fix_text(field)
+    else:
+        return field

csv_metadata_quality/util.py

@@ -1,3 +1,8 @@
+# SPDX-License-Identifier: GPL-3.0-only
+
+from ftfy.badness import sequence_weirdness
+
+
 def is_nfc(field):
     """Utility function to check whether a string is using normalized Unicode.
     Python's built-in unicodedata library has the is_normalized() function, but
@@ -12,3 +17,35 @@ def is_nfc(field):
     from unicodedata import normalize
 
     return field == normalize("NFC", field)
+
+
+def is_mojibake(field):
+    """Determines whether a string contains mojibake.
+
+    We commonly deal with CSV files that were *encoded* in UTF-8, but decoded
+    as something else like CP-1252 (Windows Latin). This manifests in the form
+    of "mojibake", for example:
+
+        - CIAT Publicaçao
+        - CIAT Publicación
+
+    This uses the excellent "fixes text for you" (ftfy) library to determine
+    whether a string contains characters that have been encoded in one encoding
+    and decoded in another.
+
+    Inspired by this code snippet from Martijn Pieters on StackOverflow:
+    https://stackoverflow.com/questions/29071995/identify-garbage-unicode-string-using-python
+
+    Return boolean.
+    """
+    if not sequence_weirdness(field):
+        # Nothing weird, should be okay
+        return False
+    try:
+        field.encode("sloppy-windows-1252")
+    except UnicodeEncodeError:
+        # Not CP-1252 encodable, probably fine
+        return False
+    else:
+        # Encodable as CP-1252, Mojibake alert level high
+        return True

csv_metadata_quality/version.py

@@ -1 +1,3 @@
-VERSION = "0.4.7"
+# SPDX-License-Identifier: GPL-3.0-only
+
+VERSION = "0.4.8-dev"

data/test.csv

@@ -1,34 +1,36 @@
-dc.title,dcterms.issued,dc.identifier.issn,dc.identifier.isbn,dcterms.language,dcterms.subject,cg.coverage.country,filename,dcterms.license,dcterms.type
- Leading space,2019-07-29,,,,,,,,
-Trailing space ,2019-07-29,,,,,,,,
-Excessive  space,2019-07-29,,,,,,,,
-Miscellaenous ||whitespace | issues ,2019-07-29,,,,,,,,
-Duplicate||Duplicate,2019-07-29,,,,,,,,
-Invalid ISSN,2019-07-29,2321-2302,,,,,,,
-Invalid ISBN,2019-07-29,,978-0-306-40615-6,,,,,,
-Multiple valid ISSNs,2019-07-29,0378-5955||0024-9319,,,,,,,
-Multiple valid ISBNs,2019-07-29,,99921-58-10-7||978-0-306-40615-7,,,,,,
-Invalid date,2019-07-260,,,,,,,,
-Multiple dates,2019-07-26||2019-01-10,,,,,,,,
-Invalid multi-value separator,2019-07-29,0378-5955|0024-9319,,,,,,,
-Unnecessary Unicode​,2019-07-29,,,,,,,,
-Suspicious character||foreˆt,2019-07-29,,,,,,,,
-Invalid ISO 639-1 (alpha 2) language,2019-07-29,,,jp,,,,,
-Invalid ISO 639-3 (alpha 3) language,2019-07-29,,,chi,,,,,
-Invalid language,2019-07-29,,,Span,,,,,
-Invalid AGROVOC subject,2019-07-29,,,,FOREST,,,,
+dc.title,dcterms.issued,dc.identifier.issn,dc.identifier.isbn,dcterms.language,dcterms.subject,cg.coverage.country,filename,dcterms.license,dcterms.type,dcterms.bibliographicCitation,cg.identifier.doi
+ Leading space,2019-07-29,,,,,,,,,,
+Trailing space ,2019-07-29,,,,,,,,,,
+Excessive  space,2019-07-29,,,,,,,,,,
+Miscellaenous ||whitespace | issues ,2019-07-29,,,,,,,,,,
+Duplicate||Duplicate,2019-07-29,,,,,,,,,,
+Invalid ISSN,2019-07-29,2321-2302,,,,,,,,,
+Invalid ISBN,2019-07-29,,978-0-306-40615-6,,,,,,,,
+Multiple valid ISSNs,2019-07-29,0378-5955||0024-9319,,,,,,,,,
+Multiple valid ISBNs,2019-07-29,,99921-58-10-7||978-0-306-40615-7,,,,,,,,
+Invalid date,2019-07-260,,,,,,,,,,
+Multiple dates,2019-07-26||2019-01-10,,,,,,,,,,
+Invalid multi-value separator,2019-07-29,0378-5955|0024-9319,,,,,,,,,
+Unnecessary Unicode​,2019-07-29,,,,,,,,,,
+Suspicious character||foreˆt,2019-07-29,,,,,,,,,,
+Invalid ISO 639-1 (alpha 2) language,2019-07-29,,,jp,,,,,,,
+Invalid ISO 639-3 (alpha 3) language,2019-07-29,,,chi,,,,,,,
+Invalid language,2019-07-29,,,Span,,,,,,,
+Invalid AGROVOC subject,2019-07-29,,,,FOREST,,,,,,
 Newline (LF),2019-07-30,,,,"TANZA
-NIA",,,,
-Missing date,,,,,,,,,
-Invalid country,2019-08-01,,,,,KENYAA,,,
-Uncommon filename extension,2019-08-10,,,,,,file.pdf.lck,,
-Unneccesary unicode (U+002D + U+00AD),2019-08-10,,978-­92-­9043-­823-­6,,,,,,
-"Missing space,after comma",2019-08-27,,,,,,,,
-Incorrect ISO 639-1 language,2019-09-26,,,es,,,,,
-Incorrect ISO 639-3 language,2019-09-26,,,spa,,,,,
-Composéd Unicode,2020-01-14,,,,,,,,
-Decomposéd Unicode,2020-01-14,,,,,,,,
-Unnecessary multi-value separator,2021-01-03,0378-5955||,,,,,,,
-Invalid SPDX license identifier,2021-03-11,,,,,,,CC-BY,
-Duplicate Title,2021-03-17,,,,,,,,Report
-Duplicate Title,2021-03-17,,,,,,,,Report
+NIA",,,,,,
+Missing date,,,,,,,,,,,
+Invalid country,2019-08-01,,,,,KENYAA,,,,,
+Uncommon filename extension,2019-08-10,,,,,,file.pdf.lck,,,,
+Unneccesary unicode (U+002D + U+00AD),2019-08-10,,978-­92-­9043-­823-­6,,,,,,,,
+"Missing space,after comma",2019-08-27,,,,,,,,,,
+Incorrect ISO 639-1 language,2019-09-26,,,es,,,,,,,
+Incorrect ISO 639-3 language,2019-09-26,,,spa,,,,,,,
+Composéd Unicode,2020-01-14,,,,,,,,,,
+Decomposéd Unicode,2020-01-14,,,,,,,,,,
+Unnecessary multi-value separator,2021-01-03,0378-5955||,,,,,,,,,
+Invalid SPDX license identifier,2021-03-11,,,,,,,CC-BY,,,
+Duplicate Title,2021-03-17,,,,,,,,Report,,
+Duplicate Title,2021-03-17,,,,,,,,Report,,
+Mojibake,2021-03-18,,,,Publicaçao CIAT,,,,Report,,
+"DOI in citation, but missing cg.identifier.doi",2021-10-06,,,,,,,,,"Orth, A. 2021. Testing all the things. doi: 10.1186/1743-422X-9-218",

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "csv-metadata-quality"
-version = "0.4.7"
+version = "0.4.8-dev"
 description="A simple, but opinionated CSV quality checking and fixing pipeline for CSVs in the DSpace ecosystem."
 authors = ["Alan Orth <alan.orth@gmail.com>"]
 license="GPL-3.0-only"
@@ -16,18 +16,20 @@ pandas = "^1.0.4"
 python-stdnum = "^1.13"
 xlrd = "^1.2.0"
 requests = "^2.23.0"
-requests-cache = "^0.5.2"
+requests-cache = "~0.6.4"
 pycountry = "^19.8.18"
 langid = "^1.1.6"
 colorama = "^0.4.4"
 spdx-license-list = "^0.5.2"
+ftfy = "^5.9"
+SQLAlchemy = ">=1.3.3,<1.4.23"
 
 [tool.poetry.dev-dependencies]
 pytest = "^6.1.1"
 ipython = { version = "^7.18.1", python = "^3.7" }
 flake8 = "^3.8.4"
-pytest-clarity = "^0.3.0-alpha.0"
-black = "20.8b1"
+pytest-clarity = "^1.0.1"
+black = "^21.6b0"
 isort = "^5.5.4"
 csvkit = "^1.0.5"
 

requirements-dev.txt

@@ -1,75 +1,82 @@
 agate-dbf==0.2.2
-agate-excel==0.2.3
-agate-sql==0.5.6
-agate==1.6.2
-appdirs==1.4.4; python_version >= "3.6"
+agate-excel==0.2.5
+agate-sql==0.5.8
+agate==1.6.3
 appnope==0.1.2; python_version >= "3.7" and python_version < "4.0" and sys_platform == "darwin"
 atomicwrites==1.4.0; python_version >= "3.6" and python_full_version < "3.0.0" and sys_platform == "win32" and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6") or sys_platform == "win32" and python_version >= "3.6" and python_full_version >= "3.4.0" and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6")
-attrs==20.3.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
-babel==2.9.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0"
+attrs==21.2.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.6"
+babel==2.9.1; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0"
 backcall==0.2.0; python_version >= "3.7" and python_version < "4.0"
-black==20.8b1; python_version >= "3.6"
-certifi==2020.12.5; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
-chardet==4.0.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
-click==7.1.2; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.6"
+black==21.9b0; python_full_version >= "3.6.2"
+certifi==2021.5.30; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6"
+charset-normalizer==2.0.6; python_full_version >= "3.6.0" and python_version >= "3.6"
+click==8.0.1; python_version >= "3.6" and python_full_version >= "3.6.2"
 colorama==0.4.4; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0")
-csvkit==1.0.5
+commonmark==0.9.1; python_full_version >= "3.6.2" and python_full_version < "4.0.0" and (python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0")
+csvkit==1.0.6
 dbfread==2.0.7
-decorator==4.4.2; python_version >= "3.7" and python_full_version < "3.0.0" and python_version < "4.0" or python_version >= "3.7" and python_version < "4.0" and python_full_version >= "3.2.0"
-et-xmlfile==1.0.1; python_version >= "3.6"
-flake8==3.9.0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0")
-greenlet==1.0.0; python_version >= "3" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3"
-idna==2.10; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
-importlib-metadata==3.7.3; python_version < "3.8" and python_version >= "3.6" and (python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.8" or python_full_version >= "3.5.0" and python_version < "3.8" and python_version >= "3.6") and (python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.8" or python_full_version >= "3.4.0" and python_version >= "3.6" and python_version < "3.8") and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6") and (python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.8" or python_full_version >= "3.6.0" and python_version < "3.8" and python_version >= "3.6")
+decorator==5.1.0; python_version >= "3.7" and python_version < "4.0"
+et-xmlfile==1.1.0; python_version >= "3.6"
+flake8==3.9.2; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0")
+ftfy==5.9; python_version >= "3.5"
+future==0.18.2; python_version >= "2.6" and python_full_version < "3.0.0" or python_full_version >= "3.3.0"
+greenlet==1.1.2; python_version >= "3" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3"
+idna==3.2; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6"
+importlib-metadata==4.8.1; python_full_version >= "3.6.2" and python_version < "3.8" and python_version >= "3.6" and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6") and (python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.8" or python_full_version >= "3.6.0" and python_version < "3.8" and python_version >= "3.6")
 iniconfig==1.1.1; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
-ipython-genutils==0.2.0; python_version >= "3.7" and python_version < "4.0"
-ipython==7.21.0; python_version >= "3.7" and python_version < "4.0"
+ipython==7.28.0; python_version >= "3.7" and python_version < "4.0"
 isodate==0.6.0
-isort==5.7.0; python_version >= "3.6" and python_version < "4.0"
+isort==5.9.3; python_full_version >= "3.6.1" and python_version < "4.0"
+itsdangerous==2.0.1; python_version >= "3.6"
 jedi==0.18.0; python_version >= "3.7" and python_version < "4.0"
 langid==1.1.6
 leather==0.3.3
+matplotlib-inline==0.1.3; python_version >= "3.7" and python_version < "4.0"
 mccabe==0.6.1; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
-mypy-extensions==0.4.3; python_version >= "3.6"
-numpy==1.20.1; python_version >= "3.7" and python_full_version >= "3.7.1"
-openpyxl==3.0.7; python_version >= "3.6"
-packaging==20.9; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
-pandas==1.2.3; python_full_version >= "3.7.1"
-parsedatetime==2.6
-parso==0.8.1; python_version >= "3.7" and python_version < "4.0"
-pathspec==0.8.1; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.6"
+mypy-extensions==0.4.3; python_full_version >= "3.6.2"
+numpy==1.21.1; python_version >= "3.7" and python_full_version >= "3.7.1"
+olefile==0.46; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0"
+openpyxl==3.0.9; python_version >= "3.6"
+packaging==21.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
+pandas==1.3.3; python_full_version >= "3.7.1"
+parsedatetime==2.4
+parso==0.8.2; python_version >= "3.7" and python_version < "4.0"
+pathspec==0.9.0; python_full_version >= "3.6.2"
 pexpect==4.8.0; python_version >= "3.7" and python_version < "4.0" and sys_platform != "win32"
 pickleshare==0.7.5; python_version >= "3.7" and python_version < "4.0"
-pluggy==0.13.1; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
-prompt-toolkit==3.0.17; python_version >= "3.7" and python_version < "4.0" and python_full_version >= "3.6.1"
+platformdirs==2.4.0; python_version >= "3.6" and python_full_version >= "3.6.2"
+pluggy==1.0.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
+pprintpp==0.4.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0"
+prompt-toolkit==3.0.20; python_version >= "3.7" and python_version < "4.0" and python_full_version >= "3.6.2"
 ptyprocess==0.7.0; python_version >= "3.7" and python_version < "4.0" and sys_platform != "win32"
 py==1.10.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
 pycodestyle==2.7.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
 pycountry==19.8.18
-pyflakes==2.3.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
-pygments==2.8.1; python_version >= "3.7" and python_version < "4.0"
-pyicu==2.6
-pyparsing==2.4.7; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
-pytest-clarity==0.3.0a0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")
-pytest==6.2.2; python_version >= "3.6"
-python-dateutil==2.8.1; python_full_version >= "3.7.1"
-python-slugify==4.0.1; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
-python-stdnum==1.16
+pyflakes==2.3.1; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
+pygments==2.10.0; python_version >= "3.7" and python_version < "4.0" and python_full_version >= "3.6.2" and python_full_version < "4.0.0" and (python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0")
+pyparsing==2.4.7; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.3.0" and python_version >= "3.6"
+pytest-clarity==1.0.1; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")
+pytest==6.2.5; python_version >= "3.6"
+python-dateutil==2.8.2; python_full_version >= "3.7.1"
+python-slugify==5.0.2; python_version >= "3.6"
+python-stdnum==1.17
 pytimeparse==1.1.8
-pytz==2021.1; python_full_version >= "3.7.1"
-regex==2020.11.13; python_version >= "3.6"
-requests-cache==0.5.2
-requests==2.25.1; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0")
-six==1.15.0; python_full_version >= "3.7.1"
+pytz==2021.3; python_full_version >= "3.7.1"
+regex==2021.9.30; python_full_version >= "3.6.2"
+requests-cache==0.6.4; python_version >= "3.6"
+requests==2.26.0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.6.0")
+rich==10.12.0; python_full_version >= "3.6.2" and python_full_version < "4.0.0" and (python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0")
+six==1.16.0; python_full_version >= "3.7.1" and python_version >= "3.6" and (python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.3.0")
 spdx-license-list==0.5.2
-sqlalchemy==1.4.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.6.0"
-termcolor==1.1.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0"
-text-unidecode==1.3; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
+sqlalchemy==1.4.22; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.6.0")
+text-unidecode==1.3; python_version >= "3.6"
 toml==0.10.2; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
-traitlets==5.0.5; python_version >= "3.7" and python_version < "4.0"
-typed-ast==1.4.2; python_version >= "3.6"
-typing-extensions==3.7.4.3; python_version < "3.8" and python_version >= "3.6"
-urllib3==1.26.4; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version < "4"
-wcwidth==0.2.5; python_version >= "3.7" and python_version < "4.0" and python_full_version >= "3.6.1"
+tomli==1.2.1; python_version >= "3.6" and python_full_version >= "3.6.2"
+traitlets==5.1.0; python_version >= "3.7" and python_version < "4.0"
+typed-ast==1.4.3; python_version < "3.8" and python_full_version >= "3.6.2"
+typing-extensions==3.10.0.2
+url-normalize==1.4.3; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6"
+urllib3==1.26.7; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version < "4" and python_version >= "3.6"
+wcwidth==0.2.5; python_version >= "3.7" and python_version < "4.0" and python_full_version >= "3.6.2"
 xlrd==1.2.0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")
-zipp==3.4.1; python_version < "3.8" and python_version >= "3.6"
+zipp==3.6.0; python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.8" or python_full_version >= "3.6.0" and python_version < "3.8" and python_version >= "3.6"

requirements.txt

@@ -1,17 +1,26 @@
-certifi==2020.12.5; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
-chardet==4.0.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
+certifi==2021.5.30; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6"
+charset-normalizer==2.0.6; python_full_version >= "3.6.0" and python_version >= "3.6"
 colorama==0.4.4; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0")
-idna==2.10; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
+ftfy==5.9; python_version >= "3.5"
+greenlet==1.1.2; python_version >= "3" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3"
+idna==3.2; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6"
+importlib-metadata==4.8.1; python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.8" or python_full_version >= "3.6.0" and python_version < "3.8" and python_version >= "3.6"
+itsdangerous==2.0.1; python_version >= "3.6"
 langid==1.1.6
-numpy==1.20.1; python_version >= "3.7" and python_full_version >= "3.7.1"
-pandas==1.2.3; python_full_version >= "3.7.1"
+numpy==1.21.1; python_version >= "3.7" and python_full_version >= "3.7.1"
+pandas==1.3.3; python_full_version >= "3.7.1"
 pycountry==19.8.18
-python-dateutil==2.8.1; python_full_version >= "3.7.1"
-python-stdnum==1.16
-pytz==2021.1; python_full_version >= "3.7.1"
-requests-cache==0.5.2
-requests==2.25.1; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0")
-six==1.15.0; python_full_version >= "3.7.1"
+python-dateutil==2.8.2; python_full_version >= "3.7.1"
+python-stdnum==1.17
+pytz==2021.3; python_full_version >= "3.7.1"
+requests-cache==0.6.4; python_version >= "3.6"
+requests==2.26.0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.6.0")
+six==1.16.0; python_full_version >= "3.7.1" and python_version >= "3.6"
 spdx-license-list==0.5.2
-urllib3==1.26.4; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version < "4"
+sqlalchemy==1.4.22; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.6.0")
+typing-extensions==3.10.0.2; python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.8" or python_full_version >= "3.6.0" and python_version < "3.8" and python_version >= "3.6"
+url-normalize==1.4.3; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6"
+urllib3==1.26.7; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version < "4" and python_version >= "3.6"
+wcwidth==0.2.5; python_version >= "3.5"
 xlrd==1.2.0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")
+zipp==3.6.0; python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.8" or python_full_version >= "3.6.0" and python_version < "3.8" and python_version >= "3.6"

setup.py

@@ -14,7 +14,7 @@ install_requires = [
 
 setuptools.setup(
     name="csv-metadata-quality",
-    version="0.4.7",
+    version="0.4.8-dev",
     author="Alan Orth",
     author_email="aorth@mjanja.ch",
     description="A simple, but opinionated CSV quality checking and fixing pipeline for CSVs in the DSpace ecosystem.",

tests/test_check.py

@@ -1,3 +1,5 @@
+# SPDX-License-Identifier: GPL-3.0-only
+
 import pandas as pd
 from colorama import Fore
 
@@ -339,3 +341,70 @@ def test_check_duplicate_item(capsys):
         captured.out
         == f"{Fore.YELLOW}Possible duplicate (dc.title): {Fore.RESET}{item_title}\n"
     )
+
+
+def test_check_no_mojibake():
+    """Test string with no mojibake."""
+
+    field = "CIAT Publicaçao"
+    field_name = "dcterms.isPartOf"
+
+    result = check.mojibake(field, field_name)
+
+    assert result == None
+
+
+def test_check_mojibake(capsys):
+    """Test string with mojibake."""
+
+    field = "CIAT Publicaçao"
+    field_name = "dcterms.isPartOf"
+
+    result = check.mojibake(field, field_name)
+
+    captured = capsys.readouterr()
+    assert (
+        captured.out
+        == f"{Fore.YELLOW}Possible encoding issue ({field_name}): {Fore.RESET}{field}\n"
+    )
+
+
+def test_check_doi_field():
+    """Test an item with a DOI field."""
+
+    doi = "https://doi.org/10.1186/1743-422X-9-218"
+    citation = "Orth, A. 2021. Testing all the things. doi: 10.1186/1743-422X-9-218"
+
+    # Emulate a column in a transposed dataframe (which is just a series), with
+    # the citation and an empty DOI field.
+    d = {
+        "cg.identifier.doi": doi,
+        "dcterms.bibliographicCitation": citation
+    }
+    series = pd.Series(data=d)
+
+    result = check.citation_doi(series)
+
+    assert result == None
+
+
+def test_check_doi_only_in_citation(capsys):
+    """Test an item with a DOI in its citation, but no DOI field."""
+
+    citation = "Orth, A. 2021. Testing all the things. doi: 10.1186/1743-422X-9-218"
+
+    # Emulate a column in a transposed dataframe (which is just a series), with
+    # an empty DOI field and a citation containing a DOI.
+    d = {
+        "cg.identifier.doi": None,
+        "dcterms.bibliographicCitation": citation
+    }
+    series = pd.Series(data=d)
+
+    check.citation_doi(series)
+
+    captured = capsys.readouterr()
+    assert (
+        captured.out
+        == f"{Fore.YELLOW}DOI in citation, but missing a DOI field: {Fore.RESET}{citation}\n"
+    )

tests/test_fix.py

@@ -1,3 +1,5 @@
+# SPDX-License-Identifier: GPL-3.0-only
+
 import csv_metadata_quality.fix as fix
 
 
@@ -108,3 +110,12 @@ def test_fix_decomposed_unicode():
     field_name = "dc.contributor.author"
 
     assert fix.normalize_unicode(value, field_name) == "Ouédraogo, Mathieu"
+
+
+def test_fix_mojibake():
+    """Test string with no mojibake."""
+
+    field = "CIAT Publicaçao"
+    field_name = "dcterms.isPartOf"
+
+    assert fix.mojibake(field, field_name) == "CIAT Publicaçao"