Version 0.5.0

Generated with poetry export:

    $ poetry export --without-hashes -f requirements.txt > requirements.txt
    $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt

I am trying `--without-hashes` to work around an error on pip install
when running in CI:

    ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==.

.drone.yml

@@ -1,3 +1,20 @@
+---
+kind: pipeline
+type: docker
+name: python310
+
+steps:
+- name: test
+  image: python:3.10-slim
+  commands:
+  - id
+  - python -V
+  - apt update && apt install -y gcc g++ libicu-dev pkg-config
+  - pip install -r requirements-dev.txt
+  - pytest
+  - python setup.py install
+  - csv-metadata-quality -i data/test.csv -o /tmp/test.csv -e -u --agrovoc-fields dcterms.subject,cg.coverage.country
+
 ---
 kind: pipeline
 type: docker
@@ -13,7 +30,7 @@ steps:
   - pip install -r requirements-dev.txt
   - pytest
   - python setup.py install
-  - csv-metadata-quality -i data/test.csv -o /tmp/test.csv -e -u --agrovoc-fields dc.subject,cg.coverage.country
+  - csv-metadata-quality -i data/test.csv -o /tmp/test.csv -e -u --agrovoc-fields dcterms.subject,cg.coverage.country
 
 ---
 kind: pipeline
@@ -30,7 +47,7 @@ steps:
   - pip install -r requirements-dev.txt
   - pytest
   - python setup.py install
-  - csv-metadata-quality -i data/test.csv -o /tmp/test.csv -e -u --agrovoc-fields dc.subject,cg.coverage.country
+  - csv-metadata-quality -i data/test.csv -o /tmp/test.csv -e -u --agrovoc-fields dcterms.subject,cg.coverage.country
 
 ---
 kind: pipeline
@@ -47,6 +64,6 @@ steps:
   - pip install -r requirements-dev.txt
   - pytest
   - python setup.py install
-  - csv-metadata-quality -i data/test.csv -o /tmp/test.csv -e -u --agrovoc-fields dc.subject,cg.coverage.country
+  - csv-metadata-quality -i data/test.csv -o /tmp/test.csv -e -u --agrovoc-fields dcterms.subject,cg.coverage.country
 
 # vim: ts=2 sw=2 et

.github/workflows/python-app.yml

@@ -16,10 +16,10 @@ jobs:
 
     steps:
     - uses: actions/checkout@v2
-    - name: Set up Python 3.8
+    - name: Set up Python 3.9
       uses: actions/setup-python@v2
       with:
-        python-version: 3.8
+        python-version: 3.9
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
@@ -38,4 +38,4 @@ jobs:
     - name: Test CLI
       run: |
         python setup.py install
-        csv-metadata-quality -i data/test.csv -o /tmp/test.csv -e -u --agrovoc-fields dc.subject,cg.coverage.country
+        csv-metadata-quality -i data/test.csv -o /tmp/test.csv -e -u --agrovoc-fields dcterms.subject,cg.coverage.country

CHANGELOG.md

@@ -4,6 +4,44 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.5.0] - 2021-12-08
+### Added
+- Ability to check for, and fix, "mojibake" characters using [ftfy](https://github.com/LuminosoInsight/python-ftfy)
+- Ability to check if the item's title exists in the citation
+- Ability to check if an item has countries, but no matching regions (only
+suggests missing regions if there is a region field in the CSV)
+
+### Updated
+- Python dependencies
+
+### Fixed
+- Regular expression to match all citation fields (dc.identifier.citation as
+well as dcterms.bibliographicCitation) in `experimental.correct_language()`
+- Regular expression to match dc.title and dcterms.title, but
+ignore dc.title.alternative `check.duplicate_items()`
+- Missing field name in `fix.newlines()` output
+
+## [0.4.7] - 2021-03-17
+### Changed
+- Fixing invalid multi-value separators like `|` and `|||` is no longer class-
+ified as "unsafe" as I have yet to see a case where this was intentional
+- Not user visible, but now checks only print a warning to the screen instead
+of returning a value and re-writing the DataFrame, which should be faster and
+use less memory
+
+### Added
+- Configurable directory for AGROVOC requests cache (to allow running the web
+version from Google App Engine where we can only write to /tmp)
+- Ability to check for duplicate items in the data set (uses a combination of
+the title, type, and date issued to determine uniqueness)
+
+### Removed
+- Checks for invalid and unnecessary multi-value separators because now I fix
+them whenever I see them, so there is no need to have checks for them
+
+### Updated
+- Run `poetry update` to update project dependencies
+
 ## [0.4.6] - 2021-03-11
 ### Added
 - Validation of dcterms.license field against SPDX license identifiers 

CITATION.cff

@@ -0,0 +1,19 @@
+cff-version: "1.1.0"
+abstract: "A simple but opinionated metadata quality checker and fixer designed to work with CSVs in the DSpace ecosystem."
+authors: 
+  -
+    affiliation: "International Livestock Research Institute"
+    family-names: Orth
+    given-names: "Alan S."
+    orcid: "https://orcid.org/0000-0002-1735-7458"
+date-released: 2019-07-26
+doi: "10568/110997"
+keywords: 
+  - dspace
+  - "dublin-core"
+  - csv
+  - metadata
+license: "GPL-3.0-only"
+message: "If you use this software, please cite it using these metadata."
+repository-code: "https://github.com/ilri/csv-metadata-quality"
+title: "DSpace CSV Metadata Quality Checker"

README.md

@@ -1,7 +1,14 @@
-# DSpace CSV Metadata Quality Checker ![GitHub Actions](https://github.com/ilri/csv-metadata-quality/workflows/Build%20and%20Test/badge.svg) [![Build Status](https://ci.mjanja.ch/api/badges/alanorth/csv-metadata-quality/status.svg)](https://ci.mjanja.ch/alanorth/csv-metadata-quality)
+<h1 align="center">DSpace CSV Metadata Quality Checker</h1>
+
+<p align="center">
+  <a href="https://ci.mjanja.ch/alanorth/csv-metadata-quality"><img alt="Build Status" src="https://ci.mjanja.ch/api/badges/alanorth/csv-metadata-quality/status.svg"></a>
+  <a href="https://github.com/ilri/csv-metadata-quality/actions"><img alt="Build and Test" src="https://github.com/ilri/csv-metadata-quality/workflows/Build%20and%20Test/badge.svg"></a>
+  <a href="https://github.com/psf/black"><img alt="Code style: black" src="https://img.shields.io/badge/code%20style-black-000000.svg"></a>
+</p>
+
 A simple, but opinionated metadata quality checker and fixer designed to work with CSVs in the DSpace ecosystem (though it could theoretically work on any CSV that uses Dublin Core fields as columns). The implementation is essentially a pipeline of checks and fixes that begins with splitting multi-value fields on the standard DSpace "||" separator, trimming leading/trailing whitespace, and then proceeding to more specialized cases like ISSNs, ISBNs, languages, unnecessary Unicode, AGROVOC terms, etc.
 
-Requires Python 3.7 or greater (3.8 recommended). CSV and Excel support comes from the [Pandas](https://pandas.pydata.org/) library, though your mileage may vary with Excel because this is much less tested.
+Requires Python 3.7.1 or greater (3.8+ recommended). CSV and Excel support comes from the [Pandas](https://pandas.pydata.org/) library, though your mileage may vary with Excel because this is much less tested.
 
 If you use the DSpace CSV metadata quality checker please cite:
 
@@ -13,13 +20,16 @@ If you use the DSpace CSV metadata quality checker please cite:
 - Validate languages against ISO 639-1 (alpha2) and ISO 639-3 (alpha3)
 - Experimental validation of titles and abstracts against item's Dublin Core language field
 - Validate subjects against the AGROVOC REST API (see the `--agrovoc-fields` option)
+- Validation of licenses against the list of [SPDX license identifiers](https://spdx.org/licenses)
 - Fix leading, trailing, and excessive (ie, more than one) whitespace
-- Fix invalid and unnecessary multi-value separators (`|`) using `--unsafe-fixes`
+- Fix invalid and unnecessary multi-value separators (`|`)
 - Fix problematic newlines (line feeds) using `--unsafe-fixes`
+- Perform [Unicode normalization](https://withblue.ink/2019/03/11/why-you-need-to-normalize-unicode-strings.html) on strings using `--unsafe-fixes`
 - Remove unnecessary Unicode like [non-breaking spaces](https://en.wikipedia.org/wiki/Non-breaking_space), [replacement characters](https://en.wikipedia.org/wiki/Specials_(Unicode_block)#Replacement_character), etc
 - Check for "suspicious" characters that indicate encoding or copy/paste issues, for example "foreˆt" should be "forêt"
+- Check for "mojibake" characters (and attempt to fix with `--unsafe-fixes`)
 - Remove duplicate metadata values
-- Perform [Unicode normalization](https://withblue.ink/2019/03/11/why-you-need-to-normalize-unicode-strings.html) on strings using `--unsafe-fixes`
+- Check for duplicate items, using the title, type, and date issued as an indicator
 
 ## Installation
 The easiest way to install CSV Metadata Quality is with [poetry](https://python-poetry.org):
@@ -54,14 +64,14 @@ To validate and clean a CSV file you must specify input and output files using t
 $ csv-metadata-quality -i data/test.csv -o /tmp/test.csv
 ```
 
-## Unsafe Fixes
-You can enable several "unsafe" fixes with the `--unsafe-fixes` option. Currently this will attempt to fix invalid multi-value separators and remove newlines.
-
-### Invalid Multi-Value Separators
-This is considered "unsafe" because it is *theoretically* possible for a single `|` character to be used legitimately in a metadata value, though in my experience it is always a typo. For example, if a user mistakenly writes `Kenya|Tanzania` when attempting to indicate two countries, the result will be one metadata value with the literal text `Kenya|Tanzania`. The `--unsafe-fixes` option will correct the invalid multi-value separator so that there are two metadata values, ie `Kenya||Tanzania`.
+## Invalid Multi-Value Separators
+While it is *theoretically* possible for a single `|` character to be used legitimately in a metadata value, in my experience it is always a typo. For example, if a user mistakenly writes `Kenya|Tanzania` when attempting to indicate two countries, the result will be one metadata value with the literal text `Kenya|Tanzania`. This utility will correct the invalid multi-value separator so that there are two metadata values, ie `Kenya||Tanzania`.
 
 This will also remove unnecessary trailing multi-value separators, for example `Kenya||Tanzania||`.
 
+## Unsafe Fixes
+You can enable several "unsafe" fixes with the `--unsafe-fixes` option. Currently this will remove newlines, perform Unicode normalization, and attempt to fix "mojibake" characters.
+
 ### Newlines
 This is considered "unsafe" because some systems give special importance to vertical space and render it properly. DSpace does not support rendering newlines in its XMLUI and has, at times, suffered from parsing errors that cause the import process to fail if an input file had newlines. The `--unsafe-fixes` option strips Unix line feeds (U+000A).
 
@@ -73,6 +83,14 @@ This is considered "unsafe" because some systems give special importance to vert
 
 Read more about [Unicode normalization](https://withblue.ink/2019/03/11/why-you-need-to-normalize-unicode-strings.html).
 
+### Encoding Issues aka "Mojibake"
+[Mojibake](https://en.wikipedia.org/wiki/Mojibake) is a phenomenon that occurs when text is decoded using an unintended character encoding. This usually presents itself in the form of strange, garbled characters in the text. Enabling "unsafe" fixes will attempt to correct these, for example:
+
+- CIAT Publicaçao → CIAT Publicaçao
+- CIAT Publicación → CIAT Publicación
+
+Pay special attention to the output of the script as well as the resulting file to make sure no new issues have been introduced. The ideal way to solve these issues is to avoid it in the first place. See [this guide about opening CSVs in UTF-8 format in Excel](https://www.itg.ias.edu/content/how-import-csv-file-uses-utf-8-character-encoding-0).
+
 ## AGROVOC Validation
 You can enable validation of metadata values in certain fields against the AGROVOC REST API with the `--agrovoc-fields` option. For example, in addition to agricultural subjects, many countries and regions are also present AGROVOC. Enable this validation by specifying a comma-separated list of fields:
 
@@ -111,10 +129,10 @@ This currently uses the [Python langid](https://github.com/saffsd/langid.py) lib
 - Add configurable field validation, like specify a field name and a validation file?
   - Perhaps like --validate=field.name,filename
 - Add some row-based item sanity checks and fixes:
-    - Warn if item is Open Access, but missing a filename or URL
-    - Warn if item is Open Access, but missing a license
-    - Warn if item has an ISSN but no journal title
-    - Update journal titles from ISSN
+  - Warn if item is Open Access, but missing a filename or URL
+  - Warn if item is Open Access, but missing a license
+  - Warn if item has an ISSN but no journal title
+  - Update journal titles from ISSN
 
 ## License
 This work is licensed under the [GPLv3](https://www.gnu.org/licenses/gpl-3.0.en.html).

csv_metadata_quality/__main__.py

@@ -1,3 +1,5 @@
+# SPDX-License-Identifier: GPL-3.0-only
+
 from sys import argv
 
 from csv_metadata_quality import app

csv_metadata_quality/app.py

@@ -1,3 +1,5 @@
+# SPDX-License-Identifier: GPL-3.0-only
+
 import argparse
 import re
 import signal
@@ -17,7 +19,7 @@ def parse_args(argv):
     parser.add_argument(
         "--agrovoc-fields",
         "-a",
-        help="Comma-separated list of fields to validate against AGROVOC, for example: dc.subject,cg.coverage.country",
+        help="Comma-separated list of fields to validate against AGROVOC, for example: dcterms.subject,cg.coverage.country",
     )
     parser.add_argument(
         "--experimental-checks",
@@ -48,7 +50,7 @@ def parse_args(argv):
     parser.add_argument(
         "--exclude-fields",
         "-x",
-        help="Comma-separated list of fields to skip, for example: dc.contributor.author,dc.identifier.citation",
+        help="Comma-separated list of fields to skip, for example: dc.contributor.author,dcterms.bibliographicCitation",
     )
     args = parser.parse_args()
 
@@ -87,7 +89,7 @@ def run(argv):
 
         # Fix: newlines
         if args.unsafe_fixes:
-            df[column] = df[column].apply(fix.newlines)
+            df[column] = df[column].apply(fix.newlines, field_name=column)
 
         # Fix: missing space after comma. Only run on author and citation
         # fields for now, as this problem is mostly an issue in names.
@@ -104,17 +106,19 @@ def run(argv):
         # Fix: unnecessary Unicode
         df[column] = df[column].apply(fix.unnecessary_unicode)
 
-        # Check: invalid and unnecessary multi-value separators
-        df[column] = df[column].apply(check.separators, field_name=column)
-
         # Check: suspicious characters
-        df[column] = df[column].apply(check.suspicious_characters, field_name=column)
+        df[column].apply(check.suspicious_characters, field_name=column)
+
+        # Fix: mojibake. If unsafe fixes are not enabled then we only check.
+        if args.unsafe_fixes:
+            df[column] = df[column].apply(fix.mojibake, field_name=column)
+        else:
+            df[column].apply(check.mojibake, field_name=column)
 
         # Fix: invalid and unnecessary multi-value separators
-        if args.unsafe_fixes:
-            df[column] = df[column].apply(fix.separators, field_name=column)
-            # Run whitespace fix again after fixing invalid separators
-            df[column] = df[column].apply(fix.whitespace, field_name=column)
+        df[column] = df[column].apply(fix.separators, field_name=column)
+        # Run whitespace fix again after fixing invalid separators
+        df[column] = df[column].apply(fix.whitespace, field_name=column)
 
         # Fix: duplicate metadata values
         df[column] = df[column].apply(fix.duplicates, field_name=column)
@@ -124,36 +128,51 @@ def run(argv):
             # Identify fields the user wants to validate against AGROVOC
             for field in args.agrovoc_fields.split(","):
                 if column == field:
-                    df[column] = df[column].apply(check.agrovoc, field_name=column)
+                    df[column].apply(check.agrovoc, field_name=column)
 
         # Check: invalid language
         match = re.match(r"^.*?language.*$", column)
         if match is not None:
-            df[column] = df[column].apply(check.language)
+            df[column].apply(check.language)
 
         # Check: invalid ISSN
         match = re.match(r"^.*?issn.*$", column)
         if match is not None:
-            df[column] = df[column].apply(check.issn)
+            df[column].apply(check.issn)
 
         # Check: invalid ISBN
         match = re.match(r"^.*?isbn.*$", column)
         if match is not None:
-            df[column] = df[column].apply(check.isbn)
+            df[column].apply(check.isbn)
 
         # Check: invalid date
         match = re.match(r"^.*?(date|dcterms\.issued).*$", column)
         if match is not None:
-            df[column] = df[column].apply(check.date, field_name=column)
+            df[column].apply(check.date, field_name=column)
 
         # Check: filename extension
         if column == "filename":
-            df[column] = df[column].apply(check.filename_extension)
+            df[column].apply(check.filename_extension)
 
         # Check: SPDX license identifier
         match = re.match(r"dcterms\.license.*$", column)
         if match is not None:
-            df[column] = df[column].apply(check.spdx_license_identifier)
+            df[column].apply(check.spdx_license_identifier)
+
+    ### End individual column checks ###
+
+    # Check: duplicate items
+    # We extract just the title, type, and date issued columns to analyze
+    try:
+        duplicates_df = df.filter(
+            regex=r"dcterms\.title|dc\.title|dcterms\.type|dc\.type|dcterms\.issued|dc\.date\.issued"
+        )
+        check.duplicate_items(duplicates_df)
+
+        # Delete the temporary duplicates DataFrame
+        del duplicates_df
+    except IndexError:
+        pass
 
     ##
     # Perform some checks on rows so we can consider items as a whole rather
@@ -166,11 +185,22 @@ def run(argv):
     # column. For now it will have to do.
     ##
 
-    if args.experimental_checks:
-        # Transpose the DataFrame so we can consider each row as a column
-        df_transposed = df.T
+    # Transpose the DataFrame so we can consider each row as a column
+    df_transposed = df.T
 
-        for column in df_transposed.columns:
+    # Remember, here a "column" is an item (previously row). Perhaps I
+    # should rename column in this for loop...
+    for column in df_transposed.columns:
+        # Check: citation DOI
+        check.citation_doi(df_transposed[column])
+
+        # Check: title in citation
+        check.title_in_citation(df_transposed[column])
+
+        # Check: countries match regions
+        check.countries_match_regions(df_transposed[column])
+
+        if args.experimental_checks:
             experimental.correct_language(df_transposed[column])
 
     # Write

csv_metadata_quality/check.py

@@ -1,6 +1,10 @@
+# SPDX-License-Identifier: GPL-3.0-only
+
+import os
 import re
 from datetime import datetime, timedelta
 
+import country_converter as coco
 import pandas as pd
 import requests
 import requests_cache
@@ -10,6 +14,8 @@ from pycountry import languages
 from stdnum import isbn as stdnum_isbn
 from stdnum import issn as stdnum_issn
 
+from csv_metadata_quality.util import is_mojibake
+
 
 def issn(field):
     """Check if an ISSN is valid.
@@ -31,7 +37,7 @@ def issn(field):
         if not stdnum_issn.is_valid(value):
             print(f"{Fore.RED}Invalid ISSN: {Fore.RESET}{value}")
 
-    return field
+    return
 
 
 def isbn(field):
@@ -54,43 +60,7 @@ def isbn(field):
         if not stdnum_isbn.is_valid(value):
             print(f"{Fore.RED}Invalid ISBN: {Fore.RESET}{value}")
 
-    return field
-
-
-def separators(field, field_name):
-    """Check for invalid and unnecessary multi-value separators, for example:
-
-        value|value
-        value|||value
-        value||value||
-
-    Prints the field with the invalid multi-value separator.
-    """
-
-    # Skip fields with missing values
-    if pd.isna(field):
-        return
-
-    # Try to split multi-value field on "||" separator
-    for value in field.split("||"):
-        # Check if the current value is blank
-        if value == "":
-            print(
-                f"{Fore.RED}Unnecessary multi-value separator ({field_name}): {Fore.RESET}{field}"
-            )
-
-            continue
-
-        # After splitting, see if there are any remaining "|" characters
-        match = re.findall(r"^.*?\|.*$", value)
-
-        # Check if there was a match
-        if match:
-            print(
-                f"{Fore.RED}Invalid multi-value separator ({field_name}): {Fore.RESET}{field}"
-            )
-
-    return field
+    return
 
 
 def date(field, field_name):
@@ -118,13 +88,13 @@ def date(field, field_name):
             f"{Fore.RED}Multiple dates not allowed ({field_name}): {Fore.RESET}{field}"
         )
 
-        return field
+        return
 
     try:
         # Check if date is valid YYYY format
         datetime.strptime(field, "%Y")
 
-        return field
+        return
     except ValueError:
         pass
 
@@ -132,7 +102,7 @@ def date(field, field_name):
         # Check if date is valid YYYY-MM format
         datetime.strptime(field, "%Y-%m")
 
-        return field
+        return
     except ValueError:
         pass
 
@@ -140,7 +110,7 @@ def date(field, field_name):
         # Check if date is valid YYYY-MM-DD format
         datetime.strptime(field, "%Y-%m-%d")
 
-        return field
+        return
     except ValueError:
         pass
 
@@ -148,11 +118,11 @@ def date(field, field_name):
         # Check if date is valid YYYY-MM-DDTHH:MM:SSZ format
         datetime.strptime(field, "%Y-%m-%dT%H:%M:%SZ")
 
-        return field
+        return
     except ValueError:
         print(f"{Fore.RED}Invalid date ({field_name}): {Fore.RESET}{field}")
 
-        return field
+        return
 
 
 def suspicious_characters(field, field_name):
@@ -186,7 +156,7 @@ def suspicious_characters(field, field_name):
             suspicious_character_msg = f"{Fore.YELLOW}Suspicious character ({field_name}): {Fore.RESET}{field_subset}"
             print(f"{suspicious_character_msg:1.80}")
 
-    return field
+    return
 
 
 def language(field):
@@ -209,17 +179,13 @@ def language(field):
         if len(value) == 2:
             if not languages.get(alpha_2=value):
                 print(f"{Fore.RED}Invalid ISO 639-1 language: {Fore.RESET}{value}")
-
-                pass
         elif len(value) == 3:
             if not languages.get(alpha_3=value):
                 print(f"{Fore.RED}Invalid ISO 639-3 language: {Fore.RESET}{value}")
-
-                pass
         else:
             print(f"{Fore.RED}Invalid language: {Fore.RESET}{value}")
 
-    return field
+    return
 
 
 def agrovoc(field, field_name):
@@ -242,10 +208,16 @@ def agrovoc(field, field_name):
 
     # enable transparent request cache with thirty days expiry
     expire_after = timedelta(days=30)
-    requests_cache.install_cache("agrovoc-response-cache", expire_after=expire_after)
+    # Allow overriding the location of the requests cache, just in case we are
+    # running in an environment where we can't write to the current working di-
+    # rectory (for example from csv-metadata-quality-web).
+    REQUESTS_CACHE_DIR = os.environ.get("REQUESTS_CACHE_DIR", ".")
+    requests_cache.install_cache(
+        f"{REQUESTS_CACHE_DIR}/agrovoc-response-cache", expire_after=expire_after
+    )
 
     # prune old cache entries
-    requests_cache.core.remove_expired_responses()
+    requests_cache.remove_expired_responses()
 
     # Try to split multi-value field on "||" separator
     for value in field.split("||"):
@@ -261,7 +233,7 @@ def agrovoc(field, field_name):
             if len(data["results"]) == 0:
                 print(f"{Fore.RED}Invalid AGROVOC ({field_name}): {Fore.RESET}{value}")
 
-    return field
+    return
 
 
 def filename_extension(field):
@@ -312,7 +284,7 @@ def filename_extension(field):
         if filename_extension_match is False:
             print(f"{Fore.YELLOW}Filename with uncommon extension: {Fore.RESET}{value}")
 
-    return field
+    return
 
 
 def spdx_license_identifier(field):
@@ -330,6 +302,220 @@ def spdx_license_identifier(field):
         if value not in spdx_license_list.LICENSES:
             print(f"{Fore.YELLOW}Non-SPDX license identifier: {Fore.RESET}{value}")
 
-            pass
+    return
 
-    return field
+
+def duplicate_items(df):
+    """Attempt to identify duplicate items.
+
+    First we check the total number of titles and compare it with the number of
+    unique titles. If there are less unique titles than total titles we expand
+    the search by creating a key (of sorts) for each item that includes their
+    title, type, and date issued, and compare it with all the others. If there
+    are multiple occurrences of the same title, type, date string then it's a
+    very good indicator that the items are duplicates.
+    """
+
+    # Extract the names of the title, type, and date issued columns so we can
+    # reference them later. First we filter columns by likely patterns, then
+    # we extract the name from the first item of the resulting object, ie:
+    #
+    #   Index(['dcterms.title[en_US]'], dtype='object')
+    #
+    # But, we need to consider that dc.title.alternative might come before the
+    # main title in the CSV, so use a negative lookahead to eliminate that.
+    #
+    # See: https://regex101.com/r/elyXkW/1
+    title_column_name = df.filter(
+        regex=r"^(dc|dcterms)\.title(?!\.alternative).*$"
+    ).columns[0]
+    type_column_name = df.filter(regex=r"^(dcterms\.type|dc\.type).*$").columns[0]
+    date_column_name = df.filter(
+        regex=r"^(dcterms\.issued|dc\.date\.accessioned).*$"
+    ).columns[0]
+
+    items_count_total = df[title_column_name].count()
+    items_count_unique = df[title_column_name].nunique()
+
+    if items_count_unique < items_count_total:
+        # Create a list to hold our items while we check for duplicates
+        items = list()
+
+        for index, row in df.iterrows():
+            item_title_type_date = f"{row[title_column_name]}{row[type_column_name]}{row[date_column_name]}"
+
+            if item_title_type_date in items:
+                print(
+                    f"{Fore.YELLOW}Possible duplicate ({title_column_name}): {Fore.RESET}{row[title_column_name]}"
+                )
+            else:
+                items.append(item_title_type_date)
+
+
+def mojibake(field, field_name):
+    """Check for mojibake (text that was encoded in one encoding and decoded in
+    in another, perhaps multiple times). See util.py.
+
+    Prints the string if it contains suspected mojibake.
+    """
+
+    # Skip fields with missing values
+    if pd.isna(field):
+        return
+
+    if is_mojibake(field):
+        print(
+            f"{Fore.YELLOW}Possible encoding issue ({field_name}): {Fore.RESET}{field}"
+        )
+
+    return
+
+
+def citation_doi(row):
+    """Check for the scenario where an item has a DOI listed in its citation,
+    but does not have a cg.identifier.doi field.
+
+    Function prints a warning if the DOI field is missing, but there is a DOI
+    in the citation.
+    """
+    # Initialize some variables at global scope so that we can set them in the
+    # loop scope below and still be able to access them afterwards.
+    citation = ""
+
+    # Iterate over the labels of the current row's values to check if a DOI
+    # exists. If not, then we extract the citation to see if there is a DOI
+    # listed there.
+    for label in row.axes[0]:
+        # Skip fields with missing values
+        if pd.isna(row[label]):
+            continue
+
+        # If a DOI field exists we don't need to check the citation
+        match = re.match(r"^.*?doi.*$", label)
+        if match is not None:
+            return
+
+        # Get the name of the citation field
+        match = re.match(r"^.*?[cC]itation.*$", label)
+        if match is not None:
+            citation = row[label]
+
+    if citation != "":
+        # Check the citation for "doi: 10.1186/1743-422X-9-218"
+        doi_match1 = re.match(r"^.*?doi:\s.*$", citation)
+        # Check the citation for a DOI URL (doi.org, dx.doi.org, etc)
+        doi_match2 = re.match(r"^.*?doi\.org.*$", citation)
+        if doi_match1 is not None or doi_match2 is not None:
+            print(
+                f"{Fore.YELLOW}DOI in citation, but missing a DOI field: {Fore.RESET}{citation}"
+            )
+
+    return
+
+
+def title_in_citation(row):
+    """Check for the scenario where an item's title is missing from its cita-
+    tion. This could mean that it is missing entirely, or perhaps just exists
+    in a different format (whitespace, accents, etc).
+
+    Function prints a warning if the title does not appear in the citation.
+    """
+    # Initialize some variables at global scope so that we can set them in the
+    # loop scope below and still be able to access them afterwards.
+    title = ""
+    citation = ""
+
+    # Iterate over the labels of the current row's values to get the names of
+    # the title and citation columns. Then we check if the title is present in
+    # the citation.
+    for label in row.axes[0]:
+        # Skip fields with missing values
+        if pd.isna(row[label]):
+            continue
+
+        # Find the name of the title column
+        match = re.match(r"^(dc|dcterms)\.title.*$", label)
+        if match is not None:
+            title = row[label]
+
+        # Find the name of the citation column
+        match = re.match(r"^.*?[cC]itation.*$", label)
+        if match is not None:
+            citation = row[label]
+
+    if citation != "":
+        if title not in citation:
+            print(f"{Fore.YELLOW}Title is not present in citation: {Fore.RESET}{title}")
+
+    return
+
+
+def countries_match_regions(row):
+    """Check for the scenario where an item has country coverage metadata, but
+    does not have the corresponding region metadata. For example, an item that
+    has country coverage "Kenya" should also have region "Eastern Africa" acc-
+    ording to the UN M.49 classification scheme.
+
+    See: https://unstats.un.org/unsd/methodology/m49/
+
+    Function prints a warning if the appropriate region is not present.
+    """
+    # Initialize some variables at global scope so that we can set them in the
+    # loop scope below and still be able to access them afterwards.
+    country_column_name = ""
+    region_column_name = ""
+    title_column_name = ""
+
+    # Iterate over the labels of the current row's values to get the names of
+    # the title and citation columns. Then we check if the title is present in
+    # the citation.
+    for label in row.axes[0]:
+        # Find the name of the country column
+        match = re.match(r"^.*?country.*$", label)
+        if match is not None:
+            country_column_name = label
+
+        # Find the name of the region column
+        match = re.match(r"^.*?region.*$", label)
+        if match is not None:
+            region_column_name = label
+
+        # Find the name of the title column
+        match = re.match(r"^(dc|dcterms)\.title.*$", label)
+        if match is not None:
+            title_column_name = label
+
+    # Make sure we found the country and region columns
+    if country_column_name != "" and region_column_name != "":
+        # If we don't have any countries then we should return early before
+        # suggesting regions.
+        if row[country_column_name] is not None:
+            countries = row[country_column_name].split("||")
+        else:
+            return
+
+        if row[region_column_name] is not None:
+            regions = row[region_column_name].split("||")
+        else:
+            regions = list()
+
+        # An empty list for our regions so we can keep track for all countries
+        missing_regions = list()
+
+        for country in countries:
+            # Look up the UN M.49 regions for this country code. CoCo seems to
+            # only list the direct region, ie Western Africa, rather than all
+            # the parent regions ("Sub-Saharan Africa", "Africa", "World")
+            un_region = coco.convert(names=country, to="UNRegion")
+
+            if un_region not in regions:
+                if un_region not in missing_regions:
+                    missing_regions.append(un_region)
+
+        if len(missing_regions) > 0:
+            for missing_region in missing_regions:
+                print(
+                    f"{Fore.YELLOW}Missing region ({missing_region}): {Fore.RESET}{row[title_column_name]}"
+                )
+
+    return

csv_metadata_quality/experimental.py

@@ -1,5 +1,11 @@
+# SPDX-License-Identifier: GPL-3.0-only
+
+import re
+
+import langid
 import pandas as pd
 from colorama import Fore
+from pycountry import languages
 
 
 def correct_language(row):
@@ -11,11 +17,6 @@ def correct_language(row):
     language and returns the value in the language field if it does match.
     """
 
-    import re
-
-    import langid
-    from pycountry import languages
-
     # Initialize some variables at global scope so that we can set them in the
     # loop scope below and still be able to access them afterwards.
     language = ""
@@ -51,7 +52,7 @@ def correct_language(row):
             sample_strings.append(row[label])
 
         # Extract citation if it is present
-        match = re.match(r"^.*?citation.*$", label)
+        match = re.match(r"^.*?[cC]itation.*$", label)
         if match is not None:
             sample_strings.append(row[label])
 
@@ -94,4 +95,4 @@ def correct_language(row):
         )
 
     else:
-        return language
+        return

csv_metadata_quality/fix.py

@@ -1,10 +1,13 @@
+# SPDX-License-Identifier: GPL-3.0-only
+
 import re
 from unicodedata import normalize
 
 import pandas as pd
 from colorama import Fore
+from ftfy import fix_text
 
-from csv_metadata_quality.util import is_nfc
+from csv_metadata_quality.util import is_mojibake, is_nfc
 
 
 def whitespace(field, field_name):
@@ -177,7 +180,7 @@ def duplicates(field, field_name):
     return new_field
 
 
-def newlines(field):
+def newlines(field, field_name):
     """Fix newlines.
 
     Single metadata values should not span multiple lines because this is not
@@ -202,7 +205,7 @@ def newlines(field):
     match = re.findall(r"\n", field)
 
     if match:
-        print(f"{Fore.GREEN}Removing newline: {Fore.RESET}{field}")
+        print(f"{Fore.GREEN}Removing newline ({field_name}): {Fore.RESET}{field}")
         field = field.replace("\n", "")
 
     return field
@@ -253,3 +256,22 @@ def normalize_unicode(field, field_name):
         field = normalize("NFC", field)
 
     return field
+
+
+def mojibake(field, field_name):
+    """Attempts to fix mojibake (text that was encoded in one encoding and deco-
+    ded in another, perhaps multiple times). See util.py.
+
+    Return fixed string.
+    """
+
+    # Skip fields with missing values
+    if pd.isna(field):
+        return field
+
+    if is_mojibake(field):
+        print(f"{Fore.GREEN}Fixing encoding issue ({field_name}): {Fore.RESET}{field}")
+
+        return fix_text(field)
+    else:
+        return field

csv_metadata_quality/util.py

@@ -1,3 +1,8 @@
+# SPDX-License-Identifier: GPL-3.0-only
+
+from ftfy.badness import sequence_weirdness
+
+
 def is_nfc(field):
     """Utility function to check whether a string is using normalized Unicode.
     Python's built-in unicodedata library has the is_normalized() function, but
@@ -12,3 +17,35 @@ def is_nfc(field):
     from unicodedata import normalize
 
     return field == normalize("NFC", field)
+
+
+def is_mojibake(field):
+    """Determines whether a string contains mojibake.
+
+    We commonly deal with CSV files that were *encoded* in UTF-8, but decoded
+    as something else like CP-1252 (Windows Latin). This manifests in the form
+    of "mojibake", for example:
+
+        - CIAT Publicaçao
+        - CIAT Publicación
+
+    This uses the excellent "fixes text for you" (ftfy) library to determine
+    whether a string contains characters that have been encoded in one encoding
+    and decoded in another.
+
+    Inspired by this code snippet from Martijn Pieters on StackOverflow:
+    https://stackoverflow.com/questions/29071995/identify-garbage-unicode-string-using-python
+
+    Return boolean.
+    """
+    if not sequence_weirdness(field):
+        # Nothing weird, should be okay
+        return False
+    try:
+        field.encode("sloppy-windows-1252")
+    except UnicodeEncodeError:
+        # Not CP-1252 encodable, probably fine
+        return False
+    else:
+        # Encodable as CP-1252, Mojibake alert level high
+        return True

csv_metadata_quality/version.py

@@ -1 +1,3 @@
-VERSION = "0.4.6"
+# SPDX-License-Identifier: GPL-3.0-only
+
+VERSION = "0.5.0"

data/test.csv

@@ -1,32 +1,38 @@
-dc.title,dcterms.issued,dc.identifier.issn,dc.identifier.isbn,dcterms.language,dcterms.subject,cg.coverage.country,filename,dcterms.license
- Leading space,2019-07-29,,,,,,,
-Trailing space ,2019-07-29,,,,,,,
-Excessive  space,2019-07-29,,,,,,,
-Miscellaenous ||whitespace | issues ,2019-07-29,,,,,,,
-Duplicate||Duplicate,2019-07-29,,,,,,,
-Invalid ISSN,2019-07-29,2321-2302,,,,,,
-Invalid ISBN,2019-07-29,,978-0-306-40615-6,,,,,
-Multiple valid ISSNs,2019-07-29,0378-5955||0024-9319,,,,,,
-Multiple valid ISBNs,2019-07-29,,99921-58-10-7||978-0-306-40615-7,,,,,
-Invalid date,2019-07-260,,,,,,,
-Multiple dates,2019-07-26||2019-01-10,,,,,,,
-Invalid multi-value separator,2019-07-29,0378-5955|0024-9319,,,,,,
-Unnecessary Unicode​,2019-07-29,,,,,,,
-Suspicious character||foreˆt,2019-07-29,,,,,,,
-Invalid ISO 639-1 (alpha 2) language,2019-07-29,,,jp,,,,
-Invalid ISO 639-3 (alpha 3) language,2019-07-29,,,chi,,,,
-Invalid language,2019-07-29,,,Span,,,,
-Invalid AGROVOC subject,2019-07-29,,,,FOREST,,,
+dc.title,dcterms.issued,dc.identifier.issn,dc.identifier.isbn,dcterms.language,dcterms.subject,cg.coverage.country,filename,dcterms.license,dcterms.type,dcterms.bibliographicCitation,cg.identifier.doi,cg.coverage.region
+ Leading space,2019-07-29,,,,,,,,,,,
+Trailing space ,2019-07-29,,,,,,,,,,,
+Excessive  space,2019-07-29,,,,,,,,,,,
+Miscellaenous ||whitespace | issues ,2019-07-29,,,,,,,,,,,
+Duplicate||Duplicate,2019-07-29,,,,,,,,,,,
+Invalid ISSN,2019-07-29,2321-2302,,,,,,,,,,
+Invalid ISBN,2019-07-29,,978-0-306-40615-6,,,,,,,,,
+Multiple valid ISSNs,2019-07-29,0378-5955||0024-9319,,,,,,,,,,
+Multiple valid ISBNs,2019-07-29,,99921-58-10-7||978-0-306-40615-7,,,,,,,,,
+Invalid date,2019-07-260,,,,,,,,,,,
+Multiple dates,2019-07-26||2019-01-10,,,,,,,,,,,
+Invalid multi-value separator,2019-07-29,0378-5955|0024-9319,,,,,,,,,,
+Unnecessary Unicode​,2019-07-29,,,,,,,,,,,
+Suspicious character||foreˆt,2019-07-29,,,,,,,,,,,
+Invalid ISO 639-1 (alpha 2) language,2019-07-29,,,jp,,,,,,,,
+Invalid ISO 639-3 (alpha 3) language,2019-07-29,,,chi,,,,,,,,
+Invalid language,2019-07-29,,,Span,,,,,,,,
+Invalid AGROVOC subject,2019-07-29,,,,FOREST,,,,,,,
 Newline (LF),2019-07-30,,,,"TANZA
-NIA",,,
-Missing date,,,,,,,,
-Invalid country,2019-08-01,,,,,KENYAA,,
-Uncommon filename extension,2019-08-10,,,,,,file.pdf.lck,
-Unneccesary unicode (U+002D + U+00AD),2019-08-10,,978-­92-­9043-­823-­6,,,,,
-"Missing space,after comma",2019-08-27,,,,,,,
-Incorrect ISO 639-1 language,2019-09-26,,,es,,,,
-Incorrect ISO 639-3 language,2019-09-26,,,spa,,,,
-Composéd Unicode,2020-01-14,,,,,,,
-Decomposéd Unicode,2020-01-14,,,,,,,
-Unnecessary multi-value separator,2021-01-03,0378-5955||,,,,,,
-Invalid SPDX license identifier,2021-03-11,,,,,,,CC-BY
+NIA",,,,,,,
+Missing date,,,,,,,,,,,,
+Invalid country,2019-08-01,,,,,KENYAA,,,,,,
+Uncommon filename extension,2019-08-10,,,,,,file.pdf.lck,,,,,
+Unneccesary unicode (U+002D + U+00AD),2019-08-10,,978-­92-­9043-­823-­6,,,,,,,,,
+"Missing space,after comma",2019-08-27,,,,,,,,,,,
+Incorrect ISO 639-1 language,2019-09-26,,,es,,,,,,,,
+Incorrect ISO 639-3 language,2019-09-26,,,spa,,,,,,,,
+Composéd Unicode,2020-01-14,,,,,,,,,,,
+Decomposéd Unicode,2020-01-14,,,,,,,,,,,
+Unnecessary multi-value separator,2021-01-03,0378-5955||,,,,,,,,,,
+Invalid SPDX license identifier,2021-03-11,,,,,,,CC-BY,,,,
+Duplicate Title,2021-03-17,,,,,,,,Report,,,
+Duplicate Title,2021-03-17,,,,,,,,Report,,,
+Mojibake,2021-03-18,,,,Publicaçao CIAT,,,,Report,,,
+"DOI in citation, but missing cg.identifier.doi",2021-10-06,,,,,,,,,"Orth, A. 2021. DOI in citation, but missing cg.identifier.doi. doi: 10.1186/1743-422X-9-218",,
+Title missing from citation,2021-12-05,,,,,,,,,"Orth, A. 2021. Title missing f rom citation.",,
+Country missing region,2021-12-08,,,,,Kenya,,,,,,

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "csv-metadata-quality"
-version = "0.4.6"
+version = "0.5.0"
 description="A simple, but opinionated CSV quality checking and fixing pipeline for CSVs in the DSpace ecosystem."
 authors = ["Alan Orth <alan.orth@gmail.com>"]
 license="GPL-3.0-only"
@@ -11,23 +11,26 @@ homepage = "https://github.com/ilri/csv-metadata-quality"
 csv-metadata-quality = 'csv_metadata_quality.__main__:main'
 
 [tool.poetry.dependencies]
-python = "^3.8"
+python = "^3.7.1"
 pandas = "^1.0.4"
 python-stdnum = "^1.13"
 xlrd = "^1.2.0"
 requests = "^2.23.0"
-requests-cache = "^0.5.2"
+requests-cache = "~0.6.4"
 pycountry = "^19.8.18"
 langid = "^1.1.6"
 colorama = "^0.4.4"
 spdx-license-list = "^0.5.2"
+ftfy = "^5.9"
+SQLAlchemy = ">=1.3.3,<1.4.23"
+country-converter = "^0.7.4"
 
 [tool.poetry.dev-dependencies]
 pytest = "^6.1.1"
 ipython = { version = "^7.18.1", python = "^3.7" }
 flake8 = "^3.8.4"
-pytest-clarity = "^0.3.0-alpha.0"
-black = "20.8b1"
+pytest-clarity = "^1.0.1"
+black = "^21.6b0"
 isort = "^5.5.4"
 csvkit = "^1.0.5"
 

requirements-dev.txt

@@ -1,72 +1,82 @@
 agate-dbf==0.2.2
-agate-excel==0.2.3
-agate-sql==0.5.5
-agate==1.6.2
-appdirs==1.4.4; python_version >= "3.6"
+agate-excel==0.2.5
+agate-sql==0.5.8
+agate==1.6.3
 appnope==0.1.2; python_version >= "3.7" and python_version < "4.0" and sys_platform == "darwin"
 atomicwrites==1.4.0; python_version >= "3.6" and python_full_version < "3.0.0" and sys_platform == "win32" and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6") or sys_platform == "win32" and python_version >= "3.6" and python_full_version >= "3.4.0" and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6")
-attrs==20.3.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
-babel==2.9.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0"
+attrs==21.2.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.6"
+babel==2.9.1; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0"
 backcall==0.2.0; python_version >= "3.7" and python_version < "4.0"
-black==20.8b1; python_version >= "3.6"
-certifi==2020.12.5; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
-chardet==4.0.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
-click==7.1.2; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.6"
+black==21.12b0; python_full_version >= "3.6.2"
+certifi==2021.10.8; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6"
+charset-normalizer==2.0.9; python_full_version >= "3.6.0" and python_version >= "3.6"
+click==8.0.3; python_version >= "3.6" and python_full_version >= "3.6.2"
 colorama==0.4.4; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0")
-csvkit==1.0.5
+commonmark==0.9.1; python_full_version >= "3.6.2" and python_full_version < "4.0.0" and (python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0")
+country-converter==0.7.4
+csvkit==1.0.6
 dbfread==2.0.7
-decorator==4.4.2; python_version >= "3.7" and python_full_version < "3.0.0" and python_version < "4.0" or python_version >= "3.7" and python_version < "4.0" and python_full_version >= "3.2.0"
-et-xmlfile==1.0.1; python_version >= "3.6"
-flake8==3.8.4; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")
-idna==2.10; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
+decorator==5.1.0; python_version >= "3.7" and python_version < "4.0"
+et-xmlfile==1.1.0; python_version >= "3.6"
+flake8==3.9.2; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0")
+ftfy==5.9; python_version >= "3.5"
+future==0.18.2; python_version >= "2.6" and python_full_version < "3.0.0" or python_full_version >= "3.3.0"
+greenlet==1.1.2; python_version >= "3" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3"
+idna==3.3; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6"
+importlib-metadata==4.8.2; python_full_version >= "3.6.2" and python_version < "3.8" and python_version >= "3.6" and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6") and (python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.8" or python_full_version >= "3.6.0" and python_version < "3.8" and python_version >= "3.6")
 iniconfig==1.1.1; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
-ipython-genutils==0.2.0; python_version >= "3.7" and python_version < "4.0"
-ipython==7.21.0; python_version >= "3.7" and python_version < "4.0"
+ipython==7.30.1; python_version >= "3.7" and python_version < "4.0"
 isodate==0.6.0
-isort==5.7.0; python_version >= "3.6" and python_version < "4.0"
-jedi==0.18.0; python_version >= "3.7" and python_version < "4.0"
+isort==5.10.1; python_full_version >= "3.6.1" and python_version < "4.0"
+itsdangerous==2.0.1; python_version >= "3.6"
+jedi==0.18.1; python_version >= "3.7" and python_version < "4.0"
 langid==1.1.6
-leather==0.3.3
-mccabe==0.6.1; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0"
-mypy-extensions==0.4.3; python_version >= "3.6"
-numpy==1.20.1; python_version >= "3.7" and python_full_version >= "3.7.1"
-openpyxl==3.0.7; python_version >= "3.6"
-packaging==20.9; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
-pandas==1.2.3; python_full_version >= "3.7.1"
-parsedatetime==2.6
-parso==0.8.1; python_version >= "3.7" and python_version < "4.0"
-pathspec==0.8.1; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.6"
+leather==0.3.4
+matplotlib-inline==0.1.3; python_version >= "3.7" and python_version < "4.0"
+mccabe==0.6.1; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
+mypy-extensions==0.4.3; python_full_version >= "3.6.2"
+numpy==1.21.1
+olefile==0.46; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0"
+openpyxl==3.0.9; python_version >= "3.6"
+packaging==21.3; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
+pandas==1.3.4; python_full_version >= "3.7.1"
+parsedatetime==2.4
+parso==0.8.3; python_version >= "3.7" and python_version < "4.0"
+pathspec==0.9.0; python_full_version >= "3.6.2"
 pexpect==4.8.0; python_version >= "3.7" and python_version < "4.0" and sys_platform != "win32"
 pickleshare==0.7.5; python_version >= "3.7" and python_version < "4.0"
-pluggy==0.13.1; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
-prompt-toolkit==3.0.16; python_version >= "3.7" and python_version < "4.0" and python_full_version >= "3.6.1"
+platformdirs==2.4.0; python_version >= "3.6" and python_full_version >= "3.6.2"
+pluggy==1.0.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
+pprintpp==0.4.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0"
+prompt-toolkit==3.0.23; python_version >= "3.7" and python_version < "4.0" and python_full_version >= "3.6.2"
 ptyprocess==0.7.0; python_version >= "3.7" and python_version < "4.0" and sys_platform != "win32"
-py==1.10.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
-pycodestyle==2.6.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0"
+py==1.11.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.6"
+pycodestyle==2.7.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
 pycountry==19.8.18
-pyflakes==2.2.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0"
-pygments==2.8.1; python_version >= "3.7" and python_version < "4.0"
-pyicu==2.6
-pyparsing==2.4.7; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
-pytest-clarity==0.3.0a0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")
-pytest==6.2.2; python_version >= "3.6"
-python-dateutil==2.8.1; python_full_version >= "3.7.1"
-python-slugify==4.0.1; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
-python-stdnum==1.16
+pyflakes==2.3.1; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
+pygments==2.10.0; python_version >= "3.7" and python_version < "4.0" and python_full_version >= "3.6.2" and python_full_version < "4.0.0" and (python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0")
+pyparsing==3.0.6; python_version >= "3.6"
+pytest-clarity==1.0.1; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")
+pytest==6.2.5; python_version >= "3.6"
+python-dateutil==2.8.2; python_full_version >= "3.7.1"
+python-slugify==5.0.2; python_version >= "3.6"
+python-stdnum==1.17
 pytimeparse==1.1.8
-pytz==2021.1; python_full_version >= "3.7.1"
-regex==2020.11.13; python_version >= "3.6"
-requests-cache==0.5.2
-requests==2.25.1; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0")
-six==1.15.0; python_full_version >= "3.7.1"
+pytz==2021.3; python_full_version >= "3.7.1"
+requests-cache==0.6.4; python_version >= "3.6"
+requests==2.26.0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.6.0")
+rich==10.15.2; python_full_version >= "3.6.2" and python_full_version < "4.0.0" and (python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0")
+six==1.16.0; python_full_version >= "3.7.1" and python_version >= "3.6" and (python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.3.0")
 spdx-license-list==0.5.2
-sqlalchemy==1.3.23; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0"
-termcolor==1.1.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0"
-text-unidecode==1.3; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
+sqlalchemy==1.4.22; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.6.0")
+text-unidecode==1.3; python_version >= "3.6"
 toml==0.10.2; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
-traitlets==5.0.5; python_version >= "3.7" and python_version < "4.0"
-typed-ast==1.4.2; python_version >= "3.6"
-typing-extensions==3.7.4.3; python_version >= "3.6"
-urllib3==1.26.3; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version < "4"
-wcwidth==0.2.5; python_version >= "3.7" and python_version < "4.0" and python_full_version >= "3.6.1"
+tomli==1.2.2; python_version >= "3.6" and python_full_version >= "3.6.2"
+traitlets==5.1.1; python_version >= "3.7" and python_version < "4.0"
+typed-ast==1.5.1; python_version < "3.8" and implementation_name == "cpython" and python_full_version >= "3.6.2" and python_version >= "3.6"
+typing-extensions==4.0.1
+url-normalize==1.4.3; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6"
+urllib3==1.26.7; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version < "4" and python_version >= "3.6"
+wcwidth==0.2.5; python_version >= "3.7" and python_version < "4.0" and python_full_version >= "3.6.2"
 xlrd==1.2.0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")
+zipp==3.6.0; python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.8" or python_full_version >= "3.6.0" and python_version < "3.8" and python_version >= "3.6"

requirements.txt

@@ -1,17 +1,201 @@
-certifi==2020.12.5; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
-chardet==4.0.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
-colorama==0.4.4; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0")
-idna==2.10; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
-langid==1.1.6
-numpy==1.20.1; python_version >= "3.7" and python_full_version >= "3.7.1"
-pandas==1.2.3; python_full_version >= "3.7.1"
-pycountry==19.8.18
-python-dateutil==2.8.1; python_full_version >= "3.7.1"
-python-stdnum==1.16
-pytz==2021.1; python_full_version >= "3.7.1"
-requests-cache==0.5.2
-requests==2.25.1; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0")
-six==1.15.0; python_full_version >= "3.7.1"
-spdx-license-list==0.5.2
-urllib3==1.26.3; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version < "4"
-xlrd==1.2.0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")
+certifi==2021.10.8; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6" \
+    --hash=sha256:d62a0163eb4c2344ac042ab2bdf75399a71a2d8c7d47eac2e2ee91b9d6339569 \
+    --hash=sha256:78884e7c1d4b00ce3cea67b44566851c4343c120abd683433ce934a68ea58872
+charset-normalizer==2.0.9; python_full_version >= "3.6.0" and python_version >= "3.6" \
+    --hash=sha256:b0b883e8e874edfdece9c28f314e3dd5badf067342e42fb162203335ae61aa2c \
+    --hash=sha256:1eecaa09422db5be9e29d7fc65664e6c33bd06f9ced7838578ba40d58bdf3721
+colorama==0.4.4; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0") \
+    --hash=sha256:9f47eda37229f68eee03b24b9748937c7dc3868f906e8ba69fbcbdd3bc5dc3e2 \
+    --hash=sha256:5941b2b48a20143d2267e95b1c2a7603ce057ee39fd88e7329b0c292aa16869b
+country-converter==0.7.4 \
+    --hash=sha256:0291cc91c4a4efe7f128a11c8c6e4cb761f7fea7cde2517f8677c7c56da334d3
+ftfy==5.9; python_version >= "3.5" \
+    --hash=sha256:8c4fb2863c0b82eae2ab3cf353d9ade268dfbde863d322f78d6a9fd5cefb31e9
+greenlet==1.1.2; python_version >= "3" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3" \
+    --hash=sha256:58df5c2a0e293bf665a51f8a100d3e9956febfbf1d9aaf8c0677cf70218910c6 \
+    --hash=sha256:aec52725173bd3a7b56fe91bc56eccb26fbdff1386ef123abb63c84c5b43b63a \
+    --hash=sha256:833e1551925ed51e6b44c800e71e77dacd7e49181fdc9ac9a0bf3714d515785d \
+    --hash=sha256:aa5b467f15e78b82257319aebc78dd2915e4c1436c3c0d1ad6f53e47ba6e2713 \
+    --hash=sha256:40b951f601af999a8bf2ce8c71e8aaa4e8c6f78ff8afae7b808aae2dc50d4c40 \
+    --hash=sha256:95e69877983ea39b7303570fa6760f81a3eec23d0e3ab2021b7144b94d06202d \
+    --hash=sha256:356b3576ad078c89a6107caa9c50cc14e98e3a6c4874a37c3e0273e4baf33de8 \
+    --hash=sha256:8639cadfda96737427330a094476d4c7a56ac03de7265622fcf4cfe57c8ae18d \
+    --hash=sha256:97e5306482182170ade15c4b0d8386ded995a07d7cc2ca8f27958d34d6736497 \
+    --hash=sha256:e6a36bb9474218c7a5b27ae476035497a6990e21d04c279884eb10d9b290f1b1 \
+    --hash=sha256:abb7a75ed8b968f3061327c433a0fbd17b729947b400747c334a9c29a9af6c58 \
+    --hash=sha256:14d4f3cd4e8b524ae9b8aa567858beed70c392fdec26dbdb0a8a418392e71708 \
+    --hash=sha256:17ff94e7a83aa8671a25bf5b59326ec26da379ace2ebc4411d690d80a7fbcf23 \
+    --hash=sha256:9f3cba480d3deb69f6ee2c1825060177a22c7826431458c697df88e6aeb3caee \
+    --hash=sha256:fa877ca7f6b48054f847b61d6fa7bed5cebb663ebc55e018fda12db09dcc664c \
+    --hash=sha256:7cbd7574ce8e138bda9df4efc6bf2ab8572c9aff640d8ecfece1b006b68da963 \
+    --hash=sha256:903bbd302a2378f984aef528f76d4c9b1748f318fe1294961c072bdc7f2ffa3e \
+    --hash=sha256:049fe7579230e44daef03a259faa24511d10ebfa44f69411d99e6a184fe68073 \
+    --hash=sha256:dd0b1e9e891f69e7675ba5c92e28b90eaa045f6ab134ffe70b52e948aa175b3c \
+    --hash=sha256:7418b6bfc7fe3331541b84bb2141c9baf1ec7132a7ecd9f375912eca810e714e \
+    --hash=sha256:f9d29ca8a77117315101425ec7ec2a47a22ccf59f5593378fc4077ac5b754fce \
+    --hash=sha256:21915eb821a6b3d9d8eefdaf57d6c345b970ad722f856cd71739493ce003ad08 \
+    --hash=sha256:eff9d20417ff9dcb0d25e2defc2574d10b491bf2e693b4e491914738b7908168 \
+    --hash=sha256:32ca72bbc673adbcfecb935bb3fb1b74e663d10a4b241aaa2f5a75fe1d1f90aa \
+    --hash=sha256:f0214eb2a23b85528310dad848ad2ac58e735612929c8072f6093f3585fd342d \
+    --hash=sha256:b92e29e58bef6d9cfd340c72b04d74c4b4e9f70c9fa7c78b674d1fec18896dc4 \
+    --hash=sha256:fdcec0b8399108577ec290f55551d926d9a1fa6cad45882093a7a07ac5ec147b \
+    --hash=sha256:93f81b134a165cc17123626ab8da2e30c0455441d4ab5576eed73a64c025b25c \
+    --hash=sha256:1e12bdc622676ce47ae9abbf455c189e442afdde8818d9da983085df6312e7a1 \
+    --hash=sha256:8c790abda465726cfb8bb08bd4ca9a5d0a7bd77c7ac1ca1b839ad823b948ea28 \
+    --hash=sha256:f276df9830dba7a333544bd41070e8175762a7ac20350786b322b714b0e654f5 \
+    --hash=sha256:64e6175c2e53195278d7388c454e0b30997573f3f4bd63697f88d855f7a6a1fc \
+    --hash=sha256:b11548073a2213d950c3f671aa88e6f83cda6e2fb97a8b6317b1b5b33d850e06 \
+    --hash=sha256:9633b3034d3d901f0a46b7939f8c4d64427dfba6bbc5a36b1a67364cf148a1b0 \
+    --hash=sha256:eb6ea6da4c787111adf40f697b4e58732ee0942b5d3bd8f435277643329ba627 \
+    --hash=sha256:f3acda1924472472ddd60c29e5b9db0cec629fbe3c5c5accb74d6d6d14773478 \
+    --hash=sha256:e859fcb4cbe93504ea18008d1df98dee4f7766db66c435e4882ab35cf70cac43 \
+    --hash=sha256:00e44c8afdbe5467e4f7b5851be223be68adb4272f44696ee71fe46b7036a711 \
+    --hash=sha256:ec8c433b3ab0419100bd45b47c9c8551248a5aee30ca5e9d399a0b57ac04651b \
+    --hash=sha256:288c6a76705dc54fba69fbcb59904ae4ad768b4c768839b8ca5fdadec6dd8cfd \
+    --hash=sha256:8d2f1fb53a421b410751887eb4ff21386d119ef9cde3797bf5e7ed49fb51a3b3 \
+    --hash=sha256:166eac03e48784a6a6e0e5f041cfebb1ab400b394db188c48b3a84737f505b67 \
+    --hash=sha256:572e1787d1460da79590bf44304abbc0a2da944ea64ec549188fa84d89bba7ab \
+    --hash=sha256:be5f425ff1f5f4b3c1e33ad64ab994eed12fc284a6ea71c5243fd564502ecbe5 \
+    --hash=sha256:b1692f7d6bc45e3200844be0dba153612103db241691088626a33ff1f24a0d88 \
+    --hash=sha256:7227b47e73dedaa513cdebb98469705ef0d66eb5a1250144468e9c3097d6b59b \
+    --hash=sha256:7ff61ff178250f9bb3cd89752df0f1dd0e27316a8bd1465351652b1b4a4cdfd3 \
+    --hash=sha256:f70a9e237bb792c7cc7e44c531fd48f5897961701cdaa06cf22fc14965c496cf \
+    --hash=sha256:013d61294b6cd8fe3242932c1c5e36e5d1db2c8afb58606c5a67efce62c1f5fd \
+    --hash=sha256:e30f5ea4ae2346e62cedde8794a56858a67b878dd79f7df76a0767e356b1744a
+idna==3.3; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6" \
+    --hash=sha256:84d9dd047ffa80596e0f246e2eab0b391788b0503584e8945f2368256d2735ff \
+    --hash=sha256:9d643ff0a55b762d5cdb124b8eaa99c66322e2157b69160bc32796e824360e6d
+importlib-metadata==4.8.2; python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.8" or python_full_version >= "3.6.0" and python_version < "3.8" and python_version >= "3.6" \
+    --hash=sha256:53ccfd5c134223e497627b9815d5030edf77d2ed573922f7a0b8f8bb81a1c100 \
+    --hash=sha256:75bdec14c397f528724c1bfd9709d660b33a4d2e77387a3358f20b848bb5e5fb
+itsdangerous==2.0.1; python_version >= "3.6" \
+    --hash=sha256:5174094b9637652bdb841a3029700391451bd092ba3db90600dea710ba28e97c \
+    --hash=sha256:9e724d68fc22902a1435351f84c3fb8623f303fffcc566a4cb952df8c572cff0
+langid==1.1.6 \
+    --hash=sha256:044bcae1912dab85c33d8e98f2811b8f4ff1213e5e9a9e9510137b84da2cb293
+numpy==1.21.1 \
+    --hash=sha256:38e8648f9449a549a7dfe8d8755a5979b45b3538520d1e735637ef28e8c2dc50 \
+    --hash=sha256:fd7d7409fa643a91d0a05c7554dd68aa9c9bb16e186f6ccfe40d6e003156e33a \
+    --hash=sha256:a75b4498b1e93d8b700282dc8e655b8bd559c0904b3910b144646dbbbc03e062 \
+    --hash=sha256:1412aa0aec3e00bc23fbb8664d76552b4efde98fb71f60737c83efbac24112f1 \
+    --hash=sha256:e46ceaff65609b5399163de5893d8f2a82d3c77d5e56d976c8b5fb01faa6b671 \
+    --hash=sha256:c6a2324085dd52f96498419ba95b5777e40b6bcbc20088fddb9e8cbb58885e8e \
+    --hash=sha256:73101b2a1fef16602696d133db402a7e7586654682244344b8329cdcbbb82172 \
+    --hash=sha256:7a708a79c9a9d26904d1cca8d383bf869edf6f8e7650d85dbc77b041e8c5a0f8 \
+    --hash=sha256:95b995d0c413f5d0428b3f880e8fe1660ff9396dcd1f9eedbc311f37b5652e16 \
+    --hash=sha256:635e6bd31c9fb3d475c8f44a089569070d10a9ef18ed13738b03049280281267 \
+    --hash=sha256:4a3d5fb89bfe21be2ef47c0614b9c9c707b7362386c9a3ff1feae63e0267ccb6 \
+    --hash=sha256:8a326af80e86d0e9ce92bcc1e65c8ff88297de4fa14ee936cb2293d414c9ec63 \
+    --hash=sha256:791492091744b0fe390a6ce85cc1bf5149968ac7d5f0477288f78c89b385d9af \
+    --hash=sha256:0318c465786c1f63ac05d7c4dbcecd4d2d7e13f0959b01b534ea1e92202235c5 \
+    --hash=sha256:9a513bd9c1551894ee3d31369f9b07460ef223694098cf27d399513415855b68 \
+    --hash=sha256:91c6f5fc58df1e0a3cc0c3a717bb3308ff850abdaa6d2d802573ee2b11f674a8 \
+    --hash=sha256:978010b68e17150db8765355d1ccdd450f9fc916824e8c4e35ee620590e234cd \
+    --hash=sha256:9749a40a5b22333467f02fe11edc98f022133ee1bfa8ab99bda5e5437b831214 \
+    --hash=sha256:d7a4aeac3b94af92a9373d6e77b37691b86411f9745190d2c351f410ab3a791f \
+    --hash=sha256:d9e7912a56108aba9b31df688a4c4f5cb0d9d3787386b87d504762b6754fbb1b \
+    --hash=sha256:25b40b98ebdd272bc3020935427a4530b7d60dfbe1ab9381a39147834e985eac \
+    --hash=sha256:8a92c5aea763d14ba9d6475803fc7904bda7decc2a0a68153f587ad82941fec1 \
+    --hash=sha256:05a0f648eb28bae4bcb204e6fd14603de2908de982e761a2fc78efe0f19e96e1 \
+    --hash=sha256:f01f28075a92eede918b965e86e8f0ba7b7797a95aa8d35e1cc8821f5fc3ad6a \
+    --hash=sha256:88c0b89ad1cc24a5efbb99ff9ab5db0f9a86e9cc50240177a571fbe9c2860ac2 \
+    --hash=sha256:01721eefe70544d548425a07c80be8377096a54118070b8a62476866d5208e33 \
+    --hash=sha256:2d4d1de6e6fb3d28781c73fbde702ac97f03d79e4ffd6598b880b2d95d62ead4 \
+    --hash=sha256:dff4af63638afcc57a3dfb9e4b26d434a7a602d225b42d746ea7fe2edf1342fd
+pandas==1.3.4; python_full_version >= "3.7.1" \
+    --hash=sha256:9707bdc1ea9639c886b4d3be6e2a45812c1ac0c2080f94c31b71c9fa35556f9b \
+    --hash=sha256:c2f44425594ae85e119459bb5abb0748d76ef01d9c08583a667e3339e134218e \
+    --hash=sha256:372d72a3d8a5f2dbaf566a5fa5fa7f230842ac80f29a931fb4b071502cf86b9a \
+    --hash=sha256:d99d2350adb7b6c3f7f8f0e5dfb7d34ff8dd4bc0a53e62c445b7e43e163fce63 \
+    --hash=sha256:4acc28364863127bca1029fb72228e6f473bb50c32e77155e80b410e2068eeac \
+    --hash=sha256:c2646458e1dce44df9f71a01dc65f7e8fa4307f29e5c0f2f92c97f47a5bf22f5 \
+    --hash=sha256:5298a733e5bfbb761181fd4672c36d0c627320eb999c59c65156c6a90c7e1b4f \
+    --hash=sha256:22808afb8f96e2269dcc5b846decacb2f526dd0b47baebc63d913bf847317c8f \
+    --hash=sha256:b528e126c13816a4374e56b7b18bfe91f7a7f6576d1aadba5dee6a87a7f479ae \
+    --hash=sha256:fe48e4925455c964db914b958f6e7032d285848b7538a5e1b19aeb26ffaea3ec \
+    --hash=sha256:eaca36a80acaacb8183930e2e5ad7f71539a66805d6204ea88736570b2876a7b \
+    --hash=sha256:42493f8ae67918bf129869abea8204df899902287a7f5eaf596c8e54e0ac7ff4 \
+    --hash=sha256:a388960f979665b447f0847626e40f99af8cf191bce9dc571d716433130cb3a7 \
+    --hash=sha256:5ba0aac1397e1d7b654fccf263a4798a9e84ef749866060d19e577e927d66e1b \
+    --hash=sha256:f567e972dce3bbc3a8076e0b675273b4a9e8576ac629149cf8286ee13c259ae5 \
+    --hash=sha256:c1aa4de4919358c5ef119f6377bc5964b3a7023c23e845d9db7d9016fa0c5b1c \
+    --hash=sha256:dd324f8ee05925ee85de0ea3f0d66e1362e8c80799eb4eb04927d32335a3e44a \
+    --hash=sha256:d47750cf07dee6b55d8423471be70d627314277976ff2edd1381f02d52dbadf9 \
+    --hash=sha256:2d1dc09c0013d8faa7474574d61b575f9af6257ab95c93dcf33a14fd8d2c1bab \
+    --hash=sha256:10e10a2527db79af6e830c3d5842a4d60383b162885270f8cffc15abca4ba4a9 \
+    --hash=sha256:35c77609acd2e4d517da41bae0c11c70d31c87aae8dd1aabd2670906c6d2c143 \
+    --hash=sha256:003ba92db58b71a5f8add604a17a059f3068ef4e8c0c365b088468d0d64935fd \
+    --hash=sha256:a51528192755f7429c5bcc9e80832c517340317c861318fea9cea081b57c9afd \
+    --hash=sha256:a2aa18d3f0b7d538e21932f637fbfe8518d085238b429e4790a35e1e44a96ffc
+pycountry==19.8.18 \
+    --hash=sha256:3c57aa40adcf293d59bebaffbe60d8c39976fba78d846a018dc0c2ec9c6cb3cb
+python-dateutil==2.8.2; python_full_version >= "3.7.1" \
+    --hash=sha256:0123cacc1627ae19ddf3c27a5de5bd67ee4586fbdd6440d9748f8abb483d3e86 \
+    --hash=sha256:961d03dc3453ebbc59dbdea9e4e11c5651520a876d0f4db161e8674aae935da9
+python-stdnum==1.17 \
+    --hash=sha256:374e2b5e13912ccdbf50b0b23fca2c3e0531174805c32d74e145f37756328340 \
+    --hash=sha256:a46e6cf9652807314d369b654b255c86a59f93d18be2834f3d567ed1a346c547
+pytz==2021.3; python_full_version >= "3.7.1" \
+    --hash=sha256:3672058bc3453457b622aab7a1c3bfd5ab0bdae451512f6cf25f64ed37f5b87c \
+    --hash=sha256:acad2d8b20a1af07d4e4c9d2e9285c5ed9104354062f275f3fcd88dcef4f1326
+requests-cache==0.6.4; python_version >= "3.6" \
+    --hash=sha256:dd9120a4ab7b8128cba9b6b120d8b5560d566a3cd0f828cced3d3fd60a42ec40 \
+    --hash=sha256:1102daa13a804abe23fad62d694e7dee58d6063a35d94bf6e8c9821e22e5a78b
+requests==2.26.0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.6.0") \
+    --hash=sha256:6c1246513ecd5ecd4528a0906f910e8f0f9c6b8ec72030dc9fd154dc1a6efd24 \
+    --hash=sha256:b8aa58f8cf793ffd8782d3d8cb19e66ef36f7aba4353eec859e74678b01b07a7
+six==1.16.0; python_full_version >= "3.7.1" and python_version >= "3.6" \
+    --hash=sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254 \
+    --hash=sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926
+spdx-license-list==0.5.2 \
+    --hash=sha256:1b338470c7b403dbecceca563a316382c7977516128ca6c1e8f7078e3ed6e7b0 \
+    --hash=sha256:952996f72ab807972dc2278bb9b91e5294767211e51f09aad9c0e2ff5b82a31b
+sqlalchemy==1.4.22; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.6.0") \
+    --hash=sha256:488608953385d6c127d2dcbc4b11f8d7f2f30b89f6bd27c01b042253d985cc2f \
+    --hash=sha256:5d856cc50fd26fc8dd04892ed5a5a3d7eeb914fea2c2e484183e2d84c14926e0 \
+    --hash=sha256:a00d9c6d3a8afe1d1681cd8a5266d2f0ed684b0b44bada2ca82403b9e8b25d39 \
+    --hash=sha256:5908ea6c652a050d768580d01219c98c071e71910ab8e7b42c02af4010608397 \
+    --hash=sha256:b7fb937c720847879c7402fe300cfdb2aeff22349fa4ea3651bca4e2d6555939 \
+    --hash=sha256:9bfe882d5a1bbde0245dca0bd48da0976bd6634cf2041d2fdf0417c5463e40e5 \
+    --hash=sha256:eedd76f135461cf237534a6dc0d1e0f6bb88a1dc193678fab48a11d223462da5 \
+    --hash=sha256:6a16c7c4452293da5143afa3056680db2d187b380b3ef4d470d4e29885720de3 \
+    --hash=sha256:44d23ea797a5e0be71bc5454b9ae99158ea0edc79e2393c6e9a2354de88329c0 \
+    --hash=sha256:a5e14cb0c0a4ac095395f24575a0e7ab5d1be27f5f9347f1762f21505e3ba9f1 \
+    --hash=sha256:bc34a007e604091ca3a4a057525efc4cefd2b7fe970f44d20b9cfa109ab1bddb \
+    --hash=sha256:756f5d2f5b92d27450167247fb574b09c4cd192a3f8c2e493b3e518a204ee543 \
+    --hash=sha256:9fcbb4b4756b250ed19adc5e28c005b8ed56fdb5c21efa24c6822c0575b4964d \
+    --hash=sha256:09dbb4bc01a734ccddbf188deb2a69aede4b3c153a72b6d5c6900be7fb2945b1 \
+    --hash=sha256:f028ef6a1d828bc754852a022b2160e036202ac8658a6c7d34875aafd14a9a15 \
+    --hash=sha256:68393d3fd31469845b6ba11f5b4209edbea0b58506be0e077aafbf9aa2e21e11 \
+    --hash=sha256:891927a49b2363a4199763a9d436d97b0b42c65922a4ea09025600b81a00d17e \
+    --hash=sha256:fd2102a8f8a659522719ed73865dff3d3cc76eb0833039dc473e0ad3041d04be \
+    --hash=sha256:4014978de28163cd8027434916a92d0f5bb1a3a38dff5e8bf8bff4d9372a9117 \
+    --hash=sha256:f814d80844969b0d22ea63663da4de5ca1c434cfbae226188901e5d368792c17 \
+    --hash=sha256:d09a760b0a045b4d799102ae7965b5491ccf102123f14b2a8cc6c01d1021a2d9 \
+    --hash=sha256:26daa429f039e29b1e523bf763bfab17490556b974c77b5ca7acb545b9230e9a \
+    --hash=sha256:12bac5fa1a6ea870bdccb96fe01610641dd44ebe001ed91ef7fcd980e9702db5 \
+    --hash=sha256:39b5d36ab71f73c068cdcf70c38075511de73616e6c7fdd112d6268c2704d9f5 \
+    --hash=sha256:5102b9face693e8b2db3b2539c7e1a5d9a5b4dc0d79967670626ffd2f710d6e6 \
+    --hash=sha256:c9373ef67a127799027091fa53449125351a8c943ddaa97bec4e99271dbb21f4 \
+    --hash=sha256:36a089dc604032d41343d86290ce85d4e6886012eea73faa88001260abf5ff81 \
+    --hash=sha256:b48148ceedfb55f764562e04c00539bb9ea72bf07820ca15a594a9a049ff6b0e \
+    --hash=sha256:1fdae7d980a2fa617d119d0dc13ecb5c23cc63a8b04ffcb5298f2c59d86851e9 \
+    --hash=sha256:ec1be26cdccd60d180359a527d5980d959a26269a2c7b1b327a1eea0cab37ed8
+typing-extensions==4.0.1; python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.8" or python_full_version >= "3.6.0" and python_version < "3.8" and python_version >= "3.6" \
+    --hash=sha256:7f001e5ac290a0c0401508864c7ec868be4e701886d5b573a9528ed3973d9d3b \
+    --hash=sha256:4ca091dea149f945ec56afb48dae714f21e8692ef22a395223bcd328961b6a0e
+url-normalize==1.4.3; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6" \
+    --hash=sha256:d23d3a070ac52a67b83a1c59a0e68f8608d1cd538783b401bc9de2c0fac999b2 \
+    --hash=sha256:ec3c301f04e5bb676d333a7fa162fa977ad2ca04b7e652bfc9fac4e405728eed
+urllib3==1.26.7; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version < "4" and python_version >= "3.6" \
+    --hash=sha256:c4fdf4019605b6e5423637e01bc9fe4daef873709a7973e195ceba0a62bbc844 \
+    --hash=sha256:4987c65554f7a2dbf30c18fd48778ef124af6fab771a377103da0585e2336ece
+wcwidth==0.2.5; python_version >= "3.5" \
+    --hash=sha256:beb4802a9cebb9144e99086eff703a642a13d6a0052920003a230f3294bbe784 \
+    --hash=sha256:c4d647b99872929fdb7bdcaa4fbe7f01413ed3d98077df798530e5b04f116c83
+xlrd==1.2.0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0") \
+    --hash=sha256:e551fb498759fa3a5384a94ccd4c3c02eb7c00ea424426e212ac0c57be9dfbde \
+    --hash=sha256:546eb36cee8db40c3eaa46c351e67ffee6eeb5fa2650b71bc4c758a29a1b29b2
+zipp==3.6.0; python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.8" or python_full_version >= "3.6.0" and python_version < "3.8" and python_version >= "3.6" \
+    --hash=sha256:9fe5ea21568a0a70e50f273397638d39b03353731e6cbbb3fd8502a33fec40bc \
+    --hash=sha256:71c644c5369f4a6e07636f0aa966270449561fcea2e3d6747b8d23efaa9d7832

setup.py

@@ -14,7 +14,7 @@ install_requires = [
 
 setuptools.setup(
     name="csv-metadata-quality",
-    version="0.4.6",
+    version="0.5.0",
     author="Alan Orth",
     author_email="aorth@mjanja.ch",
     description="A simple, but opinionated CSV quality checking and fixing pipeline for CSVs in the DSpace ecosystem.",
@@ -28,7 +28,6 @@ setuptools.setup(
         "Programming Language :: Python :: 3.9",
         "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
         "Operating System :: OS Independent",
-        "Development Status :: 4 - Beta",
     ],
     packages=["csv_metadata_quality"],
     entry_points={

tests/test_check.py

@@ -1,3 +1,5 @@
+# SPDX-License-Identifier: GPL-3.0-only
+
 import pandas as pd
 from colorama import Fore
 
@@ -23,7 +25,7 @@ def test_check_valid_issn():
 
     result = check.issn(value)
 
-    assert result == value
+    assert result == None
 
 
 def test_check_invalid_isbn(capsys):
@@ -44,51 +46,7 @@ def test_check_valid_isbn():
 
     result = check.isbn(value)
 
-    assert result == value
-
-
-def test_check_invalid_separators(capsys):
-    """Test checking invalid multi-value separators."""
-
-    value = "Alan|Orth"
-
-    field_name = "dc.contributor.author"
-
-    check.separators(value, field_name)
-
-    captured = capsys.readouterr()
-    assert (
-        captured.out
-        == f"{Fore.RED}Invalid multi-value separator ({field_name}): {Fore.RESET}{value}\n"
-    )
-
-
-def test_check_unnecessary_separators(capsys):
-    """Test checking unnecessary multi-value separators."""
-
-    field = "Alan||Orth||"
-
-    field_name = "dc.contributor.author"
-
-    check.separators(field, field_name)
-
-    captured = capsys.readouterr()
-    assert (
-        captured.out
-        == f"{Fore.RED}Unnecessary multi-value separator ({field_name}): {Fore.RESET}{field}\n"
-    )
-
-
-def test_check_valid_separators():
-    """Test checking valid multi-value separators."""
-
-    value = "Alan||Orth"
-
-    field_name = "dc.contributor.author"
-
-    result = check.separators(value, field_name)
-
-    assert result == value
+    assert result == None
 
 
 def test_check_missing_date(capsys):
@@ -144,7 +102,7 @@ def test_check_valid_date():
 
     result = check.date(value, field_name)
 
-    assert result == value
+    assert result == None
 
 
 def test_check_suspicious_characters(capsys):
@@ -170,7 +128,7 @@ def test_check_valid_iso639_1_language():
 
     result = check.language(value)
 
-    assert result == value
+    assert result == None
 
 
 def test_check_valid_iso639_3_language():
@@ -180,7 +138,7 @@ def test_check_valid_iso639_3_language():
 
     result = check.language(value)
 
-    assert result == value
+    assert result == None
 
 
 def test_check_invalid_iso639_1_language(capsys):
@@ -243,7 +201,7 @@ def test_check_valid_agrovoc():
 
     result = check.agrovoc(value, field_name)
 
-    assert result == value
+    assert result == None
 
 
 def test_check_uncommon_filename_extension(capsys):
@@ -267,7 +225,7 @@ def test_check_common_filename_extension():
 
     result = check.filename_extension(value)
 
-    assert result == value
+    assert result == None
 
 
 def test_check_incorrect_iso_639_1_language(capsys):
@@ -320,7 +278,7 @@ def test_check_correct_iso_639_1_language():
 
     result = experimental.correct_language(series)
 
-    assert result == language
+    assert result == None
 
 
 def test_check_correct_iso_639_3_language():
@@ -335,7 +293,7 @@ def test_check_correct_iso_639_3_language():
 
     result = experimental.correct_language(series)
 
-    assert result == language
+    assert result == None
 
 
 def test_check_valid_spdx_license_identifier():
@@ -345,7 +303,7 @@ def test_check_valid_spdx_license_identifier():
 
     result = check.spdx_license_identifier(license)
 
-    assert result == license
+    assert result == None
 
 
 def test_check_invalid_spdx_license_identifier(capsys):
@@ -360,3 +318,163 @@ def test_check_invalid_spdx_license_identifier(capsys):
         captured.out
         == f"{Fore.YELLOW}Non-SPDX license identifier: {Fore.RESET}{license}\n"
     )
+
+
+def test_check_duplicate_item(capsys):
+    """Test item with duplicate title, type, and date."""
+
+    item_title = "Title"
+    item_type = "Report"
+    item_date = "2021-03-17"
+
+    d = {
+        "dc.title": [item_title, item_title],
+        "dcterms.type": [item_type, item_type],
+        "dcterms.issued": [item_date, item_date],
+    }
+    df = pd.DataFrame(data=d)
+
+    result = check.duplicate_items(df)
+
+    captured = capsys.readouterr()
+    assert (
+        captured.out
+        == f"{Fore.YELLOW}Possible duplicate (dc.title): {Fore.RESET}{item_title}\n"
+    )
+
+
+def test_check_no_mojibake():
+    """Test string with no mojibake."""
+
+    field = "CIAT Publicaçao"
+    field_name = "dcterms.isPartOf"
+
+    result = check.mojibake(field, field_name)
+
+    assert result == None
+
+
+def test_check_mojibake(capsys):
+    """Test string with mojibake."""
+
+    field = "CIAT Publicaçao"
+    field_name = "dcterms.isPartOf"
+
+    result = check.mojibake(field, field_name)
+
+    captured = capsys.readouterr()
+    assert (
+        captured.out
+        == f"{Fore.YELLOW}Possible encoding issue ({field_name}): {Fore.RESET}{field}\n"
+    )
+
+
+def test_check_doi_field():
+    """Test an item with a DOI field."""
+
+    doi = "https://doi.org/10.1186/1743-422X-9-218"
+    citation = "Orth, A. 2021. Testing all the things. doi: 10.1186/1743-422X-9-218"
+
+    # Emulate a column in a transposed dataframe (which is just a series), with
+    # the citation and a DOI field.
+    d = {"cg.identifier.doi": doi, "dcterms.bibliographicCitation": citation}
+    series = pd.Series(data=d)
+
+    result = check.citation_doi(series)
+
+    assert result == None
+
+
+def test_check_doi_only_in_citation(capsys):
+    """Test an item with a DOI in its citation, but no DOI field."""
+
+    citation = "Orth, A. 2021. Testing all the things. doi: 10.1186/1743-422X-9-218"
+
+    # Emulate a column in a transposed dataframe (which is just a series), with
+    # an empty DOI field and a citation containing a DOI.
+    d = {"cg.identifier.doi": None, "dcterms.bibliographicCitation": citation}
+    series = pd.Series(data=d)
+
+    check.citation_doi(series)
+
+    captured = capsys.readouterr()
+    assert (
+        captured.out
+        == f"{Fore.YELLOW}DOI in citation, but missing a DOI field: {Fore.RESET}{citation}\n"
+    )
+
+
+def test_title_in_citation():
+    """Test an item with its title in the citation."""
+
+    title = "Testing all the things"
+    citation = "Orth, A. 2021. Testing all the things."
+
+    # Emulate a column in a transposed dataframe (which is just a series), with
+    # the title and citation.
+    d = {"dc.title": title, "dcterms.bibliographicCitation": citation}
+    series = pd.Series(data=d)
+
+    result = check.title_in_citation(series)
+
+    assert result == None
+
+
+def test_title_not_in_citation(capsys):
+    """Test an item with its title missing from the citation."""
+
+    title = "Testing all the things"
+    citation = "Orth, A. 2021. Testing all teh things."
+
+    # Emulate a column in a transposed dataframe (which is just a series), with
+    # the title and citation.
+    d = {"dc.title": title, "dcterms.bibliographicCitation": citation}
+    series = pd.Series(data=d)
+
+    check.title_in_citation(series)
+
+    captured = capsys.readouterr()
+    assert (
+        captured.out
+        == f"{Fore.YELLOW}Title is not present in citation: {Fore.RESET}{title}\n"
+    )
+
+
+def test_country_matches_region():
+    """Test an item with regions matching its country list."""
+
+    country = "Kenya"
+    region = "Eastern Africa"
+
+    # Emulate a column in a transposed dataframe (which is just a series)
+    d = {"cg.coverage.country": country, "cg.coverage.region": region}
+    series = pd.Series(data=d)
+
+    result = check.countries_match_regions(series)
+
+    assert result == None
+
+
+def test_country_not_matching_region(capsys):
+    """Test an item with regions not matching its country list."""
+
+    title = "Testing an item with no matching region."
+    country = "Kenya"
+    region = ""
+    missing_region = "Eastern Africa"
+
+    # Emulate a column in a transposed dataframe (which is just a series)
+    d = {
+        "dc.title": title,
+        "cg.coverage.country": country,
+        "cg.coverage.region": region,
+    }
+    series = pd.Series(data=d)
+
+    check.countries_match_regions(series)
+
+    captured = capsys.readouterr()
+    assert (
+        captured.out
+        == f"{Fore.YELLOW}Missing region ({missing_region}): {Fore.RESET}{title}\n"
+    )

tests/test_fix.py

@@ -1,3 +1,5 @@
+# SPDX-License-Identifier: GPL-3.0-only
+
 import csv_metadata_quality.fix as fix
 
 
@@ -74,8 +76,9 @@ def test_fix_newlines():
 
     value = """Ken
 ya"""
+    field_name = "dcterms.subject"
 
-    assert fix.newlines(value) == "Kenya"
+    assert fix.newlines(value, field_name) == "Kenya"
 
 
 def test_fix_comma_space():
@@ -108,3 +111,12 @@ def test_fix_decomposed_unicode():
     field_name = "dc.contributor.author"
 
     assert fix.normalize_unicode(value, field_name) == "Ouédraogo, Mathieu"
+
+
+def test_fix_mojibake():
+    """Test string with no mojibake."""
+
+    field = "CIAT Publicaçao"
+    field_name = "dcterms.isPartOf"
+
+    assert fix.mojibake(field, field_name) == "CIAT Publicaçao"