1
0
mirror of https://github.com/ilri/csv-metadata-quality.git synced 2025-05-12 16:16:02 +02:00

60 Commits

Author SHA1 Message Date
8a2c567d1f Version 0.7.0 2025-01-31 10:12:43 +03:00
42eb9437e3 pyproject.toml: bump a few deps 2025-01-31 10:11:38 +03:00
5400bcb19b Remove pytest-clarity dependency
I think pytest itself has gotten much more readable over the years.
2025-01-31 10:10:33 +03:00
febea54f1b Remove poetry.lock
We switched to rye and then uv, so this is not needed.
2025-01-31 10:03:03 +03:00
b5565124de .github/workflows: fix pytest task 2025-01-30 11:56:27 +03:00
2869919507 pyproject.toml: minimum Python version is 3.10
Apparently this is due to ipython.
2025-01-30 11:55:17 +03:00
f7d66947f7 Use uv instead of rye 2025-01-30 11:55:09 +03:00
1d701f4056 renovate.json: use pep621 manager
We are not using pip requirements files, but I'm wondering if we
need to switch to uv instead.

See: https://docs.renovatebot.com/modules/manager/pep621/
2025-01-30 11:38:49 +03:00
1e339609a6 chore(config): migrate config renovate.json 2025-01-30 11:34:33 +03:00
2b0568de30 .github/workflows: use ubuntu-latest
This is now Ubuntu 24.04.

See: https://github.com/actions/runner-images
2025-01-30 11:28:32 +03:00
9903ada97a pyproject.toml: update isort to v6.0.0 2025-01-30 11:27:51 +03:00
d4b20e282c Python 3.13 2025-01-30 11:27:19 +03:00
9785c18301 Merge pull request #64 from ilri/renovate/ftfy-6.x 2024-10-11 20:02:53 +03:00
de5e292f1a chore(deps): update dependency ftfy to ~=6.3.0 2024-10-11 00:32:38 +00:00
2675cd288e Remove and re-sync rye requirements
Seems there is no way to force rye to sync?
2024-09-26 12:24:36 +03:00
78dc1336d0 requirements-dev.lock: run rye sync 2024-09-26 12:22:09 +03:00
28bbb919ce pyproject.toml: remove fixit from deps
I read about this online and was testing it but never used it.
2024-09-06 15:13:59 +03:00
b1de9552a4 Remove Mjanja CI (drone) 2024-09-06 12:54:16 +03:00
81e3ca3d9c .github/workflows: use rye in CI
Use rye instead of poetry in CI.
2024-08-21 18:56:09 +03:00
c470f8b375 requirements-dev.lock: run rye sync 2024-08-21 17:41:49 +03:00
0f45448517 pyproject.toml: bump dev dependencies 2024-08-21 17:41:36 +03:00
7dd52ca491 requirements: run rye sync 2024-07-29 19:58:42 -07:00
92ff0ee51b Normalize DOIs with %2f
These seem to be incorrectly URL encoded.
2024-06-25 11:54:09 +03:00
ae38a826ec csv_metadata_quality/fix.py: minor update to DOI fix
Normalize www.doi.org to doi.org in DOI field.
2024-06-25 11:48:45 +03:00
c1f630c298 Bump dependencies
All tests passing...
2024-06-18 22:17:38 +03:00
82b056f0ea Use py3langid v0.3.0 2024-06-18 21:51:32 +03:00
7fca981b95 Add .python-version
This was created with:

    rye pin --relaxed cpython@3.12

Rye will now always use the latest 3.12.x apparently.

See: https://rye-up.com/guide/toolchains/
2024-05-23 10:11:10 +03:00
1a9424197b Run rye lock --update-all and rye sync 2024-05-23 10:04:43 +03:00
f6c6c94a1e pyproject.toml: use ~= for dependencies
These are the closest to semantic versioning in Python that I can
find with PEP 621 syntax. For example:

> ~=3.1: version 3.1 or later, but not version 4.0 or later.
> ~=3.1.2: version 3.1.2 or later, but not version 3.2.0 or later.

For most cases I want to bump the minor and micro / patch.

See: https://packaging.python.org/en/latest/specifications/version-specifiers/#examples
2024-05-23 10:01:46 +03:00
f500fac64b pyproject.toml: remove scalene from dev deps 2024-05-23 10:00:01 +03:00
8143a7d978 pyproject.toml: align better with PEP 621
This PEP was approved years ago and has become a standard for the
way pyproject.toml file is laid out. We need to make some changes
to the license, URLs, add classifiers, etc.

See: https://peps.python.org/pep-0621/
2024-05-23 09:44:16 +03:00
94cec080d6 pyproject.toml: remove Hatch direct-references
Apparently I copied this from somewhere but it's not needed in this
project because we are not using direct dependency references (which
seem to be local packages).
2024-05-23 09:43:08 +03:00
9402af1e30 pyproject.toml: add comment about packages
Important for Hatch.
2024-05-23 09:42:11 +03:00
d71ff9082b pyproject.toml: add comment about backend 2024-05-23 09:41:08 +03:00
f309b694c4 Run rye sync to update lockfiles 2024-05-22 23:19:20 +03:00
4d879f6d13 pyproject.toml: remove black
rye bundles ruff so we can use that instead via `rye fmt`.
2024-05-22 23:19:20 +03:00
a30fefcd52 pyproject.toml: update formatting
rye requires a slightly different formatting.
2024-05-22 23:19:14 +03:00
2341c56c40 poetry.lock: run poetry update 2024-04-25 12:50:30 +03:00
5be2195325 Add fix for normalizing DOIs 2024-04-25 12:49:19 +03:00
736948ed2c csv_metadata_quality/check.py: run rye fmt 2024-04-12 13:40:55 +03:00
ee0b448355 csv_metadata_quality/check.py: remove unused import 2024-04-12 11:07:36 +03:00
4f3174a543 CHANGELOG.md: add note about SPDX license list
All checks were successful
continuous-integration/drone/push Build is passing
2024-03-02 10:39:00 +03:00
d5c25f82fa Update SPDX license list
From: https://github.com/spdx/license-list-data/blob/main/json/licenses.json
2024-03-02 10:38:27 +03:00
7b3e2b4e68 Merge pull request #43 from ilri/renovate/pytest-7.x-lockfile
All checks were successful
continuous-integration/drone/push Build is passing
chore(deps): update dependency pytest to v7.4.4
2024-01-05 16:40:13 +03:00
f92b2fe206 Merge pull request #44 from ilri/renovate/flake8-7.x
chore(deps): update dependency flake8 to v7
2024-01-05 16:25:22 +03:00
df040b70c7 chore(deps): update dependency flake8 to v7
All checks were successful
continuous-integration/drone/push Build is passing
2024-01-05 00:58:28 +00:00
10bc8f3e14 chore(deps): update dependency pytest to v7.4.4
All checks were successful
continuous-integration/drone/push Build is passing
2023-12-31 13:47:46 +00:00
7e6e92ecaa poetry.lock: run poetry lock
All checks were successful
continuous-integration/drone/push Build is passing
2023-12-28 14:12:03 +03:00
a21ffb0fa8 Use py3langid instead of langid
Faster and more modern code for Python 3 as a drop-in replacement.

See: https://adrien.barbaresi.eu/blog/language-detection-langid-py-faster.html
2023-12-28 14:11:21 +03:00
fb341dd9fa Merge pull request #37 from ilri/renovate/actions-setup-python-5.x
chore(deps): update actions/setup-python action to v5
2023-12-28 09:02:41 +03:00
2e943ee4db Merge pull request #39 from ilri/renovate/isort-5.x-lockfile
chore(deps): update dependency isort to v5.13.2
2023-12-28 09:01:48 +03:00
6d3a9870d6 Merge pull request #41 from ilri/renovate/pycountry-23.x-lockfile
fix(deps): update dependency pycountry to v23.12.11
2023-12-28 09:01:21 +03:00
82ecf7119a Merge pull request #42 from ilri/renovate/black-23.x-lockfile
chore(deps): update dependency black to v23.12.1
2023-12-28 09:00:39 +03:00
1db21cf275 chore(deps): update dependency black to v23.12.1
All checks were successful
continuous-integration/drone/push Build is passing
2023-12-23 00:35:13 +00:00
bcd1408798 chore(deps): update dependency isort to v5.13.2
All checks were successful
continuous-integration/drone/push Build is passing
2023-12-13 22:21:38 +00:00
ee8d255811 fix(deps): update dependency pycountry to v23.12.11
All checks were successful
continuous-integration/drone/push Build is passing
2023-12-11 21:50:09 +00:00
2cc2dbe952 tests: apply fixes from fixit
All checks were successful
continuous-integration/drone/push Build is passing
RewriteToLiteral: It's slower to call list() than using the empty literal
2023-12-09 12:20:35 +03:00
940a325d61 poetry.lock: run poetry lock 2023-12-09 12:05:26 +03:00
59b3b307c9 pyproject.toml: use official pycountry
The project is moving again and has all the latest data from the
iso-codes project.
2023-12-09 12:04:14 +03:00
96a486471c Update actions/setup-python action to v5
All checks were successful
continuous-integration/drone/push Build is passing
2023-12-06 13:13:11 +00:00
20 changed files with 3945 additions and 2722 deletions

View File

@ -1,91 +0,0 @@
---
kind: pipeline
type: docker
name: python311
steps:
- name: test
image: python:3.11-slim
commands:
- id
- python -V
- apt update && apt install -y gcc g++ libicu-dev pkg-config git
- python -m pip install poetry
- poetry install
- poetry run pytest
# Basic test
- poetry run csv-metadata-quality -i data/test.csv -o /tmp/test.csv
# Basic test with unsafe fixes
- poetry run csv-metadata-quality -i data/test.csv -o /tmp/test.csv -u
# Geography test
- poetry run csv-metadata-quality -i data/test-geography.csv -o /tmp/test.csv
# Geography test with unsafe fixes
- poetry run csv-metadata-quality -i data/test-geography.csv -o /tmp/test.csv -u
# Test with experimental checks
- poetry run csv-metadata-quality -i data/test.csv -o /tmp/test.csv -e
# Test with AGROVOC validation
- poetry run csv-metadata-quality -i data/test.csv -o /tmp/test.csv --agrovoc-fields dcterms.subject
# Test with AGROVOC validation (and dropping invalid)
- poetry run csv-metadata-quality -i data/test.csv -o /tmp/test.csv --agrovoc-fields dcterms.subject -d
---
kind: pipeline
type: docker
name: python310
steps:
- name: test
image: python:3.10-slim
commands:
- id
- python -V
- apt update && apt install -y gcc g++ libicu-dev pkg-config git
- python -m pip install poetry
- poetry install
- poetry run pytest
# Basic test
- poetry run csv-metadata-quality -i data/test.csv -o /tmp/test.csv
# Basic test with unsafe fixes
- poetry run csv-metadata-quality -i data/test.csv -o /tmp/test.csv -u
# Geography test
- poetry run csv-metadata-quality -i data/test-geography.csv -o /tmp/test.csv
# Geography test with unsafe fixes
- poetry run csv-metadata-quality -i data/test-geography.csv -o /tmp/test.csv -u
# Test with experimental checks
- poetry run csv-metadata-quality -i data/test.csv -o /tmp/test.csv -e
# Test with AGROVOC validation
- poetry run csv-metadata-quality -i data/test.csv -o /tmp/test.csv --agrovoc-fields dcterms.subject
# Test with AGROVOC validation (and dropping invalid)
- poetry run csv-metadata-quality -i data/test.csv -o /tmp/test.csv --agrovoc-fields dcterms.subject -d
---
kind: pipeline
type: docker
name: python39
steps:
- name: test
image: python:3.9-slim
commands:
- id
- python -V
- apt update && apt install -y gcc g++ libicu-dev pkg-config git
- python -m pip install poetry
- poetry install
- poetry run pytest
# Basic test
- poetry run csv-metadata-quality -i data/test.csv -o /tmp/test.csv
# Basic test with unsafe fixes
- poetry run csv-metadata-quality -i data/test.csv -o /tmp/test.csv -u
# Geography test
- poetry run csv-metadata-quality -i data/test-geography.csv -o /tmp/test.csv
# Geography test with unsafe fixes
- poetry run csv-metadata-quality -i data/test-geography.csv -o /tmp/test.csv -u
# Test with experimental checks
- poetry run csv-metadata-quality -i data/test.csv -o /tmp/test.csv -e
# Test with AGROVOC validation
- poetry run csv-metadata-quality -i data/test.csv -o /tmp/test.csv --agrovoc-fields dcterms.subject
# Test with AGROVOC validation (and dropping invalid)
- poetry run csv-metadata-quality -i data/test.csv -o /tmp/test.csv --agrovoc-fields dcterms.subject -d
# vim: ts=2 sw=2 et

View File

@ -12,34 +12,26 @@ on:
jobs: jobs:
build: build:
runs-on: ubuntu-22.04 runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
- name: Install poetry - name: Install uv
run: pipx install poetry uses: astral-sh/setup-uv@v5
- uses: actions/setup-python@v4
with: with:
python-version: '3.11' version: 'latest'
cache: 'poetry' - run: uv sync
- run: poetry install
- name: Lint with flake8
run: |
# stop the build if there are Python syntax errors or undefined names
poetry run flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
poetry run flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
- name: Test with pytest - name: Test with pytest
run: poetry run pytest run: uv run pytest
- name: Test CLI - name: Test CLI
run: | run: |
# Basic test # Basic test
poetry run csv-metadata-quality -i data/test.csv -o /tmp/test.csv uv run csv-metadata-quality -i data/test.csv -o /tmp/test.csv
# Test with unsafe fixes # Test with unsafe fixes
poetry run csv-metadata-quality -i data/test.csv -o /tmp/test.csv -u uv run csv-metadata-quality -i data/test.csv -o /tmp/test.csv -u
# Test with experimental checks # Test with experimental checks
poetry run csv-metadata-quality -i data/test.csv -o /tmp/test.csv -e uv run csv-metadata-quality -i data/test.csv -o /tmp/test.csv -e
# Test with AGROVOC validation # Test with AGROVOC validation
poetry run csv-metadata-quality -i data/test.csv -o /tmp/test.csv --agrovoc-fields dcterms.subject uv run csv-metadata-quality -i data/test.csv -o /tmp/test.csv --agrovoc-fields dcterms.subject
# Test with AGROVOC validation (and dropping invalid) # Test with AGROVOC validation (and dropping invalid)
poetry run csv-metadata-quality -i data/test.csv -o /tmp/test.csv --agrovoc-fields dcterms.subject -d uv run csv-metadata-quality -i data/test.csv -o /tmp/test.csv --agrovoc-fields dcterms.subject -d

1
.python-version Normal file
View File

@ -0,0 +1 @@
3.13

View File

@ -4,7 +4,10 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## Unreleased ## [0.7.0] - 2025-01-31
### Added
- Ability to normalize DOIs to https://doi.org URI format
### Fixed ### Fixed
- Fixed regex so we don't run the invalid multi-value separator fix on - Fixed regex so we don't run the invalid multi-value separator fix on
`dcterms.bibliographicCitation` fields `dcterms.bibliographicCitation` fields
@ -15,9 +18,13 @@ fields
### Changed ### Changed
- Don't run newline fix on description fields - Don't run newline fix on description fields
- Install requests-cache in main run() function instead of check.agrovoc() function so we only incur the overhead once - Install requests-cache in main run() function instead of check.agrovoc() function so we only incur the overhead once
- Use py3langid instead of langid, see: [How to make language detection with langid.py faster](https://adrien.barbaresi.eu/blog/language-detection-langid-py-faster.html)
- Use uv instead of rye
- Remove pytest-clarity — I think pytest itself has gotten much better in the past few years
### Updated ### Updated
- Python dependencies, including Pandas 2.0.0 and [Arrow-backed dtypes](https://datapythonista.me/blog/pandas-20-and-the-arrow-revolution-part-i) - Python dependencies, including Pandas 2.0.0 and [Arrow-backed dtypes](https://datapythonista.me/blog/pandas-20-and-the-arrow-revolution-part-i)
- SPDX license list
## [0.6.1] - 2023-02-23 ## [0.6.1] - 2023-02-23
### Fixed ### Fixed

View File

@ -1,7 +1,6 @@
<h1 align="center">DSpace CSV Metadata Quality Checker</h1> <h1 align="center">DSpace CSV Metadata Quality Checker</h1>
<p align="center"> <p align="center">
<a href="https://ci.mjanja.ch/alanorth/csv-metadata-quality"><img alt="Build Status" src="https://ci.mjanja.ch/api/badges/alanorth/csv-metadata-quality/status.svg"></a>
<a href="https://github.com/ilri/csv-metadata-quality/actions"><img alt="Build and Test" src="https://github.com/ilri/csv-metadata-quality/workflows/Build%20and%20Test/badge.svg"></a> <a href="https://github.com/ilri/csv-metadata-quality/actions"><img alt="Build and Test" src="https://github.com/ilri/csv-metadata-quality/workflows/Build%20and%20Test/badge.svg"></a>
<a href="https://github.com/psf/black"><img alt="Code style: black" src="https://img.shields.io/badge/code%20style-black-000000.svg"></a> <a href="https://github.com/psf/black"><img alt="Code style: black" src="https://img.shields.io/badge/code%20style-black-000000.svg"></a>
</p> </p>
@ -31,24 +30,25 @@ If you use the DSpace CSV metadata quality checker please cite:
- Check for countries with missing regions (and attempt to fix with `--unsafe-fixes`) - Check for countries with missing regions (and attempt to fix with `--unsafe-fixes`)
- Remove duplicate metadata values - Remove duplicate metadata values
- Check for duplicate items, using the title, type, and date issued as an indicator - Check for duplicate items, using the title, type, and date issued as an indicator
- [Normalize DOIs](https://www.crossref.org/documentation/member-setup/constructing-your-dois/) to https://doi.org URI format
## Installation ## Installation
The easiest way to install CSV Metadata Quality is with [poetry](https://python-poetry.org): The easiest way to install CSV Metadata Quality is with [uv](https://docs.astral.sh/uv/):
``` ```
$ git clone https://github.com/ilri/csv-metadata-quality.git $ git clone https://github.com/ilri/csv-metadata-quality.git
$ cd csv-metadata-quality $ cd csv-metadata-quality
$ poetry install $ uv sync
$ poetry shell $ source .venv/bin/activate
``` ```
Otherwise, if you don't have poetry, you can use a vanilla Python virtual environment: Otherwise, if you don't have uv, you can use a vanilla Python virtual environment:
``` ```
$ git clone https://github.com/ilri/csv-metadata-quality.git $ git clone https://github.com/ilri/csv-metadata-quality.git
$ cd csv-metadata-quality $ cd csv-metadata-quality
$ python3 -m venv venv $ python3 -m venv .venv
$ source venv/bin/activate $ source .venv/bin/activate
$ pip install -r requirements.txt $ pip install -r requirements.txt
``` ```
@ -125,7 +125,6 @@ This currently uses the [Python langid](https://github.com/saffsd/langid.py) lib
- Better logging, for example with INFO, WARN, and ERR levels - Better logging, for example with INFO, WARN, and ERR levels
- Verbose, debug, or quiet options - Verbose, debug, or quiet options
- Warn if an author is shorter than 3 characters? - Warn if an author is shorter than 3 characters?
- Validate DOIs? Normalize to https://doi.org format? Or use just the DOI part: 10.1016/j.worlddev.2010.06.006
- Warn if two items use the same file in `filename` column - Warn if two items use the same file in `filename` column
- Add tests for application invocation, ie `tests/test_app.py`? - Add tests for application invocation, ie `tests/test_app.py`?
- Validate ISSNs or journal titles against CrossRef API? - Validate ISSNs or journal titles against CrossRef API?

View File

@ -141,6 +141,11 @@ def run(argv):
# Fix: unnecessary Unicode # Fix: unnecessary Unicode
df[column] = df[column].apply(fix.unnecessary_unicode) df[column] = df[column].apply(fix.unnecessary_unicode)
# Fix: normalize DOIs
match = re.match(r"^.*?identifier\.doi.*$", column)
if match is not None:
df[column] = df[column].apply(fix.normalize_dois)
# Fix: invalid and unnecessary multi-value separators. Skip the title # Fix: invalid and unnecessary multi-value separators. Skip the title
# and abstract fields because "|" is used to indicate something like # and abstract fields because "|" is used to indicate something like
# a subtitle. # a subtitle.

View File

@ -2,7 +2,7 @@
import logging import logging
import re import re
from datetime import datetime, timedelta from datetime import datetime
import country_converter as coco import country_converter as coco
import pandas as pd import pandas as pd
@ -133,7 +133,7 @@ def suspicious_characters(field, field_name):
return return
# List of suspicious characters, for example: ́ˆ~` # List of suspicious characters, for example: ́ˆ~`
suspicious_characters = ["\u00B4", "\u02C6", "\u007E", "\u0060"] suspicious_characters = ["\u00b4", "\u02c6", "\u007e", "\u0060"]
for character in suspicious_characters: for character in suspicious_characters:
# Find the position of the suspicious character in the string # Find the position of the suspicious character in the string

File diff suppressed because it is too large Load Diff

View File

@ -2,8 +2,8 @@
import re import re
import langid
import pandas as pd import pandas as pd
import py3langid as langid
from colorama import Fore from colorama import Fore
from pycountry import languages from pycountry import languages

View File

@ -395,3 +395,88 @@ def countries_match_regions(row, exclude):
row[region_column_name] = "||".join(missing_regions) row[region_column_name] = "||".join(missing_regions)
return row return row
def normalize_dois(field):
"""Normalize DOIs.
DOIs are meant to be globally unique identifiers. They are case insensitive,
but in order to compare them robustly they should be normalized to a common
format:
- strip leading and trailing whitespace
- lowercase all ASCII characters
- convert all variations to https://doi.org/10.xxxx/xxxx URI format
Return string with normalized DOI.
See: https://www.crossref.org/documentation/member-setup/constructing-your-dois/
"""
# Skip fields with missing values
if pd.isna(field):
return
# Try to split multi-value field on "||" separator
values = field.split("||")
# Initialize an empty list to hold the de-duplicated values
new_values = []
# Iterate over all values (most items will only have one DOI)
for value in values:
# Strip leading and trailing whitespace
new_value = value.strip()
new_value = new_value.lower()
# Convert to HTTPS
pattern = re.compile(r"^http://")
match = re.findall(pattern, new_value)
if match:
new_value = re.sub(pattern, "https://", new_value)
# Convert dx.doi.org to doi.org
pattern = re.compile(r"dx\.doi\.org")
match = re.findall(pattern, new_value)
if match:
new_value = re.sub(pattern, "doi.org", new_value)
# Convert www.doi.org to doi.org
pattern = re.compile(r"www\.doi\.org")
match = re.findall(pattern, new_value)
if match:
new_value = re.sub(pattern, "doi.org", new_value)
# Convert erroneous %2f to /
pattern = re.compile("%2f")
match = re.findall(pattern, new_value)
if match:
new_value = re.sub(pattern, "/", new_value)
# Replace values like doi: 10.11648/j.jps.20140201.14
pattern = re.compile(r"^doi: 10\.")
match = re.findall(pattern, new_value)
if match:
new_value = re.sub(pattern, "https://doi.org/10.", new_value)
# Replace values like 10.3390/foods12010115
pattern = re.compile(r"^10\.")
match = re.findall(pattern, new_value)
if match:
new_value = re.sub(pattern, "https://doi.org/10.", new_value)
if new_value != value:
print(f"{Fore.GREEN}Normalized DOI: {Fore.RESET}{value}")
new_values.append(new_value)
new_field = "||".join(new_values)
return new_field

View File

@ -1,3 +1,3 @@
# SPDX-License-Identifier: GPL-3.0-only # SPDX-License-Identifier: GPL-3.0-only
VERSION = "0.6.1" VERSION = "0.7.0"

View File

@ -37,3 +37,7 @@ Mojibake,2021-03-18,,,,Publicaçao CIAT,,,,Report,,,,
Title missing from citation,2021-12-05,,,,,,,,,"Orth, A. 2021. Title missing f rom citation.",,, Title missing from citation,2021-12-05,,,,,,,,,"Orth, A. 2021. Title missing f rom citation.",,,
Country missing region,2021-12-08,,,,,Kenya,,,,,,, Country missing region,2021-12-08,,,,,Kenya,,,,,,,
Subregion field shouldnt trigger region checks,2022-12-07,,,,,Kenya,,,,,,Eastern Africa,Baringo Subregion field shouldnt trigger region checks,2022-12-07,,,,,Kenya,,,,,,Eastern Africa,Baringo
DOI with HTTP and dx.doi.org,2024-04-23,,,,,,,,,,http://dx.doi.org/10.1016/j.envc.2023.100794,,
DOI with colon,2024-04-23,,,,,,,,,,doi: 10.11648/j.jps.20140201.14,,
Upper case bare DOI,2024-04-23,,,,,,,,,,10.19103/AS.2018.0043.16,,
DOI with %2f,2024-06-25,,,,,,,,,,https://doi.org/10.1016%2fj.envc.2023.100794,,

1 dc.title dcterms.issued dc.identifier.issn dc.identifier.isbn dcterms.language dcterms.subject cg.coverage.country filename dcterms.license dcterms.type dcterms.bibliographicCitation cg.identifier.doi cg.coverage.region cg.coverage.subregion
37 Country missing region 2021-12-08 Kenya
38 Subregion field shouldn’t trigger region checks 2022-12-07 Kenya Eastern Africa Baringo
39 DOI with HTTP and dx.doi.org 2024-04-23 http://dx.doi.org/10.1016/j.envc.2023.100794
40 DOI with colon 2024-04-23 doi: 10.11648/j.jps.20140201.14
41 Upper case bare DOI 2024-04-23 10.19103/AS.2018.0043.16
42 DOI with %2f 2024-06-25 https://doi.org/10.1016%2fj.envc.2023.100794
43

1937
poetry.lock generated

File diff suppressed because it is too large Load Diff

View File

@ -1,40 +1,60 @@
[tool.poetry] [project]
name = "csv-metadata-quality" name = "csv-metadata-quality"
version = "0.6.1" version = "0.7.0"
description="A simple, but opinionated CSV quality checking and fixing pipeline for CSVs in the DSpace ecosystem." description="A simple, but opinionated CSV quality checking and fixing pipeline for CSVs in the DSpace ecosystem."
authors = ["Alan Orth <alan.orth@gmail.com>"] authors = [
license="GPL-3.0-only" { name = "Alan Orth", email = "alan.orth@gmail.com" }
]
license= { file = "LICENSE.txt" }
dependencies = [
"pandas[feather,performance]~=2.2.3",
"python-stdnum~=1.20",
"requests~=2.32.3",
"requests-cache~=1.2.1",
"colorama~=0.4",
"ftfy~=6.3.0",
"country-converter~=1.3",
"pycountry~=24.6.1",
"py3langid~=0.3",
]
readme = "README.md"
requires-python = ">= 3.10"
classifiers = [
"License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
"Natural Language :: English",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Programming Language :: Python :: Implementation :: CPython",
]
[project.urls]
repository = "https://github.com/ilri/csv-metadata-quality" repository = "https://github.com/ilri/csv-metadata-quality"
homepage = "https://github.com/ilri/csv-metadata-quality" homepage = "https://github.com/ilri/csv-metadata-quality"
[tool.poetry.scripts] [project.scripts]
csv-metadata-quality = 'csv_metadata_quality.__main__:main' csv-metadata-quality = 'csv_metadata_quality.__main__:main'
[tool.poetry.dependencies] # So uv doesn't fall back to setuptools
python = "^3.9" # See: https://packaging.python.org/en/latest/tutorials/packaging-projects/#choosing-build-backend
pandas = {version = "^2.0.2", extras = ["feather", "performance"]}
python-stdnum = "^1.18"
requests = "^2.28.2"
requests-cache = "^1.0.0"
langid = "^1.1.6"
colorama = "^0.4.6"
ftfy = "^6.1.1"
country-converter = "~1.1.0"
pycountry = {git = "https://github.com/alanorth/pycountry", rev = "iso-codes-4.15.0"}
[tool.poetry.group.dev.dependencies]
pytest = "^7.2.1"
flake8 = "^6.0.0"
pytest-clarity = "^1.0.1"
black = "^23.1.0"
isort = "^5.12.0"
csvkit = "^1.1.0"
ipython = "^8.10.0"
fixit = "^2.1.0"
[build-system] [build-system]
requires = ["poetry>=0.12"] requires = ["hatchling"]
build-backend = "poetry.masonry.api" build-backend = "hatchling.build"
[dependency-groups]
dev = [
"pytest~=8.3",
"isort~=6.0",
"csvkit~=2.0",
"ipython~=8.31",
]
# So hatch doesn't try to build other top-level directories like "data"
[tool.hatch.build.targets.wheel]
packages = ["csv_metadata_quality"]
[tool.isort] [tool.isort]
profile = "black" profile = "black"

View File

@ -1,9 +1,9 @@
{ {
"$schema": "https://docs.renovatebot.com/renovate-schema.json", "$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": [ "extends": [
"config:base" "config:recommended"
], ],
"pip_requirements": { "pep621": {
"enabled": false "enabled": false
} }
} }

View File

@ -1,82 +0,0 @@
agate-dbf==0.2.2 ; python_version >= "3.9" and python_version < "4.0"
agate-excel==0.2.5 ; python_version >= "3.9" and python_version < "4.0"
agate-sql==0.5.9 ; python_version >= "3.9" and python_version < "4.0"
agate==1.7.1 ; python_version >= "3.9" and python_version < "4.0"
appdirs==1.4.4 ; python_version >= "3.9" and python_version < "4.0"
appnope==0.1.3 ; python_version >= "3.9" and python_version < "4.0" and sys_platform == "darwin"
asttokens==2.2.1 ; python_version >= "3.9" and python_version < "4.0"
attrs==23.1.0 ; python_version >= "3.9" and python_version < "4.0"
babel==2.12.1 ; python_version >= "3.9" and python_version < "4.0"
backcall==0.2.0 ; python_version >= "3.9" and python_version < "4.0"
black==23.3.0 ; python_version >= "3.9" and python_version < "4.0"
cattrs==22.2.0 ; python_version >= "3.9" and python_version < "4.0"
certifi==2022.12.7 ; python_version >= "3.9" and python_version < "4.0"
charset-normalizer==3.1.0 ; python_version >= "3.9" and python_version < "4.0"
click==8.1.3 ; python_version >= "3.9" and python_version < "4.0"
colorama==0.4.6 ; python_version >= "3.9" and python_version < "4.0"
country-converter==1.0.0 ; python_version >= "3.9" and python_version < "4.0"
csvkit==1.1.1 ; python_version >= "3.9" and python_version < "4.0"
dbfread==2.0.7 ; python_version >= "3.9" and python_version < "4.0"
decorator==5.1.1 ; python_version >= "3.9" and python_version < "4.0"
et-xmlfile==1.1.0 ; python_version >= "3.9" and python_version < "4.0"
exceptiongroup==1.1.1 ; python_version >= "3.9" and python_version < "3.11"
executing==1.2.0 ; python_version >= "3.9" and python_version < "4.0"
flake8==6.0.0 ; python_version >= "3.9" and python_version < "4.0"
ftfy==6.1.1 ; python_version >= "3.9" and python_version < "4"
greenlet==2.0.2 ; python_version >= "3.9" and platform_machine == "aarch64" and python_version < "4.0" or python_version >= "3.9" and platform_machine == "ppc64le" and python_version < "4.0" or python_version >= "3.9" and platform_machine == "x86_64" and python_version < "4.0" or python_version >= "3.9" and platform_machine == "amd64" and python_version < "4.0" or python_version >= "3.9" and platform_machine == "AMD64" and python_version < "4.0" or python_version >= "3.9" and platform_machine == "win32" and python_version < "4.0" or python_version >= "3.9" and platform_machine == "WIN32" and python_version < "4.0"
idna==3.4 ; python_version >= "3.9" and python_version < "4.0"
iniconfig==2.0.0 ; python_version >= "3.9" and python_version < "4.0"
ipython==8.13.1 ; python_version >= "3.9" and python_version < "4.0"
isodate==0.6.1 ; python_version >= "3.9" and python_version < "4.0"
isort==5.12.0 ; python_version >= "3.9" and python_version < "4.0"
jedi==0.18.2 ; python_version >= "3.9" and python_version < "4.0"
langid==1.1.6 ; python_version >= "3.9" and python_version < "4.0"
leather==0.3.4 ; python_version >= "3.9" and python_version < "4.0"
markdown-it-py==2.2.0 ; python_version >= "3.9" and python_version < "4.0"
matplotlib-inline==0.1.6 ; python_version >= "3.9" and python_version < "4.0"
mccabe==0.7.0 ; python_version >= "3.9" and python_version < "4.0"
mdurl==0.1.2 ; python_version >= "3.9" and python_version < "4.0"
mypy-extensions==1.0.0 ; python_version >= "3.9" and python_version < "4.0"
numpy==1.24.3 ; python_version >= "3.9" and python_version < "4.0"
olefile==0.46 ; python_version >= "3.9" and python_version < "4.0"
openpyxl==3.1.2 ; python_version >= "3.9" and python_version < "4.0"
packaging==23.1 ; python_version >= "3.9" and python_version < "4.0"
pandas==2.0.1 ; python_version >= "3.9" and python_version < "4.0"
parsedatetime==2.6 ; python_version >= "3.9" and python_version < "4.0"
parso==0.8.3 ; python_version >= "3.9" and python_version < "4.0"
pathspec==0.11.1 ; python_version >= "3.9" and python_version < "4.0"
pexpect==4.8.0 ; python_version >= "3.9" and python_version < "4.0" and sys_platform != "win32"
pickleshare==0.7.5 ; python_version >= "3.9" and python_version < "4.0"
platformdirs==3.5.0 ; python_version >= "3.9" and python_version < "4.0"
pluggy==1.0.0 ; python_version >= "3.9" and python_version < "4.0"
pprintpp==0.4.0 ; python_version >= "3.9" and python_version < "4.0"
prompt-toolkit==3.0.38 ; python_version >= "3.9" and python_version < "4.0"
ptyprocess==0.7.0 ; python_version >= "3.9" and python_version < "4.0" and sys_platform != "win32"
pure-eval==0.2.2 ; python_version >= "3.9" and python_version < "4.0"
pyarrow==11.0.0 ; python_version >= "3.9" and python_version < "4.0"
pycodestyle==2.10.0 ; python_version >= "3.9" and python_version < "4.0"
pycountry @ git+https://github.com/alanorth/pycountry@iso-codes-4.13.0 ; python_version >= "3.9" and python_version < "4.0"
pyflakes==3.0.1 ; python_version >= "3.9" and python_version < "4.0"
pygments==2.15.1 ; python_version >= "3.9" and python_version < "4.0"
pytest-clarity==1.0.1 ; python_version >= "3.9" and python_version < "4.0"
pytest==7.3.1 ; python_version >= "3.9" and python_version < "4.0"
python-dateutil==2.8.2 ; python_version >= "3.9" and python_version < "4.0"
python-slugify==8.0.1 ; python_version >= "3.9" and python_version < "4.0"
python-stdnum==1.18 ; python_version >= "3.9" and python_version < "4.0"
pytimeparse==1.1.8 ; python_version >= "3.9" and python_version < "4.0"
pytz==2023.3 ; python_version >= "3.9" and python_version < "4.0"
requests-cache==0.9.8 ; python_version >= "3.9" and python_version < "4.0"
requests==2.29.0 ; python_version >= "3.9" and python_version < "4.0"
rich==13.3.5 ; python_version >= "3.9" and python_version < "4.0"
six==1.16.0 ; python_version >= "3.9" and python_version < "4.0"
sqlalchemy==1.4.48 ; python_version >= "3.9" and python_version < "4.0"
stack-data==0.6.2 ; python_version >= "3.9" and python_version < "4.0"
text-unidecode==1.3 ; python_version >= "3.9" and python_version < "4.0"
tomli==2.0.1 ; python_version >= "3.9" and python_version < "3.11"
traitlets==5.9.0 ; python_version >= "3.9" and python_version < "4.0"
typing-extensions==4.5.0 ; python_version >= "3.9" and python_version < "3.10"
tzdata==2023.3 ; python_version >= "3.9" and python_version < "4.0"
url-normalize==1.4.3 ; python_version >= "3.9" and python_version < "4.0"
urllib3==1.26.15 ; python_version >= "3.9" and python_version < "4.0"
wcwidth==0.2.6 ; python_version >= "3.9" and python_version < "4"
xlrd==2.0.1 ; python_version >= "3.9" and python_version < "4.0"

View File

@ -1,25 +1,356 @@
appdirs==1.4.4 ; python_version >= "3.9" and python_version < "4.0" # This file was autogenerated by uv via the following command:
attrs==23.1.0 ; python_version >= "3.9" and python_version < "4.0" # uv export --no-dev
cattrs==22.2.0 ; python_version >= "3.9" and python_version < "4.0" -e .
certifi==2022.12.7 ; python_version >= "3.9" and python_version < "4.0" attrs==25.1.0 \
charset-normalizer==3.1.0 ; python_version >= "3.9" and python_version < "4.0" --hash=sha256:1c97078a80c814273a76b2a298a932eb681c87415c11dee0a6921de7f1b02c3e \
colorama==0.4.6 ; python_version >= "3.9" and python_version < "4.0" --hash=sha256:c75a69e28a550a7e93789579c22aa26b0f5b83b75dc4e08fe092980051e1090a
country-converter==1.0.0 ; python_version >= "3.9" and python_version < "4.0" bottleneck==1.4.2 \
exceptiongroup==1.1.1 ; python_version >= "3.9" and python_version < "3.11" --hash=sha256:037315c56605128a39f77d19af6a6019dc8c21a63694a4bfef3c026ed963be2e \
ftfy==6.1.1 ; python_version >= "3.9" and python_version < "4" --hash=sha256:070d22f2f62ab81297380a89492cca931e4d9443fa4b84c2baeb52db09c3b1b4 \
idna==3.4 ; python_version >= "3.9" and python_version < "4.0" --hash=sha256:122845e3106c85465551d4a9a3777841347cfedfbebb3aa985cca110e07030b1 \
langid==1.1.6 ; python_version >= "3.9" and python_version < "4.0" --hash=sha256:125436df93751a226eab1732783aa8f6125e88e779587aa61be071fb66e41f9d \
numpy==1.24.3 ; python_version >= "3.9" and python_version < "4.0" --hash=sha256:1f61658ebdf5a178298544336b65020730bf86cc092dab5f6579a99a86bd888b \
pandas==2.0.1 ; python_version >= "3.9" and python_version < "4.0" --hash=sha256:1fc4e7645bd425c05e05acd5541e9e09cb4179e71164e862f082561bf4509eac \
pyarrow==11.0.0 ; python_version >= "3.9" and python_version < "4.0" --hash=sha256:26b5f0531f7044befaad95c20365dd666372e66bdacbfaf009ff65d60285534d \
pycountry @ git+https://github.com/alanorth/pycountry@iso-codes-4.13.0 ; python_version >= "3.9" and python_version < "4.0" --hash=sha256:2c9dbaf737b605b30c81611f2c1d197c2fd2e46c33f605876c1d332d3360c4fc \
python-dateutil==2.8.2 ; python_version >= "3.9" and python_version < "4.0" --hash=sha256:2db287f6ecdbb1c998085eca9b717fec2bfc48a4ab6ae070a9820ba8ab59c90b \
python-stdnum==1.18 ; python_version >= "3.9" and python_version < "4.0" --hash=sha256:2e2fe327dc2d0564e295a5857a252755103f8c6e05b07d3ff80a69afaa9f5065 \
pytz==2023.3 ; python_version >= "3.9" and python_version < "4.0" --hash=sha256:48c6b9d9287c4102b803fcb01ae66ae7ef6b310b711b4b7b7e23bf952894dc05 \
requests-cache==0.9.8 ; python_version >= "3.9" and python_version < "4.0" --hash=sha256:4c6df9a60ec6ab88fec934ca864266ba95edd89c490af71dc9cd8afb2a54ebd9 \
requests==2.29.0 ; python_version >= "3.9" and python_version < "4.0" --hash=sha256:6282fa925ac3768f66e3547f89a512376d3f9de7ef53bdd37aa29232fd864054 \
six==1.16.0 ; python_version >= "3.9" and python_version < "4.0" --hash=sha256:6b7790ca8658cd69e3cc0d0e4ff0e9829d60849bf7945fbd7344fbce05b2bbb8 \
tzdata==2023.3 ; python_version >= "3.9" and python_version < "4.0" --hash=sha256:7363b3c8ce6ca433779cd7e96bcb94c0e516dcacadff0011adcbf0b3ac86bc9d \
url-normalize==1.4.3 ; python_version >= "3.9" and python_version < "4.0" --hash=sha256:7c7d29c044a3511b36fd744503c3e697e279c273a8477a6d91a2831d04fd19e0 \
urllib3==1.26.15 ; python_version >= "3.9" and python_version < "4.0" --hash=sha256:7ebbcbe5d4062e37507b9a81e2aacdb1fcccc6193f7feff124ef2b5a6a5eb740 \
wcwidth==0.2.6 ; python_version >= "3.9" and python_version < "4" --hash=sha256:89651ef18c06616850203bf8875c958c5d316ea48d8ba60d9b450199d39ae391 \
--hash=sha256:964f6ac4118ddab3bbbac79d4f726b093459be751baba73ee0aa364666e8068e \
--hash=sha256:99778329331d5fae8df19772a019e8b73ba4d9d1650f110cd995ab7657114db0 \
--hash=sha256:a74ddd0417f42eeaba37375f0fc065b28451e0fba45cb2f99e88880b10b3fa43 \
--hash=sha256:b6902ebf3e85315b481bc084f10c5770f8240275ad1e039ac69c7c8d2013b040 \
--hash=sha256:c1c885ad02a6a8fa1f7ee9099f29b9d4c03eb1da2c7ab25839482d5cce739021 \
--hash=sha256:c2fd34b9b490204f95288f0dd35d37042486a95029617246c88c0f94a0ab49fe \
--hash=sha256:c663cbba8f52011fd82ee08c6a85c93b34b19e0e7ebba322d2d67809f34e0597 \
--hash=sha256:e56a206fbf48e3b8054a964398bf1ed843e9625d3c6bdbeb7898cb48bf97441b \
--hash=sha256:e7a1b023de1de3d84b18826462718fba548fed41870df44354f9ab6a414ea82f \
--hash=sha256:eb0c611d15b0fd8f511d288e8964e4725b4b3b0d9d310880cf0ff6b8dd03c859 \
--hash=sha256:fa8e8e1799dea5483ce6669462660f9d9a95649f6f98a80d315b84ec89f449f4
cattrs==24.1.2 \
--hash=sha256:67c7495b760168d931a10233f979b28dc04daf853b30752246f4f8471c6d68d0 \
--hash=sha256:8028cfe1ff5382df59dd36474a86e02d817b06eaf8af84555441bac915d2ef85
certifi==2024.12.14 \
--hash=sha256:1275f7a45be9464efc1173084eaa30f866fe2e47d389406136d332ed4967ec56 \
--hash=sha256:b650d30f370c2b724812bee08008be0c4163b163ddaec3f2546c1caf65f191db
charset-normalizer==3.4.1 \
--hash=sha256:0924e81d3d5e70f8126529951dac65c1010cdf117bb75eb02dd12339b57749dd \
--hash=sha256:09b26ae6b1abf0d27570633b2b078a2a20419c99d66fb2823173d73f188ce601 \
--hash=sha256:09b5e6733cbd160dcc09589227187e242a30a49ca5cefa5a7edd3f9d19ed53fd \
--hash=sha256:0f55e69f030f7163dffe9fd0752b32f070566451afe180f99dbeeb81f511ad8d \
--hash=sha256:22e14b5d70560b8dd51ec22863f370d1e595ac3d024cb8ad7d308b4cd95f8313 \
--hash=sha256:234ac59ea147c59ee4da87a0c0f098e9c8d169f4dc2a159ef720f1a61bbe27cd \
--hash=sha256:2369eea1ee4a7610a860d88f268eb39b95cb588acd7235e02fd5a5601773d4fa \
--hash=sha256:237bdbe6159cff53b4f24f397d43c6336c6b0b42affbe857970cefbb620911c8 \
--hash=sha256:28bf57629c75e810b6ae989f03c0828d64d6b26a5e205535585f96093e405ed1 \
--hash=sha256:2967f74ad52c3b98de4c3b32e1a44e32975e008a9cd2a8cc8966d6a5218c5cb2 \
--hash=sha256:2bdfe3ac2e1bbe5b59a1a63721eb3b95fc9b6817ae4a46debbb4e11f6232428d \
--hash=sha256:2d074908e1aecee37a7635990b2c6d504cd4766c7bc9fc86d63f9c09af3fa11b \
--hash=sha256:3a3bd0dcd373514dcec91c411ddb9632c0d7d92aed7093b8c3bbb6d69ca74408 \
--hash=sha256:44251f18cd68a75b56585dd00dae26183e102cd5e0f9f1466e6df5da2ed64ea3 \
--hash=sha256:44ecbf16649486d4aebafeaa7ec4c9fed8b88101f4dd612dcaf65d5e815f837f \
--hash=sha256:4532bff1b8421fd0a320463030c7520f56a79c9024a4e88f01c537316019005a \
--hash=sha256:4d86f7aff21ee58f26dcf5ae81a9addbd914115cdebcbb2217e4f0ed8982e146 \
--hash=sha256:5777ee0881f9499ed0f71cc82cf873d9a0ca8af166dfa0af8ec4e675b7df48e6 \
--hash=sha256:5df196eb874dae23dcfb968c83d4f8fdccb333330fe1fc278ac5ceeb101003a9 \
--hash=sha256:6ff8a4a60c227ad87030d76e99cd1698345d4491638dfa6673027c48b3cd395f \
--hash=sha256:73d94b58ec7fecbc7366247d3b0b10a21681004153238750bb67bd9012414545 \
--hash=sha256:7461baadb4dc00fd9e0acbe254e3d7d2112e7f92ced2adc96e54ef6501c5f176 \
--hash=sha256:804a4d582ba6e5b747c625bf1255e6b1507465494a40a2130978bda7b932c90b \
--hash=sha256:80ed5e856eb7f30115aaf94e4a08114ccc8813e6ed1b5efa74f9f82e8509858f \
--hash=sha256:8417cb1f36cc0bc7eaba8ccb0e04d55f0ee52df06df3ad55259b9a323555fc8b \
--hash=sha256:8436c508b408b82d87dc5f62496973a1805cd46727c34440b0d29d8a2f50a6c9 \
--hash=sha256:8bfa33f4f2672964266e940dd22a195989ba31669bd84629f05fab3ef4e2d125 \
--hash=sha256:91b36a978b5ae0ee86c394f5a54d6ef44db1de0815eb43de826d41d21e4af3de \
--hash=sha256:955f8851919303c92343d2f66165294848d57e9bba6cf6e3625485a70a038d11 \
--hash=sha256:9b23ca7ef998bc739bf6ffc077c2116917eabcc901f88da1b9856b210ef63f35 \
--hash=sha256:9f0b8b1c6d84c8034a44893aba5e767bf9c7a211e313a9605d9c617d7083829f \
--hash=sha256:aabfa34badd18f1da5ec1bc2715cadc8dca465868a4e73a0173466b688f29dda \
--hash=sha256:b010a7a4fd316c3c484d482922d13044979e78d1861f0e0650423144c616a46a \
--hash=sha256:b1ac5992a838106edb89654e0aebfc24f5848ae2547d22c2c3f66454daa11971 \
--hash=sha256:bc2722592d8998c870fa4e290c2eec2c1569b87fe58618e67d38b4665dfa680d \
--hash=sha256:c0429126cf75e16c4f0ad00ee0eae4242dc652290f940152ca8c75c3a4b6ee8f \
--hash=sha256:c30197aa96e8eed02200a83fba2657b4c3acd0f0aa4bdc9f6c1af8e8962e0757 \
--hash=sha256:c4c3e6da02df6fa1410a7680bd3f63d4f710232d3139089536310d027950696a \
--hash=sha256:c75cb2a3e389853835e84a2d8fb2b81a10645b503eca9bcb98df6b5a43eb8886 \
--hash=sha256:c96836c97b1238e9c9e3fe90844c947d5afbf4f4c92762679acfe19927d81d77 \
--hash=sha256:d7f50a1f8c450f3925cb367d011448c39239bb3eb4117c36a6d354794de4ce76 \
--hash=sha256:d973f03c0cb71c5ed99037b870f2be986c3c05e63622c017ea9816881d2dd247 \
--hash=sha256:d98b1668f06378c6dbefec3b92299716b931cd4e6061f3c875a71ced1780ab85 \
--hash=sha256:d9c3cdf5390dcd29aa8056d13e8e99526cda0305acc038b96b30352aff5ff2bb \
--hash=sha256:dad3e487649f498dd991eeb901125411559b22e8d7ab25d3aeb1af367df5efd7 \
--hash=sha256:e218488cd232553829be0664c2292d3af2eeeb94b32bea483cf79ac6a694e037 \
--hash=sha256:e358e64305fe12299a08e08978f51fc21fac060dcfcddd95453eabe5b93ed0e1 \
--hash=sha256:eab677309cdb30d047996b36d34caeda1dc91149e4fdca0b1a039b3f79d9a807 \
--hash=sha256:eb8178fe3dba6450a3e024e95ac49ed3400e506fd4e9e5c32d30adda88cbd407 \
--hash=sha256:eea6ee1db730b3483adf394ea72f808b6e18cf3cb6454b4d86e04fa8c4327a12 \
--hash=sha256:f08ff5e948271dc7e18a35641d2f11a4cd8dfd5634f55228b691e62b37125eb3 \
--hash=sha256:fa88b843d6e211393a37219e6a1c1df99d35e8fd90446f1118f4216e307e48cd \
--hash=sha256:fd4ec41f914fa74ad1b8304bbc634b3de73d2a0889bd32076342a573e0779e00 \
--hash=sha256:ffc9202a29ab3920fa812879e95a9e78b2465fd10be7fcbd042899695d75e616
colorama==0.4.6 \
--hash=sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44 \
--hash=sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6
country-converter==1.3 \
--hash=sha256:006958c83adeada455d2f178921fdd051def736259ff250fada912eaf3ca8cf1 \
--hash=sha256:f6a1a14d1f98112ca90a5198f645f4e60bb73840e98f3f733893ff5b617c2f38
exceptiongroup==1.2.2 ; python_full_version < '3.11' \
--hash=sha256:3111b9d131c238bec2f8f516e123e14ba243563fb135d3fe885990585aa7795b \
--hash=sha256:47c2edf7c6738fafb49fd34290706d1a1a2f4d1c6df275526b62cbb4aa5393cc
ftfy==6.3.1 \
--hash=sha256:7c70eb532015cd2f9adb53f101fb6c7945988d023a085d127d1573dc49dd0083 \
--hash=sha256:9b3c3d90f84fb267fe64d375a07b7f8912d817cf86009ae134aa03e1819506ec
idna==3.10 \
--hash=sha256:12f65c9b470abda6dc35cf8e63cc574b1c52b11df2c86030af0ac09b01b13ea9 \
--hash=sha256:946d195a0d259cbba61165e88e65941f16e9b36ea6ddb97f00452bae8b1287d3
llvmlite==0.44.0 \
--hash=sha256:07667d66a5d150abed9157ab6c0b9393c9356f229784a4385c02f99e94fc94d4 \
--hash=sha256:1d671a56acf725bf1b531d5ef76b86660a5ab8ef19bb6a46064a705c6ca80aad \
--hash=sha256:2fb7c4f2fb86cbae6dca3db9ab203eeea0e22d73b99bc2341cdf9de93612e930 \
--hash=sha256:319bddd44e5f71ae2689859b7203080716448a3cd1128fb144fe5c055219d516 \
--hash=sha256:40526fb5e313d7b96bda4cbb2c85cd5374e04d80732dd36a282d72a560bb6408 \
--hash=sha256:41e3839150db4330e1b2716c0be3b5c4672525b4c9005e17c7597f835f351ce2 \
--hash=sha256:46224058b13c96af1365290bdfebe9a6264ae62fb79b2b55693deed11657a8bf \
--hash=sha256:5f79a728e0435493611c9f405168682bb75ffd1fbe6fc360733b850c80a026db \
--hash=sha256:7202b678cdf904823c764ee0fe2dfe38a76981f4c1e51715b4cb5abb6cf1d9e8 \
--hash=sha256:9c58867118bad04a0bb22a2e0068c693719658105e40009ffe95c7000fcde88e \
--hash=sha256:9fbadbfba8422123bab5535b293da1cf72f9f478a65645ecd73e781f962ca614 \
--hash=sha256:aa0097052c32bf721a4efc03bd109d335dfa57d9bffb3d4c24cc680711b8b4fc \
--hash=sha256:ace564d9fa44bb91eb6e6d8e7754977783c68e90a471ea7ce913bff30bd62427 \
--hash=sha256:c0143a5ef336da14deaa8ec26c5449ad5b6a2b564df82fcef4be040b9cacfea9 \
--hash=sha256:c5d22c3bfc842668168a786af4205ec8e3ad29fb1bc03fd11fd48460d0df64c1 \
--hash=sha256:cccf8eb28f24840f2689fb1a45f9c0f7e582dd24e088dcf96e424834af11f791 \
--hash=sha256:d752f89e31b66db6f8da06df8b39f9b91e78c5feea1bf9e8c1fba1d1c24c065d \
--hash=sha256:d8489634d43c20cd0ad71330dde1d5bc7b9966937a263ff1ec1cebb90dc50955 \
--hash=sha256:eae7e2d4ca8f88f89d315b48c6b741dcb925d6a1042da694aa16ab3dd4cbd3a1 \
--hash=sha256:eed7d5f29136bda63b6d7804c279e2b72e08c952b7c5df61f45db408e0ee52f3 \
--hash=sha256:f01a394e9c9b7b1d4e63c327b096d10f6f0ed149ef53d38a09b3749dcf8c9610
numba==0.61.0 \
--hash=sha256:074cd38c5b1f9c65a4319d1f3928165f48975ef0537ad43385b2bd908e6e2e35 \
--hash=sha256:0ebbd4827091384ab8c4615ba1b3ca8bc639a3a000157d9c37ba85d34cd0da1b \
--hash=sha256:152146ecdbb8d8176f294e9f755411e6f270103a11c3ff50cecc413f794e52c8 \
--hash=sha256:21c2fe25019267a608e2710a6a947f557486b4b0478b02e45a81cf606a05a7d4 \
--hash=sha256:43aa4d7d10c542d3c78106b8481e0cbaaec788c39ee8e3d7901682748ffdf0b4 \
--hash=sha256:44240e694d4aa321430c97b21453e46014fe6c7b8b7d932afa7f6a88cc5d7e5e \
--hash=sha256:46c5ae094fb3706f5adf9021bfb7fc11e44818d61afee695cdee4eadfed45e98 \
--hash=sha256:550d389573bc3b895e1ccb18289feea11d937011de4d278b09dc7ed585d1cdcb \
--hash=sha256:5cafa6095716fcb081618c28a8d27bf7c001e09696f595b41836dec114be2905 \
--hash=sha256:5f6c452dca1de8e60e593f7066df052dd8da09b243566ecd26d2b796e5d3087d \
--hash=sha256:6fb74e81aa78a2303e30593d8331327dfc0d2522b5db05ac967556a26db3ef87 \
--hash=sha256:74250b26ed6a1428763e774dc5b2d4e70d93f73795635b5412b8346a4d054574 \
--hash=sha256:764f0e47004f126f58c3b28e0a02374c420a9d15157b90806d68590f5c20cc89 \
--hash=sha256:888d2e89b8160899e19591467e8fdd4970e07606e1fbc248f239c89818d5f925 \
--hash=sha256:9cab9783a700fa428b1a54d65295122bc03b3de1d01fb819a6b9dbbddfdb8c43 \
--hash=sha256:9f25f7fef0206d55c1cfb796ad833cbbc044e2884751e56e798351280038484c \
--hash=sha256:b72bbc8708e98b3741ad0c63f9929c47b623cc4ee86e17030a4f3e301e8401ac \
--hash=sha256:b96fafbdcf6f69b69855273e988696aae4974115a815f6818fef4af7afa1f6b8 \
--hash=sha256:bf64c2d0f3d161af603de3825172fb83c2600bcb1d53ae8ea568d4c53ba6ac08 \
--hash=sha256:de5aa7904741425f28e1028b85850b31f0a245e9eb4f7c38507fb893283a066c \
--hash=sha256:ffe9fe373ed30638d6e20a0269f817b2c75d447141f55a675bfcf2d1fe2e87fb
numexpr==2.10.2 \
--hash=sha256:0db5ff5183935d1612653559c319922143e8fa3019007696571b13135f216458 \
--hash=sha256:15f59655458056fdb3a621b1bb8e071581ccf7e823916c7568bb7c9a3e393025 \
--hash=sha256:3bf01ec502d89944e49e9c1b5cc7c7085be8ca2eb9dd46a0eafd218afbdbd5f5 \
--hash=sha256:3fc2b8035a0c2cdc352e58c3875cb668836018065cbf5752cb531015d9a568d8 \
--hash=sha256:4213a92efa9770bc28e3792134e27c7e5c7e97068bdfb8ba395baebbd12f991b \
--hash=sha256:5191ba8f2975cb9703afc04ae845a929e193498c0e8bcd408ecb147b35978470 \
--hash=sha256:57b59cbb5dcce4edf09cd6ce0b57ff60312479930099ca8d944c2fac896a1ead \
--hash=sha256:5b3f814437d5a10797f8d89d2037cca2c9d9fa578520fc911f894edafed6ea3e \
--hash=sha256:6b360eb8d392483410fe6a3d5a7144afa298c9a0aa3e9fe193e89590b47dd477 \
--hash=sha256:81d1dde7dd6166d8ff5727bb46ab42a6b0048db0e97ceb84a121334a404a800f \
--hash=sha256:83fcb11988b57cc25b028a36d285287d706d1f536ebf2662ea30bd990e0de8b9 \
--hash=sha256:9309f2e43fe6e4560699ef5c27d7a848b3ff38549b6b57194207cf0e88900527 \
--hash=sha256:97298b14f0105a794bea06fd9fbc5c423bd3ff4d88cbc618860b83eb7a436ad6 \
--hash=sha256:a37d6a51ec328c561b2ca8a2bef07025642eca995b8553a5267d0018c732976d \
--hash=sha256:a42963bd4c62d8afa4f51e7974debfa39a048383f653544ab54f50a2f7ec6c42 \
--hash=sha256:b0aff6b48ebc99d2f54f27b5f73a58cb92fde650aeff1b397c71c8788b4fff1a \
--hash=sha256:b5323a46e75832334f1af86da1ef6ff0add00fbacdd266250be872b438bdf2be \
--hash=sha256:b5b0e82d2109c1d9e63fcd5ea177d80a11b881157ab61178ddbdebd4c561ea46 \
--hash=sha256:ba85371c9a8d03e115f4dfb6d25dfbce05387002b9bc85016af939a1da9624f0 \
--hash=sha256:cb845b2d4f9f8ef0eb1c9884f2b64780a85d3b5ae4eeb26ae2b0019f489cd35e \
--hash=sha256:ce8cccf944339051e44a49a124a06287fe3066d0acbff33d1aa5aee10a96abb7 \
--hash=sha256:d7a3fc83c959288544db3adc70612475d8ad53a66c69198105c74036182d10dd \
--hash=sha256:d9a42f5c24880350d88933c4efee91b857c378aaea7e8b86221fff569069841e \
--hash=sha256:deb64235af9eeba59fcefa67e82fa80cfc0662e1b0aa373b7118a28da124d51d \
--hash=sha256:e2d0ae24b0728e4bc3f1d3f33310340d67321d36d6043f7ce26897f4f1042db0 \
--hash=sha256:ebb73b93f5c4d6994f357fa5a47a9f7a5485577e633b3c46a603cb01445bbb19 \
--hash=sha256:ebdbef5763ca057eea0c2b5698e4439d084a0505d9d6e94f4804f26e8890c45e \
--hash=sha256:ec04c9a3c050c175348801e27c18c68d28673b7bfb865ef88ce333be523bbc01 \
--hash=sha256:f9d7805ccb6be2d3b0f7f6fad3707a09ac537811e8e9964f4074d28cb35543db
numpy==2.1.3 \
--hash=sha256:016d0f6f5e77b0f0d45d77387ffa4bb89816b57c835580c3ce8e099ef830befe \
--hash=sha256:02135ade8b8a84011cbb67dc44e07c58f28575cf9ecf8ab304e51c05528c19f0 \
--hash=sha256:08788d27a5fd867a663f6fc753fd7c3ad7e92747efc73c53bca2f19f8bc06f48 \
--hash=sha256:0d30c543f02e84e92c4b1f415b7c6b5326cbe45ee7882b6b77db7195fb971e3a \
--hash=sha256:0fa14563cc46422e99daef53d725d0c326e99e468a9320a240affffe87852564 \
--hash=sha256:13138eadd4f4da03074851a698ffa7e405f41a0845a6b1ad135b81596e4e9958 \
--hash=sha256:14e253bd43fc6b37af4921b10f6add6925878a42a0c5fe83daee390bca80bc17 \
--hash=sha256:15cb89f39fa6d0bdfb600ea24b250e5f1a3df23f901f51c8debaa6a5d122b2f0 \
--hash=sha256:17ee83a1f4fef3c94d16dc1802b998668b5419362c8a4f4e8a491de1b41cc3ee \
--hash=sha256:2312b2aa89e1f43ecea6da6ea9a810d06aae08321609d8dc0d0eda6d946a541b \
--hash=sha256:2564fbdf2b99b3f815f2107c1bbc93e2de8ee655a69c261363a1172a79a257d4 \
--hash=sha256:3522b0dfe983a575e6a9ab3a4a4dfe156c3e428468ff08ce582b9bb6bd1d71d4 \
--hash=sha256:4394bc0dbd074b7f9b52024832d16e019decebf86caf909d94f6b3f77a8ee3b6 \
--hash=sha256:45966d859916ad02b779706bb43b954281db43e185015df6eb3323120188f9e4 \
--hash=sha256:4d1167c53b93f1f5d8a139a742b3c6f4d429b54e74e6b57d0eff40045187b15d \
--hash=sha256:4f2015dfe437dfebbfce7c85c7b53d81ba49e71ba7eadbf1df40c915af75979f \
--hash=sha256:50ca6aba6e163363f132b5c101ba078b8cbd3fa92c7865fd7d4d62d9779ac29f \
--hash=sha256:50d18c4358a0a8a53f12a8ba9d772ab2d460321e6a93d6064fc22443d189853f \
--hash=sha256:5641516794ca9e5f8a4d17bb45446998c6554704d888f86df9b200e66bdcce56 \
--hash=sha256:576a1c1d25e9e02ed7fa5477f30a127fe56debd53b8d2c89d5578f9857d03ca9 \
--hash=sha256:6a4825252fcc430a182ac4dee5a505053d262c807f8a924603d411f6718b88fd \
--hash=sha256:72dcc4a35a8515d83e76b58fdf8113a5c969ccd505c8a946759b24e3182d1f23 \
--hash=sha256:747641635d3d44bcb380d950679462fae44f54b131be347d5ec2bce47d3df9ed \
--hash=sha256:762479be47a4863e261a840e8e01608d124ee1361e48b96916f38b119cfda04a \
--hash=sha256:78574ac2d1a4a02421f25da9559850d59457bac82f2b8d7a44fe83a64f770098 \
--hash=sha256:825656d0743699c529c5943554d223c021ff0494ff1442152ce887ef4f7561a1 \
--hash=sha256:8637dcd2caa676e475503d1f8fdb327bc495554e10838019651b76d17b98e512 \
--hash=sha256:96fe52fcdb9345b7cd82ecd34547fca4321f7656d500eca497eb7ea5a926692f \
--hash=sha256:973faafebaae4c0aaa1a1ca1ce02434554d67e628b8d805e61f874b84e136b09 \
--hash=sha256:996bb9399059c5b82f76b53ff8bb686069c05acc94656bb259b1d63d04a9506f \
--hash=sha256:a38c19106902bb19351b83802531fea19dee18e5b37b36454f27f11ff956f7fc \
--hash=sha256:a6b46587b14b888e95e4a24d7b13ae91fa22386c199ee7b418f449032b2fa3b8 \
--hash=sha256:a9f7f672a3388133335589cfca93ed468509cb7b93ba3105fce780d04a6576a0 \
--hash=sha256:aa08e04e08aaf974d4458def539dece0d28146d866a39da5639596f4921fd761 \
--hash=sha256:b0df3635b9c8ef48bd3be5f862cf71b0a4716fa0e702155c45067c6b711ddcef \
--hash=sha256:b47fbb433d3260adcd51eb54f92a2ffbc90a4595f8970ee00e064c644ac788f5 \
--hash=sha256:baed7e8d7481bfe0874b566850cb0b85243e982388b7b23348c6db2ee2b2ae8e \
--hash=sha256:bc6f24b3d1ecc1eebfbf5d6051faa49af40b03be1aaa781ebdadcbc090b4539b \
--hash=sha256:c006b607a865b07cd981ccb218a04fc86b600411d83d6fc261357f1c0966755d \
--hash=sha256:c181ba05ce8299c7aa3125c27b9c2167bca4a4445b7ce73d5febc411ca692e43 \
--hash=sha256:c7662f0e3673fe4e832fe07b65c50342ea27d989f92c80355658c7f888fcc83c \
--hash=sha256:c80e4a09b3d95b4e1cac08643f1152fa71a0a821a2d4277334c88d54b2219a41 \
--hash=sha256:c894b4305373b9c5576d7a12b473702afdf48ce5369c074ba304cc5ad8730dff \
--hash=sha256:d7aac50327da5d208db2eec22eb11e491e3fe13d22653dce51b0f4109101b408 \
--hash=sha256:d89dd2b6da69c4fff5e39c28a382199ddedc3a5be5390115608345dec660b9e2 \
--hash=sha256:d9beb777a78c331580705326d2367488d5bc473b49a9bc3036c154832520aca9 \
--hash=sha256:dc258a761a16daa791081d026f0ed4399b582712e6fc887a95af09df10c5ca57 \
--hash=sha256:e14e26956e6f1696070788252dcdff11b4aca4c3e8bd166e0df1bb8f315a67cb \
--hash=sha256:e6988e90fcf617da2b5c78902fe8e668361b43b4fe26dbf2d7b0f8034d4cafb9 \
--hash=sha256:e711e02f49e176a01d0349d82cb5f05ba4db7d5e7e0defd026328e5cfb3226d3 \
--hash=sha256:ea4dedd6e394a9c180b33c2c872b92f7ce0f8e7ad93e9585312b0c5a04777a4a \
--hash=sha256:ecc76a9ba2911d8d37ac01de72834d8849e55473457558e12995f4cd53e778e0 \
--hash=sha256:f55ba01150f52b1027829b50d70ef1dafd9821ea82905b63936668403c3b471e \
--hash=sha256:f653490b33e9c3a4c1c01d41bc2aef08f9475af51146e4a7710c450cf9761598 \
--hash=sha256:fa2d1337dc61c8dc417fbccf20f6d1e139896a30721b7f1e832b2bb6ef4eb6c4
pandas==2.2.3 \
--hash=sha256:062309c1b9ea12a50e8ce661145c6aab431b1e99530d3cd60640e255778bd43a \
--hash=sha256:15c0e1e02e93116177d29ff83e8b1619c93ddc9c49083f237d4312337a61165d \
--hash=sha256:1948ddde24197a0f7add2bdc4ca83bf2b1ef84a1bc8ccffd95eda17fd836ecb5 \
--hash=sha256:1db71525a1538b30142094edb9adc10be3f3e176748cd7acc2240c2f2e5aa3a4 \
--hash=sha256:22a9d949bfc9a502d320aa04e5d02feab689d61da4e7764b62c30b991c42c5f0 \
--hash=sha256:29401dbfa9ad77319367d36940cd8a0b3a11aba16063e39632d98b0e931ddf32 \
--hash=sha256:3508d914817e153ad359d7e069d752cdd736a247c322d932eb89e6bc84217f28 \
--hash=sha256:37e0aced3e8f539eccf2e099f65cdb9c8aa85109b0be6e93e2baff94264bdc6f \
--hash=sha256:381175499d3802cde0eabbaf6324cce0c4f5d52ca6f8c377c29ad442f50f6348 \
--hash=sha256:38cf8125c40dae9d5acc10fa66af8ea6fdf760b2714ee482ca691fc66e6fcb18 \
--hash=sha256:3b71f27954685ee685317063bf13c7709a7ba74fc996b84fc6821c59b0f06468 \
--hash=sha256:3fc6873a41186404dad67245896a6e440baacc92f5b716ccd1bc9ed2995ab2c5 \
--hash=sha256:4f18ba62b61d7e192368b84517265a99b4d7ee8912f8708660fb4a366cc82667 \
--hash=sha256:56534ce0746a58afaf7942ba4863e0ef81c9c50d3f0ae93e9497d6a41a057645 \
--hash=sha256:59ef3764d0fe818125a5097d2ae867ca3fa64df032331b7e0917cf5d7bf66b13 \
--hash=sha256:5de54125a92bb4d1c051c0659e6fcb75256bf799a732a87184e5ea503965bce3 \
--hash=sha256:61c5ad4043f791b61dd4752191d9f07f0ae412515d59ba8f005832a532f8736d \
--hash=sha256:6374c452ff3ec675a8f46fd9ab25c4ad0ba590b71cf0656f8b6daa5202bca3fb \
--hash=sha256:63cc132e40a2e084cf01adf0775b15ac515ba905d7dcca47e9a251819c575ef3 \
--hash=sha256:66108071e1b935240e74525006034333f98bcdb87ea116de573a6a0dccb6c039 \
--hash=sha256:6dfcb5ee8d4d50c06a51c2fffa6cff6272098ad6540aed1a76d15fb9318194d8 \
--hash=sha256:7c2875855b0ff77b2a64a0365e24455d9990730d6431b9e0ee18ad8acee13dbd \
--hash=sha256:800250ecdadb6d9c78eae4990da62743b857b470883fa27f652db8bdde7f6659 \
--hash=sha256:86976a1c5b25ae3f8ccae3a5306e443569ee3c3faf444dfd0f41cda24667ad57 \
--hash=sha256:a5a1595fe639f5988ba6a8e5bc9649af3baf26df3998a0abe56c02609392e0a4 \
--hash=sha256:ad5b65698ab28ed8d7f18790a0dc58005c7629f227be9ecc1072aa74c0c1d43a \
--hash=sha256:b1d432e8d08679a40e2a6d8b2f9770a5c21793a6f9f47fdd52c5ce1948a5a8a9 \
--hash=sha256:b8661b0238a69d7aafe156b7fa86c44b881387509653fdf857bebc5e4008ad42 \
--hash=sha256:ba96630bc17c875161df3818780af30e43be9b166ce51c9a18c1feae342906c2 \
--hash=sha256:c124333816c3a9b03fbeef3a9f230ba9a737e9e5bb4060aa2107a86cc0a497fc \
--hash=sha256:cd8d0c3be0515c12fed0bdbae072551c8b54b7192c7b1fda0ba56059a0179698 \
--hash=sha256:d9c45366def9a3dd85a6454c0e7908f2b3b8e9c138f5dc38fed7ce720d8453ed \
--hash=sha256:f00d1345d84d8c86a63e476bb4955e46458b304b9575dcf71102b5c705320015 \
--hash=sha256:f3a255b2c19987fbbe62a9dfd6cff7ff2aa9ccab3fc75218fd4b7530f01efa24 \
--hash=sha256:fffb8ae78d8af97f849404f21411c95062db1496aeb3e56f146f0355c9989319
platformdirs==4.3.6 \
--hash=sha256:357fb2acbc885b0419afd3ce3ed34564c13c9b95c89360cd9563f73aa5e2b907 \
--hash=sha256:73e575e1408ab8103900836b97580d5307456908a03e92031bab39e4554cc3fb
py3langid==0.3.0 \
--hash=sha256:0a875a031a58aaf9dbda7bb8285fd75e801a7bd276216ffabe037901d4b449ec \
--hash=sha256:38f022eec31cf9a2bf6f142acb2a9b350fd7d0d5ae7762b1392c6d3567401fd3
pyarrow==19.0.0 \
--hash=sha256:239ca66d9a05844bdf5af128861af525e14df3c9591bcc05bac25918e650d3a2 \
--hash=sha256:2795064647add0f16563e57e3d294dbfc067b723f0fd82ecd80af56dad15f503 \
--hash=sha256:29cd86c8001a94f768f79440bf83fee23963af5e7bc68ce3a7e5f120e17edf89 \
--hash=sha256:2a0144a712d990d60f7f42b7a31f0acaccf4c1e43e957f7b1ad58150d6f639c1 \
--hash=sha256:2a1a109dfda558eb011e5f6385837daffd920d54ca00669f7a11132d0b1e6042 \
--hash=sha256:2b6d3ce4288793350dc2d08d1e184fd70631ea22a4ff9ea5c4ff182130249d9b \
--hash=sha256:2f672f5364b2d7829ef7c94be199bb88bf5661dd485e21d2d37de12ccb78a136 \
--hash=sha256:450a7d27e840e4d9a384b5c77199d489b401529e75a3b7a3799d4cd7957f2f9c \
--hash=sha256:4624c89d6f777c580e8732c27bb8e77fd1433b89707f17c04af7635dd9638351 \
--hash=sha256:4d8b0c0de0a73df1f1bf439af1b60f273d719d70648e898bc077547649bb8352 \
--hash=sha256:5418d4d0fab3a0ed497bad21d17a7973aad336d66ad4932a3f5f7480d4ca0c04 \
--hash=sha256:5e8a28b918e2e878c918f6d89137386c06fe577cd08d73a6be8dafb317dc2d73 \
--hash=sha256:62ef8360ff256e960f57ce0299090fb86423afed5e46f18f1225f960e05aae3d \
--hash=sha256:66732e39eaa2247996a6b04c8aa33e3503d351831424cdf8d2e9a0582ac54b34 \
--hash=sha256:8d47c691765cf497aaeed4954d226568563f1b3b74ff61139f2d77876717084b \
--hash=sha256:8e3a839bf36ec03b4315dc924d36dcde5444a50066f1c10f8290293c0427b46a \
--hash=sha256:9348a0137568c45601b031a8d118275069435f151cbb77e6a08a27e8125f59d4 \
--hash=sha256:a08e2a8a039a3f72afb67a6668180f09fddaa38fe0d21f13212b4aba4b5d2451 \
--hash=sha256:a218670b26fb1bc74796458d97bcab072765f9b524f95b2fccad70158feb8b17 \
--hash=sha256:a22a4bc0937856263df8b94f2f2781b33dd7f876f787ed746608e06902d691a5 \
--hash=sha256:a7bbe7109ab6198688b7079cbad5a8c22de4d47c4880d8e4847520a83b0d1b68 \
--hash=sha256:a92aff08e23d281c69835e4a47b80569242a504095ef6a6223c1f6bb8883431d \
--hash=sha256:b34d3bde38eba66190b215bae441646330f8e9da05c29e4b5dd3e41bde701098 \
--hash=sha256:b903afaa5df66d50fc38672ad095806443b05f202c792694f3a604ead7c6ea6e \
--hash=sha256:be686bf625aa7b9bada18defb3a3ea3981c1099697239788ff111d87f04cd263 \
--hash=sha256:c318eda14f6627966997a7d8c374a87d084a94e4e38e9abbe97395c215830e0c \
--hash=sha256:c3b78eff5968a1889a0f3bc81ca57e1e19b75f664d9c61a42a604bf9d8402aae \
--hash=sha256:c751c1c93955b7a84c06794df46f1cec93e18610dcd5ab7d08e89a81df70a849 \
--hash=sha256:ce42275097512d9e4e4a39aade58ef2b3798a93aa3026566b7892177c266f735 \
--hash=sha256:cf3bf0ce511b833f7bc5f5bb3127ba731e97222023a444b7359f3a22e2a3b463 \
--hash=sha256:e675a3ad4732b92d72e4d24009707e923cab76b0d088e5054914f11a797ebe44 \
--hash=sha256:e82c3d5e44e969c217827b780ed8faf7ac4c53f934ae9238872e749fa531f7c9 \
--hash=sha256:f094742275586cdd6b1a03655ccff3b24b2610c3af76f810356c4c71d24a2a6c \
--hash=sha256:f208c3b58a6df3b239e0bb130e13bc7487ed14f39a9ff357b6415e3f6339b560 \
--hash=sha256:f43f5aef2a13d4d56adadae5720d1fed4c1356c993eda8b59dace4b5983843c1
pycountry==24.6.1 \
--hash=sha256:b61b3faccea67f87d10c1f2b0fc0be714409e8fcdcc1315613174f6466c10221 \
--hash=sha256:f1a4fb391cd7214f8eefd39556d740adcc233c778a27f8942c8dca351d6ce06f
python-dateutil==2.9.0.post0 \
--hash=sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3 \
--hash=sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427
python-stdnum==1.20 \
--hash=sha256:111008e10391d54fb2afad2a10df70d5cb0c6c0a7ec82fec6f022cb8712961d3 \
--hash=sha256:ad2a2cf2eb025de408210235f36b4ae31252de3186240ccaa8126e117cb82690
pytz==2024.2 \
--hash=sha256:2aa355083c50a0f93fa581709deac0c9ad65cca8a9e9beac660adcbd493c798a \
--hash=sha256:31c7c1817eb7fae7ca4b8c7ee50c72f93aa2dd863de768e1ef4245d426aa0725
requests==2.32.3 \
--hash=sha256:55365417734eb18255590a9ff9eb97e9e1da868d4ccd6402399eaf68af20a760 \
--hash=sha256:70761cfe03c773ceb22aa2f671b4757976145175cdfca038c02654d061d6dcc6
requests-cache==1.2.1 \
--hash=sha256:1285151cddf5331067baa82598afe2d47c7495a1334bfe7a7d329b43e9fd3603 \
--hash=sha256:68abc986fdc5b8d0911318fbb5f7c80eebcd4d01bfacc6685ecf8876052511d1
six==1.17.0 \
--hash=sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274 \
--hash=sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81
typing-extensions==4.12.2 ; python_full_version < '3.11' \
--hash=sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d \
--hash=sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8
tzdata==2025.1 \
--hash=sha256:24894909e88cdb28bd1636c6887801df64cb485bd593f2fd83ef29075a81d694 \
--hash=sha256:7e127113816800496f027041c570f50bcd464a020098a3b6b199517772303639
url-normalize==1.4.3 \
--hash=sha256:d23d3a070ac52a67b83a1c59a0e68f8608d1cd538783b401bc9de2c0fac999b2 \
--hash=sha256:ec3c301f04e5bb676d333a7fa162fa977ad2ca04b7e652bfc9fac4e405728eed
urllib3==2.3.0 \
--hash=sha256:1cee9ad369867bfdbbb48b7dd50374c0967a0bb7710050facf0dd6911440e3df \
--hash=sha256:f8c5449b3cf0861679ce7e0503c7b44b5ec981bec0d1d3795a07f1ba96f0204d
wcwidth==0.2.13 \
--hash=sha256:3da69048e4540d84af32131829ff948f1e022c1c6bdb8d6102117aac784f6859 \
--hash=sha256:72ea0c06399eb286d978fdedb6923a9eb47e1c486ce63e9b4e64fc18303972b5

View File

@ -257,7 +257,7 @@ def test_check_incorrect_iso_639_1_language(capsys):
title = "A randomised vaccine field trial in Kenya demonstrates protection against wildebeest-associated malignant catarrhal fever in cattle" title = "A randomised vaccine field trial in Kenya demonstrates protection against wildebeest-associated malignant catarrhal fever in cattle"
language = "es" language = "es"
exclude = list() exclude = []
# Create a dictionary to mimic Pandas series # Create a dictionary to mimic Pandas series
row = {"dc.title": title, "dc.language.iso": language} row = {"dc.title": title, "dc.language.iso": language}
@ -277,7 +277,7 @@ def test_check_incorrect_iso_639_3_language(capsys):
title = "A randomised vaccine field trial in Kenya demonstrates protection against wildebeest-associated malignant catarrhal fever in cattle" title = "A randomised vaccine field trial in Kenya demonstrates protection against wildebeest-associated malignant catarrhal fever in cattle"
language = "spa" language = "spa"
exclude = list() exclude = []
# Create a dictionary to mimic Pandas series # Create a dictionary to mimic Pandas series
row = {"dc.title": title, "dc.language.iso": language} row = {"dc.title": title, "dc.language.iso": language}
@ -297,7 +297,7 @@ def test_check_correct_iso_639_1_language():
title = "A randomised vaccine field trial in Kenya demonstrates protection against wildebeest-associated malignant catarrhal fever in cattle" title = "A randomised vaccine field trial in Kenya demonstrates protection against wildebeest-associated malignant catarrhal fever in cattle"
language = "en" language = "en"
exclude = list() exclude = []
# Create a dictionary to mimic Pandas series # Create a dictionary to mimic Pandas series
row = {"dc.title": title, "dc.language.iso": language} row = {"dc.title": title, "dc.language.iso": language}
@ -313,7 +313,7 @@ def test_check_correct_iso_639_3_language():
title = "A randomised vaccine field trial in Kenya demonstrates protection against wildebeest-associated malignant catarrhal fever in cattle" title = "A randomised vaccine field trial in Kenya demonstrates protection against wildebeest-associated malignant catarrhal fever in cattle"
language = "eng" language = "eng"
exclude = list() exclude = []
# Create a dictionary to mimic Pandas series # Create a dictionary to mimic Pandas series
row = {"dc.title": title, "dc.language.iso": language} row = {"dc.title": title, "dc.language.iso": language}
@ -407,7 +407,7 @@ def test_check_doi_field():
# the citation and a DOI field. # the citation and a DOI field.
d = {"cg.identifier.doi": doi, "dcterms.bibliographicCitation": citation} d = {"cg.identifier.doi": doi, "dcterms.bibliographicCitation": citation}
series = pd.Series(data=d) series = pd.Series(data=d)
exclude = list() exclude = []
result = check.citation_doi(series, exclude) result = check.citation_doi(series, exclude)
@ -418,7 +418,7 @@ def test_check_doi_only_in_citation(capsys):
"""Test an item with a DOI in its citation, but no DOI field.""" """Test an item with a DOI in its citation, but no DOI field."""
citation = "Orth, A. 2021. Testing all the things. doi: 10.1186/1743-422X-9-218" citation = "Orth, A. 2021. Testing all the things. doi: 10.1186/1743-422X-9-218"
exclude = list() exclude = []
# Emulate a column in a transposed dataframe (which is just a series), with # Emulate a column in a transposed dataframe (which is just a series), with
# an empty DOI field and a citation containing a DOI. # an empty DOI field and a citation containing a DOI.
@ -439,7 +439,7 @@ def test_title_in_citation():
title = "Testing all the things" title = "Testing all the things"
citation = "Orth, A. 2021. Testing all the things." citation = "Orth, A. 2021. Testing all the things."
exclude = list() exclude = []
# Emulate a column in a transposed dataframe (which is just a series), with # Emulate a column in a transposed dataframe (which is just a series), with
# the title and citation. # the title and citation.
@ -456,7 +456,7 @@ def test_title_not_in_citation(capsys):
title = "Testing all the things" title = "Testing all the things"
citation = "Orth, A. 2021. Testing all teh things." citation = "Orth, A. 2021. Testing all teh things."
exclude = list() exclude = []
# Emulate a column in a transposed dataframe (which is just a series), with # Emulate a column in a transposed dataframe (which is just a series), with
# the title and citation. # the title and citation.
@ -477,7 +477,7 @@ def test_country_matches_region():
country = "Kenya" country = "Kenya"
region = "Eastern Africa" region = "Eastern Africa"
exclude = list() exclude = []
# Emulate a column in a transposed dataframe (which is just a series) # Emulate a column in a transposed dataframe (which is just a series)
d = {"cg.coverage.country": country, "cg.coverage.region": region} d = {"cg.coverage.country": country, "cg.coverage.region": region}
@ -495,7 +495,7 @@ def test_country_not_matching_region(capsys):
country = "Kenya" country = "Kenya"
region = "" region = ""
missing_region = "Eastern Africa" missing_region = "Eastern Africa"
exclude = list() exclude = []
# Emulate a column in a transposed dataframe (which is just a series) # Emulate a column in a transposed dataframe (which is just a series)
d = { d = {

View File

@ -131,7 +131,7 @@ def test_fix_country_not_matching_region():
country = "Kenya" country = "Kenya"
region = "" region = ""
missing_region = "Eastern Africa" missing_region = "Eastern Africa"
exclude = list() exclude = []
# Emulate a column in a transposed dataframe (which is just a series) # Emulate a column in a transposed dataframe (which is just a series)
d = { d = {
@ -152,3 +152,11 @@ def test_fix_country_not_matching_region():
series_correct = pd.Series(data=d_correct) series_correct = pd.Series(data=d_correct)
pd.testing.assert_series_equal(result, series_correct) pd.testing.assert_series_equal(result, series_correct)
def test_fix_normalize_dois():
"""Test normalizing a DOI."""
value = "doi: 10.11648/j.jps.20140201.14"
assert fix.normalize_dois(value) == "https://doi.org/10.11648/j.jps.20140201.14"

1180
uv.lock generated Normal file

File diff suppressed because it is too large Load Diff