alanorth/csv-metadata-quality
1
0
Fork 0
mirror of https://github.com/ilri/csv-metadata-quality.git synced 2025-05-09 22:56:01 +02:00
Code Releases Activity

csv_metadata_quality/app.py: skip abstract in separator check

Browse Source 
Also skip abstract in the separator check, since it's rare to have
any "|" here, but more likely that if one is present then it's for
a reason.
This commit is contained in:
Alan Orth
2023-02-13 10:37:33 +03:00
parent fbb625be5c
commit 8db1e36a6d
1 changed files with 3 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
csv_metadata_quality/app.py
View File

@ -124,9 +124,9 @@ def run(argv):
df[column] = df[column].apply(fix.unnecessary_unicode)
# Fix: invalid and unnecessary multi-value separators. Skip the title
# field because sometimes "|" is used to indicate something like a
# subtitle.
match = re.match(r"^.*?title.*$", column)
# and abstract fields because "|" is used to indicate something like
# a subtitle.
match = re.match(r"^.*?(abstract|title).*$", column)
if match is None:
df[column] = df[column].apply(fix.separators, field_name=column)
# Run whitespace fix again after fixing invalid separators