alanorth/csv-metadata-quality
1
0
Fork 0
mirror of https://github.com/ilri/csv-metadata-quality.git synced 2024-11-22 22:05:03 +01:00
Code Releases Activity
d93c2aae13
csv-metadata-quality/tests/test_fix.py

59 lines
1.1 KiB
Python
Raw Normal View History

Add initial tests For now only test fixes because they return changed data. I'm not sure how to test the checks, because they don't return data and I can't modify them to return boolean values without breaking the app.
2019-07-26 23:36:40 +02:00
import csv_metadata_quality.fix as fix
Fix whitespace errors found by flake8
2019-07-28 16:47:28 +02:00
Add initial tests For now only test fixes because they return changed data. I'm not sure how to test the checks, because they don't return data and I can't modify them to return boolean values without breaking the app.
2019-07-26 23:36:40 +02:00
def test_fix_leading_whitespace():
'''Test fixing leading whitespace.'''
value = ' Alan'
assert fix.whitespace(value) == 'Alan'
Fix whitespace errors found by flake8
2019-07-28 16:47:28 +02:00
Add initial tests For now only test fixes because they return changed data. I'm not sure how to test the checks, because they don't return data and I can't modify them to return boolean values without breaking the app.
2019-07-26 23:36:40 +02:00
def test_fix_trailing_whitespace():
'''Test fixing trailing whitespace.'''
value = 'Alan '
assert fix.whitespace(value) == 'Alan'
Fix whitespace errors found by flake8
2019-07-28 16:47:28 +02:00
Add initial tests For now only test fixes because they return changed data. I'm not sure how to test the checks, because they don't return data and I can't modify them to return boolean values without breaking the app.
2019-07-26 23:36:40 +02:00
def test_fix_excessive_whitespace():
'''Test fixing excessive whitespace.'''
value = 'Alan Orth'
assert fix.whitespace(value) == 'Alan Orth'
Add "unsafe fixes" runtime option In this case it fixes occurences of invalid multi-value separators. DSpace uses "||" to separate multiple values in one field, but our editors sometimes give us files with mistakes like "|". We can fix these to be correct multi-value separators if we are sure that the metadata is not actually using "|" for some legitimate purpose.
2019-07-28 21:53:39 +02:00
def test_fix_invalid_separators():
'''Test fixing invalid multi-value separators.'''
value = 'Alan|Orth'
assert fix.separators(value) == 'Alan||Orth'
Add support for fixing "unnecessary" Unicode These are things like non-breaking spaces, "replacement" characters, etc that add nothing to the metadata and often cause errors during parsing or displaying in a UI.
2019-07-29 15:38:10 +02:00
def test_fix_unnecessary_unicode():
'''Test fixing unnecessary Unicode.'''
value = 'Alan​ Orth'
assert fix.unnecessary_unicode(value) == 'Alan Orth'
Add fix for duplicate metadata values
2019-07-29 17:05:03 +02:00
def test_fix_duplicates():
'''Test fixing duplicate metadata values.'''
value = 'Kenya||Kenya'
assert fix.duplicates(value) == 'Kenya'
Add support for removing newlines This was tricky because of the nature of newlines. In actuality we are removing Unix line feeds here (U+000A) because Windows carriage returns are actually already removed by the string stripping in the whitespace fix. Creating the test case in Vim was difficult because I couldn't fig- ure out how to manually enter a line feed character. In the end I used a search and replace on a known pattern like "ALAN", replacing it with \r. Neither entering the Unicode code point (U+000A) direc- tly or typing an "Enter" character after ^V worked. Grrr.
2019-07-30 19:05:12 +02:00
def test_fix_newlines():
'''Test fixing newlines.'''
value = '''Ken
ya'''
assert fix.newlines(value) == 'Kenya'
Copy Permalink