alanorth/csv-metadata-quality
1
0
Fork 0
mirror of https://github.com/ilri/csv-metadata-quality.git synced 2024-07-01 10:33:46 +02:00
Code Releases Activity
98a6a19e12
csv-metadata-quality/data/test.csv

31 lines
1.4 KiB
Plaintext
Raw Normal View History

data/test.csv: Change birthdate column to dc.date.issued More accurately reflects actual data we will be validating.
2019-09-26 13:15:48 +02:00
dc.title,dc.date.issued,dc.identifier.issn,dc.identifier.isbn,dc.language.iso,dc.subject,cg.coverage.country,filename
data/test.csv: Add test case for uncommon filename extension
2019-08-10 22:46:56 +02:00
Leading space,2019-07-29,,,,,,
Trailing space ,2019-07-29,,,,,,
Excessive space,2019-07-29,,,,,,
Miscellaenous ||whitespace | issues ,2019-07-29,,,,,,
Duplicate||Duplicate,2019-07-29,,,,,,
Invalid ISSN,2019-07-29,2321-2302,,,,,
Invalid ISBN,2019-07-29,,978-0-306-40615-6,,,,
Multiple valid ISSNs,2019-07-29,0378-5955||0024-9319,,,,,
Multiple valid ISBNs,2019-07-29,,99921-58-10-7||978-0-306-40615-7,,,,
Invalid date,2019-07-260,,,,,,
Multiple dates,2019-07-26||2019-01-10,,,,,,
Invalid multi-value separator,2019-07-29,0378-5955|0024-9319,,,,,
Unnecessary Unicode​,2019-07-29,,,,,,
Suspicious character||foreˆt,2019-07-29,,,,,,
data/test.csv: Update titles of language tests ISO 639-1 is alpha 2 and ISO 639-3 is alpha 3. See: https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes
2019-09-26 06:40:27 +02:00
Invalid ISO 639-1 (alpha 2) language,2019-07-29,,,jp,,,
Invalid ISO 639-3 (alpha 3) language,2019-07-29,,,chi,,,
data/test.csv: Add test case for uncommon filename extension
2019-08-10 22:46:56 +02:00
Invalid language,2019-07-29,,,Span,,,
Invalid AGROVOC subject,2019-07-29,,,,FOREST,,
data/test.csv: Clarify that newline is a line feed
2019-07-31 12:03:43 +02:00
Newline (LF),2019-07-30,,,,"TANZA
data/test.csv: Add test case for uncommon filename extension
2019-08-10 22:46:56 +02:00
NIA",,
Missing date,,,,,,,
Invalid country,2019-08-01,,,,,KENYAA,
Uncommon filename extension,2019-08-10,,,,,,file.pdf.lck
data/test.csv: Add test for replacement of unneccessary Unicode
2019-08-10 23:08:44 +02:00
Unneccesary unicode (U+002D + U+00AD),2019-08-10,,978-­92-­9043-­823-­6,,,,
data/test.csv: Add sample for missing space after comma
2019-08-27 23:08:26 +02:00
"Missing space,after comma",2019-08-27,,,,,,
Experimental language detection using langid Works decenty well assuming the title, abstract, and citation fields are an accurate representation of the language as identified by the language field. Handles ISO 639-1 (alpha 2) and ISO 639-3 (alpha 3) values seamlessly. This includes updated pipenv environment, test data, pytest tests for both correct and incorrect ISO 639-1 and ISO 639-3 languages, and a new command line option "-e".
2019-09-24 17:55:05 +02:00
Incorrect ISO 639-1 language,2019-09-26,,,es,,,
Incorrect ISO 639-3 language,2019-09-26,,,spa,,,
Add Unicode normalization This will check all strings for un-normalized Unicode characters. Normalization is done using NFC. This includes tests and updated sample data (data/test.csv). See: https://withblue.ink/2019/03/11/why-you-need-to-normalize-unicode-strings.html
2020-01-15 10:37:54 +01:00
Composéd Unicode,2020-01-14,,,,,,
Decomposéd Unicode,2020-01-14,,,,,,
Copy Permalink