Add notes for 2019-07-09

content/posts/2019-07.md

@@ -161,5 +161,42 @@ $ ./fix-metadata-values.py -i 2019-07-04-update-orcids.csv -db dspace -u dspace
 - Meeting with AgroKnow and CTA about their new ICT Update story telling thing
   - AgroKnow has developed a React application to display tag clouds based on harvesting metadata and full text from CGSpace items
   - We discussed how to host it technically, perhaps we purchase a server to run it on and just give AgroKnow guys access
+- Playing with the idea of using [xsv](https://github.com/BurntSushi/xsv) to do some basic batch quality checks on CSVs, for example to find items that might be duplicates if they have the same DOI or title:
+
+```
+$ xsv frequency --select cg.identifier.doi --no-nulls cgspace_metadata_africaRice-11to73_ay_id.csv | grep -v -E ',1'
+field,value,count
+cg.identifier.doi,https://doi.org/10.1016/j.agwat.2018.06.018,2
+$ xsv frequency --select dc.title --no-nulls cgspace_metadata_africaRice-11to73_ay_id.csv | grep -v -E ',1'         
+field,value,count
+dc.title,Reference evapotranspiration prediction using hybridized fuzzy model with firefly algorithm: Regional case study in Burkina Faso,2
+```
+
+- Or perhaps if DOIs are valid or not (having doi.org in the URL):
+
+```
+$ xsv frequency --select cg.identifier.doi --no-nulls cgspace_metadata_africaRice-11to73_ay_id.csv | grep -v -E 'doi.org'
+field,value,count
+cg.identifier.doi,https://hdl.handle.net/10520/EJC-1236ac700f,1
+```
+
+- Or perhaps items with invalid ISSNs (according to the [ISSN code format](https://en.wikipedia.org/wiki/International_Standard_Serial_Number#Code_format)):
+
+```
+$ xsv select dc.identifier.issn cgspace_metadata_africaRice-11to73_ay_id.csv | grep -v '"' | grep -v -E '^[0-9]{4}-[0-9]{3}[0-9xX]$'
+dc.identifier.issn
+978-3-319-71997-9
+978-3-319-71997-9
+978-3-319-71997-9
+978-3-319-58789-9
+2320-7035 
+ 2593-9173
+```
+
+## 2019-07-09
+
+- Thinking about data cleaning automation again and found some resources about Python and Pandas:
+  - https://realpython.com/python-data-cleaning-numpy-pandas/
+  - https://mode.com/blog/python-data-cleaning-libraries
 
 <!-- vim: set sw=2 ts=2: -->