Add notes for 2022-06-29

content/posts/2022-06.md

@@ -200,4 +200,59 @@ $ xsv join --full alpha2 /tmp/clarisa-un-cgspace-xsv-full.csv alpha2 /tmp/mel-co
 
 - Start a harvest on AReS
 
+## 2022-06-28
+
+- Start working on the CGSpace subject export for FAO
+- First I exported a list of all metadata in our `dcterms.subject` and other center-specific subject fields with their counts:
+
+```console
+localhost/dspacetest= ☘ \COPY (SELECT DISTINCT text_value AS "subject", count(*) FROM metadatavalue WHERE dspace_object_id in (SELECT dspace_object_id FROM item) AND metadata_field_id IN (187, 120, 210, 122, 215, 127, 208, 124, 128, 123, 125, 135, 203, 236, 238, 119) GROUP BY "subject" ORDER BY count DESC) to /tmp/2022-06-28-cgspace-subjects.csv WITH CSV HEADER;
+COPY 27010
+```
+
+- Then I extracted the subjects and looked them up against AGROVOC:
+
+```console
+$ csvcut -c subject /tmp/2022-06-28-cgspace-subjects.csv | sed '1d' > /tmp/2022-06-28-cgspace-subjects.txt
+$ ./ilri/agrovoc-lookup.py -i /tmp/2022-06-28-cgspace-subjects.txt -o /tmp/2022-06-28-cgspace-subjects-results.csv
+```
+
+- I keep getting timeouts after every five or ten requests, so this will not be feasible for 27,000 subjects!
+- I think I will have to write some custom script to use the AGROVOC RDF file
+  - Using rdflib to open the 1.2GB `agrovoc_lod.rdf` file takes several minutes and doesn't seem very efficient
+- I tried using [lightrdf](https://github.com/ozekik/lightrdf) and it's much quicker, but the documentation is limiting and I'm not sure how to search yet
+  - I had to try in different Python versions because 3.10.x is apparently too new
+- For future reference I was able to search with lightrdf:
+
+```console
+import lightrdf
+parser = lightrdf.Parser()
+# prints millions of lines
+for triple in parser.parse("./agrovoc_lod.rdf", base_iri=None):
+     print(triple)
+agrovoc = lightrdf.RDFDocument('agrovoc_lod.rdf');
+# all results for prefix http://aims.fao.org/aos/agrovoc/c_5
+for triple in agrovoc.search_triples('http://aims.fao.org/aos/agrovoc/c_5', None, None):
+     print(triple)
+('http://aims.fao.org/aos/agrovoc/c_5', 'http://www.w3.org/2004/02/skos/core#altLabel', '"Abalone"@de')
+('http://aims.fao.org/aos/agrovoc/c_5', 'http://www.w3.org/2004/02/skos/core#prefLabel', '"abalones"@en')
+# all stuff for abalones in English
+for triple in agrovoc.search_triples(None, None, '"abalones"@en'):
+     print(triple)
+```
+
+- I ran the `agrovoc-lookup.py` from a Linode server and it completed without issues... hmmm
+
+## 2022-06-29
+
+- Continue working on the list of non-AGROVOC subject to report to FAO
+  - I got a one liner to get the list of non-AGROVOC subjects and join them with their counts:
+
+```console
+$ csvgrep -c 'number of matches' -m 0 /tmp/2022-06-28-cgspace-subjects-results.csv \
+  | csvcut -c subject \
+  | csvjoin -c subject /tmp/2022-06-28-cgspace-subjects.csv - \
+  > /tmp/2022-06-28-cgspace-non-agrovoc.csv
+```
+
 <!-- vim: set sw=2 ts=2: -->