Add notes for 2024-05-27

content/posts/2024-05.md

@@ -106,4 +106,86 @@ Continue working through alternative duplicate matching for IFPRI
   - One thing I think I can say for sure is that the default similarity factor in my script is 0.6, and I rarely see legitimate duplicates with such similarity so I might increase this to 0.7 to reduce the number of items I have to check
   - Also, the difference in issue dates is currently 365, but I should reduce that a bit, perhaps to 270 days (9 months)
 
+## 2024-05-22
+
+- Finalize and upload the IFPRI 2020–2021 batch set
+  - I used a new technique to get missing licenses via Crossref (it's Python 2 because of OpenRefine's Jython):
+
+```python
+import urllib2
+
+doi = cells['cg.identifier.doi[en_US]'].value
+url = "https://api.crossref.org/works/" + doi
+useragent = "Python (mailto:a.o@cgiar.org)"
+
+request = urllib2.Request(url.encode("utf-8"), headers={"User-Agent" : useragent})
+get = urllib2.urlopen(request)
+
+return get.read().decode('utf-8')
+```
+
+## 2024-05-23
+
+- Finalize last of the duplicates I found for the IFPRI 2020–2021 batch set (those that we missed initially due to mismatched types)
+- Export a new list of IFPRI redirects from CONTENTdm:
+
+```console
+$ csvgrep -c 'dc.description.provenance[en_US]' -r 'Original URLs? from IFPRI CONTENTdm' cgspace.csv \
+  | csvcut -c 'id,dc.description.provenance[en_US],dc.identifier.uri[en_US]' \
+  | tee /tmp/ifpri-redirects.csv \
+  | csvstat --count
+4004
+```
+
+I found a way to get abstracts from PLOS
+  - They offer an API that returns XML including the JATS-formatted abstracts
+  - I created a new column in OpenRefine by fetching specially crafted URLs based on the DOIs using this GREL:
+
+```console
+"https://journals.plos.org/plosone/article/file?id=" + cells['doi'].value + '&type=manuscript'
+```
+
+Then used `value.parseXml()` on the resulting text to extract the abstract's text:
+
+```console
+value.parseXml().select("abstract")[0].xmlText()
+```
+
+This doesn't preserve `<p>` tags though...
+  - Oh, nice, this does!
+
+```console
+forEach(value.parseHtml().select("abstract p"), i, i.htmlText()).join("\r\n\r\n")
+```
+
+For each paragraph inside an abstract, get the inner text and join them as one string separated by two newlines...
+  - Ah, some articles have multiple abstracts, for example: https://journals.plos.org/plosone/article/file?id=https://doi.org/10.1371/journal.pntd.0001859&type=manuscript
+  - I need to select the abstract that does **not** have any attributes (using [Jsoup selector syntax](https://jsoup.org/apidocs/org/jsoup/select/Selector.html))
+
+```console
+forEach(value.parseXml().select("abstract:not([*]) p"), i, i.xmlText()).join("\r\n\r\n")
+```
+
+Testing `xsv` (Rust) versus `csvkit` (Python) to filter all items with DOIs from a DSpace dump with 118,000 items:
+
+```console
+$ time xsv search -s doi 'doi\.org' /tmp/cgspace-minimal.csv | xsv select doi | xsv count
+27339
+xsv search -s doi 'doi\.org' /tmp/cgspace-minimal.csv  0.06s user 0.03s system 98% cpu 0.091 total
+xsv select doi  0.02s user 0.02s system 40% cpu 0.091 total
+xsv count  0.01s user 0.00s system 9% cpu 0.090 total
+$ time csvgrep -c doi -m 'doi.org' /tmp/cgspace-minimal.csv | csvcut -c doi | csvstat --count
+27339
+csvgrep -c doi -m 'doi.org' /tmp/cgspace-minimal.csv  1.15s user 0.06s system 95% cpu 1.273 total
+csvcut -c doi  0.42s user 0.05s system 36% cpu 1.283 total
+csvstat --count  0.20s user 0.03s system 18% cpu 1.298 total
+```
+
+## 2024-05-27
+
+- Working on IFPRI datasets batch migration
+  - 732 items total
+  - 6 duplicates on CGSpace
+  - 6 duplicates within set that need investigation
+
 <!-- vim: set sw=2 ts=2: -->