Update notes for 2020-07-03

content/posts/2020-07.md

@@ -178,5 +178,45 @@ Typhoeus
   - Atmire says they are able to build fine, so I tried again and noticed that I had been building with `-Denv=dspacetest.cgiar.org`, which is not necessary for DSpace 6 of course
   - Once I removed that it builds fine
 - I quickly re-applied the Font Awesome 5 changes to use SVG+JS instead of web fonts (from 2020-04) and things are looking good!
+- Run all system updates on DSpace Test (linode26), deploy latest `6_x-dev-atmire-modules` branch, and reboot it
+
+## 2020-07-02
+
+- I need to export some Solr statistics data from CGSpace to test Salem's modifications to the dspace-statistics-api
+  - He modified it to query Solr on the fly instead of indexing it, which will be heavier and slower, but allows us to get more granular stats and countries/cities
+  - Because have so many records I want to use solr-import-export-json to get several months at a time with a date range, but it seems there are first issues with curl (need to disable globbing with `-g` and URL encode the range)
+  - For reference, the [Solr 4.10.x DateField docs](https://lucene.apache.org/solr/4_10_2/solr-core/org/apache/solr/schema/DateField.html)
+  - This range works in Solr UI: `[2019-01-01T00:00:00Z TO 2019-06-30T23:59:59Z]`
+  - As well in curl:
+
+```
+$ curl -g -s 'http://localhost:8081/solr/statistics-2019/select?q=*:*&fq=time:%5B2019-01-01T00%3A00%3A00Z%20TO%202019-06-30T23%3A59%3A59Z%5D&rows=0&wt=json&indent=true'
+{
+  "responseHeader":{
+    "status":0,
+    "QTime":0,
+    "params":{
+      "q":"*:*",
+      "indent":"true",
+      "fq":"time:[2019-01-01T00:00:00Z TO 2019-06-30T23:59:59Z]",
+      "rows":"0",
+      "wt":"json"}},
+  "response":{"numFound":7784285,"start":0,"docs":[]
+  }}
+```
+
+- But not in solr-import-export-json... hmmm... seems we need to URL encode *only* the date range itself, but not the brackets:
+
+```
+$ ./run.sh -s http://localhost:8081/solr/statistics-2019 -a export -o /tmp/statistics-2019-1.json -f 'time:%5B2019-01-01T00%3A00%3A00Z%20TO%202019-06-30T23%3A59%3A59Z]' -k uid
+$ zstd /tmp/statistics-2019-1.json
+```
+
+- Then import it on my local dev environment:
+
+```
+$ zstd -d statistics-2019-1.json.zst
+$ ./run.sh -s http://localhost:8080/solr/statistics -a import -o ~/Downloads/statistics-2019-1.json -k uid
+```
 
 <!-- vim: set sw=2 ts=2: -->