Update notes for 2018-05-15

content/posts/2018-05.md

@@ -175,6 +175,7 @@ $ lein run /tmp/crps.csv id
 ## 2018-05-14
 
 - Send a message to the OpenRefine mailing list about the bug with reconciling multi-value cells
+- Help Silvia Alonso get a list of all her publications since 2013 from Listings and Reports
 
 ## 2018-05-15
 
@@ -200,3 +201,52 @@ return "blank"
 - More information and good examples here: https://programminghistorian.org/lessons/fetch-and-parse-data-with-openrefine
 - Finish looking at the 2,640 CIFOR records on DSpace Test ([10568/92904](https://dspacetest.cgiar.org/handle/10568/92904)), cleaning up authors and adding collection mappings
 - They can now be moved to CGSpace as far as I'm concerned, but I don't know if Sisay will do it or me
+- I was checking the CIFOR data for duplicates using Atmire's Metadata Quality Module (and found some duplicates actually), but then DSpace died...
+- I didn't see anything in the Tomcat, DSpace, or Solr logs, but I saw this in `dmest -T`:
+
+```
+[Tue May 15 12:10:01 2018] Out of memory: Kill process 3763 (java) score 706 or sacrifice child
+[Tue May 15 12:10:01 2018] Killed process 3763 (java) total-vm:14667688kB, anon-rss:5705268kB, file-rss:0kB, shmem-rss:0kB
+[Tue May 15 12:10:01 2018] oom_reaper: reaped process 3763 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
+```
+
+- So the Linux kernel killed Java...
+- Maria from Bioversity mailed to say she got an error while submitting an item on CGSpace:
+
+```
+Unable to load Submission Information, since WorkspaceID (ID:S96060) is not a valid in-process submission
+```
+
+- Looking in the DSpace log I see something related:
+
+```
+2018-05-15 12:35:30,858 INFO  org.dspace.submit.step.CompleteStep @ m.garruccio@cgiar.org:session_id=8AC4499945F38B45EF7A1226E3042DAE:submission_complete:Completed submission with id=96060
+```
+
+- So I'm not sure...
+- I finally figured out how to get OpenRefine to reconcile values from Solr via [conciliator](https://github.com/codeforkjeff/conciliator):
+- The trick was to use a more appropriate Solr fieldType `text_en` instead of `text_general` so that more terms match, for example uppercase and lower case:
+
+```
+$ ./bin/solr start
+$ ./bin/solr create_core -c countries
+$ ./bin/post -c countries ~/src/git/DSpace/2018-05-10-countries.csv
+$ curl -X POST -H 'Content-type:application/json' --data-binary '{"add-field": {"name":"country", "type":"text_en", "multiValued":false, "stored":true}}' http://localhost:8983/solr/countries/schema
+```
+
+- It still doesn't catch simple mistakes like "ALBANI" or "AL BANIA" for "ALBANIA", and it doesn't return scores, so I have to select matches manually:
+
+![OpenRefine reconciling countries from local Solr](/cgspace-notes/2018/05/openrefine-solr-conciliator.png)
+
+- I should probably make a general copy field and set it to be the default search field, like DSpace's search core does (see schema.xml):
+
+```
+<defaultSearchField>search_text</defaultSearchField>
+...
+<copyField source="*" dest="search_text"/>
+```
+
+- Actually, I wonder how much of their schema I could just copy...
+- Apparently the default search field is the `df` parameter and you could technically just add it to the query string, so no need to bother with that in the schema now
+- I copied over the DSpace `search_text` field type from the DSpace Solr config (had to remove some properties so Solr would start) but it doesn't seem to be any better at matching than the `text_en` type
+- I think I need to focus on trying to return scores with conciliator