Update posts

content/posts/2020-10.md

@@ -724,7 +724,7 @@ org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error whil
 ```
 
 - So basically, as I saw at this same step in 2020-05, there are some documents that have IDs that have *not* been converted to UUID, and have *not* been labeled as "unmigrated" either...
-  - I see there are about 217,000 of them, 99% of which are of `type: 5` which is "search"
+  - I see there are about 217,000 of them, 99% of which are of `type: 5` which is "site"
   - I purged them:
 
 ```
@@ -745,7 +745,7 @@ java.lang.OutOfMemoryError: Java heap space
 ```
 
 - I had the same problem when I processed the statistics-2018 core in 2020-07 and 2020-08
-  - I will try to purge some unmigrated records (around 460,000), most of which are of `type: 5` (search) so not relevant to our views and downloads anyways:
+  - I will try to purge some unmigrated records (around 460,000), most of which are of `type: 5` (site) ~~so not relevant to our views and downloads anyways~~:
 
 ```console
 $ curl -s "http://localhost:8083/solr/statistics-2018/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>id:/.+-unmigrated/</query></delete>"

content/posts/2020-11.md

@@ -201,9 +201,62 @@ $ sudo systemctl start tomcat7
 
 ```
 $ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    211m30.726s
+user    134m40.124s
+sys     2m17.979s
+```
+
+- Towards the end of the indexing there were a few dozen of these messages:
+
+```
+2020-11-15 13:23:21,685 INFO  com.atmire.dspace.discovery.service.AtmireSolrService @ Removed Item: null from Index
 ```
 
 - I updated all the Ansible infrastructure and DSpace branches to be the DSpace 6 ones
 - I will wait until the Discovery indexing is finished to start doing the Solr statistics migration
+- I tested the email functionality and it seems to need more configuration:
+
+```
+$ dspace test-email
+
+About to send test email:
+ - To: blah@cgiar.org
+ - Subject: DSpace test email
+ - Server: smtp.office365.com
+
+Error sending email:
+ - Error: com.sun.mail.smtp.SMTPSendFailedException: 451 5.7.3 STARTTLS is required to send mail [AM4PR0701CA0003.eurprd07.prod.outlook.com]
+```
+
+- I copied the `mail.extraproperties = mail.smtp.starttls.enable=true` setting from the old DSpace 5 `dspace.cfg` and now the emails are working
+- After the Discovery indexing finished I started processing the Solr stats one core and 2.5 million records at a time:
+
+```
+$ export JAVA_OPTS='-Dfile.encoding=UTF-8 -Xmx2048m'
+$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 2500000 -i statistics
+```
+
+- After about 6,000,000 records I got the same error that I've gotten every time I test this migration process:
+
+```
+Exception: Error while creating field 'p_group_id{type=uuid,properties=indexed,stored,multiValued}' from value '10'
+org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error while creating field 'p_group_id{type=uuid,properties=indexed,stored,multiValued}' from value '10'
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
+        at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
+        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:68)
+        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:54)
+        at org.dspace.util.SolrUpgradePre6xStatistics.batchUpdateStats(SolrUpgradePre6xStatistics.java:161)
+        at org.dspace.util.SolrUpgradePre6xStatistics.run(SolrUpgradePre6xStatistics.java:456)
+        at org.dspace.util.SolrUpgradePre6xStatistics.main(SolrUpgradePre6xStatistics.java:365)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+```
 
 <!-- vim: set sw=2 ts=2: -->