diff --git a/content/posts/2020-10.md b/content/posts/2020-10.md index 57f1b813e..f462b9fcf 100644 --- a/content/posts/2020-10.md +++ b/content/posts/2020-10.md @@ -724,7 +724,7 @@ org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error whil ``` - So basically, as I saw at this same step in 2020-05, there are some documents that have IDs that have *not* been converted to UUID, and have *not* been labeled as "unmigrated" either... - - I see there are about 217,000 of them, 99% of which are of `type: 5` which is "search" + - I see there are about 217,000 of them, 99% of which are of `type: 5` which is "site" - I purged them: ``` @@ -745,7 +745,7 @@ java.lang.OutOfMemoryError: Java heap space ``` - I had the same problem when I processed the statistics-2018 core in 2020-07 and 2020-08 - - I will try to purge some unmigrated records (around 460,000), most of which are of `type: 5` (search) so not relevant to our views and downloads anyways: + - I will try to purge some unmigrated records (around 460,000), most of which are of `type: 5` (site) ~~so not relevant to our views and downloads anyways~~: ```console $ curl -s "http://localhost:8083/solr/statistics-2018/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "id:/.+-unmigrated/" diff --git a/content/posts/2020-11.md b/content/posts/2020-11.md index 729155757..b3ac797e5 100644 --- a/content/posts/2020-11.md +++ b/content/posts/2020-11.md @@ -201,9 +201,62 @@ $ sudo systemctl start tomcat7 ``` $ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b + +real 211m30.726s +user 134m40.124s +sys 2m17.979s +``` + +- Towards the end of the indexing there were a few dozen of these messages: + +``` +2020-11-15 13:23:21,685 INFO com.atmire.dspace.discovery.service.AtmireSolrService @ Removed Item: null from Index ``` - I updated all the Ansible infrastructure and DSpace branches to be the DSpace 6 ones - I will wait until the Discovery indexing is finished to start doing the Solr statistics migration +- I tested the email functionality and it seems to need more configuration: + +``` +$ dspace test-email + +About to send test email: + - To: blah@cgiar.org + - Subject: DSpace test email + - Server: smtp.office365.com + +Error sending email: + - Error: com.sun.mail.smtp.SMTPSendFailedException: 451 5.7.3 STARTTLS is required to send mail [AM4PR0701CA0003.eurprd07.prod.outlook.com] +``` + +- I copied the `mail.extraproperties = mail.smtp.starttls.enable=true` setting from the old DSpace 5 `dspace.cfg` and now the emails are working +- After the Discovery indexing finished I started processing the Solr stats one core and 2.5 million records at a time: + +``` +$ export JAVA_OPTS='-Dfile.encoding=UTF-8 -Xmx2048m' +$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 2500000 -i statistics +``` + +- After about 6,000,000 records I got the same error that I've gotten every time I test this migration process: + +``` +Exception: Error while creating field 'p_group_id{type=uuid,properties=indexed,stored,multiValued}' from value '10' +org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error while creating field 'p_group_id{type=uuid,properties=indexed,stored,multiValued}' from value '10' + at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552) + at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210) + at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206) + at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124) + at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:68) + at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:54) + at org.dspace.util.SolrUpgradePre6xStatistics.batchUpdateStats(SolrUpgradePre6xStatistics.java:161) + at org.dspace.util.SolrUpgradePre6xStatistics.run(SolrUpgradePre6xStatistics.java:456) + at org.dspace.util.SolrUpgradePre6xStatistics.main(SolrUpgradePre6xStatistics.java:365) + at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) + at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) + at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) + at java.lang.reflect.Method.invoke(Method.java:498) + at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229) + at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81) +```