alanorth/cgspace-notes
1
0
Fork 0
mirror of https://github.com/alanorth/cgspace-notes.git synced 2024-11-21 14:15:02 +01:00
Code Issues Projects Releases Wiki Activity

Update posts

Browse Source
This commit is contained in:
Alan Orth 2020-11-16 10:53:45 +02:00
parent 1b829672ee
commit a447a14116
Signed by: alanorth
GPG Key ID: 0FB860CC9C45B1B9
2 changed files with 55 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
content/posts/2020-10.md
View File

@ -724,7 +724,7 @@ org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error whil
```
- So basically, as I saw at this same step in 2020-05, there are some documents that have IDs that have *not* been converted to UUID, and have *not* been labeled as "unmigrated" either...
- I see there are about 217,000 of them, 99% of which are of `type: 5` which is "search"
- I see there are about 217,000 of them, 99% of which are of `type: 5` which is "site"
- I purged them:
```
@ -745,7 +745,7 @@ java.lang.OutOfMemoryError: Java heap space
```
- I had the same problem when I processed the statistics-2018 core in 2020-07 and 2020-08
- I will try to purge some unmigrated records (around 460,000), most of which are of `type: 5` (search) so not relevant to our views and downloads anyways:
- I will try to purge some unmigrated records (around 460,000), most of which are of `type: 5` (site) ~~so not relevant to our views and downloads anyways~~:
```console
$ curl -s "http://localhost:8083/solr/statistics-2018/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>id:/.+-unmigrated/</query></delete>"

53
content/posts/2020-11.md
View File

@ -201,9 +201,62 @@ $ sudo systemctl start tomcat7
```
$ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
real 211m30.726s
user 134m40.124s
sys 2m17.979s
```
- Towards the end of the indexing there were a few dozen of these messages:
```
2020-11-15 13:23:21,685 INFO com.atmire.dspace.discovery.service.AtmireSolrService @ Removed Item: null from Index
```
- I updated all the Ansible infrastructure and DSpace branches to be the DSpace 6 ones
- I will wait until the Discovery indexing is finished to start doing the Solr statistics migration
- I tested the email functionality and it seems to need more configuration:
```
$ dspace test-email
About to send test email:
- To: blah@cgiar.org
- Subject: DSpace test email
- Server: smtp.office365.com
Error sending email:
- Error: com.sun.mail.smtp.SMTPSendFailedException: 451 5.7.3 STARTTLS is required to send mail [AM4PR0701CA0003.eurprd07.prod.outlook.com]
```
- I copied the `mail.extraproperties = mail.smtp.starttls.enable=true` setting from the old DSpace 5 `dspace.cfg` and now the emails are working
- After the Discovery indexing finished I started processing the Solr stats one core and 2.5 million records at a time:
```
$ export JAVA_OPTS='-Dfile.encoding=UTF-8 -Xmx2048m'
$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 2500000 -i statistics
```
- After about 6,000,000 records I got the same error that I've gotten every time I test this migration process:
```
Exception: Error while creating field 'p_group_id{type=uuid,properties=indexed,stored,multiValued}' from value '10'
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error while creating field 'p_group_id{type=uuid,properties=indexed,stored,multiValued}' from value '10'
at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:68)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:54)
at org.dspace.util.SolrUpgradePre6xStatistics.batchUpdateStats(SolrUpgradePre6xStatistics.java:161)
at org.dspace.util.SolrUpgradePre6xStatistics.run(SolrUpgradePre6xStatistics.java:456)
at org.dspace.util.SolrUpgradePre6xStatistics.main(SolrUpgradePre6xStatistics.java:365)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
```
<!-- vim: set sw=2 ts=2: -->