alanorth/cgspace-notes
1
0
Fork 0
mirror of https://github.com/alanorth/cgspace-notes.git synced 2025-01-27 05:49:12 +01:00
Code Issues Projects Releases Wiki Activity

Regenerate docs

Browse Source
This commit is contained in:
Alan Orth
2020-11-16 10:54:00 +02:00
parent a447a14116
commit a50ad3ac76
93 changed files with 464 additions and 469 deletions
Download Patch File Download Diff File Expand all files Collapse all files

62
docs/2020-11/index.html
View File

@ -17,7 +17,7 @@ So far we’ve spent at least fifty hours to process the statistics and stat
<meta property="og:type" content="article" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2020-11/" />
<meta property="article:published_time" content="2020-11-01T13:11:54+02:00" />
<meta property="article:modified_time" content="2020-11-13T09:22:18+02:00" />
<meta property="article:modified_time" content="2020-11-16T10:53:45+02:00" />
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="November, 2020"/>
@ -39,9 +39,9 @@ So far we&rsquo;ve spent at least fifty hours to process the statistics and stat
"@type": "BlogPosting",
"headline": "November, 2020",
"url": "https://alanorth.github.io/cgspace-notes/2020-11/",
"wordCount": "1565",
"wordCount": "1762",
"datePublished": "2020-11-01T13:11:54+02:00",
"dateModified": "2020-11-13T09:22:18+02:00",
"dateModified": "2020-11-16T10:53:45+02:00",
"author": {
"@type": "Person",
"name": "Alan Orth"
@ -59,20 +59,18 @@ So far we&rsquo;ve spent at least fifty hours to process the statistics and stat
<!-- combined, minified CSS -->
<link href="https://alanorth.github.io/cgspace-notes/css/style.6da5c906cc7a8fbb93f31cd2316c5dbe3f19ac4aa6bfb066f1243045b8f6061e.css" rel="stylesheet" integrity="sha256-baXJBsx6j7uT8xzSMWxdvj8ZrEqmv7Bm8SQwRbj2Bh4=" crossorigin="anonymous">
<link href="https://alanorth.github.io/cgspace-notes/css/style.d20c61b183eb27beb5b2c48f70a38b91c8bb5fb929e77b447d5f77c7285221ad.css" rel="stylesheet" integrity="sha256-0gxhsYPrJ761ssSPcKOLkci7X7kp53tEfV93xyhSIa0=" crossorigin="anonymous">
<!-- minified Font Awesome for SVG icons -->
<script defer src="https://alanorth.github.io/cgspace-notes/js/fontawesome.min.f3d2a1f5980bab30ddd0d8cadbd496475309fc48e2b1d052c5c09e6facffcb0f.js" integrity="sha256-89Kh9ZgLqzDd0NjK29SWR1MJ/EjisdBSxcCeb6z/yw8=" crossorigin="anonymous"></script>
<script defer src="https://alanorth.github.io/cgspace-notes/js/fontawesome.min.4ed405d7c7002b970d34cbe6026ff44a556b0808cb98a9db4008752110ed964b.js" integrity="sha256-TtQF18cAK5cNNMvmAm/0SlVrCAjLmKnbQAh1IRDtlks=" crossorigin="anonymous"></script>
<!-- RSS 2.0 feed -->
</head>
<body>
@ -109,7 +107,9 @@ So far we&rsquo;ve spent at least fifty hours to process the statistics and stat
<article class="blog-post">
<header>
<h2 class="blog-post-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/2020-11/">November, 2020</a></h2>
<p class="blog-post-meta"><time datetime="2020-11-01T13:11:54+02:00">Sun Nov 01, 2020</time> by Alan Orth in
<p class="blog-post-meta">
<time datetime="2020-11-01T13:11:54+02:00">Sun Nov 01, 2020</time>
in
<span class="fas fa-folder" aria-hidden="true"></span>&nbsp;<a href="/cgspace-notes/categories/notes/" rel="category tag">Notes</a>
@ -343,11 +343,55 @@ $ sudo systemctl start tomcat7
<li>Start a full Discovery re-indexing:</li>
</ul>
<pre><code>$ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
real 211m30.726s
user 134m40.124s
sys 2m17.979s
</code></pre><ul>
<li>Towards the end of the indexing there were a few dozen of these messages:</li>
</ul>
<pre><code>2020-11-15 13:23:21,685 INFO com.atmire.dspace.discovery.service.AtmireSolrService @ Removed Item: null from Index
</code></pre><ul>
<li>I updated all the Ansible infrastructure and DSpace branches to be the DSpace 6 ones</li>
<li>I will wait until the Discovery indexing is finished to start doing the Solr statistics migration</li>
<li>I tested the email functionality and it seems to need more configuration:</li>
</ul>
<!-- raw HTML omitted -->
<pre><code>$ dspace test-email
About to send test email:
- To: blah@cgiar.org
- Subject: DSpace test email
- Server: smtp.office365.com
Error sending email:
- Error: com.sun.mail.smtp.SMTPSendFailedException: 451 5.7.3 STARTTLS is required to send mail [AM4PR0701CA0003.eurprd07.prod.outlook.com]
</code></pre><ul>
<li>I copied the <code>mail.extraproperties = mail.smtp.starttls.enable=true</code> setting from the old DSpace 5 <code>dspace.cfg</code> and now the emails are working</li>
<li>After the Discovery indexing finished I started processing the Solr stats one core and 2.5 million records at a time:</li>
</ul>
<pre><code>$ export JAVA_OPTS='-Dfile.encoding=UTF-8 -Xmx2048m'
$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 2500000 -i statistics
</code></pre><ul>
<li>After about 6,000,000 records I got the same error that I&rsquo;ve gotten every time I test this migration process:</li>
</ul>
<pre><code>Exception: Error while creating field 'p_group_id{type=uuid,properties=indexed,stored,multiValued}' from value '10'
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error while creating field 'p_group_id{type=uuid,properties=indexed,stored,multiValued}' from value '10'
at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:68)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:54)
at org.dspace.util.SolrUpgradePre6xStatistics.batchUpdateStats(SolrUpgradePre6xStatistics.java:161)
at org.dspace.util.SolrUpgradePre6xStatistics.run(SolrUpgradePre6xStatistics.java:456)
at org.dspace.util.SolrUpgradePre6xStatistics.main(SolrUpgradePre6xStatistics.java:365)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
</code></pre><!-- raw HTML omitted -->