alanorth/cgspace-notes
1
0
Fork 0
mirror of https://github.com/alanorth/cgspace-notes.git synced 2024-06-09 13:25:10 +02:00
Code Issues Projects Releases Wiki Activity

Add notes for 2020-07-23

Browse Source
This commit is contained in:
Alan Orth 2020-07-23 12:32:11 +03:00
parent bb1b29170c
commit 8c8e87828e
Signed by: alanorth
GPG Key ID: 0FB860CC9C45B1B9
20 changed files with 68 additions and 25 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

22
content/posts/2020-07.md
View File

@ -648,5 +648,27 @@ OpenRXV harvesting bot; https://github.com/ilri/OpenRXV
- [COUNTER_Robots_list.json: Remove anchors from okhttp](https://github.com/atmire/COUNTER-Robots/pull/33)
- I will update the agent patterns on the CGSpace `5_x-prod` and 6.x branches
- Make some changes to the Bootstrap CSS and HTML configuration to improve readability and style on the CG Core v2 metadata reference guide and send a pull request to Marie ([#29](https://github.com/AgriculturalSemantics/cg-core/pull/29))
- The `solr-upgrade-statistics-6x` tool keeps crashing due to memory issues when processing 2018 stats
- I reduced the number of records per batch from 10,000 to 5,000 and increased the memory to 3072 and it still crashes...
- I reduced the number of records per batch to 1,000 and it works, but still took like twenty minutes before it even started!
- Eventually after processing a few million records it crashed with this error:
```
Exception: Error while creating field 'p_group_id{type=uuid,properties=indexed,stored,multiValued}' from value '10'
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error while creating field 'p_group_id{type=uuid,properties=indexed,stored,multiValued}' from value '10'
```
- There were four records so I deleted them:
```
$ curl -s "http://localhost:8081/solr/statistics-2018/update?softCommit=true" -H "Content-Type: text/xml" --data-binary '<delete><query>id:10</query></delete>'
```
- Meeting with Moayad and Peter and Abenet to discuss the latest AReS changes
## 2020-07-23
- I closed all issues in the [OpenRXV](https://github.com/ilri/OpenRXV/issues) and [AReS](https://github.com/ilri/AReS/issues) GitHub repositories with screenshots so that Moayad can use them for his invoice
- The statistics-2018 core always crashes with the same error even after I deleted the "id:10" records...
<!-- vim: set sw=2 ts=2: -->

27
docs/2020-07/index.html
View File

@ -20,7 +20,7 @@ Since I was restarting Tomcat anyways I decided to redeploy the latest changes f
<meta property="og:type" content="article" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2020-07/" />
<meta property="article:published_time" content="2020-07-01T10:53:54+03:00" />
<meta property="article:modified_time" content="2020-07-21T13:44:52+03:00" />
<meta property="article:modified_time" content="2020-07-22T11:00:40+03:00" />
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="July, 2020"/>
@ -45,9 +45,9 @@ Since I was restarting Tomcat anyways I decided to redeploy the latest changes f
"@type": "BlogPosting",
"headline": "July, 2020",
"url": "https://alanorth.github.io/cgspace-notes/2020-07/",
"wordCount": "4195",
"wordCount": "4352",
"datePublished": "2020-07-01T10:53:54+03:00",
"dateModified": "2020-07-21T13:44:52+03:00",
"dateModified": "2020-07-22T11:00:40+03:00",
"author": {
"@type": "Person",
"name": "Alan Orth"
@ -782,6 +782,27 @@ $ chrt -b 0 dspace solr-upgrade-statistics-6x -n 1000000 -i statistics-2018
</li>
<li>I will update the agent patterns on the CGSpace <code>5_x-prod</code> and 6.x branches</li>
<li>Make some changes to the Bootstrap CSS and HTML configuration to improve readability and style on the CG Core v2 metadata reference guide and send a pull request to Marie (<a href="https://github.com/AgriculturalSemantics/cg-core/pull/29">#29</a>)</li>
<li>The <code>solr-upgrade-statistics-6x</code> tool keeps crashing due to memory issues when processing 2018 stats
<ul>
<li>I reduced the number of records per batch from 10,000 to 5,000 and increased the memory to 3072 and it still crashes&hellip;</li>
<li>I reduced the number of records per batch to 1,000 and it works, but still took like twenty minutes before it even started!</li>
<li>Eventually after processing a few million records it crashed with this error:</li>
</ul>
</li>
</ul>
<pre><code>Exception: Error while creating field 'p_group_id{type=uuid,properties=indexed,stored,multiValued}' from value '10'
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error while creating field 'p_group_id{type=uuid,properties=indexed,stored,multiValued}' from value '10'
</code></pre><ul>
<li>There were four records so I deleted them:</li>
</ul>
<pre><code>$ curl -s &quot;http://localhost:8081/solr/statistics-2018/update?softCommit=true&quot; -H &quot;Content-Type: text/xml&quot; --data-binary '&lt;delete&gt;&lt;query&gt;id:10&lt;/query&gt;&lt;/delete&gt;'
</code></pre><ul>
<li>Meeting with Moayad and Peter and Abenet to discuss the latest AReS changes</li>
</ul>
<h2 id="2020-07-23">2020-07-23</h2>
<ul>
<li>I closed all issues in the <a href="https://github.com/ilri/OpenRXV/issues">OpenRXV</a> and <a href="https://github.com/ilri/AReS/issues">AReS</a> GitHub repositories with screenshots so that Moayad can use them for his invoice</li>
<li>The statistics-2018 core always crashes with the same error even after I deleted the &ldquo;id:10&rdquo; records&hellip;</li>
</ul>
<!-- raw HTML omitted -->

2
docs/categories/index.html
View File

@ -9,7 +9,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/" />
<meta property="og:updated_time" content="2020-07-21T13:44:52+03:00" />
<meta property="og:updated_time" content="2020-07-22T11:00:40+03:00" />
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Categories"/>

2
docs/categories/notes/index.html
View File

@ -9,7 +9,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
<meta property="og:updated_time" content="2020-07-21T13:44:52+03:00" />
<meta property="og:updated_time" content="2020-07-22T11:00:40+03:00" />
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Notes"/>

2
docs/categories/notes/page/2/index.html
View File

@ -9,7 +9,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
<meta property="og:updated_time" content="2020-07-21T13:44:52+03:00" />
<meta property="og:updated_time" content="2020-07-22T11:00:40+03:00" />
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Notes"/>

2
docs/categories/notes/page/3/index.html
View File

@ -9,7 +9,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
<meta property="og:updated_time" content="2020-07-21T13:44:52+03:00" />
<meta property="og:updated_time" content="2020-07-22T11:00:40+03:00" />
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Notes"/>

2
docs/categories/notes/page/4/index.html
View File

@ -9,7 +9,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
<meta property="og:updated_time" content="2020-07-21T13:44:52+03:00" />
<meta property="og:updated_time" content="2020-07-22T11:00:40+03:00" />
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Notes"/>

2
docs/index.html
View File

@ -9,7 +9,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
<meta property="og:updated_time" content="2020-07-21T13:44:52+03:00" />
<meta property="og:updated_time" content="2020-07-22T11:00:40+03:00" />
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="CGSpace Notes"/>

2
docs/page/2/index.html
View File

@ -9,7 +9,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
<meta property="og:updated_time" content="2020-07-21T13:44:52+03:00" />
<meta property="og:updated_time" content="2020-07-22T11:00:40+03:00" />
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="CGSpace Notes"/>

2
docs/page/3/index.html
View File

@ -9,7 +9,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
<meta property="og:updated_time" content="2020-07-21T13:44:52+03:00" />
<meta property="og:updated_time" content="2020-07-22T11:00:40+03:00" />
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="CGSpace Notes"/>

2
docs/page/4/index.html
View File

@ -9,7 +9,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
<meta property="og:updated_time" content="2020-07-21T13:44:52+03:00" />
<meta property="og:updated_time" content="2020-07-22T11:00:40+03:00" />
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="CGSpace Notes"/>

2
docs/page/5/index.html
View File

@ -9,7 +9,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
<meta property="og:updated_time" content="2020-07-21T13:44:52+03:00" />
<meta property="og:updated_time" content="2020-07-22T11:00:40+03:00" />
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="CGSpace Notes"/>

2
docs/page/6/index.html
View File

@ -9,7 +9,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
<meta property="og:updated_time" content="2020-07-21T13:44:52+03:00" />
<meta property="og:updated_time" content="2020-07-22T11:00:40+03:00" />
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="CGSpace Notes"/>

2
docs/posts/index.html
View File

@ -9,7 +9,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
<meta property="og:updated_time" content="2020-07-21T13:44:52+03:00" />
<meta property="og:updated_time" content="2020-07-22T11:00:40+03:00" />
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Posts"/>

2
docs/posts/page/2/index.html
View File

@ -9,7 +9,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
<meta property="og:updated_time" content="2020-07-21T13:44:52+03:00" />
<meta property="og:updated_time" content="2020-07-22T11:00:40+03:00" />
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Posts"/>

2
docs/posts/page/3/index.html
View File

@ -9,7 +9,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
<meta property="og:updated_time" content="2020-07-21T13:44:52+03:00" />
<meta property="og:updated_time" content="2020-07-22T11:00:40+03:00" />
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Posts"/>

2
docs/posts/page/4/index.html
View File

@ -9,7 +9,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
<meta property="og:updated_time" content="2020-07-21T13:44:52+03:00" />
<meta property="og:updated_time" content="2020-07-22T11:00:40+03:00" />
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Posts"/>

2
docs/posts/page/5/index.html
View File

@ -9,7 +9,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
<meta property="og:updated_time" content="2020-07-21T13:44:52+03:00" />
<meta property="og:updated_time" content="2020-07-22T11:00:40+03:00" />
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Posts"/>

2
docs/posts/page/6/index.html
View File

@ -9,7 +9,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
<meta property="og:updated_time" content="2020-07-21T13:44:52+03:00" />
<meta property="og:updated_time" content="2020-07-22T11:00:40+03:00" />
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Posts"/>

10
docs/sitemap.xml
View File

@ -4,27 +4,27 @@
<url>
<loc>https://alanorth.github.io/cgspace-notes/categories/</loc>
<lastmod>2020-07-21T13:44:52+03:00</lastmod>
<lastmod>2020-07-22T11:00:40+03:00</lastmod>
</url>
<url>
<loc>https://alanorth.github.io/cgspace-notes/</loc>
<lastmod>2020-07-21T13:44:52+03:00</lastmod>
<lastmod>2020-07-22T11:00:40+03:00</lastmod>
</url>
<url>
<loc>https://alanorth.github.io/cgspace-notes/2020-07/</loc>
<lastmod>2020-07-21T13:44:52+03:00</lastmod>
<lastmod>2020-07-22T11:00:40+03:00</lastmod>
</url>
<url>
<loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
<lastmod>2020-07-21T13:44:52+03:00</lastmod>
<lastmod>2020-07-22T11:00:40+03:00</lastmod>
</url>
<url>
<loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
<lastmod>2020-07-21T13:44:52+03:00</lastmod>
<lastmod>2020-07-22T11:00:40+03:00</lastmod>
</url>
<url>