mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2024-11-22 06:35:03 +01:00
Update notes for 2018-09-25
This commit is contained in:
parent
bc869de57d
commit
025016667d
GPG Key ID:
0FB860CC9C45B1B9
@ -513,9 +513,7 @@ http://localhost:8081/solr/statistics/update?commit=true&stream.body=<delete><qu
|
||||
- After a few hours the Solr statistics core is down to 44GB on CGSpace!
|
||||
- I did a *major* refactor and logic fix in the DSpace Statistics API's `indexer.py`
|
||||
- Basically, it turns out that using `facet.mincount=1` is really beneficial for me because it reduces the size of the Solr result set, reduces the amount of data we need to ingest into PostgreSQL, and the API returns HTTP 404 Not Found for items without views or downloads anyways
|
||||
- I deployed the new version on CGSpace and now there are only 92,000 pages of item views, which is half that it was before... but still seems like way too many (we only have about 74,000 items, which I'd assume would be 740 pages of 100)
|
||||
- Anyways, the indexing crashed kinda... I think
|
||||
- But systemd seems to have restarted it, and I now see:
|
||||
- I deployed the new version on CGSpace and now it looks pretty good!
|
||||
|
||||
```
|
||||
Indexing item views (page 28 of 753)
|
||||
@ -523,6 +521,6 @@ Indexing item views (page 28 of 753)
|
||||
Indexing item downloads (page 260 of 260)
|
||||
```
|
||||
|
||||
- So that looks really awesome! I wonder if the first crash was due to some caching shit in Solr?
|
||||
- And now it's fast as hell due to the muuuuch smaller Solr statistics core
|
||||
|
||||
<!-- vim: set sw=2 ts=2: -->
|
||||
|
||||
@ -18,7 +18,7 @@ I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I
|
||||
" />
|
||||
<meta property="og:type" content="article" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2018-09/" /><meta property="article:published_time" content="2018-09-02T09:55:54+03:00"/>
|
||||
<meta property="article:modified_time" content="2018-09-25T23:54:29+03:00"/>
|
||||
<meta property="article:modified_time" content="2018-09-26T03:24:46+03:00"/>
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="September, 2018"/>
|
||||
<meta name="twitter:description" content="2018-09-02
|
||||
@ -41,9 +41,9 @@ I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I
|
||||
"@type": "BlogPosting",
|
||||
"headline": "September, 2018",
|
||||
"url": "https://alanorth.github.io/cgspace-notes/2018-09/",
|
||||
"wordCount": "4129",
|
||||
"wordCount": "4073",
|
||||
"datePublished": "2018-09-02T09:55:54+03:00",
|
||||
"dateModified": "2018-09-25T23:54:29+03:00",
|
||||
"dateModified": "2018-09-26T03:24:46+03:00",
|
||||
"author": {
|
||||
"@type": "Person",
|
||||
"name": "Alan Orth"
|
||||
@ -698,9 +698,7 @@ dspacestatistics-> (id INT PRIMARY KEY, views INT DEFAULT 0, downloads INT DE
|
||||
<li>After a few hours the Solr statistics core is down to 44GB on CGSpace!</li>
|
||||
<li>I did a <em>major</em> refactor and logic fix in the DSpace Statistics API’s <code>indexer.py</code></li>
|
||||
<li>Basically, it turns out that using <code>facet.mincount=1</code> is really beneficial for me because it reduces the size of the Solr result set, reduces the amount of data we need to ingest into PostgreSQL, and the API returns HTTP 404 Not Found for items without views or downloads anyways</li>
|
||||
<li>I deployed the new version on CGSpace and now there are only 92,000 pages of item views, which is half that it was before… but still seems like way too many (we only have about 74,000 items, which I’d assume would be 740 pages of 100)</li>
|
||||
<li>Anyways, the indexing crashed kinda… I think</li>
|
||||
<li>But systemd seems to have restarted it, and I now see:</li>
|
||||
<li>I deployed the new version on CGSpace and now it looks pretty good!</li>
|
||||
</ul>
|
||||
|
||||
<pre><code>Indexing item views (page 28 of 753)
|
||||
@ -709,7 +707,7 @@ Indexing item downloads (page 260 of 260)
|
||||
</code></pre>
|
||||
|
||||
<ul>
|
||||
<li>So that looks really awesome! I wonder if the first crash was due to some caching shit in Solr?</li>
|
||||
<li>And now it’s fast as hell due to the muuuuch smaller Solr statistics core</li>
|
||||
</ul>
|
||||
|
||||
<!-- vim: set sw=2 ts=2: -->
|
||||
|
||||
@ -4,7 +4,7 @@
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/2018-09/</loc>
|
||||
<lastmod>2018-09-25T23:54:29+03:00</lastmod>
|
||||
<lastmod>2018-09-26T03:24:46+03:00</lastmod>
|
||||
</url>
|
||||
|
||||
<url>
|
||||
@ -184,7 +184,7 @@
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/</loc>
|
||||
<lastmod>2018-09-25T23:54:29+03:00</lastmod>
|
||||
<lastmod>2018-09-26T03:24:46+03:00</lastmod>
|
||||
<priority>0</priority>
|
||||
</url>
|
||||
|
||||
@ -195,7 +195,7 @@
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
|
||||
<lastmod>2018-09-25T23:54:29+03:00</lastmod>
|
||||
<lastmod>2018-09-26T03:24:46+03:00</lastmod>
|
||||
<priority>0</priority>
|
||||
</url>
|
||||
|
||||
@ -207,13 +207,13 @@
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
|
||||
<lastmod>2018-09-25T23:54:29+03:00</lastmod>
|
||||
<lastmod>2018-09-26T03:24:46+03:00</lastmod>
|
||||
<priority>0</priority>
|
||||
</url>
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/tags/</loc>
|
||||
<lastmod>2018-09-25T23:54:29+03:00</lastmod>
|
||||
<lastmod>2018-09-26T03:24:46+03:00</lastmod>
|
||||
<priority>0</priority>
|
||||
</url>
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user