alanorth/cgspace-notes
1
0
Fork 0
mirror of https://github.com/alanorth/cgspace-notes.git synced 2024-11-22 14:45:03 +01:00
Code Issues Projects Releases Wiki Activity

Update notes for 2018-09-25

Browse Source
This commit is contained in:
Alan Orth 2018-09-26 03:24:46 +03:00
parent fd2cca6fd5
commit bc869de57d
Signed by: alanorth
GPG Key ID: 0FB860CC9C45B1B9
3 changed files with 37 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
content/posts/2018-09.md
View File

@ -510,5 +510,19 @@ http://localhost:8081/solr/statistics/update?commit=true&stream.body=<delete><qu
```
- And magically all those 81,000 documents are gone!
- After a few hours the Solr statistics core is down to 44GB on CGSpace!
- I did a *major* refactor and logic fix in the DSpace Statistics API's `indexer.py`
- Basically, it turns out that using `facet.mincount=1` is really beneficial for me because it reduces the size of the Solr result set, reduces the amount of data we need to ingest into PostgreSQL, and the API returns HTTP 404 Not Found for items without views or downloads anyways
- I deployed the new version on CGSpace and now there are only 92,000 pages of item views, which is half that it was before... but still seems like way too many (we only have about 74,000 items, which I'd assume would be 740 pages of 100)
- Anyways, the indexing crashed kinda... I think
- But systemd seems to have restarted it, and I now see:
```
Indexing item views (page 28 of 753)
...
Indexing item downloads (page 260 of 260)
```
- So that looks really awesome! I wonder if the first crash was due to some caching shit in Solr?
<!-- vim: set sw=2 ts=2: -->

21
docs/2018-09/index.html
View File

@ -18,7 +18,7 @@ I&rsquo;m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I
" />
<meta property="og:type" content="article" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2018-09/" /><meta property="article:published_time" content="2018-09-02T09:55:54&#43;03:00"/>
<meta property="article:modified_time" content="2018-09-25T22:06:05&#43;03:00"/>
<meta property="article:modified_time" content="2018-09-25T23:54:29&#43;03:00"/>
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="September, 2018"/>
<meta name="twitter:description" content="2018-09-02
@ -41,9 +41,9 @@ I&rsquo;m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I
"@type": "BlogPosting",
"headline": "September, 2018",
"url": "https://alanorth.github.io/cgspace-notes/2018-09/",
"wordCount": "3955",
"wordCount": "4129",
"datePublished": "2018-09-02T09:55:54&#43;03:00",
"dateModified": "2018-09-25T22:06:05&#43;03:00",
"dateModified": "2018-09-25T23:54:29&#43;03:00",
"author": {
"@type": "Person",
"name": "Alan Orth"
@ -695,6 +695,21 @@ dspacestatistics-&gt; (id INT PRIMARY KEY, views INT DEFAULT 0, downloads INT DE
<ul>
<li>And magically all those 81,000 documents are gone!</li>
<li>After a few hours the Solr statistics core is down to 44GB on CGSpace!</li>
<li>I did a <em>major</em> refactor and logic fix in the DSpace Statistics API&rsquo;s <code>indexer.py</code></li>
<li>Basically, it turns out that using <code>facet.mincount=1</code> is really beneficial for me because it reduces the size of the Solr result set, reduces the amount of data we need to ingest into PostgreSQL, and the API returns HTTP 404 Not Found for items without views or downloads anyways</li>
<li>I deployed the new version on CGSpace and now there are only 92,000 pages of item views, which is half that it was before&hellip; but still seems like way too many (we only have about 74,000 items, which I&rsquo;d assume would be 740 pages of 100)</li>
<li>Anyways, the indexing crashed kinda&hellip; I think</li>
<li>But systemd seems to have restarted it, and I now see:</li>
</ul>
<pre><code>Indexing item views (page 28 of 753)
...
Indexing item downloads (page 260 of 260)
</code></pre>
<ul>
<li>So that looks really awesome! I wonder if the first crash was due to some caching shit in Solr?</li>
</ul>
<!-- vim: set sw=2 ts=2: -->

10
docs/sitemap.xml
View File

@ -4,7 +4,7 @@
<url>
<loc>https://alanorth.github.io/cgspace-notes/2018-09/</loc>
<lastmod>2018-09-25T22:06:05+03:00</lastmod>
<lastmod>2018-09-25T23:54:29+03:00</lastmod>
</url>
<url>
@ -184,7 +184,7 @@
<url>
<loc>https://alanorth.github.io/cgspace-notes/</loc>
<lastmod>2018-09-25T22:06:05+03:00</lastmod>
<lastmod>2018-09-25T23:54:29+03:00</lastmod>
<priority>0</priority>
</url>
@ -195,7 +195,7 @@
<url>
<loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
<lastmod>2018-09-25T22:06:05+03:00</lastmod>
<lastmod>2018-09-25T23:54:29+03:00</lastmod>
<priority>0</priority>
</url>
@ -207,13 +207,13 @@
<url>
<loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
<lastmod>2018-09-25T22:06:05+03:00</lastmod>
<lastmod>2018-09-25T23:54:29+03:00</lastmod>
<priority>0</priority>
</url>
<url>
<loc>https://alanorth.github.io/cgspace-notes/tags/</loc>
<lastmod>2018-09-25T22:06:05+03:00</lastmod>
<lastmod>2018-09-25T23:54:29+03:00</lastmod>
<priority>0</priority>
</url>