mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2024-11-22 06:35:03 +01:00
Update notes for 2017-12-20
This commit is contained in:
parent
7e9e49f4b2
commit
0652179a0f
GPG Key ID:
0FB860CC9C45B1B9
@ -423,18 +423,18 @@ $ schedtool -D -e ionice -c2 -n7 nice -n19 dspace index-discovery
|
||||
- Then I create a global `Resource` in the main Tomcat _server.xml_ (inside `GlobalNamingResources`):
|
||||
|
||||
```
|
||||
<Resource name="jdbc/dspace" auth="Container" type="javax.sql.DataSource"
|
||||
driverClassName="org.postgresql.Driver"
|
||||
url="jdbc:postgresql://localhost:5432/dspace"
|
||||
username="dspace"
|
||||
password="dspace"
|
||||
initialSize='5'
|
||||
maxActive='50'
|
||||
maxIdle='15'
|
||||
minIdle='5'
|
||||
maxWait='5000'
|
||||
validationQuery='SELECT 1'
|
||||
testOnBorrow='true' />
|
||||
<Resource name="jdbc/dspace" auth="Container" type="javax.sql.DataSource"
|
||||
driverClassName="org.postgresql.Driver"
|
||||
url="jdbc:postgresql://localhost:5432/dspace"
|
||||
username="dspace"
|
||||
password="dspace"
|
||||
initialSize='5'
|
||||
maxActive='50'
|
||||
maxIdle='15'
|
||||
minIdle='5'
|
||||
maxWait='5000'
|
||||
validationQuery='SELECT 1'
|
||||
testOnBorrow='true' />
|
||||
```
|
||||
|
||||
- Most of the parameters are from comments by Mark Wood about his JNDI setup: https://jira.duraspace.org/browse/DS-3564
|
||||
@ -623,3 +623,30 @@ javax.naming.NoInitialContextException: Need to specify class name in environmen
|
||||
- Oh cool! `select * from pg_stat_activity` shows "PostgreSQL JDBC Driver" for the application name! That's how you know it's working!
|
||||
- If you monitor the `pg_stat_activity` while you run `dspace database info` you can see that it doesn't use the JNDI and creates ~9 extra PostgreSQL connections!
|
||||
- And in the middle of all of this Linode sends an alert that CGSpace has high CPU usage from 2 to 4 PM
|
||||
|
||||
## 2017-12-20
|
||||
|
||||
- The database connection pooling is definitely better!
|
||||
|
||||
|
||||
|
||||
- Now there is only one set of idle connections shared among all the web applications, instead of 10+ per application
|
||||
- There are short bursts of connections up to 10, but it generally stays around 5
|
||||
- Test and import 13 records to CGSpace for Abenet:
|
||||
|
||||
```
|
||||
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx512m -XX:+TieredCompilation -XX:TieredStopAtLevel=1"
|
||||
$ dspace import -a -e aorth@mjanja.ch -s /home/aorth/cg_system_20Dec/SimpleArchiveFormat -m systemoffice.map &> systemoffice.log
|
||||
```
|
||||
|
||||
- The fucking database went from 47 to 72 to 121 connections while I was importing so it stalled.
|
||||
- Since I had to restart Tomcat anyways, I decided to just deploy the new JNDI connection pooling stuff on CGSpace
|
||||
- There was an initial connection storm of 50 PostgreSQL connections, but then it settled down to 7
|
||||
- After that CGSpace came up fine and I was able to import the 13 items just fine:
|
||||
|
||||
```
|
||||
$ dspace import -a -e aorth@mjanja.ch -s /home/aorth/cg_system_20Dec/SimpleArchiveFormat -m systemoffice.map &> systemoffice.log
|
||||
$ schedtool -D -e ionice -c2 -n7 nice -n19 dspace filter-media -i 10568/89287
|
||||
```
|
||||
|
||||
- The final code for the JNDI work in the Ansible infrastructure scripts is here: https://github.com/ilri/rmg-ansible-public/commit/1959d9cb7a0e7a7318c77f769253e5e029bdfa3b
|
||||
|
||||
@ -23,7 +23,7 @@ The list of connections to XMLUI and REST API for today:
|
||||
|
||||
<meta property="article:published_time" content="2017-12-01T13:53:54+03:00"/>
|
||||
|
||||
<meta property="article:modified_time" content="2017-12-19T18:33:19+02:00"/>
|
||||
<meta property="article:modified_time" content="2017-12-19T19:16:47+02:00"/>
|
||||
|
||||
|
||||
|
||||
@ -56,9 +56,9 @@ The list of connections to XMLUI and REST API for today:
|
||||
"@type": "BlogPosting",
|
||||
"headline": "December, 2017",
|
||||
"url": "https://alanorth.github.io/cgspace-notes/2017-12/",
|
||||
"wordCount": "3411",
|
||||
"wordCount": "3593",
|
||||
"datePublished": "2017-12-01T13:53:54+03:00",
|
||||
"dateModified": "2017-12-19T18:33:19+02:00",
|
||||
"dateModified": "2017-12-19T19:16:47+02:00",
|
||||
"author": {
|
||||
"@type": "Person",
|
||||
"name": "Alan Orth"
|
||||
@ -591,18 +591,18 @@ $ schedtool -D -e ionice -c2 -n7 nice -n19 dspace index-discovery
|
||||
<li>Then I create a global <code>Resource</code> in the main Tomcat <em>server.xml</em> (inside <code>GlobalNamingResources</code>):</li>
|
||||
</ul>
|
||||
|
||||
<pre><code> <Resource name="jdbc/dspace" auth="Container" type="javax.sql.DataSource"
|
||||
driverClassName="org.postgresql.Driver"
|
||||
url="jdbc:postgresql://localhost:5432/dspace"
|
||||
username="dspace"
|
||||
password="dspace"
|
||||
initialSize='5'
|
||||
maxActive='50'
|
||||
maxIdle='15'
|
||||
minIdle='5'
|
||||
maxWait='5000'
|
||||
validationQuery='SELECT 1'
|
||||
testOnBorrow='true' />
|
||||
<pre><code><Resource name="jdbc/dspace" auth="Container" type="javax.sql.DataSource"
|
||||
driverClassName="org.postgresql.Driver"
|
||||
url="jdbc:postgresql://localhost:5432/dspace"
|
||||
username="dspace"
|
||||
password="dspace"
|
||||
initialSize='5'
|
||||
maxActive='50'
|
||||
maxIdle='15'
|
||||
minIdle='5'
|
||||
maxWait='5000'
|
||||
validationQuery='SELECT 1'
|
||||
testOnBorrow='true' />
|
||||
</code></pre>
|
||||
|
||||
<ul>
|
||||
@ -811,6 +811,39 @@ javax.naming.NoInitialContextException: Need to specify class name in environmen
|
||||
<li>And in the middle of all of this Linode sends an alert that CGSpace has high CPU usage from 2 to 4 PM</li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2017-12-20">2017-12-20</h2>
|
||||
|
||||
<ul>
|
||||
<li>The database connection pooling is definitely better!</li>
|
||||
</ul>
|
||||
|
||||
<p><img src="/cgspace-notes/2017/12/postgres-connections-week-dspacetest.png" alt="PostgreSQL connection pooling on DSpace Test" /></p>
|
||||
|
||||
<ul>
|
||||
<li>Now there is only one set of idle connections shared among all the web applications, instead of 10+ per application</li>
|
||||
<li>There are short bursts of connections up to 10, but it generally stays around 5</li>
|
||||
<li>Test and import 13 records to CGSpace for Abenet:</li>
|
||||
</ul>
|
||||
|
||||
<pre><code>$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx512m -XX:+TieredCompilation -XX:TieredStopAtLevel=1"
|
||||
$ dspace import -a -e aorth@mjanja.ch -s /home/aorth/cg_system_20Dec/SimpleArchiveFormat -m systemoffice.map &> systemoffice.log
|
||||
</code></pre>
|
||||
|
||||
<ul>
|
||||
<li>The fucking database went from 47 to 72 to 121 connections while I was importing so it stalled.</li>
|
||||
<li>Since I had to restart Tomcat anyways, I decided to just deploy the new JNDI connection pooling stuff on CGSpace</li>
|
||||
<li>There was an initial connection storm of 50 PostgreSQL connections, but then it settled down to 7</li>
|
||||
<li>After that CGSpace came up fine and I was able to import the 13 items just fine:</li>
|
||||
</ul>
|
||||
|
||||
<pre><code>$ dspace import -a -e aorth@mjanja.ch -s /home/aorth/cg_system_20Dec/SimpleArchiveFormat -m systemoffice.map &> systemoffice.log
|
||||
$ schedtool -D -e ionice -c2 -n7 nice -n19 dspace filter-media -i 10568/89287
|
||||
</code></pre>
|
||||
|
||||
<ul>
|
||||
<li>The final code for the JNDI work in the Ansible infrastructure scripts is here: <a href="https://github.com/ilri/rmg-ansible-public/commit/1959d9cb7a0e7a7318c77f769253e5e029bdfa3b">https://github.com/ilri/rmg-ansible-public/commit/1959d9cb7a0e7a7318c77f769253e5e029bdfa3b</a></li>
|
||||
</ul>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
BIN
public/2017/12/postgres-connections-week-dspacetest.png
Normal file
BIN
public/2017/12/postgres-connections-week-dspacetest.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 8.7 KiB |
@ -4,7 +4,7 @@
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/2017-12/</loc>
|
||||
<lastmod>2017-12-19T18:33:19+02:00</lastmod>
|
||||
<lastmod>2017-12-19T19:16:47+02:00</lastmod>
|
||||
</url>
|
||||
|
||||
<url>
|
||||
@ -139,7 +139,7 @@
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/</loc>
|
||||
<lastmod>2017-12-19T18:33:19+02:00</lastmod>
|
||||
<lastmod>2017-12-19T19:16:47+02:00</lastmod>
|
||||
<priority>0</priority>
|
||||
</url>
|
||||
|
||||
@ -150,7 +150,7 @@
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
|
||||
<lastmod>2017-12-19T18:33:19+02:00</lastmod>
|
||||
<lastmod>2017-12-19T19:16:47+02:00</lastmod>
|
||||
<priority>0</priority>
|
||||
</url>
|
||||
|
||||
@ -162,13 +162,13 @@
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/post/</loc>
|
||||
<lastmod>2017-12-19T18:33:19+02:00</lastmod>
|
||||
<lastmod>2017-12-19T19:16:47+02:00</lastmod>
|
||||
<priority>0</priority>
|
||||
</url>
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/tags/</loc>
|
||||
<lastmod>2017-12-19T18:33:19+02:00</lastmod>
|
||||
<lastmod>2017-12-19T19:16:47+02:00</lastmod>
|
||||
<priority>0</priority>
|
||||
</url>
|
||||
|
||||
|
||||
BIN
static/2017/12/postgres-connections-week-dspacetest.png
Normal file
BIN
static/2017/12/postgres-connections-week-dspacetest.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 8.7 KiB |
Loading…
Reference in New Issue
Block a user