mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2025-01-27 05:49:12 +01:00
Add notes for 2020-01-27
This commit is contained in:
@ -27,7 +27,7 @@ The logs say “Timeout waiting for idle object”
|
||||
PostgreSQL activity says there are 115 connections currently
|
||||
The list of connections to XMLUI and REST API for today:
|
||||
"/>
|
||||
<meta name="generator" content="Hugo 0.62.2" />
|
||||
<meta name="generator" content="Hugo 0.63.1" />
|
||||
|
||||
|
||||
|
||||
@ -57,7 +57,7 @@ The list of connections to XMLUI and REST API for today:
|
||||
|
||||
<!-- combined, minified CSS -->
|
||||
|
||||
<link href="https://alanorth.github.io/cgspace-notes/css/style.a20c1a4367639632cdb341d23c27ca44fedcc75b0f8b3cbea6203010da153d3c.css" rel="stylesheet" integrity="sha256-ogwaQ2djljLNs0HSPCfKRP7cx1sPizy+piAwENoVPTw=" crossorigin="anonymous">
|
||||
<link href="https://alanorth.github.io/cgspace-notes/css/style.23e2c3298bcc8c1136c19aba330c211ec94c36f7c4454ea15cf4d3548370042a.css" rel="stylesheet" integrity="sha256-I+LDKYvMjBE2wZq6MwwhHslMNvfERU6hXPTTVINwBCo=" crossorigin="anonymous">
|
||||
|
||||
|
||||
<!-- RSS 2.0 feed -->
|
||||
@ -104,7 +104,7 @@ The list of connections to XMLUI and REST API for today:
|
||||
<header>
|
||||
<h2 class="blog-post-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/2017-12/">December, 2017</a></h2>
|
||||
<p class="blog-post-meta"><time datetime="2017-12-01T13:53:54+03:00">Fri Dec 01, 2017</time> by Alan Orth in
|
||||
<i class="fa fa-folder" aria-hidden="true"></i> <a href="/cgspace-notes/categories/notes" rel="category tag">Notes</a>
|
||||
<span class="fas fa-folder" aria-hidden="true"></span> <a href="/cgspace-notes/categories/notes" rel="category tag">Notes</a>
|
||||
|
||||
|
||||
</p>
|
||||
@ -128,7 +128,7 @@ The list of connections to XMLUI and REST API for today:
|
||||
4007 70.32.83.92
|
||||
6061 45.5.184.196
|
||||
</code></pre><ul>
|
||||
<li>The number of DSpace sessions isn't even that high:</li>
|
||||
<li>The number of DSpace sessions isn’t even that high:</li>
|
||||
</ul>
|
||||
<pre><code>$ cat /home/cgspace.cgiar.org/log/dspace.log.2017-12-01 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
|
||||
5815
|
||||
@ -148,7 +148,7 @@ The list of connections to XMLUI and REST API for today:
|
||||
314 2.86.122.76
|
||||
</code></pre><ul>
|
||||
<li>What the fuck is going on?</li>
|
||||
<li>I've never seen this 2.86.122.76 before, it has made quite a few unique Tomcat sessions today:</li>
|
||||
<li>I’ve never seen this 2.86.122.76 before, it has made quite a few unique Tomcat sessions today:</li>
|
||||
</ul>
|
||||
<pre><code>$ grep 2.86.122.76 /home/cgspace.cgiar.org/log/dspace.log.2017-12-01 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
|
||||
822
|
||||
@ -169,20 +169,20 @@ The list of connections to XMLUI and REST API for today:
|
||||
319 2001:4b99:1:1:216:3eff:fe76:205b
|
||||
</code></pre><h2 id="2017-12-03">2017-12-03</h2>
|
||||
<ul>
|
||||
<li>Linode alerted that CGSpace's load was 327.5% from 6 to 8 AM again</li>
|
||||
<li>Linode alerted that CGSpace’s load was 327.5% from 6 to 8 AM again</li>
|
||||
</ul>
|
||||
<h2 id="2017-12-04">2017-12-04</h2>
|
||||
<ul>
|
||||
<li>Linode alerted that CGSpace's load was 255.5% from 8 to 10 AM again</li>
|
||||
<li>Linode alerted that CGSpace’s load was 255.5% from 8 to 10 AM again</li>
|
||||
<li>I looked at the Munin stats on DSpace Test (linode02) again to see how the PostgreSQL tweaks from a few weeks ago were holding up:</li>
|
||||
</ul>
|
||||
<p><img src="/cgspace-notes/2017/12/postgres-connections-month.png" alt="DSpace Test PostgreSQL connections month"></p>
|
||||
<ul>
|
||||
<li>The results look fantastic! So the <code>random_page_cost</code> tweak is massively important for informing the PostgreSQL scheduler that there is no “cost” to accessing random pages, as we're on an SSD!</li>
|
||||
<li>The results look fantastic! So the <code>random_page_cost</code> tweak is massively important for informing the PostgreSQL scheduler that there is no “cost” to accessing random pages, as we’re on an SSD!</li>
|
||||
<li>I guess we could probably even reduce the PostgreSQL connections in DSpace / PostgreSQL after using this</li>
|
||||
<li>Run system updates on DSpace Test (linode02) and reboot it</li>
|
||||
<li>I'm going to enable the PostgreSQL <code>random_page_cost</code> tweak on CGSpace</li>
|
||||
<li>For reference, here is the past month's connections:</li>
|
||||
<li>I’m going to enable the PostgreSQL <code>random_page_cost</code> tweak on CGSpace</li>
|
||||
<li>For reference, here is the past month’s connections:</li>
|
||||
</ul>
|
||||
<p><img src="/cgspace-notes/2017/12/postgres-connections-month-cgspace.png" alt="CGSpace PostgreSQL connections month"></p>
|
||||
<h2 id="2017-12-05">2017-12-05</h2>
|
||||
@ -196,8 +196,8 @@ The list of connections to XMLUI and REST API for today:
|
||||
<li>Linode alerted again that the CPU usage on CGSpace was high this morning from 6 to 8 AM</li>
|
||||
<li>Uptime Robot alerted that the server went down and up around 8:53 this morning</li>
|
||||
<li>Uptime Robot alerted that CGSpace was down and up again a few minutes later</li>
|
||||
<li>I don't see any errors in the DSpace logs but I see in nginx's access.log that UptimeRobot was returned with HTTP 499 status (Client Closed Request)</li>
|
||||
<li>Looking at the REST API logs I see some new client IP I haven't noticed before:</li>
|
||||
<li>I don’t see any errors in the DSpace logs but I see in nginx’s access.log that UptimeRobot was returned with HTTP 499 status (Client Closed Request)</li>
|
||||
<li>Looking at the REST API logs I see some new client IP I haven’t noticed before:</li>
|
||||
</ul>
|
||||
<pre><code># cat /var/log/nginx/rest.log /var/log/nginx/rest.log.1 | grep -E "6/Dec/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
|
||||
18 95.108.181.88
|
||||
@ -233,7 +233,7 @@ The list of connections to XMLUI and REST API for today:
|
||||
2662 66.249.66.219
|
||||
5110 124.17.34.60
|
||||
</code></pre><ul>
|
||||
<li>We've never seen 124.17.34.60 yet, but it's really hammering us!</li>
|
||||
<li>We’ve never seen 124.17.34.60 yet, but it’s really hammering us!</li>
|
||||
<li>Apparently it is from China, and here is one of its user agents:</li>
|
||||
</ul>
|
||||
<pre><code>Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.2; Win64; x64; Trident/7.0; LCTE)
|
||||
@ -243,7 +243,7 @@ The list of connections to XMLUI and REST API for today:
|
||||
<pre><code>$ grep 124.17.34.60 /home/cgspace.cgiar.org/log/dspace.log.2017-12-07 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
|
||||
4574
|
||||
</code></pre><ul>
|
||||
<li>I've adjusted the nginx IP mapping that I set up last month to account for 124.17.34.60 and 124.17.34.59 using a regex, as it's the same bot on the same subnet</li>
|
||||
<li>I’ve adjusted the nginx IP mapping that I set up last month to account for 124.17.34.60 and 124.17.34.59 using a regex, as it’s the same bot on the same subnet</li>
|
||||
<li>I was running the DSpace cleanup task manually and it hit an error:</li>
|
||||
</ul>
|
||||
<pre><code>$ /home/cgspace.cgiar.org/bin/dspace cleanup -v
|
||||
@ -261,7 +261,7 @@ UPDATE 1
|
||||
</ul>
|
||||
<h2 id="2017-12-16">2017-12-16</h2>
|
||||
<ul>
|
||||
<li>Re-work the XMLUI base theme to allow child themes to override the header logo's image and link destination: <a href="https://github.com/ilri/DSpace/pull/349">#349</a></li>
|
||||
<li>Re-work the XMLUI base theme to allow child themes to override the header logo’s image and link destination: <a href="https://github.com/ilri/DSpace/pull/349">#349</a></li>
|
||||
<li>This required a little bit of work to restructure the XSL templates</li>
|
||||
<li>Optimize PNG and SVG image assets in the CGIAR base theme using pngquant and svgo: <a href="https://github.com/ilri/DSpace/pull/350">#350</a></li>
|
||||
</ul>
|
||||
@ -276,7 +276,7 @@ UPDATE 1
|
||||
<li>I also had to add the .jpg to the thumbnail string in the CSV</li>
|
||||
<li>The thumbnail11.jpg is missing</li>
|
||||
<li>The dates are in super long ISO8601 format (from Excel?) like <code>2016-02-07T00:00:00Z</code> so I converted them to simpler forms in GREL: <code>value.toString("yyyy-MM-dd")</code></li>
|
||||
<li>I trimmed the whitespaces in a few fields but it wasn't many</li>
|
||||
<li>I trimmed the whitespaces in a few fields but it wasn’t many</li>
|
||||
<li>Rename her thumbnail column to filename, and format it so SAFBuilder adds the files to the thumbnail bundle with this GREL in OpenRefine: <code>value + "__bundle:THUMBNAIL"</code></li>
|
||||
<li>Rename dc.identifier.status and dc.identifier.url columns to cg.identifier.status and cg.identifier.url</li>
|
||||
<li>Item 4 has weird characters in citation, ie: Nagoya et de Trait</li>
|
||||
@ -289,7 +289,7 @@ UPDATE 1
|
||||
</ul>
|
||||
<pre><code>$ JAVA_OPTS="-Xmx512m -Dfile.encoding=UTF-8" ~/dspace/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568/89338 --source /Users/aorth/Downloads/2016\ bulk\ upload\ thumbnails/SimpleArchiveFormat --mapfile=/tmp/ccafs.map &> /tmp/ccafs.log
|
||||
</code></pre><ul>
|
||||
<li>It's the same on DSpace Test, I can't import the SAF bundle without specifying the collection:</li>
|
||||
<li>It’s the same on DSpace Test, I can’t import the SAF bundle without specifying the collection:</li>
|
||||
</ul>
|
||||
<pre><code>$ dspace import --add --eperson=aorth@mjanja.ch --mapfile=/tmp/ccafs.map --source=/tmp/ccafs-2016/SimpleArchiveFormat
|
||||
No collections given. Assuming 'collections' file inside item directory
|
||||
@ -317,7 +317,7 @@ Elapsed time: 2 secs (2559 msecs)
|
||||
<pre><code>-Dlog4j.configuration=file:/Users/aorth/dspace/config/log4j-console.properties -Ddspace.log.init.disable=true
|
||||
</code></pre><ul>
|
||||
<li>… but the error message was the same, just with more INFO noise around it</li>
|
||||
<li>For now I'll import into a collection in DSpace Test but I'm really not sure what's up with this!</li>
|
||||
<li>For now I’ll import into a collection in DSpace Test but I’m really not sure what’s up with this!</li>
|
||||
<li>Linode alerted that CGSpace was using high CPU from 4 to 6 PM</li>
|
||||
<li>The logs for today show the CORE bot (137.108.70.7) being active in XMLUI:</li>
|
||||
</ul>
|
||||
@ -347,7 +347,7 @@ Elapsed time: 2 secs (2559 msecs)
|
||||
4014 70.32.83.92
|
||||
11030 45.5.184.196
|
||||
</code></pre><ul>
|
||||
<li>That's probably ok, as I don't think the REST API connections use up a Tomcat session…</li>
|
||||
<li>That’s probably ok, as I don’t think the REST API connections use up a Tomcat session…</li>
|
||||
<li>CIP emailed a few days ago to ask about unique IDs for authors and organizations, and if we can provide them via an API</li>
|
||||
<li>Regarding the import issue above it seems to be a known issue that has a patch in DSpace 5.7:
|
||||
<ul>
|
||||
@ -355,7 +355,7 @@ Elapsed time: 2 secs (2559 msecs)
|
||||
<li><a href="https://jira.duraspace.org/browse/DS-3583">https://jira.duraspace.org/browse/DS-3583</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>We're on DSpace 5.5 but there is a one-word fix to the addItem() function here: <a href="https://github.com/DSpace/DSpace/pull/1731">https://github.com/DSpace/DSpace/pull/1731</a></li>
|
||||
<li>We’re on DSpace 5.5 but there is a one-word fix to the addItem() function here: <a href="https://github.com/DSpace/DSpace/pull/1731">https://github.com/DSpace/DSpace/pull/1731</a></li>
|
||||
<li>I will apply it on our branch but I need to make a note to NOT cherry-pick it when I rebase on to the latest 5.x upstream later</li>
|
||||
<li>Pull request: <a href="https://github.com/ilri/DSpace/pull/351">#351</a></li>
|
||||
</ul>
|
||||
@ -393,7 +393,7 @@ Elapsed time: 2 secs (2559 msecs)
|
||||
<li>I need to keep an eye on this issue because it has nice fixes for reducing the number of database connections in DSpace 5.7: <a href="https://jira.duraspace.org/browse/DS-3551">https://jira.duraspace.org/browse/DS-3551</a></li>
|
||||
<li>Update text on CGSpace about page to give some tips to developers about using the resources more wisely (<a href="https://github.com/ilri/DSpace/pull/352">#352</a>)</li>
|
||||
<li>Linode alerted that CGSpace was using 396.3% CPU from 12 to 2 PM</li>
|
||||
<li>The REST and OAI API logs look pretty much the same as earlier this morning, but there's a new IP harvesting XMLUI:</li>
|
||||
<li>The REST and OAI API logs look pretty much the same as earlier this morning, but there’s a new IP harvesting XMLUI:</li>
|
||||
</ul>
|
||||
<pre><code># cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -E "18/Dec/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
|
||||
360 95.108.181.88
|
||||
@ -416,8 +416,8 @@ Elapsed time: 2 secs (2559 msecs)
|
||||
<pre><code>$ grep 2.86.72.181 dspace.log.2017-12-18 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
|
||||
1
|
||||
</code></pre><ul>
|
||||
<li>I guess there's nothing I can do to them for now</li>
|
||||
<li>In other news, I am curious how many PostgreSQL connection pool errors we've had in the last month:</li>
|
||||
<li>I guess there’s nothing I can do to them for now</li>
|
||||
<li>In other news, I am curious how many PostgreSQL connection pool errors we’ve had in the last month:</li>
|
||||
</ul>
|
||||
<pre><code>$ grep -c "Cannot get a connection, pool error Timeout waiting for idle object" dspace.log.2017-1* | grep -v :0
|
||||
dspace.log.2017-11-07:15695
|
||||
@ -430,9 +430,9 @@ dspace.log.2017-12-01:1601
|
||||
dspace.log.2017-12-02:1274
|
||||
dspace.log.2017-12-07:2769
|
||||
</code></pre><ul>
|
||||
<li>I made a small fix to my <code>move-collections.sh</code> script so that it handles the case when a “to” or “from” community doesn't exist</li>
|
||||
<li>I made a small fix to my <code>move-collections.sh</code> script so that it handles the case when a “to” or “from” community doesn’t exist</li>
|
||||
<li>The script lives here: <a href="https://gist.github.com/alanorth/e60b530ed4989df0c731afbb0c640515">https://gist.github.com/alanorth/e60b530ed4989df0c731afbb0c640515</a></li>
|
||||
<li>Major reorganization of four of CTA's French collections</li>
|
||||
<li>Major reorganization of four of CTA’s French collections</li>
|
||||
<li>Basically moving their items into the English ones, then moving the English ones to the top-level of the CTA community, and deleting the old sub-communities</li>
|
||||
<li>Move collection 10568/51821 from 10568/42212 to 10568/42211</li>
|
||||
<li>Move collection 10568/51400 from 10568/42214 to 10568/42211</li>
|
||||
@ -457,21 +457,21 @@ $ schedtool -D -e ionice -c2 -n7 nice -n19 dspace index-discovery
|
||||
<h2 id="2017-12-19">2017-12-19</h2>
|
||||
<ul>
|
||||
<li>Briefly had PostgreSQL connection issues on CGSpace for the millionth time</li>
|
||||
<li>I'm fucking sick of this!</li>
|
||||
<li>I’m fucking sick of this!</li>
|
||||
<li>The connection graph on CGSpace shows shit tons of connections idle</li>
|
||||
</ul>
|
||||
<p><img src="/cgspace-notes/2017/12/postgres-connections-month-cgspace-2.png" alt="Idle PostgreSQL connections on CGSpace"></p>
|
||||
<ul>
|
||||
<li>And I only now just realized that DSpace's <code>db.maxidle</code> parameter is not seconds, but number of idle connections to allow.</li>
|
||||
<li>And I only now just realized that DSpace’s <code>db.maxidle</code> parameter is not seconds, but number of idle connections to allow.</li>
|
||||
<li>So theoretically, because each webapp has its own pool, this could be 20 per app—so no wonder we have 50 idle connections!</li>
|
||||
<li>I notice that this number will be set to 10 by default in DSpace 6.1 and 7.0: <a href="https://jira.duraspace.org/browse/DS-3564">https://jira.duraspace.org/browse/DS-3564</a></li>
|
||||
<li>So I'm going to reduce ours from 20 to 10 and start trying to figure out how the hell to supply a database pool using Tomcat JNDI</li>
|
||||
<li>So I’m going to reduce ours from 20 to 10 and start trying to figure out how the hell to supply a database pool using Tomcat JNDI</li>
|
||||
<li>I re-deployed the <code>5_x-prod</code> branch on CGSpace, applied all system updates, and restarted the server</li>
|
||||
<li>Looking through the dspace.log I see this error:</li>
|
||||
</ul>
|
||||
<pre><code>2017-12-19 08:17:15,740 ERROR org.dspace.statistics.SolrLogger @ Error CREATEing SolrCore 'statistics-2010': Unable to create core [statistics-2010] Caused by: Lock obtain timed out: NativeFSLock@/home/cgspace.cgiar.org/solr/statistics-2010/data/index/write.lock
|
||||
</code></pre><ul>
|
||||
<li>I don't have time now to look into this but the Solr sharding has long been an issue!</li>
|
||||
<li>I don’t have time now to look into this but the Solr sharding has long been an issue!</li>
|
||||
<li>Looking into using JDBC / JNDI to provide a database pool to DSpace</li>
|
||||
<li>The <a href="https://wiki.duraspace.org/display/DSDOC6x/Configuration+Reference">DSpace 6.x configuration docs</a> have more notes about setting up the database pool than the 5.x ones (which actually have none!)</li>
|
||||
<li>First, I uncomment <code>db.jndi</code> in <em>dspace/config/dspace.cfg</em></li>
|
||||
@ -496,7 +496,7 @@ $ schedtool -D -e ionice -c2 -n7 nice -n19 dspace index-discovery
|
||||
<pre><code><ResourceLink global="jdbc/dspace" name="jdbc/dspace" type="javax.sql.DataSource"/>
|
||||
</code></pre><ul>
|
||||
<li>I am not sure why several guides show configuration snippets for <em>server.xml</em> and web application contexts that use a Local and Global jdbc…</li>
|
||||
<li>When DSpace can't find the JNDI context (for whatever reason) you will see this in the dspace logs:</li>
|
||||
<li>When DSpace can’t find the JNDI context (for whatever reason) you will see this in the dspace logs:</li>
|
||||
</ul>
|
||||
<pre><code>2017-12-19 13:12:08,796 ERROR org.dspace.storage.rdbms.DatabaseManager @ Error retrieving JNDI context: jdbc/dspace
|
||||
javax.naming.NameNotFoundException: Name [jdbc/dspace] is not bound in this Context. Unable to find [jdbc].
|
||||
@ -547,31 +547,31 @@ javax.naming.NameNotFoundException: Name [jdbc/dspace] is not bound in this Cont
|
||||
<version>9.1-901-1.jdbc4</version>
|
||||
</dependency>
|
||||
</code></pre><ul>
|
||||
<li>So WTF? Let's try copying one to Tomcat's lib folder and restarting Tomcat:</li>
|
||||
<li>So WTF? Let’s try copying one to Tomcat’s lib folder and restarting Tomcat:</li>
|
||||
</ul>
|
||||
<pre><code>$ cp ~/dspace/lib/postgresql-9.1-901-1.jdbc4.jar /usr/local/opt/tomcat@7/libexec/lib
|
||||
</code></pre><ul>
|
||||
<li>Oh that's fantastic, now at least Tomcat doesn't print an error during startup so I guess it succeeds to create the JNDI pool</li>
|
||||
<li>DSpace starts up but I have no idea if it's using the JNDI configuration because I see this in the logs:</li>
|
||||
<li>Oh that’s fantastic, now at least Tomcat doesn’t print an error during startup so I guess it succeeds to create the JNDI pool</li>
|
||||
<li>DSpace starts up but I have no idea if it’s using the JNDI configuration because I see this in the logs:</li>
|
||||
</ul>
|
||||
<pre><code>2017-12-19 13:26:54,271 INFO org.dspace.storage.rdbms.DatabaseManager @ DBMS is '{}'PostgreSQL
|
||||
2017-12-19 13:26:54,277 INFO org.dspace.storage.rdbms.DatabaseManager @ DBMS driver version is '{}'9.5.10
|
||||
2017-12-19 13:26:54,293 INFO org.dspace.storage.rdbms.DatabaseUtils @ Loading Flyway DB migrations from: filesystem:/Users/aorth/dspace/etc/postgres, classpath:org.dspace.storage.rdbms.sqlmigration.postgres, classpath:org.dspace.storage.rdbms.migration
|
||||
2017-12-19 13:26:54,306 INFO org.flywaydb.core.internal.dbsupport.DbSupportFactory @ Database: jdbc:postgresql://localhost:5432/dspacetest (PostgreSQL 9.5)
|
||||
</code></pre><ul>
|
||||
<li>Let's try again, but this time explicitly blank the PostgreSQL connection parameters in dspace.cfg and see if DSpace starts…</li>
|
||||
<li>Wow, ok, that works, but having to copy the PostgreSQL JDBC JAR to Tomcat's lib folder totally blows</li>
|
||||
<li>Also, it's likely this is only a problem on my local macOS + Tomcat test environment</li>
|
||||
<li>Ubuntu's Tomcat distribution will probably handle this differently</li>
|
||||
<li>Let’s try again, but this time explicitly blank the PostgreSQL connection parameters in dspace.cfg and see if DSpace starts…</li>
|
||||
<li>Wow, ok, that works, but having to copy the PostgreSQL JDBC JAR to Tomcat’s lib folder totally blows</li>
|
||||
<li>Also, it’s likely this is only a problem on my local macOS + Tomcat test environment</li>
|
||||
<li>Ubuntu’s Tomcat distribution will probably handle this differently</li>
|
||||
<li>So for reference I have:
|
||||
<ul>
|
||||
<li>a <code><Resource></code> defined globally in server.xml</li>
|
||||
<li>a <code><ResourceLink></code> defined in each web application's context XML</li>
|
||||
<li>a <code><ResourceLink></code> defined in each web application’s context XML</li>
|
||||
<li>unset the <code>db.url</code>, <code>db.username</code>, and <code>db.password</code> parameters in dspace.cfg</li>
|
||||
<li>set the <code>db.jndi</code> in dspace.cfg to the name specified in the web application context</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>After adding the <code>Resource</code> to <em>server.xml</em> on Ubuntu I get this in Catalina's logs:</li>
|
||||
<li>After adding the <code>Resource</code> to <em>server.xml</em> on Ubuntu I get this in Catalina’s logs:</li>
|
||||
</ul>
|
||||
<pre><code>SEVERE: Unable to create initial connections of pool.
|
||||
java.sql.SQLException: org.postgresql.Driver
|
||||
@ -579,8 +579,8 @@ java.sql.SQLException: org.postgresql.Driver
|
||||
Caused by: java.lang.ClassNotFoundException: org.postgresql.Driver
|
||||
</code></pre><ul>
|
||||
<li>The username and password are correct, but maybe I need to copy the fucking lib there too?</li>
|
||||
<li>I tried installing Ubuntu's <code>libpostgresql-jdbc-java</code> package but Tomcat still can't find the class</li>
|
||||
<li>Let me try to symlink the lib into Tomcat's libs:</li>
|
||||
<li>I tried installing Ubuntu’s <code>libpostgresql-jdbc-java</code> package but Tomcat still can’t find the class</li>
|
||||
<li>Let me try to symlink the lib into Tomcat’s libs:</li>
|
||||
</ul>
|
||||
<pre><code># ln -sv /usr/share/java/postgresql.jar /usr/share/tomcat7/lib
|
||||
</code></pre><ul>
|
||||
@ -589,17 +589,17 @@ Caused by: java.lang.ClassNotFoundException: org.postgresql.Driver
|
||||
<pre><code>SEVERE: Exception sending context initialized event to listener instance of class org.dspace.app.util.DSpaceContextListener
|
||||
java.lang.AbstractMethodError: Method org/postgresql/jdbc3/Jdbc3ResultSet.isClosed()Z is abstract
|
||||
</code></pre><ul>
|
||||
<li>Could be a version issue or something since the Ubuntu package provides 9.2 and DSpace's are 9.1…</li>
|
||||
<li>Let me try to remove it and copy in DSpace's:</li>
|
||||
<li>Could be a version issue or something since the Ubuntu package provides 9.2 and DSpace’s are 9.1…</li>
|
||||
<li>Let me try to remove it and copy in DSpace’s:</li>
|
||||
</ul>
|
||||
<pre><code># rm /usr/share/tomcat7/lib/postgresql.jar
|
||||
# cp [dspace]/webapps/xmlui/WEB-INF/lib/postgresql-9.1-901-1.jdbc4.jar /usr/share/tomcat7/lib/
|
||||
</code></pre><ul>
|
||||
<li>Wow, I think that actually works…</li>
|
||||
<li>I wonder if I could get the JDBC driver from postgresql.org instead of relying on the one from the DSpace build: <a href="https://jdbc.postgresql.org/">https://jdbc.postgresql.org/</a></li>
|
||||
<li>I notice our version is 9.1-901, which isn't even available anymore! The latest in the archived versions is 9.1-903</li>
|
||||
<li>I notice our version is 9.1-901, which isn’t even available anymore! The latest in the archived versions is 9.1-903</li>
|
||||
<li>Also, since I commented out all the db parameters in DSpace.cfg, how does the command line <code>dspace</code> tool work?</li>
|
||||
<li>Let's try the upstream JDBC driver first:</li>
|
||||
<li>Let’s try the upstream JDBC driver first:</li>
|
||||
</ul>
|
||||
<pre><code># rm /usr/share/tomcat7/lib/postgresql-9.1-901-1.jdbc4.jar
|
||||
# wget https://jdbc.postgresql.org/download/postgresql-42.1.4.jar -O /usr/share/tomcat7/lib/postgresql-42.1.4.jar
|
||||
@ -648,8 +648,8 @@ javax.naming.NoInitialContextException: Need to specify class name in environmen
|
||||
</code></pre><ul>
|
||||
<li>If I add the db values back to dspace.cfg the <code>dspace database info</code> command succeeds but the log still shows errors retrieving the JNDI connection</li>
|
||||
<li>Perhaps something to report to the dspace-tech mailing list when I finally send my comments</li>
|
||||
<li>Oh cool! <code>select * from pg_stat_activity</code> shows “PostgreSQL JDBC Driver” for the application name! That's how you know it's working!</li>
|
||||
<li>If you monitor the <code>pg_stat_activity</code> while you run <code>dspace database info</code> you can see that it doesn't use the JNDI and creates ~9 extra PostgreSQL connections!</li>
|
||||
<li>Oh cool! <code>select * from pg_stat_activity</code> shows “PostgreSQL JDBC Driver” for the application name! That’s how you know it’s working!</li>
|
||||
<li>If you monitor the <code>pg_stat_activity</code> while you run <code>dspace database info</code> you can see that it doesn’t use the JNDI and creates ~9 extra PostgreSQL connections!</li>
|
||||
<li>And in the middle of all of this Linode sends an alert that CGSpace has high CPU usage from 2 to 4 PM</li>
|
||||
</ul>
|
||||
<h2 id="2017-12-20">2017-12-20</h2>
|
||||
@ -678,14 +678,14 @@ $ schedtool -D -e ionice -c2 -n7 nice -n19 dspace filter-media -i 10568/89287
|
||||
<h2 id="2017-12-24">2017-12-24</h2>
|
||||
<ul>
|
||||
<li>Linode alerted that CGSpace was using high CPU this morning around 6 AM</li>
|
||||
<li>I'm playing with reading all of a month's nginx logs into goaccess:</li>
|
||||
<li>I’m playing with reading all of a month’s nginx logs into goaccess:</li>
|
||||
</ul>
|
||||
<pre><code># find /var/log/nginx -type f -newermt "2017-12-01" | xargs zcat --force | goaccess --log-format=COMBINED -
|
||||
</code></pre><ul>
|
||||
<li>I can see interesting things using this approach, for example:
|
||||
<ul>
|
||||
<li>50.116.102.77 checked our status almost 40,000 times so far this month—I think it's the CGNet uptime tool</li>
|
||||
<li>Also, we've handled 2.9 million requests this month from 172,000 unique IP addresses!</li>
|
||||
<li>50.116.102.77 checked our status almost 40,000 times so far this month—I think it’s the CGNet uptime tool</li>
|
||||
<li>Also, we’ve handled 2.9 million requests this month from 172,000 unique IP addresses!</li>
|
||||
<li>Total bandwidth so far this month is 640GiB</li>
|
||||
<li>The user that made the most requests so far this month is 45.5.184.196 (267,000 requests)</li>
|
||||
</ul>
|
||||
@ -720,13 +720,13 @@ UPDATE 5
|
||||
# delete from metadatavalue where resource_type_id=2 and metadata_field_id=38 and text_value ~ '(dc.language.iso|CGIAR Challenge Program on Water and Food)';
|
||||
DELETE 20
|
||||
</code></pre><ul>
|
||||
<li>I need to figure out why we have records with language <code>in</code> because that's not a language!</li>
|
||||
<li>I need to figure out why we have records with language <code>in</code> because that’s not a language!</li>
|
||||
</ul>
|
||||
<h2 id="2017-12-30">2017-12-30</h2>
|
||||
<ul>
|
||||
<li>Linode alerted that CGSpace was using 259% CPU from 4 to 6 AM</li>
|
||||
<li>Uptime Robot noticed that the server went down for 1 minute a few hours later, around 9AM</li>
|
||||
<li>Here's the XMLUI logs:</li>
|
||||
<li>Here’s the XMLUI logs:</li>
|
||||
</ul>
|
||||
<pre><code># cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -E "30/Dec/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
|
||||
637 207.46.13.106
|
||||
@ -740,14 +740,14 @@ DELETE 20
|
||||
1586 66.249.64.78
|
||||
3653 66.249.64.91
|
||||
</code></pre><ul>
|
||||
<li>Looks pretty normal actually, but I don't know who 54.175.208.220 is</li>
|
||||
<li>Looks pretty normal actually, but I don’t know who 54.175.208.220 is</li>
|
||||
<li>They identify as “com.plumanalytics”, which Google says is associated with Elsevier</li>
|
||||
<li>They only seem to have used one Tomcat session so that's good, I guess I don't need to add them to the Tomcat Crawler Session Manager valve:</li>
|
||||
<li>They only seem to have used one Tomcat session so that’s good, I guess I don’t need to add them to the Tomcat Crawler Session Manager valve:</li>
|
||||
</ul>
|
||||
<pre><code>$ grep 54.175.208.220 dspace.log.2017-12-30 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
|
||||
1
|
||||
</code></pre><ul>
|
||||
<li>216.244.66.245 seems to be moz.com's DotBot</li>
|
||||
<li>216.244.66.245 seems to be moz.com’s DotBot</li>
|
||||
</ul>
|
||||
<h2 id="2017-12-31">2017-12-31</h2>
|
||||
<ul>
|
||||
|
||||
Reference in New Issue
Block a user