diff --git a/content/post/2017-12.md b/content/post/2017-12.md index 10eb5d060..56d9ce5d1 100644 --- a/content/post/2017-12.md +++ b/content/post/2017-12.md @@ -417,3 +417,116 @@ $ schedtool -D -e ionice -c2 -n7 nice -n19 dspace index-discovery ``` - I don't have time now to look into this but the Solr sharding has long been an issue! +- Looking into using JDBC / JNDI to provide a database pool to DSpace +- The [DSpace 6.x configuration docs](https://wiki.duraspace.org/display/DSDOC6x/Configuration+Reference) have more notes about setting up the database pool than the 5.x ones (which actually have none!) +- First, I uncomment `db.jndi` in _dspace/config/dspace.cfg_ +- Then I create a global `Resource` in the main Tomcat _server.xml_ (inside `GlobalNamingResources`): + +``` + +``` + +- Most of the parameters are from comments by Mark Wood about his JNDI setup: https://jira.duraspace.org/browse/DS-3564 +- Then I add a `ResourceLink` to each web application context: + +``` + +``` + +- I am not sure why several guides show configuration snippets for _server.xml_ and web application contexts that use a Local and Global jdbc... +- When DSpace can't find the JNDI context (for whatever reason) you will see this in the dspace logs: + +``` +2017-12-19 13:12:08,796 ERROR org.dspace.storage.rdbms.DatabaseManager @ Error retrieving JNDI context: jdbc/dspace +javax.naming.NameNotFoundException: Name [jdbc/dspace] is not bound in this Context. Unable to find [jdbc]. + at org.apache.naming.NamingContext.lookup(NamingContext.java:825) + at org.apache.naming.NamingContext.lookup(NamingContext.java:173) + at org.dspace.storage.rdbms.DatabaseManager.initDataSource(DatabaseManager.java:1414) + at org.dspace.storage.rdbms.DatabaseManager.initialize(DatabaseManager.java:1331) + at org.dspace.storage.rdbms.DatabaseManager.getDataSource(DatabaseManager.java:648) + at org.dspace.storage.rdbms.DatabaseManager.getConnection(DatabaseManager.java:627) + at org.dspace.core.Context.init(Context.java:121) + at org.dspace.core.Context.(Context.java:95) + at org.dspace.app.util.AbstractDSpaceWebapp.register(AbstractDSpaceWebapp.java:79) + at org.dspace.app.util.DSpaceContextListener.contextInitialized(DSpaceContextListener.java:128) + at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5110) + at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5633) + at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) + at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:1015) + at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:991) + at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) + at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:712) + at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:2002) + at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) + at java.util.concurrent.FutureTask.run(FutureTask.java:266) + at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) + at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) + at java.lang.Thread.run(Thread.java:748) +2017-12-19 13:12:08,798 INFO org.dspace.storage.rdbms.DatabaseManager @ Unable to locate JNDI dataSource: jdbc/dspace +2017-12-19 13:12:08,798 INFO org.dspace.storage.rdbms.DatabaseManager @ Falling back to creating own Database pool +``` + +- And indeed the Catalina logs show that it failed to set up the JDBC driver: + +``` +org.apache.tomcat.dbcp.dbcp.SQLNestedException: Cannot load JDBC driver class 'org.postgresql.Driver' +``` + +- There are several copies of the PostgreSQL driver installed by DSpace: + +``` +$ find ~/dspace/ -iname "postgresql*jdbc*.jar" +/Users/aorth/dspace/webapps/jspui/WEB-INF/lib/postgresql-9.1-901-1.jdbc4.jar +/Users/aorth/dspace/webapps/oai/WEB-INF/lib/postgresql-9.1-901-1.jdbc4.jar +/Users/aorth/dspace/webapps/xmlui/WEB-INF/lib/postgresql-9.1-901-1.jdbc4.jar +/Users/aorth/dspace/webapps/rest/WEB-INF/lib/postgresql-9.1-901-1.jdbc4.jar +/Users/aorth/dspace/lib/postgresql-9.1-901-1.jdbc4.jar +``` + +- These apparently come from the main DSpace `pom.xml`: + +``` + + postgresql + postgresql + 9.1-901-1.jdbc4 + +``` + +- So WTF? Let's try copying one to Tomcat's lib folder and restarting Tomcat: + +``` +$ cp ~/dspace/lib/postgresql-9.1-901-1.jdbc4.jar /usr/local/opt/tomcat@7/libexec/lib +``` + +- Oh that's fantastic, now at least Tomcat doesn't print an error during startup so I guess it succeeds to create the JNDI pool +- DSpace starts up but I have no idea if it's using the JNDI configuration because I see this in the logs: + +``` +2017-12-19 13:26:54,271 INFO org.dspace.storage.rdbms.DatabaseManager @ DBMS is '{}'PostgreSQL +2017-12-19 13:26:54,277 INFO org.dspace.storage.rdbms.DatabaseManager @ DBMS driver version is '{}'9.5.10 +2017-12-19 13:26:54,293 INFO org.dspace.storage.rdbms.DatabaseUtils @ Loading Flyway DB migrations from: filesystem:/Users/aorth/dspace/etc/postgres, classpath:org.dspace.storage.rdbms.sqlmigration.postgres, classpath:org.dspace.storage.rdbms.migration +2017-12-19 13:26:54,306 INFO org.flywaydb.core.internal.dbsupport.DbSupportFactory @ Database: jdbc:postgresql://localhost:5432/dspacetest (PostgreSQL 9.5) +``` + +- Let's try again, but this time explicitly blank the PostgreSQL connection parameters in dspace.cfg and see if DSpace starts... +- Wow, ok, that works, but having to copy the PostgreSQL JDBC JAR to Tomcat's lib folder totally blows +- Also, it's likely this is only a problem on my local macOS + Tomcat test environment +- Ubuntu's Tomcat distribution will probably handle this differently +- So for reference I have: + - a `` defined globally in server.xml + - a `` defined in each web application's context XML + - unset the `db.url`, `db.username`, and `db.password` parameters in dspace.cfg + - set the `db.jndi` in dspace.cfg to the name specified in the web application context + diff --git a/public/2017-12/index.html b/public/2017-12/index.html index b319daa4d..aac70bf7e 100644 --- a/public/2017-12/index.html +++ b/public/2017-12/index.html @@ -23,7 +23,7 @@ The list of connections to XMLUI and REST API for today: - + @@ -56,9 +56,9 @@ The list of connections to XMLUI and REST API for today: "@type": "BlogPosting", "headline": "December, 2017", "url": "https://alanorth.github.io/cgspace-notes/2017-12/", - "wordCount": "2421", + "wordCount": "2915", "datePublished": "2017-12-01T13:53:54+03:00", - "dateModified": "2017-12-19T10:14:51+02:00", + "dateModified": "2017-12-19T10:38:57+02:00", "author": { "@type": "Person", "name": "Alan Orth" @@ -585,6 +585,129 @@ $ schedtool -D -e ionice -c2 -n7 nice -n19 dspace index-discovery
  • I don’t have time now to look into this but the Solr sharding has long been an issue!
  • +
  • Looking into using JDBC / JNDI to provide a database pool to DSpace
  • +
  • The DSpace 6.x configuration docs have more notes about setting up the database pool than the 5.x ones (which actually have none!)
  • +
  • First, I uncomment db.jndi in dspace/config/dspace.cfg
  • +
  • Then I create a global Resource in the main Tomcat server.xml (inside GlobalNamingResources):
  • +
+ +
	<Resource name="jdbc/dspace" auth="Container" type="javax.sql.DataSource"
+		  driverClassName="org.postgresql.Driver"
+		  url="jdbc:postgresql://localhost:5432/dspace"
+		  username="dspace"
+		  password="dspace"
+          initialSize='5'
+          maxActive='50'
+          maxIdle='15'
+          minIdle='5'
+          maxWait='5000'
+          validationQuery='SELECT 1'
+          testOnBorrow='true' />
+
+ + + +
<ResourceLink global="jdbc/dspace" name="jdbc/dspace" type="javax.sql.DataSource"/>
+
+ +
    +
  • I am not sure why several guides show configuration snippets for server.xml and web application contexts that use a Local and Global jdbc…
  • +
  • When DSpace can’t find the JNDI context (for whatever reason) you will see this in the dspace logs:
  • +
+ +
2017-12-19 13:12:08,796 ERROR org.dspace.storage.rdbms.DatabaseManager @ Error retrieving JNDI context: jdbc/dspace
+javax.naming.NameNotFoundException: Name [jdbc/dspace] is not bound in this Context. Unable to find [jdbc].
+        at org.apache.naming.NamingContext.lookup(NamingContext.java:825)
+        at org.apache.naming.NamingContext.lookup(NamingContext.java:173)
+        at org.dspace.storage.rdbms.DatabaseManager.initDataSource(DatabaseManager.java:1414)
+        at org.dspace.storage.rdbms.DatabaseManager.initialize(DatabaseManager.java:1331)
+        at org.dspace.storage.rdbms.DatabaseManager.getDataSource(DatabaseManager.java:648)
+        at org.dspace.storage.rdbms.DatabaseManager.getConnection(DatabaseManager.java:627)
+        at org.dspace.core.Context.init(Context.java:121)
+        at org.dspace.core.Context.<init>(Context.java:95)
+        at org.dspace.app.util.AbstractDSpaceWebapp.register(AbstractDSpaceWebapp.java:79)
+        at org.dspace.app.util.DSpaceContextListener.contextInitialized(DSpaceContextListener.java:128)
+        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5110)
+        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5633)
+        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
+        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:1015)
+        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:991)
+        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
+        at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:712)
+        at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:2002)
+        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
+        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
+        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
+        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
+        at java.lang.Thread.run(Thread.java:748)
+2017-12-19 13:12:08,798 INFO  org.dspace.storage.rdbms.DatabaseManager @ Unable to locate JNDI dataSource: jdbc/dspace
+2017-12-19 13:12:08,798 INFO  org.dspace.storage.rdbms.DatabaseManager @ Falling back to creating own Database pool
+
+ +
    +
  • And indeed the Catalina logs show that it failed to set up the JDBC driver:
  • +
+ +
org.apache.tomcat.dbcp.dbcp.SQLNestedException: Cannot load JDBC driver class 'org.postgresql.Driver'
+
+ +
    +
  • There are several copies of the PostgreSQL driver installed by DSpace:
  • +
+ +
$ find ~/dspace/ -iname "postgresql*jdbc*.jar"
+/Users/aorth/dspace/webapps/jspui/WEB-INF/lib/postgresql-9.1-901-1.jdbc4.jar
+/Users/aorth/dspace/webapps/oai/WEB-INF/lib/postgresql-9.1-901-1.jdbc4.jar
+/Users/aorth/dspace/webapps/xmlui/WEB-INF/lib/postgresql-9.1-901-1.jdbc4.jar
+/Users/aorth/dspace/webapps/rest/WEB-INF/lib/postgresql-9.1-901-1.jdbc4.jar
+/Users/aorth/dspace/lib/postgresql-9.1-901-1.jdbc4.jar
+
+ +
    +
  • These apparently come from the main DSpace pom.xml:
  • +
+ +
<dependency>
+   <groupId>postgresql</groupId>
+   <artifactId>postgresql</artifactId>
+   <version>9.1-901-1.jdbc4</version>
+</dependency>
+
+ +
    +
  • So WTF? Let’s try copying one to Tomcat’s lib folder and restarting Tomcat:
  • +
+ +
$ cp ~/dspace/lib/postgresql-9.1-901-1.jdbc4.jar /usr/local/opt/tomcat@7/libexec/lib
+
+ +
    +
  • Oh that’s fantastic, now at least Tomcat doesn’t print an error during startup so I guess it succeeds to create the JNDI pool
  • +
  • DSpace starts up but I have no idea if it’s using the JNDI configuration because I see this in the logs:
  • +
+ +
2017-12-19 13:26:54,271 INFO  org.dspace.storage.rdbms.DatabaseManager @ DBMS is '{}'PostgreSQL
+2017-12-19 13:26:54,277 INFO  org.dspace.storage.rdbms.DatabaseManager @ DBMS driver version is '{}'9.5.10
+2017-12-19 13:26:54,293 INFO  org.dspace.storage.rdbms.DatabaseUtils @ Loading Flyway DB migrations from: filesystem:/Users/aorth/dspace/etc/postgres, classpath:org.dspace.storage.rdbms.sqlmigration.postgres, classpath:org.dspace.storage.rdbms.migration
+2017-12-19 13:26:54,306 INFO  org.flywaydb.core.internal.dbsupport.DbSupportFactory @ Database: jdbc:postgresql://localhost:5432/dspacetest (PostgreSQL 9.5)
+
+ +
    +
  • Let’s try again, but this time explicitly blank the PostgreSQL connection parameters in dspace.cfg and see if DSpace starts…
  • +
  • Wow, ok, that works, but having to copy the PostgreSQL JDBC JAR to Tomcat’s lib folder totally blows
  • +
  • Also, it’s likely this is only a problem on my local macOS + Tomcat test environment
  • +
  • Ubuntu’s Tomcat distribution will probably handle this differently
  • +
  • So for reference I have: + +
      +
    • a <Resource> defined globally in server.xml
    • +
    • a <ResourceLink> defined in each web application’s context XML
    • +
    • unset the db.url, db.username, and db.password parameters in dspace.cfg
    • +
    • set the db.jndi in dspace.cfg to the name specified in the web application context
    • +
diff --git a/public/2017/12/postgres-connections-month-cgspace-2.png b/public/2017/12/postgres-connections-month-cgspace-2.png new file mode 100644 index 000000000..11731d504 Binary files /dev/null and b/public/2017/12/postgres-connections-month-cgspace-2.png differ diff --git a/public/robots.txt b/public/robots.txt index d2ae9994d..7c86be3fe 100644 --- a/public/robots.txt +++ b/public/robots.txt @@ -30,7 +30,7 @@ Disallow: /cgspace-notes/2015-12/ Disallow: /cgspace-notes/2015-11/ Disallow: /cgspace-notes/ Disallow: /cgspace-notes/categories/ -Disallow: /cgspace-notes/tags/notes/ Disallow: /cgspace-notes/categories/notes/ +Disallow: /cgspace-notes/tags/notes/ Disallow: /cgspace-notes/post/ Disallow: /cgspace-notes/tags/ diff --git a/public/sitemap.xml b/public/sitemap.xml index 9500729d1..163c01278 100644 --- a/public/sitemap.xml +++ b/public/sitemap.xml @@ -4,7 +4,7 @@ https://alanorth.github.io/cgspace-notes/2017-12/ - 2017-12-19T10:14:51+02:00 + 2017-12-19T10:38:57+02:00 @@ -139,7 +139,7 @@ https://alanorth.github.io/cgspace-notes/ - 2017-12-19T10:14:51+02:00 + 2017-12-19T10:38:57+02:00 0 @@ -148,27 +148,27 @@ 0 - - https://alanorth.github.io/cgspace-notes/tags/notes/ - 2017-12-19T10:14:51+02:00 - 0 - - https://alanorth.github.io/cgspace-notes/categories/notes/ 2017-09-28T12:00:49+03:00 0 + + https://alanorth.github.io/cgspace-notes/tags/notes/ + 2017-12-19T10:38:57+02:00 + 0 + + https://alanorth.github.io/cgspace-notes/post/ - 2017-12-19T10:14:51+02:00 + 2017-12-19T10:38:57+02:00 0 https://alanorth.github.io/cgspace-notes/tags/ - 2017-12-19T10:14:51+02:00 + 2017-12-19T10:38:57+02:00 0 diff --git a/static/2017/12/postgres-connections-month-cgspace-2.png b/static/2017/12/postgres-connections-month-cgspace-2.png new file mode 100644 index 000000000..11731d504 Binary files /dev/null and b/static/2017/12/postgres-connections-month-cgspace-2.png differ