cgspace-notes/public/2016-08/index.html

436 lines
18 KiB
HTML
Raw Normal View History

<!DOCTYPE html>
<html lang="en-us">
<head prefix="og: http://ogp.me/ns#">
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1" />
<meta property="og:title" content=" August, 2016 &middot; CGSpace Notes" />
<meta property="og:site_name" content="CGSpace Notes" />
2016-08-03 09:09:36 +02:00
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2016-08/" />
<meta property="og:type" content="article" />
<meta property="og:article:published_time" content="2016-08-01T15:53:00&#43;03:00" />
<meta property="og:article:tag" content="notes" />
<title>
August, 2016 &middot; CGSpace Notes
</title>
2016-08-03 09:09:36 +02:00
<link rel="stylesheet" href="https://alanorth.github.io/cgspace-notes/css/bootstrap.min.css" />
<link rel="stylesheet" href="https://alanorth.github.io/cgspace-notes/css/main.css" />
<link rel="stylesheet" href="https://alanorth.github.io/cgspace-notes/css/font-awesome.min.css" />
<link rel="stylesheet" href="https://alanorth.github.io/cgspace-notes/css/github.css" />
<link rel="stylesheet" href="//fonts.googleapis.com/css?family=Source+Sans+Pro:200,300,400" type="text/css">
2016-08-03 09:09:36 +02:00
<link rel="shortcut icon" href="https://alanorth.github.io/cgspace-notes/images/favicon.ico" />
<link rel="apple-touch-icon" href="https://alanorth.github.io/cgspace-notes/images/apple-touch-icon.png" />
</head>
<body>
<header class="global-header" style="background-image:url(../images/bg.jpg )">
<section class="header-text">
2016-08-03 09:09:36 +02:00
<h1><a href="https://alanorth.github.io/cgspace-notes/">CGSpace Notes</a></h1>
<div class="sns-links hidden-print">
</div>
2016-08-03 09:09:36 +02:00
<a href="https://alanorth.github.io/cgspace-notes/" class="btn-header btn-back hidden-xs">
<i class="fa fa-angle-left" aria-hidden="true"></i>
&nbsp;Home
</a>
</section>
</header>
<main class="container">
<article>
<header>
<h1 class="text-primary">August, 2016</h1>
<div class="post-meta clearfix">
<div class="post-date pull-left">
Posted on
<time datetime="2016-08-01T15:53:00&#43;03:00">
Aug 1, 2016
</time>
</div>
<div class="pull-right">
2016-08-03 09:09:36 +02:00
<span class="post-tag small"><a href="https://alanorth.github.io/cgspace-notes//tags/notes">#notes</a></span>
</div>
</div>
</header>
<section>
2016-08-03 09:09:36 +02:00
<h2 id="2016-08-01">2016-08-01</h2>
<ul>
<li>Add updated distribution license from Sisay (<a href="https://github.com/ilri/DSpace/issues/259">#259</a>)</li>
2016-08-02 00:06:47 +02:00
<li>Play with upgrading Mirage 2 dependencies in <code>bower.json</code> because most are several versions of out date</li>
<li>Bootstrap is at 3.3.0 but upstream is at 3.3.7, and upgrading to anything beyond 3.3.1 breaks glyphicons and probably more</li>
<li>bower stuff is a dead end, waste of time, too many issues</li>
<li>Anything after Bootstrap 3.3.1 makes glyphicons disappear (HTTP 404 trying to access from incorrect path of <code>fonts</code>)</li>
<li>Start working on DSpace 5.15.5 port:</li>
</ul>
<pre><code>$ git checkout -b 55new 5_x-prod
$ git reset --hard ilri/5_x-prod
$ git rebase -i dspace-5.5
</code></pre>
<ul>
<li>Lots of conflicts that don&rsquo;t make sense (ie, shouldn&rsquo;t conflict!)</li>
<li>This file in particular conflicts almost 10 times: <code>dspace/modules/xmlui-mirage2/src/main/webapp/themes/CGIAR/styles/_style.scss</code></li>
<li>Checking out a clean branch at 5.5 and cherry-picking our commits works where that file would normally have a conflict</li>
<li>Seems to be related to merge commits</li>
<li><code>git rebase --preserve-merges</code> doesn&rsquo;t seem to help</li>
<li>Eventually I just turned on git rerere and solved the conflicts and completed the 403 commit rebase</li>
<li>The 5.5 code now builds but doesn&rsquo;t run (white page in Tomcat)</li>
2016-08-02 14:03:28 +02:00
</ul>
2016-08-03 09:09:36 +02:00
<h2 id="2016-08-02">2016-08-02</h2>
2016-08-02 14:03:28 +02:00
<ul>
<li>Ask Atmire for help with DSpace 5.5 issue</li>
<li>Vanilla DSpace 5.5 deploys and runs fine</li>
<li>Playing with DSpace in Ubuntu 16.04 and Tomcat 7</li>
<li>Everything is still fucked up, even vanilla DSpace 5.5</li>
2016-08-05 13:05:32 +02:00
</ul>
<h2 id="2016-08-04">2016-08-04</h2>
<ul>
<li>Ask on DSpace mailing list about duplicate authors, Discovery and author text values</li>
<li>Atmire responded with some new DSpace 5.5 ready versions to try for their modules</li>
</ul>
2016-08-05 15:38:43 +02:00
<h2 id="2016-08-05">2016-08-05</h2>
<ul>
<li>Fix item display incorrectly displaying Species when Breeds were present (<a href="https://github.com/ilri/DSpace/pull/260">#260</a>)</li>
<li>Experiment with fixing more authors, like Delia Grace:</li>
</ul>
<pre><code>dspacetest=# update metadatavalue set authority='0b4fcbc1-d930-4319-9b4d-ea1553cca70b', confidence=600 where metadata_field_id=3 and text_value='Grace, D.';
</code></pre>
2016-08-06 15:55:17 +02:00
<h2 id="2016-08-06">2016-08-06</h2>
<ul>
<li>Finally figured out how to remove &ldquo;View/Open&rdquo; and &ldquo;Bitstreams&rdquo; from the item view</li>
</ul>
2016-08-08 22:22:50 +02:00
<h2 id="2016-08-07">2016-08-07</h2>
<ul>
<li>Start working on Ubuntu 16.04 Ansible playbook for Tomcat 8, PostgreSQL 9.5, Oracle 8, etc</li>
</ul>
<h2 id="2016-08-08">2016-08-08</h2>
<ul>
<li>Still troubleshooting Atmire modules on DSpace 5.5</li>
<li>Vanilla DSpace 5.5 works on Tomcat 7&hellip;</li>
<li>Ooh, and vanilla DSpace 5.5 works on Tomcat 8 with Java 8!</li>
<li>Some notes about setting up Tomcat 8, since it&rsquo;s new on this machine&hellip;</li>
<li>Install latest Oracle Java 8 JDK</li>
<li>Create <code>setenv.sh</code> in Tomcat 8 <code>libexec/bin</code> directory:
<br /></li>
</ul>
<pre><code>CATALINA_OPTS=&quot;-Djava.awt.headless=true -Xms3072m -Xmx3072m -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -Dfile.encoding=UTF-8&quot;
CATALINA_OPTS=&quot;$CATALINA_OPTS -Djava.library.path=/opt/brew/Cellar/tomcat-native/1.2.8/lib&quot;
JRE_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_102.jdk/Contents/Home
</code></pre>
<ul>
<li>Edit Tomcat 8 <code>server.xml</code> to add regular HTTP listener for solr</li>
<li>Symlink webapps:</li>
</ul>
<pre><code>$ rm -rf /opt/brew/Cellar/tomcat/8.5.4/libexec/webapps/ROOT
$ ln -sv ~/dspace/webapps/xmlui /opt/brew/Cellar/tomcat/8.5.4/libexec/webapps/ROOT
$ ln -sv ~/dspace/webapps/oai /opt/brew/Cellar/tomcat/8.5.4/libexec/webapps/oai
$ ln -sv ~/dspace/webapps/jspui /opt/brew/Cellar/tomcat/8.5.4/libexec/webapps/jspui
$ ln -sv ~/dspace/webapps/rest /opt/brew/Cellar/tomcat/8.5.4/libexec/webapps/rest
$ ln -sv ~/dspace/webapps/solr /opt/brew/Cellar/tomcat/8.5.4/libexec/webapps/solr
</code></pre>
2016-08-09 11:31:06 +02:00
<h2 id="2016-08-09">2016-08-09</h2>
<ul>
<li>More tests of Atmire&rsquo;s 5.5 modules on a clean, working instance of <code>5_x-prod</code></li>
<li>Still fails, though perhaps differently than before (Flyway): <a href="https://gist.github.com/alanorth/5d49c45a16efd7c6bc1e6642e66118b2">https://gist.github.com/alanorth/5d49c45a16efd7c6bc1e6642e66118b2</a></li>
<li>More work on Tomcat 8 and Java 8 stuff for Ansible playbooks</li>
2016-08-10 12:19:46 +02:00
</ul>
<h2 id="2016-08-10">2016-08-10</h2>
<ul>
<li>Turns out DSpace 5.x isn&rsquo;t ready for Tomcat 8: <a href="https://jira.duraspace.org/browse/DS-3092">https://jira.duraspace.org/browse/DS-3092</a></li>
<li>So we&rsquo;ll need to use Tomcat 7 + Java 8 on Ubuntu 16.04</li>
<li>More work on the Ansible stuff for this, allowing Tomcat 7 to use Java 8</li>
<li>Merge pull request for fixing the type Discovery index to use <code>dc.type</code> (<a href="https://github.com/ilri/DSpace/pull/262">#262</a>)</li>
<li>Merge pull request for removing &ldquo;Bitstream&rdquo; text from item display, as it confuses users and isn&rsquo;t necessary (<a href="https://github.com/ilri/DSpace/pull/263">#263</a>)</li>
2016-08-09 11:31:06 +02:00
</ul>
2016-08-11 07:50:33 +02:00
<h2 id="2016-08-11">2016-08-11</h2>
<ul>
<li>Finally got DSpace (5.5) running on Ubuntu 16.04, Tomcat 7, Java 8, PostgreSQL 9.5 via the updated Ansible stuff</li>
</ul>
<p><img src="../images/2016/08/dspace55-ubuntu16.04.png" alt="DSpace 5.5 on Ubuntu 16.04, Tomcat 7, Java 8, PostgreSQL 9.5" /></p>
2016-08-15 10:34:55 +02:00
<h2 id="2016-08-14">2016-08-14</h2>
<ul>
<li>Update Mirage 2 build notes for Ubuntu 16.04: <a href="https://gist.github.com/alanorth/2cf9c15834dc68a514262fcb04004cb0">https://gist.github.com/alanorth/2cf9c15834dc68a514262fcb04004cb0</a></li>
</ul>
<h2 id="2016-08-15">2016-08-15</h2>
<ul>
<li>Notes on NodeJS + nginx + systemd: <a href="https://gist.github.com/alanorth/51acd476891c67dfe27725848cf5ace1">https://gist.github.com/alanorth/51acd476891c67dfe27725848cf5ace1</a></li>
</ul>
<p><img src="../images/2016/08/nodejs-nginx.png" alt="ExpressJS running behind nginx" /></p>
2016-08-17 06:19:11 +02:00
<h2 id="2016-08-16">2016-08-16</h2>
<ul>
<li>Troubleshoot Paramiko connection issues with Ansible on ILRI servers: <a href="https://github.com/ilri/rmg-ansible-public/issues/37">#37</a></li>
<li>Turns out we need to add some MACs to our <code>sshd_config</code>: hmac-sha2-512,hmac-sha2-256</li>
<li>Update DSpace Test&rsquo;s Java to version 8 to start testing this configuration (<a href="https://wiki.apache.org/solr/ShawnHeisey">seeing as Solr recommends it</a>)</li>
2016-08-17 13:40:16 +02:00
</ul>
<h2 id="2016-08-17">2016-08-17</h2>
<ul>
<li>More work on Let&rsquo;s Encrypt stuff for Ansible roles</li>
<li>Yesterday Atmire responded about DSpace 5.5 issues and asked me to try the <code>dspace database repair</code> command to fix Flyway issues</li>
<li>The <code>dspace database</code> command doesn&rsquo;t even run: <a href="https://gist.github.com/alanorth/c43c8d89e8df346d32c0ee938be90cd5">https://gist.github.com/alanorth/c43c8d89e8df346d32c0ee938be90cd5</a></li>
<li>Oops, it looks like the missing classes causing <code>dspace database</code> to fail were coming from the old <code>~/dspace/config/spring</code> folder</li>
<li>After removing the spring folder and running ant install again, <code>dspace database</code> works</li>
<li>I see there are missing and pending Flyway migrations, but running <code>dspace database repair</code> and <code>dspace database migrate</code> does nothing: <a href="https://gist.github.com/alanorth/41ed5abf2ff32d8ac9eedd1c3d015d70">https://gist.github.com/alanorth/41ed5abf2ff32d8ac9eedd1c3d015d70</a></li>
2016-08-17 06:19:11 +02:00
</ul>
2016-08-18 14:48:32 +02:00
<h2 id="2016-08-18">2016-08-18</h2>
<ul>
<li>Fix &ldquo;CONGO,DR&rdquo; country name in <code>input-forms.xml</code> (<a href="https://github.com/ilri/DSpace/pull/264">#264</a>)</li>
<li>Also need to fix existing records using the incorrect form in the database:</li>
</ul>
<pre><code>dspace=# update metadatavalue set text_value='CONGO, DR' where resource_type_id=2 and metadata_field_id=228 and text_value='CONGO,DR';
</code></pre>
2016-08-18 15:35:08 +02:00
<ul>
<li>I asked a question on the DSpace mailing list about updating &ldquo;preferred&rdquo; forms of author names from ORCID</li>
2016-08-21 16:10:08 +02:00
</ul>
<h2 id="2016-08-21">2016-08-21</h2>
<ul>
<li>A few days ago someone on the DSpace mailing list suggested I try <code>dspace dsrun org.dspace.authority.UpdateAuthorities</code> to update preferred author names from ORCID</li>
<li>If you set <code>auto-update-items=true</code> in <code>dspace/config/modules/solrauthority.cfg</code> it is supposed to update records it finds automatically</li>
<li>I updated my name format on ORCID and I&rsquo;ve been running that script a few times per day since then but nothing has changed</li>
2016-08-21 17:10:52 +02:00
<li>Still troubleshooting Atmire modules on DSpace 5.5</li>
<li>I sent them some new verbose logs: <a href="https://gist.github.com/alanorth/700748995649688148ceba89d760253e">https://gist.github.com/alanorth/700748995649688148ceba89d760253e</a></li>
2016-08-22 18:29:35 +02:00
</ul>
<h2 id="2016-08-22">2016-08-22</h2>
<ul>
<li>Database migrations are fine on DSpace 5.1:</li>
</ul>
<pre><code>$ ~/dspace/bin/dspace database info
Database URL: jdbc:postgresql://localhost:5432/dspacetest
Database Schema: public
Database Software: PostgreSQL version 9.3.14
Database Driver: PostgreSQL Native Driver version PostgreSQL 9.1 JDBC4 (build 901)
+----------------+----------------------------+---------------------+---------+
| Version | Description | Installed on | State |
+----------------+----------------------------+---------------------+---------+
| 1.1 | Initial DSpace 1.1 databas | | PreInit |
| 1.2 | Upgrade to DSpace 1.2 sche | | PreInit |
| 1.3 | Upgrade to DSpace 1.3 sche | | PreInit |
| 1.3.9 | Drop constraint for DSpace | | PreInit |
| 1.4 | Upgrade to DSpace 1.4 sche | | PreInit |
| 1.5 | Upgrade to DSpace 1.5 sche | | PreInit |
| 1.5.9 | Drop constraint for DSpace | | PreInit |
| 1.6 | Upgrade to DSpace 1.6 sche | | PreInit |
| 1.7 | Upgrade to DSpace 1.7 sche | | PreInit |
| 1.8 | Upgrade to DSpace 1.8 sche | | PreInit |
| 3.0 | Upgrade to DSpace 3.x sche | | PreInit |
| 4.0 | Initializing from DSpace 4 | 2015-11-20 12:42:52 | Success |
| 5.0.2014.08.08 | DS-1945 Helpdesk Request a | 2015-11-20 12:42:53 | Success |
| 5.0.2014.09.25 | DS 1582 Metadata For All O | 2015-11-20 12:42:55 | Success |
| 5.0.2014.09.26 | DS-1582 Metadata For All O | 2015-11-20 12:42:55 | Success |
| 5.0.2015.01.27 | MigrateAtmireExtraMetadata | 2015-11-20 12:43:29 | Success |
| 5.1.2015.12.03 | Atmire CUA 4 migration | 2016-03-21 17:10:41 | Success |
| 5.1.2015.12.03 | Atmire MQM migration | 2016-03-21 17:10:42 | Success |
+----------------+----------------------------+---------------------+---------+
</code></pre>
<ul>
<li>So I&rsquo;m not sure why they have problems when we move to DSpace 5.5 (even the 5.1 migrations themselves show as &ldquo;Missing&rdquo;)</li>
2016-08-23 15:16:10 +02:00
</ul>
<h2 id="2016-08-23">2016-08-23</h2>
<ul>
<li>Help Paola from CCAFS with her thumbnails again</li>
<li>Talk to Atmire about the DSpace 5.5 issue, and it seems to be caused by a bug in FlywayDB</li>
<li>They said I should delete the Atmire migrations
<br /></li>
</ul>
<pre><code>dspacetest=# delete from schema_version where description = 'Atmire CUA 4 migration' and version='5.1.2015.12.03.2';
dspacetest=# delete from schema_version where description = 'Atmire MQM migration' and version='5.1.2015.12.03.3';
</code></pre>
<ul>
<li>After that DSpace starts up by XMLUI now has unrelated issues that I need to solve!</li>
</ul>
<pre><code>org.apache.avalon.framework.configuration.ConfigurationException: Type 'ThemeResourceReader' does not exist for 'map:read' at jndi:/localhost/themes/0_CGIAR/sitemap.xmap:136:77
context:/jndi:/localhost/themes/0_CGIAR/sitemap.xmap - 136:77
</code></pre>
<ul>
<li>Looks like we&rsquo;re missing some stuff in the XMLUI module&rsquo;s <code>sitemap.xmap</code>, as well as in each of our XMLUI themes</li>
<li>Diff them with these to get the <code>ThemeResourceReader</code> changes:
<ul>
<li><code>dspace-xmlui/src/main/webapp/sitemap.xmap</code></li>
<li><code>dspace-xmlui-mirage2/src/main/webapp/sitemap.xmap</code></li>
</ul></li>
<li>Then we had some NullPointerException from the SolrLogger class, which is apparently part of Atmire&rsquo;s CUA module</li>
<li>I tried with a small version bump to CUA but it didn&rsquo;t work (version <code>5.5-4.1.1-0</code>)</li>
<li>Also, I started looking into huge pages to prepare for PostgreSQL 9.5, but it seems Linode&rsquo;s kernels don&rsquo;t enable them</li>
2016-08-18 15:35:08 +02:00
</ul>
2016-08-25 10:09:07 +02:00
<h2 id="2016-08-24">2016-08-24</h2>
<ul>
<li>Clean up and import 48 CCAFS records into DSpace Test</li>
<li>SQL to get all journal titles from dc.source (55), since it&rsquo;s apparently used for internal DSpace filename shit, but we moved all our journal titles there a few months ago:</li>
</ul>
<pre><code>dspacetest=# select distinct text_value from metadatavalue where metadata_field_id=55 and text_value !~ '.*(\.pdf|\.png|\.PDF|\.Pdf|\.JPEG|\.jpg|\.JPG|\.jpeg|\.xls|\.rtf|\.docx?|\.potx|\.dotx|\.eqa|\.tiff|\.mp4|\.mp3|\.gif|\.zip|\.txt|\.pptx|\.indd|\.PNG|\.bmp|\.exe|org\.dspace\.app\.mediafilter).*';
</code></pre>
<h2 id="2016-08-25">2016-08-25</h2>
<ul>
<li>Atmire suggested adding a missing bean to <code>dspace/config/spring/api/atmire-cua.xml</code> but it doesn&rsquo;t help:</li>
</ul>
<pre><code>...
Error creating bean with name 'MetadataStorageInfoService'
...
2016-08-25 13:59:59 +02:00
</code></pre>
<ul>
<li>Atmire sent an updated version of <code>dspace/config/spring/api/atmire-cua.xml</code> and now XMLUI starts but gives a null pointer exception:</li>
</ul>
<pre><code>Java stacktrace: java.lang.NullPointerException
at org.dspace.app.xmlui.aspect.statistics.Navigation.addOptions(Navigation.java:129)
at org.dspace.app.xmlui.wing.AbstractWingTransformer.startElement(AbstractWingTransformer.java:228)
at sun.reflect.GeneratedMethodAccessor126.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
at com.sun.proxy.$Proxy103.startElement(Unknown Source)
at org.apache.cocoon.environment.internal.EnvironmentChanger.startElement(EnvironmentStack.java:140)
at org.apache.cocoon.environment.internal.EnvironmentChanger.startElement(EnvironmentStack.java:140)
at org.apache.cocoon.xml.AbstractXMLPipe.startElement(AbstractXMLPipe.java:94)
...
</code></pre>
<ul>
<li>Import the 47 CCAFS records to CGSpace, creating the SimpleArchiveFormat bundles and importing like:</li>
</ul>
<pre><code>$ ./safbuilder.sh -c /tmp/Thumbnails\ to\ Upload\ to\ CGSpace/3546.csv
$ JAVA_OPTS=&quot;-Dfile.encoding=UTF-8 -Xmx512m&quot; /home/cgspace.cgiar.org/bin/dspace import -a -e aorth@mjanja.ch -c 10568/3546 -s /tmp/Thumbnails\ to\ Upload\ to\ CGSpace/SimpleArchiveFormat -m 3546.map
2016-08-25 10:09:07 +02:00
</code></pre>
</section>
<footer>
<section class="author-info row">
<div class="author-avatar col-md-2">
</div>
<div class="author-meta col-md-6">
<h1 class="author-name text-primary">Alan Orth</h1>
</div>
</section>
<ul class="pager">
2016-08-03 09:09:36 +02:00
<li class="previous"><a href="https://alanorth.github.io/cgspace-notes/2016-07/"><span aria-hidden="true">&larr;</span> Older</a></li>
<li class="next disabled"><a href="#">Newer <span aria-hidden="true">&rarr;</span></a></li>
</ul>
</footer>
</article>
</main>
<footer class="container global-footer">
<div class="copyright-note pull-left">
</div>
<div class="sns-links hidden-print">
</div>
</footer>
2016-08-03 09:09:36 +02:00
<script src="https://alanorth.github.io/cgspace-notes/js/highlight.pack.js"></script>
<script>
hljs.initHighlightingOnLoad();
</script>
</body>
</html>