alanorth/cgspace-notes
1
0
Fork 0
mirror of https://github.com/alanorth/cgspace-notes.git synced 2024-11-22 22:55:04 +01:00
Code Issues Projects Releases Wiki Activity
202d3fb88f
cgspace-notes/content/posts/2016-08.md

277 lines
13 KiB
Markdown
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

+++
date = "2016-08-01T15:53:00+03:00"
author = "Alan Orth"
title = "August, 2016"
tags = ["Notes"]
+++
## 2016-08-01
- Add updated distribution license from Sisay ([#259](https://github.com/ilri/DSpace/issues/259))
- Play with upgrading Mirage 2 dependencies in `bower.json` because most are several versions of out date
- Bootstrap is at 3.3.0 but upstream is at 3.3.7, and upgrading to anything beyond 3.3.1 breaks glyphicons and probably more
- bower stuff is a dead end, waste of time, too many issues
- Anything after Bootstrap 3.3.1 makes glyphicons disappear (HTTP 404 trying to access from incorrect path of `fonts`)
- Start working on DSpace 5.1 → 5.5 port:
```
$ git checkout -b 55new 5_x-prod
$ git reset --hard ilri/5_x-prod
$ git rebase -i dspace-5.5
```
<!--more-->
- Lots of conflicts that don't make sense (ie, shouldn't conflict!)
- This file in particular conflicts almost 10 times: `dspace/modules/xmlui-mirage2/src/main/webapp/themes/CGIAR/styles/_style.scss`
- Checking out a clean branch at 5.5 and cherry-picking our commits works where that file would normally have a conflict
- Seems to be related to merge commits
- `git rebase --preserve-merges` doesn't seem to help
- Eventually I just turned on git rerere and solved the conflicts and completed the 403 commit rebase
- The 5.5 code now builds but doesn't run (white page in Tomcat)
## 2016-08-02
- Ask Atmire for help with DSpace 5.5 issue
- Vanilla DSpace 5.5 deploys and runs fine
- Playing with DSpace in Ubuntu 16.04 and Tomcat 7
- Everything is still fucked up, even vanilla DSpace 5.5
## 2016-08-04
- Ask on DSpace mailing list about duplicate authors, Discovery and author text values
- Atmire responded with some new DSpace 5.5 ready versions to try for their modules
## 2016-08-05
- Fix item display incorrectly displaying Species when Breeds were present ([#260](https://github.com/ilri/DSpace/pull/260))
- Experiment with fixing more authors, like Delia Grace:
```
dspacetest=# update metadatavalue set authority='0b4fcbc1-d930-4319-9b4d-ea1553cca70b', confidence=600 where metadata_field_id=3 and text_value='Grace, D.';
```
## 2016-08-06
- Finally figured out how to remove "View/Open" and "Bitstreams" from the item view
## 2016-08-07
- Start working on Ubuntu 16.04 Ansible playbook for Tomcat 8, PostgreSQL 9.5, Oracle 8, etc
## 2016-08-08
- Still troubleshooting Atmire modules on DSpace 5.5
- Vanilla DSpace 5.5 works on Tomcat 7...
- Ooh, and vanilla DSpace 5.5 works on Tomcat 8 with Java 8!
- Some notes about setting up Tomcat 8, since it's new on this machine...
- Install latest Oracle Java 8 JDK
- Create `setenv.sh` in Tomcat 8 `libexec/bin` directory:
```
CATALINA_OPTS="-Djava.awt.headless=true -Xms3072m -Xmx3072m -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -Dfile.encoding=UTF-8"
CATALINA_OPTS="$CATALINA_OPTS -Djava.library.path=/opt/brew/Cellar/tomcat-native/1.2.8/lib"
JRE_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_102.jdk/Contents/Home
```
- Edit Tomcat 8 `server.xml` to add regular HTTP listener for solr
- Symlink webapps:
```
$ rm -rf /opt/brew/Cellar/tomcat/8.5.4/libexec/webapps/ROOT
$ ln -sv ~/dspace/webapps/xmlui /opt/brew/Cellar/tomcat/8.5.4/libexec/webapps/ROOT
$ ln -sv ~/dspace/webapps/oai /opt/brew/Cellar/tomcat/8.5.4/libexec/webapps/oai
$ ln -sv ~/dspace/webapps/jspui /opt/brew/Cellar/tomcat/8.5.4/libexec/webapps/jspui
$ ln -sv ~/dspace/webapps/rest /opt/brew/Cellar/tomcat/8.5.4/libexec/webapps/rest
$ ln -sv ~/dspace/webapps/solr /opt/brew/Cellar/tomcat/8.5.4/libexec/webapps/solr
```
## 2016-08-09
- More tests of Atmire's 5.5 modules on a clean, working instance of `5_x-prod`
- Still fails, though perhaps differently than before (Flyway): https://gist.github.com/alanorth/5d49c45a16efd7c6bc1e6642e66118b2
- More work on Tomcat 8 and Java 8 stuff for Ansible playbooks
## 2016-08-10
- Turns out DSpace 5.x isn't ready for Tomcat 8: https://jira.duraspace.org/browse/DS-3092
- So we'll need to use Tomcat 7 + Java 8 on Ubuntu 16.04
- More work on the Ansible stuff for this, allowing Tomcat 7 to use Java 8
- Merge pull request for fixing the type Discovery index to use `dc.type` ([#262](https://github.com/ilri/DSpace/pull/262))
- Merge pull request for removing "Bitstream" text from item display, as it confuses users and isn't necessary ([#263](https://github.com/ilri/DSpace/pull/263))
## 2016-08-11
- Finally got DSpace (5.5) running on Ubuntu 16.04, Tomcat 7, Java 8, PostgreSQL 9.5 via the updated Ansible stuff
![DSpace 5.5 on Ubuntu 16.04, Tomcat 7, Java 8, PostgreSQL 9.5](/cgspace-notes/2016/08/dspace55-ubuntu16.04.png)
## 2016-08-14
- Update Mirage 2 build notes for Ubuntu 16.04: https://gist.github.com/alanorth/2cf9c15834dc68a514262fcb04004cb0
## 2016-08-15
- Notes on NodeJS + nginx + systemd: https://gist.github.com/alanorth/51acd476891c67dfe27725848cf5ace1
![ExpressJS running behind nginx](/cgspace-notes/2016/08/nodejs-nginx.png)
## 2016-08-16
- Troubleshoot Paramiko connection issues with Ansible on ILRI servers: [#37](https://github.com/ilri/rmg-ansible-public/issues/37)
- Turns out we need to add some MACs to our `sshd_config`: hmac-sha2-512,hmac-sha2-256
- Update DSpace Test's Java to version 8 to start testing this configuration ([seeing as Solr recommends it](https://wiki.apache.org/solr/ShawnHeisey))
## 2016-08-17
- More work on Let's Encrypt stuff for Ansible roles
- Yesterday Atmire responded about DSpace 5.5 issues and asked me to try the `dspace database repair` command to fix Flyway issues
- The `dspace database` command doesn't even run: https://gist.github.com/alanorth/c43c8d89e8df346d32c0ee938be90cd5
- Oops, it looks like the missing classes causing `dspace database` to fail were coming from the old `~/dspace/config/spring` folder
- After removing the spring folder and running ant install again, `dspace database` works
- I see there are missing and pending Flyway migrations, but running `dspace database repair` and `dspace database migrate` does nothing: https://gist.github.com/alanorth/41ed5abf2ff32d8ac9eedd1c3d015d70
## 2016-08-18
- Fix "CONGO,DR" country name in `input-forms.xml` ([#264](https://github.com/ilri/DSpace/pull/264))
- Also need to fix existing records using the incorrect form in the database:
```
dspace=# update metadatavalue set text_value='CONGO, DR' where resource_type_id=2 and metadata_field_id=228 and text_value='CONGO,DR';
```
- I asked a question on the DSpace mailing list about updating "preferred" forms of author names from ORCID
## 2016-08-21
- A few days ago someone on the DSpace mailing list suggested I try `dspace dsrun org.dspace.authority.UpdateAuthorities` to update preferred author names from ORCID
- If you set `auto-update-items=true` in `dspace/config/modules/solrauthority.cfg` it is supposed to update records it finds automatically
- I updated my name format on ORCID and I've been running that script a few times per day since then but nothing has changed
- Still troubleshooting Atmire modules on DSpace 5.5
- I sent them some new verbose logs: https://gist.github.com/alanorth/700748995649688148ceba89d760253e
## 2016-08-22
- Database migrations are fine on DSpace 5.1:
```
$ ~/dspace/bin/dspace database info
Database URL: jdbc:postgresql://localhost:5432/dspacetest
Database Schema: public
Database Software: PostgreSQL version 9.3.14
Database Driver: PostgreSQL Native Driver version PostgreSQL 9.1 JDBC4 (build 901)
+----------------+----------------------------+---------------------+---------+
| Version | Description | Installed on | State |
+----------------+----------------------------+---------------------+---------+
| 1.1 | Initial DSpace 1.1 databas | | PreInit |
| 1.2 | Upgrade to DSpace 1.2 sche | | PreInit |
| 1.3 | Upgrade to DSpace 1.3 sche | | PreInit |
| 1.3.9 | Drop constraint for DSpace | | PreInit |
| 1.4 | Upgrade to DSpace 1.4 sche | | PreInit |
| 1.5 | Upgrade to DSpace 1.5 sche | | PreInit |
| 1.5.9 | Drop constraint for DSpace | | PreInit |
| 1.6 | Upgrade to DSpace 1.6 sche | | PreInit |
| 1.7 | Upgrade to DSpace 1.7 sche | | PreInit |
| 1.8 | Upgrade to DSpace 1.8 sche | | PreInit |
| 3.0 | Upgrade to DSpace 3.x sche | | PreInit |
| 4.0 | Initializing from DSpace 4 | 2015-11-20 12:42:52 | Success |
| 5.0.2014.08.08 | DS-1945 Helpdesk Request a | 2015-11-20 12:42:53 | Success |
| 5.0.2014.09.25 | DS 1582 Metadata For All O | 2015-11-20 12:42:55 | Success |
| 5.0.2014.09.26 | DS-1582 Metadata For All O | 2015-11-20 12:42:55 | Success |
| 5.0.2015.01.27 | MigrateAtmireExtraMetadata | 2015-11-20 12:43:29 | Success |
| 5.1.2015.12.03 | Atmire CUA 4 migration | 2016-03-21 17:10:41 | Success |
| 5.1.2015.12.03 | Atmire MQM migration | 2016-03-21 17:10:42 | Success |
+----------------+----------------------------+---------------------+---------+
```
- So I'm not sure why they have problems when we move to DSpace 5.5 (even the 5.1 migrations themselves show as "Missing")
## 2016-08-23
- Help Paola from CCAFS with her thumbnails again
- Talk to Atmire about the DSpace 5.5 issue, and it seems to be caused by a bug in FlywayDB
- They said I should delete the Atmire migrations
```
dspacetest=# delete from schema_version where description = 'Atmire CUA 4 migration' and version='5.1.2015.12.03.2';
dspacetest=# delete from schema_version where description = 'Atmire MQM migration' and version='5.1.2015.12.03.3';
```
- After that DSpace starts up by XMLUI now has unrelated issues that I need to solve!
```
org.apache.avalon.framework.configuration.ConfigurationException: Type 'ThemeResourceReader' does not exist for 'map:read' at jndi:/localhost/themes/0_CGIAR/sitemap.xmap:136:77
context:/jndi:/localhost/themes/0_CGIAR/sitemap.xmap - 136:77
```
- Looks like we're missing some stuff in the XMLUI module's `sitemap.xmap`, as well as in each of our XMLUI themes
- Diff them with these to get the `ThemeResourceReader` changes:
- `dspace-xmlui/src/main/webapp/sitemap.xmap`
- `dspace-xmlui-mirage2/src/main/webapp/sitemap.xmap`
- Then we had some NullPointerException from the SolrLogger class, which is apparently part of Atmire's CUA module
- I tried with a small version bump to CUA but it didn't work (version `5.5-4.1.1-0`)
- Also, I started looking into huge pages to prepare for PostgreSQL 9.5, but it seems Linode's kernels don't enable them
## 2016-08-24
- Clean up and import 48 CCAFS records into DSpace Test
- SQL to get all journal titles from dc.source (55), since it's apparently used for internal DSpace filename shit, but we moved all our journal titles there a few months ago:
```
dspacetest=# select distinct text_value from metadatavalue where metadata_field_id=55 and text_value !~ '.*(\.pdf|\.png|\.PDF|\.Pdf|\.JPEG|\.jpg|\.JPG|\.jpeg|\.xls|\.rtf|\.docx?|\.potx|\.dotx|\.eqa|\.tiff|\.mp4|\.mp3|\.gif|\.zip|\.txt|\.pptx|\.indd|\.PNG|\.bmp|\.exe|org\.dspace\.app\.mediafilter).*';
```
## 2016-08-25
- Atmire suggested adding a missing bean to `dspace/config/spring/api/atmire-cua.xml` but it doesn't help:
```
...
Error creating bean with name 'MetadataStorageInfoService'
...
```
- Atmire sent an updated version of `dspace/config/spring/api/atmire-cua.xml` and now XMLUI starts but gives a null pointer exception:
```
Java stacktrace: java.lang.NullPointerException
at org.dspace.app.xmlui.aspect.statistics.Navigation.addOptions(Navigation.java:129)
at org.dspace.app.xmlui.wing.AbstractWingTransformer.startElement(AbstractWingTransformer.java:228)
at sun.reflect.GeneratedMethodAccessor126.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
at com.sun.proxy.$Proxy103.startElement(Unknown Source)
at org.apache.cocoon.environment.internal.EnvironmentChanger.startElement(EnvironmentStack.java:140)
at org.apache.cocoon.environment.internal.EnvironmentChanger.startElement(EnvironmentStack.java:140)
at org.apache.cocoon.xml.AbstractXMLPipe.startElement(AbstractXMLPipe.java:94)
...
```
- Import the 47 CCAFS records to CGSpace, creating the SimpleArchiveFormat bundles and importing like:
```
$ ./safbuilder.sh -c /tmp/Thumbnails\ to\ Upload\ to\ CGSpace/3546.csv
$ JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx512m" /home/cgspace.cgiar.org/bin/dspace import -a -e aorth@mjanja.ch -c 10568/3546 -s /tmp/Thumbnails\ to\ Upload\ to\ CGSpace/SimpleArchiveFormat -m 3546.map
```
- Finally got DSpace 5.5 working with the Atmire modules after a few rounds of back and forth with Atmire devs
## 2016-08-26
- CGSpace had issues tonight, not entirely crashing, but becoming unresponsive
- The dspace log had this:
```
2016-08-26 20:48:05,040 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error - org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
```
- Related to /rest no doubt
## 2016-08-27
- Run corrections for Delia Grace and `CONGO, DR`, and deploy August changes to CGSpace
- Run all system updates and reboot the server
Reference in New Issue View Git Blame Copy Permalink