- And it seems that we need to enabled `pg_crypto` now (used for UUIDs):
```
$ psql -h localhost -U postgres dspace63
dspace63=# CREATE EXTENSION pgcrypto;
CREATE EXTENSION pgcrypto;
```
- I tried importing a PostgreSQL snapshot from CGSpace and had errors due to missing Atmire database migrations
- If I try to run `dspace database migrate` I get the IDs of the migrations that are missing
- I delete them manually in psql:
```
dspace63=# DELETE FROM schema_version WHERE version IN ('5.0.2015.01.27', '5.6.2015.12.03.2', '5.6.2016.08.08', '5.0.2017.04.28', '5.0.2017.09.25', '5.8.2015.12.03.3');
```
- Then I ran `dspace database migrate` and got an error:
Statement : ALTER TABLE metadatavalue DROP COLUMN IF EXISTS resource_id
at org.flywaydb.core.internal.dbsupport.SqlScript.execute(SqlScript.java:117)
at org.flywaydb.core.internal.resolver.sql.SqlMigrationExecutor.execute(SqlMigrationExecutor.java:71)
at org.flywaydb.core.internal.command.DbMigrate.doMigrate(DbMigrate.java:352)
at org.flywaydb.core.internal.command.DbMigrate.access$1100(DbMigrate.java:47)
at org.flywaydb.core.internal.command.DbMigrate$4.doInTransaction(DbMigrate.java:308)
at org.flywaydb.core.internal.util.jdbc.TransactionTemplate.execute(TransactionTemplate.java:72)
at org.flywaydb.core.internal.command.DbMigrate.applyMigration(DbMigrate.java:305)
at org.flywaydb.core.internal.command.DbMigrate.access$1000(DbMigrate.java:47)
at org.flywaydb.core.internal.command.DbMigrate$2.doInTransaction(DbMigrate.java:230)
at org.flywaydb.core.internal.command.DbMigrate$2.doInTransaction(DbMigrate.java:173)
at org.flywaydb.core.internal.util.jdbc.TransactionTemplate.execute(TransactionTemplate.java:72)
at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:173)
at org.flywaydb.core.Flyway$1.execute(Flyway.java:959)
at org.flywaydb.core.Flyway$1.execute(Flyway.java:917)
at org.flywaydb.core.Flyway.execute(Flyway.java:1373)
at org.flywaydb.core.Flyway.migrate(Flyway.java:917)
at org.dspace.storage.rdbms.DatabaseUtils.updateDatabase(DatabaseUtils.java:662)
... 8 more
Caused by: org.postgresql.util.PSQLException: ERROR: cannot drop table metadatavalue column resource_id because other objects depend on it
Detail: view eperson_metadata depends on table metadatavalue column resource_id
Hint: Use DROP ... CASCADE to drop the dependent objects too.
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2422)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2167)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:307)
at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:293)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:270)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:266)
at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:291)
at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:291)
at org.flywaydb.core.internal.dbsupport.JdbcTemplate.executeStatement(JdbcTemplate.java:238)
at org.flywaydb.core.internal.dbsupport.SqlScript.execute(SqlScript.java:114)
... 24 more
```
- I think I might need to update the sequences first... nope
- Perhaps it's due to some missing bitstream IDs and I need to run `dspace cleanup` on CGSpace and take a new PostgreSQL dump... nope
- A thread on the dspace-tech mailing list regarding this migration noticed that his database had some views created that were using the `resource_id` column
- Our database had the same issue, where the `eperson_metadata` view was created by something (Atmire module?) but has no references in the vanilla DSpace code, so I dropped it and tried the migration again:
```
dspace63=# DROP VIEW eperson_metadata;
DROP VIEW
```
- After that the migration was successful and DSpace starts up successfully and begins indexing
- xmlui, solr, jspui, rest, and oai are working (rest was redirecting to HTTPS, so I set the Tomcat connector to `secure="true"` and it fixed it on localhost, but caused other issues so I disabled it for now)
dspace63=# DELETE FROM schema_version WHERE version IN ('5.0.2015.01.27', '5.6.2015.12.03.2', '5.6.2016.08.08', '5.0.2017.04.28', '5.0.2017.09.25', '5.8.2015.12.03.3');
- I notice that the indexing doesn't work correctly if I start it manually with `dspace index-discovery -b` (search.resourceid becomes an integer!)
- If I induce an indexing by touching `dspace/solr/search/conf/reindex.flag` the search.resourceid are all UUIDs...
- Speaking of database stuff, there was a performance-related update for the [indexes that we used in DSpace 5](https://github.com/DSpace/DSpace/pull/1791/)
- We might want to [apply it in DSpace 6](https://github.com/DSpace/DSpace/pull/1792), as it was never merged to 6.x, but it helped with the performance of `/submissions` in XMLUI for us in [2018-03]({{< relref path="2018-03.md" >}})
- The indexing issue I was having yesterday seems to only present itself the first time a new installation is running DSpace 6
- Once the indexing induced by touching `dspace/solr/search/conf/reindex.flag` has finished, subsequent manual invocations of `dspace index-discovery -b` work as expected
- Nevertheless, I sent a message to the dspace-tech mailing list describing the issue to see if anyone has any comments
- I am seeing that the number of important commits on the unreleased DSpace 6.4 are really numerous and it might be better for us to target that version
- I did a simple test and it's easy to rebase my current 6.3 branch on top of the upstream `dspace-6_x` branch:
```
$ git checkout -b 6_x-dev64 6_x-dev
$ git rebase -i upstream/dspace-6_x
```
- I finally understand why our themes show all the "Browse by" buttons on community and collection pages in DSpace 6.x
- The code in `./dspace-xmlui/src/main/java/org/dspace/app/xmlui/aspect/browseArtifacts/CommunityBrowse.java` iterates over all the browse indexes and prints them when it is called
- The XMLUI theme code in `dspace/modules/xmlui-mirage2/src/main/webapp/themes/0_CGIAR/xsl/preprocess/browse.xsl` calls the template because the id of the div matches "aspect.browseArtifacts.CommunityBrowse.list.community-browse"
- I checked the DRI of a community page on my local 6.x and DSpace Test 5.x by appending `?XML` to the URL and I see the ID is missing on DSpace 5.x
- The issue is the same with the ordering of the "My Account" link, but in Navigation.java
- I tried modifying `preprocess/browse.xsl` but it always ends up printing some default list of browse by links...
- I'm starting to wonder if Atmire's modules somehow override this, as I don't see how `CommunityBrowse.java` can behave like ours on DSpace 5.x unless they have overridden it (as the open source code is the same in 5.x and 6.x)
- At least the "account" link in the sidebar is overridden in our 5.x branch because Atmire copied a modified `Navigation.java` to the local xmlui modules folder... so that explains that (and it's easy to replicate in 6.x)
- Checking out the DSpace 6.x REST API query client
- There is a [tutorial](https://terrywbrady.github.io/restReportTutorial/intro) that explains how it works and I see it is very powerful because you can export a CSV of results in order to fix and re-upload them with batch import!
- Custom queries can be added in `dspace-rest/src/main/webapp/static/reports/restQueryReport.js`
- I noticed two new bots in the logs with the following user agents:
-`Jersey/2.6 (HttpUrlConnection 1.8.0_152)`
-`magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)`
- I filed an [issue to add Jersey to the COUNTER-Robots](https://github.com/atmire/COUNTER-Robots/issues/30) list
- Peter noticed that the statlets on community, collection, and item pages aren't working on CGSpace
- I thought it might be related to the fact that the yearly sharding didn't complete successfully this year so the `statistics-2019` core is empty
- I removed the `statistics-2019` core and had to restart Tomcat like six times before all cores would load properly (ugh!!!!)
- After that the statlets were working properly...
- Run all system updates on DSpace Test (linode19) and restart it
dspace63=# DELETE FROM schema_version WHERE version IN ('5.0.2015.01.27', '5.6.2015.12.03.2', '5.6.2016.08.08', '5.0.2017.04.28', '5.0.2017.09.25', '5.8.2015.12.03.3');
dspace63=# DROP VIEW eperson_metadata;
dspace63=# \q
```
- I purged ~33,000 hits from the "Jersey/2.6" bot in CGSpace's statistics using my `check-spider-hits.sh` script:
- I noticed another user agen in the logs that we should add to the list:
```
ReactorNetty/0.9.2.RELEASE
```
- I made [an issue on the COUNTER-Robots repository](https://github.com/atmire/COUNTER-Robots/issues/31)
- I found a [nice tool for exporting and importing Solr records](https://github.com/freedev/solr-import-export-json) and it seems to workfor exporting our 2019 stats from the large statistics core!
- Follow up with [Atmire about DSpace 6.x upgrade](https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=706)
- I raised the issue of targetting 6.4-SNAPSHOT as well as the Discovery indexing performance issues in 6.x
## 2020-02-11
- Maria from Bioversity asked me to add some ORCID iDs to our controlled vocabulary so I combined them with our existing ones and updated the names from the ORCID API:
- Udana from IWMI asked about the OAI base URL for their community on CGSpace
- I think it should be this: https://cgspace.cgiar.org/oai/request?verb=ListRecords&metadataPrefix=oai_dc&set=com_10568_16814
## 2020-02-19
- I noticed a thread on the mailing list about the Tomcat header size and Solr max boolean clauses error
- The solution is to do as we have done and increase the headers / boolean clauses, or to simply [disable access rights awareness](https://wiki.lyrasis.org/display/DSPACE/TechnicalFaq#TechnicalFAQ-I'mgetting%22SolrException:BadRequest%22followedbyalongqueryora%22tooManyClauses%22Exception) in Discovery
- I applied the fix to the `5_x-prod` branch and cherry-picked it to `6_x-dev`
- Upgrade Tomcat from 7.0.99 to 7.0.100 in [Ansible infrastructure playbooks](https://github.com/ilri/rmg-ansible-public)
- Upgrade PostgreSQL JDBC driver from 42.2.9 to 42.2.10 in [Ansible infrastructure playbooks](https://github.com/ilri/rmg-ansible-public)
- Run Tomcat and PostgreSQL JDBC driver updates on DSpace Test (linode19)
- And removing the extra jfreechart library and restarting Tomcat I was able to load the usage statistics graph on DSpace Test...
- Hmm, actually I think this is an Java bug, perhaps introduced or at [least present in 18.04](https://bugs.openjdk.java.net/browse/JDK-8204862), with lots of [references](https://code-maven.com/slides/jenkins-intro/no-graph-error) to it [happening in other](https://issues.jenkins-ci.org/browse/JENKINS-39636) configurations like Debian 9 with Jenkins, etc...
- Apparently if you use the *non-headless* version of openjdk this doesn't happen... but that pulls in X11 stuff so no thanks
- Also, I see dozens of occurences of this going back over one month (we have logs for about that period):
```
# grep -c 'initialize class org.jfree.chart.JFreeChart' dspace.log.2020-0*
dspace.log.2020-01-12:4
dspace.log.2020-01-13:66
dspace.log.2020-01-14:4
dspace.log.2020-01-15:36
dspace.log.2020-01-16:88
dspace.log.2020-01-17:4
dspace.log.2020-01-18:4
dspace.log.2020-01-19:4
dspace.log.2020-01-20:4
dspace.log.2020-01-21:4
...
```
- I deployed the fix on CGSpace (linode18) and I was able to see the graphs in the Atmire CUA Usage Statistics...
- On an unrelated note there is something weird going on in that I see millions of hits from IP 34.218.226.147 in Solr statistics, but if I remember correctly that IP belongs to CodeObia's AReS explorer, but it should only be using REST and therefore no Solr statistics...?
- I'm a little suspicious of the 2012, 2013, and 2014 numbers, though
- I should facet those years by IP and see if any stand out...
- The next thing I need to do is figure out why the nginx IP to bot mapping isn't working...
- Actually, and I've probably learned this before, but the bot mapping is working, but nginx only logs the real user agent (of course!), as I'm only using the mapped one in the proxy pass...
- This trick for adding a header with the mapped "ua" variable is nice:
```
add_header X-debug-message "ua is $ua" always;
```
- Then in the HTTP response you see:
```
X-debug-message: ua is bot
```
- So the IP to bot mapping is working, phew.
- More bad news, I checked the remaining IPs in our existing bot IP mapping, and there are statistics registered for them!
- For example, ciat.cgiar.org was previously 104.196.152.243, but it is now 35.237.175.180, which I had noticed as a "mystery" client on Google Cloud in 2018-09
- Others I should probably add to the nginx bot map list are:
- wle.cgiar.org (70.32.90.172)
- ccafs.cgiar.org (205.186.128.185)
- another CIAT scraper using the PHP GuzzleHttp library (45.5.184.72)