From ad60e0525ecc42a7d33cba5bdc7a4a24ff038113 Mon Sep 17 00:00:00 2001 From: Alan Orth Date: Sun, 15 Nov 2020 11:35:31 +0200 Subject: [PATCH] Add notes for 2020-11-15 --- content/posts/2020-11.md | 51 ++++++++++++++++++++ docs/2020-11/index.html | 62 +++++++++++++++++++++++-- docs/categories/index.html | 2 +- docs/categories/notes/index.html | 2 +- docs/categories/notes/page/2/index.html | 2 +- docs/categories/notes/page/3/index.html | 2 +- docs/categories/notes/page/4/index.html | 2 +- docs/index.html | 2 +- docs/page/2/index.html | 2 +- docs/page/3/index.html | 2 +- docs/page/4/index.html | 2 +- docs/page/5/index.html | 2 +- docs/page/6/index.html | 2 +- docs/page/7/index.html | 2 +- docs/posts/index.html | 2 +- docs/posts/page/2/index.html | 2 +- docs/posts/page/3/index.html | 2 +- docs/posts/page/4/index.html | 2 +- docs/posts/page/5/index.html | 2 +- docs/posts/page/6/index.html | 2 +- docs/posts/page/7/index.html | 2 +- docs/sitemap.xml | 10 ++-- 22 files changed, 133 insertions(+), 28 deletions(-) diff --git a/content/posts/2020-11.md b/content/posts/2020-11.md index 1d9e73844..729155757 100644 --- a/content/posts/2020-11.md +++ b/content/posts/2020-11.md @@ -127,6 +127,8 @@ dspace=# COMMIT; - I did some tests to add a usage statistics chart to the item views on DSpace Test - It is inspired by Salem's work on WorldFish's repository, and it hits the dspace-statistics-api for the current item and displays a graph - I got it working very easily for all-time statistics with Chart.js, but I think I will need to use Highcharts or something else because Chart.js is HTML5 canvas and doesn't allow theming via CSS (so our Bootstrap brand colors for each theme won't work) + - Hmm, Highcharts is not licensed under and open source license so I will not use it + - Perhaps I'll use Chartist with the popover plugin... - I think I'll pursue this after the DSpace 6 upgrade... ## 2020-11-12 @@ -155,4 +157,53 @@ facet=true&facet.field=owningComm&facet.mincount=1&facet.limit=100&facet.offset= - I hadn't seen monit before, but the others are already in DSpace's spider agents lists for some time so probably only appear in older stats cores - The issue with purging these using `check-spider-hits.sh` is that it can't do case-insensitive regexes and some metacharacters like `\s` don't work so I added case-sensitive patterns to a local agents file and purged them with the script +## 2020-11-15 + +- Upgrade CGSpace to DSpace 6.3 + - First build, update, and migrate the database: + +``` +$ dspace cleanup -v +$ git checkout origin/6_x-dev-atmire-modules +$ npm install -g yarn +$ chrt -b 0 mvn -U -Dmirage2.on=true -Dmirage2.deps.included=false -P \!dspace-lni,\!dspace-rdf,\!dspace-sword,\!dspace-swordv2,\!dspace-jspui clean package +$ sudo su - postgres +$ psql dspace -c 'CREATE EXTENSION pgcrypto;' +$ psql dspace -c "DELETE FROM schema_version WHERE version IN ('5.8.2015.12.03.3');" +$ exit +$ rm -rf /home/cgspace/config/spring +$ ant update +$ dspace database info +$ dspace database migrate +$ sudo systemctl start tomcat7 +``` + +- After starting Tomcat DSpace should start up OK and begin Discovery indexing, but I want to also upgrade from PostgreSQL 9.6 to 10 + - I installed and configured PostgreSQL 10 using the Ansible playbooks, then migrated the database manually: + +``` +# systemctl stop tomcat7 +# pg_ctlcluster 9.6 main stop +# tar -cvzpf var-lib-postgresql-9.6.tar.gz /var/lib/postgresql/9.6 +# tar -cvzpf etc-postgresql-9.6.tar.gz /etc/postgresql/9.6 +# pg_ctlcluster 10 main stop +# pg_dropcluster 10 main +# pg_upgradecluster 9.6 main +# pg_dropcluster 9.6 main +# systemctl start postgresql +# dpkg -l | grep postgresql | grep 9.6 | awk '{print $2}' | xargs dpkg -r +``` + +- Then I ran all system updates and rebooted the server... +- After the server came back up I re-ran the Ansible playbook to make sure all configs and services were updated +- I disabled the dspace-statistsics-api for now because it won't work until I migrate all the Solr statistics anyways +- Start a full Discovery re-indexing: + +``` +$ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b +``` + +- I updated all the Ansible infrastructure and DSpace branches to be the DSpace 6 ones +- I will wait until the Discovery indexing is finished to start doing the Solr statistics migration + diff --git a/docs/2020-11/index.html b/docs/2020-11/index.html index 0b739ed46..9e80fde15 100644 --- a/docs/2020-11/index.html +++ b/docs/2020-11/index.html @@ -17,7 +17,7 @@ So far we’ve spent at least fifty hours to process the statistics and stat - + @@ -39,9 +39,9 @@ So far we’ve spent at least fifty hours to process the statistics and stat "@type": "BlogPosting", "headline": "November, 2020", "url": "https://alanorth.github.io/cgspace-notes/2020-11/", - "wordCount": "1266", + "wordCount": "1565", "datePublished": "2020-11-01T13:11:54+02:00", - "dateModified": "2020-11-10T17:00:02+02:00", + "dateModified": "2020-11-13T09:22:18+02:00", "author": { "@type": "Person", "name": "Alan Orth" @@ -257,7 +257,12 @@ dspace=# COMMIT;
  • I did some tests to add a usage statistics chart to the item views on DSpace Test
    • @@ -293,6 +298,55 @@ dspace=# COMMIT; +

    2020-11-15

    + +
    $ dspace cleanup -v
+$ git checkout origin/6_x-dev-atmire-modules
+$ npm install -g yarn
+$ chrt -b 0 mvn -U -Dmirage2.on=true -Dmirage2.deps.included=false -P \!dspace-lni,\!dspace-rdf,\!dspace-sword,\!dspace-swordv2,\!dspace-jspui clean package
+$ sudo su - postgres
+$ psql dspace -c 'CREATE EXTENSION pgcrypto;'
+$ psql dspace -c "DELETE FROM schema_version WHERE version IN ('5.8.2015.12.03.3');"
+$ exit
+$ rm -rf /home/cgspace/config/spring
+$ ant update
+$ dspace database info
+$ dspace database migrate
+$ sudo systemctl start tomcat7
+
    +
    # systemctl stop tomcat7
+# pg_ctlcluster 9.6 main stop
+# tar -cvzpf var-lib-postgresql-9.6.tar.gz /var/lib/postgresql/9.6
+# tar -cvzpf etc-postgresql-9.6.tar.gz /etc/postgresql/9.6
+# pg_ctlcluster 10 main stop
+# pg_dropcluster 10 main
+# pg_upgradecluster 9.6 main
+# pg_dropcluster 9.6 main
+# systemctl start postgresql
+# dpkg -l | grep postgresql | grep 9.6 | awk '{print $2}' | xargs dpkg -r
+
    +
    $ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
    diff --git a/docs/categories/index.html b/docs/categories/index.html index 5f0b67863..d9beb3f92 100644 --- a/docs/categories/index.html +++ b/docs/categories/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/categories/notes/index.html b/docs/categories/notes/index.html index 717996bee..7f49da826 100644 --- a/docs/categories/notes/index.html +++ b/docs/categories/notes/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/categories/notes/page/2/index.html b/docs/categories/notes/page/2/index.html index fdb267c95..5fcbc420e 100644 --- a/docs/categories/notes/page/2/index.html +++ b/docs/categories/notes/page/2/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/categories/notes/page/3/index.html b/docs/categories/notes/page/3/index.html index e7ab6f81a..8cda71291 100644 --- a/docs/categories/notes/page/3/index.html +++ b/docs/categories/notes/page/3/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/categories/notes/page/4/index.html b/docs/categories/notes/page/4/index.html index f18ec8ec0..00e7a4eb9 100644 --- a/docs/categories/notes/page/4/index.html +++ b/docs/categories/notes/page/4/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/index.html b/docs/index.html index 4e3cb818e..3be34bba0 100644 --- a/docs/index.html +++ b/docs/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/page/2/index.html b/docs/page/2/index.html index 184160d27..88787c306 100644 --- a/docs/page/2/index.html +++ b/docs/page/2/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/page/3/index.html b/docs/page/3/index.html index 14dbc9027..910f13a34 100644 --- a/docs/page/3/index.html +++ b/docs/page/3/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/page/4/index.html b/docs/page/4/index.html index e31f8fe2c..813735e21 100644 --- a/docs/page/4/index.html +++ b/docs/page/4/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/page/5/index.html b/docs/page/5/index.html index e93688575..2d8a8009a 100644 --- a/docs/page/5/index.html +++ b/docs/page/5/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/page/6/index.html b/docs/page/6/index.html index f26af7ea6..4b16f168f 100644 --- a/docs/page/6/index.html +++ b/docs/page/6/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/page/7/index.html b/docs/page/7/index.html index e6f1e64e2..b3c182d72 100644 --- a/docs/page/7/index.html +++ b/docs/page/7/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/posts/index.html b/docs/posts/index.html index 201f0ce61..cecc32f30 100644 --- a/docs/posts/index.html +++ b/docs/posts/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/posts/page/2/index.html b/docs/posts/page/2/index.html index e0d56313b..37fb781cb 100644 --- a/docs/posts/page/2/index.html +++ b/docs/posts/page/2/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/posts/page/3/index.html b/docs/posts/page/3/index.html index af1632184..8c783f22a 100644 --- a/docs/posts/page/3/index.html +++ b/docs/posts/page/3/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/posts/page/4/index.html b/docs/posts/page/4/index.html index 38d0e92e4..c1cccb820 100644 --- a/docs/posts/page/4/index.html +++ b/docs/posts/page/4/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/posts/page/5/index.html b/docs/posts/page/5/index.html index c4bd7702e..936a67627 100644 --- a/docs/posts/page/5/index.html +++ b/docs/posts/page/5/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/posts/page/6/index.html b/docs/posts/page/6/index.html index 89c0d123b..9c8cc8bd1 100644 --- a/docs/posts/page/6/index.html +++ b/docs/posts/page/6/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/posts/page/7/index.html b/docs/posts/page/7/index.html index 110193aaa..63623abf4 100644 --- a/docs/posts/page/7/index.html +++ b/docs/posts/page/7/index.html @@ -9,7 +9,7 @@ - + diff --git a/docs/sitemap.xml b/docs/sitemap.xml index e592f0970..20af8c4a0 100644 --- a/docs/sitemap.xml +++ b/docs/sitemap.xml @@ -4,27 +4,27 @@ https://alanorth.github.io/cgspace-notes/categories/ - 2020-11-10T17:00:02+02:00 + 2020-11-13T09:22:18+02:00 https://alanorth.github.io/cgspace-notes/ - 2020-11-10T17:00:02+02:00 + 2020-11-13T09:22:18+02:00 https://alanorth.github.io/cgspace-notes/categories/notes/ - 2020-11-10T17:00:02+02:00 + 2020-11-13T09:22:18+02:00 https://alanorth.github.io/cgspace-notes/2020-11/ - 2020-11-10T17:00:02+02:00 + 2020-11-13T09:22:18+02:00 https://alanorth.github.io/cgspace-notes/posts/ - 2020-11-10T17:00:02+02:00 + 2020-11-13T09:22:18+02:00