From 51aa83e6c3ca8714c2c92ad76066576a7a1d39b9 Mon Sep 17 00:00:00 2001
From: Alan Orth
Date: Thu, 8 Nov 2018 09:02:20 +0200
Subject: [PATCH] Update notes for 2018-11-08
---
content/posts/2018-11.md | 6 +
docs/2015-11/index.html | 8 +-
docs/2015-12/index.html | 8 +-
docs/2016-01/index.html | 8 +-
docs/2016-02/index.html | 8 +-
docs/2016-03/index.html | 8 +-
docs/2016-04/index.html | 8 +-
docs/2016-05/index.html | 8 +-
docs/2016-06/index.html | 8 +-
docs/2016-07/index.html | 8 +-
docs/2016-08/index.html | 8 +-
docs/2016-09/index.html | 8 +-
docs/2016-10/index.html | 8 +-
docs/2016-11/index.html | 8 +-
docs/2016-12/index.html | 8 +-
docs/2017-01/index.html | 8 +-
docs/2017-02/index.html | 8 +-
docs/2017-03/index.html | 8 +-
docs/2017-04/index.html | 8 +-
docs/2017-05/index.html | 2 +-
docs/2017-06/index.html | 2 +-
docs/2017-07/index.html | 8 +-
docs/2017-08/index.html | 8 +-
docs/2017-09/index.html | 8 +-
docs/2017-10/index.html | 8 +-
docs/2017-11/index.html | 8 +-
docs/2017-12/index.html | 8 +-
docs/2018-01/index.html | 8 +-
docs/2018-02/index.html | 8 +-
docs/2018-03/index.html | 8 +-
docs/2018-04/index.html | 8 +-
docs/2018-05/index.html | 8 +-
docs/2018-06/index.html | 8 +-
docs/2018-07/index.html | 8 +-
docs/2018-08/index.html | 8 +-
docs/2018-09/index.html | 8 +-
docs/2018-10/index.html | 8 +-
docs/2018-11/index.html | 22 ++--
docs/404.html | 2 +-
docs/categories/index.html | 22 +---
docs/categories/notes/index.html | 4 +-
docs/categories/notes/index.xml | 4 +-
docs/categories/page/2/index.html | 18 +--
docs/categories/page/3/index.html | 22 +---
docs/categories/page/4/index.html | 18 +--
docs/cgiar-library-migration/index.html | 4 +-
docs/index.html | 22 +---
docs/index.xml | 144 ++++++------------------
docs/page/2/index.html | 18 +--
docs/page/3/index.html | 22 +---
docs/page/4/index.html | 18 +--
docs/posts/index.html | 22 +---
docs/posts/index.xml | 144 ++++++------------------
docs/posts/page/2/index.html | 18 +--
docs/posts/page/3/index.html | 22 +---
docs/posts/page/4/index.html | 18 +--
docs/robots.txt | 2 +-
docs/sitemap.xml | 20 ++--
docs/tags/index.html | 22 +---
docs/tags/notes/index.html | 22 +---
docs/tags/notes/index.xml | 140 ++++++-----------------
docs/tags/notes/page/2/index.html | 18 +--
docs/tags/notes/page/3/index.html | 22 +---
docs/tags/notes/page/4/index.html | 16 +--
docs/tags/page/2/index.html | 18 +--
docs/tags/page/3/index.html | 22 +---
docs/tags/page/4/index.html | 18 +--
67 files changed, 196 insertions(+), 970 deletions(-)
diff --git a/content/posts/2018-11.md b/content/posts/2018-11.md
index 60d69d1a1..cabad2045 100644
--- a/content/posts/2018-11.md
+++ b/content/posts/2018-11.md
@@ -274,4 +274,10 @@ $ time ./rest-find-collections.py 10568/27629 --rest-url https://dspacetest.cgia
- Update my [dspace-statistics-api](https://github.com/ilri/dspace-statistics-api) to use a database management class with Python contexts so that connections and cursors are automatically opened and closed
- Tag version 0.7.0 of the dspace-statistics-api
+## 2018-11-08
+
+- I deployed verison 0.7.0 of the dspace-statistics-api on DSpace Test (linode19) so I can test it for a few days (and check the Munin stats to see the change in database connections) before deploying on CGSpace
+- I also enabled systemd's persistent journal by setting [`Storage=persistent` in *journald.conf*](https://www.freedesktop.org/software/systemd/man/journald.conf.html)
+- Apparently [Ubuntu 16.04 defaulted to using rsyslog for boot records until early 2018](https://www.freedesktop.org/software/systemd/man/journald.conf.html), so I removed `rsyslog` too
+
diff --git a/docs/2015-11/index.html b/docs/2015-11/index.html
index 7777389e1..0bd512169 100644
--- a/docs/2015-11/index.html
+++ b/docs/2015-11/index.html
@@ -16,8 +16,6 @@ Last week I had increased the limit from 30 to 60, which seemed to help, but now
$ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
78
-
-
" />
@@ -35,10 +33,8 @@ Last week I had increased the limit from 30 to 60, which seemed to help, but now
$ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
78
-
-
"/>
-
+
@@ -128,8 +124,6 @@ $ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspac
78
-
-
- For now I have increased the limit from 60 to 90, run updates, and rebooted the server
diff --git a/docs/2015-12/index.html b/docs/2015-12/index.html
index e20e43735..9f44e2584 100644
--- a/docs/2015-12/index.html
+++ b/docs/2015-12/index.html
@@ -17,8 +17,6 @@ Replace lzop with xz in log compression cron jobs on DSpace Test—it uses less
-rw-rw-r-- 1 tomcat7 tomcat7 2.0M Nov 18 23:59 dspace.log.2015-11-18
-rw-rw-r-- 1 tomcat7 tomcat7 387K Nov 18 23:59 dspace.log.2015-11-18.lzo
-rw-rw-r-- 1 tomcat7 tomcat7 169K Nov 18 23:59 dspace.log.2015-11-18.xz
-
-
" />
@@ -37,10 +35,8 @@ Replace lzop with xz in log compression cron jobs on DSpace Test—it uses less
-rw-rw-r-- 1 tomcat7 tomcat7 2.0M Nov 18 23:59 dspace.log.2015-11-18
-rw-rw-r-- 1 tomcat7 tomcat7 387K Nov 18 23:59 dspace.log.2015-11-18.lzo
-rw-rw-r-- 1 tomcat7 tomcat7 169K Nov 18 23:59 dspace.log.2015-11-18.xz
-
-
"/>
-
+
@@ -131,8 +127,6 @@ Replace lzop with xz in log compression cron jobs on DSpace Test—it uses less
-rw-rw-r-- 1 tomcat7 tomcat7 169K Nov 18 23:59 dspace.log.2015-11-18.xz
-
-
- I had used lrzip once, but it needs more memory and is harder to use as it requires the lrztar wrapper
- Need to remember to go check if everything is ok in a few days and then change CGSpace
diff --git a/docs/2016-01/index.html b/docs/2016-01/index.html
index 247b70f3a..ee12ccce6 100644
--- a/docs/2016-01/index.html
+++ b/docs/2016-01/index.html
@@ -12,8 +12,6 @@
Move ILRI collection 10568/12503 from 10568/27869 to 10568/27629 using the move_collections.sh script I wrote last year.
I realized it is only necessary to clear the Cocoon cache after moving collections—rather than reindexing—as no metadata has changed, and therefore no search or browse indexes need to be updated.
Update GitHub wiki for documentation of maintenance tasks.
-
-
" />
@@ -27,10 +25,8 @@ Update GitHub wiki for documentation of maintenance tasks.
Move ILRI collection 10568/12503 from 10568/27869 to 10568/27629 using the move_collections.sh script I wrote last year.
I realized it is only necessary to clear the Cocoon cache after moving collections—rather than reindexing—as no metadata has changed, and therefore no search or browse indexes need to be updated.
Update GitHub wiki for documentation of maintenance tasks.
-
-
"/>
-
+
@@ -116,8 +112,6 @@ Update GitHub wiki for documentation of maintenance tasks.
- Update GitHub wiki for documentation of maintenance tasks.
-
-
2016-01-14
diff --git a/docs/2016-02/index.html b/docs/2016-02/index.html
index 45e17d3e0..a4200f30e 100644
--- a/docs/2016-02/index.html
+++ b/docs/2016-02/index.html
@@ -19,8 +19,6 @@ I noticed we have a very interesting list of countries on CGSpace:
Not only are there 49,000 countries, we have some blanks (25)…
Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE”
-
-
" />
@@ -41,10 +39,8 @@ I noticed we have a very interesting list of countries on CGSpace:
Not only are there 49,000 countries, we have some blanks (25)…
Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE”
-
-
"/>
-
+
@@ -137,8 +133,6 @@ Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE&r
- Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE”
-
-
2016-02-06
diff --git a/docs/2016-03/index.html b/docs/2016-03/index.html
index 900f8eb9f..7146f599d 100644
--- a/docs/2016-03/index.html
+++ b/docs/2016-03/index.html
@@ -12,8 +12,6 @@
Looking at issues with author authorities on CGSpace
For some reason we still have the index-lucene-update cron job active on CGSpace, but I’m pretty sure we don’t need it as of the latest few versions of Atmire’s Listings and Reports module
Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Java JDK 1.7 to match environment on CGSpace server
-
-
" />
@@ -27,10 +25,8 @@ Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Ja
Looking at issues with author authorities on CGSpace
For some reason we still have the index-lucene-update cron job active on CGSpace, but I’m pretty sure we don’t need it as of the latest few versions of Atmire’s Listings and Reports module
Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Java JDK 1.7 to match environment on CGSpace server
-
-
"/>
-
+
@@ -116,8 +112,6 @@ Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Ja
- Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Java JDK 1.7 to match environment on CGSpace server
-
-
2016-03-07
diff --git a/docs/2016-04/index.html b/docs/2016-04/index.html
index 921585703..75604db80 100644
--- a/docs/2016-04/index.html
+++ b/docs/2016-04/index.html
@@ -14,8 +14,6 @@ We are backing up all logs in the log folder, including useless stuff like solr,
After running DSpace for over five years I’ve never needed to look in any other log file than dspace.log, leave alone one from last year!
This will save us a few gigs of backup space we’re paying for on S3
Also, I noticed the checker log has some errors we should pay attention to:
-
-
" />
@@ -31,10 +29,8 @@ We are backing up all logs in the log folder, including useless stuff like solr,
After running DSpace for over five years I’ve never needed to look in any other log file than dspace.log, leave alone one from last year!
This will save us a few gigs of backup space we’re paying for on S3
Also, I noticed the checker log has some errors we should pay attention to:
-
-
"/>
-
+
@@ -122,8 +118,6 @@ Also, I noticed the checker log has some errors we should pay attention to:
- Also, I noticed the
checker
log has some errors we should pay attention to:
-
-
Run start time: 03/06/2016 04:00:22
Error retrieving bitstream ID 71274 from asset store.
java.io.FileNotFoundException: /home/cgspace.cgiar.org/assetstore/64/29/06/64290601546459645925328536011917633626 (Too many open files)
diff --git a/docs/2016-05/index.html b/docs/2016-05/index.html
index b2bef7aa9..2175f48ab 100644
--- a/docs/2016-05/index.html
+++ b/docs/2016-05/index.html
@@ -16,8 +16,6 @@ There are 3,000 IPs accessing the REST API in a 24-hour period!
# awk '{print $1}' /var/log/nginx/rest.log | uniq | wc -l
3168
-
-
" />
@@ -35,10 +33,8 @@ There are 3,000 IPs accessing the REST API in a 24-hour period!
# awk '{print $1}' /var/log/nginx/rest.log | uniq | wc -l
3168
-
-
"/>
-
+
@@ -128,8 +124,6 @@ There are 3,000 IPs accessing the REST API in a 24-hour period!
3168
-
-
- The two most often requesters are in Ethiopia and Colombia: 213.55.99.121 and 181.118.144.29
- 100% of the requests coming from Ethiopia are like this and result in an HTTP 500:
diff --git a/docs/2016-06/index.html b/docs/2016-06/index.html
index 158b88d18..bba599b73 100644
--- a/docs/2016-06/index.html
+++ b/docs/2016-06/index.html
@@ -15,8 +15,6 @@ After reading the OAI documentation and testing with an OAI validator I found ou
This is their publications set: http://ebrary.ifpri.org/oai/oai.php?verb=ListRecords&from=2016-01-01&set=p15738coll2&metadataPrefix=oai_dc
You can see the others by using the OAI ListSets verb: http://ebrary.ifpri.org/oai/oai.php?verb=ListSets
Working on second phase of metadata migration, looks like this will work for moving CPWF-specific data in dc.identifier.fund to cg.identifier.cpwfproject and then the rest to dc.description.sponsorship
-
-
" />
@@ -33,10 +31,8 @@ After reading the OAI documentation and testing with an OAI validator I found ou
This is their publications set: http://ebrary.ifpri.org/oai/oai.php?verb=ListRecords&from=2016-01-01&set=p15738coll2&metadataPrefix=oai_dc
You can see the others by using the OAI ListSets verb: http://ebrary.ifpri.org/oai/oai.php?verb=ListSets
Working on second phase of metadata migration, looks like this will work for moving CPWF-specific data in dc.identifier.fund to cg.identifier.cpwfproject and then the rest to dc.description.sponsorship
-
-
"/>
-
+
@@ -125,8 +121,6 @@ Working on second phase of metadata migration, looks like this will work for mov
- Working on second phase of metadata migration, looks like this will work for moving CPWF-specific data in
dc.identifier.fund
to cg.identifier.cpwfproject
and then the rest to dc.description.sponsorship
-
-
dspacetest=# update metadatavalue set metadata_field_id=130 where metadata_field_id=75 and (text_value like 'PN%' or text_value like 'PHASE%' or text_value = 'CBA' or text_value = 'IA');
UPDATE 497
dspacetest=# update metadatavalue set metadata_field_id=29 where metadata_field_id=75;
diff --git a/docs/2016-07/index.html b/docs/2016-07/index.html
index a8977b88e..dcdbd6e3e 100644
--- a/docs/2016-07/index.html
+++ b/docs/2016-07/index.html
@@ -23,8 +23,6 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
In this case the select query was showing 95 results before the update
-
-
" />
@@ -49,10 +47,8 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
In this case the select query was showing 95 results before the update
-
-
"/>
-
+
@@ -149,8 +145,6 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
In this case the select query was showing 95 results before the update
-
-
2016-07-02
diff --git a/docs/2016-08/index.html b/docs/2016-08/index.html
index d98f14cc6..d2a8bb2cd 100644
--- a/docs/2016-08/index.html
+++ b/docs/2016-08/index.html
@@ -20,8 +20,6 @@ Start working on DSpace 5.1 → 5.5 port:
$ git checkout -b 55new 5_x-prod
$ git reset --hard ilri/5_x-prod
$ git rebase -i dspace-5.5
-
-
" />
@@ -43,10 +41,8 @@ Start working on DSpace 5.1 → 5.5 port:
$ git checkout -b 55new 5_x-prod
$ git reset --hard ilri/5_x-prod
$ git rebase -i dspace-5.5
-
-
"/>
-
+
@@ -140,8 +136,6 @@ $ git reset --hard ilri/5_x-prod
$ git rebase -i dspace-5.5
-
-
- Lots of conflicts that don’t make sense (ie, shouldn’t conflict!)
- This file in particular conflicts almost 10 times:
dspace/modules/xmlui-mirage2/src/main/webapp/themes/CGIAR/styles/_style.scss
diff --git a/docs/2016-09/index.html b/docs/2016-09/index.html
index e17bd8591..4be0ccc50 100644
--- a/docs/2016-09/index.html
+++ b/docs/2016-09/index.html
@@ -16,8 +16,6 @@ It looks like we might be able to use OUs now, instead of DCs:
$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=org" -D "admigration1@cgiarad.org" -W "(sAMAccountName=admigration1)"
-
-
" />
@@ -35,10 +33,8 @@ It looks like we might be able to use OUs now, instead of DCs:
$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=org" -D "admigration1@cgiarad.org" -W "(sAMAccountName=admigration1)"
-
-
"/>
-
+
@@ -128,8 +124,6 @@ $ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=or
$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=org" -D "admigration1@cgiarad.org" -W "(sAMAccountName=admigration1)"
-
-
- User who has been migrated to the root vs user still in the hierarchical structure:
diff --git a/docs/2016-10/index.html b/docs/2016-10/index.html
index 12b2bc8f2..4bbae5b19 100644
--- a/docs/2016-10/index.html
+++ b/docs/2016-10/index.html
@@ -20,8 +20,6 @@ I exported a random item’s metadata as CSV, deleted all columns except id
0000-0002-6115-0956||0000-0002-3812-8793||0000-0001-7462-405X
-
-
" />
@@ -43,10 +41,8 @@ I exported a random item’s metadata as CSV, deleted all columns except id
0000-0002-6115-0956||0000-0002-3812-8793||0000-0001-7462-405X
-
-
"/>
-
+
@@ -140,8 +136,6 @@ I exported a random item’s metadata as CSV, deleted all columns except id
0000-0002-6115-0956||0000-0002-3812-8793||0000-0001-7462-405X
-
-
- Hmm, with the
dc.contributor.author
column removed, DSpace doesn’t detect any changes
- With a blank
dc.contributor.author
column, DSpace wants to remove all non-ORCID authors and add the new ORCID authors
diff --git a/docs/2016-11/index.html b/docs/2016-11/index.html
index 9af26dd32..5befdbf47 100644
--- a/docs/2016-11/index.html
+++ b/docs/2016-11/index.html
@@ -12,8 +12,6 @@
Add dc.type to the output options for Atmire’s Listings and Reports module (#286)
-
-
" />
@@ -27,10 +25,8 @@ Add dc.type to the output options for Atmire’s Listings and Reports module
Add dc.type to the output options for Atmire’s Listings and Reports module (#286)
-
-
"/>
-
+
@@ -116,8 +112,6 @@ Add dc.type to the output options for Atmire’s Listings and Reports module
-
-
2016-11-02
diff --git a/docs/2016-12/index.html b/docs/2016-12/index.html
index 37b6256db..51fa6896e 100644
--- a/docs/2016-12/index.html
+++ b/docs/2016-12/index.html
@@ -24,8 +24,6 @@ While looking in the logs for errors, I see tons of warnings about Atmire MQM:
I see thousands of them in the logs for the last few months, so it’s not related to the DSpace 5.5 upgrade
I’ve raised a ticket with Atmire to ask
Another worrying error from dspace.log is:
-
-
" />
@@ -51,10 +49,8 @@ While looking in the logs for errors, I see tons of warnings about Atmire MQM:
I see thousands of them in the logs for the last few months, so it’s not related to the DSpace 5.5 upgrade
I’ve raised a ticket with Atmire to ask
Another worrying error from dspace.log is:
-
-
"/>
-
+
@@ -152,8 +148,6 @@ Another worrying error from dspace.log is:
- Another worrying error from dspace.log is:
-
-
org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.NoSuchMethodError: com.atmire.statistics.generator.DSpaceObjectDatasetGenerator.toDatasetQuery(Lorg/dspace/core/Context;)Lcom/atmire/statistics/content/DatasetQuery;
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:972)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
diff --git a/docs/2017-01/index.html b/docs/2017-01/index.html
index 0d1c31f3e..99a514dc2 100644
--- a/docs/2017-01/index.html
+++ b/docs/2017-01/index.html
@@ -12,8 +12,6 @@
I checked to see if the Solr sharding task that is supposed to run on January 1st had run and saw there was an error
I tested on DSpace Test as well and it doesn’t work there either
I asked on the dspace-tech mailing list because it seems to be broken, and actually now I’m not sure if we’ve ever had the sharding task run successfully over all these years
-
-
" />
@@ -27,10 +25,8 @@ I asked on the dspace-tech mailing list because it seems to be broken, and actua
I checked to see if the Solr sharding task that is supposed to run on January 1st had run and saw there was an error
I tested on DSpace Test as well and it doesn’t work there either
I asked on the dspace-tech mailing list because it seems to be broken, and actually now I’m not sure if we’ve ever had the sharding task run successfully over all these years
-
-
"/>
-
+
@@ -116,8 +112,6 @@ I asked on the dspace-tech mailing list because it seems to be broken, and actua
I asked on the dspace-tech mailing list because it seems to be broken, and actually now I’m not sure if we’ve ever had the sharding task run successfully over all these years
-
-
2017-01-04
diff --git a/docs/2017-02/index.html b/docs/2017-02/index.html
index 0f9dcc9d7..a610ffb11 100644
--- a/docs/2017-02/index.html
+++ b/docs/2017-02/index.html
@@ -26,8 +26,6 @@ DELETE 1
Create issue on GitHub to track the addition of CCAFS Phase II project tags (#301)
Looks like we’ll be using cg.identifier.ccafsprojectpii as the field name
-
-
" />
@@ -55,10 +53,8 @@ DELETE 1
Create issue on GitHub to track the addition of CCAFS Phase II project tags (#301)
Looks like we’ll be using cg.identifier.ccafsprojectpii as the field name
-
-
"/>
-
+
@@ -158,8 +154,6 @@ DELETE 1
- Looks like we’ll be using
cg.identifier.ccafsprojectpii
as the field name
-
-
2017-02-08
diff --git a/docs/2017-03/index.html b/docs/2017-03/index.html
index 74b0357a5..e69ad631a 100644
--- a/docs/2017-03/index.html
+++ b/docs/2017-03/index.html
@@ -28,8 +28,6 @@ Interestingly, it seems DSpace 4.x’s thumbnails were sRGB, but forcing reg
$ identify ~/Desktop/alc_contrastes_desafios.jpg
/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000
-
-
" />
@@ -59,10 +57,8 @@ Interestingly, it seems DSpace 4.x’s thumbnails were sRGB, but forcing reg
$ identify ~/Desktop/alc_contrastes_desafios.jpg
/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000
-
-
"/>
-
+
@@ -164,8 +160,6 @@ $ identify ~/Desktop/alc_contrastes_desafios.jpg
/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000
-
-
- This results in discolored thumbnails when compared to the original PDF, for example sRGB and CMYK:
diff --git a/docs/2017-04/index.html b/docs/2017-04/index.html
index 37de98611..af77e4e1c 100644
--- a/docs/2017-04/index.html
+++ b/docs/2017-04/index.html
@@ -21,8 +21,6 @@ Testing the CMYK patch on a collection with 650 items:
$ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Thumbnail" -v >& /tmp/filter-media-cmyk.txt
-
-
" />
@@ -45,10 +43,8 @@ Testing the CMYK patch on a collection with 650 items:
$ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Thumbnail" -v >& /tmp/filter-media-cmyk.txt
-
-
"/>
-
+
@@ -143,8 +139,6 @@ $ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Th
$ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Thumbnail" -v >& /tmp/filter-media-cmyk.txt
-
-
2017-04-03
diff --git a/docs/2017-05/index.html b/docs/2017-05/index.html
index 5d80efd59..a5f0aa547 100644
--- a/docs/2017-05/index.html
+++ b/docs/2017-05/index.html
@@ -14,7 +14,7 @@
-
+
diff --git a/docs/2017-06/index.html b/docs/2017-06/index.html
index 383c2307d..ba40ce510 100644
--- a/docs/2017-06/index.html
+++ b/docs/2017-06/index.html
@@ -14,7 +14,7 @@
-
+
diff --git a/docs/2017-07/index.html b/docs/2017-07/index.html
index 162904bff..0e6fb5d1b 100644
--- a/docs/2017-07/index.html
+++ b/docs/2017-07/index.html
@@ -18,8 +18,6 @@ Run system updates and reboot DSpace Test
Merge changes for WLE Phase II theme rename (#329)
Looking at extracting the metadata registries from ICARDA’s MEL DSpace database so we can compare fields with CGSpace
We can use PostgreSQL’s extended output format (-x) plus sed to format the output into quasi XML:
-
-
" />
@@ -39,10 +37,8 @@ Run system updates and reboot DSpace Test
Merge changes for WLE Phase II theme rename (#329)
Looking at extracting the metadata registries from ICARDA’s MEL DSpace database so we can compare fields with CGSpace
We can use PostgreSQL’s extended output format (-x) plus sed to format the output into quasi XML:
-
-
"/>
-
+
@@ -134,8 +130,6 @@ We can use PostgreSQL’s extended output format (-x) plus sed to format the
- We can use PostgreSQL’s extended output format (
-x
) plus sed
to format the output into quasi XML:
-
-
$ psql dspacenew -x -c 'select element, qualifier, scope_note from metadatafieldregistry where metadata_schema_id=5 order by element, qualifier;' | sed -r 's:^-\[ RECORD (.*) \]-+$:</dc-type>\n<dc-type>\n<schema>cg</schema>:;s:([^ ]*) +\| (.*): <\1>\2</\1>:;s:^$:</dc-type>:;1s:</dc-type>\n::'
diff --git a/docs/2017-08/index.html b/docs/2017-08/index.html
index 2d076bddf..c363609aa 100644
--- a/docs/2017-08/index.html
+++ b/docs/2017-08/index.html
@@ -28,8 +28,6 @@ Abenet pointed out that the CGIAR Library Historical Archive collection I sent J
This was due to newline characters in the dc.description.abstract column, which caused OpenRefine to choke when exporting the CSV
I exported a new CSV from the collection on DSpace Test and then manually removed the characters in vim using g/^$/d
Then I cleaned up the author authorities and HTML characters in OpenRefine and sent the file back to Abenet
-
-
" />
@@ -59,10 +57,8 @@ Abenet pointed out that the CGIAR Library Historical Archive collection I sent J
This was due to newline characters in the dc.description.abstract column, which caused OpenRefine to choke when exporting the CSV
I exported a new CSV from the collection on DSpace Test and then manually removed the characters in vim using g/^$/d
Then I cleaned up the author authorities and HTML characters in OpenRefine and sent the file back to Abenet
-
-
"/>
-
+
@@ -164,8 +160,6 @@ Then I cleaned up the author authorities and HTML characters in OpenRefine and s
Then I cleaned up the author authorities and HTML characters in OpenRefine and sent the file back to Abenet
-
-
2017-08-02
diff --git a/docs/2017-09/index.html b/docs/2017-09/index.html
index fcecceb4a..a283a13b3 100644
--- a/docs/2017-09/index.html
+++ b/docs/2017-09/index.html
@@ -16,8 +16,6 @@ Linode sent an alert that CGSpace (linode18) was using 261% CPU for the past two
Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account is both in the approvers step as well as the group
-
-
" />
@@ -35,10 +33,8 @@ Linode sent an alert that CGSpace (linode18) was using 261% CPU for the past two
Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account is both in the approvers step as well as the group
-
-
"/>
-
+
@@ -128,8 +124,6 @@ Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account
- Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account is both in the approvers step as well as the group
-
-
2017-09-10
diff --git a/docs/2017-10/index.html b/docs/2017-10/index.html
index 2830088ea..0e48d84dc 100644
--- a/docs/2017-10/index.html
+++ b/docs/2017-10/index.html
@@ -18,8 +18,6 @@ http://hdl.handle.net/10568/78495||http://hdl.handle.net/10568/79336
There appears to be a pattern but I’ll have to look a bit closer and try to clean them up automatically, either in SQL or in OpenRefine
Add Katherine Lutz to the groups for content submission and edit steps of the CGIAR System collections
-
-
" />
@@ -39,10 +37,8 @@ http://hdl.handle.net/10568/78495||http://hdl.handle.net/10568/79336
There appears to be a pattern but I’ll have to look a bit closer and try to clean them up automatically, either in SQL or in OpenRefine
Add Katherine Lutz to the groups for content submission and edit steps of the CGIAR System collections
-
-
"/>
-
+
@@ -134,8 +130,6 @@ Add Katherine Lutz to the groups for content submission and edit steps of the CG
- Add Katherine Lutz to the groups for content submission and edit steps of the CGIAR System collections
-
-
2017-10-02
diff --git a/docs/2017-11/index.html b/docs/2017-11/index.html
index 39a950d10..2e6d33590 100644
--- a/docs/2017-11/index.html
+++ b/docs/2017-11/index.html
@@ -28,8 +28,6 @@ Generate list of authors on CGSpace for Peter to go through and correct:
dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors.csv with csv;
COPY 54701
-
-
" />
@@ -59,10 +57,8 @@ Generate list of authors on CGSpace for Peter to go through and correct:
dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors.csv with csv;
COPY 54701
-
-
"/>
-
+
@@ -164,8 +160,6 @@ COPY 54701
COPY 54701
-
-
- Abenet asked if it would be possible to generate a report of items in Listing and Reports that had “International Fund for Agricultural Development” as the only investor
- I opened a ticket with Atmire to ask if this was possible: https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=540
diff --git a/docs/2017-12/index.html b/docs/2017-12/index.html
index 79031ef2f..8f2d935ca 100644
--- a/docs/2017-12/index.html
+++ b/docs/2017-12/index.html
@@ -13,8 +13,6 @@ Uptime Robot noticed that CGSpace went down
The logs say “Timeout waiting for idle object”
PostgreSQL activity says there are 115 connections currently
The list of connections to XMLUI and REST API for today:
-
-
" />
@@ -29,10 +27,8 @@ Uptime Robot noticed that CGSpace went down
The logs say “Timeout waiting for idle object”
PostgreSQL activity says there are 115 connections currently
The list of connections to XMLUI and REST API for today:
-
-
"/>
-
+
@@ -119,8 +115,6 @@ The list of connections to XMLUI and REST API for today:
- The list of connections to XMLUI and REST API for today:
-
-
# cat /var/log/nginx/rest.log /var/log/nginx/rest.log.1 /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -E "1/Dec/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
763 2.86.122.76
907 207.46.13.94
diff --git a/docs/2018-01/index.html b/docs/2018-01/index.html
index d037fd830..5f9fc5909 100644
--- a/docs/2018-01/index.html
+++ b/docs/2018-01/index.html
@@ -82,8 +82,6 @@ dspace.log.2018-01-02:34
Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let’s Encrypt if it’s just a handful of domains
-
-
" />
@@ -167,10 +165,8 @@ dspace.log.2018-01-02:34
Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let’s Encrypt if it’s just a handful of domains
-
-
"/>
-
+
@@ -326,8 +322,6 @@ dspace.log.2018-01-02:34
Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let’s Encrypt if it’s just a handful of domains
-
-
2018-01-03
diff --git a/docs/2018-02/index.html b/docs/2018-02/index.html
index fcd5cc0b1..78d887994 100644
--- a/docs/2018-02/index.html
+++ b/docs/2018-02/index.html
@@ -13,8 +13,6 @@ Peter gave feedback on the dc.rights proof of concept that I had sent him last w
We don’t need to distinguish between internal and external works, so that makes it just a simple list
Yesterday I figured out how to monitor DSpace sessions using JMX
I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu’s munin-plugins-java package and used the stuff I discovered about JMX in 2018-01
-
-
" />
@@ -29,10 +27,8 @@ Peter gave feedback on the dc.rights proof of concept that I had sent him last w
We don’t need to distinguish between internal and external works, so that makes it just a simple list
Yesterday I figured out how to monitor DSpace sessions using JMX
I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu’s munin-plugins-java package and used the stuff I discovered about JMX in 2018-01
-
-
"/>
-
+
@@ -119,8 +115,6 @@ I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu’s munin-pl
- I copied the logic in the
jmx_tomcat_dbpools
provided by Ubuntu’s munin-plugins-java
package and used the stuff I discovered about JMX in 2018-01
-
-
diff --git a/docs/2018-03/index.html b/docs/2018-03/index.html
index fdcc4e6a4..29a0de3f2 100644
--- a/docs/2018-03/index.html
+++ b/docs/2018-03/index.html
@@ -10,8 +10,6 @@
Export a CSV of the IITA community metadata for Martin Mueller
-
-
" />
@@ -23,10 +21,8 @@ Export a CSV of the IITA community metadata for Martin Mueller
Export a CSV of the IITA community metadata for Martin Mueller
-
-
"/>
-
+
@@ -110,8 +106,6 @@ Export a CSV of the IITA community metadata for Martin Mueller
- Export a CSV of the IITA community metadata for Martin Mueller
-
-
2018-03-06
diff --git a/docs/2018-04/index.html b/docs/2018-04/index.html
index 643afff0e..e67663a5e 100644
--- a/docs/2018-04/index.html
+++ b/docs/2018-04/index.html
@@ -11,8 +11,6 @@
I tried to test something on DSpace Test but noticed that it’s down since god knows when
Catalina logs at least show some memory errors yesterday:
-
-
" />
@@ -25,10 +23,8 @@ Catalina logs at least show some memory errors yesterday:
I tried to test something on DSpace Test but noticed that it’s down since god knows when
Catalina logs at least show some memory errors yesterday:
-
-
"/>
-
+
@@ -113,8 +109,6 @@ Catalina logs at least show some memory errors yesterday:
- Catalina logs at least show some memory errors yesterday:
-
-
Mar 31, 2018 10:26:42 PM org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor run
SEVERE: Unexpected death of background thread ContainerBackgroundProcessor[StandardEngine[Catalina]]
java.lang.OutOfMemoryError: Java heap space
diff --git a/docs/2018-05/index.html b/docs/2018-05/index.html
index 93c3cfee0..cb7c57c53 100644
--- a/docs/2018-05/index.html
+++ b/docs/2018-05/index.html
@@ -17,8 +17,6 @@ http://localhost:3000/solr/statistics/update?stream.body=%3Ccommit/%3E
Then I reduced the JVM heap size from 6144 back to 5120m
Also, I switched it to use OpenJDK instead of Oracle Java, as well as re-worked the Ansible infrastructure scripts to support hosts choosing which distribution they want to use
-
-
" />
@@ -37,10 +35,8 @@ http://localhost:3000/solr/statistics/update?stream.body=%3Ccommit/%3E
Then I reduced the JVM heap size from 6144 back to 5120m
Also, I switched it to use OpenJDK instead of Oracle Java, as well as re-worked the Ansible infrastructure scripts to support hosts choosing which distribution they want to use
-
-
"/>
-
+
@@ -131,8 +127,6 @@ Also, I switched it to use OpenJDK instead of Oracle Java, as well as re-worked
Also, I switched it to use OpenJDK instead of Oracle Java, as well as re-worked the Ansible infrastructure scripts to support hosts choosing which distribution they want to use
-
-
2018-05-02
diff --git a/docs/2018-06/index.html b/docs/2018-06/index.html
index 0b2e780ef..a0f2b9ddc 100644
--- a/docs/2018-06/index.html
+++ b/docs/2018-06/index.html
@@ -31,8 +31,6 @@ $ time schedtool -D -e ionice -c2 -n7 nice -n19 [dspace]/bin/dspace index-discov
real 74m42.646s
user 8m5.056s
sys 2m7.289s
-
-
" />
@@ -65,10 +63,8 @@ $ time schedtool -D -e ionice -c2 -n7 nice -n19 [dspace]/bin/dspace index-discov
real 74m42.646s
user 8m5.056s
sys 2m7.289s
-
-
"/>
-
+
@@ -173,8 +169,6 @@ user 8m5.056s
sys 2m7.289s
-
-
2018-06-06
diff --git a/docs/2018-07/index.html b/docs/2018-07/index.html
index 2f1a4e3da..899c74866 100644
--- a/docs/2018-07/index.html
+++ b/docs/2018-07/index.html
@@ -20,8 +20,6 @@ During the mvn package stage on the 5.8 branch I kept getting issues with java r
There is insufficient memory for the Java Runtime Environment to continue.
-
-
" />
@@ -43,10 +41,8 @@ During the mvn package stage on the 5.8 branch I kept getting issues with java r
There is insufficient memory for the Java Runtime Environment to continue.
-
-
"/>
-
+
@@ -140,8 +136,6 @@ There is insufficient memory for the Java Runtime Environment to continue.
There is insufficient memory for the Java Runtime Environment to continue.
-
-
- As the machine only has 8GB of RAM, I reduced the Tomcat memory heap from 5120m to 4096m so I could try to allocate more to the build process:
diff --git a/docs/2018-08/index.html b/docs/2018-08/index.html
index 5dad194df..08a52459d 100644
--- a/docs/2018-08/index.html
+++ b/docs/2018-08/index.html
@@ -24,8 +24,6 @@ I’m not sure why Tomcat didn’t crash with an OutOfMemoryError&hellip
Anyways, perhaps I should increase the JVM heap from 5120m to 6144m like we did a few months ago when we tried to run the whole CGSpace Solr core
The server only has 8GB of RAM so we’ll eventually need to upgrade to a larger one because we’ll start starving the OS, PostgreSQL, and command line batch processes
I ran all system updates on DSpace Test and rebooted it
-
-
" />
@@ -51,10 +49,8 @@ I’m not sure why Tomcat didn’t crash with an OutOfMemoryError&hellip
Anyways, perhaps I should increase the JVM heap from 5120m to 6144m like we did a few months ago when we tried to run the whole CGSpace Solr core
The server only has 8GB of RAM so we’ll eventually need to upgrade to a larger one because we’ll start starving the OS, PostgreSQL, and command line batch processes
I ran all system updates on DSpace Test and rebooted it
-
-
"/>
-
+
@@ -152,8 +148,6 @@ I ran all system updates on DSpace Test and rebooted it
I ran all system updates on DSpace Test and rebooted it
-
-
- I started looking over the latest round of IITA batch records from Sisay on DSpace Test: IITA July_30
diff --git a/docs/2018-09/index.html b/docs/2018-09/index.html
index 4622aaae2..504bc5a68 100644
--- a/docs/2018-09/index.html
+++ b/docs/2018-09/index.html
@@ -13,8 +13,6 @@ New PostgreSQL JDBC driver version 42.2.5
I’ll update the DSpace role in our Ansible infrastructure playbooks and run the updated playbooks on CGSpace and DSpace Test
Also, I’ll re-run the postgresql tasks because the custom PostgreSQL variables are dynamic according to the system’s RAM, and we never re-ran them after migrating to larger Linodes last month
I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I’m getting those autowire errors in Tomcat 8.5.30 again:
-
-
" />
@@ -29,10 +27,8 @@ New PostgreSQL JDBC driver version 42.2.5
I’ll update the DSpace role in our Ansible infrastructure playbooks and run the updated playbooks on CGSpace and DSpace Test
Also, I’ll re-run the postgresql tasks because the custom PostgreSQL variables are dynamic according to the system’s RAM, and we never re-ran them after migrating to larger Linodes last month
I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I’m getting those autowire errors in Tomcat 8.5.30 again:
-
-
"/>
-
+
@@ -119,8 +115,6 @@ I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I
- I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I’m getting those autowire errors in Tomcat 8.5.30 again:
-
-
02-Sep-2018 11:18:52.678 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [org.dspace.servicemanager.servlet.DSpaceKernelServletContextListener]
java.lang.RuntimeException: Failure during filter init: Failed to startup the DSpace Service Manager: failure starting up spring service manager: Error creating bean with name 'conversionService' defined in file [/home/dspacetest.cgiar.org/config/spring/xmlui/spring-dspace-addon-cua-services.xml]: Cannot create inner bean 'com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter#4c5d5a2' of type [com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter] while setting bean property 'converters' with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter#4c5d5a2': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$FilterConverter com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter.filterConverter; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No matching bean of type [com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$FilterConverter] found for dependency: expected at least 1 bean which qualifies as autowire candidate for this dependency. Dependency annotations: {@org.springframework.beans.factory.annotation.Autowired(required=true)}
at org.dspace.servicemanager.servlet.DSpaceKernelServletContextListener.contextInitialized(DSpaceKernelServletContextListener.java:92)
diff --git a/docs/2018-10/index.html b/docs/2018-10/index.html
index 678b2423d..eec0cbda7 100644
--- a/docs/2018-10/index.html
+++ b/docs/2018-10/index.html
@@ -11,8 +11,6 @@
Phil Thornton got an ORCID identifier so we need to add it to the list on CGSpace and tag his existing items
I created a GitHub issue to track this #389, because I’m super busy in Nairobi right now
-
-
" />
@@ -25,10 +23,8 @@ I created a GitHub issue to track this #389, because I’m super busy in Nai
Phil Thornton got an ORCID identifier so we need to add it to the list on CGSpace and tag his existing items
I created a GitHub issue to track this #389, because I’m super busy in Nairobi right now
-
-
"/>
-
+
@@ -113,8 +109,6 @@ I created a GitHub issue to track this #389, because I’m super busy in Nai
I created a GitHub issue to track this #389, because I’m super busy in Nairobi right now
-
-
2018-10-03
diff --git a/docs/2018-11/index.html b/docs/2018-11/index.html
index 41997bdb0..8706059d6 100644
--- a/docs/2018-11/index.html
+++ b/docs/2018-11/index.html
@@ -18,12 +18,10 @@ Send a note about my dspace-statistics-api to the dspace-tech mailing list
Linode has been sending mails a few times a day recently that CGSpace (linode18) has had high CPU usage
Today these are the top 10 IPs:
-
-
" />
-
+
@@ -39,10 +37,8 @@ Send a note about my dspace-statistics-api to the dspace-tech mailing list
Linode has been sending mails a few times a day recently that CGSpace (linode18) has had high CPU usage
Today these are the top 10 IPs:
-
-
"/>
-
+
@@ -52,9 +48,9 @@ Today these are the top 10 IPs:
"@type": "BlogPosting",
"headline": "November, 2018",
"url": "https://alanorth.github.io/cgspace-notes/2018-11/",
- "wordCount": "1244",
+ "wordCount": "1310",
"datePublished": "2018-11-01T16:41:30+02:00",
- "dateModified": "2018-11-06T18:03:44+02:00",
+ "dateModified": "2018-11-07T19:20:25+02:00",
"author": {
"@type": "Person",
"name": "Alan Orth"
@@ -134,8 +130,6 @@ Today these are the top 10 IPs:
- Today these are the top 10 IPs:
-
-
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "03/Nov/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
1300 66.249.64.63
1384 35.237.175.180
@@ -420,6 +414,14 @@ Today these are the top 10 IPs:
Tag version 0.7.0 of the dspace-statistics-api
+2018-11-08
+
+
+
diff --git a/docs/404.html b/docs/404.html
index 303ed1f8f..0ad28f0bf 100644
--- a/docs/404.html
+++ b/docs/404.html
@@ -13,7 +13,7 @@
-
+
diff --git a/docs/categories/index.html b/docs/categories/index.html
index f0adf6a89..d6bb6de30 100644
--- a/docs/categories/index.html
+++ b/docs/categories/index.html
@@ -13,7 +13,7 @@
-
+
@@ -109,8 +109,6 @@
Linode has been sending mails a few times a day recently that CGSpace (linode18) has had high CPU usage
Today these are the top 10 IPs:
-
-
Read more →
@@ -134,8 +132,6 @@
Phil Thornton got an ORCID identifier so we need to add it to the list on CGSpace and tag his existing items
I created a GitHub issue to track this #389, because I’m super busy in Nairobi right now
-
-
Read more →
@@ -161,8 +157,6 @@
Also, I’ll re-run the postgresql
tasks because the custom PostgreSQL variables are dynamic according to the system’s RAM, and we never re-ran them after migrating to larger Linodes last month
I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I’m getting those autowire errors in Tomcat 8.5.30 again:
-
-
Read more →
@@ -199,8 +193,6 @@
The server only has 8GB of RAM so we’ll eventually need to upgrade to a larger one because we’ll start starving the OS, PostgreSQL, and command line batch processes
I ran all system updates on DSpace Test and rebooted it
-
-
Read more →
@@ -233,8 +225,6 @@
There is insufficient memory for the Java Runtime Environment to continue.
-
-
Read more →
@@ -278,8 +268,6 @@ real 74m42.646s
user 8m5.056s
sys 2m7.289s
-
-
Read more →
@@ -309,8 +297,6 @@ sys 2m7.289s
Then I reduced the JVM heap size from 6144 back to 5120m
Also, I switched it to use OpenJDK instead of Oracle Java, as well as re-worked the Ansible infrastructure scripts to support hosts choosing which distribution they want to use
-
-
Read more →
@@ -334,8 +320,6 @@ sys 2m7.289s
I tried to test something on DSpace Test but noticed that it’s down since god knows when
Catalina logs at least show some memory errors yesterday:
-
-
Read more →
@@ -358,8 +342,6 @@ sys 2m7.289s
- Export a CSV of the IITA community metadata for Martin Mueller
-
-
Read more →
@@ -385,8 +367,6 @@ sys 2m7.289s
Yesterday I figured out how to monitor DSpace sessions using JMX
I copied the logic in the jmx_tomcat_dbpools
provided by Ubuntu’s munin-plugins-java
package and used the stuff I discovered about JMX in 2018-01
-
-
Read more →
diff --git a/docs/categories/notes/index.html b/docs/categories/notes/index.html
index b0585df82..35ab84955 100644
--- a/docs/categories/notes/index.html
+++ b/docs/categories/notes/index.html
@@ -14,7 +14,7 @@
-
+
@@ -84,8 +84,6 @@
Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called CGIAR System Organization.
-
-
Read more →
diff --git a/docs/categories/notes/index.xml b/docs/categories/notes/index.xml
index 88149ec10..15ff6958d 100644
--- a/docs/categories/notes/index.xml
+++ b/docs/categories/notes/index.xml
@@ -17,9 +17,7 @@
Mon, 18 Sep 2017 16:38:35 +0300
https://alanorth.github.io/cgspace-notes/cgiar-library-migration/
- <p>Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called <em>CGIAR System Organization</em>.</p>
-
-<p></p>
+ <p>Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called <em>CGIAR System Organization</em>.</p>
diff --git a/docs/categories/page/2/index.html b/docs/categories/page/2/index.html
index 5e57bf1be..acd4ad6e9 100644
--- a/docs/categories/page/2/index.html
+++ b/docs/categories/page/2/index.html
@@ -14,7 +14,7 @@
-
+
@@ -175,8 +175,6 @@ dspace.log.2018-01-02:34
- Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let’s Encrypt if it’s just a handful of domains
-
-
Read more →
@@ -202,8 +200,6 @@ dspace.log.2018-01-02:34
PostgreSQL activity says there are 115 connections currently
The list of connections to XMLUI and REST API for today:
-
-
Read more →
@@ -244,8 +240,6 @@ dspace.log.2018-01-02:34
dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors.csv with csv;
COPY 54701
-
-
Read more →
@@ -276,8 +270,6 @@ COPY 54701
There appears to be a pattern but I’ll have to look a bit closer and try to clean them up automatically, either in SQL or in OpenRefine
Add Katherine Lutz to the groups for content submission and edit steps of the CGIAR System collections
-
-
Read more →
@@ -296,8 +288,6 @@ COPY 54701
Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called CGIAR System Organization.
-
-
Read more →
@@ -326,8 +316,6 @@ COPY 54701
- Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account is both in the approvers step as well as the group
-
-
Read more →
@@ -368,8 +356,6 @@ COPY 54701
I exported a new CSV from the collection on DSpace Test and then manually removed the characters in vim using g/^$/d
Then I cleaned up the author authorities and HTML characters in OpenRefine and sent the file back to Abenet
-
-
Read more →
@@ -400,8 +386,6 @@ COPY 54701
Looking at extracting the metadata registries from ICARDA’s MEL DSpace database so we can compare fields with CGSpace
We can use PostgreSQL’s extended output format (-x
) plus sed
to format the output into quasi XML:
-
-
Read more →
diff --git a/docs/categories/page/3/index.html b/docs/categories/page/3/index.html
index 0b42823c2..8b0fa09d7 100644
--- a/docs/categories/page/3/index.html
+++ b/docs/categories/page/3/index.html
@@ -14,7 +14,7 @@
-
+
@@ -114,8 +114,6 @@
$ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Thumbnail" -v >& /tmp/filter-media-cmyk.txt
-
-
Read more →
@@ -156,8 +154,6 @@
$ identify ~/Desktop/alc_contrastes_desafios.jpg
/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000
-
-
Read more →
@@ -196,8 +192,6 @@ DELETE 1
Create issue on GitHub to track the addition of CCAFS Phase II project tags (#301)
Looks like we’ll be using cg.identifier.ccafsprojectpii
as the field name
-
-
Read more →
@@ -222,8 +216,6 @@ DELETE 1
I tested on DSpace Test as well and it doesn’t work there either
I asked on the dspace-tech mailing list because it seems to be broken, and actually now I’m not sure if we’ve ever had the sharding task run successfully over all these years
-
-
Read more →
@@ -260,8 +252,6 @@ DELETE 1
I’ve raised a ticket with Atmire to ask
Another worrying error from dspace.log is:
-
-
Read more →
@@ -286,8 +276,6 @@ DELETE 1
-
-
Read more →
@@ -320,8 +308,6 @@ DELETE 1
0000-0002-6115-0956||0000-0002-3812-8793||0000-0001-7462-405X
-
-
Read more →
@@ -350,8 +336,6 @@ DELETE 1
$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=org" -D "admigration1@cgiarad.org" -W "(sAMAccountName=admigration1)"
-
-
Read more →
@@ -384,8 +368,6 @@ DELETE 1
$ git reset --hard ilri/5_x-prod
$ git rebase -i dspace-5.5
-
-
Read more →
@@ -421,8 +403,6 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
- In this case the select query was showing 95 results before the update
-
-
Read more →
diff --git a/docs/categories/page/4/index.html b/docs/categories/page/4/index.html
index b1f8d25a0..09828816f 100644
--- a/docs/categories/page/4/index.html
+++ b/docs/categories/page/4/index.html
@@ -14,7 +14,7 @@
-
+
@@ -108,8 +108,6 @@
You can see the others by using the OAI ListSets
verb: http://ebrary.ifpri.org/oai/oai.php?verb=ListSets
Working on second phase of metadata migration, looks like this will work for moving CPWF-specific data in dc.identifier.fund
to cg.identifier.cpwfproject
and then the rest to dc.description.sponsorship
-
-
Read more →
@@ -138,8 +136,6 @@
# awk '{print $1}' /var/log/nginx/rest.log | uniq | wc -l
3168
-
-
Read more →
@@ -166,8 +162,6 @@
This will save us a few gigs of backup space we’re paying for on S3
Also, I noticed the checker
log has some errors we should pay attention to:
-
-
Read more →
@@ -192,8 +186,6 @@
For some reason we still have the index-lucene-update
cron job active on CGSpace, but I’m pretty sure we don’t need it as of the latest few versions of Atmire’s Listings and Reports module
Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Java JDK 1.7 to match environment on CGSpace server
-
-
Read more →
@@ -225,8 +217,6 @@
Not only are there 49,000 countries, we have some blanks (25)…
Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE”
-
-
Read more →
@@ -251,8 +241,6 @@
I realized it is only necessary to clear the Cocoon cache after moving collections—rather than reindexing—as no metadata has changed, and therefore no search or browse indexes need to be updated.
Update GitHub wiki for documentation of maintenance tasks.
-
-
Read more →
@@ -282,8 +270,6 @@
-rw-rw-r-- 1 tomcat7 tomcat7 387K Nov 18 23:59 dspace.log.2015-11-18.lzo
-rw-rw-r-- 1 tomcat7 tomcat7 169K Nov 18 23:59 dspace.log.2015-11-18.xz
-
-
Read more →
@@ -312,8 +298,6 @@
$ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
78
-
-
Read more →
diff --git a/docs/cgiar-library-migration/index.html b/docs/cgiar-library-migration/index.html
index 60ebec85c..60aae80f0 100644
--- a/docs/cgiar-library-migration/index.html
+++ b/docs/cgiar-library-migration/index.html
@@ -14,7 +14,7 @@
-
+
@@ -95,8 +95,6 @@
Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called CGIAR System Organization.
-
-
Pre-migration Technical TODOs
Things that need to happen before the migration:
diff --git a/docs/index.html b/docs/index.html
index 23bec0f5d..154e1cc35 100644
--- a/docs/index.html
+++ b/docs/index.html
@@ -14,7 +14,7 @@
-
+
@@ -111,8 +111,6 @@
Linode has been sending mails a few times a day recently that CGSpace (linode18) has had high CPU usage
Today these are the top 10 IPs:
-
-
Read more →
@@ -136,8 +134,6 @@
Phil Thornton got an ORCID identifier so we need to add it to the list on CGSpace and tag his existing items
I created a GitHub issue to track this #389, because I’m super busy in Nairobi right now
-
-
Read more →
@@ -163,8 +159,6 @@
Also, I’ll re-run the postgresql
tasks because the custom PostgreSQL variables are dynamic according to the system’s RAM, and we never re-ran them after migrating to larger Linodes last month
I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I’m getting those autowire errors in Tomcat 8.5.30 again:
-
-
Read more →
@@ -201,8 +195,6 @@
The server only has 8GB of RAM so we’ll eventually need to upgrade to a larger one because we’ll start starving the OS, PostgreSQL, and command line batch processes
I ran all system updates on DSpace Test and rebooted it
-
-
Read more →
@@ -235,8 +227,6 @@
There is insufficient memory for the Java Runtime Environment to continue.
-
-
Read more →
@@ -280,8 +270,6 @@ real 74m42.646s
user 8m5.056s
sys 2m7.289s
-
-
Read more →
@@ -311,8 +299,6 @@ sys 2m7.289s
Then I reduced the JVM heap size from 6144 back to 5120m
Also, I switched it to use OpenJDK instead of Oracle Java, as well as re-worked the Ansible infrastructure scripts to support hosts choosing which distribution they want to use
-
-
Read more →
@@ -336,8 +322,6 @@ sys 2m7.289s
I tried to test something on DSpace Test but noticed that it’s down since god knows when
Catalina logs at least show some memory errors yesterday:
-
-
Read more →
@@ -360,8 +344,6 @@ sys 2m7.289s
- Export a CSV of the IITA community metadata for Martin Mueller
-
-
Read more →
@@ -387,8 +369,6 @@ sys 2m7.289s
Yesterday I figured out how to monitor DSpace sessions using JMX
I copied the logic in the jmx_tomcat_dbpools
provided by Ubuntu’s munin-plugins-java
package and used the stuff I discovered about JMX in 2018-01
-
-
Read more →
diff --git a/docs/index.xml b/docs/index.xml
index 6ca38da91..698443a04 100644
--- a/docs/index.xml
+++ b/docs/index.xml
@@ -29,9 +29,7 @@
<ul>
<li>Linode has been sending mails a few times a day recently that CGSpace (linode18) has had high CPU usage</li>
<li>Today these are the top 10 IPs:</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -45,9 +43,7 @@
<ul>
<li>Phil Thornton got an ORCID identifier so we need to add it to the list on CGSpace and tag his existing items</li>
<li>I created a GitHub issue to track this <a href="https://github.com/ilri/DSpace/issues/389">#389</a>, because I’m super busy in Nairobi right now</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -63,9 +59,7 @@
<li>I’ll update the DSpace role in our <a href="https://github.com/ilri/rmg-ansible-public">Ansible infrastructure playbooks</a> and run the updated playbooks on CGSpace and DSpace Test</li>
<li>Also, I’ll re-run the <code>postgresql</code> tasks because the custom PostgreSQL variables are dynamic according to the system’s RAM, and we never re-ran them after migrating to larger Linodes last month</li>
<li>I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I’m getting those autowire errors in Tomcat 8.5.30 again:</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -92,9 +86,7 @@
<li>Anyways, perhaps I should increase the JVM heap from 5120m to 6144m like we did a few months ago when we tried to run the whole CGSpace Solr core</li>
<li>The server only has 8GB of RAM so we’ll eventually need to upgrade to a larger one because we’ll start starving the OS, PostgreSQL, and command line batch processes</li>
<li>I ran all system updates on DSpace Test and rebooted it</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -117,9 +109,7 @@
</ul>
<pre><code>There is insufficient memory for the Java Runtime Environment to continue.
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -153,9 +143,7 @@
real 74m42.646s
user 8m5.056s
sys 2m7.289s
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -175,9 +163,7 @@ sys 2m7.289s
</ul></li>
<li>Then I reduced the JVM heap size from 6144 back to 5120m</li>
<li>Also, I switched it to use OpenJDK instead of Oracle Java, as well as re-worked the <a href="https://github.com/ilri/rmg-ansible-public">Ansible infrastructure scripts</a> to support hosts choosing which distribution they want to use</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -191,9 +177,7 @@ sys 2m7.289s
<ul>
<li>I tried to test something on DSpace Test but noticed that it’s down since god knows when</li>
<li>Catalina logs at least show some memory errors yesterday:</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -206,9 +190,7 @@ sys 2m7.289s
<ul>
<li>Export a CSV of the IITA community metadata for Martin Mueller</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -224,9 +206,7 @@ sys 2m7.289s
<li>We don’t need to distinguish between internal and external works, so that makes it just a simple list</li>
<li>Yesterday I figured out how to monitor DSpace sessions using JMX</li>
<li>I copied the logic in the <code>jmx_tomcat_dbpools</code> provided by Ubuntu’s <code>munin-plugins-java</code> package and used the stuff I discovered about JMX <a href="https://alanorth.github.io/cgspace-notes/cgspace-notes/2018-01/">in 2018-01</a></li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -311,9 +291,7 @@ dspace.log.2018-01-02:34
<ul>
<li>Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let’s Encrypt if it’s just a handful of domains</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -329,9 +307,7 @@ dspace.log.2018-01-02:34
<li>The logs say “Timeout waiting for idle object”</li>
<li>PostgreSQL activity says there are 115 connections currently</li>
<li>The list of connections to XMLUI and REST API for today:</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -362,9 +338,7 @@ dspace.log.2018-01-02:34
<pre><code>dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors.csv with csv;
COPY 54701
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -385,9 +359,7 @@ COPY 54701
<ul>
<li>There appears to be a pattern but I’ll have to look a bit closer and try to clean them up automatically, either in SQL or in OpenRefine</li>
<li>Add Katherine Lutz to the groups for content submission and edit steps of the CGIAR System collections</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -396,9 +368,7 @@ COPY 54701
Mon, 18 Sep 2017 16:38:35 +0300
https://alanorth.github.io/cgspace-notes/cgiar-library-migration/
- <p>Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called <em>CGIAR System Organization</em>.</p>
-
-<p></p>
+ <p>Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called <em>CGIAR System Organization</em>.</p>
-
@@ -417,9 +387,7 @@ COPY 54701
<ul>
<li>Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account is both in the approvers step as well as the group</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -450,9 +418,7 @@ COPY 54701
<li>This was due to newline characters in the <code>dc.description.abstract</code> column, which caused OpenRefine to choke when exporting the CSV</li>
<li>I exported a new CSV from the collection on DSpace Test and then manually removed the characters in vim using <code>g/^$/d</code></li>
<li>Then I cleaned up the author authorities and HTML characters in OpenRefine and sent the file back to Abenet</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -473,9 +439,7 @@ COPY 54701
<li>Merge changes for WLE Phase II theme rename (<a href="https://github.com/ilri/DSpace/pull/329">#329</a>)</li>
<li>Looking at extracting the metadata registries from ICARDA’s MEL DSpace database so we can compare fields with CGSpace</li>
<li>We can use PostgreSQL’s extended output format (<code>-x</code>) plus <code>sed</code> to format the output into quasi XML:</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -517,9 +481,7 @@ COPY 54701
</ul>
<pre><code>$ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Thumbnail" -v >& /tmp/filter-media-cmyk.txt
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -550,9 +512,7 @@ COPY 54701
<pre><code>$ identify ~/Desktop/alc_contrastes_desafios.jpg
/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -581,9 +541,7 @@ DELETE 1
<ul>
<li>Create issue on GitHub to track the addition of CCAFS Phase II project tags (<a href="https://github.com/ilri/DSpace/issues/301">#301</a>)</li>
<li>Looks like we’ll be using <code>cg.identifier.ccafsprojectpii</code> as the field name</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -598,9 +556,7 @@ DELETE 1
<li>I checked to see if the Solr sharding task that is supposed to run on January 1st had run and saw there was an error</li>
<li>I tested on DSpace Test as well and it doesn’t work there either</li>
<li>I asked on the dspace-tech mailing list because it seems to be broken, and actually now I’m not sure if we’ve ever had the sharding task run successfully over all these years</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -627,9 +583,7 @@ DELETE 1
<li>I see thousands of them in the logs for the last few months, so it’s not related to the DSpace 5.5 upgrade</li>
<li>I’ve raised a ticket with Atmire to ask</li>
<li>Another worrying error from dspace.log is:</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -644,9 +598,7 @@ DELETE 1
<li>Add <code>dc.type</code> to the output options for Atmire’s Listings and Reports module (<a href="https://github.com/ilri/DSpace/pull/286">#286</a>)</li>
</ul>
-<p><img src="https://alanorth.github.io/cgspace-notes/cgspace-notes/2016/11/listings-and-reports.png" alt="Listings and Reports with output type" /></p>
-
-<p></p>
+<p><img src="https://alanorth.github.io/cgspace-notes/cgspace-notes/2016/11/listings-and-reports.png" alt="Listings and Reports with output type" /></p>
-
@@ -669,9 +621,7 @@ DELETE 1
</ul>
<pre><code>0000-0002-6115-0956||0000-0002-3812-8793||0000-0001-7462-405X
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -690,9 +640,7 @@ DELETE 1
</ul>
<pre><code>$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=org" -D "admigration1@cgiarad.org" -W "(sAMAccountName=admigration1)"
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -715,9 +663,7 @@ DELETE 1
<pre><code>$ git checkout -b 55new 5_x-prod
$ git reset --hard ilri/5_x-prod
$ git rebase -i dspace-5.5
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -743,9 +689,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
<ul>
<li>In this case the select query was showing 95 results before the update</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -763,9 +707,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
<li>This is their publications set: <a href="http://ebrary.ifpri.org/oai/oai.php?verb=ListRecords&from=2016-01-01&set=p15738coll2&metadataPrefix=oai_dc">http://ebrary.ifpri.org/oai/oai.php?verb=ListRecords&from=2016-01-01&set=p15738coll2&metadataPrefix=oai_dc</a></li>
<li>You can see the others by using the OAI <code>ListSets</code> verb: <a href="http://ebrary.ifpri.org/oai/oai.php?verb=ListSets">http://ebrary.ifpri.org/oai/oai.php?verb=ListSets</a></li>
<li>Working on second phase of metadata migration, looks like this will work for moving CPWF-specific data in <code>dc.identifier.fund</code> to <code>cg.identifier.cpwfproject</code> and then the rest to <code>dc.description.sponsorship</code></li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -784,9 +726,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
<pre><code># awk '{print $1}' /var/log/nginx/rest.log | uniq | wc -l
3168
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -803,9 +743,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
<li>After running DSpace for over five years I’ve never needed to look in any other log file than dspace.log, leave alone one from last year!</li>
<li>This will save us a few gigs of backup space we’re paying for on S3</li>
<li>Also, I noticed the <code>checker</code> log has some errors we should pay attention to:</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -820,9 +758,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
<li>Looking at issues with author authorities on CGSpace</li>
<li>For some reason we still have the <code>index-lucene-update</code> cron job active on CGSpace, but I’m pretty sure we don’t need it as of the latest few versions of Atmire’s Listings and Reports module</li>
<li>Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Java JDK 1.7 to match environment on CGSpace server</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -844,9 +780,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
<ul>
<li>Not only are there 49,000 countries, we have some blanks (25)…</li>
<li>Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE”</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -861,9 +795,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
<li>Move ILRI collection <code>10568/12503</code> from <code>10568/27869</code> to <code>10568/27629</code> using the <a href="https://gist.github.com/alanorth/392c4660e8b022d99dfa">move_collections.sh</a> script I wrote last year.</li>
<li>I realized it is only necessary to clear the Cocoon cache after moving collections—rather than reindexing—as no metadata has changed, and therefore no search or browse indexes need to be updated.</li>
<li>Update GitHub wiki for documentation of <a href="https://github.com/ilri/DSpace/wiki/Maintenance-Tasks">maintenance tasks</a>.</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -883,9 +815,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
-rw-rw-r-- 1 tomcat7 tomcat7 2.0M Nov 18 23:59 dspace.log.2015-11-18
-rw-rw-r-- 1 tomcat7 tomcat7 387K Nov 18 23:59 dspace.log.2015-11-18.lzo
-rw-rw-r-- 1 tomcat7 tomcat7 169K Nov 18 23:59 dspace.log.2015-11-18.xz
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -904,9 +834,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
<pre><code>$ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
78
-</code></pre>
-
-<p></p>
+</code></pre>
diff --git a/docs/page/2/index.html b/docs/page/2/index.html
index a9f723757..3ab66ce9a 100644
--- a/docs/page/2/index.html
+++ b/docs/page/2/index.html
@@ -14,7 +14,7 @@
-
+
@@ -175,8 +175,6 @@ dspace.log.2018-01-02:34
- Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let’s Encrypt if it’s just a handful of domains
-
-
Read more →
@@ -202,8 +200,6 @@ dspace.log.2018-01-02:34
PostgreSQL activity says there are 115 connections currently
The list of connections to XMLUI and REST API for today:
-
-
Read more →
@@ -244,8 +240,6 @@ dspace.log.2018-01-02:34
dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors.csv with csv;
COPY 54701
-
-
Read more →
@@ -276,8 +270,6 @@ COPY 54701
There appears to be a pattern but I’ll have to look a bit closer and try to clean them up automatically, either in SQL or in OpenRefine
Add Katherine Lutz to the groups for content submission and edit steps of the CGIAR System collections
-
-
Read more →
@@ -296,8 +288,6 @@ COPY 54701
Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called CGIAR System Organization.
-
-
Read more →
@@ -326,8 +316,6 @@ COPY 54701
- Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account is both in the approvers step as well as the group
-
-
Read more →
@@ -368,8 +356,6 @@ COPY 54701
I exported a new CSV from the collection on DSpace Test and then manually removed the characters in vim using g/^$/d
Then I cleaned up the author authorities and HTML characters in OpenRefine and sent the file back to Abenet
-
-
Read more →
@@ -400,8 +386,6 @@ COPY 54701
Looking at extracting the metadata registries from ICARDA’s MEL DSpace database so we can compare fields with CGSpace
We can use PostgreSQL’s extended output format (-x
) plus sed
to format the output into quasi XML:
-
-
Read more →
diff --git a/docs/page/3/index.html b/docs/page/3/index.html
index 7b5b2da3b..e32f297e9 100644
--- a/docs/page/3/index.html
+++ b/docs/page/3/index.html
@@ -14,7 +14,7 @@
-
+
@@ -114,8 +114,6 @@
$ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Thumbnail" -v >& /tmp/filter-media-cmyk.txt
-
-
Read more →
@@ -156,8 +154,6 @@
$ identify ~/Desktop/alc_contrastes_desafios.jpg
/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000
-
-
Read more →
@@ -196,8 +192,6 @@ DELETE 1
Create issue on GitHub to track the addition of CCAFS Phase II project tags (#301)
Looks like we’ll be using cg.identifier.ccafsprojectpii
as the field name
-
-
Read more →
@@ -222,8 +216,6 @@ DELETE 1
I tested on DSpace Test as well and it doesn’t work there either
I asked on the dspace-tech mailing list because it seems to be broken, and actually now I’m not sure if we’ve ever had the sharding task run successfully over all these years
-
-
Read more →
@@ -260,8 +252,6 @@ DELETE 1
I’ve raised a ticket with Atmire to ask
Another worrying error from dspace.log is:
-
-
Read more →
@@ -286,8 +276,6 @@ DELETE 1
-
-
Read more →
@@ -320,8 +308,6 @@ DELETE 1
0000-0002-6115-0956||0000-0002-3812-8793||0000-0001-7462-405X
-
-
Read more →
@@ -350,8 +336,6 @@ DELETE 1
$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=org" -D "admigration1@cgiarad.org" -W "(sAMAccountName=admigration1)"
-
-
Read more →
@@ -384,8 +368,6 @@ DELETE 1
$ git reset --hard ilri/5_x-prod
$ git rebase -i dspace-5.5
-
-
Read more →
@@ -421,8 +403,6 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
- In this case the select query was showing 95 results before the update
-
-
Read more →
diff --git a/docs/page/4/index.html b/docs/page/4/index.html
index 85344c20f..caf0a14c1 100644
--- a/docs/page/4/index.html
+++ b/docs/page/4/index.html
@@ -14,7 +14,7 @@
-
+
@@ -108,8 +108,6 @@
You can see the others by using the OAI ListSets
verb: http://ebrary.ifpri.org/oai/oai.php?verb=ListSets
Working on second phase of metadata migration, looks like this will work for moving CPWF-specific data in dc.identifier.fund
to cg.identifier.cpwfproject
and then the rest to dc.description.sponsorship
-
-
Read more →
@@ -138,8 +136,6 @@
# awk '{print $1}' /var/log/nginx/rest.log | uniq | wc -l
3168
-
-
Read more →
@@ -166,8 +162,6 @@
This will save us a few gigs of backup space we’re paying for on S3
Also, I noticed the checker
log has some errors we should pay attention to:
-
-
Read more →
@@ -192,8 +186,6 @@
For some reason we still have the index-lucene-update
cron job active on CGSpace, but I’m pretty sure we don’t need it as of the latest few versions of Atmire’s Listings and Reports module
Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Java JDK 1.7 to match environment on CGSpace server
-
-
Read more →
@@ -225,8 +217,6 @@
Not only are there 49,000 countries, we have some blanks (25)…
Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE”
-
-
Read more →
@@ -251,8 +241,6 @@
I realized it is only necessary to clear the Cocoon cache after moving collections—rather than reindexing—as no metadata has changed, and therefore no search or browse indexes need to be updated.
Update GitHub wiki for documentation of maintenance tasks.
-
-
Read more →
@@ -282,8 +270,6 @@
-rw-rw-r-- 1 tomcat7 tomcat7 387K Nov 18 23:59 dspace.log.2015-11-18.lzo
-rw-rw-r-- 1 tomcat7 tomcat7 169K Nov 18 23:59 dspace.log.2015-11-18.xz
-
-
Read more →
@@ -312,8 +298,6 @@
$ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
78
-
-
Read more →
diff --git a/docs/posts/index.html b/docs/posts/index.html
index 3cef573e3..8b858b9ed 100644
--- a/docs/posts/index.html
+++ b/docs/posts/index.html
@@ -14,7 +14,7 @@
-
+
@@ -111,8 +111,6 @@
Linode has been sending mails a few times a day recently that CGSpace (linode18) has had high CPU usage
Today these are the top 10 IPs:
-
-
Read more →
@@ -136,8 +134,6 @@
Phil Thornton got an ORCID identifier so we need to add it to the list on CGSpace and tag his existing items
I created a GitHub issue to track this #389, because I’m super busy in Nairobi right now
-
-
Read more →
@@ -163,8 +159,6 @@
Also, I’ll re-run the postgresql
tasks because the custom PostgreSQL variables are dynamic according to the system’s RAM, and we never re-ran them after migrating to larger Linodes last month
I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I’m getting those autowire errors in Tomcat 8.5.30 again:
-
-
Read more →
@@ -201,8 +195,6 @@
The server only has 8GB of RAM so we’ll eventually need to upgrade to a larger one because we’ll start starving the OS, PostgreSQL, and command line batch processes
I ran all system updates on DSpace Test and rebooted it
-
-
Read more →
@@ -235,8 +227,6 @@
There is insufficient memory for the Java Runtime Environment to continue.
-
-
Read more →
@@ -280,8 +270,6 @@ real 74m42.646s
user 8m5.056s
sys 2m7.289s
-
-
Read more →
@@ -311,8 +299,6 @@ sys 2m7.289s
Then I reduced the JVM heap size from 6144 back to 5120m
Also, I switched it to use OpenJDK instead of Oracle Java, as well as re-worked the Ansible infrastructure scripts to support hosts choosing which distribution they want to use
-
-
Read more →
@@ -336,8 +322,6 @@ sys 2m7.289s
I tried to test something on DSpace Test but noticed that it’s down since god knows when
Catalina logs at least show some memory errors yesterday:
-
-
Read more →
@@ -360,8 +344,6 @@ sys 2m7.289s
- Export a CSV of the IITA community metadata for Martin Mueller
-
-
Read more →
@@ -387,8 +369,6 @@ sys 2m7.289s
Yesterday I figured out how to monitor DSpace sessions using JMX
I copied the logic in the jmx_tomcat_dbpools
provided by Ubuntu’s munin-plugins-java
package and used the stuff I discovered about JMX in 2018-01
-
-
Read more →
diff --git a/docs/posts/index.xml b/docs/posts/index.xml
index f84ea936a..ff20bc24c 100644
--- a/docs/posts/index.xml
+++ b/docs/posts/index.xml
@@ -29,9 +29,7 @@
<ul>
<li>Linode has been sending mails a few times a day recently that CGSpace (linode18) has had high CPU usage</li>
<li>Today these are the top 10 IPs:</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -45,9 +43,7 @@
<ul>
<li>Phil Thornton got an ORCID identifier so we need to add it to the list on CGSpace and tag his existing items</li>
<li>I created a GitHub issue to track this <a href="https://github.com/ilri/DSpace/issues/389">#389</a>, because I’m super busy in Nairobi right now</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -63,9 +59,7 @@
<li>I’ll update the DSpace role in our <a href="https://github.com/ilri/rmg-ansible-public">Ansible infrastructure playbooks</a> and run the updated playbooks on CGSpace and DSpace Test</li>
<li>Also, I’ll re-run the <code>postgresql</code> tasks because the custom PostgreSQL variables are dynamic according to the system’s RAM, and we never re-ran them after migrating to larger Linodes last month</li>
<li>I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I’m getting those autowire errors in Tomcat 8.5.30 again:</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -92,9 +86,7 @@
<li>Anyways, perhaps I should increase the JVM heap from 5120m to 6144m like we did a few months ago when we tried to run the whole CGSpace Solr core</li>
<li>The server only has 8GB of RAM so we’ll eventually need to upgrade to a larger one because we’ll start starving the OS, PostgreSQL, and command line batch processes</li>
<li>I ran all system updates on DSpace Test and rebooted it</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -117,9 +109,7 @@
</ul>
<pre><code>There is insufficient memory for the Java Runtime Environment to continue.
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -153,9 +143,7 @@
real 74m42.646s
user 8m5.056s
sys 2m7.289s
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -175,9 +163,7 @@ sys 2m7.289s
</ul></li>
<li>Then I reduced the JVM heap size from 6144 back to 5120m</li>
<li>Also, I switched it to use OpenJDK instead of Oracle Java, as well as re-worked the <a href="https://github.com/ilri/rmg-ansible-public">Ansible infrastructure scripts</a> to support hosts choosing which distribution they want to use</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -191,9 +177,7 @@ sys 2m7.289s
<ul>
<li>I tried to test something on DSpace Test but noticed that it’s down since god knows when</li>
<li>Catalina logs at least show some memory errors yesterday:</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -206,9 +190,7 @@ sys 2m7.289s
<ul>
<li>Export a CSV of the IITA community metadata for Martin Mueller</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -224,9 +206,7 @@ sys 2m7.289s
<li>We don’t need to distinguish between internal and external works, so that makes it just a simple list</li>
<li>Yesterday I figured out how to monitor DSpace sessions using JMX</li>
<li>I copied the logic in the <code>jmx_tomcat_dbpools</code> provided by Ubuntu’s <code>munin-plugins-java</code> package and used the stuff I discovered about JMX <a href="https://alanorth.github.io/cgspace-notes/cgspace-notes/2018-01/">in 2018-01</a></li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -311,9 +291,7 @@ dspace.log.2018-01-02:34
<ul>
<li>Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let’s Encrypt if it’s just a handful of domains</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -329,9 +307,7 @@ dspace.log.2018-01-02:34
<li>The logs say “Timeout waiting for idle object”</li>
<li>PostgreSQL activity says there are 115 connections currently</li>
<li>The list of connections to XMLUI and REST API for today:</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -362,9 +338,7 @@ dspace.log.2018-01-02:34
<pre><code>dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors.csv with csv;
COPY 54701
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -385,9 +359,7 @@ COPY 54701
<ul>
<li>There appears to be a pattern but I’ll have to look a bit closer and try to clean them up automatically, either in SQL or in OpenRefine</li>
<li>Add Katherine Lutz to the groups for content submission and edit steps of the CGIAR System collections</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -396,9 +368,7 @@ COPY 54701
Mon, 18 Sep 2017 16:38:35 +0300
https://alanorth.github.io/cgspace-notes/cgiar-library-migration/
- <p>Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called <em>CGIAR System Organization</em>.</p>
-
-<p></p>
+ <p>Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called <em>CGIAR System Organization</em>.</p>
-
@@ -417,9 +387,7 @@ COPY 54701
<ul>
<li>Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account is both in the approvers step as well as the group</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -450,9 +418,7 @@ COPY 54701
<li>This was due to newline characters in the <code>dc.description.abstract</code> column, which caused OpenRefine to choke when exporting the CSV</li>
<li>I exported a new CSV from the collection on DSpace Test and then manually removed the characters in vim using <code>g/^$/d</code></li>
<li>Then I cleaned up the author authorities and HTML characters in OpenRefine and sent the file back to Abenet</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -473,9 +439,7 @@ COPY 54701
<li>Merge changes for WLE Phase II theme rename (<a href="https://github.com/ilri/DSpace/pull/329">#329</a>)</li>
<li>Looking at extracting the metadata registries from ICARDA’s MEL DSpace database so we can compare fields with CGSpace</li>
<li>We can use PostgreSQL’s extended output format (<code>-x</code>) plus <code>sed</code> to format the output into quasi XML:</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -517,9 +481,7 @@ COPY 54701
</ul>
<pre><code>$ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Thumbnail" -v >& /tmp/filter-media-cmyk.txt
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -550,9 +512,7 @@ COPY 54701
<pre><code>$ identify ~/Desktop/alc_contrastes_desafios.jpg
/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -581,9 +541,7 @@ DELETE 1
<ul>
<li>Create issue on GitHub to track the addition of CCAFS Phase II project tags (<a href="https://github.com/ilri/DSpace/issues/301">#301</a>)</li>
<li>Looks like we’ll be using <code>cg.identifier.ccafsprojectpii</code> as the field name</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -598,9 +556,7 @@ DELETE 1
<li>I checked to see if the Solr sharding task that is supposed to run on January 1st had run and saw there was an error</li>
<li>I tested on DSpace Test as well and it doesn’t work there either</li>
<li>I asked on the dspace-tech mailing list because it seems to be broken, and actually now I’m not sure if we’ve ever had the sharding task run successfully over all these years</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -627,9 +583,7 @@ DELETE 1
<li>I see thousands of them in the logs for the last few months, so it’s not related to the DSpace 5.5 upgrade</li>
<li>I’ve raised a ticket with Atmire to ask</li>
<li>Another worrying error from dspace.log is:</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -644,9 +598,7 @@ DELETE 1
<li>Add <code>dc.type</code> to the output options for Atmire’s Listings and Reports module (<a href="https://github.com/ilri/DSpace/pull/286">#286</a>)</li>
</ul>
-<p><img src="https://alanorth.github.io/cgspace-notes/cgspace-notes/2016/11/listings-and-reports.png" alt="Listings and Reports with output type" /></p>
-
-<p></p>
+<p><img src="https://alanorth.github.io/cgspace-notes/cgspace-notes/2016/11/listings-and-reports.png" alt="Listings and Reports with output type" /></p>
-
@@ -669,9 +621,7 @@ DELETE 1
</ul>
<pre><code>0000-0002-6115-0956||0000-0002-3812-8793||0000-0001-7462-405X
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -690,9 +640,7 @@ DELETE 1
</ul>
<pre><code>$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=org" -D "admigration1@cgiarad.org" -W "(sAMAccountName=admigration1)"
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -715,9 +663,7 @@ DELETE 1
<pre><code>$ git checkout -b 55new 5_x-prod
$ git reset --hard ilri/5_x-prod
$ git rebase -i dspace-5.5
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -743,9 +689,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
<ul>
<li>In this case the select query was showing 95 results before the update</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -763,9 +707,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
<li>This is their publications set: <a href="http://ebrary.ifpri.org/oai/oai.php?verb=ListRecords&from=2016-01-01&set=p15738coll2&metadataPrefix=oai_dc">http://ebrary.ifpri.org/oai/oai.php?verb=ListRecords&from=2016-01-01&set=p15738coll2&metadataPrefix=oai_dc</a></li>
<li>You can see the others by using the OAI <code>ListSets</code> verb: <a href="http://ebrary.ifpri.org/oai/oai.php?verb=ListSets">http://ebrary.ifpri.org/oai/oai.php?verb=ListSets</a></li>
<li>Working on second phase of metadata migration, looks like this will work for moving CPWF-specific data in <code>dc.identifier.fund</code> to <code>cg.identifier.cpwfproject</code> and then the rest to <code>dc.description.sponsorship</code></li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -784,9 +726,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
<pre><code># awk '{print $1}' /var/log/nginx/rest.log | uniq | wc -l
3168
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -803,9 +743,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
<li>After running DSpace for over five years I’ve never needed to look in any other log file than dspace.log, leave alone one from last year!</li>
<li>This will save us a few gigs of backup space we’re paying for on S3</li>
<li>Also, I noticed the <code>checker</code> log has some errors we should pay attention to:</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -820,9 +758,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
<li>Looking at issues with author authorities on CGSpace</li>
<li>For some reason we still have the <code>index-lucene-update</code> cron job active on CGSpace, but I’m pretty sure we don’t need it as of the latest few versions of Atmire’s Listings and Reports module</li>
<li>Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Java JDK 1.7 to match environment on CGSpace server</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -844,9 +780,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
<ul>
<li>Not only are there 49,000 countries, we have some blanks (25)…</li>
<li>Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE”</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -861,9 +795,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
<li>Move ILRI collection <code>10568/12503</code> from <code>10568/27869</code> to <code>10568/27629</code> using the <a href="https://gist.github.com/alanorth/392c4660e8b022d99dfa">move_collections.sh</a> script I wrote last year.</li>
<li>I realized it is only necessary to clear the Cocoon cache after moving collections—rather than reindexing—as no metadata has changed, and therefore no search or browse indexes need to be updated.</li>
<li>Update GitHub wiki for documentation of <a href="https://github.com/ilri/DSpace/wiki/Maintenance-Tasks">maintenance tasks</a>.</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -883,9 +815,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
-rw-rw-r-- 1 tomcat7 tomcat7 2.0M Nov 18 23:59 dspace.log.2015-11-18
-rw-rw-r-- 1 tomcat7 tomcat7 387K Nov 18 23:59 dspace.log.2015-11-18.lzo
-rw-rw-r-- 1 tomcat7 tomcat7 169K Nov 18 23:59 dspace.log.2015-11-18.xz
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -904,9 +834,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
<pre><code>$ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
78
-</code></pre>
-
-<p></p>
+</code></pre>
diff --git a/docs/posts/page/2/index.html b/docs/posts/page/2/index.html
index c85a3bcff..2657609a0 100644
--- a/docs/posts/page/2/index.html
+++ b/docs/posts/page/2/index.html
@@ -14,7 +14,7 @@
-
+
@@ -175,8 +175,6 @@ dspace.log.2018-01-02:34
- Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let’s Encrypt if it’s just a handful of domains
-
-
Read more →
@@ -202,8 +200,6 @@ dspace.log.2018-01-02:34
PostgreSQL activity says there are 115 connections currently
The list of connections to XMLUI and REST API for today:
-
-
Read more →
@@ -244,8 +240,6 @@ dspace.log.2018-01-02:34
dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors.csv with csv;
COPY 54701
-
-
Read more →
@@ -276,8 +270,6 @@ COPY 54701
There appears to be a pattern but I’ll have to look a bit closer and try to clean them up automatically, either in SQL or in OpenRefine
Add Katherine Lutz to the groups for content submission and edit steps of the CGIAR System collections
-
-
Read more →
@@ -296,8 +288,6 @@ COPY 54701
Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called CGIAR System Organization.
-
-
Read more →
@@ -326,8 +316,6 @@ COPY 54701
- Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account is both in the approvers step as well as the group
-
-
Read more →
@@ -368,8 +356,6 @@ COPY 54701
I exported a new CSV from the collection on DSpace Test and then manually removed the characters in vim using g/^$/d
Then I cleaned up the author authorities and HTML characters in OpenRefine and sent the file back to Abenet
-
-
Read more →
@@ -400,8 +386,6 @@ COPY 54701
Looking at extracting the metadata registries from ICARDA’s MEL DSpace database so we can compare fields with CGSpace
We can use PostgreSQL’s extended output format (-x
) plus sed
to format the output into quasi XML:
-
-
Read more →
diff --git a/docs/posts/page/3/index.html b/docs/posts/page/3/index.html
index 026a26f14..3f54449a2 100644
--- a/docs/posts/page/3/index.html
+++ b/docs/posts/page/3/index.html
@@ -14,7 +14,7 @@
-
+
@@ -114,8 +114,6 @@
$ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Thumbnail" -v >& /tmp/filter-media-cmyk.txt
-
-
Read more →
@@ -156,8 +154,6 @@
$ identify ~/Desktop/alc_contrastes_desafios.jpg
/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000
-
-
Read more →
@@ -196,8 +192,6 @@ DELETE 1
Create issue on GitHub to track the addition of CCAFS Phase II project tags (#301)
Looks like we’ll be using cg.identifier.ccafsprojectpii
as the field name
-
-
Read more →
@@ -222,8 +216,6 @@ DELETE 1
I tested on DSpace Test as well and it doesn’t work there either
I asked on the dspace-tech mailing list because it seems to be broken, and actually now I’m not sure if we’ve ever had the sharding task run successfully over all these years
-
-
Read more →
@@ -260,8 +252,6 @@ DELETE 1
I’ve raised a ticket with Atmire to ask
Another worrying error from dspace.log is:
-
-
Read more →
@@ -286,8 +276,6 @@ DELETE 1
-
-
Read more →
@@ -320,8 +308,6 @@ DELETE 1
0000-0002-6115-0956||0000-0002-3812-8793||0000-0001-7462-405X
-
-
Read more →
@@ -350,8 +336,6 @@ DELETE 1
$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=org" -D "admigration1@cgiarad.org" -W "(sAMAccountName=admigration1)"
-
-
Read more →
@@ -384,8 +368,6 @@ DELETE 1
$ git reset --hard ilri/5_x-prod
$ git rebase -i dspace-5.5
-
-
Read more →
@@ -421,8 +403,6 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
- In this case the select query was showing 95 results before the update
-
-
Read more →
diff --git a/docs/posts/page/4/index.html b/docs/posts/page/4/index.html
index 3236a49f0..7400f0ccd 100644
--- a/docs/posts/page/4/index.html
+++ b/docs/posts/page/4/index.html
@@ -14,7 +14,7 @@
-
+
@@ -108,8 +108,6 @@
You can see the others by using the OAI ListSets
verb: http://ebrary.ifpri.org/oai/oai.php?verb=ListSets
Working on second phase of metadata migration, looks like this will work for moving CPWF-specific data in dc.identifier.fund
to cg.identifier.cpwfproject
and then the rest to dc.description.sponsorship
-
-
Read more →
@@ -138,8 +136,6 @@
# awk '{print $1}' /var/log/nginx/rest.log | uniq | wc -l
3168
-
-
Read more →
@@ -166,8 +162,6 @@
This will save us a few gigs of backup space we’re paying for on S3
Also, I noticed the checker
log has some errors we should pay attention to:
-
-
Read more →
@@ -192,8 +186,6 @@
For some reason we still have the index-lucene-update
cron job active on CGSpace, but I’m pretty sure we don’t need it as of the latest few versions of Atmire’s Listings and Reports module
Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Java JDK 1.7 to match environment on CGSpace server
-
-
Read more →
@@ -225,8 +217,6 @@
Not only are there 49,000 countries, we have some blanks (25)…
Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE”
-
-
Read more →
@@ -251,8 +241,6 @@
I realized it is only necessary to clear the Cocoon cache after moving collections—rather than reindexing—as no metadata has changed, and therefore no search or browse indexes need to be updated.
Update GitHub wiki for documentation of maintenance tasks.
-
-
Read more →
@@ -282,8 +270,6 @@
-rw-rw-r-- 1 tomcat7 tomcat7 387K Nov 18 23:59 dspace.log.2015-11-18.lzo
-rw-rw-r-- 1 tomcat7 tomcat7 169K Nov 18 23:59 dspace.log.2015-11-18.xz
-
-
Read more →
@@ -312,8 +298,6 @@
$ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
78
-
-
Read more →
diff --git a/docs/robots.txt b/docs/robots.txt
index 3eebe0dcf..b8338064a 100644
--- a/docs/robots.txt
+++ b/docs/robots.txt
@@ -41,7 +41,7 @@ Disallow: /cgspace-notes/2015-12/
Disallow: /cgspace-notes/2015-11/
Disallow: /cgspace-notes/
Disallow: /cgspace-notes/categories/
-Disallow: /cgspace-notes/tags/notes/
Disallow: /cgspace-notes/categories/notes/
+Disallow: /cgspace-notes/tags/notes/
Disallow: /cgspace-notes/posts/
Disallow: /cgspace-notes/tags/
diff --git a/docs/sitemap.xml b/docs/sitemap.xml
index 6bff85414..c00bfbc92 100644
--- a/docs/sitemap.xml
+++ b/docs/sitemap.xml
@@ -4,7 +4,7 @@
https://alanorth.github.io/cgspace-notes/2018-11/
- 2018-11-06T18:03:44+02:00
+ 2018-11-07T19:20:25+02:00
@@ -194,7 +194,7 @@
https://alanorth.github.io/cgspace-notes/
- 2018-11-06T18:03:44+02:00
+ 2018-11-07T19:20:25+02:00
0
@@ -203,27 +203,27 @@
0
-
- https://alanorth.github.io/cgspace-notes/tags/notes/
- 2018-11-06T18:03:44+02:00
- 0
-
-
https://alanorth.github.io/cgspace-notes/categories/notes/
2018-03-09T22:10:33+02:00
0
+
+ https://alanorth.github.io/cgspace-notes/tags/notes/
+ 2018-11-07T19:20:25+02:00
+ 0
+
+
https://alanorth.github.io/cgspace-notes/posts/
- 2018-11-06T18:03:44+02:00
+ 2018-11-07T19:20:25+02:00
0
https://alanorth.github.io/cgspace-notes/tags/
- 2018-11-06T18:03:44+02:00
+ 2018-11-07T19:20:25+02:00
0
diff --git a/docs/tags/index.html b/docs/tags/index.html
index a4d0067b8..343c20866 100644
--- a/docs/tags/index.html
+++ b/docs/tags/index.html
@@ -14,7 +14,7 @@
-
+
@@ -111,8 +111,6 @@
Linode has been sending mails a few times a day recently that CGSpace (linode18) has had high CPU usage
Today these are the top 10 IPs:
-
-
Read more →
@@ -136,8 +134,6 @@
Phil Thornton got an ORCID identifier so we need to add it to the list on CGSpace and tag his existing items
I created a GitHub issue to track this #389, because I’m super busy in Nairobi right now
-
-
Read more →
@@ -163,8 +159,6 @@
Also, I’ll re-run the postgresql
tasks because the custom PostgreSQL variables are dynamic according to the system’s RAM, and we never re-ran them after migrating to larger Linodes last month
I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I’m getting those autowire errors in Tomcat 8.5.30 again:
-
-
Read more →
@@ -201,8 +195,6 @@
The server only has 8GB of RAM so we’ll eventually need to upgrade to a larger one because we’ll start starving the OS, PostgreSQL, and command line batch processes
I ran all system updates on DSpace Test and rebooted it
-
-
Read more →
@@ -235,8 +227,6 @@
There is insufficient memory for the Java Runtime Environment to continue.
-
-
Read more →
@@ -280,8 +270,6 @@ real 74m42.646s
user 8m5.056s
sys 2m7.289s
-
-
Read more →
@@ -311,8 +299,6 @@ sys 2m7.289s
Then I reduced the JVM heap size from 6144 back to 5120m
Also, I switched it to use OpenJDK instead of Oracle Java, as well as re-worked the Ansible infrastructure scripts to support hosts choosing which distribution they want to use
-
-
Read more →
@@ -336,8 +322,6 @@ sys 2m7.289s
I tried to test something on DSpace Test but noticed that it’s down since god knows when
Catalina logs at least show some memory errors yesterday:
-
-
Read more →
@@ -360,8 +344,6 @@ sys 2m7.289s
- Export a CSV of the IITA community metadata for Martin Mueller
-
-
Read more →
@@ -387,8 +369,6 @@ sys 2m7.289s
Yesterday I figured out how to monitor DSpace sessions using JMX
I copied the logic in the jmx_tomcat_dbpools
provided by Ubuntu’s munin-plugins-java
package and used the stuff I discovered about JMX in 2018-01
-
-
Read more →
diff --git a/docs/tags/notes/index.html b/docs/tags/notes/index.html
index 6854409e6..f47e3e53e 100644
--- a/docs/tags/notes/index.html
+++ b/docs/tags/notes/index.html
@@ -14,7 +14,7 @@
-
+
@@ -96,8 +96,6 @@
Linode has been sending mails a few times a day recently that CGSpace (linode18) has had high CPU usage
Today these are the top 10 IPs:
-
-
Read more →
@@ -121,8 +119,6 @@
Phil Thornton got an ORCID identifier so we need to add it to the list on CGSpace and tag his existing items
I created a GitHub issue to track this #389, because I’m super busy in Nairobi right now
-
-
Read more →
@@ -148,8 +144,6 @@
Also, I’ll re-run the postgresql
tasks because the custom PostgreSQL variables are dynamic according to the system’s RAM, and we never re-ran them after migrating to larger Linodes last month
I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I’m getting those autowire errors in Tomcat 8.5.30 again:
-
-
Read more →
@@ -186,8 +180,6 @@
The server only has 8GB of RAM so we’ll eventually need to upgrade to a larger one because we’ll start starving the OS, PostgreSQL, and command line batch processes
I ran all system updates on DSpace Test and rebooted it
-
-
Read more →
@@ -220,8 +212,6 @@
There is insufficient memory for the Java Runtime Environment to continue.
-
-
Read more →
@@ -265,8 +255,6 @@ real 74m42.646s
user 8m5.056s
sys 2m7.289s
-
-
Read more →
@@ -296,8 +284,6 @@ sys 2m7.289s
Then I reduced the JVM heap size from 6144 back to 5120m
Also, I switched it to use OpenJDK instead of Oracle Java, as well as re-worked the Ansible infrastructure scripts to support hosts choosing which distribution they want to use
-
-
Read more →
@@ -321,8 +307,6 @@ sys 2m7.289s
I tried to test something on DSpace Test but noticed that it’s down since god knows when
Catalina logs at least show some memory errors yesterday:
-
-
Read more →
@@ -345,8 +329,6 @@ sys 2m7.289s
- Export a CSV of the IITA community metadata for Martin Mueller
-
-
Read more →
@@ -372,8 +354,6 @@ sys 2m7.289s
Yesterday I figured out how to monitor DSpace sessions using JMX
I copied the logic in the jmx_tomcat_dbpools
provided by Ubuntu’s munin-plugins-java
package and used the stuff I discovered about JMX in 2018-01
-
-
Read more →
diff --git a/docs/tags/notes/index.xml b/docs/tags/notes/index.xml
index b6f0a76d6..b0f93b0c9 100644
--- a/docs/tags/notes/index.xml
+++ b/docs/tags/notes/index.xml
@@ -29,9 +29,7 @@
<ul>
<li>Linode has been sending mails a few times a day recently that CGSpace (linode18) has had high CPU usage</li>
<li>Today these are the top 10 IPs:</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -45,9 +43,7 @@
<ul>
<li>Phil Thornton got an ORCID identifier so we need to add it to the list on CGSpace and tag his existing items</li>
<li>I created a GitHub issue to track this <a href="https://github.com/ilri/DSpace/issues/389">#389</a>, because I’m super busy in Nairobi right now</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -63,9 +59,7 @@
<li>I’ll update the DSpace role in our <a href="https://github.com/ilri/rmg-ansible-public">Ansible infrastructure playbooks</a> and run the updated playbooks on CGSpace and DSpace Test</li>
<li>Also, I’ll re-run the <code>postgresql</code> tasks because the custom PostgreSQL variables are dynamic according to the system’s RAM, and we never re-ran them after migrating to larger Linodes last month</li>
<li>I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I’m getting those autowire errors in Tomcat 8.5.30 again:</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -92,9 +86,7 @@
<li>Anyways, perhaps I should increase the JVM heap from 5120m to 6144m like we did a few months ago when we tried to run the whole CGSpace Solr core</li>
<li>The server only has 8GB of RAM so we’ll eventually need to upgrade to a larger one because we’ll start starving the OS, PostgreSQL, and command line batch processes</li>
<li>I ran all system updates on DSpace Test and rebooted it</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -117,9 +109,7 @@
</ul>
<pre><code>There is insufficient memory for the Java Runtime Environment to continue.
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -153,9 +143,7 @@
real 74m42.646s
user 8m5.056s
sys 2m7.289s
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -175,9 +163,7 @@ sys 2m7.289s
</ul></li>
<li>Then I reduced the JVM heap size from 6144 back to 5120m</li>
<li>Also, I switched it to use OpenJDK instead of Oracle Java, as well as re-worked the <a href="https://github.com/ilri/rmg-ansible-public">Ansible infrastructure scripts</a> to support hosts choosing which distribution they want to use</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -191,9 +177,7 @@ sys 2m7.289s
<ul>
<li>I tried to test something on DSpace Test but noticed that it’s down since god knows when</li>
<li>Catalina logs at least show some memory errors yesterday:</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -206,9 +190,7 @@ sys 2m7.289s
<ul>
<li>Export a CSV of the IITA community metadata for Martin Mueller</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -224,9 +206,7 @@ sys 2m7.289s
<li>We don’t need to distinguish between internal and external works, so that makes it just a simple list</li>
<li>Yesterday I figured out how to monitor DSpace sessions using JMX</li>
<li>I copied the logic in the <code>jmx_tomcat_dbpools</code> provided by Ubuntu’s <code>munin-plugins-java</code> package and used the stuff I discovered about JMX <a href="https://alanorth.github.io/cgspace-notes/cgspace-notes/2018-01/">in 2018-01</a></li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -311,9 +291,7 @@ dspace.log.2018-01-02:34
<ul>
<li>Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let’s Encrypt if it’s just a handful of domains</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -329,9 +307,7 @@ dspace.log.2018-01-02:34
<li>The logs say “Timeout waiting for idle object”</li>
<li>PostgreSQL activity says there are 115 connections currently</li>
<li>The list of connections to XMLUI and REST API for today:</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -362,9 +338,7 @@ dspace.log.2018-01-02:34
<pre><code>dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors.csv with csv;
COPY 54701
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -385,9 +359,7 @@ COPY 54701
<ul>
<li>There appears to be a pattern but I’ll have to look a bit closer and try to clean them up automatically, either in SQL or in OpenRefine</li>
<li>Add Katherine Lutz to the groups for content submission and edit steps of the CGIAR System collections</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -406,9 +378,7 @@ COPY 54701
<ul>
<li>Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account is both in the approvers step as well as the group</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -439,9 +409,7 @@ COPY 54701
<li>This was due to newline characters in the <code>dc.description.abstract</code> column, which caused OpenRefine to choke when exporting the CSV</li>
<li>I exported a new CSV from the collection on DSpace Test and then manually removed the characters in vim using <code>g/^$/d</code></li>
<li>Then I cleaned up the author authorities and HTML characters in OpenRefine and sent the file back to Abenet</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -462,9 +430,7 @@ COPY 54701
<li>Merge changes for WLE Phase II theme rename (<a href="https://github.com/ilri/DSpace/pull/329">#329</a>)</li>
<li>Looking at extracting the metadata registries from ICARDA’s MEL DSpace database so we can compare fields with CGSpace</li>
<li>We can use PostgreSQL’s extended output format (<code>-x</code>) plus <code>sed</code> to format the output into quasi XML:</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -506,9 +472,7 @@ COPY 54701
</ul>
<pre><code>$ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Thumbnail" -v >& /tmp/filter-media-cmyk.txt
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -539,9 +503,7 @@ COPY 54701
<pre><code>$ identify ~/Desktop/alc_contrastes_desafios.jpg
/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -570,9 +532,7 @@ DELETE 1
<ul>
<li>Create issue on GitHub to track the addition of CCAFS Phase II project tags (<a href="https://github.com/ilri/DSpace/issues/301">#301</a>)</li>
<li>Looks like we’ll be using <code>cg.identifier.ccafsprojectpii</code> as the field name</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -587,9 +547,7 @@ DELETE 1
<li>I checked to see if the Solr sharding task that is supposed to run on January 1st had run and saw there was an error</li>
<li>I tested on DSpace Test as well and it doesn’t work there either</li>
<li>I asked on the dspace-tech mailing list because it seems to be broken, and actually now I’m not sure if we’ve ever had the sharding task run successfully over all these years</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -616,9 +574,7 @@ DELETE 1
<li>I see thousands of them in the logs for the last few months, so it’s not related to the DSpace 5.5 upgrade</li>
<li>I’ve raised a ticket with Atmire to ask</li>
<li>Another worrying error from dspace.log is:</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -633,9 +589,7 @@ DELETE 1
<li>Add <code>dc.type</code> to the output options for Atmire’s Listings and Reports module (<a href="https://github.com/ilri/DSpace/pull/286">#286</a>)</li>
</ul>
-<p><img src="https://alanorth.github.io/cgspace-notes/cgspace-notes/2016/11/listings-and-reports.png" alt="Listings and Reports with output type" /></p>
-
-<p></p>
+<p><img src="https://alanorth.github.io/cgspace-notes/cgspace-notes/2016/11/listings-and-reports.png" alt="Listings and Reports with output type" /></p>
-
@@ -658,9 +612,7 @@ DELETE 1
</ul>
<pre><code>0000-0002-6115-0956||0000-0002-3812-8793||0000-0001-7462-405X
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -679,9 +631,7 @@ DELETE 1
</ul>
<pre><code>$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=org" -D "admigration1@cgiarad.org" -W "(sAMAccountName=admigration1)"
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -704,9 +654,7 @@ DELETE 1
<pre><code>$ git checkout -b 55new 5_x-prod
$ git reset --hard ilri/5_x-prod
$ git rebase -i dspace-5.5
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -732,9 +680,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
<ul>
<li>In this case the select query was showing 95 results before the update</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -752,9 +698,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
<li>This is their publications set: <a href="http://ebrary.ifpri.org/oai/oai.php?verb=ListRecords&from=2016-01-01&set=p15738coll2&metadataPrefix=oai_dc">http://ebrary.ifpri.org/oai/oai.php?verb=ListRecords&from=2016-01-01&set=p15738coll2&metadataPrefix=oai_dc</a></li>
<li>You can see the others by using the OAI <code>ListSets</code> verb: <a href="http://ebrary.ifpri.org/oai/oai.php?verb=ListSets">http://ebrary.ifpri.org/oai/oai.php?verb=ListSets</a></li>
<li>Working on second phase of metadata migration, looks like this will work for moving CPWF-specific data in <code>dc.identifier.fund</code> to <code>cg.identifier.cpwfproject</code> and then the rest to <code>dc.description.sponsorship</code></li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -773,9 +717,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
<pre><code># awk '{print $1}' /var/log/nginx/rest.log | uniq | wc -l
3168
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -792,9 +734,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
<li>After running DSpace for over five years I’ve never needed to look in any other log file than dspace.log, leave alone one from last year!</li>
<li>This will save us a few gigs of backup space we’re paying for on S3</li>
<li>Also, I noticed the <code>checker</code> log has some errors we should pay attention to:</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -809,9 +749,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
<li>Looking at issues with author authorities on CGSpace</li>
<li>For some reason we still have the <code>index-lucene-update</code> cron job active on CGSpace, but I’m pretty sure we don’t need it as of the latest few versions of Atmire’s Listings and Reports module</li>
<li>Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Java JDK 1.7 to match environment on CGSpace server</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -833,9 +771,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
<ul>
<li>Not only are there 49,000 countries, we have some blanks (25)…</li>
<li>Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE”</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -850,9 +786,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
<li>Move ILRI collection <code>10568/12503</code> from <code>10568/27869</code> to <code>10568/27629</code> using the <a href="https://gist.github.com/alanorth/392c4660e8b022d99dfa">move_collections.sh</a> script I wrote last year.</li>
<li>I realized it is only necessary to clear the Cocoon cache after moving collections—rather than reindexing—as no metadata has changed, and therefore no search or browse indexes need to be updated.</li>
<li>Update GitHub wiki for documentation of <a href="https://github.com/ilri/DSpace/wiki/Maintenance-Tasks">maintenance tasks</a>.</li>
-</ul>
-
-<p></p>
+</ul>
-
@@ -872,9 +806,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
-rw-rw-r-- 1 tomcat7 tomcat7 2.0M Nov 18 23:59 dspace.log.2015-11-18
-rw-rw-r-- 1 tomcat7 tomcat7 387K Nov 18 23:59 dspace.log.2015-11-18.lzo
-rw-rw-r-- 1 tomcat7 tomcat7 169K Nov 18 23:59 dspace.log.2015-11-18.xz
-</code></pre>
-
-<p></p>
+</code></pre>
-
@@ -893,9 +825,7 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
<pre><code>$ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
78
-</code></pre>
-
-<p></p>
+</code></pre>
diff --git a/docs/tags/notes/page/2/index.html b/docs/tags/notes/page/2/index.html
index e3d0340ee..5a9ecff78 100644
--- a/docs/tags/notes/page/2/index.html
+++ b/docs/tags/notes/page/2/index.html
@@ -14,7 +14,7 @@
-
+
@@ -160,8 +160,6 @@ dspace.log.2018-01-02:34
- Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let’s Encrypt if it’s just a handful of domains
-
-
Read more →
@@ -187,8 +185,6 @@ dspace.log.2018-01-02:34
PostgreSQL activity says there are 115 connections currently
The list of connections to XMLUI and REST API for today:
-
-
Read more →
@@ -229,8 +225,6 @@ dspace.log.2018-01-02:34
dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors.csv with csv;
COPY 54701
-
-
Read more →
@@ -261,8 +255,6 @@ COPY 54701
There appears to be a pattern but I’ll have to look a bit closer and try to clean them up automatically, either in SQL or in OpenRefine
Add Katherine Lutz to the groups for content submission and edit steps of the CGIAR System collections
-
-
Read more →
@@ -291,8 +283,6 @@ COPY 54701
- Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account is both in the approvers step as well as the group
-
-
Read more →
@@ -333,8 +323,6 @@ COPY 54701
I exported a new CSV from the collection on DSpace Test and then manually removed the characters in vim using g/^$/d
Then I cleaned up the author authorities and HTML characters in OpenRefine and sent the file back to Abenet
-
-
Read more →
@@ -365,8 +353,6 @@ COPY 54701
Looking at extracting the metadata registries from ICARDA’s MEL DSpace database so we can compare fields with CGSpace
We can use PostgreSQL’s extended output format (-x
) plus sed
to format the output into quasi XML:
-
-
Read more →
@@ -436,8 +422,6 @@ COPY 54701
$ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Thumbnail" -v >& /tmp/filter-media-cmyk.txt
-
-
Read more →
diff --git a/docs/tags/notes/page/3/index.html b/docs/tags/notes/page/3/index.html
index 6bc16f95a..492efa756 100644
--- a/docs/tags/notes/page/3/index.html
+++ b/docs/tags/notes/page/3/index.html
@@ -14,7 +14,7 @@
-
+
@@ -106,8 +106,6 @@
$ identify ~/Desktop/alc_contrastes_desafios.jpg
/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000
-
-
Read more →
@@ -146,8 +144,6 @@ DELETE 1
Create issue on GitHub to track the addition of CCAFS Phase II project tags (#301)
Looks like we’ll be using cg.identifier.ccafsprojectpii
as the field name
-
-
Read more →
@@ -172,8 +168,6 @@ DELETE 1
I tested on DSpace Test as well and it doesn’t work there either
I asked on the dspace-tech mailing list because it seems to be broken, and actually now I’m not sure if we’ve ever had the sharding task run successfully over all these years
-
-
Read more →
@@ -210,8 +204,6 @@ DELETE 1
I’ve raised a ticket with Atmire to ask
Another worrying error from dspace.log is:
-
-
Read more →
@@ -236,8 +228,6 @@ DELETE 1
-
-
Read more →
@@ -270,8 +260,6 @@ DELETE 1
0000-0002-6115-0956||0000-0002-3812-8793||0000-0001-7462-405X
-
-
Read more →
@@ -300,8 +288,6 @@ DELETE 1
$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=org" -D "admigration1@cgiarad.org" -W "(sAMAccountName=admigration1)"
-
-
Read more →
@@ -334,8 +320,6 @@ DELETE 1
$ git reset --hard ilri/5_x-prod
$ git rebase -i dspace-5.5
-
-
Read more →
@@ -371,8 +355,6 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
- In this case the select query was showing 95 results before the update
-
-
Read more →
@@ -400,8 +382,6 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
You can see the others by using the OAI ListSets
verb: http://ebrary.ifpri.org/oai/oai.php?verb=ListSets
Working on second phase of metadata migration, looks like this will work for moving CPWF-specific data in dc.identifier.fund
to cg.identifier.cpwfproject
and then the rest to dc.description.sponsorship
-
-
Read more →
diff --git a/docs/tags/notes/page/4/index.html b/docs/tags/notes/page/4/index.html
index 80c7edf2b..29d495f54 100644
--- a/docs/tags/notes/page/4/index.html
+++ b/docs/tags/notes/page/4/index.html
@@ -14,7 +14,7 @@
-
+
@@ -94,8 +94,6 @@
# awk '{print $1}' /var/log/nginx/rest.log | uniq | wc -l
3168
-
-
Read more →
@@ -122,8 +120,6 @@
This will save us a few gigs of backup space we’re paying for on S3
Also, I noticed the checker
log has some errors we should pay attention to:
-
-
Read more →
@@ -148,8 +144,6 @@
For some reason we still have the index-lucene-update
cron job active on CGSpace, but I’m pretty sure we don’t need it as of the latest few versions of Atmire’s Listings and Reports module
Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Java JDK 1.7 to match environment on CGSpace server
-
-
Read more →
@@ -181,8 +175,6 @@
Not only are there 49,000 countries, we have some blanks (25)…
Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE”
-
-
Read more →
@@ -207,8 +199,6 @@
I realized it is only necessary to clear the Cocoon cache after moving collections—rather than reindexing—as no metadata has changed, and therefore no search or browse indexes need to be updated.
Update GitHub wiki for documentation of maintenance tasks.
-
-
Read more →
@@ -238,8 +228,6 @@
-rw-rw-r-- 1 tomcat7 tomcat7 387K Nov 18 23:59 dspace.log.2015-11-18.lzo
-rw-rw-r-- 1 tomcat7 tomcat7 169K Nov 18 23:59 dspace.log.2015-11-18.xz
-
-
Read more →
@@ -268,8 +256,6 @@
$ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
78
-
-
Read more →
diff --git a/docs/tags/page/2/index.html b/docs/tags/page/2/index.html
index a573329dc..8bd9c4fa3 100644
--- a/docs/tags/page/2/index.html
+++ b/docs/tags/page/2/index.html
@@ -14,7 +14,7 @@
-
+
@@ -175,8 +175,6 @@ dspace.log.2018-01-02:34
- Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let’s Encrypt if it’s just a handful of domains
-
-
Read more →
@@ -202,8 +200,6 @@ dspace.log.2018-01-02:34
PostgreSQL activity says there are 115 connections currently
The list of connections to XMLUI and REST API for today:
-
-
Read more →
@@ -244,8 +240,6 @@ dspace.log.2018-01-02:34
dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors.csv with csv;
COPY 54701
-
-
Read more →
@@ -276,8 +270,6 @@ COPY 54701
There appears to be a pattern but I’ll have to look a bit closer and try to clean them up automatically, either in SQL or in OpenRefine
Add Katherine Lutz to the groups for content submission and edit steps of the CGIAR System collections
-
-
Read more →
@@ -296,8 +288,6 @@ COPY 54701
Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called CGIAR System Organization.
-
-
Read more →
@@ -326,8 +316,6 @@ COPY 54701
- Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account is both in the approvers step as well as the group
-
-
Read more →
@@ -368,8 +356,6 @@ COPY 54701
I exported a new CSV from the collection on DSpace Test and then manually removed the characters in vim using g/^$/d
Then I cleaned up the author authorities and HTML characters in OpenRefine and sent the file back to Abenet
-
-
Read more →
@@ -400,8 +386,6 @@ COPY 54701
Looking at extracting the metadata registries from ICARDA’s MEL DSpace database so we can compare fields with CGSpace
We can use PostgreSQL’s extended output format (-x
) plus sed
to format the output into quasi XML:
-
-
Read more →
diff --git a/docs/tags/page/3/index.html b/docs/tags/page/3/index.html
index adf78f4d0..4d893c72b 100644
--- a/docs/tags/page/3/index.html
+++ b/docs/tags/page/3/index.html
@@ -14,7 +14,7 @@
-
+
@@ -114,8 +114,6 @@
$ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Thumbnail" -v >& /tmp/filter-media-cmyk.txt
-
-
Read more →
@@ -156,8 +154,6 @@
$ identify ~/Desktop/alc_contrastes_desafios.jpg
/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000
-
-
Read more →
@@ -196,8 +192,6 @@ DELETE 1
Create issue on GitHub to track the addition of CCAFS Phase II project tags (#301)
Looks like we’ll be using cg.identifier.ccafsprojectpii
as the field name
-
-
Read more →
@@ -222,8 +216,6 @@ DELETE 1
I tested on DSpace Test as well and it doesn’t work there either
I asked on the dspace-tech mailing list because it seems to be broken, and actually now I’m not sure if we’ve ever had the sharding task run successfully over all these years
-
-
Read more →
@@ -260,8 +252,6 @@ DELETE 1
I’ve raised a ticket with Atmire to ask
Another worrying error from dspace.log is:
-
-
Read more →
@@ -286,8 +276,6 @@ DELETE 1
-
-
Read more →
@@ -320,8 +308,6 @@ DELETE 1
0000-0002-6115-0956||0000-0002-3812-8793||0000-0001-7462-405X
-
-
Read more →
@@ -350,8 +336,6 @@ DELETE 1
$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=org" -D "admigration1@cgiarad.org" -W "(sAMAccountName=admigration1)"
-
-
Read more →
@@ -384,8 +368,6 @@ DELETE 1
$ git reset --hard ilri/5_x-prod
$ git rebase -i dspace-5.5
-
-
Read more →
@@ -421,8 +403,6 @@ dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and
- In this case the select query was showing 95 results before the update
-
-
Read more →
diff --git a/docs/tags/page/4/index.html b/docs/tags/page/4/index.html
index 043952b68..e88417e62 100644
--- a/docs/tags/page/4/index.html
+++ b/docs/tags/page/4/index.html
@@ -14,7 +14,7 @@
-
+
@@ -108,8 +108,6 @@
You can see the others by using the OAI ListSets
verb: http://ebrary.ifpri.org/oai/oai.php?verb=ListSets
Working on second phase of metadata migration, looks like this will work for moving CPWF-specific data in dc.identifier.fund
to cg.identifier.cpwfproject
and then the rest to dc.description.sponsorship
-
-
Read more →
@@ -138,8 +136,6 @@
# awk '{print $1}' /var/log/nginx/rest.log | uniq | wc -l
3168
-
-
Read more →
@@ -166,8 +162,6 @@
This will save us a few gigs of backup space we’re paying for on S3
Also, I noticed the checker
log has some errors we should pay attention to:
-
-
Read more →
@@ -192,8 +186,6 @@
For some reason we still have the index-lucene-update
cron job active on CGSpace, but I’m pretty sure we don’t need it as of the latest few versions of Atmire’s Listings and Reports module
Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Java JDK 1.7 to match environment on CGSpace server
-
-
Read more →
@@ -225,8 +217,6 @@
Not only are there 49,000 countries, we have some blanks (25)…
Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE”
-
-
Read more →
@@ -251,8 +241,6 @@
I realized it is only necessary to clear the Cocoon cache after moving collections—rather than reindexing—as no metadata has changed, and therefore no search or browse indexes need to be updated.
Update GitHub wiki for documentation of maintenance tasks.
-
-
Read more →
@@ -282,8 +270,6 @@
-rw-rw-r-- 1 tomcat7 tomcat7 387K Nov 18 23:59 dspace.log.2015-11-18.lzo
-rw-rw-r-- 1 tomcat7 tomcat7 169K Nov 18 23:59 dspace.log.2015-11-18.xz
-
-
Read more →
@@ -312,8 +298,6 @@
$ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
78
-
-
Read more →