2018-02-11 17:28:23 +01:00
<!DOCTYPE html>
2019-10-11 10:19:42 +02:00
< html lang = "en" >
2018-02-11 17:28:23 +01:00
< head >
< meta charset = "utf-8" >
< meta name = "viewport" content = "width=device-width, initial-scale=1, shrink-to-fit=no" >
< meta property = "og:title" content = "June, 2017" / >
2020-01-27 15:20:44 +01:00
< meta property = "og:description" content = "2017-06-01 After discussion with WLE and CGSpace content people, we decided to just add one metadata field for the WLE Research Themes The cg.identifier.wletheme field will be used for both Phase I and Phase II Research Themes Then we’ll create a new sub-community for Phase II and create collections for the research themes there The current “Research Themes” community will be renamed to “WLE Phase I Research Themes” Tagged all items in the current Phase I collections with their appropriate themes Create pull request to add Phase II research themes to the submission form: #328 Add cg." / >
2018-02-11 17:28:23 +01:00
< meta property = "og:type" content = "article" / >
2019-02-02 13:12:57 +01:00
< meta property = "og:url" content = "https://alanorth.github.io/cgspace-notes/2017-06/" / >
2019-08-08 17:10:44 +02:00
< meta property = "article:published_time" content = "2017-06-01T10:14:52+03:00" / >
< meta property = "article:modified_time" content = "2018-03-09T22:10:33+02:00" / >
2018-09-30 07:23:48 +02:00
2018-02-11 17:28:23 +01:00
< meta name = "twitter:card" content = "summary" / >
< meta name = "twitter:title" content = "June, 2017" / >
2020-01-27 15:20:44 +01:00
< meta name = "twitter:description" content = "2017-06-01 After discussion with WLE and CGSpace content people, we decided to just add one metadata field for the WLE Research Themes The cg.identifier.wletheme field will be used for both Phase I and Phase II Research Themes Then we’ll create a new sub-community for Phase II and create collections for the research themes there The current “Research Themes” community will be renamed to “WLE Phase I Research Themes” Tagged all items in the current Phase I collections with their appropriate themes Create pull request to add Phase II research themes to the submission form: #328 Add cg." / >
2020-03-22 13:35:20 +01:00
< meta name = "generator" content = "Hugo 0.68.0" / >
2018-02-11 17:28:23 +01:00
< script type = "application/ld+json" >
{
"@context": "http://schema.org",
"@type": "BlogPosting",
"headline": "June, 2017",
2019-04-13 11:15:55 +02:00
"url": "https:\/\/alanorth.github.io\/cgspace-notes\/2017-06\/",
2018-02-11 17:28:23 +01:00
"wordCount": "1261",
2019-10-11 10:19:42 +02:00
"datePublished": "2017-06-01T10:14:52+03:00",
"dateModified": "2018-03-09T22:10:33+02:00",
2018-02-11 17:28:23 +01:00
"author": {
"@type": "Person",
"name": "Alan Orth"
},
"keywords": "Notes"
}
< / script >
< link rel = "canonical" href = "https://alanorth.github.io/cgspace-notes/2017-06/" >
< title > June, 2017 | CGSpace Notes< / title >
2019-10-11 10:19:42 +02:00
2018-02-11 17:28:23 +01:00
<!-- combined, minified CSS -->
2020-01-23 19:19:38 +01:00
2020-01-28 11:01:42 +01:00
< link href = "https://alanorth.github.io/cgspace-notes/css/style.6da5c906cc7a8fbb93f31cd2316c5dbe3f19ac4aa6bfb066f1243045b8f6061e.css" rel = "stylesheet" integrity = "sha256-baXJBsx6j7uT8xzSMWxdvj8ZrEqmv7Bm8SQwRbj2Bh4=" crossorigin = "anonymous" >
2019-10-11 10:19:42 +02:00
2018-02-11 17:28:23 +01:00
2020-01-28 11:01:42 +01:00
<!-- minified Font Awesome for SVG icons -->
< script defer src = "https://alanorth.github.io/cgspace-notes/js/fontawesome.min.90e14c13cee52929ac33e1c21694a3cc95063a194eb22aad9f7976434e1a9125.js" integrity = "sha256-kOFME87lKSmsM+HCFpSjzJUGOhlOsiqtn3l2Q04akSU=" crossorigin = "anonymous" > < / script >
2019-04-14 15:59:47 +02:00
<!-- RSS 2.0 feed -->
2018-02-11 17:28:23 +01:00
< / head >
< body >
< div class = "blog-masthead" >
< div class = "container" >
< nav class = "nav blog-nav" >
< a class = "nav-link " href = "https://alanorth.github.io/cgspace-notes/" > Home< / a >
< / nav >
< / div >
< / div >
2018-12-19 12:20:39 +01:00
2018-02-11 17:28:23 +01:00
< header class = "blog-header" >
< div class = "container" >
2019-10-11 10:19:42 +02:00
< h1 class = "blog-title" dir = "auto" > < a href = "https://alanorth.github.io/cgspace-notes/" rel = "home" > CGSpace Notes< / a > < / h1 >
< p class = "lead blog-description" dir = "auto" > Documenting day-to-day work on the < a href = "https://cgspace.cgiar.org" > CGSpace< / a > repository.< / p >
2018-02-11 17:28:23 +01:00
< / div >
< / header >
2018-12-19 12:20:39 +01:00
2018-02-11 17:28:23 +01:00
< div class = "container" >
< div class = "row" >
< div class = "col-sm-8 blog-main" >
< article class = "blog-post" >
< header >
2019-10-11 10:19:42 +02:00
< h2 class = "blog-post-title" dir = "auto" > < a href = "https://alanorth.github.io/cgspace-notes/2017-06/" > June, 2017< / a > < / h2 >
2018-02-11 17:28:23 +01:00
< p class = "blog-post-meta" > < time datetime = "2017-06-01T10:14:52+03:00" > Thu Jun 01, 2017< / time > by Alan Orth in
2020-01-28 11:01:42 +01:00
< span class = "fas fa-tag" aria-hidden = "true" > < / span > < a href = "/cgspace-notes/tags/notes/" rel = "tag" > Notes< / a >
2018-02-11 17:28:23 +01:00
< / p >
< / header >
2019-12-17 13:49:24 +01:00
< h2 id = "2017-06-01" > 2017-06-01< / h2 >
2018-02-11 17:28:23 +01:00
< ul >
< li > After discussion with WLE and CGSpace content people, we decided to just add one metadata field for the WLE Research Themes< / li >
< li > The < code > cg.identifier.wletheme< / code > field will be used for both Phase I and Phase II Research Themes< / li >
2020-01-27 15:20:44 +01:00
< li > Then we’ ll create a new sub-community for Phase II and create collections for the research themes there< / li >
2018-02-11 17:28:23 +01:00
< li > The current “ Research Themes” community will be renamed to “ WLE Phase I Research Themes” < / li >
< li > Tagged all items in the current Phase I collections with their appropriate themes< / li >
< li > Create pull request to add Phase II research themes to the submission form: < a href = "https://github.com/ilri/DSpace/pull/328" > #328< / a > < / li >
< li > Add < code > cg.subject.system< / code > to CGSpace metadata registry, for subject from the upcoming CGIAR Library migration< / li >
< / ul >
2019-12-17 13:49:24 +01:00
< h2 id = "2017-06-04" > 2017-06-04< / h2 >
2018-02-11 17:28:23 +01:00
< ul >
< li > After adding < code > cg.identifier.wletheme< / code > to 1106 WLE items I can see the field on XMLUI but not in REST!< / li >
< li > Strangely it happens on DSpace Test AND on CGSpace!< / li >
2020-01-27 15:20:44 +01:00
< li > I tried to re-index Discovery but it didn’ t fix it< / li >
2018-02-11 17:28:23 +01:00
< li > Run all system updates on DSpace Test and reboot the server< / li >
< li > After rebooting the server (and therefore restarting Tomcat) the new metadata field is available< / li >
2020-01-27 15:20:44 +01:00
< li > I’ ve sent a message to the dspace-tech mailing list to ask if this is a bug and whether I should file a Jira ticket< / li >
2018-02-11 17:28:23 +01:00
< / ul >
2019-12-17 13:49:24 +01:00
< h2 id = "2016-06-05" > 2016-06-05< / h2 >
2018-02-11 17:28:23 +01:00
< ul >
2020-01-27 15:20:44 +01:00
< li > Rename WLE’ s “ Research Themes” sub-community to “ WLE Phase I Research Themes” on DSpace Test so Macaroni Bros can continue their testing< / li >
< li > Macaroni Bros tested it and said it’ s fine, so I renamed it on CGSpace as well< / li >
2018-02-11 17:28:23 +01:00
< li > Working on how to automate the extraction of the CIAT Book chapters, doing some magic in OpenRefine to extract page from–
< ul >
< li > cg.identifier.url: < code > value.split(" page=" , " " )[1]< / code > < / li >
< li > dc.format.extent: < code > value.replace(" p. " , " " ).split(" -" )[1].toNumber() - value.replace(" p. " , " " ).split(" -" )[0].toNumber()< / code > < / li >
2019-11-28 16:30:45 +01:00
< / ul >
< / li >
< li > Finally, after some filtering to see which small outliers there were (based on dc.format.extent using “ p. 1-14” vs “ 29 p." ), create a new column with last page number:
2018-02-11 17:28:23 +01:00
< ul >
< li > < code > cells[" dc.page.from" ].value.toNumber() + cells[" dc.format.pages" ].value.toNumber()< / code > < / li >
2019-11-28 16:30:45 +01:00
< / ul >
< / li >
2018-02-11 17:28:23 +01:00
< li > Then create a new, unique file name to be used in the output, based on a SHA1 of the dc.title and with a description:
< ul >
< li > dc.page.to: < code > value.split(" " )[0].replace(" ," ," " ).toLowercase() + " -" + sha1(value).get(1,9) + " .pdf__description:" + cells[" dc.type" ].value< / code > < / li >
2019-11-28 16:30:45 +01:00
< / ul >
< / li >
2018-02-11 17:28:23 +01:00
< li > Start processing 769 records after filtering the following (there are another 159 records that have some other format, or for example they have their own PDF which I will process later), using a modified < code > generate-thumbnails.py< / code > script to read certain fields and then pass to GhostScript:
< ul >
< li > cg.identifier.url: < code > value.contains(" page=" )< / code > < / li >
< li > dc.format.extent: < code > or(value.contains(" p. " ),value.contains(" p." ))< / code > < / li >
< li > Command like: < code > $ gs -dNOPAUSE -dBATCH -dFirstPage=14 -dLastPage=27 -sDEVICE=pdfwrite -sOutputFile=beans.pdf -f 12605-1.pdf< / code > < / li >
2019-11-28 16:30:45 +01:00
< / ul >
< / li >
2018-02-11 17:28:23 +01:00
< li > 17 of the items have issues with incorrect page number ranges, and upon closer inspection they do not appear in the referenced PDF< / li >
2020-01-27 15:20:44 +01:00
< li > I’ ve flagged them and proceeded without them (752 total) on DSpace Test:< / li >
2019-11-28 16:30:45 +01:00
< / ul >
2018-02-11 17:28:23 +01:00
< pre > < code > $ JAVA_OPTS=" -Xmx1024m -Dfile.encoding=UTF-8" [dspace]/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568/93843 --source /home/aorth/src/CIAT-Books/SimpleArchiveFormat/ --mapfile=/tmp/ciat-books.map & > /tmp/ciat-books.log
2019-11-28 16:30:45 +01:00
< / code > < / pre > < ul >
< li > I went and did some basic sanity checks on the remaining items in the CIAT Book Chapters and decided they are mostly fine (except one duplicate and the flagged ones), so I imported them to DSpace Test too (162 items)< / li >
< li > Total items in CIAT Book Chapters is 914, with the others being flagged for some reason, and we should send that back to CIAT< / li >
< li > Restart Tomcat on CGSpace so that the < code > cg.identifier.wletheme< / code > field is available on REST API for Macaroni Bros< / li >
2018-02-11 17:28:23 +01:00
< / ul >
2019-12-17 13:49:24 +01:00
< h2 id = "2017-06-07" > 2017-06-07< / h2 >
2018-02-11 17:28:23 +01:00
< ul >
2020-01-27 15:20:44 +01:00
< li > Testing < a href = "https://github.com/ilri/DSpace/pull/319" > Atmire’ s patch for the CUA Workflow Statistics again< / a > < / li >
< li > Still doesn’ t seem to give results I’ d expect, like there are no results for Maria Garruccio, or for the ILRI community!< / li >
< li > Then I’ ll file an update to the issue on Atmire’ s tracker< / li >
2018-02-11 17:28:23 +01:00
< li > Created a new branch with just the relevant changes, so I can send it to them< / li >
2019-11-28 16:30:45 +01:00
< li > One thing I noticed is that there is a failed database migration related to CUA:< / li >
< / ul >
2018-02-11 17:28:23 +01:00
< pre > < code > +----------------+----------------------------+---------------------+---------+
| Version | Description | Installed on | State |
+----------------+----------------------------+---------------------+---------+
| 1.1 | Initial DSpace 1.1 databas | | PreInit |
| 1.2 | Upgrade to DSpace 1.2 sche | | PreInit |
| 1.3 | Upgrade to DSpace 1.3 sche | | PreInit |
| 1.3.9 | Drop constraint for DSpace | | PreInit |
| 1.4 | Upgrade to DSpace 1.4 sche | | PreInit |
| 1.5 | Upgrade to DSpace 1.5 sche | | PreInit |
| 1.5.9 | Drop constraint for DSpace | | PreInit |
| 1.6 | Upgrade to DSpace 1.6 sche | | PreInit |
| 1.7 | Upgrade to DSpace 1.7 sche | | PreInit |
| 1.8 | Upgrade to DSpace 1.8 sche | | PreInit |
| 3.0 | Upgrade to DSpace 3.x sche | | PreInit |
| 4.0 | Initializing from DSpace 4 | 2015-11-20 12:42:52 | Success |
| 5.0.2014.08.08 | DS-1945 Helpdesk Request a | 2015-11-20 12:42:53 | Success |
| 5.0.2014.09.25 | DS 1582 Metadata For All O | 2015-11-20 12:42:55 | Success |
| 5.0.2014.09.26 | DS-1582 Metadata For All O | 2015-11-20 12:42:55 | Success |
| 5.0.2015.01.27 | MigrateAtmireExtraMetadata | 2015-11-20 12:43:29 | Success |
| 5.0.2017.04.28 | CUA eperson metadata migra | 2017-06-07 11:07:28 | OutOrde |
| 5.5.2015.12.03 | Atmire CUA 4 migration | 2016-11-27 06:39:05 | OutOrde |
| 5.5.2015.12.03 | Atmire MQM migration | 2016-11-27 06:39:06 | OutOrde |
| 5.6.2016.08.08 | CUA emailreport migration | 2017-01-29 11:18:56 | OutOrde |
+----------------+----------------------------+---------------------+---------+
2019-11-28 16:30:45 +01:00
< / code > < / pre > < ul >
< li > Merge the pull request for < a href = "https://github.com/ilri/DSpace/pull/328" > WLE Phase II themes< / a > < / li >
2018-02-11 17:28:23 +01:00
< / ul >
2019-12-17 13:49:24 +01:00
< h2 id = "2017-06-18" > 2017-06-18< / h2 >
2018-02-11 17:28:23 +01:00
< ul >
< li > Redeploy CGSpace with latest changes from < code > 5_x-prod< / code > , run system updates, and reboot the server< / li >
< li > Continue working on ansible infrastructure changes for CGIAR Library< / li >
< / ul >
2019-12-17 13:49:24 +01:00
< h2 id = "2017-06-20" > 2017-06-20< / h2 >
2018-02-11 17:28:23 +01:00
< ul >
2020-01-27 15:20:44 +01:00
< li > Import Abenet and Peter’ s changes to the CGIAR Library CRP community< / li >
2018-02-11 17:28:23 +01:00
< li > Due to them using Windows and renaming some columns there were formatting, encoding, and duplicate metadata value issues< / li >
< li > I had to remove some fields from the CSV and rename some back to, ie, < code > dc.subject[en_US]< / code > just so DSpace would detect changes properly< / li >
< li > Now it looks much better: < a href = "https://dspacetest.cgiar.org/handle/10947/2517" > https://dspacetest.cgiar.org/handle/10947/2517< / a > < / li >
< li > Removing the HTML tags and HTML/XML entities using the following GREL:
< ul >
< li > < code > replace(value,/< \/?\w+((\s+\w+(\s*=\s*(?:" .*?" |'.*?'|[^'" > \s]+))?)+\s*|\s*)\/?> /,'')< / code > < / li >
< li > < code > value.unescape(" html" ).unescape(" xml" )< / code > < / li >
2019-11-28 16:30:45 +01:00
< / ul >
< / li >
< li > Finally import 914 CIAT Book Chapters to CGSpace in two batches:< / li >
< / ul >
2018-02-11 17:28:23 +01:00
< pre > < code > $ JAVA_OPTS=" -Xmx1024m -Dfile.encoding=UTF-8" [dspace]/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568/35701 --source /home/aorth/CIAT-Books/SimpleArchiveFormat/ --mapfile=/tmp/ciat-books.map & > /tmp/ciat-books.log
$ JAVA_OPTS=" -Xmx1024m -Dfile.encoding=UTF-8" [dspace]/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568/35701 --source /home/aorth/CIAT-Books/SimpleArchiveFormat/ --mapfile=/tmp/ciat-books2.map & > /tmp/ciat-books2.log
2019-12-17 13:49:24 +01:00
< / code > < / pre > < h2 id = "2017-06-25" > 2017-06-25< / h2 >
2018-02-11 17:28:23 +01:00
< ul >
< li > WLE has said that one of their Phase II research themes is being renamed from < code > Regenerating Degraded Landscapes< / code > to < code > Restoring Degraded Landscapes< / code > < / li >
< li > Pull request with the changes to < code > input-forms.xml< / code > : < a href = "https://github.com/ilri/DSpace/pull/329" > #329< / a > < / li >
2020-01-27 15:20:44 +01:00
< li > As of now it doesn’ t look like there are any items using this research theme so we don’ t need to do any updates:< / li >
2019-11-28 16:30:45 +01:00
< / ul >
2018-02-11 17:28:23 +01:00
< pre > < code > dspace=# select text_value from metadatavalue where resource_type_id=2 and metadata_field_id=237 and text_value like 'Regenerating Degraded Landscapes%';
2019-11-28 16:30:45 +01:00
text_value
2018-02-11 17:28:23 +01:00
------------
(0 rows)
2019-11-28 16:30:45 +01:00
< / code > < / pre > < ul >
< li > Marianne from WLE asked if they can have both Phase I and II research themes together in the item submission form< / li >
< li > Perhaps we can add them together in the same question for < code > cg.identifier.wletheme< / code > < / li >
2018-02-11 17:28:23 +01:00
< / ul >
2019-12-17 13:49:24 +01:00
< h2 id = "2017-06-30" > 2017-06-30< / h2 >
2018-02-11 17:28:23 +01:00
< ul >
2019-11-28 16:30:45 +01:00
< li > CGSpace went down briefly, I see lots of these errors in the dspace logs:< / li >
< / ul >
2018-02-11 17:28:23 +01:00
< pre > < code > Java stacktrace: java.util.NoSuchElementException: Timeout waiting for idle object
2019-11-28 16:30:45 +01:00
< / code > < / pre > < ul >
< li > After looking at the Tomcat logs, Munin graphs, and PostgreSQL connection stats, it seems there is just a high load< / li >
2020-01-27 15:20:44 +01:00
< li > Might be a good time to adjust DSpace’ s database connection settings, like I first mentioned in April, 2017 after reading the < a href = "https://wiki.duraspace.org/display/cmtygp/DCAT+Meeting+April+2017" > 2017-04 DCAT comments< / a > < / li >
< li > I’ ve adjusted the following in CGSpace’ s config:
2018-02-11 17:28:23 +01:00
< ul >
2020-01-27 15:20:44 +01:00
< li > < code > db.maxconnections< / code > 30→70 (the default PostgreSQL config allows 100 connections, so DSpace’ s default of 30 is quite low)< / li >
2018-02-11 17:28:23 +01:00
< li > < code > db.maxwait< / code > 5000→10000< / li >
< li > < code > db.maxidle< / code > 8→20 (DSpace default is -1, unlimited, but we had set it to 8 earlier)< / li >
< / ul >
2019-11-28 16:30:45 +01:00
< / li >
2020-01-27 15:20:44 +01:00
< li > We will need to adjust this again (as well as the < code > pg_hba.conf< / code > settings) when we deploy tsega’ s REST API< / li >
2019-11-28 16:30:45 +01:00
< li > Whip up a test for Marianne of WLE to be able to show both their Phase I and II research themes in the CGSpace item submission form:< / li >
< / ul >
< p > < img src = "/cgspace-notes/2017/06/wle-theme-test-a.png" alt = "Test A for displaying the Phase I and II research themes" >
< img src = "/cgspace-notes/2017/06/wle-theme-test-b.png" alt = "Test B for displaying the Phase I and II research themes" > < / p >
2018-02-11 17:28:23 +01:00
< / article >
< / div > <!-- /.blog - main -->
< aside class = "col-sm-3 ml-auto blog-sidebar" >
< section class = "sidebar-module" >
< h4 > Recent Posts< / h4 >
< ol class = "list-unstyled" >
2020-03-02 11:38:10 +01:00
< li > < a href = "/cgspace-notes/2020-03/" > March, 2020< / a > < / li >
2020-02-02 16:15:48 +01:00
< li > < a href = "/cgspace-notes/2020-02/" > February, 2020< / a > < / li >
2020-01-14 19:40:41 +01:00
< li > < a href = "/cgspace-notes/2020-01/" > January, 2020< / a > < / li >
2019-12-01 10:29:49 +01:00
< li > < a href = "/cgspace-notes/2019-12/" > December, 2019< / a > < / li >
2019-11-04 15:41:19 +01:00
< li > < a href = "/cgspace-notes/2019-11/" > November, 2019< / a > < / li >
2018-02-11 17:28:23 +01:00
< / ol >
< / section >
< section class = "sidebar-module" >
< h4 > Links< / h4 >
< ol class = "list-unstyled" >
< li > < a href = "https://cgspace.cgiar.org" > CGSpace< / a > < / li >
< li > < a href = "https://dspacetest.cgiar.org" > DSpace Test< / a > < / li >
< li > < a href = "https://github.com/ilri/DSpace" > CGSpace @ GitHub< / a > < / li >
< / ol >
< / section >
< / aside >
< / div > <!-- /.row -->
< / div > <!-- /.container -->
< footer class = "blog-footer" >
2019-10-11 10:19:42 +02:00
< p dir = "auto" >
2018-02-11 17:28:23 +01:00
Blog template created by < a href = "https://twitter.com/mdo" > @mdo< / a > , ported to Hugo by < a href = 'https://twitter.com/mralanorth' > @mralanorth< / a > .
< / p >
< p >
< a href = "#" > Back to top< / a >
< / p >
< / footer >
< / body >
< / html >
