2018-02-11 17:28:23 +01:00
<!DOCTYPE html>
2019-10-11 10:19:42 +02:00
< html lang = "en" >
2018-02-11 17:28:23 +01:00
< head >
< meta charset = "utf-8" >
< meta name = "viewport" content = "width=device-width, initial-scale=1, shrink-to-fit=no" >
2020-12-06 15:53:29 +01:00
2018-02-11 17:28:23 +01:00
< meta property = "og:title" content = "February, 2016" / >
< meta property = "og:description" content = "2016-02-05
Looking at some DAGRIS data for Abenet Yabowork
Lots of issues with spaces, newlines, etc causing the import to fail
I noticed we have a very interesting list of countries on CGSpace:
Not only are there 49,000 countries, we have some blanks (25)…
Also, lots of things like “ COTE D`LVOIRE” and “ COTE D IVOIRE”
" />
< meta property = "og:type" content = "article" / >
2019-02-02 13:12:57 +01:00
< meta property = "og:url" content = "https://alanorth.github.io/cgspace-notes/2016-02/" / >
2019-08-08 17:10:44 +02:00
< meta property = "article:published_time" content = "2016-02-05T13:18:00+03:00" / >
< meta property = "article:modified_time" content = "2018-03-09T22:10:33+02:00" / >
2018-09-30 07:23:48 +02:00
2020-12-06 15:53:29 +01:00
2018-02-11 17:28:23 +01:00
< meta name = "twitter:card" content = "summary" / >
< meta name = "twitter:title" content = "February, 2016" / >
< meta name = "twitter:description" content = "2016-02-05
Looking at some DAGRIS data for Abenet Yabowork
Lots of issues with spaces, newlines, etc causing the import to fail
I noticed we have a very interesting list of countries on CGSpace:
Not only are there 49,000 countries, we have some blanks (25)…
Also, lots of things like “ COTE D`LVOIRE” and “ COTE D IVOIRE”
"/>
2023-02-09 06:50:54 +01:00
< meta name = "generator" content = "Hugo 0.110.0" >
2018-02-11 17:28:23 +01:00
< script type = "application/ld+json" >
{
"@context": "http://schema.org",
"@type": "BlogPosting",
"headline": "February, 2016",
2020-04-02 09:55:42 +02:00
"url": "https://alanorth.github.io/cgspace-notes/2016-02/",
2018-04-30 18:05:39 +02:00
"wordCount": "1657",
2019-10-11 10:19:42 +02:00
"datePublished": "2016-02-05T13:18:00+03:00",
"dateModified": "2018-03-09T22:10:33+02:00",
2018-02-11 17:28:23 +01:00
"author": {
"@type": "Person",
"name": "Alan Orth"
},
"keywords": "Notes"
}
< / script >
< link rel = "canonical" href = "https://alanorth.github.io/cgspace-notes/2016-02/" >
< title > February, 2016 | CGSpace Notes< / title >
2019-10-11 10:19:42 +02:00
2018-02-11 17:28:23 +01:00
<!-- combined, minified CSS -->
2020-01-23 19:19:38 +01:00
2022-09-12 10:35:57 +02:00
< link href = "https://alanorth.github.io/cgspace-notes/css/style.c6ba80bc50669557645abe05f86b73cc5af84408ed20f1551a267bc19ece8228.css" rel = "stylesheet" integrity = "sha256-xrqAvFBmlVdkWr4F+GtzzFr4RAjtIPFVGiZ7wZ7Ogig=" crossorigin = "anonymous" >
2019-10-11 10:19:42 +02:00
2018-02-11 17:28:23 +01:00
2020-01-28 11:01:42 +01:00
<!-- minified Font Awesome for SVG icons -->
2021-09-28 09:32:32 +02:00
< script defer src = "https://alanorth.github.io/cgspace-notes/js/fontawesome.min.f5072c55a0721857184db93a50561d7dc13975b4de2e19db7f81eb5f3fa57270.js" integrity = "sha256-9QcsVaByGFcYTbk6UFYdfcE5dbTeLhnbf4HrXz+lcnA=" crossorigin = "anonymous" > < / script >
2020-01-28 11:01:42 +01:00
2019-04-14 15:59:47 +02:00
<!-- RSS 2.0 feed -->
2018-02-11 17:28:23 +01:00
< / head >
< body >
< div class = "blog-masthead" >
< div class = "container" >
< nav class = "nav blog-nav" >
< a class = "nav-link " href = "https://alanorth.github.io/cgspace-notes/" > Home< / a >
< / nav >
< / div >
< / div >
2018-12-19 12:20:39 +01:00
2018-02-11 17:28:23 +01:00
< header class = "blog-header" >
< div class = "container" >
2019-10-11 10:19:42 +02:00
< h1 class = "blog-title" dir = "auto" > < a href = "https://alanorth.github.io/cgspace-notes/" rel = "home" > CGSpace Notes< / a > < / h1 >
< p class = "lead blog-description" dir = "auto" > Documenting day-to-day work on the < a href = "https://cgspace.cgiar.org" > CGSpace< / a > repository.< / p >
2018-02-11 17:28:23 +01:00
< / div >
< / header >
2018-12-19 12:20:39 +01:00
2018-02-11 17:28:23 +01:00
< div class = "container" >
< div class = "row" >
< div class = "col-sm-8 blog-main" >
< article class = "blog-post" >
< header >
2019-10-11 10:19:42 +02:00
< h2 class = "blog-post-title" dir = "auto" > < a href = "https://alanorth.github.io/cgspace-notes/2016-02/" > February, 2016< / a > < / h2 >
2020-11-16 09:54:00 +01:00
< p class = "blog-post-meta" >
< time datetime = "2016-02-05T13:18:00+03:00" > Fri Feb 05, 2016< / time >
in
2018-02-11 17:28:23 +01:00
2022-06-23 07:40:53 +02:00
< span class = "fas fa-tag" aria-hidden = "true" > < / span > < a href = "/tags/notes/" rel = "tag" > Notes< / a >
2018-02-11 17:28:23 +01:00
< / p >
< / header >
2019-12-17 13:49:24 +01:00
< h2 id = "2016-02-05" > 2016-02-05< / h2 >
2018-02-11 17:28:23 +01:00
< ul >
< li > Looking at some DAGRIS data for Abenet Yabowork< / li >
< li > Lots of issues with spaces, newlines, etc causing the import to fail< / li >
< li > I noticed we have a very < em > interesting< / em > list of countries on CGSpace:< / li >
< / ul >
2019-11-28 16:30:45 +01:00
< p > < img src = "/cgspace-notes/2016/02/cgspace-countries.png" alt = "CGSpace country list" > < / p >
2018-02-11 17:28:23 +01:00
< ul >
< li > Not only are there 49,000 countries, we have some blanks (25)… < / li >
< li > Also, lots of things like “ COTE D`LVOIRE” and “ COTE D IVOIRE” < / li >
< / ul >
2019-12-17 13:49:24 +01:00
< h2 id = "2016-02-06" > 2016-02-06< / h2 >
2018-02-11 17:28:23 +01:00
< ul >
< li > Found a way to get items with null/empty metadata values from SQL< / li >
2019-11-28 16:30:45 +01:00
< li > First, find the < code > metadata_field_id< / code > for the field you want from the < code > metadatafieldregistry< / code > table:< / li >
< / ul >
2021-09-13 15:21:16 +02:00
< pre tabindex = "0" > < code > dspacetest=# select * from metadatafieldregistry;
2019-11-28 16:30:45 +01:00
< / code > < / pre > < ul >
< li > In this case our country field is 78< / li >
< li > Now find all resources with type 2 (item) that have null/empty values for that field:< / li >
< / ul >
2022-03-04 13:30:06 +01:00
< pre tabindex = "0" > < code > dspacetest=# select resource_id from metadatavalue where resource_type_id=2 and metadata_field_id=78 and (text_value=' ' OR text_value IS NULL);
2019-11-28 16:30:45 +01:00
< / code > < / pre > < ul >
< li > Then you can find the handle that owns it from its < code > resource_id< / code > :< / li >
< / ul >
2022-03-04 13:30:06 +01:00
< pre tabindex = "0" > < code > dspacetest=# select handle from item, handle where handle.resource_id = item.item_id AND item.item_id = ' 22678' ;
2019-11-28 16:30:45 +01:00
< / code > < / pre > < ul >
2020-01-27 15:20:44 +01:00
< li > It’ s 25 items so editing in the web UI is annoying, let’ s try SQL!< / li >
2019-11-28 16:30:45 +01:00
< / ul >
2022-03-04 13:30:06 +01:00
< pre tabindex = "0" > < code > dspacetest=# delete from metadatavalue where metadata_field_id=78 and text_value=' ' ;
2018-02-11 17:28:23 +01:00
DELETE 25
2019-11-28 16:30:45 +01:00
< / code > < / pre > < ul >
< li > After that perhaps a regular < code > dspace index-discovery< / code > (no -b) < em > should< / em > suffice… < / li >
< li > Hmm, I indexed, cleared the Cocoon cache, and restarted Tomcat but the 25 “ |||” countries are still there< / li >
< li > Maybe I need to do a full re-index… < / li >
< li > Yep! The full re-index seems to work.< / li >
< li > Process the empty countries on CGSpace< / li >
2018-02-11 17:28:23 +01:00
< / ul >
2019-12-17 13:49:24 +01:00
< h2 id = "2016-02-07" > 2016-02-07< / h2 >
2018-02-11 17:28:23 +01:00
< ul >
2020-01-27 15:20:44 +01:00
< li > Working on cleaning up Abenet’ s DAGRIS data with OpenRefine< / li >
2018-02-11 17:28:23 +01:00
< li > I discovered two really nice functions in OpenRefine: < code > value.trim()< / code > and < code > value.escape(" javascript" )< / code > which shows whitespace characters like < code > \r\n< / code > !< / li >
< li > For some reason when you import an Excel file into OpenRefine it exports dates like 1949 to 1949.0 in the CSV< / li >
< li > I re-import the resulting CSV and run a GREL on the date issued column: < code > value.replace(" \.0" , " " )< / code > < / li >
< li > I need to start running DSpace in Mac OS X instead of a Linux VM< / li >
2019-11-28 16:30:45 +01:00
< li > Install PostgreSQL from homebrew, then configure and import CGSpace database dump:< / li >
< / ul >
2021-09-13 15:21:16 +02:00
< pre tabindex = "0" > < code > $ postgres -D /opt/brew/var/postgres
2018-02-11 17:28:23 +01:00
$ createuser --superuser postgres
$ createuser --pwprompt dspacetest
$ createdb -O dspacetest --encoding=UNICODE dspacetest
$ psql postgres
postgres=# alter user dspacetest createuser;
postgres=# \q
$ pg_restore -O -U dspacetest -d dspacetest ~/Downloads/cgspace_2016-02-07.backup
$ psql postgres
postgres=# alter user dspacetest nocreateuser;
postgres=# \q
$ vacuumdb dspacetest
$ psql -U dspacetest -f ~/src/git/DSpace/dspace/etc/postgres/update-sequences.sql dspacetest -h localhost
2019-11-28 16:30:45 +01:00
< / code > < / pre > < ul >
2020-01-27 15:20:44 +01:00
< li > After building and running a < code > fresh_install< / code > I symlinked the webapps into Tomcat’ s webapps folder:< / li >
2019-11-28 16:30:45 +01:00
< / ul >
2021-09-13 15:21:16 +02:00
< pre tabindex = "0" > < code > $ mv /opt/brew/Cellar/tomcat/8.0.30/libexec/webapps/ROOT /opt/brew/Cellar/tomcat/8.0.30/libexec/webapps/ROOT.orig
2018-02-11 17:28:23 +01:00
$ ln -sfv ~/dspace/webapps/xmlui /opt/brew/Cellar/tomcat/8.0.30/libexec/webapps/ROOT
$ ln -sfv ~/dspace/webapps/rest /opt/brew/Cellar/tomcat/8.0.30/libexec/webapps/rest
$ ln -sfv ~/dspace/webapps/jspui /opt/brew/Cellar/tomcat/8.0.30/libexec/webapps/jspui
$ ln -sfv ~/dspace/webapps/oai /opt/brew/Cellar/tomcat/8.0.30/libexec/webapps/oai
$ ln -sfv ~/dspace/webapps/solr /opt/brew/Cellar/tomcat/8.0.30/libexec/webapps/solr
$ /opt/brew/Cellar/tomcat/8.0.30/bin/catalina start
2019-11-28 16:30:45 +01:00
< / code > < / pre > < ul >
< li > Add CATALINA_OPTS in < code > /opt/brew/Cellar/tomcat/8.0.30/libexec/bin/setenv.sh< / code > , as this script is sourced by the < code > catalina< / code > startup script< / li >
< li > For example:< / li >
< / ul >
2022-03-04 13:30:06 +01:00
< pre tabindex = "0" > < code > CATALINA_OPTS=" -Djava.awt.headless=true -Xms2048m -Xmx2048m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -Dfile.encoding=UTF-8"
2019-11-28 16:30:45 +01:00
< / code > < / pre > < ul >
< li > After verifying that the site is working, start a full index:< / li >
2019-05-05 15:45:12 +02:00
< / ul >
2021-09-13 15:21:16 +02:00
< pre tabindex = "0" > < code > $ ~/dspace/bin/dspace index-discovery -b
2019-12-17 13:49:24 +01:00
< / code > < / pre > < h2 id = "2016-02-08" > 2016-02-08< / h2 >
2018-02-11 17:28:23 +01:00
< ul >
< li > Finish cleaning up and importing ~400 DAGRIS items into CGSpace< / li >
2020-01-27 15:20:44 +01:00
< li > Whip up some quick CSS to make the button in the submission workflow use the XMLUI theme’ s brand colors (< a href = "https://github.com/ilri/DSpace/issues/154" > #154< / a > )< / li >
2018-02-11 17:28:23 +01:00
< / ul >
2019-11-28 16:30:45 +01:00
< p > < img src = "/cgspace-notes/2016/02/submit-button-ilri.png" alt = "ILRI submission buttons" >
< img src = "/cgspace-notes/2016/02/submit-button-drylands.png" alt = "Drylands submission buttons" > < / p >
2019-12-17 13:49:24 +01:00
< h2 id = "2016-02-09" > 2016-02-09< / h2 >
2018-02-11 17:28:23 +01:00
< ul >
< li > Re-sync DSpace Test with CGSpace< / li >
< li > Help Sisay with OpenRefine< / li >
2020-01-27 15:20:44 +01:00
< li > Enable HTTPS on DSpace Test using Let’ s Encrypt:< / li >
2019-11-28 16:30:45 +01:00
< / ul >
2021-09-13 15:21:16 +02:00
< pre tabindex = "0" > < code > $ cd ~/src/git
2018-02-11 17:28:23 +01:00
$ git clone https://github.com/letsencrypt/letsencrypt
$ cd letsencrypt
$ sudo service nginx stop
# add port 443 to firewall rules
$ ./letsencrypt-auto certonly --standalone -d dspacetest.cgiar.org
$ sudo service nginx start
$ ansible-playbook dspace.yml -l linode02 -t nginx,firewall -u aorth --ask-become-pass
2019-11-28 16:30:45 +01:00
< / code > < / pre > < ul >
< li > We should install it in /opt/letsencrypt and then script the renewal script, but first we have to wire up some variables and template stuff based on the script here: < a href = "https://letsencrypt.org/howitworks/" > https://letsencrypt.org/howitworks/< / a > < / li >
< li > I had to export some CIAT items that were being cleaned up on the test server and I noticed their < code > dc.contributor.author< / code > fields have DSpace 5 authority index UUIDs… < / li >
< li > To clean those up in OpenRefine I used this GREL expression: < code > value.replace(/::\w{8}-\w{4}-\w{4}-\w{4}-\w{12}::600/," " )< / code > < / li >
< li > Getting more and more hangs on DSpace Test, seemingly random but also during CSV import< / li >
2020-01-27 15:20:44 +01:00
< li > Logs don’ t always show anything right when it fails, but eventually one of these appears:< / li >
2019-11-28 16:30:45 +01:00
< / ul >
2021-09-13 15:21:16 +02:00
< pre tabindex = "0" > < code > org.dspace.discovery.SearchServiceException: Error while processing facet fields: java.lang.OutOfMemoryError: Java heap space
2019-11-28 16:30:45 +01:00
< / code > < / pre > < ul >
< li > or< / li >
< / ul >
2021-09-13 15:21:16 +02:00
< pre tabindex = "0" > < code > Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
2019-11-28 16:30:45 +01:00
< / code > < / pre > < ul >
2020-01-27 15:20:44 +01:00
< li > Right now DSpace Test’ s Tomcat heap is set to 1536m and we have quite a bit of free RAM:< / li >
2019-11-28 16:30:45 +01:00
< / ul >
2021-09-13 15:21:16 +02:00
< pre tabindex = "0" > < code > # free -m
2019-11-28 16:30:45 +01:00
total used free shared buffers cached
2018-02-11 17:28:23 +01:00
Mem: 3950 3902 48 9 37 1311
-/+ buffers/cache: 2552 1397
Swap: 255 57 198
2019-11-28 16:30:45 +01:00
< / code > < / pre > < ul >
2020-01-27 15:20:44 +01:00
< li > So I’ ll bump up the Tomcat heap to 2048 (CGSpace production server is using 3GB)< / li >
2018-02-11 17:28:23 +01:00
< / ul >
2019-12-17 13:49:24 +01:00
< h2 id = "2016-02-11" > 2016-02-11< / h2 >
2018-02-11 17:28:23 +01:00
< ul >
< li > Massaging some CIAT data in OpenRefine< / li >
< li > There are 1200 records that have PDFs, and will need to be imported into CGSpace< / li >
2019-11-28 16:30:45 +01:00
< li > I created a < code > filename< / code > column based on the < code > dc.identifier.url< / code > column using the following transform:< / li >
< / ul >
2022-03-04 13:30:06 +01:00
< pre tabindex = "0" > < code > value.split(' /' )[-1]
2019-11-28 16:30:45 +01:00
< / code > < / pre > < ul >
< li > Then I wrote a tool called < a href = "https://gist.github.com/alanorth/2206f24483fe5f0454fc" > < code > generate-thumbnails.py< / code > < / a > to download the PDFs and generate thumbnails for them, for example:< / li >
< / ul >
2021-09-13 15:21:16 +02:00
< pre tabindex = "0" > < code > $ ./generate-thumbnails.py ciat-reports.csv
2018-02-11 17:28:23 +01:00
Processing 64661.pdf
> Downloading 64661.pdf
> Creating thumbnail for 64661.pdf
Processing 64195.pdf
> Downloading 64195.pdf
> Creating thumbnail for 64195.pdf
2019-12-17 13:49:24 +01:00
< / code > < / pre > < h2 id = "2016-02-12" > 2016-02-12< / h2 >
2018-02-11 17:28:23 +01:00
< ul >
2020-01-27 15:20:44 +01:00
< li > Looking at CIAT’ s records again, there are some problems with a dozen or so files (out of 1200)< / li >
2018-02-11 17:28:23 +01:00
< li > A few items are using the same exact PDF< / li >
< li > A few items are using HTM or DOC files< / li >
2020-01-27 15:20:44 +01:00
< li > A few items link to PDFs on IFPRI’ s e-Library or Research Gate< / li >
2018-02-11 17:28:23 +01:00
< li > A few items have no item< / li >
2020-01-27 15:20:44 +01:00
< li > Also, I’ m not sure if we import these items, will be remove the < code > dc.identifier.url< / code > field from the records?< / li >
2018-02-11 17:28:23 +01:00
< / ul >
2019-12-17 13:49:24 +01:00
< h2 id = "2016-02-12-1" > 2016-02-12< / h2 >
2018-02-11 17:28:23 +01:00
< ul >
2020-01-27 15:20:44 +01:00
< li > Looking at CIAT’ s records again, there are some files linking to PDFs on Slide Share, Embrapa, UEA UK, and Condesan, so I’ m not sure if we can use those< / li >
2019-11-28 16:30:45 +01:00
< li > 265 items have dirty, URL-encoded filenames:< / li >
< / ul >
2022-03-04 13:30:06 +01:00
< pre tabindex = "0" > < code > $ ls | grep -c -E " %"
2018-02-11 17:28:23 +01:00
265
2019-11-28 16:30:45 +01:00
< / code > < / pre > < ul >
< li > I suggest that we import ~850 or so of the clean ones first, then do the rest after I can find a clean/reliable way to decode the filenames< / li >
< li > This python2 snippet seems to work in the CLI, but not so well in OpenRefine:< / li >
< / ul >
2022-03-04 13:30:06 +01:00
< pre tabindex = "0" > < code > $ python -c " import urllib, sys; print urllib.unquote(sys.argv[1])" CIAT_COLOMBIA_000169_T%C3%A9cnicas_para_el_aislamiento_y_cultivo_de_protoplastos_de_yuca.pdf
2018-02-11 17:28:23 +01:00
CIAT_COLOMBIA_000169_Técnicas_para_el_aislamiento_y_cultivo_de_protoplastos_de_yuca.pdf
2019-11-28 16:30:45 +01:00
< / code > < / pre > < ul >
< li > Merge pull requests for submission form theming (< a href = "https://github.com/ilri/DSpace/pull/178" > #178< / a > ) and missing center subjects in XMLUI item views (< a href = "https://github.com/ilri/DSpace/pull/176" > #176< / a > )< / li >
< li > They will be deployed on CGSpace the next time I re-deploy< / li >
2018-02-11 17:28:23 +01:00
< / ul >
2019-12-17 13:49:24 +01:00
< h2 id = "2016-02-16" > 2016-02-16< / h2 >
2018-02-11 17:28:23 +01:00
< ul >
2019-11-28 16:30:45 +01:00
< li > Turns out OpenRefine has an unescape function!< / li >
< / ul >
2022-03-04 13:30:06 +01:00
< pre tabindex = "0" > < code > value.unescape(" url" )
2019-11-28 16:30:45 +01:00
< / code > < / pre > < ul >
< li > This turns the URLs into human-readable versions that we can use as proper filenames< / li >
< li > Run web server and system updates on DSpace Test and reboot< / li >
2020-01-27 15:20:44 +01:00
< li > To merge < code > dc.identifier.url< / code > and < code > dc.identifier.url[]< / code > , rename the second column so it doesn’ t have the brackets, like < code > dc.identifier.url2< / code > < / li >
2019-11-28 16:30:45 +01:00
< li > Then you create a facet for blank values on each column, show the rows that have values for one and not the other, then transform each independently to have the contents of the other, with “ ||” in between< / li >
< li > Work on Python script for parsing and downloading PDF records from < code > dc.identifier.url< / code > < / li >
< li > To get filenames from < code > dc.identifier.url< / code > , create a new column based on this transform: < code > forEach(value.split('||'), v, v.split('/')[-1]).join('||')< / code > < / li >
2022-03-04 13:30:06 +01:00
< li > This also works for records that have multiple URLs (separated by “ ||” )< / li >
2018-02-11 17:28:23 +01:00
< / ul >
2019-12-17 13:49:24 +01:00
< h2 id = "2016-02-17" > 2016-02-17< / h2 >
2018-02-11 17:28:23 +01:00
< ul >
< li > Re-deploy CGSpace, run all system updates, and reboot< / li >
< li > More work on CIAT data, cleaning and doing a last metadata-only import into DSpace Test< / li >
< li > SAFBuilder has a bug preventing it from processing filenames containing more than one underscore< / li >
< li > Need to re-process the filename column to replace multiple underscores with one: < code > value.replace(/_{2,}/, " _" )< / code > < / li >
< / ul >
2019-12-17 13:49:24 +01:00
< h2 id = "2016-02-20" > 2016-02-20< / h2 >
2018-02-11 17:28:23 +01:00
< ul >
2020-01-27 15:20:44 +01:00
< li > Turns out the “ bug” in SAFBuilder isn’ t a bug, it’ s a feature that allows you to encode extra information like the destintion bundle in the filename< / li >
< li > Also, it seems DSpace’ s SAF import tool doesn’ t like importing filenames that have accents in them:< / li >
2019-11-28 16:30:45 +01:00
< / ul >
2021-09-13 15:21:16 +02:00
< pre tabindex = "0" > < code > java.io.FileNotFoundException: /usr/share/tomcat7/SimpleArchiveFormat/item_1021/CIAT_COLOMBIA_000075_Medición_de_palatabilidad_en_forrajes.pdf (No such file or directory)
2019-11-28 16:30:45 +01:00
< / code > < / pre > < ul >
< li > Need to rename files to have no accents or umlauts, etc… < / li >
< li > Useful custom text facet for URLs ending with “ .pdf” : < code > value.endsWith(" .pdf" )< / code > < / li >
2018-02-11 17:28:23 +01:00
< / ul >
2019-12-17 13:49:24 +01:00
< h2 id = "2016-02-22" > 2016-02-22< / h2 >
2018-02-11 17:28:23 +01:00
< ul >
2019-11-28 16:30:45 +01:00
< li > To change Spanish accents to ASCII in OpenRefine:< / li >
< / ul >
2022-03-04 13:30:06 +01:00
< pre tabindex = "0" > < code > value.replace(' ó' ,' o' ).replace(' í' ,' i' ).replace(' á' ,' a' ).replace(' é' ,' e' ).replace(' ñ' ,' n' )
2019-11-28 16:30:45 +01:00
< / code > < / pre > < ul >
< li > But actually, the accents might not be an issue, as I can successfully import files containing Spanish accents on my Mac< / li >
< li > On closer inspection, I can import files with the following names on Linux (DSpace Test):< / li >
< / ul >
2021-09-13 15:21:16 +02:00
< pre tabindex = "0" > < code > Bitstream: tést.pdf
2018-02-11 17:28:23 +01:00
Bitstream: tést señora.pdf
Bitstream: tést señora alimentación.pdf
2019-11-28 16:30:45 +01:00
< / code > < / pre > < ul >
2020-01-27 15:20:44 +01:00
< li > Seems it could be something with the HFS+ filesystem actually, as it’ s not UTF-8 (< a href = "http://www.cio.com/article/2868393/linus-torvalds-apples-hfs-is-probably-the-worst-file-system-ever.html" > it’ s something like UCS-2< / a > )< / li >
< li > HFS+ stores filenames as a string, and filenames with accents get stored as < a href = "https://blog.vrypan.net/2012/11/13/hfsplus-unicode-and-accented-chars/" > character+accent< / a > whereas Linux’ s ext4 stores them as an array of bytes< / li >
< li > Running the SAFBuilder on Mac OS X works if you’ re going to import the resulting bundle on Mac OS X, but if your DSpace is running on Linux you need to run the SAFBuilder there where the filesystem’ s encoding matches< / li >
2018-02-11 17:28:23 +01:00
< / ul >
2019-12-17 13:49:24 +01:00
< h2 id = "2016-02-29" > 2016-02-29< / h2 >
2018-02-11 17:28:23 +01:00
< ul >
2020-01-27 15:20:44 +01:00
< li > Got notified by some CIFOR colleagues that the Google Scholar team had contacted them about CGSpace’ s incorrect ordering of authors in Google Scholar metadata< / li >
2018-02-11 17:28:23 +01:00
< li > Turns out there is a patch, and it was merged in DSpace 5.4: < a href = "https://jira.duraspace.org/browse/DS-2679" > https://jira.duraspace.org/browse/DS-2679< / a > < / li >
2020-01-27 15:20:44 +01:00
< li > I’ ve merged it into our < code > 5_x-prod< / code > branch that is currently based on DSpace 5.1< / li >
2018-02-11 17:28:23 +01:00
< li > We found a bug when a user searches from the homepage, sorts the results, and then tries to click “ View More” in a sidebar facet< / li >
< li > I am not sure what causes it yet, but I opened an issue for it: < a href = "https://github.com/ilri/DSpace/issues/179" > https://github.com/ilri/DSpace/issues/179< / a > < / li >
< li > Have more problems with SAFBuilder on Mac OS X< / li >
2020-01-27 15:20:44 +01:00
< li > Now it doesn’ t recognize description hints in the filename column, like: < code > test.pdf__description:Blah< / code > < / li >
2018-02-11 17:28:23 +01:00
< li > But on Linux it works fine< / li >
2020-01-27 15:20:44 +01:00
< li > Trying to test Atmire’ s series of stats and CUA fixes from January and February, but their branch history is really messy and it’ s hard to see what’ s going on< / li >
< li > Rebasing their branch on top of our production branch results in a broken Tomcat, so I’ m going to tell them to fix their history and make a proper pull request< / li >
2018-02-11 17:28:23 +01:00
< li > Looking at the filenames for the CIAT Reports, some have some really ugly characters, like: < code > '< / code > or < code > ,< / code > or < code > =< / code > or < code > [< / code > or < code > ]< / code > or < code > (< / code > or < code > )< / code > or < code > _.pdf< / code > or < code > ._< / code > etc< / li >
2020-01-27 15:20:44 +01:00
< li > It’ s tricky to parse those things in some programming languages so I’ d rather just get rid of the weird stuff now in OpenRefine:< / li >
2019-11-28 16:30:45 +01:00
< / ul >
2022-03-04 13:30:06 +01:00
< pre tabindex = "0" > < code > value.replace(" ' " ,' ' ).replace(' _=_' ,' _' ).replace(' ,' ,' ' ).replace(' [' ,' ' ).replace(' ]' ,' ' ).replace(' (' ,' ' ).replace(' )' ,' ' ).replace(' _.pdf' ,' .pdf' ).replace(' ._' ,' _' )
2019-11-28 16:30:45 +01:00
< / code > < / pre > < ul >
< li > Finally import the 1127 CIAT items into CGSpace: < a href = "https://cgspace.cgiar.org/handle/10568/35710" > https://cgspace.cgiar.org/handle/10568/35710< / a > < / li >
2020-01-27 15:20:44 +01:00
< li > Re-deploy CGSpace with the Google Scholar fix, but I’ m waiting on the Atmire fixes for now, as the branch history is ugly< / li >
2018-02-11 17:28:23 +01:00
< / ul >
< / article >
< / div > <!-- /.blog - main -->
< aside class = "col-sm-3 ml-auto blog-sidebar" >
< section class = "sidebar-module" >
< h4 > Recent Posts< / h4 >
< ol class = "list-unstyled" >
2023-02-09 06:50:54 +01:00
< li > < a href = "/cgspace-notes/2023-02/" > February, 2023< / a > < / li >
2023-01-01 09:12:13 +01:00
< li > < a href = "/cgspace-notes/2023-01/" > January, 2023< / a > < / li >
2022-12-03 08:46:29 +01:00
< li > < a href = "/cgspace-notes/2022-12/" > December, 2022< / a > < / li >
2022-11-01 20:12:24 +01:00
< li > < a href = "/cgspace-notes/2022-11/" > November, 2022< / a > < / li >
2022-10-01 18:47:37 +02:00
< li > < a href = "/cgspace-notes/2022-10/" > October, 2022< / a > < / li >
2018-02-11 17:28:23 +01:00
< / ol >
< / section >
< section class = "sidebar-module" >
< h4 > Links< / h4 >
< ol class = "list-unstyled" >
< li > < a href = "https://cgspace.cgiar.org" > CGSpace< / a > < / li >
< li > < a href = "https://dspacetest.cgiar.org" > DSpace Test< / a > < / li >
< li > < a href = "https://github.com/ilri/DSpace" > CGSpace @ GitHub< / a > < / li >
< / ol >
< / section >
< / aside >
< / div > <!-- /.row -->
< / div > <!-- /.container -->
< footer class = "blog-footer" >
2019-10-11 10:19:42 +02:00
< p dir = "auto" >
2018-02-11 17:28:23 +01:00
Blog template created by < a href = "https://twitter.com/mdo" > @mdo< / a > , ported to Hugo by < a href = 'https://twitter.com/mralanorth' > @mralanorth< / a > .
< / p >
< p >
< a href = "#" > Back to top< / a >
< / p >
< / footer >
< / body >
< / html >
