mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2025-01-27 05:49:12 +01:00
Add notes for 2019-05-05
This commit is contained in:
@ -15,22 +15,22 @@ Test the DSpace 5.8 module upgrades from Atmire (#378)
|
||||
There seems to be a problem with the CUA and L&R versions in pom.xml because they are using SNAPSHOT and it doesn’t build
|
||||
|
||||
I added the new CCAFS Phase II Project Tag PII-FP1_PACCA2 and merged it into the 5_x-prod branch (#379)
|
||||
I proofed and tested the ILRI author corrections that Peter sent back to me this week:
|
||||
|
||||
I proofed and tested the ILRI author corrections that Peter sent back to me this week:
|
||||
|
||||
$ ./fix-metadata-values.py -i /tmp/2018-05-30-Correct-660-authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t correct -m 3 -n
|
||||
|
||||
|
||||
|
||||
I think a sane proofing workflow in OpenRefine is to apply the custom text facets for check/delete/remove and illegal characters that I developed in March, 2018
|
||||
Time to index ~70,000 items on CGSpace:
|
||||
|
||||
Time to index ~70,000 items on CGSpace:
|
||||
|
||||
$ time schedtool -D -e ionice -c2 -n7 nice -n19 [dspace]/bin/dspace index-discovery -b
|
||||
|
||||
real 74m42.646s
|
||||
user 8m5.056s
|
||||
sys 2m7.289s
|
||||
|
||||
" />
|
||||
<meta property="og:type" content="article" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2018-06/" />
|
||||
@ -48,24 +48,24 @@ Test the DSpace 5.8 module upgrades from Atmire (#378)
|
||||
There seems to be a problem with the CUA and L&R versions in pom.xml because they are using SNAPSHOT and it doesn’t build
|
||||
|
||||
I added the new CCAFS Phase II Project Tag PII-FP1_PACCA2 and merged it into the 5_x-prod branch (#379)
|
||||
I proofed and tested the ILRI author corrections that Peter sent back to me this week:
|
||||
|
||||
I proofed and tested the ILRI author corrections that Peter sent back to me this week:
|
||||
|
||||
$ ./fix-metadata-values.py -i /tmp/2018-05-30-Correct-660-authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t correct -m 3 -n
|
||||
|
||||
|
||||
|
||||
I think a sane proofing workflow in OpenRefine is to apply the custom text facets for check/delete/remove and illegal characters that I developed in March, 2018
|
||||
Time to index ~70,000 items on CGSpace:
|
||||
|
||||
Time to index ~70,000 items on CGSpace:
|
||||
|
||||
$ time schedtool -D -e ionice -c2 -n7 nice -n19 [dspace]/bin/dspace index-discovery -b
|
||||
|
||||
real 74m42.646s
|
||||
user 8m5.056s
|
||||
sys 2m7.289s
|
||||
|
||||
"/>
|
||||
<meta name="generator" content="Hugo 0.55.3" />
|
||||
<meta name="generator" content="Hugo 0.55.5" />
|
||||
|
||||
|
||||
|
||||
@ -153,23 +153,23 @@ sys 2m7.289s
|
||||
<li>There seems to be a problem with the CUA and L&R versions in <code>pom.xml</code> because they are using SNAPSHOT and it doesn’t build</li>
|
||||
</ul></li>
|
||||
<li>I added the new CCAFS Phase II Project Tag <code>PII-FP1_PACCA2</code> and merged it into the <code>5_x-prod</code> branch (<a href="https://github.com/ilri/DSpace/pull/379">#379</a>)</li>
|
||||
<li>I proofed and tested the ILRI author corrections that Peter sent back to me this week:</li>
|
||||
</ul>
|
||||
|
||||
<li><p>I proofed and tested the ILRI author corrections that Peter sent back to me this week:</p>
|
||||
|
||||
<pre><code>$ ./fix-metadata-values.py -i /tmp/2018-05-30-Correct-660-authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t correct -m 3 -n
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>I think a sane proofing workflow in OpenRefine is to apply the custom text facets for check/delete/remove and illegal characters that I developed in <a href="/cgspace-notes/2018-03/">March, 2018</a></li>
|
||||
<li>Time to index ~70,000 items on CGSpace:</li>
|
||||
</ul>
|
||||
<li><p>I think a sane proofing workflow in OpenRefine is to apply the custom text facets for check/delete/remove and illegal characters that I developed in <a href="/cgspace-notes/2018-03/">March, 2018</a></p></li>
|
||||
|
||||
<li><p>Time to index ~70,000 items on CGSpace:</p>
|
||||
|
||||
<pre><code>$ time schedtool -D -e ionice -c2 -n7 nice -n19 [dspace]/bin/dspace index-discovery -b
|
||||
|
||||
real 74m42.646s
|
||||
user 8m5.056s
|
||||
sys 2m7.289s
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2018-06-06">2018-06-06</h2>
|
||||
|
||||
@ -198,32 +198,29 @@ sys 2m7.289s
|
||||
<li>Universit F lix Houphouet-Boigny</li>
|
||||
</ul></li>
|
||||
<li>I uploaded fixes for all those now, but I will continue with the rest of the data later</li>
|
||||
<li>Regarding the SQL migration errors, Atmire told me I need to run some migrations manually in PostgreSQL:</li>
|
||||
</ul>
|
||||
|
||||
<li><p>Regarding the SQL migration errors, Atmire told me I need to run some migrations manually in PostgreSQL:</p>
|
||||
|
||||
<pre><code>delete from schema_version where version = '5.6.2015.12.03.2';
|
||||
update schema_version set version = '5.6.2015.12.03.2' where version = '5.5.2015.12.03.2';
|
||||
update schema_version set version = '5.8.2015.12.03.3' where version = '5.5.2015.12.03.3';
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>And then I need to ignore the ignored ones:</li>
|
||||
</ul>
|
||||
<li><p>And then I need to ignore the ignored ones:</p>
|
||||
|
||||
<pre><code>$ ~/dspace/bin/dspace database migrate ignored
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>Now DSpace starts up properly!</li>
|
||||
<li>Gabriela from CIP got back to me about the author names we were correcting on CGSpace</li>
|
||||
<li>I did a quick sanity check on them and then did a test import with my <a href="https://gist.github.com/alanorth/df92cbfb54d762ba21b28f7cd83b6897"><code>fix-metadata-value.py</code></a> script:</li>
|
||||
</ul>
|
||||
<li><p>Now DSpace starts up properly!</p></li>
|
||||
|
||||
<li><p>Gabriela from CIP got back to me about the author names we were correcting on CGSpace</p></li>
|
||||
|
||||
<li><p>I did a quick sanity check on them and then did a test import with my <a href="https://gist.github.com/alanorth/df92cbfb54d762ba21b28f7cd83b6897"><code>fix-metadata-value.py</code></a> script:</p>
|
||||
|
||||
<pre><code>$ ./fix-metadata-values.py -i /tmp/2018-06-08-CIP-Authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t correct -m 3
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>I will apply them on CGSpace tomorrow I think…</li>
|
||||
<li><p>I will apply them on CGSpace tomorrow I think…</p></li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2018-06-09">2018-06-09</h2>
|
||||
@ -238,17 +235,18 @@ update schema_version set version = '5.8.2015.12.03.3' where version = '5.5.2015
|
||||
|
||||
<ul>
|
||||
<li>I spent some time removing the Atmire Metadata Quality Module (MQM) from the proposed DSpace 5.8 changes</li>
|
||||
<li>After removing all code mentioning MQM, mqm, metadata-quality, batchedit, duplicatechecker, etc, I think I got most of it removed, but there is a Spring error during Tomcat startup:</li>
|
||||
</ul>
|
||||
|
||||
<pre><code> INFO [org.dspace.servicemanager.DSpaceServiceManager] Shutdown DSpace core service manager
|
||||
<li><p>After removing all code mentioning MQM, mqm, metadata-quality, batchedit, duplicatechecker, etc, I think I got most of it removed, but there is a Spring error during Tomcat startup:</p>
|
||||
|
||||
<pre><code>INFO [org.dspace.servicemanager.DSpaceServiceManager] Shutdown DSpace core service manager
|
||||
Failed to startup the DSpace Service Manager: failure starting up spring service manager: Error creating bean with name 'org.dspace.servicemanager.spring.DSpaceBeanPostProcessor#0' defined in class path resource [spring/spring-dspace-applicationContext.xml]: Unsatisfied dependency expressed through constructor argument with index 0 of type [org.dspace.servicemanager.config.DSpaceConfigurationService]: : Cannot find class [com.atmire.dspace.discovery.ItemCollectionPlugin] for bean with name 'itemCollectionPlugin' defined in file [/home/aorth/dspace/config/spring/api/discovery.xml];
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>I can fix this by commenting out the <code>ItemCollectionPlugin</code> line of <code>discovery.xml</code>, but from looking at the git log I’m not actually sure if that is related to MQM or not</li>
|
||||
<li>I will have to ask Atmire</li>
|
||||
<li>I continued to look at Sisay’s IITA records from last week
|
||||
<li><p>I can fix this by commenting out the <code>ItemCollectionPlugin</code> line of <code>discovery.xml</code>, but from looking at the git log I’m not actually sure if that is related to MQM or not</p></li>
|
||||
|
||||
<li><p>I will have to ask Atmire</p></li>
|
||||
|
||||
<li><p>I continued to look at Sisay’s IITA records from last week</p>
|
||||
|
||||
<ul>
|
||||
<li>I normalized all DOIs to use HTTPS and “doi.org” instead of “dx.doi.org”</li>
|
||||
@ -263,7 +261,8 @@ Failed to startup the DSpace Service Manager: failure starting up spring service
|
||||
<li>dÕpassÕ</li>
|
||||
<li>Also the abstracts have missing accents, ie “recherche sur le d veloppement”</li>
|
||||
</ul></li>
|
||||
<li>I will have to tell IITA people to redo these entirely I think…</li>
|
||||
|
||||
<li><p>I will have to tell IITA people to redo these entirely I think…</p></li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2018-06-11">2018-06-11</h2>
|
||||
@ -301,7 +300,8 @@ Failed to startup the DSpace Service Manager: failure starting up spring service
|
||||
<li>The style sheet obfuscates the data, but if you look at the source it is all there, including information about pagination of results</li>
|
||||
<li>Regarding Udana’s Book Chapters and Reports on DSpace Test last week, Abenet told him to fix some character encoding and CRP issues, then I told him I’d check them after that</li>
|
||||
<li>The latest batch of IITA’s 200 records (based on Abenet’s version <code>Mercy1805_AY.xls</code>) are now in the <a href="https://dspacetest.cgiar.org/handle/10568/96071">IITA_Jan_9_II_Ab</a> collection</li>
|
||||
<li>So here are some corrections:
|
||||
|
||||
<li><p>So here are some corrections:</p>
|
||||
|
||||
<ul>
|
||||
<li>use of Unicode smart quote (hex 2019) in countries and affiliations, for example “COTE D’IVOIRE” and “Institut d’Economic Rurale, Mali”</li>
|
||||
@ -338,27 +338,27 @@ Failed to startup the DSpace Service Manager: failure starting up spring service
|
||||
<li>“LEGUMINOUS COVER CROP” and “LEGUMINOUS COVER CROPS”</li>
|
||||
<li>“MATÉRIEL DE PLANTATION” and “MATÉRIELS DE PLANTATION”</li>
|
||||
<li>I noticed that some records do have encoding errors in the <code>dc.description.abstract</code> field, but only four of them so probably not from Abenet’s handling of the XLS file</li>
|
||||
<li>Based on manually eyeballing the text I used a custom text facet with this GREL to identify the records:</li>
|
||||
</ul></li>
|
||||
</ul>
|
||||
|
||||
<li><p>Based on manually eyeballing the text I used a custom text facet with this GREL to identify the records:</p>
|
||||
|
||||
<pre><code>or(
|
||||
value.contains('€'),
|
||||
value.contains('6g'),
|
||||
value.contains('6m'),
|
||||
value.contains('6d'),
|
||||
value.contains('6e')
|
||||
value.contains('€'),
|
||||
value.contains('6g'),
|
||||
value.contains('6m'),
|
||||
value.contains('6d'),
|
||||
value.contains('6e')
|
||||
)
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>So IITA should double check the abstracts for these:
|
||||
<li><p>So IITA should double check the abstracts for these:</p></li>
|
||||
|
||||
<ul>
|
||||
<li><a href="https://dspacetest.cgiar.org/10568/96184">https://dspacetest.cgiar.org/10568/96184</a></li>
|
||||
<li><a href="https://dspacetest.cgiar.org/10568/96141">https://dspacetest.cgiar.org/10568/96141</a></li>
|
||||
<li><a href="https://dspacetest.cgiar.org/10568/96118">https://dspacetest.cgiar.org/10568/96118</a></li>
|
||||
<li><a href="https://dspacetest.cgiar.org/10568/96113">https://dspacetest.cgiar.org/10568/96113</a></li>
|
||||
<li><p><a href="https://dspacetest.cgiar.org/10568/96184">https://dspacetest.cgiar.org/10568/96184</a></p></li>
|
||||
|
||||
<li><p><a href="https://dspacetest.cgiar.org/10568/96141">https://dspacetest.cgiar.org/10568/96141</a></p></li>
|
||||
|
||||
<li><p><a href="https://dspacetest.cgiar.org/10568/96118">https://dspacetest.cgiar.org/10568/96118</a></p></li>
|
||||
|
||||
<li><p><a href="https://dspacetest.cgiar.org/10568/96113">https://dspacetest.cgiar.org/10568/96113</a></p></li>
|
||||
</ul></li>
|
||||
</ul>
|
||||
|
||||
@ -366,38 +366,33 @@ Failed to startup the DSpace Service Manager: failure starting up spring service
|
||||
|
||||
<ul>
|
||||
<li>Elizabeth from CIAT contacted me to ask if I could add ORCID identifiers to all of Robin Buruchara’s items</li>
|
||||
<li>I used my <a href="https://gist.githubusercontent.com/alanorth/a49d85cd9c5dea89cddbe809813a7050/raw/f67b6e45a9a940732882ae4bb26897a9b245ef31/add-orcid-identifiers-csv.py">add-orcid-identifiers-csv.py</a> script:</li>
|
||||
</ul>
|
||||
|
||||
<li><p>I used my <a href="https://gist.githubusercontent.com/alanorth/a49d85cd9c5dea89cddbe809813a7050/raw/f67b6e45a9a940732882ae4bb26897a9b245ef31/add-orcid-identifiers-csv.py">add-orcid-identifiers-csv.py</a> script:</p>
|
||||
|
||||
<pre><code>$ ./add-orcid-identifiers-csv.py -i 2018-06-13-Robin-Buruchara.csv -db dspace -u dspace -p 'fuuu'
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>The contents of <code>2018-06-13-Robin-Buruchara.csv</code> were:</li>
|
||||
</ul>
|
||||
<li><p>The contents of <code>2018-06-13-Robin-Buruchara.csv</code> were:</p>
|
||||
|
||||
<pre><code>dc.contributor.author,cg.creator.id
|
||||
"Buruchara, Robin",Robin Buruchara: 0000-0003-0934-1218
|
||||
"Buruchara, Robin A.",Robin Buruchara: 0000-0003-0934-1218
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>On a hunch I checked to see if CGSpace’s bitstream cleanup was working properly and of course it’s broken:</li>
|
||||
</ul>
|
||||
<li><p>On a hunch I checked to see if CGSpace’s bitstream cleanup was working properly and of course it’s broken:</p>
|
||||
|
||||
<pre><code>$ dspace cleanup -v
|
||||
...
|
||||
Error: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
|
||||
Detail: Key (bitstream_id)=(152402) is still referenced from table "bundle".
|
||||
</code></pre>
|
||||
Detail: Key (bitstream_id)=(152402) is still referenced from table "bundle".
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>As always, the solution is to delete that ID manually in PostgreSQL:</li>
|
||||
</ul>
|
||||
<li><p>As always, the solution is to delete that ID manually in PostgreSQL:</p>
|
||||
|
||||
<pre><code>$ psql dspace -c 'update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (152402);'
|
||||
UPDATE 1
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2018-06-14">2018-06-14</h2>
|
||||
|
||||
@ -411,39 +406,47 @@ UPDATE 1
|
||||
<h2 id="2018-06-24">2018-06-24</h2>
|
||||
|
||||
<ul>
|
||||
<li>I was restoring a PostgreSQL dump on my test machine and found a way to restore the CGSpace dump as the <code>postgres</code> user, but have the owner of the schema be the <code>dspacetest</code> user:</li>
|
||||
</ul>
|
||||
<li><p>I was restoring a PostgreSQL dump on my test machine and found a way to restore the CGSpace dump as the <code>postgres</code> user, but have the owner of the schema be the <code>dspacetest</code> user:</p>
|
||||
|
||||
<pre><code>$ dropdb -h localhost -U postgres dspacetest
|
||||
$ createdb -h localhost -U postgres -O dspacetest --encoding=UNICODE dspacetest
|
||||
$ psql -h localhost -U postgres dspacetest -c 'alter user dspacetest superuser;'
|
||||
$ pg_restore -h localhost -U postgres -d dspacetest -O --role=dspacetest -h localhost /tmp/cgspace_2018-06-24.backup
|
||||
$ psql -h localhost -U postgres dspacetest -c 'alter user dspacetest nosuperuser;'
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>The <code>-O</code> option to <code>pg_restore</code> makes the import process ignore ownership specified in the dump itself, and instead makes the schema owned by the user doing the restore</li>
|
||||
<li>I always prefer to use the <code>postgres</code> user locally because it’s just easier than remembering the <code>dspacetest</code> user’s password, but then I couldn’t figure out why the resulting schema was owned by <code>postgres</code></li>
|
||||
<li>So with this you connect as the <code>postgres</code> superuser and then switch roles to <code>dspacetest</code> (also, make sure this user has <code>superuser</code> privileges before the restore)</li>
|
||||
<li>Last week Linode emailed me to say that our Linode 8192 instance used for DSpace Test qualified for an upgrade</li>
|
||||
<li>Apparently they announced some <a href="https://blog.linode.com/2018/05/17/updated-linode-plans-new-larger-linodes/">upgrades to most of their plans in 2018-05</a></li>
|
||||
<li>After the upgrade I see we have more disk space available in the instance’s dashboard, so I shut the instance down and resized it from 98GB to 160GB</li>
|
||||
<li>The resize was very quick (less than one minute) and after booting the instance back up I now have 160GB for the root filesystem!</li>
|
||||
<li>I will move the DSpace installation directory back to the root file system and delete the extra 300GB block storage, as it was actually kinda slow when we put Solr there and now we don’t actually need it anymore because running the production Solr on this instance didn’t work well with 8GB of RAM</li>
|
||||
<li>Also, the larger instance we’re using for CGSpace will go from 24GB of RAM to 32, and will also get a storage increase from 320GB to 640GB… that means we don’t need to consider using block storage right now!</li>
|
||||
<li>The smaller instances get increased storage and network speed but I doubt many are actually using much of their current allocations so we probably don’t need to bother with upgrading them</li>
|
||||
<li>Last week Abenet asked if we could add <code>dc.language.iso</code> to the advanced search filters</li>
|
||||
<li>There is already a search filter for this field defined in <code>discovery.xml</code> but we aren’t using it, so I quickly enabled and tested it, then merged it to the <code>5_x-prod</code> branch (<a href="https://github.com/ilri/DSpace/pull/380">#380</a>)</li>
|
||||
<li>Back to testing the DSpace 5.8 changes from Atmire, I had another issue with SQL migrations:</li>
|
||||
</ul>
|
||||
<li><p>The <code>-O</code> option to <code>pg_restore</code> makes the import process ignore ownership specified in the dump itself, and instead makes the schema owned by the user doing the restore</p></li>
|
||||
|
||||
<li><p>I always prefer to use the <code>postgres</code> user locally because it’s just easier than remembering the <code>dspacetest</code> user’s password, but then I couldn’t figure out why the resulting schema was owned by <code>postgres</code></p></li>
|
||||
|
||||
<li><p>So with this you connect as the <code>postgres</code> superuser and then switch roles to <code>dspacetest</code> (also, make sure this user has <code>superuser</code> privileges before the restore)</p></li>
|
||||
|
||||
<li><p>Last week Linode emailed me to say that our Linode 8192 instance used for DSpace Test qualified for an upgrade</p></li>
|
||||
|
||||
<li><p>Apparently they announced some <a href="https://blog.linode.com/2018/05/17/updated-linode-plans-new-larger-linodes/">upgrades to most of their plans in 2018-05</a></p></li>
|
||||
|
||||
<li><p>After the upgrade I see we have more disk space available in the instance’s dashboard, so I shut the instance down and resized it from 98GB to 160GB</p></li>
|
||||
|
||||
<li><p>The resize was very quick (less than one minute) and after booting the instance back up I now have 160GB for the root filesystem!</p></li>
|
||||
|
||||
<li><p>I will move the DSpace installation directory back to the root file system and delete the extra 300GB block storage, as it was actually kinda slow when we put Solr there and now we don’t actually need it anymore because running the production Solr on this instance didn’t work well with 8GB of RAM</p></li>
|
||||
|
||||
<li><p>Also, the larger instance we’re using for CGSpace will go from 24GB of RAM to 32, and will also get a storage increase from 320GB to 640GB… that means we don’t need to consider using block storage right now!</p></li>
|
||||
|
||||
<li><p>The smaller instances get increased storage and network speed but I doubt many are actually using much of their current allocations so we probably don’t need to bother with upgrading them</p></li>
|
||||
|
||||
<li><p>Last week Abenet asked if we could add <code>dc.language.iso</code> to the advanced search filters</p></li>
|
||||
|
||||
<li><p>There is already a search filter for this field defined in <code>discovery.xml</code> but we aren’t using it, so I quickly enabled and tested it, then merged it to the <code>5_x-prod</code> branch (<a href="https://github.com/ilri/DSpace/pull/380">#380</a>)</p></li>
|
||||
|
||||
<li><p>Back to testing the DSpace 5.8 changes from Atmire, I had another issue with SQL migrations:</p>
|
||||
|
||||
<pre><code>Caused by: org.flywaydb.core.api.FlywayException: Validate failed. Found differences between applied migrations and available migrations: Detected applied migration missing on the classpath: 5.8.2015.12.03.3
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>It took me a while to figure out that this migration is for MQM, which I removed after Atmire’s original advice about the migrations so we actually need to delete this migration instead up updating it</li>
|
||||
<li>So I need to make sure to run the following during the DSpace 5.8 upgrade:</li>
|
||||
</ul>
|
||||
<li><p>It took me a while to figure out that this migration is for MQM, which I removed after Atmire’s original advice about the migrations so we actually need to delete this migration instead up updating it</p></li>
|
||||
|
||||
<li><p>So I need to make sure to run the following during the DSpace 5.8 upgrade:</p>
|
||||
|
||||
<pre><code>-- Delete existing CUA 4 migration if it exists
|
||||
delete from schema_version where version = '5.6.2015.12.03.2';
|
||||
@ -453,49 +456,45 @@ update schema_version set version = '5.6.2015.12.03.2' where version = '5.5.2015
|
||||
|
||||
-- Delete MQM migration since we're no longer using it
|
||||
delete from schema_version where version = '5.5.2015.12.03.3';
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>After that you can run the migrations manually and then DSpace should work fine:</li>
|
||||
</ul>
|
||||
<li><p>After that you can run the migrations manually and then DSpace should work fine:</p>
|
||||
|
||||
<pre><code>$ ~/dspace/bin/dspace database migrate ignored
|
||||
...
|
||||
Done.
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>Elizabeth from CIAT contacted me to ask if I could add ORCID identifiers to all of Andy Jarvis’ items on CGSpace</li>
|
||||
<li>I used my <a href="https://gist.githubusercontent.com/alanorth/a49d85cd9c5dea89cddbe809813a7050/raw/f67b6e45a9a940732882ae4bb26897a9b245ef31/add-orcid-identifiers-csv.py">add-orcid-identifiers-csv.py</a> script:</li>
|
||||
</ul>
|
||||
<li><p>Elizabeth from CIAT contacted me to ask if I could add ORCID identifiers to all of Andy Jarvis’ items on CGSpace</p></li>
|
||||
|
||||
<li><p>I used my <a href="https://gist.githubusercontent.com/alanorth/a49d85cd9c5dea89cddbe809813a7050/raw/f67b6e45a9a940732882ae4bb26897a9b245ef31/add-orcid-identifiers-csv.py">add-orcid-identifiers-csv.py</a> script:</p>
|
||||
|
||||
<pre><code>$ ./add-orcid-identifiers-csv.py -i 2018-06-24-andy-jarvis-orcid.csv -db dspacetest -u dspacetest -p 'fuuu'
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>The contents of <code>2018-06-24-andy-jarvis-orcid.csv</code> were:</li>
|
||||
</ul>
|
||||
<li><p>The contents of <code>2018-06-24-andy-jarvis-orcid.csv</code> were:</p>
|
||||
|
||||
<pre><code>dc.contributor.author,cg.creator.id
|
||||
"Jarvis, A.",Andy Jarvis: 0000-0001-6543-0798
|
||||
"Jarvis, Andy",Andy Jarvis: 0000-0001-6543-0798
|
||||
"Jarvis, Andrew",Andy Jarvis: 0000-0001-6543-0798
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2018-06-26">2018-06-26</h2>
|
||||
|
||||
<ul>
|
||||
<li>Atmire got back to me to say that we can remove the <code>itemCollectionPlugin</code> and <code>HasBitstreamsSSIPlugin</code> beans from DSpace’s <code>discovery.xml</code> file, as they are used by the Metadata Quality Module (MQM) that we are not using anymore</li>
|
||||
<li>I removed both those beans and did some simple tests to check item submission, media-filter of PDFs, REST API, but got an error “No matches for the query” when listing records in OAI</li>
|
||||
<li>This warning appears in the DSpace log:</li>
|
||||
</ul>
|
||||
|
||||
<li><p>This warning appears in the DSpace log:</p>
|
||||
|
||||
<pre><code>2018-06-26 16:58:12,052 WARN org.dspace.xoai.services.impl.xoai.DSpaceRepositoryConfiguration @ { OAI 2.0 :: DSpace } Not able to retrieve the dspace.oai.url property from oai.cfg. Falling back to request address
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>It’s actually only a warning and it also appears in the logs on DSpace Test (which is currently running DSpace 5.5), so I need to keep troubleshooting</li>
|
||||
<li>Ah, I think I just need to run <code>dspace oai import</code></li>
|
||||
<li><p>It’s actually only a warning and it also appears in the logs on DSpace Test (which is currently running DSpace 5.5), so I need to keep troubleshooting</p></li>
|
||||
|
||||
<li><p>Ah, I think I just need to run <code>dspace oai import</code></p></li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2018-06-27">2018-06-27</h2>
|
||||
@ -503,8 +502,8 @@ Done.
|
||||
<ul>
|
||||
<li>Vika from CIFOR sent back his annotations on the duplicates for the “CIFOR_May_9” archive import that I sent him last week</li>
|
||||
<li>I’ll have to figure out how to separate those we’re keeping, deleting, and mapping into CIFOR’s archive collection</li>
|
||||
<li>First, get the 62 deletes from Vika’s file and remove them from the collection:</li>
|
||||
</ul>
|
||||
|
||||
<li><p>First, get the 62 deletes from Vika’s file and remove them from the collection:</p>
|
||||
|
||||
<pre><code>$ grep delete 2018-06-22-cifor-duplicates.txt | grep -o -E '[0-9]{5}\/[0-9]{5}' > cifor-handle-to-delete.txt
|
||||
$ wc -l cifor-handle-to-delete.txt
|
||||
@ -514,51 +513,53 @@ $ wc -l 10568-92904.csv
|
||||
$ while read line; do sed -i "\#$line#d" 10568-92904.csv; done < cifor-handle-to-delete.txt
|
||||
$ wc -l 10568-92904.csv
|
||||
2399 10568-92904.csv
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>This iterates over the handles for deletion and uses <code>sed</code> with an alternative pattern delimiter of ‘#’ (which must be escaped), because the pattern itself contains a ‘/’</li>
|
||||
<li>The mapped ones will be difficult because we need their internal IDs in order to map them, and there are 50 of them:</li>
|
||||
</ul>
|
||||
<li><p>This iterates over the handles for deletion and uses <code>sed</code> with an alternative pattern delimiter of ‘#’ (which must be escaped), because the pattern itself contains a ‘/’</p></li>
|
||||
|
||||
<li><p>The mapped ones will be difficult because we need their internal IDs in order to map them, and there are 50 of them:</p>
|
||||
|
||||
<pre><code>$ grep map 2018-06-22-cifor-duplicates.txt | grep -o -E '[0-9]{5}\/[0-9]{5}' > cifor-handle-to-map.txt
|
||||
$ wc -l cifor-handle-to-map.txt
|
||||
50 cifor-handle-to-map.txt
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>I can either get them from the databse, or programatically export the metadata using <code>dspace metadata-export -i 10568/xxxxx</code>…</li>
|
||||
<li>Oooh, I can export the items one by one, concatenate them together, remove the headers, and extract the <code>id</code> and <code>collection</code> columns using <a href="https://csvkit.readthedocs.io/">csvkit</a>:</li>
|
||||
</ul>
|
||||
<li><p>I can either get them from the databse, or programatically export the metadata using <code>dspace metadata-export -i 10568/xxxxx</code>…</p></li>
|
||||
|
||||
<li><p>Oooh, I can export the items one by one, concatenate them together, remove the headers, and extract the <code>id</code> and <code>collection</code> columns using <a href="https://csvkit.readthedocs.io/">csvkit</a>:</p>
|
||||
|
||||
<pre><code>$ while read line; do filename=${line/\//-}.csv; dspace metadata-export -i $line -f $filename; done < /tmp/cifor-handle-to-map.txt
|
||||
$ sed '/^id/d' 10568-*.csv | csvcut -c 1,2 > map-to-cifor-archive.csv
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>Then I can use Open Refine to add the “CIFOR Archive” collection to the mappings</li>
|
||||
<li>Importing the 2398 items via <code>dspace metadata-import</code> ends up with a Java garbage collection error, so I think I need to do it in batches of 1,000</li>
|
||||
<li>After deleting the 62 duplicates, mapping the 50 items from elsewhere in CGSpace, and uploading 2,398 unique items, there are a total of 2,448 items added in this batch</li>
|
||||
<li>I’ll let Abenet take one last look and then move them to CGSpace</li>
|
||||
<li><p>Then I can use Open Refine to add the “CIFOR Archive” collection to the mappings</p></li>
|
||||
|
||||
<li><p>Importing the 2398 items via <code>dspace metadata-import</code> ends up with a Java garbage collection error, so I think I need to do it in batches of 1,000</p></li>
|
||||
|
||||
<li><p>After deleting the 62 duplicates, mapping the 50 items from elsewhere in CGSpace, and uploading 2,398 unique items, there are a total of 2,448 items added in this batch</p></li>
|
||||
|
||||
<li><p>I’ll let Abenet take one last look and then move them to CGSpace</p></li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2018-06-28">2018-06-28</h2>
|
||||
|
||||
<ul>
|
||||
<li>DSpace Test appears to have crashed last night</li>
|
||||
<li>There is nothing in the Tomcat or DSpace logs, but I see the following in <code>dmesg -T</code>:</li>
|
||||
</ul>
|
||||
|
||||
<li><p>There is nothing in the Tomcat or DSpace logs, but I see the following in <code>dmesg -T</code>:</p>
|
||||
|
||||
<pre><code>[Thu Jun 28 00:00:30 2018] Out of memory: Kill process 14501 (java) score 701 or sacrifice child
|
||||
[Thu Jun 28 00:00:30 2018] Killed process 14501 (java) total-vm:14926704kB, anon-rss:5693608kB, file-rss:0kB, shmem-rss:0kB
|
||||
[Thu Jun 28 00:00:30 2018] oom_reaper: reaped process 14501 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>Look over IITA’s <a href="https://dspacetest.cgiar.org/handle/10568/96071">IITA_Jan_9_II_Ab</a> collection from earlier this month on DSpace Test</li>
|
||||
<li>Bosede fixed a few things (and seems to have removed many French IITA subjects like <code>AMÉLIORATION DES PLANTES</code> and <code>SANTÉ DES PLANTES</code>)</li>
|
||||
<li>I still see at least one issue with author affiliations, and I didn’t bother to check the AGROVOC subjects because it’s such a mess aanyways</li>
|
||||
<li>I suggested that IITA provide an updated list of subject to us so we can include their controlled vocabulary in CGSpace, which would also make it easier to do automated validation</li>
|
||||
<li><p>Look over IITA’s <a href="https://dspacetest.cgiar.org/handle/10568/96071">IITA_Jan_9_II_Ab</a> collection from earlier this month on DSpace Test</p></li>
|
||||
|
||||
<li><p>Bosede fixed a few things (and seems to have removed many French IITA subjects like <code>AMÉLIORATION DES PLANTES</code> and <code>SANTÉ DES PLANTES</code>)</p></li>
|
||||
|
||||
<li><p>I still see at least one issue with author affiliations, and I didn’t bother to check the AGROVOC subjects because it’s such a mess aanyways</p></li>
|
||||
|
||||
<li><p>I suggested that IITA provide an updated list of subject to us so we can include their controlled vocabulary in CGSpace, which would also make it easier to do automated validation</p></li>
|
||||
</ul>
|
||||
|
||||
<!-- vim: set sw=2 ts=2: -->
|
||||
|
||||
Reference in New Issue
Block a user