I think a sane proofing workflow in OpenRefine is to apply the custom text facets for check/delete/remove and illegal characters that I developed in March, 2018
I think a sane proofing workflow in OpenRefine is to apply the custom text facets for check/delete/remove and illegal characters that I developed in March, 2018
<li>Test the <ahref="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=560">DSpace 5.8 module upgrades from Atmire</a> (<ahref="https://github.com/ilri/DSpace/pull/378">#378</a>)
<li>There seems to be a problem with the CUA and L&R versions in <code>pom.xml</code> because they are using SNAPSHOT and it doesn’t build</li>
<li>I added the new CCAFS Phase II Project Tag <code>PII-FP1_PACCA2</code> and merged it into the <code>5_x-prod</code> branch (<ahref="https://github.com/ilri/DSpace/pull/379">#379</a>)</li>
<li>I think a sane proofing workflow in OpenRefine is to apply the custom text facets for check/delete/remove and illegal characters that I developed in <ahref="/cgspace-notes/2018-03/">March, 2018</a></li>
<li>Now Maven and Ant run properly, but I’m getting SQL migration errors in <code>dspace.log</code> after starting Tomcat</li>
<li>I’ve updated my ticket on Atmire’s bug tracker: <ahref="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=560">https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=560</a></li>
<li>Gabriela from CIP got back to me about the author names we were correcting on CGSpace</li>
<li>I did a quick sanity check on them and then did a test import with my <ahref="https://gist.github.com/alanorth/df92cbfb54d762ba21b28f7cd83b6897"><code>fix-metadata-value.py</code></a> script:</li>
<li>After removing all code mentioning MQM, mqm, metadata-quality, batchedit, duplicatechecker, etc, I think I got most of it removed, but there is a Spring error during Tomcat startup:</li>
</ul>
<pre><code> INFO [org.dspace.servicemanager.DSpaceServiceManager] Shutdown DSpace core service manager
Failed to startup the DSpace Service Manager: failure starting up spring service manager: Error creating bean with name 'org.dspace.servicemanager.spring.DSpaceBeanPostProcessor#0' defined in class path resource [spring/spring-dspace-applicationContext.xml]: Unsatisfied dependency expressed through constructor argument with index 0 of type [org.dspace.servicemanager.config.DSpaceConfigurationService]: : Cannot find class [com.atmire.dspace.discovery.ItemCollectionPlugin] for bean with name 'itemCollectionPlugin' defined in file [/home/aorth/dspace/config/spring/api/discovery.xml];
<li>I can fix this by commenting out the <code>ItemCollectionPlugin</code> line of <code>discovery.xml</code>, but from looking at the git log I’m not actually sure if that is related to MQM or not</li>
<li>I normalized all DOIs to use HTTPS and “doi.org” instead of “dx.doi.org”</li>
<li>I cleaned up white space in <code>cg.subject.iita</code> and <code>dc.subject</code></li>
<li>Even a bunch of IITA and AGROVOC subjects are missing accents, ie “FERTILIT DU SOL”</li>
<li>More organization names in <code>dc.description.sponsorship</code> are incorrect (ie, missing accents) or inconsistent (ie, CGIAR centers should be spelled in English or multiple spellings of the same one, like “Rockefeller Foundation” and “Rockefeller foundation”)</li>
<li>Authorship types: “CGIAR ans advanced research institute”, “CGAIR and advanced research institute”, “CGIAR and advanced research institutes”, “CGAIR single center”</li>
<li>“Institut des Recherches Agricoles du Bénin” and “Institut National des Recherche Agricoles du Benin” and “National Agricultural Research Institute, Benin”</li>
<li>International Insitute of Tropical Agriculture</li>
<li>Centro Internacional de Agricultura Tropical</li>
<li>“Rivers State University of Science and Technology” and “Rivers State University”</li>
<li>“Institut de la Recherche Agronomique, Cameroon” and “Institut de Recherche Agronomique, Cameroon”</li>
<li>A few DOIs with spaces or invalid characters</li>
<li>Inconsistency in IITA subjects, for example “PRODUCTION VEGETALE” and “PRODUCTION VÉGÉTALE” and several others</li>
<li>I ran <code>value.unescape('javascript')</code> on the abstract and citation fields because it looks like this data came from a SQL database and some stuff was escaped</li>
<li>It turns out that Abenet actually did a lot of small corrections on this data so when Sisay uses Bosede’s original file it doesn’t have all those corrections</li>
<li>So I told Sisay to re-create the collection using Abenet’s XLS from last week (<code>Mercy1805_AY.xls</code>)</li>
<li>I was curious to see if I could create a GREL for use with a custom text facet in Open Refine to find cells with two or more consecutive spaces</li>
<li>I always use the built-in trim and collapse transformations anyways, but this seems to work to find the offending cells: <code>isNotNull(value.match(/.*?\s{2,}.*?/))</code></li>
<li>I wonder if I should start checking for “smart” quotes like ’ (hex 2019)</li>
<li>I think it should be this: <ahref="https://cgspace.cgiar.org/oai/request?verb=ListRecords&metadataPrefix=oai_dc&set=com_10568_16814">https://cgspace.cgiar.org/oai/request?verb=ListRecords&metadataPrefix=oai_dc&set=com_10568_16814</a></li>
<li>The style sheet obfuscates the data, but if you look at the source it is all there, including information about pagination of results</li>
<li>Regarding Udana’s Book Chapters and Reports on DSpace Test last week, Abenet told him to fix some character encoding and CRP issues, then I told him I’d check them after that</li>
<li>The latest batch of IITA’s 200 records (based on Abenet’s version <code>Mercy1805_AY.xls</code>) are now in the <ahref="https://dspacetest.cgiar.org/handle/10568/96071">IITA_Jan_9_II_Ab</a> collection</li>
<li>use of Unicode smart quote (hex 2019) in countries and affiliations, for example “COTE D’IVOIRE” and “Institut d’Economic Rurale, Mali”</li>
<li>“Centro Internacional de Agricultura Tropical” and “Centro International de Agricultura Tropical” should use the English name of CIAT (International Center for Tropical Agriculture)</li>
<li>“Institut International d’Agriculture Tropicale” should use the English name of IITA (International Institute of Tropical Agriculture)</li>
<li>“East and Southern Africa Regional Center” and “Eastern and Southern Africa Regional Centre”</li>
<li>“Institut de la Recherche Agronomique, Cameroon” and “Institut de Recherche Agronomique, Cameroon”</li>
<li>“Institut des Recherches Agricoles du Bénin” and “Institut National des Recherche Agricoles du Benin” and “National Agricultural Research Institute, Benin”</li>
<li>“Institute of Agronomic Research, Cameroon” and “Institute of Agronomy Research, Cameroon”</li>
<li>“Rivers State University” and “Rivers State University of Science and Technology”</li>
<li>“Universität Hannover” and “University of Hannover”</li>
<li>“AMELIORATION DES PLANTES” and “AMÉLIORATION DES PLANTES”</li>
<li>“PRODUCTION VEGETALE” and “PRODUCTION VÉGÉTALE”</li>
<li>“CONTRÔLE DE MALADIES” and “CONTROLE DES MALADIES”</li>
<li>“HANDLING, TRANSPORT, STORAGE AND PROTECTION OF AGRICULTURAL PRODUCT” and “HANDLING, TRANSPORT, STORAGE AND PROTECTION OF AGRICULTURAL PRODUCTS”</li>
<li>“RAVAGEURS DE PLANTES” and “RAVAGEURS DES PLANTES”</li>
<li>“SANTE DES PLANTES” and “SANTÉ DES PLANTES”</li>
<li>“SOCIOECONOMIE” and “SOCIOECONOMY”</li>
<li>I noticed that some records do have encoding errors in the <code>dc.description.abstract</code> field, but only four of them so probably not from Abenet’s handling of the XLS file</li>
<li>I used my <ahref="https://gist.githubusercontent.com/alanorth/a49d85cd9c5dea89cddbe809813a7050/raw/f67b6e45a9a940732882ae4bb26897a9b245ef31/add-orcid-identifiers-csv.py">add-orcid-identifiers-csv.py</a> script:</li>
<li>There were only some minor whitespace and one or two syntax errors, but they look very good otherwise</li>
<li>I uploaded the twenty-four reports to the IWMI Reports collection: <ahref="https://cgspace.cgiar.org/handle/10568/36188">https://cgspace.cgiar.org/handle/10568/36188</a></li>
<li>I uploaded the seventy-six book chapters to the IWMI Book Chapters collection: <ahref="https://cgspace.cgiar.org/handle/10568/36178">https://cgspace.cgiar.org/handle/10568/36178</a></li>
<li>I was restoring a PostgreSQL dump on my test machine and found a way to restore the CGSpace dump as the <code>postgres</code> user, but have the owner of the schema be the <code>dspacetest</code> user:</li>
<li>The <code>-O</code> option to <code>pg_restore</code> makes the import process ignore ownership specified in the dump itself, and instead makes the schema owned by the user doing the restore</li>
<li>I always prefer to use the <code>postgres</code> user locally because it’s just easier than remembering the <code>dspacetest</code> user’s password, but then I couldn’t figure out why the resulting schema was owned by <code>postgres</code></li>
<li>So with this you connect as the <code>postgres</code> superuser and then switch roles to <code>dspacetest</code> (also, make sure this user has <code>superuser</code> privileges before the restore)</li>
<li>Last week Linode emailed me to say that our Linode 8192 instance used for DSpace Test qualified for an upgrade</li>
<li>Apparently they announced some <ahref="https://blog.linode.com/2018/05/17/updated-linode-plans-new-larger-linodes/">upgrades to most of their plans in 2018-05</a></li>
<li>After the upgrade I see we have more disk space available in the instance’s dashboard, so I shut the instance down and resized it from 98GB to 160GB</li>
<li>I will move the DSpace installation directory back to the root file system and delete the extra 300GB block storage, as it was actually kinda slow when we put Solr there and now we don’t actually need it anymore because running the production Solr on this instance didn’t work well with 8GB of RAM</li>
<li>Also, the larger instance we’re using for CGSpace will go from 24GB of RAM to 32, and will also get a storage increase from 320GB to 640GB… that means we don’t need to consider using block storage right now!</li>
<li>The smaller instances get increased storage and network speed but I doubt many are actually using much of their current allocations so we probably don’t need to bother with upgrading them</li>
<li>There is already a search filter for this field defined in <code>discovery.xml</code> but we aren’t using it, so I quickly enabled and tested it, then merged it to the <code>5_x-prod</code> branch (<ahref="https://github.com/ilri/DSpace/pull/380">#380</a>)</li>
<pre><code>Caused by: org.flywaydb.core.api.FlywayException: Validate failed. Found differences between applied migrations and available migrations: Detected applied migration missing on the classpath: 5.8.2015.12.03.3
<li>It took me a while to figure out that this migration is for MQM, which I removed after Atmire’s original advice about the migrations so we actually need to delete this migration instead up updating it</li>
<li>I used my <ahref="https://gist.githubusercontent.com/alanorth/a49d85cd9c5dea89cddbe809813a7050/raw/f67b6e45a9a940732882ae4bb26897a9b245ef31/add-orcid-identifiers-csv.py">add-orcid-identifiers-csv.py</a> script:</li>
<li>Atmire got back to me to say that we can remove the <code>itemCollectionPlugin</code> and <code>HasBitstreamsSSIPlugin</code> beans from DSpace’s <code>discovery.xml</code> file, as they are used by the Metadata Quality Module (MQM) that we are not using anymore</li>
<li>I removed both those beans and did some simple tests to check item submission, media-filter of PDFs, REST API, but got an error “No matches for the query” when listing records in OAI</li>
<pre><code>2018-06-26 16:58:12,052 WARN org.dspace.xoai.services.impl.xoai.DSpaceRepositoryConfiguration @ { OAI 2.0 :: DSpace } Not able to retrieve the dspace.oai.url property from oai.cfg. Falling back to request address
<li>It’s actually only a warning and it also appears in the logs on DSpace Test (which is currently running DSpace 5.5), so I need to keep troubleshooting</li>
<li>This iterates over the handles for deletion and uses <code>sed</code> with an alternative pattern delimiter of ‘#’ (which must be escaped), because the pattern itself contains a ‘/’</li>
<li>The mapped ones will be difficult because we need their internal IDs in order to map them, and there are 50 of them:</li>
<li>I can either get them from the databse, or programatically export the metadata using <code>dspace metadata-export -i 10568/xxxxx</code>…</li>
<li>Oooh, I can export the items one by one, concatenate them together, remove the headers, and extract the <code>id</code> and <code>collection</code> columns using <ahref="https://csvkit.readthedocs.io/">csvkit</a>:</li>
<li>Then I can use Open Refine to add the “CIFOR Archive” collection to the mappings</li>
<li>Importing the 2398 items via <code>dspace metadata-import</code> ends up with a Java garbage collection error, so I think I need to do it in batches of 1,000</li>
<li>After deleting the 62 duplicates, mapping the 50 items from elsewhere in CGSpace, and uploading 2,398 unique items, there are a total of 2,448 items added in this batch</li>
<li>Look over IITA’s <ahref="https://dspacetest.cgiar.org/handle/10568/96071">IITA_Jan_9_II_Ab</a> collection from earlier this month on DSpace Test</li>
<li>Bosede fixed a few things (and seems to have removed many French IITA subjects like <code>AMÉLIORATION DES PLANTES</code> and <code>SANTÉ DES PLANTES</code>)</li>
<li>I still see at least one issue with author affiliations, and I didn’t bother to check the AGROVOC subjects because it’s such a mess aanyways</li>
<li>I suggested that IITA provide an updated list of subject to us so we can include their controlled vocabulary in CGSpace, which would also make it easier to do automated validation</li>
