Add notes for 2019-07-25

2024-11-22 06:35:03 +01:00 · 2019-07-25 16:58:23 +03:00 · 2019-07-25 16:58:23 +03:00 · cec7499c76
commit cec7499c76
parent 38394ed782
3 changed files with 90 additions and 10 deletions
--- a/content/posts/2019-07.md
+++ b/content/posts/2019-07.md
@ -302,7 +302,7 @@ $ dspace user --add --givenname Lionelle --surname Samnick --email blah@blah.com
 ```

 - I added her as a submitter to [CTA ISF Pro-Agro series](https://cgspace.cgiar.org/handle/10568/74536)
- Start looking at 1492 records for the Bioversity batch import
+- Start looking at 1429 records for the Bioversity batch import
  - Multiple authors should be specified with multi-value separatator (||) instead of ;
  - We don't use "(eds)" as an author
  - Same issue with dc.publisher using ";" for multiple values
@ -334,4 +334,42 @@ $ dspace user --add --givenname Lionelle --surname Samnick --email blah@blah.com
 - I left a note saying that DSpace is technically limited to a flat schema so we use `cg.coverage.country: Kenya`
 - Do a little more work on CG Core v2 in the input forms

+## 2019-07-25
+
+- Generate a list of the ORCID identifiers that we added to CGSpace in 2019 for Sara Jani at ICARDA
+- Bioversity sent a new file for their migration to CGSpace
+  - There is always a blank row and blank column at the end
+  - One invalid type (Brie)
+  - 824 items with leading/trailing spaces in dc.identifier.citation
+  - 175 items with a trailing comma in dc.identifier.citation (using custom text facet with GREL `value.endsWith(',').toString()`)
+  - Fix them with GREL transform: `value.replace(/,$/, '')`
+  - A few strange publishers after splitting multi-value cells, like "(Belgium)"
+  - Deleted four ISSNs that are actually ISBNs and are already present in the ISBN field
+  - Eight invalid ISBNs 
+  - Convert all DOIs to "https://doi.org" format and fix one invalid DOI
+  - Fix a handful of incorrect CRPs that seem to have been split on comma ","
+  - Lots of strange values in cg.link.reference, and I normalized all DOIs to https://doi.org format
+    - There are lots of invalid links here, like "36" and "recordlink:publications:2606" and "t3://record?identifier=publications&amp;uid=2606"
+    - Also there are hundreds of items that use the same value for cg.link.reference AND cg.link.dataurl
+  - Use https:// for all Bioversity links (reference, data url, permalink)
+
+- I might be able to use [isbnlib](https://pypi.org/project/isbnlib/) to validate ISBNs in Python:
+
+```
+if isbnlib.is_isbn10('9966-955-07-0') or isbnlib.is_isbn13('9966-955-07-0'):
+    print("Yes")
+else:
+    print("No")
+```
+
+- Or with [python-stdnum](https://github.com/arthurdejong/python-stdnum):
+
+```
+from stdnum import isbn
+from stdnum import issn
+
+isbn.validate('978-92-9043-389-7')
+issn.validate('1020-3362')
+```
+
 <!-- vim: set sw=2 ts=2: -->
--- a/docs/2019-07/index.html
+++ b/docs/2019-07/index.html
@ -21,7 +21,7 @@ Abenet had another similar issue a few days ago when trying to find the stats fo
 <meta property="og:type" content="article" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2019-07/" />
 <meta property="article:published_time" content="2019-07-01T12:13:51&#43;03:00"/>
-<meta property="article:modified_time" content="2019-07-20T18:24:48&#43;03:00"/>
+<meta property="article:modified_time" content="2019-07-22T15:54:32&#43;03:00"/>

 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="July, 2019"/>
@ -47,9 +47,9 @@ Abenet had another similar issue a few days ago when trying to find the stats fo
  "@type": "BlogPosting",
  "headline": "July, 2019",
  "url": "https:\/\/alanorth.github.io\/cgspace-notes\/2019-07\/",
-  "wordCount": "1906",
+  "wordCount": "2119",
  "datePublished": "2019-07-01T12:13:51\x2b03:00",
-  "dateModified": "2019-07-20T18:24:48\x2b03:00",
+  "dateModified": "2019-07-22T15:54:32\x2b03:00",
  "author": {
    "@type": "Person",
    "name": "Alan Orth"
@ -478,7 +478,7 @@ Please see the DSpace documentation for assistance.

 <li><p>I added her as a submitter to <a href="https://cgspace.cgiar.org/handle/10568/74536">CTA ISF Pro-Agro series</a></p></li>

-<li><p>Start looking at 1492 records for the Bioversity batch import</p>
+<li><p>Start looking at 1429 records for the Bioversity batch import</p>

 <ul>
 <li>Multiple authors should be specified with multi-value separatator (||) instead of ;</li>
@ -519,6 +519,48 @@ Please see the DSpace documentation for assistance.
 <li><p>Do a little more work on CG Core v2 in the input forms</p></li>
 </ul>

+<h2 id="2019-07-25">2019-07-25</h2>
+
+<ul>
+<li>Generate a list of the ORCID identifiers that we added to CGSpace in 2019 for Sara Jani at ICARDA</li>
+
+<li><p>Bioversity sent a new file for their migration to CGSpace</p>
+
+<ul>
+<li>There is always a blank row and blank column at the end</li>
+<li>One invalid type (Brie)</li>
+<li>824 items with leading/trailing spaces in dc.identifier.citation</li>
+<li>175 items with a trailing comma in dc.identifier.citation (using custom text facet with GREL <code>value.endsWith(',').toString()</code>)</li>
+<li>Fix them with GREL transform: <code>value.replace(/,$/, '')</code></li>
+<li>A few strange publishers after splitting multi-value cells, like &ldquo;(Belgium)&rdquo;</li>
+<li>Deleted four ISSNs that are actually ISBNs and are already present in the ISBN field</li>
+<li>Eight invalid ISBNs</li>
+<li>Convert all DOIs to &ldquo;<a href="https://doi.org&quot;">https://doi.org&quot;</a> format and fix one invalid DOI</li>
+<li>Fix a handful of incorrect CRPs that seem to have been split on comma &ldquo;,&rdquo;</li>
+<li>Lots of strange values in cg.link.reference, and I normalized all DOIs to <a href="https://doi.org">https://doi.org</a> format</li>
+<li>There are lots of invalid links here, like &ldquo;36&rdquo; and &ldquo;recordlink:publications:2606&rdquo; and &ldquo;t3://record?identifier=publications&amp;uid=2606&rdquo;</li>
+<li>Also there are hundreds of items that use the same value for cg.link.reference AND cg.link.dataurl</li>
+<li>Use https:// for all Bioversity links (reference, data url, permalink)</li>
+</ul></li>
+
+<li><p>I might be able to use <a href="https://pypi.org/project/isbnlib/">isbnlib</a> to validate ISBNs in Python:</p>
+
+<pre><code>if isbnlib.is_isbn10('9966-955-07-0') or isbnlib.is_isbn13('9966-955-07-0'):
+print(&quot;Yes&quot;)
+else:
+print(&quot;No&quot;)
+</code></pre></li>
+
+<li><p>Or with <a href="https://github.com/arthurdejong/python-stdnum">python-stdnum</a>:</p>
+
+<pre><code>from stdnum import isbn
+from stdnum import issn
+
+isbn.validate('978-92-9043-389-7')
+issn.validate('1020-3362')
+</code></pre></li>
+</ul>
+
 <!-- vim: set sw=2 ts=2: -->

  
--- a/docs/sitemap.xml
+++ b/docs/sitemap.xml
@ -4,30 +4,30 @@
  
  <url>
    <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2019-07-20T18:24:48+03:00</lastmod>
+    <lastmod>2019-07-22T15:54:32+03:00</lastmod>
    <priority>0</priority>
  </url>
  
  <url>
    <loc>https://alanorth.github.io/cgspace-notes/2019-07/</loc>
-    <lastmod>2019-07-20T18:24:48+03:00</lastmod>
+    <lastmod>2019-07-22T15:54:32+03:00</lastmod>
  </url>
  
  <url>
    <loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
-    <lastmod>2019-07-20T18:24:48+03:00</lastmod>
+    <lastmod>2019-07-22T15:54:32+03:00</lastmod>
    <priority>0</priority>
  </url>
  
  <url>
    <loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
-    <lastmod>2019-07-20T18:24:48+03:00</lastmod>
+    <lastmod>2019-07-22T15:54:32+03:00</lastmod>
    <priority>0</priority>
  </url>
  
  <url>
    <loc>https://alanorth.github.io/cgspace-notes/tags/</loc>
-    <lastmod>2019-07-20T18:24:48+03:00</lastmod>
+    <lastmod>2019-07-22T15:54:32+03:00</lastmod>
    <priority>0</priority>
  </url>