alanorth/cgspace-notes
1
0
Fork 0
mirror of https://github.com/alanorth/cgspace-notes.git synced 2024-11-22 06:35:03 +01:00
Code Issues Projects Releases Wiki Activity

Add notes for 2023-08-23

Browse Source
This commit is contained in:
Alan Orth 2023-08-24 09:03:46 +03:00
parent d2ad21bde1
commit 33061dbe3a
Signed by: alanorth
GPG Key ID: 0FB860CC9C45B1B9
32 changed files with 100 additions and 37 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

30
content/posts/2023-08.md
View File

@ -172,5 +172,35 @@ $ ./run.sh -s http://localhost:8081/solr/statistics -a import -o /tmp/statistics
- This time looking at the thumbnail responses in item endpoints
- According to [the documentation](https://github.com/DSpace/RestContract/blob/main/items.md#main-thumbnail) the API will respond with HTTP 200 if there is a thumbnail, and HTTP 204 if there is no content
- That means we need to make the request before we can even find out!
- Tim on DSpace Slack pointed out the DSpace 7 REST API's [projections](https://github.com/DSpace/RestContract/blob/main/projections.md)
- This means we can embed resources like thumbnail and owningCollection in the item (and other) requests, for example: https://dspace7test.ilri.org/server/api/discover/search/objects?dsoType=item&embed=thumbnail,owningCollection
## 2023-08-23
- I benchmarked the DSpace 7 REST API with the new embeds and it took four hours and seventeen minutes to get all 106,000 items on DSpace 7 Test
- So this is much slower than the results I saw earlier this week, but maybe slightly faster than DSpace 6?
- Maria from Alliance contacted me to say they have agreed to use UN M.49 regions more strictly in TIP, so they want to replace our non-standard "Latin America" region with "Latin America and the Caribbean", "Caribbean" and "Americas" on all Alliance outputs
- I exported their community on CGSpace and fixed the metadata in OpenRefine
- I tried to run `dspace cleanup -v` on CGSpace, but got this error:
```
Caused by: org.postgresql.util.PSQLException: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
Detail: Key (uuid)=(61bff7da-c8e3-420f-841c-ec5e8238d716) is still referenced from table "bundle".
```
- The solution, as always, is to delete those IDs manually in PostgreSQL:
```
$ psql -d dspace -c "UPDATE bundle SET primary_bitstream_id=NULL WHERE primary_bitstream_id IN ('61bff7da-c8e3-420f-841c-ec5e8238d716');"
UPDATE 1
```
- I also tried to delete all users who haven't logged in since 2017 using the groomer script, but it crashes due to those users still having items or workflows or whatever:
```console
$ dspace dsrun org.dspace.eperson.Groomer -a -b 08/23/2017 -d
```
- I see that it is now [possible in DSpace 7 to delete such users](https://github.com/DSpace/DSpace/pull/2229) so we will have to wait
<!-- vim: set sw=2 ts=2: -->

39
docs/2023-08/index.html
View File

@ -19,7 +19,7 @@ Start working on some batch uploads for IFPRI
<meta property="og:type" content="article" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2023-08/" />
<meta property="article:published_time" content="2023-08-03T11:18:36+03:00" />
<meta property="article:modified_time" content="2023-08-18T23:54:07+03:00" />
<meta property="article:modified_time" content="2023-08-22T17:28:49+03:00" />
@ -44,9 +44,9 @@ Start working on some batch uploads for IFPRI
"@type": "BlogPosting",
"headline": "August, 2023",
"url": "https://alanorth.github.io/cgspace-notes/2023-08/",
"wordCount": "1254",
"wordCount": "1514",
"datePublished": "2023-08-03T11:18:36+03:00",
"dateModified": "2023-08-18T23:54:07+03:00",
"dateModified": "2023-08-22T17:28:49+03:00",
"author": {
"@type": "Person",
"name": "Alan Orth"
@ -322,6 +322,39 @@ Start working on some batch uploads for IFPRI
<li>That means we need to make the request before we can even find out!</li>
</ul>
</li>
<li>Tim on DSpace Slack pointed out the DSpace 7 REST API&rsquo;s <a href="https://github.com/DSpace/RestContract/blob/main/projections.md">projections</a>
<ul>
<li>This means we can embed resources like thumbnail and owningCollection in the item (and other) requests, for example: <a href="https://dspace7test.ilri.org/server/api/discover/search/objects?dsoType=item&amp;embed=thumbnail,owningCollection">https://dspace7test.ilri.org/server/api/discover/search/objects?dsoType=item&amp;embed=thumbnail,owningCollection</a></li>
</ul>
</li>
</ul>
<h2 id="2023-08-23">2023-08-23</h2>
<ul>
<li>I benchmarked the DSpace 7 REST API with the new embeds and it took four hours and seventeen minutes to get all 106,000 items on DSpace 7 Test
<ul>
<li>So this is much slower than the results I saw earlier this week, but maybe slightly faster than DSpace 6?</li>
</ul>
</li>
<li>Maria from Alliance contacted me to say they have agreed to use UN M.49 regions more strictly in TIP, so they want to replace our non-standard &ldquo;Latin America&rdquo; region with &ldquo;Latin America and the Caribbean&rdquo;, &ldquo;Caribbean&rdquo; and &ldquo;Americas&rdquo; on all Alliance outputs
<ul>
<li>I exported their community on CGSpace and fixed the metadata in OpenRefine</li>
</ul>
</li>
<li>I tried to run <code>dspace cleanup -v</code> on CGSpace, but got this error:</li>
</ul>
<pre tabindex="0"><code>Caused by: org.postgresql.util.PSQLException: ERROR: update or delete on table &#34;bitstream&#34; violates foreign key constraint &#34;bundle_primary_bitstream_id_fkey&#34; on table &#34;bundle&#34;
Detail: Key (uuid)=(61bff7da-c8e3-420f-841c-ec5e8238d716) is still referenced from table &#34;bundle&#34;.
</code></pre><ul>
<li>The solution, as always, is to delete those IDs manually in PostgreSQL:</li>
</ul>
<pre tabindex="0"><code>$ psql -d dspace -c &#34;UPDATE bundle SET primary_bitstream_id=NULL WHERE primary_bitstream_id IN (&#39;61bff7da-c8e3-420f-841c-ec5e8238d716&#39;);&#34;
UPDATE 1
</code></pre><ul>
<li>I also tried to delete all users who haven&rsquo;t logged in since 2017 using the groomer script, but it crashes due to those users still having items or workflows or whatever:</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ dspace dsrun org.dspace.eperson.Groomer -a -b 08/23/2017 -d
</span></span></code></pre></div><ul>
<li>I see that it is now <a href="https://github.com/DSpace/DSpace/pull/2229">possible in DSpace 7 to delete such users</a> so we will have to wait</li>
</ul>
<!-- raw HTML omitted -->

2
docs/categories/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/categories/notes/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/categories/notes/page/2/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/categories/notes/page/3/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/categories/notes/page/4/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/categories/notes/page/5/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/categories/notes/page/6/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/categories/notes/page/7/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/categories/notes/page/8/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/page/10/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/page/2/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/page/3/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/page/4/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/page/5/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/page/6/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/page/7/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/page/8/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/page/9/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/posts/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/posts/page/10/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/posts/page/2/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/posts/page/3/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/posts/page/4/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/posts/page/5/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/posts/page/6/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/posts/page/7/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/posts/page/8/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

2
docs/posts/page/9/index.html
View File

@ -10,7 +10,7 @@
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
<meta property="og:updated_time" content="2023-08-18T23:54:07+03:00" />
<meta property="og:updated_time" content="2023-08-22T17:28:49+03:00" />

10
docs/sitemap.xml
View File

@ -3,19 +3,19 @@
xmlns:xhtml="http://www.w3.org/1999/xhtml">
<url>
<loc>https://alanorth.github.io/cgspace-notes/2023-08/</loc>
<lastmod>2023-08-18T23:54:07+03:00</lastmod>
<lastmod>2023-08-22T17:28:49+03:00</lastmod>
</url><url>
<loc>https://alanorth.github.io/cgspace-notes/categories/</loc>
<lastmod>2023-08-18T23:54:07+03:00</lastmod>
<lastmod>2023-08-22T17:28:49+03:00</lastmod>
</url><url>
<loc>https://alanorth.github.io/cgspace-notes/</loc>
<lastmod>2023-08-18T23:54:07+03:00</lastmod>
<lastmod>2023-08-22T17:28:49+03:00</lastmod>
</url><url>
<loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
<lastmod>2023-08-18T23:54:07+03:00</lastmod>
<lastmod>2023-08-22T17:28:49+03:00</lastmod>
</url><url>
<loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
<lastmod>2023-08-18T23:54:07+03:00</lastmod>
<lastmod>2023-08-22T17:28:49+03:00</lastmod>
</url><url>
<loc>https://alanorth.github.io/cgspace-notes/2023-07/</loc>
<lastmod>2023-08-02T23:04:11+03:00</lastmod>