mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2024-12-19 11:42:19 +01:00
515 lines
26 KiB
HTML
515 lines
26 KiB
HTML
<!DOCTYPE html>
|
|
<html lang="en" >
|
|
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
|
|
|
|
|
|
<meta property="og:title" content="August, 2023" />
|
|
<meta property="og:description" content="2023-08-03
|
|
|
|
I finally got around to working on Peter’s cleanups for affiliations, authors, and donors from last week
|
|
|
|
I did some minor cleanups myself and applied them to CGSpace
|
|
|
|
|
|
Start working on some batch uploads for IFPRI
|
|
" />
|
|
<meta property="og:type" content="article" />
|
|
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2023-08/" />
|
|
<meta property="article:published_time" content="2023-08-03T11:18:36+03:00" />
|
|
<meta property="article:modified_time" content="2023-09-01T08:10:02+03:00" />
|
|
|
|
|
|
|
|
<meta name="twitter:card" content="summary"/>
|
|
<meta name="twitter:title" content="August, 2023"/>
|
|
<meta name="twitter:description" content="2023-08-03
|
|
|
|
I finally got around to working on Peter’s cleanups for affiliations, authors, and donors from last week
|
|
|
|
I did some minor cleanups myself and applied them to CGSpace
|
|
|
|
|
|
Start working on some batch uploads for IFPRI
|
|
"/>
|
|
<meta name="generator" content="Hugo 0.118.2">
|
|
|
|
|
|
|
|
<script type="application/ld+json">
|
|
{
|
|
"@context": "http://schema.org",
|
|
"@type": "BlogPosting",
|
|
"headline": "August, 2023",
|
|
"url": "https://alanorth.github.io/cgspace-notes/2023-08/",
|
|
"wordCount": "1921",
|
|
"datePublished": "2023-08-03T11:18:36+03:00",
|
|
"dateModified": "2023-09-01T08:10:02+03:00",
|
|
"author": {
|
|
"@type": "Person",
|
|
"name": "Alan Orth"
|
|
},
|
|
"keywords": "Notes"
|
|
}
|
|
</script>
|
|
|
|
|
|
|
|
<link rel="canonical" href="https://alanorth.github.io/cgspace-notes/2023-08/">
|
|
|
|
<title>August, 2023 | CGSpace Notes</title>
|
|
|
|
|
|
<!-- combined, minified CSS -->
|
|
|
|
<link href="https://alanorth.github.io/cgspace-notes/css/style.c6ba80bc50669557645abe05f86b73cc5af84408ed20f1551a267bc19ece8228.css" rel="stylesheet" integrity="sha256-xrqAvFBmlVdkWr4F+GtzzFr4RAjtIPFVGiZ7wZ7Ogig=" crossorigin="anonymous">
|
|
|
|
|
|
<!-- minified Font Awesome for SVG icons -->
|
|
|
|
<script defer src="https://alanorth.github.io/cgspace-notes/js/fontawesome.min.f5072c55a0721857184db93a50561d7dc13975b4de2e19db7f81eb5f3fa57270.js" integrity="sha256-9QcsVaByGFcYTbk6UFYdfcE5dbTeLhnbf4HrXz+lcnA=" crossorigin="anonymous"></script>
|
|
|
|
<!-- RSS 2.0 feed -->
|
|
|
|
|
|
|
|
|
|
</head>
|
|
|
|
<body>
|
|
|
|
|
|
<div class="blog-masthead">
|
|
<div class="container">
|
|
<nav class="nav blog-nav">
|
|
<a class="nav-link " href="https://alanorth.github.io/cgspace-notes/">Home</a>
|
|
</nav>
|
|
</div>
|
|
</div>
|
|
|
|
|
|
|
|
|
|
<header class="blog-header">
|
|
<div class="container">
|
|
<h1 class="blog-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/" rel="home">CGSpace Notes</a></h1>
|
|
<p class="lead blog-description" dir="auto">Documenting day-to-day work on the <a href="https://cgspace.cgiar.org">CGSpace</a> repository.</p>
|
|
</div>
|
|
</header>
|
|
|
|
|
|
|
|
|
|
<div class="container">
|
|
<div class="row">
|
|
<div class="col-sm-8 blog-main">
|
|
|
|
|
|
|
|
|
|
<article class="blog-post">
|
|
<header>
|
|
<h2 class="blog-post-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/2023-08/">August, 2023</a></h2>
|
|
<p class="blog-post-meta">
|
|
<time datetime="2023-08-03T11:18:36+03:00">Thu Aug 03, 2023</time>
|
|
in
|
|
<span class="fas fa-folder" aria-hidden="true"></span> <a href="/categories/notes/" rel="category tag">Notes</a>
|
|
|
|
|
|
</p>
|
|
</header>
|
|
<h2 id="2023-08-03">2023-08-03</h2>
|
|
<ul>
|
|
<li>I finally got around to working on Peter’s cleanups for affiliations, authors, and donors from last week
|
|
<ul>
|
|
<li>I did some minor cleanups myself and applied them to CGSpace</li>
|
|
</ul>
|
|
</li>
|
|
<li>Start working on some batch uploads for IFPRI</li>
|
|
</ul>
|
|
<h2 id="2023-08-04">2023-08-04</h2>
|
|
<ul>
|
|
<li>Minor cleanups on IFPRI’s batch uploads
|
|
<ul>
|
|
<li>I also did a duplicate check and found thirteen items that seem to be duplicates, so I sent them to Leigh to check</li>
|
|
</ul>
|
|
</li>
|
|
<li>I read this <a href="https://www.endpointdev.com/blog/2012/06/logstatement-postgres-all-full-logging/">interesting blog post about PostgreSQL’s <code>log_statement</code> function</a>
|
|
<ul>
|
|
<li>Someone pointed out that this also lets you take advantage of <a href="https://github.com/darold/pgbadger">PgBadger</a> analysis</li>
|
|
<li>I enabled statement logging on DSpace Test and I will check it in a few days</li>
|
|
</ul>
|
|
</li>
|
|
<li>Reading about DSpace 7 REST API again
|
|
<ul>
|
|
<li>Here is how to get the first page of 100 items: <a href="https://dspace7test.ilri.org/server/api/discover/search/objects?dsoType=item&page=1&size=100">https://dspace7test.ilri.org/server/api/discover/search/objects?dsoType=item&page=1&size=100</a></li>
|
|
<li>I really want to benchmark this to see how fast we can get all the pages</li>
|
|
<li>Another thing I notice is that the bitstreams are not here, so that will be an extra call…</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<h2 id="2023-08-05">2023-08-05</h2>
|
|
<ul>
|
|
<li>Export CGSpace to check for missing Initiative collection mappings</li>
|
|
<li>Start a harvest on AReS</li>
|
|
</ul>
|
|
<h2 id="2023-08-07">2023-08-07</h2>
|
|
<ul>
|
|
<li>I’m checking the PostgreSQL logs now that statement logging has been enabled for a few days on DSpace Test
|
|
<ul>
|
|
<li>I see the logs are about 7 or 8 GB, which is larger than expected—and this is the test server!</li>
|
|
<li>I will now play with pgbadger to see if it gives any useful insights</li>
|
|
<li>Hmm, it sems the <code>log_statement</code> advice was old as pgbadger itself says:</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<blockquote>
|
|
<p>Do not enable log_statement as its log format will not be parsed by pgBadger.</p>
|
|
</blockquote>
|
|
<p>… and:</p>
|
|
<blockquote>
|
|
<p>Warning: Do not enable both log_min_duration_statement, log_duration and log_statement all together, this will result in wrong counter values. Note that this will also increase drastically the size of your log. log_min_duration_statement should always be preferred.</p>
|
|
</blockquote>
|
|
<ul>
|
|
<li>So we need to follow pgbadger’s instructions rather to get a suitable log file
|
|
<ul>
|
|
<li>After enabling the new settings I see that our log file is going to be reaallllly big… hmmmm will check tomorrow morning</li>
|
|
</ul>
|
|
</li>
|
|
<li>More work on the IFPRI batch uploads</li>
|
|
</ul>
|
|
<h2 id="2023-08-08">2023-08-08</h2>
|
|
<ul>
|
|
<li>Apply more corrections to authors from Peter on CGSpace</li>
|
|
<li>I finally figured out a <code>log_line_prefix</code> for PostgreSQL that works for pgBadger:</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>log_line_prefix = '%t [%p]: user=%u,db=%d,app=%a,client=%h '
|
|
</span></span></code></pre></div><ul>
|
|
<li>Now I can generate reports:</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span># /usr/bin/pgbadger -I -q /var/log/postgresql/postgresql-14-main.log -O /srv/www/pgbadger
|
|
</span></span></code></pre></div><ul>
|
|
<li>Ideally we would run this incremental report every day on the postgresql-14-main.log.1 aka yesterday’s version of the log file after it is rotated
|
|
<ul>
|
|
<li>Now I have to see how large the file will be…</li>
|
|
</ul>
|
|
</li>
|
|
<li>I did some final updates to the ninety IFPRI records and uploaded them to DSpace Test first, then to CGSpace</li>
|
|
</ul>
|
|
<h2 id="2023-08-11">2023-08-11</h2>
|
|
<ul>
|
|
<li>Fix bug with header background on DSpace 7 on mobile</li>
|
|
</ul>
|
|
<h2 id="2023-08-12">2023-08-12</h2>
|
|
<ul>
|
|
<li>Export CGSpace to check for missing Initiative collection mappings</li>
|
|
<li>I deployed the latest OpenRXV master branch with Angular v14 and backend updates on the server</li>
|
|
<li>Start a harvest on AReS</li>
|
|
</ul>
|
|
<h2 id="2023-08-14">2023-08-14</h2>
|
|
<ul>
|
|
<li>I ported the DSpace 6.x REST API patch to allow specifying a bundle name when POSTing a bitstream to the legacy REST API in DSpace 7.6</li>
|
|
</ul>
|
|
<h2 id="2023-08-16">2023-08-16</h2>
|
|
<ul>
|
|
<li>I noticed that the DSpace statistics pages don’t seem to work on communities or collections
|
|
<ul>
|
|
<li>I finally took time to look in the DSpace log file and found this for one:</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>2023-08-16 14:30:31,873 WARN dace8f96-f034-488e-b38c-9f2eb5d0e002 6cbd0b18-6852-4294-99a5-02dfcab0a469 org.dspace.app.rest.exception.DSpaceApiExceptionControllerAdvice @ Request is invalid or incorrect (status:400 exception: Invalid UUID string: -1 at: java.base/java.util.UUID.fromString1(UUID.java:280))
|
|
</span></span></code></pre></div><ul>
|
|
<li>I’m surprised to see this because those should have been dealt with when we upgraded to DSpace 6
|
|
<ul>
|
|
<li>Looking in the Solr statistics core I see ~1,000,000 documents with the ID <code>-1</code>, and about 57,000,000 that don’t</li>
|
|
<li>Also interesting, faceting by <code>dateYear</code> I see:
|
|
<ul>
|
|
<li>2023: 209566</li>
|
|
<li>2022: 403871</li>
|
|
<li>2021: 336548</li>
|
|
<li>2020: 31659</li>
|
|
<li>… none before 2020</li>
|
|
</ul>
|
|
</li>
|
|
<li>They are all type 5, which is “Site” aka the home page, according to <code>dspace-api/src/main/java/org/dspace/core/Constants.java</code></li>
|
|
<li>Ah hah, and I can see in my DSpace 7 test Solr there are a bunch of hits with <code>type: 5</code> that have “-1” of course, but also newer ones that have an actual UUID</li>
|
|
<li>I used the <code>/server/api/dso/find?uuid=3945ec23-2426-4fce-a2ea-48b38b91547f</code> endpoint to find out that there is a new <code>/server/api/core/sites</code> endpoint listing exactly one site (the home page) with this ID</li>
|
|
<li>So for now I can replace all the “-1” documents with this ID on the test server at least, then I will have to remember to do that during the migration of the production instance</li>
|
|
<li>I did a new export from DSpace 6 using solr-import-export-json with a query limiting it to documents of type 5 and negative 1 ID:</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ chrt -b <span style="color:#ae81ff">0</span> ./run.sh -s http://localhost:8081/solr/statistics -a export -o /tmp/statistics-fix-uuid.json -f <span style="color:#e6db74">'id:\-1 AND type:5 AND time:[2020-01-01T00\:00\:00Z TO 2023-12-31T23\:59\:59Z]'</span> -k uid -S actingGroupId,actingGroupParentId,actorMemberGroupId,author_mtdt,author_mtdt_search,bitstreamCount,bitstreamId,complete_query,complete_query_search,containerBitstream,containerCollection,containerCommunity,containerItem,core_update_run_nb,countryCode_ngram,countryCode_search,cua_version,dateYear,dateYearMonth,file_id,filterquery,first_name,geoipcountrycode,geoIpCountryCode,group_id,group_map,group_name,ip_ngram,ip_search,isArchived,isInternal,iso_mtdt,iso_mtdt_search,isWithdrawn,last_name,name,ngram_query_search,ngram_simplequery_search,orphaned,parent_count,p_communities_id,p_communities_map,p_communities_name,p_group_id,p_group_map,p_group_name,range,rangeDescription,rangeDescription_ngram,rangeDescription_search,range_ngram,range_search,referrer_ngram,referrer_search,simple_query,simple_query_search,solr_update_time_stamp,storage_nb_of_bitstreams,storage_size,storage_statistics_type,subject_mtdt,subject_mtdt_search,text,userAgent_ngram,userAgent_search,version_id,workflowItemId
|
|
</span></span></code></pre></div><ul>
|
|
<li>Then I replaced the IDs with the UUID of the site homepage on DSpace 7 Test:</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ sed -i <span style="color:#e6db74">'s/"id":"-1"/"id":"3945ec23-2426-4fce-a2ea-48b38b91547f"/'</span> /tmp/statistics-fix-uuid.json
|
|
</span></span></code></pre></div><ul>
|
|
<li>I re-imported those records and I no longer see the “-1” IDs, but still get the same error in the log
|
|
<ul>
|
|
<li>I don’t understand, maybe there is some voodoo, so I rebooted the server</li>
|
|
<li>Hmm, no, it’s not a voodoo cache issue, so I really need to debug this:</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>2023-08-16 15:44:07,122 WARN dace8f96-f034-488e-b38c-9f2eb5d0e002 036b88e6-7548-4852-9646-f345ce3bfcc2 org.dspace.app.rest.exception.DSpaceApiExceptionControllerAdvice @ Request is invalid or incorrect (status:400 exception: Invalid UUID string: -1 at: java.base/java.util.UUID.fromString1(UUID.java:280))
|
|
</span></span></code></pre></div><ul>
|
|
<li>On a related note, I figured out that the root site already has a UUID in DSpace 6, and it’s exactly the one above (3945ec23-2426-4fce-a2ea-48b38b91547f)
|
|
<ul>
|
|
<li>I noticed it while looking at the <a href="https://cgspace.cgiar.org/rest/hierarchy">DSpace 6 REST API’s hierarchy page</a></li>
|
|
<li>So I can update these “-1” IDs with “type:5” in our production I think…</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<h2 id="2023-08-17">2023-08-17</h2>
|
|
<ul>
|
|
<li>I decided to update the “-1” IDs in Solr on DSpace 6
|
|
<ul>
|
|
<li>Unfortunately, in Solr there is no way to update only documents matching a query, so we have to export and re-import</li>
|
|
<li>I exported all documents with “type:5” (Homepage) and replaced the ID in the JSON:</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ chrt -b <span style="color:#ae81ff">0</span> ./run.sh -s http://localhost:8081/solr/statistics -a export -o /tmp/statistics-fix-uuid.json -f <span style="color:#e6db74">'type:5'</span> -k uid -S actingGroupId,actingGroupParentId,actorMemberGroupId,author_mtdt,author_mtdt_search,bitstreamCount,bitstreamId,complete_query,complete_query_search,containerBitstream,containerCollection,containerCommunity,containerItem,core_update_run_nb,countryCode_ngram,countryCode_search,cua_version,dateYear,dateYearMonth,file_id,filterquery,first_name,geoipcountrycode,geoIpCountryCode,group_id,group_map,group_name,ip_ngram,ip_search,isArchived,isInternal,iso_mtdt,iso_mtdt_search,isWithdrawn,last_name,name,ngram_query_search,ngram_simplequery_search,orphaned,parent_count,p_communities_id,p_communities_map,p_communities_name,p_group_id,p_group_map,p_group_name,range,rangeDescription,rangeDescription_ngram,rangeDescription_search,range_ngram,range_search,referrer_ngram,referrer_search,simple_query,simple_query_search,solr_update_time_stamp,storage_nb_of_bitstreams,storage_size,storage_statistics_type,subject_mtdt,subject_mtdt_search,text,userAgent_ngram,userAgent_search,version_id,workflowItemId
|
|
</span></span><span style="display:flex;"><span>$ sed -i <span style="color:#e6db74">'s/"id":"-1"/"id":"3945ec23-2426-4fce-a2ea-48b38b91547f"/'</span> /tmp/statistics-fix-uuid.json
|
|
</span></span></code></pre></div><ul>
|
|
<li>(Oops, skipping the fields above was not necessary, since I’m importing back into DSpace 6 where those fields exist)</li>
|
|
<li>Then I re-imported:</li>
|
|
</ul>
|
|
<pre tabindex="0"><code>$ ./run.sh -s http://localhost:8081/solr/statistics -a import -o /tmp/statistics-fix-uuid.json -k uid
|
|
</code></pre><ul>
|
|
<li>This worked, but I still see new records coming in that have “id:-1” so I will need to repeat this during the migration.</li>
|
|
<li>I also notice many stats records that have erroneous cities:
|
|
<ul>
|
|
<li><code>"city":"com.maxmind.geoip2.record.City [ {} ]"</code></li>
|
|
<li><code>"city":"com.maxmind.geoip2.record.City [ {\"geoname_id\":1002145,\"names\":{\"de\":\"George\",\"en\":\"George\",\"ru\":\"Джордж\",\"fr\":\"George\",\"ja\":\"ジョージ\"}} ]"</code></li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<h2 id="2023-08-18">2023-08-18</h2>
|
|
<ul>
|
|
<li>Export CGSpace to check for missing Initiative collection mappings</li>
|
|
</ul>
|
|
<h2 id="2023-08-19">2023-08-19</h2>
|
|
<ul>
|
|
<li>Start a harvest on AReS</li>
|
|
</ul>
|
|
<h2 id="2023-08-21">2023-08-21</h2>
|
|
<ul>
|
|
<li>Experiment with the DSpace 7 REST API
|
|
<ul>
|
|
<li>I wrote a Python script to benchmark harvesting all 100,000+ items using the <code>/api/discover/search/objects</code> endpoint 100 items at a time</li>
|
|
<li>I was able to harvest the entire 106,000 items in fifty-two minutes, which seems slow, but that’s about ten times faster than with the legacy REST API…</li>
|
|
<li>Still, I need to benchmark a bit more, as the item response doesn’t include collection mappings or thumbnails</li>
|
|
</ul>
|
|
</li>
|
|
<li>Reading the <a href="https://github.com/DSpace/RestContract/blob/main/README.md#etags--conditional-headers">API docs</a> it seems that we should be able to use the standard <code>If-Modified-Since</code> header for some endpoints
|
|
<ul>
|
|
<li>I tried it on the <code>/api/discover/search/objects</code> and <code>/api/core/items</code> endpoints, but apparently those don’t support this header because I don’t see a <code>Last-Modified</code> header in the response</li>
|
|
<li>According to the docs, it means that these endpoints indeed don’t support it…</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<h2 id="2023-08-22">2023-08-22</h2>
|
|
<ul>
|
|
<li>I was experimenting with the DSpace 7 REST API again
|
|
<ul>
|
|
<li>This time looking at the thumbnail responses in item endpoints</li>
|
|
<li>According to <a href="https://github.com/DSpace/RestContract/blob/main/items.md#main-thumbnail">the documentation</a> the API will respond with HTTP 200 if there is a thumbnail, and HTTP 204 if there is no content</li>
|
|
<li>That means we need to make the request before we can even find out!</li>
|
|
</ul>
|
|
</li>
|
|
<li>Tim on DSpace Slack pointed out the DSpace 7 REST API’s <a href="https://github.com/DSpace/RestContract/blob/main/projections.md">projections</a>
|
|
<ul>
|
|
<li>This means we can embed resources like thumbnail and owningCollection in the item (and other) requests, for example: <a href="https://dspace7test.ilri.org/server/api/discover/search/objects?dsoType=item&embed=thumbnail,owningCollection">https://dspace7test.ilri.org/server/api/discover/search/objects?dsoType=item&embed=thumbnail,owningCollection</a></li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<h2 id="2023-08-23">2023-08-23</h2>
|
|
<ul>
|
|
<li>I benchmarked the DSpace 7 REST API with the new embeds and it took four hours and seventeen minutes to get all 106,000 items on DSpace 7 Test
|
|
<ul>
|
|
<li>So this is much slower than the results I saw earlier this week, but maybe slightly faster than DSpace 6?</li>
|
|
</ul>
|
|
</li>
|
|
<li>Maria from Alliance contacted me to say they have agreed to use UN M.49 regions more strictly in TIP, so they want to replace our non-standard “Latin America” region with “Latin America and the Caribbean”, “Caribbean” and “Americas” on all Alliance outputs
|
|
<ul>
|
|
<li>I exported their community on CGSpace and fixed the metadata in OpenRefine</li>
|
|
</ul>
|
|
</li>
|
|
<li>I tried to run <code>dspace cleanup -v</code> on CGSpace, but got this error:</li>
|
|
</ul>
|
|
<pre tabindex="0"><code>Caused by: org.postgresql.util.PSQLException: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
|
|
Detail: Key (uuid)=(61bff7da-c8e3-420f-841c-ec5e8238d716) is still referenced from table "bundle".
|
|
</code></pre><ul>
|
|
<li>The solution, as always, is to delete those IDs manually in PostgreSQL:</li>
|
|
</ul>
|
|
<pre tabindex="0"><code>$ psql -d dspace -c "UPDATE bundle SET primary_bitstream_id=NULL WHERE primary_bitstream_id IN ('61bff7da-c8e3-420f-841c-ec5e8238d716');"
|
|
UPDATE 1
|
|
</code></pre><ul>
|
|
<li>I also tried to delete all users who haven’t logged in since 2017 using the groomer script, but it crashes due to those users still having items or workflows or whatever:</li>
|
|
</ul>
|
|
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ dspace dsrun org.dspace.eperson.Groomer -a -b 08/23/2017 -d
|
|
</span></span></code></pre></div><ul>
|
|
<li>I see that it is now <a href="https://github.com/DSpace/DSpace/pull/2229">possible in DSpace 7 to delete such users</a> so we will have to wait</li>
|
|
</ul>
|
|
<h2 id="2023-08-24">2023-08-24</h2>
|
|
<ul>
|
|
<li>I spent some time trying to get themes to extend in DSpace 7
|
|
<ul>
|
|
<li>I finally got a basic ILRI theme working, but there is a bug that causes theme components to get duplicated</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<h2 id="2023-08-25">2023-08-25</h2>
|
|
<ul>
|
|
<li>Meeting with Altmetric about the next phase of their integration with CGSpace</li>
|
|
<li>A bit of cleanup on CGSpace metadata
|
|
<ul>
|
|
<li>I fixed DOIs, licenses, dates, subjects, affiliations, titles, publishers, types, and titles in 1,240 items</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<h2 id="2023-08-26">2023-08-26</h2>
|
|
<ul>
|
|
<li>A few weeks ago we received a request from the Fruits and Vegetables Initiative saying that they’ve gotten approval to begin using the long name instead of the short one everywhere, apparently for SEO reasons
|
|
<ul>
|
|
<li>After communicating with PRMS and other teams working on systems using this metadata I finally updated them in CGSpace</li>
|
|
</ul>
|
|
</li>
|
|
<li>Export CGSpace to check for missing Initiative collection mappings</li>
|
|
<li>Start a harvest on AReS</li>
|
|
<li>I fixed ~200 titles with new lines, excessive whitespace, and Unicode FFFD characters
|
|
<ul>
|
|
<li>There are many more with 00A0, 200B, etc, but those need more careful inspection</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<h2 id="2023-08-28">2023-08-28</h2>
|
|
<ul>
|
|
<li>Day one of CGSpace partners meeting in Addis</li>
|
|
<li>Oh this is a game changer, I just realized that we can use Solr query syntax in the DSpace 7 REST API, so we can do this for example:</li>
|
|
</ul>
|
|
<pre tabindex="0"><code>https://dspace7test.ilri.org/server/api/discover/search/objects?query=lastModified%3A%5B2023-08-01T00%3A00%3A00Z%20TO%20%2A%5D
|
|
</code></pre><ul>
|
|
<li>Which is this query: <code>lastModified:[2023-08-01T00:00:00Z TO *]</code>
|
|
<ul>
|
|
<li>The queries need to be URL encoded of course</li>
|
|
</ul>
|
|
</li>
|
|
<li>Oh nice, and we can do the same for accession date:</li>
|
|
</ul>
|
|
<pre tabindex="0"><code>https://dspace7test.ilri.org/server/api/discover/search/objects?query=dc.date.accessioned_dt%3A%5B2023-08-01T00%3A00%3A00Z%20TO%20%2A%5D'
|
|
</code></pre><ul>
|
|
<li>That is this query: <code>dc.date.accessioned_dt:[2023-08-01T00:00:00Z TO *]</code>
|
|
<ul>
|
|
<li>We need to use the dt version of the accession date because that is the one that has a date type</li>
|
|
<li>This query give 290 results, which should be the items submitted in August!</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<h2 id="2023-08-29">2023-08-29</h2>
|
|
<ul>
|
|
<li>Day two of CGSpace partners meeting in Addis</li>
|
|
</ul>
|
|
<h2 id="2023-08-30">2023-08-30</h2>
|
|
<ul>
|
|
<li>Day three of CGSpace partners meeting in Addis</li>
|
|
<li>I did a lot of work on the CGSpace Angular theme for DSpace 7
|
|
<ul>
|
|
<li>Many changes to Discovery filters and search results</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<h2 id="2023-08-31">2023-08-31</h2>
|
|
<ul>
|
|
<li>Day four of CGSpace partners meeting in Addis</li>
|
|
<li>I removed the old Bioversity and CIAT subjects from Discovery facets on CGSpace
|
|
<ul>
|
|
<li>Maria and Leroy said they are no longer using them so we don’t need to keep indexing and displaying them</li>
|
|
</ul>
|
|
</li>
|
|
<li>I did a lot of work on the CGSpace Angular theme for DSpace 7
|
|
<ul>
|
|
<li>Now we have clickable keywords that go to Discovery instead of browse, as well as some new icons</li>
|
|
<li>We don’t need to use the clunky browse links to get clickable links any more so I will disable those</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<!-- raw HTML omitted -->
|
|
|
|
|
|
|
|
|
|
|
|
</article>
|
|
|
|
|
|
|
|
</div> <!-- /.blog-main -->
|
|
|
|
<aside class="col-sm-3 ml-auto blog-sidebar">
|
|
|
|
|
|
|
|
<section class="sidebar-module">
|
|
<h4>Recent Posts</h4>
|
|
<ol class="list-unstyled">
|
|
|
|
|
|
<li><a href="/cgspace-notes/2023-09/">September, 2023</a></li>
|
|
|
|
<li><a href="/cgspace-notes/2023-08/">August, 2023</a></li>
|
|
|
|
<li><a href="/cgspace-notes/2023-07/">July, 2023</a></li>
|
|
|
|
<li><a href="/cgspace-notes/2023-06/">June, 2023</a></li>
|
|
|
|
<li><a href="/cgspace-notes/2023-05/">May, 2023</a></li>
|
|
|
|
</ol>
|
|
</section>
|
|
|
|
|
|
|
|
|
|
<section class="sidebar-module">
|
|
<h4>Links</h4>
|
|
<ol class="list-unstyled">
|
|
|
|
<li><a href="https://cgspace.cgiar.org">CGSpace</a></li>
|
|
|
|
<li><a href="https://dspacetest.cgiar.org">DSpace Test</a></li>
|
|
|
|
<li><a href="https://github.com/ilri/DSpace">CGSpace @ GitHub</a></li>
|
|
|
|
</ol>
|
|
</section>
|
|
|
|
</aside>
|
|
|
|
|
|
</div> <!-- /.row -->
|
|
</div> <!-- /.container -->
|
|
|
|
|
|
|
|
<footer class="blog-footer">
|
|
<p dir="auto">
|
|
|
|
Blog template created by <a href="https://twitter.com/mdo">@mdo</a>, ported to Hugo by <a href='https://twitter.com/mralanorth'>@mralanorth</a>.
|
|
|
|
</p>
|
|
<p>
|
|
<a href="#">Back to top</a>
|
|
</p>
|
|
</footer>
|
|
|
|
|
|
</body>
|
|
|
|
</html>
|