Add notes for 2020-09-23
This commit is contained in:
parent
53cdf18622
commit
448097a539
GPG Key ID:
0FB860CC9C45B1B9
|
|
@ -344,6 +344,8 @@ DELETE 12
|
|||
dspace=# COMMIT;
|
||||
```
|
||||
|
||||
- I added all CG center- and CRP-specific subject fields and mapped them to `dc.subject` in AReS
|
||||
- After forcing a re-harvesting now the review status is much cleaner and the missing subjects are available
|
||||
- Last week Natalia from CIAT had asked me to download all the PDFs for a certain query:
|
||||
- items with status "Open Access"
|
||||
- items with type "Journal Article"
|
||||
|
|
@ -351,4 +353,40 @@ dspace=# COMMIT;
|
|||
- The resulting OpenSearch query is: https://cgspace.cgiar.org/open-search/discover?query=type:"Journal Article" AND status:"Open Access" AND Water Land Ecosystems trade offs&rpp=1
|
||||
- There were 241 results with a total of 208 PDFs, which I downloaded with my `get-wle-pdfs.py` script and shared to her via bashupload.com
|
||||
|
||||
## 2020-09-23
|
||||
|
||||
- Peter said he was having problems submitting items to CGSpace
|
||||
- On a hunch I looked at the PostgreSQL locks in Munin and indeed the normal issue with locks is back (though I haven't seen it in a few months?)
|
||||
|
||||
|
||||
|
||||
- Instead of restarting Tomcat I restarted the PostgreSQL service and then Peter said he was able to submit the item...
|
||||
- Experiment with doing direct queries for items in the [dspace-statistics-api](https://github.com/ilri/dspace-statistics-api)
|
||||
- I tested querying a handful of item UUIDs with a date range and returning their hits faceted by `id`
|
||||
- Assuming a list of item UUIDs was posted to the REST API we could prepare them for a Solr query by joining them into a string with "OR" and escaping the hyphens:
|
||||
|
||||
```
|
||||
...
|
||||
item_ids = ['0079470a-87a1-4373-beb1-b16e3f0c4d81', '007a9df1-0871-4612-8b28-5335982198cb']
|
||||
item_ids_str = ' OR '.join(item_ids).replace('-', '\-')
|
||||
...
|
||||
solr_query_params = {
|
||||
"q": f"id:({item_ids_str})",
|
||||
"fq": "type:2 AND isBot:false AND statistics_type:view AND time:[2020-01-01T00:00:00Z TO 2020-09-02T00:00:00Z]",
|
||||
"facet": "true",
|
||||
"facet.field": "id",
|
||||
"facet.mincount": 1,
|
||||
"facet.limit": 1,
|
||||
"facet.offset": 0,
|
||||
"stats": "true",
|
||||
"stats.field": "id",
|
||||
"stats.calcdistinct": "true",
|
||||
"shards": shards,
|
||||
"rows": 0,
|
||||
"wt": "json",
|
||||
}
|
||||
```
|
||||
|
||||
- The date range format for Solr is important, but it seems we only need to add `T00:00:00Z` to the normal ISO 8601 YYYY-MM-DD strings
|
||||
|
||||
<!-- vim: set sw=2 ts=2: -->
|
||||
|
|
|
|||
|
|
@ -25,7 +25,7 @@ I filed an issue on OpenRXV to make some minor edits to the admin UI: https://gi
|
|||
<meta property="og:type" content="article" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2020-09/" />
|
||||
<meta property="article:published_time" content="2020-09-02T15:35:54+03:00" />
|
||||
<meta property="article:modified_time" content="2020-09-22T12:38:53+03:00" />
|
||||
<meta property="article:modified_time" content="2020-09-22T14:14:18+03:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="September, 2020"/>
|
||||
|
|
@ -55,9 +55,9 @@ I filed an issue on OpenRXV to make some minor edits to the admin UI: https://gi
|
|||
"@type": "BlogPosting",
|
||||
"headline": "September, 2020",
|
||||
"url": "https://alanorth.github.io/cgspace-notes/2020-09/",
|
||||
"wordCount": "2380",
|
||||
"wordCount": "2608",
|
||||
"datePublished": "2020-09-02T15:35:54+03:00",
|
||||
"dateModified": "2020-09-22T12:38:53+03:00",
|
||||
"dateModified": "2020-09-22T14:14:18+03:00",
|
||||
"author": {
|
||||
"@type": "Person",
|
||||
"name": "Alan Orth"
|
||||
|
|
@ -539,6 +539,8 @@ dspace=# DELETE FROM metadatavalue WHERE text_value='Report' AND resource_type_i
|
|||
DELETE 12
|
||||
dspace=# COMMIT;
|
||||
</code></pre><ul>
|
||||
<li>I added all CG center- and CRP-specific subject fields and mapped them to <code>dc.subject</code> in AReS</li>
|
||||
<li>After forcing a re-harvesting now the review status is much cleaner and the missing subjects are available</li>
|
||||
<li>Last week Natalia from CIAT had asked me to download all the PDFs for a certain query:
|
||||
<ul>
|
||||
<li>items with status “Open Access”</li>
|
||||
|
|
@ -549,6 +551,46 @@ dspace=# COMMIT;
|
|||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<h2 id="2020-09-23">2020-09-23</h2>
|
||||
<ul>
|
||||
<li>Peter said he was having problems submitting items to CGSpace
|
||||
<ul>
|
||||
<li>On a hunch I looked at the PostgreSQL locks in Munin and indeed the normal issue with locks is back (though I haven’t seen it in a few months?)</li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<p><img src="/cgspace-notes/2020/09/postgres_connections_ALL-day.png" alt="PostgreSQL connections day"></p>
|
||||
<ul>
|
||||
<li>Instead of restarting Tomcat I restarted the PostgreSQL service and then Peter said he was able to submit the item…</li>
|
||||
<li>Experiment with doing direct queries for items in the <a href="https://github.com/ilri/dspace-statistics-api">dspace-statistics-api</a>
|
||||
<ul>
|
||||
<li>I tested querying a handful of item UUIDs with a date range and returning their hits faceted by <code>id</code></li>
|
||||
<li>Assuming a list of item UUIDs was posted to the REST API we could prepare them for a Solr query by joining them into a string with “OR” and escaping the hyphens:</li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<pre><code>...
|
||||
item_ids = ['0079470a-87a1-4373-beb1-b16e3f0c4d81', '007a9df1-0871-4612-8b28-5335982198cb']
|
||||
item_ids_str = ' OR '.join(item_ids).replace('-', '\-')
|
||||
...
|
||||
solr_query_params = {
|
||||
"q": f"id:({item_ids_str})",
|
||||
"fq": "type:2 AND isBot:false AND statistics_type:view AND time:[2020-01-01T00:00:00Z TO 2020-09-02T00:00:00Z]",
|
||||
"facet": "true",
|
||||
"facet.field": "id",
|
||||
"facet.mincount": 1,
|
||||
"facet.limit": 1,
|
||||
"facet.offset": 0,
|
||||
"stats": "true",
|
||||
"stats.field": "id",
|
||||
"stats.calcdistinct": "true",
|
||||
"shards": shards,
|
||||
"rows": 0,
|
||||
"wt": "json",
|
||||
}
|
||||
</code></pre><ul>
|
||||
<li>The date range format for Solr is important, but it seems we only need to add <code>T00:00:00Z</code> to the normal ISO 8601 YYYY-MM-DD strings</li>
|
||||
</ul>
|
||||
<!-- raw HTML omitted -->
|
||||
|
||||
|
||||
|
|
|
|||
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
|
|
@ -9,7 +9,7 @@
|
|||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/" />
|
||||
<meta property="og:updated_time" content="2020-09-22T12:38:53+03:00" />
|
||||
<meta property="og:updated_time" content="2020-09-22T14:14:18+03:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="Categories"/>
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
|
||||
<meta property="og:updated_time" content="2020-09-22T12:38:53+03:00" />
|
||||
<meta property="og:updated_time" content="2020-09-22T14:14:18+03:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="Notes"/>
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
|
||||
<meta property="og:updated_time" content="2020-09-22T12:38:53+03:00" />
|
||||
<meta property="og:updated_time" content="2020-09-22T14:14:18+03:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="Notes"/>
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
|
||||
<meta property="og:updated_time" content="2020-09-22T12:38:53+03:00" />
|
||||
<meta property="og:updated_time" content="2020-09-22T14:14:18+03:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="Notes"/>
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
|
||||
<meta property="og:updated_time" content="2020-09-22T12:38:53+03:00" />
|
||||
<meta property="og:updated_time" content="2020-09-22T14:14:18+03:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="Notes"/>
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
|
||||
<meta property="og:updated_time" content="2020-09-22T12:38:53+03:00" />
|
||||
<meta property="og:updated_time" content="2020-09-22T14:14:18+03:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="CGSpace Notes"/>
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
|
||||
<meta property="og:updated_time" content="2020-09-22T12:38:53+03:00" />
|
||||
<meta property="og:updated_time" content="2020-09-22T14:14:18+03:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="CGSpace Notes"/>
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
|
||||
<meta property="og:updated_time" content="2020-09-22T12:38:53+03:00" />
|
||||
<meta property="og:updated_time" content="2020-09-22T14:14:18+03:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="CGSpace Notes"/>
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
|
||||
<meta property="og:updated_time" content="2020-09-22T12:38:53+03:00" />
|
||||
<meta property="og:updated_time" content="2020-09-22T14:14:18+03:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="CGSpace Notes"/>
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
|
||||
<meta property="og:updated_time" content="2020-09-22T12:38:53+03:00" />
|
||||
<meta property="og:updated_time" content="2020-09-22T14:14:18+03:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="CGSpace Notes"/>
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
|
||||
<meta property="og:updated_time" content="2020-09-22T12:38:53+03:00" />
|
||||
<meta property="og:updated_time" content="2020-09-22T14:14:18+03:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="CGSpace Notes"/>
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
|
||||
<meta property="og:updated_time" content="2020-09-22T12:38:53+03:00" />
|
||||
<meta property="og:updated_time" content="2020-09-22T14:14:18+03:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="CGSpace Notes"/>
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
|
||||
<meta property="og:updated_time" content="2020-09-22T12:38:53+03:00" />
|
||||
<meta property="og:updated_time" content="2020-09-22T14:14:18+03:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="Posts"/>
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
|
||||
<meta property="og:updated_time" content="2020-09-22T12:38:53+03:00" />
|
||||
<meta property="og:updated_time" content="2020-09-22T14:14:18+03:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="Posts"/>
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
|
||||
<meta property="og:updated_time" content="2020-09-22T12:38:53+03:00" />
|
||||
<meta property="og:updated_time" content="2020-09-22T14:14:18+03:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="Posts"/>
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
|
||||
<meta property="og:updated_time" content="2020-09-22T12:38:53+03:00" />
|
||||
<meta property="og:updated_time" content="2020-09-22T14:14:18+03:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="Posts"/>
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
|
||||
<meta property="og:updated_time" content="2020-09-22T12:38:53+03:00" />
|
||||
<meta property="og:updated_time" content="2020-09-22T14:14:18+03:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="Posts"/>
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
|
||||
<meta property="og:updated_time" content="2020-09-22T12:38:53+03:00" />
|
||||
<meta property="og:updated_time" content="2020-09-22T14:14:18+03:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="Posts"/>
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
|
||||
<meta property="og:updated_time" content="2020-09-22T12:38:53+03:00" />
|
||||
<meta property="og:updated_time" content="2020-09-22T14:14:18+03:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="Posts"/>
|
||||
|
|
|
|||
|
|
@ -4,27 +4,27 @@
|
|||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/categories/</loc>
|
||||
<lastmod>2020-09-22T12:38:53+03:00</lastmod>
|
||||
<lastmod>2020-09-22T14:14:18+03:00</lastmod>
|
||||
</url>
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/</loc>
|
||||
<lastmod>2020-09-22T12:38:53+03:00</lastmod>
|
||||
<lastmod>2020-09-22T14:14:18+03:00</lastmod>
|
||||
</url>
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
|
||||
<lastmod>2020-09-22T12:38:53+03:00</lastmod>
|
||||
<lastmod>2020-09-22T14:14:18+03:00</lastmod>
|
||||
</url>
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
|
||||
<lastmod>2020-09-22T12:38:53+03:00</lastmod>
|
||||
<lastmod>2020-09-22T14:14:18+03:00</lastmod>
|
||||
</url>
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/2020-09/</loc>
|
||||
<lastmod>2020-09-22T12:38:53+03:00</lastmod>
|
||||
<lastmod>2020-09-22T14:14:18+03:00</lastmod>
|
||||
</url>
|
||||
|
||||
<url>
|
||||
|
|
|
|||
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
Loading…
Reference in New Issue