mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2024-09-30 06:04:16 +02:00
51 lines
2.3 KiB
Markdown
51 lines
2.3 KiB
Markdown
---
|
|
title: "August, 2023"
|
|
date: 2023-08-03T11:18:36+03:00
|
|
author: "Alan Orth"
|
|
categories: ["Notes"]
|
|
---
|
|
|
|
## 2023-08-03
|
|
|
|
- I finally got around to working on Peter's cleanups for affiliations, authors, and donors from last week
|
|
- I did some minor cleanups myself and applied them to CGSpace
|
|
- Start working on some batch uploads for IFPRI
|
|
|
|
<!--more-->
|
|
|
|
## 2023-08-04
|
|
|
|
- Minor cleanups on IFPRI's batch uploads
|
|
- I also did a duplicate check and found thirteen items that seem to be duplicates, so I sent them to Leigh to check
|
|
- I read this [interesting blog post about PostgreSQL's `log_statement` function](https://www.endpointdev.com/blog/2012/06/logstatement-postgres-all-full-logging/)
|
|
- Someone pointed out that this also lets you take advantage of [PgBadger](https://github.com/darold/pgbadger) analysis
|
|
- I enabled statement logging on DSpace Test and I will check it in a few days
|
|
- Reading about DSpace 7 REST API again
|
|
- Here is how to get the first page of 100 items: https://dspace7test.ilri.org/server/api/discover/search/objects?dsoType=item&page=1&size=100
|
|
- I really want to benchmark this to see how fast we can get all the pages
|
|
- Another thing I notice is that the bitstreams are not here, so that will be an extra call...
|
|
|
|
## 2023-08-05
|
|
|
|
- Export CGSpace to check for missing Initiative collection mappings
|
|
- Start a harvest on AReS
|
|
|
|
## 2023-08-07
|
|
|
|
- I'm checking the PostgreSQL logs now that statement logging has been enabled for a few days on DSpace Test
|
|
- I see the logs are about 7 or 8 GB, which is larger than expected—and this is the test server!
|
|
- I will now play with pgbadger to see if it gives any useful insights
|
|
- Hmm, it sems the `log_statement` advice was old as pgbadger itself says:
|
|
|
|
> Do not enable log_statement as its log format will not be parsed by pgBadger.
|
|
|
|
... and:
|
|
|
|
> Warning: Do not enable both log_min_duration_statement, log_duration and log_statement all together, this will result in wrong counter values. Note that this will also increase drastically the size of your log. log_min_duration_statement should always be preferred.
|
|
|
|
- So we need to follow pgbadger's instructions rather to get a suitable log file
|
|
- After enabling the new settings I see that our log file is going to be reaallllly big... hmmmm will check tomorrow morning
|
|
- More work on the IFPRI batch uploads
|
|
|
|
<!-- vim: set sw=2 ts=2: -->
|