<li>I finally got around to working on Peter’s cleanups for affiliations, authors, and donors from last week
<ul>
<li>I did some minor cleanups myself and applied them to CGSpace</li>
</ul>
</li>
<li>Start working on some batch uploads for IFPRI</li>
</ul>
<h2id="2023-08-04">2023-08-04</h2>
<ul>
<li>Minor cleanups on IFPRI’s batch uploads
<ul>
<li>I also did a duplicate check and found thirteen items that seem to be duplicates, so I sent them to Leigh to check</li>
</ul>
</li>
<li>I read this <ahref="https://www.endpointdev.com/blog/2012/06/logstatement-postgres-all-full-logging/">interesting blog post about PostgreSQL’s <code>log_statement</code> function</a>
<ul>
<li>Someone pointed out that this also lets you take advantage of <ahref="https://github.com/darold/pgbadger">PgBadger</a> analysis</li>
<li>I enabled statement logging on DSpace Test and I will check it in a few days</li>
</ul>
</li>
<li>Reading about DSpace 7 REST API again
<ul>
<li>Here is how to get the first page of 100 items: <ahref="https://dspace7test.ilri.org/server/api/discover/search/objects?dsoType=item&page=1&size=100">https://dspace7test.ilri.org/server/api/discover/search/objects?dsoType=item&page=1&size=100</a></li>
<li>I really want to benchmark this to see how fast we can get all the pages</li>
<li>Another thing I notice is that the bitstreams are not here, so that will be an extra call…</li>
<li>I’m checking the PostgreSQL logs now that statement logging has been enabled for a few days on DSpace Test
<ul>
<li>I see the logs are about 7 or 8 GB, which is larger than expected—and this is the test server!</li>
<li>I will now play with pgbadger to see if it gives any useful insights</li>
<li>Hmm, it sems the <code>log_statement</code> advice was old as pgbadger itself says:</li>
</ul>
</li>
</ul>
<blockquote>
<p>Do not enable log_statement as its log format will not be parsed by pgBadger.</p>
</blockquote>
<p>… and:</p>
<blockquote>
<p>Warning: Do not enable both log_min_duration_statement, log_duration and log_statement all together, this will result in wrong counter values. Note that this will also increase drastically the size of your log. log_min_duration_statement should always be preferred.</p>
</blockquote>
<ul>
<li>So we need to follow pgbadger’s instructions rather to get a suitable log file
<ul>
<li>After enabling the new settings I see that our log file is going to be reaallllly big… hmmmm will check tomorrow morning</li>
<li>Ideally we would run this incremental report every day on the postgresql-14-main.log.1 aka yesterday’s version of the log file after it is rotated
<ul>
<li>Now I have to see how large the file will be…</li>
</ul>
</li>
<li>I did some final updates to the ninety IFPRI records and uploaded them to DSpace Test first, then to CGSpace</li>
