alanorth/cgspace-notes
1
0
Fork 0
mirror of https://github.com/alanorth/cgspace-notes.git synced 2024-11-24 23:50:17 +01:00
Code Issues Projects Releases Wiki Activity

Add notes for 2018-02-27

Browse Source
This commit is contained in:
Alan Orth 2018-02-27 17:34:48 +02:00
parent 3e58e2f81d
commit 703f92a005
Signed by: alanorth
GPG Key ID: 0FB860CC9C45B1B9
44 changed files with 140 additions and 50 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

42
content/post/2018-02.md
View File

@ -903,3 +903,45 @@ Nor Azwadi: 0000-0001-9634-1958
- I don't know why it would take so long, but this logic kinda makes sense
- I think I should increase the `removeAbandonedTimeout` from 90 to something like 180 and continue observing
- I also reduced the timeout for the API pool back to 60 because those interfaces are only used by bots
## 2018-02-27
- Peter is still having problems with "Socket closed" on his submissions page
- I have disabled `removeAbandoned` for now because that's the only thing I changed in the last few weeks since he started having issues
- I think the real line of logic to follow here is why the submissions page is so slow for him (presumably because of loading all his submissions?)
- I need to see which SQL queries are run during that time
- And only a few hours after I disabled the `removeAbandoned` thing CGSpace went down and lo and behold, there were 264 connections, most of which were idle:
```
$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
5 dspaceApi
279 dspaceWeb
$ psql -c 'select * from pg_stat_activity' | grep dspaceWeb | grep -c "idle in transaction"
218
```
- So I'm re-enabling the `removeAbandoned` setting
- I grabbed a snapshot of the active connections in `pg_stat_activity` for all queries running longer than 2 minutes:
```
dspace=# \copy (SELECT now() - query_start as "runtime", application_name, usename, datname, waiting, state, query
FROM pg_stat_activity
WHERE now() - query_start > '2 minutes'::interval
ORDER BY runtime DESC) to /tmp/2018-02-27-postgresql.txt
COPY 263
```
- 100 of these idle in transaction connections are the following query:
```
SELECT * FROM resourcepolicy WHERE resource_type_id= $1 AND resource_id= $2 AND action_id= $3
```
- ... but according to the [pg_locks documentation](https://www.postgresql.org/docs/9.5/static/view-pg-locks.html) I should have done this to correlate the locks with the activity:
```
SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;
```
- Finally finished the [orcid-authority-to-item.py](https://gist.github.com/alanorth/6d7489b50f06a6a1f04ae1c8b899cb6e) script!
- It successfully mapped 2600 ORCID identifiers to items!

2
docs/2015-11/index.html
View File

@ -53,7 +53,7 @@ $ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspac
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2015-12/index.html
View File

@ -55,7 +55,7 @@ Replace lzop with xz in log compression cron jobs on DSpace Test—it uses less
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2016-01/index.html
View File

@ -45,7 +45,7 @@ Update GitHub wiki for documentation of maintenance tasks.
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2016-02/index.html
View File

@ -59,7 +59,7 @@ Also, lots of things like &ldquo;COTE D`LVOIRE&rdquo; and &ldquo;COTE D IVOIRE&r
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2016-03/index.html
View File

@ -45,7 +45,7 @@ Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Ja
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2016-04/index.html
View File

@ -49,7 +49,7 @@ Also, I noticed the checker log has some errors we should pay attention to:
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2016-05/index.html
View File

@ -53,7 +53,7 @@ There are 3,000 IPs accessing the REST API in a 24-hour period!
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2016-06/index.html
View File

@ -51,7 +51,7 @@ Working on second phase of metadata migration, looks like this will work for mov
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2016-07/index.html
View File

@ -67,7 +67,7 @@ In this case the select query was showing 95 results before the update
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2016-08/index.html
View File

@ -61,7 +61,7 @@ $ git rebase -i dspace-5.5
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2016-09/index.html
View File

@ -53,7 +53,7 @@ $ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b &quot;dc=cgiarad,dc=or
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2016-10/index.html
View File

@ -61,7 +61,7 @@ I exported a random item&rsquo;s metadata as CSV, deleted all columns except id
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2016-11/index.html
View File

@ -45,7 +45,7 @@ Add dc.type to the output options for Atmire&rsquo;s Listings and Reports module
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2016-12/index.html
View File

@ -69,7 +69,7 @@ Another worrying error from dspace.log is:
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2017-01/index.html
View File

@ -45,7 +45,7 @@ I asked on the dspace-tech mailing list because it seems to be broken, and actua
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2017-02/index.html
View File

@ -73,7 +73,7 @@ Looks like we&rsquo;ll be using cg.identifier.ccafsprojectpii as the field name
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2017-03/index.html
View File

@ -77,7 +77,7 @@ $ identify ~/Desktop/alc_contrastes_desafios.jpg
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2017-04/index.html
View File

@ -63,7 +63,7 @@ $ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p &quot;ImageMagick PDF Th
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2017-05/index.html
View File

@ -29,7 +29,7 @@
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="May, 2017"/>
<meta name="twitter:description" content="2017-05-01 ICARDA apparently started working on CG Core on their MEL repository They have done a few cg.* fields, but not very consistent and even copy some of CGSpace items: https://mel.cgiar.org/xmlui/handle/20.500.11766/6911?show=full https://cgspace.cgiar.org/handle/10568/73683 2017-05-02 Atmire got back about the Workflow Statistics issue, and apparently it&rsquo;s a bug in the CUA module so they will send us a pull request 2017-05-04 Sync DSpace Test with database and assetstore from CGSpace Re-deploy DSpace Test with Atmire&rsquo;s CUA patch for workflow statistics, run system updates, and restart the server Now I can see the workflow statistics and am able to select users, but everything returns 0 items Megan says there are still some mapped items are not appearing since last week, so I forced a full index-discovery -b Need to remember to check if the collection has more items (currently 39 on CGSpace, but 118 on the freshly reindexed DSPace Test) tomorrow: https://cgspace."/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2017-06/index.html
View File

@ -29,7 +29,7 @@
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="June, 2017"/>
<meta name="twitter:description" content="2017-06-01 After discussion with WLE and CGSpace content people, we decided to just add one metadata field for the WLE Research Themes The cg.identifier.wletheme field will be used for both Phase I and Phase II Research Themes Then we&rsquo;ll create a new sub-community for Phase II and create collections for the research themes there The current &ldquo;Research Themes&rdquo; community will be renamed to &ldquo;WLE Phase I Research Themes&rdquo; Tagged all items in the current Phase I collections with their appropriate themes Create pull request to add Phase II research themes to the submission form: #328 Add cg."/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2017-07/index.html
View File

@ -57,7 +57,7 @@ We can use PostgreSQL&rsquo;s extended output format (-x) plus sed to format the
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2017-08/index.html
View File

@ -77,7 +77,7 @@ Then I cleaned up the author authorities and HTML characters in OpenRefine and s
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2017-09/index.html
View File

@ -53,7 +53,7 @@ Ask Sisay to clean up the WLE approvers a bit, as Marianne&rsquo;s user account
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2017-10/index.html
View File

@ -57,7 +57,7 @@ Add Katherine Lutz to the groups for content submission and edit steps of the CG
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2017-11/index.html
View File

@ -77,7 +77,7 @@ COPY 54701
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2017-12/index.html
View File

@ -47,7 +47,7 @@ The list of connections to XMLUI and REST API for today:
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/2018-01/index.html
View File

@ -185,7 +185,7 @@ Danny wrote to ask for help renewing the wildcard ilri.org certificate and I adv
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

56
docs/2018-02/index.html
View File

@ -23,7 +23,7 @@ I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu&rsquo;s munin-pl
<meta property="article:published_time" content="2018-02-01T16:28:54&#43;02:00"/>
<meta property="article:modified_time" content="2018-02-26T16:41:28&#43;02:00"/>
<meta property="article:modified_time" content="2018-02-26T18:12:27&#43;02:00"/>
@ -47,7 +47,7 @@ I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu&rsquo;s munin-pl
"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />
@ -57,9 +57,9 @@ I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu&rsquo;s munin-pl
"@type": "BlogPosting",
"headline": "February, 2018",
"url": "https://alanorth.github.io/cgspace-notes/2018-02/",
"wordCount": "5609",
"wordCount": "5873",
"datePublished": "2018-02-01T16:28:54&#43;02:00",
"dateModified": "2018-02-26T16:41:28&#43;02:00",
"dateModified": "2018-02-26T18:12:27&#43;02:00",
"author": {
"@type": "Person",
"name": "Alan Orth"
@ -1136,6 +1136,54 @@ Nor Azwadi: 0000-0001-9634-1958
<li>I also reduced the timeout for the API pool back to 60 because those interfaces are only used by bots</li>
</ul>
<h2 id="2018-02-27">2018-02-27</h2>
<ul>
<li>Peter is still having problems with &ldquo;Socket closed&rdquo; on his submissions page</li>
<li>I have disabled <code>removeAbandoned</code> for now because that&rsquo;s the only thing I changed in the last few weeks since he started having issues</li>
<li>I think the real line of logic to follow here is why the submissions page is so slow for him (presumably because of loading all his submissions?)</li>
<li>I need to see which SQL queries are run during that time</li>
<li>And only a few hours after I disabled the <code>removeAbandoned</code> thing CGSpace went down and lo and behold, there were 264 connections, most of which were idle:</li>
</ul>
<pre><code>$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
5 dspaceApi
279 dspaceWeb
$ psql -c 'select * from pg_stat_activity' | grep dspaceWeb | grep -c &quot;idle in transaction&quot;
218
</code></pre>
<ul>
<li>So I&rsquo;m re-enabling the <code>removeAbandoned</code> setting</li>
<li>I grabbed a snapshot of the active connections in <code>pg_stat_activity</code> for all queries running longer than 2 minutes:</li>
</ul>
<pre><code>dspace=# \copy (SELECT now() - query_start as &quot;runtime&quot;, application_name, usename, datname, waiting, state, query
FROM pg_stat_activity
WHERE now() - query_start &gt; '2 minutes'::interval
ORDER BY runtime DESC) to /tmp/2018-02-27-postgresql.txt
COPY 263
</code></pre>
<ul>
<li>100 of these idle in transaction connections are the following query:</li>
</ul>
<pre><code>SELECT * FROM resourcepolicy WHERE resource_type_id= $1 AND resource_id= $2 AND action_id= $3
</code></pre>
<ul>
<li>&hellip; but according to the <a href="https://www.postgresql.org/docs/9.5/static/view-pg-locks.html">pg_locks documentation</a> I should have done this to correlate the locks with the activity:</li>
</ul>
<pre><code>SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;
</code></pre>
<ul>
<li>Finally finished the <a href="https://gist.github.com/alanorth/6d7489b50f06a6a1f04ae1c8b899cb6e">orcid-authority-to-item.py</a> script!</li>
<li>It successfully mapped 2600 ORCID identifiers to items!</li>
</ul>

2
docs/404.html
View File

@ -26,7 +26,7 @@
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="404 Page not found"/>
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/categories/index.html
View File

@ -26,7 +26,7 @@
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Categories"/>
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/categories/notes/index.html
View File

@ -26,7 +26,7 @@
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Notes"/>
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/cgiar-library-migration/index.html
View File

@ -29,7 +29,7 @@
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="CGIAR Library Migration"/>
<meta name="twitter:description" content="Notes on the migration of the CGIAR Library to CGSpace"/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/index.html
View File

@ -26,7 +26,7 @@
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="CGSpace Notes"/>
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/page/2/index.html
View File

@ -26,7 +26,7 @@
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="CGSpace Notes"/>
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/page/3/index.html
View File

@ -26,7 +26,7 @@
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="CGSpace Notes"/>
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/post/index.html
View File

@ -26,7 +26,7 @@
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Posts"/>
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/post/page/2/index.html
View File

@ -26,7 +26,7 @@
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Posts"/>
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/post/page/3/index.html
View File

@ -26,7 +26,7 @@
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Posts"/>
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

10
docs/sitemap.xml
View File

@ -4,7 +4,7 @@
<url>
<loc>https://alanorth.github.io/cgspace-notes/2018-02/</loc>
<lastmod>2018-02-26T16:41:28+02:00</lastmod>
<lastmod>2018-02-26T18:12:27+02:00</lastmod>
</url>
<url>
@ -149,7 +149,7 @@
<url>
<loc>https://alanorth.github.io/cgspace-notes/</loc>
<lastmod>2018-02-26T16:41:28+02:00</lastmod>
<lastmod>2018-02-26T18:12:27+02:00</lastmod>
<priority>0</priority>
</url>
@ -160,7 +160,7 @@
<url>
<loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
<lastmod>2018-02-26T16:41:28+02:00</lastmod>
<lastmod>2018-02-26T18:12:27+02:00</lastmod>
<priority>0</priority>
</url>
@ -172,13 +172,13 @@
<url>
<loc>https://alanorth.github.io/cgspace-notes/post/</loc>
<lastmod>2018-02-26T16:41:28+02:00</lastmod>
<lastmod>2018-02-26T18:12:27+02:00</lastmod>
<priority>0</priority>
</url>
<url>
<loc>https://alanorth.github.io/cgspace-notes/tags/</loc>
<lastmod>2018-02-26T16:41:28+02:00</lastmod>
<lastmod>2018-02-26T18:12:27+02:00</lastmod>
<priority>0</priority>
</url>

2
docs/tags/index.html
View File

@ -26,7 +26,7 @@
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Tags"/>
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/tags/notes/index.html
View File

@ -26,7 +26,7 @@
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Notes"/>
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/tags/notes/page/2/index.html
View File

@ -26,7 +26,7 @@
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Notes"/>
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />

2
docs/tags/notes/page/3/index.html
View File

@ -26,7 +26,7 @@
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Notes"/>
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
<meta name="generator" content="Hugo 0.36.1" />
<meta name="generator" content="Hugo 0.37" />