Add notes for 2017-10-26

This commit is contained in:
Alan Orth 2017-10-26 17:23:36 +03:00
parent ee2a713c2a
commit 85b84e6458
Signed by: alanorth
GPG Key ID: 0FB860CC9C45B1B9
4 changed files with 72 additions and 14 deletions

View File

@ -159,3 +159,30 @@ http://library.cgiar.org/browse?value=Intellectual%20Assets%20Reports&type=subje
- Re-deploy CGSpace from latest `5_x-prod` (adds ISI Journal to search filters and adds Discovery index for CGIAR Library `systemsubject`)
- Deploy nginx redirect fixes to catch CGIAR Library browse links (redirect to their community and translate subject→systemsubject)
- Run migration of CGSpace server (linode18) for Linode security alert, which took 42 minutes of downtime
## 2017-10-26
- In the last 24 hours we've gotten a few alerts from Linode that there was high CPU and outgoing traffic on CGSpace
- Uptime Robot even noticed CGSpace go "down" for a few minutes
- In other news, I was trying to look at a question about stats raised by Magdalena and then CGSpace went down due to SQL connection pool
- Looking at the PostgreSQL activity I see there are 93 connections, but after a minute or two they went down and CGSpace came back up
- Annnd I reloaded the Atmire Usage Stats module and the connections shot back up and CGSpace went down again
- Still not sure where the load is coming from right now, but it's clear why there were so many alerts yesterday on the 25th!
```
# grep -o -E 'session_id=[A-Z0-9]{32}' dspace.log.2017-10-25 | sort -n | uniq | wc -l
18022
```
- Compared to other days there were two or three times the number of requests yesterday!
```
# grep -o -E 'session_id=[A-Z0-9]{32}' dspace.log.2017-10-23 | sort -n | uniq | wc -l
3141
# grep -o -E 'session_id=[A-Z0-9]{32}' dspace.log.2017-10-26 | sort -n | uniq | wc -l
7851
```
- I still have no idea what was causing the load to go up today
- I finally investigated Magdalena's issue with the item download stats and now I can't reproduce it: I get the same number of downloads reported in the stats widget on the item page, the "Most Popular Items" page, and in Usage Stats
- I think it might have been an issue with the statistics not being fresh

View File

@ -28,7 +28,7 @@ Add Katherine Lutz to the groups for content sumission and edit steps of the CGI
<meta property="article:published_time" content="2017-10-01T08:07:54&#43;03:00"/>
<meta property="article:modified_time" content="2017-10-22T23:51:48&#43;03:00"/>
<meta property="article:modified_time" content="2017-10-23T09:58:33&#43;03:00"/>
@ -66,9 +66,9 @@ Add Katherine Lutz to the groups for content sumission and edit steps of the CGI
"@type": "BlogPosting",
"headline": "October, 2017",
"url": "https://alanorth.github.io/cgspace-notes/2017-10/",
"wordCount": "1194",
"wordCount": "1451",
"datePublished": "2017-10-01T08:07:54&#43;03:00",
"dateModified": "2017-10-22T23:51:48&#43;03:00",
"dateModified": "2017-10-23T09:58:33&#43;03:00",
"author": {
"@type": "Person",
"name": "Alan Orth"
@ -318,6 +318,37 @@ Add Katherine Lutz to the groups for content sumission and edit steps of the CGI
<li>Run migration of CGSpace server (linode18) for Linode security alert, which took 42 minutes of downtime</li>
</ul>
<h2 id="2017-10-26">2017-10-26</h2>
<ul>
<li>In the last 24 hours we&rsquo;ve gotten a few alerts from Linode that there was high CPU and outgoing traffic on CGSpace</li>
<li>Uptime Robot even noticed CGSpace go &ldquo;down&rdquo; for a few minutes</li>
<li>In other news, I was trying to look at a question about stats raised by Magdalena and then CGSpace went down due to SQL connection pool</li>
<li>Looking at the PostgreSQL activity I see there are 93 connections, but after a minute or two they went down and CGSpace came back up</li>
<li>Annnd I reloaded the Atmire Usage Stats module and the connections shot back up and CGSpace went down again</li>
<li>Still not sure where the load is coming from right now, but it&rsquo;s clear why there were so many alerts yesterday on the 25th!</li>
</ul>
<pre><code># grep -o -E 'session_id=[A-Z0-9]{32}' dspace.log.2017-10-25 | sort -n | uniq | wc -l
18022
</code></pre>
<ul>
<li>Compared to other days there were two or three times the number of requests yesterday!</li>
</ul>
<pre><code># grep -o -E 'session_id=[A-Z0-9]{32}' dspace.log.2017-10-23 | sort -n | uniq | wc -l
3141
# grep -o -E 'session_id=[A-Z0-9]{32}' dspace.log.2017-10-26 | sort -n | uniq | wc -l
7851
</code></pre>
<ul>
<li>I still have no idea what was causing the load to go up today</li>
<li>I finally investigated Magdalena&rsquo;s issue with the item download stats and now I can&rsquo;t reproduce it: I get the same number of downloads reported in the stats widget on the item page, the &ldquo;Most Popular Items&rdquo; page, and in Usage Stats</li>
<li>I think it might have been an issue with the statistics not being fresh</li>
</ul>

View File

@ -28,7 +28,7 @@ Disallow: /cgspace-notes/2015-12/
Disallow: /cgspace-notes/2015-11/
Disallow: /cgspace-notes/
Disallow: /cgspace-notes/categories/
Disallow: /cgspace-notes/tags/notes/
Disallow: /cgspace-notes/categories/notes/
Disallow: /cgspace-notes/tags/notes/
Disallow: /cgspace-notes/post/
Disallow: /cgspace-notes/tags/

View File

@ -4,7 +4,7 @@
<url>
<loc>https://alanorth.github.io/cgspace-notes/2017-10/</loc>
<lastmod>2017-10-22T23:51:48+03:00</lastmod>
<lastmod>2017-10-23T09:58:33+03:00</lastmod>
</url>
<url>
@ -129,7 +129,7 @@
<url>
<loc>https://alanorth.github.io/cgspace-notes/</loc>
<lastmod>2017-10-22T23:51:48+03:00</lastmod>
<lastmod>2017-10-23T09:58:33+03:00</lastmod>
<priority>0</priority>
</url>
@ -138,27 +138,27 @@
<priority>0</priority>
</url>
<url>
<loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
<lastmod>2017-10-22T23:51:48+03:00</lastmod>
<priority>0</priority>
</url>
<url>
<loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
<lastmod>2017-09-28T12:00:49+03:00</lastmod>
<priority>0</priority>
</url>
<url>
<loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
<lastmod>2017-10-23T09:58:33+03:00</lastmod>
<priority>0</priority>
</url>
<url>
<loc>https://alanorth.github.io/cgspace-notes/post/</loc>
<lastmod>2017-10-22T23:51:48+03:00</lastmod>
<lastmod>2017-10-23T09:58:33+03:00</lastmod>
<priority>0</priority>
</url>
<url>
<loc>https://alanorth.github.io/cgspace-notes/tags/</loc>
<lastmod>2017-10-22T23:51:48+03:00</lastmod>
<lastmod>2017-10-23T09:58:33+03:00</lastmod>
<priority>0</priority>
</url>