alanorth/cgspace-notes
1
0
Fork 0
mirror of https://github.com/alanorth/cgspace-notes.git synced 2024-11-22 14:45:03 +01:00
Code Issues Projects Releases Wiki Activity

Add notes for 2019-04-09

Browse Source
This commit is contained in:
Alan Orth 2019-04-09 09:33:52 +03:00
parent b64cc467ad
commit b890d3cb03
Signed by: alanorth
GPG Key ID: 0FB860CC9C45B1B9
3 changed files with 107 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

47
content/posts/2019-04.md
View File

@ -507,4 +507,51 @@ org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exe
370 51.254.16.223
```
## 2019-04-09
- Linode sent an alert that CGSpace (linode18) was 440% CPU for the last two hours this morning
- Here are the top IPs in the web server logs around that time:
```
# zcat --force /var/log/nginx/{rest,oai}.log /var/log/nginx/{rest,oai}.log.1 | grep -E "09/Apr/2019:(06|07|08)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
18 66.249.79.139
21 157.55.39.160
29 66.249.79.137
38 66.249.79.135
50 34.200.212.137
54 66.249.79.133
100 102.128.190.18
1166 45.5.184.72
4251 45.5.186.2
4895 205.186.128.185
# zcat --force /var/log/nginx/{access,library-access}.log /var/log/nginx/{access,library-access}.log.1 | grep -E "09/Apr/2019:(06|07|08)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
200 144.48.242.108
202 207.46.13.185
206 18.194.46.84
239 66.249.79.139
246 18.196.196.108
274 31.6.77.23
289 66.249.79.137
312 157.55.39.160
441 66.249.79.135
856 66.249.79.133
```
- `45.5.186.2` is at CIAT in Colombia and I see they are mostly making requests to the REST API, but also to XMLUI with the following user agent:
```
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36
```
- Database connection usage looks fine:
```
$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
5 dspaceApi
7 dspaceCli
11 dspaceWeb
```
- Ironically I do still see some 2 to 10% of CPU steal in `iostat 1 10`
<!-- vim: set sw=2 ts=2: -->

58
docs/2019-04/index.html
View File

@ -38,7 +38,7 @@ $ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace
<meta property="og:type" content="article" />
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2019-04/" />
<meta property="article:published_time" content="2019-04-01T09:00:43&#43;03:00"/>
<meta property="article:modified_time" content="2019-04-08T20:03:58&#43;03:00"/>
<meta property="article:modified_time" content="2019-04-08T20:22:40&#43;03:00"/>
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="April, 2019"/>
@ -81,9 +81,9 @@ $ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace
"@type": "BlogPosting",
"headline": "April, 2019",
"url": "https://alanorth.github.io/cgspace-notes/2019-04/",
"wordCount": "2869",
"wordCount": "3073",
"datePublished": "2019-04-01T09:00:43&#43;03:00",
"dateModified": "2019-04-08T20:03:58&#43;03:00",
"dateModified": "2019-04-08T20:22:40&#43;03:00",
"author": {
"@type": "Person",
"name": "Alan Orth"
@ -750,6 +750,58 @@ org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exe
370 51.254.16.223
</code></pre>
<h2 id="2019-04-09">2019-04-09</h2>
<ul>
<li>Linode sent an alert that CGSpace (linode18) was 440% CPU for the last two hours this morning</li>
<li>Here are the top IPs in the web server logs around that time:</li>
</ul>
<pre><code># zcat --force /var/log/nginx/{rest,oai}.log /var/log/nginx/{rest,oai}.log.1 | grep -E &quot;09/Apr/2019:(06|07|08)&quot; | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
18 66.249.79.139
21 157.55.39.160
29 66.249.79.137
38 66.249.79.135
50 34.200.212.137
54 66.249.79.133
100 102.128.190.18
1166 45.5.184.72
4251 45.5.186.2
4895 205.186.128.185
# zcat --force /var/log/nginx/{access,library-access}.log /var/log/nginx/{access,library-access}.log.1 | grep -E &quot;09/Apr/2019:(06|07|08)&quot; | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
200 144.48.242.108
202 207.46.13.185
206 18.194.46.84
239 66.249.79.139
246 18.196.196.108
274 31.6.77.23
289 66.249.79.137
312 157.55.39.160
441 66.249.79.135
856 66.249.79.133
</code></pre>
<ul>
<li><code>45.5.186.2</code> is at CIAT in Colombia and I see they are mostly making requests to the REST API, but also to XMLUI with the following user agent:</li>
</ul>
<pre><code>Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36
</code></pre>
<ul>
<li>Database connection usage looks fine:</li>
</ul>
<pre><code>$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
5 dspaceApi
7 dspaceCli
11 dspaceWeb
</code></pre>
<ul>
<li>Ironically I do still see some 2 to 10% of CPU steal in <code>iostat 1 10</code></li>
</ul>
<!-- vim: set sw=2 ts=2: -->

10
docs/sitemap.xml
View File

@ -4,7 +4,7 @@
<url>
<loc>https://alanorth.github.io/cgspace-notes/2019-04/</loc>
<lastmod>2019-04-08T20:03:58+03:00</lastmod>
<lastmod>2019-04-08T20:22:40+03:00</lastmod>
</url>
<url>
@ -219,7 +219,7 @@
<url>
<loc>https://alanorth.github.io/cgspace-notes/</loc>
<lastmod>2019-04-08T20:03:58+03:00</lastmod>
<lastmod>2019-04-08T20:22:40+03:00</lastmod>
<priority>0</priority>
</url>
@ -230,7 +230,7 @@
<url>
<loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
<lastmod>2019-04-08T20:03:58+03:00</lastmod>
<lastmod>2019-04-08T20:22:40+03:00</lastmod>
<priority>0</priority>
</url>
@ -242,13 +242,13 @@
<url>
<loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
<lastmod>2019-04-08T20:03:58+03:00</lastmod>
<lastmod>2019-04-08T20:22:40+03:00</lastmod>
<priority>0</priority>
</url>
<url>
<loc>https://alanorth.github.io/cgspace-notes/tags/</loc>
<lastmod>2019-04-08T20:03:58+03:00</lastmod>
<lastmod>2019-04-08T20:22:40+03:00</lastmod>
<priority>0</priority>
</url>