diff --git a/content/posts/2019-02.md b/content/posts/2019-02.md
index f55d6da2c..cb29cec6f 100644
--- a/content/posts/2019-02.md
+++ b/content/posts/2019-02.md
@@ -296,4 +296,64 @@ $ dspace metadata-import -f /tmp/2019-02-06-CTA-multiple-subjects.csv
5102 45.5.186.2
```
+## 2019-02-07
+
+- Linode sent an alert last night that the load on CGSpace (linode18) was over 300%
+- Here are the top IPs in the web server and API logs before, during, and after that time, respectively:
+
+```
+# zcat --force /var/log/nginx/{oai,rest,statistics}.log /var/log/nginx/{oai,rest,statistics}.log.1 | grep -E "06/Feb/2019:(17|18|19|20|23)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+ 5 66.249.66.209
+ 6 2a01:4f8:210:51ef::2
+ 6 40.77.167.75
+ 9 104.198.9.108
+ 9 157.55.39.192
+ 10 157.55.39.244
+ 12 66.249.66.221
+ 20 95.108.181.88
+ 27 66.249.66.219
+ 2381 45.5.186.2
+# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -E "06/Feb/2019:(17|18|19|20|23)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+ 455 45.5.186.2
+ 506 40.77.167.75
+ 559 54.70.40.11
+ 825 157.55.39.244
+ 871 2a01:4f8:140:3192::2
+ 938 157.55.39.192
+ 1058 85.25.237.71
+ 1416 5.9.6.51
+ 1606 66.249.66.219
+ 1718 35.237.175.180
+```
+
+- Then again this morning another alert:
+
+```
+# zcat --force /var/log/nginx/{oai,rest,statistics}.log /var/log/nginx/{oai,rest,statistics}.log.1 | grep -E "07/Feb/2019:0(5|6|7|8|9)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+ 5 66.249.66.223
+ 8 104.198.9.108
+ 13 110.54.160.222
+ 24 66.249.66.219
+ 25 175.158.217.98
+ 214 34.218.226.147
+ 346 45.5.184.72
+ 4529 45.5.186.2
+ 4661 205.186.128.185
+ 4661 70.32.83.92
+# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -E "07/Feb/2019:0(5|6|7|8|9)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+ 145 157.55.39.237
+ 154 66.249.66.221
+ 214 34.218.226.147
+ 261 35.237.175.180
+ 273 2a01:4f8:140:3192::2
+ 300 169.48.66.92
+ 487 5.143.231.39
+ 766 5.9.6.51
+ 771 85.25.237.71
+ 848 66.249.66.219
+```
+
+- So it seems that the load issue comes from the REST API, not the XMLUI
+- I could probably rate limit the REST API, or maybe just keep increasing the alert threshold so I don't get alert spam (this is probably the correct approach because it seems like the REST API can keep up with the requests and is returning HTTP 200 status as far as I can tell)
+
diff --git a/docs/2019-02/index.html b/docs/2019-02/index.html
index afb1b8165..61829dfc0 100644
--- a/docs/2019-02/index.html
+++ b/docs/2019-02/index.html
@@ -42,7 +42,7 @@ sys 0m1.979s
-
+
@@ -89,9 +89,9 @@ sys 0m1.979s
"@type": "BlogPosting",
"headline": "February, 2019",
"url": "https://alanorth.github.io/cgspace-notes/2019-02/",
- "wordCount": "1435",
+ "wordCount": "1724",
"datePublished": "2019-02-01T21:37:30+02:00",
- "dateModified": "2019-02-05T09:22:06+02:00",
+ "dateModified": "2019-02-06T16:50:39+02:00",
"author": {
"@type": "Person",
"name": "Alan Orth"
@@ -485,6 +485,70 @@ PESCAS E AQUACULTURE,PISCICULTURA||AQUACULTURE
5102 45.5.186.2
+2019-02-07
+
+
+- Linode sent an alert last night that the load on CGSpace (linode18) was over 300%
+- Here are the top IPs in the web server and API logs before, during, and after that time, respectively:
+
+
+# zcat --force /var/log/nginx/{oai,rest,statistics}.log /var/log/nginx/{oai,rest,statistics}.log.1 | grep -E "06/Feb/2019:(17|18|19|20|23)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+ 5 66.249.66.209
+ 6 2a01:4f8:210:51ef::2
+ 6 40.77.167.75
+ 9 104.198.9.108
+ 9 157.55.39.192
+ 10 157.55.39.244
+ 12 66.249.66.221
+ 20 95.108.181.88
+ 27 66.249.66.219
+ 2381 45.5.186.2
+# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -E "06/Feb/2019:(17|18|19|20|23)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+ 455 45.5.186.2
+ 506 40.77.167.75
+ 559 54.70.40.11
+ 825 157.55.39.244
+ 871 2a01:4f8:140:3192::2
+ 938 157.55.39.192
+ 1058 85.25.237.71
+ 1416 5.9.6.51
+ 1606 66.249.66.219
+ 1718 35.237.175.180
+
+
+
+- Then again this morning another alert:
+
+
+# zcat --force /var/log/nginx/{oai,rest,statistics}.log /var/log/nginx/{oai,rest,statistics}.log.1 | grep -E "07/Feb/2019:0(5|6|7|8|9)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+ 5 66.249.66.223
+ 8 104.198.9.108
+ 13 110.54.160.222
+ 24 66.249.66.219
+ 25 175.158.217.98
+ 214 34.218.226.147
+ 346 45.5.184.72
+ 4529 45.5.186.2
+ 4661 205.186.128.185
+ 4661 70.32.83.92
+# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -E "07/Feb/2019:0(5|6|7|8|9)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+ 145 157.55.39.237
+ 154 66.249.66.221
+ 214 34.218.226.147
+ 261 35.237.175.180
+ 273 2a01:4f8:140:3192::2
+ 300 169.48.66.92
+ 487 5.143.231.39
+ 766 5.9.6.51
+ 771 85.25.237.71
+ 848 66.249.66.219
+
+
+
+- So it seems that the load issue comes from the REST API, not the XMLUI
+- I could probably rate limit the REST API, or maybe just keep increasing the alert threshold so I don’t get alert spam (this is probably the correct approach because it seems like the REST API can keep up with the requests and is returning HTTP 200 status as far as I can tell)
+
+
diff --git a/docs/sitemap.xml b/docs/sitemap.xml
index c18b78bbd..4e8bcafe4 100644
--- a/docs/sitemap.xml
+++ b/docs/sitemap.xml
@@ -4,7 +4,7 @@
https://alanorth.github.io/cgspace-notes/2019-02/
- 2019-02-05T09:22:06+02:00
+ 2019-02-06T16:50:39+02:00
@@ -209,7 +209,7 @@
https://alanorth.github.io/cgspace-notes/
- 2019-02-05T09:22:06+02:00
+ 2019-02-06T16:50:39+02:00
0
@@ -220,7 +220,7 @@
https://alanorth.github.io/cgspace-notes/tags/notes/
- 2019-02-05T09:22:06+02:00
+ 2019-02-06T16:50:39+02:00
0
@@ -232,13 +232,13 @@
https://alanorth.github.io/cgspace-notes/posts/
- 2019-02-05T09:22:06+02:00
+ 2019-02-06T16:50:39+02:00
0
https://alanorth.github.io/cgspace-notes/tags/
- 2019-02-05T09:22:06+02:00
+ 2019-02-06T16:50:39+02:00
0