diff --git a/content/posts/2018-03.md b/content/posts/2018-03.md
index 984e97dc5..632d18925 100644
--- a/content/posts/2018-03.md
+++ b/content/posts/2018-03.md
@@ -237,3 +237,52 @@ $ ./fix-metadata-values.py -i Correct-21-CRPs-2018-03-16.csv -db dspace -u dspac
- Tezira has been having problems accessing CGSpace from the ILRI Nairobi campus since last week
- She is getting an HTTPS error apparently
- It's working outside, and Ethiopian users seem to be having no issues so I've asked ICT to have a look
+- CGSpace crashed this morning for about seven minutes and Dani restarted Tomcat
+- Around that time there were an increase of SQL errors:
+
+```
+2018-03-19 09:10:54,856 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL QueryTable Error -
+...
+2018-03-19 09:10:54,862 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL query singleTable Error -
+```
+
+- But these errors, I don't even know what they mean, because a handful of them happen every day:
+
+```
+$ grep -c 'ERROR org.dspace.storage.rdbms.DatabaseManager' dspace.log.2018-03-1*
+dspace.log.2018-03-10:13
+dspace.log.2018-03-11:15
+dspace.log.2018-03-12:13
+dspace.log.2018-03-13:13
+dspace.log.2018-03-14:14
+dspace.log.2018-03-15:13
+dspace.log.2018-03-16:13
+dspace.log.2018-03-17:13
+dspace.log.2018-03-18:15
+dspace.log.2018-03-19:90
+```
+
+- There wasn't even a lot of traffic at the time (8–9 AM):
+
+```
+# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "19/Mar/2018:0[89]:" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+ 92 40.77.167.197
+ 92 83.103.94.48
+ 96 40.77.167.175
+ 116 207.46.13.178
+ 122 66.249.66.153
+ 140 95.108.181.88
+ 196 213.55.99.121
+ 206 197.210.168.174
+ 207 104.196.152.243
+ 294 54.198.169.202
+```
+
+- ICT responded that they "fixed" the CGSpace connectivity issue in Nairobi without telling me the problem
+- When I asked, Robert Okal said CGNET messed up when updating the DNS for cgspace.cgiar.org last week
+- I told him that my request last week was for dspacetest.cgiar.org, not cgspace.cgiar.org!
+- So they updated the wrong fucking DNS records
+- Magdalena from CCAFS wrote to ask about one record that has a bunch of metadata missing in her Listings and Reports export
+- It appears to be this one: https://cgspace.cgiar.org/handle/10568/83473?show=full
+- The title is "Untitled" and there is some metadata but indeed the citation is missing
+- I don't know what would cause that
diff --git a/docs/2018-03/index.html b/docs/2018-03/index.html
index 3aefd7af0..b9a0e8748 100644
--- a/docs/2018-03/index.html
+++ b/docs/2018-03/index.html
@@ -20,7 +20,7 @@ Export a CSV of the IITA community metadata for Martin Mueller
-
+
@@ -51,9 +51,9 @@ Export a CSV of the IITA community metadata for Martin Mueller
"@type": "BlogPosting",
"headline": "March, 2018",
"url": "https://alanorth.github.io/cgspace-notes/2018-03/",
- "wordCount": "1407",
+ "wordCount": "1645",
"datePublished": "2018-03-02T16:07:54+02:00",
- "dateModified": "2018-03-16T18:58:36+02:00",
+ "dateModified": "2018-03-19T08:30:32+02:00",
"author": {
"@type": "Person",
"name": "Alan Orth"
@@ -389,6 +389,58 @@ COPY 21
Tezira has been having problems accessing CGSpace from the ILRI Nairobi campus since last week
She is getting an HTTPS error apparently
It’s working outside, and Ethiopian users seem to be having no issues so I’ve asked ICT to have a look
+CGSpace crashed this morning for about seven minutes and Dani restarted Tomcat
+Around that time there were an increase of SQL errors:
+
+
+2018-03-19 09:10:54,856 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL QueryTable Error -
+...
+2018-03-19 09:10:54,862 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL query singleTable Error -
+
+
+
+- But these errors, I don’t even know what they mean, because a handful of them happen every day:
+
+
+$ grep -c 'ERROR org.dspace.storage.rdbms.DatabaseManager' dspace.log.2018-03-1*
+dspace.log.2018-03-10:13
+dspace.log.2018-03-11:15
+dspace.log.2018-03-12:13
+dspace.log.2018-03-13:13
+dspace.log.2018-03-14:14
+dspace.log.2018-03-15:13
+dspace.log.2018-03-16:13
+dspace.log.2018-03-17:13
+dspace.log.2018-03-18:15
+dspace.log.2018-03-19:90
+
+
+
+- There wasn’t even a lot of traffic at the time (8–9 AM):
+
+
+# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "19/Mar/2018:0[89]:" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+ 92 40.77.167.197
+ 92 83.103.94.48
+ 96 40.77.167.175
+ 116 207.46.13.178
+ 122 66.249.66.153
+ 140 95.108.181.88
+ 196 213.55.99.121
+ 206 197.210.168.174
+ 207 104.196.152.243
+ 294 54.198.169.202
+
+
+
+- ICT responded that they “fixed” the CGSpace connectivity issue in Nairobi without telling me the problem
+- When I asked, Robert Okal said CGNET messed up when updating the DNS for cgspace.cgiar.org last week
+- I told him that my request last week was for dspacetest.cgiar.org, not cgspace.cgiar.org!
+- So they updated the wrong fucking DNS records
+- Magdalena from CCAFS wrote to ask about one record that has a bunch of metadata missing in her Listings and Reports export
+- It appears to be this one: https://cgspace.cgiar.org/handle/10568/83473?show=full
+- The title is “Untitled” and there is some metadata but indeed the citation is missing
+- I don’t know what would cause that
diff --git a/docs/sitemap.xml b/docs/sitemap.xml
index 989ef4aaa..2a4a87eaa 100644
--- a/docs/sitemap.xml
+++ b/docs/sitemap.xml
@@ -4,7 +4,7 @@
https://alanorth.github.io/cgspace-notes/2018-03/
- 2018-03-16T18:58:36+02:00
+ 2018-03-19T08:30:32+02:00
@@ -154,7 +154,7 @@
https://alanorth.github.io/cgspace-notes/
- 2018-03-16T18:58:36+02:00
+ 2018-03-19T08:30:32+02:00
0
@@ -165,7 +165,7 @@
https://alanorth.github.io/cgspace-notes/tags/notes/
- 2018-03-16T18:58:36+02:00
+ 2018-03-19T08:30:32+02:00
0
@@ -177,13 +177,13 @@
https://alanorth.github.io/cgspace-notes/posts/
- 2018-03-16T18:58:36+02:00
+ 2018-03-19T08:30:32+02:00
0
https://alanorth.github.io/cgspace-notes/tags/
- 2018-03-16T18:58:36+02:00
+ 2018-03-19T08:30:32+02:00
0