Update notes for 2018-03-19

content/posts/2018-03.md

@@ -237,3 +237,52 @@ $ ./fix-metadata-values.py -i Correct-21-CRPs-2018-03-16.csv -db dspace -u dspac
 - Tezira has been having problems accessing CGSpace from the ILRI Nairobi campus since last week
 - She is getting an HTTPS error apparently
 - It's working outside, and Ethiopian users seem to be having no issues so I've asked ICT to have a look
+- CGSpace crashed this morning for about seven minutes and Dani restarted Tomcat
+- Around that time there were an increase of SQL errors:
+
+```
+2018-03-19 09:10:54,856 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL QueryTable Error -
+...
+2018-03-19 09:10:54,862 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL query singleTable Error -
+```
+
+- But these errors, I don't even know what they mean, because a handful of them happen every day:
+
+```
+$ grep -c 'ERROR org.dspace.storage.rdbms.DatabaseManager' dspace.log.2018-03-1*
+dspace.log.2018-03-10:13
+dspace.log.2018-03-11:15
+dspace.log.2018-03-12:13
+dspace.log.2018-03-13:13
+dspace.log.2018-03-14:14
+dspace.log.2018-03-15:13
+dspace.log.2018-03-16:13
+dspace.log.2018-03-17:13
+dspace.log.2018-03-18:15
+dspace.log.2018-03-19:90
+```
+
+- There wasn't even a lot of traffic at the time (8–9 AM):
+
+```
+# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "19/Mar/2018:0[89]:" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+     92 40.77.167.197
+     92 83.103.94.48
+     96 40.77.167.175
+    116 207.46.13.178
+    122 66.249.66.153
+    140 95.108.181.88
+    196 213.55.99.121
+    206 197.210.168.174
+    207 104.196.152.243
+    294 54.198.169.202
+```
+
+- ICT responded that they "fixed" the CGSpace connectivity issue in Nairobi without telling me the problem
+- When I asked, Robert Okal said CGNET messed up when updating the DNS for cgspace.cgiar.org last week
+- I told him that my request last week was for dspacetest.cgiar.org, not cgspace.cgiar.org!
+- So they updated the wrong fucking DNS records
+- Magdalena from CCAFS wrote to ask about one record that has a bunch of metadata missing in her Listings and Reports export
+- It appears to be this one: https://cgspace.cgiar.org/handle/10568/83473?show=full
+- The title is "Untitled" and there is some metadata but indeed the citation is missing
+- I don't know what would cause that

docs/2018-03/index.html

@@ -20,7 +20,7 @@ Export a CSV of the IITA community metadata for Martin Mueller
 
 <meta property="article:published_time" content="2018-03-02T16:07:54&#43;02:00"/>
 
-<meta property="article:modified_time" content="2018-03-16T18:58:36&#43;02:00"/>
+<meta property="article:modified_time" content="2018-03-19T08:30:32&#43;02:00"/>
 
 
 
@@ -51,9 +51,9 @@ Export a CSV of the IITA community metadata for Martin Mueller
   "@type": "BlogPosting",
   "headline": "March, 2018",
   "url": "https://alanorth.github.io/cgspace-notes/2018-03/",
-  "wordCount": "1407",
+  "wordCount": "1645",
   "datePublished": "2018-03-02T16:07:54&#43;02:00",
-  "dateModified": "2018-03-16T18:58:36&#43;02:00",
+  "dateModified": "2018-03-19T08:30:32&#43;02:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"
@@ -389,6 +389,58 @@ COPY 21
 <li>Tezira has been having problems accessing CGSpace from the ILRI Nairobi campus since last week</li>
 <li>She is getting an HTTPS error apparently</li>
 <li>It&rsquo;s working outside, and Ethiopian users seem to be having no issues so I&rsquo;ve asked ICT to have a look</li>
+<li>CGSpace crashed this morning for about seven minutes and Dani restarted Tomcat</li>
+<li>Around that time there were an increase of SQL errors:</li>
+</ul>
+
+<pre><code>2018-03-19 09:10:54,856 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL QueryTable Error -
+...
+2018-03-19 09:10:54,862 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL query singleTable Error -
+</code></pre>
+
+<ul>
+<li>But these errors, I don&rsquo;t even know what they mean, because a handful of them happen every day:</li>
+</ul>
+
+<pre><code>$ grep -c 'ERROR org.dspace.storage.rdbms.DatabaseManager' dspace.log.2018-03-1*
+dspace.log.2018-03-10:13
+dspace.log.2018-03-11:15
+dspace.log.2018-03-12:13
+dspace.log.2018-03-13:13
+dspace.log.2018-03-14:14
+dspace.log.2018-03-15:13
+dspace.log.2018-03-16:13
+dspace.log.2018-03-17:13
+dspace.log.2018-03-18:15
+dspace.log.2018-03-19:90
+</code></pre>
+
+<ul>
+<li>There wasn&rsquo;t even a lot of traffic at the time (8–9 AM):</li>
+</ul>
+
+<pre><code># zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E &quot;19/Mar/2018:0[89]:&quot; | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+     92 40.77.167.197
+     92 83.103.94.48
+     96 40.77.167.175
+    116 207.46.13.178
+    122 66.249.66.153
+    140 95.108.181.88
+    196 213.55.99.121
+    206 197.210.168.174
+    207 104.196.152.243
+    294 54.198.169.202
+</code></pre>
+
+<ul>
+<li>ICT responded that they &ldquo;fixed&rdquo; the CGSpace connectivity issue in Nairobi without telling me the problem</li>
+<li>When I asked, Robert Okal said CGNET messed up when updating the DNS for cgspace.cgiar.org last week</li>
+<li>I told him that my request last week was for dspacetest.cgiar.org, not cgspace.cgiar.org!</li>
+<li>So they updated the wrong fucking DNS records</li>
+<li>Magdalena from CCAFS wrote to ask about one record that has a bunch of metadata missing in her Listings and Reports export</li>
+<li>It appears to be this one: <a href="https://cgspace.cgiar.org/handle/10568/83473?show=full">https://cgspace.cgiar.org/handle/10568/83473?show=full</a></li>
+<li>The title is &ldquo;Untitled&rdquo; and there is some metadata but indeed the citation is missing</li>
+<li>I don&rsquo;t know what would cause that</li>
 </ul>
 
   

docs/sitemap.xml

@@ -4,7 +4,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2018-03/</loc>
-    <lastmod>2018-03-16T18:58:36+02:00</lastmod>
+    <lastmod>2018-03-19T08:30:32+02:00</lastmod>
   </url>
   
   <url>
@@ -154,7 +154,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2018-03-16T18:58:36+02:00</lastmod>
+    <lastmod>2018-03-19T08:30:32+02:00</lastmod>
     <priority>0</priority>
   </url>
   
@@ -165,7 +165,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
-    <lastmod>2018-03-16T18:58:36+02:00</lastmod>
+    <lastmod>2018-03-19T08:30:32+02:00</lastmod>
     <priority>0</priority>
   </url>
   
@@ -177,13 +177,13 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
-    <lastmod>2018-03-16T18:58:36+02:00</lastmod>
+    <lastmod>2018-03-19T08:30:32+02:00</lastmod>
     <priority>0</priority>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/tags/</loc>
-    <lastmod>2018-03-16T18:58:36+02:00</lastmod>
+    <lastmod>2018-03-19T08:30:32+02:00</lastmod>
     <priority>0</priority>
   </url>