Add notes

content/posts/2021-04.md

@@ -331,5 +331,154 @@ $ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid =
 
 - I deleted the ilri/AReS repository on GitHub since we haven't updated it in two years
   - All development is happening in https://github.com/ilri/openRXV now
+- 10PM and the server is down again, with locks through the roof:
+
+```console
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+12198
+```
+
+- I see that there are tons of PostgreSQL connections getting abandoned today, compared to very few in the past few weeks:
+
+```console
+$ journalctl -u tomcat7 --since=today | grep -c 'ConnectionPool abandon'
+1838
+$ journalctl -u tomcat7 --since=2021-03-20 --until=2021-04-05 | grep -c 'ConnectionPool abandon'
+3
+```
+
+- I even restarted the server and connections were low for a few minutes until they shot back up:
+
+```console
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+13
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+8651
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+8940
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+10504
+```
+
+- I had to go to bed and I bet it will crash and be down for hours until I wake up...
+- What the hell is this user agent?
+
+```
+54.197.119.143 - - [06/Apr/2021:19:18:11 +0200] "GET /handle/10568/16499 HTTP/1.1" 499 0 "-" "GetUrl/1.0 wdestiny@umich.edu (Linux)"
+```
+
+## 2021-04-07
+
+- CGSpace was still down from last night of course, with tons of database locks:
+
+```console
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+12168
+```
+
+- I restarted the server again and the locks came back
+- Atmire responded to the message from yesterday
+  - The noticed something in the logs about emails failing to be sent
+  - There appears to be an issue sending mails on workflow tasks when a user in that group has an invalid email address:
+
+```console
+2021-04-01 12:45:11,414 WARN  org.dspace.workflowbasic.BasicWorkflowServiceImpl @ a.akwarandu@cgiar.org:session_id=2F20F20D4A8C36DB53D42DE45DFA3CCE:notifyGroupofTask:cannot email user group_id=aecf811b-b7e9-4b6f-8776-3d372e6a048b workflow_item_id=33085\colon;  Invalid Addresses (com.sun.mail.smtp.SMTPAddressFailedException\colon; 501 5.1.3 Invalid address
+```
+
+- The issue is not the named user above, but a member of the group...
+- And the group does have users with invalid email addresses (probably accounts created automatically after authenticating with LDAP):
+
+![DSpace group](/cgspace-notes/2021/04/group-invalid-email.png)
+
+- I extracted all the group IDs from recent logs that had users with invalid email addresses:
+
+```console
+$ grep -a -E 'email user group_id=\b[0-9a-f]{8}\b-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-\b[0-9a-f]{12}\b' /home/cgspace.cgiar.org/log/dspace.log.* | grep -o -E '\b[0-9a-f]{8}\b-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-\b[0-9a-f]{12}\b' | sort | uniq
+0a30d6ae-74a6-4eee-a8f5-ee5d15192ee6
+1769137c-36d4-42b2-8fec-60585e110db7
+203c8614-8a97-4ac8-9686-d9d62cb52acc
+294603de-3d09-464e-a5b0-09e452c6b5ab
+35878555-9623-4679-beb8-bb3395fdf26e
+3d8a5efa-5509-4bf9-9374-2bc714aceb99
+4238208a-f848-47cb-9dd2-43f9f954a4af
+44939b84-1894-41e7-b3e6-8c8d1781057b
+49ba087e-75a3-45ce-805c-69eeda0f786b
+4a6606ce-0284-421d-bf80-4dafddba2d42
+527de6aa-9cd0-4988-bf5f-c9c92ba2ac10
+54cd1b16-65bf-4041-9d84-fb2ea3301d6d
+58982847-5f7c-4b8b-a7b0-4d4de702136e
+5f0b85be-bd23-47de-927d-bca368fa1fbc
+646ada17-e4ef-49f6-9378-af7e58596ce1
+7e2f4bf8-fbc9-4b2f-97a4-75e5427bef90
+8029fd53-f9f5-4107-bfc3-8815507265cf
+81faa934-c602-4608-bf45-de91845dfea7
+8611a462-210c-4be1-a5bb-f87a065e6113
+8855c903-ef86-433c-b0be-c12300eb0f84
+8c7ece98-3598-4de7-a885-d61fd033bea8
+8c9a0d01-2d12-4a99-84f9-cdc25ac072f9
+8f9f888a-b501-41f3-a462-4da16150eebf
+94168f0e-9f45-4112-ac8d-3ba9be917842
+96998038-f381-47dc-8488-ff7252703627
+9768f4a8-3018-44e9-bf58-beba4296327c
+9a99e8d2-558e-4fc1-8011-e4411f658414
+a34e6400-78ed-45c0-a751-abc039eed2e6
+a9da5af3-4ec7-4a9b-becb-6e3d028d594d
+abf5201c-8be5-4dee-b461-132203dd51cb
+adb5658c-cef3-402f-87b6-b498f580351c
+aecf811b-b7e9-4b6f-8776-3d372e6a048b
+ba5aae61-ea34-4ac1-9490-4645acf2382f
+bf7f3638-c7c6-4a8f-893d-891a6d3dafff
+c617ada0-09d1-40ed-b479-1c4860a4f724
+cff91d44-a855-458c-89e5-bd48c17d1a54
+e65171ae-a2bf-4043-8f54-f8457bc9174b
+e7098b40-4701-4ca2-b9a9-3a1282f67044
+e904f122-71dc-439b-b877-313ef62486d7
+ede59734-adac-4c01-8691-b45f19088d37
+f88bd6bb-f93f-41cb-872f-ff26f6237068
+f985f5fb-be5c-430b-a8f1-cf86ae4fc49a
+fe800006-aaec-4f9e-9ab4-f9475b4cbdc3
+```
+
+## 2021-04-08
+
+- I can't believe it but the server has been down for twelve hours or so
+  - The locks have not changed since I went to bed last night:
+
+```console
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+12070
+```
+
+- I restarted PostgreSQL and Tomcat and the locks go straight back up!
+
+```console
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+13
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+986
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+1194
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+1212
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+1489
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+2124
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+5934
+```
+
+## 2021-04-09
+
+- Atmire managed to get CGSpace back up by killing all the PostgreSQL connections yesterday
+  - I don't know how they did it...
+  - They also think it's weird that restarting PostgreSQL didn't kill the connections
+  - They asked some more questions, like for example if there were also issues on DSpace Test
+  - Strangely enough, I checked DSpace Test and notice a clear spike in PostgreSQL locks on the morning of April 6th as well!
+
+![PostgreSQL locks week CGSpace](/cgspace-notes/2021/04/postgres_locks_ALL-week-PROD.png)
+![PostgreSQL locks week DSpace Test](/cgspace-notes/2021/04/postgres_locks_ALL-week-TEST.png)
+
+- I definitely need to look into that!
 
 <!-- vim: set sw=2 ts=2: -->

docs/2021-04/index.html

@@ -24,7 +24,7 @@ Perhaps one of the containers crashed, I should have looked closer but I was in
 <meta property="og:type" content="article" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2021-04/" />
 <meta property="article:published_time" content="2021-04-01T09:50:54+03:00" />
-<meta property="article:modified_time" content="2021-04-06T22:33:43+03:00" />
+<meta property="article:modified_time" content="2021-04-06T22:48:44+03:00" />
 
 
 
@@ -54,9 +54,9 @@ Perhaps one of the containers crashed, I should have looked closer but I was in
   "@type": "BlogPosting",
   "headline": "April, 2021",
   "url": "https://alanorth.github.io/cgspace-notes/2021-04/",
-  "wordCount": "1824",
+  "wordCount": "2530",
   "datePublished": "2021-04-01T09:50:54+03:00",
-  "dateModified": "2021-04-06T22:33:43+03:00",
+  "dateModified": "2021-04-06T22:48:44+03:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"
@@ -462,6 +462,143 @@ $ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid =
 <li>All development is happening in <a href="https://github.com/ilri/openRXV">https://github.com/ilri/openRXV</a> now</li>
 </ul>
 </li>
+<li>10PM and the server is down again, with locks through the roof:</li>
+</ul>
+<pre><code class="language-console" data-lang="console">$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+12198
+</code></pre><ul>
+<li>I see that there are tons of PostgreSQL connections getting abandoned today, compared to very few in the past few weeks:</li>
+</ul>
+<pre><code class="language-console" data-lang="console">$ journalctl -u tomcat7 --since=today | grep -c 'ConnectionPool abandon'
+1838
+$ journalctl -u tomcat7 --since=2021-03-20 --until=2021-04-05 | grep -c 'ConnectionPool abandon'
+3
+</code></pre><ul>
+<li>I even restarted the server and connections were low for a few minutes until they shot back up:</li>
+</ul>
+<pre><code class="language-console" data-lang="console">$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+13
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+8651
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+8940
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+10504
+</code></pre><ul>
+<li>I had to go to bed and I bet it will crash and be down for hours until I wake up&hellip;</li>
+<li>What the hell is this user agent?</li>
+</ul>
+<pre><code>54.197.119.143 - - [06/Apr/2021:19:18:11 +0200] &quot;GET /handle/10568/16499 HTTP/1.1&quot; 499 0 &quot;-&quot; &quot;GetUrl/1.0 wdestiny@umich.edu (Linux)&quot;
+</code></pre><h2 id="2021-04-07">2021-04-07</h2>
+<ul>
+<li>CGSpace was still down from last night of course, with tons of database locks:</li>
+</ul>
+<pre><code class="language-console" data-lang="console">$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+12168
+</code></pre><ul>
+<li>I restarted the server again and the locks came back</li>
+<li>Atmire responded to the message from yesterday
+<ul>
+<li>The noticed something in the logs about emails failing to be sent</li>
+<li>There appears to be an issue sending mails on workflow tasks when a user in that group has an invalid email address:</li>
+</ul>
+</li>
+</ul>
+<pre><code class="language-console" data-lang="console">2021-04-01 12:45:11,414 WARN  org.dspace.workflowbasic.BasicWorkflowServiceImpl @ a.akwarandu@cgiar.org:session_id=2F20F20D4A8C36DB53D42DE45DFA3CCE:notifyGroupofTask:cannot email user group_id=aecf811b-b7e9-4b6f-8776-3d372e6a048b workflow_item_id=33085\colon;  Invalid Addresses (com.sun.mail.smtp.SMTPAddressFailedException\colon; 501 5.1.3 Invalid address
+</code></pre><ul>
+<li>The issue is not the named user above, but a member of the group&hellip;</li>
+<li>And the group does have users with invalid email addresses (probably accounts created automatically after authenticating with LDAP):</li>
+</ul>
+<p><img src="/cgspace-notes/2021/04/group-invalid-email.png" alt="DSpace group"></p>
+<ul>
+<li>I extracted all the group IDs from recent logs that had users with invalid email addresses:</li>
+</ul>
+<pre><code class="language-console" data-lang="console">$ grep -a -E 'email user group_id=\b[0-9a-f]{8}\b-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-\b[0-9a-f]{12}\b' /home/cgspace.cgiar.org/log/dspace.log.* | grep -o -E '\b[0-9a-f]{8}\b-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-\b[0-9a-f]{12}\b' | sort | uniq
+0a30d6ae-74a6-4eee-a8f5-ee5d15192ee6
+1769137c-36d4-42b2-8fec-60585e110db7
+203c8614-8a97-4ac8-9686-d9d62cb52acc
+294603de-3d09-464e-a5b0-09e452c6b5ab
+35878555-9623-4679-beb8-bb3395fdf26e
+3d8a5efa-5509-4bf9-9374-2bc714aceb99
+4238208a-f848-47cb-9dd2-43f9f954a4af
+44939b84-1894-41e7-b3e6-8c8d1781057b
+49ba087e-75a3-45ce-805c-69eeda0f786b
+4a6606ce-0284-421d-bf80-4dafddba2d42
+527de6aa-9cd0-4988-bf5f-c9c92ba2ac10
+54cd1b16-65bf-4041-9d84-fb2ea3301d6d
+58982847-5f7c-4b8b-a7b0-4d4de702136e
+5f0b85be-bd23-47de-927d-bca368fa1fbc
+646ada17-e4ef-49f6-9378-af7e58596ce1
+7e2f4bf8-fbc9-4b2f-97a4-75e5427bef90
+8029fd53-f9f5-4107-bfc3-8815507265cf
+81faa934-c602-4608-bf45-de91845dfea7
+8611a462-210c-4be1-a5bb-f87a065e6113
+8855c903-ef86-433c-b0be-c12300eb0f84
+8c7ece98-3598-4de7-a885-d61fd033bea8
+8c9a0d01-2d12-4a99-84f9-cdc25ac072f9
+8f9f888a-b501-41f3-a462-4da16150eebf
+94168f0e-9f45-4112-ac8d-3ba9be917842
+96998038-f381-47dc-8488-ff7252703627
+9768f4a8-3018-44e9-bf58-beba4296327c
+9a99e8d2-558e-4fc1-8011-e4411f658414
+a34e6400-78ed-45c0-a751-abc039eed2e6
+a9da5af3-4ec7-4a9b-becb-6e3d028d594d
+abf5201c-8be5-4dee-b461-132203dd51cb
+adb5658c-cef3-402f-87b6-b498f580351c
+aecf811b-b7e9-4b6f-8776-3d372e6a048b
+ba5aae61-ea34-4ac1-9490-4645acf2382f
+bf7f3638-c7c6-4a8f-893d-891a6d3dafff
+c617ada0-09d1-40ed-b479-1c4860a4f724
+cff91d44-a855-458c-89e5-bd48c17d1a54
+e65171ae-a2bf-4043-8f54-f8457bc9174b
+e7098b40-4701-4ca2-b9a9-3a1282f67044
+e904f122-71dc-439b-b877-313ef62486d7
+ede59734-adac-4c01-8691-b45f19088d37
+f88bd6bb-f93f-41cb-872f-ff26f6237068
+f985f5fb-be5c-430b-a8f1-cf86ae4fc49a
+fe800006-aaec-4f9e-9ab4-f9475b4cbdc3
+</code></pre><h2 id="2021-04-08">2021-04-08</h2>
+<ul>
+<li>I can&rsquo;t believe it but the server has been down for twelve hours or so
+<ul>
+<li>The locks have not changed since I went to bed last night:</li>
+</ul>
+</li>
+</ul>
+<pre><code class="language-console" data-lang="console">$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+12070
+</code></pre><ul>
+<li>I restarted PostgreSQL and Tomcat and the locks go straight back up!</li>
+</ul>
+<pre><code class="language-console" data-lang="console">$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+13
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+986
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+1194
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+1212
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+1489
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+2124
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+5934
+</code></pre><h2 id="2021-04-09">2021-04-09</h2>
+<ul>
+<li>Atmire managed to get CGSpace back up by killing all the PostgreSQL connections yesterday
+<ul>
+<li>I don&rsquo;t know how they did it&hellip;</li>
+<li>They also think it&rsquo;s weird that restarting PostgreSQL didn&rsquo;t kill the connections</li>
+<li>They asked some more questions, like for example if there were also issues on DSpace Test</li>
+<li>Strangely enough, I checked DSpace Test and notice a clear spike in PostgreSQL locks on the morning of April 6th as well!</li>
+</ul>
+</li>
+</ul>
+<p><img src="/cgspace-notes/2021/04/postgres_locks_ALL-week-PROD.png" alt="PostgreSQL locks week CGSpace">
+<img src="/cgspace-notes/2021/04/postgres_locks_ALL-week-TEST.png" alt="PostgreSQL locks week DSpace Test"></p>
+<ul>
+<li>I definitely need to look into that!</li>
 </ul>
 <!-- raw HTML omitted -->
 

docs/categories/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/" />
-<meta property="og:updated_time" content="2021-04-06T22:33:43+03:00" />
+<meta property="og:updated_time" content="2021-04-06T22:48:44+03:00" />
 
 
 

docs/categories/notes/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-04-06T22:33:43+03:00" />
+<meta property="og:updated_time" content="2021-04-06T22:48:44+03:00" />
 
 
 

docs/categories/notes/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-04-06T22:33:43+03:00" />
+<meta property="og:updated_time" content="2021-04-06T22:48:44+03:00" />
 
 
 

docs/categories/notes/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-04-06T22:33:43+03:00" />
+<meta property="og:updated_time" content="2021-04-06T22:48:44+03:00" />
 
 
 

docs/categories/notes/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-04-06T22:33:43+03:00" />
+<meta property="og:updated_time" content="2021-04-06T22:48:44+03:00" />
 
 
 

docs/categories/notes/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-04-06T22:33:43+03:00" />
+<meta property="og:updated_time" content="2021-04-06T22:48:44+03:00" />
 
 
 

docs/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-04-06T22:33:43+03:00" />
+<meta property="og:updated_time" content="2021-04-06T22:48:44+03:00" />
 
 
 

docs/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-04-06T22:33:43+03:00" />
+<meta property="og:updated_time" content="2021-04-06T22:48:44+03:00" />
 
 
 

docs/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-04-06T22:33:43+03:00" />
+<meta property="og:updated_time" content="2021-04-06T22:48:44+03:00" />
 
 
 

docs/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-04-06T22:33:43+03:00" />
+<meta property="og:updated_time" content="2021-04-06T22:48:44+03:00" />
 
 
 

docs/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-04-06T22:33:43+03:00" />
+<meta property="og:updated_time" content="2021-04-06T22:48:44+03:00" />
 
 
 

docs/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-04-06T22:33:43+03:00" />
+<meta property="og:updated_time" content="2021-04-06T22:48:44+03:00" />
 
 
 

docs/page/7/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-04-06T22:33:43+03:00" />
+<meta property="og:updated_time" content="2021-04-06T22:48:44+03:00" />
 
 
 

docs/posts/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-04-06T22:33:43+03:00" />
+<meta property="og:updated_time" content="2021-04-06T22:48:44+03:00" />
 
 
 

docs/posts/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-04-06T22:33:43+03:00" />
+<meta property="og:updated_time" content="2021-04-06T22:48:44+03:00" />
 
 
 

docs/posts/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-04-06T22:33:43+03:00" />
+<meta property="og:updated_time" content="2021-04-06T22:48:44+03:00" />
 
 
 

docs/posts/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-04-06T22:33:43+03:00" />
+<meta property="og:updated_time" content="2021-04-06T22:48:44+03:00" />
 
 
 

docs/posts/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-04-06T22:33:43+03:00" />
+<meta property="og:updated_time" content="2021-04-06T22:48:44+03:00" />
 
 
 

docs/posts/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-04-06T22:33:43+03:00" />
+<meta property="og:updated_time" content="2021-04-06T22:48:44+03:00" />
 
 
 

docs/posts/page/7/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-04-06T22:33:43+03:00" />
+<meta property="og:updated_time" content="2021-04-06T22:48:44+03:00" />
 
 
 

docs/sitemap.xml

@@ -3,19 +3,19 @@
   xmlns:xhtml="http://www.w3.org/1999/xhtml">
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2021-04/</loc>
-    <lastmod>2021-04-06T22:33:43+03:00</lastmod>
+    <lastmod>2021-04-06T22:48:44+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/</loc>
-    <lastmod>2021-04-06T22:33:43+03:00</lastmod>
+    <lastmod>2021-04-06T22:48:44+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2021-04-06T22:33:43+03:00</lastmod>
+    <lastmod>2021-04-06T22:48:44+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
-    <lastmod>2021-04-06T22:33:43+03:00</lastmod>
+    <lastmod>2021-04-06T22:48:44+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
-    <lastmod>2021-04-06T22:33:43+03:00</lastmod>
+    <lastmod>2021-04-06T22:48:44+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/2021-03/</loc>
     <lastmod>2021-04-05T19:36:44+03:00</lastmod>