Add notes for 2021-07-17

content/posts/2021-07.md

@@ -181,4 +181,254 @@ $ ./ilri/agrovoc-lookup.py -i /tmp/2021-07-06-all-subjects.txt -o /tmp/2021-07-0
 
 - Start an indexing on AReS
 
+## 2021-07-17
+
+- I'm in Cyprus mostly offline, but I noticed that CGSpace was down
+  - I checked and there was a blank white page with HTTP 200
+  - There are thousands of locks in PostgreSQL:
+
+```console
+postgres@linode18:~$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+2302
+postgres@linode18:~$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+2564
+postgres@linode18:~$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+2530
+```
+
+- The locks are held by XMLUI, not REST API or OAI:
+
+```console
+postgres@linode18:~$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | grep -o -E '(dspaceWeb|dspaceApi)' | sort | uniq -c | sort -n
+     57 dspaceApi
+   2671 dspaceWeb
+```
+
+- I ran all updates on the server (linode18) and restarted it, then DSpace came back up
+- I sent a message to Atmire, as I never heard from them last week when we blocked access to the REST API for two days for them to investigate the server issues
+- Clone the `openrxv-items-temp` index on AReS and re-run all the plugins, but most of the "dspace_add_missing_items" tasks failed so I will just run a full re-harvest
+- The load on CGSpace is 45.00... the nginx access.log is going so fast I can't even read it
+  - I see lots of IPs from AS206485 that are changing their user agents (Linux, Windows, macOS...)
+  - This is finegroupservers.com aka "UGB - UGB Hosting OU"
+  - I will get a list of their IP blocks from [ipinfo.app](https://asn.ipinfo.app/AS206485) and block them in nginx
+  - There is another group of IPs that are owned by an ISP called "TrafficTransitSolution LLC" that does not have its own ASN unfortunately
+  - "TrafficTransitSolution LLC" seems to be affiliated with AS206485 (UGB Hosting OU) anyways, but they sometimes use AS49453 Global Layer B.V.G also
+  - I found a tool that lets you grep a file by CIDR, so I can use that to purge hits from Solr eventually:
+
+```console
+# grepcidr 91.243.191.0/24 /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -n
+     32 91.243.191.124
+     33 91.243.191.129
+     33 91.243.191.200
+     34 91.243.191.115
+     34 91.243.191.154
+     34 91.243.191.234
+     34 91.243.191.56
+     35 91.243.191.187
+     35 91.243.191.91
+     36 91.243.191.58
+     37 91.243.191.209
+     39 91.243.191.119
+     39 91.243.191.144
+     39 91.243.191.55
+     40 91.243.191.112
+     40 91.243.191.182
+     40 91.243.191.57
+     40 91.243.191.98
+     41 91.243.191.106
+     44 91.243.191.79
+     45 91.243.191.151
+     46 91.243.191.103
+     56 91.243.191.172
+```
+
+- I found a few people complaining about these Russian attacks too:
+  - https://community.cloudflare.com/t/russian-ddos-completley-unmitigated-by-cloudflare/284578
+  - https://vklader.com/ddos-2020-may/
+- According to AbuseIPDB.com and whois data provided by the asn tool, I see these organizations, networks, and ISPs all seem to be related:
+  - Sharktech
+  - LIR LLC / lir.am
+  - TrafficTransitSolution LLC / traffictransitsolution.us
+  - Fine Group Servers Solutions LLC / finegroupservers.com
+  - UGB
+  - Bulgakov Alexey Yurievich
+  - Dmitry Vorozhtsov / fitz-isp.uk / UGB
+  - Auction LLC / dauction.ru / UGB / traffictransitsolution.us
+  - Alax LLC / alaxona.com / finegroupservers.com
+  - Sysoev Aleksey Anatolevich / jobbuzzactiv.com / traffictransitsolution.us
+  - Bulgakov Alexey Yurievich / UGB / blockchainnetworksolutions.co.uk / info@finegroupservers.com
+  - UAB Rakrejus
+- I looked in the nginx log and copied a few IP addresses that were suspicious
+  - First I looked them up in AbuseIPDB.com to get the ISP name and website
+  - Then I looked them up with the [asn](https://github.com/nitefood/asn) tool, ie:
+
+```console
+$ ./asn -n 45.80.217.235  
+
+╭──────────────────────────────╮
+│ ASN lookup for 45.80.217.235 │
+╰──────────────────────────────╯
+
+ 45.80.217.235 ┌PTR -
+               ├ASN 46844 (ST-BGP, US)
+               ├ORG Sharktech
+               ├NET 45.80.217.0/24 (TrafficTransitSolutionNet)
+               ├ABU info@traffictransitsolution.us
+               ├ROA ✓ VALID (1 ROA found)
+               ├TYP  Proxy host   Hosting/DC 
+               ├GEO Los Angeles, California (US)
+               └REP ✓ NONE
+```
+
+- Slowly slowly I manually built up a list of the IPs, ISP names, and network blocks, for example:
+
+```csv
+IP, Organization, Website, Network
+45.148.126.246, TrafficTransitSolution LLC, traffictransitsolution.us, 45.148.126.0/24 (Net-traffictransitsolution-15)
+45.138.102.253, TrafficTransitSolution LLC, traffictransitsolution.us, 45.138.102.0/24 (Net-traffictransitsolution-11)
+45.140.205.104, Bulgakov Alexey Yurievich, finegroupservers.com, 45.140.204.0/23 (CHINA_NETWORK)
+185.68.247.63, Fine Group Servers Solutions LLC, finegroupservers.com, 185.68.247.0/24 (Net-finegroupservers-18)
+213.232.123.188, Fine Group Servers Solutions LLC, finegroupservers.com, 213.232.123.0/24 (Net-finegroupservers-12)
+45.80.217.235, TrafficTransitSolution LLC, traffictransitsolution.us, 45.80.217.0/24 (TrafficTransitSolutionNet)
+185.81.144.202, Fine Group Servers Solutions LLC, finegroupservers.com, 185.81.144.0/24 (Net-finegroupservers-19)
+109.106.22.114, TrafficTransitSolution LLC, traffictransitsolution.us, 109.106.22.0/24 (TrafficTransitSolutionNet)
+185.68.247.200, Fine Group Servers Solutions LLC, finegroupservers.com, 185.68.247.0/24 (Net-finegroupservers-18)
+45.80.105.252, Bulgakov Alexey Yurievich, finegroupservers.com, 45.80.104.0/23 (NET-BNSL2-1)
+185.233.187.156, Dmitry Vorozhtsov, mgn-host.ru, 185.233.187.0/24 (GB-FITZISP-20181106)
+185.88.100.75, TrafficTransitSolution LLC, traffictransitsolution.us, 185.88.100.0/24 (Net-traffictransitsolution-17)
+194.104.8.154, TrafficTransitSolution LLC, traffictransitsolution.us, 194.104.8.0/24 (Net-traffictransitsolution-37)
+185.102.112.46, Fine Group Servers Solutions LLC, finegroupservers.com, 185.102.112.0/24 (Net-finegroupservers-13)
+212.193.12.64, Fine Group Servers Solutions LLC, finegroupservers.com, 212.193.12.0/24 (FINE_GROUP_SERVERS_SOLUTIONS_LLC)
+91.243.191.129, Auction LLC, dauction.ru, 91.243.191.0/24 (TR-QN-20180917)
+45.148.232.161, Nikolaeva Ekaterina Sergeevna, blockchainnetworksolutions.co.uk, 45.148.232.0/23 (LONDON_NETWORK)
+147.78.181.191, TrafficTransitSolution LLC, traffictransitsolution.us, 147.78.181.0/24 (Net-traffictransitsolution-58)
+77.83.27.90, Alax LLC, alaxona.com, 77.83.27.0/24 (FINEGROUPSERVERS-LEASE)
+185.250.46.119, Dmitry Vorozhtsov, mgn-host.ru, 185.250.46.0/23 (GB-FITZISP-20181106)
+94.231.219.106, LIR LLC, lir.am, 94.231.219.0/24 (CN-NET-219)
+45.12.65.56, Sysoev Aleksey Anatolevich, jobbuzzactiv.com / traffictransitsolution.us, 45.12.65.0/24 (TrafficTransitSolutionNet)
+45.140.206.31, Bulgakov Alexey Yurievich, blockchainnetworksolutions.co.uk / info@finegroupservers.com, 45.140.206.0/23 (FRANKFURT_NETWORK)
+84.54.57.130, LIR LLC, lir.am / traffictransitsolution.us, 84.54.56.0/23 (CN-FTNET-5456)
+178.20.214.235, Alaxona Internet Inc., alaxona.com / finegroupservers.com, 178.20.214.0/24 (FINEGROUPSERVERS-LEASE)
+37.44.253.204, Atex LLC, atex.ru / blockchainnetworksolutions.co.uk, 37.44.252.0/23 (NL-FTN-44252)
+46.161.61.242, Petersburg Internet Network Ltd., pinspb.ru / abusemail@depo40.ru, 46.161.61.0/24 (FineTransitDE)
+194.87.113.141, Fine Group Servers Solutions LLC, finegroupservers.com, 194.87.113.0/24 (FINE_GROUP_SERVERS_SOLUTIONS_LLC)
+109.94.223.217, LIR LLC, lir.am / traffictransitsolution.us, 109.94.223.0/24 (CN-NET-223)
+94.231.217.115, LIR LLC, lir.am / traffictransitsolution.us, 94.231.217.0/24 (TR-NET-217)
+146.185.202.214, Petersburg Internet Network Ltd., pinspb.ru / abusemail@depo40.ru / abuse@ripe.net, 146.185.202.0/24 (FineTransitRU)
+194.58.68.110, Fine Group Servers Solutions LLC, finegroupservers.com, 194.58.68.0/24 (FINE_GROUP_SERVERS_SOLUTIONS_LLC)
+94.154.131.237, TrafficTransitSolution LLC, traffictransitsolution.us, 94.154.131.0/24 (TrafficTransitSolutionNet)
+193.202.8.245, Fine Group Servers Solutions LLC, finegroupservers.com, 193.202.8.0/21 (FTL5)
+212.192.27.33, Fine Group Servers Solutions LLC, finegroupservers.com, 212.192.27.0/24 (FINE_GROUP_SERVERS_SOLUTIONS_LLC)
+193.202.87.218, Fine Group Servers Solutions LLC, finegroupservers.com, 193.202.84.0/22 (FTEL-2)
+146.185.200.52, Petersburg Internet Network Ltd., pinspb.ru / abusemail@depo40.ru / abuse@ripe.net, 146.185.200.0/24 (FineTransitRU)
+194.104.11.11, TrafficTransitSolution LLC, traffictransitsolution.us, 194.104.11.0/24 (Net-traffictransitsolution-40)
+185.50.250.145, ATOMOHOST LLC, atomohost.com, 185.50.250.0/24 (Silverstar_Invest_Limited)
+37.9.46.68, Petersburg Internet Network Ltd., pinspb.ru / abusemail@depo40.ru / abuse@ripe.net / , 37.9.44.0/22 (QUALITYNETWORK)
+185.81.145.14, Fine Group Servers Solutions LLC, finegroupservers.com, 185.81.145.0/24 (Net-finegroupservers-20)
+5.183.255.72, TrafficTransitSolution LLC, traffictransitsolution.us, 5.183.255.0/24 (Net-traffictransitsolution-32)
+84.54.58.204, LIR LLC, lir.am / traffictransitsolution.us, 84.54.58.0/24 (GB-BTTGROUP-20181119)
+109.236.55.175, Mosnet LLC, mosnetworks.ru / info@traffictransitsolution.us, 109.236.55.0/24 (CN-NET-55)
+5.133.123.184, Mosnet LLC, mosnet.ru / abuse@blockchainnetworksolutions.co.uk, 5.133.123.0/24 (DE-NET5133123)
+5.181.168.90, Fine Group Servers Solutions LLC, finegroupservers.com, 5.181.168.0/24 (Net-finegroupservers-5)
+185.61.217.86, TrafficTransitSolution LLC, traffictransitsolution.us, 185.61.217.0/24 (Net-traffictransitsolution-46)
+217.145.227.84, TrafficTransitSolution LLC, traffictransitsolution.us, 217.145.227.0/24 (Net-traffictransitsolution-64)
+193.56.75.29, Auction LLC, dauction.ru / abuse@blockchainnetworksolutions.co.uk, 193.56.75.0/24 (CN-NET-75)
+45.132.184.212, TrafficTransitSolution LLC, traffictransitsolution.us, 45.132.184.0/24 (Net-traffictransitsolution-5)
+45.10.167.239, TrafficTransitSolution LLC, traffictransitsolution.us, 45.10.167.0/24 (Net-traffictransitsolution-28)
+109.94.222.106, Express Courier LLC, expcourier.ru / info@traffictransitsolution.us, 109.94.222.0/24 (IN-NET-222)
+62.76.232.218, Fine Group Servers Solutions LLC, finegroupservers.com, 62.76.232.0/24 (FINE_GROUP_SERVERS_SOLUTIONS_LLC)
+147.78.183.221, TrafficTransitSolution LLC, traffictransitsolution.us, 147.78.183.0/24 (Net-traffictransitsolution-60)
+94.158.22.202, Auction LLC, dauction.ru / info@traffictransitsolution.us, 94.158.22.0/24 (FR-QN-20180917)
+85.202.194.33, Mosnet LLC, mosnet.ru / info@traffictransitsolution.us, 85.202.194.0/24 (DE-QN-20180917)
+193.187.93.150, Fine Group Servers Solutions LLC, finegroupservers.com, 193.187.92.0/22 (FTL3)
+185.250.45.149, Dmitry Vorozhtsov, mgn-host.ru / abuse@fitz-isp.uk, 185.250.44.0/23 (GB-FITZISP-20181106)
+185.50.251.75, ATOMOHOST LLC, atomohost.com, 185.50.251.0/24 (Silverstar_Invest_Limited)
+5.183.254.117, TrafficTransitSolution LLC, traffictransitsolution.us, 5.183.254.0/24 (Net-traffictransitsolution-31)
+45.132.186.187, TrafficTransitSolution LLC, traffictransitsolution.us, 45.132.186.0/24 (Net-traffictransitsolution-7)
+83.171.252.105, Teleport LLC, teleport.az / abuse@blockchainnetworksolutions.co.uk, 83.171.252.0/23 (DE-FTNET-252)
+45.148.127.37, TrafficTransitSolution LLC, traffictransitsolution.us, 45.148.127.0/24 (Net-traffictransitsolution-16)
+194.87.115.133, Fine Group Servers Solutions LLC, finegroupservers.com, 194.87.115.0/24 (FINE_GROUP_SERVERS_SOLUTIONS_LLC)
+193.233.250.100, OOO Freenet Group, free.net / abuse@vmage.ru, 193.233.250.0/24 (TrafficTransitSolutionNet)
+194.87.116.246, Fine Group Servers Solutions LLC, finegroupservers.com, 194.87.116.0/24 (FINE_GROUP_SERVERS_SOLUTIONS_LLC)
+195.133.25.244, Fine Group Servers Solutions LLC, finegroupservers.com, 195.133.25.0/24 (FINE_GROUP_SERVERS_SOLUTIONS_LLC)
+77.220.194.159, Fine Group Servers Solutions LLC, finegroupservers.com, 77.220.194.0/24 (Net-finegroupservers-3)
+185.89.101.177, ATOMOHOST LLC, atomohost.com, 185.89.100.0/23 (QUALITYNETWORK)
+193.151.191.133, Alax LLC, alaxona.com / info@finegroupservers.com, 193.151.191.0/24 (FINEGROUPSERVERS-LEASE)
+5.181.170.147, Fine Group Servers Solutions LLC, finegroupservers.com, 5.181.170.0/24 (Net-finegroupservers-7)
+193.233.249.167, OOO Freenet Group, free.net / abuse@vmage.ru, 193.233.249.0/24 (TrafficTransitSolutionNet)
+46.161.59.90, Petersburg Internet Network Ltd., pinspb.ru / abusemail@depo40.ru, 46.161.59.0/24 (FineTransitJP)
+213.108.3.74, TrafficTransitSolution LLC, traffictransitsolution.us, 213.108.3.0/24 (Net-traffictransitsolution-24)
+193.233.251.238, OOO Freenet Group, free.net / abuse@vmage.ru, 193.233.251.0/24 (TrafficTransitSolutionNet)
+178.20.215.224, Alaxona Internet Inc., alaxona.com / info@finegroupservers.com, 178.20.215.0/24 (FINEGROUPSERVERS-LEASE)
+45.159.22.199, Server LLC, ixserv.ru / info@finegroupservers.com, 45.159.22.0/24 (FINEGROUPSERVERS-LEASE)
+109.236.53.244, Mosnet LLC, mosnet.ru, info@traffictransitsolution.us, 109.236.53.0/24 (TR-NET-53)
+```
+
+- I found a better way to get the ASNs using my `resolve-addresses-geoip2.py` script
+  - First, get a list of all IPs making requests to nginx today:
+
+```console
+# grep -v -E "(mahider|Googlebot|Turnitin|Grammarly|Unpaywall|UptimeRobot|bot)" /var/log/nginx/access.log | awk '{print $1}' | sort | uniq  > /tmp/ips-sorted.txt
+# wc -l /tmp/ips-sorted.txt 
+10776 /tmp/ips-sorted.txt
+```
+
+- Then resolve them all:
+
+```console:
+$ ./ilri/resolve-addresses-geoip2.py -i /tmp/ips-sorted.txt -o /tmp/out.csv
+```
+
+- Then get the top 10 organizations and top ten ASNs:
+
+```console
+$ csvcut -c 2 /tmp/out.csv | sed 1d | sort | uniq -c | sort -n | tail -n 10
+    213 AMAZON-AES
+    218 ASN-QUADRANET-GLOBAL
+    246 Silverstar Invest Limited
+    347 Ethiopian Telecommunication Corporation
+    475 DEDIPATH-LLC
+    504 AS-COLOCROSSING
+    598 UAB Rakrejus
+    814 UGB Hosting OU
+   1010 ST-BGP
+   1757 Global Layer B.V.
+$ csvcut -c 3 /tmp/out.csv | sed 1d | sort | uniq -c | sort -n | tail -n 10
+    213 14618
+    218 8100
+    246 35624
+    347 24757
+    475 35913
+    504 36352
+    598 62282
+    814 206485
+   1010 46844
+   1757 49453
+```
+
+- I will download blocklists for all these except Ethiopian Telecom, Quadranet, and Amazon, though I'm concerned about Global Layer because it's a huge ASN that seems to have legit hosts too...?
+
+```console
+$ wget https://asn.ipinfo.app/api/text/nginx/AS49453
+$ wget https://asn.ipinfo.app/api/text/nginx/AS46844
+$ wget https://asn.ipinfo.app/api/text/nginx/AS206485
+$ wget https://asn.ipinfo.app/api/text/nginx/AS62282
+$ wget https://asn.ipinfo.app/api/text/nginx/AS36352
+$ wget https://asn.ipinfo.app/api/text/nginx/AS35624
+$ cat AS* | sort | uniq > /tmp/abusive-networks.txt
+$ wc -l /tmp/abusive-networks.txt 
+2276 /tmp/abusive-networks.txt
+```
+
+- Combining with my existing rules and filtering uniques:
+
+```console
+$ cat roles/dspace/templates/nginx/abusive-networks.conf.j2 /tmp/abusive-networks.txt | grep deny | sort | uniq | wc -l
+2298
+```
+
+- [According to Scamlytics all these are high risk ISPs](https://scamalytics.com/ip/isp) (as recently as 2021-06) so I will just keep blocking them
+- I deployed the block list on CGSpace (linode18) and the load is down to 1.0 but I see there are still some DDoS IPs getting through... sigh
+- The next thing I need to do is purge all the IPs from Solr using grepcidr...
+
 <!-- vim: set sw=2 ts=2: -->

docs/categories/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/" />
-<meta property="og:updated_time" content="2021-07-11T15:56:57+03:00" />
+<meta property="og:updated_time" content="2021-07-11T15:59:16+03:00" />
 
 
 

docs/categories/notes/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-07-11T15:56:57+03:00" />
+<meta property="og:updated_time" content="2021-07-11T15:59:16+03:00" />
 
 
 

docs/categories/notes/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-07-11T15:56:57+03:00" />
+<meta property="og:updated_time" content="2021-07-11T15:59:16+03:00" />
 
 
 

docs/categories/notes/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-07-11T15:56:57+03:00" />
+<meta property="og:updated_time" content="2021-07-11T15:59:16+03:00" />
 
 
 

docs/categories/notes/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-07-11T15:56:57+03:00" />
+<meta property="og:updated_time" content="2021-07-11T15:59:16+03:00" />
 
 
 

docs/categories/notes/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
-<meta property="og:updated_time" content="2021-07-11T15:56:57+03:00" />
+<meta property="og:updated_time" content="2021-07-11T15:59:16+03:00" />
 
 
 

docs/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-07-11T15:56:57+03:00" />
+<meta property="og:updated_time" content="2021-07-11T15:59:16+03:00" />
 
 
 

docs/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-07-11T15:56:57+03:00" />
+<meta property="og:updated_time" content="2021-07-11T15:59:16+03:00" />
 
 
 

docs/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-07-11T15:56:57+03:00" />
+<meta property="og:updated_time" content="2021-07-11T15:59:16+03:00" />
 
 
 

docs/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-07-11T15:56:57+03:00" />
+<meta property="og:updated_time" content="2021-07-11T15:59:16+03:00" />
 
 
 

docs/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-07-11T15:56:57+03:00" />
+<meta property="og:updated_time" content="2021-07-11T15:59:16+03:00" />
 
 
 

docs/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-07-11T15:56:57+03:00" />
+<meta property="og:updated_time" content="2021-07-11T15:59:16+03:00" />
 
 
 

docs/page/7/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-07-11T15:56:57+03:00" />
+<meta property="og:updated_time" content="2021-07-11T15:59:16+03:00" />
 
 
 

docs/page/8/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/" />
-<meta property="og:updated_time" content="2021-07-11T15:56:57+03:00" />
+<meta property="og:updated_time" content="2021-07-11T15:59:16+03:00" />
 
 
 

docs/posts/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-07-11T15:56:57+03:00" />
+<meta property="og:updated_time" content="2021-07-11T15:59:16+03:00" />
 
 
 

docs/posts/page/2/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-07-11T15:56:57+03:00" />
+<meta property="og:updated_time" content="2021-07-11T15:59:16+03:00" />
 
 
 

docs/posts/page/3/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-07-11T15:56:57+03:00" />
+<meta property="og:updated_time" content="2021-07-11T15:59:16+03:00" />
 
 
 

docs/posts/page/4/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-07-11T15:56:57+03:00" />
+<meta property="og:updated_time" content="2021-07-11T15:59:16+03:00" />
 
 
 

docs/posts/page/5/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-07-11T15:56:57+03:00" />
+<meta property="og:updated_time" content="2021-07-11T15:59:16+03:00" />
 
 
 

docs/posts/page/6/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-07-11T15:56:57+03:00" />
+<meta property="og:updated_time" content="2021-07-11T15:59:16+03:00" />
 
 
 

docs/posts/page/7/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-07-11T15:56:57+03:00" />
+<meta property="og:updated_time" content="2021-07-11T15:59:16+03:00" />
 
 
 

docs/posts/page/8/index.html

@@ -10,7 +10,7 @@
 <meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
 <meta property="og:type" content="website" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/posts/" />
-<meta property="og:updated_time" content="2021-07-11T15:56:57+03:00" />
+<meta property="og:updated_time" content="2021-07-11T15:59:16+03:00" />
 
 
 

docs/sitemap.xml

@@ -3,22 +3,22 @@
   xmlns:xhtml="http://www.w3.org/1999/xhtml">
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/</loc>
-    <lastmod>2021-07-11T15:56:57+03:00</lastmod>
+    <lastmod>2021-07-11T15:59:16+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2021-07-11T15:56:57+03:00</lastmod>
+    <lastmod>2021-07-11T15:59:16+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/2021-06/</loc>
     <lastmod>2021-07-01T08:53:21+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
-    <lastmod>2021-07-11T15:56:57+03:00</lastmod>
+    <lastmod>2021-07-11T15:59:16+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
-    <lastmod>2021-07-11T15:56:57+03:00</lastmod>
+    <lastmod>2021-07-11T15:59:16+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/2021-06/</loc>
-    <lastmod>2021-07-11T15:56:57+03:00</lastmod>
+    <lastmod>2021-07-11T15:59:16+03:00</lastmod>
   </url><url>
     <loc>https://alanorth.github.io/cgspace-notes/2021-05/</loc>
     <lastmod>2021-07-06T17:03:55+03:00</lastmod>