diff --git a/content/posts/2020-12.md b/content/posts/2020-12.md
index c39827bb7..3e337e700 100644
--- a/content/posts/2020-12.md
+++ b/content/posts/2020-12.md
@@ -389,8 +389,85 @@ dspace=# COMMIT;
```console
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx512m"
$ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real 265m11.224s
+user 171m29.141s
+sys 2m41.097s
```
- Udana sent a report that the WLE approver is experiencing the same issue Peter highlighted a few weeks ago: they are unable to save metadata edits in the workflow
+- Yesterday Atmire responded about the owningComm and owningColl duplicates in Solr saying they didn't see any anymore...
+ - Indeed I spent a few minutes looking randomly and I didn't find any either...
+ - I did, however, see lots of duplicates in countryCode_search, countryCode_ngram, ip_search, ip_ngram, userAgent_search, userAgent_ngram, referrer_search, referrer_ngram fields
+ - I sent feedback to them
+- On the database locking front we haven't had issues in over a week and the Munin graphs look normal:
+
+
+
+
+- After the Discovery re-indexing finished on CGSpace I prepared to start re-harvesting AReS by making sure the `openrxv-items-temp` index was empty and that the backup index I made yesterday was still there:
+
+```console
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp?pretty'
+{
+ "acknowledged" : true
+}
+$ curl -s 'http://localhost:9200/openrxv-items-final/_count?q=*&pretty'
+{
+ "count" : 0,
+ "_shards" : {
+ "total" : 1,
+ "successful" : 1,
+ "skipped" : 0,
+ "failed" : 0
+ }
+}
+$ curl -s 'http://localhost:9200/openrxv-items-2020-12-14/_count?q=*&pretty'
+{
+ "count" : 99992,
+ "_shards" : {
+ "total" : 1,
+ "successful" : 1,
+ "skipped" : 0,
+ "failed" : 0
+ }
+}
+```
+
+## 2020-12-16
+
+- The harvesting on AReS finished last night so this morning I manually cloned the `openrxv-items-temp` index to `openrxv-items`
+ - First check the number of items in the temp index, then set it to read only, then delete the items index, then delete the temp index:
+
+```console
+$ curl -s 'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'
+{
+ "count" : 100046,
+ "_shards" : {
+ "total" : 1,
+ "successful" : 1,
+ "skipped" : 0,
+ "failed" : 0
+ }
+}
+$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings?pretty" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -XDELETE 'http://localhost:9200/openrxv-items?pretty'
+$ curl -s -X POST "http://localhost:9200/openrxv-items-temp/_clone/openrxv-items?pretty"
+$ curl -s 'http://localhost:9200/openrxv-items/_count?q=*&pretty'
+{
+ "count" : 100046,
+ "_shards" : {
+ "total" : 1,
+ "successful" : 1,
+ "skipped" : 0,
+ "failed" : 0
+ }
+}
+$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings?pretty" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": false}}'
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp?pretty'
+```
+
+- Interestingly [the item](https://hdl.handle.net/10568/110447) that we noticed was duplicated now only appears once
+- The [missing item](https://hdl.handle.net/10568/110133) is still missing
diff --git a/docs/2020-12/index.html b/docs/2020-12/index.html
index 427762f51..5970e2562 100644
--- a/docs/2020-12/index.html
+++ b/docs/2020-12/index.html
@@ -20,7 +20,7 @@ I started processing those (about 411,000 records):
-
+
@@ -46,9 +46,9 @@ I started processing those (about 411,000 records):
"@type": "BlogPosting",
"headline": "December, 2020",
"url": "https://alanorth.github.io/cgspace-notes/2020-12/",
- "wordCount": "2037",
+ "wordCount": "2609",
"datePublished": "2020-12-01T11:32:54+02:00",
- "dateModified": "2020-12-13T16:16:10+02:00",
+ "dateModified": "2020-12-15T13:13:18+02:00",
"author": {
"@type": "Person",
"name": "Alan Orth"
@@ -490,7 +490,122 @@ $ query-json '.items | length' /tmp/policy2.json
$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings?pretty" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items-2020-12-14
$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings?pretty" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": false}}'
-
+2020-12-15
+
+- After the re-harvest last night there were 200,000 items in the
openrxv-items-temp
index again
+
+- I cleared the core and started a re-harvest, but Peter sent me a bunch of author corrections for CGSpace so I decided to cancel it until after I apply them and re-index Discovery
+
+
+- I checked the 1,534 fixes in Open Refine (had to fix a few UTF-8 errors, as always from Peter’s CSVs) and then applied them using the
fix-metadata-values.py
script:
+
+$ ./fix-metadata-values.py -i /tmp/2020-10-28-fix-1534-Authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t 'correct' -m 3
+$ ./delete-metadata-values.py -i /tmp/2020-10-28-delete-2-Authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -m 3
+
+- Since I was re-indexing Discovery anyways I decided to check for any uppercase AGROVOC and lowercase them:
+
+dspace=# BEGIN;
+BEGIN
+dspace=# UPDATE metadatavalue SET text_value=LOWER(text_value) WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=57 AND text_value ~ '[[:upper:]]';
+UPDATE 406
+dspace=# COMMIT;
+COMMIT
+
+- I also updated the Font Awesome icon classes for version 5 syntax:
+
+dspace=# BEGIN;
+dspace=# UPDATE metadatavalue SET text_value = REGEXP_REPLACE(text_value, 'fa fa-rss','fas fa-rss', 'g') WHERE text_value LIKE '%fa fa-rss%';
+UPDATE 74
+dspace=# UPDATE metadatavalue SET text_value = REGEXP_REPLACE(text_value, 'fa fa-at','fas fa-at', 'g') WHERE text_value LIKE '%fa fa-at%';
+UPDATE 74
+dspace=# COMMIT;
+
+- Then I started a full Discovery re-index:
+
+$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx512m"
+$ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real 265m11.224s
+user 171m29.141s
+sys 2m41.097s
+
+- Udana sent a report that the WLE approver is experiencing the same issue Peter highlighted a few weeks ago: they are unable to save metadata edits in the workflow
+- Yesterday Atmire responded about the owningComm and owningColl duplicates in Solr saying they didn’t see any anymore…
+
+- Indeed I spent a few minutes looking randomly and I didn’t find any either…
+- I did, however, see lots of duplicates in countryCode_search, countryCode_ngram, ip_search, ip_ngram, userAgent_search, userAgent_ngram, referrer_search, referrer_ngram fields
+- I sent feedback to them
+
+
+- On the database locking front we haven’t had issues in over a week and the Munin graphs look normal:
+
+
+
+
+- After the Discovery re-indexing finished on CGSpace I prepared to start re-harvesting AReS by making sure the
openrxv-items-temp
index was empty and that the backup index I made yesterday was still there:
+
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp?pretty'
+{
+ "acknowledged" : true
+}
+$ curl -s 'http://localhost:9200/openrxv-items-final/_count?q=*&pretty'
+{
+ "count" : 0,
+ "_shards" : {
+ "total" : 1,
+ "successful" : 1,
+ "skipped" : 0,
+ "failed" : 0
+ }
+}
+$ curl -s 'http://localhost:9200/openrxv-items-2020-12-14/_count?q=*&pretty'
+{
+ "count" : 99992,
+ "_shards" : {
+ "total" : 1,
+ "successful" : 1,
+ "skipped" : 0,
+ "failed" : 0
+ }
+}
+
2020-12-16
+
+- The harvesting on AReS finished last night so this morning I manually cloned the
openrxv-items-temp
index to openrxv-items
+
+- First check the number of items in the temp index, then set it to read only, then delete the items index, then delete the temp index:
+
+
+
+$ curl -s 'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'
+{
+ "count" : 100046,
+ "_shards" : {
+ "total" : 1,
+ "successful" : 1,
+ "skipped" : 0,
+ "failed" : 0
+ }
+}
+$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings?pretty" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -XDELETE 'http://localhost:9200/openrxv-items?pretty'
+$ curl -s -X POST "http://localhost:9200/openrxv-items-temp/_clone/openrxv-items?pretty"
+$ curl -s 'http://localhost:9200/openrxv-items/_count?q=*&pretty'
+{
+ "count" : 100046,
+ "_shards" : {
+ "total" : 1,
+ "successful" : 1,
+ "skipped" : 0,
+ "failed" : 0
+ }
+}
+$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings?pretty" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": false}}'
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp?pretty'
+
+- Interestingly the item that we noticed was duplicated now only appears once
+- The missing item is still missing
+
+
diff --git a/docs/2020/12/postgres_connections_ALL-week2.png b/docs/2020/12/postgres_connections_ALL-week2.png
new file mode 100644
index 000000000..43e75793a
Binary files /dev/null and b/docs/2020/12/postgres_connections_ALL-week2.png differ
diff --git a/docs/2020/12/postgres_locks_ALL-week2.png b/docs/2020/12/postgres_locks_ALL-week2.png
new file mode 100644
index 000000000..f3e257ca0
Binary files /dev/null and b/docs/2020/12/postgres_locks_ALL-week2.png differ
diff --git a/docs/categories/index.html b/docs/categories/index.html
index dc5fd8090..b845c9af9 100644
--- a/docs/categories/index.html
+++ b/docs/categories/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/categories/notes/index.html b/docs/categories/notes/index.html
index 13f16b0e8..ed4f9866f 100644
--- a/docs/categories/notes/index.html
+++ b/docs/categories/notes/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/categories/notes/page/2/index.html b/docs/categories/notes/page/2/index.html
index ad9f0bf3f..4e54f4427 100644
--- a/docs/categories/notes/page/2/index.html
+++ b/docs/categories/notes/page/2/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/categories/notes/page/3/index.html b/docs/categories/notes/page/3/index.html
index c30ef4942..237faf052 100644
--- a/docs/categories/notes/page/3/index.html
+++ b/docs/categories/notes/page/3/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/categories/notes/page/4/index.html b/docs/categories/notes/page/4/index.html
index 84acd820d..53990d757 100644
--- a/docs/categories/notes/page/4/index.html
+++ b/docs/categories/notes/page/4/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/categories/notes/page/5/index.html b/docs/categories/notes/page/5/index.html
index 13ff454c8..50401e9d4 100644
--- a/docs/categories/notes/page/5/index.html
+++ b/docs/categories/notes/page/5/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/index.html b/docs/index.html
index b0927d9b1..10d083385 100644
--- a/docs/index.html
+++ b/docs/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/page/2/index.html b/docs/page/2/index.html
index db2a86f3d..5b43ca620 100644
--- a/docs/page/2/index.html
+++ b/docs/page/2/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/page/3/index.html b/docs/page/3/index.html
index 1f1cc8983..fc7bc8b71 100644
--- a/docs/page/3/index.html
+++ b/docs/page/3/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/page/4/index.html b/docs/page/4/index.html
index dba8579df..1555a7df4 100644
--- a/docs/page/4/index.html
+++ b/docs/page/4/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/page/5/index.html b/docs/page/5/index.html
index 362ba51fd..d38f6c48d 100644
--- a/docs/page/5/index.html
+++ b/docs/page/5/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/page/6/index.html b/docs/page/6/index.html
index 668449f25..c2d8e7909 100644
--- a/docs/page/6/index.html
+++ b/docs/page/6/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/page/7/index.html b/docs/page/7/index.html
index 5f74603cf..0d6b69f7d 100644
--- a/docs/page/7/index.html
+++ b/docs/page/7/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/index.html b/docs/posts/index.html
index 9447796f0..8f7ddca68 100644
--- a/docs/posts/index.html
+++ b/docs/posts/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/page/2/index.html b/docs/posts/page/2/index.html
index 86df01283..1993dc377 100644
--- a/docs/posts/page/2/index.html
+++ b/docs/posts/page/2/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/page/3/index.html b/docs/posts/page/3/index.html
index 396756935..645dc1684 100644
--- a/docs/posts/page/3/index.html
+++ b/docs/posts/page/3/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/page/4/index.html b/docs/posts/page/4/index.html
index b2f8e56cd..e5cbbdd4d 100644
--- a/docs/posts/page/4/index.html
+++ b/docs/posts/page/4/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/page/5/index.html b/docs/posts/page/5/index.html
index 73c23112c..3ca1553ab 100644
--- a/docs/posts/page/5/index.html
+++ b/docs/posts/page/5/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/page/6/index.html b/docs/posts/page/6/index.html
index 452eb9da4..fc79ef858 100644
--- a/docs/posts/page/6/index.html
+++ b/docs/posts/page/6/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/posts/page/7/index.html b/docs/posts/page/7/index.html
index aeca6cfc4..2be119d9e 100644
--- a/docs/posts/page/7/index.html
+++ b/docs/posts/page/7/index.html
@@ -10,7 +10,7 @@
-
+
diff --git a/docs/sitemap.xml b/docs/sitemap.xml
index 81b5da041..0be7e190b 100644
--- a/docs/sitemap.xml
+++ b/docs/sitemap.xml
@@ -4,27 +4,27 @@
https://alanorth.github.io/cgspace-notes/categories/
- 2020-12-13T16:16:10+02:00
+ 2020-12-15T13:13:18+02:00
https://alanorth.github.io/cgspace-notes/
- 2020-12-13T16:16:10+02:00
+ 2020-12-15T13:13:18+02:00
https://alanorth.github.io/cgspace-notes/2020-12/
- 2020-12-13T16:16:10+02:00
+ 2020-12-15T13:13:18+02:00
https://alanorth.github.io/cgspace-notes/categories/notes/
- 2020-12-13T16:16:10+02:00
+ 2020-12-15T13:13:18+02:00
https://alanorth.github.io/cgspace-notes/posts/
- 2020-12-13T16:16:10+02:00
+ 2020-12-15T13:13:18+02:00
diff --git a/static/2020/12/postgres_connections_ALL-week2.png b/static/2020/12/postgres_connections_ALL-week2.png
new file mode 100644
index 000000000..43e75793a
Binary files /dev/null and b/static/2020/12/postgres_connections_ALL-week2.png differ
diff --git a/static/2020/12/postgres_locks_ALL-week2.png b/static/2020/12/postgres_locks_ALL-week2.png
new file mode 100644
index 000000000..f3e257ca0
Binary files /dev/null and b/static/2020/12/postgres_locks_ALL-week2.png differ