diff --git a/content/posts/2023-07.md b/content/posts/2023-07.md
index 03bd3ab15..cde7bbd22 100644
--- a/content/posts/2023-07.md
+++ b/content/posts/2023-07.md
@@ -206,4 +206,92 @@ $ chrt -b 0 ./run.sh -s http://localhost:8081/solr/statistics -a export -o /tmp/
- After importing the Solr statistics into DSpace 7 I realized that my DSpace Statistics API will work fine
- I made some minor modifications to the Ansible infrastructure scripts to make sure it is enabled and then activated it on DSpace 7 Test
+## 2023-07-26
+
+- Debugging lock issues on CGSpace
+ - I see the blocking PIDs for some long-held locks are "idle in transaction":
+
+```console
+$ ps auxw | grep -E "(1864132|1659487)"
+postgres 1659487 0.0 0.5 3269900 197120 ? Ss Jul25 0:03 postgres: 14/main: cgspace cgspace 127.0.0.1(61648) idle in transaction
+postgres 1864132 0.1 0.7 3275704 254528 ? Ss 07:27 0:08 postgres: 14/main: cgspace cgspace 127.0.0.1(36998) idle in transaction
+postgres 1880388 0.0 0.0 9208 2432 pts/3 S+ 08:48 0:00 grep -E (1864132|1659487)
+```
+
+- I used some other scripts and found that those processes were executing the following statement:
+
+```console
+select nextval ('public.tasklistitem_seq')
+```
+
+- I don't know why these can get blocked for hours without resolution, but for now I just killed them
+ - For what it's worth [these sequences were removed in DSpace 7.0](https://github.com/DSpace/DSpace/commit/16ae96b4c3d833c2a4acd1f05985d424c3a52bd7) along with the "traditional" item workflow—maybe that means we won't have such contention issues in DSpace 7!
+- I wrote a slightly longer regex to match locks that have been stuck for more than 1 hour based on the output of the `locks-age.sql` script and killed them:
+
+```console
+$ psql < locks-age.sql | awk -F"|" '/ [[:digit:]][1-9]:[[:digit:]]{2}:[[:digit:]]{2}\./ {print $10}' | sort -u | xargs kill
+```
+
+- I filed [an issue for missing Altmetric badges on DSpace 7 Angular](https://github.com/DSpace/dspace-angular/issues/2400)
+
+## 2023-07-27
+
+- Export CGSpace to check countries, regions, types, and Initiatives
+ - There were a few minor issues in countries and regions, and I noticed 186 items without types!
+ - Then I ran the file through csv-metadata-quality to make sure items with countries have appropriate regions
+- Brief discussion about OpenRXV bugs and fixes with Moayad
+- I was toying with the idea of using an expanded whitespace check/fix based on [ESLint's no-irregular-whitespace](https://eslint.org/docs/latest/rules/no-irregular-whitespace) rule in csv-metadata-quality
+ - I found 176 items in CGSpace with such whitespace in their titles alone
+ - I compared the results of removing these characters and replacing them with a space
+ - In _most_ cases removing it is the correct thing to do, for example "Pesticides : une arme à double tranchant" → "Pesticides: une arme à double tranchant"
+ - But in some items it is tricky, for example "L'environnement juridique est-il propice à la gestion" → "L'environnement juridique est-il propice àla gestion"
+ - I guess it would really need some good heuristics or a human to verify...
+- I upgraded OpenRXV to Angular v14
+
+## 2023-07-28
+
+- After a bit more testing I merged the [Angular v14 changes to OpenRXV master](https://github.com/ilri/OpenRXV/pull/184)
+- I am getting an error trying to import the 2020 Solr statistics from CGSpace to DSpace 7:
+
+```console
+Exception in thread "main" org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error from server at http://localhost:8983/solr/statistics: ERROR: [doc=0008a7c1-e552-4a4e-93e4-4d23bf39964b] Error adding field 'workflowItemId'='0812be47-1bfe-45e2-9208-5bf10ee46f81' msg=For input string: "0812be47-1bfe-45e2-9208-5bf10ee46f81"
+ at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:745)
+ at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
+ at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:240)
+ at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:234)
+ at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:102)
+ at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:69)
+ at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:82)
+ at it.damore.solr.importexport.App.insertBatch(App.java:295)
+ at it.damore.solr.importexport.App.lambda$writeAllDocuments$10(App.java:276)
+ at it.damore.solr.importexport.BatchCollector.lambda$accumulator$0(BatchCollector.java:71)
+ at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
+ at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
+ at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845)
+ at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
+ at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
+ at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
+ at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
+ at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
+ at it.damore.solr.importexport.App.writeAllDocuments(App.java:252)
+ at it.damore.solr.importexport.App.main(App.java:150)
+```
+
+- Ahhhh, in DSpace 6 this field was a string in the Solr statistics schema, but in DSpace 7 it is an integer...?
+ - Oh, it seems to be an Atmire change in our DSpace 6... hmmm, so we need to ignore the `workflowItemId` field when exporting
+ - Upstream: https://github.com/DSpace/DSpace/blob/dspace-6_x/dspace/solr/statistics/conf/schema.xml#L328
+ - ILRI: https://github.com/ilri/DSpace/blob/6_x-prod/dspace/solr/statistics/conf/schema.xml#L344
+ - I am wondering if we can skip all these workflow fields since I don't think we are using any aspects of statistics related to workflows
+ - I diffed our Solr statistics schema with the one from vanilla DSpace 6 and got a list of all the fields that were different:
+
+```
+isInternal,workflowItemId,containerCommunity,containerCollection,containerItem,containerBitstream,dateYear,dateYearMonth,filterquery,complete_query,simple_query,complete_query_search,simple_query_search,ngram_query_search,ngram_simplequery_search,text,storage_statistics_type,storage_size,storage_nb_of_bitstreams,name,first_name,last_name,p_communities_id,p_communities_name,p_communities_map,p_group_id,p_group_name,p_group_map,group_id,group_name,group_map,parent_count,bitstreamId,bitstreamCount,actingGroupId,actorMemberGroupId,actingGroupParentId,rangeDescription,range,version_id,file_id,cua_version,core_update_run_nb,orphaned
+```
+
+- I will combine it with the other fields I was skipping above and try the export again:
+
+```console
+$ chrt -b 0 ./run.sh -s http://localhost:8081/solr/statistics -a export -o /tmp/statistics-2020.json -f 'time:[2020-01-01T00\:00\:00Z TO 2020-12-31T23\:59\:59Z]' -k uid -S actingGroupId,actingGroupParentId,actorMemberGroupId,author_mtdt,author_mtdt_search,bitstreamCount,bitstreamId,complete_query,complete_query_search,containerBitstream,containerCollection,containerCommunity,containerItem,core_update_run_nb,countryCode_ngram,countryCode_search,cua_version,dateYear,dateYearMonth,file_id,filterquery,first_name,geoipcountrycode,geoIpCountryCode,group_id,group_map,group_name,ip_ngram,ip_search,isArchived,isInternal,iso_mtdt,iso_mtdt_search,isWithdrawn,last_name,name,ngram_query_search,ngram_simplequery_search,orphaned,parent_count,p_communities_id,p_communities_map,p_communities_name,p_group_id,p_group_map,p_group_name,range,rangeDescription,rangeDescription_ngram,rangeDescription_search,range_ngram,range_search,referrer_ngram,referrer_search,simple_query,simple_query_search,solr_update_time_stamp,storage_nb_of_bitstreams,storage_size,storage_statistics_type,subject_mtdt,subject_mtdt_search,text,userAgent_ngram,userAgent_search,version_id,workflowItemId
+```
+
diff --git a/docs/2023-07/index.html b/docs/2023-07/index.html
index ab62b7a55..bd76c6cca 100644
--- a/docs/2023-07/index.html
+++ b/docs/2023-07/index.html
@@ -11,7 +11,7 @@
-
+
@@ -28,9 +28,9 @@
"@type": "BlogPosting",
"headline": "July, 2023",
"url": "https://alanorth.github.io/cgspace-notes/2023-07/",
- "wordCount": "1503",
+ "wordCount": "2096",
"datePublished": "2023-07-01T17:14:36+03:00",
- "dateModified": "2023-07-22T09:19:48+03:00",
+ "dateModified": "2023-07-25T23:54:53+03:00",
"author": {
"@type": "Person",
"name": "Alan Orth"
@@ -337,7 +337,100 @@
-
+2023-07-26
+
+- Debugging lock issues on CGSpace
+
+- I see the blocking PIDs for some long-held locks are “idle in transaction”:
+
+
+
+$ ps auxw | grep -E "(1864132|1659487)"
+postgres 1659487 0.0 0.5 3269900 197120 ? Ss Jul25 0:03 postgres: 14/main: cgspace cgspace 127.0.0.1(61648) idle in transaction
+postgres 1864132 0.1 0.7 3275704 254528 ? Ss 07:27 0:08 postgres: 14/main: cgspace cgspace 127.0.0.1(36998) idle in transaction
+postgres 1880388 0.0 0.0 9208 2432 pts/3 S+ 08:48 0:00 grep -E (1864132|1659487)
+
+- I used some other scripts and found that those processes were executing the following statement:
+
+select nextval ('public.tasklistitem_seq')
+
+- I don’t know why these can get blocked for hours without resolution, but for now I just killed them
+
+
+- I wrote a slightly longer regex to match locks that have been stuck for more than 1 hour based on the output of the
locks-age.sql script and killed them:
+
+$ psql < locks-age.sql | awk -F"|" '/ [[:digit:]][1-9]:[[:digit:]]{2}:[[:digit:]]{2}\./ {print $10}' | sort -u | xargs kill
+
2023-07-27
+
+- Export CGSpace to check countries, regions, types, and Initiatives
+
+- There were a few minor issues in countries and regions, and I noticed 186 items without types!
+- Then I ran the file through csv-metadata-quality to make sure items with countries have appropriate regions
+
+
+- Brief discussion about OpenRXV bugs and fixes with Moayad
+- I was toying with the idea of using an expanded whitespace check/fix based on ESLint’s no-irregular-whitespace rule in csv-metadata-quality
+
+- I found 176 items in CGSpace with such whitespace in their titles alone
+- I compared the results of removing these characters and replacing them with a space
+- In most cases removing it is the correct thing to do, for example “Pesticides : une arme à double tranchant” → “Pesticides: une arme à double tranchant”
+- But in some items it is tricky, for example “L’environnement juridique est-il propice à la gestion” → “L’environnement juridique est-il propice àla gestion”
+- I guess it would really need some good heuristics or a human to verify…
+
+
+- I upgraded OpenRXV to Angular v14
+
+2023-07-28
+
+Exception in thread "main" org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error from server at http://localhost:8983/solr/statistics: ERROR: [doc=0008a7c1-e552-4a4e-93e4-4d23bf39964b] Error adding field 'workflowItemId'='0812be47-1bfe-45e2-9208-5bf10ee46f81' msg=For input string: "0812be47-1bfe-45e2-9208-5bf10ee46f81"
+ at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:745)
+ at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
+ at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:240)
+ at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:234)
+ at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:102)
+ at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:69)
+ at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:82)
+ at it.damore.solr.importexport.App.insertBatch(App.java:295)
+ at it.damore.solr.importexport.App.lambda$writeAllDocuments$10(App.java:276)
+ at it.damore.solr.importexport.BatchCollector.lambda$accumulator$0(BatchCollector.java:71)
+ at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
+ at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
+ at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845)
+ at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
+ at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
+ at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
+ at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
+ at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
+ at it.damore.solr.importexport.App.writeAllDocuments(App.java:252)
+ at it.damore.solr.importexport.App.main(App.java:150)
+
+- Ahhhh, in DSpace 6 this field was a string in the Solr statistics schema, but in DSpace 7 it is an integer…?
+
+- Oh, it seems to be an Atmire change in our DSpace 6… hmmm, so we need to ignore the
workflowItemId field when exporting
+
+
+- I am wondering if we can skip all these workflow fields since I don’t think we are using any aspects of statistics related to workflows
+- I diffed our Solr statistics schema with the one from vanilla DSpace 6 and got a list of all the fields that were different:
+
+
+
+isInternal,workflowItemId,containerCommunity,containerCollection,containerItem,containerBitstream,dateYear,dateYearMonth,filterquery,complete_query,simple_query,complete_query_search,simple_query_search,ngram_query_search,ngram_simplequery_search,text,storage_statistics_type,storage_size,storage_nb_of_bitstreams,name,first_name,last_name,p_communities_id,p_communities_name,p_communities_map,p_group_id,p_group_name,p_group_map,group_id,group_name,group_map,parent_count,bitstreamId,bitstreamCount,actingGroupId,actorMemberGroupId,actingGroupParentId,rangeDescription,range,version_id,file_id,cua_version,core_update_run_nb,orphaned
+
+- I will combine it with the other fields I was skipping above and try the export again:
+
+$ chrt -b 0 ./run.sh -s http://localhost:8081/solr/statistics -a export -o /tmp/statistics-2020.json -f 'time:[2020-01-01T00\:00\:00Z TO 2020-12-31T23\:59\:59Z]' -k uid -S actingGroupId,actingGroupParentId,actorMemberGroupId,author_mtdt,author_mtdt_search,bitstreamCount,bitstreamId,complete_query,complete_query_search,containerBitstream,containerCollection,containerCommunity,containerItem,core_update_run_nb,countryCode_ngram,countryCode_search,cua_version,dateYear,dateYearMonth,file_id,filterquery,first_name,geoipcountrycode,geoIpCountryCode,group_id,group_map,group_name,ip_ngram,ip_search,isArchived,isInternal,iso_mtdt,iso_mtdt_search,isWithdrawn,last_name,name,ngram_query_search,ngram_simplequery_search,orphaned,parent_count,p_communities_id,p_communities_map,p_communities_name,p_group_id,p_group_map,p_group_name,range,rangeDescription,rangeDescription_ngram,rangeDescription_search,range_ngram,range_search,referrer_ngram,referrer_search,simple_query,simple_query_search,solr_update_time_stamp,storage_nb_of_bitstreams,storage_size,storage_statistics_type,subject_mtdt,subject_mtdt_search,text,userAgent_ngram,userAgent_search,version_id,workflowItemId
+