mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2025-01-27 05:49:12 +01:00
Add notes for 2023-08-18
This commit is contained in:
@ -80,4 +80,76 @@ log_line_prefix = '%t [%p]: user=%u,db=%d,app=%a,client=%h '
|
||||
|
||||
- I ported the DSpace 6.x REST API patch to allow specifying a bundle name when POSTing a bitstream to the legacy REST API in DSpace 7.6
|
||||
|
||||
## 2023-08-16
|
||||
|
||||
- I noticed that the DSpace statistics pages don't seem to work on communities or collections
|
||||
- I finally took time to look in the DSpace log file and found this for one:
|
||||
|
||||
```console
|
||||
2023-08-16 14:30:31,873 WARN dace8f96-f034-488e-b38c-9f2eb5d0e002 6cbd0b18-6852-4294-99a5-02dfcab0a469 org.dspace.app.rest.exception.DSpaceApiExceptionControllerAdvice @ Request is invalid or incorrect (status:400 exception: Invalid UUID string: -1 at: java.base/java.util.UUID.fromString1(UUID.java:280))
|
||||
```
|
||||
|
||||
- I'm surprised to see this because those should have been dealt with when we upgraded to DSpace 6
|
||||
- Looking in the Solr statistics core I see ~1,000,000 documents with the ID `-1`, and about 57,000,000 that don't
|
||||
- Also interesting, faceting by `dateYear` I see:
|
||||
- 2023: 209566
|
||||
- 2022: 403871
|
||||
- 2021: 336548
|
||||
- 2020: 31659
|
||||
- ... none before 2020
|
||||
- They are all type 5, which is "Site" aka the home page, according to `dspace-api/src/main/java/org/dspace/core/Constants.java`
|
||||
- Ah hah, and I can see in my DSpace 7 test Solr there are a bunch of hits with `type: 5` that have "-1" of course, but also newer ones that have an actual UUID
|
||||
- I used the `/server/api/dso/find?uuid=3945ec23-2426-4fce-a2ea-48b38b91547f` endpoint to find out that there is a new `/server/api/core/sites` endpoint listing exactly one site (the home page) with this ID
|
||||
- So for now I can replace all the "-1" documents with this ID on the test server at least, then I will have to remember to do that during the migration of the production instance
|
||||
- I did a new export from DSpace 6 using solr-import-export-json with a query limiting it to documents of type 5 and negative 1 ID:
|
||||
|
||||
```console
|
||||
$ chrt -b 0 ./run.sh -s http://localhost:8081/solr/statistics -a export -o /tmp/statistics-fix-uuid.json -f 'id:\-1 AND type:5 AND time:[2020-01-01T00\:00\:00Z TO 2023-12-31T23\:59\:59Z]' -k uid -S actingGroupId,actingGroupParentId,actorMemberGroupId,author_mtdt,author_mtdt_search,bitstreamCount,bitstreamId,complete_query,complete_query_search,containerBitstream,containerCollection,containerCommunity,containerItem,core_update_run_nb,countryCode_ngram,countryCode_search,cua_version,dateYear,dateYearMonth,file_id,filterquery,first_name,geoipcountrycode,geoIpCountryCode,group_id,group_map,group_name,ip_ngram,ip_search,isArchived,isInternal,iso_mtdt,iso_mtdt_search,isWithdrawn,last_name,name,ngram_query_search,ngram_simplequery_search,orphaned,parent_count,p_communities_id,p_communities_map,p_communities_name,p_group_id,p_group_map,p_group_name,range,rangeDescription,rangeDescription_ngram,rangeDescription_search,range_ngram,range_search,referrer_ngram,referrer_search,simple_query,simple_query_search,solr_update_time_stamp,storage_nb_of_bitstreams,storage_size,storage_statistics_type,subject_mtdt,subject_mtdt_search,text,userAgent_ngram,userAgent_search,version_id,workflowItemId
|
||||
```
|
||||
|
||||
- Then I replaced the IDs with the UUID of the site homepage on DSpace 7 Test:
|
||||
|
||||
```console
|
||||
$ sed -i 's/"id":"-1"/"id":"3945ec23-2426-4fce-a2ea-48b38b91547f"/' /tmp/statistics-fix-uuid.json
|
||||
```
|
||||
|
||||
- I re-imported those records and I no longer see the "-1" IDs, but still get the same error in the log
|
||||
- I don't understand, maybe there is some voodoo, so I rebooted the server
|
||||
- Hmm, no, it's not a voodoo cache issue, so I really need to debug this:
|
||||
|
||||
```console
|
||||
2023-08-16 15:44:07,122 WARN dace8f96-f034-488e-b38c-9f2eb5d0e002 036b88e6-7548-4852-9646-f345ce3bfcc2 org.dspace.app.rest.exception.DSpaceApiExceptionControllerAdvice @ Request is invalid or incorrect (status:400 exception: Invalid UUID string: -1 at: java.base/java.util.UUID.fromString1(UUID.java:280))
|
||||
```
|
||||
|
||||
- On a related note, I figured out that the root site already has a UUID in DSpace 6, and it's exactly the one above (3945ec23-2426-4fce-a2ea-48b38b91547f)
|
||||
- I noticed it while looking at the [DSpace 6 REST API's hierarchy page](https://cgspace.cgiar.org/rest/hierarchy)
|
||||
- So I can update these "-1" IDs with "type:5" in our production I think...
|
||||
|
||||
## 2023-08-17
|
||||
|
||||
- I decided to update the "-1" IDs in Solr on DSpace 6
|
||||
- Unfortunately, in Solr there is no way to update only documents matching a query, so we have to export and re-import
|
||||
- I exported all documents with "type:5" (Homepage) and replaced the ID in the JSON:
|
||||
|
||||
```console
|
||||
$ chrt -b 0 ./run.sh -s http://localhost:8081/solr/statistics -a export -o /tmp/statistics-fix-uuid.json -f 'type:5' -k uid -S actingGroupId,actingGroupParentId,actorMemberGroupId,author_mtdt,author_mtdt_search,bitstreamCount,bitstreamId,complete_query,complete_query_search,containerBitstream,containerCollection,containerCommunity,containerItem,core_update_run_nb,countryCode_ngram,countryCode_search,cua_version,dateYear,dateYearMonth,file_id,filterquery,first_name,geoipcountrycode,geoIpCountryCode,group_id,group_map,group_name,ip_ngram,ip_search,isArchived,isInternal,iso_mtdt,iso_mtdt_search,isWithdrawn,last_name,name,ngram_query_search,ngram_simplequery_search,orphaned,parent_count,p_communities_id,p_communities_map,p_communities_name,p_group_id,p_group_map,p_group_name,range,rangeDescription,rangeDescription_ngram,rangeDescription_search,range_ngram,range_search,referrer_ngram,referrer_search,simple_query,simple_query_search,solr_update_time_stamp,storage_nb_of_bitstreams,storage_size,storage_statistics_type,subject_mtdt,subject_mtdt_search,text,userAgent_ngram,userAgent_search,version_id,workflowItemId
|
||||
$ sed -i 's/"id":"-1"/"id":"3945ec23-2426-4fce-a2ea-48b38b91547f"/' /tmp/statistics-fix-uuid.json
|
||||
```
|
||||
|
||||
- (Oops, skipping the fields above was not necessary, since I'm importing back into DSpace 6 where those fields exist)
|
||||
- Then I re-imported:
|
||||
|
||||
```
|
||||
$ ./run.sh -s http://localhost:8081/solr/statistics -a import -o /tmp/statistics-fix-uuid.json -k uid
|
||||
```
|
||||
|
||||
- This worked, but I still see new records coming in that have "id:-1" so I will need to repeat this during the migration.
|
||||
- I also notice many stats records that have erroneous cities:
|
||||
- `"city":"com.maxmind.geoip2.record.City [ {} ]"`
|
||||
- `"city":"com.maxmind.geoip2.record.City [ {\"geoname_id\":1002145,\"names\":{\"de\":\"George\",\"en\":\"George\",\"ru\":\"Джордж\",\"fr\":\"George\",\"ja\":\"ジョージ\"}} ]"`
|
||||
|
||||
## 2023-08-18
|
||||
|
||||
- Export CGSpace to check for missing Initiative collection mappings
|
||||
|
||||
<!-- vim: set sw=2 ts=2: -->
|
||||
|
||||
Reference in New Issue
Block a user