alanorth/cgspace-notes
1
0
Fork 0
mirror of https://github.com/alanorth/cgspace-notes.git synced 2025-01-27 05:49:12 +01:00
Code Issues Projects Releases Wiki Activity

Add notes for 2020-11-18

Browse Source
This commit is contained in:
Alan Orth
2020-11-18 23:15:06 +02:00
parent 2557931751
commit efbfbf46af
28 changed files with 372 additions and 55 deletions
Download Patch File Download Diff File Expand all files Collapse all files

45
content/posts/2020-11.md
View File

@ -352,4 +352,49 @@ $ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid =
2071
```
## 2020-11-18
- I decided to enable the `rollbackOnReturn=true` option in [Tomcat's JDBC connection pool parameters](https://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html) because I noticed that all of the "idle in transaction" connections waiting for locks were SELECT queries
- There are many posts on the Internet about people having this issue with Hibernate
- The locks are lower now, but Peter and Abenet are still having issues approving items and Tezira forwarded one strange case where an item was "approved" and was assigned a handle, but it doesn't exist...
- I sent another mail to the dspace-tech mailing list to ask for help
- I reverted the `rollbackOnReturn` change in Tomcat...
- I sent a message to Atmire to ask for urgent help
- Call with IWMI and Abenet about them potentially moving from InMagic to CGSpace
- They have questions about the reporting on AReS
- We told them that we can use collections to infer Strategic Priorities and Research Groups and WLE Flagships
- It sounds like we will create this structure under the top-level IWMI community:
- IWMI Strategic Priorities (sub-community)
- Water, Food and Ecosystems (sub-community)
- Sustainable and Resilient Food Production Systems (collection)
- Sustainable Water infrastructure and Ecosystems (collection)
- Integrated Basin and Aquifer Management
- Water, Climate Change and Resilience (sub-community)
- Climate Change Adaptation and Resilience (collection)
- etc...
- They will submit items to their normal output type collections and map to these
- In other news I finally finished processing the Solr statistics for UUIDs and re-indexed the stats with the dspace-statistics-api
- I started the Atmire stats processing, notes in the dedicated [CGSpace DSpace 6 Upgrade section]({{< relref "cgspace-dspace6-upgrade.md" >}})
- Peter got a strange message this evening when trying to update metadata:
```
2020-11-18 16:57:33,309 ERROR org.hibernate.engine.jdbc.batch.internal.BatchingBatch @ HHH000315: Exception executing batch [Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1]
2020-11-18 16:57:33,316 ERROR org.hibernate.engine.jdbc.batch.internal.BatchingBatch @ HHH000315: Exception executing batch [Batch update returned unexpected row count from update [13]; actual row count: 0; expected: 1]
2020-11-18 16:57:33,385 INFO org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl @ HHH000010: On release of batch it still contained JDBC statements
```
- Minor bug fixes to limit parameter in DSpace Statistics API
- Release [version 1.3.2](https://github.com/ilri/dspace-statistics-api/releases/tag/v1.3.2)
- Send a list of potential ToRs for a next phase of OpenRXV development to Michael Victor for feedback:
- Enable advanced reporting templates using "Angular expressions" in Docxtemplater (would be used immediately for IWMI and Bioversity–CIAT)
- Enable embedding of charts like world map and word cloud in reports
- Enable embedding of item thumbnails in reports, similar to the "list of information products"
- Enable something like the "Statistics" Excel report Peter wanted in 2019 so we can get community and collection statistics reports
- Add a new "metrics" block with statistics about top authors and items by number of views and downloads for the current search terms
- Add ability to change the explorer UI to "Usage Statistics" mode where lists of authors, affiliations, sponsors, CRPs, communities, collections, etc are sorted according to the number of views or downloads for the current search results, rather than by number of occurrences of metadata values
- Add ability to "drill down" or modify search filter terms by clicking on countries in the map
- Enable date-based usage statistics (currently only "all time" statistics are available)
- Fixing minor bugs for all issues filed on GitHub
- I also added GitHub issues for each of them
<!-- vim: set sw=2 ts=2: -->

137
content/posts/cgspace-dspace6-upgrade.md
View File

@ -11,7 +11,8 @@ Notes about the DSpace 6 upgrade on CGSpace in 2020-11.
<!--more-->
- [Processing Solr Statistics With solr-upgrade-statistics-6x](#processing-solr-statistics-with-solr-upgrade-statistics-6x)
- [Re-import OAI with clean index](#re-import-oai-with-clean-index)
- [Processing Solr statistics with solr-upgrade-statistics-6x](#processing-solr-statistics-with-solr-upgrade-statistics-6x)
- [Current year's statistics core](#statistics)
- [statistics-2019 core](#statistics-2019)
- [statistics-2018 core](#statistics-2018)
@ -20,11 +21,28 @@ Notes about the DSpace 6 upgrade on CGSpace in 2020-11.
- [statistics-2015 core](#statistics-2015)
- [statistics-2014 core](#statistics-2014)
- [statistics-2013 core](#statistics-2013)
- [statistics-2013 core](#statistics-2012)
- [statistics-2013 core](#statistics-2011)
- [statistics-2013 core](#statistics-2010)
- [Processing Solr statistics with AtomicStatisticsUpdateCLI](processing-solr-statistics-with-atomicstatisticsupdatecli)
## Processing Solr Statistics With solr-upgrade-statistics-6x
### Re-import OAI with clean index
After the upgrade is complete, re-index all items into OAI with a clean index:
```console
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx2048m"
$ dspace oai -c import
```
The process ran out of memory several times so I had to keep trying again with more JVM heap memory.
### Processing Solr Statistics With solr-upgrade-statistics-6x
After the main upgrade process was finished and DSpace was running I started processing the Solr statistics with `solr-upgrade-statistics-6x` to migrate all IDs to UUIDs.
### statistics
## statistics
First process the current year's statistics core:
```console
@ -57,7 +75,7 @@ After several rounds of processing it finished. Here are some statistics about u
$ curl -s "http://localhost:8081/solr/statistics/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>*:* NOT id:/.{36}/</query></delete>"
```
### statistics-2019
## statistics-2019
Processing the statistics-2019 core:
```console
@ -89,7 +107,7 @@ After several rounds of processing it finished. Here are some statistics about u
$ curl -s "http://localhost:8081/solr/statistics-2019/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>*:* NOT id:/.{36}/</query></delete>"
```
### statistics-2018
## statistics-2018
Processing the statistics-2018 core:
```console
@ -161,7 +179,7 @@ Eventually the processing finished. Here are some statistics about unmigrated do
$ curl -s "http://localhost:8081/solr/statistics-2017/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>*:* NOT id:/.{36}/</query></delete>"
```
### statistics-2016
## statistics-2016
Processing the statistics-2016 core:
@ -192,7 +210,8 @@ $ chrt -b 0 dspace solr-upgrade-statistics-6x -n 2500000 -i statistics-2016
$ curl -s "http://localhost:8081/solr/statistics-2016/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>*:* NOT id:/.{36}/</query></delete>"
```
### statistics-2015
## statistics-2015
Processing the statistics-2015 core:
@ -225,6 +244,7 @@ Summary of stats after processing:
$ curl -s "http://localhost:8081/solr/statistics-2015/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>*:* NOT id:/.{36}/</query></delete>"
```
## statistics-2014
Processing the statistics-2014 core:
@ -259,6 +279,7 @@ Summary of unmigrated documents after processing:
$ curl -s "http://localhost:8081/solr/statistics-2014/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>*:* NOT id:/.{36}/</query></delete>"
```
## statistics-2013
Processing the statistics-2013 core:
@ -292,3 +313,105 @@ Summary of unmigrated docs after processing:
```console
$ curl -s "http://localhost:8081/solr/statistics-2013/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>*:* NOT id:/.{36}/</query></delete>"
```
## statistics-2012
Processing the statistics-2012 core:
```console
$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 2500000 -i statistics-2012
...
=================================================================
*** Statistics Records with Legacy Id ***
2,229,332 Item View
913,577 Bistream View
215,577 Collection View
104,734 Community View
--------------------------------------
3,463,220 TOTAL
=================================================================
```
Summary of unmigrated docs after processing:
- 0: `(*:* NOT id:/.{36}/) AND (*:* NOT id:/.+-unmigrated/)`
- 33,161: `id:/.+-unmigrated/`
- 33,161: `*:* NOT id:/.{36}/`
- 33,161 are `type: 3` (COLLECTION), which is different than I've seen previously... but I suppose I still have to purge them because there will be errors in the Atmire modules otherwise:
```console
$ curl -s "http://localhost:8081/solr/statistics-2012/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>*:* NOT id:/.{36}/</query></delete>"
```
## statistics-2011
Processing the statistics-2011 core:
```console
$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 2500000 -i statistics-2011
...
=================================================================
*** Statistics Records with Legacy Id ***
904,896 Item View
385,789 Bistream View
154,356 Collection View
62,978 Community View
--------------------------------------
1,508,019 TOTAL
=================================================================
```
Summary of unmigrated docs after processing:
- 0: `(*:* NOT id:/.{36}/) AND (*:* NOT id:/.+-unmigrated/)`
- 17,551: `id:/.+-unmigrated/`
- 17,551: `*:* NOT id:/.{36}/`
- 12,116 are `type: 3` (COLLECTION), which is different than I've seen previously... but I suppose I still have to purge them because there will be errors in the Atmire modules otherwise:
```console
$ curl -s "http://localhost:8081/solr/statistics-2011/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>*:* NOT id:/.{36}/</query></delete>"
```
## statistics-2010
Processing the statistics-2010 core:
```console
$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 2500000 -i statistics-2010
...
=================================================================
*** Statistics Records with Legacy Id ***
26,067 Item View
15,615 Bistream View
4,116 Collection View
1,094 Community View
--------------------------------------
46,892 TOTAL
=================================================================
```
Summary of unmigrated docs after processing:
- 0: `(*:* NOT id:/.{36}/) AND (*:* NOT id:/.+-unmigrated/)`
- 1,012: `id:/.+-unmigrated/`
- 1,012: `*:* NOT id:/.{36}/`
- 654 are `type: 3` (COLLECTION), which is different than I've seen previously... but I suppose I still have to purge them because there will be errors in the Atmire modules otherwise:
```console
$ curl -s "http://localhost:8081/solr/statistics-2010/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>*:* NOT id:/.{36}/</query></delete>"
```
### Processing Solr statistics with AtomicStatisticsUpdateCLI
On 2020-11-18 I finished processing the Solr statistics with solr-upgrade-statistics-6x and I started processing them with AtomicStatisticsUpdateCLI:
```
$ chrt -b 0 dspace dsrun com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI -t 12 -c statistics
```