alanorth
/
cgspace-notes
mirror of https://github.com/alanorth/cgspace-notes.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
cgspace-notes/content/posts/2023-04.md

33 KiB
Raw Blame History

title date author categories
April, 2023 2023-04-02T08:19:36+03:00 Alan Orth
Notes

2023-04-02

  • Run all system updates on CGSpace and reboot it
  • I exported CGSpace to CSV to check for any missing Initiative collection mappings
    • I also did a check for missing country/region mappings with csv-metadata-quality
  • Start a harvest on AReS
  • I'm starting to get annoyed at my shell script for doing ImageMagick tests and looking to re-write it in something object oriented like Python
    • There doesn't seem to be an official ImageMagick Python binding on pypi.org, perhaps I can use Wand?
  • Testing Wand in Python:
from wand.image import Image

with Image(filename='data/10568-103447.pdf[0]', resolution=144) as first_page:
    print(first_page.height)
  • I spent more time re-working my thumbnail scripts to compare the resized images and other minor changes
    • I am realizing that doing the thumbnails directly from the source improves the ssimulacra2 score by 1-3% points compared to DSpace's method of creating a lossy supersample followed by a lossy resized thumbnail

2023-04-03

  • The harvest on AReS that I started yesterday never finished, and actually seems to have died...
    • Also, Fabio and Patrizio from Alliance emailed me to ask if there is something wrong with the REST API because they are having problems
    • I stopped the harvest and started the plugins to get the remaining items via the sitemap...

2023-04-04

  • Presentation about CGSpace metadata, controlled vocabularies, and curation to Pooja's communications and development team at UNEP
  • Someone from the system organization contacted me to ask how to download a few thousand PDFs from a spreadsheet with DOIs and Handles
$ csvcut -c Handle ~/Downloads/2023-04-04-Donald.csv \
    | sed \
        -e 1d \
        -e 's_https://hdl.handle.net/__' \
        -e 's_https://cgspace.cgiar.org/handle/__' \
        -e 's_http://hdl.handle.net/__' \
    | sort -u > /tmp/handles.txt
  • Then I used the get_dspace_pdfs.py script to download them

2023-04-05

  • After some cleanup on Donald's DOIs I started the get_scihub_pdfs.py script

2023-04-06

  • I did some more work to cleanup and streamline my next generation of DSpace thumbnail testing scripts
    • I think I found a bug in ImageMagick 7.1.1.5 where CMYK to sRGB conversion fails if we use image operations like -density or -define before reading the input file
    • I started a discussion on the ImageMagick GitHub to ask
  • Yesterday I started downloading the rest of the PDFs from Donald, those that had DOIs
    • As a measure of caution, I extracted the list of DOIs and used my crossref_doi_lookup.py script to get their licenses from Crossref:
$ ./ilri/crossref_doi_lookup.py -e xxxx@i.org -i /tmp/dois.txt -o /tmp/donald-crossref-dois.csv -d
  • Then I did some CSV manipulation to extract the DOIs that were Creative Commons licensed, excluding any that were "No Derivatives", and re-formatting the DOIs:
$ csvcut -c doi,license /tmp/donald-crossref-dois.csv \
  | csvgrep -c license -m 'creativecommons' \
  | csvgrep -c license -i -r 'by-(nd|nc-nd)' \
  | sed -e 's_^10_https://doi.org/10_' \
    -e 's/\(am\|tdm\|unspecified\|vor\): //' \
  | tee /tmp/donald-open-dois.csv \
  | wc -l
4268
  • From those I filtered for the DOIs for which I had downloaded PDFs, in the filename column of the Sci-Hub script and copied them to a separate directory:
$ for file in $(csvjoin -c doi /tmp/donald-doi-pdfs.csv /tmp/donald-open-dois.csv | csvgrep -c filename -i -r '^$' | csvcut -c filename | sed 1d); do cp --reflink=always "$file" "creative-commons-licensed/$file"; done
  • I used BTRFS copy-on-write via reflinks to make sure I didn't duplicate the files :-D
  • I ran out of time and had to stop the process around 3,127 PDFs
    • I zipped them up and sent them to the others, along with a CSV of the DOIs, PDF filenames, and licenses

2023-04-17

  • Abenet noticed a weird issue with this item
    • The item has metadata, but the page is blank
    • When I try to edit the item's authorization policies in XMLUI I get a nullPointerException:
Java stacktrace: java.lang.NullPointerException
	at org.dspace.app.xmlui.aspect.administrative.authorization.EditItemPolicies.addBody(EditItemPolicies.java:166)
	at org.dspace.app.xmlui.wing.AbstractWingTransformer.startElement(AbstractWingTransformer.java:234)
	at sun.reflect.GeneratedMethodAccessor347.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy201.startElement(Unknown Source)
	at org.apache.cocoon.components.sax.XMLTeePipe.startElement(XMLTeePipe.java:87)
	at org.apache.cocoon.xml.AbstractXMLPipe.startElement(AbstractXMLPipe.java:94)
	at org.dspace.app.xmlui.wing.AbstractWingTransformer.startElement(AbstractWingTransformer.java:251)
	at sun.reflect.GeneratedMethodAccessor347.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy203.startElement(Unknown Source)
	at org.apache.cocoon.xml.AbstractXMLPipe.startElement(AbstractXMLPipe.java:94)
	at org.dspace.app.xmlui.wing.AbstractWingTransformer.startElement(AbstractWingTransformer.java:251)
	at sun.reflect.GeneratedMethodAccessor347.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy203.startElement(Unknown Source)
	at org.apache.cocoon.environment.internal.EnvironmentChanger.startElement(EnvironmentStack.java:140)
	at org.apache.cocoon.components.sax.XMLTeePipe.startElement(XMLTeePipe.java:87)
	at org.apache.cocoon.xml.AbstractXMLPipe.startElement(AbstractXMLPipe.java:94)
	at org.dspace.app.xmlui.wing.AbstractWingTransformer.startElement(AbstractWingTransformer.java:251)
	at sun.reflect.GeneratedMethodAccessor347.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy203.startElement(Unknown Source)
	at org.apache.cocoon.environment.internal.EnvironmentChanger.startElement(EnvironmentStack.java:140)
	at org.apache.cocoon.components.sax.XMLTeePipe.startElement(XMLTeePipe.java:87)
	at org.apache.cocoon.components.sax.AbstractXMLByteStreamInterpreter.parse(AbstractXMLByteStreamInterpreter.java:117)
	at org.apache.cocoon.components.sax.XMLByteStreamInterpreter.deserialize(XMLByteStreamInterpreter.java:44)
	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processXMLPipeline(AbstractCachingProcessingPipeline.java:324)
	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:750)
	at sun.reflect.GeneratedMethodAccessor438.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy191.process(Unknown Source)
	at org.apache.cocoon.components.source.impl.SitemapSource.toSAX(SitemapSource.java:362)
	at org.apache.cocoon.components.source.util.SourceUtil.toSAX(SourceUtil.java:111)
	at org.apache.cocoon.components.source.util.SourceUtil.parse(SourceUtil.java:294)
	at org.apache.cocoon.generation.FileGenerator.generate(FileGenerator.java:136)
	at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy198.generate(Unknown Source)
	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processXMLPipeline(AbstractCachingProcessingPipeline.java:326)
	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:750)
	at sun.reflect.GeneratedMethodAccessor438.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy191.process(Unknown Source)
	at org.apache.cocoon.components.source.impl.SitemapSource.toSAX(SitemapSource.java:362)
	at org.apache.cocoon.components.source.util.SourceUtil.toSAX(SourceUtil.java:111)
	at org.apache.cocoon.components.source.util.SourceUtil.parse(SourceUtil.java:294)
	at org.apache.cocoon.generation.FileGenerator.generate(FileGenerator.java:136)
	at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy198.generate(Unknown Source)
	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processXMLPipeline(AbstractCachingProcessingPipeline.java:326)
	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:750)
	at sun.reflect.GeneratedMethodAccessor438.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy191.process(Unknown Source)
	at org.apache.cocoon.components.source.impl.SitemapSource.toSAX(SitemapSource.java:362)
	at org.apache.cocoon.components.source.util.SourceUtil.toSAX(SourceUtil.java:111)
	at org.apache.cocoon.components.source.util.SourceUtil.parse(SourceUtil.java:294)
	at org.apache.cocoon.generation.FileGenerator.generate(FileGenerator.java:136)
	at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy198.generate(Unknown Source)
	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.processXMLPipeline(AbstractProcessingPipeline.java:544)
	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processXMLPipeline(AbstractCachingProcessingPipeline.java:273)
	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:750)
	at sun.reflect.GeneratedMethodAccessor438.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy191.process(Unknown Source)
	at org.apache.cocoon.components.source.impl.SitemapSource.toSAX(SitemapSource.java:362)
	at org.apache.cocoon.components.source.util.SourceUtil.toSAX(SourceUtil.java:111)
	at org.apache.cocoon.components.source.util.SourceUtil.parse(SourceUtil.java:294)
	at org.apache.cocoon.generation.FileGenerator.generate(FileGenerator.java:136)
	at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy198.generate(Unknown Source)
	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.processXMLPipeline(AbstractProcessingPipeline.java:544)
	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processXMLPipeline(AbstractCachingProcessingPipeline.java:273)
	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:750)
	at sun.reflect.GeneratedMethodAccessor438.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy191.process(Unknown Source)
	at org.apache.cocoon.components.source.impl.SitemapSource.toSAX(SitemapSource.java:362)
	at org.apache.cocoon.components.source.util.SourceUtil.toSAX(SourceUtil.java:111)
	at org.apache.cocoon.components.source.util.SourceUtil.parse(SourceUtil.java:294)
	at org.apache.cocoon.generation.FileGenerator.generate(FileGenerator.java:136)
	at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy198.generate(Unknown Source)
	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.processXMLPipeline(AbstractProcessingPipeline.java:544)
	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processXMLPipeline(AbstractCachingProcessingPipeline.java:273)
	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:750)
	at sun.reflect.GeneratedMethodAccessor438.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy191.process(Unknown Source)
	at org.apache.cocoon.components.source.impl.SitemapSource.toSAX(SitemapSource.java:362)
	at org.apache.cocoon.components.source.util.SourceUtil.toSAX(SourceUtil.java:111)
	at org.apache.cocoon.components.source.util.SourceUtil.parse(SourceUtil.java:294)
	at org.apache.cocoon.generation.FileGenerator.generate(FileGenerator.java:136)
	at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy198.generate(Unknown Source)
	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.processXMLPipeline(AbstractProcessingPipeline.java:544)
	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processXMLPipeline(AbstractCachingProcessingPipeline.java:273)
	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:750)
	at sun.reflect.GeneratedMethodAccessor438.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy191.process(Unknown Source)
	at org.apache.cocoon.components.source.impl.SitemapSource.toSAX(SitemapSource.java:362)
	at org.apache.cocoon.components.source.util.SourceUtil.toSAX(SourceUtil.java:111)
	at org.apache.cocoon.components.source.util.SourceUtil.parse(SourceUtil.java:294)
	at org.apache.cocoon.generation.FileGenerator.generate(FileGenerator.java:136)
	at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy198.generate(Unknown Source)
	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.processXMLPipeline(AbstractProcessingPipeline.java:544)
	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processXMLPipeline(AbstractCachingProcessingPipeline.java:273)
	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:750)
	at sun.reflect.GeneratedMethodAccessor438.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy191.process(Unknown Source)
	at org.apache.cocoon.components.source.impl.SitemapSource.toSAX(SitemapSource.java:362)
	at org.apache.cocoon.components.source.util.SourceUtil.toSAX(SourceUtil.java:111)
	at org.apache.cocoon.components.source.util.SourceUtil.parse(SourceUtil.java:294)
	at org.apache.cocoon.generation.FileGenerator.generate(FileGenerator.java:136)
	at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy198.generate(Unknown Source)
	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.processXMLPipeline(AbstractProcessingPipeline.java:544)
	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processXMLPipeline(AbstractCachingProcessingPipeline.java:273)
	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:750)
	at sun.reflect.GeneratedMethodAccessor438.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy191.process(Unknown Source)
	at org.apache.cocoon.components.source.impl.SitemapSource.toSAX(SitemapSource.java:362)
	at org.apache.cocoon.components.source.util.SourceUtil.toSAX(SourceUtil.java:111)
	at org.apache.cocoon.components.source.util.SourceUtil.parse(SourceUtil.java:294)
	at org.apache.cocoon.generation.FileGenerator.generate(FileGenerator.java:136)
	at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy198.generate(Unknown Source)
	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.processXMLPipeline(AbstractProcessingPipeline.java:544)
	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processXMLPipeline(AbstractCachingProcessingPipeline.java:273)
	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:750)
	at sun.reflect.GeneratedMethodAccessor438.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy191.process(Unknown Source)
	at org.apache.cocoon.components.source.impl.SitemapSource.toSAX(SitemapSource.java:362)
	at org.apache.cocoon.components.source.util.SourceUtil.toSAX(SourceUtil.java:111)
	at org.apache.cocoon.components.source.util.SourceUtil.parse(SourceUtil.java:294)
	at org.apache.cocoon.generation.FileGenerator.generate(FileGenerator.java:136)
	at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy198.generate(Unknown Source)
	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.processXMLPipeline(AbstractProcessingPipeline.java:544)
	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processXMLPipeline(AbstractCachingProcessingPipeline.java:273)
	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:750)
	at sun.reflect.GeneratedMethodAccessor438.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy191.process(Unknown Source)
	at org.apache.cocoon.components.source.impl.SitemapSource.toSAX(SitemapSource.java:362)
	at org.apache.cocoon.components.source.util.SourceUtil.toSAX(SourceUtil.java:111)
	at org.apache.cocoon.components.source.util.SourceUtil.parse(SourceUtil.java:294)
	at org.apache.cocoon.generation.FileGenerator.generate(FileGenerator.java:136)
	at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy198.generate(Unknown Source)
	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.processXMLPipeline(AbstractProcessingPipeline.java:544)
	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processXMLPipeline(AbstractCachingProcessingPipeline.java:273)
	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:439)
	at sun.reflect.GeneratedMethodAccessor255.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
	at com.sun.proxy.$Proxy191.process(Unknown Source)
	at org.apache.cocoon.components.treeprocessor.sitemap.SerializeNode.invoke(SerializeNode.java:147)
	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:55)
	at org.apache.cocoon.components.treeprocessor.sitemap.MatchNode.invoke(MatchNode.java:87)
	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
	at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:143)
	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
	at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:81)
	at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:239)
	at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:171)
	at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:247)
	at org.apache.cocoon.components.treeprocessor.sitemap.MountNode.invoke(MountNode.java:117)
	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:55)
	at org.apache.cocoon.components.treeprocessor.sitemap.MatchNode.invoke(MatchNode.java:87)
	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
	at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:143)
	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
	at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:81)
	at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:239)
	at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:171)
	at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:247)
	at org.apache.cocoon.components.treeprocessor.sitemap.MountNode.invoke(MountNode.java:117)
	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
	at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:143)
	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
	at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:81)
	at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:239)
	at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:171)
	at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:247)
	at org.apache.cocoon.servlet.RequestProcessor.process(RequestProcessor.java:351)
	at org.apache.cocoon.servlet.RequestProcessor.service(RequestProcessor.java:169)
	at org.apache.cocoon.sitemap.SitemapServlet.service(SitemapServlet.java:84)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
	at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:468)
	at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:443)
	at org.apache.cocoon.servletservice.spring.ServletFactoryBean$ServiceInterceptor.invoke(ServletFactoryBean.java:264)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at com.sun.proxy.$Proxy186.service(Unknown Source)
	at org.dspace.springmvc.CocoonView.render(CocoonView.java:113)
	at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1216)
	at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1001)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:945)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:853)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.dspace.app.xmlui.cocoon.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:113)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.dspace.app.xmlui.cocoon.DSpaceCocoonServletFilter.doFilter(DSpaceCocoonServletFilter.java:160)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.dspace.app.xmlui.cocoon.servlet.multipart.DSpaceMultipartFilter.doFilter(DSpaceMultipartFilter.java:119)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.dspace.utils.servlet.DSpaceWebappServletFilter.doFilter(DSpaceWebappServletFilter.java:78)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:165)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
	at org.apache.catalina.valves.CrawlerSessionManagerValve.invoke(CrawlerSessionManagerValve.java:235)
	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:451)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1201)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:654)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:317)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:750)
  • I don't see anything on the DSpace issue tracker or mailing list so I asked about it on the DSpace Slack...
  • Peter said CGSpace was slow and I see a lot of locks from the XMLUI
    • I looked and found many locks that were many hours and days old so I killed some:
$ psql < locks-age.sql | grep -E "[[:digit:]] days" | awk -F\| '{print $10}' | sort -u
 1050672
 1053773
 1054602
 1054702
 1056782
 1057629
 1057630
$ psql < locks-age.sql | grep -E "[[:digit:]] days" | awk -F\| '{print $10}' | sort -u | xargs kill
  • I'm also running a dspace cleanup -v, but it doesn't seem to be finishing
    • I recall something like there being errors in the logs rather than on the command line in DSpace 6...
    • I found it in the DSpace log:
2023-04-17 21:09:46,004 ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper @ ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
  Detail: Key (uuid)=(a7ddf477-1c04-4de0-9c7a-4d3c84a875bc) is still referenced from table "bundle".
  • If I mark the primary bitstream as null manually the cleanup script continues until it finds a few more
    • I ended up with a long list of UUIDs to fix before the script would complete:
$ psql -d dspace -c "update bundle set primary_bitstream_id=NULL where primary_bitstream_id in ('a7ddf477-1c04-4de0-9c7a-4d3c84a875bc', '9582b661-9c2d-4c86-be22-c3b0942b646a', '210a4d5d-3af9-46f0-84cc-682dd1431762')"

2023-04-18

  • Regarding the item Abenet noticed yesterday that has a blank page and a nullPointerException
    • It appears OK on DSpace Test! https://dspacetest.cgiar.org/handle/10568/75611
    • And according to the REST API on CGSpace the item was modified on 2023-04-11, so last week...
    • According to the DSpace logs it was Francesca who edited the item last week, so I asked her for more information before I troubleshoot more