mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2024-12-22 21:22:19 +01:00
417 lines
19 KiB
HTML
417 lines
19 KiB
HTML
<!DOCTYPE html>
|
||
<html lang="en">
|
||
|
||
<head>
|
||
<meta charset="utf-8">
|
||
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
|
||
|
||
<meta property="og:title" content="September, 2018" />
|
||
<meta property="og:description" content="2018-09-02
|
||
|
||
|
||
New PostgreSQL JDBC driver version 42.2.5
|
||
I’ll update the DSpace role in our Ansible infrastructure playbooks and run the updated playbooks on CGSpace and DSpace Test
|
||
Also, I’ll re-run the postgresql tasks because the custom PostgreSQL variables are dynamic according to the system’s RAM, and we never re-ran them after migrating to larger Linodes last month
|
||
I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I’m getting those autowire errors in Tomcat 8.5.30 again:
|
||
|
||
|
||
" />
|
||
<meta property="og:type" content="article" />
|
||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2018-09/" /><meta property="article:published_time" content="2018-09-02T09:55:54+03:00"/>
|
||
<meta property="article:modified_time" content="2018-09-12T13:49:33+03:00"/>
|
||
<meta name="twitter:card" content="summary"/>
|
||
<meta name="twitter:title" content="September, 2018"/>
|
||
<meta name="twitter:description" content="2018-09-02
|
||
|
||
|
||
New PostgreSQL JDBC driver version 42.2.5
|
||
I’ll update the DSpace role in our Ansible infrastructure playbooks and run the updated playbooks on CGSpace and DSpace Test
|
||
Also, I’ll re-run the postgresql tasks because the custom PostgreSQL variables are dynamic according to the system’s RAM, and we never re-ran them after migrating to larger Linodes last month
|
||
I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I’m getting those autowire errors in Tomcat 8.5.30 again:
|
||
|
||
|
||
"/>
|
||
<meta name="generator" content="Hugo 0.48" />
|
||
|
||
|
||
|
||
<script type="application/ld+json">
|
||
{
|
||
"@context": "http://schema.org",
|
||
"@type": "BlogPosting",
|
||
"headline": "September, 2018",
|
||
"url": "https://alanorth.github.io/cgspace-notes/2018-09/",
|
||
"wordCount": "1338",
|
||
"datePublished": "2018-09-02T09:55:54+03:00",
|
||
"dateModified": "2018-09-12T13:49:33+03:00",
|
||
"author": {
|
||
"@type": "Person",
|
||
"name": "Alan Orth"
|
||
},
|
||
"keywords": "Notes"
|
||
}
|
||
</script>
|
||
|
||
|
||
|
||
<link rel="canonical" href="https://alanorth.github.io/cgspace-notes/2018-09/">
|
||
|
||
<title>September, 2018 | CGSpace Notes</title>
|
||
|
||
<!-- combined, minified CSS -->
|
||
<link href="https://alanorth.github.io/cgspace-notes/css/style.css" rel="stylesheet" integrity="sha384-Upm5uY/SXdvbjuIGH6fBjF5vOYUr9DguqBskM+EQpLBzO9U+9fMVmWEt+TTlGrWQ" crossorigin="anonymous">
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
</head>
|
||
|
||
<body>
|
||
|
||
|
||
<div class="blog-masthead">
|
||
<div class="container">
|
||
<nav class="nav blog-nav">
|
||
<a class="nav-link " href="https://alanorth.github.io/cgspace-notes/">Home</a>
|
||
</nav>
|
||
</div>
|
||
</div>
|
||
|
||
|
||
|
||
<header class="blog-header">
|
||
<div class="container">
|
||
<h1 class="blog-title"><a href="https://alanorth.github.io/cgspace-notes/" rel="home">CGSpace Notes</a></h1>
|
||
<p class="lead blog-description">Documenting day-to-day work on the <a href="https://cgspace.cgiar.org">CGSpace</a> repository.</p>
|
||
</div>
|
||
</header>
|
||
|
||
|
||
|
||
<div class="container">
|
||
<div class="row">
|
||
<div class="col-sm-8 blog-main">
|
||
|
||
|
||
|
||
|
||
<article class="blog-post">
|
||
<header>
|
||
<h2 class="blog-post-title"><a href="https://alanorth.github.io/cgspace-notes/2018-09/">September, 2018</a></h2>
|
||
<p class="blog-post-meta"><time datetime="2018-09-02T09:55:54+03:00">Sun Sep 02, 2018</time> by Alan Orth in
|
||
|
||
<i class="fa fa-tag" aria-hidden="true"></i> <a href="/cgspace-notes/tags/notes" rel="tag">Notes</a>
|
||
|
||
</p>
|
||
</header>
|
||
<h2 id="2018-09-02">2018-09-02</h2>
|
||
|
||
<ul>
|
||
<li>New <a href="https://jdbc.postgresql.org/documentation/changelog.html#version_42.2.5">PostgreSQL JDBC driver version 42.2.5</a></li>
|
||
<li>I’ll update the DSpace role in our <a href="https://github.com/ilri/rmg-ansible-public">Ansible infrastructure playbooks</a> and run the updated playbooks on CGSpace and DSpace Test</li>
|
||
<li>Also, I’ll re-run the <code>postgresql</code> tasks because the custom PostgreSQL variables are dynamic according to the system’s RAM, and we never re-ran them after migrating to larger Linodes last month</li>
|
||
<li>I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I’m getting those autowire errors in Tomcat 8.5.30 again:</li>
|
||
</ul>
|
||
|
||
<p></p>
|
||
|
||
<pre><code>02-Sep-2018 11:18:52.678 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [org.dspace.servicemanager.servlet.DSpaceKernelServletContextListener]
|
||
java.lang.RuntimeException: Failure during filter init: Failed to startup the DSpace Service Manager: failure starting up spring service manager: Error creating bean with name 'conversionService' defined in file [/home/dspacetest.cgiar.org/config/spring/xmlui/spring-dspace-addon-cua-services.xml]: Cannot create inner bean 'com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter#4c5d5a2' of type [com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter] while setting bean property 'converters' with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter#4c5d5a2': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$FilterConverter com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter.filterConverter; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No matching bean of type [com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$FilterConverter] found for dependency: expected at least 1 bean which qualifies as autowire candidate for this dependency. Dependency annotations: {@org.springframework.beans.factory.annotation.Autowired(required=true)}
|
||
at org.dspace.servicemanager.servlet.DSpaceKernelServletContextListener.contextInitialized(DSpaceKernelServletContextListener.java:92)
|
||
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4776)
|
||
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5240)
|
||
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
|
||
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
|
||
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
|
||
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
|
||
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:629)
|
||
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1838)
|
||
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
|
||
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
|
||
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
|
||
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
|
||
at java.lang.Thread.run(Thread.java:748)
|
||
Caused by: java.lang.RuntimeException: Failed to startup the DSpace Service Manager: failure starting up spring service manager: Error creating bean with name 'conversionService' defined in file [/home/dspacetest.cgiar.org/config/spring/xmlui/spring-dspace-addon-cua-services.xml]: Cannot create inner bean 'com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter#4c5d5a2' of type [com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter] while setting bean property 'converters' with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter#4c5d5a2': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$FilterConverter com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter.filterConverter; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No matching bean of type [com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$FilterConverter] found for dependency: expected at least 1 bean which qualifies as autowire candidate for this dependency. Dependency annotations:
|
||
</code></pre>
|
||
|
||
<ul>
|
||
<li>Full log here: <a href="https://gist.github.com/alanorth/1e4ae567b853fea9d9dbf1a030ecd8c2">https://gist.github.com/alanorth/1e4ae567b853fea9d9dbf1a030ecd8c2</a></li>
|
||
<li>XMLUI fails to load, but the REST, SOLR, JSPUI, etc work</li>
|
||
<li>The old <code>5_x-prod-dspace-5.5</code> branch does work in Ubuntu 18.04 with Tomcat 8.5.30-1ubuntu1.4, however!</li>
|
||
<li>And the <code>5_x-prod</code> DSpace 5.8 branch does work in Tomcat 8.5.x on my Arch Linux laptop…</li>
|
||
<li>I’m not sure where the issue is then!</li>
|
||
</ul>
|
||
|
||
<h2 id="2018-09-03">2018-09-03</h2>
|
||
|
||
<ul>
|
||
<li>Abenet says she’s getting three emails about periodic statistics reports every day since the DSpace 5.8 upgrade last week</li>
|
||
<li>They are from the CUA module</li>
|
||
<li>Two of them have “no data” and one has a “null” title</li>
|
||
<li>The last one is a report of the top downloaded items, and includes a graph</li>
|
||
<li>She will try to click the “Unsubscribe” link in the first two to see if it works, otherwise we should contact Atmire</li>
|
||
<li>The only one she remembers subscribing to is the top downloads one</li>
|
||
</ul>
|
||
|
||
<h2 id="2018-09-04">2018-09-04</h2>
|
||
|
||
<ul>
|
||
<li>I’m looking over the latest round of IITA records from Sisay: <a href="https://dspacetest.cgiar.org/handle/10568/104230">Mercy1806_August_29</a>
|
||
|
||
<ul>
|
||
<li>All fields are split with multiple columns like <code>cg.authorship.types</code> and <code>cg.authorship.types[]</code></li>
|
||
<li>This makes it super annoying to do the checks and cleanup, so I will merge them (also time consuming)</li>
|
||
<li>Five items had <code>dc.date.issued</code> values like <code>2013-5</code> so I corrected them to be <code>2013-05</code></li>
|
||
<li>Several metadata fields had values with newlines in them (even in some titles!), which I fixed by trimming the consecutive whitespaces in Open Refine</li>
|
||
<li>Many (91!) items from before 2011 are indicated as having a CRP, but CRPs didn’t exist then so this is impossible</li>
|
||
<li>I got all items that were from 2011 and onwards using a custom facet with this GREL on the <code>dc.date.issued</code> column: <code>isNotNull(value.match(/201[1-8].*/))</code> and then blanking their CRPs</li>
|
||
<li>Some affiliations with only one separator (|) for multiple values</li>
|
||
<li>I replaced smart quotes like <code>’</code> with plain ones</li>
|
||
<li>Some inconsistencies in <code>cg.subject.iita</code> like COWPEA and COWPEAS, and YAM and YAMS, etc, as well as some spelling mistakes like IMPACT ASSESSMENTN</li>
|
||
<li>Some values in the <code>dc.identifier.isbn</code> are actually ISSNs so I moved them to the <code>dc.identifier.issn</code> column</li>
|
||
<li>I found one invalid ISSN using a custom text facet with the regex from the <a href="https://en.wikipedia.org/wiki/International_Standard_Serial_Number#Code_format">ISSN page on Wikipedia</a>: <code>isNotBlank(value.match(/^\d{4}-\d{3}[\dxX]$/))</code></li>
|
||
<li>One invalid value for <code>dc.type</code></li>
|
||
</ul></li>
|
||
<li>Abenet says she hasn’t received any more subscription emails from the CUA module since she unsubscribed yesterday, so I think we don’t need create an issue on Atmire’s bug tracker anymore</li>
|
||
</ul>
|
||
|
||
<h2 id="2018-09-10">2018-09-10</h2>
|
||
|
||
<ul>
|
||
<li>Playing with <a href="https://github.com/eykhagen/strest">strest</a> to test the DSpace REST API programatically</li>
|
||
<li>For example, given this <code>test.yaml</code>:</li>
|
||
</ul>
|
||
|
||
<pre><code>version: 1
|
||
|
||
requests:
|
||
test:
|
||
method: GET
|
||
url: https://dspacetest.cgiar.org/rest/test
|
||
validate:
|
||
raw: "REST api is running."
|
||
|
||
login:
|
||
url: https://dspacetest.cgiar.org/rest/login
|
||
method: POST
|
||
data:
|
||
json: {"email":"test@dspace","password":"thepass"}
|
||
|
||
status:
|
||
url: https://dspacetest.cgiar.org/rest/status
|
||
method: GET
|
||
headers:
|
||
rest-dspace-token: Value(login)
|
||
|
||
logout:
|
||
url: https://dspacetest.cgiar.org/rest/logout
|
||
method: POST
|
||
headers:
|
||
rest-dspace-token: Value(login)
|
||
|
||
# vim: set sw=2 ts=2:
|
||
</code></pre>
|
||
|
||
<ul>
|
||
<li>Works pretty well, though the DSpace <code>logout</code> always returns an HTTP 415 error for some reason</li>
|
||
<li>We could eventually use this to test sanity of the API for creating collections etc</li>
|
||
<li>A user is getting an error in her workflow:</li>
|
||
</ul>
|
||
|
||
<pre><code>2018-09-10 07:26:35,551 ERROR org.dspace.submit.step.CompleteStep @ Caught exception in submission step:
|
||
org.dspace.authorize.AuthorizeException: Authorization denied for action WORKFLOW_STEP_1 on COLLECTION:2 by user 3819
|
||
</code></pre>
|
||
|
||
<ul>
|
||
<li>Seems to be during submit step, because it’s workflow step 1…?</li>
|
||
<li>Move some top-level CRP communities to be below the new <a href="https://cgspace.cgiar.org/handle/10568/97114">CGIAR Research Programs and Platforms</a> community:</li>
|
||
</ul>
|
||
|
||
<pre><code>$ dspace community-filiator --set -p 10568/97114 -c 10568/51670
|
||
$ dspace community-filiator --set -p 10568/97114 -c 10568/35409
|
||
$ dspace community-filiator --set -p 10568/97114 -c 10568/3112
|
||
</code></pre>
|
||
|
||
<ul>
|
||
<li>Valerio contacted me to point out some issues with metadata on CGSpace, which I corrected in PostgreSQL:</li>
|
||
</ul>
|
||
|
||
<pre><code>update metadatavalue set text_value='ISI Journal' where resource_type_id=2 and metadata_field_id=226 and text_value='ISI Juornal';
|
||
UPDATE 1
|
||
update metadatavalue set text_value='ISI Journal' where resource_type_id=2 and metadata_field_id=226 and text_value='ISI journal';
|
||
UPDATE 23
|
||
update metadatavalue set text_value='ISI Journal' where resource_type_id=2 and metadata_field_id=226 and text_value='YES';
|
||
UPDATE 1
|
||
delete from metadatavalue where resource_type_id=2 and metadata_field_id=226 and text_value='NO';
|
||
DELETE 17
|
||
update metadatavalue set text_value='ISI Journal' where resource_type_id=2 and metadata_field_id=226 and text_value='ISI';
|
||
UPDATE 15
|
||
</code></pre>
|
||
|
||
<ul>
|
||
<li>Start working on adding metadata for access and usage rights that we started earlier in 2018 (and also in 2017)</li>
|
||
<li>The current <code>cg.identifier.status</code> field will become “Access rights” and <code>dc.rights</code> will become “Usage rights”</li>
|
||
<li>I have some work in progress on the <a href="https://github.com/alanorth/DSpace/tree/5_x-rights"><code>5_x-rights</code> branch</a></li>
|
||
<li>Linode said that CGSpace (linode18) had a high CPU load earlier today</li>
|
||
<li>When I looked, I see it’s the same Russian IP that I noticed last month:</li>
|
||
</ul>
|
||
|
||
<pre><code># zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "10/Sep/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
|
||
1459 157.55.39.202
|
||
1579 95.108.181.88
|
||
1615 157.55.39.147
|
||
1714 66.249.64.91
|
||
1924 50.116.102.77
|
||
3696 157.55.39.106
|
||
3763 157.55.39.148
|
||
4470 70.32.83.92
|
||
4724 35.237.175.180
|
||
14132 5.9.6.51
|
||
</code></pre>
|
||
|
||
<ul>
|
||
<li>And this bot is still creating more Tomcat sessions than Nginx requests (WTF?):</li>
|
||
</ul>
|
||
|
||
<pre><code># grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=5.9.6.51' dspace.log.2018-09-10
|
||
14133
|
||
</code></pre>
|
||
|
||
<ul>
|
||
<li>The user agent is still the same:</li>
|
||
</ul>
|
||
|
||
<pre><code>Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)
|
||
</code></pre>
|
||
|
||
<ul>
|
||
<li>I added <code>.*crawl.*</code> to the Tomcat Session Crawler Manager Valve, so I’m not sure why the bot is creating so many sessions…</li>
|
||
<li>I just tested that user agent on CGSpace and it <em>does not</em> create a new session:</li>
|
||
</ul>
|
||
|
||
<pre><code>$ http --print Hh https://cgspace.cgiar.org 'User-Agent:Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)'
|
||
GET / HTTP/1.1
|
||
Accept: */*
|
||
Accept-Encoding: gzip, deflate
|
||
Connection: keep-alive
|
||
Host: cgspace.cgiar.org
|
||
User-Agent: Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)
|
||
|
||
HTTP/1.1 200 OK
|
||
Connection: keep-alive
|
||
Content-Encoding: gzip
|
||
Content-Language: en-US
|
||
Content-Type: text/html;charset=utf-8
|
||
Date: Mon, 10 Sep 2018 20:43:04 GMT
|
||
Server: nginx
|
||
Strict-Transport-Security: max-age=15768000
|
||
Transfer-Encoding: chunked
|
||
Vary: Accept-Encoding
|
||
X-Cocoon-Version: 2.2.0
|
||
X-Content-Type-Options: nosniff
|
||
X-Frame-Options: SAMEORIGIN
|
||
X-XSS-Protection: 1; mode=block
|
||
</code></pre>
|
||
|
||
<ul>
|
||
<li>I will have to keep an eye on it and perhaps add it to the list of “bad bots” that get rate limited</li>
|
||
</ul>
|
||
|
||
<h2 id="2018-09-12">2018-09-12</h2>
|
||
|
||
<ul>
|
||
<li>Merge AReS explorer changes to nginx config and deploy on CGSpace so CodeObia can start testing more</li>
|
||
<li>Re-create my local Docker container for PostgreSQL data, but using a volume for the database data:</li>
|
||
</ul>
|
||
|
||
<pre><code>$ sudo docker volume create --name dspacetest_data
|
||
$ sudo docker run --name dspacedb -v dspacetest_data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:9.6-alpine
|
||
</code></pre>
|
||
|
||
<ul>
|
||
<li>Sisay is still having problems with the controlled vocabulary for top authors</li>
|
||
<li>I took a look at the submission template and Firefox complains that the XML file is missing a root element</li>
|
||
<li>I guess it’s because Firefox is receiving an empty XML file</li>
|
||
<li>I told Sisay to run the XML file through tidy</li>
|
||
<li>More testing of the access and usage rights changes</li>
|
||
</ul>
|
||
|
||
<!-- vim: set sw=2 ts=2: -->
|
||
|
||
|
||
|
||
|
||
|
||
</article>
|
||
|
||
|
||
|
||
</div> <!-- /.blog-main -->
|
||
|
||
<aside class="col-sm-3 ml-auto blog-sidebar">
|
||
|
||
|
||
|
||
<section class="sidebar-module">
|
||
<h4>Recent Posts</h4>
|
||
<ol class="list-unstyled">
|
||
|
||
|
||
<li><a href="/cgspace-notes/2018-09/">September, 2018</a></li>
|
||
|
||
<li><a href="/cgspace-notes/2018-08/">August, 2018</a></li>
|
||
|
||
<li><a href="/cgspace-notes/2018-07/">July, 2018</a></li>
|
||
|
||
<li><a href="/cgspace-notes/2018-06/">June, 2018</a></li>
|
||
|
||
<li><a href="/cgspace-notes/2018-05/">May, 2018</a></li>
|
||
|
||
</ol>
|
||
</section>
|
||
|
||
|
||
|
||
|
||
<section class="sidebar-module">
|
||
<h4>Links</h4>
|
||
<ol class="list-unstyled">
|
||
|
||
<li><a href="https://cgspace.cgiar.org">CGSpace</a></li>
|
||
|
||
<li><a href="https://dspacetest.cgiar.org">DSpace Test</a></li>
|
||
|
||
<li><a href="https://github.com/ilri/DSpace">CGSpace @ GitHub</a></li>
|
||
|
||
</ol>
|
||
</section>
|
||
|
||
</aside>
|
||
|
||
|
||
</div> <!-- /.row -->
|
||
</div> <!-- /.container -->
|
||
|
||
|
||
|
||
<footer class="blog-footer">
|
||
<p>
|
||
|
||
Blog template created by <a href="https://twitter.com/mdo">@mdo</a>, ported to Hugo by <a href='https://twitter.com/mralanorth'>@mralanorth</a>.
|
||
|
||
</p>
|
||
<p>
|
||
<a href="#">Back to top</a>
|
||
</p>
|
||
</footer>
|
||
|
||
|
||
</body>
|
||
|
||
</html>
|