mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2025-01-27 05:49:12 +01:00
Add notes for 2020-01-27
This commit is contained in:
@ -9,9 +9,9 @@
|
||||
<meta property="og:description" content="2018-09-02
|
||||
|
||||
New PostgreSQL JDBC driver version 42.2.5
|
||||
I'll update the DSpace role in our Ansible infrastructure playbooks and run the updated playbooks on CGSpace and DSpace Test
|
||||
Also, I'll re-run the postgresql tasks because the custom PostgreSQL variables are dynamic according to the system's RAM, and we never re-ran them after migrating to larger Linodes last month
|
||||
I'm testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I'm getting those autowire errors in Tomcat 8.5.30 again:
|
||||
I’ll update the DSpace role in our Ansible infrastructure playbooks and run the updated playbooks on CGSpace and DSpace Test
|
||||
Also, I’ll re-run the postgresql tasks because the custom PostgreSQL variables are dynamic according to the system’s RAM, and we never re-ran them after migrating to larger Linodes last month
|
||||
I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I’m getting those autowire errors in Tomcat 8.5.30 again:
|
||||
" />
|
||||
<meta property="og:type" content="article" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2018-09/" />
|
||||
@ -23,11 +23,11 @@ I'm testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I&#
|
||||
<meta name="twitter:description" content="2018-09-02
|
||||
|
||||
New PostgreSQL JDBC driver version 42.2.5
|
||||
I'll update the DSpace role in our Ansible infrastructure playbooks and run the updated playbooks on CGSpace and DSpace Test
|
||||
Also, I'll re-run the postgresql tasks because the custom PostgreSQL variables are dynamic according to the system's RAM, and we never re-ran them after migrating to larger Linodes last month
|
||||
I'm testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I'm getting those autowire errors in Tomcat 8.5.30 again:
|
||||
I’ll update the DSpace role in our Ansible infrastructure playbooks and run the updated playbooks on CGSpace and DSpace Test
|
||||
Also, I’ll re-run the postgresql tasks because the custom PostgreSQL variables are dynamic according to the system’s RAM, and we never re-ran them after migrating to larger Linodes last month
|
||||
I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I’m getting those autowire errors in Tomcat 8.5.30 again:
|
||||
"/>
|
||||
<meta name="generator" content="Hugo 0.62.2" />
|
||||
<meta name="generator" content="Hugo 0.63.1" />
|
||||
|
||||
|
||||
|
||||
@ -57,7 +57,7 @@ I'm testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I&#
|
||||
|
||||
<!-- combined, minified CSS -->
|
||||
|
||||
<link href="https://alanorth.github.io/cgspace-notes/css/style.a20c1a4367639632cdb341d23c27ca44fedcc75b0f8b3cbea6203010da153d3c.css" rel="stylesheet" integrity="sha256-ogwaQ2djljLNs0HSPCfKRP7cx1sPizy+piAwENoVPTw=" crossorigin="anonymous">
|
||||
<link href="https://alanorth.github.io/cgspace-notes/css/style.23e2c3298bcc8c1136c19aba330c211ec94c36f7c4454ea15cf4d3548370042a.css" rel="stylesheet" integrity="sha256-I+LDKYvMjBE2wZq6MwwhHslMNvfERU6hXPTTVINwBCo=" crossorigin="anonymous">
|
||||
|
||||
|
||||
<!-- RSS 2.0 feed -->
|
||||
@ -104,7 +104,7 @@ I'm testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I&#
|
||||
<header>
|
||||
<h2 class="blog-post-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/2018-09/">September, 2018</a></h2>
|
||||
<p class="blog-post-meta"><time datetime="2018-09-02T09:55:54+03:00">Sun Sep 02, 2018</time> by Alan Orth in
|
||||
<i class="fa fa-folder" aria-hidden="true"></i> <a href="/cgspace-notes/categories/notes" rel="category tag">Notes</a>
|
||||
<span class="fas fa-folder" aria-hidden="true"></span> <a href="/cgspace-notes/categories/notes" rel="category tag">Notes</a>
|
||||
|
||||
|
||||
</p>
|
||||
@ -112,9 +112,9 @@ I'm testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I&#
|
||||
<h2 id="2018-09-02">2018-09-02</h2>
|
||||
<ul>
|
||||
<li>New <a href="https://jdbc.postgresql.org/documentation/changelog.html#version_42.2.5">PostgreSQL JDBC driver version 42.2.5</a></li>
|
||||
<li>I'll update the DSpace role in our <a href="https://github.com/ilri/rmg-ansible-public">Ansible infrastructure playbooks</a> and run the updated playbooks on CGSpace and DSpace Test</li>
|
||||
<li>Also, I'll re-run the <code>postgresql</code> tasks because the custom PostgreSQL variables are dynamic according to the system's RAM, and we never re-ran them after migrating to larger Linodes last month</li>
|
||||
<li>I'm testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I'm getting those autowire errors in Tomcat 8.5.30 again:</li>
|
||||
<li>I’ll update the DSpace role in our <a href="https://github.com/ilri/rmg-ansible-public">Ansible infrastructure playbooks</a> and run the updated playbooks on CGSpace and DSpace Test</li>
|
||||
<li>Also, I’ll re-run the <code>postgresql</code> tasks because the custom PostgreSQL variables are dynamic according to the system’s RAM, and we never re-ran them after migrating to larger Linodes last month</li>
|
||||
<li>I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I’m getting those autowire errors in Tomcat 8.5.30 again:</li>
|
||||
</ul>
|
||||
<pre><code>02-Sep-2018 11:18:52.678 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [org.dspace.servicemanager.servlet.DSpaceKernelServletContextListener]
|
||||
java.lang.RuntimeException: Failure during filter init: Failed to startup the DSpace Service Manager: failure starting up spring service manager: Error creating bean with name 'conversionService' defined in file [/home/dspacetest.cgiar.org/config/spring/xmlui/spring-dspace-addon-cua-services.xml]: Cannot create inner bean 'com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter#4c5d5a2' of type [com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter] while setting bean property 'converters' with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter#4c5d5a2': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$FilterConverter com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter.filterConverter; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No matching bean of type [com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$FilterConverter] found for dependency: expected at least 1 bean which qualifies as autowire candidate for this dependency. Dependency annotations: {@org.springframework.beans.factory.annotation.Autowired(required=true)}
|
||||
@ -138,11 +138,11 @@ Caused by: java.lang.RuntimeException: Failed to startup the DSpace Service Mana
|
||||
<li>XMLUI fails to load, but the REST, SOLR, JSPUI, etc work</li>
|
||||
<li>The old <code>5_x-prod-dspace-5.5</code> branch does work in Ubuntu 18.04 with Tomcat 8.5.30-1ubuntu1.4, however!</li>
|
||||
<li>And the <code>5_x-prod</code> DSpace 5.8 branch does work in Tomcat 8.5.x on my Arch Linux laptop…</li>
|
||||
<li>I'm not sure where the issue is then!</li>
|
||||
<li>I’m not sure where the issue is then!</li>
|
||||
</ul>
|
||||
<h2 id="2018-09-03">2018-09-03</h2>
|
||||
<ul>
|
||||
<li>Abenet says she's getting three emails about periodic statistics reports every day since the DSpace 5.8 upgrade last week</li>
|
||||
<li>Abenet says she’s getting three emails about periodic statistics reports every day since the DSpace 5.8 upgrade last week</li>
|
||||
<li>They are from the CUA module</li>
|
||||
<li>Two of them have “no data” and one has a “null” title</li>
|
||||
<li>The last one is a report of the top downloaded items, and includes a graph</li>
|
||||
@ -151,13 +151,13 @@ Caused by: java.lang.RuntimeException: Failed to startup the DSpace Service Mana
|
||||
</ul>
|
||||
<h2 id="2018-09-04">2018-09-04</h2>
|
||||
<ul>
|
||||
<li>I'm looking over the latest round of IITA records from Sisay: <a href="https://dspacetest.cgiar.org/handle/10568/104230">Mercy1806_August_29</a>
|
||||
<li>I’m looking over the latest round of IITA records from Sisay: <a href="https://dspacetest.cgiar.org/handle/10568/104230">Mercy1806_August_29</a>
|
||||
<ul>
|
||||
<li>All fields are split with multiple columns like <code>cg.authorship.types</code> and <code>cg.authorship.types[]</code></li>
|
||||
<li>This makes it super annoying to do the checks and cleanup, so I will merge them (also time consuming)</li>
|
||||
<li>Five items had <code>dc.date.issued</code> values like <code>2013-5</code> so I corrected them to be <code>2013-05</code></li>
|
||||
<li>Several metadata fields had values with newlines in them (even in some titles!), which I fixed by trimming the consecutive whitespaces in Open Refine</li>
|
||||
<li>Many (91!) items from before 2011 are indicated as having a CRP, but CRPs didn't exist then so this is impossible
|
||||
<li>Many (91!) items from before 2011 are indicated as having a CRP, but CRPs didn’t exist then so this is impossible
|
||||
<ul>
|
||||
<li>I got all items that were from 2011 and onwards using a custom facet with this GREL on the <code>dc.date.issued</code> column: <code>isNotNull(value.match(/201[1-8].*/))</code> and then blanking their CRPs</li>
|
||||
</ul>
|
||||
@ -170,7 +170,7 @@ Caused by: java.lang.RuntimeException: Failed to startup the DSpace Service Mana
|
||||
<li>One invalid value for <code>dc.type</code></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>Abenet says she hasn't received any more subscription emails from the CUA module since she unsubscribed yesterday, so I think we don't need create an issue on Atmire's bug tracker anymore</li>
|
||||
<li>Abenet says she hasn’t received any more subscription emails from the CUA module since she unsubscribed yesterday, so I think we don’t need create an issue on Atmire’s bug tracker anymore</li>
|
||||
</ul>
|
||||
<h2 id="2018-09-10">2018-09-10</h2>
|
||||
<ul>
|
||||
@ -213,7 +213,7 @@ requests:
|
||||
<pre><code>2018-09-10 07:26:35,551 ERROR org.dspace.submit.step.CompleteStep @ Caught exception in submission step:
|
||||
org.dspace.authorize.AuthorizeException: Authorization denied for action WORKFLOW_STEP_1 on COLLECTION:2 by user 3819
|
||||
</code></pre><ul>
|
||||
<li>Seems to be during submit step, because it's workflow step 1…?</li>
|
||||
<li>Seems to be during submit step, because it’s workflow step 1…?</li>
|
||||
<li>Move some top-level CRP communities to be below the new <a href="https://cgspace.cgiar.org/handle/10568/97114">CGIAR Research Programs and Platforms</a> community:</li>
|
||||
</ul>
|
||||
<pre><code>$ dspace community-filiator --set -p 10568/97114 -c 10568/51670
|
||||
@ -237,7 +237,7 @@ UPDATE 15
|
||||
<li>The current <code>cg.identifier.status</code> field will become “Access rights” and <code>dc.rights</code> will become “Usage rights”</li>
|
||||
<li>I have some work in progress on the <a href="https://github.com/alanorth/DSpace/tree/5_x-rights"><code>5_x-rights</code> branch</a></li>
|
||||
<li>Linode said that CGSpace (linode18) had a high CPU load earlier today</li>
|
||||
<li>When I looked, I see it's the same Russian IP that I noticed last month:</li>
|
||||
<li>When I looked, I see it’s the same Russian IP that I noticed last month:</li>
|
||||
</ul>
|
||||
<pre><code># zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "10/Sep/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
|
||||
1459 157.55.39.202
|
||||
@ -260,7 +260,7 @@ UPDATE 15
|
||||
</ul>
|
||||
<pre><code>Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)
|
||||
</code></pre><ul>
|
||||
<li>I added <code>.*crawl.*</code> to the Tomcat Session Crawler Manager Valve, so I'm not sure why the bot is creating so many sessions…</li>
|
||||
<li>I added <code>.*crawl.*</code> to the Tomcat Session Crawler Manager Valve, so I’m not sure why the bot is creating so many sessions…</li>
|
||||
<li>I just tested that user agent on CGSpace and it <em>does not</em> create a new session:</li>
|
||||
</ul>
|
||||
<pre><code>$ http --print Hh https://cgspace.cgiar.org 'User-Agent:Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)'
|
||||
@ -298,17 +298,17 @@ $ sudo docker run --name dspacedb -v dspacetest_data:/var/lib/postgresql/data -e
|
||||
</code></pre><ul>
|
||||
<li>Sisay is still having problems with the controlled vocabulary for top authors</li>
|
||||
<li>I took a look at the submission template and Firefox complains that the XML file is missing a root element</li>
|
||||
<li>I guess it's because Firefox is receiving an empty XML file</li>
|
||||
<li>I guess it’s because Firefox is receiving an empty XML file</li>
|
||||
<li>I told Sisay to run the XML file through tidy</li>
|
||||
<li>More testing of the access and usage rights changes</li>
|
||||
</ul>
|
||||
<h2 id="2018-09-13">2018-09-13</h2>
|
||||
<ul>
|
||||
<li>Peter was communicating with Altmetric about the OAI mapping issue for item <a href="https://cgspace.cgiar.org/oai/request?verb=GetRecord&metadataPrefix=oai_dc&identifier=oai:cgspace.cgiar.org:10568/82810">10568/82810</a> again</li>
|
||||
<li>Altmetric said it was somehow related to the OAI <code>dateStamp</code> not getting updated when the mappings changed, but I said that back in <a href="/cgspace-notes/2018-07/">2018-07</a> when this happened it was because the OAI was actually just not reflecting all the item's mappings</li>
|
||||
<li>Altmetric said it was somehow related to the OAI <code>dateStamp</code> not getting updated when the mappings changed, but I said that back in <a href="/cgspace-notes/2018-07/">2018-07</a> when this happened it was because the OAI was actually just not reflecting all the item’s mappings</li>
|
||||
<li>After forcing a complete re-indexing of OAI the mappings were fine</li>
|
||||
<li>The <code>dateStamp</code> is most probably only updated when the item's metadata changes, not its mappings, so if Altmetric is relying on that we're in a tricky spot</li>
|
||||
<li>We need to make sure that our OAI isn't publicizing stale data… I was going to post something on the dspace-tech mailing list, but never did</li>
|
||||
<li>The <code>dateStamp</code> is most probably only updated when the item’s metadata changes, not its mappings, so if Altmetric is relying on that we’re in a tricky spot</li>
|
||||
<li>We need to make sure that our OAI isn’t publicizing stale data… I was going to post something on the dspace-tech mailing list, but never did</li>
|
||||
<li>Linode says that CGSpace (linode18) has had high CPU for the past two hours</li>
|
||||
<li>The top IP addresses today are:</li>
|
||||
</ul>
|
||||
@ -331,8 +331,8 @@ $ sudo docker run --name dspacedb -v dspacetest_data:/var/lib/postgresql/data -e
|
||||
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=50.116.102.77' dspace.log.2018-09-13 | sort | uniq
|
||||
2
|
||||
</code></pre><ul>
|
||||
<li>So I'm not sure what's going on</li>
|
||||
<li>Valerio asked me if there's a way to get the page views and downloads from CGSpace</li>
|
||||
<li>So I’m not sure what’s going on</li>
|
||||
<li>Valerio asked me if there’s a way to get the page views and downloads from CGSpace</li>
|
||||
<li>I said no, but that we might be able to piggyback on the Atmire statlet REST API</li>
|
||||
<li>For example, when you expand the “statlet” at the bottom of an item like <a href="https://cgspace.cgiar.org/handle/10568/97103">10568/97103</a> you can see the following request in the browser console:</li>
|
||||
</ul>
|
||||
@ -340,12 +340,12 @@ $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=50.116.102.77' dspace.log.2018-09-
|
||||
</code></pre><ul>
|
||||
<li>That JSON file has the total page views and item downloads for the item…</li>
|
||||
<li>Abenet forwarded a request by CIP that item thumbnails be included in RSS feeds</li>
|
||||
<li>I had a quick look at the DSpace 5.x manual and it doesn't not seem that this is possible (you can only add metadata)</li>
|
||||
<li>Testing the new LDAP server the CGNET says will be replacing the old one, it doesn't seem that they are using the global catalog on port 3269 anymore, now only 636 is open</li>
|
||||
<li>I had a quick look at the DSpace 5.x manual and it doesn’t not seem that this is possible (you can only add metadata)</li>
|
||||
<li>Testing the new LDAP server the CGNET says will be replacing the old one, it doesn’t seem that they are using the global catalog on port 3269 anymore, now only 636 is open</li>
|
||||
<li>I did a clean deploy of DSpace 5.8 on Ubuntu 18.04 with some stripped down Tomcat 8 configuration and actually managed to get it up and running without the autowire errors that I had previously experienced</li>
|
||||
<li>I realized that it always works on my local machine with Tomcat 8.5.x, but not when I do the deployment from Ansible in Ubuntu 18.04</li>
|
||||
<li>So there must be something in my Tomcat 8 <code>server.xml</code> template</li>
|
||||
<li>Now I re-deployed it with the normal server template and it's working, WTF?</li>
|
||||
<li>Now I re-deployed it with the normal server template and it’s working, WTF?</li>
|
||||
<li>Must have been something like an old DSpace 5.5 file in the spring folder… weird</li>
|
||||
<li>But yay, this means we can update DSpace Test to Ubuntu 18.04, Tomcat 8, PostgreSQL 9.6, etc…</li>
|
||||
</ul>
|
||||
@ -357,7 +357,7 @@ $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=50.116.102.77' dspace.log.2018-09-
|
||||
<h2 id="2018-09-16">2018-09-16</h2>
|
||||
<ul>
|
||||
<li>Add the DSpace build.properties as a template into my <a href="https://github.com/ilri/rmg-ansible-public">Ansible infrastructure scripts</a> for configuring DSpace machines</li>
|
||||
<li>One stupid thing there is that I add all the variables in a private vars file, which is apparently higher precedence than host vars, meaning that I can't override them (like SMTP server) on a per-host basis</li>
|
||||
<li>One stupid thing there is that I add all the variables in a private vars file, which is apparently higher precedence than host vars, meaning that I can’t override them (like SMTP server) on a per-host basis</li>
|
||||
<li>Discuss access and usage rights with Peter</li>
|
||||
<li>I suggested that we leave access rights (<code>cg.identifier.access</code>) as it is now, with “Open Access” or “Limited Access”, and then simply re-brand that as “Access rights” in the UIs and relevant drop downs</li>
|
||||
<li>Then we continue as planned to add <code>dc.rights</code> as “Usage rights”</li>
|
||||
@ -374,26 +374,26 @@ $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=50.116.102.77' dspace.log.2018-09-
|
||||
</li>
|
||||
<li>Update these immediately, but talk to CodeObia to create a mapping between the old and new values</li>
|
||||
<li>Finalize <code>dc.rights</code> “Usage rights” with seven combinations of Creative Commons, plus the others</li>
|
||||
<li>Need to double check the new <a href="https://cgspace.cgiar.org/handle/10568/97114">CRP community</a> to see why the collection counts aren't updated after we moved the communities there last week
|
||||
<li>Need to double check the new <a href="https://cgspace.cgiar.org/handle/10568/97114">CRP community</a> to see why the collection counts aren’t updated after we moved the communities there last week
|
||||
<ul>
|
||||
<li>I forced a full Discovery re-index and now the community shows 1,600 items</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>Check if it's possible to have items deposited via REST use a workflow so we can perhaps tell ICARDA to use that from MEL</li>
|
||||
<li>Agree that we'll publicize AReS explorer on the week before the Big Data Platform workshop
|
||||
<li>Check if it’s possible to have items deposited via REST use a workflow so we can perhaps tell ICARDA to use that from MEL</li>
|
||||
<li>Agree that we’ll publicize AReS explorer on the week before the Big Data Platform workshop
|
||||
<ul>
|
||||
<li>Put a link and or picture on the CGSpace homepage saying “Visualized CGSpace research” or something, and post a message on Yammer</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>I want to explore creating a thin API to make the item view and download stats available from Solr so CodeObia can use them in the AReS explorer</li>
|
||||
<li>Currently CodeObia is exploring using the Atmire statlets internal API, but I don't really like that…</li>
|
||||
<li>Currently CodeObia is exploring using the Atmire statlets internal API, but I don’t really like that…</li>
|
||||
<li>There are some example queries on the <a href="https://wiki.duraspace.org/display/DSPACE/Solr">DSpace Solr wiki</a></li>
|
||||
<li>For example, this query returns 1655 rows for item <a href="https://cgspace.cgiar.org/handle/10568/10630">10568/10630</a>:</li>
|
||||
</ul>
|
||||
<pre><code>$ http 'http://localhost:3000/solr/statistics/select?indent=on&rows=0&q=type:0+owningItem:11576&fq=isBot:false'
|
||||
</code></pre><ul>
|
||||
<li>The id in the Solr query is the item's database id (get it from the REST API or something)</li>
|
||||
<li>Next, I adopted a query to get the downloads and it shows 889, which is similar to the number Atmire's statlet shows, though the query logic here is confusing:</li>
|
||||
<li>The id in the Solr query is the item’s database id (get it from the REST API or something)</li>
|
||||
<li>Next, I adopted a query to get the downloads and it shows 889, which is similar to the number Atmire’s statlet shows, though the query logic here is confusing:</li>
|
||||
</ul>
|
||||
<pre><code>$ http 'http://localhost:3000/solr/statistics/select?indent=on&rows=0&q=type:0+owningItem:11576&fq=isBot:false&fq=-(bundleName:[*+TO+*]-bundleName:ORIGINAL)&fq=-(statistics_type:[*+TO+*]+-statistics_type:view)'
|
||||
</code></pre><ul>
|
||||
@ -404,7 +404,7 @@ $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=50.116.102.77' dspace.log.2018-09-
|
||||
<li><code>-(bundleName:[*+TO+*]-bundleName:ORIGINAL)</code> seems to be a <a href="https://wiki.apache.org/solr/NegativeQueryProblems">negative query starting with all documents</a>, subtracting those with <code>bundleName:ORIGINAL</code>, and then negating the whole thing… meaning only documents from <code>bundleName:ORIGINAL</code>?</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>What the shit, I think I'm right: the simplified logic in <em>this</em> query returns the same 889:</li>
|
||||
<li>What the shit, I think I’m right: the simplified logic in <em>this</em> query returns the same 889:</li>
|
||||
</ul>
|
||||
<pre><code>$ http 'http://localhost:3000/solr/statistics/select?indent=on&rows=0&q=type:0+owningItem:11576&fq=isBot:false&fq=bundleName:ORIGINAL&fq=-(statistics_type:[*+TO+*]+-statistics_type:view)'
|
||||
</code></pre><ul>
|
||||
@ -412,12 +412,12 @@ $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=50.116.102.77' dspace.log.2018-09-
|
||||
</ul>
|
||||
<pre><code>$ http 'http://localhost:3000/solr/statistics/select?indent=on&rows=0&q=type:0+owningItem:11576&fq=isBot:false&fq=bundleName:ORIGINAL&fq=statistics_type:view'
|
||||
</code></pre><ul>
|
||||
<li>As for item views, I suppose that's just the same query, minus the <code>bundleName:ORIGINAL</code>:</li>
|
||||
<li>As for item views, I suppose that’s just the same query, minus the <code>bundleName:ORIGINAL</code>:</li>
|
||||
</ul>
|
||||
<pre><code>$ http 'http://localhost:3000/solr/statistics/select?indent=on&rows=0&q=type:0+owningItem:11576&fq=isBot:false&fq=-bundleName:ORIGINAL&fq=statistics_type:view'
|
||||
</code></pre><ul>
|
||||
<li>That one returns 766, which is exactly 1655 minus 889…</li>
|
||||
<li>Also, Solr's <code>fq</code> is similar to the regular <code>q</code> query parameter, but it is considered for the Solr query cache so it should be faster for multiple queries</li>
|
||||
<li>Also, Solr’s <code>fq</code> is similar to the regular <code>q</code> query parameter, but it is considered for the Solr query cache so it should be faster for multiple queries</li>
|
||||
</ul>
|
||||
<h2 id="2018-09-18">2018-09-18</h2>
|
||||
<ul>
|
||||
@ -432,7 +432,7 @@ $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=50.116.102.77' dspace.log.2018-09-
|
||||
"views": 15
|
||||
}
|
||||
</code></pre><ul>
|
||||
<li>The numbers are different than those that come from Atmire's statlets for some reason, but as I'm querying Solr directly, I have no idea where their numbers come from!</li>
|
||||
<li>The numbers are different than those that come from Atmire’s statlets for some reason, but as I’m querying Solr directly, I have no idea where their numbers come from!</li>
|
||||
<li>Moayad from CodeObia asked if I could make the API be able to paginate over all items, for example: /statistics?limit=100&page=1</li>
|
||||
<li>Getting all the item IDs from PostgreSQL is certainly easy:</li>
|
||||
</ul>
|
||||
@ -443,7 +443,7 @@ $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=50.116.102.77' dspace.log.2018-09-
|
||||
<h2 id="2018-09-19">2018-09-19</h2>
|
||||
<ul>
|
||||
<li>I emailed Jane Poole to ask if there is some money we can use from the Big Data Platform (BDP) to fund the purchase of some Atmire credits for CGSpace</li>
|
||||
<li>I learned that there is an efficient way to do <a href="http://yonik.com/solr/paging-and-deep-paging/">“deep paging” in large Solr results sets by using <code>cursorMark</code></a>, but it doesn't work with faceting</li>
|
||||
<li>I learned that there is an efficient way to do <a href="http://yonik.com/solr/paging-and-deep-paging/">“deep paging” in large Solr results sets by using <code>cursorMark</code></a>, but it doesn’t work with faceting</li>
|
||||
</ul>
|
||||
<h2 id="2018-09-20">2018-09-20</h2>
|
||||
<ul>
|
||||
@ -464,21 +464,21 @@ $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=50.116.102.77' dspace.log.2018-09-
|
||||
<h2 id="2018-09-21">2018-09-21</h2>
|
||||
<ul>
|
||||
<li>I see that there was a nice optimization to the ImageMagick PDF CMYK detection in the upstream <code>dspace-5_x</code> branch: <a href="https://github.com/DSpace/DSpace/pull/2204">DS-3664</a></li>
|
||||
<li>The fix will go into DSpace 5.10, and we are currently on DSpace 5.8 but I think I'll cherry-pick that fix into our <code>5_x-prod</code> branch:
|
||||
<li>The fix will go into DSpace 5.10, and we are currently on DSpace 5.8 but I think I’ll cherry-pick that fix into our <code>5_x-prod</code> branch:
|
||||
<ul>
|
||||
<li>4e8c7b578bdbe26ead07e36055de6896bbf02f83: ImageMagick: Only execute “identify” on first page</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>I think it would also be nice to cherry-pick the fixes for <a href="https://github.com/DSpace/DSpace/pull/2020">DS-3883</a>, which is related to optimizing the XMLUI item display of items with many bitstreams
|
||||
<ul>
|
||||
<li>a0ea20bd1821720b111e2873b08e03ce2bf93307: DS-3883: Don't loop through original bitstreams if only displaying thumbnails</li>
|
||||
<li>a0ea20bd1821720b111e2873b08e03ce2bf93307: DS-3883: Don’t loop through original bitstreams if only displaying thumbnails</li>
|
||||
<li>8d81e825dee62c2aa9d403a505e4a4d798964e8d: DS-3883: If only including thumbnails, only load the main item thumbnail.</li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<h2 id="2019-09-23">2019-09-23</h2>
|
||||
<ul>
|
||||
<li>I did more work on my <a href="https://github.com/alanorth/cgspace-statistics-api">cgspace-statistics-api</a>, fixing some item view counts and adding indexing via SQLite (I'm trying to avoid having to set up <em>yet another</em> database, user, password, etc) during deployment</li>
|
||||
<li>I did more work on my <a href="https://github.com/alanorth/cgspace-statistics-api">cgspace-statistics-api</a>, fixing some item view counts and adding indexing via SQLite (I’m trying to avoid having to set up <em>yet another</em> database, user, password, etc) during deployment</li>
|
||||
<li>I created a new branch called <code>5_x-upstream-cherry-picks</code> to test and track those cherry-picks from the upstream 5.x branch</li>
|
||||
<li>Also, I need to test the new LDAP server, so I will deploy that on DSpace Test today</li>
|
||||
<li>Rename my cgspace-statistics-api to <a href="https://github.com/alanorth/dspace-statistics-api">dspace-statistics-api</a> on GitHub</li>
|
||||
@ -486,7 +486,7 @@ $ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=50.116.102.77' dspace.log.2018-09-
|
||||
<h2 id="2018-09-24">2018-09-24</h2>
|
||||
<ul>
|
||||
<li>Trying to figure out how to get item views and downloads from SQLite in a join</li>
|
||||
<li>It appears SQLite doesn't support <code>FULL OUTER JOIN</code> so some people on StackOverflow have emulated it with <code>LEFT JOIN</code> and <code>UNION</code>:</li>
|
||||
<li>It appears SQLite doesn’t support <code>FULL OUTER JOIN</code> so some people on StackOverflow have emulated it with <code>LEFT JOIN</code> and <code>UNION</code>:</li>
|
||||
</ul>
|
||||
<pre><code>> SELECT views.views, views.id, downloads.downloads, downloads.id FROM itemviews views
|
||||
LEFT JOIN itemdownloads downloads USING(id)
|
||||
@ -495,7 +495,7 @@ SELECT views.views, views.id, downloads.downloads, downloads.id FROM itemdownloa
|
||||
LEFT JOIN itemviews views USING(id)
|
||||
WHERE views.id IS NULL;
|
||||
</code></pre><ul>
|
||||
<li>This “works” but the resulting rows are kinda messy so I'd have to do extra logic in Python</li>
|
||||
<li>This “works” but the resulting rows are kinda messy so I’d have to do extra logic in Python</li>
|
||||
<li>Maybe we can use one “items” table with defaults values and UPSERT (aka insert… on conflict … do update):</li>
|
||||
</ul>
|
||||
<pre><code>sqlite> CREATE TABLE items(id INT PRIMARY KEY, views INT DEFAULT 0, downloads INT DEFAULT 0);
|
||||
@ -507,9 +507,9 @@ sqlite> INSERT INTO items(id, views) VALUES(0, 3) ON CONFLICT(id) DO UPDATE S
|
||||
sqlite> INSERT INTO items(id, views) VALUES(0, 7) ON CONFLICT(id) DO UPDATE SET downloads=excluded.views;
|
||||
</code></pre><ul>
|
||||
<li>This totally works!</li>
|
||||
<li>Note the special <code>excluded.views</code> form! See <a href="https://www.sqlite.org/lang_UPSERT.html">SQLite's lang_UPSERT documentation</a></li>
|
||||
<li>Oh nice, I finally finished the Falcon API route to page through all the results using SQLite's amazing <code>LIMIT</code> and <code>OFFSET</code> support</li>
|
||||
<li>But when I deployed it on my Ubuntu 16.04 environment I realized Ubuntu's SQLite is old and doesn't support <code>UPSERT</code>, so my indexing doesn't work…</li>
|
||||
<li>Note the special <code>excluded.views</code> form! See <a href="https://www.sqlite.org/lang_UPSERT.html">SQLite’s lang_UPSERT documentation</a></li>
|
||||
<li>Oh nice, I finally finished the Falcon API route to page through all the results using SQLite’s amazing <code>LIMIT</code> and <code>OFFSET</code> support</li>
|
||||
<li>But when I deployed it on my Ubuntu 16.04 environment I realized Ubuntu’s SQLite is old and doesn’t support <code>UPSERT</code>, so my indexing doesn’t work…</li>
|
||||
<li>Apparently <code>UPSERT</code> came in SQLite 3.24.0 (2018-06-04), and Ubuntu 16.04 has 3.11.0</li>
|
||||
<li>Ok this is hilarious, I manually downloaded the <a href="https://packages.ubuntu.com/cosmic/libsqlite3-0">libsqlite3 3.24.0 deb from Ubuntu 18.10 “cosmic”</a> and installed it in Ubnutu 16.04 and now the Python <code>indexer.py</code> works</li>
|
||||
<li>This is definitely a dirty hack, but the list of packages we use that depend on <code>libsqlite3-0</code> in Ubuntu 16.04 are actually pretty few:</li>
|
||||
@ -543,28 +543,28 @@ dspacestatistics-> (id INT PRIMARY KEY, views INT DEFAULT 0, downloads INT DE
|
||||
</code></pre><h2 id="2018-09-25">2018-09-25</h2>
|
||||
<ul>
|
||||
<li>I deployed the DSpace statistics API on CGSpace, but when I ran the indexer it wanted to index 180,000 pages of item views</li>
|
||||
<li>I'm not even sure how that's possible, as we only have 74,000 items!</li>
|
||||
<li>I’m not even sure how that’s possible, as we only have 74,000 items!</li>
|
||||
<li>I need to inspect the <code>id</code> values that are returned for views and cross check them with the <code>owningItem</code> values for bitstream downloads…</li>
|
||||
<li>Also, I could try to check all IDs against the items table to see if they are actually items (perhaps the Solr <code>id</code> field doesn't correspond with <em>actual</em> DSpace items?)</li>
|
||||
<li>I want to purge the bot hits from the Solr statistics core, as I am now realizing that I don't give a shit about tens of millions of hits by Google and Bing indexing my shit every day (at least not in Solr!)</li>
|
||||
<li>CGSpace's Solr core has 150,000,000 documents in it… and it's still pretty fast to query, but it's really a maintenance and backup burden</li>
|
||||
<li>DSpace Test currently has about 2,000,000 documents with <code>isBot:true</code> in its Solr statistics core, and the size on disk is 2GB (it's not much, but I have to test this somewhere!)</li>
|
||||
<li>According to the <a href="https://wiki.duraspace.org/display/DSDOC5x/SOLR+Statistics+Maintenance">DSpace 5.x Solr documentation</a> I can use <code>dspace stats-util -f</code>, so let's try it:</li>
|
||||
<li>Also, I could try to check all IDs against the items table to see if they are actually items (perhaps the Solr <code>id</code> field doesn’t correspond with <em>actual</em> DSpace items?)</li>
|
||||
<li>I want to purge the bot hits from the Solr statistics core, as I am now realizing that I don’t give a shit about tens of millions of hits by Google and Bing indexing my shit every day (at least not in Solr!)</li>
|
||||
<li>CGSpace’s Solr core has 150,000,000 documents in it… and it’s still pretty fast to query, but it’s really a maintenance and backup burden</li>
|
||||
<li>DSpace Test currently has about 2,000,000 documents with <code>isBot:true</code> in its Solr statistics core, and the size on disk is 2GB (it’s not much, but I have to test this somewhere!)</li>
|
||||
<li>According to the <a href="https://wiki.duraspace.org/display/DSDOC5x/SOLR+Statistics+Maintenance">DSpace 5.x Solr documentation</a> I can use <code>dspace stats-util -f</code>, so let’s try it:</li>
|
||||
</ul>
|
||||
<pre><code>$ dspace stats-util -f
|
||||
</code></pre><ul>
|
||||
<li>The command comes back after a few seconds and I still see 2,000,000 documents in the statistics core with <code>isBot:true</code></li>
|
||||
<li>I was just writing a message to the dspace-tech mailing list and then I decided to check the number of bot view events on DSpace Test again, and now it's 201 instead of 2,000,000, and statistics core is only 30MB now!</li>
|
||||
<li>I was just writing a message to the dspace-tech mailing list and then I decided to check the number of bot view events on DSpace Test again, and now it’s 201 instead of 2,000,000, and statistics core is only 30MB now!</li>
|
||||
<li>I will set the <code>logBots = false</code> property in <code>dspace/config/modules/usage-statistics.cfg</code> on DSpace Test and check if the number of <code>isBot:true</code> events goes up any more…</li>
|
||||
<li>I restarted the server with <code>logBots = false</code> and after it came back up I see 266 events with <code>isBots:true</code> (maybe they were buffered)… I will check again tomorrow</li>
|
||||
<li>After a few hours I see there are still only 266 view events with <code>isBot:true</code> on DSpace Test's Solr statistics core, so I'm definitely going to deploy this on CGSpace soon</li>
|
||||
<li>Also, CGSpace currently has 60,089,394 view events with <code>isBot:true</code> in it's Solr statistics core and it is 124GB!</li>
|
||||
<li>After a few hours I see there are still only 266 view events with <code>isBot:true</code> on DSpace Test’s Solr statistics core, so I’m definitely going to deploy this on CGSpace soon</li>
|
||||
<li>Also, CGSpace currently has 60,089,394 view events with <code>isBot:true</code> in it’s Solr statistics core and it is 124GB!</li>
|
||||
<li>Amazing! After running <code>dspace stats-util -f</code> on CGSpace the Solr statistics core went from 124GB to 60GB, and now there are only 700 events with <code>isBot:true</code> so I should really disable logging of bot events!</li>
|
||||
<li>I'm super curious to see how the JVM heap usage changes…</li>
|
||||
<li>I’m super curious to see how the JVM heap usage changes…</li>
|
||||
<li>I made (and merged) a pull request to disable bot logging on the <code>5_x-prod</code> branch (<a href="https://github.com/ilri/DSpace/pull/387">#387</a>)</li>
|
||||
<li>Now I'm wondering if there are other bot requests that aren't classified as bots because the IP lists or user agents are outdated</li>
|
||||
<li>Now I’m wondering if there are other bot requests that aren’t classified as bots because the IP lists or user agents are outdated</li>
|
||||
<li>DSpace ships a list of spider IPs, for example: <code>config/spiders/iplists.com-google.txt</code></li>
|
||||
<li>I checked the list against all the IPs we've seen using the “Googlebot” useragent on CGSpace's nginx access logs</li>
|
||||
<li>I checked the list against all the IPs we’ve seen using the “Googlebot” useragent on CGSpace’s nginx access logs</li>
|
||||
<li>The first thing I learned is that shit tons of IPs in Russia, Ukraine, Ireland, Brazil, Portugal, the US, Canada, etc are pretending to be “Googlebot”…</li>
|
||||
<li>According to the <a href="https://support.google.com/webmasters/answer/80553">Googlebot FAQ</a> the domain name in the reverse DNS lookup should contain either <code>googlebot.com</code> or <code>google.com</code></li>
|
||||
<li>In Solr this appears to be an appropriate query that I can maybe use later (returns 81,000 documents):</li>
|
||||
@ -577,7 +577,7 @@ dspacestatistics-> (id INT PRIMARY KEY, views INT DEFAULT 0, downloads INT DE
|
||||
</code></pre><ul>
|
||||
<li>And magically all those 81,000 documents are gone!</li>
|
||||
<li>After a few hours the Solr statistics core is down to 44GB on CGSpace!</li>
|
||||
<li>I did a <em>major</em> refactor and logic fix in the DSpace Statistics API's <code>indexer.py</code></li>
|
||||
<li>I did a <em>major</em> refactor and logic fix in the DSpace Statistics API’s <code>indexer.py</code></li>
|
||||
<li>Basically, it turns out that using <code>facet.mincount=1</code> is really beneficial for me because it reduces the size of the Solr result set, reduces the amount of data we need to ingest into PostgreSQL, and the API returns HTTP 404 Not Found for items without views or downloads anyways</li>
|
||||
<li>I deployed the new version on CGSpace and now it looks pretty good!</li>
|
||||
</ul>
|
||||
@ -585,14 +585,14 @@ dspacestatistics-> (id INT PRIMARY KEY, views INT DEFAULT 0, downloads INT DE
|
||||
...
|
||||
Indexing item downloads (page 260 of 260)
|
||||
</code></pre><ul>
|
||||
<li>And now it's fast as hell due to the muuuuch smaller Solr statistics core</li>
|
||||
<li>And now it’s fast as hell due to the muuuuch smaller Solr statistics core</li>
|
||||
</ul>
|
||||
<h2 id="2018-09-26">2018-09-26</h2>
|
||||
<ul>
|
||||
<li>Linode emailed to say that CGSpace (linode18) was using 30Mb/sec of outward bandwidth for two hours around midnight</li>
|
||||
<li>I don't see anything unusual in the nginx logs, so perhaps it was the cron job that syncs the Solr database to Amazon S3?</li>
|
||||
<li>I don’t see anything unusual in the nginx logs, so perhaps it was the cron job that syncs the Solr database to Amazon S3?</li>
|
||||
<li>It could be that the bot purge yesterday changed the core significantly so there was a lot to change?</li>
|
||||
<li>I don't see any drop in JVM heap size in CGSpace's munin stats since I did the Solr cleanup, but this looks pretty good:</li>
|
||||
<li>I don’t see any drop in JVM heap size in CGSpace’s munin stats since I did the Solr cleanup, but this looks pretty good:</li>
|
||||
</ul>
|
||||
<p><img src="/cgspace-notes/2018/09/tomcat_maxtime-week.png" alt="Tomcat max processing time week"></p>
|
||||
<ul>
|
||||
@ -610,16 +610,16 @@ real 77m3.755s
|
||||
user 7m39.785s
|
||||
sys 2m18.485s
|
||||
</code></pre><ul>
|
||||
<li>I told Peter it's better to do the access rights before the usage rights because the git branches are conflicting with each other and it's actually a pain in the ass to keep changing the values as we discuss, rebase, merge, fix conflicts…</li>
|
||||
<li>I told Peter it’s better to do the access rights before the usage rights because the git branches are conflicting with each other and it’s actually a pain in the ass to keep changing the values as we discuss, rebase, merge, fix conflicts…</li>
|
||||
<li>Udana and Mia from WLE were asking some questions about their <a href="https://feeds.feedburner.com/WLEcgspace">WLE Feedburner feed</a></li>
|
||||
<li>It's pretty confusing, because until recently they were entering issue dates as only YYYY (like 2018) and their feeds were all showing items in the wrong order</li>
|
||||
<li>I'm not exactly sure what their problem now is, though (confusing)</li>
|
||||
<li>I updated the dspace-statistiscs-api to use psycopg2's <code>execute_values()</code> to insert batches of 100 values into PostgreSQL instead of doing every insert individually</li>
|
||||
<li>It’s pretty confusing, because until recently they were entering issue dates as only YYYY (like 2018) and their feeds were all showing items in the wrong order</li>
|
||||
<li>I’m not exactly sure what their problem now is, though (confusing)</li>
|
||||
<li>I updated the dspace-statistiscs-api to use psycopg2’s <code>execute_values()</code> to insert batches of 100 values into PostgreSQL instead of doing every insert individually</li>
|
||||
<li>On CGSpace this reduces the total run time of <code>indexer.py</code> from 432 seconds to 400 seconds (most of the time is actually spent in getting the data from Solr though)</li>
|
||||
</ul>
|
||||
<h2 id="2018-09-27">2018-09-27</h2>
|
||||
<ul>
|
||||
<li>Linode emailed to say that CGSpace's (linode19) CPU load was high for a few hours last night</li>
|
||||
<li>Linode emailed to say that CGSpace’s (linode19) CPU load was high for a few hours last night</li>
|
||||
<li>Looking in the nginx logs around that time I see some new IPs that look like they are harvesting things:</li>
|
||||
</ul>
|
||||
<pre><code># zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "26/Sep/2018:(19|20|21)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
|
||||
@ -643,7 +643,7 @@ sys 2m18.485s
|
||||
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=68.6.87.12' dspace.log.2018-09-26 | sort | uniq
|
||||
758
|
||||
</code></pre><ul>
|
||||
<li>I will add their IPs to the list of bad bots in nginx so we can add a “bot” user agent to them and let Tomcat's Crawler Session Manager Valve handle them</li>
|
||||
<li>I will add their IPs to the list of bad bots in nginx so we can add a “bot” user agent to them and let Tomcat’s Crawler Session Manager Valve handle them</li>
|
||||
<li>I asked Atmire to prepare an invoice for 125 credits</li>
|
||||
</ul>
|
||||
<h2 id="2018-09-29">2018-09-29</h2>
|
||||
@ -670,7 +670,7 @@ $ ./fix-metadata-values.py -i 2018-09-29-fix-authors.csv -db dspace -u dspace -p
|
||||
</ul>
|
||||
<pre><code>dspace=# \copy (select distinct text_value, count(*) from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'language' and qualifier = 'iso') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/2018-09-30-languages.csv with csv;
|
||||
</code></pre><ul>
|
||||
<li>Then I can simply delete the “Other” and “other” ones because that's not useful at all:</li>
|
||||
<li>Then I can simply delete the “Other” and “other” ones because that’s not useful at all:</li>
|
||||
</ul>
|
||||
<pre><code>dspace=# DELETE FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'language' and qualifier = 'iso') AND text_value='Other';
|
||||
DELETE 6
|
||||
|
||||
Reference in New Issue
Block a user