Notes on CGSpace Notes

November, 2019

Mon, 04 Nov 2019 12:20:30 +0200

2019-11-04

Peter noticed that there were 5.2 million hits on CGSpace in 2019-10 according to the Atmire usage statistics
- I looked in the nginx logs and see 4.6 million in the access logs, and 1.2 million in the API logs:
```
# zcat --force /var/log/nginx/*access.log.*.gz | grep -cE "[0-9]{1,2}/Oct/2019"
4671942
# zcat --force /var/log/nginx/{rest,oai,statistics}.log.*.gz | grep -cE "[0-9]{1,2}/Oct/2019"
1277694
```
So 4.6 million from XMLUI and another 1.2 million from API requests

Let’s see how many of the REST API requests were for bitstreams (because they are counted in Solr stats):

# zcat --force /var/log/nginx/rest.log.*.gz | grep -c -E "[0-9]{1,2}/Oct/2019"
1183456 
# zcat --force /var/log/nginx/rest.log.*.gz | grep -E "[0-9]{1,2}/Oct/2019" | grep -c -E "/rest/bitstreams"
106781

CGSpace CG Core v2 Migration

Mon, 28 Oct 2019 13:27:35 +0200

Possible changes to CGSpace metadata fields to align more with DC, QDC, and DCTERMS as well as CG Core v2.

With reference to CG Core v2 draft standard by Marie-Angélique as well as DCMI DCTERMS.

October, 2019

Tue, 01 Oct 2019 13:20:51 +0300

2019-10-01 Udana from IWMI asked me for a CSV export of their community on CGSpace I exported it, but a quick run through the csv-metadata-quality tool shows that there are some low-hanging fruits we can fix before I send him the data I will limit the scope to the titles, regions, subregions, and river basins for now to manually fix some non-breaking spaces (U+00A0) there that would otherwise be removed by the csv-metadata-quality script’s “unneccesary Unicode” fix:

September, 2019

Sun, 01 Sep 2019 10:17:51 +0300

2019-09-01

Linode emailed to say that CGSpace (linode18) had a high rate of outbound traffic for several hours this morning

Here are the top ten IPs in the nginx XMLUI and REST/OAI logs this morning:

# zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep -E "01/Sep/2019:0" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
440 17.58.101.255
441 157.55.39.101
485 207.46.13.43
728 169.60.128.125
730 207.46.13.108
758 157.55.39.9
808 66.160.140.179
814 207.46.13.212
2472 163.172.71.23
6092 3.94.211.189
# zcat --force /var/log/nginx/rest.log /var/log/nginx/rest.log.1 /var/log/nginx/oai.log /var/log/nginx/oai.log.1 | grep -E "01/Sep/2019:0" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
 33 2a01:7e00::f03c:91ff:fe16:fcb
 57 3.83.192.124
 57 3.87.77.25
 57 54.82.1.8
822 2a01:9cc0:47:1:1a:4:0:2
1223 45.5.184.72
1633 172.104.229.92
5112 205.186.128.185
7249 2a01:7e00::f03c:91ff:fe18:7396
9124 45.5.186.2

August, 2019

Sat, 03 Aug 2019 12:39:51 +0300

2019-08-03

Look at Bioversity’s latest migration CSV and now I see that Francesco has cleaned up the extra columns and the newline at the end of the file, but many of the column headers have an extra space in the name…

2019-08-04

Deploy ORCID identifier updates requested by Bioversity to CGSpace
Run system updates on CGSpace (linode18) and reboot it
- Before updating it I checked Solr and verified that all statistics cores were loaded properly…
- After rebooting, all statistics cores were loaded… wow, that’s lucky.
Run system updates on DSpace Test (linode19) and reboot it

July, 2019

Mon, 01 Jul 2019 12:13:51 +0300

2019-07-01

Create an “AfricaRice books and book chapters” collection on CGSpace for AfricaRice
Last month Sisay asked why the following “most popular” statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace:
- DSpace Test
- CGSpace
Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community

June, 2019

Sun, 02 Jun 2019 10:57:51 +0300

2019-06-02

Merge the Solr filterCache and XMLUI ISI journal changes to the 5_x-prod branch and deploy on CGSpace
Run system updates on CGSpace (linode18) and reboot it

2019-06-03

Skype with Marie-Angélique and Abenet about CG Core v2

May, 2019

Wed, 01 May 2019 07:37:43 +0300

2019-05-01

Help CCAFS with regenerating some item thumbnails after they uploaded new PDFs to some items on CGSpace
A user on the dspace-tech mailing list offered some suggestions for troubleshooting the problem with the inability to delete certain items
- Apparently if the item is in the workflowitem table it is submitted to a workflow
- And if it is in the workspaceitem table it is in the pre-submitted state
The item seems to be in a pre-submitted state, so I tried to delete it from there:
```
dspace=# DELETE FROM workspaceitem WHERE item_id=74648;
DELETE 1
```
But after this I tried to delete the item from the XMLUI and it is still present…

April, 2019

Mon, 01 Apr 2019 09:00:43 +0300

2019-04-01

Meeting with AgroKnow to discuss CGSpace, ILRI data, AReS, GARDIAN, etc
- They asked if we had plans to enable RDF support in CGSpace
There have been 4,400 more downloads of the CTA Spore publication from those strange Amazon IP addresses today
- I suspected that some might not be successful, because the stats show less, but today they were all HTTP 200!
```
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep 'Spore-192-EN-web.pdf' | grep -E '(18.196.196.108|18.195.78.144|18.195.218.6)' | awk '{print $9}' | sort | uniq -c | sort -n | tail -n 5
4432 200
```
In the last two weeks there have been 47,000 downloads of this same exact PDF by these three IP addresses

Apply country and region corrections and deletions on DSpace Test and CGSpace:

$ ./fix-metadata-values.py -i /tmp/2019-02-21-fix-9-countries.csv -db dspace -u dspace -p 'fuuu' -f cg.coverage.country -m 228 -t ACTION -d
$ ./fix-metadata-values.py -i /tmp/2019-02-21-fix-4-regions.csv -db dspace -u dspace -p 'fuuu' -f cg.coverage.region -m 231 -t action -d
$ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-2-countries.csv -db dspace -u dspace -p 'fuuu' -m 228 -f cg.coverage.country -d
$ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace -u dspace -p 'fuuu' -m 231 -f cg.coverage.region -d

March, 2019

Fri, 01 Mar 2019 12:16:30 +0100

2019-03-01

I checked IITA’s 259 Feb 14 records from last month for duplicates using Atmire’s Duplicate Checker on a fresh snapshot of CGSpace on my local machine and everything looks good
I am now only waiting to hear from her about where the items should go, though I assume Journal Articles go to IITA Journal Articles collection, etc…
Looking at the other half of Udana’s WLE records from 2018-11
- I finished the ones for Restoring Degraded Landscapes (RDL), but these are for Variability, Risks and Competing Uses (VRC)
- I did the usual cleanups for whitespace, added regions where they made sense for certain countries, cleaned up the DOI link formats, added rights information based on the publications page for a few items
- Most worryingly, there are encoding errors in the abstracts for eleven items, for example:
- 68.15% � 9.45 instead of 68.15% ± 9.45
- 2003�2013 instead of 2003–2013
I think I will need to ask Udana to re-copy and paste the abstracts with more care using Google Docs

February, 2019

Fri, 01 Feb 2019 21:37:30 +0200

2019-02-01

Linode has alerted a few times since last night that the CPU usage on CGSpace (linode18) was high despite me increasing the alert threshold last week from 250% to 275%—I might need to increase it again!

The top IPs before, during, and after this latest alert tonight were:

# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "01/Feb/2019:(17|18|19|20|21)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
245 207.46.13.5
332 54.70.40.11
385 5.143.231.38
405 207.46.13.173
405 207.46.13.75
1117 66.249.66.219
1121 35.237.175.180
1546 5.9.6.51
2474 45.5.186.2
5490 85.25.237.71

85.25.237.71 is the “Linguee Bot” that I first saw last month
The Solr statistics the past few months have been very high and I was wondering if the web server logs also showed an increase

There were just over 3 million accesses in the nginx logs last month:

# time zcat --force /var/log/nginx/* | grep -cE "[0-9]{1,2}/Jan/2019"
3018243

real    0m19.873s
user    0m22.203s
sys     0m1.979s

January, 2019

Wed, 02 Jan 2019 09:48:30 +0200

2019-01-02

Linode alerted that CGSpace (linode18) had a higher outbound traffic rate than normal early this morning

I don’t see anything interesting in the web server logs around that time though:

# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "02/Jan/2019:0(1|2|3)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
 92 40.77.167.4
 99 210.7.29.100
120 38.126.157.45
177 35.237.175.180
177 40.77.167.32
216 66.249.75.219
225 18.203.76.93
261 46.101.86.248
357 207.46.13.1
903 54.70.40.11

December, 2018

Sun, 02 Dec 2018 02:09:30 +0200

2018-12-01

Switch CGSpace (linode18) to use OpenJDK instead of Oracle JDK
I manually installed OpenJDK, then removed Oracle JDK, then re-ran the Ansible playbook to update all configuration files, etc
Then I ran all system updates and restarted the server

2018-12-02

I noticed that there is another issue with PDF thumbnails on CGSpace, and I see there was another Ghostscript vulnerability last week

November, 2018

Thu, 01 Nov 2018 16:41:30 +0200

2018-11-01

Finalize AReS Phase I and Phase II ToRs
Send a note about my dspace-statistics-api to the dspace-tech mailing list

2018-11-03

Linode has been sending mails a few times a day recently that CGSpace (linode18) has had high CPU usage
Today these are the top 10 IPs:

October, 2018

Mon, 01 Oct 2018 22:31:54 +0300

2018-10-01

Phil Thornton got an ORCID identifier so we need to add it to the list on CGSpace and tag his existing items
I created a GitHub issue to track this #389, because I’m super busy in Nairobi right now

September, 2018

Sun, 02 Sep 2018 09:55:54 +0300

2018-09-02

New PostgreSQL JDBC driver version 42.2.5
I’ll update the DSpace role in our Ansible infrastructure playbooks and run the updated playbooks on CGSpace and DSpace Test
Also, I’ll re-run the postgresql tasks because the custom PostgreSQL variables are dynamic according to the system’s RAM, and we never re-ran them after migrating to larger Linodes last month
I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I’m getting those autowire errors in Tomcat 8.5.30 again:

August, 2018

Wed, 01 Aug 2018 11:52:54 +0300

2018-08-01

DSpace Test had crashed at some point yesterday morning and I see the following in dmesg:

[Tue Jul 31 00:00:41 2018] Out of memory: Kill process 1394 (java) score 668 or sacrifice child
[Tue Jul 31 00:00:41 2018] Killed process 1394 (java) total-vm:15601860kB, anon-rss:5355528kB, file-rss:0kB, shmem-rss:0kB
[Tue Jul 31 00:00:41 2018] oom_reaper: reaped process 1394 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Judging from the time of the crash it was probably related to the Discovery indexing that starts at midnight
From the DSpace log I see that eventually Solr stopped responding, so I guess the java process that was OOM killed above was Tomcat’s
I’m not sure why Tomcat didn’t crash with an OutOfMemoryError…
Anyways, perhaps I should increase the JVM heap from 5120m to 6144m like we did a few months ago when we tried to run the whole CGSpace Solr core
The server only has 8GB of RAM so we’ll eventually need to upgrade to a larger one because we’ll start starving the OS, PostgreSQL, and command line batch processes
I ran all system updates on DSpace Test and rebooted it

July, 2018

Sun, 01 Jul 2018 12:56:54 +0300

2018-07-01

I want to upgrade DSpace Test to DSpace 5.8 so I took a backup of its current database just in case:
```
$ pg_dump -b -v -o --format=custom -U dspace -f dspace-2018-07-01.backup dspace
```
During the mvn package stage on the 5.8 branch I kept getting issues with java running out of memory:
```
There is insufficient memory for the Java Runtime Environment to continue.
```

June, 2018

Mon, 04 Jun 2018 19:49:54 -0700

2018-06-04

Test the DSpace 5.8 module upgrades from Atmire (#378)
- There seems to be a problem with the CUA and L&R versions in pom.xml because they are using SNAPSHOT and it doesn’t build
I added the new CCAFS Phase II Project Tag PII-FP1_PACCA2 and merged it into the 5_x-prod branch (#379)

I proofed and tested the ILRI author corrections that Peter sent back to me this week:

$ ./fix-metadata-values.py -i /tmp/2018-05-30-Correct-660-authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t correct -m 3 -n

I think a sane proofing workflow in OpenRefine is to apply the custom text facets for check/delete/remove and illegal characters that I developed in March, 2018

Time to index ~70,000 items on CGSpace:

$ time schedtool -D -e ionice -c2 -n7 nice -n19 [dspace]/bin/dspace index-discovery -b                                  

real    74m42.646s
user    8m5.056s
sys     2m7.289s

May, 2018

Tue, 01 May 2018 16:43:54 +0300

2018-05-01

I cleared the Solr statistics core on DSpace Test by issuing two commands directly to the Solr admin interface:
- http://localhost:3000/solr/statistics/update?stream.body=%3Cdelete%3E%3Cquery%3E*:*%3C/query%3E%3C/delete%3E
- http://localhost:3000/solr/statistics/update?stream.body=%3Ccommit/%3E
Then I reduced the JVM heap size from 6144 back to 5120m
Also, I switched it to use OpenJDK instead of Oracle Java, as well as re-worked the Ansible infrastructure scripts to support hosts choosing which distribution they want to use

April, 2018

Sun, 01 Apr 2018 16:13:54 +0200

2018-04-01

I tried to test something on DSpace Test but noticed that it’s down since god knows when
Catalina logs at least show some memory errors yesterday:

March, 2018

Fri, 02 Mar 2018 16:07:54 +0200

2018-03-02

Export a CSV of the IITA community metadata for Martin Mueller

February, 2018

Thu, 01 Feb 2018 16:28:54 +0200

2018-02-01

Peter gave feedback on the dc.rights proof of concept that I had sent him last week
We don’t need to distinguish between internal and external works, so that makes it just a simple list
Yesterday I figured out how to monitor DSpace sessions using JMX
I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu’s munin-plugins-java package and used the stuff I discovered about JMX in 2018-01

January, 2018

Tue, 02 Jan 2018 08:35:54 -0800

2018-01-02

Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time
I didn’t get any load alerts from Linode and the REST and XMLUI logs don’t show anything out of the ordinary
The nginx logs show HTTP 200s until 02/Jan/2018:11:27:17 +0000 when Uptime Robot got an HTTP 500
In dspace.log around that time I see many errors like “Client closed the connection before file download was complete”

And just before that I see this:

Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000].

Ah hah! So the pool was actually empty!
I need to increase that, let’s try to bump it up from 50 to 75
After that one client got an HTTP 499 but then the rest were HTTP 200, so I don’t know what the hell Uptime Robot saw

I notice this error quite a few times in dspace.log:

2018-01-02 01:21:19,137 ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ Error while searching for sidebar facets
org.dspace.discovery.SearchServiceException: org.apache.solr.search.SyntaxError: Cannot parse 'dateIssued_keyword:[1976+TO+1979]': Encountered " "]" "] "" at line 1, column 32.

And there are many of these errors every day for the past month:

$ grep -c "Error while searching for sidebar facets" dspace.log.*
dspace.log.2017-11-21:4
dspace.log.2017-11-22:1
dspace.log.2017-11-23:4
dspace.log.2017-11-24:11
dspace.log.2017-11-25:0
dspace.log.2017-11-26:1
dspace.log.2017-11-27:7
dspace.log.2017-11-28:21
dspace.log.2017-11-29:31
dspace.log.2017-11-30:15
dspace.log.2017-12-01:15
dspace.log.2017-12-02:20
dspace.log.2017-12-03:38
dspace.log.2017-12-04:65
dspace.log.2017-12-05:43
dspace.log.2017-12-06:72
dspace.log.2017-12-07:27
dspace.log.2017-12-08:15
dspace.log.2017-12-09:29
dspace.log.2017-12-10:35
dspace.log.2017-12-11:20
dspace.log.2017-12-12:44
dspace.log.2017-12-13:36
dspace.log.2017-12-14:59
dspace.log.2017-12-15:104
dspace.log.2017-12-16:53
dspace.log.2017-12-17:66
dspace.log.2017-12-18:83
dspace.log.2017-12-19:101
dspace.log.2017-12-20:74
dspace.log.2017-12-21:55
dspace.log.2017-12-22:66
dspace.log.2017-12-23:50
dspace.log.2017-12-24:85
dspace.log.2017-12-25:62
dspace.log.2017-12-26:49
dspace.log.2017-12-27:30
dspace.log.2017-12-28:54
dspace.log.2017-12-29:68
dspace.log.2017-12-30:89
dspace.log.2017-12-31:53
dspace.log.2018-01-01:45
dspace.log.2018-01-02:34

Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let’s Encrypt if it’s just a handful of domains

December, 2017

Fri, 01 Dec 2017 13:53:54 +0300

2017-12-01

Uptime Robot noticed that CGSpace went down
The logs say “Timeout waiting for idle object”
PostgreSQL activity says there are 115 connections currently
The list of connections to XMLUI and REST API for today:

November, 2017

Thu, 02 Nov 2017 09:37:54 +0200

2017-11-01

The CORE developers responded to say they are looking into their bot not respecting our robots.txt

2017-11-02

Today there have been no hits by CORE and no alerts from Linode (coincidence?)
```
# grep -c "CORE" /var/log/nginx/access.log
0
```

Generate list of authors on CGSpace for Peter to go through and correct:

dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors.csv with csv;
COPY 54701

October, 2017

Sun, 01 Oct 2017 08:07:54 +0300

2017-10-01

Peter emailed to point out that many items in the ILRI archive collection have multiple handles:
```
http://hdl.handle.net/10568/78495||http://hdl.handle.net/10568/79336
```
There appears to be a pattern but I’ll have to look a bit closer and try to clean them up automatically, either in SQL or in OpenRefine
Add Katherine Lutz to the groups for content submission and edit steps of the CGIAR System collections

CGIAR Library Migration

Mon, 18 Sep 2017 16:38:35 +0300

Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called CGIAR System Organization.