Notes on CGSpace Notes

November, 2021

Tue, 02 Nov 2021 22:27:07 +0200

2021-11-02

I experimented with manually sharding the Solr statistics on DSpace Test
First I exported all the 2019 stats from CGSpace:

$ ./run.sh -s http://localhost:8081/solr/statistics -f 'time:2019-*' -a export -o statistics-2019.json -k uid
$ zstd statistics-2019.json

October, 2021

Fri, 01 Oct 2021 11:14:07 +0300

2021-10-01

Export all affiliations on CGSpace and run them against the latest RoR data dump:

localhost/dspace63= > \COPY (SELECT DISTINCT text_value as "cg.contributor.affiliation", count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC) to /tmp/2021-10-01-affiliations.csv WITH CSV HEADER;
$ csvcut -c 1 /tmp/2021-10-01-affiliations.csv | sed 1d > /tmp/2021-10-01-affiliations.txt
$ ./ilri/ror-lookup.py -i /tmp/2021-10-01-affiliations.txt -r 2021-09-23-ror-data.json -o /tmp/2021-10-01-affili
ations-matching.csv
$ csvgrep -c matched -m true /tmp/2021-10-01-affiliations-matching.csv | sed 1d | wc -l 
1879
$ wc -l /tmp/2021-10-01-affiliations.txt 
7100 /tmp/2021-10-01-affiliations.txt

So we have 1879/7100 (26.46%) matching already

September, 2021

Wed, 01 Sep 2021 09:14:07 +0300

2021-09-02

Troubleshooting the missing Altmetric scores on AReS
- Turns out that I didn’t actually fix them last month because the check for content.altmetric still exists, and I can’t access the DOIs using _h.source.DOI for some reason
- I can access all other kinds of item metadata using the Elasticsearch label, but not DOI!!!
- I will change DOI to tomato in the repository setup and start a re-harvest… I need to see if this is some kind of reserved word or something…
- Even as tomato I can’t access that field as _h.source.tomato in Angular, but it does work as a filter source… sigh
I’m having problems using the OpenRXV API
- The syntax Moayad showed me last month doesn’t seem to honor the search query properly…

August, 2021

Sun, 01 Aug 2021 09:01:07 +0300

2021-08-01

Update Docker images on AReS server (linode20) and reboot the server:

# docker images | grep -v ^REPO | sed 's/ \+/:/g' | cut -d: -f1,2 | grep -v none | xargs -L1 docker pull

I decided to upgrade linode20 from Ubuntu 18.04 to 20.04

July, 2021

Thu, 01 Jul 2021 08:53:07 +0300

2021-07-01

Export another list of ALL subjects on CGSpace, including AGROVOC and non-AGROVOC for Enrico:

localhost/dspace63= > \COPY (SELECT DISTINCT LOWER(text_value) AS subject, count(*) FROM metadatavalue WHERE dspace_object_id in (SELECT dspace_object_id FROM item) AND metadata_field_id IN (119, 120, 127, 122, 128, 125, 135, 203, 208, 210, 215, 123, 236, 242, 187) GROUP BY subject ORDER BY count DESC) to /tmp/2021-07-01-all-subjects.csv WITH CSV HEADER;
COPY 20994

June, 2021

Tue, 01 Jun 2021 10:51:07 +0300

2021-06-01

IWMI notified me that AReS was down with an HTTP 502 error
- Looking at UptimeRobot I see it has been down for 33 hours, but I never got a notification
- I don’t see anything in the Elasticsearch container logs, or the systemd journal on the host, but I notice that the angular_nginx container isn’t running
- I simply started it and AReS was running again:

May, 2021

Sun, 02 May 2021 09:50:54 +0300

2021-05-01

I looked at the top user agents and IPs in the Solr statistics for last month and I see these user agents:
- “RI/1.0”, 1337
- “Microsoft Office Word 2014”, 941
I will add the RI/1.0 pattern to our DSpace agents overload and purge them from Solr (we had previously seen this agent with 9,000 hits or so in 2020-09), but I think I will leave the Microsoft Word one… as that’s an actual user…

April, 2021

Thu, 01 Apr 2021 09:50:54 +0300

2021-04-01

I wrote a script to query Sherpa’s API for our ISSNs: sherpa-issn-lookup.py
- I’m curious to see how the results compare with the results from Crossref yesterday
AReS Explorer was down since this morning, I didn’t see anything in the systemd journal
- I simply took everything down with docker-compose and then back up, and then it was OK
- Perhaps one of the containers crashed, I should have looked closer but I was in a hurry

March, 2021

Mon, 01 Mar 2021 10:13:54 +0200

2021-03-01

Discuss some OpenRXV issues with Abdullah from CodeObia
- He’s trying to work on the DSpace 6+ metadata schema autoimport using the DSpace 6+ REST API
- Also, we found some issues building and running OpenRXV currently due to ecosystem shift in the Node.js dependencies

CGSpace CG Core v2 Migration

Sun, 21 Feb 2021 13:27:35 +0200

Changes to CGSpace metadata fields to align more with DC, QDC, and DCTERMS as well as CG Core v2. Implemented on 2021-02-21.

With reference to CG Core v2 draft standard by Marie-Angélique as well as DCMI DCTERMS.

February, 2021

Mon, 01 Feb 2021 10:13:54 +0200

2021-02-01

Abenet said that CIP found more duplicate records in their export from AReS
- I re-opened the issue on OpenRXV where we had previously noticed this
- The shared link where the duplicates are is here: https://cgspace.cgiar.org/explorer/shared/heEOz3YBnXdK69bR2ra6
I had a call with CodeObia to discuss the work on OpenRXV
Check the results of the AReS harvesting from last night:

$ curl -s 'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'
{
  "count" : 100875,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  }
}

January, 2021

Sun, 03 Jan 2021 10:13:54 +0200

2021-01-03

Peter notified me that some filters on AReS were broken again
- It’s the same issue with the field names getting .keyword appended to the end that I already filed an issue on OpenRXV about last month
- I fixed the broken filters (careful to not edit any others, lest they break too!)
Fix an issue with start page number for the DSpace REST API and statistics API in OpenRXV
- The start page had been “1” in the UI, but in the backend they were doing some gymnastics to adjust to the zero-based offset/limit/page of the DSpace REST API and the statistics API
- I adjusted it to default to 0 and added a note to the admin screen
- I realized that this issue was actually causing the first page of 100 statistics to be missing…
- For example, this item has 51 views on CGSpace, but 0 on AReS

December, 2020

Tue, 01 Dec 2020 11:32:54 +0200

2020-12-01

Atmire responded about the issue with duplicate data in our Solr statistics
- They noticed that some records in the statistics-2015 core haven’t been migrated with the AtomicStatisticsUpdateCLI tool yet and assumed that I haven’t migrated any of the records yet
- That’s strange, as I checked all ten cores and 2015 is the only one with some unmigrated documents, as according to the cua_version field
- I started processing those (about 411,000 records):

CGSpace DSpace 6 Upgrade

Sun, 15 Nov 2020 13:27:35 +0200

Notes about the DSpace 6 upgrade on CGSpace in 2020-11.

November, 2020

Sun, 01 Nov 2020 13:11:54 +0200

2020-11-01

Continue with processing the statistics-2019 Solr core with the AtomicStatisticsUpdateCLI tool on DSpace Test
- So far we’ve spent at least fifty hours to process the statistics and statistics-2019 core… wow.

October, 2020

Tue, 06 Oct 2020 16:55:54 +0300

2020-10-06

Add tests for the new /items POST handlers to the DSpace 6.x branch of my dspace-statistics-api
- It took a bit of extra work because I had to learn how to mock the responses for when Solr is not available
- Tag and release version 1.3.0 on GitHub: https://github.com/ilri/dspace-statistics-api/releases/tag/v1.3.0
Trying to test the changes Atmire sent last week but I had to re-create my local database from a recent CGSpace dump
- During the FlywayDB migration I got an error:

September, 2020

Wed, 02 Sep 2020 15:35:54 +0300

2020-09-02

Replace Marissa van Epp for Rhys Bucknall in the CCAFS groups on CGSpace because Marissa no longer works at CCAFS
The AReS Explorer hasn’t updated its index since 2020-08-22 when I last forced it
- I restarted it again now and told Moayad that the automatic indexing isn’t working
Add Alliance of Bioversity International and CIAT to affiliations on CGSpace
Abenet told me that the general search text on AReS doesn’t get reset when you use the “Reset Filters” button
- I filed a bug on OpenRXV: https://github.com/ilri/OpenRXV/issues/39
I filed an issue on OpenRXV to make some minor edits to the admin UI: https://github.com/ilri/OpenRXV/issues/40

August, 2020

Sun, 02 Aug 2020 15:35:54 +0300

2020-08-02

I spent a few days working on a Java-based curation task to tag items with ISO 3166-1 Alpha2 country codes based on their cg.coverage.country text values
- It looks up the names in ISO 3166-1 first, and then in our CGSpace countries mapping (which has five or so of Peter’s preferred “display” country names)
- It implements a “force” mode too that will clear existing country codes and re-tag everything
- It is class based so I can easily add support for other vocabularies, and the technique could even be used for organizations with mappings to ROR and Clarisa…

July, 2020

Wed, 01 Jul 2020 10:53:54 +0300

2020-07-01

A few users noticed that CGSpace wasn’t loading items today, item pages seem blank
- I looked at the PostgreSQL locks but they don’t seem unusual
- I guess this is the same “blank item page” issue that we had a few times in 2019 that we never solved
- I restarted Tomcat and PostgreSQL and the issue was gone
Since I was restarting Tomcat anyways I decided to redeploy the latest changes from the 5_x-prod branch and I added a note about COVID-19 items to the CGSpace frontpage at Peter’s request

June, 2020

Mon, 01 Jun 2020 13:55:39 +0300

2020-06-01

I tried to run the AtomicStatisticsUpdateCLI CUA migration script on DSpace Test (linode26) again and it is still going very slowly and has tons of errors like I noticed yesterday
- I sent Atmire the dspace.log from today and told them to log into the server to debug the process
In other news, I checked the statistics API on DSpace 6 and it’s working
I tried to build the OAI registry on the freshly migrated DSpace 6 on DSpace Test and I get an error:

May, 2020

Sat, 02 May 2020 09:52:04 +0300

2020-05-02

Peter said that CTA is having problems submitting an item to CGSpace
- Looking at the PostgreSQL stats it seems to be the same issue that Tezira was having last week, as I see the number of connections in ‘idle in transaction’ and ‘waiting for lock’ state are increasing again
- I see that CGSpace (linode18) is still using PostgreSQL JDBC driver version 42.2.11, and there were some bugs related to transactions fixed in 42.2.12 (which I had updated in the Ansible playbooks, but not deployed yet)

April, 2020

Thu, 02 Apr 2020 10:53:24 +0300

2020-04-02

Maria asked me to update Charles Staver’s ORCID iD in the submission template and on CGSpace, as his name was lower case before, and now he has corrected it
- I updated the fifty-eight existing items on CGSpace
Looking into the items Udana had asked about last week that were missing Altmetric donuts:
- The first is still missing its DOI, so I added it and tweeted its handle (after a few hours there was a donut with score 222)
- The second item now has a donut with score 2 since I tweeted its handle last week
- The third item now has a donut with score 1 since I tweeted it last week
On the same note, the one item Abenet pointed out last week now has a donut with score of 104 after I tweeted it last week

March, 2020

Mon, 02 Mar 2020 12:31:30 +0200

2020-03-02

Update dspace-statistics-api for DSpace 6+ UUIDs
- Tag version 1.2.0 on GitHub
Test migrating legacy Solr statistics to UUIDs with the as-of-yet unreleased SolrUpgradePre6xStatistics.java
- You need to download this into the DSpace 6.x source and compile it

February, 2020

Sun, 02 Feb 2020 11:56:30 +0200

2020-02-02

Continue working on porting CGSpace’s DSpace 5 code to DSpace 6.3 that I started yesterday
- Sign up for an account with MaxMind so I can get the GeoLite2-City.mmdb database
- I still need to wire up the API credentials and cron job into the Ansible infrastructure playbooks
- Fix some minor issues in the config and XMLUI themes, like removing Atmire stuff
- The code finally builds and runs with a fresh install

January, 2020

Mon, 06 Jan 2020 10:48:30 +0200

2020-01-06

Open a ticket with Atmire to request a quote for the upgrade to DSpace 6
Last week Altmetric responded about the item that had a lower score than than its DOI
- The score is now linked to the DOI
- Another item that had the same problem in 2019 has now also linked to the score for its DOI
- Another item that had the same problem in 2019 has also been fixed

2020-01-07

Peter Ballantyne highlighted one more WLE item that is missing the Altmetric score that its DOI has
- The DOI has a score of 259, but the Handle has no score at all
- I tweeted the CGSpace repository link

December, 2019

Sun, 01 Dec 2019 11:22:30 +0200

2019-12-01

Upgrade CGSpace (linode18) to Ubuntu 18.04:
- Check any packages that have residual configs and purge them:
- # dpkg -l | grep -E ‘^rc’ | awk ‘{print $2}’ | xargs dpkg -P
- Make sure all packages are up to date and the package manager is up to date, then reboot:

# apt update && apt full-upgrade
# apt-get autoremove && apt-get autoclean
# dpkg -C
# reboot

November, 2019

Mon, 04 Nov 2019 12:20:30 +0200

2019-11-04

Peter noticed that there were 5.2 million hits on CGSpace in 2019-10 according to the Atmire usage statistics
- I looked in the nginx logs and see 4.6 million in the access logs, and 1.2 million in the API logs:

# zcat --force /var/log/nginx/*access.log.*.gz | grep -cE "[0-9]{1,2}/Oct/2019"
4671942
# zcat --force /var/log/nginx/{rest,oai,statistics}.log.*.gz | grep -cE "[0-9]{1,2}/Oct/2019"
1277694

So 4.6 million from XMLUI and another 1.2 million from API requests
Let’s see how many of the REST API requests were for bitstreams (because they are counted in Solr stats):

# zcat --force /var/log/nginx/rest.log.*.gz | grep -c -E "[0-9]{1,2}/Oct/2019"
1183456 
# zcat --force /var/log/nginx/rest.log.*.gz | grep -E "[0-9]{1,2}/Oct/2019" | grep -c -E "/rest/bitstreams"
106781

October, 2019

Tue, 01 Oct 2019 13:20:51 +0300

2019-10-01 Udana from IWMI asked me for a CSV export of their community on CGSpace I exported it, but a quick run through the csv-metadata-quality tool shows that there are some low-hanging fruits we can fix before I send him the data I will limit the scope to the titles, regions, subregions, and river basins for now to manually fix some non-breaking spaces (U+00A0) there that would otherwise be removed by the csv-metadata-quality script’s “unneccesary Unicode” fix: $ csvcut -c 'id,dc.

September, 2019

Sun, 01 Sep 2019 10:17:51 +0300

2019-09-01

Linode emailed to say that CGSpace (linode18) had a high rate of outbound traffic for several hours this morning
Here are the top ten IPs in the nginx XMLUI and REST/OAI logs this morning:

# zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep -E "01/Sep/2019:0" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
    440 17.58.101.255
    441 157.55.39.101
    485 207.46.13.43
    728 169.60.128.125
    730 207.46.13.108
    758 157.55.39.9
    808 66.160.140.179
    814 207.46.13.212
   2472 163.172.71.23
   6092 3.94.211.189
# zcat --force /var/log/nginx/rest.log /var/log/nginx/rest.log.1 /var/log/nginx/oai.log /var/log/nginx/oai.log.1 | grep -E "01/Sep/2019:0" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
     33 2a01:7e00::f03c:91ff:fe16:fcb
     57 3.83.192.124
     57 3.87.77.25
     57 54.82.1.8
    822 2a01:9cc0:47:1:1a:4:0:2
   1223 45.5.184.72
   1633 172.104.229.92
   5112 205.186.128.185
   7249 2a01:7e00::f03c:91ff:fe18:7396
   9124 45.5.186.2

August, 2019

Sat, 03 Aug 2019 12:39:51 +0300

2019-08-03

Look at Bioversity’s latest migration CSV and now I see that Francesco has cleaned up the extra columns and the newline at the end of the file, but many of the column headers have an extra space in the name…

2019-08-04

Deploy ORCID identifier updates requested by Bioversity to CGSpace
Run system updates on CGSpace (linode18) and reboot it
- Before updating it I checked Solr and verified that all statistics cores were loaded properly…
- After rebooting, all statistics cores were loaded… wow, that’s lucky.
Run system updates on DSpace Test (linode19) and reboot it

July, 2019

Mon, 01 Jul 2019 12:13:51 +0300

2019-07-01

Create an “AfricaRice books and book chapters” collection on CGSpace for AfricaRice
Last month Sisay asked why the following “most popular” statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace:
- DSpace Test
- CGSpace
Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community

June, 2019

Sun, 02 Jun 2019 10:57:51 +0300

2019-06-02

Merge the Solr filterCache and XMLUI ISI journal changes to the 5_x-prod branch and deploy on CGSpace
Run system updates on CGSpace (linode18) and reboot it

2019-06-03

Skype with Marie-Angélique and Abenet about CG Core v2

May, 2019

Wed, 01 May 2019 07:37:43 +0300

2019-05-01

Help CCAFS with regenerating some item thumbnails after they uploaded new PDFs to some items on CGSpace
A user on the dspace-tech mailing list offered some suggestions for troubleshooting the problem with the inability to delete certain items
- Apparently if the item is in the workflowitem table it is submitted to a workflow
- And if it is in the workspaceitem table it is in the pre-submitted state
The item seems to be in a pre-submitted state, so I tried to delete it from there:

dspace=# DELETE FROM workspaceitem WHERE item_id=74648;
DELETE 1

But after this I tried to delete the item from the XMLUI and it is still present…

April, 2019

Mon, 01 Apr 2019 09:00:43 +0300

2019-04-01

Meeting with AgroKnow to discuss CGSpace, ILRI data, AReS, GARDIAN, etc
- They asked if we had plans to enable RDF support in CGSpace
There have been 4,400 more downloads of the CTA Spore publication from those strange Amazon IP addresses today
- I suspected that some might not be successful, because the stats show less, but today they were all HTTP 200!

# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep 'Spore-192-EN-web.pdf' | grep -E '(18.196.196.108|18.195.78.144|18.195.218.6)' | awk '{print $9}' | sort | uniq -c | sort -n | tail -n 5
   4432 200

In the last two weeks there have been 47,000 downloads of this same exact PDF by these three IP addresses
Apply country and region corrections and deletions on DSpace Test and CGSpace:

$ ./fix-metadata-values.py -i /tmp/2019-02-21-fix-9-countries.csv -db dspace -u dspace -p 'fuuu' -f cg.coverage.country -m 228 -t ACTION -d
$ ./fix-metadata-values.py -i /tmp/2019-02-21-fix-4-regions.csv -db dspace -u dspace -p 'fuuu' -f cg.coverage.region -m 231 -t action -d
$ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-2-countries.csv -db dspace -u dspace -p 'fuuu' -m 228 -f cg.coverage.country -d
$ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace -u dspace -p 'fuuu' -m 231 -f cg.coverage.region -d

March, 2019

Fri, 01 Mar 2019 12:16:30 +0100

2019-03-01

I checked IITA’s 259 Feb 14 records from last month for duplicates using Atmire’s Duplicate Checker on a fresh snapshot of CGSpace on my local machine and everything looks good
I am now only waiting to hear from her about where the items should go, though I assume Journal Articles go to IITA Journal Articles collection, etc…
Looking at the other half of Udana’s WLE records from 2018-11
- I finished the ones for Restoring Degraded Landscapes (RDL), but these are for Variability, Risks and Competing Uses (VRC)
- I did the usual cleanups for whitespace, added regions where they made sense for certain countries, cleaned up the DOI link formats, added rights information based on the publications page for a few items
- Most worryingly, there are encoding errors in the abstracts for eleven items, for example:
- 68.15% � 9.45 instead of 68.15% ± 9.45
- 2003�2013 instead of 2003–2013
I think I will need to ask Udana to re-copy and paste the abstracts with more care using Google Docs

February, 2019

Fri, 01 Feb 2019 21:37:30 +0200

2019-02-01

Linode has alerted a few times since last night that the CPU usage on CGSpace (linode18) was high despite me increasing the alert threshold last week from 250% to 275%—I might need to increase it again!
The top IPs before, during, and after this latest alert tonight were:

# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "01/Feb/2019:(17|18|19|20|21)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
    245 207.46.13.5
    332 54.70.40.11
    385 5.143.231.38
    405 207.46.13.173
    405 207.46.13.75
   1117 66.249.66.219
   1121 35.237.175.180
   1546 5.9.6.51
   2474 45.5.186.2
   5490 85.25.237.71

85.25.237.71 is the “Linguee Bot” that I first saw last month
The Solr statistics the past few months have been very high and I was wondering if the web server logs also showed an increase
There were just over 3 million accesses in the nginx logs last month:

# time zcat --force /var/log/nginx/* | grep -cE "[0-9]{1,2}/Jan/2019"
3018243

real    0m19.873s
user    0m22.203s
sys     0m1.979s

January, 2019

Wed, 02 Jan 2019 09:48:30 +0200

2019-01-02

Linode alerted that CGSpace (linode18) had a higher outbound traffic rate than normal early this morning
I don’t see anything interesting in the web server logs around that time though:

# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "02/Jan/2019:0(1|2|3)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
     92 40.77.167.4
     99 210.7.29.100
    120 38.126.157.45
    177 35.237.175.180
    177 40.77.167.32
    216 66.249.75.219
    225 18.203.76.93
    261 46.101.86.248
    357 207.46.13.1
    903 54.70.40.11

December, 2018

Sun, 02 Dec 2018 02:09:30 +0200

2018-12-01

Switch CGSpace (linode18) to use OpenJDK instead of Oracle JDK
I manually installed OpenJDK, then removed Oracle JDK, then re-ran the Ansible playbook to update all configuration files, etc
Then I ran all system updates and restarted the server

2018-12-02

I noticed that there is another issue with PDF thumbnails on CGSpace, and I see there was another Ghostscript vulnerability last week

November, 2018

Thu, 01 Nov 2018 16:41:30 +0200

2018-11-01

Finalize AReS Phase I and Phase II ToRs
Send a note about my dspace-statistics-api to the dspace-tech mailing list

2018-11-03

Linode has been sending mails a few times a day recently that CGSpace (linode18) has had high CPU usage
Today these are the top 10 IPs:

October, 2018

Mon, 01 Oct 2018 22:31:54 +0300

2018-10-01

Phil Thornton got an ORCID identifier so we need to add it to the list on CGSpace and tag his existing items
I created a GitHub issue to track this #389, because I’m super busy in Nairobi right now

September, 2018

Sun, 02 Sep 2018 09:55:54 +0300

2018-09-02

New PostgreSQL JDBC driver version 42.2.5
I’ll update the DSpace role in our Ansible infrastructure playbooks and run the updated playbooks on CGSpace and DSpace Test
Also, I’ll re-run the postgresql tasks because the custom PostgreSQL variables are dynamic according to the system’s RAM, and we never re-ran them after migrating to larger Linodes last month
I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I’m getting those autowire errors in Tomcat 8.5.30 again:

August, 2018

Wed, 01 Aug 2018 11:52:54 +0300

2018-08-01

DSpace Test had crashed at some point yesterday morning and I see the following in dmesg:

[Tue Jul 31 00:00:41 2018] Out of memory: Kill process 1394 (java) score 668 or sacrifice child
[Tue Jul 31 00:00:41 2018] Killed process 1394 (java) total-vm:15601860kB, anon-rss:5355528kB, file-rss:0kB, shmem-rss:0kB
[Tue Jul 31 00:00:41 2018] oom_reaper: reaped process 1394 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Judging from the time of the crash it was probably related to the Discovery indexing that starts at midnight
From the DSpace log I see that eventually Solr stopped responding, so I guess the java process that was OOM killed above was Tomcat’s
I’m not sure why Tomcat didn’t crash with an OutOfMemoryError…
Anyways, perhaps I should increase the JVM heap from 5120m to 6144m like we did a few months ago when we tried to run the whole CGSpace Solr core
The server only has 8GB of RAM so we’ll eventually need to upgrade to a larger one because we’ll start starving the OS, PostgreSQL, and command line batch processes
I ran all system updates on DSpace Test and rebooted it

July, 2018

Sun, 01 Jul 2018 12:56:54 +0300

2018-07-01

I want to upgrade DSpace Test to DSpace 5.8 so I took a backup of its current database just in case:

$ pg_dump -b -v -o --format=custom -U dspace -f dspace-2018-07-01.backup dspace

During the mvn package stage on the 5.8 branch I kept getting issues with java running out of memory:

There is insufficient memory for the Java Runtime Environment to continue.

June, 2018

Mon, 04 Jun 2018 19:49:54 -0700

2018-06-04

Test the DSpace 5.8 module upgrades from Atmire (#378)
- There seems to be a problem with the CUA and L&R versions in pom.xml because they are using SNAPSHOT and it doesn’t build
I added the new CCAFS Phase II Project Tag PII-FP1_PACCA2 and merged it into the 5_x-prod branch (#379)
I proofed and tested the ILRI author corrections that Peter sent back to me this week:

$ ./fix-metadata-values.py -i /tmp/2018-05-30-Correct-660-authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t correct -m 3 -n

I think a sane proofing workflow in OpenRefine is to apply the custom text facets for check/delete/remove and illegal characters that I developed in March, 2018
Time to index ~70,000 items on CGSpace:

$ time schedtool -D -e ionice -c2 -n7 nice -n19 [dspace]/bin/dspace index-discovery -b                                  

real    74m42.646s
user    8m5.056s
sys     2m7.289s

May, 2018

Tue, 01 May 2018 16:43:54 +0300

2018-05-01

I cleared the Solr statistics core on DSpace Test by issuing two commands directly to the Solr admin interface:
- http://localhost:3000/solr/statistics/update?stream.body=%3Cdelete%3E%3Cquery%3E*:*%3C/query%3E%3C/delete%3E
- http://localhost:3000/solr/statistics/update?stream.body=%3Ccommit/%3E
Then I reduced the JVM heap size from 6144 back to 5120m
Also, I switched it to use OpenJDK instead of Oracle Java, as well as re-worked the Ansible infrastructure scripts to support hosts choosing which distribution they want to use

April, 2018

Sun, 01 Apr 2018 16:13:54 +0200

2018-04-01

I tried to test something on DSpace Test but noticed that it’s down since god knows when
Catalina logs at least show some memory errors yesterday:

March, 2018

Fri, 02 Mar 2018 16:07:54 +0200

2018-03-02

Export a CSV of the IITA community metadata for Martin Mueller

February, 2018

Thu, 01 Feb 2018 16:28:54 +0200

2018-02-01

Peter gave feedback on the dc.rights proof of concept that I had sent him last week
We don’t need to distinguish between internal and external works, so that makes it just a simple list
Yesterday I figured out how to monitor DSpace sessions using JMX
I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu’s munin-plugins-java package and used the stuff I discovered about JMX in 2018-01

January, 2018

Tue, 02 Jan 2018 08:35:54 -0800

2018-01-02

Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time
I didn’t get any load alerts from Linode and the REST and XMLUI logs don’t show anything out of the ordinary
The nginx logs show HTTP 200s until 02/Jan/2018:11:27:17 +0000 when Uptime Robot got an HTTP 500
In dspace.log around that time I see many errors like “Client closed the connection before file download was complete”
And just before that I see this:

Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000].

Ah hah! So the pool was actually empty!
I need to increase that, let’s try to bump it up from 50 to 75
After that one client got an HTTP 499 but then the rest were HTTP 200, so I don’t know what the hell Uptime Robot saw
I notice this error quite a few times in dspace.log:

2018-01-02 01:21:19,137 ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ Error while searching for sidebar facets
org.dspace.discovery.SearchServiceException: org.apache.solr.search.SyntaxError: Cannot parse 'dateIssued_keyword:[1976+TO+1979]': Encountered " "]" "] "" at line 1, column 32.

And there are many of these errors every day for the past month:

$ grep -c "Error while searching for sidebar facets" dspace.log.*
dspace.log.2017-11-21:4
dspace.log.2017-11-22:1
dspace.log.2017-11-23:4
dspace.log.2017-11-24:11
dspace.log.2017-11-25:0
dspace.log.2017-11-26:1
dspace.log.2017-11-27:7
dspace.log.2017-11-28:21
dspace.log.2017-11-29:31
dspace.log.2017-11-30:15
dspace.log.2017-12-01:15
dspace.log.2017-12-02:20
dspace.log.2017-12-03:38
dspace.log.2017-12-04:65
dspace.log.2017-12-05:43
dspace.log.2017-12-06:72
dspace.log.2017-12-07:27
dspace.log.2017-12-08:15
dspace.log.2017-12-09:29
dspace.log.2017-12-10:35
dspace.log.2017-12-11:20
dspace.log.2017-12-12:44
dspace.log.2017-12-13:36
dspace.log.2017-12-14:59
dspace.log.2017-12-15:104
dspace.log.2017-12-16:53
dspace.log.2017-12-17:66
dspace.log.2017-12-18:83
dspace.log.2017-12-19:101
dspace.log.2017-12-20:74
dspace.log.2017-12-21:55
dspace.log.2017-12-22:66
dspace.log.2017-12-23:50
dspace.log.2017-12-24:85
dspace.log.2017-12-25:62
dspace.log.2017-12-26:49
dspace.log.2017-12-27:30
dspace.log.2017-12-28:54
dspace.log.2017-12-29:68
dspace.log.2017-12-30:89
dspace.log.2017-12-31:53
dspace.log.2018-01-01:45
dspace.log.2018-01-02:34

Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let’s Encrypt if it’s just a handful of domains

December, 2017

Fri, 01 Dec 2017 13:53:54 +0300

2017-12-01

Uptime Robot noticed that CGSpace went down
The logs say “Timeout waiting for idle object”
PostgreSQL activity says there are 115 connections currently
The list of connections to XMLUI and REST API for today:

November, 2017

Thu, 02 Nov 2017 09:37:54 +0200

2017-11-01

The CORE developers responded to say they are looking into their bot not respecting our robots.txt

2017-11-02

Today there have been no hits by CORE and no alerts from Linode (coincidence?)

# grep -c "CORE" /var/log/nginx/access.log
0

Generate list of authors on CGSpace for Peter to go through and correct:

dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors.csv with csv;
COPY 54701

October, 2017

Sun, 01 Oct 2017 08:07:54 +0300

2017-10-01

Peter emailed to point out that many items in the ILRI archive collection have multiple handles:

http://hdl.handle.net/10568/78495||http://hdl.handle.net/10568/79336

There appears to be a pattern but I’ll have to look a bit closer and try to clean them up automatically, either in SQL or in OpenRefine
Add Katherine Lutz to the groups for content submission and edit steps of the CGIAR System collections

CGIAR Library Migration

Mon, 18 Sep 2017 16:38:35 +0300

Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called CGIAR System Organization.