- I added Phil Thornton and Sonal Henson's ORCID identifiers to the controlled vocabulary for `cg.creator.orcid` and then re-generated the names using my [resolve-orcids.py](https://gist.github.com/alanorth/57a88379126d844563c1410bd7b8d12b) script:
- But in super positive news, he says they are using my new [dspace-statistics-api](https://github.com/alanorth/dspace-statistics-api) and it's MUCH faster than using Atmire CUA's internal "restlet" API
- I don't recognize the `138.201.49.199` IP, but it is in Germany (Hetzner) and appears to be paginating over some browse pages and downloading bitstreams:
- I tagged all of Sonal and Phil's items with their ORCID identifiers on CGSpace using my [add-orcid-identifiers.py](https://gist.github.com/alanorth/a49d85cd9c5dea89cddbe809813a7050) script:
- Salem raised an issue that the dspace-statistics-api reports downloads for some items that have no bitstreams (like many limited access items)
- Every item has at least a `LICENSE` bundle, and some have a `THUMBNAIL` bundle, but the indexing code is specifically checking for downloads from the `ORIGINAL` bundle
- [10568/97460](https://cgspace.cgiar.org/handle/10568/97460) (100550): has a thumbnail bitstream
- [10568/96112](https://cgspace.cgiar.org/handle/10568/96112) (96736): has only a LICENSE bitstream
- I see there are other bundles we might need to pay attention to: `TEXT`, `@_LOGO-COLLECTION_@`, `@_LOGO-COMMUNITY_@`, etc...
- On a hunch I dropped the statistics table and re-indexed and now those two items above have no downloads
- So it's fixed, but I'm not sure why!
- Peter wants to know the number of API requests per month, which was about 250,000 in September (exluding statlet requests):
- AgriKnowledge says they're going to add the `dc.identifier.uri` to their item view in November when they update their website software
## 2018-10-10
- Peter noticed that some recently added PDFs don't have thumbnails
- When I tried to force them to be generated I got an error that I've never seen before:
```
$ dspace filter-media -v -f -i 10568/97613
org.im4java.core.InfoException: org.im4java.core.CommandException: org.im4java.core.CommandException: identify: not authorized `/tmp/impdfthumb5039464037201498062.pdf' @ error/constitute.c/ReadImage/412.
```
- I see there was an update to Ubuntu's ImageMagick on 2018-10-05, so maybe something changed or broke?
- I get the same error when forcing `filter-media` to run on DSpace Test too, so it's gotta be an ImageMagic bug
- The ImageMagick version is currently 8:6.8.9.9-7ubuntu5.13, and there is an [Ubuntu Security Notice from 2018-10-04](https://usn.ubuntu.com/3785-1/)
- Wow, someone on [Twitter posted about this breaking his web application](https://twitter.com/rosscampbell/status/1048268966819319808) (and it was retweeted by the ImageMagick acount!)
- I commented out the line that disables PDF thumbnails in `/etc/ImageMagick-6/policy.xml`:
- I emailed DuraSpace to update [our entry in their DSpace registry](https://duraspace.org/registry/entry/4188/?gvid=178) (the data was still on DSpace 3, JSPUI, etc)
- Generate a list of the top 1500 values for `dc.subject` so Sisay can start making a controlled vocabulary for it:
```
dspace=# \COPY (SELECT DISTINCT text_value, count(*) FROM metadatavalue WHERE metadata_field_id = 57 AND resource_type_id = 2 GROUP BY text_value ORDER BY count DESC LIMIT 1500) to /tmp/2018-10-11-top-1500-subject.csv WITH CSV HEADER;
- Give WorldFish advice about Handles because they are talking to some company called KnowledgeArc who recommends they do not use Handles!
- Last week I emailed Altmetric to ask if their software would notice mentions of our Handle in the format "handle:10568/80775" because I noticed that the [Land Portal does this](https://landportal.org/library/resources/handle1056880775/unlocking-farming-potential-bangladesh%E2%80%99-polders)
- Altmetric support responded to say no, but the reason is that Land Portal is doing even more strange stuff by not using `<meta>` tags in their page header, and using "dct:identifier" property instead of "dc:identifier"
- I re-created my local DSpace databse container using [podman](https://github.com/containers/libpod) instead of Docker:
- With a few changes to my local Maven `settings.xml` it is working well
- Generate a list of the top 10,000 authors for Peter Ballantyne to look through:
```
dspace=# \COPY (SELECT DISTINCT text_value, count(*) FROM metadatavalue WHERE metadata_field_id = 3 AND resource_type_id = 2 GROUP BY text_value ORDER BY count DESC LIMIT 10000) to /tmp/2018-10-11-top-10000-authors.csv WITH CSV HEADER;
- CTA uploaded some infographics that are very tall and their thumbnails disrupt the item lists on the front page and in their communities and collections
- I decided to constrain the max height of these to 200px using CSS ([#392](https://github.com/ilri/DSpace/pull/392))
- I will apply these on CGSpace when I do the other updates tomorrow, as well as double check the high scoring ones to see if they are correct in Sisay's author controlled vocabulary
- Merge the authors controlled vocabulary ([#393](https://github.com/ilri/DSpace/pull/393)), usage rights ([#394](https://github.com/ilri/DSpace/pull/394)), and the upstream DSpace 5.x cherry-picks ([#394](https://github.com/ilri/DSpace/pull/395)) into our `5_x-prod` branch
- Switch to new CGIAR LDAP server on CGSpace, as it's been running (at least for authentication) on DSpace Test for the last few weeks, and I think they old one will be deprecated soon (today?)
- Apply Peter's 746 author corrections on CGSpace and DSpace Test using my [fix-metadata-values.py](https://gist.github.com/alanorth/df92cbfb54d762ba21b28f7cd83b6897) script:
- After rebooting the server I noticed that Handles are not resolving, and the `dspace-handle-server` systemd service is not running (or rather, it exited with success)
- Restarting the service with systemd works for a few seconds, then the java process quits
- I suspect that the systemd service type needs to be `forking` rather than `simple`, because the service calls the default DSpace `start-handle-server` shell script, which uses `nohup` and `&` to background the java process
- It would be nice if there was a cleaner way to start the service and then just log to the systemd journal rather than all this hiding and log redirecting
- Email the Landportal.org people to ask if they would consider Dublin Core metadata tags in their page's header, rather than the HTML properties they are using in their body
- Peter pointed out that some thumbnails were still not getting generated
- When I tried to generate them manually I noticed that the path to the CMYK profile had changed because Ubuntu upgraded Ghostscript from 9.18 to 9.25 last week... WTF?
- Looks like I can use `/usr/share/ghostscript/current` instead of `/usr/share/ghostscript/9.25`...
- I limited the tall thumbnails even further to 170px because Peter said CTA's were still too tall at 200px ([#396](https://github.com/ilri/DSpace/pull/396))
- Tomcat on DSpace Test (linode19) has somehow stopped running all the DSpace applications
- I don't see anything in the Catalina logs or `dmesg`, and the Tomcat manager shows XMLUI, REST, OAI, etc all "Running: false"
- Actually, now I remember that yesterday when I deployed the latest changes from git on DSpace Test I noticed a syntax error in one XML file when I was doing the discovery reindexing
- I fixed it so that I could reindex, but I guess the rest of DSpace actually didn't start up...
- Generate a list of the schema on CGSpace so CodeObia can compare with MELSpace:
```
dspace=# \copy (SELECT (CASE when metadata_schema_id=1 THEN 'dc' WHEN metadata_schema_id=2 THEN 'cg' END) AS schema, element, qualifier, scope_note FROM metadatafieldregistry where metadata_schema_id IN (1,2)) TO /tmp/cgspace-schema.csv WITH CSV HEADER;
- Talking to the CodeObia guys about the REST API I started to wonder why it's so slow and how I can quantify it in order to ask the dspace-tech mailing list for help profiling it
- Interestingly, the speed doesn't get better after you request the same thing multiple times–it's consistently bad on both CGSpace and DSpace Test!
```
$ time http --print h 'https://cgspace.cgiar.org/rest/items?expand=metadata,bitstreams,parentCommunityList&limit=100&offset=0'
...
0.35s user 0.06s system 1% cpu 25.133 total
0.31s user 0.04s system 1% cpu 25.223 total
0.27s user 0.06s system 1% cpu 27.858 total
0.20s user 0.05s system 1% cpu 23.838 total
0.30s user 0.05s system 1% cpu 24.301 total
$ time http --print h 'https://dspacetest.cgiar.org/rest/items?expand=metadata,bitstreams,parentCommunityList&limit=100&offset=0'
...
0.22s user 0.03s system 1% cpu 17.248 total
0.23s user 0.02s system 1% cpu 16.856 total
0.23s user 0.04s system 1% cpu 16.460 total
0.24s user 0.04s system 1% cpu 21.043 total
0.22s user 0.04s system 1% cpu 17.132 total
```
- I should note that at this time CGSpace is using Oracle Java and DSpace Test is using OpenJDK (both version 8)
- I wonder if the Java garbage collector is important here, or if there are missing indexes in PostgreSQL?
- I switched DSpace Test to the G1GC garbage collector and tried again and now the results are worse!
```
$ time http --print h 'https://dspacetest.cgiar.org/rest/items?expand=metadata,bitstreams,parentCommunityList&limit=100&offset=0'
...
0.20s user 0.03s system 0% cpu 25.017 total
0.23s user 0.02s system 1% cpu 23.299 total
0.24s user 0.02s system 1% cpu 22.496 total
0.22s user 0.03s system 1% cpu 22.720 total
0.23s user 0.03s system 1% cpu 22.632 total
```
- If I make a request without the expands it is ten time faster:
```
$ time http --print h 'https://dspacetest.cgiar.org/rest/items?limit=100&offset=0'
...
0.20s user 0.03s system 7% cpu 3.098 total
0.22s user 0.03s system 8% cpu 2.896 total
0.21s user 0.05s system 9% cpu 2.787 total
0.23s user 0.02s system 8% cpu 2.896 total
```
- I sent a mail to dspace-tech to ask how to profile this...
- I decided to update most of the existing metadata values that we have in `dc.rights` on CGSpace to be machine readable in SPDX format (with Creative Commons version if it was included)
- Most of the are from Bioversity, and I asked Maria for permission before updating them
- I manually went through and looked at the existing values and updated them in several batches:
```
UPDATE metadatavalue SET text_value='CC-BY-4.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value LIKE '%CC BY %';
UPDATE metadatavalue SET text_value='CC-BY-NC-ND-4.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value LIKE '%4.0%' AND text_value LIKE '%BY-NC-ND%' AND text_value LIKE '%by-nc-nd%';
UPDATE metadatavalue SET text_value='CC-BY-NC-SA-4.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value LIKE '%4.0%' AND text_value LIKE '%BY-NC-SA%' AND text_value LIKE '%by-nc-sa%';
UPDATE metadatavalue SET text_value='CC-BY-3.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value LIKE '%3.0%' AND text_value LIKE '%/by/%';
UPDATE metadatavalue SET text_value='CC-BY-4.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value LIKE '%4.0%' AND text_value LIKE '%/by/%' AND text_value NOT LIKE '%zero%';
UPDATE metadatavalue SET text_value='CC-BY-NC-2.5' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value LIKE
'%/by-nc%' AND text_value LIKE '%2.5%';
UPDATE metadatavalue SET text_value='CC-BY-NC-4.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value LIKE '%/by-nc%' AND text_value LIKE '%4.0%';
UPDATE metadatavalue SET text_value='CC-BY-4.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value LIKE '%4.0%' AND text_value LIKE '%Attribution %' AND text_value NOT LIKE '%zero%';
UPDATE metadatavalue SET text_value='CC-BY-NC-SA-4.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value NOT LIKE '%zero%' AND text_value LIKE '%4.0%' AND text_value LIKE '%Attribution-NonCommercial-ShareAlike%';
UPDATE metadatavalue SET text_value='CC-BY-NC-4.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value LIKE '%4.0%' AND text_value NOT LIKE '%zero%' AND text_value LIKE '%Attribution-NonCommercial %';
UPDATE metadatavalue SET text_value='CC-BY-NC-3.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value LIKE '%3.0%' AND text_value NOT LIKE '%zero%' AND text_value LIKE '%Attribution-NonCommercial %';
UPDATE metadatavalue SET text_value='CC-BY-3.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value LIKE '%3.0%' AND text_value NOT LIKE '%zero%' AND text_value LIKE '%Attribution %';
UPDATE metadatavalue SET text_value='CC-BY-ND-4.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND resource_id=78184;
UPDATE metadatavalue SET text_value='CC-BY' WHERE resource_type_id=2 AND metadata_field_id=53 AND text_value NOT LIKE '%zero%' AND text_value NOT LIKE '%CC0%' AND text_value LIKE '%Attribution %' AND text_value NOT LIKE '%CC-%';
UPDATE metadatavalue SET text_value='CC-BY-NC-4.0' WHERE resource_type_id=2 AND metadata_field_id=53 AND resource_id=78564;
- We also need to re-think the `dc.rights` field in the submission form: we should probably use a popup controlled vocabulary and list the Creative Commons values with version numbers and allow the user to enter their own (like the ORCID identifier field)
- Ask Jane if we can use some of the BDP money to host AReS explorer on a more powerful server
- IWMI sent me a list of new ORCID identifiers for their staff so I combined them with our list, updated the names with my [resolve-orcids.py](https://gist.github.com/alanorth/57a88379126d844563c1410bd7b8d12b) script, and regenerated the controlled vocabulary:
- I also decided to add the ORCID identifiers that MEL had sent us a few months ago...
- One problem I had with the `resolve-orcids.py` script is that one user seems to have disabled their profile data since we last updated:
```
Looking up the names associated with ORCID iD: 0000-0001-7930-5752
Given Names Deactivated Family Name Deactivated: 0000-0001-7930-5752
```
- So I need to handle that situation in the script for sure, but I'm not sure what to do organizationally or ethically, since that user disabled their name! Do we remove him from the list?
- I upgraded PostgreSQL to 9.6 on DSpace Test using Ansible, then had to manually [migrate from 9.5 to 9.6](https://wiki.postgresql.org/wiki/Using_pg_upgrade_on_Ubuntu/Debian):
- 5.9.6.51 is MegaIndex, which I've seen before...
## 2018-10-20
- I was going to try to run Solr in Docker because I learned I can run Docker on Travis-CI (for testing my dspace-statistics-api), but the oldest official Solr images are for 5.5, and DSpace's Solr configuration is for 4.9
- This means our existing Solr configuration doesn't run in Solr 5.5:
- Last month I added "crawl" to the Tomcat Crawler Session Manager Valve's regular expression matching, and it seems to be working for MegaIndex's user agent:
- Post message to Yammer about usage rights (dc.rights)
- Change `build.properties` to use HTTPS for Handles in our [Ansible infrastructure playbooks](https://github.com/ilri/rmg-ansible-public)
- We will still need to do a batch update of the `dc.identifier.uri` and other fields in the database:
```
dspace=# UPDATE metadatavalue SET text_value=replace(text_value, 'http://', 'https://') WHERE resource_type_id=2 AND text_value LIKE 'http://hdl.handle.net%';
```
- While I was doing that I found two items using CGSpace URLs instead of handles in their `dc.identifier.uri` so I corrected those
- I also found several items that had invalid characters or multiple Handles in some related URL field like `cg.link.reference` so I corrected those too
- Improve the usage rights on the submission form by adding a default selection with no value as well as a better hint to look for the CC license on the publisher page or in the PDF ([#398](https://github.com/ilri/DSpace/pull/398))
- I deployed the changes on CGSpace, ran all system updates, and rebooted the server
- Also, I updated all Handles in the database to use HTTPS:
```
dspace=# UPDATE metadatavalue SET text_value=replace(text_value, 'http://', 'https://') WHERE resource_type_id=2 AND text_value LIKE 'http://hdl.handle.net%';
- Skype with Peter about ToRs for the AReS open source work and future plans to develop tools around the DSpace ecosystem
- Help CGSpace users with some issues related to usage rights
## 2018-10-23
- Improve the usage rights (dc.rights) on CGSpace again by adding the long names in the submission form, as well as adding versio 3.0 and Creative Commons Zero (CC0) public domain license ([#399](https://github.com/ilri/DSpace/pull/399))
- Add "usage rights" to the XMLUI item display ([#400](https://github.com/ilri/DSpace/pull/400))
- Re-work the [dspace-statistics-api](https://github.com/alanorth/dspace-statistics-api) to use Python's native json instead of ujson to make it easier to deploy in places where we don't have—or don't want to have—Python headers and a compiler (like containers)
- Re-work the deployment of the API to use systemd's `EnvironmentFile` to read the environment variables instead of `Environment` in the [RMG Ansible infrastructure scripts](https://github.com/ilri/rmg-ansible-public)
- Send Peter and Jane a list of technical ToRs for AReS open source work:
- Basic version of AReS that works with metadata fields present in default DSpace 5.x/6.x (for example author, date issued, type, subjects)
- Ability to harvest multiple repositories
- Configurable list of extra fields to harvest, per repository
- Configurable list of field and value mappings for consistent display/search with multiple repositories
- Configurable list of graphs/blocks to display on homepage
- Optional harvesting of DSpace view/download statistics if dspace-statistics-api is available on repository
- Optional harvesting of Altmetric mentions
- Configurable scheduling of harvesting (daily, weekly, etc)
- High-quality README.md on GitHub with description, requirements, deployment instructions, and license (GPLv3 unless ICARDA has a problem with that)
- Maria asked if we can add publisher (`dc.publisher`) to the advanced search filters, so I created a [GitHub issue](https://github.com/ilri/DSpace/issues/401) to track it
- I forked the [SolrClient library and updated its kazoo dependency to be version 2.5.0](https://github.com/alanorth/SolrClient/tree/kazoo-2.5.0) so we stop getting errors about "async" being a reserved keyword in Python 3.7
- Then I re-generated the `requirements.txt` in the [dspace-statistics-library](https://github.com/alanorth/dspace-statistics-api) and released version 0.5.2
- Then I re-deployed the API on DSpace Test, ran all system updates on the server, and rebooted it
- I tested my hack of depositing to one collection where the default item and bistream READ policies are restricted and then mapping the item to another collection, but the item retains its default policies so Anonymous cannot see them in the mapped collection either
- Perhaps we need to try moving the item and inheriting the target collection's policies?
- I merged the changes for adding publisher (`dc.publisher`) to the advanced search to the `5_x-prod` branch ([#402](https://github.com/ilri/DSpace/pull/402))
- I merged the changes for adding versionless Creative Commons licenses to the submission form to the `5_x-prod` branch ([#403](https://github.com/ilri/DSpace/pull/403))
- Meet with the COPO guys to walk them through the CGSpace submission workflow and discuss CG core, REST API, etc
- I suggested that they look into submitting via the [SWORDv2](https://wiki.duraspace.org/display/DSDOC5x/SWORDv2+Server) protocol because it respects the workflows
- They said that they're not too worried about the hierarchical CG core schema, that they would just flatten metadata like affiliations when depositing to a DSpace repository
- I said that it might be time to engage the DSpace community to add support for more advanced schemas in DSpace 7+ (perhaps partnership with Atmire?)
## 2018-10-31
- More discussion and planning for AReS open sourcing and Amman meeting in 2019-10
- I did some work to clean up and improve the dspace-statistics-api README.md and project structure and [moved it to the ILRI organization on GitHub](https://github.com/ilri/dspace-statistics-api)
- Now the API serves some basic documentation on the root route
- I want to announce it to the dspace-tech mailing list soon
