Update notes for 2018-10-17

content/posts/2018-10.md

@@ -298,4 +298,52 @@ $ psql -h localhost -U postgres dspacetest -c 'alter user dspacetest nosuperuser
 dspace=# \copy (SELECT (CASE when metadata_schema_id=1 THEN 'dc' WHEN metadata_schema_id=2 THEN 'cg' END) AS schema, element, qualifier, scope_note FROM metadatafieldregistry where metadata_schema_id IN (1,2)) TO /tmp/cgspace-schema.csv WITH CSV HEADER;
 ```
 
+- Talking to the CodeObia guys about the REST API I started to wonder why it's so slow and how I can quantify it in order to ask the dspace-tech mailing list for help profiling it
+- Interestingly, the speed doesn't get better after you request the same thing multiple times–it's consistently bad on both CGSpace and DSpace Test!
+
+```
+$ time http --print h 'https://cgspace.cgiar.org/rest/items?expand=metadata,bitstreams,parentCommunityList&limit=100&offset=0'
+...
+0.35s user 0.06s system 1% cpu 25.133 total
+0.31s user 0.04s system 1% cpu 25.223 total
+0.27s user 0.06s system 1% cpu 27.858 total
+0.20s user 0.05s system 1% cpu 23.838 total
+0.30s user 0.05s system 1% cpu 24.301 total
+
+$ time http --print h 'https://dspacetest.cgiar.org/rest/items?expand=metadata,bitstreams,parentCommunityList&limit=100&offset=0'
+...
+0.22s user 0.03s system 1% cpu 17.248 total
+0.23s user 0.02s system 1% cpu 16.856 total
+0.23s user 0.04s system 1% cpu 16.460 total
+0.24s user 0.04s system 1% cpu 21.043 total
+0.22s user 0.04s system 1% cpu 17.132 total
+```
+
+- I should note that at this time CGSpace is using Oracle Java and DSpace Test is using OpenJDK (both version 8)
+- I wonder if the Java garbage collector is important here, or if there are missing indexes in PostgreSQL?
+- I switched DSpace Test to the G1GC garbage collector and tried again and now the results are worse!
+
+```
+$ time http --print h 'https://dspacetest.cgiar.org/rest/items?expand=metadata,bitstreams,parentCommunityList&limit=100&offset=0'
+...
+0.20s user 0.03s system 0% cpu 25.017 total
+0.23s user 0.02s system 1% cpu 23.299 total
+0.24s user 0.02s system 1% cpu 22.496 total
+0.22s user 0.03s system 1% cpu 22.720 total
+0.23s user 0.03s system 1% cpu 22.632 total
+```
+
+- If I make a request without the expands it is ten time faster:
+
+```
+$ time http --print h 'https://dspacetest.cgiar.org/rest/items?limit=100&offset=0'
+...
+0.20s user 0.03s system 7% cpu 3.098 total
+0.22s user 0.03s system 8% cpu 2.896 total
+0.21s user 0.05s system 9% cpu 2.787 total
+0.23s user 0.02s system 8% cpu 2.896 total
+```
+
+- I sent a mail to dspace-tech to ask how to profile this...
+
 <!-- vim: set sw=2 ts=2: -->