Add notes for 2020-02-09

content/posts/2020-02.md

@@ -339,4 +339,49 @@ $ perf-java-flames 11359
 - If the width of the stacks indicates time, then it's clear that Hibernate takes longer...
 - Apparently there is a "flame diff" tool, I wonder if we can use that to compare!
 
+## 2020-02-09
+
+- This weekend I did a lot more testing of indexing performance with our DSpace 5.8 branch, vanilla DSpace 5.10, and vanilla DSpace 6.4-SNAPSHOT:
+
+```
+# CGSpace 5.8
+schedtool -D -e ~/dspace/bin/dspace index-discovery -b  385.72s user 131.16s system 19% cpu 43:21.18 total
+schedtool -D -e ~/dspace/bin/dspace index-discovery -b  382.95s user 127.31s system 20% cpu 42:10.07 total
+schedtool -D -e ~/dspace/bin/dspace index-discovery -b  368.56s user 143.97s system 20% cpu 42:22.66 total
+schedtool -D -e ~/dspace/bin/dspace index-discovery -b  360.09s user 104.03s system 19% cpu 39:24.41 total
+
+# Vanilla DSpace 5.10
+schedtool -D -e ~/dspace510/bin/dspace index-discovery -b  236.19s user 59.70s system 3% cpu 2:03:31.14 total
+schedtool -D -e ~/dspace510/bin/dspace index-discovery -b  232.41s user 50.38s system 3% cpu 2:04:16.00 total
+
+# Vanilla DSpace 6.4-SNAPSHOT
+schedtool -D -e ~/dspace63/bin/dspace index-discovery -b  5112.96s user 127.80s system 40% cpu 3:36:53.98 total
+schedtool -D -e ~/dspace63/bin/dspace index-discovery -b  5112.96s user 127.80s system 40% cpu 3:21:0.0 total
+```
+
+- I generated better flame graphs for the DSpace indexing process by using `perf-record-stack` and filtering out the java process:
+
+```
+$ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk
+$ export PERF_RECORD_SECONDS=60
+$ export JAVA_OPTS="-XX:+PreserveFramePointer"
+$ time schedtool -D -e ~/dspace/bin/dspace index-discovery -b &
+# process id of java indexing process (not Tomcat)
+$ perf-java-record-stack 169639
+$ sudo perf script -i /tmp/perf-169639.data > out.dspace510-1
+$ cat out.dspace510-1 | ../FlameGraph/stackcollapse-perf.pl | grep -E '^java' | ../FlameGraph/flamegraph.pl --color=java --hash > out.dspace510-1.svg
+```
+- All data recorded on my laptop with the same kernel, same boot, etc.
+- CGSpace 5.8 (with Atmire patches):
+
+![DSpace 5.8 (with Atmire modules) index-discovery flame graph](/cgspace-notes/2020/02/out.dspace58-2.svg)
+
+- Vanilla DSpace 5.10:
+
+![Vanilla DSpace 5.10 index-discovery flame graph](/cgspace-notes/2020/02/out.dspace510-3.svg)
+
+- Vanilla DSpace 6.4-SNAPSHOT:
+
+![Vanilla DSpace 6.4-SNAPSHOT index-discovery flame graph](/cgspace-notes/2020/02/out.dspace64-3.svg)
+
 <!-- vim: set sw=2 ts=2: -->

docs/2020-02/index.html

@@ -20,7 +20,7 @@ The code finally builds and runs with a fresh install
 <meta property="og:type" content="article" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2020-02/" />
 <meta property="article:published_time" content="2020-02-02T11:56:30+02:00" />
-<meta property="article:modified_time" content="2020-02-06T16:54:41+02:00" />
+<meta property="article:modified_time" content="2020-02-07T14:44:08+02:00" />
 
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="February, 2020"/>
@@ -45,9 +45,9 @@ The code finally builds and runs with a fresh install
   "@type": "BlogPosting",
   "headline": "February, 2020",
   "url": "https:\/\/alanorth.github.io\/cgspace-notes\/2020-02\/",
-  "wordCount": "2254",
+  "wordCount": "2497",
   "datePublished": "2020-02-02T11:56:30+02:00",
-  "dateModified": "2020-02-06T16:54:41+02:00",
+  "dateModified": "2020-02-07T14:44:08+02:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"
@@ -460,6 +460,47 @@ $ perf-java-flames 11359
 <li>If the width of the stacks indicates time, then it&rsquo;s clear that Hibernate takes longer&hellip;</li>
 <li>Apparently there is a &ldquo;flame diff&rdquo; tool, I wonder if we can use that to compare!</li>
 </ul>
+<h2 id="2020-02-09">2020-02-09</h2>
+<ul>
+<li>This weekend I did a lot more testing of indexing performance with our DSpace 5.8 branch, vanilla DSpace 5.10, and vanilla DSpace 6.4-SNAPSHOT:</li>
+</ul>
+<pre><code># CGSpace 5.8
+schedtool -D -e ~/dspace/bin/dspace index-discovery -b  385.72s user 131.16s system 19% cpu 43:21.18 total
+schedtool -D -e ~/dspace/bin/dspace index-discovery -b  382.95s user 127.31s system 20% cpu 42:10.07 total
+schedtool -D -e ~/dspace/bin/dspace index-discovery -b  368.56s user 143.97s system 20% cpu 42:22.66 total
+schedtool -D -e ~/dspace/bin/dspace index-discovery -b  360.09s user 104.03s system 19% cpu 39:24.41 total
+
+# Vanilla DSpace 5.10
+schedtool -D -e ~/dspace510/bin/dspace index-discovery -b  236.19s user 59.70s system 3% cpu 2:03:31.14 total
+schedtool -D -e ~/dspace510/bin/dspace index-discovery -b  232.41s user 50.38s system 3% cpu 2:04:16.00 total
+
+# Vanilla DSpace 6.4-SNAPSHOT
+schedtool -D -e ~/dspace63/bin/dspace index-discovery -b  5112.96s user 127.80s system 40% cpu 3:36:53.98 total
+schedtool -D -e ~/dspace63/bin/dspace index-discovery -b  5112.96s user 127.80s system 40% cpu 3:21:0.0 total
+</code></pre><ul>
+<li>I generated better flame graphs for the DSpace indexing process by using <code>perf-record-stack</code> and filtering out the java process:</li>
+</ul>
+<pre><code>$ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk
+$ export PERF_RECORD_SECONDS=60
+$ export JAVA_OPTS=&quot;-XX:+PreserveFramePointer&quot;
+$ time schedtool -D -e ~/dspace/bin/dspace index-discovery -b &amp;
+# process id of java indexing process (not Tomcat)
+$ perf-java-record-stack 169639
+$ sudo perf script -i /tmp/perf-169639.data &gt; out.dspace510-1
+$ cat out.dspace510-1 | ../FlameGraph/stackcollapse-perf.pl | grep -E '^java' | ../FlameGraph/flamegraph.pl --color=java --hash &gt; out.dspace510-1.svg
+</code></pre><ul>
+<li>All data recorded on my laptop with the same kernel, same boot, etc.</li>
+<li>CGSpace 5.8 (with Atmire patches):</li>
+</ul>
+<p><img src="/cgspace-notes/2020/02/out.dspace58-2.svg" alt="DSpace 5.8 (with Atmire modules) index-discovery flame graph"></p>
+<ul>
+<li>Vanilla DSpace 5.10:</li>
+</ul>
+<p><img src="/cgspace-notes/2020/02/out.dspace510-3.svg" alt="Vanilla DSpace 5.10 index-discovery flame graph"></p>
+<ul>
+<li>Vanilla DSpace 6.4-SNAPSHOT:</li>
+</ul>
+<p><img src="/cgspace-notes/2020/02/out.dspace64-3.svg" alt="Vanilla DSpace 6.4-SNAPSHOT index-discovery flame graph"></p>
 <!-- raw HTML omitted -->
 
   

docs/sitemap.xml

@@ -4,27 +4,27 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/</loc>
-    <lastmod>2020-02-06T16:54:41+02:00</lastmod>
+    <lastmod>2020-02-07T14:44:08+02:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2020-02-06T16:54:41+02:00</lastmod>
+    <lastmod>2020-02-07T14:44:08+02:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2020-02/</loc>
-    <lastmod>2020-02-06T16:54:41+02:00</lastmod>
+    <lastmod>2020-02-07T14:44:08+02:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
-    <lastmod>2020-02-06T16:54:41+02:00</lastmod>
+    <lastmod>2020-02-07T14:44:08+02:00</lastmod>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
-    <lastmod>2020-02-06T16:54:41+02:00</lastmod>
+    <lastmod>2020-02-07T14:44:08+02:00</lastmod>
   </url>
   
   <url>