Add notes for 2018-01-24

content/post/2018-01.md

@@ -1047,3 +1047,70 @@ $ ./rest-find-collections.py 10568/1 | grep -i untitled
 ```
 
 - I can definitely design a test plan on this!
+
+## 2018-01-24
+
+- Looking at the REST requests, most of them are to expand all or metadata, but 5% are for retrieving bitstreams:
+
+```
+# zcat --force /var/log/nginx/access.log.3.gz /var/log/nginx/access.log.4.gz /var/log/nginx/library-access.log.3.gz /var/log/nginx/library-access.log.4.gz /var/log/nginx/rest.log.3.gz /var/log/nginx/rest.log.4.gz /var/log/nginx/oai.log.3.gz /var/log/nginx/oai.log.4.gz /var/log/nginx/error.log.3.gz /var/log/nginx/error.log.4.gz | grep "21/Jan/2018" | grep "GET " | grep -v "/admin" | awk '{print $7}' | grep -E "^/rest" | grep -Eo "(retrieve|expand=[a-z].*)" | sort | uniq -c | sort -n
+      1 expand=collections
+     16 expand=all&limit=1
+     45 expand=items
+    775 retrieve
+   5675 expand=all
+   8633 expand=metadata
+```
+
+- I finished creating the test plan for DSpace Test and ran it from my Linode with:
+
+```
+$ jmeter -n -t DSpacePerfTest-dspacetest.cgiar.org.jmx -l 2018-01-24-1.jtl
+```
+
+- Atmire responded to [my issue from two weeks ago](https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=560) and said they will start looking into DSpace 5.8 compatibility for CGSpace
+- I set up a new Arch Linux Linode instance with 8192 MB of RAM and ran the test plan a few times to get a baseline:    
+
+```
+# lscpu
+# lscpu 
+Architecture:        x86_64
+CPU op-mode(s):      32-bit, 64-bit
+Byte Order:          Little Endian
+CPU(s):              4
+On-line CPU(s) list: 0-3
+Thread(s) per core:  1
+Core(s) per socket:  1
+Socket(s):           4
+NUMA node(s):        1
+Vendor ID:           GenuineIntel
+CPU family:          6
+Model:               63
+Model name:          Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
+Stepping:            2
+CPU MHz:             2499.994
+BogoMIPS:            5001.32
+Hypervisor vendor:   KVM
+Virtualization type: full
+L1d cache:           32K
+L1i cache:           32K
+L2 cache:            4096K
+L3 cache:            16384K
+NUMA node0 CPU(s):   0-3
+Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault invpcid_single pti retpoline fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
+# free -m
+              total        used        free      shared  buff/cache   available
+Mem:           7970         107        7759           1         103        7771
+Swap:           255           0         255
+# pacman -Syu
+# pacman -S git wget jre8-openjdk-headless mosh htop tmux
+# useradd -m test
+# su - test
+$ git clone -b ilri https://github.com/alanorth/dspace-performance-test.git
+$ wget http://www-us.apache.org/dist//jmeter/binaries/apache-jmeter-3.3.tgz
+$ tar xf apache-jmeter-3.3.tgz
+$ cd apache-jmeter-3.3/bin
+$ ./jmeter -n -t ~/dspace-performance-test/DSpacePerfTest-dspacetest.cgiar.org.jmx -l ~/dspace-performance-test/2018-01-24-linode5451120-baseline.jtl -j ~/dspace-performance-test/2018-01-24-linode5451120-baseline.log
+$ ./jmeter -n -t ~/dspace-performance-test/DSpacePerfTest-dspacetest.cgiar.org.jmx -l ~/dspace-performance-test/2018-01-24-linode5451120-baseline2.jtl -j ~/dspace-performance-test/2018-01-24-linode5451120-baseline2.log
+$ ./jmeter -n -t ~/dspace-performance-test/DSpacePerfTest-dspacetest.cgiar.org.jmx -l ~/dspace-performance-test/2018-01-24-linode5451120-baseline3.jtl -j ~/dspace-performance-test/2018-01-24-linode5451120-baseline3.log
+```

public/2018-01/index.html

@@ -92,7 +92,7 @@ Danny wrote to ask for help renewing the wildcard ilri.org certificate and I adv
 
 <meta property="article:published_time" content="2018-01-02T08:35:54-08:00"/>
 
-<meta property="article:modified_time" content="2018-01-23T00:08:32&#43;02:00"/>
+<meta property="article:modified_time" content="2018-01-23T18:08:57&#43;02:00"/>
 
 
 
@@ -194,9 +194,9 @@ Danny wrote to ask for help renewing the wildcard ilri.org certificate and I adv
   "@type": "BlogPosting",
   "headline": "January, 2018",
   "url": "https://alanorth.github.io/cgspace-notes/2018-01/",
-  "wordCount": "5977",
+  "wordCount": "6355",
   "datePublished": "2018-01-02T08:35:54-08:00",
-  "dateModified": "2018-01-23T00:08:32&#43;02:00",
+  "dateModified": "2018-01-23T18:08:57&#43;02:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"
@@ -1418,6 +1418,76 @@ $ ./rest-find-collections.py 10568/1 | grep -i untitled
 <li>I can definitely design a test plan on this!</li>
 </ul>
 
+<h2 id="2018-01-24">2018-01-24</h2>
+
+<ul>
+<li>Looking at the REST requests, most of them are to expand all or metadata, but 5% are for retrieving bitstreams:</li>
+</ul>
+
+<pre><code># zcat --force /var/log/nginx/access.log.3.gz /var/log/nginx/access.log.4.gz /var/log/nginx/library-access.log.3.gz /var/log/nginx/library-access.log.4.gz /var/log/nginx/rest.log.3.gz /var/log/nginx/rest.log.4.gz /var/log/nginx/oai.log.3.gz /var/log/nginx/oai.log.4.gz /var/log/nginx/error.log.3.gz /var/log/nginx/error.log.4.gz | grep &quot;21/Jan/2018&quot; | grep &quot;GET &quot; | grep -v &quot;/admin&quot; | awk '{print $7}' | grep -E &quot;^/rest&quot; | grep -Eo &quot;(retrieve|expand=[a-z].*)&quot; | sort | uniq -c | sort -n
+      1 expand=collections
+     16 expand=all&amp;limit=1
+     45 expand=items
+    775 retrieve
+   5675 expand=all
+   8633 expand=metadata
+</code></pre>
+
+<ul>
+<li>I finished creating the test plan for DSpace Test and ran it from my Linode with:</li>
+</ul>
+
+<pre><code>$ jmeter -n -t DSpacePerfTest-dspacetest.cgiar.org.jmx -l 2018-01-24-1.jtl
+</code></pre>
+
+<ul>
+<li>Atmire responded to <a href="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=560">my issue from two weeks ago</a> and said they will start looking into DSpace 5.8 compatibility for CGSpace</li>
+<li>I set up a new Arch Linux Linode instance with 8192 MB of RAM and ran the test plan a few times to get a baseline:<br /></li>
+</ul>
+
+<pre><code># lscpu
+# lscpu 
+Architecture:        x86_64
+CPU op-mode(s):      32-bit, 64-bit
+Byte Order:          Little Endian
+CPU(s):              4
+On-line CPU(s) list: 0-3
+Thread(s) per core:  1
+Core(s) per socket:  1
+Socket(s):           4
+NUMA node(s):        1
+Vendor ID:           GenuineIntel
+CPU family:          6
+Model:               63
+Model name:          Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
+Stepping:            2
+CPU MHz:             2499.994
+BogoMIPS:            5001.32
+Hypervisor vendor:   KVM
+Virtualization type: full
+L1d cache:           32K
+L1i cache:           32K
+L2 cache:            4096K
+L3 cache:            16384K
+NUMA node0 CPU(s):   0-3
+Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault invpcid_single pti retpoline fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
+# free -m
+              total        used        free      shared  buff/cache   available
+Mem:           7970         107        7759           1         103        7771
+Swap:           255           0         255
+# pacman -Syu
+# pacman -S git wget jre8-openjdk-headless mosh htop tmux
+# useradd -m test
+# su - test
+$ git clone -b ilri https://github.com/alanorth/dspace-performance-test.git
+$ wget http://www-us.apache.org/dist//jmeter/binaries/apache-jmeter-3.3.tgz
+$ tar xf apache-jmeter-3.3.tgz
+$ cd apache-jmeter-3.3/bin
+$ ./jmeter -n -t ~/dspace-performance-test/DSpacePerfTest-dspacetest.cgiar.org.jmx -l ~/dspace-performance-test/2018-01-24-linode5451120-baseline.jtl -j ~/dspace-performance-test/2018-01-24-linode5451120-baseline.log
+$ ./jmeter -n -t ~/dspace-performance-test/DSpacePerfTest-dspacetest.cgiar.org.jmx -l ~/dspace-performance-test/2018-01-24-linode5451120-baseline2.jtl -j ~/dspace-performance-test/2018-01-24-linode5451120-baseline2.log
+$ ./jmeter -n -t ~/dspace-performance-test/DSpacePerfTest-dspacetest.cgiar.org.jmx -l ~/dspace-performance-test/2018-01-24-linode5451120-baseline3.jtl -j ~/dspace-performance-test/2018-01-24-linode5451120-baseline3.log
+</code></pre>
+
   
 
   

public/sitemap.xml

@@ -4,7 +4,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2018-01/</loc>
-    <lastmod>2018-01-23T00:08:32+02:00</lastmod>
+    <lastmod>2018-01-23T18:08:57+02:00</lastmod>
   </url>
   
   <url>
@@ -144,7 +144,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2018-01-23T00:08:32+02:00</lastmod>
+    <lastmod>2018-01-23T18:08:57+02:00</lastmod>
     <priority>0</priority>
   </url>
   
@@ -155,7 +155,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
-    <lastmod>2018-01-23T00:08:32+02:00</lastmod>
+    <lastmod>2018-01-23T18:08:57+02:00</lastmod>
     <priority>0</priority>
   </url>
   
@@ -167,13 +167,13 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/post/</loc>
-    <lastmod>2018-01-23T00:08:32+02:00</lastmod>
+    <lastmod>2018-01-23T18:08:57+02:00</lastmod>
     <priority>0</priority>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/tags/</loc>
-    <lastmod>2018-01-23T00:08:32+02:00</lastmod>
+    <lastmod>2018-01-23T18:08:57+02:00</lastmod>
     <priority>0</priority>
   </url>