diff --git a/content/post/2018-01.md b/content/post/2018-01.md index 5617ebeef..e9d8fcdf6 100644 --- a/content/post/2018-01.md +++ b/content/post/2018-01.md @@ -1047,3 +1047,70 @@ $ ./rest-find-collections.py 10568/1 | grep -i untitled ``` - I can definitely design a test plan on this! + +## 2018-01-24 + +- Looking at the REST requests, most of them are to expand all or metadata, but 5% are for retrieving bitstreams: + +``` +# zcat --force /var/log/nginx/access.log.3.gz /var/log/nginx/access.log.4.gz /var/log/nginx/library-access.log.3.gz /var/log/nginx/library-access.log.4.gz /var/log/nginx/rest.log.3.gz /var/log/nginx/rest.log.4.gz /var/log/nginx/oai.log.3.gz /var/log/nginx/oai.log.4.gz /var/log/nginx/error.log.3.gz /var/log/nginx/error.log.4.gz | grep "21/Jan/2018" | grep "GET " | grep -v "/admin" | awk '{print $7}' | grep -E "^/rest" | grep -Eo "(retrieve|expand=[a-z].*)" | sort | uniq -c | sort -n + 1 expand=collections + 16 expand=all&limit=1 + 45 expand=items + 775 retrieve + 5675 expand=all + 8633 expand=metadata +``` + +- I finished creating the test plan for DSpace Test and ran it from my Linode with: + +``` +$ jmeter -n -t DSpacePerfTest-dspacetest.cgiar.org.jmx -l 2018-01-24-1.jtl +``` + +- Atmire responded to [my issue from two weeks ago](https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=560) and said they will start looking into DSpace 5.8 compatibility for CGSpace +- I set up a new Arch Linux Linode instance with 8192 MB of RAM and ran the test plan a few times to get a baseline: + +``` +# lscpu +# lscpu +Architecture: x86_64 +CPU op-mode(s): 32-bit, 64-bit +Byte Order: Little Endian +CPU(s): 4 +On-line CPU(s) list: 0-3 +Thread(s) per core: 1 +Core(s) per socket: 1 +Socket(s): 4 +NUMA node(s): 1 +Vendor ID: GenuineIntel +CPU family: 6 +Model: 63 +Model name: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz +Stepping: 2 +CPU MHz: 2499.994 +BogoMIPS: 5001.32 +Hypervisor vendor: KVM +Virtualization type: full +L1d cache: 32K +L1i cache: 32K +L2 cache: 4096K +L3 cache: 16384K +NUMA node0 CPU(s): 0-3 +Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault invpcid_single pti retpoline fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat +# free -m + total used free shared buff/cache available +Mem: 7970 107 7759 1 103 7771 +Swap: 255 0 255 +# pacman -Syu +# pacman -S git wget jre8-openjdk-headless mosh htop tmux +# useradd -m test +# su - test +$ git clone -b ilri https://github.com/alanorth/dspace-performance-test.git +$ wget http://www-us.apache.org/dist//jmeter/binaries/apache-jmeter-3.3.tgz +$ tar xf apache-jmeter-3.3.tgz +$ cd apache-jmeter-3.3/bin +$ ./jmeter -n -t ~/dspace-performance-test/DSpacePerfTest-dspacetest.cgiar.org.jmx -l ~/dspace-performance-test/2018-01-24-linode5451120-baseline.jtl -j ~/dspace-performance-test/2018-01-24-linode5451120-baseline.log +$ ./jmeter -n -t ~/dspace-performance-test/DSpacePerfTest-dspacetest.cgiar.org.jmx -l ~/dspace-performance-test/2018-01-24-linode5451120-baseline2.jtl -j ~/dspace-performance-test/2018-01-24-linode5451120-baseline2.log +$ ./jmeter -n -t ~/dspace-performance-test/DSpacePerfTest-dspacetest.cgiar.org.jmx -l ~/dspace-performance-test/2018-01-24-linode5451120-baseline3.jtl -j ~/dspace-performance-test/2018-01-24-linode5451120-baseline3.log +``` diff --git a/public/2018-01/index.html b/public/2018-01/index.html index 169934994..4006dca68 100644 --- a/public/2018-01/index.html +++ b/public/2018-01/index.html @@ -92,7 +92,7 @@ Danny wrote to ask for help renewing the wildcard ilri.org certificate and I adv - + @@ -194,9 +194,9 @@ Danny wrote to ask for help renewing the wildcard ilri.org certificate and I adv "@type": "BlogPosting", "headline": "January, 2018", "url": "https://alanorth.github.io/cgspace-notes/2018-01/", - "wordCount": "5977", + "wordCount": "6355", "datePublished": "2018-01-02T08:35:54-08:00", - "dateModified": "2018-01-23T00:08:32+02:00", + "dateModified": "2018-01-23T18:08:57+02:00", "author": { "@type": "Person", "name": "Alan Orth" @@ -1418,6 +1418,76 @@ $ ./rest-find-collections.py 10568/1 | grep -i untitled
# zcat --force /var/log/nginx/access.log.3.gz /var/log/nginx/access.log.4.gz /var/log/nginx/library-access.log.3.gz /var/log/nginx/library-access.log.4.gz /var/log/nginx/rest.log.3.gz /var/log/nginx/rest.log.4.gz /var/log/nginx/oai.log.3.gz /var/log/nginx/oai.log.4.gz /var/log/nginx/error.log.3.gz /var/log/nginx/error.log.4.gz | grep "21/Jan/2018" | grep "GET " | grep -v "/admin" | awk '{print $7}' | grep -E "^/rest" | grep -Eo "(retrieve|expand=[a-z].*)" | sort | uniq -c | sort -n
+ 1 expand=collections
+ 16 expand=all&limit=1
+ 45 expand=items
+ 775 retrieve
+ 5675 expand=all
+ 8633 expand=metadata
+
+
+$ jmeter -n -t DSpacePerfTest-dspacetest.cgiar.org.jmx -l 2018-01-24-1.jtl
+
+
+# lscpu
+# lscpu
+Architecture: x86_64
+CPU op-mode(s): 32-bit, 64-bit
+Byte Order: Little Endian
+CPU(s): 4
+On-line CPU(s) list: 0-3
+Thread(s) per core: 1
+Core(s) per socket: 1
+Socket(s): 4
+NUMA node(s): 1
+Vendor ID: GenuineIntel
+CPU family: 6
+Model: 63
+Model name: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
+Stepping: 2
+CPU MHz: 2499.994
+BogoMIPS: 5001.32
+Hypervisor vendor: KVM
+Virtualization type: full
+L1d cache: 32K
+L1i cache: 32K
+L2 cache: 4096K
+L3 cache: 16384K
+NUMA node0 CPU(s): 0-3
+Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault invpcid_single pti retpoline fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
+# free -m
+ total used free shared buff/cache available
+Mem: 7970 107 7759 1 103 7771
+Swap: 255 0 255
+# pacman -Syu
+# pacman -S git wget jre8-openjdk-headless mosh htop tmux
+# useradd -m test
+# su - test
+$ git clone -b ilri https://github.com/alanorth/dspace-performance-test.git
+$ wget http://www-us.apache.org/dist//jmeter/binaries/apache-jmeter-3.3.tgz
+$ tar xf apache-jmeter-3.3.tgz
+$ cd apache-jmeter-3.3/bin
+$ ./jmeter -n -t ~/dspace-performance-test/DSpacePerfTest-dspacetest.cgiar.org.jmx -l ~/dspace-performance-test/2018-01-24-linode5451120-baseline.jtl -j ~/dspace-performance-test/2018-01-24-linode5451120-baseline.log
+$ ./jmeter -n -t ~/dspace-performance-test/DSpacePerfTest-dspacetest.cgiar.org.jmx -l ~/dspace-performance-test/2018-01-24-linode5451120-baseline2.jtl -j ~/dspace-performance-test/2018-01-24-linode5451120-baseline2.log
+$ ./jmeter -n -t ~/dspace-performance-test/DSpacePerfTest-dspacetest.cgiar.org.jmx -l ~/dspace-performance-test/2018-01-24-linode5451120-baseline3.jtl -j ~/dspace-performance-test/2018-01-24-linode5451120-baseline3.log
+
+
diff --git a/public/sitemap.xml b/public/sitemap.xml
index d037301a4..69d728ae8 100644
--- a/public/sitemap.xml
+++ b/public/sitemap.xml
@@ -4,7 +4,7 @@