mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2025-01-27 05:49:12 +01:00
Add notes for 2020-09-03
This commit is contained in:
@ -9,7 +9,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
|
||||
<meta property="og:updated_time" content="2020-08-22T13:29:08+03:00" />
|
||||
<meta property="og:updated_time" content="2020-09-02T13:39:11+03:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="Notes"/>
|
||||
@ -80,6 +80,44 @@
|
||||
|
||||
|
||||
|
||||
<article class="blog-post">
|
||||
<header>
|
||||
<h2 class="blog-post-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/2019-11/">November, 2019</a></h2>
|
||||
<p class="blog-post-meta"><time datetime="2019-11-04T12:20:30+02:00">Mon Nov 04, 2019</time> by Alan Orth in
|
||||
<span class="fas fa-folder" aria-hidden="true"></span> <a href="/cgspace-notes/categories/notes/" rel="category tag">Notes</a>
|
||||
|
||||
|
||||
</p>
|
||||
</header>
|
||||
<h2 id="2019-11-04">2019-11-04</h2>
|
||||
<ul>
|
||||
<li>Peter noticed that there were 5.2 million hits on CGSpace in 2019-10 according to the Atmire usage statistics
|
||||
<ul>
|
||||
<li>I looked in the nginx logs and see 4.6 million in the access logs, and 1.2 million in the API logs:</li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<pre><code># zcat --force /var/log/nginx/*access.log.*.gz | grep -cE "[0-9]{1,2}/Oct/2019"
|
||||
4671942
|
||||
# zcat --force /var/log/nginx/{rest,oai,statistics}.log.*.gz | grep -cE "[0-9]{1,2}/Oct/2019"
|
||||
1277694
|
||||
</code></pre><ul>
|
||||
<li>So 4.6 million from XMLUI and another 1.2 million from API requests</li>
|
||||
<li>Let’s see how many of the REST API requests were for bitstreams (because they are counted in Solr stats):</li>
|
||||
</ul>
|
||||
<pre><code># zcat --force /var/log/nginx/rest.log.*.gz | grep -c -E "[0-9]{1,2}/Oct/2019"
|
||||
1183456
|
||||
# zcat --force /var/log/nginx/rest.log.*.gz | grep -E "[0-9]{1,2}/Oct/2019" | grep -c -E "/rest/bitstreams"
|
||||
106781
|
||||
</code></pre>
|
||||
<a href='https://alanorth.github.io/cgspace-notes/2019-11/'>Read more →</a>
|
||||
</article>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<article class="blog-post">
|
||||
<header>
|
||||
<h2 class="blog-post-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/cgspace-cgcorev2-migration/">CGSpace CG Core v2 Migration</a></h2>
|
||||
@ -355,51 +393,6 @@ $ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace
|
||||
|
||||
|
||||
|
||||
|
||||
<article class="blog-post">
|
||||
<header>
|
||||
<h2 class="blog-post-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/2019-02/">February, 2019</a></h2>
|
||||
<p class="blog-post-meta"><time datetime="2019-02-01T21:37:30+02:00">Fri Feb 01, 2019</time> by Alan Orth in
|
||||
<span class="fas fa-folder" aria-hidden="true"></span> <a href="/cgspace-notes/categories/notes/" rel="category tag">Notes</a>
|
||||
|
||||
|
||||
</p>
|
||||
</header>
|
||||
<h2 id="2019-02-01">2019-02-01</h2>
|
||||
<ul>
|
||||
<li>Linode has alerted a few times since last night that the CPU usage on CGSpace (linode18) was high despite me increasing the alert threshold last week from 250% to 275%—I might need to increase it again!</li>
|
||||
<li>The top IPs before, during, and after this latest alert tonight were:</li>
|
||||
</ul>
|
||||
<pre><code># zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "01/Feb/2019:(17|18|19|20|21)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
|
||||
245 207.46.13.5
|
||||
332 54.70.40.11
|
||||
385 5.143.231.38
|
||||
405 207.46.13.173
|
||||
405 207.46.13.75
|
||||
1117 66.249.66.219
|
||||
1121 35.237.175.180
|
||||
1546 5.9.6.51
|
||||
2474 45.5.186.2
|
||||
5490 85.25.237.71
|
||||
</code></pre><ul>
|
||||
<li><code>85.25.237.71</code> is the “Linguee Bot” that I first saw last month</li>
|
||||
<li>The Solr statistics the past few months have been very high and I was wondering if the web server logs also showed an increase</li>
|
||||
<li>There were just over 3 million accesses in the nginx logs last month:</li>
|
||||
</ul>
|
||||
<pre><code># time zcat --force /var/log/nginx/* | grep -cE "[0-9]{1,2}/Jan/2019"
|
||||
3018243
|
||||
|
||||
real 0m19.873s
|
||||
user 0m22.203s
|
||||
sys 0m1.979s
|
||||
</code></pre>
|
||||
<a href='https://alanorth.github.io/cgspace-notes/2019-02/'>Read more →</a>
|
||||
</article>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<nav class="blog-pagination">
|
||||
|
||||
<a class="btn btn-outline-primary" href="/cgspace-notes/categories/notes/" rel="prev" role="button">Previous page</a>
|
||||
@ -424,6 +417,8 @@ sys 0m1.979s
|
||||
<ol class="list-unstyled">
|
||||
|
||||
|
||||
<li><a href="/cgspace-notes/2020-09/">September, 2020</a></li>
|
||||
|
||||
<li><a href="/cgspace-notes/2020-08/">August, 2020</a></li>
|
||||
|
||||
<li><a href="/cgspace-notes/2020-07/">July, 2020</a></li>
|
||||
@ -432,8 +427,6 @@ sys 0m1.979s
|
||||
|
||||
<li><a href="/cgspace-notes/2020-05/">May, 2020</a></li>
|
||||
|
||||
<li><a href="/cgspace-notes/2020-04/">April, 2020</a></li>
|
||||
|
||||
</ol>
|
||||
</section>
|
||||
|
||||
|
||||
@ -9,7 +9,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
|
||||
<meta property="og:updated_time" content="2020-08-22T13:29:08+03:00" />
|
||||
<meta property="og:updated_time" content="2020-09-02T13:39:11+03:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="Notes"/>
|
||||
@ -80,6 +80,51 @@
|
||||
|
||||
|
||||
|
||||
<article class="blog-post">
|
||||
<header>
|
||||
<h2 class="blog-post-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/2019-02/">February, 2019</a></h2>
|
||||
<p class="blog-post-meta"><time datetime="2019-02-01T21:37:30+02:00">Fri Feb 01, 2019</time> by Alan Orth in
|
||||
<span class="fas fa-folder" aria-hidden="true"></span> <a href="/cgspace-notes/categories/notes/" rel="category tag">Notes</a>
|
||||
|
||||
|
||||
</p>
|
||||
</header>
|
||||
<h2 id="2019-02-01">2019-02-01</h2>
|
||||
<ul>
|
||||
<li>Linode has alerted a few times since last night that the CPU usage on CGSpace (linode18) was high despite me increasing the alert threshold last week from 250% to 275%—I might need to increase it again!</li>
|
||||
<li>The top IPs before, during, and after this latest alert tonight were:</li>
|
||||
</ul>
|
||||
<pre><code># zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "01/Feb/2019:(17|18|19|20|21)" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
|
||||
245 207.46.13.5
|
||||
332 54.70.40.11
|
||||
385 5.143.231.38
|
||||
405 207.46.13.173
|
||||
405 207.46.13.75
|
||||
1117 66.249.66.219
|
||||
1121 35.237.175.180
|
||||
1546 5.9.6.51
|
||||
2474 45.5.186.2
|
||||
5490 85.25.237.71
|
||||
</code></pre><ul>
|
||||
<li><code>85.25.237.71</code> is the “Linguee Bot” that I first saw last month</li>
|
||||
<li>The Solr statistics the past few months have been very high and I was wondering if the web server logs also showed an increase</li>
|
||||
<li>There were just over 3 million accesses in the nginx logs last month:</li>
|
||||
</ul>
|
||||
<pre><code># time zcat --force /var/log/nginx/* | grep -cE "[0-9]{1,2}/Jan/2019"
|
||||
3018243
|
||||
|
||||
real 0m19.873s
|
||||
user 0m22.203s
|
||||
sys 0m1.979s
|
||||
</code></pre>
|
||||
<a href='https://alanorth.github.io/cgspace-notes/2019-02/'>Read more →</a>
|
||||
</article>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<article class="blog-post">
|
||||
<header>
|
||||
<h2 class="blog-post-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/2019-01/">January, 2019</a></h2>
|
||||
@ -338,28 +383,6 @@ sys 2m7.289s
|
||||
|
||||
|
||||
|
||||
|
||||
<article class="blog-post">
|
||||
<header>
|
||||
<h2 class="blog-post-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/2018-04/">April, 2018</a></h2>
|
||||
<p class="blog-post-meta"><time datetime="2018-04-01T16:13:54+02:00">Sun Apr 01, 2018</time> by Alan Orth in
|
||||
<span class="fas fa-folder" aria-hidden="true"></span> <a href="/cgspace-notes/categories/notes/" rel="category tag">Notes</a>
|
||||
|
||||
|
||||
</p>
|
||||
</header>
|
||||
<h2 id="2018-04-01">2018-04-01</h2>
|
||||
<ul>
|
||||
<li>I tried to test something on DSpace Test but noticed that it’s down since god knows when</li>
|
||||
<li>Catalina logs at least show some memory errors yesterday:</li>
|
||||
</ul>
|
||||
<a href='https://alanorth.github.io/cgspace-notes/2018-04/'>Read more →</a>
|
||||
</article>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<nav class="blog-pagination">
|
||||
|
||||
<a class="btn btn-outline-primary" href="/cgspace-notes/categories/notes/page/2/" rel="prev" role="button">Previous page</a>
|
||||
@ -384,6 +407,8 @@ sys 2m7.289s
|
||||
<ol class="list-unstyled">
|
||||
|
||||
|
||||
<li><a href="/cgspace-notes/2020-09/">September, 2020</a></li>
|
||||
|
||||
<li><a href="/cgspace-notes/2020-08/">August, 2020</a></li>
|
||||
|
||||
<li><a href="/cgspace-notes/2020-07/">July, 2020</a></li>
|
||||
@ -392,8 +417,6 @@ sys 2m7.289s
|
||||
|
||||
<li><a href="/cgspace-notes/2020-05/">May, 2020</a></li>
|
||||
|
||||
<li><a href="/cgspace-notes/2020-04/">April, 2020</a></li>
|
||||
|
||||
</ol>
|
||||
</section>
|
||||
|
||||
|
||||
@ -9,7 +9,7 @@
|
||||
<meta property="og:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository." />
|
||||
<meta property="og:type" content="website" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/categories/notes/" />
|
||||
<meta property="og:updated_time" content="2020-08-22T13:29:08+03:00" />
|
||||
<meta property="og:updated_time" content="2020-09-02T13:39:11+03:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="Notes"/>
|
||||
@ -80,6 +80,28 @@
|
||||
|
||||
|
||||
|
||||
<article class="blog-post">
|
||||
<header>
|
||||
<h2 class="blog-post-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/2018-04/">April, 2018</a></h2>
|
||||
<p class="blog-post-meta"><time datetime="2018-04-01T16:13:54+02:00">Sun Apr 01, 2018</time> by Alan Orth in
|
||||
<span class="fas fa-folder" aria-hidden="true"></span> <a href="/cgspace-notes/categories/notes/" rel="category tag">Notes</a>
|
||||
|
||||
|
||||
</p>
|
||||
</header>
|
||||
<h2 id="2018-04-01">2018-04-01</h2>
|
||||
<ul>
|
||||
<li>I tried to test something on DSpace Test but noticed that it’s down since god knows when</li>
|
||||
<li>Catalina logs at least show some memory errors yesterday:</li>
|
||||
</ul>
|
||||
<a href='https://alanorth.github.io/cgspace-notes/2018-04/'>Read more →</a>
|
||||
</article>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<article class="blog-post">
|
||||
<header>
|
||||
<h2 class="blog-post-title" dir="auto"><a href="https://alanorth.github.io/cgspace-notes/2018-03/">March, 2018</a></h2>
|
||||
@ -335,6 +357,8 @@ COPY 54701
|
||||
<ol class="list-unstyled">
|
||||
|
||||
|
||||
<li><a href="/cgspace-notes/2020-09/">September, 2020</a></li>
|
||||
|
||||
<li><a href="/cgspace-notes/2020-08/">August, 2020</a></li>
|
||||
|
||||
<li><a href="/cgspace-notes/2020-07/">July, 2020</a></li>
|
||||
@ -343,8 +367,6 @@ COPY 54701
|
||||
|
||||
<li><a href="/cgspace-notes/2020-05/">May, 2020</a></li>
|
||||
|
||||
<li><a href="/cgspace-notes/2020-04/">April, 2020</a></li>
|
||||
|
||||
</ol>
|
||||
</section>
|
||||
|
||||
|
||||
Reference in New Issue
Block a user