mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2024-12-23 21:44:30 +01:00
Add notes for 2018-12-06
This commit is contained in:
parent
edc35dcda9
commit
97c4ce04f0
GPG Key ID:
0FB860CC9C45B1B9
@ -308,4 +308,42 @@ $ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.79.71' dspace.log.2018-12-03
|
||||
|
||||
- Discuss RSS issues with IWMI and WLE people
|
||||
|
||||
## 2018-12-06
|
||||
|
||||
- Linode sent a message that the CPU usage of CGSpace (linode18) is too high last night
|
||||
- I looked in the logs and there's nothing particular going on:
|
||||
|
||||
```
|
||||
# zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "05/Dec/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
|
||||
1225 157.55.39.177
|
||||
1240 207.46.13.12
|
||||
1261 207.46.13.101
|
||||
1411 207.46.13.157
|
||||
1529 34.218.226.147
|
||||
2085 50.116.102.77
|
||||
3334 2a01:7e00::f03c:91ff:fe0a:d645
|
||||
3733 66.249.70.27
|
||||
3815 35.237.175.180
|
||||
7669 54.70.40.11
|
||||
```
|
||||
|
||||
- `54.70.40.11` is some new bot with the following user agent:
|
||||
|
||||
```
|
||||
Mozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler)
|
||||
```
|
||||
|
||||
- But Tomcat is forcing them to re-use their Tomcat sessions with the Crawler Session Manager valve:
|
||||
|
||||
```
|
||||
$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=54.70.40.11' dspace.log.2018-12-05
|
||||
6980
|
||||
$ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=54.70.40.11' dspace.log.2018-12-05 | sort | uniq | wc -l
|
||||
1156
|
||||
```
|
||||
|
||||
- `2a01:7e00::f03c:91ff:fe0a:d645` appears to be the CKM dev server where Danny is testing harvesting via Drupal
|
||||
- It seems they are hitting the XMLUI's OpenSearch a bit, but mostly on the REST API so no issues here yet
|
||||
- `Drupal` is already in the Tomcat Crawler Session Manager Valve's regex so that's good!
|
||||
|
||||
<!-- vim: set sw=2 ts=2: -->
|
||||
|
||||
@ -21,7 +21,7 @@ I noticed that there is another issue with PDF thumbnails on CGSpace, and I see
|
||||
" />
|
||||
<meta property="og:type" content="article" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2018-12/" /><meta property="article:published_time" content="2018-12-02T02:09:30+02:00"/>
|
||||
<meta property="article:modified_time" content="2018-12-04T10:02:51+02:00"/>
|
||||
<meta property="article:modified_time" content="2018-12-05T17:20:48+02:00"/>
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="December, 2018"/>
|
||||
@ -48,9 +48,9 @@ I noticed that there is another issue with PDF thumbnails on CGSpace, and I see
|
||||
"@type": "BlogPosting",
|
||||
"headline": "December, 2018",
|
||||
"url": "https://alanorth.github.io/cgspace-notes/2018-12/",
|
||||
"wordCount": "1835",
|
||||
"wordCount": "2009",
|
||||
"datePublished": "2018-12-02T02:09:30+02:00",
|
||||
"dateModified": "2018-12-04T10:02:51+02:00",
|
||||
"dateModified": "2018-12-05T17:20:48+02:00",
|
||||
"author": {
|
||||
"@type": "Person",
|
||||
"name": "Alan Orth"
|
||||
@ -457,6 +457,49 @@ $ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=78.46.79.71' dspace.log.2018-12-03
|
||||
<li>Discuss RSS issues with IWMI and WLE people</li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2018-12-06">2018-12-06</h2>
|
||||
|
||||
<ul>
|
||||
<li>Linode sent a message that the CPU usage of CGSpace (linode18) is too high last night</li>
|
||||
<li>I looked in the logs and there’s nothing particular going on:</li>
|
||||
</ul>
|
||||
|
||||
<pre><code># zcat --force /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "05/Dec/2018" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
|
||||
1225 157.55.39.177
|
||||
1240 207.46.13.12
|
||||
1261 207.46.13.101
|
||||
1411 207.46.13.157
|
||||
1529 34.218.226.147
|
||||
2085 50.116.102.77
|
||||
3334 2a01:7e00::f03c:91ff:fe0a:d645
|
||||
3733 66.249.70.27
|
||||
3815 35.237.175.180
|
||||
7669 54.70.40.11
|
||||
</code></pre>
|
||||
|
||||
<ul>
|
||||
<li><code>54.70.40.11</code> is some new bot with the following user agent:</li>
|
||||
</ul>
|
||||
|
||||
<pre><code>Mozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler)
|
||||
</code></pre>
|
||||
|
||||
<ul>
|
||||
<li>But Tomcat is forcing them to re-use their Tomcat sessions with the Crawler Session Manager valve:</li>
|
||||
</ul>
|
||||
|
||||
<pre><code>$ grep -c -E 'session_id=[A-Z0-9]{32}:ip_addr=54.70.40.11' dspace.log.2018-12-05
|
||||
6980
|
||||
$ grep -o -E 'session_id=[A-Z0-9]{32}:ip_addr=54.70.40.11' dspace.log.2018-12-05 | sort | uniq | wc -l
|
||||
1156
|
||||
</code></pre>
|
||||
|
||||
<ul>
|
||||
<li><code>2a01:7e00::f03c:91ff:fe0a:d645</code> appears to be the CKM dev server where Danny is testing harvesting via Drupal</li>
|
||||
<li>It seems they are hitting the XMLUI’s OpenSearch a bit, but mostly on the REST API so no issues here yet</li>
|
||||
<li><code>Drupal</code> is already in the Tomcat Crawler Session Manager Valve’s regex so that’s good!</li>
|
||||
</ul>
|
||||
|
||||
<!-- vim: set sw=2 ts=2: -->
|
||||
|
||||
|
||||
|
||||
@ -42,7 +42,7 @@ Disallow: /cgspace-notes/2015-12/
|
||||
Disallow: /cgspace-notes/2015-11/
|
||||
Disallow: /cgspace-notes/
|
||||
Disallow: /cgspace-notes/categories/
|
||||
Disallow: /cgspace-notes/categories/notes/
|
||||
Disallow: /cgspace-notes/tags/notes/
|
||||
Disallow: /cgspace-notes/categories/notes/
|
||||
Disallow: /cgspace-notes/posts/
|
||||
Disallow: /cgspace-notes/tags/
|
||||
|
||||
@ -4,7 +4,7 @@
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/2018-12/</loc>
|
||||
<lastmod>2018-12-04T10:02:51+02:00</lastmod>
|
||||
<lastmod>2018-12-05T17:20:48+02:00</lastmod>
|
||||
</url>
|
||||
|
||||
<url>
|
||||
@ -199,7 +199,7 @@
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/</loc>
|
||||
<lastmod>2018-12-04T10:02:51+02:00</lastmod>
|
||||
<lastmod>2018-12-05T17:20:48+02:00</lastmod>
|
||||
<priority>0</priority>
|
||||
</url>
|
||||
|
||||
@ -208,27 +208,27 @@
|
||||
<priority>0</priority>
|
||||
</url>
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
|
||||
<lastmod>2018-12-05T17:20:48+02:00</lastmod>
|
||||
<priority>0</priority>
|
||||
</url>
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
|
||||
<lastmod>2018-03-09T22:10:33+02:00</lastmod>
|
||||
<priority>0</priority>
|
||||
</url>
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
|
||||
<lastmod>2018-12-04T10:02:51+02:00</lastmod>
|
||||
<priority>0</priority>
|
||||
</url>
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
|
||||
<lastmod>2018-12-04T10:02:51+02:00</lastmod>
|
||||
<lastmod>2018-12-05T17:20:48+02:00</lastmod>
|
||||
<priority>0</priority>
|
||||
</url>
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/tags/</loc>
|
||||
<lastmod>2018-12-04T10:02:51+02:00</lastmod>
|
||||
<lastmod>2018-12-05T17:20:48+02:00</lastmod>
|
||||
<priority>0</priority>
|
||||
</url>
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user