mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2024-11-22 14:45:03 +01:00
Add notes for 2018-01-03
This commit is contained in:
parent
4a7805f50f
commit
17a70cb502
@ -79,3 +79,109 @@ dspace.log.2018-01-02:34
|
|||||||
- Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let's Encrypt if it's just a handful of domains
|
- Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let's Encrypt if it's just a handful of domains
|
||||||
|
|
||||||
<!--more-->
|
<!--more-->
|
||||||
|
|
||||||
|
## 2018-01-03
|
||||||
|
|
||||||
|
- I woke up to more up and down of CGSpace, this time UptimeRobot noticed a few rounds of up and down of a few minutes each and Linode also notified of high CPU load from 12 to 2 PM
|
||||||
|
- Looks like I need to increase the database pool size again:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ grep -c "Timeout: Pool empty." dspace.log.2018-01-*
|
||||||
|
dspace.log.2018-01-01:0
|
||||||
|
dspace.log.2018-01-02:1972
|
||||||
|
dspace.log.2018-01-03:1909
|
||||||
|
```
|
||||||
|
|
||||||
|
- For some reason there were a lot of "active" connections last night:
|
||||||
|
|
||||||
|
![CGSpace PostgreSQL connections](/cgspace-notes/2018/01/postgres_connections-day.png)
|
||||||
|
|
||||||
|
- The active IPs in XMLUI are:
|
||||||
|
|
||||||
|
```
|
||||||
|
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -E "3/Jan/2018" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
|
||||||
|
607 40.77.167.141
|
||||||
|
611 2a00:23c3:8c94:7800:392c:a491:e796:9c50
|
||||||
|
663 188.226.169.37
|
||||||
|
759 157.55.39.245
|
||||||
|
887 68.180.229.254
|
||||||
|
1037 157.55.39.175
|
||||||
|
1068 216.244.66.245
|
||||||
|
1495 66.249.64.91
|
||||||
|
1934 104.196.152.243
|
||||||
|
2219 134.155.96.78
|
||||||
|
```
|
||||||
|
|
||||||
|
- 134.155.96.78 appears to be at the University of Mannheim in Germany
|
||||||
|
- They identify as: Mozilla/5.0 (compatible; heritrix/3.2.0 +http://ifm.uni-mannheim.de)
|
||||||
|
- This appears to be the [Internet Archive's open source bot](https://github.com/internetarchive/heritrix3)
|
||||||
|
- They seem to be re-using their Tomcat session so I don't need to do anything to them just yet:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ grep 134.155.96.78 dspace.log.2018-01-03 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
|
||||||
|
2
|
||||||
|
```
|
||||||
|
|
||||||
|
- The API logs show the normal users:
|
||||||
|
|
||||||
|
```
|
||||||
|
# cat /var/log/nginx/rest.log /var/log/nginx/rest.log.1 /var/log/nginx/oai.log /var/log/nginx/oai.log.1 | grep -E "3/Jan/2018" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
|
||||||
|
32 207.46.13.182
|
||||||
|
38 40.77.167.132
|
||||||
|
38 68.180.229.254
|
||||||
|
43 66.249.64.91
|
||||||
|
46 40.77.167.141
|
||||||
|
49 157.55.39.245
|
||||||
|
79 157.55.39.175
|
||||||
|
1533 50.116.102.77
|
||||||
|
4069 70.32.83.92
|
||||||
|
9355 45.5.184.196
|
||||||
|
```
|
||||||
|
|
||||||
|
- In other related news I see a sizeable amount of requests coming from python-requests
|
||||||
|
- For example, just in the last day there were 1700!
|
||||||
|
|
||||||
|
```
|
||||||
|
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -c python-requests
|
||||||
|
1773
|
||||||
|
```
|
||||||
|
|
||||||
|
- But they come from hundreds of IPs, many of which are 54.x.x.x:
|
||||||
|
|
||||||
|
```
|
||||||
|
# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep python-requests | awk '{print $1}' | sort -n | uniq -c | sort -h | tail -n 30
|
||||||
|
9 54.144.87.92
|
||||||
|
9 54.146.222.143
|
||||||
|
9 54.146.249.249
|
||||||
|
9 54.158.139.206
|
||||||
|
9 54.161.235.224
|
||||||
|
9 54.163.41.19
|
||||||
|
9 54.163.4.51
|
||||||
|
9 54.196.195.107
|
||||||
|
9 54.198.89.134
|
||||||
|
9 54.80.158.113
|
||||||
|
10 54.198.171.98
|
||||||
|
10 54.224.53.185
|
||||||
|
10 54.226.55.207
|
||||||
|
10 54.227.8.195
|
||||||
|
10 54.242.234.189
|
||||||
|
10 54.242.238.209
|
||||||
|
10 54.80.100.66
|
||||||
|
11 54.161.243.121
|
||||||
|
11 54.205.154.178
|
||||||
|
11 54.234.225.84
|
||||||
|
11 54.87.23.173
|
||||||
|
11 54.90.206.30
|
||||||
|
12 54.196.127.62
|
||||||
|
12 54.224.242.208
|
||||||
|
12 54.226.199.163
|
||||||
|
13 54.162.149.249
|
||||||
|
13 54.211.182.255
|
||||||
|
19 50.17.61.150
|
||||||
|
21 54.211.119.107
|
||||||
|
139 164.39.7.62
|
||||||
|
```
|
||||||
|
|
||||||
|
- I have no idea what these are but they seem to be coming from Amazon...
|
||||||
|
- I guess for now I just have to increase the database connection pool's max active
|
||||||
|
- It's currently 75 and normally I'd just bump it by 25 but let me be a bit daring and push it by 50 to 125, because I used to see at least 121 connections in pg_stat_activity before when we were using the shitty default pooling
|
||||||
|
@ -52,7 +52,7 @@ $ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspac
|
|||||||
|
|
||||||
|
|
||||||
"/>
|
"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -54,7 +54,7 @@ Replace lzop with xz in log compression cron jobs on DSpace Test—it uses less
|
|||||||
|
|
||||||
|
|
||||||
"/>
|
"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -44,7 +44,7 @@ Update GitHub wiki for documentation of maintenance tasks.
|
|||||||
|
|
||||||
|
|
||||||
"/>
|
"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -58,7 +58,7 @@ Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE&r
|
|||||||
|
|
||||||
|
|
||||||
"/>
|
"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -44,7 +44,7 @@ Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Ja
|
|||||||
|
|
||||||
|
|
||||||
"/>
|
"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -48,7 +48,7 @@ Also, I noticed the checker log has some errors we should pay attention to:
|
|||||||
|
|
||||||
|
|
||||||
"/>
|
"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -52,7 +52,7 @@ There are 3,000 IPs accessing the REST API in a 24-hour period!
|
|||||||
|
|
||||||
|
|
||||||
"/>
|
"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -50,7 +50,7 @@ Working on second phase of metadata migration, looks like this will work for mov
|
|||||||
|
|
||||||
|
|
||||||
"/>
|
"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -66,7 +66,7 @@ In this case the select query was showing 95 results before the update
|
|||||||
|
|
||||||
|
|
||||||
"/>
|
"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -60,7 +60,7 @@ $ git rebase -i dspace-5.5
|
|||||||
|
|
||||||
|
|
||||||
"/>
|
"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -52,7 +52,7 @@ $ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=or
|
|||||||
|
|
||||||
|
|
||||||
"/>
|
"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -60,7 +60,7 @@ I exported a random item’s metadata as CSV, deleted all columns except id
|
|||||||
|
|
||||||
|
|
||||||
"/>
|
"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -44,7 +44,7 @@ Add dc.type to the output options for Atmire’s Listings and Reports module
|
|||||||
|
|
||||||
|
|
||||||
"/>
|
"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -68,7 +68,7 @@ Another worrying error from dspace.log is:
|
|||||||
|
|
||||||
|
|
||||||
"/>
|
"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -44,7 +44,7 @@ I asked on the dspace-tech mailing list because it seems to be broken, and actua
|
|||||||
|
|
||||||
|
|
||||||
"/>
|
"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -72,7 +72,7 @@ Looks like we’ll be using cg.identifier.ccafsprojectpii as the field name
|
|||||||
|
|
||||||
|
|
||||||
"/>
|
"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -76,7 +76,7 @@ $ identify ~/Desktop/alc_contrastes_desafios.jpg
|
|||||||
|
|
||||||
|
|
||||||
"/>
|
"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -62,7 +62,7 @@ $ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Th
|
|||||||
|
|
||||||
|
|
||||||
"/>
|
"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -28,7 +28,7 @@
|
|||||||
|
|
||||||
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="May, 2017"/>
|
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="May, 2017"/>
|
||||||
<meta name="twitter:description" content="2017-05-01 ICARDA apparently started working on CG Core on their MEL repository They have done a few cg.* fields, but not very consistent and even copy some of CGSpace items: https://mel.cgiar.org/xmlui/handle/20.500.11766/6911?show=full https://cgspace.cgiar.org/handle/10568/73683 2017-05-02 Atmire got back about the Workflow Statistics issue, and apparently it’s a bug in the CUA module so they will send us a pull request 2017-05-04 Sync DSpace Test with database and assetstore from CGSpace Re-deploy DSpace Test with Atmire’s CUA patch for workflow statistics, run system updates, and restart the server Now I can see the workflow statistics and am able to select users, but everything returns 0 items Megan says there are still some mapped items are not appearing since last week, so I forced a full index-discovery -b Need to remember to check if the collection has more items (currently 39 on CGSpace, but 118 on the freshly reindexed DSPace Test) tomorrow: https://cgspace."/>
|
<meta name="twitter:description" content="2017-05-01 ICARDA apparently started working on CG Core on their MEL repository They have done a few cg.* fields, but not very consistent and even copy some of CGSpace items: https://mel.cgiar.org/xmlui/handle/20.500.11766/6911?show=full https://cgspace.cgiar.org/handle/10568/73683 2017-05-02 Atmire got back about the Workflow Statistics issue, and apparently it’s a bug in the CUA module so they will send us a pull request 2017-05-04 Sync DSpace Test with database and assetstore from CGSpace Re-deploy DSpace Test with Atmire’s CUA patch for workflow statistics, run system updates, and restart the server Now I can see the workflow statistics and am able to select users, but everything returns 0 items Megan says there are still some mapped items are not appearing since last week, so I forced a full index-discovery -b Need to remember to check if the collection has more items (currently 39 on CGSpace, but 118 on the freshly reindexed DSPace Test) tomorrow: https://cgspace."/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -28,7 +28,7 @@
|
|||||||
|
|
||||||
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="June, 2017"/>
|
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="June, 2017"/>
|
||||||
<meta name="twitter:description" content="2017-06-01 After discussion with WLE and CGSpace content people, we decided to just add one metadata field for the WLE Research Themes The cg.identifier.wletheme field will be used for both Phase I and Phase II Research Themes Then we’ll create a new sub-community for Phase II and create collections for the research themes there The current “Research Themes” community will be renamed to “WLE Phase I Research Themes” Tagged all items in the current Phase I collections with their appropriate themes Create pull request to add Phase II research themes to the submission form: #328 Add cg."/>
|
<meta name="twitter:description" content="2017-06-01 After discussion with WLE and CGSpace content people, we decided to just add one metadata field for the WLE Research Themes The cg.identifier.wletheme field will be used for both Phase I and Phase II Research Themes Then we’ll create a new sub-community for Phase II and create collections for the research themes there The current “Research Themes” community will be renamed to “WLE Phase I Research Themes” Tagged all items in the current Phase I collections with their appropriate themes Create pull request to add Phase II research themes to the submission form: #328 Add cg."/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -56,7 +56,7 @@ We can use PostgreSQL’s extended output format (-x) plus sed to format the
|
|||||||
|
|
||||||
|
|
||||||
"/>
|
"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -76,7 +76,7 @@ Then I cleaned up the author authorities and HTML characters in OpenRefine and s
|
|||||||
|
|
||||||
|
|
||||||
"/>
|
"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -52,7 +52,7 @@ Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account
|
|||||||
|
|
||||||
|
|
||||||
"/>
|
"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -56,7 +56,7 @@ Add Katherine Lutz to the groups for content submission and edit steps of the CG
|
|||||||
|
|
||||||
|
|
||||||
"/>
|
"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -76,7 +76,7 @@ COPY 54701
|
|||||||
|
|
||||||
|
|
||||||
"/>
|
"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -46,7 +46,7 @@ The list of connections to XMLUI and REST API for today:
|
|||||||
|
|
||||||
|
|
||||||
"/>
|
"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -184,7 +184,7 @@ Danny wrote to ask for help renewing the wildcard ilri.org certificate and I adv
|
|||||||
|
|
||||||
|
|
||||||
"/>
|
"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@ -194,7 +194,7 @@ Danny wrote to ask for help renewing the wildcard ilri.org certificate and I adv
|
|||||||
"@type": "BlogPosting",
|
"@type": "BlogPosting",
|
||||||
"headline": "January, 2018",
|
"headline": "January, 2018",
|
||||||
"url": "https://alanorth.github.io/cgspace-notes/2018-01/",
|
"url": "https://alanorth.github.io/cgspace-notes/2018-01/",
|
||||||
"wordCount": "282",
|
"wordCount": "731",
|
||||||
"datePublished": "2018-01-02T08:35:54-08:00",
|
"datePublished": "2018-01-02T08:35:54-08:00",
|
||||||
"dateModified": "2018-01-02T09:30:34-08:00",
|
"dateModified": "2018-01-02T09:30:34-08:00",
|
||||||
"author": {
|
"author": {
|
||||||
@ -339,6 +339,122 @@ dspace.log.2018-01-02:34
|
|||||||
|
|
||||||
<p></p>
|
<p></p>
|
||||||
|
|
||||||
|
<h2 id="2018-01-03">2018-01-03</h2>
|
||||||
|
|
||||||
|
<ul>
|
||||||
|
<li>I woke up to more up and down of CGSpace, this time UptimeRobot noticed a few rounds of up and down of a few minutes each and Linode also notified of high CPU load from 12 to 2 PM</li>
|
||||||
|
<li>Looks like I need to increase the database pool size again:</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<pre><code>$ grep -c "Timeout: Pool empty." dspace.log.2018-01-*
|
||||||
|
dspace.log.2018-01-01:0
|
||||||
|
dspace.log.2018-01-02:1972
|
||||||
|
dspace.log.2018-01-03:1909
|
||||||
|
</code></pre>
|
||||||
|
|
||||||
|
<ul>
|
||||||
|
<li>For some reason there were a lot of “active” connections last night:</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<p><img src="/cgspace-notes/2018/01/postgres_connections-day.png" alt="CGSpace PostgreSQL connections" /></p>
|
||||||
|
|
||||||
|
<ul>
|
||||||
|
<li>The active IPs in XMLUI are:</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<pre><code># cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -E "3/Jan/2018" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
|
||||||
|
607 40.77.167.141
|
||||||
|
611 2a00:23c3:8c94:7800:392c:a491:e796:9c50
|
||||||
|
663 188.226.169.37
|
||||||
|
759 157.55.39.245
|
||||||
|
887 68.180.229.254
|
||||||
|
1037 157.55.39.175
|
||||||
|
1068 216.244.66.245
|
||||||
|
1495 66.249.64.91
|
||||||
|
1934 104.196.152.243
|
||||||
|
2219 134.155.96.78
|
||||||
|
</code></pre>
|
||||||
|
|
||||||
|
<ul>
|
||||||
|
<li>134.155.96.78 appears to be at the University of Mannheim in Germany</li>
|
||||||
|
<li>They identify as: Mozilla/5.0 (compatible; heritrix/3.2.0 +<a href="http://ifm.uni-mannheim.de">http://ifm.uni-mannheim.de</a>)</li>
|
||||||
|
<li>This appears to be the <a href="https://github.com/internetarchive/heritrix3">Internet Archive’s open source bot</a></li>
|
||||||
|
<li>They seem to be re-using their Tomcat session so I don’t need to do anything to them just yet:</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<pre><code>$ grep 134.155.96.78 dspace.log.2018-01-03 | grep -o -E 'session_id=[A-Z0-9]{32}' | sort -n | uniq | wc -l
|
||||||
|
2
|
||||||
|
</code></pre>
|
||||||
|
|
||||||
|
<ul>
|
||||||
|
<li>The API logs show the normal users:</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<pre><code># cat /var/log/nginx/rest.log /var/log/nginx/rest.log.1 /var/log/nginx/oai.log /var/log/nginx/oai.log.1 | grep -E "3/Jan/2018" | awk '{print $1}' | sort -n | uniq -c | sort -h | tail
|
||||||
|
32 207.46.13.182
|
||||||
|
38 40.77.167.132
|
||||||
|
38 68.180.229.254
|
||||||
|
43 66.249.64.91
|
||||||
|
46 40.77.167.141
|
||||||
|
49 157.55.39.245
|
||||||
|
79 157.55.39.175
|
||||||
|
1533 50.116.102.77
|
||||||
|
4069 70.32.83.92
|
||||||
|
9355 45.5.184.196
|
||||||
|
</code></pre>
|
||||||
|
|
||||||
|
<ul>
|
||||||
|
<li>In other related news I see a sizeable amount of requests coming from python-requests</li>
|
||||||
|
<li>For example, just in the last day there were 1700!</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<pre><code># cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep -c python-requests
|
||||||
|
1773
|
||||||
|
</code></pre>
|
||||||
|
|
||||||
|
<ul>
|
||||||
|
<li>But they come from hundreds of IPs, many of which are 54.x.x.x:</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<pre><code># cat /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/library-access.log /var/log/nginx/library-access.log.1 | grep python-requests | awk '{print $1}' | sort -n | uniq -c | sort -h | tail -n 30
|
||||||
|
9 54.144.87.92
|
||||||
|
9 54.146.222.143
|
||||||
|
9 54.146.249.249
|
||||||
|
9 54.158.139.206
|
||||||
|
9 54.161.235.224
|
||||||
|
9 54.163.41.19
|
||||||
|
9 54.163.4.51
|
||||||
|
9 54.196.195.107
|
||||||
|
9 54.198.89.134
|
||||||
|
9 54.80.158.113
|
||||||
|
10 54.198.171.98
|
||||||
|
10 54.224.53.185
|
||||||
|
10 54.226.55.207
|
||||||
|
10 54.227.8.195
|
||||||
|
10 54.242.234.189
|
||||||
|
10 54.242.238.209
|
||||||
|
10 54.80.100.66
|
||||||
|
11 54.161.243.121
|
||||||
|
11 54.205.154.178
|
||||||
|
11 54.234.225.84
|
||||||
|
11 54.87.23.173
|
||||||
|
11 54.90.206.30
|
||||||
|
12 54.196.127.62
|
||||||
|
12 54.224.242.208
|
||||||
|
12 54.226.199.163
|
||||||
|
13 54.162.149.249
|
||||||
|
13 54.211.182.255
|
||||||
|
19 50.17.61.150
|
||||||
|
21 54.211.119.107
|
||||||
|
139 164.39.7.62
|
||||||
|
</code></pre>
|
||||||
|
|
||||||
|
<ul>
|
||||||
|
<li>I have no idea what these are but they seem to be coming from Amazon…</li>
|
||||||
|
<li>I guess for now I just have to increase the database connection pool’s max active</li>
|
||||||
|
<li>It’s currently 75 and normally I’d just bump it by 25 but let me be a bit daring and push it by 50 to 125, because I used to see at least 121 connections in pg_stat_activity before when we were using the shitty default pooling</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
BIN
public/2018/01/postgres_connections-day.png
Normal file
BIN
public/2018/01/postgres_connections-day.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 15 KiB |
@ -25,7 +25,7 @@
|
|||||||
|
|
||||||
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="Notes"/>
|
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="Notes"/>
|
||||||
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
|
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -28,7 +28,7 @@
|
|||||||
|
|
||||||
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="CGIAR Library Migration"/>
|
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="CGIAR Library Migration"/>
|
||||||
<meta name="twitter:description" content="Notes on the migration of the CGIAR Library to CGSpace"/>
|
<meta name="twitter:description" content="Notes on the migration of the CGIAR Library to CGSpace"/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -25,7 +25,7 @@
|
|||||||
|
|
||||||
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="CGSpace Notes"/>
|
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="CGSpace Notes"/>
|
||||||
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
|
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -25,7 +25,7 @@
|
|||||||
|
|
||||||
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="CGSpace Notes"/>
|
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="CGSpace Notes"/>
|
||||||
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
|
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -25,7 +25,7 @@
|
|||||||
|
|
||||||
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="CGSpace Notes"/>
|
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="CGSpace Notes"/>
|
||||||
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
|
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -25,7 +25,7 @@
|
|||||||
|
|
||||||
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="Posts"/>
|
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="Posts"/>
|
||||||
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
|
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -25,7 +25,7 @@
|
|||||||
|
|
||||||
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="Posts"/>
|
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="Posts"/>
|
||||||
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
|
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -25,7 +25,7 @@
|
|||||||
|
|
||||||
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="Posts"/>
|
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="Posts"/>
|
||||||
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
|
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -31,7 +31,7 @@ Disallow: /cgspace-notes/2015-12/
|
|||||||
Disallow: /cgspace-notes/2015-11/
|
Disallow: /cgspace-notes/2015-11/
|
||||||
Disallow: /cgspace-notes/
|
Disallow: /cgspace-notes/
|
||||||
Disallow: /cgspace-notes/categories/
|
Disallow: /cgspace-notes/categories/
|
||||||
Disallow: /cgspace-notes/categories/notes/
|
|
||||||
Disallow: /cgspace-notes/tags/notes/
|
Disallow: /cgspace-notes/tags/notes/
|
||||||
|
Disallow: /cgspace-notes/categories/notes/
|
||||||
Disallow: /cgspace-notes/post/
|
Disallow: /cgspace-notes/post/
|
||||||
Disallow: /cgspace-notes/tags/
|
Disallow: /cgspace-notes/tags/
|
||||||
|
@ -154,14 +154,14 @@
|
|||||||
</url>
|
</url>
|
||||||
|
|
||||||
<url>
|
<url>
|
||||||
<loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
|
<loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
|
||||||
<lastmod>2017-09-28T12:00:49+03:00</lastmod>
|
<lastmod>2018-01-02T09:30:34-08:00</lastmod>
|
||||||
<priority>0</priority>
|
<priority>0</priority>
|
||||||
</url>
|
</url>
|
||||||
|
|
||||||
<url>
|
<url>
|
||||||
<loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
|
<loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
|
||||||
<lastmod>2018-01-02T09:30:34-08:00</lastmod>
|
<lastmod>2017-09-28T12:00:49+03:00</lastmod>
|
||||||
<priority>0</priority>
|
<priority>0</priority>
|
||||||
</url>
|
</url>
|
||||||
|
|
||||||
|
@ -25,7 +25,7 @@
|
|||||||
|
|
||||||
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="Notes"/>
|
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="Notes"/>
|
||||||
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
|
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -25,7 +25,7 @@
|
|||||||
|
|
||||||
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="Notes"/>
|
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="Notes"/>
|
||||||
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
|
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@ -25,7 +25,7 @@
|
|||||||
|
|
||||||
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="Notes"/>
|
<meta name="twitter:card" content="summary"/><meta name="twitter:title" content="Notes"/>
|
||||||
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
|
<meta name="twitter:description" content="Documenting day-to-day work on the [CGSpace](https://cgspace.cgiar.org) repository."/>
|
||||||
<meta name="generator" content="Hugo 0.32.1" />
|
<meta name="generator" content="Hugo 0.32.2" />
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
BIN
static/2018/01/postgres_connections-day.png
Normal file
BIN
static/2018/01/postgres_connections-day.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 15 KiB |
Loading…
Reference in New Issue
Block a user