From 238ae1678f0cf1b52436455881c8283f66e6df64 Mon Sep 17 00:00:00 2001 From: Alan Orth Date: Tue, 19 Feb 2019 12:42:33 -0800 Subject: [PATCH] Add notes for 2019-02-19 --- content/posts/2019-02.md | 49 +++++++++++++++++++++++++++ docs/2019-02/index.html | 60 +++++++++++++++++++++++++++++++-- docs/2019/02/usage-stats.png | Bin 0 -> 3790 bytes docs/sitemap.xml | 10 +++--- static/2019/02/usage-stats.png | Bin 0 -> 3790 bytes 5 files changed, 111 insertions(+), 8 deletions(-) create mode 100644 docs/2019/02/usage-stats.png create mode 100644 static/2019/02/usage-stats.png diff --git a/content/posts/2019-02.md b/content/posts/2019-02.md index c059daf43..3a12be7ed 100644 --- a/content/posts/2019-02.md +++ b/content/posts/2019-02.md @@ -884,4 +884,53 @@ $ tidy -xml -utf8 -iq -m -w 0 dspace/config/controlled-vocabularies/cg-creator-i - I merged the changes to the `5_x-prod` branch and they will go live the next time we re-deploy CGSpace ([#412](https://github.com/ilri/DSpace/pull/412)) +## 2019-02-19 + +- Linode sent another alert about CPU usage on CGSpace (linode18) averaging 417% this morning +- Unfortunately, I don't see any strange activity in the web server API or XMLUI logs at that time in particular +- So far today the top ten IPs in the XMLUI logs are: + +``` +# zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -E "19/Feb/2019:" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10 + 11541 18.212.208.240 + 11560 3.81.136.184 + 11562 3.88.237.84 + 11569 34.230.15.139 + 11572 3.80.128.247 + 11573 3.91.17.126 + 11586 54.82.89.217 + 11610 54.209.39.13 + 11657 54.175.90.13 + 14686 143.233.242.130 +``` + +- 143.233.242.130 is in Greece and using the user agent "Indy Library", like the top IP yesterday (94.71.244.172) +- That user agent is in our Tomcat list of crawlers so at least its resource usage is controlled by forcing it to use a single Tomcat session, but I don't know if DSpace recognizes if this is a bot or not, so the logs are probably skewed because of this +- The user is requesting only things like `/handle/10568/56199?show=full` so it's nothing malicious, only annoying +- Otherwise there are still shit loads of IPs from Amazon still hammering the server, though I see HTTP 503 errors now after yesterday's nginx rate limiting updates + - I should really try to script something around [ipapi.co](https://ipapi.co/api/) to get these quickly and easily +- The top requests in the API logs today are: + +``` +# zcat --force /var/log/nginx/{oai,rest,statistics}.log /var/log/nginx/{oai,rest,statistics}.log.1 | grep -E "19/Feb/2019:" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10 + 42 66.249.66.221 + 44 156.156.81.215 + 55 3.85.54.129 + 76 66.249.66.219 + 87 34.209.213.122 + 1550 34.218.226.147 + 2127 50.116.102.77 + 4684 205.186.128.185 + 11429 45.5.186.2 + 12360 2a01:7e00::f03c:91ff:fe0a:d645 +``` + +- `2a01:7e00::f03c:91ff:fe0a:d645` is on Linode, and I can see from the XMLUI access logs that it is Drupal, so I assume it is part of the new ILRI website harvester... +- Jesus, Linode just sent another alert as we speak that the load on CGSpace (linode18) has been at 450% the last two hours! I'm so fucking sick of this +- Our usage stats have exploded the last few months: + +![Usage stats](/cgspace-notes/2019/02/usage-stats.png) + +- I need to follow up with the DSpace developers and Atmire to see how they classify which requests are bots so we can try to estimate the impact caused by these users and perhaps try to update the list to make the stats more accurate + diff --git a/docs/2019-02/index.html b/docs/2019-02/index.html index 2face16dd..75b32090c 100644 --- a/docs/2019-02/index.html +++ b/docs/2019-02/index.html @@ -42,7 +42,7 @@ sys 0m1.979s - + @@ -89,9 +89,9 @@ sys 0m1.979s "@type": "BlogPosting", "headline": "February, 2019", "url": "https://alanorth.github.io/cgspace-notes/2019-02/", - "wordCount": "4900", + "wordCount": "5236", "datePublished": "2019-02-01T21:37:30+02:00", - "dateModified": "2019-02-18T15:00:47-08:00", + "dateModified": "2019-02-18T16:30:34-08:00", "author": { "@type": "Person", "name": "Alan Orth" @@ -1158,6 +1158,60 @@ $ tidy -xml -utf8 -iq -m -w 0 dspace/config/controlled-vocabularies/cg-creator-i
  • I merged the changes to the 5_x-prod branch and they will go live the next time we re-deploy CGSpace (#412)
  • +

    2019-02-19

    + + + +
    # zcat --force /var/log/nginx/{access,error,library-access}.log /var/log/nginx/{access,error,library-access}.log.1 | grep -E "19/Feb/2019:" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
    +  11541 18.212.208.240
    +  11560 3.81.136.184
    +  11562 3.88.237.84
    +  11569 34.230.15.139
    +  11572 3.80.128.247
    +  11573 3.91.17.126
    +  11586 54.82.89.217
    +  11610 54.209.39.13
    +  11657 54.175.90.13
    +  14686 143.233.242.130
    +
    + + + +
    # zcat --force /var/log/nginx/{oai,rest,statistics}.log /var/log/nginx/{oai,rest,statistics}.log.1 | grep -E "19/Feb/2019:" | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 10
    +     42 66.249.66.221
    +     44 156.156.81.215
    +     55 3.85.54.129
    +     76 66.249.66.219
    +     87 34.209.213.122
    +   1550 34.218.226.147
    +   2127 50.116.102.77
    +   4684 205.186.128.185
    +  11429 45.5.186.2
    +  12360 2a01:7e00::f03c:91ff:fe0a:d645
    +
    + + + +

    Usage stats

    + diff --git a/docs/2019/02/usage-stats.png b/docs/2019/02/usage-stats.png new file mode 100644 index 0000000000000000000000000000000000000000..a0d17434c05e2d5c0e629ab23d765aa6b9de58db GIT binary patch literal 3790 zcmZ`+c|4SB8@42jCOVd5Nsf+qEiJZWi9y6v8e2ngh$KXx=}5-DwFoJ5Xl#v!l+pCF zpRq3)8DnhO4Q0>HSQGQTL-jeQ-#7EeGtcw9&wby|bzS#;zw;1bb`}Z|hp@4+K}}80 zSg^5i0BmgRaXg&hO8yHE6>uVtIB#XdWHLiSLPkbLP$<;g+{nj|-QWXE=Df0fTvb)o z+}t?$5f>LXGBTQ&nBd~#l8}%vG&KDB_3QlnyrZMz$B(^C=F-)xSNZt(yuH0092}rf zC?1b5C@2sS5m{SXcyo!p70s{jD1O&pv!*g?UJ3Bk6RO+!~$F#JxOifL*va-y~%;MwY z?d9v{+hNj*X4=_4VZz zmEmwW831t>1k=&?n{4S=0HMkwneV{2oC+;p=Q7%9U)-w{x_p{rLgzE>9fls zLO6tQY;}L!Wj2_|RwyN(3r@Qr6VvG7Z?9Dp-=}G!PM&lw9HG_Aub#|C-YXy;Xz1gI zU(v8&C}OFUuTsS7p$JIhpYrzuU1bZ>QtT*=Wyb+a#3d@S{!9Rq-dP8|2sO zAKPrb$Hezx5Qg%U{Ni>0K&qWi&-3g<7bpd!Q;Jyr;bAG%N}LPUri_q)9Km}?)Yb7d zi$Ccnw6-EO{cf3AJ|#qN`J`d$>Z{9^A`V8S`M#C>OOk=IpuLgcUWv2x^Q7OEx9vK4 zQhiSR3FnmYdD%Dh^(n7!OnaX|7(hqiGPe)pK0Ru*d=aS=+IepekFOvq(aszT(2)1X0OC#JXl>b zkjeDxdSC1Olvpn(#m#x-8aBRpW!ZJZh)ktV%GtIS93hnulLB(+wzr#sn9u1=IUn-K zuEw8cyhU&KoB}q1M1Eg-W3+m^&7A%I8Tw~Un{VaG#KLs)*NK?r<9do3tE-Rhm0w{hV*MEQ7s(X|e#Po0!5!X8`llDN)y|L7sR z(k``Typ3zm$&}LL^zhYFkAKNKm?jio7jNiK(kYKu)3wdgwixf}H&dFMrc9QSMrhTA zqANIjKIz(Zw-HA1T#B8?nZ?d_zpgnBn%xW5}WKE?*9pSLp$++lj&j?D_G9u|rtd!+{aVI=I?!6k+oF!t=mi zxFNj1V(K!fQJW73(DKN(QY9p(%Np*c&RehE(aE8p%NixD6~F;q(QEy_gg4Vn_!mGj zHJUipX`L=xXiz*l8N9i)Bv%o(ZUI}5_4HgX51d46r=|#v70pcLXyP%?%xiS;?>Ku9 z+;Se|ynM&*dH_QRZTsAXrrJGtDVa2f-Y<{jj6|s;PJP-7CO0c)85`XDb`fdpX%kdtrbUUb!HL2w3X}B9`_k{v7@lrYcY<3*X&5}s7Z@RWB zkL15W>J3IXUs*UtUiyP*HsN-6LKZ-B8d(z$>0xazzXVNDI$)n7B)02} z08Ze44tIP@!=z#Cl>ZpG-MQXGKQ5kXi+iP&-)n9#LFF*&!M^GAjif0dFsVJ|QP!|@ z)pM?wTCs%$FDgIBa_h}u1ZB<#$SP=zn-W|U6gPv8Mc2aRLY9rt}i5cfjl`nRn_vPJ$t?mLL=kR3tb zfW{6_{|_K`Lr0OtGv=1Wy-~$>^rB@G;k@F0vbL+@O@#R9WYKANN(ZxY=ue%DQTOW@ z-O8MIwd#1aU1x*)@c#mE$LBXUy93ycX5quW6s~fN;%5v2zdv)@k`|p9=!)5kVTlev zo|TM38!7-kg78!Y8H5!AqW)SKTrIbosg$lnEpuXawDru~3bAiPlqH`4N{2{-A;Z2| z8Uz8nJbdnlqP|J!XFl2CCc)@wr3_j=AB&Uh=LeB_XXRr+60Hz$!T?+Xs(~vAeOj8- zFx!sJR3(HxZD~h`iA9PTak=zHsOYoK(5BZKsBJ2s7Fw`e-w_9O3M%EFoPzrOCa9^-*^&p)A1$mQacMQO4j-HmYztPrzf5 zf?nUP|D;jSE6+}!G_ocBVMoa+dud1ap?E*X;7{Szwo@AvJ-3SOp0cU5{g-QJJQ6^J zKOl^|d?+3E!Sa)|)}u&|NRnreeYlvBCO${3QJfWEW5-Gj*ABG1Zu#>azY%?JIjf## z(1%t6fw>!O(qDAw9EjE*fM?@Pwf>Ny$TrqO0!>L3qDtX!0z9X3!HgK1p$1)foC&6h zR{oDX+^wAN`Tt$T;6*V-t>;Kj%|L(@fV#~BFOd2u2%H{akpBtC*!Ne=K9Lh&a6Z?A885bx@`iU91Jds*j0f7DTBI9zWzZl*V}_;9e!lx?h3RM zK`Yp0wo8yO2EGDhns57d>*xzSH~@kgu&o^anY(6u82{t{>E5yB&Dwr6n4jt+SWBDh`|H6Rx1Xtzr)lFU@+@7r%jvbjrK;h>L zaN7A-HrKxL;Ff-+m>Y1+QCC(axG+Jm*PXsc=0RD%;C6ZIB#hvgrLGjK!d;7smFYg$ z>mVQraNoa*LbUT?4;3)1LF^s_VBb}v{x8ql1tzl@;pM-1^hNspyuUHFev*HZ(DS%` zv@NJ2CT6^&>C94N2aNY)=D5enFV-fwrTs2{+VlI8O8V>Wix3*?Fa%+Fs^|T$UFt)% zq;%b80BjT@iy#JH*z2}@Vjdw9389P4DQ6dsBR7I_Oxb~y|9HS2PHRBEHG^GZ`01zE zWgrc-6w;>qeE@j@XGGNU^yGBeAoIS!U}|bHnfe9V>mJ>++}Dzzdc9-yRv(H?J?5)S z4mPBT4vu#=Jwxa*Z9-Hb(y7A=NWme7hu-5+ch*ZlTy^Y5tqNL9unlUozRoSNdf#P9 zMK(hsYLUUO)5PC6;eFmjLuhndYPxi64BR{UZBLB|?zw&2yU{O|(AxMc&I_P4>vjLF zr({?_el8`tV^eVVy9Vop81%Z&XXH6l`!_Yjy|r6%V5!+|A@C2lyMr+%S(5It3Tbw} zj9+;WhuKGq7WA(bd#Bu0B5L-Geq!t&KDuExHjb)qR(D0`Zdu8mIVVZ& zczOJg@1l{h$AH|p@q)q|x6h|*yH(#7ciz68+{S=PSzeB;0ftgJ+WUFNVVeBI{328@a>P7p!g^edR{T<<5sWGeO-DwSS1ueJcI!Zt?Kk9mJ=oKL8d6<3H!ig?D&_T;nQ6i?Nu;!Vrti6?pU zwx`+lrFp-wlECQTTMR>zWA{ciYY$^bUJ+`dKPkqL**1BrWagSaw5vv8BAY!|IWF;r zSv-FDV$h|Pb3i+G7;*L(kFE$3j+cf_aQy8T96-`7MT%H#E%wN=^BusT!wzdPR>NQ2 zDi=NHZTSm3vdu{&TW#<~Z zy(B0G6m4c5Ituqe20!9DcfI%4#v2Uj;%z9V`c=ys(N*@&>GbY_`Ehtnf2)_e8?X|3AG4oSN@#`61K>&kmK4jS{Y_hPH>h6Yaj6 gL~@NnrY^f8*OO6~PL3+@Zz!9ok=Yrdfn(Tz0fVm3(*OVf literal 0 HcmV?d00001 diff --git a/docs/sitemap.xml b/docs/sitemap.xml index aef222c19..a71195d76 100644 --- a/docs/sitemap.xml +++ b/docs/sitemap.xml @@ -4,7 +4,7 @@ https://alanorth.github.io/cgspace-notes/2019-02/ - 2019-02-18T15:00:47-08:00 + 2019-02-18T16:30:34-08:00 @@ -209,7 +209,7 @@ https://alanorth.github.io/cgspace-notes/ - 2019-02-18T15:00:47-08:00 + 2019-02-18T16:30:34-08:00 0 @@ -220,7 +220,7 @@ https://alanorth.github.io/cgspace-notes/tags/notes/ - 2019-02-18T15:00:47-08:00 + 2019-02-18T16:30:34-08:00 0 @@ -232,13 +232,13 @@ https://alanorth.github.io/cgspace-notes/posts/ - 2019-02-18T15:00:47-08:00 + 2019-02-18T16:30:34-08:00 0 https://alanorth.github.io/cgspace-notes/tags/ - 2019-02-18T15:00:47-08:00 + 2019-02-18T16:30:34-08:00 0 diff --git a/static/2019/02/usage-stats.png b/static/2019/02/usage-stats.png new file mode 100644 index 0000000000000000000000000000000000000000..a0d17434c05e2d5c0e629ab23d765aa6b9de58db GIT binary patch literal 3790 zcmZ`+c|4SB8@42jCOVd5Nsf+qEiJZWi9y6v8e2ngh$KXx=}5-DwFoJ5Xl#v!l+pCF zpRq3)8DnhO4Q0>HSQGQTL-jeQ-#7EeGtcw9&wby|bzS#;zw;1bb`}Z|hp@4+K}}80 zSg^5i0BmgRaXg&hO8yHE6>uVtIB#XdWHLiSLPkbLP$<;g+{nj|-QWXE=Df0fTvb)o z+}t?$5f>LXGBTQ&nBd~#l8}%vG&KDB_3QlnyrZMz$B(^C=F-)xSNZt(yuH0092}rf zC?1b5C@2sS5m{SXcyo!p70s{jD1O&pv!*g?UJ3Bk6RO+!~$F#JxOifL*va-y~%;MwY z?d9v{+hNj*X4=_4VZz zmEmwW831t>1k=&?n{4S=0HMkwneV{2oC+;p=Q7%9U)-w{x_p{rLgzE>9fls zLO6tQY;}L!Wj2_|RwyN(3r@Qr6VvG7Z?9Dp-=}G!PM&lw9HG_Aub#|C-YXy;Xz1gI zU(v8&C}OFUuTsS7p$JIhpYrzuU1bZ>QtT*=Wyb+a#3d@S{!9Rq-dP8|2sO zAKPrb$Hezx5Qg%U{Ni>0K&qWi&-3g<7bpd!Q;Jyr;bAG%N}LPUri_q)9Km}?)Yb7d zi$Ccnw6-EO{cf3AJ|#qN`J`d$>Z{9^A`V8S`M#C>OOk=IpuLgcUWv2x^Q7OEx9vK4 zQhiSR3FnmYdD%Dh^(n7!OnaX|7(hqiGPe)pK0Ru*d=aS=+IepekFOvq(aszT(2)1X0OC#JXl>b zkjeDxdSC1Olvpn(#m#x-8aBRpW!ZJZh)ktV%GtIS93hnulLB(+wzr#sn9u1=IUn-K zuEw8cyhU&KoB}q1M1Eg-W3+m^&7A%I8Tw~Un{VaG#KLs)*NK?r<9do3tE-Rhm0w{hV*MEQ7s(X|e#Po0!5!X8`llDN)y|L7sR z(k``Typ3zm$&}LL^zhYFkAKNKm?jio7jNiK(kYKu)3wdgwixf}H&dFMrc9QSMrhTA zqANIjKIz(Zw-HA1T#B8?nZ?d_zpgnBn%xW5}WKE?*9pSLp$++lj&j?D_G9u|rtd!+{aVI=I?!6k+oF!t=mi zxFNj1V(K!fQJW73(DKN(QY9p(%Np*c&RehE(aE8p%NixD6~F;q(QEy_gg4Vn_!mGj zHJUipX`L=xXiz*l8N9i)Bv%o(ZUI}5_4HgX51d46r=|#v70pcLXyP%?%xiS;?>Ku9 z+;Se|ynM&*dH_QRZTsAXrrJGtDVa2f-Y<{jj6|s;PJP-7CO0c)85`XDb`fdpX%kdtrbUUb!HL2w3X}B9`_k{v7@lrYcY<3*X&5}s7Z@RWB zkL15W>J3IXUs*UtUiyP*HsN-6LKZ-B8d(z$>0xazzXVNDI$)n7B)02} z08Ze44tIP@!=z#Cl>ZpG-MQXGKQ5kXi+iP&-)n9#LFF*&!M^GAjif0dFsVJ|QP!|@ z)pM?wTCs%$FDgIBa_h}u1ZB<#$SP=zn-W|U6gPv8Mc2aRLY9rt}i5cfjl`nRn_vPJ$t?mLL=kR3tb zfW{6_{|_K`Lr0OtGv=1Wy-~$>^rB@G;k@F0vbL+@O@#R9WYKANN(ZxY=ue%DQTOW@ z-O8MIwd#1aU1x*)@c#mE$LBXUy93ycX5quW6s~fN;%5v2zdv)@k`|p9=!)5kVTlev zo|TM38!7-kg78!Y8H5!AqW)SKTrIbosg$lnEpuXawDru~3bAiPlqH`4N{2{-A;Z2| z8Uz8nJbdnlqP|J!XFl2CCc)@wr3_j=AB&Uh=LeB_XXRr+60Hz$!T?+Xs(~vAeOj8- zFx!sJR3(HxZD~h`iA9PTak=zHsOYoK(5BZKsBJ2s7Fw`e-w_9O3M%EFoPzrOCa9^-*^&p)A1$mQacMQO4j-HmYztPrzf5 zf?nUP|D;jSE6+}!G_ocBVMoa+dud1ap?E*X;7{Szwo@AvJ-3SOp0cU5{g-QJJQ6^J zKOl^|d?+3E!Sa)|)}u&|NRnreeYlvBCO${3QJfWEW5-Gj*ABG1Zu#>azY%?JIjf## z(1%t6fw>!O(qDAw9EjE*fM?@Pwf>Ny$TrqO0!>L3qDtX!0z9X3!HgK1p$1)foC&6h zR{oDX+^wAN`Tt$T;6*V-t>;Kj%|L(@fV#~BFOd2u2%H{akpBtC*!Ne=K9Lh&a6Z?A885bx@`iU91Jds*j0f7DTBI9zWzZl*V}_;9e!lx?h3RM zK`Yp0wo8yO2EGDhns57d>*xzSH~@kgu&o^anY(6u82{t{>E5yB&Dwr6n4jt+SWBDh`|H6Rx1Xtzr)lFU@+@7r%jvbjrK;h>L zaN7A-HrKxL;Ff-+m>Y1+QCC(axG+Jm*PXsc=0RD%;C6ZIB#hvgrLGjK!d;7smFYg$ z>mVQraNoa*LbUT?4;3)1LF^s_VBb}v{x8ql1tzl@;pM-1^hNspyuUHFev*HZ(DS%` zv@NJ2CT6^&>C94N2aNY)=D5enFV-fwrTs2{+VlI8O8V>Wix3*?Fa%+Fs^|T$UFt)% zq;%b80BjT@iy#JH*z2}@Vjdw9389P4DQ6dsBR7I_Oxb~y|9HS2PHRBEHG^GZ`01zE zWgrc-6w;>qeE@j@XGGNU^yGBeAoIS!U}|bHnfe9V>mJ>++}Dzzdc9-yRv(H?J?5)S z4mPBT4vu#=Jwxa*Z9-Hb(y7A=NWme7hu-5+ch*ZlTy^Y5tqNL9unlUozRoSNdf#P9 zMK(hsYLUUO)5PC6;eFmjLuhndYPxi64BR{UZBLB|?zw&2yU{O|(AxMc&I_P4>vjLF zr({?_el8`tV^eVVy9Vop81%Z&XXH6l`!_Yjy|r6%V5!+|A@C2lyMr+%S(5It3Tbw} zj9+;WhuKGq7WA(bd#Bu0B5L-Geq!t&KDuExHjb)qR(D0`Zdu8mIVVZ& zczOJg@1l{h$AH|p@q)q|x6h|*yH(#7ciz68+{S=PSzeB;0ftgJ+WUFNVVeBI{328@a>P7p!g^edR{T<<5sWGeO-DwSS1ueJcI!Zt?Kk9mJ=oKL8d6<3H!ig?D&_T;nQ6i?Nu;!Vrti6?pU zwx`+lrFp-wlECQTTMR>zWA{ciYY$^bUJ+`dKPkqL**1BrWagSaw5vv8BAY!|IWF;r zSv-FDV$h|Pb3i+G7;*L(kFE$3j+cf_aQy8T96-`7MT%H#E%wN=^BusT!wzdPR>NQ2 zDi=NHZTSm3vdu{&TW#<~Z zy(B0G6m4c5Ituqe20!9DcfI%4#v2Uj;%z9V`c=ys(N*@&>GbY_`Ehtnf2)_e8?X|3AG4oSN@#`61K>&kmK4jS{Y_hPH>h6Yaj6 gL~@NnrY^f8*OO6~PL3+@Zz!9ok=Yrdfn(Tz0fVm3(*OVf literal 0 HcmV?d00001