Update notes for 2019-04-06

content/posts/2019-04.md

@@ -155,6 +155,16 @@ GET /handle/10568/72970/discover?filtertype_0=type&filtertype_1=author&filter_re
   22077 /handle/10568/72970/discover
 ```
 
+- Yesterday they made 43,000 requests and we actually blocked most of them:
+
+```
+# zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/access.log.2.gz | grep "05/Apr/2019" | grep 45.5.184.72 | grep -oE '/handle/[0-9]+/[0-9]+/discover' | sort | uniq -c 
+  43631 /handle/10568/72970/discover
+# zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/access.log.2.gz | grep "05/Apr/2019" | grep 45.5.184.72 | grep -E '/handle/[0-9]+/[0-9]+/discover' | awk '{print $9}' | sort | uniq -c 
+    142 200
+  43489 503
+```
+
 - I need to find a contact at CIAT to tell them to use the REST API rather than crawling Discover
 - Maria from Bioversity recommended that we use the phrase "AGROVOC subject" instead of "Subject" in Listings and Reports
   - I made a pull request to update this and merged it to the `5_x-prod` branch ([#418](https://github.com/ilri/DSpace/pull/418))

docs/2019-04/index.html

@@ -38,7 +38,7 @@ $ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace
 <meta property="og:type" content="article" />
 <meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2019-04/" />
 <meta property="article:published_time" content="2019-04-01T09:00:43&#43;03:00"/>
-<meta property="article:modified_time" content="2019-04-05T23:07:30&#43;03:00"/>
+<meta property="article:modified_time" content="2019-04-06T11:47:45&#43;03:00"/>
 
 <meta name="twitter:card" content="summary"/>
 <meta name="twitter:title" content="April, 2019"/>
@@ -81,9 +81,9 @@ $ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace
   "@type": "BlogPosting",
   "headline": "April, 2019",
   "url": "https://alanorth.github.io/cgspace-notes/2019-04/",
-  "wordCount": "980",
+  "wordCount": "1044",
   "datePublished": "2019-04-01T09:00:43&#43;03:00",
-  "dateModified": "2019-04-05T23:07:30&#43;03:00",
+  "dateModified": "2019-04-06T11:47:45&#43;03:00",
   "author": {
     "@type": "Person",
     "name": "Alan Orth"
@@ -338,6 +338,17 @@ $ ./delete-metadata-values.py -i /tmp/2019-02-21-delete-1-region.csv -db dspace
   22077 /handle/10568/72970/discover
 </code></pre>
 
+<ul>
+<li>Yesterday they made 43,000 requests and we actually blocked most of them:</li>
+</ul>
+
+<pre><code># zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/access.log.2.gz | grep &quot;05/Apr/2019&quot; | grep 45.5.184.72 | grep -oE '/handle/[0-9]+/[0-9]+/discover' | sort | uniq -c 
+  43631 /handle/10568/72970/discover
+# zcat --force /var/log/nginx/access.log /var/log/nginx/access.log.1 /var/log/nginx/access.log.2.gz | grep &quot;05/Apr/2019&quot; | grep 45.5.184.72 | grep -E '/handle/[0-9]+/[0-9]+/discover' | awk '{print $9}' | sort | uniq -c 
+    142 200
+  43489 503
+</code></pre>
+
 <ul>
 <li>I need to find a contact at CIAT to tell them to use the REST API rather than crawling Discover</li>
 <li>Maria from Bioversity recommended that we use the phrase &ldquo;AGROVOC subject&rdquo; instead of &ldquo;Subject&rdquo; in Listings and Reports

docs/sitemap.xml

@@ -4,7 +4,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/2019-04/</loc>
-    <lastmod>2019-04-05T23:07:30+03:00</lastmod>
+    <lastmod>2019-04-06T11:47:45+03:00</lastmod>
   </url>
   
   <url>
@@ -219,7 +219,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/</loc>
-    <lastmod>2019-04-05T23:07:30+03:00</lastmod>
+    <lastmod>2019-04-06T11:47:45+03:00</lastmod>
     <priority>0</priority>
   </url>
   
@@ -230,7 +230,7 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/tags/notes/</loc>
-    <lastmod>2019-04-05T23:07:30+03:00</lastmod>
+    <lastmod>2019-04-06T11:47:45+03:00</lastmod>
     <priority>0</priority>
   </url>
   
@@ -242,13 +242,13 @@
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
-    <lastmod>2019-04-05T23:07:30+03:00</lastmod>
+    <lastmod>2019-04-06T11:47:45+03:00</lastmod>
     <priority>0</priority>
   </url>
   
   <url>
     <loc>https://alanorth.github.io/cgspace-notes/tags/</loc>
-    <lastmod>2019-04-05T23:07:30+03:00</lastmod>
+    <lastmod>2019-04-06T11:47:45+03:00</lastmod>
     <priority>0</priority>
   </url>