mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2024-11-26 00:18:21 +01:00
Update notes for 2019-11-07
This commit is contained in:
parent
7177841e29
commit
65d9c40103
@ -177,5 +177,39 @@ $ http --print b 'http://localhost:8081/solr/statistics/select?q=userAgent:www.g
|
||||
- CCAFS finally confirmed that they do indeed need the confusing new project tag that looks like a duplicate
|
||||
- They had proposed a batch of new tags in 2019-09 and we never merged them due to this uncertainty
|
||||
- I have now merged the changes in to the `5_x-prod` branch ([#432](https://github.com/ilri/DSpace/pull/432))
|
||||
- I am reconsidering the move of `cg.identifier.dataurl` to `cg.hasMetadata` in CG Core v2
|
||||
- The values of this field are mostly links to data sets on Dataverse and partner sites
|
||||
- I opened an [issue on GitHub](https://github.com/AgriculturalSemantics/cg-core/issues/10) to ask Marie-Angelique for clarification
|
||||
- Looking into CGSpace statistics again
|
||||
- I searched for hits in Solr from the BUbiNG bot and found 63,000 in the `statistics-2018` core:
|
||||
|
||||
```
|
||||
$ http --print b 'http://localhost:8081/solr/statistics-2018/select?facet=true&facet.field=ip&facet.mincount=1&type:0&q=userAgent:BUbiNG*' | xmllint --format - | grep numFound
|
||||
<result name="response" numFound="62944" start="0">
|
||||
```
|
||||
|
||||
- Similar for com.plumanalytics, Grammarly, and ltx71!
|
||||
|
||||
```
|
||||
$ http --print b 'http://localhost:8081/solr/statistics-2018/select?facet=true&facet.field=ip&facet.mincount=1&type:0&q=userAgent:
|
||||
*com.plumanalytics*' | xmllint --format - | grep numFound
|
||||
<result name="response" numFound="28256" start="0">
|
||||
$ http --print b 'http://localhost:8081/solr/statistics-2018/select?facet=true&facet.field=ip&facet.mincount=1&type:0&q=userAgent:*Grammarly*' | xmllint --format - | grep numFound
|
||||
<result name="response" numFound="6288" start="0">
|
||||
$ http --print b 'http://localhost:8081/solr/statistics-2018/select?facet=true&facet.field=ip&facet.mincount=1&type:0&q=userAgent:*ltx71*' | xmllint --format - | grep numFound
|
||||
<result name="response" numFound="105663" start="0">
|
||||
```
|
||||
|
||||
- Deleting these seems to work, for example the 105,000 ltx71 records from 2018:
|
||||
|
||||
```
|
||||
$ http --print b 'http://localhost:8081/solr/statistics-2018/update?stream.body=<delete><query>userAgent:*ltx71*</query><query>type:0</query></delete>&commit=true'
|
||||
$ http --print b 'http://localhost:8081/solr/statistics-2018/select?facet=true&facet.field=ip&facet.mincount=1&type:0&q=userAgent:*ltx71*' | xmllint --format - | grep numFound
|
||||
<result name="response" numFound="0" start="0"/>
|
||||
```
|
||||
|
||||
- I wrote a quick bash script to check all these user agents against the CGSpace Solr statistics cores
|
||||
- For years 2010 until 2019 there are 1.6 million hits from these spider user agents
|
||||
- For 2019 alone there are 740,000, over half of which come from Unpaywall!
|
||||
|
||||
<!-- vim: set sw=2 ts=2: -->
|
||||
|
@ -34,7 +34,7 @@ Let’s see how many of the REST API requests were for bitstreams (because t
|
||||
<meta property="og:type" content="article" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2019-11/" />
|
||||
<meta property="article:published_time" content="2019-11-04T12:20:30+02:00" />
|
||||
<meta property="article:modified_time" content="2019-11-06T09:35:51+02:00" />
|
||||
<meta property="article:modified_time" content="2019-11-07T12:40:25+02:00" />
|
||||
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="November, 2019"/>
|
||||
@ -73,9 +73,9 @@ Let’s see how many of the REST API requests were for bitstreams (because t
|
||||
"@type": "BlogPosting",
|
||||
"headline": "November, 2019",
|
||||
"url": "https:\/\/alanorth.github.io\/cgspace-notes\/2019-11\/",
|
||||
"wordCount": "1088",
|
||||
"wordCount": "1293",
|
||||
"datePublished": "2019-11-04T12:20:30+02:00",
|
||||
"dateModified": "2019-11-06T09:35:51+02:00",
|
||||
"dateModified": "2019-11-07T12:40:25+02:00",
|
||||
"author": {
|
||||
"@type": "Person",
|
||||
"name": "Alan Orth"
|
||||
@ -323,21 +323,56 @@ $ http –print Hh ‘<a href="https://dspacetest.cgiar.org/handle/10568
|
||||
$ http –print b ‘<a href="http://localhost:8081/solr/statistics/select?q=userAgent:www.gnip.com&fq=dateYearMonth%3A2019-11'">http://localhost:8081/solr/statistics/select?q=userAgent:www.gnip.com&fq=dateYearMonth%3A2019-11'</a> | xmllint –format - | grep numFound
|
||||
<result name="response" numFound="0" start="0"/>
|
||||
$ http –print b ‘<a href="http://localhost:8081/solr/statistics/select?q=userAgent:www.gnyp.com&fq=dateYearMonth%3A2019-11'">http://localhost:8081/solr/statistics/select?q=userAgent:www.gnyp.com&fq=dateYearMonth%3A2019-11'</a> | xmllint –format - | grep numFound
|
||||
<result name="response" numFound="1" start="0">
|
||||
<result name="response" numFound="1" start="0"></p>
|
||||
|
||||
<pre><code>
|
||||
- So the blocking seems to be working because "www\.gnip\.com" is one of the new patterns added to the spiders file...
|
||||
|
||||
## 2019-11-07
|
||||
|
||||
- CCAFS finally confirmed that they do indeed need the confusing new project tag that looks like a duplicate
|
||||
- They had proposed a batch of new tags in 2019-09 and we never merged them due to this uncertainty
|
||||
- I have now merged the changes in to the `5_x-prod` branch ([#432](https://github.com/ilri/DSpace/pull/432))
|
||||
- I am reconsidering the move of `cg.identifier.dataurl` to `cg.hasMetadata` in CG Core v2
|
||||
- The values of this field are mostly links to data sets on Dataverse and partner sites
|
||||
- I opened an [issue on GitHub](https://github.com/AgriculturalSemantics/cg-core/issues/10) to ask Marie-Angelique for clarification
|
||||
- Looking into CGSpace statistics again
|
||||
- I searched for hits in Solr from the BUbiNG bot and found 63,000 in the `statistics-2018` core:
|
||||
|
||||
</code></pre>
|
||||
|
||||
<p>$ http –print b ‘<a href="http://localhost:8081/solr/statistics-2018/select?facet=true&facet.field=ip&facet.mincount=1&type:0&q=userAgent:BUbiNG*'">http://localhost:8081/solr/statistics-2018/select?facet=true&facet.field=ip&facet.mincount=1&type:0&q=userAgent:BUbiNG*'</a> | xmllint –format - | grep numFound
|
||||
<result name="response" numFound="62944" start="0"></p>
|
||||
|
||||
<pre><code>
|
||||
- Similar for com.plumanalytics, Grammarly, and ltx71!
|
||||
|
||||
</code></pre>
|
||||
|
||||
<p>$ http –print b ‘<a href="http://localhost:8081/solr/statistics-2018/select?facet=true&facet.field=ip&facet.mincount=1&type:0&q=userAgent:">http://localhost:8081/solr/statistics-2018/select?facet=true&facet.field=ip&facet.mincount=1&type:0&q=userAgent:</a>
|
||||
<em>com.plumanalytics</em>’ | xmllint –format - | grep numFound
|
||||
<result name="response" numFound="28256" start="0">
|
||||
$ http –print b ‘<a href="http://localhost:8081/solr/statistics-2018/select?facet=true&facet.field=ip&facet.mincount=1&type:0&q=userAgent:*Grammarly*'">http://localhost:8081/solr/statistics-2018/select?facet=true&facet.field=ip&facet.mincount=1&type:0&q=userAgent:*Grammarly*'</a> | xmllint –format - | grep numFound
|
||||
<result name="response" numFound="6288" start="0">
|
||||
$ http –print b ‘<a href="http://localhost:8081/solr/statistics-2018/select?facet=true&facet.field=ip&facet.mincount=1&type:0&q=userAgent:*ltx71*'">http://localhost:8081/solr/statistics-2018/select?facet=true&facet.field=ip&facet.mincount=1&type:0&q=userAgent:*ltx71*'</a> | xmllint –format - | grep numFound
|
||||
<result name="response" numFound="105663" start="0"></p>
|
||||
|
||||
<pre><code>
|
||||
- Deleting these seems to work, for example the 105,000 ltx71 records from 2018:
|
||||
|
||||
</code></pre>
|
||||
|
||||
<p>$ http –print b ‘<a href="http://localhost:8081/solr/statistics-2018/update?stream.body=">http://localhost:8081/solr/statistics-2018/update?stream.body=</a><delete><query>userAgent:<em>ltx71</em></query><query>type:0</query></delete>&commit=true’
|
||||
$ http –print b ‘<a href="http://localhost:8081/solr/statistics-2018/select?facet=true&facet.field=ip&facet.mincount=1&type:0&q=userAgent:*ltx71*'">http://localhost:8081/solr/statistics-2018/select?facet=true&facet.field=ip&facet.mincount=1&type:0&q=userAgent:*ltx71*'</a> | xmllint –format - | grep numFound
|
||||
<result name="response" numFound="0" start="0"/>
|
||||
```</p>
|
||||
|
||||
<ul>
|
||||
<li>So the blocking seems to be working because “www.gnip.com” is one of the new patterns added to the spiders file…</li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2019-11-07">2019-11-07</h2>
|
||||
<li>I wrote a quick bash script to check all these user agents against the CGSpace Solr statistics cores
|
||||
|
||||
<ul>
|
||||
<li>CCAFS finally confirmed that they do indeed need the confusing new project tag that looks like a duplicate
|
||||
|
||||
<ul>
|
||||
<li>They had proposed a batch of new tags in 2019-09 and we never merged them due to this uncertainty</li>
|
||||
<li>I have now merged the changes in to the <code>5_x-prod</code> branch (<a href="https://github.com/ilri/DSpace/pull/432">#432</a>)</li>
|
||||
<li>For years 2010 until 2019 there are 1.6 million hits from these spider user agents</li>
|
||||
<li>For 2019 alone there are 740,000, over half of which come from Unpaywall!</li>
|
||||
</ul></li>
|
||||
</ul>
|
||||
|
||||
|
@ -4,27 +4,27 @@
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/categories/</loc>
|
||||
<lastmod>2019-11-06T09:35:51+02:00</lastmod>
|
||||
<lastmod>2019-11-07T12:40:25+02:00</lastmod>
|
||||
</url>
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/</loc>
|
||||
<lastmod>2019-11-06T09:35:51+02:00</lastmod>
|
||||
<lastmod>2019-11-07T12:40:25+02:00</lastmod>
|
||||
</url>
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/categories/notes/</loc>
|
||||
<lastmod>2019-11-06T09:35:51+02:00</lastmod>
|
||||
<lastmod>2019-11-07T12:40:25+02:00</lastmod>
|
||||
</url>
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/2019-11/</loc>
|
||||
<lastmod>2019-11-06T09:35:51+02:00</lastmod>
|
||||
<lastmod>2019-11-07T12:40:25+02:00</lastmod>
|
||||
</url>
|
||||
|
||||
<url>
|
||||
<loc>https://alanorth.github.io/cgspace-notes/posts/</loc>
|
||||
<lastmod>2019-11-06T09:35:51+02:00</lastmod>
|
||||
<lastmod>2019-11-07T12:40:25+02:00</lastmod>
|
||||
</url>
|
||||
|
||||
<url>
|
||||
|
Loading…
Reference in New Issue
Block a user