mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2025-01-27 05:49:12 +01:00
Add notes for 2022-03-04
This commit is contained in:
@ -50,7 +50,7 @@ For example, this item has 51 views on CGSpace, but 0 on AReS
|
||||
|
||||
|
||||
"/>
|
||||
<meta name="generator" content="Hugo 0.92.2" />
|
||||
<meta name="generator" content="Hugo 0.93.1" />
|
||||
|
||||
|
||||
|
||||
@ -160,29 +160,29 @@ For example, this item has 51 views on CGSpace, but 0 on AReS
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp'</span>
|
||||
# start indexing in AReS
|
||||
</code></pre></div><ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp'</span>
|
||||
</span></span><span style="display:flex;"><span># start indexing in AReS
|
||||
</span></span></code></pre></div><ul>
|
||||
<li>Then, the next morning when it’s done, check the results of the harvesting, backup the current <code>openrxv-items</code> index, and clone the <code>openrxv-items-temp</code> index to <code>openrxv-items</code>:</li>
|
||||
</ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ curl -s <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'</span>
|
||||
{
|
||||
"count" : 100278,
|
||||
"_shards" : {
|
||||
"total" : 1,
|
||||
"successful" : 1,
|
||||
"skipped" : 0,
|
||||
"failed" : 0
|
||||
}
|
||||
}
|
||||
$ curl -X PUT <span style="color:#e6db74">"localhost:9200/openrxv-items/_settings"</span> -H <span style="color:#e6db74">'Content-Type: application/json'</span> -d<span style="color:#e6db74">'{"settings": {"index.blocks.write": true}}'</span>
|
||||
$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-2021-01-04
|
||||
$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items'</span>
|
||||
$ curl -X PUT <span style="color:#e6db74">"localhost:9200/openrxv-items-temp/_settings"</span> -H <span style="color:#e6db74">'Content-Type: application/json'</span> -d<span style="color:#e6db74">'{"settings": {"index.blocks.write": true}}'</span>
|
||||
$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items
|
||||
$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp'</span>
|
||||
$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-2021-01-04'</span>
|
||||
</code></pre></div><h2 id="2021-01-04">2021-01-04</h2>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ curl -s <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'</span>
|
||||
</span></span><span style="display:flex;"><span>{
|
||||
</span></span><span style="display:flex;"><span> "count" : 100278,
|
||||
</span></span><span style="display:flex;"><span> "_shards" : {
|
||||
</span></span><span style="display:flex;"><span> "total" : 1,
|
||||
</span></span><span style="display:flex;"><span> "successful" : 1,
|
||||
</span></span><span style="display:flex;"><span> "skipped" : 0,
|
||||
</span></span><span style="display:flex;"><span> "failed" : 0
|
||||
</span></span><span style="display:flex;"><span> }
|
||||
</span></span><span style="display:flex;"><span>}
|
||||
</span></span><span style="display:flex;"><span>$ curl -X PUT <span style="color:#e6db74">"localhost:9200/openrxv-items/_settings"</span> -H <span style="color:#e6db74">'Content-Type: application/json'</span> -d<span style="color:#e6db74">'{"settings": {"index.blocks.write": true}}'</span>
|
||||
</span></span><span style="display:flex;"><span>$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-2021-01-04
|
||||
</span></span><span style="display:flex;"><span>$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items'</span>
|
||||
</span></span><span style="display:flex;"><span>$ curl -X PUT <span style="color:#e6db74">"localhost:9200/openrxv-items-temp/_settings"</span> -H <span style="color:#e6db74">'Content-Type: application/json'</span> -d<span style="color:#e6db74">'{"settings": {"index.blocks.write": true}}'</span>
|
||||
</span></span><span style="display:flex;"><span>$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items
|
||||
</span></span><span style="display:flex;"><span>$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp'</span>
|
||||
</span></span><span style="display:flex;"><span>$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-2021-01-04'</span>
|
||||
</span></span></code></pre></div><h2 id="2021-01-04">2021-01-04</h2>
|
||||
<ul>
|
||||
<li>There is one item that appears twice in AReS: <a href="https://cgspace.cgiar.org/handle/10568/66839">10568/66839</a>
|
||||
<ul>
|
||||
@ -214,8 +214,8 @@ $ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-i
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ ./doi-to-handle.py -db dspace -u dspace -p <span style="color:#e6db74">'fuuu'</span> -i /tmp/dois.txt -o /tmp/out.csv
|
||||
</code></pre></div><ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ ./doi-to-handle.py -db dspace -u dspace -p <span style="color:#e6db74">'fuuu'</span> -i /tmp/dois.txt -o /tmp/out.csv
|
||||
</span></span></code></pre></div><ul>
|
||||
<li>Help Udana export IWMI records from AReS
|
||||
<ul>
|
||||
<li>He wanted me to give him CSV export permissions on CGSpace, but I told him that this requires super admin so I’m not comfortable with it</li>
|
||||
@ -261,28 +261,28 @@ $ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-i
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">2021-01-10 10:03:27,692 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY, SubjectType=ITEM, SubjectID=1e8fb96c-b994-4fe2-8f0c-0a98ab138be0, ObjectType=(Unknown), ObjectID=null, TimeStamp=1610269383279, dispatcher=1544803905, detail=[null], transactionID="TX35636856957739531161091194485578658698")
|
||||
</code></pre></div><ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>2021-01-10 10:03:27,692 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY, SubjectType=ITEM, SubjectID=1e8fb96c-b994-4fe2-8f0c-0a98ab138be0, ObjectType=(Unknown), ObjectID=null, TimeStamp=1610269383279, dispatcher=1544803905, detail=[null], transactionID="TX35636856957739531161091194485578658698")
|
||||
</span></span></code></pre></div><ul>
|
||||
<li>I filed <a href="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=907">a bug on Atmire’s issue tracker</a></li>
|
||||
<li>Peter asked me to move the CGIAR Gender Platform community to the top level of CGSpace, but I get an error when I use the community-filiator command:</li>
|
||||
</ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ dspace community-filiator --remove --parent<span style="color:#f92672">=</span>10568/66598 --child<span style="color:#f92672">=</span>10568/106605
|
||||
Loading @mire database changes for module MQM
|
||||
Changes have been processed
|
||||
Exception: null
|
||||
java.lang.UnsupportedOperationException
|
||||
at java.util.AbstractList.remove(AbstractList.java:161)
|
||||
at java.util.AbstractList$Itr.remove(AbstractList.java:374)
|
||||
at java.util.AbstractCollection.remove(AbstractCollection.java:293)
|
||||
at org.dspace.administer.CommunityFiliator.defiliate(CommunityFiliator.java:264)
|
||||
at org.dspace.administer.CommunityFiliator.main(CommunityFiliator.java:164)
|
||||
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
|
||||
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
|
||||
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
|
||||
at java.lang.reflect.Method.invoke(Method.java:498)
|
||||
at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
|
||||
at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
|
||||
</code></pre></div><ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ dspace community-filiator --remove --parent<span style="color:#f92672">=</span>10568/66598 --child<span style="color:#f92672">=</span>10568/106605
|
||||
</span></span><span style="display:flex;"><span>Loading @mire database changes for module MQM
|
||||
</span></span><span style="display:flex;"><span>Changes have been processed
|
||||
</span></span><span style="display:flex;"><span>Exception: null
|
||||
</span></span><span style="display:flex;"><span>java.lang.UnsupportedOperationException
|
||||
</span></span><span style="display:flex;"><span> at java.util.AbstractList.remove(AbstractList.java:161)
|
||||
</span></span><span style="display:flex;"><span> at java.util.AbstractList$Itr.remove(AbstractList.java:374)
|
||||
</span></span><span style="display:flex;"><span> at java.util.AbstractCollection.remove(AbstractCollection.java:293)
|
||||
</span></span><span style="display:flex;"><span> at org.dspace.administer.CommunityFiliator.defiliate(CommunityFiliator.java:264)
|
||||
</span></span><span style="display:flex;"><span> at org.dspace.administer.CommunityFiliator.main(CommunityFiliator.java:164)
|
||||
</span></span><span style="display:flex;"><span> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
|
||||
</span></span><span style="display:flex;"><span> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
|
||||
</span></span><span style="display:flex;"><span> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
|
||||
</span></span><span style="display:flex;"><span> at java.lang.reflect.Method.invoke(Method.java:498)
|
||||
</span></span><span style="display:flex;"><span> at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)
|
||||
</span></span><span style="display:flex;"><span> at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)
|
||||
</span></span></code></pre></div><ul>
|
||||
<li>There is apparently <a href="https://jira.lyrasis.org/browse/DS-3914">a bug</a> in DSpace 6.x that makes community-filiator not work
|
||||
<ul>
|
||||
<li>There is <a href="https://github.com/DSpace/DSpace/pull/2178">a patch</a> for the as-of-yet unreleased DSpace 6.4 so I will try that</li>
|
||||
@ -301,24 +301,24 @@ java.lang.UnsupportedOperationException
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp'</span>
|
||||
# start indexing in AReS
|
||||
... after ten hours
|
||||
$ curl -s <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'</span>
|
||||
{
|
||||
"count" : 100411,
|
||||
"_shards" : {
|
||||
"total" : 1,
|
||||
"successful" : 1,
|
||||
"skipped" : 0,
|
||||
"failed" : 0
|
||||
}
|
||||
}
|
||||
$ curl -X PUT <span style="color:#e6db74">"localhost:9200/openrxv-items-temp/_settings?pretty"</span> -H <span style="color:#e6db74">'Content-Type: application/json'</span> -d<span style="color:#e6db74">'{"settings": {"index.blocks.write": true}}'</span>
|
||||
$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items'</span>
|
||||
$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items
|
||||
$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp'</span>
|
||||
</code></pre></div><ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp'</span>
|
||||
</span></span><span style="display:flex;"><span># start indexing in AReS
|
||||
</span></span><span style="display:flex;"><span>... after ten hours
|
||||
</span></span><span style="display:flex;"><span>$ curl -s <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'</span>
|
||||
</span></span><span style="display:flex;"><span>{
|
||||
</span></span><span style="display:flex;"><span> "count" : 100411,
|
||||
</span></span><span style="display:flex;"><span> "_shards" : {
|
||||
</span></span><span style="display:flex;"><span> "total" : 1,
|
||||
</span></span><span style="display:flex;"><span> "successful" : 1,
|
||||
</span></span><span style="display:flex;"><span> "skipped" : 0,
|
||||
</span></span><span style="display:flex;"><span> "failed" : 0
|
||||
</span></span><span style="display:flex;"><span> }
|
||||
</span></span><span style="display:flex;"><span>}
|
||||
</span></span><span style="display:flex;"><span>$ curl -X PUT <span style="color:#e6db74">"localhost:9200/openrxv-items-temp/_settings?pretty"</span> -H <span style="color:#e6db74">'Content-Type: application/json'</span> -d<span style="color:#e6db74">'{"settings": {"index.blocks.write": true}}'</span>
|
||||
</span></span><span style="display:flex;"><span>$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items'</span>
|
||||
</span></span><span style="display:flex;"><span>$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items
|
||||
</span></span><span style="display:flex;"><span>$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp'</span>
|
||||
</span></span></code></pre></div><ul>
|
||||
<li>Looking over the last month of Solr stats I see a familiar bot that <em>should</em> have been marked as a bot months ago:</li>
|
||||
</ul>
|
||||
<blockquote>
|
||||
@ -331,9 +331,9 @@ $ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-i
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ cat log/dspace.log.2020-12-2* | grep -E <span style="color:#e6db74">'session_id=[A-Z0-9]{32}:ip_addr=64.62.202.71'</span> | sort | uniq | wc -l
|
||||
0
|
||||
</code></pre></div><ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ cat log/dspace.log.2020-12-2* | grep -E <span style="color:#e6db74">'session_id=[A-Z0-9]{32}:ip_addr=64.62.202.71'</span> | sort | uniq | wc -l
|
||||
</span></span><span style="display:flex;"><span>0
|
||||
</span></span></code></pre></div><ul>
|
||||
<li>So now I should really add it to the DSpace spider agent list so it doesn’t create Solr hits
|
||||
<ul>
|
||||
<li>I added it to the “ilri” lists of spider agent patterns</li>
|
||||
@ -341,8 +341,8 @@ $ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-i
|
||||
</li>
|
||||
<li>I purged the existing hits using my <code>check-spider-ip-hits.sh</code> script:</li>
|
||||
</ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ ./check-spider-ip-hits.sh -d -f /tmp/ips -s http://localhost:8081/solr -s statistics -p
|
||||
</code></pre></div><h2 id="2021-01-11">2021-01-11</h2>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ ./check-spider-ip-hits.sh -d -f /tmp/ips -s http://localhost:8081/solr -s statistics -p
|
||||
</span></span></code></pre></div><h2 id="2021-01-11">2021-01-11</h2>
|
||||
<ul>
|
||||
<li>The AReS indexing finished this morning and I moved the <code>openrxv-items-temp</code> core to <code>openrxv-items</code> (see above)
|
||||
<ul>
|
||||
@ -351,8 +351,8 @@ $ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-i
|
||||
</li>
|
||||
<li>I deployed the community-filiator fix on CGSpace and moved the Gender Platform community to the top level of CGSpace:</li>
|
||||
</ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ dspace community-filiator --remove --parent<span style="color:#f92672">=</span>10568/66598 --child<span style="color:#f92672">=</span>10568/106605
|
||||
</code></pre></div><h2 id="2021-01-12">2021-01-12</h2>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ dspace community-filiator --remove --parent<span style="color:#f92672">=</span>10568/66598 --child<span style="color:#f92672">=</span>10568/106605
|
||||
</span></span></code></pre></div><h2 id="2021-01-12">2021-01-12</h2>
|
||||
<ul>
|
||||
<li>IWMI is really pressuring us to have a periodic CSV export of their community
|
||||
<ul>
|
||||
@ -393,29 +393,29 @@ $ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-i
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp'</span>
|
||||
# start indexing in AReS
|
||||
</code></pre></div><ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp'</span>
|
||||
</span></span><span style="display:flex;"><span># start indexing in AReS
|
||||
</span></span></code></pre></div><ul>
|
||||
<li>Then, the next morning when it’s done, check the results of the harvesting, backup the current <code>openrxv-items</code> index, and clone the <code>openrxv-items-temp</code> index to <code>openrxv-items</code>:</li>
|
||||
</ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ curl -s <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'</span>
|
||||
{
|
||||
"count" : 100540,
|
||||
"_shards" : {
|
||||
"total" : 1,
|
||||
"successful" : 1,
|
||||
"skipped" : 0,
|
||||
"failed" : 0
|
||||
}
|
||||
}
|
||||
$ curl -X PUT <span style="color:#e6db74">"localhost:9200/openrxv-items/_settings"</span> -H <span style="color:#e6db74">'Content-Type: application/json'</span> -d<span style="color:#e6db74">'{"settings": {"index.blocks.write": true}}'</span>
|
||||
$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-2021-01-18
|
||||
$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items'</span>
|
||||
$ curl -X PUT <span style="color:#e6db74">"localhost:9200/openrxv-items-temp/_settings"</span> -H <span style="color:#e6db74">'Content-Type: application/json'</span> -d<span style="color:#e6db74">'{"settings": {"index.blocks.write": true}}'</span>
|
||||
$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items
|
||||
$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp'</span>
|
||||
$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-2021-01-18'</span>
|
||||
</code></pre></div><h2 id="2021-01-18">2021-01-18</h2>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ curl -s <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'</span>
|
||||
</span></span><span style="display:flex;"><span>{
|
||||
</span></span><span style="display:flex;"><span> "count" : 100540,
|
||||
</span></span><span style="display:flex;"><span> "_shards" : {
|
||||
</span></span><span style="display:flex;"><span> "total" : 1,
|
||||
</span></span><span style="display:flex;"><span> "successful" : 1,
|
||||
</span></span><span style="display:flex;"><span> "skipped" : 0,
|
||||
</span></span><span style="display:flex;"><span> "failed" : 0
|
||||
</span></span><span style="display:flex;"><span> }
|
||||
</span></span><span style="display:flex;"><span>}
|
||||
</span></span><span style="display:flex;"><span>$ curl -X PUT <span style="color:#e6db74">"localhost:9200/openrxv-items/_settings"</span> -H <span style="color:#e6db74">'Content-Type: application/json'</span> -d<span style="color:#e6db74">'{"settings": {"index.blocks.write": true}}'</span>
|
||||
</span></span><span style="display:flex;"><span>$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-2021-01-18
|
||||
</span></span><span style="display:flex;"><span>$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items'</span>
|
||||
</span></span><span style="display:flex;"><span>$ curl -X PUT <span style="color:#e6db74">"localhost:9200/openrxv-items-temp/_settings"</span> -H <span style="color:#e6db74">'Content-Type: application/json'</span> -d<span style="color:#e6db74">'{"settings": {"index.blocks.write": true}}'</span>
|
||||
</span></span><span style="display:flex;"><span>$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items
|
||||
</span></span><span style="display:flex;"><span>$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp'</span>
|
||||
</span></span><span style="display:flex;"><span>$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-2021-01-18'</span>
|
||||
</span></span></code></pre></div><h2 id="2021-01-18">2021-01-18</h2>
|
||||
<ul>
|
||||
<li>Finish the indexing on AReS that I started yesterday</li>
|
||||
<li>Udana from IWMI emailed me to ask why the iwmi.csv doesn’t include items he approved to CGSpace this morning
|
||||
@ -462,9 +462,9 @@ localhost/dspace63= > COMMIT;
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ docker exec -it api /bin/bash
|
||||
# apt update <span style="color:#f92672">&&</span> apt install unoconv
|
||||
</code></pre></div><ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ docker exec -it api /bin/bash
|
||||
</span></span><span style="display:flex;"><span># apt update <span style="color:#f92672">&&</span> apt install unoconv
|
||||
</span></span></code></pre></div><ul>
|
||||
<li>Help Peter get a list of titles and DOIs for CGSpace items that Altmetric does not have an attention score for
|
||||
<ul>
|
||||
<li>He generated a list from their dashboard and I extracted the DOIs in OpenRefine (because it was WINDOWS-1252 and csvcut couldn’t do it)</li>
|
||||
@ -512,30 +512,30 @@ localhost/dspace63= > COMMIT;
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp'</span>
|
||||
# start indexing in AReS
|
||||
</code></pre></div><ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp'</span>
|
||||
</span></span><span style="display:flex;"><span># start indexing in AReS
|
||||
</span></span></code></pre></div><ul>
|
||||
<li>Then, the next morning when it’s done, check the results of the harvesting, backup the current <code>openrxv-items</code> index, and clone the <code>openrxv-items-temp</code> index to <code>openrxv-items</code>:</li>
|
||||
</ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ curl -s <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'</span>
|
||||
{
|
||||
"count" : 100699,
|
||||
"_shards" : {
|
||||
"total" : 1,
|
||||
"successful" : 1,
|
||||
"skipped" : 0,
|
||||
"failed" : 0
|
||||
}
|
||||
}
|
||||
$ curl -X PUT <span style="color:#e6db74">"localhost:9200/openrxv-items/_settings"</span> -H <span style="color:#e6db74">'Content-Type: application/json'</span> -d<span style="color:#960050;background-color:#1e0010">'</span><span style="color:#f92672">{</span><span style="color:#e6db74">"settings"</span>: <span style="color:#f92672">{</span><span style="color:#e6db74">"index.b
|
||||
</span><span style="color:#e6db74"></span>locks.write":true}}'
|
||||
$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-2021-01-25
|
||||
$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items'</span>
|
||||
$ curl -X PUT <span style="color:#e6db74">"localhost:9200/openrxv-items-temp/_settings"</span> -H <span style="color:#e6db74">'Content-Type: application/json'</span> -d<span style="color:#e6db74">'{"settings": {"index.blocks.write": true}}'</span>
|
||||
$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items
|
||||
$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp'</span>
|
||||
$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-2021-01-25'</span>
|
||||
</code></pre></div><ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ curl -s <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp/_count?q=*&pretty'</span>
|
||||
</span></span><span style="display:flex;"><span>{
|
||||
</span></span><span style="display:flex;"><span> "count" : 100699,
|
||||
</span></span><span style="display:flex;"><span> "_shards" : {
|
||||
</span></span><span style="display:flex;"><span> "total" : 1,
|
||||
</span></span><span style="display:flex;"><span> "successful" : 1,
|
||||
</span></span><span style="display:flex;"><span> "skipped" : 0,
|
||||
</span></span><span style="display:flex;"><span> "failed" : 0
|
||||
</span></span><span style="display:flex;"><span> }
|
||||
</span></span><span style="display:flex;"><span>}
|
||||
</span></span><span style="display:flex;"><span>$ curl -X PUT <span style="color:#e6db74">"localhost:9200/openrxv-items/_settings"</span> -H <span style="color:#e6db74">'Content-Type: application/json'</span> -d<span style="color:#960050;background-color:#1e0010">'</span><span style="color:#f92672">{</span><span style="color:#e6db74">"settings"</span>: <span style="color:#f92672">{</span><span style="color:#e6db74">"index.b
|
||||
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74"></span>locks.write":true}}'
|
||||
</span></span><span style="display:flex;"><span>$ curl -s -X POST http://localhost:9200/openrxv-items/_clone/openrxv-items-2021-01-25
|
||||
</span></span><span style="display:flex;"><span>$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items'</span>
|
||||
</span></span><span style="display:flex;"><span>$ curl -X PUT <span style="color:#e6db74">"localhost:9200/openrxv-items-temp/_settings"</span> -H <span style="color:#e6db74">'Content-Type: application/json'</span> -d<span style="color:#e6db74">'{"settings": {"index.blocks.write": true}}'</span>
|
||||
</span></span><span style="display:flex;"><span>$ curl -s -X POST http://localhost:9200/openrxv-items-temp/_clone/openrxv-items
|
||||
</span></span><span style="display:flex;"><span>$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp'</span>
|
||||
</span></span><span style="display:flex;"><span>$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-2021-01-25'</span>
|
||||
</span></span></code></pre></div><ul>
|
||||
<li>Resume working on CG Core v2, I realized a few things:
|
||||
<ul>
|
||||
<li>We are trying to move from <code>dc.identifier.issn</code> (and ISBN) to <code>cg.issn</code>, but this is currently implemented as a “qualdrop” input in DSpace’s submission form, which only works to fill in the qualifier (ie <code>dc.identier.xxxx</code>)
|
||||
@ -601,12 +601,12 @@ java.lang.IllegalArgumentException: Invalid character found in the request targe
|
||||
<li>I <a href="https://jira.lyrasis.org/browse/DS-4566">filed a bug</a> on DSpace’s issue tracker (though I accidentally hit Enter and submitted it before I finished, and there is no edit function)</li>
|
||||
<li>Looking into Linode report that the load outbound traffic rate was high this morning:</li>
|
||||
</ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console"># grep -E <span style="color:#e6db74">'26/Jan/2021:(08|09|10|11|12)'</span> /var/log/nginx/rest.log | goaccess --log-format<span style="color:#f92672">=</span>COMBINED -
|
||||
</code></pre></div><ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span># grep -E <span style="color:#e6db74">'26/Jan/2021:(08|09|10|11|12)'</span> /var/log/nginx/rest.log | goaccess --log-format<span style="color:#f92672">=</span>COMBINED -
|
||||
</span></span></code></pre></div><ul>
|
||||
<li>The culprit seems to be the ILRI publications importer, so that’s OK</li>
|
||||
<li>But I also see an IP in Jordan hitting the REST API 1,100 times today:</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>80.10.12.54 - - [26/Jan/2021:09:43:42 +0100] "GET /rest/rest/bitstreams/98309f17-a831-48ed-8f0a-2d3244cc5a1c/retrieve HTTP/2.0" 302 138 "http://wp.local/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36"
|
||||
<pre tabindex="0"><code>80.10.12.54 - - [26/Jan/2021:09:43:42 +0100] "GET /rest/rest/bitstreams/98309f17-a831-48ed-8f0a-2d3244cc5a1c/retrieve HTTP/2.0" 302 138 "http://wp.local/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36"
|
||||
</code></pre><ul>
|
||||
<li>Seems to be someone from CodeObia working on WordPress
|
||||
<ul>
|
||||
@ -615,8 +615,8 @@ java.lang.IllegalArgumentException: Invalid character found in the request targe
|
||||
</li>
|
||||
<li>I purged all ~3,000 statistics hits that have the “<a href="http://wp.local/%22">http://wp.local/"</a> referrer:</li>
|
||||
</ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ curl -s <span style="color:#e6db74">"http://localhost:8081/solr/statistics/update?softCommit=true"</span> -H <span style="color:#e6db74">"Content-Type: text/xml"</span> --data-binary <span style="color:#e6db74">"<delete><query>referrer:http\:\/\/wp\.local\/</query></delete>"</span>
|
||||
</code></pre></div><ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ curl -s <span style="color:#e6db74">"http://localhost:8081/solr/statistics/update?softCommit=true"</span> -H <span style="color:#e6db74">"Content-Type: text/xml"</span> --data-binary <span style="color:#e6db74">"<delete><query>referrer:http\:\/\/wp\.local\/</query></delete>"</span>
|
||||
</span></span></code></pre></div><ul>
|
||||
<li>Tag version 0.4.3 of the csv-metadata-quality tool on GitHub: <a href="https://github.com/ilri/csv-metadata-quality/releases/tag/v0.4.3">https://github.com/ilri/csv-metadata-quality/releases/tag/v0.4.3</a>
|
||||
<ul>
|
||||
<li>I just realized that I never submitted this to CGSpace as a Big Data Platform output</li>
|
||||
@ -661,9 +661,9 @@ java.lang.IllegalArgumentException: Invalid character found in the request targe
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp'</span>
|
||||
# start indexing in AReS
|
||||
</code></pre></div><ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-console" data-lang="console"><span style="display:flex;"><span>$ curl -XDELETE <span style="color:#e6db74">'http://localhost:9200/openrxv-items-temp'</span>
|
||||
</span></span><span style="display:flex;"><span># start indexing in AReS
|
||||
</span></span></code></pre></div><ul>
|
||||
<li>Sent out emails about CG Core v2 to Macaroni Bros, Fabio, Hector at CCAFS, Dani and Tariku</li>
|
||||
<li>A bit more minor work on testing the series/report/journal changes for CG Core v2</li>
|
||||
</ul>
|
||||
|
||||
Reference in New Issue
Block a user