mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2025-01-27 05:49:12 +01:00
Add notes for 2022-03-04
This commit is contained in:
@ -48,7 +48,7 @@ The third item now has a donut with score 1 since I tweeted it last week
|
||||
|
||||
On the same note, the one item Abenet pointed out last week now has a donut with score of 104 after I tweeted it last week
|
||||
"/>
|
||||
<meta name="generator" content="Hugo 0.92.2" />
|
||||
<meta name="generator" content="Hugo 0.93.1" />
|
||||
|
||||
|
||||
|
||||
@ -171,14 +171,14 @@ On the same note, the one item Abenet pointed out last week now has a donut with
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>$ psql -h localhost -U postgres dspace -c "DELETE FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=240 AND text_value LIKE '%Ballantyne%';"
|
||||
<pre tabindex="0"><code>$ psql -h localhost -U postgres dspace -c "DELETE FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=240 AND text_value LIKE '%Ballantyne%';"
|
||||
DELETE 97
|
||||
$ ./add-orcid-identifiers-csv.py -i 2020-04-07-peter-orcids.csv -db dspace -u dspace -p 'fuuu' -d
|
||||
$ ./add-orcid-identifiers-csv.py -i 2020-04-07-peter-orcids.csv -db dspace -u dspace -p 'fuuu' -d
|
||||
</code></pre><ul>
|
||||
<li>I used this CSV with the script (all records with his name have the name standardized like this):</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>dc.contributor.author,cg.creator.id
|
||||
"Ballantyne, Peter G.","Peter G. Ballantyne: 0000-0001-9346-2893"
|
||||
"Ballantyne, Peter G.","Peter G. Ballantyne: 0000-0001-9346-2893"
|
||||
</code></pre><ul>
|
||||
<li>Then I tried another way, to identify all duplicate ORCID identifiers for a given resource ID and group them so I can see if count is greater than 1:</li>
|
||||
</ul>
|
||||
@ -188,31 +188,31 @@ COPY 15209
|
||||
<li>Of those, about nine authors had duplicate ORCID identifiers over about thirty records, so I created a CSV with all their name variations and ORCID identifiers:</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>dc.contributor.author,cg.creator.id
|
||||
"Ballantyne, Peter G.","Peter G. Ballantyne: 0000-0001-9346-2893"
|
||||
"Ramirez-Villegas, Julian","Julian Ramirez-Villegas: 0000-0002-8044-583X"
|
||||
"Villegas-Ramirez, J","Julian Ramirez-Villegas: 0000-0002-8044-583X"
|
||||
"Ishitani, Manabu","Manabu Ishitani: 0000-0002-6950-4018"
|
||||
"Manabu, Ishitani","Manabu Ishitani: 0000-0002-6950-4018"
|
||||
"Ishitani, M.","Manabu Ishitani: 0000-0002-6950-4018"
|
||||
"Ishitani, M.","Manabu Ishitani: 0000-0002-6950-4018"
|
||||
"Buruchara, Robin A.","Robin Buruchara: 0000-0003-0934-1218"
|
||||
"Buruchara, Robin","Robin Buruchara: 0000-0003-0934-1218"
|
||||
"Jarvis, Andy","Andy Jarvis: 0000-0001-6543-0798"
|
||||
"Jarvis, Andrew","Andy Jarvis: 0000-0001-6543-0798"
|
||||
"Jarvis, A.","Andy Jarvis: 0000-0001-6543-0798"
|
||||
"Tohme, Joseph M.","Joe Tohme: 0000-0003-2765-7101"
|
||||
"Hansen, James","James Hansen: 0000-0002-8599-7895"
|
||||
"Hansen, James W.","James Hansen: 0000-0002-8599-7895"
|
||||
"Asseng, Senthold","Senthold Asseng: 0000-0002-7583-3811"
|
||||
"Ballantyne, Peter G.","Peter G. Ballantyne: 0000-0001-9346-2893"
|
||||
"Ramirez-Villegas, Julian","Julian Ramirez-Villegas: 0000-0002-8044-583X"
|
||||
"Villegas-Ramirez, J","Julian Ramirez-Villegas: 0000-0002-8044-583X"
|
||||
"Ishitani, Manabu","Manabu Ishitani: 0000-0002-6950-4018"
|
||||
"Manabu, Ishitani","Manabu Ishitani: 0000-0002-6950-4018"
|
||||
"Ishitani, M.","Manabu Ishitani: 0000-0002-6950-4018"
|
||||
"Ishitani, M.","Manabu Ishitani: 0000-0002-6950-4018"
|
||||
"Buruchara, Robin A.","Robin Buruchara: 0000-0003-0934-1218"
|
||||
"Buruchara, Robin","Robin Buruchara: 0000-0003-0934-1218"
|
||||
"Jarvis, Andy","Andy Jarvis: 0000-0001-6543-0798"
|
||||
"Jarvis, Andrew","Andy Jarvis: 0000-0001-6543-0798"
|
||||
"Jarvis, A.","Andy Jarvis: 0000-0001-6543-0798"
|
||||
"Tohme, Joseph M.","Joe Tohme: 0000-0003-2765-7101"
|
||||
"Hansen, James","James Hansen: 0000-0002-8599-7895"
|
||||
"Hansen, James W.","James Hansen: 0000-0002-8599-7895"
|
||||
"Asseng, Senthold","Senthold Asseng: 0000-0002-7583-3811"
|
||||
</code></pre><ul>
|
||||
<li>Then I deleted <em>all</em> their existing ORCID identifier records:</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>dspace=# DELETE FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=240 AND text_value SIMILAR TO '%(0000-0001-6543-0798|0000-0001-9346-2893|0000-0002-6950-4018|0000-0002-7583-3811|0000-0002-8044-583X|0000-0002-8599-7895|0000-0003-0934-1218|0000-0003-2765-7101)%';
|
||||
<pre tabindex="0"><code>dspace=# DELETE FROM metadatavalue WHERE resource_type_id=2 AND metadata_field_id=240 AND text_value SIMILAR TO '%(0000-0001-6543-0798|0000-0001-9346-2893|0000-0002-6950-4018|0000-0002-7583-3811|0000-0002-8044-583X|0000-0002-8599-7895|0000-0003-0934-1218|0000-0003-2765-7101)%';
|
||||
DELETE 994
|
||||
</code></pre><ul>
|
||||
<li>And then I added them again using the <code>add-orcid-identifiers</code> records:</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>$ ./add-orcid-identifiers-csv.py -i 2020-04-07-fix-duplicate-orcids.csv -db dspace -u dspace -p 'fuuu' -d
|
||||
<pre tabindex="0"><code>$ ./add-orcid-identifiers-csv.py -i 2020-04-07-fix-duplicate-orcids.csv -db dspace -u dspace -p 'fuuu' -d
|
||||
</code></pre><ul>
|
||||
<li>I ran the fixes on DSpace Test and CGSpace as well</li>
|
||||
<li>I started testing the <a href="https://github.com/ilri/DSpace/pull/445">pull request</a> sent by Atmire yesterday
|
||||
@ -230,7 +230,7 @@ DELETE 994
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>dspace63=# DELETE FROM schema_version WHERE version IN ('5.8.2015.12.03.3');
|
||||
<pre tabindex="0"><code>dspace63=# DELETE FROM schema_version WHERE version IN ('5.8.2015.12.03.3');
|
||||
dspace63=# CREATE EXTENSION pgcrypto;
|
||||
</code></pre><ul>
|
||||
<li>Then DSpace 6.3 started up OK and I was able to see some statistics in the Content and Usage Analysis (CUA) module, but not on community, collection, or item pages
|
||||
@ -243,7 +243,7 @@ dspace63=# CREATE EXTENSION pgcrypto;
|
||||
</code></pre><ul>
|
||||
<li>And I remembered I actually need to run the DSpace 6.4 Solr UUID migrations:</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>$ export JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8"
|
||||
<pre tabindex="0"><code>$ export JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8"
|
||||
$ ~/dspace63/bin/dspace solr-upgrade-statistics-6x
|
||||
</code></pre><ul>
|
||||
<li>Run system updates on DSpace Test (linode26) and reboot it</li>
|
||||
@ -258,7 +258,7 @@ $ ~/dspace63/bin/dspace solr-upgrade-statistics-6x
|
||||
<li>I realized that <code>solr-upgrade-statistics-6x</code> only processes 100,000 records by default so I think we actually need to finish running it for all legacy Solr records before asking Atmire why CUA statlets and detailed statistics aren’t working</li>
|
||||
<li>For now I am just doing 250,000 records at a time on my local environment:</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>$ export JAVA_OPTS="-Xmx2000m -Dfile.encoding=UTF-8"
|
||||
<pre tabindex="0"><code>$ export JAVA_OPTS="-Xmx2000m -Dfile.encoding=UTF-8"
|
||||
$ ~/dspace63/bin/dspace solr-upgrade-statistics-6x -n 250000
|
||||
</code></pre><ul>
|
||||
<li>Despite running the migration for all of my local 1.5 million Solr records, I still see a few hundred thousand like <code>-1</code> and <code>0-unmigrated</code>
|
||||
@ -284,7 +284,7 @@ $ podman start artifactory
|
||||
<ul>
|
||||
<li>A few days ago Peter asked me to update an author’s name on CGSpace and in the controlled vocabularies:</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>dspace=# UPDATE metadatavalue SET text_value='Knight-Jones, Theodore J.D.' WHERE resource_type_id=2 AND metadata_field_id=3 AND text_value='Knight-Jones, T.J.D.';
|
||||
<pre tabindex="0"><code>dspace=# UPDATE metadatavalue SET text_value='Knight-Jones, Theodore J.D.' WHERE resource_type_id=2 AND metadata_field_id=3 AND text_value='Knight-Jones, T.J.D.';
|
||||
</code></pre><ul>
|
||||
<li>I updated his existing records on CGSpace, changed the controlled lists, added his ORCID identifier to the controlled list, and tagged his thirty-nine items with the ORCID iD</li>
|
||||
<li>The new DSpace 6 stuff that Atmire sent modifies the Mirage 2’s <code>pom.xml</code> to copy the each theme’s resulting <code>node_modules</code> to each theme after building and installing with <code>ant update</code> because they moved some packages from bower to npm and now reference them in <code>page-structure.xsl</code>
|
||||
@ -315,7 +315,7 @@ $ podman start artifactory
|
||||
<ul>
|
||||
<li>Looking into a high rate of outgoing bandwidth from yesterday on CGSpace (linode18):</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code># cat /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "19/Apr/2020:0[6789]" | goaccess --log-format=COMBINED -
|
||||
<pre tabindex="0"><code># cat /var/log/nginx/*.log /var/log/nginx/*.log.1 | grep -E "19/Apr/2020:0[6789]" | goaccess --log-format=COMBINED -
|
||||
</code></pre><ul>
|
||||
<li>One host in Russia (91.241.19.70) download 23GiB over those few hours in the morning
|
||||
<ul>
|
||||
@ -325,7 +325,7 @@ $ podman start artifactory
|
||||
</ul>
|
||||
<pre tabindex="0"><code># grep -c 91.241.19.70 /var/log/nginx/access.log.1
|
||||
8900
|
||||
# grep 91.241.19.70 /var/log/nginx/access.log.1 | grep -c '10568/35187'
|
||||
# grep 91.241.19.70 /var/log/nginx/access.log.1 | grep -c '10568/35187'
|
||||
8900
|
||||
</code></pre><ul>
|
||||
<li>I thought the host might have been Yandex misbehaving, but its user agent is:</li>
|
||||
@ -343,20 +343,20 @@ Total number of bot hits purged: 8909
|
||||
</code></pre><ul>
|
||||
<li>While investigating that I noticed ORCID identifiers missing from a few authors names, so I added them with my <code>add-orcid-identifiers.py</code> script:</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>$ ./add-orcid-identifiers-csv.py -i 2020-04-20-add-orcids.csv -db dspace -u dspace -p 'fuuu' -d
|
||||
<pre tabindex="0"><code>$ ./add-orcid-identifiers-csv.py -i 2020-04-20-add-orcids.csv -db dspace -u dspace -p 'fuuu' -d
|
||||
</code></pre><ul>
|
||||
<li>The contents of <code>2020-04-20-add-orcids.csv</code> was:</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>dc.contributor.author,cg.creator.id
|
||||
"Schut, Marc","Marc Schut: 0000-0002-3361-4581"
|
||||
"Schut, M.","Marc Schut: 0000-0002-3361-4581"
|
||||
"Kamau, G.","Geoffrey Kamau: 0000-0002-6995-4801"
|
||||
"Kamau, G","Geoffrey Kamau: 0000-0002-6995-4801"
|
||||
"Triomphe, Bernard","Bernard Triomphe: 0000-0001-6657-3002"
|
||||
"Waters-Bayer, Ann","Ann Waters-Bayer: 0000-0003-1887-7903"
|
||||
"Klerkx, Laurens","Laurens Klerkx: 0000-0002-1664-886X"
|
||||
"Schut, Marc","Marc Schut: 0000-0002-3361-4581"
|
||||
"Schut, M.","Marc Schut: 0000-0002-3361-4581"
|
||||
"Kamau, G.","Geoffrey Kamau: 0000-0002-6995-4801"
|
||||
"Kamau, G","Geoffrey Kamau: 0000-0002-6995-4801"
|
||||
"Triomphe, Bernard","Bernard Triomphe: 0000-0001-6657-3002"
|
||||
"Waters-Bayer, Ann","Ann Waters-Bayer: 0000-0003-1887-7903"
|
||||
"Klerkx, Laurens","Laurens Klerkx: 0000-0002-1664-886X"
|
||||
</code></pre><ul>
|
||||
<li>I confirmed some of the authors' names from the report itself, then by looking at their profiles on ORCID.org</li>
|
||||
<li>I confirmed some of the authors’ names from the report itself, then by looking at their profiles on ORCID.org</li>
|
||||
<li>Add new ILRI subject “COVID19” to the <code>5_x-prod</code> branch</li>
|
||||
<li>Add new CCAFS Phase II project tags to the <code>5_x-prod</code> branch</li>
|
||||
<li>I will deploy these to CGSpace in the next few days</li>
|
||||
@ -387,17 +387,17 @@ Total number of bot hits purged: 8909
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx1024m"
|
||||
<pre tabindex="0"><code>$ export JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx1024m"
|
||||
$ time chrt -i 0 ionice -c2 -n7 nice -n19 dspace index-discovery -b
|
||||
</code></pre><ul>
|
||||
<li>I ran the <code>dspace cleanup -v</code> process on CGSpace and got an error:</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>Error: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
|
||||
Detail: Key (bitstream_id)=(184980) is still referenced from table "bundle".
|
||||
<pre tabindex="0"><code>Error: ERROR: update or delete on table "bitstream" violates foreign key constraint "bundle_primary_bitstream_id_fkey" on table "bundle"
|
||||
Detail: Key (bitstream_id)=(184980) is still referenced from table "bundle".
|
||||
</code></pre><ul>
|
||||
<li>The solution is, as always:</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>$ psql -d dspace -U dspace -c 'update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (183996);'
|
||||
<pre tabindex="0"><code>$ psql -d dspace -U dspace -c 'update bundle set primary_bitstream_id=NULL where primary_bitstream_id in (183996);'
|
||||
UPDATE 1
|
||||
</code></pre><ul>
|
||||
<li>I spent some time working on the XMLUI themes in DSpace 6
|
||||
@ -413,7 +413,7 @@ UPDATE 1
|
||||
</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>.breadcrumb > li + li:before {
|
||||
content: "/\00a0";
|
||||
content: "/\00a0";
|
||||
}
|
||||
</code></pre><h2 id="2020-04-27">2020-04-27</h2>
|
||||
<ul>
|
||||
@ -421,9 +421,9 @@ UPDATE 1
|
||||
<li>My changes to DSpace XMLUI Mirage 2 build process mean that we don’t need Ruby gems at all anymore! We can completely build without them!</li>
|
||||
<li>Trying to test the <code>com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdateCLI</code> script but there is an error:</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>Exception: org.apache.solr.search.SyntaxError: Cannot parse 'cua_version:${cua.version.number}': Encountered " "}" "} "" at line 1, column 32.
|
||||
<pre tabindex="0"><code>Exception: org.apache.solr.search.SyntaxError: Cannot parse 'cua_version:${cua.version.number}': Encountered " "}" "} "" at line 1, column 32.
|
||||
Was expecting one of:
|
||||
"TO" ...
|
||||
"TO" ...
|
||||
<RANGE_QUOTED> ...
|
||||
<RANGE_GOOP> ...
|
||||
</code></pre><ul>
|
||||
@ -473,7 +473,7 @@ atmire-cua.version.number=${cua.version.number}
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>Record uid: ee085cc0-0110-42c5-80b9-0fad4015ed9f couldn't be processed
|
||||
<pre tabindex="0"><code>Record uid: ee085cc0-0110-42c5-80b9-0fad4015ed9f couldn't be processed
|
||||
com.atmire.statistics.util.update.atomic.ProcessingException: something went wrong while processing record uid: ee085cc0-0110-42c5-80b9-0fad4015ed9f, an error occured in the com.atmire.statistics.util.update.atomic.processor.ContainerOwnerDBProcessor
|
||||
at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.applyProcessors(AtomicStatisticsUpdater.java:304)
|
||||
at com.atmire.statistics.util.update.atomic.AtomicStatisticsUpdater.processRecords(AtomicStatisticsUpdater.java:176)
|
||||
@ -508,7 +508,7 @@ Caused by: java.lang.NullPointerException
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>$ grep ERROR dspace.log.2020-04-29 | cut -f 3- -d' ' | sort | uniq -c | sort -n
|
||||
<pre tabindex="0"><code>$ grep ERROR dspace.log.2020-04-29 | cut -f 3- -d' ' | sort | uniq -c | sort -n
|
||||
1 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL findByUnique Error -
|
||||
1 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL find Error -
|
||||
1 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL query singleTable Error -
|
||||
@ -524,25 +524,25 @@ Caused by: java.lang.NullPointerException
|
||||
<ul>
|
||||
<li>Database connections do seem high:</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
|
||||
<pre tabindex="0"><code>$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
|
||||
5 dspaceApi
|
||||
6 dspaceCli
|
||||
88 dspaceWeb
|
||||
</code></pre><ul>
|
||||
<li>Most of those are idle in transaction:</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>$ psql -c 'select * from pg_stat_activity' | grep 'dspaceWeb' | grep -c "idle in transaction"
|
||||
<pre tabindex="0"><code>$ psql -c 'select * from pg_stat_activity' | grep 'dspaceWeb' | grep -c "idle in transaction"
|
||||
67
|
||||
</code></pre><ul>
|
||||
<li>I don’t see anything in the PostgreSQL or Tomcat logs suggesting anything is wrong… I think the solution to clear these idle connections is probably to just restart Tomcat</li>
|
||||
<li>I looked at the Solr stats for this month and see lots of suspicious IPs:</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>$ curl -s 'http://localhost:8081/solr/statistics/select?q=*:*&fq=dateYearMonth:2020-04&rows=0&wt=json&indent=true&facet=true&facet.field=ip
|
||||
<pre tabindex="0"><code>$ curl -s 'http://localhost:8081/solr/statistics/select?q=*:*&fq=dateYearMonth:2020-04&rows=0&wt=json&indent=true&facet=true&facet.field=ip
|
||||
|
||||
"88.99.115.53",23621, # Hetzner, using XMLUI and REST API with no user agent
|
||||
"104.154.216.0",11865,# Google cloud, scraping XMLUI with no user agent
|
||||
"104.198.96.245",4925,# Google cloud, using REST API with no user agent
|
||||
"52.34.238.26",2907, # EcoSearch on XMLUI, user agent: EcoSearch (+https://search.ecointernet.org/)
|
||||
"88.99.115.53",23621, # Hetzner, using XMLUI and REST API with no user agent
|
||||
"104.154.216.0",11865,# Google cloud, scraping XMLUI with no user agent
|
||||
"104.198.96.245",4925,# Google cloud, using REST API with no user agent
|
||||
"52.34.238.26",2907, # EcoSearch on XMLUI, user agent: EcoSearch (+https://search.ecointernet.org/)
|
||||
</code></pre><ul>
|
||||
<li>And a bunch more… ugh…
|
||||
<ul>
|
||||
@ -561,10 +561,10 @@ $ ./check-spider-ip-hits.sh -f /tmp/ips -s statistics -p
|
||||
<li>Then I added a few of them to the bot mapping in the nginx config because it appears they are regular harvesters since 2018</li>
|
||||
<li>Looking through the Solr stats faceted by the <code>userAgent</code> field I see some interesting ones:</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>$ curl 'http://localhost:8081/solr/statistics/select?q=*%3A*&rows=0&wt=json&indent=true&facet=true&facet.field=userAgent'
|
||||
<pre tabindex="0"><code>$ curl 'http://localhost:8081/solr/statistics/select?q=*%3A*&rows=0&wt=json&indent=true&facet=true&facet.field=userAgent'
|
||||
...
|
||||
"Delphi 2009",50725,
|
||||
"OgScrper/1.0.0",12421,
|
||||
"Delphi 2009",50725,
|
||||
"OgScrper/1.0.0",12421,
|
||||
</code></pre><ul>
|
||||
<li>Delphi is only used by IP addresses in Greece, so that’s obviously the GARDIAN people harvesting us…</li>
|
||||
<li>I have no idea what OgScrper is, but it’s not a user!</li>
|
||||
@ -586,11 +586,11 @@ $ ./check-spider-hits.sh -f /tmp/agents -s statistics -p
|
||||
<li>That’s about 300,000 hits purged…</li>
|
||||
<li>Then remove the ones with spaces manually, checking the query syntax first, then deleting in yearly cores and the statistics core:</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>$ curl -s "http://localhost:8081/solr/statistics/select" -d "q=userAgent:/Delphi 2009/&rows=0"
|
||||
<pre tabindex="0"><code>$ curl -s "http://localhost:8081/solr/statistics/select" -d "q=userAgent:/Delphi 2009/&rows=0"
|
||||
...
|
||||
<lst name="responseHeader"><int name="status">0</int><int name="QTime">52</int><lst name="params"><str name="q">userAgent:/Delphi 2009/</str><str name="rows">0</str></lst></lst><result name="response" numFound="38760" start="0"></result>
|
||||
$ for year in {2010..2019}; do curl -s "http://localhost:8081/solr/statistics-$year/update?softCommit=true" -H "Content-Type: text/xml" --data-binary '<delete><query>userAgent:"Delphi 2009"</query></delete>'; done
|
||||
$ curl -s "http://localhost:8081/solr/statistics/update?softCommit=true" -H "Content-Type: text/xml" --data-binary '<delete><query>userAgent:"Delphi 2009"</query></delete>'
|
||||
<lst name="responseHeader"><int name="status">0</int><int name="QTime">52</int><lst name="params"><str name="q">userAgent:/Delphi 2009/</str><str name="rows">0</str></lst></lst><result name="response" numFound="38760" start="0"></result>
|
||||
$ for year in {2010..2019}; do curl -s "http://localhost:8081/solr/statistics-$year/update?softCommit=true" -H "Content-Type: text/xml" --data-binary '<delete><query>userAgent:"Delphi 2009"</query></delete>'; done
|
||||
$ curl -s "http://localhost:8081/solr/statistics/update?softCommit=true" -H "Content-Type: text/xml" --data-binary '<delete><query>userAgent:"Delphi 2009"</query></delete>'
|
||||
</code></pre><ul>
|
||||
<li>Quoting them works for now until I can look into it and handle it properly in the script</li>
|
||||
<li>This was about 400,000 hits in total purged from the Solr statistics</li>
|
||||
@ -607,7 +607,7 @@ $ curl -s "http://localhost:8081/solr/statistics/update?softCommit=true&quo
|
||||
</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code># mv /etc/letsencrypt /etc/letsencrypt.bak
|
||||
# /opt/certbot-auto certonly --standalone --email fu@m.com -d dspacetest.cgiar.org --standalone --pre-hook "/bin/systemctl stop nginx" --post-hook "/bin/systemctl start nginx"
|
||||
# /opt/certbot-auto certonly --standalone --email fu@m.com -d dspacetest.cgiar.org --standalone --pre-hook "/bin/systemctl stop nginx" --post-hook "/bin/systemctl start nginx"
|
||||
# /opt/certbot-auto revoke --cert-path /etc/letsencrypt.bak/live/dspacetest.cgiar.org/cert.pem
|
||||
# rm -rf /etc/letsencrypt.bak
|
||||
</code></pre><ul>
|
||||
@ -618,11 +618,11 @@ $ curl -s "http://localhost:8081/solr/statistics/update?softCommit=true&quo
|
||||
<ul>
|
||||
<li>But I don’t see a lot of connections in PostgreSQL itself:</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
|
||||
<pre tabindex="0"><code>$ psql -c 'select * from pg_stat_activity' | grep -o -E '(dspaceWeb|dspaceApi|dspaceCli)' | sort | uniq -c
|
||||
5 dspaceApi
|
||||
6 dspaceCli
|
||||
14 dspaceWeb
|
||||
$ psql -c 'select * from pg_stat_activity' | wc -l
|
||||
$ psql -c 'select * from pg_stat_activity' | wc -l
|
||||
30
|
||||
</code></pre><ul>
|
||||
<li>Tezira said she cleared her browser cache and then was able to submit again
|
||||
|
||||
Reference in New Issue
Block a user