mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2025-01-27 05:49:12 +01:00
Add notes for 2022-03-04
This commit is contained in:
@ -42,7 +42,7 @@ You need to download this into the DSpace 6.x source and compile it
|
||||
|
||||
|
||||
"/>
|
||||
<meta name="generator" content="Hugo 0.92.2" />
|
||||
<meta name="generator" content="Hugo 0.93.1" />
|
||||
|
||||
|
||||
|
||||
@ -141,7 +141,7 @@ You need to download this into the DSpace 6.x source and compile it
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>$ export JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8"
|
||||
<pre tabindex="0"><code>$ export JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8"
|
||||
$ ~/dspace63/bin/dspace solr-upgrade-statistics-6x
|
||||
</code></pre><h2 id="2020-03-03">2020-03-03</h2>
|
||||
<ul>
|
||||
@ -160,7 +160,7 @@ $ ~/dspace63/bin/dspace solr-upgrade-statistics-6x
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>$ ./fix-metadata-values.py -i 2020-03-04-fix-1-ilri-subject.csv -db dspace -u dspace -p 'fuuu' -f cg.subject.ilri -m 203 -t correct -d
|
||||
<pre tabindex="0"><code>$ ./fix-metadata-values.py -i 2020-03-04-fix-1-ilri-subject.csv -db dspace -u dspace -p 'fuuu' -f cg.subject.ilri -m 203 -t correct -d
|
||||
</code></pre><ul>
|
||||
<li>But I have not run it on CGSpace yet because we want to ask Peter if he is sure about it…</li>
|
||||
<li>Send a message to Macaroni Bros to ask them about their Drupal module and its readiness for DSpace 6 UUIDs</li>
|
||||
@ -179,16 +179,16 @@ $ ~/dspace63/bin/dspace solr-upgrade-statistics-6x
|
||||
</ul>
|
||||
<pre tabindex="0"><code>$ ./run.sh -s http://localhost:8081/solr/statistics-2010 -a export -o /tmp/statistics-2010.json -k uid
|
||||
$ ./run.sh -s http://localhost:8081/solr/statistics -a import -o /tmp/statistics-2010.json -k uid
|
||||
$ curl -s "http://localhost:8081/solr/statistics-2010/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>time:2010*</query></delete>"
|
||||
$ curl -s "http://localhost:8081/solr/statistics-2010/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>time:2010*</query></delete>"
|
||||
$ ./run.sh -s http://localhost:8081/solr/statistics-2011 -a export -o /tmp/statistics-2011.json -k uid
|
||||
$ ./run.sh -s http://localhost:8081/solr/statistics -a import -o /tmp/statistics-2011.json -k uid
|
||||
$ curl -s "http://localhost:8081/solr/statistics-2011/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>time:2011*</query></delete>"
|
||||
$ curl -s "http://localhost:8081/solr/statistics-2011/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>time:2011*</query></delete>"
|
||||
$ ./run.sh -s http://localhost:8081/solr/statistics -a import -o /tmp/statistics-2012.json -k uid
|
||||
$ curl -s 'http://localhost:8081/solr/statistics/select?q=time:2012*&rows=0&wt=json&indent=true' | grep numFound
|
||||
"response":{"numFound":3761989,"start":0,"docs":[]
|
||||
$ curl -s 'http://localhost:8081/solr/statistics-2012/select?q=time:2012*&rows=0&wt=json&indent=true' | grep numFound
|
||||
"response":{"numFound":3761989,"start":0,"docs":[]
|
||||
$ curl -s "http://localhost:8081/solr/statistics-2012/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>time:2012*</query></delete>"
|
||||
$ curl -s 'http://localhost:8081/solr/statistics/select?q=time:2012*&rows=0&wt=json&indent=true' | grep numFound
|
||||
"response":{"numFound":3761989,"start":0,"docs":[]
|
||||
$ curl -s 'http://localhost:8081/solr/statistics-2012/select?q=time:2012*&rows=0&wt=json&indent=true' | grep numFound
|
||||
"response":{"numFound":3761989,"start":0,"docs":[]
|
||||
$ curl -s "http://localhost:8081/solr/statistics-2012/update?softCommit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>time:2012*</query></delete>"
|
||||
</code></pre><ul>
|
||||
<li>I will do this for as many cores as I can (disk space limited) and then monitor the effect on the system and JVM memory usage
|
||||
<ul>
|
||||
@ -196,7 +196,7 @@ $ curl -s "http://localhost:8081/solr/statistics-2012/update?softCommit=tru
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>$ ./run.sh -s http://localhost:8081/solr/statistics-2014 -a export -o /tmp/statistics-2014-1.json -k uid -f 'time:/2014-0[1-6].*/'
|
||||
<pre tabindex="0"><code>$ ./run.sh -s http://localhost:8081/solr/statistics-2014 -a export -o /tmp/statistics-2014-1.json -k uid -f 'time:/2014-0[1-6].*/'
|
||||
</code></pre><ul>
|
||||
<li>Upgrade PostgreSQL from 9.6 to 10 on DSpace Test (linode19)
|
||||
<ul>
|
||||
@ -213,7 +213,7 @@ $ curl -s "http://localhost:8081/solr/statistics-2012/update?softCommit=tru
|
||||
# pg_dropcluster 10 main
|
||||
# pg_upgradecluster 9.6 main
|
||||
# pg_dropcluster 9.6 main
|
||||
# dpkg -l | grep postgresql | grep 9.6 | awk '{print $2}' | xargs dpkg -r
|
||||
# dpkg -l | grep postgresql | grep 9.6 | awk '{print $2}' | xargs dpkg -r
|
||||
</code></pre><h2 id="2020-03-09">2020-03-09</h2>
|
||||
<ul>
|
||||
<li>Peter noticed that the Solr stats were not showing anything before 2020
|
||||
@ -250,7 +250,7 @@ $ curl -s "http://localhost:8081/solr/statistics-2012/update?softCommit=tru
|
||||
<li>In Solr the IP is 127.0.0.1, but in the nginx logs I can luckily see the real IP (64.225.40.66), which is on Digital Ocean</li>
|
||||
<li>I will purge them from Solr statistics:</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>$ curl -s "http://localhost:8081/solr/statistics/update?softCommit=true" -H "Content-Type: text/xml" --data-binary '<delete><query>userAgent:"Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 (.NET CLR 3.5.30729)"</query></delete>'
|
||||
<pre tabindex="0"><code>$ curl -s "http://localhost:8081/solr/statistics/update?softCommit=true" -H "Content-Type: text/xml" --data-binary '<delete><query>userAgent:"Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 (.NET CLR 3.5.30729)"</query></delete>'
|
||||
</code></pre><ul>
|
||||
<li>Another user agent that seems to be a bot is:</li>
|
||||
</ul>
|
||||
@ -258,14 +258,14 @@ $ curl -s "http://localhost:8081/solr/statistics-2012/update?softCommit=tru
|
||||
</code></pre><ul>
|
||||
<li>In Solr the IP is 127.0.0.1 because of the misconfiguration, but in nginx’s logs I see it belongs to three IPs on Online.net in France:</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code># zcat /var/log/nginx/access.log.*.gz /var/log/nginx/rest.log.*.gz | grep 'Mozilla/5.0 ((Windows; U; Windows NT 6.1; fr; rv:1.9.2) Gecko/20100115 Firefox/3.6)' | awk '{print $1}' | sort | uniq -c
|
||||
<pre tabindex="0"><code># zcat /var/log/nginx/access.log.*.gz /var/log/nginx/rest.log.*.gz | grep 'Mozilla/5.0 ((Windows; U; Windows NT 6.1; fr; rv:1.9.2) Gecko/20100115 Firefox/3.6)' | awk '{print $1}' | sort | uniq -c
|
||||
63090 163.172.68.99
|
||||
183428 163.172.70.248
|
||||
147608 163.172.71.24
|
||||
</code></pre><ul>
|
||||
<li>It is making 10,000 to 40,000 requests to XMLUI per day…</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code># zgrep -c 'Mozilla/5.0 ((Windows; U; Windows NT 6.1; fr; rv:1.9.2) Gecko/20100115 Firefox/3.6)' /var/log/nginx/access.log.{1..9}
|
||||
<pre tabindex="0"><code># zgrep -c 'Mozilla/5.0 ((Windows; U; Windows NT 6.1; fr; rv:1.9.2) Gecko/20100115 Firefox/3.6)' /var/log/nginx/access.log.{1..9}
|
||||
/var/log/nginx/access.log.30.gz:18687
|
||||
/var/log/nginx/access.log.31.gz:28936
|
||||
/var/log/nginx/access.log.32.gz:36402
|
||||
@ -284,7 +284,7 @@ $ curl -s "http://localhost:8081/solr/statistics-2012/update?softCommit=tru
|
||||
</code></pre><ul>
|
||||
<li>I will purge those hits too!</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>$ curl -s "http://localhost:8081/solr/statistics/update?softCommit=true" -H "Content-Type: text/xml" --data-binary '<delete><query>userAgent:"Mozilla/5.0 ((Windows; U; Windows NT 6.1; fr; rv:1.9.2) Gecko/20100115 Firefox/3.6)"</query></delete>'
|
||||
<pre tabindex="0"><code>$ curl -s "http://localhost:8081/solr/statistics/update?softCommit=true" -H "Content-Type: text/xml" --data-binary '<delete><query>userAgent:"Mozilla/5.0 ((Windows; U; Windows NT 6.1; fr; rv:1.9.2) Gecko/20100115 Firefox/3.6)"</query></delete>'
|
||||
</code></pre><ul>
|
||||
<li>Shit, and something happened and a few thousand hits from user agents with “Bot” in their user agent got through
|
||||
<ul>
|
||||
@ -348,7 +348,7 @@ Purging 62 hits from [Ss]pider in statistics
|
||||
</ul>
|
||||
<pre tabindex="0"><code>dspace=# \COPY (SELECT DISTINCT text_value, count(*) FROM metadatavalue WHERE resource_type_id = 2 AND metadata_field_id = 211 GROUP BY text_value ORDER BY count DESC LIMIT 1500) to /tmp/2020-03-12-affiliations.csv WITH CSV HEADER;`
|
||||
dspace=# \q
|
||||
$ csvcut -l -c 0 /tmp/2020-03-12-affiliations.csv | sed -e 's/^line_number/id/' -e 's/text_value/name/' > /tmp/affiliations.csv
|
||||
$ csvcut -l -c 0 /tmp/2020-03-12-affiliations.csv | sed -e 's/^line_number/id/' -e 's/text_value/name/' > /tmp/affiliations.csv
|
||||
$ lein run /tmp/affiliations.csv name id
|
||||
</code></pre><ul>
|
||||
<li>I always forget how to copy the reconciled values in OpenRefine, but you need to make a new column and populate it using this GREL: <code>if(cell.recon.matched, cell.recon.match.name, value)</code></li>
|
||||
@ -417,7 +417,7 @@ $ lein run /tmp/affiliations.csv name id
|
||||
<li>Update Tomcat to version 7.0.103 in the Ansible infrastrcutrue playbooks and deploy on DSpace Test (linode26)</li>
|
||||
<li>Maria sent me a few new ORCID identifiers from Bioversity so I combined them with our existing ones, filtered the unique ones, and then resolved their names using my <code>resolve-orcids.py</code> script:</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>$ cat ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-id.xml /tmp/bioversity-orcids | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort | uniq > /tmp/2020-03-26-combined-orcids.txt
|
||||
<pre tabindex="0"><code>$ cat ~/src/git/DSpace/dspace/config/controlled-vocabularies/cg-creator-id.xml /tmp/bioversity-orcids | grep -oE '[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}-[A-Z0-9]{4}' | sort | uniq > /tmp/2020-03-26-combined-orcids.txt
|
||||
$ ./resolve-orcids.py -i /tmp/2020-03-26-combined-orcids.txt -o /tmp/2020-03-26-combined-names.txt -d
|
||||
# sort names, copy to cg-creator-id.xml, add XML formatting, and then format with tidy (preserving accents)
|
||||
$ tidy -xml -utf8 -iq -m -w 0 dspace/config/controlled-vocabularies/cg-creator-id.xml
|
||||
@ -425,16 +425,16 @@ $ tidy -xml -utf8 -iq -m -w 0 dspace/config/controlled-vocabularies/cg-creator-i
|
||||
<li>I checked the database for likely matches to the author name and then created a CSV with the author names and ORCID iDs:</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>dc.contributor.author,cg.creator.id
|
||||
"King, Brian","Brian King: 0000-0002-7056-9214"
|
||||
"Ortiz-Crespo, Berta","Berta Ortiz-Crespo: 0000-0002-6664-0815"
|
||||
"Ekesa, Beatrice","Beatrice Ekesa: 0000-0002-2630-258X"
|
||||
"Ekesa, B.","Beatrice Ekesa: 0000-0002-2630-258X"
|
||||
"Ekesa, B.N.","Beatrice Ekesa: 0000-0002-2630-258X"
|
||||
"Gullotta, G.","Gaia Gullotta: 0000-0002-2240-3869"
|
||||
"King, Brian","Brian King: 0000-0002-7056-9214"
|
||||
"Ortiz-Crespo, Berta","Berta Ortiz-Crespo: 0000-0002-6664-0815"
|
||||
"Ekesa, Beatrice","Beatrice Ekesa: 0000-0002-2630-258X"
|
||||
"Ekesa, B.","Beatrice Ekesa: 0000-0002-2630-258X"
|
||||
"Ekesa, B.N.","Beatrice Ekesa: 0000-0002-2630-258X"
|
||||
"Gullotta, G.","Gaia Gullotta: 0000-0002-2240-3869"
|
||||
</code></pre><ul>
|
||||
<li>Running the <code>add-orcid-identifiers-csv.py</code> script I added 32 ORCID iDs to items on CGSpace!</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>$ ./add-orcid-identifiers-csv.py -i /tmp/2020-03-26-ciat-orcids.csv -db dspace -u dspace -p 'fuuu'
|
||||
<pre tabindex="0"><code>$ ./add-orcid-identifiers-csv.py -i /tmp/2020-03-26-ciat-orcids.csv -db dspace -u dspace -p 'fuuu'
|
||||
</code></pre><ul>
|
||||
<li>Udana from IWMI asked about some items that are missing Altmetric donuts on CGSpace
|
||||
<ul>
|
||||
@ -447,13 +447,13 @@ $ tidy -xml -utf8 -iq -m -w 0 dspace/config/controlled-vocabularies/cg-creator-i
|
||||
</ul>
|
||||
<h2 id="2020-03-29">2020-03-29</h2>
|
||||
<ul>
|
||||
<li>Add two more Bioversity ORCID iDs to CGSpace and then tag ~70 of the authors' existing publications in the database using this CSV with my <code>add-orcid-identifiers-csv.py</code> script:</li>
|
||||
<li>Add two more Bioversity ORCID iDs to CGSpace and then tag ~70 of the authors’ existing publications in the database using this CSV with my <code>add-orcid-identifiers-csv.py</code> script:</li>
|
||||
</ul>
|
||||
<pre tabindex="0"><code>dc.contributor.author,cg.creator.id
|
||||
"Snook, L.K.","Laura Snook: 0000-0002-9168-1301"
|
||||
"Snook, L.","Laura Snook: 0000-0002-9168-1301"
|
||||
"Zheng, S.J.","Sijun Zheng: 0000-0003-1550-3738"
|
||||
"Zheng, S.","Sijun Zheng: 0000-0003-1550-3738"
|
||||
"Snook, L.K.","Laura Snook: 0000-0002-9168-1301"
|
||||
"Snook, L.","Laura Snook: 0000-0002-9168-1301"
|
||||
"Zheng, S.J.","Sijun Zheng: 0000-0003-1550-3738"
|
||||
"Zheng, S.","Sijun Zheng: 0000-0003-1550-3738"
|
||||
</code></pre><ul>
|
||||
<li>Deploy latest Bioversity and CIAT updates on CGSpace (linode18) and DSpace Test (linode26)</li>
|
||||
<li>Deploy latest Ansible infrastructure playbooks on CGSpace and DSpace Test to get the latest dspace-statistics-api (v1.1.1) and Tomcat (7.0.103) versions</li>
|
||||
|
||||
Reference in New Issue
Block a user