mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2025-01-27 05:49:12 +01:00
Add notes for 2019-05-05
This commit is contained in:
@ -12,10 +12,11 @@
|
||||
Discuss helping CCAFS with some batch tagging of ORCID IDs for their authors
|
||||
Discuss how the migration of CGIAR’s Active Directory to a flat structure will break our LDAP groups in DSpace
|
||||
We had been using DC=ILRI to determine whether a user was ILRI or not
|
||||
|
||||
It looks like we might be able to use OUs now, instead of DCs:
|
||||
|
||||
|
||||
$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=org" -D "admigration1@cgiarad.org" -W "(sAMAccountName=admigration1)"
|
||||
|
||||
" />
|
||||
<meta property="og:type" content="article" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2016-09/" />
|
||||
@ -30,12 +31,13 @@ $ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=or
|
||||
Discuss helping CCAFS with some batch tagging of ORCID IDs for their authors
|
||||
Discuss how the migration of CGIAR’s Active Directory to a flat structure will break our LDAP groups in DSpace
|
||||
We had been using DC=ILRI to determine whether a user was ILRI or not
|
||||
|
||||
It looks like we might be able to use OUs now, instead of DCs:
|
||||
|
||||
|
||||
$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=org" -D "admigration1@cgiarad.org" -W "(sAMAccountName=admigration1)"
|
||||
|
||||
"/>
|
||||
<meta name="generator" content="Hugo 0.55.3" />
|
||||
<meta name="generator" content="Hugo 0.55.5" />
|
||||
|
||||
|
||||
|
||||
@ -120,29 +122,27 @@ $ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=or
|
||||
<li>Discuss helping CCAFS with some batch tagging of ORCID IDs for their authors</li>
|
||||
<li>Discuss how the migration of CGIAR’s Active Directory to a flat structure will break our LDAP groups in DSpace</li>
|
||||
<li>We had been using <code>DC=ILRI</code> to determine whether a user was ILRI or not</li>
|
||||
<li>It looks like we might be able to use OUs now, instead of DCs:</li>
|
||||
</ul>
|
||||
|
||||
<li><p>It looks like we might be able to use OUs now, instead of DCs:</p>
|
||||
|
||||
<pre><code>$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=org" -D "admigration1@cgiarad.org" -W "(sAMAccountName=admigration1)"
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
</ul>
|
||||
|
||||
<ul>
|
||||
<li>User who has been migrated to the root vs user still in the hierarchical structure:</li>
|
||||
</ul>
|
||||
<li><p>User who has been migrated to the root vs user still in the hierarchical structure:</p>
|
||||
|
||||
<pre><code>distinguishedName: CN=Last\, First (ILRI),OU=ILRI Kenya Employees,OU=ILRI Kenya,OU=ILRIHUB,DC=CGIARAD,DC=ORG
|
||||
distinguishedName: CN=Last\, First (ILRI),OU=ILRI Ethiopia Employees,OU=ILRI Ethiopia,DC=ILRI,DC=CGIARAD,DC=ORG
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>Changing the DSpace LDAP config to use <code>OU=ILRIHUB</code> seems to work:</li>
|
||||
<li><p>Changing the DSpace LDAP config to use <code>OU=ILRIHUB</code> seems to work:</p></li>
|
||||
</ul>
|
||||
|
||||
<p><img src="/cgspace-notes/2016/09/ilri-ldap-users.png" alt="DSpace groups based on LDAP DN" /></p>
|
||||
|
||||
<ul>
|
||||
<li>Notes for local PostgreSQL database recreation from production snapshot:</li>
|
||||
</ul>
|
||||
<li><p>Notes for local PostgreSQL database recreation from production snapshot:</p>
|
||||
|
||||
<pre><code>$ dropdb dspacetest
|
||||
$ createdb -O dspacetest --encoding=UNICODE dspacetest
|
||||
@ -151,96 +151,83 @@ $ pg_restore -O -U dspacetest -d dspacetest ~/Downloads/cgspace_2016-09-01.backu
|
||||
$ psql dspacetest -c 'alter user dspacetest nocreateuser;'
|
||||
$ psql -U dspacetest -f ~/src/git/DSpace/dspace/etc/postgres/update-sequences.sql dspacetest -h localhost
|
||||
$ vacuumdb dspacetest
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>Some names that I thought I fixed in July seem not to be:</li>
|
||||
</ul>
|
||||
<li><p>Some names that I thought I fixed in July seem not to be:</p>
|
||||
|
||||
<pre><code>dspacetest=# select distinct text_value, authority, confidence from metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value like 'Poole, %';
|
||||
text_value | authority | confidence
|
||||
text_value | authority | confidence
|
||||
-----------------------+--------------------------------------+------------
|
||||
Poole, Elizabeth Jane | b6efa27f-8829-4b92-80fe-bc63e03e3ccb | 600
|
||||
Poole, Elizabeth Jane | 41628f42-fc38-4b38-b473-93aec9196326 | 600
|
||||
Poole, Elizabeth Jane | 83b82da0-f652-4ebc-babc-591af1697919 | 600
|
||||
Poole, Elizabeth Jane | c3a22456-8d6a-41f9-bba0-de51ef564d45 | 600
|
||||
Poole, E.J. | c3a22456-8d6a-41f9-bba0-de51ef564d45 | 600
|
||||
Poole, E.J. | 0fbd91b9-1b71-4504-8828-e26885bf8b84 | 600
|
||||
Poole, Elizabeth Jane | b6efa27f-8829-4b92-80fe-bc63e03e3ccb | 600
|
||||
Poole, Elizabeth Jane | 41628f42-fc38-4b38-b473-93aec9196326 | 600
|
||||
Poole, Elizabeth Jane | 83b82da0-f652-4ebc-babc-591af1697919 | 600
|
||||
Poole, Elizabeth Jane | c3a22456-8d6a-41f9-bba0-de51ef564d45 | 600
|
||||
Poole, E.J. | c3a22456-8d6a-41f9-bba0-de51ef564d45 | 600
|
||||
Poole, E.J. | 0fbd91b9-1b71-4504-8828-e26885bf8b84 | 600
|
||||
(6 rows)
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>At least a few of these actually have the correct ORCID, but I will unify the authority to be c3a22456-8d6a-41f9-bba0-de51ef564d45</li>
|
||||
</ul>
|
||||
<li><p>At least a few of these actually have the correct ORCID, but I will unify the authority to be c3a22456-8d6a-41f9-bba0-de51ef564d45</p>
|
||||
|
||||
<pre><code>dspacetest=# update metadatavalue set authority='c3a22456-8d6a-41f9-bba0-de51ef564d45', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Poole, %';
|
||||
UPDATE 69
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>And for Peter Ballantyne:</li>
|
||||
</ul>
|
||||
<li><p>And for Peter Ballantyne:</p>
|
||||
|
||||
<pre><code>dspacetest=# select distinct text_value, authority, confidence from metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value like 'Ballantyne, %';
|
||||
text_value | authority | confidence
|
||||
text_value | authority | confidence
|
||||
-------------------+--------------------------------------+------------
|
||||
Ballantyne, Peter | 2dcbcc7b-47b0-4fd7-bef9-39d554494081 | 600
|
||||
Ballantyne, Peter | 4f04ca06-9a76-4206-bd9c-917ca75d278e | 600
|
||||
Ballantyne, P.G. | 4f04ca06-9a76-4206-bd9c-917ca75d278e | 600
|
||||
Ballantyne, Peter | ba5f205b-b78b-43e5-8e80-0c9a1e1ad2ca | 600
|
||||
Ballantyne, Peter | 20f21160-414c-4ecf-89ca-5f2cb64e75c1 | 600
|
||||
Ballantyne, Peter | 2dcbcc7b-47b0-4fd7-bef9-39d554494081 | 600
|
||||
Ballantyne, Peter | 4f04ca06-9a76-4206-bd9c-917ca75d278e | 600
|
||||
Ballantyne, P.G. | 4f04ca06-9a76-4206-bd9c-917ca75d278e | 600
|
||||
Ballantyne, Peter | ba5f205b-b78b-43e5-8e80-0c9a1e1ad2ca | 600
|
||||
Ballantyne, Peter | 20f21160-414c-4ecf-89ca-5f2cb64e75c1 | 600
|
||||
(5 rows)
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>Again, a few have the correct ORCID, but there should only be one authority…</li>
|
||||
</ul>
|
||||
<li><p>Again, a few have the correct ORCID, but there should only be one authority…</p>
|
||||
|
||||
<pre><code>dspacetest=# update metadatavalue set authority='4f04ca06-9a76-4206-bd9c-917ca75d278e', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Ballantyne, %';
|
||||
UPDATE 58
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>And for me:</li>
|
||||
</ul>
|
||||
<li><p>And for me:</p>
|
||||
|
||||
<pre><code>dspacetest=# select distinct text_value, authority, confidence from metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value like 'Orth, A%';
|
||||
text_value | authority | confidence
|
||||
text_value | authority | confidence
|
||||
------------+--------------------------------------+------------
|
||||
Orth, Alan | 4884def0-4d7e-4256-9dd4-018cd60a5871 | 600
|
||||
Orth, A. | 4884def0-4d7e-4256-9dd4-018cd60a5871 | 600
|
||||
Orth, A. | 1a1943a0-3f87-402f-9afe-e52fb46a513e | 600
|
||||
Orth, Alan | 4884def0-4d7e-4256-9dd4-018cd60a5871 | 600
|
||||
Orth, A. | 4884def0-4d7e-4256-9dd4-018cd60a5871 | 600
|
||||
Orth, A. | 1a1943a0-3f87-402f-9afe-e52fb46a513e | 600
|
||||
(3 rows)
|
||||
dspacetest=# update metadatavalue set authority='1a1943a0-3f87-402f-9afe-e52fb46a513e', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Orth, %';
|
||||
UPDATE 11
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>And for CCAFS author Bruce Campbell that I had discussed with CCAFS earlier this week:</li>
|
||||
</ul>
|
||||
<li><p>And for CCAFS author Bruce Campbell that I had discussed with CCAFS earlier this week:</p>
|
||||
|
||||
<pre><code>dspacetest=# update metadatavalue set authority='0e414b4c-4671-4a23-b570-6077aca647d8', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Campbell, B%';
|
||||
UPDATE 166
|
||||
dspacetest=# select distinct text_value, authority, confidence from metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value like 'Campbell, B%';
|
||||
text_value | authority | confidence
|
||||
text_value | authority | confidence
|
||||
------------------------+--------------------------------------+------------
|
||||
Campbell, Bruce | 0e414b4c-4671-4a23-b570-6077aca647d8 | 600
|
||||
Campbell, Bruce Morgan | 0e414b4c-4671-4a23-b570-6077aca647d8 | 600
|
||||
Campbell, B. | 0e414b4c-4671-4a23-b570-6077aca647d8 | 600
|
||||
Campbell, B.M. | 0e414b4c-4671-4a23-b570-6077aca647d8 | 600
|
||||
Campbell, Bruce | 0e414b4c-4671-4a23-b570-6077aca647d8 | 600
|
||||
Campbell, Bruce Morgan | 0e414b4c-4671-4a23-b570-6077aca647d8 | 600
|
||||
Campbell, B. | 0e414b4c-4671-4a23-b570-6077aca647d8 | 600
|
||||
Campbell, B.M. | 0e414b4c-4671-4a23-b570-6077aca647d8 | 600
|
||||
(4 rows)
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>After updating the Authority indexes (<code>bin/dspace index-authority</code>) everything looks good</li>
|
||||
<li>Run authority updates on CGSpace</li>
|
||||
<li><p>After updating the Authority indexes (<code>bin/dspace index-authority</code>) everything looks good</p></li>
|
||||
|
||||
<li><p>Run authority updates on CGSpace</p></li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2016-09-05">2016-09-05</h2>
|
||||
|
||||
<ul>
|
||||
<li>After one week of logging TLS connections on CGSpace:</li>
|
||||
</ul>
|
||||
<li><p>After one week of logging TLS connections on CGSpace:</p>
|
||||
|
||||
<pre><code># zgrep "DES-CBC3" /var/log/nginx/cgspace.cgiar.org-access-ssl.log* | wc -l
|
||||
217
|
||||
@ -249,18 +236,16 @@ dspacetest=# select distinct text_value, authority, confidence from metadatavalu
|
||||
# zgrep "DES-CBC3" /var/log/nginx/cgspace.cgiar.org-access-ssl.log* | awk '{print $6}' | sort | uniq
|
||||
TLSv1/DES-CBC3-SHA
|
||||
TLSv1/EDH-RSA-DES-CBC3-SHA
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>So this represents <code>0.02%</code> of 1.16M connections over a one-week period</li>
|
||||
<li>Transforming some filenames in OpenRefine so they can have a useful description for SAFBuilder:</li>
|
||||
</ul>
|
||||
<li><p>So this represents <code>0.02%</code> of 1.16M connections over a one-week period</p></li>
|
||||
|
||||
<li><p>Transforming some filenames in OpenRefine so they can have a useful description for SAFBuilder:</p>
|
||||
|
||||
<pre><code>value + "__description:" + cells["dc.type"].value
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>This gives you, for example: <code>Mainstreaming gender in agricultural R&D.pdf__description:Brief</code></li>
|
||||
<li><p>This gives you, for example: <code>Mainstreaming gender in agricultural R&D.pdf__description:Brief</code></p></li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2016-09-06">2016-09-06</h2>
|
||||
@ -283,28 +268,31 @@ TLSv1/EDH-RSA-DES-CBC3-SHA
|
||||
<li>See: <a href="http://www.fileformat.info/info/unicode/char/e1/index.htm">http://www.fileformat.info/info/unicode/char/e1/index.htm</a></li>
|
||||
<li>See: <a href="http://demo.icu-project.org/icu-bin/nbrowser?t=%C3%A1&s=&uv=0">http://demo.icu-project.org/icu-bin/nbrowser?t=%C3%A1&s=&uv=0</a></li>
|
||||
<li>If I unzip the original zip from CIAT on Windows, re-zip it with 7zip on Windows, and then unzip it on Linux directly, the file names seem to be proper UTF-8</li>
|
||||
<li>We should definitely clean filenames so they don’t use characters that are tricky to process in CSV and shell scripts, like: <code>,</code>, <code>'</code>, and <code>"</code></li>
|
||||
</ul>
|
||||
|
||||
<li><p>We should definitely clean filenames so they don’t use characters that are tricky to process in CSV and shell scripts, like: <code>,</code>, <code>'</code>, and <code>"</code></p>
|
||||
|
||||
<pre><code>value.replace("'","").replace(",","").replace('"','')
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>I need to write a Python script to match that for renaming files in the file system</li>
|
||||
<li>When importing SAF bundles it seems you can specify the target collection on the command line using <code>-c 10568/4003</code> or in the <code>collections</code> file inside each item in the bundle</li>
|
||||
<li>Seems that the latter method causes a null pointer exception, so I will just have to use the former method</li>
|
||||
<li>In the end I was able to import the files after unzipping them ONLY on Linux
|
||||
<li><p>I need to write a Python script to match that for renaming files in the file system</p></li>
|
||||
|
||||
<li><p>When importing SAF bundles it seems you can specify the target collection on the command line using <code>-c 10568/4003</code> or in the <code>collections</code> file inside each item in the bundle</p></li>
|
||||
|
||||
<li><p>Seems that the latter method causes a null pointer exception, so I will just have to use the former method</p></li>
|
||||
|
||||
<li><p>In the end I was able to import the files after unzipping them ONLY on Linux</p>
|
||||
|
||||
<ul>
|
||||
<li>The CSV file was giving file names in UTF-8, and unzipping the zip on Mac OS X and transferring it was converting the file names to Unicode equivalence like I saw above</li>
|
||||
</ul></li>
|
||||
<li>Import CIAT Gender Network records to CGSpace, first creating the SAF bundles as my user, then importing as the <code>tomcat7</code> user, and deleting the bundle, for each collection’s items:</li>
|
||||
</ul>
|
||||
|
||||
<li><p>Import CIAT Gender Network records to CGSpace, first creating the SAF bundles as my user, then importing as the <code>tomcat7</code> user, and deleting the bundle, for each collection’s items:</p>
|
||||
|
||||
<pre><code>$ ./safbuilder.sh -c /home/aorth/ciat-gender-2016-09-06/66601.csv
|
||||
$ JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx512m" /home/cgspace.cgiar.org/bin/dspace import -a -e aorth@mjanja.ch -c 10568/66601 -s /home/aorth/ciat-gender-2016-09-06/SimpleArchiveFormat -m 66601.map
|
||||
$ rm -rf ~/ciat-gender-2016-09-06/SimpleArchiveFormat/
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2016-09-07">2016-09-07</h2>
|
||||
|
||||
@ -313,132 +301,117 @@ $ rm -rf ~/ciat-gender-2016-09-06/SimpleArchiveFormat/
|
||||
<li>Reading about PostgreSQL maintenance and it seems manual vacuuming is only for certain workloads, such as heavy update/write loads</li>
|
||||
<li>I suggest we disable our nightly manual vacuum task, as we’re a mostly read workload, and I’d rather stick as close to the documentation as possible since we haven’t done any testing/observation of PostgreSQL</li>
|
||||
<li>See: <a href="https://www.postgresql.org/docs/9.3/static/routine-vacuuming.html">https://www.postgresql.org/docs/9.3/static/routine-vacuuming.html</a></li>
|
||||
<li>CGSpace went down and the error seems to be the same as always (lately):</li>
|
||||
</ul>
|
||||
|
||||
<li><p>CGSpace went down and the error seems to be the same as always (lately):</p>
|
||||
|
||||
<pre><code>2016-09-07 11:39:23,162 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error -
|
||||
org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
|
||||
...
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>Since CGSpace had crashed I quickly deployed the new LDAP settings before restarting Tomcat</li>
|
||||
<li><p>Since CGSpace had crashed I quickly deployed the new LDAP settings before restarting Tomcat</p></li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2016-09-13">2016-09-13</h2>
|
||||
|
||||
<ul>
|
||||
<li>CGSpace crashed twice today, errors from <code>catalina.out</code>:</li>
|
||||
</ul>
|
||||
<li><p>CGSpace crashed twice today, errors from <code>catalina.out</code>:</p>
|
||||
|
||||
<pre><code>org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
|
||||
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
|
||||
</code></pre>
|
||||
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>I enabled logging of requests to <code>/rest</code> again</li>
|
||||
<li><p>I enabled logging of requests to <code>/rest</code> again</p></li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2016-09-14">2016-09-14</h2>
|
||||
|
||||
<ul>
|
||||
<li>CGSpace crashed again, errors from <code>catalina.out</code>:</li>
|
||||
</ul>
|
||||
<li><p>CGSpace crashed again, errors from <code>catalina.out</code>:</p>
|
||||
|
||||
<pre><code>org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
|
||||
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
|
||||
</code></pre>
|
||||
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>I restarted Tomcat and it was ok again</li>
|
||||
<li>CGSpace crashed a few hours later, errors from <code>catalina.out</code>:</li>
|
||||
</ul>
|
||||
<li><p>I restarted Tomcat and it was ok again</p></li>
|
||||
|
||||
<li><p>CGSpace crashed a few hours later, errors from <code>catalina.out</code>:</p>
|
||||
|
||||
<pre><code>Exception in thread "http-bio-127.0.0.1-8081-exec-25" java.lang.OutOfMemoryError: Java heap space
|
||||
at java.lang.StringCoding.decode(StringCoding.java:215)
|
||||
</code></pre>
|
||||
at java.lang.StringCoding.decode(StringCoding.java:215)
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>We haven’t seen that in quite a while…</li>
|
||||
<li>Indeed, in a month of logs it only occurs 15 times:</li>
|
||||
</ul>
|
||||
<li><p>We haven’t seen that in quite a while…</p></li>
|
||||
|
||||
<li><p>Indeed, in a month of logs it only occurs 15 times:</p>
|
||||
|
||||
<pre><code># grep -rsI "OutOfMemoryError" /var/log/tomcat7/catalina.* | wc -l
|
||||
15
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>I also see a bunch of errors from dspace.log:</li>
|
||||
</ul>
|
||||
<li><p>I also see a bunch of errors from dspace.log:</p>
|
||||
|
||||
<pre><code>2016-09-14 12:23:07,981 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error -
|
||||
org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>Looking at REST requests, it seems there is one IP hitting us nonstop:</li>
|
||||
</ul>
|
||||
<li><p>Looking at REST requests, it seems there is one IP hitting us nonstop:</p>
|
||||
|
||||
<pre><code># awk '{print $1}' /var/log/nginx/rest.log | sort -n | uniq -c | sort -h | tail -n 3
|
||||
820 50.87.54.15
|
||||
12872 70.32.99.142
|
||||
25744 70.32.83.92
|
||||
820 50.87.54.15
|
||||
12872 70.32.99.142
|
||||
25744 70.32.83.92
|
||||
# awk '{print $1}' /var/log/nginx/rest.log.1 | sort -n | uniq -c | sort -h | tail -n 3
|
||||
7966 181.118.144.29
|
||||
54706 70.32.99.142
|
||||
109412 70.32.83.92
|
||||
</code></pre>
|
||||
7966 181.118.144.29
|
||||
54706 70.32.99.142
|
||||
109412 70.32.83.92
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>Those are the same IPs that were hitting us heavily in July, 2016 as well…</li>
|
||||
<li>I think the stability issues are definitely from REST</li>
|
||||
<li>Crashed AGAIN, errors from dspace.log:</li>
|
||||
</ul>
|
||||
<li><p>Those are the same IPs that were hitting us heavily in July, 2016 as well…</p></li>
|
||||
|
||||
<li><p>I think the stability issues are definitely from REST</p></li>
|
||||
|
||||
<li><p>Crashed AGAIN, errors from dspace.log:</p>
|
||||
|
||||
<pre><code>2016-09-14 14:31:43,069 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL connection Error -
|
||||
org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>And more heap space errors:</li>
|
||||
</ul>
|
||||
<li><p>And more heap space errors:</p>
|
||||
|
||||
<pre><code># grep -rsI "OutOfMemoryError" /var/log/tomcat7/catalina.* | wc -l
|
||||
19
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>There are no more rest requests since the last crash, so maybe there are other things causing this.</li>
|
||||
<li>Hmm, I noticed a shitload of IPs from 180.76.0.0/16 are connecting to both CGSpace and DSpace Test (58 unique IPs concurrently!)</li>
|
||||
<li>They seem to be coming from Baidu, and so far during today alone account for <sup>1</sup>⁄<sub>6</sub> of every connection:</li>
|
||||
</ul>
|
||||
<li><p>There are no more rest requests since the last crash, so maybe there are other things causing this.</p></li>
|
||||
|
||||
<li><p>Hmm, I noticed a shitload of IPs from 180.76.0.0/16 are connecting to both CGSpace and DSpace Test (58 unique IPs concurrently!)</p></li>
|
||||
|
||||
<li><p>They seem to be coming from Baidu, and so far during today alone account for <sup>1</sup>⁄<sub>6</sub> of every connection:</p>
|
||||
|
||||
<pre><code># grep -c ip_addr= /home/cgspace.cgiar.org/log/dspace.log.2016-09-14
|
||||
29084
|
||||
# grep -c ip_addr=180.76.15 /home/cgspace.cgiar.org/log/dspace.log.2016-09-14
|
||||
5192
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>Other recent days are the same… hmmm.</li>
|
||||
<li>From the activity control panel I can see 58 unique IPs hitting the site <em>concurrently</em>, which has GOT to hurt our stability</li>
|
||||
<li>A list of all 2000 unique IPs from CGSpace logs today:</li>
|
||||
</ul>
|
||||
<li><p>Other recent days are the same… hmmm.</p></li>
|
||||
|
||||
<li><p>From the activity control panel I can see 58 unique IPs hitting the site <em>concurrently</em>, which has GOT to hurt our stability</p></li>
|
||||
|
||||
<li><p>A list of all 2000 unique IPs from CGSpace logs today:</p>
|
||||
|
||||
<pre><code># grep ip_addr= /home/cgspace.cgiar.org/log/dspace.log.2016-09-11 | awk -F: '{print $5}' | sort -n | uniq -c | sort -h | tail -n 100
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>Looking at the top 20 IPs or so, most are Yahoo, MSN, Google, Baidu, TurnitIn (iParadigm), etc… do we have any real users?</li>
|
||||
<li>Generate a list of all author affiliations for Peter Ballantyne to go through, make corrections, and create a lookup list from:</li>
|
||||
</ul>
|
||||
<li><p>Looking at the top 20 IPs or so, most are Yahoo, MSN, Google, Baidu, TurnitIn (iParadigm), etc… do we have any real users?</p></li>
|
||||
|
||||
<li><p>Generate a list of all author affiliations for Peter Ballantyne to go through, make corrections, and create a lookup list from:</p>
|
||||
|
||||
<pre><code>dspacetest=# \copy (select text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=211 group by text_value order by count desc) to /tmp/affiliations.csv with csv;
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>Looking into the Catalina logs again around the time of the first crash, I see:</li>
|
||||
</ul>
|
||||
<li><p>Looking into the Catalina logs again around the time of the first crash, I see:</p>
|
||||
|
||||
<pre><code>Wed Sep 14 09:47:27 UTC 2016 | Query:id: 78581 AND type:2
|
||||
Wed Sep 14 09:47:28 UTC 2016 | Updating : 6/6 docs.
|
||||
@ -446,12 +419,11 @@ Commit
|
||||
Commit done
|
||||
dn:CN=Haman\, Magdalena (CIAT-CCAFS),OU=Standard,OU=Users,OU=HQ,OU=CIATHUB,dc=cgiarad,dc=org
|
||||
Exception in thread "http-bio-127.0.0.1-8081-exec-193" java.lang.OutOfMemoryError: Java heap space
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>And after that I see a bunch of “pool error Timeout waiting for idle object”</li>
|
||||
<li>Later, near the time of the next crash I see:</li>
|
||||
</ul>
|
||||
<li><p>And after that I see a bunch of “pool error Timeout waiting for idle object”</p></li>
|
||||
|
||||
<li><p>Later, near the time of the next crash I see:</p>
|
||||
|
||||
<pre><code>dn:CN=Haman\, Magdalena (CIAT-CCAFS),OU=Standard,OU=Users,OU=HQ,OU=CIATHUB,dc=cgiarad,dc=org
|
||||
Wed Sep 14 11:29:55 UTC 2016 | Query:id: 79078 AND type:2
|
||||
@ -462,27 +434,24 @@ Sep 14, 2016 11:32:22 AM com.sun.jersey.server.wadl.generators.WadlGeneratorJAXB
|
||||
SEVERE: Failed to generate the schema for the JAX-B elements
|
||||
com.sun.xml.bind.v2.runtime.IllegalAnnotationsException: 2 counts of IllegalAnnotationExceptions
|
||||
java.util.Map is an interface, and JAXB can't handle interfaces.
|
||||
this problem is related to the following location:
|
||||
at java.util.Map
|
||||
at public java.util.Map com.atmire.dspace.rest.common.Statlet.getRender()
|
||||
at com.atmire.dspace.rest.common.Statlet
|
||||
this problem is related to the following location:
|
||||
at java.util.Map
|
||||
at public java.util.Map com.atmire.dspace.rest.common.Statlet.getRender()
|
||||
at com.atmire.dspace.rest.common.Statlet
|
||||
java.util.Map does not have a no-arg default constructor.
|
||||
this problem is related to the following location:
|
||||
at java.util.Map
|
||||
at public java.util.Map com.atmire.dspace.rest.common.Statlet.getRender()
|
||||
at com.atmire.dspace.rest.common.Statlet
|
||||
</code></pre>
|
||||
this problem is related to the following location:
|
||||
at java.util.Map
|
||||
at public java.util.Map com.atmire.dspace.rest.common.Statlet.getRender()
|
||||
at com.atmire.dspace.rest.common.Statlet
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>Then 20 minutes later another outOfMemoryError:</li>
|
||||
</ul>
|
||||
<li><p>Then 20 minutes later another outOfMemoryError:</p>
|
||||
|
||||
<pre><code>Exception in thread "http-bio-127.0.0.1-8081-exec-25" java.lang.OutOfMemoryError: Java heap space
|
||||
at java.lang.StringCoding.decode(StringCoding.java:215)
|
||||
</code></pre>
|
||||
at java.lang.StringCoding.decode(StringCoding.java:215)
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>Perhaps these particular issues <em>are</em> memory issues, the munin graphs definitely show some weird purging/allocating behavior starting this week</li>
|
||||
<li><p>Perhaps these particular issues <em>are</em> memory issues, the munin graphs definitely show some weird purging/allocating behavior starting this week</p></li>
|
||||
</ul>
|
||||
|
||||
<p><img src="/cgspace-notes/2016/09/tomcat_jvm-day.png" alt="Tomcat JVM usage day" />
|
||||
@ -492,15 +461,15 @@ java.util.Map does not have a no-arg default constructor.
|
||||
<ul>
|
||||
<li>And really, we did reduce the memory of CGSpace in late 2015, so maybe we should just increase it again, now that our usage is higher and we are having memory errors in the logs</li>
|
||||
<li>Oh great, the configuration on the actual server is different than in configuration management!</li>
|
||||
<li>Seems we added a bunch of settings to the <code>/etc/default/tomcat7</code> in December, 2015 and never updated our ansible repository:</li>
|
||||
</ul>
|
||||
|
||||
<li><p>Seems we added a bunch of settings to the <code>/etc/default/tomcat7</code> in December, 2015 and never updated our ansible repository:</p>
|
||||
|
||||
<pre><code>JAVA_OPTS="-Djava.awt.headless=true -Xms3584m -Xmx3584m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -Dfile.encoding=UTF-8 -XX:-UseGCOverheadLimit -XX:MaxGCPauseMillis=250 -XX:GCTimeRatio=9 -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled -XX:G1HeapRegionSize=8m -XX:InitiatingHeapOccupancyPercent=75 -XX:+UseLargePages -XX:+AggressiveOpts"
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>So I’m going to bump the heap +512m and remove all the other experimental shit (and update ansible!)</li>
|
||||
<li>Increased JVM heap to 4096m on CGSpace (linode01)</li>
|
||||
<li><p>So I’m going to bump the heap +512m and remove all the other experimental shit (and update ansible!)</p></li>
|
||||
|
||||
<li><p>Increased JVM heap to 4096m on CGSpace (linode01)</p></li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2016-09-15">2016-09-15</h2>
|
||||
@ -514,8 +483,7 @@ java.util.Map does not have a no-arg default constructor.
|
||||
<h2 id="2016-09-16">2016-09-16</h2>
|
||||
|
||||
<ul>
|
||||
<li>CGSpace crashed again, and there are TONS of heap space errors but the datestamps aren’t on those lines so I’m not sure if they were yesterday:</li>
|
||||
</ul>
|
||||
<li><p>CGSpace crashed again, and there are TONS of heap space errors but the datestamps aren’t on those lines so I’m not sure if they were yesterday:</p>
|
||||
|
||||
<pre><code>dn:CN=Orentlicher\, Natalie (CIAT),OU=Standard,OU=Users,OU=HQ,OU=CIATHUB,dc=cgiarad,dc=org
|
||||
Thu Sep 15 18:45:25 UTC 2016 | Query:id: 55785 AND type:2
|
||||
@ -533,41 +501,38 @@ Exception in thread "http-bio-127.0.0.1-8081-exec-263" java.lang.OutOf
|
||||
Exception in thread "http-bio-127.0.0.1-8081-exec-280" java.lang.OutOfMemoryError: Java heap space
|
||||
Exception in thread "Thread-54216" org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Exception writing document id 7feaa95d-8e1f-4f45-80bb
|
||||
-e14ef82ee224 to the index; possible analysis error.
|
||||
at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
|
||||
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
|
||||
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
|
||||
at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
|
||||
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
|
||||
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
|
||||
at com.atmire.statistics.SolrLogThread.run(SourceFile:25)
|
||||
</code></pre>
|
||||
at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
|
||||
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
|
||||
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
|
||||
at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
|
||||
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
|
||||
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
|
||||
at com.atmire.statistics.SolrLogThread.run(SourceFile:25)
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>I bumped the heap space from 4096m to 5120m to see if this is <em>really</em> about heap speace or not.</li>
|
||||
<li>Looking into some of these errors that I’ve seen this week but haven’t noticed before:</li>
|
||||
</ul>
|
||||
<li><p>I bumped the heap space from 4096m to 5120m to see if this is <em>really</em> about heap speace or not.</p></li>
|
||||
|
||||
<li><p>Looking into some of these errors that I’ve seen this week but haven’t noticed before:</p>
|
||||
|
||||
<pre><code># zcat -f -- /var/log/tomcat7/catalina.* | grep -c 'Failed to generate the schema for the JAX-B elements'
|
||||
113
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>I’ve sent a message to Atmire about the Solr error to see if it’s related to their batch update module</li>
|
||||
<li><p>I’ve sent a message to Atmire about the Solr error to see if it’s related to their batch update module</p></li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2016-09-19">2016-09-19</h2>
|
||||
|
||||
<ul>
|
||||
<li>Work on cleanups for author affiliations after Peter sent me his list of corrections/deletions:</li>
|
||||
</ul>
|
||||
<li><p>Work on cleanups for author affiliations after Peter sent me his list of corrections/deletions:</p>
|
||||
|
||||
<pre><code>$ ./fix-metadata-values.py -i affiliations_pb-322-corrections.csv -f cg.contributor.affiliation -t correct -m 211 -d dspace -u dspace -p fuuu
|
||||
$ ./delete-metadata-values.py -f cg.contributor.affiliation -i affiliations_pb-2-deletions.csv -m 211 -u dspace -d dspace -p fuuu
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>After that we need to take the top ~300 and make a controlled vocabulary for it</li>
|
||||
<li>I dumped a list of the top 300 affiliations from the database, sorted it alphabetically in OpenRefine, and created a controlled vocabulary for it (<a href="https://github.com/ilri/DSpace/pull/267">#267</a>)</li>
|
||||
<li><p>After that we need to take the top ~300 and make a controlled vocabulary for it</p></li>
|
||||
|
||||
<li><p>I dumped a list of the top 300 affiliations from the database, sorted it alphabetically in OpenRefine, and created a controlled vocabulary for it (<a href="https://github.com/ilri/DSpace/pull/267">#267</a>)</p></li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2016-09-20">2016-09-20</h2>
|
||||
@ -587,42 +552,42 @@ $ ./delete-metadata-values.py -f cg.contributor.affiliation -i affiliations_pb-2
|
||||
|
||||
<ul>
|
||||
<li>Turns out the Solr search logic switched from OR to AND in DSpace 6.0 and the change is easy to backport: <a href="https://jira.duraspace.org/browse/DS-2809">https://jira.duraspace.org/browse/DS-2809</a></li>
|
||||
<li>We just need to set this in <code>dspace/solr/search/conf/schema.xml</code>:</li>
|
||||
</ul>
|
||||
|
||||
<li><p>We just need to set this in <code>dspace/solr/search/conf/schema.xml</code>:</p>
|
||||
|
||||
<pre><code><solrQueryParser defaultOperator="AND"/>
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>It actually works really well, and search results return much less hits now (before, after):</li>
|
||||
<li><p>It actually works really well, and search results return much less hits now (before, after):</p></li>
|
||||
</ul>
|
||||
|
||||
<p><img src="/cgspace-notes/2016/09/cgspace-search.png" alt="CGSpace search with "OR" boolean logic" />
|
||||
<img src="/cgspace-notes/2016/09/dspacetest-search.png" alt="DSpace Test search with "AND" boolean logic" /></p>
|
||||
|
||||
<ul>
|
||||
<li>Found a way to improve the configuration of Atmire’s Content and Usage Analysis (CUA) module for date fields</li>
|
||||
</ul>
|
||||
<li><p>Found a way to improve the configuration of Atmire’s Content and Usage Analysis (CUA) module for date fields</p>
|
||||
|
||||
<pre><code>-content.analysis.dataset.option.8=metadata:dateAccessioned:discovery
|
||||
+content.analysis.dataset.option.8=metadata:dc.date.accessioned:date(month)
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>This allows the module to treat the field as a date rather than a text string, so we can interrogate it more intelligently</li>
|
||||
<li>Add <code>dc.date.accessioned</code> to XMLUI Discovery search filters</li>
|
||||
<li>Major CGSpace crash because ILRI forgot to pay the Linode bill</li>
|
||||
<li>45 minutes of downtime!</li>
|
||||
<li>Start processing the fixes to <code>dc.description.sponsorship</code> from Peter Ballantyne:</li>
|
||||
</ul>
|
||||
<li><p>This allows the module to treat the field as a date rather than a text string, so we can interrogate it more intelligently</p></li>
|
||||
|
||||
<li><p>Add <code>dc.date.accessioned</code> to XMLUI Discovery search filters</p></li>
|
||||
|
||||
<li><p>Major CGSpace crash because ILRI forgot to pay the Linode bill</p></li>
|
||||
|
||||
<li><p>45 minutes of downtime!</p></li>
|
||||
|
||||
<li><p>Start processing the fixes to <code>dc.description.sponsorship</code> from Peter Ballantyne:</p>
|
||||
|
||||
<pre><code>$ ./fix-metadata-values.py -i sponsors-fix-23.csv -f dc.description.sponsorship -t correct -m 29 -d dspace -u dspace -p fuuu
|
||||
$ ./delete-metadata-values.py -i sponsors-delete-8.csv -f dc.description.sponsorship -m 29 -d dspace -u dspace -p fuuu
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>I need to run these and the others from a few days ago on CGSpace the next time we run updates</li>
|
||||
<li>Also, I need to update the controlled vocab for sponsors based on these</li>
|
||||
<li><p>I need to run these and the others from a few days ago on CGSpace the next time we run updates</p></li>
|
||||
|
||||
<li><p>Also, I need to update the controlled vocab for sponsors based on these</p></li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2016-09-22">2016-09-22</h2>
|
||||
@ -639,18 +604,19 @@ $ ./delete-metadata-values.py -i sponsors-delete-8.csv -f dc.description.sponsor
|
||||
<li>Merge updates to sponsorship controlled vocabulary (<a href="https://github.com/ilri/DSpace/pull/277">#277</a>)</li>
|
||||
<li>I’ve been trying to add a search filter for <code>dc.description</code> so the IITA people can search for some tags they use there, but for some reason the filter never shows up in Atmire’s CUA</li>
|
||||
<li>Not sure if it’s something like we already have too many filters there (30), or the filter name is reserved, etc…</li>
|
||||
<li>Generate a list of ILRI subjects for Peter and Abenet to look through/fix:</li>
|
||||
</ul>
|
||||
|
||||
<li><p>Generate a list of ILRI subjects for Peter and Abenet to look through/fix:</p>
|
||||
|
||||
<pre><code>dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where resource_type_id=2 and metadata_field_id=203 group by text_value order by count desc) to /tmp/ilrisubjects.csv with csv;
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>Regenerate Discovery indexes a few times after playing with <code>discovery.xml</code> index definitions (syntax, parameters, etc).</li>
|
||||
<li>Merge changes to boolean logic in Solr search (<a href="https://github.com/ilri/DSpace/pull/274">#274</a>)</li>
|
||||
<li>Run all sponsorship and affiliation fixes on CGSpace, deploy latest <code>5_x-prod</code> branch, and re-index Discovery on CGSpace</li>
|
||||
<li>Tested OCSP stapling on DSpace Test’s nginx and it works:</li>
|
||||
</ul>
|
||||
<li><p>Regenerate Discovery indexes a few times after playing with <code>discovery.xml</code> index definitions (syntax, parameters, etc).</p></li>
|
||||
|
||||
<li><p>Merge changes to boolean logic in Solr search (<a href="https://github.com/ilri/DSpace/pull/274">#274</a>)</p></li>
|
||||
|
||||
<li><p>Run all sponsorship and affiliation fixes on CGSpace, deploy latest <code>5_x-prod</code> branch, and re-index Discovery on CGSpace</p></li>
|
||||
|
||||
<li><p>Tested OCSP stapling on DSpace Test’s nginx and it works:</p>
|
||||
|
||||
<pre><code>$ openssl s_client -connect dspacetest.cgiar.org:443 -servername dspacetest.cgiar.org -tls1_2 -tlsextdebug -status
|
||||
...
|
||||
@ -658,48 +624,48 @@ OCSP response:
|
||||
======================================
|
||||
OCSP Response Data:
|
||||
...
|
||||
Cert Status: good
|
||||
</code></pre>
|
||||
Cert Status: good
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>I’ve been monitoring this for almost two years in this GitHub issue: <a href="https://github.com/ilri/DSpace/issues/38">https://github.com/ilri/DSpace/issues/38</a></li>
|
||||
<li><p>I’ve been monitoring this for almost two years in this GitHub issue: <a href="https://github.com/ilri/DSpace/issues/38">https://github.com/ilri/DSpace/issues/38</a></p></li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2016-09-27">2016-09-27</h2>
|
||||
|
||||
<ul>
|
||||
<li>Discuss fixing some ORCIDs for CCAFS author Sonja Vermeulen with Magdalena Haman</li>
|
||||
<li>This author has a few variations:</li>
|
||||
</ul>
|
||||
|
||||
<li><p>This author has a few variations:</p>
|
||||
|
||||
<pre><code>dspacetest=# select distinct text_value, authority, confidence from metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value like 'Vermeu
|
||||
len, S%';
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>And it looks like <code>fe4b719f-6cc4-4d65-8504-7a83130b9f83</code> is the authority with the correct ORCID linked</li>
|
||||
</ul>
|
||||
<li><p>And it looks like <code>fe4b719f-6cc4-4d65-8504-7a83130b9f83</code> is the authority with the correct ORCID linked</p>
|
||||
|
||||
<pre><code>dspacetest=# update metadatavalue set authority='fe4b719f-6cc4-4d65-8504-7a83130b9f83w', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Vermeulen, S%';
|
||||
UPDATE 101
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>Hmm, now her name is missing from the authors facet and only shows the authority ID</li>
|
||||
<li>On the production server there is an item with her ORCID but it is using a different authority: f01f7b7b-be3f-4df7-a61d-b73c067de88d</li>
|
||||
<li>Maybe I used the wrong one… I need to look again at the production database</li>
|
||||
<li>On a clean snapshot of the database I see the correct authority should be <code>f01f7b7b-be3f-4df7-a61d-b73c067de88d</code>, not <code>fe4b719f-6cc4-4d65-8504-7a83130b9f83</code></li>
|
||||
<li>Updating her authorities again and reindexing:</li>
|
||||
</ul>
|
||||
<li><p>Hmm, now her name is missing from the authors facet and only shows the authority ID</p></li>
|
||||
|
||||
<li><p>On the production server there is an item with her ORCID but it is using a different authority: f01f7b7b-be3f-4df7-a61d-b73c067de88d</p></li>
|
||||
|
||||
<li><p>Maybe I used the wrong one… I need to look again at the production database</p></li>
|
||||
|
||||
<li><p>On a clean snapshot of the database I see the correct authority should be <code>f01f7b7b-be3f-4df7-a61d-b73c067de88d</code>, not <code>fe4b719f-6cc4-4d65-8504-7a83130b9f83</code></p></li>
|
||||
|
||||
<li><p>Updating her authorities again and reindexing:</p>
|
||||
|
||||
<pre><code>dspacetest=# update metadatavalue set authority='f01f7b7b-be3f-4df7-a61d-b73c067de88d', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Vermeulen, S%';
|
||||
UPDATE 101
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>Use GitHub icon from Font Awesome instead of a PNG to save one extra network request</li>
|
||||
<li>We can also replace the RSS and mail icons in community text!</li>
|
||||
<li>Fix reference to <code>dc.type.*</code> in Atmire CUA module, as we now only index <code>dc.type</code> for “Output type”</li>
|
||||
<li><p>Use GitHub icon from Font Awesome instead of a PNG to save one extra network request</p></li>
|
||||
|
||||
<li><p>We can also replace the RSS and mail icons in community text!</p></li>
|
||||
|
||||
<li><p>Fix reference to <code>dc.type.*</code> in Atmire CUA module, as we now only index <code>dc.type</code> for “Output type”</p></li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2016-09-28">2016-09-28</h2>
|
||||
@ -711,22 +677,23 @@ UPDATE 101
|
||||
<li>Going to try to update Sonja Vermeulen’s authority to 2b4166b7-6e4d-4f66-9d8b-ddfbec9a6ae0, as that seems to be one of her authorities that has an ORCID</li>
|
||||
<li>Merge Font Awesome changes (<a href="https://github.com/ilri/DSpace/pull/279">#279</a>)</li>
|
||||
<li>Minor fix to a string in Atmire’s CUA module (<a href="https://github.com/ilri/DSpace/pull/280">#280</a>)</li>
|
||||
<li>This seems to be what I’ll need to do for Sonja Vermeulen (but with <code>2b4166b7-6e4d-4f66-9d8b-ddfbec9a6ae0</code> instead on the live site):</li>
|
||||
</ul>
|
||||
|
||||
<li><p>This seems to be what I’ll need to do for Sonja Vermeulen (but with <code>2b4166b7-6e4d-4f66-9d8b-ddfbec9a6ae0</code> instead on the live site):</p>
|
||||
|
||||
<pre><code>dspacetest=# update metadatavalue set authority='09e4da69-33a3-45ca-b110-7d3f82d2d6d2', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Vermeulen, S%';
|
||||
dspacetest=# update metadatavalue set authority='09e4da69-33a3-45ca-b110-7d3f82d2d6d2', confidence=600 where metadata_field_id=3 and resource_type_id=2 and text_value like 'Vermeulen SJ%';
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
|
||||
<ul>
|
||||
<li>And then update Discovery and Authority indexes</li>
|
||||
<li>Minor fix for “Subject” string in Discovery search and Atmire modules (<a href="https://github.com/ilri/DSpace/pull/281">#281</a>)</li>
|
||||
<li>Start testing batch fixes for ILRI subject from Peter:</li>
|
||||
</ul>
|
||||
<li><p>And then update Discovery and Authority indexes</p></li>
|
||||
|
||||
<li><p>Minor fix for “Subject” string in Discovery search and Atmire modules (<a href="https://github.com/ilri/DSpace/pull/281">#281</a>)</p></li>
|
||||
|
||||
<li><p>Start testing batch fixes for ILRI subject from Peter:</p>
|
||||
|
||||
<pre><code>$ ./fix-metadata-values.py -i ilrisubjects-fix-32.csv -f cg.subject.ilri -t correct -m 203 -d dspace -u dspace -p fuuuu
|
||||
$ ./delete-metadata-values.py -i ilrisubjects-delete-13.csv -f cg.subject.ilri -m 203 -d dspace -u dspace -p fuuu
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2016-09-29">2016-09-29</h2>
|
||||
|
||||
@ -734,11 +701,12 @@ $ ./delete-metadata-values.py -i ilrisubjects-delete-13.csv -f cg.subject.ilri -
|
||||
<li>Add <code>cg.identifier.ciatproject</code> to metadata registry in preparation for CIAT project tag</li>
|
||||
<li>Merge changes for CIAT project tag (<a href="https://github.com/ilri/DSpace/pull/282">#282</a>)</li>
|
||||
<li>DSpace Test (linode02) became unresponsive for some reason, I had to hard reboot it from the Linode console</li>
|
||||
<li>People on DSpace mailing list gave me a query to get authors from certain collections:</li>
|
||||
</ul>
|
||||
|
||||
<li><p>People on DSpace mailing list gave me a query to get authors from certain collections:</p>
|
||||
|
||||
<pre><code>dspacetest=# select distinct text_value from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 AND resource_id IN (select item_id from collection2item where collection_id IN (select resource_id from handle where handle in ('10568/5472', '10568/5473')));
|
||||
</code></pre>
|
||||
</code></pre></li>
|
||||
</ul>
|
||||
|
||||
<h2 id="2016-09-30">2016-09-30</h2>
|
||||
|
||||
|
||||
Reference in New Issue
Block a user