alanorth/cgspace-notes
1
0
Fork 0
mirror of https://github.com/alanorth/cgspace-notes.git synced 2025-01-27 05:49:12 +01:00
Code Issues Projects Releases Wiki Activity

Add notes for 2019-12-17

Browse Source
This commit is contained in:
Alan Orth
2019-12-17 14:49:24 +02:00
parent d83c951532
commit d54e5b69f1
90 changed files with 1420 additions and 1377 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
docs/2016-09/index.html
View File

@ -31,7 +31,7 @@ It looks like we might be able to use OUs now, instead of DCs:
$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=org" -D "admigration1@cgiarad.org" -W "(sAMAccountName=admigration1)"
"/>
<meta name="generator" content="Hugo 0.60.1" />
<meta name="generator" content="Hugo 0.61.0" />
@ -112,7 +112,7 @@ $ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b &quot;dc=cgiarad,dc=or
</p>
</header>
<h2 id="20160901">2016-09-01</h2>
<h2 id="2016-09-01">2016-09-01</h2>
<ul>
<li>Discuss helping CCAFS with some batch tagging of ORCID IDs for their authors</li>
<li>Discuss how the migration of CGIAR's Active Directory to a flat structure will break our LDAP groups in DSpace</li>
@ -203,7 +203,7 @@ dspacetest=# select distinct text_value, authority, confidence from metadatavalu
<li>After updating the Authority indexes (<code>bin/dspace index-authority</code>) everything looks good</li>
<li>Run authority updates on CGSpace</li>
</ul>
<h2 id="20160905">2016-09-05</h2>
<h2 id="2016-09-05">2016-09-05</h2>
<ul>
<li>After one week of logging TLS connections on CGSpace:</li>
</ul>
@ -222,7 +222,7 @@ TLSv1/EDH-RSA-DES-CBC3-SHA
</code></pre><ul>
<li>This gives you, for example: <code>Mainstreaming gender in agricultural R&amp;D.pdf__description:Brief</code></li>
</ul>
<h2 id="20160906">2016-09-06</h2>
<h2 id="2016-09-06">2016-09-06</h2>
<ul>
<li>Trying to import the records for CIAT from yesterday, but having filename encoding issues from their zip file</li>
<li>Create a zip on Mac OS X from a SAF bundle containing only one record with one PDF:
@ -258,7 +258,7 @@ TLSv1/EDH-RSA-DES-CBC3-SHA
<pre><code>$ ./safbuilder.sh -c /home/aorth/ciat-gender-2016-09-06/66601.csv
$ JAVA_OPTS=&quot;-Dfile.encoding=UTF-8 -Xmx512m&quot; /home/cgspace.cgiar.org/bin/dspace import -a -e aorth@mjanja.ch -c 10568/66601 -s /home/aorth/ciat-gender-2016-09-06/SimpleArchiveFormat -m 66601.map
$ rm -rf ~/ciat-gender-2016-09-06/SimpleArchiveFormat/
</code></pre><h2 id="20160907">2016-09-07</h2>
</code></pre><h2 id="2016-09-07">2016-09-07</h2>
<ul>
<li>Erase and rebuild DSpace Test based on latest Ubuntu 16.04, PostgreSQL 9.5, and Java 8 stuff</li>
<li>Reading about PostgreSQL maintenance and it seems manual vacuuming is only for certain workloads, such as heavy update/write loads</li>
@ -272,7 +272,7 @@ org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error
</code></pre><ul>
<li>Since CGSpace had crashed I quickly deployed the new LDAP settings before restarting Tomcat</li>
</ul>
<h2 id="20160913">2016-09-13</h2>
<h2 id="2016-09-13">2016-09-13</h2>
<ul>
<li>CGSpace crashed twice today, errors from <code>catalina.out</code>:</li>
</ul>
@ -281,7 +281,7 @@ org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error
</code></pre><ul>
<li>I enabled logging of requests to <code>/rest</code> again</li>
</ul>
<h2 id="20160914">2016-09-14</h2>
<h2 id="2016-09-14">2016-09-14</h2>
<ul>
<li>CGSpace crashed again, errors from <code>catalina.out</code>:</li>
</ul>
@ -399,12 +399,12 @@ java.util.Map does not have a no-arg default constructor.
<li>So I'm going to bump the heap +512m and remove all the other experimental shit (and update ansible!)</li>
<li>Increased JVM heap to 4096m on CGSpace (linode01)</li>
</ul>
<h2 id="20160915">2016-09-15</h2>
<h2 id="2016-09-15">2016-09-15</h2>
<ul>
<li>Looking at Google Webmaster Tools again, it seems the work I did on URL query parameters and blocking via the <code>X-Robots-Tag</code> HTTP header in March, 2016 seem to have had a positive effect on Google's index for CGSpace</li>
</ul>
<p><img src="/cgspace-notes/2016/09/google-webmaster-tools-index.png" alt="Google Webmaster Tools for CGSpace"></p>
<h2 id="20160916">2016-09-16</h2>
<h2 id="2016-09-16">2016-09-16</h2>
<ul>
<li>CGSpace crashed again, and there are TONS of heap space errors but the datestamps aren't on those lines so I'm not sure if they were yesterday:</li>
</ul>
@ -440,7 +440,7 @@ Exception in thread &quot;Thread-54216&quot; org.apache.solr.client.solrj.impl.H
</code></pre><ul>
<li>I've sent a message to Atmire about the Solr error to see if it's related to their batch update module</li>
</ul>
<h2 id="20160919">2016-09-19</h2>
<h2 id="2016-09-19">2016-09-19</h2>
<ul>
<li>Work on cleanups for author affiliations after Peter sent me his list of corrections/deletions:</li>
</ul>
@ -450,7 +450,7 @@ $ ./delete-metadata-values.py -f cg.contributor.affiliation -i affiliations_pb-2
<li>After that we need to take the top ~300 and make a controlled vocabulary for it</li>
<li>I dumped a list of the top 300 affiliations from the database, sorted it alphabetically in OpenRefine, and created a controlled vocabulary for it (<a href="https://github.com/ilri/DSpace/pull/267">#267</a>)</li>
</ul>
<h2 id="20160920">2016-09-20</h2>
<h2 id="2016-09-20">2016-09-20</h2>
<ul>
<li>Run all system updates on DSpace Test and reboot the server</li>
<li>Merge changes for sponsorship and affiliation controlled vocabularies (<a href="https://github.com/ilri/DSpace/pull/267">#267</a>, <a href="https://github.com/ilri/DSpace/pull/268">#268</a>)</li>
@ -461,7 +461,7 @@ $ ./delete-metadata-values.py -f cg.contributor.affiliation -i affiliations_pb-2
<li>I need to read the docs and ask on the mailing list to see if we can tweak that</li>
<li>Generate a new list of sponsors from the database for Peter Ballantyne so we can clean them up and update the controlled vocabulary</li>
</ul>
<h2 id="20160921">2016-09-21</h2>
<h2 id="2016-09-21">2016-09-21</h2>
<ul>
<li>Turns out the Solr search logic switched from OR to AND in DSpace 6.0 and the change is easy to backport: <a href="https://jira.duraspace.org/browse/DS-2809">https://jira.duraspace.org/browse/DS-2809</a></li>
<li>We just need to set this in <code>dspace/solr/search/conf/schema.xml</code>:</li>
@ -490,11 +490,11 @@ $ ./delete-metadata-values.py -i sponsors-delete-8.csv -f dc.description.sponsor
<li>I need to run these and the others from a few days ago on CGSpace the next time we run updates</li>
<li>Also, I need to update the controlled vocab for sponsors based on these</li>
</ul>
<h2 id="20160922">2016-09-22</h2>
<h2 id="2016-09-22">2016-09-22</h2>
<ul>
<li>Update controlled vocabulary for sponsorship based on the latest corrected values from the database</li>
</ul>
<h2 id="20160925">2016-09-25</h2>
<h2 id="2016-09-25">2016-09-25</h2>
<ul>
<li>Merge accession date improvements for CUA module (<a href="https://github.com/ilri/DSpace/pull/275">#275</a>)</li>
<li>Merge addition of accession date to Discovery search filters (<a href="https://github.com/ilri/DSpace/pull/276">#276</a>)</li>
@ -520,7 +520,7 @@ OCSP Response Data:
</code></pre><ul>
<li>I've been monitoring this for almost two years in this GitHub issue: <a href="https://github.com/ilri/DSpace/issues/38">https://github.com/ilri/DSpace/issues/38</a></li>
</ul>
<h2 id="20160927">2016-09-27</h2>
<h2 id="2016-09-27">2016-09-27</h2>
<ul>
<li>Discuss fixing some ORCIDs for CCAFS author Sonja Vermeulen with Magdalena Haman</li>
<li>This author has a few variations:</li>
@ -546,7 +546,7 @@ UPDATE 101
<li>We can also replace the RSS and mail icons in community text!</li>
<li>Fix reference to <code>dc.type.*</code> in Atmire CUA module, as we now only index <code>dc.type</code> for &ldquo;Output type&rdquo;</li>
</ul>
<h2 id="20160928">2016-09-28</h2>
<h2 id="2016-09-28">2016-09-28</h2>
<ul>
<li>Make a placeholder pull request for <code>discovery.xml</code> changes (<a href="https://github.com/ilri/DSpace/pull/278">#278</a>), as I still need to test their effect on Atmire content analysis module</li>
<li>Make a placeholder pull request for Font Awesome changes (<a href="https://github.com/ilri/DSpace/pull/279">#279</a>), which replaces the GitHub image in the footer with an icon, and add style for RSS and @ icons that I will start replacing in community/collection HTML intros</li>
@ -565,7 +565,7 @@ dspacetest=# update metadatavalue set authority='09e4da69-33a3-45ca-b110-7d3f82d
</ul>
<pre><code>$ ./fix-metadata-values.py -i ilrisubjects-fix-32.csv -f cg.subject.ilri -t correct -m 203 -d dspace -u dspace -p fuuuu
$ ./delete-metadata-values.py -i ilrisubjects-delete-13.csv -f cg.subject.ilri -m 203 -d dspace -u dspace -p fuuu
</code></pre><h2 id="20160929">2016-09-29</h2>
</code></pre><h2 id="2016-09-29">2016-09-29</h2>
<ul>
<li>Add <code>cg.identifier.ciatproject</code> to metadata registry in preparation for CIAT project tag</li>
<li>Merge changes for CIAT project tag (<a href="https://github.com/ilri/DSpace/pull/282">#282</a>)</li>
@ -573,7 +573,7 @@ $ ./delete-metadata-values.py -i ilrisubjects-delete-13.csv -f cg.subject.ilri -
<li>People on DSpace mailing list gave me a query to get authors from certain collections:</li>
</ul>
<pre><code>dspacetest=# select distinct text_value from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 AND resource_id IN (select item_id from collection2item where collection_id IN (select resource_id from handle where handle in ('10568/5472', '10568/5473')));
</code></pre><h2 id="20160930">2016-09-30</h2>
</code></pre><h2 id="2016-09-30">2016-09-30</h2>
<ul>
<li>Deny access to REST API's <code>find-by-metadata-field</code> endpoint to protect against an upstream security issue (DS-3250)</li>
<li>There is a patch but it is only for 5.5 and doesn't apply cleanly to 5.1</li>