alanorth/cgspace-notes
1
0
Fork 0
mirror of https://github.com/alanorth/cgspace-notes.git synced 2025-01-27 05:49:12 +01:00
Code Issues Projects Releases Wiki Activity

Add notes for 2020-03-22

Browse Source
This commit is contained in:
Alan Orth
2020-03-22 14:35:20 +02:00
parent 711b76f386
commit f613ae9f7a
89 changed files with 147 additions and 134 deletions
Download Patch File Download Diff File Expand all files Collapse all files

54
docs/cgiar-library-migration/index.html
View File

@ -15,7 +15,7 @@
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="CGIAR Library Migration"/>
<meta name="twitter:description" content="Notes on the migration of the CGIAR Library to CGSpace"/>
<meta name="generator" content="Hugo 0.67.1" />
<meta name="generator" content="Hugo 0.68.0" />
@ -108,26 +108,26 @@
<h2 id="pre-migration-technical-todos">Pre-migration Technical TODOs</h2>
<p>Things that need to happen before the migration:</p>
<ul>
<li><input checked="" disabled="" type="checkbox">Create top-level community on CGSpace to hold the CGIAR Library content: <code>10568/83389</code>
<li><input checked="" disabled="" type="checkbox"> Create top-level community on CGSpace to hold the CGIAR Library content: <code>10568/83389</code>
<ul>
<li><input checked="" disabled="" type="checkbox">Update nginx redirects in ansible templates</li>
<li><input checked="" disabled="" type="checkbox">Update handle in DSpace XMLUI config</li>
<li><input checked="" disabled="" type="checkbox"> Update nginx redirects in ansible templates</li>
<li><input checked="" disabled="" type="checkbox"> Update handle in DSpace XMLUI config</li>
</ul>
</li>
<li>Set up nginx redirects for URLs like:
<ul>
<li><input checked="" disabled="" type="checkbox"><a href="https://library.cgiar.org/bitstream/handle/10947/2699/CGIAR_Branding_Guidelines_and_Toolkit.pdf">https://library.cgiar.org/bitstream/handle/10947/2699/CGIAR_Branding_Guidelines_and_Toolkit.pdf</a></li>
<li><input checked="" disabled="" type="checkbox"><a href="https://library.cgiar.org/handle/10947/4258">https://library.cgiar.org/handle/10947/4258</a></li>
<li><input checked="" disabled="" type="checkbox"> <a href="https://library.cgiar.org/bitstream/handle/10947/2699/CGIAR_Branding_Guidelines_and_Toolkit.pdf">https://library.cgiar.org/bitstream/handle/10947/2699/CGIAR_Branding_Guidelines_and_Toolkit.pdf</a></li>
<li><input checked="" disabled="" type="checkbox"> <a href="https://library.cgiar.org/handle/10947/4258">https://library.cgiar.org/handle/10947/4258</a></li>
</ul>
</li>
<li><input checked="" disabled="" type="checkbox">Merge <a href="https://github.com/ilri/DSpace/pull/339">#339</a> to <code>5_x-prod</code> branch and rebuild DSpace</li>
<li><input checked="" disabled="" type="checkbox">Increase <code>max_connections</code> in <code>/etc/postgresql/9.5/main/postgresql.conf</code> by ~10
<li><input checked="" disabled="" type="checkbox"> Merge <a href="https://github.com/ilri/DSpace/pull/339">#339</a> to <code>5_x-prod</code> branch and rebuild DSpace</li>
<li><input checked="" disabled="" type="checkbox"> Increase <code>max_connections</code> in <code>/etc/postgresql/9.5/main/postgresql.conf</code> by ~10
<ul>
<li><code>SELECT * FROM pg_stat_activity;</code> seems to show ~6 extra connections used by the command line tools during import</li>
</ul>
</li>
<li><input checked="" disabled="" type="checkbox">Temporarily disable nightly <code>index-discovery</code> cron job because the import process will be taking place during some of this time and I don&rsquo;t want them to be competing to update the Solr index</li>
<li><input checked="" disabled="" type="checkbox">Copy HTTPS certificate key pair from CGIAR Library server&rsquo;s Tomcat keystore:</li>
<li><input checked="" disabled="" type="checkbox"> Temporarily disable nightly <code>index-discovery</code> cron job because the import process will be taking place during some of this time and I don&rsquo;t want them to be competing to update the Solr index</li>
<li><input checked="" disabled="" type="checkbox"> Copy HTTPS certificate key pair from CGIAR Library server&rsquo;s Tomcat keystore:</li>
</ul>
<pre><code>$ keytool -list -keystore tomcat.keystore
$ keytool -importkeystore -srckeystore tomcat.keystore -destkeystore library.cgiar.org.p12 -deststoretype PKCS12 -srcalias tomcat
@ -152,13 +152,13 @@ $ dspace packager -d -a -t AIP -e aorth@mjanja.ch -i 10568/93760 10568-93760/105
$ dspace packager -d -a -t AIP -e aorth@mjanja.ch -i 10947/1 10947-1/10947-1.zip
</code></pre><p><strong>Import to CGSpace (also see <a href="http://alanorth.github.io/cgspace-notes/2017-05/#2017-05-10">notes from 2017-05-10</a>):</strong></p>
<ul>
<li><input checked="" disabled="" type="checkbox">Copy all exports from DSpace Test</li>
<li><input checked="" disabled="" type="checkbox">Add ingestion overrides to <code>dspace.cfg</code> before import:</li>
<li><input checked="" disabled="" type="checkbox"> Copy all exports from DSpace Test</li>
<li><input checked="" disabled="" type="checkbox"> Add ingestion overrides to <code>dspace.cfg</code> before import:</li>
</ul>
<pre><code>mets.dspaceAIP.ingest.crosswalk.METSRIGHTS = NIL
mets.dspaceAIP.ingest.crosswalk.DSPACE-ROLES = NIL
</code></pre><ul>
<li><input checked="" disabled="" type="checkbox">Import communities and collections, paying attention to options to skip missing parents and ignore handles:</li>
<li><input checked="" disabled="" type="checkbox"> Import communities and collections, paying attention to options to skip missing parents and ignore handles:</li>
</ul>
<pre><code>$ export JAVA_OPTS=&quot;-Dfile.encoding=UTF-8 -Xmx3072m -XX:-UseGCOverheadLimit -XX:+TieredCompilation -XX:TieredStopAtLevel=1&quot;
$ export PATH=$PATH:/home/cgspace.cgiar.org/bin
@ -179,9 +179,9 @@ $ for item in 10947-1/ITEM@10947-*; do dspace packager -r -f -u -t AIP -e aorth@
</code></pre><p>This submits AIP hierarchies recursively (-r) and suppresses errors when an item&rsquo;s parent collection hasn&rsquo;t been created yet—for example, if the item is mapped. The large historic archive (10947/1) is created in several steps because it requires a lot of memory and often crashes.</p>
<p><strong>Create new subcommunities and collections for content we reorganized into new hierarchies from the original:</strong></p>
<ul>
<li><input checked="" disabled="" type="checkbox">Create <em>CGIAR System Management Board</em> sub-community: <code>10568/83536</code>
<li><input checked="" disabled="" type="checkbox"> Create <em>CGIAR System Management Board</em> sub-community: <code>10568/83536</code>
<ul>
<li><input checked="" disabled="" type="checkbox">Content from <em>CGIAR System Management Board documents</em> collection (<code>10947/4561</code>) goes here</li>
<li><input checked="" disabled="" type="checkbox"> Content from <em>CGIAR System Management Board documents</em> collection (<code>10947/4561</code>) goes here</li>
<li>Import collection hierarchy first and then the items:</li>
</ul>
</li>
@ -189,9 +189,9 @@ $ for item in 10947-1/ITEM@10947-*; do dspace packager -r -f -u -t AIP -e aorth@
<pre><code>$ dspace packager -r -t AIP -o ignoreHandle=false -e aorth@mjanja.ch -p 10568/83536 10568-93760/COLLECTION@10947-4651.zip
$ for item in 10568-93760/ITEM@10947-465*; do dspace packager -r -f -u -t AIP -e aorth@mjanja.ch $item; done
</code></pre><ul>
<li><input checked="" disabled="" type="checkbox">Create <em>CGIAR System Management Office</em> sub-community: <code>10568/83537</code>
<li><input checked="" disabled="" type="checkbox"> Create <em>CGIAR System Management Office</em> sub-community: <code>10568/83537</code>
<ul>
<li><input checked="" disabled="" type="checkbox">Create <em>CGIAR System Management Office documents</em> collection: <code>10568/83538</code></li>
<li><input checked="" disabled="" type="checkbox"> Create <em>CGIAR System Management Office documents</em> collection: <code>10568/83538</code></li>
<li>Import items to collection individually in replace mode (-r) while explicitly preserving handles and ignoring parents:</li>
</ul>
</li>
@ -209,11 +209,11 @@ $ for item in 10568-93760/ITEM@10947-465*; do dspace packager -r -f -u -t AIP -e
<pre><code>$ for item in 10947-latest/*.zip; do dspace packager -r -u -t AIP -e aorth@mjanja.ch $item; done
</code></pre><h2 id="post-migration">Post Migration</h2>
<ul>
<li><input checked="" disabled="" type="checkbox">Shut down Tomcat and run <code>update-sequences.sql</code> as the system&rsquo;s <code>postgres</code> user</li>
<li><input checked="" disabled="" type="checkbox">Remove ingestion overrides from <code>dspace.cfg</code></li>
<li><input checked="" disabled="" type="checkbox">Reset PostgreSQL <code>max_connections</code> to 183</li>
<li><input checked="" disabled="" type="checkbox">Enable nightly <code>index-discovery</code> cron job</li>
<li><input checked="" disabled="" type="checkbox">Adjust CGSpace&rsquo;s <code>handle-server/config.dct</code> to add the new prefix alongside our existing 10568, ie:</li>
<li><input checked="" disabled="" type="checkbox"> Shut down Tomcat and run <code>update-sequences.sql</code> as the system&rsquo;s <code>postgres</code> user</li>
<li><input checked="" disabled="" type="checkbox"> Remove ingestion overrides from <code>dspace.cfg</code></li>
<li><input checked="" disabled="" type="checkbox"> Reset PostgreSQL <code>max_connections</code> to 183</li>
<li><input checked="" disabled="" type="checkbox"> Enable nightly <code>index-discovery</code> cron job</li>
<li><input checked="" disabled="" type="checkbox"> Adjust CGSpace&rsquo;s <code>handle-server/config.dct</code> to add the new prefix alongside our existing 10568, ie:</li>
</ul>
<pre><code>&quot;server_admins&quot; = (
&quot;300:0.NA/10568&quot;
@ -231,15 +231,15 @@ $ for item in 10568-93760/ITEM@10947-465*; do dspace packager -r -f -u -t AIP -e
)
</code></pre><p>I had been regenerated the <code>sitebndl.zip</code> file on the CGIAR Library server and sent it to the Handle.net admins but they said that there were mismatches between the public and private keys, which I suspect is due to <code>make-handle-config</code> not being very flexible. After discussing our scenario with the Handle.net admins they said we actually don&rsquo;t need to send an updated <code>sitebndl.zip</code> for this type of change, and the above <code>config.dct</code> edits are all that is required. I guess they just did something on their end by setting the authoritative IP address for the 10947 prefix to be the same as ours&hellip;</p>
<ul>
<li><input checked="" disabled="" type="checkbox">Update DNS records:
<li><input checked="" disabled="" type="checkbox"> Update DNS records:
<ul>
<li>CNAME: cgspace.cgiar.org</li>
</ul>
</li>
<li><input checked="" disabled="" type="checkbox">Re-deploy DSpace from freshly built <code>5_x-prod</code> branch</li>
<li><input checked="" disabled="" type="checkbox">Merge <code>cgiar-library</code> branch to <code>master</code> and re-run ansible nginx templates</li>
<li><input checked="" disabled="" type="checkbox">Run system updates and reboot server</li>
<li><input checked="" disabled="" type="checkbox">Switch to Let&rsquo;s Encrypt HTTPS certificates (after DNS is updated and server isn&rsquo;t busy):</li>
<li><input checked="" disabled="" type="checkbox"> Re-deploy DSpace from freshly built <code>5_x-prod</code> branch</li>
<li><input checked="" disabled="" type="checkbox"> Merge <code>cgiar-library</code> branch to <code>master</code> and re-run ansible nginx templates</li>
<li><input checked="" disabled="" type="checkbox"> Run system updates and reboot server</li>
<li><input checked="" disabled="" type="checkbox"> Switch to Let&rsquo;s Encrypt HTTPS certificates (after DNS is updated and server isn&rsquo;t busy):</li>
</ul>
<pre><code>$ sudo systemctl stop nginx
$ /opt/certbot-auto certonly --standalone -d library.cgiar.org