alanorth/cgspace-notes
1
0
Fork 0
mirror of https://github.com/alanorth/cgspace-notes.git synced 2025-01-27 05:49:12 +01:00
Code Issues Projects Releases Wiki Activity

Add notes for 2019-12-17

Browse Source
This commit is contained in:
Alan Orth
2019-12-17 14:49:24 +02:00
parent d83c951532
commit d54e5b69f1
90 changed files with 1420 additions and 1377 deletions
Download Patch File Download Diff File Expand all files Collapse all files

44
docs/2018-02/index.html
View File

@ -27,7 +27,7 @@ We don't need to distinguish between internal and external works, so that ma
Yesterday I figured out how to monitor DSpace sessions using JMX
I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu's munin-plugins-java package and used the stuff I discovered about JMX in 2018-01
"/>
<meta name="generator" content="Hugo 0.60.1" />
<meta name="generator" content="Hugo 0.61.0" />
@ -108,7 +108,7 @@ I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu&#39;s munin-plug
</p>
</header>
<h2 id="20180201">2018-02-01</h2>
<h2 id="2018-02-01">2018-02-01</h2>
<ul>
<li>Peter gave feedback on the <code>dc.rights</code> proof of concept that I had sent him last week</li>
<li>We don't need to distinguish between internal and external works, so that makes it just a simple list</li>
@ -124,7 +124,7 @@ I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu&#39;s munin-plug
v_.value 223
v_jspui.value 1
v_oai.value 0
</code></pre><h2 id="20180203">2018-02-03</h2>
</code></pre><h2 id="2018-02-03">2018-02-03</h2>
<ul>
<li>Bram from Atmire responded about the high load caused by the Solr updater script and said it will be fixed with the updates to DSpace 5.8 compatibility: <a href="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=566">https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=566</a></li>
<li>We will close that ticket for now and wait for the 5.8 stuff: <a href="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=560">https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=560</a></li>
@ -155,7 +155,7 @@ COPY 3723
real 0m23.839s
user 0m27.225s
sys 0m1.905s
</code></pre><h2 id="20180205">2018-02-05</h2>
</code></pre><h2 id="2018-02-05">2018-02-05</h2>
<ul>
<li>Toying with correcting authors with trailing spaces via PostgreSQL:</li>
</ul>
@ -168,7 +168,7 @@ UPDATE 20
</ul>
<pre><code>dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors-2018-02-05.csv with csv;
COPY 55630
</code></pre><h2 id="20180206">2018-02-06</h2>
</code></pre><h2 id="2018-02-06">2018-02-06</h2>
<ul>
<li>UptimeRobot says CGSpace is down this morning around 9:15</li>
<li>I see 308 PostgreSQL connections in <code>pg_stat_activity</code></li>
@ -213,7 +213,7 @@ Tue Feb 6 09:30:32 UTC 2018
<li>I'm not actually sure if the Solr web application uses the database though, so I'll have to check later and remove it if necessary</li>
<li>I deployed the changes on DSpace Test only for now, so I will monitor and make them on CGSpace later this week</li>
</ul>
<h2 id="20180207">2018-02-07</h2>
<h2 id="2018-02-07">2018-02-07</h2>
<ul>
<li>Abenet wrote to ask a question about the ORCiD lookup not working for one CIAT user on CGSpace</li>
<li>I tried on DSpace Test and indeed the lookup just doesn't work!</li>
@ -363,7 +363,7 @@ $ grep 46.229.168 dspace.log.2018-02-07 | grep -o -E 'session_id=[A-Z0-9]{32}' |
<li>I cherry-picked all the commits for DS-3551 but it won't build on our current DSpace 5.5!</li>
<li>I sent a message to the dspace-tech mailing list asking why DSpace thinks these connections are busy when PostgreSQL says they are idle</li>
</ul>
<h2 id="20180210">2018-02-10</h2>
<h2 id="2018-02-10">2018-02-10</h2>
<ul>
<li>I tried to disable ORCID lookups but keep the existing authorities</li>
<li>This item has an ORCID for Ralf Kiese: http://localhost:8080/handle/10568/89897</li>
@ -378,7 +378,7 @@ $ grep 46.229.168 dspace.log.2018-02-07 | grep -o -E 'session_id=[A-Z0-9]{32}' |
</code></pre><ul>
<li>So I don't think we can disable the ORCID lookup function and keep the ORCID badges</li>
</ul>
<h2 id="20180211">2018-02-11</h2>
<h2 id="2018-02-11">2018-02-11</h2>
<ul>
<li>Magdalena from CCAFS emailed to ask why one of their items has such a weird thumbnail: <a href="https://cgspace.cgiar.org/handle/10568/90735">10568/90735</a></li>
</ul>
@ -442,7 +442,7 @@ dspace=# commit;
<li>I don't know how to add ORCID IDs to existing items yet&hellip; some more querying of PostgreSQL for authority values perhaps?</li>
<li>I added the script to the <a href="https://github.com/ilri/DSpace/wiki/Scripts">ILRI DSpace wiki on GitHub</a></li>
</ul>
<h2 id="20180212">2018-02-12</h2>
<h2 id="2018-02-12">2018-02-12</h2>
<ul>
<li>Follow up with Atmire on the <a href="https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=560">DSpace 5.8 Compatibility ticket</a> to ask again if they want me to send them a DSpace 5.8 branch to work on</li>
<li>Abenet asked if there was a way to get the number of submissions she and Bizuwork did</li>
@ -464,7 +464,7 @@ dspace=# commit;
<li>I think I'd probably just attach the block storage volume and mount it on /home/dspace</li>
<li>Ask Peter about <code>dc.rights</code> on DSpace Test again, if he likes it then we should move it to CGSpace soon</li>
</ul>
<h2 id="20180213">2018-02-13</h2>
<h2 id="2018-02-13">2018-02-13</h2>
<ul>
<li>Peter said he was getting a &ldquo;socket closed&rdquo; error on CGSpace</li>
<li>I looked in the dspace.log.2018-02-13 and saw one recent one:</li>
@ -497,7 +497,7 @@ dspace.log.2018-02-13:4
</ul>
<pre><code>Feb 13, 2018 2:05:42 PM org.apache.tomcat.jdbc.pool.ConnectionPool abandon
WARNING: Connection has been abandoned PooledConnection[org.postgresql.jdbc.PgConnection@22e107be]:java.lang.Exception
</code></pre><h2 id="20180214">2018-02-14</h2>
</code></pre><h2 id="2018-02-14">2018-02-14</h2>
<ul>
<li>Skype with Peter and the Addis team to discuss what we need to do for the ORCIDs in the immediate future</li>
<li>We said we'd start with a controlled vocabulary for <code>cg.creator.id</code> on the DSpace Test submission form, where we store the author name and the ORCID in some format like: Alan S. Orth (0000-0002-1735-7458)</li>
@ -552,7 +552,7 @@ UPDATE 1
</code></pre><ul>
<li>Then the cleanup process will continue for awhile and hit another foreign key conflict, and eventually it will complete after you manually resolve them all</li>
</ul>
<h2 id="20180215">2018-02-15</h2>
<h2 id="2018-02-15">2018-02-15</h2>
<ul>
<li>Altmetric seems to be indexing DSpace Test for some reason:
<ul>
@ -596,7 +596,7 @@ UPDATE 1
1512 207.46.13.59
1554 207.46.13.157
2018 104.196.152.243
</code></pre><h2 id="20180217">2018-02-17</h2>
</code></pre><h2 id="2018-02-17">2018-02-17</h2>
<ul>
<li>Peter pointed out that we had an incorrect sponsor in the controlled vocabulary: <code>U.S. Agency for International Development</code> → <code>United States Agency for International Development</code></li>
<li>I made a pull request to fix it ((#354)[<a href="https://github.com/ilri/DSpace/pull/354">https://github.com/ilri/DSpace/pull/354</a>])</li>
@ -604,7 +604,7 @@ UPDATE 1
</ul>
<pre><code>dspace=# update metadatavalue set text_value='United States Agency for International Development' where resource_type_id=2 and metadata_field_id=29 and text_value like '%U.S. Agency for International Development%';
UPDATE 2
</code></pre><h2 id="20180218">2018-02-18</h2>
</code></pre><h2 id="2018-02-18">2018-02-18</h2>
<ul>
<li>ICARDA's Mohamed Salem pointed out that it would be easiest to format the <code>cg.creator.id</code> field like &ldquo;Alan Orth: 0000-0002-1735-7458&rdquo; because no name will have a &ldquo;:&rdquo; so it's easier to split on</li>
<li>I finally figured out a few ways to extract ORCID iDs from metadata using XSLT and display them in the XMLUI:</li>
@ -665,7 +665,7 @@ org.springframework.web.util.NestedServletException: Handler processing failed;
<li>I have no idea what caused this crash</li>
<li>In other news, I adjusted the ORCID badge size on the XMLUI item display and sent it back to Peter for feedback</li>
</ul>
<h2 id="20180219">2018-02-19</h2>
<h2 id="2018-02-19">2018-02-19</h2>
<ul>
<li>Combined list of CGIAR author ORCID iDs is up to 1,500:</li>
</ul>
@ -708,7 +708,7 @@ TypeError: 'NoneType' object is not subscriptable
</code></pre><ul>
<li>According to ORCID that identifier's entire name block is null!</li>
</ul>
<h2 id="20180220">2018-02-20</h2>
<h2 id="2018-02-20">2018-02-20</h2>
<ul>
<li>Send Abenet an email about getting a purchase requisition for a new DSpace Test server on Linode</li>
<li>Discuss some of the issues with null values and poor-quality names in some ORCID identifiers with Abenet and I think we'll now only use ORCID iDs that have been sent to use from partners, not those extracted via keyword searches on orcid.org</li>
@ -756,7 +756,7 @@ TypeError: 'NoneType' object is not subscriptable
<li>Remove CPWF project number and Humidtropics subject from submission form (<a href="https://github.com/alanorth/DSpace/pull/3">#3</a>)</li>
<li>I accidentally merged it into my own repository, oops</li>
</ul>
<h2 id="20180222">2018-02-22</h2>
<h2 id="2018-02-22">2018-02-22</h2>
<ul>
<li>CGSpace was apparently down today around 13:00 server time and I didn't get any emails on my phone, but saw them later on the computer</li>
<li>It looks like Sisay restarted Tomcat because I was offline</li>
@ -803,11 +803,11 @@ TypeError: 'NoneType' object is not subscriptable
</code></pre><ul>
<li>It seems to re-use its user agent but makes tons of useless requests and I wonder if I should add &ldquo;.<em>spider.</em>&rdquo; to the Tomcat Crawler Session Manager valve?</li>
</ul>
<h2 id="20180223">2018-02-23</h2>
<h2 id="2018-02-23">2018-02-23</h2>
<ul>
<li>Atmire got back to us with a quote about their DSpace 5.8 upgrade</li>
</ul>
<h2 id="20180225">2018-02-25</h2>
<h2 id="2018-02-25">2018-02-25</h2>
<ul>
<li>A few days ago Abenet sent me the list of ORCID iDs from CCAFS</li>
<li>We currently have 988 unique identifiers:</li>
@ -872,7 +872,7 @@ Alan S. Orth: 0000-0002-1735-7458
Ibrahim Mohammed: 0000-0001-5199-5528
Nor Azwadi: 0000-0001-9634-1958
./resolve-orcids.py -i orcid-test-values.txt -o /tmp/orcid-names 0.23s user 0.05s system 8% cpu 3.046 total
</code></pre><h2 id="20180226">2018-02-26</h2>
</code></pre><h2 id="2018-02-26">2018-02-26</h2>
<ul>
<li>Peter is having problems with &ldquo;Socket closed&rdquo; on his submissions page again</li>
<li>He says his personal account loads much faster than his CGIAR account, which could be because the CGIAR account has potentially thousands of submissions over the last few years</li>
@ -880,7 +880,7 @@ Nor Azwadi: 0000-0001-9634-1958
<li>I think I should increase the <code>removeAbandonedTimeout</code> from 90 to something like 180 and continue observing</li>
<li>I also reduced the timeout for the API pool back to 60 because those interfaces are only used by bots</li>
</ul>
<h2 id="20180227">2018-02-27</h2>
<h2 id="2018-02-27">2018-02-27</h2>
<ul>
<li>Peter is still having problems with &ldquo;Socket closed&rdquo; on his submissions page</li>
<li>I have disabled <code>removeAbandoned</code> for now because that's the only thing I changed in the last few weeks since he started having issues</li>
@ -923,7 +923,7 @@ COPY 263
<li>It successfully mapped 2600 ORCID identifiers to items in my tests</li>
<li>I will run it on DSpace Test</li>
</ul>
<h2 id="20180228">2018-02-28</h2>
<h2 id="2018-02-28">2018-02-28</h2>
<ul>
<li>CGSpace crashed today, the first HTTP 499 in nginx's access.log was around 09:12</li>
<li>There's nothing interesting going on in nginx's logs around that time:</li>