cgspace-notes/public/post/index.xml

<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Posts on CGSpace Notes</title>
    <link>https://alanorth.github.io/cgspace-notes/post/</link>
    <description>Recent content in Posts on CGSpace Notes</description>
    <generator>Hugo -- gohugo.io</generator>
    <language>en-us</language>
    <lastBuildDate>Thu, 02 Nov 2017 09:37:54 +0200</lastBuildDate>
    
	<atom:link href="https://alanorth.github.io/cgspace-notes/post/index.xml" rel="self" type="application/rss+xml" />
    
    
    <item>
      <title>November, 2017</title>
      <link>https://alanorth.github.io/cgspace-notes/2017-11/</link>
      <pubDate>Thu, 02 Nov 2017 09:37:54 +0200</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/2017-11/</guid>
      <description>&lt;h2 id=&#34;2017-11-01&#34;&gt;2017-11-01&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;The CORE developers responded to say they are looking into their bot not respecting our robots.txt&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&#34;2017-11-02&#34;&gt;2017-11-02&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Today there have been no hits by CORE and no alerts from Linode (coincidence?)&lt;/li&gt;
&lt;/ul&gt;

&lt;pre&gt;&lt;code&gt;# grep -c &amp;quot;CORE&amp;quot; /var/log/nginx/access.log
0
&lt;/code&gt;&lt;/pre&gt;

&lt;ul&gt;
&lt;li&gt;Generate list of authors on CGSpace for Peter to go through and correct:&lt;/li&gt;
&lt;/ul&gt;

&lt;pre&gt;&lt;code&gt;dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = &#39;contributor&#39; and qualifier = &#39;author&#39;) AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors.csv with csv;
COPY 54701
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>October, 2017</title>
      <link>https://alanorth.github.io/cgspace-notes/2017-10/</link>
      <pubDate>Sun, 01 Oct 2017 08:07:54 +0300</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/2017-10/</guid>
      <description>&lt;h2 id=&#34;2017-10-01&#34;&gt;2017-10-01&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Peter emailed to point out that many items in the &lt;a href=&#34;https://cgspace.cgiar.org/handle/10568/2703&#34;&gt;ILRI archive collection&lt;/a&gt; have multiple handles:&lt;/li&gt;
&lt;/ul&gt;

&lt;pre&gt;&lt;code&gt;http://hdl.handle.net/10568/78495||http://hdl.handle.net/10568/79336
&lt;/code&gt;&lt;/pre&gt;

&lt;ul&gt;
&lt;li&gt;There appears to be a pattern but I&amp;rsquo;ll have to look a bit closer and try to clean them up automatically, either in SQL or in OpenRefine&lt;/li&gt;
&lt;li&gt;Add Katherine Lutz to the groups for content submission and edit steps of the CGIAR System collections&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>CGIAR Library Migration</title>
      <link>https://alanorth.github.io/cgspace-notes/cgiar-library-migration/</link>
      <pubDate>Mon, 18 Sep 2017 16:38:35 +0300</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/cgiar-library-migration/</guid>
      <description>&lt;p&gt;Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called &lt;em&gt;CGIAR System Organization&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>September, 2017</title>
      <link>https://alanorth.github.io/cgspace-notes/2017-09/</link>
      <pubDate>Thu, 07 Sep 2017 16:54:52 +0700</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/2017-09/</guid>
      <description>&lt;h2 id=&#34;2017-09-06&#34;&gt;2017-09-06&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Linode sent an alert that CGSpace (linode18) was using 261% CPU for the past two hours&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&#34;2017-09-07&#34;&gt;2017-09-07&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Ask Sisay to clean up the WLE approvers a bit, as Marianne&amp;rsquo;s user account is both in the approvers step as well as the group&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>August, 2017</title>
      <link>https://alanorth.github.io/cgspace-notes/2017-08/</link>
      <pubDate>Tue, 01 Aug 2017 11:51:52 +0300</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/2017-08/</guid>
      <description>&lt;h2 id=&#34;2017-08-01&#34;&gt;2017-08-01&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Linode sent an alert that CGSpace (linode18) was using 350% CPU for the past two hours&lt;/li&gt;
&lt;li&gt;I looked in the Activity pane of the Admin Control Panel and it seems that Google, Baidu, Yahoo, and Bing are all crawling with massive numbers of bots concurrently (~100 total, mostly Baidu and Google)&lt;/li&gt;
&lt;li&gt;The good thing is that, according to &lt;code&gt;dspace.log.2017-08-01&lt;/code&gt;, they are all using the same Tomcat session&lt;/li&gt;
&lt;li&gt;This means our Tomcat Crawler Session Valve is working&lt;/li&gt;
&lt;li&gt;But many of the bots are browsing dynamic URLs like:

&lt;ul&gt;
&lt;li&gt;/handle/10568/3353/discover&lt;/li&gt;
&lt;li&gt;/handle/10568/16510/browse&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;li&gt;The &lt;code&gt;robots.txt&lt;/code&gt; only blocks the top-level &lt;code&gt;/discover&lt;/code&gt; and &lt;code&gt;/browse&lt;/code&gt; URLs&amp;hellip; we will need to find a way to forbid them from accessing these!&lt;/li&gt;
&lt;li&gt;Relevant issue from DSpace Jira (semi resolved in DSpace 6.0): &lt;a href=&#34;https://jira.duraspace.org/browse/DS-2962&#34;&gt;https://jira.duraspace.org/browse/DS-2962&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;It turns out that we&amp;rsquo;re already adding the &lt;code&gt;X-Robots-Tag &amp;quot;none&amp;quot;&lt;/code&gt; HTTP header, but this only forbids the search engine from &lt;em&gt;indexing&lt;/em&gt; the page, not crawling it!&lt;/li&gt;
&lt;li&gt;Also, the bot has to successfully browse the page first so it can receive the HTTP header&amp;hellip;&lt;/li&gt;
&lt;li&gt;We might actually have to &lt;em&gt;block&lt;/em&gt; these requests with HTTP 403 depending on the user agent&lt;/li&gt;
&lt;li&gt;Abenet pointed out that the CGIAR Library Historical Archive collection I sent July 20th only had ~100 entries, instead of 2415&lt;/li&gt;
&lt;li&gt;This was due to newline characters in the &lt;code&gt;dc.description.abstract&lt;/code&gt; column, which caused OpenRefine to choke when exporting the CSV&lt;/li&gt;
&lt;li&gt;I exported a new CSV from the collection on DSpace Test and then manually removed the characters in vim using &lt;code&gt;g/^$/d&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Then I cleaned up the author authorities and HTML characters in OpenRefine and sent the file back to Abenet&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>July, 2017</title>
      <link>https://alanorth.github.io/cgspace-notes/2017-07/</link>
      <pubDate>Sat, 01 Jul 2017 18:03:52 +0300</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/2017-07/</guid>
      <description>&lt;h2 id=&#34;2017-07-01&#34;&gt;2017-07-01&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Run system updates and reboot DSpace Test&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&#34;2017-07-04&#34;&gt;2017-07-04&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Merge changes for WLE Phase II theme rename (&lt;a href=&#34;https://github.com/ilri/DSpace/pull/329&#34;&gt;#329&lt;/a&gt;)&lt;/li&gt;
&lt;li&gt;Looking at extracting the metadata registries from ICARDA&amp;rsquo;s MEL DSpace database so we can compare fields with CGSpace&lt;/li&gt;
&lt;li&gt;We can use PostgreSQL&amp;rsquo;s extended output format (&lt;code&gt;-x&lt;/code&gt;) plus &lt;code&gt;sed&lt;/code&gt; to format the output into quasi XML:&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>June, 2017</title>
      <link>https://alanorth.github.io/cgspace-notes/2017-06/</link>
      <pubDate>Thu, 01 Jun 2017 10:14:52 +0300</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/2017-06/</guid>
      <description>2017-06-01  After discussion with WLE and CGSpace content people, we decided to just add one metadata field for the WLE Research Themes The cg.identifier.wletheme field will be used for both Phase I and Phase II Research Themes Then we&amp;rsquo;ll create a new sub-community for Phase II and create collections for the research themes there The current &amp;ldquo;Research Themes&amp;rdquo; community will be renamed to &amp;ldquo;WLE Phase I Research Themes&amp;rdquo; Tagged all items in the current Phase I collections with their appropriate themes Create pull request to add Phase II research themes to the submission form: #328 Add cg.</description>
    </item>
    
    <item>
      <title>May, 2017</title>
      <link>https://alanorth.github.io/cgspace-notes/2017-05/</link>
      <pubDate>Mon, 01 May 2017 16:21:52 +0200</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/2017-05/</guid>
      <description>2017-05-01  ICARDA apparently started working on CG Core on their MEL repository They have done a few cg.* fields, but not very consistent and even copy some of CGSpace items:  https://mel.cgiar.org/xmlui/handle/20.500.11766/6911?show=full https://cgspace.cgiar.org/handle/10568/73683   2017-05-02  Atmire got back about the Workflow Statistics issue, and apparently it&amp;rsquo;s a bug in the CUA module so they will send us a pull request  2017-05-04  Sync DSpace Test with database and assetstore from CGSpace Re-deploy DSpace Test with Atmire&amp;rsquo;s CUA patch for workflow statistics, run system updates, and restart the server Now I can see the workflow statistics and am able to select users, but everything returns 0 items Megan says there are still some mapped items are not appearing since last week, so I forced a full index-discovery -b Need to remember to check if the collection has more items (currently 39 on CGSpace, but 118 on the freshly reindexed DSPace Test) tomorrow: https://cgspace.</description>
    </item>
    
    <item>
      <title>April, 2017</title>
      <link>https://alanorth.github.io/cgspace-notes/2017-04/</link>
      <pubDate>Sun, 02 Apr 2017 17:08:52 +0200</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/2017-04/</guid>
      <description>&lt;h2 id=&#34;2017-04-02&#34;&gt;2017-04-02&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Merge one change to CCAFS flagships that I had forgotten to remove last month (&amp;ldquo;MANAGING CLIMATE RISK&amp;rdquo;): &lt;a href=&#34;https://github.com/ilri/DSpace/pull/317&#34;&gt;https://github.com/ilri/DSpace/pull/317&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Quick proof-of-concept hack to add &lt;code&gt;dc.rights&lt;/code&gt; to the input form, including some inline instructions/hints:&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;img src=&#34;https://alanorth.github.io/cgspace-notes/cgspace-notes/2017/04/dc-rights.png&#34; alt=&#34;dc.rights in the submission form&#34; /&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Remove redundant/duplicate text in the DSpace submission license&lt;/li&gt;
&lt;li&gt;Testing the CMYK patch on a collection with 650 items:&lt;/li&gt;
&lt;/ul&gt;

&lt;pre&gt;&lt;code&gt;$ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p &amp;quot;ImageMagick PDF Thumbnail&amp;quot; -v &amp;gt;&amp;amp; /tmp/filter-media-cmyk.txt
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>March, 2017</title>
      <link>https://alanorth.github.io/cgspace-notes/2017-03/</link>
      <pubDate>Wed, 01 Mar 2017 17:08:52 +0200</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/2017-03/</guid>
      <description>&lt;h2 id=&#34;2017-03-01&#34;&gt;2017-03-01&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Run the 279 CIAT author corrections on CGSpace&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&#34;2017-03-02&#34;&gt;2017-03-02&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Skype with Michael and Peter, discussing moving the CGIAR Library to CGSpace&lt;/li&gt;
&lt;li&gt;CGIAR people possibly open to moving content, redirecting library.cgiar.org to CGSpace and letting CGSpace resolve their handles&lt;/li&gt;
&lt;li&gt;They might come in at the top level in one &amp;ldquo;CGIAR System&amp;rdquo; community, or with several communities&lt;/li&gt;
&lt;li&gt;I need to spend a bit of time looking at the multiple handle support in DSpace and see if new content can be minted in both handles, or just one?&lt;/li&gt;
&lt;li&gt;Need to send Peter and Michael some notes about this in a few days&lt;/li&gt;
&lt;li&gt;Also, need to consider talking to Atmire about hiring them to bring ORCiD metadata to REST / OAI&lt;/li&gt;
&lt;li&gt;Filed an issue on DSpace issue tracker for the &lt;code&gt;filter-media&lt;/code&gt; bug that causes it to process JPGs even when limiting to the PDF thumbnail plugin: &lt;a href=&#34;https://jira.duraspace.org/browse/DS-3516&#34;&gt;DS-3516&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Discovered that the ImageMagic &lt;code&gt;filter-media&lt;/code&gt; plugin creates JPG thumbnails with the CMYK colorspace when the source PDF is using CMYK&lt;/li&gt;
&lt;li&gt;Interestingly, it seems DSpace 4.x&amp;rsquo;s thumbnails were sRGB, but forcing regeneration using DSpace 5.x&amp;rsquo;s ImageMagick plugin creates CMYK JPGs if the source PDF was CMYK (see &lt;a href=&#34;https://cgspace.cgiar.org/handle/10568/51999&#34;&gt;&lt;sup&gt;10568&lt;/sup&gt;&amp;frasl;&lt;sub&gt;51999&lt;/sub&gt;&lt;/a&gt;):&lt;/li&gt;
&lt;/ul&gt;

&lt;pre&gt;&lt;code&gt;$ identify ~/Desktop/alc_contrastes_desafios.jpg
/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>February, 2017</title>
      <link>https://alanorth.github.io/cgspace-notes/2017-02/</link>
      <pubDate>Tue, 07 Feb 2017 07:04:52 -0800</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/2017-02/</guid>
      <description>&lt;h2 id=&#34;2017-02-07&#34;&gt;2017-02-07&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;An item was mapped twice erroneously again, so I had to remove one of the mappings manually:&lt;/li&gt;
&lt;/ul&gt;

&lt;pre&gt;&lt;code&gt;dspace=# select * from collection2item where item_id = &#39;80278&#39;;
  id   | collection_id | item_id
-------+---------------+---------
 92551 |           313 |   80278
 92550 |           313 |   80278
 90774 |          1051 |   80278
(3 rows)
dspace=# delete from collection2item where id = 92551 and item_id = 80278;
DELETE 1
&lt;/code&gt;&lt;/pre&gt;

&lt;ul&gt;
&lt;li&gt;Create issue on GitHub to track the addition of CCAFS Phase II project tags (&lt;a href=&#34;https://github.com/ilri/DSpace/issues/301&#34;&gt;#301&lt;/a&gt;)&lt;/li&gt;
&lt;li&gt;Looks like we&amp;rsquo;ll be using &lt;code&gt;cg.identifier.ccafsprojectpii&lt;/code&gt; as the field name&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>January, 2017</title>
      <link>https://alanorth.github.io/cgspace-notes/2017-01/</link>
      <pubDate>Mon, 02 Jan 2017 10:43:00 +0300</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/2017-01/</guid>
      <description>&lt;h2 id=&#34;2017-01-02&#34;&gt;2017-01-02&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;I checked to see if the Solr sharding task that is supposed to run on January 1st had run and saw there was an error&lt;/li&gt;
&lt;li&gt;I tested on DSpace Test as well and it doesn&amp;rsquo;t work there either&lt;/li&gt;
&lt;li&gt;I asked on the dspace-tech mailing list because it seems to be broken, and actually now I&amp;rsquo;m not sure if we&amp;rsquo;ve ever had the sharding task run successfully over all these years&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>December, 2016</title>
      <link>https://alanorth.github.io/cgspace-notes/2016-12/</link>
      <pubDate>Fri, 02 Dec 2016 10:43:00 +0300</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/2016-12/</guid>
      <description>&lt;h2 id=&#34;2016-12-02&#34;&gt;2016-12-02&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;CGSpace was down for five hours in the morning while I was sleeping&lt;/li&gt;
&lt;li&gt;While looking in the logs for errors, I see tons of warnings about Atmire MQM:&lt;/li&gt;
&lt;/ul&gt;

&lt;pre&gt;&lt;code&gt;2016-12-02 03:00:32,352 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=CREATE, SubjectType=BUNDLE, SubjectID=70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632305, dispatcher=1544803905, detail=[null], transactionID=&amp;quot;TX157907838689377964651674089851855413607&amp;quot;)
2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY_METADATA, SubjectType=BUNDLE, SubjectID =70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632309, dispatcher=1544803905, detail=&amp;quot;dc.title&amp;quot;, transactionID=&amp;quot;TX157907838689377964651674089851855413607&amp;quot;)
2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=ITEM, SubjectID=80044, Object Type=BUNDLE, ObjectID=70316, TimeStamp=1480647632311, dispatcher=1544803905, detail=&amp;quot;THUMBNAIL&amp;quot;, transactionID=&amp;quot;TX157907838689377964651674089851855413607&amp;quot;)
2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=BUNDLE, SubjectID=70316, Obje ctType=BITSTREAM, ObjectID=86715, TimeStamp=1480647632318, dispatcher=1544803905, detail=&amp;quot;-1&amp;quot;, transactionID=&amp;quot;TX157907838689377964651674089851855413607&amp;quot;)
2016-12-02 03:00:32,353 WARN  com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY, SubjectType=ITEM, SubjectID=80044, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632351, dispatcher=1544803905, detail=[null], transactionID=&amp;quot;TX157907838689377964651674089851855413607&amp;quot;)
&lt;/code&gt;&lt;/pre&gt;

&lt;ul&gt;
&lt;li&gt;I see thousands of them in the logs for the last few months, so it&amp;rsquo;s not related to the DSpace 5.5 upgrade&lt;/li&gt;
&lt;li&gt;I&amp;rsquo;ve raised a ticket with Atmire to ask&lt;/li&gt;
&lt;li&gt;Another worrying error from dspace.log is:&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>November, 2016</title>
      <link>https://alanorth.github.io/cgspace-notes/2016-11/</link>
      <pubDate>Tue, 01 Nov 2016 09:21:00 +0300</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/2016-11/</guid>
      <description>&lt;h2 id=&#34;2016-11-01&#34;&gt;2016-11-01&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Add &lt;code&gt;dc.type&lt;/code&gt; to the output options for Atmire&amp;rsquo;s Listings and Reports module (&lt;a href=&#34;https://github.com/ilri/DSpace/pull/286&#34;&gt;#286&lt;/a&gt;)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;img src=&#34;https://alanorth.github.io/cgspace-notes/cgspace-notes/2016/11/listings-and-reports.png&#34; alt=&#34;Listings and Reports with output type&#34; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>October, 2016</title>
      <link>https://alanorth.github.io/cgspace-notes/2016-10/</link>
      <pubDate>Mon, 03 Oct 2016 15:53:00 +0300</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/2016-10/</guid>
      <description>&lt;h2 id=&#34;2016-10-03&#34;&gt;2016-10-03&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Testing adding &lt;a href=&#34;https://wiki.duraspace.org/display/DSDOC5x/ORCID+Integration#ORCIDIntegration-EditingexistingitemsusingBatchCSVEditing&#34;&gt;ORCIDs to a CSV&lt;/a&gt; file for a single item to see if the author orders get messed up&lt;/li&gt;
&lt;li&gt;Need to test the following scenarios to see how author order is affected:

&lt;ul&gt;
&lt;li&gt;ORCIDs only&lt;/li&gt;
&lt;li&gt;ORCIDs plus normal authors&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;li&gt;I exported a random item&amp;rsquo;s metadata as CSV, deleted &lt;em&gt;all columns&lt;/em&gt; except id and collection, and made a new coloum called &lt;code&gt;ORCID:dc.contributor.author&lt;/code&gt; with the following random ORCIDs from the ORCID registry:&lt;/li&gt;
&lt;/ul&gt;

&lt;pre&gt;&lt;code&gt;0000-0002-6115-0956||0000-0002-3812-8793||0000-0001-7462-405X
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>September, 2016</title>
      <link>https://alanorth.github.io/cgspace-notes/2016-09/</link>
      <pubDate>Thu, 01 Sep 2016 15:53:00 +0300</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/2016-09/</guid>
      <description>&lt;h2 id=&#34;2016-09-01&#34;&gt;2016-09-01&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Discuss helping CCAFS with some batch tagging of ORCID IDs for their authors&lt;/li&gt;
&lt;li&gt;Discuss how the migration of CGIAR&amp;rsquo;s Active Directory to a flat structure will break our LDAP groups in DSpace&lt;/li&gt;
&lt;li&gt;We had been using &lt;code&gt;DC=ILRI&lt;/code&gt; to determine whether a user was ILRI or not&lt;/li&gt;
&lt;li&gt;It looks like we might be able to use OUs now, instead of DCs:&lt;/li&gt;
&lt;/ul&gt;

&lt;pre&gt;&lt;code&gt;$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b &amp;quot;dc=cgiarad,dc=org&amp;quot; -D &amp;quot;admigration1@cgiarad.org&amp;quot; -W &amp;quot;(sAMAccountName=admigration1)&amp;quot;
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>August, 2016</title>
      <link>https://alanorth.github.io/cgspace-notes/2016-08/</link>
      <pubDate>Mon, 01 Aug 2016 15:53:00 +0300</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/2016-08/</guid>
      <description>&lt;h2 id=&#34;2016-08-01&#34;&gt;2016-08-01&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Add updated distribution license from Sisay (&lt;a href=&#34;https://github.com/ilri/DSpace/issues/259&#34;&gt;#259&lt;/a&gt;)&lt;/li&gt;
&lt;li&gt;Play with upgrading Mirage 2 dependencies in &lt;code&gt;bower.json&lt;/code&gt; because most are several versions of out date&lt;/li&gt;
&lt;li&gt;Bootstrap is at 3.3.0 but upstream is at 3.3.7, and upgrading to anything beyond 3.3.1 breaks glyphicons and probably more&lt;/li&gt;
&lt;li&gt;bower stuff is a dead end, waste of time, too many issues&lt;/li&gt;
&lt;li&gt;Anything after Bootstrap 3.3.1 makes glyphicons disappear (HTTP 404 trying to access from incorrect path of &lt;code&gt;fonts&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;Start working on DSpace 5.1 → 5.5 port:&lt;/li&gt;
&lt;/ul&gt;

&lt;pre&gt;&lt;code&gt;$ git checkout -b 55new 5_x-prod
$ git reset --hard ilri/5_x-prod
$ git rebase -i dspace-5.5
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>July, 2016</title>
      <link>https://alanorth.github.io/cgspace-notes/2016-07/</link>
      <pubDate>Fri, 01 Jul 2016 10:53:00 +0300</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/2016-07/</guid>
      <description>&lt;h2 id=&#34;2016-07-01&#34;&gt;2016-07-01&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Add &lt;code&gt;dc.description.sponsorship&lt;/code&gt; to Discovery sidebar facets and make investors clickable in item view (&lt;a href=&#34;https://github.com/ilri/DSpace/issues/232&#34;&gt;#232&lt;/a&gt;)&lt;/li&gt;
&lt;li&gt;I think this query should find and replace all authors that have &amp;ldquo;,&amp;rdquo; at the end of their names:&lt;/li&gt;
&lt;/ul&gt;

&lt;pre&gt;&lt;code&gt;dspacetest=# update metadatavalue set text_value = regexp_replace(text_value, &#39;(^.+?),$&#39;, &#39;\1&#39;) where metadata_field_id=3 and resource_type_id=2 and text_value ~ &#39;^.+?,$&#39;;
UPDATE 95
dspacetest=# select text_value from  metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value ~ &#39;^.+?,$&#39;;
 text_value
------------
(0 rows)
&lt;/code&gt;&lt;/pre&gt;

&lt;ul&gt;
&lt;li&gt;In this case the select query was showing 95 results before the update&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>June, 2016</title>
      <link>https://alanorth.github.io/cgspace-notes/2016-06/</link>
      <pubDate>Wed, 01 Jun 2016 10:53:00 +0300</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/2016-06/</guid>
      <description>&lt;h2 id=&#34;2016-06-01&#34;&gt;2016-06-01&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Experimenting with IFPRI OAI (we want to harvest their publications)&lt;/li&gt;
&lt;li&gt;After reading the &lt;a href=&#34;https://www.oclc.org/support/services/contentdm/help/server-admin-help/oai-support.en.html&#34;&gt;ContentDM documentation&lt;/a&gt; I found IFPRI&amp;rsquo;s OAI endpoint: &lt;a href=&#34;http://ebrary.ifpri.org/oai/oai.php&#34;&gt;http://ebrary.ifpri.org/oai/oai.php&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;After reading the &lt;a href=&#34;https://www.openarchives.org/OAI/openarchivesprotocol.html&#34;&gt;OAI documentation&lt;/a&gt; and testing with an &lt;a href=&#34;http://validator.oaipmh.com/&#34;&gt;OAI validator&lt;/a&gt; I found out how to get their publications&lt;/li&gt;
&lt;li&gt;This is their publications set: &lt;a href=&#34;http://ebrary.ifpri.org/oai/oai.php?verb=ListRecords&amp;amp;from=2016-01-01&amp;amp;set=p15738coll2&amp;amp;metadataPrefix=oai_dc&#34;&gt;http://ebrary.ifpri.org/oai/oai.php?verb=ListRecords&amp;amp;from=2016-01-01&amp;amp;set=p15738coll2&amp;amp;metadataPrefix=oai_dc&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;You can see the others by using the OAI &lt;code&gt;ListSets&lt;/code&gt; verb: &lt;a href=&#34;http://ebrary.ifpri.org/oai/oai.php?verb=ListSets&#34;&gt;http://ebrary.ifpri.org/oai/oai.php?verb=ListSets&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Working on second phase of metadata migration, looks like this will work for moving CPWF-specific data in &lt;code&gt;dc.identifier.fund&lt;/code&gt; to &lt;code&gt;cg.identifier.cpwfproject&lt;/code&gt; and then the rest to &lt;code&gt;dc.description.sponsorship&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>May, 2016</title>
      <link>https://alanorth.github.io/cgspace-notes/2016-05/</link>
      <pubDate>Sun, 01 May 2016 23:06:00 +0300</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/2016-05/</guid>
      <description>&lt;h2 id=&#34;2016-05-01&#34;&gt;2016-05-01&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Since yesterday there have been 10,000 REST errors and the site has been unstable again&lt;/li&gt;
&lt;li&gt;I have blocked access to the API now&lt;/li&gt;
&lt;li&gt;There are 3,000 IPs accessing the REST API in a 24-hour period!&lt;/li&gt;
&lt;/ul&gt;

&lt;pre&gt;&lt;code&gt;# awk &#39;{print $1}&#39; /var/log/nginx/rest.log  | uniq | wc -l
3168
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>April, 2016</title>
      <link>https://alanorth.github.io/cgspace-notes/2016-04/</link>
      <pubDate>Mon, 04 Apr 2016 11:06:00 +0300</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/2016-04/</guid>
      <description>&lt;h2 id=&#34;2016-04-04&#34;&gt;2016-04-04&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Looking at log file use on CGSpace and notice that we need to work on our cron setup a bit&lt;/li&gt;
&lt;li&gt;We are backing up all logs in the log folder, including useless stuff like solr, cocoon, handle-plugin, etc&lt;/li&gt;
&lt;li&gt;After running DSpace for over five years I&amp;rsquo;ve never needed to look in any other log file than dspace.log, leave alone one from last year!&lt;/li&gt;
&lt;li&gt;This will save us a few gigs of backup space we&amp;rsquo;re paying for on S3&lt;/li&gt;
&lt;li&gt;Also, I noticed the &lt;code&gt;checker&lt;/code&gt; log has some errors we should pay attention to:&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>March, 2016</title>
      <link>https://alanorth.github.io/cgspace-notes/2016-03/</link>
      <pubDate>Wed, 02 Mar 2016 16:50:00 +0300</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/2016-03/</guid>
      <description>&lt;h2 id=&#34;2016-03-02&#34;&gt;2016-03-02&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Looking at issues with author authorities on CGSpace&lt;/li&gt;
&lt;li&gt;For some reason we still have the &lt;code&gt;index-lucene-update&lt;/code&gt; cron job active on CGSpace, but I&amp;rsquo;m pretty sure we don&amp;rsquo;t need it as of the latest few versions of Atmire&amp;rsquo;s Listings and Reports module&lt;/li&gt;
&lt;li&gt;Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Java JDK 1.7 to match environment on CGSpace server&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>February, 2016</title>
      <link>https://alanorth.github.io/cgspace-notes/2016-02/</link>
      <pubDate>Fri, 05 Feb 2016 13:18:00 +0300</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/2016-02/</guid>
      <description>&lt;h2 id=&#34;2016-02-05&#34;&gt;2016-02-05&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Looking at some DAGRIS data for Abenet Yabowork&lt;/li&gt;
&lt;li&gt;Lots of issues with spaces, newlines, etc causing the import to fail&lt;/li&gt;
&lt;li&gt;I noticed we have a very &lt;em&gt;interesting&lt;/em&gt; list of countries on CGSpace:&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;img src=&#34;https://alanorth.github.io/cgspace-notes/cgspace-notes/2016/02/cgspace-countries.png&#34; alt=&#34;CGSpace country list&#34; /&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Not only are there 49,000 countries, we have some blanks (25)&amp;hellip;&lt;/li&gt;
&lt;li&gt;Also, lots of things like &amp;ldquo;COTE D`LVOIRE&amp;rdquo; and &amp;ldquo;COTE D IVOIRE&amp;rdquo;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>January, 2016</title>
      <link>https://alanorth.github.io/cgspace-notes/2016-01/</link>
      <pubDate>Wed, 13 Jan 2016 13:18:00 +0300</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/2016-01/</guid>
      <description>&lt;h2 id=&#34;2016-01-13&#34;&gt;2016-01-13&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Move ILRI collection &lt;code&gt;10568/12503&lt;/code&gt; from &lt;code&gt;10568/27869&lt;/code&gt; to &lt;code&gt;10568/27629&lt;/code&gt; using the &lt;a href=&#34;https://gist.github.com/alanorth/392c4660e8b022d99dfa&#34;&gt;move_collections.sh&lt;/a&gt; script I wrote last year.&lt;/li&gt;
&lt;li&gt;I realized it is only necessary to clear the Cocoon cache after moving collections—rather than reindexing—as no metadata has changed, and therefore no search or browse indexes need to be updated.&lt;/li&gt;
&lt;li&gt;Update GitHub wiki for documentation of &lt;a href=&#34;https://github.com/ilri/DSpace/wiki/Maintenance-Tasks&#34;&gt;maintenance tasks&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>December, 2015</title>
      <link>https://alanorth.github.io/cgspace-notes/2015-12/</link>
      <pubDate>Wed, 02 Dec 2015 13:18:00 +0300</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/2015-12/</guid>
      <description>&lt;h2 id=&#34;2015-12-02&#34;&gt;2015-12-02&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Replace &lt;code&gt;lzop&lt;/code&gt; with &lt;code&gt;xz&lt;/code&gt; in log compression cron jobs on DSpace Test—it uses less space:&lt;/li&gt;
&lt;/ul&gt;

&lt;pre&gt;&lt;code&gt;# cd /home/dspacetest.cgiar.org/log
# ls -lh dspace.log.2015-11-18*
-rw-rw-r-- 1 tomcat7 tomcat7 2.0M Nov 18 23:59 dspace.log.2015-11-18
-rw-rw-r-- 1 tomcat7 tomcat7 387K Nov 18 23:59 dspace.log.2015-11-18.lzo
-rw-rw-r-- 1 tomcat7 tomcat7 169K Nov 18 23:59 dspace.log.2015-11-18.xz
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;&lt;/p&gt;</description>
    </item>
    
    <item>
      <title>November, 2015</title>
      <link>https://alanorth.github.io/cgspace-notes/2015-11/</link>
      <pubDate>Mon, 23 Nov 2015 17:00:57 +0300</pubDate>
      
      <guid>https://alanorth.github.io/cgspace-notes/2015-11/</guid>
      <description>&lt;h2 id=&#34;2015-11-22&#34;&gt;2015-11-22&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;CGSpace went down&lt;/li&gt;
&lt;li&gt;Looks like DSpace exhausted its PostgreSQL connection pool&lt;/li&gt;
&lt;li&gt;Last week I had increased the limit from 30 to 60, which seemed to help, but now there are many more idle connections:&lt;/li&gt;
&lt;/ul&gt;

&lt;pre&gt;&lt;code&gt;$ psql -c &#39;SELECT * from pg_stat_activity;&#39; | grep idle | grep -c cgspace
78
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;&lt;/p&gt;</description>
    </item>
    
  </channel>
</rss>