Update notes for 2017-01-24

content/post/2017-01.md

@@ -194,8 +194,7 @@ value + "__description:" + cells["dc.type"].value
 - Test importing of the new CIAT records (actually there are 232, not 234):
 
 ```
-$ JAVA_OPTS="-Xmx512m -Dfile.encoding=UTF-8" /home/dspacetest.cgiar.org/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568
+$ JAVA_OPTS="-Xmx512m -Dfile.encoding=UTF-8" /home/dspacetest.cgiar.org/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568/79042 --source /home/aorth/CIAT_234/SimpleArchiveFormat/ --mapfile=/tmp/ciat.map &> /tmp/ciat.log
-/79042 --source /home/aorth/CIAT_234/SimpleArchiveFormat/ --mapfile=/tmp/ciat.map &> /tmp/ciat.log
 ```
 
 - Many of the PDFs are 20, 30, 40, 50+ MB, which makes a total of 4GB
@@ -246,3 +245,12 @@ $ for community in 10568/171 10568/27868 10568/231 10568/27869 10568/150 10568/2
 ```
 $ ./fix-metadata-values.py -i /tmp/fix-49-journal-titles.csv -f dc.source -t correct -m 55 -d dspace -u dspace -p 'password'
 ```
+
+- Create a new list of the top 500 journal titles from the database:
+
+```
+dspace-# \copy (select distinct text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=55 group by text_value order by count desc limit 500) to /tmp/journal-titles.csv with csv;
+```
+
+- Then sort them in OpenRefine and create a controlled vocabulary by manually adding the XML markup, pull request ([#298](https://github.com/ilri/DSpace/pull/298))
+- This would be the last issue remaining to close the meta issue about switching to controlled vocabularies ([#69](https://github.com/ilri/DSpace/pull/69))

public/2017-01/index.html

@@ -59,7 +59,7 @@ I asked on the dspace-tech mailing list because it seems to be broken, and actua
   
   "headline": "January, 2017",
   "url": "https://alanorth.github.io/cgspace-notes/2017-01/",
-  "wordCount": "1327",
+  "wordCount": "1400",
   
   
   "datePublished": "2017-01-02T10:43:00+03:00",
@@ -299,8 +299,7 @@ UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 15:
 <li>Now get the top 500 journal titles:</li>
 </ul>
 
-<pre><code>dspace-# \copy (select distinct text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=55 group by text_value order by cou
+<pre><code>dspace-# \copy (select distinct text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=55 group by text_value order by count desc limit 500) to /tmp/journal-titles.csv with csv;
-nt desc limit 500) to /tmp/journal-titles.csv with csv;
 </code></pre>
 
 <ul>
@@ -351,8 +350,7 @@ delete from collection2item where id = '91082';
 <li>Test importing of the new CIAT records (actually there are 232, not 234):</li>
 </ul>
 
-<pre><code>$ JAVA_OPTS=&quot;-Xmx512m -Dfile.encoding=UTF-8&quot; /home/dspacetest.cgiar.org/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568
+<pre><code>$ JAVA_OPTS=&quot;-Xmx512m -Dfile.encoding=UTF-8&quot; /home/dspacetest.cgiar.org/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568/79042 --source /home/aorth/CIAT_234/SimpleArchiveFormat/ --mapfile=/tmp/ciat.map &amp;&gt; /tmp/ciat.log
-/79042 --source /home/aorth/CIAT_234/SimpleArchiveFormat/ --mapfile=/tmp/ciat.map &amp;&gt; /tmp/ciat.log
 </code></pre>
 
 <ul>
@@ -413,6 +411,18 @@ $ gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/ebook -dNOPAUSE -
 <pre><code>$ ./fix-metadata-values.py -i /tmp/fix-49-journal-titles.csv -f dc.source -t correct -m 55 -d dspace -u dspace -p 'password'
 </code></pre>
 
+<ul>
+<li>Create a new list of the top 500 journal titles from the database:</li>
+</ul>
+
+<pre><code>dspace-# \copy (select distinct text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=55 group by text_value order by count desc limit 500) to /tmp/journal-titles.csv with csv;
+</code></pre>
+
+<ul>
+<li>Then sort them in OpenRefine and create a controlled vocabulary by manually adding the XML markup, pull request (<a href="https://github.com/ilri/DSpace/pull/298">#298</a>)</li>
+<li>This would be the last issue remaining to close the meta issue about switching to controlled vocabularies (<a href="https://github.com/ilri/DSpace/pull/69">#69</a>)</li>
+</ul>
+
   
 
   

public/index.xml

@@ -180,8 +180,7 @@ UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4&
 <li>Now get the top 500 journal titles:</li>
 </ul>
 
-<pre><code>dspace-# \copy (select distinct text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=55 group by text_value order by cou
+<pre><code>dspace-# \copy (select distinct text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=55 group by text_value order by count desc limit 500) to /tmp/journal-titles.csv with csv;
-nt desc limit 500) to /tmp/journal-titles.csv with csv;
 </code></pre>
 
 <ul>
@@ -232,8 +231,7 @@ delete from collection2item where id = '91082';
 <li>Test importing of the new CIAT records (actually there are 232, not 234):</li>
 </ul>
 
-<pre><code>$ JAVA_OPTS="-Xmx512m -Dfile.encoding=UTF-8" /home/dspacetest.cgiar.org/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568
+<pre><code>$ JAVA_OPTS="-Xmx512m -Dfile.encoding=UTF-8" /home/dspacetest.cgiar.org/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568/79042 --source /home/aorth/CIAT_234/SimpleArchiveFormat/ --mapfile=/tmp/ciat.map &> /tmp/ciat.log
-/79042 --source /home/aorth/CIAT_234/SimpleArchiveFormat/ --mapfile=/tmp/ciat.map &> /tmp/ciat.log
 </code></pre>
 
 <ul>
@@ -292,7 +290,19 @@ $ gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/ebook -dNOPAUSE -
 </ul>
 
 <pre><code>$ ./fix-metadata-values.py -i /tmp/fix-49-journal-titles.csv -f dc.source -t correct -m 55 -d dspace -u dspace -p 'password'
-&lt;/code&gt;&lt;/pre&gt;</description>
+&lt;/code&gt;&lt;/pre&gt;
+
+&lt;ul&gt;
+&lt;li&gt;Create a new list of the top 500 journal titles from the database:&lt;/li&gt;
+&lt;/ul&gt;
+
+&lt;pre&gt;&lt;code&gt;dspace-# \copy (select distinct text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=55 group by text_value order by count desc limit 500) to /tmp/journal-titles.csv with csv;
+&lt;/code&gt;&lt;/pre&gt;
+
+&lt;ul&gt;
+&lt;li&gt;Then sort them in OpenRefine and create a controlled vocabulary by manually adding the XML markup, pull request (&lt;a href=&#34;https://github.com/ilri/DSpace/pull/298&#34;&gt;#298&lt;/a&gt;)&lt;/li&gt;
+&lt;li&gt;This would be the last issue remaining to close the meta issue about switching to controlled vocabularies (&lt;a href=&#34;https://github.com/ilri/DSpace/pull/69&#34;&gt;#69&lt;/a&gt;)&lt;/li&gt;
+&lt;/ul&gt;</description>
     </item>
     
     <item>

public/post/index.xml

@@ -180,8 +180,7 @@ UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4&
 <li>Now get the top 500 journal titles:</li>
 </ul>
 
-<pre><code>dspace-# \copy (select distinct text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=55 group by text_value order by cou
+<pre><code>dspace-# \copy (select distinct text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=55 group by text_value order by count desc limit 500) to /tmp/journal-titles.csv with csv;
-nt desc limit 500) to /tmp/journal-titles.csv with csv;
 </code></pre>
 
 <ul>
@@ -232,8 +231,7 @@ delete from collection2item where id = '91082';
 <li>Test importing of the new CIAT records (actually there are 232, not 234):</li>
 </ul>
 
-<pre><code>$ JAVA_OPTS="-Xmx512m -Dfile.encoding=UTF-8" /home/dspacetest.cgiar.org/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568
+<pre><code>$ JAVA_OPTS="-Xmx512m -Dfile.encoding=UTF-8" /home/dspacetest.cgiar.org/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568/79042 --source /home/aorth/CIAT_234/SimpleArchiveFormat/ --mapfile=/tmp/ciat.map &> /tmp/ciat.log
-/79042 --source /home/aorth/CIAT_234/SimpleArchiveFormat/ --mapfile=/tmp/ciat.map &> /tmp/ciat.log
 </code></pre>
 
 <ul>
@@ -292,7 +290,19 @@ $ gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/ebook -dNOPAUSE -
 </ul>
 
 <pre><code>$ ./fix-metadata-values.py -i /tmp/fix-49-journal-titles.csv -f dc.source -t correct -m 55 -d dspace -u dspace -p 'password'
-&lt;/code&gt;&lt;/pre&gt;</description>
+&lt;/code&gt;&lt;/pre&gt;
+
+&lt;ul&gt;
+&lt;li&gt;Create a new list of the top 500 journal titles from the database:&lt;/li&gt;
+&lt;/ul&gt;
+
+&lt;pre&gt;&lt;code&gt;dspace-# \copy (select distinct text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=55 group by text_value order by count desc limit 500) to /tmp/journal-titles.csv with csv;
+&lt;/code&gt;&lt;/pre&gt;
+
+&lt;ul&gt;
+&lt;li&gt;Then sort them in OpenRefine and create a controlled vocabulary by manually adding the XML markup, pull request (&lt;a href=&#34;https://github.com/ilri/DSpace/pull/298&#34;&gt;#298&lt;/a&gt;)&lt;/li&gt;
+&lt;li&gt;This would be the last issue remaining to close the meta issue about switching to controlled vocabularies (&lt;a href=&#34;https://github.com/ilri/DSpace/pull/69&#34;&gt;#69&lt;/a&gt;)&lt;/li&gt;
+&lt;/ul&gt;</description>
     </item>
     
     <item>

public/tags/notes/index.xml

@@ -179,8 +179,7 @@ UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4&
 <li>Now get the top 500 journal titles:</li>
 </ul>
 
-<pre><code>dspace-# \copy (select distinct text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=55 group by text_value order by cou
+<pre><code>dspace-# \copy (select distinct text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=55 group by text_value order by count desc limit 500) to /tmp/journal-titles.csv with csv;
-nt desc limit 500) to /tmp/journal-titles.csv with csv;
 </code></pre>
 
 <ul>
@@ -231,8 +230,7 @@ delete from collection2item where id = '91082';
 <li>Test importing of the new CIAT records (actually there are 232, not 234):</li>
 </ul>
 
-<pre><code>$ JAVA_OPTS="-Xmx512m -Dfile.encoding=UTF-8" /home/dspacetest.cgiar.org/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568
+<pre><code>$ JAVA_OPTS="-Xmx512m -Dfile.encoding=UTF-8" /home/dspacetest.cgiar.org/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568/79042 --source /home/aorth/CIAT_234/SimpleArchiveFormat/ --mapfile=/tmp/ciat.map &> /tmp/ciat.log
-/79042 --source /home/aorth/CIAT_234/SimpleArchiveFormat/ --mapfile=/tmp/ciat.map &> /tmp/ciat.log
 </code></pre>
 
 <ul>
@@ -291,7 +289,19 @@ $ gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/ebook -dNOPAUSE -
 </ul>
 
 <pre><code>$ ./fix-metadata-values.py -i /tmp/fix-49-journal-titles.csv -f dc.source -t correct -m 55 -d dspace -u dspace -p 'password'
-&lt;/code&gt;&lt;/pre&gt;</description>
+&lt;/code&gt;&lt;/pre&gt;
+
+&lt;ul&gt;
+&lt;li&gt;Create a new list of the top 500 journal titles from the database:&lt;/li&gt;
+&lt;/ul&gt;
+
+&lt;pre&gt;&lt;code&gt;dspace-# \copy (select distinct text_value, count(*) from metadatavalue where resource_type_id=2 and metadata_field_id=55 group by text_value order by count desc limit 500) to /tmp/journal-titles.csv with csv;
+&lt;/code&gt;&lt;/pre&gt;
+
+&lt;ul&gt;
+&lt;li&gt;Then sort them in OpenRefine and create a controlled vocabulary by manually adding the XML markup, pull request (&lt;a href=&#34;https://github.com/ilri/DSpace/pull/298&#34;&gt;#298&lt;/a&gt;)&lt;/li&gt;
+&lt;li&gt;This would be the last issue remaining to close the meta issue about switching to controlled vocabularies (&lt;a href=&#34;https://github.com/ilri/DSpace/pull/69&#34;&gt;#69&lt;/a&gt;)&lt;/li&gt;
+&lt;/ul&gt;</description>
     </item>
     
     <item>