mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2025-01-27 05:49:12 +01:00
Add notes
This commit is contained in:
@ -21,7 +21,7 @@ We agreed to try to do more alignment of affiliations/funders with ROR
|
||||
<meta property="og:type" content="article" />
|
||||
<meta property="og:url" content="https://alanorth.github.io/cgspace-notes/2022-02/" />
|
||||
<meta property="article:published_time" content="2022-02-01T14:06:54+02:00" />
|
||||
<meta property="article:modified_time" content="2022-02-14T09:40:59+03:00" />
|
||||
<meta property="article:modified_time" content="2022-02-14T16:43:12+03:00" />
|
||||
|
||||
|
||||
|
||||
@ -38,7 +38,7 @@ We agreed to try to do more alignment of affiliations/funders with ROR
|
||||
|
||||
|
||||
"/>
|
||||
<meta name="generator" content="Hugo 0.92.1" />
|
||||
<meta name="generator" content="Hugo 0.92.2" />
|
||||
|
||||
|
||||
|
||||
@ -48,9 +48,9 @@ We agreed to try to do more alignment of affiliations/funders with ROR
|
||||
"@type": "BlogPosting",
|
||||
"headline": "February, 2022",
|
||||
"url": "https://alanorth.github.io/cgspace-notes/2022-02/",
|
||||
"wordCount": "2194",
|
||||
"wordCount": "2868",
|
||||
"datePublished": "2022-02-01T14:06:54+02:00",
|
||||
"dateModified": "2022-02-14T09:40:59+03:00",
|
||||
"dateModified": "2022-02-14T16:43:12+03:00",
|
||||
"author": {
|
||||
"@type": "Person",
|
||||
"name": "Alan Orth"
|
||||
@ -532,6 +532,139 @@ $ csvjoin -c id /tmp/2022-02-14-tac-batch3-401-700.csv /tmp/tac3-filenames.csv &
|
||||
</code></pre></div><ul>
|
||||
<li>I sent these 300 items to Gaia…</li>
|
||||
</ul>
|
||||
<h2 id="2022-02-16">2022-02-16</h2>
|
||||
<ul>
|
||||
<li>Upgrade PostgreSQL on DSpace Test from version 10 to 12
|
||||
<ul>
|
||||
<li>First, I installed the new version of PostgreSQL via the Ansible playbook scripts</li>
|
||||
<li>Then I stopped Tomcat and all PostgreSQL clusters and used <code>pg_upgrade</code> to upgrade the old version:</li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console"># systemctl stop tomcat7
|
||||
# pg_ctlcluster <span style="color:#ae81ff">10</span> main stop
|
||||
# tar -cvzpf var-lib-postgresql-10.tar.gz /var/lib/postgresql/10
|
||||
# tar -cvzpf etc-postgresql-10.tar.gz /etc/postgresql/10
|
||||
# pg_ctlcluster <span style="color:#ae81ff">12</span> main stop
|
||||
# pg_dropcluster <span style="color:#ae81ff">12</span> main
|
||||
# pg_upgradecluster <span style="color:#ae81ff">10</span> main
|
||||
# pg_ctlcluster <span style="color:#ae81ff">12</span> main start
|
||||
</code></pre></div><ul>
|
||||
<li>After that I <a href="https://adamj.eu/tech/2021/04/13/reindexing-all-tables-after-upgrading-to-postgresql-13/">re-indexed the database indexes using a query</a>:</li>
|
||||
</ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">$ su - postgres
|
||||
$ cat /tmp/generate-reindex.sql
|
||||
SELECT 'REINDEX TABLE CONCURRENTLY ' || quote_ident(relname) || ' /*' || pg_size_pretty(pg_total_relation_size(C.oid)) || '*/;'
|
||||
FROM pg_class C
|
||||
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
|
||||
WHERE nspname = 'public'
|
||||
AND C.relkind = 'r'
|
||||
AND nspname !~ '^pg_toast'
|
||||
ORDER BY pg_total_relation_size(C.oid) ASC;
|
||||
$ psql dspace < /tmp/generate-reindex.sql > /tmp/reindex.sql
|
||||
$ <trim the extra stuff from /tmp/reindex.sql>
|
||||
$ psql dspace < /tmp/reindex.sql
|
||||
</code></pre></div><ul>
|
||||
<li>I saw that the index on <code>metadatavalue</code> shrunk by about 200MB!</li>
|
||||
<li>After testing a few things I dropped the old cluster:</li>
|
||||
</ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console"># pg_dropcluster <span style="color:#ae81ff">10</span> main
|
||||
# dpkg -l | grep postgresql-10 | awk <span style="color:#e6db74">'{print $2}'</span> | xargs dpkg -r
|
||||
</code></pre></div><h2 id="2022-02-17">2022-02-17</h2>
|
||||
<ul>
|
||||
<li>I updated my <code>migrate-fields.sh</code> script to use field names instead of IDs
|
||||
<ul>
|
||||
<li>The script now looks up the appropriate <code>metadata_field_id</code> values for each field in the metadata registry</li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<h2 id="2022-02-18">2022-02-18</h2>
|
||||
<ul>
|
||||
<li>Normalize the <code>text_lang</code> attributes of metadata on CGSpace:</li>
|
||||
</ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">dspace=# SELECT DISTINCT text_lang, count(text_lang) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) GROUP BY text_lang ORDER BY count DESC;
|
||||
text_lang | count
|
||||
-----------+---------
|
||||
en_US | 2838588
|
||||
en | 1082
|
||||
| 801
|
||||
fr | 2
|
||||
vn | 2
|
||||
en_US. | 1
|
||||
sp | 1
|
||||
| 0
|
||||
(8 rows)
|
||||
dspace=# UPDATE metadatavalue SET text_lang='en_US' WHERE dspace_object_id IN (SELECT uuid FROM item) AND text_lang IN ('en', 'en_US.', '');
|
||||
UPDATE 1884
|
||||
dspace=# UPDATE metadatavalue SET text_lang='vi' WHERE dspace_object_id IN (SELECT uuid FROM item) AND text_lang IN ('vn');
|
||||
UPDATE 2
|
||||
dspace=# UPDATE metadatavalue SET text_lang='es' WHERE dspace_object_id IN (SELECT uuid FROM item) AND text_lang IN ('sp');
|
||||
UPDATE 1
|
||||
</code></pre></div><ul>
|
||||
<li>I then exported the entire repository and did some cleanup on DOIs
|
||||
<ul>
|
||||
<li>I found ~1,200 items with no <code>cg.identifier.doi</code>, but which had a DOI in their citation</li>
|
||||
<li>I cleaned up and normalized a few hundred others to use <a href="https://doi.org">https://doi.org</a> format</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>I’m debating using the Crossref API to search for our DOIs and improve our metadata
|
||||
<ul>
|
||||
<li>For example: <a href="https://api.crossref.org/works/10.1016/j.ecolecon.2008.03.011">https://api.crossref.org/works/10.1016/j.ecolecon.2008.03.011</a></li>
|
||||
<li>There is good data on publishers, issue dates, volume/issue, and sometimes even licenses</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>I cleaned up ~1,200 URLs that were using HTTP instead of HTTPS, fixed a bunch of handles, removed some handles from DOI field, etc</li>
|
||||
</ul>
|
||||
<h2 id="2022-02-20">2022-02-20</h2>
|
||||
<ul>
|
||||
<li>Yesterday I wrote a script to check our DOIs against Crossref’s API and the did some investigation on dates, volumes, issues, pages, and types
|
||||
<ul>
|
||||
<li>While investigating issue dates in OpenRefine I created a new column using this GREL to show the number of days between Crossref’s date and ours:</li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">abs(diff(toDate(cells["issued"].value),toDate(cells["dcterms.issued[en_US]"].value), "days"))
|
||||
</code></pre></div><ul>
|
||||
<li>In <em>most</em> cases Crossref’s dates are more correct than ours, though there are a few odd cases that I don’t know what strategy I want to use yet</li>
|
||||
<li>Start a full harvest on AReS</li>
|
||||
</ul>
|
||||
<h2 id="2022-02-21">2022-02-21</h2>
|
||||
<ul>
|
||||
<li>I added support for checking the license of DOIs to my Crossref script
|
||||
<ul>
|
||||
<li>I exported ~2,800 DOIs and ran a check on them, then merged the CGSpace CSV with the results of the script to inspect in OpenRefine</li>
|
||||
<li>There are hundreds of DOIs missing licenses in our data, even in this small subset of ~2,800 (out of 19,000 on CGSpace)</li>
|
||||
<li>I spot checked a few dozen in Crossref’s data and found some incorrect ones, like on Elsevier, Wiley, and Sage journals</li>
|
||||
<li>I ended up using a series of GREL expressions in OpenRefine that ended up filtering out DOIs from these prefixes:</li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-console" data-lang="console">or(
|
||||
value.contains("10.1017"),
|
||||
value.contains("10.1007"),
|
||||
value.contains("10.1016"),
|
||||
value.contains("10.1098"),
|
||||
value.contains("10.1111"),
|
||||
value.contains("10.1002"),
|
||||
value.contains("10.1046"),
|
||||
value.contains("10.2135"),
|
||||
value.contains("10.1006"),
|
||||
value.contains("10.1177"),
|
||||
value.contains("10.1079"),
|
||||
value.contains("10.2298"),
|
||||
value.contains("10.1186"),
|
||||
value.contains("10.3835"),
|
||||
value.contains("10.1128"),
|
||||
value.contains("10.3732"),
|
||||
value.contains("10.2134")
|
||||
)
|
||||
</code></pre></div><ul>
|
||||
<li>Many many of Crossref’s records are correct where we have no license, and in some cases more correct when we have a different license
|
||||
<ul>
|
||||
<li>I ran license updates on ~167 DOIs in the end on CGSpace</li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<!-- raw HTML omitted -->
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user