Remove cg.subject.wle and cg.identifier.wletheme from CGSpace input form after confirming with IWMI colleagues that they no longer need them (WLE closed in 2021)
iso-codes 4.13.0 was released, which incorporates my changes to the common names for Iran, Laos, and Syria
Remove cg.subject.wle and cg.identifier.wletheme from CGSpace input form after confirming with IWMI colleagues that they no longer need them (WLE closed in 2021)
iso-codes 4.13.0 was released, which incorporates my changes to the common names for Iran, Laos, and Syria
<li>Remove <code>cg.subject.wle</code> and <code>cg.identifier.wletheme</code> from CGSpace input form after confirming with IWMI colleagues that they no longer need them (WLE closed in 2021)</li>
<li><ahref="https://salsa.debian.org/iso-codes-team/iso-codes/-/blob/main/CHANGELOG.md#4130-2023-02-28">iso-codes 4.13.0 was released</a>, which incorporates my changes to the common names for Iran, Laos, and Syria</li>
<li>I can’t put my finger on it, but the input form has to be formatted very particularly, for example if your rows have more than two fields in them with out a sufficient Bootstrap grid style, or if you use a <code>twobox</code>, etc, the entire form step appears blank</li>
</ul>
<h2id="2023-03-02">2023-03-02</h2>
<ul>
<li>I did some experiments with the new <ahref="https://datapythonista.me/blog/pandas-20-and-the-arrow-revolution-part-i">Pandas 2.0.0rc0 Apache Arrow support</a>
<ul>
<li>There is a change to the way nulls are handled and it causes my tests for <code>pd.isna(field)</code> to fail</li>
<li>I think we need consider blanks as null, but I’m not sure</li>
</ul>
</li>
<li>I made some adjustments to the Discovery sidebar facets on DSpace 6 while I was looking at the DSpace 7 configuration
<ul>
<li>I downgraded CIFOR subject, Humidtropics subject, Drylands subject, ICARDA subject, and Language from DiscoverySearchFilterFacet to DiscoverySearchFilter in <code>discovery.xml</code> since we are no longer using them in sidebar facets</li>
</ul>
</li>
</ul>
<h2id="2023-03-03">2023-03-03</h2>
<ul>
<li>Atmire merged one of my old pull requests into COUNTER-Robots:
<ul>
<li><ahref="https://github.com/atmire/COUNTER-Robots/pull/54">COUNTER_Robots_list.json: Add new bots</a></li>
</ul>
</li>
<li>I will update the local ILRI overrides in our DSpace spider agents file</li>
</ul>
<h2id="2023-03-04">2023-03-04</h2>
<ul>
<li>Submit a <ahref="https://github.com/flyingcircusio/pycountry/pull/156">pull request on pycountry to use iso-codes 4.13.0</a></li>
</ul>
<h2id="2023-03-05">2023-03-05</h2>
<ul>
<li>Start a harvest on AReS</li>
</ul>
<h2id="2023-03-06">2023-03-06</h2>
<ul>
<li>Export CGSpace to do Initiative collection mappings
<ul>
<li>There were thirty-three that needed updating</li>
</ul>
</li>
<li>Send Abenet and Sam a list of twenty-one CAS publications that had been marked as “multiple documents” that we uploaded as metadata-only items
<ul>
<li>Goshu will download the PDFs for each and upload them to the items on CGSpace manually</li>
</ul>
</li>
<li>I spent some time trying to get csv-metadata-quality working with the new Arrow backend for Pandas 2.0.0rc0
<ul>
<li>It seems there is a problem recognizing empty strings as na with <code>pd.isna()</code></li>
<li>If I do <code>pd.isna(field) or field == ""</code> then it works as expected, but that feels hacky</li>
<li>I’m going to test again on the next release…</li>
<li>Note that I had been setting both of these global options:</li>
<li>Peter sent me a list of items that had ILRI affiation on Altmetric, but that didn’t have Handles
<ul>
<li>I ran a duplicate check on them to find if they exist or if we can import them</li>
<li>There were about ninety matches, but a few dozen of those were pre-prints!</li>
<li>After excluding those there were about sixty-one items we already have on CGSpace so I will add their DOIs to the existing items
<ul>
<li>After joining these with the records from CGSpace and inspecting the DOIs I found that only forty-four were new DOIs</li>
<li>Surprisingly some of the DOIs on Altmetric were not working, though we also had some that were not working (specifically the Journal of Agricultural Economics seems to have reassigned DOIs)</li>
<li>For the rest of the ~359 items I extracted their DOIs and looked up the metadata on Crossref using my <code>crossref-doi-lookup.py</code> script
<ul>
<li>After spending some time cleaning the data in OpenRefine I realized we don’t get access status from Crossref</li>
<li>We can imply it if the item is Creative Commons, but otherwise I might be able to use <ahref="https://unpaywall.org/products/api">Unpaywall’s API</a></li>
<li>I found some false positives in Unpaywall, so I might only use their data when it says the DOI is not OA…</li>
<li>During this process I updated my <code>crossref-doi-lookup.py</code> script to get more information from Crossref like ISSNs, ISBNs, full journal title, and subjects</li>
<li>I am wondering how to speed up PostgreSQL trgm searches more
<ul>
<li>I see my local PostgreSQL is using vanilla configuration and I should update some configs:</li>
</ul>
</li>
</ul>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span>localhost/dspacetest= ☘ SELECT setting, unit FROM pg_settings WHERE name = 'shared_buffers';
</span></span><spanstyle="display:flex;"><span> setting │ unit
<li>Then I created a GiST <ahref="https://alexklibisz.com/2022/02/18/optimizing-postgres-trigram-search">index on the <code>metadatavalue</code> table to try to speed up the trgm similarity operations</a>:</li>
</ul>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span>localhost/dspacetest= ☘ CREATE INDEX metadatavalue_text_value_trgm_gist_idx ON metadatavalue USING gist(text_value gist_trgm_ops(siglen=64)); # \di+ shows index size is 795MB
</span></span></code></pre></div><ul>
<li>That took a few minutes to build… then the duplicate checker ran in 12 minutes: <code>0.07s user 0.02s system 0% cpu 12:43.08 total</code></li>
<li>On a hunch, I tried with a GIN index:</li>
</ul>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span>localhost/dspacetest= ☘ CREATE INDEX metadatavalue_text_value_trgm_gin_idx ON metadatavalue USING gin(text_value gin_trgm_ops); # \di+ shows index size is 274MB
</span></span></code></pre></div><ul>
<li>This ran in 19 minutes: <code>0.08s user 0.01s system 0% cpu 19:49.73 total</code>
<ul>
<li>So clearly the GiST index is better for this task</li>
<li>I am curious if I increase the signature length in the GiST index from 64 to 256 (which will for sure increase the size taken):</li>
</ul>
</li>
</ul>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span>localhost/dspacetest= ☘ CREATE INDEX metadatavalue_text_value_trgm_gist_idx ON metadatavalue USING gist(text_value gist_trgm_ops(siglen=256)); # \di+ shows index size is 716MB, which is less than the previous GiST index...
</span></span></code></pre></div><ul>
<li>This one finished in ten minutes: <code>0.07s user 0.02s system 0% cpu 10:04.04 total</code></li>
<li>I might also want to <ahref="https://stackoverflow.com/questions/43008382/postgresql-gin-index-slower-than-gist-for-pg-trgm">increase my <code>work_mem</code></a> (default 4MB):</li>
</ul>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span>localhost/dspacetest= ☘ SELECT setting, unit FROM pg_settings WHERE name = 'work_mem';
</span></span><spanstyle="display:flex;"><span> setting │ unit
<li>After updating my Crossref lookup script and checking the remaining ~359 items I found a eight more duplicates already existing on CGSpace</li>
<li>Wow, I found a <ahref="https://programminghistorian.org/en/lessons/fetch-and-parse-data-with-openrefine#example-1-fetching-and-parsing-html">really cool way to fetch URLs in OpenRefine</a>
<ul>
<li>I used this to fetch the open access status for each DOI from Unpaywall</li>
</ul>
</li>
<li>First, create a new column called “url” based on the DOI that builds the request URL. I used a Jython expression:</li>
<li>Then create a new column based on fetching the values in that column. I called it “unpaywall_status”</li>
<li>Then you get a JSON blob in each and you can extract the Open Access status with a GREL like <code>value.parseJson()['is_oa']</code>
<ul>
<li>I checked a handful of results manually and found that the limited access status was more trustworthy from Unpaywall than the open access, so I will just tag the limited access ones</li>
</ul>
</li>
<li>I merged the funders and affiliations from Altmetric into my file, then used the same technique to get Crossref data for open access items directly into OpenRefine and parsed the abstracts
<ul>
<li>The syntax was hairy because it’s marked up with tags like <code><jats:p></code>, but this got me most of the way there:</li>
<li>Meeting with FAO AGRIS team about how to detect duplicates
<ul>
<li>They are currently using a sha256 hash on titles, which will work, but will only return exact matches</li>
<li>I told them to try to normalize the string, drop stop words, etc to increase the possibility that the hash matches</li>
</ul>
</li>
<li>Meeting with Abenet to discuss CGSpace issues
<ul>
<li>She reminded me about needing a metadata field for first author when the affiliation is ILRI</li>
<li>I said I prefer to write a small script for her that will check the first author and first affiliation… I could do it easily in Python, but would need to put a web frontend on it for her</li>
<li>Unless we could do that in AReS reports somehow</li>
<li>Apply a bunch of corrections to authors, affiliations, and donors on the new items on DSpace Test</li>
<li>Meeting with Peter and Abenet about future OpenRXV developments, DSpace 7, etc
<ul>
<li>I submitted an <ahref="https://github.com/CodeObia/MEL/issues/11173">issue on MEL asking them to add provenance metadata when submitting to CGSpace</a></li>
