Remove cg.subject.wle and cg.identifier.wletheme from CGSpace input form after confirming with IWMI colleagues that they no longer need them (WLE closed in 2021)
iso-codes 4.13.0 was released, which incorporates my changes to the common names for Iran, Laos, and Syria
Remove cg.subject.wle and cg.identifier.wletheme from CGSpace input form after confirming with IWMI colleagues that they no longer need them (WLE closed in 2021)
iso-codes 4.13.0 was released, which incorporates my changes to the common names for Iran, Laos, and Syria
<li>Remove <code>cg.subject.wle</code> and <code>cg.identifier.wletheme</code> from CGSpace input form after confirming with IWMI colleagues that they no longer need them (WLE closed in 2021)</li>
<li><ahref="https://salsa.debian.org/iso-codes-team/iso-codes/-/blob/main/CHANGELOG.md#4130-2023-02-28">iso-codes 4.13.0 was released</a>, which incorporates my changes to the common names for Iran, Laos, and Syria</li>
<li>I can’t put my finger on it, but the input form has to be formatted very particularly, for example if your rows have more than two fields in them with out a sufficient Bootstrap grid style, or if you use a <code>twobox</code>, etc, the entire form step appears blank</li>
</ul>
<h2id="2023-03-02">2023-03-02</h2>
<ul>
<li>I did some experiments with the new <ahref="https://datapythonista.me/blog/pandas-20-and-the-arrow-revolution-part-i">Pandas 2.0.0rc0 Apache Arrow support</a>
<ul>
<li>There is a change to the way nulls are handled and it causes my tests for <code>pd.isna(field)</code> to fail</li>
<li>I think we need consider blanks as null, but I’m not sure</li>
</ul>
</li>
<li>I made some adjustments to the Discovery sidebar facets on DSpace 6 while I was looking at the DSpace 7 configuration
<ul>
<li>I downgraded CIFOR subject, Humidtropics subject, Drylands subject, ICARDA subject, and Language from DiscoverySearchFilterFacet to DiscoverySearchFilter in <code>discovery.xml</code> since we are no longer using them in sidebar facets</li>
</ul>
</li>
</ul>
<h2id="2023-03-03">2023-03-03</h2>
<ul>
<li>Atmire merged one of my old pull requests into COUNTER-Robots:
<ul>
<li><ahref="https://github.com/atmire/COUNTER-Robots/pull/54">COUNTER_Robots_list.json: Add new bots</a></li>
</ul>
</li>
<li>I will update the local ILRI overrides in our DSpace spider agents file</li>
</ul>
<h2id="2023-03-04">2023-03-04</h2>
<ul>
<li>Submit a <ahref="https://github.com/flyingcircusio/pycountry/pull/156">pull request on pycountry to use iso-codes 4.13.0</a></li>
</ul>
<h2id="2023-03-05">2023-03-05</h2>
<ul>
<li>Start a harvest on AReS</li>
</ul>
<h2id="2023-03-06">2023-03-06</h2>
<ul>
<li>Export CGSpace to do Initiative collection mappings
<ul>
<li>There were thirty-three that needed updating</li>
</ul>
</li>
<li>Send Abenet and Sam a list of twenty-one CAS publications that had been marked as “multiple documents” that we uploaded as metadata-only items
<ul>
<li>Goshu will download the PDFs for each and upload them to the items on CGSpace manually</li>
</ul>
</li>
<li>I spent some time trying to get csv-metadata-quality working with the new Arrow backend for Pandas 2.0.0rc0
<ul>
<li>It seems there is a problem recognizing empty strings as na with <code>pd.isna()</code></li>
<li>If I do <code>pd.isna(field) or field == ""</code> then it works as expected, but that feels hacky</li>
<li>I’m going to test again on the next release…</li>
<li>Note that I had been setting both of these global options:</li>
<li>Peter sent me a list of items that had ILRI affiation on Altmetric, but that didn’t have Handles
<ul>
<li>I ran a duplicate check on them to find if they exist or if we can import them</li>
<li>There were about ninety matches, but a few dozen of those were pre-prints!</li>
<li>After excluding those there were about sixty-one items we already have on CGSpace so I will add their DOIs to the existing items
<ul>
<li>After joining these with the records from CGSpace and inspecting the DOIs I found that only forty-four were new DOIs</li>
<li>Surprisingly some of the DOIs on Altmetric were not working, though we also had some that were not working (specifically the Journal of Agricultural Economics seems to have reassigned DOIs)</li>
<li>For the rest of the ~359 items I extracted their DOIs and looked up the metadata on Crossref using my <code>crossref-doi-lookup.py</code> script
<ul>
<li>After spending some time cleaning the data in OpenRefine I realized we don’t get access status from Crossref</li>
<li>We can imply it if the item is Creative Commons, but otherwise I might be able to use <ahref="https://unpaywall.org/products/api">Unpaywall’s API</a></li>
<li>I found some false positives in Unpaywall, so I might only use their data when it says the DOI is not OA…</li>
<li>During this process I updated my <code>crossref-doi-lookup.py</code> script to get more information from Crossref like ISSNs, ISBNs, full journal title, and subjects</li>
<li>I am wondering how to speed up PostgreSQL trgm searches more
<ul>
<li>I see my local PostgreSQL is using vanilla configuration and I should update some configs:</li>
</ul>
</li>
</ul>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span>localhost/dspacetest= ☘ SELECT setting, unit FROM pg_settings WHERE name = 'shared_buffers';
</span></span><spanstyle="display:flex;"><span> setting │ unit
<li>Then I created a GiST <ahref="https://alexklibisz.com/2022/02/18/optimizing-postgres-trigram-search">index on the <code>metadatavalue</code> table to try to speed up the trgm similarity operations</a>:</li>
</ul>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span>localhost/dspacetest= ☘ CREATE INDEX metadatavalue_text_value_trgm_gist_idx ON metadatavalue USING gist(text_value gist_trgm_ops(siglen=64)); # \di+ shows index size is 795MB
</span></span></code></pre></div><ul>
<li>That took a few minutes to build… then the duplicate checker ran in 12 minutes: <code>0.07s user 0.02s system 0% cpu 12:43.08 total</code></li>
<li>On a hunch, I tried with a GIN index:</li>
</ul>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span>localhost/dspacetest= ☘ CREATE INDEX metadatavalue_text_value_trgm_gin_idx ON metadatavalue USING gin(text_value gin_trgm_ops); # \di+ shows index size is 274MB
</span></span></code></pre></div><ul>
<li>This ran in 19 minutes: <code>0.08s user 0.01s system 0% cpu 19:49.73 total</code>
<ul>
<li>So clearly the GiST index is better for this task</li>
<li>I am curious if I increase the signature length in the GiST index from 64 to 256 (which will for sure increase the size taken):</li>
</ul>
</li>
</ul>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span>localhost/dspacetest= ☘ CREATE INDEX metadatavalue_text_value_trgm_gist_idx ON metadatavalue USING gist(text_value gist_trgm_ops(siglen=256)); # \di+ shows index size is 716MB, which is less than the previous GiST index...
</span></span></code></pre></div><ul>
<li>This one finished in ten minutes: <code>0.07s user 0.02s system 0% cpu 10:04.04 total</code></li>
<li>I might also want to <ahref="https://stackoverflow.com/questions/43008382/postgresql-gin-index-slower-than-gist-for-pg-trgm">increase my <code>work_mem</code></a> (default 4MB):</li>
</ul>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span>localhost/dspacetest= ☘ SELECT setting, unit FROM pg_settings WHERE name = 'work_mem';
</span></span><spanstyle="display:flex;"><span> setting │ unit
<li>After updating my Crossref lookup script and checking the remaining ~359 items I found a eight more duplicates already existing on CGSpace</li>
<li>Wow, I found a <ahref="https://programminghistorian.org/en/lessons/fetch-and-parse-data-with-openrefine#example-1-fetching-and-parsing-html">really cool way to fetch URLs in OpenRefine</a>
<ul>
<li>I used this to fetch the open access status for each DOI from Unpaywall</li>
</ul>
</li>
<li>First, create a new column called “url” based on the DOI that builds the request URL. I used a Jython expression:</li>
<li>Then create a new column based on fetching the values in that column. I called it “unpaywall_status”</li>
<li>Then you get a JSON blob in each and you can extract the Open Access status with a GREL like <code>value.parseJson()['is_oa']</code>
<ul>
<li>I checked a handful of results manually and found that the limited access status was more trustworthy from Unpaywall than the open access, so I will just tag the limited access ones</li>
</ul>
</li>
<li>I merged the funders and affiliations from Altmetric into my file, then used the same technique to get Crossref data for open access items directly into OpenRefine and parsed the abstracts
<ul>
<li>The syntax was hairy because it’s marked up with tags like <code><jats:p></code>, but this got me most of the way there:</li>
<li>Meeting with FAO AGRIS team about how to detect duplicates
<ul>
<li>They are currently using a sha256 hash on titles, which will work, but will only return exact matches</li>
<li>I told them to try to normalize the string, drop stop words, etc to increase the possibility that the hash matches</li>
</ul>
</li>
<li>Meeting with Abenet to discuss CGSpace issues
<ul>
<li>She reminded me about needing a metadata field for first author when the affiliation is ILRI</li>
<li>I said I prefer to write a small script for her that will check the first author and first affiliation… I could do it easily in Python, but would need to put a web frontend on it for her</li>
<li>Unless we could do that in AReS reports somehow</li>
<li>Apply a bunch of corrections to authors, affiliations, and donors on the new items on DSpace Test</li>
<li>Meeting with Peter and Abenet about future OpenRXV developments, DSpace 7, etc
<ul>
<li>I submitted an <ahref="https://github.com/CodeObia/MEL/issues/11173">issue on MEL asking them to add provenance metadata when submitting to CGSpace</a></li>
<li>CKM is getting ready to launch their new website and they display CGSpace thumbnails at 255x362px
<ul>
<li>Our thumbnails are 300px so they get up-scaled and look bad</li>
<li>I realized that the last time we <ahref="https://github.com/ilri/DSpace/commit/5de61e220124c1d0441c87cd7d36d18cb2293c03">increased the size of our thumbnails was in 2013</a>, from 94x130 to 300px</li>
<li>I offered to CKM that we increase them again to 400 or 600px</li>
<li>I did some tests to check the thumbnail file sizes for 300px, 400px, 500px, and 600px on <ahref="https://hdl.handle.net/10568/126388">this item</a>:</li>
</ul>
</li>
</ul>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span>$ ls -lh 10568-126388-*
</span></span><spanstyle="display:flex;"><span>-rw-r--r-- 1 aorth aorth 31K Mar 10 12:42 10568-126388-300px.jpg
</span></span><spanstyle="display:flex;"><span>-rw-r--r-- 1 aorth aorth 52K Mar 10 12:41 10568-126388-400px.jpg
</span></span><spanstyle="display:flex;"><span>-rw-r--r-- 1 aorth aorth 76K Mar 10 12:43 10568-126388-500px.jpg
</span></span><spanstyle="display:flex;"><span>-rw-r--r-- 1 aorth aorth 106K Mar 10 12:44 10568-126388-600px.jpg
</span></span></code></pre></div><ul>
<li>Seems like 600px is 3 to 4 times larger file size, so maybe we should shoot for 400px or 500px
<li>On that note, I also re-worked the XMLUI item display to show larger thumbnails (from a max-width of 128px to 200px)</li>
<li>And now that I’m looking at thumbnails I am curious what it would take to get DSpace to generate WebP or AVIF thumbnails</li>
<li>Peter sent me citations and ILRI subjects for the 350 new ILRI publications
<ul>
<li>I guess he edited it in Excel because there are a bunch of encoding issues with accents</li>
<li>I merged Peter’s citations and subjects with the other metadata, ran one last duplicate check (and found one item!), then ran the items through csv-metadata-quality and uploaded them to CGSpace</li>
<li>In the end it was only 348 items for some reason…</li>
<li>Extract a list of DOIs from the Creative Commons licensed ILRI journal articles that I uploaded last week, skipping any that are “no derivatives” (ND):</li>
</ul>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span>$ csvgrep -c <spanstyle="color:#e6db74">'dc.description.provenance[en]'</span> -m <spanstyle="color:#e6db74">'Made available in DSpace on 2023-03-10'</span> /tmp/ilri-articles.csv <spanstyle="color:#ae81ff">\
<li>I tried in both the <code>ORIGINAL</code> and <code>THUMBNAIL</code> bundle, and with different filenames</li>
<li>I tried manually on the command line with <code>http</code> and both PDF and PNG work… hmmmm</li>
<li>Hmm, this seems to have been due to some difference in behavior between the <code>files</code> and <code>data</code> parameters of <code>requests.get()</code></li>
<li>I finalized the <code>post_bitstreams.py</code> script and uploaded eighty-five PDF thumbnails</li>
</ul>
</li>
<li>It seems Bizu uploaded covers for a handful so I deleted them and ran them through the script to get proper thumbnails</li>
<li>Jawoo was asking about possibilities to harvest PDFs from CGSpace for some kind of AI chatbot integration
<ul>
<li>I see we have 45,000 PDFs (format ID 2)</li>
</ul>
</li>
</ul>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span>localhost/dspacetest= ☘ SELECT COUNT(*) FROM bitstream WHERE NOT deleted AND bitstream_format_id=2;
<li>Rework some of my Python scripts to use a common <code>db_connect</code> function from util</li>
<li>I reworked my <code>post_bitstreams.py</code> script to be able to overwrite bitstreams if requested
<ul>
<li>The use case is to upload thumbnails for all the journal articles where we have these horrible pixelated journal covers</li>
<li>I replaced JPEG thumbnails for ~896 ILRI publications by exporting a list of DOIs from the 10568/3 collection that were CC-BY, getting their PDFs from Sci-Hub, and then posting them with my new script</li>
</ul>
</li>
</ul>
<h2id="2023-03-16">2023-03-16</h2>
<ul>
<li>Continue working on the ILRI publication thumbnails
<ul>
<li>There were about sixty-four that had existing PNG “journal cover” thumbnails that didn’t get replaced because I only overwrote the JPEG ones yesterday</li>
<li>Now I generated a list of those bitstream UUIDs and deleted them with a shell script via the REST API</li>
</ul>
</li>
<li>I made a <ahref="https://github.com/DSpace/DSpace/pull/8722">pull request on DSpace 7 to update the bitstream format registry for PNG, WebP, and AVIF</a></li>
<li>Export CGSpace to perform mappings to Initiatives collections</li>
<li>I also used this export to find CC-BY items with DOIs that had JPEGs or PNGs in their provenance, meaning that the submitter likely submitted a low-quality “journal cover” for the item
<ul>
<li>I found about 330 of them and got most of their PDFs from Sci-Hub and replaced the crappy thumbnails with real ones where Sci-Hub had them (~245)</li>
</ul>
</li>
<li>In related news, I realized you can get an <ahref="https://stackoverflow.com/questions/59202176/python-download-papers-from-sciencedirect-by-doi-with-requests">API key from Elsevier and download the PDFs from their API</a>:</li>
<li>The question is, how do we know if a DOI is Elsevier or not…</li>
<li>CGIAR Repositories Working Group meeting
<ul>
<li>We discussed controlled vocabularies for funders</li>
<li>I suggested checking our combined lists against Crossref and ROR</li>
</ul>
</li>
<li>Export a list of donors from <code>cg.contributor.donor</code> on CGSpace:</li>
</ul>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span>localhost/dspacetest= ☘ \COPY (SELECT DISTINCT(text_value) FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=248) to /tmp/2023-03-16-donors.txt;
<li>That’s a 31% hit rate, but I see some simple things like “Bill and Melinda Gates Foundation” instead of “Bill & Melinda Gates Foundation”</li>
</ul>
<h2id="2023-03-17">2023-03-17</h2>
<ul>
<li>I did the same lookup of CGSpace donors on ROR’s 2022-12-01 data dump:</li>
<li>On a related note, I remembered that DOI.org has a list of DOI prefixes and publishers: <ahref="https://doi.crossref.org/getPrefixPublisher">https://doi.crossref.org/getPrefixPublisher</a>
<ul>
<li>In Python I can look up publishers by prefix easily, here with a nested list comprehension:</li>
</ul>
</li>
</ul>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span>In [10]: [publisher for publisher in publishers if '10.3390' in publisher['prefixes']]
<li>I learned how to use Python’s built-in <code>logging</code> module and it simplifies all my debug and info printing
<ul>
<li>I re-factored a few scripts to use the new logging</li>
</ul>
</li>
</ul>
<h2id="2023-03-18">2023-03-18</h2>
<ul>
<li>I applied changes for publishers on 16,000 items in batches of 5,000</li>
<li>While working on my <code>post_bitstreams.py</code> script I realized the Tomcat Crawler Session Manager valve that groups bot user agents into sessions is causing my login to fail the first time, every time
<ul>
<li>I’ve disabled it for now and will check the Munin session graphs after some time to see if it makes a difference</li>
<li>In any case I have much better spider user agent lists in DSpace now than I did years ago when I started using the Crawler Session Manager valve</li>
</span></span><spanstyle="display:flex;"><span># pg_ctlcluster <spanstyle="color:#ae81ff">12</span> main stop
</span></span><spanstyle="display:flex;"><span># tar -cvzpf var-lib-postgresql-12.tar.gz /var/lib/postgresql/12
</span></span><spanstyle="display:flex;"><span># tar -cvzpf etc-postgresql-12.tar.gz /etc/postgresql/12
</span></span><spanstyle="display:flex;"><span># pg_ctlcluster <spanstyle="color:#ae81ff">14</span> main stop
</span></span><spanstyle="display:flex;"><span># pg_dropcluster <spanstyle="color:#ae81ff">14</span> main
</span></span><spanstyle="display:flex;"><span># pg_upgradecluster <spanstyle="color:#ae81ff">12</span> main
</span></span><spanstyle="display:flex;"><span># pg_ctlcluster <spanstyle="color:#ae81ff">14</span> main start
</span></span></code></pre></div><ul>
<li>After that I <ahref="https://adamj.eu/tech/2021/04/13/reindexing-all-tables-after-upgrading-to-postgresql-13/">re-indexed the database indexes using a query</a>:</li>
</ul>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span>$ su - postgres
<li>The index on <code>metadatavalue</code> shrunk by 90MB, and others a bit less
<ul>
<li>This is nice, but not as drastic as I noticed last year when upgrading to PostgreSQL 12</li>
</ul>
</li>
</ul>
<h2id="2023-03-21">2023-03-21</h2>
<ul>
<li>Leigh sent me a list of IFPRI authors with ORCID identifiers so I combined them with our list and resolved all their names with <code>resolve_orcids.py</code>
<ul>
<li>It adds 154 new ORCID identifiers</li>
</ul>
</li>
<li>I did a follow up to the publisher names from last week using the list from doi.org
<ul>
<li>Last week I only updated items with a DOI that had <em>no</em> publisher, but now I was curious to see how our existing publisher information compared</li>
<li>I checked a dozen or so manually and, other than CIFOR/ICRAF and CIAT/Alliance, the metadata was better than our existing data, so I overwrote them</li>
<li>I spent some time trying to figure out how to get ssimulacra2 running so I could compare thumbnails in JPEG and WebP
<ul>
<li>I realized that we can’t directly compare JPEG to WebP, we need to convert to JPEG/WebP, then convert each to lossless PNG</li>
<li>Also, we shouldn’t be comparing the resulting images against each other, but rather the original, so I need to a straight PDF to lossless PNG version also</li>
<li>After playing with WebP at Q82 and Q92, I see it has lower ssimulacra2 scores than JPEG Q92 for the dozen test files</li>
<li>Could it just be something with ImageMagick?</li>
<li>I updated csv-metadata-quality to use pandas 2.0.0rc1 and everything seems to work…?
<ul>
<li>So the issues with nulls (isna) when I tried the first release candidate a few weeks ago were resolved?</li>
</ul>
</li>
<li>Meeting with Jawoo and others about a “ChatGPT-like” thing for CGIAR data using CGSpace documents and metadata</li>
</ul>
<h2id="2023-03-23">2023-03-23</h2>
<ul>
<li>Add a missing IFPRI ORCID identifier to CGSpace and tag his items on CGSpace</li>
<li>A super unscientific comparison between csv-metadata-quality’s pytest regimen using Pandas 1.5.3 and Pandas 2.0.0rc1
<ul>
<li>The data was gathered using <ahref="https://justine.lol/rusage">rusage</a>, and this is the results of the last of three consecutive runs:</li>
</ul>
</li>
</ul>
<pretabindex="0"><code># Pandas 1.5.3
RL: took 1,585,999µs wall time
RL: ballooned to 272,380kb in size
RL: needed 2,093,947µs cpu (25% kernel)
RL: caused 55,856 page faults (100% memcpy)
RL: 699 context switches (1% consensual)
RL: performed 0 reads and 16 write i/o operations
# Pandas 2.0.0rc1
RL: took 1,625,718µs wall time
RL: ballooned to 262,116kb in size
RL: needed 2,148,425µs cpu (24% kernel)
RL: caused 63,934 page faults (100% memcpy)
RL: 461 context switches (2% consensual)
RL: performed 0 reads and 16 write i/o operations
</code></pre><ul>
<li>So it seems that Pandas 2.0.0rc1 took ten megabytes less RAM… interesting to see that the PyArrow-backed dtypes make a measurable difference even on my small test set
<ul>
<li>I should try to compare runs of larger input files</li>
</ul>
</li>
</ul>
<h2id="2023-03-24">2023-03-24</h2>
<ul>
<li>I added a Flyway SQL migration for the PNG bitstream format registry changes on DSpace 7.6</li>
