<li>A bit of work on the “Mapping CG Core–CGSpace–MEL–MARLO Types” spreadsheet</li>
<li>I tested an item submission on DSpace Test with the Cocoon <code>org.apache.cocoon.uploads.autosave=false</code> change
<ul>
<li>The submission works as expected</li>
</ul>
</li>
<li>Start debugging some region-related issues with csv-metadata-quality
<ul>
<li>I created a new test file <code>test-geography.csv</code> with some different scenarios</li>
<li>I also fixed a few bugs and improved the region-matching logic</li>
</ul>
</li>
</ul>
<ul>
<li>I filed <ahref="https://github.com/konstantinstadler/country_converter/issues/115">an issue for the “South-eastern Asia” case mismatch in country_converter</a> on GitHub</li>
<li>Meeting with Moayad to discuss OpenRXV developments
<ul>
<li>He demoed his new multiple dashboards feature and I helped him rebase those changes to master so we can test them more</li>
</ul>
</li>
</ul>
<h2id="2022-09-02">2022-09-02</h2>
<ul>
<li>I worked a bit more on exclusion and skipping logic in csv-metadata-quality
<ul>
<li>I also pruned and updated all the Python dependencies</li>
<li>Then I released <ahref="https://github.com/ilri/csv-metadata-quality/releases/tag/v0.6.0">version 0.6.0</a> now that the excludes and region matching support is working way better</li>
</ul>
</li>
</ul>
<h2id="2022-09-05">2022-09-05</h2>
<ul>
<li>Started a harvest on AReS last night</li>
<li>Looking over the Solr statistics from last month I see many user agents that look suspicious:
<ul>
<li>Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; WOW64; Trident/7.0; .NET4.0E; .NET4.0C)</li>
<li>Mozilla / 5.0(Windows NT 10.0; Win64; x64) AppleWebKit / 537.36(KHTML, like Gecko) Chrome / 77.0.3865.90 Safari / 537.36</li>
<li>Mozilla/5.0 (Windows NT 10.0; WOW64; Rv:50.0) Gecko/20100101 Firefox/50.0</li>
<li>Mozilla/5.0 (X11; Linux i686; rv:2.0b12pre) Gecko/20110204 Firefox/4.0b12pre</li>
<li>Mozilla/5.0 (Windows NT 10.0; Win64; x64; Xbox; Xbox One) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36 Edge/44.18363.8131</li>
<li>Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)</li>
<li>Mozilla/4.0 (compatible; MSIE 4.5; Windows 98;)</li>
<li>curb</li>
<li>bitdiscovery</li>
<li>omgili/0.5 +http://omgili.com</li>
<li>Mozilla/5.0 (compatible)</li>
<li>Vizzit</li>
<li>Mozilla/5.0 (Windows NT 5.1; rv:52.0) Gecko/20100101 Firefox/52.0</li>
<li>For example, some are apparently using versions of Firefox that are over ten years old, and some are obviously trying to look like valid user agents, but making typos (<code>Mozilla / 5.0</code>)</li>
<li>I see some new Hetzner ranges that I hadn’t blocked yet apparently?
<ul>
<li>I got a <ahref="https://www.ipqualityscore.com/asn-details/AS24940/hetzner-online-gmbh">list of Hetzner’s IPs from IP Quality Score</a> then added them to the existing ones in my Ansible playbooks:</li>
<li>I will add this new list to nginx’s <code>bot-networks.conf</code> so they get throttled on scraping XMLUI and get classified as bots in Solr statistics</li>
<li>Then I purged hits from the following user agents:</li>
</span></span></span><spanstyle="display:flex;"><span><spanstyle="color:#960050;background-color:#1e0010"></span>Total number of hits from bots: 12220
</span></span></code></pre></div><ul>
<li>Then I will add these user agents to the ILRI spider override in DSpace</li>
</ul>
<h2id="2022-09-06">2022-09-06</h2>
<ul>
<li>I’m testing dspace-statistics-api with our DSpace 7 test server
<ul>
<li>After setting up the env and the database the <code>python -m dspace_statistics_api.indexer</code> runs without issues</li>
<li>While playing with Solr I tried to search for statistics from this month using <code>time:2022-09*</code> but I get this error: “Can’t run prefix queries on numeric fields”</li>
<li>I guess that the syntax in Solr changed since 4.10…</li>
<li>This works, but is super annoying: <code>time:[2022-09-01T00:00:00Z TO 2022-09-30T23:59:59Z]</code></li>
</ul>
</li>
</ul>
<h2id="2022-09-07">2022-09-07</h2>
<ul>
<li>I tested the controlled-vocabulary changes on DSpace 6 and they work fine
<ul>
<li>Last week I found that DSpace 7 is more strict with controlled vocabularies and requires IDs for all node values</li>
<li>This is a pain because it means I have to re-do the IDs in each file every time I update them</li>
<li>If I add <code>id="0000"</code> to each, then I can use <ahref="https://vim.fandom.com/wiki/Making_a_list_of_numbers#Substitute_with_ascending_numbers">this vim expression</a><code>let i=0001 | g/0000/s//\=i/ | let i=i+1</code> to replace the numbers with increments starting from 1</li>
</ul>
</li>
<li>Meeting with Marie Angelique, Abenet, Sarа, аnd Margarita to continue the discussion about Types from last week
<ul>
<li>We made progress with concrete actions and will continue next week</li>
</ul>
</li>
</ul>
<h2id="2022-09-08">2022-09-08</h2>
<ul>
<li>I had a meeting with Nicky from UNEP to discuss issues they are having with their DSpace
<ul>
<li>I told her about the meeting of DSpace community people that we’re planning at ILRI in the next few weeks</li>
</ul>
</li>
</ul>
<h2id="2022-09-09">2022-09-09</h2>
<ul>
<li>Add some value mappings to AReS because I see a lot of incorrect regions and countries</li>
<li>I also found some values that were blank in CGSpace so I deleted them:</li>
</span></span><spanstyle="display:flex;"><span>dspace=# DELETE FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND text_value='';
<li>The top is still Google, but all the requests are HTTP 503 because I classified them as bots for XMLUI at least</li>
<li>Then there’s 80.248.237.167, which is using a normal user agent and scraping Discovery
<ul>
<li>That IP is on Internet Vikings aka Internetbolaget and we are already marking that subnet as ‘bot’ for XMLUI so most of these requests are HTTP 503</li>
</ul>
</li>
<li>On another note, I’m curious to explore enabling caching of certain REST API responses
<ul>
<li>For example, where the use is for harvesting rather than actual clients getting bitstreams or thumbnails, it seems there might be a benefit to speeding these up for subsequent requestors:</li>
<li>I specifically have to not cache things like requests for bitstreams because those are from actual users and we need to keep the real requests so we get the statistics hit
<ul>
<li>Will be interesting to check the results above as the day goes on (now 10AM)</li>
<li>To estimate the potential savings from caching I will check how many non-bitstream requests are made versus how many are made more than once (updated the next morning using yesterday’s log):</li>
<li>In the afternoon I started a harvest on AReS (which should affect the numbers above also)</li>
<li>I enabled an nginx proxy cache on DSpace Test for this location regex: <code>location ~ /rest/(handle|items|collections|communities)/.+</code></li>
</ul>
<h2id="2022-09-12">2022-09-12</h2>
<ul>
<li>I am testing harvesting DSpace Test via AReS with the nginx proxy cache enabled
<ul>
<li>I had to tune the regular expression in nginx a bit because the REST requests OpenRXV uses weren’t matching</li>
<li>Now I’m trying this one: <code>/rest/(handle|items|collections|communities)/?</code></li>
<li>Testing in <ahref="https://regex101.com/r/vPz11y/1">regex101.com</a> with this test string:</li>
<li>I estimate that it will take about 1GB of cache to harvest 100,000 items from CGSpace with OpenRXV (10,000 pages)</li>
<li>Basically all but 4 and 5 (bitstreams) should match</li>
<li>Upload 682 OICRs from MARLO to CGSpace
<ul>
<li>We had tested these on DSpace Test last month along with the MELIAs, Policies, and Innovations, but we decided to upload the OICRs first so that other things can link against them as related items</li>
</ul>
</li>
</ul>
<h2id="2022-09-14">2022-09-14</h2>
<ul>
<li>Meeting with Peter, Abenet, Indira, and Michael about CGSpace rollout plan for the Initiatives</li>
</ul>
<h2id="2022-09-16">2022-09-16</h2>
<ul>
<li>Meeting with Marie-Angeqlique, Abenet, Margarita, and Sara about types for CG Core
<ul>
<li>We are about halfway through the list of types now, with concrete actions for CG Core and CGSpace</li>
<li>We will meet next in two weeks to hopefully finalize the list, then we can move on to definitions</li>
</ul>
</li>
</ul>
<h2id="2022-09-18">2022-09-18</h2>
<ul>
<li>Deploy the <code>org.apache.cocoon.uploads.autosave=false</code> change on CGSpace</li>
<li>Start a harvest on AReS</li>
</ul>
<h2id="2022-09-19">2022-09-19</h2>
<ul>
<li>Deploy the nginx proxy cache for /rest requests on CGSpace
<ul>
<li>I had tested this last week on DSpace Test</li>
<li>By my counts on CGSpace yesterday (Sunday, a busy day for the REST API), we had 5,654 URLs that were requested more than twice, and it tails off after that towards two, three, four, etc:</li>
<li>For now I guess requests that were done two or three times by different clients will be cached and that’s a win, and I expect more and more REST API activity soon when initiatives and One CGIAR stuff picks up</li>
</ul>
<h2id="2022-09-20">2022-09-20</h2>
<ul>
<li>I checked the status of the nginx REST API cache on CGSpace and it was stuck at 7,083 items for hours:</li>
</ul>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span># find /var/cache/nginx/rest_cache/ -type f | wc -l
<li>The proxy cache key zone is currently 1m, which can store ~8,000 keys, so that could be what we’re running into
<ul>
<li>I increased it to 2m and will keep monitoring it</li>
</ul>
</li>
<li>CIP webmaster contacted me to say they are having problems harvesting CGSpace from their WordPress
<ul>
<li>I am not sure if there are issues due to the REST API caching I enabled…</li>
</ul>
</li>
</ul>
<h2id="2022-09-21">2022-09-21</h2>
<ul>
<li>Planning the Nairobi DSpace Users meeting with Abenet</li>
<li>Planning to have a call about MEL submitting to CGSpace on Monday with Mohammed Salem
<ul>
<li>I created two collections on DSpace Test: one with a workflow, and one without</li>
<li>According to my notes from <ahref="/cgspace-notes/2020-10/">2020-10</a> the account must be in the admin group in order to submit via the REST API, so I added it to the admin group of each collection</li>
</ul>
</li>
</ul>
<h2id="2022-09-22">2022-09-22</h2>
<ul>
<li>Nairobi DSpace users meeting at ILRI</li>
<li>I found a few users that didn’t have ORCID iDs and were missing tags on CGSpace so I tagged them:</li>
<li>Some of these authors were not in the controlled vocabulary so I added them</li>
</ul>
</li>
</ul>
<h2id="2022-09-23">2022-09-23</h2>
<ul>
<li>Tag some more ORCID metdata (amended above)</li>
<li>Meeting with Peter and Abenet to discuss CGSpace issues
<ul>
<li>We found a workable solution to the MEL submission issue: they can submit to a dedicated MEL-only collection with no workflow and we will map or move the items after</li>
</ul>
</li>
<li>Pascal says that they have made a <ahref="https://github.com/DSpace/DSpace/pull/8415">pull request for their duplicate checker on DSpace 7</a> yayyyy</li>
</ul>
<h2id="2022-09-24">2022-09-24</h2>
<ul>
<li>Found some more ORCID identifiers to tag so I added them to the list above</li>
<li>Start a harvest on AReS around 8PM on Saturday night</li>
</ul>
<h2id="2022-09-25">2022-09-25</h2>
<ul>
<li>The harvest on AReS finished and now the load on CGSpace server is still high like always on Sunday mornings
<ul>
<li>UptimeRobot says it’s down sigh…</li>
</ul>
</li>
<li>I had an idea to include the HTTP Accept header in the nginx proxy cache key to fix the issue we had with CIP last week
<li>This effectively makes our cache half as effective, but hopefully as more people start harvesting the number of requests handled by it will go up</li>
<li>I will enable this on CGSpace and email Moises from CIP to check if their harvester is working</li>
</ul>
<h2id="2022-09-26">2022-09-26</h2>
<ul>
<li>Update welcome text on CGSpace after our meeting last week</li>
<li>I found another dozen or so ORCIDs for top authors on ILRI’s community on CGSpace and tagged them (~1,100 more metadata fields)</li>
<li>Last week we discussed moving <code>cg.identifier.googleurl</code> to <code>cg.identifier.url</code> since there is no need to treat Google Books URLs specially anymore as far as we know
<ul>
<li>I made the changes to the submission form and the XMLUI item displays, then moved all existing metadata in PostgreSQL:</li>
</ul>
</li>
</ul>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span>dspace= ☘ UPDATE metadatavalue SET metadata_field_id=219 WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=222;
<li>Then I deleted <code>cg.identifier.googleurl</code> from the metadata registry</li>
<li>Meeting with Salem, Svetlana, Valentina, and Abenet about MEL depositing to CGSpace for the initiatives
<ul>
<li>Submitting to a collection without a workflow works as expected, and we can even select another collection (with a workflow) to map the item to from the MEL submission</li>
<li>The three minor issues we found were:
<ul>
<li>MEL still doesn’t send the bitstream</li>
<li>MEL sends metadata with a download URL on mel.cgiar.org</li>
<li>MEL sends a JPEG that says “no thumbnail” when an item doesn’t have a thumbnail</li>
</ul>
</li>
<li>I still need to send feedback to the group</li>
</ul>
</li>
</ul>
<h2id="2022-09-27">2022-09-27</h2>
<ul>
<li>Find a few more ORCID identifiers missing for ILRI authors and add them to the controlled vocabulary and tag the authors on CGSpace</li>
<li>Moises from CIP says the WordPress importer worked fine with the current nginx proxy cache settings so it seems adding the HTTP Accept header to the cache key worked</li>
<li>Update my DSpace 7 environments to 7.4-SNAPSHOT
<ul>
<li>I see they have added thumbnails in some places now</li>
<li>Oh nice, they also added the “recent submissions” to the home page</li>
</ul>
</li>
<li>While talking with Salem about the MEL depositing to CGSpace we discovered an issue with HTTP DELETE on <code>/items/{item id}/bitstreams/{bitstream id}</code> or <code>/bitstreams/{bitstream id}</code>
<ul>
<li>DSpace removes the bitstream but keeps the empty <code>THUMBNAIL</code> bundle, which breaks the display in XMLUI</li>
</ul>
</li>
<li>Meeting with Enrico et al about PRMS reporting for the initiatives</li>
</ul>
<h2id="2022-09-28">2022-09-28</h2>
<ul>
<li>I was reading the source code for DSpace 6’s REST API and found that it’s <ahref="https://github.com/DSpace/DSpace/blob/dspace-6.4/dspace-rest/src/main/java/org/dspace/rest/ItemsResource.java#L427">not possible to specify a bundle while POSTing a bitstream</a>
<ul>
<li>I asked Salem how they do it on MEL and he said they pretend to be a human and do it via XMLUI!</li>
</ul>
</li>
<li>I added a few new ILRI subjects to the input forms on CGSpace
<ul>
<li>Both “bushmeat” and “wildlife conservation” are AGROVOC terms, but “wild meat” is not</li>
<li>The distinction ILRI would like to start making is:</li>
</ul>
</li>
</ul>
<blockquote>
<p>Meat comes from any animal, and when at ILRI we specifically make
reference to it in the context of livestock. However the word bushmeat
refers to illegal harvesting of meat. wild meat is being used as legal
harvesting of meat from wildlife and not from livestock.</p>
</blockquote>
<ul>
<li>I added a few more CGIAR authors ORCID identifiers to our controlled vocabulary and tagged them on CGSpace (~450 more metadata fields)</li>
<li>Talking to Salem about ORCID identifiers, we compared list and they have a bunch that we don’t have:</li>
<li>I wrote a script <code>update-orcids.py</code> to read a list of names and ORCID identifiers and update existing metadata in the database to the latest name format</li>
<li>Also on that note I’m trying to implement a workaround for a potential caching issue that causes MEL to not be able to update items on DSpace Test
<ul>
<li>I <em>think</em> we might need to allow requests with a JSESSIONID to bypass the cache, but I have to verify with Salem</li>
<li>We can do this with an nginx map:</li>
</ul>
</li>
</ul>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span># Check <spanstyle="color:#66d9ef">if</span> the JSESSIONID cookie is present and contains a 32-character hex
</span></span><spanstyle="display:flex;"><span># value, which would mean that a user is actively attempting to re-use their
</span></span><spanstyle="display:flex;"><span># Tomcat session. Then we set the $active_user_session variable and use it
</span></span><spanstyle="display:flex;"><span># to bypass the nginx proxy cache in REST requests.
<li>Then in the location block where we do the proxy cache:</li>
</ul>
<divclass="highlight"><pretabindex="0"style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><codeclass="language-console"data-lang="console"><spanstyle="display:flex;"><span> # Don't cache when user Shift-refreshes (Cache-Control: no-cache) or
</span></span><spanstyle="display:flex;"><span> # when a client has an active session (see the $cookie_jsessionid map).
</span></span></span><spanstyle="display:flex;"><span><spanstyle="color:#960050;background-color:#1e0010"></span>Total number of bot hits purged: 33027
</span></span></code></pre></div><ul>
<li>So it seems we’ve seen this bot before and the total number is much higher than the 10,000 this month</li>
<li>I had a call with Salem and we verified that the nginx cache bypass for clients who provide a JSESSIONID fixes their issue with updating items/bitstreams from MEL
<ul>
<li>The issue was that they delete all metadata and bitstreams, then add them again to make sure everything is up to date, and in that process they also re-request the item with all expands to get the bitstreams, which ends up getting cached and then they try to delete the old bitstream</li>
</ul>
</li>
<li>I also noticed that someone made a <ahref="https://github.com/DSpace/DSpace/pull/8343">pull request to enable POSTing bitstreams to a particular bundle</a> and it works, so that’s awesome!</li>
</ul>
<h2id="2022-09-30">2022-09-30</h2>
<ul>
<li>I applied <ahref="https://github.com/DSpace/DSpace/pull/8343">the patch for POSTing bitstreams to other bundles</a> on CGSpace</li>
<li>Testing a few other DSpace 6.4 patches on DSpace Test:
<ul>
<li><ahref="https://github.com/DSpace/DSpace/pull/1901">DS-3791 Make sure the “yearDifference” takes into account that a gap of 10 year contains 11 years</a></li>
<li><ahref="https://github.com/DSpace/DSpace/pull/2501">DS-3873 Limit the usage of PDFBoxThumbnail to PDFs</a></li>
