From 7916af54177c76915dedaf4fe520fb07641dd42c Mon Sep 17 00:00:00 2001 From: Alan Orth Date: Fri, 13 May 2022 10:28:23 +0300 Subject: [PATCH] Add notes for 2022-05-13 --- content/posts/2022-05.md | 22 ++++++++++++++++++++- docs/2022-05/index.html | 26 ++++++++++++++++++++----- docs/categories/index.html | 2 +- docs/categories/notes/index.html | 2 +- docs/categories/notes/page/2/index.html | 2 +- docs/categories/notes/page/3/index.html | 2 +- docs/categories/notes/page/4/index.html | 2 +- docs/categories/notes/page/5/index.html | 2 +- docs/categories/notes/page/6/index.html | 2 +- docs/index.html | 2 +- docs/page/2/index.html | 2 +- docs/page/3/index.html | 2 +- docs/page/4/index.html | 2 +- docs/page/5/index.html | 2 +- docs/page/6/index.html | 2 +- docs/page/7/index.html | 2 +- docs/page/8/index.html | 2 +- docs/page/9/index.html | 2 +- docs/posts/index.html | 2 +- docs/posts/page/2/index.html | 2 +- docs/posts/page/3/index.html | 2 +- docs/posts/page/4/index.html | 2 +- docs/posts/page/5/index.html | 2 +- docs/posts/page/6/index.html | 2 +- docs/posts/page/7/index.html | 2 +- docs/posts/page/8/index.html | 2 +- docs/posts/page/9/index.html | 2 +- docs/sitemap.xml | 10 +++++----- 28 files changed, 72 insertions(+), 36 deletions(-) diff --git a/content/posts/2022-05.md b/content/posts/2022-05.md index e87d7b285..e468ed79a 100644 --- a/content/posts/2022-05.md +++ b/content/posts/2022-05.md @@ -102,7 +102,7 @@ localhost/dspacetest= ☘ SELECT EXTRACT(year from TO_DATE(text_value, 'YYYY-MM- - We discussed the future of CGSpace and DSpace in general in the new One CGIAR - We discussed how to prepare for bringing in content from the Initiatives, whether we need new metadata fields to support people from IFPRI etc - We discussed the need for good quality Drupal and WordPress modules so sites can harvest content from the repository - - Peter asked me to send him a list of investors/funders/donors so he can clean it up, but also to try to align it with RoR and evntually do something like we do with country codes, adding the RoR IDs and potentially showing the badge on item views + - Peter asked me to send him a list of investors/funders/donors so he can clean it up, but also to try to align it with ROR and evntually do something like we do with country codes, adding the ROR IDs and potentially showing the badge on item views - We also discussed removing some Mirage 2 themes for old programs and CRPs that don't have custom branding, ie only Google Analytics - Export a list of donors for Peter to clean up: @@ -135,4 +135,24 @@ $ grep -c "Adding ORCID" /tmp/add-orcids.log # zcat --force /var/log/nginx/access.log.* | grep 'GET /explorer' | goaccess --log-format=COMBINED - -o /tmp/ares_report.html ``` +## 2022-05-13 + +- Peter finalized the corrections on donors from yesterday so I extracted them into fix/delete CSVs and ran them on CGSpace: + +```console +$ ./ilri/fix-metadata-values.py -i 2022-05-13-fix-CGSpace-Donors.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.donor -m 248 -t correct -d +$ ./ilri/delete-metadata-values.py -i 2022-05-13-delete-CGSpace-Donors.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.donor -m 248 -d +``` + +- I cleaned up a few records manually (like some that had \r\n) then re-exported the donors and checked against the latest ROR dump: + +```console +$ ./ilri/ror-lookup.py -i /tmp/2022-05-13-donors.csv -r v1.0-2022-03-17-ror-data.json -o /tmp/2022-05-13-ror.csv +$ csvgrep -c matched -m true /tmp/2022-05-13-ror.csv | wc -l +230 +$ csvgrep -c matched -m false /tmp/2022-05-13-ror.csv | csvcut -c organization > /tmp/2022-05-13-ror-unmatched.csv +``` + +- Then I sent Peter a list so he can try to update some from ROR + diff --git a/docs/2022-05/index.html b/docs/2022-05/index.html index f93c70757..f476d52ce 100644 --- a/docs/2022-05/index.html +++ b/docs/2022-05/index.html @@ -35,7 +35,7 @@ I purged 93,974 hits from these IPs using my check-spider-ip-hits.sh script - + @@ -76,9 +76,9 @@ I purged 93,974 hits from these IPs using my check-spider-ip-hits.sh script "@type": "BlogPosting", "headline": "May, 2022", "url": "https://alanorth.github.io/cgspace-notes/2022-05/", - "wordCount": "947", + "wordCount": "1070", "datePublished": "2022-05-04T09:13:39+03:00", - "dateModified": "2022-05-10T16:35:50+03:00", + "dateModified": "2022-05-13T08:39:15+03:00", "author": { "@type": "Person", "name": "Alan Orth" @@ -269,7 +269,7 @@ I purged 93,974 hits from these IPs using my check-spider-ip-hits.sh script
  • We discussed the future of CGSpace and DSpace in general in the new One CGIAR
  • We discussed how to prepare for bringing in content from the Initiatives, whether we need new metadata fields to support people from IFPRI etc
  • We discussed the need for good quality Drupal and WordPress modules so sites can harvest content from the repository
    • -
  • Peter asked me to send him a list of investors/funders/donors so he can clean it up, but also to try to align it with RoR and evntually do something like we do with country codes, adding the RoR IDs and potentially showing the badge on item views
    • +
  • Peter asked me to send him a list of investors/funders/donors so he can clean it up, but also to try to align it with ROR and evntually do something like we do with country codes, adding the ROR IDs and potentially showing the badge on item views
  • We also discussed removing some Mirage 2 themes for old programs and CRPs that don’t have custom branding, ie only Google Analytics
    • @@ -300,7 +300,23 @@ I purged 93,974 hits from these IPs using my check-spider-ip-hits.sh script
  • I generated a report from the AReS nginx logs on linode18:
    • # zcat --force /var/log/nginx/access.log.* | grep 'GET /explorer' | goaccess --log-format=COMBINED - -o /tmp/ares_report.html
-
    +

    2022-05-13

    + +
    $ ./ilri/fix-metadata-values.py -i 2022-05-13-fix-CGSpace-Donors.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.donor -m 248 -t correct -d
+$ ./ilri/delete-metadata-values.py -i 2022-05-13-delete-CGSpace-Donors.csv -db dspace -u dspace -p 'fuuu' -f cg.contributor.donor -m 248 -d
+
    +
    $ ./ilri/ror-lookup.py -i /tmp/2022-05-13-donors.csv -r v1.0-2022-03-17-ror-data.json -o /tmp/2022-05-13-ror.csv
+$ csvgrep -c matched -m true /tmp/2022-05-13-ror.csv | wc -l
+230
+$ csvgrep -c matched -m false /tmp/2022-05-13-ror.csv | csvcut -c organization > /tmp/2022-05-13-ror-unmatched.csv
+
    + diff --git a/docs/categories/index.html b/docs/categories/index.html index 59402a126..e19cf8e95 100644 --- a/docs/categories/index.html +++ b/docs/categories/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/index.html b/docs/categories/notes/index.html index f77e20a31..738176431 100644 --- a/docs/categories/notes/index.html +++ b/docs/categories/notes/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/2/index.html b/docs/categories/notes/page/2/index.html index c44c3b677..632ed48e1 100644 --- a/docs/categories/notes/page/2/index.html +++ b/docs/categories/notes/page/2/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/3/index.html b/docs/categories/notes/page/3/index.html index aa0b8e515..f3f3f5f2d 100644 --- a/docs/categories/notes/page/3/index.html +++ b/docs/categories/notes/page/3/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/4/index.html b/docs/categories/notes/page/4/index.html index b2b9cbac1..288518cbd 100644 --- a/docs/categories/notes/page/4/index.html +++ b/docs/categories/notes/page/4/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/5/index.html b/docs/categories/notes/page/5/index.html index 1e952fb58..8929047fd 100644 --- a/docs/categories/notes/page/5/index.html +++ b/docs/categories/notes/page/5/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/categories/notes/page/6/index.html b/docs/categories/notes/page/6/index.html index 697f2932a..305958a36 100644 --- a/docs/categories/notes/page/6/index.html +++ b/docs/categories/notes/page/6/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/index.html b/docs/index.html index aade452cd..0f984a203 100644 --- a/docs/index.html +++ b/docs/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/2/index.html b/docs/page/2/index.html index 350f5846e..9b787e0ec 100644 --- a/docs/page/2/index.html +++ b/docs/page/2/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/3/index.html b/docs/page/3/index.html index 2f0dc8803..dd7b31e2a 100644 --- a/docs/page/3/index.html +++ b/docs/page/3/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/4/index.html b/docs/page/4/index.html index 87c511c0d..508a0909d 100644 --- a/docs/page/4/index.html +++ b/docs/page/4/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/5/index.html b/docs/page/5/index.html index 8e014cc9d..77c03617c 100644 --- a/docs/page/5/index.html +++ b/docs/page/5/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/6/index.html b/docs/page/6/index.html index 215e74a44..d2d79c1c5 100644 --- a/docs/page/6/index.html +++ b/docs/page/6/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/7/index.html b/docs/page/7/index.html index 00da4ea35..60dc6848a 100644 --- a/docs/page/7/index.html +++ b/docs/page/7/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/8/index.html b/docs/page/8/index.html index 94afe79b3..41a6693f3 100644 --- a/docs/page/8/index.html +++ b/docs/page/8/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/page/9/index.html b/docs/page/9/index.html index afc9ba68b..5d39eae37 100644 --- a/docs/page/9/index.html +++ b/docs/page/9/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/index.html b/docs/posts/index.html index 59ad55b16..d4d81f903 100644 --- a/docs/posts/index.html +++ b/docs/posts/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/2/index.html b/docs/posts/page/2/index.html index c6741a771..09f7fd366 100644 --- a/docs/posts/page/2/index.html +++ b/docs/posts/page/2/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/3/index.html b/docs/posts/page/3/index.html index 09bfe6979..67fd52ac1 100644 --- a/docs/posts/page/3/index.html +++ b/docs/posts/page/3/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/4/index.html b/docs/posts/page/4/index.html index 7814164b1..f1fab8e51 100644 --- a/docs/posts/page/4/index.html +++ b/docs/posts/page/4/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/5/index.html b/docs/posts/page/5/index.html index 56b64b855..d6dcb3f5b 100644 --- a/docs/posts/page/5/index.html +++ b/docs/posts/page/5/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/6/index.html b/docs/posts/page/6/index.html index 96b3bd433..bc432f17b 100644 --- a/docs/posts/page/6/index.html +++ b/docs/posts/page/6/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/7/index.html b/docs/posts/page/7/index.html index d1e03257a..c7c480a32 100644 --- a/docs/posts/page/7/index.html +++ b/docs/posts/page/7/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/8/index.html b/docs/posts/page/8/index.html index b33d15de7..157dfafe0 100644 --- a/docs/posts/page/8/index.html +++ b/docs/posts/page/8/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/posts/page/9/index.html b/docs/posts/page/9/index.html index 697f3b4c8..eb2a24c03 100644 --- a/docs/posts/page/9/index.html +++ b/docs/posts/page/9/index.html @@ -10,7 +10,7 @@ - + diff --git a/docs/sitemap.xml b/docs/sitemap.xml index 791cc1e0c..eac5af829 100644 --- a/docs/sitemap.xml +++ b/docs/sitemap.xml @@ -3,19 +3,19 @@ xmlns:xhtml="http://www.w3.org/1999/xhtml"> https://alanorth.github.io/cgspace-notes/categories/ - 2022-05-12T12:51:45+03:00 + 2022-05-13T08:39:15+03:00 https://alanorth.github.io/cgspace-notes/ - 2022-05-12T12:51:45+03:00 + 2022-05-13T08:39:15+03:00 https://alanorth.github.io/cgspace-notes/2022-05/ - 2022-05-10T16:35:50+03:00 + 2022-05-13T08:39:15+03:00 https://alanorth.github.io/cgspace-notes/categories/notes/ - 2022-05-12T12:51:45+03:00 + 2022-05-13T08:39:15+03:00 https://alanorth.github.io/cgspace-notes/posts/ - 2022-05-12T12:51:45+03:00 + 2022-05-13T08:39:15+03:00 https://alanorth.github.io/cgspace-notes/2022-04/ 2022-05-04T11:09:45+03:00