CGSpace Notes

Documenting day-to-day work on the CGSpace repository.

January, 2024

in  Notes

2024-01-02

  • Work on preparation of new server for DSpace 7 migration
    • I’m not quite sure what we need to do for the Handle server
    • For now I just ran the dspace make-handle-config script and diffed it with the one from DSpace 6
    • I sent the bundle to the Handle admins to make sure it’s OK before we do the migration
  • Continue testing and debugging the cgspace-java-helpers on DSpace 7
  • Work on IFPRI ISNAR archive cleanup

2024-01-03

  • I haven’t heard from the Handle admins so I’m preparing a backup solution using nginx streams
  • This seems to work in my simple tests (this must be outside the http {} block):
stream {
    upstream handle_tcp_9000 {
       server 188.34.177.10:9000;
    }

    server {
        listen 9000;
        proxy_connect_timeout 1s;
        proxy_timeout 3s;
        proxy_pass handle_tcp_9000;
    }
}
  • Here I forwarded a test TCP port 9000 from one server to another and was able to retrieve a test HTML that was running on the target
    • I will have to do TCP and UDP on port 2641, and TCP/HTTP on port 8000.
  • I did some more minor work on the IFPRI ISNAR archive
    • I got some PDFs from the UMN AgEcon search and fixed some metadata
    • Then I did some duplicate checking and found five items already on CGSpace

2024-01-04

  • Upload 692 items for the ISNAR archive to CGSpace: https://cgspace.cgiar.org/handle/10568/136192
  • Help Peter proof and upload 252 items from the 2023 Gender conference to CGSpace
  • Meeting with IFPRI to discuss their migration to CGSpace
    • We agreed to add two new fields, one for IFPRI project and one for IFPRI publication ranking
    • Most likely we will use cg.identifier.project as a general field and consolidate other project fields there
    • Not sure which field to use for the publication rank…

2024-01-05

  • Proof and upload 51 items in bulk for IFPRI
  • I did a big cleanup of user groups in anticipation of complaints about slow workflow tasks etc in DSpace 7
    • I removed ILRI editors from all the dozens of CCAFS community and collection groups, and I should do the same for other CRPs since they are closed for two years now

2024-01-06

  • Migrate CGSpace to DSpace 7

2024-01-07

  • High load on the server and UptimeRobot saying the frontend is flapping
    • I noticed tons of logs from pm2 in the systemd journal, so I disabled those in the systemd unit because they are available from pm2’s log directory anyway
    • I also noticed the same for Solr, so I disabled stdout for that systemd unit as well
  • I spent a lot of time bringing back the nginx rate limits we used in DSpace 6 and it seems to have helped
  • I see some client doing weird HEAD requests to search pages:
47.76.35.19 - - [07/Jan/2024:00:00:02 +0100] "HEAD /search/?f.accessRights=Open+Access%2Cequals&f.actionArea=Resilient+Agrifood+Systems%2Cequals&f.author=Burkart%2C+Stefan%2Cequals&f.country=Kenya%2Cequals&f.impactArea=Climate+adaptation+and+mitigation%2Cequals&f.itemtype=Brief%2Cequals&f.publisher=CGIAR+System+Organization%2Cequals&f.region=Asia%2Cequals&f.sdg=SDG+12+-+Responsible+consumption+and+production%2Cequals&f.sponsorship=CGIAR+Trust+Fund%2Cequals&f.subject=environmental+factors%2Cequals&spc.page=1 HTTP/1.1" 499 0 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.2504.63 Safari/537.36"
  • I will add their network blocks (AS45102) and regenerate my list of bot networks:
$ wget https://asn.ipinfo.app/api/text/list/AS16276 \
     https://asn.ipinfo.app/api/text/list/AS23576 \
     https://asn.ipinfo.app/api/text/list/AS24940 \
     https://asn.ipinfo.app/api/text/list/AS13238 \
     https://asn.ipinfo.app/api/text/list/AS14061 \
     https://asn.ipinfo.app/api/text/list/AS12876 \
     https://asn.ipinfo.app/api/text/list/AS55286 \
     https://asn.ipinfo.app/api/text/list/AS203020 \
     https://asn.ipinfo.app/api/text/list/AS204287 \
     https://asn.ipinfo.app/api/text/list/AS50245 \
     https://asn.ipinfo.app/api/text/list/AS6939 \
     https://asn.ipinfo.app/api/text/list/AS45102 \
     https://asn.ipinfo.app/api/text/list/AS21859
$ cat AS* | sort | uniq | wc -l
4897
$ cat AS* | ~/go/bin/mapcidr -a > /tmp/networks.txt
$ wc -l /tmp/networks.txt
2017 /tmp/networks.txt
  • I’m surprised to see the number of networks reduced from my current ones… hmmm.
  • I will also update my list of Bing networks:
$ ./ilri/bing-networks-to-ips.sh
$ ~/go/bin/mapcidr -a < /tmp/bing-ips.txt > /tmp/bing-networks.txt
$ wc -l /tmp/bing-networks.txt
250 /tmp/bing-networks.txt