- Linode sent an alert that CGSpace (linode18) was using 350% CPU for the past two hours
- I looked in the Activity pane of the Admin Control Panel and it seems that Google, Baidu, Yahoo, and Bing are all crawling with massive numbers of bots concurrently (~100 total, mostly Baidu and Google)
- The good thing is that, according to `dspace.log.2017-08-01`, they are all using the same Tomcat session
- This means our Tomcat Crawler Session Valve is working
- But many of the bots are browsing dynamic URLs like:
- /handle/10568/3353/discover
- /handle/10568/16510/browse
- The `robots.txt` only blocks the top-level `/discover` and `/browse` URLs... we will need to find a way to forbid them from accessing these!
- Relevant issue from DSpace Jira (semi resolved in DSpace 6.0): https://jira.duraspace.org/browse/DS-2962
- It turns out that we're already adding the `X-Robots-Tag "none"` HTTP header, but this only forbids the search engine from _indexing_ the page, not crawling it!
- Also, the bot has to successfully browse the page first so it can receive the HTTP header...
- Magdalena from CCAFS asked if there was a way to get the top ten items published in 2016 (note: not the top items in 2016!)
- I think Atmire's Content and Usage Analysis module should be able to do this but I will have to look at the configuration and maybe email Atmire if I can't figure it out
- I had a look at the moduel configuration and couldn't figure out a way to do this, so I [opened a ticket on the Atmire tracker](https://tracker.atmire.com/tickets-cgiar-ilri/view-tickets)
- Atmire responded about the [missing workflow statistics issue](https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=500) a few weeks ago but I didn't see it for some reason
- They said they added a publication and saw the workflow stat for the user, so I should try again and let them know
- Usman from CIFOR emailed to ask about the status of our OAI tests for harvesting their DSpace repository
- I told him that the OAI appears to not be harvesting properly after the first sync, and that the control panel shows an "Internal error" for that collection:
- I don't see anything related in our logs, so I asked him to check for our server's IP in their logs
- Also, in the mean time I stopped the harvesting process, reset the status, and restarted the process via the Admin control panel (note: I didn't reset the collection, just the harvester status!)
- Apply Abenet's corrections for the CGIAR Library's Consortium subcommunity (697 records)
- I had to fix a few small things, like moving the `dc.title` column away from the beginning of the row, delete blank spaces in the abstract in vim using `:g/^$/d`, add the `dc.subject[en_US]` column back, as she had deleted it and DSpace didn't detect the changes made there (we needed to blank the values instead)
- Apply last updates to the CGIAR Library's Fund community (812 items)
- Had to do some quality checks and column renames before importing, as either Sisay or Abenet renamed a few columns and the metadata importer wanted to remove/add new metadata for title, abstract, etc.
- Also I applied the HTML entities unescape transform on the abstract column in Open Refine
- I need to get an author list from the database for only the CGIAR Library community to send to Peter
- Alan to follow up with ICARDA about depositing in CGSpace, we want ICARD and Drylands legacy content but not duplicates
- Alan to follow up on dc.rights, where are we?
- Alan to follow up with Atmire about a dedicated field for ORCIDs, based on the discussion in the [June, 2017 DCAT meeting](https://wiki.duraspace.org/display/cmtygp/DCAT+Meeting+June+2017)
- Alan to ask about how to query external services like AGROVOC in the DSpace submission form
- CGSpace had load issues and was throwing errors related to PostgreSQL
- I told Tsega to reduce the max connections from 70 to 40 because actually each web application gets that limit and so for xmlui, oai, jspui, rest, etc it could be 70 x 4 = 280 connections depending on the load, and the PostgreSQL config itself is only 100!
- I learned this on a recent discussion on the DSpace wiki
- I need to either look into setting up a database pool through JNDI or increase the PostgreSQL max connections
- Also, I need to find out where the load is coming from (rest?) and possibly block bots from accessing dynamic pages like Browse and Discover instead of just sending an X-Robots-Tag HTTP header
- I noticed that Google has bitstreams from the `rest` interface in the search index. I need to ask on the dspace-tech mailing list to see what other people are doing about this, and maybe start issuing an `X-Robots-Tag: none` there!
- The top offender is 70.32.83.92 which is actually the same IP as ccafs.cgiar.org, so I will email the Macaroni Bros to see if they can test on DSpace Test instead
- I've enabled logging of `/oai` requests on nginx as well so we can potentially determine bad actors here (also to see if anyone is actually using OAI!)
- Thinking about resource limits for PostgreSQL again after last week's CGSpace crash and related to a recently discussion I had in the comments of the [April, 2017 DCAT meeting notes](https://wiki.duraspace.org/display/cmtygp/DCAT+Meeting+April+2017)
- In that thread Chris Wilper suggests a new default of 35 max connections for `db.maxconnections` (from the current default of 30), knowing that _each DSpace web application_ gets to use up to this many on its own
- It would be good to approximate what the theoretical maximum number of connections on a busy server would be, perhaps by looking to see which apps use SQL:
```
$ grep -rsI SQLException dspace-jspui | wc -l
473
$ grep -rsI SQLException dspace-oai | wc -l
63
$ grep -rsI SQLException dspace-rest | wc -l
139
$ grep -rsI SQLException dspace-solr | wc -l
0
$ grep -rsI SQLException dspace-xmlui | wc -l
866
```
- Of those five applications we're running, only `solr` appears not to use the database directly
- And JSPUI is only used internally (so it doesn't really count), leaving us with OAI, REST, and XMLUI
- Assuming each takes a theoretical maximum of 35 connections during a heavy load (35 * 3 = 105), that would put the connections well above PostgreSQL's default max of 100 connections (remember a handful of connections are reserved for the PostgreSQL super user, see `superuser_reserved_connections`)
- So we should adjust PostgreSQL's max connections to be DSpace's `db.maxconnections` * 3 + 3
- This would allow each application to use up to `db.maxconnections` and not to go over the system's PostgreSQL limit
- Perhaps since CGSpace is a busy site with lots of resources we could actually use something like 40 for `db.maxconnections`
- Also worth looking into is to set up a database pool using JNDI, as apparently DSpace's `db.poolname` hasn't been used since around DSpace 1.7 (according to Chris Wilper's comments in the thread)
- Need to go check the PostgreSQL connection stats in Munin on CGSpace from the past week to get an idea if 40 is appropriate
- Unfortunately I don't have the breakdown of which DSpace apps are making those connections (I'll assume XMLUI)
- So I guess a limit of 30 (DSpace default) is too low, but 70 causes problems when the load increases and the system's PostgreSQL `max_connections` is too low
- For now I think maybe setting DSpace's `db.maxconnections` to 40 and adjusting the system's `max_connections` might be a good starting point: 40 * 3 + 4 = 123
- I wanted to merge the various field variations like `cg.subject.system` and `cg.subject.system[en_US]` in OpenRefine but I realized it would be easier in PostgreSQL:
```
dspace=# select distinct text_value, text_lang from metadatavalue where resource_type_id=2 and metadata_field_id=254;
```
- And actually, we can do it for other generic fields for items in those collections, for example `dc.description.abstract`:
- And on others like `dc.language.iso`, `dc.relation.ispartofseries`, `dc.type`, `dc.title`, etc...
- Also, to move fields from `dc.identifier.url` to `cg.identifier.url[en_US]` (because we don't use the Dublin Core one for some reason):
```
dspace=# update metadatavalue set metadata_field_id = 219, text_lang = 'en_US' where resource_type_id = 2 AND metadata_field_id = 237;
UPDATE 15
```
- Set the text_lang of all `dc.identifier.uri` (Handle) fields to be NULL, just like default DSpace does:
```
dspace=# update metadatavalue set text_lang=NULL where resource_type_id = 2 and metadata_field_id = 25 and text_value like 'http://hdl.handle.net/10947/%';
UPDATE 4248
```
- Also update the text_lang of `dc.contributor.author` fields for metadata in these collections:
- Unfortunately when you fix these in OpenRefine and then submit the metadata to DSpace it doesn't detect any changes, so you have to edit them all manually via DSpace's "Edit Item"
- Ooh! And an even more interesting regex would match _any_ duplicated author:
```
isNotNull(value.match(/(.+?)\|\|\1/))
```
- Which means it can also be used to find items with duplicate `dc.subject` fields...
- Finally sent Peter the final dump of the CGIAR System Organization community so he can have a last look at it
