alanorth/cgspace-notes
1
0
Fork 0
mirror of https://github.com/alanorth/cgspace-notes.git synced 2024-07-01 02:23:51 +02:00
Code Issues Projects Releases Wiki Activity
c49ccb8b20
cgspace-notes/content/post/2017-11.md

123 lines
5.6 KiB
Markdown
Raw Normal View History

Add notes for 2017-11-02
2017-11-02 08:37:59 +01:00
---
title: "November, 2017"
date: 2017-11-02T09:37:54+02:00
author: "Alan Orth"
tags: ["Notes"]
---
## 2017-11-01
- The CORE developers responded to say they are looking into their bot not respecting our robots.txt
## 2017-11-02
- Today there have been no hits by CORE and no alerts from Linode (coincidence?)
```
# grep -c "CORE" /var/log/nginx/access.log
0
```
Add notes for 2017-11-02
2017-11-02 09:07:34 +01:00
- Generate list of authors on CGSpace for Peter to go through and correct:
```
dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors.csv with csv;
COPY 54701
```
Add notes for 2017-11-02
2017-11-02 08:37:59 +01:00
<!--more-->
Update notes for 2017-11-02
2017-11-02 16:05:37 +01:00
- Abenet asked if it would be possible to generate a report of items in Listing and Reports that had "International Fund for Agricultural Development" as the *only* investor
- I opened a ticket with Atmire to ask if this was possible: https://tracker.atmire.com/tickets-cgiar-ilri/view-ticket?id=540
- Work on making the thumbnails in the item view clickable
- Basically, once you read the METS XML for an item it becomes easy to trace the structure to find the bitstream link
```
//mets:fileSec/mets:fileGrp[@USE='CONTENT']/mets:file/mets:FLocat[@LOCTYPE='URL']/@xlink:href
```
- METS XML is available for all items with this pattern: /metadata/handle/10568/95947/mets.xml
- I whipped up a quick hack to print a clickable link with this URL on the thumbnail but it needs to check a few corner cases, like when there is a thumbnail but no content bitstream!
Update notes for 2017-11-02
2017-11-02 16:30:54 +01:00
- Help proof fifty-three CIAT records for Sisay: https://dspacetest.cgiar.org/handle/10568/95895
- A handful of issues with `cg.place` using format like "Lima, PE" instead of "Lima, Peru"
- Also, some dates like with completely invalid format like "2010- 06" and "2011-3-28"
- I also collapsed some consecutive whitespace on a handful of fields
Add notes for 2017-11-03
2017-11-03 17:15:40 +01:00
## 2017-11-03
- Atmire got back to us to say that they estimate it will take two days of labor to implement the change to Listings and Reports
- I said I'd ask Abenet if she wants that feature
Add notes for 2017-11-04
2017-11-04 15:44:23 +01:00
## 2017-11-04
- I finished looking through Sisay's CIAT records for the "Alianzas de Aprendizaje" data
- I corrected about half of the authors to standardize them
- Linode emailed this morning to say that the CPU usage was high again, this time at 6:14AM
- It's the first time in a few days that this has happened
- I had a look to see what was going on, but it isn't the CORE bot:
```
# awk '{print $1}' /var/log/nginx/access.log | sort -n | uniq -c | sort -h | tail
306 68.180.229.31
323 61.148.244.116
414 66.249.66.91
507 40.77.167.16
618 157.55.39.161
652 207.46.13.103
666 157.55.39.254
1173 104.196.152.243
1737 66.249.66.90
23101 138.201.52.218
```
- 138.201.52.218 is from some Hetzner server, and I see it making 40,000 requests yesterday too, but none before that:
```
# zgrep -c 138.201.52.218 /var/log/nginx/access.log*
/var/log/nginx/access.log:24403
/var/log/nginx/access.log.1:45958
/var/log/nginx/access.log.2.gz:0
/var/log/nginx/access.log.3.gz:0
/var/log/nginx/access.log.4.gz:0
/var/log/nginx/access.log.5.gz:0
/var/log/nginx/access.log.6.gz:0
```
- It's clearly a bot as it's making tens of thousands of requests, but it's using a "normal" user agent:
```
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36
```
- For now I don't know what this user is!
Add notes for 2017-11-05
2017-11-05 14:06:22 +01:00
## 2017-11-05
- Peter asked if I could fix the appearance of "International Livestock Research Institute" in the author lookup during item submission
- It looks to be just an issue with the user interface expecting authors to have both a first and last name:
![Author lookup](/cgspace-notes/2017/11/author-lookup.png)
![Add author](/cgspace-notes/2017/11/add-author.png)
- But in the database the authors are correct (none with weird `, /` characters):
```
dspace=# select distinct text_value, authority, confidence from metadatavalue value where resource_type_id=2 and metadata_field_id=3 and text_value like 'International Livestock Research Institute%';
text_value | authority | confidence
--------------------------------------------+--------------------------------------+------------
International Livestock Research Institute | 8f3865dc-d056-4aec-90b7-77f49ab4735c | 0
International Livestock Research Institute | f4db1627-47cd-4699-b394-bab7eba6dadc | 0
International Livestock Research Institute | | -1
International Livestock Research Institute | 8f3865dc-d056-4aec-90b7-77f49ab4735c | 600
International Livestock Research Institute | f4db1627-47cd-4699-b394-bab7eba6dadc | -1
International Livestock Research Institute | | 600
International Livestock Research Institute | 8f3865dc-d056-4aec-90b7-77f49ab4735c | -1
International Livestock Research Institute | 8f3865dc-d056-4aec-90b7-77f49ab4735c | 500
(8 rows)
```
- So I'm not sure if this is just a graphical glitch or if editors have to edit this metadata field prior to approval
Update notes for 2017-11-05
2017-11-05 14:53:35 +01:00
- Looking at monitoring Tomcat's JVM heap with Prometheus, it looks like we need to use JMX + [jmx_exporter](https://github.com/prometheus/jmx_exporter)
- This guide shows how to [enable JMX in Tomcat](https://geekflare.com/enable-jmx-tomcat-to-monitor-administer/) by modifying `CATALINA_OPTS`
- I was able to successfully connect to my local Tomcat with jconsole!
Reference in New Issue Copy Permalink