alanorth/cgspace-notes
1
0
Fork 0
mirror of https://github.com/alanorth/cgspace-notes.git synced 2024-11-18 20:57:03 +01:00
Code Issues Projects Releases Wiki Activity
cfe5796b3a
cgspace-notes/content/post/2016-11.md

98 lines
5.0 KiB
Markdown
Raw Normal View History

Add notes for 2016-11-01
2016-11-01 08:23:34 +01:00
+++
date = "2016-11-01T09:21:00+03:00"
author = "Alan Orth"
title = "November, 2016"
tags = ["Notes"]
+++
Add notes for 2016-11-02
2016-11-02 11:27:37 +01:00
## 2016-11-01
Add notes for 2016-11-01
2016-11-01 08:23:34 +01:00
Update notes for 2016-11-01
2016-11-01 08:35:09 +01:00
- Add `dc.type` to the output options for Atmire's Listings and Reports module ([#286](https://github.com/ilri/DSpace/pull/286))
Add notes for 2016-11-01
2016-11-01 08:23:34 +01:00
![Listings and Reports with output type](2016/11/listings-and-reports.png)
Add notes for 2016-11-02
2016-11-02 11:27:37 +01:00
## 2016-11-02
Add notes for 2016-11-06
2016-11-06 12:47:08 +01:00
- Migrate DSpace Test to DSpace 5.5 ([notes](https://gist.github.com/alanorth/61013895c6efe7095d7f81000953d1cf))
Add notes for 2016-11-02
2016-11-02 11:27:37 +01:00
- Run all updates on DSpace Test and reboot the server
Update notes for 2016-11-02
2016-11-02 12:51:42 +01:00
- Looks like the OAI bug from DSpace 5.1 that caused validation at Base Search to fail is now fixed and DSpace Test passes validation! ([#63](https://github.com/ilri/DSpace/issues/63))
Update notes for 2016-11-02
2016-11-02 16:19:02 +01:00
- Indexing Discovery on DSpace Test took 332 minutes, which is like five times as long as it usually takes
- At the end it appeared to finish correctly but there were lots of errors right after it finished:
```
2016-11-02 15:09:48,578 INFO com.atmire.dspace.discovery.AtmireSolrService @ Wrote Collection: 10568/76454 to Index
2016-11-02 15:09:48,584 INFO com.atmire.dspace.discovery.AtmireSolrService @ Wrote Community: 10568/3202 to Index
2016-11-02 15:09:48,589 INFO com.atmire.dspace.discovery.AtmireSolrService @ Wrote Collection: 10568/76455 to Index
2016-11-02 15:09:48,590 INFO com.atmire.dspace.discovery.AtmireSolrService @ Wrote Community: 10568/51693 to Index
2016-11-02 15:09:48,590 INFO org.dspace.discovery.IndexClient @ Done with indexing
2016-11-02 15:09:48,600 INFO com.atmire.dspace.discovery.AtmireSolrService @ Wrote Collection: 10568/76456 to Index
2016-11-02 15:09:48,613 INFO org.dspace.discovery.SolrServiceImpl @ Wrote Item: 10568/55536 to Index
2016-11-02 15:09:48,616 INFO com.atmire.dspace.discovery.AtmireSolrService @ Wrote Collection: 10568/76457 to Index
2016-11-02 15:09:48,634 ERROR com.atmire.dspace.discovery.AtmireSolrService @
java.lang.NullPointerException
at org.dspace.discovery.SearchUtils.getDiscoveryConfiguration(SourceFile:57)
at org.dspace.discovery.SolrServiceImpl.buildDocument(SolrServiceImpl.java:824)
at com.atmire.dspace.discovery.AtmireSolrService.indexContent(AtmireSolrService.java:821)
at com.atmire.dspace.discovery.AtmireSolrService.updateIndex(AtmireSolrService.java:898)
at org.dspace.discovery.SolrServiceImpl.createIndex(SolrServiceImpl.java:370)
at org.dspace.storage.rdbms.DatabaseUtils$ReindexerThread.run(DatabaseUtils.java:945)
```
- DSpace is still up, and a few minutes later I see the default DSpace indexer is still running
- Sure enough, looking back before the first one finished, I see output from both indexers interleaved in the log:
```
2016-11-02 15:09:28,545 INFO org.dspace.discovery.SolrServiceImpl @ Wrote Item: 10568/47242 to Index
2016-11-02 15:09:28,633 INFO org.dspace.discovery.SolrServiceImpl @ Wrote Item: 10568/60785 to Index
2016-11-02 15:09:28,678 INFO com.atmire.dspace.discovery.AtmireSolrService @ Processing (55695 of 55722): 43557
2016-11-02 15:09:28,688 INFO com.atmire.dspace.discovery.AtmireSolrService @ Processing (55703 of 55722): 34476
```
- I will raise a ticket with Atmire to ask them
Add notes for 2016-11-06
2016-11-06 12:47:08 +01:00
## 2016-11-06
- After re-deploying and re-indexing I didn't see the same issue, and the indexing completed in 85 minutes, which is about how long it is supposed to take
Update notes for 2016-11-07
2016-11-07 15:46:42 +01:00
## 2016-11-07
- Horrible one liner to get Linode ID from certain Ansible host vars:
```
$ grep -A 3 contact_info * | grep -E "(Orth|Sisay|Peter|Daniel|Tsega)" | awk -F'-' '{print $1}' | grep linode | uniq | xargs grep linode_id
```
- I noticed some weird CRPs in the database, and they don't show up in Discovery for some reason, perhaps the `:`
- I'll export these and fix them in batch:
```
dspace=# \copy (select distinct text_value, count(*) from metadatavalue where metadata_field_id=230 group by text_value order by count desc) to /tmp/crp.csv with csv;
COPY 22
```
- Test running the replacements:
```
$ ./fix-metadata-values.py -i /tmp/CRPs.csv -f cg.contributor.crp -t correct -m 230 -d dspace -u dspace -p 'fuuu'
```
- Add `AMR` to ILRI subjects and remove one duplicate instance of IITA in author affiliations controlled vocabulary ([#288](https://github.com/ilri/DSpace/pull/288))
Add notes for 2016-11-08
2016-11-08 10:27:36 +01:00
## 2016-11-08
- Atmire's Listings and Reports module seems to be broken on DSpace 5.5
![Listings and Reports broken in DSpace 5.5](2016/11/listings-and-reports-55.png)
- I've filed a ticket with Atmire
Update notes for 2016-11-08
2016-11-08 11:44:29 +01:00
- Thinking about batch updates for ORCIDs and authors
- Playing with [SolrClient](https://github.com/moonlitesolutions/SolrClient) in Python to query Solr
- All records in the authority core are either `authority_type:orcid` or `authority_type:person`
- There is a `deleted` field and all items seem to be `false`, but might be important sanity check to remember
- The way to go is probably to have a CSV of author names and authority IDs, then to batch update them in PostgreSQL
- Dump of the top ~200 authors in CGSpace:
```
dspace=# \copy (select distinct text_value, count(*) from metadatavalue where metadata_field_id=3 group by text_value order by count desc limit 210) to /tmp/210-authors.csv with csv;
```
Reference in New Issue Copy Permalink