March, 2018
+ +2018-03-02
+ +-
+
- Export a CSV of the IITA community metadata for Martin Mueller +
CGIAR Library Migration
@@ -123,6 +369,119 @@ +September, 2017
+ +2017-09-06
+ +-
+
- Linode sent an alert that CGSpace (linode18) was using 261% CPU for the past two hours +
2017-09-07
+ +-
+
- Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account is both in the approvers step as well as the group +
August, 2017
+ +2017-08-01
+ +-
+
- Linode sent an alert that CGSpace (linode18) was using 350% CPU for the past two hours +
- I looked in the Activity pane of the Admin Control Panel and it seems that Google, Baidu, Yahoo, and Bing are all crawling with massive numbers of bots concurrently (~100 total, mostly Baidu and Google) +
- The good thing is that, according to
dspace.log.2017-08-01, they are all using the same Tomcat session
+ - This means our Tomcat Crawler Session Valve is working +
- But many of the bots are browsing dynamic URLs like:
+
+
-
+
- /handle/10568/3353/discover +
- /handle/10568/16510/browse +
+ - The
robots.txtonly blocks the top-level/discoverand/browseURLs… we will need to find a way to forbid them from accessing these!
+ - Relevant issue from DSpace Jira (semi resolved in DSpace 6.0): https://jira.duraspace.org/browse/DS-2962 +
- It turns out that we’re already adding the
X-Robots-Tag "none"HTTP header, but this only forbids the search engine from indexing the page, not crawling it!
+ - Also, the bot has to successfully browse the page first so it can receive the HTTP header… +
- We might actually have to block these requests with HTTP 403 depending on the user agent +
- Abenet pointed out that the CGIAR Library Historical Archive collection I sent July 20th only had ~100 entries, instead of 2415 +
- This was due to newline characters in the
dc.description.abstractcolumn, which caused OpenRefine to choke when exporting the CSV
+ - I exported a new CSV from the collection on DSpace Test and then manually removed the characters in vim using
g/^$/d
+ - Then I cleaned up the author authorities and HTML characters in OpenRefine and sent the file back to Abenet +
July, 2017
+ +2017-07-01
+ +-
+
- Run system updates and reboot DSpace Test +
2017-07-04
+ +-
+
- Merge changes for WLE Phase II theme rename (#329) +
- Looking at extracting the metadata registries from ICARDA’s MEL DSpace database so we can compare fields with CGSpace +
- We can use PostgreSQL’s extended output format (
-x) plussedto format the output into quasi XML:
+
+
+
+
+
+
+
+
+
+
+ CGSpace Notes
+Documenting day-to-day work on the CGSpace repository.
+
+
+
+
+
+
+
+
+
+
+
diff --git a/docs/categories/notes/page/3/index.html b/docs/categories/notes/page/3/index.html
new file mode 100644
index 000000000..1885802fa
--- /dev/null
+++ b/docs/categories/notes/page/3/index.html
@@ -0,0 +1,481 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 2017-06-01 After discussion with WLE and CGSpace content people, we decided to just add one metadata field for the WLE Research Themes The cg.identifier.wletheme field will be used for both Phase I and Phase II Research Themes Then we’ll create a new sub-community for Phase II and create collections for the research themes there The current “Research Themes” community will be renamed to “WLE Phase I Research Themes” Tagged all items in the current Phase I collections with their appropriate themes Create pull request to add Phase II research themes to the submission form: #328 Add cg.
+ Read more →
+
+
+
+
+
+
+
+
+
+
+ 2017-05-01 ICARDA apparently started working on CG Core on their MEL repository They have done a few cg.* fields, but not very consistent and even copy some of CGSpace items: https://mel.cgiar.org/xmlui/handle/20.500.11766/6911?show=full https://cgspace.cgiar.org/handle/10568/73683 2017-05-02 Atmire got back about the Workflow Statistics issue, and apparently it’s a bug in the CUA module so they will send us a pull request 2017-05-04 Sync DSpace Test with database and assetstore from CGSpace Re-deploy DSpace Test with Atmire’s CUA patch for workflow statistics, run system updates, and restart the server Now I can see the workflow statistics and am able to select users, but everything returns 0 items Megan says there are still some mapped items are not appearing since last week, so I forced a full index-discovery -b Need to remember to check if the collection has more items (currently 39 on CGSpace, but 118 on the freshly reindexed DSPace Test) tomorrow: https://cgspace.
+ Read more →
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ June, 2017
+ +May, 2017
+ +April, 2017
+ +2017-04-02
+ +-
+
- Merge one change to CCAFS flagships that I had forgotten to remove last month (“MANAGING CLIMATE RISK”): https://github.com/ilri/DSpace/pull/317 +
- Quick proof-of-concept hack to add
dc.rightsto the input form, including some inline instructions/hints:
+
-
+
- Remove redundant/duplicate text in the DSpace submission license +
- Testing the CMYK patch on a collection with 650 items: +
$ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Thumbnail" -v >& /tmp/filter-media-cmyk.txt
+March, 2017
+ +2017-03-01
+ +-
+
- Run the 279 CIAT author corrections on CGSpace +
2017-03-02
+ +-
+
- Skype with Michael and Peter, discussing moving the CGIAR Library to CGSpace +
- CGIAR people possibly open to moving content, redirecting library.cgiar.org to CGSpace and letting CGSpace resolve their handles +
- They might come in at the top level in one “CGIAR System” community, or with several communities +
- I need to spend a bit of time looking at the multiple handle support in DSpace and see if new content can be minted in both handles, or just one? +
- Need to send Peter and Michael some notes about this in a few days +
- Also, need to consider talking to Atmire about hiring them to bring ORCiD metadata to REST / OAI +
- Filed an issue on DSpace issue tracker for the
filter-mediabug that causes it to process JPGs even when limiting to the PDF thumbnail plugin: DS-3516
+ - Discovered that the ImageMagic
filter-mediaplugin creates JPG thumbnails with the CMYK colorspace when the source PDF is using CMYK
+ - Interestingly, it seems DSpace 4.x’s thumbnails were sRGB, but forcing regeneration using DSpace 5.x’s ImageMagick plugin creates CMYK JPGs if the source PDF was CMYK (see 10568⁄51999): +
$ identify ~/Desktop/alc_contrastes_desafios.jpg
+/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000
+February, 2017
+ +2017-02-07
+ +-
+
- An item was mapped twice erroneously again, so I had to remove one of the mappings manually: +
dspace=# select * from collection2item where item_id = '80278';
+ id | collection_id | item_id
+-------+---------------+---------
+ 92551 | 313 | 80278
+ 92550 | 313 | 80278
+ 90774 | 1051 | 80278
+(3 rows)
+dspace=# delete from collection2item where id = 92551 and item_id = 80278;
+DELETE 1
+-
+
- Create issue on GitHub to track the addition of CCAFS Phase II project tags (#301) +
- Looks like we’ll be using
cg.identifier.ccafsprojectpiias the field name
+
January, 2017
+ +2017-01-02
+ +-
+
- I checked to see if the Solr sharding task that is supposed to run on January 1st had run and saw there was an error +
- I tested on DSpace Test as well and it doesn’t work there either +
- I asked on the dspace-tech mailing list because it seems to be broken, and actually now I’m not sure if we’ve ever had the sharding task run successfully over all these years +
December, 2016
+ +2016-12-02
+ +-
+
- CGSpace was down for five hours in the morning while I was sleeping +
- While looking in the logs for errors, I see tons of warnings about Atmire MQM: +
2016-12-02 03:00:32,352 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=CREATE, SubjectType=BUNDLE, SubjectID=70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632305, dispatcher=1544803905, detail=[null], transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY_METADATA, SubjectType=BUNDLE, SubjectID =70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632309, dispatcher=1544803905, detail="dc.title", transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=ITEM, SubjectID=80044, Object Type=BUNDLE, ObjectID=70316, TimeStamp=1480647632311, dispatcher=1544803905, detail="THUMBNAIL", transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=BUNDLE, SubjectID=70316, Obje ctType=BITSTREAM, ObjectID=86715, TimeStamp=1480647632318, dispatcher=1544803905, detail="-1", transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY, SubjectType=ITEM, SubjectID=80044, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632351, dispatcher=1544803905, detail=[null], transactionID="TX157907838689377964651674089851855413607")
+-
+
- I see thousands of them in the logs for the last few months, so it’s not related to the DSpace 5.5 upgrade +
- I’ve raised a ticket with Atmire to ask +
- Another worrying error from dspace.log is: +
November, 2016
+ +2016-11-01
+ +-
+
- Add
dc.typeto the output options for Atmire’s Listings and Reports module (#286)
+
October, 2016
+ +2016-10-03
+ +-
+
- Testing adding ORCIDs to a CSV file for a single item to see if the author orders get messed up +
- Need to test the following scenarios to see how author order is affected:
+
+
-
+
- ORCIDs only +
- ORCIDs plus normal authors +
+ - I exported a random item’s metadata as CSV, deleted all columns except id and collection, and made a new coloum called
ORCID:dc.contributor.authorwith the following random ORCIDs from the ORCID registry:
+
0000-0002-6115-0956||0000-0002-3812-8793||0000-0001-7462-405X
+September, 2016
+ +2016-09-01
+ +-
+
- Discuss helping CCAFS with some batch tagging of ORCID IDs for their authors +
- Discuss how the migration of CGIAR’s Active Directory to a flat structure will break our LDAP groups in DSpace +
- We had been using
DC=ILRIto determine whether a user was ILRI or not
+ - It looks like we might be able to use OUs now, instead of DCs: +
$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=org" -D "admigration1@cgiarad.org" -W "(sAMAccountName=admigration1)"
+
+
+
+
+
+
+
+
+
+
+ CGSpace Notes
+Documenting day-to-day work on the CGSpace repository.
+
+
+
+
+
+
+
+
+
+
+
diff --git a/docs/categories/notes/page/4/index.html b/docs/categories/notes/page/4/index.html
new file mode 100644
index 000000000..faf203b7d
--- /dev/null
+++ b/docs/categories/notes/page/4/index.html
@@ -0,0 +1,191 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ August, 2016
+ +2016-08-01
+ +-
+
- Add updated distribution license from Sisay (#259) +
- Play with upgrading Mirage 2 dependencies in
bower.jsonbecause most are several versions of out date
+ - Bootstrap is at 3.3.0 but upstream is at 3.3.7, and upgrading to anything beyond 3.3.1 breaks glyphicons and probably more +
- bower stuff is a dead end, waste of time, too many issues +
- Anything after Bootstrap 3.3.1 makes glyphicons disappear (HTTP 404 trying to access from incorrect path of
fonts)
+ - Start working on DSpace 5.1 → 5.5 port: +
$ git checkout -b 55new 5_x-prod
+$ git reset --hard ilri/5_x-prod
+$ git rebase -i dspace-5.5
+July, 2016
+ +2016-07-01
+ +-
+
- Add
dc.description.sponsorshipto Discovery sidebar facets and make investors clickable in item view (#232)
+ - I think this query should find and replace all authors that have “,” at the end of their names: +
dspacetest=# update metadatavalue set text_value = regexp_replace(text_value, '(^.+?),$', '\1') where metadata_field_id=3 and resource_type_id=2 and text_value ~ '^.+?,$';
+UPDATE 95
+dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value ~ '^.+?,$';
+ text_value
+------------
+(0 rows)
+-
+
- In this case the select query was showing 95 results before the update +
June, 2016
+ +2016-06-01
+ +-
+
- Experimenting with IFPRI OAI (we want to harvest their publications) +
- After reading the ContentDM documentation I found IFPRI’s OAI endpoint: http://ebrary.ifpri.org/oai/oai.php +
- After reading the OAI documentation and testing with an OAI validator I found out how to get their publications +
- This is their publications set: http://ebrary.ifpri.org/oai/oai.php?verb=ListRecords&from=2016-01-01&set=p15738coll2&metadataPrefix=oai_dc +
- You can see the others by using the OAI
ListSetsverb: http://ebrary.ifpri.org/oai/oai.php?verb=ListSets
+ - Working on second phase of metadata migration, looks like this will work for moving CPWF-specific data in
dc.identifier.fundtocg.identifier.cpwfprojectand then the rest todc.description.sponsorship
+
May, 2016
+ +2016-05-01
+ +-
+
- Since yesterday there have been 10,000 REST errors and the site has been unstable again +
- I have blocked access to the API now +
- There are 3,000 IPs accessing the REST API in a 24-hour period! +
# awk '{print $1}' /var/log/nginx/rest.log | uniq | wc -l
+3168
+April, 2016
+ +2016-04-04
+ +-
+
- Looking at log file use on CGSpace and notice that we need to work on our cron setup a bit +
- We are backing up all logs in the log folder, including useless stuff like solr, cocoon, handle-plugin, etc +
- After running DSpace for over five years I’ve never needed to look in any other log file than dspace.log, leave alone one from last year! +
- This will save us a few gigs of backup space we’re paying for on S3 +
- Also, I noticed the
checkerlog has some errors we should pay attention to:
+
March, 2016
+ +2016-03-02
+ +-
+
- Looking at issues with author authorities on CGSpace +
- For some reason we still have the
index-lucene-updatecron job active on CGSpace, but I’m pretty sure we don’t need it as of the latest few versions of Atmire’s Listings and Reports module
+ - Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Java JDK 1.7 to match environment on CGSpace server +
February, 2016
+ +2016-02-05
+ +-
+
- Looking at some DAGRIS data for Abenet Yabowork +
- Lots of issues with spaces, newlines, etc causing the import to fail +
- I noticed we have a very interesting list of countries on CGSpace: +
-
+
- Not only are there 49,000 countries, we have some blanks (25)… +
- Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE” +
January, 2016
+ +2016-01-13
+ +-
+
- Move ILRI collection
10568/12503from10568/27869to10568/27629using the move_collections.sh script I wrote last year.
+ - I realized it is only necessary to clear the Cocoon cache after moving collections—rather than reindexing—as no metadata has changed, and therefore no search or browse indexes need to be updated. +
- Update GitHub wiki for documentation of maintenance tasks. +
December, 2015
+ +2015-12-02
+ +-
+
- Replace
lzopwithxzin log compression cron jobs on DSpace Test—it uses less space:
+
# cd /home/dspacetest.cgiar.org/log
+# ls -lh dspace.log.2015-11-18*
+-rw-rw-r-- 1 tomcat7 tomcat7 2.0M Nov 18 23:59 dspace.log.2015-11-18
+-rw-rw-r-- 1 tomcat7 tomcat7 387K Nov 18 23:59 dspace.log.2015-11-18.lzo
+-rw-rw-r-- 1 tomcat7 tomcat7 169K Nov 18 23:59 dspace.log.2015-11-18.xz
+November, 2015
+ +2015-11-22
+ +-
+
- CGSpace went down +
- Looks like DSpace exhausted its PostgreSQL connection pool +
- Last week I had increased the limit from 30 to 60, which seemed to help, but now there are many more idle connections: +
$ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
+78
+
+
+
+
+
+
+
+
+
+
+ CGSpace Notes
+Documenting day-to-day work on the CGSpace repository.
+
+
+
+
+
+
+
+
+
+
+
diff --git a/docs/categories/page/2/index.html b/docs/categories/page/2/index.html
new file mode 100644
index 000000000..c86e25129
--- /dev/null
+++ b/docs/categories/page/2/index.html
@@ -0,0 +1,484 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Read more →
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Posts
+ +
+
+
+
+
+
+
+
+
+
+ CGSpace Notes
+Documenting day-to-day work on the CGSpace repository.
+
+
+
+
+
+
+
+
+
+
+
diff --git a/docs/categories/page/3/index.html b/docs/categories/page/3/index.html
new file mode 100644
index 000000000..66191a251
--- /dev/null
+++ b/docs/categories/page/3/index.html
@@ -0,0 +1,481 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 2017-06-01 After discussion with WLE and CGSpace content people, we decided to just add one metadata field for the WLE Research Themes The cg.identifier.wletheme field will be used for both Phase I and Phase II Research Themes Then we’ll create a new sub-community for Phase II and create collections for the research themes there The current “Research Themes” community will be renamed to “WLE Phase I Research Themes” Tagged all items in the current Phase I collections with their appropriate themes Create pull request to add Phase II research themes to the submission form: #328 Add cg.
+ Read more →
+
+
+
+
+
+
+
+
+
+
+ 2017-05-01 ICARDA apparently started working on CG Core on their MEL repository They have done a few cg.* fields, but not very consistent and even copy some of CGSpace items: https://mel.cgiar.org/xmlui/handle/20.500.11766/6911?show=full https://cgspace.cgiar.org/handle/10568/73683 2017-05-02 Atmire got back about the Workflow Statistics issue, and apparently it’s a bug in the CUA module so they will send us a pull request 2017-05-04 Sync DSpace Test with database and assetstore from CGSpace Re-deploy DSpace Test with Atmire’s CUA patch for workflow statistics, run system updates, and restart the server Now I can see the workflow statistics and am able to select users, but everything returns 0 items Megan says there are still some mapped items are not appearing since last week, so I forced a full index-discovery -b Need to remember to check if the collection has more items (currently 39 on CGSpace, but 118 on the freshly reindexed DSPace Test) tomorrow: https://cgspace.
+ Read more →
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ June, 2017
+ +May, 2017
+ +April, 2017
+ +2017-04-02
+ +-
+
- Merge one change to CCAFS flagships that I had forgotten to remove last month (“MANAGING CLIMATE RISK”): https://github.com/ilri/DSpace/pull/317 +
- Quick proof-of-concept hack to add
dc.rightsto the input form, including some inline instructions/hints:
+
-
+
- Remove redundant/duplicate text in the DSpace submission license +
- Testing the CMYK patch on a collection with 650 items: +
$ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Thumbnail" -v >& /tmp/filter-media-cmyk.txt
+March, 2017
+ +2017-03-01
+ +-
+
- Run the 279 CIAT author corrections on CGSpace +
2017-03-02
+ +-
+
- Skype with Michael and Peter, discussing moving the CGIAR Library to CGSpace +
- CGIAR people possibly open to moving content, redirecting library.cgiar.org to CGSpace and letting CGSpace resolve their handles +
- They might come in at the top level in one “CGIAR System” community, or with several communities +
- I need to spend a bit of time looking at the multiple handle support in DSpace and see if new content can be minted in both handles, or just one? +
- Need to send Peter and Michael some notes about this in a few days +
- Also, need to consider talking to Atmire about hiring them to bring ORCiD metadata to REST / OAI +
- Filed an issue on DSpace issue tracker for the
filter-mediabug that causes it to process JPGs even when limiting to the PDF thumbnail plugin: DS-3516
+ - Discovered that the ImageMagic
filter-mediaplugin creates JPG thumbnails with the CMYK colorspace when the source PDF is using CMYK
+ - Interestingly, it seems DSpace 4.x’s thumbnails were sRGB, but forcing regeneration using DSpace 5.x’s ImageMagick plugin creates CMYK JPGs if the source PDF was CMYK (see 10568⁄51999): +
$ identify ~/Desktop/alc_contrastes_desafios.jpg
+/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000
+February, 2017
+ +2017-02-07
+ +-
+
- An item was mapped twice erroneously again, so I had to remove one of the mappings manually: +
dspace=# select * from collection2item where item_id = '80278';
+ id | collection_id | item_id
+-------+---------------+---------
+ 92551 | 313 | 80278
+ 92550 | 313 | 80278
+ 90774 | 1051 | 80278
+(3 rows)
+dspace=# delete from collection2item where id = 92551 and item_id = 80278;
+DELETE 1
+-
+
- Create issue on GitHub to track the addition of CCAFS Phase II project tags (#301) +
- Looks like we’ll be using
cg.identifier.ccafsprojectpiias the field name
+
January, 2017
+ +2017-01-02
+ +-
+
- I checked to see if the Solr sharding task that is supposed to run on January 1st had run and saw there was an error +
- I tested on DSpace Test as well and it doesn’t work there either +
- I asked on the dspace-tech mailing list because it seems to be broken, and actually now I’m not sure if we’ve ever had the sharding task run successfully over all these years +
December, 2016
+ +2016-12-02
+ +-
+
- CGSpace was down for five hours in the morning while I was sleeping +
- While looking in the logs for errors, I see tons of warnings about Atmire MQM: +
2016-12-02 03:00:32,352 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=CREATE, SubjectType=BUNDLE, SubjectID=70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632305, dispatcher=1544803905, detail=[null], transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY_METADATA, SubjectType=BUNDLE, SubjectID =70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632309, dispatcher=1544803905, detail="dc.title", transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=ITEM, SubjectID=80044, Object Type=BUNDLE, ObjectID=70316, TimeStamp=1480647632311, dispatcher=1544803905, detail="THUMBNAIL", transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=BUNDLE, SubjectID=70316, Obje ctType=BITSTREAM, ObjectID=86715, TimeStamp=1480647632318, dispatcher=1544803905, detail="-1", transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY, SubjectType=ITEM, SubjectID=80044, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632351, dispatcher=1544803905, detail=[null], transactionID="TX157907838689377964651674089851855413607")
+-
+
- I see thousands of them in the logs for the last few months, so it’s not related to the DSpace 5.5 upgrade +
- I’ve raised a ticket with Atmire to ask +
- Another worrying error from dspace.log is: +
November, 2016
+ +2016-11-01
+ +-
+
- Add
dc.typeto the output options for Atmire’s Listings and Reports module (#286)
+
October, 2016
+ +2016-10-03
+ +-
+
- Testing adding ORCIDs to a CSV file for a single item to see if the author orders get messed up +
- Need to test the following scenarios to see how author order is affected:
+
+
-
+
- ORCIDs only +
- ORCIDs plus normal authors +
+ - I exported a random item’s metadata as CSV, deleted all columns except id and collection, and made a new coloum called
ORCID:dc.contributor.authorwith the following random ORCIDs from the ORCID registry:
+
0000-0002-6115-0956||0000-0002-3812-8793||0000-0001-7462-405X
+September, 2016
+ +2016-09-01
+ +-
+
- Discuss helping CCAFS with some batch tagging of ORCID IDs for their authors +
- Discuss how the migration of CGIAR’s Active Directory to a flat structure will break our LDAP groups in DSpace +
- We had been using
DC=ILRIto determine whether a user was ILRI or not
+ - It looks like we might be able to use OUs now, instead of DCs: +
$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=org" -D "admigration1@cgiarad.org" -W "(sAMAccountName=admigration1)"
+
+
+
+
+
+
+
+
+
+
+ CGSpace Notes
+Documenting day-to-day work on the CGSpace repository.
+
+
+
+
+
+
+
+
+
+
+
diff --git a/docs/categories/page/4/index.html b/docs/categories/page/4/index.html
new file mode 100644
index 000000000..6edb57e97
--- /dev/null
+++ b/docs/categories/page/4/index.html
@@ -0,0 +1,191 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ August, 2016
+ +2016-08-01
+ +-
+
- Add updated distribution license from Sisay (#259) +
- Play with upgrading Mirage 2 dependencies in
bower.jsonbecause most are several versions of out date
+ - Bootstrap is at 3.3.0 but upstream is at 3.3.7, and upgrading to anything beyond 3.3.1 breaks glyphicons and probably more +
- bower stuff is a dead end, waste of time, too many issues +
- Anything after Bootstrap 3.3.1 makes glyphicons disappear (HTTP 404 trying to access from incorrect path of
fonts)
+ - Start working on DSpace 5.1 → 5.5 port: +
$ git checkout -b 55new 5_x-prod
+$ git reset --hard ilri/5_x-prod
+$ git rebase -i dspace-5.5
+July, 2016
+ +2016-07-01
+ +-
+
- Add
dc.description.sponsorshipto Discovery sidebar facets and make investors clickable in item view (#232)
+ - I think this query should find and replace all authors that have “,” at the end of their names: +
dspacetest=# update metadatavalue set text_value = regexp_replace(text_value, '(^.+?),$', '\1') where metadata_field_id=3 and resource_type_id=2 and text_value ~ '^.+?,$';
+UPDATE 95
+dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value ~ '^.+?,$';
+ text_value
+------------
+(0 rows)
+-
+
- In this case the select query was showing 95 results before the update +
June, 2016
+ +2016-06-01
+ +-
+
- Experimenting with IFPRI OAI (we want to harvest their publications) +
- After reading the ContentDM documentation I found IFPRI’s OAI endpoint: http://ebrary.ifpri.org/oai/oai.php +
- After reading the OAI documentation and testing with an OAI validator I found out how to get their publications +
- This is their publications set: http://ebrary.ifpri.org/oai/oai.php?verb=ListRecords&from=2016-01-01&set=p15738coll2&metadataPrefix=oai_dc +
- You can see the others by using the OAI
ListSetsverb: http://ebrary.ifpri.org/oai/oai.php?verb=ListSets
+ - Working on second phase of metadata migration, looks like this will work for moving CPWF-specific data in
dc.identifier.fundtocg.identifier.cpwfprojectand then the rest todc.description.sponsorship
+
May, 2016
+ +2016-05-01
+ +-
+
- Since yesterday there have been 10,000 REST errors and the site has been unstable again +
- I have blocked access to the API now +
- There are 3,000 IPs accessing the REST API in a 24-hour period! +
# awk '{print $1}' /var/log/nginx/rest.log | uniq | wc -l
+3168
+April, 2016
+ +2016-04-04
+ +-
+
- Looking at log file use on CGSpace and notice that we need to work on our cron setup a bit +
- We are backing up all logs in the log folder, including useless stuff like solr, cocoon, handle-plugin, etc +
- After running DSpace for over five years I’ve never needed to look in any other log file than dspace.log, leave alone one from last year! +
- This will save us a few gigs of backup space we’re paying for on S3 +
- Also, I noticed the
checkerlog has some errors we should pay attention to:
+
March, 2016
+ +2016-03-02
+ +-
+
- Looking at issues with author authorities on CGSpace +
- For some reason we still have the
index-lucene-updatecron job active on CGSpace, but I’m pretty sure we don’t need it as of the latest few versions of Atmire’s Listings and Reports module
+ - Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Java JDK 1.7 to match environment on CGSpace server +
February, 2016
+ +2016-02-05
+ +-
+
- Looking at some DAGRIS data for Abenet Yabowork +
- Lots of issues with spaces, newlines, etc causing the import to fail +
- I noticed we have a very interesting list of countries on CGSpace: +
-
+
- Not only are there 49,000 countries, we have some blanks (25)… +
- Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE” +
January, 2016
+ +2016-01-13
+ +-
+
- Move ILRI collection
10568/12503from10568/27869to10568/27629using the move_collections.sh script I wrote last year.
+ - I realized it is only necessary to clear the Cocoon cache after moving collections—rather than reindexing—as no metadata has changed, and therefore no search or browse indexes need to be updated. +
- Update GitHub wiki for documentation of maintenance tasks. +
December, 2015
+ +2015-12-02
+ +-
+
- Replace
lzopwithxzin log compression cron jobs on DSpace Test—it uses less space:
+
# cd /home/dspacetest.cgiar.org/log
+# ls -lh dspace.log.2015-11-18*
+-rw-rw-r-- 1 tomcat7 tomcat7 2.0M Nov 18 23:59 dspace.log.2015-11-18
+-rw-rw-r-- 1 tomcat7 tomcat7 387K Nov 18 23:59 dspace.log.2015-11-18.lzo
+-rw-rw-r-- 1 tomcat7 tomcat7 169K Nov 18 23:59 dspace.log.2015-11-18.xz
+November, 2015
+ +2015-11-22
+ +-
+
- CGSpace went down +
- Looks like DSpace exhausted its PostgreSQL connection pool +
- Last week I had increased the limit from 30 to 60, which seemed to help, but now there are many more idle connections: +
$ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
+78
+
+
+
+
+
+
+
+
+
+
+ CGSpace Notes
+Documenting day-to-day work on the CGSpace repository.
+
+
+
+
+
+
+
+
+
+
+
diff --git a/docs/cgiar-library-migration/index.html b/docs/cgiar-library-migration/index.html
index 6de53c8c1..a989f1009 100644
--- a/docs/cgiar-library-migration/index.html
+++ b/docs/cgiar-library-migration/index.html
@@ -14,7 +14,7 @@
-
+
@@ -41,7 +41,7 @@
"url": "https://alanorth.github.io/cgspace-notes/cgiar-library-migration/",
"wordCount": "1278",
"datePublished": "2017-09-18T16:38:35+03:00",
- "dateModified": "2017-09-28T12:00:49+03:00",
+ "dateModified": "2018-03-09T22:10:33+02:00",
"author": {
"@type": "Person",
"name": "Alan Orth"
@@ -77,8 +77,6 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Read more →
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Posts
+ +
+
+
+
+
+
+
+
+
+
+ CGSpace Notes
+Documenting day-to-day work on the CGSpace repository.
+
+
+
+
+
+
+
+
+
+
+
diff --git a/docs/post/page/1/index.html b/docs/post/page/1/index.html
deleted file mode 100644
index c4d1d9390..000000000
--- a/docs/post/page/1/index.html
+++ /dev/null
@@ -1 +0,0 @@
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Read more →
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Posts
+ +
+
+
+
+
+
+
+
+
+
+ CGSpace Notes
+Documenting day-to-day work on the CGSpace repository.
+
+
+
+
+
+
+
+
+
+
+
diff --git a/docs/posts/page/3/index.html b/docs/posts/page/3/index.html
new file mode 100644
index 000000000..6cf4917ae
--- /dev/null
+++ b/docs/posts/page/3/index.html
@@ -0,0 +1,481 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 2017-06-01 After discussion with WLE and CGSpace content people, we decided to just add one metadata field for the WLE Research Themes The cg.identifier.wletheme field will be used for both Phase I and Phase II Research Themes Then we’ll create a new sub-community for Phase II and create collections for the research themes there The current “Research Themes” community will be renamed to “WLE Phase I Research Themes” Tagged all items in the current Phase I collections with their appropriate themes Create pull request to add Phase II research themes to the submission form: #328 Add cg.
+ Read more →
+
+
+
+
+
+
+
+
+
+
+ 2017-05-01 ICARDA apparently started working on CG Core on their MEL repository They have done a few cg.* fields, but not very consistent and even copy some of CGSpace items: https://mel.cgiar.org/xmlui/handle/20.500.11766/6911?show=full https://cgspace.cgiar.org/handle/10568/73683 2017-05-02 Atmire got back about the Workflow Statistics issue, and apparently it’s a bug in the CUA module so they will send us a pull request 2017-05-04 Sync DSpace Test with database and assetstore from CGSpace Re-deploy DSpace Test with Atmire’s CUA patch for workflow statistics, run system updates, and restart the server Now I can see the workflow statistics and am able to select users, but everything returns 0 items Megan says there are still some mapped items are not appearing since last week, so I forced a full index-discovery -b Need to remember to check if the collection has more items (currently 39 on CGSpace, but 118 on the freshly reindexed DSPace Test) tomorrow: https://cgspace.
+ Read more →
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ June, 2017
+ +May, 2017
+ +April, 2017
+ +2017-04-02
+ +-
+
- Merge one change to CCAFS flagships that I had forgotten to remove last month (“MANAGING CLIMATE RISK”): https://github.com/ilri/DSpace/pull/317 +
- Quick proof-of-concept hack to add
dc.rightsto the input form, including some inline instructions/hints:
+
-
+
- Remove redundant/duplicate text in the DSpace submission license +
- Testing the CMYK patch on a collection with 650 items: +
$ [dspace]/bin/dspace filter-media -f -i 10568/16498 -p "ImageMagick PDF Thumbnail" -v >& /tmp/filter-media-cmyk.txt
+March, 2017
+ +2017-03-01
+ +-
+
- Run the 279 CIAT author corrections on CGSpace +
2017-03-02
+ +-
+
- Skype with Michael and Peter, discussing moving the CGIAR Library to CGSpace +
- CGIAR people possibly open to moving content, redirecting library.cgiar.org to CGSpace and letting CGSpace resolve their handles +
- They might come in at the top level in one “CGIAR System” community, or with several communities +
- I need to spend a bit of time looking at the multiple handle support in DSpace and see if new content can be minted in both handles, or just one? +
- Need to send Peter and Michael some notes about this in a few days +
- Also, need to consider talking to Atmire about hiring them to bring ORCiD metadata to REST / OAI +
- Filed an issue on DSpace issue tracker for the
filter-mediabug that causes it to process JPGs even when limiting to the PDF thumbnail plugin: DS-3516
+ - Discovered that the ImageMagic
filter-mediaplugin creates JPG thumbnails with the CMYK colorspace when the source PDF is using CMYK
+ - Interestingly, it seems DSpace 4.x’s thumbnails were sRGB, but forcing regeneration using DSpace 5.x’s ImageMagick plugin creates CMYK JPGs if the source PDF was CMYK (see 10568⁄51999): +
$ identify ~/Desktop/alc_contrastes_desafios.jpg
+/Users/aorth/Desktop/alc_contrastes_desafios.jpg JPEG 464x600 464x600+0+0 8-bit CMYK 168KB 0.000u 0:00.000
+February, 2017
+ +2017-02-07
+ +-
+
- An item was mapped twice erroneously again, so I had to remove one of the mappings manually: +
dspace=# select * from collection2item where item_id = '80278';
+ id | collection_id | item_id
+-------+---------------+---------
+ 92551 | 313 | 80278
+ 92550 | 313 | 80278
+ 90774 | 1051 | 80278
+(3 rows)
+dspace=# delete from collection2item where id = 92551 and item_id = 80278;
+DELETE 1
+-
+
- Create issue on GitHub to track the addition of CCAFS Phase II project tags (#301) +
- Looks like we’ll be using
cg.identifier.ccafsprojectpiias the field name
+
January, 2017
+ +2017-01-02
+ +-
+
- I checked to see if the Solr sharding task that is supposed to run on January 1st had run and saw there was an error +
- I tested on DSpace Test as well and it doesn’t work there either +
- I asked on the dspace-tech mailing list because it seems to be broken, and actually now I’m not sure if we’ve ever had the sharding task run successfully over all these years +
December, 2016
+ +2016-12-02
+ +-
+
- CGSpace was down for five hours in the morning while I was sleeping +
- While looking in the logs for errors, I see tons of warnings about Atmire MQM: +
2016-12-02 03:00:32,352 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=CREATE, SubjectType=BUNDLE, SubjectID=70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632305, dispatcher=1544803905, detail=[null], transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY_METADATA, SubjectType=BUNDLE, SubjectID =70316, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632309, dispatcher=1544803905, detail="dc.title", transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=ITEM, SubjectID=80044, Object Type=BUNDLE, ObjectID=70316, TimeStamp=1480647632311, dispatcher=1544803905, detail="THUMBNAIL", transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=ADD, SubjectType=BUNDLE, SubjectID=70316, Obje ctType=BITSTREAM, ObjectID=86715, TimeStamp=1480647632318, dispatcher=1544803905, detail="-1", transactionID="TX157907838689377964651674089851855413607")
+2016-12-02 03:00:32,353 WARN com.atmire.metadataquality.batchedit.BatchEditConsumer @ BatchEditConsumer should not have been given this kind of Subject in an event, skipping: org.dspace.event.Event(eventType=MODIFY, SubjectType=ITEM, SubjectID=80044, ObjectType=(Unknown), ObjectID=-1, TimeStamp=1480647632351, dispatcher=1544803905, detail=[null], transactionID="TX157907838689377964651674089851855413607")
+-
+
- I see thousands of them in the logs for the last few months, so it’s not related to the DSpace 5.5 upgrade +
- I’ve raised a ticket with Atmire to ask +
- Another worrying error from dspace.log is: +
November, 2016
+ +2016-11-01
+ +-
+
- Add
dc.typeto the output options for Atmire’s Listings and Reports module (#286)
+
October, 2016
+ +2016-10-03
+ +-
+
- Testing adding ORCIDs to a CSV file for a single item to see if the author orders get messed up +
- Need to test the following scenarios to see how author order is affected:
+
+
-
+
- ORCIDs only +
- ORCIDs plus normal authors +
+ - I exported a random item’s metadata as CSV, deleted all columns except id and collection, and made a new coloum called
ORCID:dc.contributor.authorwith the following random ORCIDs from the ORCID registry:
+
0000-0002-6115-0956||0000-0002-3812-8793||0000-0001-7462-405X
+September, 2016
+ +2016-09-01
+ +-
+
- Discuss helping CCAFS with some batch tagging of ORCID IDs for their authors +
- Discuss how the migration of CGIAR’s Active Directory to a flat structure will break our LDAP groups in DSpace +
- We had been using
DC=ILRIto determine whether a user was ILRI or not
+ - It looks like we might be able to use OUs now, instead of DCs: +
$ ldapsearch -x -H ldaps://svcgroot2.cgiarad.org:3269/ -b "dc=cgiarad,dc=org" -D "admigration1@cgiarad.org" -W "(sAMAccountName=admigration1)"
+
+
+
+
+
+
+
+
+
+
+ CGSpace Notes
+Documenting day-to-day work on the CGSpace repository.
+
+
+
+
+
+
+
+
+
+
+
diff --git a/docs/posts/page/4/index.html b/docs/posts/page/4/index.html
new file mode 100644
index 000000000..164dbd742
--- /dev/null
+++ b/docs/posts/page/4/index.html
@@ -0,0 +1,191 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ August, 2016
+ +2016-08-01
+ +-
+
- Add updated distribution license from Sisay (#259) +
- Play with upgrading Mirage 2 dependencies in
bower.jsonbecause most are several versions of out date
+ - Bootstrap is at 3.3.0 but upstream is at 3.3.7, and upgrading to anything beyond 3.3.1 breaks glyphicons and probably more +
- bower stuff is a dead end, waste of time, too many issues +
- Anything after Bootstrap 3.3.1 makes glyphicons disappear (HTTP 404 trying to access from incorrect path of
fonts)
+ - Start working on DSpace 5.1 → 5.5 port: +
$ git checkout -b 55new 5_x-prod
+$ git reset --hard ilri/5_x-prod
+$ git rebase -i dspace-5.5
+July, 2016
+ +2016-07-01
+ +-
+
- Add
dc.description.sponsorshipto Discovery sidebar facets and make investors clickable in item view (#232)
+ - I think this query should find and replace all authors that have “,” at the end of their names: +
dspacetest=# update metadatavalue set text_value = regexp_replace(text_value, '(^.+?),$', '\1') where metadata_field_id=3 and resource_type_id=2 and text_value ~ '^.+?,$';
+UPDATE 95
+dspacetest=# select text_value from metadatavalue where metadata_field_id=3 and resource_type_id=2 and text_value ~ '^.+?,$';
+ text_value
+------------
+(0 rows)
+-
+
- In this case the select query was showing 95 results before the update +
June, 2016
+ +2016-06-01
+ +-
+
- Experimenting with IFPRI OAI (we want to harvest their publications) +
- After reading the ContentDM documentation I found IFPRI’s OAI endpoint: http://ebrary.ifpri.org/oai/oai.php +
- After reading the OAI documentation and testing with an OAI validator I found out how to get their publications +
- This is their publications set: http://ebrary.ifpri.org/oai/oai.php?verb=ListRecords&from=2016-01-01&set=p15738coll2&metadataPrefix=oai_dc +
- You can see the others by using the OAI
ListSetsverb: http://ebrary.ifpri.org/oai/oai.php?verb=ListSets
+ - Working on second phase of metadata migration, looks like this will work for moving CPWF-specific data in
dc.identifier.fundtocg.identifier.cpwfprojectand then the rest todc.description.sponsorship
+
May, 2016
+ +2016-05-01
+ +-
+
- Since yesterday there have been 10,000 REST errors and the site has been unstable again +
- I have blocked access to the API now +
- There are 3,000 IPs accessing the REST API in a 24-hour period! +
# awk '{print $1}' /var/log/nginx/rest.log | uniq | wc -l
+3168
+April, 2016
+ +2016-04-04
+ +-
+
- Looking at log file use on CGSpace and notice that we need to work on our cron setup a bit +
- We are backing up all logs in the log folder, including useless stuff like solr, cocoon, handle-plugin, etc +
- After running DSpace for over five years I’ve never needed to look in any other log file than dspace.log, leave alone one from last year! +
- This will save us a few gigs of backup space we’re paying for on S3 +
- Also, I noticed the
checkerlog has some errors we should pay attention to:
+
March, 2016
+ +2016-03-02
+ +-
+
- Looking at issues with author authorities on CGSpace +
- For some reason we still have the
index-lucene-updatecron job active on CGSpace, but I’m pretty sure we don’t need it as of the latest few versions of Atmire’s Listings and Reports module
+ - Reinstall my local (Mac OS X) DSpace stack with Tomcat 7, PostgreSQL 9.3, and Java JDK 1.7 to match environment on CGSpace server +
February, 2016
+ +2016-02-05
+ +-
+
- Looking at some DAGRIS data for Abenet Yabowork +
- Lots of issues with spaces, newlines, etc causing the import to fail +
- I noticed we have a very interesting list of countries on CGSpace: +
-
+
- Not only are there 49,000 countries, we have some blanks (25)… +
- Also, lots of things like “COTE D`LVOIRE” and “COTE D IVOIRE” +
January, 2016
+ +2016-01-13
+ +-
+
- Move ILRI collection
10568/12503from10568/27869to10568/27629using the move_collections.sh script I wrote last year.
+ - I realized it is only necessary to clear the Cocoon cache after moving collections—rather than reindexing—as no metadata has changed, and therefore no search or browse indexes need to be updated. +
- Update GitHub wiki for documentation of maintenance tasks. +
December, 2015
+ +2015-12-02
+ +-
+
- Replace
lzopwithxzin log compression cron jobs on DSpace Test—it uses less space:
+
# cd /home/dspacetest.cgiar.org/log
+# ls -lh dspace.log.2015-11-18*
+-rw-rw-r-- 1 tomcat7 tomcat7 2.0M Nov 18 23:59 dspace.log.2015-11-18
+-rw-rw-r-- 1 tomcat7 tomcat7 387K Nov 18 23:59 dspace.log.2015-11-18.lzo
+-rw-rw-r-- 1 tomcat7 tomcat7 169K Nov 18 23:59 dspace.log.2015-11-18.xz
+November, 2015
+ +2015-11-22
+ +-
+
- CGSpace went down +
- Looks like DSpace exhausted its PostgreSQL connection pool +
- Last week I had increased the limit from 30 to 60, which seemed to help, but now there are many more idle connections: +
$ psql -c 'SELECT * from pg_stat_activity;' | grep idle | grep -c cgspace
+78
+
+
+
+
+
+
+
+
+
+
+ CGSpace Notes
+Documenting day-to-day work on the CGSpace repository.
+
+
+
+
+
+
+
+
+
+
+
diff --git a/docs/robots.txt b/docs/robots.txt
index 9d63922ff..118e7de64 100644
--- a/docs/robots.txt
+++ b/docs/robots.txt
@@ -35,5 +35,5 @@ Disallow: /cgspace-notes/
Disallow: /cgspace-notes/categories/
Disallow: /cgspace-notes/tags/notes/
Disallow: /cgspace-notes/categories/notes/
-Disallow: /cgspace-notes/post/
+Disallow: /cgspace-notes/posts/
Disallow: /cgspace-notes/tags/
diff --git a/docs/sitemap.xml b/docs/sitemap.xml
index 7ab32ef56..3e290afd7 100644
--- a/docs/sitemap.xml
+++ b/docs/sitemap.xml
@@ -4,157 +4,157 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Read more →
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Posts
+ +March, 2018
+ +2018-03-02
+ +-
+
- Export a CSV of the IITA community metadata for Martin Mueller +
February, 2018
+ +2018-02-01
+ +-
+
- Peter gave feedback on the
dc.rightsproof of concept that I had sent him last week
+ - We don’t need to distinguish between internal and external works, so that makes it just a simple list +
- Yesterday I figured out how to monitor DSpace sessions using JMX +
- I copied the logic in the
jmx_tomcat_dbpoolsprovided by Ubuntu’smunin-plugins-javapackage and used the stuff I discovered about JMX in 2018-01
+
January, 2018
+ +2018-01-02
+ +-
+
- Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time +
- I didn’t get any load alerts from Linode and the REST and XMLUI logs don’t show anything out of the ordinary +
- The nginx logs show HTTP 200s until
02/Jan/2018:11:27:17 +0000when Uptime Robot got an HTTP 500
+ - In dspace.log around that time I see many errors like “Client closed the connection before file download was complete” +
- And just before that I see this: +
Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000].
+-
+
- Ah hah! So the pool was actually empty! +
- I need to increase that, let’s try to bump it up from 50 to 75 +
- After that one client got an HTTP 499 but then the rest were HTTP 200, so I don’t know what the hell Uptime Robot saw +
- I notice this error quite a few times in dspace.log: +
2018-01-02 01:21:19,137 ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ Error while searching for sidebar facets
+org.dspace.discovery.SearchServiceException: org.apache.solr.search.SyntaxError: Cannot parse 'dateIssued_keyword:[1976+TO+1979]': Encountered " "]" "] "" at line 1, column 32.
+-
+
- And there are many of these errors every day for the past month: +
$ grep -c "Error while searching for sidebar facets" dspace.log.*
+dspace.log.2017-11-21:4
+dspace.log.2017-11-22:1
+dspace.log.2017-11-23:4
+dspace.log.2017-11-24:11
+dspace.log.2017-11-25:0
+dspace.log.2017-11-26:1
+dspace.log.2017-11-27:7
+dspace.log.2017-11-28:21
+dspace.log.2017-11-29:31
+dspace.log.2017-11-30:15
+dspace.log.2017-12-01:15
+dspace.log.2017-12-02:20
+dspace.log.2017-12-03:38
+dspace.log.2017-12-04:65
+dspace.log.2017-12-05:43
+dspace.log.2017-12-06:72
+dspace.log.2017-12-07:27
+dspace.log.2017-12-08:15
+dspace.log.2017-12-09:29
+dspace.log.2017-12-10:35
+dspace.log.2017-12-11:20
+dspace.log.2017-12-12:44
+dspace.log.2017-12-13:36
+dspace.log.2017-12-14:59
+dspace.log.2017-12-15:104
+dspace.log.2017-12-16:53
+dspace.log.2017-12-17:66
+dspace.log.2017-12-18:83
+dspace.log.2017-12-19:101
+dspace.log.2017-12-20:74
+dspace.log.2017-12-21:55
+dspace.log.2017-12-22:66
+dspace.log.2017-12-23:50
+dspace.log.2017-12-24:85
+dspace.log.2017-12-25:62
+dspace.log.2017-12-26:49
+dspace.log.2017-12-27:30
+dspace.log.2017-12-28:54
+dspace.log.2017-12-29:68
+dspace.log.2017-12-30:89
+dspace.log.2017-12-31:53
+dspace.log.2018-01-01:45
+dspace.log.2018-01-02:34
+-
+
- Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let’s Encrypt if it’s just a handful of domains +
December, 2017
+ +2017-12-01
+ +-
+
- Uptime Robot noticed that CGSpace went down +
- The logs say “Timeout waiting for idle object” +
- PostgreSQL activity says there are 115 connections currently +
- The list of connections to XMLUI and REST API for today: +
November, 2017
+ +2017-11-01
+ +-
+
- The CORE developers responded to say they are looking into their bot not respecting our robots.txt +
2017-11-02
+ +-
+
- Today there have been no hits by CORE and no alerts from Linode (coincidence?) +
# grep -c "CORE" /var/log/nginx/access.log
+0
+-
+
- Generate list of authors on CGSpace for Peter to go through and correct: +
dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors.csv with csv;
+COPY 54701
+October, 2017
+ +2017-10-01
+ +-
+
- Peter emailed to point out that many items in the ILRI archive collection have multiple handles: +
http://hdl.handle.net/10568/78495||http://hdl.handle.net/10568/79336
+-
+
- There appears to be a pattern but I’ll have to look a bit closer and try to clean them up automatically, either in SQL or in OpenRefine +
- Add Katherine Lutz to the groups for content submission and edit steps of the CGIAR System collections +
CGIAR Library Migration
+ +Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called CGIAR System Organization.
+ + + Read more → +September, 2017
+ +2017-09-06
+ +-
+
- Linode sent an alert that CGSpace (linode18) was using 261% CPU for the past two hours +
2017-09-07
+ +-
+
- Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account is both in the approvers step as well as the group +
August, 2017
+ +2017-08-01
+ +-
+
- Linode sent an alert that CGSpace (linode18) was using 350% CPU for the past two hours +
- I looked in the Activity pane of the Admin Control Panel and it seems that Google, Baidu, Yahoo, and Bing are all crawling with massive numbers of bots concurrently (~100 total, mostly Baidu and Google) +
- The good thing is that, according to
dspace.log.2017-08-01, they are all using the same Tomcat session
+ - This means our Tomcat Crawler Session Valve is working +
- But many of the bots are browsing dynamic URLs like:
+
+
-
+
- /handle/10568/3353/discover +
- /handle/10568/16510/browse +
+ - The
robots.txtonly blocks the top-level/discoverand/browseURLs… we will need to find a way to forbid them from accessing these!
+ - Relevant issue from DSpace Jira (semi resolved in DSpace 6.0): https://jira.duraspace.org/browse/DS-2962 +
- It turns out that we’re already adding the
X-Robots-Tag "none"HTTP header, but this only forbids the search engine from indexing the page, not crawling it!
+ - Also, the bot has to successfully browse the page first so it can receive the HTTP header… +
- We might actually have to block these requests with HTTP 403 depending on the user agent +
- Abenet pointed out that the CGIAR Library Historical Archive collection I sent July 20th only had ~100 entries, instead of 2415 +
- This was due to newline characters in the
dc.description.abstractcolumn, which caused OpenRefine to choke when exporting the CSV
+ - I exported a new CSV from the collection on DSpace Test and then manually removed the characters in vim using
g/^$/d
+ - Then I cleaned up the author authorities and HTML characters in OpenRefine and sent the file back to Abenet +
July, 2017
+ +2017-07-01
+ +-
+
- Run system updates and reboot DSpace Test +
2017-07-04
+ +-
+
- Merge changes for WLE Phase II theme rename (#329) +
- Looking at extracting the metadata registries from ICARDA’s MEL DSpace database so we can compare fields with CGSpace +
- We can use PostgreSQL’s extended output format (
-x) plussedto format the output into quasi XML:
+
CGIAR Library Migration
+ +Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called CGIAR System Organization.
+ + + Read more → +