pom.xml: Release version 5.2

We should make sure we are catching .JPG and .jpg. Also, we should
check for Generated Thumbnails as well as IM Thumbnail.

.github/workflows/maven.yml

@@ -1,24 +0,0 @@
-# This workflow will build a Java project with Maven
-# For more information see: https://help.github.com/actions/language-and-framework-guides/building-and-testing-java-with-maven
-
-name: Build
-
-on:
-  push:
-    branches: [ dspace6 ]
-  pull_request:
-    branches: [ dspace6 ]
-
-jobs:
-  build:
-
-    runs-on: ubuntu-latest
-
-    steps:
-    - uses: actions/checkout@v2
-    - name: Set up JDK 1.8
-      uses: actions/setup-java@v1
-      with:
-        java-version: 1.8
-    - name: Build with Maven
-      run: mvn -B package --file pom.xml

.travis.yml

@@ -0,0 +1,8 @@
+dist: bionic
+language: java
+jdk:
+  - openjdk8
+script:
+  - mvn package -B
+
+# vim: ts=2 sw=2 et

README.md

@@ -1,10 +1,10 @@
-# CGSpace Java Helpers [![GitHub Actions](https://github.com/ilri/cgspace-java-helpers/workflows/Build/badge.svg)](https://github.com/ilri/cgspace-java-helpers/actions)
+# CGSpace Java Helpers [![Build Status](https://travis-ci.org/ilri/cgspace-java-helpers.svg?branch=dspace5)](https://travis-ci.org/ilri/dspace-curation-tasks)
 DSpace curation tasks and other Java-based helpers used on the [CGSpace](https://cgspace.cgiar.org) institutional repository:
 
 - **CountryCodeTagger**: add ISO 3166-1 Alpha2 country codes to items based on their existing country metadata
 - **FixJpgJpgThumbnails**: Fix low-quality ".jpg.jpg" thumbnails by replacing them with their originals
 
-Tested on DSpace 6.3. Read more about the [DSpace curation system](https://wiki.lyrasis.org/display/DSDOC5x/Curation+System).
+Tested on DSpace 5.8. Read more about the [DSpace curation system](https://wiki.lyrasis.org/display/DSDOC5x/Curation+System).
 
 ## Build and Install
 
@@ -15,7 +15,7 @@ To use these curation tasks in a DSpace project add the following dependency to
 <dependency>
   <groupId>io.github.ilri.cgspace</groupId>
   <artifactId>cgspace-java-helpers</artifactId>
-  <version>6.1-SNAPSHOT</version>
+  <version>5.1</version>
 </dependency>
 ```
 
@@ -31,18 +31,42 @@ $ mvn package
 Copy the resulting jar to the DSpace `lib` directory:
 
 ```
-$ cp target/cgspace-java-helpers-6.1-SNAPSHOT.jar ~/dspace/lib/
+$ cp target/cgspace-java-helpers-5.1.jar ~/dspace/lib
 ```
 
 ## Configuration
-Please refer to the appropriate README.md file:
+Add the curation task to DSpace's `config/modules/curate.cfg`:
 
-- Curation Tasks: [src/main/java/io/github/ilri/cgspace/ctasks/README.md](https://github.com/ilri/cgspace-java-helpers/blob/dspace6/src/main/java/io/github/ilri/cgspace/ctasks/README.md)
+```
-- Scripts: [src/main/java/io/github/ilri/cgspace/scripts/README.md](https://github.com/ilri/cgspace-java-helpers/blob/dspace6/src/main/java/io/github/ilri/cgspace/scripts/README.md)
+plugin.named.org.dspace.curate.CurationTask = \
+...
+    io.github.ilri.cgspace.ctasks.CountryCodeTagger = countrycodetagger \
+    io.github.ilri.cgspace.ctasks.CountryCodeTagger = countrycodetagger.force
+```
 
-## Todo
+And then add a configuration file for the task in `config/modules/countrycodetagger.cfg`:
 
-- Add a curation task to normalize DOIs to "https://doi.org" format
+```
+# name of the field containing ISO 3166-1 country names
+iso3166.field = cg.coverage.country
+
+# name of the field containing ISO 3166-1 Alpha2 country codes
+iso3166-alpha2.field = cg.coverage.iso3166-alpha2
+
+# only add country codes if an item doesn't have any (default false)
+#forceupdate = false
+```
+
+*Note*: DSpace's curation system supports "profiles" where you can use the same task with different options, for example above I have a normal country code tagger and a "force" variant. To use the "force" variant you create a new configuration file with the overridden options in `config/modules/countrycodetagger.force.cfg`. The "force" profile clears all existing country codes and updates everything.
+
+## Invocation
+Once the jar is installed and you have added appropriate configuration in `~/dspace/config/modules`:
+
+```
+$ ~/dspace/bin/dspace curate -t countrycodetagger -i 10568/3 -r - -l 500 -s object
+```
+
+*Note*: it is very important to set the cache limit (`-l`) and the database transaction scope to something sensible (`object`) if you're curating a community or collection with more than a few hundred items.
 
 ## Notes
 This project was initially created according to the [Maven Getting Started Guide](https://maven.apache.org/guides/getting-started/):
@@ -51,6 +75,12 @@ This project was initially created according to the [Maven Getting Started Guide
 $ mvn -B archetype:generate -DgroupId=io.github.ilri.cgspace -DartifactId=cgspace-java-helpers -DarchetypeArtifactId=maven-archetype-quickstart -DarchetypeVersion=1.4
 ```
 
+## TODO
+
+- Make sure this doesn't work on items in the workflow
+- Check for existence of metadata field before trying to add metadata
+- Add tests
+
 ## License
 This work is licensed under the [GPLv3](https://www.gnu.org/licenses/gpl-3.0.en.html).
 

pom.xml

@@ -6,7 +6,7 @@
 
   <groupId>io.github.ilri.cgspace</groupId>
   <artifactId>cgspace-java-helpers</artifactId>
-  <version>6.1-SNAPSHOT</version>
+  <version>5.2</version>
 
   <name>cgspace-java-helpers</name>
   <url>https://github.com/ilri/cgspace-java-helpers</url>
@@ -42,12 +42,12 @@
     <dependency>
       <groupId>com.google.code.gson</groupId>
       <artifactId>gson</artifactId>
-      <version>2.6.1</version>
+      <version>2.2.1</version>
     </dependency>
     <dependency>
       <groupId>org.dspace</groupId>
       <artifactId>dspace-api</artifactId>
-      <version>6.3</version>
+      <version>5.8</version>
       <scope>provided</scope>
     </dependency>
   </dependencies>

src/main/java/io/github/ilri/cgspace/ctasks/CountryCodeTagger.java

@@ -23,7 +23,7 @@ import org.apache.log4j.Logger;
 import org.dspace.authorize.AuthorizeException;
 import org.dspace.content.DSpaceObject;
 import org.dspace.content.Item;
-import org.dspace.content.MetadataValue;
+import org.dspace.content.Metadatum;
 import org.dspace.core.Constants;
 import org.dspace.curate.AbstractCurationTask;
 import org.dspace.curate.Curator;
@@ -81,11 +81,7 @@ public class CountryCodeTagger extends AbstractCurationTask
 
             Item item = (Item)dso;
 
-            try {
+            alpha2Result = performAlpha2(item, config);
-                alpha2Result = performAlpha2(item, config);
-            } catch (SQLException throwables) {
-                throwables.printStackTrace();
-            }
 
             setResult(alpha2Result.getResult());
             report(alpha2Result.getResult());
@@ -94,14 +90,15 @@ public class CountryCodeTagger extends AbstractCurationTask
 		return alpha2Result.getStatus();
     }
 
-    public CountryCodeTaggerResult performAlpha2(Item item, CountryCodeTaggerConfig config) throws IOException, SQLException {
+    public CountryCodeTaggerResult performAlpha2(Item item, CountryCodeTaggerConfig config) throws IOException
+    {
         CountryCodeTaggerResult alpha2Result = new CountryCodeTaggerResult();
         String itemHandle = item.getHandle();
 
-        List<MetadataValue> itemCountries = itemService.getMetadataByMetadataString(item, config.iso3166Field);
+        Metadatum[] itemCountries = item.getMetadataByMetadataString(config.iso3166Field);
 
         // skip items that don't have country metadata
-        if (itemCountries.size() == 0) {
+        if (itemCountries.length == 0) {
             alpha2Result.setResult(itemHandle + ": no countries, skipping.");
             alpha2Result.setStatus(Curator.CURATE_SKIP);
         } else {
@@ -120,25 +117,25 @@ public class CountryCodeTagger extends AbstractCurationTask
             String[] iso3166Alpha2FieldParts = config.iso3166Alpha2Field.split("\\.");
 
             if (config.forceupdate) {
-                itemService.clearMetadata(Curator.curationContext(), item, iso3166Alpha2FieldParts[0], iso3166Alpha2FieldParts[1], iso3166Alpha2FieldParts[2], Item.ANY);
+                item.clearMetadata(iso3166Alpha2FieldParts[0], iso3166Alpha2FieldParts[1], iso3166Alpha2FieldParts[2], Item.ANY);
             }
 
             // check the item's country codes, if any
-            List<MetadataValue> itemAlpha2CountryCodes = itemService.getMetadataByMetadataString(item, config.iso3166Alpha2Field);
+            Metadatum[] itemAlpha2CountryCodes = item.getMetadataByMetadataString(config.iso3166Alpha2Field);
 
-            if (itemAlpha2CountryCodes.size() == 0) {
+            if (itemAlpha2CountryCodes.length == 0) {
                 List<String> newAlpha2Codes = new ArrayList<String>();
-                for (MetadataValue itemCountry : itemCountries) {
+                for (Metadatum itemCountry : itemCountries) {
                     //check ISO 3166-1 countries
                     for (CountriesVocabulary.Country country : isocodesCountriesJson.countries) {
-                        if (itemCountry.getValue().equalsIgnoreCase(country.getName()) || itemCountry.getValue().equalsIgnoreCase(country.get_official_name()) || itemCountry.getValue().equalsIgnoreCase(country.get_common_name())) {
+                        if (itemCountry.value.equalsIgnoreCase(country.getName()) || itemCountry.value.equalsIgnoreCase(country.get_official_name()) || itemCountry.value.equalsIgnoreCase(country.get_common_name())) {
                             newAlpha2Codes.add(country.getAlpha_2());
                         }
                     }
 
                     //check CGSpace countries
                     for (CountriesVocabulary.Country country : cgspaceCountriesJson.countries) {
-                        if (itemCountry.getValue().equalsIgnoreCase(country.getCgspace_name())) {
+                        if (itemCountry.value.equalsIgnoreCase(country.getCgspace_name())) {
                             newAlpha2Codes.add(country.getAlpha_2());
                         }
                     }
@@ -146,8 +143,9 @@ public class CountryCodeTagger extends AbstractCurationTask
 
                 if (newAlpha2Codes.size() > 0) {
                     try {
-                        itemService.addMetadata(Curator.curationContext(), item, iso3166Alpha2FieldParts[0], iso3166Alpha2FieldParts[1], iso3166Alpha2FieldParts[2], "en_US", newAlpha2Codes);
+                        // add metadata values (casting the List<String> to an array)
-                        itemService.update(Curator.curationContext(), item);
+                        item.addMetadata(iso3166Alpha2FieldParts[0], iso3166Alpha2FieldParts[1], iso3166Alpha2FieldParts[2], "en_US", newAlpha2Codes.toArray(new String[0]));
+                        item.update();
                     } catch (SQLException | AuthorizeException sqle) {
                         config.log.debug(sqle.getMessage());
                         alpha2Result.setResult(itemHandle + ": error");

src/main/java/io/github/ilri/cgspace/ctasks/README.md

@@ -1,72 +0,0 @@
-# Curation Tasks
-DSpace curation tasks used on the [CGSpace](https://cgspace.cgiar.org) institutional repository:
-
-- **CountryCodeTagger**: add ISO 3166-1 Alpha2 country codes to items based on their existing country metadata
-
-Tested on DSpace 6.3. Read more about the [DSpace curation system](https://wiki.lyrasis.org/display/DSDOC5x/Curation+System).
-
-## Build and Install
-
-### Integrate into DSpace Build
-To use these curation tasks in a DSpace project add the following dependency to `dspace/modules/additions/pom.xml`:
-
-```
-<dependency>
-  <groupId>io.github.ilri.cgspace</groupId>
-  <artifactId>cgspace-java-helpers</artifactId>
-  <version>6.1-SNAPSHOT</version>
-</dependency>
-```
-
-The jar will be copied to all DSpace applications.
-
-### Manual Build and Install
-To build the standalone jar:
-
-```
-$ mvn package
-```
-
-Copy the resulting jar to the DSpace `lib` directory:
-
-```
-$ cp target/cgspace-java-helpers-6.1-SNAPSHOT.jar ~/dspace/lib/
-```
-
-## Configuration
-Add the curation task to DSpace's `config/modules/curate.cfg`:
-
-```
-plugin.named.org.dspace.curate.CurationTask = io.github.ilri.cgspace.ctasks.CountryCodeTagger = countrycodetagger
-plugin.named.org.dspace.curate.CurationTask = io.github.ilri.cgspace.ctasks.CountryCodeTagger = countrycodetagger.force
-```
-
-And then add the following variables to your `local.cfg` or some other [configuration file that is included](https://wiki.lyrasis.org/display/DSDOC6x/Configuration+Reference#ConfigurationReference-IncludingotherPropertyFiles):
-
-```
-# name of the field containing ISO 3166-1 country names
-countrycodetagger.iso3166.field = cg.coverage.country
-
-# name of the field containing ISO 3166-1 Alpha2 country codes
-countrycodetagger.iso3166-alpha2.field = cg.coverage.iso3166-alpha2
-
-# only add country codes if an item doesn't have any (default false)
-#countrycodetagger.forceupdate = false
-```
-
-*Note*: DSpace's curation system supports "profiles" where you can use the same task with different options, for example above I have a normal country code tagger task and a "force" variant. The "force" variant is the same task, but it looks for configuration variables using the `countrycodetagger.force` instead. To use the "force" variant you simply need to add these new variables with the `forceupdate` parameter overridden to the same configuration file where you put the other variables. The "force" profile clears all existing country codes and updates everything.
-
-## Invocation
-Once the jar is installed and you have added appropriate configuration in `~/dspace/config/modules`:
-
-```
-$ ~/dspace/bin/dspace curate -t countrycodetagger -i 10568/3 -r - -s object
-```
-
-*Note*: it is very important to set the database transaction scope to something sensible (`object`) if you're curating a community or collection with more than a few hundred items.
-
-## TODO
-
-- Make sure this doesn't work on items in the workflow
-- Check for existence of metadata field before trying to add metadata
-- Add tests

src/main/java/io/github/ilri/cgspace/scripts/FixJpgJpgThumbnails.java

@@ -5,28 +5,18 @@ import org.dspace.authorize.AuthorizeException;
 import org.dspace.content.*;
 import org.dspace.core.Constants;
 import org.dspace.core.Context;
-import org.dspace.content.factory.ContentServiceFactory;
+import org.dspace.handle.HandleManager;
-import org.dspace.content.service.ItemService;
-import org.dspace.handle.factory.HandleServiceFactory;
-import org.dspace.handle.service.HandleService;
-import org.dspace.content.service.BundleService;
 
 import java.io.IOException;
 import java.sql.SQLException;
-import java.util.Iterator;
-import java.util.List;
 
 /**
  * @author Andrea Schweer schweer@waikato.ac.nz for the LCoNZ Institutional Research Repositories
  * @author Alan Orth for the International Livestock Research Institute
- * @version 6.1
+ * @version 5.1-SNAPSHOT
- * @since 5.1
+ * @since 5.1-SNAPSHOT
  */
 public class FixJpgJpgThumbnails {
-	//note: static members belong to the class itself, not any one instance
-	public static ItemService itemService = ContentServiceFactory.getInstance().getItemService();
-	public static HandleService handleService = HandleServiceFactory.getInstance().getHandleService();
-	public static BundleService bundleService = ContentServiceFactory.getInstance().getBundleService();
 
 	public static void main(String[] args) {
 		String parentHandle = null;
@@ -40,25 +30,25 @@ public class FixJpgJpgThumbnails {
 			context.turnOffAuthorisationSystem();
 
 			if (StringUtils.isBlank(parentHandle)) {
-				process(context, itemService.findAll(context));
+				process(context, Item.findAll(context));
 			} else {
-				DSpaceObject parent = handleService.resolveToObject(context, parentHandle);
+				DSpaceObject parent = HandleManager.resolveToObject(context, parentHandle);
 				if (parent != null) {
 					switch (parent.getType()) {
 						case Constants.COLLECTION:
-							process(context, itemService.findByCollection(context, (Collection) parent));
+							process(context, ((Collection) parent).getAllItems()); // getAllItems because we want to work on non-archived ones as well
 							break;
 						case Constants.COMMUNITY:
-							List<Collection> collections = ((Community) parent).getCollections();
+							Collection[] collections = ((Community) parent).getCollections();
 							for (Collection collection : collections) {
-								process(context, itemService.findAllByCollection(context, collection));
+								process(context, collection.getAllItems()); // getAllItems because we want to work on non-archived ones as well
 							}
 							break;
 						case Constants.SITE:
-							process(context, itemService.findAll(context));
+							process(context, Item.findAll(context));
 							break;
 						case Constants.ITEM:
-							processItem(context, (Item) parent);
+							processItem((Item) parent);
 							context.commit();
 							break;
 					}
@@ -73,59 +63,36 @@ public class FixJpgJpgThumbnails {
 		}
 	}
 
-	private static void process(Context context, Iterator<Item> items) throws SQLException, IOException, AuthorizeException {
+	private static void process(Context context, ItemIterator items) throws SQLException, IOException, AuthorizeException {
 		while (items.hasNext()) {
 			Item item = items.next();
-			processItem(context, item);
+			processItem(item);
-			itemService.update(context, item);
+			context.commit();
+			item.decache();
 		}
 	}
 
-	private static void processItem(Context context, Item item) throws SQLException, AuthorizeException, IOException {
+	private static void processItem(Item item) throws SQLException, AuthorizeException, IOException {
-		// Some bitstreams like Infographics are large JPGs and put in the ORIGINAL bundle on purpose so we shouldn't
+		Bundle[] thumbnailBundles = item.getBundles("THUMBNAIL");
-		// swap them.
-		List<MetadataValue> itemTypes = itemService.getMetadataByMetadataString(item, "dcterms.type");
-		boolean itemHasInfographic = false;
-		for (MetadataValue itemType: itemTypes) {
-			if (itemType.getValue().equals("Infographic")) {
-				itemHasInfographic = true;
-			}
-		}
-
-		List<Bundle> thumbnailBundles = item.getBundles("THUMBNAIL");
 		for (Bundle thumbnailBundle : thumbnailBundles) {
-			List<Bitstream> thumbnailBundleBitstreams = thumbnailBundle.getBitstreams();
+			Bitstream[] thumbnailBundleBitstreams = thumbnailBundle.getBitstreams();
 			for (Bitstream thumbnailBitstream : thumbnailBundleBitstreams) {
 				String thumbnailName = thumbnailBitstream.getName();
 
 				if (thumbnailName.toLowerCase().contains(".jpg.jpg")) {
-					List<Bundle> originalBundles = item.getBundles("ORIGINAL");
+					Bundle[] originalBundles = item.getBundles("ORIGINAL");
 					for (Bundle originalBundle : originalBundles) {
-						List<Bitstream> originalBundleBitstreams = originalBundle.getBitstreams();
+						Bitstream[] originalBundleBitstreams = originalBundle.getBitstreams();
 
-						for (Bitstream originalBitstream : originalBundleBitstreams) {
+						for(Bitstream originalBitstream : originalBundleBitstreams) {
 							String originalName = originalBitstream.getName();
 
-							long originalBitstreamBytes = originalBitstream.getSize();
+							//check if the original file name is the same as the thumbnail name minus the extra ".jpg"
-
+							if (originalName.equalsIgnoreCase(StringUtils.removeEndIgnoreCase(thumbnailName, ".jpg")) && ("Generated Thumbnail".equals(thumbnailBitstream.getDescription()) || "IM Thumbnail".equals(thumbnailBitstream.getDescription()))) {
-							/*
-							- check if the original file name is the same as the thumbnail name minus the extra ".jpg"
-							- check if the thumbnail description indicates it was automatically generated
-							- check if the item has dc.type Infographic (JPG could be the "real" item!)
-							- check if the original bitstream is less than ~100KiB
-							    - Note: in my tests there were 4022 items with ".jpg.jpg" thumbnails totaling 394549249
-							      bytes for an average of about 98KiB so ~100KiB seems like a good cut off
-							*/
-							if (
-									originalName.equalsIgnoreCase(StringUtils.removeEndIgnoreCase(thumbnailName, ".jpg"))
-									&& ("Generated Thumbnail".equals(thumbnailBitstream.getDescription()) || "IM Thumbnail".equals(thumbnailBitstream.getDescription()))
-									&& !itemHasInfographic
-									&& originalBitstreamBytes < 100000
-							) {
 								System.out.println(item.getHandle() + ": replacing " + thumbnailName + " with " + originalName);
 
 								//add the original bitstream to the THUMBNAIL bundle
-								bundleService.addBitstream(context, thumbnailBundle, originalBitstream);
+								thumbnailBundle.addBitstream(originalBitstream);
 								//remove the original bitstream from the ORIGINAL bundle
 								originalBundle.removeBitstream(originalBitstream);
 								//remove the JpgJpg bitstream from the THUMBNAIL bundle

src/main/java/io/github/ilri/cgspace/scripts/README.md

@@ -1,41 +0,0 @@
-# Scripts
-Java-based helpers used on the [CGSpace](https://cgspace.cgiar.org) institutional repository:
-
-- **FixJpgJpgThumbnails**: Fix low-quality ".jpg.jpg" thumbnails by replacing them with their originals
-
-Tested on DSpace 6.3. Read more about the [DSpace curation system](https://wiki.lyrasis.org/display/DSDOC5x/Curation+System).
-
-## Build and Install
-
-### Integrate into DSpace Build
-To use these curation tasks in a DSpace project add the following dependency to `dspace/modules/additions/pom.xml`:
-
-```
-<dependency>
-  <groupId>io.github.ilri.cgspace</groupId>
-  <artifactId>cgspace-java-helpers</artifactId>
-  <version>6.1-SNAPSHOT</version>
-</dependency>
-```
-
-The jar will be copied to all DSpace applications.
-
-### Manual Build and Install
-To build the standalone jar:
-
-```
-$ mvn package
-```
-
-Copy the resulting jar to the DSpace `lib` directory:
-
-```
-$ cp target/cgspace-java-helpers-6.1-SNAPSHOT.jar ~/dspace/lib/
-```
-
-## Invocation
-The script only takes one argument, which is a community, collection, or item:
-
-```
-$ dspace dsrun io.github.ilri.cgspace.scripts.FixJpgJpgThumbnails 10568/83389
-```