Version 5.3

Make sure we don't modify thumbnails if the item is an Infographic
because the JPG in the ORIGINAL bundle might actually be the "real"
file, in which case the THUMBNAIL bundle would have a legitimate
".jpg.jpg" file.

Also, limit the criteria for replacement to original bitstreams
that are less than 100KiB. In my tests I found that we had 4,022
items with ".jpg.jpg" thumbnails, and the average file size of the
originals in those items was 98KiB. Without considering the large
inforgraphics, which are several megabytes apiece, the average of
the remaining 3,765 originals was ~20KiB so 100KiB should be very
safe.

README.md

@@ -1,10 +1,10 @@
-# CGSpace Java Helpers [![Build Status](https://travis-ci.org/ilri/cgspace-java-helpers.svg?branch=dspace6)](https://travis-ci.org/ilri/cgspace-java-helpers)
+# CGSpace Java Helpers [![Build Status](https://travis-ci.org/ilri/cgspace-java-helpers.svg?branch=dspace5)](https://travis-ci.org/ilri/dspace-curation-tasks)
 DSpace curation tasks and other Java-based helpers used on the [CGSpace](https://cgspace.cgiar.org) institutional repository:
 
 - **CountryCodeTagger**: add ISO 3166-1 Alpha2 country codes to items based on their existing country metadata
 - **FixJpgJpgThumbnails**: Fix low-quality ".jpg.jpg" thumbnails by replacing them with their originals
 
-Tested on DSpace 6.3. Read more about the [DSpace curation system](https://wiki.lyrasis.org/display/DSDOC5x/Curation+System).
+Tested on DSpace 5.8. Read more about the [DSpace curation system](https://wiki.lyrasis.org/display/DSDOC5x/Curation+System).
 
 ## Build and Install
 
@@ -15,7 +15,7 @@ To use these curation tasks in a DSpace project add the following dependency to
 <dependency>
   <groupId>io.github.ilri.cgspace</groupId>
   <artifactId>cgspace-java-helpers</artifactId>
-  <version>6.0</version>
+  <version>5.3</version>
 </dependency>
 ```
 
@@ -31,31 +31,33 @@ $ mvn package
 Copy the resulting jar to the DSpace `lib` directory:
 
 ```
-$ cp target/cgspace-java-helpers-6.0.jar ~/dspace/lib/
+$ cp target/cgspace-java-helpers-5.3.jar ~/dspace/lib
 ```
 
 ## Configuration
 Add the curation task to DSpace's `config/modules/curate.cfg`:
 
 ```
-plugin.named.org.dspace.curate.CurationTask = io.github.ilri.cgspace.ctasks.CountryCodeTagger = countrycodetagger
+plugin.named.org.dspace.curate.CurationTask = \
-plugin.named.org.dspace.curate.CurationTask = io.github.ilri.cgspace.ctasks.CountryCodeTagger = countrycodetagger.force
+...
+    io.github.ilri.cgspace.ctasks.CountryCodeTagger = countrycodetagger \
+    io.github.ilri.cgspace.ctasks.CountryCodeTagger = countrycodetagger.force
 ```
 
-And then add the following variables to your `local.cfg` or some other [configuration file that is included](https://wiki.lyrasis.org/display/DSDOC6x/Configuration+Reference#ConfigurationReference-IncludingotherPropertyFiles):
+And then add a configuration file for the task in `config/modules/countrycodetagger.cfg`:
 
 ```
 # name of the field containing ISO 3166-1 country names
-countrycodetagger.iso3166.field = cg.coverage.country
+iso3166.field = cg.coverage.country
 
 # name of the field containing ISO 3166-1 Alpha2 country codes
-countrycodetagger.iso3166-alpha2.field = cg.coverage.iso3166-alpha2
+iso3166-alpha2.field = cg.coverage.iso3166-alpha2
 
 # only add country codes if an item doesn't have any (default false)
-#countrycodetagger.forceupdate = false
+#forceupdate = false
 ```
 
-*Note*: DSpace's curation system supports "profiles" where you can use the same task with different options, for example above I have a normal country code tagger task and a "force" variant. The "force" variant is the same task, but it looks for configuration variables using the `countrycodetagger.force` instead. To use the "force" variant you simply need to add these new variables with the `forceupdate` parameter overridden to the same configuration file where you put the other variables. The "force" profile clears all existing country codes and updates everything.
+*Note*: DSpace's curation system supports "profiles" where you can use the same task with different options, for example above I have a normal country code tagger and a "force" variant. To use the "force" variant you create a new configuration file with the overridden options in `config/modules/countrycodetagger.force.cfg`. The "force" profile clears all existing country codes and updates everything.
 
 ## Invocation
 Once the jar is installed and you have added appropriate configuration in `~/dspace/config/modules`:

pom.xml

@@ -6,7 +6,7 @@
 
   <groupId>io.github.ilri.cgspace</groupId>
   <artifactId>cgspace-java-helpers</artifactId>
-  <version>6.0</version>
+  <version>5.3</version>
 
   <name>cgspace-java-helpers</name>
   <url>https://github.com/ilri/cgspace-java-helpers</url>
@@ -42,12 +42,12 @@
     <dependency>
       <groupId>com.google.code.gson</groupId>
       <artifactId>gson</artifactId>
-      <version>2.6.1</version>
+      <version>2.2.1</version>
     </dependency>
     <dependency>
       <groupId>org.dspace</groupId>
       <artifactId>dspace-api</artifactId>
-      <version>6.3</version>
+      <version>5.8</version>
       <scope>provided</scope>
     </dependency>
   </dependencies>

src/main/java/io/github/ilri/cgspace/ctasks/CountryCodeTagger.java

@@ -23,7 +23,7 @@ import org.apache.log4j.Logger;
 import org.dspace.authorize.AuthorizeException;
 import org.dspace.content.DSpaceObject;
 import org.dspace.content.Item;
-import org.dspace.content.MetadataValue;
+import org.dspace.content.Metadatum;
 import org.dspace.core.Constants;
 import org.dspace.curate.AbstractCurationTask;
 import org.dspace.curate.Curator;
@@ -81,11 +81,7 @@ public class CountryCodeTagger extends AbstractCurationTask
 
             Item item = (Item)dso;
 
-            try {
             alpha2Result = performAlpha2(item, config);
-            } catch (SQLException throwables) {
-                throwables.printStackTrace();
-            }
 
             setResult(alpha2Result.getResult());
             report(alpha2Result.getResult());
@@ -94,14 +90,15 @@ public class CountryCodeTagger extends AbstractCurationTask
 		return alpha2Result.getStatus();
     }
 
-    public CountryCodeTaggerResult performAlpha2(Item item, CountryCodeTaggerConfig config) throws IOException, SQLException {
+    public CountryCodeTaggerResult performAlpha2(Item item, CountryCodeTaggerConfig config) throws IOException
+    {
         CountryCodeTaggerResult alpha2Result = new CountryCodeTaggerResult();
         String itemHandle = item.getHandle();
 
-        List<MetadataValue> itemCountries = itemService.getMetadataByMetadataString(item, config.iso3166Field);
+        Metadatum[] itemCountries = item.getMetadataByMetadataString(config.iso3166Field);
 
         // skip items that don't have country metadata
-        if (itemCountries.size() == 0) {
+        if (itemCountries.length == 0) {
             alpha2Result.setResult(itemHandle + ": no countries, skipping.");
             alpha2Result.setStatus(Curator.CURATE_SKIP);
         } else {
@@ -120,25 +117,25 @@ public class CountryCodeTagger extends AbstractCurationTask
             String[] iso3166Alpha2FieldParts = config.iso3166Alpha2Field.split("\\.");
 
             if (config.forceupdate) {
-                itemService.clearMetadata(Curator.curationContext(), item, iso3166Alpha2FieldParts[0], iso3166Alpha2FieldParts[1], iso3166Alpha2FieldParts[2], Item.ANY);
+                item.clearMetadata(iso3166Alpha2FieldParts[0], iso3166Alpha2FieldParts[1], iso3166Alpha2FieldParts[2], Item.ANY);
             }
 
             // check the item's country codes, if any
-            List<MetadataValue> itemAlpha2CountryCodes = itemService.getMetadataByMetadataString(item, config.iso3166Alpha2Field);
+            Metadatum[] itemAlpha2CountryCodes = item.getMetadataByMetadataString(config.iso3166Alpha2Field);
 
-            if (itemAlpha2CountryCodes.size() == 0) {
+            if (itemAlpha2CountryCodes.length == 0) {
                 List<String> newAlpha2Codes = new ArrayList<String>();
-                for (MetadataValue itemCountry : itemCountries) {
+                for (Metadatum itemCountry : itemCountries) {
                     //check ISO 3166-1 countries
                     for (CountriesVocabulary.Country country : isocodesCountriesJson.countries) {
-                        if (itemCountry.getValue().equalsIgnoreCase(country.getName()) || itemCountry.getValue().equalsIgnoreCase(country.get_official_name()) || itemCountry.getValue().equalsIgnoreCase(country.get_common_name())) {
+                        if (itemCountry.value.equalsIgnoreCase(country.getName()) || itemCountry.value.equalsIgnoreCase(country.get_official_name()) || itemCountry.value.equalsIgnoreCase(country.get_common_name())) {
                             newAlpha2Codes.add(country.getAlpha_2());
                         }
                     }
 
                     //check CGSpace countries
                     for (CountriesVocabulary.Country country : cgspaceCountriesJson.countries) {
-                        if (itemCountry.getValue().equalsIgnoreCase(country.getCgspace_name())) {
+                        if (itemCountry.value.equalsIgnoreCase(country.getCgspace_name())) {
                             newAlpha2Codes.add(country.getAlpha_2());
                         }
                     }
@@ -146,8 +143,9 @@ public class CountryCodeTagger extends AbstractCurationTask
 
                 if (newAlpha2Codes.size() > 0) {
                     try {
-                        itemService.addMetadata(Curator.curationContext(), item, iso3166Alpha2FieldParts[0], iso3166Alpha2FieldParts[1], iso3166Alpha2FieldParts[2], "en_US", newAlpha2Codes);
+                        // add metadata values (casting the List<String> to an array)
-                        itemService.update(Curator.curationContext(), item);
+                        item.addMetadata(iso3166Alpha2FieldParts[0], iso3166Alpha2FieldParts[1], iso3166Alpha2FieldParts[2], "en_US", newAlpha2Codes.toArray(new String[0]));
+                        item.update();
                     } catch (SQLException | AuthorizeException sqle) {
                         config.log.debug(sqle.getMessage());
                         alpha2Result.setResult(itemHandle + ": error");

src/main/java/io/github/ilri/cgspace/scripts/FixJpgJpgThumbnails.java

@@ -5,28 +5,18 @@ import org.dspace.authorize.AuthorizeException;
 import org.dspace.content.*;
 import org.dspace.core.Constants;
 import org.dspace.core.Context;
-import org.dspace.content.factory.ContentServiceFactory;
+import org.dspace.handle.HandleManager;
-import org.dspace.content.service.ItemService;
-import org.dspace.handle.factory.HandleServiceFactory;
-import org.dspace.handle.service.HandleService;
-import org.dspace.content.service.BundleService;
 
 import java.io.IOException;
 import java.sql.SQLException;
-import java.util.Iterator;
-import java.util.List;
 
 /**
  * @author Andrea Schweer schweer@waikato.ac.nz for the LCoNZ Institutional Research Repositories
  * @author Alan Orth for the International Livestock Research Institute
- * @version 6.0
+ * @version 5.3
  * @since 5.1
  */
 public class FixJpgJpgThumbnails {
-	//note: static members belong to the class itself, not any one instance
-	public static ItemService itemService = ContentServiceFactory.getInstance().getItemService();
-	public static HandleService handleService = HandleServiceFactory.getInstance().getHandleService();
-	public static BundleService bundleService = ContentServiceFactory.getInstance().getBundleService();
 
 	public static void main(String[] args) {
 		String parentHandle = null;
@@ -40,25 +30,25 @@ public class FixJpgJpgThumbnails {
 			context.turnOffAuthorisationSystem();
 
 			if (StringUtils.isBlank(parentHandle)) {
-				process(context, itemService.findAll(context));
+				process(context, Item.findAll(context));
 			} else {
-				DSpaceObject parent = handleService.resolveToObject(context, parentHandle);
+				DSpaceObject parent = HandleManager.resolveToObject(context, parentHandle);
 				if (parent != null) {
 					switch (parent.getType()) {
 						case Constants.COLLECTION:
-							process(context, itemService.findByCollection(context, (Collection) parent));
+							process(context, ((Collection) parent).getAllItems()); // getAllItems because we want to work on non-archived ones as well
 							break;
 						case Constants.COMMUNITY:
-							List<Collection> collections = ((Community) parent).getCollections();
+							Collection[] collections = ((Community) parent).getCollections();
 							for (Collection collection : collections) {
-								process(context, itemService.findAllByCollection(context, collection));
+								process(context, collection.getAllItems()); // getAllItems because we want to work on non-archived ones as well
 							}
 							break;
 						case Constants.SITE:
-							process(context, itemService.findAll(context));
+							process(context, Item.findAll(context));
 							break;
 						case Constants.ITEM:
-							processItem(context, (Item) parent);
+							processItem((Item) parent);
 							context.commit();
 							break;
 					}
@@ -73,40 +63,41 @@ public class FixJpgJpgThumbnails {
 		}
 	}
 
-	private static void process(Context context, Iterator<Item> items) throws SQLException, IOException, AuthorizeException {
+	private static void process(Context context, ItemIterator items) throws SQLException, IOException, AuthorizeException {
 		while (items.hasNext()) {
 			Item item = items.next();
-			processItem(context, item);
+			processItem(item);
-			itemService.update(context, item);
+			context.commit();
+			item.decache();
 		}
 	}
 
-	private static void processItem(Context context, Item item) throws SQLException, AuthorizeException, IOException {
+	private static void processItem(Item item) throws SQLException, AuthorizeException, IOException {
 		// Some bitstreams like Infographics are large JPGs and put in the ORIGINAL bundle on purpose so we shouldn't
 		// swap them.
-		List<MetadataValue> itemTypes = itemService.getMetadataByMetadataString(item, "dc.type");
+		Metadatum[] itemTypes = item.getMetadataByMetadataString("dc.type");
-		boolean itemHasInfographic = false;
+		Boolean itemHasInfographic = false;
-		for (MetadataValue itemType: itemTypes) {
+		for (Metadatum itemType: itemTypes) {
-			if (itemType.getValue().equals("Infographic")) {
+			if (itemType.value.equals("Infographic")) {
 				itemHasInfographic = true;
 			}
 		}
 
-		List<Bundle> thumbnailBundles = item.getBundles("THUMBNAIL");
+		Bundle[] thumbnailBundles = item.getBundles("THUMBNAIL");
 		for (Bundle thumbnailBundle : thumbnailBundles) {
-			List<Bitstream> thumbnailBundleBitstreams = thumbnailBundle.getBitstreams();
+			Bitstream[] thumbnailBundleBitstreams = thumbnailBundle.getBitstreams();
 			for (Bitstream thumbnailBitstream : thumbnailBundleBitstreams) {
 				String thumbnailName = thumbnailBitstream.getName();
 
 				if (thumbnailName.toLowerCase().contains(".jpg.jpg")) {
-					List<Bundle> originalBundles = item.getBundles("ORIGINAL");
+					Bundle[] originalBundles = item.getBundles("ORIGINAL");
 					for (Bundle originalBundle : originalBundles) {
-						List<Bitstream> originalBundleBitstreams = originalBundle.getBitstreams();
+						Bitstream[] originalBundleBitstreams = originalBundle.getBitstreams();
 
 						for (Bitstream originalBitstream : originalBundleBitstreams) {
 							String originalName = originalBitstream.getName();
 
-							long originalBitstreamBytes = originalBitstream.getSize();
+							Long originalBitstreamBytes = originalBitstream.getSize();
 
 							/*
 							- check if the original file name is the same as the thumbnail name minus the extra ".jpg"
@@ -125,7 +116,7 @@ public class FixJpgJpgThumbnails {
 								System.out.println(item.getHandle() + ": replacing " + thumbnailName + " with " + originalName);
 
 								//add the original bitstream to the THUMBNAIL bundle
-								bundleService.addBitstream(context, thumbnailBundle, originalBitstream);
+								thumbnailBundle.addBitstream(originalBitstream);
 								//remove the original bitstream from the ORIGINAL bundle
 								originalBundle.removeBitstream(originalBitstream);
 								//remove the JpgJpg bitstream from the THUMBNAIL bundle