CHANGELOG.md: Update changes and move to 0.4.2

This was inserting correctly on the first run, but subsequent runs
were inserting into the incorrect column on conflict. This made it
seem like there were downloads for items where there were none.

CHANGELOG.md

@@ -4,6 +4,15 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.4.2] - 2018-10-04
+### Changed
+- README.md introduction and requirements
+- Use ujson instead of json
+- Iterate directly on SQL cursor in `/items` route
+
+### Fixed
+- Logic error in SQL for item views
+
 ## [0.4.1] - 2018-09-26
 ### Changed
 - Use execute_values() to batch insert records to PostgreSQL

README.md

@@ -1,16 +1,28 @@
 # DSpace Statistics API
-A quick and dirty REST API to expose Solr view and download statistics for items in a DSpace repository.
+A simple REST API to expose Solr view and download statistics for items in a DSpace repository. This project contains a standalone indexing component and a WSGI application.
 
-Written and tested in Python 3.5, 3.6, and 3.7. Requires PostgreSQL version 9.5 or greater for [`UPSERT` support](https://wiki.postgresql.org/wiki/UPSERT).
+## Requirements
 
-## Installation
-Create a virtual environment and run it:
+- Python 3.5+
+- PostgreSQL version 9.5+ (due to [`UPSERT` support](https://wiki.postgresql.org/wiki/UPSERT))
+- DSpace 4+ with [Solr usage statistics enabled](https://wiki.duraspace.org/display/DSDOC5x/SOLR+Statistics)
+
+## Installation and Testing
+Create a Python virtual environment and install the dependencies:
 
     $ python -m venv venv
     $ . venv/bin/activate
     $ pip install -r requirements.txt
+
+Set up the environment variables Solr and PostgreSQL:
+
+    $ export SOLR_SERVER=http://localhost:8080/solr
+    $ 
     $ gunicorn app:api
 
+## Deployment
+There are example systemd service and timer units in the `contrib` directory.
+
 ## Using the API
 The API exposes the following endpoints:
 
@@ -25,6 +37,9 @@ The API exposes the following endpoints:
 - Close up DB connection when gunicorn shuts down gracefully
 - Better logging
 - Tests
+- Check if database exists (try/except)
+- Version API
+- Use JSON in PostgreSQL
 
 ## License
 This work is licensed under the [GPLv3](https://www.gnu.org/licenses/gpl-3.0.en.html).

app.py

@@ -22,16 +22,16 @@ class AllItemsResource:
 
         # get statistics, ordered by id, and use limit and offset to page through results
         cursor.execute('SELECT id, views, downloads FROM items ORDER BY id ASC LIMIT {} OFFSET {}'.format(limit, offset))
-        results = cursor.fetchmany(limit)
-        cursor.close()
 
         # create a list to hold dicts of item stats
         statistics = list()
 
         # iterate over results and build statistics object
-        for item in results:
+        for item in cursor:
             statistics.append({ 'id': item['id'], 'views': item['views'], 'downloads': item['downloads'] })
 
+        cursor.close()
+
         message = {
                 'currentPage': page,
                 'totalPages': pages,

indexer.py

@@ -31,7 +31,7 @@
 # See: https://wiki.duraspace.org/display/DSPACE/Solr
 
 from database import database_connection
-import json
+import ujson
 import psycopg2.extras
 from solr import solr_connection
 
@@ -56,7 +56,7 @@ def index_views():
     }, rows=0)
 
     # get total number of distinct facets (countDistinct)
-    results_totalNumFacets = json.loads(res.get_json())['stats']['stats_fields']['id']['countDistinct']
+    results_totalNumFacets = ujson.loads(res.get_json())['stats']['stats_fields']['id']['countDistinct']
 
     # divide results into "pages" (cast to int to effectively round down)
     results_per_page = 100
@@ -88,7 +88,7 @@ def index_views():
             data.append((item_id, item_views))
 
         # do a batch insert of values from the current "page" of results
-        sql = 'INSERT INTO items(id, views) VALUES %s ON CONFLICT(id) DO UPDATE SET downloads=excluded.views'
+        sql = 'INSERT INTO items(id, views) VALUES %s ON CONFLICT(id) DO UPDATE SET views=excluded.views'
         psycopg2.extras.execute_values(cursor, sql, data, template='(%s, %s)')
         db.commit()
 
@@ -115,7 +115,7 @@ def index_downloads():
     }, rows=0)
 
     # get total number of distinct facets (countDistinct)
-    results_totalNumFacets = json.loads(res.get_json())['stats']['stats_fields']['owningItem']['countDistinct']
+    results_totalNumFacets = ujson.loads(res.get_json())['stats']['stats_fields']['owningItem']['countDistinct']
 
     # divide results into "pages" (cast to int to effectively round down)
     results_per_page = 100

requirements.txt

@@ -9,4 +9,5 @@ python-mimeparse==1.6.0
 requests==2.19.1
 six==1.11.0
 SolrClient==0.2.1
+ujson==1.35
 urllib3==1.23