CHANGELOD.md: Add note about ujson

Falcon can optionally use ujson to speed up JSON (de)serialization,
but Falcon's already really fast and requiring ujson actually makes
deployment trickier in some cases (for example in Docker containers
that are based on Alpine Linux).

Here are some tests of Falcon 1.4.1 on Python 3.5 from my laptop:

    1. falcon...............60172 req/sec or 16.62 μs/req (36x)
    2. falcon-ext...........34186 req/sec or 29.25 μs/req (20x)
    3. bottle...............32924 req/sec or 30.37 μs/req (20x)
    4. werkzeug.............11948 req/sec or 83.70 μs/req (7x)
    5. flask.................6654 req/sec or 150.30 μs/req (4x)
    6. django................4565 req/sec or 219.04 μs/req (3x)
    7. pecan.................1672 req/sec or 598.19 μs/req (1x)

The tests were conducted with Falcon's official Docker benchmarking
tools on my Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz on Arch Linux.

See: https://github.com/falconry/falcon/tree/master/docker

.travis.yml

@@ -2,8 +2,10 @@ language: python
 python:
   - "3.5"
   - "3.6"
-  - "3.7"
-install:
-  - pip install -r requirements.txt
+  - "3.7-dev"
+script: pip install -r requirements.txt
+branches:
+  only:
+    - master
 
 # vim: ts=2 sw=2 et

CHANGELOG.md

@@ -4,6 +4,46 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+### [0.5.1] - 2018-10-24
+## Changed
+- Use Python's native json instead of ujson
+
+### [0.5.0] - 2018-10-24
+## Added
+- Example nginx configuration to README.md
+
+## Changed
+- Don't initialize Solr connection in API
+
+### [0.4.3] - 2018-10-17
+## Changed
+- Use pip install as script for Travis CI
+
+## Improved
+- Documentation for deployment and testing
+
+## [0.4.2] - 2018-10-04
+### Changed
+- README.md introduction and requirements
+- Use ujson instead of json
+- Iterate directly on SQL cursor in `/items` route
+
+### Fixed
+- Logic error in SQL for item views
+
+## [0.4.1] - 2018-09-26
+### Changed
+- Use execute_values() to batch insert records to PostgreSQL
+
+## [0.4.0] - 2018-09-25
+### Fixed
+- Invalid OnCalendar syntax in dspace-statistics-indexer.timer
+- Major logic error in indexer.py
+
+## [0.3.2] - 2018-09-25
+## Changed
+- /item/id route now returns HTTP 404 if an item is not found
+
 ## [0.3.1] - 2018-09-25
 ### Changed
 - Force SolrClient's kazoo dependency to version 2.5.0 to work with Python 3.7

README.md

@@ -1,31 +1,79 @@
-# DSpace Statistics API
-A quick and dirty REST API to expose Solr view and download statistics for items in a DSpace repository.
+# DSpace Statistics API [![Build Status](https://travis-ci.org/alanorth/dspace-statistics-api.svg?branch=master)](https://travis-ci.org/alanorth/dspace-statistics-api)
+A simple REST API to expose Solr view and download statistics for items in a DSpace repository. This project contains a standalone indexing component and a WSGI application.
 
-Written and tested in Python 3.5, 3.6, and 3.7. Requires PostgreSQL version 9.5 or greater for [`UPSERT` support](https://wiki.postgresql.org/wiki/UPSERT).
+## Requirements
 
-## Installation
-Create a virtual environment and run it:
+- Python 3.5+
+- PostgreSQL version 9.5+ (due to [`UPSERT` support](https://wiki.postgresql.org/wiki/UPSERT))
+- DSpace 4+ with [Solr usage statistics enabled](https://wiki.duraspace.org/display/DSDOC5x/SOLR+Statistics)
+
+## Installation and Testing
+Create a Python virtual environment and install the dependencies:
 
     $ python -m venv venv
     $ . venv/bin/activate
     $ pip install -r requirements.txt
+
+Set up the environment variables for Solr and PostgreSQL:
+
+    $ export SOLR_SERVER=http://localhost:8080/solr
+    $ export DATABASE_NAME=dspacestatistics
+    $ export DATABASE_USER=dspacestatistics
+    $ export DATABASE_PASS=dspacestatistics
+    $ export DATABASE_HOST=localhost
+
+Index the Solr statistics core to populate the PostgreSQL database:
+
+    $ ./indexer.py
+
+Run the REST API:
+
     $ gunicorn app:api
 
+Test to see if there are any statistics:
+
+    $ curl 'http://localhost:8000/items?limit=1'
+
+## Deployment
+There are example systemd service and timer units in the `contrib` directory. The API service listens on localhost by default so you will need to expose it publicly using a web server like nginx.
+
+An example nginx configuration is:
+
+```
+server {
+    #...
+
+    location ~ /rest/statistics/?(.*) {
+        access_log /var/log/nginx/statistics.log;
+        proxy_pass http://statistics_api/$1$is_args$args;
+    }
+}
+
+upstream statistics_api {
+    server 127.0.0.1:5000;
+}
+```
+
+This would expose the API at `/rest/statistics`.
+
 ## Using the API
 The API exposes the following endpoints:
 
   - GET `/items` — return views and downloads for all items that Solr knows about¹. Accepts `limit` and `page` query parameters for pagination of results.
-  - GET `/item/id` — return views and downloads for a single item (*id* must be a positive integer).
+  - GET `/item/id` — return views and downloads for a single item (*id* must be a positive integer). Returns HTTP 404 if an item id is not found.
 
-¹ We are querying the Solr statistics core, which technically only knows about all items that have either views or downloads.
+¹ We are querying the Solr statistics core, which technically only knows about items that have either views or downloads.
 
 ## Todo
 
 - Add API documentation
-- Close up DB connection when gunicorn shuts down gracefully
+- Close DB connection when gunicorn shuts down gracefully
 - Better logging
-- Return HTTP 404 when item_id is nonexistent
 - Tests
+- Check if database exists (try/except)
+- Version API
+- Use JSON in PostgreSQL
+- Switch to [Python 3.6+ f-string syntax](https://realpython.com/python-f-strings/)
 
 ## License
 This work is licensed under the [GPLv3](https://www.gnu.org/licenses/gpl-3.0.en.html).

app.py

@@ -1,10 +1,8 @@
 from database import database_connection
 import falcon
-from solr import solr_connection
 
 db = database_connection()
 db.set_session(readonly=True)
-solr = solr_connection()
 
 class AllItemsResource:
     def on_get(self, req, resp):
@@ -22,16 +20,16 @@ class AllItemsResource:
 
         # get statistics, ordered by id, and use limit and offset to page through results
         cursor.execute('SELECT id, views, downloads FROM items ORDER BY id ASC LIMIT {} OFFSET {}'.format(limit, offset))
-        results = cursor.fetchmany(limit)
-        cursor.close()
 
         # create a list to hold dicts of item stats
         statistics = list()
 
         # iterate over results and build statistics object
-        for item in results:
+        for item in cursor:
             statistics.append({ 'id': item['id'], 'views': item['views'], 'downloads': item['downloads'] })
 
+        cursor.close()
+
         message = {
                 'currentPage': page,
                 'totalPages': pages,
@@ -47,16 +45,26 @@ class ItemResource:
 
         cursor = db.cursor()
         cursor.execute('SELECT views, downloads FROM items WHERE id={}'.format(item_id))
-        results = cursor.fetchone()
+        if cursor.rowcount == 0:
+            raise falcon.HTTPNotFound(
+                    title='Item not found',
+                    description='The item with id "{}" was not found.'.format(item_id)
+            )
+        else:
+            results = cursor.fetchone()
+
+            statistics = {
+                'id': item_id,
+                'views': results['views'],
+                'downloads': results['downloads']
+            }
+
+            resp.media = statistics
+
         cursor.close()
 
-        statistics = {
-            'id': item_id,
-            'views': results['views'],
-            'downloads': results['downloads']
-        }
-
-        resp.media = statistics
+def on_exit(api):
+    print("Shutting down DB")
 
 api = falcon.API()
 api.add_route('/items', AllItemsResource())

contrib/dspace-statistics-api.service

@@ -9,8 +9,8 @@ Environment=DATABASE_PASS=dspacestatistics
 Environment=DATABASE_HOST=localhost
 User=nobody
 Group=nogroup
-WorkingDirectory=/opt/ilri/dspace-statistics-api
-ExecStart=/opt/ilri/dspace-statistics-api/venv/bin/gunicorn \
+WorkingDirectory=/var/lib/dspace-statistics-api
+ExecStart=/var/lib/dspace-statistics-api/venv/bin/gunicorn \
           --bind 127.0.0.1:5000                             \
           app:api
 ExecReload=/bin/kill -s HUP $MAINPID

contrib/dspace-statistics-indexer.service

@@ -10,8 +10,8 @@ Environment=DATABASE_PASS=dspacestatistics
 Environment=DATABASE_HOST=localhost
 User=nobody
 Group=nogroup
-WorkingDirectory=/opt/ilri/dspace-statistics-api
-ExecStart=/opt/ilri/dspace-statistics-api/venv/bin/python indexer.py
+WorkingDirectory=/var/lib/dspace-statistics-api
+ExecStart=/var/lib/dspace-statistics-api/venv/bin/python indexer.py
 
 [Install]
 WantedBy=multi-user.target

contrib/dspace-statistics-indexer.timer

@@ -3,7 +3,7 @@ Description=DSpace Statistics Indexer
 
 [Timer]
 # twice a day, at 6AM and 6PM
-OnCalendar=*-*-* 06:00:00,18:00:00
+OnCalendar=*-*-* 06,18:00:00
 # Add a random delay of 0–3600 seconds
 RandomizedDelaySec=3600
 Persistent=true

database.py

@@ -2,8 +2,7 @@ from config import DATABASE_NAME
 from config import DATABASE_USER
 from config import DATABASE_PASS
 from config import DATABASE_HOST
-import psycopg2
-import psycopg2.extras
+import psycopg2, psycopg2.extras
 
 def database_connection():
     connection = psycopg2.connect("dbname={} user={} password={} host='{}'".format(DATABASE_NAME, DATABASE_USER, DATABASE_PASS, DATABASE_HOST), cursor_factory=psycopg2.extras.DictCursor)

indexer.py

@@ -20,111 +20,140 @@
 # ---
 #
 # Connects to a DSpace Solr statistics core and ingests item views and downloads
-# into a Postgres database for use with other applications (an API, for example).
+# into a PostgreSQL database for use by other applications (like an API).
 #
-# This script is written for Python 3 and requires several modules that you can
-# install with pip (I recommend setting up a Python virtual environment first):
+# This script is written for Python 3.5+ and requires several modules that you
+# can install with pip (I recommend using a Python virtual environment):
 #
-#   $ pip install SolrClient
+#   $ pip install SolrClient psycopg2-binary
 #
 # See: https://solrclient.readthedocs.io/en/latest/SolrClient.html
 # See: https://wiki.duraspace.org/display/DSPACE/Solr
-#
-# Tested with Python 3.5 and 3.6.
 
 from database import database_connection
+import json
+import psycopg2.extras
 from solr import solr_connection
 
 def index_views():
-    print("Populating database with item views.")
-
-    # determine the total number of items with views (aka Solr's numFound)
+    # get total number of distinct facets for items with a minimum of 1 view,
+    # otherwise Solr returns all kinds of weird ids that are actually not in
+    # the database. Also, stats are expensive, but we need stats.calcdistinct
+    # so we can get the countDistinct summary.
+    #
+    # see: https://lucene.apache.org/solr/guide/6_6/the-stats-component.html
     res = solr.query('statistics', {
         'q':'type:2',
         'fq':'isBot:false AND statistics_type:view',
         'facet':True,
         'facet.field':'id',
+        'facet.mincount':1,
+        'facet.limit':1,
+        'facet.offset':0,
+        'stats':True,
+        'stats.field':'id',
+        'stats.calcdistinct':True
     }, rows=0)
 
-    # divide results into "pages" (numFound / 100)
-    results_numFound = res.get_num_found()
+    # get total number of distinct facets (countDistinct)
+    results_totalNumFacets = json.loads(res.get_json())['stats']['stats_fields']['id']['countDistinct']
+
+    # divide results into "pages" (cast to int to effectively round down)
     results_per_page = 100
-    results_num_pages = round(results_numFound / results_per_page)
+    results_num_pages = int(results_totalNumFacets / results_per_page)
     results_current_page = 0
 
     cursor = db.cursor()
 
+    # create an empty list to store values for batch insertion
+    data = []
+
     while results_current_page <= results_num_pages:
-        print('Page {} of {}.'.format(results_current_page, results_num_pages))
+        print('Indexing item views (page {} of {})'.format(results_current_page, results_num_pages))
 
         res = solr.query('statistics', {
             'q':'type:2',
             'fq':'isBot:false AND statistics_type:view',
             'facet':True,
             'facet.field':'id',
+            'facet.mincount':1,
             'facet.limit':results_per_page,
             'facet.offset':results_current_page * results_per_page
-        })
+        }, rows=0)
 
-        # make sure total number of results > 0
-        if res.get_num_found() > 0:
-            # SolrClient's get_facets() returns a dict of dicts
-            views = res.get_facets()
-            # in this case iterate over the 'id' dict and get the item ids and views
-            for item_id, item_views in views['id'].items():
-                cursor.execute('''INSERT INTO items(id, views) VALUES(%s, %s)
-                               ON CONFLICT(id) DO UPDATE SET downloads=excluded.views''',
-                               (item_id, item_views))
+        # SolrClient's get_facets() returns a dict of dicts
+        views = res.get_facets()
+        # in this case iterate over the 'id' dict and get the item ids and views
+        for item_id, item_views in views['id'].items():
+            data.append((item_id, item_views))
 
+        # do a batch insert of values from the current "page" of results
+        sql = 'INSERT INTO items(id, views) VALUES %s ON CONFLICT(id) DO UPDATE SET views=excluded.views'
+        psycopg2.extras.execute_values(cursor, sql, data, template='(%s, %s)')
         db.commit()
 
+        # clear all items from the list so we can populate it with the next batch
+        data.clear()
+
         results_current_page += 1
 
     cursor.close()
 
 def index_downloads():
-    print("Populating database with item downloads.")
-
-    # determine the total number of items with downloads (aka Solr's numFound)
+    # get the total number of distinct facets for items with at least 1 download
     res = solr.query('statistics', {
         'q':'type:0',
         'fq':'isBot:false AND statistics_type:view AND bundleName:ORIGINAL',
         'facet':True,
         'facet.field':'owningItem',
+        'facet.mincount':1,
+        'facet.limit':1,
+        'facet.offset':0,
+        'stats':True,
+        'stats.field':'owningItem',
+        'stats.calcdistinct':True
     }, rows=0)
 
-    # divide results into "pages" (numFound / 100)
-    results_numFound = res.get_num_found()
+    # get total number of distinct facets (countDistinct)
+    results_totalNumFacets = json.loads(res.get_json())['stats']['stats_fields']['owningItem']['countDistinct']
+
+    # divide results into "pages" (cast to int to effectively round down)
     results_per_page = 100
-    results_num_pages = round(results_numFound / results_per_page)
+    results_num_pages = int(results_totalNumFacets / results_per_page)
     results_current_page = 0
 
     cursor = db.cursor()
 
+    # create an empty list to store values for batch insertion
+    data = []
+
     while results_current_page <= results_num_pages:
-        print('Page {} of {}.'.format(results_current_page, results_num_pages))
+        print('Indexing item downloads (page {} of {})'.format(results_current_page, results_num_pages))
 
         res = solr.query('statistics', {
             'q':'type:0',
             'fq':'isBot:false AND statistics_type:view AND bundleName:ORIGINAL',
             'facet':True,
             'facet.field':'owningItem',
+            'facet.mincount':1,
             'facet.limit':results_per_page,
             'facet.offset':results_current_page * results_per_page
-        })
+        }, rows=0)
 
-        # make sure total number of results > 0
-        if res.get_num_found() > 0:
-            # SolrClient's get_facets() returns a dict of dicts
-            downloads = res.get_facets()
-            # in this case iterate over the 'owningItem' dict and get the item ids and downloads
-            for item_id, item_downloads in downloads['owningItem'].items():
-                cursor.execute('''INSERT INTO items(id, downloads) VALUES(%s, %s)
-                               ON CONFLICT(id) DO UPDATE SET downloads=excluded.downloads''',
-                               (item_id, item_downloads))
+        # SolrClient's get_facets() returns a dict of dicts
+        downloads = res.get_facets()
+        # in this case iterate over the 'owningItem' dict and get the item ids and downloads
+        for item_id, item_downloads in downloads['owningItem'].items():
+            data.append((item_id, item_downloads))
 
+        # do a batch insert of values from the current "page" of results
+        sql = 'INSERT INTO items(id, downloads) VALUES %s ON CONFLICT(id) DO UPDATE SET downloads=excluded.downloads'
+        psycopg2.extras.execute_values(cursor, sql, data, template='(%s, %s)')
         db.commit()
 
+        # clear all items from the list so we can populate it with the next batch
+        data.clear()
+
         results_current_page += 1
 
     cursor.close()