CHANGELOG.md: Add changes for v0.6.0

This was added accidentally when I refactored. I was trying to see
if I could use Falcon's on_exit() hook.

.travis.yml

@@ -0,0 +1,11 @@
+language: python
+python:
+  - "3.5"
+  - "3.6"
+  - "3.7-dev"
+script: pip install -r requirements.txt
+branches:
+  only:
+    - master
+
+# vim: ts=2 sw=2 et

CHANGELOG.md

@@ -4,6 +4,68 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+### [0.6.0] - 2018-10-31
+## Changed
+- Refactor project structure (note breaking changes to API and indexing invocation, see contrib and README.md)
+
+### [0.5.2] - 2018-10-28
+## Changed
+- Update library versions in requirements.txt
+
+### [0.5.1] - 2018-10-24
+## Changed
+- Use Python's native json instead of ujson
+
+### [0.5.0] - 2018-10-24
+## Added
+- Example nginx configuration to README.md
+
+## Changed
+- Don't initialize Solr connection in API
+
+### [0.4.3] - 2018-10-17
+## Changed
+- Use pip install as script for Travis CI
+
+## Improved
+- Documentation for deployment and testing
+
+## [0.4.2] - 2018-10-04
+### Changed
+- README.md introduction and requirements
+- Use ujson instead of json
+- Iterate directly on SQL cursor in `/items` route
+
+### Fixed
+- Logic error in SQL for item views
+
+## [0.4.1] - 2018-09-26
+### Changed
+- Use execute_values() to batch insert records to PostgreSQL
+
+## [0.4.0] - 2018-09-25
+### Fixed
+- Invalid OnCalendar syntax in dspace-statistics-indexer.timer
+- Major logic error in indexer.py
+
+## [0.3.2] - 2018-09-25
+## Changed
+- /item/id route now returns HTTP 404 if an item is not found
+
+## [0.3.1] - 2018-09-25
+### Changed
+- Force SolrClient's kazoo dependency to version 2.5.0 to work with Python 3.7
+- Add Python 3.7 to Travis CI configuration
+
+## [0.3.0] - 2018-09-25
+### Added
+- requirements.txt for pip
+- Travis CI build configuration for Python 3.5 and 3.6
+- Documentation on using the API
+
+### Changed
+- The "all items" route from / to /items
+
 ## [0.2.1] - 2018-09-24
 ### Changed
 - Environment settings in example systemd unit files

README.md

@@ -1,22 +1,79 @@
-# DSpace Statistics API
-A quick and dirty REST API to expose Solr view and download statistics for items in a DSpace repository.
+# DSpace Statistics API [![Build Status](https://travis-ci.org/alanorth/dspace-statistics-api.svg?branch=master)](https://travis-ci.org/alanorth/dspace-statistics-api)
+A simple REST API to expose Solr view and download statistics for items in a DSpace repository. This project contains a standalone indexing component and a WSGI application.
 
-Written and tested in Python 3.6. SolrClient (0.2.1) does not currently run in Python 3.7.0. Requires PostgreSQL version 9.5 or greater for [`UPSERT` support](https://wiki.postgresql.org/wiki/UPSERT).
+## Requirements
 
-## Installation
-Create a virtual environment and run it:
+- Python 3.5+
+- PostgreSQL version 9.5+ (due to [`UPSERT` support](https://wiki.postgresql.org/wiki/UPSERT))
+- DSpace 4+ with [Solr usage statistics enabled](https://wiki.duraspace.org/display/DSDOC5x/SOLR+Statistics)
 
-    $ virtualenv -p /usr/bin/python3.6 venv
+## Installation and Testing
+Create a Python virtual environment and install the dependencies:
+
+    $ python -m venv venv
     $ . venv/bin/activate
-    $ pip install falcon gunicorn SolrClient psycopg2-binary
-    $ gunicorn app:api
+    $ pip install -r requirements.txt
+
+Set up the environment variables for Solr and PostgreSQL:
+
+    $ export SOLR_SERVER=http://localhost:8080/solr
+    $ export DATABASE_NAME=dspacestatistics
+    $ export DATABASE_USER=dspacestatistics
+    $ export DATABASE_PASS=dspacestatistics
+    $ export DATABASE_HOST=localhost
+
+Index the Solr statistics core to populate the PostgreSQL database:
+
+    $ python -m dspace_statistics_api.indexer
+
+Run the REST API:
+
+    $ gunicorn dspace_statistics_api.app
+
+Test to see if there are any statistics:
+
+    $ curl 'http://localhost:8000/items?limit=1'
+
+## Deployment
+There are example systemd service and timer units in the `contrib` directory. The API service listens on localhost by default so you will need to expose it publicly using a web server like nginx.
+
+An example nginx configuration is:
+
+```
+server {
+    #...
+
+    location ~ /rest/statistics/?(.*) {
+        access_log /var/log/nginx/statistics.log;
+        proxy_pass http://statistics_api/$1$is_args$args;
+    }
+}
+
+upstream statistics_api {
+    server 127.0.0.1:5000;
+}
+```
+
+This would expose the API at `/rest/statistics`.
+
+## Using the API
+The API exposes the following endpoints:
+
+  - GET `/items` — return views and downloads for all items that Solr knows about¹. Accepts `limit` and `page` query parameters for pagination of results.
+  - GET `/item/id` — return views and downloads for a single item (*id* must be a positive integer). Returns HTTP 404 if an item id is not found.
+
+¹ We are querying the Solr statistics core, which technically only knows about items that have either views or downloads.
 
 ## Todo
 
 - Add API documentation
-- Close up DB connection when gunicorn shuts down gracefully
+- Close DB connection when gunicorn shuts down gracefully
 - Better logging
-- Return HTTP 404 when item_id is nonexistent
+- Tests
+- Check if database exists (try/except)
+- Version API
+- Use JSON in PostgreSQL
+- Switch to [Python 3.6+ f-string syntax](https://realpython.com/python-f-strings/)
 
 ## License
 This work is licensed under the [GPLv3](https://www.gnu.org/licenses/gpl-3.0.en.html).

contrib/dspace-statistics-api.service

@@ -9,10 +9,10 @@ Environment=DATABASE_PASS=dspacestatistics
 Environment=DATABASE_HOST=localhost
 User=nobody
 Group=nogroup
-WorkingDirectory=/opt/ilri/dspace-statistics-api
-ExecStart=/opt/ilri/dspace-statistics-api/venv/bin/gunicorn \
+WorkingDirectory=/var/lib/dspace-statistics-api
+ExecStart=/var/lib/dspace-statistics-api/venv/bin/gunicorn \
           --bind 127.0.0.1:5000                             \
-          app:api
+          dspace_statistics_api.app
 ExecReload=/bin/kill -s HUP $MAINPID
 ExecStop=/bin/kill -s TERM $MAINPID
 

contrib/dspace-statistics-indexer.service

@@ -10,8 +10,8 @@ Environment=DATABASE_PASS=dspacestatistics
 Environment=DATABASE_HOST=localhost
 User=nobody
 Group=nogroup
-WorkingDirectory=/opt/ilri/dspace-statistics-api
-ExecStart=/opt/ilri/dspace-statistics-api/venv/bin/python indexer.py
+WorkingDirectory=/var/lib/dspace-statistics-api
+ExecStart=/var/lib/dspace-statistics-api/venv/bin/python -m dspace_statistics_api.indexer
 
 [Install]
 WantedBy=multi-user.target

contrib/dspace-statistics-indexer.timer

@@ -3,7 +3,7 @@ Description=DSpace Statistics Indexer
 
 [Timer]
 # twice a day, at 6AM and 6PM
-OnCalendar=*-*-* 06:00:00,18:00:00
+OnCalendar=*-*-* 06,18:00:00
 # Add a random delay of 0–3600 seconds
 RandomizedDelaySec=3600
 Persistent=true

dspace_statistics_api/app.py

@@ -1,10 +1,8 @@
-from database import database_connection
+from .database import database_connection
 import falcon
-from solr import solr_connection
 
 db = database_connection()
 db.set_session(readonly=True)
-solr = solr_connection()
 
 class AllItemsResource:
     def on_get(self, req, resp):
@@ -22,16 +20,16 @@ class AllItemsResource:
 
         # get statistics, ordered by id, and use limit and offset to page through results
         cursor.execute('SELECT id, views, downloads FROM items ORDER BY id ASC LIMIT {} OFFSET {}'.format(limit, offset))
-        results = cursor.fetchmany(limit)
-        cursor.close()
 
         # create a list to hold dicts of item stats
         statistics = list()
 
         # iterate over results and build statistics object
-        for item in results:
+        for item in cursor:
             statistics.append({ 'id': item['id'], 'views': item['views'], 'downloads': item['downloads'] })
 
+        cursor.close()
+
         message = {
                 'currentPage': page,
                 'totalPages': pages,
@@ -47,19 +45,26 @@ class ItemResource:
 
         cursor = db.cursor()
         cursor.execute('SELECT views, downloads FROM items WHERE id={}'.format(item_id))
-        results = cursor.fetchone()
+        if cursor.rowcount == 0:
+            raise falcon.HTTPNotFound(
+                    title='Item not found',
+                    description='The item with id "{}" was not found.'.format(item_id)
+            )
+        else:
+            results = cursor.fetchone()
+
+            statistics = {
+                'id': item_id,
+                'views': results['views'],
+                'downloads': results['downloads']
+            }
+
+            resp.media = statistics
+
         cursor.close()
 
-        statistics = {
-            'id': item_id,
-            'views': results['views'],
-            'downloads': results['downloads']
-        }
-
-        resp.media = statistics
-
-api = falcon.API()
-api.add_route('/', AllItemsResource())
+api = application = falcon.API()
+api.add_route('/items', AllItemsResource())
 api.add_route('/item/{item_id:int}', ItemResource())
 
 # vim: set sw=4 ts=4 expandtab:

dspace_statistics_api/database.py

@@ -1,9 +1,8 @@
-from config import DATABASE_NAME
-from config import DATABASE_USER
-from config import DATABASE_PASS
-from config import DATABASE_HOST
-import psycopg2
-import psycopg2.extras
+from .config import DATABASE_NAME
+from .config import DATABASE_USER
+from .config import DATABASE_PASS
+from .config import DATABASE_HOST
+import psycopg2, psycopg2.extras
 
 def database_connection():
     connection = psycopg2.connect("dbname={} user={} password={} host='{}'".format(DATABASE_NAME, DATABASE_USER, DATABASE_PASS, DATABASE_HOST), cursor_factory=psycopg2.extras.DictCursor)

dspace_statistics_api/indexer.py

@@ -0,0 +1,172 @@
+#
+# indexer.py
+#
+# Copyright 2018 Alan Orth.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# ---
+#
+# Connects to a DSpace Solr statistics core and ingests item views and downloads
+# into a PostgreSQL database for use by other applications (like an API).
+#
+# This script is written for Python 3.5+ and requires several modules that you
+# can install with pip (I recommend using a Python virtual environment):
+#
+#   $ pip install SolrClient psycopg2-binary
+#
+# See: https://solrclient.readthedocs.io/en/latest/SolrClient.html
+# See: https://wiki.duraspace.org/display/DSPACE/Solr
+
+from .database import database_connection
+import json
+import psycopg2.extras
+from .solr import solr_connection
+
+def index_views():
+    # get total number of distinct facets for items with a minimum of 1 view,
+    # otherwise Solr returns all kinds of weird ids that are actually not in
+    # the database. Also, stats are expensive, but we need stats.calcdistinct
+    # so we can get the countDistinct summary.
+    #
+    # see: https://lucene.apache.org/solr/guide/6_6/the-stats-component.html
+    res = solr.query('statistics', {
+        'q':'type:2',
+        'fq':'isBot:false AND statistics_type:view',
+        'facet':True,
+        'facet.field':'id',
+        'facet.mincount':1,
+        'facet.limit':1,
+        'facet.offset':0,
+        'stats':True,
+        'stats.field':'id',
+        'stats.calcdistinct':True
+    }, rows=0)
+
+    # get total number of distinct facets (countDistinct)
+    results_totalNumFacets = json.loads(res.get_json())['stats']['stats_fields']['id']['countDistinct']
+
+    # divide results into "pages" (cast to int to effectively round down)
+    results_per_page = 100
+    results_num_pages = int(results_totalNumFacets / results_per_page)
+    results_current_page = 0
+
+    cursor = db.cursor()
+
+    # create an empty list to store values for batch insertion
+    data = []
+
+    while results_current_page <= results_num_pages:
+        print('Indexing item views (page {} of {})'.format(results_current_page, results_num_pages))
+
+        res = solr.query('statistics', {
+            'q':'type:2',
+            'fq':'isBot:false AND statistics_type:view',
+            'facet':True,
+            'facet.field':'id',
+            'facet.mincount':1,
+            'facet.limit':results_per_page,
+            'facet.offset':results_current_page * results_per_page
+        }, rows=0)
+
+        # SolrClient's get_facets() returns a dict of dicts
+        views = res.get_facets()
+        # in this case iterate over the 'id' dict and get the item ids and views
+        for item_id, item_views in views['id'].items():
+            data.append((item_id, item_views))
+
+        # do a batch insert of values from the current "page" of results
+        sql = 'INSERT INTO items(id, views) VALUES %s ON CONFLICT(id) DO UPDATE SET views=excluded.views'
+        psycopg2.extras.execute_values(cursor, sql, data, template='(%s, %s)')
+        db.commit()
+
+        # clear all items from the list so we can populate it with the next batch
+        data.clear()
+
+        results_current_page += 1
+
+    cursor.close()
+
+def index_downloads():
+    # get the total number of distinct facets for items with at least 1 download
+    res = solr.query('statistics', {
+        'q':'type:0',
+        'fq':'isBot:false AND statistics_type:view AND bundleName:ORIGINAL',
+        'facet':True,
+        'facet.field':'owningItem',
+        'facet.mincount':1,
+        'facet.limit':1,
+        'facet.offset':0,
+        'stats':True,
+        'stats.field':'owningItem',
+        'stats.calcdistinct':True
+    }, rows=0)
+
+    # get total number of distinct facets (countDistinct)
+    results_totalNumFacets = json.loads(res.get_json())['stats']['stats_fields']['owningItem']['countDistinct']
+
+    # divide results into "pages" (cast to int to effectively round down)
+    results_per_page = 100
+    results_num_pages = int(results_totalNumFacets / results_per_page)
+    results_current_page = 0
+
+    cursor = db.cursor()
+
+    # create an empty list to store values for batch insertion
+    data = []
+
+    while results_current_page <= results_num_pages:
+        print('Indexing item downloads (page {} of {})'.format(results_current_page, results_num_pages))
+
+        res = solr.query('statistics', {
+            'q':'type:0',
+            'fq':'isBot:false AND statistics_type:view AND bundleName:ORIGINAL',
+            'facet':True,
+            'facet.field':'owningItem',
+            'facet.mincount':1,
+            'facet.limit':results_per_page,
+            'facet.offset':results_current_page * results_per_page
+        }, rows=0)
+
+        # SolrClient's get_facets() returns a dict of dicts
+        downloads = res.get_facets()
+        # in this case iterate over the 'owningItem' dict and get the item ids and downloads
+        for item_id, item_downloads in downloads['owningItem'].items():
+            data.append((item_id, item_downloads))
+
+        # do a batch insert of values from the current "page" of results
+        sql = 'INSERT INTO items(id, downloads) VALUES %s ON CONFLICT(id) DO UPDATE SET downloads=excluded.downloads'
+        psycopg2.extras.execute_values(cursor, sql, data, template='(%s, %s)')
+        db.commit()
+
+        # clear all items from the list so we can populate it with the next batch
+        data.clear()
+
+        results_current_page += 1
+
+    cursor.close()
+
+db = database_connection()
+solr = solr_connection()
+
+# create table to store item views and downloads
+cursor = db.cursor()
+cursor.execute('''CREATE TABLE IF NOT EXISTS items
+                  (id INT PRIMARY KEY, views INT DEFAULT 0, downloads INT DEFAULT 0)''')
+index_views()
+index_downloads()
+
+db.close()
+
+# vim: set sw=4 ts=4 expandtab:

dspace_statistics_api/solr.py

@@ -1,4 +1,4 @@
-from config import SOLR_SERVER
+from .config import SOLR_SERVER
 from SolrClient import SolrClient
 
 def solr_connection():

indexer.py

@@ -1,144 +0,0 @@
-#!/usr/bin/env python
-#
-# indexer.py
-#
-# Copyright 2018 Alan Orth.
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-#
-# ---
-#
-# Connects to a DSpace Solr statistics core and ingests item views and downloads
-# into a Postgres database for use with other applications (an API, for example).
-#
-# This script is written for Python 3 and requires several modules that you can
-# install with pip (I recommend setting up a Python virtual environment first):
-#
-#   $ pip install SolrClient
-#
-# See: https://solrclient.readthedocs.io/en/latest/SolrClient.html
-# See: https://wiki.duraspace.org/display/DSPACE/Solr
-#
-# Tested with Python 3.5 and 3.6.
-
-from database import database_connection
-from solr import solr_connection
-
-def index_views():
-    print("Populating database with item views.")
-
-    # determine the total number of items with views (aka Solr's numFound)
-    res = solr.query('statistics', {
-        'q':'type:2',
-        'fq':'isBot:false AND statistics_type:view',
-        'facet':True,
-        'facet.field':'id',
-    }, rows=0)
-
-    # divide results into "pages" (numFound / 100)
-    results_numFound = res.get_num_found()
-    results_per_page = 100
-    results_num_pages = round(results_numFound / results_per_page)
-    results_current_page = 0
-
-    cursor = db.cursor()
-
-    while results_current_page <= results_num_pages:
-        print('Page {} of {}.'.format(results_current_page, results_num_pages))
-
-        res = solr.query('statistics', {
-            'q':'type:2',
-            'fq':'isBot:false AND statistics_type:view',
-            'facet':True,
-            'facet.field':'id',
-            'facet.limit':results_per_page,
-            'facet.offset':results_current_page * results_per_page
-        })
-
-        # make sure total number of results > 0
-        if res.get_num_found() > 0:
-            # SolrClient's get_facets() returns a dict of dicts
-            views = res.get_facets()
-            # in this case iterate over the 'id' dict and get the item ids and views
-            for item_id, item_views in views['id'].items():
-                cursor.execute('''INSERT INTO items(id, views) VALUES(%s, %s)
-                               ON CONFLICT(id) DO UPDATE SET downloads=excluded.views''',
-                               (item_id, item_views))
-
-        db.commit()
-
-        results_current_page += 1
-
-    cursor.close()
-
-def index_downloads():
-    print("Populating database with item downloads.")
-
-    # determine the total number of items with downloads (aka Solr's numFound)
-    res = solr.query('statistics', {
-        'q':'type:0',
-        'fq':'isBot:false AND statistics_type:view AND bundleName:ORIGINAL',
-        'facet':True,
-        'facet.field':'owningItem',
-    }, rows=0)
-
-    # divide results into "pages" (numFound / 100)
-    results_numFound = res.get_num_found()
-    results_per_page = 100
-    results_num_pages = round(results_numFound / results_per_page)
-    results_current_page = 0
-
-    cursor = db.cursor()
-
-    while results_current_page <= results_num_pages:
-        print('Page {} of {}.'.format(results_current_page, results_num_pages))
-
-        res = solr.query('statistics', {
-            'q':'type:0',
-            'fq':'isBot:false AND statistics_type:view AND bundleName:ORIGINAL',
-            'facet':True,
-            'facet.field':'owningItem',
-            'facet.limit':results_per_page,
-            'facet.offset':results_current_page * results_per_page
-        })
-
-        # make sure total number of results > 0
-        if res.get_num_found() > 0:
-            # SolrClient's get_facets() returns a dict of dicts
-            downloads = res.get_facets()
-            # in this case iterate over the 'owningItem' dict and get the item ids and downloads
-            for item_id, item_downloads in downloads['owningItem'].items():
-                cursor.execute('''INSERT INTO items(id, downloads) VALUES(%s, %s)
-                               ON CONFLICT(id) DO UPDATE SET downloads=excluded.downloads''',
-                               (item_id, item_downloads))
-
-        db.commit()
-
-        results_current_page += 1
-
-    cursor.close()
-
-db = database_connection()
-solr = solr_connection()
-
-# create table to store item views and downloads
-cursor = db.cursor()
-cursor.execute('''CREATE TABLE IF NOT EXISTS items
-                  (id INT PRIMARY KEY, views INT DEFAULT 0, downloads INT DEFAULT 0)''')
-index_views()
-index_downloads()
-
-db.close()
-
-# vim: set sw=4 ts=4 expandtab:

requirements.txt

@@ -0,0 +1,12 @@
+certifi==2018.10.15
+chardet==3.0.4
+falcon==1.4.1
+gunicorn==19.9.0
+idna==2.7
+kazoo==2.5.0
+psycopg2-binary==2.7.5
+python-mimeparse==1.6.0
+requests==2.20.0
+six==1.11.0
+-e git://github.com/alanorth/SolrClient.git@c629e3475be37c82770b2be61748be7e29882648#egg=SolrClient
+urllib3==1.24