CHANGELOG.md: Move unreleased changes to version 0.4.0

Basically Solr's numFound has nothing to do with the actual number
of distinct facets that are returned. You need to use Solr's stats
component to get the number of distinct facets, aka countDistinct.
This is apparently deprecated in newer Solr versions, but we're on
version 4.10 and it works there.

Also, I realized that there is no need to return facets for items
without any views or downloads. Using facet.mincount=1 reduces the
result set size and also means we can store less data in the data-
base. The API returns HTTP 404 Not Found if an item is not in the
database anyways.

I can't figure it out exactly, but there is some weird issue with
Solr's facet results when you don't use facet.mincount=1. For some
reason you get tons of results with an id that doesn't even exist
in the document database, let alone as an actual DSpace item!

See: https://lucene.apache.org/solr/guide/6_6/the-stats-component.html

.gitignore

@@ -1,3 +1,2 @@
 __pycache__
 venv
-*.db

.travis.yml

@@ -0,0 +1,9 @@
+language: python
+python:
+  - "3.5"
+  - "3.6"
+  - "3.7"
+install:
+  - pip install -r requirements.txt
+
+# vim: ts=2 sw=2 et

CHANGELOG.md

@@ -4,6 +4,38 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.4.0] - 2018-09-25
+### Fixed
+- Invalid OnCalendar syntax in dspace-statistics-indexer.timer
+- Major logic error in indexer.py
+
+## [0.3.2] - 2018-09-25
+## Changed
+- /item/id route now returns HTTP 404 if an item is not found
+
+## [0.3.1] - 2018-09-25
+### Changed
+- Force SolrClient's kazoo dependency to version 2.5.0 to work with Python 3.7
+- Add Python 3.7 to Travis CI configuration
+
+## [0.3.0] - 2018-09-25
+### Added
+- requirements.txt for pip
+- Travis CI build configuration for Python 3.5 and 3.6
+- Documentation on using the API
+
+### Changed
+- The "all items" route from / to /items
+
+## [0.2.1] - 2018-09-24
+### Changed
+- Environment settings in example systemd unit files
+- Use psycopg2.extras.DictCursor for PostgreSQL connection
+
+## [0.2.0] - 2018-09-24
+### Changed
+- Use PostgreSQL instead of SQLite because UPSERT support needs a very new libsqlite3 whereas it's already in PostgreSQL 9.5+
+
 ## [0.1.0] - 2018-09-24
 ### Changed
 - Rename project to "DSpace Statistics API"

README.md

@@ -1,22 +1,30 @@
 # DSpace Statistics API
 A quick and dirty REST API to expose Solr view and download statistics for items in a DSpace repository.
 
-Written and tested in Python 3.6. SolrClient (0.2.1) does not currently run in Python 3.7.0.
+Written and tested in Python 3.5, 3.6, and 3.7. Requires PostgreSQL version 9.5 or greater for [`UPSERT` support](https://wiki.postgresql.org/wiki/UPSERT).
 
 ## Installation
 Create a virtual environment and run it:
 
-    $ virtualenv -p /usr/bin/python3.6 venv
+    $ python -m venv venv
     $ . venv/bin/activate
-    $ pip install falcon gunicorn SolrClient
+    $ pip install -r requirements.txt
     $ gunicorn app:api
 
+## Using the API
+The API exposes the following endpoints:
+
+  - GET `/items` — return views and downloads for all items that Solr knows about¹. Accepts `limit` and `page` query parameters for pagination of results.
+  - GET `/item/id` — return views and downloads for a single item (*id* must be a positive integer). Returns HTTP 404 if an item id is not found.
+
+¹ We are querying the Solr statistics core, which technically only knows about items that have either views or downloads.
+
 ## Todo
 
 - Add API documentation
 - Close up DB connection when gunicorn shuts down gracefully
 - Better logging
-- Return HTTP 404 when item_id is nonexistent
+- Tests
 
 ## License
 This work is licensed under the [GPLv3](https://www.gnu.org/licenses/gpl-3.0.en.html).

app.py

@@ -1,12 +1,9 @@
-# Tested with Python 3.6
-# See DSpace Solr docs for tips about parameters
-# https://wiki.duraspace.org/display/DSPACE/Solr
-
-from database import database_connection_ro
+from database import database_connection
 import falcon
 from solr import solr_connection
 
-db = database_connection_ro()
+db = database_connection()
+db.set_session(readonly=True)
 solr = solr_connection()
 
 class AllItemsResource:
@@ -24,7 +21,7 @@ class AllItemsResource:
         pages = round(cursor.fetchone()[0] / limit)
 
         # get statistics, ordered by id, and use limit and offset to page through results
-        cursor.execute('SELECT id, views, downloads FROM items ORDER BY id ASC LIMIT {0} OFFSET {1}'.format(limit, offset))
+        cursor.execute('SELECT id, views, downloads FROM items ORDER BY id ASC LIMIT {} OFFSET {}'.format(limit, offset))
         results = cursor.fetchmany(limit)
         cursor.close()
 
@@ -49,9 +46,14 @@ class ItemResource:
         """Handles GET requests"""
 
         cursor = db.cursor()
-        cursor.execute('SELECT views, downloads FROM items WHERE id={0}'.format(item_id))
+        cursor.execute('SELECT views, downloads FROM items WHERE id={}'.format(item_id))
+        if cursor.rowcount == 0:
+            raise falcon.HTTPNotFound(
+                    title='Item not found',
+                    description='The item with id "{}" was not found.'.format(item_id)
+            )
+        else:
             results = cursor.fetchone()
-        cursor.close()
 
             statistics = {
                 'id': item_id,
@@ -61,8 +63,10 @@ class ItemResource:
 
             resp.media = statistics
 
+        cursor.close()
+
 api = falcon.API()
-api.add_route('/', AllItemsResource())
+api.add_route('/items', AllItemsResource())
 api.add_route('/item/{item_id:int}', ItemResource())
 
 # vim: set sw=4 ts=4 expandtab:

config.py

@@ -3,6 +3,9 @@ import os
 # Check if Solr connection information was provided in the environment
 SOLR_SERVER = os.environ.get('SOLR_SERVER', 'http://localhost:8080/solr')
 
-SQLITE_DB = os.environ.get('SQLITE_DB', 'statistics.db')
+DATABASE_NAME = os.environ.get('DATABASE_NAME', 'dspacestatistics')
+DATABASE_USER = os.environ.get('DATABASE_USER', 'dspacestatistics')
+DATABASE_PASS = os.environ.get('DATABASE_PASS', 'dspacestatistics')
+DATABASE_HOST = os.environ.get('DATABASE_HOST', 'localhost')
 
 # vim: set sw=4 ts=4 expandtab:

contrib/dspace-statistics-api.service

@@ -3,7 +3,10 @@ Description=DSpace Statistics API
 After=network.target
 
 [Service]
-Environment=SOLR_SERVER=http://localhost:8081/solr
+Environment=DATABASE_NAME=dspacestatistics
+Environment=DATABASE_USER=dspacestatistics
+Environment=DATABASE_PASS=dspacestatistics
+Environment=DATABASE_HOST=localhost
 User=nobody
 Group=nogroup
 WorkingDirectory=/opt/ilri/dspace-statistics-api

contrib/dspace-statistics-indexer.service

@@ -4,6 +4,10 @@ After=tomcat7.target
 
 [Service]
 Environment=SOLR_SERVER=http://localhost:8081/solr
+Environment=DATABASE_NAME=dspacestatistics
+Environment=DATABASE_USER=dspacestatistics
+Environment=DATABASE_PASS=dspacestatistics
+Environment=DATABASE_HOST=localhost
 User=nobody
 Group=nogroup
 WorkingDirectory=/opt/ilri/dspace-statistics-api

contrib/dspace-statistics-indexer.timer

@@ -3,7 +3,7 @@ Description=DSpace Statistics Indexer
 
 [Timer]
 # twice a day, at 6AM and 6PM
-OnCalendar=*-*-* 06:00:00,18:00:00
+OnCalendar=*-*-* 06,18:00:00
 # Add a random delay of 0–3600 seconds
 RandomizedDelaySec=3600
 Persistent=true

database.py

@@ -1,17 +1,12 @@
-from config import SQLITE_DB
-import sqlite3
+from config import DATABASE_NAME
+from config import DATABASE_USER
+from config import DATABASE_PASS
+from config import DATABASE_HOST
+import psycopg2
+import psycopg2.extras
 
-def database_connection_rw():
-    connection = sqlite3.connect(SQLITE_DB)
-    # allow iterating over row results by column key
-    connection.row_factory = sqlite3.Row
-
-    return connection
-
-def database_connection_ro():
-    connection = sqlite3.connect('file:{0}?mode=ro'.format(SQLITE_DB), uri=True)
-    # allow iterating over row results by column key
-    connection.row_factory = sqlite3.Row
+def database_connection():
+    connection = psycopg2.connect("dbname={} user={} password={} host='{}'".format(DATABASE_NAME, DATABASE_USER, DATABASE_PASS, DATABASE_HOST), cursor_factory=psycopg2.extras.DictCursor)
 
     return connection
 

indexer.py

@@ -20,57 +20,71 @@
 # ---
 #
 # Connects to a DSpace Solr statistics core and ingests item views and downloads
-# into a SQLite database for use with other applications (an API, for example).
+# into a PostgreSQL database for use by other applications (like an API).
 #
-# This script is written for Python 3 and requires several modules that you can
-# install with pip (I recommend setting up a Python virtual environment first):
+# This script is written for Python 3.5+ and requires several modules that you
+# can install with pip (I recommend using a Python virtual environment):
 #
-#   $ pip install SolrClient
+#   $ pip install SolrClient psycopg2-binary
 #
 # See: https://solrclient.readthedocs.io/en/latest/SolrClient.html
 # See: https://wiki.duraspace.org/display/DSPACE/Solr
-#
-# Tested with Python 3.5 and 3.6.
 
-from database import database_connection_rw
+from database import database_connection
+import json
 from solr import solr_connection
 
 def index_views():
-    print("Populating database with item views.")
-
-    # determine the total number of items with views (aka Solr's numFound)
+    # get total number of distinct facets for items with a minimum of 1 view,
+    # otherwise Solr returns all kinds of weird ids that are actually not in
+    # the database. Also, stats are expensive, but we need stats.calcdistinct
+    # so we can get the countDistinct summary.
+    #
+    # see: https://lucene.apache.org/solr/guide/6_6/the-stats-component.html
     res = solr.query('statistics', {
         'q':'type:2',
         'fq':'isBot:false AND statistics_type:view',
         'facet':True,
         'facet.field':'id',
+        'facet.mincount':1,
+        'facet.limit':1,
+        'facet.offset':0,
+        'stats':True,
+        'stats.field':'id',
+        'stats.calcdistinct':True
     }, rows=0)
 
-    # divide results into "pages" (numFound / 100)
-    results_numFound = res.get_num_found()
+    # get total number of distinct facets (countDistinct)
+    results_totalNumFacets = json.loads(res.get_json())['stats']['stats_fields']['id']['countDistinct']
+
+    # divide results into "pages" (cast to int to effectively round down)
     results_per_page = 100
-    results_num_pages = round(results_numFound / results_per_page)
+    results_num_pages = int(results_totalNumFacets / results_per_page)
     results_current_page = 0
 
+    cursor = db.cursor()
+
     while results_current_page <= results_num_pages:
-        print('Page {0} of {1}.'.format(results_current_page, results_num_pages))
+        print('Indexing item views (page {} of {})'.format(results_current_page, results_num_pages))
 
         res = solr.query('statistics', {
             'q':'type:2',
             'fq':'isBot:false AND statistics_type:view',
             'facet':True,
             'facet.field':'id',
+            'facet.mincount':1,
             'facet.limit':results_per_page,
             'facet.offset':results_current_page * results_per_page
-        })
+        }, rows=0)
+
+        # check number of facets returned in the last query
+        #results_currentNumFacets = len(res.get_facets()['id'])
 
-        # make sure total number of results > 0
-        if res.get_num_found() > 0:
         # SolrClient's get_facets() returns a dict of dicts
         views = res.get_facets()
         # in this case iterate over the 'id' dict and get the item ids and views
         for item_id, item_views in views['id'].items():
-                db.execute('''INSERT INTO items(id, views) VALUES(?, ?)
+            cursor.execute('''INSERT INTO items(id, views) VALUES(%s, %s)
                            ON CONFLICT(id) DO UPDATE SET downloads=excluded.views''',
                            (item_id, item_views))
 
@@ -78,42 +92,51 @@ def index_views():
 
         results_current_page += 1
 
-def index_downloads():
-    print("Populating database with item downloads.")
+    cursor.close()
 
-    # determine the total number of items with downloads (aka Solr's numFound)
+def index_downloads():
+    # get the total number of distinct facets for items with at least 1 download
     res = solr.query('statistics', {
         'q':'type:0',
         'fq':'isBot:false AND statistics_type:view AND bundleName:ORIGINAL',
         'facet':True,
         'facet.field':'owningItem',
+        'facet.mincount':1,
+        'facet.limit':1,
+        'facet.offset':0,
+        'stats':True,
+        'stats.field':'owningItem',
+        'stats.calcdistinct':True
     }, rows=0)
 
-    # divide results into "pages" (numFound / 100)
-    results_numFound = res.get_num_found()
+    # get total number of distinct facets (countDistinct)
+    results_totalNumFacets = json.loads(res.get_json())['stats']['stats_fields']['owningItem']['countDistinct']
+
+    # divide results into "pages" (cast to int to effectively round down)
     results_per_page = 100
-    results_num_pages = round(results_numFound / results_per_page)
+    results_num_pages = int(results_totalNumFacets / results_per_page)
     results_current_page = 0
 
+    cursor = db.cursor()
+
     while results_current_page <= results_num_pages:
-        print('Page {0} of {1}.'.format(results_current_page, results_num_pages))
+        print('Indexing item downloads (page {} of {})'.format(results_current_page, results_num_pages))
 
         res = solr.query('statistics', {
             'q':'type:0',
             'fq':'isBot:false AND statistics_type:view AND bundleName:ORIGINAL',
             'facet':True,
             'facet.field':'owningItem',
+            'facet.mincount':1,
             'facet.limit':results_per_page,
             'facet.offset':results_current_page * results_per_page
-        })
+        }, rows=0)
 
-        # make sure total number of results > 0
-        if res.get_num_found() > 0:
         # SolrClient's get_facets() returns a dict of dicts
         downloads = res.get_facets()
         # in this case iterate over the 'owningItem' dict and get the item ids and downloads
         for item_id, item_downloads in downloads['owningItem'].items():
-                db.execute('''INSERT INTO items(id, downloads) VALUES(?, ?)
+            cursor.execute('''INSERT INTO items(id, downloads) VALUES(%s, %s)
                            ON CONFLICT(id) DO UPDATE SET downloads=excluded.downloads''',
                            (item_id, item_downloads))
 
@@ -121,11 +144,14 @@ def index_downloads():
 
         results_current_page += 1
 
-db = database_connection_rw()
+    cursor.close()
+
+db = database_connection()
 solr = solr_connection()
 
 # create table to store item views and downloads
-db.execute('''CREATE TABLE IF NOT EXISTS items
+cursor = db.cursor()
+cursor.execute('''CREATE TABLE IF NOT EXISTS items
                   (id INT PRIMARY KEY, views INT DEFAULT 0, downloads INT DEFAULT 0)''')
 index_views()
 index_downloads()

requirements.txt

@@ -0,0 +1,12 @@
+certifi==2018.8.24
+chardet==3.0.4
+falcon==1.4.1
+gunicorn==19.9.0
+idna==2.7
+kazoo==2.5.0
+psycopg2-binary==2.7.5
+python-mimeparse==1.6.0
+requests==2.19.1
+six==1.11.0
+SolrClient==0.2.1
+urllib3==1.23