CHANGELOG.md: Add changes for v0.6.0

This was added accidentally when I refactored. I was trying to see
if I could use Falcon's on_exit() hook.

.travis.yml

@@ -2,8 +2,10 @@ language: python
 python:
   - "3.5"
   - "3.6"
-  - "3.7"
+  - "3.7-dev"
-install:
+script: pip install -r requirements.txt
-  - pip install -r requirements.txt
+branches:
+  only:
+    - master
 
 # vim: ts=2 sw=2 et

CHANGELOG.md

@@ -4,6 +4,32 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+### [0.6.0] - 2018-10-31
+## Changed
+- Refactor project structure (note breaking changes to API and indexing invocation, see contrib and README.md)
+
+### [0.5.2] - 2018-10-28
+## Changed
+- Update library versions in requirements.txt
+
+### [0.5.1] - 2018-10-24
+## Changed
+- Use Python's native json instead of ujson
+
+### [0.5.0] - 2018-10-24
+## Added
+- Example nginx configuration to README.md
+
+## Changed
+- Don't initialize Solr connection in API
+
+### [0.4.3] - 2018-10-17
+## Changed
+- Use pip install as script for Travis CI
+
+## Improved
+- Documentation for deployment and testing
+
 ## [0.4.2] - 2018-10-04
 ### Changed
 - README.md introduction and requirements

README.md

@@ -1,4 +1,4 @@
-# DSpace Statistics API
+# DSpace Statistics API [![Build Status](https://travis-ci.org/alanorth/dspace-statistics-api.svg?branch=master)](https://travis-ci.org/alanorth/dspace-statistics-api)
 A simple REST API to expose Solr view and download statistics for items in a DSpace repository. This project contains a standalone indexing component and a WSGI application.
 
 ## Requirements
@@ -14,14 +14,47 @@ Create a Python virtual environment and install the dependencies:
     $ . venv/bin/activate
     $ pip install -r requirements.txt
 
-Set up the environment variables Solr and PostgreSQL:
+Set up the environment variables for Solr and PostgreSQL:
 
     $ export SOLR_SERVER=http://localhost:8080/solr
-    $ 
+    $ export DATABASE_NAME=dspacestatistics
-    $ gunicorn app:api
+    $ export DATABASE_USER=dspacestatistics
+    $ export DATABASE_PASS=dspacestatistics
+    $ export DATABASE_HOST=localhost
+
+Index the Solr statistics core to populate the PostgreSQL database:
+
+    $ python -m dspace_statistics_api.indexer
+
+Run the REST API:
+
+    $ gunicorn dspace_statistics_api.app
+
+Test to see if there are any statistics:
+
+    $ curl 'http://localhost:8000/items?limit=1'
 
 ## Deployment
-There are example systemd service and timer units in the `contrib` directory.
+There are example systemd service and timer units in the `contrib` directory. The API service listens on localhost by default so you will need to expose it publicly using a web server like nginx.
+
+An example nginx configuration is:
+
+```
+server {
+    #...
+
+    location ~ /rest/statistics/?(.*) {
+        access_log /var/log/nginx/statistics.log;
+        proxy_pass http://statistics_api/$1$is_args$args;
+    }
+}
+
+upstream statistics_api {
+    server 127.0.0.1:5000;
+}
+```
+
+This would expose the API at `/rest/statistics`.
 
 ## Using the API
 The API exposes the following endpoints:
@@ -34,12 +67,13 @@ The API exposes the following endpoints:
 ## Todo
 
 - Add API documentation
-- Close up DB connection when gunicorn shuts down gracefully
+- Close DB connection when gunicorn shuts down gracefully
 - Better logging
 - Tests
 - Check if database exists (try/except)
 - Version API
 - Use JSON in PostgreSQL
+- Switch to [Python 3.6+ f-string syntax](https://realpython.com/python-f-strings/)
 
 ## License
 This work is licensed under the [GPLv3](https://www.gnu.org/licenses/gpl-3.0.en.html).

contrib/dspace-statistics-api.service

@@ -9,10 +9,10 @@ Environment=DATABASE_PASS=dspacestatistics
 Environment=DATABASE_HOST=localhost
 User=nobody
 Group=nogroup
-WorkingDirectory=/opt/ilri/dspace-statistics-api
+WorkingDirectory=/var/lib/dspace-statistics-api
-ExecStart=/opt/ilri/dspace-statistics-api/venv/bin/gunicorn \
+ExecStart=/var/lib/dspace-statistics-api/venv/bin/gunicorn \
           --bind 127.0.0.1:5000                             \
-          app:api
+          dspace_statistics_api.app
 ExecReload=/bin/kill -s HUP $MAINPID
 ExecStop=/bin/kill -s TERM $MAINPID
 

contrib/dspace-statistics-indexer.service

@@ -10,8 +10,8 @@ Environment=DATABASE_PASS=dspacestatistics
 Environment=DATABASE_HOST=localhost
 User=nobody
 Group=nogroup
-WorkingDirectory=/opt/ilri/dspace-statistics-api
+WorkingDirectory=/var/lib/dspace-statistics-api
-ExecStart=/opt/ilri/dspace-statistics-api/venv/bin/python indexer.py
+ExecStart=/var/lib/dspace-statistics-api/venv/bin/python -m dspace_statistics_api.indexer
 
 [Install]
 WantedBy=multi-user.target

dspace_statistics_api/app.py

@@ -1,10 +1,8 @@
-from database import database_connection
+from .database import database_connection
 import falcon
-from solr import solr_connection
 
 db = database_connection()
 db.set_session(readonly=True)
-solr = solr_connection()
 
 class AllItemsResource:
     def on_get(self, req, resp):
@@ -65,7 +63,7 @@ class ItemResource:
 
         cursor.close()
 
-api = falcon.API()
+api = application = falcon.API()
 api.add_route('/items', AllItemsResource())
 api.add_route('/item/{item_id:int}', ItemResource())
 

dspace_statistics_api/database.py

@@ -1,7 +1,7 @@
-from config import DATABASE_NAME
+from .config import DATABASE_NAME
-from config import DATABASE_USER
+from .config import DATABASE_USER
-from config import DATABASE_PASS
+from .config import DATABASE_PASS
-from config import DATABASE_HOST
+from .config import DATABASE_HOST
 import psycopg2, psycopg2.extras
 
 def database_connection():

dspace_statistics_api/indexer.py

@@ -1,4 +1,3 @@
-#!/usr/bin/env python
 #
 # indexer.py
 #
@@ -30,10 +29,10 @@
 # See: https://solrclient.readthedocs.io/en/latest/SolrClient.html
 # See: https://wiki.duraspace.org/display/DSPACE/Solr
 
-from database import database_connection
+from .database import database_connection
-import ujson
+import json
 import psycopg2.extras
-from solr import solr_connection
+from .solr import solr_connection
 
 def index_views():
     # get total number of distinct facets for items with a minimum of 1 view,
@@ -56,7 +55,7 @@ def index_views():
     }, rows=0)
 
     # get total number of distinct facets (countDistinct)
-    results_totalNumFacets = ujson.loads(res.get_json())['stats']['stats_fields']['id']['countDistinct']
+    results_totalNumFacets = json.loads(res.get_json())['stats']['stats_fields']['id']['countDistinct']
 
     # divide results into "pages" (cast to int to effectively round down)
     results_per_page = 100
@@ -115,7 +114,7 @@ def index_downloads():
     }, rows=0)
 
     # get total number of distinct facets (countDistinct)
-    results_totalNumFacets = ujson.loads(res.get_json())['stats']['stats_fields']['owningItem']['countDistinct']
+    results_totalNumFacets = json.loads(res.get_json())['stats']['stats_fields']['owningItem']['countDistinct']
 
     # divide results into "pages" (cast to int to effectively round down)
     results_per_page = 100

dspace_statistics_api/solr.py

@@ -1,4 +1,4 @@
-from config import SOLR_SERVER
+from .config import SOLR_SERVER
 from SolrClient import SolrClient
 
 def solr_connection():

requirements.txt

@@ -1,4 +1,4 @@
-certifi==2018.8.24
+certifi==2018.10.15
 chardet==3.0.4
 falcon==1.4.1
 gunicorn==19.9.0
@@ -6,8 +6,7 @@ idna==2.7
 kazoo==2.5.0
 psycopg2-binary==2.7.5
 python-mimeparse==1.6.0
-requests==2.19.1
+requests==2.20.0
 six==1.11.0
-SolrClient==0.2.1
+-e git://github.com/alanorth/SolrClient.git@c629e3475be37c82770b2be61748be7e29882648#egg=SolrClient
-ujson==1.35
+urllib3==1.24
-urllib3==1.23