1
0
mirror of https://github.com/ilri/dspace-statistics-api.git synced 2025-05-18 10:41:55 +02:00

Compare commits

..

34 Commits

Author SHA1 Message Date
ef0991e352 Update requirements
All checks were successful
continuous-integration/drone/push Build is passing
Generated with poetry export:

    $ poetry export --without-hashes -f requirements.txt > requirements.txt
    $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt

The `--without-hashes` is required to work around an issue with
gunicorn pulling in a dependency on setuptools that poetry ignores.

See: https://github.com/python-poetry/poetry/issues/1584
2021-06-22 10:11:57 +03:00
4502d6053c poetry.lock: run poetry update
The following packages were updated:

> markupsafe (2.0.0 -> 2.0.1)
> certifi (2020.12.5 -> 2021.5.30)
> click (8.0.0 -> 8.0.1)
> decorator (5.0.7 -> 5.0.9)
> jinja2 (3.0.0 -> 3.0.1)
> prompt-toolkit (3.0.18 -> 3.0.19)
> urllib3 (1.26.4 -> 1.26.5)
> ipython (7.23.1 -> 7.24.1)
> psycopg2-binary (2.8.6 -> 2.9.1)
2021-06-22 10:10:29 +03:00
a524068cf6 Bump version to 1.4.3-dev
All checks were successful
continuous-integration/drone/push Build is passing
2021-04-15 14:44:44 +03:00
964d5dff06 Version 1.4.2 2021-04-15 14:23:07 +03:00
a9252d1771 Update requirements-dev.txt
Generated with poetry export:

    $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt

The `--without-hashes` is required to work around an issue with
gunicorn pulling in a dependency on setuptools that poetry ignores.

See: https://github.com/python-poetry/poetry/issues/1584
2021-04-15 14:19:48 +03:00
a63687d516 poetry.lock: Run poetry update 2021-04-15 14:17:17 +03:00
73dc3a292e README.md: Remove TODO about Swagger
All checks were successful
continuous-integration/drone/push Build is passing
I added the SwaggerUI interface a few months ago.
2021-04-06 20:28:10 +03:00
1e742bad41 CHANGELOG.md: Add note about valid page tests
All checks were successful
continuous-integration/drone/push Build is passing
2021-04-06 09:07:51 +03:00
164008981e CHANGELOG.md: Add notes about Falcon 3.0.0
All checks were successful
continuous-integration/drone/push Build is passing
2021-04-06 08:58:00 +03:00
dd1769b954 tests: Fix totalPages
A few months ago I fixed the totalPages display to show 1 when we
only have one page of results (the page itself is still 0), but I
didn't update the tests.

See: 4f8cd1097bd0b1384540518304cbfdb88f73d151
2021-04-06 08:54:54 +03:00
b009820fb4 Update requirements
Generated with poetry export:

    $ poetry export --without-hashes -f requirements.txt > requirements.txt
    $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt

The `--without-hashes` is required to work around an issue with
gunicorn pulling in a dependency on setuptools that poetry ignores.

See: https://github.com/python-poetry/poetry/issues/1584
2021-04-06 08:32:22 +03:00
9830295978 poetry.lock: Run poetry update 2021-04-06 08:31:50 +03:00
c93a4d7455 pyproject.toml: Falcon 3.0.0
Release notes: https://falcon.readthedocs.io/en/latest/changes/3.0.0.html
2021-04-06 08:31:39 +03:00
2f8e4f8a0a Changes for Falcon 3.0.0
Mostly it seems we just need to use resp.text instead of resp.body,
including in falcon-swagger-ui (I forked the upstream one to make
this change).

See: https://falcon.readthedocs.io/en/latest/changes/3.0.0.html
2021-04-06 08:30:28 +03:00
0650c5985e Add SPDX short license identifier to all Python files
All checks were successful
continuous-integration/drone/push Build is passing
See: https://spdx.github.io/spdx-spec/appendix-V-using-SPDX-short-identifiers-in-source-files/
2021-03-22 13:42:42 +02:00
d814f1c4f0 CHANGELOG.md: Fix heading
All checks were successful
continuous-integration/drone/push Build is passing
2021-03-21 19:50:39 +02:00
00f30591c4 CHANGELOG.md: Add notes about GitHub Actions 2021-03-21 19:49:35 +02:00
acfe87b91a Add GitHub Actions badge and remove sr.ht
All checks were successful
continuous-integration/drone/push Build is passing
2021-03-21 11:48:05 +02:00
bc6d84dda2 Add GitHub Actions workflow
My first time setting up a PostgreSQL service container on GitHub
actions. Note that there are two different kinds of environment
variables: those passed to the Docker container, and those used by
the PostgreSQL utilities.

See: https://docs.github.com/en/actions/guides/creating-postgresql-service-containers
See: https://hub.docker.com/_/postgres
2021-03-21 11:44:39 +02:00
889fb2f74a Update requirements
All checks were successful
continuous-integration/drone/push Build is passing
Generated with poetry export:

    $ poetry export --without-hashes -f requirements.txt > requirements.txt
    $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt

The `--without-hashes` is required to work around an issue with
gunicorn pulling in a dependency on setuptools that poetry ignores.

See: https://github.com/python-poetry/poetry/issues/1584
2021-03-21 08:59:41 +02:00
c42cd7a818 poetry.lock: Run poetry update 2021-03-21 08:59:04 +02:00
f8bba59d66 .gitignore: Ignore .egg-info
All checks were successful
continuous-integration/drone/push Build is passing
2021-03-14 21:50:47 +02:00
b8cb752a29 CHANGELOG.md: Add note about updated poetry deps
All checks were successful
continuous-integration/drone/push Build is passing
2021-03-11 11:23:18 +02:00
09496aa2b5 Update requirements
Generated with poetry export:

    $ poetry export --without-hashes -f requirements.txt > requirements.txt
    $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt

The `--without-hashes` is required to work around an issue with
gunicorn pulling in a dependency on setuptools that poetry ignores.

See: https://github.com/python-poetry/poetry/issues/1584
2021-03-11 11:22:05 +02:00
ff5dc7506d poetry.lock: Run poetry update 2021-03-11 11:21:02 +02:00
80a11ead97 Version 1.4.1
All checks were successful
continuous-integration/drone/push Build is passing
2021-01-14 14:19:50 +02:00
a282c95933 CHANGELOG.md: Minor syntax fix 2021-01-14 14:15:57 +02:00
fd7cc36306 Update requirements-dev.txt
Generated with poetry export:

    $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt

The `--without-hashes` is required to work around an issue with
gunicorn pulling in a dependency on setuptools that poetry ignores.

See: https://github.com/python-poetry/poetry/issues/1584
2021-01-14 14:14:16 +02:00
a20ff09570 poetry.lock: Run poetry update
All tests still pass.
2021-01-14 14:13:32 +02:00
fdc0e73088 tests: Sort imports with isort 2021-01-14 14:12:59 +02:00
b15afc9f39 CHANGELOG.md: Add note about UUIDs
All checks were successful
continuous-integration/drone/push Build is passing
2021-01-05 12:41:21 +02:00
2bc18ef719 README.md: Make a note about migrating UUIDs 2021-01-05 12:35:23 +02:00
49751b53f0 dspace_statistics_api/indexer.py: Limit to UUIDs
We need to make sure that the indexer only tries to index UUIDs, as
opposed to legacy IDs that may have been left over from a migration
from earlier DSpace versions. For example, "98110-unmigrated", "-1"
etc.

For matching the UUIDs in Solr I decided that it is sufficient for
our use case to simply match thirty-six characters, where a UUID is
composed of thirty-two hexadecimal characters and four dashes. We
don't need to do any verification of "real" UUIDs because it would
be needlessly complex in our case.

See: https://github.com/ilri/dspace-statistics-api/issues/12
2021-01-05 12:30:27 +02:00
d1c177e146 .drone.yml: Add git to python container
All checks were successful
continuous-integration/drone/push Build is passing
Now that I am installing my own fork of falcon-swagger-ui we need
to have git so we can install it with pip.
2020-12-27 14:22:23 +02:00
21 changed files with 555 additions and 472 deletions

View File

@ -1,21 +0,0 @@
image: archlinux
packages:
- python-poetry
- postgresql
sources:
- https://git.sr.ht/~alanorth/dspace-statistics-api
tasks:
- setup: |
id
psql --version
sudo su - postgres -c "initdb --locale en_US.UTF-8 -E UTF8 -D '/var/lib/postgres/data'"
sudo systemctl start postgresql
createuser -U postgres dspacestatistics
psql -U postgres -c "ALTER USER dspacestatistics WITH PASSWORD 'dspacestatistics'"
createdb -U postgres -O dspacestatistics --encoding=UNICODE dspacestatistics
cd dspace-statistics-api
psql -U postgres -d dspacestatistics < tests/dspacestatistics.sql
poetry install --no-root
- test: |
cd dspace-statistics-api
poetry run pytest

View File

@ -24,7 +24,7 @@ steps:
commands: commands:
- id - id
- python -V - python -V
- apt update && apt install -y gcc - apt update && apt install -y gcc git
- pip install -r requirements-dev.txt - pip install -r requirements-dev.txt
- pytest - pytest
@ -71,6 +71,7 @@ steps:
commands: commands:
- id - id
- python -V - python -V
- apt update && apt install -y git
- pip install -r requirements-dev.txt - pip install -r requirements-dev.txt
- pytest - pytest
@ -109,6 +110,7 @@ steps:
commands: commands:
- id - id
- python -V - python -V
- apt update && apt install -y git
- pip install -r requirements-dev.txt - pip install -r requirements-dev.txt
- pytest - pytest
@ -147,6 +149,7 @@ steps:
commands: commands:
- id - id
- python -V - python -V
- apt update && apt install -y git
- pip install -r requirements-dev.txt - pip install -r requirements-dev.txt
- pytest - pytest

61
.github/workflows/python-app.yml vendored Normal file
View File

@ -0,0 +1,61 @@
# This workflow will install Python dependencies, run tests and lint with a single version of Python
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
name: Build and Test
on: ['push', 'pull_request']
jobs:
build:
runs-on: ubuntu-latest
services:
database:
image: postgres:10-alpine
env:
# password for postgres user in the Docker container
POSTGRES_PASSWORD: postgres
# default database to create
POSTGRES_DB: dspacestatistics
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5432:5432
steps:
- uses: actions/checkout@v2
- name: Set up Python 3.9
uses: actions/setup-python@v2
with:
python-version: 3.9
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install flake8 pytest
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
if [ -f requirements-dev.txt ]; then pip install -r requirements-dev.txt; fi
- name: Lint with flake8
run: |
# stop the build if there are Python syntax errors or undefined names
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
- name: Set up PostgreSQL
run: |
pg_isready -U postgres -d dspacestatistics
createuser -U postgres dspacestatistics
psql -U postgres -c "ALTER USER dspacestatistics WITH PASSWORD 'dspacestatistics'"
psql -U postgres -d dspacestatistics < tests/dspacestatistics.sql
env:
PGHOST: localhost
PGPASSWORD: postgres
- name: Test with pytest
run: |
pytest
env:
PGHOST: localhost
PGPASSWORD: dspacestatistics

1
.gitignore vendored
View File

@ -1,2 +1,3 @@
__pycache__ __pycache__
venv venv
*.egg-info

View File

@ -4,6 +4,25 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## 1.4.2 - 2021-04-14
### Updated
- Update dependencies with `poetry update`
- Falcon 3.0.0, a minor change for us, but good to be using a current upstream
version
### Fixed
- Bug in several of the "valid page" tests
### Added
- GitHub Actions workflow to build and test the API
## [1.4.1] - 2021-01-14
### Changed
- Limit Solr query to UUIDs to avoid errors with unmigrated legacy stats (https://github.com/ilri/dspace-statistics-api/issues/12)
### Updated
- Dev dependencies
## [1.4.0] - 2020-12-27 ## [1.4.0] - 2020-12-27
### Added ### Added
- indexer.py now indexes views and downloads for communities and collections - indexer.py now indexes views and downloads for communities and collections
@ -72,7 +91,7 @@ and gunicorn 20.0.4
- Minor syntax issues highlighted by flake8 - Minor syntax issues highlighted by flake8
## [1.1.0] - 2019-05-05 ## [1.1.0] - 2019-05-05
## Updated ### Updated
- Falcon 2.0.0 (@alanorth) - Falcon 2.0.0 (@alanorth)
## [1.0.0] - 2019-04-15 ## [1.0.0] - 2019-04-15
@ -90,7 +109,7 @@ and gunicorn 20.0.4
## [0.9.0] - 2019-01-22 ## [0.9.0] - 2019-01-22
### Updated ### Updated
- pytest version 4.0.0 - pytest version 4.0.0
- Fix indexing of sharded statistics cores ([#10)) - Fix indexing of sharded statistics cores (#10)
- Handle case of missing views/downloads gracefully - Handle case of missing views/downloads gracefully
## [0.8.1] - 2018-11-14 ## [0.8.1] - 2018-11-14

View File

@ -1,8 +1,9 @@
# DSpace Statistics API [![Build Status](https://ci.mjanja.ch/api/badges/alanorth/dspace-statistics-api/status.svg?ref=refs/heads/v6_x)](https://ci.mjanja.ch/alanorth/dspace-statistics-api) [![builds.sr.ht status](https://builds.sr.ht/~alanorth/dspace-statistics-api.svg)](https://builds.sr.ht/~alanorth/dspace-statistics-api?) # DSpace Statistics API [![Build Status](https://ci.mjanja.ch/api/badges/alanorth/dspace-statistics-api/status.svg?ref=refs/heads/v6_x)](https://ci.mjanja.ch/alanorth/dspace-statistics-api) [![Build and Test](https://github.com/ilri/dspace-statistics-api/actions/workflows/python-app.yml/badge.svg)](https://github.com/ilri/dspace-statistics-api/actions/workflows/python-app.yml)
DSpace stores item view and download events in a Solr "statistics" core. This information is available for use in the various DSpace user interfaces, but is not exposed externally via any APIs. The DSpace 4/5/6 [REST API](https://wiki.lyrasis.org/display/DSDOC5x/REST+API), for example, only exposes _metadata_ about communities, collections, items, and bitstreams. DSpace stores item view and download events in a Solr "statistics" core. This information is available for use in the various DSpace user interfaces, but is not exposed externally via any APIs. The DSpace 4/5/6 [REST API](https://wiki.lyrasis.org/display/DSDOC5x/REST+API), for example, only exposes _metadata_ about communities, collections, items, and bitstreams.
- If your DSpace is version 4 or 5, use [dspace-statistics-api v1.1.1](https://github.com/ilri/dspace-statistics-api/releases/tag/v1.1.1) - If your DSpace is version 4 or 5, use [dspace-statistics-api v1.1.1](https://github.com/ilri/dspace-statistics-api/releases/tag/v1.1.1)
- If your DSpace is version 6+, use [dspace-statistics-api v1.2.0 or greater](https://github.com/ilri/dspace-statistics-api/releases/tag/v1.2.0) - If your DSpace is version 6+, use [dspace-statistics-api v1.2.0 or greater](https://github.com/ilri/dspace-statistics-api/releases/tag/v1.2.0)
- Please make sure your statistics have been migrated from integers to UUIDs with the [solr-upgrade-statistics-6x](https://wiki.lyrasis.org/display/DSDOC6x/SOLR+Statistics+Maintenance) command
This project contains an indexer and a [Falcon-based](https://falcon.readthedocs.io/) web application to make the item, community, and collection statistics available via a simple REST API. You can read more about the Solr queries used to gather the item view and download statistics on the [DSpace wiki](https://wiki.lyrasis.org/display/DSPACE/Solr). This project contains an indexer and a [Falcon-based](https://falcon.readthedocs.io/) web application to make the item, community, and collection statistics available via a simple REST API. You can read more about the Solr queries used to gather the item view and download statistics on the [DSpace wiki](https://wiki.lyrasis.org/display/DSPACE/Solr).
@ -119,7 +120,6 @@ The id is the *internal* UUID for an item, community, or collection. You can get
- Use JSON in PostgreSQL - Use JSON in PostgreSQL
- Add top items endpoint, perhaps `/top/items` or `/items/top`? - Add top items endpoint, perhaps `/top/items` or `/items/top`?
- Actually we could add `/items?limit=10&sort=views` - Actually we could add `/items?limit=10&sort=views`
- Add Swagger with OpenAPI 3.0.x with [falcon-swagger-ui](https://github.com/rdidyk/falcon-swagger-ui)
## License ## License
This work is licensed under the [GPLv3](https://www.gnu.org/licenses/gpl-3.0.en.html). This work is licensed under the [GPLv3](https://www.gnu.org/licenses/gpl-3.0.en.html).

View File

@ -1,3 +1,5 @@
# SPDX-License-Identifier: GPL-3.0-only
import json import json
import math import math
@ -29,7 +31,7 @@ class RootResource:
"</html" "</html"
) )
resp.body = docs_html resp.text = docs_html
class StatusResource: class StatusResource:
@ -61,7 +63,7 @@ class OpenAPIJSONResource:
# Set the version in the schema so Swagger UI can display it # Set the version in the schema so Swagger UI can display it
data["info"]["version"] = VERSION data["info"]["version"] = VERSION
resp.body = json.dumps(data) resp.text = json.dumps(data)
class AllStatisticsResource: class AllStatisticsResource:
@ -213,24 +215,24 @@ class SingleStatisticsResource:
resp.media = statistics resp.media = statistics
api = application = falcon.API() app = application = falcon.App()
api.add_route("/", RootResource()) app.add_route("/", RootResource())
api.add_route("/status", StatusResource()) app.add_route("/status", StatusResource())
# Item routes # Item routes
api.add_route("/items", AllStatisticsResource()) app.add_route("/items", AllStatisticsResource())
api.add_route("/item/{id_:uuid}", SingleStatisticsResource()) app.add_route("/item/{id_:uuid}", SingleStatisticsResource())
# Community routes # Community routes
api.add_route("/communities", AllStatisticsResource()) app.add_route("/communities", AllStatisticsResource())
api.add_route("/community/{id_:uuid}", SingleStatisticsResource()) app.add_route("/community/{id_:uuid}", SingleStatisticsResource())
# Collection routes # Collection routes
api.add_route("/collections", AllStatisticsResource()) app.add_route("/collections", AllStatisticsResource())
api.add_route("/collection/{id_:uuid}", SingleStatisticsResource()) app.add_route("/collection/{id_:uuid}", SingleStatisticsResource())
# Route to the Swagger UI OpenAPI schema # Route to the Swagger UI Openapp schema
api.add_route("/docs/openapi.json", OpenAPIJSONResource()) app.add_route("/docs/openapi.json", OpenAPIJSONResource())
# Path to host the Swagger UI. Keep in mind that Falcon will add a route for # Path to host the Swagger UI. Keep in mind that Falcon will add a route for
# this automatically when we register Swagger and the path will be relative # this automatically when we register Swagger and the path will be relative
@ -240,12 +242,12 @@ SWAGGERUI_PATH = "/swagger"
# The *absolute* path to the OpenJSON schema. This must be absolute because # The *absolute* path to the OpenJSON schema. This must be absolute because
# it will be requested by the client and must resolve absolutely. Note: the # it will be requested by the client and must resolve absolutely. Note: the
# name of this variable is misleading because it is actually the schema URL # name of this variable is misleading because it is actually the schema URL
# but we pass it into the register_swaggerui_app() function as the api_url # but we pass it into the register_swaggerui_app() function as the app_url
# parameter. # parameter.
SWAGGERUI_API_URL = f"{DSPACE_STATISTICS_API_URL}/docs/openapi.json" SWAGGERUI_API_URL = f"{DSPACE_STATISTICS_API_URL}/docs/openapi.json"
register_swaggerui_app( register_swaggerui_app(
api, app,
SWAGGERUI_PATH, SWAGGERUI_PATH,
SWAGGERUI_API_URL, SWAGGERUI_API_URL,
config={ config={

View File

@ -1,3 +1,5 @@
# SPDX-License-Identifier: GPL-3.0-only
import os import os
# Check if Solr connection information was provided in the environment # Check if Solr connection information was provided in the environment
@ -16,6 +18,6 @@ DATABASE_PORT = os.environ.get("DATABASE_PORT", "5432")
# the vanilla DSpace REST API. # the vanilla DSpace REST API.
DSPACE_STATISTICS_API_URL = os.environ.get("DSPACE_STATISTICS_API_URL", "") DSPACE_STATISTICS_API_URL = os.environ.get("DSPACE_STATISTICS_API_URL", "")
VERSION = "1.4.0" VERSION = "1.4.3-dev"
# vim: set sw=4 ts=4 expandtab: # vim: set sw=4 ts=4 expandtab:

View File

@ -1,3 +1,5 @@
# SPDX-License-Identifier: GPL-3.0-only
import falcon import falcon
import psycopg2 import psycopg2
import psycopg2.extras import psycopg2.extras

View File

@ -1,7 +1,7 @@
{ {
"openapi": "3.0.3", "openapi": "3.0.3",
"info": { "info": {
"version": "1.4.0", "version": "1.4.3-dev",
"title": "DSpace Statistics API", "title": "DSpace Statistics API",
"description": "A [Falcon-based](https://falcon.readthedocs.io/) web application to make DSpace's item, community, and collection statistics available via a simple REST API. This Swagger interface is powered by [falcon-swagger-ui](https://github.com/rdidyk/falcon-swagger-ui).", "description": "A [Falcon-based](https://falcon.readthedocs.io/) web application to make DSpace's item, community, and collection statistics available via a simple REST API. This Swagger interface is powered by [falcon-swagger-ui](https://github.com/rdidyk/falcon-swagger-ui).",
"license": { "license": {

View File

@ -1,23 +1,7 @@
# SPDX-License-Identifier: GPL-3.0-only
# #
# indexer.py # indexer.py
# #
# Copyright 2018 Alan Orth.
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
# ---
#
# Connects to a DSpace Solr statistics core and ingests views and downloads for # Connects to a DSpace Solr statistics core and ingests views and downloads for
# communities, collections, and items into a PostgreSQL database. # communities, collections, and items into a PostgreSQL database.
# #
@ -47,7 +31,7 @@ def index_views(indexType: str, facetField: str):
# #
# see: https://lucene.apache.org/solr/guide/6_6/the-stats-component.html # see: https://lucene.apache.org/solr/guide/6_6/the-stats-component.html
solr_query_params = { solr_query_params = {
"q": "type:2", "q": f"type:2 AND {facetField}:/.{{36}}/",
"fq": "-isBot:true AND statistics_type:view", "fq": "-isBot:true AND statistics_type:view",
"fl": facetField, "fl": facetField,
"facet": "true", "facet": "true",
@ -94,7 +78,7 @@ def index_views(indexType: str, facetField: str):
) )
solr_query_params = { solr_query_params = {
"q": "type:2", "q": f"type:2 AND {facetField}:/.{{36}}/",
"fq": "-isBot:true AND statistics_type:view", "fq": "-isBot:true AND statistics_type:view",
"fl": facetField, "fl": facetField,
"facet": "true", "facet": "true",
@ -130,7 +114,7 @@ def index_views(indexType: str, facetField: str):
def index_downloads(indexType: str, facetField: str): def index_downloads(indexType: str, facetField: str):
# get the total number of distinct facets for items with at least 1 download # get the total number of distinct facets for items with at least 1 download
solr_query_params = { solr_query_params = {
"q": "type:0", "q": f"type:0 AND {facetField}:/.{{36}}/",
"fq": "-isBot:true AND statistics_type:view AND bundleName:ORIGINAL", "fq": "-isBot:true AND statistics_type:view AND bundleName:ORIGINAL",
"fl": facetField, "fl": facetField,
"facet": "true", "facet": "true",
@ -176,7 +160,7 @@ def index_downloads(indexType: str, facetField: str):
) )
solr_query_params = { solr_query_params = {
"q": "type:0", "q": f"type:0 AND {facetField}:/.{{36}}/",
"fq": "-isBot:true AND statistics_type:view AND bundleName:ORIGINAL", "fq": "-isBot:true AND statistics_type:view AND bundleName:ORIGINAL",
"fl": facetField, "fl": facetField,
"facet": "true", "facet": "true",

View File

@ -1,3 +1,5 @@
# SPDX-License-Identifier: GPL-3.0-only
import requests import requests
from .config import SOLR_SERVER from .config import SOLR_SERVER

View File

@ -1,3 +1,5 @@
# SPDX-License-Identifier: GPL-3.0-only
import datetime import datetime
import json import json
import re import re

672
poetry.lock generated

File diff suppressed because it is too large Load Diff

View File

@ -1,6 +1,6 @@
[tool.poetry] [tool.poetry]
name = "dspace-statistics-api" name = "dspace-statistics-api"
version = "1.4.0" version = "1.4.3-dev"
description = "A simple REST API to expose Solr view and download statistics for items, communities, and collections in a DSpace repository." description = "A simple REST API to expose Solr view and download statistics for items, communities, and collections in a DSpace repository."
authors = ["Alan Orth <aorth@mjanja.ch>"] authors = ["Alan Orth <aorth@mjanja.ch>"]
license = "GPL-3.0-only" license = "GPL-3.0-only"
@ -8,10 +8,10 @@ license = "GPL-3.0-only"
[tool.poetry.dependencies] [tool.poetry.dependencies]
python = "^3.6" python = "^3.6"
gunicorn = "^20.0.4" gunicorn = "^20.0.4"
falcon = "^2.0.0" falcon = "3.0.0"
psycopg2-binary = "^2.8.6" psycopg2-binary = "^2.8.6"
requests = "^2.24.0" requests = "^2.24.0"
falcon-swagger-ui = {git = "https://github.com/alanorth/falcon-swagger-ui.git"} falcon-swagger-ui = {git = "https://github.com/alanorth/falcon-swagger-ui.git", rev="falcon-300b1"}
[tool.poetry.dev-dependencies] [tool.poetry.dev-dependencies]
ipython = { version = "^7.18.1", python = "^3.7" } ipython = { version = "^7.18.1", python = "^3.7" }

View File

@ -1,51 +1,52 @@
appdirs==1.4.4 appdirs==1.4.4; python_version >= "3.6"
appnope==0.1.2; python_version >= "3.7" and python_version < "4.0" and sys_platform == "darwin" appnope==0.1.2; python_version >= "3.7" and python_version < "4.0" and sys_platform == "darwin"
atomicwrites==1.4.0; sys_platform == "win32" atomicwrites==1.4.0; python_version >= "3.6" and python_full_version < "3.0.0" and sys_platform == "win32" or sys_platform == "win32" and python_version >= "3.6" and python_full_version >= "3.4.0"
attrs==20.3.0 attrs==21.2.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.6"
backcall==0.2.0; python_version >= "3.7" and python_version < "4.0" backcall==0.2.0; python_version >= "3.7" and python_version < "4.0"
black==20.8b1 black==20.8b1; python_version >= "3.6"
certifi==2020.12.5 certifi==2021.5.30; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
chardet==4.0.0 chardet==4.0.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
click==7.1.2 click==8.0.1; python_version >= "3.6"
colorama==0.4.4; python_version >= "3.7" and python_version < "4.0" and sys_platform == "win32" or sys_platform == "win32" colorama==0.4.4; python_version >= "3.7" and python_full_version < "3.0.0" and python_version < "4.0" and sys_platform == "win32" and platform_system == "Windows" or python_version >= "3.7" and python_version < "4.0" and sys_platform == "win32" and python_full_version >= "3.5.0" and platform_system == "Windows"
dataclasses==0.6; python_version < "3.7" dataclasses==0.8; python_version >= "3.6" and python_version < "3.7"
decorator==4.4.2; python_version >= "3.7" and python_version < "4.0" decorator==5.0.9; python_version >= "3.7" and python_version < "4.0"
falcon==2.0.0 falcon-swagger-ui @ git+https://github.com/alanorth/falcon-swagger-ui.git@falcon-300b1
-e git+https://github.com/alanorth/falcon-swagger-ui.git@a44244c85dceccfcd249b62fea4ee82a8221e3d2#egg=falcon-swagger-ui falcon==3.0.0; python_version >= "3.5"
flake8==3.8.4 flake8==3.9.2; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0")
gunicorn==20.0.4 gunicorn==20.1.0; python_version >= "3.5"
idna==2.10 idna==2.10; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
importlib-metadata==3.3.0; python_version < "3.8" importlib-metadata==4.5.0; python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.8" or python_full_version >= "3.5.0" and python_version < "3.8" and python_version >= "3.6"
iniconfig==1.1.1 iniconfig==1.1.1; python_version >= "3.6"
ipython==7.19.0; python_version >= "3.7" and python_version < "4.0"
ipython-genutils==0.2.0; python_version >= "3.7" and python_version < "4.0" ipython-genutils==0.2.0; python_version >= "3.7" and python_version < "4.0"
isort==5.6.4 ipython==7.24.1; python_version >= "3.7" and python_version < "4.0"
jedi==0.17.2; python_version >= "3.7" and python_version < "4.0" isort==5.8.0; python_version >= "3.6" and python_version < "4.0"
jinja2==2.11.2 jedi==0.18.0; python_version >= "3.7" and python_version < "4.0"
markupsafe==1.1.1 jinja2==3.0.1; python_version >= "3.6"
mccabe==0.6.1 markupsafe==2.0.1; python_version >= "3.6"
mypy-extensions==0.4.3 matplotlib-inline==0.1.2; python_version >= "3.7" and python_version < "4.0"
packaging==20.8 mccabe==0.6.1; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
parso==0.7.1; python_version >= "3.7" and python_version < "4.0" mypy-extensions==0.4.3; python_version >= "3.6"
pathspec==0.8.1 packaging==20.9; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
parso==0.8.2; python_version >= "3.7" and python_version < "4.0"
pathspec==0.8.1; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.6"
pexpect==4.8.0; python_version >= "3.7" and python_version < "4.0" and sys_platform != "win32" pexpect==4.8.0; python_version >= "3.7" and python_version < "4.0" and sys_platform != "win32"
pickleshare==0.7.5; python_version >= "3.7" and python_version < "4.0" pickleshare==0.7.5; python_version >= "3.7" and python_version < "4.0"
pluggy==0.13.1 pluggy==0.13.1; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
prompt-toolkit==3.0.8; python_version >= "3.7" and python_version < "4.0" prompt-toolkit==3.0.19; python_version >= "3.7" and python_version < "4.0" and python_full_version >= "3.6.1"
psycopg2-binary==2.8.6 psycopg2-binary==2.9.1; python_version >= "3.6"
ptyprocess==0.6.0; python_version >= "3.7" and python_version < "4.0" and sys_platform != "win32" ptyprocess==0.7.0; python_version >= "3.7" and python_version < "4.0" and sys_platform != "win32"
py==1.10.0 py==1.10.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
pycodestyle==2.6.0 pycodestyle==2.7.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
pyflakes==2.2.0 pyflakes==2.3.1; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
pygments==2.7.3; python_version >= "3.7" and python_version < "4.0" pygments==2.9.0; python_version >= "3.7" and python_version < "4.0"
pyparsing==2.4.7 pyparsing==2.4.7; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
pytest==6.2.1 pytest==6.2.4; python_version >= "3.6"
regex==2020.11.13 regex==2021.4.4; python_version >= "3.6"
requests==2.25.1 requests==2.25.1; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0")
toml==0.10.2 toml==0.10.2; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.3.0" and python_version >= "3.6"
traitlets==5.0.5; python_version >= "3.7" and python_version < "4.0" traitlets==5.0.5; python_version >= "3.7" and python_version < "4.0"
typed-ast==1.4.1 typed-ast==1.4.3; python_version >= "3.6"
typing-extensions==3.7.4.3 typing-extensions==3.10.0.0; python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.8" or python_full_version >= "3.5.0" and python_version < "3.8" and python_version >= "3.6"
urllib3==1.26.2 urllib3==1.26.5; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version < "4"
wcwidth==0.2.5; python_version >= "3.7" and python_version < "4.0" wcwidth==0.2.5; python_version >= "3.7" and python_version < "4.0" and python_full_version >= "3.6.1"
zipp==3.4.0; python_version < "3.8" zipp==3.4.1; python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.8" or python_full_version >= "3.5.0" and python_version < "3.8" and python_version >= "3.6"

View File

@ -1,11 +1,11 @@
certifi==2020.12.5 certifi==2021.5.30; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
chardet==4.0.0 chardet==4.0.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
falcon==2.0.0 falcon-swagger-ui @ git+https://github.com/alanorth/falcon-swagger-ui.git@falcon-300b1
-e git+https://github.com/alanorth/falcon-swagger-ui.git@a44244c85dceccfcd249b62fea4ee82a8221e3d2#egg=falcon-swagger-ui falcon==3.0.0; python_version >= "3.5"
gunicorn==20.0.4 gunicorn==20.1.0; python_version >= "3.5"
idna==2.10 idna==2.10; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
jinja2==2.11.2 jinja2==3.0.1; python_version >= "3.6"
markupsafe==1.1.1 markupsafe==2.0.1; python_version >= "3.6"
psycopg2-binary==2.8.6 psycopg2-binary==2.9.1; python_version >= "3.6"
requests==2.25.1 requests==2.25.1; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0")
urllib3==1.26.2 urllib3==1.26.5; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version < "4"

View File

@ -1,14 +1,17 @@
from falcon import testing # SPDX-License-Identifier: GPL-3.0-only
import json import json
import pytest
from unittest.mock import patch from unittest.mock import patch
from dspace_statistics_api.app import api import pytest
from falcon import testing
from dspace_statistics_api.app import app
@pytest.fixture @pytest.fixture
def client(): def client():
return testing.TestClient(api) return testing.TestClient(app)
def test_get_collection(client): def test_get_collection(client):
@ -309,7 +312,7 @@ def test_post_collections_valid_page(client):
assert response.status_code == 200 assert response.status_code == 200
assert response.json["limit"] == 100 assert response.json["limit"] == 100
assert response.json["currentPage"] == 0 assert response.json["currentPage"] == 0
assert response.json["totalPages"] == 0 assert response.json["totalPages"] == 1
assert len(response.json["statistics"]) == 2 assert len(response.json["statistics"]) == 2
assert isinstance(response.json["statistics"][0]["views"], int) assert isinstance(response.json["statistics"][0]["views"], int)
assert isinstance(response.json["statistics"][0]["downloads"], int) assert isinstance(response.json["statistics"][0]["downloads"], int)

View File

@ -1,14 +1,17 @@
from falcon import testing # SPDX-License-Identifier: GPL-3.0-only
import json import json
import pytest
from unittest.mock import patch from unittest.mock import patch
from dspace_statistics_api.app import api import pytest
from falcon import testing
from dspace_statistics_api.app import app
@pytest.fixture @pytest.fixture
def client(): def client():
return testing.TestClient(api) return testing.TestClient(app)
def test_get_community(client): def test_get_community(client):
@ -309,7 +312,7 @@ def test_post_communities_valid_page(client):
assert response.status_code == 200 assert response.status_code == 200
assert response.json["limit"] == 100 assert response.json["limit"] == 100
assert response.json["currentPage"] == 0 assert response.json["currentPage"] == 0
assert response.json["totalPages"] == 0 assert response.json["totalPages"] == 1
assert len(response.json["statistics"]) == 2 assert len(response.json["statistics"]) == 2
assert isinstance(response.json["statistics"][0]["views"], int) assert isinstance(response.json["statistics"][0]["views"], int)
assert isinstance(response.json["statistics"][0]["downloads"], int) assert isinstance(response.json["statistics"][0]["downloads"], int)

View File

@ -1,12 +1,14 @@
from falcon import testing # SPDX-License-Identifier: GPL-3.0-only
import pytest
from dspace_statistics_api.app import api import pytest
from falcon import testing
from dspace_statistics_api.app import app
@pytest.fixture @pytest.fixture
def client(): def client():
return testing.TestClient(api) return testing.TestClient(app)
def test_get_docs(client): def test_get_docs(client):

View File

@ -1,14 +1,17 @@
from falcon import testing # SPDX-License-Identifier: GPL-3.0-only
import json import json
import pytest
from unittest.mock import patch from unittest.mock import patch
from dspace_statistics_api.app import api import pytest
from falcon import testing
from dspace_statistics_api.app import app
@pytest.fixture @pytest.fixture
def client(): def client():
return testing.TestClient(api) return testing.TestClient(app)
def test_get_item(client): def test_get_item(client):
@ -309,7 +312,7 @@ def test_post_items_valid_page(client):
assert response.status_code == 200 assert response.status_code == 200
assert response.json["limit"] == 100 assert response.json["limit"] == 100
assert response.json["currentPage"] == 0 assert response.json["currentPage"] == 0
assert response.json["totalPages"] == 0 assert response.json["totalPages"] == 1
assert len(response.json["statistics"]) == 2 assert len(response.json["statistics"]) == 2
assert isinstance(response.json["statistics"][0]["views"], int) assert isinstance(response.json["statistics"][0]["views"], int)
assert isinstance(response.json["statistics"][0]["downloads"], int) assert isinstance(response.json["statistics"][0]["downloads"], int)