The "totalPages" value in our response is calculated incorrectly.
Instead of casting to int and rounding, we should rather round up
to the next integer with math.ceil. This is a more correct way to
get the value.
Also update the indexer to use the same logic, although there the
values are printed with +1 so they are more readable.
Don't use a static HTML file anymore. Now I simply print an XHTML
page from the Falcon resource. This way I can use variables to add
in the API version as well as a link to the Swagger UI.
The list of API calls is still present on the README.md, though in
the long run I might move them to some dedicated documentation or
a GitHub wiki.
continuous-integration/drone/push Build is passingDetails
It turns out that Swagger UI mostly does the "right" thing for our
use cases here, but it assumes that API paths are relative to the
root of the host where it is being served. This works in the local
development environment because we are serving on "/", but it does
not work in production where the API is deployed beneath the DSpace
REST API, for example at "/rest/statistics".
The solution here is to allow configuration of the DSpace Statistics
API path and use that when registering the Swagger UI as well as in
a new "server" block in the OpenAPI JSON schema.
By default it is configured to work out of the box in a development
environment. Set the DSPACE_STATISTICS_API_URL environment variable
to something like "/rest/statistics" when running in production.
This includes a Swagger UI with an OpenAPI 3.0 JSON schema for easy
interactive demonstration and testing of the API. The JSON schema
was created with the standalone swagger-editor. Includes tests to
make sure that the /swagger and /docs/openapi.json paths are acce-
ssible.
The basic logic is similar to items, where you can request single
item statistics with a UUID, all item statistics, and item statis-
tics for a list of items (optionally with a date range). Most of
the item code was re-purposed to work on "elements", which can be
items, communities, or collections depending on the request, with
the use of Falcon's `before` hooks to set the statistics scope so
we know how to behave for the current request.
Other than the minor difference in facet fields, another issue I
had with communities and collections is that the owningComm and
owningColl fields are multi-valued (unlike items' id field). This
means that, when you facet the results of your query, Solr returns
ids that seem unrelated, but are actually present in the field, so
I had to make sure I checked all returned ids to see if they were
in the user's POSTed elements list.
TODO:
- Add tests
- Revise docstrings
- Refactor items.py as it is now generic
A few months ago I had an issue setting up mocking because I was
trying to be clever importing these libraries only when I needed
them rather than at the global scope. Someone pointed out to me
that if the imports are at the top of the file Falcon will load
them once when the WSGI server starts, whereas if they are in the
on_get() or on_post() they will load for every request! Also, it
seems that PEP8 recommends keeping imports at the top of the file
anyways, so I will just do that.
Imports sorted with isort.
See: https://www.python.org/dev/peps/pep-0008/#imports
I thought it was clever to only import these in the on_post handler
because they aren't needed elsewhere, but it turns out that this is
not a common pattern and even causes problems with testability.
First, if the imports are at the top of the file as PEP8 recommends,
then the WSGI server will import them once when it loads the app and
they remain in memory for the lifecycle of the app. If the imports
are in the on_post handler they would be re-imported on every request!
Second, this pattern of importing in a method makes it tricky to use
object patching in mocks.
See: https://www.python.org/dev/peps/pep-0008/#imports
You can now POST a JSON request to /items with a list of items and
a date range. This allows the possibility to get view and download
statistics for arbitrary items and arbitrary date ranges.
The JSON request should be in the following format:
{
"limit": 100,
"page": 0,
"dateFrom": "2020-01-01T00:00:00Z",
"dateTo": "2020-09-09T00:00:00Z",
"items": [
"f44cf173-2344-4eb2-8f00-ee55df32c76f",
"2324aa41-e9de-4a2b-bc36-16241464683e",
"8542f9da-9ce1-4614-abf4-f2e3fdb4b305",
"0fe573e7-042a-4240-a4d9-753b61233908"
]
}
The limit, page, and date parameters are all optional. By default
it will use a limit of 100, page 0, and [* TO *] Solr date range.
We had previously been avoiding the f-strings because we needed to
run on Python 3.5 and they were only available in Python 3.6+, but
now the black formatter requires Python 3.6 and all our systems are
running Python 3.6+ anyways.
DSpace 6+ uses a UUID for item identifiers instead of an integer so
we need to adapt our PostgreSQL queries to use those. Note that we
can no longer sort results in the "all items" endpoint by ID. Also,
we need to use parameterized psycopg2 queries instead of strings to
support queries with UUIDs properly. To use the Python UUID objects
elsewhere in the code we need to make sure that we cast them to str.
Flake8 validates code style against PEP 8 in order to encourage the
writing of idiomatic Python. For reference, I am currently ignoring
errors about line length (E501) because I feel it makes code harder
to read.
This is the invocation I am using:
$ flake8 --ignore E501 dspace_statistics_api
I had imagined plugging in an interactive Swagger or OpenAPI instance
here, but that's actually much more involved in Falcon than I want to
deal with right now.
Alan Orth