alanorth/dspace-statistics-api
1
0
Fork 0
mirror of https://github.com/ilri/dspace-statistics-api.git synced 2024-11-13 01:57:03 +01:00
Code Releases Activity
85 Commits 8 Branches 37 Tags 1.6 MiB
385a34e5d0
Commit Graph

85 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Alan Orth
385a34e5d0
indexer.py: Use psycopg2's execute_values to batch inserts 
Batch inserts are much faster than a series of individual inserts
because they drastically reduce the overhead caused by round-trip
communication with the server. My tests in development confirm:

  - cursor.execute(): 19 seconds
  - execute_values(): 14 seconds

I'm currently only working with 4,500 rows, but I will experiment
with larger data sets, as well as larger batches. For example, on
the PostgreSQL mailing list a user reports doing 10,000 rows with
a page size of 100.

See: http://initd.org/psycopg/docs/extras.html#psycopg2.extras.execute_values
See: https://github.com/psycopg/psycopg2/issues/491#issuecomment-276551038
 2018-09-26 23:10:29 +03:00
Alan Orth
d0ea62d2bd
database.py: Use one line for psycopg2 imports 2018-09-26 22:23:24 +03:00
Alan Orth
366ae25b8e
README.md: Add link to psycopg2 issue about batch inserts 2018-09-26 22:23:08 +03:00
Alan Orth
0f3054ae03
README.md: Add TODO about batch DB inserts 2018-09-26 16:31:13 +03:00
Alan Orth
6bf34235d4
CHANGELOG.md: Move unreleased changes to version 0.4.0 2018-09-26 02:51:27 +03:00
Alan Orth
e604d8ca81
indexer.py: Major refactor 
Basically Solr's numFound has nothing to do with the actual number
of distinct facets that are returned. You need to use Solr's stats
component to get the number of distinct facets, aka countDistinct.
This is apparently deprecated in newer Solr versions, but we're on
version 4.10 and it works there.

Also, I realized that there is no need to return facets for items
without any views or downloads. Using facet.mincount=1 reduces the
result set size and also means we can store less data in the data-
base. The API returns HTTP 404 Not Found if an item is not in the
database anyways.

I can't figure it out exactly, but there is some weird issue with
Solr's facet results when you don't use facet.mincount=1. For some
reason you get tons of results with an id that doesn't even exist
in the document database, let alone as an actual DSpace item!

See: https://lucene.apache.org/solr/guide/6_6/the-stats-component.html
 2018-09-26 02:41:10 +03:00
Alan Orth
fc35b816f3
CHANGELOG.md: Add unreleased changes 2018-09-25 23:09:44 +03:00
Alan Orth
9e6a2f7559
contrib/dspace-statistics-indexer.timer: Fix syntax 
You can test OnCalendar strings using systemd-analyze calendar, eg:

    # systemd-analyze calendar '*-*-* 06:00:00,18:00:00'
    Failed to parse calendar specification '*-*-* 06:00:00,18:00:00':
    Invalid argument
    # systemd-analyze calendar '*-*-* 06,18:00:00'
    Normalized form: *-*-* 06,18:00:00
        Next elapse: Wed 2018-09-26 06:00:00 EEST
           (in UTC): Wed 2018-09-26 03:00:00 UTC
           From now: 6h left
 2018-09-25 23:07:03 +03:00
Alan Orth
46cfc3ffbc
CHANGELOG.md: Release version 0.3.2 2018-09-25 13:14:08 +03:00
Alan Orth
2850035a4c
Return HTTP 404 when an item id is not found 2018-09-25 13:12:53 +03:00
Alan Orth
c0b550109a
README.md: Improve wording 2018-09-25 12:24:52 +03:00
Alan Orth
bfceffd84d
indexer.py: Improve inline documentation 2018-09-25 12:23:31 +03:00
Alan Orth
d0552f5047
CHANGELOG.md: Move unreleased changes to version 0.3.1 2018-09-25 12:18:26 +03:00
Alan Orth
c3a0bf7f44
CHANGELOG.md: Add Python 3.7 to Travis CI config 2018-09-25 12:17:49 +03:00
Alan Orth
6e47e9c9ee
.travis.yml: Add Python 3.7 2018-09-25 12:17:20 +03:00
Alan Orth
cd90d618d6
CHANGELOG.md: Fix error in old release 2018-09-25 12:17:01 +03:00
Alan Orth
280d211d56
CHANGELOG.md: Add note about kazoo 2.5.0 2018-09-25 12:12:10 +03:00
Alan Orth
806d63137f
requirements.txt: Use kazoo 2.5.0 
SolrClient 0.2.1 currently depends on kazoo 2.2.1, but there is an
issue with Python 3.7 in kazoo <= 2.5.0. Kazoo 2.5.0 fixes the is-
sue with Python 3.7, and for my limited usage of SolrClient it se-
ems to work fine.

See: https://github.com/moonlitesolutions/SolrClient/issues/79
 2018-09-25 12:08:28 +03:00
Alan Orth
f7c7390e4f
README.md: Add note about Python 3.7 2018-09-25 12:07:58 +03:00
Alan Orth
702724e8a4
CHANGELOG.md: Move unreleased changes to version 0.3.0 2018-09-25 11:38:36 +03:00
Alan Orth
36818d03ef
CHANGELOG.md: Update unreleased changes 2018-09-25 11:37:56 +03:00
Alan Orth
4cf8656b35
Change / route to /items 
I think it's more obvious if the "all items" route is plural. Also,
this will allow me to eventually put documentation at the root.
 2018-09-25 11:34:07 +03:00
Alan Orth
f30a464cd1
README.md: Add notes about API endpoints 2018-09-25 11:28:12 +03:00
Alan Orth
93ae12e313
README.md: Update introduction 2018-09-25 11:15:12 +03:00
Alan Orth
dc978e9333
CHANGELOG.md: Add note about requirements.txt and Travis CI 2018-09-25 11:09:02 +03:00
Alan Orth
295436fea0
Add .travis.yml 2018-09-25 11:08:01 +03:00
Alan Orth
46a1476ab0
Add requirements.txt 
Generated with `pip freeze`. This is so I can pin the versions of
packages that I've tested with as well as to allow Travis to test
whether the project runs on various Pythons and to let GitHub in-
form me of vulnerabilities in some libraries.
 2018-09-25 11:02:50 +03:00
Alan Orth
87dbb6c4df
CHANGELOG.md: Release version 0.2.1 2018-09-25 02:21:44 +03:00
Alan Orth
3160c44566
app.py: Remove comment 
This comment was added when I first began the application and the
testing status is documented in the README now.
 2018-09-25 02:20:51 +03:00
Alan Orth
4b72f626d9
Update string substitution format 
Instead of doing numbered strings I will just depend on the order,
at least to be consistent.
 2018-09-25 02:19:29 +03:00
Alan Orth
2d3b7620e3
CHANGELOG.md: Add note about psycopg2.extras.DictCursor 2018-09-25 02:08:54 +03:00
Alan Orth
6e4bc630f7
database.py: Use psycopg2.extras.DictCursor 
This allows us to access records using their column name. I didn't
notice that this was not working, as I had been testing the wrong
server!

See: http://initd.org/psycopg/docs/extras.html
 2018-09-25 02:06:29 +03:00
Alan Orth
44884140e5
CHANGELOG.md: Add new unreleased changes 2018-09-25 01:11:37 +03:00
Alan Orth
74ff86ee3b
contrib: Update environment settings in system units 2018-09-25 01:10:14 +03:00
Alan Orth
3327884f21
Update docs to remove SQLite stuff 
I've decided to use PostgreSQL instead of SQLite because the UPSERT
support is available in versions of PostgreSQL we're alread running,
whereas SQLite needs a VERY new (3.24.0) version that is not avail-
able on any recent long-term support Ubuntu releases.
 2018-09-25 00:56:01 +03:00
Alan Orth
8f7450f67a
Use PostgreSQL instead of SQLite 
I was very surprised how easy and fast and robust SQLite was, but in
the end I realized that its UPSERT support only came in version 3.24
and both Ubuntu 16.04 and 18.04 have older versions than that! I did
manage to install libsqlite3-0 from Ubuntu 18.04 cosmic on my xenial
host, but that feels dirty.

PostgreSQL has support for UPSERT since 9.5, not to mention the same
nice LIMIT and OFFSET clauses.
 2018-09-25 00:49:47 +03:00
Alan Orth
28d61fb041
README.md: Add notes about Python and SQLite versions 2018-09-24 17:26:48 +03:00
Alan Orth
cbc98991b4
CHANGELOG.md: Move unreleased notes to version 0.1.0 2018-09-24 16:14:14 +03:00
Alan Orth
6c28be0463
README.md: Add note about route for all items 2018-09-24 16:13:26 +03:00
Alan Orth
42e8f17305
CHANGELOG.md: Add note about route for all items 2018-09-24 16:13:05 +03:00
Alan Orth
19a45f3f6f
app.py: Add route to page through all item statistics 
This route exposes all item statistics and uses the limit and offset
parameters to control paging throug the result set. The logic here
is extremely easy thanks to the brilliant LIMIT and OFFSET features
of SQLite (of course the SQL query sorts the results by some unique
field to ensure the order is already the same).
 2018-09-24 16:07:26 +03:00
Alan Orth
505ef31101
CHANGELOG.md: Add note about UPSERT 2018-09-24 14:31:05 +03:00
Alan Orth
1543cacc54
app.py: Update SQL logic to use single table 
The indexer.py script was updated to use a single table because I
learned about UPSERT. This simplifies the database schema and the
Python logic, and makes it easier to page all views and downloads
at once without complicated JOIN queries.
 2018-09-24 14:28:00 +03:00
Alan Orth
2cab456f16
indexer.py: Use single items table with UPSERT 
I was using two separate tables for item views and downloads without
realizing that SQLite didn't support FULL OUTER JOIN, which would be
needed to get views and downloads for a given item in a single query.

Instead I can use one table with a default value of 0 for both views
and downloads, and then use "UPSERT" to populate the statistics. This
is a newish SQL concept that allows you to attempt an INSERT and then
specify an action to perform in case of conflict. This works well in
SQLite and actually simplifies my Python logic greatly!

Note that the "excluded" table qualifier is a special keyword that
allows you to reference the value that would have been inserted.

See: https://www.sqlite.org/lang_UPSERT.html
 2018-09-24 14:19:50 +03:00
Alan Orth
53615dea2d
indexer.py: Add license and documentation 2018-09-24 09:18:50 +03:00
Alan Orth
2d8d1e6833
README.md: Add TODO for nonexistent items 2018-09-24 00:48:02 +03:00
Alan Orth
e26e595ea1
README.md: Add more TODOs 2018-09-24 00:35:00 +03:00
Alan Orth
a9151b5bbf
CHANGELOG.md: Update unreleased notes 2018-09-24 00:30:58 +03:00
Alan Orth
76833d6f5f
contrib: Update some old CGSpace references to DSpace 2018-09-24 00:30:26 +03:00
Alan Orth
a51422273c
Remove SOLR_CORE configuration variable 
This parameter is not customizable. All DSpace instances use this
name for the Solr statistics core.
 2018-09-24 00:20:54 +03:00