236 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Alan Orth	560fa6056d	README.md: Remove batch inserts from TODO	2018-09-26 23:25:35 +03:00
Alan Orth	385a34e5d0	indexer.py: Use psycopg2's execute_values to batch inserts ... Batch inserts are much faster than a series of individual inserts because they drastically reduce the overhead caused by round-trip communication with the server. My tests in development confirm: - cursor.execute(): 19 seconds - execute_values(): 14 seconds I'm currently only working with 4,500 rows, but I will experiment with larger data sets, as well as larger batches. For example, on the PostgreSQL mailing list a user reports doing 10,000 rows with a page size of 100. See: http://initd.org/psycopg/docs/extras.html#psycopg2.extras.execute_values See: https://github.com/psycopg/psycopg2/issues/491#issuecomment-276551038	2018-09-26 23:10:29 +03:00
Alan Orth	d0ea62d2bd	database.py: Use one line for psycopg2 imports	2018-09-26 22:23:24 +03:00
Alan Orth	366ae25b8e	README.md: Add link to psycopg2 issue about batch inserts	2018-09-26 22:23:08 +03:00
Alan Orth	0f3054ae03	README.md: Add TODO about batch DB inserts	2018-09-26 16:31:13 +03:00
Alan Orth	6bf34235d4	CHANGELOG.md: Move unreleased changes to version 0.4.0	2018-09-26 02:51:27 +03:00
Alan Orth	e604d8ca81	indexer.py: Major refactor ... Basically Solr's numFound has nothing to do with the actual number of distinct facets that are returned. You need to use Solr's stats component to get the number of distinct facets, aka countDistinct. This is apparently deprecated in newer Solr versions, but we're on version 4.10 and it works there. Also, I realized that there is no need to return facets for items without any views or downloads. Using facet.mincount=1 reduces the result set size and also means we can store less data in the data- base. The API returns HTTP 404 Not Found if an item is not in the database anyways. I can't figure it out exactly, but there is some weird issue with Solr's facet results when you don't use facet.mincount=1. For some reason you get tons of results with an id that doesn't even exist in the document database, let alone as an actual DSpace item! See: https://lucene.apache.org/solr/guide/6_6/the-stats-component.html	2018-09-26 02:41:10 +03:00
Alan Orth	fc35b816f3	CHANGELOG.md: Add unreleased changes	2018-09-25 23:09:44 +03:00
Alan Orth	9e6a2f7559	contrib/dspace-statistics-indexer.timer: Fix syntax ... You can test OnCalendar strings using systemd-analyze calendar, eg: # systemd-analyze calendar '*-*-* 06:00:00,18:00:00' Failed to parse calendar specification '*-*-* 06:00:00,18:00:00': Invalid argument # systemd-analyze calendar '*-*-* 06,18:00:00' Normalized form: *-*-* 06,18:00:00 Next elapse: Wed 2018-09-26 06:00:00 EEST (in UTC): Wed 2018-09-26 03:00:00 UTC From now: 6h left	2018-09-25 23:07:03 +03:00
Alan Orth	46cfc3ffbc	CHANGELOG.md: Release version 0.3.2	2018-09-25 13:14:08 +03:00
Alan Orth	2850035a4c	Return HTTP 404 when an item id is not found	2018-09-25 13:12:53 +03:00
Alan Orth	c0b550109a	README.md: Improve wording	2018-09-25 12:24:52 +03:00
Alan Orth	bfceffd84d	indexer.py: Improve inline documentation	2018-09-25 12:23:31 +03:00
Alan Orth	d0552f5047	CHANGELOG.md: Move unreleased changes to version 0.3.1	2018-09-25 12:18:26 +03:00
Alan Orth	c3a0bf7f44	CHANGELOG.md: Add Python 3.7 to Travis CI config	2018-09-25 12:17:49 +03:00
Alan Orth	6e47e9c9ee	.travis.yml: Add Python 3.7	2018-09-25 12:17:20 +03:00
Alan Orth	cd90d618d6	CHANGELOG.md: Fix error in old release	2018-09-25 12:17:01 +03:00
Alan Orth	280d211d56	CHANGELOG.md: Add note about kazoo 2.5.0	2018-09-25 12:12:10 +03:00
Alan Orth	806d63137f	requirements.txt: Use kazoo 2.5.0 ... SolrClient 0.2.1 currently depends on kazoo 2.2.1, but there is an issue with Python 3.7 in kazoo <= 2.5.0. Kazoo 2.5.0 fixes the is- sue with Python 3.7, and for my limited usage of SolrClient it se- ems to work fine. See: https://github.com/moonlitesolutions/SolrClient/issues/79	2018-09-25 12:08:28 +03:00
Alan Orth	f7c7390e4f	README.md: Add note about Python 3.7	2018-09-25 12:07:58 +03:00
Alan Orth	702724e8a4	CHANGELOG.md: Move unreleased changes to version 0.3.0	2018-09-25 11:38:36 +03:00
Alan Orth	36818d03ef	CHANGELOG.md: Update unreleased changes	2018-09-25 11:37:56 +03:00
Alan Orth	4cf8656b35	Change / route to /items ... I think it's more obvious if the "all items" route is plural. Also, this will allow me to eventually put documentation at the root.	2018-09-25 11:34:07 +03:00
Alan Orth	f30a464cd1	README.md: Add notes about API endpoints	2018-09-25 11:28:12 +03:00
Alan Orth	93ae12e313	README.md: Update introduction	2018-09-25 11:15:12 +03:00
Alan Orth	dc978e9333	CHANGELOG.md: Add note about requirements.txt and Travis CI	2018-09-25 11:09:02 +03:00
Alan Orth	295436fea0	Add .travis.yml	2018-09-25 11:08:01 +03:00
Alan Orth	46a1476ab0	Add requirements.txt ... Generated with `pip freeze`. This is so I can pin the versions of packages that I've tested with as well as to allow Travis to test whether the project runs on various Pythons and to let GitHub in- form me of vulnerabilities in some libraries.	2018-09-25 11:02:50 +03:00
Alan Orth	87dbb6c4df	CHANGELOG.md: Release version 0.2.1	2018-09-25 02:21:44 +03:00
Alan Orth	3160c44566	app.py: Remove comment ... This comment was added when I first began the application and the testing status is documented in the README now.	2018-09-25 02:20:51 +03:00
Alan Orth	4b72f626d9	Update string substitution format ... Instead of doing numbered strings I will just depend on the order, at least to be consistent.	2018-09-25 02:19:29 +03:00
Alan Orth	2d3b7620e3	CHANGELOG.md: Add note about psycopg2.extras.DictCursor	2018-09-25 02:08:54 +03:00
Alan Orth	6e4bc630f7	database.py: Use psycopg2.extras.DictCursor ... This allows us to access records using their column name. I didn't notice that this was not working, as I had been testing the wrong server! See: http://initd.org/psycopg/docs/extras.html	2018-09-25 02:06:29 +03:00
Alan Orth	44884140e5	CHANGELOG.md: Add new unreleased changes	2018-09-25 01:11:37 +03:00
Alan Orth	74ff86ee3b	contrib: Update environment settings in system units	2018-09-25 01:10:14 +03:00
Alan Orth	3327884f21	Update docs to remove SQLite stuff ... I've decided to use PostgreSQL instead of SQLite because the UPSERT support is available in versions of PostgreSQL we're alread running, whereas SQLite needs a VERY new (3.24.0) version that is not avail- able on any recent long-term support Ubuntu releases.	2018-09-25 00:56:01 +03:00
Alan Orth	8f7450f67a	Use PostgreSQL instead of SQLite ... I was very surprised how easy and fast and robust SQLite was, but in the end I realized that its UPSERT support only came in version 3.24 and both Ubuntu 16.04 and 18.04 have older versions than that! I did manage to install libsqlite3-0 from Ubuntu 18.04 cosmic on my xenial host, but that feels dirty. PostgreSQL has support for UPSERT since 9.5, not to mention the same nice LIMIT and OFFSET clauses.	2018-09-25 00:49:47 +03:00
Alan Orth	28d61fb041	README.md: Add notes about Python and SQLite versions	2018-09-24 17:26:48 +03:00
Alan Orth	cbc98991b4	CHANGELOG.md: Move unreleased notes to version 0.1.0	2018-09-24 16:14:14 +03:00
Alan Orth	6c28be0463	README.md: Add note about route for all items	2018-09-24 16:13:26 +03:00
Alan Orth	42e8f17305	CHANGELOG.md: Add note about route for all items	2018-09-24 16:13:05 +03:00
Alan Orth	19a45f3f6f	app.py: Add route to page through all item statistics ... This route exposes all item statistics and uses the limit and offset parameters to control paging throug the result set. The logic here is extremely easy thanks to the brilliant LIMIT and OFFSET features of SQLite (of course the SQL query sorts the results by some unique field to ensure the order is already the same).	2018-09-24 16:07:26 +03:00
Alan Orth	505ef31101	CHANGELOG.md: Add note about UPSERT	2018-09-24 14:31:05 +03:00
Alan Orth	1543cacc54	app.py: Update SQL logic to use single table ... The indexer.py script was updated to use a single table because I learned about UPSERT. This simplifies the database schema and the Python logic, and makes it easier to page all views and downloads at once without complicated JOIN queries.	2018-09-24 14:28:00 +03:00
Alan Orth	2cab456f16	indexer.py: Use single items table with UPSERT ... I was using two separate tables for item views and downloads without realizing that SQLite didn't support FULL OUTER JOIN, which would be needed to get views and downloads for a given item in a single query. Instead I can use one table with a default value of 0 for both views and downloads, and then use "UPSERT" to populate the statistics. This is a newish SQL concept that allows you to attempt an INSERT and then specify an action to perform in case of conflict. This works well in SQLite and actually simplifies my Python logic greatly! Note that the "excluded" table qualifier is a special keyword that allows you to reference the value that would have been inserted. See: https://www.sqlite.org/lang_UPSERT.html	2018-09-24 14:19:50 +03:00
Alan Orth	53615dea2d	indexer.py: Add license and documentation	2018-09-24 09:18:50 +03:00
Alan Orth	2d8d1e6833	README.md: Add TODO for nonexistent items	2018-09-24 00:48:02 +03:00
Alan Orth	e26e595ea1	README.md: Add more TODOs	2018-09-24 00:35:00 +03:00
Alan Orth	a9151b5bbf	CHANGELOG.md: Update unreleased notes	2018-09-24 00:30:58 +03:00
Alan Orth	76833d6f5f	contrib: Update some old CGSpace references to DSpace	2018-09-24 00:30:26 +03:00