Switch to a personal fork of SolrClient so that we can use kazoo 2.5.0
and get rid of the error about the 'async' keyword on Python 3.7. Also
this bumps some of the other libraries to their latest versions.
Falcon can optionally use ujson to speed up JSON (de)serialization,
but Falcon's already really fast and requiring ujson actually makes
deployment trickier in some cases (for example in Docker containers
that are based on Alpine Linux).
Here are some tests of Falcon 1.4.1 on Python 3.5 from my laptop:
1. falcon...............60172 req/sec or 16.62 μs/req (36x)
2. falcon-ext...........34186 req/sec or 29.25 μs/req (20x)
3. bottle...............32924 req/sec or 30.37 μs/req (20x)
4. werkzeug.............11948 req/sec or 83.70 μs/req (7x)
5. flask.................6654 req/sec or 150.30 μs/req (4x)
6. django................4565 req/sec or 219.04 μs/req (3x)
7. pecan.................1672 req/sec or 598.19 μs/req (1x)
The tests were conducted with Falcon's official Docker benchmarking
tools on my Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz on Arch Linux.
See: https://github.com/falconry/falcon/tree/master/docker
This was inserting correctly on the first run, but subsequent runs
were inserting into the incorrect column on conflict. This made it
seem like there were downloads for items where there were none.
We don't need to create an intermediate variable for the results of
the SQL query because psycopg2's cursor is iterable.
See: http://initd.org/psycopg/docs/cursor.html
Batch inserts are much faster than a series of individual inserts
because they drastically reduce the overhead caused by round-trip
communication with the server. My tests in development confirm:
- cursor.execute(): 19 seconds
- execute_values(): 14 seconds
I'm currently only working with 4,500 rows, but I will experiment
with larger data sets, as well as larger batches. For example, on
the PostgreSQL mailing list a user reports doing 10,000 rows with
a page size of 100.
See: http://initd.org/psycopg/docs/extras.html#psycopg2.extras.execute_values
See: https://github.com/psycopg/psycopg2/issues/491#issuecomment-276551038
Alan Orth