This was inserting correctly on the first run, but subsequent runs
were inserting into the incorrect column on conflict. This made it
seem like there were downloads for items where there were none.
We don't need to create an intermediate variable for the results of
the SQL query because psycopg2's cursor is iterable.
See: http://initd.org/psycopg/docs/cursor.html
Batch inserts are much faster than a series of individual inserts
because they drastically reduce the overhead caused by round-trip
communication with the server. My tests in development confirm:
- cursor.execute(): 19 seconds
- execute_values(): 14 seconds
I'm currently only working with 4,500 rows, but I will experiment
with larger data sets, as well as larger batches. For example, on
the PostgreSQL mailing list a user reports doing 10,000 rows with
a page size of 100.
See: http://initd.org/psycopg/docs/extras.html#psycopg2.extras.execute_values
See: https://github.com/psycopg/psycopg2/issues/491#issuecomment-276551038
Basically Solr's numFound has nothing to do with the actual number
of distinct facets that are returned. You need to use Solr's stats
component to get the number of distinct facets, aka countDistinct.
This is apparently deprecated in newer Solr versions, but we're on
version 4.10 and it works there.
Also, I realized that there is no need to return facets for items
without any views or downloads. Using facet.mincount=1 reduces the
result set size and also means we can store less data in the data-
base. The API returns HTTP 404 Not Found if an item is not in the
database anyways.
I can't figure it out exactly, but there is some weird issue with
Solr's facet results when you don't use facet.mincount=1. For some
reason you get tons of results with an id that doesn't even exist
in the document database, let alone as an actual DSpace item!
See: https://lucene.apache.org/solr/guide/6_6/the-stats-component.html
You can test OnCalendar strings using systemd-analyze calendar, eg:
# systemd-analyze calendar '*-*-* 06:00:00,18:00:00'
Failed to parse calendar specification '*-*-* 06:00:00,18:00:00':
Invalid argument
# systemd-analyze calendar '*-*-* 06,18:00:00'
Normalized form: *-*-* 06,18:00:00
Next elapse: Wed 2018-09-26 06:00:00 EEST
(in UTC): Wed 2018-09-26 03:00:00 UTC
From now: 6h left
SolrClient 0.2.1 currently depends on kazoo 2.2.1, but there is an
issue with Python 3.7 in kazoo <= 2.5.0. Kazoo 2.5.0 fixes the is-
sue with Python 3.7, and for my limited usage of SolrClient it se-
ems to work fine.
See: https://github.com/moonlitesolutions/SolrClient/issues/79
Generated with `pip freeze`. This is so I can pin the versions of
packages that I've tested with as well as to allow Travis to test
whether the project runs on various Pythons and to let GitHub in-
form me of vulnerabilities in some libraries.
Alan Orth