alanorth/csv-metadata-quality-web
1
0
Fork 0
mirror of https://github.com/ilri/csv-metadata-quality-web.git synced 2024-11-29 18:08:23 +01:00
Code Issues Projects Releases Wiki Activity
68 Commits 1 Branch 1 Tag 608 KiB
73a13145b6
Commit Graph

68 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Alan Orth
73a13145b6
Add runtime.txt 
Apparently to deploy on Heroku we need this. And they only support
Python 3.7? Damn...
 2021-03-14 12:24:03 +02:00
Alan Orth
8f6d337d2d
Move style.min.css to css/v1/style.min.css 
Trying to break Google App Engine's aggressive caching.
 2021-03-14 12:04:28 +02:00
Alan Orth
aabb783d99
app.yaml: Set Cache-Control header to private for CSS 
Google App Engine agressively caches stuff. They are currently serving
a 24-hour old version of my CSS after multiple updates and re-deploys.
Ughhh. From their docs:

> After a file is transmitted with a given expiration time, there is
> generally no way to clear it out of web-proxy caches, even if the user
> clears their own browser cache. Re-deploying a new version of the app
> will not reset any caches. Therefore, if you ever plan to modify a
> static file, it should have a short (less than one hour) expiration
> time. In most cases, the default 10-minute expiration time is
> appropriate.

The only way to break this for now is to change the CSS *directory*.
In the future I think we have to be sure to set the private cache
control header, which lets browsers cache it, but not public CDNs.

See: https://cloud.google.com/appengine/docs/standard/python3/how-requests-are-handled
 2021-03-14 11:58:57 +02:00
Alan Orth
bd31ac912e
Regenerate static assets 2021-03-14 11:42:07 +02:00
Alan Orth
61040ea4a5
Use an ILRI theme 2021-03-14 11:41:48 +02:00
Alan Orth
d483f7fc0b
.gitignore: Ignore sqlite requests response cache 2021-03-14 11:36:46 +02:00
Alan Orth
c07716abcb
.gcloudignore: Ignore sqlite requests response cache 2021-03-14 11:36:09 +02:00
Alan Orth
36b43a06b9
templates/index.html: Add link to test.csv 
Offer the user to test with this file.
 2021-03-14 11:34:55 +02:00
Alan Orth
25aa74ba14
app.yaml: Try to serve static CSS directly 
Google App Engine is currently caching an old version of my CSS, so
I am trying to get it to use the correct version. Let's try serving
it directly from the filesystem.

See: https://cloud.google.com/appengine/docs/standard/php/getting-started/serving-static-files
 2021-03-14 10:36:57 +02:00
Alan Orth
a2ace2d331
.gcloudignore: Don't upload source dir 
We only need this during build.
 2021-03-14 10:31:46 +02:00
Alan Orth
94cdd7fb50
Regenerate requirements 
Generated using poetry:

  $ poetry export --without-hashes -f requirements.txt > requirements.txt
  $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt
 2021-03-14 10:09:18 +02:00
Alan Orth
74bde23567
poetry.lock: Run poetry update 2021-03-14 10:08:44 +02:00
Alan Orth
4e52d1bcc9
Add configurable requests cache directory 
As I expected, on Google App Engine we can't write the cache file
to the current working directory. I modified csv-metadata-quality
CLI to check for the REQUESTS_CACHE_DIR environment variable so we
don't really have to do anything different other than setting the
variable.
 2021-03-14 10:06:39 +02:00
Alan Orth
e7dd8d1421
Add AGROVOC lookup support 
This works locally, but I don't think it will work on App Engine
because csv-metadata-quality uses requests-cache and creates the
agrovoc-response-cache.sqlite file in the current working directory.
 2021-03-13 23:49:24 +02:00
Alan Orth
9aab2ae83f
templates/index.html: Remove basic and experimental labels 2021-03-13 23:34:34 +02:00
Alan Orth
6c3804d55b
Add support for skipping fields ("-x") 2021-03-13 23:34:11 +02:00
Alan Orth
122d9fd53c
Add support for experimental checks ("-e") 2021-03-13 23:01:11 +02:00
Alan Orth
198acdb1a7
Major refactor 
Re-work upload and file processing so they are in the same Python
function. Now I will start exposing other command line options in
the form, like unsafe fixes, excluding fields, etc. Now I see tha
t it is easier to save the POSTed file and process it in the same
function so I don't have to pass around the other POSTed form val
ues as URL query parameters.

Now, as a result of changing the flow above, I also had to make a
change to the way I show the results page. Instead of processing
the file and returning the rendered results to the user directly,
I process the file, save the rendered results to /tmp, and return
a redirect to the user to the results page.
 2021-03-13 22:11:26 +02:00
Alan Orth
adc2d06094
main.py: Add note about using /tmp 
I originally wanted to use an "uploads" directory or something, but
it seems we can only write to /tmp on Google App Engine. They really
want you to buy storage or database services! This is memory mapped
so it disappears when you re-deploy.
 2021-03-13 22:09:17 +02:00
Alan Orth
bc256b242d
static/css/style.min.css: Regenerate 2021-03-13 22:07:53 +02:00
Alan Orth
f82cb6ce05
source/scss/main.scss: Increase container width 
We need more space for the log on the results page and we actually
don't even need to worry about people running this on a phone.
 2021-03-13 22:06:43 +02:00
Alan Orth
4bdec3b889
app.yaml: Use Python 3.9 
Python 3.9 is apparently now generally available.
 2021-03-13 14:12:14 +02:00
Alan Orth
f79be86361
templates/index.html: Use Bootstrap form components 2021-03-13 13:50:46 +02:00
Alan Orth
3715c5e976
Use new commit for csv-metadata-quality 
This one doesn't treat the fixing of invalid multi-value separators
as "unsafe".
 2021-03-13 13:02:47 +02:00
Alan Orth
8603ec4bca
main.py: Rework command line args 
Turns out we only need to use sys.argv when we were trying to run
the csv-metadata-quality module directly in Python by importing it.
 2021-03-13 12:52:22 +02:00
Alan Orth
0471820f3a
main.py: Actually use sys.argv 
I set this but never actually passed it to the subprocess. Now I'm
wondering if I actually need it, or if that was just when I was tr-
ying to import the csv-metadata-quality module?
 2021-03-13 12:49:01 +02:00
Alan Orth
f1de00d7ca
Regenerate requirements 
Generated using poetry:

  $ poetry export --without-hashes -f requirements.txt > requirements.txt
  $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt
 2021-03-12 23:11:33 +02:00
Alan Orth
31e699bb8b
poetry.lock: Sync lockfile 2021-03-12 23:10:55 +02:00
Alan Orth
982b47459c
Use gunicorn to run on Google App Engine 
See: https://cloud.google.com/appengine/docs/flexible/python/quickstart
 2021-03-12 23:10:29 +02:00
Alan Orth
38ace23f97
.gitignore: Add __pycache__ 2021-03-12 23:09:29 +02:00
Alan Orth
e5f57802f3
Don't ignore uploads directory 
We're using /tmp now since we apparently can't create and upload to
arbitrary directories on Google App Engine (?).
 2021-03-12 23:08:28 +02:00
Alan Orth
69abc3d764
Add downloading of cleaned CSV file 2021-03-12 23:00:48 +02:00
Alan Orth
37cee1d3f0
Add configuration for isort 
See: https://sourcery.ai/blog/python-best-practices/
 2021-03-12 22:59:57 +02:00
Alan Orth
126bf74ea0
main.py: Remove comments 2021-03-12 21:28:24 +02:00
Alan Orth
54ea340b80
poetry.lock: Run poetry update 2021-03-12 21:24:51 +02:00
Alan Orth
75c5e48aa3
templates/header.html: Link to home in navbar 2021-03-12 21:24:31 +02:00
Alan Orth
4de4388b1e
Encode filenames to base64 and change URL to /result 2021-03-12 21:22:03 +02:00
Alan Orth
032fa5f2e7
pyproject.toml: Use default csv-metadata-quality release 
Now that we are just running the command line version via a Python
subprocess we don't have to do anything special.
 2021-03-12 21:18:50 +02:00
Alan Orth
633fb8419a
Regenerate requirements.txt 
Generated using poetry:

  $ poetry export --without-hashes -f requirements.txt > requirements.txt
  $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt
 2021-03-12 19:43:06 +02:00
Alan Orth
c1d79c147c
poetry.lock: Update lockfile 2021-03-12 19:41:39 +02:00
Alan Orth
477b5bcd42
Regenerate static assets 2021-03-12 19:40:49 +02:00
Alan Orth
e13d63bf6b
Major refactor 
Refactor the templates to include a header, use csv-metadata-quality
in a subshell instead of trying to import and pass args to it (which
I actually managed to do, but then trying to capture the output was
an issue), and use ansi2html to prepare the CLI output as the HTML.
 2021-03-12 19:14:49 +02:00
Alan Orth
ed747b2cef
templates/footer.html: Add link to csv-metadata-quality on GitHub 2021-03-12 19:07:29 +02:00
Alan Orth
1f2589f35c
templates/footer.html: Retab 2021-03-12 19:06:52 +02:00
Alan Orth
f1060e79b8
pyproject.toml: Use the web branch of csv-metadata-quality 
I'm still trying to figure out what modifications we need to make
to the standalone command line tool to use it here from Flask.
 2021-03-12 19:03:51 +02:00
Alan Orth
1a92b99c68
templates: Reduce top margin 2021-03-11 22:54:46 +02:00
Alan Orth
18643ca1ae
Regenerate requirements.txt 
Generated using poetry:

  $ poetry export --without-hashes -f requirements.txt > requirements.txt
 2021-03-11 22:49:52 +02:00
Alan Orth
0eeea43f17
Use csv-metadata-quality 0.4.6-dev 2021-03-11 22:49:01 +02:00
Alan Orth
0136d99046
Add route for processing the input file 2021-03-11 22:42:59 +02:00
Alan Orth
207d5af64a
Split index.html template into header, index, and footer 2021-03-11 22:41:22 +02:00