alanorth/csv-metadata-quality-web
1
0
Fork 0
mirror of https://github.com/ilri/csv-metadata-quality-web.git synced 2025-05-07 13:46:01 +02:00
Code Issues Projects Releases Wiki Activity
61 Commits 1 Branch 1 Tag
36b43a06b9353f683f491b115703f55e8136c38b
Commit Graph

61 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Alan Orth
36b43a06b9 templates/index.html: Add link to test.csv 
Offer the user to test with this file.
 2021-03-14 11:34:55 +02:00
Alan Orth
25aa74ba14 app.yaml: Try to serve static CSS directly 
Google App Engine is currently caching an old version of my CSS, so
I am trying to get it to use the correct version. Let's try serving
it directly from the filesystem.

See: https://cloud.google.com/appengine/docs/standard/php/getting-started/serving-static-files
 2021-03-14 10:36:57 +02:00
Alan Orth
a2ace2d331 .gcloudignore: Don't upload source dir 
We only need this during build.
 2021-03-14 10:31:46 +02:00
Alan Orth
94cdd7fb50 Regenerate requirements 
Generated using poetry:

  $ poetry export --without-hashes -f requirements.txt > requirements.txt
  $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt
 2021-03-14 10:09:18 +02:00
Alan Orth
74bde23567 poetry.lock: Run poetry update 2021-03-14 10:08:44 +02:00
Alan Orth
4e52d1bcc9 Add configurable requests cache directory 
As I expected, on Google App Engine we can't write the cache file
to the current working directory. I modified csv-metadata-quality
CLI to check for the REQUESTS_CACHE_DIR environment variable so we
don't really have to do anything different other than setting the
variable.
 2021-03-14 10:06:39 +02:00
Alan Orth
e7dd8d1421 Add AGROVOC lookup support 
This works locally, but I don't think it will work on App Engine
because csv-metadata-quality uses requests-cache and creates the
agrovoc-response-cache.sqlite file in the current working directory.
 2021-03-13 23:49:24 +02:00
Alan Orth
9aab2ae83f templates/index.html: Remove basic and experimental labels 2021-03-13 23:34:34 +02:00
Alan Orth
6c3804d55b Add support for skipping fields ("-x") 2021-03-13 23:34:11 +02:00
Alan Orth
122d9fd53c Add support for experimental checks ("-e") 2021-03-13 23:01:11 +02:00
Alan Orth
198acdb1a7 Major refactor 
Re-work upload and file processing so they are in the same Python
function. Now I will start exposing other command line options in
the form, like unsafe fixes, excluding fields, etc. Now I see tha
t it is easier to save the POSTed file and process it in the same
function so I don't have to pass around the other POSTed form val
ues as URL query parameters.

Now, as a result of changing the flow above, I also had to make a
change to the way I show the results page. Instead of processing
the file and returning the rendered results to the user directly,
I process the file, save the rendered results to /tmp, and return
a redirect to the user to the results page.
 2021-03-13 22:11:26 +02:00
Alan Orth
adc2d06094 main.py: Add note about using /tmp 
I originally wanted to use an "uploads" directory or something, but
it seems we can only write to /tmp on Google App Engine. They really
want you to buy storage or database services! This is memory mapped
so it disappears when you re-deploy.
 2021-03-13 22:09:17 +02:00
Alan Orth
bc256b242d static/css/style.min.css: Regenerate 2021-03-13 22:07:53 +02:00
Alan Orth
f82cb6ce05 source/scss/main.scss: Increase container width 
We need more space for the log on the results page and we actually
don't even need to worry about people running this on a phone.
 2021-03-13 22:06:43 +02:00
Alan Orth
4bdec3b889 app.yaml: Use Python 3.9 
Python 3.9 is apparently now generally available.
 2021-03-13 14:12:14 +02:00
Alan Orth
f79be86361 templates/index.html: Use Bootstrap form components 2021-03-13 13:50:46 +02:00
Alan Orth
3715c5e976 Use new commit for csv-metadata-quality 
This one doesn't treat the fixing of invalid multi-value separators
as "unsafe".
 2021-03-13 13:02:47 +02:00
Alan Orth
8603ec4bca main.py: Rework command line args 
Turns out we only need to use sys.argv when we were trying to run
the csv-metadata-quality module directly in Python by importing it.
 2021-03-13 12:52:22 +02:00
Alan Orth
0471820f3a main.py: Actually use sys.argv 
I set this but never actually passed it to the subprocess. Now I'm
wondering if I actually need it, or if that was just when I was tr-
ying to import the csv-metadata-quality module?
 2021-03-13 12:49:01 +02:00
Alan Orth
f1de00d7ca Regenerate requirements 
Generated using poetry:

  $ poetry export --without-hashes -f requirements.txt > requirements.txt
  $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt
 2021-03-12 23:11:33 +02:00
Alan Orth
31e699bb8b poetry.lock: Sync lockfile 2021-03-12 23:10:55 +02:00
Alan Orth
982b47459c Use gunicorn to run on Google App Engine 
See: https://cloud.google.com/appengine/docs/flexible/python/quickstart
 2021-03-12 23:10:29 +02:00
Alan Orth
38ace23f97 .gitignore: Add __pycache__ 2021-03-12 23:09:29 +02:00
Alan Orth
e5f57802f3 Don't ignore uploads directory 
We're using /tmp now since we apparently can't create and upload to
arbitrary directories on Google App Engine (?).
 2021-03-12 23:08:28 +02:00
Alan Orth
69abc3d764 Add downloading of cleaned CSV file 2021-03-12 23:00:48 +02:00
Alan Orth
37cee1d3f0 Add configuration for isort 
See: https://sourcery.ai/blog/python-best-practices/
 2021-03-12 22:59:57 +02:00
Alan Orth
126bf74ea0 main.py: Remove comments 2021-03-12 21:28:24 +02:00
Alan Orth
54ea340b80 poetry.lock: Run poetry update 2021-03-12 21:24:51 +02:00
Alan Orth
75c5e48aa3 templates/header.html: Link to home in navbar 2021-03-12 21:24:31 +02:00
Alan Orth
4de4388b1e Encode filenames to base64 and change URL to /result 2021-03-12 21:22:03 +02:00
Alan Orth
032fa5f2e7 pyproject.toml: Use default csv-metadata-quality release 
Now that we are just running the command line version via a Python
subprocess we don't have to do anything special.
 2021-03-12 21:18:50 +02:00
Alan Orth
633fb8419a Regenerate requirements.txt 
Generated using poetry:

  $ poetry export --without-hashes -f requirements.txt > requirements.txt
  $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt
 2021-03-12 19:43:06 +02:00
Alan Orth
c1d79c147c poetry.lock: Update lockfile 2021-03-12 19:41:39 +02:00
Alan Orth
477b5bcd42 Regenerate static assets 2021-03-12 19:40:49 +02:00
Alan Orth
e13d63bf6b Major refactor 
Refactor the templates to include a header, use csv-metadata-quality
in a subshell instead of trying to import and pass args to it (which
I actually managed to do, but then trying to capture the output was
an issue), and use ansi2html to prepare the CLI output as the HTML.
 2021-03-12 19:14:49 +02:00
Alan Orth
ed747b2cef templates/footer.html: Add link to csv-metadata-quality on GitHub 2021-03-12 19:07:29 +02:00
Alan Orth
1f2589f35c templates/footer.html: Retab 2021-03-12 19:06:52 +02:00
Alan Orth
f1060e79b8 pyproject.toml: Use the web branch of csv-metadata-quality 
I'm still trying to figure out what modifications we need to make
to the standalone command line tool to use it here from Flask.
 2021-03-12 19:03:51 +02:00
Alan Orth
1a92b99c68 templates: Reduce top margin 2021-03-11 22:54:46 +02:00
Alan Orth
18643ca1ae Regenerate requirements.txt 
Generated using poetry:

  $ poetry export --without-hashes -f requirements.txt > requirements.txt
 2021-03-11 22:49:52 +02:00
Alan Orth
0eeea43f17 Use csv-metadata-quality 0.4.6-dev 2021-03-11 22:49:01 +02:00
Alan Orth
0136d99046 Add route for processing the input file 2021-03-11 22:42:59 +02:00
Alan Orth
207d5af64a Split index.html template into header, index, and footer 2021-03-11 22:41:22 +02:00
Alan Orth
08b83caeb4 .gcloudignore: Ignore some local files 2021-03-11 21:15:39 +02:00
Alan Orth
4f1d6fad38 main.py: Import csv_metadata_quality 
Now I understand how I can use this, for example to print the vers-
ion of the CLI tool in the HTML template.
 2021-03-11 20:29:43 +02:00
Alan Orth
ee54096561 templates/index.html: Improve layout 
Based on the Bootstrap v5.0-beta2 sticky footer example.

See: https://getbootstrap.com/docs/5.0/examples/sticky-footer/
 2021-03-11 20:28:42 +02:00
Alan Orth
7e14635b99 .gitignore: Ignore node_modules 2021-03-11 20:27:32 +02:00
Alan Orth
14d592d0ea Build Bootstrap v5.0-beta2 CSS 2021-03-11 20:26:33 +02:00
Alan Orth
0a8096788d templates/index.html: More boilerplate 2021-03-11 17:59:51 +02:00
Alan Orth
5086ea2376 templates/index.html: Use url_for to get CSS 
This is apparently the way you do it in Flask.
 2021-03-11 17:55:19 +02:00