75 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Alan Orth	e8dc08bcac	README.md: Update text	2021-03-14 13:54:03 +02:00
Alan Orth	f4ddf4a7b5	README.md: Update intro text	2021-03-14 13:11:39 +02:00
Alan Orth	d37654206f	README.md: Fix screenshot link	2021-03-14 13:08:50 +02:00
Alan Orth	69501cbacb	Add README.md with screenshot and license	2021-03-14 13:07:24 +02:00
Alan Orth	2a9ec1c3f3	LICENSE.txt: Use GPLv3 instead of AGPLv3 ... I would rather have the source publishing requirements be triggered on distribution than on web hosting.	2021-03-14 13:05:32 +02:00
Alan Orth	be9143204c	Add Procfile for Heroku ... See: https://devcenter.heroku.com/articles/python-gunicorn	2021-03-14 12:29:28 +02:00
Alan Orth	55815cf4c0	runtime.txt: Use Python 3.9.2 ... Actually it seems they do have Python 3.9.2. See: https://devcenter.heroku.com/articles/python-support	2021-03-14 12:25:49 +02:00
Alan Orth	73a13145b6	Add runtime.txt ... Apparently to deploy on Heroku we need this. And they only support Python 3.7? Damn...	2021-03-14 12:24:03 +02:00
Alan Orth	8f6d337d2d	Move style.min.css to css/v1/style.min.css ... Trying to break Google App Engine's aggressive caching.	2021-03-14 12:04:28 +02:00
Alan Orth	aabb783d99	app.yaml: Set Cache-Control header to private for CSS ... Google App Engine agressively caches stuff. They are currently serving a 24-hour old version of my CSS after multiple updates and re-deploys. Ughhh. From their docs: > After a file is transmitted with a given expiration time, there is > generally no way to clear it out of web-proxy caches, even if the user > clears their own browser cache. Re-deploying a new version of the app > will not reset any caches. Therefore, if you ever plan to modify a > static file, it should have a short (less than one hour) expiration > time. In most cases, the default 10-minute expiration time is > appropriate. The only way to break this for now is to change the CSS *directory*. In the future I think we have to be sure to set the private cache control header, which lets browsers cache it, but not public CDNs. See: https://cloud.google.com/appengine/docs/standard/python3/how-requests-are-handled	2021-03-14 11:58:57 +02:00
Alan Orth	bd31ac912e	Regenerate static assets	2021-03-14 11:42:07 +02:00
Alan Orth	61040ea4a5	Use an ILRI theme	2021-03-14 11:41:48 +02:00
Alan Orth	d483f7fc0b	.gitignore: Ignore sqlite requests response cache	2021-03-14 11:36:46 +02:00
Alan Orth	c07716abcb	.gcloudignore: Ignore sqlite requests response cache	2021-03-14 11:36:09 +02:00
Alan Orth	36b43a06b9	templates/index.html: Add link to test.csv ... Offer the user to test with this file.	2021-03-14 11:34:55 +02:00
Alan Orth	25aa74ba14	app.yaml: Try to serve static CSS directly ... Google App Engine is currently caching an old version of my CSS, so I am trying to get it to use the correct version. Let's try serving it directly from the filesystem. See: https://cloud.google.com/appengine/docs/standard/php/getting-started/serving-static-files	2021-03-14 10:36:57 +02:00
Alan Orth	a2ace2d331	.gcloudignore: Don't upload source dir ... We only need this during build.	2021-03-14 10:31:46 +02:00
Alan Orth	94cdd7fb50	Regenerate requirements ... Generated using poetry: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt	2021-03-14 10:09:18 +02:00
Alan Orth	74bde23567	poetry.lock: Run poetry update	2021-03-14 10:08:44 +02:00
Alan Orth	4e52d1bcc9	Add configurable requests cache directory ... As I expected, on Google App Engine we can't write the cache file to the current working directory. I modified csv-metadata-quality CLI to check for the REQUESTS_CACHE_DIR environment variable so we don't really have to do anything different other than setting the variable.	2021-03-14 10:06:39 +02:00
Alan Orth	e7dd8d1421	Add AGROVOC lookup support ... This works locally, but I don't think it will work on App Engine because csv-metadata-quality uses requests-cache and creates the agrovoc-response-cache.sqlite file in the current working directory.	2021-03-13 23:49:24 +02:00
Alan Orth	9aab2ae83f	templates/index.html: Remove basic and experimental labels	2021-03-13 23:34:34 +02:00
Alan Orth	6c3804d55b	Add support for skipping fields ("-x")	2021-03-13 23:34:11 +02:00
Alan Orth	122d9fd53c	Add support for experimental checks ("-e")	2021-03-13 23:01:11 +02:00
Alan Orth	198acdb1a7	Major refactor ... Re-work upload and file processing so they are in the same Python function. Now I will start exposing other command line options in the form, like unsafe fixes, excluding fields, etc. Now I see tha t it is easier to save the POSTed file and process it in the same function so I don't have to pass around the other POSTed form val ues as URL query parameters. Now, as a result of changing the flow above, I also had to make a change to the way I show the results page. Instead of processing the file and returning the rendered results to the user directly, I process the file, save the rendered results to /tmp, and return a redirect to the user to the results page.	2021-03-13 22:11:26 +02:00
Alan Orth	adc2d06094	main.py: Add note about using /tmp ... I originally wanted to use an "uploads" directory or something, but it seems we can only write to /tmp on Google App Engine. They really want you to buy storage or database services! This is memory mapped so it disappears when you re-deploy.	2021-03-13 22:09:17 +02:00
Alan Orth	bc256b242d	static/css/style.min.css: Regenerate	2021-03-13 22:07:53 +02:00
Alan Orth	f82cb6ce05	source/scss/main.scss: Increase container width ... We need more space for the log on the results page and we actually don't even need to worry about people running this on a phone.	2021-03-13 22:06:43 +02:00
Alan Orth	4bdec3b889	app.yaml: Use Python 3.9 ... Python 3.9 is apparently now generally available.	2021-03-13 14:12:14 +02:00
Alan Orth	f79be86361	templates/index.html: Use Bootstrap form components	2021-03-13 13:50:46 +02:00
Alan Orth	3715c5e976	Use new commit for csv-metadata-quality ... This one doesn't treat the fixing of invalid multi-value separators as "unsafe".	2021-03-13 13:02:47 +02:00
Alan Orth	8603ec4bca	main.py: Rework command line args ... Turns out we only need to use sys.argv when we were trying to run the csv-metadata-quality module directly in Python by importing it.	2021-03-13 12:52:22 +02:00
Alan Orth	0471820f3a	main.py: Actually use sys.argv ... I set this but never actually passed it to the subprocess. Now I'm wondering if I actually need it, or if that was just when I was tr- ying to import the csv-metadata-quality module?	2021-03-13 12:49:01 +02:00
Alan Orth	f1de00d7ca	Regenerate requirements ... Generated using poetry: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt	2021-03-12 23:11:33 +02:00
Alan Orth	31e699bb8b	poetry.lock: Sync lockfile	2021-03-12 23:10:55 +02:00
Alan Orth	982b47459c	Use gunicorn to run on Google App Engine ... See: https://cloud.google.com/appengine/docs/flexible/python/quickstart	2021-03-12 23:10:29 +02:00
Alan Orth	38ace23f97	.gitignore: Add __pycache__	2021-03-12 23:09:29 +02:00
Alan Orth	e5f57802f3	Don't ignore uploads directory ... We're using /tmp now since we apparently can't create and upload to arbitrary directories on Google App Engine (?).	2021-03-12 23:08:28 +02:00
Alan Orth	69abc3d764	Add downloading of cleaned CSV file	2021-03-12 23:00:48 +02:00
Alan Orth	37cee1d3f0	Add configuration for isort ... See: https://sourcery.ai/blog/python-best-practices/	2021-03-12 22:59:57 +02:00
Alan Orth	126bf74ea0	main.py: Remove comments	2021-03-12 21:28:24 +02:00
Alan Orth	54ea340b80	poetry.lock: Run poetry update	2021-03-12 21:24:51 +02:00
Alan Orth	75c5e48aa3	templates/header.html: Link to home in navbar	2021-03-12 21:24:31 +02:00
Alan Orth	4de4388b1e	Encode filenames to base64 and change URL to /result	2021-03-12 21:22:03 +02:00
Alan Orth	032fa5f2e7	pyproject.toml: Use default csv-metadata-quality release ... Now that we are just running the command line version via a Python subprocess we don't have to do anything special.	2021-03-12 21:18:50 +02:00
Alan Orth	633fb8419a	Regenerate requirements.txt ... Generated using poetry: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --dev -f requirements.txt > requirements-dev.txt	2021-03-12 19:43:06 +02:00
Alan Orth	c1d79c147c	poetry.lock: Update lockfile	2021-03-12 19:41:39 +02:00
Alan Orth	477b5bcd42	Regenerate static assets	2021-03-12 19:40:49 +02:00
Alan Orth	e13d63bf6b	Major refactor ... Refactor the templates to include a header, use csv-metadata-quality in a subshell instead of trying to import and pass args to it (which I actually managed to do, but then trying to capture the output was an issue), and use ansi2html to prepare the CLI output as the HTML.	2021-03-12 19:14:49 +02:00
Alan Orth	ed747b2cef	templates/footer.html: Add link to csv-metadata-quality on GitHub	2021-03-12 19:07:29 +02:00