Add dockerized development environment; pypgstac testing; CI - fix issue with ingest #18
Conversation
This adds the entry to the migrations table which was missing prior, and would cause "pypgstac migrate" to fail on new databases
|
@lossyrob thanks for pulling this all together. I definitely like this approach. One thing that I don't have a strong opinion on. I had been using Poetry to build & publish to PyPi. In the Makefile the setup.py was actually generated from the Poetry build artifacts in the make command. If we stick with Poetry, we should add the dev requirements to the pyproject.toml rather than as a requirements-dev.txt and we should use Poetry for the build/publishing steps. If we don't want to use Poetry, we should get rid of the pyproject.toml and just use setup.py + requirements files. I was also doing a couple steps in the Makefile to set the version in a few places and to concatenate the sql scripts into the pgstac-x.x.x.sql file that gets distributed with pypgstac. Not sure I was handling that in the best way, but we should make sure that we are getting the version number set appropriately in all the right places. |
This causes the required egg-info to be created in the source directory, which is mounted in when running the dev container and can cause issues. Because the source is on the PYTHONPATH any edits mounted into the container will be used over the installed source
I don't have experience with poetry, but am cool with learning it and integrating it if that's the preference. @geospatial-jeff had mentioned he had opinions on the use of poetry in stac-fastapi, would be curious of your thoughts on which direction to go here Jeff.
Makes sense - perhaps we can either keep the Makefile or put this logic into a script, and then have these generated files committed as part of a PR during the release process? Usually right before a release there's a version change and a CHANGELOG turnover PR that's the last step before a release (e.g. stac-utils/pystac#532), so we could have those files generated as part of a release PR like that? |
For consistency, I'd be inclined to add to the scripts. I can do that as a follow on PR and then once we get that and the docker push in, we can get rid of the makefile. |
There was a problem hiding this comment.
There are some follow on tasks that we can address now or in future PRs, but this is definitely a great step forward for managability.
Things we should follow up on:
Docker push CI
Version tagging script / process documentation including checklist to create version to version migration and change log.
Decision and cleanup on whether to use poetry.
Get PGTap SQL tests running/passing with CI.
|
Having a way to lock down dependencies (poetry/pipenv) beyond what vanilla pip provides is really nice for dependency management. Ever since using poetry/pipenv I've never really had the "it works on my machine" sort of problem that you can run into with vanilla pip. The biggest difference is that pipenv is slow to lock but fast to install while poetry is fast to lock but slow to install. I think poetry is a little more ergonomic as well. I'd recommend poetry over pipenv for this project. |
Just +1 I've had a good experience with Poetry lately. The next version of Poetry (or the 1.2.0 alpha) will make local development easier with support for editable installs like |
Oh, that's awesome! That's why the makefile had a weird step that it extracted the setup.py from the build. |
|
Made follow up issues, merging |
Related Issues: #6
This PR adds a dockerized development environment that allows consistent testing across dev environments.
This follows the "Scripts To Rule Them All" pattern for scripts that control the development environment. It makes it very easy for developers to clone the repository and get started working with the project.
A docker-compose defines the development database (which uses the existing Dockerfile in the repo) and a dev container (which uses a new Docker.dev), and all scripts run through those docker containers.
This also adds black as a formatter, mypy as a type checker, and flake8 as a linter to the python project. This set of dev tools are become standard across python projects in stac-utils.
The following modifications were made to pypgstac:
There is no database schema updates, but I did change
pgstac.sqlto include running thesql/999_version.sqlagainst new databases. Otherwise there's no entry in the migration table, and so apypgstac migratecommand will assume it's a new database, and fail on creation of tables. I also fixed that sql to include pypgstac in the SEARCH_PATH.GitHub Workflows are created to perform CI on pull requests and on the main branch. It will also publish pypgstac to pypi on tags. This will allow us to publish only via tagging releases.
I was unable to get
test/test.shworking - it seems like there's some missing files (e.g. test/test.sql) that perhaps were not committed.I didn't touch the Makefile, but I suggest after this PR goes in to remove that file. The only thing not covered by these STRTA/CI is the publishing to dockerhub for pgstac images, which should be added in a follow up PR.
Testing:
scripts/setupto build a new docker dev environmentscripts/testto run all the testsscripts/consoleto drop into a dev console, which will allow you to runpypgstacagainst the development db from the command linescripts/console --dbto drop into apsqlinteractive session to inspect the databasescripts/formatto format the python codescripts/migrateto migrate the db (which should be a no-op)scripts/updateto rebuild the docker containers.