-
Notifications
You must be signed in to change notification settings - Fork 324
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
V52 database migration breaks existing Marquez installations #2298
Comments
The problem here are the repeatable migrations https://flywaydb.org/documentation/concepts/migrations.html#repeatable-migrations which according to the docs:
This in fact means that we do not test it properly as our clean docker environments do not run repeatable migrations before the versioned migrations. In other words, our tests never experienced scenario with a view being present which is a cause of this bug. Solution to this could be:
I agree we need a more extensive tests that test migration at different states. |
A possible approach would be to spin up an existing marquez docker image (like the last release or something), then try to apply flyway migration to the database afterward. That should catch at least structural incompatibilities (it won't catch issues in the data). |
The migration introduced in #2217 is not backward compatible with existing marquez installations. The following exception is thrown during the flyway migration.
Database migrations can't change the types of columns in existing views in order to change the type, it is necessary to drop the existing view, then alter the columns, and recreate the view.
Marquez is missing a way of testing these migrations against existing installations. We should figure out a way to validate these scripts before merging to main.
The text was updated successfully, but these errors were encountered: