Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Orchestrate db migrations for 2 simultaneous starting pods in a k8s environment #206

Closed
BorntraegerMarc opened this issue May 12, 2020 · 3 comments

Comments

@BorntraegerMarc
Copy link

Is your feature request related to a problem? Please describe.
I'm wondering whether this library takes care of a problem that we're currently having in production environments (it's a very common problem). What is happening:

  1. In our kubernetes production cluster we have of a Deployment 2 replica pods running (that means that our server runs 2 instances for load balancing).
  2. On every server startup we check if DB migrations need to be run
  3. Now, if we release a new version including a new DB migration the 2 pods would simultaneously start & run the exact same DB migration. Which is not good because DB migrations should only run once.

This is especially a problem for migrations that take multiple seconds to finish.

Describe the solution you'd like
I honestly don't have a solution for this problem 🙂

Only thing I can say is that ideally this lib "knows" when a pod has already started a migration and doesn't start it again. Some of the "bigger" db migration tools take care of this.

Describe alternatives you've considered
None :(

Additional context

@seppevs
Copy link
Owner

seppevs commented Jul 21, 2020

Sorry, but I'm going to close this issue. I don't think it's a problem migrate-mongo should solve.

@seppevs seppevs closed this as completed Jul 21, 2020
@205g0
Copy link

205g0 commented Jun 27, 2021

@BorntraegerMarc have you found a solution to your problem?

@BorntraegerMarc
Copy link
Author

BorntraegerMarc commented Jul 12, 2021

I definitely think this is a problem that the db migration tool should sove.

Since the owner said that migrate-mongo will not be solving this problem I'm relying on a different db migration tool: https://github.com/db-migrate/node-db-migrate/

In the next major version we'll get a feature State Manager: db-migrate/node-db-migrate#538

@205g0 The other obvious solution is to always remove all pods except for 1 and then run the db migration in only that pod and then to start up all other pods again. Depending on your workload of the cluster this might not work for you though in production

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants