Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Master more robust with state persistence #58

Open
nilesh-c opened this issue Aug 20, 2014 · 0 comments
Open

Make Master more robust with state persistence #58

nilesh-c opened this issue Aug 20, 2014 · 0 comments

Comments

@nilesh-c
Copy link
Member

If the Master dies and restarts, we lose state.

Akka supports something called PersistentActor that lets us save actor state. Worker-side state is strictly unimportant. If a Worker dies and restarts, the corresponding job is just re-enqueued by the Master and everything goes on as usual.

If a Master fails we have a problem. We loose the queue, mirrors statuses, worker progress reports, and worker download size (bytes) distribution. Using PersistentActor can help us handle this easily. It uses an append-only "journal" to record state (events rather!) to.

I am yet to come across a situation where my Master actor died, but this is certainly a possibility that we should handle. Any loose Exception can kill the Master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant