Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Brigade Worker on Transient Fault Handling #625

Closed
jeehwancho opened this issue Sep 10, 2018 · 2 comments
Closed

Brigade Worker on Transient Fault Handling #625

jeehwancho opened this issue Sep 10, 2018 · 2 comments

Comments

@jeehwancho
Copy link
Contributor

As part of evaluating autoscaling, I found out that workers (the calls from kubernetes-client, specifically) don't handle transient faults, which occur when a cluster is being scaled.
While the transient faults caused by the AKS's autoscale has its own problem to be resolved from their end (Azure/AKS#381), I think a client like brigade workers should handle such problems regardless.
According to kubernetes-client, transient faults are left for user to handle at this moment. (kubernetes-client/javascript#96).
So I updated k8s.ts with the exponential backoff retry, which seems to work in a scaling scenario where timeout occurs time to time.
I'm planning to make a PR regarding this, and please let me know if you have any thoughts.

Cheers,
Jeehwan
@jeehwancho jeehwancho mentioned this issue Sep 10, 2018
Merged
@jeehwancho
Copy link
Contributor Author

Just created a PR. It'd be appreciated if anyone could review it.

technosophos pushed a commit that referenced this issue Sep 28, 2018
@jeehwancho @technosophos
fix(worker): handle transient faults (#625) (#626)
d6f2a59 
* fix(worker): handle transient faults (#625)
fix(worker): handle JSON.parse exception caused by transient faults in
watch
feat(worker): support KUBECONFIG (feedback from #596)

* Specify types for pod and request in worker (#626)
Make sure watch stream stay connected
@jeehwancho
Copy link
Contributor Author

close via #626

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jeehwancho