Resilient watches #67
Conversation
|
|
||
| func (e *FatalError) Error() string { return e.Err.Error() } | ||
| func (*FatalError) Fatal() bool { return true } | ||
|
|
There was a problem hiding this comment.
This whole error thing I think would look cleaner with embedding:
type FatalError struct {
error
}
func (*FatalError) Fatal() bool { return true }
func main() {
e := &FatalError{errors.New("some error")}
fmt.Printf("error: %v\n", e)
fmt.Printf("error: is fatal: %v\n", e.Fatal())
}
Try: http://play.golang.org/p/UHCiHEnDcJ, It would eliminate the needless pass-through, and a few lines of code.
There was a problem hiding this comment.
Good call! Fixed.
|
👍 |
| } | ||
| } | ||
| w.cordCtx.Log(metafora.LogLevelError, "%s Unexpected error unmarshalling etcd response: %+v", w.path, err) | ||
| return |
There was a problem hiding this comment.
Would it be safer to just do a continue startWatch here too, just so the node doesn't go down? Maybe throw in a sleep and retry count too to prevent it from thrashing?
There was a problem hiding this comment.
Metafora itself does that here: https://github.com/lytics/metafora/blob/master/metafora.go#L202-L212
My current strategy has been to only handle specific errors that I know how to handle here (timeouts and 401/EcodeExpiredIndex), and leave catchall error handling up to the consumer which can have configurable error handling logic added.
That being said maybe it's currently backwards: maybe coordinator implementations should handle all non-fatal errors internally and metafora's core consumer should just shutdown when coordinators actually return errors?
Feel free to open an issue if you have an opinion. I can't think of a clear winning strategy at the moment.
|
👍 |
|
Merged a rebased version of this branch/PR. |
Got some help in #coreos to find out what to do about 401 errors. Turns out the answer is to restart the watch with a fresh Get. That ensures metafora will never miss events.
The diff looks like an absolute mess. Basically all I did was put another for loop around the initial Get when we start watching, and have 401s continue that outer loop.
bleh