Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BackgroundWorkerSink crashes if target sink throws an exception #44

Closed
silviucotilici opened this issue Jun 6, 2019 · 1 comment · Fixed by #45
Closed

BackgroundWorkerSink crashes if target sink throws an exception #44

silviucotilici opened this issue Jun 6, 2019 · 1 comment · Fixed by #45
Labels

Comments

@silviucotilici
Copy link

silviucotilici commented Jun 6, 2019

Hello guys,

I am having the following setup: I am using async-sink that targets azuretablestorage-sink. All nice and good until azuretablestorage starts throwing some timeout exceptions (for whatever reasons).

The thing is that when a timeout occurs (or any other exception) in the target sink,
the entire worker thread crashes (BackgroundWorkerSink.Pump()) and events from the queue are no longer dequeued and emitted trough the pipeline, so the queue quickly reaches max capacity and then drops out incoming events. Basically all logging mechanism is dead because of a single timeout.

I was wondering if wouldn't be nice to catch exceptions from the target sink and write them to selflog and let the worker thread continue pumping the rest of the events in the queue?
Eventually failed events could be stored and retried later.
Or exceptions like this should be treated in the target sink? what do you think?

@silviucotilici silviucotilici changed the title BackgroundWorkerSink crashes if next sink in the pipeline throws an exception BackgroundWorkerSink crashes if target sink throws an exception Jun 6, 2019
@nblumhardt
Copy link
Member

Thanks for the report. This is a regression introduced in #19, I'll PR a fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants