-
Notifications
You must be signed in to change notification settings - Fork 463
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow for a Graceful Shutdown of the Worker #109
Conversation
Add a new method to the worker requestShutdown that allows the worker to gracefully shutdown all record processors. The graceful shutdown gives the record processors a last chance to checkpoint before they're terminated. To use these new features the record processor must implement IShutdownnotificationaware. Some cleanup to try and make the state transitions of the record processor more clear.
Handle some possible race conditions during the shutdown process. Added more comments clarifying the how shutdown works, and the race conditions it can face.
Added some additional comments clarifying the shutdown check miss.
Updated the log message to include the the count awaiting final shutdown.
* General * Allow disabling shard synchronization at startup. * Applications can disable shard synchronization at startup. Disabling shard synchronization can application startup times for very large streams. * [PR awslabs#102](awslabs#102) * Applications can now request a graceful shutdown, and record processors that implement the IShutdownNotificationAware will be given a chance to checkpoint before being shutdown. * This adds a [new interface](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/interfaces/v2/IShutdownNotificationAware.java), and a [new method on Worker](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/Worker.java#L539). * [PR awslabs#109](awslabs#109) * Solves [Issue awslabs#79](awslabs#79) * MultiLangDaemon * Applications can now use credential provides that accept string parameters. * [PR awslabs#99](awslabs#99) * Applications can now use different credentials for each service. * [PR awslabs#111](awslabs#111)
* General * Allow disabling shard synchronization at startup. * Applications can disable shard synchronization at startup. Disabling shard synchronization can application startup times for very large streams. * [PR #102](#102) * Applications can now request a graceful shutdown, and record processors that implement the IShutdownNotificationAware will be given a chance to checkpoint before being shutdown. * This adds a [new interface](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/interfaces/v2/IShutdownNotificationAware.java), and a [new method on Worker](https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/Worker.java#L539). * [PR #109](#109) * Solves [Issue #79](#79) * MultiLangDaemon * Applications can now use credential provides that accept string parameters. * [PR #99](#99) * Applications can now use different credentials for each service. * [PR #111](#111)
What is the difference between the shutdown and shutdownRequested? What is the use case for both of them? Can you provide an example for both? Do we need to call requestShutdown then shutdown or does requestShutdown should shutdown the worker as well? Thanks. @pfifer |
@DavidDeCoding The In general I would recommend using |
Thanks @pfifer. I thought so too. |
…heckpoint - instance doesn't hold the lease for this shard" errors. Spark's KinesisRecordProcessor to implement Kinesis IShutdownNotificationAware interface to be notified before processor will be shutdown so that it can update checkpoint before shutdown. See more details from: - awslabs/amazon-kinesis-client#79 - awslabs/amazon-kinesis-client#109
Add a new method to the worker requestShutdown that allows the worker to
gracefully shutdown all record processors. The graceful shutdown gives
the record processors a last chance to checkpoint before they're
terminated.
To use these new features the record processor must implement
IShutdownnotificationaware.
Some cleanup to try and make the state transitions of the record
processor more clear.