Create table resilience #41
Conversation
|
Can we get this reviewed and merged please? We manage our DynamoDB table separately to our KCL app, and we use very strict IAM roles for access to the table. It's kinda silly that we have to grant the KCL app permission to create a table we know already exists! :) |
|
sorry for pulling you in @pfifer, I saw you were being active on the other PRs. I work on the same team as Sam and we really want this PR to get merged so that we don't have the infrastructure overhead of managing extra IAM roles. |
|
A version of this is already merged: 8fc5dd5 This PR, and Fix #36 Avoid failure on start due to CreateTable are very similar. I'm starting the process of release to Maven today, and hope to have v1.6.5 out soon. |
|
The change is committed, and will be released to Maven soon. |
…el of the shard hierarchy (awslabs#41) * Update NonEmptyLeaseTableSynchronizer algorithm to create leases for no more than one level of the shard hierarchy * Add and fix unit tests for LATEST, TRIM_HORIZON, AT_TIMESTAMP * Update unit tests * Fix edge case of reading from TRIM with partial lease for one child shard * Revert child shards optimization for TRIM/TIMESTAMP case
This breaks the dependency of worker start on the CreateTable API in DynamoDB, which prevented KCL workers from starting during the us-east-1 DynamoDB outage today.