-
Notifications
You must be signed in to change notification settings - Fork 132
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix batch_put ttl issue #457
Fix batch_put ttl issue #457
Conversation
86e80bc
to
d751fcc
Compare
00f6ff1
to
2d24519
Compare
Signed-off-by: limbooverlambda <schakra1@gmail.com>
2d24519
to
767e930
Compare
src/raw/requests.rs
Outdated
// Maintain a map of the pair and its associated ttl | ||
let kvs = self.pairs.clone(); | ||
let kv_pair = kvs.into_iter().map(Into::<KvPair>::into); | ||
let kv_ttl = kv_pair.zip(self.ttls.clone()).collect::<HashMap<_, _>>(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
store_stream_for_keys
is able to handle data of keys along with values and ttls. Mapping by Hash map is not necessary, and it's not efficient as well.
Please refer to PrewriteRequest::shards
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the review. Didn't realize store_stream_for_keys
has this capability. Removed the HashMap to collect the ttls for the kv_pair.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @pingyu, I have made the changes. Please let me know if there's anything else I need to consider for this PR.
Signed-off-by: limbooverlambda <schakra1@gmail.com>
Signed-off-by: limbooverlambda <schakra1@gmail.com>
Signed-off-by: limbooverlambda <schakra1@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rest LGTM.
Signed-off-by: limbooverlambda <schakra1@gmail.com>
Thanks for helping me clean up this PR @pingyu. Addressed your comments. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm sorry I missed another suggestion in last comments. The clone
in comparation should also be un-necessary.
src/raw/requests.rs
Outdated
@@ -7,10 +7,12 @@ use std::time::Duration; | |||
|
|||
use async_trait::async_trait; | |||
use futures::stream::BoxStream; | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done.
src/raw/requests.rs
Outdated
.zip(ttls) | ||
.map(|(kv, ttl)| KvPairTTL(kv, ttl)) | ||
.collect(); | ||
kv_ttl.sort_by(|a, b| a.0.key.clone().cmp(&b.0.key)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
kv_ttl.sort_by(|a, b| a.0.key.clone().cmp(&b.0.key)); | |
kv_ttl.sort_by(|a, b| a.0.key.cmp(&b.0.key)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed.
Signed-off-by: limbooverlambda <schakra1@gmail.com>
Yeah, I should have caught that. Removed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM~
Thanks for you contribution !
There is a bug in the client where the ttls are not properly sharded for RawBatchPuts. This essentially means that if the batch is spread across multiple regions, the ttls for the batch specific to the shard is not properly aligned. This is an issue when the request ends up in the TiKV node specific to the region and fails because of the misalignment of the
Vec<TTLs
> with its correspondingVec<KvPair>
. The fix was to change the sharding logic for RawBatchPutRequest so that the TTLs are aligned with the KvPair. Maintaining a mapping of the kvpair to its corresponding ttl in the Shard output.