Tool to determine mapping from vindex and value to shard #17290
Conversation
…ndex and shard ranges Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Review ChecklistHello reviewers! 👋 Please follow this checklist when reviewing this Pull Request. General
Tests
Documentation
New flags
If a workflow is added or modified:
Backward compatibility
|
vitess-bot
bot
added
NeedsBackportReason
rohit-nayak-ps
added
Type: Feature
Component: Cluster management
and removed
NeedsDescriptionUpdate
… reported for a keyspace id Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #17290 +/- ##
==========================================
- Coverage 67.40% 67.39% -0.02%
==========================================
Files 1574 1574
Lines 253205 253251 +46
==========================================
- Hits 170676 170673 -3
- Misses 82529 82578 +49 ☔ View full report in Codecov by Sentry. |
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
rohit-nayak-ps
removed
NeedsWebsiteDocsUpdate
rohit-nayak-ps
requested review from
harshit-gangal,
GuptaManan100 and
a team
|
Do we have to print the ksid? |
| func getValue(valueStr string) (sqltypes.Value, int64, error) { | ||
| var value sqltypes.Value | ||
| valueInt, err := strconv.ParseInt(valueStr, 10, 64) | ||
| if err == nil { | ||
| value = sqltypes.NewInt64(int64(valueInt)) | ||
| } else { | ||
| valueUint, err := strconv.ParseUint(valueStr, 10, 64) | ||
| if err == nil { | ||
| value = sqltypes.NewUint64(valueUint) | ||
| } else { | ||
| value = sqltypes.NewVarChar(valueStr) | ||
| } | ||
| } | ||
| return value, valueInt, err | ||
| } |
There was a problem hiding this comment.
we should take value type as input using cli
| allShards := make([]*topodata.ShardReference, 0, len(shards)) | ||
| for shard, keyRange := range shards { | ||
| allShards = append(allShards, &topodata.ShardReference{ | ||
| Name: shard, | ||
| KeyRange: keyRange, | ||
| }) | ||
| } |
There was a problem hiding this comment.
This can be done as part of a struct init and passed along to methods
| foundShard, err := mapShard(shards, ksid) | ||
| if err != nil { | ||
| return "", nil, fmt.Errorf("failed to map shard: %w", err) | ||
| } |
There was a problem hiding this comment.
we should add the original value as well to the error message
| var start, end []byte | ||
| if len(parts) > 0 && parts[0] != "" { | ||
| start, err = hex.DecodeString(parts[0]) | ||
| if err != nil { | ||
| log.Fatalf("failed to decode shard start: %v", err) | ||
| } | ||
| } | ||
| if len(parts) > 1 && parts[1] != "" { | ||
| end, err = hex.DecodeString(parts[1]) | ||
| if err != nil { | ||
| log.Fatalf("failed to decode shard end: %v", err) | ||
| } | ||
| } |
There was a problem hiding this comment.
can use method key.ParseKeyRangeParts to create *topodatapb.KeyRange
There was a problem hiding this comment.
You can also look at topo.ValidateShardName method.
There was a problem hiding this comment.
Used topo.ValidateShardName
certainly not needed, but it doesn't hurt, easy enough to work around like this: |
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Refactored code and added a unit test. I would prefer to keep this in |
There was a problem hiding this comment.
oops, not sure how that got committed. Good catch, removed!
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Description
This is a separate tool in the vitess
toolsdirectory which can be used as a separate binary without needing to setup a cluster. It can be used to map a list of values from the stdin and outputs a csv format giving you the relatedkeyspace idand theshardin which it will reside. The inputs are the vindex type and the sharding scheme.I tested this by comparing about 2K results of vexplain by specifying a test vschema file like
vtexplain --shards 128 --vschema-file test-vschema.json --sql "select * from users where id = $1" --schema-file "test-schema.sql" --output-mode json | jq -r '.[].TabletActions | keys[] | split("/") | .[1]' | awk -v id="$id" '{print id "," $0}'.Features
hash,xxhash).stdinand outputs the corresponding shard information, so it can beused to map values from a file or another program.
Usage
Flags
--vindex: Specifies the name of the vindex to use (e.g.,hash,xxhash) (defaultxxhash)One (and only one) of these is required:
--shards: Comma-separated list of shard ranges--total_shards: Total number of shards, only if shards are uniformly distributedRelated Issue(s)
#17292
Checklist
Deployment Notes