idx_migration tool #425
idx_migration tool #425
Conversation
|
@Dieterbe I think we need to cleanup the partitioning in tsdb-gw before we merge this tool. I have opened https://github.com/raintank/tsdb-gw/issues/16 to address this. |
|
I want to have a consistent naming scheme for tools that works well with tab completion . maybe something like: mt-index-migrate |
woodsaj
force-pushed
the
idxMigration
branch
from
b885aea
to
214b96a
Compare
define a generic partitioner interface for getting the partition id of a metricData or metricDefinition struct. Impliment initial KafkaPartitioner that can partition based on orgId or series name.
woodsaj
force-pushed
the
idxMigration
branch
from
214b96a
to
e828372
Compare
woodsaj
force-pushed
the
idxMigration
branch
from
57f8531
to
d49747a
Compare
woodsaj
force-pushed
the
idxMigration
branch
from
0194132
to
7433b89
Compare
|
Now use 4b buf for byorg. May be incompat with previous approach I think but not a problem since all current deployments just consume all partitions. On phone excuse brevity |
| return k.Partitioner.Partition(&sarama.ProducerMessage{Key: sarama.ByteEncoder(key)}, numPartitions) | ||
| } | ||
|
|
||
| func (k *KafkaPartitioner) GetPartitionKey(m schema.PartitionedMetric, b []byte) ([]byte, error) { |
There was a problem hiding this comment.
Callable by other packages needed ?
There was a problem hiding this comment.
yes, it is called by tsdb-gw
| "gopkg.in/raintank/schema.v1" | ||
| ) | ||
|
|
||
| const table_schema = `CREATE TABLE IF NOT EXISTS %s.metric_idx ( |
There was a problem hiding this comment.
Seems safer to reuse the table and index schema defined in idx package
| p, err := partitioner.Partition(&mdef, int32(*numPartitions)) | ||
| if err != nil { | ||
| log.Error(3, "failed to get partition id of metric. %s", err) | ||
| mdef.Partition = 0 |
There was a problem hiding this comment.
This condition seems like a big problem warranting an abort instead of proceeding with 0 partition
re-use cassandraIdx keyspace consts in mt-index-migrate
|
we only need 4bytes for the orgId. hashing a 8byte slice padded with 0's is a waste of CPU cycles. Changing the partitioning in existing deployments wont have any impact as we are not running sharded clusters anywhere. |
No description provided.