Skip to content
This repository has been archived by the owner on Aug 23, 2023. It is now read-only.

major kafka ingestion format changes + some other stuff

Compare
Choose a tag to compare
@Dieterbe Dieterbe released this 16 May 10:19
· 3245 commits to master since this release
e797a9f

kafka format upgrade

support for new MetricPoint optimized data format in the kafka mdm topic, resulting in less kafka io, disk usage, GC workload, metrictank and kafka cpu usage, faster backfills. #876 , #885 #890, #891, #894, #911
this also comes with:

  • new dashboard
  • updates of internal representations of keys, archives, index structures, intra-cluster requests, etc.(so you must do a colored upgrade or new cluster deployment)
  • metrics changes for metrics_received, metrics_invalid changed see 01dabf9, a772c10
  • removal of mt-replicator-via-tsdb tool
  • deprecation of .Metric field in MetricData and MetricDefinition. it is now ignored in incoming MetricData and in the cassandra index tables.
  • mt-schemas-explain: also show built-in default
  • various update to some of the docker stacks (use latest graphite, control x-org-id authentication, use prometheus docker monitoring, update for latest fakemetrics and tsdb-gw, etc)

other

  • upgrade to latest sarama (kafka) library. #905
  • remove pressure.idx and pressure.tank metrics, too much overhead and too little use. #905
  • sarama reduce default kafka-mdm channel-buffer-size #886
  • refactor chaos testing and carbon end2end test into a unified testing library and test functions. #830
  • migrate "public org id" functionality from magic org=-1 to a >0 orgid to be specified via public-org config setting. #880, #883
  • fix cluster load balancing #884
  • initialize cassandra schemas via template files, support scylladb schemas. #898
  • support adding arbitrary extra tags in jaeger traces #904
  • Accept bool strings for ArgBool #895