Bug Report: vttablet w/--enable-tx-throttler starts with disabled tx throttler

[timvaillancourt]

Overview of the Issue

👋 Vitess,

I've noticed that vttablet will successfully start when the user has enabled the transaction throttler (using --enable-tx-throttler) and the throttler fails to start up.

Instead of failing to start as I expected, vttablet logs an error (below) and continues to appear healthy/operational.

Example due to missing, required initial_rate field in -tx-throttler-config:

E0220 09:43:58.985198    3187 tx_throttler.go:88] Error creating transaction throttler. Transaction throttling will be disabled. Error: initial_rate must be >= 1

NOTE: after this error vttablet continues to start as normal

I believe this behaviour was implemented intentionally, however I feel it is unsafe and confusing to the user that asked for something to be enabled that is nearly-silently disabled in reality. It took quite a while for me to notice this was the cause of my throttling problems, for example.

My preference is that vttablet fails to start in this scenario, which I feel would be a valid, but "breaking change" to the tx throttler 🤔

Assuming agreement this is a bug/problem, I am happy to contribute a fix for this myself 👍

Reproduction Steps

1. Start vttablet with transaction throttler and an incomplete tx-throttler configuration
   • Example:

     vttablet -tablet-path us_east_1e-123456 \
       -mycnf-file /mnt/vitess/mysql/datadir/my.cnf -topo_implementation "consul" \
       -topo_global_server_address "consul.redacted.com:8500" -topo_global_root "vitess/global" \
       -init_keyspace test -init_shard -80 -init_tablet_type "replica" \
       -enable-tx-throttler 
     

   • Notice the required tx-throttler flag --tx-throttler-healthcheck-cells is not specified
2. Noticed that Error creating transaction throttler. Transaction throttling will be disabled. is logged, but vttablet continues to startup

Full example

$ vttablet -tablet-path us_east_1e-123456 \
    -mycnf-file /etc/my.cnf -topo_implementation "consul" \
    -topo_global_server_address "consul.redacted.com:8500" -topo_global_root "vitess/global" \
    -init_keyspace test -init_shard -80 -init_tablet_type "replica" \
    -enable-tx-throttler 
E0221 10:13:18.843054    6307 tx_throttler.go:88] Error creating transaction throttler. Transaction throttling will be disabled. Error: empty healthCheckCells given. &{enabled:true topoServer:0xc0012319c0 throttlerConfig:0xc001217950 healthCheckCells:[]}
W0221 10:13:18.970389    6307 tm_init.go:706] creating keyspace and shard failed (Get "http://consul.redacted.com:8500/v1/kv/vitess/global/keyspaces/test/shards/-80/Shard": dial tcp: lookup consul.redacted.com on 127.0.0.1:53: no such host), backing off 1s before retrying
W0221 10:13:19.973236    6307 tm_init.go:706] creating keyspace and shard failed (Get "http://consul.redacted.com:8500/v1/kv/vitess/global/keyspaces/test/shards/-80/Shard": dial tcp: lookup consul.redacted.com on 127.0.0.1:53: no such host), backing off 1.125602181s before retrying
W0221 10:13:21.101325    6307 tm_init.go:706] creating keyspace and shard failed (Get "http://consul.redacted.com:8500/v1/kv/vitess/global/keyspaces/test/shards/-80/Shard": dial tcp: lookup consul.redacted.com on 127.0.0.1:53: no such host), backing off 1.295018994s before retrying
^C

Binary Version

vttablet binary built from slackhq/vitess (fork) based on vitessio/vitess tag: v12.0.5

Our fork has no changes to go/vt/vttablet/tabletserver/txthrottler code

I expect this issue exists on v12+ as well

Operating System and Environment details

- Ubuntu 18.04.6 LTS
- Linux 5.4.0-1096-aws
- x86_64

Log Fragments

E0220 09:43:58.985198    3187 tx_throttler.go:88] Error creating transaction throttler. Transaction throttling will be disabled. Error: initial_rate must be >= 1

[shlomi-noach]

I agree this is a bug. The correct solution is to have a sane default for cells, and the default would be "all cells". And I see you've done exactly that in #12435, which I'll review next.