Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backport #13130: Tablet throttler: throttler-config-via-topo defaults 'true', deprecation message for old flags #13237

Merged
merged 1 commit into from
Jun 5, 2023
Merged

Backport #13130: Tablet throttler: throttler-config-via-topo defaults 'true', deprecation message for old flags #13237

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions changelog/17.0/17.0.0/summary.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
- **[New command line flags and behavior](#new-flag)**
- [Builtin backup: read buffering flags](#builtin-backup-read-buffering-flags)
- [Manifest backup external decompressor command](#manifest-backup-external-decompressor-command)
- [Throttler config via topo enabled by default](#throttler-config-via-topo)
- **[New stats](#new-stats)**
- [Detailed backup and restore stats](#detailed-backup-and-restore-stats)
- [VTtablet Error count with code](#vttablet-error-count-with-code)
Expand Down Expand Up @@ -164,6 +165,12 @@ This feature enables the following flow:
```
2. Restore that backup with a mere `Restore` command, without having to specify `--external-decompressor`.

#### <a id="throttler-config-via-topo" />vttablet --throttler-config-via-topo

This flag was introduced in v16 and defaulted to `false`. In v17 it defaults to `true`, and there is no need to supply it.

Note that this flag overrides `--enable-lag-throttler` and `--throttle-threshold`, which now give warnings, and will be removed in v18.

### <a id="new-stats"/>New stats

#### <a id="detailed-backup-and-restore-stats"/>Detailed backup and restore stats
Expand Down
2 changes: 1 addition & 1 deletion go/flags/endtoend/vttablet.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -316,7 +316,7 @@ Usage of vttablet:
--throttle_metrics_threshold float Override default throttle threshold, respective to --throttle_metrics_query (default 1.7976931348623157e+308)
--throttle_tablet_types string Comma separated VTTablet types to be considered by the throttler. default: 'replica'. example: 'replica,rdonly'. 'replica' aways implicitly included (default "replica")
--throttle_threshold duration Replication lag threshold for default lag throttling (default 1s)
--throttler-config-via-topo When 'true', read config from topo service and ignore throttle_threshold, throttle_metrics_threshold, throttle_metrics_query, throttle_check_as_check_self
--throttler-config-via-topo When 'true', read config from topo service and ignore throttle_threshold, throttle_metrics_threshold, throttle_metrics_query, throttle_check_as_check_self (default true)
--topo_consul_lock_delay duration LockDelay for consul session. (default 15s)
--topo_consul_lock_session_checks string List of checks for consul session. (default "serfHealth")
--topo_consul_lock_session_ttl string TTL for consul session.
Expand Down
7 changes: 4 additions & 3 deletions go/test/endtoend/onlineddl/ghost/onlineddl_ghost_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ import (

"vitess.io/vitess/go/test/endtoend/cluster"
"vitess.io/vitess/go/test/endtoend/onlineddl"
"vitess.io/vitess/go/test/endtoend/throttler"

"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
Expand Down Expand Up @@ -166,9 +167,6 @@ func TestMain(m *testing.M) {
}

clusterInstance.VtTabletExtraArgs = []string{
"--enable-lag-throttler",
"--throttle_threshold", "1s",
"--heartbeat_enable",
"--heartbeat_interval", "250ms",
"--heartbeat_on_demand_duration", "5s",
"--migration_check_interval", "5s",
Expand Down Expand Up @@ -218,6 +216,9 @@ func TestSchemaChange(t *testing.T) {
defer cluster.PanicHandler(t)
shards = clusterInstance.Keyspaces[0].Shards
assert.Equal(t, 2, len(shards))

throttler.EnableLagThrottlerAndWaitForStatus(t, clusterInstance, time.Second)

testWithInitialSchema(t)
t.Run("create non_online", func(t *testing.T) {
_ = testOnlineDDLStatement(t, alterTableNormalStatement, string(schema.DDLStrategyDirect), "vtctl", "non_online", "")
Expand Down
6 changes: 3 additions & 3 deletions go/test/endtoend/onlineddl/revert/onlineddl_revert_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ import (

"vitess.io/vitess/go/test/endtoend/cluster"
"vitess.io/vitess/go/test/endtoend/onlineddl"
"vitess.io/vitess/go/test/endtoend/throttler"

"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
Expand Down Expand Up @@ -150,9 +151,6 @@ func TestMain(m *testing.M) {
}

clusterInstance.VtTabletExtraArgs = []string{
"--enable-lag-throttler",
"--throttle_threshold", "1s",
"--heartbeat_enable",
"--heartbeat_interval", "250ms",
"--heartbeat_on_demand_duration", "5s",
"--migration_check_interval", "5s",
Expand Down Expand Up @@ -205,6 +203,8 @@ func TestSchemaChange(t *testing.T) {
shards = clusterInstance.Keyspaces[0].Shards
require.Equal(t, 1, len(shards))

throttler.EnableLagThrottlerAndWaitForStatus(t, clusterInstance, time.Second)

t.Run("revertible", testRevertible)
t.Run("revert", testRevert)
}
Expand Down
7 changes: 4 additions & 3 deletions go/test/endtoend/onlineddl/scheduler/onlineddl_scheduler_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ import (

"vitess.io/vitess/go/test/endtoend/cluster"
"vitess.io/vitess/go/test/endtoend/onlineddl"
"vitess.io/vitess/go/test/endtoend/throttler"

"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
Expand Down Expand Up @@ -187,9 +188,6 @@ func TestMain(m *testing.M) {
}

clusterInstance.VtTabletExtraArgs = []string{
"--enable-lag-throttler",
"--throttle_threshold", "1s",
"--heartbeat_enable",
"--heartbeat_interval", "250ms",
"--heartbeat_on_demand_duration", "5s",
"--watch_replication_stream",
Expand Down Expand Up @@ -234,6 +232,9 @@ func TestMain(m *testing.M) {
}

func TestSchemaChange(t *testing.T) {

throttler.EnableLagThrottlerAndWaitForStatus(t, clusterInstance, time.Second)

t.Run("scheduler", testScheduler)
t.Run("singleton", testSingleton)
t.Run("declarative", testDeclarative)
Expand Down
4 changes: 1 addition & 3 deletions go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,8 +171,6 @@ func TestMain(m *testing.M) {
}

clusterInstance.VtTabletExtraArgs = []string{
"--throttler-config-via-topo",
"--heartbeat_enable",
"--heartbeat_interval", "250ms",
"--heartbeat_on_demand_duration", "5s",
"--migration_check_interval", "5s",
Expand Down Expand Up @@ -259,7 +257,7 @@ func TestSchemaChange(t *testing.T) {
err := clusterInstance.WaitForTabletsToHealthyInVtgate()
require.NoError(t, err)

_, err = throttler.UpdateThrottlerTopoConfig(clusterInstance, true, false, 0, "", false)
_, err = throttler.UpdateThrottlerTopoConfig(clusterInstance, true, false, 0, "")
require.NoError(t, err)

for _, ks := range clusterInstance.Keyspaces {
Expand Down
6 changes: 3 additions & 3 deletions go/test/endtoend/onlineddl/vrepl_stress/onlineddl_vrepl_mini_stress_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ import (

"vitess.io/vitess/go/test/endtoend/cluster"
"vitess.io/vitess/go/test/endtoend/onlineddl"
"vitess.io/vitess/go/test/endtoend/throttler"

"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
Expand Down Expand Up @@ -177,9 +178,6 @@ func TestMain(m *testing.M) {
}

clusterInstance.VtTabletExtraArgs = []string{
"--enable-lag-throttler",
"--throttle_threshold", "1s",
"--heartbeat_enable",
"--heartbeat_interval", "250ms",
"--heartbeat_on_demand_duration", "5s",
"--migration_check_interval", "5s",
Expand Down Expand Up @@ -232,6 +230,8 @@ func TestSchemaChange(t *testing.T) {
shards = clusterInstance.Keyspaces[0].Shards
require.Equal(t, 1, len(shards))

throttler.EnableLagThrottlerAndWaitForStatus(t, clusterInstance, time.Second)

t.Run("create schema", func(t *testing.T) {
assert.Equal(t, 1, len(clusterInstance.Keyspaces[0].Shards))
testWithInitialSchema(t)
Expand Down
6 changes: 3 additions & 3 deletions go/test/endtoend/onlineddl/vrepl_stress_suite/onlineddl_vrepl_stress_suite_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ import (

"vitess.io/vitess/go/test/endtoend/cluster"
"vitess.io/vitess/go/test/endtoend/onlineddl"
"vitess.io/vitess/go/test/endtoend/throttler"

"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
Expand Down Expand Up @@ -429,9 +430,6 @@ func TestMain(m *testing.M) {
// thereby examining lastPK on vcopier side. We will be iterating tables using non-PK order throughout
// this test suite, and so the low setting ensures we hit the more interesting code paths.
clusterInstance.VtTabletExtraArgs = []string{
"--enable-lag-throttler",
"--throttle_threshold", "1s",
"--heartbeat_enable",
"--heartbeat_interval", "250ms",
"--heartbeat_on_demand_duration", "5s",
"--migration_check_interval", "5s",
Expand Down Expand Up @@ -485,6 +483,8 @@ func TestSchemaChange(t *testing.T) {
shards = clusterInstance.Keyspaces[0].Shards
require.Equal(t, 1, len(shards))

throttler.EnableLagThrottlerAndWaitForStatus(t, clusterInstance, time.Second)

for _, testcase := range testCases {
require.NotEmpty(t, testcase.name)
t.Run(testcase.name, func(t *testing.T) {
Expand Down
6 changes: 3 additions & 3 deletions go/test/endtoend/onlineddl/vrepl_suite/onlineddl_vrepl_suite_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ import (

"vitess.io/vitess/go/test/endtoend/cluster"
"vitess.io/vitess/go/test/endtoend/onlineddl"
"vitess.io/vitess/go/test/endtoend/throttler"

"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
Expand Down Expand Up @@ -82,9 +83,6 @@ func TestMain(m *testing.M) {
}

clusterInstance.VtTabletExtraArgs = []string{
"--enable-lag-throttler",
"--throttle_threshold", "1s",
"--heartbeat_enable",
"--heartbeat_interval", "250ms",
"--heartbeat_on_demand_duration", "5s",
"--migration_check_interval", "5s",
Expand Down Expand Up @@ -134,6 +132,8 @@ func TestSchemaChange(t *testing.T) {
shards := clusterInstance.Keyspaces[0].Shards
require.Equal(t, 1, len(shards))

throttler.EnableLagThrottlerAndWaitForStatus(t, clusterInstance, time.Second)

files, err := os.ReadDir(testDataPath)
require.NoError(t, err)
for _, f := range files {
Expand Down
7 changes: 4 additions & 3 deletions go/test/endtoend/schemadiff/vrepl/schemadiff_vrepl_suite_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ import (
"regexp"
"strings"
"testing"
"time"

"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
Expand All @@ -33,6 +34,7 @@ import (
"vitess.io/vitess/go/sqltypes"
"vitess.io/vitess/go/test/endtoend/cluster"
"vitess.io/vitess/go/test/endtoend/onlineddl"
"vitess.io/vitess/go/test/endtoend/throttler"
"vitess.io/vitess/go/vt/schemadiff"
"vitess.io/vitess/go/vt/sqlparser"
)
Expand Down Expand Up @@ -87,9 +89,6 @@ func TestMain(m *testing.M) {
}

clusterInstance.VtTabletExtraArgs = []string{
"--enable-lag-throttler",
"--throttle_threshold", "1s",
"--heartbeat_enable",
"--heartbeat_interval", "250ms",
"--heartbeat_on_demand_duration", "5s",
"--migration_check_interval", "5s",
Expand Down Expand Up @@ -139,6 +138,8 @@ func TestSchemaChange(t *testing.T) {
shards := clusterInstance.Keyspaces[0].Shards
require.Equal(t, 1, len(shards))

throttler.EnableLagThrottlerAndWaitForStatus(t, clusterInstance, time.Second)

files, err := os.ReadDir(testDataPath)
require.NoError(t, err)
for _, f := range files {
Expand Down
1 change: 0 additions & 1 deletion go/test/endtoend/tabletmanager/tablegc/tablegc_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,6 @@ func TestMain(m *testing.M) {
"--lock_tables_timeout", "5s",
"--watch_replication_stream",
"--enable_replication_reporter",
"--heartbeat_enable",
"--heartbeat_interval", "250ms",
"--gc_check_interval", gcCheckInterval.String(),
"--gc_purge_check_interval", gcPurgeCheckInterval.String(),
Expand Down
2 changes: 1 addition & 1 deletion go/test/endtoend/tabletmanager/throttler/throttler_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,12 +96,12 @@ func TestMain(m *testing.M) {

// Set extra tablet args for lock timeout
clusterInstance.VtTabletExtraArgs = []string{
"--throttler-config-via-topo=false",
"--lock_tables_timeout", "5s",
"--watch_replication_stream",
"--enable_replication_reporter",
"--enable-lag-throttler",
"--throttle_threshold", throttlerThreshold.String(),
"--heartbeat_enable",
"--heartbeat_interval", "250ms",
"--heartbeat_on_demand_duration", onDemandHeartbeatDuration.String(),
"--disable_active_reparents",
Expand Down
2 changes: 1 addition & 1 deletion go/test/endtoend/tabletmanager/throttler_custom_config/throttler_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,14 +98,14 @@ func TestMain(m *testing.M) {

// Set extra tablet args for lock timeout
clusterInstance.VtTabletExtraArgs = []string{
"--throttler-config-via-topo=false",
"--lock_tables_timeout", "5s",
"--watch_replication_stream",
"--enable_replication_reporter",
"--enable-lag-throttler",
"--throttle_metrics_query", "show global status like 'threads_running'",
"--throttle_metrics_threshold", fmt.Sprintf("%d", testThreshold),
"--throttle_check_as_check_self",
"--heartbeat_enable",
"--heartbeat_interval", "250ms",
}

Expand Down
Loading