Dynamic tablet throttler config: enable/disable, set metrics query/th…

…reshold (#11604) * Support syntax: ALTER VITESS_MIGRATION {DISABLE|ENABLE|THRESHOLD='...') Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * vtgate supports ALTER VITESS_THROTTLER syntax Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * initial support for AlterThrottler in query executor Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * towards enable/disable in throttler Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * no cleanup ticker for caches, single initialization Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * init metrics query and threshold (temporary) Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * replace if-else with switch Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * parser: support 'show vitess_throttler status' command Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * vtgate: support 'show vitess_throttler status' command Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * implement SHOW VITESS_THROTTLER STATUS Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * ugly quick hack for reducing flakiness Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * undo hack Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * IsEnabled() now implies IsOpen Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * THRESHOLD is decimal/integer Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * experimental, accessing topo for config Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * make proto Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * store throttler config is SrvKeyspace in all cells Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * apply throttler config Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * AlterThrottler does not get handled by QueryExecutor Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * no schema, removed file Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * unsupport ALTER VITESS_THROTTLER syntax Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * repltracker enables heartbeats if ondemand is positive Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * throttler does not requests heartbeats when disabled Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * endtoend test to cover throttler config by topo Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * make generate_ci_workflows Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * graceful sleep Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * Enable throttler in Open() when not reading from topo (maintain backwards compatibility) Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * improve test logic with WaitGroup Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * adapt test logic to new on-demand behavior. Add more test cases Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * removed AlterThrottler from vtgate Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * removed AlterThrottler from ast Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * update test Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * test custom config Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * validate return from custom query to default query Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * updated workflow Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * update endtoend cli docs Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * update endtoend cli docs: --throttle_config_via_topo Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * vtadmin web proto Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * s/throttle_config_via_topo/throttler_config_via_topo Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * do not listen on KeyspaceSrv if not configured to read from topo Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * reverse --throttle_threshold default increase Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * dashes: --throttler-config-via-topo Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * throttler check: flag for skipping heartbeat request (avoids renewing lease) Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * wait-based status checks Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * wait-based TestLag Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * reorganizing, cleanup Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * remove excessive sleep Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * vtctl subflags use dashes Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * validate mutually exclusive flags Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * fix cli test Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * empty commit to kick CI Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * disable active reparents on tests Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * do not return list of cells. Instead, just return exit code Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * improve error message Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * typo Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * copyright year Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * vtctldclient: support UpdateThrottlerConfig Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * vt admin proto Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * refactored throttler code to its own file Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * simplify: options are of UpdateThrottlerConfigRequest type Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * added keyspace Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * implementing server-side UpdateThrottlerConfig Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * validating that vtctldclient works Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * copyright year Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * safer Open() flow that does not block execution Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * expected CLI Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * vt admin proto Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * CustomQuerySet: identify existence of flag Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * vt admin proto Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * update ci workflow Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * go mod tidy Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * check 'Changed' after parsing Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * more verbose message; do give throttler timt to update config Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * metricsQuery is atomic; report metrics query in SHOW VITESS_THROTTLER STATUS Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * cobra comment Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * goimport Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * fix CLI docs Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * release notes Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
vitessio · Nov 20, 2022 · 9fa7784 · 9fa7784
1 parent 1d61c11
commit 9fa7784
Show file tree

Hide file tree

Showing 49 changed files with 13,070 additions and 9,485 deletions.
diff --git a/.github/workflows/cluster_endtoend_tabletmanager_throttler_topo.yml b/.github/workflows/cluster_endtoend_tabletmanager_throttler_topo.yml
@@ -0,0 +1,134 @@
+# DO NOT MODIFY: THIS FILE IS GENERATED USING "make generate_ci_workflows"
+
+name: Cluster (tabletmanager_throttler_topo)
+on: [push, pull_request]
+concurrency:
+  group: format('{0}-{1}', ${{ github.ref }}, 'Cluster (tabletmanager_throttler_topo)')
+  cancel-in-progress: true
+
+env:
+  LAUNCHABLE_ORGANIZATION: "vitess"
+  LAUNCHABLE_WORKSPACE: "vitess-app"
+  GITHUB_PR_HEAD_SHA: "${{ github.event.pull_request.head.sha }}"
+
+jobs:
+  build:
+    name: Run endtoend tests on Cluster (tabletmanager_throttler_topo)
+    runs-on: ubuntu-20.04
+
+    steps:
+    - name: Skip CI
+      run: |
+        if [[ "${{contains( github.event.pull_request.labels.*.name, 'Skip CI')}}" == "true" ]]; then
+          echo "skipping CI due to the 'Skip CI' label"
+          exit 1
+        fi
+
+    - name: Check if workflow needs to be skipped
+      id: skip-workflow
+      run: |
+        skip='false'
+        if [[ "${{github.event.pull_request}}" ==  "" ]] && [[ "${{github.ref}}" != "refs/heads/main" ]] && [[ ! "${{github.ref}}" =~ ^refs/heads/release-[0-9]+\.[0-9]$ ]] && [[ ! "${{github.ref}}" =~ "refs/tags/.*" ]]; then
+          skip='true'
+        fi
+        echo Skip ${skip}
+        echo "::set-output name=skip-workflow::${skip}"
+
+    - name: Check out code
+      if: steps.skip-workflow.outputs.skip-workflow == 'false'
+      uses: actions/checkout@v3
+
+    - name: Check for changes in relevant files
+      if: steps.skip-workflow.outputs.skip-workflow == 'false'
+      uses: frouioui/paths-filter@main
+      id: changes
+      with:
+        token: ''
+        filters: |
+          end_to_end:
+            - 'go/**/*.go'
+            - 'test.go'
+            - 'Makefile'
+            - 'build.env'
+            - 'go.[sumod]'
+            - 'proto/*.proto'
+            - 'tools/**'
+            - 'config/**'
+            - 'bootstrap.sh'
+            - '.github/workflows/cluster_endtoend_tabletmanager_throttler_topo.yml'
+
+    - name: Set up Go
+      if: steps.skip-workflow.outputs.skip-workflow == 'false' && steps.changes.outputs.end_to_end == 'true'
+      uses: actions/setup-go@v3
+      with:
+        go-version: 1.19.3
+
+    - name: Set up python
+      if: steps.skip-workflow.outputs.skip-workflow == 'false' && steps.changes.outputs.end_to_end == 'true'
+      uses: actions/setup-python@v4
+
+    - name: Tune the OS
+      if: steps.skip-workflow.outputs.skip-workflow == 'false' && steps.changes.outputs.end_to_end == 'true'
+      run: |
+        echo '1024 65535' | sudo tee -a /proc/sys/net/ipv4/ip_local_port_range
+        # Increase the asynchronous non-blocking I/O. More information at https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_use_native_aio
+        echo "fs.aio-max-nr = 1048576" | sudo tee -a /etc/sysctl.conf
+        sudo sysctl -p /etc/sysctl.conf
+
+    - name: Get dependencies
+      if: steps.skip-workflow.outputs.skip-workflow == 'false' && steps.changes.outputs.end_to_end == 'true'
+      run: |
+        
+        # Get key to latest MySQL repo
+        sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 467B942D3A79BD29
+        # Setup MySQL 8.0
+        wget -c https://dev.mysql.com/get/mysql-apt-config_0.8.20-1_all.deb
+        echo mysql-apt-config mysql-apt-config/select-server select mysql-8.0 | sudo debconf-set-selections
+        sudo DEBIAN_FRONTEND="noninteractive" dpkg -i mysql-apt-config*
+        sudo apt-get update
+        # Install everything else we need, and configure
+        sudo apt-get install -y mysql-server mysql-client make unzip g++ etcd curl git wget eatmydata xz-utils
+
+        sudo service mysql stop
+        sudo service etcd stop
+        sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
+        sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
+        go mod download
+
+        # install JUnit report formatter
+        go install github.com/vitessio/go-junit-report@HEAD
+
+    - name: Setup launchable dependencies
+      if: steps.skip-workflow.outputs.skip-workflow == 'false' && steps.changes.outputs.end_to_end == 'true'
+      run: |
+        # Get Launchable CLI installed. If you can, make it a part of the builder image to speed things up
+        pip3 install --user launchable~=1.0 > /dev/null
+
+        # verify that launchable setup is all correct.
+        launchable verify || true
+
+        # Tell Launchable about the build you are producing and testing
+        launchable record build --name "$GITHUB_RUN_ID" --source .
+
+    - name: Run cluster endtoend test
+      if: steps.skip-workflow.outputs.skip-workflow == 'false' && steps.changes.outputs.end_to_end == 'true'
+      timeout-minutes: 45
+      run: |
+        # We set the VTDATAROOT to the /tmp folder to reduce the file path of mysql.sock file
+        # which musn't be more than 107 characters long.
+        export VTDATAROOT="/tmp/"
+        source build.env
+
+        set -x
+
+        # run the tests however you normally do, then produce a JUnit XML file
+        eatmydata -- go run test.go -docker=false -follow -shard tabletmanager_throttler_topo | tee -a output.txt | go-junit-report -set-exit-code > report.xml
+
+    - name: Print test output and Record test result in launchable
+      if: steps.skip-workflow.outputs.skip-workflow == 'false' && steps.changes.outputs.end_to_end == 'true' && always()
+      run: |
+        # send recorded tests to launchable
+        launchable record tests --build "$GITHUB_RUN_ID" go-test . || true
+
+        # print test output
+        cat output.txt
diff --git a/doc/releasenotes/16_0_0_summary.md b/doc/releasenotes/16_0_0_summary.md
@@ -15,6 +15,14 @@
 
 In [PR #11103](https://github.com/vitessio/vitess/pull/11103) we introduced the ability to resume a `VTGate` [`VStream` copy operation](https://vitess.io/docs/design-docs/vreplication/vstream/vscopy/). This is useful when a [`VStream` copy operation](https://vitess.io/docs/design-docs/vreplication/vstream/vscopy/) is interrupted due to e.g. a network failure or a server restart. The `VStream` copy operation can be resumed by specifying each table's last seen primary key value in the `VStream` request. Please see the [`VStream` docs](https://vitess.io/docs/16.0/reference/vreplication/vstream/) for more details.
 
+### Tablet throttler
+
+The tablet throttler can now be configured dynamically. Configuration is now found in the topo service, and applies to all tablets in all shards and cells of a given keyspace. For backwards compatibility `v16` still supports `vttablet`-based command line flags for throttler ocnfiguration.
+
+It is possible to enable/disable, to change throttling threshold as well as the throttler query.
+
+See https://github.com/vitessio/vitess/pull/11604
+
 ### Breaking Changes
 
 #### Orchestrator Integration Deletion
@@ -43,6 +51,37 @@ Other aspects of the VReplication copy-phase logic are preserved:
 
  Other phases, catchup, fast-forward, and replicating/"running", are unchanged.
 
+### vttablet --throttler-config-via-topo
+
+The flag `--throttler-config-via-topo` switches throttler configuration from `vttablet`-flags to the topo service. This flag is `false` by default, for backwards compatibility. It will default to `true` in future versions.
+
+### vtctldclient UpdateThrottlerConfig
+
+Tablet throttler configuration is now supported in `topo`. Updating the throttler configuration is done via `vtctldclient UpdateThrottlerConfig` and applies to all tablet in all cells for a given keyspace.
+
+Examples:
+```shell
+# disable throttler; all throttler checks will return with "200 OK"
+$ vtctldclient UpdateThrottlerConfig --disable commerce
+
+# enable throttler; checks are responded with appropriate status per current metrics
+$ vtctldclient UpdateThrottlerConfig --enable commerce
+
+# Both enable and set threshold in same command. Since no query is indicated, we assume the default check for replication lag
+$ vtctldclient UpdateThrottlerConfig --enable --threshold 5.0 commerce
+
+# Change threshold. Does not affect enabled/disabled state of the throttler
+$ vtctldclient UpdateThrottlerConfig --threshold 1.5 commerce
+
+# Use a custom query
+$ vtctldclient UpdateThrottlerConfig --custom_query "show global status like 'threads_running'" --check_as_check_self --threshold 50 commerce
+
+# Restore default query and threshold
+$ vtctldclient UpdateThrottlerConfig --custom_query "" --check_as_check_shard --threshold 1.5 commerce
+```
+
+See https://github.com/vitessio/vitess/pull/11604
+
 ### Important bug fixes
 
 #### Corrupted results for non-full-group-by queries with JOINs

diff --git a/go/cmd/vtctldclient/command/throttler.go b/go/cmd/vtctldclient/command/throttler.go
@@ -0,0 +1,61 @@
+/*
+Copyright 2022 The Vitess Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+	http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package command
+
+import (
+	"github.com/spf13/cobra"
+
+	"vitess.io/vitess/go/cmd/vtctldclient/cli"
+
+	vtctldatapb "vitess.io/vitess/go/vt/proto/vtctldata"
+)
+
+var (
+	// UpdateThrottlerConfig makes a UpdateThrottlerConfig gRPC call to a vtctld.
+	UpdateThrottlerConfig = &cobra.Command{
+		Use:                   "UpdateThrottlerConfig [--enable|--disable] [--threshold=<float64>] [--custom-query=<query>] [--check-as-check-self|--check-as-check-shard] <keyspace>",
+		Short:                 "Update the table throttler configuration for all cells and tablets of a given keyspace",
+		DisableFlagsInUseLine: true,
+		Args:                  cobra.ExactArgs(1),
+		RunE:                  commandUpdateThrottlerConfig,
+	}
+)
+
+var updateThrottlerConfigOptions vtctldatapb.UpdateThrottlerConfigRequest
+
+func commandUpdateThrottlerConfig(cmd *cobra.Command, args []string) error {
+	keyspace := cmd.Flags().Arg(0)
+	cli.FinishedParsing(cmd)
+
+	updateThrottlerConfigOptions.CustomQuerySet = cmd.Flags().Changed("custom-query")
+	updateThrottlerConfigOptions.Keyspace = keyspace
+	_, err := client.UpdateThrottlerConfig(commandCtx, &updateThrottlerConfigOptions)
+	if err != nil {
+		return err
+	}
+	return nil
+}
+
+func init() {
+	UpdateThrottlerConfig.Flags().BoolVar(&updateThrottlerConfigOptions.Enable, "enable", false, "Enable the throttler")
+	UpdateThrottlerConfig.Flags().BoolVar(&updateThrottlerConfigOptions.Disable, "disable", false, "Disable the throttler")
+	UpdateThrottlerConfig.Flags().Float64Var(&updateThrottlerConfigOptions.Threshold, "threshold", 0, "threshold for the either default check (replication lag seconds) or custom check")
+	UpdateThrottlerConfig.Flags().StringVar(&updateThrottlerConfigOptions.CustomQuery, "custom-query", "", "custom throttler check query")
+	UpdateThrottlerConfig.Flags().BoolVar(&updateThrottlerConfigOptions.CheckAsCheckSelf, "check-as-check-self", false, "/throttler/check requests behave as is /throttler/check-self was called")
+	UpdateThrottlerConfig.Flags().BoolVar(&updateThrottlerConfigOptions.CheckAsCheckShard, "check-as-check-shard", false, "use standard behavior for /throttler/check requests")
+	Root.AddCommand(UpdateThrottlerConfig)
+}
diff --git a/go/flags/endtoend/vtctldclient.txt b/go/flags/endtoend/vtctldclient.txt
@@ -79,6 +79,7 @@ Available Commands:
   TabletExternallyReparented  Updates the topology record for the tablet's shard to acknowledge that an external tool made this tablet the primary.
   UpdateCellInfo              Updates the content of a CellInfo with the provided parameters, creating the CellInfo if it does not exist.
   UpdateCellsAlias            Updates the content of a CellsAlias with the provided parameters, creating the CellsAlias if it does not exist.
+  UpdateThrottlerConfig       Update the table throttler configuration for all cells and tablets of a given keyspace
   Validate                    Validates that all nodes reachable from the global replication graph, as well as all tablets in discoverable cells, are consistent.
   ValidateKeyspace            Validates that all nodes reachable from the specified keyspace are consistent.
   ValidateSchemaKeyspace      Validates that the schema on the primary tablet for shard 0 matches the schema on all other tablets in the keyspace.

diff --git a/go/flags/endtoend/vttablet.txt b/go/flags/endtoend/vttablet.txt
@@ -302,6 +302,7 @@ Usage of vttablet:
       --throttle_metrics_threshold float                                 Override default throttle threshold, respective to -throttle_metrics_query (default 1.7976931348623157e+308)
       --throttle_tablet_types string                                     Comma separated VTTablet types to be considered by the throttler. default: 'replica'. example: 'replica,rdonly'. 'replica' aways implicitly included (default "replica")
       --throttle_threshold duration                                      Replication lag threshold for default lag throttling (default 1s)
+      --throttler-config-via-topo                                        When 'true', read config from topo service and ignore throttle_threshold, throttle_metrics_threshold, throttle_metrics_query, throttle_check_as_check_self
       --topo_consul_lock_delay duration                                  LockDelay for consul session. (default 15s)
       --topo_consul_lock_session_checks string                           List of checks for consul session. (default "serfHealth")
       --topo_consul_lock_session_ttl string                              TTL for consul session.

diff --git a/go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go b/go/test/endtoend/onlineddl/vrepl/onlineddl_vrepl_test.go
@@ -546,7 +546,6 @@ func TestSchemaChange(t *testing.T) {
 				err := clusterInstance.VtctlclientProcess.ExecuteCommand("PlannedReparentShard", "--", "--keyspace_shard", keyspaceName+"/-80", "--new_primary", shards[0].Vttablets[reparentTabletIndex].Alias)
 				require.NoError(t, err, "failed PRS: %v", err)
 			})
-
 			t.Run("unthrottle and expect completion", func(t *testing.T) {
 				for i := range shards {
 					var body string