Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ticdc: version 8.3.0 of TiCDC is not compatible with version 8.1.0 of TiKV #11507

Closed
wlwilliamx opened this issue Aug 22, 2024 · 1 comment · Fixed by #11580
Closed

ticdc: version 8.3.0 of TiCDC is not compatible with version 8.1.0 of TiKV #11507

wlwilliamx opened this issue Aug 22, 2024 · 1 comment · Fixed by #11580
Labels
area/ticdc Issues or PRs related to TiCDC. severity/moderate type/bug The issue is confirmed as a bug.

Comments

@wlwilliamx
Copy link
Contributor

wlwilliamx commented Aug 22, 2024

What did you do?

Description:
While performing integration testing with TiCDC's master branch (version 8.3.0), I encountered an issue where the batch_add_table test fails. The TiCDC node crashes because the puller module panics during DDL execution.

Steps to Reproduce:

  1. Run make prepare_test_binaries community=true to download TiDB, TiKV, PD, and other components. This pulls version 8.1.0 of the components (I'll explain why below).
  2. Execute make integration_test_build.
  3. Execute make integration_test to start the integration test.

Observed Behavior:
During the batch_add_table integration test, TiCDC's puller module panics when processing DDL, causing the TiCDC node to crash.

Cause Analysis:
The reason TiCDC 8.3.0 pulled version 8.1.0 of TiDB, TiKV, PD, etc., is that when running make prepare_test_binaries with the community=true option, if the version is not manually specified, it defaults to downloading version 8.1.0 of all TiDB components. (related to #11508)

After manually replacing the TiDB, TiKV, PD components with the latest versions, the batch_add_table integration test passed successfully. This indicates that TiCDC 8.3.0 currently does not maintain backward compatibility with TiKV 8.1.0.

What did you expect to see?

I expected the batch_add_table integration test to pass without any issues when using TiCDC 8.3.0 with TiDB, TiKV, and PD components (even if they are from version 8.1.0). The system should handle version differences gracefully, or at the very least, the prepare_test_binaries step should automatically pull compatible versions of all components.

What did you see instead?

img_v3_02e0_1c59243e-9039-4edf-bbf7-7592e91e009g

panic: runtime error: index out of range [1] with length 1

goroutine 809 [running]:
[github.com/pingcap/tiflow/cdc/puller.(*ddlJobPullerImpl).handleJob](http://github.com/pingcap/tiflow/cdc/puller.(*ddlJobPullerImpl).handleJob)(0xc0040b7980, 0xc001756820)
        /root/tiflow/cdc/puller/ddl_puller.go:410 +0x4c92
[github.com/pingcap/tiflow/cdc/puller.(*ddlJobPullerImpl).handleRawKVEntry](http://github.com/pingcap/tiflow/cdc/puller.(*ddlJobPullerImpl).handleRawKVEntry)(0xc0040b7980, {0xbde3480, 0xc0031b5db0}, 0xc00123b1f0)
        /root/tiflow/cdc/puller/ddl_puller.go:206 +0x35c
[github.com/pingcap/tiflow/cdc/puller.(*ddlJobPullerImpl).Run.func3()](http://github.com/pingcap/tiflow/cdc/puller.(*ddlJobPullerImpl).Run.func3())
        /root/tiflow/cdc/puller/ddl_puller.go:153 +0x218
[golang.org/x/sync/errgroup.(*Group).Go.func1()](http://golang.org/x/sync/errgroup.(*Group).Go.func1())
        /root/go/pkg/mod/[golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78](http://golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78) +0x92
created by [golang.org/x/sync/errgroup.(*Group).Go](http://golang.org/x/sync/errgroup.(*Group).Go) in goroutine 805
        /root/go/pkg/mod/[golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75](http://golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75) +0x125
~

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

8.1.0

Upstream TiKV version (execute tikv-server --version):

8.1.0

TiCDC version (execute cdc version):

8.3.0
@fubinzh
Copy link

fubinzh commented Aug 23, 2024

/severity moderate

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/ticdc Issues or PRs related to TiCDC. severity/moderate type/bug The issue is confirmed as a bug.
Projects
2 participants