tidb panic after inject one of AZ network partition with pitr running

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、run pitr
2、run tpcc with 2w warehouse and 32 threads
3、inject network partition between one of az and other az last for 10mins
fault start time：2024-04-12 23:16:25
fault end time：2024-04-12 23:26:32

2. What did you expect to see? (Required)

no panic

3. What did you see instead (Required)

tidb panic after fault recover

2024-04-12 23:27:44 log="/tidb-server --store=tikv --advertise-address=tc-tidb-1.tc-tidb-peer.endless-ha-test-oltp-pitr-tps-7504383-1-172.svc --host=0.0.0.0 --path=tc-pd:2379 --config=/etc/tidb/tidb.toml\n" 2024-04-12 23:27:44 log="start tidb-server ...\n" 2024-04-12 23:27:33 log="\t/workspace/source/tidb/br/pkg/streamhelper/flush_subscriber.go:150 +0x78\n" 2024-04-12 23:27:33 log="created by github.com/pingcap/tidb/br/pkg/streamhelper.spawnJoinable in goroutine 16842\n" 2024-04-12 23:27:33 log="\t/workspace/source/tidb/br/pkg/streamhelper/flush_subscriber.go:152 +0x4d\n" 2024-04-12 23:27:33 log="github.com/pingcap/tidb/br/pkg/streamhelper.spawnJoinable.func1()\n" 2024-04-12 23:27:33 log="\t/workspace/source/tidb/br/pkg/streamhelper/flush_subscriber.go:237 +0x27\n" 2024-04-12 23:27:33 log="github.com/pingcap/tidb/br/pkg/streamhelper.(*subscription).doConnect.func1()\n" 2024-04-12 23:27:33 log="\t/workspace/source/tidb/br/pkg/streamhelper/flush_subscriber.go:280 +0xac5\n" 2024-04-12 23:27:33 log="github.com/pingcap/tidb/br/pkg/streamhelper.(*subscription).listenOver(0xc0b24bf600, {0x66287d8, 0xc086a1e420}, {0x663d180, 0xc057c9afc0})\n" 2024-04-12 23:27:33 log="goroutine 1090341181 [running]:\n" 2024-04-12 23:27:33 log="\n" 2024-04-12 23:27:33 log="panic: send on closed channel\n"

4. What is your TiDB version? (Required)

./tidb-server -V
Release Version: v8.1.0
Edition: Community
Git Commit Hash: 6bf6755
Git Branch: HEAD
UTC Build Time: 2024-04-12 11:10:02
GoVersion: go1.21.4
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2024-04-12T23:06:17.157+0800

[Lily2025]

/assign YuJuncen
/component backup-restore

[ti-chi-bot]

@Lily2025: The label(s) component/backup-restore cannot be applied, because the repository doesn't have them.

In response to this:

/assign YuJuncen
/component backup-restore

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[YuJuncen]

Could you upload the logs from subscription_manager.go and advancer*.go?