checksum mismatch when restored backup with checksum v2 and table id changes

[fubinzh]

What did you do?

1. create testbed, upstream tidb with ticdc, downstream tidb, kafka cluster
2. run same backup with 32 data tables and 32 index tables to upstream and downstreasm tidb
3. enable rowchecksum on upstream and downstream tidb (set global tidb_enable_row_level_checksum=true)
4. Create kafka simple protocol changefeed
5. run consumer to consumer kafka message to downstream tidb
6. Run workload, workloads contains:
   1): insert and update
   2): add column/index for index table, by: creating index_v2 table with new column/index, copy data from index to index_v2, drop index table, rename index_v2 to index
7. stop workload, wait all data sync to downstream tidb, and do upstream and downstream data consistency check => pass
8. clean the environment. (drop the database from upstream and downstream tidb, delete changefeed, stop consumer.)
9. repeat the testing, restore the backup, create changefeed again, and run workload, this time only run insert and update workload. (add column/index workload not execute.)

What did you expect to see?

after step 9, cdc changefeed not run into checksum mismatch error.

What did you see instead?

CDC changefeed run into checksum mismatch error:

[2024/11/04 16:25:40.795 +08:00] [ERROR] [mounter.go:642] ["raw bytes checksum mismatch"] [expected=1621378861] [obtained=400491149] [tableInfo="TableInfo, ID: 1853, Name:uxx2.Data9, ColNum: 4, IdxNum: 0, PKIsHa
ndle: true"] [columns="[{\"column_id\":1,\"value\":1459507293},{\"column_id\":2,\"value\":45415},{\"column_id\":3,\"value\":\"dnV6YWd0eWJnemh1cXZ6YnZmYnR2ZXdhZ2VldXl5dWhjeGJlc2lhc3FmbWxweW16bmdrb253bnphZmF2cHFmY
mZqeHdjc2JqbWtza3VkY2ZjYmZwcnJ6aGdibGV0cmRwZXB2Z2thbGd2YWttbnJ1ZGhtanRsbmV3aXNyeHV5cWx3ZXBzaWRld3psdnpzbWNia3VnZnh5cGhydXBkcGtzYnhpZGtocmNzcGtsempicXZkZ3F5dXpiY21peWJlaW14dnVkbWpmcWV0aWNjZXN4ZmFtaHp2cHR3YWdsbHVq
eHd2aGpsd3VpdXljb2RtYWVwcm9uZW1laXlzb29ra2h1bXFleW1ya2NydWxzeWFtZHJpcGxodWlxY3pudG1vbmRtbWhmY2VoaWh5cHp6emp5bnh1bGRxam9ibm5teWNneHhoa29ka2tycWpscWZkbHFzYWpub3dmcWh1Z3V6anBiaHJtcnBrZnFsdnNpbW9uenNic2VmbGlteGx3eHh
3dGp0cXVkb3hmYXVubWVicXB6YWp0ZGFzbHNpeHhmY2pyaWZtZ3ljd3Nvbnp4ZGRxa3hibXl3aWN5dGZhamV3aW5pdHJhZ2dyY3RmeHNxem92aWtyb3prdXd1Z3F6bHBndmZ4dmdlcXFxeHpmamZ5ZndnendmeWxuYnl2cXVtdHR6cGxhYWZ6anBha3dzY3NscXh2cGF6YXZ2Z2phYX
R4emV6bWF1bnh5cWN4anJpaXdodnhqc2d6bGRoYmtuZHBjYXBqeXJiZnBsa2hmaHlienZhbWlsYWluc2JrY3V1bmJ6d2VzaGFidXZsbGdrZmp5dWd5c2dsbHJmbHptYXJ6YmRjYXppaXl6ZmJybHFpeXZrcXJuaW94aGhhcnJ5bGNleHhrdGFhbWdwYXRpZ3FwZmtkbmdwZWF3a2J2Z
nF1ZXB6amF4cmV3dm9tbGJtenZ4bmRrdHpxcmRzaHZrbHRpZG5qanNydGxrZHRneHFra3ZtZ3VmeXBxYWRuc2x4ZmhucGxseWFodmN0YmV4dW93amF6enN1bnZkYmpmbmljZnpleHlraXhsdXd2cWVyZWVicXpnamJ1aWF2aW5qbG1oc3hobmx5ZWFpemtoeWdpdHl1bnZlcHdwbmF5
ZWxsaG1mZ3Rtdmtpc21uZ2dhcGZpbWxpaXlyZ2dkb3J5dXJ3ZW5yaHB1ZHF5c3FxcnJybGJ5bXdnaWlkeWlpbWNqZ2RvY2Zxcnhib3ZtaG1jbndqcXBvY2hlcHdhZ2hsY25lZGJkZGFpcWxta3d0cG9sam9nb3dhdmRsbHRieGxxamVieGl1em1lbXFsenpqa2pqbm9zcWZmdmN5cnp
wdWxwb3FyZGdxZ3F3YnF1Y3RjeGJsd3lhY3htcG93cmt5YWZyamZscWZneXphaXF3bW10dm91a2luZmlzbXZ6cWJweXBsdHd4ZHpubXRwZ3hma2Z6eGVyanJhYm9oaGRidG53ZWlrYXZlZGh5dmVpbmJxZmtna2FreHNodXRmYmF6bG9nYnJxeHd1cXVib3lsanFyc3JscXhxZXpien
B1Y3RscW9ibWRzZGVmZnd6dWtxZ2txcGpzdGJ1d2RvdGFydHlmYW9ybXl5bXFubWtocnppd3d0b2Fjb252YXplb2JyYWRxenJldW52andra3hoYXpvZWxtbXF1cGNyYmV0dnpzaWFndm1zcmRwZWd1aXl0bHByd2hhb2Z4eWd3Zm90aG1mZHpkYXFwZ2VzdWdudWRlcXhhaXZlZnJrc
HBvc2V1a3pld3FsY3R4c3VkZG5pbmNnaXVsYnJ6Z3B6b2V4cWllaGNyd2Zhb3JlanB1Y3JkdmNiZ3VrZG95bWZ6eWl0dHFidWRleGFpZ2h0eXJkb2VxYXN3aXlpdm1scm91dm9ianVnZmd4eGtveXh5c21yZGNxbHNyYmpwYWNsaWtwbXVlZHVnaGVucGpta2xpaHhtcW1lbm5ieHFu
Z3Vic2h1ZGlrcW9panFrd3VndWdhamlhdG50c3duemtlbnVhaXpzZmJldHV6aHl4bmxsbG15eHNsaXFwcnJ3aGJnbm5zZG90Zm9senF6bGZmb2dzYXpudWp5Y2hwaXNkcmRhZXF2d2JobWFuZW1wdXh5YWt4bXR0Y3pjcWNsemh3cHN6bnByYnBleXRxZmRmeWd5c2V2b2Npb3F2dHJ
mZWJkbHRka3FrZHBvaGVqeW15Z29leGpkaW5qamZ0eHdybWZ1eGhzdG5ncnZiaHR4YmpkY2xkcm5iZWd1bG15bndrcWRodXBjcmh3ZnhxeG5uaXNqdWNyY2dsbXRobm50c3NhZG9zaGpucGRra3ljamhrdXFraXl3cmtpdWZyc2h2aHpna2Vpd2JlcmNuYnRyZmtqbXZpdmVnc2hnen
JrcW1jd2V1YWNkbXl3b3FueXZkZGtmeHVlZGZmYnZlenl4Ymtvem5pbWhjYXNlZnZpZnNlcmRsdmFkaXZvbmthbmFvaGVzeWhma2lpZnRodmxnc2hpdHZxb21pd2p3Z2Nqc3F0a3B1ampmcXhsendvZGlubXViZ2VobWZtbXltd2hvdm95eW16eXhpZ3pha2p5bmpldHdwd2pkdm52b
nZzYnV5cGt3YXhlc2d3cm9jc2hjd3FqZ3NzZ2R6cGFjdHpwcnlqYXViaWJjbWJsaGN4a2FrbW9ibXJ2amFoa294Y3drcWhhZ3lzbHlqYmh4ZXN3eGJ3cWdnbG1tanNpYmR6bHVscWt6bWp4Y3lieWFlYW9mZnpxemF3a3hxeHhqZ2pmaWtseWVmY2piZm52ZmdoaWxxbmxuZWp5YnR6
ZXRneXptY2VtYXltc2R4d2d1Z2Zyb3F6bXpkeGF3ZWVqemxwbmVlZ3B4Z3huYWJ1cXBkcml1dXVmZWVpbW9kd2FvaWZrdW9ldWl3Z21yaGpmYmZpdmFzaXVnenRsZ2tod3phdXBlaGhuYmRmbWl4bGVydXphanFhZWF0aXFqd3pleXV0ZmxndnRkZHVtZXp6ZnRycG13eHpobHRqZml
uYXBudHBvaWF3dXNkaGRpaW9zeW1xbXlmZm9wZHdvYnF1dXRva2p4Y2J4eWd0cmZrc3B1Z2xwbWdtZXh0bm1pcmFodWl4d2x6cmN2anp1cGlqbXliZXljY3hsZWNxcW1seGtrZ2N3YnVjdmZqbGpod2Z0dHBmcGVwcXh4cGVkc3hlYWpqa2doYXBsdG1ldWZwc3Nod2N0Znh3aWdveG
h0dWZtaXpka2huamhnZ2JranhiZnh5d3NodmtkeXhqYXN3aXp2b2ZwaG93c3Z5Y2hxY3pwYmNuZXhtdGtuY3h1ZXdvZW15aGRxanRocmN5YXh0Z3pjaWh6ZGxhcnljcWxnYWNrbHdocHZ1dm15cGF1ZnhwZWd3eGJlcXlidXdpbm5haXZxb25nZXNhaHN5bWplbHN6Z3Rzc2lrdnZna
Xd1YXpxZGV5dWlwdWl0a2FnZ2p4ZHR1Y21heGFuYmNqaGtueHNpcnR5ZW9vdXJ3aXl4ZWV4dGFlZG5seGRzcmdmandldm9pYnNrdXpwbHRyaWxsbHZpa3lqcG1jbWd4dG52a21zb21xYm9xbWJqaXRqanpxY3l3YnFhanVseWV3ZWNrbWJwbWNhZXRybGZycG9vcnV1bmxwaGNrZGpr
YmZjZ2p3dGhicnZldmN2amlsbmhwbWZtbXJlb3Zzc3lnbmJvaGh5dmhib3pjamdtbm14c2Vtb2d5cXlrdHFodW1scmx5ZWJ0c3FuaHdha3Jyc2pvd3Ntcml4dnhpbWR4YWZlaG5yYnJyYWdub2ttdnh4d3dpYXJzeHd4c3pua25vdHphbnZsdmJjbHlna3ZiaHBpcG9pY3Zxdmd3YWl
wemdubHhsY3RncHJ6YnlqY2xobnF0a2FwYm9qb2FtaWtzdHZua2N1dWNhaGNmZmdzcmNteXV5eGJ0dWV3b25sc2JyaGdkb3lqdXVkeHVoYmNiZnFwaW5xbHhva3V5Z2dmam91c2hnZWhkaHhscm53YW5wdG93YmJqeHdwbGlzeXpheWNtcHZyanJtaXpzbnJobXFncmZkYWxlbHRhZn
FxcW92cXh1ZHdkdHV2bXpwbmhubm50dWNyaGtkdXRvbWxmZGNteW9ucXFyaGhsd2N2dWRja2lmYm1keWhtYnN6eHpldnppc2tqdXppeHBua3Z1a2JqdW5lZnpuaGhsd2FrdHRydG1sY3JzcGptbnpuc3JvaHpxd3BtZmNyYXFyZ2lja2JxcmJmeHZ1Y2tkZnRuc3N2aWlyaWV2eGt1Y
W1vd2hiYnZpanV6bG95YXVveWVjaWFkanN4bXpocmRxYWl2bG1oZHR1cXhpd3lzZGphYWN6aGdvaWFmenVzZXlidnJ2bmVsb3pqbHZlYXFqbHN3a2Z5bGhycnNobXhzYmtkcnJjbWt4Y3BianZoa2FtcWVtcXVsdWpibmJnY2d1YW9ndHdjZmJseXVtc3hocXptbmJ6YXdudmx5b2li
cHFyamFqeWxqbGlmcHBud2dxam9ydGRiaG1scW1zbW1iYXJ3a2xueWduYWR6bXZsZGp6enBlanlwamtpZWFzeXVreWJqYXdibnJ0Ymp4dHFvbnJxZ25tdmV5c3RkcnpqYXF0cGh5cWRqeGhnYnNodHhwZ214eG9pb2RjdmNwYnB2dnBsc3lud3Fsb2Zrcmpza2tnc3ZwcnRwY2tnaHl
yY3Jrb21haGR5YW9pcXh1YmRveW54Y2x2b2dnbGl0eGl4am9kdnh3bG5peXd5dmdxem91cHhmY2RyZXllY2JyamF5amtneWtrYXBmdmF0YXhudmFubmx3cm9qcXJjeWVldnBkY2hzaHl6cnJ6ZWRkZGVuenJpb3N5b2JtaG9qend4b2FubndlZWFueW56dGJycmZocmpuemt1eHl3Zm
RkcWxtYWVtbGt4cGloemh6Y2xvZ2R0c2lteW1wa3ZqbWZna3JzbXRhY211cXhwenRzd3FybWN1amFxeWdkaXVlb3RoaWhsZ3ZoemN3ZWloZnVjcXpmZmRqZ2xuZXFxc3JiZXBic3l3dnR1bnZkbmJmanV5c255dm1semV1ZHZ3aXh0dGZocWV5dmtha3VmeHV5cXRmcmx1ZHhudHhkd
m5lY3FzaGdyd2ZhZm9sdXFycnRxbW16ZWZ3cG9tYW5kbXd1b3JmanVpenliaXhnbmF2Z3RxdXVvZmZxY3JrYXZ1eXN3aWpuZWhwbm9mdGhhdmFibmltcnB1b2pobW1oYXFldWhmamNqb21idGdxbmFoY2pmcndpZmFlanhpcnNnYWxrcWZjc3RicXRnbmdyamZzbHRyemlkcGNzeXln
aWJmbGt3Z2VxbmdvbHNkeWRycmx0aGpvd3hjbG5rcmFvcHd0aWJzYXJpZG5ucWlkcmtkd3NqandrZHBkdm1qdGduY3VhZ3BuaXpvYnVseXhraWFoc2hkcm1nd25xY3pibWlpdWtwdmR2YnBrdWd5cW9qZGhlbXp0anVqa2hja2d0YWhtYWp1Zmlzd2licHNjYWxxeGN4bnFnZHZnaHN
0dmlqZ3ZhdXVmcmdwZGhwa3lwa2ZzYWJmanJmY3RwaGxjd3J1dWtyb3FkZHFxamdwbW9pbHpvcGZxenBicHF1ZGRzZHd0Z2tpZnVoZ3l4dWRpZnBjaGZqdmhucmJ1ZWVoZG5qcGdmbmlpZGZva2xwc3dwcHNqc2h0amN0YXVpbmR1emRxenNpdHhnenZkbHRxd256ZnpxcHd1bHJ6em
piZ21maGpoc2Nod2t0cndiZXV5c3dtem14bGt6eXpseHludXptZW1kc2Nwc3hoenpzZHV2bWxxZ2praHF5emFid2RwZmxtb3FleGp2anBranZyc3VvYW1kdHBkYWd4Z21leXd6bmRyZ21icndzbHh1a2lpa3lmY3dlcGlmY3JoeW9zYmdxem5jY2ZrdXRwb3J5ZHlxaGN5aWl1a25nb
WRhZmFnbndvcHhjYnd0aW5jZ3BtemV3Y3hia3JneGl4aHh4d3JhaGVzY255aGJldWNuanJkcGlxbnV0aWdsdGdiZGx5cWhuenhzeXZ0dnhuYXR6cHZrcml4cXByanFmZXVvcGt1dnZzcnVqeXRpbXhkc3B2Y2ZrZnl0cmJ1b3JudXJrdmhidXFmdnh0dnNodmhtaHZ5Zm9jenFqYm1p
eHBvY3hkcWJoY3RucmFjdHVydWNmaGJheHBlcWt2dmJoc2Z5ZXh5cmxpaHZzbm9jd2Z5c2VrenV1ZGR5Y29keWJ0ZXBrZWpkc2x5ZWpvemVnc3N0b2p4cGFpYXBoeXdrbW52d2FkbXhxaW9zb2plYWZ5ZmR4amxoa2JmYnhsb3ZsdGN1ZXV6c3doeHdremF1dWNid3FtdHl1cXhycGp
kdWxwcHdhcW52ZHF3bHdhbm5lY2h4ZGJ3bHhjbHN4c2p0empyaHZhYnN3b2lzdXpscnZmdGd0anhmbGZ2ZHNsaWpkdnloaGlwZ2VoZGJhY2FsbGtneHdqdWVvZ214d25panZhZGdqbWh0ZXhxZHNxbWdleG15d2lreG9ydm5qdmxsZmtkZmtua2pzenNoY3J0YXZxcHlyb21kc3libW
Zma3FicXdmd3FqZnluY3RlZW5uZ3hkYXphYWhlY2tra2g=\"},{\"column_id\":4,\"value\":1351208411}]"] [key=74800000000000073d5f72800000000000b167] [tz=Asia/Shanghai]

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client): v8.4.0

(# /tidb-server -V
Release Version: v8.4.0
Edition: Community
Git Commit Hash: 7e87622511efa3113cbd771600d4272fecad79a9

Upstream TiKV version (execute tikv-server --version): v8.4.0

TiKV 
Release Version:   8.4.0-alpha
Edition:           Community
Git Commit Hash:   1cc62578757f50dddc32a8ae089414091b6d3be7

TiCDC version (execute cdc version): v8.4.0

Release Version: v8.4.0-pr11712
Git Commit Hash: 3c954ff9325d7ea9c0bb9169a95f681e7162653f

[fubinzh]

/severity critical

[fubinzh]

/label affects-8.4

[ti-chi-bot]

@fubinzh: The label(s) affects-8.4 cannot be applied, because the repository doesn't have them.

In response to this:

/label affects-8.4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[fubinzh]

/assign @3AceShowHand