TiFlash may crash under very frequent truncate table, change column type and insert data

[lidezhu]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

It may occurs with a small probability under the following scenario

1. change column type to a wider column type
2. insert some data into the new column that exceeds the range of the old column type
3. doing many truncate table
   The reason may be that if a table is truncated, then it become a tombstone table in tiflash. And tiflash has two schema sync mode: incremental sync and full sync. If the schema version changed a lot, then it will trigger a full sync. But the tombstone table schema change is not showed in the full sync list, so it's schema change cannot be seen. And tiflash cannot use the old schema to decode the new data.

2. What did you expect to see? (Required)

tiflash works normally

3. What did you see instead (Required)

tiflash crash

4. What is your TiFlash version? (Required)

the problem may influence all version

[ilovesoup]

Is the recoverable? @lidezhu

[lidezhu]

Is the recoverable? @lidezhu

This can be recovered by drop the relevant tables in tiflash side because the tables that may cause this problem are already dropped in tidb. And tiflash keep them just for the recover table feature, so it will cause little impact to drop them.

[JaySon-Huang]

Similar issue with the same root cause #1664

[flowbehappy]

Lower the severity since it is a very rare case. Only seen in testing env. And it is more like a design flaw.