tiflash crash and restart all the way

[fzhedu]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

happen by accadence

2. What did you expect to see? (Required)

3. What did you see instead (Required)

[2022/04/19 19:29:45.294 +08:00] [ERROR] [<unknown>] [BaseDaemon:########################################] [thread_id=16]
[2022/04/19 19:29:45.294 +08:00] [ERROR] [<unknown>] ["BaseDaemon:(from thread 14) Received signal Floating point exception (8)."] [thread_id=16]
[2022/04/19 19:29:45.294 +08:00] [ERROR] [<unknown>] ["BaseDaemon:Integer divide by zero."] [thread_id=16]
[2022/04/19 19:29:47.506 +08:00] [ERROR] [<unknown>] ["BaseDaemon:\n0x5a54041\tfaultSignalHandler(int, siginfo_t*, void*) [tiflash+94715969]\n         \tlibs/libdaemon/src/BaseDaemon.cpp:226\n0x7f10ed38b630\t<unknown symbol> [libpthread.so.0+63024]\n0xae640df\tDB::FramedChecksumReadBuffer<DB::Digest::XXH3>::doSeek(long, int) [tiflash+182862047]\n         \tdbms/src/IO/ChecksumBuffer.h:402\n0xb4b771a\tDB::DM::DMFilePackFilter::loadIndex(std::__1::unordered_map<long, DB::DM::RSIndex, std::__1::hash<long>, std::__1::equal_to<long>, std::__1::allocator<std::__1::pair<long const, DB::DM::RSIndex> > >&, std::__1::shared_ptr<DB::DM::DMFile> const&, std::__1::shared_ptr<DB::FileProvider> const&, std::__1::shared_ptr<DB::DM::MinMaxIndexCache> const&, long, std::__1::shared_ptr<DB::ReadLimiter> const&)::'lambda'()::operator()() const [tiflash+189495066]\n         \tdbms/src/Storages/DeltaMerge/File/DMFilePackFilter.h:244\n0xb4b7ebb\tstd::__1::pair<std::__1::shared_ptr<DB::DM::MinMaxIndex>, bool> DB::LRUCache<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DB::DM::MinMaxIndex, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, DB::DM::MinMaxIndexWeightFunction>::getOrSet<DB::DM::DMFilePackFilter::loadIndex(std::__1::unordered_map<long, DB::DM::RSIndex, std::__1::hash<long>, std::__1::equal_to<long>, std::__1::allocator<std::__1::pair<long const, DB::DM::RSIndex> > >&, std::__1::shared_ptr<DB::DM::DMFile> const&, std::__1::shared_ptr<DB::FileProvider> const&, std::__1::shared_ptr<DB::DM::MinMaxIndexCache> const&, long, std::__1::shared_ptr<DB::ReadLimiter> const&)::'lambda'()&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::DM::DMFilePackFilter::loadIndex(std::__1::unordered_map<long, DB::DM::RSIndex, std::__1::hash<long>, std::__1::equal_to<long>, std::__1::allocator<std::__1::pair<long const, DB::DM::RSIndex> > >&, std::__1::shared_ptr<DB::DM::DMFile> const&, std::__1::shared_ptr<DB::FileProvider> const&, std::__1::shared_ptr<DB::DM::MinMaxIndexCache> const&, long, std::__1::shared_ptr<DB::ReadLimiter> const&)::'lambda'()&) [tiflash+189497019]\n         \tdbms/src/Common/LRUCache.h:127\n0xb4b6f39\tDB::DM::DMFilePackFilter::loadIndex(std::__1::unordered_map<long, DB::DM::RSIndex, std::__1::hash<long>, std::__1::equal_to<long>, std::__1::allocator<std::__1::pair<long const, DB::DM::RSIndex> > >&, std::__1::shared_ptr<DB::DM::DMFile> const&, std::__1::shared_ptr<DB::FileProvider> const&, std::__1::shared_ptr<DB::DM::MinMaxIndexCache> const&, long, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+189493049]\n         \tdbms/src/Storages/DeltaMerge/File/DMFilePackFilter.h:255\n0xb4b65a8\tDB::DM::DMFilePackFilter::tryLoadIndex(long) [tiflash+189490600]\n         \tdbms/src/Storages/DeltaMerge/File/DMFilePackFilter.h:272\n0xb4b5479\tDB::DM::DMFilePackFilter::init() [tiflash+189486201]\n         \tdbms/src/Storages/DeltaMerge/File/DMFilePackFilter.h:129\n0xb4f91d4\tDB::DM::DMFileReader::DMFileReader(std::__1::shared_ptr<DB::DM::DMFile> const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > const&, bool, unsigned long, std::__1::vector<DB::DM::RowKeyRange, std::__1::allocator<DB::DM::RowKeyRange> > const&, std::__1::shared_ptr<DB::DM::RSOperator> const&, std::__1::shared_ptr<std::__1::set<unsigned long, std::__1::less<unsigned long>, std::__1::allocator<unsigned long> > > const&, unsigned long, std::__1::shared_ptr<DB::MarkCache> const&, std::__1::shared_ptr<DB::DM::MinMaxIndexCache> const&, bool, std::__1::shared_ptr<DB::DM::ColumnCache> const&, unsigned long, unsigned long, std::__1::shared_ptr<DB::FileProvider> const&, std::__1::shared_ptr<DB::ReadLimiter> const&, unsigned long, bool) [tiflash+189764052]\n         \tdbms/src/Storages/DeltaMerge/File/DMFileReader.cpp:249\n0x5a3aabc\tDB::DM::DMFileBlockInputStream::DMFileBlockInputStream(DB::Context const&, unsigned long, bool, unsigned long, std::__1::shared_ptr<DB::DM::DMFile> const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > const&, std::__1::vector<DB::DM::RowKeyRange, std::__1::allocator<DB::DM::RowKeyRange> > const&, std::__1::shared_ptr<DB::DM::RSOperator> const&, std::__1::shared_ptr<DB::DM::ColumnCache> const&, std::__1::shared_ptr<std::__1::set<unsigned long, std::__1::less<unsigned long>, std::__1::allocator<unsigned long> > > const&, unsigned long, bool) [tiflash+94612156]\n         \tdbms/src/Storages/DeltaMerge/File/DMFileBlockInputStream.h:41\n0xb4b225d\tDB::DM::StableValueSpace::Snapshot::getInputStream(DB::DM::DMContext const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > const&, std::__1::vector<DB::DM::RowKeyRange, std::__1::allocator<DB::DM::RowKeyRange> > const&, std::__1::shared_ptr<DB::DM::RSOperator> const&, unsigned long, unsigned long, bool) [tiflash+189473373]\n         \tdbms/src/Storages/DeltaMerge/StableValueSpace.cpp:337\n0xb47a637\tstd::__1::shared_ptr<DB::DM::SkippableBlockInputStream> DB::DM::Segment::getPlacedStream<false, DB::DM::DTCompactedEntries<55ul, 20ul, 3ul>::Iterator>(DB::DM::DMContext const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > const&, std::__1::vector<DB::DM::RowKeyRange, std::__1::allocator<DB::DM::RowKeyRange> > const&, std::__1::shared_ptr<DB::DM::RSOperator> const&, std::__1::shared_ptr<DB::DM::StableValueSpace::Snapshot> const&, std::__1::shared_ptr<DB::DM::DeltaValueReader> const&, DB::DM::DTCompactedEntries<55ul, 20ul, 3ul>::Iterator const&, DB::DM::DTCompactedEntries<55ul, 20ul, 3ul>::Iterator const&, unsigned long, unsigned long) [tiflash+189244983]\n         \tdbms/src/Storages/DeltaMerge/Segment.cpp:1395\n0xb47e3ad\tbool DB::DM::Segment::placeDelete<true>(DB::DM::DMContext const&, std::__1::shared_ptr<DB::DM::StableValueSpace::Snapshot> const&, std::__1::shared_ptr<DB::DM::DeltaValueReader> const&, DB::DM::RowKeyRange const&, DB::DM::DeltaTree<DB::DM::EmptyValueSpace, 55ul, 20ul, 3ul, DB::ArenaWithFreeLists>&, DB::DM::RowKeyRange const&, bool) const [tiflash+189260717]\n         \tdbms/src/Storages/DeltaMerge/Segment.cpp:1601\n0xb475694\tDB::DM::Segment::ensurePlace(DB::DM::DMContext const&, std::__1::shared_ptr<DB::DM::StableValueSpace::Snapshot> const&, std::__1::shared_ptr<DB::DM::DeltaValueReader> const&, std::__1::vector<DB::DM::RowKeyRange, std::__1::allocator<DB::DM::RowKeyRange> > const&, unsigned long) const [tiflash+189224596]\n         \tdbms/src/Storages/DeltaMerge/Segment.cpp:1479\n0xb4673cd\tDB::DM::Segment::getReadInfo(DB::DM::DMContext const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > const&, std::__1::shared_ptr<DB::DM::SegmentSnapshot> const&, std::__1::vector<DB::DM::RowKeyRange, std::__1::allocator<DB::DM::RowKeyRange> > const&, unsigned long) const [tiflash+189166541]\n         \tdbms/src/Storages/DeltaMerge/Segment.cpp:1337\n0xb467f37\tDB::DM::Segment::getInputStreamForDataExport(DB::DM::DMContext const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > const&, std::__1::shared_ptr<DB::DM::SegmentSnapshot> const&, DB::DM::RowKeyRange const&, unsigned long, bool) const [tiflash+189169463]\n         \tdbms/src/Storages/DeltaMerge/Segment.cpp:465\n0xb469457\tDB::DM::Segment::prepareMergeDelta(DB::DM::DMContext&, std::__1::shared_ptr<std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > > const&, std::__1::shared_ptr<DB::DM::SegmentSnapshot> const&, DB::DM::WriteBatches&) const [tiflash+189174871]\n         \tdbms/src/Storages/DeltaMerge/Segment.cpp:599\n0xb438528\tDB::DM::DeltaMergeStore::segmentMergeDelta(DB::DM::DMContext&, std::__1::shared_ptr<DB::DM::Segment> const&, DB::DM::DeltaMergeStore::TaskRunThread, std::__1::shared_ptr<DB::DM::SegmentSnapshot>) [tiflash+188974376]\n         \tdbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp:1980\n0xb43c190\tDB::DM::DeltaMergeStore::handleBackgroundTask(bool) [tiflash+188989840]\n         \tdbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp:1440\n0xb379d41\tDB::BackgroundProcessingPool::threadFunction() [tiflash+188194113]\n         \tdbms/src/Storages/BackgroundProcessingPool.cpp:225\n0xb37a401\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int)::$_1> >(void*) [tiflash+188195841]\n         \t/usr/local/bin/../include/c++/v1/thread:291\n0x7f10ed383ea5\tstart_thread [libpthread.so.0+32421]"] [thread_id=16]

4. What is your TiFlash version? (Required)

v6.0.0

[lidezhu]

It happens because degrade master version to a previous release version. Introduced in this pr: #4500. And only affect master branch and will be fixed later.