[bugfix](memtable) arena is freed early and will cause use after free #46997
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
yiguolei
changed the title
|
run buildall |
TPC-H: Total hot run time: 32807 ms |
|
TeamCity be ut coverage result: |
TPC-DS: Total hot run time: 195253 ms |
ClickBench: Total hot run time: 31.87 s |
|
run buildall |
TPC-H: Total hot run time: 33934 ms |
|
TeamCity be ut coverage result: |
TPC-DS: Total hot run time: 194656 ms |
ClickBench: Total hot run time: 30.98 s |
|
PR approved by anyone and no changes requested. |
This was referenced Jan 15, 2025
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
the
approved
github-actions bot
pushed a commit
that referenced
this pull request
Jan 15, 2025
…#46997) ### What problem does this PR solve? Related PR: #40912 Problem Summary: Do not reset _arena in MemTable::to_block(), because it is still used in ~MemTable() when releasing agg places Fix the following use-after-free Use: ==3628099==ERROR: AddressSanitizer: heap-use-after-free on address 0x52100381be60 at pc 0x5648f30893f8 bp 0x7f8842433310 sp 0x7f8842433308 READ of size 8 at 0x52100381be60 thread T4767 (wg_flush_broker) #0 0x5648f30893f7 in phmap::priv::raw_hash_set<phmap::priv::FlatHashSetPolicy<unsigned long>, phmap::Hash<unsigned long>, phmap::EqualTo<unsigned long>, std::allocator<unsigned long>>::destroy_slots() doris/thirdparty/installed/include/parallel_hashmap/phmap.h:1992:14 #1 0x5648f30936f6 in phmap::priv::raw_hash_set<phmap::priv::FlatHashSetPolicy<unsigned long>, phmap::Hash<unsigned long>, phmap::EqualTo<unsigned long>, std::allocator<unsigned long>>::~raw_hash_set() doris/thirdparty/installed/include/parallel_hashmap/phmap.h:1236:23 #2 0x5648f3089276 in phmap::flat_hash_set<unsigned long, phmap::Hash<unsigned long>, phmap::EqualTo<unsigned long>, std::allocator<unsigned long>>::~flat_hash_set() doris/thirdparty/installed/include/parallel_hashmap/phmap.h:4577:7 #3 0x5648f308922a in doris::BitmapValue::~BitmapValue() doris/be/src/util/bitmap_value.h:824:7 #4 0x56490d319fa6 in doris::vectorized::AggregateFunctionBitmapData<doris::vectorized::AggregateFunctionBitmapUnionOp>::~AggregateFunctionBitmapData() doris/be/src/vec/aggregate_functions/aggregate_function_bitmap.h:127:8 #5 0x56490d49636a in doris::vectorized::IAggregateFunctionDataHelper<doris::vectorized::AggregateFunctionBitmapData<doris::vectorized::AggregateFunctionBitmapUnionOp>, doris::vectorized::AggregateFunctionBitmapOp<doris::vectorized::AggregateFunctionBitmapUnionOp>>::destroy(char*) const doris/be/src/vec/aggregate_functions/aggregate_function.h:563:92 #6 0x5648f68376e9 in doris::MemTable::~MemTable() doris/be/src/olap/memtable.cpp:159:27 Free: 0x52100381be60 is located 352 bytes inside of 4096-byte region [0x52100381bd00,0x52100381cd00) freed by thread T4767 (wg_flush_broker) here: #0 0x5648f2f3ee46 in free (doris/output/be/lib/doris_be+0x57418e46) (BuildId: 298b9c91a1ec8fe0) #1 0x5648f3080dfc in DefaultMemoryAllocator::free(void*) doris/be/src/vec/common/allocator.h:108:41 #2 0x5648f3080b3f in Allocator<false, false, false, DefaultMemoryAllocator>::free(void*, unsigned long) doris/be/src/vec/common/allocator.h:323:13 #3 0x5648f30b6dee in doris::vectorized::Arena::Chunk::~Chunk() doris/be/src/vec/common/arena.h:77:31 #4 0x5648f30b6d1f in doris::vectorized::Arena::~Arena() doris/be/src/vec/common/arena.h:151:16 #5 0x5648f30b695a in std::default_delete<doris::vectorized::Arena>::operator()(doris::vectorized::Arena*) const env/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:99:2 #6 0x5648f30b67c8 in std::__uniq_ptr_impl<doris::vectorized::Arena, std::default_delete<doris::vectorized::Arena>>::reset(doris::vectorized::Arena*) env/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:211:4 #7 0x5648f30b5d8c in std::unique_ptr<doris::vectorized::Arena, std::default_delete<doris::vectorized::Arena>>::reset(doris::vectorized::Arena*) env/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:509:7 #8 0x5648f684253b in doris::MemTable::_to_block(std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block>>*) doris/be/src/olap/memtable.cpp:522:12 #9 0x5648f6842ac5 in doris::MemTable::to_block(std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block>>*) doris/be/src/olap/memtable.cpp:528:5 #10 0x5648f6907a72 in doris::FlushToken::_do_flush_memtable(doris::MemTable*, int, long*) doris/be/src/olap/memtable_flush_executor.cpp:144:9 #11 0x5648f690932c in doris::FlushToken::_flush_memtable(std::shared_ptr<doris::MemTable>, int, long) doris/be/src/olap/memtable_flush_executor.cpp:183:16 #12 0x5648f6915d18 in doris::MemtableFlushTask::run() doris/be/src/olap/memtable_flush_executor.cpp:60:20
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Related PR: #40912
Problem Summary:
Do not reset _arena in MemTable::to_block(), because it is still used in ~MemTable() when releasing agg places
Fix the following use-after-free
Use:
==3628099==ERROR: AddressSanitizer: heap-use-after-free on address 0x52100381be60 at pc 0x5648f30893f8 bp 0x7f8842433310 sp 0x7f8842433308
READ of size 8 at 0x52100381be60 thread T4767 (wg_flush_broker)
#0 0x5648f30893f7 in phmap::priv::raw_hash_set<phmap::priv::FlatHashSetPolicy, phmap::Hash, phmap::EqualTo, std::allocator>::destroy_slots() doris/thirdparty/installed/include/parallel_hashmap/phmap.h:1992:14
#1 0x5648f30936f6 in phmap::priv::raw_hash_set<phmap::priv::FlatHashSetPolicy, phmap::Hash, phmap::EqualTo, std::allocator>::~raw_hash_set() doris/thirdparty/installed/include/parallel_hashmap/phmap.h:1236:23
#2 0x5648f3089276 in phmap::flat_hash_set<unsigned long, phmap::Hash, phmap::EqualTo, std::allocator>::~flat_hash_set() doris/thirdparty/installed/include/parallel_hashmap/phmap.h:4577:7
#3 0x5648f308922a in doris::BitmapValue::~BitmapValue() doris/be/src/util/bitmap_value.h:824:7
#4 0x56490d319fa6 in doris::vectorized::AggregateFunctionBitmapDatadoris::vectorized::AggregateFunctionBitmapUnionOp::~AggregateFunctionBitmapData() doris/be/src/vec/aggregate_functions/aggregate_function_bitmap.h:127:8
#5 0x56490d49636a in doris::vectorized::IAggregateFunctionDataHelper<doris::vectorized::AggregateFunctionBitmapDatadoris::vectorized::AggregateFunctionBitmapUnionOp, doris::vectorized::AggregateFunctionBitmapOpdoris::vectorized::AggregateFunctionBitmapUnionOp>::destroy(char*) const doris/be/src/vec/aggregate_functions/aggregate_function.h:563:92
#6 0x5648f68376e9 in doris::MemTable::~MemTable() doris/be/src/olap/memtable.cpp:159:27
Free:
0x52100381be60 is located 352 bytes inside of 4096-byte region [0x52100381bd00,0x52100381cd00)
freed by thread T4767 (wg_flush_broker) here:
#0 0x5648f2f3ee46 in free (doris/output/be/lib/doris_be+0x57418e46) (BuildId: 298b9c91a1ec8fe0)
#1 0x5648f3080dfc in DefaultMemoryAllocator::free(void*) doris/be/src/vec/common/allocator.h:108:41
#2 0x5648f3080b3f in Allocator<false, false, false, DefaultMemoryAllocator>::free(void*, unsigned long) doris/be/src/vec/common/allocator.h:323:13
#3 0x5648f30b6dee in doris::vectorized::Arena::Chunk::~Chunk() doris/be/src/vec/common/arena.h:77:31
#4 0x5648f30b6d1f in doris::vectorized::Arena::~Arena() doris/be/src/vec/common/arena.h:151:16
#5 0x5648f30b695a in std::default_deletedoris::vectorized::Arena::operator()(doris::vectorized::Arena*) const env/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:99:2
#6 0x5648f30b67c8 in std::__uniq_ptr_impl<doris::vectorized::Arena, std::default_deletedoris::vectorized::Arena>::reset(doris::vectorized::Arena*) env/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:211:4
#7 0x5648f30b5d8c in std::unique_ptr<doris::vectorized::Arena, std::default_deletedoris::vectorized::Arena>::reset(doris::vectorized::Arena*) env/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/unique_ptr.h:509:7
#8 0x5648f684253b in doris::MemTable::_to_block(std::unique_ptr<doris::vectorized::Block, std::default_deletedoris::vectorized::Block>) doris/be/src/olap/memtable.cpp:522:12
#9 0x5648f6842ac5 in doris::MemTable::to_block(std::unique_ptr<doris::vectorized::Block, std::default_deletedoris::vectorized::Block>) doris/be/src/olap/memtable.cpp:528:5
#10 0x5648f6907a72 in doris::FlushToken::_do_flush_memtable(doris::MemTable*, int, long*) doris/be/src/olap/memtable_flush_executor.cpp:144:9
#11 0x5648f690932c in doris::FlushToken::_flush_memtable(std::shared_ptrdoris::MemTable, int, long) doris/be/src/olap/memtable_flush_executor.cpp:183:16
#12 0x5648f6915d18 in doris::MemtableFlushTask::run() doris/be/src/olap/memtable_flush_executor.cpp:60:20
Release note
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)