[FLASH-461/471] Basic read/write StorageDeltaMerge's table through Raft #217

JaySon-Huang · 2019-09-03T13:36:36Z

Changes

Add enum StorageEngine for indicating which storage engine to use
DBGInvoke mock_tidb_table now support Engine=DeltaMerge
We can specify default storage engine by raft.storage_engine in tiflash.xml
Use IManageableStorage as interface for Storages synced from TiDB
TMTStorages now store ptr to IManageableStore instead of StorageMergeTree
Add StorageDeltaMerge::deleteRange
Support applySnapshot by using StorageDeltaMerge::deleteRange
Use ::DB::MutableSupport for constant naming in DeltaMergeDefines.h
Note that we can NOT read data in KVStore by now, we must flush data to StorageDeltaMerge by using some commands like DBGInvoke try_flush_region

Steps to debug for DeltaMerge table

See tests/delta_merge/raft/sync_table_from_raft.test for detail

Create a DeltaMerge table and bind to a region
- using mock_tidb_table, refresh_schemas, put_region
Insert data and flush to DM table
- using raft_insert_row, try_flush_region
Select data from DM table as normal

TODO

Support UInt64 handle type. ApplySnapshot.cpp (FLASH-392) @marsishandsome
- Ensure that getHandleMapByRange<Int64> can handle all non UInt64 pk-type
- ~~Inside RegionDataMover::getHandleMapByRange, call createBlockInputStreamFromRange. Is DeltaMerge table support SELRAW NOKVSTORE ... ?~~ This is only called by TMT.
Flush data in KVStore to StorageDeltaMerge (FLASH-463)
Read with sepcify schema version (FLASH-469)
Learner read? (FLASH-470)

~~5. Bug: Try to lock mutex after mutex is free (FLASH-468)~~
Done @ b00a3a1
~~6. Bug: If write throw exception, that region can not write any more~~
Done @ b054cdf, merged to master in this commit 1509cfe

dbms/src/Storages/DeltaMerge/tests/gtest_dm_storage_delta_merge.cpp

dbms/src/Storages/Transaction/applySnapshot.cpp

dbms/src/Storages/Transaction/RegionDataMover.cpp

JaySon-Huang · 2019-09-04T04:26:24Z

dbms/src/Interpreters/InterpreterSelectQuery.cpp

@@ -208,6 +208,7 @@ void InterpreterSelectQuery::getAndLockStorageWithSchemaVersion(const String & d
        if (!storage_)
            return std::make_tuple(nullptr, nullptr, DEFAULT_UNSPECIFIED_SCHEMA_VERSION, false);

+        // TODO handle if storage_ is a DeltaMerge?


flowbehappy · 2019-09-04T06:52:14Z

dbms/src/Storages/DeltaMerge/DeltaMergeDefines.h


-static const ColumnDefine VERSION_COLUMN_DEFINE{VERSION_COLUMN_ID, VERSION_COLUMN_NAME, VERSION_COLUMN_TYPE};
-static const ColumnDefine TAG_COLUMN_DEFINE{TAG_COLUMN_ID, TAG_COLUMN_NAME, TAG_COLUMN_TYPE};
+inline ColumnDefine getVersionColumnDefine()


Why not use static object?

Column name and type are defined in class ::DB::MutableSupport now, and C++ don't ensure the order of init for static object in different compile unit.
If we use static object here, we will get empty column name and nullptr as column type.

Maybe static local object? Like

inline const ColumnDefine & EXTRA_HANDLE_COLUMN() { static ColumnDefine VERSION_COLUMN_DEFINE_{VERSION_COLUMN_ID, VERSION_COLUMN_NAME, VERSION_COLUMN_TYPE}; return VERSION_COLUMN_DEFINE_; }

JaySon-Huang · 2019-09-05T08:51:16Z

/run-integration-tests

JaySon-Huang · 2019-09-05T09:27:18Z

/run-integration-tests

JaySon-Huang · 2019-09-05T10:28:41Z

/run-integration-tests

JaySon-Huang · 2019-09-06T14:17:51Z

/run-integration-tests

JaySon-Huang · 2019-09-07T06:50:17Z

/run-integration-tests

JaySon-Huang · 2019-09-09T15:13:30Z

/rebuild

JaySon-Huang · 2019-09-10T02:32:50Z

/rebuild

JaySon-Huang · 2019-09-10T03:44:52Z

/run-integration-tests

flowbehappy · 2019-09-12T08:28:12Z

LGTM

MutableSupport::storage_name -> MutableSupport::mmt_storage_name

JaySon-Huang · 2019-09-12T11:01:27Z

/run-integration-tests

…ft (pingcap#217) ## Changes * Add enum StorageEngine for indicating which storage engine to use * `DBGInvoke mock_tidb_table` now support Engine=DeltaMerge * We can specify default storage engine by raft.storage_engine in tiflash.xml * Use `IManageableStorage` as interface for Storages synced from TiDB * TMTStorages now store ptr to IManageableStore instead of StorageMergeTree * Add `StorageDeltaMerge::deleteRange` * Support `applySnapshot` by using `StorageDeltaMerge::deleteRange` * Use ::DB::MutableSupport for constant naming in DeltaMergeDefines.h * Note that we can NOT read data in KVStore by now, we must flush data to StorageDeltaMerge by using some commands like `DBGInvoke try_flush_region`

…ft (pingcap#217) * Add enum StorageEngine for indicating which storage engine to use * `DBGInvoke mock_tidb_table` now support Engine=DeltaMerge * We can specify default storage engine by raft.storage_engine in tiflash.xml * Use `IManageableStorage` as interface for Storages synced from TiDB * TMTStorages now store ptr to IManageableStore instead of StorageMergeTree * Add `StorageDeltaMerge::deleteRange` * Support `applySnapshot` by using `StorageDeltaMerge::deleteRange` * Use ::DB::MutableSupport for constant naming in DeltaMergeDefines.h * Note that we can NOT read data in KVStore by now, we must flush data to StorageDeltaMerge by using some commands like `DBGInvoke try_flush_region`

…ft (#217) * Add enum StorageEngine for indicating which storage engine to use * `DBGInvoke mock_tidb_table` now support Engine=DeltaMerge * We can specify default storage engine by raft.storage_engine in tiflash.xml * Use `IManageableStorage` as interface for Storages synced from TiDB * TMTStorages now store ptr to IManageableStore instead of StorageMergeTree * Add `StorageDeltaMerge::deleteRange` * Support `applySnapshot` by using `StorageDeltaMerge::deleteRange` * Use ::DB::MutableSupport for constant naming in DeltaMergeDefines.h * Note that we can NOT read data in KVStore by now, we must flush data to StorageDeltaMerge by using some commands like `DBGInvoke try_flush_region`

…ft (pingcap#217) * Add enum StorageEngine for indicating which storage engine to use * `DBGInvoke mock_tidb_table` now support Engine=DeltaMerge * We can specify default storage engine by raft.storage_engine in tiflash.xml * Use `IManageableStorage` as interface for Storages synced from TiDB * TMTStorages now store ptr to IManageableStore instead of StorageMergeTree * Add `StorageDeltaMerge::deleteRange` * Support `applySnapshot` by using `StorageDeltaMerge::deleteRange` * Use ::DB::MutableSupport for constant naming in DeltaMergeDefines.h * Note that we can NOT read data in KVStore by now, we must flush data to StorageDeltaMerge by using some commands like `DBGInvoke try_flush_region`

…ft (#217) * Add enum StorageEngine for indicating which storage engine to use * `DBGInvoke mock_tidb_table` now support Engine=DeltaMerge * We can specify default storage engine by raft.storage_engine in tiflash.xml * Use `IManageableStorage` as interface for Storages synced from TiDB * TMTStorages now store ptr to IManageableStore instead of StorageMergeTree * Add `StorageDeltaMerge::deleteRange` * Support `applySnapshot` by using `StorageDeltaMerge::deleteRange` * Use ::DB::MutableSupport for constant naming in DeltaMergeDefines.h * Note that we can NOT read data in KVStore by now, we must flush data to StorageDeltaMerge by using some commands like `DBGInvoke try_flush_region`

…ft (pingcap#217) * Add enum StorageEngine for indicating which storage engine to use * `DBGInvoke mock_tidb_table` now support Engine=DeltaMerge * We can specify default storage engine by raft.storage_engine in tiflash.xml * Use `IManageableStorage` as interface for Storages synced from TiDB * TMTStorages now store ptr to IManageableStore instead of StorageMergeTree * Add `StorageDeltaMerge::deleteRange` * Support `applySnapshot` by using `StorageDeltaMerge::deleteRange` * Use ::DB::MutableSupport for constant naming in DeltaMergeDefines.h * Note that we can NOT read data in KVStore by now, we must flush data to StorageDeltaMerge by using some commands like `DBGInvoke try_flush_region`

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

JaySon-Huang commented Sep 4, 2019

View reviewed changes

dbms/src/Storages/DeltaMerge/tests/gtest_dm_storage_delta_merge.cpp Show resolved Hide resolved

JaySon-Huang commented Sep 4, 2019

View reviewed changes

dbms/src/Storages/Transaction/applySnapshot.cpp Outdated Show resolved Hide resolved

JaySon-Huang commented Sep 4, 2019

View reviewed changes

dbms/src/Storages/Transaction/RegionDataMover.cpp Show resolved Hide resolved

JaySon-Huang commented Sep 4, 2019

View reviewed changes

JaySon-Huang requested a review from flowbehappy September 4, 2019 06:15

flowbehappy reviewed Sep 4, 2019

View reviewed changes

JaySon-Huang force-pushed the DeltaMergeEngine branch from afb8ffc to 0f63c48 Compare September 5, 2019 07:30

JaySon-Huang force-pushed the dm-raft-integrate branch from b25da45 to 8ded428 Compare September 5, 2019 08:19

JaySon-Huang changed the title ~~Basic read/write StorageDeltaMerge's table through Raft~~ [FLASH-461/471] Basic read/write StorageDeltaMerge's table through Raft Sep 5, 2019

JaySon-Huang force-pushed the dm-raft-integrate branch from b00a3a1 to 3d2ebf7 Compare September 6, 2019 06:07

JaySon-Huang added 11 commits September 12, 2019 17:48

use IManageableStorage instead of StorageMergeTree in TMTStorages

9227266

refine ColumnId define && some include headers

ab985d1

MutableSupport::storage_name -> MutableSupport::mmt_storage_name

simple read/write data to DM table through Raft

35e5be4

fix some gtest cases

7a94d0e

use "{WORD}" to replace some output in test cases

ad5a96a

fix bug: the sequence of static var init

579e145

fix bug: init TMTContext in gtests

003cb62

add shutdown

235333d

handle the cases by casting to TMT

7453bbe

move enum StorageEngine to a single .h file

97a2a69

bug fix

7f6ac9c

JaySon-Huang added 9 commits September 12, 2019 17:56

use ::TiDB::StorageEngine

4e61e02

Raft apply snapshot, use delete range for StorageDeltaMerge

37e44db

enhance run-test for verbose and inline comment

df13a65

bug fix: test cases for --schema

dea18e4

make ci run delta-merge-test

9166472

fix bug: FLASH-468

8f17307

address comment

9a42dba

minor fix

e2b2ca9

add debug function for force flushing all regions

5c7e00a

JaySon-Huang force-pushed the dm-raft-integrate branch from 2e25424 to 5c7e00a Compare September 12, 2019 10:33

remove useless code

0b8aac0

JaySon-Huang merged commit 22df6d3 into pingcap:DeltaMergeEngine Sep 12, 2019

CalvinNeo mentioned this pull request Nov 8, 2022

update proxy of master to raftstore-proxy #6270

Merged

12 tasks

JaySon-Huang pushed a commit to JaySon-Huang/tiflash that referenced this pull request Aug 2, 2024

Cherry-pick: Class memory trace for PageStorage (pingcap#217)

d1288ed

Signed-off-by: CalvinNeo <calvinneo1995@gmail.com>

CalvinNeo mentioned this pull request Nov 7, 2024

update proxy of master to raftstore-proxy #9590

Closed

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLASH-461/471] Basic read/write StorageDeltaMerge's table through Raft #217

[FLASH-461/471] Basic read/write StorageDeltaMerge's table through Raft #217

JaySon-Huang commented Sep 3, 2019 •

edited

Loading

JaySon-Huang Sep 4, 2019

flowbehappy Sep 4, 2019

JaySon-Huang Sep 4, 2019

flowbehappy Sep 9, 2019

JaySon-Huang Sep 9, 2019

JaySon-Huang commented Sep 5, 2019

JaySon-Huang commented Sep 5, 2019

JaySon-Huang commented Sep 5, 2019

JaySon-Huang commented Sep 6, 2019

JaySon-Huang commented Sep 7, 2019

JaySon-Huang commented Sep 9, 2019

JaySon-Huang commented Sep 10, 2019

JaySon-Huang commented Sep 10, 2019

flowbehappy commented Sep 12, 2019

JaySon-Huang commented Sep 12, 2019

[FLASH-461/471] Basic read/write StorageDeltaMerge's table through Raft #217

[FLASH-461/471] Basic read/write StorageDeltaMerge's table through Raft #217

Conversation

JaySon-Huang commented Sep 3, 2019 • edited Loading

Changes

Steps to debug for DeltaMerge table

TODO

JaySon-Huang Sep 4, 2019

Choose a reason for hiding this comment

flowbehappy Sep 4, 2019

Choose a reason for hiding this comment

JaySon-Huang Sep 4, 2019

Choose a reason for hiding this comment

flowbehappy Sep 9, 2019

Choose a reason for hiding this comment

JaySon-Huang Sep 9, 2019

Choose a reason for hiding this comment

JaySon-Huang commented Sep 5, 2019

JaySon-Huang commented Sep 5, 2019

JaySon-Huang commented Sep 5, 2019

JaySon-Huang commented Sep 6, 2019

JaySon-Huang commented Sep 7, 2019

JaySon-Huang commented Sep 9, 2019

JaySon-Huang commented Sep 10, 2019

JaySon-Huang commented Sep 10, 2019

flowbehappy commented Sep 12, 2019

JaySon-Huang commented Sep 12, 2019

JaySon-Huang commented Sep 3, 2019 •

edited

Loading