[improve](txn insert) make sub transactions visible #41362

mymeiyi · 2024-09-26T11:07:09Z

this pr does not support mow table.

mow support is in #42089 or #43011

doris-robot · 2024-09-26T11:07:14Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

mymeiyi · 2024-09-26T11:14:29Z

run buildall

doris-robot · 2024-09-26T13:03:02Z

TeamCity be ut coverage result:
Function Coverage: 37.31% (9635/25827)
Line Coverage: 28.70% (79715/277800)
Region Coverage: 28.12% (41217/146573)
Branch Coverage: 24.75% (20989/84804)
Coverage Report: http://coverage.selectdb-in.cc/coverage/468f4a1ee092489aeb84d542f4c98fb7d4593b82_468f4a1ee092489aeb84d542f4c98fb7d4593b82/report/index.html

mymeiyi · 2024-09-27T06:26:37Z

run buildall

doris-robot · 2024-09-27T08:16:43Z

TeamCity be ut coverage result:
Function Coverage: 37.29% (9627/25815)
Line Coverage: 28.68% (79665/277737)
Region Coverage: 28.12% (41191/146504)
Branch Coverage: 24.74% (20973/84788)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7646c207b62a29bf04a45757bfe5eef2e8e9489b_7646c207b62a29bf04a45757bfe5eef2e8e9489b/report/index.html

dataroaring · 2024-09-27T13:33:09Z

fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java

@@ -1743,6 +1745,9 @@ public void setEnableLeftZigZag(boolean enableLeftZigZag) {
    @VariableMgr.VarAttr(name = GROUP_COMMIT, needForward = true)
    public String groupCommit = "off_mode";

+    @VariableMgr.VarAttr(name = ENABLE_QUERY_IN_TRANSACTION_LOAD, needForward = true)
+    public boolean enableQueryInTransactionLoad = false;


default should be true.

enable_transaction_write_visibility is a better name.

dataroaring · 2024-09-27T13:41:09Z

fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java

+                        ConnectContext.get().getTxnEntry().getTabletSubTxnIds(olapTable.getId(), tablet));
+                LOG.info("table={}, partition={}, tablet={}, sub txn ids={}", olapTable.getId(), partition.getId(),
+                        tablet.getId(), paloRange.getSubTxnIds());
+            }



We should assure that select in txn is forwarded to master too.

the observer also knows the sub txn info

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PruneEmptyPartition.java

cloud/src/meta-service/meta_service.cpp

dataroaring · 2024-09-27T14:11:48Z

be/src/olap/tablet.cpp

+        DCHECK(rowset != nullptr) << " rowset is nullptr for sub_txn_id=" << sub_txn_ids[i]
+                                  << ", partition_id=" << partition_id()
+                                  << ", tablet=" << tablet_id();
+        int64_t tmp_version = version + i + 1;


version has already been added in prallel_scanner_builder _build_scanners_by_rowid.

_load is called firstly, now version is not changed

be/src/cloud/cloud_tablet.cpp

fe/fe-core/src/main/java/org/apache/doris/transaction/TransactionEntry.java

mymeiyi · 2024-09-29T07:17:30Z

run buildall

doris-robot · 2024-09-29T09:02:08Z

TeamCity be ut coverage result:
Function Coverage: 37.29% (9627/25820)
Line Coverage: 28.67% (79675/277888)
Region Coverage: 28.11% (41208/146593)
Branch Coverage: 24.73% (20982/84848)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3b07d43331c05277cc24a34fb911921877e73809_3b07d43331c05277cc24a34fb911921877e73809/report/index.html

mymeiyi · 2024-09-29T12:01:46Z

run buildall

doris-robot · 2024-09-29T12:55:30Z

TeamCity be ut coverage result:
Function Coverage: 37.28% (9625/25820)
Line Coverage: 28.68% (79689/277888)
Region Coverage: 28.11% (41202/146593)
Branch Coverage: 24.72% (20976/84848)
Coverage Report: http://coverage.selectdb-in.cc/coverage/78739c1820daadfd7345065e11f836ab27e2a2a7_78739c1820daadfd7345065e11f836ab27e2a2a7/report/index.html

mymeiyi · 2024-09-30T08:22:33Z

run buildall

doris-robot · 2024-09-30T10:11:03Z

TeamCity be ut coverage result:
Function Coverage: 37.31% (9631/25811)
Line Coverage: 28.68% (79853/278428)
Region Coverage: 28.13% (41301/146824)
Branch Coverage: 24.75% (21043/85018)
Coverage Report: http://coverage.selectdb-in.cc/coverage/db8cdc062e4c6790ffc93255722c460ee3a49c5a_db8cdc062e4c6790ffc93255722c460ee3a49c5a/report/index.html

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PruneEmptyPartition.java

github-actions

clang-tidy made some suggestions

github-actions · 2024-10-12T03:36:07Z

be/src/olap/base_tablet.cpp

@@ -128,6 +128,42 @@ BaseTablet::~BaseTablet() {
    g_total_tablet_num << -1;
 }

+Status BaseTablet::capture_sub_txn_rs_readers(int64_t version,


warning: method 'capture_sub_txn_rs_readers' can be made const [readability-make-member-function-const]

be/src/olap/base_tablet.cpp:132:

- std::vector<RowSetSplits>* rs_splits) { + std::vector<RowSetSplits>* rs_splits) const {

github-actions · 2024-10-12T03:36:08Z

be/src/olap/tablet_reader.cpp

-            // unique keys with merge on write, no need to merge sort keys in rowset
-            need_ordered_result = false;
+            if (read_params.query_mow_in_mor) {
+                need_ordered_result = true;


warning: redundant boolean literal in conditional assignment [readability-simplify-boolean-expr]

need_ordered_result = true; ^

github-actions · 2024-10-12T03:36:08Z

be/src/vec/olap/vcollect_iterator.cpp

@@ -71,7 +71,11 @@ void VCollectIterator::init(TabletReader* reader, bool ori_data_overlapping, boo
        (_reader->_direct_mode || _reader->_tablet->keys_type() == KeysType::DUP_KEYS ||
         (_reader->_tablet->keys_type() == KeysType::UNIQUE_KEYS &&
          _reader->_tablet->enable_unique_key_merge_on_write()))) {
-        _merge = false;
+        if (_reader->_reader_context.query_mow_in_mor) {
+            _merge = true;


warning: redundant boolean literal in conditional assignment [readability-simplify-boolean-expr]

be/src/vec/olap/vcollect_iterator.cpp:73:

- if (_reader->_reader_context.query_mow_in_mor) { - _merge = true; - } else { - _merge = false; - } + _merge = static_cast<bool>(_reader->_reader_context.query_mow_in_mor);

github-actions · 2024-10-12T03:36:08Z

cloud/src/meta-service/meta_service.cpp

@@ -1410,6 +1410,108 @@ static bool try_fetch_and_parse_schema(Transaction* txn, RowsetMetaCloudPB& rows
    return true;
 }

+void MetaServiceImpl::get_tmp_rowset(::google::protobuf::RpcController* controller,


warning: function 'get_tmp_rowset' exceeds recommended size/complexity thresholds [readability-function-size]

void MetaServiceImpl::get_tmp_rowset(::google::protobuf::RpcController* controller, ^

Additional context

cloud/src/meta-service/meta_service.cpp:1412: 97 lines including whitespace and comments (threshold 80)

void MetaServiceImpl::get_tmp_rowset(::google::protobuf::RpcController* controller, ^

github-actions · 2024-10-12T03:36:08Z

cloud/src/meta-service/meta_service.cpp

+            return;
+        }
+        // set tablet schema
+        if (rowset_meta->has_tablet_schema()) continue;


warning: statement should be inside braces [readability-braces-around-statements]

Suggested change

if (rowset_meta->has_tablet_schema()) continue;

if (rowset_meta->has_tablet_schema()) { continue;

}

mymeiyi · 2024-10-12T03:38:31Z

run buildall

doris-robot · 2024-10-12T05:24:49Z

TeamCity be ut coverage result:
Function Coverage: 37.39% (9659/25834)
Line Coverage: 28.64% (80131/279747)
Region Coverage: 28.07% (41435/147608)
Branch Coverage: 24.67% (21103/85526)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e4fe4482ba7182ab619af8fb1fe75db39aac9762_e4fe4482ba7182ab619af8fb1fe75db39aac9762/report/index.html

mymeiyi · 2024-10-12T07:14:18Z

run buildall

mymeiyi · 2024-10-31T04:01:45Z

run buildall

doris-robot · 2024-10-31T05:42:33Z

TeamCity be ut coverage result:
Function Coverage: 37.92% (9850/25978)
Line Coverage: 29.17% (82053/281274)
Region Coverage: 28.44% (42347/148918)
Branch Coverage: 25.02% (21517/86014)
Coverage Report: http://coverage.selectdb-in.cc/coverage/26b0fdc8ea349ff089d07e1bb2b715aa59b661c8_26b0fdc8ea349ff089d07e1bb2b715aa59b661c8/report/index.html

mymeiyi · 2024-10-31T08:10:33Z

run p0

mymeiyi · 2024-11-04T02:28:49Z

run buildall

doris-robot · 2024-11-04T04:18:36Z

TeamCity be ut coverage result:
Function Coverage: 37.80% (9818/25972)
Line Coverage: 28.95% (81558/281685)
Region Coverage: 28.23% (42126/149214)
Branch Coverage: 24.81% (21373/86142)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ee8e7345872a92faf689d92b91b4e07122f36afc_ee8e7345872a92faf689d92b91b4e07122f36afc/report/index.html

mymeiyi · 2024-11-08T02:35:25Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-11-08T02:40:53Z

be/src/olap/base_tablet.cpp

@@ -129,6 +129,42 @@ BaseTablet::~BaseTablet() {
    g_total_tablet_num << -1;
 }

+Status BaseTablet::capture_sub_txn_rs_readers(int64_t version,


warning: method 'capture_sub_txn_rs_readers' can be made const [readability-make-member-function-const]

be/src/olap/base_tablet.cpp:133:

- std::vector<RowSetSplits>* rs_splits) { + std::vector<RowSetSplits>* rs_splits) const {

doris-robot · 2024-11-08T04:36:58Z

TeamCity be ut coverage result:
Function Coverage: 37.89% (9864/26031)
Line Coverage: 29.06% (82069/282456)
Region Coverage: 28.27% (42266/149485)
Branch Coverage: 24.85% (21436/86278)
Coverage Report: http://coverage.selectdb-in.cc/coverage/21597e29e7b140083f2e25ca1a8834c3a9ba5716_21597e29e7b140083f2e25ca1a8834c3a9ba5716/report/index.html

dataroaring · 2024-11-11T14:00:38Z

run buildall

be/src/olap/tablet.cpp

mymeiyi · 2024-11-12T07:04:39Z

run buildall

mymeiyi · 2024-11-12T14:09:21Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-11-12T14:14:50Z

cloud/src/meta-service/meta_service.cpp

@@ -1418,6 +1418,108 @@ static bool try_fetch_and_parse_schema(Transaction* txn, RowsetMetaCloudPB& rows
    return true;
 }

+void MetaServiceImpl::get_tmp_rowset(::google::protobuf::RpcController* controller,


warning: function 'get_tmp_rowset' exceeds recommended size/complexity thresholds [readability-function-size]

void MetaServiceImpl::get_tmp_rowset(::google::protobuf::RpcController* controller, ^

Additional context

cloud/src/meta-service/meta_service.cpp:1420: 97 lines including whitespace and comments (threshold 80)

void MetaServiceImpl::get_tmp_rowset(::google::protobuf::RpcController* controller, ^

mymeiyi · 2024-11-13T08:39:54Z

run buildall

dataroaring

LGTM

github-actions · 2024-11-19T06:53:48Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-11-19T06:53:49Z

PR approved by anyone and no changes requested.

mymeiyi force-pushed the sub-txn-visible branch from 468f4a1 to 7646c20 Compare September 27, 2024 06:25

dataroaring reviewed Sep 27, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PruneEmptyPartition.java Show resolved Hide resolved

dataroaring reviewed Sep 27, 2024

View reviewed changes

cloud/src/meta-service/meta_service.cpp Show resolved Hide resolved

dataroaring reviewed Sep 27, 2024

View reviewed changes

be/src/cloud/cloud_tablet.cpp Outdated Show resolved Hide resolved

dataroaring reviewed Sep 27, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/transaction/TransactionEntry.java Show resolved Hide resolved

mymeiyi force-pushed the sub-txn-visible branch from 58694e1 to 3b07d43 Compare September 29, 2024 07:10

mymeiyi force-pushed the sub-txn-visible branch 2 times, most recently from b730606 to db8cdc0 Compare September 30, 2024 08:19

morrySnow reviewed Oct 10, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PruneEmptyPartition.java Outdated Show resolved Hide resolved

mymeiyi force-pushed the sub-txn-visible branch 2 times, most recently from 393fb19 to e4fe448 Compare October 12, 2024 03:33

github-actions bot reviewed Oct 12, 2024

View reviewed changes

mymeiyi force-pushed the sub-txn-visible branch from e4fe448 to f8e33fe Compare October 12, 2024 07:13

mymeiyi mentioned this pull request Oct 31, 2024

[fix](txn) make sub transactions visible for mow #43011

Closed

mymeiyi force-pushed the sub-txn-visible branch from 26b0fdc to ee8e734 Compare November 4, 2024 02:28

mymeiyi force-pushed the sub-txn-visible branch from ee8e734 to 21597e2 Compare November 8, 2024 02:35

github-actions bot reviewed Nov 8, 2024

View reviewed changes

dataroaring force-pushed the sub-txn-visible branch from 21597e2 to 148ee0b Compare November 11, 2024 14:00

dataroaring reviewed Nov 11, 2024

View reviewed changes

be/src/olap/tablet.cpp Show resolved Hide resolved

mymeiyi force-pushed the sub-txn-visible branch from 148ee0b to 1495454 Compare November 12, 2024 07:03

mymeiyi force-pushed the sub-txn-visible branch from 1495454 to 548ca97 Compare November 12, 2024 14:09

github-actions bot reviewed Nov 12, 2024

View reviewed changes

mymeiyi and others added 4 commits November 13, 2024 16:19

[improve](txn insert) make sub transactions visible

bc3937b

fix

8c3de52

remove subtransaction type

9eb80c9

add case

63ef3c1

mymeiyi force-pushed the sub-txn-visible branch from 548ca97 to 63ef3c1 Compare November 13, 2024 08:36

dataroaring approved these changes Nov 19, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 19, 2024

github-actions bot added the reviewed label Nov 19, 2024

	if (rowset_meta->has_tablet_schema()) continue;
	if (rowset_meta->has_tablet_schema()) { continue;
	}

[improve](txn insert) make sub transactions visible #41362

Are you sure you want to change the base?

[improve](txn insert) make sub transactions visible #41362

Conversation

mymeiyi commented Sep 26, 2024 • edited Loading

doris-robot commented Sep 26, 2024

mymeiyi commented Sep 26, 2024

doris-robot commented Sep 26, 2024

mymeiyi commented Sep 27, 2024

doris-robot commented Sep 27, 2024

dataroaring Sep 27, 2024

Choose a reason for hiding this comment

dataroaring Nov 11, 2024

Choose a reason for hiding this comment

dataroaring Sep 27, 2024 • edited Loading

Choose a reason for hiding this comment

mymeiyi Sep 29, 2024

Choose a reason for hiding this comment

dataroaring Sep 27, 2024

Choose a reason for hiding this comment

mymeiyi Sep 29, 2024

Choose a reason for hiding this comment

mymeiyi commented Sep 29, 2024

doris-robot commented Sep 29, 2024

mymeiyi commented Sep 29, 2024

doris-robot commented Sep 29, 2024

mymeiyi commented Sep 30, 2024

doris-robot commented Sep 30, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Oct 12, 2024

Choose a reason for hiding this comment

github-actions bot Oct 12, 2024

Choose a reason for hiding this comment

github-actions bot Oct 12, 2024

Choose a reason for hiding this comment

github-actions bot Oct 12, 2024

Choose a reason for hiding this comment

github-actions bot Oct 12, 2024

Choose a reason for hiding this comment

mymeiyi commented Oct 12, 2024

doris-robot commented Oct 12, 2024

mymeiyi commented Oct 12, 2024

mymeiyi commented Oct 31, 2024

doris-robot commented Oct 31, 2024

mymeiyi commented Oct 31, 2024

mymeiyi commented Nov 4, 2024

doris-robot commented Nov 4, 2024

mymeiyi commented Nov 8, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Nov 8, 2024

Choose a reason for hiding this comment

doris-robot commented Nov 8, 2024

dataroaring commented Nov 11, 2024

mymeiyi commented Nov 12, 2024

mymeiyi commented Nov 12, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Nov 12, 2024

Choose a reason for hiding this comment

mymeiyi commented Nov 13, 2024

dataroaring left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 19, 2024

github-actions bot commented Nov 19, 2024

mymeiyi commented Sep 26, 2024 •

edited

Loading

dataroaring Sep 27, 2024 •

edited

Loading