[Fix](hive-writer) Fix partition column orders issue when the partition fields inserted into the target table are inconsistent with the field order of the query source table and the schema field order of the query source table. #35347
Conversation
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
kaka11chen
force-pushed
the
fix_partition_column_orders_issue
branch
from
55c2cd2 to
b877d98
Compare
|
clang-tidy review says "All clean, LGTM! 👍" |
1 similar comment
|
clang-tidy review says "All clean, LGTM! 👍" |
kaka11chen
force-pushed
the
fix_partition_column_orders_issue
branch
from
b877d98 to
de0dffc
Compare
|
clang-tidy review says "All clean, LGTM! 👍" |
|
clang-tidy review says "All clean, LGTM! 👍" |
kaka11chen
changed the title
|
run buildall |
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
the
approved
|
PR approved by anyone and no changes requested. |
github-actions
bot
removed
the
approved
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| @@ -82,8 +82,17 @@ Status VHiveTableWriter::open(RuntimeState* state, RuntimeProfile* profile) { | |||
|
|
|||
| Status VHiveTableWriter::write(vectorized::Block& block) { | |||
There was a problem hiding this comment.
warning: function 'write' has cognitive complexity of 83 (threshold 50) [readability-function-cognitive-complexity]
Status VHiveTableWriter::write(vectorized::Block& block) {
^Additional context
be/src/vec/sink/writer/vhive_table_writer.cpp:85: +1, including nesting penalty of 0, nesting level increased to 1
if (block.rows() == 0) {
^be/src/vec/sink/writer/vhive_table_writer.cpp:89: +1, including nesting penalty of 0, nesting level increased to 1
RETURN_IF_ERROR(vectorized::VExprContext::get_output_block_after_execute_exprs(
^be/src/common/status.h:614: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:89: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(vectorized::VExprContext::get_output_block_after_execute_exprs(
^be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:97: +1, including nesting penalty of 0, nesting level increased to 1
if (_partition_columns_input_index.empty()) {
^be/src/vec/sink/writer/vhive_table_writer.cpp:102: +2, including nesting penalty of 1, nesting level increased to 2
if (writer_iter == _partitions_to_writers.end()) {
^be/src/vec/sink/writer/vhive_table_writer.cpp:105: +3, including nesting penalty of 2, nesting level increased to 3
} catch (doris::Exception& e) {
^be/src/vec/sink/writer/vhive_table_writer.cpp:109: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(writer->open(_state, _profile));
^be/src/common/status.h:614: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:109: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(writer->open(_state, _profile));
^be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:110: +1, nesting level increased to 2
} else {
^be/src/vec/sink/writer/vhive_table_writer.cpp:111: +3, including nesting penalty of 2, nesting level increased to 3
if (writer_iter->second->written_len() > config::hive_sink_max_file_size) {
^be/src/vec/sink/writer/vhive_table_writer.cpp:122: +4, including nesting penalty of 3, nesting level increased to 4
} catch (doris::Exception& e) {
^be/src/vec/sink/writer/vhive_table_writer.cpp:126: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(writer->open(_state, _profile));
^be/src/common/status.h:614: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:126: +5, including nesting penalty of 4, nesting level increased to 5
RETURN_IF_ERROR(writer->open(_state, _profile));
^be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:127: +1, nesting level increased to 3
} else {
^be/src/vec/sink/writer/vhive_table_writer.cpp:134: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(writer->write(output_block));
^be/src/common/status.h:614: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:134: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(writer->write(output_block));
^be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:140: +1, including nesting penalty of 0, nesting level increased to 1
for (int i = 0; i < output_block.rows(); ++i) {
^be/src/vec/sink/writer/vhive_table_writer.cpp:144: +2, including nesting penalty of 1, nesting level increased to 2
} catch (doris::Exception& e) {
^be/src/vec/sink/writer/vhive_table_writer.cpp:151: nesting level increased to 2
[&](const std::string& partition_name, int position,
^be/src/vec/sink/writer/vhive_table_writer.cpp:157: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(writer->open(_state, _profile));
^be/src/common/status.h:614: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:157: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(writer->open(_state, _profile));
^be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:163: +3, including nesting penalty of 2, nesting level increased to 3
} catch (doris::Exception& e) {
^be/src/vec/sink/writer/vhive_table_writer.cpp:170: +2, including nesting penalty of 1, nesting level increased to 2
if (writer_iter == _partitions_to_writers.end()) {
^be/src/vec/sink/writer/vhive_table_writer.cpp:172: +3, including nesting penalty of 2, nesting level increased to 3
if (_partitions_to_writers.size() + 1 >
^be/src/vec/sink/writer/vhive_table_writer.cpp:178: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(create_and_open_writer(partition_name, i, nullptr, 0, writer));
^be/src/common/status.h:614: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:178: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(create_and_open_writer(partition_name, i, nullptr, 0, writer));
^be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:179: +1, nesting level increased to 2
} else {
^be/src/vec/sink/writer/vhive_table_writer.cpp:181: +3, including nesting penalty of 2, nesting level increased to 3
if (writer_iter->second->written_len() > config::hive_sink_max_file_size) {
^be/src/vec/sink/writer/vhive_table_writer.cpp:190: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(create_and_open_writer(partition_name, i, &file_name,
^be/src/common/status.h:614: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:190: +5, including nesting penalty of 4, nesting level increased to 5
RETURN_IF_ERROR(create_and_open_writer(partition_name, i, &file_name,
^be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:192: +1, nesting level increased to 3
} else {
^be/src/vec/sink/writer/vhive_table_writer.cpp:196: +3, including nesting penalty of 2, nesting level increased to 3
if (writer_pos_iter == writer_positions.end()) {
^be/src/vec/sink/writer/vhive_table_writer.cpp:200: +1, nesting level increased to 3
} else {
^
|
run buildall |
kaka11chen
force-pushed
the
fix_partition_column_orders_issue
branch
from
71cc694 to
d298e5c
Compare
|
run buildall |
| @@ -82,8 +83,17 @@ Status VHiveTableWriter::open(RuntimeState* state, RuntimeProfile* profile) { | |||
|
|
|||
| Status VHiveTableWriter::write(vectorized::Block& block) { | |||
There was a problem hiding this comment.
warning: function 'write' has cognitive complexity of 83 (threshold 50) [readability-function-cognitive-complexity]
Status VHiveTableWriter::write(vectorized::Block& block) {
^Additional context
be/src/vec/sink/writer/vhive_table_writer.cpp:86: +1, including nesting penalty of 0, nesting level increased to 1
if (block.rows() == 0) {
^be/src/vec/sink/writer/vhive_table_writer.cpp:90: +1, including nesting penalty of 0, nesting level increased to 1
RETURN_IF_ERROR(vectorized::VExprContext::get_output_block_after_execute_exprs(
^be/src/common/status.h:614: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:90: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(vectorized::VExprContext::get_output_block_after_execute_exprs(
^be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:98: +1, including nesting penalty of 0, nesting level increased to 1
if (_partition_columns_input_index.empty()) {
^be/src/vec/sink/writer/vhive_table_writer.cpp:103: +2, including nesting penalty of 1, nesting level increased to 2
if (writer_iter == _partitions_to_writers.end()) {
^be/src/vec/sink/writer/vhive_table_writer.cpp:106: +3, including nesting penalty of 2, nesting level increased to 3
} catch (doris::Exception& e) {
^be/src/vec/sink/writer/vhive_table_writer.cpp:110: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(writer->open(_state, _profile));
^be/src/common/status.h:614: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:110: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(writer->open(_state, _profile));
^be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:111: +1, nesting level increased to 2
} else {
^be/src/vec/sink/writer/vhive_table_writer.cpp:112: +3, including nesting penalty of 2, nesting level increased to 3
if (writer_iter->second->written_len() > config::hive_sink_max_file_size) {
^be/src/vec/sink/writer/vhive_table_writer.cpp:123: +4, including nesting penalty of 3, nesting level increased to 4
} catch (doris::Exception& e) {
^be/src/vec/sink/writer/vhive_table_writer.cpp:127: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(writer->open(_state, _profile));
^be/src/common/status.h:614: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:127: +5, including nesting penalty of 4, nesting level increased to 5
RETURN_IF_ERROR(writer->open(_state, _profile));
^be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:128: +1, nesting level increased to 3
} else {
^be/src/vec/sink/writer/vhive_table_writer.cpp:135: +2, including nesting penalty of 1, nesting level increased to 2
RETURN_IF_ERROR(writer->write(output_block));
^be/src/common/status.h:614: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:135: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(writer->write(output_block));
^be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:141: +1, including nesting penalty of 0, nesting level increased to 1
for (int i = 0; i < output_block.rows(); ++i) {
^be/src/vec/sink/writer/vhive_table_writer.cpp:145: +2, including nesting penalty of 1, nesting level increased to 2
} catch (doris::Exception& e) {
^be/src/vec/sink/writer/vhive_table_writer.cpp:152: nesting level increased to 2
[&](const std::string& partition_name, int position,
^be/src/vec/sink/writer/vhive_table_writer.cpp:158: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(writer->open(_state, _profile));
^be/src/common/status.h:614: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:158: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(writer->open(_state, _profile));
^be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:164: +3, including nesting penalty of 2, nesting level increased to 3
} catch (doris::Exception& e) {
^be/src/vec/sink/writer/vhive_table_writer.cpp:171: +2, including nesting penalty of 1, nesting level increased to 2
if (writer_iter == _partitions_to_writers.end()) {
^be/src/vec/sink/writer/vhive_table_writer.cpp:173: +3, including nesting penalty of 2, nesting level increased to 3
if (_partitions_to_writers.size() + 1 >
^be/src/vec/sink/writer/vhive_table_writer.cpp:179: +3, including nesting penalty of 2, nesting level increased to 3
RETURN_IF_ERROR(create_and_open_writer(partition_name, i, nullptr, 0, writer));
^be/src/common/status.h:614: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:179: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(create_and_open_writer(partition_name, i, nullptr, 0, writer));
^be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:180: +1, nesting level increased to 2
} else {
^be/src/vec/sink/writer/vhive_table_writer.cpp:182: +3, including nesting penalty of 2, nesting level increased to 3
if (writer_iter->second->written_len() > config::hive_sink_max_file_size) {
^be/src/vec/sink/writer/vhive_table_writer.cpp:191: +4, including nesting penalty of 3, nesting level increased to 4
RETURN_IF_ERROR(create_and_open_writer(partition_name, i, &file_name,
^be/src/common/status.h:614: expanded from macro 'RETURN_IF_ERROR'
do { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:191: +5, including nesting penalty of 4, nesting level increased to 5
RETURN_IF_ERROR(create_and_open_writer(partition_name, i, &file_name,
^be/src/common/status.h:616: expanded from macro 'RETURN_IF_ERROR'
if (UNLIKELY(!_status_.ok())) { \
^be/src/vec/sink/writer/vhive_table_writer.cpp:193: +1, nesting level increased to 3
} else {
^be/src/vec/sink/writer/vhive_table_writer.cpp:197: +3, including nesting penalty of 2, nesting level increased to 3
if (writer_pos_iter == writer_positions.end()) {
^be/src/vec/sink/writer/vhive_table_writer.cpp:201: +1, nesting level increased to 3
} else {
^
kaka11chen
force-pushed
the
fix_partition_column_orders_issue
branch
from
d298e5c to
dcfabac
Compare
|
run buildall |
|
TeamCity be ut coverage result: |
TPC-H: Total hot run time: 41896 ms |
TPC-DS: Total hot run time: 172159 ms |
ClickBench: Total hot run time: 30.82 s |
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
the
approved
…on fields inserted into the target table are inconsistent with the field order of the query source table and the schema field order of the query source table. (apache#35347) Fix partition column orders issue when the partition fields inserted into the target table are inconsistent with the field order of the query source table and the schema field order of the query source table. Please look at `regression-test/suites/external_table_p0/hive/ddl/test_hive_write_type.groovy` in the PR.
…on fields inserted into the target table are inconsistent with the field order of the query source table and the schema field order of the query source table. (#35543) ## Proposed changes backport #35347 ## Further comments If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
…on fields inserted into the target table are inconsistent with the field order of the query source table and the schema field order of the query source table. (#35347) Fix partition column orders issue when the partition fields inserted into the target table are inconsistent with the field order of the query source table and the schema field order of the query source table. Please look at `regression-test/suites/external_table_p0/hive/ddl/test_hive_write_type.groovy` in the PR.
Proposed changes
#31442
Fix partition column orders issue when the partition fields inserted into the target table are inconsistent with the field order of the query source table and the schema field order of the query source table.
Please look at
regression-test/suites/external_table_p0/hive/ddl/test_hive_write_type.groovyin the PR.Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...