[regression](partial update) Add cases when the deleted rows have non nullable columns without default value #26776
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
run buildall |
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
zhannngchen
changed the title
bobhan1
force-pushed
the
add-case-for-delete
branch
from
5c45b72 to
e809fd9
Compare
|
run buildall |
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
seawinde
pushed a commit
to seawinde/doris
that referenced
this pull request
Nov 13, 2023
… nullable columns without default value (apache#26776)
seawinde
pushed a commit
to seawinde/doris
that referenced
this pull request
Nov 14, 2023
… nullable columns without default value (apache#26776)
eldenmoon
added a commit
that referenced
this pull request
Nov 23, 2023
* [keyword](decimalv2) Add DecimalV2 keyword #26283 (#26319) * [fix](planner) Fix sample partition table #25912 (#26399) In the past, two conditions needed to be met when sampling a partitioned table: 1. Data is evenly distributed between partitions; 2. Data is evenly distributed between buckets. Finally, the number of sampled rows in each partition and each bucket is the same. Now, sampling will be proportional to the number of partitioned and bucketed rows. * [fix](spark-load)fix-Unique-key-with-MOR-by-sparkload #26383 (#26414) * [fix](nereids)fix bug of select mv in nereids #26235 (#26415) * [improvement](show trash) Fix be restart slow when too many trash files #26147 (#26417) * [fix](planner)should keep at least one slot materialized in agg node #26116 (#26419) * [fix](multi-catalog)add the FAQ for Aliyun DLF and add the fs.xx.impl check #25594 (#26422) * [coverage](pipeline) Remove unless code and add call method for coverage #25552 (#26423) Remove unless code and add call method for coverage * [Fix](statistics)Fix analyze min max sql syntax error. #26240 (#26443) backport #26240 * [fix](auditlog) fix without lock in QueryStatisticsRecvr find (#26441) * [fix](invert index) Fix the timing error when opening the searcher #26401 (#26472) * [fix](nereids)only enable colocate scan for one phase global parttion topn in some condition #26473 (#26481) * [branch-2.0](cherry-pick) Add more indexed column reader be unit test #25652 (#26430) * [enhancement](regression) fault injection for segcompaction test (#25709) (#26305) 1. generalized debug point facilities from docker suites for fault-injection/stubbing cases 2. add segcompaction fault-injection cases for demonstration 3. add -238 TOO_MANY_SEGMENTS fault-injection case for good Co-authored-by: zhengyu <freeman.zhang1992@gmail.com> * [fix](case) rm non-visiable charactor null in out file (#26540) * [fix](load) fix merged row number miscounting because of race condition (#26516) row numbers miscounting because of race condition, will cause load to fail sometimes with warning 'the rows number written doesn't match'. Signed-off-by: freemandealer <freeman.zhang1992@gmail.com> * [test](regression) Add more regression test for FE (#26539) * [test](coverage) Improve test coverage for runtime filter (#26314) (#26547) * [fix](Nereids) RewriteCteChildren not work with cost based rewritter (#26326) (#26530) we use a map to record rewrite cte children result to avoid rewrite twice in cost based rewritter. However, we record cte outer and inner in one map, and use null as outer result's key, use cte id as inner result's key. This is wrong, because every anchor has an outer, and we could only record one outer. So when we use the cache in cost based rewritter, we get wrong outer plan from the cache. Then the error will be thrown as below: ``` Caused by: java.lang.IllegalArgumentException: Stats for CTE: CTEId#1 not found at com.google.common.base.Preconditions.checkArgument(Preconditions.java:143) ~[guava-32.1.2-jre.jar:?] at org.apache.doris.nereids.stats.StatsCalculator.visitLogicalCTEConsumer(StatsCalculator.java:1049) ~[classes/:?] at org.apache.doris.nereids.stats.StatsCalculator.visitLogicalCTEConsumer(StatsCalculator.java:147) ~[classes/:?] at org.apache.doris.nereids.trees.plans.logical.LogicalCTEConsumer.accept(LogicalCTEConsumer.java:111) ~[classes/:?] at org.apache.doris.nereids.stats.StatsCalculator.estimate(StatsCalculator.java:222) ~[classes/:?] at org.apache.doris.nereids.stats.StatsCalculator.estimate(StatsCalculator.java:200) ~[classes/:?] at org.apache.doris.nereids.jobs.cascades.DeriveStatsJob.execute(DeriveStatsJob.java:108) ~[classes/:?] at org.apache.doris.nereids.jobs.scheduler.SimpleJobScheduler.executeJobPool(SimpleJobScheduler.java:39) ~[classes/:?] at org.apache.doris.nereids.jobs.executor.Optimizer.execute(Optimizer.java:51) ~[classes/:?] at org.apache.doris.nereids.jobs.rewrite.CostBasedRewriteJob.getCost(CostBasedRewriteJob.java:98) ~[classes/:?] at org.apache.doris.nereids.jobs.rewrite.CostBasedRewriteJob.execute(CostBasedRewriteJob.java:64) ~[classes/:?] at org.apache.doris.nereids.jobs.executor.AbstractBatchJobExecutor.execute(AbstractBatchJobExecutor.java:119) ~[classes/:?] at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.visit(RewriteCteChildren.java:72) ~[classes/:?] at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.visit(RewriteCteChildren.java:56) ~[classes/:?] at org.apache.doris.nereids.trees.plans.visitor.PlanVisitor.visitLogicalSink(PlanVisitor.java:118) ~[classes/:?] at org.apache.doris.nereids.trees.plans.visitor.SinkVisitor.visitLogicalResultSink(SinkVisitor.java:72) ~[classes/:?] at org.apache.doris.nereids.trees.plans.logical.LogicalResultSink.accept(LogicalResultSink.java:58) ~[classes/:?] at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.visitLogicalCTEAnchor(RewriteCteChildren.java:86) ~[classes/:?] at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.visitLogicalCTEAnchor(RewriteCteChildren.java:56) ~[classes/:?] at org.apache.doris.nereids.trees.plans.logical.LogicalCTEAnchor.accept(LogicalCTEAnchor.java:60) ~[classes/:?] at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.visitLogicalCTEAnchor(RewriteCteChildren.java:86) ~[classes/:?] at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.visitLogicalCTEAnchor(RewriteCteChildren.java:56) ~[classes/:?] at org.apache.doris.nereids.trees.plans.logical.LogicalCTEAnchor.accept(LogicalCTEAnchor.java:60) ~[classes/:?] at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.rewriteRoot(RewriteCteChildren.java:67) ~[classes/:?] at org.apache.doris.nereids.jobs.rewrite.CustomRewriteJob.execute(CustomRewriteJob.java:58) ~[classes/:?] at org.apache.doris.nereids.jobs.executor.AbstractBatchJobExecutor.execute(AbstractBatchJobExecutor.java:119) ~[classes/:?] at org.apache.doris.nereids.NereidsPlanner.rewrite(NereidsPlanner.java:275) ~[classes/:?] at org.apache.doris.nereids.NereidsPlanner.plan(NereidsPlanner.java:218) ~[classes/:?] at org.apache.doris.nereids.NereidsPlanner.plan(NereidsPlanner.java:118) ~[classes/:?] at org.apache.doris.nereids.trees.plans.commands.ExplainCommand.run(ExplainCommand.java:81) ~[classes/:?] at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:550) ~[classes/:?] ``` * [fix](Nereids) could not run query with repeat node in cte (#26330) (#26531) pick from master PR: #26330 commit id: a89477e ExpressionDeepCopier not process VirtualReference, so we generate inline plan with mistake. * [opt](Nereids) remove Nondeterministic trait from date related functions (#26444) (#26568) * change version to 2.0.3-rc03dev * [fix](regression-test) Fix regiressin test syncer suit use master fe directly (#26456) (#26583) Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com> * Revert "[improvement](scanner_schedule) reduce memory consumption of scanner #24199 (#25547)" (#26613) This reverts commit 9a19581 to investigate ANALYZE TABLE WITH SYNC problem * [enhancement](Nereids): add LOG info to show the phase of NereidsPlanner. (#26542) * [opt](regression test) Add string-like column order by test #26379 (#26533) * [Feature](auditloader) Plugin auditloader use auth token to avoid using cleartext passwords in config (#26278) (#26532) Doris FE will check if stream load http request has auth token after checking password failed; Plugin audit-log loader can use auth token if plugin config set use_auth_token to true Co-authored-by: Mingyu Chen <morningman.cmy@gmail.com> * [branch-2.0](JdbcCatalog) fix that the predicate column name does not have back quote when querying the JDBC appearance (#26479) (#26560) master pr: #26479 * [fix](prepare statement) Not supported such prepared statement if prepare a forward master sql (#26512) (#26638) * [Pick-2.0](regression) add failure injection in inverted index writer #26121 (#26376) * [fix](regression) fix regression framework bug: if real test result is negative, it will miss check test result #25734 (#25734) (#26551) * [Branch-2.0](regression-test) Add tvf regression tests #26322 #26455 (#26566) * [fix](BE)Branch-2.0 unknown runtime filter when get filter from _consumer_map (#26570) * [regression-test](framework) support Non concurrent mode #26487 (#26574) * [regression-test](fix) fix case bug #26561 (#26578) * [fix](backup) Add repo id to local meta/info files to avoid overwriting #26536 (#26622) The local meta/info files generated during backup are not distinguished by repo names. If two backup jobs with the same name are submitted to different repos at the same time, meta/info may be overwritten by another backup job. * [cases](regression-test) Add backup & restore test case #26490 #26491 (#26623) * [case](regression) Adapt show create table and views to 2.0 (#26624) * [fix](regression-test) add more check to address flaky test_partial_update_with_delete_stmt #26474 (#26628) * [feature](Nereids): push down topN through join #24720 (#26634) Push TopN through Join. JoinType just can be left/right outer join or cross join, because data of their one child can't be filtered. new TopN is (original limit + original offset, 0) as limit and offset. (cherry picked from commit 3c9ff7a) * [Test](statistics) Add test cases for external table statistics #26511 (#26636) 1. Test for close and open auto collection for external catalog. 2. Test for analyze table table_name (column) and whole table. * [fix](runtime filter) append late arrival runtime filters in vfilecanner #25996 (#26640) `VFileScanner` will try to append late arrival runtime filters in each loop of `ScannerScheduler::_scanner_scan`. However, `VFileScanner::_get_next_reader` only generates the `_push_down_conjuncts` in the first loop, so the late arrival runtime filters are ignored. * [fix](information_schema)fix bug that metadata_name_ids error tableid and append information_schema case #26238 (#26646) fix bug that #24059 . Added some information_schema scanner tests. files schema_privileges table_privileges partitions rowsets statistics table_constraints Based on infodb_support_ext_catalog=false, it currently includes tests for all tables under the information_schema database. * [Improve](map)Map impli cast #26126 (#26654) * [chore](regression) Do stale resource reclaim before executing cold heat separation p2 case #26596 (#26660) * fix shrink in topN for complext type #26609 (#26661) * [fix](planner) Fix decimal precision and scale wrong when create table like #25802 (#26666) Use field datatype such as decimal(10, 0) to create table like. Because the scale is 0, the precision and scale will lost when create table like done. this will fix the bug. **Before fix, create table with following SQL**: CREATE TABLE IF NOT EXISTS db_test.table_test ( `name` varchar COMMENT "1m size", `id` SMALLINT COMMENT "[-32768, 32767]", `timestamp0` decimal null comment "c0", `timestamp1` decimal(38, 0) null comment "c1" ) DISTRIBUTED BY HASH(`id`) BUCKETS 1 PROPERTIES ('replication_num' = '1'); **and Then run** CREATE TABLE db_test.table_test_like LIKE db_test.table_test SHOW CREATE TABLE db_test.table_test_like; the field `timestamp1` will be decimal(9, 0), it's wrong. this will fix it. Co-authored-by: JingDas <114388747+JingDas@users.noreply.github.com> * [fix](test) fix sql block rule test (#26671) * [Coverage](BE) Delete vinfo_func in BE #26562 (#26674) * [Fix](partial update) Fix core when successfully schema change and load during a partial update #26210 (#26518) * [typo] copy branch master docs to branch-2.0 (#26703) * [typo] update sql-functions to upper-case (#26706) * [Bug](cherry-pick) Add status dispose in branch 2.0 beta rowset reader (#26684) * (selectdb-cloud) Reduce FE db lock range for ShowDataStmt #26588 (#26621) Reduce read lock critical sections and avoid execution timeouts * [brach-2.0](pick)use 2 phase agg above union all #26245 (#26664) * [bug](bitmap) fix bitmap value copy operator not call reset #26451 (#26681) when a empty bitmap assign to other bitmap the other bitmap should reset self firstly, and then set empty type. * [fix](planner)isnull predicate can't be safely constant folded in inlineview #25377 (#26685) * [fix](nereids)unnest in-subquery with agg node in proper condition #25800 (#26687) * [fix](nereids)add visitMarkJoinReference method in ExpressionDeepCopier #25874 (#26688) * [fix](nereids)don't normalize column name for base index #26476 (#26690) * [fix](planner)cast floating point type to bigint for bit functions #26598 (#26691) * [fix](Nereids) storage later agg rule process agg children by mistake #26101 (#26698) pick from master PR #26101 commit id c0ed5f7 update Project#findProject agg function's children could be any expression rather than only slot. we use Project#findProject to process them. But this util could only process slot. This PR update this util to let it could process all type expression. * [fix](Nereids) time extract function constant folding core (#26292) (#26699) pick from master PR: #26292 commit id: 74fd5da some time extract function changed return type in the previous PR #18369 but it is not change FE constant folding function signature. This is let them have same signature to avoid BE core. * [fix](Nereids) only search internal funcftion when dbName is empty (#26296) (#26700) pick from master PR: #26296 commit id: 6892fc9 if call function with database name. we should only search UDF * [fix](Nereids) ban right outer, right anti, full outer with bucket shuffle (#26529) (#26702) pick from master PR: #26529 commit id: f80495d if left bucket has no data, we do not generate left bucket instance. These join should reserve all right side data. But because left instance is not exists. So right data will be discard since no dest be set. We ban these join temporarily until we could generate all instance for left side in Coordinator. * [test](statistics)Add hive statistics all data type p0 test (#26676) (#26715) * [test](serialisation) Serialise some cases and enable str_to_date tests #26651 (#26716) 1 enable the cases about str_to_date, which have been muted because some parallel config influence. 2 serialise some cases which called admin set config * Revert "[Coverage](BE) Delete vinfo_func in BE #26562 (#26674)" (#26724) This reverts commit 22eafa4. * [fix](regression-test) add tests for jdbc catalog (#26608) (#26719) * [fix](nereids)SimplifyRange rule may mess up and/or predicate #26304 (#26693) * [Fix](fs_benchmark_tools) Fix `run_fs_benchmark.sh` classpath issue. (#26183) (#26704) Backport from #26183. * [Fix](partial update) Fix core when doing partial update on tables with row column after schema change #26632 (#26695) * [Opt](orc-reader) Optimize orc string dict filter in not_single_conjunct case. (#26386) (#26696) Optimize orc/parquet string dict filter in not_single_conjunct case. We can optimize this processing to filter block firstly by dict code, then filter by not_single_conjunct. Because dict code is int, it will filter faster than string. For example: ``` select count(l_receiptdate) from lineitem_date_as_string where l_shipmode in ('MAIL', 'SHIP') and l_commitdate < l_receiptdate and l_receiptdate >= '1994-01-01' and l_receiptdate < '1995-01-01'; ``` `l_receiptdate` and `l_shipmode` will using string dict filtering, and `l_commitdate < l_receiptdate` is the an not_single_conjunct which contains dict filter field. We can optimize this processing to filter block firstly by dict code, then filter by not_single_conjunct. Because dict code is int, it will filter faster than string. Before: mysql> select count(l_receiptdate) from lineitem_date_as_string where l_shipmode in ('MAIL', 'SHIP') and l_commitdate < l_receiptdate and l_receiptdate >= '1994-01-01' and l_receiptdate < '1995-01-01'; +----------------------+ | count(l_receiptdate) | +----------------------+ | 49314694 | +----------------------+ 1 row in set (6.87 sec) After: mysql> select count(l_receiptdate) from lineitem_date_as_string where l_shipmode in ('MAIL', 'SHIP') and l_commitdate < l_receiptdate and l_receiptdate >= '1994-01-01' and l_receiptdate < '1995-01-01'; +----------------------+ | count(l_receiptdate) | +----------------------+ | 49314694 | +----------------------+ 1 row in set (4.85 sec) * [docs](docs) Update Files of Branch-2.0 (#26737) * [date](parser) Support DateV1 keyword (#25414) (#26746) * [Fix](orc-reader) Fix orc complex types when late materialization was turned on by disabling late materialization in this case. (#26548) (#26743) Fix orc complex types when late materialization was turned on in orc reader by disabling late materialization in this case. * [fix](udf)java udf does not support overloaded evaluate method (#22681) (#26768) Co-authored-by: HB <hubiao01@corp.netease.com> * [fix](show_proc) fix show statistic proc dir to ensure that result only contains dbs in internal catalog (#26254) (#26763) backport #26254 Co-authored-by: caiconghui <55968745+caiconghui@users.noreply.github.com> * [Enhancement](sql-cache) Use update time of hive to avoid cache miss through multi fe nodes. (#26424) (#26762) backport #26424 * [Fix](partial update) Fix partial update info loss when the delete bitmaps of the committed transactions are calculated by the compaction #26556 (#26735) * [hotfix](editlog) Fix upsert replay on follower not contains loadedTableIndexIds (#26597) (#26756) * [chore](regression-test) Fix error add partition operation due to duplicate partition range #26742 (#26758) * [Bug](materialized-view) fix some bugs on create mv with percentile_approx (#26528) (#26764) 1. percentile_approx have wrong symbol 2. fnCall.getParams() get obsolete childrens * [Bug](agg-state) fix file load insert wrong data to agg_state (#26581) (#26765) * [Bug](decimalv2) getCmpType return decimalv2 when lhs/rhs type both is decimalv2 (#26705) (#26767) * [fix](Nereids) fix plan shape of query64 unstable (#26012) (#26775) don't remove the physical plan after optimizing the plan in dphyper. * [FIX](complextype) fxi array nested struct literal #26270 (#26778) * [improvement](disk balance) Prevent duplicate disk balance tasks afte… (#25990) (#26745) * [branch-2.0](transaction) Fix publish txn wait too long when not meet quorum #26659 (#26759) * [bugfix](clickhouse) fix datetime convert error. (#26128) (#26766) Co-authored-by: Guangdong Liu <liugddx@gmail.com> * [Fix](row store) cache invalidate key should not include sequence column #26771 (#26780) * [branch-2.0](pick) support HTTP request with chunked transfer (#26520) (#26785) * [feature](nestedType) add nested data type to create table tool (#26787) * [fix](hudi) fix wrong schema when query hudi table on obs #26789 (#26791) * [fix](decimal) fix undefined behaviour of divide by zero when cast string to decimal (#26792) * [fix](refresh) fix priv issue of refresh database and table operation #26793 (#26794) * [minor] add disable swap command tip (#26798) * [fix](information_schema) fix test_query_sys_tables schema_privileges regression case #26753 (#26800) * [branch-2.0] fix test result (#26801) fix output error from #26743 On master branch, the value in struct field is wrapped by quota, but on branch 2.0, the value in struct field is NOT wrapped by quota * fix: restore load job progress before retry load task (#26802) Co-authored-by: chenboyang.922 <chenboyang.922@bytedance.com> * [fix](thrift)limit be and fe thrift server max pkg size,avoid accepting error or too large package causing OOM #26179 (#26805) * [fix](Planner): don't push down isNull predicate into view (#26288) (#26773) * [opt](scanner) increase the connection num of s3 client #26795 (#26796) * [enhancement](metrics) enhance visibility of flush thread pool (#26544) (#26819) * [fix](regression) move fault-injection data to the right place (#26825) Signed-off-by: freemandealer <freeman.zhang1992@gmail.com> * [feature](binlog) Add ingest_binlog/http_get_snapshot limit download speed && Add async ingest_binlog (#26323) (#26733) * [fix](jdbc catalog) fix mysql zero date (#26569) (#26837) * [ci](pipeline) add tpch sff100 test on branch-2.0 (#26824) * [pick](nerieds) make AGG_SCALAR_SUBQUERY_TO_WINDOW_FUNCTION rewrite rule #25969 (#26852) * [enhancement](230) print max version and spec version when -230 happens (#26643) (#26854) * [chore](fs) Don't print the stack for file system and it's derived class #26814 (#26838) * [compile](gcc) fix gcc compile error #26863 * [test](jdbc) pick some jdbc test from branch master (#26860) * [pipeline](exec) disable shared scan in default and disable shared scan in limit with where scan (#25952) (#26815) * [regression](partial update) Add cases when the deleted rows have non nullable columns without default value #26776 (#26848) * [feature](fe) Add coverage tool for FE UT (#26203) (#26857) * [fix](map) the implementation of ColumnMap::replicate was incorrect (#26647) (#26868) * [fix](broker load) pass loadToSingleTablet to olapTableSink (#26680) (#26869) * [regression-test](framework) Support running tests multiple times and reporting correctly to TeamCity (#26606) (#26871) * [refactor](stats) refactor collection logic and opt some config #26163 (#26858) picked from #26163 * [bug](user login)fix PASSWORD_LOCK_TIME setting UNBOUNDED does not take effect #26585 (#26859) * [Improvement](statistics)Improve stats sample strategy (#26435) (#26890) backport #26435 Improve the accuracy of sample stats collection. For non distribution columns, use `n*d / (n - f1 + f1*n/N)` where `f1` is the number of distinct values that occurred exactly once in our sample of n rows (from a total of N), and `d` is the total number of distinct values in the sample. For distribution columns, use `ndv(n) * fraction of tablets sampled` for NDV. For very large tablet to sample, use limit to control the total lines to scan (for non key column only, because key column is sorted and will be inaccurate using limit). * [fix](partial update) Fix NPE when the query statement of an update statement is a point query in OriginPlanner #26881 (#26900) * [bug](function) add signature for precentile function (#26867) (#26926) Co-authored-by: zhangstar333 <87313068+zhangstar333@users.noreply.github.com> * enable pipeline and nereids in test-pipeline (#26918) * [Fix](Planner) fix varchar does not show real length #25171 (#26850) * [improvement](statistics)Multi bucket columns using DUJ1 to collect ndv #26950 (#26976) backport #26950 * [fix](statistics)Fix external table show column stats type bug #26910 (#26921) backport: #26910 * [minor](stats) rename stats related session variable name #26936 (#26928) * [nereids](datetime) fix wrong result type of datetime add with interval as first arg (#26957) (#26987) * [fix](Nereids) column pruning under union broken unexpectedly (#26884) (#26985) * [fix](catalog) Fix ClickHouse DataTime64 precision parsing (#26980) * [opt](MergeIO) use equivalent merge size to measure merge effectiveness (#26741) (#26923) backport #26741 * add defensive code in runtime predicate to avoid crash due to column not in tablet schema #26990 (#26991) * [fix](stats) fix auto collector always create sample job no matter the table size #26968 (#26972) * [Enhance](regression)enhance docker network by add docker network subnet (#26872) * [fix](case) regression-test/suites/show_p0/test_show_statistic_proc.groovy (#26925) Co-authored-by: stephen <hello-stephen@qq.com> * [fix](auth) fix overwrite logic of user with domain (#27003) backport #27002 * [Branch-2.0](Serde) Fix content displayed by complex types in MySQL Client (#26880) backport #25946 and #26301 * [test](tvf) append tvf read hive_text file regression case. (#26790) (#26989) backport #26790 * [test](information_schema)append information_schema external_table_p0 case. (#27029) backport : #26846 * [fix](parquet) compressed_page_size has the same meaning in page v1 and v2 (#26783) (#26922) backport #26783 * [BugFix](JDBC Catalog) fix jdbc catalog query bitmap may cause be core sometimes (#26933) (#27018) * [Enhance](regression) skip test_information_schema_external (#27058) * [improvement](pipeline) task group scan entity (#19924) (#27040) Co-authored-by: Lijia Liu <liutang123@yeah.net> * [opt](pipeline) Return InternalError to FE instead of doing a useless DCHECK in ExecNode #27035 (#27057) Effect: Client will see error message like below when BE meeting plan logical error. RROR 1105 (HY000): errCode = 2, detailMessage = ([xxx]())[CANCELLED]Logical error during processing VNewOlapScanNode(dr_case_tag), output of projections 2 mismatches with exec node output 3 * [fix](nereids)Fix nereids fail to parse tablesample bug (#26982) backport #26981 * [branch2.0](test) fix external table test case with nested type display (#27092) * [fix](load) skip cancel already cancelled channels (#27109) * [fix](Nereids) store user variable in connect context (#26655) (#26920) pick from master #26655 1. user variable should be case insensitive 2. user variable should be cleared after the connection reset * [test](parquet)append parquet reader byte_array_decimal and rle_bool case (#26751) (#27026) backport #26751 * [fix](nereids) support uncorrelated subquery in join condition (#26893) pick from master #26672 commit id: 17b1108 * [Bug](pipeline) try fix the exchange sink buffer result error (#27087) * [fix](function)return NULL rather than 'null' if path not found #25880 (#26823) * [enhancement](nereids)make error message more readable when bind logicalRepeat node #26744 (#26895) * [regression](delete) add delete case for every type (#26961) * [branch-2.0](paimon)disable paimon decimal case (#26971) * [regression](partial update) Add row store cases for all existing partial update cases #26924 (#27017) * [fix](statistics) fix updated rows incorrect due to typo in code #26979 (#27034) * [fix](typo) Use minutes as auto analyze schedule interval #26968 (#27041) * [Improvement](function) opt for case when #23068 (#27054) * [fix](planner)scan node should project all required expr from parent node #26886 (#27096) * [fix](nereids)count in correlated subquery shoud not output null value #27064 (#27097) * [fix](load) add lock in active_memtable_mem_consumption #25207 (#27100) * [branch-2.0](suites) Enable test_cast_with_scale_type since Nereids is ON (#26986) * [Fix](multi-catalog) Fix NPE when replaying hms events #26803 (#26997) Co-authored-by: wangxiangyu <wangxiangyu@360shuke.com> * [Opt](scanner-scheduler) Optimize `BlockingQueue`, `BlockingPriorityQueue` and change remote scan thread pool #26784 (#27053) - Optimize `BlockingQueue`, `BlockingPriorityQueue` by swapping `notify` and `unlock` to reduce lock competition. Ref: https://www.boost.org/doc/libs/1_54_0/boost/thread/sync_bounded_queue.hpp - Change remote scan thread pool to `PriorityQueue`. * [fix](errmsg) fix multiple FE processes start err msg (#27009) (#27080) * [FIX](regresstest) fix test_load_with_map_nested_array csv for id #27105 (#27107) * [FIX](map)fix map nested decimal with element at #27030 (#27110) * [feature](tvf)(jni-avro)jni-avro scanner add complex data types (#26236) (#26731) * [fix](nereids)fix bug that query infomation_schema.rowsets fe send fragment to one of muilti be. (#27025) (#27090) Fixed the bug of incomplete query results when querying information_schema.rowsets in the case of multiple BEs. The reason is that the schema scanner sends the scan fragment to one of multiple bes, and be queries the information of fe through rpc. Since the rowsets information requires information about all BEs, the scan fragment needs to be sent to all BEs. * [Config](statistics)Set enable_auto_analyze default value to true. #27146 * [branch2.0](test) fix doris jdbc catalog test case (#27150) 1. Fix doris_jdbc_catalog test case out file 2. Add log to debug 2 unstable test cases: pg_jdbc_catalog and oracle_jdbc_catalog * [bugfix](tablet)fix the tablet will be deleted when clone due to concurrency #25784 (#26777) * [fix](sink) crash caused by wild pointer of counter in VDataStreamSender (#26947) (#27148) If preparation fails, the counter _peak_memory_usage_counter will be a wild pointer. *** SIGSEGV address not mapped to object (@0x454d49545f) received by PID 16992 (TID 18856 OR 0x7f4d05444700) from PID 1296651359; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:417 1# os::Linux::chained_handler(int, siginfo*, void*) in /app/doris/Nexchip-doris-1.2.4.2-bin-x86_64/java8/jre/lib/amd64/server/libjvm.so 2# JVM_handle_linux_signal in /app/doris/Nexchip-doris-1.2.4.2-bin-x86_64/java8/jre/lib/amd64/server/libjvm.so 3# signalHandler(int, siginfo*, void*) in /app/doris/Nexchip-doris-1.2.4.2-bin-x86_64/java8/jre/lib/amd64/server/libjvm.so 4# 0x00007F55C85B9400 in /lib64/libc.so.6 5# doris::vectorized::VDataStreamSender::close(doris::RuntimeState*, doris::Status) at /root/doris/be/src/vec/sink/vdata_stream_sender.cpp:734 6# doris::PlanFragmentExecutor::close() at /root/doris/be/src/runtime/plan_fragment_executor.cpp:543 7# doris::PlanFragmentExecutor::~PlanFragmentExecutor() at /root/doris/be/src/runtime/plan_fragment_executor.cpp:95 8# doris::FragmentExecState::~FragmentExecState() at /root/doris/be/src/runtime/fragment_mgr.cpp:112 9# std::_Sp_counted_ptr<doris::FragmentExecState*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() at /root/ldb/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:348 10# doris::FragmentMgr::exec_plan_fragment(doris::TExecPlanFragmentParams const&, std::function<void (doris::RuntimeState*, doris::Status*)> const&) at /root/doris/be/src/runtime/fragment_mgr.cpp:855 11# doris::FragmentMgr::exec_plan_fragment(doris::TExecPlanFragmentParams const&) at /root/doris/be/src/runtime/fragment_mgr.cpp:592 12# doris::PInternalServiceImpl::_exec_plan_fragment_impl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, doris::PFragmentRequestVersion, bool) at /root/doris/be/src/service/internal_service.cpp:463 13# doris::PInternalServiceImpl::_exec_plan_fragment_in_pthread(google::protobuf::RpcController*, doris::PExecPlanFragmentRequest const*, doris::PExecPlanFragmentResult*, google::protobuf::Closure*) at /root/doris/be/src/service/internal_service.cpp:305 14# doris::WorkThreadPool<false>::work_thread(int) at /root/doris/be/src/util/work_thread_pool.hpp:160 15# execute_native_thread_routine at ../../../../../libstdc++-v3/src/c++11/thread.cc:84 16# start_thread in /lib64/libpthread.so.0 17# clone in /lib64/libc.so.6 * [improvement](log) log desensitization without displaying user info (#26912) (#27167) * [branch-2.0](cherry-pick) add chunked transfer json test (#26902) (#27164) * [fix](statistics)Fix alter column stats bug (#27093) (#27189) backport #27093 * (fix)[schema change] fix incorrect setting of schema change jobstate when replay editlog (#26992) (#27139) * [fix](jni) avoid BE crash and NPE when close paimon reader #27129 (#27204) bp #27129 * [enhancement](jdbc catalog) Add lowercase column name mapping to Jdbc data source & optimize database and table mapping #27124 (#27130) * [case] Load json data with enable_simdjson_reader=false (#26601) (#27158) Co-authored-by: HowardQin <hao.qin@esgyn.cn> * [fix](function) fix error when use negative number in explode_numbers #27020 (#27180) * [fix](iceberg) iceberg use customer method to encode special characters of field name (#27108) (#27205) Fix two bugs: 1. Missing column is case sensitive, change the column name to lower case in FE for hive/iceberg/hudi 2. Iceberg use custom method to encode special characters in column name. Decode the column name to match the right column in parquet reader. * [enhancement](binlog) Add dbName && tableName in CreateTableRecord (#26901) (#27208) * [Branch2.0](Export) add show export regression testes #27140 (#27160) * [log](tablet invert) add preconditition check failed log (#26770) (#27171) * [branch-2.0](publish version) publish version task no need return VERSION_NOT_EXIST #27005 (#27174) * [minor](stats) Add start/end time for analyze job, precise to seconds of TableStats update time #27123 (#27185) * [test](regression) Add more alter stmt regression case (#26988) (#27193) * [test](external_table_p0)append log in external_table_p0 for debug unknown table case #27212 (#27213) * [Improve](txn) Add some fuzzy test stub in txn (#26712) (#27144) * [branch-2.0](fe ut) fix decommission test #27082 (#27175) * [Fix](multi-catalog) Fix complex type crash when using dict filter facility in the parquet-reader. (#27151) (#27187) - Fix complex type crash when using the dict filter facility in the parquet-reader by turning off the dict filter facility in this case. - Add orc complex types regression test. * [Optimize](point query) clear names to reduce mem consumption and cpu cost related to block column name (#26931) (#27157) * [fix](fe) Fix `enable_nereids_planner` forward not take effect (#26782) (#27159) * The java reflection method `getFields()` only return public fields, but enable_nereids_planner is private * [fix](fe ut) Fix borrow oject throw npe (#27072) (#27207) occasional failure of fe ut, borrowObject throw npe ``` get agent task request. type: CREATE, signature: 10008, fe addr: null java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) at org.apache.commons.pool2.impl.GenericKeyedObjectPool.register(GenericKeyedObjectPool.java:1079) at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:347) get agent task request. type: CREATE, signature: 10012, fe addr: TNetworkAddress(hostname:127.0.0.1, port:56072) at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:277) at org.apache.doris.common.GenericPool.borrowObject(GenericPool.java:99) at org.apache.doris.utframe.MockedBackendFactory$DefaultBeThriftServiceImpl$1.run(MockedBackendFactory.java:219) at java.lang.Thread.run(Thread.java:750) ``` * [regression](conf) Make checkpoint/clean thread trigger more frequent (#26883) (#27194) * When run p0, we want some checkpoint/clean thread in FE work more frequently * [hotfix](priv) Fix restore snapshot user priv with add cluster in UserIdentity (#26969) (#27210) Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com> * [branch-2.0](fe ut) fix unstable test DecommissionBackendTest (#27173) * [fix](disk migrate) migrate ignore not exists tablet (#26779) (#27172) * [fix](build)macos clang 15 version compilation error (#25457) * [fix](tablet sched) fix sched delete stale remain replica (#27050) (#27179) * Revert "[Branch2.0](Export) add show export regression testes #27140 (#27160)" (#27217) This reverts commit d76581d, since it caused test_show_export testcase fail. * Revert "[test](regression) Add more alter stmt regression case (#26988) (#27193)" (#27216) This reverts commit 42d4806, since it caused test_alter_table_drop_column and test_alter_table_modify_column testcases fail. * [fix](nereids)remove literal partition by and order by expression in window function #26899 (#27214) * [fix](agg) fix coredump of multi distinct of decimal128I (#27014) (#27228) * [fix](agg) fix coredump of multi distinct of decimal128 * fix * Revert "[enhancement](jdbc catalog) Add lowercase column name mapping to Jdbc data source & optimize database and table mapping #27124 (#27130)" (#27230) This reverts commit 087fccd. * [feature](Nereids): eliminate sort under subquery (#26993) (#27218) * [fix](ccr) Mark getBinlog,getBinlogLag,getMeta,getBackendMeta as from master (#27211) (#27227) Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com> * [fix](Nereids): NullSafeEqual should be in HashJoinCondition #27127 (#27232) * [fix](planner)the data type should be the same between input slot and sort slot #27137 (#27215) * [branch2.0](nereids)Pick #26873 #25769: partition prune fix (#27222) * [improvement](fe and broker) support specify broker to getSplits, check isSplitable, file scan for HMS Multi-catalog (#24830) (#27236) bp #24830 * [fix](fe ut) fix unstable ut TabletRepairAndBalanceTest (#27044) (#27239) * [minor](stats) Report error with more friendly meesage when timeout #27197 (#27240) * [fix](build index) Fix inverted index hardlink leak and missing problem #26903 (#27244) * [fix](multi-catalog)add the max compute fe ut and fix download expired #27007 (#27220) bp #27007 * [cherry-pick](regression) add hms catalog broker scan case (#25453) (#27253) * [cherry-pick](fe) select BE local broker to scan Hive table when 'broker.name' in hms catalog is specified (#27122) (#27252) Since #24830 introduce `broker.name` in hms catalog, data scan will run on specified brokers. And [doris operator](https://github.com/selectdb/doris-operator) support BE and broker deployed in same pod, BE access local broker is the fastest approach to access data. In previous logic, every inputSplit will select one BE to execute, then randomly select one broker for actual data access, BE and related broker are always located on separate K8S pod. This pr optimizes the broker select strategy to prioritize BE-local broker when `broker.name` is specified in hms catalog. * [improvement](statistics)Use count as ndv for unique/agg olap table single key column (#27186) (#27275) Single key column of unique/agg olap table has the same value of count and ndv, for this kind of column, don't need to calculate ndv, simply use count as ndv. backport #27186 * [minor](stats) Fix potential npe when loading stats #27200 (#27241) * [fix](tablesample) Fix computeSampleTabletIds NullPointerException (#27165) (#27258) * [fix](partial update) keep case insensitivity and use the columns' origin names in partialUpdateCols in origin planner #27223 (#27255) * [chore](fix) sync check-pr-if-need-run-build.sh with master branch (#27250) * [fix](compile) fix BE compile failure on Mac (#27206) (#27281) * [chore](clucene) coverage compilation option added #27162 (#27284) * [FIX]Fix complex type meta schema in information database #27203 (#27286) * [feature](Nereids): Pushdown LimitDistinct Through Join (#25113) (#27288) Push down limit-distinct through left/right outer join or cross join. such as select t1.c1 from t1 left join t2 on t1.c1 = t2.c1 order by t1.c1 limit 1; * [fix](inverted index) reset fs_writer to nullptr before throw exception (#27202) (#27289) * [fix](planner)output slot should be materialized as intermediate slot in agg node #27282 (#27285) * [FIX](complextype)Fix complex nested and add regress test #26973 (#27293) * [fix](test) disable forbid_unknown_col_stats (#27303) * [fix](stats) Release analyze tasks once job finished #27310 (#27309) * [doc](fix) a new docs for k8s deploy by operator to 2.0 (#26927) * [doc](fix) fix date trunc doc (#27320) * [Fix](statistics)Fix analyze sql including key word bug #27321 (#27322) backport #27321 * [cherry-pick](function) improve compoundPred optimization work with children is nullable #26160 (#27354) * Revert "[improvement](routine-load) add routine load rows check (#25818)" (#27336) * [refactor](planner) filter empty partitions in a unified location (#27190) (#27256) * [fix](hms) fix compatibility issue of hive metastore client #27327 (#27328) * [Branch2.0](Export) add show export regression testes (#27330) * [fix](stats) Fix thread leaks when doing checkpoint #27334 #27335 * [fix](stats) Fix creating too many tasks on new env (#27362) * [fix](build index) fix core when build index for a new column which without data (#27276) * change version to 2.0.3-rc04 (#27392) * fix merge * update clucene --------- Signed-off-by: freemandealer <freeman.zhang1992@gmail.com> Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com> Co-authored-by: Gabriel <gabrielleebuaa@gmail.com> Co-authored-by: Xinyi Zou <zouxinyi02@gmail.com> Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com> Co-authored-by: starocean999 <40539150+starocean999@users.noreply.github.com> Co-authored-by: deardeng <565620795@qq.com> Co-authored-by: slothever <18522955+wsjz@users.noreply.github.com> Co-authored-by: HappenLee <happenlee@hotmail.com> Co-authored-by: Jibing-Li <64681310+Jibing-Li@users.noreply.github.com> Co-authored-by: Mryange <59914473+Mryange@users.noreply.github.com> Co-authored-by: zzzxl <33418555+zzzxl1993@users.noreply.github.com> Co-authored-by: abmdocrt <Yukang.Lian2022@gmail.com> Co-authored-by: HHoflittlefish777 <77738092+HHoflittlefish777@users.noreply.github.com> Co-authored-by: zhengyu <freeman.zhang1992@gmail.com> Co-authored-by: Dongyang Li <hello_stephen@qq.com> Co-authored-by: walter <w41ter.l@gmail.com> Co-authored-by: morrySnow <101034200+morrySnow@users.noreply.github.com> Co-authored-by: Kang <kxiao.tiger@gmail.com> Co-authored-by: Jack Drogon <jack.xsuperman@gmail.com> Co-authored-by: jakevin <jakevingoo@gmail.com> Co-authored-by: zhiqiang <seuhezhiqiang@163.com> Co-authored-by: Mingyu Chen <morningman.cmy@gmail.com> Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com> Co-authored-by: meiyi <myimeiyi@gmail.com> Co-authored-by: airborne12 <airborne08@gmail.com> Co-authored-by: TengJianPing <18241664+jacktengg@users.noreply.github.com> Co-authored-by: minghong <englefly@gmail.com> Co-authored-by: shuke <37901441+shuke987@users.noreply.github.com> Co-authored-by: walter <patricknicholas@foxmail.com> Co-authored-by: zhannngchen <48427519+zhannngchen@users.noreply.github.com> Co-authored-by: Ashin Gau <AshinGau@users.noreply.github.com> Co-authored-by: daidai <2017501503@qq.com> Co-authored-by: amory <wangqiannan@selectdb.com> Co-authored-by: AlexYue <yj976240184@gmail.com> Co-authored-by: seawinde <149132972+seawinde@users.noreply.github.com> Co-authored-by: JingDas <114388747+JingDas@users.noreply.github.com> Co-authored-by: zclllyybb <zhaochangle@selectdb.com> Co-authored-by: bobhan1 <bh2444151092@outlook.com> Co-authored-by: Jeffrey <color.dove@gmail.com> Co-authored-by: zhangstar333 <87313068+zhangstar333@users.noreply.github.com> Co-authored-by: Qi Chen <kaka11.chen@gmail.com> Co-authored-by: KassieZ <139741991+KassieZ@users.noreply.github.com> Co-authored-by: Mingyu Chen <morningman@163.com> Co-authored-by: HB <hubiao01@corp.netease.com> Co-authored-by: Pxl <pxl290@qq.com> Co-authored-by: 谢健 <jianxie0@gmail.com> Co-authored-by: yujun <yu.jun.reach@gmail.com> Co-authored-by: Guangdong Liu <liugddx@gmail.com> Co-authored-by: zfr95 <87513668+zfr9527@users.noreply.github.com> Co-authored-by: chen <czjourney@163.com> Co-authored-by: TsukiokaKogane <cby141994@gmail.com> Co-authored-by: chenboyang.922 <chenboyang.922@bytedance.com> Co-authored-by: ryanzryu <143597717+ryanzryu@users.noreply.github.com> Co-authored-by: Siyang Tang <82279870+TangSiyang2001@users.noreply.github.com> Co-authored-by: zy-kkk <zhongyk10@gmail.com> Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com> Co-authored-by: Lei Zhang <27994433+SWJTU-ZhangLei@users.noreply.github.com> Co-authored-by: Jerry Hu <mrhhsg@gmail.com> Co-authored-by: qiye <jianliang5669@gmail.com> Co-authored-by: AKIRA <33112463+Kikyou1997@users.noreply.github.com> Co-authored-by: Liqf <109049295+LemonLiTree@users.noreply.github.com> Co-authored-by: LiBinfeng <46676950+LiBinfeng-01@users.noreply.github.com> Co-authored-by: zhangguoqiang <18372634969@163.com> Co-authored-by: stephen <hello-stephen@qq.com> Co-authored-by: GoGoWen <82132356+GoGoWen@users.noreply.github.com> Co-authored-by: wangbo <wangbo@apache.org> Co-authored-by: Lijia Liu <liutang123@yeah.net> Co-authored-by: Kaijie Chen <ckj@apache.org> Co-authored-by: Yulei-Yang <yulei.yang0699@gmail.com> Co-authored-by: lsy3993 <110876560+lsy3993@users.noreply.github.com> Co-authored-by: zhangdong <493738387@qq.com> Co-authored-by: Xiangyu Wang <dut.xiangyu@gmail.com> Co-authored-by: wangxiangyu <wangxiangyu@360shuke.com> Co-authored-by: wudongliang <46414265+DongLiang-0@users.noreply.github.com> Co-authored-by: Houliang Qi <neuyilan@163.com> Co-authored-by: Luwei <814383175@qq.com> Co-authored-by: HowardQin <hao.qin@esgyn.cn> Co-authored-by: jiafeng.zhang <zhangjf1@gmail.com> Co-authored-by: DuRipeng <453243496@qq.com> Co-authored-by: catpineapple <42031973+catpineapple@users.noreply.github.com> Co-authored-by: YueW <45946325+Tanya-W@users.noreply.github.com>
gnehil
pushed a commit
to gnehil/doris
that referenced
this pull request
Dec 4, 2023
… nullable columns without default value apache#26776 (apache#26848)
XuJianxu
pushed a commit
to XuJianxu/doris
that referenced
this pull request
Dec 14, 2023
… nullable columns without default value (apache#26776)
yiguolei
pushed a commit
that referenced
this pull request
Dec 27, 2023
add conditions for mor table removed in #26776
CalvinKirs
added a commit
to CalvinKirs/incubator-doris
that referenced
this pull request
Dec 27, 2023
* [Performance](point query)Opimize partition prune for point query (apache#28150) * [Performance](point query)Opimize partition prune for point query * [fix](stacktrace) ignore stacktrace for error code INVALID_ARGUMENT INVERTED_INDEX_NOT_IMPLEMENTED (apache#27898) (apache#28598) * ignore stacktrace for error INVALID_ARGUMENT INVERTED_INDEX_NOT_IMPLEMENTED * AndBlockColumnPredicate::evaluate * [bugfix](topn) fix coredump in copy_column_data_to_block when nullable mismatch (apache#28597) * [bugfix](topn) fix coredump in copy_column_data_to_block when nullable mismatch return RuntimeError if copy_column_data_to_block nullable mismatch to avoid coredump in input_col_ptr->filter_by_selector(sel_rowid_idx, select_size, raw_res_ptr) . The problem is reported by a doris user but I can not reproduce it, so there is no testcase added currently. * clang format * [feature](Nereids): eliminate semi join (apache#28588) Eliminate Semi/Anti Join which is FALSE or TRUE. * [feature](Nereids) support datev1 and datetimev1 (apache#28581) * [improvement](http) add show_table_data http api (apache#28380) In some cases, users need to get the data size of single replica of a table, and evaluate certain actions based on this, such as estimating the precise backup size. Signed-off-by: nextdreamblue <zxw520blue1@163.com> * [Enhance](regression)enhance jdbc case to adapt to use case concurrency (apache#28565) enhance jdbc case to adapt to use case concurrency * [opt](Nereids)when both Nereids and old parsers report errors, prompt error messages for the Nereids (apache#28580) * [Improvement](regression) change compound predicate regression case name to make it more clear (apache#28612) * [regression](memtable) add case for memtable flush error handle (apache#28285) Co-authored-by: ziyang zhang <zhangziyang@stu.cdut.edu.cn> * [regression-test](memtable) test memtable flush is high priority for vtable writerV1 (apache#28502) * enhance performance for broken tablet checking under multi-core scenario with a coarse-grained read lock (apache#28552) * [fix](load) fix memtracking orphan too large (apache#28600) * [fix](memtable-limiter) do not block write if load mem usage is low (apache#28602) Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com> * [refactor](renamefile) rename some files according to the class names (apache#28606) * [feature](mtmv)create mtmv support refresh_partition_num (apache#28566) - create/alter mtmv support refresh_partition_num - mtmv task according to refresh_partition_num executes refresh tasks in batches - `tasks` tvf add column `CompletedPartitions` and `progress` - fix mtmv can not `show temp partition` and `drop temp partition` - fix task can not get error msg when insert overwrite error - fix when the partition field is capitalized, the verification of creating a mtmv does not pass * [fix](Nereids) stats estimation of lessThan apache#28444 * [fix](regression)Change analyze_timeout to global. (apache#28587) Fix hive statistics regression case. analyze_timeout is a global session variable. * [chore] Add bvar for meta operations of BE (apache#28374) * [Bug](cooldown) Fix problem that followers may never completely cooldown (apache#28561) * [Improve](tvf)jni-avro support split file (apache#27933) * [feature](mtmv)after creating a materialized view, if other operations fail, roll back (apache#28621) after create mtmv, if create job failed,need drop mtmv * [fix](meta) update killed query state (#) (apache#25917) * [refactor](profile&names) using dst_id in pipelinex profile to be same as non pipeline; rename some function names (apache#28626) Co-authored-by: yiguolei <yiguolei@gmail.com> * [feature][executor]support workload schedule policy (apache#28443) * [feature](expr) Support kill query by query_id (apache#28530) Issue Number: open apache#28517 * [chore](user) Add user property `parallel_fragment_exec_instance_num` (apache#28447) * [feature](inverted index) add ignore_above property to prevent long s… (apache#28585) When string is too long, clucene will throw an error. And the string is too long to analyze. So we ignore the string in index process when the string is longer than 256 bytes by default. We add an poperty `ignore_above` for user to customize. * [Enhance](broker) add inputstream expire scheduled checker to avoid memory leak for broker scan (apache#28589) This pr introduces 2 broker conf: 1. enable_input_stream_expire_check: which indicates whether enable inputStream expire check. 2. input_stream_expire_seconds: which indicates the timeout seconds for inputStream since last update. * [fix](hive) add support for `quoteChar` and `seperatorChar` for hive (apache#28613) add support for quoteChar and seperatorChar . * [Fix](transactional-hive) Fix hive transactional table return empty result. (apache#28518) * [Fix](memtable) fix `shrink_memtable_by_agg` without duplicated keys (apache#28660) remove duplicated logic: ``` vectorized::Block in_block = _input_mutable_block.to_block(); _put_into_output(in_block); ``` `_input_mutable_block.to_block()` will move `_input_mutable_block`, and lead to `flush` with empty block * [enhance](partition_id) check partition id before store meta (apache#28055) * [opt](task-assignment) use consistent hash as default task assigner and cache the consistent hash ring (apache#28522) 1. Use consistent hash algo as the default assigner for file query scan node A consistent assignment can better utilize the page cache of BE node. 2. Cache the consistent hash ring Init a consistent hash ring is time-consuming because there a thousands of virtual node need to be added. So cache it for better performance * [Fix](Job)Fixed job scheduling missing certain time window schedules (apache#28659) Since scheduling itself consumes a certain amount of time, the start time of the time window should not be the current time, but the end time of the last schedule. * [fix](test) fix ccr test cases (apache#28664) * [fix](regression) fix test_set_replica_status due to force_olap_table_replication_num=3 (apache#28573) * [fix](regression) restore reserve num replicas (apache#28541) * [fix](regression) fix test_alter_colocate_table due to force_olap_table_replication_num=3 (apache#28575) * [improvement](transaction) reduce publish txn log (apache#28277) * [fix](stream_load)fix bug for stream (apache#27752) 1. forbid thed stream_load without content-length or chunked Transfer Encoding 2. forbid thed stream_load both with content-length and chunked Transfer Encoding Co-authored-by: xingying01 <xingying01@corp.netease.com> * [Revert](partial update) Revert "Fix missing rowsets during doing alignment when flushing memtable due to compaction (apache#28062)" (apache#28674) This reverts commit 485d7db. * [feat](Nereids) support outer join and aggregate bitmap rewrite by mv (apache#28596) - Support left outer join rewrite by materialized view - Support bitmap_union roll up to imp count(distinct) - Support partition materialized view rewrite * [refactor](runtimefilter) do not use QueryContext in runtime filter (apache#28559) * [fix](planner)should bind expr using no slot to correct tuple (apache#28656) * [fix](planner) ctas update datev1 to datev2 should use equals (apache#28641) * [fix](Nereids) should only do bind relation in view analyzer (apache#28637) * [code](pipelineX) refine some pipelineX code (apache#28570) * [fix](partial update) report error directly if missing rowsets during doing alignment when flushing memtable due to compaction (apache#28677) * [opt](inverted index) Add null document interface to optimize empty string indexing (apache#28661) * [feature](invert index) match_regexp feature added (apache#28257) * [Feature](datatype) update be ut codes and fix bugs for IPv4/v6 (apache#28670) * [test](Nereids): remove shape plan project and distribute in eager test (apache#28701) * [fix](statistics)Fix drop stats fail silently bug. (apache#28635) Drop stats use IN predicate to filter the column stats to delete. The default length of IN predicate is 1024, drop table stats with more than 1024 columns may fail. This pr is to split the delete sql based on the IN predicate length. * [fix](mtmv) fix insert overwrite getExecTimeout error (apache#28700) should use InsertTimeout but QueryTimeout * [Enhancement](auto-partition) change the behaviour when insert overwrite an auto partition table apache#28683 If we specific target partition(s) when inserting overwrite an auto partition table, before: could create new partition now: behalf just like non-auto partition table * [FIX](regresstest) fix case with double value apache#28668 double value in case has 27 figures after the point, which will make not standable output * [test](Nereids): add test for scalar agg (apache#28712) * [fix](meta_scanner) fix meta_scanner process ColumnNullable (apache#28711) * [fix](nereids) Fix data wrong using mv rewrite and ignore case when getting mv related partition table (apache#28699) 1. Fix data wrong using mv rewrite 2. Ignore case when getting mv related partition table 3. Enable infer expression column name without alias when create mv * [test](regression-test) order by decs should only make effect on its nearest column apache#28728 * [refine](pipelineX)Make the 'set ready' logic of SenderQueue in pipelineX the same as that in the pipeline (apache#28488) * [fix](Nereids) delete partition failed (apache#28717) 1. parser's partitionSpec changed unexpectly by PR apache#26492 2. delete without using should support un-equals expression * [fix](tvf)Fixed the avro-scanner projection pushdown failing to query on multiple BEs (apache#28709) * [fix](block) fix nullptr in MutableBlock::allocated_bytes (apache#28738) * [fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE (apache#28723) when replay addTaskResult log,will create one ConnectContext,and set Env.getCurrentEnv,then store this ctx in ConnectContext.threadLocalInfo,threadLocalInfo is static,so this ctx can not be recycling，Env of replay thread also can not be recycling * [Fix](json reader) fix json reader crash due to `fmt::format_to` (apache#28737) ``` 4# __gnu_cxx::__verbose_terminate_handler() [clone .cold] at ../../../../libstdc++-v3/libsupc++/vterminate.cc:75 5# __cxxabiv1::__terminate(void (*)()) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48 6# 0x00005622F33D22B1 in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be 7# 0x00005622F33D2404 in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be 8# fmt::v7::detail::error_handler::on_error(char const*) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be 9# char const* fmt::v7::detail::parse_replacement_field<char, fmt::v7::detail::format_handler<fmt::v7::detail::buffer_appender<char>, char, fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<char>, char> >&>(char const*, char const*, fmt::v7::detail::format_handler<fmt::v7::detail::buffer_appender<char>, char, fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<char>, char> >&) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be 10# void fmt::v7::detail::vformat_to<char>(fmt::v7::detail::buffer<char>&, fmt::v7::basic_string_view<char>, fmt::v7::basic_format_args<fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<fmt::v7::type_identity<char>::type>, fmt::v7::type_identity<char>::type> >, fmt::v7::detail::locale_ref) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be 11# doris::vectorized::NewJsonReader::_append_error_msg(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:924 12# doris::vectorized::NewJsonReader::_set_column_value ``` * [bug](coredump) Fix coredump in aggregation node's destruction(apache#28684) fix coredump in aggregation node's destruction * [pipelineX](improvement) Support global runtime filter (apache#28692) * [FIX](explode)fix explode array decimal (apache#28744) * fix explode with array<decimal> has specific precision at old planner * [refactor](broadcastbuffer) using a queue to remove ref and unref codes (apache#28698) Co-authored-by: yiguolei <yiguolei@gmail.com>Add a new class broadcastbufferholderqueue to manage holders Using shared ptr to manage holders, not use ref and unref, it is too difficult to maintain. * [enhance](blacklist) seperate blacklist conf from heartbeat (apache#28638) There is a circuit breaker lasting for 2 minutes in grpc, then if a be is down and up again, send fragments to the be fails lasting for 2 minutes. * [fix](stmt):fix CreateTableStmt toSql placed comment in wrong place (apache#27504) Issue Number: close apache#27474 Co-authored-by: tongyang.han <tongyang.han@jiduauto.com> * [Improve](compile) add `__AVX2__` macro for JsonbParser (apache#28754) * [Improve](compile) add `__AVX2__` macro for JsonbParser * throw exception instead of CHECK * [Feature](materialized-view) support match logicalAggregate(logicalProject(logicalFilter(logicalOlapScan())) without agg (apache#28747) support match logicalAggregate(logicalProject(logicalFilter(logicalOlapScan())) without agg * [Fix] (schema change) forbid adding time type column (apache#28751) * [fix](group_commit) fix group commit cancel stuck (apache#28749) * [fix](scanner) fix concurrency bugs when scanner is stopped or finished (apache#28650) `ScannerContext` will schedule scanners even after stopped, and confused with `_is_finished` and `_should_stop`. Only Fix the concurrency bugs when scanner is stopped or finished reported in apache#28384 * [fix](regression) fix stream load properties case fail (apache#28680) * [fix](fe ut) fix PropertyConverterTest (apache#28722) * [improve](rpc) Log channel state before shutdown backend service client (apache#28667) * [fix](publish version) fix publish fail but return ok (apache#28425) * [test](Nereids): remove shape plan project and distribute in eager test (apache#28724) * [refactor](nereids) make NormalizeAggregate rule more clear and readable (apache#28607) * [FIX](type) fix matchExactType for complex type (apache#28233) fe matchExactType function should call type.matchTypes for its own logic, do not switch case to do special logic otherwise we may meet core in be like this. ``` F20231208 18:54:39.359673 680131 block.h:535] Check failed: _data_types[i]->is_nullable() target type: Struct(l_info:Nullable(Array(Nullable(String)))) src type: Struct(col:Nullable(Array(Nullable(UInt8)))) *** Check failure stack trace: *** @ 0x5584e952b926 google::LogMessage::SendToLog() @ 0x5584e9527ef0 google::LogMessage::Flush() @ 0x5584e952c169 google::LogMessageFatal::~LogMessageFatal() @ 0x5584cf17201e doris::vectorized::MutableBlock::merge_impl<>() @ 0x5584ceac4b1d doris::vectorized::MutableBlock::merge<>() @ 0x5584d4dd7de3 doris::vectorized::VUnionNode::get_next_const() @ 0x5584d4dd9a45 doris::vectorized::VUnionNode::get_next() @ 0x5584bce469bd std::__invoke_impl<>() @ 0x5584bce466d0 std::__invoke<>() @ 0x5584bce465c7 _ZNSt5_BindIFMN5doris8ExecNodeEFNS0_6StatusEPNS0_12RuntimeStateEPNS0_10vectorized5BlockEPbEPS1_St12_PlaceholderILi1EESC_ILi2EESC_ILi3EEEE6__callIS2_JOS4_OS7_OS8_EJLm0ELm1ELm2ELm3EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE @ 0x5584bce46358 std::_Bind<>::operator()<>() @ 0x5584bce46208 std::__invoke_impl<>() @ 0x5584bce46178 _ZSt10__invoke_rIN5doris6StatusERSt5_BindIFMNS0_8ExecNodeEFS1_PNS0_12RuntimeStateEPNS0_10vectorized5BlockEPbEPS3_St12_PlaceholderILi1EESD_ILi2EESD_ILi3EEEEJS5_S8_S9_EENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESL_E4typeEOSM_DpOSN_ @ 0x5584bce45c18 std::_Function_handler<>::_M_invoke() @ 0x5584bce6412f std::function<>::operator()() @ 0x5584bce56382 doris::ExecNode::get_next_after_projects() @ 0x5584bce26218 doris::PlanFragmentExecutor::get_vectorized_internal() @ 0x5584bce2431b doris::PlanFragmentExecutor::open_vectorized_internal() @ 0x5584bce22a96 doris::PlanFragmentExecutor::open() @ 0x5584bce27c9d doris::PlanFragmentExecutor::execute() @ 0x5584bcbdb3f8 doris::FragmentMgr::_exec_actual() @ 0x5584bcbf982f doris::FragmentMgr::exec_plan_fragment()::$_0::operator()() @ 0x5584bcbf9715 std::__invoke_impl<>() @ 0x5584bcbf96b5 _ZSt10__invoke_rIvRZN5doris11FragmentMgr18exec_plan_fragmentERKNS0_23TExecPlanFragmentParamsERKSt8functionIFvPNS0_12RuntimeStateEPNS0_6StatusEEEE3$_0JEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESH_E4typeEOSI_DpOSJ_ @ 0x5584bcbf942d std::_Function_handler<>::_M_invoke() @ 0x5584b9dfd883 std::function<>::operator()() @ 0x5584bd6e3929 doris::FunctionRunnable::run() @ 0x5584bd6cf8ce doris::ThreadPool::dispatch_thread() ``` * [test](Nereids): add assert debug log for TopnToMaxTest (apache#28755) * [improve](move-memtable) limit task num in load stream flush token (apache#28748) * [feature](Nereids) elimite inner join by foreign key (apache#28486) * [enhancement](backup/restore) support alter s3 repo info about ak/sk/token (apache#27027) in some cases: s3.session_token/AWS_TOKEN will be expired after hours, and may be upload snapshot job will failed if data is big large; as same reason, repo will be expired too when RepositoryMgr exec repo ping so it need support alter s3 repo properties about ak/sk/token and update upload snapshot job properties to continue backup. Signed-off-by: nextdreamblue <zxw520blue1@163.com> * [improve](transaction) extend abort transaction time (apache#28662) * [regression](p2) fix test cases result (apache#28768) regression-test/data/external_table_p2/hive/test_hive_hudi.out regression-test/data/external_table_p2/hive/test_hive_to_array.out regression-test/suites/external_table_p2/tvf/test_local_tvf_compression.groovy regression-test/suites/external_table_p2/tvf/test_path_partition_keys.groovy regression-test/data/external_table_p2/hive/test_hive_text_complex_type.out * (fix)[meta][export] fix replay export NPE issue (apache#28752) The ConnectionContext does not exist in replay thread * [Bug fix][metrics] correct fe collector type for jvm_gc (apache#28784) Co-authored-by: 胥剑旭 <xujianxu@xujianxudeMacBook-Pro.local> * [Chore](decimal) set check_overflow_for_decimal to true when alter table(apache#28777) set check_overflow_for_decimal to true when alter table * [pipelineX](fix) Fix TPCH Q2 (apache#28783) * (topN)runtime_predicate is only triggered when the column name is obtained (apache#28419) Issue Number: close apache#27485 * [fix](planner)fix bug of bound conjunct to wrong tuple (apache#28811) this fix bug introduced by apache#28656 * [bug](pipelineX) Fix pipelineX bug on multiple BE (apache#28792) * [opt](inverted index) ignore_above only affects untokenized strings (apache#28819) * [improve](load) reduce lock scope in MemTableWriter active consumption (apache#28790) * [opt](Nereids) improve Shape check cases (apache#28124) * tmplate update * log tpcds stats when check shape * [fix](group commit)Fix the issue of duplicate addition of wal path when encouter exception (apache#28691) * [fix](compaction) compaction should catch exception when vertical block reader read next block (apache#28625) * [fix](Nereids) join order is not right after sql parsing (apache#28721) for sql ``` t1, t2 join t3 ``` we should generate plan like: ``` t1 join (t2 join t3) ``` but we generate: ``` (t1 join t2) join t3 ``` to follow legancy planner. * [fix](metrics) fix bvar memtable_input_block_allocated_size (apache#28725) * [enhancement](audit-log) add status code and error msg to audit log for proxy stmt (apache#27607) * [improvement](external catalog)Change log level to debug for getRowCountFromExternalSource. (apache#28801) * [fix](partial update) only report error when in strict mode partial update when finding missing rowsets during flushing memtable (apache#28764) related pr: apache#28062, apache#28674, apache#28677 fix apache#28677 * [fix](function) Fix from_second functions overflow and wrong result (apache#28685) * [Enhancement](load) Limit the number of incorrect data drops and add documents (apache#27727) In the load process, if there are problems with the original data, we will store the error data in an error_log file on the disk for subsequent debugging. However, if there are many error data, it will occupy a lot of disk space. Now we want to limit the number of error data that is saved to the disk. Be familiar with the usage of doris' import function and internal implementation process Add a new be configuration item load_error_log_limit_bytes = default value 200MB Use the newly added threshold to limit the amount of data that RuntimeState::append_error_msg_to_file writes to disk Write regression cases for testing and verification Co-authored-by: xy720 <22125576+xy720@users.noreply.github.com> * [test](partial update) add complex type regression cases for partial update (apache#28758) NOTE: There's some issue for MAP type to work with row store, so in this PR we don't have cases for MAP type Will add the support for MAP type in future. * [fix](test)fix test_create_table test case for nereids (apache#28693) * [improve](config) set mutable and masterOnly in FE config stream_load_default_memtable_on_sink_node (apache#28835) * [fix](group-commit) check if wal need recovery is abnormal (apache#28769) * [Fix](Variant) fix variant predicate rewrite OrToIn with wrong plan (apache#28695) using the name without paths info will lead to wrong In plan, e.g. ``` where cast(v:a as text) = 'hello' or cast(v:b as text) = 'world' ``` will be rewrite to: ``` where cast(v as text) in ('hello', 'world') `` This is wrong, because they are different slots * [refactor](buffer) remove download buffer since it is no longer useful (apache#28832) remove download buffer since it is no longer useful * [Feature](Variant) Implement variant new sub column access method (apache#28484) * [Feature](Variant) Implement variant new sub column access method The query SELECT v["a"]["b"] from simple_var WHERE cast(v["a"]["b"] as int) = 1 encompasses three primary testing scenarios: ``` 1. A basic test involving the variant data type. 2. A scenario dealing with GitHub event data in the context of a variant. 3. A case related to the TPC-H benchmark using a variant. ``` * [fix](memory) Add thread asynchronous purge jemalloc dirty pages (apache#28655) jemallctl purge all arena dirty pages may take several seconds, which will block memory GC and cause OOM. So purge asynchronously in a thread. * [fix](regression) fix regression error of test_compress_type (apache#28826) * [improvement](executor) Add tvf and regression test for Workload Scheduler (apache#28733) 1 Add select workload schedule policy tvf 2 Add reg test * [chore](error msg) print type info when colocate with ddl failed due to type mismatch (apache#28773) * [opt](query cancel) optimization for query cancel apache#28778 * [Fix](multi-catalog) skip hms events if hms table is not supported. (apache#28644) Co-authored-by: wangxiangyu <wangxiangyu@360shuke.com> * [Enhancement](job) No need to query some backends which are not alive. (apache#28608) No need to execute some jobs if backend is not alive * [fix](paimon)fix type convert for paimon (apache#28774) fix type convert for paimon * [fix](stream load)add test case and doc for arrow type of stream load (apache#28098) add test case and doc for arrow type of stream load * [feature](mtmv)mtmv partition refresh case (apache#28787) * (enhance)(InternalQuery) Support to collect profile for intenal query (apache#28762) * [optimize](count) optimize pk exact query without reading data (apache#28494) * [opt](sessionVar)show changed sessoin var first apache#28840 “show variables” command list changed vars before not changed vars, * check stats and log memo for ds46 (apache#28396) * [bug](sharedscan) Fix shared scan bug (apache#28841) * [fix](hash join) fix stack overflow caused by evaluate case expr on huge build block (apache#28851) * [refactor](executor)remove scan group apache#28847 * [pipelineX](refactor) rename functions (apache#28846) * [Feature](inverted index) add lowercase option for inverted index analyzer (apache#28704) * [feature](Nereids) Pull up join from union all (apache#28682) * [fix](ci) tpch pipeline should not re-load data (apache#28874) * [fix](ci) tpch pipeline should not re-load data * 2 --------- Co-authored-by: stephen <hello-stephen@qq.com> * [exec](compress) use FragmentTransmissionCompressionCodec control the exchange compress behavior (apache#28818) * [improve](move-memtable) tweak load stream flush token num and max tasks (apache#28884) * [improve](load) remove extra layer of heavy work pool in tablet_writer_add_block (apache#28550) * [improve](load) limit delta writer flush task parallelism (apache#28883) * [fix](multi-catalog)filter impala generated path (apache#28786) file impala generated dir _imapala_insert_staging * [enhancement](udf) add prepare function for java-udf (apache#28750) * [refactor](test)Refactor workload group/schedule policy test apache#28888 [refactor](test)Refactor workload group/schedule policy test apache#28888 * [improve](move-memtable) avoid using heavy work pool during append data (apache#28745) * [enhancement](broker-load) fix-move-memtable-session-var-for-s3 (apache#28894) * [exec](load) change default parallel num from 1 to 8 in no pipeline exec engine (apache#28864) * [fix](segcompaction) disable segcompaction by default (apache#28906) Signed-off-by: freemandealer <freeman.zhang1992@gmail.com> * [fix](pipelineX) fix cannot runtime obtain profile on pipelineX apache#28795 * [fix](mtmv) fix failed to specify the number of buckets when bucket auto (apache#28854) Issue Number: close #xxx - fix failed to specify the number of buckets when bucket auto - delete unused SessionVariable - if mtmv used external table ,check `isMaterializedViewRewriteEnableContainForeignTable` * [bugfix](scannercore) scanner will core in deconstructor during collect profile (apache#28727) * [FIX](map)fix map with rowstore table (apache#28877) * [feature](mtmv)MTMV pause and resume (apache#28887) - PAUSE MATERIALIZED VIEW JOB ON mv1 - RESUME MATERIALIZED VIEW JOB ON mv1 - fix when drop db,not drop job - add lock for one materialized view can only run one task at a time * [feature](mtmv)add more test case1 (apache#28910) * [fix](block) add block columns size dcheck (apache#28539) * [fix](chore) update dcheck to avoid core during stress test (apache#28895) * [fix](doc) Add the usage example of bos to the documentation of s3 tvf (apache#28899) * [fix](mtmv)fix can not create mtmv all use default value (apache#28922) * [fix](parquet) the end offset of column chunk may be wrong in parquet metadata (apache#28891) * [fix](paimon)fix `like` predicate (apache#28803) fix like predict * [fix](merge-on-write) migration may cause duplicate keys for mow table (apache#28923) * [fix](mtmv) Related partition exclude null generate column when increment build materialized view (apache#28855) Infer partition column by materialized view partition column, exclude null generate column in join when increment build materialized view * [nereids] fix join fd computing bug (apache#28849) * [Fix](statistics) Fix partition name NPE and sample for all table during auto analyze (apache#28916) Fix partition name NPE and sample for all table during auto analyze. Sample for all tables because getData may have latency, which may cause full analyze a huge table and use too much resource. Sample for all tables to avoid this. Will improve the strategy later. * [fix](nereids) Fix query mv rewrite fail when mv cache build quickly (apache#28876) * [optimize](zonemap) skip zonemap if predicate does not support_zonemap (apache#28595) * [optimize](zonemap) skip zonemap if predicate does not support_zonemap apache#27608 (apache#28506) * Revert "[bugfix](scannercore) scanner will core in deconstructor during collect profile (apache#28727)" (apache#28931) This reverts commit 4066de3. * [fix] (nereids) Catch exception when mv fail and fix the npe (apache#28932) * [fix](regression-test) test_partial_update_native_insert_stmt_complex is flaky (apache#28927) * [improvement](group commit) make get column function more reliable when replaying wal (apache#28900) * [opt](nereids) convert or to inpredicate and optimize inpredicate partition prune (apache#28316) * [enhancement](bulk-load) cancel loading tasks directly without retrying when timeout exceeded (apache#28666) * [chore](test) Add testing util sync point (apache#28924) * [chore](prompt) Optimize tablet and replica prompt by pointing out what the numbers mean (apache#28925) * [fix](join) incorrect result of left semi/anti join with empty build side (apache#28898) * [docs] (DebugPoints) Update docs about Debug Points (apache#28347) --------- Co-authored-by: qinhao <qinhao@newland.com.cn> * [feature](api) add profile text api (apache#28697) * [opt](invert index) Empty strings are not written to the index in the case of TOKENIZED (apache#28822) * [feature](pipelineX) control exchange sink by memory usage (apache#28814) * [fix](pipelineX) fix use global rf when there no shared_scans (apache#28869) * [refactor](pipelineX)do not override dependency() function in pipelineX (apache#28848) * [fix](log) regularise some BE error type and fix a load task check apache#28729 * [chore](test) correct create table statement (apache#28863) * [fix](nereids)subquery unnest need handle subquery in Not expr correnctly (apache#28713) * [feature](nereids)support decimalv2 (apache#28726) * [fix](nereids)group by expr may be bound twice in bind agg slot (apache#28771) * [doc](insert) Add group commit docs (apache#25949) * [performance](variant) support topn 2phase read for variant column (apache#28318) [performance](variant) support topn 2phase read for variant column * return residual expr of join (apache#28760) * [fix](load) fix nullptr when getting memtable flush running count (apache#28942) * [fix](load) fix nullptr when getting memtable flush running count * style * (enhance)(regression) Support `force_olap_table_replication_num=3` run test_insert_random_distribution_table case (apache#28903) * [Test](Job)Add test case (apache#28481) * [fix](doc) typo fix in auto-partition page (apache#28512) * [feature](load) enable memtable on sink node by default (apache#28963) * [Chore](Job)print log before task execute (apache#28962) * [fix](hdfs) Fix HdfsFileSystem::exists_impl crash (apache#28952) Calling hdfsGetLastExceptionRootCause without initializing ThreadLocalState will crash. This PR modifies the condition for determining the existence of a hdfs file, because hdfsExists will set errno to ENOENT when the file does not exist, we can use this condition to check whether a file existence rather than check the existence of the root cause. * [fix](block) fix be core while mutable block merge may cause different row size between columns in origin block (apache#27943) * [fix](doc) typo fix in dynamic-partition page (apache#28511) * [fix](hash join) fix column ref DCHECK failure of hash join node block mem reuse (apache#28991) Introduced by apache#28851, after evaluating build side expr, some columns in resulting block may be referenced more than once in the same block. e.g. coalesce(col_a, 'string') if col_a is nullable but actually contains no null values, in this case funcition coalesce will insert a new nullable column which references the original col_a. * [opt](assert_num_rows) support filter in AssertNumRows operator and fix some explain (apache#28935) * NEED * Update pipeline x * fix pipelinex compile * [fix](parquet_reader) misused bool pointer (apache#28986) Signed-off-by: pengyu <pengyu@selectdb.com> * [doc](multi-catalog)add krb and some ertificates FAQs (apache#28858) add some security docs * [fix](planner)should save original select list item before analyze (apache#28187) * [fix](planner)should save original select list item before analyze * fix test case * fix failed case * [bug](storage) Fix gc rowset bug (apache#28979) * [fix](mtmv)add log for resolve pending task (apache#28999) * add lock for resolve pending task * add lock for resolve pending task * [improvement](nereids) Get partition related table disable nullable field and complete agg matched pattern mv rules. (apache#28973) * [improvement] (nereids) Get partition related table disable nullable field and modify regression test, complete agg mv rules. * make filed not null to create partition mv * [chore](config) modify `tablet_schema_cache_recycle_interval` from 24h to 1h (apache#28980) To prevent from too many tablet schema cache in memory and lead to performance issue when hold lock to erase item * [opt](scanner) optimize the number of threads of scanners (apache#28640) 1. Remove `doris_max_remote_scanner_thread_pool_thread_num`, use `doris_scanner_thread_pool_thread_num` only. 2. Set the default value `doris_scanner_thread_pool_thread_num` as `std::max(48, CpuInfo::num_cores() * 4)` * [chore](config) modify `variant_ratio_of_defaults_as_sparse_column` from 0.95 to 1 (apache#28984) since sparse column is not stable at present * [feature](nereids)support partition property in nereids (apache#28982) * [feature](Nereids) support table valued function http_stream (apache#29004) * [feature](Load)(step2)support nereids load job schedule (apache#26356) We will Integrate new load job manager into new job scheduling framework so that the insert into task can be scheduled after the broker load sql is converted to insert into TVF(table value function) sql. issue: apache#24221 Now support: 1. load data by tvf insert into sql, but just for simple load（columns need to be defined in the table） 2. show load stmt - job id, label name, job state, time info - simple progress 3. cancel load from db 4. support that enable new load through Config.enable_nereids_load 5. can replay job after restarting doris TODO: - support partition insert job - support show statistics from BE - support multiple task and collect task statistic - support transactional task - need add ut case * [enhance](tablet) Reduce log in tablet meta (apache#28719) * [fix](case) Add sync to test case (apache#29034) * [opt](compound) Optimize by deleting the compound expr after obtaining the final result (apache#28934) * [hotfix](jdbc catalog) fix load table and column names npe (apache#28865) This fix is for the npe situation that occurs when FE of non-Master nodes initializes Jdbc Catalog metadata. * [opt](Nereids) support cast bewteen numeric and boolean in FE (apache#29006) * [improvement](jdbc catalog) Optimize connection pool caching logic (apache#28859) In the old caching logic, we only used jdbcurl, user, and password as cache keys. This may cause the old link to be still used when replacing the jar package, so we should concatenate all the parameters required for the connection pool as the key. * [fix](Nereids) runtime filter push down failed (apache#28997) project child not always NamedExpression failed msg ``` org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = class org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral cannot be cast to class org.apache.doris.nereids.trees.expressions.NamedExpression (org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral and org.apache.doris.nereids.trees.expressions.NamedExpression are in unnamed module of loader 'app') at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:623) ~[classes/:?] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:478) ~[classes/:?] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:457) ~[classes/:?] at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:245) ~[classes/:?] at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:166) ~[classes/:?] at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:193) ~[classes/:?] at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:246) ~[classes/:?] at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[classes/:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:829) ~[?:?] Caused by: java.lang.ClassCastException: class org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral cannot be cast to class org.apache.doris.nereids.trees.expressions.NamedExpression (org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral and org.apache.doris.nereids.trees.expressions.NamedExpression are in unnamed module of loader 'app') at org.apache.doris.nereids.trees.plans.physical.PhysicalSetOperation.pushDownRuntimeFilter(PhysicalSetOperation.java:178) ~[classes/:?] at org.apache.doris.nereids.trees.plans.physical.PhysicalHashJoin.pushDownRuntimeFilter(PhysicalHashJoin.java:229) ~[classes/:?] at org.apache.doris.nereids.processor.post.RuntimeFilterGenerator.pushDownRuntimeFilterCommon(RuntimeFilterGenerator.java:386) ~[classes/:?] ``` * [fix](Nereids) generating function should not folding to NullLiteral (apache#29003) should not fold table generating function to null when do constant folding. we should remove Generate node and replaced it by project later. * [improve](load) add profile for WaitFlushLimitTime (apache#29013) * [opt](Nereids) let inverted index work with top opt (apache#29000) * [feature](Nereids) support values inline table in query (apache#28972) * [enhencement](config) change default memtable size & loadStreamPerNode & default load parallelism (apache#28977) We change memtable size from 200MB to 100MB to achieve smoother flush performance. We change loadStreamPerNode from 20 to 60 to avoid stream rpc to be the bottleneck when enable memtable_on_sink_node. We change default s3&broker load parallelsim to make the most of CPUs on moderm multi-core systems. Signed-off-by: freemandealer <freeman.zhang1992@gmail.com> * [improve](move-memtable) increase load_stream_flush_token_max_tasks (apache#29011) * [feature](scan) Implement parallel scanning by dividing the tablets based on the row range (apache#28967) * [feature](scan) parallel scann on dup/mow mode * fix bugs * [refactor](create tablet) default create tablet round robin (apache#28911) * [Bug] optimize the collection import Lists dependency packge apache#28523 (apache#28579) * [fix](regression) Fix flaky test test_partial_update_2pc_schema_change (apache#29037) * [Bug](security) BE download_files function exists log print sensitive msg apache#28592 (apache#28594) * Revert "[feature](load) enable memtable on sink node by default (apache#28963)" (apache#29090) This reverts commit 17917a0. * [improvement](statistics)Remove retry load when load stats cache fail (apache#28904) Remove retry load when load stats cache fail. This case usually happens when BE is down or BE OOM, retry doesn't work in these cases and may increase BE work load. * [fix](pipeline) sort_merge should throw exception in has_next_block if got failed status (apache#29076) Test in regression-test/suites/datatype_p0/decimalv3/test_decimalv3_overflow.groovy::249 sometimes failed when there are multiple BEs and FE process report status slowly for some reason. explain select k1, k2, k1 * k2 from test_decimal128_overflow2 order by 1,2,3 -------------- +----------------------------------------------------------------------------------------------------------------------------+ | Explain String(Nereids Planner) | +----------------------------------------------------------------------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | k1[#5] | | k2[#6] | | (k1 * k2)[#7] | | PARTITION: UNPARTITIONED | | | | HAS_COLO_PLAN_NODE: false | | | | VRESULT SINK | | MYSQL_PROTOCAL | | | | 111:VMERGING-EXCHANGE | | offset: 0 | | | | PLAN FRAGMENT 1 | | | | PARTITION: HASH_PARTITIONED: k1[#0], k2[#1] | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 111 | | UNPARTITIONED | | | | 108:VSORT | | | order by: k1[#5] ASC, k2[#6] ASC, (k1 * k2)[#7] ASC | | | offset: 0 | | | | | 102:VOlapScanNode | | TABLE: regression_test_datatype_p0_decimalv3.test_decimal128_overflow2(test_decimal128_overflow2), PREAGGREGATION: ON | | partitions=1/1 (test_decimal128_overflow2), tablets=8/8, tabletList=22841,22843,22845 ... | | cardinality=6, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | projections: k1[#0], k2[#1], (k1[#0] * k2[#1]) | | project output tuple id: 1 | +----------------------------------------------------------------------------------------------------------------------------+ 36 rows in set (0.03 sec) Why failed: Multiple BEs Fragments 0 and 1 are MUST on different BEs Pipeline task of VOlapScanNode which executes k1*k2 failed sets query status to cancelled Pipeline task of VSort call try close, send Cancelled status to VMergeExchange sort_curso did not throw exception when it meets error * [feature](Nereids): support infer join when comapring mv (apache#28988) * [fix](nereids) "not is null" stats estimation fix (apache#28860) * fix not is null stats * [opt](nereids)expr normalize after filter pushdown (apache#28743) normalize expression after filter push down * [Enhancement](Wal)Support dynamic wal space limit (apache#27726) * [fix](doc) spell errors fixes query profile docs (apache#28437) * [opt](Nerieds) add infer props to expression (apache#28953) * [fix](read) remove logic of estimating count of rows to read in segment iterator to avoid wrong result of unique key. (apache#29109) * [fix](paimon) read batch by doris' batch size (apache#29039) * [fix](mtmv)add log for resolve pending task (apache#29078) * [Refact](inverted index) refactor inverted index writer init (apache#29072) * [fix](paimon)Remove the static attribute of the source for paimon (apache#29032) * [fix](planner) Fix delete statement on MOR unique table (apache#28968) add conditions for mor table removed in apache#26776 * [fix][compile]unused variable (apache#28992) * [Fix](WorkFlow)Auto replay not work * [Fix](WorkFlow)Auto replay not work * [Fix](WorkFlow)Auto replay not work --------- Signed-off-by: nextdreamblue <zxw520blue1@163.com> Signed-off-by: freemandealer <freeman.zhang1992@gmail.com> Signed-off-by: pengyu <pengyu@selectdb.com> Co-authored-by: lihangyu <15605149486@163.com> Co-authored-by: Kang <kxiao.tiger@gmail.com> Co-authored-by: jakevin <jakevingoo@gmail.com> Co-authored-by: morrySnow <101034200+morrySnow@users.noreply.github.com> Co-authored-by: xueweizhang <zxw520blue1@163.com> Co-authored-by: zhangguoqiang <18372634969@163.com> Co-authored-by: zhangdong <493738387@qq.com> Co-authored-by: airborne12 <airborne08@gmail.com> Co-authored-by: Ma1oneZhang <73012001+Ma1oneZhang@users.noreply.github.com> Co-authored-by: ziyang zhang <zhangziyang@stu.cdut.edu.cn> Co-authored-by: Siyang Tang <82279870+TangSiyang2001@users.noreply.github.com> Co-authored-by: Kaijie Chen <ckj@apache.org> Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com> Co-authored-by: yiguolei <676222867@qq.com> Co-authored-by: minghong <englefly@gmail.com> Co-authored-by: Jibing-Li <64681310+Jibing-Li@users.noreply.github.com> Co-authored-by: Gavin Chou <gavineaglechou@gmail.com> Co-authored-by: plat1ko <platonekosama@gmail.com> Co-authored-by: wudongliang <46414265+DongLiang-0@users.noreply.github.com> Co-authored-by: Nitin-Kashyap <66766227+Nitin-Kashyap@users.noreply.github.com> Co-authored-by: yiguolei <yiguolei@gmail.com> Co-authored-by: wangbo <wangbo@apache.org> Co-authored-by: nanfeng <42513321+nanfeng1999@users.noreply.github.com> Co-authored-by: Xinyi Zou <zouxinyi02@gmail.com> Co-authored-by: qiye <jianliang5669@gmail.com> Co-authored-by: DuRipeng <453243496@qq.com> Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com> Co-authored-by: Qi Chen <kaka11.chen@gmail.com> Co-authored-by: Mingyu Chen <morningman@163.com> Co-authored-by: Xin Liao <liaoxinbit@126.com> Co-authored-by: yujun <yu.jun.reach@gmail.com> Co-authored-by: walter <w41ter.l@gmail.com> Co-authored-by: xy <whuxingying@163.com> Co-authored-by: xingying01 <xingying01@corp.netease.com> Co-authored-by: seawinde <149132972+seawinde@users.noreply.github.com> Co-authored-by: Mryange <59914473+Mryange@users.noreply.github.com> Co-authored-by: starocean999 <40539150+starocean999@users.noreply.github.com> Co-authored-by: bobhan1 <bh2444151092@outlook.com> Co-authored-by: zzzxl <33418555+zzzxl1993@users.noreply.github.com> Co-authored-by: yangshijie <sjyang2022@zju.edu.cn> Co-authored-by: zclllyybb <zhaochangle@selectdb.com> Co-authored-by: amory <wangqiannan@selectdb.com> Co-authored-by: zhiqiang <seuhezhiqiang@163.com> Co-authored-by: xy720 <22125576+xy720@users.noreply.github.com> Co-authored-by: Gabriel <gabrielleebuaa@gmail.com> Co-authored-by: htyoung <hty551020@hotmail.com> Co-authored-by: Pxl <pxl290@qq.com> Co-authored-by: Luwei <814383175@qq.com> Co-authored-by: meiyi <myimeiyi@gmail.com> Co-authored-by: Ashin Gau <AshinGau@users.noreply.github.com> Co-authored-by: HHoflittlefish777 <77738092+HHoflittlefish777@users.noreply.github.com> Co-authored-by: 谢健 <jianxie0@gmail.com> Co-authored-by: XuJianxu <xjxlzlpzl@163.com> Co-authored-by: 胥剑旭 <xujianxu@xujianxudeMacBook-Pro.local> Co-authored-by: Yoko <53637394+TomaYoko@users.noreply.github.com> Co-authored-by: huanghaibin <284824253@qq.com> Co-authored-by: lw112 <131352377+felixwluo@users.noreply.github.com> Co-authored-by: zhannngchen <48427519+zhannngchen@users.noreply.github.com> Co-authored-by: Xiangyu Wang <dut.xiangyu@gmail.com> Co-authored-by: wangxiangyu <wangxiangyu@360shuke.com> Co-authored-by: AlexYue <yj976240184@gmail.com> Co-authored-by: TengJianPing <18241664+jacktengg@users.noreply.github.com> Co-authored-by: xzj7019 <131111794+xzj7019@users.noreply.github.com> Co-authored-by: Dongyang Li <hello_stephen@qq.com> Co-authored-by: stephen <hello-stephen@qq.com> Co-authored-by: HappenLee <happenlee@hotmail.com> Co-authored-by: slothever <18522955+wsjz@users.noreply.github.com> Co-authored-by: zhangstar333 <87313068+zhangstar333@users.noreply.github.com> Co-authored-by: zhengyu <freeman.zhang1992@gmail.com> Co-authored-by: Jerry Hu <mrhhsg@gmail.com> Co-authored-by: HowardQin <hao.qin@esgyn.cn> Co-authored-by: qinhao <qinhao@newland.com.cn> Co-authored-by: deardeng <565620795@qq.com> Co-authored-by: caiconghui <55968745+caiconghui@users.noreply.github.com> Co-authored-by: py023 <137761033+py023@users.noreply.github.com> Co-authored-by: zy-kkk <zhongyk10@gmail.com> Co-authored-by: Guangming Lu <71873108+LuGuangming@users.noreply.github.com> Co-authored-by: abmdocrt <Yukang.Lian2022@gmail.com>
hello-stephen
pushed a commit
to hello-stephen/doris
that referenced
this pull request
Dec 28, 2023
add conditions for mor table removed in apache#26776
HappenLee
pushed a commit
to HappenLee/incubator-doris
that referenced
this pull request
Jan 12, 2024
add conditions for mor table removed in apache#26776
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...