v1.2.648-nightly

What's Changed

Thoughtful Bug Fix 🔧

• fix(query): check_timestamp/date -> clamp_timestamp/date by @TCeason in #16634
• fix: deadlock in table lock by @zhyass in #16632
• fix(query): sort spilling use arrow file format painc by @forsaken628 in #16658

Code Refactor 🎉

• refactor: delete user_procedure and refactor procedure interpreter by @TCeason in #16532

Others 📒

• chore(query): add warn log if vacuum temporary file is error by @zhang2014 in #16646
• chore: resolve todo in dynamic sample by @xudong963 in #16629
• chore(query): enable loser tree merge sort default by @forsaken628 in #16649
• chore(query): replace arrow2 with arrow-rs in serialize_column and deserialize_column by @forsaken628 in #16655

Full Changelog: v1.2.647-nightly...v1.2.648-nightly

v1.2.647-nightly

What's Changed

Exciting New Features ✨

• feat: support sampling table with block and row level simultaneously by @xudong963 in #16613
• feat: add unload options overwrite, include_query_id and use_raw_path. by @youngsofun in #16614
• feat(function): Add mode Aggregate function. by @Freejww in #16627
• feat(query): Support use parquet format when spilling by @forsaken628 in #16612
• feat(query): support json_object_insert function by @b41sh in #16636
• feat: http handler support request forwarding. by @youngsofun in #16637
• feat: Implemented a new sql explain analyze graphical by @Maricaya in #16543

Thoughtful Bug Fix 🔧

• fix: lock mgr may failed to release lock in time by @dantengsky in #16611
• fix(query): support subquery in pivot by @Dragonliu2018 in #16631

Code Refactor 🎉

• refactor: Unify the set and unset structures by @TCeason in #16622
• refactor(query): make delta table support partition prunning by @sundy-li in #16621
• refactor(storage): inverted index support calculate the score by @b41sh in #16609
• refactor: some http APIs do not need sticky. by @youngsofun in #16644

Others 📒

• chore: reset recluster block size by @zhyass in #16623
• chore(query): add http address info to cluster node info by @zhang2014 in #16620
• chore: drop guard when acquire lock failed by @zhyass in #16616
• chore: revert “lock mgr may failed to release lock in time (#16611)" by @dantengsky in #16630
• chore(query): reduce unsafe codes in kernels by @sundy-li in #16633

New Contributors

• @Dragonliu2018 made their first contribution in #16631
• @Maricaya made their first contribution in #16543

Full Changelog: v1.2.646-nightly...v1.2.647-nightly

v1.2.646-nightly

What's Changed

Exciting New Features ✨

• feat(query): Reduce serialisation memory usage when spilling to local disk by @forsaken628 in #16580
• feat(query): Support show dictionaries DQL. by @Winnie-Hong0927 in #16602

Thoughtful Bug Fix 🔧

• fix: subquery returns wrong results under cluster by @xudong963 in #16606
• fix(query): ntile frame should be unbounded by @TCeason in #16604

Code Refactor 🎉

• refactor: isolate session_id and session states of diff users. by @youngsofun in #16592

Others 📒

• chore: refine table sample tests by @xudong963 in #16600
• chore(query): add more logs on aggregation by @sundy-li in #16552

Full Changelog: v1.2.645-nightly...v1.2.646-nightly

v1.2.645-nightly

What's Changed

Exciting New Features ✨

• feat(query): add date_diff function by @chagelo in #16560

Thoughtful Bug Fix 🔧

• fix(query): fix left outer join wrong result by @Dousir9 in #16601

Build/Testing/CI Infra Changes 🔌

• ci: Add iceberg tpch sqllogictests by @Xuanwo in #16598

Others 📒

• chore(query): improve variable parser by @sundy-li in #16599

New Contributors

• @chagelo made their first contribution in #16560

Full Changelog: v1.2.644-nightly...v1.2.645-nightly

v1.2.644-nightly

What's Changed

Exciting New Features ✨

• feat(query): window partition by spill to disk by @forsaken628 in #16441
• feat(query): Simplified version Pattern S/L/D/G for number to_char by @forsaken628 in #16569
• feat(query): add json array lambda functions by @sundy-li in #16573
• feat(query): support describe user statement by @b41sh in #16579
• feat: copy option 'pattern' support variable. by @youngsofun in #16525
• feat(query): support json array functions by @b41sh in #16571

Thoughtful Bug Fix 🔧

• fix: direct io syscall not available on mac by @forsaken628 in #16574
• fix: window in scalar subquery returns wrong results by @xudong963 in #16567
• fix: avoid eliminating the left child of union in RuleEliminateUnion by @xudong963 in #16584
• fix(query): fix inverted index panic caused by null value by @b41sh in #16576
• fix(query): fix cannot obtain stack trace when segmentation fault by @zhang2014 in #16566
• fix: merge into panic by @zhyass in #16581
• fix(query): list_dictionaries should not ignore db_id by @TCeason in #16587
• fix(iceberg): Make sure iceberg table has been loaded by @Xuanwo in #16593
• fix: Disable prewhere for iceberg table for now by @Xuanwo in #16597

Code Refactor 🎉

• refactor: Automatically chunk large key sets in KVPbApi methods by @drmingdrmer in #16561
• refactor: simplify get_tables_history() by @drmingdrmer in #16572
• refactor(iceberg): Use iceberg file IO instead of our own operator by @Xuanwo in #16577
• refactor: add SchemaApi::get_table_in_db by @drmingdrmer in #16582
• refactor: SchemaApi::get_table_meta_history() by @drmingdrmer in #16586
• refactor: fix error message. by @youngsofun in #16578
• refactor(storage): inverted index use merge io to read data by @b41sh in #16589

Others 📒

• chore(query): SHOW/DESC is not DDL by @TCeason in #16583
• chore: check lock revision before watch delete by @zhyass in #16590
• chore(ci): flaky test by @zhyass in #16596

Full Changelog: v1.2.643-nightly...v1.2.644-nightly

v1.2.643-nightly

What's Changed

Exciting New Features ✨

• feat: unset table options by @dantengsky in #16544

Thoughtful Bug Fix 🔧

• fix: move TableInfo building out of SchemaApi by @drmingdrmer in #16548
• fix: identify failed vacuum db by id by @drmingdrmer in #16553
• fix(query): fix json object compare cause panic by @b41sh in #16555

Code Refactor 🎉

• refactor: get_history_tables_for_gc() should not return TableInfo, but just table name, id and values by @drmingdrmer in #16545
• refactor(storage): improve inverted index match phrase query by @b41sh in #16547

Others 📒

• chore(storage): refactor compact source by @zhyass in #16527

Full Changelog: v1.2.642-nightly...v1.2.643-nightly

v1.2.642-nightly

What's Changed

Exciting New Features ✨

• feat: short sql length setting by @arkzuse in #16502

Code Refactor 🎉

• refactor: SchemaAPI::do_get_table_history by @drmingdrmer in #16540
• refactor: DroppedId for listing db/tables for gc by @drmingdrmer in #16542

Others 📒

• chore: update repo for workflows by @everpcpc in #16546

New Contributors

• @arkzuse made their first contribution in #16502

Full Changelog: v1.2.641-nightly...v1.2.642-nightly

v1.2.641-nightly

What's Changed

Exciting New Features ✨

• feat: add fast path for cross join to optimize tpcds q23 (~50x) by @xudong963 in #16504
• feat(catalog): ignore error when listing databases by @BohuTANG in #16514
• feat: limit stage name to plain_identifier by @youngsofun in #16522
• feat(query): Initial implementation of postgres-compatible number to_char function by @forsaken628 in #16507
• feat(query): Procedure support drop if exists and create or replace by @TCeason in #16500
• feat(query): DATE_ADD Functions to support week as a unit by @TracyZYJ in #16530
• feat: http handler add X-DATABEND-VERSION in each response. by @youngsofun in #16518

Thoughtful Bug Fix 🔧

• chore: add extra bracket for and and or to make explain clear by @xudong963 in #16494
• fix(query): fix planner cache cause modify async function does not take effect by @b41sh in #16510
• fix: ignore_result doesn't have an effect on union by @xudong963 in #16515
• fix(query): fix copy into table with disable distributed copy into by @zhang2014 in #16529
• fix(query): fix column leaf_index by @Dousir9 in #16537
• fix: Only when all tables are included, the db can be dropped by @drmingdrmer in #16539

Code Refactor 🎉

• refactor(query): refactor license manager by @zhang2014 in #16492
• refactor: simplify DroppedId. Replace manual comparison with string comparison by @drmingdrmer in #16495
• refactor: update dependency datafusion-orc. by @youngsofun in #16491
• refactor: get_tableinfos_by_ids() by @drmingdrmer in #16503
• chore(query): refactor like function by @sundy-li in #16493
• chore: simplify SchemaApi::truncate_table() by @drmingdrmer in #16506
• refactor: remove get_pb_value() by @drmingdrmer in #16516
• refactor: Output whole error message with source while retry by @Xuanwo in #16519
• refactor: unify path of write segment by @SkyFan2002 in #16517
• refactor: Improve gc_dropped_table_by_id() method by @drmingdrmer in #16528
• refactor: dropped table listing and GC by @drmingdrmer in #16531
• refactor: gc_dropped_db_by_id() by @drmingdrmer in #16538

Others 📒

• chore: remove logs from block writer by @dantengsky in #16497
• chore: simplify get_gc_table_info() by @drmingdrmer in #16499
• chore(query): replace to datafuse-extras/dtparse by @TCeason in #16509
• chore(storage): hook recluster avoid scan all segments by @zhyass in #16498
• chore: update source repo for async-compression by @everpcpc in #16513
• chore(query): move recursion & backtrace crate to datafuse-extras by @zhang2014 in #16520
• chore(query): add more asserts by @sundy-li in #16536

Full Changelog: v1.2.640-nightly...v1.2.641-nightly

v1.2.640-nightly

What's Changed

Exciting New Features ✨

• feat(query): push rank limit into aggregate partial node by @sundy-li in #16466
• feat: add eliminate union optimizer rule by @xudong963 in #16478
• feat(query): parallel k way merge sort by @forsaken628 in #16340

Code Refactor 🎉

• refactor: add DatabaseMeta.gc_in_progress by @drmingdrmer in #16464
• refactor(query): use pipeline exchange to refactor window scatter by @zhang2014 in #16471
• refactor(query): simplify expression kernel by @sundy-li in #16482
• refactor: add KVPbCrudApi as abstraction layer for commonly used CRUD operations by @drmingdrmer in #16479
• refactor: add KVPbCrudApi::crud_try_insert() and crud_try_upsert() by @drmingdrmer in #16484
• refactor: SchemaApi::list_databases() by @drmingdrmer in #16485
• refactor: SchemaApi::get_database_history() by @drmingdrmer in #16487
• refactor: test SchemaApi::get_tenant_history_database() by @drmingdrmer in #16488
• refactor(query): refactor license manager by @zhang2014 in #16489

Others 📒

• chore(ci): remove hdfs release by @everpcpc in #16477
• chore(query): null first behavior by @sundy-li in #16475
• chore: update nom-rule by @andylokandy in #16483
• chore: bump proj4rs 0.1.4 by @b41sh in #16474

Full Changelog: v1.2.639-nightly...v1.2.640-nightly

v1.2.639-nightly

What's Changed

Thoughtful Bug Fix 🔧

• fix: typo for query & profile logging config by @everpcpc in #16476

Full Changelog: v1.2.638-nightly...v1.2.639-nightly