[Feature] Z-Order Implement #7149

xinghuayu007 · 2021-11-18T13:57:05Z

Proposed changes

For issue: #6359

Background

Z-Order

Application Situation

Grammar

CREATE TABLE table2 (
siteid int(11) NULL DEFAULT "10" COMMENT "",
citycode int(11) NULL COMMENT "",
username varchar(32) NULL DEFAULT "" COMMENT "",
pv bigint(20) NULL DEFAULT "0" COMMENT ""
) ENGINE=OLAP
DUPLICATE KEY(siteid, citycode)
COMMENT "OLAP"
DISTRIBUTED BY HASH(siteid) BUCKETS 1
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"data_sort.sort_type" = "ZORDER",
"data_sort.col_num" = "2",
"in_memory" = "false",
"storage_format" = "V2"
);

data_sort.sort_type: support lexical/z-order sort type, default is lexical sort type
data_sort.col_num: take the pre-columns as sort key

Performance Test

Load Performance
Env: ssb scale 100, stream load

Query Performance
Env: TPCH scale 25
Table:

CREATE TABLE LINEITEM (
L_PARTKEY int(11) NOT NULL COMMENT "",
L_SUPPKEY int(11) NOT NULL COMMENT "",
L_ORDERKEY int(11) NOT NULL COMMENT "",
L_LINENUMBER int(11) NOT NULL COMMENT "",
L_QUANTITY decimal(15, 2) NOT NULL COMMENT "",
L_EXTENDEDPRICE decimal(15, 2) NOT NULL COMMENT "",
L_DISCOUNT decimal(15, 2) NOT NULL COMMENT "",
L_TAX decimal(15, 2) NOT NULL COMMENT "",
L_RETURNFLAG char(1) NOT NULL COMMENT "",
L_LINESTATUS char(1) NOT NULL COMMENT "",
L_SHIPDATE date NOT NULL COMMENT "",
L_COMMITDATE date NOT NULL COMMENT "",
L_RECEIPTDATE date NOT NULL COMMENT "",
L_SHIPINSTRUCT char(25) NOT NULL COMMENT "",
L_SHIPMODE char(10) NOT NULL COMMENT "",
L_COMMENT varchar(44) NOT NULL COMMENT "",
L_DEFAULT varchar(44) NULL COMMENT ""
) ENGINE=OLAP
DUPLICATE KEY(L_PARTKEY, L_SUPPKEY)
COMMENT "OLAP"
DISTRIBUTED BY HASH(L_PARTKEY) BUCKETS 1
PROPERTIES (
"replication_num" = "3",
"in_memory" = "false",
"storage_format" = "V2"
);

Q1: select count(1) from LINEITEM where L_PARTKEY=999852;
Q2: select count(1) from LINEITEM where L_SUPPKEY=125019;

Limitation

only duplicate model supports z-order
short key index is not available for z-order
only V2 storage format supports z-order

Future work

aggregate/unique model supports z-order
try to use hilbert-curve to make data more clustering

Types of changes

What types of changes does your code introduce to Doris?
Put an x in the boxes that apply

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation Update (if none of the other choices apply)
Code refactor (Modify the code structure, format the code, etc...)
Optimization. Including functional usability improvements and performance improvements.
Dependency. Such as changes related to third-party components.
Other.

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail
Compiling and unit tests pass locally with my changes
I have added tests that prove my fix is effective or that my feature works
If these changes need document changes, I have updated the document
Any dependent changes have been merged

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

morningman · 2021-11-18T13:58:55Z

Nice work!

be/src/olap/skiplist.h

be/src/util/tuple_row_zorder_compare.h

hf200012 · 2021-11-19T01:02:48Z

apache carbondata also has this min/max index，nice

morningman · 2021-11-26T04:08:18Z

be/test/olap/skiplist_test.cpp

@@ -268,7 +271,7 @@ class ConcurrentTest {
    ConcurrentTest()
            : _mem_tracker(new MemTracker(-1)),
              _mem_pool(new MemPool(_mem_tracker.get())),
-              _list(TestComparator(), _mem_pool.get(), false) {}
+              _list(new TestComparator(), _mem_pool.get(), false) {}


Add ~ConcurrentTest() to delete _list

_list is an object, is it needs to delete it?

Yes, you pass a raw pointer to the skiplist, and nowhere to delete this TestComparator

fe/fe-core/src/main/java/org/apache/doris/analysis/DataSortInfo.java

morningman · 2021-11-26T04:11:42Z

fe/fe-core/src/main/java/org/apache/doris/analysis/DataSortInfo.java

+        return true;
+    }
+
+    public String getProperties() {


toString() maybe better?

getProperties keeps the same style with before, like dynamic partition

Oh, So I think we should change the dynamic partition too...
It is more like toString() or toSql()

modify it to toSql()

fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java

morningman · 2021-11-26T04:25:40Z

fe/fe-core/src/main/java/org/apache/doris/catalog/TableProperty.java

@@ -137,6 +151,11 @@ public void modifyTableProperties(Map<String, String> modifyProperties) {
        properties.putAll(modifyProperties);
    }

+    public void modifyTableProperties(DataSortInfo dataSortInfo) {


Change the method name

modifyTableProperties keeps the same style with before

Not same, the origin 2 modifyTableProperties() method pass the key-value, which means it can modify any of table properties.
But here you just want to modify the data sort info.

modify it to modifyDataSortInfoProperties()

fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java

morningman · 2021-11-26T04:34:03Z

fe/fe-core/src/main/java/org/apache/doris/task/CreateReplicaTask.java

@@ -165,6 +202,10 @@ public TCreateTabletReq toThrift() {
        tSchema.setSchemaHash(schemaHash);
        tSchema.setKeysType(keysType.toThrift());
        tSchema.setStorageType(storageType);
+        if (dataSortInfo != null) {
+            tSchema.setSortType(dataSortInfo.getSortType());
+            tSchema.setSortColNum(dataSortInfo.getColNum());


Better use default value instead of optional setting?

dataSortInfo maybe null, will cause toThrift() failed

morningman · 2021-11-26T04:36:13Z

I tested my own and it works well.
3 more things:

Some litter problem which I have commented.
rebase to resolve the conflict.
Add document for this new feature.

morningman · 2021-11-29T08:34:49Z

fe/fe-core/src/main/java/org/apache/doris/analysis/DataSortInfo.java

+        return true;
+    }
+
+    public String getProperties() {


Oh, So I think we should change the dynamic partition too...
It is more like toString() or toSql()

morningman · 2021-11-29T08:36:00Z

fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java

+    public boolean isZOrderSort() {
+        return tableProperty != null
+                && tableProperty.getDataSortInfo() != null
+                && tableProperty.getDataSortInfo().getSortType() != TSortType.LEXICAL;


Suggested change

&& tableProperty.getDataSortInfo().getSortType() != TSortType.LEXICAL;

&& tableProperty.getDataSortInfo().getSortType() == TSortType.ZORDER;

So that we don't need to modify it if we add some other sort type later.

morningman · 2021-11-29T08:39:03Z

fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java

@@ -1714,6 +1729,13 @@ public TStorageFormat getStorageFormat() {
        return tableProperty.getStorageFormat();
    }

+    public DataSortInfo getDataSortInfo() {
+        if (tableProperty == null) {
+            return new DataSortInfo();


Better assign the default value to the fields of DataSortInfo, so that we can make this return more semantically accurate

morningman · 2021-11-29T08:40:34Z

fe/fe-core/src/main/java/org/apache/doris/catalog/TableProperty.java

@@ -137,6 +151,11 @@ public void modifyTableProperties(Map<String, String> modifyProperties) {
        properties.putAll(modifyProperties);
    }

+    public void modifyTableProperties(DataSortInfo dataSortInfo) {


Not same, the origin 2 modifyTableProperties() method pass the key-value, which means it can modify any of table properties.
But here you just want to modify the data sort info.

fe/fe-core/src/main/java/org/apache/doris/analysis/DataSortInfo.java

xinghuayu007 · 2021-11-29T09:17:30Z

I tested my own and it works well. 3 more things:

Some litter problem which I have commented.

rebase to resolve the conflict.

Add document for this new feature.

documents are added in 'create table' doc; Alter table is prohibited for z-order table

morningman

LGTM

github-actions · 2021-11-30T11:40:46Z

PR approved by at least one committer and no changes requested.

github-actions · 2021-11-30T11:40:48Z

PR approved by anyone and no changes requested.

morningman

LGTM

github-actions · 2021-12-01T04:50:42Z

PR approved by at least one committer and no changes requested.

morningman added kind/improvement kind/feature Categorizes issue or PR as related to a new feature. labels Nov 18, 2021

morningman reviewed Nov 18, 2021

View reviewed changes

be/src/olap/skiplist.h Outdated Show resolved Hide resolved

be/src/util/tuple_row_zorder_compare.h Outdated Show resolved Hide resolved

be/src/util/tuple_row_zorder_compare.h Outdated Show resolved Hide resolved

xinghuayu007 force-pushed the new_zorder_index branch from b588425 to fc81ccd Compare November 19, 2021 02:32

morningman reviewed Nov 26, 2021

View reviewed changes

xinghuayu007 force-pushed the new_zorder_index branch 3 times, most recently from 282981f to d677397 Compare November 29, 2021 04:06

morningman reviewed Nov 29, 2021

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/analysis/DataSortInfo.java Outdated Show resolved Hide resolved

xinghuayu007 added 2 commits November 30, 2021 14:07

z-order implement

9a278d9

refactor code

a7cecc3

xinghuayu007 force-pushed the new_zorder_index branch from bcb86e6 to a7cecc3 Compare November 30, 2021 06:08

remove unused method

0059ea3

morningman previously approved these changes Nov 30, 2021

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 30, 2021

github-actions bot added the reviewed label Nov 30, 2021

xinghuayu007 dismissed morningman’s stale review via 4e70bb2 December 1, 2021 04:44

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 1, 2021

delete comparator

3bc0a9e

xinghuayu007 force-pushed the new_zorder_index branch from 4e70bb2 to 3bc0a9e Compare December 1, 2021 04:47

morningman approved these changes Dec 1, 2021

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 1, 2021

morningman merged commit dd36ccc into apache:master Dec 2, 2021

morningman mentioned this pull request Jan 2, 2022

Doris Roadmap 2022 #7502

Closed

fengfuyuyang mentioned this pull request Jun 21, 2023

[Bug] Zorder排序表第一列作为过滤列，查询结果错误 #21051

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Z-Order Implement #7149

[Feature] Z-Order Implement #7149

xinghuayu007 commented Nov 18, 2021 •

edited

Loading

morningman commented Nov 18, 2021

hf200012 commented Nov 19, 2021

morningman Nov 26, 2021

xinghuayu007 Nov 29, 2021

morningman Nov 29, 2021

xinghuayu007 Nov 29, 2021

morningman Nov 26, 2021

xinghuayu007 Nov 29, 2021

morningman Nov 29, 2021

xinghuayu007 Nov 29, 2021

morningman Nov 26, 2021

xinghuayu007 Nov 29, 2021

morningman Nov 29, 2021

xinghuayu007 Nov 29, 2021

morningman Nov 26, 2021

xinghuayu007 Nov 29, 2021

morningman commented Nov 26, 2021

morningman Nov 29, 2021

morningman Nov 29, 2021

morningman Nov 29, 2021

morningman Nov 29, 2021

xinghuayu007 commented Nov 29, 2021

morningman left a comment

github-actions bot commented Nov 30, 2021

github-actions bot commented Nov 30, 2021

morningman left a comment

github-actions bot commented Dec 1, 2021

	&& tableProperty.getDataSortInfo().getSortType() != TSortType.LEXICAL;
	&& tableProperty.getDataSortInfo().getSortType() == TSortType.ZORDER;

[Feature] Z-Order Implement #7149

[Feature] Z-Order Implement #7149

Conversation

xinghuayu007 commented Nov 18, 2021 • edited Loading

Proposed changes

Background

Z-Order

Application Situation

Grammar

Performance Test

Limitation

Future work

Types of changes

Checklist

Further comments

morningman commented Nov 18, 2021

hf200012 commented Nov 19, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

morningman commented Nov 26, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xinghuayu007 commented Nov 29, 2021

morningman left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 30, 2021

github-actions bot commented Nov 30, 2021

morningman left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 1, 2021

xinghuayu007 commented Nov 18, 2021 •

edited

Loading