HLL support default value #1825

HangyuanLiu · 2019-09-18T10:47:50Z

When load date don't have HLL value , HLL value is a empty HyperLogLog

eg:
CREATE TABLE test_uv_9 (
pin_id bigint(20) NOT NULL COMMENT "",
id bigint(20) NULL COMMENT "",
uv1 hll HLL_UNION NOT NULL COMMENT "",
uv2 hll HLL_UNION NOT NULL COMMENT ""
) ENGINE=OLAP
AGGREGATE KEY(pin_id, id)
DISTRIBUTED BY HASH(pin_id) BUCKETS 16
PROPERTIES (
"storage_type" = "COLUMN"
)
1、curl --location-trusted -u root:123456 -H column_separator:, -H label:test_uv_14 -H "columns:pin_id,idx,u1,u2,id=12 ,uv1=hll_hash(u1)" -T uv_test http://11.40.166.162:8030/api/test/test_uv_9/_stream_load
Result ：hll_union_agg(uv2) should be 0，And hll_union_agg(uv1) has values > 0

2、curl --location-trusted -u root:123456 -H column_separator:, -H label:test_uv_15 -H "columns:pin_id,idx,u1,u2,id=12" -T uv_test http://11.40.166.162:8030/api/test/test_uv_10/_stream_load
Result ：hll_union_agg(uv1) and hll_union_agg(uv2) should be 0

imay · 2019-09-18T14:38:11Z

@HangyuanLiu
Can you add some document or usage example for this change?

kangkaisen · 2019-09-19T03:07:32Z

be/src/olap/aggregate_func.h

@@ -408,7 +408,11 @@ struct AggregateFuncTraits<OLAP_FIELD_AGGREGATION_HLL_UNION, OLAP_FIELD_TYPE_HLL
        dst_slice->size = sizeof(HyperLogLog);
        // use 'placement new' to allocate HyperLogLog on arena, so that we can control the memory usage.
        char* mem = arena->Allocate(dst_slice->size);
-        dst_slice->data = (char*) new (mem) HyperLogLog(src_slice->data);
+        if (src_slice->empty()) {
+            dst_slice->data = (char*) new (mem) HyperLogLog();


When the slice size is empty?

When HLL columns are NULL in the LOAD data

HangyuanLiu · 2019-09-19T04:02:55Z

Update example @imay

kangkaisen · 2019-09-19T06:16:24Z

curl --location-trusted -u root:123456 -H column_separator:, -H label:test_uv_15 -H "columns:pin_id,idx,u1,u2,id=12" -T uv_test http://11.40.166.162:8030/api/test/test_uv_10/_stream_load
Result ：hll_union_agg(uv1) and hll_union_agg(uv2) should be 0

I think we don't support this usage.

When load data with HLL column, we must use hll_hash function, hll_hash function will handle null value.

HangyuanLiu · 2019-09-19T06:29:47Z

This PR will support this capability.
There is a scenario where a Doris table may have multiple HLL columns that may come from different streams. The A stream may not contain the required HLL columns in the B stream and may not have NULL fields. The number of fields in this stream may be less than the number of fields in the Doris table
@kangkaisen

kangkaisen · 2019-09-19T07:02:10Z

@HangyuanLiu Hi, I See.
Could we add a empty_hll function support this capability? add then, we could still ensure the HLL input in aggregate_func.h will not null.

HangyuanLiu · 2019-09-19T07:20:44Z

@kangkaisen
If we add a empty_hll function , so we should write load command like .

LOAD LABEL test.uv
DATA INFILE ("hdfs://ns1017/streamA/*") INTO TABLE test_uv
(pin, id, u1)
set (
uv1=hll_hash(u1),
uv2= empty_hll(),
);

LOAD LABEL test.uv
DATA INFILE ("hdfs://ns1017/streamB/*") INTO TABLE test_uv
(pin, id, u2)
set (
uv1= empty_hll(),
uv2= hll_hash(u2),
);

uv2 may be another business stream
It is unreasonable to let the import data for A business care about the other B business HLL type.

morningman · 2019-09-21T04:35:25Z

@HangyuanLiu I think using empty_hll() is more reasonable. If business A does not care not column uv2, It should leave uv2 unchanged by explicitly assigning a function means unchanged, which is what empty_hll() will do.

If user assign nothing to HLL column, HLL column's content will be undefined. Because HLL is not like other types of column, which can be filled by default value or null.

empty_hll() to HLL is like 0 to SUM.

And also, should will add a function empty_bitmap() for BITMAP column? @kangkaisen

kangkaisen · 2019-09-21T04:43:12Z

And also, should will and a function empty_bitmap() for BITMAP column

@morningman I agree with you.

imay · 2019-09-21T05:44:37Z

And also, should will and a function empty_bitmap() for BITMAP column

@morningman I agree with you.

I agree too

@HangyuanLiu

Will you add a hll_empty() function? And it will be better if you can create an issue to record this improvement.

HangyuanLiu · 2019-09-21T11:53:36Z

OK,I can add a function

…to hll_default Conflicts: be/src/olap/aggregate_func.h

imay · 2019-09-23T10:57:08Z

be/src/exprs/hll_function.cpp

+    const int HLL_EMPTY_SIZE = 1;
+    std::string buf;
+    std::unique_ptr<HyperLogLog> hll;
+    hll.reset(new HyperLogLog());


A stack variable is enough

# Conflicts: # be/src/exprs/hll_function.cpp

imay · 2019-09-24T13:46:11Z

be/src/olap/hll.h

@@ -103,6 +103,14 @@ class HyperLogLog {

    int64_t estimate_cardinality();

+    std::string empty() {


why not make this function static?

imay · 2019-09-24T13:46:22Z

docs/documentation/cn/sql-reference/sql-statements/Data Definition/HLL.md

@@ -18,6 +18,9 @@

      HLL_HASH(column_name)
      生成HLL列类型，用于insert或导入的时候，导入的使用见相关说明
+
+      EMPTY_HLL()


english doc

HangyuanLiu · 2019-09-27T06:39:41Z

#1895

hll default value

3eb2d82

kangkaisen reviewed Sep 19, 2019

View reviewed changes

HangyuanLiu added 3 commits September 23, 2019 18:33

add empty_hll

9996ebe

Merge branch 'master' of https://github.com/apache/incubator-doris in…

839b31f

…to hll_default Conflicts: be/src/olap/aggregate_func.h

fix

6ed3dfb

imay reviewed Sep 23, 2019

View reviewed changes

HangyuanLiu force-pushed the hll_default branch from 6ed3dfb to c8de048 Compare September 24, 2019 13:35

fix

7105674

HangyuanLiu force-pushed the hll_default branch from c8de048 to 7105674 Compare September 24, 2019 13:36

HangyuanLiu added 2 commits September 24, 2019 21:37

Add empty_hll doc

9bf922f

Merge remote-tracking branch 'origin/hll_default' into hll_default

fe966b0

# Conflicts: # be/src/exprs/hll_function.cpp

imay reviewed Sep 24, 2019

View reviewed changes

fix

25f3a6a

HangyuanLiu force-pushed the hll_default branch from fe966b0 to 25f3a6a Compare September 24, 2019 14:17

HangyuanLiu added 3 commits September 24, 2019 22:27

Merge remote-tracking branch 'origin/hll_default' into hll_default

6e2978a

Add empty_hll doc

c403e5d

Add empty_hll doc

cbe3153

imay approved these changes Sep 25, 2019

View reviewed changes

imay merged commit 40b9c35 into apache:master Sep 25, 2019

imay mentioned this pull request Sep 26, 2019

Release Notes 0.11.0 #1891

Closed

SWJTU-ZhangLei pushed a commit to SWJTU-ZhangLei/incubator-doris that referenced this pull request Jul 25, 2023

(selectdb-cloud) Update recycler regression tests (apache#1825)

00b4f7a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HLL support default value #1825

HLL support default value #1825

HangyuanLiu commented Sep 18, 2019 •

edited

Loading

imay commented Sep 18, 2019

kangkaisen Sep 19, 2019

HangyuanLiu Sep 19, 2019

HangyuanLiu commented Sep 19, 2019

kangkaisen commented Sep 19, 2019

HangyuanLiu commented Sep 19, 2019

kangkaisen commented Sep 19, 2019

HangyuanLiu commented Sep 19, 2019 •

edited

Loading

morningman commented Sep 21, 2019 •

edited

Loading

kangkaisen commented Sep 21, 2019

imay commented Sep 21, 2019

HangyuanLiu commented Sep 21, 2019

imay Sep 23, 2019

imay Sep 24, 2019

imay Sep 24, 2019

HangyuanLiu commented Sep 27, 2019

		@@ -103,6 +103,14 @@ class HyperLogLog {

		int64_t estimate_cardinality();

		std::string empty() {

HLL support default value #1825

HLL support default value #1825

Conversation

HangyuanLiu commented Sep 18, 2019 • edited Loading

imay commented Sep 18, 2019

kangkaisen Sep 19, 2019

Choose a reason for hiding this comment

HangyuanLiu Sep 19, 2019

Choose a reason for hiding this comment

HangyuanLiu commented Sep 19, 2019

kangkaisen commented Sep 19, 2019

HangyuanLiu commented Sep 19, 2019

kangkaisen commented Sep 19, 2019

HangyuanLiu commented Sep 19, 2019 • edited Loading

morningman commented Sep 21, 2019 • edited Loading

kangkaisen commented Sep 21, 2019

imay commented Sep 21, 2019

HangyuanLiu commented Sep 21, 2019

imay Sep 23, 2019

Choose a reason for hiding this comment

imay Sep 24, 2019

Choose a reason for hiding this comment

imay Sep 24, 2019

Choose a reason for hiding this comment

HangyuanLiu commented Sep 27, 2019

HangyuanLiu commented Sep 18, 2019 •

edited

Loading

HangyuanLiu commented Sep 19, 2019 •

edited

Loading

morningman commented Sep 21, 2019 •

edited

Loading