support for apache datasketches

[nagarajatantry]

Hello databend community,

1. Does databend provide support for Apache Datasketches, more specifically HLL Sketch (unique count) and Doublesketch (quantiles)?
2. If yes, can we store the data as sketch datatypes allowing us to slice/dice on various dimensions?

[sundy-li]

Hello, We don't support apache datasketches. I just notice apache datasketches on first time, I will take a look at this

[sundy-li]

Looks like it stored aggregating state as an additional datatype from link : https://datasketches.apache.org/docs/HLL/HllHiveUDFs.html

We can have this in the aggregating index, it's an easy task to try.

@ariesdevil can you enable approx_count_distinct aggregating function in the aggregating index ?

[ariesdevil]

Sure, let me do it and add some tests.

[nagarajatantry]

Thank you for looking into this.
@sundy-li Yes.. It allows us to store the sketch data. In the case of HLL, for our use case due to compliance issue, we cannot store the raw data. So we generate a sketch and do merge the sketches to compute unique count.

Similarly, for Double Sketch (quantiles), we store the sketch data.. It allows us to slice and dice on various dimensions dynamically and compute quantiles. Also helps us roll up the data to daily granularity and store it for a longer duration without losing the ability to compute percentiles dynamically

[sundy-li]

Now we don't have the ability to merge sketches into a merged sketch during compaction (Rollup is not supported now). So if you insert N rows into databend, it'll always be N rows, we can do state merging during the query stage.

[nagarajatantry]

got it.. that is okay I think. If I can merge it during query time natively, that would be fantastic.