chunked insertion (by size of blocks)

[dantengsky]

Summary

Original:

while taking in data blocks during Table::append, chunks several small data blocks into a larger one if possible.

Updated (2022-03-30):

Although this issue has been fixed by PR #3122, later a strategy of merging small blocks into bigger ones by the number of rows is used.

It would be better if both bock_size(in bytes, before compression, imo) and number_rows are taken into account while merging small blocks into larger ones. such that the size of memory used to construct the block will no longer increase proportionally to the number of columns of the table.

current impl:

https://github.com/datafuselabs/databend/blob/50226158b19380f06d36edb759f98482027c51db/query/src/storages/fuse/io/write/block_stream_writer.rs#L278-L304

[dantengsky]

addressed in PR #3122

[dantengsky]

Updated (2022-03-30):

Although this issue has been fixed by PR #3122, later a strategy of merging small blocks into bigger ones by the number of rows is used.

It would be better if both bock_size(in bytes, before compression, imo) and nubmer_of_rows are taken into account while merging small blocks into larger ones. such that the size of memory used to construct the block will no longer increase proportionally to the number of columns of the table.

current impl:

https://github.com/datafuselabs/databend/blob/50226158b19380f06d36edb759f98482027c51db/query/src/storages/fuse/io/write/block_stream_writer.rs#L278-L304

[BohuTANG]

Hello @dantengsky ,
is it related to async insert #4577?

[dantengsky]

Hello @dantengsky , is it related to async insert #4577?

Hope I get it right, in issue #4577, expressions (values of batched insert statements) are going to be compacted into blocks of proper size if applicable and then insert into a table by calling Table::append.

• apparently, the config /setting of the block size could be shared.
• It may also be worth introducing another param to Table::append, as a hint that indicates the blocks being appended are well-sized.
• There seems to be something in common between
  • compacting expressions into blocks of the proper size
  • compacting blocks into blocks of the proper size
  but frankly speaking, I do not know how to extract them. @sundy-li please help

[sundy-li]

Async Insert is used to aggregate multi-client's small insert.

Seems it's pretty different with this I think.

[BohuTANG]

I got, thank you man!

[sundy-li]

@dantengsky We already have TransformBlockCompact which has setting max_row_per_block, min_row_per_block. maybe you can introduce more settings into this struct.

[dantengsky]

staled