Control the max number of row and bytes lighting sends to tidb

[guoshouyan]

Feature Request

Is your feature request related to a problem? Please describe:

In lightning physical backend, we can set the store-write-bwlimit to control how much data lightning send to tikv.
Can we have similar feature in logical mode?
We already have minDeliverBytes here but it's not configurable.
Can we make it configurable and also have another config like minDeliverRows to control the rows it sends?

Describe the feature you'd like:

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

[lance6716]

Hi, not sure if I understand it correctly. In logical backend lightning will try to send data every 1MB. (be.MaxChunkSize() always return 1MB)

tidb/br/pkg/lightning/backend/tidb/tidb.go

Lines 614 to 617 in a427113

	for _, r := range rows.SplitIntoChunks(be.MaxChunkSize()) {
	for i := 0; i < writeRowsMaxRetryTimes; i++ {
	// Write in the batch mode first.
	err = be.WriteBatchRowsToDB(ctx, tableName, columnNames, r)

with the concurrency of lightning.region-concurrency from the configuration file. I think you want to make 1MB configurable?

And can you explain why the default value is not good enough?

[D3Hunter]

Hi, not sure if I understand it correctly. In logical backend lightning will try to send data every 1MB. (be.MaxChunkSize() always return 1MB)

tidb/br/pkg/lightning/backend/tidb/tidb.go

Lines 614 to 617 in a427113

	for _, r := range rows.SplitIntoChunks(be.MaxChunkSize()) {
	for i := 0; i < writeRowsMaxRetryTimes; i++ {
	// Write in the batch mode first.
	err = be.WriteBatchRowsToDB(ctx, tableName, columnNames, r)

with the concurrency of lightning.region-concurrency from the configuration file. I think you want to make 1MB configurable?

And can you explain why the default value is not good enough?

this 1M is useless, as deliverLoop send in 96K batch, see #46997 (comment)