how to fix data loss when using the transaction and some kafka partitions are down

[jackjoesh]

backgroud:
As we all know, inflightMessages just add tx-commit record. That means if we start a local transaction as the following:
begin transaction
insert A (no tx commit）
insert B （tx commit）
end transaction

And if "insert A" and "insert B" dispatch to different kafka partitions, "insert B" completes callbacking before "insert A" out of order.
The event timeline is as the following:

1 "insert B" completes callbacking and updates the this tx's position （tx commit）
2 some kafka partitions are down
3 partitons don't receive "insert A" data yet

Finally, dose it main we lose the data "insert A" ? Is there any way to fix this problem?

[osheroff]

The design of maxwell's InflightMessage buffer is towards at-least-once delivery: all rows have to commit before maxwell.positions is incremented

So if you have:

row 1 <- succeeded
row 2 <- blocked
row 3 <- succeeded

rows 1 and 3 will reach kafka but maxwell.positions will not commit. If the partition stays blocked long enough, maxwell's buffers will fill and the process will stop. If maxwell crashes and row 2 is still not committed, maxwell will replay from row 1.

[jackjoesh]

hi osheroff, it seems that you don't get my meaning, let me explain again

InflightMessage doesn't record no final tx-commit action in a local transaction even it's a insert action as the following

so in my above case:
begin transaction
insert A (no tx commit）
insert B （tx commit）
end transaction

insert A (no tx commit) doesn't be added in InflightMessage.

so if "insert A " 's position is 1, insert B （tx commit）'s position is 2.

Then the all process is as the following timeline:
1 owing to different partitions' out of order, "insert B" completes callbacking before "insert A" and updates the tx's position to 2（tx commit）
2 some kafka partitions are down
3 partitons don't receive "insert A" data yet （position 1 actually doesn't send kafka successfully）

At last, when maxwell restarts, it will continue from position 2(insert B) but not position 1(insert A), and insert A doesn't send kafka message successfully, right? @osheroff

[osheroff]

ah, yeah, sorry. I missed that the rows inside a transaction are sent to different positions.

#2019 addresses this, but I do want to be a bit cautious about the increased memory usage possible here.

[jackjoesh]

ah, yeah, sorry. I missed that the rows inside a transaction are sent to different positions.

#2019 addresses this, but I do want to be a bit cautious about the increased memory usage possible here.

thank you , I get it. Would you also make DEFAULT_CAPACITY in InflightMessageList can be configured? That means we need to increase this capactity