[HUDI-4717] CompactionCommitEvent message corrupted when sent by compact_task

[nonggialiang]

…act_task

Change Logs

Use mail-box executor to send compaction-commit-event in CompactFunction.

Impact

Potential bug fixed.

Risk level: none | low | medium | high

Choose one. If medium or high, explain what verification was done to mitigate the risks.

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[danny0405]

Thanks, we may need to figure out more simpler way for collector concurrency.

[nonggialiang]

Just pushed a new implementaion using AsynFunction of flink.
And fixed the same potential problem in clustering operator.

[hudi-bot]

CI report:

• e257940 Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[codope]

@nonggialiang Can you please rebase?

[littleeleventhwolf]

@nonggialiang After using AsyncFunction to rewrite the asynchronous compaction logic, have you solved the problem described in HUDI-4717？

[danny0405]

Does #7399 solve your problem here ?

[littleeleventhwolf]

Does #7399 solve your problem here ?

Yeah, we cherry-pick #7399. But if user enable latency-marker, there is still thread-safety problem.

[danny0405]

Does #7399 solve your problem here ?

Yeah, we cherry-pick #7399. But if user enable latency-marker, there is still thread-safety problem.

Does the Operator has any interface to handle the latency-marker yet ? Just like what we do to watermark.

[littleeleventhwolf]

Does #7399 solve your problem here ?

Yeah, we cherry-pick #7399. But if user enable latency-marker, there is still thread-safety problem.

Does the Operator has any interface to handle the latency-marker yet ? Just like what we do to watermark.

We can provide an empty method processLatencyMarker(LatencyMarker latencyMarker) to not propagate the latency-marker. But I don't think this is an elegant way, for two reasons:
1) if Flink generates a new StreamElement in future version, we also need to provide an empty method for compatibility;
2) if users want to use Watermark or LatencyMarker do some other work in Operator Sink:compact_commit, an empty method that does not propagate these events cannot support users' requirement.

[yihua]

@danny0405 is this PR still useful?

[danny0405]

yeah, just need to reach concensus.