[HUDI-3404] Automatically adjust write configs based on metadata table and write concurrency mode

[yihua]

What is the purpose of the pull request

This PR adds logic to automatically adjust write configs based on metadata table enablement and write concurrency mode, to prevent possible missing configs that are required for enabling metadata table and multi-writer. Write configs in scope are:

hoodie.metadata.enable (read-only in the new logic)
hoodie.write.concurrency.mode (can be auto updated in the new logic)
hoodie.write.lock.provider (can be auto updated in the new logic)
hoodie.cleaner.policy.failed.writes (can be auto updated in the new logic)

The following scenarios are checked:

Metadata Table	Async Table Services	Lock Provider Set by User	Action
Enabled	Enabled	No	Set hoodie.write.concurrency.mode=optimistic_concurrency_control, hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.InProcessLockProvider and hoodie.cleaner.policy.failed.writes=LAZY
Enabled	Enabled	Yes	Set hoodie.cleaner.policy.failed.writes=LAZY if user has set hoodie.write.concurrency.mode=optimistic_concurrency_control
Enabled	Disabled	Yes/No	Set hoodie.cleaner.policy.failed.writes=LAZY if user has set hoodie.write.concurrency.mode=optimistic_concurrency_control
Disabled	Enabled/Disabled	Yes/No	Set hoodie.cleaner.policy.failed.writes=LAZY if user has set hoodie.write.concurrency.mode=optimistic_concurrency_control

Note that the logic also depends on Hudi table type, since by default, MOR table has async compaction enabled, while COW table has all inline table services.

Brief change log

• Adds auto config adjustment to in HoodieWriteConfig.
• Adds unit tests for different user-defined config scenarios in TestHoodieWriteConfig to make sure the logic is correct.

Verify this pull request

This change adds tests in TestHoodieWriteConfig to verify the logic.

Committer checklist

• Has a corresponding JIRA in PR title & commit

• Commit message is descriptive of the change

• CI is green

• Necessary doc changes done or have another open PR

• For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

[nsivabalan]

if I am not wrong, we discussed that w/ deltatstreamer, its not easy to distinguish if async table services are enabled since no explicit configs are set. So, don't we need some fix in deltastreamer class towards this ?

[nsivabalan]

based on our discussion, lets add table type check and check for compaction only for MOR.
bcoz, incase of Deltastreamer, no explicit compaction configs are set, but still we need to detect if async table services are enabled or not.

[yihua]

based on our discussion, lets add table type check and check for compaction only for MOR. bcoz, incase of Deltastreamer, no explicit compaction configs are set, but still we need to detect if async table services are enabled or not.

I added the logic based on table type. By default COW table should not have any async table service enabled, while MOR table have async compaction enabled. So the auto config adjustment for metadata table is different for different types of tables.

[hudi-bot]

CI report:

• 1bc5f11 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[nsivabalan]

Good job on the patch Ethan!