[HUDI-5251] Split GitHub actions CI by spark and flink

[xushiyan]

Change Logs

flink bundle use 1.10.0 avro version, but hudi common modes were generated during build process controlled by avro.version property, which is not overwritten in flink profiles. This would resulted in generated models being incompatible with included avro lib in flink-bundle.

This change separate out spark and flink build process in GH actions CI so that spark and flink profiles do not interfere each other. For example, spark 3.3 profile uses avro 1.11 which should not be used for flink bundle.

Related issues

• [SUPPORT] The hudi-flink1.15-bundle on maven repository still not usable #7259
• [SUPPORT] HoodieFlinkCompactor with NoSuchMethodError: org.apache.hudi.org.apache.avro.specific.SpecificRecordBuilderBase #7047

Follow up

We need to make sure release artifacts not to be impacted by the profile settings.

• [HUDI-5357] Optimize deployment of release artifacts #7419

Impact

Build process for flink bundles has to be separated out from other bundles. Dev and release processes should be updated accordingly.

Risk level

low

Documentation Update

• build instructions to be updated
• release process to be updated

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[danny0405]

+1, i can not validate locally because of the compile error on MacPro m1, but the direction is correct.

[xushiyan]

@danny0405 to land this pr, there is a bigger implication: we have to restructure the way CI is built and run. We need to separate flink and spark build process as spark 2.4 and 3.1 uses avro 1.8.2 which conflicts with flink profiles. The integration test is trickier where all bundles are put together. We can only do it when using spark 3.2 profile (avro 1.10.2) to fit with flink profiles. I have to get back to this later. but we'll still need to resolve this for 0.12.2

[nsivabalan]

@xushiyan : do we need to target this for 0.12.2 ?

[xushiyan]

@xushiyan : do we need to target this for 0.12.2 ?

yes this issue is a show-stopper for flink bundle

[xushiyan]

@nsivabalan @danny0405 overwriting avro.version in flink profiles causes conflicts in how we setup CI today. I've kept the old way of overwriting version but make flink-bundle built differently for release. pls review #7419

[hudi-bot]

CI report:

• 21cb894 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build