[SUPPORT] [Metadata table] Metadata table heartbeat for instant has expired, last heartbeat 0

[BruceKellan]

Describe the problem you faced

I am using hudi to write table.
When I stop the application at 11:13 AM，and restart the application at 12: 00 AM.
The application start failed. It seem like some failed happen.

Flink Configuration
The configuration related to metadata table are same as default configuration.
The configuration like this:

(
  'connector' = 'hudi',
  'path' = 'oss://xxxx/xxx',
  'table.type' = 'COPY_ON_WRITE',
  'write.operation' = 'insert',
  'write.bucket_assign.tasks' = '1',
  'write.tasks' = '12',
  'write.task.max.size' = '1024',
  'hive_sync.enable' = 'true',
  'hive_sync.mode' = 'hms',
  'hive_sync.db' = 'test_db',
  'hive_sync.table' = 'test_table',
  'hive_sync.file_format' = 'HFILE',
  'hive_sync.partition_fields' = 'day,type',
  'hive_sync.partition_extractor_class' = 'org.apache.hudi.hive.MultiPartKeysValueExtractor',
  'hoodie.datasource.write.recordkey.field' = 'uniqueKey',
  'hoodie.datasource.write.partitionpath.field' = 'day,type',
  'hoodie.datasource.write.hive_style_partitioning' = 'true',
  'hoodie.datasource.write.keygenerator.type' = 'COMPLEX',
  'hoodie.parquet.compression.codec' = 'snappy',
  'write.insert.cluster' = 'true',
  'write.parquet.max.file.size' = '50',
  'write.parquet.block.size' = '70',
  'hoodie.parquet.small.file.limit' = '52428800',
  'clean.retain_commits' = '480',
  'hoodie.archive.merge.enable' = 'true',
  'hoodie.archive.merge.small.file.limit.bytes' = '62914560',
  'metadata.enabled' = 'true'
)

Environment Description

• Flink version : 1.13

• Hudi version : 0.11.0

• Hive version : 2.3.6

• Hadoop version : 2.8.5

• Storage (HDFS/S3/GCS..) : Aliyun OSS

Stacktrace

2022-05-12 12:15:50
org.apache.flink.util.FlinkException: Global failure triggered by OperatorCoordinator for 'stream_write' (operator e45d100e470b831662a314d7036975b3).
	at org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder$LazyInitializedCoordinatorContext.failJob(OperatorCoordinatorHolder.java:594)
	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$start$0(StreamWriteOperatorCoordinator.java:180)
	at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$execute$0(NonThrownExecutor.java:103)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hudi.exception.HoodieException: Executor executes action [initialize instant 20220512111342536] error
	... 5 more
Caused by: org.apache.hudi.exception.HoodieException: Heartbeat for instant 20220512111342536 has expired, last heartbeat 0
	at org.apache.hudi.client.heartbeat.HeartbeatUtils.abortIfHeartbeatExpired(HeartbeatUtils.java:92)
	at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:229)
	at org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:112)
	at org.apache.hudi.metadata.FlinkHoodieBackedTableMetadataWriter.commit(FlinkHoodieBackedTableMetadataWriter.java:152)
	at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.processAndCommit(HoodieBackedTableMetadataWriter.java:803)
	at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java:870)
	at org.apache.hudi.client.HoodieFlinkWriteClient.lambda$writeTableMetadata$1(HoodieFlinkWriteClient.java:262)
	at org.apache.hudi.common.util.Option.ifPresent(Option.java:97)
	at org.apache.hudi.client.HoodieFlinkWriteClient.writeTableMetadata(HoodieFlinkWriteClient.java:260)
	at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:269)
	at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:234)
	at org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:112)
	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:516)
	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:492)
	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:463)
	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$initInstant$6(StreamWriteOperatorCoordinator.java:392)
	at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$execute$0(NonThrownExecutor.java:93)
	... 3 more

[nsivabalan]

@danny0405 @wangxianghu : Is metadata table tested/certified w/ flink. can you chime in here please.

[danny0405]

Have fired a fix here: #5583

[BruceKellan]

Thanks, I will do some test with this patch.

[BruceKellan]

I will close this issue. If there are still problems I will reopen this issue.