Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BitSail][Common]Fix ArrayIndexOutOfBoundsException when hive table is non-partitioned #495

Merged

Conversation

humengyu2012
Copy link
Contributor

We ran a task to transfer data from MySQL to Hive. Hive is a non-partitioned table, and we encountered the following error:

he cause of the job failure maybe due to:
java.lang.RuntimeException: Error while calling HiveMetaClientUtil::addPartition. Retrying failed to complete successfully after 5 attempts.
	at com.bytedance.bitsail.shaded.hive.client.HiveMetaClientUtil.addPartition(HiveMetaClientUtil.java:482)
	at com.bytedance.bitsail.connector.legacy.hive.sink.HiveOutputFormat.onSuccessComplete(HiveOutputFormat.java:450)
	at com.bytedance.bitsail.flink.core.writer.PluginableOutputFormatDAGBuilder.commit(PluginableOutputFormatDAGBuilder.java:133)
	at com.bytedance.bitsail.flink.core.execution.FlinkExecutionEnviron.doGlobalCommit(FlinkExecutionEnviron.java:289)
	at com.bytedance.bitsail.flink.core.execution.FlinkExecutionEnviron.run(FlinkExecutionEnviron.java:171)
	at com.bytedance.bitsail.core.api.program.UnifiedProgram.submit(UnifiedProgram.java:122)
	at com.bytedance.bitsail.core.Engine.run(Engine.java:96)
	at com.bytedance.bitsail.core.Engine.start(Engine.java:65)
	at com.bytedance.bitsail.core.Engine.main(Engine.java:55)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
	at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:98)
	at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:843)
	at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:240)
	at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1087)
	at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1165)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1772)
	at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
	at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1165)
Caused by: com.github.rholder.retry.RetryException: Retrying failed to complete successfully after 5 attempts.
	at com.github.rholder.retry.Retryer.call(Retryer.java:174)
	at com.bytedance.bitsail.shaded.hive.client.HiveMetaClientUtil.addPartition(HiveMetaClientUtil.java:440)
	... 24 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
	at com.bytedance.bitsail.shaded.hive.client.HiveMetaClientUtil.lambda$addPartition$11(HiveMetaClientUtil.java:451)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:206)
	at com.bytedance.bitsail.common.util.timelimit.SimpleTimeLimiter.callWithTimeout(SimpleTimeLimiter.java:181)
	at com.bytedance.bitsail.common.util.timelimit.FixedAttemptTimeLimit.call(FixedAttemptTimeLimit.java:52)
	at com.github.rholder.retry.Retryer.call(Retryer.java:160)
	... 25 more

Copy link
Collaborator

@garyli1019 garyli1019 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@garyli1019 garyli1019 merged commit 28c41a2 into bytedance:master Sep 12, 2023
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants