Skip to content
This repository has been archived by the owner on Jan 5, 2024. It is now read-only.

Unable to use hudi sink #22

Open
ankitdimania-eligible opened this issue Oct 8, 2019 · 2 comments
Open

Unable to use hudi sink #22

ankitdimania-eligible opened this issue Oct 8, 2019 · 2 comments

Comments

@ankitdimania-eligible
Copy link

I'm trying to use Hudi sink but the write to Hudi-client is not working. Has anyone seen this issue before:

2019-10-08 20:29:46 INFO  HoodieTableConfig:69 - Loading dataset properties from /tmp/hoodie/payment_reports_18/.hoodie/hoodie.properties
2019-10-08 20:29:46 INFO  HoodieTableMetaClient:94 - Finished Loading Table of type COPY_ON_WRITE from /tmp/hoodie/payment_reports_18
2019-10-08 20:29:46 INFO  HoodieTableMetaClient:96 - Loading Active commit timeline for /tmp/hoodie/payment_reports_18
2019-10-08 20:29:46 INFO  HoodieActiveTimeline:77 - Loaded instants java.util.stream.ReferencePipeline$Head@1a850eab
2019-10-08 20:29:46 INFO  HoodieActiveTimeline:212 - Marking instant complete [==>20191008202942__commit__INFLIGHT]
2019-10-08 20:29:46 INFO  HoodieActiveTimeline:377 - Created a new file in meta path: /tmp/hoodie/payment_reports_18/.hoodie/20191008202942.inflight
2019-10-08 20:29:46 ERROR JobDag:188 - Failed in JobDag
com.uber.hoodie.exception.HoodieIOException: Could not rename /tmp/hoodie/payment_reports_18/.hoodie/20191008202942.inflight to /tmp/hoodie/payment_reports_18/.hoodie/20191008202942.commit
        at com.uber.hoodie.common.table.timeline.HoodieActiveTimeline.transitionState(HoodieActiveTimeline.java:330)
        at com.uber.hoodie.common.table.timeline.HoodieActiveTimeline.saveAsComplete(HoodieActiveTimeline.java:215)
        at com.uber.hoodie.HoodieWriteClient.commit(HoodieWriteClient.java:518)
        at com.uber.hoodie.HoodieWriteClient.commit(HoodieWriteClient.java:491)
        at com.uber.marmaray.common.sinks.hoodie.HoodieSink$HoodieWriteClientWrapper.commit(HoodieSink.java:489)
        at com.uber.marmaray.common.sinks.hoodie.HoodieSink.commit(HoodieSink.java:290)
        at com.uber.marmaray.common.sinks.hoodie.HoodieSink.commit(HoodieSink.java:267)
        at com.uber.marmaray.common.sinks.hoodie.HoodieSink.write(HoodieSink.java:184)
        at com.uber.marmaray.common.sinks.hoodie.HoodieSink.write(HoodieSink.java:161)
        at com.uber.marmaray.common.job.SingleSinkSubDag.executeNode(SingleSinkSubDag.java:51)
        at com.uber.marmaray.common.job.JobSubDag.execute(JobSubDag.java:149)
        at com.uber.marmaray.common.job.JobDag.execute(JobDag.java:171)
        at com.uber.marmaray.common.job.JobManager.lambda$null$0(JobManager.java:207)
        at com.uber.marmaray.common.job.ThreadPoolService$ThreadPoolServiceCallable.call(ThreadPoolService.java:415)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
@tooptoop4
Copy link

@ankitdimania-eligible did u solve? i get similar

2020-02-10 18:04:24,707 [main] INFO com.uber.hoodie.common.table.timeline.HoodieActiveTimeline - Marking instant complete [==>20200210180348__commit__INFLIGHT]
Exception in thread "main" com.uber.hoodie.exception.HoodieIOException: Could not rename s3a://x/.hoodie/20200210180348.inflight to s3a://x/.hoodie/20200210180348.commit
at com.uber.hoodie.common.table.timeline.HoodieActiveTimeline.transitionState(HoodieActiveTimeline.java:328)
at com.uber.hoodie.common.table.timeline.HoodieActiveTimeline.saveAsComplete(HoodieActiveTimeline.java:213)
at com.uber.hoodie.HoodieWriteClient.commit(HoodieWriteClient.java:529)
at com.uber.hoodie.HoodieWriteClient.commit(HoodieWriteClient.java:489)
at com.uber.hoodie.HoodieWriteClient.commit(HoodieWriteClient.java:480)

@ankitdimania-eligible
Copy link
Author

@tooptoop4 yeah, the above issue is bause of double commits. I had auto commit set to true and then I was mistakenly trying to commit again.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants