New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prevent race condition in BQ sink jobId generation #877

Merged

feast-ci-bot merged 2 commits into feast-dev:master from pyalex:jobid-race-condition

Jul 16, 2020

Collaborator

pyalex commented Jul 13, 2020 •

edited

Loading

What this PR does / why we need it:

Getting rid of any DoFn state completely, so there won't be any race conditions.
TempFilePrefix is generated globally, since each data flush will be accompanied with generated UUID.
JobIdPrefix is generated in the end of the window (so Combine.globally without triggers can be used) since it's not needed earlier.

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?:

pyalex requested review from davidheryanto, khorshuheng, woop and zhilingc as code owners

July 13, 2020 14:24

feast-ci-bot added approved needs-kind size/M labels

pyalex added the kind/bug label

feast-ci-bot removed the needs-kind label

Collaborator Author

pyalex commented Jul 13, 2020

/test test-end-to-end-batch-dataflow

5 similar comments

Collaborator Author

pyalex commented Jul 13, 2020

/test test-end-to-end-batch-dataflow

Collaborator Author

pyalex commented Jul 13, 2020

/test test-end-to-end-batch-dataflow

Collaborator Author

pyalex commented Jul 13, 2020

/test test-end-to-end-batch-dataflow

Collaborator Author

pyalex commented Jul 13, 2020

/test test-end-to-end-batch-dataflow

Collaborator Author

pyalex commented Jul 14, 2020

/test test-end-to-end-batch-dataflow

feast-ci-bot added size/L and removed size/M labels

Collaborator Author

pyalex commented Jul 14, 2020

/test test-end-to-end-batch-dataflow

5 similar comments

Collaborator Author

pyalex commented Jul 14, 2020

/test test-end-to-end-batch-dataflow

Collaborator Author

pyalex commented Jul 14, 2020

/test test-end-to-end-batch-dataflow

Collaborator Author

pyalex commented Jul 14, 2020

/test test-end-to-end-batch-dataflow

Collaborator Author

pyalex commented Jul 14, 2020

/test test-end-to-end-batch-dataflow

Collaborator Author

pyalex commented Jul 14, 2020

/test test-end-to-end-batch-dataflow

pyalex force-pushed the jobid-race-condition branch from 4e10d19 to 9d49852 Compare

July 14, 2020 10:32

feast-ci-bot added size/XL and removed size/L labels


          fix race condition

88a2a0e

pyalex force-pushed the jobid-race-condition branch from 9d49852 to 88a2a0e Compare

July 14, 2020 10:33

feast-ci-bot added size/M and removed size/XL labels

Collaborator Author

pyalex commented Jul 14, 2020

/retest


          [bq] temp file prefix in global window

e7c4c0b

feast-ci-bot added size/L and removed size/M labels

Collaborator Author

pyalex commented Jul 16, 2020

/test publish-docker-images

Collaborator Author

pyalex commented Jul 16, 2020

/test test-end-to-end-batch-dataflow

Collaborator

khorshuheng commented Jul 16, 2020

/lgtm

feast-ci-bot assigned khorshuheng

feast-ci-bot added the lgtm label

khorshuheng approved these changes

View reviewed changes

Collaborator

feast-ci-bot commented Jul 16, 2020

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: khorshuheng, pyalex

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [khorshuheng,pyalex]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

feast-ci-bot merged commit 5b96c04 into feast-dev:master

pyalex pushed a commit to pyalex/feast that referenced this pull request


          Prevent race condition in BQ sink jobId generation (feast-dev#877)

8ce45c7

* fix race condition

* [bq] temp file prefix in global window

pyalex pushed a commit that referenced this pull request


          Prevent race condition in BQ sink jobId generation (#877)

17473f7

* fix race condition

* [bq] temp file prefix in global window

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved kind/bug lgtm size/L