Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: BQIO fails to keep track of offset correctly, causing write exceptions when using Storage Write API in batch mode #26520

Closed
2 of 15 tasks
johnjcasey opened this issue May 3, 2023 · 1 comment
Closed
2 of 15 tasks

[Bug]: BQIO fails to keep track of offset correctly, causing write exceptions when using Storage Write API in batch mode #26520

johnjcasey opened this issue May 3, 2023 · 1 comment
Labels
awaiting triage bigquery bug io java P1
Milestone
2.48.0 Release

Comments

@johnjcasey
Copy link
Contributor

johnjcasey commented May 3, 2023 •
edited
Loading

What happened?

BQIO was lazily creating streams, which means that all our appends to a given stream would be appended at index 0.
This meant that for any second append to a stream, we would get an exception from BQ, and be forced to retry our work. This results in bad performance by itself, but combined with #26521 resulted in data loss for beam versions 2.44 - 2.47 inclusive.

Issue Priority

Priority: 1 (data loss / total loss of function)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
@github-actions github-actions bot added the java label May 3, 2023
This was referenced May 3, 2023
Closed
Closed
Merged
@johnjcasey
Copy link
Contributor Author

Fixed by #26503

@github-actions github-actions bot added this to the 2.48.0 Release milestone May 3, 2023
@tunetheweb tunetheweb mentioned this issue Jul 9, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
awaiting triage bigquery bug io java P1
Projects
None yet
Milestone
2.48.0 Release
Development

No branches or pull requests
1 participant
@johnjcasey