Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: storage write api veneer #328

Merged
merged 63 commits into from
Aug 10, 2023
Merged

feat: storage write api veneer #328

merged 63 commits into from
Aug 10, 2023

Conversation

alvarowolfx
Copy link
Contributor

This PR adds a layer on top of the GAPIC generated BigQuery Storage Write API client to make it easier to use. Some of the benefits of using the managed writer layer:

  • Automatic connection management.
  • Promise like interface to wait for writes.
  • Make easier to send protobufs. Do not require users to write an equivalent proto file and proto descriptor. A package is going to be added to convert a BigQuery Table Schema to a proto descriptor. More on feat: adapt package to convert TableSchema to ProtoDescriptor #326

Towards internal b/214063522
Supersedes #300

Lo Ferris and others added 27 commits August 30, 2022 15:08
writer_client_sandbox
d744328
refactored pseudo classes
a86c3ae
abstracting out helpers
db3fe28
first pass mvp
36b207f
draft writer_client
bea869c
updating v1 package with writer_client
3c9a921
finalizing draft class
1e48acc
Merge branch 'main' of github.com:googleapis/nodejs-bigquery-storage …
738357f 
…into writer_veneer_sandbox

merging local and remote
removing sandbox
ddf5329
veneer refactor
3ffa986
update
9cc6e66
working client, rough draft
18731b3
working version
009b7ef
move veneer out of v1
11bfb89
refactor of managed client
ce4fafa
rewrite to writer client
cae103d
stashing test progress
f7f5dda
changing constructor logic
316accc
adding initialization test
84a0746
client with added abstraction for connections and main tests
f47539b
rewriting tests to load protos from JSON
e8301d5
refactor of acceptable protobuf messages
b19f095
full happy path test suite
d90e788
refactor test fixtures
a666747
@loferris
Merge branch 'main' into writer_veneer_sandbox
37677d2
@alvarowolfx
feat: add ManagedStream and PendingWrite abstractions
828a487
@alvarowolfx
feat: add test for managed writer against real api
2e9928d
@product-auto-label product-auto-label bot added size: xl Pull request size is extra large. api: bigquerystorage Issues related to the googleapis/nodejs-bigquery-storage API. labels Feb 27, 2023
@punteroo punteroo mentioned this pull request Jul 4, 2023
Closed
@alvarowolfx alvarowolfx added the owlbot:ignore instruct owl-bot to ignore a PR label Aug 1, 2023
@alvarowolfx alvarowolfx added owlbot:run Add this label to trigger the Owlbot post processor. and removed owlbot:ignore instruct owl-bot to ignore a PR labels Aug 2, 2023
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Aug 2, 2023
@alvarowolfx alvarowolfx requested review from feywind, alexander-fenster and loferris and removed request for alexander-fenster and feywind August 2, 2023 14:54
loferris
loferris reviewed Aug 9, 2023
View reviewed changes
Copy link
Contributor

@loferris loferris left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some outstanding questions about test coverage, that I noted, but otherwise LGTM.

@loferris loferris self-requested a review August 9, 2023 19:46
loferris
loferris approved these changes Aug 9, 2023
View reviewed changes
Copy link
Contributor

@loferris loferris left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Questions resolved, LGTM.

@alvarowolfx alvarowolfx requested review from bcoe and feywind and removed request for feywind, alexander-fenster and bcoe August 10, 2023 14:20
feywind
feywind approved these changes Aug 10, 2023
View reviewed changes
Copy link

@feywind feywind left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I trust Lo's review :)

samples/append_rows_buffered.js
@@ -0,0 +1,123 @@
// Copyright 2022 Google LLC
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might have to change this to 2023 for the linter to like it.

@alvarowolfx alvarowolfx merged commit 146141e into googleapis:main Aug 10, 2023
@release-please release-please bot mentioned this pull request Aug 10, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bcoe bcoe bcoe left review comments

@shollyman shollyman shollyman approved these changes

@loferris loferris loferris approved these changes

@feywind feywind feywind approved these changes

Assignees

@alvarowolfx alvarowolfx

Labels
api: bigquerystorage Issues related to the googleapis/nodejs-bigquery-storage API. size: xl Pull request size is extra large.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@alvarowolfx @rhodgkins @bcoe @shollyman @loferris @feywind @yoshi-kokoro