feat: integrate jobs.query and stateless query for faster queries #1337

alvarowolfx · 2024-02-16T15:40:42Z

Integrate with jobs.query endpoint for faster small queries. Keep using jobs.getQueryResults when pagination is needed. Fallback to current method of creating a job + jobs.getQueryResults when not possible to use jobs.query.

feywind

Someone with more BQ experience should give this a look too, but I don't see major Node issues :)

src/bigquery.ts

feywind · 2024-03-20T18:32:46Z

src/bigquery.ts

+    params: Query['params'],
+    types: Query['types']
+  ): {
+    parameterMode: 'positional' | 'named' | undefined;


Might be good to pull this type out into an interface, but YMMV.

feywind · 2024-03-20T18:37:24Z

src/bigquery.ts

+    const queryParameters: bigquery.IQueryParameter[] = [];
+    if (parameterMode === 'named') {
+      const namedParams = params as {[param: string]: any};
+      for (const namedParameter in namedParams) {


Might want to use for (const namedParameter of Object.getOwnPropertyNames(namedParams)) or check each namedParameter against namedParams.hasOwnProperty().

src/bigquery.ts

feywind · 2024-03-20T18:50:14Z

src/bigquery.ts

+        let rows: any = [];
+        if (res.schema && res.rows) {
+          rows = BigQuery.mergeSchemaWithRows_(res.schema, res.rows, {
+            wrapIntegers: options.wrapIntegers!, // TODO: fix default value


Same as above (re: not sure if this was moved from elsewhere or new code) but I wanted to ping in case the TODO isn't intentional.

feywind · 2024-03-20T18:54:40Z

src/bigquery.ts

-      // The Job is important for the `queryAsStream_` method, so a new query
-      // isn't created each time results are polled for.
-      options = extend({job}, queryOpts, options);
+      if (options.timeoutMs) {


You can just delete, no need for the if.

src/bigquery.ts

shollyman · 2024-03-25T19:04:51Z

src/bigquery.ts

+      requestId: uuid.v4(),
+      jobCreationMode: 'JOB_CREATION_OPTIONAL',
+    };
+    if (req.maxResults) {


Temporary workaround?

gonna remove that

src/job.ts

src/logger.ts

src/bigquery.ts

shollyman

Looks like reasonable coverage for the expansion here, thanks for putting it together.

feywind

All still looks fine. I wonder if we have a code style standard for foo_ vs _foo. Not really something specific to this PR.

alex-statsig · 2024-08-09T21:37:07Z

Hey @alvarowolfx , thanks for contributing this!

I tried using this out but noticed it seems like the types might be incorrect. In particular, when I use jobs.query(query, options) (or jobs.query(query)), it seems to return 3 items of the shape [ResultsArray, {job: IJob}, JobStatistics]. The types indicate the return type is [ResultsArray, IJob] (note in particular that job is not nested under a key "job" for the second value). Does that sound correct to you?

alex-statsig · 2024-08-09T21:52:05Z

Also do you know if there is a reason jobTimeoutMs cannot be set? Seems like a nice safety net for runaway queries that is unavailable here (unless timeoutMs implicitly stops the query, but it doesn't seem like it)

edit: Also timeoutMs doesn't seem to behave as I would expect. It seems like its a timeout for the initial request only, but then the option is deleted and it will wait for the results again still which may take much longer? I want to ensure my requests don't exceed 1s, but it seems like for a slow query it will still wait 10+s. The "job not complete" log fires fairly fast, but the results still don't resolve until the query actually finishes

feat: integrate jobs.query and jobless query for faster queries

c2a6841

product-auto-label bot added size: m Pull request size is medium. api: bigquery Issues related to the googleapis/nodejs-bigquery API. labels Feb 16, 2024

alvarowolfx added 5 commits February 22, 2024 11:10

feat: rename probeFastPath method to convert queryRequests

344c2ce

fix: avoid promisify of buildQueryRequest method

ed4381c

Merge branch 'main' into feat-fast-query-path

af93d2b

feat: re-generate types

1a3be39

feat: support stateless queries

0717724

alvarowolfx added the owlbot:run Add this label to trigger the Owlbot post processor. label Feb 29, 2024

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Feb 29, 2024

feat: fully build QueryRequest object for jobs.query

970e08f

product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Mar 11, 2024

fix: lint issues

bce8c48

alvarowolfx mentioned this pull request Mar 15, 2024

Use Template Strings for GoogleSQL raw queries #1342

Open

alvarowolfx changed the title ~~feat: integrate jobs.query and jobless query for faster queries~~ feat: integrate jobs.query and stateless query for faster queries Mar 15, 2024

alvarowolfx marked this pull request as ready for review March 18, 2024 19:17

alvarowolfx requested review from a team as code owners March 18, 2024 19:17

alvarowolfx requested a review from farhan0102 March 18, 2024 19:17

alvarowolfx added the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 18, 2024

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 18, 2024

Merge branch 'main' into feat-fast-query-path

3757849

alvarowolfx requested review from tswast, shollyman and feywind March 19, 2024 17:59

feat: only use jobCreationMode with preview feature enable

9a2cce5

alvarowolfx added the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 19, 2024

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 19, 2024

alvarowolfx added the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 19, 2024

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 19, 2024

feywind approved these changes Mar 20, 2024

View reviewed changes

shollyman reviewed Mar 25, 2024

View reviewed changes

fix: address code review comments

7531a12

alvarowolfx requested review from feywind and shollyman March 26, 2024 15:43

alvarowolfx added the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 26, 2024

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 26, 2024

shollyman removed the request for review from farhan0102 March 26, 2024 17:44

shollyman approved these changes Mar 26, 2024

View reviewed changes

feywind approved these changes Mar 26, 2024

View reviewed changes

Merge branch 'main' into feat-fast-query-path

7141834

alvarowolfx added the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 27, 2024

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 27, 2024

Merge branch 'main' into feat-fast-query-path

f6b19df

alvarowolfx added the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 27, 2024

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Mar 27, 2024

alvarowolfx merged commit 74aa150 into googleapis:main Mar 27, 2024
15 checks passed

release-please bot mentioned this pull request Mar 27, 2024

chore(main): release 7.6.0 #1349

Merged

This was referenced Apr 23, 2024

April 22, 2024 kitta65/bq-extension-vscode#374

Closed

April 22, 2024 kitta65/prettier-plugin-bq#364

Closed

April 22, 2024 kitta65/bq2cst#349

Closed

alvarowolfx mentioned this pull request Apr 25, 2024

Direct usage of queryParameters broke with 7.6.0 #1357

Closed

WontonSam mentioned this pull request Jul 17, 2024

[Snyk] Upgrade @google-cloud/bigquery from 6.0.3 to 7.8.0 WontonSam/repo-automation-bots#191

Open

Pyloris mentioned this pull request Jul 31, 2024

[Snyk] Upgrade @google-cloud/bigquery from 7.5.1 to 7.8.0 Pyloris/nango#4

Open

alex-statsig mentioned this pull request Aug 30, 2024

Several issues with stateless queries #1399

Closed

WontonSam mentioned this pull request Sep 20, 2024

[Snyk] Upgrade: , , , gaxios, , , , , , ajv, c8, jsonwebtoken, lru-cache, yargs, gcf-utils, gts, mocha, nock, sinon, typescript WontonSam/repo-automation-bots#511

Open

WontonSam mentioned this pull request Oct 3, 2024

[Snyk] Upgrade @google-cloud/bigquery from 6.0.3 to 7.9.0 WontonSam/repo-automation-bots#588

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: integrate jobs.query and stateless query for faster queries #1337

feat: integrate jobs.query and stateless query for faster queries #1337

alvarowolfx commented Feb 16, 2024

feywind left a comment

feywind Mar 20, 2024

feywind Mar 20, 2024

feywind Mar 20, 2024

feywind Mar 20, 2024

shollyman Mar 25, 2024

alvarowolfx Mar 25, 2024

shollyman left a comment

feywind left a comment

alex-statsig commented Aug 9, 2024 •

edited

Loading

alex-statsig commented Aug 9, 2024 •

edited

Loading

feat: integrate jobs.query and stateless query for faster queries #1337

feat: integrate jobs.query and stateless query for faster queries #1337

Conversation

alvarowolfx commented Feb 16, 2024

feywind left a comment

Choose a reason for hiding this comment

feywind Mar 20, 2024

Choose a reason for hiding this comment

feywind Mar 20, 2024

Choose a reason for hiding this comment

feywind Mar 20, 2024

Choose a reason for hiding this comment

feywind Mar 20, 2024

Choose a reason for hiding this comment

shollyman Mar 25, 2024

Choose a reason for hiding this comment

alvarowolfx Mar 25, 2024

Choose a reason for hiding this comment

shollyman left a comment

Choose a reason for hiding this comment

feywind left a comment

Choose a reason for hiding this comment

alex-statsig commented Aug 9, 2024 • edited Loading

alex-statsig commented Aug 9, 2024 • edited Loading

alex-statsig commented Aug 9, 2024 •

edited

Loading

alex-statsig commented Aug 9, 2024 •

edited

Loading