BigQuery: Add dry_run option to BigQuery magic

[shubha-rajan]

See #8143
Adding the --dry_run flag returns a QueryJob instead of a pandas DataFrame as requested in the original issue. The QueryJob can also be stored in a variable if the destination_var magic argument is present

[plamut]

Thanks for the PR! It generally it looks good to me, apart from the dry_run argument help string.

I noticed, however, than when using the dry_run mode, an error is printed to the console:

In [16]: %%bigquery --dry_run 
    ...: SELECT * FROM `bigquery-public-data.samples.shakespeare` LIMIT 5; 
    ...:  
    ...:                                                                                                                                                                                                                                                                                                                                                                                                                          
Executing query with job ID: None

ERROR:
 404 GET https://www.googleapis.com/bigquery/v2/projects/precise-truck-742/queries/None?maxResults=0&location=US: Not found: Job precise-truck-742:US.None

(job ID: None)

                   -----Query Job SQL Follows-----                    

    |    .    |    .    |    .    |    .    |    .    |    .    |
   1:SELECT * FROM `bigquery-public-data.samples.shakespeare` LIMIT 5;
   2:
    |    .    |    .    |    .    |    .    |    .    |    .    |

This can be confusing for the users, and would be good to have it fixed. However, if fixing _run_query() proves to be too complex in the scope of this PR, we can make these changes separately.

[shubha-rajan]

I'll investigate that error being printed out. I agree that it would be confusing to users, and don't think it's outside of the scope of the PR

[tswast]

When you run a dry run query, a job is not actually created, so fetching the results fails.

[tswast]

Have you run this in a notebook, yet? I'm curious what the output looks like. We probably want to improve the __str__ or __repr__ functions on QueryJob to at least show estimated_bytes_processed, because that's what people running dryrun queries are going to care about.

[shubha-rajan]

@tswast as of now, running it in a notebook is silent. I tried modifying _run_query() and that made the error @plamut was referring to go away, but I still couldn't print out the estimated bytes without it clearing. I think the next step is to look at modifying those functions in QueryJob like you said

[shubha-rajan]

6efa408 allows the cell to output the total bytes processed. adding a variable argument allows the user to inspect values the returned QueryJob's properties including total_bytes_processed and total_bytes_billed

[plamut]

The new changes look good IMO.

I still have two questions, though:

• If destination_var is used and an error occurs, should the failed query job still be stored for introspection? Currently it isn't:

  In [179]: %%bigquery result --dry_run 
      ...: SELECT SELECT * FROM `bigquery-public-data.samples.shakespeare` LIMIT 5; 
      ...:  
      ...:  
      ...:                                                                                                                                                                                                                                                                                                                                                                                                                         
  
  ERROR:
  400 POST https://www.googleapis.com/bigquery/v2/projects/precise-truck-742/jobs: Syntax error: Unexpected keyword SELECT at [1:8]
  
  In [180]: "result" in locals()                                                                                                                                                                                                                                                                                                                                                                                                    
  Out[180]: False
  

• If destination_var is not specified and an error occurs, it would IMO still be useful to print a query even on dry runs, especially on syntax errors. Currently this information is omitted in dry runs, is this intentional?

  In [188]: %%bigquery --dry_run 
      ...: SELECT SELECT * FROM `bigquery-public-data.samples.shakespeare` LIMIT 5; 
      ...:  
      ...:  
      ...:                                                                                                                                                                                                                                                                                                                                                                                                                         
  
  ERROR:
  400 POST https://www.googleapis.com/bigquery/v2/projects/precise-truck-742/jobs: Syntax error: Unexpected keyword SELECT at [1:8]
  

  Without the --dry-run option, the query gets printed as a part of the error message.

[shubha-rajan]

So currently, destination_var doesn't return anything if an error occurs by default, even when the --dry_run flag isn't present. I investigated adding in that functionality, but the problem is that if calling _run_query raises an exception, it won't return a QueryJob. I suspect getting around that would mean making changes to exception handling in QueryJob and/or Client and idk what might end up breaking because of that 😅

As for printing out the SQL queries along with error messages, that's working as of the last commit.

[plamut]

So currently, destination_var doesn't return anything if an error occurs by default, even when the --dry_run flag isn't present.

Fine then, the --dry_run option does not have to deal with that, either. 👍

The error output is now much more useful, looks good from my side.

[tswast]

Looking great! Love the new error message behavior.

FYI: I've filed #9091 for consideration of how to handle failures with destination_var.