Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BigQuery: ability to set default job / table / dataset configs in client class #5183

Closed
tswast opened this issue Apr 11, 2018 · 5 comments · Fixed by #6088
Closed

BigQuery: ability to set default job / table / dataset configs in client class #5183

tswast opened this issue Apr 11, 2018 · 5 comments · Fixed by #6088
Assignees
Labels
api: bigquery Issues related to the BigQuery API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@tswast
Copy link
Contributor

tswast commented Apr 11, 2018

The location is considered a global parameter (by the the BigQuery engineering team and the bq command-line tool), and we are tracking the feature to set a default location at #5148.
There are other non-global properties where it would be useful to set a user-configurable default.

For example:

  • Labels on jobs. It could be useful to set default labels so that folks can track more easily which jobs are getting run by which application.
  • use_legacy_sql on QueryJobConfig. Right now this defaults to standard SQL syntax, but some people might prefer to use legacy SQL for existing queries.
  • The destination_encryption_configuration in jobs and the encryption_configuration in tables. Maybe this one actually makes sense as a global parameter, but it's not considered on in the bq command-line tool.
@tswast tswast added type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. api: bigquery Issues related to the BigQuery API. priority: p2 Moderately-important priority. Fix may not be included in next release. labels Apr 11, 2018
@blainehansen
Copy link
Contributor

I'm trying to put together a pull request for this, to get us to the behavior we're looking for that was mentioned in #6042 .

However, when trying to run nox, I'm getting errors that I wouldn't expect. For example:

$ nox -f bigquery/nox.py -s "unit(py='3.6')"
nox > Running session unit(py='3.6')
nox > Creating virtualenv using python3 in /home/blaine/experiments/google-cloud-python/.nox/unit-py-3-6
nox > Session unit(py='3.6') raised exception AttributeError("'Session' object has no attribute 'interpreter'",)
Traceback (most recent call last):
  File "/home/blaine/.local/lib/python3.6/site-packages/nox/sessions.py", line 297, in execute
    self.func(session)
  File "/home/blaine/.local/lib/python3.6/site-packages/nox/_parametrize.py", line 92, in call_wrapper
    return func(*args, **kwargs)
  File "/home/blaine/experiments/google-cloud-python/bigquery/nox.py", line 76, in unit
    session.interpreter = 'python{}'.format(py)
AttributeError: 'Session' object has no attribute 'interpreter'
nox > Session unit(py='3.6') failed.

This makes sense, since it seems nox sessions don't have an interpreter attribute (nor a virtualenv_dirname for that matter, which is also set in that method). But it's surprising to me that these errors are popping up for me? Am I running in a context I shouldn't be? Why is the nox file setting non-existent attributes?

@tseaver
Copy link
Contributor

tseaver commented Sep 22, 2018

@blainehansen I was just able to run the tests using the command line you provided. Note that we would normally run from inside the bigquery directory). Do you have python3.6 on your $PATH when you start?

@tswast
Copy link
Contributor Author

tswast commented Sep 24, 2018

I think the Nox error may be due to nox (new incompatible package) versus nox-automation package (what google-cloud-python still uses AFAIK)

@blainehansen
Copy link
Contributor

blainehansen commented Sep 24, 2018

⬆️ This sounds right. The error I'm getting is a basic python problem (can't assign to a non-existent attribute) so it suggests a misalignment between the code I'm running in the project and the version of nox I have.

I don't know if this is helpful but:

$ nox --version
2018.9.14

Also:

$ which python3.6
/usr/bin/python3.6
$ echo $PATH
...redacted
:/usr/bin:
...redacted

@blainehansen
Copy link
Contributor

Yep, installing and using nox-automation fixed it :) 🎊 🎊

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the BigQuery API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants