Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BigQuery: Use configurable bucket name for GCS samples data in systems tests. #8783

Merged
merged 3 commits into from
Jul 29, 2019

Conversation

tswast
Copy link
Contributor

@tswast tswast commented Jul 25, 2019

This allows the VPC-SC team to use their private mirror which is within
the security boundary when testing BigQuery VPC-SC support.

Closes #8550.

This allows the VPC-SC team to use their private mirror which is within
the security boundary when testing BigQuery VPC-SC support.
@tswast tswast added testing api: bigquery Issues related to the BigQuery API. type: process A process-related concern. May include testing, release, or the like. labels Jul 25, 2019
@tswast tswast requested a review from a team July 25, 2019 18:32
@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label Jul 25, 2019
@tswast tswast requested a review from shollyman July 25, 2019 18:33
# The VPC-SC team maintains a mirror of the GCS bucket used for code
# samples. The public bucket crosses the configured security boundary.
# See: https://github.com/googleapis/google-cloud-python/issues/8550
SAMPLES_BUCKET = os.environ.get("GCLOUD_TEST_SAMPLES_BUCKET", "cloud-samples-data")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to update noxfile.py to set this value in the environment?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, the default of cloud-samples-data is fine for our purposes. The VPC-SC team has a separate Kokoro (or maybe something similar?) config where they'll set the necessary environment variable.

@tswast
Copy link
Contributor Author

tswast commented Jul 25, 2019

___________________ test_bq_to_arrow_array_w_special_floats ____________________
module_under_test = <module 'google.cloud.bigquery._pandas_helpers' from '/tmpfs/src/github/google-cloud-python/bigquery/google/cloud/bigquery/_pandas_helpers.py'>
    @pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
    @pytest.mark.skipif(pyarrow is None, reason="Requires `pyarrow`")
    def test_bq_to_arrow_array_w_special_floats(module_under_test):
        bq_field = schema.SchemaField("field_name", "FLOAT64")
        rows = [float("-inf"), float("nan"), float("inf"), None]
        series = pandas.Series(rows, dtype="object")
        arrow_array = module_under_test.bq_to_arrow_array(series, bq_field)
        roundtrip = arrow_array.to_pylist()
        assert len(rows) == len(roundtrip)
        assert roundtrip[0] == float("-inf")
>       assert roundtrip[1] != roundtrip[1]  # NaN doesn't equal itself.
E       assert None != None

failure is unrelated. I think it's a behavior change / regression from pyarrow.

@tseaver
Copy link
Contributor

tseaver commented Jul 25, 2019

@tswast How should we track that regression? All BigQuery CI is b0rked ATM.

@tswast
Copy link
Contributor Author

tswast commented Jul 25, 2019

@tswast How should we track that regression? All BigQuery CI is b0rked ATM.

Let's pin to pyarrow==0.13.0 in the noxfile until I can reproduce the regression and file an issue on the Apache Arrow JIRA.

@tswast
Copy link
Contributor Author

tswast commented Jul 25, 2019

Sent #8784.

@tswast tswast merged commit bbe686d into googleapis:master Jul 29, 2019
@tswast tswast deleted the issue8550-bq-vpc-sc-tests branch July 29, 2019 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the BigQuery API. cla: yes This human has signed the Contributor License Agreement. testing type: process A process-related concern. May include testing, release, or the like.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BigQuery: test_create_routine test fails on VPC-SC tests
4 participants