Flaky BigQuery tests

[kwigley]

🚨 Flaky BigQuery tests alert 🚨

There appears to be flakiness around seed with BigQuery integration tests. I'm unsure of root cause, but we have access to a few knobs to hopefully make these tests less flaky.

• increasing timeout
• mark all BQ tests as flaky, this will add retry logic to them and report when tests are being flaky. This is a pattern we use elsewhere, but we can start using the flaky report to track down root cause. https://github.com/dbt-labs/dbt/blob/0a935855f3ef7382c01164e8058f623f84a61998/test/integration/100_rpc_test/test_rpc.py#L481
• this could be related to running too many jobs in parallel against BQ, we could update the concurrency logic in the GHA workflow to only run one BQ job at a time https://github.com/dbt-labs/dbt/blob/0a935855f3ef7382c01164e8058f623f84a61998/.github/workflows/integration.yml#L136

Example failures:

1. Failure caused by seed for simple run test

• test/integration/022_bigquery_test/test_simple_bigquery_view.py::TestSimpleBigQueryRun::test__bigquery_simple_run
• https://github.com/dbt-labs/dbt/runs/3425029001?check_suite_focus=true#step:12:249

...
Completed with 1 error and 0 warnings:

Runtime Error in seed incremental_overwrite_date_expected (data/incremental_overwrite_date_expected.csv)
  BigQuery Timeout Exceeded
...

2. Failure caused by seed for database override test

• test/integration/040_override_database_test/test_override_database.py::TestProjectModelOverride::test_bigquery_database_override
• https://github.com/dbt-labs/dbt/runs/3425029001?check_suite_focus=true#step:12:249

...
Completed with 1 error and 0 warnings:

403 PUT bigquery.googleapis.com/upload/bigquery/v2/projects/dbt-test-env/jobs?uploadType=resumable&upload_id=ADPycdvFfY8AwbDXNN4llBuQfDjiyOf4c753Zt-oRenhK-Oe-5lR6Qp0CSIUxw9kw_DUnkGb-P3G7uJJ0KZkDsg6fWI: Access Denied: Table dbt-test-env:test16299067873390981321_override_database_040.seed: Permission bigquery.tables.updateData denied on table dbt-test-env:test16299067873390981321_override_database_040.seed (or it may not exist).
...

3. Failure caused by seed for source freshness test

• test/integration/042_sources_test/test_sources.py::TestSourceFreshness::test_bigquery_source_freshness
• https://github.com/dbt-labs/dbt/runs/3425662600#step:12:240

...
Completed with 1 error and 0 warnings:

Runtime Error in seed other_source_table (data/other_source_table.csv)
  BigQuery Timeout Exceeded
...

4. Failure caused by seed for test testing hooks on seeds

• 'test/integration/014_hook_tests/test_model_hooks_bq.py::TestBigqueryPrePostModelHooksOnSeeds::test_hooks_on_seeds_bigquery'
• https://github.com/dbt-labs/dbt/runs/3425997183#step:12:232

...
Completed with 1 error and 0 warnings:

Runtime Error in seed example_seed (data/example_seed.csv)
  BigQuery Timeout Exceeded
...

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.