Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(bigquery): support filtering by _TABLE_SUFFIX when using a wildcard table name #9375

Merged
merged 1 commit into from
Jun 13, 2024
Merged

feat(bigquery): support filtering by _TABLE_SUFFIX when using a wildcard table name #9375

merged 1 commit into from
Jun 13, 2024

Conversation

cpcloud
Copy link
Member

@cpcloud cpcloud commented Jun 13, 2024

Add support for BigQuery sharded tables and use of _TABLE_SUFFIX pseudo-column for filtering shards. Closes #9371.

@cpcloud cpcloud added this to the 9.1 milestone Jun 13, 2024
@cpcloud cpcloud added feature Features or general enhancements bigquery The BigQuery backend labels Jun 13, 2024
@cpcloud cpcloud modified the milestones: 9.1, 9.2 Jun 13, 2024
@cpcloud
Copy link
Member Author

cpcloud commented Jun 13, 2024

BigQuery is passing:

❯ pytest -m bigquery -n 8 --dist loadgroup -q
bringing up nodes...
..x....x.....ssssssssssssssssssssssx.xxxxxx.....xx.xxxxx..xxxx.xx.....x....x...x...xxxxxx.x.......x..xxxxxxxxxx...xxxxxxx.xxxxxxxxxxxx.xxxxxxxxxxxxxxxx.xxxxxxxxxxxxxxxxxxx...xxxxxxxxxxxxxxxxxxxxxxxxx [ 10%]
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx......x...........................xx.........x...........x...........x...x................................................x.................x..................s.... [ 20%]
...............................s....................s......xxx.....xxx...xxxxx..x.x...x.x..............x.............sxx..........xxxxxxx...x..x.x........xx.x.x........x..x.x...x......x.........x.... [ 30%]
..x...x.......x.........x....x......x.....x..xx.........x..x...x...x......x.x......x......x..xs.................x..s...x.................xx...x..x.xx........s.........x.s......x...xx...x.......x....x [ 41%]
...x........x...x......................x.x..x.x..........x....x...x.x..........x.x....x......x....x......x...x........xx....xxx.....................x...x.................x............x..xx..x.....x.. [ 51%]
.........x......xx....x.......xxx...x................x.x....x..x...........x.x.......x.x.x......xx....x..x................x.......................................x.................................... [ 61%]
.......................xx.............................x..........x................x...x.......x.........xx.x...........x.....x...x..................xx...x....x.x.........x......x...x..........xxx.... [ 71%]
...............x....x..x................x..........x..x................x..................x.x.....x..x.......x.x...xxx....x....................x........x.............................................. [ 82%]
........x...................x.............................................................................................................................................x............................ [ 92%]
.................................................................................................................................................. [100%]
1570 passed, 30 skipped, 337 xfailed in 711.96s (0:11:51)

@cpcloud cpcloud requested a review from jcrist June 13, 2024 12:26
gforsyth
gforsyth approved these changes Jun 13, 2024
View reviewed changes
Copy link
Member

@gforsyth gforsyth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@cpcloud cpcloud modified the milestones: 9.2, 9.1 Jun 13, 2024
@gforsyth gforsyth merged commit 62a25c4 into ibis-project:main Jun 13, 2024
80 checks passed
@cpcloud cpcloud deleted the bigquery-wildcard-table-suffix branch July 17, 2024 11:23
@jcrist jcrist mentioned this pull request Sep 6, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gforsyth gforsyth gforsyth approved these changes

@jcrist jcrist Awaiting requested review from jcrist

Assignees
No one assigned
Labels
bigquery The BigQuery backend feature Features or general enhancements
Projects
None yet
Milestone
9.1
Development

Successfully merging this pull request may close these issues.

Support for sharded BigQuery tables
2 participants
@cpcloud @gforsyth