Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(bigquery): non-nullable schema support for embedded fields in struct #10189

Merged
merged 1 commit into from
Sep 21, 2024

Conversation

ssabdb
Copy link
Contributor

@ssabdb ssabdb commented Sep 20, 2024

Bigquery supports embedded non-nullable checks inside bigquery schemas, allowing any non-nullable fields in the schema to persist down into bigquery.

Description of changes

Override _from_ibis_Struct for bigquery to allow the adding of a non-nullable field constraint

Issues closed

Resolves #10188, at least for bigquery. There might be opportunities to implement this change in other backends as well.

@cpcloud cpcloud added this to the 10.0 milestone Sep 21, 2024
@cpcloud cpcloud added the feature Features or general enhancements label Sep 21, 2024
Copy link
Member

@cpcloud cpcloud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BigQuery tests look good:

…/ibis on  ssabdb/main:main is 📦 v9.5.0 via 🐍 v3.12.5 via ❄️   impure (ibis-3.12.5-env)
❯ pytest -m bigquery -n auto --dist loadgroup --snapshot-update -q
bringing up nodes...
..................sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss...sssssssssssssssssssss.....x................s........x............................x.............xx...x.....x...x..xxxx..xx.xxxxx...xx..x.xx.xxxx..x.x.xxxxx..x.xxxxxx.xx.xxxxxxxxxx.xxxxx.xxxxxxxx..xx.xxxxxxxxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.xx.x.xx.xxxxxxxxxxxxxxxxxxxxx [ 17%]
xxxxxxxxx..xxxxxx...x.x.x.......x......x.......xx.....x...............x............x.......x.....................................................x.............x............x...............x......x...x......xxx.....xx.........x......x...x....xxx..X.x....X..x..x.x.x.....x...x....X..x...xx.x.....xxx.xx........xX.x.x.xx.............x.xxx......x........x.x....x...x........x.xxx.x..x..x.xx..x [ 35%]
..x.....x...xx...x......x....x.................................x........................x............................x.............................................................x..x....x...........x........xxx...x...................x......x.....................x........xx..x.....x.......x..x....x...x....x.....................................x.......x........x................x.......x. [ 52%]
..............x..................x..................x..xx...x.....x...xxx........xx........x.........x.x.x..xx...............x...........x..........................xxxxx...xxx.xxx...x.....x...x..xx.x.....x.........s...x.........xs.....x......s.....x.................................x...........x.......x...xx.....xx.x.....x..x......x........xx..x.x.x.....x....xxx.x.....xx..........xx.x... [ 70%]
......xxx...........x...xxx.......x...xx.................................xx.......x.............x.......x......x.x......x.......x...Xs.x......s.xsx.xs.................................................................................................................................................................................x............................................................. [ 88%]
...........x..............s..........s....s...........x..........................................................x.....x..................................................................................................................s...............x..........                                                                                                                                 [100%]
1719 passed, 132 skipped, 350 xfailed, 5 xpassed in 362.04s (0:06:02)

@cpcloud
Copy link
Member

cpcloud commented Sep 21, 2024

Thanks for the PR!

@cpcloud cpcloud merged commit b16db6a into ibis-project:main Sep 21, 2024
79 checks passed
@ssabdb
Copy link
Contributor Author

ssabdb commented Sep 21, 2024

Thanks for the fast merge!

ncclementi pushed a commit to ncclementi/ibis that referenced this pull request Sep 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bigquery The BigQuery backend feature Features or general enhancements tests Issues or PRs related to tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

(bug/feat?): nested non-nullable datatypes aren't created as non-nullable types
2 participants