Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: Batch Snowflake materialization queries to obey Snowpark 100 fea… #3406

Merged
merged 1 commit into from
Apr 21, 2023
Merged

fix: Batch Snowflake materialization queries to obey Snowpark 100 fea… #3406

merged 1 commit into from
Apr 21, 2023

Conversation

sfc-gh-madkins
Copy link
Collaborator

…ture limit

Signed-off-by: miles.adkins miles.adkins@snowflake.com

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #3380

@sfc-gh-madkins
Copy link
Collaborator Author

/ok-to-test

@sfc-gh-madkins sfc-gh-madkins enabled auto-merge (squash) December 20, 2022 03:38
@sfc-gh-madkins sfc-gh-madkins force-pushed the snow_engine_scale_fix branch 3 times, most recently from 46289ea to 61523f3 Compare December 20, 2022 21:55
@sfc-gh-madkins sfc-gh-madkins force-pushed the snow_engine_scale_fix branch 3 times, most recently from a8f8d17 to fe1a60b Compare March 1, 2023 22:18
@sfc-gh-madkins
Copy link
Collaborator Author

Ok -- this one should be good to push through

@sfc-gh-madkins
Copy link
Collaborator Author

@adchia good to merge I think

@sfc-gh-madkins sfc-gh-madkins force-pushed the snow_engine_scale_fix branch 3 times, most recently from 57463a1 to 7461459 Compare March 12, 2023 00:05
@sfc-gh-madkins sfc-gh-madkins force-pushed the snow_engine_scale_fix branch 2 times, most recently from 2472a24 to 7cc955f Compare March 12, 2023 22:06
@sfc-gh-madkins
fix: Batch Snowflake materialization queries to obey Snowpark 100 limit
44526f5 
Signed-off-by: Miles Adkins <miles.adkins@snowflake.com>
@sfc-gh-madkins
Copy link
Collaborator Author

@adchia ok take a look at this one ... added pbar for snowflake.online as well

adchia
adchia approved these changes Apr 21, 2023
View reviewed changes
Copy link
Collaborator

@adchia adchia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@feast-ci-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: adchia, sfc-gh-madkins

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [adchia,sfc-gh-madkins]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sfc-gh-madkins sfc-gh-madkins merged commit f9862b5 into feast-dev:master Apr 21, 2023
felixwang9817 pushed a commit that referenced this pull request Apr 21, 2023
@feast-ci-bot
chore(release): release 0.31.0
902f23f 
# [0.31.0](v0.30.0...v0.31.0) (2023-04-21)

### Bug Fixes

* Add Stream Feature Views to helper that collect Feature View names ([#3582](#3582)) ([7854f63](7854f63))
* Add StreamFeatureViewSpec to FeastObjectSpecProto convenience type ([#3550](#3550)) ([3cefd6c](3cefd6c))
* Batch Snowflake materialization queries to obey Snowpark 100 fea… ([#3406](#3406)) ([f9862b5](f9862b5))
* Bytewax materializer security context ([#3573](#3573)) ([6794338](6794338))
* **cI:** Install coreutils in mac github workers for smoke test ([#3563](#3563)) ([e7421c1](e7421c1))
* Fix bug with no SqlRegistryConfig class ([#3586](#3586)) ([6dc1368](6dc1368))
* Fix Snowflake template ([#3584](#3584)) ([6c09c39](6c09c39))
* Make snowflake to remote tables temporary ([#3588](#3588)) ([ad48146](ad48146))
* Remove snowflake source warehouse tech debt ([#3422](#3422)) ([7da0580](7da0580))
* Snowflake remote storage ([#3574](#3574)) ([f8d3890](f8d3890))
* Support param timeout when persisting ([#3593](#3593)) ([01a98f0](01a98f0))
* Use pyarrow in a way that works across versions ([#3562](#3562)) ([1289f3f](1289f3f))
* Wrap the bigquery table name with backtick. ([#3577](#3577)) ([09f0e7e](09f0e7e))

### Features

* Add AWS Redshift Serverless support ([#3595](#3595)) ([58ce148](58ce148))
* Add Hazelcast as an online store ([#3523](#3523)) ([b05d50b](b05d50b))
* Cache Bigtable client ([#3602](#3602)) ([b27472f](b27472f))
* Relax aws extras requirements ([#3585](#3585)) ([7e77382](7e77382))
* Show bigquery datasource table and query on UI ([#3600](#3600)) ([58d63f7](58d63f7))
* Update snowflake offline store job output formats -- added arrow ([#3589](#3589)) ([be3e349](be3e349))
zerafachris pushed a commit to zerafachris/feast that referenced this pull request Mar 5, 2024
@sfc-gh-madkins @zerafachris
fix: Batch Snowflake materialization queries to obey Snowpark 100 fea… (
72f93bd 
feast-dev#3406)

fix: Batch Snowflake materialization queries to obey Snowpark 100 feature limit

Signed-off-by: miles.adkins <miles.adkins@snowflake.com>

Signed-off-by: miles.adkins <miles.adkins@snowflake.com>
Signed-off-by: zerafachris PERSONAL <zerafachris@gmail.com>
zerafachris pushed a commit to zerafachris/feast that referenced this pull request Mar 5, 2024
@feast-ci-bot @zerafachris
chore(release): release 0.31.0
e7cc13a 
# [0.31.0](feast-dev/feast@v0.30.0...v0.31.0) (2023-04-21)

### Bug Fixes

* Add Stream Feature Views to helper that collect Feature View names ([feast-dev#3582](feast-dev#3582)) ([7854f63](feast-dev@7854f63))
* Add StreamFeatureViewSpec to FeastObjectSpecProto convenience type ([feast-dev#3550](feast-dev#3550)) ([3cefd6c](feast-dev@3cefd6c))
* Batch Snowflake materialization queries to obey Snowpark 100 fea… ([feast-dev#3406](feast-dev#3406)) ([f9862b5](feast-dev@f9862b5))
* Bytewax materializer security context ([feast-dev#3573](feast-dev#3573)) ([6794338](feast-dev@6794338))
* **cI:** Install coreutils in mac github workers for smoke test ([feast-dev#3563](feast-dev#3563)) ([e7421c1](feast-dev@e7421c1))
* Fix bug with no SqlRegistryConfig class ([feast-dev#3586](feast-dev#3586)) ([6dc1368](feast-dev@6dc1368))
* Fix Snowflake template ([feast-dev#3584](feast-dev#3584)) ([6c09c39](feast-dev@6c09c39))
* Make snowflake to remote tables temporary ([feast-dev#3588](feast-dev#3588)) ([ad48146](feast-dev@ad48146))
* Remove snowflake source warehouse tech debt ([feast-dev#3422](feast-dev#3422)) ([7da0580](feast-dev@7da0580))
* Snowflake remote storage ([feast-dev#3574](feast-dev#3574)) ([f8d3890](feast-dev@f8d3890))
* Support param timeout when persisting ([feast-dev#3593](feast-dev#3593)) ([01a98f0](feast-dev@01a98f0))
* Use pyarrow in a way that works across versions ([feast-dev#3562](feast-dev#3562)) ([1289f3f](feast-dev@1289f3f))
* Wrap the bigquery table name with backtick. ([feast-dev#3577](feast-dev#3577)) ([09f0e7e](feast-dev@09f0e7e))

### Features

* Add AWS Redshift Serverless support ([feast-dev#3595](feast-dev#3595)) ([58ce148](feast-dev@58ce148))
* Add Hazelcast as an online store ([feast-dev#3523](feast-dev#3523)) ([b05d50b](feast-dev@b05d50b))
* Cache Bigtable client ([feast-dev#3602](feast-dev#3602)) ([b27472f](feast-dev@b27472f))
* Relax aws extras requirements ([feast-dev#3585](feast-dev#3585)) ([7e77382](feast-dev@7e77382))
* Show bigquery datasource table and query on UI ([feast-dev#3600](feast-dev#3600)) ([58d63f7](feast-dev@58d63f7))
* Update snowflake offline store job output formats -- added arrow ([feast-dev#3589](feast-dev#3589)) ([be3e349](feast-dev@be3e349))

Signed-off-by: zerafachris PERSONAL <zerafachris@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@adchia adchia adchia approved these changes

Assignees

@adchia adchia

Labels
approved lgtm ok-to-test size/M
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Snowflake Materialization can only handle 128 features at a time
3 participants
@sfc-gh-madkins @feast-ci-bot @adchia