feat: Push to Redshift batch source offline store directly #2819

kevjumba · 2022-06-21T00:33:07Z

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

felixwang9817

looks great overall!

sdk/python/feast/feature_store.py

sdk/python/feast/infra/offline_stores/redshift.py

sdk/python/feast/infra/utils/aws_utils.py

sdk/python/tests/conftest.py

sdk/python/tests/integration/offline_store/test_offline_write.py

codecov-commenter · 2022-06-21T15:57:11Z

Codecov Report

Merging #2819 (9f53598) into master (17303d3) will decrease coverage by 0.04%.
The diff coverage is 81.66%.

@@            Coverage Diff             @@
##           master    #2819      +/-   ##
==========================================
- Coverage   79.63%   79.59%   -0.05%     
==========================================
  Files         174      174              
  Lines       15492    15534      +42     
==========================================
+ Hits        12337    12364      +27     
- Misses       3155     3170      +15

Flag	Coverage Δ
integrationtests	`69.55% <81.66%> (-0.09%)`	⬇️
unittests	`59.55% <48.33%> (-0.08%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdk/python/feast/infra/utils/aws_utils.py	`71.34% <37.50%> (-1.74%)`	⬇️
sdk/python/tests/conftest.py	`77.77% <83.33%> (+0.07%)`	⬆️
sdk/python/feast/infra/offline_stores/redshift.py	`92.34% <87.50%> (-0.63%)`	⬇️
sdk/python/feast/feature_store.py	`86.06% <100.00%> (+0.01%)`	⬆️
sdk/python/feast/infra/passthrough_provider.py	`100.00% <100.00%> (ø)`
...ts/integration/feature_repos/repo_configuration.py	`85.63% <100.00%> (+0.08%)`	⬆️
...ts/integration/offline_store/test_offline_write.py	`100.00% <100.00%> (ø)`
sdk/python/tests/utils/online_read_write_test.py	`93.54% <0.00%> (-6.46%)`	⬇️
.../integration/online_store/test_online_retrieval.py	`96.84% <0.00%> (-3.16%)`	⬇️
sdk/python/feast/infra/online_stores/datastore.py	`87.96% <0.00%> (-1.39%)`	⬇️
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 17303d3...9f53598. Read the comment docs.

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

felixwang9817 · 2022-06-22T18:18:35Z

sdk/python/feast/infra/passthrough_provider.py

        data: pa.Table,
        progress: Optional[Callable[[int], Any]],
    ) -> None:
        set_usage_attribute("provider", self.__class__.__name__)

        if self.offline_store:
-            self.offline_store.offline_write_batch(config, table, data, progress)
+            self.offline_store.__class__.offline_write_batch(


is this necessary?

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

felixwang9817

/lgtm

feast-ci-bot · 2022-06-22T18:33:08Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: felixwang9817, kevjumba

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [felixwang9817,kevjumba]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

# [0.22.0](v0.21.0...v0.22.0) (2022-06-23) ### Bug Fixes * Add columns for user metadata in the tables ([#2760](#2760)) ([269055e](269055e)) * Add project columns in the SQL Registry ([#2784](#2784)) ([336fdd1](336fdd1)) * Add S3FS dependency (which Dask depends on for S3 files) ([#2701](#2701)) ([5d6fa94](5d6fa94)) * Bugfixes for how registry is loaded ([#2768](#2768)) ([ecb8b2a](ecb8b2a)) * Conversion of null timestamp from proto to python ([#2814](#2814)) ([cb23648](cb23648)) * Correct feature statuses during feature logging test ([#2709](#2709)) ([cebf609](cebf609)) * Dynamodb drops missing entities when batching ([#2802](#2802)) ([a2e9209](a2e9209)) * Enable faulthandler and disable flaky tests ([#2815](#2815)) ([4934d84](4934d84)) * Explicitly translate errors when instantiating the go fs ([#2842](#2842)) ([7a2c4cd](7a2c4cd)) * Fix broken roadmap links ([#2690](#2690)) ([b3ba8aa](b3ba8aa)) * Fix bugs in applying stream feature view and retrieving online features ([#2754](#2754)) ([d024e5e](d024e5e)) * Fix Feast UI failure with new way of specifying entities ([#2773](#2773)) ([0d1ac01](0d1ac01)) * Fix feature view __getitem__ for feature services ([#2769](#2769)) ([88cc47d](88cc47d)) * Fix issue when user specifies a port for feast ui ([#2692](#2692)) ([1c621fe](1c621fe)) * Fix on demand feature view crash from inference when it uses df.apply ([#2713](#2713)) ([c5539fd](c5539fd)) * Fix SparkKafkaProcessor `query_timeout` parameter ([#2789](#2789)) ([a8d282d](a8d282d)) * Fixed custom S3 endpoint read fail ([#2786](#2786)) ([6fec431](6fec431)) * Hydrate infra object in the sql registry proto() method ([#2782](#2782)) ([452dcd3](452dcd3)) * Implement apply_materialization and infra methods in sql registry ([#2775](#2775)) ([4ed107c](4ed107c)) * Minor refactor to format exception message ([#2764](#2764)) ([da763c6](da763c6)) * Prefer installing gopy from feast's fork as opposed to upstream ([#2839](#2839)) ([34c997d](34c997d)) * Python server is not correctly starting in integration tests ([#2706](#2706)) ([7583a0b](7583a0b)) * Random port allocation for python server in tests ([#2710](#2710)) ([dee8090](dee8090)) * Refactor test to reuse LocalRegistryFile ([#2763](#2763)) ([4339c0a](4339c0a)) * Support push sources in stream feature views ([#2704](#2704)) ([0d60eaa](0d60eaa)) * Update roadmap with stream feature view rfc ([#2824](#2824)) ([fc8f890](fc8f890)) * Update udf tests and add base functions to streaming fcos and fix some nonetype errors ([#2776](#2776)) ([331a214](331a214)) ### Features * Add file write_to_offline_store functionality ([#2808](#2808)) ([c0e2ad7](c0e2ad7)) * Add http endpoint to the Go feature server ([#2658](#2658)) ([3347a57](3347a57)) * Add StreamProcessor and SparkKafkaProcessor as contrib ([#2777](#2777)) ([83ab682](83ab682)) * Added Spark support for Delta and Avro ([#2757](#2757)) ([7d16516](7d16516)) * CLI interface for validation of logged features ([#2718](#2718)) ([c8b11b3](c8b11b3)) * Enable stream feature view materialization ([#2798](#2798)) ([a06700d](a06700d)) * Enable stream feature view materialization ([#2807](#2807)) ([7d57724](7d57724)) * Implement `offline_write_batch` for BigQuery and Snowflake ([#2840](#2840)) ([97444e4](97444e4)) * Offline push endpoint for pushing to offline stores ([#2837](#2837)) ([a88cd30](a88cd30)) * Push to Redshift batch source offline store directly ([#2819](#2819)) ([5748a8b](5748a8b)) * Scaffold for unified push api ([#2796](#2796)) ([1bd0930](1bd0930)) * SQLAlchemy Registry Support ([#2734](#2734)) ([b3fe39c](b3fe39c)) * Stream Feature View FCOS ([#2750](#2750)) ([0cf3c92](0cf3c92)) * Update stream fcos to have watermark and sliding interval ([#2765](#2765)) ([3256952](3256952)) * Validating logged features via Python SDK ([#2640](#2640)) ([2874fc5](2874fc5))

# [0.22.0](v0.21.0...v0.22.0) (2022-06-24) ### Bug Fixes * Add columns for user metadata in the tables ([#2760](#2760)) ([269055e](269055e)) * Add project columns in the SQL Registry ([#2784](#2784)) ([336fdd1](336fdd1)) * Add S3FS dependency (which Dask depends on for S3 files) ([#2701](#2701)) ([5d6fa94](5d6fa94)) * Bugfixes for how registry is loaded ([#2768](#2768)) ([ecb8b2a](ecb8b2a)) * Conversion of null timestamp from proto to python ([#2814](#2814)) ([cb23648](cb23648)) * Correct feature statuses during feature logging test ([#2709](#2709)) ([cebf609](cebf609)) * Dynamodb drops missing entities when batching ([#2802](#2802)) ([a2e9209](a2e9209)) * Enable faulthandler and disable flaky tests ([#2815](#2815)) ([4934d84](4934d84)) * Explicitly translate errors when instantiating the go fs ([#2842](#2842)) ([7a2c4cd](7a2c4cd)) * Fix broken roadmap links ([#2690](#2690)) ([b3ba8aa](b3ba8aa)) * Fix bugs in applying stream feature view and retrieving online features ([#2754](#2754)) ([d024e5e](d024e5e)) * Fix Feast UI failure with new way of specifying entities ([#2773](#2773)) ([0d1ac01](0d1ac01)) * Fix feature view __getitem__ for feature services ([#2769](#2769)) ([88cc47d](88cc47d)) * Fix issue when user specifies a port for feast ui ([#2692](#2692)) ([1c621fe](1c621fe)) * Fix on demand feature view crash from inference when it uses df.apply ([#2713](#2713)) ([c5539fd](c5539fd)) * Fix SparkKafkaProcessor `query_timeout` parameter ([#2789](#2789)) ([a8d282d](a8d282d)) * Fixed custom S3 endpoint read fail ([#2786](#2786)) ([6fec431](6fec431)) * Hydrate infra object in the sql registry proto() method ([#2782](#2782)) ([452dcd3](452dcd3)) * Implement apply_materialization and infra methods in sql registry ([#2775](#2775)) ([4ed107c](4ed107c)) * Minor refactor to format exception message ([#2764](#2764)) ([da763c6](da763c6)) * Prefer installing gopy from feast's fork as opposed to upstream ([#2839](#2839)) ([34c997d](34c997d)) * Python server is not correctly starting in integration tests ([#2706](#2706)) ([7583a0b](7583a0b)) * Random port allocation for python server in tests ([#2710](#2710)) ([dee8090](dee8090)) * Refactor test to reuse LocalRegistryFile ([#2763](#2763)) ([4339c0a](4339c0a)) * Revert "chore(release): release 0.22.0" ([#2852](#2852)) ([e6a4636](e6a4636)) * Support push sources in stream feature views ([#2704](#2704)) ([0d60eaa](0d60eaa)) * Update roadmap with stream feature view rfc ([#2824](#2824)) ([fc8f890](fc8f890)) * Update udf tests and add base functions to streaming fcos and fix some nonetype errors ([#2776](#2776)) ([331a214](331a214)) ### Features * Add file write_to_offline_store functionality ([#2808](#2808)) ([c0e2ad7](c0e2ad7)) * Add http endpoint to the Go feature server ([#2658](#2658)) ([3347a57](3347a57)) * Add StreamProcessor and SparkKafkaProcessor as contrib ([#2777](#2777)) ([83ab682](83ab682)) * Added Spark support for Delta and Avro ([#2757](#2757)) ([7d16516](7d16516)) * CLI interface for validation of logged features ([#2718](#2718)) ([c8b11b3](c8b11b3)) * Enable stream feature view materialization ([#2798](#2798)) ([a06700d](a06700d)) * Enable stream feature view materialization ([#2807](#2807)) ([7d57724](7d57724)) * Implement `offline_write_batch` for BigQuery and Snowflake ([#2840](#2840)) ([97444e4](97444e4)) * Offline push endpoint for pushing to offline stores ([#2837](#2837)) ([a88cd30](a88cd30)) * Push to Redshift batch source offline store directly ([#2819](#2819)) ([5748a8b](5748a8b)) * Scaffold for unified push api ([#2796](#2796)) ([1bd0930](1bd0930)) * SQLAlchemy Registry Support ([#2734](#2734)) ([b3fe39c](b3fe39c)) * Stream Feature View FCOS ([#2750](#2750)) ([0cf3c92](0cf3c92)) * Update stream fcos to have watermark and sliding interval ([#2765](#2765)) ([3256952](3256952)) * Validating logged features via Python SDK ([#2640](#2640)) ([2874fc5](2874fc5))

# [0.22.0](v0.21.0...v0.22.0) (2022-06-24) ### Bug Fixes * Add columns for user metadata in the tables ([#2760](#2760)) ([269055e](269055e)) * Add project columns in the SQL Registry ([#2784](#2784)) ([336fdd1](336fdd1)) * Add S3FS dependency (which Dask depends on for S3 files) ([#2701](#2701)) ([5d6fa94](5d6fa94)) * Bugfixes for how registry is loaded ([#2768](#2768)) ([ecb8b2a](ecb8b2a)) * Conversion of null timestamp from proto to python ([#2814](#2814)) ([cb23648](cb23648)) * Correct feature statuses during feature logging test ([#2709](#2709)) ([cebf609](cebf609)) * Correctly generate projects-list.json when calling feast ui and using postgres as a source ([#2845](#2845)) ([bee8076](bee8076)) * Dynamodb drops missing entities when batching ([#2802](#2802)) ([a2e9209](a2e9209)) * Enable faulthandler and disable flaky tests ([#2815](#2815)) ([4934d84](4934d84)) * Explicitly translate errors when instantiating the go fs ([#2842](#2842)) ([7a2c4cd](7a2c4cd)) * Fix broken roadmap links ([#2690](#2690)) ([b3ba8aa](b3ba8aa)) * Fix bugs in applying stream feature view and retrieving online features ([#2754](#2754)) ([d024e5e](d024e5e)) * Fix Feast UI failure with new way of specifying entities ([#2773](#2773)) ([0d1ac01](0d1ac01)) * Fix feature view __getitem__ for feature services ([#2769](#2769)) ([88cc47d](88cc47d)) * Fix issue when user specifies a port for feast ui ([#2692](#2692)) ([1c621fe](1c621fe)) * Fix on demand feature view crash from inference when it uses df.apply ([#2713](#2713)) ([c5539fd](c5539fd)) * Fix SparkKafkaProcessor `query_timeout` parameter ([#2789](#2789)) ([a8d282d](a8d282d)) * Fixed custom S3 endpoint read fail ([#2786](#2786)) ([6fec431](6fec431)) * Hydrate infra object in the sql registry proto() method ([#2782](#2782)) ([452dcd3](452dcd3)) * Implement apply_materialization and infra methods in sql registry ([#2775](#2775)) ([4ed107c](4ed107c)) * Minor refactor to format exception message ([#2764](#2764)) ([da763c6](da763c6)) * Prefer installing gopy from feast's fork as opposed to upstream ([#2839](#2839)) ([34c997d](34c997d)) * Python server is not correctly starting in integration tests ([#2706](#2706)) ([7583a0b](7583a0b)) * Random port allocation for python server in tests ([#2710](#2710)) ([dee8090](dee8090)) * Refactor test to reuse LocalRegistryFile ([#2763](#2763)) ([4339c0a](4339c0a)) * Revert "chore(release): release 0.22.0" ([#2852](#2852)) ([e6a4636](e6a4636)) * Support push sources in stream feature views ([#2704](#2704)) ([0d60eaa](0d60eaa)) * Update roadmap with stream feature view rfc ([#2824](#2824)) ([fc8f890](fc8f890)) * Update udf tests and add base functions to streaming fcos and fix some nonetype errors ([#2776](#2776)) ([331a214](331a214)) ### Features * Add feast repo-upgrade for automated repo upgrades ([#2733](#2733)) ([a3304d4](a3304d4)) * Add file write_to_offline_store functionality ([#2808](#2808)) ([c0e2ad7](c0e2ad7)) * Add http endpoint to the Go feature server ([#2658](#2658)) ([3347a57](3347a57)) * Add StreamProcessor and SparkKafkaProcessor as contrib ([#2777](#2777)) ([83ab682](83ab682)) * Added Spark support for Delta and Avro ([#2757](#2757)) ([7d16516](7d16516)) * CLI interface for validation of logged features ([#2718](#2718)) ([c8b11b3](c8b11b3)) * Enable stream feature view materialization ([#2798](#2798)) ([a06700d](a06700d)) * Enable stream feature view materialization ([#2807](#2807)) ([7d57724](7d57724)) * Implement `offline_write_batch` for BigQuery and Snowflake ([#2840](#2840)) ([97444e4](97444e4)) * Offline push endpoint for pushing to offline stores ([#2837](#2837)) ([a88cd30](a88cd30)) * Push to Redshift batch source offline store directly ([#2819](#2819)) ([5748a8b](5748a8b)) * Scaffold for unified push api ([#2796](#2796)) ([1bd0930](1bd0930)) * SQLAlchemy Registry Support ([#2734](#2734)) ([b3fe39c](b3fe39c)) * Stream Feature View FCOS ([#2750](#2750)) ([0cf3c92](0cf3c92)) * Update stream fcos to have watermark and sliding interval ([#2765](#2765)) ([3256952](3256952)) * Validating logged features via Python SDK ([#2640](#2640)) ([2874fc5](2874fc5))

# [0.22.0](v0.21.0...v0.22.0) (2022-06-25) ### Bug Fixes * Add columns for user metadata in the tables ([#2760](#2760)) ([269055e](269055e)) * Add project columns in the SQL Registry ([#2784](#2784)) ([336fdd1](336fdd1)) * Add S3FS dependency (which Dask depends on for S3 files) ([#2701](#2701)) ([5d6fa94](5d6fa94)) * Bugfixes for how registry is loaded ([#2768](#2768)) ([ecb8b2a](ecb8b2a)) * Conversion of null timestamp from proto to python ([#2814](#2814)) ([cb23648](cb23648)) * Correct feature statuses during feature logging test ([#2709](#2709)) ([cebf609](cebf609)) * Correctly generate projects-list.json when calling feast ui and using postgres as a source ([#2845](#2845)) ([bee8076](bee8076)) * Dynamodb drops missing entities when batching ([#2802](#2802)) ([a2e9209](a2e9209)) * Enable faulthandler and disable flaky tests ([#2815](#2815)) ([4934d84](4934d84)) * Explicitly translate errors when instantiating the go fs ([#2842](#2842)) ([7a2c4cd](7a2c4cd)) * Fix broken roadmap links ([#2690](#2690)) ([b3ba8aa](b3ba8aa)) * Fix bugs in applying stream feature view and retrieving online features ([#2754](#2754)) ([d024e5e](d024e5e)) * Fix Feast UI failure with new way of specifying entities ([#2773](#2773)) ([0d1ac01](0d1ac01)) * Fix feature view __getitem__ for feature services ([#2769](#2769)) ([88cc47d](88cc47d)) * Fix issue when user specifies a port for feast ui ([#2692](#2692)) ([1c621fe](1c621fe)) * Fix on demand feature view crash from inference when it uses df.apply ([#2713](#2713)) ([c5539fd](c5539fd)) * Fix SparkKafkaProcessor `query_timeout` parameter ([#2789](#2789)) ([a8d282d](a8d282d)) * Fixed custom S3 endpoint read fail ([#2786](#2786)) ([6fec431](6fec431)) * Hydrate infra object in the sql registry proto() method ([#2782](#2782)) ([452dcd3](452dcd3)) * Implement apply_materialization and infra methods in sql registry ([#2775](#2775)) ([4ed107c](4ed107c)) * Minor refactor to format exception message ([#2764](#2764)) ([da763c6](da763c6)) * Prefer installing gopy from feast's fork as opposed to upstream ([#2839](#2839)) ([34c997d](34c997d)) * Python server is not correctly starting in integration tests ([#2706](#2706)) ([7583a0b](7583a0b)) * Random port allocation for python server in tests ([#2710](#2710)) ([dee8090](dee8090)) * Refactor test to reuse LocalRegistryFile ([#2763](#2763)) ([4339c0a](4339c0a)) * Revert "chore(release): release 0.22.0" ([#2852](#2852)) ([e6a4636](e6a4636)) * Support push sources in stream feature views ([#2704](#2704)) ([0d60eaa](0d60eaa)) * Update roadmap with stream feature view rfc ([#2824](#2824)) ([fc8f890](fc8f890)) * Update udf tests and add base functions to streaming fcos and fix some nonetype errors ([#2776](#2776)) ([331a214](331a214)) ### Features * Add feast repo-upgrade for automated repo upgrades ([#2733](#2733)) ([a3304d4](a3304d4)) * Add file write_to_offline_store functionality ([#2808](#2808)) ([c0e2ad7](c0e2ad7)) * Add http endpoint to the Go feature server ([#2658](#2658)) ([3347a57](3347a57)) * Add StreamProcessor and SparkKafkaProcessor as contrib ([#2777](#2777)) ([83ab682](83ab682)) * Added Spark support for Delta and Avro ([#2757](#2757)) ([7d16516](7d16516)) * CLI interface for validation of logged features ([#2718](#2718)) ([c8b11b3](c8b11b3)) * Enable stream feature view materialization ([#2798](#2798)) ([a06700d](a06700d)) * Enable stream feature view materialization ([#2807](#2807)) ([7d57724](7d57724)) * Implement `offline_write_batch` for BigQuery and Snowflake ([#2840](#2840)) ([97444e4](97444e4)) * Offline push endpoint for pushing to offline stores ([#2837](#2837)) ([a88cd30](a88cd30)) * Push to Redshift batch source offline store directly ([#2819](#2819)) ([5748a8b](5748a8b)) * Scaffold for unified push api ([#2796](#2796)) ([1bd0930](1bd0930)) * SQLAlchemy Registry Support ([#2734](#2734)) ([b3fe39c](b3fe39c)) * Stream Feature View FCOS ([#2750](#2750)) ([0cf3c92](0cf3c92)) * Update stream fcos to have watermark and sliding interval ([#2765](#2765)) ([3256952](3256952)) * Validating logged features via Python SDK ([#2640](#2640)) ([2874fc5](2874fc5))

# [0.22.0](v0.21.0...v0.22.0) (2022-06-28) ### Bug Fixes * Add columns for user metadata in the tables ([#2760](#2760)) ([269055e](269055e)) * Add project columns in the SQL Registry ([#2784](#2784)) ([336fdd1](336fdd1)) * Add S3FS dependency (which Dask depends on for S3 files) ([#2701](#2701)) ([5d6fa94](5d6fa94)) * Bugfixes for how registry is loaded ([#2768](#2768)) ([ecb8b2a](ecb8b2a)) * Conversion of null timestamp from proto to python ([#2814](#2814)) ([cb23648](cb23648)) * Correct feature statuses during feature logging test ([#2709](#2709)) ([cebf609](cebf609)) * Correctly generate projects-list.json when calling feast ui and using postgres as a source ([#2845](#2845)) ([bee8076](bee8076)) * Dynamodb drops missing entities when batching ([#2802](#2802)) ([a2e9209](a2e9209)) * Enable faulthandler and disable flaky tests ([#2815](#2815)) ([4934d84](4934d84)) * Explicitly translate errors when instantiating the go fs ([#2842](#2842)) ([7a2c4cd](7a2c4cd)) * Fix broken roadmap links ([#2690](#2690)) ([b3ba8aa](b3ba8aa)) * Fix bugs in applying stream feature view and retrieving online features ([#2754](#2754)) ([d024e5e](d024e5e)) * Fix Feast UI failure with new way of specifying entities ([#2773](#2773)) ([0d1ac01](0d1ac01)) * Fix feature view __getitem__ for feature services ([#2769](#2769)) ([88cc47d](88cc47d)) * Fix issue when user specifies a port for feast ui ([#2692](#2692)) ([1c621fe](1c621fe)) * Fix on demand feature view crash from inference when it uses df.apply ([#2713](#2713)) ([c5539fd](c5539fd)) * Fix SparkKafkaProcessor `query_timeout` parameter ([#2789](#2789)) ([a8d282d](a8d282d)) * Fixed custom S3 endpoint read fail ([#2786](#2786)) ([6fec431](6fec431)) * Go install gopy instead using go mod tidy ([#2863](#2863)) ([2f2b519](2f2b519)) * Hydrate infra object in the sql registry proto() method ([#2782](#2782)) ([452dcd3](452dcd3)) * Implement apply_materialization and infra methods in sql registry ([#2775](#2775)) ([4ed107c](4ed107c)) * Minor refactor to format exception message ([#2764](#2764)) ([da763c6](da763c6)) * Prefer installing gopy from feast's fork as opposed to upstream ([#2839](#2839)) ([34c997d](34c997d)) * Python server is not correctly starting in integration tests ([#2706](#2706)) ([7583a0b](7583a0b)) * Random port allocation for python server in tests ([#2710](#2710)) ([dee8090](dee8090)) * Refactor test to reuse LocalRegistryFile ([#2763](#2763)) ([4339c0a](4339c0a)) * Revert "chore(release): release 0.22.0" ([#2852](#2852)) ([e6a4636](e6a4636)) * Support push sources in stream feature views ([#2704](#2704)) ([0d60eaa](0d60eaa)) * Update roadmap with stream feature view rfc ([#2824](#2824)) ([fc8f890](fc8f890)) * Update udf tests and add base functions to streaming fcos and fix some nonetype errors ([#2776](#2776)) ([331a214](331a214)) ### Features * Add feast repo-upgrade for automated repo upgrades ([#2733](#2733)) ([a3304d4](a3304d4)) * Add file write_to_offline_store functionality ([#2808](#2808)) ([c0e2ad7](c0e2ad7)) * Add http endpoint to the Go feature server ([#2658](#2658)) ([3347a57](3347a57)) * Add StreamProcessor and SparkKafkaProcessor as contrib ([#2777](#2777)) ([83ab682](83ab682)) * Added Spark support for Delta and Avro ([#2757](#2757)) ([7d16516](7d16516)) * CLI interface for validation of logged features ([#2718](#2718)) ([c8b11b3](c8b11b3)) * Enable stream feature view materialization ([#2798](#2798)) ([a06700d](a06700d)) * Enable stream feature view materialization ([#2807](#2807)) ([7d57724](7d57724)) * Implement `offline_write_batch` for BigQuery and Snowflake ([#2840](#2840)) ([97444e4](97444e4)) * Offline push endpoint for pushing to offline stores ([#2837](#2837)) ([a88cd30](a88cd30)) * Push to Redshift batch source offline store directly ([#2819](#2819)) ([5748a8b](5748a8b)) * Scaffold for unified push api ([#2796](#2796)) ([1bd0930](1bd0930)) * SQLAlchemy Registry Support ([#2734](#2734)) ([b3fe39c](b3fe39c)) * Stream Feature View FCOS ([#2750](#2750)) ([0cf3c92](0cf3c92)) * Update stream fcos to have watermark and sliding interval ([#2765](#2765)) ([3256952](3256952)) * Validating logged features via Python SDK ([#2640](#2640)) ([2874fc5](2874fc5)) ### Reverts * Revert "Create main.yml" (#2867) ([47922a4](47922a4)), closes [#2867](#2867)

# [0.22.0](v0.21.0...v0.22.0) (2022-06-28) ### Bug Fixes * Add columns for user metadata in the tables ([#2760](#2760)) ([269055e](269055e)) * Add project columns in the SQL Registry ([#2784](#2784)) ([336fdd1](336fdd1)) * Add S3FS dependency (which Dask depends on for S3 files) ([#2701](#2701)) ([5d6fa94](5d6fa94)) * Bugfixes for how registry is loaded ([#2768](#2768)) ([ecb8b2a](ecb8b2a)) * Conversion of null timestamp from proto to python ([#2814](#2814)) ([cb23648](cb23648)) * Correct feature statuses during feature logging test ([#2709](#2709)) ([cebf609](cebf609)) * Correctly generate projects-list.json when calling feast ui and using postgres as a source ([#2845](#2845)) ([bee8076](bee8076)) * Dynamodb drops missing entities when batching ([#2802](#2802)) ([a2e9209](a2e9209)) * Enable faulthandler and disable flaky tests ([#2815](#2815)) ([4934d84](4934d84)) * Explicitly translate errors when instantiating the go fs ([#2842](#2842)) ([7a2c4cd](7a2c4cd)) * Fix broken roadmap links ([#2690](#2690)) ([b3ba8aa](b3ba8aa)) * Fix bugs in applying stream feature view and retrieving online features ([#2754](#2754)) ([d024e5e](d024e5e)) * Fix Feast UI failure with new way of specifying entities ([#2773](#2773)) ([0d1ac01](0d1ac01)) * Fix feature view __getitem__ for feature services ([#2769](#2769)) ([88cc47d](88cc47d)) * Fix issue when user specifies a port for feast ui ([#2692](#2692)) ([1c621fe](1c621fe)) * Fix on demand feature view crash from inference when it uses df.apply ([#2713](#2713)) ([c5539fd](c5539fd)) * Fix SparkKafkaProcessor `query_timeout` parameter ([#2789](#2789)) ([a8d282d](a8d282d)) * Fix workflow syntax error ([#2869](#2869)) ([fae45a1](fae45a1)) * Fixed custom S3 endpoint read fail ([#2786](#2786)) ([6fec431](6fec431)) * Go install gopy instead using go mod tidy ([#2863](#2863)) ([2f2b519](2f2b519)) * Hydrate infra object in the sql registry proto() method ([#2782](#2782)) ([452dcd3](452dcd3)) * Implement apply_materialization and infra methods in sql registry ([#2775](#2775)) ([4ed107c](4ed107c)) * Minor refactor to format exception message ([#2764](#2764)) ([da763c6](da763c6)) * Prefer installing gopy from feast's fork as opposed to upstream ([#2839](#2839)) ([34c997d](34c997d)) * Python server is not correctly starting in integration tests ([#2706](#2706)) ([7583a0b](7583a0b)) * Random port allocation for python server in tests ([#2710](#2710)) ([dee8090](dee8090)) * Refactor test to reuse LocalRegistryFile ([#2763](#2763)) ([4339c0a](4339c0a)) * Revert "chore(release): release 0.22.0" ([#2852](#2852)) ([e6a4636](e6a4636)) * Support push sources in stream feature views ([#2704](#2704)) ([0d60eaa](0d60eaa)) * Sync publish and build_wheels workflow to fix verify wheel error. ([#2871](#2871)) ([b0f050a](b0f050a)) * Update roadmap with stream feature view rfc ([#2824](#2824)) ([fc8f890](fc8f890)) * Update udf tests and add base functions to streaming fcos and fix some nonetype errors ([#2776](#2776)) ([331a214](331a214)) ### Features * Add feast repo-upgrade for automated repo upgrades ([#2733](#2733)) ([a3304d4](a3304d4)) * Add file write_to_offline_store functionality ([#2808](#2808)) ([c0e2ad7](c0e2ad7)) * Add http endpoint to the Go feature server ([#2658](#2658)) ([3347a57](3347a57)) * Add StreamProcessor and SparkKafkaProcessor as contrib ([#2777](#2777)) ([83ab682](83ab682)) * Added Spark support for Delta and Avro ([#2757](#2757)) ([7d16516](7d16516)) * CLI interface for validation of logged features ([#2718](#2718)) ([c8b11b3](c8b11b3)) * Enable stream feature view materialization ([#2798](#2798)) ([a06700d](a06700d)) * Enable stream feature view materialization ([#2807](#2807)) ([7d57724](7d57724)) * Implement `offline_write_batch` for BigQuery and Snowflake ([#2840](#2840)) ([97444e4](97444e4)) * Offline push endpoint for pushing to offline stores ([#2837](#2837)) ([a88cd30](a88cd30)) * Push to Redshift batch source offline store directly ([#2819](#2819)) ([5748a8b](5748a8b)) * Scaffold for unified push api ([#2796](#2796)) ([1bd0930](1bd0930)) * SQLAlchemy Registry Support ([#2734](#2734)) ([b3fe39c](b3fe39c)) * Stream Feature View FCOS ([#2750](#2750)) ([0cf3c92](0cf3c92)) * Update stream fcos to have watermark and sliding interval ([#2765](#2765)) ([3256952](3256952)) * Validating logged features via Python SDK ([#2640](#2640)) ([2874fc5](2874fc5)) ### Reverts * Revert "chore(release): release 0.22.0" (#2870) ([ffb0892](ffb0892)), closes [#2870](#2870) * Revert "Create main.yml" (#2867) ([47922a4](47922a4)), closes [#2867](#2867)

# [0.22.0](v0.21.0...v0.22.0) (2022-06-29) ### Bug Fixes * Add columns for user metadata in the tables ([#2760](#2760)) ([269055e](269055e)) * Add project columns in the SQL Registry ([#2784](#2784)) ([336fdd1](336fdd1)) * Add S3FS dependency (which Dask depends on for S3 files) ([#2701](#2701)) ([5d6fa94](5d6fa94)) * Bugfixes for how registry is loaded ([#2768](#2768)) ([ecb8b2a](ecb8b2a)) * Conversion of null timestamp from proto to python ([#2814](#2814)) ([cb23648](cb23648)) * Correct feature statuses during feature logging test ([#2709](#2709)) ([cebf609](cebf609)) * Correctly generate projects-list.json when calling feast ui and using postgres as a source ([#2845](#2845)) ([bee8076](bee8076)) * Dynamodb drops missing entities when batching ([#2802](#2802)) ([a2e9209](a2e9209)) * Enable faulthandler and disable flaky tests ([#2815](#2815)) ([4934d84](4934d84)) * Explicitly translate errors when instantiating the go fs ([#2842](#2842)) ([7a2c4cd](7a2c4cd)) * Fix broken roadmap links ([#2690](#2690)) ([b3ba8aa](b3ba8aa)) * Fix bugs in applying stream feature view and retrieving online features ([#2754](#2754)) ([d024e5e](d024e5e)) * Fix Feast UI failure with new way of specifying entities ([#2773](#2773)) ([0d1ac01](0d1ac01)) * Fix feature view __getitem__ for feature services ([#2769](#2769)) ([88cc47d](88cc47d)) * Fix issue when user specifies a port for feast ui ([#2692](#2692)) ([1c621fe](1c621fe)) * Fix macos wheel version for 310 and also checkout edited go files ([#2890](#2890)) ([bdf170f](bdf170f)) * Fix on demand feature view crash from inference when it uses df.apply ([#2713](#2713)) ([c5539fd](c5539fd)) * Fix SparkKafkaProcessor `query_timeout` parameter ([#2789](#2789)) ([a8d282d](a8d282d)) * Fix workflow syntax error ([#2869](#2869)) ([fae45a1](fae45a1)) * Fixed custom S3 endpoint read fail ([#2786](#2786)) ([6fec431](6fec431)) * Go install gopy instead using go mod tidy ([#2863](#2863)) ([2f2b519](2f2b519)) * Hydrate infra object in the sql registry proto() method ([#2782](#2782)) ([452dcd3](452dcd3)) * Implement apply_materialization and infra methods in sql registry ([#2775](#2775)) ([4ed107c](4ed107c)) * Minor refactor to format exception message ([#2764](#2764)) ([da763c6](da763c6)) * Prefer installing gopy from feast's fork as opposed to upstream ([#2839](#2839)) ([34c997d](34c997d)) * Python server is not correctly starting in integration tests ([#2706](#2706)) ([7583a0b](7583a0b)) * Random port allocation for python server in tests ([#2710](#2710)) ([dee8090](dee8090)) * Refactor test to reuse LocalRegistryFile ([#2763](#2763)) ([4339c0a](4339c0a)) * Revert "chore(release): release 0.22.0" ([#2852](#2852)) ([e6a4636](e6a4636)) * Stop running go mod tidy in setup.py ([#2877](#2877)) ([676ecbb](676ecbb)), closes [/github.com/pypa/cibuildwheel/issues/189#issuecomment-549933912](https://github.com//github.com/pypa/cibuildwheel/issues/189/issues/issuecomment-549933912) * Support push sources in stream feature views ([#2704](#2704)) ([0d60eaa](0d60eaa)) * Sync publish and build_wheels workflow to fix verify wheel error. ([#2871](#2871)) ([b0f050a](b0f050a)) * Update roadmap with stream feature view rfc ([#2824](#2824)) ([fc8f890](fc8f890)) * Update udf tests and add base functions to streaming fcos and fix some nonetype errors ([#2776](#2776)) ([331a214](331a214)) ### Features * Add feast repo-upgrade for automated repo upgrades ([#2733](#2733)) ([a3304d4](a3304d4)) * Add file write_to_offline_store functionality ([#2808](#2808)) ([c0e2ad7](c0e2ad7)) * Add http endpoint to the Go feature server ([#2658](#2658)) ([3347a57](3347a57)) * Add simple TLS support in Go RedisOnlineStore ([#2860](#2860)) ([521488d](521488d)) * Add StreamProcessor and SparkKafkaProcessor as contrib ([#2777](#2777)) ([83ab682](83ab682)) * Added Spark support for Delta and Avro ([#2757](#2757)) ([7d16516](7d16516)) * CLI interface for validation of logged features ([#2718](#2718)) ([c8b11b3](c8b11b3)) * Enable stream feature view materialization ([#2798](#2798)) ([a06700d](a06700d)) * Enable stream feature view materialization ([#2807](#2807)) ([7d57724](7d57724)) * Implement `offline_write_batch` for BigQuery and Snowflake ([#2840](#2840)) ([97444e4](97444e4)) * Offline push endpoint for pushing to offline stores ([#2837](#2837)) ([a88cd30](a88cd30)) * Push to Redshift batch source offline store directly ([#2819](#2819)) ([5748a8b](5748a8b)) * Scaffold for unified push api ([#2796](#2796)) ([1bd0930](1bd0930)) * SQLAlchemy Registry Support ([#2734](#2734)) ([b3fe39c](b3fe39c)) * Stream Feature View FCOS ([#2750](#2750)) ([0cf3c92](0cf3c92)) * Update stream fcos to have watermark and sliding interval ([#2765](#2765)) ([3256952](3256952)) * Validating logged features via Python SDK ([#2640](#2640)) ([2874fc5](2874fc5)) ### Reverts * Revert "chore(release): release 0.22.0" (#2891) ([e5abf58](e5abf58)), closes [#2891](#2891) * Revert "chore(release): release 0.22.0" (#2870) ([ffb0892](ffb0892)), closes [#2870](#2870) * Revert "Create main.yml" (#2867) ([47922a4](47922a4)), closes [#2867](#2867)

feast-ci-bot added approved size/L labels Jun 21, 2022

kevjumba changed the title ~~feat: push to offline store directly through redshift batch source.~~ feat: Push to redshift batch source offline store directly Jun 21, 2022

kevjumba added the ok-to-test label Jun 21, 2022

kevjumba assigned felixwang9817 Jun 21, 2022

felixwang9817 changed the title ~~feat: Push to redshift batch source offline store directly~~ feat: Push to Redshift batch source offline store directly Jun 21, 2022

felixwang9817 reviewed Jun 21, 2022

View reviewed changes

kevjumba requested a review from felixwang9817 June 21, 2022 15:32

kevjumba force-pushed the redshift branch from c13349f to a987c46 Compare June 21, 2022 18:48

feast-ci-bot added size/XL and removed size/L labels Jun 21, 2022

kevjumba added ok-to-test and removed ok-to-test labels Jun 21, 2022

kevjumba force-pushed the redshift branch from 6033ffc to a312ef7 Compare June 21, 2022 22:52

feast-ci-bot added size/L and removed size/XL labels Jun 21, 2022

kevjumba added 13 commits June 22, 2022 10:50

Skaffolding for offline store push

d3e740b

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

LInt

0b41400

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Fix

b426888

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

File source offline push

5187a19

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Fix

42133bd

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Fix

cd45c2c

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Fix

0a62346

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Fix

e022dc4

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Fix

5e0a699

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Fix

b138673

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Address review comments

f57a129

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Add redshift function

1800276

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Add redshift

b277eb1

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

kevjumba added 9 commits June 22, 2022 10:50

Fix

646cf3d

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Lint

1d72fd1

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

fix

89a0ff0

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

fix

14919d8

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Fix errors

01f80d6

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Fix test

c52b476

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Fix test

ce10921

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Fix test

d0c679c

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

add back in

eea45c9

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

kevjumba force-pushed the redshift branch from a312ef7 to eea45c9 Compare June 22, 2022 17:51

kevjumba added 2 commits June 22, 2022 10:52

Fix

d094834

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

lint

9f53598

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

felixwang9817 reviewed Jun 22, 2022

View reviewed changes

kevjumba added 2 commits June 22, 2022 11:30

Address review comments

80df266

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Fix

aa4794d

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

felixwang9817 approved these changes Jun 22, 2022

View reviewed changes

feast-ci-bot added the lgtm label Jun 22, 2022

feast-ci-bot merged commit 5748a8b into feast-dev:master Jun 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Push to Redshift batch source offline store directly #2819

feat: Push to Redshift batch source offline store directly #2819

kevjumba commented Jun 21, 2022

felixwang9817 left a comment

codecov-commenter commented Jun 21, 2022 •

edited

Loading

felixwang9817 Jun 22, 2022

felixwang9817 left a comment

feast-ci-bot commented Jun 22, 2022

feat: Push to Redshift batch source offline store directly #2819

feat: Push to Redshift batch source offline store directly #2819

Conversation

kevjumba commented Jun 21, 2022

felixwang9817 left a comment

Choose a reason for hiding this comment

codecov-commenter commented Jun 21, 2022 • edited Loading

Codecov Report

felixwang9817 Jun 22, 2022

Choose a reason for hiding this comment

felixwang9817 left a comment

Choose a reason for hiding this comment

feast-ci-bot commented Jun 22, 2022

codecov-commenter commented Jun 21, 2022 •

edited

Loading