feat: Updating FeatureViewProjection and OnDemandFeatureView to add batch_source and entities #4530

franciscojavierarceo · 2024-09-17T19:17:54Z

What this PR does / why we need it:

This PR addresses some gaps in the FeatureViewProjection, BaseFeatureView, and OnDemandFeatureview classes. In particular, this PR aims to allow the data source to be included in the BaseFeatureView and all data sources in a FeatureViewProjection. I make the assumption that a single BaseFeatureView can only have one datasource, which is a reasonable assumption.

This structure allows users to express an explicit dependency graph with the BaseFeatureView being ultimately tied to one single, foundational data source which I believe is the right pattern.

Along with storing the datasources, we also store relevant batch_source data in the Projection.

More specificially, this PR update the following:

FeatureViewProjection to include the underlying data sources for OnDemandFeatureView. This allows for much richer lineage when constructing metadata about an OnDemandFeatureViews and its data sources.
OnDemandFeatureView to optionally support entities, which will be required to write an OnDemandFeatureView to the online store
BaseFeatureView to include sources to optionally include bath_source
OnDemandFeatureView to include write_to_online_store, a boolean to be used in a follow-up PR (not used in this PR)

Which issue(s) this PR fixes:

This is a step along the path of solving #4376

Misc

N/A

franciscojavierarceo · 2024-09-18T20:23:58Z

sdk/python/feast/feature_view_projection.py

+                name_alias=None,
+                features=base_feature_view.features,
+                desired_features=[],
+                timestamp_field=base_feature_view.batch_source.created_timestamp_column  # type:ignore[attr-defined]


This is admittedly a hack and I don't like it but this should be refactored in 1.0.0

My overall learning from this is that FeatureViews should be really derived from the same object with the same class parameters and make them optionally instantiated.

This should be described more thoroughly for 1.0.0

Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>

…otobuf Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>

Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>

…ing to put a workaround Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>

Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>

tokoko · 2024-09-22T17:31:09Z

A few questions from me:

I make the assumption that a single BaseFeatureView can only have one datasource, which is a reasonable assumption.

I'm not sure this will hold true actually... BatchFeatureViews or whatever we will call them will need to rely on multiple sources for example.

FeatureViewProjection to include the underlying data sources for OnDemandFeatureView. This allows for much richer lineage when constructing metadata about an OnDemandFeatureViews and its data sources.

Can you elaborate on why we need this? My understanding is that FeatureViewProjection objects hold additional info (request-specific modifications) about a specific FeatureView, what's the point of copying some of those field over here when they can already be accessed from FeatureView objects anyway. Feels like unnecessary duplication to me.

OnDemandFeatureView to optionally support entities, which will be required to write an OnDemandFeatureView to the online store

I sort of understand the rationale behind this, but wouldn't it better for us to try to keep OnDemandFeatureViews entity-agnostic? I guess I have concerns similar to FeatureViewProjection ones above. odfv already has dependency on feature views and their entities as a result. Relisting them here feels redundant and would break a single of version of truth. What is a user applies an odfv whose entities do not match the entities of relevant feature views?

My idea about a odfv caching was not to make them retrievable by any set of entities, but rather by some sort of key that would hold information about all the input fields.

franciscojavierarceo · 2024-09-22T17:59:20Z

ODFVs today can behave the old way by not using the optional forthcoming optional parameter that configures writes. The benefit of storing the data here for batch will be the backfilling and seeing the full lineage of the data sources that's used to compute the ODFV.

In a simple way:

ODFV = f(Data source 1, data source 2, feature view 3)

In practice, combinations like this may occur and we get this flexibility and lineage. Having the write option basically acts as a precomputation option for faster retrieval.

franciscojavierarceo · 2024-09-22T18:01:10Z

I sort of understand the rationale behind this, but wouldn't it better for us to try to keep OnDemandFeatureViews entity-agnostic? I guess I have concerns similar to FeatureViewProjection ones above. odfv already has dependency on feature views and their entities as a result.

In order to write an ODFV that can be stored in the online store for retrieval, you need an entity.

This is also important for backfilling.

Relisting them here feels redundant and would break a single of version of truth. What if a user applies an odfv whose entities do not match the entities of relevant feature views?

Not redundant as things can change at different points in time.

HaoXuAI · 2024-09-22T20:51:46Z

Instead of directly updating the BaseFeatureView with DataSource, is it possible to change OndemandFeatureView in inherit from FeatureView which has the source attribute already?
I'm not saying this is a better idea since all types of concrete feature views have the source attribute. Just in case updating the BaseFeatureView might cause breaking changes

franciscojavierarceo · 2024-09-22T21:23:01Z

It won't be a breaking change as it's optional. I do think we should revisit all of these for free 1.0.0 release and make StreamFeatureView, FeatureView, and OnDemandFeatureView all inherit from BaseFeatureView with a fixed set of parameters and make things much cleaner. They have lots of duplicate code all over the place and it's very unnecessary...but I'd rather do that full cleanup once we have this functionality done. Wdyt?

HaoXuAI

LGTM!

tokoko · 2024-09-23T18:46:28Z

Sorry, couldn't follow up sooner.

Not redundant as things can change at different points in time.

I think that's precisely my point. we are allowing essentially same concepts to be defined twice in unrelated places that can easily diverge from one another.

For example, after this PR what's the difference between entities field of an odfvs vs retrieving the list of entities from underlying feature views:

odfv = store.get_on_demand_feature_view('transformed_conv_rate')

entities_1 = odfv.entities
entities_2 = [store.registry.get_any_feature_view(fvp.name).entities for fvp in odfv.source_feature_view_projections]

Is there any scenario where it makes sense for entities_1 and entities_2 not to be exactly the same? If not, why do we need them defined twice?

P.S. the same applied to the FeatureViewProjection. Those fields can easily be retrieved from the underlying FeatureView imho.

# [0.41.0](v0.40.0...v0.41.0) (2024-10-26) * chore!: Update @elastic/eui and @emotion/react in Feast UI ([#4597](#4597)) ([b9ddbf9](b9ddbf9)) ### Bug Fixes * Add --chdir to test_workflow.py ([#4453](#4453)) ([6b2f026](6b2f026)) * Add feast-operator files to semantic-release script ([#4382](#4382)) ([8eceff2](8eceff2)) * Add feast-operator Makefile to semantic-release script ([#4424](#4424)) ([d18d01d](d18d01d)) * Added Offline Store Arrow client errors handler ([#4524](#4524)) ([7535b40](7535b40)) * Added Online Store REST client errors handler ([#4488](#4488)) ([2118719](2118719)) * Added Permission API docs ([#4485](#4485)) ([2bd03fa](2bd03fa)) * Added support for multiple name patterns to Permissions ([#4633](#4633)) ([f05e928](f05e928)) * Adding protobuf<5 as a required dependency due to snowflake limitations ([#4537](#4537)) ([cecca83](cecca83)) * Avoid the python 3.9+ threadpool cleanup bug ([#4627](#4627)) ([ba05893](ba05893)) * Bigquery dataset create table disposition ([#4649](#4649)) ([58e03d1](58e03d1)) * Changes template file path to relative path ([#4624](#4624)) ([3e313b1](3e313b1)) * Check for snowflake functions when setting up materialization engine ([#4456](#4456)) ([c365b4e](c365b4e)) * Correctly handle list values in _python_value_to_proto_value ([#4608](#4608)) ([c0a1026](c0a1026)) * Default to pandas mode if not specified in ODFV proto in database ([#4420](#4420)) ([d235832](d235832)) * Deleting data from feast_metadata when we delete project ([#4550](#4550)) ([351a2d0](351a2d0)) * Disable active_timer When registry_ttl_sec is 0 ([#4499](#4499)) ([c94f32f](c94f32f)) * Escape special characters in the Postgres password ([#4394](#4394)) ([419ca5e](419ca5e)) * FeastExtrasDependencyImportError when using SparkOfflineStore without S3 ([#4594](#4594)) ([1ba94f7](1ba94f7)) * Fix Feast project name test ([#4685](#4685)) ([9f41fd6](9f41fd6)) * Fix for SQL registry initialization fails [#4543](#4543) ([#4544](#4544)) ([4e2eacc](4e2eacc)) * Fix gitignore issue ([#4674](#4674)) ([2807dfa](2807dfa)) * Fix online pg import ([#4581](#4581)) ([1f17caa](1f17caa)) * Fix the mypy type check issue. ([#4498](#4498)) ([7ecc615](7ecc615)) * Fix vector store config ([#4583](#4583)) ([11c00d4](11c00d4)) * Fixes validator field access for 'project_id' in BigQuery offline Store ([#4509](#4509)) ([9a0398e](9a0398e)) * Fixing failure of protos during ODFV transformations for missing entities ([#4667](#4667)) ([41aaeeb](41aaeeb)) * Fixing the master branch build failure. ([#4563](#4563)) ([0192b2e](0192b2e)) * Hao xu request source timestamp_field ([#4495](#4495)) ([96344b2](96344b2)) * Ignore the type check as both functions calls are not belonging to Feast code. ([#4500](#4500)) ([867f532](867f532)) * Import grpc only for type checking in errors.py ([#4533](#4533)) ([f308572](f308572)) * Initial commit targetting grpc registry server ([#4458](#4458)) ([484240c](484240c)), closes [#4465](#4465) * Links to the RBAC documentation under Concepts and Components ([#4430](#4430)) ([0a48f7b](0a48f7b)) * Locate feature_store.yaml from __file__ ([#4443](#4443)) ([20290ce](20290ce)) * Logger settings for feature servers and updated logger for permission flow ([#4531](#4531)) ([50b8f23](50b8f23)) * Move tslib from devDependencies to dependencies in Feast UI ([#4525](#4525)) ([c5a4d90](c5a4d90)) * Null value compatibility for unit timestamp list value type ([#4378](#4378)) ([8f264b6](8f264b6)) * Patch FAISS online return signature ([#4671](#4671)) ([0d45e95](0d45e95)) * Quickstart documentation changes ([#4618](#4618)) ([7ac0908](7ac0908)) * Refactor auth_client_manager_factory.py in function get_auth_client_m… ([#4505](#4505)) ([def8633](def8633)) * Remote apply using offline store ([#4559](#4559)) ([ac62a32](ac62a32)) * Remove Feast UI TypeScript dependencies from `peerDependencies` and `dependencies` ([#4554](#4554)) ([e781e16](e781e16)) * Remove unnecessary peer dependencies from Feast UI ([#4577](#4577)) ([9ac7f4e](9ac7f4e)) * Removed protobuf as a required dependency ([#4535](#4535)) ([0fb76e9](0fb76e9)) * Removed the k8s dependency from required dependencies ([#4519](#4519)) ([3073ea5](3073ea5)) * Removed usage of pull_request_target as much as possible to prevent security concerns ([#4549](#4549)) ([3198371](3198371)) * Replaced ClusterRoles with local RoleBindings ([#4625](#4625)) ([ca9fb9b](ca9fb9b)) * Retire pytz library ([#4406](#4406)) ([23c6c86](23c6c86)) * Typos related to k8s ([#4442](#4442)) ([dda0088](dda0088)) * Update java testcontainers to use Compose V2 ([#4381](#4381)) ([9a33fce](9a33fce)) * Update min versions for pyarrow and protobuf ([#4646](#4646)) ([c7ddd4b](c7ddd4b)) * Update react-router-dom to 6.3.0 and restrict its version in Feast UI ([#4556](#4556)) ([4293608](4293608)), closes [#3794](#3794) [/github.com/remix-run/react-router/blob/main/CHANGELOG.md#v630](https://github.com//github.com/remix-run/react-router/blob/main/CHANGELOG.md/issues/v630) * Update the base image for feature-server. ([#4576](#4576)) ([0390d8a](0390d8a)) * Update the base image of materilization engine. ([#4580](#4580)) ([f8592d8](f8592d8)) * Updated README link ([#4669](#4669)) ([35fbdc9](35fbdc9)) * Updating the documentation and adding tests for project length ([#4628](#4628)) ([945b0fa](945b0fa)) * Using get_type_hints instead of inspect signature for udf return annotation ([#4391](#4391)) ([3a32e8a](3a32e8a)) * Using repo_config parameter in teardown to allow for feature-store-yaml overrides ([#4413](#4413)) ([0baeeb5](0baeeb5)) * Validating permission to update an existing request on both the new and the old instance ([#4449](#4449)) ([635a01b](635a01b)) ### Features * Add boto3 session based auth for dynamodb online store for cross account access ([#4606](#4606)) ([00eaf74](00eaf74)) * Add cli list/describe for SavedDatasets, StreamFeatureViews, & … ([#4487](#4487)) ([7b250e5](7b250e5)) * Add connection_name field to Snowflake config ([#4600](#4600)) ([10ce2aa](10ce2aa)) * Add health check service to registry server ([#4421](#4421)) ([46655f0](46655f0)) * Add more __repr__ methods ([#4676](#4676)) ([e726c09](e726c09)) * Add registry methods for dealing with all FV types ([#4435](#4435)) ([ac381b2](ac381b2)) * Added Project object to Feast Objects ([#4475](#4475)) ([4a6b663](4a6b663)) * Added support for reading from Reader Endpoints for AWS Aurora use cases ([#4494](#4494)) ([d793c77](d793c77)) * Adding documentation for On Demand Feature Transformations with writes ([#4607](#4607)) ([8e0c1b5](8e0c1b5)) * Adding mode='python' for get_historical_features on ODFVs ([#4653](#4653)) ([c40d539](c40d539)) * Adding registry cache support for get_on_demand_feature_view ([#4572](#4572)) ([354c059](354c059)) * Adding SSL support for online server ([#4677](#4677)) ([80a5b3c](80a5b3c)) * Adding write capability to online store to on demand feature views ([#4585](#4585)) ([ef9e0bb](ef9e0bb)), closes [#4603](#4603) * Allow feast snowflake to read in byte string for private-key authentication ([#4384](#4384)) ([5215a21](5215a21)) * An action to test operator at PR time ([#4635](#4635)) ([14c1000](14c1000)) * Create ADOPTERS.md ([#4410](#4410)) ([721ec74](721ec74)) * Create initial structure of Feast Go Operator ([#4596](#4596)) ([b5ab6c7](b5ab6c7)) * Faiss and In memory store ([#4464](#4464)) ([a1ff129](a1ff129)) * Feast Security Model (aka RBAC) ([#4380](#4380)) ([1771f66](1771f66)), closes [#36](#36) * Instrument Feast using Prometheus and OpenTelemetry ([#4366](#4366)) ([a571e08](a571e08)) * Intra server to server communication ([#4433](#4433)) ([729c874](729c874)) * Publish TypeScript types in Feast UI package ([#4551](#4551)) ([334e5d7](334e5d7)) * Refactoring code to get oidc end points from discovery URL. ([#4429](#4429)) ([896360a](896360a)) * Return entity key in the retrieval document api ([#4511](#4511)) ([5f5caf0](5f5caf0)) * Update roadmap.md ([#4445](#4445)) ([34238d2](34238d2)) * Update sqlite-vec package ([#4389](#4389)) ([b734cb1](b734cb1)) * Updated Feast model Inference Architecture ([#4570](#4570)) ([8cd0dcf](8cd0dcf)) * Updating docs to include model inference guidelines ([#4416](#4416)) ([cebbe04](cebbe04)) * Updating FeatureViewProjection and OnDemandFeatureView to add batch_source and entities ([#4530](#4530)) ([0795496](0795496)) * Upgrade React from 17.0.2 to 18.3.1 in Feast UI ([#4620](#4620)) ([d6f3cb8](d6f3cb8)) ### Performance Improvements * Add init and cleanup of long lived resources ([#4642](#4642)) ([47dc04d](47dc04d)) * Added indexes to sql tables to optimize query execution ([#4538](#4538)) ([9688790](9688790)) * Default to async endpoints, use threadpool for sync ([#4647](#4647)) ([c1f1912](c1f1912)) * Implement dynamo write_batch_async ([#4675](#4675)) ([ba4404c](ba4404c)) * Make /push async ([#4650](#4650)) ([61abf89](61abf89)) * Parallelize read calls by table and batch ([#4619](#4619)) ([043eff1](043eff1)) ### BREAKING CHANGES * Consuming apps that use @elastic/eui should update it to a compatible version. If you use @elastic/eui components that have been renamed or replaced with others, you'll need to update your code accordingly. Signed-off-by: Harri Lehtola <peruukki@hotmail.com> * chore: Update Node version from 17 to 20 in UI unit tests Node 17 is not an LTS (long-term support) version and apparently rejected by the latest versions of Elastic UI: > error @elastic/eui@95.12.0: The engine "node" is incompatible with > this module. Expected version "16.x || 18.x || >=20.x". Got "17.9.1" Let's try with the latest LTS version. Signed-off-by: Harri Lehtola <peruukki@hotmail.com>

# [0.41.0](feast-dev/feast@v0.40.0...v0.41.0) (2024-10-26) * chore!: Update @elastic/eui and @emotion/react in Feast UI ([feast-dev#4597](feast-dev#4597)) ([b9ddbf9](feast-dev@b9ddbf9)) ### Bug Fixes * Add --chdir to test_workflow.py ([feast-dev#4453](feast-dev#4453)) ([6b2f026](feast-dev@6b2f026)) * Add feast-operator files to semantic-release script ([feast-dev#4382](feast-dev#4382)) ([8eceff2](feast-dev@8eceff2)) * Add feast-operator Makefile to semantic-release script ([feast-dev#4424](feast-dev#4424)) ([d18d01d](feast-dev@d18d01d)) * Added Offline Store Arrow client errors handler ([feast-dev#4524](feast-dev#4524)) ([7535b40](feast-dev@7535b40)) * Added Online Store REST client errors handler ([feast-dev#4488](feast-dev#4488)) ([2118719](feast-dev@2118719)) * Added Permission API docs ([feast-dev#4485](feast-dev#4485)) ([2bd03fa](feast-dev@2bd03fa)) * Added support for multiple name patterns to Permissions ([feast-dev#4633](feast-dev#4633)) ([f05e928](feast-dev@f05e928)) * Adding protobuf<5 as a required dependency due to snowflake limitations ([feast-dev#4537](feast-dev#4537)) ([cecca83](feast-dev@cecca83)) * Avoid the python 3.9+ threadpool cleanup bug ([feast-dev#4627](feast-dev#4627)) ([ba05893](feast-dev@ba05893)) * Bigquery dataset create table disposition ([feast-dev#4649](feast-dev#4649)) ([58e03d1](feast-dev@58e03d1)) * Changes template file path to relative path ([feast-dev#4624](feast-dev#4624)) ([3e313b1](feast-dev@3e313b1)) * Check for snowflake functions when setting up materialization engine ([feast-dev#4456](feast-dev#4456)) ([c365b4e](feast-dev@c365b4e)) * Correctly handle list values in _python_value_to_proto_value ([feast-dev#4608](feast-dev#4608)) ([c0a1026](feast-dev@c0a1026)) * Default to pandas mode if not specified in ODFV proto in database ([feast-dev#4420](feast-dev#4420)) ([d235832](feast-dev@d235832)) * Deleting data from feast_metadata when we delete project ([feast-dev#4550](feast-dev#4550)) ([351a2d0](feast-dev@351a2d0)) * Disable active_timer When registry_ttl_sec is 0 ([feast-dev#4499](feast-dev#4499)) ([c94f32f](feast-dev@c94f32f)) * Escape special characters in the Postgres password ([feast-dev#4394](feast-dev#4394)) ([419ca5e](feast-dev@419ca5e)) * FeastExtrasDependencyImportError when using SparkOfflineStore without S3 ([feast-dev#4594](feast-dev#4594)) ([1ba94f7](feast-dev@1ba94f7)) * Fix Feast project name test ([feast-dev#4685](feast-dev#4685)) ([9f41fd6](feast-dev@9f41fd6)) * Fix for SQL registry initialization fails [feast-dev#4543](feast-dev#4543) ([feast-dev#4544](feast-dev#4544)) ([4e2eacc](feast-dev@4e2eacc)) * Fix gitignore issue ([feast-dev#4674](feast-dev#4674)) ([2807dfa](feast-dev@2807dfa)) * Fix online pg import ([feast-dev#4581](feast-dev#4581)) ([1f17caa](feast-dev@1f17caa)) * Fix the mypy type check issue. ([feast-dev#4498](feast-dev#4498)) ([7ecc615](feast-dev@7ecc615)) * Fix vector store config ([feast-dev#4583](feast-dev#4583)) ([11c00d4](feast-dev@11c00d4)) * Fixes validator field access for 'project_id' in BigQuery offline Store ([feast-dev#4509](feast-dev#4509)) ([9a0398e](feast-dev@9a0398e)) * Fixing failure of protos during ODFV transformations for missing entities ([feast-dev#4667](feast-dev#4667)) ([41aaeeb](feast-dev@41aaeeb)) * Fixing the master branch build failure. ([feast-dev#4563](feast-dev#4563)) ([0192b2e](feast-dev@0192b2e)) * Hao xu request source timestamp_field ([feast-dev#4495](feast-dev#4495)) ([96344b2](feast-dev@96344b2)) * Ignore the type check as both functions calls are not belonging to Feast code. ([feast-dev#4500](feast-dev#4500)) ([867f532](feast-dev@867f532)) * Import grpc only for type checking in errors.py ([feast-dev#4533](feast-dev#4533)) ([f308572](feast-dev@f308572)) * Initial commit targetting grpc registry server ([feast-dev#4458](feast-dev#4458)) ([484240c](feast-dev@484240c)), closes [feast-dev#4465](feast-dev#4465) * Links to the RBAC documentation under Concepts and Components ([feast-dev#4430](feast-dev#4430)) ([0a48f7b](feast-dev@0a48f7b)) * Locate feature_store.yaml from __file__ ([feast-dev#4443](feast-dev#4443)) ([20290ce](feast-dev@20290ce)) * Logger settings for feature servers and updated logger for permission flow ([feast-dev#4531](feast-dev#4531)) ([50b8f23](feast-dev@50b8f23)) * Move tslib from devDependencies to dependencies in Feast UI ([feast-dev#4525](feast-dev#4525)) ([c5a4d90](feast-dev@c5a4d90)) * Null value compatibility for unit timestamp list value type ([feast-dev#4378](feast-dev#4378)) ([8f264b6](feast-dev@8f264b6)) * Patch FAISS online return signature ([feast-dev#4671](feast-dev#4671)) ([0d45e95](feast-dev@0d45e95)) * Quickstart documentation changes ([feast-dev#4618](feast-dev#4618)) ([7ac0908](feast-dev@7ac0908)) * Refactor auth_client_manager_factory.py in function get_auth_client_m… ([feast-dev#4505](feast-dev#4505)) ([def8633](feast-dev@def8633)) * Remote apply using offline store ([feast-dev#4559](feast-dev#4559)) ([ac62a32](feast-dev@ac62a32)) * Remove Feast UI TypeScript dependencies from `peerDependencies` and `dependencies` ([feast-dev#4554](feast-dev#4554)) ([e781e16](feast-dev@e781e16)) * Remove unnecessary peer dependencies from Feast UI ([feast-dev#4577](feast-dev#4577)) ([9ac7f4e](feast-dev@9ac7f4e)) * Removed protobuf as a required dependency ([feast-dev#4535](feast-dev#4535)) ([0fb76e9](feast-dev@0fb76e9)) * Removed the k8s dependency from required dependencies ([feast-dev#4519](feast-dev#4519)) ([3073ea5](feast-dev@3073ea5)) * Removed usage of pull_request_target as much as possible to prevent security concerns ([feast-dev#4549](feast-dev#4549)) ([3198371](feast-dev@3198371)) * Replaced ClusterRoles with local RoleBindings ([feast-dev#4625](feast-dev#4625)) ([ca9fb9b](feast-dev@ca9fb9b)) * Retire pytz library ([feast-dev#4406](feast-dev#4406)) ([23c6c86](feast-dev@23c6c86)) * Typos related to k8s ([feast-dev#4442](feast-dev#4442)) ([dda0088](feast-dev@dda0088)) * Update java testcontainers to use Compose V2 ([feast-dev#4381](feast-dev#4381)) ([9a33fce](feast-dev@9a33fce)) * Update min versions for pyarrow and protobuf ([feast-dev#4646](feast-dev#4646)) ([c7ddd4b](feast-dev@c7ddd4b)) * Update react-router-dom to 6.3.0 and restrict its version in Feast UI ([feast-dev#4556](feast-dev#4556)) ([4293608](feast-dev@4293608)), closes [feast-dev#3794](feast-dev#3794) [/github.com/remix-run/react-router/blob/main/CHANGELOG.md#v630](https://github.com//github.com/remix-run/react-router/blob/main/CHANGELOG.md/issues/v630) * Update the base image for feature-server. ([feast-dev#4576](feast-dev#4576)) ([0390d8a](feast-dev@0390d8a)) * Update the base image of materilization engine. ([feast-dev#4580](feast-dev#4580)) ([f8592d8](feast-dev@f8592d8)) * Updated README link ([feast-dev#4669](feast-dev#4669)) ([35fbdc9](feast-dev@35fbdc9)) * Updating the documentation and adding tests for project length ([feast-dev#4628](feast-dev#4628)) ([945b0fa](feast-dev@945b0fa)) * Using get_type_hints instead of inspect signature for udf return annotation ([feast-dev#4391](feast-dev#4391)) ([3a32e8a](feast-dev@3a32e8a)) * Using repo_config parameter in teardown to allow for feature-store-yaml overrides ([feast-dev#4413](feast-dev#4413)) ([0baeeb5](feast-dev@0baeeb5)) * Validating permission to update an existing request on both the new and the old instance ([feast-dev#4449](feast-dev#4449)) ([635a01b](feast-dev@635a01b)) ### Features * Add boto3 session based auth for dynamodb online store for cross account access ([feast-dev#4606](feast-dev#4606)) ([00eaf74](feast-dev@00eaf74)) * Add cli list/describe for SavedDatasets, StreamFeatureViews, & … ([feast-dev#4487](feast-dev#4487)) ([7b250e5](feast-dev@7b250e5)) * Add connection_name field to Snowflake config ([feast-dev#4600](feast-dev#4600)) ([10ce2aa](feast-dev@10ce2aa)) * Add health check service to registry server ([feast-dev#4421](feast-dev#4421)) ([46655f0](feast-dev@46655f0)) * Add more __repr__ methods ([feast-dev#4676](feast-dev#4676)) ([e726c09](feast-dev@e726c09)) * Add registry methods for dealing with all FV types ([feast-dev#4435](feast-dev#4435)) ([ac381b2](feast-dev@ac381b2)) * Added Project object to Feast Objects ([feast-dev#4475](feast-dev#4475)) ([4a6b663](feast-dev@4a6b663)) * Added support for reading from Reader Endpoints for AWS Aurora use cases ([feast-dev#4494](feast-dev#4494)) ([d793c77](feast-dev@d793c77)) * Adding documentation for On Demand Feature Transformations with writes ([feast-dev#4607](feast-dev#4607)) ([8e0c1b5](feast-dev@8e0c1b5)) * Adding mode='python' for get_historical_features on ODFVs ([feast-dev#4653](feast-dev#4653)) ([c40d539](feast-dev@c40d539)) * Adding registry cache support for get_on_demand_feature_view ([feast-dev#4572](feast-dev#4572)) ([354c059](feast-dev@354c059)) * Adding SSL support for online server ([feast-dev#4677](feast-dev#4677)) ([80a5b3c](feast-dev@80a5b3c)) * Adding write capability to online store to on demand feature views ([feast-dev#4585](feast-dev#4585)) ([ef9e0bb](feast-dev@ef9e0bb)), closes [feast-dev#4603](feast-dev#4603) * Allow feast snowflake to read in byte string for private-key authentication ([feast-dev#4384](feast-dev#4384)) ([5215a21](feast-dev@5215a21)) * An action to test operator at PR time ([feast-dev#4635](feast-dev#4635)) ([14c1000](feast-dev@14c1000)) * Create ADOPTERS.md ([feast-dev#4410](feast-dev#4410)) ([721ec74](feast-dev@721ec74)) * Create initial structure of Feast Go Operator ([feast-dev#4596](feast-dev#4596)) ([b5ab6c7](feast-dev@b5ab6c7)) * Faiss and In memory store ([feast-dev#4464](feast-dev#4464)) ([a1ff129](feast-dev@a1ff129)) * Feast Security Model (aka RBAC) ([feast-dev#4380](feast-dev#4380)) ([1771f66](feast-dev@1771f66)), closes [feast-dev#36](feast-dev#36) * Instrument Feast using Prometheus and OpenTelemetry ([feast-dev#4366](feast-dev#4366)) ([a571e08](feast-dev@a571e08)) * Intra server to server communication ([feast-dev#4433](feast-dev#4433)) ([729c874](feast-dev@729c874)) * Publish TypeScript types in Feast UI package ([feast-dev#4551](feast-dev#4551)) ([334e5d7](feast-dev@334e5d7)) * Refactoring code to get oidc end points from discovery URL. ([feast-dev#4429](feast-dev#4429)) ([896360a](feast-dev@896360a)) * Return entity key in the retrieval document api ([feast-dev#4511](feast-dev#4511)) ([5f5caf0](feast-dev@5f5caf0)) * Update roadmap.md ([feast-dev#4445](feast-dev#4445)) ([34238d2](feast-dev@34238d2)) * Update sqlite-vec package ([feast-dev#4389](feast-dev#4389)) ([b734cb1](feast-dev@b734cb1)) * Updated Feast model Inference Architecture ([feast-dev#4570](feast-dev#4570)) ([8cd0dcf](feast-dev@8cd0dcf)) * Updating docs to include model inference guidelines ([feast-dev#4416](feast-dev#4416)) ([cebbe04](feast-dev@cebbe04)) * Updating FeatureViewProjection and OnDemandFeatureView to add batch_source and entities ([feast-dev#4530](feast-dev#4530)) ([0795496](feast-dev@0795496)) * Upgrade React from 17.0.2 to 18.3.1 in Feast UI ([feast-dev#4620](feast-dev#4620)) ([d6f3cb8](feast-dev@d6f3cb8)) ### Performance Improvements * Add init and cleanup of long lived resources ([feast-dev#4642](feast-dev#4642)) ([47dc04d](feast-dev@47dc04d)) * Added indexes to sql tables to optimize query execution ([feast-dev#4538](feast-dev#4538)) ([9688790](feast-dev@9688790)) * Default to async endpoints, use threadpool for sync ([feast-dev#4647](feast-dev#4647)) ([c1f1912](feast-dev@c1f1912)) * Implement dynamo write_batch_async ([feast-dev#4675](feast-dev#4675)) ([ba4404c](feast-dev@ba4404c)) * Make /push async ([feast-dev#4650](feast-dev#4650)) ([61abf89](feast-dev@61abf89)) * Parallelize read calls by table and batch ([feast-dev#4619](feast-dev#4619)) ([043eff1](feast-dev@043eff1)) ### BREAKING CHANGES * Consuming apps that use @elastic/eui should update it to a compatible version. If you use @elastic/eui components that have been renamed or replaced with others, you'll need to update your code accordingly. Signed-off-by: Harri Lehtola <peruukki@hotmail.com> * chore: Update Node version from 17 to 20 in UI unit tests Node 17 is not an LTS (long-term support) version and apparently rejected by the latest versions of Elastic UI: > error @elastic/eui@95.12.0: The engine "node" is incompatible with > this module. Expected version "16.x || 18.x || >=20.x". Got "17.9.1" Let's try with the latest LTS version. Signed-off-by: Harri Lehtola <peruukki@hotmail.com>

franciscojavierarceo commented Sep 18, 2024

View reviewed changes

franciscojavierarceo added the ok-to-test label Sep 18, 2024

franciscojavierarceo changed the title ~~feat: Updating protos for Projections to include more info~~ feat: Updating OnDemandFeatureView to add Entities and batch_source Sep 18, 2024

franciscojavierarceo mentioned this pull request Sep 21, 2024

chore: Adding unit test to test feature view dummy entity serialization after apply() #4553

Merged

franciscojavierarceo changed the title ~~feat: Updating OnDemandFeatureView to add Entities and batch_source~~ feat: Updating FeatureViewProjection and OnDemandFeatureView to add batch_source and entities Sep 21, 2024

franciscojavierarceo added 16 commits September 21, 2024 11:31

feat: Updating protos for Projections to include more info

b791284

Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>

adding unit test

54ca376

Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>

adding type checking where batch source is already serialized into pr…

cf613bd

…otobuf Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>

almost got everything working and type validation behaving

44066d2

Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>

cleaned up and have tests behaving

2837c1c

Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>

removed comment

52d4253

Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>

updated FeatureViewProjection batch_source serialization

34f99b9

Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>

trying to debug a test

fcf2917

Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>

handling snowflake issue, cant confirm why it is happening so just go…

3c7812b

…ing to put a workaround Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>

linter

6c9a21f

Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>

trying to handle it correctly

1103af7

Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>

handling the else case for from_feature_view_definition

987c690

Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>

adding print

c5a7ea3

Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>

adding test of issue

0d489a9

Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>

think i got everything working now

a85854e

Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>

removing print

cf86862

Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>

franciscojavierarceo force-pushed the fv-projections branch from f50923f to cf86862 Compare September 21, 2024 17:56

franciscojavierarceo marked this pull request as ready for review September 21, 2024 18:23

franciscojavierarceo requested review from tokoko, shuchu, HaoXuAI, a team, adchia and felixwang9817 and removed request for a team September 21, 2024 18:23

HaoXuAI approved these changes Sep 23, 2024

View reviewed changes

HaoXuAI merged commit 0795496 into master Sep 23, 2024
17 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Updating FeatureViewProjection and OnDemandFeatureView to add batch_source and entities #4530

feat: Updating FeatureViewProjection and OnDemandFeatureView to add batch_source and entities #4530

franciscojavierarceo commented Sep 17, 2024 •

edited

Loading

franciscojavierarceo Sep 18, 2024

tokoko commented Sep 22, 2024

franciscojavierarceo commented Sep 22, 2024

franciscojavierarceo commented Sep 22, 2024 •

edited

Loading

HaoXuAI commented Sep 22, 2024

franciscojavierarceo commented Sep 22, 2024

HaoXuAI left a comment

tokoko commented Sep 23, 2024 •

edited

Loading

feat: Updating FeatureViewProjection and OnDemandFeatureView to add batch_source and entities #4530

feat: Updating FeatureViewProjection and OnDemandFeatureView to add batch_source and entities #4530

Conversation

franciscojavierarceo commented Sep 17, 2024 • edited Loading

What this PR does / why we need it:

Which issue(s) this PR fixes:

Misc

franciscojavierarceo Sep 18, 2024

Choose a reason for hiding this comment

tokoko commented Sep 22, 2024

franciscojavierarceo commented Sep 22, 2024

franciscojavierarceo commented Sep 22, 2024 • edited Loading

HaoXuAI commented Sep 22, 2024

franciscojavierarceo commented Sep 22, 2024

HaoXuAI left a comment

Choose a reason for hiding this comment

tokoko commented Sep 23, 2024 • edited Loading

franciscojavierarceo commented Sep 17, 2024 •

edited

Loading

franciscojavierarceo commented Sep 22, 2024 •

edited

Loading

tokoko commented Sep 23, 2024 •

edited

Loading