direct data ingestion into Online store #1939

vas28r13 · 2021-10-14T03:11:27Z

Signed-off-by: Vitaly Sergeyev vsergeyev@better.com

Issue docs: #1863

Directly ingest data into the Online store.

Use Case
a model process/service will get a stream of data and would need to keep track of the new/updated features in the Online feature store

Code example

feature_store_object.ingest_df("my_feature_view_name", dataframe_object)

Process 1

# load feature store
feature_store = _get_feature_store()
for feature_data in _kafka_consume():
    # clean the data
    features_df = _clean_features(feature_data)

    # ingest the data directly into the online store
    feature_store.ingest_df("my_feature_view_name", features_df)

Process 2 (materialization job)

feature_store = _get_feature_store()

# materialization from Offline sources
feature_store.materialize_incremental(end_date=datetime.now(timezone.utc))

feast-ci-bot · 2021-10-14T03:11:28Z

@vas28r13: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

feast-ci-bot · 2021-10-14T03:11:37Z

Hi @vas28r13. Thanks for your PR.

I'm waiting for a feast-dev member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

felixwang9817 · 2021-10-14T05:00:11Z

/ok-to-test

felixwang9817 · 2021-10-14T05:00:14Z

/kind feature

codecov-commenter · 2021-10-14T05:10:37Z

Codecov Report

Merging #1939 (161be8d) into master (e8a1519) will decrease coverage by 0.18%.
The diff coverage is 97.05%.

@@            Coverage Diff             @@
##           master    #1939      +/-   ##
==========================================
- Coverage   82.26%   82.08%   -0.19%     
==========================================
  Files          96       99       +3     
  Lines        7669     7958     +289     
==========================================
+ Hits         6309     6532     +223     
- Misses       1360     1426      +66

Flag	Coverage Δ
integrationtests	`74.14% <50.00%> (-0.01%)`	⬇️
unittests	`59.77% <88.23%> (+0.38%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdk/python/feast/infra/provider.py	`89.34% <50.00%> (-0.66%)`	⬇️
sdk/python/feast/feature_store.py	`91.31% <100.00%> (-3.24%)`	⬇️
sdk/python/feast/flags.py	`100.00% <100.00%> (ø)`
sdk/python/feast/flags_helper.py	`90.47% <100.00%> (+1.00%)`	⬆️
sdk/python/feast/infra/passthrough_provider.py	`100.00% <100.00%> (ø)`
...sts/integration/registration/test_feature_store.py	`99.37% <100.00%> (+0.05%)`	⬆️
...ts/integration/feature_repos/repo_configuration.py	`93.18% <0.00%> (-6.82%)`	⬇️
sdk/python/tests/conftest.py	`91.66% <0.00%> (-3.65%)`	⬇️
sdk/python/feast/infra/utils/aws_utils.py	`70.37% <0.00%> (-0.65%)`	⬇️
... and 26 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e8a1519...161be8d. Read the comment docs.

adchia

looks good!

adchia · 2021-10-14T20:50:23Z

sdk/python/feast/feature_store.py

@@ -766,6 +766,19 @@ def tqdm_builder(length):
                feature_view, self.project, start_date, end_date
            )

+    @log_exceptions_and_usage
+    def ingest_df(


mind calling this write_to_online_store instead? ingest can be confusing for folks who think this might also mean writing to the offline store

the name is fine for the provider since i think this naming will work for those once we want to ingest to both offline and online stores. But as an external facing API, should be more clear to start.

also, mind making this an experimental feature?

e.g. like https://github.com/feast-dev/feast/blob/master/sdk/python/feast/feature_store.py#L399

sdk/python/feast/feature_store.py

sdk/python/feast/infra/provider.py

adchia · 2021-10-14T21:50:26Z

sdk/python/feast/feature_store.py

@@ -766,6 +766,19 @@ def tqdm_builder(length):
                feature_view, self.project, start_date, end_date
            )

+    @log_exceptions_and_usage
+    def ingest_df(


also, mind making this an experimental feature?

e.g. like https://github.com/feast-dev/feast/blob/master/sdk/python/feast/feature_store.py#L399

vas28r13 · 2021-10-21T00:37:30Z

/retest

adchia · 2021-10-22T17:49:51Z

sdk/python/tests/integration/registration/test_feature_store.py

@@ -424,6 +426,46 @@ def test_apply_remote_repo():
    )


+@pytest.mark.parametrize(


this is fine for now. mind adding a TODO to move this to test_universal_online? that parametrizes across online stores

adchia

/lgtm

adchia

Leaving a quick request changes to add the TODO i mentioned (even better if you move it to test_universal_online as per https://docs.feast.dev/how-to-guides/adding-or-reusing-tests

felixwang9817 · 2021-10-22T18:53:39Z

Hey @vas28r13, thanks for this PR. We just fixed the issue with the linter that's been blocking all PRs for the last two days. Would you mind rebasing your changes on master and then force pushing your changes up? Thanks!

Signed-off-by: Vitaly Sergeyev <vsergeyev@better.com>

adchia

/lgtm

sdk/python/tests/integration/feature_repos/repo_configuration.py

Signed-off-by: Vitaly Sergeyev <vsergeyev@better.com>

adchia

/lgtm

feast-ci-bot · 2021-10-25T20:12:31Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: adchia, vas28r13

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [adchia]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

benny-hal · 2021-11-16T11:58:55Z

sdk/python/feast/feature_store.py

@@ -809,6 +809,26 @@ def tqdm_builder(length):
                feature_view, self.project, start_date, end_date
            )

+    @log_exceptions_and_usage
+    def write_to_online_store(


@vas28r13 thx for adding this capability, we use this capability so we can write inference data in a real-time flow. However, the get_feature_view does not use the allow_cache option of the registry. This causes a lookup to refresh the registry for every call to write_to_online_store. This blocks the new capability from being used real-time. Would it make sense to add a PR that fixes that? thx...

would you mind making an issue so we can track this?

vas28r13 requested review from achals and adchia as code owners October 14, 2021 03:11

vas28r13 requested review from felixwang9817, tsotnet, woop and a team as code owners October 14, 2021 03:11

feast-ci-bot added do-not-merge/release-note-label-needed needs-kind needs-ok-to-test labels Oct 14, 2021

feast-ci-bot added the size/M label Oct 14, 2021

felixwang9817 assigned achals Oct 14, 2021

feast-ci-bot added ok-to-test kind/feature New feature or request and removed needs-ok-to-test needs-kind labels Oct 14, 2021

adchia reviewed Oct 14, 2021

View reviewed changes

vas28r13 force-pushed the 1863-direct-ingestion branch from 560d4fc to 161be8d Compare October 21, 2021 18:10

adchia reviewed Oct 22, 2021

View reviewed changes

adchia approved these changes Oct 22, 2021

View reviewed changes

feast-ci-bot assigned adchia Oct 22, 2021

feast-ci-bot added lgtm approved labels Oct 22, 2021

adchia requested changes Oct 22, 2021

View reviewed changes

feast-ci-bot removed the approved label Oct 22, 2021

feast-ci-bot removed the lgtm label Oct 25, 2021

Vitaly Sergeyev added 6 commits October 25, 2021 15:11

direct data ingestion into Online store

6f9d0f6

Signed-off-by: Vitaly Sergeyev <vsergeyev@better.com>

feature flags and formatting

f1202bb

Signed-off-by: Vitaly Sergeyev <vsergeyev@better.com>

add test for write_to_online_store

c2eec8b

Signed-off-by: Vitaly Sergeyev <vsergeyev@better.com>

formatting

55c77d5

Signed-off-by: Vitaly Sergeyev <vsergeyev@better.com>

move test to test_universal_online

6f2aca2

Signed-off-by: Vitaly Sergeyev <vsergeyev@better.com>

formatting

9bfa450

Signed-off-by: Vitaly Sergeyev <vsergeyev@better.com>

vas28r13 force-pushed the 1863-direct-ingestion branch from 6106e20 to 9bfa450 Compare October 25, 2021 19:11

remove unused import

5cbfa2a

Signed-off-by: Vitaly Sergeyev <vsergeyev@better.com>

adchia requested changes Oct 25, 2021

View reviewed changes

sdk/python/tests/integration/feature_repos/repo_configuration.py Outdated Show resolved Hide resolved

feast-ci-bot added lgtm approved labels Oct 25, 2021

test flags not needed

6065adf

Signed-off-by: Vitaly Sergeyev <vsergeyev@better.com>

feast-ci-bot removed the lgtm label Oct 25, 2021

adchia approved these changes Oct 25, 2021

View reviewed changes

feast-ci-bot added the lgtm label Oct 25, 2021

feast-ci-bot merged commit 6728f2a into feast-dev:master Oct 25, 2021

vas28r13 deleted the 1863-direct-ingestion branch October 26, 2021 01:57

benny-hal reviewed Nov 16, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

direct data ingestion into Online store #1939

direct data ingestion into Online store #1939

vas28r13 commented Oct 14, 2021

feast-ci-bot commented Oct 14, 2021

feast-ci-bot commented Oct 14, 2021

felixwang9817 commented Oct 14, 2021

felixwang9817 commented Oct 14, 2021

codecov-commenter commented Oct 14, 2021 •

edited

Loading

adchia left a comment

adchia Oct 14, 2021

adchia Oct 14, 2021

adchia Oct 14, 2021

adchia Oct 14, 2021

vas28r13 commented Oct 21, 2021

adchia Oct 22, 2021

adchia left a comment

adchia left a comment

felixwang9817 commented Oct 22, 2021

adchia left a comment

adchia left a comment

feast-ci-bot commented Oct 25, 2021

benny-hal Nov 16, 2021

adchia Nov 16, 2021

benny-hal Nov 17, 2021

		@@ -424,6 +426,46 @@ def test_apply_remote_repo():
		)


		@pytest.mark.parametrize(

direct data ingestion into Online store #1939

direct data ingestion into Online store #1939

Conversation

vas28r13 commented Oct 14, 2021

feast-ci-bot commented Oct 14, 2021

feast-ci-bot commented Oct 14, 2021

felixwang9817 commented Oct 14, 2021

felixwang9817 commented Oct 14, 2021

codecov-commenter commented Oct 14, 2021 • edited Loading

Codecov Report

adchia left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vas28r13 commented Oct 21, 2021

Choose a reason for hiding this comment

adchia left a comment

Choose a reason for hiding this comment

adchia left a comment

Choose a reason for hiding this comment

felixwang9817 commented Oct 22, 2021

adchia left a comment

Choose a reason for hiding this comment

adchia left a comment

Choose a reason for hiding this comment

feast-ci-bot commented Oct 25, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Oct 14, 2021 •

edited

Loading