fix: Update file api #2470

kevjumba · 2022-03-30T22:35:38Z

What this PR does / why we need it:
Change file options param file_url to uri. Deprecates file_url immediately because file_url is not user facing and is only used in FileSource.
DOES NOT update the API for java(i.e java still uses fileurl).

Which issue(s) this PR fixes:

Fixes #

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

achals · 2022-03-30T22:41:46Z

protos/feast/core/DataSource.proto

@@ -89,10 +89,18 @@ message DataSource {
    // s3://path/to/file for AWS S3 storage
    // gs://path/to/file for GCP GCS storage
    // file:///path/to/file for local storage
+    // Deprecated in Feast 0.23.0
    string file_url  = 2;


You should change this in place; serialized protobufs only use the index number not the field name.

Ah I didn't know that but that makes a lot of sense! Thanks for the suggestion that makes the update so much easier.

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

achals · 2022-03-31T16:39:42Z

sdk/python/feast/infra/offline_stores/file_source.py

    def __init__(
        self,
        file_format: Optional[FileFormat],
-        file_url: Optional[str],
        s3_endpoint_override: Optional[str],
+        uri: Optional[str],


Wanted to confirm that this is not user facing?

Yup isn't called anymore but maybe @adchia can also confirm?

achals

just a question, but looks good otherwise.

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

codecov-commenter · 2022-03-31T17:31:12Z

Codecov Report

Merging #2470 (bab7542) into master (6b96b21) will decrease coverage by 26.27%.
The diff coverage is 90.90%.

@@             Coverage Diff             @@
##           master    #2470       +/-   ##
===========================================
- Coverage   84.87%   58.60%   -26.28%     
===========================================
  Files         127      126        -1     
  Lines       10888    10836       -52     
===========================================
- Hits         9241     6350     -2891     
- Misses       1647     4486     +2839

Flag	Coverage Δ
integrationtests	`?`
unittests	`58.60% <90.90%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdk/python/feast/infra/offline_stores/file.py	`86.22% <ø> (-10.72%)`	⬇️
...k/python/feast/infra/offline_stores/file_source.py	`82.85% <90.90%> (-7.77%)`	⬇️
.../integration/online_store/test_online_retrieval.py	`17.39% <0.00%> (-82.61%)`	⬇️
sdk/python/tests/utils/online_read_write_test.py	`20.68% <0.00%> (-79.32%)`	⬇️
.../integration/online_store/test_universal_online.py	`14.48% <0.00%> (-78.84%)`	⬇️
...fline_store/test_universal_historical_retrieval.py	`23.50% <0.00%> (-76.50%)`	⬇️
sdk/python/feast/infra/online_stores/redis.py	`27.04% <0.00%> (-69.19%)`	⬇️
...gration/registration/test_feature_service_apply.py	`31.25% <0.00%> (-68.75%)`	⬇️
sdk/python/tests/integration/e2e/test_usage_e2e.py	`33.87% <0.00%> (-66.13%)`	⬇️
sdk/python/tests/data/data_creator.py	`34.78% <0.00%> (-65.22%)`	⬇️
... and 66 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6b96b21...bab7542. Read the comment docs.

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

achals · 2022-03-31T18:28:08Z

sdk/python/feast/infra/offline_stores/file_source.py

+        self._uri = uri
+        if not self._uri:
+            self._uri = self._file_url


you can do self._uri = uri or file_url and probably not store self._file_url at all

Great catch! thanks!

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

achals

/lgtm

feast-ci-bot · 2022-03-31T18:39:27Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: achals, kevjumba

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [achals,kevjumba]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

# [0.20.0](v0.19.0...v0.20.0) (2022-04-14) ### Bug Fixes * Add inlined data sources to the top level registry ([#2456](#2456)) ([356788a](356788a)) * Add new value types to types.ts for web ui ([#2463](#2463)) ([ad5694e](ad5694e)) * Add PushSource proto and Python class ([#2428](#2428)) ([9a4bd63](9a4bd63)) * Add spark to lambda dockerfile ([#2480](#2480)) ([514666f](514666f)) * Added private_key auth for Snowflake ([#2508](#2508)) ([c42c9b0](c42c9b0)) * Added Redshift and Spark typecheck to data_source event_timestamp_col inference ([#2389](#2389)) ([04dea73](04dea73)) * Building of go extension fails ([#2448](#2448)) ([7d1efd5](7d1efd5)) * Bump the number of versions bumps expected to 27 ([#2549](#2549)) ([ecc9938](ecc9938)) * Create __init__ files for the proto-generated python dirs ([#2410](#2410)) ([e17028d](e17028d)) * Don't prevent apply from running given duplicate empty names in data sources. Also fix repeated apply of Spark data source. ([#2415](#2415)) ([b95f441](b95f441)) * Dynamodb deduplicate batch write request by partition keys ([#2515](#2515)) ([70d4a13](70d4a13)) * Ensure that __init__ files exist in proto dirs ([#2433](#2433)) ([9b94f7b](9b94f7b)) * Fix DataSource constructor to unbreak custom data sources ([#2492](#2492)) ([712653e](712653e)) * Fix default feast apply path without any extras ([#2373](#2373)) ([6ba7fc7](6ba7fc7)) * Fix definitions.py with new definition ([#2541](#2541)) ([eefc34a](eefc34a)) * Fix entity row to use join key instead of name ([#2521](#2521)) ([c22fa2c](c22fa2c)) * Fix Java Master ([#2499](#2499)) ([e083458](e083458)) * Fix registry proto ([#2435](#2435)) ([ea6a9b2](ea6a9b2)) * Fix some inconsistencies in the docs and comments in the code ([#2444](#2444)) ([ad008bf](ad008bf)) * Fix spark docs ([#2382](#2382)) ([d4a606a](d4a606a)) * Fix Spark template to work correctly on feast init -t spark ([#2393](#2393)) ([ae133fd](ae133fd)) * Fix the feature repo fixture used by java tests ([#2469](#2469)) ([32e925e](32e925e)) * Fix unhashable Snowflake and Redshift sources ([cd8f1c9](cd8f1c9)) * Fixed bug in passing config file params to snowflake python connector ([#2503](#2503)) ([34f2b59](34f2b59)) * Fixing Spark template to include source name ([#2381](#2381)) ([a985f1d](a985f1d)) * Make name a keyword arg for the Entity class ([#2467](#2467)) ([43847de](43847de)) * Making a name for data sources not a breaking change ([#2379](#2379)) ([71d7ae2](71d7ae2)) * Minor link fix in `CONTRIBUTING.md` ([#2481](#2481)) ([2917e27](2917e27)) * Preserve ordering of features in _get_column_names ([#2457](#2457)) ([495b435](495b435)) * Relax click python requirement to >=7 ([#2450](#2450)) ([f202f92](f202f92)) * Remove date partition column field from datasources that don't s… ([#2478](#2478)) ([ce35835](ce35835)) * Remove docker step from unit test workflow ([#2535](#2535)) ([6f22f22](6f22f22)) * Remove spark from the AWS Lambda dockerfile ([#2498](#2498)) ([6abae16](6abae16)) * Request data api update ([#2488](#2488)) ([0c9e5b7](0c9e5b7)) * Schema update ([#2509](#2509)) ([cf7bbc2](cf7bbc2)) * Simplify DataSource.from_proto logic ([#2424](#2424)) ([6bda4d2](6bda4d2)) * Snowflake api update ([#2487](#2487)) ([1181a9e](1181a9e)) * Support passing batch source to streaming sources for backfills ([#2523](#2523)) ([90db1d1](90db1d1)) * Timestamp update ([#2486](#2486)) ([bf23111](bf23111)) * Typos in Feast UI error message ([#2432](#2432)) ([e14369d](e14369d)) * Update feature view APIs to prefer keyword args ([#2472](#2472)) ([7c19cf7](7c19cf7)) * Update file api ([#2470](#2470)) ([83a11c6](83a11c6)) * Update Makefile to cd into python dir before running commands ([#2437](#2437)) ([ca32155](ca32155)) * Update redshift api ([#2479](#2479)) ([4fa73a9](4fa73a9)) * Update some fields optional in UI parser ([#2380](#2380)) ([cff7ac3](cff7ac3)) * Use a single version of jackson libraries and upgrade to 2.12.6.1 ([#2473](#2473)) ([5be1cc6](5be1cc6)) * Use dateutil parser to parse materialization times ([#2464](#2464)) ([6c55e49](6c55e49)) * Use the correct dockerhub image tag when building feature servers ([#2372](#2372)) ([0d62c1d](0d62c1d)) ### Features * Add `/write-to-online-store` method to the python feature server ([#2423](#2423)) ([d2fb048](d2fb048)) * Add description, tags, owner fields to all feature view classes ([#2440](#2440)) ([ed5e928](ed5e928)) * Add DQM Logging on GRPC Server with FileLogStorage for Testing ([#2403](#2403)) ([57a97d8](57a97d8)) * Add Feast types in preparation for changing type system ([#2475](#2475)) ([4864252](4864252)) * Add Field class ([#2500](#2500)) ([1279612](1279612)) * Add support for DynamoDB online_read in batches ([#2371](#2371)) ([702ec49](702ec49)) * Add Support for DynamodbOnlineStoreConfig endpoint_url parameter ([#2485](#2485)) ([7b863d1](7b863d1)) * Add templating for dynamodb table name ([#2394](#2394)) ([f591088](f591088)) * Allow local feature server to use Go feature server if enabled ([#2538](#2538)) ([a2ef375](a2ef375)) * Allow using entity's join_key in get_online_features ([#2420](#2420)) ([068c765](068c765)) * Data Source Api Update ([#2468](#2468)) ([6b96b21](6b96b21)) * Go server ([#2339](#2339)) ([d12e7ef](d12e7ef)), closes [#2354](#2354) [#2361](#2361) [#2332](#2332) [#2356](#2356) [#2363](#2363) [#2349](#2349) [#2355](#2355) [#2336](#2336) [#2361](#2361) [#2363](#2363) [#2344](#2344) [#2354](#2354) [#2347](#2347) [#2350](#2350) [#2356](#2356) [#2355](#2355) [#2349](#2349) [#2352](#2352) [#2341](#2341) [#2336](#2336) [#2373](#2373) [#2315](#2315) [#2372](#2372) [#2332](#2332) [#2349](#2349) [#2336](#2336) [#2361](#2361) [#2363](#2363) [#2344](#2344) [#2354](#2354) [#2347](#2347) [#2350](#2350) [#2356](#2356) [#2355](#2355) [#2349](#2349) [#2352](#2352) [#2341](#2341) [#2336](#2336) [#2373](#2373) [#2379](#2379) [#2380](#2380) [#2382](#2382) [#2364](#2364) [#2366](#2366) [#2386](#2386) * Graduate write_to_online_store out of experimental status ([#2426](#2426)) ([e7dd4b7](e7dd4b7)) * Make feast PEP 561 compliant ([#2405](#2405)) ([3c41f94](3c41f94)), closes [#2420](#2420) [#2418](#2418) [#2425](#2425) [#2426](#2426) [#2427](#2427) [#2431](#2431) [#2433](#2433) [#2420](#2420) [#2418](#2418) [#2425](#2425) [#2426](#2426) [#2427](#2427) [#2431](#2431) [#2433](#2433) * Makefile for contrib for Issue [#2364](#2364) ([#2366](#2366)) ([a02325b](a02325b)) * Support on demand feature views in go feature server ([#2494](#2494)) ([6edd274](6edd274)) * Switch from Feature to Field ([#2514](#2514)) ([6a03bed](6a03bed)) * Use a daemon thread to monitor the go feature server exclusively ([#2391](#2391)) ([0bb5e8c](0bb5e8c))

kevjumba added 4 commits March 30, 2022 15:19

Update file

fc0d160

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Fix

d79d9d8

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Fix

13adf24

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Lint

9d5fe71

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

kevjumba added area/core kind/housekeeping python Pull requests that update Python code labels Mar 30, 2022

kevjumba requested a review from a team as a code owner March 30, 2022 22:35

kevjumba requested review from mavysavydav and removed request for a team March 30, 2022 22:35

feast-ci-bot added the approved label Mar 30, 2022

kevjumba requested a review from achals March 30, 2022 22:35

feast-ci-bot added the size/M label Mar 30, 2022

github-actions bot assigned felixwang9817 Mar 30, 2022

kevjumba removed the request for review from mavysavydav March 30, 2022 22:39

achals requested changes Mar 30, 2022

View reviewed changes

kevjumba added 3 commits March 30, 2022 15:59

Update the schema

3c91830

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Fix lint

aad3322

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Fix lint

c59a7d6

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

kevjumba force-pushed the update_file branch from 4e317c9 to c59a7d6 Compare March 30, 2022 23:02

achals reviewed Mar 31, 2022

View reviewed changes

Fix unit tests

3727bd2

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

kevjumba added 3 commits March 31, 2022 11:18

Keep file_url param

4f6ffef

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Fix

291463a

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

Fix lint

64c740b

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

achals reviewed Mar 31, 2022

View reviewed changes

Fix

bab7542

Signed-off-by: Kevin Zhang <kzhang@tecton.ai>

achals approved these changes Mar 31, 2022

View reviewed changes

feast-ci-bot assigned achals Mar 31, 2022

feast-ci-bot added the lgtm label Mar 31, 2022

feast-ci-bot merged commit 83a11c6 into feast-dev:master Mar 31, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Update file api #2470

fix: Update file api #2470

kevjumba commented Mar 30, 2022

achals Mar 30, 2022

kevjumba Mar 30, 2022

achals Mar 31, 2022

kevjumba Mar 31, 2022

achals left a comment

codecov-commenter commented Mar 31, 2022 •

edited

Loading

achals Mar 31, 2022

kevjumba Mar 31, 2022

achals left a comment

feast-ci-bot commented Mar 31, 2022

fix: Update file api #2470

fix: Update file api #2470

Conversation

kevjumba commented Mar 30, 2022

achals Mar 30, 2022

Choose a reason for hiding this comment

kevjumba Mar 30, 2022

Choose a reason for hiding this comment

achals Mar 31, 2022

Choose a reason for hiding this comment

kevjumba Mar 31, 2022

Choose a reason for hiding this comment

achals left a comment

Choose a reason for hiding this comment

codecov-commenter commented Mar 31, 2022 • edited Loading

Codecov Report

achals Mar 31, 2022

Choose a reason for hiding this comment

kevjumba Mar 31, 2022

Choose a reason for hiding this comment

achals left a comment

Choose a reason for hiding this comment

feast-ci-bot commented Mar 31, 2022

codecov-commenter commented Mar 31, 2022 •

edited

Loading