Support printing features and returning keys when getting features from registry #886

enya-yx · 2022-11-29T13:06:06Z

Description

Provide function to show features details (including keys) in json format and return a [feature_name] = [typed keys] dict when 'get_features_from_registry'

How was this PR tested?

Tested locally, and got log like below:

- Modified one of existing test cases to print out features details and to return key dicts

Does this PR introduce any user-facing changes?

No. You can skip the rest of this section.
Yes. Make sure to clarify your proposed changes.

* Add working gradle build * Set up pdl support * Working PDL java code gen * With pdl files from metadata models * With pdl files from compute model * Fix compile for all pdl files * Add working gradle build * Migrate frame-config module into feathr * Migrate fcm graph module to feathr * Add FCM offline execution code, includes FDS metadata code * Add needed jars for feathr-config tests * Switch client to FeathrClient2 for local tests and fix config errors * Fix SWA test * Add gradle wrapper jar * Change name of git PR test from sbt to gradle * Switch python client to use FCM client * Exclude json from dependency * Add hacky solution to handle json dependency conflict in cloud * Add json to local dependency * Add log to debug cloud jar * Add json as dependency * Another attempt at resolving json dependency * Resolve json via shading * Fix json shading * Remove log * Shade typesafe config for cloud jar * Add maven publish code to build.gradle * Add working local maven build and rename frame-config to feathr-config to avoid namespace conflict * Modify sonatype creds * Change so no need to sign if releasing snapshot version * Update build.gradle to allow publishing of all modules * Removed FDS handling from Feathr * All tests working * Deleted FR stuff * Remove dimension and other tensor related stuff * Remove mlfeatureversionurn from defaultvalueresolver * Remove mlfeatureversionurn and featureref * Remove featuredefinition files * Remove featureRef and typedRef * final cleanup * Fix merge conflict bugs * Fix guava error * udf plugin for swa features * row-transformations optimization * fix bug * fix another bug * always execute agg nodes first * Add SWA log * reverse order of execution * group by datasource * Fix bug * Merge main into fcm branch * Remove insecure URLs * Add back removed files * Add back removed files * Add back removed files * Change PR build system to gradle * Change sbt job to gradle jobb * Change sbt workflow:wq * Update maven github workflow to use gradle * fix failing test * remove sbt project module * Remove sbt related files * Change docs to reflect gradle * Remove keywords * Create a single jar * 1. Fix jar not getting populated\n 2. Fix documentation bugs * pubishToMavenLocal Working * With FFE integrated * maven upload working * Update docs and code clean up * add gradle-wrapper file * Push all dependency jars * Update docs * Docs cleanup * Update github workflow commands * Update github workflow * Update workflow syntax * Update version * Add gradle version to github workflow * Update gradle version w/o quotes * Remove github gradle version * Github workflow fix * Github workflow fix-2 * Github workflow fix-4 Co-authored-by: Bozhong Hu <bobhu@linkedin.com> Co-authored-by: rkashyap <rkashyap@linkedin.com>

… arg order to make it work with old codes (feathr-ai#890) Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com> Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

Bumps [minimatch](https://github.com/isaacs/minimatch) and [recursive-readdir](https://github.com/jergason/recursive-readdir). These dependencies needed to be updated together. Updates `minimatch` from 3.0.4 to 3.1.2 - [Release notes](https://github.com/isaacs/minimatch/releases) - [Changelog](https://github.com/isaacs/minimatch/blob/main/changelog.md) - [Commits](isaacs/minimatch@v3.0.4...v3.1.2) Updates `recursive-readdir` from 2.2.2 to 2.2.3 - [Release notes](https://github.com/jergason/recursive-readdir/releases) - [Changelog](https://github.com/jergason/recursive-readdir/blob/master/CHANGELOG.md) - [Commits](https://github.com/jergason/recursive-readdir/commits/v2.2.3) --- updated-dependencies: - dependency-name: minimatch dependency-type: indirect - dependency-name: recursive-readdir dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add docs for checking/improving test coverage

feathr_project/test/test_feature_registry.py

feathr_project/feathr/client.py

* registry-changes * update purview * remove delete functionality for now * update tests * remove unused import * update endpoints * fix locking issue * Update _feature_registry_purview.py * remove cascading delete * Update feature_registry.py * update access control * update status code to 412

Signed-off-by: Yuqing Wei <weiyuqing021@outlook.com>

* add sql credential pass through doc Signed-off-by: Yuqing Wei <weiyuqing021@outlook.com> * fix comments Signed-off-by: Yuqing Wei <weiyuqing021@outlook.com> * fix comments Signed-off-by: Yuqing Wei <weiyuqing021@outlook.com> Signed-off-by: Yuqing Wei <weiyuqing021@outlook.com>

* update registry test, modify log Signed-off-by: Yuqing Wei <weiyuqing021@outlook.com> * fix comment Signed-off-by: Yuqing Wei <weiyuqing021@outlook.com> Signed-off-by: Yuqing Wei <weiyuqing021@outlook.com>

feathr_project/feathr/client.py

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com> Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

* Add feature embedding example. Update README Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com> * Add feature embedding notebook test Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com> * increase notebook's spark job timeout Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com> Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

Bumps [decode-uri-component](https://github.com/SamVerschueren/decode-uri-component) from 0.2.0 to 0.2.2. - [Release notes](https://github.com/SamVerschueren/decode-uri-component/releases) - [Commits](SamVerschueren/decode-uri-component@v0.2.0...v0.2.2) --- updated-dependencies: - dependency-name: decode-uri-component dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…thr-ai#858) * Add 'postfixPath' to support time pattern in the middle of paths * Expose 'postfix_path' to datasource API * Add test cases & documents

xiaoyongzhu · 2022-12-02T14:58:28Z

LGTM, thanks @enya-yx for this!

feathr_project/feathr/client.py

… into enya/print-features

Support printing features function when getting features from registry

33e16d9

enya-yx added the safe to test Tag to execute build pipeline for a PR from forked repo label Nov 29, 2022

rakeshkashyap123 and others added 9 commits November 29, 2022 20:48

Add job_tag to materialization job submission. Change get_result_df's…

2d6da56

… arg order to make it work with old codes (feathr-ai#890) Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com> Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

Merge branch 'main' into enya/print-features

66165ca

Merge branch 'main' into enya/print-features

5a3dbf1

Add return keys function

892d5d0

Add return keys function

8bd5cc6

quick fix

1e6aebb

quick fix

3e3fe7e

enya-yx changed the title ~~Support printing features function when getting features from registry~~ Support printing features and returning keys when getting features from registry Dec 1, 2022

enya-yx requested review from xiaoyongzhu, blrchen, edwincheung, jainr and Yuqing-cat December 1, 2022 06:33

Add docs for checking/improving test coverage (feathr-ai#884)

cae6a99

* Add docs for checking/improving test coverage

xiaoyongzhu reviewed Dec 1, 2022

View reviewed changes

feathr_project/test/test_feature_registry.py Outdated Show resolved Hide resolved

xiaoyongzhu reviewed Dec 1, 2022

View reviewed changes

feathr_project/feathr/client.py Show resolved Hide resolved

enya-yx added 2 commits December 1, 2022 18:31

add composite keys in test cases

69849b5

add composite keys in test cases

d2be7d9

blrchen reviewed Dec 1, 2022

View reviewed changes

feathr_project/feathr/client.py Outdated Show resolved Hide resolved

aabbasi-hbo and others added 6 commits December 1, 2022 08:43

Add KeyError for Key and Feature Type (feathr-ai#877)

0c00490

Signed-off-by: Yuqing Wei <weiyuqing021@outlook.com>

use logger to print

71462ac

use logger to print

f5ee741

update registry test, modify log (feathr-ai#892)

bb36583

* update registry test, modify log Signed-off-by: Yuqing Wei <weiyuqing021@outlook.com> * fix comment Signed-off-by: Yuqing Wei <weiyuqing021@outlook.com> Signed-off-by: Yuqing Wei <weiyuqing021@outlook.com>

xiaoyongzhu reviewed Dec 2, 2022

View reviewed changes

feathr_project/feathr/client.py Outdated Show resolved Hide resolved

loomlike and others added 10 commits December 2, 2022 00:53

Fix empty job tag (feathr-ai#895)

1233653

Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com> Signed-off-by: Jun Ki Min <42475935+loomlike@users.noreply.github.com>

quick change

8da9af7

quick change

c4ce150

quick fix

0614863

quick fix

24c5a59

Add 'postfixPath' to support time pattern in the middle of paths (fea…

2ee97ce

…thr-ai#858) * Add 'postfixPath' to support time pattern in the middle of paths * Expose 'postfix_path' to datasource API * Add test cases & documents

Merge branch 'main' into enya/print-features

d196023

Merge branch 'main' into enya/print-features

8a3066a

xiaoyongzhu previously approved these changes Dec 2, 2022

View reviewed changes

loomlike reviewed Dec 2, 2022

View reviewed changes

feathr_project/feathr/client.py Outdated Show resolved Hide resolved

loomlike reviewed Dec 2, 2022

View reviewed changes

feathr_project/feathr/client.py Show resolved Hide resolved

jaymo001 force-pushed the main branch from d24d4e6 to 635e33f Compare December 7, 2022 17:12

jaymo001 requested a review from donegjookim as a December 7, 2022 17:12

jaymo001 force-pushed the main branch from 635e33f to 77571fd Compare December 7, 2022 17:28

jaymo001 dismissed xiaoyongzhu’s stale review via d196023 December 8, 2022 04:42

jaymo001 force-pushed the enya/print-features branch from 8a3066a to d196023 Compare December 8, 2022 04:42

enya-yx added 4 commits December 13, 2022 15:00

Merge branch 'enya/print-features' of https://github.com/enya-yx/feathr…

e3d9ae7

… into enya/print-features

Merge branch 'main' into enya/print-features

ddce381

Quick fix and merge with main

0b65caa

Remove extra files

ede7f70

blrchen approved these changes Dec 15, 2022

View reviewed changes

Yuqing-cat approved these changes Dec 15, 2022

View reviewed changes

enya-yx merged commit 8c8dfde into feathr-ai:main Dec 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support printing features and returning keys when getting features from registry #886

Support printing features and returning keys when getting features from registry #886

enya-yx commented Nov 29, 2022 •

edited

Loading

xiaoyongzhu commented Dec 2, 2022

Support printing features and returning keys when getting features from registry #886

Support printing features and returning keys when getting features from registry #886

Conversation

enya-yx commented Nov 29, 2022 • edited Loading

Description

How was this PR tested?

Does this PR introduce any user-facing changes?

xiaoyongzhu commented Dec 2, 2022

enya-yx commented Nov 29, 2022 •

edited

Loading