-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support TFRecord as one of the output formats for historical feature retrieval #1222
Conversation
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: khorshuheng The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
1ca8d28
to
7d62dec
Compare
|
||
|
||
def test_dataproc_job_tfrecord_output( | ||
dataproc_launcher: DataprocClusterLauncher, # noqa: F811 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is with all these noqa: F811
? Can it be fixed please?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's because we are importing the fixtures from another module, and flake8 doesn't understand pytest fixtures.
One way to fix it would be not to import the fixture and include it directly in test_launchers
, though that would mean it's harder for us to reuse the fixture across different test files.
I will research for way on how this can be circumvented without explicitly adding # noqa: F811
, but i am not certain if it is possible.
7bb80e3
to
2d019f1
Compare
"Args": ["spark-submit", pyspark_script_path] | ||
+ args | ||
+ ["--packages", ",".join(packages)] | ||
if packages | ||
else [], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"Args": ["spark-submit", pyspark_script_path] | |
+ args | |
+ ["--packages", ",".join(packages)] | |
if packages | |
else [], | |
"Args": ["spark-submit", pyspark_script_path] | |
+ args | |
+ (["--packages", ",".join(packages)] | |
if packages | |
else []), |
1238e6c
to
0937a1b
Compare
/test test-end-to-end-sparkop |
3 similar comments
/test test-end-to-end-sparkop |
/test test-end-to-end-sparkop |
/test test-end-to-end-sparkop |
/test test-end-to-end-azure |
…retrieval Signed-off-by: Khor Shu Heng <khor.heng@gojek.com>
Signed-off-by: Khor Shu Heng <khor.heng@gojek.com>
Signed-off-by: Khor Shu Heng <khor.heng@gojek.com>
Signed-off-by: Khor Shu Heng <khor.heng@gojek.com>
Signed-off-by: Khor Shu Heng <khor.heng@gojek.com>
Signed-off-by: Khor Shu Heng <khor.heng@gojek.com>
Signed-off-by: Khor Shu Heng <khor.heng@gojek.com>
Signed-off-by: Khor Shu Heng <khor.heng@gojek.com>
Signed-off-by: Khor Shu Heng <khor.heng@gojek.com>
Signed-off-by: Khor Shu Heng <khor.heng@gojek.com>
Signed-off-by: Khor Shu Heng <khor.heng@gojek.com>
39840ec
to
5d80975
Compare
Signed-off-by: Khor Shu Heng <khor.heng@gojek.com>
@khorshuheng: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
/test test-end-to-end |
What this PR does / why we need it:
This PR allows user to specify
tfrecord
as the output format of historical feature retrieval, which is useful in cases where the users wish to generate statistics from the retrieved dataset using tfdv, or if the user's machine learning model is based on tensorflow.Which issue(s) this PR fixes:
Fixes #
Does this PR introduce a user-facing change?: