CLI command to start/stop/list streaming ingestion job on emr #1040
Conversation
oavdeev
requested review from
davidheryanto,
khorshuheng,
pyalex,
woop and
zhilingc
as code owners
oavdeev
force-pushed
the
aws-streaming-job
branch
from
70da00b
to
4e00729
Compare
|
/kind feature |
|
|
||
| client = Client() | ||
| table = client.get_feature_table(feature_table) | ||
| feast.pyspark.aws.jobs.start_stream_to_online(client, table, [jar] if jar else []) |
There was a problem hiding this comment.
Should we be bypassing the client here? What would happen in the case where we use the job service? or will that be added later?
There was a problem hiding this comment.
The job service will have to talk to the core service to get the metadata too, so it would need a client (for example, to find the feature tables so it can kick off streaming ingestion jobs)
There was a problem hiding this comment.
Oh I meant moving all requests through the client, even starting jobs. But you've already answered this.
|
@oavdeev can we add a release note to this PR please (as in PR template) |
|
/retest |
|
Did you rebase this PR @oavdeev. It's strange that its looking for a job controller. |
Signed-off-by: Oleg Avdeev <oleg.v.avdeev@gmail.com>
oavdeev
force-pushed
the
aws-streaming-job
branch
from
4e00729
to
cc5eeec
Compare
Added but i expect the actual interface may still change a bit before the release |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: oavdeev, woop The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/lgtm |
What this PR does / why we need it:
Add SDK glue code and the corresponding CLI command to kick off the streaming ingestion on EMR, check if it is already running, stop it and list running jobs.