Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement JobService API calls & connect it to SDK #1129

Merged
merged 13 commits into from
Nov 4, 2020
Merged

Implement JobService API calls & connect it to SDK #1129

merged 13 commits into from
Nov 4, 2020

Commits on Oct 29, 2020

  1. Implement half of JobService functionality 

    Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
    @tsotnet
    tsotnet committed Oct 29, 2020
    Configuration menu
    Copy the full SHA
    221badc View commit details
    Browse the repository at this point in the history

  2. Python lint 

    Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
    @tsotnet
    tsotnet committed Oct 29, 2020
    Configuration menu
    Copy the full SHA
    2c055f7 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into job-service-server 

    Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
    @tsotnet
    tsotnet committed Oct 29, 2020
    Configuration menu
    Copy the full SHA
    0296462 View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2020

  1. Correct implementation of all jobservice functions tested on standalo… 

    …ne mode

* New API calls (start_offline_to_online_ingestion, start_stream_to_online_ingestion) now return Remote Jobs instead of job ids
* Implement list_jobs & get_job for standalone mode (looks like Spark is running in local mode and we can't get job statuses so we have to keep cache in memory)
* Wire up list_jobs & get_job on client side with job service
* Tested locally on Feast 101 notebook, everything works

Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
    @tsotnet
    tsotnet committed Oct 30, 2020
    Configuration menu
    Copy the full SHA
    7e68055 View commit details
    Browse the repository at this point in the history

  2. Fix list_jobs when include_terminated=False 

    Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
    @tsotnet
    tsotnet committed Oct 30, 2020
    Configuration menu
    Copy the full SHA
    f59dc02 View commit details
    Browse the repository at this point in the history

Commits on Nov 2, 2020

  1. e2e tests 

    Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
    @tsotnet
    tsotnet committed Nov 2, 2020
    Configuration menu
    Copy the full SHA
    e5506e7 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into job-service-server 

    Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
    @tsotnet
    tsotnet committed Nov 2, 2020
    Configuration menu
    Copy the full SHA
    84f6e4c View commit details
    Browse the repository at this point in the history

  3. Format python 

    Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
    @tsotnet
    tsotnet committed Nov 2, 2020
    Configuration menu
    Copy the full SHA
    afe6e00 View commit details
    Browse the repository at this point in the history

Commits on Nov 3, 2020

  1. Merge branch 'master' into job-service-server 

    Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
    @tsotnet
    tsotnet committed Nov 3, 2020
    Configuration menu
    Copy the full SHA
    6cb6587 View commit details
    Browse the repository at this point in the history

  2. Add docstring 

    Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
    @tsotnet
    tsotnet committed Nov 3, 2020
    Configuration menu
    Copy the full SHA
    7e43530 View commit details
    Browse the repository at this point in the history

  3. Remove spark extra configs and hardcode spark jars/conf in stadalone … 

    …mode

Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
    @tsotnet
    tsotnet committed Nov 3, 2020
    Configuration menu
    Copy the full SHA
    3335ae4 View commit details
    Browse the repository at this point in the history

  4. Remove extra spark params from dockerfile 

    Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
    @tsotnet
    tsotnet committed Nov 3, 2020
    Configuration menu
    Copy the full SHA
    ca43800 View commit details
    Browse the repository at this point in the history

Commits on Nov 4, 2020

  1. Use start_stream_to_online_ingestion from launcher in job service 

    Signed-off-by: Tsotne Tabidze <tsotnet@gmail.com>
    @tsotnet
    tsotnet committed Nov 4, 2020
    Configuration menu
    Copy the full SHA
    dc92728 View commit details
    Browse the repository at this point in the history