Skip to content
This repository has been archived by the owner on Jan 9, 2020. It is now read-only.

[WIP] Submission client redesign to use a step-based builder pattern #363

Merged
merged 2 commits into from
Jun 30, 2017

Commits on Jun 29, 2017

  1. Submission client redesign to use a step-based builder pattern.

    This change overhauls the underlying architecture of the submission
    client, but it is intended to entirely preserve existing behavior of
    Spark applications. Therefore users will find this to be an invisible
    change.
    
    The philosophy behind this design is to reconsider the breakdown of the
    submission process. It operates off the abstraction of "submission
    steps", which are transformation functions that take the previous state
    of the driver and return the new state of the driver. The driver's state
    includes its Spark configurations and the Kubernetes resources that will
    be used to deploy it.
    
    Such a refactor moves away from a features-first API design, which
    considers different containers to serve a set of features. The previous
    design, for example, had a container files resolver API object that
    returned different resolutions of the dependencies added by the user.
    However, it was up to the main Client to know how to intelligently
    invoke all of those APIs. Therefore the API surface area of the file
    resolver became untenably large and it was not intuitive of how it was
    to be used or extended.
    
    This design changes the encapsulation layout; every module is now
    responsible for changing the driver specification directly. An
    orchestrator builds the correct chain of steps and hands it to the
    client, which then calls it verbatim. The main client then makes any
    final modifications that put the different pieces of the driver
    together, particularly to attach the driver container itself to the pod
    and to apply the Spark configuration as command-line arguments.
    mccheah committed Jun 29, 2017
    Configuration menu
    Copy the full SHA
    5499f6d View commit details
    Browse the repository at this point in the history

Commits on Jun 30, 2017

  1. Configuration menu
    Copy the full SHA
    e103225 View commit details
    Browse the repository at this point in the history