Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nomad exec part 1: plumbing and docker driver #5632

Merged
merged 15 commits into from
May 9, 2019

Commits on Apr 30, 2019

  1. aux: helper method that returns token as well as ACL policy

    This helper returns the token as well as the ACL policy, to be used in a later
    commit for logging the token info associated with nomad exec invocation.
    Mahmood Ali committed Apr 30, 2019
    Configuration menu
    Copy the full SHA
    43cd7a7 View commit details
    Browse the repository at this point in the history
  2. test helper for registering jobs with acl

    Test helper that allows registration of jobs when ACL is activated.
    Mahmood Ali committed Apr 30, 2019
    Configuration menu
    Copy the full SHA
    75349d6 View commit details
    Browse the repository at this point in the history
  3. drivers/mock: extract command related operations

    Extract command parsing and execution mocking into a separate struct.  Also,
    allow mocking of different fs_isolation for testing.
    Mahmood Ali committed Apr 30, 2019
    Configuration menu
    Copy the full SHA
    e79ce1f View commit details
    Browse the repository at this point in the history
  4. Add ACL capabilities for nomad exec

    This adds `alloc-exec` capability to allow operator to execute command into a
    running task.  Furthermore, it adds `alloc-node-exec` capability, required when
    the alloc task is raw_exec or a driver with no FSIsolation.
    Mahmood Ali committed Apr 30, 2019
    Configuration menu
    Copy the full SHA
    4f7bd68 View commit details
    Browse the repository at this point in the history
  5. add nomad streaming exec core data structures and interfaces

    In this commit, we add two driver interfaces for supporting `nomad exec`
    invocation:
    
    * A high level `ExecTaskStreamingDriver`, that operates on io reader/writers.
      Drivers should prefer using this interface
    * A low level `ExecTaskStreamingRawDriver` that operates on the raw stream of
      input structs; useful when a driver delegates handling to driver backend (e.g.
      across RPC/grpc).
    
    The interfaces are optional for a driver, as `nomad exec` support is opt-in.
    Existing drivers continue to compile without exec support, until their
    maintainer add such support.
    
    Furthermore, we create protobuf structures to represent exec stream entities:
    `ExecTaskStreamingRequest` and `ExecTaskStreamingResponse`.  We aim to reuse the
    protobuf generated code as much as possible, without translation to avoid
    conversion overhead.
    
    `ExecTaskStream` abstract fetching and sending stream entities.  It's influenced
    by the grpc bi-directional stream interface, to avoid needing any adapter.  I
    considered using channels, but the asynchronisity and concurrency makes buffer
    reuse too complicated, which would put more pressure on GC and slows exec operation.
    Mahmood Ali committed Apr 30, 2019
    Configuration menu
    Copy the full SHA
    6d711d0 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2019

  1. implemment streaming exec handling in driver grpc handlers

    Also add a helper that converts the adapts the high level interface to the
    low-level interface of nomad exec interfaces.
    Mahmood Ali committed May 9, 2019
    Configuration menu
    Copy the full SHA
    94ed649 View commit details
    Browse the repository at this point in the history
  2. drivers/mock: implement nomad exec interface

    Mahmood Ali committed May 9, 2019
    Configuration menu
    Copy the full SHA
    cb4ad3f View commit details
    Browse the repository at this point in the history
  3. implement client endpoint of nomad exec

    Add a client streaming RPC endpoint for processing nomad exec tasks, by invoking
    the relevant task handler for execution.
    Mahmood Ali committed May 9, 2019
    Configuration menu
    Copy the full SHA
    979a6a1 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    a77a3ba View commit details
    Browse the repository at this point in the history
  5. agent: add websocket handler for nomad exec

    This adds a websocket endpoint for handling `nomad exec`.
    
    The endpoint is a websocket interface, as we require a bi-directional
    streaming (to handle both input and output), which is not very appropriate for
    plain HTTP 1.0. Using websocket makes implementing the web ui a bit simpler. I
    considered using golang http hijack capability to treat http request as a plain
    connection, but the web interface would be too complicated potentially.
    
    Furthermore, the API endpoint operates against the raw core nomad exec streaming
    datastructures, defined in protobuf, with json serializer.  Our APIs use json
    interfaces in general, and protobuf generates json friendly golang structs.
    Reusing the structs here simplify interface and reduce conversion overhead.
    Mahmood Ali committed May 9, 2019
    Configuration menu
    Copy the full SHA
    bfb4f0c View commit details
    Browse the repository at this point in the history
  6. vendor github.com/gorilla/websocket

    Mahmood Ali committed May 9, 2019
    Configuration menu
    Copy the full SHA
    f908d6b View commit details
    Browse the repository at this point in the history
  7. Add basic drivers conformance tests

    Add consolidated testing package to serve as conformance tests for all drivers.
    Mahmood Ali committed May 9, 2019
    Configuration menu
    Copy the full SHA
    3a77502 View commit details
    Browse the repository at this point in the history
  8. drivers/docker: implement streaming exec

    Mahmood Ali committed May 9, 2019
    Configuration menu
    Copy the full SHA
    67160a6 View commit details
    Browse the repository at this point in the history
  9. add api support for nomad exec

    Adds nomad exec support in our API, by hitting the websocket endpoint.
    
    We introduce API structs that correspond to the drivers streaming exec structs.
    
    For creating the websocket connection, we reuse the transport setting from api
    http client.
    Mahmood Ali committed May 9, 2019
    Configuration menu
    Copy the full SHA
    6168cc5 View commit details
    Browse the repository at this point in the history
  10. add e2e tests for nomad exec

    Mahmood Ali committed May 9, 2019
    Configuration menu
    Copy the full SHA
    980e4f5 View commit details
    Browse the repository at this point in the history