Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-25957][K8S] Make building alternate language binding docker images optional #23053

Closed
wants to merge 6 commits into from

Commits on Nov 16, 2018

  1. [SPARK-25957][K8S] Add ability to skip building optional k8s docker i…

    …mages
    
    bin/docker-image-tool.sh tries to build all docker images (JVM, PySpark
    and SparkR) by default. But not all spark distributions are built with
    SparkR and hence this script will fail on such distros.
    
    With this change,
    - We should be able to skip building optional docker
      images (PySpark and SparkR) by specifying -pskip or -Rskip flags.
    - We autodetect if SparkR is not installed in the build and skip building
      SparkR docker image.
    - We skip pushing docker images that are not available locally.
    
    Tested following scenarios.
    - On source code and distro with SparkR support
      - Run bin/docker-image-tool.sh -r <repo> -t <tag> build. Verify that JVM, PySpark and SparkR docker images are built.
      - Run bin/docker-image-tool.sh -r <repo> -t <tag> -Rskip -pskip build. Verify that only JVM docker image is built. Building PySpark and SparkR images is skipped.
    - On source code and distro without SparkR support
      - Run bin/docker-image-tool.sh -r <repo> -t <tag> build. Verify that only JVM, PySpark docker images are built. Building SparkR image is skipped.
    - On system with JVM, PySpark and SparkR images built,
      - Run bin/docker-image-tool.sh -r <repo> -t <tag> push. Verify that all images are pushed to docker registry.
    - On system with only JVM and PySpark images built.
      - Run bin/docker-image-tool.sh -r <repo> -t <tag> push. Verify that only JVM and PySpark images are pushed. Pushing SparkR images is skipped.
    ramaddepally committed Nov 16, 2018
    Configuration menu
    Copy the full SHA
    ae1d743 View commit details
    Browse the repository at this point in the history
  2. Address review comments

    * Update usage to specify how to skip building docker image
    * explicitly log message when building R image is skipped due to R packages not found
    ramaddepally committed Nov 16, 2018
    Configuration menu
    Copy the full SHA
    8887b5a View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2018

  1. Configuration menu
    Copy the full SHA
    11080c9 View commit details
    Browse the repository at this point in the history
  2. Update usage

    ramaddepally committed Nov 20, 2018
    Configuration menu
    Copy the full SHA
    5ec7cb7 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    7601cbc View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2018

  1. Configuration menu
    Copy the full SHA
    86a11a8 View commit details
    Browse the repository at this point in the history