Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: Build multi-platform Docker images #727

Merged
merged 27 commits into from
Oct 2, 2020
Merged

CI: Build multi-platform Docker images #727

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
6a71d5a
:recycle: refactor Makefile
rinx Sep 24, 2020
2d5ab05
:recycle: add TAG variable
rinx Sep 25, 2020
2905eca
:sparkles: add docker buildx
rinx Sep 25, 2020
cadd26b
:recycle: use golang:xxx-buster image as base
rinx Sep 25, 2020
00783bd
:fire: remove some platforms
rinx Sep 25, 2020
84bf1a0
:green_heart: fix CI build
rinx Sep 28, 2020
1510e81
:whale: remove armv7 because of cmake 3.16 bug
rinx Sep 28, 2020
e3d9e7f
:bug: fix extldflags
rinx Oct 1, 2020
b8d9cec
:green_heart: enable build cache
rinx Oct 1, 2020
5c31d0e
:green_heart: add PACKAGE_USER, PACKAGE_TOKEN
rinx Oct 1, 2020
624cfb0
:label: add labels
rinx Oct 1, 2020
5c6d837
:green_heart: use max mode build cache
rinx Oct 1, 2020
f6c35bd
:whale: multi-platform images for agent-sidecar
rinx Oct 1, 2020
98817de
:whale: build NGT before copying .git dir
rinx Oct 1, 2020
3fdb165
:whale: use buildx for base, ci-container, dev-container images
rinx Oct 1, 2020
0cdb3c5
:green_heart: use independent cache for vald-base
rinx Oct 1, 2020
6e6749c
:zap: remove exporting cache layers for vald-base
rinx Oct 1, 2020
a9128de
:whale: build arm64 images for other images
rinx Oct 1, 2020
9c3c508
:whale: remove useless arm images
rinx Oct 2, 2020
1a48cbe
:whale: use BASE_TAG arg
rinx Oct 2, 2020
c751b20
:whale: fix build command for helm-operator
rinx Oct 2, 2020
cd46067
:whale: fix build of dev-container
rinx Oct 2, 2020
45c1faa
:art: build image for amd64 only when PR builds
rinx Oct 2, 2020
9019b6c
:whale: revise PATH
rinx Oct 2, 2020
eb10d45
:whale: revise base golang dirs
rinx Oct 2, 2020
ec8bf00
:green_heart: use --load options for vald-base build
rinx Oct 2, 2020
1122fd9
:green_heart: use --push options for vald-base build
rinx Oct 2, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 79 additions & 32 deletions .github/workflows/dockers-agent-ngt-image.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,64 +42,111 @@ jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: actions/checkout@v2
- name: Setup QEMU
uses: docker/setup-qemu-action@v1
with:
fetch-depth: 10
- name: Overwrite version name
if: github.event_name == 'pull_request'
run: |
pr_num=`cat $GITHUB_EVENT_PATH | jq -r ".number"`
echo "PR-${pr_num}" > versions/VALD_VERSION
- name: Build the Docker image
run: |
make docker/build/agent-ngt
env:
DOCKER_BUILDKIT: 1
- name: login to DockerHub
platforms: all
- name: Setup Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v1
with:
buildkitd-flags: "--debug"
- name: Cache Docker layers (base)
uses: actions/cache@v2
with:
path: /tmp/.buildx-cache-base
key: ${{ runner.os }}-buildx-vald-base-${{ github.sha }}
restore-keys: |
${{ runner.os }}-buildx-vald-base-
- name: Cache Docker layers
uses: actions/cache@v2
with:
path: /tmp/.buildx-cache
key: ${{ runner.os }}-buildx-vald-agent-ngt-${{ github.sha }}
restore-keys: |
${{ runner.os }}-buildx-vald-agent-ngt-
- name: Login to DockerHub
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_PASS }}
- name: push to DockerHub (master)
- name: Login to GitHub Container Registry
uses: docker/login-action@v1
with:
registry: ghcr.io
username: ${{ secrets.PACKAGE_USER }}
password: ${{ secrets.PACKAGE_TOKEN }}
- name: Image name
id: image_name
run: |
image_name=`make docker/name/agent-ngt`
base_platforms=`make docker/platforms`
echo "::set-env name=IMAGE_NAME::${image_name}"
echo "::set-output name=IMAGE_NAME::${image_name}"
echo "::set-output name=BASE_PLATFORMS::${base_platforms}"
- name: Determine tag name (master)
if: github.ref == 'refs/heads/master'
run: |
imagename=`make docker/name/agent-ngt`
docker tag ${imagename} ${imagename}:nightly
docker push ${imagename}:nightly
- name: push to DockerHub (pull request)
echo "::set-env name=PRIMARY_TAG::nightly"
echo "::set-env name=PLATFORMS::${PLATFORMS}"
env:
PLATFORMS: linux/amd64,linux/arm64
- name: Determine tag name (pull request)
if: github.event_name == 'pull_request'
run: |
imagename=`make docker/name/agent-ngt`
pr_num=`cat $GITHUB_EVENT_PATH | jq -r ".number"`
docker tag ${imagename} ${imagename}:pr-${pr_num}
docker push ${imagename}:pr-${pr_num}
- name: push to DockerHub (tags)
echo "PR-${pr_num}" > versions/VALD_VERSION
echo "::set-env name=PRIMARY_TAG::pr-${pr_num}"
echo "::set-env name=PLATFORMS::${PLATFORMS}"
env:
PLATFORMS: linux/amd64
- name: Determine tag name (tags)
if: startsWith( github.ref, 'refs/tags/')
id: push_to_dockerhub_tags
id: determine_tag
run: |
imagename=`make docker/name/agent-ngt`
docker push ${imagename}:latest
tag_name=`echo $GITHUB_REF | sed -e 's:^refs/tags/::'`
docker tag ${imagename} ${imagename}:${tag_name}
docker push ${imagename}:${tag_name}
docker tag ${imagename} ${imagename}:nightly
docker push ${imagename}:nightly
echo "::set-output name=IMAGE_NAME::${imagename}"
echo "::set-output name=TAG_NAME::${tag_name}"
echo "::set-env name=PRIMARY_TAG::${tag_name}"
echo "::set-env name=PLATFORMS::${PLATFORMS}"
env:
PLATFORMS: linux/amd64,linux/arm64
- name: Build and Push
run: |
make \
DOCKER="docker buildx" \
DOCKER_OPTS="--platform ${PLATFORMS} --builder ${BUILDER} ${CACHE_OPTS} ${LABEL_OPTS} --push" \
DOCKER_OPTS_BASE="--platform ${BASE_PLATFORMS} --builder ${BUILDER} ${CACHE_OPTS_BASE} --push" \
TAG="${PRIMARY_TAG}" \
docker/build/agent-ngt
make \
REPO="ghcr.io/vdaas/vald" \
DOCKER="docker buildx" \
DOCKER_OPTS="--platform ${PLATFORMS} --builder ${BUILDER} ${CACHE_OPTS} ${LABEL_OPTS} --push" \
DOCKER_OPTS_BASE="--platform ${BASE_PLATFORMS} --builder ${BUILDER} ${CACHE_OPTS_BASE} --push" \
TAG="${PRIMARY_TAG}" \
docker/build/agent-ngt
env:
DOCKER_BUILDKIT: 1
BASE_PLATFORMS: ${{ steps.image_name.outputs.BASE_PLATFORMS }}
BUILDER: ${{ steps.buildx.outputs.name }}
CACHE_OPTS: "--cache-from=type=local,src=/tmp/.buildx-cache --cache-to=type=local,mode=max,dest=/tmp/.buildx-cache"
CACHE_OPTS_BASE: "--cache-from=type=local,src=/tmp/.buildx-cache-base"
LABEL_OPTS: "--label org.opencontainers.image.url=${{ github.event.repository.html_url }} --label org.opencontainers.image.source=${{ github.event.repository.html_url }} --label org.opencontainers.image.revision=${{ github.sha }}"
- name: Initialize CodeQL
if: startsWith( github.ref, 'refs/tags/')
uses: github/codeql-action/init@v1
- name: Run vulnerability scanner (table)
if: startsWith( github.ref, 'refs/tags/')
uses: aquasecurity/trivy-action@master
with:
image-ref: "${{ steps.push_to_dockerhub_tags.outputs.IMAGE_NAME }}:${{ steps.push_to_dockerhub_tags.outputs.TAG_NAME }}"
image-ref: "${{ steps.image_name.outputs.IMAGE_NAME }}:${{ steps.determine_tag.outputs.TAG_NAME }}"
format: 'table'
- name: Run vulnerability scanner (sarif)
if: startsWith( github.ref, 'refs/tags/')
uses: aquasecurity/trivy-action@master
with:
image-ref: "${{ steps.push_to_dockerhub_tags.outputs.IMAGE_NAME }}:${{ steps.push_to_dockerhub_tags.outputs.TAG_NAME }}"
image-ref: "${{ steps.image_name.outputs.IMAGE_NAME }}:${{ steps.determine_tag.outputs.TAG_NAME }}"
format: 'template'
template: '@/contrib/sarif.tpl'
output: 'trivy-results.sarif'
Expand Down
111 changes: 79 additions & 32 deletions .github/workflows/dockers-agent-sidecar-image.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,64 +42,111 @@ jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: actions/checkout@v2
- name: Setup QEMU
uses: docker/setup-qemu-action@v1
with:
fetch-depth: 10
- name: Overwrite version name
if: github.event_name == 'pull_request'
run: |
pr_num=`cat $GITHUB_EVENT_PATH | jq -r ".number"`
echo "PR-${pr_num}" > versions/VALD_VERSION
- name: Build the Docker image
run: |
make docker/build/agent-sidecar
env:
DOCKER_BUILDKIT: 1
- name: login to DockerHub
platforms: all
- name: Setup Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v1
with:
buildkitd-flags: "--debug"
- name: Cache Docker layers (base)
uses: actions/cache@v2
with:
path: /tmp/.buildx-cache-base
key: ${{ runner.os }}-buildx-vald-base-${{ github.sha }}
restore-keys: |
${{ runner.os }}-buildx-vald-base-
- name: Cache Docker layers
uses: actions/cache@v2
with:
path: /tmp/.buildx-cache
key: ${{ runner.os }}-buildx-vald-agent-sidecar-${{ github.sha }}
restore-keys: |
${{ runner.os }}-buildx-vald-agent-sidecar-
- name: Login to DockerHub
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_PASS }}
- name: push to DockerHub (master)
- name: Login to GitHub Container Registry
uses: docker/login-action@v1
with:
registry: ghcr.io
username: ${{ secrets.PACKAGE_USER }}
password: ${{ secrets.PACKAGE_TOKEN }}
- name: Image name
id: image_name
run: |
image_name=`make docker/name/agent-sidecar`
base_platforms=`make docker/platforms`
echo "::set-env name=IMAGE_NAME::${image_name}"
echo "::set-output name=IMAGE_NAME::${image_name}"
echo "::set-output name=BASE_PLATFORMS::${base_platforms}"
- name: Determine tag name (master)
if: github.ref == 'refs/heads/master'
run: |
imagename=`make docker/name/agent-sidecar`
docker tag ${imagename} ${imagename}:nightly
docker push ${imagename}:nightly
- name: push to DockerHub (pull request)
echo "::set-env name=PRIMARY_TAG::nightly"
echo "::set-env name=PLATFORMS::${PLATFORMS}"
env:
PLATFORMS: linux/amd64,linux/arm64
- name: Determine tag name (pull request)
if: github.event_name == 'pull_request'
run: |
imagename=`make docker/name/agent-sidecar`
pr_num=`cat $GITHUB_EVENT_PATH | jq -r ".number"`
docker tag ${imagename} ${imagename}:pr-${pr_num}
docker push ${imagename}:pr-${pr_num}
- name: push to DockerHub (tags)
echo "PR-${pr_num}" > versions/VALD_VERSION
echo "::set-env name=PRIMARY_TAG::pr-${pr_num}"
echo "::set-env name=PLATFORMS::${PLATFORMS}"
env:
PLATFORMS: linux/amd64
- name: Determine tag name (tags)
if: startsWith( github.ref, 'refs/tags/')
id: push_to_dockerhub_tags
id: determine_tag
run: |
imagename=`make docker/name/agent-sidecar`
docker push ${imagename}:latest
tag_name=`echo $GITHUB_REF | sed -e 's:^refs/tags/::'`
docker tag ${imagename} ${imagename}:${tag_name}
docker push ${imagename}:${tag_name}
docker tag ${imagename} ${imagename}:nightly
docker push ${imagename}:nightly
echo "::set-output name=IMAGE_NAME::${imagename}"
echo "::set-output name=TAG_NAME::${tag_name}"
echo "::set-env name=PRIMARY_TAG::${tag_name}"
echo "::set-env name=PLATFORMS::${PLATFORMS}"
env:
PLATFORMS: linux/amd64,linux/arm64
- name: Build and Push
run: |
make \
DOCKER="docker buildx" \
DOCKER_OPTS="--platform ${PLATFORMS} --builder ${BUILDER} ${CACHE_OPTS} ${LABEL_OPTS} --push" \
DOCKER_OPTS_BASE="--platform ${BASE_PLATFORMS} --builder ${BUILDER} ${CACHE_OPTS_BASE} --push" \
TAG="${PRIMARY_TAG}" \
docker/build/agent-sidecar
make \
REPO="ghcr.io/vdaas/vald" \
DOCKER="docker buildx" \
DOCKER_OPTS="--platform ${PLATFORMS} --builder ${BUILDER} ${CACHE_OPTS} ${LABEL_OPTS} --push" \
DOCKER_OPTS_BASE="--platform ${BASE_PLATFORMS} --builder ${BUILDER} ${CACHE_OPTS_BASE} --push" \
TAG="${PRIMARY_TAG}" \
docker/build/agent-sidecar
env:
DOCKER_BUILDKIT: 1
BASE_PLATFORMS: ${{ steps.image_name.outputs.BASE_PLATFORMS }}
BUILDER: ${{ steps.buildx.outputs.name }}
CACHE_OPTS: "--cache-from=type=local,src=/tmp/.buildx-cache --cache-to=type=local,mode=max,dest=/tmp/.buildx-cache"
CACHE_OPTS_BASE: "--cache-from=type=local,src=/tmp/.buildx-cache-base"
LABEL_OPTS: "--label org.opencontainers.image.url=${{ github.event.repository.html_url }} --label org.opencontainers.image.source=${{ github.event.repository.html_url }} --label org.opencontainers.image.revision=${{ github.sha }}"
- name: Initialize CodeQL
if: startsWith( github.ref, 'refs/tags/')
uses: github/codeql-action/init@v1
- name: Run vulnerability scanner (table)
if: startsWith( github.ref, 'refs/tags/')
uses: aquasecurity/trivy-action@master
with:
image-ref: "${{ steps.push_to_dockerhub_tags.outputs.IMAGE_NAME }}:${{ steps.push_to_dockerhub_tags.outputs.TAG_NAME }}"
image-ref: "${{ steps.image_name.outputs.IMAGE_NAME }}:${{ steps.determine_tag.outputs.TAG_NAME }}"
format: 'table'
- name: Run vulnerability scanner (sarif)
if: startsWith( github.ref, 'refs/tags/')
uses: aquasecurity/trivy-action@master
with:
image-ref: "${{ steps.push_to_dockerhub_tags.outputs.IMAGE_NAME }}:${{ steps.push_to_dockerhub_tags.outputs.TAG_NAME }}"
image-ref: "${{ steps.image_name.outputs.IMAGE_NAME }}:${{ steps.determine_tag.outputs.TAG_NAME }}"
format: 'template'
template: '@/contrib/sarif.tpl'
output: 'trivy-results.sarif'
Expand Down
Loading