Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Publish Docker images to docker hub when released #13631

Merged
merged 1 commit into from
Feb 27, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 20 additions & 7 deletions .github/workflows/publish-docker.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,9 @@ on:
push:
branches:
- dev

env:
HUB: ghcr.io/apache/dolphinscheduler
release:
types:
- released

jobs:
build:
Expand All @@ -41,12 +41,25 @@ jobs:
key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-
- name: Set environment variables
run: |
if [[ ${{ github.event_name }} == "release" ]]; then
echo "DOCKER_REGISTRY=docker.io" >> $GITHUB_ENV
echo "DOCKER_USERNAME=${{ secrets.DOCKERHUB_USER }}" >> $GITHUB_ENV
echo "DOCKER_PASSWORD=${{ secrets.DOCKERHUB_TOKEN }}" >> $GITHUB_ENV
echo "HUB=apache" >> $GITHUB_ENV
else
echo "DOCKER_REGISTRY=ghcr.io/apache/dolphinscheduler" >> $GITHUB_ENV
echo "DOCKER_USERNAME=${{ github.actor }}" >> $GITHUB_ENV
echo "DOCKER_PASSWORD=${{ secrets.GITHUB_TOKEN }}" >> $GITHUB_ENV
echo "HUB=ghcr.io/apache/dolphinscheduler" >> $GITHUB_ENV
fi
- name: Log in to the Container registry
uses: docker/login-action@v1.10.0
uses: docker/login-action@v2
with:
registry: ${{ env.HUB }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
registry: ${{ env.DOCKER_REGISTRY }}
username: ${{ env.DOCKER_USERNAME }}
password: ${{ env.DOCKER_PASSWORD }}
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
- name: Set up Docker Buildx
Expand Down
33 changes: 26 additions & 7 deletions .github/workflows/publish-helm-chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@ on:
push:
branches:
- dev
release:
types:
- released

env:
HUB: ghcr.io/apache/dolphinscheduler
Expand All @@ -34,17 +37,33 @@ jobs:
timeout-minutes: 30
steps:
- uses: actions/checkout@v2
- name: Set environment variables
run: |
# TODO
if [[ ${{ github.event_name }} == "release" ]]; then
echo "HUB=registry-1.docker.io/apache" >> $GITHUB_ENV
echo "DOCKER_REGISTRY=docker.io" >> $GITHUB_ENV
echo "DOCKER_USERNAME=${{ secrets.DOCKERHUB_USER }}" >> $GITHUB_ENV
echo "DOCKER_PASSWORD=${{ secrets.DOCKERHUB_TOKEN }}" >> $GITHUB_ENV
else
echo "HUB=ghcr.io/apache/dolphinscheduler" >> $GITHUB_ENV
echo "DOCKER_REGISTRY=c" >> $GITHUB_ENV
echo "DOCKER_USERNAME=${{ github.actor }}" >> $GITHUB_ENV
echo "DOCKER_PASSWORD=${{ secrets.GITHUB_TOKEN }}" >> $GITHUB_ENV
fi
- name: Log in to the Container registry
uses: docker/login-action@v1.10.0
uses: docker/login-action@v2
with:
registry: ${{ env.HUB }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
registry: ${{ env.DOCKER_REGISTRY }}
username: ${{ env.DOCKER_USERNAME }}
password: ${{ env.DOCKER_PASSWORD }}
- name: Publish Helm Chart
working-directory: deploy/kubernetes
run: |
VERSION=0.0.0-$(git rev-parse --short HEAD)
sed -i "s/^version: .*/version: $VERSION/" dolphinscheduler/Chart.yaml
if [[ ${{ env.HUB }} == "ghcr.io/apache/dolphinscheduler" ]]; then
VERSION=0.0.0-$(git rev-parse --short HEAD)
sed -i "s/^version: .*/version: $VERSION/" dolphinscheduler/Chart.yaml
fi
helm dep up dolphinscheduler
helm package dolphinscheduler
helm push dolphinscheduler-helm-$VERSION.tgz oci://${{ env.HUB }}
helm push dolphinscheduler-helm-*.tgz oci://${{ env.HUB }}
55 changes: 6 additions & 49 deletions docs/docs/en/contribute/release.md
Original file line number Diff line number Diff line change
Expand Up @@ -492,56 +492,13 @@ the release version is `<VERSION>`, the following updates are required(note it w
issue template have **Version** selection bottom. So after released we should add the new `<VERSION>` to
bug-report.yml

### Publish Docker Image
### Publish Docker Image and Helm Chart

we already have the exists CI to publish the latest Docker image to GitHub container register with [config](https://github.com/apache/dolphinscheduler/blob/d80cf21456265c9d84e642bdb4db4067c7577fc6/.github/workflows/publish-docker.yaml#L55-L63).
We could reuse the main command the CI run and publish our Docker images to Docker Hub by single command.

It is highly recommended to build and test docker images locally first before push to docker hub

```shell
# Checkout and create to target tag
git checkout -b "${VERSION}" "${VERSION}"

# Build docker images locally
./mvnw -B clean package \
-Dmaven.test.skip \
-Dmaven.javadoc.skip \
-Dmaven.checkstyle.skip \
-Dmaven.deploy.skip \
-Ddocker.tag="${VERSION}" \
-Pdocker,release

# You should test whether the standalone-server images work or not
docker run --name dolphinscheduler-standalone-server -p 12345:12345 -p 25333:25333 -d apache/dolphinscheduler-standalone-server:"${DOLPHINSCHEDULER_VERSION}"
```

> Note: To push to dockerhub, you must have Apache organization permission of dockerhub. If you don’t you need to require
> from Apache infra Jira. You can refer to here to submit an application from [here](https://issues.apache.org/jira/projects/INFRA/issues/INFRA-23314)

After verifying the Docker images works as expected, you need to publish the Docker images by the following command:

```shell
./mvnw -B clean deploy \
-Dmaven.test.skip \
-Dmaven.javadoc.skip \
-Dmaven.checkstyle.skip \
-Dmaven.deploy.skip \
-Ddocker.tag="${VERSION}" \
-Ddocker.hub=apache \
-Pdocker,release
```

## Publish Helm Chart

We will also publish the Helm Chart to Docker Hub so that users don't need to download our source codes just in order
to install DolphinScheduler with Helm, run the following command to publish the Helm Chart to Docker Hub.

```bash
cd deploy/kubernetes
helm package dolphinscheduler
helm push dolphinscheduler-helm-$VERSION.tgz oci://registry-1.docker.io/apache
```
We have a [workflow](../../../../.github/workflows/publish-docker.yaml) to automatically publish Docker images
and a [workflow](../../../../.github/workflows/publish-helm-chart.yaml) to automatically publish Helm Chart to Docker Hub,
after you change the release from "pre-release" to "release", the workflow will be triggered. All you need to do
is to observe the aforementioned workflows, and after they are completed, you can pull the Docker images locally and
verify that they work as expected.

### Send Announcement E-mail Community

Expand Down
51 changes: 4 additions & 47 deletions docs/docs/zh/contribute/release.md
Original file line number Diff line number Diff line change
Expand Up @@ -495,53 +495,10 @@ git push --delete "${GH_REMOTE}" "${VERSION}-prepare"

### 发布 Docker Image

我们已经有了现有的 CI,可以将最新的 Docker image 发布到 GitHub 容器注册 [config](https://github.com/apache/dolphinscheduler/blob/d80cf21456265c9d84e642bdb4db4067c7577fc6/.github/workflows/publish-docker.yaml#L55- L63).
我们可以重用 CI 运行的主要命令,并通过单个命令将 Docker iamge 发布到 DockerHub。

强烈建议在推送到 docker hub 之前先在本地构建和测试 docker 镜像

```shell
# Checkout and create to target tag
git checkout -b "${VERSION}" "${VERSION}"

# Build docker images locally
./mvnw -B clean package \
-Dmaven.test.skip \
-Dmaven.javadoc.skip \
-Dmaven.checkstyle.skip \
-Dmaven.deploy.skip \
-Ddocker.tag="${VERSION}" \
-Pdocker,release

# You should test whether the standalone-server images work or not
docker run --name dolphinscheduler-standalone-server -p 12345:12345 -p 25333:25333 -d apache/dolphinscheduler-standalone-server:"${DOLPHINSCHEDULER_VERSION}"
```

> 注意:推送到 dockerhub,必须有 dockerhub 的 Apache 组织权限。 如果你没有权限,你需要向 Apache infra Jira 申请。 您可以参考
> [此处](https://issues.apache.org/jira/projects/INFRA/issues/INFRA-23314)提交申请

如果您确保 docker 镜像正常工作,您需要通过以下命令将镜像发布到 Docker Hub

```shell
./mvnw -B clean deploy \
-Dmaven.test.skip \
-Dmaven.javadoc.skip \
-Dmaven.checkstyle.skip \
-Dmaven.deploy.skip \
-Ddocker.tag="${VERSION}" \
-Ddocker.hub=apache \
-Pdocker,release
```

## 发布 Helm Chart

我们也会将 Helm Chart 发布到 Docker Hub,这样用户就不需要下载我们的源码就可以使用 Helm 安装 DolphinScheduler,运行以下命令发布 Helm Chart 到 Docker Hub。

```bash
cd deploy/kubernetes
helm package dolphinscheduler
helm push dolphinscheduler-helm-$VERSION.tgz oci://registry-1.docker.io/apache
```
我们有一个 [工作流](../../../../.github/workflows/publish-docker.yaml) 来自动发布 Docker 镜像,
以及一个 [工作流](../../../../.github/workflows/publish-helm-chart.yaml) 来自动发布 Helm Chart 到 Docker Hub。
当你将发版从 "pre-release" 改为 "release" 后,这两个工作流就会被触发。你需要做的就是观察上述的工作流,
当它们完成后,你可以在本地拉取 Docker 镜像并验证它们是否按预期工作。

### 发送公告邮件通知社区

Expand Down