Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide dockerx setup for docker builds #1194

Merged
merged 27 commits into from
Oct 22, 2024
Merged
Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
1eb9953
Created buildx_and_push.sh script in ui directory to create multi-pla…
sikehish Oct 17, 2024
5ce6bda
Fixed content related issue in buildx_and_push.sh
sikehish Oct 17, 2024
de3b0b2
Merge branch 'DAGWorks-Inc:main' into main
sikehish Oct 17, 2024
28799a9
buildx_and_push.sh: Added functionality to fecth the latest version f…
sikehish Oct 18, 2024
19d2d98
Merge branch 'main' of https://github.com/sikehish/hamilton
sikehish Oct 18, 2024
7f118a8
buildx_and_push.sh: Added check_buildx_installed
sikehish Oct 18, 2024
7cd0deb
buildx_and_push.sh: Added check_buildx_installed
sikehish Oct 18, 2024
b8586bd
buildx_and_push.sh: Enhanced error handling(checking if jq exists, cu…
sikehish Oct 18, 2024
df8e12e
Adding build args to buildx_and_push.sh
sikehish Oct 19, 2024
f2bf479
buildx_and_push.sh: Changes made to test a new workflow
sikehish Oct 19, 2024
3b90f00
Created a new workflow: hamilton-ui-build-and-push
sikehish Oct 19, 2024
f23ea2a
buildx_and_push.sh: echo statement added to debug
sikehish Oct 19, 2024
4e05999
buildx_and_push.sh: cd'ing to the directory(ui) where the shell scrip…
sikehish Oct 19, 2024
fe32d18
hamilton-ui-build-and-push.yml: Added Docker Hub login step
sikehish Oct 19, 2024
174f453
buildx_and_push.sh: added dagworks dockerhub username; workflow worke…
sikehish Oct 19, 2024
81ea2f0
hamilton-ui-build-and-push.yml: Replaced previous version from cache …
sikehish Oct 20, 2024
1e1a170
buildx_and_push.sh: Changed dockerhub username for testing
sikehish Oct 20, 2024
ed7f5f7
hamilton-ui-build-and-push.yml: Minor change in the docker registry U…
sikehish Oct 20, 2024
53e7c3b
hamilton-ui-build-and-push.yml: Minor change in the "Fetch highest ve…
sikehish Oct 20, 2024
de07d82
hamilton-ui-build-and-push.yml: Replacing deprecated set-output with …
sikehish Oct 20, 2024
c9c7f19
hamilton-ui-build-and-push.yml: Conditional execution of steps implem…
sikehish Oct 20, 2024
a68343d
Undid change in dockerhub username
sikehish Oct 20, 2024
4dc82e5
Update ui/buildx_and_push.sh
skrawcz Oct 22, 2024
2acb408
Update ui/buildx_and_push.sh
skrawcz Oct 22, 2024
624fe02
chore: fix pre-commit whitespace issues
sikehish Oct 22, 2024
68947c7
Merge branch 'main' of https://github.com/sikehish/hamilton
sikehish Oct 22, 2024
7efa0c3
Merge branch 'DAGWorks-Inc:main' into main
sikehish Oct 22, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 83 additions & 0 deletions .github/workflows/hamilton-ui-build-and-push.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
name: Building and pushing UI frontend and backend images

on:
schedule:
- cron: '0 0 * * *'
workflow_dispatch:

jobs:
check_and_build:
runs-on: ubuntu-latest

steps:
- name: Checkout code
uses: actions/checkout@v3

- name: Log in to Docker Hub
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_TOKEN }}

- name: Install jq
run: sudo apt-get update && sudo apt-get install -y jq

- name: Check latest version of sf-hamilton-ui
id: check_version
run: |
response=$(curl -s https://pypi.org/pypi/sf-hamilton-ui/json)
version=$(echo "$response" | jq -r '.info.version')
echo "latest_version=$version" >> $GITHUB_ENV

- name: Fetch highest version from Docker Hub
id: fetch_tags
run: |
# get the list of tags for the frontend image
tags=$(curl -s https://registry.hub.docker.com/v2/repositories/dagworks/ui-frontend/tags?page_size=100 | jq -r '.results[].name')

# find the highest version tag
highest_version="none"
for tag in $tags; do
if [[ "$tag" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
if [ "$highest_version" == "none" ] || [ "$(printf '%s\n' "$tag" "$highest_version" | sort -V | tail -n1)" == "$tag" ]; then
highest_version="$tag"
fi
fi
done

echo "Highest version on Docker Hub: $highest_version"
echo "highest_version=$highest_version" >> $GITHUB_OUTPUT

- name: Compare versions
id: compare_versions
run: |
echo "Current version: $latest_version"
highest_version=${{ steps.fetch_tags.outputs.highest_version }}
echo "Highest version found: $highest_version"

if [[ "$latest_version" == "$highest_version" ]]; then
echo "No new version found. Exiting."
echo "skip_build=true" >> $GITHUB_ENV
else
echo "New version detected: $latest_version"
echo "skip_build=false" >> $GITHUB_ENV
fi

- name: Run build and push script
if: env.skip_build != 'true'
run: |
chmod +x ./ui/buildx_and_push.sh
./ui/buildx_and_push.sh $latest_version

- name: Save the latest version to a file
if: env.skip_build != 'true'
run: |
mkdir -p version_cache
echo "$latest_version" > version_cache/previous_version.txt

- name: Cache the latest version
if: env.skip_build != 'true'
uses: actions/cache@v4
with:
path: ./version_cache
key: hamilton-ui-version-cache
69 changes: 69 additions & 0 deletions ui/buildx_and_push.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
#!/bin/bash

export DOCKER_CLI_EXPERIMENTAL=enabled

# check if Docker Buildx is installed
check_buildx_installed() {
if ! docker buildx version &> /dev/null; then
echo "Error: Docker Buildx is not installed. Please install Docker Buildx to proceed."
exit 1
fi
}

# check if jq is installed
check_jq_installed() {
if ! jq --version &> /dev/null; then
echo "Error: jq is not installed. Please install jq to proceed."
exit 1
fi
}

# fetches the latest version of sf-hamilton-ui from PyPI
get_latest_version() {
response=$(curl -s https://pypi.org/pypi/sf-hamilton-ui/json)

# check if curl succeeded and the response is not empty
if [ $? -ne 0 ] || [ -z "$response" ]; then
echo "Error: Failed to fetch data from PyPI. Please check your internet connection or the URL."
exit 1
fi

# extract version using jq and handle potential errors
version=$(echo "$response" | jq -r '.info.version')
if [ "$version" == "null" ]; then
echo "Error: Unable to extract version from the response."
exit 1
fi

echo "$version"
}

# check if Docker Buildx and jq are installed
check_buildx_installed
check_jq_installed

VERSION=$(get_latest_version)

echo "Using sf-hamilton-ui version: $VERSION"

# check if Buildx is already enabled; create a builder instance if not
docker buildx inspect hamilton-builder > /dev/null 2>&1 || \
sikehish marked this conversation as resolved.
Show resolved Hide resolved
docker buildx create --use --name hamilton-builder

FRONTEND_IMAGE="dagworks/ui-frontend"
BACKEND_IMAGE="dagworks/ui-backend"

# define common platforms/architectures
PLATFORMS="linux/amd64,linux/arm64"

cd "$(dirname "$0")" # cd into the directory where this script is present(i.e. ui)

docker buildx build --platform $PLATFORMS \
-t $BACKEND_IMAGE:$VERSION -t $BACKEND_IMAGE:latest \
--push -f backend/Dockerfile.backend backend/
skrawcz marked this conversation as resolved.
Show resolved Hide resolved

docker buildx build --platform $PLATFORMS \
-t $FRONTEND_IMAGE:$VERSION -t $FRONTEND_IMAGE:latest \
--push -f frontend/Dockerfile.frontend frontend/ \
skrawcz marked this conversation as resolved.
Show resolved Hide resolved
--build-arg REACT_APP_AUTH_MODE=local \
--build-arg REACT_APP_USE_POSTHOG=false \