Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

build(docker): simplification of assembly to facilitate further support #24504

Merged
merged 1 commit into from
Jun 24, 2023
Merged

build(docker): simplification of assembly to facilitate further support #24504

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/docker.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ jobs:
mkdir -p ./build
echo ${{ github.sha }} > ./build/SHA
echo ${{ github.event.pull_request.number }} > ./build/PR-NUM
docker build --target ci -t ${{ github.sha }} -t "pr-${{ github.event.pull_request.number }}" .
DOCKER_BUILDKIT=1 docker build --target ci -t ${{ github.sha }} -t "pr-${{ github.event.pull_request.number }}" .
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The runner already uses docker-buildx by default. Have a look here. So I don't think it is necessary setting this variable DOCKER_BUILDKIT=1 explicitly. Please correct me if I am wrong.

Copy link
Contributor Author

@alekseyolg alekseyolg Jun 24, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sebastianliebscher Hello!
Originally my code didn't include this, but without it, an error pops up that this variable is not set and a build error occurs.
Apparently this feature is not used by default.

docker save ${{ github.sha }} | gzip > ./build/${{ github.sha }}.tar.gz

- name: Upload build artifacts
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/docker_build_push.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ EOF
#
# Build the "lean" image
#
docker build --target lean \
DOCKER_BUILDKIT=1 docker build --target lean \
-t "${REPO_NAME}:${SHA}" \
-t "${REPO_NAME}:${REFSPEC}" \
-t "${REPO_NAME}:${LATEST_TAG}" \
Expand All @@ -59,7 +59,7 @@ docker build --target lean \
#
# Build the "lean310" image
#
docker build --target lean \
DOCKER_BUILDKIT=1 docker build --target lean \
-t "${REPO_NAME}:${SHA}-py310" \
-t "${REPO_NAME}:${REFSPEC}-py310" \
-t "${REPO_NAME}:${LATEST_TAG}-py310" \
Expand All @@ -73,7 +73,7 @@ docker build --target lean \
#
# Build the "websocket" image
#
docker build \
DOCKER_BUILDKIT=1 docker build \
-t "${REPO_NAME}:${SHA}-websocket" \
-t "${REPO_NAME}:${REFSPEC}-websocket" \
-t "${REPO_NAME}:${LATEST_TAG}-websocket" \
Expand All @@ -86,7 +86,7 @@ docker build \
#
# Build the dev image
#
docker build --target dev \
DOCKER_BUILDKIT=1 docker build --target dev \
-t "${REPO_NAME}:${SHA}-dev" \
-t "${REPO_NAME}:${REFSPEC}-dev" \
-t "${REPO_NAME}:${LATEST_TAG}-dev" \
Expand Down
64 changes: 24 additions & 40 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,17 +29,17 @@ ENV BUILD_CMD=${NPM_BUILD_CMD}
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true

# NPM ci first, as to NOT invalidate previous steps except for when package.json changes
RUN mkdir -p /app/superset-frontend
WORKDIR /app/superset-frontend

COPY ./docker/frontend-mem-nag.sh /
RUN /frontend-mem-nag.sh

WORKDIR /app/superset-frontend/
RUN /frontend-mem-nag.sh

COPY superset-frontend/package*.json ./

RUN npm ci

COPY ./superset-frontend .
COPY ./superset-frontend ./

# This seems to be the most expensive step
RUN npm run ${BUILD_CMD}
Expand All @@ -49,6 +49,7 @@ RUN npm run ${BUILD_CMD}
######################################################################
FROM python:${PY_VER} AS lean

WORKDIR /app
ENV LANG=C.UTF-8 \
LC_ALL=C.UTF-8 \
FLASK_ENV=production \
Expand All @@ -59,8 +60,8 @@ ENV LANG=C.UTF-8 \

RUN mkdir -p ${PYTHONPATH} \
&& useradd --user-group -d ${SUPERSET_HOME} -m --no-log-init --shell /bin/bash superset \
&& apt-get update -y \
&& apt-get install -y --no-install-recommends \
&& apt-get update -q \
&& apt-get install -yq --no-install-recommends \
build-essential \
curl \
default-libmysqlclient-dev \
Expand All @@ -71,11 +72,8 @@ RUN mkdir -p ${PYTHONPATH} \
libldap2-dev \
&& rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY --chown=superset:superset ./requirements/*.txt requirements/
COPY --chown=superset:superset setup.py MANIFEST.in README.md ./

# setup.py uses the version information in package.json
COPY --chown=superset:superset superset-frontend/package.json superset-frontend/

Expand All @@ -84,16 +82,14 @@ RUN mkdir -p superset/static \
&& pip install --no-cache-dir -r requirements/local.txt

COPY --chown=superset:superset --from=superset-node /app/superset/static/assets superset/static/assets

## Lastly, let's install superset itself
COPY --chown=superset:superset superset superset

RUN chown -R superset:superset ./* \
&& pip install --no-cache-dir -e . \
&& flask fab babel-compile --target superset/translations

COPY ./docker/run-server.sh /usr/bin/
RUN chmod a+x /usr/bin/run-server.sh

COPY --chmod=755 ./docker/run-server.sh /usr/bin/
USER superset

HEALTHCHECK CMD curl -f "http://localhost:$SUPERSET_PORT/health"
Expand All @@ -109,47 +105,35 @@ FROM lean AS dev
ARG GECKODRIVER_VERSION=v0.32.0
ARG FIREFOX_VERSION=106.0.3

COPY ./requirements/*.txt ./docker/requirements-*.txt/ /app/requirements/

USER root

RUN apt-get update -y \
&& apt-get install -y --no-install-recommends \
RUN apt-get update -q \
&& apt-get install -yq --no-install-recommends \
libnss3 \
libdbus-glib-1-2 \
libgtk-3-0 \
libx11-xcb1 \
libasound2 \
libxtst6 \
wget

# Install GeckoDriver WebDriver
RUN wget https://github.com/mozilla/geckodriver/releases/download/${GECKODRIVER_VERSION}/geckodriver-${GECKODRIVER_VERSION}-linux64.tar.gz -O /tmp/geckodriver.tar.gz && \
tar xvfz /tmp/geckodriver.tar.gz -C /tmp && \
mv /tmp/geckodriver /usr/local/bin/geckodriver && \
rm /tmp/geckodriver.tar.gz

# Install Firefox
RUN wget https://download-installer.cdn.mozilla.net/pub/firefox/releases/${FIREFOX_VERSION}/linux-x86_64/en-US/firefox-${FIREFOX_VERSION}.tar.bz2 -O /opt/firefox.tar.bz2 && \
tar xvf /opt/firefox.tar.bz2 -C /opt && \
ln -s /opt/firefox/firefox /usr/local/bin/firefox
wget \
# Install GeckoDriver WebDriver
&& wget https://github.com/mozilla/geckodriver/releases/download/${GECKODRIVER_VERSION}/geckodriver-${GECKODRIVER_VERSION}-linux64.tar.gz -O - | tar xfz - -C /usr/local/bin \
# Install Firefox
&& wget https://download-installer.cdn.mozilla.net/pub/firefox/releases/${FIREFOX_VERSION}/linux-x86_64/en-US/firefox-${FIREFOX_VERSION}.tar.bz2 -O - | tar xfj - -C /opt \
&& ln -s /opt/firefox/firefox /usr/local/bin/firefox \
&& apt-get autoremove -yqq --purge wget && rm -rf /var/lib/apt/lists/* && apt-get clean

COPY ./requirements/*.txt ./docker/requirements-*.txt/ /app/requirements/
# Cache everything for dev purposes...
RUN cd /app \
&& pip install --no-cache -r requirements/docker.txt \
&& pip install --no-cache -r requirements/requirements-local.txt || true
USER superset

RUN pip install --no-cache-dir -r /app/requirements/docker.txt \
&& pip install --no-cache-dir -r /app/requirements/requirements-local.txt || true

USER superset
######################################################################
# CI image...
######################################################################
FROM lean AS ci

COPY --chown=superset ./docker/docker-bootstrap.sh /app/docker/
COPY --chown=superset ./docker/docker-init.sh /app/docker/
COPY --chown=superset ./docker/docker-ci.sh /app/docker/

RUN chmod a+x /app/docker/*.sh
COPY --chown=superset --chmod=755 ./docker/*.sh /app/docker/

CMD /app/docker/docker-ci.sh
CMD ["/app/docker/docker-ci.sh"]
38 changes: 23 additions & 15 deletions docker/docker-bootstrap.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,22 +30,30 @@ fi
#
if [ -f "${REQUIREMENTS_LOCAL}" ]; then
echo "Installing local overrides at ${REQUIREMENTS_LOCAL}"
pip install -r "${REQUIREMENTS_LOCAL}"
pip install --no-cache-dir -r "${REQUIREMENTS_LOCAL}"
else
echo "Skipping local overrides"
fi

if [[ "${1}" == "worker" ]]; then
echo "Starting Celery worker..."
celery --app=superset.tasks.celery_app:app worker -O fair -l INFO
elif [[ "${1}" == "beat" ]]; then
echo "Starting Celery beat..."
rm -f /tmp/celerybeat.pid
celery --app=superset.tasks.celery_app:app beat --pidfile /tmp/celerybeat.pid -l INFO -s "${SUPERSET_HOME}"/celerybeat-schedule
elif [[ "${1}" == "app" ]]; then
echo "Starting web app..."
flask run -p 8088 --with-threads --reload --debugger --host=0.0.0.0
elif [[ "${1}" == "app-gunicorn" ]]; then
echo "Starting web app..."
/usr/bin/run-server.sh
fi
case "${1}" in
worker)
echo "Starting Celery worker..."
celery --app=superset.tasks.celery_app:app worker -O fair -l INFO
;;
beat)
echo "Starting Celery beat..."
rm -f /tmp/celerybeat.pid
celery --app=superset.tasks.celery_app:app beat --pidfile /tmp/celerybeat.pid -l INFO -s "${SUPERSET_HOME}"/celerybeat-schedule
;;
app)
echo "Starting web app..."
flask run -p 8088 --with-threads --reload --debugger --host=0.0.0.0
;;
app-gunicorn)
echo "Starting web app..."
/usr/bin/run-server.sh
;;
*)
echo "Unknown Operation!!!"
;;
esac