Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: optimize docker build by better utilizing build cache #21583

Merged
merged 2 commits into from
Oct 6, 2022
Merged

build: optimize docker build by better utilizing build cache #21583

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
16b37c7
misc: optimize docker build by better utilizing build cache
pw618 Sep 26, 2022
3d0888e
Update Dockerfile
pwang2 Oct 6, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 23 additions & 41 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,60 +15,34 @@
# limitations under the License.
#

######################################################################
# PY stage that simply does a pip install on our requirements
######################################################################
ARG PY_VER=3.8.13-slim
FROM python:${PY_VER} AS superset-py

RUN mkdir /app \
&& apt-get update -y \
&& apt-get install -y --no-install-recommends \
build-essential \
default-libmysqlclient-dev \
libpq-dev \
libsasl2-dev \
libecpg-dev \
&& rm -rf /var/lib/apt/lists/*

# First, we just wanna install requirements, which will allow us to utilize the cache
# in order to only build if and only if requirements change
COPY ./requirements/*.txt /app/requirements/
COPY setup.py MANIFEST.in README.md /app/
COPY superset-frontend/package.json /app/superset-frontend/
RUN cd /app \
&& mkdir -p superset/static \
&& touch superset/static/version_info.json \
&& pip install --no-cache -r requirements/local.txt


######################################################################
# Node stage to deal with static asset construction
######################################################################
ARG PY_VER=3.8.13-slim
zhaoyongjie marked this conversation as resolved.
Show resolved Hide resolved
FROM node:16-slim AS superset-node

ARG NPM_BUILD_CMD="build"
ENV BUILD_CMD=${NPM_BUILD_CMD}

# NPM ci first, as to NOT invalidate previous steps except for when package.json changes
RUN mkdir -p /app/superset-frontend
RUN mkdir -p /app/superset/assets

COPY ./docker/frontend-mem-nag.sh /
COPY ./superset-frontend /app/superset-frontend
RUN /frontend-mem-nag.sh \
&& cd /app/superset-frontend \
&& npm ci
RUN /frontend-mem-nag.sh

# This seems to be the most expensive step
RUN cd /app/superset-frontend \
&& npm run ${BUILD_CMD} \
&& rm -rf node_modules
WORKDIR /app/superset-frontend/

COPY superset-frontend/package*.json ./
RUN npm ci

COPY ./superset-frontend .

# This seems to be the most expensive step
RUN npm run ${BUILD_CMD}

######################################################################
# Final lean image...
######################################################################
ARG PY_VER=3.8.13-slim
FROM python:${PY_VER} AS lean

ENV LANG=C.UTF-8 \
Expand All @@ -85,16 +59,24 @@ RUN mkdir -p ${PYTHONPATH} \
&& apt-get install -y --no-install-recommends \
build-essential \
default-libmysqlclient-dev \
libsasl2-dev \
libsasl2-modules-gssapi-mit \
libpq-dev \
libecpg-dev \
&& rm -rf /var/lib/apt/lists/*

COPY --from=superset-py /usr/local/lib/python3.8/site-packages/ /usr/local/lib/python3.8/site-packages/
# Copying site-packages doesn't move the CLIs, so let's copy them one by one
COPY --from=superset-py /usr/local/bin/gunicorn /usr/local/bin/celery /usr/local/bin/flask /usr/bin/
COPY ./requirements/*.txt /app/requirements/
COPY setup.py MANIFEST.in README.md /app/

# setup.py uses the version information in package.json
COPY superset-frontend/package.json /app/superset-frontend/
Comment on lines +71 to +72
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO setup.py to read package.json isn't a good idea but exists a long time in Superset, might change this logic in the separated PR.


RUN cd /app \
&& mkdir -p superset/static \
&& touch superset/static/version_info.json \
&& pip install --no-cache -r requirements/local.txt

COPY --from=superset-node /app/superset/static/assets /app/superset/static/assets
COPY --from=superset-node /app/superset-frontend /app/superset-frontend

## Lastly, let's install superset itself
COPY superset /app/superset
Expand Down