Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds Dockerfile to build cross-platform images for binaries used by the certification test harness. (Take 2) #21140

Merged
merged 7 commits into from
Jul 27, 2022
Merged

Adds Dockerfile to build cross-platform images for binaries used by the certification test harness. (Take 2) #21140

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
dba81e1
Add Dockerfile for building multiplatform certification binaries.
davidgoogle Jul 20, 2022
5d44171
Updates and documentation for chip-cert-bins image.
davidgoogle Jul 23, 2022
b144416
Adds all-clusters-minimal and other updates to chip-cert-bins.
davidgoogle Jul 26, 2022
0abaf3f
Restyled by prettier-markdown
restyled-commits Jul 26, 2022
fdb5626
Fix spell-checking issues
mikaelhm Jul 26, 2022
8cb15c0
Merge pull request #17 from mikaelhm/patch-1
davidgoogle Jul 26, 2022
4f39133
Update .wordlist.txt (#18)
mikaelhm Jul 26, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion integrations/docker/images/build-all.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
# https://github.com/project-chip/connectedhomeip/issues/710
#
set -e
find "$(git rev-parse --show-toplevel)"/integrations/docker/images/ -name Dockerfile | while read -r dockerfile; do
find "$(git rev-parse --show-toplevel)"/integrations/docker/images/ -name Dockerfile ! -path "*chip-cert-bins/*" | while read -r dockerfile; do
pushd "$(dirname "$dockerfile")" >/dev/null
./build.sh "$@"
popd >/dev/null
Expand Down
273 changes: 273 additions & 0 deletions integrations/docker/images/chip-cert-bins/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,273 @@
# Stage 1: Setup dependencies (based on chip-build).
FROM ubuntu:focal as chip-build-cert
davidgoogle marked this conversation as resolved.
Show resolved Hide resolved
ARG TARGETPLATFORM
# BRANCH defines the target branch or tag to build from.
ARG BRANCH=sve
VOLUME "/var/source"
davidgoogle marked this conversation as resolved.
Show resolved Hide resolved

# Ensure TARGETPLATFORM is set
RUN case ${TARGETPLATFORM} in \
"linux/amd64") \
echo "Building for linux/amd64" \
;; \
"linux/arm64") \
echo "Building for linux/arm64" \
;; \
*) \
if [ -z "$TARGETPLATFORM" ] ;\
then \
echo "TARGETPLATFORM not defined! Please run from buildkit (buildx)." \
&& return 1 ;\
else \
echo "Unsupported platform ${TARGETPLATFORM}." \
&& return 1 ;\
fi \
;; \
esac

# Below should be the same as chip-build except arm64 logic for cmake and node.

# base build and check tools and libraries layer
RUN set -x \
&& apt-get update \
&& DEBIAN_FRONTEND=noninteractive apt-get install -fy \
autoconf \
automake \
bison \
bridge-utils \
clang \
clang-format \
clang-tidy \
curl \
flex \
g++ \
git \
gperf \
iproute2 \
jq \
lcov \
libavahi-client-dev \
libavahi-common-dev \
libcairo2-dev \
libdbus-1-dev \
libdbus-glib-1-dev \
libgif-dev \
libglib2.0-dev \
libical-dev \
libjpeg-dev \
libdmalloc-dev \
libmbedtls-dev \
libncurses5-dev \
libncursesw5-dev \
libnspr4-dev \
libpango1.0-dev \
libpixman-1-dev \
libreadline-dev \
libssl-dev \
libtool \
libudev-dev \
libusb-1.0-0 \
libusb-dev \
libxml2-dev \
make \
net-tools \
ninja-build \
openjdk-8-jdk \
pkg-config \
python3.9 \
python3.9-dev \
python3.9-venv \
rsync \
shellcheck \
strace \
systemd \
udev \
unzip \
wget \
git-lfs \
zlib1g-dev \
&& rm -rf /var/lib/apt/lists/ \
&& git lfs install \
&& : # last line

# Cmake (Mbed OS requires >=3.19.0-rc3 version which is not available in Ubuntu 20.04 repository)
RUN case ${TARGETPLATFORM} in \
"linux/amd64") \
set -x \
&& (cd /tmp \
&& wget --progress=dot:giga https://github.com/Kitware/CMake/releases/download/v3.19.3/cmake-3.19.3-Linux-x86_64.sh \
&& sh cmake-3.19.3-Linux-x86_64.sh --exclude-subdir --prefix=/usr/local \
&& rm -rf cmake-3.19.3-Linux-x86_64.sh) \
&& exec bash \
;; \
"linux/arm64") \
set -x \
&& (cd /tmp \
&& wget --progress=dot:giga https://github.com/Kitware/CMake/releases/download/v3.19.3/cmake-3.19.3-Linux-aarch64.sh \
&& sh cmake-3.19.3-Linux-aarch64.sh --exclude-subdir --prefix=/usr/local \
&& rm -rf cmake-3.19.3-Linux-aarch64.sh) \
&& exec bash \
;; \
*) \
test -n "$TARGETPLATFORM" \
echo "Unsupported platform ${TARGETPLATFORM}" \
;; \
esac

# Python 3.9 and PIP
RUN set -x \
&& DEBIAN_FRONTEND=noninteractive apt-get update \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y libgirepository1.0-dev \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y software-properties-common \
&& add-apt-repository universe \
&& curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py \
&& python3.9 get-pip.py \
&& update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.9 1 \
&& rm -rf /var/lib/apt/lists/ \
&& : # last line

RUN set -x \
&& pip3 install attrs coloredlogs PyGithub pygit future portpicker mobly click cxxfilt ghapi pandas tabulate \
&& : # last line

# build and install gn
RUN set -x \
&& git clone https://gn.googlesource.com/gn \
&& cd gn \
&& python3 build/gen.py \
&& ninja -C out \
&& cp out/gn /usr/local/bin \
&& cd .. \
&& rm -rf gn \
&& : # last line

# Install bloat comparison tools
RUN set -x \
&& git clone https://github.com/google/bloaty.git \
&& mkdir -p bloaty/build \
&& cd bloaty/build \
&& cmake ../ \
&& make -j8 \
&& make install \
&& cd ../.. \
&& rm -rf bloaty \
&& : # last line

# NodeJS: install a newer version than what apt-get would read
# This installs the latest LTS version of nodejs
RUN case ${TARGETPLATFORM} in \
"linux/amd64") \
set -x \
&& mkdir node_js \
&& cd node_js \
&& wget https://nodejs.org/dist/v12.19.0/node-v12.19.0-linux-x64.tar.xz \
&& tar xfvJ node-v12.19.0-linux-x64.tar.xz \
&& mv node-v12.19.0-linux-x64 /opt/ \
&& ln -s /opt/node-v12.19.0-linux-x64 /opt/node \
&& ln -s /opt/node/bin/* /usr/bin \
&& cd .. \
&& rm -rf node_js \
;; \
"linux/arm64")\
set -x \
&& mkdir node_js \
&& cd node_js \
&& wget https://nodejs.org/dist/v12.19.0/node-v12.19.0-linux-arm64.tar.xz \
&& tar xfvJ node-v12.19.0-linux-arm64.tar.xz \
&& mv node-v12.19.0-linux-arm64 /opt/ \
&& ln -s /opt/node-v12.19.0-linux-arm64 /opt/node \
&& ln -s /opt/node/bin/* /usr/bin \
&& cd .. \
&& rm -rf node_js \
;; \
*) ;; \
esac

# Stage 1.5: Bootstrap Matter.
# Set python to python3
RUN update-alternatives --install /usr/bin/python python /usr/bin/python3 10

RUN mkdir /root/connectedhomeip
RUN git clone --depth 1 --single-branch --branch ${BRANCH} https://github.com/project-chip/connectedhomeip.git /root/connectedhomeip
davidgoogle marked this conversation as resolved.
Show resolved Hide resolved
WORKDIR /root/connectedhomeip/
RUN scripts/build/gn_bootstrap.sh
RUN gn gen out/debug --args='chip_mdns="platform" chip_inet_config_enable_ipv4=false'
RUN ninja -C out/debug

# Stage 2: Build.
from chip-build-cert as chip-build-cert-bins
SHELL ["/bin/bash", "-c"]
# Records Matter SDK commit hash to include in the image.
RUN git rev-parse HEAD > /root/.sdk-sha-version
RUN case ${TARGETPLATFORM} in \
"linux/amd64") \
set -x \
&& source scripts/activate.sh \
&& scripts/build/build_examples.py --target linux-x64-all-clusters-ipv6only build \
&& mv out/linux-x64-all-clusters-ipv6only/chip-all-clusters-app out/chip-all-clusters-app \
&& scripts/build/build_examples.py --target linux-x64-bridge-ipv6only build \
davidgoogle marked this conversation as resolved.
Show resolved Hide resolved
&& mv out/linux-x64-bridge-ipv6only/chip-bridge-app out/chip-bridge-app \
&& scripts/build/build_examples.py --target linux-x64-tv-app-ipv6only build \
&& mv out/linux-x64-tv-app-ipv6only/chip-tv-app out/chip-tv-app \
&& scripts/build/build_examples.py --target linux-x64-tv-casting-app-ipv6only build \
&& mv out/linux-x64-tv-casting-app-ipv6only/chip-tv-casting-app out/chip-tv-casting-app \
&& scripts/build/build_examples.py --target linux-x64-light-ipv6only build \
&& mv out/linux-x64-light-ipv6only/chip-lighting-app out/chip-lighting-app \
&& scripts/build/build_examples.py --target linux-x64-thermostat-ipv6only build \
&& mv out/linux-x64-thermostat-ipv6only/thermostat-app out/thermostat-app \
&& scripts/build/build_examples.py --target linux-x64-ota-provider-ipv6only build \
&& mv out/linux-x64-ota-provider-ipv6only/chip-ota-provider-app out/chip-ota-provider-app \
&& scripts/build/build_examples.py --target linux-x64-ota-requestor-ipv6only build \
&& mv out/linux-x64-ota-requestor-ipv6only/chip-ota-requestor-app out/chip-ota-requestor-app \
&& scripts/build/build_examples.py --target linux-x64-lock-ipv6only build \
&& mv out/linux-x64-lock-ipv6only/chip-lock-app out/chip-lock-app \
;; \
"linux/arm64")\
set -x \
&& source scripts/activate.sh \
&& scripts/build/build_examples.py --target linux-arm64-all-clusters-ipv6only build \
&& mv out/linux-arm64-all-clusters-ipv6only/chip-all-clusters-app out/chip-all-clusters-app \
&& scripts/build/build_examples.py --target linux-arm64-bridge-ipv6only build \
&& mv out/linux-arm64-bridge-ipv6only/chip-bridge-app out/chip-bridge-app \
&& scripts/build/build_examples.py --target linux-arm64-tv-app-ipv6only build \
&& mv out/linux-arm64-tv-app-ipv6only/chip-tv-app out/chip-tv-app \
&& scripts/build/build_examples.py --target linux-arm64-tv-casting-app-ipv6only build \
&& mv out/linux-arm64-tv-casting-app-ipv6only/chip-tv-casting-app out/chip-tv-casting-app \
&& scripts/build/build_examples.py --target linux-arm64-light-ipv6only build \
&& mv out/linux-arm64-light-ipv6only/chip-lighting-app out/chip-lighting-app \
&& scripts/build/build_examples.py --target linux-arm64-thermostat-ipv6only build \
&& mv out/linux-arm64-thermostat-ipv6only/thermostat-app out/thermostat-app \
&& scripts/build/build_examples.py --target linux-arm64-ota-provider-ipv6only build \
&& mv out/linux-arm64-ota-provider-ipv6only/chip-ota-provider-app out/chip-ota-provider-app \
&& scripts/build/build_examples.py --target linux-arm64-ota-requestor-ipv6only build \
&& mv out/linux-arm64-ota-requestor-ipv6only/chip-ota-requestor-app out/chip-ota-requestor-app \
&& scripts/build/build_examples.py --target linux-arm64-lock-ipv6only build \
&& mv out/linux-arm64-lock-ipv6only/chip-lock-app out/chip-lock-app \
;; \
*) ;; \
esac

RUN npm --prefix third_party/zap/repo/ ci
RUN scripts/examples/gn_build_test_example.sh app1

# Stage 3: Copy relevant cert bins to a minimal image to reduce size.
FROM ubuntu:21.10
ENV TZ=Etc/UTC
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
RUN apt-get update -y
RUN apt-get install -y libssl-dev libdbus-1-dev libglib2.0-dev libavahi-client-dev avahi-utils iproute2
WORKDIR /root/
COPY --from=chip-build-cert-bins /root/.sdk-sha-version .sdk-sha-version
COPY --from=chip-build-cert-bins /root/connectedhomeip/out/debug/chip-tool chip-tool
COPY --from=chip-build-cert-bins /root/connectedhomeip/out/debug/chip-shell chip-shell
COPY --from=chip-build-cert-bins /root/connectedhomeip/out/debug/chip-cert chip-cert
COPY --from=chip-build-cert-bins /root/connectedhomeip/out/chip-all-clusters-app chip-all-clusters-app
COPY --from=chip-build-cert-bins /root/connectedhomeip/out/chip-lighting-app chip-lighting-app
COPY --from=chip-build-cert-bins /root/connectedhomeip/out/chip-tv-casting-app chip-tv-casting-app
COPY --from=chip-build-cert-bins /root/connectedhomeip/out/chip-tv-app chip-tv-app
COPY --from=chip-build-cert-bins /root/connectedhomeip/out/chip-bridge-app chip-bridge-app
COPY --from=chip-build-cert-bins /root/connectedhomeip/out/thermostat-app thermostat-app
COPY --from=chip-build-cert-bins /root/connectedhomeip/out/chip-ota-provider-app chip-ota-provider-app
COPY --from=chip-build-cert-bins /root/connectedhomeip/out/chip-ota-requestor-app chip-ota-requestor-app
COPY --from=chip-build-cert-bins /root/connectedhomeip/out/chip-lock-app chip-lock-app
COPY --from=chip-build-cert-bins /root/connectedhomeip/out/app1/chip-app1 chip-app1
davidgoogle marked this conversation as resolved.
Show resolved Hide resolved
55 changes: 55 additions & 0 deletions integrations/docker/images/chip-cert-bins/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Docker image for Matter Certification Test Harness

The Dockerfile here helps build multi-platform Docker images containing the executable binaries necessary for the Matter Test Harness. It utilizes the BuildKit toolkit and Buildx, included within Docker since version 18.06.

## Running

In order to properly run the binaries, avahi must be properly set up and passed to the container.

Prerequisites:

* Host must support and enable IPv6 and be on a network that has IPv6.
* IPv6 must be enabled within avahi config on the host. `use-ipv6=yes` in avahi-daemon.conf
* Sometimes there are stale avahi entries, so restarting avahi-daemon between runs may be necessary.

The host network and dbus must be exposed to the container for avahi to work properly. So for an interactive prompt, use:

```
docker run -it --network host -v /var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket chip-cert-bins
```

## Building

The Dockerfile requires building using the Buildx plugin, included within docker. It is used to build for both the amd64 and arm64 architectures, so the image may be cross-built and ran directly on a Raspberry Pi or other arm64 based environment. If your docker installation does not have the Buildx plugin, please update docker or install Buildx manually.

Prerequisites:

* A recent docker installation.
* Create a buildx builder: `docker buildx create --use --name mybuild`
* Install the Binfmt cross-platform Docker emulators: `docker run --privileged --rm tonistiigi/binfmt --install all`

### Example: Building for the host platform and loading into Docker

```
docker buildx build --load .
```

The above command will build the image and load them into your local Docker instance.

### Example: Building for another platform and exporting to a tar

```
docker buildx build --platform linux/arm64 --output "dest=/full/path/to/dest/chipcertbins.tar,type=docker" .
```

The above command will build the image and export it to a tar file. You may copy the tar file to a RaspberryPi and import the image by using:

```
docker load --input chipcertbins.tar
```

### Example: Creating a multi-platform image and pushing to the Docker registry

```
docker buildx build --platform linux/amd64,linux/arm64 --tag chip-cert-bins:tag1 --push .
```