Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ported Marvell armhf build on amd64 host for debian buster to use cross-comp… #8035

Merged
merged 37 commits into from
Jul 21, 2022
Merged

Ported Marvell armhf build on amd64 host for debian buster to use cross-comp… #8035

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
b8e21f6
Ported Marvell armhf build on x86 for debian buster to use cross-comp…
Jun 30, 2021
c090db1
Fixed final Sonic image build with dockers inside
Aug 15, 2021
0575054
Update Dockerfile.j2
gregshpit Aug 26, 2021
eba55b6
Update cross-build-arm-python-reqirements.sh
gregshpit Sep 12, 2021
1acb980
Update Makefile
gregshpit Sep 14, 2021
3da08d9
Reviewer's @qiluo-msft requests done
Nov 17, 2021
fcb2b8b
Added new radius/pam patch for arm64 support
Nov 21, 2021
1e079dc
Update slave.mk
gregshpit Nov 21, 2021
c66756d
Added libgtest-dev: libgmock-dev: to the buster Dockerfile.j2. Fixed …
Dec 2, 2021
e153741
Merged with the upstream
Dec 2, 2021
a8e69e5
Added missing armhf/arm64 entries in /etc/apt/sources.list
Dec 6, 2021
b4376b2
Merge remote-tracking branch 'upstream/master'
Dec 9, 2021
6a912bb
fix libc-bin core dump issue from xumia:fix-libc-bin-install-issue co…
Dec 16, 2021
3e32281
Merge branch 'master' into master
gregshpit Dec 16, 2021
a5c6ba3
Removed unnecessary 'apt-get update' from sonic-slave-buster/Dockerfi…
Dec 19, 2021
cedf62c
Fixed saiarcot895 reviewer's requests
Mar 7, 2022
3ef6968
Fixed README and replaced 'sed/awk' with patches
Mar 17, 2022
4f69553
Fixed ntp build to use openssl
Mar 27, 2022
80446dd
Unuse sonic-slave-buster/cross-build-arm-python-reqirements.sh script…
Apr 5, 2022
c095ecd
Merged with the upstream
Apr 24, 2022
e44472e
Clean armhf cross-compilation build fixes
May 3, 2022
bdc6c47
Ported cross-compilation armhf build to bullseye
Jun 16, 2022
6d37a00
Additional change for bullseye
Jun 21, 2022
c38b5d0
Merged with Azure upstream
Jun 21, 2022
d10c812
Set CROSS_BUILD_ENVIRON default value n
Jun 21, 2022
db50dbf
Removed python2 references
Jun 22, 2022
ec639e8
Fixes after merge with the upstream
Jun 26, 2022
9d7ab8c
Deleted unused sonic-slave-buster/cross-build-arm-python-reqirements.…
Jun 27, 2022
544993d
Fixed 2 @saiarcot895 requests
Jun 29, 2022
e7d1fa5
Fixed @saiarcot895 reviewer's requests
Jun 30, 2022
844a2d9
Removed use of prebuilt python wheels
Jul 6, 2022
ca07e21
Merge branch 'master' into master
gregshpit Jul 6, 2022
5290e07
Incorporated saiarcot895 CC/CXX and other simplification/generalizati…
Jul 13, 2022
ca03d67
Fixed saiarcot895 reviewer's additional requests
Jul 18, 2022
f6ef895
merge with the upstream
Jul 18, 2022
26f479e
src/libyang/patch/debian-packaging-files.patch
Jul 18, 2022
c20e081
Removed --no-deps option when installing wheels. Removed unnecessary …
Jul 20, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions Makefile.work
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,9 +128,14 @@ SLAVE_BASE_IMAGE = $(SLAVE_DIR)
MULTIARCH_QEMU_ENVIRON = n
else ifneq ($(CONFIGURED_ARCH),)
SLAVE_BASE_IMAGE = $(SLAVE_DIR)-march-$(CONFIGURED_ARCH)
saiarcot895 marked this conversation as resolved.
Show resolved Hide resolved
ifneq ($(CROSS_BLDENV),)
MULTIARCH_QEMU_ENVIRON = n
CROSS_BUILD_ENVIRON = y
Copy link
Collaborator

@qiluo-msft qiluo-msft Sep 27, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CROSS_BUILD_ENVIRON

The 2 variable names are confusing: CROSS_BUILD_ENVIRON vs CROSS_BLDENV. What is the difference? Can we use one? #Closed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CROSS_BLDENV is used only on the building/configuring command line to trigger cross-compilation build instead of the default qemu build. Like this:
NOJESSIE=1 NOSTRETCH=1 BLDENV=buster CROSS_BLDENV=1 make target/sonic-marvell-armhf.bin

CROSS_BUILD_ENVIRON is internal variabled used everywhere in the Makefiles.

Please see the PR description above for more info.

else
MULTIARCH_QEMU_ENVIRON = y
endif
endif
endif
SLAVE_IMAGE = $(SLAVE_BASE_IMAGE)-$(USER_LC)

# Generate the version control build info
Expand All @@ -139,8 +144,8 @@ $(shell SONIC_VERSION_CONTROL_COMPONENTS=$(SONIC_VERSION_CONTROL_COMPONENTS) \
scripts/generate_buildinfo_config.sh)

# Generate the slave Dockerfile, and prepare build info for it
$(shell CONFIGURED_ARCH=$(CONFIGURED_ARCH) MULTIARCH_QEMU_ENVIRON=$(MULTIARCH_QEMU_ENVIRON) DOCKER_EXTRA_OPTS=$(DOCKER_EXTRA_OPTS) j2 $(SLAVE_DIR)/Dockerfile.j2 > $(SLAVE_DIR)/Dockerfile)
$(shell CONFIGURED_ARCH=$(CONFIGURED_ARCH) MULTIARCH_QEMU_ENVIRON=$(MULTIARCH_QEMU_ENVIRON) j2 $(SLAVE_DIR)/Dockerfile.user.j2 > $(SLAVE_DIR)/Dockerfile.user)
$(shell CONFIGURED_ARCH=$(CONFIGURED_ARCH) MULTIARCH_QEMU_ENVIRON=$(MULTIARCH_QEMU_ENVIRON) CROSS_BUILD_ENVIRON=$(CROSS_BUILD_ENVIRON) DOCKER_EXTRA_OPTS=$(DOCKER_EXTRA_OPTS) j2 $(SLAVE_DIR)/Dockerfile.j2 > $(SLAVE_DIR)/Dockerfile)
$(shell CONFIGURED_ARCH=$(CONFIGURED_ARCH) MULTIARCH_QEMU_ENVIRON=$(MULTIARCH_QEMU_ENVIRON) CROSS_BUILD_ENVIRON=$(CROSS_BUILD_ENVIRON) j2 $(SLAVE_DIR)/Dockerfile.user.j2 > $(SLAVE_DIR)/Dockerfile.user)
$(shell BUILD_SLAVE=y scripts/prepare_docker_buildinfo.sh $(SLAVE_BASE_IMAGE) $(SLAVE_DIR)/Dockerfile $(CONFIGURED_ARCH) "" $(BLDENV))

# Add the versions in the tag, if the version change, need to rebuild the slave
Expand Down Expand Up @@ -188,7 +193,7 @@ ifeq ($(SONIC_CONFIG_USE_NATIVE_DOCKERD_FOR_BUILD), y)
DOCKER_RUN += -v /var/run/docker.sock:/var/run/docker.sock
endif

ifeq ($(MULTIARCH_QEMU_ENVIRON), y)
ifneq ($(filter y, $(MULTIARCH_QEMU_ENVIRON) $(CROSS_BUILD_ENVIRON)),)
Copy link
Contributor

@vivekrnv vivekrnv Sep 30, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @gregshpit,

Why should the slave container be using /var/run/march/docker.sock, /var/run/march/docker.pid instead of /var/run/docker.sock & /var/run/docker.pid as it seems you were pulling the slave docker FROM debian:buster as opposed to the qemu compilation which uses FROM multiarch/debian-debootstrap:arm64-buster

My build server (amd64), which i've traditionally used to compile amd64 targets doesn't have /var/run/march/docker.sock & /var/run/march/docker.pid and is failing when trying to build any DOCKER_TARGETS using cross compilation.

But it is working, when i'm restoring this condition to the original i.e. using the /var/run/docker.sock & /var/run/docker.pid even when doing the cross compilation

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/var/run/march/docker.sock docker daemon should start automatically. But there is a bug in Makefile.work. To make the automation work in cross-compilation mode please replace line 297 in Makefile.work:
-ifeq ($(MULTIARCH_QEMU_ENVIRON), y)

with
+ifneq ($(filter y, $(MULTIARCH_QEMU_ENVIRON) $(CROSS_BUILD_ENVIRON)),)

This should start the march docker daemon automatically.
Let me know if it helped.

Copy link
Contributor

@vivekrnv vivekrnv Oct 3, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my case though, it didn't work with the new condition. i.e using this command:

docker run --rm=true --privileged --init -v <build_dir>/sonic-buildimage:/sonic -w /sonic -e http_proxy= -e https_proxy= -e no_proxy= -it -v /var/cache/sonic/artifacts:/dpkg_cache:rw -v /var/run/march/docker.sock:/var/run/docker.sock -v /var/run/march/docker.pid:/var/run/docker.pid -v /var/run/march/docker:/var/run/docker -v /var/lib/march/docker:/var/lib/docker

probably because i didn't have the /var/run/march/docker.sock setup. I've raised a query in the community. if you have some pointers on this, please do share.

#8898

ifeq ($(DOCKER_DATA_ROOT_FOR_MULTIARCH),)
DOCKER_DATA_ROOT_FOR_MULTIARCH := /var/lib/march/docker
endif
Expand Down Expand Up @@ -249,6 +254,7 @@ SONIC_BUILD_INSTRUCTION := make \
PLATFORM=$(PLATFORM) \
PLATFORM_ARCH=$(PLATFORM_ARCH) \
MULTIARCH_QEMU_ENVIRON=$(MULTIARCH_QEMU_ENVIRON) \
CROSS_BUILD_ENVIRON=$(CROSS_BUILD_ENVIRON) \
BUILD_NUMBER=$(BUILD_NUMBER) \
BUILD_TIMESTAMP=$(BUILD_TIMESTAMP) \
SONIC_IMAGE_VERSION=$(SONIC_IMAGE_VERSION) \
Expand Down Expand Up @@ -294,7 +300,7 @@ SONIC_BUILD_INSTRUCTION := make \
.DEFAULT_GOAL := all

%::
ifeq ($(MULTIARCH_QEMU_ENVIRON), y)
ifneq ($(filter y, $(MULTIARCH_QEMU_ENVIRON) $(CROSS_BUILD_ENVIRON)),)
@$(DOCKER_MULTIARCH_CHECK)
ifneq ($(BLDENV), )
@$(DOCKER_SERVICE_MULTIARCH_CHECK)
Expand Down
12 changes: 12 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,18 @@ To build Arm32 bit for (ARMHF) platform

make target/sonic-marvell-armhf.bin

To build Arm32 bit for (ARMHF) Marvell platform on amd64 host for debian buster using cross-compilation run the
following commands:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick: can these words be on the same line as a previous line?


# Execute make configure once to configure ASIC and ARCH for cross-compilation build

NOJESSIE=1 NOSTRETCH=1 BLDENV=buster CROSS_BLDENV=1 make configure PLATFORM=marvell-armhf PLATFORM_ARCH=armhf

# Execute Arm32 build using cross-compilation environment

NOJESSIE=1 NOSTRETCH=1 BLDENV=buster CROSS_BLDENV=1 make target/sonic-marvell-armhf.bin
Copy link
Contributor

@saiarcot895 saiarcot895 Jun 30, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be updated for Bullseye, I think?


Running the above Arm32 build using cross-compilation instead of qemu emulator drastically reduces the build time.


To build Arm64 bit for platform
Expand Down
6 changes: 5 additions & 1 deletion build_debian.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,10 @@ sudo LANG=C chroot $FILESYSTEM_ROOT apt-get -y upgrade
echo '[INFO] Install packages for building image'
sudo LANG=C chroot $FILESYSTEM_ROOT apt-get -y install makedev psmisc

if [[ $CROSS_BUILD_ENVIRON == y ]]; then
sudo LANG=C chroot $FILESYSTEM_ROOT dpkg --add-architecture $CONFIGURED_ARCH
fi

## Create device files
echo '[INFO] MAKEDEV'
if [[ $CONFIGURED_ARCH == armhf || $CONFIGURED_ARCH == arm64 ]]; then
Expand Down Expand Up @@ -592,7 +596,7 @@ fi
# ALERT: This bit of logic tears down the qemu based build environment used to
# perform builds for the ARM architecture. This must be the last step in this
# script before creating the Sonic installer payload zip file.
if [ $MULTIARCH_QEMU_ENVIRON == y ]; then
if [[ $MULTIARCH_QEMU_ENVIRON == y || $CROSS_BUILD_ENVIRON == y ]]; then
# Remove qemu arm bin executable used for cross-building
sudo rm -f $FILESYSTEM_ROOT/usr/bin/qemu*static || true
DOCKERFS_PATH=../dockerfs/
Expand Down
4 changes: 2 additions & 2 deletions dockers/docker-base-buster/Dockerfile.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{% from "dockers/dockerfile-macros.j2" import install_debian_packages, install_python_wheels, copy_files %}
{% if CONFIGURED_ARCH == "armhf" and MULTIARCH_QEMU_ENVIRON == "y" %}
{% if CONFIGURED_ARCH == "armhf" and (MULTIARCH_QEMU_ENVIRON == "y" or CROSS_BUILD_ENVIRON == "y") %}
FROM multiarch/debian-debootstrap:armhf-buster
{% elif CONFIGURED_ARCH == "arm64" and MULTIARCH_QEMU_ENVIRON == "y" %}
{% elif CONFIGURED_ARCH == "arm64" and (MULTIARCH_QEMU_ENVIRON == "y" or CROSS_BUILD_ENVIRON == "y") %}
FROM multiarch/debian-debootstrap:arm64-buster
{% else %}
FROM {{DOCKER_BASE_ARCH}}/debian:buster
Expand Down
8 changes: 8 additions & 0 deletions dockers/docker-config-engine-buster/Dockerfile.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,14 @@ RUN apt-get install -y \
libz-dev
{%- endif %}

{%- if CROSS_BUILD_ENVIRON == "y" %}
RUN mkdir prebuilt-python-wheels
COPY prebuilt-python-wheels/* prebuilt-python-wheels/
RUN pip3 install prebuilt-python-wheels/lxml*cp37*.whl
RUN pip3 install prebuilt-python-wheels/bitarray*cp37*.whl
RUN rm -rf prebuilt-python-wheels
{%- endif %}

# For sonic-config-engine Python 3 package
# Explicitly install pyangbind here, as pyangbind causes enum34 to be installed.
# enum34 causes Python 're' package to not work properly as it redefines an incompatible enum.py module
Expand Down
8 changes: 8 additions & 0 deletions dockers/docker-platform-monitor/Dockerfile.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,14 @@ RUN pip2 install --upgrade 'pip<21'
RUN apt-get purge -y python-pip
RUN pip2 install 'setuptools==40.8.0'

{%- if CROSS_BUILD_ENVIRON == "y" %}
RUN mkdir prebuilt-python-wheels
COPY prebuilt-python-wheels/* prebuilt-python-wheels/
RUN pip2 install prebuilt-python-wheels/lxml*cp27*.whl
RUN pip3 install prebuilt-python-wheels/lxml*cp37*.whl
RUN rm -rf prebuilt-python-wheels
{%- endif %}

# On Arista devices, the sonic_platform wheel is not installed in the container.
# Instead, the installation directory is mounted from the host OS. However, this method
# doesn't ensure all dependencies are installed in the container. So here we
Expand Down
7 changes: 7 additions & 0 deletions dockers/docker-sonic-mgmt-framework/Dockerfile.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,13 @@ ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && \
apt-get install -y g++ python3-dev libxml2 libcurl3-gnutls libcjson-dev

{%- if CROSS_BUILD_ENVIRON == "y" %}
RUN mkdir prebuilt-python-wheels
COPY prebuilt-python-wheels/* prebuilt-python-wheels/
RUN pip3 install prebuilt-python-wheels/grpcio*cp37*.whl
RUN rm -rf prebuilt-python-wheels
{%- endif %}

RUN pip3 install connexion==2.7.0 \
setuptools==21.0.0 \
grpcio-tools==1.20.0 \
Expand Down
6 changes: 3 additions & 3 deletions files/build_templates/sonic_debian_extension.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ sudo bash -c "echo \"DOCKER_OPTS=\"--storage-driver=overlay2\"\" >> $FILESYSTEM_
# Copy docker start script to be able to start docker in chroot
sudo mkdir -p "$FILESYSTEM_ROOT/$DOCKER_CTL_DIR"
sudo cp $DOCKER_SCRIPTS_DIR/docker "$FILESYSTEM_ROOT/$DOCKER_CTL_SCRIPT"
if [ $MULTIARCH_QEMU_ENVIRON == y ]; then
if [[ $MULTIARCH_QEMU_ENVIRON == y || $CROSS_BUILD_ENVIRON == y ]]; then
DOCKER_HOST="unix:///dockerfs/var/run/docker.sock"
SONIC_NATIVE_DOCKERD_FOR_DOCKERFS_PID="cat `pwd`/dockerfs/var/run/docker.pid"
else
Expand Down Expand Up @@ -628,7 +628,7 @@ EOF
{% endif %}

## Bind docker path
if [ $MULTIARCH_QEMU_ENVIRON == y ]; then
if [[ $MULTIARCH_QEMU_ENVIRON == y || $CROSS_BUILD_ENVIRON == y ]]; then
sudo mkdir -p $FILESYSTEM_ROOT/dockerfs
sudo mount --bind dockerfs $FILESYSTEM_ROOT/dockerfs
fi
Expand Down Expand Up @@ -711,7 +711,7 @@ sudo LANG=C DOCKER_HOST="$DOCKER_HOST" chroot $FILESYSTEM_ROOT sonic-package-man

sudo umount $FILESYSTEM_ROOT/target
sudo rm -r $FILESYSTEM_ROOT/target
if [ $MULTIARCH_QEMU_ENVIRON == y ]; then
if [[ $MULTIARCH_QEMU_ENVIRON == y || $CROSS_BUILD_ENVIRON == y ]]; then
sudo umount $FILESYSTEM_ROOT/dockerfs
sudo rm -fr $FILESYSTEM_ROOT/dockerfs
sudo kill -9 `sudo $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS_PID` || true
Expand Down
2 changes: 2 additions & 0 deletions platform/marvell-armhf/prestera/debian/rules
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,9 @@
include /usr/share/dpkg/pkg-info.mk

PACKAGE_PRE_NAME := mrvlprestera
ifneq ($(CROSS_BUILD_ENVIRON), y)
KVERSION ?= $(shell uname -r)
endif
KERNEL_SRC := /lib/modules/$(KVERSION)
MOD_SRC_DIR:= $(shell pwd)
MODULE_DIR := mrvl-prestera/cpssEnabler/linuxNoKernelModule/drivers
Expand Down
2 changes: 1 addition & 1 deletion scripts/build_debian_base_system.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ generate_version_file()

if [ "$ENABLE_VERSION_CONTROL_DEB" != "y" ]; then
if [[ $CONFIGURED_ARCH == armhf || $CONFIGURED_ARCH == arm64 ]]; then
if [ $MULTIARCH_QEMU_ENVIRON == "y" ]; then
if [[ $MULTIARCH_QEMU_ENVIRON == y || $CROSS_BUILD_ENVIRON == y ]]; then
# qemu arm bin executable for cross-building
sudo mkdir -p $FILESYSTEM_ROOT/usr/bin
sudo cp /usr/bin/qemu*static $FILESYSTEM_ROOT/usr/bin || true
Expand Down
10 changes: 10 additions & 0 deletions scripts/prepare_docker_buildinfo.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,20 @@ DISTRO=$5
DOCKERFILE_PATH=$(dirname "$DOCKERFILE_TARGE")
BUILDINFO_PATH="${DOCKERFILE_PATH}/buildinfo"
BUILDINFO_VERSION_PATH="${BUILDINFO_PATH}/versions"
if [[ $CROSS_BUILD_ENVIRON == y ]]; then
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$CROSS_BUILD_ENVIRON

This file is to implement reproducible feature sonic-net/SONiC#684. Why you add here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to copy prebuilt (cross-compiled) python wheels to the dockers directory in order to be installed in the dockers Docker.j2 file. See for example dockers/docker-config-engine-buster/Dockerfile.j2.
Please suggest what and where to do it if this file is inappropriate.
Thanks.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you treat them as normal Makefile target and normal dependencies of others?

PREBUILT_PYTHON_WHEELS_PATH="${DOCKERFILE_PATH}/prebuilt-python-wheels"
fi

[ -d $BUILDINFO_PATH ] && rm -rf $BUILDINFO_PATH
mkdir -p $BUILDINFO_VERSION_PATH

if [[ $CROSS_BUILD_ENVIRON == y ]]; then
# Copy pre-compiled (cross-compiled) python wheels used in dockers
[ -d $PREBUILT_PYTHON_WHEELS_PATH ] && rm -rf $PREBUILT_PYTHON_WHEELS_PATH
mkdir -p $PREBUILT_PYTHON_WHEELS_PATH
cp -rf /PRE_BUILT_TARGET/* $PREBUILT_PYTHON_WHEELS_PATH
fi

# Get the debian distribution from the docker base image
if [ -z "$DISTRO" ]; then
DOCKER_BASE_IMAGE=$(grep "^FROM" $DOCKERFILE | head -n 1 | awk '{print $2}')
Expand Down
65 changes: 59 additions & 6 deletions slave.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ ifeq ($(CONFIGURED_ARCH),arm64)
override DOCKER_BASE_ARCH = arm64v8
endif
endif

IMAGE_DISTRO := bullseye
IMAGE_DISTRO_DEBS_PATH = $(TARGET_PATH)/debs/$(IMAGE_DISTRO)
IMAGE_DISTRO_FILES_PATH = $(TARGET_PATH)/files/$(IMAGE_DISTRO)
Expand All @@ -80,6 +81,7 @@ export IMAGE_DISTRO
export IMAGE_DISTRO_DEBS_PATH
export MULTIARCH_QEMU_ENVIRON
export DOCKER_BASE_ARCH
export CROSS_BUILD_ENVIRON

###############################################################################
## Utility rules
Expand Down Expand Up @@ -233,6 +235,41 @@ ifeq ($(CONFIGURED_PLATFORM),vs)
export BUILD_MULTIASIC_KVM=$(BUILD_MULTIASIC_KVM)
endif

ifeq ($(CROSS_BUILD_ENVIRON),y)
DEB_BUILD_OPTIONS_GENERIC += nocheck
export $(dpkg-architecture -a$(CONFIGURED_ARCH))
ANT_DEB_CROSS_OPT := -a$(CONFIGURED_ARCH) -Pcross,nocheck
ANT_DEB_CONFIG := CONFIG_SITE=/etc/dpkg-cross/cross-config.$(CONFIGURED_ARCH)

VIRTENV_BASE_CROSS_PYTHON2 = /python_virtualenv/env2/
VIRTENV_BASE_CROSS_PYTHON3 = /python_virtualenv/env3/
VIRTENV_BIN_CROSS_PYTHON2 = $(VIRTENV_BASE_CROSS_PYTHON2)/bin/
VIRTENV_BIN_CROSS_PYTHON3 = $(VIRTENV_BASE_CROSS_PYTHON3)/bin/
VIRTENV_LIB_CROSS_PYTHON2 = $(VIRTENV_BASE_CROSS_PYTHON2)/lib/
VIRTENV_LIB_CROSS_PYTHON3 = $(VIRTENV_BASE_CROSS_PYTHON3)/lib/

ifeq ($(CONFIGURED_ARCH),armhf)
CROSS_HOST_TYPE = arm-linux-gnueabihf
GOARCH=arm
else ifeq ($(CONFIGURED_ARCH),arm64)
CROSS_HOST_TYPE = aarch64-linux-gnu
GOARCH=arm64
endif

CROSS_COMPILE = $(CROSS_HOST_TYPE)-
CROSS_CC = $(CROSS_COMPILE)gcc
CROSS_CXX = $(CROSS_COMPILE)g++
CROSS_AR = $(CROSS_COMPILE)ar
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of using CROSS_CC/CROSS_CXX/CROSS_AR, can CC/CXX/AR be used directly, and exported as environment variables? That way, multiple places later on won't need lines like CC=$(CROSS_CC).

CROSS_LIB_PATH = /usr/$(CROSS_HOST_TYPE)/lib/
CROSS_BIN_PATH = /usr/$(CROSS_HOST_TYPE)/bin/
CROSS_PKGS_LIB_PATH = /usr/lib/$(CROSS_HOST_TYPE)

CROSS_LIBPERL_VERSION = $(shell dpkg -s libperl-dev:armhf|grep Version|awk '{split($$2,v,"-"); print v[1];}')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be libperl-dev:$arch

CROSS_PERL_CORE_PATH = $(CROSS_PKGS_LIB_PATH)/perl/$(CROSS_LIBPERL_VERSION)/CORE/

CROSS_COMPILE_FLAGS := CGO_ENABLED=1 GOOS=linux GOARCH=$(GOARCH) CROSS_COMPILE=$(CROSS_COMPILE) OVERRIDE_HOST_TYPE=$(CROSS_HOST_TYPE) CROSS_CC=$(CROSS_CC) CROSS_CXX=$(CROSS_CXX) CROSS_LIB_PATH=$(CROSS_LIB_PATH) CROSS_BIN_PATH=$(CROSS_BIN_PATH) CROSS_HOST_TYPE=$(CROSS_HOST_TYPE) CROSS_PKGS_LIB_PATH=$(CROSS_PKGS_LIB_PATH) CROSS_PERL_CORE_PATH=$(CROSS_PERL_CORE_PATH)
endif

###############################################################################
## Routing stack related exports
###############################################################################
Expand Down Expand Up @@ -301,6 +338,7 @@ $(info "SONIC_VERSION_CONTROL_COMPONENTS": "$(SONIC_VERSION_CONTROL_COMPONENTS)"
ifeq ($(CONFIGURED_PLATFORM),vs)
$(info "BUILD_MULTIASIC_KVM" : "$(BUILD_MULTIASIC_KVM)")
endif
$(info "CROSS_BUILD_ENVIRON" : "$(CROSS_BUILD_ENVIRON)")
$(info )
else
$(info SONiC Build System for $(CONFIGURED_PLATFORM):$(CONFIGURED_ARCH))
Expand Down Expand Up @@ -469,7 +507,7 @@ $(addprefix $(DEBS_PATH)/, $(SONIC_MAKE_DEBS)) : $(DEBS_PATH)/% : .platform $$(a
if [ -f $($*_SRC_PATH).patch/series ]; then pushd $($*_SRC_PATH) && QUILT_PATCHES=../$(notdir $($*_SRC_PATH)).patch quilt push -a; popd; fi
# Build project and take package
$(SETUP_OVERLAYFS_FOR_DPKG_ADMINDIR)
DEB_BUILD_OPTIONS="${DEB_BUILD_OPTIONS_GENERIC}" make -j$(SONIC_CONFIG_MAKE_JOBS) DEST=$(shell pwd)/$(DEBS_PATH) -C $($*_SRC_PATH) $(shell pwd)/$(DEBS_PATH)/$* $(LOG)
DEB_BUILD_OPTIONS="${DEB_BUILD_OPTIONS_GENERIC}" $(ANT_DEB_CONFIG) $(CROSS_COMPILE_FLAGS) make -j$(SONIC_CONFIG_MAKE_JOBS) DEST=$(shell pwd)/$(DEBS_PATH) -C $($*_SRC_PATH) $(shell pwd)/$(DEBS_PATH)/$* $(LOG)
# Clean up
if [ -f $($*_SRC_PATH).patch/series ]; then pushd $($*_SRC_PATH) && quilt pop -a -f; [ -d .pc ] && rm -rf .pc; popd; fi

Expand Down Expand Up @@ -511,8 +549,8 @@ $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(a
if [ -f ./autogen.sh ]; then ./autogen.sh $(LOG); fi
$(SETUP_OVERLAYFS_FOR_DPKG_ADMINDIR)
$(if $($*_DPKG_TARGET),
${$*_BUILD_ENV} DEB_BUILD_OPTIONS="${DEB_BUILD_OPTIONS_GENERIC} ${$*_DEB_BUILD_OPTIONS}" dpkg-buildpackage -rfakeroot -b -us -uc -tc -j$(SONIC_CONFIG_MAKE_JOBS) --as-root -T$($*_DPKG_TARGET) --admindir $$mergedir $(LOG),
${$*_BUILD_ENV} DEB_BUILD_OPTIONS="${DEB_BUILD_OPTIONS_GENERIC} ${$*_DEB_BUILD_OPTIONS}" dpkg-buildpackage -rfakeroot -b -us -uc -tc -j$(SONIC_CONFIG_MAKE_JOBS) --admindir $$mergedir $(LOG)
${$*_BUILD_ENV} DEB_BUILD_OPTIONS="${DEB_BUILD_OPTIONS_GENERIC} ${$*_DEB_BUILD_OPTIONS}" $(ANT_DEB_CONFIG) $(CROSS_COMPILE_FLAGS) dpkg-buildpackage -rfakeroot -b $(ANT_DEB_CROSS_OPT) -us -uc -tc -j$(SONIC_CONFIG_MAKE_JOBS) --as-root -T$($*_DPKG_TARGET) --admindir $$mergedir $(LOG),
${$*_BUILD_ENV} DEB_BUILD_OPTIONS="${DEB_BUILD_OPTIONS_GENERIC} ${$*_DEB_BUILD_OPTIONS}" $(ANT_DEB_CONFIG) $(CROSS_COMPILE_FLAGS) dpkg-buildpackage -rfakeroot -b $(ANT_DEB_CROSS_OPT) -us -uc -tc -j$(SONIC_CONFIG_MAKE_JOBS) --admindir $$mergedir $(LOG)
)
Copy link
Collaborator

@qiluo-msft qiluo-msft Oct 11, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

)

Seems an extra ).
This embeded if-else-endif is so difficult to read. Could you improve? #Closed

popd $(LOG_SIMPLE)
# Clean up
Expand Down Expand Up @@ -579,7 +617,12 @@ $(SONIC_INSTALL_DEBS) : $(DEBS_PATH)/%-install : .platform $$(addsuffix -install
{ while dpkg -s $(firstword $(subst _, ,$(basename $(deb)))) | grep "^Version: $(word 2, $(subst _, ,$(basename $(deb))))" &> /dev/null; do echo "waiting for $(deb) to be uninstalled" $(LOG); sleep 1; done } )
# put a lock here because dpkg does not allow installing packages in parallel
if mkdir $(DEBS_PATH)/dpkg_lock &> /dev/null; then
ifneq ($(CROSS_BUILD_ENVIRON),y)
{ sudo DEBIAN_FRONTEND=noninteractive dpkg -i $(DEBS_PATH)/$* $(LOG) && rm -d $(DEBS_PATH)/dpkg_lock && break; } || { rm -d $(DEBS_PATH)/dpkg_lock && sudo lsof /var/lib/dpkg/lock-frontend && ps aux && exit 1 ; }
else
# Relocate debian packages python libraries to the cross python virtual env location
{ sudo DEBIAN_FRONTEND=noninteractive dpkg -i $(if $(findstring $(LINUX_HEADERS),$*),--force-depends) $(DEBS_PATH)/$* $(LOG) && rm -rf tmp && mkdir tmp && dpkg -x $(DEBS_PATH)/$* tmp && (sudo cp -rf tmp/usr/lib/python2.7/dist-packages/* /python_virtualenv/env2/lib/python2.7/site-packages/ 2>/dev/null || true) && (sudo cp -rf tmp/usr/lib/python3/dist-packages/* /python_virtualenv/env3/lib/python3.7/site-packages/ 2>/dev/null || true) && rm -d $(DEBS_PATH)/dpkg_lock && break; } || { rm -d $(DEBS_PATH)/dpkg_lock && sudo lsof /var/lib/dpkg/lock-frontend && ps aux && exit 1 ; }
Copy link
Collaborator

@qiluo-msft qiluo-msft May 20, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

  	{ sudo DEBIAN_FR [](http://example.com/codeflow?start=0&length=19)

This line is super long. Could you reformat? #Closed

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are some parts in this long line same as above branch. Suggest fine granularity reuse.

Copy link
Collaborator

@qiluo-msft qiluo-msft May 20, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LINUX_HEADERS

Why treat these packages differently? Possible no special action. #WontFix

Copy link
Collaborator

@qiluo-msft qiluo-msft May 20, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2.7

why hardcode version 2.7? #Closed

Copy link
Collaborator

@qiluo-msft qiluo-msft May 20, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

python3.7

why hardcode version 3.7? It will change along time. #Closed

endif
fi
done
$(FOOTER)
Expand Down Expand Up @@ -651,9 +694,14 @@ $(addprefix $(PYTHON_WHEELS_PATH)/, $(SONIC_PYTHON_WHEELS)) : $(PYTHON_WHEELS_PA
pushd $($*_SRC_PATH) $(LOG_SIMPLE)
# apply series of patches if exist
if [ -f ../$(notdir $($*_SRC_PATH)).patch/series ]; then QUILT_PATCHES=../$(notdir $($*_SRC_PATH)).patch quilt push -a; fi
if [ ! "$($*_TEST)" = "n" ]; then
ifneq ($(CROSS_BUILD_ENVIRON),y)
# Use pip instead of later setup.py to install dependencies into user home, but uninstall self
pip$($*_PYTHON_VERSION) install . && pip$($*_PYTHON_VERSION) uninstall --yes `python$($*_PYTHON_VERSION) setup.py --name`
if [ ! "$($*_TEST)" = "n" ]; then python$($*_PYTHON_VERSION) setup.py test $(LOG); fi
saiarcot895 marked this conversation as resolved.
Show resolved Hide resolved
python$($*_PYTHON_VERSION) setup.py test $(LOG); fi
else
PATH=$(VIRTENV_BIN_CROSS_PYTHON$($*_PYTHON_VERSION)):${PATH} python$($*_PYTHON_VERSION) setup.py test $(LOG); fi
endif
python$($*_PYTHON_VERSION) setup.py bdist_wheel $(LOG)
# clean up
if [ -f ../$(notdir $($*_SRC_PATH)).patch/series ]; then quilt pop -a -f; [ -d .pc ] && rm -rf .pc; fi
Expand All @@ -680,7 +728,12 @@ $(SONIC_INSTALL_WHEELS) : $(PYTHON_WHEELS_PATH)/%-install : .platform $$(addsuff
# put a lock here to avoid race conditions
while true; do
if mkdir $(PYTHON_WHEELS_PATH)/pip_lock &> /dev/null; then
ifneq ($(CROSS_BUILD_ENVIRON),y)
{ sudo -E pip$($*_PYTHON_VERSION) install $(PYTHON_WHEELS_PATH)/$* $(LOG) && rm -d $(PYTHON_WHEELS_PATH)/pip_lock && break; } || { rm -d $(PYTHON_WHEELS_PATH)/pip_lock && exit 1 ; }
else
# Link python script and data expected location to the cross python virtual env istallation locations
{ PATH=$(VIRTENV_BIN_CROSS_PYTHON$($*_PYTHON_VERSION)):${PATH} sudo -E $(VIRTENV_BIN_CROSS_PYTHON$($*_PYTHON_VERSION))/pip$($*_PYTHON_VERSION) install --no-deps $(PYTHON_WHEELS_PATH)/$* $(LOG) && $(if $(findstring $(SONIC_CONFIG_ENGINE_PY3),$*),(sudo ln -s $(VIRTENV_BIN_CROSS_PYTHON$($*_PYTHON_VERSION))/sonic-cfggen /usr/local/bin/sonic-cfggen 2>/dev/null || true), true ) && $(if $(findstring $(SONIC_YANG_MODELS_PY3),$*),(sudo ln -s $(VIRTENV_BASE_CROSS_PYTHON3)/yang-models /usr/local/yang-models 2>/dev/null || true), true ) && rm -d $(PYTHON_WHEELS_PATH)/pip_lock && break; } || { rm -d $(PYTHON_WHEELS_PATH)/pip_lock && exit 1 ; }
Copy link
Collaborator

@qiluo-msft qiluo-msft Sep 27, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SONIC_CONFIG_ENGINE_PY3

Why treat package differently? slave just provides a common environment to build any packages in general. #Closed

Copy link
Contributor Author

@gregshpit gregshpit Sep 30, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is wrong here that python3 version appears explicitly or that I do some specific handling for SONIC_CONFIG_ENGINE_PY3 in slave.mk ?
SONIC_CONFIG_ENGINE_PY3 and also SONIC_YANG_MODELS_PY3 are special because components using them expect them to be installed in /usr/local/bin/sonic-cfggen and /usr/local/yang-models respectively. However when using cross-compilation these python scripts are installed in some other python virtual environment location. In order to keep them accessible through expected /usr/local/... location I linked the built scripts to these locations.
This should be done every time SONIC_CONFIG_ENGINE_PY3 or SONIC_YANG_MODELS_PY3 wheels are installed.
I could have done it inside these wheels building environment but I did not find any hook for "wheel installation" there except for that in slave.mk .

Please advise how to improve this implementation.
Thanks.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding is that you only need virtual env in build, not runtime. How about buiding the python wheel with virtual env, but let slave container install the wheel outside virtual env.

Not an expert on the solution. The concern is that this is not general and not scalable.

endif
fi
done
$(FOOTER)
Expand Down Expand Up @@ -803,7 +856,7 @@ $(addprefix $(TARGET_PATH)/, $(DOCKER_IMAGES)) : $(TARGET_PATH)/%.gz : .platform
$(eval export $(subst -,_,$(notdir $($*.gz_PATH)))_whls=$(shell printf "$(subst $(SPACE),\n,$(call expand,$($*.gz_PYTHON_WHEELS)))\n" | awk '!a[$$0]++'))
$(eval export $(subst -,_,$(notdir $($*.gz_PATH)))_dbgs=$(shell printf "$(subst $(SPACE),\n,$(call expand,$($*.gz_DBG_PACKAGES)))\n" | awk '!a[$$0]++'))
$(eval export $(subst -,_,$(notdir $($*.gz_PATH)))_pkgs=$(shell printf "$(subst $(SPACE),\n,$(call expand,$($*.gz_APT_PACKAGES)))\n" | awk '!a[$$0]++'))
if [ -d $($*.gz_PATH)/cli-plugin-tests/ ]; then pushd $($*.gz_PATH)/cli-plugin-tests; pytest-$($(SONIC_UTILITIES_PY3)_PYTHON_VERSION) -v $(LOG); popd; fi
if [ -d $($*.gz_PATH)/cli-plugin-tests/ ]; then pushd $($*.gz_PATH)/cli-plugin-tests; PATH=$(VIRTENV_BIN_CROSS_PYTHON$($(SONIC_UTILITIES_PY3)_PYTHON_VERSION)):${PATH} pytest-$($(SONIC_UTILITIES_PY3)_PYTHON_VERSION) -v $(LOG); popd; fi
# Label docker image with componenets versions
$(eval export $(subst -,_,$(notdir $($*.gz_PATH)))_labels=$(foreach component,\
$(call expand,$($*.gz_DEPENDS),RDEPENDS) \
Expand Down Expand Up @@ -1086,7 +1139,6 @@ $(addprefix $(TARGET_PATH)/, $(SONIC_INSTALLERS)) : $(TARGET_PATH)/% : \
)
)


# Exported variables are used by sonic_debian_extension.sh
export installer_start_scripts="$(foreach docker, $($*_DOCKERS),$(addsuffix .sh, $($(docker:-dbg.gz=.gz)_CONTAINER_NAME)))"
export feature_vs_image_names="$(foreach docker, $($*_DOCKERS), $(addsuffix :, $($(docker:-dbg.gz=.gz)_CONTAINER_NAME):$(docker:-dbg.gz=.gz)))"
Expand Down Expand Up @@ -1136,6 +1188,7 @@ $(addprefix $(TARGET_PATH)/, $(SONIC_INSTALLERS)) : $(TARGET_PATH)/% : \
TRUSTED_GPG_URLS=$(TRUSTED_GPG_URLS) \
PACKAGE_URL_PREFIX=$(PACKAGE_URL_PREFIX) \
MULTIARCH_QEMU_ENVIRON=$(MULTIARCH_QEMU_ENVIRON) \
CROSS_BUILD_ENVIRON=$(CROSS_BUILD_ENVIRON) \
./build_debian.sh $(LOG)

USERNAME="$(USERNAME)" \
Expand Down
Loading