Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add mariner build images #832

Merged
merged 22 commits into from
Mar 30, 2023
Merged

Add mariner build images #832

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
0c35da6
Add mariner build images
sbomer Mar 24, 2023
3b86c36
Don't inline long signing key
sbomer Mar 24, 2023
33548d5
Add trailing newlines
sbomer Mar 24, 2023
b3c3373
Remove arm and arm-alpine images for now
sbomer Mar 24, 2023
59c407b
Cleanup
sbomer Mar 24, 2023
b88433f
Reduce llvm version duplication, remove sources on x86
sbomer Mar 24, 2023
5d962aa
Extract llm version to variable
sbomer Mar 24, 2023
4dc2a9f
Fix tar extraction
sbomer Mar 26, 2023
82b4b45
Fix newline
sbomer Mar 27, 2023
a54d50d
Try defining separate stage for artifacts
sbomer Mar 28, 2023
ae3a84d
Fix mariner 1.0 base image
sbomer Mar 28, 2023
4f0d25d
Order manifest entries by dependency order
sbomer Mar 28, 2023
042a821
Fix order
sbomer Mar 28, 2023
c7604ae
Increase timeout
sbomer Mar 28, 2023
510b806
Merge remote-tracking branch 'origin/main' into marinerBuildImages
sbomer Mar 28, 2023
b6ebffa
Avoid local-only tags
sbomer Mar 28, 2023
ffc872d
Optimize image size
sbomer Mar 29, 2023
8327d79
Ensure qemu-user-static is available
sbomer Mar 29, 2023
1fce64b
Use local-only images, fix indentation
sbomer Mar 29, 2023
49178c5
Fix capitalization
sbomer Mar 29, 2023
ec3af4c
Fix newlines
sbomer Mar 29, 2023
8a2a90d
Add back non-local version tags
sbomer Mar 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions eng/pipelines/dotnet-buildtools-prereqs-mariner-pr.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,3 +24,6 @@ stages:
parameters:
internalProjectName: ${{ variables.internalProjectName }}
publicProjectName: ${{ variables.publicProjectName }}
linuxAmdBuildJobTimeout: 360
customBuildInitSteps:
- template: /eng/pipelines/steps/install-cross-build-prereqs.yml
3 changes: 3 additions & 0 deletions eng/pipelines/dotnet-buildtools-prereqs-mariner.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,3 +24,6 @@ stages:
parameters:
internalProjectName: ${{ variables.internalProjectName }}
publicProjectName: ${{ variables.publicProjectName }}
linuxAmdBuildJobTimeout: 360
customBuildInitSteps:
- template: /eng/pipelines/steps/install-cross-build-prereqs.yml
2 changes: 1 addition & 1 deletion src/cbl-mariner/1.0/helix/amd64/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM cblmariner.azurecr.io/base/core:1.0
FROM mcr.microsoft.com/cbl-mariner/base/core:1.0

# Install Helix Dependencies

Expand Down
10 changes: 10 additions & 0 deletions src/cbl-mariner/2.0/cross/amd64-alpine/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-builder-local AS builder
ARG ROOTFS_DIR=/crossrootfs/x64

RUN /scripts/eng/common/cross/build-rootfs.sh x64 alpine3.13 --skipunmount


FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-amd64-local
ARG ROOTFS_DIR=/crossrootfs/x64

COPY --from=builder $ROOTFS_DIR $ROOTFS_DIR
14 changes: 14 additions & 0 deletions src/cbl-mariner/2.0/cross/amd64/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-builder-local AS builder
ARG ROOTFS_DIR=/crossrootfs/x64

RUN /scripts/eng/common/cross/build-rootfs.sh x64 xenial --skipunmount


FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-amd64-local
ARG ROOTFS_DIR=/crossrootfs/x64

COPY --from=builder $ROOTFS_DIR $ROOTFS_DIR

RUN LLVM_VERSION=12.0.1 LLVM_VERSION_MAJOR="${LLVM_VERSION%%.*}" && \
Copy link
Member

@am11 am11 Mar 29, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can directly use 15.0.7, which has been validated in industry and revised seven times since its initial release (has tons of fixes and improvements). 16 is very new and 12 is very old. We are already using 15 for linux-musl official builds.

cc @jkotas, @hoyosjs

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our current plan is to ship building with LLVM 16, which will be supported by mariner. Once they add support we should be able to stop building it ourselves. 16 also won't be quite as new by the time we ship .NET 8.

I plan to make the update to 16 fairly soon after this goes in - the reason I'm not doing it here is that it's not quite as simple as updating the version number (there were some missing dependencies for the compiler-rt builds), and I wanted to prioritize getting to a point where I can run dotnet/runtime ci legs with the new images. I am sure there will be problems I haven't caught with my local testing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious, why v12.0.1 specifically even as a temporary solution? If it was v9.0.x (our go-to llvm version in official legs) or v15.0.x (modern version used in latest linux-musl legs) that would make more sense for a temporary solution. AFAIK, we aren't using 12.0.x explicitly for anything.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I chose it because it matches the version available in the cbl-mariner repos, so we can just install it for the x64 and x86 builds (crossdeps-amd64 does this). It is how I discovered that the current mariner package doesn't support targeting arm/arm64.

mkdir -p $ROOTFS_DIR/usr/lib/llvm-${LLVM_VERSION_MAJOR}/lib/clang/${LLVM_VERSION}/lib/linux/ && \
cp /usr/lib/clang/${LLVM_VERSION}/lib/linux/libclang_rt.profile-x86_64.a $ROOTFS_DIR/usr/lib/llvm-${LLVM_VERSION_MAJOR}/lib/clang/${LLVM_VERSION}/lib/linux/
10 changes: 10 additions & 0 deletions src/cbl-mariner/2.0/cross/arm64-alpine/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-builder-local AS builder
ARG ROOTFS_DIR=/crossrootfs/arm64

RUN /scripts/eng/common/cross/build-rootfs.sh arm64 alpine3.13 --skipunmount


FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-arm-local
ARG ROOTFS_DIR=/crossrootfs/arm64

COPY --from=builder $ROOTFS_DIR $ROOTFS_DIR
38 changes: 38 additions & 0 deletions src/cbl-mariner/2.0/cross/arm64/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-builder-local AS builder
ARG ROOTFS_DIR=/crossrootfs/arm64

RUN /scripts/eng/common/cross/build-rootfs.sh arm64 xenial --skipunmount

# Build compiler-rt profile library for PGO instrumentation
RUN mkdir compiler-rt_build && cd compiler-rt_build && \
BUILD_FLAGS="-v --sysroot=$ROOTFS_DIR" \
TARGET_TRIPLE=aarch64-linux-gnu && \
cmake ../llvm-project.src/compiler-rt \
-DCOMPILER_RT_BUILD_PROFILE=ON \
-DCOMPILER_RT_BUILD_BUILTINS=OFF \
-DCOMPILER_RT_BUILD_SANITIZERS=OFF \
-DCOMPILER_RT_BUILD_XRAY=OFF \
-DCOMPILER_RT_BUILD_LIBFUZZER=OFF \
\
-DCMAKE_C_COMPILER=clang \
-DCMAKE_CXX_COMPILER=clang++ \
-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" \
-DCMAKE_C_COMPILER_TARGET=${TARGET_TRIPLE} \
-DCMAKE_CXX_COMPILER_TARGET=${TARGET_TRIPLE} \
-DLLVM_CONFIG_PATH=llvm-config \
-DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=$ROOTFS_DIR/usr \
-DCMAKE_C_FLAGS="${BUILD_FLAGS}" \
-DCMAKE_CXX_FLAGS="${BUILD_FLAGS}" && \
make -j $(getconf _NPROCESSORS_ONLN)

RUN LLVM_VERSION=12.0.1 LLVM_VERSION_MAJOR="${LLVM_VERSION%%.*}" && \
mkdir -p $ROOTFS_DIR/usr/lib/llvm-${LLVM_VERSION_MAJOR}/lib/clang/${LLVM_VERSION}/lib/linux/ && \
cp compiler-rt_build/lib/linux/libclang_rt.profile-aarch64.a $ROOTFS_DIR/usr/lib/llvm-${LLVM_VERSION_MAJOR}/lib/clang/${LLVM_VERSION}/lib/linux/


FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-arm-local
ARG ROOTFS_DIR=/crossrootfs/arm64

COPY --from=builder $ROOTFS_DIR $ROOTFS_DIR
43 changes: 43 additions & 0 deletions src/cbl-mariner/2.0/cross/x86/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-builder-local AS builder
ARG ROOTFS_DIR=/crossrootfs/x86

# Provides LLVMConfig.cmake, needed to build compiler-rt. Only needed on x86.
# x64 doesn't need this because we use compiler-rt from the mariner package.
# arm/arm64 don't need this because LLVMConfig.cmake is provided by the LLVM we build from source.
RUN tdnf install -y llvm-devel

RUN /scripts/eng/common/cross/build-rootfs.sh x86 xenial --skipunmount

# Build compiler-rt profile library for PGO instrumentation
RUN mkdir compiler-rt_build && cd compiler-rt_build && \
BUILD_FLAGS="-v --sysroot=$ROOTFS_DIR" \
TARGET_TRIPLE=i386-linux-gnu && \
cmake ../llvm-project.src/compiler-rt \
-DCOMPILER_RT_BUILD_PROFILE=ON \
-DCOMPILER_RT_BUILD_BUILTINS=OFF \
-DCOMPILER_RT_BUILD_SANITIZERS=OFF \
-DCOMPILER_RT_BUILD_XRAY=OFF \
-DCOMPILER_RT_BUILD_LIBFUZZER=OFF \
\
-DCMAKE_C_COMPILER=clang \
-DCMAKE_CXX_COMPILER=clang++ \
-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" \
-DCMAKE_C_COMPILER_TARGET=${TARGET_TRIPLE} \
-DCMAKE_CXX_COMPILER_TARGET=${TARGET_TRIPLE} \
-DLLVM_CONFIG_PATH=llvm-config \
-DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=$ROOTFS_DIR/usr \
-DCMAKE_C_FLAGS="${BUILD_FLAGS}" \
-DCMAKE_CXX_FLAGS="${BUILD_FLAGS}" && \
make -j $(getconf _NPROCESSORS_ONLN)

RUN LLVM_VERSION=12.0.1 LLVM_VERSION_MAJOR="${LLVM_VERSION%%.*}" && \
mkdir -p $ROOTFS_DIR/usr/lib/llvm-${LLVM_VERSION_MAJOR}/lib/clang/${LLVM_VERSION}/lib/linux/ && \
cp compiler-rt_build/lib/linux/libclang_rt.profile-i386.a $ROOTFS_DIR/usr/lib/llvm-${LLVM_VERSION_MAJOR}/lib/clang/${LLVM_VERSION}/lib/linux/


FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-amd64-local
ARG ROOTFS_DIR=/crossrootfs/x86

COPY --from=builder $ROOTFS_DIR $ROOTFS_DIR
6 changes: 6 additions & 0 deletions src/cbl-mariner/2.0/crossdeps-amd64/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-local

RUN tdnf install -y \
clang \
lld \
compiler-rt
6 changes: 6 additions & 0 deletions src/cbl-mariner/2.0/crossdeps-arm/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-builder-local AS builder

FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-local

# Install LLVM that we built from source
COPY --from=builder /usr/local /usr/local
64 changes: 64 additions & 0 deletions src/cbl-mariner/2.0/crossdeps-builder/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-crossdeps-local

# TODO: remove this once debootstrap is available in the base repos.
COPY mariner-extended.repo /etc/yum.repos.d/

RUN tdnf install -y \
# Rootfs build dependencies
debootstrap \
# LLVM build dependencies
texinfo \
diffutils \
binutils

# Obtain ubuntu package signing key (for use by debootstrap)
# 1. Add public key used to sign the ubuntu keyring
COPY dimitri_john_ledkov.asc .
RUN gpg --output dimitri_john_ledkov.gpg --dearmor dimitri_john_ledkov.asc && \
rm dimitri_john_ledkov.asc && \
# 2. Download the ubuntu keyrings
wget https://mirrors.edge.kernel.org/ubuntu/pool/main/u/ubuntu-keyring/ubuntu-keyring_2021.03.26.tar.gz && \
echo "492eed5c06408c6f632577adb0796130af5d6542013ef418f47187a209e49bb1 ubuntu-keyring_2021.03.26.tar.gz" | sha256sum -c && \
tar xf ubuntu-keyring_2021.03.26.tar.gz && \
rm ubuntu-keyring_2021.03.26.tar.gz && \
# 3. Verify keyrings
pushd ubuntu-keyring-2021.03.26 && \
gpg --keyring /dimitri_john_ledkov.gpg --output SHA512SUMS.txt --decrypt SHA512SUMS.txt.asc && \
rm /dimitri_john_ledkov.gpg && \
sha512sum -c SHA512SUMS.txt && \
# 4. Install the needed keyring and delete the rest
mkdir -p /usr/share/keyrings && \
mv keyrings/ubuntu-archive-keyring.gpg /usr/share/keyrings && \
popd && \
rm -r ubuntu-keyring-2021.03.26

# 1. Obtain signing keys used to sign llvm sources
RUN wget https://releases.llvm.org/release-keys.asc && \
echo "f181a90697e3ea4b7782f1ee48314a570aef058505b4f3a0ab0611094ec13241 release-keys.asc" | sha256sum -c && \
gpg --output release-keys.gpg --dearmor release-keys.asc && \
rm release-keys.asc && \
# 2. Download llvm sources and signature, and verify signature
LLVM_VERSION=12.0.1 && \
wget -O llvm-project.src.tar.xz.sig https://github.com/llvm/llvm-project/releases/download/llvmorg-${LLVM_VERSION}/llvm-project-${LLVM_VERSION}.src.tar.xz.sig && \
echo "6cc956d622a7d3d746de0d71d8ca616a6c291e2c561703ac7a9535f38b999955 llvm-project.src.tar.xz.sig" | sha256sum -c && \
wget -O llvm-project.src.tar.xz https://github.com/llvm/llvm-project/releases/download/llvmorg-${LLVM_VERSION}/llvm-project-${LLVM_VERSION}.src.tar.xz && \
echo "129cb25cd13677aad951ce5c2deb0fe4afc1e9d98950f53b51bdcfb5a73afa0e llvm-project.src.tar.xz" | sha256sum -c && \
gpg --keyring /release-keys.gpg --verify llvm-project.src.tar.xz.sig && \
rm llvm-project.src.tar.xz.sig

# Build LLVM cross-toolchain (with support for targeting arm architectures)
RUN mkdir llvm-project.src && \
sbomer marked this conversation as resolved.
Show resolved Hide resolved
tar -xf llvm-project.src.tar.xz --directory llvm-project.src --strip-components=1 && \
rm llvm-project.src.tar.xz && \
mkdir build && cd build && \
cmake ../llvm-project.src/llvm \
-DCMAKE_BUILD_TYPE=Release \
-DLLVM_TARGETS_TO_BUILD="host;AArch64;ARM" \
-Wno-dev \
-DLLVM_ENABLE_PROJECTS="clang;lld" && \
Copy link
Member

@am11 am11 Mar 29, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
-DLLVM_ENABLE_PROJECTS="clang;lld" && \
-DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;lld;lldb" && \

This is to make sure we have lld-12, lldb-12, llvm-objcopy-12, llvm-objdump-12, llvm-ar-12, llvm-nm-12, llvm-ranlib-12, llvm-link-12, llvm-dwarfdump-12 and llvm-symbolizer-12 available in PATH. Please double check that these prerequisites, used by various runtime CI legs, are available in the final image's host layer: find /usr -type f -path '*bin*' -name 'llvm*' (crossrootfs only needs llvm-libs which is handled by build-rootfs.sh).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those tools (without version suffix) all get installed with just the "clang;lld" projects:

# find /usr -type f -path '*bin*' -name 'llvm*'
/usr/local/bin/llvm-pdbutil
/usr/local/bin/llvm-diff
/usr/local/bin/llvm-as
/usr/local/bin/llvm-size
/usr/local/bin/llvm-bcanalyzer
/usr/local/bin/llvm-undname
/usr/local/bin/llvm-objdump
/usr/local/bin/llvm-exegesis
/usr/local/bin/llvm-mca
/usr/local/bin/llvm-dwarfdump
/usr/local/bin/llvm-ml
/usr/local/bin/llvm-cxxdump
/usr/local/bin/llvm-c-test
/usr/local/bin/llvm-reduce
/usr/local/bin/llvm-lipo
/usr/local/bin/llvm-extract
/usr/local/bin/llvm-objcopy
/usr/local/bin/llvm-split
/usr/local/bin/llvm-mt
/usr/local/bin/llvm-elfabi
/usr/local/bin/llvm-opt-report
/usr/local/bin/llvm-link
/usr/local/bin/llvm-cxxfilt
/usr/local/bin/llvm-modextract
/usr/local/bin/llvm-profdata
/usr/local/bin/llvm-rc
/usr/local/bin/llvm-jitlink
/usr/local/bin/llvm-rtdyld
/usr/local/bin/llvm-mc
/usr/local/bin/llvm-nm
/usr/local/bin/llvm-readobj
/usr/local/bin/llvm-cfi-verify
/usr/local/bin/llvm-libtool-darwin
/usr/local/bin/llvm-symbolizer
/usr/local/bin/llvm-xray
/usr/local/bin/llvm-cat
/usr/local/bin/llvm-lto
/usr/local/bin/llvm-cov
/usr/local/bin/llvm-profgen
/usr/local/bin/llvm-tblgen
/usr/local/bin/llvm-cxxmap
/usr/local/bin/llvm-config
/usr/local/bin/llvm-lto2
/usr/local/bin/llvm-cvtres
/usr/local/bin/llvm-stress
/usr/local/bin/llvm-ar
/usr/local/bin/llvm-gsymutil
/usr/local/bin/llvm-ifs
/usr/local/bin/llvm-strings
/usr/local/bin/llvm-dwp
/usr/local/bin/llvm-dis

Are clang-tools-extra and lldb additionally required on the host?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for checking.

lldb is required in diagnostics repo's CI (since forever) and in runtime repo (since dotnet/runtime#82867) to troubleshoot stuff in the very image reporting a crash.

clang-tools-extra is not required but would be nice to put in the image while we are at it, so we don't have to rely on random (unmaintained) blob for clang-format and clang-tidy: https://github.com/dotnet/jitutils/blob/main/bootstrap.sh#L90 which CI in runtime end-up using in jit-diff legs.

llvm-ranlib seems to be missing in the list. From alpine-arm64 prereq image:

$ docker run -it mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-cross-arm64 bash -c 'find /usr -name llvm* | grep ranlib'

/usr/lib/llvm-15/bin/llvm-ranlib
/usr/bin/llvm-ranlib-15

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, sorry - llvm-ranlib is a symlink:

# ls -al /usr/local/bin/llvm-ranlib 
lrwxrwxrwx 1 root root 7 Mar 29 17:36 /usr/local/bin/llvm-ranlib -> llvm-ar

I don't plan to update the jit formatting jobs as part of this series of changes so my preference would be to leave out clang-format and clang-tidy for now. We can add them later when we update to recent formatting tools.

For lldb, I'd like to focus on just the dotnet/runtime build dependencies in this change. I think you're right that we will need it for dotnet/diagnostics and the runtime helix images, but I'd like to do that as a separate change if that sounds ok to you.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. 👍

make -j $(getconf _NPROCESSORS_ONLN) && \
make install

# Obtain arcade scripts used to build rootfs
RUN git config --global user.email builder@dotnet-buildtools-prereqs-docker && \
git clone --depth 1 --single-branch https://github.com/dotnet/arcade /scripts
Loading