Add aws-k8s-1.21-nvidia variant #1799

arnaldo2792 · 2021-11-02T02:31:00Z

Issue number:
N / A

Description of changes:

Draft PR for aws-k8s-1.21-nvidia variant.

This PR adds the aws-k8s-1.21-nvidia variant, along with all the building blocks required to support NVIDIA GPUs in future releases. This variant includes all the configurations and software required to leverage GPUs in containerized workloads. In this variant, the default containerd runtime's name is nvidia, and uses the shimpei helper program as the OCI runtime binary.

kmod-5.10-nvidia

The drivers are subpackages of kmod-5.10-nvidia, since we don't want to have a spec file per driver type per driver version. The spec file for kmod-5.10-nvidia will hold the drivers that are compatible with the 5.10 kernel. New spec files should be added for newer kernel and driver versions. The kmod-5.10-nvidia package provides a tmpfilesd conf file to create the lib modules directory where the kernel modules will be created.

Each subpackage installs the libraries and binaries underneath %{_cross_bindir}/nvidia/<type> and %{_cross_libdir}/nvidia/<type> respectively to prevent collisions while building the subpackages. Kernel module objects are installed in %{_cross_datadir}/nvidia/<type>/modules, so that driverdog can link them at runtime.

Each subpackage provides a drop-in configuration file for containerd, that sets the NVIDIA_PATH environment variable. This environment variable must be set to the directory that contains the NVIDIA userland tools, which will be mounted on the containers by libnvidia-container. The environment variable is set for containerd, since libnvidia-container is called by the runtime to set up the containers.

variants: add aws-k8s-1.21-nvidia
actions: add matrix `fetch-upstream` variable
containerd: add configuration for NVIDIA runtime
packages: add NVIDIA tesla 470 driver

Remaining tasks:

Use oci-add-hooks instead of nvidia-container-runtime
Add documentation for driverdog
Check that Node Feature Discovery, GPU feature Discovery, and DCGM work as expected
Check aarch64 build works
Add documentation for variant

Testing done:
I launched p3/g4dn/g5g instances and deployed the vectoradd-cuda sample app, I had access to nvidia-smi and I ran the sample app:

apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
spec:
  restartPolicy: OnFailure
  containers:
  - name: cuda-vectoradd
    image: nvidia/samples:vectoradd-cuda11.2.1
    resources:
      limits:
         nvidia.com/gpu: 1
    command: ['sh', '-c', 'sleep infinity']

root@cuda-vectoradd:/# nvidia-smi
Tue Nov  2 01:59:52 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  Off  | 00000000:00:1E.0 Off |                    0 |
| N/A   34C    P0    22W / 300W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

root@cuda-vectoradd:/# /tmp/sample
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

arnaldo2792 · 2021-11-03T20:11:43Z

Forced push includes:

Cleaned containerd-config-toml_k8s, there were some leftover configurations
Remove nvidia-container-runtime and use shimpei instead
Modify containerd.service to read /etc/containerd/containerd.env file for additional env variables
Comments in kmod-5.10-nvidia spec file that explains the included files
Use shimpei + oci-add-hooks as the nvidia runtime for aws-k8s-1.21-nvidia
Documentation for driverdog
Cleaned driverdog, there were some leftover errors

arnaldo2792 · 2021-11-04T23:46:57Z

Forced push includes:

Update Github Actions to build the variant
Add missing symlinks for NVIDIA libraries, used by workloads like the GPU Feature Discovery
Add NVIDIA tesla 470 license to package
Add /etc/kubernetes/node-feature-discovery/features.d for GPU Feature Discovery to work
Remove unnecessary patch in libnvidia-container
Create the nvidia-container-toolkit configuration in /etc/nvidia-container-runtime/config.toml, otherwise it isn't read, even though it was specified with the -config flag

arnaldo2792 · 2021-11-05T00:35:17Z

Forced push includes:

Removed leftover packages from kmod-5.10-nvidia Cargo.toml
Removed driver-tmpfiles.conf leftover file from os package
Cleaned up kmod-5.10-nvidia spec file and driverdog Cargo.toml

packages/binutils/Cargo.toml

packages/binutils/binutils.spec

packages/containerd/containerd.spec

sources/driverdog/src/main.rs

packages/nvidia-container-toolkit/nvidia-container-toolkit-config.toml

samuelkarp · 2021-11-08T18:41:15Z

packages/release/lib-modules.mount.in

+What=overlay
+Where=PREFIX/lib/modules
+Type=overlay
+Options=noatime,nosuid,nodev,lowerdir=/lib/modules,upperdir=/var/lib/kernel-modules/upper,workdir=/var/lib/kernel-modules/work,context=system_u:object_r:state_t:s0


@bcressey should we add anything specific to the SELinux policy covering files in /var/lib/kernel-modules?

Yes, this would be good to do for all the host directories we create for overlayfs mechanics.

samuelkarp · 2021-11-08T21:24:07Z

sources/driverdog/src/main.rs

@@ -0,0 +1,359 @@
+/*!
+  driverdog is a tool to link kernel modules at runtime. It uses a toml configuration file with the following shape:


Why the extra indent? That shows up in the readme too.

samuelkarp · 2021-11-09T00:16:48Z

sources/shimpei/src/main.rs

+/// Path to hooks definition, provided by runtime "vendor"
+const HOOK_CONFIG_PATH: &str = "/usr/share/oci-add-hooks/hook.json";


Since this is actually vendor-specific, it might be worthwhile to hard-code the (only) vendor path we have today: /usr/share/oci-add-hooks/nvidia.json. When we get to the point of adding a second vendor, we'd need a second compiled version of shimpei pointing to that second path anyway.

I could make this change, but I would prefer if we don't hard-code the vendor specific name in shimpei. I understand that the contents of the hook.json file are vendor specific, but there won't be any case in which we have two "vendor runtimes" enabled in the same AMI, and shimpei doesn't really care about the hook that will be executed, it only knows that there is a hook.json file that oci-add-hooks will use.

The hook.json file can be provided by a kmod-<kernel>-<vendor> package, just like I did with kmod-5.10-nvidia, and we will save ourselves the N shimpei compilations, per vendor.

We could avoid a second copy of shimpei by doing the multicall / busybox thing where we inspect arg0 and see if we're called as nvidia-oci-hook, amdgpupro-hook, etc, and use the matching /usr/share/oci-add-hooks/<prefix>.json as our input.

Then packages that wanted to use shimpei could just install a symlink, e.g. /usr/bin/nvidia-oci-hook -> /usr/bin/shimpei.

packages/libnvidia-container/0004-Use-NVIDIA_PATH-to-look-up-binaries.patch

samuelkarp · 2021-11-09T00:23:53Z

packages/oci-add-hooks/oci-add-hooks.spec

+
+%build
+%cross_go_configure %{goimport}
+GO111MODULE=off go build -o oci-add-hooks -ldflags "-linkmode=external"


GO111MODULE=off go build -buildmode=pie -ldflags="-linkmode=external" -o oci-add-hooks .

samuelkarp · 2021-11-09T00:25:59Z

packages/containerd/containerd-config-toml_k8s_nvidia

+[plugins."io.containerd.grpc.v1.cri".containerd]
+default_runtime_name = "nvidia"


Why are we setting this as the default? Wouldn't the typical experience be to have it available as an option set through RuntimeClass?

The typical experience with EKS Optimzed AMI, CO'OS, and the GPU Operator is that the users don't have to set a RuntimeClass to run their workloads. This settings helps us have parity with at least the EKS Optimized AMI.

packages/containerd/containerd.spec

arnaldo2792 · 2021-11-11T20:02:40Z

Forced pushed includes:

Use the same binutils version as the one in the SDK, along with a check to prevent drifts
Optional nvidia.env file in containerd.service
Change dependency order for oci-add-hooks and nvidia-container-toolkit
Replaced /etc with %_corss_sysconfigdir in glibc.spec
Set IPMI_HANDLER as a module in the kernel config
Templated files in kmod-5.10-nvidia package
Leave just the bare minimum NVIDIA libraries for me to test, the libraries will be added in a different PR
Removed nvidia-persistenced, will be added in a different PR
Load i2c_core and impi_msghandler at boot with a modules-load.d drop-in
Changed driver's configuration file shape
Separate linking and loading kernel modules sets in driverdog, and use two subsequent oneshot services
Add node-feature-discovery/features.d to all k8s variants
Use the SDK's LD linker in libnvidia-containerd
Moved drop-in to delay ldconfig.service to kmod-5.10-nvidia

packages/binutils/binutils.spec

macros/shared

packages/binutils/binutils.spec

packages/nvidia-container-toolkit/nvidia-container-toolkit.spec

packages/oci-add-hooks/oci-add-hooks.spec

packages/os/link-kernel-modules.service

packages/os/load-kernel-modules.service

variants/aws-k8s-1.21-nvidia/Cargo.toml

arnaldo2792 · 2021-11-24T02:09:11Z

Forced push includes:

Remove variant from github actions
Fix both bintuils.spec description and licenses
Add a check to fail if the binutils version is different from the one in the SDK
Exclude files in bintuils.spec instead of deleting them
Change shimpei to use the 0 argument as a prefix to look for the <prefix>-hook.json file
Add the nvidia-oci symlink to shimpei in nvidia-container-toolkit
Update containerd's configuration file to use nvidia-oci as the runtime binary
Update libnvidia-container's licenses
Use the ld and strip binaries in the SDK
Install NVIDIA binaries in %{_cross_libexecdir}
Use templated files in kmod-5.10-nvidia
Build nvidia-drm but exclude it in kmod-5.10-nvidia
List all the objects instead of using the * in kmod-5.10-nvidia
Delay ldconfig.service in release.spec instead of kmod-5.10-nvidia
Move link-kernel-modules and load-kernel-modules to preconfigured.target
driverdog now reads all the files in /etc/drivers , instead of looking for one specific file
Add binutils as a dependency of release but only include it in variants that include driverdog

arnaldo2792 · 2021-11-25T00:47:24Z

Rebased develop to fix conflicts

bcressey

Reviewed everything except the kmod packaging and driverdog.

packages/containerd/containerd-config-toml_k8s_nvidia

packages/oci-add-hooks/oci-add-hooks.spec

packages/nvidia-container-toolkit/nvidia-container-toolkit.spec

packages/release/ldconfig-service.conf

sources/shimpei/src/main.rs

variants/aws-k8s-1.21-nvidia/Cargo.toml

packages/kmod-5.10-nvidia/nvidia-tesla-tmpfiles.conf.in

sources/driverdog/src/main.rs

packages/kmod-5.10-nvidia/nvidia-tesla-build-config.toml.in

packages/kmod-5.10-nvidia/kmod-5.10-nvidia.spec

arnaldo2792 · 2021-12-02T19:03:42Z

Forced push includes:

Fixed containerd's configuration file
Removed unused linked files from nvidia-5.10-kmod.spec file
Simplified tmpfilesd conf file to copy ld.so.conf.d files
Added underscores to templated variables
Refactored driverdog to simplify build configuration file
Removed binutils and nvidia-container-toolkit from specs where they aren't needed
Added missing build flag in oci-add-hooks spec file
Use expect in shimpei constant strings

arnaldo2792 · 2022-01-20T03:06:29Z

Forced push to rebase changes in libnvidia-container + k8s-device plugin

arnaldo2792 · 2022-01-20T23:17:37Z

Forced push to fix build for aarch64 + add k8s device plugin to nvidia variant

bcressey · 2022-01-20T23:27:06Z

.github/workflows/build.yml

        include:
          - variant: aws-dev
            arch: x86_64
            supported: false
+            fetch-upstream: "false"


Do we have to override this for each item in the list, even if the default isn't changing?

I could try removing it and see if it works 👍

It didn't 😄 , I changed it back per our offline conversation.

bcressey · 2022-01-21T00:14:35Z

packages/kmod-5.10-nvidia/kmod-5.10-nvidia.spec

+%global spdx_id %(bottlerocket-license-tool -l $PWD/rpmbuild/BUILD/Licenses.toml spdx-id nvidia)
+%global license_file %(bottlerocket-license-tool -l $PWD/rpmbuild/BUILD/Licenses.toml path nvidia -p ./licenses)


Suggested change

%global spdx_id %(bottlerocket-license-tool -l $PWD/rpmbuild/BUILD/Licenses.toml spdx-id nvidia)

%global license_file %(bottlerocket-license-tool -l $PWD/rpmbuild/BUILD/Licenses.toml path nvidia -p ./licenses)

%global spdx_id %(bottlerocket-license-tool -l %{_builddir}/Licenses.toml spdx-id nvidia)

%global license_file %(bottlerocket-license-tool -l %{_builddir}/Licenses.toml path nvidia -p ./licenses)

bcressey · 2022-01-21T00:20:36Z

packages/kmod-5.10-nvidia/kmod-5.10-nvidia.spec

+# NVIDIA .run scripts from 0 to 199
+Source0: https://us.download.nvidia.com/tesla/%{nvidia_tesla_470_version}/NVIDIA-Linux-%{_cross_arch}-%{nvidia_tesla_470_version}.run


nit: it is considered bad form to conditionally include files like this in the spec file since then a source RPM won't be complete - whether it includes the x86 or arm64 run file will depend on what it was built for rather than what the end user of the srpm wants to build for.

In the context of our distro, it's fine since we don't build source RPMs, but I'd still prefer to include both .run files and conditionally invoke the right one, just as a stylistic twitch.

packages/kmod-5.10-nvidia/kmod-5.10-nvidia.spec

bcressey · 2022-01-21T00:26:52Z

packages/kmod-5.10-nvidia/nvidia-tmpfiles.conf.in

@@ -0,0 +1 @@
+D /lib/modules/KERNEL_VERSION/kernel/drivers/extra/video/nvidia/tesla 0755 root root -


nit:

Suggested change

D /lib/modules/KERNEL_VERSION/kernel/drivers/extra/video/nvidia/tesla 0755 root root -

D /lib/modules/__KERNEL_VERSION__/kernel/drivers/extra/video/nvidia/tesla 0755 root root -

bcressey · 2022-01-21T00:28:40Z

packages/kmod-5.10-nvidia/kmod-5.10-nvidia.spec

+install -d %{buildroot}%{_cross_factorydir}%{_cross_sysconfdir}/{drivers,ld.so.conf.d}
+
+KERNEL_VERSION=$(cat %{kernel_sources}/include/config/kernel.release)
+sed -e "s|KERNEL_VERSION|$KERNEL_VERSION|" %{S:200} > nvidia.conf


obligatory nit:

Suggested change

sed -e "s|KERNEL_VERSION|$KERNEL_VERSION|" %{S:200} > nvidia.conf

sed -e "s|KERNEL_VERSION|${KERNEL_VERSION}|" %{S:200} > nvidia.conf

packages/kmod-5.10-nvidia/kmod-5.10-nvidia.spec

bcressey · 2022-01-21T00:32:27Z

packages/kmod-5.10-nvidia/kmod-5.10-nvidia.spec

+install -m 755 nvidia-ngx-updater %{buildroot}%{_cross_libexecdir}/nvidia/tesla/bin/%{nvidia_tesla_470_version}
+%endif
+
+# TODO: add remaining libraries once we implement "image per variant" in the rpm2img script


Suggested change

# TODO: add remaining libraries once we implement "image per variant" in the rpm2img script

# TODO: add remaining libraries

bcressey · 2022-01-21T00:32:49Z

packages/kmod-5.10-nvidia/kmod-5.10-nvidia.spec

+# TODO: add remaining libraries once we implement "image per variant" in the rpm2img script
+# misc
+# Add libnvidia-ml.so for testing purposes
+install  -m755 libnvidia-ml.so.%{nvidia_tesla_470_version} %{buildroot}%{_cross_libdir}/nvidia/tesla/%{nvidia_tesla_470_version}


nit:

Suggested change

install -m755 libnvidia-ml.so.%{nvidia_tesla_470_version} %{buildroot}%{_cross_libdir}/nvidia/tesla/%{nvidia_tesla_470_version}

install -m755 libnvidia-ml.so.%{nvidia_tesla_470_version} %{buildroot}%{_cross_libdir}/nvidia/tesla/%{nvidia_tesla_470_version}

arnaldo2792 · 2022-01-21T02:09:23Z

Forced push includes documentation changes

arnaldo2792 · 2022-01-21T02:23:31Z

Forced push to rebase the upstream

bcressey · 2022-01-21T19:26:27Z

BUILDING.md

+1. Fetch the drivers `.run` archive from the URL provided in the [kmod-5.10-nvidia package](packages/kmod-5.10-nvidia/Cargo.toml), i. e.:
+
+```shell
+curl -LO https://us.download.nvidia.com/tesla/470.82.01/NVIDIA-Linux-x86_64-470.82.01.run
+```
+
+2. Extract the sources and copy the `LICENSE` file to the `licenses` directory in your Bottlerocket root directory:


It seems cleaner just to document using the HTML license file; then we won't have driver links that go out of date.

bcressey · 2022-01-21T19:28:06Z

QUICKSTART-EKS.md

+Also be aware that when operating in GovCloud the IAM ARNs will need to be updated to the following: `arn:aws-us-gov`.
+For example:
+ `arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy`
+ will be updated to:


nit: I like the cleanup but I tend to want it in a separate commit - can still be in this PR though

bcressey · 2022-01-21T19:30:38Z

QUICKSTART-EKS.md

+
+The `aws-k8s-1.21-nvidia` variant includes the required packages and configurations to leverage NVIDIA GPUs.
+It comes with the [NVIDIA Tesla driver](https://docs.nvidia.com/datacenter/tesla/drivers/index.html) along with the libraries required by the [CUDA toolkit](https://developer.nvidia.com/cuda-toolkit) included in your orchestrated containers.
+It also includes the [NVIDIA k8s device plugin](https://github.com/NVIDIA/k8s-device-plugin), so please make sure you don't have a daemonset running the device plugin in your cluster before launching a new node using this variant.


Suggested change

It also includes the [NVIDIA k8s device plugin](https://github.com/NVIDIA/k8s-device-plugin), so please make sure you don't have a daemonset running the device plugin in your cluster before launching a new node using this variant.

It also includes the [NVIDIA k8s device plugin](https://github.com/NVIDIA/k8s-device-plugin).

If you already have a daemonset for the device plugin in your cluster, you may need to use taints and tolerations to keep it from running on Bottlerocket nodes.

bcressey · 2022-01-21T19:32:55Z

QUICKSTART-EKS.md

+It comes with the [NVIDIA Tesla driver](https://docs.nvidia.com/datacenter/tesla/drivers/index.html) along with the libraries required by the [CUDA toolkit](https://developer.nvidia.com/cuda-toolkit) included in your orchestrated containers.
+It also includes the [NVIDIA k8s device plugin](https://github.com/NVIDIA/k8s-device-plugin), so please make sure you don't have a daemonset running the device plugin in your cluster before launching a new node using this variant.
+
+With this variant, most of the existing NVIDIA tools (like [DCGM](https://github.com/NVIDIA/dcgm-exporter) and the [GPU Feature Discovery](https://github.com/NVIDIA/gpu-feature-discovery)) work as you would expect, you can install them in your cluster following the instructions provided for each project.


Suggested change

With this variant, most of the existing NVIDIA tools (like [DCGM](https://github.com/NVIDIA/dcgm-exporter) and the [GPU Feature Discovery](https://github.com/NVIDIA/gpu-feature-discovery)) work as you would expect, you can install them in your cluster following the instructions provided for each project.

Additional NVIDIA tools such as [DCGM](https://github.com/NVIDIA/dcgm-exporter) and [GPU Feature Discovery](https://github.com/NVIDIA/gpu-feature-discovery) will work as expected.

You can install them in your cluster by following the `helm install` instructions provided for each project.

bcressey · 2022-01-21T19:39:06Z

QUICKSTART-EKS.md

+It also includes the [NVIDIA k8s device plugin](https://github.com/NVIDIA/k8s-device-plugin), so please make sure you don't have a daemonset running the device plugin in your cluster before launching a new node using this variant.
+
+With this variant, most of the existing NVIDIA tools (like [DCGM](https://github.com/NVIDIA/dcgm-exporter) and the [GPU Feature Discovery](https://github.com/NVIDIA/gpu-feature-discovery)) work as you would expect, you can install them in your cluster following the instructions provided for each project.
+Even though you could use the [GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/getting-started.html#install-nvidia-gpu-operator) to install these tools, we recommend installing these tools individually via `helm install` since the Operator could break your workloads on upgrades.


This is hard to understand without more details. I'd suggest saying something like:

Suggested change

Even though you could use the [GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/getting-started.html#install-nvidia-gpu-operator) to install these tools, we recommend installing these tools individually via `helm install` since the Operator could break your workloads on upgrades.

The [GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/getting-started.html#install-nvidia-gpu-operator) can also be used to install these tools.

However, it is cumbersome to select the right subset of features to avoid conflicts with the software included in the variant.

Therefore we recommend installing the tools individually if they are required.

bcressey · 2022-01-21T19:41:30Z

README.md

@@ -54,6 +54,9 @@ The following variants support EKS, as described above:
 - `aws-k8s-1.19`
 - `aws-k8s-1.20`
 - `aws-k8s-1.21`
+- `aws-k8s-1.21-nvidia`
+
+Please refer to [this document](QUICKSTART-EKS.md#aws-k8s-121-nvidia-variant) to learn more about the `aws-k8s-1.21-nvidia` variant.


I'd prefer not to link to variant specifics here, or else we'll be doing that everywhere on this page. In general, we need to figure out a more structured approach to getting started with particular variants, since this isn't scaling very well.

bcressey · 2022-01-21T19:43:09Z

SECURITY_FEATURES.md

@@ -134,6 +134,8 @@ All binaries are linked with the following options:

 Together these enable [full RELRO support](https://www.redhat.com/en/blog/hardening-elf-binaries-using-relocation-read-only-relro) which makes [ROP](https://en.wikipedia.org/wiki/Return-oriented_programming) attacks more difficult to execute.

+**Note:** the `aws-k8s-1.21-nvidia` variant includes the NVIDIA k8s device plugin which is compiled without the `-wl,-z,now` flags, and other precompiled NVIDIA libraries that could haven't been compiled with hardening flags.


Suggested change

**Note:** the `aws-k8s-1.21-nvidia` variant includes the NVIDIA k8s device plugin which is compiled without the `-wl,-z,now` flags, and other precompiled NVIDIA libraries that could haven't been compiled with hardening flags.

**Note:** Certain variants, such as the ones for NVIDIA, include precompiled binaries that may not have been built with these hardening flags.

There are some rpm commands in the Dockerfile executed as 'root', which results in macros like `%{_builddir}` to point to `/root` instead of `/home/builder`. By changing `%_topdir` to `/home/builder`, macros that rely on this value will always point to the build directory. Signed-off-by: Arnaldo Garcia Rincon <agarrcia@amazon.com>

arnaldo2792 · 2022-01-24T18:35:53Z

Forced push uses %{_builddir} macro in kmod-5.10-nvidia spec

bcressey · 2022-01-24T18:44:40Z

BUILDING.md

+cargo make fetch-licenses -e BUILDSYS_UPSTREAM_LICENSES_FETCH=true
+```
+
+3. Build your image, setting the `BUILDSYS_UPSTREAM_SOURCE_FALLBACK` flag to `true`, if you haven't cache the driver's sources:


Suggested change

3. Build your image, setting the `BUILDSYS_UPSTREAM_SOURCE_FALLBACK` flag to `true`, if you haven't cache the driver's sources:

3. Build your image, setting the `BUILDSYS_UPSTREAM_SOURCE_FALLBACK` flag to `true`, if you haven't cached the driver's sources:

This commit adds the NVIDIA tesla 470 driver. The drivers are subpackages of `kmod-5.10-nvidia`, since we don't want to have a spec file per driver type per driver version. The spec file for `kmod-5.10-nvidia` holds the drivers that are compatible with the 5.10 kernel. New spec files should be added for newer kernel and driver versions. The `kmod-5.10-nvidia` package provides a tmpfilesd conf file to create the lib modules directory where the kernel modules will be created. Each subpackage installs the libraries and binaries underneath `%{_cross_bindir}/nvidia/<type>` and `%{_cross_libdir}/nvidia/<type>` respectively to prevent collisions while building the subpackages. Kernel module objects are installed in `%{_cross_datadir}/nvidia/<type>/modules`, so that `driverdog` can compile them at runtime. Each subpackage provides a drop-in configuration file for containerd, that sets the `NVIDIA_PATH` environment variable. This environment variable must be set to the directory that contains the NVIDIA userland tools, which will be mounted on the containers by `libnvidia-container`. The environment variable is set for containerd, since `libnvidia-container` is called by the runtime to set up the containers. Signed-off-by: Arnaldo Garcia Rincon <agarrcia@amazon.com>

The *-nvidia variants use a different runtime instead of runc to inject prestart hooks using shimpei Signed-off-by: Arnaldo Garcia Rincon <agarrcia@amazon.com>

The `fetch-upstream` variable is be used to fetch upstream sources when they aren't provided in the lookaside cache. Signed-off-by: Arnaldo Garcia Rincon <agarrcia@amazon.com>

Signed-off-by: Arnaldo Garcia Rincon <agarrcia@amazon.com>

arnaldo2792 · 2022-01-24T19:32:03Z

Forced push fixes:

KERNEL_VERSION to KERNEL_VERSION
Type in BUILDING.md

jpculp

We should include a releases-url, but not necessarily this PR.

jpculp · 2022-01-24T20:40:13Z

.github/workflows/build.yml

@@ -27,30 +27,59 @@ jobs:
        variant: [aws-k8s-1.18, aws-k8s-1.19, aws-k8s-1.20, aws-k8s-1.21, aws-ecs-1]
        arch: [x86_64, aarch64]
        supported: [true]
+        fetch-upstream: ["false"]


nit. our previous booleans (and values) were unquoted, but these ones are quoted. I think I would unquote these for consistency.

The previous booleans values were used within the context of the github workflows, which are literal booleans. Since GH worklflows are defined in YAML, I can't use false and expect it to get translated to "false", I could get a 1, or another value that doesn't work for the environment variables passed to cargo make

zmrow

🌃

arnaldo2792 requested a review from bcressey November 2, 2021 02:31

arnaldo2792 force-pushed the nvidia-variant branch from 41ff0f8 to 95a3bdf Compare November 3, 2021 20:04

arnaldo2792 force-pushed the nvidia-variant branch from 95a3bdf to aab3bcd Compare November 4, 2021 23:41

arnaldo2792 force-pushed the nvidia-variant branch from aab3bcd to d5fc8a0 Compare November 5, 2021 00:32

bcressey reviewed Nov 5, 2021

View reviewed changes

samuelkarp reviewed Nov 9, 2021

View reviewed changes

arnaldo2792 force-pushed the nvidia-variant branch from d5fc8a0 to 7ced1ff Compare November 11, 2021 19:51

arnaldo2792 requested review from bcressey and samuelkarp November 11, 2021 20:04

bcressey reviewed Nov 12, 2021

View reviewed changes

arnaldo2792 mentioned this pull request Nov 12, 2021

Alternate image layout in rpm2img #1821

Closed

arnaldo2792 force-pushed the nvidia-variant branch from 7ced1ff to 9bfdfb8 Compare November 24, 2021 01:55

arnaldo2792 force-pushed the nvidia-variant branch from 9bfdfb8 to 6e3af9e Compare November 25, 2021 00:46

arnaldo2792 requested a review from bcressey November 29, 2021 18:03

bcressey reviewed Nov 29, 2021

View reviewed changes

bcressey reviewed Nov 30, 2021

View reviewed changes

packages/kmod-5.10-nvidia/kmod-5.10-nvidia.spec Outdated Show resolved Hide resolved

bcressey mentioned this pull request Nov 30, 2021

system directories should be owned by root bottlerocket-os/bottlerocket-sdk#64

Closed

arnaldo2792 force-pushed the nvidia-variant branch from 6e3af9e to 3780bbb Compare December 2, 2021 18:58

arnaldo2792 requested a review from bcressey December 2, 2021 19:04

This was referenced Dec 9, 2021

libelf: propagate zlib development dependency #1859

Merged

Mount /lib/modules as overlay filesystem #1860

Merged

Change when ldconfig runs and what files it looks for #1861

Merged

arnaldo2792 force-pushed the nvidia-variant branch from 657d8e1 to 93d010e Compare January 20, 2022 23:15

arnaldo2792 force-pushed the nvidia-variant branch from 93d010e to 2a7cbdb Compare January 20, 2022 23:25

bcressey approved these changes Jan 21, 2022

View reviewed changes

arnaldo2792 force-pushed the nvidia-variant branch from 2a7cbdb to 9182ec2 Compare January 21, 2022 02:09

arnaldo2792 marked this pull request as ready for review January 21, 2022 02:11

arnaldo2792 force-pushed the nvidia-variant branch from 9182ec2 to 69ee262 Compare January 21, 2022 02:22

arnaldo2792 requested a review from bcressey January 21, 2022 18:53

bcressey reviewed Jan 21, 2022

View reviewed changes

arnaldo2792 force-pushed the nvidia-variant branch from 69ee262 to 36e2eff Compare January 24, 2022 02:33

arnaldo2792 force-pushed the nvidia-variant branch from 36e2eff to 06847f1 Compare January 24, 2022 18:30

arnaldo2792 force-pushed the nvidia-variant branch from 06847f1 to d9c1e57 Compare January 24, 2022 18:52

bcressey reviewed Jan 24, 2022

View reviewed changes

arnaldo2792 added 4 commits January 24, 2022 19:10

containerd: add configuration for NVIDIA runtime

75541bc

The *-nvidia variants use a different runtime instead of runc to inject prestart hooks using shimpei Signed-off-by: Arnaldo Garcia Rincon <agarrcia@amazon.com>

actions: add matrix fetch-upstream variable

8896f88

The `fetch-upstream` variable is be used to fetch upstream sources when they aren't provided in the lookaside cache. Signed-off-by: Arnaldo Garcia Rincon <agarrcia@amazon.com>

variants: add aws-k8s-1.21-nvidia

54415cb

Signed-off-by: Arnaldo Garcia Rincon <agarrcia@amazon.com>

arnaldo2792 force-pushed the nvidia-variant branch from d9c1e57 to 54415cb Compare January 24, 2022 19:31

bcressey approved these changes Jan 24, 2022

View reviewed changes

jpculp approved these changes Jan 24, 2022

View reviewed changes

zmrow approved these changes Jan 26, 2022

View reviewed changes

arnaldo2792 merged commit 8772387 into bottlerocket-os:develop Jan 26, 2022

arnaldo2792 deleted the nvidia-variant branch January 26, 2022 21:38

bcressey mentioned this pull request Jun 28, 2022

investigate gVisor support #811

Open

5 tasks

		@@ -0,0 +1,359 @@
		/*!
		driverdog is a tool to link kernel modules at runtime. It uses a toml configuration file with the following shape:

		/// Path to hooks definition, provided by runtime "vendor"
		const HOOK_CONFIG_PATH: &str = "/usr/share/oci-add-hooks/hook.json";

		[plugins."io.containerd.grpc.v1.cri".containerd]
		default_runtime_name = "nvidia"

		%global spdx_id %(bottlerocket-license-tool -l $PWD/rpmbuild/BUILD/Licenses.toml spdx-id nvidia)
		%global license_file %(bottlerocket-license-tool -l $PWD/rpmbuild/BUILD/Licenses.toml path nvidia -p ./licenses)

		# NVIDIA .run scripts from 0 to 199
		Source0: https://us.download.nvidia.com/tesla/%{nvidia_tesla_470_version}/NVIDIA-Linux-%{_cross_arch}-%{nvidia_tesla_470_version}.run

		@@ -0,0 +1 @@
		D /lib/modules/KERNEL_VERSION/kernel/drivers/extra/video/nvidia/tesla 0755 root root -

	D /lib/modules/KERNEL_VERSION/kernel/drivers/extra/video/nvidia/tesla 0755 root root -
	D /lib/modules/__KERNEL_VERSION__/kernel/drivers/extra/video/nvidia/tesla 0755 root root -

	sed -e "s\|KERNEL_VERSION\|$KERNEL_VERSION\|" %{S:200} > nvidia.conf
	sed -e "s\|KERNEL_VERSION\|${KERNEL_VERSION}\|" %{S:200} > nvidia.conf

	# TODO: add remaining libraries once we implement "image per variant" in the rpm2img script
	# TODO: add remaining libraries

	install -m755 libnvidia-ml.so.%{nvidia_tesla_470_version} %{buildroot}%{_cross_libdir}/nvidia/tesla/%{nvidia_tesla_470_version}
	install -m755 libnvidia-ml.so.%{nvidia_tesla_470_version} %{buildroot}%{_cross_libdir}/nvidia/tesla/%{nvidia_tesla_470_version}

	It also includes the [NVIDIA k8s device plugin](https://github.com/NVIDIA/k8s-device-plugin), so please make sure you don't have a daemonset running the device plugin in your cluster before launching a new node using this variant.
	It also includes the [NVIDIA k8s device plugin](https://github.com/NVIDIA/k8s-device-plugin).
	If you already have a daemonset for the device plugin in your cluster, you may need to use taints and tolerations to keep it from running on Bottlerocket nodes.

	With this variant, most of the existing NVIDIA tools (like [DCGM](https://github.com/NVIDIA/dcgm-exporter) and the [GPU Feature Discovery](https://github.com/NVIDIA/gpu-feature-discovery)) work as you would expect, you can install them in your cluster following the instructions provided for each project.
	Additional NVIDIA tools such as [DCGM](https://github.com/NVIDIA/dcgm-exporter) and [GPU Feature Discovery](https://github.com/NVIDIA/gpu-feature-discovery) will work as expected.
	You can install them in your cluster by following the `helm install` instructions provided for each project.

-Even though you could use the [GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/getting-started.html#install-nvidia-gpu-operator) to install these tools, we recommend installing these tools individually via `helm install` since the Operator could break your workloads on upgrades.
+The [GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/getting-started.html#install-nvidia-gpu-operator) can also be used to install these tools.
+However, it is cumbersome to select the right subset of features to avoid conflicts with the software included in the variant.
+Therefore we recommend installing the tools individually if they are required.

		@@ -134,6 +134,8 @@ All binaries are linked with the following options:

		Together these enable [full RELRO support](https://www.redhat.com/en/blog/hardening-elf-binaries-using-relocation-read-only-relro) which makes [ROP](https://en.wikipedia.org/wiki/Return-oriented_programming) attacks more difficult to execute.

		Note: the `aws-k8s-1.21-nvidia` variant includes the NVIDIA k8s device plugin which is compiled without the `-wl,-z,now` flags, and other precompiled NVIDIA libraries that could haven't been compiled with hardening flags.

	Note: the `aws-k8s-1.21-nvidia` variant includes the NVIDIA k8s device plugin which is compiled without the `-wl,-z,now` flags, and other precompiled NVIDIA libraries that could haven't been compiled with hardening flags.
	Note: Certain variants, such as the ones for NVIDIA, include precompiled binaries that may not have been built with these hardening flags.

	3. Build your image, setting the `BUILDSYS_UPSTREAM_SOURCE_FALLBACK` flag to `true`, if you haven't cache the driver's sources:
	3. Build your image, setting the `BUILDSYS_UPSTREAM_SOURCE_FALLBACK` flag to `true`, if you haven't cached the driver's sources:

Add aws-k8s-1.21-nvidia variant #1799

Add aws-k8s-1.21-nvidia variant #1799

Conversation

arnaldo2792 commented Nov 2, 2021 • edited Loading

kmod-5.10-nvidia

arnaldo2792 commented Nov 3, 2021

arnaldo2792 commented Nov 4, 2021

arnaldo2792 commented Nov 5, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arnaldo2792 commented Nov 11, 2021

arnaldo2792 commented Nov 24, 2021

arnaldo2792 commented Nov 25, 2021

bcressey left a comment

Choose a reason for hiding this comment

arnaldo2792 commented Dec 2, 2021

arnaldo2792 commented Jan 20, 2022

arnaldo2792 commented Jan 20, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arnaldo2792 commented Jan 21, 2022

arnaldo2792 commented Jan 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arnaldo2792 commented Jan 24, 2022

Choose a reason for hiding this comment

arnaldo2792 commented Jan 24, 2022

jpculp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zmrow left a comment

Choose a reason for hiding this comment

arnaldo2792 commented Nov 2, 2021 •

edited

Loading

arnaldo2792 commented Jan 21, 2022 •

edited

Loading