How do I purge or prune old containerd images from Bottlerocket?

[trjate]

The disks on my k8s worker nodes are filling up with docker images, but I've noticed there is no docker client installed to use for pruning, so what's the recommended way for clearing out aging images?

[zmrow]

Hi @trjate - Bottlerocket includes the ctr command line tool. ctr doesn't include a "prune" command, but you can remove images via ctr --address "/run/dockershim.sock" image rm <image name>

Edit: fixed the address arg (thanks @etungsten! )

[trjate]

@zmrow Thanks Zac.

Follow-up question: do I need to change some initial launch setting because I'm currently getting a connection refused error?

bash-5.0# whoami
root
bash-5.0# pwd
/run/containerd
bash-5.0# ls
containerd.sock  fifo  io.containerd.grpc.v1.cri  io.containerd.runtime.v1.linux  io.containerd.runtime.v2.task  runc  s
bash-5.0# ctr --address "/run/containerd/containerd.sock" image ls
ctr: failed to dial "/run/containerd/containerd.sock": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: connection refused"

[etungsten]

The containerd socket on Bottlerocket is located at /run/dockershim.sock for compatibility purposes.

bottlerocket/packages/containerd/containerd-config-toml_k8s_nvidia

Line 13 in 7282bf5

address = "/run/dockershim.sock"

ctr --address "/run/dockershim.sock" -n k8s.io image ls should work.

[trjate]

Magnificent! Thank you both.

The other thing I was excluding was the --namespace flag.

Cheers.

[bcressey]

kubelet should be cleaning up unused container images rather than letting the filesystem fill up entirely. By default it will not run garbage collection if disk usage is below 80%, and it will always run it if disk usage is above 85%.

If this doesn't work for your use case, we might need to expose the relevant fields - imageGCLowThresholdPercent, imageGCHighThresholdPercent, imageMinimumGCAge - via the settings API. But it would be good to understand what's going wrong, and to make sure kubelet is working as expected, before trying to adjust its behavior.

[JohnnyBoy04]

I have been unable to find documentation on how to change the default from say 85 to 75. Any help would be welcomed.

[arnaldo2792]

Hi @JohnnyBoy04 , based on the default value you used (85) I assume you refer to the imageGCHighThresholdPercent setting, which can be changed via this API:

[settings.kubernetes]
image-gc-high-threshold-percent = 75

If you are setting both imageGCHighThresholdPercent and imageGCLowThresholdPercent, make sure the former is not lower then the latter and vice versa otherwise the kubelet will refuse to start.

[JohnnyBoy04]

You are correct this is what Im looking to change. Where do I actually make this change. Is it in some type of config file? yml file? I'm fairly new to all of this. I would imagine that where the changes take place depend on versions of the kubernetes cluster.

kubernetes version: v1.23.7+rke2r2 (rke2).
Using containerd

[jpmcb]

You can interact with bottlerocket's settings via either:

• The apiclient from within the host/admin container

$ apiclient set kubernetes.image-gc-high-threshold-percent=75

• Through TOML provided via the user-data:

[settings.kubernetes]
image-gc-high-threshold-percent = 75

Under our description of settings:

settings.kubernetes.image-gc-high-threshold-percent: The percent of disk usage after which image garbage collection is always run.

[arnaldo2792]

@JohnnyBoy04 , I apologize I assumed you were familiar with the Bottlerocket ecosystem 😅 . There are two ways to configure a Bottlerocket host:

• Through user data: this is a toml configuration file read on early stages of the boot and it is only read on first boot.
• Through apiclient (as @jpmcb commented): this is the entrypoint to configure Bottlerocket hosts on subsequent boots. To experiment and play around with the API, I recommend you access the host via the admin/control container and call apiclient set <settings>. Once you know what values work for your requirements, create a toml formatted user data file and use it to configure your fleet 👍 .

FWIW, there are a few things in the API that shouldn't be configured through user data but that may be required before the orchestrator starts the workloads (e.g custom CA certificates). For this type of configurations, we recommend the use of bootstrap containers and you can make use of apiclient from within this type of containers.

You can even combine user data + bootstrap containers if you wanted to! You can have a user data file with a single configuration for an essential bootstrap container that runs once, and use this bootstrap container to make multiple apiclient calls.

Please let us know if you have any feedback on the public documentation that can help you better understand the Bottlerocket environment.