Best way to update registries.yaml

[boris-savic]

On our development cluster, we re-deploy our "latest-dev" tag quite a few times a day. We have automated this via CI/CD pipelines, but with k3s we face a problem, that containerd seems to cache the "latest-dev" image and never pull it from the remote registry.

After some research it seems to me, that the issue originats from the default config always deployed to worker nodes inside worker_install_script.sh

cat > /etc/rancher/k3s/registries.yaml <<EOF
mirrors:
  "*":
EOF

I tried manually deleting the file from a few nodes and restarting k3s and it seems to me that in that case the new "latest-dev" image is being pulled.

I do have image pull policy set to Always on the Deployment, but that doesnt seem to work without going in and deleting the registries.yaml file.

Has anyone faced similar problem and what would be the best way to automate the configuration of the registries.yaml to not cache the images?

[vitobotta]

Hi, just updating the same tag when code changes is widely considered a bad practice for exactly this reasons. Images are supposed to be cached, and the correct way of handling updates is to change the tag. So I would recommend against making any changes to the registries configuration and instead change your CI/CD pipelines to properly set a new tag for new images. A common practice is to use the Git commit SHA as the tag or just the Git tag if you use those, which has the side benefit of uniquely linking a specific container image to a specific version of the code in your Git management tool.