model: container-registry.mirrors different settings representation

[etungsten]

Issue number:
Fixes #1780

Description of changes:

Author: Erikson Tung <etung@amazon.com>
Date:   Wed Oct 27 19:02:01 2021 -0700

    model: change `container-registry.mirrors` type
    
    Changes the model for `container-registry.mirrors` from a `HashMap` to a vector of
    structs.
    Adds a custom deserializer to deserialize from either a TOML table or a
    TOML sequence.

Author: Erikson Tung <etung@amazon.com>
Date:   Thu Oct 28 12:03:20 2021 -0700

    migrations: 'container-registry.mirrors' model type migration

Testing done:
Unit test passes.
On a live host, I can specify registry mirrors in both ways. Either as a map of registry to endpoints or an array of tables with registry and endpoints as object fields.

With the following userdata:

[[settings.container-registry.mirrors]]
registry = "*"
endpoint = ["https://registry-0.acme.com","https://registry-1.acme.com"]

[[settings.container-registry.mirrors]]
registry = "example"
endpoint = [ "hello", "hellohello"]

On K8s variants, the containerd config registry mirrors configuration gets rendered correctly:

bash-5.0# cat /var/lib/bottlerocket/datastore/current/live/settings/container-registry/mirrors 
[{"endpoint":["https://registry-0.acme.com","https://registry-1.acme.com"],"registry":"*"},{"endpoint":["hello","hellohello"],"registry":"example"}]

bash-5.0# cat /etc/containerd/config.toml 
...
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."*"]
endpoint = ["https://registry-0.acme.com", "https://registry-1.acme.com"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."example"]
endpoint = ["hello", "hellohello"]

On ECS variants, the docker daemon json file generated correctly for docker.io registry mirrors and docker runs fine:
userdata:

[[settings.container-registry.mirrors]]
registry = "docker.io"
endpoint = ["https://registry-0.acme.com","https://registry-1.acme.com"]

[[settings.container-registry.mirrors]]
registry = "example"
endpoint = [ "hello", "hellohello"]

docker daemon.json:

{
  "log-driver": "journald",
  "live-restore": true,
  "max-concurrent-downloads": 10,
  "storage-driver": "overlay2",
  "exec-opts": ["native.cgroupdriver=cgroupfs"],
  "data-root": "/var/lib/docker",
  "selinux-enabled": true,
  "default-ulimits": { "nofile": { "Name": "nofile", "Soft": 1024, "Hard": 4096 } }
      ,
  "registry-mirrors": ["https://registry-0.acme.com", "https://registry-1.acme.com"]
}

host-containers services registry mirrors config also gets generated correctly:

bash-5.0# cat /etc/host-containers/host-ctr.toml 
[mirrors."a"]
endpoints = ["aaaa", "bbb"]
[mirrors."b"]
endpoints = ["cccc", "dddd"]
[mirrors."docker.io"]
endpoints = ["https://registry-0.acme.com", "https://registry-1.acme.com"]

Test migration upgrades and downgrades and the container-registry.mirrors setting gets migrated successfully:
With the following userdata:

[settings.container-registry.mirrors]
"docker.io" = ["https://registry-0.acme.com","https://registry-1.acme.com"]
"a" = ["aaaa","bbb"]
"b" = ["cccc","dddd"]

On current version Bottlerocket:

bash-5.0# cat /var/lib/bottlerocket/datastore/current/live/settings/container-registry/mirrors/
a            b            docker%2Eio  
bash-5.0# cat /var/lib/bottlerocket/datastore/current/live/settings/container-registry/mirrors/a 
["aaaa","bbb"]
bash-5.0#cat /var/lib/bottlerocket/datastore/current/live/settings/container-registry/mirrors/docker%2Eio 
["https://registry-0.acme.com","https://registry-1.acme.com"]
bash-5.0# updog update -r -n
Starting update to 1.4.0
Cannot schedule shutdown without logind support, proceeding with immediate shutdown.
Update applied: aws-k8s-1.21 1.4.0

Checking after upgrade:

bash-5.0# cat /var/lib/bottlerocket/datastore/current/live/settings/container-registry/mirrors 
[{"endpoint":["cccc","dddd"],"registry":"b"},{"endpoint":["aaaa","bbb"],"registry":"a"},{"endpoint":["https://registry-0.acme.com",bash-5.0# signpost rollback-to-inactivey":"docker.io"}]

Checking containerd config:

bash-5.0# cat /etc/containerd/config.toml 
....

[plugins."io.containerd.grpc.v1.cri".registry.mirrors."a"]
endpoint = ["aaaa", "bbb"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."b"]
endpoint = ["cccc", "dddd"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-0.acme.com", "https://registry-1.acme.com"]

After downgrading back:

bash-5.0# cat /var/lib/bottlerocket/datastore/current/live/settings/container-registry/mirrors/docker%2Eio 
["https://registry-0.acme.com","https://registry-1.acme.com"]

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[tjkirch]

Nice, this seems like a friendlier solution.

[tjkirch]

LGTM assuming ECS testing is OK; couple small items remain above.

[etungsten]

Pushes above addresses @tjkirch 's comments. Adds more testing to the PR description.

[etungsten]

Push above removes snafu as a dependency from the migration.

[zmrow]

🔩

Wow nice work!

[etungsten]

Sorry about the repeat force pushes. Please take a look at the backward and forward migrations for the new changes.

Pushes above addresses @bcressey 's comment. I also realized cases where the migration would fail if the registry contained . in the name or if they were quoted so I had to adjust the forward migration to fix it. Tested the migrations fully and they work as expected.

Updated the PR description with updated migration testing info.

(important change to code)

(important change to code)

[etungsten]

push above addresses @tjkirch 's comments. I retested the migrations and they worked as expected. registry set to * gets dropped on downgrade:

bash-5.0# cat /etc/containerd/config.toml 
...
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."b"]
endpoint = ["cccc", "dddd"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."*"]
endpoint = ["aaaa", "bbb"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-0.acme.com", "https://registry-1.acme.com"]
bash-5.0# signpost rollback-to-inactive
bash-5.0# reboot
...

On next boot:

bash-5.0# cat /etc/containerd/config.toml 
...
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."b"]
endpoint = ["cccc", "dddd"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-0.acme.com", "https://registry-1.acme.com"]

[etungsten]

Push above removes an outdated comment and a redundant to_string() call

[zmrow]

🧦