V1: Prevent configs from becoming outdated when scaling down

[chrisseto]

0dd40b5 move setup-envtest to nix

This commit moves the setup-envtest shell scripts downloaded by
taskfile to the go binary equivalent and instead utilizes nix to install
it.

c84a283 V1: Prevent configs from becoming outdated when scaling down

Previously a collection of configurations (Seed Brokers, PandaProxy
Client, and SchemaRegistry Client) could become out of date when scaling
a cluster down. For example 5 replicas to 3. A special case of the
config hasher would ignore these fields to prevent needless restarts in
the case of scaling up which left stale values when scaling down. These
stale configurations would prevent various components of redpanda from
working correctly.

To mitigate these issues, this commit:

• Changes the PandaProxy and SchemaRegistry clients configurations to
  instead always point at redpanda's headless service.
• Caps the list of seed brokers to a maximum of 3.

While these changes will prevent needless restarts in the future, the
upgrade to this new configuration WILL result in a single rolling
restart.

Fixes #125

[BenPope]

When we get the set of replicas, are they in any particular order?

1. When scaling down, will the first replicas in this list be the ones that remain?
2. When doing a rolling upgrade of infrastructure (e.g., new K8s version), will this list be sufficient if the nodes are all going away?
3. Is cr.EmptySeedStartsCluster = False?
4. How does this work when there is multi-AZ?
5. I don't remember what happens when I node in the seed_server list can't be found, is that why the list has been truncated to 3?

[chrisseto]

When scaling down, will the first replicas in this list be the ones that remain?

Yes, this list will always be replicas 0, 1, and 2. Thanks to the STS's ordering guarantees we can rely on them always being around.

When doing a rolling upgrade of infrastructure (e.g., new K8s version), will this list be sufficient if the nodes are all going away?

The listed nodes will always exist (again STS ordinal system). It should be fine as long as nodes are migrated one at a time and this wouldn't really function any differently than what was previously present. Especially in the case of 3 node clusters.

Is cr.EmptySeedStartsCluster = False?

That's handled on the line above and it looks like it's always set to false if it's a supported redpanda version.

How does this work when there is multi-AZ?

I don't really see how that's relevant but it would function the same way it previously worked. This logic only capped the length of the list. The contents is otherwise computed the exact same way.

[chrisseto]

The failure is from decommission-on-delete :/ I'll try running it a few more times but It might be time to skip that test and toss an issue in the backlog.

[RafalKorepta]

Currently kuttl/harness/additional-configuration end to end test is failing.

https://buildkite.com/redpanda/redpanda-operator/builds/1440#018f7b4f-7f3a-45da-80a2-cf015134e2a2/1262-4470

logger.go:42: 09:41:07 \| additional-configuration/3-verify-config-changed \| Difference:
--
  | logger.go:42: 09:41:07 \| additional-configuration/3-verify-config-changed \| 13c13
  | logger.go:42: 09:41:07 \| additional-configuration/3-verify-config-changed \| <   - address: additional-configuration.redpanda-system.svc.cluster.local.
  | logger.go:42: 09:41:07 \| additional-configuration/3-verify-config-changed \| ---
  | logger.go:42: 09:41:07 \| additional-configuration/3-verify-config-changed \| >   - address: additional-configuration-0.additional-configuration.redpanda-system.svc.cluster.local.

It should be fixed with 1844219

[chrisseto]

Dang! I missed that one. Thanks for fixing!