Provide stronger isolation between experimental and standard release channels

[robscott]

I believe that the current release channel model is inherently flawed and is going to result in painful outages for some Gateway API users and the increased inaccessibility of experimental channel, particularly on managed clusters. Consider the following example of how things could go wrong:

1. Joe installs Gateway API v1.2 standard channel
2. Sue wants to use an experimental authorization feature so installs experimental HTTPRoute CRD
3. Joe upgrades to Gateway v1.3 using standard channel
4. Experimental authorization feature disappears - everything gets through to Sue’s app

This example is particularly relevant because authorization is very much in consideration for v1.2. In the example, everyone made rational decisions, and they still ended up with what would likely be a painful security incident.

To avoid these kinds of issues where installing standard channel CRDs can cause you to silently lose critical configuration, I propose separating experimental and standard channel resources by API group and/or resource name. This would allow both experimental and standard channel CRDs to safely coexist within a cluster without trampling over each other.

This discussion has been preceded by lots of well thought out discussion in both #2912 and today's community meeting that was exclusively focused on this topic. If you're interested in reading some of the prior discussion, I'd recommend reading through #2912. I'll also try to provide a high level summary of the outcomes of the meeting earlier today:

1. All options are awful. Each have problems, and we're stuck trying to determine which is the least bad. We all share the same goals of making experimental channel widely available and ensuring that Gateway API as a whole is as safe as possible by minimizing the room for unexpected/unpredictable failure modes.
2. The discussion in Splitting Experimental CRDs into separate API Group and Names #2912 has gotten hard to follow and it's likely more effective to transition to a GH discussion (that's why I'm creating this).
3. With the proposal described above, it would be fairly trivial for a controller to exclusively support one release channel, and only moderately complex to support one or the other independently - ie contour-experimental and contour-standard.
4. It would be quite challenging with the current k8s clients and apimachinery to concurrently support both standard and experimental channel.
5. It's ok if not every implementation supports experimental channel. We need a strong set of 5-10 implementations, but we don't need everyone. This would not represent any additional work for implementations that chose not to support experimental channel.
6. If we could somehow develop a library/tooling/something that could make this approach manageable for controllers, it seems like we could get at least 5-10 implementations on board with this.

The last point is the big one - I would not want to proceed with this kind of approach if it was completely unworkable from an implementation perspective. I think the minimum viable threshold for us to proceed with this approach would be if we had 5+ implementations that were on board with the idea (and ideally also no strong pushback from others).

[howardjohn]

Joe installs Gateway API v1.2 standard channel
Sue wants to use an experimental authorization feature so installs experimental HTTPRoute CRD
Joe upgrades to Gateway v1.3 using standard channel
Experimental authorization feature disappears - everything gets through to Sue’s app

Turn this into:

Joe installs Acme v1.2
Sue wants to use an new authorization feature so installs Acme v1.4 CRD
Joe isn't paying attention to Sue and installs Acme v1.3
Experimental authorization feature disappears - everything gets through to Sue’s app

And its now not specific to GW API and solution is clear IMO: expect cluster admins to do competent things with cluster-wide resources, and/or provide guardrails to ensure they do not do bad things. (guardrails could maybe be a validator that blocks downgrading a CRD or something I suppose...).

More contrived examples are "Joe runs kubectl delete all -A" 🙂

Why does the same not apply to Gateway API?

[howardjohn]

Strong enough guard rails can prevent the failure mode entirely though. A CEL validation policy could block installing standard on top of experimental, for instance.

[youngnick]

@howardjohn, so in the case we did add that CEL validation, to change from Experimental to Standard would require removing the CRDs from the cluster entirely, then adding them back?

[howardjohn]

Depends what you want the behavior for someone migrating to actually be. You could require removing the CEL policy (it would be in an external VAP resource, so it could be deleted) (probably a bad idea), add an annotation i-really-know-what-i-am-doing, delete + recreate, etc. Lots of options.

I don't necessarily think such guardrails are necessary, so its hard for me to pick one that makes much sense to me

[mikemorris]

A similar failure scenario not solved by splitting the experimental channel into a separate API group is the issue of breaking changes within the experimental channel possibly being disruptive - what if the experimental AuthZ feature added in 1.3 in the example given was removed in 1.4, and Joe later installs a 1.4 experimental HTTPRoute for some unrelated new feature?

I agree that with global CRDs, you really need centralized control over versioning and upgrades (e.g. managed by a vendor or a platform/infra team) if clusters are multi-tenant, or smaller single-tenant clusters where each team can more reasonably manage their own requirements.

An alternative approach to this problem would be recognizing that for multi-tenant cluster usage (multiple teams with different requirements) decentralized global CRD management is inherently problematic, and centralized management can quickly becoming limiting (by requiring significant coordination to upgrade). In the SIG-Multicluster meeting last week there was a presentation of an open "workspaces" project to group a set of tenant namespaces into a single workspace, and one of the eventual possibilities discussed was workspace-scoped CRDs, which could be an alternative path forward for this problem and complementary to validation tooling to allow individual teams in a multi-tenant cluster to move at their own pace.

[robscott]

what if the experimental AuthZ feature added in 1.3 in the example given was removed in 1.4, and Joe later installs a 1.4 experimental HTTPRoute for some unrelated new feature?

I don't think we can ever offer safe upgrades within experimental channel. My primary goal is to make sure that installing newer versions of standard channel CRDs is always safe. That could be accomplished with an API version split.

workspace-scoped CRDs, which could be an alternative path forward for this problem and complementary to validation tooling to allow individual teams in a multi-tenant cluster to move at their own pace

+1, that sounds like a great improvement for the broader Gateway API ecosystem.

A CEL validation policy could block installing standard on top of experimental, for instance

While I think adding guardrails like this with a validation policy could certainly help avoid some pain points, they could also reinforce the idea that you can get stuck in experimental channel with no way out. I can't think of a way for a ValidationAdmissionPolicy to actually check if any experimental/unknown fields are in use across a broad set of resources, so instead, I think we'd have to use some pretty broad restrictions which could trap a lot of users in experimental channel. In my opinion, I think it's important for us as a project to be able to offer some guarantees that installing the latest version of standard channel CRDs should always be safe, and I can't think of a way to do that with a ValidationAdmissionPolicy.