The future of system CRDs and how they're made available to workspaces

[ncdc]

We will reserve (although it's not documented) kcp.dev CRDs for kcp system use (when #772 is implemented). We put all the system CRDs in the system:system-crds logical cluster. To avoid copying them into all the other logical clusters that use them, we have custom CRD handling logic to make system CRDs available in certain types of workspaces. Which workspaces see which system CRDs is currently hard-coded:

kcp/pkg/server/apiextensions.go

Lines 75 to 99 in f4f4a73

	rootCRDs: sets.NewString(
	clusters.ToClusterAwareKey(SystemCRDLogicalCluster, "clusterworkspaces.tenancy.kcp.dev"),
	clusters.ToClusterAwareKey(SystemCRDLogicalCluster, "clusterworkspacetypes.tenancy.kcp.dev"),
	clusters.ToClusterAwareKey(SystemCRDLogicalCluster, "clusterworkspaceshards.tenancy.kcp.dev"),
	// the following is installed to get discovery and OpenAPI right. But it is actually
	// served by a native rest storage, projecting the clusterworkspaces.
	clusters.ToClusterAwareKey(SystemCRDLogicalCluster, "workspaces.tenancy.kcp.dev"),
	),
	orgCRDs: sets.NewString(
	clusters.ToClusterAwareKey(SystemCRDLogicalCluster, "clusterworkspaces.tenancy.kcp.dev"),
	clusters.ToClusterAwareKey(SystemCRDLogicalCluster, "clusterworkspacetypes.tenancy.kcp.dev"),
	// the following is installed to get discovery and OpenAPI right. But it is actually
	// served by a native rest storage, projecting the clusterworkspaces.
	clusters.ToClusterAwareKey(SystemCRDLogicalCluster, "workspaces.tenancy.kcp.dev"),
	),
	universalCRDs: sets.NewString(
	clusters.ToClusterAwareKey(SystemCRDLogicalCluster, "apiresourceimports.apiresource.kcp.dev"),
	clusters.ToClusterAwareKey(SystemCRDLogicalCluster, "negotiatedapiresources.apiresource.kcp.dev"),
	clusters.ToClusterAwareKey(SystemCRDLogicalCluster, "workloadclusters.workload.kcp.dev"),
	clusters.ToClusterAwareKey(SystemCRDLogicalCluster, "apiexports.apis.kcp.dev"),
	clusters.ToClusterAwareKey(SystemCRDLogicalCluster, "apibindings.apis.kcp.dev"),
	clusters.ToClusterAwareKey(SystemCRDLogicalCluster, "apiresourceschemas.apis.kcp.dev"),
	),

This is not going to scale, so we'll presumably want to make this more flexible and configurable. Some thoughts on how to do this:

1. Consider making only the APIExport/APIBinding-related CRDs available to every workspace. Any additional types would be injected via individual APIBindings per workspace.
2. Remove the hard-coded list and move it into ClusterWorkspace.Spec.
3. Some combination of the above

WDYT? Other ideas? Don't bother? 😄 Would love to hear opinions.

[sttts]

Would love to see (1). We could place the APIExport for that in some system workspace. The interesting bit: then those need a shared identity between different shards such that in the future (when we have wildcard authz for serviceaccount with identities as audience) service account can access these kcp-wide.

[ncdc]

Does (1) increase the "cost" of a workspace because it now needs 1..n APIBinding CRs, replicated times thousands of workspaces?

[sttts]

true. We could have system bindings, that are magically applied, but don't exist as objects :-) dejavu.

[ncdc]

We could also move the hard-coding into spec fields on ClusterWorkspace or something. Not a great solution, but at least not in go code.