ephemeral csi driver

[kfox1111]

Describe the feature request

As an alternate to the CNI driver which does not work well with initContainers, an ephemeral CSI driver could be written that:

• creates a socket on which to listen for traffic available in the ephemeral volume
• deploy the driver via daemonset
• pods include the csi ephemeral Volume
• the istio sidecar mounts the csi volume. When it is ready to receive traffic, it sends a message to the daemon listening on the unix socket
• when the unix socket receives a message, the driver nsenter's the network namespace of the pod and iptable configures the network.

This, in addition to the sidecar pod feature should allow seamless transitioning of the network, while still separating privilege like the cni driver.

Describe alternatives you've considered

The existing solutions

Additional context

[howardjohn]

@rlenglet

[rlenglet]

I don't see what problem this would solve. @kfox1111 can you clarify what you're trying to achieve here? Esp. can you clarify this statement:

CNI driver which does not work well with initContainers

[kfox1111]

https://istio.io/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers

1. cni configures the pod network
2. init containers run
3. istio sidecar starts.

step 2 in the middle has connectivity issues. The workarounds for whitelisting particular ip's can sometimes mitigate this issue. Sometimes not.

The proposal would provide a mechanism for istio itself to trigger the pod network to be setup but only after its ready while maintaining privilege separation. This would eliminate the need of the white list so that initContainers always just work.

The workflow proposed looks like:

1. init containers run
2. istio sidecar starts. pokes privileged agent via csi driver to setup pod network to speak through istio
3. containers start

[rlenglet]

The workflow proposed looks like:

1. init containers run
2. istio sidecar starts. pokes privileged agent via csi driver to setup pod network to speak through istio
3. containers start

The Istio sidecar can't start until the init containers have terminated, i.e., your steps 2. and 3. would happen concurrently. Therefore, this would not guarantee that network redirection is setup when application container start. Therefore, connections to/from the application containers would bypass Istio until the sidecar proxy is configured. Therefore, those connections would bypass authn and authz, which is a security risk.

Any design that doesn't guarantee that connections to/from the application containers can't bypass Istio are not acceptable from a security viewpoint.

[kfox1111]

Hmm... So one of the concerns is blocking incoming requests... That makes a lot of sense.

Is outgoing traffic also similarly constrained? Is it expected that all outgoing traffic will go out through istio?

[howardjohn]

Is outgoing traffic also similarly constrained? Is it expected that all outgoing traffic will go out through istio?

We do not guarantee this. Since its running in the same pod, the application can do various things to bypass envoy on outbound traffic. If you want to enforce this you need to use egress gateway + network policies

[kfox1111]

Ok. Then while there may still be a bit of value in maybe:

• cni (or maybe csi driver) sets up ingress filters so no incoming traffic can happen
• init containers run
• sidecar istio comes up and does csi driver socket thing to setup the full network redirection through sidecar
• rest of the containers come up

Not sure its worth the effort though.

[howardjohn]

I think we will not be able to implement any solution that doesn't ensure traffic is captured. When I said outbound traffic, I meant we cannot ensure 100% that it is captured if a user tries to intentionally bypass it, but we cannot have a startup race condition. The sidecar KEP will solve these ordering problems, I suggest we focus efforts there: kubernetes/enhancements#1913