Switch to Istio native sidecars

[mjnagel]

Is your feature request related to a problem? Please describe.

Currently we use istio's sidecars across all pods in the mesh. Due to existing istio issues we have also implemented a Pepr watch action to handle properly terminating sidecars for jobs. If we switch to "native" sidecars we will no longer need this action and may have some other benefits with regards to init containers accessing the network (common with things that validate dependencies that may be in cluster).

Describe the solution you'd like

Istio should be deployed with native sidecars by default (ENABLE_NATIVE_SIDECARS env on pilot). When upgrading this should be a seamless rollout - unsure if we will need to cycle all pods or something similar here (may be worth evaluating this issue first).

Describe alternatives you've considered

We could continue to use non-native sidecars + our pepr capability.

Additional context

This functionality in kubernetes is available in 1.28 behind a feature flag, and 1.29 by default. We should be fairly safe to enable this but should document this requirement well (maybe in our k8s/distro support page). In theory someone could use overrides to switch to non-native sidecars but they would be missing our pepr capability unless we maintain that behind a flag as well.

[adam-defenseunicorns]

Slack Thread Request - 9 July @Racer159