Internal apiserver calls blocked by Istio-- only on AKS

[krancour]

Cross posting: knative/serving#1730

[jeremyrickard]

This is also happening on my clusters, happy to provide cluster info.

Note that the error messages in main issue description both relate to one of the types. Other types seem to work.

The generated error messages are coming from the k8s libraries. The connection refused message seems self explanatory. The second message is emitted when a status code of >= 500 is received: see https://github.com/knative/serving/blob/fed3bc3f41a6f5aa87480a53a32f97cab79f1f5f/vendor/k8s.io/apimachinery/pkg/api/errors/errors.go#L379

[krancour]

Another datapoint: Using these instructions, I've been unable to find any apiserver log entries that appear to relate to this.

[sauryadas]

We are working on enabling cluster autoscaler in the next 2 weeks. Havent had a chance to try out knative serving autoscaling yet. For any feature requests/enhancements please use user voice ..

https://feedback.azure.com/forums/914020-azure-kubernetes-service-aks

[krancour]

@sauryadas, this wasn't a feature request. It was meant to call attention to an unidentified platform issue. I have a new datapoint, so I'm resurrecting this thread.

Knative itself cannot be implicated in this issue. However, by default, Knative makes extensive use of Istio and istio/istio#8696 notes that pods utilizing an istio-proxy sidecar are unable to communicate with the Kubernetes apiserver when running in AKS. I can confirm through experimentation that disabling Istio resolves this problem from a Knative perspective.

So... I'll rename this issue and reframe it as an AKS + Istio problem instead of an AKS + Knative problem.

It's unclear to me whether the root cause lies within AKS or Istio. Perhaps it's a combination fo the two: i.e. AKS violates some assumption that Istio makes.

At any rate, I'm re-opening so that we can try to bring some resolution to this.

[palma21]

Fixed