All traffic sent to mixer as TCP #3099
Comments
|
And in grafana we get nothhing, further confirming mixer isn't working :/
I turned on trace logging in mixer and see nothing, so all the side cars are failing to / not sending to it. |
|
A little more randomness, this is one of my app pods, i noticed in the However if I go onto the pod, I can curl that fine: So this is something to do with the side car making that call. It totally feels like something to do with mtls to me. |
Stono
changed the title
|
@Stono what does your deployment look like? The logs indicate that Mixer is not able to start serving gRPC, which would prevent it from receiving any Reports(). That indicates that something went wrong with deployment. |
|
@douglas-reid that's a very broad question! We are:
As this cluster isn't live, I went for a disruptive but fully clean installed, I totally removed I've tested rolling back to Everything seems to have deployed fine: Mixer config at boot: |
|
Happy to provide any more information, logs etc if you tell me what you need. All the ports seem to be listening on mixer: |
|
@douglas-reid it feels like theres something funky going on with mtls too If i go onto the mixer pod directly: Absolutely fine On a pod which doesn't have an envoy sidecar: Again, fine. However on a pod that does have envoy: And the fact that loads of envoys are doing this: Not really understanding the architecture enough though i don't know if i'm just chasing shadows. |
|
And the plot thickens... |
|
@rkpagadala do you have any idea what may be going wrong with the proxy setup here for Mixer (and mTLS) ? |
|
More than happy to send over my terraform & istio installation files if needed, just give me an email address. |
|
Is this possibly related to #3139? |
|
I'm not sure they're related @douglas-reid - I don't have any errors about it failing to read the certs. Here are my logs from istio-proxy: I find it strange however that the config file isn't there. so this is on a working pod: And this is on the mixers istio-proxy: |
|
Actually you know, you may be onto something. I'm unable to hit other mtls services from the mixer pod (it's my understanding that i should be able to). In this example And the mixer log is just littered with: |
|
@douglas-reid do we understand the problem yet? |
|
@sakshigoel12 I'm hoping that @wattli can take a look and see if anything obvious jumps out here. |
|
Happy to get on calls, screen share, provide logs, click my heels... anything i can do to help debug just ask. |
|
the pilot error should not cause you any harm. BTW, I was able to get metrics in istio dashboard with istio 0.5 on ibm cloud container service. |
|
make sure you use manual inject to inject the side car... |
|
@linsun I have injected the side car :-) It's not the pilot errors that concern me to be honest, it's the gRPC errors in Mixer. Out of curiosity what is your setup? As we're specifically using MUTUAL_TLS on an RBAC enabled cluster |
|
@Stono , how about we disable mixer, is there still an issue? |
|
@Stono mTLS + RBAC should work out of the box (we have a series of e2e tests specifically for this use case: https://prow.istio.io/?job=e2e-suite-rbac-auth). They should be exercising this functionality fairly heavily. When you installed 0.5.0 is it possible that any installation steps were missed ? |
|
@jeffmendoza any updates on this? |
|
Hey, |
|
This is a pretty old bug ... we need to make some progress here. It looks like the protocol selection happens here. So the ports appear to be named correctly (starting with @kyessenov would you mind taking a look? The problem is that all traffic shows up in mixer as |
|
I too am seeing a similar behaviour with 0.5.1 with mTLS. Our services in the mesh talking to services outside the mesh (such as cassandra or kafka) fail with |
nmittler
changed the title
|
@mandarjog would you mind taking a look? |
|
That last comment looks like traffic being sent to mixer not using gRPC. What does the filter on the egress look like? |
|
I have no defined any egress rules for this |
|
Im not entirely sure if cassandra traffic is gRPC anyway |
|
@douglas-reid @nixgadget is that not a different issue to this one? This issue is about http services sending metrics as TCP. The gRPC issue you're experiencing went away for me when I did a "clean" install of istio (which is great as I was never able to reproduce it). |
|
The grpc issue is most likely because of mtls mismatch. Mixer was expecting mtls and envoy was not sending it mtls. |
|
@Stono sorry for the late reply. Just coming back to this.
|
|
@mandarjog We now have an nginx ingress test that might help in your testing. See #4569 |
|
Going back thru the PR, I can confirm that the configuration generated by pilot is incorrect. The outbound listener is correctly identified as 'http', but since metrics reporting happens at inbound, we see continuing investigation ... |
|
Just as an FYI on this never ending thread (lol), I've upgrade to 0.7.1 and this issue is still there. Is the name of the port the only thing which defines if it is http or tcp? |
|
@Stono can you get the latest |
|
I would also like to see route rules and destination policies that pertain to |
|
And istio-pilot logs, thanks. |
|
Sent to your email |
|
@Stono can you use I think port 8088 is a red herring. The issue probably is listener collision... - name: https-web-mtls
port: 443
targetPort: http-web |
|
Hey, Anyway; there are collisions in the logs but as you're aware the I've sent you both sets of logs via email to look into yourself. |
|
I decided to try and test your hypothesis... I edited the service and removed the I then put the mtls port back, and edited the deployment to use I changed the port back to 8375 and it stopped working again. I am happy to demonstrate this on hangouts for you later! |
|
Oh what fun this has been, but thanks to @mandarjog we have finally got to the bottom of it! @mandarjog may want to elaborate more but at a high level all of our services exposed two ports This multiplexing is not actually supported by istio and resulted in a "port collision", only one of them could be used and the code base basically took the first one from the list, and that first one was sometimes The @mandarjog up to you if you want to close this issue or leave it open and link your PR with improvements to logging to help others debug this issue more easily in the future? Thanks! |
Hey,
On a fresh install of istio 0.5.0 on k8s 1.8.6 in GKE with mTLS enabled, i'm seeing errors in mixer:
and errors in pilot:
The text was updated successfully, but these errors were encountered: