Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scope-probe appears to have connection to everything #1929

Closed
rade opened this issue Oct 18, 2016 · 7 comments
Closed

scope-probe appears to have connection to everything #1929

rade opened this issue Oct 18, 2016 · 7 comments
Assignees
Labels
bug Broken end user or developer functionality; not working as the developers intended it

Comments

@rade
Copy link
Member

rade commented Oct 18, 2016

screenshot_2016-10-18_03-58-47

I wonder whether this is somehow a consequence of #1863 and associated DNS tracking PRs.

@rade rade added the bug Broken end user or developer functionality; not working as the developers intended it label Oct 18, 2016
@2opremio
Copy link
Contributor

@rade Where did this happen? Do you have a report?

@rade
Copy link
Member Author

rade commented Oct 18, 2016

@2opremio
Copy link
Contributor

The probes are doing a good job tagging the containers as being in the host network and tagging the pause container as not making connections

screen shot 2016-10-21 at 13 00 55

screen shot 2016-10-21 at 13 12 04

screen shot 2016-10-21 at 13 11 33

screen shot 2016-10-21 at 13 15 10

so, I am suspecting a regression on #1653 (or perhaps #1597 )

@2opremio 2opremio self-assigned this Oct 21, 2016
@2opremio
Copy link
Contributor

I found out what's happening. We are taking Pod IPs as unique when matching short-lived connections against them. This is perfectly fine for pods using their own networking namespace like the following:

screen shot 2016-10-21 at 16 09 58

However, pods in the host networking namespace inherit their IP from the host. For instance:

screen shot 2016-10-21 at 16 09 31

This causes Scope to attribute all the host connections to a single POD running in the host networking namespace.

No regression here, this has been broken since we started tracking POD connections.

The way to fix this is to check whether the pod runs in the host networking namespace to discard the short-lived connections from/to its IP. Unfortunately, this will also discard legitimate short-lived connections coming from the POD.

@rade
Copy link
Member Author

rade commented Oct 21, 2016

Hmm. We should be able to disambiguate these connections, and "assign" them to the right pod, by taking into account the port - the IP+port combo of the source of outbound connections and the destination of inbound connections should be associated with an individual process, which in turn will give us the pod.

@2opremio
Copy link
Contributor

We don't have PIDs for short-lived connections. So we have the exact same problem for Pods as we have for containers.

In other words, we can't currently disambiguate them. ebpf should change this.

@2opremio
Copy link
Contributor

You have a more thorough explanation in #1260

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Broken end user or developer functionality; not working as the developers intended it
Projects
None yet
Development

No branches or pull requests

2 participants