Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix panic when running ICMPv4 probe with DontFragment #686

Merged
merged 1 commit into from
Aug 29, 2020

Conversation

mem
Copy link
Contributor

@mem mem commented Aug 27, 2020

A recent change modified the way an IPv4 raw socket is created and BBE
is panicing when ICMPv4 is used with DontFragment set.

golang.org/x/net doesn't seem to have a way to create the necessary
socket, so fall back to the previous method for that case in particular
(ICMP, IPv4, DontFragment=true).

Fixes #685

Signed-off-by: Marcelo E. Magallon marcelo.magallon@grafana.com

@mem
Copy link
Contributor Author

mem commented Aug 27, 2020

this is running this change with capabilities set:

ts=2020-08-27T21:35:12.024Z caller=main.go:169 module=ping_df_v4 target=localhost level=debug msg="Beginning probe" probe=icmp timeout_seconds=119.5
ts=2020-08-27T21:35:12.024Z caller=main.go:169 module=ping_df_v4 target=localhost level=debug msg="Resolving target address" ip_protocol=ip4
ts=2020-08-27T21:35:12.025Z caller=main.go:169 module=ping_df_v4 target=localhost level=debug msg="Resolved target address" ip=127.0.0.1
ts=2020-08-27T21:35:12.025Z caller=main.go:169 module=ping_df_v4 target=localhost level=debug msg="Creating socket"
ts=2020-08-27T21:35:12.025Z caller=main.go:169 module=ping_df_v4 target=localhost level=debug msg="Creating ICMP packet" seq=55056 id=31912
ts=2020-08-27T21:35:12.026Z caller=main.go:169 module=ping_df_v4 target=localhost level=debug msg="Writing out packet"
ts=2020-08-27T21:35:12.026Z caller=main.go:169 module=ping_df_v4 target=localhost level=debug msg="Waiting for reply packets"
ts=2020-08-27T21:35:12.026Z caller=main.go:169 module=ping_df_v4 target=localhost level=debug msg="Found matching reply packet"
ts=2020-08-27T21:35:12.026Z caller=main.go:169 module=ping_df_v4 target=localhost level=debug msg="Probe succeeded" duration_seconds=0.001902335
ts=2020-08-27T21:35:22.209Z caller=main.go:169 module=ping_f_v4 target=localhost level=debug msg="Beginning probe" probe=icmp timeout_seconds=119.5
ts=2020-08-27T21:35:22.210Z caller=main.go:169 module=ping_f_v4 target=localhost level=debug msg="Resolving target address" ip_protocol=ip4
ts=2020-08-27T21:35:22.210Z caller=main.go:169 module=ping_f_v4 target=localhost level=debug msg="Resolved target address" ip=127.0.0.1
ts=2020-08-27T21:35:22.210Z caller=main.go:169 module=ping_f_v4 target=localhost level=debug msg="Creating socket"
ts=2020-08-27T21:35:22.210Z caller=main.go:169 module=ping_f_v4 target=localhost level=debug msg="Creating ICMP packet" seq=55057 id=31912
ts=2020-08-27T21:35:22.210Z caller=main.go:169 module=ping_f_v4 target=localhost level=debug msg="Writing out packet"
ts=2020-08-27T21:35:22.211Z caller=main.go:169 module=ping_f_v4 target=localhost level=debug msg="Waiting for reply packets"
ts=2020-08-27T21:35:22.211Z caller=main.go:169 module=ping_f_v4 target=localhost level=debug msg="Found matching reply packet"
ts=2020-08-27T21:35:22.211Z caller=main.go:169 module=ping_f_v4 target=localhost level=debug msg="Probe succeeded" duration_seconds=0.001515964
ts=2020-08-27T21:35:32.834Z caller=main.go:169 module=ping_f_v6 target=::1 level=debug msg="Beginning probe" probe=icmp timeout_seconds=119.5
ts=2020-08-27T21:35:32.834Z caller=main.go:169 module=ping_f_v6 target=::1 level=debug msg="Resolving target address" ip_protocol=ip6
ts=2020-08-27T21:35:32.834Z caller=main.go:169 module=ping_f_v6 target=::1 level=debug msg="Resolved target address" ip=::1
ts=2020-08-27T21:35:32.834Z caller=main.go:169 module=ping_f_v6 target=::1 level=debug msg="Creating socket"
ts=2020-08-27T21:35:32.834Z caller=main.go:169 module=ping_f_v6 target=::1 level=debug msg="Creating ICMP packet" seq=55058 id=31912
ts=2020-08-27T21:35:32.834Z caller=main.go:169 module=ping_f_v6 target=::1 level=debug msg="Writing out packet"
ts=2020-08-27T21:35:32.834Z caller=main.go:169 module=ping_f_v6 target=::1 level=debug msg="Waiting for reply packets"
ts=2020-08-27T21:35:32.835Z caller=main.go:169 module=ping_f_v6 target=::1 level=debug msg="Found matching reply packet"
ts=2020-08-27T21:35:32.835Z caller=main.go:169 module=ping_f_v6 target=::1 level=debug msg="Probe succeeded" duration_seconds=0.00033165
ts=2020-08-27T21:35:37.967Z caller=main.go:169 module=ping_df_v6 target=::1 level=debug msg="Beginning probe" probe=icmp timeout_seconds=119.5
ts=2020-08-27T21:35:37.967Z caller=main.go:169 module=ping_df_v6 target=::1 level=debug msg="Resolving target address" ip_protocol=ip6
ts=2020-08-27T21:35:37.967Z caller=main.go:169 module=ping_df_v6 target=::1 level=debug msg="Resolved target address" ip=::1
ts=2020-08-27T21:35:37.968Z caller=main.go:169 module=ping_df_v6 target=::1 level=debug msg="Creating socket"
ts=2020-08-27T21:35:37.968Z caller=main.go:169 module=ping_df_v6 target=::1 level=debug msg="Creating ICMP packet" seq=55059 id=31912
ts=2020-08-27T21:35:37.968Z caller=main.go:169 module=ping_df_v6 target=::1 level=debug msg="Writing out packet"
ts=2020-08-27T21:35:37.968Z caller=main.go:169 module=ping_df_v6 target=::1 level=debug msg="Waiting for reply packets"
ts=2020-08-27T21:35:37.968Z caller=main.go:169 module=ping_df_v6 target=::1 level=debug msg="Found matching reply packet"
ts=2020-08-27T21:35:37.968Z caller=main.go:169 module=ping_df_v6 target=::1 level=debug msg="Probe succeeded" duration_seconds=0.000917358

and this is running this code without capabilities set:

ts=2020-08-27T21:36:31.267Z caller=main.go:169 module=ping_df_v6 target=::1 level=debug msg="Beginning probe" probe=icmp timeout_seconds=119.5
ts=2020-08-27T21:36:31.267Z caller=main.go:169 module=ping_df_v6 target=::1 level=debug msg="Resolving target address" ip_protocol=ip6
ts=2020-08-27T21:36:31.267Z caller=main.go:169 module=ping_df_v6 target=::1 level=debug msg="Resolved target address" ip=::1
ts=2020-08-27T21:36:31.267Z caller=main.go:169 module=ping_df_v6 target=::1 level=debug msg="Creating socket"
ts=2020-08-27T21:36:31.267Z caller=main.go:169 module=ping_df_v6 target=::1 level=debug msg="Creating ICMP packet" seq=40949 id=36974
ts=2020-08-27T21:36:31.268Z caller=main.go:169 module=ping_df_v6 target=::1 level=debug msg="Writing out packet"
ts=2020-08-27T21:36:31.268Z caller=main.go:169 module=ping_df_v6 target=::1 level=debug msg="Waiting for reply packets"
ts=2020-08-27T21:36:31.268Z caller=main.go:169 module=ping_df_v6 target=::1 level=debug msg="Found matching reply packet"
ts=2020-08-27T21:36:31.268Z caller=main.go:169 module=ping_df_v6 target=::1 level=debug msg="Probe succeeded" duration_seconds=0.000963329
ts=2020-08-27T21:36:36.964Z caller=main.go:169 module=ping_f_v6 target=::1 level=debug msg="Beginning probe" probe=icmp timeout_seconds=119.5
ts=2020-08-27T21:36:36.965Z caller=main.go:169 module=ping_f_v6 target=::1 level=debug msg="Resolving target address" ip_protocol=ip6
ts=2020-08-27T21:36:36.965Z caller=main.go:169 module=ping_f_v6 target=::1 level=debug msg="Resolved target address" ip=::1
ts=2020-08-27T21:36:36.965Z caller=main.go:169 module=ping_f_v6 target=::1 level=debug msg="Creating socket"
ts=2020-08-27T21:36:36.965Z caller=main.go:169 module=ping_f_v6 target=::1 level=debug msg="Creating ICMP packet" seq=40950 id=36974
ts=2020-08-27T21:36:36.965Z caller=main.go:169 module=ping_f_v6 target=::1 level=debug msg="Writing out packet"
ts=2020-08-27T21:36:36.966Z caller=main.go:169 module=ping_f_v6 target=::1 level=debug msg="Waiting for reply packets"
ts=2020-08-27T21:36:36.966Z caller=main.go:169 module=ping_f_v6 target=::1 level=debug msg="Found matching reply packet"
ts=2020-08-27T21:36:36.966Z caller=main.go:169 module=ping_f_v6 target=::1 level=debug msg="Probe succeeded" duration_seconds=0.001217941
ts=2020-08-27T21:36:52.868Z caller=main.go:169 module=ping_f_v4 target=localhost level=debug msg="Beginning probe" probe=icmp timeout_seconds=119.5
ts=2020-08-27T21:36:52.868Z caller=main.go:169 module=ping_f_v4 target=localhost level=debug msg="Resolving target address" ip_protocol=ip4
ts=2020-08-27T21:36:52.868Z caller=main.go:169 module=ping_f_v4 target=localhost level=debug msg="Resolved target address" ip=127.0.0.1
ts=2020-08-27T21:36:52.868Z caller=main.go:169 module=ping_f_v4 target=localhost level=debug msg="Creating socket"
ts=2020-08-27T21:36:52.869Z caller=main.go:169 module=ping_f_v4 target=localhost level=debug msg="Creating ICMP packet" seq=40951 id=36974
ts=2020-08-27T21:36:52.869Z caller=main.go:169 module=ping_f_v4 target=localhost level=debug msg="Writing out packet"
ts=2020-08-27T21:36:52.869Z caller=main.go:169 module=ping_f_v4 target=localhost level=debug msg="Waiting for reply packets"
ts=2020-08-27T21:36:52.869Z caller=main.go:169 module=ping_f_v4 target=localhost level=debug msg="Found matching reply packet"
ts=2020-08-27T21:36:52.869Z caller=main.go:169 module=ping_f_v4 target=localhost level=debug msg="Probe succeeded" duration_seconds=0.001453394
ts=2020-08-27T21:36:57.044Z caller=main.go:169 module=ping_df_v4 target=localhost level=debug msg="Beginning probe" probe=icmp timeout_seconds=119.5
ts=2020-08-27T21:36:57.045Z caller=main.go:169 module=ping_df_v4 target=localhost level=debug msg="Resolving target address" ip_protocol=ip4
ts=2020-08-27T21:36:57.045Z caller=main.go:169 module=ping_df_v4 target=localhost level=debug msg="Resolved target address" ip=127.0.0.1
ts=2020-08-27T21:36:57.045Z caller=main.go:169 module=ping_df_v4 target=localhost level=debug msg="Creating socket"
ts=2020-08-27T21:36:57.045Z caller=main.go:169 module=ping_df_v4 target=localhost level=debug msg="Error listening to socket" err="listen ip4:icmp 0.0.0.0: socket: operation not permitted"
ts=2020-08-27T21:36:57.045Z caller=main.go:169 module=ping_df_v4 target=localhost level=debug msg="Probe failed" duration_seconds=0.000818169

the case that fails (last line) is the one that requires privileges, so the failure is expected.

@mem
Copy link
Contributor Author

mem commented Aug 27, 2020

@dgl could you please take a look?

@brian-brazil
Copy link
Contributor

Thanks for finding, and fixing this. This looks like the right fix to me, @dgl do you have any comments before I merge?

prober/icmp.go Outdated
rc, err := ipv4.NewRawConn(icmpConn)
if err != nil {
level.Error(logger).Log("msg", "Error creating raw connection", "err", err)
return
}
socket = &v4Conn{c: rc, df: true}
} else {
var icmpConn *icmp.PacketConn
// If the user has set the don't fragment option we cannot use unprivileged
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move comment to the other branch of the if, probably update it to say something like "Need to use RawConn rather than ICMP socket in order to set IP header level options."

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, done!

@dgl
Copy link
Member

dgl commented Aug 28, 2020

LGTM, aside from my comment about the comment.

A recent change modified the way an IPv4 raw socket is created and BBE
is panicing when ICMPv4 is used with DontFragment set.

golang.org/x/net doesn't seem to have a way to create the necessary
socket, so fall back to the previous method for that case in particular
(ICMP, IPv4, DontFragment=true).

Fixes prometheus#685

Signed-off-by: Marcelo E. Magallon <marcelo.magallon@grafana.com>
@brian-brazil brian-brazil merged commit 04d136f into prometheus:master Aug 29, 2020
@brian-brazil
Copy link
Contributor

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

panic when setting dont_fragment
3 participants