Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add retry logic to DNS prober #1267

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add retry logic to DNS prober #1267

wants to merge 1 commit into from

Conversation

mem
Copy link
Contributor

@mem mem commented Jul 19, 2024

With DNS, since it's possible use UDP traffic, depending on the relative network location of the server under test, sometimes packets are dropped and the probe ends up timing out. It becomes hard to distinguish between this and an actual failure in the server.

Introduce an optional retry mechanism where requests are retried for a limited number of times. Each request gets its own timeout that still respects the timeout configured for the probe.

For example, if you configure 3 retries, each of them with a 3 second timeout, and the timeout for the probe is 5 seconds, the first attempt might consume 3 seconds, leaving 2 seconds until the deadline. The second request will still use a timeout of 3 seconds, but the probe itself will timeout and it will fail as usual.

Add a new metric that reports the number of attempts. Note that the probe_duration_seconds metric will now report the total time until an answer was obtained. In the above example, if the second attempt produces a response within 1 second, the reported duration will be ~ 4 seconds. The rtt will be ~ 1 second. Both these metrics allow the user to differentiate and track requests that require continous retries, indicating some kind of issue with the server or the network path.

With DNS, since it's possible use UDP traffic, depending on the relative
network location of the server under test, sometimes packets are dropped
and the probe ends up timing out. It becomes hard to distinguish between
this and an actual failure in the server.

Introduce an optional retry mechanism where requests are retried for a
limited number of times. Each request gets its own timeout that still
respects the timeout configured for the probe.

For example, if you configure 3 retries, each of them with a 3 second
timeout, and the timeout for the probe is 5 seconds, the first attempt
might consume 3 seconds, leaving 2 seconds until the deadline. The
second request will still use a timeout of 3 seconds, but the probe
itself will timeout and it will fail as usual.

Add a new metric that reports the number of attempts. Note that the
probe_duration_seconds metric will now report the total time until an
answer was obtained. In the above example, if the second attempt
produces a response within 1 second, the reported duration will be ~ 4
seconds. The rtt will be ~ 1 second. Both these metrics allow the user
to differentiate and track requests that require continous retries,
indicating some kind of issue with the server or the network path.

Signed-off-by: Marcelo E. Magallon <marcelo.magallon@grafana.com>
@mem
Copy link
Contributor Author

mem commented Jul 19, 2024

@SuperQ I would appreciate your opinion here, particularly regarding the metrics. I tried explaining the case that I'm trying to deal with.

@mem mem requested a review from SuperQ July 20, 2024 14:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant