feat(prometheus): expose controlplane connectivity state as a gauge #14020
+132
−4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
core/clustering
plugins/prometheus
cherry-pick kong-ee
aryan9600
force-pushed
the
cp-conn-prom-metric
branch
3 times, most recently
from
c2d6278 to
697c1e6
Compare
aryan9600
force-pushed
the
cp-conn-prom-metric
branch
from
697c1e6 to
de4a868
Compare
hbagdi
reviewed
Jan 7, 2025
aryan9600
force-pushed
the
cp-conn-prom-metric
branch
from
de4a868 to
ee922f2
Compare
flrgh
reviewed
Jan 11, 2025
flrgh
reviewed
Jan 11, 2025
flrgh
reviewed
Jan 11, 2025
flrgh
reviewed
Jan 11, 2025
flrgh
reviewed
Jan 11, 2025
flrgh
reviewed
Jan 11, 2025
flrgh
reviewed
Jan 11, 2025
flrgh
reviewed
Jan 11, 2025
gszr
approved these changes
Jan 22, 2025
flrgh
reviewed
Jan 22, 2025
kong/clustering/data_plane.lua
Outdated
Comment on lines
77
to
78
| local function set_control_plane_connected(reachable, ttl) | ||
| local ok, err = ngx.shared.kong:safe_set("control_plane_connected", reachable, ttl) |
There was a problem hiding this comment.
I think we should just hard-code the ttl as PING_WAIT since it's what we should use in all cases.
Suggested change
| local function set_control_plane_connected(reachable, ttl) | |
| local ok, err = ngx.shared.kong:safe_set("control_plane_connected", reachable, ttl) | |
| local function set_control_plane_connected(reachable) | |
| local ok, err = ngx.shared.kong:safe_set("control_plane_connected", reachable, PING_WAIT) |
flrgh
reviewed
Jan 22, 2025
kong/clustering/data_plane.lua
Outdated
| local function set_control_plane_connected(reachable, ttl) | ||
| local ok, err = ngx.shared.kong:safe_set("control_plane_connected", reachable, ttl) | ||
| if not ok then | ||
| ngx_log(ngx_ERR, _log_prefix, "failed to set controlplane_reachable key in shm to ", reachable, " :", err) |
There was a problem hiding this comment.
nit: update the log line to match the current name of the SHM key
Suggested change
| ngx_log(ngx_ERR, _log_prefix, "failed to set controlplane_reachable key in shm to ", reachable, " :", err) | |
| ngx_log(ngx_ERR, _log_prefix, "failed to set \"control_plane_connected\" key in shm to ", reachable, " :", err) |
flrgh
reviewed
Jan 22, 2025
kong/plugins/prometheus/exporter.lua
Outdated
Comment on lines
70
to
74
| metrics.cp_connected = prometheus:gauge("control_plane_connected", | ||
| "Kong connected to control plane, " .. | ||
| "0 is unconnected", | ||
| nil, | ||
| prometheus.LOCAL_STORAGE) |
There was a problem hiding this comment.
nit: formatting
Suggested change
| metrics.cp_connected = prometheus:gauge("control_plane_connected", | |
| "Kong connected to control plane, " .. | |
| "0 is unconnected", | |
| nil, | |
| prometheus.LOCAL_STORAGE) | |
| metrics.cp_connected = prometheus:gauge("control_plane_connected", | |
| "Kong connected to control plane, " .. | |
| "0 is unconnected", | |
| nil, | |
| prometheus.LOCAL_STORAGE) |
flrgh
reviewed
Jan 22, 2025
Comment on lines
601
to
603
| -- it takes some time for the cp<->dp connection to get established and the | ||
| -- metric to reflect that, so set the timeout to 10 secs. | ||
| assert.with_timeout(10).eventually(function() |
There was a problem hiding this comment.
Suggesting a more lenient timeout here to prevent the test from being flaky. It's not uncommon for stuff to be extra slow in CI.
Suggested change
| -- it takes some time for the cp<->dp connection to get established and the | |
| -- metric to reflect that, so set the timeout to 10 secs. | |
| assert.with_timeout(10).eventually(function() | |
| -- it takes some time for the cp<->dp connection to get established and the | |
| -- metric to reflect that. On failure, re-connection attempts are spaced out | |
| -- in `math.random(5, 10)` second intervals, so a generous timeout is used | |
| -- in case we get unlucky and have to wait multiple retry cycles | |
| assert.with_timeout(30).eventually(function() |
Add a new Prometheus gauge metric `control_plane_connected`. Similar to `datastore_reachable` gauge, 0 means the connection is not healthy; 1 means that the connection is healthy. We mark the connection as unhealthy under the following circumstances: * Failure while establihing a websocket connection * Failure while sending basic information to controlplane * Failure while sending ping to controlplane * Failure while receiving a packet from the websocket connection This is helpful for users running a signficant number of gateways to be alerted about potential issues any gateway(s) may be facing while talking to the controlplane. Signed-off-by: Sanskar Jaiswal <sanskar.jaiswal@konghq.com>
aryan9600
force-pushed
the
cp-conn-prom-metric
branch
from
1f1c1a3 to
cf5b522
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Add a new Prometheus gauge metric
control_plane_connected. Similar todatastore_reachablegauge, 0 means the connection is not healthy; 1 means that the connection is healthy. We mark the connection as unhealthy under the following circumstances:This is helpful for users running a signficant number of gateways to be alerted about potential issues any gateway(s) may be facing while talking to the controlplane.
Checklist
changelog/unreleased/kongorskip-changeloglabel added on PR if changelog is unnecessary. README.mdIssue reference
Fix #[issue number]