Fix server_port_subscribers metric in Destination
#10820
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes #10764 `GetProfile' streams create a `server_port_subscribers` gauge that tracks the number of listeners interested in a given Server. Because of an oversight, the gauge was only being registered until the second listener was added. For just one listener the gauge was absent. But whenever the `GetProfile` stream ended, the gauge was deleted which resulted in this error if it wasn't registered to begin with: ``` level=warning msg="unable to delete server_port_subscribers metric with labels map[name:voting namespace:emojivoto port:4191]" addr=":8086" component=server ``` One can check that the gauge wasn't being created by installing viz and emojivoto, and checking the following returns empty: ```bash $ linkerd diagnostics controller-metrics|ag server.port.subscribers ``` After this fix, one can see the metric getting populated: ```bash $ linkerd diagnostics controller-metrics | grep server.port.subscribers # HELP server_port_subscribers Number of subscribers to Server changes associated with a pod's port. # TYPE server_port_subscribers gauge server_port_subscribers{name="emoji",namespace="emojivoto",port="4191"} 1 server_port_subscribers{name="linkerd",namespace="linkerd",port="4191"} 1 server_port_subscribers{name="linkerd",namespace="linkerd",port="9990"} 1 server_port_subscribers{name="linkerd",namespace="linkerd",port="9995"} 1 server_port_subscribers{name="linkerd",namespace="linkerd",port="9996"} 1 server_port_subscribers{name="linkerd",namespace="linkerd",port="9997"} 1 server_port_subscribers{name="metrics",namespace="linkerd-viz",port="4191"} 1 server_port_subscribers{name="metrics",namespace="linkerd-viz",port="9995"} 1 server_port_subscribers{name="tap",namespace="linkerd-viz",port="4191"} 1 server_port_subscribers{name="tap",namespace="linkerd-viz",port="9995"} 1 server_port_subscribers{name="tap",namespace="linkerd-viz",port="9998"} 1 server_port_subscribers{name="vote",namespace="emojivoto",port="4191"} 1 server_port_subscribers{name="voting",namespace="emojivoto",port="4191"} 1 server_port_subscribers{name="web",namespace="emojivoto",port="4191"} 1 server_port_subscribers{name="web",namespace="linkerd-viz",port="4191"} 1 server_port_subscribers{name="web",namespace="linkerd-viz",port="9994"} 1 ``` And when scaling down the voting deployment, one can see how the metric with `name="voting"` is removed.
adleong
approved these changes
Apr 25, 2023
mateiidavid
approved these changes
Apr 26, 2023
risingspiral
pushed a commit
that referenced
this pull request
May 4, 2023
Fixes #10764 `GetProfile` streams create a `server_port_subscribers` gauge that tracks the number of listeners interested in a given Server. Because of an oversight, the gauge was only being registered until the second listener was added. For just one listener the gauge was absent. But whenever the `GetProfile` stream ended, the gauge was deleted which resulted in this error if it wasn't registered to begin with: ``` level=warning msg="unable to delete server_port_subscribers metric with labels map[name:voting namespace:emojivoto port:4191]" addr=":8086" component=server ``` One can check that the gauge wasn't being created by installing viz and emojivoto, and checking the following returns empty: ```bash $ linkerd diagnostics controller-metrics | grep server_port_subscribers ``` After this fix, one can see the metric getting populated: ```bash $ linkerd diagnostics controller-metrics | grep server_port_subscribers # HELP server_port_subscribers Number of subscribers to Server changes associated with a pod's port. # TYPE server_port_subscribers gauge server_port_subscribers{name="emoji",namespace="emojivoto",port="4191"} 1 server_port_subscribers{name="linkerd",namespace="linkerd",port="4191"} 1 server_port_subscribers{name="linkerd",namespace="linkerd",port="9990"} 1 server_port_subscribers{name="linkerd",namespace="linkerd",port="9995"} 1 server_port_subscribers{name="linkerd",namespace="linkerd",port="9996"} 1 server_port_subscribers{name="linkerd",namespace="linkerd",port="9997"} 1 server_port_subscribers{name="metrics",namespace="linkerd-viz",port="4191"} 1 server_port_subscribers{name="metrics",namespace="linkerd-viz",port="9995"} 1 server_port_subscribers{name="tap",namespace="linkerd-viz",port="4191"} 1 server_port_subscribers{name="tap",namespace="linkerd-viz",port="9995"} 1 server_port_subscribers{name="tap",namespace="linkerd-viz",port="9998"} 1 server_port_subscribers{name="vote",namespace="emojivoto",port="4191"} 1 server_port_subscribers{name="voting",namespace="emojivoto",port="4191"} 1 server_port_subscribers{name="web",namespace="emojivoto",port="4191"} 1 server_port_subscribers{name="web",namespace="linkerd-viz",port="4191"} 1 server_port_subscribers{name="web",namespace="linkerd-viz",port="9994"} 1 ``` And when scaling down the voting deployment, one can see how the metric with `name="voting"` is removed.
risingspiral
pushed a commit
that referenced
this pull request
May 5, 2023
Fixes #10764 `GetProfile` streams create a `server_port_subscribers` gauge that tracks the number of listeners interested in a given Server. Because of an oversight, the gauge was only being registered until the second listener was added. For just one listener the gauge was absent. But whenever the `GetProfile` stream ended, the gauge was deleted which resulted in this error if it wasn't registered to begin with: ``` level=warning msg="unable to delete server_port_subscribers metric with labels map[name:voting namespace:emojivoto port:4191]" addr=":8086" component=server ``` One can check that the gauge wasn't being created by installing viz and emojivoto, and checking the following returns empty: ```bash $ linkerd diagnostics controller-metrics | grep server_port_subscribers ``` After this fix, one can see the metric getting populated: ```bash $ linkerd diagnostics controller-metrics | grep server_port_subscribers # HELP server_port_subscribers Number of subscribers to Server changes associated with a pod's port. # TYPE server_port_subscribers gauge server_port_subscribers{name="emoji",namespace="emojivoto",port="4191"} 1 server_port_subscribers{name="linkerd",namespace="linkerd",port="4191"} 1 server_port_subscribers{name="linkerd",namespace="linkerd",port="9990"} 1 server_port_subscribers{name="linkerd",namespace="linkerd",port="9995"} 1 server_port_subscribers{name="linkerd",namespace="linkerd",port="9996"} 1 server_port_subscribers{name="linkerd",namespace="linkerd",port="9997"} 1 server_port_subscribers{name="metrics",namespace="linkerd-viz",port="4191"} 1 server_port_subscribers{name="metrics",namespace="linkerd-viz",port="9995"} 1 server_port_subscribers{name="tap",namespace="linkerd-viz",port="4191"} 1 server_port_subscribers{name="tap",namespace="linkerd-viz",port="9995"} 1 server_port_subscribers{name="tap",namespace="linkerd-viz",port="9998"} 1 server_port_subscribers{name="vote",namespace="emojivoto",port="4191"} 1 server_port_subscribers{name="voting",namespace="emojivoto",port="4191"} 1 server_port_subscribers{name="web",namespace="emojivoto",port="4191"} 1 server_port_subscribers{name="web",namespace="linkerd-viz",port="4191"} 1 server_port_subscribers{name="web",namespace="linkerd-viz",port="9994"} 1 ``` And when scaling down the voting deployment, one can see how the metric with `name="voting"` is removed. Signed-off-by: Eric Anderson <eric@buoyant.io>
|
Hello, we have this problem in a recent update with version 2.13.4. Is it possible that this bug persists? |
|
We're also seeing this issue on version 2.13.5. |
|
Able to confirm that this appears to be getting reported in a few additional user environments. |
|
Based on the reports here, we have re-opened #10764. It would be really helpful if anyone who is experiencing this issue on version 2.13.5 could share destination controller logs in a comment on that issue. This will let us see the exact text of the error and help give us the context to reproduce and ultimately fix the error. Thanks! @jbucar @SansDK @deusxanima |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #10764
GetProfilestreams create aserver_port_subscribersgauge that tracks the number of listeners interested in a given Server. Because of an oversight, the gauge was only being registered until the second listener was added. For just one listener the gauge was absent. But whenever theGetProfilestream ended, the gauge was deleted which resulted in this error if it wasn't registered to begin with:One can check that the gauge wasn't being created by installing viz and emojivoto, and checking the following returns empty:
$ linkerd diagnostics controller-metrics | grep server_port_subscribersAfter this fix, one can see the metric getting populated:
And when scaling down the voting deployment, one can see how the metric with
name="voting"is removed.