Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix/Sentinel switch-master-count metric #280

Merged
merged 2 commits into from
Nov 16, 2023

Conversation

slopezz
Copy link
Member

@slopezz slopezz commented Nov 16, 2023

With recent migrations we saw that failover alert do not work on the first failover upon saas-operator pod creation.

The reason is, there is a timeseries database for every redis_server, on latest migrations the failover orcurs on a new redis server instance, passing the counter from non-exist to 1, so prometheus rate does not get it.

In the next image, filtering per shard and sentinel, there are 3 timeseriesdb with 0 value (the ones from old redis_servers), and one timeseriesdb with value 1 (new redis_server).
image

This PR removes the redis_server label from switchMasterCount metric, the same already done at failoverAbortNoGoodSlaveCount, which is the same case, we want a metric per shard only.

/kind bug
/kind release
/priority important-soon
/assign

@3scale-robot 3scale-robot added kind/bug Categorizes issue or PR as related to a bug. kind/release Categorizes issue or PR as related to a new release. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next sprint. labels Nov 16, 2023
@3scale-robot 3scale-robot added needs-size Indicates a PR or issue lacks a `size/foo` label and requires one. size/S Requires less than a day to complete the PR or the issue. and removed needs-size Indicates a PR or issue lacks a `size/foo` label and requires one. labels Nov 16, 2023
@3scale-robot 3scale-robot added the lgtm Indicates that a PR is ready to be merged. label Nov 16, 2023
@3scale-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: bca0e243bd944861c0694fb6242b87a4c09a0a23

@raelga
Copy link
Contributor

raelga commented Nov 16, 2023

/lgtm

@slopezz slopezz force-pushed the fix/sentinel-switchMasterCount-metric branch from fbf6443 to 2c2b0c1 Compare November 16, 2023 11:13
@3scale-robot 3scale-robot removed the lgtm Indicates that a PR is ready to be merged. label Nov 16, 2023
@3scale-robot 3scale-robot requested a review from roivaz November 16, 2023 11:13
@slopezz
Copy link
Member Author

slopezz commented Nov 16, 2023

/approve

@3scale-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: slopezz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@3scale-robot 3scale-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 16, 2023
@3scale-robot 3scale-robot added the lgtm Indicates that a PR is ready to be merged. label Nov 16, 2023
@3scale-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 36d19eb2ca8b9238ac00a9ea8a55a0c162539aac

@3scale-robot 3scale-robot merged commit 64dac09 into main Nov 16, 2023
4 of 5 checks passed
@3scale-robot 3scale-robot deleted the fix/sentinel-switchMasterCount-metric branch November 16, 2023 11:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. kind/bug Categorizes issue or PR as related to a bug. kind/release Categorizes issue or PR as related to a new release. lgtm Indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next sprint. size/S Requires less than a day to complete the PR or the issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants