Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Platform] New Health check for Replication status in 2dc setup #4843

Closed
rao-vasireddy opened this issue Jun 21, 2020 · 0 comments
Closed

[Platform] New Health check for Replication status in 2dc setup #4843

rao-vasireddy opened this issue Jun 21, 2020 · 0 comments
Assignees
@daniel-yb @nmantrav2
Labels
area/cdc Change Data Capture area/platform Yugabyte Platform kind/enhancement This is an enhancement of an existing feature
Milestone
2.5.1.0

Comments

@rao-vasireddy
Copy link
Contributor

rao-vasireddy commented Jun 21, 2020 •
edited by chirag-yb
Loading

Raise an alert when the replication lag between two DC set-up exceeds the threshold.

Aha! Link: https://yugabyte-test.aha.io/features/PLATFORM-264
@rao-vasireddy rao-vasireddy added kind/enhancement This is an enhancement of an existing feature area/cdc Change Data Capture area/platform Yugabyte Platform labels Jun 21, 2020
@streddy-yb streddy-yb assigned daniel-yb and unassigned andrewc-dev and chirag-yb Oct 12, 2020
@streddy-yb streddy-yb added this to the v2.3 milestone Oct 12, 2020
@daniel-yb daniel-yb mentioned this issue Oct 26, 2020
Open
@streddy-yb streddy-yb modified the milestones: v2.3, 2.5.1.0 Nov 11, 2020
daniel-yb added a commit that referenced this issue Nov 12, 2020
@daniel-yb
[Platform] New Health check for Replication status in 2dc setup #4843
12a69e4 
Summary:
Add ability to configure alerts to be triggered based off of prometheus queries.

Also add UI for replication lag alert. Configurable to be turned on/off at runtime as well as have the threshold (in ms) configurable by the user. Default is 3m.

Test Plan:
Use a proxy metric (cpu_utime) to test the end-to-end flow including UI + triggering alert + resolving alert (through flipping the ">" to "<" in the query) + verifying corresponding emails are sent if the alert is triggered + resolved/not sent if the setting is disabled in the UI.

```
---------- MESSAGE FOLLOWS ----------
Content-Type: multipart/alternative;
 boundary="===============1491229132144892902=="
MIME-Version: 1.0
Subject: Yugabyte Platform Alert - <[admin][admin]>
From:
To: daniel@yugabyte.com
X-Peer: 127.0.0.1

--===============1491229132144892902==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit

Replication Lag Alert for daniel-test-backup-alert-oct-20-4 is firing.
--===============1491229132144892902==--
------------ END MESSAGE ------------
```

{F14427}

{F14428}

{F14429}

Reviewers: arnav, wesley, spotachev, sanketh, andrew

Reviewed By: sanketh, andrew

Subscribers: andrew, sshevchenko, jenkins-bot

Differential Revision: https://phabricator.dev.yugabyte.com/D9726
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@daniel-yb daniel-yb

@nmantrav2 nmantrav2

Labels
area/cdc Change Data Capture area/platform Yugabyte Platform kind/enhancement This is an enhancement of an existing feature
Projects
None yet
Milestone
2.5.1.0
Development

No branches or pull requests
7 participants
@ramkumarvs @rao-vasireddy @andrewc-dev @daniel-yb @nmantrav2 @chirag-yb @streddy-yb