Skip to content
This repository has been archived by the owner on Sep 30, 2024. It is now read-only.

GTID errant transaction detection #607

Merged
merged 13 commits into from
Sep 12, 2018

Conversation

shlomi-noach
Copy link
Collaborator

In this PR orchestrator auto-analyzes errant transactions for GTID-based replicas.

At this time it does nothing about it, merely presents it in the web UI.

Heuristic for avoiding false positives

orchestrator polls different servers at different times. It may be that it first polled the master and then its replica. In such case, the replica will naturally be seen to have more entries than the master.
If the replica also happens to have real errant transactions, how can we tell apart the two?

orchestrator ignores the GTID set that relates to the replica's immediate master UUID. At this time orchestrator does not further ignore upstream UUIDs, and this is TBD.

This PR is to be considered as POC.

screen shot 2018-09-06 at 15 32 50

In future PRs orchestrator will allow the user to fix errant transactions across the cluster, in a click of a button, or via command line. To be carefully thought of.

@shlomi-noach
Copy link
Collaborator Author

Right now build fails on go fmt, unrelated to anything, other that my upgrading to go1.11 on my local box. Will follow up on that.

@shlomi-noach shlomi-noach temporarily deployed to production/mysql_cluster=concertmaster September 8, 2018 10:09 Inactive
@shlomi-noach shlomi-noach temporarily deployed to production/mysql_cluster=concertmaster September 8, 2018 10:40 Inactive
@shlomi-noach
Copy link
Collaborator Author

shlomi-noach commented Sep 8, 2018

Errant GTIDs now also show up:

  • As a structure warning in replication-analysis
    • But not if promotion-rule is downtimed
  • Corollary as a structure warning in Web interface cluster page and cluster analysis
  • A problem in /api/problems, and corollary in problems sidebar
  • A red Globe-icon indicator, see screenshot

screen shot 2018-09-08 at 13 43 57

@shlomi-noach shlomi-noach temporarily deployed to production/mysql_cluster=concertmaster September 8, 2018 10:50 Inactive
@shlomi-noach shlomi-noach temporarily deployed to production/mysql_cluster=concertmaster September 8, 2018 18:05 Inactive
@shlomi-noach shlomi-noach temporarily deployed to production/mysql_cluster=concertmaster September 8, 2018 18:06 Inactive
@shlomi-noach shlomi-noach temporarily deployed to production/mysql_cluster=concertmaster September 8, 2018 18:11 Inactive
@shlomi-noach shlomi-noach temporarily deployed to production/mysql_cluster=conductor September 12, 2018 03:44 Inactive
@shlomi-noach shlomi-noach merged commit 33254f0 into master Sep 12, 2018
@shlomi-noach shlomi-noach deleted the gtid-errant-transaction-detection branch September 12, 2018 04:47
@shlomi-noach shlomi-noach restored the gtid-errant-transaction-detection branch September 12, 2018 07:09
@shlomi-noach shlomi-noach deleted the gtid-errant-transaction-detection branch September 12, 2018 08:52
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant