negative slave lag can cause insertion issues

[sjmudd]

While theoretically impossible I recently saw some errors in orchestrator. The cause was the slave lag was negative. The problem is later orchestrator tries to insert this value into the db and fails as it is out of bounds (bigint unsigned), so the replication topology does not show up properly as the master info is not updated.

I patched the code to correct the value and generate a warning:

commit 1bcfb69d901037d1271abef9c38f530bcdc20f72
Author: Simon Mudd <simon.mudd@booking.com>
Date:   Mon May 30 14:16:25 2016 +0200

    Catch negative replication delays and adjust to 0 if found to avoid breakage later

diff --git a/go/inst/instance_dao.go b/go/inst/instance_dao.go
index 15f1a43..a727865 100644
--- a/go/inst/instance_dao.go
+++ b/go/inst/instance_dao.go
@@ -386,6 +386,13 @@ func TimedReadTopologyInstance(instanceKey *InstanceKey, latency *stopwatch.Name
                instance.SecondsBehindMaster = m.GetNullInt64("Seconds_Behind_Master")
                // And until told otherwise:
                instance.SlaveLagSeconds = instance.SecondsBehindMaster
+               // This should not happen but can. If it does we  will
+               // fail to insert the value (being negative) and the
+               // topology will not be recognised properly.
+               if instance.SlaveLagSeconds.Valid && instance.SlaveLagSeconds.Int64 < 0 {
+                       log.Warningf("Host: %+v, instance.SecondsBehindMaster < 0 [%+v], correcting to 0", instanceKey, instance.SlaveLagSeconds)
+                       instance.SlaveLagSeconds.Int64 = 0
+               }
                instance.AllowTLS = (m.GetString("Master_SSL_Allowed") == "Yes")
                // Not breaking the flow even on error
                slaveStatusFound = true

So maybe it is worth adding this check to allow some data to be stored?

[sjmudd]

Note: I've not provided a pull request as I'm a bit out of sync with mainstream orchestrator.

[shlomi-noach]

Need to apply the same for reading config.Config.SlaveLagQuery

[shlomi-noach]

Fixed downstream, merging shortly