Pseudo-GTID relocate failures when replicating 5.6 to 5.7

[sjmudd]

This is a bit of a long issue (in terms of the description). Sorry.

I run masters running 5.6 while there are now slaves running 5.7 under them and use pseudo-GTID mode. Most of the time orchestrator is able to relocate the servers this way.

5.7 introduced a new event in the binlog stream the "anonymous event". This confuses orchestrator so following a suggestion I added to the orchestrator config the following setting:

  "SkipBinlogEventsContaining": [
    "SET @@SESSION.GTID_NEXT= 'ANONYMOUS'"
  ],

Again this "mostly works". However, not always.
The reason for this is that sometimes the position to be checked will be "just at an anonymous event" and the current orchestrator logic hides these events and skips through the binlogs or relay logs and doesn't seem them. This has the effect that when it scans back to the previous pseudo-gtid event it finds it find but when it scans forward again on the slave and the "new master the slave will move to" it may "jump over" the anonymous position so think it's failed.

The description may seem somewhat vague but I hope Shlomi you follow here.

My thought on solving this (not looked at doing that yet is):
(a). the single setting used above is not ideal. If you replicate 5.7 --> 5.7 there's no need to filter out these events, so the setting should not be global but should be an attribute perhaps of the replication chain or applied on "filtered boxes" somehow.
(b). if you don't completely ignore the event but just skip over it if necessary then most of the logic should work fine. That is search back to the previous pseudo-gtid event, and then skip forward events "as needed" jumping over any "anonymous" events if you need to find the "next one".

The logic in (b) is split into several places so the change is not as trivial as I'd hoped. I still need to look but I'm also interested in solving (a) as filtering out stuff should only be done "if necessary" and not completely unconditionally as now.

Also note the filter used above also breaks relocate moving from 5.7 to 5.7 boxes as the "reference event" may get skipped over so I am keen to remove the global setting.

[shlomi-noach]

The description may seem somewhat vague but I hope Shlomi you follow here.

Yes, unfortunately I follow. I think this will be a tough one to crack.