Skip to content

Ignoring mismatches

Josh Hiles edited this page Dec 22, 2023 · 1 revision

During the early stages of an experiment, it's possible that some of your code will always generate a mismatch for reasons you know and understand but haven't yet fixed. Instead of these known cases always showing up as mismatches in your metrics or analysis, you can tell an experiment whether or not to ignore a mismatch using the Ignore method. You may include more than one block if needed:

public bool CanAccess(IUser user)
{
    return Scientist.Science<bool>("widget-permissions", experiment =>
    {
        experiment.Use(() => IsCollaborator(user));
        experiment.Try(() => HasAccess(user));

        // user is staff, always an admin in the new system
        experiment.Ignore((control, candidate) => user.IsStaff);
        // new system doesn't handle unconfirmed users yet
        experiment.Ignore((control, candidate) => control && !candidate && !user.ConfirmedEmail);
    });
}

The ignore blocks are only called if the values don't match. If one observation raises an exception and the other doesn't, it's always considered a mismatch. If both observations raise different exceptions, that is also considered a mismatch.