-
Notifications
You must be signed in to change notification settings - Fork 95
Ignoring mismatches
Josh Hiles edited this page Dec 22, 2023
·
1 revision
During the early stages of an experiment, it's possible that some of your code will always generate a mismatch for reasons you know and understand but haven't yet fixed. Instead of these known cases always showing up as mismatches in your metrics or analysis, you can tell an experiment whether or not to ignore a mismatch using the Ignore
method. You may include more than one block if needed:
public bool CanAccess(IUser user)
{
return Scientist.Science<bool>("widget-permissions", experiment =>
{
experiment.Use(() => IsCollaborator(user));
experiment.Try(() => HasAccess(user));
// user is staff, always an admin in the new system
experiment.Ignore((control, candidate) => user.IsStaff);
// new system doesn't handle unconfirmed users yet
experiment.Ignore((control, candidate) => control && !candidate && !user.ConfirmedEmail);
});
}
The ignore blocks are only called if the values don't match. If one observation raises an exception and the other doesn't, it's always considered a mismatch. If both observations raise different exceptions, that is also considered a mismatch.
- Publishing results
- Controlling comparison
- Adding context
- Expensive setup
- Keeping it clean
- Ignoring mismatches
- Enabling or disabling experiments
- Ramping up experiments
- Running candidates in parallel (asynchronous)
- Testing
- Handling errors
- Designing an experiment
- Finishing an experiment
Sometimes scientists just gotta do weird stuff. We understand.