How does stickiness work when scaling down, instead of up?

[steffly]

Unleash allows that when scaling up a feature in a gradual rollout, the same users or entities will keep the same experience.

https://docs.getunleash.io/reference/stickiness

E.g., if you're in the first 10% who received the feature, you'll also still have the feature enabled when the rollout is increased to 20%.

(1) Does stickiness apply in the reverse direction?

I.e., if you received the feature after it rolled out to 100%, you'll continue to receive the feature even after the percentage decreases all the way to 0%, which will only apply to new user IDs or entity IDs not yet seen by Unleash.

(2) If the answer to the above is no, then is there an option to get this functionality?

[gastonfournier]

Hi @steffly, great question! It will be sticky in the reverse direction. The way it works is using a hash function that turns the field into a number from 1 to 100 in a consistent way: https://docs.getunleash.io/reference/stickiness#consistency that way whenever you change the rollout percentage (say from 50% to 30%), the hash value a user receives won't change. If for example Unleash associated the value 27 to a user, it will always be 27 (always considering the field on which the feature is sticky on does not change for that user). So when you do 30% rollout the user will see the feature, also when you roll up to 50%. If you roll down to 29% it will still see the feature, while if you roll down to 26% it will stop seeing the feature.

This value is not something you can easily calculate to tell whether user X will see feature Y when rolling up to 30%, but you should be able to use the playground to test this. Because the hash is the same, it should yield the same result.

I hope that covers your question! If you're curious and want to test this out, I've created this small gist: https://gist.github.com/gastonfournier/c20a1b57df3af69c7590c37c0dd2a037

[steffly]

@gastonfournier Hello, I think I didn't explain well enough.

Our requirement is this: Once a use has seen a feature, they should always see the feature in the future, even when the % goes down, and once a user has not seen a feature, they should always not see the feature in the future, even when the % goes up. (That's what I thought "stickiness" meant but maybe not?)

[gastonfournier]

Sorry, I think I misunderstood your question! But anyway that's kind of the gist with Feature Flags, giving you the ability to disable a malfunctioning feature. What you describe is probably common in AB testing. Because both have similarities, there might be some workaround, but let me forward this to the team because I'll be OOO for 2 weeks

[steffly]

@gastonfournier Hello, any update on this yet?

[gastonfournier]

Hey @steffly, sorry for the delay. I came back from vacations and forgot to check this.

TL;DR; that's not how feature flags work and I don't think that's a feature we'd consider for Unleash. But there are options.

If I understood correctly, the expectation is that if a feature was initially flawed, it should continue to be flawed for the users already exposed to it. And the problem is that the purpose of feature flags, is to allow you to disable a feature if it's found to be flawed. Modifying this fundamental behavior is not something we'd consider.

You still have a few options, if this is what your use case requires:

1. You cannot roll back a feature once it's been rolled out to a subset of users (e.g., if you roll out to 10% of customers, you cannot prevent it from reaching the entire 10% group). However, a slower rollout pace can help keep the impact minimal if issues arise.
2. You don't use gradual rollout and you use segments or other strategy
3. You can add a wrapper to unleash.isEnabled as some sort of persistent cache, something like:

if (cached(unleash.isEnabled('myFeature'))) {
    // do new stuff
} else {
    // do old stuff
}

But this last option can mess up metrics cause you will only hit unleash the first time for each user, so it will only count the first interaction, and you'll probably have to use the same context you send to unleash as the key for the cache.

Let me know if you want to chat about it a bit further, but I don't think this is something you'll be able to solve "just" with Unleash, because this idea collides with some of our design principles. In particular, having this shared cache across all instances can become a bottleneck, which goes against https://docs.getunleash.io/topics/feature-flags/scale-horizontally and will also require us to propagate potential PII data, which goes against https://docs.getunleash.io/topics/feature-flags/never-expose-pii

If you have some suggestion that didn't cross my mind, please let me know.
Best!

[steffly]

Thanks for the response!

We will be using the cache approach.