[RFC] Progressive Rollout

[markphelps]

ℹ️ This discussion is to gather feedback about a potential new feature we will add that will require API + UI changes. Please feel free to add your thoughts as comments below.

Problem

While Flipt doesn’t enforce that you configure and use Flags, Variants, Segmentation, etc in a specific way within your application, there are essentially two ways that Flags can be used:

1. Simple on/off toggles where you create a Flag and then call GetFlag from your application, using Flipt as a sort of kill-switch to quickly turn on/off code paths
2. Evaluation/segmentation where Flags can have multiple variants that can be distributed using Rules/Segments/etc. In this case, users would call Evaluate given a flag key and request context. see: [Evaluation API Docs]

One design decision we made early on is that for Evaluation to occur, a Flag must be enabled.

That is this scenario for the disabled flag asdfda:

Will return the following response if a user tries to perform an Evaluate Request for this flag:

{
  "requestId": "78bf3b1f-8b8a-4754-b30c-cb26227c9555",
  "entityId": "e3628981-3211-4e0b-9956-d5d224e8c119",
  "requestContext": {},
  "match": false,
  "flagKey": "asdfda",
  "segmentKey": "",
  "timestamp": "2023-03-14T17:22:12.882162514Z",
  "value": "",
  "requestDurationMillis": 2.000542,
  "attachment": "",
  "reason": "FLAG_DISABLED_EVALUATION_REASON"
}

The main things to note are:

• match will always be false
• reason will be FLAG_DISABLED_EVALUATION_REASON

While this might make intuitive sense, that we don’t want to allow Evaluation for flags that have been ‘killed’ as a safety measure, it does make certain uses of Flipt more difficult.

The main use-case that is not currently easily achievable is ‘Progressive Rollout’.

Progressive Rollout

Progressive Rollout allows a user to have a Flag enabled for X% of requests/users/etc, and disabled for the rest. For example, when testing a new feature in their application, engineers may want to only enable a Flipt flag for 20% of their user base until the feature is fully vetted.

Currently, there is no easy way to do this in Flipt. A common workaround is to configure a Flag / Variant / Segment / Role combo like so:

Here the flag is created with two variants: ON and Off. Then the user sets up a rule to return Off in 80% of the cases and On for the remaining 20%.

On the evaluation side, the response then looks like this:

{
  "requestId": "5d7246d3-5566-4685-9fe5-562a45966ba9",
  "entityId": "adsfdsf",
  "requestContext": {},
  "match": true,
  "flagKey": "asdfda",
  "segmentKey": "all-users",
  "timestamp": "2023-03-14T17:37:40.557381221Z",
  "value": "OFF",
  "requestDurationMillis": 1.881417,
  "attachment": "",
  "reason": "MATCH_EVALUATION_REASON"
}

Notice that:

• match is true
• value is OFF
• reason is MATCH_EVALUATION_REASON

As you can see, this is not very intuitive for the user, nor does it allow the user to quickly ‘ratchet’ up or down the rollout %. They have to find the rule, edit the percentages and save the rule for any changes to take effect.

There is slightly more context available in this GitHub issue as well: #368.

Ideal Solution

Given the above problem(s), an ideal solution would have the following properties:

• Ability to quickly update rollout percentage of ‘On’ vs ‘Off’ or ‘Enabled’ vs ‘Disabled’ in the UI (ie: with a slider in the UI)
• Make ‘rollout’ type use cases seem more native/intuitive from an evaluation perspective
• Prevent users from having to hardcode ‘On’ and ‘Off’ in their codebase for using Flipt for flag rollouts (evaluation/client side)
• Prevent users from having to hardcode ‘On’ and ‘Off’ in their codebase if they are using the API to manage/create flags/variants
• Be backward compatible with the existing UI/API. Meaning any solution would still allow users to see and manage their flags/variants/etc as they do now, without having to update their existing code in their applications.
• Still allow for segmentation/rules/constraints/etc as they exist today. We still want to allow our users to specify ‘how’ the rollout ‘on’ / ‘off’ gets distributed based on properties they define.

Potential Solutions

Introduce Flag Type

This solution requires us to add a type field to our Flag model. For this example, we will refer to the new type as type: rollout. For backward compatibility, all Flags that have been created without a type (all that currently exist) will be `type: multi-variant (or similarly named).

Here is a quick overview of the two types:

• Multi-Variant Support on/off but only for gating evaluation as exists today. Variants are added/updated/deleted as they are today. Still supports rules/distributions as exists today for percentage rollouts of variants

  use-cases: Experimentation, Permissions Gating, Continuous Config, Kill Switches (flag with 0 variants)

• Rollout Still allows for setting enabled on/off. This would act as the ‘kill switch’ in case of rollout gone poorly. Would not allow adding variants by the user. Would have 2 hardcoded: “true” and “false” or similar.

use-cases: Progressive Rollout

Rollout-type flags would NOT allow adding or deleting variants. We may potentially allow users to rename them, however, we need to make it clear which value represents ‘truthy’ and which ‘falsey’. Users could add a description and potential ‘attachment’ field to these variants as they can today.

🍦Note: In this solution, we are basically creating ‘variants’ on behalf of the user automatically. One that represents ‘true’ and one ‘false’. Then we are adding some UI sugar allowing them to quickly change the distribution of these values using a ‘slider’ or similar. In the backend, API, etc all existing functionality still applies.

UI Changes

Flag Creation

The UI would need to change at flag creation time, allowing users to specify the ‘flag type’. Multi-variant would be the default, however, if they selected ‘rollout’ then the Variants subheading would change to show only the two allowed ‘truthy’ and ‘falsey’ values that are hard coded.

⚠️ Important: Flag types would be immutable after creation. Users would not be allowed to change flag types as this would require us to have to make assumptions or lead to additional complexity.

Rollout / Targeting

We could likely make the ‘rollout slider’ available rule/distribution edit slide over. This would allow the user to quickly update a rollout over time, without having to manually adjust percentages.

Evaluation Console

The evaluation console would not likely have to change as the Flag could still be ‘Evaluated’, however, the only values returned would be: "true" and "false". This is discussed further in API changes next.

API Changes

This section describes both the GRPC and REST APIs as one is driven by the other thanks to GRPC-Gateway. In the example request/responses, I’ll be using REST notation/examples because I find it easier to use for the purpose of documenting potential changes.

The API would need to maintain our backward compatibility promise, therefor all changes would be additive.

Flag

https://www.flipt.io/docs/reference/flags/get-flag

The Flag request/response type would add the new type or similarly named field:

{
  "key": "asdf",
  "name": "asdf",
  "description": "",
  "type": "FLAG_TYPE_ROLLOUT",
  "enabled": true,
  "createdAt": "2023-03-14T19:47:37.117547964Z",
  "updatedAt": "2023-03-14T19:47:37.117547964Z",
  "variants": []
}

• Create Flag

https://www.flipt.io/docs/reference/flags/create-flag

The CreateFlag request object would include type and require one of our new enum values. If none was provided it would default to type: FLAG_TYPE_MULTIVARIANT or similar.

• Update Flag

https://www.flipt.io/docs/reference/flags/update-flag

UpdateFlag would disallow changing a flag’s type, much like we disallow changing a flag’s key today.

Variants

The variants API would not likely change much, the major change would be that users would not be allowed to issue AddVariant requests for flags that have type: rollout.

Depending on further discussion, we could potentially allow users to change the ‘truthy’ and ‘falsey’ variant keys, however, this would require us to internally track which variant represents which.

Segments/Constraints/Rules/Distributions

The other major API endpoints would likely also not need to change much. Since variants drive rules and distributions, and we are basically creating ‘default’ variants on behalf of the user, there really isn't much additional functionality needed for these endpoints.

Evaluation

The evaluation API is where it could get a bit tricky. We could go the simple route and leave the API as is. The responses would look something like this:

• True Rollout Variant

{
  "requestId": "5d7246d3-5566-4685-9fe5-562a45966ba9",
  "entityId": "adsfdsf",
  "requestContext": {},
  "match": true, // will always be true as this is determined by the 'rule'
  "flagKey": "asdfda",
  "segmentKey": "all-users",
  "timestamp": "2023-03-14T17:37:40.557381221Z",
  "value": "true", // or whatever the truthy key is
  "requestDurationMillis": 1.881417,
  "attachment": "",
  "reason": "MATCH_EVALUATION_REASON"
}

• False Rollout Variant

{
  "requestId": "5d7246d3-5566-4685-9fe5-562a45966ba9",
  "entityId": "adsfdsf",
  "requestContext": {},
  "match": true, // will always be true as this is determined by the 'rule'
  "flagKey": "asdfda",
  "segmentKey": "all-users",
  "timestamp": "2023-03-14T17:37:40.557381221Z",
  "value": "false", // or whatever the falsey key is
  "requestDurationMillis": 1.881417,
  "attachment": "",
  "reason": "MATCH_EVALUATION_REASON"
}

• Adding Flag Type

I think it might not be a bad idea to also add flagType to the evaluation response as well. This would likely enable users to check this field first, so then they could determine what the expected values in value would be (either ‘truthy’, ‘falsey’ or some user-provided variant).

{
  "requestId": "5d7246d3-5566-4685-9fe5-562a45966ba9",
  "entityId": "adsfdsf",
  "requestContext": {},
  "match": true, // will always be true as this is determined by the 'rule'
  "flagKey": "asdfda",
  "flagType": "FLAG_TYPE_ROLLOUT", // new field, would default to 'FLAG_TYPE_MULTIVARIANT'
  "segmentKey": "all-users",
  "timestamp": "2023-03-14T17:37:40.557381221Z",
  "value": "false", // or whatever the falsey key is
  "requestDurationMillis": 1.881417,
  "attachment": "",
  "reason": "MATCH_EVALUATION_REASON"
}

Another idea would be to create a new API endpoint that only supported type: rollout flags, which could potentially make the EvaluationResponse more intuitive, however, I’m leaning on the side of simplicity and think we could just use what we have.

Client Changes

As our clients are mostly all auto-generated, we would have limited customization that we could provide here. One of the goals in the [Ideal Solution] section was to:

• Prevent users from having to hardcode ‘On’ and ‘Off’ in their codebase for using Flipt for flag rollouts (evaluation/client side)

Unfortunately, this solution does not accomplish this goal since the value field is of type string, so users will still have to evaluate this value into a boolean themselves or compare it to some known values in their code. This would be doubly true if we allowed users to change the variant key for the truthy and falsey values.

Data Changes

The majority of changes other than the UI and API would be in the database/migrations and business logic.

For the database side, we would need to add a new type enum field to the flags table and default all existing flags to the ‘multivariant’ or whichever name we choose.

The internal business logic and validation would need to change, for example, to only allow variants to be added to flags of type multivariant, however, I don’t think this would be too challenging.

Pros / Cons

Pros:

• Not a huge feature/change to our existing application/data model.
• Accomplishes most of the goals set out in Ideal Solution

Cons:

• Evaluation is not super intuitive still in my opinion
• Forces users to think upfront about how they are going to use a flag when creating it since the type is immutable.
• Does not solve the problem of requiring users to hardcode ‘on’ / ‘off’ or similar for comparing evaluation results in their calling code