Distributed Feature Flags
This is pre-release software. The Consul backend support has been used in production at VSCO for a year.
As of version 0.4.0-rc0, the etcd backend is no longer supported.
Decider is a feature flag system with adaptable backends. It supports both percentile
and boolean
flags for controlled infrastructure rollouts and kill switches. Decider is built to be adaptable to any backing datastore. At the moment, Consul and Redis are supported.
Decider has four major components.
Client
for use within your Go applicationsServer
for accessing features over HTTPWatcher
observes change in the datastore and writes them to diskCLI
for managing features, watches, and starting the server
Each of these components are comprised of lower level libraries that you can use to suit your system's specific needs.
Feature flags have many use cases and there are many implementations. With Decider, the three supported types of flags are boolean
, percentile
, and scalar
. For our purposes at VSCO, these have been enough to handle our needs.
An example use case for a boolean
flag would be an API kill switch that could alleviate load for a backing database.
disable-load-heavy-api-endpoint => true
Your code would look something like this.
if dcdr.IsAvailable("disable-load-heavy-api-endpoint") {
// DB is having a bad day, please check back later
} else {
// All good, go about your day
}
Percentiles work much the same way but allow you to stress features or new infrastructure with a percentage of the request volume.
A common use case for percentile
features would be testing out a new backend store with dual write percentage.
rollout-new-fancy-db-dual-write => 0.1
// Handle the write to the existing store
if dcdr.IsAvailableForID("rollout-new-fancy-db-dual-write", user.Id) {
// If the `user.Id` falls into 10% of requests do the dual write
}
Percentiles have an added bonus: you may use their float64
values as scalars in certain cases.
Here, we'll use the float value to scale the wait time for DB inserts between 0-1000ms.
daemon-db-insert-wait-ms => 0.1
// waitMS would be 1000*0.1 => 100
waitMS := dcdr.ScaleValue("daemon-db-insert-wait-ms", 0, 1000)
time.Sleep(waitMS * time.Millisecond)
Read more on how to use the Client
.
Feature flags and remote configuration are hard problems to solve in the general sense. Most organizations will have many corner cases unique to their own infrastructure and policies that are cumbersome to solve in an abstract way. Decider is an extracted set of flexible libraries that we at VSCO have developed over the past year that have worked well for us in solving these issues.
This package does not set out to provide features like authentication or ACLs, but it does aim to provide enough of the tooling and libraries so that you can do so yourself.
In order to allow for expanding use cases and to avoid naming collisions, Decider provides arbitrary scoping of feature flags. An example use case would be providing separate features sets according to country code or mobile platform. Additionally, multiple Decider instances can be run within a cluster with separate namespaces and key sets by configuring config.hcl
.
Due to the sensitive nature of configuration management, knowing the who, what, and when of changes can be very important. Decider uses git
to handle this responsibility. By easily specifying a git
repository and its origin in config.hcl
, Decider will export your keyspace as a JSON
file and then commit and push the changeset to the specified origin. Of course, this is all optional if you enjoy living dangerously.
It's nice to know when changes are happening. Decider can be configured to emit Statsd events when changes occur. Custom event tags can be sent as well if your collector supports them. Included in this package is a DataDog adapter with Event and Tag support. Custom stats can also be configured by supplying a custom stats.IFace
implementation.
- Install via
go get
go get -a github.com/vsco/dcdr
- Install via release
- Build from source
git clone git@github.com:vsco/dcdr.git
cd dcdr
script/bootstrap
script/install
The dcdr
CLI has comprehensive help system for all commands.
dcdr help [command]" for more information about a command.
Features have several fields that are accessible via set
command.
-n, --name="flag_name"
the name of the flag to set
-v, --value=0.0-1.0 or true|false
the value of the flag
-c, --comment="flag description"
an optional comment or description
-s, --scope="users/beta"
an optional scope to nest the flag within
dcdr set -n new-feature -v 0.1 -c "some new feature" -s user-groups/beta
The above command sets the key dcdr/features/user-groups/beta/new-feature
equal to 0.1
and commits the update to the audit repo.
Listing features can be filtered by a given scope
and prefix
. Any further fanciness can be handled by piping the output to grep
or less
.
-p, --prefix="<flag-prefix>"
List only flags with matching prefix.
-s, --scope="<flag-scope>"
List only flags within a scope.
dcdr list -p new -s user-groups/beta
Features are removed using the dcdr delete
command and take a name
and scope
parameters. If no scope
is provided the default
scope is assumed. Once deleted and if you have a repository configured, Decider will commit the changeset and push it to origin.
-p, --prefix="<flag-prefix>"
Name of the flag to delete
-s, --scope="<flag-scope>"
Optional scope to delete the flag from
dcdr delete -n another-feature -s user-groups/beta
The watch
command is central to how Decider features are distributed to nodes in a cluster. It observes the configured namespace and writes a JSON
file containing the exported structure to the Server:OutputPath
.
By default the Decider configuration and watch path are located in /etc/dcdr
. If this path does not exist you will need to create it.
sudo mkdir /etc/dcdr
sudo chown `whoami` /etc/dcdr
You can override this location by setting the DCDR_CONFIG_DIR
environment variable. More on configuration can be found here.
Decider has many moving parts that require orchestration in order to do a proper demonstration of how they all work together.
So thanks to Docker Compose we can bundle this up quite easily. This example uses a
Consul backend and starts a dcdr watch
and dcdr server
for you.
./script/compose
This starts a Consul agent for the backend, a Decider server, and a Decider Watcher.
With these services running you can now interact with the dcdr
CLI via
docker-compose exec
from another terminal window through the dcdr_server
container.
command to add/modify a feature flag:
$ docker-compose exec dcdr_server dcdr set -n feat-foo -v true
[dcdr] set flag 'dcdr/features/default/feat-foo'
result logs from running docker-compose log:
dcdr_watch_1 | [dcdr] 2018/03/13 18:25:48 wrote changes to: /etc/dcdr/decider.json
command to fetch feature flags from Decider server API:
$ docker-compose exec dcdr_server curl 127.0.0.1:8000/dcdr.json
{
"dcdr": {
"info": {},
"features": {
"feat-foo": true
}
}
}
result logs from running docker-compose log:
dcdr_server_1 | 127.0.0.1 - - [13/Mar/2018:18:36:51 +0000] "GET /dcdr.json HTTP/1.1" 200 102 "" "curl/7.52.1"
Now you can update features, CURL the results from the server, and see the changes update in realtime.
This example uses the Consul backend. If you need instructions for getting Consul installed, check their Getting Started page.
Let's start a development consul agent
with an empty feature set and see how this all works together. For simplicity we can use the default Decider configuration without a git repository or stats.
consul agent -bind "127.0.0.1" -dev
This will start a local Consul agent ready to accept connections on http://127.0.0.1:8500
. Decider should now be able to connect to this instance and set features.
# check that we can talk to the local agent
~ → dcdr list
[dcdr] no feature flags found in namespace: dcdr
# set a feature into the 'default' scope.
~ → dcdr set -n example-feature -v false
[dcdr] set flag 'dcdr/features/default/example-feature'
# set a feature into the 'user-groups/beta' scope.
~ → dcdr set -n example-feature -v true -s user-groups/beta
[dcdr] set flag 'dcdr/features/user-groups/beta/example-feature'
~ → dcdr list
Name Type Value Comment Scope Updated By
example-feature boolean false default chrisb
example-feature boolean true user-groups/beta chrisb
Here we have set the feature example-feature
into two separate scopes. In the 'default' scope the value is false
and in the 'user-groups/beta' scope it has been set to true.
# start the watcher
~ → dcdr watch
[dcdr] 2016/03/09 17:56:17.250948 watching namespace: dcdr
[dcdr] 2016/03/09 17:56:17.365362 wrote changes to /etc/dcdr/decider.json
The watcher is now observing your «Namespace» and writing all changes to the Server:OutputPath
default /etc/dcdr/decider.json
.
The easiest way to view your feature flags is with dcdr server
. This is a bare bones implementation of how to access features over HTTP. There is no authentication, so unless your use case is for internal access only you should include the server
package into a new project and assemble your own. The server is built with gorilla/mux router and is extensible by adding additional middleware. Read more on custom servers here.
# start the server
~ → dcdr server
[dcdr] started watching /etc/dcdr/decider.json
[dcdr] 2016/03/09 18:03:46.211150 serving /dcdr.json on :8000
The server is now running on :8000
and features can be accessed by curling :8000/dcdr.json
. In order to access your scopes the server accepts a x-dcdr-scopes
header. This is a comma-delimited, priority-ordered list of scopes. Meaning that the scopes should provided with the highest priority first. For now we only have one scope so let's start simple.
# curl with no scopes
~ → curl -s :8000/dcdr.json
{
"info": {},
"dcdr": {
"example-feature": false
}
}
Here we see that the default value of false is returned. The info
key is where information like the current SHA of the repository would be if one was configured. Next, if we add the scope header we can access our scoped values.
~ → curl -sH "x-dcdr-scopes: user-groups/beta" :8000/dcdr.json
{
"dcdr": {
"info": {},
"features": {
"example-feature": true
}
}
}
Included in this package is a Go client. By default this client uses the same config.hcl
for its configuration. You may also provide custom your own custom configuration as well using config.Config
and the client.New
method. For this example we will assume the defaults are still in place and that the features from the above example have been set.
import "github.com/vsco/dcdr/client"
// Initialize a client with the default configuration
client, err := client.NewDefault()
if err != nil {
panic(err)
}
The client has three main methods for interacting with flags IsAvailable(feature string)
. IsAvailableForID(feature string, id uint64)
, and ScaleValue(feature string, min float64, max float64)
.
This method is for checking boolean
features or 'kill switches'. If a percentile
feature is passed to this method it will always return false. So don't do that.
# set the default feature
dcdr set -n example-feature -v false
client, err := client.NewDefault()
if err != nil {
panic(err)
}
// example-feature would be false
if client.IsAvailable("example-feature") {
fmt.Println("example-feature enabled")
} else {
fmt.Println("example-feature disabled")
}
This example initializes a new Client
and begins watching the 'default' feature scope. It then checks the example-feature
and runs the appropriate code path given the current value of the feature.
To initialize a Decider Client
into a given set of scopes use the WithScopes(scopes ...string)
method. This method creates a new Client
with an underlying feature set that has the provided scope
values merged onto the default set. If a feature does not exist in any of the provided scopes the client will fallback to the 'default' scope
to find the value. If the feature does not exist in any scope the client simply returns false
.
# set the scoped feature
dcdr set -n example-feature -v true -s user-groups/beta
client, err := client.NewDefault()
scopedClient := client.WithScopes("user-groups/beta")
if err != nil {
panic(err)
}
// example-feature would be true
if scopedClient.IsAvailable("example-feature") {
fmt.Println("example-feature enabled")
} else {
fmt.Println("example-feature disabled")
}
# set a feature that does not exist in user-groups/beta
dcdr set -n another-feature -v true
client, err := client.NewDefault()
scopedClient := client.WithScopes("user-groups/beta")
if err != nil {
panic(err)
}
// another-feature would be true
if scopedClient.IsAvailable("another-feature") {
fmt.Println("another-feature enabled")
} else {
fmt.Println("another-feature disabled")
}
This method is used when a feature needs to be rolled out to only a percentage of requests. Functionally IsAvailableForID
works exactly as IsAvailable
with the exception of its id
argument. Both the feature
and id
arguments are joined to generate a uint64
using hash/crc32
. Which when combined with the float64
value of feature
can compute into what percentile a given request falls.
See the Client#withinPercentile
method for more details.
# set a feature to 50%
dcdr set -n new-feature-rollout -v 0.5
client, err := client.NewDefault()
if err != nil {
panic(err)
}
userId := unint64(5)
// new-feature-rollout would be true
if client.IsAvailableForID("new-feature-rollout", userId) {
fmt.Println("new-feature-rollout enabled")
} else {
fmt.Println("new-feature-rollout disabled")
}
ScaleValue
uses the same float64
values as IsAvailableForID
but in this case these values are used to obtain a new value scaled between a min
and a max
.
For instance:
Given the feature db-insert-wait-ms => 0.5
. When provided to the ScaleValue
method would result in the following.
for {
insertWaitMs := dcdr.ScaleValue("db-insert-wait-ms", 0, 1000)
time.Sleep(insertWaitMs * time.Millisecond) // waits for 500ms
db.Insert("some-value")
}
Exposing your feature flags to the open internet would be a terrible idea in most cases. The default server will work fine as long as access is restricted to internal network clients but what if we want to allow access to mobile devices? Since there are entirely too many auth strategies to cover and we are kind of lazy, Decider Server
allows you to add middleware to customize its behavior to suit your authentication needs.
Below is an example of how to do authentication very poorly. However, if you look close enough you can imagine exactly where you might add a DB lookup for an OAuth token or something similar. The Use
method takes variadic server.Middleware
as a param to allow chainable customizations.
// Middleware helper type for handlers that receive a `Client`.
type Middleware func(client.IFace) func(http.Handler) http.Handler
const AuthorizationHeader = "Authorization"
// MockAuth example authentication middleware.
// Checks for any value in the http Authorization header.
// If no value is found a 401 status is sent.
func MockAuth(c client.IFace) func(http.Handler) http.Handler {
return func(h http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(AuthorizationHeader) != "" {
h.ServeHTTP(w, r)
} else {
w.WriteHeader(http.StatusUnauthorized)
}
}
return http.HandlerFunc(fn)
}
}
func main() {
// Create a new Server and Client
srv, err := server.NewDefault()
if err != nil {
panic(err)
}
// Add the MockAuth to the middleware chain
srv.Use(MockAuth)
// Begin serving on :8000
// curl -sH "Authorization: authorized" :8000/dcdr.json
srv.Serve()
}
Here is a bit more useful example. This ScopedCountryCode
middleware takes the X-Country
header and appends it to x-dcdr-scopes
. We then use the country-code/<cc>
scope to look up per country feature flags from the K/V store.
const CountryCodeHeader = "X-Country"
const DcdrScopesHeader = "x-dcdr-scopes"
func ScopedCountryCode(c client.IFace) func(http.Handler) http.Handler {
return func(h http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
cc := strings.ToLower(r.Header.Get(CountryCodeHeader))
if cc != "" {
// Check for existing scopes and append 'country-code/xx'
scopes := strings.Split(r.Header.Get(DcdrScopesHeader), ",")
scopes = append(scopes, fmt.Sprintf("country-codes/%s", cc))
r.Header.Set(DcdrScopesHeader, strings.Join(scopes, ","))
}
h.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
}
A full working example can be found in server/demo/main.go.
All configuration lives in config.hcl
. By default Decider looks for this file in /etc/dcdr/config.hcl
. You will need to create the /etc/dcdr
directory. Your permissions depending on the machine may differ but to get started locally do the following.
sudo mkdir /etc/dcdr
sudo chown `whoami` /etc/dcdr
dcdr init
The default config path can also be overriden by setting the DCDR_CONFIG_PATH
environment variable to a location of your choosing.
Running dcdr init
will create a default config file for you if one does not already exist. Once you have edited this file with your, statsd, and git repo configurations you can view this info by running the dcdr info
command.
To create a new repository from scratch. Configure the config.hcl
file with your RepoPath
and RepoURL
and then run dcdr init --create
. This will create the repo add an empty JSON
file and attempt to push it to the specified origin.
Username = "twoism"
Namespace = "dcdr"
Storage = "consul" // redis
Consul {
Address = "127.0.0.1:8500"
}
//Redis {
// Address = ":6379"
//}
Watcher {
OutputPath = "/etc/dcdr/decider.json"
}
Server {
JsonRoot = "dcdr"
Endpoint = "/dcdr.json"
}
Git {
RepoURL = "git@github.com:vsco/decider-test-config.git"
RepoPath = "/etc/dcdr/audit"
}
Stats {
Namespace = "dcdr"
Host = "127.0.0.1"
Port = 8126
}