Tracking proposal

[toddbaert]

Tracking proposal

This proposal roughly outlines support for tracking events in OpenFeature.
Particularly when doing experimentation, it may be desirable to associate particular user actions (goals) with feature flag evaluations.
For example, to understand if a UI change controlled by a feature flag increases the likelihood of a user adding an item to their cart, a tracking event which records the fact that a user has added an item is required.
It's important to note that this action doesn't occur in the same transaction (read: HTTP request) as the feature flag evaluation; hence the need for this additional API.

Tracking API (Application-author)

A primitive example of the tracking API presented to application authors:

// tracking without explicit evaluation context (merged similarly to evaluation context in flag evaluations)
// the argument is an event identifier
client.track('user-added-item-to-cart');

// tracking with event data
// the first argument is an event identifier, the second auxiliary event data
// note that the `value` is a special "well-defined" property (similar to `targetingKey`) which can be readily mapped to equivalents in various SDKs (`eventValue` (Split), `metricValue` (LaunchDarkly), `value` (DevCycle))
client.track('user-added-item-to-cart', {
  value: 49.99, // in this example, the value corresponds to the cart subtotal
  itemSku: 123456 
});

// tracking with event data and explicit evaluation context
// the first argument is an event identifier, the second auxiliary event data, third is the evaluation context
// note that the `value` is a special "well-defined" property (similar to `targetingKey`) which can be readily mapped to equivalents in various SDKs (`eventValue` (Split), `metricValue` (LaunchDarkly), `value` (DevCycle))
client.track('user-added-item-to-cart', {
    value: 49.99, // in this example, the value corresponds to the cart subtotal
    itemSku: 123456
  }, {
    targetingKey: 'john.doe@email.net',
    age: 52,
});

Tracking interface

For simplicity, the tracking functionality is added to the existing provider objects, either by extending the existing provider interface defined in the implementation, or adding a new interface.
The implementation of the tracking method is optional; provider authors need not support tracking.
SDK implementations must ensure that providers which don't implement track result in a no-op when client.track(...) is called.

/**
 * Minimalist tracking payload
 */
export interface TrackingData {

  /**
   * Optional numeric value for the tracking event.
   * This serves as a well-known value to easily map to the metric-values existing in many SDKs. 
   */
  value?: number;

}

Dynamic context example (server-side)

/**
 * Interface to implement in order to support tracking in your provider (or this could be an optional extension of the existing provider interface)
 */
export interface Tracking {

  /**
   * Process a tracking event.
   */
  track(eventName: string, context: EvaluationContext, data: TrackingData): void;

}

Static context example (client-side)

/**
 * Interface to implement in order to support tracking in your provider.
 */
export interface Tracking {

  /**
   * Process a tracking event.
   */
  track(eventName: string, context: EvaluationContext, data: TrackingData): void;

}

Supporting heteromorphic tracking

Native tracking support

For flag management systems which implement tracking, implementation simply amounts to utilizing the tracking API of the underlying system:

- export MyProvider implements Provider {
+ export MyProvider implements Provider, Tracking {

  // constructor may accept options related to tracking configuration (caching, batching, aggregation, etc)  
  constructor(options: MyOptions) {
    myUnderlyingSdk = new UnderLyingSdk(options);  
  }

  // existing implementation...

+ // tracking method implementation
+ track(eventName: string, context: EvaluationContext) {
+   // call the underlying method, note that the context transformation or other data manipulation might necessary
+   await myUnderlyingSdk.trackingMethod(eventName, contextTransformer(context));
+ }

}

Tracking providers

Some flag management systems do not support tracking as a concept at all.
Despite this, OpenFeature users using these systems might be interested in leveraging tracking.
This is a unique benefit that OpenFeature can offer to users; the ability to use OpenFeature as a facade which combines feature flagging with user-analytics.
To support this, tracking providers which only implement the Track interface can be created to supplement the functionality of the flag provider.
Combined with hooks which report feature flag evaluations to the tracking/analytics platform in question, a tracking provider can allow for robust experimentation even for flag management systems that don't support tracking directly.

Below is a simplified diagram depicting how hooks and the tracking API can be combined to facilitate feature flag experiments, even if the flag management platform in use doesn't support it directly. Providers are omitted for simplicity.

sequenceDiagram
  Evaluation API->>+Tracking Hook: evaluate
  Tracking Hook->>Analytics Platform: before
  Tracking Hook->>Analytics Platform: after
  Tracking Hook->>-Evaluation API: evaluate
  Evaluation API->>Analytics Platform: track

Loading

Tracking provider wrapper

To allow users to combine the functionality of a flag provider and a tracking provider, a simple utility can be created which aggregates the two into a single object. Alternatively, the existing multi-provider could also be leveraged for this purpose.

// Use the multi-provider to add tracking functionality to a provider which doesn't natively support it
OpenFeature.setProvider(new MultiProvider([
  {
    provider: new MyProviderWithoutTracking(),
  },
  {
    provider: new MyDedicatedTrackingProvider()
  }
]));

Life-cycle management

The specification already rigorously defines a lifecycle.
This proposal aims to leverage this existing lifecycle to prevent the addition of additional complexity.
The existing provider init, shutdown, etc, can be used for relevant operations of a tracking provider.
For example:

• init: establish connections, setup caches
• shutdown: flush buffers complete batches

Open questions:

• Are the proposed tracking API parameters sufficient?
  • Do we need more than an event name/id and an optional context?
  • Do we need to differentiate "event details" from evaluation context? Should we add another param for this?
  • Relevant spec from Segment: https://segment.com/docs/connections/spec/track/
• Should the track() function be synchronous for the static context paradigm and asynchronous for the dynamic paradigm, similar to the evaluation methods?

[kinyoklion]

Do we need to differentiate "event details" from evaluation context? Should we add another param for this?

I think we need at least an optional data/properties parameter. In the more full example context and properties are independent in the segment track spec.

Copying the complete track call here for comparison:

{
  "anonymousId": "23adfd82-aa0f-45a7-a756-24f2a7a4c895",
  "context": {
    "library": {
      "name": "analytics.js",
      "version": "2.11.1"
    },
    "page": {
      "path": "/academy/",
      "referrer": "",
      "search": "",
      "title": "Analytics Academy",
      "url": "https://segment.com/academy/"
    },
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36",
    "ip": "108.0.78.21"
  },
  "event": "Course Clicked",
  "integrations": {},
  "messageId": "ajs-f8ca1e4de5024d9430b3928bd8ac6b96",
  "properties": {
    "title": "Intro to Analytics"
  },
  "receivedAt": "2015-12-12T19:11:01.266Z",
  "sentAt": "2015-12-12T19:11:01.169Z",
  "timestamp": "2015-12-12T19:11:01.249Z",
  "type": "track",
  "userId": "AiUGstSDIg",
  "originalTimestamp": "2015-12-12T19:11:01.152Z"
}

In LD we have a "metricValue" and "data" any combination of which can be provided. If there are properties, then we can potentially add some rules internal to the provider. Like a key of "metricValue" will be the associated metric and everything else would be "data". If you want any meaningful measurement then you are doing to need some kind of properties to measure anything other than a binary outcome.

[moredip]

+1; I can think of many cases where you want to include additional info about the event itself, distinct from evaluation context.

In fact, I'm a bit surprised that we'd include the evaluation context when tracking events. Maybe there're good reasons to do so, but FWIW it was a bit surprising to me.

My assumption would be that the api would be client.track('event-name', eventProperties, evalContext)

[toddbaert]

+1; I can think of many cases where you want to include additional info about the event itself, distinct from evaluation context.

In fact, I'm a bit surprised that we'd include the evaluation context when tracking events. Maybe there're good reasons to do so, but FWIW it was a bit surprising to me.

My assumption would be that the api would be client.track('event-name', eventProperties, evalContext)

I think I agree. I will update the API proposal accordingly.

I'm not yet sure about the metricsValue.

[toddbaert]

One of the reasons I chose to keep context is because the segment spec has context as a dedicated property besides the event properties:

https://segment.com/docs/connections/spec/track/#example

[jonathannorris]

I also think having a context is likely to be important for Feature Flag / Experimentation vendors as we want to associate an event with some trackable user data. For example, at DevCycle we require user and the event in our track() method. I imagine that would also be a requirement of analytics vendors as you need to tie your analytics events to a specific user.

I also wonder if we should include optional default types like a numerical value field; we also have a target field, and all other properties are populated under metaData. It wouldn't be difficult for us to map those fields internally in our provider, but I do wonder if common fields like value / target / date or anything like that should be defined as optional top-level properties, or do we just leave EventData as a totally open schema.

[moredip]

I'm a bit surprised that we'd include the evaluation context when tracking events.

I just looked again and saw that we have examples for both dynamic and static contexts. For dynamic contexts it makes sense that we'd want to provide evaluation context at the point of the track call. But for static contexts why wouldn't it work the same as flag evaluation itself, where you provide the context once, and don't need to provide it for each track call?

[toddbaert]

Yes, I think that's a good catch. I will remove it from that call.

[toddbaert]

I've left the context arg for the provider, since even in our static context, the provider methods receive a context; it's just that the client API doesn't take one - the static context is passed to the resolver methods though, even on the client: https://github.com/open-feature/js-sdk/blob/a621e6d80497d9daab44bbfc31c9639ab9498940/packages/client/src/provider/provider.ts#L38

[moredip]

got it, makes sense. I sometimes get mixed up between the author-facing client API vs the provider API.

[moredip]

Do we want to also take on responsibility for sending a flag-evaluation tracking event into a provider whenever the OF SDK performs an evaluation, or do we want to leave that as a responsibility of the provider. This seems like a key decision for us to make.

Leaving it a responsibility of the provider keeps the OF SDK simpler - and prevents us from having to spec out potentially complex semantics. Taking on responsibility for reporting flag evaluation would allow integrators to bolt analytics capabilities onto a flagging platform that doesn't support it natively, which is actually pretty compelling IMO.

[jonathannorris]

I think this would potentially make for a good optional method on the Tracking interface, as most flagging vendors likely track this already internally when their evaluation methods are called, but could a cleaner integration for pure analytics providers.

[toddbaert]

For providers that do not support analytics, I think this is a perfect use-case for hooks (ie: a Segment hook that sends evaluations to segment, combined with a Segment-specific tracking provider). I tried to capture that in the mermaid diagram above. For providers which do, that's up to their internal implementation; at least that's how I was thinking about it.

[moredip]

I wonder if we want to provide some sort of escape-hatch in the track API to allow provider-specific parameters.

For example, LD's track API takes an optional metricValue parameter. Would we want to provide some way for an application author who's using OF on top of LD to pass this metricValue into the OF track call?

One (perhaps clunky) approach would be an optional providerParams arg:
track('an-event',eventProperties,evaluationContext,providerParams) 😒

[kinyoklion]

In LD we have a "metricValue" and "data" any combination of which can be provided. If there are properties, then we can potentially add some rules internal to the provider. Like a key of "metricValue" will be the associated metric and everything else would be "data". If you want any meaningful measurement then you are doing to need some kind of properties to measure anything other than a binary outcome.

Is my expectation of how we would likely handle this. It isn't 100% ideal, but not worse than the way we need to encode multi-contexts into an OF context.

[kinyoklion]

It would, of course, be nicer to have a direct metricValue.

[toddbaert]

I think what you describe in the quote above makes sense; it's similar to what we do with evaluation context transformation now.

I am totally open to having metricValue be a special case, or have some dedicated equivalent, but I think I want to understand a bit more about this field and whether other providers have similar fields.

[jonathannorris]

I responded similarly above. But for DevCycle we would likely let our customers know how we would be mapping value / target / date fields out of the EventData, similarly to how we do it for Context properties to DevCycleUser.

[beeme1mr]

• Should the track() function be synchronous for the static context paradigm and asynchronous for the dynamic paradigm, similar to the evaluation methods?

Is there a reason why track() couldn't be synchronous in both paradigms?

[beeme1mr]

I would expect most implementations would synchronously be putting things into some buffer.

Yeah, I agree. This will likely be the case with OFREP as well. I would image we'll just batch events and flush the queue at a regular interval.

[toddbaert]

Sounds like we have a bit of a disagreement here. Maybe we can discuss further at the community meeting.

[moredip]

I guess I'm a bit worried since we can always call sync methods from async, but if we mark it sync and some providers require async for some reason, we'll have painted ourselves into a corner.

Even if they did require async (which feels like a very surprising design decision for an analytics/o11y type API) then you could just make a fire-and-forget async call from a synchronous function. Having a tracking call be anything other than fire-and-forget doesn't seem very plausible.

I get the desire to stick with the API that could handle both cases, but in this case I would argue for the nicer ergonomics of a synchronous API, given that I think almost every underlying implementation would also have a sync API.

[toddbaert]

I tend to agree.

[jonathannorris]

yea, I strongly agree that these functions should be sync and fire-and-forget, error handling can be done through a hook if required.

If we are assuming that most platforms will be batching and sending these events up later, we may want to add support for an optional flushEvents() method or something like it that could be async with a callback that events are actually flushed. Example from us: https://docs.devcycle.com/sdk/client-side-sdks/javascript/javascript-usage#tracking-events

[moredip]

both LD and Split's track API has a dedicated metricValue param. Perhaps we could establish a convention (and even bake into a spec) that an event property of metricValue maps to a provider's special metric value value, if they have one.

My 2c is that we should establish some sort of convention, to encourage providers to do the mapping consistently, but avoid baking it into a spec.

[toddbaert]

@jonathannorris see the above point(s). I think this is similar to what you're mentioning.

[moredip]

I can't find doc from Split on this. Can you reference it?

Sorry I was a bit loose with my language - Split. call it VALUE or eventValue, not metricValue.

https://help.split.io/hc/en-us/articles/360058730852-Browser-SDK#track

[toddbaert]

Ah, perfect. That certainly seems analogous. I think I'm convinced we need a dedicated equivalent for this, at least.

[weyert]

Wouldn't be possible to include it in the properties? Similar to some SDK would have special meaning to properties, e.g.

client.capture(
  'event_name',
  {
    $set: { name: 'Max Münster'  },
    $set_once: { initial_url: '/blog' },
  }
)

[toddbaert]

Yes, we could include it in the properties. I think the remaining question is if we want 1 or 2 "well-defined" properties that can be easily mapped between providers (similar to how our existing context specifies a well-defined targetingKey for identification purposes).

[lukas-reining]

I have one thought about the separation of concerns.
If tracking would always be called from the application and not the provider itself, I would favor a TrackingProvider as a separate object.
In many corps, there are different systems for tracking and evaluation.
In this case as you said @toddbaert we could use the multi provider but this means we might have two "half" providers.
This means that we would always have to document if the provider implements feature flagging and/or tracking.
If we always had a MySystemProvider and a MySystemTrackingProvider which could be set by OpenFeature.setProvider and OpenFeature.setTrackingProvider, this would be nicely separated.
I think these semantics would be easier to understand and better to combine systems and capabilities for different domains.

In the case that the OpenFeature SDK would implement automatic feature flag tracking, it could rely on this.
It could e.g. do a feature flag evaluation with the registered provider for a specific domain and then use the domain registered or default tracking provider to track that evaluation.

If we decide, that the providers should implement feature flag event racking themself, I think having both tracking and flagging in the same provider makes sense.

[toddbaert]

This was my initial feeling too, but I leaned against it after some thought. My reasoning is this:

I think from both an SDK implementation perspective and a usage perspective, the implementation you're proposing will be more complex. We will have to support 2 lifecycles (one for the flag provider and one for the tracking provider). What happens if only 1 of the 2 is ready, or if only 1 of the 2 errors? Is the client usable? I think most users who really care about tracking will be using providers/vendors which support it naively; so this should be our main use-case. In this case, the author can just register their provider which does it all. The case of users who want to "patch in" tracking to a provider which doesn't support it, is, in my opinion, a secondary concern. It will be "some assembly required"; authors will need to register a hook to push evaluation data for their tracking software anyway, and also mix-in a tracking provider as I've described. Yes; this is more complex, but I think it's acceptable and simpler overall. I'm quite scared of having to somehow implement a life-cylce for 2 different types of providers. 😬

I could be wrong though, if there's a simple way to model 2 separate life-cycles (tracking and flagging) and make it clear to people what's going on, I'm OK with that. I'm just worried we're optimizing for the edge cases and not the common ones.

[lukas-reining]

Oh totally missed to answer here.
I get that @toddbaert, and while I am still worried about the ergonomics in the case one wants to use Provider A for flagging and Provider B for tracking, I think we are fine moving on as it is to lower the complexity of the first level use case!
My largest concern is that I have seen several large companies that "historically grew" and with them their ecosystem, where for both cases totally different systems have been used. But I am also with you that those are maybe not the larger group of users.

[liran2000]

As track API can be called with a bulk of events, should it be updated to accept a collection of events ?

Example scenario is described by @thomaspoignant when using cache, if considering it similar to "data collection":

We should not call the relay proxy API ... for every evaluation, but call it in bulk with a periodic threshold.

The example scenario is a bit different than the described diagram here, as the tracking call is not always called immediately after each remote evaluation, and moreover, when evaluated from cache, tracking event may be called even if no remote evaluation is done.

[toddbaert]

This was discussed a bit in our meeting and captured here (see non-goals). We decided not to handle aggregate data in our API (we summarized this as dealing with "facts not metrics").

Of course, implementations are free to cache and batch things as they want internally, and our lifecycle will support this, but the consensus so far has that the API itself should only deal with disaggregated data.

[hksekai]

When it come to tracking for experimentation, we are capturing the evaluated flag variant id as an indicator to the tracking event that some feature was triggered by the openfeature evaluation API. Within the controller, there may be multiple evaluation API interactions which will each result in a qualification or nonqualification. Regardless of how many evaluations are made, our implementation is only sending the tracking event at the end of the request. Our backend data processing pipeline does most of the heavy lifting while we leverage on a tracking publisher to write some basic metadata to the tracking event.
Some clarifications I would like to get regarding the current project goals:

1. What is the purpose of predefining a tracking payload? In our case, the user of the evaluation API may not necessarily know the specifics of what goes into tracking. They may only want to add some simple information such as "my feature was used"
2. In our case we only want to add tracking data to the tracking request context and will allow the tracking platform to take care of publishing the tracking events. Therefore, from my use case perspective the tracking provider may be coupled with a tracking publisher of our choice but it should only be responsible for adding to the tracking context and "flushing" all of the tags at the end of the transaction should be an implementation step taken by the platform.
3. Regarding the data we are looking to write to tracking, we only want to add to tracking events when a flag evaluation has returned some expected values. Based on the track API signature suggested above I only see some user input tracking data along with the context used for evaluation but no support for the actual evaluation results. Will we be able to support including the evaluation result?

[weyert]

Wouldn't you be able to use a hook to capture the evaluation results by using this new capture functionality within the hook?

[toddbaert]

@hksekai have you had a change to review my response? If you have time I'd like to meet with you to understand what your perspective and patterns.

[toddbaert]

Wouldn't you be able to use a hook to capture the evaluation results by using this new capture functionality within the hook?

@weyert , possibly, but it's quite possible that the tracking "event" happens on a totally different transaction (read: HTTP request) from an flag evaluation... perhaps a user is testing that the flag for a banner color impacts shopping cart subtotals... the "checkout" request probably has no direct flag evaluation for the banner color - these two pieces of information are only connected by the user-session (targetingKey).

[lukas-reining]

Regarding the data we are looking to write to tracking, we only want to add to tracking events when a flag evaluation has returned some expected values. Based on the track API signature suggested above I only see some user input tracking data along with the context used for evaluation but no support for the actual evaluation results. Will we be able to support including the evaluation result?

I'm reading this as a requirement to set "static" event context data that will persist on the tracking provider until it is unset. I can see how this is a reasonable thing in client / static context environments but I don't think it's that easy to accomplish in a dynamic context environment or at least I'm not seeing the use case?

@nicklasl as @toddbaert said, I think for the dynamic context this can be accomplished by using something like the transaction context propagator. There, after evaluation we could set the value to the transaction experiment context and send it.
But I might have gotten the requirement wrong :D

Wouldn't you be able to use a hook to capture the evaluation results by using this new capture functionality within the hook?

@weyert , possibly, but it's quite possible that the tracking "event" happens on a totally different transaction (read: HTTP request) from an flag evaluation... perhaps a user is testing that the flag for a banner color impacts shopping cart subtotals... the "checkout" request probably has no direct flag evaluation for the banner color - these two pieces of information are only connected by the user-session (targetingKey).

If that is the goal, I think it is out of context for OF and is part of the experimentation system right @toddbaert?
Accumulating information about evaluated flags during a request could still be done with transaction context, but the use case is a bit different then.

[toddbaert]

If that is the goal, I think it is out of context for OF and is part of the experimentation system right @toddbaert?
Accumulating information about evaluated flags during a request could still be done with transaction context, but the use case is a bit different then.

Yes, this is more toward the realm of experimentation than pure "feature flagging" but I think that we also have to accept that experimentation is associated with feature flagging. OpenFeature is not an experimentation solution, but it could be part of one, and for that I think we need this basic integration. To support this assertion, I will point out how many vendors have this basic tracking functionality (LaunchDarkly, DevCycle, Split, etc). I don't think we will ever get into sampling, analytics, or anything like that, but I think we need to expose this bare minimal API so we can enable feature-flag-related integration with tools that do offer such things.

[toddbaert]

@moredip @weyert @jonathannorris @nicklasl @kinyoklion and anyone else I missed.

I've made some updates to this proposal based on the discussions here. Please re-review. If this looks "close enough" I will move towards a specification enhancement so we can work on the nitty-gritty.

[nicklasl]

This looks good to me!

I am not totally sold on value due to its ambiguity (see example below). Its semantic is very dependent on the context of the event being tracked. However I don't want to hold this proposal back.

client.track('user-added-item-to-cart', {
  value: 49.99, // does value corresponds to the cart subtotal or to item cost?
  itemSku: 123456 
});

On this question:

Do we need to differentiate "event details" from evaluation context? Should we add another param for this?

We've had this discussion internally to some extent and I think the current API in this proposal (keep event details separate from eval context) would provide most flexibility for provider authors to adhere to their underlying systems. 💯

[toddbaert]

client.track('user-added-item-to-cart', {
  value: 49.99, // does value corresponds to the cart subtotal or to item cost?
  itemSku: 123456 
});

@nicklasl

This field can be used for any numeric value. My thinking is because so many vendors have a special numeric "value" field of some sort, we may need to identify a well-known key to map from (this is similar to how we use targetingKey as a well-known field to map into id/userId/etc).

Split, LaunchDarkly, DevCycle, as well as Segment's event spec all have similar fields.

Of course, we could omit this from the spec. That would mean that every provider would have to denote their own mapping from some field in the loosely-typed OpenFeature tracking event details, to their special field. The impact would be less coherence acress providers, since they all might map this prop differently, but of course the spec would be simpler which is its own benefit.

[weyert]

I think having the value-property optional is good I think. This would mean we have two semantic property names targetingKey and value. Even for providers that really have this concept of a value which still benefit for it in a similar providers that do support it. The user would only need some manual effort to process it.

I think this define would also cover the use cases of pageView that is being tracked an event with a particular name, e.g. $pageView etc. The only thing we could consider after this first iteration, if we need or want and how we could support group() and identify(). If we don't need dedicated support for, maybe some advisory how to implement in the provider would be a good idea.

[kinyoklion]

Looks good to me.

[toddbaert]

I've opened a PR in the spec here: open-feature/spec#268