RFC: Trace Infrastructure

[schmichael]

I've been toying with verifying Metafora's correctness via etcd (in part using https://github.com/lytics/etcdlog), but in order to analyze the correctness of Metafora's behavior I need to be able to compare etcd's view-of-the-world with Metafora's.

To this end I'd like to add machine readable Trace events similar to the responses returned from an etcd watch. This should allow us to construct a Call-Me-Maybe-esque history to verify such constraints as:

• Tasks are run on exactly one node at a time
  • During normal operation
  • During partitioning from broker, tasks are released before claim expires
• Tasks are started in a timely fashion (any latency over 1s from task submission to claim seems fishy)

Types

type Trace struct {
  Timestamp time.Time
  Host string
  Namespace string
  TaskID string
  Description string
  State statemachine.State `json:"state,omitempty"`
  Progress interface{} //TODO what type?
}

type Tracer interface {
  // return an error for testing/debugging, but don't expect people to check it.
  // nothing for callers to do about failures
  Emit(t Trace) error
}