Fix data race caused by sharing sd.Event between observers

[toffaletti]

sd/internal/instance/cache.go causes sd.Event.Instances to be shared between multiple observers:

https://github.com/go-kit/kit/blob/master/sd/internal/instance/cache.go#L71

func (r registry) broadcast(event sd.Event) {
	for c := range r {
		c <- event
	}
}

but observers frequently directly modify the sd.Event.Instances they receive, for example sd/endpoint_cache.go contains this:

https://github.com/go-kit/kit/blob/master/sd/endpoint_cache.go#L54

...
	// Happy path.
	if event.Err == nil {
		c.updateCache(event.Instances)
...
func (c *endpointCache) updateCache(instances []string) {
	// Deterministic order (for later).
	sort.Strings(instances)

The fix is to deep copy sd.Event every time it is shared. This requirement should probably be called out in the documentation as well if this is the expected behavior.

Test output before the fix:

WARNING: DATA RACE
Read at 0x00c000126350 by goroutine 13:
  sort.(*StringSlice).Less()
      /usr/local/go/src/sort/sort.go:298 +0x92
  sort.doPivot()
      /usr/local/go/src/sort/sort.go:117 +0x564
  sort.quickSort()
      /usr/local/go/src/sort/sort.go:190 +0xa7
  sort.Sort()
      /usr/local/go/src/sort/sort.go:218 +0x8a
  github.com/go-kit/kit/sd.(*endpointCache).updateCache()
      /usr/local/go/src/sort/sort.go:314 +0x88
  github.com/go-kit/kit/sd.(*endpointCache).Update()
      /Users/jason/go/src/github.com/go-kit/kit/sd/endpoint_cache.go:54 +0x3f5
  github.com/go-kit/kit/sd.(*DefaultEndpointer).receive()
      /Users/jason/go/src/github.com/go-kit/kit/sd/endpointer.go:77 +0x132

Previous write at 0x00c000126350 by goroutine 12:
  sort.(*StringSlice).Swap()
      /usr/local/go/src/sort/sort.go:299 +0x144
  sort.medianOfThree()
      /usr/local/go/src/sort/sort.go:77 +0x14b
  sort.doPivot()
      /usr/local/go/src/sort/sort.go:105 +0x9b
  sort.quickSort()
      /usr/local/go/src/sort/sort.go:190 +0xa7
  sort.quickSort()
      /usr/local/go/src/sort/sort.go:197 +0x140
  sort.Sort()
      /usr/local/go/src/sort/sort.go:218 +0x8a
  github.com/go-kit/kit/sd.(*endpointCache).updateCache()
      /usr/local/go/src/sort/sort.go:314 +0x88
  github.com/go-kit/kit/sd.(*endpointCache).Update()
      /Users/jason/go/src/github.com/go-kit/kit/sd/endpoint_cache.go:54 +0x3f5
  github.com/go-kit/kit/sd.(*DefaultEndpointer).receive()
      /Users/jason/go/src/github.com/go-kit/kit/sd/endpointer.go:77 +0x132

Goroutine 13 (running) created at:
  github.com/go-kit/kit/sd.NewEndpointer()
      /Users/jason/go/src/github.com/go-kit/kit/sd/endpointer.go:38 +0x33c
  github.com/go-kit/kit/sd/internal/instance.TestDataRace()
      /Users/jason/go/src/github.com/go-kit/kit/sd/internal/instance/cache_test.go:109 +0x44c
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:865 +0x163

Goroutine 12 (running) created at:
  github.com/go-kit/kit/sd.NewEndpointer()
      /Users/jason/go/src/github.com/go-kit/kit/sd/endpointer.go:38 +0x33c
  github.com/go-kit/kit/sd/internal/instance.TestDataRace()
      /Users/jason/go/src/github.com/go-kit/kit/sd/internal/instance/cache_test.go:108 +0x3fc
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:865 +0x163

[peterbourgon]

Can you rebase this on master?

[toffaletti]

@peterbourgon done

[peterbourgon]

Vielen Dank!