chaos testing + memberlist clustering changes + govendor -> dep

[Dieterbe]

needs a bit more work, mostly sharing for anyone who's curious what i'm up to.
see README.md and chaos_test.go for the highlevel overview

[Dieterbe]

observed problems so far:

1. often series will have multiple NaN's at the beginning or end.

• data lag? -> added lag monitoring
• for now ignoring few points at begin and end, so we can focus on clustering issues where series are completely incorrect

2. timeouts that i have to figure out further:

=== RUN   TestClusterStartup
--- PASS: TestClusterStartup (24.94s)
=== RUN   TestClusterBaseIngestWorkload
--- PASS: TestClusterBaseIngestWorkload (14.40s)
=== RUN   TestQueryWorkload
--- PASS: TestQueryWorkload (60.01s)
=== RUN   TestIsolateOneInstance
--- FAIL: TestIsolateOneInstance (60.00s)
        chaos_test.go:143: expected mt4 to return either correct or erroring responses. got (chaos.checkResults) {
                 Mutex: (sync.Mutex) {
                  state: (int32) 0,
                  sema: (uint32) 0
                 },
                 valid: ([]int) (len=2 cap=2) {
                  (int) 664, // correct responses
                  (int) 532 // 503 responses
                 },
                 empty: (int) 0,
                 timeout: (int) 4,
                 other: (int) 0,
                 firstOther: (*chaos.response)(<nil>)
                }

test suite detected 4 timeouts, in mt4 logs i see 2 requests that would have timed out the client.

$ docker logs dockerchaos_metrictank4_1 2>&1 | grep -c 'Completed /render.*503 Service Unavailable in 5.[0-9]*s'
532
$ docker logs dockerchaos_metrictank4_1 2>&1 | grep Completed | rv 'µs' | rv 'ms' | rv 'Completed /render.*503 Service Unavailable in 5\.[0-9]*s' | rv '200 OK.*[0-4]\.[0-9]*s'
[Macaron] 2017-11-23 00:12:43: Completed /render?target=sum(some.id.of.a.metric.*)&format=json&from=-15s 200 OK in 5.017100176s
[Macaron] 2017-11-23 00:13:05: Completed /render?target=sum(some.id.of.a.metric.*)&format=json&from=-15s 503 Service Unavailable in 53.436061928s
[Macaron] 2017-11-23 00:13:05: Completed /render?target=sum(some.id.of.a.metric.*)&format=json&from=-15s 503 Service Unavailable in 53.186765272s

maybe the completed lines were not logged? but:

~/g/s/g/g/m/chaos ❯❯❯ docker logs dockerchaos_metrictank4_1 2>&1 | grep -c 'Started GET /render'
2200
~/g/s/g/g/m/chaos ❯❯❯ docker logs dockerchaos_metrictank4_1 2>&1 | grep -c 'Completed /render'
2200

[Dieterbe]

one thing I've just learned is with current approach where we cut off all traffic from mt4 to the others, the others will remove mt4, and mt4 keeps getting messages from the others, but refutes them, and while it fails instances after ping timeouts, it doesn't seem to remove them from its list. so after the partition recovers, mt4 just talks to them again and the cluster is healed.
I will need to update this to make the failure work both ways

[Dieterbe]

I would like to merge this PR, mainly because of these particular generally useful commits:
0bb76bc
c80a8ed
7955734
7dd7c20

Note :

• the vendor dir is currently a bit of a mess and scripts/vendor_health.sh complains and breaks the build. I will fix before merging (obviously)
• 2 commits have DO NOT MERGE in the message. these are only useful for chaos diagnosis and i'll remove those out of the branch that will ultimately be merged

If you're interested in running the chaos testing stuff, i've been doing it by running this command in the metrictank repo root:

(cd docker/docker-chaos && docker-compose down) && (cd chaos && go test -v -timeout 0)

(in my workflow I combine the output of the go tests with docker container logs and the included grafana dashboards in http://localhost:3000 )

[woodsaj]

You can get the traceID with

span.Context().(jaeger.SpanContext).TraceId().String()

this saves you having to encode the spanId to a string, then split the traceId out from the span string.

[Dieterbe]

how does this look now? @woodsaj @replay . mergey mergey?

[Dieterbe]

@DanCech mentioned another issue: bind-address is needlessly validated when it's not set, or something like that

[DanCech]

This is still being tested in every case though it's only used for manual, we should either only test it for manual config or use it for every config type

[Dieterbe]

fixed in b49c5e8