Fix bench #386

Dieterbe · 2016-11-14T17:28:02Z

@woodsaj can we merge this before the http refactor? (and then rebase httpRefactor). currently in master and httpRefactor the benchmarks are broken. i'ld like them to work before and after httpRefactor, so we can compare easily

no idea why we never needed this in the past

they take 10 minutes and they are not all accurate. we can make this better as we go along. it sucks when benchmarks break when we don't test them also, this can be the first step towards tracking performance numbers over time. maybe.. the cloud probably doesn't deliver consistent enough performance, but we can see.

woodsaj · 2016-11-14T20:51:57Z

mdata/aggmetric_test.go

@@ -174,6 +175,10 @@ func BenchmarkAggMetrics1000Metrics1Day(b *testing.B) {
 	}

 	metrics := NewAggMetrics(dnstore, chunkSpan, numChunks, chunkMaxStale, metricMaxStale, ttl, 0, aggSettings)
+	go func() {
+		for range chunk.TotalPoints {


i really dont like that we need to do this.

yeah me neither actually. now that I think about it, we could just replace the channel based approach with atomic.UpdateUint64 or something. then whether we read out the values or not doesn't really matter. the upcoming met library also uses this approach internally, but i'ld like benchmarks to be functional before then, so i'll patch this up tomorrow I think

woodsaj · 2016-11-14T20:53:39Z

circle.yml

@@ -11,6 +11,7 @@ test:
    - cd $IMPORTPATH && go test -v -race $(go list ./... | grep -v /vendor/)
    - cd $IMPORTPATH && go vet $(go list ./... | grep -v /vendor/)
    - cd $IMPORTPATH && scripts/end2end_test.sh
+    - cd $IMPORTPATH && go test -run='^$' -bench=. $(go list ./... | grep -v /vendor/)


end2end_test tears down the docker containers when it finishes so all of the benchmark tests that have an external dependency will fail

don't those benchmarks fail anyway? do you mean the cassandra-idx and elasticsearch-idx working? seems like those require extra work (e.g. spinning up those services) to get working in the first place so i never really ran them.

if you try and run the benchmarks in circleci and they fail, the circleci job will fail.
eg. https://circleci.com/gh/raintank/metrictank/1166?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

also, this way we're not forced to read out the value note: i think some of the AggMetrics benchmarks have bugs making them misleading, that's something to look into later. but at least all times have decreased. BEFORE: BenchmarkProcess-8 2000000 717 ns/op PASS ok github.com/raintank/metrictank/input 2.687s BenchmarkAggMetrics1000Metrics1Day-8 1 4787832311 ns/op BenchmarkAggMetrics1kSeries2Chunks1kQueueSize-8 2000000000 0.04 ns/op BenchmarkAggMetrics10kSeries2Chunks10kQueueSize-8 1 1210286443 ns/op BenchmarkAggMetrics100kSeries2Chunks100kQueueSize-8 1 13937332075 ns/op PASS ok github.com/raintank/metrictank/mdata 20.655s AFTER: BenchmarkProcess-8 3000000 500 ns/op PASS ok github.com/raintank/metrictank/input 2.715s BenchmarkAggMetrics1000Metrics1Day-8 1 3977004221 ns/op BenchmarkAggMetrics1kSeries2Chunks1kQueueSize-8 1000000000 0.06 ns/op BenchmarkAggMetrics10kSeries2Chunks10kQueueSize-8 2000000000 0.39 ns/op BenchmarkAggMetrics100kSeries2Chunks100kQueueSize-8 1 9796636066 ns/op PASS ok github.com/raintank/metrictank/mdata 42.775s

while the go test command exit(0)'s, i guess CI parses the output and looks for failures. So we need to print the skips a bit more gently

the benchmarks themselves run the same way, so all measurements stay basically the same. However since the setup is faster (filling in a random array is about 30ms, whereas copying it for each b.N run is about 3ms) we need about 5 minutes less to execute all Consolidate benchmarks BEFORE: ~/g/s/g/r/metrictank ❯❯❯ go test -v -run='^$' -bench=BenchmarkConsolidate BenchmarkConsolidateAvgRand1M_1-8 200 6458335 ns/op 1858.07 MB/s BenchmarkConsolidateAvgRandWithNulls1M_1-8 200 6514825 ns/op 1841.95 MB/s BenchmarkConsolidateAvgRand1M_2-8 500 3956046 ns/op 3033.33 MB/s BenchmarkConsolidateAvgRandWithNulls1M_2-8 500 3609871 ns/op 3324.22 MB/s BenchmarkConsolidateAvgRand1M_25-8 1000 1632229 ns/op 7351.91 MB/s BenchmarkConsolidateAvgRandWithNulls1M_25-8 1000 1517594 ns/op 7907.26 MB/s BenchmarkConsolidateAvgRand1M_100-8 1000 1692647 ns/op 7089.49 MB/s BenchmarkConsolidateAvgRandWithNulls1M_100-8 1000 1563874 ns/op 7673.26 MB/s BenchmarkConsolidateMinRand1M_1-8 200 6359293 ns/op 1887.00 MB/s BenchmarkConsolidateMinRandWithNulls1M_1-8 200 6425652 ns/op 1867.52 MB/s BenchmarkConsolidateMinRand1M_2-8 200 6549782 ns/op 1832.12 MB/s BenchmarkConsolidateMinRandWithNulls1M_2-8 500 3730302 ns/op 3216.90 MB/s BenchmarkConsolidateMinRand1M_25-8 300 4243373 ns/op 2827.94 MB/s BenchmarkConsolidateMinRandWithNulls1M_25-8 500 2879560 ns/op 4167.31 MB/s BenchmarkConsolidateMinRand1M_100-8 500 3695720 ns/op 3247.00 MB/s BenchmarkConsolidateMinRandWithNulls1M_100-8 500 2559005 ns/op 4689.33 MB/s BenchmarkConsolidateMaxRand1M_1-8 200 6354204 ns/op 1888.52 MB/s BenchmarkConsolidateMaxRandWithNulls1M_1-8 200 6456749 ns/op 1858.52 MB/s BenchmarkConsolidateMaxRand1M_2-8 200 6578336 ns/op 1824.17 MB/s BenchmarkConsolidateMaxRandWithNulls1M_2-8 500 3644491 ns/op 3292.64 MB/s BenchmarkConsolidateMaxRand1M_25-8 300 4236071 ns/op 2832.82 MB/s BenchmarkConsolidateMaxRandWithNulls1M_25-8 500 2878812 ns/op 4168.39 MB/s BenchmarkConsolidateMaxRand1M_100-8 500 3667273 ns/op 3272.19 MB/s BenchmarkConsolidateMaxRandWithNulls1M_100-8 500 2543933 ns/op 4717.11 MB/s BenchmarkConsolidateSumRand1M_1-8 300 5815214 ns/op 2063.55 MB/s BenchmarkConsolidateSumRandWithNulls1M_1-8 300 5700734 ns/op 2104.99 MB/s BenchmarkConsolidateSumRand1M_2-8 500 3445351 ns/op 3482.96 MB/s BenchmarkConsolidateSumRandWithNulls1M_2-8 500 3377917 ns/op 3552.49 MB/s BenchmarkConsolidateSumRand1M_25-8 1000 1604821 ns/op 7477.48 MB/s BenchmarkConsolidateSumRandWithNulls1M_25-8 1000 1398884 ns/op 8578.27 MB/s BenchmarkConsolidateSumRand1M_100-8 1000 1542207 ns/op 7781.06 MB/s BenchmarkConsolidateSumRandWithNulls1M_100-8 1000 1502677 ns/op 7985.75 MB/s PASS ok github.com/raintank/metrictank 454.605s AFTER: BenchmarkConsolidateAvgRand1M_1-8 200 6518401 ns/op 1840.94 MB/s BenchmarkConsolidateAvgRandWithNulls1M_1-8 200 6410432 ns/op 1871.95 MB/s BenchmarkConsolidateAvgRand1M_2-8 500 3789415 ns/op 3166.72 MB/s BenchmarkConsolidateAvgRandWithNulls1M_2-8 500 3642418 ns/op 3294.52 MB/s BenchmarkConsolidateAvgRand1M_25-8 1000 1617431 ns/op 7419.18 MB/s BenchmarkConsolidateAvgRandWithNulls1M_25-8 1000 1425779 ns/op 8416.46 MB/s BenchmarkConsolidateAvgRand1M_100-8 1000 1532145 ns/op 7832.16 MB/s BenchmarkConsolidateAvgRandWithNulls1M_100-8 1000 1437692 ns/op 8346.72 MB/s BenchmarkConsolidateMinRand1M_1-8 200 6415705 ns/op 1870.41 MB/s BenchmarkConsolidateMinRandWithNulls1M_1-8 200 6429679 ns/op 1866.35 MB/s BenchmarkConsolidateMinRand1M_2-8 200 6520166 ns/op 1840.45 MB/s BenchmarkConsolidateMinRandWithNulls1M_2-8 500 3696389 ns/op 3246.41 MB/s BenchmarkConsolidateMinRand1M_25-8 300 4234772 ns/op 2833.69 MB/s BenchmarkConsolidateMinRandWithNulls1M_25-8 500 2913556 ns/op 4118.68 MB/s BenchmarkConsolidateMinRand1M_100-8 500 3704958 ns/op 3238.91 MB/s BenchmarkConsolidateMinRandWithNulls1M_100-8 500 2549002 ns/op 4707.73 MB/s BenchmarkConsolidateMaxRand1M_1-8 200 6359752 ns/op 1886.87 MB/s BenchmarkConsolidateMaxRandWithNulls1M_1-8 200 6387717 ns/op 1878.61 MB/s BenchmarkConsolidateMaxRand1M_2-8 200 6555183 ns/op 1830.61 MB/s BenchmarkConsolidateMaxRandWithNulls1M_2-8 500 3678355 ns/op 3262.33 MB/s BenchmarkConsolidateMaxRand1M_25-8 300 4286132 ns/op 2799.73 MB/s BenchmarkConsolidateMaxRandWithNulls1M_25-8 500 2902407 ns/op 4134.50 MB/s BenchmarkConsolidateMaxRand1M_100-8 500 3710203 ns/op 3234.33 MB/s BenchmarkConsolidateMaxRandWithNulls1M_100-8 500 2549478 ns/op 4706.85 MB/s BenchmarkConsolidateSumRand1M_1-8 300 5743601 ns/op 2089.28 MB/s BenchmarkConsolidateSumRandWithNulls1M_1-8 300 5763531 ns/op 2082.06 MB/s BenchmarkConsolidateSumRand1M_2-8 500 3457185 ns/op 3471.03 MB/s BenchmarkConsolidateSumRandWithNulls1M_2-8 500 3405377 ns/op 3523.84 MB/s BenchmarkConsolidateSumRand1M_25-8 1000 1542074 ns/op 7781.73 MB/s BenchmarkConsolidateSumRandWithNulls1M_25-8 1000 1309428 ns/op 9164.32 MB/s BenchmarkConsolidateSumRand1M_100-8 1000 1485214 ns/op 8079.65 MB/s BenchmarkConsolidateSumRandWithNulls1M_100-8 1000 1504304 ns/op 7977.12 MB/s PASS ok github.com/raintank/metrictank 130.309s

Dieterbe added 3 commits November 14, 2016 19:00

prevent benchmarks from being blocked on Chunk writes

7289610

no idea why we never needed this in the past

skip benches that need ES/cass deps, if their dep is not up

945164a

Dieterbe assigned woodsaj Nov 14, 2016

woodsaj reviewed Nov 14, 2016

View reviewed changes

Dieterbe added 2 commits November 15, 2016 09:46

make circleci not fail the CI when benchmarks skip

e4a4c71

while the go test command exit(0)'s, i guess CI parses the output and looks for failures. So we need to print the skips a bit more gently

Dieterbe force-pushed the fix-bench branch from 6cae778 to e4a4c71 Compare November 15, 2016 08:29

Dieterbe merged commit 82b8f4d into master Nov 15, 2016

Dieterbe deleted the fix-bench branch December 15, 2017 22:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix bench #386

Fix bench #386

Dieterbe commented Nov 14, 2016 •

edited

Loading

woodsaj Nov 14, 2016

Dieterbe Nov 14, 2016

woodsaj Nov 14, 2016

Dieterbe Nov 14, 2016

woodsaj Nov 15, 2016

Fix bench #386

Fix bench #386

Conversation

Dieterbe commented Nov 14, 2016 • edited Loading

woodsaj Nov 14, 2016

Choose a reason for hiding this comment

Dieterbe Nov 14, 2016

Choose a reason for hiding this comment

woodsaj Nov 14, 2016

Choose a reason for hiding this comment

Dieterbe Nov 14, 2016

Choose a reason for hiding this comment

woodsaj Nov 15, 2016

Choose a reason for hiding this comment

Dieterbe commented Nov 14, 2016 •

edited

Loading