Graphitey consolidation step and 3 (redo) #570

note that whisper files and structures are untouched. We only use this extended format for metrictank. also: if failure to parse, return error

ChunkSpan, NumChunks and Ready fields are now supported, also optionally specified via the config file

so we can efficiently reference a schema or aggregation definition also make WhisperAggregation.Match public

because we want to work with them in metrictank

1) use storage-schemas.conf and storage-aggregation.conf to configure numchunks, chunkspans, ttls and aggregations. For carbon also to find out the raw interval. For other input plugins we only honor interval specified in the data Note: - persist message now need to mention which metric names, in case metrics need to be created on the other end. 2) support configurable retention schemes based on patterns, like graphite. adjust alignRequests accordingly 3) support configurable aggregation bands to control which rollups get created per metric pattern. 4) add back 'last' roll-up aggregation and runtime consolidation It was removed in #142 / 2be3546 based on a discussion in https://github.com/raintank/strategy/issues/11 on the premise that storing this extra series for all our metrics wasn't worth it and it's not queryable via consolidateBy which only does sum, avg, min and max. However: 1) now aggregations are configurable: One only enables specific aggregation bands on an as-needed basis. 2) graphite supports last rollups, so we must too. 3) for certain types of data (e.g. counters) it's simply the best (better than all the rest) approach 4) storage level aggregation is not as tied to consolidateBy() as we originally made it out to be, so that point doesn't really apply anymore. 5) we also restore the 'last' runtime consolidation code, even though we will no longer do runtime consolidation in MT for now but the upcoming normalization uses the same code. As we should use 'last' for normalization if rollups was 'last'. (see next commit for more info)

"runtime consolidation" used to be controlled by consolidateBy and affect consolidation after api processing by (and in) graphite, as well as before (when normalizing data before processing). Now these two are distinct: - only the post-function processing should be affected by consolidateBy, and should be referred to as "runtime consolidation". Since this code is currently in graphite, metrictank shouldn't really do anything. In particular we still parse out consolidateBy(..., "func") since that's what graphite-metrictank sends, but we ignore it - the only runtime consolidation metrictank currently does, is normalizing archives (making sure they are in the same resolution) before feeding them into the api processing pipeline. To avoid confusion this is now called normalizing, akin to `normalize` in graphite's functions.py This is an extension of the consolidation mechanism used to create the rollup archive. This used to be controlled by consolidateBy, but not anymore. Now: * the archive being read from is whathever is the primary (first) aggregationMethod defined in storage-aggregation.conf . * consolidation method for normalization is the same (always). Both are controlled via the consolidation property of Req. Later we can extend this to choose the method via queries, but it'll have to use a mechanism separate of consolidateBy.

note: a bunch of the handler/processing tests from the input package are removed. They were testing a whole bunch of things at once (index, aggmetrics, handler, input plugin, and libraries they depend on) unit tests should test very specific units of code, but in the case of the input package, the code they would test would become so trivial and obviously correct, because the input plugins barely do anything themselves. also: * make the tests a bit more descriptive * make testAlignRequests easier to understand by splitting it up

we can simplify code if we just work with the data normally

so that MT is ready to go via docker or packages (they already had storage-schemas.conf)

/etc/raintank was silly to begin with. Config directories should be named after the software, not after the company. Especially now we're getting rid of the raintank name

useful for debugging

@replay

before this, all regex matching dominated the cpu profile. With this, cpu usage reduced by easily 5x Though we still have: flat cumulative 70ms 0.86% 69.79% 770ms 9.49% github.com/raintank/metrictank/mdata/matchcache.(*Cache).Get due to the map locking We could further optimize this, probably, by changing the idx.AddOrUpdate signature to returning SchemaI and AggI, instead of requiring it as input as @replay suggested. This way we only have to match if it wasn't in the index already. However this requires more intensive changes to the index than I'm comfortable with right now (DefById only has the metricdef, not the properties, we could add them but then we need to adjust how we work with DefById everywhere and do we still need to store the properties in the tree, etc) I rather re-address this when the need is clearer and we have time to give this the attention it deserves.

on my laptop: MASTER: BenchmarkProcess-8 2000000 718 ns/op PASS ok github.com/raintank/metrictank/input 2.796s HEAD: go test -run='^$' -bench=. BenchmarkProcessUniqueMetrics-8 1000000 2388 ns/op BenchmarkProcessSameMetric-8 2000000 885 ns/op PASS ok github.com/raintank/metrictank/input 6.081s So we're a bit slower but carbon input should be a lot faster (for which we don't have benchmarks) since it used to do regex matching all the time

much cleaner code if I may say so. also: consistently apply a simpler "add default to catch all remaining metrics" default mechanism.

This reverts commit 3bae47f. They consume about 1% cpu (flat and cumul) under heavy ingest load and they basically just confirm it works exactly like it should. But the commit is here in case someone ever wants to re-apply it.

prints how many chunks per series in each table, with TTL's optionally filter down by metric prefix

users who have their index set to memory idx had a bug where the partition was never initially set properly, nor changed properly on updates.

* remove matchcache, just use index instead * s/schemaI/schemaId/ and s/aggI/aggId/ * track schemaId and aggId per metricdef, not per node since interval of a node can change over time and we want to support agg/schema settings based on interval as well. * make AddOrUpdate return the archive added, so you get access to schemaId and aggId "for free". also: * remove AddOrUpdateDef because it's trivial to use AddOrUpdate instead. there's no need for AddOrUpdateDef's "update even if exists" behavior. Except for updating LastUpdate which we still do.

this is marginal performance improvement (tested with 50kHz fakemetrics workload) before, 87% of cpu is used by Carbon.handle which spends 3.88/13.14s=29.5% of its time getting the interval after, 79.22% and 3.13/13.04=24% BEFORE: (pprof) top30 -cum 5.51s of 13.14s total (41.93%) Dropped 144 nodes (cum <= 0.07s) Showing top 30 nodes out of 150 (cum >= 0.51s) flat flat% sum% cum cum% 0 0% 0% 12.81s 97.49% runtime.goexit 0.08s 0.61% 1.14% 3.88s 29.53% github.com/raintank/metrictank/conf.Schemas.Match 0 0% 1.14% 3.88s 29.53% github.com/raintank/metrictank/mdata.MatchSchema 0.02s 0.15% 1.29% 3.79s 28.84% regexp.(*Regexp).MatchString 0.06s 0.46% 1.75% 3.77s 28.69% regexp.(*Regexp).doMatch 0.03s 0.23% 1.98% 3.71s 28.23% regexp.(*Regexp).doExecute 0.19s 1.45% 3.42% 3.36s 25.57% regexp.(*machine).backtrack 0.03s 0.23% 3.65% 2.77s 21.08% github.com/raintank/metrictank/input.(*DefaultHandler).Process 1.56s 11.87% 15.53% 2.77s 21.08% regexp.(*machine).tryBacktrack 0.01s 0.076% 15.60% 2.73s 20.78% github.com/raintank/metrictank/input.DefaultHandler.Process 0.02s 0.15% 15.75% 2.36s 17.96% github.com/raintank/metrictank/vendor/github.com/metrics20/go-metrics20/carbon20.ValidatePacket 0.01s 0.076% 15.83% 1.89s 14.38% github.com/raintank/metrictank/vendor/gopkg.in/raintank/schema%2ev1.(*MetricData).SetId 0 0% 15.83% 1.86s 14.16% bytes.Fields 0.77s 5.86% 21.69% 1.86s 14.16% bytes.FieldsFunc 0.01s 0.076% 21.77% 1.50s 11.42% runtime.systemstack 0.57s 4.34% 26.10% 1.17s 8.90% runtime.scanobject 0 0% 26.10% 1.13s 8.60% runtime.gcBgMarkWorker 0 0% 26.10% 1.12s 8.52% runtime.gcBgMarkWorker.func2 0.02s 0.15% 26.26% 1.12s 8.52% runtime.gcDrain 0.50s 3.81% 30.06% 1.06s 8.07% runtime.mallocgc 0.20s 1.52% 31.58% 0.97s 7.38% github.com/raintank/metrictank/mdata.(*AggMetric).Add 0.01s 0.076% 31.66% 0.89s 6.77% github.com/raintank/metrictank/idx/cassandra.(*CasIdx).AddOrUpdate 0 0% 31.66% 0.84s 6.39% fmt.Sprintf 0.04s 0.3% 31.96% 0.70s 5.33% fmt.(*pp).doPrintf 0.06s 0.46% 32.42% 0.67s 5.10% runtime.newobject 0.03s 0.23% 32.65% 0.66s 5.02% fmt.(*pp).printArg 0.59s 4.49% 37.14% 0.59s 4.49% regexp.(*bitState).push 0.12s 0.91% 38.05% 0.54s 4.11% github.com/raintank/metrictank/idx/memory.(*MemoryIdx).Get 0.51s 3.88% 41.93% 0.51s 3.88% unicode/utf8.DecodeRune (pprof) list Carbon.*handle Total: 13.14s ROUTINE ======================== github.com/raintank/metrictank/input/carbon.(*Carbon).handle in /home/dieter/go/src/github.com/raintank/metrictank/input/carbon/carbon.go 70ms 11.44s (flat, cum) 87.06% of Total . . 147: }() . . 148: // TODO c.SetTimeout(60e9) . . 149: r := bufio.NewReaderSize(conn, 4096) . . 150: for { . . 151: // note that we don't support lines longer than 4096B. that seems very reasonable.. 10ms 140ms 152: buf, _, err := r.ReadLine() . . 153: . . 154: if nil != err { . . 155: select { . . 156: case <-c.quit: . . 157: // we are shutting down. . . 158: break . . 159: default: . . 160: } . . 161: if io.EOF != err { . . 162: log.Error(4, "carbon-in: Recv error: %s", err.Error()) . . 163: } . . 164: break . . 165: } . . 166: . . 167: // no validation for m2.0 to provide a grace period in adopting new clients . 2.36s 168: key, val, ts, err := carbon20.ValidatePacket(buf, carbon20.MediumLegacy, carbon20.NoneM20) . . 169: if err != nil { . . 170: metricsDecodeErr.Inc() . . 171: log.Error(4, "carbon-in: invalid metric: %s", err.Error()) . . 172: continue . . 173: } 10ms 80ms 174: name := string(key) . 3.88s 175: _, s := mdata.MatchSchema(name) // note: also called in idx.AddOrUpdate via metrictank DefaultHandler.Process. maybe can be optimized . . 176: interval := s.Retentions[0].SecondsPerPoint . . 177: md := &schema.MetricData{ . . 178: Name: name, . . 179: Metric: name, . . 180: Interval: interval, . . 181: Value: val, . . 182: Unit: "unknown", . . 183: Time: int64(ts), . . 184: Mtype: "gauge", . 20ms 185: Tags: []string{}, 10ms 150ms 186: OrgId: 1, // admin org . . 187: } . 1.89s 188: md.SetId() 30ms 140ms 189: metricsPerMessage.ValueUint32(1) 10ms 2.78s 190: c.Handler.Process(md, int32(partitionId)) . . 191: } . . 192: c.handlerWaitGroup.Done() . . 193:} AFTER: (pprof) top30 -cum 5.39s of 13.04s total (41.33%) Dropped 164 nodes (cum <= 0.07s) Showing top 30 nodes out of 164 (cum >= 0.46s) flat flat% sum% cum cum% 0 0% 0% 12.47s 95.63% runtime.goexit 0.12s 0.92% 0.92% 10.33s 79.22% github.com/raintank/metrictank/input/carbon.(*Carbon).handle 0 0% 0.92% 3.11s 23.85% github.com/raintank/metrictank/input/carbon.(*IndexIntervalGetter).GetInterval 0.03s 0.23% 1.15% 3.11s 23.85% github.com/raintank/metrictank/input/carbon.IndexIntervalGetter.GetInterval 0.50s 3.83% 4.98% 3.07s 23.54% github.com/raintank/metrictank/idx/memory.(*MemoryIdx).Find 0.02s 0.15% 5.14% 2.33s 17.87% runtime.systemstack 0 0% 5.14% 2.29s 17.56% github.com/raintank/metrictank/input.(*DefaultHandler).Process 0.04s 0.31% 5.44% 2.29s 17.56% github.com/raintank/metrictank/input.DefaultHandler.Process 0.03s 0.23% 5.67% 2.28s 17.48% github.com/raintank/metrictank/vendor/gopkg.in/raintank/schema%2ev1.(*MetricData).SetId 0.01s 0.077% 5.75% 2.08s 15.95% github.com/raintank/metrictank/vendor/github.com/metrics20/go-metrics20/carbon20.ValidatePacket 0.07s 0.54% 6.29% 1.68s 12.88% github.com/raintank/metrictank/idx/memory.(*MemoryIdx).find 0.90s 6.90% 13.19% 1.64s 12.58% runtime.scanobject 0 0% 13.19% 1.62s 12.42% runtime.gcBgMarkWorker 0.89s 6.83% 20.02% 1.62s 12.42% runtime.mallocgc 0 0% 20.02% 1.61s 12.35% runtime.gcBgMarkWorker.func2 0.02s 0.15% 20.17% 1.61s 12.35% runtime.gcDrain 0 0% 20.17% 1.57s 12.04% bytes.Fields 0.70s 5.37% 25.54% 1.57s 12.04% bytes.FieldsFunc 0.01s 0.077% 25.61% 0.88s 6.75% fmt.Sprintf 0.07s 0.54% 26.15% 0.88s 6.75% runtime.newobject 0.25s 1.92% 28.07% 0.83s 6.37% github.com/raintank/metrictank/mdata.(*AggMetric).Add 0.12s 0.92% 28.99% 0.80s 6.13% fmt.(*pp).doPrintf 0.50s 3.83% 32.82% 0.69s 5.29% runtime.mapaccess2_faststr 0.02s 0.15% 32.98% 0.68s 5.21% fmt.(*pp).printArg 0.03s 0.23% 33.21% 0.61s 4.68% github.com/raintank/metrictank/idx/cassandra.(*CasIdx).AddOrUpdate 0.05s 0.38% 33.59% 0.52s 3.99% runtime.convT2E 0.10s 0.77% 34.36% 0.51s 3.91% runtime.makeslice 0.13s 1% 35.35% 0.48s 3.68% fmt.(*pp).printValue 0.32s 2.45% 37.81% 0.47s 3.60% runtime.mapassign 0.46s 3.53% 41.33% 0.46s 3.53% unicode/utf8.DecodeRune (pprof) list Carbon.*handle Total: 13.04s ROUTINE ======================== github.com/raintank/metrictank/input/carbon.(*Carbon).handle in /home/dieter/go/src/github.com/raintank/metrictank/input/carbon/carbon.go 120ms 10.33s (flat, cum) 79.22% of Total . . 151: }() . . 152: // TODO c.SetTimeout(60e9) . . 153: r := bufio.NewReaderSize(conn, 4096) . . 154: for { . . 155: // note that we don't support lines longer than 4096B. that seems very reasonable.. 40ms 190ms 156: buf, _, err := r.ReadLine() . . 157: . . 158: if nil != err { . . 159: select { . . 160: case <-c.quit: . . 161: // we are shutting down. . . 162: break . . 163: default: . . 164: } . . 165: if io.EOF != err { . . 166: log.Error(4, "carbon-in: Recv error: %s", err.Error()) . . 167: } . . 168: break . . 169: } . . 170: . . 171: // no validation for m2.0 to provide a grace period in adopting new clients 20ms 2.10s 172: key, val, ts, err := carbon20.ValidatePacket(buf, carbon20.MediumLegacy, carbon20.NoneM20) . . 173: if err != nil { . . 174: metricsDecodeErr.Inc() . . 175: log.Error(4, "carbon-in: invalid metric: %s", err.Error()) . . 176: continue . . 177: } 10ms 80ms 178: name := string(key) . . 179: md := &schema.MetricData{ . . 180: Name: name, . . 181: Metric: name, 20ms 3.13s 182: Interval: c.intervalGetter.GetInterval(name), . . 183: Value: val, . . 184: Unit: "unknown", . . 185: Time: int64(ts), . . 186: Mtype: "gauge", . . 187: Tags: []string{}, . 70ms 188: OrgId: 1, // admin org . . 189: } . 2.28s 190: md.SetId() 30ms 190ms 191: metricsPerMessage.ValueUint32(1) . 2.29s 192: c.Handler.Process(md, int32(partitionId)) . . 193: } . . 194: c.handlerWaitGroup.Done() . . 195:}

and let carbon use it instead of the more expensive idx.Find() Now Carbon.handle is at 66% of cpu of which 1.15/12.35=9.3% is spent finding the interval (pprof) top30 -cum 3.24s of 12.35s total (26.23%) Dropped 184 nodes (cum <= 0.06s) Showing top 30 nodes out of 146 (cum >= 0.75s) flat flat% sum% cum cum% 0 0% 0% 11.12s 90.04% runtime.goexit 0.05s 0.4% 0.4% 8.22s 66.56% github.com/raintank/metrictank/input/carbon.(*Carbon).handle 0.03s 0.24% 0.65% 2.29s 18.54% github.com/raintank/metrictank/vendor/github.com/metrics20/go-metrics20/carbon20.ValidatePacket 0.01s 0.081% 0.73% 2.27s 18.38% github.com/raintank/metrictank/input.(*DefaultHandler).Process 0.04s 0.32% 1.05% 2.26s 18.30% github.com/raintank/metrictank/input.DefaultHandler.Process 0.05s 0.4% 1.46% 2s 16.19% github.com/raintank/metrictank/vendor/gopkg.in/raintank/schema%2ev1.(*MetricData).SetId 0.01s 0.081% 1.54% 1.87s 15.14% runtime.systemstack 0 0% 1.54% 1.79s 14.49% bytes.Fields 0.82s 6.64% 8.18% 1.79s 14.49% bytes.FieldsFunc 0.72s 5.83% 14.01% 1.43s 11.58% runtime.scanobject 0 0% 14.01% 1.39s 11.26% runtime.gcBgMarkWorker 0.75s 6.07% 20.08% 1.39s 11.26% runtime.mallocgc 0 0% 20.08% 1.37s 11.09% runtime.gcBgMarkWorker.func2 0 0% 20.08% 1.37s 11.09% runtime.gcDrain 0.03s 0.24% 20.32% 1.15s 9.31% github.com/raintank/metrictank/input/carbon.(*IndexIntervalGetter).GetInterval 0 0% 20.32% 1.12s 9.07% github.com/raintank/metrictank/input/carbon.IndexIntervalGetter.GetInterval 0.41s 3.32% 23.64% 1.09s 8.83% github.com/raintank/metrictank/idx/memory.(*MemoryIdx).GetPath 0.02s 0.16% 23.81% 1.07s 8.66% github.com/raintank/metrictank/mdata.(*CassandraStore).processWriteQueue 0.01s 0.081% 23.89% 0.97s 7.85% github.com/raintank/metrictank/mdata.(*CassandraStore).insertChunk 0.17s 1.38% 25.26% 0.91s 7.37% github.com/raintank/metrictank/mdata.(*AggMetric).Add 0 0% 25.26% 0.90s 7.29% github.com/raintank/metrictank/vendor/github.com/gocql/gocql.(*Query).Exec 0 0% 25.26% 0.90s 7.29% github.com/raintank/metrictank/vendor/github.com/gocql/gocql.(*Query).Iter 0 0% 25.26% 0.90s 7.29% github.com/raintank/metrictank/vendor/github.com/gocql/gocql.(*Session).executeQuery 0 0% 25.26% 0.90s 7.29% github.com/raintank/metrictank/vendor/github.com/gocql/gocql.(*queryExecutor).executeQuery 0 0% 25.26% 0.89s 7.21% runtime.mcall 0.03s 0.24% 25.51% 0.85s 6.88% runtime.schedule 0 0% 25.51% 0.80s 6.48% fmt.Sprintf 0.04s 0.32% 25.83% 0.80s 6.48% runtime.findrunnable 0.04s 0.32% 26.15% 0.80s 6.48% runtime.newobject 0.01s 0.081% 26.23% 0.75s 6.07% github.com/raintank/metrictank/vendor/github.com/gocql/gocql.(*Conn).executeQuery (pprof) list Carbon.*handle Total: 12.35s ROUTINE ======================== github.com/raintank/metrictank/input/carbon.(*Carbon).handle in /home/dieter/go/src/github.com/raintank/metrictank/input/carbon/carbon.go 50ms 8.22s (flat, cum) 66.56% of Total . . 151: }() . . 152: // TODO c.SetTimeout(60e9) . . 153: r := bufio.NewReaderSize(conn, 4096) . . 154: for { . . 155: // note that we don't support lines longer than 4096B. that seems very reasonable.. 20ms 230ms 156: buf, _, err := r.ReadLine() . . 157: . . 158: if nil != err { . . 159: select { . . 160: case <-c.quit: . . 161: // we are shutting down. . . 162: break . . 163: default: . . 164: } . . 165: if io.EOF != err { . . 166: log.Error(4, "carbon-in: Recv error: %s", err.Error()) . . 167: } . . 168: break . . 169: } . . 170: . . 171: // no validation for m2.0 to provide a grace period in adopting new clients . 2.29s 172: key, val, ts, err := carbon20.ValidatePacket(buf, carbon20.MediumLegacy, carbon20.NoneM20) . . 173: if err != nil { . . 174: metricsDecodeErr.Inc() . . 175: log.Error(4, "carbon-in: invalid metric: %s", err.Error()) . . 176: continue . . 177: } . 40ms 178: name := string(key) . . 179: md := &schema.MetricData{ . . 180: Name: name, . . 181: Metric: name, . 1.15s 182: Interval: c.intervalGetter.GetInterval(name), . . 183: Value: val, . . 184: Unit: "unknown", . . 185: Time: int64(ts), . . 186: Mtype: "gauge", 20ms 30ms 187: Tags: []string{}, . 130ms 188: OrgId: 1, // admin org . . 189: } . 2s 190: md.SetId() . 70ms 191: metricsPerMessage.ValueUint32(1) 10ms 2.28s 192: c.Handler.Process(md, int32(partitionId)) . . 193: } . . 194: c.handlerWaitGroup.Done() . . 195:} (pprof)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Graphitey consolidation step and 3 (redo) #570

Graphitey consolidation step and 3 (redo) #570

Commits on Mar 16, 2017

Commits on Mar 17, 2017