enable delta memory profiler to reduce storage cost.

[zdyj3170101136]

introduction

use memory profiling to reduce 100x storage cost.

design

Go's memory analysis is special.

In the runtime, for a call stack, there is a bucket that stores the address of the call stack, the number of memory allocations, and the number of bytes.

All buckets are traversed through mbucket to generate memory performance files, and the new bucket is inserted into the head of mbucket.

Therefore, the memory performance file is monotonically increasing.

So we can only store the different parts of the two performance files to reduce costs.

example

For a go application, a performance analysis is obtained through sampling at time a, and a performance file is also obtained through sampling at the subsequent time b.

If the sample num in the performance files a and b is the same.
Then for any sample[i] in a and b, there is the same call stack, only the values ​​are different.

If the sample num in the performance files a and b is different.
Then for sample[i] in a, sample[i + numOfSampleB - numOfSampleA] in b must be the same.

For the same sample, we do not need to store the location, but only the delta of values ​​(most of which are all 0).

We only need to store the locations contained in the samples that only appear in b.

[darccio]

Hi @zdyj3170101136. Thanks for reaching out. Please, could you provide more detail of where would you apply this suggestion? It seems related to the Go runtime, not dd-trace-go itself.

[zdyj3170101136]

你好@zdyj3170101136。感谢您的联系。请问您能否提供更多有关您将在何处应用此建议的详细信息？它似乎与 Go 运行时有关，而不是 dd-trace-go 本身。

we only need a old profile to get delta profile:

func computeDelta(oldP *pprof.Profile, p *pprof.Profile) *pprof.Profile {
	var deltaP = &pprof.Profile{}
	var newSample []*pprof.Sample
	var delta int
	delta = len(p.Sample) - len(oldP.Sample)
	for i, s := range p.Sample {
		if i >= delta {
			oldValues := oldP.Sample[i+delta].Value
			deltaValue := make([]int64, 0, len(s.Value))
			var allzero bool = true
			for i := range s.Value {
				v := s.Value[i] - oldValues[i]
				if v != 0 {
					allzero = false
				}
				deltaValue = append(deltaValue, v)
			}
			if !allzero {
				deltaP.Sample = append(deltaP.Sample, &pprof.Sample{
					Value: deltaValue,
				})
			} 
		} else {
			newSample = append(newSample, s)
			deltaP.Sample = append(deltaP.Sample, s)
		}
	}

	return deltaP
}

We can cache the old profile in dd-trace-go. And use it to get the delta profile. Only upload the delta profile to datadog.

[nsrip-dd]

@zdyj3170101136 I would be very hesitant to rely on the exact order that records come out of the memory profiler. That's an implementation detail, and we'd need to rework our implementation (again) if that detail changes. If we put any more work into improving delta profile efficiency, I think we would try to make the improvements in the Go runtime rather than this repository. See golang/go#67942.