Add dynamic profiling to Linkerd Rust controllers

[risingspiral]

What problem are you trying to solve?

Enable users to dynamically profile the running application can aid significantly in debugging and diagnostics.

How should the problem be solved?

With pprof or something similar

Any alternatives you've considered?

Not at the moment.

How would users interact with this feature?

Network connection

Would you like to work on this feature?

maybe

[amit-62]

This enhancement looks interesting to me, so I would love to be a LFX mentee for this project. Please help me to get started with this issue.

[mateiidavid]

@amit-62, I'd start by looking at how this is handled for our controllers written in Go.

linkerd2/pkg/admin/admin.go

Lines 34 to 51 in ae46678

	func (h *handler) ServeHTTP(w http.ResponseWriter, req *http.Request) {
	debugPathPrefix := "/debug/pprof/"
	if h.enablePprof && strings.HasPrefix(req.URL.Path, debugPathPrefix) {
	switch req.URL.Path {
	case fmt.Sprintf("%scmdline", debugPathPrefix):
	pprof.Cmdline(w, req)
	case fmt.Sprintf("%sprofile", debugPathPrefix):
	pprof.Profile(w, req)
	case fmt.Sprintf("%strace", debugPathPrefix):
	pprof.Trace(w, req)
	case fmt.Sprintf("%ssymbol", debugPathPrefix):
	pprof.Symbol(w, req)
	default:
	pprof.Index(w, req)
	}
	return
	}
	switch req.URL.Path {

linkerd2/controller/cmd/destination/main.go

Line 34 in ae46678

enablePprof := cmd.Bool("enable-pprof", false, "Enable pprof endpoints on the admin server")

Each controller takes a CLI argument that enables pprof. How would we do this in Rust? What would the expected end-to-end flow look like? Would it similar to Go?

Also, in Go, pprof is (if I'm not wrong) exported by the http package. What would we use in Rust? Have a look at the post carefully, read through the problem statement, and come up with a thorough implementation plan. Make sure to let us know how users would interact with this feature (i.e UX) and how you would go about implementing it.

[amit-62]

Thank you @mateiidavid. I will start working on it.

[brancz]

I want to highlight a parallel effort that is on-going, which is to build binaries with DWARF debuginfos, which would allow profilers like perf and Parca to profile linkerd without any code modifications needed, and more importantly without using SIGPROF which can change program execution (in very high-performance use cases we've observed this to be an actual problem).

cc @stevej @olix0r

[stevej]

@brancz Thank you for chiming in! I caught up with oliver and he let me know that we've been putting debug_info into the proxy release since the end of december although I'm not sure this is DWARF format or not. If this doesn't work in your pipeline, we can improve it.

debug-info was included with this commit: linkerd/linkerd2-proxy@adf06c4#diff-deb9bb56fb122db0b605aa5b63f95a4665c905b18dd670e1fa6c877576a94ff1R114-R121

I took a quick look at a recent release from github: https://github.com/linkerd/linkerd2-proxy/releases/tag/release%2Fv2.189.0

LICENSE:            ASCII text
bin:                directory
linkerd2-proxy:     ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=7feb197a9e27026f754f3d3dbe56375dced0dacc, for GNU/Linux 3.2.0, stripped
linkerd2-proxy.dbg: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter *empty*, BuildID[sha1]=7feb197a9e27026f754f3d3dbe56375dced0dacc, for GNU/Linux 3.2.0, with debug_info, not stripped

[brancz]

Awesome! I’ll try it!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.