service,terminal,cmd/dlv: automatically guessing substitute-path config

[aarzilli]

Add command, API calls and launch.json option to automatically guess
substitute-path configuration.

[aarzilli]

cc @hyangah I've been wanting to remove the legacy mode of vscode-go adapter and I was wondering if something like this would work.

How it works

Client side it runs go list --json all and uses the output to create a mapping from module paths to their directories. Server side it tries to do the same thing by examining debug_info/debug_line. It works for vendor directories, for modules in the pkg/mod directory, for -trimpath and across windows/linux connections.

What the extension would have to do

The extension, before connecting to dlv, would have to run locally dlv substitute-path-guess-helper and copy its output into the configuration that is sent to dlv with the AttachRequest. That's all. If we want to extend the guessing algorithm to cover more situations we can then do it without
involving the extension itself.

Thoughts? Would this work?

[hyangah]

Thank you so much!

A couple of questions:

1. dlv substitute-path-guess-helper:

The extension, before connecting to dlv, would have to run locally dlv substitute-path-guess-helper ...

In case of the remote debugging, the debugged target executable is available only on the remote host. Shouldn't dlv substitute-path-guess-helper be able to connect to the headless server running remotely?

2. In the current snapshot, you added ClientMod2Dir. Does it mean that the DAP client can send this mapping (that's what client can compute locally using go list) instead of dlv substitute-path-guess-helper? I think that will work.

3. One minor detail is handling of standard libraries. Should the ClientMod2Dir have a special entry for the standard library module? Or should we specify the full package path for stdlib packages? (e.g. "runtime": "/mymachine/goroot/src/runtime")?

[aarzilli]

Thank you so much!

A couple of questions:

1. dlv substitute-path-guess-helper:

The extension, before connecting to dlv, would have to run locally dlv substitute-path-guess-helper ...

In case of the remote debugging, the debugged target executable is available only on the remote host. Shouldn't dlv substitute-path-guess-helper be able to connect to the headless server running remotely?

The idea is that dlv substitute-path-guess-helper collects the informations it needs from the local source, which is then sent to the server within the launch configuration. Then the rest of the work is done by the server, which does have access to the binary.

2. In the current snapshot, you added `ClientMod2Dir`. Does it mean that the DAP client can send this mapping (that's what client can compute locally using `go list`) instead of `dlv substitute-path-guess-helper`?  I think that will work.

The idea is that dlv substitute-path-guess-helper can be called to produce the contents of ClientMod2Dir, instead of having to replicate that part of the logic in javascript.

3. One minor detail is handling of standard libraries. Should the `ClientMod2Dir` have a special entry for the standard library module? Or should we specify the full package path for stdlib packages? (e.g. "runtime": "/mymachine/goroot/src/runtime")?

I initially did this and then backed off of it. There is no guarantee that the standard library that was used to build the executable is the same one that's available locally, so we shouldn't blindly map it. We could be smarter about it, but in practice it is rare that the user will step into the standard library (I think) so it's probably fine to leave it unmapped.

[hyangah]

Thanks. I will give it a try and test it.

3. One minor detail is handling of standard libraries. Should the `ClientMod2Dir` have a special entry for the standard library module? Or should we specify the full package path for stdlib packages? (e.g. "runtime": "/mymachine/goroot/src/runtime")?

I initially did this and then backed off of it. There is no guarantee that the standard library that was used to build the executable is the same one that's available locally, so we shouldn't blindly map it. We could be smarter about it, but in practice it is rare that the user will step into the standard library (I think) so it's probably fine to leave it unmapped.

I'd say there is also no guarantee that the user code used to build the executable is not exactly the same as the one available locally strictly speaking. I hope users of remote debugging to ensure both systems use the identical versions. (Or can we have a warning based on the Go build info?)
In DAP, the path translation occurs not only in the breakpoint setting, but many other message types that can use Source type such as stack trace, output event, ... And VS Code UI makes it really easy to navigate up and down over the stack. (I will test and report the impact of missing translation)

[aarzilli]

I'd say there is also no guarantee that the user code used to build the executable is not exactly the same as the one available locally strictly speaking.

Yes, we have a warning for this. Vscode-go doesn't but that's a UI problem.

(I will test and report the impact of missing translation)

Sure, let us know.

[hyangah]

I think we need to handle stdlib locations unfortunately.

• Often the stack trace includes locations from stdlib packages. Navigating through the stack causes the editor to attempt to open non-existing file paths.
• When requesting to pause the debugee, the editor subsequently attempts to open the location of the top stack frame. That's often in the runtime... so..

I agree that it's not perfect if local/remote go versions don't match. We can track this issue - how to notify users of this version skew - separately, independent of this PR.

There is one more issue. In the specific example I tested (https://github.com/hyangah/go-docker-alpine-remote-debug), the main package is main, not the package path. So, I had to adjust "ClientMod2Dir" like

           "ClientMod2Dir": {
                    "main": "/usr/local/google/home/hakim/projects/go-docker-alpine-remote-debug",
                    "github.com/ccampo133/go-docker-alpine-remote-debug":"/usr/local/google/home/hakim/projects/go-docker-alpine-remote-debug"
            }

I will send the copy of the binary built in case it's helpful for investigation.

[aarzilli]

Added the GOROOT mapping and fixed the example. The example did not work because there the main module contains a single package (main).

[derekparker]

LGTM

@hyangah can you approve as well? After your +1 I will merge.

[hyangah]

Thanks (and sorry for all this delay).

One more question: what is the meaning of "ImportPathOfMainPackage"?

Running dlv substitute-path-guess-helper from cmd/dlv directory, for example, returns "github.com/go-delve/delve/service/dap/daptest/gen".

Looking into the code, I wonder if we should use go list -json --deps . instead of go list -json all to include only dependencies to be compiled into the binary. (and for test targets, probably -tests, but I am not 100% sure)

[aarzilli]

Thanks (and sorry for all this delay).

One more question: what is the meaning of "ImportPathOfMainPackage"?

It's the import path of the package named 'main'. I added it because of https://github.com/hyangah/go-docker-alpine-remote-debug where without it we wouldn't know how to map the files in the main module.

Running dlv substitute-path-guess-helper from cmd/dlv directory, for example, returns "github.com/go-delve/delve/service/dap/daptest/gen".

I hadn't considered that a module could contain multiple packages named main. I have changed the code to account for that. For now I'm just discarding this information when there are multiple main packages. This would only affect a module that contains at least two main packages and zero non-main packages, I think this is unlikely and we can ignore it.

Looking into the code, I wonder if we should use go list -json --deps . instead of go list -json all to include only dependencies to be compiled into the binary. (and for test targets, probably -tests, but I am not 100% sure)

To do that we need to know the path of the main package, right? I don't think VSCode-go necessarily knows that.

[hyangah]

Thanks! LGTM.

BTW, I wonder if golang.org/x/tools/go/packages can be useful when adding support for other build systems like bazel (with GOPACKAGESDRIVER) in the future.

[hyangah]

(optional) to be consistent with other fields style, it may be json:"guessSubstitutePath,omitempty".

I'm ok with keeping the json encoding of api.GuessSubstitutePathIn fields as they are though because that's an output of dlv substitute-path-guess-helper output.

[aarzilli]

Done.

[hyangah]

since this is a public page :-) I think it's worth mention that this is an internal utility to support DAP's remote debugging guess substitute path.

[aarzilli]

Actually, I don't think that page should exist at all, since it's an hidden command. I've removed it. Thanks.

[aarzilli]

Thanks! LGTM.

BTW, I wonder if golang.org/x/tools/go/packages can be useful when adding support for other build systems like bazel (with GOPACKAGESDRIVER) in the future.

Is this something that has actually been implemented (a GOPACKAGESDRIVER for bazel) or is it planned?

[hyangah]

Thanks! LGTM.
BTW, I wonder if golang.org/x/tools/go/packages can be useful when adding support for other build systems like bazel (with GOPACKAGESDRIVER) in the future.

Is this something that has actually been implemented (a GOPACKAGESDRIVER for bazel) or is it planned?

I think https://pkg.go.dev/github.com/bazelbuild/rules_go/go/tools/gopackagesdriver is one of the open-source bazel GOPACKAGESDRIVER. Google internally also has a similar one. But, how complete, reliable they are currently (e.g. does 'module' info make sense and get populated...) needs research.