Failure to load some plugins when built with Go 1.18

[dtrudg]

Version of Singularity

master / release 3.9

Describe the bug

E2E tests with go 1.18 will fail with...

FATAL:   Plugin compile failed with error: while generating plugin manifest: while loading plugin /tmp/plugin-3139335449/src/plugin.so: plugin.Open("/tmp/plugin-3139335449/src/plugin"): plugin was built with a different version of package github.com/sylabs/singularity/internal/pkg/util/user

The plugin compiles, but on load Go is finding a package ABI hash that doesn't match.

Confirmed there is no issue under Go 1.17.8.

Did a very quick look at 'strings' output for the plugin.so vs singularity under 1.17.8 and 1.18 and there isn't any gross difference in the diffs.

internal/pkg/util/user does contain CGO code... so am suspecting something related to that at present.

Will be trying to work up a small reproducer... no luck to date... and subtract from the singularity code base to isolate this further.

[dtrudg]

Simple main + plugin reproducer that brings in github.com/sylabs/singularity/internal/pkg/util/user compiles and loads fine with Go 1.18.

Going to concentrate on subtraction from the full singularity code base now, and likely patch some debug output into the package hash / symbol calculation in Go 1.18. Likely this will drag through next week.

[dtrudg]

Wrote up some detail in an issue on the golang/go repository here: golang/go#51955

It appears this is due to a path to a CGO related temporary /tmp/gobuildxxxxx directory making its way into the strings section for github.com/sylabs/singularity/internal/pkg/util/user in the main singularity build, which does not happen with the plugin build.

I think this should probably never happen? A tmpdir like this is going to be different by definition on each compilation.

Will keep 🤞 that there we are able to get some insight through the linked issue.

[dtrudg]

Added some 1.17.8 vs 1.18 context on the Go issue: golang/go#51955 (comment)

Going to begin looking at places referencing _cgo_gotypes.go in the Go 1.18 code to see if I can work out where the extra path prefix in non-plugin buildmodes is sneaking in.

[dtrudg]

As explored on the golang/go issue, the problem with plugin builds in Go 1.18 is that our -gcflags -trimpath results in the 'automatic' trimpath rules being lost for our _cgo_gotypes.go file... and it ends up with a /tmp/gobuildxxxx path still prefixed to it.

It looks like this change from 1.17 -> 1.18 is unlikely to be addressed, as even back in 2019 when our plugin handling that exploited the -gcflags -trimpath was added, this was not supported...

The reason we need the -gcflags trimpath is so the strings in the main singularity binaries reference @v0.0.0 of github.com/sylabs/singularity. A plugin's own go.mod always references a @v0.0.0 for github.com/sylabs/singularity, so the plugin symbols will include that:

module github.com/sylabs/singularity/e2e-cli-plugin

go 1.13

require github.com/sylabs/singularity v0.0.0

replace github.com/sylabs/singularity => ./singularity_source

Without the custom gcflags trimpath....

-gcflags=github.com/sylabs/singularity/...="-trimpath $(SOURCEDIR)=>github.com/sylabs/singularity@v0.0.0

... that forces the @v0.0.0 into the main binary, we move from a mismatch on internal/pkg/util/user that uses CGO, to more general issues that report a mismatch on pkg/plugin between main binary and plugin.

The main binary strings don't include the @v0.0.0 and there is a mismatch from main binary to plugin, so the plugin cannot load.

Plugin

;github.com/sylabs/singularity@v0.0.0/pkg/plugin/manifest.goName
json:"author"Versionjson:"version"                              json:"name"Author
PluginRootDirPluginpluginPluginRootDirSymbolnjson:"description"9git.luolix.top/sylabs/singularity@v0.0.0/pkg/plugin/plugin.gManifest        CallbackCallbackInstall
                                            PluginSymbol

Main Binary

4git.luolix.top/sylabs/singularity/pkg/plugin/manifest.goName
json:"author"Versionjson:"version"                       json:"name"Author
PluginRootDirPluginpluginPluginRootDirSymbolnjson:"description"2git.luolix.top/sylabs/singularity/pkg/plugin/plugin.gManifest       CallbackCallbackInstall
                                            PluginSymbol

Seems pretty clear that our approach to build plugins out of tree like this is really abusing the way go.mod versioning etc. is meant to operate.... and we're not going to get back the 1.17 and prior gcflags trimpath behavior that allows us to do this.

I suspect we will have to go back to building plugins within the main source tree, under the scope of the main singularity go.mod.

[dtrudg]

The decision is to switch back to compiling plugins in-tree, which infers the plugin uses the singularity go.mod, for 3.10

For 3.9, whether to make the same (breaking) change, or just exclude the failing test and issue errata is under consultation:

https://groups.google.com/g/singularity-ce/c/jyrQXcqTG0I

@tri-adam and myself discussed possible approaches. We don't believe that there is a practical safe alternative going forward that preserves out of tree build for plugins. We need to be conservative as the Go developers are clear on the expectations of what should work with Go plugins, and this issue shows that if we go beyond this we are likely to hit problems with new Go releases.

The 'proper' way to deal with this would be to split SingularityCE, so that the CLI is separate from a runtime module. Both the CLI and plugins could import this runtime module, of a specific and matching version from their own go.mod. CLI and This is the right kind of structure to avoid the -gcflags -trimpath that we have been using to 'patch' the package version strings in the main singularity executables when they are compiled, so that they match those in plugin builds.