-
Notifications
You must be signed in to change notification settings - Fork 17.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
proposal: cmd/go: allow versioned go build
commands
#44469
Comments
On #44203, @jayconrod said:
If the plan is indeed to support versioned variants of multiple commands, perhaps we should use a single issue for all of them at once instead of one per command. |
I think we should definitely do that, regardless of whether we add Can we make this issue about that, and discuss generalizing |
Assuming the scope of #44203 is not limited to What will be the expected behavior of
which installs the output in |
I would expect
Although honestly, in my own use I would probably spell that as:
|
The downside of that is that it depends on POSIX Shell syntax, so it wouldn't be portable to e.g. Windows. I see little benefit of I do see the point in flags being consistent between commands, but the whole point of |
If there were such a flag, I would not want to name it |
Maybe. The flip side of the coin is that |
|
IMO yes, but I'd like to see what others think. 😅 |
I think if Agree that it's redundant with |
Currently we are using the sequence of
|
If it helps, When you say "for all platforms", are you cross-compiling? If |
In our specific case, we don't do cross compiling which simplifies things. To me
|
Do I have this right? Though I'm not sure when you'd want
|
IMO, the |
I would rather we have consistent meanings for flags than make every subcommand completely orthogonal. It's not obvious to me that “installing a differently named binary in the default location” really needs to be convenient. |
Thanks @seankhliao. Is there any command that can provide the default installation directory? If so, I am happy to use it to formulate the right |
Within a module, something like If we accept #44203, that would shorten to |
GH-89 [1] supersedes GH-78 [2] which documents how the official deprecation [3] of `gobin` [4] in favor of the new Go 1.16 `go install pkg@version` [5] syntax feature should have been handled for this project. The idea was to replace the `gobin` task runner [6] with a one that leverages "bingo" [7], a project similar to `gobin`, that comes with many great features and also allows to manage development tools on a per-module basis. The problem is that `bingo` uses some non-default and nontransparent mechanisms under the hood and automatically generates files in the repository without the option to disable this behavior. It does not make use of the `go install` command but relies on custom dependency resolution mechanisms, making it prone to future changes in the Go toolchain and therefore not a good choice for the maintainability of projects. >>> `go install` is still not perfect Support for the new `go install` features, which allow to install commands without affecting the `main` module, have already been added in GH-71 [8] as an alternative to `gobin`, but one significant problem was still not addressed: install module/package executables globally without overriding already installed executables of different versions. Since `go install` will always place compiled binaries in the path defined by `go env GOBIN`, any already existing executable with the same name will be replaced. It is not possible to install a module command with two different versions since `go install` still messes up the local user environment. >>> The Workaround: Hybrid `go install` task runner This commit therefore implements the solution through a custom `Runner` [9] that uses `go install` under the hood, but places the compiled executable in a custom cache directory instead of `go env GOBIN`. The runner checks if the executable already exists, installs it if not so, and executes it afterwards. The concept of storing dependencies locally on a per-project basis is well-known from the `node_modules` directory [10] of the "Node" [11] package manager "npm" [12]. Storing executables in a cache directory within the repository (not tracked by Git) allows to use `go install` mechanisms while not affect the global user environment and executables stored in `go env GOBIN`. The runner achieves this by changing the `GOBIN` environment variable to the custom cache directory during the execution of `go install`. This way it bypasses the need for "dirty hacks" while using a custom output path. The only known disadvantage is the increased usage of storage disk space, but since most Go executables are small in size anyway, this is perfectly acceptable compared to the clearly outweighing advantages. Note that the runner dynamically runs executables based on the given task so `Validate() error` is a NOOP. >>> Upcoming Changes The solution described above works totally fine, but is still not a clean solution that uses the Go toolchain without any special logic so as soon as the following changes are made to the Go toolchain (Go 1.17 or later), the custom runner will be removed again: - golang/go/issues#42088 [13] — tracks the process of adding support for the Go module syntax to the `go run` command. This will allow to let the Go toolchain handle the way how compiled executable are stored, located and executed. - golang/go#44469 [14] — tracks the process of making `go install` aware of the `-o` flag like the `go build` command which is the only reason why the custom runner has been implemented. >>> Further Adjustments Because the new custom task runner dynamically runs executables based on the given task, the `Bootstrap` method [15] of the `Wand` [16] reference implementation `Elder` [17] additionally allows to pass Go module import paths, optionally including a version suffix (`pkg@version`), to install executables from Go module-based `main` packages into the local cache directory. This way the local development environment can be set up, for e.g. by running it as startup task [18] in "JetBrains" IDEs. The method also ensures that the local cache directory exists and creates a `.gitignore` file that includes ignore pattern for the cache directory. [1]: #89 [2]: #78 [3]: myitcv/gobin#103 [4]: https://github.com/myitcv/gobin [5]: https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies [6]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/task/gobin#Runner [7]: https://github.com/bwplotka/bingo [8]: #71 [9]: https://pkg.go.dev/github.com/svengreb/wand/pkg/task#Runner [10]: https://docs.npmjs.com/cli/v7/configuring-npm/folders#node-modules [11]: https://nodejs.org [12]: https://www.npmjs.com [13]: golang/go#42088 [14]: golang/go#44469 (comment) [15]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/elder#Elder.Bootstrap [16]: https://pkg.go.dev/github.com/svengreb/wand#Wand [17]: https://pkg.go.dev/github.com/svengreb/wand/pkg/elder#Elder [18]: https://www.jetbrains.com/help/idea/settings-tools-startup-tasks.html GH-89
GH-89 [1] supersedes GH-78 [2] which documents how the official deprecation [3] of `gobin` [4] in favor of the new Go 1.16 `go install pkg@version` [5] syntax feature should have been handled for this project. The idea was to replace the `gobin` task runner [6] with a one that leverages "bingo" [7], a project similar to `gobin`, that comes with many great features and also allows to manage development tools on a per-module basis. The problem is that `bingo` uses some non-default and nontransparent mechanisms under the hood and automatically generates files in the repository without the option to disable this behavior. It does not make use of the `go install` command but relies on custom dependency resolution mechanisms, making it prone to future changes in the Go toolchain and therefore not a good choice for the maintainability of projects. >>> `go install` is still not perfect Support for the new `go install` features, which allow to install commands without affecting the `main` module, have already been added in GH-71 [8] as an alternative to `gobin`, but one significant problem was still not addressed: install module/package executables globally without overriding already installed executables of different versions. Since `go install` will always place compiled binaries in the path defined by `go env GOBIN`, any already existing executable with the same name will be replaced. It is not possible to install a module command with two different versions since `go install` still messes up the local user environment. >>> The Workaround: Hybrid `go install` task runner This commit therefore implements the solution through a custom `Runner` [9] that uses `go install` under the hood, but places the compiled executable in a custom cache directory instead of `go env GOBIN`. The runner checks if the executable already exists, installs it if not so, and executes it afterwards. The concept of storing dependencies locally on a per-project basis is well-known from the `node_modules` directory [10] of the "Node" [11] package manager "npm" [12]. Storing executables in a cache directory within the repository (not tracked by Git) allows to use `go install` mechanisms while not affect the global user environment and executables stored in `go env GOBIN`. The runner achieves this by changing the `GOBIN` environment variable to the custom cache directory during the execution of `go install`. This way it bypasses the need for "dirty hacks" while using a custom output path. The only known disadvantage is the increased usage of storage disk space, but since most Go executables are small in size anyway, this is perfectly acceptable compared to the clearly outweighing advantages. Note that the runner dynamically runs executables based on the given task so `Validate() error` is a NOOP. >>> Upcoming Changes The solution described above works totally fine, but is still not a clean solution that uses the Go toolchain without any special logic so as soon as the following changes are made to the Go toolchain (Go 1.17 or later), the custom runner will be removed again: - golang/go/issues#42088 [13] — tracks the process of adding support for the Go module syntax to the `go run` command. This will allow to let the Go toolchain handle the way how compiled executable are stored, located and executed. - golang/go#44469 [14] — tracks the process of making `go install` aware of the `-o` flag like the `go build` command which is the only reason why the custom runner has been implemented. >>> Further Adjustments Because the new custom task runner dynamically runs executables based on the given task, the `Bootstrap` method [15] of the `Wand` [16] reference implementation `Elder` [17] additionally allows to pass Go module import paths, optionally including a version suffix (`pkg@version`), to install executables from Go module-based `main` packages into the local cache directory. This way the local development environment can be set up, for e.g. by running it as startup task [18] in "JetBrains" IDEs. The method also ensures that the local cache directory exists and creates a `.gitignore` file that includes ignore pattern for the cache directory. [1]: #89 [2]: #78 [3]: myitcv/gobin#103 [4]: https://github.com/myitcv/gobin [5]: https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies [6]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/task/gobin#Runner [7]: https://github.com/bwplotka/bingo [8]: #71 [9]: https://pkg.go.dev/github.com/svengreb/wand/pkg/task#Runner [10]: https://docs.npmjs.com/cli/v7/configuring-npm/folders#node-modules [11]: https://nodejs.org [12]: https://www.npmjs.com [13]: golang/go#42088 [14]: golang/go#44469 (comment) [15]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/elder#Elder.Bootstrap [16]: https://pkg.go.dev/github.com/svengreb/wand#Wand [17]: https://pkg.go.dev/github.com/svengreb/wand/pkg/elder#Elder [18]: https://www.jetbrains.com/help/idea/settings-tools-startup-tasks.html GH-89
…#90) GH-89 [1] supersedes GH-78 [2] which documents how the official deprecation [3] of `gobin` [4] in favor of the new Go 1.16 `go install pkg@version` [5] syntax feature should have been handled for this project. The idea was to replace the `gobin` task runner [6] with a one that leverages "bingo" [7], a project similar to `gobin`, that comes with many great features and also allows to manage development tools on a per-module basis. The problem is that `bingo` uses some non-default and nontransparent mechanisms under the hood and automatically generates files in the repository without the option to disable this behavior. It does not make use of the `go install` command but relies on custom dependency resolution mechanisms, making it prone to future changes in the Go toolchain and therefore not a good choice for the maintainability of projects. >>> `go install` is still not perfect Support for the new `go install` features, which allow to install commands without affecting the `main` module, have already been added in GH-71 [8] as an alternative to `gobin`, but one significant problem was still not addressed: install module/package executables globally without overriding already installed executables of different versions. Since `go install` will always place compiled binaries in the path defined by `go env GOBIN`, any already existing executable with the same name will be replaced. It is not possible to install a module command with two different versions since `go install` still messes up the local user environment. >>> The Workaround: Hybrid `go install` task runner This commit therefore implements the solution through a custom `Runner` [9] that uses `go install` under the hood, but places the compiled executable in a custom cache directory instead of `go env GOBIN`. The runner checks if the executable already exists, installs it if not so, and executes it afterwards. The concept of storing dependencies locally on a per-project basis is well-known from the `node_modules` directory [10] of the "Node" [11] package manager "npm" [12]. Storing executables in a cache directory within the repository (not tracked by Git) allows to use `go install` mechanisms while not affect the global user environment and executables stored in `go env GOBIN`. The runner achieves this by changing the `GOBIN` environment variable to the custom cache directory during the execution of `go install`. This way it bypasses the need for "dirty hacks" while using a custom output path. The only known disadvantage is the increased usage of storage disk space, but since most Go executables are small in size anyway, this is perfectly acceptable compared to the clearly outweighing advantages. Note that the runner dynamically runs executables based on the given task so `Validate() error` is a NOOP. >>> Upcoming Changes The solution described above works totally fine, but is still not a clean solution that uses the Go toolchain without any special logic so as soon as the following changes are made to the Go toolchain (Go 1.17 or later), the custom runner will be removed again: - golang/go/issues#42088 [13] — tracks the process of adding support for the Go module syntax to the `go run` command. This will allow to let the Go toolchain handle the way how compiled executable are stored, located and executed. - golang/go#44469 [14] — tracks the process of making `go install` aware of the `-o` flag like the `go build` command which is the only reason why the custom runner has been implemented. >>> Further Adjustments Because the new custom task runner dynamically runs executables based on the given task, the `Bootstrap` method [15] of the `Wand` [16] reference implementation `Elder` [17] additionally allows to pass Go module import paths, optionally including a version suffix (`pkg@version`), to install executables from Go module-based `main` packages into the local cache directory. This way the local development environment can be set up, for e.g. by running it as startup task [18] in "JetBrains" IDEs. The method also ensures that the local cache directory exists and creates a `.gitignore` file that includes ignore pattern for the cache directory. [1]: #89 [2]: #78 [3]: myitcv/gobin#103 [4]: https://github.com/myitcv/gobin [5]: https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies [6]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/task/gobin#Runner [7]: https://github.com/bwplotka/bingo [8]: #71 [9]: https://pkg.go.dev/github.com/svengreb/wand/pkg/task#Runner [10]: https://docs.npmjs.com/cli/v7/configuring-npm/folders#node-modules [11]: https://nodejs.org [12]: https://www.npmjs.com [13]: golang/go#42088 [14]: golang/go#44469 (comment) [15]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/elder#Elder.Bootstrap [16]: https://pkg.go.dev/github.com/svengreb/wand#Wand [17]: https://pkg.go.dev/github.com/svengreb/wand/pkg/elder#Elder [18]: https://www.jetbrains.com/help/idea/settings-tools-startup-tasks.html Closes GH-89
…#90) GH-89 [1] supersedes GH-78 [2] which documents how the official deprecation [3] of `gobin` [4] in favor of the new Go 1.16 `go install pkg@version` [5] syntax feature should have been handled for this project. The idea was to replace the `gobin` task runner [6] with a one that leverages "bingo" [7], a project similar to `gobin`, that comes with many great features and also allows to manage development tools on a per-module basis. The problem is that `bingo` uses some non-default and nontransparent mechanisms under the hood and automatically generates files in the repository without the option to disable this behavior. It does not make use of the `go install` command but relies on custom dependency resolution mechanisms, making it prone to future changes in the Go toolchain and therefore not a good choice for the maintainability of projects. >>> `go install` is still not perfect Support for the new `go install` features, which allow to install commands without affecting the `main` module, have already been added in GH-71 [8] as an alternative to `gobin`, but one significant problem was still not addressed: install module/package executables globally without overriding already installed executables of different versions. Since `go install` will always place compiled binaries in the path defined by `go env GOBIN`, any already existing executable with the same name will be replaced. It is not possible to install a module command with two different versions since `go install` still messes up the local user environment. >>> The Workaround: Hybrid `go install` task runner This commit therefore implements the solution through a custom `Runner` [9] that uses `go install` under the hood, but places the compiled executable in a custom cache directory instead of `go env GOBIN`. The runner checks if the executable already exists, installs it if not so, and executes it afterwards. The concept of storing dependencies locally on a per-project basis is well-known from the `node_modules` directory [10] of the "Node" [11] package manager "npm" [12]. Storing executables in a cache directory within the repository (not tracked by Git) allows to use `go install` mechanisms while not affect the global user environment and executables stored in `go env GOBIN`. The runner achieves this by changing the `GOBIN` environment variable to the custom cache directory during the execution of `go install`. This way it bypasses the need for "dirty hacks" while using a custom output path. The only known disadvantage is the increased usage of storage disk space, but since most Go executables are small in size anyway, this is perfectly acceptable compared to the clearly outweighing advantages. Note that the runner dynamically runs executables based on the given task so `Validate() error` is a NOOP. >>> Upcoming Changes The solution described above works totally fine, but is still not a clean solution that uses the Go toolchain without any special logic so as soon as the following changes are made to the Go toolchain (Go 1.17 or later), the custom runner will be removed again: - golang/go/issues#42088 [13] — tracks the process of adding support for the Go module syntax to the `go run` command. This will allow to let the Go toolchain handle the way how compiled executable are stored, located and executed. - golang/go#44469 [14] — tracks the process of making `go install` aware of the `-o` flag like the `go build` command which is the only reason why the custom runner has been implemented. >>> Further Adjustments Because the new custom task runner dynamically runs executables based on the given task, the `Bootstrap` method [15] of the `Wand` [16] reference implementation `Elder` [17] additionally allows to pass Go module import paths, optionally including a version suffix (`pkg@version`), to install executables from Go module-based `main` packages into the local cache directory. This way the local development environment can be set up, for e.g. by running it as startup task [18] in "JetBrains" IDEs. The method also ensures that the local cache directory exists and creates a `.gitignore` file that includes ignore pattern for the cache directory. [1]: #89 [2]: #78 [3]: myitcv/gobin#103 [4]: https://github.com/myitcv/gobin [5]: https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies [6]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/task/gobin#Runner [7]: https://github.com/bwplotka/bingo [8]: #71 [9]: https://pkg.go.dev/github.com/svengreb/wand/pkg/task#Runner [10]: https://docs.npmjs.com/cli/v7/configuring-npm/folders#node-modules [11]: https://nodejs.org [12]: https://www.npmjs.com [13]: golang/go#42088 [14]: golang/go#44469 (comment) [15]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/elder#Elder.Bootstrap [16]: https://pkg.go.dev/github.com/svengreb/wand#Wand [17]: https://pkg.go.dev/github.com/svengreb/wand/pkg/elder#Elder [18]: https://www.jetbrains.com/help/idea/settings-tools-startup-tasks.html Closes GH-89
`go install` with the version specifier is THE way to install a binary in recent versions of Go. From go1.18, `go get` will not do build & install so shouldn't be used. For `dlv-dap` or `gocode-gomod`, we use `go get` + `go build` because `go install` does not allow to specify output path. (golang/go#44469) Use of `go install` allows us to address another issue: we no longer require a dummy main module in a temp directory, and we can run it from the workspace directory. The use of temp directory broke direnv, asdf users who configured their go to pick a different version of go based on the directory. Thus, we pinned the go binary by selecting the go binary from the current GOROOT's bin directory. Unfortunately, that broke another group of users who are using GOROOT for different purpose. (#1691) Now we can remove the hack to pin the go binary and use the go binary path as it is, and simplifies the installation logic. Reduced the verbose logging printed in the console. Instead, we log the command and the directory path so users could reproduce. In case of errors, the exception still includes all the stdout and stderr. Updates #1825 Change-Id: Idb1604e4484cd73832c24c0077220f8cc57dfaa0 Reviewed-on: https://go-review.googlesource.com/c/vscode-go/+/355974 Trust: Hyang-Ah Hana Kim <hyangah@gmail.com> Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com> TryBot-Result: kokoro <noreply+kokoro@google.com> Reviewed-by: Suzy Mueller <suzmue@golang.org>
Repeating here the problem I raised earlier in the hope to get a solution or traction: In order to take advantage of the new in 1.18 fantastic But Example: before: CGO_ENABLED=0 GOOS=windows go build -a -ldflags \
'-s -X fortio.org/fortio/version.version=1.28.0 -X "fortio.org/fortio/version.buildInfo=2022-04-26 00:40 57e9d8f01c342a4c1b5d96f883f81128b04d991a"' \
-o /tmp/fortio.exe fortio.org/fortio after/now GOPATH=/build CGO_ENABLED=0 GOOS=darwin /usr/local/go/bin/go install -a -ldflags -s fortio.org/fortio@v1.29.0-pre16
mv -f /build/bin/*_*/fortio* /build/bin
rmdir /build/bin/*_* I end up having to use GOPATH to try to set where the binary ends up as well as using some hacks to deal with native vs non native arch |
I'm working on a non-golang project that depends on a handful of golang binaries. Which unfortunately all have the name Since my project isn't a go module, I can't even use And if your concern is consistency of options, then what about making it a new option other than |
TL;DR: Usecase showing how name clashing during To leave a practical example of how cumbersome the user experience can be with Usecase: Run the tool defined at github.com/Jille/raft-grpc-example/cmd/hammer plus some other raft-related tooling I can go install Anyway, ideally I would want to build it as "raft-hammer" So this was my user experience: Lets try installing it with a different name. How do I do so?
Ok, so it takes build flags. Lets try:
Well, thats weird, takes build flags but not this one build flag. Lets google it. I ended up finding this GH Issue Fair enough, seems like this discussion went a lot around Anyway, what if we try to simple
Ok, so Well I guess we need to go for
Now I have my tool without losing the previous one |
@bcap I had the same issue, but used the workaround of @hyangah: add a You can then use
It even works for cross-compilation (go install currently refuses a custom GOBIN for cross-compilation) |
Sigh. It would be really nice if something like this existed. Reading through the objections, how about using go install -O foozle some.git.host/flibble,burble/foozle.git@v1.3-kitten That way it doesn't clash with the |
being able to specify a full destination path including directory (ie with slashes) is what I need for ci builds or predictable installs but maybe #52898 can be reopen if the direction for this one is to only address the binary basename |
I think that since we support |
@hyangah Should we turn this into a proposal issue? |
Do we need more than marking this as a proposal? |
I think we could update the first comment with the current version of the proposal? I'll mark this as a proposal. |
go build
commandsgo build
commands
I need "go install" to specify an abbreviation different from the package name, for example: go-effex-cli is gec in my project, supporting "-o" is a good idea. |
Update 2024-08-21 by matloob: The proposal has been updated to the following:
The proposal is to allow
package@version
arguments togo build
.They would behave similarly to how they do for
go run
: the module containingpackage
at the given version would be loaded outside of the context of the current module. It would use cmd/go/internal/load.PackagsAndErrorsOutsideModule, so it would load the given module as if it were a requirement of an otherwise empty module. This also enforces that the version of the module the package belongs to doesn't have replace or exclude directives.Similar to
go run
, if mulitple packages are supplied, they would be required to belong to the same module at the same version. (Though providing multiplepackage@version
arguments wouldn't be useful in practice because the main use case would be to produce the build outputs.)Alternative:allow to change the installed main package name ingo install
commandSince go1.16,
go install
accepts arguments with version suffixes.go install
places the output in$GOPATH/bin
or$GOBIN
, but does not allow the output binary's name change.When a user wants to use multiple versions of tools, the user need to be careful not to overwrite other versions.
Different versions may be coming from different module paths.
e.g. VS Code Go extension uses two different versions of
gocode
(stamblerre/gocode and mdempsky/gocode). In order to install the former, it does switchingGOBIN
to use a temporary directory, installing it, and renaming and copying it to the realGOBIN
.Alternative: There are other proposals for applying the same to
go run
(#42088) andgo list
(#44203). Extendinggo build
in the same way (or extendinggo install
) will help simplifying the tool installation workflow.cc @jayconrod @bcmills @matloob
The text was updated successfully, but these errors were encountered: