proposal: Go 2: allow importing the main function from a main package

[pdmccormick]

Go Programming Experience

Experienced

Other Languages Experience

C, Python

Has this idea, or one like it, been proposed before?

No, not to my knowledge.

Edit: Issue #4210, commit 679fd5b and CL 4126053 track when importing main became disallowed by the standard toolchain, but not explicitly prohibited in the language specification.

Does this affect error handling?

No.

Is this about generics?

No.

Proposal

Currently, using the standard Go compiler, if you try to import a main package you will get this error:

multitool.go:7:2: import "cmd/go" is a program, not an importable package

This proposal is to allow importing of main functions from Go programs under two conditions:

1. The import of a main package must be qualified with a package name, and
2. The only exported identifier is Main, which is an alias to the main function inside the main package.

Motivation

The gobusybox project uses a source-to-source transformation step to bundle up many separate package main Go programs into a single monolithic executable. Each individual subprogram can then be separately invoked, whether by being specified as the first command-line argument, or by being invoked through a symlink (with the name of the symlink corresponding to the subprogram to invoke). This can yield significant storage savings, since all the bundled programs in one executable would share a single copy of compiled common code such as the Go runtime, standard library and third party packages. The alternative is to compile each program individually, leading to the duplication of compiled code. These savings can be very meaningful in resource constrained contexts such as embedded systems with limited storage.

Further, there is now a huge ecosystem of Go programs and utilities, and many of them were not written to have any other way of importing and invoking them other than as programs via their main entrypoint, with the os package Args, Environ, Stdout, Stderr, Stdin being the expected process-oriented invocation interface. None the less, it would be incredibly useful to be able to import these existing programs without modification and bundle them into a single monolithic Go executable. There is no guarantee that those programs will "play nice" when invoked under different assumptions, but then there never is when using a third party package.

Example

package main

import (
    "fmt"
    "os"

    gocmd "cmd/go"
    gofmtcmd "cmd/gofmt"
)

var programs = map[string]func(){
    "go":    gocmd.Main,
    "gofmt": gofmtcmd.Main,

    /*
        Include your favourite programs into your Go multi-tool build. Examples:

        golang.org/x/tools/gopls
        github.com/jsha/minica
        k8s.io/kubernetes/cmd/kubectl
        github.com/docker/cli/cmd/docker
    */
}

func main() {
    if args := os.Args; len(args) > 1 {
        var (
            exe      = args[0]
            prog     = args[1]
            progArgs = args[2:]
            newArgs  = append([]string{exe}, progArgs...)
        )

        if mainFunc, ok := programs[prog]; ok {
            // This will not work for programs that `os/exec` themselves as
            // child processes.
            //
            // Use symlinks to the main executable, and dispatch based on name.
            os.Args = newArgs

            mainFunc()
            return
        }

    }

    for name := range programs {
        fmt.Println(name)
    }
}

Language Spec Changes

I could not find any text in The Go Programming Language Specification (as of language version go1.22, dated Feb 6, 2024) that would suggest that main packages MUST NOT be importable. My first question is, is the standard compiler behaviour just an implementation choice, or does that behaviour reflect something that was actually intended to be included in the Language Specification? If the current proposal is therefore not a change to the language but rather the standard compiler, then this proposal should be recategorized.

Assuming that the current behaviour was intended to be included in the Language Specification, and this proposal should be evaluated in the context as being a change to that behaviour, here are the further changes required.

In the section "Program Execution", it states:

When that function [referring to the function main in the package main of a Go program] invocation returns, the program exits. It does not wait for other (non-main) goroutines to complete.

While this statement would remain true for the actual main function of the program, it would not be true for a main that was imported and invoked like a regular function. But this proposed change would not break any existing programs.

In the section "Import declarations", the following addition is suggested:

If the imported package is a program, which is to say the identifier specified in the package clause of the imported package is main, then the ImportSpec must specify a PackageName that is not the blank identifier.

In the section "Exported identifiers", adding a 3rd point to the list:

3. the identifier is Main and the package clause of the imported package is main, in which case the identifier is a reference to the main function.

Informal Change

In Go, program execution starts by calling your main function inside a package main source file. Currently you are now allowed to import another package main, and even if you could, you would be able to call the main function, since by the rules of exported identifiers the main function does not qualify because of the lowercased starting letter. But since there are so many useful Go programs out there, what if we could get access to their main functions, and bundle their functionality into our own programs? Not all third party programs will work as expected this way (have you ever thought about what it would be like for someone else to call your program main as a function?), but compiling one big monolithic program together might be easier than having to deal with multiple executables.

Is this change backward compatible?

Yes. Since importing a main package is currently not allowed in practise, no valid programs have been doing this. Existing programs would be unaffected.

By aliasing the main function to an exported identifier Main, as a special case behaviour for main packages, the rules around exported identifiers would be unchanged.

The main function from existing programs may not necessarily be well-behaved from the perspective of being imported and called from another. But this situation is no different than importing a third party package and finding that it, for example, calls panic when returning an error would have been more convenient for the caller. It simply may not be convenient or practical to import another program and actually make it useful.

There is no requirement on programs, new or old, to change anything about how they behave or interact with the operating system. It is entirely the callers responsibility to provide a suitable state when calling main (for example, by manipulating os.Args to still be meaningful).

Orthogonality: How does this change interact or overlap with existing features?

I think it's a fairly conservative extension to current practise. Since the import must include an explicit package name, the compiler would still return an error if someone accidentally tried to import an main package without a package name qualifier. The user would have to be quite intentional to access this capability.

Would this change make Go easier or harder to learn, and why?

While I do understand the argument against being able to import another main package, it always struck me as a a bit irregular that the rules around importing main were so different to any other package. A lot of projects put a large amount of code into their main packages, because they are intending to process a program, and not a reusable/importable package. But they still intended to be invokable, as programs, which is very similar in concept to being able to call their main function.

With the rise of a robust module system and wider ecosystem, perhaps this restriction made more sense in an earlier era of Go's development?

But this proposal could lead to unintuitive behaviour for those who use it. Why are other seemingly legitimately exportable identifiers not in fact exported? What if a program contained both a main and Main function? Only the main function should be exported as Main, otherwise existing programs may be unimportable (i.e. if Main had a different function signature, or if it was not a function at all), but this could be extremely confusing.

Cost Description

I am not aware of anywhere else in Go where an identifier without an uppercase first character can be exported (i.e. main), or where one imported identifier is an alias for another (Main => main). This would require even more special case rules for how the main package is handled.

Changes to Go ToolChain

Entire compiler, from parser to linker, plus any tool that interacts with source code (such as vet, gopls, gofmt and goimports)

Performance Costs

None (to the best of my knowledge!)

Prototype

Work in progress. I'm working this out by first relaxing this error condition and seeing what happens next.

[pdmccormick]

I would humbly submit that this is not a Go 2 proposal, since the current standard compiler behaviour is not actually called for in the Go Programming Language Specification, and even if it was intended to be, this proposal is not a backwards compatibility breaking change to current practise.

[Jorropo]

Can't we already effectively do that ?

I mean, ok you can't literally do what you are proposing, but you could have the Main function but in it's own non main package and have your program and have both the mega-main and unique main packages both import that.

I imagine something like this:

// cmd/go/go/go.go
package main

import gocmd "cmd/go"

func main() {
 gocmd.Main()
}

The examples you gave would be identical.
That would allows to not complexify the current simple-ish rules around importing main.

This would require some automated rewrite step similar to what gobusybox do.

[seankhliao]

I think it is a language change, as it defines new visibility rules for main packages (an alias Main, no other "exported" identifiers actually exported).

---

I think this may be an unsafe thing to do in general:

• main programs may often be written to assume it has full control, e.g. registering global resources in package initialization, just importing the packages can lead to conflict.
• extended backwards compatibility for Go #56986 allows main control over //go:debug, but this proposal means the program may run under unintended configurations by default.

While it's true that similar issues can arise with poorly written importable packages, I'd argue that the importable / unimportable separation of intent is important, and for the effort needed to fix those main packages it isn't much more work to move the code into an importable one.

[pdmccormick]

Hi @Jorropo,

Can't we already effectively do that ?

I completely agree! For my own programs I also like to use the structure of a main function in cmd/program/main.go that just calls a single exported identifier from cmd/program/programcmd.

But what about third party programs, and being able to bundle them into a single monolithic binary? If they could be changed to adopt that structure, all the better. But what if that's not possible?

I'd even be happy if there were some kind of compiler-specific mechanism to support this use case:

go build -bundlemain cmd/go -bundlemain cmd/gofmt -bundlemain you/pkg/cmd/program ... multitool.go

... and then some kind of func RunMain(package string) call to actually invoke them. Even if there were lots of further restrictions, such as RunMain will never return to the calling function (so more like an exec style syscall), if that made things easier to implement.

[zigo101]

@griesemer ever wanted to support this, but @rsc denied it.
So the possibility is gone now.

I couldn't find the related issue now.
[edit]: found that issue. But sorry it looks that issue doesn't talk about whether or not main packages can be imported or not.

[pdmccormick]

Hi @seankhliao,

I think it is a language change, as it defines new visibility rules for main packages (an alias Main, no other "exported" identifiers actually exported).

I agree that it is a language change, but I believe it could be backwards compatibility preserving change to Go 1. Maybe I am misunderstanding the meaning of "Go 2" in this context (which I thought would refer to truly breaking changes, such as if I were proposing that the main function should in practise be renamed to Main, which I am most certainly not).

I think it would be helpful to clarify the current intention of the language spec:

1. Have I simply missed the part where it talks about the non-importability of package main?
2. Is the non-importability of main something that the language spec intentionally has no opinion about, and hence implementations are free to impose additional restrictions if they wish (as the standard compiler does)?
3. Setting this proposal aside for the moment, should the spec be updated to explicitly mention the non-importability of main, to align with current practise?

I think this may be an unsafe thing to do in general:

I completely agree! There are zero guarantees that, once your program calls into another main function (if it were possible to do so), that the function will ever return back to the caller, or even if it did, that the system won't be in a very strange state. There are also no guarantees that importing two different main's won't just by themselves have unintended consequence, such as with respect to init execution order and global mutable state.

But those issues would be the callers responsibility to sort through. It would not affect the behaviour of any existing programs.

If an open source project received a bug report about "strange behaviour when importing your main" they could quite fairly say "we don't support that, won't fix".

[zephyrtronium]

@pdmccormick The "Go 2" term in issue titles, as well as the v2 label, are to facilitate triage and search. They don't imply that a proposal is not backward compatible.

[pdmccormick]

@pdmccormick The "Go 2" term in issue titles, as well as the v2 label, are to facilitate triage and search. They don't imply that a proposal is not backward compatible.

Thank you for the clarification @zephyrtronium ! I appreciate it.

[pdmccormick]

Thank you all for you input.

Starting from the origin point of the current import "cmd/foo" is a program, not an importable package error message, and searching back through the history, I came across issue #4210 . It looks like in Go pre-1.5, it was possible to import main, as @kevinwallace commented:

I'm currently using this to build busybox-style binaries. Each command's main package exports a reference to its main function, and a wrapper command imports them all and selectively calls one, switching on os.Args[0].

Further comments in the thread refer to solving the newly introduced restriction by reorganizing programs to split between the package main and an importable implementation package.

Later in that thread, @ianlancetaylor commented:

CL 4126053 is a change to the spec, which describes the language. The language permits importing a package named main. That can be used, for example, when writing unit tests for functions in commands that use package main.

This issue here is about the go tool, not the language. The question is whether the go tool should permit packages to import a package that defines a command. The general consensus is that it should not.

CL 4126053 specifically removed a spec-level prohibition against importing a main package. Therefore I would conclude that what I am proposing is actually a behaviour change to the standard compiler, not a language specification change.

Given this, would it be worth creating a new issue proposing some kind of "busybox-style multi-program build" support in the standard compiler? Whether by permitting restricted main imports, or by some other way of invoking another programs main. Would there be any willingness to consider such a proposal, @rsc ?

[ianlancetaylor]

As you say, this seems to be a reversal of #4210. The discussion there included the busybox case. What has changed that we should reconsider that earlier decision?

[pdmccormick]

As you say, this seems to be a reversal of #4210. The discussion there included the busybox case. What has changed that we should reconsider that earlier decision?

The comment in #4210 that first mentioned Busybox-style binaries also mentions that they had the alternative remedy of splitting their main function implementations out into importable packages, which has always been an option when working within a project you have some influence over.

But what about the case of bundling together existing & unmodified third party programs, to share the cost of common code that would otherwise be duplicated across multiple binaries? With the rise of Go-based Linux userlands such as u-root and gokrazy, a Go-based Linux distribution (or embedded system image) could pull together an entire suite of system tools and useful applications with a single go build command.

I probably jumped too quickly to suggesting a specific solution (allow importing main packages again) when really the question I'm grappling with is, could the Go compiler help me in my quest to build multiple programs into a single executable, and offer a Go exec(3)-like mechanism to replace the currently running program with a new one?

At the risk of jumping to solutions again, perhaps something like the combination of allowing import _ "some/pkg/cmd/foo" from main (to pull the contents of some/pkg/cmd/foo into the current program build, and to help ensure that go mod tidy will include referenced packages in go.mod) plus a runtime API such as func ExecProgram(package string, args, environ []string) error to reboot and reinitialize the runtime?

[apparentlymart]

It seems like this proposal hinges on a more philosophical question about the design goals of the language:

Should it be possible for someone to embed someone else's standalone Go program as if it were a library in someone else's Go program without first negotiating an explicit library API with the author of the first program?

Everything else in Go so far seems to have tilted toward giving authors control over their exported API: package symbols can be either exported or unexported, and the internal directory convention allows entire packages to be either exported or not. The language does not offer any explicit means for a caller to override those decisions, although I will concede that there are some seemingly-accidental workarounds available for those willing to depend on implementation details of the Go toolchain.

As an author of one such program that has intentionally not exported any library API, I would say that I would prefer this not to be accepted in the current form. I would not be opposed to an alternative that makes it opt-in for the author of the program being embedded, but I'm not really convinced such a thing is needed because it's already possible for a Go module to offer both a package main entry point and a library endpoint at the same time if an author decides to do so. This proposal seems to be largely motivated by doing something in spite of the author of the program it's being done to, which doesn't seem consistent with the design goals of the language so far.

For the specific use-case of "build something like BusyBox", I would suggest that the author of such a thing ought to design an explicit API through which a Go module can expose one or more entry points with a specified signature, and then encourage authors of programs that would make sense to include in such a bundle to expose those entry points. Not all authors would want to participate in such a thing, but I expect that many would if the argument for it were compelling enough.

[pdmccormick]

Should it be possible for someone to embed someone else's standalone Go program as if it were a library in someone else's Go program without first negotiating an explicit library API with the author of the first program?

Thank you @apparentlymart for the thoughtful comments. You have helped persuade me, importing main is probably not the right approach for achieving my goals here, and it is certainly not a popular option either.

In the case of the program you mentioned, the one that you did not want to expose a package-level API for: you wouldn't have any objection to it be invoked as a program though, would you? i.e. as an OS-level process, with os.Args and os.Environ, possibly stdin/stdout/stderr, with the usual Go runtime startup process that calls your main function. If it were possible to interact with that runtime machinery directly, it could be used to implement a Busybox-style multi-tool executable (as opposed to dispatching to other main's through a regular function call).

[apparentlymart]

Indeed, my concern is that running a program that expects to be in full control of its process as if it were a library inside another process would likely cause it to behave differently in that context, and that those different behaviors would then implicitly become part of the functionality that program is offering and create (possibly undesirable) pressure to keep it working in that context, despite the differences.

If you were able to create an execution environment that is completely indistinguishable from the program being launched directly as a child process then I would not consider that to be problem, but I'm not convinced that the proposal as currently written would guarantee that outcome.

From what you've described in your latest comment, I'm imagining something that is more like a replacement for the Go runtime's startup code, presumably not actually written in Go so that it can run before a Go execution environment is available, which does whatever special logic it needs to decide which of many package main to use, and only then prepares the runtime in the same way that the current implementation does, taking care to ensure that there is no way that the program being launched could observe that it's being launched by that different runtime once it's up and running.

In particular (but not limited to):

• The multiple linked programs are unlikely to agree on exactly which version of each dependency they are linking. Therefore the linker and loader would need to tolerate having multiple versions of the same symbol in the program and select the appropriate versions to dynamically link based on which specific program was selected.

  (I'm assuming it would be a goal to be able to share the code for dependencies between multiple programs in cases where they do agree, because otherwise this would not have significant advantages over just executing the target programs directly using the facilities provided by the OS, such as exec on Unix systems.)

• All of the init functions must run in exactly the same sequence as they would when the program is executed normally, including the implicit init steps that result from assigning an expression to a package-level variable.

• No additional Goroutines or OS threads are running compared to what would happen if the program were executed directly.

• panic behaves in exactly the same way when not intercepted by a recover, printing exactly the same messages to the same handles and exiting with the same exit code.

I realize that this is quite a pedantic set of requirements and that many programs would not actually be affected by differences here in practice, but nonetheless all of these things could materially affect a specific program's behavior, in ways that the author of that program may not be aware of until it's too late.

[pdmccormick]

@apparentlymart Great points, especially about the need to preserve init execution order.

Keeping with the idea of a program invocation style interface, what about a mechanism for restarting the current Go runtime into a different program? Similar to how an exec(3) syscall replaces the currently running OS level process with a new one. In the runtime package, something like func ExecProgram(package string, args, environ []string) error. The only data you can pass across the call would be values for os.Args and os.Environ (plus OS level process state such as open file descriptors and namespaces).

You raise an excellent point about multiple package versions. If a project opted-in to this kind of arrangement, and split out an importable cmd package for its programs, the module system would have to work out the dependency versioning when combining multiple cmds. There's no guarantee that a program would work with those requirements compared to its own local go.mod. If the compiler needed to build multiple versions of the same packages, the compiled code sharing and executable size savings start to go out the window, so why bother with any of this.

Assuming that the bundled programs would still function correctly when their package requirements were resolved together, specifying the programs to bundle could be provided as flags to go build:

go build -bundle pkg1/cmd/foo -bundle pkg2/cmd/bar ./cmd/gopherbox/

In this program, pkg1/cmd/foo, pkg2/cmd/bar and main could be passed to ExecProgram.

Alternatively, at the risk of circling back to this topic, maybe a special case of main importing could be permitted, not for initializing side effects but just to declare that it should be bundled into the given build:

package main

import (
	"os"
	"runtime"
	_ "cmd/go"
)

func main() { runtime.ExecProgram("cmd/go", os.Args, os.Environ()) }

This is introducing an irregularity to how imports are to be treated, and that may require a change to the Language Specification (although, as I have learned in this process, the Go language spec does not actually disallow main imports, just the standard compiler). At least this way go mod tidy and tooling would be aware of the dependencies.

[ianlancetaylor]

Note that at least on Unix systems we do have https://pkg.go.dev/syscall#Exec .

[pdmccormick]

Note that at least on Unix systems we do have https://pkg.go.dev/syscall#Exec .

If I'm not mistaken, the Go runtime does not currently have a notion of multiple main entrypoints within a single executable, or does it have something along those lines?

[ianlancetaylor]

No, it does not. It always starts at main.main. Perhaps I misunderstood the suggestion.

[pdmccormick]

No, it does not. It always starts at main.main. Perhaps I misunderstood the suggestion.

I'm sorry for jumping around with ideas in this thread, but thank you for engaging with me on this.

To summarize, I'm wondering if it would be possible to enable:

1. go build to include multiple program entrypoints, and
2. Give programs the ability to teardown the current Go runtime and restart it back up again fresh but from a different entrypoint (and with new argv/envv)

So a Go runtime level exec call, if you will, all from within the same executable and same running process.

[apparentlymart]

I think I get what you're thinking about here, and maybe the following set of analogies would both allow you to confirm whether I've understood correctly, and if so help others in this thread reach a similar point of understanding.

• kexec is a Linux kernel system call that effectively makes Linux behave as a bootloader for another version of itself, effectively replacing the running kernel with a new kernel but without the overhead of rebooting the computer. The intent is that nothing about the original kernel survives, and the new kernel believes it was booted normally.

• exec is a system call on many Unix-derived systems that effectively replaces the current process memory image with a new process image and begins executing it. The intent is that nothing about the original process image survives, and the new program believes it was started normally. (In this case, it was started normally, because exec is the normal way of starting programs!)

• The evolved form of this proposal (now quite different than the original writeup) is a Go runtime function runtime.ExecProgram that replaces the running Go program with another Go program that was somehow linked into the same executable image as the caller, with the new program believing it was launched directly.

  However, this new idea differs from the previous one by promising that if two of the programs linked into the same executable agree on a particular dependency version then that dependency will be included in the executable only once, as an executable file size optimization. So it's a sort of Go-specific exec that allows the target program(s) to be bundled inside the same executable.

  Presumably that means that this new runtime function is subsuming some of the responsibilities that would normally be handled by the linker at build time, since it'll need to decide what to map into memory and fix up calls between packages at runtime instead of at build time.

(I'm intending this comment only as an attempt to confirm that I'm understanding the new proposal correctly, not as any specific opinion on the new proposal.)

[pdmccormick]

In terms of a concrete example for this Busybox-style multitool executable, and possible storage savings:

Two Docker plugins that are typically installed together are buildx and compose (I have no affiliation with Docker). When I build each of these, the binaries are 77 MiB and 75 MiB respectively. Next, I have locally modified each of these projects to move the contents of their primary main.main into an importable subpackage, github.com/docker/buildx/cmd/buildx/buildxcmd and github.com/docker/compose/v2/cmd/composecmd. Then, in a new program I have imported both of these, and dispatch to each depending on the contents of os.Args[0]. Finally, after compiling, I created symlinks, buildx => multitool and compose => multitool, which I use to invoke the specific program. This resultant binary is only 80 MiB, for a savings of 72 MiB compared to two separate executables.

This might be a bad idea for other reasons as discussed above, but it's basically those savings that I am chasing. I will be trying to see if the authors of the particular programs relevant to me would be willing to accept that kind of main/cmd split, but the motivation for this entire proposal was, could this be supported in the toolchain itself, to allow bundling together collections of Go programs into single executables.

[pdmccormick]

I think I get what you're thinking about here, and maybe the following set of analogies would both allow you to confirm whether I've understood correctly, and if so help others in this thread reach a similar point of understanding.

Yes, you've understood me perfectly, thank you!

And you're absolutely right, the discussion has moved on quite a bit from the original proposal. I am happy to withdraw it, or have it withdrawn by someone else, if that is the appropriate course of action for this process?

[pdmccormick]

• Presumably that means that this new runtime function is subsuming some of the responsibilities that would normally be handled by the linker at build time, since it'll need to decide what to map into memory and fix up calls between packages at runtime instead of at build time.

This is a great point: if any kind of runtime patching would be required to switch between programs then I'm definitely thinking that this is starting to seem like a bad idea.

If the results were not the equivalent to what I did in my Docker example above, existing programs which are modified to explicitly opt-in by adding a boilerplate main/importable cmd split, and then having those program packages pulled into a new program as dependencies, if it would not be possible to get some level of compiler support for achieving that without the boilerplate, with the least possible changes to anything else in the ecosystem or runtime, then I think this idea should be abandoned.

[adonovan]

Importing main packages was in the distant past permitted, but the tools were changed to disallow it, and the rationale for that change applies to this proposal too.

Therefore, this is a likely decline. Leaving open for four weeks for final comments.
— adonovan for the language proposal review group

[ianlancetaylor]

No further comments.