cmd/go: go install cmd@version errors out when module with main package has replace directive

[james-lawrence]

What version of Go are you using (go version)?

go version
go version go1.16 linux/amd64

Does this issue reproduce with the latest release?

yes

What did you do?

cd /tmp; go install bitbucket.org/jatone/genieql@latest

What did you expect to see?

go install finishing successfully

What did you see instead?

go install bitbucket.org/jatone/genieql@latest: bitbucket.org/jatone/genieql@v0.0.0-20210306180040-be9c7b630c18
	The go.mod file for the module providing named packages contains one or
	more replace directives. It must not contain directives that would cause
	it to be interpreted differently than if it were the main module.

given that genieql is the main module this error appears to be erroneous.

other attempts I made all of which i would have expected to work properly (and use to):

cd /tmp; go install bitbucket.org/jatone/genieql/...@latest
go install bitbucket.org/jatone/genieql/...@latest: bitbucket.org/jatone/genieql@v0.0.0-20210306180040-be9c7b630c18
	The go.mod file for the module providing named packages contains one or
	more replace directives. It must not contain directives that would cause
	it to be interpreted differently than if it were the main module.

cd /tmp; go install bitbucket.org/jatone/genieql/cmd/...@latest
go install bitbucket.org/jatone/genieql/cmd/...@latest: bitbucket.org/jatone/genieql@v0.0.0-20210306180040-be9c7b630c18
	The go.mod file for the module providing named packages contains one or
	more replace directives. It must not contain directives that would cause
	it to be interpreted differently than if it were the main module.

cd /tmp; go install bitbucket.org/jatone/genieql/cmd/genieql@latest
go install bitbucket.org/jatone/genieql/cmd/genieql@latest: bitbucket.org/jatone/genieql@v0.0.0-20210306180040-be9c7b630c18
	The go.mod file for the module providing named packages contains one or
	more replace directives. It must not contain directives that would cause
	it to be interpreted differently than if it were the main module.

[seankhliao]

L28 contains a replace directive

replace github.com/containous/yaegi => github.com/james-lawrence/yaegi v0.8.8-modules-enh

Which is not valid for go install

No module is considered the "main" module. If the module containing packages named on the command line has a go.mod file, it must not contain directives (replace and exclude) that would cause it to be interpreted
differently than if it were the main module

Closing as working as intended

[james-lawrence]

yes obviously line 28 contains the directive. see the title. and then see this thread where I was told to open this issue.

[james-lawrence]

that would cause it to be interpreted differently than if it were the main module

is the relevant line of that statement here. the fact is this directive wouldn't cause it to be interpreted differently than if it was the main module. and to any outside viewer, there is a main module, its the one being installed.

[seankhliao]

reopening, I would recommend rephrasing the issue in terms of a proposal to relax the behaviour rather than as a bug report

[james-lawrence]

@seankhliao I'm just not sure if it is a bug or a proposal, the discussion around go installs (new) behavior was long, varied, and confused. =) this particular problem was pointed out during that discussion and seemingly ignored/disregarded/left open to be revisited. so here we are.

[myitcv]

@james-lawrence

this particular problem was pointed out during that discussion and seemingly ignored/disregarded/left open to be revisited.

#40276 was indeed discussed at length. Indeed, over the years even prior to that specific proposal, this issue has been discussed in various forms, many times. To say that the question of replace directives was ignored/disregarded is incorrect. Indeed I specifically linked to a quote that pointed out the option for that particular point to be revisited. It suffices therefore to open an issue, as suggested, to start that discussion.

Deferring to @jayconrod on how best to take the discussion forward.

[james-lawrence]

I meant it was at one of those 3 at various points in that (and the related) thread. I wasn't calling out any particular stage, sorry for that confusion.

[ianthehat]

I think there may be confusion around the concepts of a main module and the package main.
The main module is normally the one in which the go command was invoked (no matter what packages are being built or installed). Modules cannot be installed, only packages. When building in package@version mode there is no main module.

In the case in this issue genieql is the package main, and the way in which it would be built changes if the module of which it is a part is the main module because of the replace statement.
For example, if you did go install bitbucket.org/jatone/genieql in another module that had bitbucket.org/jatone as a dependency it would not obey the replace directive. If you checkout bitbucket.org/jatone and do go install ./genieql then it will obey the replace directives and produce a different result. It would also obey the versions specified in the main module, which would change the results even further.

Also note that it is not something that used to work, it is a brand new feature that limits the number of cases in which it is enabled.

Maybe we should change the terminology for main module to a different word?

[james-lawrence]

Also note that it is not something that used to work

while technically true go get -u did work because it respected the vendor directory: go114 get -u bitbucket.org/jatone/genieql-go114/cmd/... see related issue #44841 and #40053 + the google groups thread in that issue.

cat ~/go/src/bitbucket.org/jatone/genieql-go114/vendor/github.com/containous/yaegi/interp/src.go | grep -i 'func pkgDir(ctx \*build.Context'

returns the patched function from the replace + vendor workflow, and if you started with a clean environment there would be no other sources of the library in question.

now did it work if trying to use a package as a dependency? no, and that was fine because it wasn't about using it as a library. many go programs are never intended to be used that way.

now the rest of your argument is attempting to explain away for current limitations after the fact that go get / go install commands were fundamentally changed to support go modules while simultaneously ignoring vendor workflows of the tools and making that argument with a distinction that never existed until the tools were changed.

now all that being said: installing a binary is different from managing dependencies. I believe this is an accepted fact and a reason go get was changed to be about managing dependencies and go install is being shift towards building and installing binaries. this fundamentally is a good change and I'm wholeheartedly behind it.

I think the question we need to be asking is if the distinction between main package of the main module during installation matter? I'm going to assert it does not and that this error is erroneous, but during dependency management where someone tried to add a dependency that had a replace directive this error would be 100% correct though I'd start out with a warning in that case vs a hard error.

[james-lawrence]

Ah I believe I have a good explanation about why this should be supported.

the replace directive is fundamentally about compilation, its meant to inform the build system how to patch dependencies. this is true both during development where you may need to patch them while debugging a problem which is when you compile your program. and during installation of a main module (previously the main package) onto a system.

so during compilation there is no distinction between the main package and the main module, however, during dependency management replace directives have always been ignored and any software that was relying on a dependency where there was a replace directive was broken (conceptually if not in practice). and this is where we should be erroring out (i.e. in go get in this new world) when we see a replace directive. not during compilation (go install).

[jayconrod]

Closing this issue since it's working as described in #40276.

That being said, at some point in the future when we have more information, it will be reasonable to open a proposal to either ignore or apply replace and exclude directives. That proposal should be based on the discussion in #40276 (and previously #30515), it must include data backing up whichever position is supported, and it must consider tradeoffs for the other side.

I know this is a lot to ask (those issues are very long), but the main point of disagreement was what to do with replace directives, and we debated that for a year and a half. The outcome was a compromise: replace directives are not allowed for now, and we have the option of changing that in the future without making an incompatible change.

To restart that discussion, we need new information: how is the new go install cmd@version form being used, and would replace directives be helpful or harmful?

[ncw]

@jayconrod wrote:

To restart that discussion, we need new information: how is the new go install cmd@version form being used, and would replace directives be helpful or harmful?

As a go user, I reasonably expect go install cmd@version to just work and install the binary cmd using the go.mod the authors of that package spent time getting exactly right.

Instead I get an extremely cryptic message like this

go install: github.com/rclone/rclone@latest (in github.com/rclone/rclone@v1.57.0):
	The go.mod file for the module providing named packages contains one or
	more replace directives. It must not contain directives that would cause
	it to be interpreted differently than if it were the main module.

As a user of said cmd, I have no idea how to fix that problem.

From the instructions for go install cmd@version: https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies need amending

No module is considered the "main" module. If the module containing packages named on the command line has a go.mod file, it must not contain directives (replace and exclude) that would cause it to be interpreted differently than if it were the main module. The module must not require a higher version of itself.

Which says the same thing as the error message, more or less.

So it is consistent, but not helpful in my opinion!

It is very hard having a large open source project without a few soft forks while you are waiting for upstreams to take your patches.

I first noticed this problem with github's gh tool, and I think you'll find that there are lot more high profile Go binaries out there with this problem.

$ go install github.com/cli/cli/cmd/gh@latest
go install: github.com/cli/cli/cmd/gh@latest (in github.com/cli/cli@v1.14.0):
	The go.mod file for the module providing named packages contains one or
	more replace directives. It must not contain directives that would cause
	it to be interpreted differently than if it were the main module.

[rsc]

Someone flagged recent discussion on this closed issue to me privately. As a general note, we don't watch closed issues very carefully - they're closed.

Replace directives remain for local development. If you want to post software that others can go install, it needs to build without the replace directives. If we respected the replace directives during go install foo@latest, then that would create commands that can be installed that way but not using 'go get foo; go install foo' in a module where you are trying to track the version of foo you are using. We are also discussing integrating tools more explicitly with a module in #48429, and again there it would be a serious problem if tools existed that worked with 'go install foo@latest' but not with a regular build graph from another module. (And it is a non-starter for one tool in your module to force the use of its own replace directives. What if you don't want those replace directives? Or what if different tools have conflicting replace directives?)

If you are using dependencies that don't function without replace directives, that may be a sign to rethink the use of those dependencies, or to upstream fixes that make the replace directives unnecessary.

For the specific case of go4.org/unsafe/assume-no-moving-gc, there are two better answers than replace:

1. (Short-term) Run 'go get go4.org/unsafe/assume-no-moving-gc@latest' in your own module, which will update beyond the version in your dependency, to get a copy that works with the latest version of Go.
2. (Long-term) Upstream fixes to what is using that package to not use, or upstream fixes to that package to not break at each new Go release.

These work within the ecosystem and keep it healthy. Replace directives fragment the ecosystem and make it brittle. My post The Principles of Versioning in Go has a worked example. It looks at forced package downgrades, but replaces are analogous andhave all the same problems.

We are aware of the problem of go4.org/unsafe/assume-no-moving-gc specifically. In May we worked with the upstream author on a change that makes it no longer break at each new Go release. So if you run 'go get go4.org/unsafe/assume-no-moving-gc@latest' in your own module one last time, you should stop seeing breakages from that package, and you can remove the replace.

[maxb]

Hi @rsc,

Thankyou for the above, but it seems to avoid addressing the key use case I and others in this issue seem to want:

The particular problem is with the statement

Replace directives remain for local development. If you want to post software that others can go install, it needs to build without the replace directives.

which ignores a somewhat common practice:

• Some project is shipping a CLI executable written in Go.
• They discover some bug in a third party dependency. They fork it, fix the bug, and use a replace directive to use the forked version until a formally released fix is available.
• This works perfectly fine in most ways, as being a CLI executable, never depended upon as a library, their module is always the main module, so can use replace directives...
• ...except when users try to use go install foo/bar/baz@version

It seems like you would class this as a misuse of replace directives based on what you said above?

The problem with that, is that the Go project has given developers an almost perfect tool - replace directives - for dealing with this, so it is unrealistic to think people would not use replace directives in this scenario. AFAIK there are no other options that don't involve needing to change import paths of the forked software in every source file that consumes it, which is inelegant in the extreme!

It's not like the users trying to use go install can go to the developers using the replace directives and ask them not to use them - they'll rightly point out that extreme inelegancy in forking without using replace directives, and decline to do that.

So you end up with unhappy users who have had the lovely convenience of go install dangled in front of them, and then snatched away.

[mitar]

If we respected the replace directives during go install foo@latest, then that would create commands that can be installed that way but not using 'go get foo; go install foo' in a module where you are trying to track the version of foo you are using.

@rsc: My understanding of this issue is that we care about the use case where you use go install outside of a go module. I think what we really are asking for is that go install should just be the same as doing git clone + go install inside the cloned repository. I think this is a well defined behavior. No surprises there. It would make the use case where you fix the program in your fork and wait for upstream to merge it possible.

Installing inside a go module is tricky as you describe. And there is no need to support it there because inside a go module you can just modify go.mod yourself and add the replace directive there.

So, could we get go install to work outside of go modules with repositories which have replace directives?

[tjsampson]

Some more anecdotal evidence as to why this is an issue:

We are using the k8s client libs, but recently this change was introduced in the openAPI Spec models:
kubernetes/kube-openapi#402
kubernetes/kube-openapi@38767cd

Now we can't upgrade k8s.io/api or k8s.io/client-go because of the model mismatches:

# k8s.io/client-go/applyconfigurations/meta/v1
vendor/k8s.io/client-go/applyconfigurations/meta/v1/unstructured.go:64:38: cannot use doc (variable of type *"github.com/google/gnostic/openapiv2".Document) as *"github.com/google/gnostic-models/openapiv2".Document value in argument to proto.NewOpenAPIData

So we are left with:
1.) Revert (and wait for transient patches)
- kubernetes/kubernetes#118340
- kubernetes-sigs/cli-utils#625
2.) Run a fork (and make the patch ourselves) 😱
3.) Use a replace directive

replace k8s.io/kube-openapi => k8s.io/kube-openapi v0.0.0-20230501164219-8b0f38b5fd1f

Being a Lazy (Pragmatic...¯_(ツ)_/¯) Programmer, I chose option 3, as I think most Go Devs would. Now I can upgrade the module I want, without any of the side effects. It just works.

...But now we can't go install the code because of the replace directive. Which is kind of annoying. I would expect go install to honor the replace directive, no different than if I clone the entire repo down and run go install from there.

I would also argue that although the replace directive may have been intended for local development purposes, it is widely used across go projects to circumvent issues exactly like the one I posted here. Dependencies can get messy, and being able to "strategically" use a replace directive when necessary is a nice way to work around (hack) those issues.

[aarzilli]

Regarding replace directives being "only for local development" reading back to the original vgo and modules proposals I find this about replace.

From vgo tour

If you do identify a problem in a dependency, you need a way to replace it with a fixed copy temporarily. Suppose we want to change something about the behavior of rsc.io/quote. Perhaps we want to work around the problem in rsc.io/sampler, or perhaps we want to do something else. (...)

From Minimal Version Selection

Minimal version selection is very simple. It achieves simplicity by eliminating all flexibility about what the answer must be: the build list is exactly the versions specified in the requirements. A real system needs more flexibility, for example the ability to exclude certain module versions or replace others.

(emphasis mine)

also from Minimal Version Selection, section Who controls your build?

The dependencies of a top-level module must be given some control over the top-level build. B 1.2 needs to be able to make sure it is built with D 1.3 or later, not with D 1.2. Otherwise we end up with the current go get's stale dependency failure mode.
(...)
In the design of module requirements, exclusions, and replacements, I've tried to balance the competing concerns of allowing dependencies enough control to ensure a succesful build without allowing them so much control that they harm the build. Minimum requirements combine without conflict, so it is feasible (even easy) to gather them from all dependencies. But exclusions and replacements can and do conflict, so we allow them to be specified only by the top-level module.

From the modules proposal:

This proposal aims to balance allowing dependencies enough control to ensure a successful build with not allowing them so much control that they break the build. Minimum requirements combine without conflict, so it is feasible (even easy) to gather them from all dependencies, and they make it impossible to pin older versions, as Kubernetes does. Minimal version selection gives the top-level module in the build additional control, allowing it to exclude specific module versions or replace others with different code, but those exclusions and replacements only apply when found in the top-level module, not when the module is a dependency in a larger build.

A module author is therefore in complete control of that module‘s build when it is the main program being built, but not in complete control of other users’ builds that depend on the module. I believe this distinction will make this proposal scale to much larger, more distributed code bases than the Bundler/Cargo/Dep approach.

(emphasis mine)

From the Q&A section of the modules proposal:

How does one patch a deep dependency without vendoring?

Response [#24301 (comment) by @kardianos.] By using a replace directive.

PS. I couldn't find any mention in the proposal or in the vgo posts about replace being only appropriate for local builds but AFAICT the proposal did not introduce the distinction between local vs non-local builds, AFAICT that was initially introduced as a consequence of implementation details of go get.

[seanbethard]

So what do you do if you encounter a replace? lol

I just installed go for the first time to use godo.

go get github.com/digitalocean/godo@v1.106.0

go: go.mod file not found in current directory or any parent directory.
	'go get' is no longer supported outside a module.
	To build and install a command, use 'go install' with a version,
	like 'go install example.com/cmd@latest'
	For more information, see https://golang.org/doc/go-get-install-deprecation
	or run 'go help get' or 'go help install'.

go install github.com/digitalocean/godo@v1.106.0

go: downloading github.com/digitalocean/godo v1.106.0
go: github.com/digitalocean/godo@v1.106.0 (in github.com/digitalocean/godo@v1.106.0):
	The go.mod file for the module providing named packages contains one or
	more replace directives. It must not contain directives that would cause
	it to be interpreted differently than if it were the main module.

I don't get it. (get it?) What's go.mod's deal anyway?

[paralin]

@seanbethard you must git clone that project and run "go install ." within the directory.

[DeedleFake]

@seanbethard you must git clone that project and run "go install ." within the directory.

This would be a lot less of a problem if the go command

1. Told you to do this when you ran into the error.
2. Had a command to clone the repo at a given version into a subdirectory of the current directory.

It would still be a problem, but it would be significantly less of one.

[icholy]

I think this bug intentionally exists to prevent people from leaning on replace directives instead of upstreaming their fixes.

[paralin]

As someone who is using a lot of replace directives, this is a quite annoying limitation for go install.

Example: https://github.com/aperturerobotics/bifrost#examples

Here I have to say - well, you could use go install, but we have replace directives, so instead clone it and build. It defeats the purpose of go install.