Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go mod -vendor prunes non-package directories #26366

Closed
ainar-g opened this issue Jul 13, 2018 · 57 comments
Closed

cmd/go: go mod -vendor prunes non-package directories #26366

ainar-g opened this issue Jul 13, 2018 · 57 comments
Labels
modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.

Comments

@ainar-g
Copy link
Contributor

ainar-g commented Jul 13, 2018

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

go version devel +8a33045 Fri Jul 13 03:53:00 2018 +0000 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ainar/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/ainar/go"
GOPROXY=""
GORACE=""
GOROOT="/home/ainar/go/gotip"
GOTMPDIR=""
GOTOOLDIR="/home/ainar/go/gotip/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build865981061=/tmp/go-build -gno-record-gcc-switches"

What did you do?

#!/bin/sh
set -e
export GO111MODULE=on
mkdir foo
cd foo
cat <<EOF > main.go
package foo // import "foo"

import "github.com/gen2brain/aac-go/aacenc"

func f() { _ = aacenc.VoPidAacMdoule }
EOF
$GO mod -init
$GO mod -vendor
$GO build -getmode vendor

What did you expect to see?

Successful build.

What did you see instead?

# github.com/gen2brain/aac-go/aacenc
vendor/github.com/gen2brain/aac-go/aacenc/aacenc.go:4:19: fatal error: voAAC.h: No such file or directory
 //#include "voAAC.h"
                   ^
compilation terminated.

The problem is that go mod -vendor pruned the external/aacenc folder, which contains the C code needed to build this package.

@myitcv myitcv added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. modules labels Jul 13, 2018
@bcmills
Copy link
Contributor

bcmills commented Jul 13, 2018

The cause here seems to be that we prune out directories that are not Go packages.

There is no voAAC.h in github.com/gen2brain/aac-go/aacenc/, but I do see one in github.com/gen2brain/aac-go/aacenc/external/aacenc/include/.

@bcmills bcmills changed the title cmd/go: go mod -vendor prunes C files cmd/go: go mod -vendor prunes non-package directories Jul 13, 2018
@kardianos
Copy link
Contributor

Side note from govendor. By default govendor only picks up pkgs referenced, but I had to add an option to grab the whole sub tree for this very reason.

Mostly cgo deps, but sometimes other required resources.

...

Right now I'm considering making a tool that can be run after go mod -vendor to trim unused packages. But I can't leave what isn't there.

@myitcv
Copy link
Member

myitcv commented Jul 14, 2018

I wonder whether in the world of modules what some/most people are actually intending to do in this situation (namely where they think they want/need to go mod -vendor) is snapshot the current module dependencies as opposed to the packages. Doing so would seem to address all of these points?

So then a go mod -modvendor (command/flags to be 🚲-shed-ed) would create a directory (say ./modvendor) that could then be used by GOPROXY=file:///path/to/modvendor.

That said, I don't use go mod -vendor and likely never will. But I do maintain an append-only cache of the modules that my CI then uses via GOPROXY=file://...

@ainar-g
Copy link
Contributor Author

ainar-g commented Jul 14, 2018

I don't think anyone actually wants to vendor everything. There are two problems here, that I think should not be mixed:

  1. Non-Go code dependencies vendoring. This is what this issue is about. Whether we like it or not, Go has CGo. And C dependencies are hard. Some approaches to solve this problem (excluding the painful choice of teaching the go tool about C dependencies... the horror):

    • Let module authors declare their non-Go code dependencies in their go.mod. Something like

      external "github.com/gen2brain/aac-go/aacenc/external/aacenc"
      
    • Define and document a convention for such dependencies (e.g. external directory, similar how we have conventions for internal and testdata).

    • Explicitly give up and document that CGo is fundamentally incompatible with go mod -vendor. After all, even if you do have the C code in your vendor, there are no guarantees that that C code doesn't have some dependencies of its own, so hoping that /vendor will save you from build failures is rather naïve.

  2. Binary tool dependency vendoring. This may or may not be a problem Go modules want to solve. @rsc said it in x/vgo: Support depending on binary modules #24051.

    It is not a goal to expand into "linux distribution" territory.

    But then this, too, should be documented explicitly. It would be a shame though, as AFAIK other systems, like Ruby's Bundler and JS's NPM support some version of it.

@kardianos
Copy link
Contributor

kardianos commented Jul 14, 2018

One way to solve the cgo problem is to require a dummy go file in the c folder that is then imported by another package that uses the c files with a "_" import.

I think this is only a problem because the c and h files live in a separate directory from the go files.

@rsc
Copy link
Contributor

rsc commented Jul 17, 2018

Sorry, but the model for go builds is that what's needed to build a package is in that directory, not subdirectories. The way gen2brain/aac-go is structured, modifying the C source code in the subdirectories will not trigger a rebuild. Instead your build will use stale cache entries. The C code needs to be moved up to the package directory to make it build properly, not to mention vendor properly.

If anyone wants to build a tool that adds to the vendor directory after go mod -vendor, you could do that by reading the modules.txt file and using commands like go list -f '{{.Dir}}' to find out where each package was copied from.

@rsc rsc closed this as completed Jul 17, 2018
@ainar-g
Copy link
Contributor Author

ainar-g commented Jul 17, 2018

@rsc I see two opportunities for improvement here. Firstly, should we explicitly clarify in the cgo documentation that the go tool notices only changes in Go packages? Right now the wording doesn't mention other directories and local includes at all. Something like

<...> Any .h, .hh, .hpp, or .hxx files will not be compiled separately, but, if these header files are changed, the C and C++ files will be recompiled. Changes in files included from other directories will not trigger a rebuild. <...>

(Emphasis = added by me.)

Secondly, should such local includes be a vet warning? There is already -cgocall, why not -cgoinclude?

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/125297 mentions this issue: cmd/cgo: document that #including source files in subdirectories is a bad idea

gopherbot pushed a commit that referenced this issue Jul 28, 2018
… bad idea

Suggested in discussion on #26366.

Change-Id: Id9ad2e429a915f88b4c4b30fc415c722eebe0ea4
Reviewed-on: https://go-review.googlesource.com/125297
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@pkieltyka
Copy link

this tool might be helpful to others: https://github.com/goware/modvendor — I wrote it to easily copy .c, .h, .s, .proto files from module pkgs into my local ./vendor. Feel free to improve :)

@im-kulikov
Copy link
Contributor

go mod vendor drops github.com/ethereum/go-ethereum/crypto/secp256k1/libsecp256k1 files 😔

@myitcv
Copy link
Member

myitcv commented Sep 11, 2018

I've just added a "proposal" under #27618 (not marked as a proposal because the modules stuff is experimental, so more a discussion issue than anything). Apologies for the modvendor name clash @pkieltyka, I just spotted that.

@nomad-software
Copy link

nomad-software commented Sep 24, 2018

I can't believe the proposed solution to this issue is to write another tool to finish what go mod vendor starts. Either write a tool to replace it or fix it so it copies everything. What is the reason for only vendoring half of the repository?

@myitcv
Copy link
Member

myitcv commented Sep 24, 2018

@nomad-software Russ outlined a response to this in #26366 (comment).

If instead you want to "vendor" the module that will give you everything. That's the subject of #27618 and a proposed approach is outlined at https://github.com/myitcv/go-modules-by-example/blob/master/012_modvendor/README.md.

@thepudds
Copy link
Contributor

@nomad-software this issue here was initially reported when the original reporter was using cgo.

Are also hitting this with cgo, vs. are you hitting this without using cgo?

If you are hitting this with cgo, and if we set aside vendoring for a moment, do you think you might be hitting problems even just during non-vendor-based building/recompiling due to the behavior described here:

https://golang.org/cmd/cgo/

When the Go tool sees that one or more Go files use the special import "C", it will look for other non-Go files in the directory and compile them as part of the Go package.
...
Note that changes to files in other directories do not cause the package to be recompiled, so all non-Go source code for the package should be stored in the package directory, not in subdirectories.

(It is worth reading the full description at that link, but wanted to include at least a snippet for convenience)

@nomad-software
Copy link

nomad-software commented Oct 22, 2018

go mod vendor drops github.com/ethereum/go-ethereum/crypto/secp256k1/libsecp256k1 files

@im-kulikov I've written a tool to fully copy the entire directory of all dependencies into the vendor folder. You can find it here:

https://github.com/nomad-software/vend

Any suggestions to improve it are welcome, just open an issue there.

@dolmen
Copy link
Contributor

dolmen commented Feb 20, 2022

Looking #26366 (comment), it seems that using a tree of C files will not work correctly.

But I still have to use code from a library that is organized as a tree. So I need to flatten that tree as files in my package directory.

Has anyone built a tool that will take a C file and resolve and inline all #include that are relative to a base directory? This is a subset of what a preprocessor does...

dolmen added a commit to dolmen-go/hid that referenced this issue Feb 20, 2022
There is a rule for go build: all sources files of a given package must
be in the directory of that package. Not in subdirectories.
Which means that {#include "hidapi/mac/hid.c"} is not correct as hid.c
is not in the package directory, so "go build" will not notice its
changes and go mod -vendor will not work (golang/go#26366).

So this patch creates 3 files (hidapi_linux.h, hidapi_windows.h,
hidapi_mac.h) which are go-generated by internal/gen.go. The generator
just inlines any includes from the libusb/ and hidapi/ directories.
This means that the libusb/ and hidapi/ are not needed anymore for
distribution (and for go mod -vendor): they are used only for the "go
generate" phase.
dolmen added a commit to dolmen-go/hid that referenced this issue Feb 20, 2022
There is a rule for go build: all sources files of a given package must
be in the directory of that package. Not in subdirectories.
Which means that {#include "hidapi/mac/hid.c"} is not correct as hid.c
is not in the package directory, so "go build" will not notice its
changes and go mod -vendor will not work (golang/go#26366).

So this patch creates 3 files (hidapi_linux.h, hidapi_windows.h,
hidapi_mac.h) which are go-generated by internal/gen.go. The generator
just inlines any includes from the libusb/ and hidapi/ directories.
This means that the libusb/ and hidapi/ are not needed anymore for
distribution (and for go mod -vendor): they are used only for the "go
generate" phase.
dolmen added a commit to dolmen-go/hid that referenced this issue Feb 20, 2022
To allow 'go mod -vendor' to take directories, we create dummy packages
in each directory.

Basically it does:
for d in $(find internal -type d); do echo "package $(basename $d)" >"$d/pkg.go"; done

A script is used to maintained those dummy package besides hidapi/libusb
upgrades: internal/dummy-packages.sh

See:
- karalabe#31
- golang/go#26366
@dolmen
Copy link
Contributor

dolmen commented Feb 20, 2022

Here is a workaround that may be useful for other projects:

For package github.com/dolmen-go/hid (which embeds C libraries libusb and hidapi) I've implemented an ad-hoc C preprocessor (triggered by go generate) that concatenates all required C files from the subdirectories and produces 3 .h files (one for each supported platform). The cgo source just includes the the local .h file corresponding to GOOS.

@williamh
Copy link

All,

I'm a packager on Gentoo, and I'm hitting this with https://git.sr.ht/~rjarry/aerc.git.

I am required to vendor to build packages under our package manager, and it doesn't look like that is going to change any time soon. Our package manager team sees allowing me to reach out to the network during a build as a massive security hole they don't want to open, and they tell me that having builds that are required to reach out to the network breaks building on unconnected systems.

So, what is the best way to deal with this issue?

Thanks,

William

@williamh
Copy link

I have packaged the vend tool from @nomad-software on Gentoo, thanks for writing it.

IMO this is a pretty significant issue since go mod vendor creates vendor directories that are unusable in some situations.

We will use vend for now, but please count me in as someone requesting that this discussion be restarted.

Thanks much for your time,

William

@xuyang2
Copy link

xuyang2 commented Mar 30, 2022

chenyt9 pushed a commit to MotorolaMobilityLLC/external-libcap that referenced this issue May 6, 2022
Remove psx_pthread_create() from libpsx - given the way -lpsx is
linked this is not needed.

Also, as pointed out by Lorenz Bauer, "go mod vendor" support
was unable to vendor a copy of psx_syscall.h because it didn't
reside in the same directory as the *.go code for the psx package.
(General discussion golang/go#26366 .)
Given that we can, avoid the use of a sub-directory in the libcap
tree.

Signed-off-by: Andrew G. Morgan <morgan@kernel.org>
@acestronautical
Copy link

Why is this closed? This is still a major issue. Go dep was able to handle this with "unused-packages = false" which means many cannot and will not switch over to go mod.

@rsc rsc removed their assignment Jun 23, 2022
tagatac added a commit to tagatac/bagoup that referenced this issue Jul 14, 2022
Use wkhtmltopdf for PDF support because it has the easiest emoji solution (via twemoji).

heic2jpg.go is untested as it is largely copied from upstream, and I have a PR open to create an interface upstream: adrium/goheif#2

goheif also has issues vendoring because there is no Go code in some of the required C++ subdirectories: golang/go#26366. This should also be fixed upstream, but as a stopgap, we need to use the cp and chmod commands (vendor makefile target).

* Enable output to PDF

* Add support or jpegs in PDFs

* Switch to wkhtmltopdf for emoji support

* Implement copying attachments

* Convert HEIC attachments to JPEG for PDF output

* Update example to match new defaults

* Add "attached" text to attachment references in outfiles

* Update dependencies

* Install wkhtmltopdf in Travis CI

* Use xvfb in Travis CI
@bcmills
Copy link
Contributor

bcmills commented Mar 21, 2023

Why is this closed?

See #26366 (comment).

@golang golang locked as resolved and limited conversation to collaborators Mar 21, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests