"zig cc" doesn't work with cgo (golang) #7342
Comments
|
So this looks like go is inserting flags before it passes And then putting that as your CC. This way zig gets invoked with |
|
Downstream issue filed: golang/go#43078 |
This indeed works, but another weird thing is, that I get these errors: Unless I use sudo, which produces a working binary: |
|
That error message would be more useful if it printed the path, sorry about that. Anyway that's a separate issue. Could you try using strace to find out which file paths are getting AccessDenied? My hunch is that it is the global cache, and go is running the compiler within a chroot or something. You can try messing with these options within your wrapper script: Make sure they are writable paths. They are allowed to be the same. |
|
strace example of what went wrong (I think, never used strace before) |
My |
|
Thanks, that strace output is exactly what I needed to see. This is a bit confusing - zig successfully was able to open
Ah, sorry about that. I forgot this was the I'll look into this myself and see if I can get it working. Thanks for your help. |
Then it's just: This is because the permissions of |
|
Thanks - I'll look into a solution for this. I'm not sure why Go is preventing Zig from using a project-local directory as a cache folder. That seems like a reasonable thing for a compiler to do. But I'll add support for configuring the cache directory via an environment variable so that it can be modified when using |
|
It's not because of a project local cache folder, it's because zig tries to create a directory inside a stdlib package. |
|
I think zig cc should put all caches inside |
|
After 2759313 (which I just pushed to master) and the following #!/bin/sh
ZIG_LOCAL_CACHE_DIR="$HOME/misc/gozig" zig cc $@(where It produced a viable binary. It gave the following harmless linker warning: You could also try passing |
I reviewed the logic for choosing a local cache directory, and I do think there is room to modify it when only compiling C objects: Lines 1641 to 1668 in 2759313 You can see that in some cases, for example |
|
I tried to compile zig from the current master source code, but I got the following errors, I followed the above procedure: Logs: |
|
This looks like your LLVM, Clang, LLD libraries were compiled with a different compiler than you used to compile zig. If you are annoyed by this, welcome to the club, this is what C++ forces us to do. |
|
Is there maybe a docker container or sth, which makes this easier? |
|
https://github.com/ziglang/docker-zig I don't think you need this though. The instructions have been tested on nearly every system out there. Anyway, in an hour or two the CI will have finished preparing a master branch tarball that includes the commit. |
|
If you want to try a master branch binary tarball, the download page is now up-to-date, including the commit that adds support for environment variables to control cache directories. |
|
Thanks it works now. (GOOS being the os e.g. All possible targets for go: Example: Or is this out of scope for zig, and I should just write a wrapper? |
|
@lu4p FWIW only some of those targets are compatible with cgo: Notably, these aren't supported: |
Hmm it's a good question. I'm not opposed to adding a bit of code to help support this use case, but I want to be careful to not do anything surprising. I'm imagining a developer who uses both Go and Zig and has GOOS/GOARCH env vars set up for their go project, and then is confused when zig starts changing its target in response to what they thought was unrelated. Another idea would be introducing an environment variable |
|
I'm mostly an interested observer here, but as a motivating use case (assuming I am reading this correctly) I believe this would enable us to build cross-platform multi-arch container images that require cgo in Your idea to use tl;dr for The minimal configuration in the linked issue would possibly resolve this limitation, and I would be able to stop hedging my recommendation to use |
|
Maybe zig could just reuse the CGO_ENABLED=1 env variable. |
That doesn't solve the problem @andrewrk mentioned, which is that a Zig developer might find it surprising if Zig starts obeying Go environment variables out of the blue. It needs to be something Zig specific to kill that surprise. I guess can see both ways on this issue. On the one hand, I agree, it's surprising to anyone unaware of it, and will be infuriating to anyone who spends potentially hours debugging it. On the other hand, But I'd go with the principle of least surprise. |
|
Question - does cgo ever try to compile C++ files? Does it look at the CXX environment variables? (Zig provides this functionality as well) |
|
Yes go also uses the CXX variable for c++ code. |
|
In this case I think golang/go#43078 will be a nice improvement, because then it will respect the sub-comands For linux targets, zig provides the ability to cross compile for both (dynamic) glibc and for (static) musl, is there any go environment variable to choose one or the other? |
There isn't. |
|
Just driving by, but thought I'd share my two cents since I previously maintained the cross-compilation toolchain for CockroachDB, which I have to imagine is somewhere near the top of the "cgo (ab)user" leaderboard. The common idiom for cross-compiling Go code that needs cgo is: In other words, it's pretty well understood that you have to specify the target for every compiler involved. It's a mouthful, but it's not the hard part, and it's nicely explicit. (The hard part is installing the cross toolchain with a working sysroot, etc., and as I understand it that's the thing that Translated directly to That seems 💯 great to me and I don't personally think there's much need for an ergonomic improvement. But I guess I wouldn't mind if any of the following did some magic introspection of All contingent on something like https://golang.org/cl/276412 landing, of course.
Yeah, you have to do this manually, e.g. by using a musl-targeting toolchain and passing |
Yes, and additionally with Zig you could do instead: To get a fully statically linked binary, even if you link in C++ code. Also you can do something like this: Here I've specifically targeted glibc 2.27, presumably to match the version on the servers I want to deploy to. Yes you heard it right not only can zig cross compile for both glibc and musl it can also target any version of glibc. You can also configure CPU features with That being said...
In the interest of discussion I went ahead and made a proof-of-concept: #7377 With this you can do: And Zig infers the target from the GOOS/GOARCH environment variables. And I also threw in a workaround for golang/go#43078 for good measure. I'm not promising this is going to land in master branch but it's there if anyone wants to toy with it. |
|
I solved the cache directory issue with f7d6006. Until https://golang.org/cl/276412 lands, zig can be used with cgo using a wrapper script, as demonstrated in #7342 (comment). After that go patch lands, zig can be used with cgo without a wrapper, as demonstrated in #7342 (comment) Feel free to discuss the topic further. |
Previously, when choosing the local cache directory, if there was no root source file, an explicitly chosen path, or other clues, zig would choose cwd + zig-cache/ as the local cache directory. This can be problematic if Zig is invoked with the CWD set to a read-only directory, or a directory unrelated to the actual source files being compiled. In the real world, we see this when using `zig cc` with CGo, which for some reason changes the current working directory to the read-only go standard library path before running the C compiler. This commit conservatively chooses to use the global cache directory as the local cache directory when there is no other reasonable choice, and no longer will rely on the cwd path to choose a local cache directory. As a reminder, the --cache-dir CLI flag and ZIG_LOCAL_CACHE_DIR environment variable are available for overriding the decision. For the zig build system, it will always choose the directory that build.zig is + zig-cache/. Closes #7342
|
I cherry-picked the new cache directory logic so that it will be available in the upcoming 0.7.1 release: 21236c0 |
Previously, when choosing the local cache directory, if there was no root source file, an explicitly chosen path, or other clues, zig would choose cwd + zig-cache/ as the local cache directory. This can be problematic if Zig is invoked with the CWD set to a read-only directory, or a directory unrelated to the actual source files being compiled. In the real world, we see this when using `zig cc` with CGo, which for some reason changes the current working directory to the read-only go standard library path before running the C compiler. This commit conservatively chooses to use the global cache directory as the local cache directory when there is no other reasonable choice, and no longer will rely on the cwd path to choose a local cache directory. As a reminder, the --cache-dir CLI flag and ZIG_LOCAL_CACHE_DIR environment variable are available for overriding the decision. For the zig build system, it will always choose the directory that build.zig is + zig-cache/. Closes ziglang#7342
|
I developed a small tool to automatically judge the compilation target based on the GOOS GOARCH environment. |
Thanks a lot for this! |
For those unaware, I believe golang/go@742dcba replaced https://golang.org/cl/276412 to fix golang/go#43078. |
|
When I cross compile from linux to macos, compile error error see https://github.com/akkuman/rotateproxy/runs/4863624753?check_suite_focus=true |
|
After reading this, I am not sure, what the current status is. Should it be possible to use cgo with zig without a wrapper script now? I was able to cross compile a go app from linux to macos without a wrapper script using zig 0.11.0. But it stopped working with zig 0.12.0 or 0.13.0. I get the error: The command I am running (that works with zig 0.11.0) is: Should this work with newer versions of zig? Should I create a separate issue for that? |
|
@ostcar yes it should work with Go >= 1.18, please open a separate issue. |
I would like to use "zig cc" for easily cross compiling cgo dependent go code, however the "drop-in" c compiler replacement doesn't work.
Example cgo code
main.go
other compilers
zig cc
The text was updated successfully, but these errors were encountered: