cmd/ld: invoke host linker for Windows cgo build

[rsc]

It should be possible to invoke the host linker for a cgo build.
That is, we'd run 6l to write a .o file for the Go pieces and then invoke gcc to do the
final link with the .syso files. This would fix many corner cases in 6l's support for
linking arbitrary ELF objects into the binary.

I would like to keep 6l's arbitrary ELF support so that one need not have a C compiler
installed to use a Go compiler, but we need the 'invoke gcc' plan as a fallback.

6l = 6l, 5l, 8l here, to avoid confusion with host ld.

[rsc]

Comment 1:

Issue #3591 has been merged into this issue.

[rsc]

Comment 8:

Labels changed: added size-l.

[rsc]

Comment 9:

Labels changed: added size-xl.

[rsc]

Comment 10:

Labels changed: removed size-l.

[rsc]

Comment 12:

Issue #4272 has been merged into this issue.

[rsc]

Comment 13:

Issue #1781 has been merged into this issue.

[rsc]

Comment 14:

Issue #3141 has been merged into this issue.

[rsc]

Comment 15:

Issue #3755 has been merged into this issue.

[rsc]

Comment 16:

An update: the beginning of this is implemented for darwin, linux, and *bsd on amd64 and
386. You can play by invoking the go build of a command with -ldflags=-hostobj but if it
breaks you get to keep both pieces.
Dwarf information is not correct, so you might need -ldflags='-w -hostobj' depending on
how paranoid the system linker is.
Windows has not been attempted yet.
ARM has not been attempted yet.
There is a real possibility that ARM and Windows will be skipped for this go-round.

[minux]

Comment 17:

how much remains to be done for cmd/5l?

[rsc]

Comment 18:

It's unclear to me how much is left in cmd/5l. The 6l and 8l linkers already had a
concept of a relocation and recorded the places that needed to be relocated, as part of
the standard linking, so that individual functions could be laid out separately and put
together only at the end. Adjusting them to emit the not-applied relocations was fairly
straightforward. 
5l does not have that flexibility: it lays out the entire code segment at once, because
I never changed it like I did 6l and 8l. So the required changes are more spread out
than in 6l/8l. Any part of the code layout in asm.c that uses ->pc or ->value is
suspect. However, the simplicity of the ARM instruction set might make it easier to find
these.
At the least, the two 'TODO: Use addrel' comments need to do that, although the one for
UNDEF could be changed into two instructions instead (MOVW $0, R0; BL R0). Also any case
listed in the table for C_SBRA or C_ADDR arguments, so cases 5, 11, 63, 64, 65, 68, 69,
74, 93, 94. It could be that most of the work is in addpool. It's all very unclear to me.

[rsc]

Comment 19:

Issue #4782 has been merged into this issue.

[eliasnaur]

Comment 20:

What is the plan for the ELF TLS sections? I have a test here that verifies whether
__thread variables work in foreign code:
https://golang.org/cl/7578043/
It currently crashes, presumably because TLS sections are missing from the -hostobj
generated executable.

[ianlancetaylor]

Comment 21:

As far as I can see, the gc ld is going to have to create a SHT_TLS section and
appropriate TLS relocations.  I don't really see a way around that.

[eliasnaur]

Comment 22:

Does that imply the relatively expensive call to __tls_get_addr in every stack size
prelude in many functions? I was hacking on support for generating .so files from Go,
but got stuck on this expensive need.

[ianlancetaylor]

Comment 23:

Calling __tls_get_addr would be required when compiling in PIC mode for code to go into
shared libraries.  However, that is not the normal mode and I don't see why we should
support it to start.  When not compiling PIC and not putting the code into a shared
library there is no need to call __tls_get_addr.  Instead, 6l would generate a
R_X86_64_TPOFF32 reloc on a "mov %fs:0,REG" instruction.  8l would generate a
R_386_TLE_LE reloc on a "mov %gs:0,REG" instruction.  Not sure about 5l; we may need a
__aeabi_read_tp call there for ARMv5.  For sufficiently new ARM processors we can use an
mrc instruction to get the value and a R_ARM_TLS_LE32 reloc to get the offset.

[rsc]

Comment 24:

I control the link order, so I am hoping I can just pass the files to gcc
in the right order and keep my current slots without any relocations. :-)

[ianlancetaylor]

Comment 25:

ELF TLS is now handled correctly on i386 and amd64.
ARM and Windows are still not done.

[DanielMorsing]

Comment 26:

Running the host linker means that the gdb python script automatically loaded.
Anyone know how to load it manually?

[ianlancetaylor]

Comment 27:

I'm not sure I entirely understand the question, but in gdb you can
type "source go/src/pkg/runtime/runtime-gdb.py".

[DanielMorsing]

Comment 28:

Oh man, I messed up that sentence.
When host linking, the .debug_gdb_scripts symbol isn't emitted, so the gdb script isn't
automatically loaded.
Manually loading the script complains about a missing type symbol with the following
error:
Loading Go Runtime support.
Traceback (most recent call last):
  File "../go/src/pkg/runtime/runtime-gdb.py", line 198, in <module>
    _rctp_type = gdb.lookup_type("struct runtime.rtype").pointer()
gdb.error: No struct type named runtime.rtype.

[DanielMorsing]

Comment 29:

Since the debug problem wont be solved before go1.1, I've created issue #5221.

[adg]

Comment 30:

The work related to Go 1.1 is done.

Labels changed: removed go1.1.

[rsc]

Comment 31:

Labels changed: added go1.3.

[robpike]

Comment 32:

Labels changed: removed go1.3.

[ianlancetaylor]

Comment 33:

Issue #5106 has been merged into this issue.

[eliasnaur]

Comment 34:

issue #5590 added host linker support to 5l and added linux/arm to the list of os/arch
targets supporting host linking.

[rsc]

Comment 35:

It sounds like we are just missing Windows support.

[alexbrainman]

Comment 36:

> ... missing Windows support.
I never understood what is required - I don't use gcc myself. So, if someone can explain
what needs to be done (step by step), I will have a go at it.
Alex

[rsc]

Comment 37:

I don't know exactly what is required. I plan to take a closer look once
some other linker cleanup is done for Go 1.3.

[minux]

Just found another benefit of external linking.
the test misc/cgo/test, when internal linked, the size is 5112832 bytes, when external linked,
it's 1MB smaller, at 4025803 bytes. (This is on windows/amd64.)

[yinhm]

Happy to see this will be fixed in Go1.5, really appreciating.
For the binary size, as long as it works on windows, we will be happy.

[chai2010]

@minux Greate, thanks for your work!
As a windows gopher, we can use static link with the C++ libs!

[alexbrainman]

I compared file sizes for misc/cgo/test with -linkmode=internal and -linkmode=external. And the difference (about 1M) seems to be in dwarf sections (all sections starting with .debug). Should we do something about it?

Alex

[minux]

Ahh, that explains the huge size difference. I will take a look. We should generate those dwarf sections when doing external linking.