cmd/go: install -linkshared invokes link commands for existing .so files since go1.10

[yunabe]

What version of Go are you using (go version)?

go version go1.10 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/yunabe/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/yunabe/local/gocode"
GORACE=""
GOROOT="/home/yunabe/go1.10"
GOTMPDIR=""
GOTOOLDIR="/home/yunabe/go1.10/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build331277736=/tmp/go-build -gno-record-gcc-switches"

What did you do?

cd $TMPDIR
go get -d golang.org/x/net/context
go install -buildmode=shared -linkshared -pkgdir `pwd`/tmppkg -linkshared std
go install -x -buildmode=shared -pkgdir `pwd`/tmppkg -linkshared golang.org/x/net/context

What did you expect to see?

The second go install command compiles context.a and link libgolang.org-x-net-context.so. It takes less than 1 sec to finish.

What did you see instead?

From go1.10, the second go install run link command to create libstd.so though libstd.so already exists in -pkgdir.
It makes the second go install command 3-4 times slower (0.77sec in go1.9.4 vs 2.8sec in go1.10)

I debugged the behavior of go tools and it seems like b.useCache in linkShared returns false to libstd.so target (https://github.com/golang/go/blob/master/src/cmd/go/internal/work/exec.go#L1032).

[FiloSottile]

@gopherbot please open a backport tracking issue, as this is a 1.10 regression, although just a performance one.

[gopherbot]

Backport issue(s) opened: #25044 (for 1.10).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

[Foxboron]

This isn't actually just a performance issue i believe. Currently this prevents users from running -buildmode=shared -linkshared as we don't have write privileges inn $GOROOT, and it attempts to write towards several files there.

With go 1.10

==> Starting build()...
WORK=/tmp/go-build138655936
rm -r $WORK/b003/
rm -r $WORK/b011/
rm -r $WORK/b007/
rm -r $WORK/b022/
mkdir -p $WORK/b010/
mkdir -p /usr/lib/go/pkg/linux_amd64_dynlink/runtime/internal/
cp /home/fox/.cache/go-build/20/203799d24b5628cee748f251b4361e400b7392a9e65f21f6345ecb0ac1e99ca2-d /usr/lib/go/pkg/linux_amd64_dynlink/runtime/internal/atomic.a
rm -r $WORK/b010/
go install runtime/internal/atomic: open /usr/lib/go/pkg/linux_amd64_dynlink/runtime/internal/atomic.a: permission denied
rm -r $WORK/b023/
rm -r $WORK/b037/
mkdir -p $WORK/b012/
mkdir -p /usr/lib/go/pkg/linux_amd64_dynlink/sync/
cp /home/fox/.cache/go-build/59/59efc6e118c62c218e3e59d3e843f2e6895024fd8cfe6cba4d14ad046db4f7c4-d /usr/lib/go/pkg/linux_amd64_dynlink/sync/atomic.a
rm -r $WORK/b047/
rm -r $WORK/b012/
go install sync/atomic: open /usr/lib/go/pkg/linux_amd64_dynlink/sync/atomic.a: permission denied
rm -r $WORK/b048/
rm -r $WORK/b052/
mkdir -p $WORK/b014/
mkdir -p /usr/lib/go/pkg/linux_amd64_dynlink/internal/
cp /home/fox/.cache/go-build/5c/5cad130c59015083c83300f528455ed005db9a7862397945d4740e5e35e8f044-d /usr/lib/go/pkg/linux_amd64_dynlink/internal/cpu.a
rm -r $WORK/b014/
go install internal/cpu: open /usr/lib/go/pkg/linux_amd64_dynlink/internal/cpu.a: permission denied
mkdir -p $WORK/b053/
mkdir -p /usr/lib/go/pkg/linux_amd64_dynlink/crypto/internal/
cp /home/fox/.cache/go-build/ae/aee318ece67832c036cdbc8be92aef902422e29e53942e16e2935a2dd7a7f0e7-d /usr/lib/go/pkg/linux_amd64_dynlink/crypto/internal/cipherhw.a
rm -r $WORK/b053/
rm -r $WORK/b077/
go install crypto/internal/cipherhw: open /usr/lib/go/pkg/linux_amd64_dynlink/crypto/internal/cipherhw.a: permission denied
rm -r $WORK/b079/
rm -r $WORK/b094/
rm -r $WORK/b100/
mkdir -p $WORK/b085/
mkdir -p /usr/lib/go/pkg/linux_amd64_dynlink/vendor/golang_org/x/crypto/
cp /home/fox/.cache/go-build/27/278eecfb4ee5a9a34edbfc24a41c030a7f28e489c9a1edcc610606a7684baf40-d /usr/lib/go/pkg/linux_amd64_dynlink/vendor/golang_org/x/crypto/curve25519.a
rm -r $WORK/b085/
mkdir -p $WORK/b084/
rm -r $WORK/b146/
cp /home/fox/.cache/go-build/a8/a883a42b6f584a222b93fd8c5df99499fd4dba6efcf90a2eb564ea90cdad6611-d /usr/lib/go/pkg/linux_amd64_dynlink/vendor/golang_org/x/crypto/poly1305.a
rm -r $WORK/b084/
go install vendor/golang_org/x/crypto/poly1305: open /usr/lib/go/pkg/linux_amd64_dynlink/vendor/golang_org/x/crypto/poly1305.a: permission denied
rm -r $WORK/b154/
rm -r $WORK/b155/
rm -r $WORK/b147/
rm -r $WORK/b156/
go install vendor/golang_org/x/crypto/curve25519: open /usr/lib/go/pkg/linux_amd64_dynlink/vendor/golang_org/x/crypto/curve25519.a: permission denied
rm -r $WORK/b187/
rm -r $WORK/b186/
rm -r $WORK/b025/
rm -r $WORK/b180/

Meanwhile go 1.9.4

==> Starting build()...
WORK=/tmp/go-build073266895
golang.org/x/net/context
mkdir -p $WORK/golang.org/x/net/context/_obj/
mkdir -p $WORK/golang.org/x/net/
cd /home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/src/golang.org/x/net/context
/usr/lib/go/pkg/tool/linux_amd64/compile -o $WORK/golang.org/x/net/context.a -trimpath $WORK -dynlink -goversion go1.9.4 -p golang.org/x/net/context -complete -installsuffix dynlink -buildid 3d063c354b94a000bba5463988c84e19a35173d6 -D _/home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/src/golang.org/x/net/context -I $WORK -pack ./context.go ./go17.go ./go19.go
mkdir -p /home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/pkg/linux_amd64_dynlink/golang.org/x/net/
cp $WORK/golang.org/x/net/context.a /home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/pkg/linux_amd64_dynlink/golang.org/x/net/context.a
cd .
/usr/lib/go/pkg/tool/linux_amd64/link -o $WORK/libgolang.org-x-net-context.so -L $WORK -L /home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/pkg/linux_amd64_dynlink -installsuffix dynlink -buildmode=shared -linkshared -w -extld=gcc golang.org/x/net/context=/home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/pkg/linux_amd64_dynlink/golang.org/x/net/context.a
echo 'libgolang.org-x-net-context.so' > /home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/pkg/linux_amd64_dynlink/golang.org/x/net/context.shlibname # internal
mkdir -p /home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/pkg/linux_amd64_dynlink/
cp $WORK/libgolang.org-x-net-context.so /home/fox/Git/GolangPKGBUILD/go-golang.org-x-net/src/gopath/pkg/linux_amd64_dynlink/libgolang.org-x-net-context.so

[rsc]

Sorry but not going to get to this for Go 1.11. Linkshared is admittedly a mess.

[yunabe]

Congratulations for go1.12 launch, Go team.

Do you think this linkShared related performance regression will be fixed in Go1.13?
https://github.com/yunabe/lgo, an interactive Go programming environment with Jupyter, heavily relies on go install -linkshared and it has a critical performance issue since go1.10 due to this bug.

Now, go1.12 was released and I want to stop using go1.9 in the project. I would appreciate it if this bug is fixed in the near future.

Thanks,
yunabe

[tbolsh]

Was this bug fixed in go 1.13 or 1.14?
Was the #30768 fixed in 1.13 or 1.14?

[mvdan]

@tbolsh this bug isn't fixed. #30768 was fixed in 1.13, which you can see via the milestone and commit.

[rezaalavi]

The bug still exists in 1.14.3

[mvdan]

@rezaalavi yes, the issue is open. Please don't comment in the issue if this is all you have to add.

[ldemailly]

being able to use golang in interactive notebooks is critical for both having people learn and teach go as well as build interactive science and quick prototypes and simulation, maybe the team can prioritize this regression a bit more?

[seankhliao]

Obsoleted by #47788