cmd/go: using golang.org/x/* from mainland China is awkward with modules

[hellojukay]

What version of Go are you using (go version)?

[jukay@mac-pro]~/log$ go version
go version go1.11 darwin/amd64
[jukay@mac-pro]~/log$

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

[jukay@mac-pro]~/log$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/jukay/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/jukay/Code/Go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.11/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.11/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/jukay/log/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/yp/_44_30ts2_xf43zpz2j72k200000gn/T/go-build712144984=/tmp/go-build -gno-record-gcc-switches -fno-common"
[jukay@mac-pro]~/log$

What did you do?

just compile this project outside GOPATH

[jukay@mac-pro]~/log$ go build

What did you expect to see?

[jukay@mac-pro]~/log$ time go build

real	0m0.316s
user	0m0.195s
sys	0m0.232s

What did you see instead?

[jukay@mac-pro]~/log$ time go build

real	1m0.316s
user	0m0.195s
sys	0m0.232s

[hellojukay]

Unable to understand such a simple code, it took a minute to compile it.

[hellojukay]

when i usego build -v,it not use cache.

[jukay@mac-pro]~/log$ go build -v
Fetching https://golang.org/x/sys?go-get=1
https fetch failed: Get https://golang.org/x/sys?go-get=1: dial tcp 216.239.37.1:443: i/o timeout
Fetching https://golang.org/x/crypto?go-get=1
https fetch failed: Get https://golang.org/x/crypto?go-get=1: dial tcp 216.239.37.1:443: i/o timeout

[mvdan]

If the cache isn't warm, the first build may take longer if it needs to download and build the dependencies. Have you tried running time go build a second time? It should take a few seconds at most.

[hellojukay]

@mvdan I run time go build many times, it cast much time every time.

[mvdan]

Is your GOCACHE directory /Users/jukay/Library/Caches/go-build writeable? Does it contain anything? What does go build -x show?

[mvdan]

Also, note that your go build -v above shows HTTP errors. Are the build succeeding or failing? If the build fails because it can't download some modules or dependencies, they will of course not be cached.

[iand]

Doesn't china block URLs with x in them? Someone mentioned it on go-nuts.

[hellojukay]

@mvdan go build alway success, it just very slow。i can go mod vendor,it create vendor directory,and download all dependencies. this is my source code
https://github.com/hellojukay/vgo-issue

[agnivade]

Hi @hellojukay - to help us investigate the issue faster, it would be great if you can answer the questions asked above. Especially -

What does your GOCACHE directory contain ?

What is the output of go build -x -v ?

Thank you.

[hellojukay]

@agnivade yes ,yestoday i try to upgrade my golang version to 1.11.2, this is output of go build -x -v

[jukay@mac-pro]~/log$ go build -x -v
WORK=/var/folders/yp/_44_30ts2_xf43zpz2j72k200000gn/T/go-build601851662
Fetching https://golang.org/x/sys?go-get=1
https fetch failed: Get https://golang.org/x/sys?go-get=1: dial tcp 216.239.37.1:443: i/o timeout
Fetching https://golang.org/x/crypto?go-get=1
https fetch failed: Get https://golang.org/x/crypto?go-get=1: dial tcp 216.239.37.1:443: i/o timeout
mkdir -p $WORK/b001/
cat >$WORK/b001/importcfg.link << 'EOF' # internal
packagefile main=/Users/jukay/Library/Caches/go-build/27/27043f87ca77211db1b891dd2bc2209ebd069dfe894370d2782c42512cb35969-d
packagefile github.com/sirupsen/logrus=/Users/jukay/Library/Caches/go-build/e7/e7275b538c9195808fa163ca3c56c5d63281712d3d135165cce1aac98cc02bf8-d
packagefile runtime=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/runtime.a
packagefile bufio=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/bufio.a
packagefile bytes=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/bytes.a
packagefile encoding/json=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/encoding/json.a
packagefile fmt=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/fmt.a
packagefile golang.org/x/crypto/ssh/terminal=/Users/jukay/Library/Caches/go-build/0e/0ebfbc4e93e20cd471ded2ecdc73841dc36970144e7957043cdc75f65e0099f8-d
packagefile io=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/io.a
packagefile log=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/log.a
packagefile os=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/os.a
packagefile reflect=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/reflect.a
packagefile sort=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/sort.a
packagefile strings=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/strings.a
packagefile sync=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/sync.a
packagefile sync/atomic=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/sync/atomic.a
packagefile time=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/time.a
packagefile internal/bytealg=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/internal/bytealg.a
packagefile internal/cpu=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/internal/cpu.a
packagefile runtime/internal/atomic=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/runtime/internal/atomic.a
packagefile runtime/internal/sys=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/runtime/internal/sys.a
packagefile errors=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/errors.a
packagefile unicode/utf8=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/unicode/utf8.a
packagefile unicode=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/unicode.a
packagefile encoding=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/encoding.a
packagefile encoding/base64=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/encoding/base64.a
packagefile math=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/math.a
packagefile strconv=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/strconv.a
packagefile unicode/utf16=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/unicode/utf16.a
packagefile golang.org/x/sys/unix=/Users/jukay/Library/Caches/go-build/7f/7f9284a7bb944dc9533f22fbc3ba192e51145ab0792f17b55a151eb6411981f5-d
packagefile internal/poll=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/internal/poll.a
packagefile internal/syscall/unix=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/internal/syscall/unix.a
packagefile internal/testlog=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/internal/testlog.a
packagefile syscall=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/syscall.a
packagefile internal/race=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/internal/race.a
packagefile encoding/binary=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/encoding/binary.a
packagefile math/bits=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/math/bits.a
EOF
mkdir -p $WORK/b001/exe/
cd .
/usr/local/Cellar/go/1.11.2/libexec/pkg/tool/darwin_amd64/link -o $WORK/b001/exe/a.out -importcfg $WORK/b001/importcfg.link -buildmode=exe -buildid=7l5ZuOc-Ol-x9AyQCky7/WlX-HbvTd8yVTtnfYvzf/J45TO0-nMm2qI1TaHJKq/7l5ZuOc-Ol-x9AyQCky7 -extld=clang /Users/jukay/Library/Caches/go-build/27/27043f87ca77211db1b891dd2bc2209ebd069dfe894370d2782c42512cb35969-d
/usr/local/Cellar/go/1.11.2/libexec/pkg/tool/darwin_amd64/buildid -w $WORK/b001/exe/a.out # internal
mv $WORK/b001/exe/a.out main
rm -r $WORK/b001/

[hellojukay]

This is my GOCACHE directory

[jukay@mac-pro]~/log$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/jukay/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/jukay/Code/Go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.11.2/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.11.2/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/jukay/log/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/yp/_44_30ts2_xf43zpz2j72k200000gn/T/go-build984680892=/tmp/go-build -gno-record-gcc-switches -fno-common"
[jukay@mac-pro]~/log$ cd /Users/jukay/Library/Caches/go-build
[jukay@mac-pro]~/Library/Caches/go-build$ ls
00       14       28       3c       50       64       78       8c       README   b3       c7       db       ef
01       15       29       3d       51       65       79       8d       a0       b4       c8       dc       f0
02       16       2a       3e       52       66       7a       8e       a1       b5       c9       dd       f1
03       17       2b       3f       53       67       7b       8f       a2       b6       ca       de       f2
04       18       2c       40       54       68       7c       90       a3       b7       cb       df       f3
05       19       2d       41       55       69       7d       91       a4       b8       cc       e0       f4
06       1a       2e       42       56       6a       7e       92       a5       b9       cd       e1       f5
07       1b       2f       43       57       6b       7f       93       a6       ba       ce       e2       f6
08       1c       30       44       58       6c       80       94       a7       bb       cf       e3       f7
09       1d       31       45       59       6d       81       95       a8       bc       d0       e4       f8
0a       1e       32       46       5a       6e       82       96       a9       bd       d1       e5       f9
0b       1f       33       47       5b       6f       83       97       aa       be       d2       e6       fa
0c       20       34       48       5c       70       84       98       ab       bf       d3       e7       fb
0d       21       35       49       5d       71       85       99       ac       c0       d4       e8       fc
0e       22       36       4a       5e       72       86       9a       ad       c1       d5       e9       fd
0f       23       37       4b       5f       73       87       9b       ae       c2       d6       ea       fe
10       24       38       4c       60       74       88       9c       af       c3       d7       eb       ff
11       25       39       4d       61       75       89       9d       b0       c4       d8       ec       log.txt
12       26       3a       4e       62       76       8a       9e       b1       c5       d9       ed       trim.txt
13       27       3b       4f       63       77       8b       9f       b2       c6       da       ee
[jukay@mac-pro]~/Library/Caches/go-build$

[jukay@mac-pro]~/Library/Caches/go-build/27$ ls
27043f87ca77211db1b891dd2bc2209ebd069dfe894370d2782c42512cb35969-d
275c760696e3a1f52ee8966821a649f4ee3a720eb6a37639cb7259c82b53825f-a
27afd4f6c43d566036c18b19d4eb17b801039ebb1692781323ce1274baaebbfb-a
27c160e9dd5b2354628a8bb5a4a70aaac2197a1d70ab57c66e8c8e2287bbb3e2-a
[jukay@mac-pro]~/Library/Caches/go-build/27$

[hellojukay]

After upgrade go go version go1.11.2 darwin/amd64,

[jukay@mac-pro]~/log$ time go build

real	1m0.434s
user	0m0.212s
sys	0m0.272s

Even if I use go mod edit replace, vgo alway try to connect to golang.org , when i run go build,until the connection is time out。it didn't seem to use the cache。

[agnivade]

There seems to be various oddities-

• The build tries to connect to https://golang.org/x/crypto and https://golang.org/x/crypto everytime.

• Your go.mod seems to replace

replace golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33 => github.com/golang/sys v0.0.0-20180905080454-ebe1bf3edb33

Yet, it is trying to connect to golang.org.

/cc @myitcv @bcmills for further insight.

And FYI - I think you mean go when you say vgo, because module functionality is integrated into the go toolchain.

[hellojukay]

@agnivade Not only on mac, but also on centos7, compiler will try to connect golang. org until connection timeout

[root@iZ2zefvnzjphkalt990sg0Z vgo-issue]# go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/root/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/lib/golang"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/root/vgo-issue/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build194230206=/tmp/go-build -gno-record-gcc-switches"

[root@iZ2zefvnzjphkalt990sg0Z vgo-issue]# time go build -x -v
WORK=/tmp/go-build973675453
Fetching https://golang.org/x/sys?go-get=1
https fetch failed: Get https://golang.org/x/sys?go-get=1: dial tcp 216.239.37.1:443: i/o timeout
Fetching https://golang.org/x/crypto?go-get=1
https fetch failed: Get https://golang.org/x/crypto?go-get=1: dial tcp 216.239.37.1:443: i/o timeout

real	1m0.108s
user	0m0.068s
sys	0m0.041s

GOCACHE

/root/.cache/go-build
[root@iZ2zefvnzjphkalt990sg0Z go-build]# ls
00  09  12  1b  24  2d  36  3f  48  51  5a  63  6c  75  7e  87  90  99  a2  ab  b4  bd  c6  cf  d8  e1  ea  f3  fc
01  0a  13  1c  25  2e  37  40  49  52  5b  64  6d  76  7f  88  91  9a  a3  ac  b5  be  c7  d0  d9  e2  eb  f4  fd
02  0b  14  1d  26  2f  38  41  4a  53  5c  65  6e  77  80  89  92  9b  a4  ad  b6  bf  c8  d1  da  e3  ec  f5  fe
03  0c  15  1e  27  30  39  42  4b  54  5d  66  6f  78  81  8a  93  9c  a5  ae  b7  c0  c9  d2  db  e4  ed  f6  ff
04  0d  16  1f  28  31  3a  43  4c  55  5e  67  70  79  82  8b  94  9d  a6  af  b8  c1  ca  d3  dc  e5  ee  f7  log.txt
05  0e  17  20  29  32  3b  44  4d  56  5f  68  71  7a  83  8c  95  9e  a7  b0  b9  c2  cb  d4  dd  e6  ef  f8  README
06  0f  18  21  2a  33  3c  45  4e  57  60  69  72  7b  84  8d  96  9f  a8  b1  ba  c3  cc  d5  de  e7  f0  f9  trim.txt
07  10  19  22  2b  34  3d  46  4f  58  61  6a  73  7c  85  8e  97  a0  a9  b2  bb  c4  cd  d6  df  e8  f1  fa
08  11  1a  23  2c  35  3e  47  50  59  62  6b  74  7d  86  8f  98  a1  aa  b3  bc  c5  ce  d7  e0  e9  f2  fb
[root@iZ2zefvnzjphkalt990sg0Z go-build]#

[myitcv]

A couple of observations

• If the replacements need to be systematic (as they appear to be, potentially for reasons of certain URLs being blocked) then some sort of proxy is going to ultimately be a cleaner solution
• The go tool checking golang.org regardless of the replace is likely related to cmd/go: satisfy import by adding require of available replacement #26241

[bcmills]

The issue with fetching golang.org is likely #27859 (which is fixed at head).

[bcmills]

@hellojukay, could you see if you can reproduce the issue with a go toolchain built from source¹ at HEAD? I'd like to figure out whether this is a duplicate of #27859 or if there is something more to it.

¹ See https://golang.org/doc/install/source for instructions.

[myitcv]

Note another instance of effectively the same problem in #28716.

Stepping back for one second, if golang.org remains blocked in China (ref https://blog.golang.org/hello-china), then I wonder whether we need to think of a different approach here.

Even though in Go 1.12 #26241 will give us comprehensive support for the concept of "wherever you see X use Y", it strikes me that users from mainland China will always need to add a replace statements for any of the golang.org/x/* modules, in every one of their projects where they are a (transitive) dependency.

This seems less than ideal. I would guess that this would impact a significant portion of users' projects.

Is there anything we can do here in the go tool itself?

cc @andybons

[hellojukay]

@myitcv I think this is a bug of the code of go modules, If I set up replace for golang.org, then it should not connect to golang.org。Isn't it? wait for me , i am building go compiler from the master branch golang 。

[myitcv]

If I set up replace for golang.org, then it should not connect to golang.org

Correct. As @bcmills mentioned, the solution for #27859 should fix the issue as far as go build is concerned. #26241 is the more general solution, however.

[ScottHuangZL]

@hellojukay @agnivade No plan to build go from master source branch. Will wait until the go1.11.3 version ready.

@agnivade @bcmills May I know when go.1.11.3 release? thanks.

[agnivade]

That fix (#27859) is not going to be backported. If you don't want to build from master, please wait for the 1.12 beta release which should be out soon (within this week).

[ScottHuangZL]

@agnivade sounds good, thanks.

[agnivade]

@panjiang wanted a wildcard replace shortcut so that one could replace all the golang.org/x repos to their github.com equivalent in one-shot. #29076 (comment)

[ScottHuangZL]

A quick way:
go mod edit -require=golang.org/x/net@v0.0.0
go mod edit -replace=golang.org/x/net@v0.0.0=github.com/golang/net@latest
go mod tidy
go mod vendor
go build -mod=vendor

Then the build speed still fast!!!

[bcmills]

I just landed the fix for #26241, which should remove the need for the go mod edit -require step (you can skip straight to the go mod edit -replace and omit the arbitrary v0.0.0 version).

[sgon00]

go module is a nightmare in China. Even if I have a very good VPN, go get -u will never complete in my life.

go mod edit won't work because you need to have the correct go.mod at first. It's like "which came first: the chicken or the egg?".

Please kill me I am sad and mad.

[hellojukay]

@sgon00 https://gocenter.io/ can help you.

[sgon00]

@hellojukay Thanks a lot for letting me know gocenter.io. I have't tried it yet and will try it later. Sorry that I completely collapsed yesterday. I was trying go get -u simplePackage inside module directory from 11:00am to 03:00am without a luck. It can never complete successfully. But running the same command outside the module directory, it can be completed quickly without any problems. This is just crazy.

[mindon]

linux
GOPROXY=https://mirrors.aliyun.com/goproxy/ go build
or win
env GOPROXY=https://mirrors.aliyun.com/goproxy/ go build

or use https://goproxy.io

[bcmills]

Given goproxy.io and other proxies with high availability in China, is there anything left to do here?

[changkun]

@oiooj @aofei

[aofei]

It's never just golang.org, github.com is also sometimes/always unreachable in some special provinces, such as Beijing and Xinjiang. In fact, people in mainland China are already used to solving this kind of problem by using a public hosted GOPROXY.

Given that Go modules are nearly mature and widely used in production, I believe most people are aware of the benefits of using a GOPROXY (not just solve this kind of problem). So I see no reason for this issue to remain open.