x/vgo: 403 response from api.github.com

[omac777]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

vgo version
go version go1.10 linux/amd64 vgo:2018-02-20.1

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

vgo env
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/dma2/Code/go"
GORACE=""
GOROOT="/home/dma2/go1.10"
GOTMPDIR=""
GOTOOLDIR="/home/dma2/go1.10/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build263091710=/tmp/go-build -gno-record-gcc-switches"
VGOMODROOT=""

What did you do?

I have an internal git server. In it I have a package called mfwlib.

.git/config
[remote "origin"]
	url = blahgituser@192.168.x.x:/mnt/archive/mfwgits/mfwlib.git

This mfwlib package only resides locally within GOPATH/src/github.com/omac777/mfwlib
but this package is not on github. I just made it look like it was locally.

when I change dir to GOPATH/src/github.com/omac777/mfwlib
and do
go build
It would build successfully without issue.

Now I try
vgo build -a -x -work

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

What did you expect to see?

It should build the package successfully.

What did you see instead?

vgo: finding vbom.ml/util v0.0.0-20170409195630-256737ac55c4
vgo: 403 response from api.github.com

GitHub applies fairly small rate limits to unauthenticated users, and
you appear to be hitting them. To authenticate, please visit
https://github.com/settings/tokens and click "Generate New Token" to
create a Personal Access Token. The token only needs "public_repo"
scope, but you can add "repo" if you want to access private
repositories too.

Although vgo gives an error here, I would have expected it to go searching within my existing GOPATH for the one I have fetched using go get a while ago. I did not. I simply failed to build.

For the time being I don't want to upload my packages to github. I just want to try out vgo build with local stuff using a local set of packages residing on a local git server mixed with some other people's packages residing on github such as vbom.ml/util, but have already been fetched with go get. This "Personal Access Token" github stuff doesn't seem straightforward to configure with vgo either. In fact it seems as though vgo imposes everyone to put all their packages on github/bitbucket in order to get anything done with vgo. How can we circumvent that and use a local git server with vgo?

[myitcv]

I think this is covered by #23955?

[kardianos]

vgo doesn't pull releases from git, but from zip files.

The local only git server isn't going to work because the tooling for that isn't ready, but is planned. For now you can use the replace directive for local modules and update the netrc file to work around the github 403 errors.

[rsc]

Yes, vgo intentionally does not look in $GOPATH for your source code anymore, nor does it invoke git at all. If the import path says github, it downloads from github. The solution for your situation is to use a replace directive to point at a local directory until the code appears on github. I'm actually a little confused, though. What does your go.mod look like and where is it in the file system?

[omac777]

Firstly, thank you for responding Mr. Cox,

You said: "Yes, vgo intentionally does not look in $GOPATH for your source code anymore, nor does it invoke git at all."

I am not particularly fond about this behaviour because for the past five years I have grown comfortable about how go just grabs everything from gopath and builds from it perfectly and so quickly.

You said: "If the import path says github, it downloads from github."
I am not particulary fond about this assumption either. The concept of directories should remain as directories. If there are other concepts introducing remote directories it should be explicitly stated. In this case that I want, I am explicitly stating I don't want a remote directory but I still want it named with a directory name github.com because in future, if the company permits we could easily push it into an enterprise github account AS A PRIVATE REPO. I have not obtained such permission/budget yet.

The way we work around this as a local repo is that there is a local git server within the lan that I connect to push commits, but it is fundamentally isolated from the golang build process. There should be no assumption that the go build tool fetches sources from a remote repo unless explicitly stated.
That's why vgo broke my existing project. The golden rule is vgo should be compatible and should not break existing projects. It should however introduce new concepts such as remote repo fetch while building without breaking existing project infrastructure. It would become more important as the lines of code of current infrastructure increase since we don't want to rejig the existing infrastructure code to introduce the vgo tool.

You said: "The solution for your situation is to use a replace directive to point at a local directory until the code appears on github. I'm actually a little confused, though. What does your go.mod look like and where is it in the file system?""

As I stated elsewhere, I'm not fond of having to add new files to an existing infrastructure.
I'm ok with vgo creating a new directory and basing everything off of that, but when importing legacy stuff, it should just make assumptions to use local directories unless explicitly stated otherwise.

The answer to your question:
my GOPATH is :
/home/dma2/Code/go/

root@arch-io-02 2018-04-13_14:04:00_EDT : /home/dma2/Code/go/src/192.168.x.x/mfwlib
# cat go.mod

module "192.168.x.x/mfwlib"

require (
	"github.com/go-sql-driver/mysql" v1.3.0
	"github.com/stretchr/powerwalk" v0.0.0-20151124150408-bceb9d014549
	"github.com/twotwotwo/sorts" v0.0.0-20160814051341-bf5c1f2b8553
	"vbom.ml/util" v0.0.0-20170409195630-256737ac55c4
)

root@arch-io-02 2018-04-13_14:04:35_EDT : /home/dma2/Code/go/src/192.168.x.x/go-mfw-tool1
cat go.mod

module "192.168.x.x/go-mfw-tool1"

You could assume I have a few reuse libs looking like mfwlib.
You could also assume I have a few other go-mfw-tool*-like projects also.
having to add go.mods everywhere for them is actually a pita. having tried to build with vgo without go.mod files and failing was also a painpoint.

Thank you for listening.

[gopherbot]

Change https://golang.org/cl/107657 mentions this issue: cmd/go/internal/modfetch: update for new gitrepo backend