Glide seems to hang with high CPU importing kubernetes API

[readams]

Create:

$GOPATH/src/example.com/example/main.go:

package main

import (
    "k8s.io/kubernetes/pkg/client/restclient"
    client "k8s.io/kubernetes/pkg/client/unversioned"
)

func main() {
    var config *restclient.Config
    client.New(config)
}

Now run:

cd $GOPATH/src/example.com/example
glide create --non-interactive
glide install

It will make slow progress for a while but eventually seems to hits a wall and seems to make no progress. It's using about 1 full CPU core while resident size increases only very slowly.

I'm using go 1.6 on Ubuntu Xenial and glide 0.12.1 from ppa.

Output looks like:

$ glide init --non-interactive
[INFO]  Generating a YAML configuration file and guessing the dependencies
[INFO]  Attempting to import from other package managers (use --skip-import to skip)
[INFO]  Scanning code to look for dependencies
[INFO]  --> Found reference to k8s.io/kubernetes/pkg/client/restclient
[INFO]  --> Adding sub-package pkg/client/unversioned to k8s.io/kubernetes
[INFO]  Writing configuration file (glide.yaml)
[INFO]  You can now edit the glide.yaml file. Consider:
[INFO]  --> Using versions and ranges. See https://glide.sh/docs/versions/
[INFO]  --> Adding additional metadata. See https://glide.sh/docs/glide.yaml/
[INFO]  --> Running the config-wizard command to improve the versions in your configuration

$ glide install
[INFO]  Lock file (glide.lock) does not exist. Performing update.
[INFO]  Downloading dependencies. Please wait...
[INFO]  --> Fetching updates for k8s.io/kubernetes.
[INFO]  Resolving imports
[INFO]  --> Fetching updates for github.com/golang/glog.
[INFO]  --> Fetching updates for github.com/google/cadvisor.
[INFO]  --> Fetching updates for github.com/davecgh/go-spew.
[INFO]  --> Fetching updates for github.com/ugorji/go.
[INFO]  --> Fetching updates for golang.org/x/net.
[INFO]  --> Fetching updates for github.com/go-openapi/spec.
[INFO]  --> Fetching updates for github.com/gogo/protobuf.
[INFO]  --> Fetching updates for github.com/google/gofuzz.
[INFO]  --> Fetching updates for github.com/juju/ratelimit.
[INFO]  --> Fetching updates for github.com/blang/semver.
[INFO]  --> Fetching updates for github.com/emicklei/go-restful.
[INFO]  --> Fetching updates for github.com/spf13/pflag.
[INFO]  --> Fetching updates for gopkg.in/inf.v0.
[INFO]  --> Fetching updates for github.com/pborman/uuid.
[INFO]  --> Fetching updates for github.com/go-openapi/jsonpointer.
[INFO]  --> Fetching updates for github.com/go-openapi/jsonreference.
[INFO]  --> Fetching updates for github.com/go-openapi/swag.
[INFO]  --> Fetching updates for github.com/docker/distribution.
[INFO]  --> Fetching updates for golang.org/x/oauth2.
[INFO]  --> Fetching updates for k8s.io/client-go.
[INFO]  --> Fetching updates for github.com/coreos/go-oidc.
[INFO]  --> Fetching updates for github.com/ghodss/yaml.
[INFO]  --> Fetching updates for github.com/PuerkitoBio/purell.
[INFO]  --> Fetching updates for github.com/mailru/easyjson.
[INFO]  --> Fetching updates for google.golang.org/appengine.
[INFO]  --> Fetching updates for cloud.google.com/go.
[INFO]  --> Fetching updates for github.com/coreos/pkg.
[INFO]  --> Fetching updates for github.com/jonboulle/clockwork.
[INFO]  --> Fetching updates for github.com/cloudfoundry-incubator/candiedyaml.
[INFO]  --> Fetching updates for github.com/PuerkitoBio/urlesc.
[INFO]  --> Fetching updates for golang.org/x/text.

[seh]

I think I've run into the same problem using Glide version version 0.12.2. Running glide install, after Glide fetches various repositories, it grinds away emitting messages like the following:

[DEBUG] Package github.com/go-openapi/spec imports github.com/go-openapi/jsonpointer
[DEBUG] Missing github.com/go-openapi/jsonpointer. Trying to resolve.
[DEBUG] Dependency github.com/go-openapi/jsonpointer has already been pinned. Fetching updates skipped.
[DEBUG] Dependency github.com/go-openapi/jsonpointer has already been pinned. Setting version skipped.
[DEBUG] Package github.com/go-openapi/spec imports github.com/go-openapi/jsonreference
[DEBUG] Missing github.com/go-openapi/jsonreference. Trying to resolve.
[DEBUG] Dependency github.com/go-openapi/jsonreference has already been pinned. Fetching updates skipped.
[DEBUG] Dependency github.com/go-openapi/jsonreference has already been pinned. Setting version skipped.
[DEBUG] Package github.com/go-openapi/spec imports github.com/go-openapi/swag
[DEBUG] Missing github.com/go-openapi/swag. Trying to resolve.
[DEBUG] Dependency github.com/go-openapi/swag has already been pinned. Fetching updates skipped.
[DEBUG] Dependency github.com/go-openapi/swag has already been pinned. Setting version skipped.
[DEBUG] Trying to open github.com/gogo/protobuf/proto (/Users/seh/.glide/cache/src/https-git.luolix.top-gogo-protobuf/proto)
[DEBUG] Trying to open github.com/gogo/protobuf/sortkeys (/Users/seh/.glide/cache/src/https-git.luolix.top-gogo-protobuf/sortkeys)
[DEBUG] Trying to open github.com/google/gofuzz (/Users/seh/.glide/cache/src/https-git.luolix.top-google-gofuzz)
[DEBUG] Trying to open k8s.io/kubernetes/pkg/conversion (/Users/seh/.glide/cache/src/https-git.luolix.top-kubernetes-kubernetes/pkg/conversion)
[DEBUG] Package k8s.io/kubernetes/pkg/conversion imports k8s.io/kubernetes/third_party/forked/golang/reflect
[DEBUG] Missing k8s.io/kubernetes/third_party/forked/golang/reflect. Trying to resolve.
[DEBUG] Dependency k8s.io/kubernetes has already been pinned. Fetching updates skipped.
[DEBUG] Dependency k8s.io/kubernetes has already been pinned. Setting version skipped.
[DEBUG] Trying to open k8s.io/kubernetes/pkg/genericapiserver/openapi/common (/Users/seh/.glide/cache/src/https-git.luolix.top-kubernetes-kubernetes/pkg/genericapiserver/openapi/common)
[DEBUG] Package k8s.io/kubernetes/pkg/genericapiserver/openapi/common imports github.com/go-openapi/spec
[DEBUG] Missing github.com/go-openapi/spec. Trying to resolve.
[DEBUG] Dependency github.com/go-openapi/spec has already been pinned. Fetching updates skipped.
[DEBUG] Dependency github.com/go-openapi/spec has already been pinned. Setting version skipped.
[DEBUG] Trying to open k8s.io/kubernetes/pkg/labels (/Users/seh/.glide/cache/src/https-git.luolix.top-kubernetes-kubernetes/pkg/labels)

I've had it running continuously now for about 15 hours, and I can't tell if it's getting any closer to finishing.

[timoreimann]

FWIW, I don't think I have experienced this problem with Glide 0.11 and my Kubernetes dependencies have been about the same for quite a while.

[seh]

Per @mattfarina's comment in the "general" channel of the "Gophers" Slack team, there's a symbolic link in the Kubernetes repository that's aggravating Glide's ability to finish:

% ls -l ~/.glide/cache/src/https-git.luolix.top-kubernetes-kubernetes/vendor/k8s.io 
total 8
lrwxr-xr-x  1 seh  staff   34 Sep 23 13:26 client-go -> ../../staging/src/k8s.io/client-go
drwxr-xr-x  4 seh  staff  136 Sep 23 13:26 heapster

[seh]

I find that the same problem arises when my project depends on just the kubernetes/client-go repository, so I don't think that this symbolic link is the problem here.

[mattfarina]

Can you try the fix to Glide in PR #622? I think this fixes it.

[seh]

I'll clone that and try it, but I may not finish before I have to give up tonight. I'll let you know by tomorrow morning.

[timoreimann]

@mattfarina @seh I just checked out PR #622 and used it to repopulate my sample Kubernetes client's dependencies (among which k8s.io/client-go is).

glide update ran to completion just fine again (whereas v0.11 would consistently loop forever). 🎉

From my perspective, this is ready to 🚢 !