k8s.io/client-go vendors k8s.io/apimachinery which breaks build #83

davecheney · 2017-01-31T00:26:47Z

TL;DR: libraries must not vendor their dependencies

It is not possible to build programs that vendor k8s.io/client-go as the latter has vendored k8s.io/apimachinery.

Sample program:

package main

import (
        "flag"
        "fmt"
        "log"
        "os"
        "path/filepath"
        "time"

        metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
        "k8s.io/client-go/tools/clientcmd"

        "k8s.io/client-go/kubernetes"
)

func main() {
        kubeconfig := flag.String("kubeconfig", filepath.Join(os.Getenv("HOME"), ".kube", "config"), "absolute path to the kubeconfig file")

        flag.Parse()
        // uses the current context in kubeconfig
        config, err := clientcmd.BuildConfigFromFlags("", *kubeconfig)
        check(err)

        clientset, err := kubernetes.NewForConfig(config)
        check(err)

        for {
                pods, err := clientset.Core().Pods("").List(metav1.ListOptions{})
                check(err)

                fmt.Printf("There are %d pods in the cluster\n", len(pods.Items))
                time.Sleep(10 * time.Second)
        }
}

func check(err error) {
        if err != nil {
                log.Fatalf("fatal: %v", err)
        }
}

This program will not compile because the vendored copy of k8s.io/apimachinery (https://github.com/kubernetes/client-go/tree/master/vendor/k8s.io/apimachinery) aliases the copy in my own project, this breaks type equality.

# stash.atlassian.com/scm/kube/rds-resource-provider/cmd/postgres-resource-provider
cmd/postgres-resource-provider/main.go:29: cannot use "stash.atlassian.com/scm/kube/rds-resource-provider/vendor/k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions literal (type "stash.atlassian.com/scm/kube/rds-resource-provider/vendor/k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions) as type "stash.atlassian.com/scm/kube/rds-resource-provider/vendor/k8s.io/client-go/vendor/k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions in argument to clientset.Core().Pods("").List
Makefile:11: recipe for target 'build' failed
make: *** [build] Error 2

The text was updated successfully, but these errors were encountered:

lavalamp · 2017-01-31T00:40:48Z

TL;DR: go's vendoring system is fundamentally broken.

We are working on switching to the new dep vendoring management tool, after which this should be acceptable. (You'll run dep ensure and it will flatten the dependencies.)

If we don't vendor, then you'll have a very difficult time getting a set of dependencies that actually works. We describe this in the top level README. This is a huge source of pain for us--we just did some code moves to other packages which are causing the current set of difficulties.

lavalamp · 2017-01-31T00:42:42Z

(And I know the go doctrine is "have stable public apis and make new repos when you break them" but we are untangling a big ball of spaghetti and that's just totally impractical for us.)

davecheney · 2017-01-31T00:42:58Z

@lavalamp no argument there, vendor/ is utterly broken for library writers. /cc @spf13

We are working on switching to the new dep vendoring management tool, after which this should be acceptable. (You'll run dep ensure and it will flatten the dependencies.)

This tool is marked pre alpha, are you sure that is prudent?

Can you at least update the vendored copy of k8s.io/apimachinery ?

deads2k · 2017-01-31T14:20:51Z

Can you at least update the vendored copy of k8s.io/apimachinery ?

apimachinery updates a lot faster than client-go because of the way the process goes. client-go is the one that needs updating, but because of the merge queue process its difficult to do. See my comment here for fixing client-go sync kubernetes/kubernetes#39504 (comment)

caesarxuchao · 2017-02-01T09:39:03Z

This program will not compile because the vendored copy of k8s.io/apimachinery (https://github.com/kubernetes/client-go/tree/master/vendor/k8s.io/apimachinery) aliases the copy in my own project

@davecheney do you use a vendor package management tool? Both godep and glide will flatten the dependencies.

However, godep won't handle version conflicts in dependencies. This is why we want to switch to dep or glide, which supports specifying a range of versions, lowing the probability of conflicts.

For now, can you use the release-2.0 branch of client-go? That branch doesn't depend on k8s.io/apimachinary.

davecheney · 2017-02-01T09:45:31Z

dpw/vendetta. I've worked around the issue for the moment. Please feel free to close this issue.

…

On Wed, Feb 1, 2017 at 8:39 PM, Chao Xu ***@***.***> wrote: This program will not compile because the vendored copy of k8s.io/apimachinery (https://github.com/kubernetes/client-go/tree/ master/vendor/k8s.io/apimachinery) aliases the copy in my own project @davecheney <https://github.com/davecheney> do you use a vendor package management tool? Both godep and glide will flatten the dependencies. However, godep won't handle version conflicts in dependencies. This is why we want to switch to dep or glide, which supports specifying a range of versions, lowing the probability of conflicts. For now, can you use the release-2.0 branch of client-go? That branch doesn't depend on k8s.io/apimachinary. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#83 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcA0CWgCsBYohiA8mIaGTztY2-Vcoyks5rYFK5gaJpZM4LyGG0> .

lavalamp · 2017-02-02T00:36:34Z

A version of client-go with the apimachinery package flattened back out is in github.com/lavalamp/client-go-flat. (go get github.com/lavalamp/client-go-flat should work.)

Is this approach something we should automate and officialize?

seh · 2017-02-02T13:54:58Z

Using Glide with its --strip-vendor flag allows me to use kubernetes/client-go together with kubernetes/apimachinery, per @caesarxuchao's suggestion above.

deads2k · 2017-02-02T15:15:39Z

@lavalamp @smarterclayton @liggitt in the experiment you've created here: https://github.com/lavalamp/client-go-flat/, how would another library writer who generated his clients using our client generator against the apimachinery repo go about creating a compatible client? The signatures for anything that uses an apimachinery type like RESTClient https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/rest/client.go#L61-L89 will be incompatible.

As we add aggregated API servers, we expect this to be more common, not less.

deads2k · 2017-02-02T15:44:13Z

@lavalamp I think we ought to try to deleting the content of vendor/k8s.io/apimachinery, but leaving its manifest entry present. With the synchronized rollout that @caesarxuchao is working on, go get-ing from master should give matched sets instead of client-go lagging. By leaving the manifest entry, anyone trying to vendor would be able to pin the exact version we want. And we wouldn't face the "we generated incompatible clients" problem.

caesarxuchao · 2017-02-03T08:13:13Z

@deads2k Leaving only apimachinery out of vendor/ is strange. If a user use git clone to download client-go, missing only apimachinery in vendor/ will cause confusion.

The synchronized rollout work (kubernetes/test-infra#1784) will keep client-go/vendor/k8s.io/apimachinery up-to-date, so keeping the whole vendor/ folder is possible.

We could also remove the entire vendor/ folder (#19 (comment)), but I see two problems: 1. go get user will get incompatible dependencies. 2. we'll need to provide dependency manifests for multiple vendor package management tools. 2 is solvable, just requires some maintenance efforts.

deads2k · 2017-02-03T12:37:11Z

@caesarxuchao is there another option that will allow different client-go libraries from other projects to inter-operate?

lavalamp · 2017-02-03T23:32:21Z

ok, I think flattening out the dependencies like we talked about at the SIG is not a workable plan.

I think our approach should be:

document
users must use a version control tool, dep, glide, or godep
continuously test so that every supported branch always always always works.

errordeveloper · 2017-02-06T06:24:18Z

I currently need to have set exact revisions of this repo and apimachinery in my glide.yaml, in fact I never wanted to care about two packages and how they relate to each other just about makes sense to me as a user... Looks like master is broken at the moment, and simply referencing latest revisions like I did a few days ago didn't work. Perhaps there are still many usability improvements that are waiting to be implemented. I think basically it's whenever you want to construct a object natively (not from YAML), then you have to pull in apimachinery.

errordeveloper · 2017-02-06T06:26:20Z

@seh I am also using glide up -v...

deads2k · 2017-02-06T13:23:48Z

I currently need to have set exact revisions of this repo and apimachinery in my glide.yaml, in fact I never wanted to care about two packages and how they relate to each other just about makes sense to me as a user... Looks like master is broken at the moment, and simply referencing latest revisions like I did a few days ago didn't work. Perhaps there are still many usability improvements that are waiting to be implemented. I think basically it's whenever you want to construct a object natively (not from YAML), then you have to pull in apimachinery.

@caesarxuchao has kubernetes/test-infra#1784 to change the client-go publishing infrastructure so that it stops lagging behind.

lavalamp · 2017-02-13T18:36:28Z

Current strategy is being worked out in https://docs.google.com/document/d/1h_IBGYPMa8FS0oih4NbVkAMAzM7YTHr76VBcKy1qFbg/edit#

rubenv · 2017-02-13T19:10:30Z

@lavalamp That's not publicly viewable?

lavalamp · 2017-02-13T19:12:38Z

It's shared with the api machinery sig group--there should be a link in the title.

…

On Mon, Feb 13, 2017 at 11:10 AM, Ruben Vermeersch ***@***.*** > wrote: @lavalamp <https://github.com/lavalamp> That's not publicly viewable? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#83 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAnglicvYPxH_Qo3JWUSIJ8NHRu4Nsqeks5rcKqngaJpZM4LyGG0> .

ericchiang · 2017-02-15T20:54:59Z

edit: redacted. I misunderstood what tags import what.

It looks like v2.0.0-alpha.1 doesn't specify a version of k8s.io/apimachinery[0] in it's Godeps file so tools like glide can't determine the version needed. @errordeveloper that's why you're glide command doesn't work.

~~Here's an attempt to compile the out of cluster example[1]:~~

$ cat glide.yaml 
package: github.com/ericchiang/client-go-example
import:
- package: k8s.io/client-go
  version: v2.0.0-alpha.1
$ ls
glide.yaml  main.go
$ glide up -v
[INFO]	Downloading dependencies. Please wait...
[INFO]	--> Fetching updates for k8s.io/client-go.
[INFO]	--> Setting version for k8s.io/client-go to v2.0.0-alpha.1.
[INFO]	Resolving imports
[INFO]	--> Fetching updates for github.com/go-openapi/spec.
[INFO]	--> Fetching updates for github.com/gogo/protobuf.
[INFO]	--> Fetching updates for github.com/google/gofuzz.
[INFO]	--> Fetching updates for k8s.io/apimachinery.
[INFO]	Found Godeps.json file in /home/eric/.glide/cache/src/https-k8s.io-client-go
[INFO]	--> Parsing Godeps metadata...
# ...
$ go install
./main.go:30: cannot use "github.com/ericchiang/client-go-example/vendor/k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions literal (type "github.com/ericchiang/client-go-example/vendor/k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions) as type "github.com/ericchiang/client-go-example/vendor/k8s.io/client-go/pkg/api/v1".ListOptions in argument to clientset.CoreV1().Pods("").List

Good news, this is fixed on master![2] So installing from that SHA works fine because glide can pull the right version of k8s.io/apimachinery.

$ cat glide.yaml 
package: github.com/ericchiang/client-go-example
import:
- package: k8s.io/client-go
  version: 8b466d64c5da37e0d3985b907d812e1f58cb41cb
$ glide up -v
$ go install
# no error

[0] https://github.com/kubernetes/client-go/blob/v2.0.0-alpha.1/Godeps/Godeps.json
[1] https://github.com/kubernetes/client-go/blob/8b466d64c5da37e0d3985b907d812e1f58cb41cb/examples/out-of-cluster/main.go
[2]

client-go/Godeps/Godeps.json

Line 319 in 8b466d6

"Rev": "7080e31e90e981181435294bca96c80a37db8941"

caesarxuchao · 2017-02-16T06:25:25Z

@ericchiang v2.0.0 doesn't depend on k8s.io/apimachinery, why would glide try to import apimachinery?

ericchiang · 2017-02-16T07:06:26Z

@caesarxuchao ah my bad.

So master imports k8s.io/apimachinery but v2.0.0-alpha.1 doesn't? I guess I'm confused about what the tag branch means.

caesarxuchao · 2017-02-16T07:56:24Z

Right. v2.0.0 is reflect kuberentes 1.5, when k8s.io/apimanichery doesn't exist yet.

sttts · 2017-02-16T08:55:03Z

Maybe most of you have seen this already: @lavalamp wrote a good summary about ways to vendor client-go: https://github.com/kubernetes/client-go/blob/master/INSTALL.md

lavalamp · 2017-02-16T19:14:00Z

Given my changes to README.md & INSTALL.md, and the fact that @caesarxuchao has cut & blessed the v2.0.0 tag and added some detail to CHANGELOG.md, I think we can finally consider this resolved. Separate issues have been filed for a few remaining tasks (e.g. CI for the install instructions).

rubenv · 2017-02-16T19:46:24Z

@lavalamp I've tried following your instructions in INSTALL.md (I'm one of those pesky users who just wants to use go get and doesn't vendor).

I don't think it's even possible. It states:

If you want to write a simple script, don't care about a reproducible client library install, don't mind getting head (which may be less stable than a particular release) and don't share any dependencies with client-go

It's not possible to not share dependencies. For instance, getting a Pod has the following signature (as part of PodInterface):

Get(name string, options meta_v1.GetOptions) (*v1.Pod, error)

meta_v1 is k8s.io/apimachinery/pkg/apis/meta/v1.

That's hardly obscure as a use case.
I might be doing something wrong, but I don't think it's currently possible to write any meaningful application with just go get.

Am I missing something?

lavalamp · 2017-02-16T19:50:56Z

@rubenv thanks for pointing that out. I forgot about it in the stack of other things that were broken.

rubenv · 2017-03-09T09:45:03Z

@lavalamp have you found any time to look into this? It's currently still impossible to use client-go without vendoring.

sttts · 2017-03-09T11:06:48Z

@rubenv this is being worked on in kubernetes/test-infra#1784 and related PRs by @caesarxuchao.

Until that is merged and runs daily, client-go master branch continues to be kind of broken. Not really satisfying now, but we are getting near to a solution.

rubenv · 2017-03-09T12:21:14Z

Cool, subscribing. Thanks a lot for the effort and apologies for the complaining :-)

caesarxuchao · 2017-04-05T17:09:53Z

Now if a user go get k8s.io/client-go in a clean GOPATH (meaning k8s.io/apimachinery doesn't already exist in the GOPATH), (s)he will not hit the problem described in the issue. I think we can consider the issue resolved.

pbarker · 2017-04-05T17:16:02Z

So, I'm sure you all are familiar with https://github.com/ericchiang/k8s, they manage to provide from what I can tell the same functionality with only two dependencies. I am curious from a design perspective what advantage your numerous dependencies have over their two dependency solution?

davecheney mentioned this issue Jan 31, 2017

third-party-resource example does not compile (again) #82

Closed

r2d4 mentioned this issue Jan 31, 2017

Update to k8s-v1.6-alpha.1 kubernetes/minikube#1058

Closed

dongsupark mentioned this issue Feb 2, 2017

kube-apiserver flag redefined: log_dir #19

Closed

sttts mentioned this issue Feb 3, 2017

[PROPOSAL] inverse use of branches: master = stable, develop = nightly snapshots #90

Closed

rubenv mentioned this issue Feb 13, 2017

Cannot use GetOptions{}? #67

Closed

seh mentioned this issue Feb 14, 2017

Document what the best supported path for using client-go is #105

Closed

ericchiang mentioned this issue Feb 15, 2017

errors.IsNotFound not working correctly with client-go #89

Closed

lavalamp closed this as completed Feb 16, 2017

lavalamp reopened this Feb 16, 2017

radeksimko mentioned this issue Feb 24, 2017

vendor: k8s.io/kubernetes/pkg/...@v1.5.3 hashicorp/terraform#12230

Merged

WilliamDenniss added the bug label Mar 15, 2017

caesarxuchao mentioned this issue Mar 21, 2017

We support go get now, update INSTALL.md #157

Merged

caesarxuchao closed this as completed Apr 5, 2017

ffromani mentioned this issue Jul 31, 2018

Conflict between kubevirt and kubernetes types when used in third party projects kubevirt/kubevirt#1395

Closed

Andrea-Campanella mentioned this issue Jun 25, 2019

Duplicate logs when Error or Warning onosproject/onos-config#393

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

k8s.io/client-go vendors k8s.io/apimachinery which breaks build #83

k8s.io/client-go vendors k8s.io/apimachinery which breaks build #83

davecheney commented Jan 31, 2017

lavalamp commented Jan 31, 2017

lavalamp commented Jan 31, 2017

davecheney commented Jan 31, 2017

deads2k commented Jan 31, 2017

caesarxuchao commented Feb 1, 2017

davecheney commented Feb 1, 2017 via email

lavalamp commented Feb 2, 2017

seh commented Feb 2, 2017

deads2k commented Feb 2, 2017

deads2k commented Feb 2, 2017

caesarxuchao commented Feb 3, 2017

deads2k commented Feb 3, 2017

lavalamp commented Feb 3, 2017

errordeveloper commented Feb 6, 2017

errordeveloper commented Feb 6, 2017

deads2k commented Feb 6, 2017

lavalamp commented Feb 13, 2017

rubenv commented Feb 13, 2017

lavalamp commented Feb 13, 2017 via email

ericchiang commented Feb 15, 2017 •

edited

Loading

caesarxuchao commented Feb 16, 2017

ericchiang commented Feb 16, 2017

caesarxuchao commented Feb 16, 2017

sttts commented Feb 16, 2017

lavalamp commented Feb 16, 2017

rubenv commented Feb 16, 2017

lavalamp commented Feb 16, 2017

rubenv commented Mar 9, 2017

sttts commented Mar 9, 2017 •

edited

Loading

rubenv commented Mar 9, 2017

caesarxuchao commented Apr 5, 2017

pbarker commented Apr 5, 2017

k8s.io/client-go vendors k8s.io/apimachinery which breaks build #83

k8s.io/client-go vendors k8s.io/apimachinery which breaks build #83

Comments

davecheney commented Jan 31, 2017

lavalamp commented Jan 31, 2017

lavalamp commented Jan 31, 2017

davecheney commented Jan 31, 2017

deads2k commented Jan 31, 2017

caesarxuchao commented Feb 1, 2017

davecheney commented Feb 1, 2017 via email

lavalamp commented Feb 2, 2017

seh commented Feb 2, 2017

deads2k commented Feb 2, 2017

deads2k commented Feb 2, 2017

caesarxuchao commented Feb 3, 2017

deads2k commented Feb 3, 2017

lavalamp commented Feb 3, 2017

errordeveloper commented Feb 6, 2017

errordeveloper commented Feb 6, 2017

deads2k commented Feb 6, 2017

lavalamp commented Feb 13, 2017

rubenv commented Feb 13, 2017

lavalamp commented Feb 13, 2017 via email

ericchiang commented Feb 15, 2017 • edited Loading

caesarxuchao commented Feb 16, 2017

ericchiang commented Feb 16, 2017

caesarxuchao commented Feb 16, 2017

sttts commented Feb 16, 2017

lavalamp commented Feb 16, 2017

rubenv commented Feb 16, 2017

lavalamp commented Feb 16, 2017

rubenv commented Mar 9, 2017

sttts commented Mar 9, 2017 • edited Loading

rubenv commented Mar 9, 2017

caesarxuchao commented Apr 5, 2017

pbarker commented Apr 5, 2017

ericchiang commented Feb 15, 2017 •

edited

Loading

sttts commented Mar 9, 2017 •

edited

Loading