⛵️ Move k8s.io/test-infra over to dep #5301

cblecker · 2017-11-01T19:20:31Z

~~I have no idea how this will go, but let's try it!~~
This actually seems like it could work!

What I did:

dep init -no-examples
rm -rf Godeps/ _vendor-* vendor/
# Edit Gopkg.toml to ignore github.com/aws/aws-sdk-go* and remove it as a contraint
dep ensure && dep prune
verify/update-bazel.sh

@BenTheElder @stevekuznetsov @fejta @Q-Lee

cjwagner · 2017-11-01T19:54:56Z

/retest

stevekuznetsov · 2017-11-01T20:42:22Z

I wouldn't necessarily have expected the BUILD files to change or for us to pull in 200k more LoC -- is that a misinterpretation of the current vendor state by dep? Are we pulling in sub-packages we don't need?

cblecker · 2017-11-01T22:56:18Z

Doing this without the prune is even worse:

+6,512,203 −19,285

spxtr · 2017-11-01T23:30:57Z

Right now we don't vendor aws stuff for the aws janitor. Is it possible to cleanly express that in go dep?

BenTheElder · 2017-11-01T23:46:08Z

Right now we don't vendor aws stuff for the aws janitor. Is it possible to cleanly express that in go dep?

https://github.com/golang/dep/blob/master/docs/Gopkg.toml.md#ignored

BenTheElder · 2017-11-02T00:07:26Z

BUILD

@@ -49,7 +49,53 @@ filegroup(
        "//testgrid/jenkins_verify:all-srcs",
        "//triage:all-srcs",
        "//velodrome:all-srcs",
-        "//vendor:all-srcs",
+        "//vendor/bitbucket.org/ww/goautoneg:all-srcs",


what changed here? why isn't it just "//vendor:all-srcs"?

I updated the PR body with the commands I used. This was changed by verify/update-bazel.sh.

Found what I needed to fix: kubernetes/kubernetes#51229

this is why we should make verify/ match k/k hack/ :-P

spxtr · 2017-11-02T00:43:03Z

Probably we should add some basic instructions to the readme.

cblecker · 2017-11-02T03:05:39Z

Okay. So this works as-is. I was tinkering with also trying to break the dependency on k/k, as the imports we use are all ones that have been split off into the apimachinery/apiserver repos. However there was some bazel funkiness with that, so it'll have to wait for another PR. This one is big enough!

@kubernetes/test-infra-maintainers PTAL :)

BenTheElder · 2017-11-02T03:23:26Z

LGTM, thanks for all your work on this! Hopefully this will move us to dep sanity :-)
/lgtm
/hold
This PR is big enough that I'd love someone else to /lgtm and /hold cancel after reviewing it themseleves @cjwagner @Kargakis.

krzyzacy · 2017-11-02T05:40:52Z

/cc @ixdy @rmmh
LGTM, and try to get more eyes on this

0xmichalis · 2017-11-02T09:04:37Z

It seems that most of the additions are test code, BUILD, or other documentation files. There is also some interesting cleanup.

/lgtm

k8s-ci-robot · 2017-11-02T09:04:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BenTheElder, cblecker, kargakis

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

~~OWNERS~~ [BenTheElder]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

stevekuznetsov · 2017-11-02T14:59:13Z

100k LOC diff is still weird -- looks like we were not vendoring tests in the past, but now we are? Is that ok? Do we want them?

stevekuznetsov · 2017-11-02T15:01:26Z

docs/dep.md

@@ -0,0 +1,26 @@
+# kubernetes/test-infra dependency management


Can we link to this from the main README so contributors can find it without digging too much?

stevekuznetsov · 2017-11-02T15:17:36Z

Of the additions, the majority are tests:

$ git diff --diff-filter=A --name-only a4c8fbf~1..a4c8fbf | wc -l
432
$ git diff --diff-filter=A --name-only a4c8fbf~1..a4c8fbf | grep -E '_test.go$' | wc -l
281

Then we also pull in a lot of new non-code files:

$ git diff --diff-filter=A --name-only a4c8fbf~1..a4c8fbf | grep -v '_test.go$' | xargs -L 1 basename | sort | uniq -c | sort -rn | head -n 15
     18 README.md
     16 .gitignore
     12 CONTRIBUTING.md
      6 .travis.yml
      6 BUILD
      6 AUTHORS
      5 CONTRIBUTORS
      4 OWNERS
      4 Makefile
      4 .gitattributes
      4 codereview.cfg
      3 README
      3 CHANGELOG.md
      3 AUTHORS.md
      2 util.go

So updated vendored code is small:

$ git diff --diff-filter=AM --name-only a4c8fbf~1..a4c8fbf | grep -v '_test.go$' | grep -E '.go$' | xargs wc -l
   829 vendor/github.com/influxdata/influxdb/client/influxdb.go
   501 vendor/github.com/influxdata/influxdb/client/v2/client.go
    44 vendor/github.com/influxdata/influxdb/errors.go
     1 vendor/github.com/influxdata/influxdb/influxdb.go
    27 vendor/github.com/influxdata/influxdb/models/inline_fnv.go
    38 vendor/github.com/influxdata/influxdb/models/inline_strconv_parse.go
  1916 vendor/github.com/influxdata/influxdb/models/points.go
   116 vendor/github.com/influxdata/influxdb/node.go
    95 vendor/github.com/influxdata/influxdb/pkg/escape/bytes.go
    39 vendor/github.com/petar/GoLLRB/llrb/avgvar.go
    93 vendor/github.com/petar/GoLLRB/llrb/iterator.go
    46 vendor/github.com/petar/GoLLRB/llrb/llrb-stats.go
   456 vendor/github.com/petar/GoLLRB/llrb/llrb.go
    17 vendor/github.com/petar/GoLLRB/llrb/util.go
    13 vendor/github.com/shurcooL/githubql/doc.go
    11 vendor/github.com/shurcooL/graphql/doc.go
    23 vendor/golang.org/x/crypto/curve25519/doc.go
    21 vendor/golang.org/x/crypto/ssh/doc.go
   156 vendor/golang.org/x/net/context/context.go
   341 vendor/golang.org/x/oauth2/oauth2.go
    76 vendor/golang.org/x/sys/unix/syscall.go
   100 vendor/golang.org/x/tools/go/gcexportdata/gcexportdata.go
  1041 vendor/golang.org/x/tools/go/gcimporter15/gcimporter.go
   112 vendor/google.golang.org/appengine/appengine.go
   355 vendor/google.golang.org/appengine/internal/urlfetch/urlfetch_service.pb.go
   210 vendor/google.golang.org/appengine/urlfetch/urlfetch.go
    20 vendor/k8s.io/kubernetes/pkg/util/doc.go
    56 vendor/k8s.io/kubernetes/pkg/util/string_flag.go
    48 vendor/k8s.io/kubernetes/pkg/util/template.go
    72 vendor/k8s.io/kubernetes/pkg/util/trace.go
    27 vendor/k8s.io/kubernetes/pkg/util/umask_windows.go
   147 vendor/k8s.io/kubernetes/pkg/util/util.go
  7047 total

So that actually makes me feel like this is a lot smaller of a move than it seems. Seems like we are updating those dependencies?

cblecker · 2017-11-02T17:04:12Z

@stevekuznetsov dep and godep function differently. When godep saves things to vendor/, it only copies the *.go code itself, and the LICENSE files (omits tests and non-go files). dep on the other hand, checks out the whole git repo to vendor. dep prune then prunes back to only the packages you're actually importing and using + LICENSE files.. but it leaves the whole directory (including non-go files and tests).

cblecker · 2017-11-02T17:29:55Z

/hold cancel

stevekuznetsov · 2017-11-02T17:34:46Z

Right -- the majority of this is due to test and non-code changes, but that is still a couple transitive version bumps in the dependencies from my quick search. Should be fine but did want to point it out.

sdboyer · 2017-11-17T17:24:44Z

also just noting over here that our prune logic is going to become a) automated b) more powerful and c) more granular pretty soon: golang/dep#944

stevekuznetsov · 2017-11-17T17:31:25Z

@sdboyer that sounds awesome!

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 1, 2017

BenTheElder mentioned this pull request Nov 1, 2017

don't build or test vendor in test-infra #5308

Merged

cblecker force-pushed the dep branch 2 times, most recently from 19a8351 to f6065d8 Compare November 1, 2017 22:38

cblecker force-pushed the dep branch from f6065d8 to bb20e3d Compare November 1, 2017 23:04

BenTheElder reviewed Nov 2, 2017

View reviewed changes

cblecker changed the title ~~[WIP] Move k8s.io/test-infra over to dep~~ ⛵️ Move k8s.io/test-infra over to dep Nov 2, 2017

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 2, 2017

cblecker force-pushed the dep branch from 7ed47bb to c1e44f1 Compare November 2, 2017 00:21

BenTheElder mentioned this pull request Nov 2, 2017

vendor tests are running in test-infra (?) #5304

Closed

cblecker requested review from cjwagner and 0xmichalis as code owners November 2, 2017 01:54

BenTheElder mentioned this pull request Nov 2, 2017

rename verify/ to hack/ for consistency with k/k #5199

Merged

cblecker force-pushed the dep branch from 8ee9b3b to 0faa3d6 Compare November 2, 2017 02:41

cblecker mentioned this pull request Nov 2, 2017

🎡 Remove k8s.io/kubernetes dependency #5311

Closed

cblecker added 3 commits November 1, 2017 19:44

Exempt vendor/ from pylint

d00db70

Always create vendor/BUILD in verify/update-bazel.sh

aab3752

⛵️ Move k8s.io/test-infra over to dep

a4c8fbf

cblecker force-pushed the dep branch 2 times, most recently from 1184e9a to 7a05c25 Compare November 2, 2017 02:54

k8s-ci-robot assigned BenTheElder Nov 2, 2017

k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Nov 2, 2017

k8s-ci-robot requested review from ixdy and rmmh November 2, 2017 05:40

k8s-ci-robot assigned 0xmichalis Nov 2, 2017

0xmichalis approved these changes Nov 2, 2017

View reviewed changes

stevekuznetsov reviewed Nov 2, 2017

View reviewed changes

Add basic docs for using dep

6bf1aa7

cblecker force-pushed the dep branch from 7a05c25 to 6bf1aa7 Compare November 2, 2017 17:06

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 2, 2017

k8s-ci-robot merged commit c258100 into kubernetes:master Nov 2, 2017

cblecker deleted the dep branch November 2, 2017 17:38

BenTheElder added kind/velocity-improvement ¯\_(ツ)_/¯ ¯\\\_(ツ)_/¯ labels Nov 2, 2017

BenTheElder mentioned this pull request Nov 3, 2017

bazel build failes locally #5331

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

⛵️ Move k8s.io/test-infra over to dep #5301

⛵️ Move k8s.io/test-infra over to dep #5301

cblecker commented Nov 1, 2017 •

edited

Loading

cjwagner commented Nov 1, 2017

stevekuznetsov commented Nov 1, 2017

cblecker commented Nov 1, 2017

spxtr commented Nov 1, 2017

BenTheElder commented Nov 1, 2017

BenTheElder Nov 2, 2017

cblecker Nov 2, 2017

cblecker Nov 2, 2017

BenTheElder Nov 2, 2017

spxtr commented Nov 2, 2017

cblecker commented Nov 2, 2017

BenTheElder commented Nov 2, 2017 •

edited

Loading

krzyzacy commented Nov 2, 2017

0xmichalis commented Nov 2, 2017

k8s-ci-robot commented Nov 2, 2017

stevekuznetsov commented Nov 2, 2017

stevekuznetsov Nov 2, 2017

cblecker Nov 2, 2017

stevekuznetsov commented Nov 2, 2017

cblecker commented Nov 2, 2017

cblecker commented Nov 2, 2017

stevekuznetsov commented Nov 2, 2017

sdboyer commented Nov 17, 2017

stevekuznetsov commented Nov 17, 2017

		@@ -0,0 +1,26 @@
		# kubernetes/test-infra dependency management

⛵️ Move k8s.io/test-infra over to dep #5301

⛵️ Move k8s.io/test-infra over to dep #5301

Conversation

cblecker commented Nov 1, 2017 • edited Loading

cjwagner commented Nov 1, 2017

stevekuznetsov commented Nov 1, 2017

cblecker commented Nov 1, 2017

spxtr commented Nov 1, 2017

BenTheElder commented Nov 1, 2017

BenTheElder Nov 2, 2017

Choose a reason for hiding this comment

cblecker Nov 2, 2017

Choose a reason for hiding this comment

cblecker Nov 2, 2017

Choose a reason for hiding this comment

BenTheElder Nov 2, 2017

Choose a reason for hiding this comment

spxtr commented Nov 2, 2017

cblecker commented Nov 2, 2017

BenTheElder commented Nov 2, 2017 • edited Loading

krzyzacy commented Nov 2, 2017

0xmichalis commented Nov 2, 2017

k8s-ci-robot commented Nov 2, 2017

stevekuznetsov commented Nov 2, 2017

stevekuznetsov Nov 2, 2017

Choose a reason for hiding this comment

cblecker Nov 2, 2017

Choose a reason for hiding this comment

stevekuznetsov commented Nov 2, 2017

cblecker commented Nov 2, 2017

cblecker commented Nov 2, 2017

stevekuznetsov commented Nov 2, 2017

sdboyer commented Nov 17, 2017

stevekuznetsov commented Nov 17, 2017

cblecker commented Nov 1, 2017 •

edited

Loading

BenTheElder commented Nov 2, 2017 •

edited

Loading