Add basic integration tests #203

corneliusweig · 2019-06-05T00:23:45Z

This is the beginning of integration tests for krew. It's in early stages and more tests need to be added, but the general look and feel should be clear. Therefore I'm eager to get some feedback what should be improved. In particular, there's some aggressive caching of the krew-index clone.

Additionally, I haven't thought about how these tests are run on Travis. That needs to be sorted out as well.

ahmetb · 2019-06-05T05:43:24Z

integration/integration_test.go

+	validPlugin = "konfig"
+)
+
+func TestInstall(t *testing.T) {


Hmm before these, maybe we should try more fundamental things like exit codes, -h/—help working, unknown subcmds failing, Wdyt? I can go ahead and merge this, and iterate later as well.

ahmetb · 2019-06-05T05:45:21Z

integration/integration_test.go

+	krewTest.WithIndex().Cmd("install", validPlugin).RunOrFailOutput()
+	krewTest.Cmd("remove", validPlugin).RunOrFailOutput()
+
+	indexFiles, err := krewTest.TempDir().List("store")


I think this is fundamentally testing the wrong thing. It’s testing the implementation detail and not the user contract.

Once you uninstall a plugin these should happen and we should test them instead.

Kubectl-foo no longer in PATH

Krew list shows it as uninstalled

Krew info fails

You are totally right here.

integration/integration_test.go

integration/krew/index.go

integration/integration_test.go

integration/krew/krew.go

ahmetb · 2019-06-05T05:55:55Z

Ideally next time please send a much smaller PR (maybe 1 test case) but focus on how we structure the test and how we integrate to travis. That provides a baseline with code that is already used, so we can iterate on it. 😊

corneliusweig · 2019-06-06T10:51:52Z

@ahmetb Thanks for your feedback, that's exactly what I was hoping for. I think for going forward, I will remove the flawed test cases and add the very basic krew help case. Then let's get this to run this on travis.

More test cases should follow in a separate PR. That should be straightforward to write down when everything else is set up.

For faster tests, the index is only cloned once per test run. Additionally, the index is peristently cached between test runs.

Also, optionally link to a krew binary as specified via the KREW_BINARY env varible.

corneliusweig · 2019-06-06T12:20:13Z

Note: marking as ready for review to enable travis, but this is still WIP.

integration/integration_test.go

ahmetb · 2019-06-06T18:07:57Z

pkg/testutil/tempdir.go

+
+// List lists all the files in the subpath of the temp directory.
+// The input path is expected to use '/' as directory separator regardless of the host OS.
+func (td *TempDir) List(path string) ([]string, error) {


seems redundant now. feel free to remove.

ahmetb · 2019-06-06T18:09:13Z

/approve
Looks like once you add a kubectl to the travis env we're good.

This is important, because importing the URI in integration tests triggers the init functions in the cmd package. This has very nasty and hard to find side-effects.

- rename package integration -> test - disable integration tests in run-tests.sh - remove obsolete tempdir helper

- Use a script run-integration-tests.sh for that - Add instructions how to run this script locally

corneliusweig · 2019-06-06T20:55:53Z

@ahmetb There was a little more work needed. But now the integration tests run on travis 🎉 🎊

ahmetb · 2019-06-10T16:51:16Z

moving this forward.
/lgtm
/approve

will open an issue about removing install_kubectl_if_needed.

k8s-ci-robot · 2019-06-10T16:51:23Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahmetb, corneliusweig

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ahmetb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Adding corneliusweig to owners. He has been consistently helping both with krew and krew-index repositories in terms of: - developing plugins himself - taking a stab at krew machinery with large scale code refactors - adding integration test suite to the project - adding more validation and test cases - increasing developer documentation Some of his notable work: - kubernetes-sigs#195 - kubernetes-sigs#183 - kubernetes-sigs#191 - kubernetes-sigs#201 - kubernetes-sigs#202 - kubernetes-sigs#203 - kubernetes-sigs#208 He is familiar with the codebase enough to officially review and approve code. Signed-off-by: Ahmet Alp Balkan <ahmetb@google.com>

Adding corneliusweig to owners. He has been consistently helping both with krew and krew-index repositories in terms of: - developing plugins himself - taking a stab at krew machinery with large scale code refactors - adding integration test suite to the project - adding more validation and test cases - increasing developer documentation Some of his notable work: - #195 - #183 - #191 - #201 - #202 - #203 - #208 He is familiar with the codebase enough to officially review and approve code. Signed-off-by: Ahmet Alp Balkan <ahmetb@google.com>

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jun 5, 2019

k8s-ci-robot requested review from ahmetb and juanvallejo June 5, 2019 00:23

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jun 5, 2019