Add a kubernetes driver #765

jlegrone · 2019-06-10T22:54:24Z

Closes #763

This PR includes new integration tests that are behind a build flag. To run them, you'll need to be authenticated to a kubernetes cluster with permissions in the default namespace. Then run go test with the integration tag:

go test -tags=integration ./...

jlegrone · 2019-06-11T00:40:04Z

cc @technosophos -- I know you've done some work towards this in #560

pkg/driver/kubernetes_driver_test.go

pkg/driver/kubernetes_driver.go

pkg/driver/kubernetes_driver_test.go

This is behind a build flag. Usage: go test -tags=integration ./...

sbawaska

I gave this a spin trying to install the riff cnab bundle. Except for the one issue mentioned below, the install was successful! 👍

sbawaska · 2019-06-12T05:50:01Z

pkg/driver/kubernetes.go

+		Labels:       labelMap,
+	}
+	// Mount SA token if a non-zero value for ServiceAccountName has been specified
+	mountServiceAccountToken := k.ServiceAccountName != ""


Suggested change

mountServiceAccountToken := k.ServiceAccountName != ""

mountServiceAccountToken := k.ServiceAccountName == ""

As per the docs credential for the default service account is mounted at /var/run/secrets/kubernetes.io/serviceaccount/token. When not using a service account, the default service account should be created. I got the exception Could not get kubernetes configuration: failed to read token file \"/var/run/secrets/kubernetes.io/serviceaccount/token\": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory when I gave this a spin.

This was an intentional decision, as we don't know that the kubernetes driver is being used to run invocation images that deploy to the same cluster, or even to kubernetes. I would like to avoid giving permissions to invocation images that haven't been explicitly declared as required by the bundle.

That said, duffle could afford to be more opinionated and set a default service account here:
https://github.com/deislabs/duffle/blob/206a2bec45ddddb9dcf558581f27249a80db835f/pkg/driver/lookup.go#L14-L15

Thoughts?

@sbawaska also did you try mounting kubeconfig as a credential as requested by the riff bundle?

we don't know that the kubernetes driver is being used to run invocation images that deploy to the same cluster, or even to kubernetes.

To me, the use case for this driver is to run the invocationImage in cluster where the bundle is to be installed. I think we should let the default docker driver handle the deploying to non-kubernetes environments.

I would like to avoid giving permissions to invocation images that haven't been explicitly declared as required by the bundle.

I think the missing piece here is a lack of contract between the driver and the bundle for the parameters needed by the driver. The SERVICE_ACCOUNT and KUBE_NAMESPACE env variables here are env variables for duffle itself, not the invocation image. The bundle can only specify parameters for invocationImages, not duffle.

I like this quote from Allan Kay:

Simple things should be simple, complex things should be possible.

In this context it would mean: provide defaults, but allow the bundle author to override the defaults. more concretely, I think we should:

provide default as the default value for SERVICE_ACCOUNT. The user could use a different service account if desired by setting the env variable.

provide default namespace from kubeconfig for KUBE_NAMESPACE, again, the user could specify a different namespace if desired.

On second though, we could iterate on these issues to make the experience better, and work on some contract between driver and duffle for parameters needed. Let's merge this!

sbawaska · 2019-06-12T05:53:17Z

pkg/driver/kubernetes.go

+// Run executes the operation inside of the invocation image.
+func (k *KubernetesDriver) Run(op *driver.Operation) error {
+	if k.Namespace == "" {
+		return fmt.Errorf("KUBE_NAMESPACE is required")


I don't think this should be mandatory, it should be ok to run the installer pod in the default namespace. One less thing for the user to worry about.

Similar idea as #765 (comment). Credentials mounted in invocation images are readable by anyone with secret list permissions in the target namespace. In a multi-tenant cluster using the default namespace would be really inappropriate IMO, so from the driver's perspective I think it's best to require a value to be set explicitly.

Installer implementations like duffle may still set a default namespace as this is a public field.

sbawaska · 2019-06-12T05:56:29Z

pkg/driver/kubernetes.go

+func (k *KubernetesDriver) Config() map[string]string {
+	return map[string]string{
+		"KUBE_NAMESPACE":  "Kubernetes namespace in which to run the invocation image",
+		"SERVICE_ACCOUNT": "Kubernetes service account to be mounted by the invocation image (if empty, no service account token will be mounted)",


Suggested change

"SERVICE_ACCOUNT": "Kubernetes service account to be mounted by the invocation image (if empty, no service account token will be mounted)",

"SERVICE_ACCOUNT": "Kubernetes service account to be mounted by the invocation image (if empty, default service account token will be mounted)",

please see below for the rationale.

glyn

LGTM except for @sbawaska's points.

pkg/driver/kubernetes.go

glyn

@sbawaska is happy to merge this PR now. We can add more features later.

silvin-lubecki · 2019-06-13T21:51:45Z

pkg/driver/kubernetes.go

+
+	conf, err := clientcmd.BuildConfigFromFlags(settings["MASTER_URL"], kubeconfig)
+	if err != nil {
+		panic(err)


KEEP CALM AND DON'T PANIC 😺
Wouldn't it be better to modify the driver.Configurable interface so SetConfig can return an error?

I actually did exactly this and then realized the interface is defined in cnab-go. I'm definitely in favor of this, just has to be a separate pr.

See #765 (comment)

Yeah, this can be addressed in a follow-on PR when we can fix the interface.

Sure I'm fine with a follow-up 👍

technosophos

I am with @glyn on this -- there are a few things that should be sorted out in follow-up PRs, but what you have here is an excellent place to start. We should get this merged.

technosophos · 2019-06-14T18:03:53Z

/brig run

technosophos · 2019-06-14T20:01:19Z

/brig run

PR cnabio#765 forgot to commit changes to vendor to sync it to the changes to Gopkg.lock. This syncs the vendor directory with those changes.

Build Fixes from PR #765

jlegrone force-pushed the feature/k8s-driver branch from 5a6700f to 2377cf0 Compare June 11, 2019 00:44

feat(driver): support kubernetes

2b94043

jlegrone force-pushed the feature/k8s-driver branch from 2377cf0 to 2b94043 Compare June 11, 2019 00:45

jlegrone commented Jun 11, 2019

View reviewed changes

pkg/driver/kubernetes_driver_test.go Outdated Show resolved Hide resolved

glyn suggested changes Jun 11, 2019

View reviewed changes

jlegrone added 2 commits June 11, 2019 14:59

Improve code comments and fix config help

602b001

Remove _driver from file names

474db9c

jlegrone force-pushed the feature/k8s-driver branch 2 times, most recently from 2d1ee4a to 729dd18 Compare June 11, 2019 21:27

jlegrone added 3 commits June 11, 2019 18:20

Add kubernetes driver integration test

0895168

This is behind a build flag. Usage: go test -tags=integration ./...

Add kubernetes driver unit test

d6d1504

Move job/log watching to separate function, remove todos

206a2be

jlegrone force-pushed the feature/k8s-driver branch from 4cfc7ee to 206a2be Compare June 11, 2019 22:20

sbawaska suggested changes Jun 12, 2019

View reviewed changes

glyn suggested changes Jun 12, 2019

View reviewed changes

sbawaska approved these changes Jun 13, 2019

View reviewed changes

glyn mentioned this pull request Jun 13, 2019

k8s driver should not result in an invocation image being restarted #771

Closed

glyn reviewed Jun 13, 2019

View reviewed changes

pkg/driver/kubernetes.go Show resolved Hide resolved

glyn approved these changes Jun 13, 2019

View reviewed changes

glyn requested review from silvin-lubecki and technosophos June 13, 2019 14:48

silvin-lubecki reviewed Jun 13, 2019

View reviewed changes

technosophos approved these changes Jun 14, 2019

View reviewed changes

technosophos merged commit 4dccb95 into cnabio:master Jun 14, 2019

jlegrone deleted the feature/k8s-driver branch June 14, 2019 20:20

carolynvs pushed a commit to carolynvs/duffle that referenced this pull request Jun 17, 2019

Sync vendor to PR cnabio#765

f62c521

PR cnabio#765 forgot to commit changes to vendor to sync it to the changes to Gopkg.lock. This syncs the vendor directory with those changes.

carolynvs mentioned this pull request Jun 17, 2019

Build Fixes from PR #765 #775

Merged

carolynvs pushed a commit that referenced this pull request Jun 17, 2019

Merge pull request #775 from carolynvs/sync-vendor

0ce61af

Build Fixes from PR #765

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a kubernetes driver #765

Add a kubernetes driver #765

jlegrone commented Jun 10, 2019 •

edited

Loading

jlegrone commented Jun 11, 2019

sbawaska left a comment

sbawaska Jun 12, 2019

jlegrone Jun 12, 2019

jlegrone Jun 12, 2019

sbawaska Jun 12, 2019 •

edited

Loading

sbawaska Jun 13, 2019

sbawaska Jun 12, 2019

jlegrone Jun 12, 2019 •

edited

Loading

sbawaska Jun 12, 2019

glyn left a comment

glyn left a comment

silvin-lubecki Jun 13, 2019

jlegrone Jun 14, 2019 •

edited

Loading

technosophos Jun 14, 2019

silvin-lubecki Jun 14, 2019

technosophos left a comment

technosophos commented Jun 14, 2019

technosophos commented Jun 14, 2019

	mountServiceAccountToken := k.ServiceAccountName != ""
	mountServiceAccountToken := k.ServiceAccountName == ""

	"SERVICE_ACCOUNT": "Kubernetes service account to be mounted by the invocation image (if empty, no service account token will be mounted)",
	"SERVICE_ACCOUNT": "Kubernetes service account to be mounted by the invocation image (if empty, default service account token will be mounted)",

Add a kubernetes driver #765

Add a kubernetes driver #765

Conversation

jlegrone commented Jun 10, 2019 • edited Loading

jlegrone commented Jun 11, 2019

sbawaska left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbawaska Jun 12, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jlegrone Jun 12, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

glyn left a comment

Choose a reason for hiding this comment

glyn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jlegrone Jun 14, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

technosophos left a comment

Choose a reason for hiding this comment

technosophos commented Jun 14, 2019

technosophos commented Jun 14, 2019

jlegrone commented Jun 10, 2019 •

edited

Loading

sbawaska Jun 12, 2019 •

edited

Loading

jlegrone Jun 12, 2019 •

edited

Loading

jlegrone Jun 14, 2019 •

edited

Loading