Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

When bootstrapping a client cert, store it with other client certs #62152

Merged
merged 1 commit into from
Apr 23, 2018
Merged

When bootstrapping a client cert, store it with other client certs #62152

merged 1 commit into from
Apr 23, 2018

Conversation

smarterclayton
Copy link
Contributor

@smarterclayton smarterclayton commented Apr 5, 2018 •
edited

The kubelet uses two different locations to store certificates on
initial bootstrap and then on subsequent rotation:

  • bootstrap: certDir/kubelet-client.(crt|key)
  • rotation: certDir/kubelet-client-(DATE|current).pem

Bootstrap also creates an initial node.kubeconfig that points to the
certs. Unfortunately, with short rotation the node.kubeconfig then
becomes out of date because it points to the initial cert/key, not the
rotated cert key.

Alter the bootstrap code to store client certs exactly as if they would
be rotated (using the same cert Store code), and reference the PEM file
containing cert/key from node.kubeconfig, which is supported by kubectl
and other Go tooling. This ensures that the node.kubeconfig continues to
be valid past the first expiration.

Example:

bootstrap:
  writes to certDir/kubelet-client-DATE.pem and symlinks to certDir/kubelet-client-current.pem
  writes node.kubeconfig pointing to certDir/kubelet-client-current.pem
rotation:
  writes to certDir/kubelet-client-DATE.pem and symlinks to certDir/kubelet-client-current.pem

This will also allow us to remove the wierd "init store with bootstrap cert" stuff, although I'd prefer to do that in a follow up.

@mikedanese @liggitt as per discussion on Slack today

The `--bootstrap-kubeconfig` argument to Kubelet previously created the first bootstrap client credentials in the certificates directory as `kubelet-client.key` and `kubelet-client.crt`.  Subsequent certificates created by cert rotation were created in a combined PEM file that was atomically rotated as `kubelet-client-DATE.pem` in that directory, which meant clients relying on the `node.kubeconfig` generated by bootstrapping would never use a rotated cert.  The initial bootstrap certificate is now generated into the cert directory as a PEM file and symlinked to `kubelet-client-current.pem` so that the generated kubeconfig remains valid after rotation.

@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Apr 5, 2018
@smarterclayton
Copy link
Contributor Author

smarterclayton commented Apr 5, 2018 •
edited

/test pull-kubernetes-verify

1 similar comment
@smarterclayton
Copy link
Contributor Author

/test pull-kubernetes-verify

@mikedanese
Copy link
Member

We've noticed weird corruptions of the kubeconfig after a bootstrap, e.g. "server.go:140] tls: failed to find any PEM data in certificate input". I wonder if the recovery in filestore will do better then the old LoadClientCert.

@smarterclayton
Copy link
Contributor Author

The atomic swap should help.

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Apr 6, 2018
mikedanese
mikedanese reviewed Apr 9, 2018
View reviewed changes
pkg/kubelet/certificate/bootstrap/bootstrap.go Outdated
}
}
}()
}
if keyData == nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

len(keyData) == 0. Should we do any other integrity checks here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably. Let me noodle it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated with sanity checking that it's a valid private key at minimum.

@mikedanese
Copy link
Member

One comment, otherwise LGTM.
/approve

@smarterclayton
When bootstrapping a client cert, store it with other client certs
3689593 
The kubelet uses two different locations to store certificates on
initial bootstrap and then on subsequent rotation:

* bootstrap: certDir/kubelet-client.(crt|key)
* rotation:  certDir/kubelet-client-(DATE|current).pem

Bootstrap also creates an initial node.kubeconfig that points to the
certs. Unfortunately, with short rotation the node.kubeconfig then
becomes out of date because it points to the initial cert/key, not the
rotated cert key.

Alter the bootstrap code to store client certs exactly as if they would
be rotated (using the same cert Store code), and reference the PEM file
containing cert/key from node.kubeconfig, which is supported by kubectl
and other Go tooling. This ensures that the node.kubeconfig continues to
be valid past the first expiration.
@mikedanese
Copy link
Member

@awly

@mikedanese
Copy link
Member

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 23, 2018
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mikedanese, smarterclayton

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@smarterclayton
Copy link
Contributor Author

/retest

1 similar comment
@smarterclayton
Copy link
Contributor Author

/retest

@k8s-github-robot
Copy link

Automatic merge from submit-queue (batch tested with PRs 63001, 62152, 61950). If you want to cherry-pick this change to another branch, please follow the instructions here.

@k8s-github-robot k8s-github-robot merged commit 939c078 into kubernetes:master Apr 23, 2018
@mikedanese mikedanese mentioned this pull request Jun 13, 2018
Merged
@weiwei04 weiwei04 mentioned this pull request Jul 16, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mikedanese mikedanese mikedanese left review comments

@liggitt liggitt Awaiting requested review from liggitt

@thockin thockin Awaiting requested review from thockin

Assignees

@mikedanese mikedanese

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@smarterclayton @mikedanese @k8s-ci-robot @k8s-github-robot