New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implementing the storage version API, Part I #84016

Closed

caesarxuchao wants to merge 15 commits into kubernetes:master from caesarxuchao:HA-storage-version-api

Member

caesarxuchao commented Oct 16, 2019 •

edited

Loading

/sig-apimachinery
/assign @deads2k @yliaog @lavalamp
/kind feature
KEP

This PR implements:

the storage version API.
Let the kube-aggregator host the storage version API.
kube-apiserver updates the storage version during bootstrap.
A filter that blocks certain requests during kube-apiserver bootstrap until the kube-apiserver has updated the storage versions.
An integration test that shows the filer works, and that the apiserver works properly after the storage version has been updated.

This PR is already huge, so I'll do the remaining tasks in follow-up PRs. Some major follow-ups are:

the filter should block write requests to "apiservices" and "crd".
the garbage collection of obsolete storage version objects.
a test showing that the API works fine in HA cluster.
custom resource handler needs to update the storage version the CR before accepting write requests.

Contributor

k8s-ci-robot commented Oct 16, 2019

@caesarxuchao: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Contributor

k8s-ci-robot commented Oct 16, 2019

@caesarxuchao: The label(s) kind/api cannot be applied. These labels are supported: api-review, community/discussion, community/maintenance, community/question, cuj/build-train-deploy, cuj/multi-user, platform/aws, platform/azure, platform/gcp, platform/minikube, platform/other

In response to this:

/sig-apimachinery
/assign
/kind feature
/kind api
WIP

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot assigned caesarxuchao

k8s-ci-robot added do-not-merge/work-in-progress kind/feature size/XL do-not-merge/release-note-label-needed cncf-cla: yes needs-sig needs-priority area/apiserver kind/api-change sig/api-machinery and removed needs-sig labels

k8s-ci-robot requested review from cblecker and cheftako

October 16, 2019 20:52

caesarxuchao force-pushed the HA-storage-version-api branch 4 times, most recently from 52ada48 to e16e33e Compare

November 6, 2019 02:13

k8s-ci-robot added size/XXL and removed size/XL labels

caesarxuchao force-pushed the HA-storage-version-api branch from e16e33e to 4e33274 Compare

December 21, 2019 00:49

k8s-ci-robot added needs-rebase sig/auth labels

k8s-ci-robot added area/test sig/testing sig/cluster-lifecycle labels

cblecker removed their request for review

February 12, 2020 05:07

caesarxuchao force-pushed the HA-storage-version-api branch from bfb5f7e to 8f896d3 Compare

February 13, 2020 21:39

Contributor

k8s-ci-robot commented Feb 25, 2020

@caesarxuchao: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
pull-kubernetes-e2e-gce-alpha-features	`d8edfa0`	link	`/test pull-kubernetes-e2e-gce-alpha-features`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Member Author

caesarxuchao commented Feb 26, 2020

The pull-kubernetes-e2e-gce-alpha-features failed because the alpha API (storageversion) was not enabled.

Member Author

caesarxuchao commented Feb 26, 2020

To fix pull-kubernetes-e2e-gce-alpha-features, we need to change its job yaml. It only runs some alpha e2e tests.

OTOH, the pull-kubernetes-local-e2e ran all e2e conformance tests. It passed with the storage version feature enabled. So I don't plan to fix pull-kubernetes-e2e-gce-alpha-features.

deads2k reviewed

View reviewed changes

staging/src/k8s.io/apiserver/pkg/storageversion/manager.go

+              	}
+              }
+              func (s *DefaultManager) addGroupResource(group, resource string) {

Contributor

deads2k Feb 26, 2020

prefer schema.GroupResource

deads2k reviewed

View reviewed changes

staging/src/k8s.io/apiserver/pkg/storageversion/manager.go

+              			return
+              		}
+              		klog.V(2).Infof("successfully updated storage version for %v", r.Resource.Name)
+              		s.RemoveResourceInfo(r)

Contributor

deads2k Feb 26, 2020

I suspect we will want to know the full list of ones we handled. Instead, can you mark it as completed.

deads2k reviewed

View reviewed changes

staging/src/k8s.io/apiserver/pkg/storageversion/manager.go

+              	mu             sync.RWMutex
+              	resources      map[*ResourceInfo]struct{}
+              	groupResources map[string]map[string]struct{}

Contributor

deads2k Feb 26, 2020 •

edited

Loading

How about groupResources map[schema.GroupResource]resourceStatus

type resourceStatus struct{
    done bool
    lastErr error
    resourceInfo ResourceInfo
}

for our future inspection.

Contributor

deads2k Feb 26, 2020

This would also eliminate the parallel maps

deads2k reviewed

View reviewed changes

staging/src/k8s.io/apiserver/pkg/storageversion/manager.go

+              // UpdateStorageVersions updates the StorageVersions. If the updates are
+              // successful, following calls to Completed() returns true.
+              func (s *DefaultManager) UpdateStorageVersions(loopbackClientConfig *rest.Config, serverID string) {

Contributor

deads2k Feb 26, 2020

kubeAPIServerClientConfig . On aggregated APIs, loopback means a different thing.

deads2k reviewed

View reviewed changes

staging/src/k8s.io/apiserver/pkg/storageversion/manager.go

+              	for r := range resources {
+              		r.DecodableVersions = decodableVersions(r.EquivalentResourceMapper, r.Group, r.Resource.Name)
+              		if err := updateStorageVersionFor(sc, serverID, r.Group+"."+r.Resource.Name, r.EncodingVersion, r.DecodableVersions); err != nil {
+              			klog.Fatalf("failed to update storage version for %v", r.Resource.Name)

Contributor

deads2k Feb 26, 2020

not Fatal. perhaps utilruntime.HandleError

deads2k reviewed

View reviewed changes

staging/src/k8s.io/apiserver/pkg/storageversion/updater.go

+              	if err != nil && !apierrors.IsNotFound(err) {
+              		return err
+              	}
+              	if err != nil && apierrors.IsNotFound(err) {

Contributor

deads2k Feb 26, 2020

err != nil isn't needed

deads2k reviewed

View reviewed changes

staging/src/k8s.io/apiserver/pkg/storageversion/updater.go

+              	return err
+              }
+              func localUpdateStorageVersion(sv *v1alpha1.StorageVersion, apiserverID, encodingVersion string, decodableVersions []string) {

Contributor

deads2k Feb 26, 2020

please refactor to build a new object to return that can be assigned instead of mutating the input.

deads2k reviewed

View reviewed changes

staging/src/k8s.io/apiserver/pkg/apis/apiserverinternal/v1alpha1/types.go

		metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`

		// Spec is omitted because there is no spec field.

Contributor

deads2k Feb 26, 2020

stub in an empty spec please, so the proto looks normal and the object doesn't look too weird.

Contributor

deads2k commented Feb 26, 2020

@caesarxuchao do you have a separate PR with just the API?

deads2k reviewed

View reviewed changes

staging/src/k8s.io/apiserver/pkg/server/genericapiserver.go

		}

		s.StorageVersion.AddResourceInfo(resources...)

Contributor

deads2k Feb 26, 2020

Does this happen before we start listening? if so, add a comment that this approach is safe because of this.

deads2k reviewed

View reviewed changes

staging/src/k8s.io/kube-aggregator/pkg/apiserver/apiserver.go

-              	apiGroupInfo := apiservicerest.NewRESTStorage(c.GenericConfig.MergedResourceConfig, c.GenericConfig.RESTOptionsGetter)
-              	if err := s.GenericAPIServer.InstallAPIGroup(&apiGroupInfo); err != nil {
+              	rp := apiserverinternalrest.StorageProvider{}
+              	apiGroupInfo, err := rp.NewRESTStorage(c.GenericConfig.MergedResourceConfig, c.GenericConfig.RESTOptionsGetter)

Contributor

deads2k Feb 26, 2020

I expected to see this in the kube-apiserver, not in the aggregator.

Member Author

caesarxuchao Feb 26, 2020

The reason I let the aggregator host it is because the storage version API contains information about all apiservers (kube-apiserver, apiextension apiserver, aggregator apiserver, and custom apiservers). It's analogous to APIService, which is also hosted by the aggrgator. WDYT?

Contributor

deads2k Feb 27, 2020

The reason I let the aggregator host it is because the storage version API contains information about all apiservers (kube-apiserver, apiextension apiserver, aggregator apiserver, and custom apiservers). It's analogous to APIService, which is also hosted by the aggrgator. WDYT?

I think this is closer to the idea of webhook admission configuration. It's provided by the kube-apiserver for consumption and usage by other servers, but aggregation is focused on its single mission

caesarxuchao mentioned this pull request

Add the storage version API #88607

Merged

liggitt removed the api-review label

sanchezl mentioned this pull request

REQUEST: New membership for sanchezl kubernetes/org#1730

Closed

6 tasks

fejta-bot commented May 27, 2020

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

k8s-ci-robot added lifecycle/stale needs-rebase labels

Contributor

k8s-ci-robot commented May 27, 2020

@caesarxuchao: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

roycaihw mentioned this pull request

Serve storage-versions API in kube-apiserver #92064

Merged

roycaihw mentioned this pull request

Add storageversion manager interface #92459

Merged

fejta-bot commented Jul 10, 2020

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

k8s-ci-robot added lifecycle/rotten and removed lifecycle/stale labels

Member Author

caesarxuchao commented Jul 10, 2020

Inherited by #92064 and #92459.

caesarxuchao closed this

roycaihw mentioned this pull request

Move the StorageVersions types to kube-apiserver #93087

Merged

roycaihw mentioned this pull request

Apiserver updates storageversions API and filters certain write requests during bootstrap #93873

Merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/apiserver area/dependency area/test cncf-cla: yes do-not-merge/release-note-label-needed kind/api-change kind/feature lifecycle/rotten needs-priority needs-rebase sig/api-machinery sig/auth sig/cluster-lifecycle sig/testing size/XXL