-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: crd provisioner use cache #176
Conversation
6c05ad8
to
6578e37
Compare
pkg/azuredisk/azuredisk_v2.go
Outdated
@@ -297,6 +298,7 @@ func (d *DriverV2) StartControllersAndDieOnExit(ctx context.Context) { | |||
} | |||
|
|||
sharedState := controller.NewSharedState() | |||
sharedState.RegisterCachedClient(mgr.GetClient()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider passing the mgr.GetClient()
to the NewSharedState()
function instead of through a new interface method.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I actually meant to insert this line once the controller manager acquires the leader lock so that the cached client is not registered for those that failed to acquire the lock.
<-mgr.Elected() |
Wouldn't it make sense to initialize sharedState first and post-append cached client in that case?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nvm. As long as we register sharedState to crdProvisioner only when leader lock is acquired, that should do.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ideally, sharedState
should be fully initialized (including setting the cached client) before being passed to its dependents through their constructor functions. Post-initialization inject of objects in makes it difficult to track down dependencies and make nil-reference bugs possible since it may not be clear at the point of use whether one can assume it was initialized. Consumers of this field have to deal with the possibility that it may be nil, increasing complexity.
I am equally concerned about the RegisterSharedState
method on the CrdProvisioner
but I don't see a good way of handling this unless we drastically change how the driver and controllers are initialized.
pkg/provisioner/crdprovisioner.go
Outdated
if c.controllerSharedState == nil { | ||
return nil | ||
} | ||
return c.getCachedClient() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
return c.controllerSharedState.GetCachedClient()
pkg/provisioner/crdprovisioner.go
Outdated
namespace string | ||
azDiskClient azDiskClientSet.Interface | ||
namespace string | ||
controllerSharedState *controller.SharedState |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider renaming this field to simply sharedState
.
pkg/controller/pod.go
Outdated
@@ -55,7 +54,7 @@ func (r *ReconcilePod) Reconcile(ctx context.Context, request reconcile.Request) | |||
var pod corev1.Pod | |||
klog.V(5).Infof("Reconcile pod %s.", request.Name) | |||
podKey := getQualifiedName(request.Namespace, request.Name) | |||
if err := r.client.Get(ctx, request.NamespacedName, &pod); err != nil { | |||
if err := r.controllerSharedState.cachedClient.Get(ctx, request.NamespacedName, &pod); err != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider renaming controllerSharedState
to simply sharedState
.
e4403c4
to
3e4d941
Compare
I have overall concerns with this approach. While it is convenient and desirable to share resources (i.e. object cache to reduce direct API server calls), The The post-Mandalay plan intends to partition the work done by the controllers (specifically for volume creation and attachment management) across multiple nodes. We may repackage the controllers into a separate image at this point. It may also be possible regardless of packaging that the driver and controllers may no longer coexist in the same process reducing the opportunity for shared state. It would also be beneficial to replace our current polling in the I am of the opinion that the @abhisheksinghbaghel @landreasyan Do you have any thoughts about this? |
I am for this opinion on adding an informer and revising the current polling method to use watch events. I think we should add caching to And maybe we can set up go channels for informers' event handlers to use to signal the |
The informers provide caching and use watches to drive the cache to eventual consistency. The cached objects are accessible through lister interfaces. e.g. obj, err := azVolumeAttachmentInformer.Lister().AzVolumeAttachments("azure-disk-csi").Get(attachmentName)
Yes. We could use the informers' event handlers to implement an interface like: func (w *conditionWatcher) WatchForObjectCondition(ctx context.Context, obj runtime.Object, conditionFunc func (obj runtime.Object) bool) (runtime.Object, error) <- chan
Internally, |
Oh, I didn't mean to say that we should add an additional caching mechanism other than the one provided by the informer. I meant that 1) necessity of caching and 2) inability for |
One additional point here is that the watch-based wait should be transparent to the |
I actually meant to say type CrdProvisioner struct {
...
conditionWatcher conditionWatcher
}
type waitEntry struct {
conditionFunc func(obj runtime.Object) (bool, error)
waitChan chan <-(runtime.Object, error)
}
type conditionWatcher struct {
informerMap map[Type]cache.SharedIndexInformer // maps object type to its informer
waitMap sync.Map // maps namespaced name to waitEntry
}
func (w *conditionWatcher) WatchForObjectCondition(ctx context.Context, objectName string, conditionFunc func (obj runtime.Object) bool) (runtime.Object, error) {
waitChan := make(chan (runtimeObject, error), 1)
entry := waitEntry {
conditionFunc: conditionFunc,
waitChan: waitChan
}
conditionWatcher.waitMap.Store(qualifiedName, conditionFunc)
defer conditionWatcher.waitMap.Delete(qualifiedName)
select {
case <-ctx.Done():
errMsg := fmt.Sprintf("context timeout for %v (%s)", reflect.TypeOf(obj), qualifiedName)
klog.Error(errMsg)
return nil, Status.Errorf(codes.Internal, errMsg)
case obj, err := <- waitChan:
return obj, err
default:
return nil, status.Errorf(Aborted, "unexpected error")
}
} And as you said, at event handling, we would check the map, evaluate the conditionFunc, and return its value to the stored channel. And for every crdProvisioner |
This looks pretty good, but the fact you're doing the wait in I had originally thought to have the caller wait on the result channel directly, but this means it also has to deal with context cancelation/timeout separately. I like how your solution handles both - it makes it easier to use. One enhancement I can think of to my original proposal would be to have type conditionWaiter struct {
objectType string
objectName string
watcher conditionWatcher
resultChan chan <- (runtime.Object, error)
}
func (w *conditionWaiter) Wait(ctx context.Context) (runtime.Object, error) {
select {
case <-ctx.Done():
errMsg := fmt.Sprintf("context timeout for %s (%s)", w.objectType, w.objectName )
klog.Error(errMsg)
return nil, Status.Errorf(codes.DeadlineExceeded, errMsg)
case obj, err := <- w.resultChan:
return obj, err
default:
return nil, status.Errorf(codes.Internal, "unexpected error")
}
}
func (w *conditionWaiter) Close() {
w.watcher.waitMap.Delete(w.objectName)
}
func (w *conditionWatcher) WatchForObjectCondition(obj runtime.Object, conditionFunc func (obj runtime.Object) bool) (conditionWaiter, error) Here's an example of usage: func (p* CrdProvisioner) CreateVolume(ctx context.Context, ...) (*v1alpha1.AzVolumeStatusParams, error) {
newAzVolume = &v1alpha1.AzVolume{ Metadata: metav1.ObjectMetadata{ Name: newVolumeName,, Namespace: ns }, ... }
waiter, err := p.conditionWatcher.WatchForCondition(newAzVolume, func (obj runtime.Object) bool {
azVol := obj.(*v1alpha1.AzVolume)
if azVol.Status.Detail != nil {
return true
}
return false
})
if err != nil {
return nil, err
}
defer waiter.Close()
// Create AzVolume CRI here
...
newAzVolume, err = waiter.Wait(ctx)
if err != nil {
return nil, err
}
// Handle AzError returns here
...
return newAzVolume.Status.Detail.ResponseObject, nil I am fine with either approach, though. |
replaced with #199 |
What type of PR is this?
/kind feature
What this PR does / why we need it:
Which issue(s) this PR fixes:
Fixes #
Requirements:
Special notes for your reviewer:
Release note: