-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
✨ support config work driver. #256
✨ support config work driver. #256
Conversation
81f461b
to
cd70817
Compare
/hold |
5ab10d3
to
ddf31a0
Compare
/assign @qiujian16 @skeeey |
go.mod
Outdated
open-cluster-management.io/api v0.13.0 | ||
open-cluster-management.io/sdk-go v0.13.0 | ||
open-cluster-management.io/ocm v0.13.1-0.20240325044258-22501d88f774 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we should avoid import ocm
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
removed ocm dependency.
6cafea7
to
0007165
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we enable the integration for cloudevents?
pkg/addonmanager/manager.go
Outdated
} | ||
|
||
// New returns a new addon manager for creating addon agents. | ||
func New(config *rest.Config, opts *ManagerOptions) (AddonManager, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this will break the existing signature of the function. the opts filed should be an optional field. And we should allow that it is not set.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
updated this to make managerOptions be optional.
cmd/example/helloworld/main.go
Outdated
@@ -66,22 +66,30 @@ func newCommand() *cobra.Command { | |||
} | |||
|
|||
func newControllerCommand() *cobra.Command { | |||
o := addonmanager.NewManagerOptions() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we should not impact the existing addon. We could build a new addon example that supports cloudevent, or select one addon as an example. It also ensure that existing addon won't be impacted by this change.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
built new example that supports cloudevent, following the existing examples.
f7a589a
to
4f7535b
Compare
test/integration-test.mk
Outdated
test-kube-integration: ensure-kubebuilder-tools | ||
go test -c ./test/integration/kube -o ./kube-integration.test | ||
./kube-integration.test -ginkgo.slowSpecThreshold=15 -ginkgo.v -ginkgo.failFast | ||
.PHONY: test-kube-integration | ||
|
||
test-cloudevents-integration: ensure-kubebuilder-tools | ||
go test -c ./test/integration/cloudevents -o ./cloudevents-integration.test | ||
./cloudevents-integration.test -ginkgo.slowSpecThreshold=15 -ginkgo.v -ginkgo.failFast | ||
.PHONY: test-cloudevents-integration | ||
|
||
test-integration: test-kube-integration test-cloudevents-integration |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
created separate folder for cloudevents integration test.
make sense? @qiujian16 @skeeey
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah, we may do some configuration like this https://github.com/open-cluster-management-io/ocm/blob/main/.github/workflows/cloudevents-integration.yml#L5
b480f3b
to
a5ebffe
Compare
@@ -90,6 +92,9 @@ func newManifestWork(addonNamespace, addonName, clusterName string, manifests [] | |||
Labels: map[string]string{ | |||
addonapiv1alpha1.AddonLabelKey: addonName, | |||
}, | |||
Annotations: map[string]string{ | |||
common.CloudEventsDataTypeAnnotationKey: payload.ManifestBundleEventDataType.String(), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need to add this annotation all the time? I do not think we should add any additionaly annotation to existing manifestwork.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, we do need this all the time. /cc @skeeey
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we need this annotation to indicate the manifest data type, but I think we may be able to enhance the sdk-go, if we don't find this annotation, we use ManifestBundle
as a default value?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and for ocm/acm cases, it should be always ManifestBundle
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, we can set default value(bundle type) in sdk-go so that we don't need set it here explicitly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
or we can give a fixed value(bundle type) now
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
updated sdk-go dependency, removed this annotation.
pkg/addonmanager/manager.go
Outdated
// New returns a new Manager for creating addon agents. | ||
func New(config *rest.Config) (AddonManager, error) { | ||
return &addonManager{ | ||
// ManagerOptions defines the flags for addon manager |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we put this Option flag in the separated dir specifically for cloudevent?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
put this to separate cloudevents package.
pkg/addonmanager/manager.go
Outdated
// ManifestWork client that implements the ManifestWorkInterface and ManifestWork informer based on different | ||
// driver configuration. | ||
// Refer to Event Based Manifestwork proposal in enhancements repo to get more details. | ||
_, config, err := cloudeventswork.NewConfigLoader(a.options.WorkDriver, a.options.WorkDriverConfig). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could we move all this cloudEvent setting to a separated func instead of the main Start func? or a separated cloudEvent manager? I am concerned about this change on the existing addon-framework breaking existing addons.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Implemented cloudeventsAddonManager
in separate package cloudevents
, so that we don't break existing addons. see: dcf9539
c5e39cb
to
dcf9539
Compare
Signed-off-by: morvencao <lcao@redhat.com>
dcf9539
to
da0d4f9
Compare
Signed-off-by: morvencao <lcao@redhat.com>
Signed-off-by: morvencao <lcao@redhat.com>
kindly ping @zhujian7 @zhiweiyin318 ~ |
Look good to me. |
return fmt.Errorf("method StartWithInformers is not implemented") | ||
} | ||
|
||
func (a *cloudeventsAddonManager) startWithInformers(ctx context.Context, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
actully, how about we have a baseManager interface with StartWithInformers method and an implementation that new all controllers. The Manager interface inlines the baseManager interface, and the manager/cloudeventmanger have different implementation of start func, but share the same implementation in baseManager. Will that work?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated this, not sure if this is more clear.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why we still need this? I thought this should be in the baseManager.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
removed this, PTAL @qiujian16
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I mean all the code to init controllers and start them. I think they are duplicated code and should exist only in baseManager. If we have any change on controllers in the future, we do not need to update multiple times.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We may not do that, due to the limitation source work client from sdk-go as I mentioned at #256 (comment)
we can't get the workinformers.SharedInformerFactory
from clientHolder
(from sdk-go), we can only get workv1informers.Informer
type, so we can't call StartWithInformers
to start the controllers in cloudevents addon manager.
a81869c
to
2b67a4d
Compare
Signed-off-by: morvencao <lcao@redhat.com>
2b67a4d
to
7ad6107
Compare
agentBroadcast: clusters/${MANAGED_CLUSTER_NAME}/agentbroadcast | ||
EOF | ||
|
||
# patch klusterlet-work-agent deployment to use mqtt as workload source driver |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can the klusterlet-work-agent be deployed in the way that the mqtt is the driver directly?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, the latest work agent image already support mqtt and grpc drivers.
return addonfactory.StructToValues(manifestConfig), nil | ||
} | ||
|
||
func AgentHealthProber() *agent.HealthProber { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is there any specific reason that we need to use the work prober to check the health for cloud events driver addons?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the cloudevents driver added in this PR is only used for addon agent deploy, no other difference from the current "kube" driver. Therefore, it use similar prober to make sure addon agent is deployed successfully on managed cluster as "kube" driver. Make sense?
} | ||
return nil | ||
} | ||
|
||
// New returns a new Manager for creating addon agents. | ||
func New(config *rest.Config) (AddonManager, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure if it is better to provider a addon manager factory func, like:
func New(mtype AddonManagerType)(AddonManager, error){
if mtype =="cloudevents"{
return ...
}
...
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this will break existing code that using addonmanager.New(Config)
to create addon manager, right?
we don't want make any change to current code, that's why we keep this unchanged.
|
||
// cloudeventsAddonManager is the implementation of AddonManager with | ||
// the base implementation and cloudevents options | ||
type cloudeventsAddonManager struct { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could you explain the difference between the cloudevents addon manager and the classic addon manager except for the building way of manifestwork client and informer? which I think is good for review.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the difference between the cloudevents addon manager and the classic addon manager is the way to build work client and work informer.
the cloudevents addon manager use cloudevents drivers(mqtt or grpc) to create work informer, so that the manifestwork used to deploy addon agent can be delivered to managed cluster through mqtt or grpc.
no other difference.
contents: read | ||
|
||
jobs: | ||
integration: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we also add an e2e to at least check that the example cloud event addons are available when deploying into a kubernetes cluster?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
e2e test was removed in this commit: d8d4f52
because the work client based on cloudevents drivers (sdk-go) are missing GC machnism, we will revisit and add e2e after GC machnism support for cloudevents drivers.
Signed-off-by: morvencao <lcao@redhat.com>
@qiujian16 @zhujian7 Another look? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/approve
workInformers := clientHolder.ManifestWorkInformer() | ||
|
||
// addonDeployController | ||
err = workInformers.Informer().AddIndexers( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: the indexer part should be put in a shared dir. It is duplicated with the manager.
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: morvencao, qiujian16 The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
added back the e2e testing for helloworld addon using cloudevents. @zhujian7 PTAL |
68ae426
to
f61bf8b
Compare
Signed-off-by: morvencao <lcao@redhat.com>
f61bf8b
to
c7ba5b9
Compare
LGTM |
/lgtm |
e703fc5
into
open-cluster-management-io:main
Summary
This PR aims to enhance the addon-manager by adding work driver configuration support.
This enables the addon manager to deploy addon agents using drivers other than
kube
.Related issue(s)
ref: https://issues.redhat.com/browse/ACM-10423