✨ add cloudevents clients #259

skeeey · 2023-08-10T09:38:28Z

No description provided.

skeeey · 2023-08-16T01:19:39Z

refer to open-cluster-management-io/enhancements#97

skeeey · 2023-08-16T01:20:00Z

/assign @qiujian16

skeeey · 2023-08-16T01:20:27Z

/assign @morvencao

skeeey · 2023-08-16T01:20:49Z

/cc @clyang82 @yanmxa

morvencao · 2023-08-17T03:13:44Z

client/cloudevents/protocol/fake/client.go

+
+	cloudevents "github.com/cloudevents/sdk-go/v2"
+
+	"github.com/cloudevents/sdk-go/v2/event"


cloudevents.Event is alias of event.Event: https://github.com/cloudevents/sdk-go/blob/main/v2/alias.go#L29
Can we just use cloudevents.Event?

using cloudevents.Event instead

morvencao · 2023-08-17T03:16:41Z

client/cloudevents/protocol/fake/client.go

+	}
+
+	for _, evt := range c.receivedEvents {
+		receiver(context.TODO(), evt)


Is this intent to use new context, instead of ctx?

fixed, it should not use a new context

morvencao · 2023-08-17T03:44:07Z

client/cloudevents/agentclient.go

+		return err
+	}
+
+	resources := &payload.ResourceVersionList{Versions: []payload.ResourceVersion{}}


Nit: can we pre-allocate the ResourceVersion slice with length of objects?

resources := &payload.ResourceVersionList{Versions: make([]payload.ResourceVersion, 0, len(objs))}

this can optimize memory usage, when there are two many objects, to avoid slice resize.
this also can apply to status hashes construction.

qiujian16 · 2023-08-17T03:47:44Z

client/cloudevents/interface.go

+	Decode(event *cloudevents.Event) (T, error)
+}
+
+type StatusHashGetter[T ResourceObject] interface {


could this be a func instead of a struct?

using a func instead

qiujian16 · 2023-08-17T03:50:43Z

client/cloudevents/agentclient.go

+	subClient, pubClient cloudevents.Client,
+	lister Lister[T],
+	statusHashGetter StatusHashGetter[T],
+	agentID, clusterName string,


needs explanation on the input var. e.g. what does agentId, clusterName means. It seems to me agentId is actually source. We should not assume the client can only be used by agent. Also what if client is use by the consumer on server, it should not set clusterName when init the client.

this only for agent, description is added

qiujian16 · 2023-08-17T03:55:46Z

client/cloudevents/agentclient.go

+}
+
+// Resync the resources spec of the current agent by sending a spec resync request to sources
+func (c *CloudEventAgentClient[T]) Resync(ctx context.Context, eventGVR types.GroupVersionResource) error {


why resync spec is public but resync status is private.

resync status is used to handle the resync status request, it's a internal logical handler, the outside client doesn't call it, rename the func to respondResyncStatusRequest to avoid confusion

qiujian16 · 2023-08-17T03:57:40Z

client/cloudevents/agentclient.go

+	evt.SetSource(c.agentID)
+	evt.SetExtension(types.ExtensionClusterName, c.clusterName)
+
+	objs, err := c.lister.List(types.ListOptions{Source: types.SourceAll})


who will sent this request? I am not quite clear on why here is sourceall but we do not set cluster either.

the agent will sent this, the agent need to resync its resource spec from all sources when it starts

qiujian16 · 2023-08-17T04:01:17Z

client/cloudevents/sourceclient.go

+	"open-cluster-management.io/api/client/cloudevents/types"
+)
+
+type CloudEventSourceClient[T ResourceObject] struct {


comment please. I think what you mean is this client is used by server but not agent？

yes, it is, the source and agent have different new func now

still need a comment on what CloudEventSourceClient indicates.

qiujian16 · 2023-08-17T04:03:11Z

client/cloudevents/sourceclient.go

+	sourceID         string
+}
+
+func NewCloudEventSourceClient[T ResourceObject](


comments on all input var

there seems to be something missing, e.g how to setup the sub/pub client for agent and source client, will it be same context or they need to use the different context, who takes the responsibility to set the context?

I try to introduce a CloudEventsClient interface to avoid using the cloudevents clients and context directly, please take a look

qiujian16 · 2023-08-17T04:04:33Z

client/cloudevents/sourceclient.go

+	return types.StatusModified, nil
+}
+
+func (c *CloudEventSourceClient[T]) getObj(resourceID string, objs []T) (obj T, exists bool) {


seems not need to be in the struct

removed from the struct

qiujian16 · 2023-08-17T04:06:05Z

client/cloudevents/sourceclient.go

+		Action:               types.ResyncRequestAction,
+	}
+
+	evt := cloudevents.NewEvent()


i think we should have a wrapper of NewEvent with generated uuid and time, and type/source as input.

add a event builder for this

qiujian16 · 2023-08-17T04:06:50Z

client/cloudevents/types/types.go

+	"strings"
+)
+
+const (


please add doc for all const.

qiujian16 · 2023-08-17T04:07:31Z

client/cloudevents/types/types.go

+}
+
+// GroupVersionResource identifies the type of the resource included in the cloud event payload.
+type GroupVersionResource struct {


we cannot use schema.GroupVersionResource directly?

using the schema.GroupVersionResource

morvencao

Looks good to me, thanks.

qiujian16 · 2023-08-21T03:53:16Z

client/cloudevents/agentclient.go

+//   - cloudEventContextGetter gets a cloudevents context when sending a event, different protocol should have its own
+//     context.
+//   - cloudEventsSubClient is a cloudevents client, it is used to receive the event from a broker.
+//   - cloudEventsPubClient is a cloudevents client, it is used to send the event from a broker.


Suggested change

// - cloudEventsPubClient is a cloudevents client, it is used to send the event from a broker.

// - cloudEventsPubClient is a cloudevents client, it is used to send the event to a broker.

qiujian16 · 2023-08-21T03:55:41Z

client/cloudevents/sourceclient.go

+	"open-cluster-management.io/api/client/cloudevents/types"
+)
+
+type CloudEventSourceClient[T ResourceObject] struct {


still need a comment on what CloudEventSourceClient indicates.

qiujian16 · 2023-08-21T03:56:37Z

client/cloudevents/sourceclient.go

+	return nil
+}
+
+// Publish a resource spec from a source to a spec.


Suggested change

// Publish a resource spec from a source to a spec.

// Publish a resource spec from a source to an agent.

how to know which agent to send?

the event includes the cluster name extension, when we send the event, the CloudEventsClient will get the cluster name to determine the event will be sent to which cluster, e.g. for mqtt client, it use the cluster name to create the topic

qiujian16 · 2023-08-21T03:58:21Z

client/cloudevents/sourceclient.go

+// Subscribe the events that are from the agent spec resync request or agent resource status request.
+// For spec resync request, source publish the current resources spec back as response.
+// For resource status request, source receives resource status and handles the status with resource handlers.
+func (c *CloudEventSourceClient[T]) Subscribe(ctx context.Context, handlers ...ResourceHandler[T]) error {


do we need to set the sourceid in the context or it will be don implicitly by the client?

we don't, the sourceid will be get from the event

a client for a certain source should only subscribe to a certain source, isn't it?

yes, for a source, it only subscribe its own resource, for mqtt, it limits by the topic, e.g. /sources/maestro/clusters/+/status

qiujian16 · 2023-08-21T04:00:19Z

client/cloudevents/sourceclient.go

+	sourceID         string
+}
+
+func NewCloudEventSourceClient[T ResourceObject](


there seems to be something missing, e.g how to setup the sub/pub client for agent and source client, will it be same context or they need to use the different context, who takes the responsibility to set the context?

qiujian16

I think should also add a README in the client/cloudevents to describe how the client can be used.

qiujian16 · 2023-08-23T04:07:56Z

client/cloudevents/protocol/protocol.go

+//
+// Available implementations:
+//   - MQTT
+type CloudEventsClient interface {


I am thinking the interface should be like

type CloudEventsClient interface { Sender() cloudevents.Client Subscriber() cloudevents.Client WithContext(ctx context.Context, opts) context.Context }

and the mqtt.options returns the interface with

options.Source()
options.Agent()

nit:

// wording Sender -> Receiver Publisher -> Subscriber // dynamic parameters? WithContext(ctx context.Context, opts ...interface{}) context.Context

Use CloudEventsSourceOptions and CloudEventsAgentOptions instead of this interface when building a cloudeventsclient

Signed-off-by: Wei Liu <liuweixa@redhat.com>

qiujian16

/approve
/lgtm
/hold

I think it is a good start. I am still thinking if statusHashGetter should be in the interface of codec, but we can decide this later. Would @morvencao and @yanmxa to review also? if there is no additional comment, feel free to unhold

openshift-ci · 2023-08-25T01:14:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: qiujian16, skeeey

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [qiujian16]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

skeeey · 2023-08-25T01:44:20Z

I think should also add a README in the client/cloudevents to describe how the client can be used.

I will open a new pr to add the README after we introduced the work part

yanmxa · 2023-08-25T02:21:13Z

Thanks!
Looks good to me.

skeeey · 2023-08-25T05:34:42Z

/unhold

openshift-ci bot added the do-not-merge/work-in-progress label Aug 10, 2023

openshift-ci bot requested review from deads2k and mdelder August 10, 2023 09:38

skeeey force-pushed the cloudevent branch 3 times, most recently from 631b92b to ca9569a Compare August 16, 2023 01:11

openshift-ci bot assigned qiujian16 Aug 16, 2023

openshift-ci bot assigned morvencao Aug 16, 2023

openshift-ci bot requested review from clyang82 and yanmxa August 16, 2023 01:20

skeeey changed the title ~~WIP: add cloudevents clients~~ add cloudevents clients Aug 16, 2023

openshift-ci bot removed the do-not-merge/work-in-progress label Aug 16, 2023

morvencao reviewed Aug 17, 2023

View reviewed changes

qiujian16 reviewed Aug 17, 2023

View reviewed changes

skeeey force-pushed the cloudevent branch from ca9569a to 2d18e00 Compare August 21, 2023 02:15

morvencao reviewed Aug 21, 2023

View reviewed changes

skeeey force-pushed the cloudevent branch from 2d18e00 to ca7e41e Compare August 21, 2023 03:48

qiujian16 reviewed Aug 21, 2023

View reviewed changes

skeeey force-pushed the cloudevent branch 2 times, most recently from 689236b to b584b04 Compare August 22, 2023 03:41

skeeey changed the title ~~add cloudevents clients~~ ✨ add cloudevents clients Aug 22, 2023

qiujian16 reviewed Aug 23, 2023

View reviewed changes

qiujian16 mentioned this pull request Aug 23, 2023

integrate Cloudevent with agent kube-orchestra/maestro#7

Open

add cloud event client

fd49c31

Signed-off-by: Wei Liu <liuweixa@redhat.com>

skeeey force-pushed the cloudevent branch from b584b04 to fd49c31 Compare August 24, 2023 09:14

qiujian16 reviewed Aug 25, 2023

View reviewed changes

openshift-ci bot added do-not-merge/hold lgtm labels Aug 25, 2023

openshift-ci bot added the approved label Aug 25, 2023

openshift-ci bot removed the do-not-merge/hold label Aug 25, 2023

openshift-merge-robot merged commit e598981 into open-cluster-management-io:main Aug 25, 2023
10 checks passed

skeeey deleted the cloudevent branch September 4, 2023 06:53


		cloudevents "github.com/cloudevents/sdk-go/v2"

		"github.com/cloudevents/sdk-go/v2/event"

	// - cloudEventsPubClient is a cloudevents client, it is used to send the event from a broker.
	// - cloudEventsPubClient is a cloudevents client, it is used to send the event to a broker.

	// Publish a resource spec from a source to a spec.
	// Publish a resource spec from a source to an agent.

✨ add cloudevents clients #259

✨ add cloudevents clients #259

Conversation

skeeey commented Aug 10, 2023

skeeey commented Aug 16, 2023

skeeey commented Aug 16, 2023

skeeey commented Aug 16, 2023

skeeey commented Aug 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

morvencao Aug 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

morvencao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qiujian16 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yanmxa Aug 23, 2023 • edited Loading

Choose a reason for hiding this comment

skeeey Aug 24, 2023 • edited Loading

Choose a reason for hiding this comment

qiujian16 left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Aug 25, 2023

skeeey commented Aug 25, 2023

yanmxa commented Aug 25, 2023 • edited Loading

skeeey commented Aug 25, 2023

morvencao Aug 17, 2023 •

edited

Loading

yanmxa Aug 23, 2023 •

edited

Loading

skeeey Aug 24, 2023 •

edited

Loading

yanmxa commented Aug 25, 2023 •

edited

Loading