Add placement DecisionStrategy API #242

serngawy · 2023-05-25T16:25:07Z

No description provided.

qiujian16 · 2023-05-26T01:17:53Z

cluster/v1beta1/types_placement.go

+	// Label to be added to the created placement Decisions
+	// +kubebuilder:validation:Required
+	// +required
+	Label string `json:"labels,omitempty"`


what does the value of this field look like

this is label value so the placementDecision will have label key as "cluster.open-cluster-management.io/decisiongrouplabel" and the user define here the value

I see, label may cause confusion. May be better to change the field to name or groupName as we don't want the user to add multiple labels.

qiujian16 · 2023-05-26T01:21:59Z

cluster/v1beta1/types_placement.go

+
+	// Decision group label that is defined in the DecisionStrategy's DecisionGroup.
+	// +optional
+	DecisionGroupLabel string `json:"decisionGroupLabel"`


shouldn't this contain key/value?

the key is API constant "cluster.open-cluster-management.io/decisiongrouplabel"

qiujian16 · 2023-05-26T01:23:42Z

cluster/v1beta1/types_placement.go

+	// +kubebuilder:validation:Enum=None;"5%";"10%";"15%";"20%";"25%";"30%";"50%";"100%"
+	// +kubebuilder:default:=None
+	// +optional
+	ClusterPercentagePerDecisionGroup string `json:"clusterPercentagePerDecisionGroup,omitempty"`


It is actually MaxCountPerGroup right? I still think we should only have one field that can be set as IntOrString

okay, I wasn't sure that we can define enum and int for type IntOrString, will push a fix for it.

https://github.com/open-cluster-management-io/api/blob/main/addon/v1alpha1/types_clustermanagementaddon.go#L260

cluster/v1beta1/types_placementdecision.go

qiujian16 · 2023-05-29T06:59:44Z

cluster/v1beta1/types_placementdecision.go

 	PlacementLabel string = "cluster.open-cluster-management.io/placement"
+	// decision group index.
+	DecisionGroupIndexLabel string = "cluster.open-cluster-management.io/decisiongroupindex"


Suggested change

DecisionGroupIndexLabel string = "cluster.open-cluster-management.io/decisiongroupindex"

DecisionGroupIndexLabel string = "cluster.open-cluster-management.io/decision-group-index"

qiujian16 · 2023-05-29T07:00:33Z

cluster/v1beta1/types_placementdecision.go

+	// decision group index.
+	DecisionGroupIndexLabel string = "cluster.open-cluster-management.io/decisiongroupindex"
+	// decision group label.
+	DecisionGroupName string = "cluster.open-cluster-management.io/decisiongroupname"


Suggested change

DecisionGroupName string = "cluster.open-cluster-management.io/decisiongroupname"

DecisionGroupName string = "cluster.open-cluster-management.io/decision-group-name"

kind of confusing the label key is decision-group-name not decision-group-labelvalue ? if yes, better to keep the fields up as group Name then ?

the label key is decision-group-name, while what you want user to set is the label value of this key. Group name is not clear since it may imply this is a resource name or a certain field, but it is actually a label value of the certain key

qiujian16 · 2023-05-29T07:02:52Z

cluster/v1beta1/types_placement.go

+	// +kubebuilder:validation:Enum=None;"5%";"10%";"15%";"20%";"25%";"30%";"50%";"100%"
+	// +kubebuilder:default:=None
+	// +optional
+	ClusterPercentagePerDecisionGroup string `json:"clusterPercentagePerDecisionGroup,omitempty"`


https://github.com/open-cluster-management-io/api/blob/main/addon/v1alpha1/types_clustermanagementaddon.go#L260

qiujian16 · 2023-05-29T07:04:48Z

cluster/v1beta1/types_placement.go

+}
+
+// Group the created placementDecision into decision groups based on the number of clusters per decision group.
+type DecisionStrategy struct {


I would think this like

type DecisionStrategy struct { GroupStrategy GroupStrategy } type GroupStrategy struct { DecisionGroups []DecisionGroup `json:"decisionGroups,omitempty"` ClustersPerDecisionGroup string `json:"clustersPerDecisionGroup,omitempty"` }

Do we need to have nested struct ? I believe decisionStrategy as single struct suppose to be good unless we will extended with something related.

nested will help us to extend in the future

qiujian16 · 2023-05-31T13:21:55Z

cluster/v1beta1/types_placement.go

+	DecisionStrategy DecisionStrategy `json:"decisionStrategy,omitempty"`
+}
+
+// Subset of the created placementDecision with label.


we should add this the clusters will be put into the decisions with a decision group label .....

qiujian16 · 2023-05-31T13:28:23Z

cluster/v1beta1/types_placement.go

@@ -94,6 +95,40 @@ type PlacementSpec struct {
 	// certain taints to be selected by placements with matching tolerations.
 	// +optional
 	Tolerations []Toleration `json:"tolerations,omitempty"`
+
+	// Decision Strategy divide the created placement decision to groups and define number of clusters per decision group.


Suggested change

// Decision Strategy divide the created placement decision to groups and define number of clusters per decision group.

// Decision Strategy divide the created placement decision to groups and define number of clusters per decision group. Decision result will be sorted by name and filled into decisions if it is not set.

qiujian16 · 2023-05-31T13:28:42Z

cluster/v1beta1/types_placement.go

+
+// Group the created placementDecision into decision groups based on the number of clusters per decision group.
+type DecisionStrategy struct {
+	// Define subsets of the created placement decisions and add labels to the placementDecision.


Suggested change

// Define subsets of the created placement decisions and add labels to the placementDecision.

// DecisionGroups represents a list of predefined groups to put decision results.

qiujian16 · 2023-05-31T13:30:32Z

cluster/v1beta1/types_placement.go

+	// ClustersPerDecisionGroup is a specific number or percentage of the total selected clusters.
+	// The specific number will divide the placementDecisions to decisionGroups each group has max number of clusters equal to that specific number.
+	// The percentage will divide the placementDecisions to decisionGroups each group has max number of clusters based on the total num of selected clusters and percentage.
+	// ex; for a total 100 clusters selected, ClustersPerDecisionGroup equal to 20% will divide the placement decision to 5 groups each group should have 20 clusters.


should clarify that If DecisionGroups is set, the matched decision result will be put into them at first. The left decision results will be put into the decision groups based on this field

qiujian16 · 2023-05-31T13:30:57Z

cluster/v1beta1/types_placement.go

+
+	// Decision group name that is defined in the DecisionStrategy's DecisionGroup.
+	// +optional
+	DecisionGroupName string `json:"decisionGroupName"`


Suggested change

DecisionGroupName string `json:"decisionGroupName"`

GroupLabelValue string `json:"groupLableValue"`

I change it above to groupName instead of name/labelValue and mention that in the CRD definition L106

qiujian16 · 2023-05-31T13:31:14Z

cluster/v1beta1/types_placement.go

+
+	// List of placement decisions names associated with the decision group
+	// +optional
+	PlacementDecisions []string `json:"placementDecisions"`


serngawy

I added the rolloutStrategy struct to the helper class as per discussion with ACM teams. The rolloutStrategy struct will be part and consumed by other APIs like manifestWorkReplicatSet, policy, Addon,...etc

serngawy · 2023-06-08T18:23:30Z

cluster/v1beta1/types_placement.go

+}
+
+// Group the created placementDecision into decision groups based on the number of clusters per decision group.
+type DecisionStrategy struct {


Do we need to have nested struct ? I believe decisionStrategy as single struct suppose to be good unless we will extended with something related.

serngawy · 2023-06-08T18:25:59Z

cluster/v1beta1/types_placement.go

+
+	// Decision group name that is defined in the DecisionStrategy's DecisionGroup.
+	// +optional
+	DecisionGroupName string `json:"decisionGroupName"`


I change it above to groupName instead of name/labelValue and mention that in the CRD definition L106

qiujian16 · 2023-06-12T07:37:42Z

cluster/v1beta1/types_placement.go

+	// ex; for a total 100 clusters selected, ClustersPerDecisionGroup equal to 20% will divide the placement decision to 5 groups each group should have 20 clusters.
+	// Default is having all clusters in a single group.
+	// If the DecisionGroups field defined, it will be considered first to create the decisionGroups then the ClustersPerDecisionGroup will be used to determine the rest of decisionGroups.
+	// +kubebuilder:validation:Enum="5%";"10%";"15%";"20%";"25%";"30%";"40%";"50%";"100%"


do not use enum here. you can have a regex to validate

qiujian16 · 2023-06-12T07:41:33Z

cluster/v1beta1/types_placement.go

+	// Group name to be added to the created placement Decisions labels with label key cluster.open-cluster-management.io/decision-group-name
+	// +kubebuilder:validation:Required
+	// +required
+	GroupName string `json:"groupName,omitempty"`


add regex validation rule here, since it is a label value.

qiujian16 · 2023-06-12T07:42:09Z

cluster/v1beta1/types_placement.go

+
+// DecisionGroup define a subset of clusters that will be added to placementDecisions with groupName label.
+type DecisionGroup struct {
+	// Group name to be added to the created placement Decisions labels with label key cluster.open-cluster-management.io/decision-group-name


Group name to be added as the value of the created PlacementDecisions labels with key ...

qiujian16 · 2023-06-14T01:16:38Z

/approve

I am good with this change

@haoqing0110 @zhiweiyin318 please take a final look and I think we can tag it.

openshift-ci · 2023-06-14T01:16:45Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: qiujian16, serngawy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [qiujian16]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

haoqing0110 · 2023-06-15T04:01:15Z

cluster/v1beta1/types_placement.go

+	// +kubebuilder:validation:Pattern="^((100|[0-9]{1,2})%|[0-9]+)$"
+	// +kubebuilder:default:="100%"
+	// +optional
+	ClustersPerDecisionGroup string `json:"clustersPerDecisionGroup,omitempty"`


Using intstr.IntOrString here?

I believe the validation pattern is good enough to do the same check.
actually I tried to set the type as intOrString but it gives me error while trying to build the API (error about having the validation pattern 100|[0-9]% with intOrString type) it did not build successful.

Using string should be fine from the perspective of functional implementation.

One benefit of using IntOrString I can think is that it's in line with user habits, like k8s deployment maxSurge and maxUnavailable, user don't need to warp "" for an integer in yaml.

The Placement "demo1" is invalid: spec.decisionStrategy.groupStrategy.clustersPerDecisionGroup: Invalid value: "integer": spec.decisionStrategy.groupStrategy.clustersPerDecisionGroup in body must be of type string: "integer"

For the format check, another method is that we can check it in code like https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/util/intstr/intstr.go#LL212C6-L212C32 and report error in placement "misconfigured" condition. Hope this can help.

Nice, wasn't aware that intOrString type has function to validate the percentage. okay, changing it to intOrString

zhujian7 · 2023-06-15T06:57:30Z

cluster/v1beta1/helpers.go

+	// Timeout defined in minutes for how long workload applier controller will wait till workload reach successful state in the cluster. Only considered for Rollout Type Progressive and ProgressivePerGroup. Default is None meaning the workload applier will not proceed apply workload to other clusters if did not reach the successful state.
+	// +kubebuilder:default:=None
+	// +optional
+	Timeout string `json:"timeout,omitempty"`


what is the unit of the Timeout field? minute? it could be "1h", "30m"? or "1hour", "30minutes"?

good point, it should be second with int32 as type commonly

good point thanks, I added a validation pattern for [0-9]+[h|m|s ] |None

should be *int32? my understanding is that None and 0 have different meanings, right?

yes, its string type to be able to set None

FYI https://github.com/open-cluster-management-io/api/blob/main/cluster/v1beta1/types_placement.go#L320 which is nil by default.
Are there user cases that would be waiting for minutes or even hours?

yes, upgrade OCP takes at least 50min. other examples; install Subscription/operator (ACM), images download (preCache) take around 5min-10min. AND the timeout string can be validated using the ParseDuration https://pkg.go.dev/time#ParseDuration

haoqing0110 · 2023-06-20T03:38:34Z

LGTM
I'm not sure if helpers.go is the right place to put the common rollout strategy API, since it will be a general API, being part of addon/policy/etc, even a duck type.
Can this PR get split and merge the placement decision strategy part first?
cc @serngawy @qiujian16

Signed-off-by: melserngawy <melserng@redhat.com>

serngawy · 2023-06-20T18:35:59Z

LGTM I'm not sure if helpers.go is the right place to put the common rollout strategy API, since it will be a general API, being part of addon/policy/etc, even a duck type. Can this PR get split and merge the placement decision strategy part first? cc @serngawy @qiujian16

As we discussed the rolloutStrategy struct is separated from the helper and moved to another PR [0]. I believe the DecisionStrategy is good to merge.

[0] #244

haoqing0110 · 2023-06-21T02:31:17Z

/lgtm

openshift-ci bot requested review from mdelder and qiujian16 May 25, 2023 16:25

serngawy changed the title ~~Add placement DecisionStrategy API~~ (WIP) Add placement DecisionStrategy API May 25, 2023

openshift-ci bot added the do-not-merge/work-in-progress label May 25, 2023

qiujian16 reviewed May 26, 2023

View reviewed changes

serngawy force-pushed the placementStrategy branch 5 times, most recently from e911c19 to 02c9225 Compare May 29, 2023 20:37

serngawy changed the title ~~(WIP) Add placement DecisionStrategy API~~ Add placement DecisionStrategy API May 30, 2023

openshift-ci bot removed the do-not-merge/work-in-progress label May 30, 2023

qiujian16 reviewed May 31, 2023

View reviewed changes

serngawy force-pushed the placementStrategy branch 2 times, most recently from 03ac89d to 5d71971 Compare June 8, 2023 18:21

serngawy commented Jun 8, 2023

View reviewed changes

qiujian16 reviewed Jun 12, 2023

View reviewed changes

serngawy force-pushed the placementStrategy branch 2 times, most recently from e15d8b3 to bed4557 Compare June 12, 2023 21:23

openshift-ci bot added the approved label Jun 14, 2023

haoqing0110 reviewed Jun 15, 2023

View reviewed changes

zhujian7 reviewed Jun 15, 2023

View reviewed changes

serngawy force-pushed the placementStrategy branch 4 times, most recently from 3aa6838 to 7f121af Compare June 16, 2023 13:29

Add placementStrategy API

7418172

Signed-off-by: melserngawy <melserng@redhat.com>

serngawy force-pushed the placementStrategy branch from 7f121af to 7418172 Compare June 20, 2023 18:31

openshift-ci bot assigned haoqing0110 Jun 21, 2023

openshift-ci bot added the lgtm label Jun 21, 2023

openshift-merge-robot merged commit 8f11dd3 into open-cluster-management-io:main Jun 21, 2023
6 checks passed

haoqing0110 mentioned this pull request Jul 26, 2023

Placement Strategy enhancement open-cluster-management-io/ocm#236

Closed

serngawy deleted the placementStrategy branch August 23, 2023 10:51

This was referenced Aug 31, 2023

Upgrade placement with placementStrategy API open-cluster-management-io/multicloud-operators-subscription#364

Merged

Upgrade placement with DecisionStrategy API open-cluster-management-io/registration-operator#360

Closed

	DecisionGroupIndexLabel string = "cluster.open-cluster-management.io/decisiongroupindex"
	DecisionGroupIndexLabel string = "cluster.open-cluster-management.io/decision-group-index"

	DecisionGroupName string = "cluster.open-cluster-management.io/decisiongroupname"
	DecisionGroupName string = "cluster.open-cluster-management.io/decision-group-name"

	// Decision Strategy divide the created placement decision to groups and define number of clusters per decision group.
	// Decision Strategy divide the created placement decision to groups and define number of clusters per decision group. Decision result will be sorted by name and filled into decisions if it is not set.

	// Define subsets of the created placement decisions and add labels to the placementDecision.
	// DecisionGroups represents a list of predefined groups to put decision results.

	DecisionGroupName string `json:"decisionGroupName"`
	GroupLabelValue string `json:"groupLableValue"`

Add placement DecisionStrategy API #242

Add placement DecisionStrategy API #242

Conversation

serngawy commented May 25, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qiujian16 Jun 2, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

serngawy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qiujian16 commented Jun 14, 2023

openshift-ci bot commented Jun 14, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haoqing0110 Jun 16, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

serngawy Jun 19, 2023 • edited Loading

Choose a reason for hiding this comment

haoqing0110 commented Jun 20, 2023

serngawy commented Jun 20, 2023

haoqing0110 commented Jun 21, 2023

qiujian16 Jun 2, 2023 •

edited

Loading

haoqing0110 Jun 16, 2023 •

edited

Loading

serngawy Jun 19, 2023 •

edited

Loading