Skip to content

Commit

Permalink
placement support decision groups
Browse files Browse the repository at this point in the history 
Signed-off-by: haoqing0110 <qhao@redhat.com>
  • Loading branch information
@haoqing0110
haoqing0110 committed Jun 28, 2023
1 parent 987d7c7 commit 59e5c68
Show file tree
Hide file tree
Showing 14 changed files with 684 additions and 131 deletions.
2 changes: 1 addition & 1 deletion go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ require (
k8s.io/kube-aggregator v0.27.2
k8s.io/utils v0.0.0-20230313181309-38a27ef9d749
open-cluster-management.io/addon-framework v0.7.1-0.20230626092851-963716af4eed
open-cluster-management.io/api v0.11.1-0.20230609103311-088e8fe86139
open-cluster-management.io/api v0.11.1-0.20230621023453-8f11dd3ede06
sigs.k8s.io/controller-runtime v0.15.0
sigs.k8s.io/kube-storage-version-migrator v0.0.5
)
Expand Down
4 changes: 2 additions & 2 deletions go.sum
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1158,8 +1158,8 @@ k8s.io/utils v0.0.0-20230313181309-38a27ef9d749 h1:xMMXJlJbsU8w3V5N2FLDQ8YgU8s1E
k8s.io/utils v0.0.0-20230313181309-38a27ef9d749/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0=
open-cluster-management.io/addon-framework v0.7.1-0.20230626092851-963716af4eed h1:fOOKf8kzVNizc5fYvMwkPy9TT/vOpojd4IIxpzh/vhw=
open-cluster-management.io/addon-framework v0.7.1-0.20230626092851-963716af4eed/go.mod h1:Cyt5knxR+sXaKvOfUKseZDAGulS2AJz6o7a9J0WXbak=
open-cluster-management.io/api v0.11.1-0.20230609103311-088e8fe86139 h1:nw/XSv4eDGqmg0ks2PHzrE2uosvjw+D314843G56xGY=
open-cluster-management.io/api v0.11.1-0.20230609103311-088e8fe86139/go.mod h1:WgKUCJ7+Bf40DsOmH1Gdkpyj3joco+QLzrlM6Ak39zE=
open-cluster-management.io/api v0.11.1-0.20230621023453-8f11dd3ede06 h1:FO+fWqXHvG2daZurmSVgOe8rpefoqmShKs7PbbVnmpw=
open-cluster-management.io/api v0.11.1-0.20230621023453-8f11dd3ede06/go.mod h1:WgKUCJ7+Bf40DsOmH1Gdkpyj3joco+QLzrlM6Ak39zE=
rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=
rsc.io/quote/v3 v3.1.0/go.mod h1:yEA65RcK8LyAZtP9Kv3t0HmxON59tX3rD+tICJqUlj0=
rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA=
Expand Down
180 changes: 180 additions & 0 deletions ...fests/cluster-manager/hub/0000_02_clusters.open-cluster-management.io_placements.crd.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,154 @@ spec:
items:
type: string
type: array
decisionStrategy:
description: DecisionStrategy divide the created placement decision
to groups and define number of clusters per decision group.
properties:
groupStrategy:
description: GroupStrategy define strategies to divide selected
clusters to decision groups.
properties:
clustersPerDecisionGroup:
anyOf:
- type: integer
- type: string
default: 100%
description: ClustersPerDecisionGroup is a specific number
or percentage of the total selected clusters. The specific
number will divide the placementDecisions to decisionGroups
each group has max number of clusters equal to that specific
number. The percentage will divide the placementDecisions
to decisionGroups each group has max number of clusters
based on the total num of selected clusters and percentage.
ex; for a total 100 clusters selected, ClustersPerDecisionGroup
equal to 20% will divide the placement decision to 5 groups
each group should have 20 clusters. Default is having all
clusters in a single group. If the DecisionGroups field
defined, it will be considered first to create the decisionGroups
then the ClustersPerDecisionGroup will be used to determine
the rest of decisionGroups.
x-kubernetes-int-or-string: true
decisionGroups:
description: DecisionGroups represents a list of predefined
groups to put decision results.
items:
description: DecisionGroup define a subset of clusters that
will be added to placementDecisions with groupName label.
properties:
groupClusterSelector:
description: LabelSelector to select clusters subset
by label.
properties:
claimSelector:
description: ClaimSelector represents a selector
of ManagedClusters by clusterClaims in status
properties:
matchExpressions:
description: matchExpressions is a list of cluster
claim selector requirements. The requirements
are ANDed.
items:
description: A label selector requirement
is a selector that contains values, a key,
and an operator that relates the key and
values.
properties:
key:
description: key is the label key that
the selector applies to.
type: string
operator:
description: operator represents a key's
relationship to a set of values. Valid
operators are In, NotIn, Exists and
DoesNotExist.
type: string
values:
description: values is an array of string
values. If the operator is In or NotIn,
the values array must be non-empty.
If the operator is Exists or DoesNotExist,
the values array must be empty. This
array is replaced during a strategic
merge patch.
items:
type: string
type: array
required:
- key
- operator
type: object
type: array
type: object
labelSelector:
description: LabelSelector represents a selector
of ManagedClusters by label
properties:
matchExpressions:
description: matchExpressions is a list of label
selector requirements. The requirements are
ANDed.
items:
description: A label selector requirement
is a selector that contains values, a key,
and an operator that relates the key and
values.
properties:
key:
description: key is the label key that
the selector applies to.
type: string
operator:
description: operator represents a key's
relationship to a set of values. Valid
operators are In, NotIn, Exists and
DoesNotExist.
type: string
values:
description: values is an array of string
values. If the operator is In or NotIn,
the values array must be non-empty.
If the operator is Exists or DoesNotExist,
the values array must be empty. This
array is replaced during a strategic
merge patch.
items:
type: string
type: array
required:
- key
- operator
type: object
type: array
matchLabels:
additionalProperties:
type: string
description: matchLabels is a map of {key,value}
pairs. A single {key,value} in the matchLabels
map is equivalent to an element of matchExpressions,
whose key field is "key", the operator is
"In", and the values array contains only "value".
The requirements are ANDed.
type: object
type: object
x-kubernetes-map-type: atomic
type: object
groupName:
description: Group name to be added as label value to
the created placement Decisions labels with label
key cluster.open-cluster-management.io/decision-group-name
pattern: ^[a-zA-Z0-9][-A-Za-z0-9_.]{0,61}[a-zA-Z0-9]$
type: string
required:
- groupClusterSelector
- groupName
type: object
type: array
required:
- clustersPerDecisionGroup
type: object
type: object
numberOfClusters:
description: NumberOfClusters represents the desired number of ManagedClusters
to be selected which meet the placement requirements. 1) If not
Expand Down Expand Up @@ -466,6 +614,38 @@ spec:
- type
type: object
type: array
decisionGroups:
description: List of decision groups determined by the placement and
DecisionStrategy.
items:
description: Present decision groups status based on the DecisionStrategy
definition.
properties:
clusterCount:
default: 0
description: Total number of clusters in the decision group.
Clusters count is equal or less than the clusterPerDecisionGroups
defined in the decision strategy.
format: int32
type: integer
decisionGroupIndex:
description: Present the decision group index. If there is no
decision strategy defined all placement decisions will be
in group index 0
format: int32
type: integer
decisionGroupName:
description: Decision group name that is defined in the DecisionStrategy's
DecisionGroup.
type: string
decisions:
description: List of placement decisions names associated with
the decision group
items:
type: string
type: array
type: object
type: array
numberOfSelectedClusters:
description: NumberOfSelectedClusters represents the number of selected
ManagedClusters
Expand Down
20 changes: 20 additions & 0 deletions pkg/placement/controllers/scheduling/schedule.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,12 @@ type Scheduler interface {
) (ScheduleResult, *framework.Status)
}

// decision groups defines the group name and cluster decisions
type DecisionGroup struct {
DecisionGroupName string
ClusterDecisions []clusterapiv1beta1.ClusterDecision
}

type ScheduleResult interface {
// FilterResults returns results for each filter
FilterResults() []FilterResult
Expand All @@ -59,6 +65,9 @@ type ScheduleResult interface {
// Decisions returns the decisions of the schedule
Decisions() []clusterapiv1beta1.ClusterDecision

// DecisionGroups returns the decision groups of the schedule
DecisionGroups() []DecisionGroup

// NumOfUnscheduled returns the number of unscheduled.
NumOfUnscheduled() int

Expand Down Expand Up @@ -431,6 +440,17 @@ func (r *scheduleResult) Decisions() []clusterapiv1beta1.ClusterDecision {
return r.scheduledDecisions
}

func (r *scheduleResult) DecisionGroups() []DecisionGroup {
// TODO: divide r.scheduledDecisions into decision groups by group strategy
decisionGroups := []DecisionGroup{}
group := DecisionGroup{
DecisionGroupName: "",
ClusterDecisions: r.scheduledDecisions,
}
decisionGroups = append(decisionGroups, group)
return decisionGroups
}

func (r *scheduleResult) NumOfUnscheduled() int {
return r.unscheduledDecisions
}
Expand Down
12 changes: 6 additions & 6 deletions pkg/placement/controllers/scheduling/schedule_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -152,12 +152,12 @@ func TestSchedule(t *testing.T) {
testinghelpers.NewClusterSet(clusterSetName).Build(),
testinghelpers.NewClusterSetBinding(placementNamespace, clusterSetName),
testinghelpers.NewPlacementDecision(placementNamespace, placementDecisionName(placementName, 1)).
WithLabel(placementLabel, placementName).
WithLabel(clusterapiv1beta1.PlacementLabel, placementName).
WithDecisions("cluster1", "cluster2").Build(),
},
decisions: []runtime.Object{
testinghelpers.NewPlacementDecision(placementNamespace, placementDecisionName(placementName, 1)).
WithLabel(placementLabel, placementName).
WithLabel(clusterapiv1beta1.PlacementLabel, placementName).
WithDecisions("cluster1", "cluster2").Build(),
},
clusters: []*clusterapiv1.ManagedCluster{
Expand Down Expand Up @@ -399,7 +399,7 @@ func TestSchedule(t *testing.T) {
testinghelpers.NewClusterSet(clusterSetName).Build(),
testinghelpers.NewClusterSetBinding(placementNamespace, clusterSetName),
testinghelpers.NewPlacementDecision(placementNamespace, placementDecisionName(placementName, 1)).
WithLabel(placementLabel, placementName).
WithLabel(clusterapiv1beta1.PlacementLabel, placementName).
WithDecisions("cluster1").Build(),
},
clusters: []*clusterapiv1.ManagedCluster{
Expand All @@ -408,7 +408,7 @@ func TestSchedule(t *testing.T) {
},
decisions: []runtime.Object{
testinghelpers.NewPlacementDecision(placementNamespace, placementDecisionName(placementName, 1)).
WithLabel(placementLabel, placementName).
WithLabel(clusterapiv1beta1.PlacementLabel, placementName).
WithDecisions("cluster1").Build(),
},
expectedDecisions: []clusterapiv1beta1.ClusterDecision{
Expand Down Expand Up @@ -497,7 +497,7 @@ func TestSchedule(t *testing.T) {
testinghelpers.NewPlacementDecision(placementNamespace, placementDecisionName("others", 2)).
WithDecisions("cluster1", "cluster2").Build(),
testinghelpers.NewPlacementDecision(placementNamespace, placementDecisionName(placementName, 1)).
WithLabel(placementLabel, placementName).
WithLabel(clusterapiv1beta1.PlacementLabel, placementName).
WithDecisions("cluster3").Build(),
},
decisions: []runtime.Object{
Expand All @@ -506,7 +506,7 @@ func TestSchedule(t *testing.T) {
testinghelpers.NewPlacementDecision(placementNamespace, placementDecisionName("others", 2)).
WithDecisions("cluster1", "cluster2").Build(),
testinghelpers.NewPlacementDecision(placementNamespace, placementDecisionName(placementName, 1)).
WithLabel(placementLabel, placementName).
WithLabel(clusterapiv1beta1.PlacementLabel, placementName).
WithDecisions("cluster3").Build(),
},
clusters: []*clusterapiv1.ManagedCluster{
Expand Down
Loading

0 comments on commit 59e5c68

Please sign in to comment.