Releases: karmada-io/karmada
karmada v1.0.4 release
Changes since v1.0.3
Bug Fixes
karmadactl
: Fixed namespace can not be customized issue. (#1826, @likakuli)karmadactl
: fixed can not taint while karmada control plane config is not located on default path. (#1837, @wuyingjun-lucky)karmada-controller-manager
: FixedJob
status might be incorrectly marked asCompleted
issue. (#2011, @Garrybest)karmada-controller-manager
/karmada-agent
: fixed panic issue when dumps error info. (#2133, @AllenZMC)
karmada v1.1.3 release
Changes since v1.1.2
Bug Fixes
karmadactl
: Fixed namespace can not be customized issue. (#1827, @likakuli )karmadactl
: fixed karmadactl can not taint while karmada control plane config is not located on default path. (#1838, @wuyingjun-lucky )karmada-controller-manager
: FixedJob
status might be incorrectly marked asCompleted
issue. (#2010, @Garrybest )karmada-controller-manager
/karmada-agent
: fixed panic issue when dumps error info. (#2126, @AllenZMC )
karmada v1.2.0 release
What's New
Significant improvement on scheduling capability and scalability
1. Karmada Descheduler
A new component karmada-descheduler
was introduced, for rebalancing the scheduling decisions over time.
One example use case is: it helps evict pending replicas (Pods) from resource-starved clusters so that karmada-scheduler
can "reschedule" these replicas (Pods) to a cluster with sufficient resources.
For more details please refer to Descheduler user guide.
(Feature contributor: @Garrybest)
2. Multi region HA support
By leveraging the newly added spread-by-region
constraint, users are now able to deploy workloads aross regions
, e.g. people may want their workloads always running on different regions for HA purposes.
We also introduced two plugins to karmada-scheduler
, which add to accurate scheduling.
ClusterLocality
is a scoring plugin that favors clusters already assigned.SpreadConstraint
is a filter plugin that filters clusters as per spread constraints.
(Feature contributors: @huone1, @gf457832386)
We are also in the progress of enhancing the multi-cluster failover mechanism. Part of the work has been included in this release.
For example:
- A new flag(
--cluster-failure-threshold
) has been added to bothkarmada-controller-manager
andkarmada-agent
, which specifies the cluster failure threshold (defaults to 30s). A cluster will be considerednot-ready
only when it stays unhealthy longer than supposed. - A new flag(
--failover-eviction-timeout
) has been added tokarmada-controller-manager
, which specifies the grace period of eviction (defaults to 5 minutes). If a cluster staysnot-ready
longer than supposed, the controller taints the cluster. (Note: The taint is essentially the eviction order and the implementation is planned for the next release.)
(Feature contributors: @Garrybest, @dddddai)
Fully adopted aggregated API
The Aggregated API
was initially introduced in Release 1.0, which allows users to access clusters through Karmada by a single aggregated API endpoint. By leveraging this feature, we introduced a lot of interesting features to karmadactl
and kubectl-karmada
.
1. The get
sub-command now supports clusters both in push
and pull
mode.
# karmadactl get deployment -n default
NAME CLUSTER READY UP-TO-DATE AVAILABLE AGE ADOPTION
nginx member1 2/2 2 2 33h N
nginx member2 1/1 1 1 4m38s Y
podinfo member3 2/2 2 2 27h N
2. The newly added logs
command prints the container logs in a specific cluster.
# ./karmadactl logs nginx-6799fc88d8-9mpxn -c nginx -C member1
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
...
3. We also added watch
and exec
commands to karmadactl
, in addition to get
and logs
. They all use the aggregated API.
(Feature contributor: @lonelyCZ)
Distributed search and analytics engine for Kubernetes resources (alpha
)
The newly introduced karmada-search
caches resources in clusters and allows users to search for resources without directly touching real clusters.
# kubectl get --raw /apis/search.karmada.io/v1alpha1/search/cache/apis/apps/v1/deployments
{
"apiVersion": "v1",
"kind": "List",
"metadata": {},
"items": [{
"apiVersion": "apps/v1",
"kind": "Deployment",
"metadata": {
"annotations": {
"cluster.karmada.io/name": "member1",
},
}
},
]
}
The karmada-search
also supports syncing cached resources to backend stores like Elasticsearch or OpenSearch. By leveraging the search engine, you can perform full-text searches with all desired features, by field, and by indice; rank results by score, sort results by field, and aggregate results.
(Feature contributors: @huntsman-li, @liys87x)
Resource Interpreter Webhook enhancement
Introduced InterpretStatus
for the Resource Interpreter Webhook
framework, which enables customized resource status collection.
Karmada can thereby learn how to collect status for your resources, especially custom resources. For example, a custom resource may have many status fields and only Karmada can collect only those you want.
Refer to [Customizing Resource Interpreter][https://github.com/karmada-io/karmada/blob/master/docs/userguide/customizing-resource-interpreter.md] for more details.
(Feature contributor: @XiShanYongYe-Chang)
Integrating verification with the ecosystem
Benefiting from the Kubernetes native APIs, Karmada can easily integrate the Kubernetes ecosystem. The following components are verified by the Karmada community:
Kyverno
: policy engine. Refer to working with kyverno for more details.Gatekeeper
: another policy engine. Refer to working with gatekeeper for more details.fluxcd
: GitOps tooling for helm chart. Refer to working with fluxcd for more details.
(Feature contributors: @Poor12, @learner0810)
Other Notable Changes
Bug Fixes
karmadactl
: Fixed the cluster joining failures in the case of legacy secrets. (@zgfh, #1306)karmadactl
: Fixed the issue that you cannot use the '-v 6' log level. (@zgfh, #1426)karmadactl
: Fixed the issue that the--namespace
flag ofinit
command did not work. (@sayaoailun, #1416)karmadactl
: Allowed namespaces to be customized. (@sayaoailun, #1449)karmadactl
: Fixed theinit
failure due to data path not clean. (@prodanlabs, #1455)karmadactl
: Fixed theinit
failure to read the KUBECONFIG environment variable. (@lonelyCZ, #1437)karmadactl
: Fixed theinit
command failure to select the default release version. (@prodanlabs, #1456)karmadactl
: Fixed the issue that thekarmada-system
namespace already exists when deployingkarmada-agent
. (@hanweisen, #1604)karmadactl
: Fixed the issue that the karmada-controller-manager args did not honor customized namespaces.` (@prodanlabs, #1683)karmadactl
: Fixed a panic due to nil annotation whenpromoting
resources to Karmada.` (@duanmengkk, #1759)karmadactl
: Fixed thepromote
command failure to migrate cluster-scoped resources. (@duanmengkk, #1766)karmadactl
: fixed the karmadactl taint failure while the karmada control plane config is not located in the default path. (@wuyingjun-lucky, #1825)helm-chart
: Fixed the karmada-agent installation failure due to the lack of permission. (@AllenZMC, #1457)helm-chart
: Fixed the issue that version constraints skip pre-releases. (@pigletfly, #1444)karmada-controller-manager
: Fixed the issue that ResourceBinding may hinder en-queue in the case of schedule failures. (@mrlihanbo, #1499)karmada-controller-manager
: Fixed the panic when the interpreter webhook returns nil patch. (@CharlesQQ, #1584)karmada-controller-manager
: Fixed the RB/CRB controller failure to aggregate status in the case of work condition update. (@mrlihanbo, #1513)karmada-aggregate-apiserver
: Fixed timeout issue when requestingcluster/proxy
with options-w
orlogs -f
fromkarmadactl get
. (@XiShanYongYe-Chang, #1620)karmada-aggregate-apiserver
: Fixed exec failed:error: unable to upgrade connection: you must specify at least 1 of stdin, stdout, stderr
. (@pangsq, #1632)
Features & Enhancements
karmada-controller-manager
: Introduced several flags to specify controller's concurrent capacities(--rate-limiter-base-delay
,--rate-limiter-max-delay
,--rate-limiter-qps
,--rate-limiter-bucket-size
). (@pigletfly, #1399)karmada-controller-manager
: The klog flags now have been grouped for better readability. (@RainbowMango, #1468)karmada-controller-manager
: Fixed theFullyApplied
condition ofResourceBinding/ClusterResourceBinding
mislabeling issue in the case of non-scheduling. (@huone1, #1512)karmada-controller-manager
: Added defaultAggregateStatus
webhook forDaemonSet
andStatefulSet
. (@Poor12, #1586)- `karmada-controll...
karmada v1.1.2 release
Changes since v1.1.1
Bug Fixes
karmadactl: fixed the
karmada-systemnamespace already existing issue when deploying
karmada-agent` issue. (#1608, @hanweisen)karmadactl: Fixed karmada-controller-manager args not honor customized namespace issue.
(#1689, @prodanlabs)karmada-controller-manager
: Fixed ResourceBinding maybe prevents en-queue in case of schedule failure. (#1507, @mrlihanbo)karmada-controller-manager
: Fixed theFullyApplied
condition ofResourceBinding/ClusterResourceBinding
mislabeling issue in case of non-scheduled. (#1517, @huone1)karmada-controller-manager
: Fixed RB/CRB controller can't aggregate status in case of work condition update issue. (#1523, @mrlihanbo)karmada-controller-manager
: Fixed panic in case of interpreter webhook returns nil patch. (#1592, @CharlesQQ)karmada-aggregate-apiserver
: Fixed timeout issue when requestcluster/proxy
with options-w
orlogs -f
fromkarmadactl get
. (#1630, @XiShanYongYe-Chang)karmada-aggregate-apiserver
: Fixed exec failed:error: unable to upgrade connection: you must specify at least 1 of stdin, stdout, stderr
. (#1642, @XiShanYongYe-Chang)
Other
- The base image
alpine
has been promoted tov3.15.1
. (#1583, @RainbowMango)
karmada v1.0.3 release
Changes since v1.0.2
Bug Fixes
karmadactl: fixed the
karmada-systemnamespace already existing issue when deploying
karmada-agent` issue. (#1609, @hanweisen)karmadactl: Fixed karmada-controller-manager args not honor customized namespace issue.
(#1690, @prodanlabs)karmada-controller-manager
: Fixed theFullyApplied
condition ofResourceBinding/ClusterResourceBinding
mislabeling issue in case of non-scheduled. (#1518, @huone1)karmada-controller-manager
: Fixed RB/CRB controller can't aggregate status in case of work condition update issue. (#1524, @mrlihanbo)karmada-controller-manager
: Fixed panic in case of interpreter webhook returns nil patch. (#1591, @CharlesQQ)karmada-aggregate-apiserver
: Fixed timeout issue when requestcluster/proxy
with options-w
orlogs -f
fromkarmadactl get
. (#1631, @XiShanYongYe-Chang)karmada-aggregate-apiserver
: Fixed exec failed:error: unable to upgrade connection: you must specify at least 1 of stdin, stdout, stderr
. (#1641, @pangsq)
Other
- The base image
alpine
has been promoted tov3.15.1
. (#1582, @RainbowMango)
karmada v1.0.2 release
Changes since v1.0.1
Bug Fixes
karmadactl
: Fixed--namespace
flag ofinit
command not work issue. (#1452, @sayaoailun)karmadactl
: Fixedinit
failure due to data path not clean issue. (#1473, @prodanlabs)karmadactl
: Fixedinit
can not select default release version issue. (#1495, @prodanlabs)karmadactl
: Fixedinit
can not read KUBECONFIG environment variable issue. (#1482, @lonelyCZ)helm chart
: Fixed version constraints skip the pre-releases issue. (#1466, @pigletfly)karmada-controller-manager
: Fixed a bug where resource binding is not created occasionally. (#1384, @dddddai)
karmada v1.1.1 release
Changes since v1.1.0
Bug Fixes
karmadactl
: Fixed--namespace
flag ofinit
command not work issue. (#1452, @sayaoailun)karmadactl
: Fixedinit
failure due to data path not clean issue. (#1473, @prodanlabs)karmadactl
: Fixedinit
can not read KUBECONFIG environment variable issue. (#1482, @lonelyCZ)karmadactl
: Fixedinit
can not select default release version issue. (#1496, @prodanlabs)helm chart
: Fixed version constraints skip the pre-releases issue. (#1466, @pigletfly)
karmada v1.1.0 release
What's New
Multi-Cluster Ingress
The newly introduced MultiClusterIngress API exposes HTTP and HTTPS routes that target multi-cluster services within the Karmada control plane. The specification of MultiClusterIngress
is compatible with Kubernetes Ingress.
Traffic routing is controlled by rules defined on the MultiClusterIngress resource, an MultiClusterIngress controller is responsible for fulfilling the ingress. The Multi-Cluster-Nginx Ingress Controller is one of the MultiClusterIngress controller implementations maintained by the community.
(Feature contributors: @GitHubxsy @XiShanYongYe-Chang)
Federated ResourceQuota
The newly introduced FederatedResourceQuota provides constraints that limit total resource consumption per namespace across all clusters
. It can limit the number of objects that can be created in a namespace by type, as well as the total amount of compute resources that may be consumed by resources in that namespace.
(Feature contributors: @RainbowMango @XiShanYongYe-Chang)
Configurability improvement for performance tuning
The default number of reconciling workers has been enlarged and configurable. A larger number of workers means higher responsiveness but heavier CPU and network load. The number of concurrent workers could be configured by the flags introduced to karmada-controller-manager
and karmada-agent
.
Flags introduced to karmada-controller-manager
:
- --concurrent-work-syncs
- --concurrent-namespace-syncs
- --concurrent-resource-template-syncs
- --concurrent-cluster-syncs
- --concurrent-clusterresourcebinding-syncs
- --concurrent-resourcebinding-syncs
Flags introduced to karmada-agent
:
- --concurrent-work-syncs
- --concurrent-cluster-syncs
(Feature contributor: @pigletfly)
Resource Interpreter Webhook Enhancement
Introduced AggregateStatus
support for the Resource Interpreter Webhook
framework, which enables customized resource status aggregating.
Introduced InterpreterOperationInterpretDependency
support for the Resource Interpreter Webhook
framework, which enables propagating workload's dependencies automatically.
Refer to Customizing Resource Interpreter for more details.
(Feature contributors: @iawia002 @mrlihanbo)
Other Notable Changes
Bug Fixes
karmadactl
andkubectl-karmada
: Fixed thatinit
cannot update theAPIService
. (@prodanlabs, #1207)karmada-controller-manager
: Fixed ApplyPolicySucceed event type mistake (should beNormal
but notWarning
). (@Garrybest, #1267)karmada-controller-manager
andkarmada-agent
: Fixed that resync slows down reconciliation. (@Garrybest, #1265)karmada-controller-manager
/karmada-agent
: Fixed continually updating cluster status due to unordered apiEnablements. (@pigletfly, #1304)karmada-controller-manager
: Fixed that Replicas set by OverridePolicy will be reset by the ReviseReplica interpreterhook. (@likakuli, #1352)karmada-controller-manager
: Fixed that ResourceBinding couldn't be created in a corner case. (@dddddai, #1368)karmada-scheduler
: Fixed inaccuracy in requested resources in the case that pod limits are specified but requests are not. (@Garrybest, #1225)karmada-scheduler
: Fixed spreadconstraints[i].MaxGroups is invalidated in some scenarios. (@huone1, #1324)
Features & Enhancements
karmadactl
: Introduced--tls-min-version
flag to specify the minimum TLS version. (@carlory, #1278)karmadactl
: Improved theget
command to show more useful information. (@lonelyCZ, #1270)karmada-controller-manager
/karmada-agent
: Introduced--resync-period
flag to specify reflector resync period (defaults to 0, meaning no resync). (@Garrybest, #1261)karmada-controller-manager
: Introduced--metrics-bind-address
flag to specify the customized address for metrics. (@pigletfly, #1341)karmada-webhook
: Introduced --metrics-bind-address and --health-probe-bind-address flags. (@mrlihanbo, #1346)
Instrumentation (Metrics and Events)
karmada-controller-manager
: Fixed ApplyPolicySucceed event type mistake (should beNormal
but notWarning
). (@Garrybest, #1267)
Deprecation
- OverridePolicy/ClusterOverridePolicy: The
.spec.targetCluster
andspec.overriders
have been deprecated in favor ofspec.overrideRules
. (@RainbowMango #1238) karmada-aggregate-apiserver
: Deprecated--master
and--karmada-config
flags. Please use--kubeconfig
instead. (@carlory, #1336)
Contributors
Thank you to everyone who contributed to this release!
Users whose commits are in this release (alphabetically by user name)
- @AllenZMC
- @ashley-rongfang
- @carlory
- @CuiDengdeng
- @dddddai
- @EvaDD
- @Fish-pro
- @Garrybest
- @helen-frank
- @huone1
- @iawia002
- @jinglina
- @kerthcet
- @liangyongzhenya
- @likakuli
- @lonelyCZ
- @mrlihanbo
- @pigletfly
- @prodanlabs
- @RainbowMango
- @RishiKumarRay
- @Tingtal
- @viniciuspietscher
- @weilaaa
- @wlp1153468871
- @XiShanYongYe-Chang
- @zach593
- @zgfh
karmada v1.0.1 release
Changes since v1.0.0
Bug Fixes
karmadactl
andkubectl-karmada
: Fixedinit
can not update theAPIService
issue. (#1207, @prodanlabs )karmada-controller-manager
: Fixed ApplyPolicySucceed event type mistake(should beNormal
butWarning
). (#1267, @Garrybest )karmada-controller-manager
andkarmada-agent
: Fixed resync slow down reconcile issue. (#1265, @Garrybest )
karmada v1.0.0 release
What's New
Aggregated Kubernetes API Endpoint
The newly introduced karmada-aggregated-apiserver
component aggregates all registered clusters and allows users to access member clusters through Karmada by the proxy endpoint, e.g.
- Retrieve `Node` from `member1`: /apis/cluster.karmada.io/v1alpha1/clusters/member1/proxy/api/v1/nodes
- Retrieve `Pod` from `member2`: /apis/cluster.karmada.io/v1alpha1/clusters/member2/proxy/api/v1/namespaces/default/pods
Please refer to user guide for more details.
(Feature contributor: @kevin-wangzefeng @GitHubxsy @XiShanYongYe-Chang @mrlihanbo @jrkeen @prodanlabs @carlory @RainbowMango)
Promoting Workloads from Legacy Clusters to Karmada
Legacy workloads running in Kubernetes now can be promoted to Karmada smoothly without container restart.
In favor of promote
commands added to Karmada CLI, any kind of Kubernetes resources can be promoted to Karmada easily, e.g.
# Promote deployment(default/nginx) from cluster1 to Karmada
kubectl karmada promote deployment nginx -n default -c cluster1
(Feature contributor: @lonelyCZ @iawia002 @dddddai)
Verified Integration with Ecosystem
Benefiting from the Kubernetes native API support, Karmada can easily integrate the single cluster ecosystem for multi-cluster, multi-cloud purpose. The following components have been verified by the Karmada community:
argo-cd
: refer to working with argo-cdFlux
: refer to propagating helm charts with fluxIstio
: refer to working with IstioFilebeat
: refer to working with FilebeatSubmariner
: refer to working with SubmarinerVelero
: refer to working with VeleroPrometheus
: refer to working with Prometheus
(Feature contributor: @lfbear @learner0810 @zirain @Rains6 @gy95 @XiShanYongYe-Chang )
OverridePolicy Improvements
By leverage of the new-introduced RuleWithCluster
fields to OverridePolicy
and ClusterOverridePolicy
, users are now able to define override policies with a single policy for specified workloads.
(Feature contributor: @iawia002 @lfbear @RainbowMango @lonelyCZ @jameszhangyukun )
Karmada Installation Improvements
Introduced init
command to Karmada CLI
. Users are now able to install Karmada by a single command.
Please refer to Installing Karmada for more details.
(Feature contributor: @prodanlabs @lonelyCZ @jrkeen )
Configuring Karmada Controllers
Now all controllers provided by Karmada work as plug-ins. Users are now able to turn off any of them from the default enabled list.
See --controllers
flag of karmada-controller-manager
and karmada-agent
for more details.
(Feature contributor: @snowplayfire @iawia002 @jameszhangyukun )
Resource Interpreter Webhook Enhancement
Introduced ReviseReplica
support for the Resource Interpreter Webhook
framework, which enables scheduling all customized workloads just like Kubernetes native ones.
Refer to Resource Interpreter Webhook Proposal for more design details.
(Feature contributor: @iawia002)
Other Notable Changes
Bug Fixes
karmada-controller-manager
: Fixed the issue that the annotation of resource template cannot be updated. (@mrlihanbo #1012)karmada-controller-manager
: Fixed the issue of generating binding reference key. (@JarHMJ #1003)karmada-controller-manager
: Fixed the inefficiency of en-queue failed task issue. (@Garrybest #1068)
Features & Enhancements
Karmada CLI
: Introduced--cluster-provider
flag tojoin
command to specify provider of joining cluster. (@2hangchen #1025)Karmada CLI
: Introducedtaint
command to set taints for clusters. (@lonelyCZ #889)Karmada CLI
: TheApplied
condition ofWork
andScheduled/FullyApplied
ofResourceBinding
are available forkubectl get
. (@lonelyCZ #1110)karmada-controller-manager
: The cluster discovery feature now supportsv1beta1
ofcluster-api
. (@iawia002 #1029)karmada-controller-manager
: TheJob
'sstartTime
andcompletionTime
now available at resource template. (@Garrybest #1034)karmada-controller-manager
: introduced--controllers
flag to enable or disable controllers. (@snowplayfire #1083)karmada-controller-manager
: Support retainownerReference
from observed objects. (@snowplayfire #1116)karmada-controller-manager
andkarmada-agent
: Introducedcluster-cache-sync-timeout
flag to specify the time waiting for cache sync. (@snowplayfire #1112)
Instrumentation (Metrics and Events)
karmada-scheduler-estimator
: Introduced/metrics
endpoint to emit metrics. (@Garrybest #1030)- Introduced
ApplyPolicy
andScheduleBinding
events for resource template. (@mrlihanbo #1070)
Deprecation
- The
ReplicaSchedulingPolicy
API deprecated at v0.9.0 now has been removed in favor ofReplicaScheduling
ofPropagationPolicy
. (@iawia002 #1161)
Contributors
Thank you to everyone who contributed to this release!
Users whose commits are in this release (alphabetically by user name)
- @2hangchen
- @aven-ai
- @BDXGD
- @carlory
- @dddddai
- @eightzero
- @fanzhihai0215
- @feeltimeQ
- @fleeto
- @Garrybest
- @ghl116
- @gy95
- @haiker2011
- @Haleygo
- @iawia002
- @imroc
- @JackZxj
- @jameszhangyukun
- @JarHMJ
- @jrkeen
- @kevin-wangzefeng
- @leonharetd
- @lfbear
- @lonelyCZ
- @mrlihanbo
- @Phil-sun
- @pigletfly
- @prodanlabs
- @RainbowMango
- @Rains6
- @Shike-Ada
- @snowplayfire
- @wawa0210
- @XiShanYongYe-Chang
- @zirain