forked from karmada-io/karmada
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Signed-off-by: guoyao <[email protected]>
- Loading branch information
Showing
6 changed files
with
283 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| # CHANGELOGs | ||
|
|
||
| - [CHANGELOG-1.1.md](./docs/CHANGELOG/CHANGELOG-1.1.md) | ||
| - [CHANGELOG-1.0.md](./docs/CHANGELOG/CHANGELOG-1.0.md) | ||
| - [CHANGELOG-0.10.md](./docs/CHANGELOG/CHANGELOG-0.10.md) | ||
| - [CHANGELOG-0.9.md](./docs/CHANGELOG/CHANGELOG-0.9.md) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| # What's New | ||
| ## Resource Interpreter Webhook | ||
| The newly introduced Resource Interpreter Webhook framework allows users to implement their own CRD plugins that will be | ||
| consulted at all parts of propagation process. With this feature, CRDs and CRs will be propagated just like Kubernetes | ||
| native resources, which means all scheduling primitives also support custom resources. An example as well as some helpful | ||
| utilities are provided to help users better understand how this framework works. | ||
|
|
||
| Refer to [Proposal](../proposals/resource-interpreter-webhook/README.md) for more details. | ||
|
|
||
|
|
||
| ## Significant Scheduling Enhancement | ||
| - Introduced `dynamicWeight` primitive to `PropagationPolicy` and `ClusterPropagationPolicy`. With this feature, replicas | ||
| could be divided by a dynamic weight list, and the weight of each cluster will be calculated based on the available | ||
| replicas during scheduling. | ||
|
|
||
| This feature can balance the cluster's utilization significantly. [#841](https://github.com/karmada-io/karmada/pull/841) | ||
|
|
||
| - Introduced `Job` schedule (divide) support. A `Job` that desires many replicas now could be divided into many clusters | ||
| just like `Deployment`. | ||
|
|
||
| This feature makes it possible to run huge Jobs across small clusters. [#898](https://github.com/karmada-io/karmada/pull/898) | ||
|
|
||
|
|
||
| ## Workloads Observation from Karmada Control Plane | ||
| After workloads (e.g. Deployments) are propagated to member clusters, users may also want to get the overall workload | ||
| status across many clusters, especially the status of each `pod`. In this release, a `get` subcommand was introduced to | ||
| the `kubectl-karmada`. With this command, user are now able get all kinds of resources deployed in member clusters from | ||
| the Karmada control plane. | ||
|
|
||
| For example (get `deployment` and `pods` across clusters): | ||
|
|
||
| ```shell | ||
| $ kubectl karmada get deployment | ||
| NAME CLUSTER READY UP-TO-DATE AVAILABLE AGE ADOPTION | ||
| nginx member2 1/1 1 1 19m Y | ||
| nginx member1 1/1 1 1 19m Y | ||
| $ kubectl karmada get pods | ||
| NAME CLUSTER READY STATUS RESTARTS AGE | ||
| nginx-6799fc88d8-vzdvt member1 1/1 Running 0 31m | ||
| nginx-6799fc88d8-l55kk member2 1/1 Running 0 31m | ||
| ``` | ||
|
|
||
|
|
||
| # Other Notable Changes | ||
| - karmada-scheduler-estimator: The number of pods becomes an important reference when calculating available replicas for | ||
| the cluster. [#777](https://github.com/karmada-io/karmada/pull/777) | ||
| - The labels (`resourcebinding.karmada.io/namespace`, `resourcebinding.karmada.io/name`, `clusterresourcebinding.karmada.io/name`) | ||
| which were previously added on the Work object now have been moved to annotations. | ||
| [#752](https://github.com/karmada-io/karmada/pull/752) | ||
| - Bugfix: Fixed the impact of cluster unjoining on resource status aggregation. [#817](https://github.com/karmada-io/karmada/pull/817) | ||
| - Instrumentation: Introduced events (`SyncFailed` and `SyncSucceed`) to the Work object. [#800](https://github.com/karmada-io/karmada/pull/800) | ||
| - Instrumentation: Introduced condition (`Scheduled`) to the `ResourceBinding` and `ClusterResourceBinding`. | ||
| [#823](https://github.com/karmada-io/karmada/pull/823) | ||
| - Instrumentation: Introduced events (`CreateExecutionNamespaceFailed` and `RemoveExecutionNamespaceFailed`) | ||
| to the Cluster object. [#749](https://github.com/karmada-io/karmada/pull/749) | ||
| - Instrumentation: Introduced several metrics (`workqueue_adds_total`, `workqueue_depth`, `workqueue_longest_running_processor_seconds`, | ||
| `workqueue_queue_duration_seconds_bucket`) for `karmada-agent` and `karmada-controller-manager`. [#831](https://github.com/karmada-io/karmada/pull/831) | ||
| - Instrumentation: Introduced condition (`FullyApplied`) to the `ResourceBinding` and `ClusterResourceBinding`. [#825](https://github.com/karmada-io/karmada/pull/825) | ||
| - karmada-scheduler: Introduced feature gates. [#805](https://github.com/karmada-io/karmada/pull/805) | ||
| - karmada-controller-manager: Deleted resources from member clusters that use "Background" as the default | ||
| delete option. [#970](https://github.com/karmada-io/karmada/pull/970) | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| # What's New | ||
| ## Upgrading support | ||
| Users are now able to upgrade from the previous version smoothly. With the | ||
| [multiple version](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/#specify-multiple-versions) | ||
| feature of CRD, | ||
| objects with different schemas can be automatically converted between versions. Karmada uses the [semantic | ||
| versioning](https://semver.org/) and will provide workarounds for inevitable breaking changes. | ||
|
|
||
| In this release, the ResourceBining and ClusterResourceBinding promote to v1alpha2 and the previous v1alpha1 | ||
| version is still available for one more release. With the | ||
| [upgrading instruction](https://github.com/karmada-io/karmada/tree/9126cffa218871e921510d3deb1b185c8a4dee3d/docs/upgrading), | ||
| the previous version of | ||
| Karmada can promote smoothly. | ||
|
|
||
| ## Introduced karmada-scheduler-estimator to facilitate end-to-end scheduling accuracy | ||
| Karmada scheduler aims to assign workload to clusters according to constraints and available resources of | ||
| each member cluster. The `kube-scheduler` working on each cluster takes the responsibility to assign Pods | ||
| to Nodes. | ||
| Even though Karmada has the capacity to reschedule failure workload between member clusters, but the | ||
| community still commits lots of effort to improve the accuracy of the end-to-end scheduling. | ||
|
|
||
| The `karmada-scheduler-estimator` is the effective assistant of `karmada-scheduler`, it provides | ||
| prediction-based scheduling decisions that can significantly improve the scheduling efficiency and | ||
| avoid the wave of rescheduling among clusters. Note that this feature is implemented as | ||
| a pluggable add-on. For the instructions please refer to | ||
| [scheduler estimator guideline](https://github.com/karmada-io/karmada/blob/9126cffa218871e921510d3deb1b185c8a4dee3d/docs/scheduler-estimator.md). | ||
|
|
||
|
|
||
| ## Maintainability improvements | ||
| A bunch of significant maintainability improvements were added to this release, including: | ||
|
|
||
| Simplified Karmada installation with | ||
| [helm chart](https://github.com/karmada-io/karmada/tree/9126cffa218871e921510d3deb1b185c8a4dee3d/charts). | ||
|
|
||
| Provided metrics to observe scheduler status, the metrics API now served at `/metrics` of `karmada-scheduler`. | ||
| With these metrics, users are now able to evaluate the scheduler's performance and identify the bottlenecks. | ||
|
|
||
| Provided events to Karmada API objects as supplemental information to debug problems. | ||
|
|
||
| # Other Notable Changes | ||
| - karmada-controller-manager: The ResourceBinding/ClusterResourceBinding won't be deleted after associate | ||
| PropagationPolicy/ClusterPropagationPolicy is removed and is still available until resource template is | ||
| removed.([#601](https://github.com/karmada-io/karmada/pull/601)) | ||
|
|
||
| - Introduced --leader-elect-resource-namespace which is used to specify the namespace of election object | ||
| to components karmada-controller-manager/karmada-scheduler/karmada-agent`. | ||
| ([#698](https://github.com/karmada-io/karmada/pull/698)) | ||
|
|
||
| - Deprecation: The API ReplicaSchedulingPolicy has been deprecated and will be removed from the following | ||
| release. The feature now has been integrated into ReplicaScheduling. | ||
| - Introduced kubectl-karmada commands as the extensions for kubectl. | ||
| ([#686](https://github.com/karmada-io/karmada/pull/686)) | ||
|
|
||
| - karmada-controller-manager introduced a version command to represent version information. | ||
| ([#717](https://github.com/karmada-io/karmada/pull/717) ) | ||
|
|
||
| - karmada-scheduler/karmada-webhook/karmada-agent/karmada-scheduler-estimator introduced a version command to | ||
| represent version information. ([#719](https://github.com/karmada-io/karmada/pull/719) ) | ||
|
|
||
| - Provided instructions about how to use the Submariner to connect the network between member | ||
| clusters. ([#737](https://github.com/karmada-io/karmada/pull/737) ) | ||
|
|
||
| - Added four metrics to the karmada-scheduler to monitor scheduler performance. | ||
| ([#747](https://github.com/karmada-io/karmada/pull/747)) | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,76 @@ | ||
| # What's New | ||
| ## Aggregated Kubernetes API Endpoint | ||
| The newly introduced `karmada-aggregated-apiserver` component aggregates all registered clusters and allows users to | ||
| access member clusters through Karmada by the proxy endpoint, e.g. | ||
|
|
||
| - Retrieve `Node` from `member1`: /apis/cluster.karmada.io/v1alpha1/clusters/member1/proxy/api/v1/nodes | ||
| - Retrieve `Pod` from `member2`: /apis/cluster.karmada.io/v1alpha1/clusters/member2/proxy/api/v1/namespaces/default/pods | ||
| Please refer to user guide for more details. | ||
|
|
||
|
|
||
| ## Promoting Workloads from Legacy Clusters to Karmada | ||
| Legacy workloads running in Kubernetes now can be promoted to Karmada smoothly without container restart. | ||
| In favor of `promote` commands added to Karmada CLI, any kind of Kubernetes resources can be promoted to Karmada easily, e.g. | ||
|
|
||
| ```shell | ||
| # Promote deployment(default/nginx) from cluster1 to Karmada | ||
| kubectl karmada promote deployment nginx -n default -c cluster1 | ||
| ``` | ||
|
|
||
|
|
||
| ## Verified Integration with Ecosystem | ||
| Benefiting from the Kubernetes native API support, Karmada can easily integrate the single cluster ecosystem for multi-cluster, | ||
| multi-cloud purpose. The following components have been verified by the Karmada community: | ||
|
|
||
| - argo-cd: refer to [working with argo-cd](../working-with-argocd.md) | ||
| - Flux: refer to [propagating helm charts with flux](https://github.com/karmada-io/karmada/issues/861#issuecomment-998540302) | ||
| - Istio: refer to [working with Istio](../istio-on-karmada.md) | ||
| - Filebeat: refer to [working with Filebeat](../working-with-filebeat.md) | ||
| - Submariner: refer to [working with Submariner](../working-with-submariner.md) | ||
| - Velero: refer to [working with Velero](../working-with-velero.md) | ||
| - Prometheus: refer to [working with Prometheus](../working-with-prometheus.md) | ||
|
|
||
|
|
||
| ## OverridePolicy Improvements | ||
| By leverage of the new-introduced `RuleWithCluster` fields to `OverridePolicy` and `ClusterOverridePolicy`, users are now | ||
| able to define override policies with a single policy for specified workloads. | ||
|
|
||
| ## Karmada Installation Improvements | ||
| Introduced `init` command to `Karmada CLI`. Users are now able to install Karmada by a single command. | ||
|
|
||
| Please refer to [Installing Karmada](../installation/installation.md) for more details. | ||
|
|
||
|
|
||
| ## Configuring Karmada Controllers | ||
| Now all controllers provided by Karmada work as plug-ins. Users are now able to turn off any of them from the default | ||
| enabled list. | ||
|
|
||
| See `--controllers` flag of `karmada-controller-manager` and `karmada-agent` for more details. | ||
|
|
||
|
|
||
| ## Resource Interpreter Webhook Enhancement | ||
| Introduced `ReviseReplica` support for the `Resource Interpreter Webhook` framework, which enables scheduling all | ||
| customized workloads just like Kubernetes native ones. | ||
|
|
||
| Refer to [Resource Interpreter Webhook Proposal](../proposals/resource-interpreter-webhook/README.md) for more design details. | ||
|
|
||
|
|
||
| # Other Notable Changes | ||
| ## Bug Fixes | ||
| - `karmada-controller-manager`: Fixed the issue that the annotation of resource template cannot be updated. [#1012](https://github.com/karmada-io/karmada/pull/1025) | ||
| - `karmada-controller-manager`: Fixed the issue of generating binding reference key. [#1003](https://github.com/karmada-io/karmada/pull/1025) | ||
| - `karmada-controller-manager`: Fixed the inefficiency of en-queue failed task issue. [#1068](https://github.com/karmada-io/karmada/pull/1025) | ||
| ## Features & Enhancements | ||
| - `Karmada CLI`: Introduced `--cluster-provider` flag to `join` command to specify provider of joining cluster. [#1025](https://github.com/karmada-io/karmada/pull/1025) | ||
| - `Karmada CLI`: Introduced `taint` command to set taints for clusters. [#889](https://github.com/karmada-io/karmada/pull/1025) | ||
| - `Karmada CLI`: The Applied condition of Work and `Scheduled/FullyApplied` of `ResourceBinding` are available for `kubectl get`. [#1110](https://github.com/karmada-io/karmada/pull/1025) | ||
| - `karmada-controller-manager`: The cluster discovery feature now supports `v1beta1` of `cluster-api`. [#1029](https://github.com/karmada-io/karmada/pull/1025) | ||
| - `karmada-controller-manager`: The `Job`'s `startTime` and `completionTime` now available at resource template. [#1034](https://github.com/karmada-io/karmada/pull/1025) | ||
| - `karmada-controller-manager`: introduced `--controllers` flag to enable or disable controllers. [#1083](https://github.com/karmada-io/karmada/pull/1025) | ||
| - `karmada-controller-manager`: Support retain `ownerReference` from observed objects. [#1116](https://github.com/karmada-io/karmada/pull/1025) | ||
| - `karmada-controller-manager` and `karmada-agent`: Introduced `cluster-cache-sync-timeout` flag to specify the time waiting for cache sync. [#1112](https://github.com/karmada-io/karmada/pull/1025) | ||
| ## Instrumentation (Metrics and Events) | ||
| - `karmada-scheduler-estimator`: Introduced `/metrics` endpoint to emit metrics. [#1030](https://github.com/karmada-io/karmada/pull/1025) | ||
| - Introduced `ApplyPolicy` and `ScheduleBinding` events for resource template. [#1070](https://github.com/karmada-io/karmada/pull/1025) | ||
| ## Deprecation | ||
| - The `ReplicaSchedulingPolicy` API deprecated at v0.9.0 now has been removed in favor of `ReplicaScheduling` of `PropagationPolicy`. [#1161](https://github.com/karmada-io/karmada/pull/1025) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| # What's New | ||
| ## Multi-Cluster Ingress | ||
| The newly introduced [MultiClusterIngress](https://github.com/karmada-io/karmada/blob/d6355ec85296daa46ed344cade6ef10a9bee58dc/pkg/apis/networking/v1alpha1/ingress_types.go#L16) | ||
| API exposes HTTP and HTTPS routes that target multi-cluster services within the Karmada control plane. The specification | ||
| of `MultiClusterIngress` is compatible with [Kubernetes Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/). | ||
|
|
||
| Traffic routing is controlled by rules defined on the MultiClusterIngress resource, an MultiClusterIngress controller is | ||
| responsible for fulfilling the ingress. The [Multi-Cluster-Nginx Ingress Controller](https://github.com/karmada-io/multi-cluster-ingress-nginx) | ||
| is one of the MultiClusterIngress controller implementations maintained by the community. | ||
|
|
||
|
|
||
| ## Federated ResourceQuota | ||
| The newly introduced [FederatedResourceQuota](https://github.com/karmada-io/karmada/blob/master/pkg/apis/policy/v1alpha1/federatedresourcequota_types.go#L14) | ||
| provides constraints that limit total resource consumption per namespace `across all clusters`. | ||
| It can limit the number of objects that can be created in a namespace by type, as well as the total amount of compute | ||
| resources that may be consumed by resources in that namespace. | ||
|
|
||
|
|
||
| ## Configurability improvement for performance tuning | ||
| The default number of reconciling workers has been enlarged and configurable. A larger number of workers means higher | ||
| responsiveness but heavier CPU and network load. The number of concurrent workers could be configured by the flags | ||
| introduced to `karmada-controller-manager` and `karmada-agent`. | ||
|
|
||
| Flags introduced to `karmada-controller-manager`: | ||
|
|
||
| - `--concurrent-work-syncs` | ||
| - `--concurrent-namespace-syncs` | ||
| - `--concurrent-resource-template-syncs` | ||
| - `--concurrent-cluster-syncs` | ||
| - `--concurrent-clusterresourcebinding-syncs` | ||
| - `--concurrent-resourcebinding-syncs` | ||
|
|
||
| Flags introduced to `karmada-agent`: | ||
|
|
||
| - `--concurrent-work-syncs` | ||
| - `--concurrent-cluster-syncs` | ||
|
|
||
|
|
||
| ## Resource Interpreter Webhook Enhancement | ||
| Introduced `AggregateStatus` support for the `Resource Interpreter Webhook` framework, which enables customized resource status aggregating. | ||
|
|
||
| Introduced `InterpreterOperationInterpretDependency` support for the `Resource Interpreter Webhook` framework, | ||
| which enables propagating workload's dependencies automatically. | ||
|
|
||
| Refer to [Customizing Resource Interpreter](../userguide/customizing-resource-interpreter.md) for more details. | ||
|
|
||
|
|
||
| # Other Notable Changes | ||
| ## Bug Fixes | ||
| - `karmadactl` and `kubectl-karmada`: Fixed that init cannot update the `APIService`. [#1207](https://github.com/karmada-io/karmada/pull/1207) | ||
| - `karmada-controller-manager`: Fixed ApplyPolicySucceed event type mistake (should be `Normal` but not `Warning`). [#1267](https://github.com/karmada-io/karmada/pull/1267) | ||
| - `karmada-controller-manager` and `karmada-agent`: Fixed that resync slows down reconciliation. [1265](https://github.com/karmada-io/karmada/pull/1265) | ||
| - `karmada-controller-manager`/`karmada-agent`: Fixed continually updating cluster status due to unordered apiEnablements. [#1304](https://github.com/karmada-io/karmada/pull/1304) | ||
| - `karmada-controller-manager`: Fixed that Replicas set by OverridePolicy will be reset by the ReviseReplica interpreterhook. [#1352](https://github.com/karmada-io/karmada/pull/1352) | ||
| - `karmada-controller-manager`: Fixed that ResourceBinding couldn't be created in a corner case. [#1368](https://github.com/karmada-io/karmada/pull/1368) | ||
| - `karmada-scheduler`: Fixed inaccuracy in requested resources in the case that pod limits are specified but requests are not. [#1225](https://github.com/karmada-io/karmada/pull/1225) | ||
| - `karmada-scheduler`: Fixed spreadconstraints[i].MaxGroups is invalidated in some scenarios. [#1324](https://github.com/karmada-io/karmada/pull/1324) | ||
| ## Features & Enhancements | ||
| - `karmadactl`: Introduced --tls-min-version flag to specify the minimum TLS version. [#1278](https://github.com/karmada-io/karmada/pull/1278) | ||
| - `karmadactl`: Improved the get command to show more useful information. [#1270](https://github.com/karmada-io/karmada/pull/1270) | ||
| - `karmada-controller-manager`/`karmada-agent`: Introduced --resync-period flag to specify reflector resync period (defaults to 0, meaning no resync). [#1261](https://github.com/karmada-io/karmada/pull/1261) | ||
| - `karmada-controller-manager`: Introduced --metrics-bind-address flag to specify the customized address for metrics. [#1341](https://github.com/karmada-io/karmada/pull/1341) | ||
| - `karmada-webhook`: Introduced --metrics-bind-address and --health-probe-bind-address flags. [#1346](https://github.com/karmada-io/karmada/pull/1346) | ||
| ## Instrumentation (Metrics and Events) | ||
| - `karmada-controller-manager`: Fixed ApplyPolicySucceed event type mistake (should be Normal but not Warning). [#1267](https://github.com/karmada-io/karmada/pull/1267) | ||
| ## Deprecation | ||
| - `OverridePolicy`/`ClusterOverridePolicy`: The `.spec.targetCluster` and `spec.overriders` have been deprecated in favor of `spec.overrideRules`. [#1238](https://github.com/karmada-io/karmada/pull/1238) | ||
| - `karmada-aggregate-apiserver`: Deprecated `--master` and `--karmada-config` flags. Please use `--kubeconfig` instead. [#1336](https://github.com/karmada-io/karmada/pull/1336) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| reviewers: | ||
| - RainbowMango | ||
| - kevin-wangzefeng | ||
| approvers: | ||
| - RainbowMango | ||
| - kevin-wangzefeng |