Skip to content

Commit

Permalink
Remove derived mcs
Browse files Browse the repository at this point in the history
Signed-off-by: zishen <softwarebtg@163.com>
  • Loading branch information
zishen committed Jul 4, 2023
1 parent 2840f91 commit 800021a
Show file tree
Hide file tree
Showing 3 changed files with 156 additions and 0 deletions.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
156 changes: 156 additions & 0 deletions docs/proposals/mcs/remove-derived-mcs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
# Remove "derived-" prefix of MCS

## Summary

As mentioned in [issue-2384](https://github.com/karmada-io/karmada/issues/2384), currently karmada uses ServiceExport, ServiceImport and other CR to form mcs-controller and manage multi-cluster services. But in the cluster where the service is not located, we need to use a different IP or a service name with a "derived-" prefix to access. We hope to reduce this difference so that different clusters can use the same domain name to access the same service.

## Motivation

Currently, in different cluster members of Karmada, if you need to access the same service across clusters, you need to use different ip or domain names with a "derived-" prefix to access. This will increase the complexity of the user's use. We hope that users can use the same domain name to achieve cross-cluster access.

### Goals

* Keep the original access method. Avoid the need to make modifications to the customer's system after the upgrade.
* Added the use of a unified domain name for cross-cluster access. The domain name is the access domain name of the cluster where the service resides.

### Non-Goals

none

## Proposal

### User Stories

According to the current Karmada's [mcs configuration](https://karmada.io/docs/userguide/service/multi-cluster-service) , if the service is currently in member1, the user needs to access the service in member1, needs to use the domain name: "serve.default.svc.cluster.local", and the domain name in member2 will be: "derived-serve.default.svc.cluster.local". As a result, when customers use the service, they need to add the prefix "derived-" to access the domain name of the service. This brings additional inconvenience. So I hope to be able to:

1. Provide a unified access method across clusters.

2. The original access method remains unchanged. So that existing business can also work normally.

### Notes/Constraints/Caveats (Optional)

1. The coredns version must be v1.9.3 or above, and the matching k8s verison is at v1.21.0 or above, otherwise the multicluster feature is not supported

### Risks and Mitigations



## Design Details

### API change

none

### Components change

The process is divided into two parts: coreDNS and Karmada.

Here is the flow diagram:

![diagrammatic drawing of remove mcs](C:/02-Data/03-tasks/diagrammatic drawing of remove mcs.png)



1) . Given that the `Service` named `foo` exists on cluster member1 and it has endpointSlice and two pods.

2). Kamarda make `derived-foo` service and its endpointSlice on member2 through `serviceExport` and `serviceImport`. We will not change these, but add value to `serviceImport`'s `ips`, and this value comes from `derived-foo`'s clusterIP.

3). Add `coreDNS` plugin `multicluster`, and config ``coreDNS` ` as: https://github.com/coredns/multicluster

Here's an example using "clusterset.local" domain:

* Add multicluster processing rules in corefile in coredns

```shell
kubectl edit configmap coredns -n kube-system
....
Corefile: |
.:53 {
errors
health {
lameduck 5s
}
ready
multicluster clusterset.local
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . /etc/resolv.conf {
max_concurrent 1000
}
cache 30
loop
reload
loadbalance
}
...
```

* RBAC permissions need to add read permissions to `serviceImports`

excute the cmd:

```shell
kubectl edit clusterrole system:coredns
```

to add the following:

```shell
...
- apiGroups:
- multicluster.x-k8s.io
resources:
- serviceimports
verbs:
- list
- watch
```

* add "clusterset.local" for dns search.

Edit the /etc/resolv.conf file to add "clusterset.local"

```shell
root@btg:/# cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local clusterset.local
nameserver 10.13.0.10
options ndots:5
```

The Karmada part flow diagram is:

![remove-derived-in-controller](C:/02-Data/03-tasks/remove-derived-in-controller.jpg)

Add the serviceImport ips in serviceImport-controller from service's clusterIP which in the cluster member.

Here I want aad this logic in the serviceImport controller to sync the clusterIP. Add here have another two ways:

First, use op to set the clusterIP for the different clusters . These need user to kown the server'sclusterip of the individual clusters.

Second, sync service's clusterIP in work. This approach will the independence of work .



How to use:

After do this, we can use `foo.default.svc.clusterset.local` to access service.

### Test Plan

- All current testing should be passed, no break change would be involved by this feature.
- Add new E2E tests to cover the feature, the scope should include:
- Use the same domain name to access service in different clusters .



## Alternatives

Here are another two ways I know:

First, using the same service name, not service with`derived-` prefix on other cluster. More details see **[bivas](https://github.com/bivas)** 's pr [proposal for native service discovery](https://github.com/karmada-io/karmada/pull/3694)

Second, install and using [submariner](https://submariner.io/) .
Binary file added docs/proposals/mcs/remove-derived-mcs.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 800021a

Please sign in to comment.