Support Connection to ResourceInterpretWebhook without DNS Service #2999

lxtywypc · 2022-12-28T02:46:21Z

What type of PR is this?
/kind feature

What this PR does / why we need it:
If we deploy karmada with Default dnsPolicy or deploy karmada in physical machine/virtual machine directly, and there is no coreDNS or other DNS services, it would not connect to resource-interpret-webhooks successfully with DefaultServiceResolver.

Which issue(s) this PR fixes:
Fixes #2998

Special notes for your reviewer:
NONE

Does this PR introduce a user-facing change?:

`karmada-controller-manager`/`karmada-agent`: support connection to resourceInterpretWebhook without DNS Service.

karmada-bot · 2022-12-28T02:46:32Z

Welcome @lxtywypc! It looks like this is your first PR to karmada-io/karmada 🎉

codecov-commenter · 2022-12-29T08:31:08Z

Codecov Report

Merging #2999 (71ffbf4) into master (7f2b660) will decrease coverage by 0.03%.
The diff coverage is n/a.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@            Coverage Diff             @@
##           master    #2999      +/-   ##
==========================================
- Coverage   51.63%   51.61%   -0.03%     
==========================================
  Files         210      210              
  Lines       18926    18926              
==========================================
- Hits         9773     9769       -4     
- Misses       8622     8625       +3     
- Partials      531      532       +1

Flag	Coverage Δ
unittests	`51.61% <ø> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 2 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

XiShanYongYe-Chang · 2023-01-05T12:28:15Z

Hi @lxtywypc, the E2E has failed in the CI, have you tested it on your site?

XiShanYongYe-Chang

Kindly ping @lxtywypc

XiShanYongYe-Chang · 2023-01-09T03:51:19Z

pkg/resourceinterpreter/customizedinterpreter/customized.go

@@ -45,8 +49,23 @@ func NewCustomizedInterpreter(informer genericmanager.SingleClusterInformerManag
 	if err != nil {
 		return nil, err
 	}
+
+	serviceInformer := informers.NewSharedInformerFactory(kubernetes.NewForConfigOrDie(config), 0).Core().V1().Services()


Why not use the input parameters informer directly?

Because it is a dynamic informer, but we need a versioned informer.
Maybe I can use indexer to build a service lister directly, but that need to add a new method to get indexer in interface SingleClusterInformerManager, I don't know if it would be better.

ok, i get it.

I think we can use the versioned informer to replace the dynamic informer, if this works, we can do it in the next pr.

lxtywypc · 2023-01-09T08:13:39Z

Hi @lxtywypc, the E2E has failed in the CI, have you tested it on your site?

Humm...
I don't know well about how e2e test works, but it may show different error after each retry, and it seems to be ok after some retries.

XiShanYongYe-Chang

LGTM
/cc @RainbowMango

RainbowMango · 2023-01-10T08:55:32Z

The E2E still failing after retrying. @XiShanYongYe-Chang

XiShanYongYe-Chang · 2023-01-11T01:27:35Z

Hi @lxtywypc, I test with this pr on my side, after I deployed Karmada, I find that the status condition of the cluster member3 (Pull mode) is Unknown, and when I delete the old karmada-controller-manager pod, the logs of new pod report an panic:

lxtywypc · 2023-01-12T13:08:34Z

Hi @lxtywypc, I test with this pr on my side, after I deployed Karmada, I find that the status condition of the cluster member3 (Pull mode) is Unknown, and when I delete the old karmada-controller-manager pod, the logs of new pod report an panic:

Ok, I will check again later

lxtywypc · 2023-01-16T12:23:38Z

@XiShanYongYe-Chang @RainbowMango
It should be ok this time.
I used a for-select loop to make sure the informer had synced, and in the early version I made it sleep 1s if the loop hadn't been done.
4bbdf74#diff-a111978a679a9f03e49c67a43ecb913299d153e59ce48d2d30f5029f27ab601aR532
It seems that some calling of resource-interpreter comes early during this period, then it causes a terrible panic of empty interpreter pointer.
This time I build service informer and wait for it synced before newing a resource-interpreter. It can work now.
But since it would cause a panic for a period of sleep during starting interpreter, I think it may still have a potential panic risk if it process too long during starting interpreter. How do you think about?

XiShanYongYe-Chang · 2023-02-11T09:59:26Z

/assign

cmd/controller-manager/app/controllermanager.go

XiShanYongYe-Chang · 2023-03-20T09:35:38Z

Thanks, @lxtywypc, can you help compress your commits into one?

Ask for @ikaven1024 @RainbowMango to retake a review.
/assign @ikaven1024 @RainbowMango

lxtywypc · 2023-03-20T09:41:50Z

Thanks, @lxtywypc, can you help compress your commits into one?

Ok, but how about vendor? @XiShanYongYe-Chang

RainbowMango

/lgtm
/approve

/hold
for the test report, since we don't have the test for this scenario.

karmada-bot · 2023-03-22T09:32:38Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: RainbowMango

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [RainbowMango]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

XiShanYongYe-Chang · 2023-03-29T08:02:56Z

Hi @lxtywypc can you help post your test result here? A simple test process report is enough.

lxtywypc · 2023-04-17T02:29:28Z

Hi @XiShanYongYe-Chang , I'm sorry for replying late.
I'll be glad to raise a test report, but what kind report would you like? Could you offer me a template?

…minate dependency of DNS Signed-off-by: lxtywypc <lxtywypc@gmail.com>

XiShanYongYe-Chang · 2023-04-17T03:34:13Z

I'll be glad to raise a test report, but what kind report would you like? Could you offer me a template?

Thanks~

I'm sorry, I don't have a specific template here either. My idea is that you can simply explain your local test process and post your test results here will be ok.

lxtywypc · 2023-04-17T03:39:17Z

My idea is that you can simply explain your local test process and post your test results here will be ok.

Ok, I'll do it later.

lxtywypc · 2023-04-20T07:56:57Z

Here is the test report:

Note: Some parts with the information of company in the example has been hidden.

Prepare: We has a workload CR whose kind is Ro, and a resource-interpreter-webhook in cluster named ro-webhook and implemented the InterpretReplica point:

$ kubectl get svc -n karmada-system ro-webhook
NAME         TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
ro-webhook   ClusterIP   10.7.93.168   <none>        443/TCP   196d
$ kubectl get resourceinterpreterwebhookconfiguration ro-webhook -o jsonpath='{"service: "}{.webhooks[0].clientConfig.service}{"\nrules: "}{.webhooks[0].rules}{"\n"}'
service: {"name":"ro-webhook","namespace":"karmada-system","path":"/interpret-ro","port":443}
rules: [{"apiGroups":["xxx.xxx"],"apiVersions":["v1"],"kinds":["Ro"],"operations":["InterpretReplica"]}]

First, we apply a workload named test with 0 replicas in namespace test, and karmada-controller-manager with image based on branch master:

$ kubectl get ro -n test test -o jsonpath='{.spec.replicas}{"\n"}'
0
$ kubectl get deploy -n karmada-system karmada-controller-manager -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
xxx.xxx/karmada/karmada-controller-manager:master

Then we scale the workload up to 3, and get it's replicas in resourcebinding:

$ kubectl scale ro -n test test --replicas=3
xxx.xxx/test scaled
$ kubectl get rb -n test test-ro -o jsonpath='{.spec.replicas}{"\n"}'

$

Then we look up the log of karmada-controller-manager:

$ kubectl exec -it karmada-controller-manager-6db886b97f-4fb5q -n karmada-system -- /bin/sh
/ # cd /var/log/karmada-controller-manager/
/var/log/karmada-controller-manager # cat runtime.log | grep test
I0420 07:08:18.189526       1 customized.go:74] Get replicas for object: xxx.xxx/v1, Kind=Ro test/test with webhook interpreter.
E0420 07:08:18.195101       1 detector.go:649] Failed to customize replicas for xxx.xxx/v1, Kind=Ro(test), Internal error occurred: failed calling webhook "ro-webhook/xxx.xxx": failed to call webhook: Post "https://ro-webhook.karmada-system.svc:443/interpret-ro?timeout=3s": dial tcp: lookup ro-webhook.karmada-system.svc on 10.5.168.10:53: no such host
E0420 07:08:18.195121       1 detector.go:472] Failed to build resourceBinding for object: xxx.xxx/v1, kind=Ro, test/test. error: Internal error occurred: failed calling webhook "ro-webhook/xxx.xxx": failed to call webhook: Post "https://ro-webhook.karmada-system.svc:443/interpret-ro?timeout=3s": dial tcp: lookup ro-webhook.karmada-system.svc on 10.5.168.10:53: no such host
I0420 07:08:18.195198       1 recorder.go:103] "events: Apply cluster policy(ro) failed: Internal error occurred: failed calling webhook \"ro-webhook/xxx.xxx\": failed to call webhook: Post \"https://ro-webhook.karmada-system.svc:443/interpret-ro?timeout=3s\": dial tcp: lookup ro-webhook.karmada-system.svc on 10.5.168.10:53: no such host" type="Warning" object={Kind:Ro Namespace:test Name:test UID:0f9fc62b-26ea-45da-8b18-085f1b167864 APIVersion:xxx.xxx/v1 ResourceVersion:643725715 FieldPath:} reason="ApplyPolicyFailed"

Now, we change the image of karmada-controller-manager to the image based on this pull request, and get the replicas in resourcebinding again:

$ kubectl get deploy -n karmada-system karmada-controller-manager -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
xxx.xxx/karmada/karmada-controller-manager:master-with-resolver
$ kubectl get rb -n test test-ro -o jsonpath='{.spec.replicas}{"\n"}'
3

Let's look up the log of karmada-controller-manager again:

$ kubectl exec -it karmada-controller-manager-57dcd97579-k9shq -n karmada-system -- /bin/sh
/ # cd /var/log/karmada-controller-manager/
/var/log/karmada-controller-manager # cat runtime.log | grep test
I0420 07:32:17.723681       1 customized.go:77] Get replicas for object: xxx.xxx/v1, Kind=Ro test/test with webhook interpreter.
I0420 07:32:17.735193       1 detector.go:510] Update ResourceBinding(test-ro) successfully.
I0420 07:32:17.735240       1 recorder.go:103] "events: Apply cluster policy(ro) succeed" type="Normal" object={Kind:Ro Namespace:test Name:test UID:0f9fc62b-26ea-45da-8b18-085f1b167864 APIVersion:xxx.xxx/v1 ResourceVersion:643725715 FieldPath:} reason="ApplyPolicySucceed"

The webhook can be accessed successfully, and we haven't met any other problems yet.

lxtywypc · 2023-04-20T07:57:38Z

/cc @XiShanYongYe-Chang @RainbowMango

XiShanYongYe-Chang · 2023-04-20T08:35:35Z

@lxtywypc a clear test report, thanks a lot.

/lgtm
/hold cancel

Can you help update the release-note, such as:

karmada-controller-manager/karmada-agent: support connection to resourceInterpretWebhook without DNS Service

lxtywypc · 2023-04-23T04:03:14Z

@XiShanYongYe-Chang I'm glad to. But I don't know clearly what I should do.
Should I cherry pick this PR to other release branches and make tags?

XiShanYongYe-Chang · 2023-04-23T06:27:39Z

I'm glad to. But I don't know clearly what I should do.

Modify this place will be ok:

Should I cherry pick this PR to other release branches and make tags?

We usually synchronize bugs to previous versions, and the current change is more like a feature to us. Do you need to use it in previous versions?

RainbowMango · 2023-04-23T06:45:14Z

I don't think we need to cherry-pick this to release branches.

lxtywypc · 2023-04-24T02:54:48Z

@XiShanYongYe-Chang @RainbowMango I see. Thanks a lot.

karmada-bot added the kind/feature Categorizes issue or PR as related to a new feature. label Dec 28, 2022

karmada-bot requested review from chaunceyjiang, Garrybest and lonelyCZ December 28, 2022 02:46

karmada-bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Dec 28, 2022

lxtywypc force-pushed the update-resolver branch from 8a9cbaf to a7c0d24 Compare December 28, 2022 08:53

XiShanYongYe-Chang reviewed Jan 9, 2023

View reviewed changes

XiShanYongYe-Chang reviewed Jan 10, 2023

View reviewed changes

karmada-bot requested a review from RainbowMango January 10, 2023 03:50

karmada-bot added the do-not-merge/contains-merge-commits Indicates a PR which contains merge commits. label Jan 12, 2023

lxtywypc force-pushed the update-resolver branch from 959fa9a to 8d1eb7a Compare January 16, 2023 10:19

karmada-bot removed the do-not-merge/contains-merge-commits Indicates a PR which contains merge commits. label Jan 16, 2023

lxtywypc force-pushed the update-resolver branch from 8d1eb7a to 4bbdf74 Compare January 16, 2023 10:33

karmada-bot assigned XiShanYongYe-Chang Feb 11, 2023

lxtywypc force-pushed the update-resolver branch from 4bbdf74 to ee33fb8 Compare February 13, 2023 08:42

ikaven1024 reviewed Mar 14, 2023

View reviewed changes

cmd/controller-manager/app/controllermanager.go Outdated Show resolved Hide resolved

cmd/controller-manager/app/controllermanager.go Outdated Show resolved Hide resolved

lxtywypc force-pushed the update-resolver branch from ee33fb8 to ec3f62e Compare March 20, 2023 09:11

karmada-bot assigned ikaven1024 and RainbowMango Mar 20, 2023

lxtywypc force-pushed the update-resolver branch from c3ab005 to fafcaac Compare March 22, 2023 09:14

karmada-bot removed the lgtm Indicates that a PR is ready to be merged. label Mar 22, 2023

RainbowMango approved these changes Mar 22, 2023

View reviewed changes

karmada-bot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. lgtm Indicates that a PR is ready to be merged. labels Mar 22, 2023

karmada-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 22, 2023

use ClusterIPServiceResolver instead of DefaultServiceResolver to eli…

71ffbf4

…minate dependency of DNS Signed-off-by: lxtywypc <lxtywypc@gmail.com>

lxtywypc force-pushed the update-resolver branch from fafcaac to 71ffbf4 Compare April 17, 2023 02:44

karmada-bot removed the lgtm Indicates that a PR is ready to be merged. label Apr 17, 2023

karmada-bot requested review from RainbowMango and XiShanYongYe-Chang April 20, 2023 07:57

karmada-bot added lgtm Indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Apr 20, 2023

karmada-bot merged commit ed2b101 into karmada-io:master Apr 20, 2023

lxtywypc deleted the update-resolver branch August 1, 2023 08:02

lxtywypc mentioned this pull request Oct 23, 2023

REQUEST: New membership for lxtywypc karmada-io/community#56

Closed

8 tasks

RainbowMango mentioned this pull request Dec 14, 2024

Fix the issue of missing workqueue metrics in the Karmada controller #5945

Open

RainbowMango added this to the v1.10 milestone Dec 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Connection to ResourceInterpretWebhook without DNS Service #2999

Support Connection to ResourceInterpretWebhook without DNS Service #2999

lxtywypc commented Dec 28, 2022 •

edited by RainbowMango

Loading

karmada-bot commented Dec 28, 2022

codecov-commenter commented Dec 29, 2022 •

edited

Loading

XiShanYongYe-Chang commented Jan 5, 2023

XiShanYongYe-Chang left a comment

XiShanYongYe-Chang Jan 9, 2023

lxtywypc Jan 9, 2023

XiShanYongYe-Chang Jan 9, 2023

XiShanYongYe-Chang Jan 10, 2023

lxtywypc commented Jan 9, 2023 •

edited

Loading

XiShanYongYe-Chang left a comment

RainbowMango commented Jan 10, 2023

XiShanYongYe-Chang commented Jan 11, 2023

lxtywypc commented Jan 12, 2023

lxtywypc commented Jan 16, 2023

XiShanYongYe-Chang commented Feb 11, 2023

XiShanYongYe-Chang commented Mar 20, 2023

lxtywypc commented Mar 20, 2023 •

edited

Loading

RainbowMango left a comment

karmada-bot commented Mar 22, 2023

XiShanYongYe-Chang commented Mar 29, 2023

lxtywypc commented Apr 17, 2023 •

edited

Loading

XiShanYongYe-Chang commented Apr 17, 2023

lxtywypc commented Apr 17, 2023 •

edited

Loading

lxtywypc commented Apr 20, 2023

lxtywypc commented Apr 20, 2023

XiShanYongYe-Chang commented Apr 20, 2023

lxtywypc commented Apr 23, 2023

XiShanYongYe-Chang commented Apr 23, 2023

RainbowMango commented Apr 23, 2023

lxtywypc commented Apr 24, 2023

Support Connection to ResourceInterpretWebhook without DNS Service #2999

Support Connection to ResourceInterpretWebhook without DNS Service #2999

Conversation

lxtywypc commented Dec 28, 2022 • edited by RainbowMango Loading

karmada-bot commented Dec 28, 2022

codecov-commenter commented Dec 29, 2022 • edited Loading

Codecov Report

XiShanYongYe-Chang commented Jan 5, 2023

XiShanYongYe-Chang left a comment

Choose a reason for hiding this comment

XiShanYongYe-Chang Jan 9, 2023

Choose a reason for hiding this comment

lxtywypc Jan 9, 2023

Choose a reason for hiding this comment

XiShanYongYe-Chang Jan 9, 2023

Choose a reason for hiding this comment

XiShanYongYe-Chang Jan 10, 2023

Choose a reason for hiding this comment

lxtywypc commented Jan 9, 2023 • edited Loading

XiShanYongYe-Chang left a comment

Choose a reason for hiding this comment

RainbowMango commented Jan 10, 2023

XiShanYongYe-Chang commented Jan 11, 2023

lxtywypc commented Jan 12, 2023

lxtywypc commented Jan 16, 2023

XiShanYongYe-Chang commented Feb 11, 2023

XiShanYongYe-Chang commented Mar 20, 2023

lxtywypc commented Mar 20, 2023 • edited Loading

RainbowMango left a comment

Choose a reason for hiding this comment

karmada-bot commented Mar 22, 2023

XiShanYongYe-Chang commented Mar 29, 2023

lxtywypc commented Apr 17, 2023 • edited Loading

XiShanYongYe-Chang commented Apr 17, 2023

lxtywypc commented Apr 17, 2023 • edited Loading

lxtywypc commented Apr 20, 2023

lxtywypc commented Apr 20, 2023

XiShanYongYe-Chang commented Apr 20, 2023

lxtywypc commented Apr 23, 2023

XiShanYongYe-Chang commented Apr 23, 2023

RainbowMango commented Apr 23, 2023

lxtywypc commented Apr 24, 2023

lxtywypc commented Dec 28, 2022 •

edited by RainbowMango

Loading

codecov-commenter commented Dec 29, 2022 •

edited

Loading

lxtywypc commented Jan 9, 2023 •

edited

Loading

lxtywypc commented Mar 20, 2023 •

edited

Loading

lxtywypc commented Apr 17, 2023 •

edited

Loading

lxtywypc commented Apr 17, 2023 •

edited

Loading