Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feature: add --working-mode flag for yurthub #483

Merged
merged 1 commit into from
Sep 22, 2021
Merged

feature: add --working-mode flag for yurthub #483

merged 1 commit into from
Sep 22, 2021

Conversation

DrmagicE
Copy link
Member

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespace from that line:
/kind bug
/kind documentation
/kind enhancement
/kind good-first-issue
/kind feature
/kind question
/kind design
/sig ai
/sig iot
/sig network
/sig storage
/sig storage

/kind feature

What this PR does / why we need it:

When YurtHub is deployed on cloud side, some features such as local disk cache, GC manager and discardcloudservice filter should be disabled. We need this option to specify where yurthub is working.

Which issue(s) this PR fixes:

Fixes #476

Special notes for your reviewer:

/assign @rambohe-ch

Does this PR introduce a user-facing change?

other Note

@openyurt-bot
Copy link
Collaborator

@DrmagicE: GitHub didn't allow me to assign the following users: your_reviewer.

Note that only openyurtio members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespace from that line:
/kind bug
/kind documentation
/kind enhancement
/kind good-first-issue
/kind feature
/kind question
/kind design
/sig ai
/sig iot
/sig network
/sig storage
/sig storage

/kind feature

What this PR does / why we need it:

When YurtHub is deployed on cloud side, some features such as local disk cache, GC manager and discardcloudservice filter should be disabled. We need this option to specify where yurthub is working.

Which issue(s) this PR fixes:

Fixes #476

Special notes for your reviewer:

/assign @rambohe-ch

Does this PR introduce a user-facing change?

other Note

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openyurt-bot openyurt-bot added the kind/feature kind/feature label Sep 21, 2021
@openyurt-bot openyurt-bot added the size/L size/L: 100-499 label Sep 21, 2021
@rambohe-ch
Copy link
Member

@DrmagicE Would you be able to upload the detail logs of yurthub running as cloud working mode?

@DrmagicE
Copy link
Member Author

@DrmagicE Would you be able to upload the detail logs of yurthub running as cloud working mode?

Sure, I run yurthub and then restart kubelet to check whether the masterservice filter is working, here is the log:

$ ./yurthub --node-name=master-1 --working-mode=cloud --server-addr=https://192.168.33.220:6443  -v=2
yurthub version: projectinfo.Info{GitVersion:"v0.4.1", GitCommit:"67c4311", BuildDate:"2021-09-21T14:07:31Z", GoVersion:"go1.14.4", Compiler:"gc", Platform:"linux/amd64"}
I0921 22:09:51.653630   24167 start.go:62] FLAG: --access-server-through-hub="true"
I0921 22:09:51.653756   24167 start.go:62] FLAG: --add_dir_header="false"
I0921 22:09:51.653761   24167 start.go:62] FLAG: --alsologtostderr="false"
I0921 22:09:51.653764   24167 start.go:62] FLAG: --bind-address="127.0.0.1"
I0921 22:09:51.653769   24167 start.go:62] FLAG: --cert-mgr-mode="hubself"
I0921 22:09:51.653772   24167 start.go:62] FLAG: --disabled-resource-filters="[]"
I0921 22:09:51.653781   24167 start.go:62] FLAG: --disk-cache-path="/etc/kubernetes/cache/"
I0921 22:09:51.653785   24167 start.go:62] FLAG: --dummy-if-ip="169.254.2.1"
I0921 22:09:51.653788   24167 start.go:62] FLAG: --dummy-if-name="yurthub-dummy0"
I0921 22:09:51.653791   24167 start.go:62] FLAG: --enable-dummy-if="true"
I0921 22:09:51.653794   24167 start.go:62] FLAG: --enable-iptables="true"
I0921 22:09:51.653797   24167 start.go:62] FLAG: --enable-resource-filter="true"
I0921 22:09:51.653800   24167 start.go:62] FLAG: --gc-frequency="120"
I0921 22:09:51.653804   24167 start.go:62] FLAG: --heartbeat-failed-retry="3"
I0921 22:09:51.653808   24167 start.go:62] FLAG: --heartbeat-healthy-threshold="2"
I0921 22:09:51.653811   24167 start.go:62] FLAG: --heartbeat-timeout-seconds="2"
I0921 22:09:51.653813   24167 start.go:62] FLAG: --help="false"
I0921 22:09:51.653816   24167 start.go:62] FLAG: --join-token=""
I0921 22:09:51.653819   24167 start.go:62] FLAG: --kubelet-ca-file="/etc/kubernetes/pki/ca.crt"
I0921 22:09:51.653823   24167 start.go:62] FLAG: --kubelet-client-certificate="/var/lib/kubelet/pki/kubelet-client-current.pem"
I0921 22:09:51.653827   24167 start.go:62] FLAG: --lb-mode="rr"
I0921 22:09:51.653829   24167 start.go:62] FLAG: --log-flush-frequency="5s"
I0921 22:09:51.653833   24167 start.go:62] FLAG: --log_backtrace_at=":0"
I0921 22:09:51.653838   24167 start.go:62] FLAG: --log_dir=""
I0921 22:09:51.653842   24167 start.go:62] FLAG: --log_file=""
I0921 22:09:51.653845   24167 start.go:62] FLAG: --log_file_max_size="1800"
I0921 22:09:51.653848   24167 start.go:62] FLAG: --logtostderr="true"
I0921 22:09:51.653851   24167 start.go:62] FLAG: --max-requests-in-flight="250"
I0921 22:09:51.653854   24167 start.go:62] FLAG: --node-name="master-1"
I0921 22:09:51.653857   24167 start.go:62] FLAG: --nodepool-name=""
I0921 22:09:51.653865   24167 start.go:62] FLAG: --profiling="true"
I0921 22:09:51.653868   24167 start.go:62] FLAG: --proxy-port="10261"
I0921 22:09:51.653871   24167 start.go:62] FLAG: --proxy-secure-port="10268"
I0921 22:09:51.653874   24167 start.go:62] FLAG: --root-dir="/var/lib/yurthub"
I0921 22:09:51.653877   24167 start.go:62] FLAG: --serve-port="10267"
I0921 22:09:51.653880   24167 start.go:62] FLAG: --server-addr="https://192.168.33.220:6443"
I0921 22:09:51.653883   24167 start.go:62] FLAG: --skip_headers="false"
I0921 22:09:51.653886   24167 start.go:62] FLAG: --skip_log_headers="false"
I0921 22:09:51.653889   24167 start.go:62] FLAG: --stderrthreshold="2"
I0921 22:09:51.653892   24167 start.go:62] FLAG: --v="2"
I0921 22:09:51.653895   24167 start.go:62] FLAG: --version="false"
I0921 22:09:51.653898   24167 start.go:62] FLAG: --vmodule=""
I0921 22:09:51.653902   24167 start.go:62] FLAG: --working-mode="cloud"
I0921 22:09:51.653923   24167 config.go:188] yurthub would connect remote servers: https://192.168.33.220:6443
I0921 22:09:51.655557   24167 restmapper.go:83] reset DynamicRESTMapper to map[apps.openyurt.io/v1alpha1, Resource=nodepool:apps.openyurt.io/v1alpha1, Kind=NodePool apps.openyurt.io/v1alpha1, Resource=nodepools:apps.openyurt.io/v1alpha1, Kind=NodePool]
I0921 22:09:51.655985   24167 filter.go:93] Filter servicetopology registered successfully
I0921 22:09:51.655994   24167 filter.go:93] Filter masterservice registered successfully
I0921 22:09:51.656000   24167 filter.go:93] Filter discardcloudservice registered successfully
I0921 22:09:51.656008   24167 start.go:72] yurthub cfg: &config.YurtHubConfiguration{LBMode:"rr", RemoteServers:[]*url.URL{(*url.URL)(0xc00033bc00)}, YurtHubServerAddr:"127.0.0.1:10267", YurtHubProxyServerAddr:"127.0.0.1:10261", YurtHubProxyServerSecureAddr:"127.0.0.1:10268", YurtHubProxyServerDummyAddr:"169.254.2.1:10261", YurtHubProxyServerSecureDummyAddr:"169.254.2.1:10268", GCFrequency:120, CertMgrMode:"hubself", KubeletRootCAFilePath:"/etc/kubernetes/pki/ca.crt", KubeletPairFilePath:"/var/lib/kubelet/pki/kubelet-client-current.pem", NodeName:"master-1", HeartbeatFailedRetry:3, HeartbeatHealthyThreshold:2, HeartbeatTimeoutSeconds:2, MaxRequestInFlight:250, JoinToken:"", RootDir:"/var/lib/yurthub", EnableProfiling:true, EnableDummyIf:true, EnableIptables:true, HubAgentDummyIfName:"yurthub-dummy0", StorageWrapper:(*cachemanager.storageWrapper)(0xc00012ba80), SerializerManager:(*serializer.SerializerManager)(0xc00012bac0), RESTMapperManager:(*meta.RESTMapperManager)(0xc00012bb00), TLSConfig:(*tls.Config)(nil), MutatedMasterServiceAddr:"169.254.2.1:10268", Filters:(*filter.Filters)(0xc00048a480), SharedFactory:(*informers.sharedInformerFactory)(0xc0006a8a50), YurtSharedFactory:(*externalversions.sharedInformerFactory)(0xc0006a8aa0), WorkingMode:"cloud"}
I0921 22:09:51.656050   24167 start.go:87] 1. register cert managers
I0921 22:09:51.656060   24167 certificate.go:60] Registered certificate manager kubelet
I0921 22:09:51.656065   24167 certificate.go:60] Registered certificate manager hubself
I0921 22:09:51.656068   24167 start.go:93] 2. create cert manager with hubself mode
I0921 22:09:51.656089   24167 cert_mgr.go:214] /var/lib/yurthub/pki/ca.crt file already exists, so skip to create ca file
I0921 22:09:51.656094   24167 cert_mgr.go:127] use /var/lib/yurthub/pki/ca.crt ca file to bootstrap yurthub
I0921 22:09:51.656213   24167 cert_mgr.go:289] yurthub bootstrap conf file already exists, skip init bootstrap
I0921 22:09:51.656245   24167 certificate_store.go:130] Loading cert/key pair from "/var/lib/yurthub/pki/yurthub-current.pem".
I0921 22:09:51.665234   24167 certificate_manager.go:282] Certificate rotation is enabled.
I0921 22:09:51.665293   24167 cert_mgr.go:412] yurthub config file already exists, skip init config file
I0921 22:09:51.665306   24167 start.go:100] 3. new transport manager
I0921 22:09:51.665317   24167 transport.go:60] use /var/lib/yurthub/pki/ca.crt ca cert file to access remote server
I0921 22:09:51.665406   24167 certificate_manager.go:553] Certificate expiration is 2022-09-09 07:27:36 +0000 UTC, rotation deadline is 2022-06-25 11:30:17.021332313 +0000 UTC
I0921 22:09:51.665436   24167 certificate_manager.go:288] Waiting 6645h20m25.355898654s for next certificate rotation
I0921 22:09:51.665533   24167 start.go:107] 4. create health checker for remote servers
I0921 22:09:51.667038   24167 connrotation.go:145] create a connection from 192.168.33.220:56578 to 192.168.33.220:6443, total 1 connections in transport manager dialer
I0921 22:09:51.685492   24167 start.go:115] 5. new restConfig manager for hubself mode
I0921 22:09:51.685509   24167 start.go:122] 6. create tls config for secure servers
I0921 22:09:51.686435   24167 config.go:114] re-fix hub rest config host successfully with server https://192.168.33.220:6443
I0921 22:09:51.688175   24167 certmanager.go:48] subject of yurthub server certificate
I0921 22:09:51.688200   24167 certificate_store.go:130] Loading cert/key pair from "/var/lib/yurthub/pki/yurthub-server-current.pem".
I0921 22:09:51.688359   24167 certificate_manager.go:282] Certificate rotation is enabled.
I0921 22:09:51.688415   24167 certificate_manager.go:553] Certificate expiration is 2022-09-09 08:05:58 +0000 UTC, rotation deadline is 2022-07-13 00:28:07.183490938 +0000 UTC
I0921 22:09:51.688431   24167 certificate_manager.go:288] Waiting 7066h18m15.495061948s for next certificate rotation
I0921 22:09:56.688534   24167 start.go:137] 7. disable cache manager for node master-1 because it is a cloud node
I0921 22:09:56.688572   24167 start.go:149] 8. disable gc manager for node master-1 because it is a cloud node
I0921 22:09:56.688579   24167 start.go:153] 9. new filter chain for mutating response body
I0921 22:09:56.688721   24167 filter.go:70] Filter servicetopology initialize successfully
I0921 22:09:56.688739   24167 filter.go:70] Filter masterservice initialize successfully
I0921 22:09:56.688746   24167 start.go:160] 10. new reverse proxy handler for remote servers
I0921 22:09:56.688783   24167 start.go:168] 11. create dummy network interface yurthub-dummy0 and init iptables manager
I0921 22:09:56.721263   24167 start.go:175] 12. new yurthub server and begin to serve, dummy proxy server: 169.254.2.1:10261, secure dummy proxy server: 169.254.2.1:10268
I0921 22:09:56.721289   24167 start.go:184] 12. new yurthub server and begin to serve, proxy server: 127.0.0.1:10261, secure proxy server: 127.0.0.1:10268, hub server: 127.0.0.1:10267
I0921 22:09:56.722128   24167 reflector.go:175] Starting reflector *v1.Service (24h0m0s) from pkg/mod/k8s.io/client-go@v0.18.8/tools/cache/reflector.go:125
I0921 22:09:56.722409   24167 reflector.go:175] Starting reflector *v1alpha1.NodePool (24h0m0s) from pkg/mod/k8s.io/client-go@v0.18.8/tools/cache/reflector.go:125
I0921 22:09:56.723305   24167 util.go:232] start proxying: get /apis/apps.openyurt.io/v1alpha1/nodepools?limit=500&resourceVersion=0, in flight requests: 1
I0921 22:09:56.723585   24167 util.go:232] start proxying: get /api/v1/services?limit=500&resourceVersion=0, in flight requests: 2
I0921 22:09:56.725512   24167 util.go:215] yurthub list nodepools: /apis/apps.openyurt.io/v1alpha1/nodepools?limit=500&resourceVersion=0 with status code 200, spent 2.16761ms
I0921 22:09:56.725714   24167 util.go:215] yurthub list services: /api/v1/services?limit=500&resourceVersion=0 with status code 200, spent 2.109334ms
I0921 22:09:56.727858   24167 util.go:232] start proxying: get /api/v1/services?allowWatchBookmarks=true&resourceVersion=351555&timeout=8m58s&timeoutSeconds=538&watch=true, in flight requests: 1
I0921 22:09:56.728257   24167 util.go:232] start proxying: get /apis/apps.openyurt.io/v1alpha1/nodepools?allowWatchBookmarks=true&resourceVersion=354410&timeout=7m25s&timeoutSeconds=445&watch=true, in flight requests: 2
I0921 22:09:57.023507   24167 util.go:232] start proxying: get /api/v1/nodes?allowWatchBookmarks=true&resourceVersion=354453&timeout=6m44s&timeoutSeconds=404&watch=true, in flight requests: 3
I0921 22:09:57.024022   24167 connrotation.go:145] create a connection from 192.168.33.220:56776 to 192.168.33.220:6443, total 2 connections in transport manager dialer
I0921 22:09:57.043346   24167 util.go:232] start proxying: get /api/v1/endpoints?allowWatchBookmarks=true&resourceVersion=354672&timeout=5m12s&timeoutSeconds=312&watch=true, in flight requests: 4
I0921 22:09:57.047187   24167 util.go:232] start proxying: get /api/v1/namespaces/monitoring/endpoints?allowWatchBookmarks=true&resourceVersion=354431&timeout=7m58s&timeoutSeconds=478&watch=true, in flight requests: 5
I0921 22:09:57.058686   24167 util.go:232] start proxying: get /api/v1/services?allowWatchBookmarks=true&labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=351555&timeout=7m26s&timeoutSeconds=446&watch=true, in flight requests: 6
I0921 22:09:57.058981   24167 util.go:232] start proxying: get /api/v1/namespaces/monitoring/endpoints?allowWatchBookmarks=true&resourceVersion=354431&timeout=5m25s&timeoutSeconds=325&watch=true, in flight requests: 7
I0921 22:09:57.059198   24167 util.go:232] start proxying: get /api/v1/namespaces/monitoring/services?allowWatchBookmarks=true&resourceVersion=351555&timeout=9m33s&timeoutSeconds=573&watch=true, in flight requests: 8
I0921 22:09:57.059372   24167 util.go:232] start proxying: get /api/v1/namespaces/kube-system/endpoints?allowWatchBookmarks=true&resourceVersion=354672&timeout=7m7s&timeoutSeconds=427&watch=true, in flight requests: 9
I0921 22:09:57.059675   24167 util.go:232] start proxying: get /api/v1/namespaces/monitoring/services?allowWatchBookmarks=true&resourceVersion=351555&timeout=5m7s&timeoutSeconds=307&watch=true, in flight requests: 10
I0921 22:09:57.059878   24167 util.go:232] start proxying: get /api/v1/namespaces?allowWatchBookmarks=true&resourceVersion=351555&timeout=7m3s&timeoutSeconds=423&watch=true, in flight requests: 11
I0921 22:09:57.060234   24167 util.go:232] start proxying: get /api/v1/namespaces/default/pods?allowWatchBookmarks=true&resourceVersion=352342&timeout=5m17s&timeoutSeconds=317&watch=true, in flight requests: 12
I0921 22:09:57.060458   24167 util.go:232] start proxying: get /api/v1/namespaces/default/endpoints?allowWatchBookmarks=true&resourceVersion=352344&timeout=8m39s&timeoutSeconds=519&watch=true, in flight requests: 13
I0921 22:09:57.061298   24167 util.go:232] start proxying: get /api/v1/namespaces/monitoring/pods?allowWatchBookmarks=true&resourceVersion=354425&timeout=8m46s&timeoutSeconds=526&watch=true, in flight requests: 14
I0921 22:09:57.061505   24167 util.go:232] start proxying: get /api/v1/namespaces/monitoring/pods?allowWatchBookmarks=true&resourceVersion=354425&timeout=9m28s&timeoutSeconds=568&watch=true, in flight requests: 15
I0921 22:09:57.061783   24167 util.go:232] start proxying: get /api/v1/namespaces/kube-system/pods?allowWatchBookmarks=true&resourceVersion=354438&timeout=5m0s&timeoutSeconds=300&watch=true, in flight requests: 16
I0921 22:09:57.062308   24167 util.go:232] start proxying: get /api/v1/namespaces/kube-system/services?allowWatchBookmarks=true&resourceVersion=351555&timeout=8m51s&timeoutSeconds=531&watch=true, in flight requests: 17
I0921 22:09:57.064728   24167 util.go:232] start proxying: get /apis/discovery.k8s.io/v1beta1/endpointslices?allowWatchBookmarks=true&labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=354432&timeout=5m44s&timeoutSeconds=344&watch=true, in flight requests: 18
I0921 22:09:57.067401   24167 util.go:232] start proxying: get /api/v1/namespaces/default/services?allowWatchBookmarks=true&resourceVersion=351555&timeout=9m43s&timeoutSeconds=583&watch=true, in flight requests: 19
I0921 22:09:57.069959   24167 util.go:232] start proxying: get /api/v1/services?allowWatchBookmarks=true&resourceVersion=351555&timeout=8m14s&timeoutSeconds=494&watch=true, in flight requests: 20
I0921 22:10:00.373246   24167 util.go:232] start proxying: post /apis/authentication.k8s.io/v1beta1/tokenreviews, in flight requests: 21
I0921 22:10:00.375434   24167 util.go:215] kube-rbac-proxy create tokenreviews: /apis/authentication.k8s.io/v1beta1/tokenreviews with status code 201, spent 2.1432ms
I0921 22:10:00.710924   24167 util.go:232] start proxying: get /api/v1/nodes?fieldSelector=metadata.name%3Dmaster-1&limit=500&resourceVersion=0, in flight requests: 21
I0921 22:10:00.711166   24167 util.go:232] start proxying: get /api/v1/pods?fieldSelector=spec.nodeName%3Dmaster-1&limit=500&resourceVersion=0, in flight requests: 22
I0921 22:10:00.711293   24167 util.go:232] start proxying: get /api/v1/services?limit=500&resourceVersion=0, in flight requests: 23
I0921 22:10:00.714851   24167 util.go:215] kubelet list nodes: /api/v1/nodes?fieldSelector=metadata.name%3Dmaster-1&limit=500&resourceVersion=0 with status code 200, spent 3.882596ms
I0921 22:10:00.714864   24167 handler.go:83] mutate master service into ClusterIP:Port=169.254.2.1:10268 for request kubelet list services: https://192.168.33.220:6443/api/v1/services?limit=500&resourceVersion=0
I0921 22:10:00.715042   24167 util.go:215] kubelet list pods: /api/v1/pods?fieldSelector=spec.nodeName%3Dmaster-1&limit=500&resourceVersion=0 with status code 200, spent 3.843424ms
I0921 22:10:00.715046   24167 util.go:215] kubelet list services: /api/v1/services?limit=500&resourceVersion=0 with status code 200, spent 3.733287ms

@rambohe-ch
Copy link
Member

@DrmagicE In cloud woking mode, It's not need to start WithListRequestSelector and WithCacheHeaderCheck request handlers.

code link: https://github.com/openyurtio/openyurt/blob/master/pkg/yurthub/proxy/proxy.go#L88-L98

@DrmagicE
Copy link
Member Author

@rambohe-ch got it, will check it later.

@DrmagicE
Copy link
Member Author

@DrmagicE In cloud woking mode, It's not need to start WithListRequestSelector and WithCacheHeaderCheck request handlers.

code link: https://github.com/openyurtio/openyurt/blob/master/pkg/yurthub/proxy/proxy.go#L88-L98

@rambohe-ch updated, please have a look.

@rambohe-ch
Copy link
Member

/lgtm
/approve

@openyurt-bot openyurt-bot added the lgtm lgtm label Sep 22, 2021
@openyurt-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: DrmagicE, rambohe-ch

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openyurt-bot openyurt-bot added the approved approved label Sep 22, 2021
@openyurt-bot openyurt-bot merged commit da69115 into openyurtio:master Sep 22, 2021
This was referenced Nov 5, 2021
Merged
Closed
Merged
@DrmagicE DrmagicE mentioned this pull request Nov 17, 2021
Closed
MrGirl pushed a commit to MrGirl/openyurt that referenced this pull request Mar 29, 2022
@DrmagicE
feature: add --working-mode flag for yurthub (openyurtio#483)
bb005bb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rambohe-ch rambohe-ch rambohe-ch approved these changes

@Fei-Guo Fei-Guo Awaiting requested review from Fei-Guo

@yixingjia yixingjia Awaiting requested review from yixingjia

Assignees

@rambohe-ch rambohe-ch

Labels
approved approved kind/feature kind/feature lgtm lgtm size/L size/L: 100-499
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[feature request] Add command-line option for YurtHub to disable local cache and GC
3 participants
@DrmagicE @openyurt-bot @rambohe-ch