-
Notifications
You must be signed in to change notification settings - Fork 407
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
e2e: add e2e test into github action #496
Conversation
@Congrool: GitHub didn't allow me to assign the following users: your_reviewer. Note that only openyurtio members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Welcome @Congrool! It looks like this is your first PR to openyurtio/openyurt 🎉 |
bdea58a
to
95aedfa
Compare
Now give the description of the error. • Failure [5.112 seconds]
[k8s.io] yurttunnel-e2e-test
/home/runner/go/pkg/mod/github.com/kubernetes/kubernetes@v1.18.8/test/e2e/framework/framework.go:698
[k8s.io] [yurttunnel-e2e-test]: pod_operate_test_on_edge
/home/runner/go/pkg/mod/github.com/kubernetes/kubernetes@v1.18.8/test/e2e/framework/framework.go:698
yurttunnel_e2e_test_pod_run_on_edge [It]
/home/runner/work/openyurt/openyurt/test/e2e/yurttunnel/yurttunnel.go:170
fail run exec:test-po-on-edge
Unexpected error:
<*errors.errorString | 0xc0005ec5d0>: {
s: "unable to upgrade connection: unable to read error from server response",
}
unable to upgrade connection: unable to read error from server response
occurred
/home/runner/work/openyurt/openyurt/test/e2e/yurttunnel/yurttunnel.go:203 I tried to use the following command to setup openyurt cluster locally. cd ${YURT_ROOT}
export REGION=cn
sudo -E bash hack/local_up_openyurt.sh And got cluster as follows $ kubectl get pod -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-74ff55c5b-5jzkp 1/1 Running 0 46m 10.244.0.3 openyurt-e2e-test-control-plane <none> <none>
kube-system coredns-74ff55c5b-s5xbp 1/1 Running 0 46m 10.244.0.4 openyurt-e2e-test-control-plane <none> <none>
kube-system etcd-openyurt-e2e-test-control-plane 1/1 Running 0 46m 172.18.0.3 openyurt-e2e-test-control-plane <none> <none>
kube-system kindnet-mklwd 1/1 Running 0 45m 172.18.0.2 openyurt-e2e-test-worker <none> <none>
kube-system kindnet-nxw69 1/1 Running 0 46m 172.18.0.3 openyurt-e2e-test-control-plane <none> <none>
kube-system kube-apiserver-openyurt-e2e-test-control-plane 1/1 Running 0 46m 172.18.0.3 openyurt-e2e-test-control-plane <none> <none>
kube-system kube-controller-manager-openyurt-e2e-test-control-plane 1/1 Running 0 41m 172.18.0.3 openyurt-e2e-test-control-plane <none> <none>
kube-system kube-proxy-76hgh 1/1 Running 0 45m 172.18.0.2 openyurt-e2e-test-worker <none> <none>
kube-system kube-proxy-l75vd 1/1 Running 0 46m 172.18.0.3 openyurt-e2e-test-control-plane <none> <none>
kube-system kube-scheduler-openyurt-e2e-test-control-plane 1/1 Running 0 46m 172.18.0.3 openyurt-e2e-test-control-plane <none> <none>
kube-system yurt-controller-manager-7dd9fc956d-sv2g4 1/1 Running 0 42m 172.18.0.3 openyurt-e2e-test-control-plane <none> <none>
kube-system yurt-hub-openyurt-e2e-test-worker 1/1 Running 0 41m 172.18.0.2 openyurt-e2e-test-worker <none> <none>
kube-system yurt-tunnel-agent-fr6jn 1/1 Running 0 40m 172.18.0.2 openyurt-e2e-test-worker <none> <none>
kube-system yurt-tunnel-server-677bdcbcf6-2gjgd 1/1 Running 0 41m 172.18.0.3 openyurt-e2e-test-control-plane <none> <none>
local-path-storage local-path-provisioner-547f784dff-8w4v8 1/1 Running 0 46m 10.244.0.2 openyurt-e2e-test-control-plane <none> <none> Then I tried to error: unable to upgrade connection: unable to read error from server response I could not logs the agent as well: $ kubectl logs yurt-tunnel-agent-fr6jn -n kube-system
error: You must be logged in to the server (the server has asked for the client to provide credentials ( pods/log yurt-tunnel-agent-fr6jn)) I set the option I0926 04:08:02.313247 1 tracereq.go:99] start handling request GET https://172.18.0.2:10250/containerLogs/kube-system/yurt-tunnel-agent-4q8cc/yurt-tunnel-agent, from 172.18.0.3:39386 to 172.18.0.2:10250
I0926 04:08:02.313300 1 handler.go:111] request header in localHostProxyMiddleware: map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]] with host: 172.18.0.2:10250 and urL: https://172.18.0.2:10250/containerLogs/kube-system/yurt-tunnel-agent-4q8cc/yurt-tunnel-agent
I0926 04:08:02.313362 1 interceptor.go:78] Sending request to "172.18.0.2:10250".
I0926 04:08:02.313791 1 tunnel.go:36] "Received request for host" method="CONNECT" host="172.18.0.2:10250" userAgent="Go-http-client/1.1"
I0926 04:08:02.314096 1 tunnel.go:71] Set pending(rand=2610529275472644968) to &{0xc0006965a0 0xc00062b200 {} 0x5994a0 true false false false 0 {0 0} <nil> {0xc00047c700 map[] true true} map[] false 0 -1 200 false false [] 0 [83 117 110 44 32 50 54 32 83 101 112 32 50 48 50 49 32 48 52 58 48 56 58 48 50 32 71 77 84] [0 0 0 0 0 0 0 0 0 0] [50 48 48] 0xc0008121c0 0}
I0926 04:08:02.314211 1 desthost_backend_manager.go:49] "Get the backend through the DestHostBackendManager" destHost="172.18.0.2"
I0926 04:08:02.317577 1 server.go:689] "Received DIAL_RSP" random=2610529275472644968 agentID="openyurt-e2e-test-worker" connectionID=1
I0926 04:08:02.318268 1 server.go:269] "Register frontend for agent" frontend=&server.ProxyClientConnection{Mode:"http-connect", Grpc:client.ProxyService_ProxyServer(nil), HTTP:(*net.UnixConn)(0xc00000e298), connected:(chan struct {})(0xc00007b920), connectID:1, agentID:"openyurt-e2e-test-worker", start:time.Time{wall:0xc04c1a6892bb1420, ext:260641359529, loc:(*time.Location)(0x23095c0)}, backend:(*server.backend)(0xc0004568e0)} agentID="openyurt-e2e-test-worker" connectionID=1
I0926 04:08:02.318607 1 tunnel.go:128] "Starting proxy to host" host="172.18.0.2:10250" agentID="openyurt-e2e-test-worker" connectionID=1
I0926 04:08:02.318824 1 tunnel.go:163] "Forwarding data on tunnel to agent" bytes=239 totalBytes=239 agentID="openyurt-e2e-test-worker" connectionID=1
I0926 04:08:02.325538 1 server.go:718] "Received data from agent" bytes=2303 agentID="openyurt-e2e-test-worker" connectionID=1
I0926 04:08:02.325697 1 server.go:727] "DATA sent to frontend"
I0926 04:08:02.328240 1 tunnel.go:163] "Forwarding data on tunnel to agent" bytes=1223 totalBytes=1462 agentID="openyurt-e2e-test-worker" connectionID=1
I0926 04:08:02.328252 1 interceptor.go:112] successfully setup TLS connection to "172.18.0.2:10250" with headers:
User-Agent: Go-http-client/1.1
I0926 04:08:02.328376 1 tunnel.go:163] "Forwarding data on tunnel to agent" bytes=186 totalBytes=1648 agentID="openyurt-e2e-test-worker" connectionID=1
I0926 04:08:02.330133 1 server.go:718] "Received data from agent" bytes=161 agentID="openyurt-e2e-test-worker" connectionID=1
I0926 04:08:02.330258 1 server.go:727] "DATA sent to frontend"
I0926 04:08:02.330497 1 interceptor.go:282] interceptor: successfully read the http response from the proxy tunnel for request https://172.18.0.2:10250/containerLogs/kube-system/yurt-tunnel-agent-4q8cc/yurt-tunnel-agent
I0926 04:08:02.330613 1 interceptor.go:323] interceptor: stop serving request https://172.18.0.2:10250/containerLogs/kube-system/yurt-tunnel-agent-4q8cc/yurt-tunnel-agent with headers: map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
I0926 04:08:02.330691 1 tracereq.go:103] stop handling request GET https://172.18.0.2:10250/containerLogs/kube-system/yurt-tunnel-agent-4q8cc/yurt-tunnel-agent, request handling lasts 17.399337ms
I0926 04:08:02.330987 1 tunnel.go:163] "Forwarding data on tunnel to agent" bytes=24 totalBytes=1672 agentID="openyurt-e2e-test-worker" connectionID=1
I0926 04:08:02.331034 1 tunnel.go:141] "EOF from host" host="172.18.0.2:10250" agentID="openyurt-e2e-test-worker" connID=1
I0926 04:08:02.332747 1 tunnel.go:170] "Stopping transfer to host" host="172.18.0.2:10250" agentID="openyurt-e2e-test-worker" connectionID=1
I0926 04:08:02.332247 1 server.go:718] "Received data from agent" bytes=24 agentID="openyurt-e2e-test-worker" connectionID=1
E0926 04:08:02.332948 1 server.go:725] "send to client stream failure" err="write unix /tmp/interceptor-proxier.sock->@: use of closed network connection"
I0926 04:08:02.332968 1 server.go:732] "Received CLOSE_RSP" connectionID=1
I0926 04:08:02.332987 1 server.go:290] "Remove frontend for agent" agentID="openyurt-e2e-test-worker" connectionID=1
I0926 04:08:02.333010 1 server.go:747] "Close streaming" agentID="openyurt-e2e-test-worker" connectionID=1
I0926 04:08:02.333708 1 server.go:732] "Received CLOSE_RSP" connectionID=1
E0926 04:08:02.333830 1 server.go:735] "could not get frontent client" err="can't find agentID openyurt-e2e-test-worker in the frontends" for tunnel-agent I0926 04:05:30.567393 1 client.go:224] "Connect to" server="c219f2f5-1d95-4349-af1e-fa0ea7b615c1"
I0926 04:05:30.567499 1 clientset.go:190] "sync added client connecting to proxy server" serverID="c219f2f5-1d95-4349-af1e-fa0ea7b615c1"
I0926 04:05:30.567624 1 client.go:326] "Start serving" serverID="c219f2f5-1d95-4349-af1e-fa0ea7b615c1"
I0926 04:08:02.316217 1 client.go:346] "[tracing] recv packet" type="DIAL_REQ"
I0926 04:08:02.316286 1 client.go:355] received DIAL_REQ
I0926 04:08:02.316871 1 client.go:412] received dial request to tcp:172.18.0.2:10250 with random=2610529275472644968 and connID=1
I0926 04:08:02.321113 1 client.go:346] "[tracing] recv packet" type="DATA"
I0926 04:08:02.321219 1 client.go:416] "received DATA" connectionID=1
I0926 04:08:02.321365 1 client.go:492] "write to remote" connID=1 lastData=239
I0926 04:08:02.325188 1 client.go:461] "received data from remote" bytes=2303 connID=1
I0926 04:08:02.328451 1 client.go:346] "[tracing] recv packet" type="DATA"
I0926 04:08:02.328483 1 client.go:416] "received DATA" connectionID=1
I0926 04:08:02.328568 1 client.go:492] "write to remote" connID=1 lastData=1223
I0926 04:08:02.328620 1 client.go:346] "[tracing] recv packet" type="DATA"
I0926 04:08:02.328647 1 client.go:416] "received DATA" connectionID=1
I0926 04:08:02.328713 1 client.go:492] "write to remote" connID=1 lastData=186
I0926 04:08:02.329756 1 client.go:461] "received data from remote" bytes=161 connID=1
I0926 04:08:02.331452 1 client.go:346] "[tracing] recv packet" type="DATA"
I0926 04:08:02.331485 1 client.go:416] "received DATA" connectionID=1
I0926 04:08:02.331565 1 client.go:492] "write to remote" connID=1 lastData=24
I0926 04:08:02.331670 1 client.go:461] "received data from remote" bytes=24 connID=1
I0926 04:08:02.331716 1 client.go:461] "received data from remote" bytes=0 connID=1
I0926 04:08:02.331731 1 client.go:464] "connection read EOF" connID=1
I0926 04:08:02.331746 1 client.go:382] "close connection" connectionID=1
I0926 04:08:02.333303 1 client.go:346] "[tracing] recv packet" type="CLOSE_REQ"
I0926 04:08:02.333332 1 client.go:427] "received CLOSE_REQ" connectionID=1 for kubectl I0926 12:11:22.501541 1090376 round_trippers.go:425] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.20.2 (linux/amd64) kubernetes/faecb19" 'https://127.0.0.1:45873/api/v1/namespaces/kube-system/pods/yurt-tunnel-agent-4q8cc/log'
I0926 12:11:22.532743 1090376 round_trippers.go:445] GET https://127.0.0.1:45873/api/v1/namespaces/kube-system/pods/yurt-tunnel-agent-4q8cc/log 401 Unauthorized in 31 milliseconds
I0926 12:11:22.532784 1090376 round_trippers.go:451] Response Headers:
I0926 12:11:22.532808 1090376 round_trippers.go:454] Cache-Control: no-cache, private
I0926 12:11:22.532828 1090376 round_trippers.go:454] Content-Type: application/json
I0926 12:11:22.532847 1090376 round_trippers.go:454] Content-Length: 274
I0926 12:11:22.532867 1090376 round_trippers.go:454] Date: Sun, 26 Sep 2021 04:11:22 GMT
I0926 12:11:22.532920 1090376 request.go:1107] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the server has asked for the client to provide credentials ( pods/log yurt-tunnel-agent-4q8cc)","reason":"Unauthorized","details":{"name":"yurt-tunnel-agent-4q8cc","kind":"pods/log"},"code":401}
I0926 12:11:22.534648 1090376 helpers.go:216] server response object: [{
"metadata": {},
"status": "Failure",
"message": "the server has asked for the client to provide credentials ( pods/log yurt-tunnel-agent-4q8cc)",
"reason": "Unauthorized",
"details": {
"name": "yurt-tunnel-agent-4q8cc",
"kind": "pods/log"
},
"code": 401
}]
F0926 12:11:22.534722 1090376 helpers.go:115] error: You must be logged in to the server (the server has asked for the client to provide credentials ( pods/log yurt-tunnel-agent-4q8cc)) It seems that yurt-tunnel-server successfully fetched the log of |
Hi, I've revised the code and updated the error log. @rambohe-ch |
@Congrool Thank you for uploading detailed info of yurt-tunnel errors. It's looks like the webhook error that comes from kubelet, because kubelet usually send webhook request(subjectaccessreview) to kube-apiserver when he received pod/logs request. |
@rambohe-ch Thank you for your help. I checked the log of kubelet and yurthub. There's actually something wrong. Sep 26 08:23:28 openyurt-e2e-test-worker kubelet[26717]: E0926 08:23:28.869863 26717 server.go:259] Unable to authenticate the request due to an error: verifying certificate SN=324686761876842496809737091112637902102, SKID=, AKID=35:23:38:84:B4:76:01:63:06:04:1C:B5:32:D3:3B:48:A9:2D:52:57 failed: x509: certificate specifies an incompatible key usage Maybe it's the error that prevent kubelet from sending subjectaccessreview request to APIServer. But I have no idea that how does it occur? |
@Congrool Sorry, maybe i have made a mistake. response status code is 401 for authentication failed, and kubelet use |
@rambohe-ch OK, I retried with root@openyurt-e2e-test-worker:/# crictl ps
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
436e5d29fdb9d b124b0cf14d4a 7 hours ago Running yurt-tunnel-agent 0 53db1f4484d1e
92f9a2c1d71ac 3c29d52ea23d1 8 hours ago Running yurt-hub 0 4bdde362f5edc
227bc01ff75a5 d1ece029a5f6a 8 hours ago Running kube-proxy 0 21749708770fe
cdb2fe84a2bbf 6de166512aa22 8 hours ago Running kindnet-cni 0 552aabc77f71b
root@openyurt-e2e-test-worker:/# crictl logs 92f9a2c1d71ac 2>&1 | grep -i tokenreview
root@openyurt-e2e-test-worker:/# |
@Congrool @SataQiu and @DrmagicE have come across the same error, you can check the issue: #508, and this bug have fixed by @DrmagicE in PR: #509 |
2f49577
to
26aac9f
Compare
@Congrool Please remove WIP from title. |
@rambohe-ch ,Hello, I'm testing on my fork. I'll remove WIP when it passes the test. |
@Congrool ok |
/lgtm |
/lgtm cancel |
/lgtm |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Congrool, rambohe-ch The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
e2e: add e2e test into github action
What type of PR is this?
/kind enhancement
What this PR does / why we need it:
test/e2e
to check its correctness.hack/local_up_openyurt.sh
, with which we can quickly setup an openyurt cluster at local host withkind
.Which issue(s) this PR fixes:
Fixes #467
Special notes for your reviewer:
Does this PR introduce a user-facing change?
other Note
Currently, this shell skips yurt-tunnel e2e test at
test/e2e/yurttunnel
, because it always failed at this test. We should find out the reason.