-
Notifications
You must be signed in to change notification settings - Fork 689
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proxying doesn't work for services with different port and targetPort #299
Comments
I've had several goes at fixing this problem and I was hopeful I nailed it last time. Can you please grab a copy of /clusters (or run the contour cli cds k exec trick) and the service document from the api and I'll look into it. Thanks |
Maybe I've mistaken about the reason of problem. Because eds gives me right endpoint (it works):
Cluster:
And route:
describe svc
But eventually I get this from envoy in its logs:
|
That looks right. Is the dashboard talking http or https?
… On 25 Mar 2018, at 08:13, Alexander Lukyanchenko ***@***.***> wrote:
Maybe I've mistaken about the reason of problem. Because eds gives me right endpoint (it works):
resources: <
[type.googleapis.com/envoy.api.v2.ClusterLoadAssignment]: <
cluster_name: "kube-system/kubernetes-dashboard"
endpoints: <
lb_endpoints: <
endpoint: <
address: <
socket_address: <
address: "10.7.40.134"
port_value: 9090
>
>
>
>
>
>
>
Cluster:
resources: <
[type.googleapis.com/envoy.api.v2.Cluster]: <
name: "kube-system/kubernetes-dashboard/80"
type: EDS
eds_cluster_config: <
eds_config: <
api_config_source: <
api_type: GRPC
cluster_names: "contour"
>
>
service_name: "kube-system/kubernetes-dashboard"
>
connect_timeout: <
nanos: 250000000
>
>
>
And route:
virtual_hosts: <
name: "dashboard.host"
domains: "dashboard.host"
routes: <
match: <
prefix: "/"
>
route: <
cluster: "kube-system/kubernetes-dashboard/80"
>
>
>
describe svc
kubectl describe svc -n kube-system kubernetes-dashboard
Name: kubernetes-dashboard
Namespace: kube-system
Labels: addonmanager.kubernetes.io/mode: Reconcile k8s-app: kubernetes-dashboard kubernetes.io/cluster-service: true
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"k8s-app":"kubernetes-dashboard"},"name":"kubernetes-dashboard","namespace":...
Selector: k8s-app=kubernetes-dashboard
Type: ClusterIP
IP: 10.100.126.135
Port: <unset> 80/TCP
TargetPort: 9090/TCP
Endpoints: 10.7.40.134 <unset>, 9090, TCP
But eventually I get this from envoy in its logs:
source/common/upstream/cluster_manager_impl.cc:755] no healthy host for HTTP connection pool
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.
|
It talks http. |
Can you port forward to the [envoy admin interface][0] and grab the relevant portion of the /cluster output. I'm interested to see if there are no endpoints registered, or if they are, but are not responding. |
This is /cluster grep dashboard:
And this is stats of dashboard with one failed request:
And envoy logs with trace type at the moment of request:
|
Ok, that’s looking like the same error that others have been reporting. Your cluster has no endpoints even though they are there in the api
… On 27 Mar 2018, at 15:48, Alexander Lukyanchenko ***@***.***> wrote:
This is /cluster grep dashboard:
kube-system/kubernetes-dashboard/80::default_priority::max_connections::1024
kube-system/kubernetes-dashboard/80::default_priority::max_pending_requests::1024
kube-system/kubernetes-dashboard/80::default_priority::max_requests::1024
kube-system/kubernetes-dashboard/80::default_priority::max_retries::3
kube-system/kubernetes-dashboard/80::high_priority::max_connections::1024
kube-system/kubernetes-dashboard/80::high_priority::max_pending_requests::1024
kube-system/kubernetes-dashboard/80::high_priority::max_requests::1024
kube-system/kubernetes-dashboard/80::high_priority::max_retries::3
kube-system/kubernetes-dashboard/80::added_via_api::true
And this is stats of dashboard with one failed request:
cluster.kube-system/kubernetes-dashboard/80.bind_errors: 0
cluster.kube-system/kubernetes-dashboard/80.internal.upstream_rq_503: 1
cluster.kube-system/kubernetes-dashboard/80.internal.upstream_rq_5xx: 1
cluster.kube-system/kubernetes-dashboard/80.lb_healthy_panic: 1
cluster.kube-system/kubernetes-dashboard/80.lb_local_cluster_not_ok: 0
cluster.kube-system/kubernetes-dashboard/80.lb_recalculate_zone_structures: 0
cluster.kube-system/kubernetes-dashboard/80.lb_subsets_active: 0
cluster.kube-system/kubernetes-dashboard/80.lb_subsets_created: 0
cluster.kube-system/kubernetes-dashboard/80.lb_subsets_fallback: 0
cluster.kube-system/kubernetes-dashboard/80.lb_subsets_removed: 0
cluster.kube-system/kubernetes-dashboard/80.lb_subsets_selected: 0
cluster.kube-system/kubernetes-dashboard/80.lb_zone_cluster_too_small: 0
cluster.kube-system/kubernetes-dashboard/80.lb_zone_no_capacity_left: 0
cluster.kube-system/kubernetes-dashboard/80.lb_zone_number_differs: 0
cluster.kube-system/kubernetes-dashboard/80.lb_zone_routing_all_directly: 0
cluster.kube-system/kubernetes-dashboard/80.lb_zone_routing_cross_zone: 0
cluster.kube-system/kubernetes-dashboard/80.lb_zone_routing_sampled: 0
cluster.kube-system/kubernetes-dashboard/80.max_host_weight: 0
cluster.kube-system/kubernetes-dashboard/80.membership_change: 0
cluster.kube-system/kubernetes-dashboard/80.membership_healthy: 0
cluster.kube-system/kubernetes-dashboard/80.membership_total: 0
cluster.kube-system/kubernetes-dashboard/80.retry_or_shadow_abandoned: 0
cluster.kube-system/kubernetes-dashboard/80.update_attempt: 1
cluster.kube-system/kubernetes-dashboard/80.update_empty: 0
cluster.kube-system/kubernetes-dashboard/80.update_failure: 0
cluster.kube-system/kubernetes-dashboard/80.update_rejected: 0
cluster.kube-system/kubernetes-dashboard/80.update_success: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_cx_active: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_cx_close_notify: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_cx_connect_attempts_exceeded: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_cx_connect_fail: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_cx_connect_timeout: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_cx_destroy: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_cx_destroy_local: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_cx_destroy_local_with_active_rq: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_cx_destroy_remote: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_cx_destroy_remote_with_active_rq: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_cx_destroy_with_active_rq: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_cx_http1_total: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_cx_http2_total: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_cx_max_requests: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_cx_none_healthy: 1
cluster.kube-system/kubernetes-dashboard/80.upstream_cx_overflow: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_cx_protocol_error: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_cx_rx_bytes_buffered: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_cx_rx_bytes_total: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_cx_total: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_cx_tx_bytes_buffered: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_cx_tx_bytes_total: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_flow_control_backed_up_total: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_flow_control_drained_total: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_flow_control_paused_reading_total: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_flow_control_resumed_reading_total: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_rq_503: 1
cluster.kube-system/kubernetes-dashboard/80.upstream_rq_5xx: 1
cluster.kube-system/kubernetes-dashboard/80.upstream_rq_active: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_rq_cancelled: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_rq_maintenance_mode: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_rq_pending_active: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_rq_pending_failure_eject: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_rq_pending_overflow: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_rq_pending_total: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_rq_per_try_timeout: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_rq_retry: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_rq_retry_overflow: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_rq_retry_success: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_rq_rx_reset: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_rq_timeout: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_rq_total: 0
cluster.kube-system/kubernetes-dashboard/80.upstream_rq_tx_reset: 0
cluster.kube-system/kubernetes-dashboard/80.version: 0
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.
|
How can I debug it and how can I understand what's wrong with EDS communication? Only using tcpdump and see what's going there? |
Try turning up the envoy logging level, you can do it via the admin web interface.
You’ve already confirmed that the entries are present in EDS via contour cli eds so either envoy is rejecting the update (the cluster entry has a version of 0 which leads me to suspect this), or the endpoints are somehow broken or pointing to the wrong thing or the contour pod cannot communicate with them and thus they are not able to leave the prewarming state after which the configuration will be applied. Turning envoys logging up to debug will help prove or disprove these hypothesises
… On 27 Mar 2018, at 16:30, Alexander Lukyanchenko ***@***.***> wrote:
How can I debug it and how can I understand what's wrong with EDS communication? Only using tcpdump and see what's going there?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.
|
It seems like it's related with #291 |
Can you please set up port forwarding to the contour admin api 0 then paste the results on this command
|
Thanks to Alexander Lukyanchenko (@Lookyan) we have increased the general gRPC limits on both the Envoy client and Contour server well above anything that should be an issue for the immediate future. The symptoms of hitting gRPC limits vary, but are basically "envoy doesn't see changes in the API server until I restart ". The underlying cause is likely to be that you have a large (more than 100, possibly 200, the exact limit is not precisely known) number of Service objects in your cluster -- these don't have to be associated with an Ingress. Currently Contour creates a CDS Cluster record for any Service object it learns about through the API, see #298. Each CDS record will cause Envoy to open a new EDS stream, one per Cluster, which can blow through the default limits that Envoy, as the gRPC client, and Contour, as the gRPC server, have set. One of the easiest ways to detect if this issue is occuring in your cluster is too look for lines about "cluster warming"
Without a matching "warming complete" message. We believe we have addressed this issue, #291, and the fixes are available now to test. These changes are in master now, and available in the gcr.io/heptio-images/contour:master image for you to try. This has been backported the release-0.4 branch and are available in a short lived image gcr.io/heptio-images/contour:release-0.4 (it's not going to be deleted, but don't expect it to continue to be updated beyond the 0.4.1 release). |
Not sure why github won't auto close this, but it's fixed. |
I found interesting problem. When we have kubernetes service like this:
Then it doesn't work because, as far as I understand, envoy tries to proxy to the port 8890, but endpoint listens 8888 port. So, we get an answer from envoy, that this service doesn't have healthy endpoints.
The text was updated successfully, but these errors were encountered: