failed to open dashboard in minikube in some situations where --extra-config is used #87

darren-fu · 2017-12-22T05:37:22Z

darrenfu:~$ kubectl version --short
Client Version: v1.9.0
Server Version: v1.8.0
darrenfu:~$ conduit version
Client version: v0.1.1
Server version: v0.1.1

darrenfu:~$ conduit dashboard
Running `kubectl proxy -p 8001`
Starting to serve on 127.0.0.1:8001

Opening [http://127.0.0.1:8001/api/v1/namespaces/conduit/services/web:http/proxy/] in the default browser

but i get nothing but an error when visit the url : http://127.0.0.1:8001/api/v1/namespaces/conduit/services/web:http/proxy/routes

# error on the page
Error: 'EOF'
Trying to reach: 'http://172.17.0.11:8084/routes'

The text was updated successfully, but these errors were encountered:

klingerf · 2017-12-22T21:55:53Z

Hi @darren-fu -- thanks for reporting this! Can you provide a bit more info about your kubernetes environment? Are you running in minikube, cloud hosted, etc? Is RBAC enabled?

Fwiw, I ran into this issue in a minikube environment that didn't have RBAC properly configured. The error looked like this in my web browser:

The issue was that some of the containers in the kube-system namespace had failed to start as a result of me starting minikube with RBAC enabled. I fixed it by applying the config found here: kubernetes/minikube#1734 (comment)

Let me know if that works for you. We should certainly make this easier to debug as well.

darren-fu · 2017-12-26T03:00:13Z

Hi, @klingerf ,yes, I use minikube.

#start minikube
minikube start --extra-config=apiserver.Authorization.Mode=RBAC

# po status
darrenfu:~$ kubectl get po --all-namespaces
NAMESPACE     NAME                          READY     STATUS    RESTARTS   AGE
conduit       controller-598cb87c47-x4lnf   5/5       Running   10         41m
conduit       prometheus-66dcf6fb5-f5dpb    2/2       Running   4          41m
conduit       web-795cd7fdcd-xbrgs          1/1       Running   2          41m
kube-system   kube-addon-manager-minikube   1/1       Running   4          4d
kube-system   kube-dns-6fc954457d-bl5wm     3/3       Running   30         22d
kube-system   kubernetes-dashboard-gtlch    1/1       Running   4          4d
kube-system   kubernetes-dashboard-zvtjc    1/1       Running   10         22d
kube-system   storage-provisioner           1/1       Running   3          4d

I apply the RBAC config you mentioned here:
kubernetes/minikube#1734 (comment)
,BTW, I didn't change anything in this config, just touch a new file role.yaml

# apply RBAC
kubectl apply -f role.yaml

when I try to access dashboard, get same error(try to restart minikube, still get this error)

Error: 'EOF'
Trying to reach: 'http://172.17.0.5:8084/'

As a new user for K8S, I don't how to fix this now, can you give me some help? thx

franziskagoltz · 2017-12-26T23:41:58Z

Hi @darren-fu,

Sorry that you're still having issues with this. Can you try starting minikube without RBAC enabled? And when you reinstall conduit, do you still get the same error?

We want to check if this is related to your minikube and kubernetes setup or if the issue is related to RBAC use with conduit.

Thank you!

darren-fu · 2017-12-27T02:53:15Z

@hi @franziskagoltz,
I reinstall conduit as you said, unfortunately still can not open the dashboard page.


darrenfu:~$ minikube start --extra-config=apiserver.AuthorizationMode=AlwaysAllow
Starting local Kubernetes v1.8.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
Kubectl is now configured to use the cluster.
Loading cached images from config file.

darrenfu:~$ kubectl get po --all-namespaces
NAMESPACE     NAME                          READY     STATUS    RESTARTS   AGE
conduit       controller-598cb87c47-x4lnf   5/5       Running   15         1d
conduit       prometheus-66dcf6fb5-f5dpb    2/2       Running   6          1d
conduit       web-795cd7fdcd-xbrgs          1/1       Running   3          1d
kube-system   kube-addon-manager-minikube   1/1       Running   5          5d
kube-system   kube-dns-6fc954457d-bl5wm     3/3       Running   33         23d
kube-system   kubernetes-dashboard-gtlch    1/1       Running   5          5d
kube-system   kubernetes-dashboard-zvtjc    1/1       Running   11         23d
kube-system   storage-provisioner           1/1       Running   4          5d

# delete conduit
darrenfu:~$ conduit install | kubectl delete -f -
namespace "conduit" deleted
serviceaccount "conduit-controller" deleted
clusterrole "conduit-controller" deleted
clusterrolebinding "conduit-controller" deleted
service "api" deleted
service "proxy-api" deleted
deployment "controller" deleted
service "web" deleted
service "prometheus" deleted

reinstall conduit

darrenfu:~$ conduit install | kubectl apply -f -
namespace "conduit" created
serviceaccount "conduit-controller" created
clusterrole "conduit-controller" created
clusterrolebinding "conduit-controller" created
service "api" created
service "proxy-api" created
deployment "controller" created
service "web" created
deployment "web" created
service "prometheus" created
deployment "prometheus" created
configmap "prometheus-config" created

darrenfu:~$ kubectl get po --all-namespaces
NAMESPACE     NAME                          READY     STATUS    RESTARTS   AGE
conduit       controller-598cb87c47-b6gkw   5/5       Running   0          31s
conduit       prometheus-66dcf6fb5-xz28k    2/2       Running   0          31s
conduit       web-7cf459866b-wj4mb          1/1       Running   0          31s
kube-system   kube-addon-manager-minikube   1/1       Running   5          5d
kube-system   kube-dns-6fc954457d-bl5wm     3/3       Running   33         23d
kube-system   kubernetes-dashboard-gtlch    1/1       Running   5          5d
kube-system   kubernetes-dashboard-zvtjc    1/1       Running   11         23d
kube-system   storage-provisioner           1/1       Running   4          5d

darrenfu:~$ conduit dashboard
Running `kubectl proxy -p 8001`
Starting to serve on 127.0.0.1:8001

Opening [http://127.0.0.1:8001/api/v1/namespaces/conduit/services/web:http/proxy/] in the default browser

get same error

Error: 'EOF'
Trying to reach: 'http://172.17.0.5:8084/'

again, version

darrenfu:~$ conduit version
Client version: v0.1.1
Server version: v0.1.1
darrenfu:~$ kubectl version --short
Client Version: v1.9.0
Server Version: v1.8.0
darrenfu:~$ minikube version
minikube version: v0.24.1

franziskagoltz · 2017-12-27T22:37:11Z

HI @darren-fu,

Thanks for the detailed set-up. It seems like the --extra-config=apiserver.AuthorizationMode=AlwaysAllow flag is causing some issues. I was able to reproduce your error when restarting minikube with the extra config.

In the meantime, if you run a minikube delete, (important step to make sure the extra-config flag isn't cached somewhere), followed by minikube start (without any extra flags), you should be able to continue the conduit install process and open the dashboard without any issues.

Let us know how this goes!

darren-fu · 2017-12-28T01:58:49Z

@franziskagoltz
Great, it works, now I can open the dashboard successfully after run minikube delete.
Thanks for the help, so I can try to add my service to Conduit.
I will close the issue.

wmorgan · 2017-12-28T07:26:14Z

It seems really strange to me that specifying AlwaysAllow would have any effect. According to https://kubernetes.io/docs/admin/kubelet-authentication-authorization/, AlwaysAllow should be the default behavior, so specifying it should be a noop...

darren-fu · 2018-01-02T01:44:57Z

@wmorgan @franziskagoltz
I think the problem is not AlwaysAllow.
Actually, I run minikube and Istio first, and I enable some alpha features
--extra-config=apiserver.Admission.PluginNames="Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,GenericAdmissionWebhook,ResourceQuota.
Then I use this kube cluster to run Conduit directly without runminikube delete.
FYI, I guess the kube cluster has some problem to run Conduit in this case, maybe some config cause this issue, hope this helps.

briansmith · 2018-02-15T04:00:50Z

It seems really strange to me that specifying AlwaysAllow would have any effect. According to https://kubernetes.io/docs/admin/kubelet-authentication-authorization/, AlwaysAllow should be the default behavior, so specifying it should be a noop...

See kubernetes/kubeadm#600 (comment).

briansmith · 2018-02-15T04:04:35Z

@darren-fu wrote:

--extra-config=apiserver.Admission.PluginNames="Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,GenericAdmissionWebhook,ResourceQuota

@darren-fu You're comment seems to be missing a closing quote. Could you clarify it please?

I reopened this. We should verify that Conduit actually works correctly in the configurations given by @darren-fu. Besides the one I quoted above in this comment:

--extra-config=apiserver.AuthorizationMode=AlwaysAllow
--extra-config=apiserver.Admission.PluginNames="Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,GenericAdmissionWebhook,ResourceQuota
--extra-config=apiserver.AuthorizationMode=AlwaysAllow.

Hopefully this is just about RBAC-related issues that were fixed in more recent versions.

darren-fu · 2018-02-20T00:53:58Z

@briansmith yes, I missing a closing quote, rhe extra-config I use with minikube as follow:

--extra-config=apiserver.Admission.PluginNames="Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,GenericAdmissionWebhook,ResourceQuota"

klingerf · 2018-02-21T23:27:55Z

I did a bit of investigation for this issue. Here are the versions that I'm running:

$ minikube version
minikube version: v0.25.0
$ kubectl version --short
Client Version: v1.8.6
Server Version: v1.9.0
$ conduit version --client
Client version: v0.3.0

When I run:

$ minikube start --extra-config=apiserver.Admission.PluginNames="Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,GenericAdmissionWebhook,ResourceQuota"

The command completes, but the kubernetes cluster is unreachable via kubectl:

 $ kubectl get po
The connection to the server 192.168.99.104:8443 was refused - did you specify the right host or port?

So I was unable to test conduit with that configuration. Maybe there's some additional step I need to take to configure kubectl? Usually this happens automatically though.

When I run:

$ minikube start --extra-config=apiserver.AuthorizationMode=AlwaysAllow"

The command completes and the kubernetes cluster is reachable via kubectl.

Then I run:

$ conduit install | kubectl apply -f -

That command also completes successfully, and conduit check succeeds:

$ conduit check
kubernetes-api: can initialize the client.......................................[ok]
kubernetes-api: can query the Kubernetes API....................................[ok]
kubernetes-api: is running the minimum Kubernetes API version...................[ok]
conduit-api: can query the Conduit API..........................................[ok]
conduit-api[telemetry]: control plane can use telemetry service.................[ok]

Status check results are [ok]

When I run conduit dashboard, the dashboard opens successfully.

Based on these results, I'm concluding that:

conduit-v0.3.0 installs with the correct RBAC permissions
minikube-v0.25.0 fixed their kube-dns RBAC issues 🎉

I still don't know if conduit works with any of the other --extra-config options, since I could not get a cluster to start with those options set. @darren-fu Any chance you can do some additional investigation and report back?

klingerf · 2018-03-23T21:24:52Z

Am going to close this due to inactivity, please re-open if it's still an issue.

`bind::BoundService` wraps a `Reconnect` service and handles its Connect errors. However, `BoundService` exposes `Reconnect`'s Error type to callers even though these errors can never be returned. Furthermore, `Reconnect` is allowed be polled after returning an error, triggering the inner service to be rebuilt. We needlessly duplicate this logic in `BoundService`. Before splitting this file up into smaller chunks, let's update `BoundService` to more narrowly adhere to `Reconnect`s API: - Only the inner error type is returned. `unreachable!` assertions have been made where error variants cannot be returned. - Do not "rebind" the stack explicitly. Instead, let `Reconnect` do this. - Now BoundService::call may panic if invoked before poll_ready. It's a programming error, since `Reconnect` requires that `poll_ready` be called first.

darren-fu closed this as completed Dec 28, 2017

briansmith changed the title ~~failed to open dashboard~~ failed to open dashboard in minikube in some situations where --extra-config is used Feb 15, 2018

briansmith added this to the Conduit 0.3 milestone Feb 15, 2018

briansmith reopened this Feb 19, 2018

klingerf self-assigned this Feb 19, 2018

klingerf removed this from the Conduit 0.3 milestone Feb 21, 2018

klingerf closed this as completed Mar 23, 2018

github-actions bot locked as resolved and limited conversation to collaborators Jul 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

failed to open dashboard in minikube in some situations where --extra-config is used #87

failed to open dashboard in minikube in some situations where --extra-config is used #87

darren-fu commented Dec 22, 2017

klingerf commented Dec 22, 2017

darren-fu commented Dec 26, 2017 •

edited

Loading

franziskagoltz commented Dec 26, 2017

darren-fu commented Dec 27, 2017 •

edited

Loading

franziskagoltz commented Dec 27, 2017

darren-fu commented Dec 28, 2017 •

edited

Loading

wmorgan commented Dec 28, 2017

darren-fu commented Jan 2, 2018

briansmith commented Feb 15, 2018

briansmith commented Feb 15, 2018

darren-fu commented Feb 20, 2018

klingerf commented Feb 21, 2018

klingerf commented Mar 23, 2018

failed to open dashboard in minikube in some situations where --extra-config is used #87

failed to open dashboard in minikube in some situations where --extra-config is used #87

Comments

darren-fu commented Dec 22, 2017

klingerf commented Dec 22, 2017

darren-fu commented Dec 26, 2017 • edited Loading

franziskagoltz commented Dec 26, 2017

darren-fu commented Dec 27, 2017 • edited Loading

franziskagoltz commented Dec 27, 2017

darren-fu commented Dec 28, 2017 • edited Loading

wmorgan commented Dec 28, 2017

darren-fu commented Jan 2, 2018

briansmith commented Feb 15, 2018

briansmith commented Feb 15, 2018

darren-fu commented Feb 20, 2018

klingerf commented Feb 21, 2018

klingerf commented Mar 23, 2018

darren-fu commented Dec 26, 2017 •

edited

Loading

darren-fu commented Dec 27, 2017 •

edited

Loading

darren-fu commented Dec 28, 2017 •

edited

Loading