Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

console 403: no RBAC policy matched for system:anonymous #411

Closed
michaelgugino opened this issue Oct 3, 2018 · 31 comments
Closed

console 403: no RBAC policy matched for system:anonymous #411

michaelgugino opened this issue Oct 3, 2018 · 31 comments

Comments

@michaelgugino
Copy link
Contributor

https://github.com/openshift/installer/blob/master/docs/dev/libvirt-howto.md#connect-to-the-cluster-console

https://${OPENSHIFT_INSTALL_CLUSTER_NAME}-api.${OPENSHIFT_INSTALL_BASE_DOMAIN}:6443/console/

does not pull up a console, just json output.

bf36c90

@wking
Copy link
Member

wking commented Oct 3, 2018

does not pull up a console, just json output.

What is the JSON output? Things have been exciting since #330 landed, there are many things that could be going wrong ;).

Also, we have a nice issue template... ;). Even if you push your issues with hub or whatever instead of going through GitHub's web UI, those are the sort of things we're looking for in bug reports.

@michaelgugino
Copy link
Contributor Author

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/console\": no RBAC policy matched",
  "reason": "Forbidden",
  "details": {
    
  },
  "code": 403
}

Yeah, it wasn't working last week either, not sure if the output was the same, but never got a console.

@wking
Copy link
Member

wking commented Oct 3, 2018

"message": "forbidden: User \"system:anonymous\" cannot get path \"/console\": no RBAC policy matched",

We think this is openshift/origin#20983.

@wking wking changed the title Cannot connect to console with libvirt instructions libvirt console 403: no RBAC policy matched for system:anonymous Oct 3, 2018
@mtnbikenc
Copy link
Member

Ran into the same problem.

Platform: libvirt
Accessing: https://test1-api.tt.testing:6443/console
Response:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/console\": no RBAC policy matched",
  "reason": "Forbidden",
  "details": {
    
  },
  "code": 403
}
$ bin/openshift-install version
bin/openshift-install v0.1.0-65-gc0331204949479b41010ff7553297075a777bc2a
Terraform v0.11.8
$ oc get pods --all-namespaces                                                                                                                                                                                                                                                [±master ●]
NAMESPACE                              NAME                                                      READY     STATUS                 RESTARTS   AGE
kube-system                            kube-apiserver-f2lzx                                      1/1       Running                0          33m
kube-system                            kube-controller-manager-b6985978d-z8tnw                   1/1       Running                0          33m
kube-system                            kube-core-operator-7f4d6b8dcf-6kzgl                       1/1       Running                0          26m
kube-system                            kube-dns-787c975867-p8dtz                                 3/3       Running                0          33m
kube-system                            kube-flannel-5rn9f                                        2/2       Running                0          27m
kube-system                            kube-flannel-fdv56                                        2/2       Running                0          27m
kube-system                            kube-flannel-z7vcb                                        2/2       Running                4          19m
kube-system                            kube-proxy-kbs8t                                          1/1       Running                0          33m
kube-system                            kube-proxy-klfqx                                          1/1       Running                0          19m
kube-system                            kube-proxy-lkwf8                                          1/1       Running                0          33m
kube-system                            kube-scheduler-78d86f9754-gdkdd                           1/1       Running                0          33m
kube-system                            metrics-server-5767bfc576-tzp4p                           2/2       Running                0          24m
kube-system                            pod-checkpointer-5847n                                    1/1       Running                0          33m
kube-system                            pod-checkpointer-5847n-test1-master-0                     1/1       Running                0          32m
kube-system                            tectonic-network-operator-qbgn7                           1/1       Running                0          33m
openshift-cluster-api                  clusterapi-apiserver-6b855f7bc5-bxqmp                     2/2       Running                0          26m
openshift-cluster-api                  clusterapi-controllers-85f6bfd9d5-96cd6                   2/2       Running                0          25m
openshift-cluster-api                  machine-api-operator-5d85454676-8tgmf                     1/1       Running                0          33m
openshift-cluster-version              bootstrap-cluster-version-operator-test1-bootstrap        1/1       Running                1          33m
openshift-cluster-version              cluster-version-operator-8bb6cff75-nkwrl                  1/1       Running                0          33m
openshift-core-operators               openshift-service-cert-signer-operator-6dbfb84555-wsm54   1/1       Running                0          6m
openshift-ingress                      default-http-backend-6985d557bb-r5w29                     1/1       Running                0          25m
openshift-ingress                      router-67b6b747c5-jp7vk                                   1/1       Running                0          25m
openshift-ingress                      tectonic-ingress-controller-operator-fcb9c6f4b-jtdcn      1/1       Running                0          26m
openshift-machine-config-operator      machine-config-controller-5b74549d47-w6q65                1/1       Running                0          26m
openshift-machine-config-operator      machine-config-daemon-jfjrz                               1/1       Running                3          19m
openshift-machine-config-operator      machine-config-daemon-mfjvb                               1/1       Running                0          24m
openshift-machine-config-operator      machine-config-operator-68fcdcb4f9-7bxqd                  1/1       Running                0          33m
openshift-machine-config-operator      machine-config-server-8rprn                               1/1       Running                0          25m
openshift-operator-lifecycle-manager   catalog-operator-7db987f97b-xdrcj                         0/1       CreateContainerError   0          33m
openshift-operator-lifecycle-manager   olm-operator-748f55cdd-m254c                              0/1       CreateContainerError   0          33m
openshift-operator-lifecycle-manager   olm-operator-867dbbb8f9-sj9dk                             0/1       CreateContainerError   0          5m
openshift-operator-lifecycle-manager   package-server-7fb4c799bc-v4h4c                           0/1       CreateContainerError   0          33m
openshift-service-cert-signer          apiservice-cabundle-injector-7bc57bccf4-hrz9z             1/1       Running                0          5m
openshift-service-cert-signer          configmap-cabundle-injector-5889f997bc-4cxmn              1/1       Running                0          5m
openshift-service-cert-signer          service-serving-cert-signer-5d867cb55d-44gxm              1/1       Running                0          5m
tectonic-system                        kube-addon-operator-784b4b6c7-d8sjh                       1/1       Running                0          26m

Was the web-console replaced by something?

@imranrazakhan
Copy link

I was getting same json issue and just run below command

oc login -u system:admin

Now i am able to access it

@praveenkumar
Copy link
Contributor

Same error I am getting today.

# kubectl get --all-namespaces pods
NAMESPACE                              NAME                                                              READY     STATUS             RESTARTS   AGE
default                                registry-6fcb8b7789-4x72w                                         1/1       Running            0          20m
kube-system                            kube-apiserver-krqjc                                              1/1       Running            0          41m
kube-system                            kube-controller-manager-b6985978d-zxhm2                           1/1       Running            0          41m
kube-system                            kube-core-operator-7f4d6b8dcf-hdfgw                               1/1       Running            0          32m
kube-system                            kube-dns-787c975867-kz8np                                         3/3       Running            0          41m
kube-system                            kube-flannel-h2kqm                                                2/2       Running            5          29m
kube-system                            kube-flannel-lt5dd                                                2/2       Running            0          36m
kube-system                            kube-flannel-t4l49                                                2/2       Running            0          36m
kube-system                            kube-proxy-5xz66                                                  1/1       Running            0          29m
kube-system                            kube-proxy-kvgxr                                                  1/1       Running            0          41m
kube-system                            kube-proxy-nbqvd                                                  1/1       Running            0          41m
kube-system                            kube-scheduler-78d86f9754-j4pr8                                   1/1       Running            0          41m
kube-system                            metrics-server-5767bfc576-67znk                                   2/2       Running            0          30m
kube-system                            pod-checkpointer-7v2xn                                            1/1       Running            0          41m
kube-system                            pod-checkpointer-7v2xn-test1-master-0                             1/1       Running            0          39m
kube-system                            tectonic-network-operator-76qt7                                   1/1       Running            0          41m
openshift-apiserver                    apiserver-kgcgc                                                   1/1       Running            0          31m
openshift-cluster-api                  clusterapi-apiserver-6b855f7bc5-s2flm                             2/2       Running            0          33m
openshift-cluster-api                  clusterapi-controllers-85f6bfd9d5-vrw2s                           2/2       Running            0          31m
openshift-cluster-api                  machine-api-operator-5d85454676-m9z9d                             1/1       Running            0          39m
openshift-cluster-version              cluster-version-operator-fqkl6                                    1/1       Running            0          41m
openshift-controller-manager           controller-manager-w9kdb                                          1/1       Running            0          30m
openshift-core-operators               openshift-cluster-openshift-apiserver-operator-5fbd49d8f7-vlthq   1/1       Running            0          39m
openshift-core-operators               openshift-cluster-openshift-controller-manager-operator-7cw26h6   1/1       Running            0          39m
openshift-core-operators               openshift-service-cert-signer-operator-6d6c6f55db-jspwh           1/1       Running            0          39m
openshift-image-registry               cluster-image-registry-operator-58c7c9bfd6-r7l92                  1/1       Running            0          29m
openshift-ingress                      tectonic-ingress-controller-operator-fcb9c6f4b-tlhzq              0/1       CrashLoopBackOff   9          31m
openshift-machine-config-operator      machine-config-controller-6948b45dd9-n8cxd                        1/1       Running            0          32m
openshift-machine-config-operator      machine-config-daemon-drsmd                                       1/1       Running            4          29m
openshift-machine-config-operator      machine-config-daemon-fsxft                                       1/1       Running            0          31m
openshift-machine-config-operator      machine-config-operator-545fcb447d-h9pfj                          1/1       Running            0          39m
openshift-machine-config-operator      machine-config-server-5dwbh                                       1/1       Running            0          31m
openshift-monitoring                   cluster-monitoring-operator-c64f5b475-tlplc                       1/1       Running            0          29m
openshift-monitoring                   prometheus-operator-5bf8644c75-xmrcm                              1/1       Running            0          20m
openshift-operator-lifecycle-manager   catalog-operator-5d5d8c7689-wf2ql                                 1/1       Running            0          39m
openshift-operator-lifecycle-manager   olm-operator-76b7f57649-8zj5v                                     1/1       Running            0          39m
openshift-operator-lifecycle-manager   package-server-f994b8699-f7hsc                                    0/1       CrashLoopBackOff   8          39m
openshift-service-cert-signer          apiservice-cabundle-injector-d4c746869-z66ks                      1/1       Running            0          32m
openshift-service-cert-signer          configmap-cabundle-injector-77bd46b-bqnd8                         1/1       Running            0          32m
openshift-service-cert-signer          service-serving-cert-signer-55fb7cc589-l6rzs                      1/1       Running            0          32m
openshift-web-console                  webconsole-86f4f55644-cl9r5                                       1/1       Running            0          20m
tectonic-system                        kube-addon-operator-784b4b6c7-fdkqj                               1/1       Running            0          32m

== web console output

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/console/\": no RBAC policy matched",
  "reason": "Forbidden",
  "details": {
    
  },
  "code": 403
}

@imranrazakhan where did you run that oc command to login?

@imranrazakhan
Copy link

@praveenkumar i just run this command on shell of VM where i installed 3.11

[root@c1-ocp ~]# oc login -u system:admin

@spadgett
Copy link
Member

This is the old console that is being removed. The new console is behind a route. You're seeing this because the master console proxy is removed (since it's no longer needed).

I believe @deads2k was going to add a redirect to the new console from the old URL, but I'm not sure if that's in place.

@spadgett
Copy link
Member

We think this is openshift/origin#20983.

It's a different cause.

@wking
Copy link
Member

wking commented Oct 22, 2018

I believe @deads2k was going to add a redirect to the new console from the old URL, but I'm not sure if that's in place.

Do you know the new URL? A redirect would be nice, but we may be able to help a bunch of people by updating our docs.

@spadgett
Copy link
Member

spadgett commented Oct 22, 2018

For new 4.0 installs, it is https://console-openshift-console.<default-routing-suffix>

cc @benjaminapetersen

@stlaz
Copy link

stlaz commented Oct 25, 2018

@spadgett I'm not entirely sure what you mean by default-routing-suffix. There does not seem to be anything at the following location:

# curl https://console.openshift-console.${OPENSHIFT_INSTALL_BASE_DOMAIN}
curl: (6) Could not resolve host: console.openshift-console.tt.testing

edit: this is calling it from the KVM host, not KVM guest

@spadgett
Copy link
Member

@stlaz Look at the route in the openshift-console namespace. You can see the hostname there. By default, console uses a generated hostname.

@stlaz
Copy link

stlaz commented Oct 25, 2018

Thanks. Here's what I did:

# oc get route -n openshift-console
NAME      HOST/PORT                                                    PATH      SERVICES   PORT      TERMINATION          WILDCARD
console   console-openshift-console.router.default.svc.cluster.local             console    https     reencrypt/Redirect   None

Curling this just gives me Could not resolve hots:. I wonder which service is used to add this to the KVM host machine DNS? I am using the dnsmasq with NetworkManager solution from the guide. For the record, the following works just fine with it:

# host $OPENSHIFT_INSTALL_CLUSTER_NAME-bootstrap.$OPENSHIFT_INSTALL_BASE_DOMAIN
test1-bootstrap.tt.testing has address 192.168.126.10

@spadgett
Copy link
Member

Yeah, we depend on the default route working. I don't think it does out of the box yet.

You could add the console hostname to the /etc/hosts or update the Console CR to another hostname. (@benjaminapetersen, does the operator currently reconcile console hostname?)

@benjaminapetersen
Copy link

The console-operator does not yet reconcile a custom hostname from its cr.yaml. Still some discussion about how that should actually work.

@wking
Copy link
Member

wking commented Nov 14, 2018

Routing workaround for AWS from @ironcladlou: https://gist.github.com/ironcladlou/784e4bd5cdd7e270ae0bea444809cbfd If anyone goes through something similar for libvirt, please post your notes here.

@jianzhangbjz
Copy link

Anyone knows how to enable console for the libvirt?

@benjaminapetersen
Copy link

benjaminapetersen commented Nov 20, 2018

See this comment: https://gist.github.com/ironcladlou/784e4bd5cdd7e270ae0bea444809cbfd#gistcomment-2764321

ATM, libvert seems to be the trickiest option. There are workarounds for AWS & GCE around.

@wking wking changed the title libvirt console 403: no RBAC policy matched for system:anonymous console 403: no RBAC policy matched for system:anonymous Nov 20, 2018
@sallyom
Copy link
Contributor

sallyom commented Dec 5, 2018

There is now a 'special user' who goes by the name kubeadmin configured. (12/5/18 this was enabled)
To access the web-console (aws):

  • install with latest installer
  • after install completes export kubeconfig, wait a few minutes (or keep obsessively trying to oc login -u kubeadmin -p 5char-5char-5char-5char until it works
  • the password is logged at end of install, also it is written to ${INSTALL_DIR}/auth/kubeadmin-password
  • grab the route from -n openshift-console, you might get an error at first when you try to access it, https://console-openshift-console.apps.clustername.devcluster.openshift.com, if you do oc delete pod/openshift-console-blahblah -n openshift-console then wait a minute and try again
  • enter username kubeadmin and password 5char-5char-5char-5char to login to console

(you can now also view prometheus/grafana console)

@spadgett
Copy link
Member

spadgett commented Dec 6, 2018

grab the route from -n openshift-console, you might get an error at first when you try to access it

This problem is fixed, so you shouldn't need to delete the pod anymore. Console should just work now on AWS. (We just need to log the URL, #782)

@praveenkumar
Copy link
Contributor

@sallyom So #411 (comment) will only for AWS side, from libvirt there is still no way to get the console?

@RobertKrawitz
Copy link
Contributor

@spadgett confirmed don't need the delete pod hack today (although even after logging in to the console, going to prometheus and using Login with OpenShift gives me a login screen).

@s-urbaniak
Copy link
Contributor

s-urbaniak commented Dec 7, 2018

To access the console from libvirt, execute the following steps:

Allow kubectl to bind to privileged ports:

$ sudo setcap CAP_NET_BIND_SERVICE=+eip /usr/bin/kubectl

Note: If you omit the above, you have to start kubectl using sudo. Next, port-forward the router-default service:

$ kubectl -n openshift-ingress port-forward svc/router-internal-default 443

Get the routes and add bind them to 127.0.0.1 in /etc/hosts:

$ kubectl get routes --all-namespaces 
NAMESPACE              NAME             HOST/PORT                                                   PATH   SERVICES         PORT    TERMINATION          WILDCARD
openshift-console      console          console-openshift-console.apps.test1.tt.testing                    console          https   reencrypt/Redirect   None
openshift-monitoring   grafana          grafana-openshift-monitoring.apps.test1.tt.testing                 grafana          https   reencrypt            None
openshift-monitoring   prometheus-k8s   prometheus-k8s-openshift-monitoring.apps.test1.tt.testing          prometheus-k8s   web     reencrypt            None

$ cat /etc/hosts
127.0.0.1 console-openshift-console.apps.test1.tt.testing           
127.0.0.1 grafana-openshift-monitoring.apps.test1.tt.testing        
127.0.0.1 prometheus-k8s-openshift-monitoring.apps.test1.tt.testing 

Use the credentials as described in #411 (comment)

Open i.e. https://grafana-openshift-monitoring.apps.test1.tt.testing or https://console-openshift-console.apps.test1.tt.testing.

@praveenkumar
Copy link
Contributor

I just tried @s-urbaniak suggestion and it works as said. Thanks for this quick workaround 👍

@ghost
Copy link

ghost commented Dec 11, 2018

To access the console from libvirt, execute the following steps:

Allow kubectl to bind to privileged ports:

$ sudo setcap CAP_NET_BIND_SERVICE=+eip /usr/bin/kubectl

Note: If you omit the above, you have to start kubectl using sudo. Next, port-forward the router-default service:

$  kubectl -n openshift-ingress port-forward svc/router-default 443

Get the routes and add bind them to 127.0.0.1 in /etc/hosts:

$ kubectl get routes --all-namespaces 
NAMESPACE              NAME             HOST/PORT                                                   PATH   SERVICES         PORT    TERMINATION          WILDCARD
openshift-console      console          console-openshift-console.apps.test1.tt.testing                    console          https   reencrypt/Redirect   None
openshift-monitoring   grafana          grafana-openshift-monitoring.apps.test1.tt.testing                 grafana          https   reencrypt            None
openshift-monitoring   prometheus-k8s   prometheus-k8s-openshift-monitoring.apps.test1.tt.testing          prometheus-k8s   web     reencrypt            None

$ cat /etc/hosts
127.0.0.1 console-openshift-console.apps.test1.tt.testing           
127.0.0.1 grafana-openshift-monitoring.apps.test1.tt.testing        
127.0.0.1 prometheus-k8s-openshift-monitoring.apps.test1.tt.testing 

Use the credentials as described in #411 (comment)

Open i.e. https://grafana-openshift-monitoring.apps.test1.tt.testing or https://console-openshift-console.apps.test1.tt.testing.

I have to mention that if the libvirt machine have no additional memory or no X to open a browser, then we have to use proxy to make the browser and terminal work on another machine with something like https://gist.github.com/corehello/6f50472d9b4f9a5624c49312c17d3e99
And setup necessary proxy env for terminal and also browser:

✗ cat proxy.sh 
export HTTPS_PROXY=http://10.66.140.70:8888/
export ALL_PROXY=socks://10.66.140.70:8888/
export ftp_proxy=http://10.66.140.70:8888/
export FTP_PROXY=http://10.66.140.70:8888/
export no_proxy=localhost,127.0.0.0/8,::1
export https_proxy=http://10.66.140.70:8888/
export HTTP_PROXY=http://10.66.140.70:8888/
export all_proxy=socks://10.66.140.70:8888/
export http_proxy=http://10.66.140.70:8888/
export NO_PROXY=localhost,127.0.0.0/8,::1

@spadgett
Copy link
Member

spadgett commented Jan 8, 2019

Since this is working out of the box now for AWS and the incorrect instructions were removed from the README, closing the issue.

/close

@openshift-ci-robot
Copy link
Contributor

@spadgett: Closing this issue.

In response to this:

Since this is working out of the box now for AWS and the incorrect instructions were removed from the README, closing the issue.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@praveenkumar
Copy link
Contributor

@s-urbaniak Since most of the folks are using you suggested way #411 (comment) to get the console and recently svc name is changed from router-default to router-internal-default , can you update that this comment with?

 $  kubectl -n openshift-ingress port-forward svc/router-internal-default 443

@ironcladlou
Copy link
Contributor

Just a quick note, and I haven't had time to dig into the background of the use case, but it's unsafe to rely on resource names like router-internal-default. We make no guarantees about that service name, or even the existence of the service — it should be considered an implementation detail. Of course when it comes to libvirt right now a working hack is probably going to proliferate anyway, but I noticed this and wanted to drop a disclaimer 😁

@nileshalhat
Copy link

@praveenkumar i just run this command on shell of VM where i installed 3.11

[root@c1-ocp ~]# oc login -u system:admin

DONE..! Thank You,,!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests