stream deploy : Message: Forbidden! User admin doesn't have permission

[tajdars]

Version
OpenShift Master:
v1.4.1+3f9807a
Kubernetes Master:
v1.4.0+776c994
spring-cloud-dataflow-server-openshift:
1.1.0.RELEASE

Followed :
https://blog.switchbit.io/spring-cloud-deployer-openshift/

Get the following Exception when deploying http-log stream :

o.s.c.d.s.c.StreamDeploymentController : Exception when deploying the app StreamAppDefinition [streamName=http-log,......
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://host:8443/api/v1/namespaces/default/pods?labelSelector=spring-app-id%3Dhttp-log-log. Message: Forbidden! User admin/host:8443 doesn't have permission..

[donovanmuller]

@tajdars Thanks for reporting this.

Perhaps this issue #11 might be similar?

To be honest I haven't had a chance to test the deployer server on OpenShift 1.4/3.4 yet.
I'll take a look when I get a chance and let you know.

[donovanmuller]

@tajdars I've spun up an OpenShift Origin 1.4.1 instance (with minishift) and deployed the Data Flow server for OpenShift (1.1.0.RELEASE). Deployed a simple stream and all worked as expected.

Since the 1.1.0 release, the preferred way to deploy the OpenShift server is using the provided Templates (the post was written quite a while before the 1.1.0 release and so is most probably out of date with regards to deploying into an OpenShift environment - have added note).

Can you please follow the "Getting Started" section in the reference documentation and see if that works for you?

[tajdars]

Hi @donovanmuller ,

Didn't try minishift yet.
This is the exception I get trying out the templates route (deploying scdf server + rabbitmq + MySQL + redis).
Looks like its unable to resolve kubernetes.default.svc.
Will continue looking.

2017-02-21 19:01:39.722 WARN 1 --- [ main] i.f.s.cloud.kubernetes.StandardPodUtils : Failed to get pod with name:[scdf-rabbitmq-1-z27n7]. You should look into this if things aren't working as you expect. Are you missing serviceaccount permissions?

io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:57) ~[kubernetes-client-1.4.26.jar!/:na]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:173) ~[kubernetes-client-1.4.26.jar!/:na]
at io.fabric8.spring.cloud.kubernetes.StandardPodUtils.internalGetPod(StandardPodUtils.java:56) [spring-cloud-kubernetes-core-0.1.3.jar!/:na]
at io.fabric8.spring.cloud.kubernetes.StandardPodUtils.lambda$new$0(StandardPodUtils.java:40) [spring-cloud-kubernetes-core-0.1.3.jar!/:na]
at io.fabric8.spring.cloud.kubernetes.LazilyInstantiate.swapper(LazilyInstantiate.java:41) ~[spring-cloud-kubernetes-core-0.1.3.jar!/:na]
at io.fabric8.spring.cloud.kubernetes.LazilyInstantiate.lambda$new$0(LazilyInstantiate.java:34) ~[spring-cloud-kubernetes-core-0.1.3.jar!/:na]
at io.fabric8.spring.cloud.kubernetes.LazilyInstantiate.get(LazilyInstantiate.java:29) ~[spring-cloud-kubernetes-core-0.1.3.jar!/:na]
at io.fabric8.spring.cloud.kubernetes.profile.KubernetesProfileApplicationListener.addKubernetesProfile(KubernetesProfileApplicationListener.java:49) ~[spring-cloud-kubernetes-core-0.1.3.jar!/:na]
at io.fabric8.spring.cloud.kubernetes.profile.KubernetesApplicationContextInitializer.initialize(KubernetesApplicationContextInitializer.java:53) ~[spring-cloud-kubernetes-core-0.1.3.jar!/:na]
at org.springframework.boot.SpringApplication.applyInitializers(SpringApplication.java:635) ~[spring-boot-1.4.1.RELEASE.jar!/:1.4.1.RELEASE]
at org.springframework.boot.SpringApplication.prepareContext(SpringApplication.java:349) ~[spring-boot-1.4.1.RELEASE.jar!/:1.4.1.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:313) ~[spring-boot-1.4.1.RELEASE.jar!/:1.4.1.RELEASE]
at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:134) ~[spring-boot-1.4.1.RELEASE.jar!/:1.4.1.RELEASE]
at org.springframework.cloud.bootstrap.BootstrapApplicationListener.bootstrapServiceContext(BootstrapApplicationListener.java:138) ~[spring-cloud-context-1.1.5.RELEASE.jar!/:1.1.5.RELEASE]
at org.springframework.cloud.bootstrap.BootstrapApplicationListener.onApplicationEvent(BootstrapApplicationListener.java:84) ~[spring-cloud-context-1.1.5.RELEASE.jar!/:1.1.5.RELEASE]
at org.springframework.cloud.bootstrap.BootstrapApplicationListener.onApplicationEvent(BootstrapApplicationListener.java:62) ~[spring-cloud-context-1.1.5.RELEASE.jar!/:1.1.5.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:166) ~[spring-context-4.3.3.RELEASE.jar!/:4.3.3.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:138) ~[spring-context-4.3.3.RELEASE.jar!/:4.3.3.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:121) ~[spring-context-4.3.3.RELEASE.jar!/:4.3.3.RELEASE]
at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:68) ~[spring-boot-1.4.1.RELEASE.jar!/:1.4.1.RELEASE]
at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:54) ~[spring-boot-1.4.1.RELEASE.jar!/:1.4.1.RELEASE]
at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:337) ~[spring-boot-1.4.1.RELEASE.jar!/:1.4.1.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:308) ~[spring-boot-1.4.1.RELEASE.jar!/:1.4.1.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1186) ~[spring-boot-1.4.1.RELEASE.jar!/:1.4.1.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1175) ~[spring-boot-1.4.1.RELEASE.jar!/:1.4.1.RELEASE]
at org.springframework.cloud.dataflow.server.openshift.OpenShiftDataFlowServer.main(OpenShiftDataFlowServer.java:15) ~[classes!/:1.1.0.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_111-internal]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_111-internal]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111-internal]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111-internal]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) ~[spring-cloud-dataflow-server-openshift.jar:1.1.0.RELEASE]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) ~[spring-cloud-dataflow-server-openshift.jar:1.1.0.RELEASE]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) ~[spring-cloud-dataflow-server-openshift.jar:1.1.0.RELEASE]
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:58) ~[spring-cloud-dataflow-server-openshift.jar:1.1.0.RELEASE]
Caused by: java.net.UnknownHostException: kubernetes.default.svc: Name does not resolve
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[na:1.8.0_111-internal]
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928) ~[na:1.8.0_111-internal]
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323) ~[na:1.8.0_111-internal]
at java.net.InetAddress.getAllByName0(InetAddress.java:1276) ~[na:1.8.0_111-internal]
at java.net.InetAddress.getAllByName(InetAddress.java:1192) ~[na:1.8.0_111-internal]
at java.net.InetAddress.getAllByName(InetAddress.java:1126) ~[na:1.8.0_111-internal]
at okhttp3.Dns$1.lookup(Dns.java:39) ~[okhttp-3.4.1.jar!/:na]
at okhttp3.internal.connection.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:172) ~[okhttp-3.4.1.jar!/:na]
at okhttp3.internal.connection.RouteSelector.nextProxy(RouteSelector.java:138) ~[okhttp-3.4.1.jar!/:na]
at okhttp3.internal.connection.RouteSelector.next(RouteSelector.java:80) ~[okhttp-3.4.1.jar!/:na]
at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:178) ~[okhttp-3.4.1.jar!/:na]
at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:129) ~[okhttp-3.4.1.jar!/:na]
at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:98) ~[okhttp-3.4.1.jar!/:na]
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) ~[okhttp-3.4.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.4.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.4.1.jar!/:na]
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:109) ~[okhttp-3.4.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.4.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.4.1.jar!/:na]
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[okhttp-3.4.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.4.1.jar!/:na]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:124) ~[okhttp-3.4.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.4.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.4.1.jar!/:na]
at io.fabric8.kubernetes.client.utils.HttpClientUtils$3.intercept(HttpClientUtils.java:102) ~[kubernetes-client-1.4.26.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.4.1.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.4.1.jar!/:na]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:170) ~[okhttp-3.4.1.jar!/:na]
at okhttp3.RealCall.execute(RealCall.java:60) ~[okhttp-3.4.1.jar!/:na]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:235) ~[kubernetes-client-1.4.26.jar!/:na]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:230) ~[kubernetes-client-1.4.26.jar!/:na]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:226) ~[kubernetes-client-1.4.26.jar!/:na]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:630) ~[kubernetes-client-1.4.26.jar!/:na]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:166) ~[kubernetes-client-1.4.26.jar!/:na]
... 32 common frames omitted

[donovanmuller]

@tajdars Can you please have a look at #11 as this error was also referenced in that issue.

It seems related to the Service Account that the Data Flow Server requires.

[tajdars]

Hi @donovanmuller ,

Following 3 checks match was pointed out in the other thread #11

1.scdf ServiceAccount has RoleBinding(edit).

2, scdf ServiceAccount is being referenced by the deployment yaml for scdf-rabbitmq

3.There are no kubernetes related params for env: JAVA_OPTS

      env:
        -
          name: KUBERNETES_NAMESPACE
          valueFrom: {fieldRef: {apiVersion: v1, fieldPath: metadata.namespace}}
        -
          name: JAVA_OPTS
          value: '-Xms128m -Xmx384m'
        -.......

[donovanmuller]

@tajdars what OpenShift project (i.e. the project name) have you deployed the Data Flow server into? If you use the following command, from the reference docs, to grant the edit role to the scdf service account:

$ oc policy add-role-to-user edit system:serviceaccount:scdf:scdf

it assumes that the project name is also scdf. Perhaps your project name is different?

[tajdars]

@donovanmuller , yup I used scdf as the project name.
Perhaps an internal DNS entry is missing ?

[donovanmuller]

Hmm, ok, I see what you mean:

Caused by: java.net.UnknownHostException: kubernetes.default.svc: Name does not resolve...

The OpenShift deployer uses the fabric8/kubernetes-client to integrate with OpenShift, which is making the call to the OpenShift API's (via okhttp).

Given the above, could it be an OpenShift networking/configuration issue?
Not sure where else I can help you?

[tajdars]

@donovanmuller
This is working now. I made the following change to deploy the template :

Added the following param to scdf-rabbitmq yaml :

        - name: KUBERNETES_MASTER
          value: 'https://kubernetes.default.svc.cluster.local'

To deploy the spring tick tock (time | log)
I Added /management prefix to the /health and /info endpoints.

Thanx a lot for all your help.

[donovanmuller]

Glad to hear it 👍