Scope 1.3.0 thinks Kubernetes 1.6 doesn't support Deployments or ReplicaSets

[omkensey]

When I start the agent DaemonSet and look at the logs for a pod, near the top I see this:

<probe> INFO: 2017/05/03 18:18:31.578773 probe starting, version 1.3.0, ID f6655448c0d274c
[...]
<probe> INFO: 2017/05/03 18:18:31.909595 Deployments and ReplicaSets are not supported by this Kubernetes version

My Kube version is:

$ kubectl get nodes
NAME            STATUS    AGE       VERSION
10.132.38.193   Ready     1d        v1.6.2+coreos.0
10.132.44.58    Ready     1d        v1.6.2+coreos.0
10.132.73.90    Ready     1d        v1.6.2+coreos.0

[2opremio]

This smells like the vendored k8s client needs updating to work properly with k8s 1.6 .

@omkensey Does Scope properly show other k8s views (e.g. Pods)?

[omkensey]

Yep, I can see Pods and Services and all my nodes show up in Hosts.

[errordeveloper]

Also note that there had been an API group change, prior to 1.6 Deployments and ReplicaSets where in extensions group, and in 1.6 they got moved to apps. This is probably related, but may be not so much.

[2opremio]

I cannot reproduce, I have just launched weaveworks/scope:latest against a local minikube 1.6.2 cluster and the deployments show up in the UI just fine (I also don't get errors in the logs).

[2opremio]

I have merged #2501 to debug the problem further.

@omkensey Would you mind testing again with weaveworks/scope:latest and let us know what error you get?

[omkensey]

<probe> INFO: 2017/05/05 16:15:13.497847 Deployments and ReplicaSets are not supported by this Kubernetes version: the server does not allow access to the requested resource (get deployments.extensions)

So I think @errordeveloper had the right idea.

[2opremio]

@omkensey How did you exactly deployed scope?

[omkensey]

Using the scope.yaml served from https://cloud.weave.works/k8s/scope.yaml -- which then redirects to https://cloud.weave.works/k8s/v1.5/scope.yaml.

[2opremio]

That's probably the issue. We now have version-specific resources for kubernetes. We need to update the installation instructions. Can you give this URL a try?

"https://cloud.weave.works/k8s.yaml?version=$(kubectl version | base64 | tr -d '\n')"

[omkensey]

I still get a redirect to a 1.5 path:

$ curl "https://cloud.weave.works/k8s.yaml?version=$(kubectl version | base64 | tr -d '\n')"
code: Redirect
message: >-
  Moved permanently to `/k8s/v1.5/weave-cloud.yaml`, assuming Kubernetes version
  is "v1.5" or older, old location (`/k8s.yaml`) will be deprecated soon. If you
  are using `curl`, you can pass `--location`, but it is best to just use the
  new URL.

Also that seems like a really brittle method of determining the version to use. Won't it get confused by changes in any of the version strings for the client or the server?

[rade]

it should be k8s-version=, not version=.

[omkensey]

That gets an error:

$ curl "https://cloud.weave.works/k8s.yaml?k8s-version=$(kubectl version | base64 | tr -d '\n')"
jse_shortmsg: >-
  Invalid kubectl version output: "Client Version: version.Info{Major:"1",
  Minor:"6", GitVersion:"v1.6.2",
  GitCommit:"477efc3cbe6a7effca06bd1452fa356e2201e1ee", GitTreeState:"clean",
  BuildDate:"2017-04-19T20:33:11Z", GoVersion:"go1.7.5", Compiler:"gc",
  Platform:"darwin/amd64"}

  Server Version: version.Info{Major:"1", Minor:"6",
  GitVersion:"v1.6.2+coreos.0",
  GitCommit:"79fee581ce4a35b7791fdd92e0fc97e02ef1d5c0", GitTreeState:"clean",
  BuildDate:"2017-04-19T23:13:34Z", GoVersion:"go1.7.5", Compiler:"gc",
  Platform:"linux/amd64"}

  ".
jse_info: {}
message: >-
  Invalid kubectl version output: "Client Version: version.Info{Major:"1",
  Minor:"6", GitVersion:"v1.6.2",
  GitCommit:"477efc3cbe6a7effca06bd1452fa356e2201e1ee", GitTreeState:"clean",
  BuildDate:"2017-04-19T20:33:11Z", GoVersion:"go1.7.5", Compiler:"gc",
  Platform:"darwin/amd64"}

  Server Version: version.Info{Major:"1", Minor:"6",
  GitVersion:"v1.6.2+coreos.0",
  GitCommit:"79fee581ce4a35b7791fdd92e0fc97e02ef1d5c0", GitTreeState:"clean",
  BuildDate:"2017-04-19T23:13:34Z", GoVersion:"go1.7.5", Compiler:"gc",
  Platform:"linux/amd64"}

  ".
statusCode: 400
body:
  code: BadRequestError
  message: >-
    Invalid Kubernetes version:
    "Q2xpZW50IFZlcnNpb246IHZlcnNpb24uSW5mb3tNYWpvcjoiMSIsIE1pbm9yOiI2IiwgR2l0VmVyc2lvbjoidjEuNi4yIiwgR2l0Q29tbWl0OiI0NzdlZmMzY2JlNmE3ZWZmY2EwNmJkMTQ1MmZhMzU2ZTIyMDFlMWVlIiwgR2l0VHJlZVN0YXRlOiJjbGVhbiIsIEJ1aWxkRGF0ZToiMjAxNy0wNC0xOVQyMDozMzoxMVoiLCBHb1ZlcnNpb246ImdvMS43LjUiLCBDb21waWxlcjoiZ2MiLCBQbGF0Zm9ybToiZGFyd2luL2FtZDY0In0KU2VydmVyIFZlcnNpb246IHZlcnNpb24uSW5mb3tNYWpvcjoiMSIsIE1pbm9yOiI2IiwgR2l0VmVyc2lvbjoidjEuNi4yK2NvcmVvcy4wIiwgR2l0Q29tbWl0OiI3OWZlZTU4MWNlNGEzNWI3NzkxZmRkOTJlMGZjOTdlMDJlZjFkNWMwIiwgR2l0VHJlZVN0YXRlOiJjbGVhbiIsIEJ1aWxkRGF0ZToiMjAxNy0wNC0xOVQyMzoxMzozNFoiLCBHb1ZlcnNpb246ImdvMS43LjUiLCBDb21waWxlcjoiZ2MiLCBQbGF0Zm9ybToibGludXgvYW1kNjQifQo=".
  rootCause: >-
    Invalid kubectl version output: "Client Version: version.Info{Major:"1",
    Minor:"6", GitVersion:"v1.6.2",
    GitCommit:"477efc3cbe6a7effca06bd1452fa356e2201e1ee", GitTreeState:"clean",
    BuildDate:"2017-04-19T20:33:11Z", GoVersion:"go1.7.5", Compiler:"gc",
    Platform:"darwin/amd64"}

    Server Version: version.Info{Major:"1", Minor:"6",
    GitVersion:"v1.6.2+coreos.0",
    GitCommit:"79fee581ce4a35b7791fdd92e0fc97e02ef1d5c0", GitTreeState:"clean",
    BuildDate:"2017-04-19T23:13:34Z", GoVersion:"go1.7.5", Compiler:"gc",
    Platform:"linux/amd64"}

    ".

[rade]

Looks like the esoteric "v1.6.2+coreos.0" broke things :( we shall have to fix that. cc @marccarre

[rade]

you can of course simply fetch the v1.6 specific yaml from https://cloud.weave.works/k8s/v1.6/scope.yaml

[omkensey]

That does successfully display the Deployments and ReplicaSets. However, I did have to fix one thing -- the weave-scope ServiceAccount is in the default namespace, but the ClusterRoleBinding tries to bind it in the kube-system namespace. Once I changed that from "kube-system" to "default", it all worked.

[rade]

The yaml is meant to be applied with kubectl apply -n kube-system -f ...

[2opremio]

it should be k8s-version=, not version=.

Uhm, I thought copied this from the Weave Cloud instructions. They show k8s-version correctly today.