Add alert on error creating a POD

[2opremio]

As a continuation of #280

We have already experienced two issues creating pods in kubernetes of which we should be alerted automatically:

• Kubernetes cannot create pods due to lack of CPU resources in the cluster (which caused Incident Incident: Kubernetes couldn't schedule pods #287 in production)
• Kubernetes cannot create pods due to lack of available IPs (experienced in dev and reported upstream Mitigate impact of docker 1.8 ErrNoAvailableIPs bug kubernetes/kubernetes#19477 )

[2opremio]

Copying a Slack conversation on this with @peterbourgon for further reference: https://weaveworks.slack.com/archives/service/p1452792646004629

​[5:30] 
how would you go about generating an alert when kubernetes fails to create new pods?

​[5:31] 
would you know how to do this with prometheus?

​[5:32] 
it already talks to the API server, so I guess it should also be receiving error events like the “no more available ips” or “I am not going to run any more pods because the CPU consumption is too high"

peter [5:41 PM] 
I guess we'd need to find out if Kubernetes exports a metric that maps to that condition

​[5:41] 
If so, Prometheus is already scraping it...

​[5:43] 
kubelet_docker_errors is a fun one

fons [5:43 PM] 
​_Exports a metric_​ <- how can we check?(edited)

peter [5:43 PM] 
well, reading the source, or seeing what timeseries pop up in the Prometheus graph dropdown:
http://monitoring.default.svc.cluster.local:9090/graph

​[5:44] 
`rate(kubelet_docker_errors[1m])` might be a good general candidate

​[5:45] 
that reads as: how many kubelet_docker_errors per second (1 minute avg) do we see?

fons [5:45 PM] 
OK, I will check

new messages
peter [5:47 PM] 
`sort_desc(sum(kubelet_docker_errors) by (operation_type))` — also a fun one

​[5:47] 
this will definitely capture it

[errordeveloper]

Is to do with spawning app pods?

[tomwilkie]

We don't need this anymore (with m/t)