Incident: launch-generator returning empty resources

[2opremio]

Launch generator doesn't work in production.

Reported by @rismusz on Slack at ~15:00

rimusz [3:35 PM]  
hey man, how are things?

[3:36]  
I'm trying to download scope manifest with `curl --silent --remote-name https://cloud.weave.works/launch/k8s/weavescope.yaml` and it returns the empty file, has that file been moved

Rimantas is using the exact same command in our instructions and I just confirmed that the output of curl is empty.

Also see #1002

[2opremio]

After Peter restarted the instances, it seemed it started working agin with redirections:

curl -s -L https://cloud.weave.works/launch/k8s/weavescope.yaml

[rade]

So the redirect points to https://cloud.weave.works/k8s/scope.yaml. Should we change docs to that (as well as adding the redirect handling)? Frankly, I am surprised that the launch generator is serving URLs on the /k8s path - that is claiming a lot of territory. Will it end up doing the same for /mesos and /swarm? @errordeveloper what is wrong with /launch/* or some other common prefix?

[rade]

Which other docs / guides / etc might be affected by this?

[rade]

After Peter restarted the instances, it seemed it started working agin with redirections:

I saw the redirects in the container logs before the restarts; I doubt the restart changed anything.

When was the redirect introduced, @errordeveloper?

[2opremio]

Note: I would had sworn that I tried following redirects first thing (with -L) and I still got an empty reapnse, but now I doubt.

[rade]

When was the redirect introduced

https://github.com/weaveworks/launch-generator/commit/d07b2a2557155177b50652b8d214df2a591cfa11 which was in turn fixed in
https://github.com/weaveworks/launch-generator/commit/179cfd57ba6c88a2f2b1df71c44133313072697c

[rade]

btw, the redirect isn't really a problem most of the time since our docs generally leave the fetching to kubectl, which does follow redirects, e.g.

$ kubectl apply -f 'https://cloud.weave.works/launch/k8s/weavescope.yaml'

[rade]

my search of our website (using google, since our search is currently broken - see weaveworks/website#418) has not revealed any more redirect-free curls to launch generator URLs, so I think the one in the scope instructions was the exception.

[2opremio]

weaveworks/scope#2010 is another situación in which the docs weren't updated properly when modifying the launch generator. Docs need to be tested every time we do so.

[jml]

What do we need to do to close this incident? Generally speaking, criteria are that all the agreed upon follow-up actions have either been done, or we have decided upon further reflection not to do them. Alternatively, it's reasonable to decide that the chance*impact of repeated incidents is so low so as not to require further work.

I'd check myself, but I can't find the follow-up actions for this (admittedly I skimmed very quickly so I might have missed something).

[jml]

We discussed this in the production meeting and decided that all actions are complete.