Multi docker stack

[Dieterbe]

refactor into several docker-compose stacks

variants:
* simple standalone with CI built docker image and binary, monitors into itself
* similar, but mounts custom built binary, monitors into itself
* clustered stack with kafka input and graphite for monitoring

also:
- remove unneeded links
- re-arrange a bit
- export cassandra port so you can query it from the host if you want
- simplify hostnames (strip -1 everywhere)

Notably missing from this work, but available in
https://github.com/raintank/raintank-docker is tooling to automatically
run given workload scripts. We can add that later, it's not a priority now .

[Dieterbe]

note i have run the docker-standard stack and it seems to work fine, just like the previous docker stack

[woodsaj]

this image does not exist. We were building it as part of raintank-docker.
Perhaps we should use https://hub.docker.com/r/raintank/graphite-stack/ for both graphite statsd and grafana.

[Dieterbe]

pushed a commit that uses graphite-stack. the problem is however that it only speaks https.

i would like the stacks to be as consistent/similar as possible, and only differ where needed. e.g. ideally the populate.sh script should work for all stacks, meaning the same datasource config should work for all stacks. but in the other docker stacks we use graphite-api+metrictank for monitoring which only accepts plain http (https://github.com/raintank/graphite-metrictank/blob/master/pkg/Dockerfile)
for docker-cluster i can route port 8000 to 443 but the requests will fail since it will refuse plain http requests.

[woodsaj]

I dont understand what you are trying to say.

There are 3 variants that can be deployed. "cluster", "dev" and "standard".

In dev and standard, statsdaemon published directly to metrictank. In the environemnt there is only a single Graphite service, graphite-api which uses metrictank as the backend and listens on 8080.

In "cluster", statsdaemon publishes to graphitemon (graphite-stack). In the environment there are two graphite services. graphitemon which listens on 443 and graphite-api which listens on 8080.

At present, you are adding 2 datasources to grafana, 1 pointed to port 8080 (graphite-api) and 1 pointed to 8000, which doesnt exist.

So why cant you just change the datasource using 8000 use port 443 and https?

[Dieterbe]

So why cant you just change the datasource using 8000 use port 443 and https?

then the script would create a DS that is broken for the dev and standard stacks.
I was trying to see if we could have the same populate.sh script for all stacks, which means the same DS configs, and we would use the different docker-compose.ymls to provide different implementations for the graphite datasource. This way you could always use the "monitoring" datasource for monitoring of the stack, and the "metrictank" datasource, to query metrictank. Even if they're powered by the same thing or not. You would be able to quickly spin up a (different) stack and use it without having to know some implementation details.
The only thing needed to make this work is if graphite-stack would serve graphite over http, which it doesn't.

But anyway it's not worth the time trying to go that route, I'll just set up different datasources based on which environment you're in.

[Dieterbe]

@woodsaj how does it look now?

note:

• the monitoring in cluster stack using graphite-stack doesn't work because the SSL certificate doesn't verify
• the extra/populate.sh script is not documented yet. the docker guide still asks people to set things up manually which I think is a good exercise. also i'm not certain yet I like the workflow and how it takes an argument. perhaps it should be run from inside the specific docker stack dir