Docker deployment support #588

meetwudi · 2015-10-05T22:16:15Z

Add docker support (see #450).

Couple things to note:

I removed imagemin task in grunt since
- It is obsolete and cannot be downloaded anymore (at least on my side). It can block building so I removed it.
- It is useless, I mean, in re:dash
Example docker-compose.yaml is added.
Made service listen on 0.0.0.0 instead of only 127.0.0.1. So docker containers can accept incoming traffic then.

grunt-contrib-imagemin seems to be broken because several dependencies are quite obsolete and cannot be downloaded.

Signed-off-by: John Wu <webmaster@leapoahead.com>

arikfr · 2015-10-06T12:30:42Z

Dockerfile

@@ -0,0 +1,25 @@
+FROM debian:wheezy


Is there a reason to use Debian and not Ubuntu as the base image?

arikfr · 2015-10-06T12:42:14Z

Thanks, @tjwudi ! This is much appreciated (and long awaited).

I've left some comments.

Signed-off-by: John Wu <webmaster@leapoahead.com>

Instead of binding to `0.0.0.0`, use `127.0.0.1` instead for security concerns. "The Python Web server is more vulnerable than nginx that proxies it." Signed-off-by: John Wu <webmaster@leapoahead.com>

Signed-off-by: John Wu <webmaster@leapoahead.com>

meetwudi · 2015-10-06T20:12:39Z

Also, @arikfr is it a good practice to use db containers? Since files modified will be discarded when container stops, then all data will be lost if we do not commit the container regularly.

By using Dockerfile `RUN` command, we can enable docker to cache our build. Also, much more easier to maintain. Signed-off-by: John Wu <webmaster@leapoahead.com>

meetwudi · 2015-10-06T20:25:06Z

Updated!

arikfr · 2015-10-06T20:25:18Z

@tjwudi that's what data volumes are for: https://docs.docker.com/userguide/dockervolumes/

arikfr · 2015-10-06T20:27:32Z

Dockerfile

+ENV FILES_BASE_URL /opt/redash/current/setup/files/
+
+# Base packages
+RUN apt-get dist-upgrade


This line is probably not needed. I had some issues with AWS Ubuntu images, and added it. But normally everything should work fine without it.

Which line is it? GitHub didn't point out which one exactly

RUN apt-get dist-upgrade

meetwudi · 2015-10-06T20:47:59Z

@arikfr then probably add data volumes in docker-compose-example.yaml?

arthurpsmith · 2015-10-08T14:25:21Z

FYI below is the docker-compose.yml file I've been using with success. Some notes on differences:

No need to expose the ports on the redis and postgres containers - these are automatically available to the redash container via the link, they do not need to be exposed on the docker host machine.
with a POSTGRES_PASSWORD environment variable, the default admin user (postgres) on the postgres server can be accessed by username/password authentication. In my "redash_start.sh" script I have a line export PGPASSWORD=${POSTGRES_PASSWORD} when I need to log in as the postgres user via psql (ie. psql -h postgres -U postgres postgres ... works by pulling the password from the environment).
Uses a volume on the host to have a persistent database between updates of the postgres container. A different volume choice might be suitable depending on your needs.
exposes the redash port (5000) via a port map on the host. If you want nginx in front you could, (that would be one more container with the nginx image + a custom config file) but it doesn't seem needed.

redis:
  image: redis

postgres:
  image: postgres
  environment:
    POSTGRES_PASSWORD: xxxxxxxx
  volumes:
   - /opt/postgres-data:/var/lib/postgresql/data

redash:
  build: .
  container_name: redash
  links:
   - redis
   - postgres 
  environment:
    POSTGRES_PASSWORD: xxxxxxxx
  ports:
   - "8080:5000"

bobrik · 2015-10-08T15:39:34Z

I played with re:dash a bit and decided that it's nice to split celery and web into separate containers. It's easier to run on marathon and scale this way and you don't have to mess with init systems.

meetwudi · 2015-10-08T18:34:28Z

So I decided to create a separate image for redash nginx. It's now live on https://hub.docker.com/r/tjwudi/redash-nginx/ and repo https://github.com/tjwudi/redash-nginx.

Signed-off-by: John Wu <webmaster@leapoahead.com>

arikfr · 2015-10-08T18:51:07Z

Why not keep the Dockerfile and nginx.conf for the nginx image in this
repo? It will make it easier to manage access and edit Docker related
configuration.

Also make sure to add daemon off directive to nginx.conf.
On Oct 8, 2015 9:34 PM, "John Wu" notifications@github.com wrote:

So I decided to create a separate image for redash nginx. It's now live on
https://hub.docker.com/r/tjwudi/redash-nginx/ and repo
https://github.com/tjwudi/redash-nginx.

—
Reply to this email directly or view it on GitHub
#588 (comment).

meetwudi · 2015-10-08T18:55:08Z

@arikfr Also makes sense. Though I prefer one image one repo, I can do it if that is what you guys want :)

meetwudi · 2015-10-08T18:55:43Z

@arikfr daemon is off by default

Signed-off-by: John Wu <webmaster@leapoahead.com>

meetwudi · 2015-10-08T19:11:20Z

Well the thing is, the latest build does not contain supervisord_docker.conf so it can fail if you try to build. Anyway, it is now downloading from latest build instead :)

Signed-off-by: John Wu <webmaster@leapoahead.com>

meetwudi · 2015-10-08T19:15:27Z

@bobrik Separating celery is definitely what we want to do next, but probably in another branch maybe? b/c I don't know the complexity to do it yet.

arikfr · 2015-10-08T20:12:54Z

I agree that we can split the images in another iteration/branch. Also the resulting image can still be used without the supervisord with custom run command.

arikfr · 2015-10-08T20:17:00Z

setup/files/docker-redash-nginx/nginx.conf

+    error_log  /var/log/nginx/log/error.log;
+
+    location / {
+      proxy_pass  http://redashapp;


Unless in newer versions of nginx those options are on by default worth adding to the server directive:

gzip on; gzip_types *; gzip_proxied any;

and to the location / directive:

proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme;

Thanks for this!

Signed-off-by: John Wu <webmaster@leapoahead.com>

arikfr · 2015-10-08T20:31:32Z

Dockerfile

+FROM ubuntu:trusty
+MAINTAINER Di Wu <diwu@yelp.com>
+
+ENV FILES_BASE_URL /opt/redash/current/setup/files/


this is no longer used.

Signed-off-by: John Wu <webmaster@leapoahead.com>

landscape-bot · 2015-10-09T03:23:51Z

Repository health increased by 0.45% when pulling e19962d on tjwudi:diwu/feature/docker-deployment into 5d1c75d on EverythingMe:master.

14 new problems were found (including 0 errors and 14 code smells).
14 problems were fixed (including 0 errors and 14 code smells).

Docker deployment support

arikfr · 2015-10-11T20:14:24Z

Merged. I will probably tweak it some more myself, to make it work with CI and everything.

Thanks a lot for the effort making this and patience during the review. :-) 👍

meetwudi · 2015-10-12T05:08:46Z

My pleasure. Learned a lot here. Will go on contributing :) @arikfr ++

arikfr · 2015-10-12T05:41:14Z

Awesome 👍

meetwudi · 2015-11-20T21:30:10Z

@arikfr are you working on separating celery tasks? If not, I am happy to take over.

arikfr · 2015-11-22T09:26:39Z

@tjwudi you mean having Celery and gunicorn in separate containers?

meetwudi added 7 commits September 28, 2015 17:14

Remove imagemin grunt task

dedae03

grunt-contrib-imagemin seems to be broken because several dependencies are quite obsolete and cannot be downloaded.

Bind to 0.0.0.0 instead of 127.0.0.1

6f91849

Docker support

1ee05e1

Ignore .env file

28ccaed

Do not ignore .env file

aa1b729

Signed-off-by: John Wu <webmaster@leapoahead.com>

Add CMD to start service since docker doesn't support init scripts

21c413f

Signed-off-by: John Wu <webmaster@leapoahead.com>

Use standalone supervisord.conf for docker deployment

46a0083

Signed-off-by: John Wu <webmaster@leapoahead.com>

arikfr reviewed Oct 6, 2015
View reviewed changes

Dockerfile

@@ -0,0 +1,25 @@

FROM debian:wheezy

Copy link

Member

arikfr Oct 6, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason to use Debian and not Ubuntu as the base image?

meetwudi added 6 commits October 6, 2015 10:35

Remove redundant database migration scripts

11682b3

Signed-off-by: John Wu <webmaster@leapoahead.com>

Remove Node.js infra

72e48a1

Revert TCP listening address

73bd83a

Instead of binding to `0.0.0.0`, use `127.0.0.1` instead for security concerns. "The Python Web server is more vulnerable than nginx that proxies it." Signed-off-by: John Wu <webmaster@leapoahead.com>

Use ubuntu instead

407a649

Signed-off-by: John Wu <webmaster@leapoahead.com>

tag postgres & redis version in docker-compose.yaml

b0eaffd

Signed-off-by: John Wu <webmaster@leapoahead.com>

Remove redis build + use postgres-client package instead of postgres

0bbcb69

Signed-off-by: John Wu <webmaster@leapoahead.com>

Migrate stuff in bootstrap_docker.sh into Dockerfile

fb00350

By using Dockerfile `RUN` command, we can enable docker to cache our build. Also, much more easier to maintain. Signed-off-by: John Wu <webmaster@leapoahead.com>

arikfr reviewed Oct 6, 2015
View reviewed changes

meetwudi added 3 commits October 8, 2015 11:38

Use built image in docker-compose

9f799f4

Signed-off-by: John Wu <webmaster@leapoahead.com>

Add nginx frontend in docker-compose

0e96072

Signed-off-by: John Wu <webmaster@leapoahead.com>

Remove unnecessarily exposed ports

e51db08

Signed-off-by: John Wu <webmaster@leapoahead.com>

meetwudi added 3 commits October 8, 2015 11:57

Add redash-nginx repo content

72804e6

Signed-off-by: John Wu <webmaster@leapoahead.com>

Rename folder

a8d7547

Download and build from latest source instead

a699c04

Use volume to store postgres data

cf6ce05

Signed-off-by: John Wu <webmaster@leapoahead.com>

arikfr reviewed Oct 8, 2015
View reviewed changes

Add some mandatory nginx directives

99b6f89

Signed-off-by: John Wu <webmaster@leapoahead.com>

arikfr reviewed Oct 8, 2015
View reviewed changes

Remove unnecessary ENV line

e19962d

Signed-off-by: John Wu <webmaster@leapoahead.com>

arikfr added a commit that referenced this pull request Oct 11, 2015

Merge pull request #588 from tjwudi/diwu/feature/docker-deployment

9e183f1

Docker deployment support

arikfr merged commit 9e183f1 into getredash:master Oct 11, 2015

meetwudi mentioned this pull request Oct 12, 2015

docker support #450

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker deployment support #588

Docker deployment support #588

meetwudi commented Oct 5, 2015

arikfr Oct 6, 2015

arikfr commented Oct 6, 2015

meetwudi commented Oct 6, 2015

meetwudi commented Oct 6, 2015

arikfr commented Oct 6, 2015

arikfr Oct 6, 2015

meetwudi Oct 6, 2015

arikfr Oct 6, 2015

meetwudi commented Oct 6, 2015

arthurpsmith commented Oct 8, 2015

bobrik commented Oct 8, 2015

meetwudi commented Oct 8, 2015

arikfr commented Oct 8, 2015

meetwudi commented Oct 8, 2015

meetwudi commented Oct 8, 2015

meetwudi commented Oct 8, 2015

meetwudi commented Oct 8, 2015

arikfr commented Oct 8, 2015

arikfr Oct 8, 2015

meetwudi Oct 8, 2015

arikfr Oct 8, 2015

meetwudi Oct 8, 2015

landscape-bot commented Oct 9, 2015

arikfr commented Oct 11, 2015

meetwudi commented Oct 12, 2015

arikfr commented Oct 12, 2015

meetwudi commented Nov 20, 2015

arikfr commented Nov 22, 2015

Docker deployment support #588

Docker deployment support #588

Conversation

meetwudi commented Oct 5, 2015

Choose a reason for hiding this comment

arikfr commented Oct 6, 2015

meetwudi commented Oct 6, 2015

meetwudi commented Oct 6, 2015

arikfr commented Oct 6, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

meetwudi commented Oct 6, 2015

arthurpsmith commented Oct 8, 2015

bobrik commented Oct 8, 2015

meetwudi commented Oct 8, 2015

arikfr commented Oct 8, 2015

meetwudi commented Oct 8, 2015

meetwudi commented Oct 8, 2015

meetwudi commented Oct 8, 2015

meetwudi commented Oct 8, 2015

arikfr commented Oct 8, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

landscape-bot commented Oct 9, 2015

arikfr commented Oct 11, 2015

meetwudi commented Oct 12, 2015

arikfr commented Oct 12, 2015

meetwudi commented Nov 20, 2015

arikfr commented Nov 22, 2015