Secure MongoDB and Redis

README.md

@@ -44,9 +44,10 @@ Below is a diagram (made with [Lucidchart](https://www.lucidchart.com)) of the c
 
 ![Diagram of Docker Containers](./doc/container-diagram.svg)
 
-### Secure your installation!
-kobo-docker **opens ports on all interfaces** to let the `frontend` containers communicate with `backend` containers.
-A firewall is **HIGHLY recommended**. You **MUST** block PostgreSQL, Redis and MongoDB ports when the server is exposed publicly.
+### Secure your installation
+This version of kobo-docker does **not** expose backend container ports, but [previous versions did](https://github.com/kobotoolbox/kobo-docker/pull/280), relying on a firewall to prevent unauthorized access. You should always verify that your database ports (by default 5432, 27017, 6379, 6380) are not accessible to the public.
+
+If you want to use kobo-docker with separate front-end and back-end servers, you will need to expose ports, and **you MUST use a firewall**. The firewall is required to allow only the `frontend` containers to access PostgreSQL, Redis, and MongoDB.
 
 
 ## Setup procedure
@@ -70,7 +71,7 @@ Already have an existing installation? Please see below.
     - `kobo-deployments/envfiles/kobocat.txt`
         ```diff
         - KOBOCAT_BROKER_URL=amqp://kobocat: kobocat@rabbit.[internal domain name]:5672/kobocat
-        + KOBOCAT_BROKER_URL =redis://redis-main.[internal domain name]:6389/2`
+        + KOBOCAT_BROKER_URL=redis://redis-main.[internal domain name]:6389/2
         ```
 
 2. **Load balancing and redundancy**
@@ -89,7 +90,7 @@ Already have an existing installation? Please see below.
             - MongoDB
             - Redis
 
-        Docker-compose for `frontend` can be started on its own server, same thing for `backend`. Users can start as many `frontend` servers they want. A load balancer can spread the traffic between `frontend` servers.
+        Docker-compose for `frontend` can be started on its own server, same thing for `backend`. Users can start as many front-end servers they want. A load balancer can spread the traffic between front-end servers.
         kobo-docker uses (private) domain names between `frontend` and `backend`.
         It's fully customizable in configuration files. Once again, [kobo-install](https://github.com/kobotoolbox/kobo-install) does simplify the job by creating the configuration files for you.
 
@@ -103,21 +104,25 @@ Already have an existing installation? Please see below.
     ![aws diagram](./doc/aws-diagram.svg)
 
 ## Usage
-It's recommended to create `*.override.yml` docker-compose files to customize your environment. It makes easier to update.
+It's recommended to create `*.override.yml` docker-compose files to customize your environment. It makes easier to update. 
+Samples are provided. Remove `.sample` extension and update them to match your environment. 
 
 - `docker-compose.frontend.override.yml`
 - `docker-compose.backend.master.override.yml`
 - `docker-compose.backend.slave.override.yml` (if a postgres replica is used)
 
-1. **Start/start containers**
+1. **Start/start containers** 
+
     ```
-    $kobo-docker> docker-compose -f docker-compose.frontend.yml [-f docker-compose.frontend.override.yml] up -d
-    $kobo-docker> docker-compose -f docker-compose.backend.master.yml [-f docker-compose.backend.master.override.yml] up -d
-    $kobo-docker> docker-compose -f docker-compose.frontend.yml [-f docker-compose.frontend.override.yml] stop
-    $kobo-docker> docker-compose -f docker-compose.backend.master.yml [-f docker-compose.backend.master.override.yml] stop
+    # Start
+    $kobo-docker> docker-compose -f docker-compose.frontend.yml -f docker-compose.frontend.override.yml up -d  
+    $kobo-docker> docker-compose -f docker-compose.backend.master.yml -f docker-compose.backend.master.override.yml up -d
+
+    # Stop
+    $kobo-docker> docker-compose -f docker-compose.frontend.yml -f docker-compose.frontend.override.yml stop  
+    $kobo-docker> docker-compose -f docker-compose.backend.master.yml -f docker-compose.backend.master.override.yml stop
     ```
 
-
 2. **Backups**
 
     Automatic, periodic backups of KoBoCAT media, MongoDB, PostgreSQL and Redis can be individually enabled by uncommenting (and optionally customizing) the `*_BACKUP_SCHEDULE` variables in your envfiles.
@@ -132,10 +137,10 @@ It's recommended to create `*.override.yml` docker-compose files to customize yo
     Backups **on disk** can also be manually triggered when kobo-docker is running by executing the the following commands:
 
     ```
-    $kobo-docker> docker-compose -f docker-compose.frontend.yml [-f docker-compose.frontend.override.yml] exec kobocat /srv/src/kobocat/docker/backup_media.bash
-    $kobo-docker> docker-compose -f docker-compose.backend.master.yml [-f docker-compose.backend.master.override.yml] exec mongo /bin/bash /kobo-docker-scripts/backup-to-disk.bash
-    $kobo-docker> docker-compose -f docker-compose.backend.master.yml [-f docker-compose.backend.master.override.yml] exec -e PGUSER=kobo postgres /bin/bash /kobo-docker-scripts/backup-to-disk.bash
-    $kobo-docker> docker-compose -f docker-compose.backend.master.yml [-f docker-compose.backend.master.override.yml] exec redis_main /bin/bash /kobo-docker-scripts/backup-to-disk.bash
+    $kobo-docker> docker-compose -f docker-compose.frontend.yml -f docker-compose.frontend.override.yml exec kobocat /srv/src/kobocat/docker/backup_media.bash
+    $kobo-docker> docker-compose -f docker-compose.backend.master.yml -f docker-compose.backend.master.override.yml exec mongo bash /kobo-docker-scripts/backup-to-disk.bash
+    $kobo-docker> docker-compose -f docker-compose.backend.master.yml -f docker-compose.backend.master.override.yml exec -e PGUSER=kobo postgres bash /kobo-docker-scripts/backup-to-disk.bash
+    $kobo-docker> docker-compose -f docker-compose.backend.master.yml -f docker-compose.backend.master.override.yml exec redis_main bash /kobo-docker-scripts/backup-to-disk.bash
     ```
 
 2. **Restore backups**
@@ -150,20 +155,20 @@ It's recommended to create `*.override.yml` docker-compose files to customize yo
 
     There is one composer file `docker-compose.maintenance.yml` can be used to put `KoBoToolbox` in maintenance mode.
 
-    `nginx` container has to be stopped before launching the maintenance container.
+    NGINX container has to be stopped before launching the maintenance container.
 
     **Start**
 
     ```
-    docker-compose -f docker-compose.frontend.yml [-f docker-compose.frontend.override.yml] stop nginx
+    docker-compose -f docker-compose.frontend.yml -f docker-compose.frontend.override.yml stop nginx
     docker-compose -f docker-compose.maintenance.yml up -d
     ```
 
     **Stop**
 
     ```
     docker-compose -f docker-compose.maintenance.yml down
-    docker-compose -f docker-compose.frontend.yml [-f docker-compose.frontend.override.yml] up -d nginx
+    docker-compose -f docker-compose.frontend.yml -f docker-compose.frontend.override.yml up -d nginx
     ```
 
     There are 4 variables that can be customized in `docker-compose.maintenance.yml`
@@ -178,13 +183,13 @@ It's recommended to create `*.override.yml` docker-compose files to customize yo
 - ### Basic troubleshooting
     You can confirm that your containers are running with `docker ps`.
     To inspect the log output from:
-
-     - the frontend containers, execute `docker-compose -f docker-compose.frontend.yml [-f docker-compose.frontend.override.yml] logs -f`
-     - the master backend containers, execute `docker-compose -f docker-compose.backend.master.yml [-f docker-compose.backend.master.override.yml] logs -f`
-     - the slaved backend container, execute `docker-compose -f docker-compose.backend.slave.yml [-f docker-compose.backend.slave.override.yml] logs -f`
-
-    For a specific container use e.g. `docker-compose -f docker-compose.backend.master.yml [-f docker-compose.backend.master.override.yml] logs -f redis_main`.
-
+     
+     - the frontend containers, execute `docker-compose -f docker-compose.frontend.yml -f docker-compose.frontend.override.yml logs -f`
+     - the master backend containers, execute `docker-compose -f docker-compose.backend.master.yml -f docker-compose.backend.master.override.yml logs -f`
+     - the slaved backend container, execute `docker-compose -f docker-compose.backend.slave.yml -f docker-compose.backend.slave.override.yml logs -f`
+       
+    For a specific container use e.g. `docker-compose -f docker-compose.backend.master.yml -f docker-compose.backend.master.override.yml logs -f redis_main`.
+    
     The documentation for Docker can be found at https://docs.docker.com.
 
 - ### Django debugging

deployments/envfiles/databases.txt

@@ -6,6 +6,11 @@
 # Please see kobocat.txt to set container variables
 KOBO_MONGO_PORT=27017
 KOBO_MONGO_HOST=mongo.domain.name
+MONGO_INITDB_ROOT_USERNAME=root
+MONGO_INITDB_ROOT_PASSWORD=kobo
+MONGO_INITDB_DATABASE=formhub
+KOBO_MONGO_USERNAME=kobo
+KOBO_MONGO_PASSWORD=kobo
 
 # Default MongoDB backup schedule is weekly at 01:00 AM UTC on Sunday.
 #MONGO_BACKUP_SCHEDULE=0 1 * * 0
@@ -20,12 +25,14 @@ KOBO_MONGO_HOST=mongo.domain.name
 # `DATABASE_URL` environment variable.
 POSTGRES_PORT=5432
 POSTGRES_HOST=postgres.domain.name
-POSTGRES_DB=kobotoolbox
 POSTGRES_USER=kobo
 POSTGRES_PASSWORD=kobo
+KC_POSTGRES_DB=kobocat
+KPI_POSTGRES_DB=koboform
 
 # Postgres database used by kpi and kobocat Django apps
-DATABASE_URL=postgis://kobo:kobo@postgres.domain.name:5432/kobotoolbox
+KC_DATABASE_URL=postgis://kobo:kobo@postgres.domain.name:5432/kobotoolbox
+KPI_DATABASE_URL=postgis://kobo:kobo@postgres.domain.name:5432/kobotoolbox
 
 # Replication. Password is mandatory
 KOBO_POSTGRES_REPLICATION_USER=kobo_replication
@@ -41,3 +48,6 @@ KOBO_POSTGRES_MASTER_ENDPOINT=primary.postgres.domain.name
 #--------------------------------------------------------------------------------
 
 #REDIS_BACKUP_SCHEDULE=0 3 * * 0
+
+REDIS_SESSION_URL=redis://:kobo@redis-cache.kobo.private:6390/2
+REDIS_PASSWORD=kobo

deployments/envfiles/kobocat.txt

@@ -5,7 +5,7 @@ USE_X_FORWARDED_HOST=False
 DJANGO_SETTINGS_MODULE=onadata.settings.kc_environ
 ENKETO_VERSION=Express
 
-KOBOCAT_BROKER_URL=redis://redis-main.domain.name:6379/2
+KOBOCAT_BROKER_URL=redis://:kobo@redis-main.domain.name:6379/2
 KOBOCAT_CELERY_LOG_FILE=/srv/logs/celery.log
 
 #ENKETO_OFFLINE_SURVEYS=True

deployments/envfiles/kpi.txt

@@ -5,7 +5,7 @@ USE_X_FORWARDED_HOST=False
 
 ENKETO_VERSION=Express
 KPI_PREFIX=/
-KPI_BROKER_URL=redis://redis-main.domain.name:6379/1
+KPI_BROKER_URL=redis://:kobo@redis-main.domain.name:6379/1
 
 KPI_MONGO_HOST=mongo.domain.name
 

docker-compose.backend.master.override.yml.sample

@@ -0,0 +1,54 @@
+# For public, HTTPS servers.
+version: '2.2'
+
+services:
+
+  postgres:
+    #environment:
+    #  - POSTGRES_BACKUP_FROM_SLAVE=True
+    # Uncomment `ports` section if you want to expose ports (e.g. use as separated servers)
+    # Before uncommenting, be sure you have a firewall configured to prevent these ports
+    # from being publicly accessible
+    #ports:
+    #  - 5432:5432
+    # Comment out `networks` section below if you want to expose ports
+    networks:
+      kobo-be-network:
+        aliases:
+          - postgres.kobo.private
+
+  mongo:
+    # Uncomment `ports` section if you want to expose ports (e.g. use as separated servers)
+    # Before uncommenting, be sure you have a firewall configured to prevent these ports
+    # from being publicly accessible
+    #ports:
+    #  - 27017:27017
+    # Comment out section below if you want to expose ports
+    networks:
+      kobo-be-network:
+        aliases:
+          - mongo.kobo.private
+
+  redis_main:
+    # Uncomment `ports` section if you want to expose ports (e.g. use as separated servers)
+    # Before uncommenting, be sure you have a firewall configured to prevent these ports
+    # from being publicly accessible
+    #ports:
+    #  - 6379:6379
+    # Comment out `networks` section below if you want to expose ports
+    networks:
+      kobo-be-network:
+        aliases:
+          - redis-main.kobo.private
+
+  redis_cache:
+    # Uncomment `ports` section if you want to expose ports (e.g. use as separated servers)
+    # Before uncommenting, be sure you have a firewall configured to prevent these ports
+    # from being publicly accessible
+    #ports:
+    #  - 6380:6380
+    # Comment out `networks` section below if you want to expose ports
+    networks:
+      kobo-be-network:
+        aliases:
+          - redis-cache.kobo.private

docker-compose.backend.master.yml

@@ -27,3 +27,7 @@ services:
       service: redis_cache
     environment:
       - KOBO_REDIS_SERVER_ROLE=cache
+
+networks:
+  kobo-be-network:
+    driver: bridge

docker-compose.backend.slave.yml

@@ -9,19 +9,6 @@ services:
     environment:
       - KOBO_POSTGRES_DB_SERVER_ROLE=slave
 
-#  mongo:
-#    extends:
-#      file: docker-compose.primary.backend.server.yml
-#      service: mongo
-#
-#  # Adapted from https://github.com/kobotoolbox/enketo-express/blob/docker/docker-compose.yml.
-#  redis_main:
-#    extends:
-#      file: docker-compose.primary.backend.server.yml
-#      service: redis_main
-#
-#  # Adapted from https://github.com/kobotoolbox/enketo-express/blob/docker/docker-compose.yml.
-#  redis_cache:
-#    extends:
-#      file: docker-compose.primary.backend.server.yml
-#      service: redis_cache
+networks:
+  kobo-be-network:
+    driver: bridge

docker-compose.backend.template.yml

@@ -14,12 +14,9 @@ services:
       - ./backups/postgres:/srv/backups
       - ./log/postgres:/srv/logs
       - ./postgres:/kobo-docker-scripts
-    command: "/bin/bash /kobo-docker-scripts/entrypoint.sh"
+    command: "bash /kobo-docker-scripts/entrypoint.sh"
     restart: on-failure
     stop_grace_period: 5m
-    # Ports should be declared in `docker-compose.backend.master.override.yml` and docker-compose.backend.slave.override.yml`
-    #ports:
-    #  - 5432:5432
 
   mongo:
     image: mongo:3.4
@@ -36,16 +33,11 @@ services:
       - ./mongo:/kobo-docker-scripts
       - ./log/mongo:/srv/logs
     restart: on-failure
-    command: "/bin/bash /kobo-docker-scripts/entrypoint.sh"
+    command: "bash /kobo-docker-scripts/entrypoint.sh"
     stop_grace_period: 5m
-    # Ports should be declared in `docker-compose.backend.master.override.yml` and docker-compose.backend.slave.override.yml`
-    #ports:
-    #  - 27017:27017
 
-  # Adapted from https://github.com/kobotoolbox/enketo-express/blob/docker/docker-compose.yml.
   redis_main:
     image: redis:3.2
-    # Map our "main" Redis config into the container.
     env_file:
       - ../kobo-deployments/envfile.txt
       - ../kobo-deployments/envfiles/databases.txt
@@ -59,27 +51,21 @@ services:
       - ./log/redis_main:/var/log/redis
     restart: on-failure
     stop_grace_period: 2m30s
-    # Ports should be declared in `docker-compose.backend.master.override.yml` and docker-compose.backend.slave.override.yml`
-    #ports:
-    #  - 6379:6379
     sysctls:
         - net.core.somaxconn=2048
-    command: "/bin/bash /tmp/redis/entrypoint.sh"
+    command: "bash /tmp/redis/entrypoint.sh"
 
-  # Adapted from https://github.com/kobotoolbox/enketo-express/blob/docker/docker-compose.yml.
   redis_cache:
     image: redis:3.2
-    # Map our "cache" Redis config into the container.
+    env_file:
+      - ../kobo-deployments/envfiles/databases.txt
     volumes:
       - ./.vols/redis_cache_data/:/data/
       - ./redis/redis-enketo-cache.conf.tmpl:/etc/redis/redis.conf.tmpl:ro
       - ./redis/entrypoint.sh:/tmp/redis/entrypoint.sh:ro
       - ./log/redis_cache:/var/log/redis
     restart: on-failure
     stop_grace_period: 2m30s
-    # Ports should be declared in `docker-compose.backend.master.override.yml` and docker-compose.backend.slave.override.yml`
-    #ports:
-    #  - 6380:6380
     sysctls:
         - net.core.somaxconn=2048
-    command: "/bin/bash /tmp/redis/entrypoint.sh"
+    command: "bash /tmp/redis/entrypoint.sh"

docker-compose.frontend.override.yml.sample

@@ -0,0 +1,67 @@
+# For public, HTTPS servers.
+version: '3'
+
+services:
+  kobocat:
+    environment:
+      # change `ENKETO_PROTOCOL` to http if HTTPS is not used
+      - ENKETO_PROTOCOL=https
+      # `NGINX_PUBLIC_PORT` is the port used to access KoBoToolbox (e.g. `https://kc.kobotoolbox.org:<NGINX_PUBLIC_PORT>`)
+      - NGINX_PUBLIC_PORT=80
+      # Uncomment the lines below to tweak uWSGI
+      #- KC_UWSGI_WORKERS_COUNT=2
+      #- KC_UWSGI_CHEAPER_WORKERS_COUNT=1
+      #- KC_UWSGI_MAX_REQUESTS=512
+      #- KC_UWSGI_CHEAPER_RSS_LIMIT_SOFT=134217728
+    networks:
+      kobo-be-network:
+        aliases:
+          - kobocat
+          - kobocat.docker.container
+
+  kpi:
+    environment:
+      # `NGINX_PUBLIC_PORT` is the port used to access KoBoToolbox (e.g. `https://kf.kobotoolbox.org:<NGINX_PUBLIC_PORT>`)
+      - NGINX_PUBLIC_PORT=80
+      # Uncomment the lines below to tweak uWSGI
+      #- KPI_UWSGI_WORKERS_COUNT=2
+      #- KPI_UWSGI_CHEAPER_WORKERS_COUNT=1
+      #- KPI_UWSGI_MAX_REQUESTS=512
+      #- KPI_UWSGI_CHEAPER_RSS_LIMIT_SOFT=134217728
+      # Comment out the line below if HTTPS is not used
+      - SECURE_PROXY_SSL_HEADER=HTTP_X_FORWARDED_PROTO, https
+    networks:
+      kobo-be-network:
+        aliases:
+          - kpi
+          - kpi.docker.container
+
+  nginx:
+    environment:
+      # `NGINX_PUBLIC_PORT` is the port used to access KoBoToolbox (e.g. `https://kf.kobotoolbox.org:<NGINX_PUBLIC_PORT>`)
+      - NGINX_PUBLIC_PORT=80
+    ports:
+      # <proxy_port>:80 . If no proxies, `proxy_port` should be the same as `NGINX_PUBLIC_PORT`
+      - 80:80
+    networks:
+      kobo-fe-network:
+        aliases:
+          - nginx
+          # These aliases must match the concatenation of `*_PUBLIC_SUBDOMAIN` and `INTERNAL_DOMAIN_NAME`
+          # found in `../kobo-deployments/envfile.txt`
+          - kf.docker.internal
+          - kc.docker.internal
+          - ee.docker.internal
+
+  enketo_express:
+    networks:
+      kobo-be-network:
+        aliases:
+          - enketo_express
+
+networks:
+  kobo-be-network:
+    external:
+      # name: <prefix>_kobo-be-network`, where `prefix` is usually the parent
+      # folder name
+      name: kobodocker_kobo-be-network