Enable the app to be built using Docker

[paulsturgess]

Hi, I saw your request for help on this app and I thought maybe getting it working via Docker might make local dev more appealing.

This is a work in progress PR, not ready for merging just yet. I thought it would be good to open it to get some early feedback.

TODO:

• Investigate why some specs are failing
• Test the app manually

[Firefishy]

👍 nice!

[tomhughes]

Do all those files really have to be in the top level directory? I'd really rather avoid polluting that with lots of noise if we can avoid it...

[tomhughes]

Also if I'm reading that right it's relying on a prebuilt image rather that using a stock image and checking out the code? So anybody that uses this is going to start with a snapshot of the code from heaven knows when rather than something current?

[pnorman]

• There should be a COPY or ADD instruction to get the local directory to the docker instance running the rails port. I'm not sure on the semantics of copy vs add, so I don't know which is preferred.
• Because the configuration docs require running a rails console the docker instructions should mention how to do that, as well as how to access the database directly or with psql, which are needed for populating with osmosis or general debugging.
• I don't feel qualified to review the docker-compose stuff

[pnorman]

You're adding Gemfile and Gemfile.lock, but shouldn't they come over when you bring all of the files from the local directory into docker?

[paulsturgess]

These are required so that bundler can install all the gem dependencies required for the app. They have to be manually added so that they are available during the image build process.

[mojodna]

The added benefit is that gems will only be reinstalled when Gemfile* changes, rather than when anything else in the source tree does.

[pnorman]

You should add && apt-get clean && rm -rf /var/lib/apt/lists/ to avoid including cache files in the docker image, inflating its size.

[paulsturgess]

👍

[pnorman]

I would still call running software directly on your machine the traditional and probably best-supported approach.

[paulsturgess]

👍

[pnorman]

Can this be automated, since we always know what the DB info is when running in docker?

[paulsturgess]

The guides I've read keep this kind of configuration separate. I would suggest leaving it out for now and it could be reviewed later.

[mojodna]

All of the configuration variables can be set using the environment: OSM_...

Defaults could be set in the Dockerfile and overwritten with -e or --env-file or using equivalent docker-compose.yml configuration.

[pnorman]

Add cleaning commands as in the Dockerfile

[pnorman]

In other images I've seen a CMD set so you get a meaningful console when you access it, e.g. CMD ["python2"]. Is it possible to do the same here to get a console? bundle exec rails console is the command used in CONFIGURE.md

[paulsturgess]

The CMD is actually declared in the docker-compose.yml as:
./script/rails s -p 3000 -b '0.0.0.0'

[tomhughes]

That script/rails is kind of ancient history - it should really have been deleted long since I think. A newly created rails app wouldn't have it.

I think something like bundle exec rails s -p 3000 -b '0.0.0.0' is more like what you should be doing these days.

[paulsturgess]

👍

[mojodna]

Including puma or similar would be nice if this image were also going to be used for "production" purposes (e.g. POSM, where we currently use puma).

[mojodna]

(Commenting here because it's the end of the file.)

COPY . /app should be included if this image is intended to be used independently of a local git clone, otherwise none of the source files will be available in the image.

(This can co-exist happily with mapping $(pwd) to /app as a volume; files present in the image will be shadowed by local paths.)

[mojodna]

Also consider adding -j $(nproc) as an argument to speed up gem installation (it will run a number of processes matching the cores available on the system doing the building).

[pnorman]

These are different than the listed minimum requirements. I can understand ruby being different since it's coming from a ruby image and the lack of postgresql, but are there reasons for the other differences?

[pnorman]

Is this needed for anything other than debugging? I'm okay with keeping it, but if it's only for debugging we should add a comment.

[paulsturgess]

I'm pretty sure there was a reason to add postgresql-client while I was getting this all up and running but I can't remember precisely off the top of my head. I'm happy to try removing it and rebuilding to see.

[paulsturgess]

OK I tried removing it and when running the migrations it bails out because it can't execute:
pg_dump -s -x -O -f /app/db/structure.sql openstreetmap

This is run because of the setting in application.rb:
config.active_record.schema_format = :sql

[pnorman]

Also if I'm reading that right it's relying on a prebuilt image rather that using a stock image and checking out the code? So anybody that uses this is going to start with a snapshot of the code from heaven knows when rather than something current?

After reviewing everything I think I understand everything, but take it with a grain of salt because I don't fully understand docker-compose. When docker builds the image it copies the Gemfile and Gemfile.lock and does a bundle install. When the image is run it copies the local directory and runs the rails port, using what was in the local directory when the image was run.

I think this works because if the gemfiles are touched it invalidates that layer of the build and requires it to be rebuilt, while the next steps (bundle install) do not depend on the rest of the website code.

If this works it means that you can make changes and test them without having to do a new bundle install

[tomhughes]

If it's copying in the code then what is in the openstreetmap-website:v1 image over and above stock ububtu?

[tomhughes]

Having said that I can't obviously find that in the registry so maybe I am misunderstanding?

[pnorman]

If it's copying in the code then what is in the openstreetmap-website:v1 image over and above stock ububtu?

The docs say it's what compose names the built image

[paulsturgess]

Hey guys lot's of comments, thanks.

I can't address them all right at the moment but to pick one:

There should be a COPY or ADD instruction to get the local directory to the docker instance running the rails port. I'm not sure on the semantics of copy vs add, so I don't know which is preferred.

This isn't required as the docker-compose.yml specifies .:/app as the volume which exposes the code locally on your machine to make it available in the container at the location /app.

This means you can make changes locally and see them reflected when you refresh the browser.

This would obviously be different you wanted to use Docker to deploy to a production server where it would need to clone down the repo. Hopefully that makes sense and apologies if i've misunderstood!

[tomhughes]

So I think I've figured it out now and the images from the hub it is using are:

• ruby:2.3.1-slim which is the 2.3.1-slim tag from https://hub.docker.com/_/ruby/
• postgres:9.4 which is the 9.4 tag from https://hub.docker.com/_/postgres/

Which seems reasonable I guess.

[paulsturgess]

@tomhughes that's exactly right. It's very easy to switch those if you prefer other versions. For instance, it would probably be a good idea to match the production server environment.

[paulsturgess]

OK so I think I've addressed the comments.

• Changed the installed packages to follow the minimum requirements specified in INSTALL.md
• Use bundle exec rails s -p 3000 -b '0.0.0.0'
• Moved docker_postgres.sh into /db to reduce the noise in the root directory (I can't move all the files due to the way docker will not ../ backwards to access files)
• Added more info to the DOCKER.md
• Tweaked the INSTALL.md to state installing directly is best supported approach.

The app boots and it appears to load ok, but I'm not familiar with it at all.

There are 4 specs that either error or fail locally. The output is below. Does anyone know what might be causing those?

Spec failure output

1) Error:
DiaryEntryTest#test_diary_entry_visible_comments:
ActiveRecord::InvalidForeignKey: PG::ForeignKeyViolation: ERROR:  insert or update on table "diary_entries" violates foreign key constraint "diary_entries_language_code_fkey"
DETAIL:  Key (language_code)=(en) is not present in table "languages".
: INSERT INTO "diary_entries" ("title", "body", "user_id", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5) RETURNING "id"
    test/models/diary_entry_test.rb:37:in `test_diary_entry_visible_comments'


  2) Error:
DiaryEntryTest#test_diary_entry_comments:
ActiveRecord::InvalidForeignKey: PG::ForeignKeyViolation: ERROR:  insert or update on table "diary_entries" violates foreign key constraint "diary_entries_language_code_fkey"
DETAIL:  Key (language_code)=(en) is not present in table "languages".
: INSERT INTO "diary_entries" ("title", "body", "user_id", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5) RETURNING "id"
    test/models/diary_entry_test.rb:30:in `test_diary_entry_comments'


  3) Error:
DiaryEntryTest#test_diary_entry_visible:
ActiveRecord::InvalidForeignKey: PG::ForeignKeyViolation: ERROR:  insert or update on table "diary_entries" violates foreign key constraint "diary_entries_language_code_fkey"
DETAIL:  Key (language_code)=(en) is not present in table "languages".
: INSERT INTO "diary_entries" ("title", "body", "user_id", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5) RETURNING "id"
    test/models/diary_entry_test.rb:21:in `test_diary_entry_visible'


  4) Failure:
TraceTest#test_xml_file [/app/test/models/trace_test.rb:145]:
--- expected
+++ actual
@@ -1 +1,2 @@
-"848caa72f2f456d1bd6a0fdf228aa1b9"
+# encoding: US-ASCII
+"d41d8cd98f00b204e9800998ecf8427e"

[pnorman]

There are 4 specs that either error or fail locally. The output is below. Does anyone know what might be causing those?

I'm not familiar enough with the osm-website code but for any encoding, language, or locale issues on Docker I tend to suspect the absurdly minimal set of locales the images come with.

[tomhughes]

The first three are apparently because the languages table isn't loaded properly. I saw something similar myself the other day after merging @gravitystorm's factory changes but I couldn't reproduce it again so decided it was a transient glitch but maybe not.

The last one is not something I've seen before and looks very odd to me as it appears that Digest::MD5#hexdigest has returned a digest prefixed with an encoding comment!

[tomhughes]

So I was going to try this, but it turns out I don't have a docker-compose command... I have a docker command, which is what I've always used in the past to start docker containers?

[tomhughes]

Found it - there's a separate docker-compose package.

[tomhughes]

So the first time I ran it the docker-compose bombed out building the web site image due to a random HTTP failure talking to the debian repos and now I can't start it because every attempt bombs out with:

Starting rails_db_1

ERROR: for db  failed to create endpoint rails_db_1 on network rails_default: Error starting userland proxy: listen tcp 0.0.0.0:5432: bind: address already in use

Presumably because the database image from the first attempt is still running, but I can't work out how to stop it - a docker-compose stop doesn't seem to help.

I can't help feeling it all seems very fragile...

[tomhughes]

Hmm no apparently that is because it is trying to bind port 5432 on the host which, unsurprisingly, is already in use by my host system's postgres daemon!

Shouldn't it be using a private network or something to connect the two containers?

[paulsturgess]

Hey @tomhughes. I'm definitely not an expert on the inner workings of Docker, but as I understand it the two containers defined in the docker-compose do talk directly to each other. Hence the need for the setting of host: db in the database.yml file – basically this translates to the IP address the db container is using without actually having to hard-code an IP.

This is possible because docker-compose is configured to make the db container available to the web container via:

depends_on:
      - db

The port 5432 is also defined in the docker-compose – this is to make it available to your local machine. As you discovered, this clashes if you also have Postgres running locally on the same port. Obviously the idea with Docker is that you don't need to run any services locally.

Hope this makes sense.

[pnorman]

Shouldn't it be using a private network or something to connect the two containers?

Yes, it should.

[paulsturgess]

OK I spotted the path to the db functions was incorrect in db/docker_postgres.sh so I've updated it.

I think the three PG::ForeignKeyViolation spec failures are intermittent. I've seen them pass locally for me and then fail with no code changes. The Travis build for this PR has also fluctuated between pass and fail.

[pnorman]

Standard practice is to have all the apt-get stuff on one line

[tomhughes]

@Firefishy where are we at with the improvements you were working on in Brussels so we can merge this?

[pnorman]

Do all those files really have to be in the top level directory? I'd really rather avoid polluting that with lots of noise if we can avoid it...

I was working with docker for something else and files mentioned in the Dockerfile need to be in ./ or a subdirectory.

[mojodna]

Can the sed command be bypassed by setting this before packages are installed? Reconfiguring locales isn't fast, so doing that earlier would make it quicker to iterate when using this for development purposes.

[mojodna]

(For those new to Docker, each Docker "command" creates a filesystem layer. Filesystem layers are (typically) cached, allowing changes to later parts of the process to skip installation steps that were already taken care of (that were also defined earlier in the Dockerfile).)

[mojodna]

(Changes to files COPY'd or ADDed into an image invalidate those layers (and subsequent ones), similar to make.)

[mojodna]

Also if I'm reading that right it's relying on a prebuilt image rather that using a stock image and checking out the code? So anybody that uses this is going to start with a snapshot of the code from heaven knows when rather than something current?

The general idea is that an image is created (using this Dockerfile) and pushed to a registry either on each commit or release within this repository (it can even be automated as though it were another continuous integration check) (I'm partial to Quay.io for this functionality over Docker Hub).

openstreetmap/openstreetmap-website:latest would always correspond to the "latest" version but additional tagged releases can also be made available if users wish to stay on "known good" releases / commits.

It would be possible for the image to check out a current version of the code each time it's run, but that runs counter to the "immutable infrastructure" philosophy that Docker espouses (and enables).

[mojodna]

Also consider creating/running as an unprivileged user.

[mojodna]

(Sorry, didn't mean to trigger a review on this.)

[mojodna]

(Commenting here because it's the end of the file.)

COPY . /app should be included if this image is intended to be used independently of a local git clone, otherwise none of the source files will be available in the image.

(This can co-exist happily with mapping $(pwd) to /app as a volume; files present in the image will be shadowed by local paths.)

[mojodna]

Also consider adding -j $(nproc) as an argument to speed up gem installation (it will run a number of processes matching the cores available on the system doing the building).

[tomhughes]

@mojodna I don't think there's any intention for this to be published to the registry, and it's certainly not something we're expecting people to use for production use. The goal here is to make it easier for people wanting to work on the code to get a development environment.

[mojodna]

Cool, that makes total sense.

[kocio-pl]

Dockerized version of osm-website would be probably useful for me, what are the chances to have this code merged?

[systemed]

Yes, it would be terrific to have this - about once every two months I want to get openstreetmap-website up and running to patch something, but struggle and eventually fail (usually due to Mac peculiarities and Ruby conflicts). @paulsturgess , @mojodna , is there anything anyone can do to help this move along?

[paulsturgess]

Unfortunately I don't have the time to work further on this or check if it still works but obviously happy for someone to take this.

Looking at my last comment over a year ago, it appeared to be done but there were some intermittent failures occurring on Travis but no-one was able to shed any light on those.

I'm not entirely sure on the resistance to merging it and iterating on it as it has no impact on the production code.

[Zverik]

Isn't vagrant enough for local development?

[kocio-pl]

It looks like Vagrant is just a starting command - you have to have all the VMs already.

[Zverik]

No, it downloads a clean ubuntu image and runs this script, so basically you do a vagrant up and have your development system in ten minutes.

[kocio-pl]

I've checked and it works with Vagrant, I was able to use the website locally reading docs. I have also found that cruise.openstreetmap.org server mentioned in CONTRIBUTING.md is unavailable (no DNS record), but that's a different issue.

So Docker version might be just not needed.

[Zverik]

Docker is still relevant, and many developers are used to it. Could we reopen this please, even if it stays unresolved a few more months?

This could probably benefit from using the same or similar files to vagrant's.

[fazlerabbi37]

Hi, is there any update or progress on this? I am getting started with docker technology, and I was hoping to build on top of this PR. What do you guys think about it?
cc @paulsturgess; @gravitystorm @tomhughes @mojodna @pnorman

[tomhughes]

@fazlerabbi37 Well you sounds like an excellent candidate to implement docker support then, because none of us uses it - of course ideally we'd likely somebody that will actually maintain the support because otherwise it is likely to bit rot if none of us are using it.

[fazlerabbi37]

@tomhughes great! I will get started 😄 .

---

I was thinking of creating a new PR. Would that be ok?

[tomhughes]

Yes of course.

[timwaters]

There has been some recent docker and osm website work over on the OpenHistoricalMap side of things on this which might help your PR @fazlerabbi37 https://github.com/OpenHistoricalMap/openhistoricalmap-docker . One note for example is the conversion of the postgres C functions to SQL versions so it can work with cloud sql instances.

[tomhughes]

That appears to be entirely unrelated to whether or not the site is running in docker though...

[fazlerabbi37]

@timwaters I was hoping to focus first on make the dev setup easier through docker. Once that is done, we can surely move on to make things production ready. What do you think? @tomhughes . Thanks for the OpenHistoricalMap link BTW, I will take a look at it soon 😄.