Try building with Docker #32

nightscape · 2015-09-17T21:16:42Z

Hi Paul,

I'm trying to get Unison to run in Docker which would make it super-easy for people to try it out by just doing a

docker run unisonweb/platform

Unfortunately I'm completely new to nix and cabal and compilation fails with changing reasons, the last one being

Preprocessing library base64-bytestring-1.0.0.1...
libtool: link: gcc -std=gnu99 -DUSE_POSIX_THREADS -g -O2 -o .libs/gettext-6-prg gettext_6_prg-gettext-6-prg.o  ../gnulib-lib/.libs/libgettextlib.so ../intl/.libs/libgnuintl.so -lc -lpthread -fopenmp -Wl,-rpath -Wl,/nix/store/lsgh3rjy7mipimjrgi5cnjz6nvq7rahy-gettext-0.18.2/lib
gcc -std=gnu99 -DLOCALEDIR=\"/nix/store/lsgh3rjy7mipimjrgi5cnjz6nvq7rahy-gettext-0.18.2/share/locale\" -DHAVE_CONFIG_H -I. -I..  -I.. -I../gnulib-lib -I../gnulib-lib -I../intl  -DUSE_POSIX_THREADS -g -O2 -c -o gettext_7_prg-gettext-7-prg.o `test -f 'gettext-7-prg.c' || echo './'`gettext-7-prg.c
/nix/store/0fmbypn0w29r3lykg5w1glbphz1pdbkz-bash-4.3-p33/bin/bash ../libtool  --tag=CC   --mode=link gcc -std=gnu99 -DUSE_POSIX_THREADS -g -O2   -o gettext-7-prg gettext_7_prg-gettext-7-prg.o ../gnulib-lib/libgettextlib.la ../intl/libgnuintl.la    -lpthread

malloc: dispose_cmd.c:249: assertion botched
free: called with already freed block argument
Aborting...Makefile:1810: recipe for target 'gettext-7-prg' failed
make[3]: *** [gettext-7-prg] Aborted
make[3]: Leaving directory '/tmp/nix-build-gettext-0.18.2.drv-0/gettext-0.18.2/gettext-tools/tests'
Makefile:1660: recipe for target 'all-recursive' failed
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory '/tmp/nix-build-gettext-0.18.2.drv-0/gettext-0.18.2/gettext-tools'
Makefile:1525: recipe for target 'all' failed
make[1]: *** [all] Error 2
make[1]: Leaving directory '/tmp/nix-build-gettext-0.18.2.drv-0/gettext-0.18.2/gettext-tools'
Makefile:315: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1
builder for ‘/nix/store/p0b9lzdxi598zq8i9fp4p6qciixxkazd-gettext-0.18.2.drv’ failed with exit code 2
cannot build derivation ‘/nix/store/dscyq920pg351zi72wqg1sg0cbp14qlc-git-2.3.6.drv’: 1 dependencies couldn't be built

Could you maybe try if a

docker build -t unison .

in this branch works on your machine?

Best
Martin

BTW: I hope it's OK that I already created a PR with this although it's not yet running correctly. I find PRs very useful for tracking code and comments in the same place.

pchiusano · 2015-09-17T23:26:29Z

Cool. I don't know much about docker at all, but I'd have no objections to
merging this if you can get it working. Maybe look at the vagrantfile and
provisioning script, those are known to work, I'm guessing docker
provisioning will be similar? See the info on this in the README.

I'll also try running that on my machine.
On Thu, Sep 17, 2015 at 5:16 PM Martin Mauch notifications@github.com
wrote:

Hi Paul,

I'm trying to get Unison to run in Docker which would make it super-easy
for people to try it out by just doing a

docker run unisonweb/platform

Unfortunately I'm completely new to nix and cabal and compilation fails
with changing reasons, the last one being

Preprocessing library base64-bytestring-1.0.0.1...
libtool: link: gcc -std=gnu99 -DUSE_POSIX_THREADS -g -O2 -o .libs/gettext-6-prg gettext_6_prg-gettext-6-prg.o ../gnulib-lib/.libs/libgettextlib.so ../intl/.libs/libgnuintl.so -lc -lpthread -fopenmp -Wl,-rpath -Wl,/nix/store/lsgh3rjy7mipimjrgi5cnjz6nvq7rahy-gettext-0.18.2/lib
gcc -std=gnu99 -DLOCALEDIR="/nix/store/lsgh3rjy7mipimjrgi5cnjz6nvq7rahy-gettext-0.18.2/share/locale" -DHAVE_CONFIG_H -I. -I.. -I.. -I../gnulib-lib -I../gnulib-lib -I../intl -DUSE_POSIX_THREADS -g -O2 -c -o gettext_7_prg-gettext-7-prg.o test -f 'gettext-7-prg.c' || echo './'gettext-7-prg.c
/nix/store/0fmbypn0w29r3lykg5w1glbphz1pdbkz-bash-4.3-p33/bin/bash ../libtool --tag=CC --mode=link gcc -std=gnu99 -DUSE_POSIX_THREADS -g -O2 -o gettext-7-prg gettext_7_prg-gettext-7-prg.o ../gnulib-lib/libgettextlib.la ../intl/libgnuintl.la -lpthread

malloc: dispose_cmd.c:249: assertion botched
free: called with already freed block argument
Aborting...Makefile:1810: recipe for target 'gettext-7-prg' failed
make[3]: *** [gettext-7-prg] Aborted
make[3]: Leaving directory '/tmp/nix-build-gettext-0.18.2.drv-0/gettext-0.18.2/gettext-tools/tests'
Makefile:1660: recipe for target 'all-recursive' failed
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory '/tmp/nix-build-gettext-0.18.2.drv-0/gettext-0.18.2/gettext-tools'
Makefile:1525: recipe for target 'all' failed
make[1]: *** [all] Error 2
make[1]: Leaving directory '/tmp/nix-build-gettext-0.18.2.drv-0/gettext-0.18.2/gettext-tools'
Makefile:315: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1
builder for ‘/nix/store/p0b9lzdxi598zq8i9fp4p6qciixxkazd-gettext-0.18.2.drv’ failed with exit code 2
cannot build derivation ‘/nix/store/dscyq920pg351zi72wqg1sg0cbp14qlc-git-2.3.6.drv’: 1 dependencies couldn't be built

Could you maybe try if a

docker build -t unison .

in this branch works on your machine?

Best

Martin

You can view, comment on, or merge this pull request online at:

#32
Commit Summary

Try building with Docker

File Changes

A .dockerignore
https://github.com/unisonweb/platform/pull/32/files#diff-0 (1)

A Dockerfile
https://github.com/unisonweb/platform/pull/32/files#diff-1 (4)

A docker-provision.sh
https://github.com/unisonweb/platform/pull/32/files#diff-2 (14)

Patch Links:

https://github.com/unisonweb/platform/pull/32.patch

https://github.com/unisonweb/platform/pull/32.diff

—
Reply to this email directly or view it on GitHub
#32.

nightscape · 2015-09-18T12:19:34Z

Yes, I've tried to more or less replicate the Vagrant setup.
I've managed to manually get unison running by logging into the container and starting shell.sh multiple times.
Currently I'm trying to do the same in the Docker build with shell.sh || shell.sh || shell.sh

Is it normal that there are so many things that need to be compiled? Are those packages not provided in the extra-binary-caches?
Ideally, we would configure an automated build on the official Docker registry which would allow people to always get the newest version of unison.
But they have limits wrt to build time (and probably also memory) which the package compilation definitely breaks.

pchiusano · 2015-09-18T15:37:56Z

Compilation takes 10-15 minutes I think the first time, but it's a one time
cost. Ryan's cache just has all reflex related deps and it's missing some
dependencies of the unison node I think.

I'm sure it is possible to build a nix cache that has everything we need,
but I haven't looked into that and we'd have to run a server somewhere.
We'd want the cache to have all Mac and Linux binaries for all deps used by
the editor and node.
On Fri, Sep 18, 2015 at 8:19 AM Martin Mauch notifications@github.com
wrote:

Yes, I've tried to more or less replicate the Vagrant setup.
I've managed to manually get unison running by logging into the container
and starting shell.sh multiple times.
Currently I'm trying to do the same in the Docker build with shell.sh ||
shell.sh || shell.sh

Is it normal that there are so many things that need to be compiled? Are
those packages not provided in the extra-binary-caches?
Ideally, we would configure an automated build on the official Docker
registry which would allow people to always get the newest version of
unison.
But they have limits wrt to build time (and probably also memory) which
the package compilation definitely breaks.

—
Reply to this email directly or view it on GitHub
#32 (comment).

Ericson2314 · 2015-09-19T22:52:44Z

commercialhaskell/stack#337
commercialhaskell/stack#749

Those describe some work getting GHCJS buildable with stack. Hopefully this means we could use a more standard set of packages than we do now, and thereby get by with the official cache.

pchiusano · 2016-01-28T17:09:21Z

@nightscape if you are still interested, you could take another look at this now that the build has been converted over to Stack (see #34) and does not use Nix anymore. It might be that this is now a lot simpler. This page in the Stack guide might be useful, not sure.

nightscape · 2016-01-30T20:40:34Z

I'm still interested, although maybe not the best person to do it due to lack of any Cabal/Stack knowledge. I've asked in the repo of the official Haskell Docker image if they can provide an example on how to best build Stack based projects. If they comment on that I'll try and apply it to Unison.

pchiusano · 2016-01-31T00:51:33Z

Cool.

On Sat, Jan 30, 2016 at 3:40 PM Martin Mauch notifications@github.com
wrote:

I'm still interested, although maybe not the best person to do it due to
lack of any Cabal/Stack knowledge. I've asked in the repo of the official
Haskell Docker image if they can provide an example on how to best build
Stack based projects. If they comment on that I'll try and apply it to
Unison.

—
Reply to this email directly or view it on GitHub
#32 (comment).

nightscape · 2016-02-03T00:59:01Z

@pchiusano Do you happen to know if one can tell stack to just download and compile the dependencies, not the code itself?
This is essential for the Docker build to get to an acceptable speed, otherwise every single change in any file will bust the Docker cache and require starting all over with the dependencies...

pchiusano · 2016-02-03T01:22:20Z

Try stack build --only-dependencies

On Tue, Feb 2, 2016 at 7:59 PM Martin Mauch notifications@github.com
wrote:

@pchiusano https://github.com/pchiusano Do you happen to know if one
can tell stack to just download and compile the dependencies, not the code
itself?
This is essential for the Docker build to get to an acceptable speed,
otherwise every single change in any file will bust the Docker cache and
require starting all over with the dependencies...

—
Reply to this email directly or view it on GitHub
#32 (comment).

nightscape · 2016-02-05T09:25:58Z

Ok, I've got everything built inside Docker. It takes ages the first time, but it should be quite fast after that until you need to touch the .cabal or (even worse) the stack.yaml files ;)

Unfortunately, the generated editor.html assumes that one can access the compiled .js files via a path on disk which won't work with Docker. Could you change that so that the .js files get served by Scotty?
I think once that is done, we should have a working Docker build 👍

nightscape · 2016-02-05T10:06:14Z

Forgot to mention: The editor.html file itself also needs to be served by Scotty (or some other way via TCP port).

nightscape · 2016-02-05T17:03:47Z

Ok, almost got it :D

nightscape · 2016-02-05T17:31:32Z

And here we go! I'm just uploading my locally built image to docker hub, once the upload is finished (it's a hefty 5.6 GB) you can try it using the following command

docker run -it -p 8080:8080 --name nightscape/unison --rm unison

and then browsing to http://localhost:8080/.

Once this PR is merged you (or I) could create an automated build on Docker Hub so that people can always get the latest version (you can also select to build only dedicated branches).

I just hope that the automated build is not detected as DoS and killed 😁

These are the limits to the automated build:

2 hours maximum build time
2 GB RAM
1 CPU
30 GB Disk Space

pchiusano · 2016-02-05T18:30:24Z

Hey, let's just get rid of the make-editor-html script now, the only reason for that was to lookup those system dependent paths. Instead let's just check in the editor.html file that refers to localhost in the path. Also please update the README accordingly.

using wai-middleware-static

Also remove obsolete make-editor-html part of build instructions

nightscape · 2016-02-05T19:31:45Z

Ok, got rid of the make-editor-script and adapted the README.
Any preferences on how the Docker image should be called? I used unisonweb/platform like the Github project in the README, but I can change it if you prefer another name.
It would be good to use the same name when building yourself and when pulling the official image from Docker Hub.

pchiusano · 2016-02-05T20:30:41Z

I would just call it 'unison' if that is available, otherwise 'unisonweb'. The 'platform' is not necessary. The GitHub org uses that just to distinguish from the unisonweb.org site, which is also under the unisonweb GitHub org.

nightscape · 2016-02-05T20:36:02Z

You have to give it a organization/name tag though. The name images are reserved for "official" Docker images, i.e. the ones listed here.

nightscape · 2016-02-05T20:37:35Z

And I'm not a 100% sure if the name of an automated build can deviate from the Github repo name, checking that now...

Try building with Docker

nightscape force-pushed the dockerize branch from f8922e5 to c4b88f9 Compare February 5, 2016 08:33

Add Docker build

e22a363

nightscape force-pushed the dockerize branch from c4b88f9 to 71bd413 Compare February 5, 2016 17:12

nightscape added 2 commits February 5, 2016 20:26

Serve static files through Scotty

6b44a24

using wai-middleware-static

Update README with Docker build instructions

f0a5ef2

Also remove obsolete make-editor-html part of build instructions

nightscape force-pushed the dockerize branch from 71bd413 to f0a5ef2 Compare February 5, 2016 19:28

pchiusano added a commit that referenced this pull request Feb 5, 2016

Merge pull request #32 from nightscape/dockerize

c3bbd63

Try building with Docker

pchiusano merged commit c3bbd63 into unisonweb:master Feb 5, 2016

nightscape deleted the dockerize branch February 5, 2016 21:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Try building with Docker #32

Try building with Docker #32

nightscape commented Sep 17, 2015

pchiusano commented Sep 17, 2015

Martin

nightscape commented Sep 18, 2015

pchiusano commented Sep 18, 2015

Ericson2314 commented Sep 19, 2015

pchiusano commented Jan 28, 2016

nightscape commented Jan 30, 2016

pchiusano commented Jan 31, 2016

nightscape commented Feb 3, 2016

pchiusano commented Feb 3, 2016

nightscape commented Feb 5, 2016

nightscape commented Feb 5, 2016

nightscape commented Feb 5, 2016

nightscape commented Feb 5, 2016

pchiusano commented Feb 5, 2016

nightscape commented Feb 5, 2016

pchiusano commented Feb 5, 2016

nightscape commented Feb 5, 2016

nightscape commented Feb 5, 2016

Try building with Docker #32

Try building with Docker #32

Conversation

nightscape commented Sep 17, 2015

pchiusano commented Sep 17, 2015

Martin

nightscape commented Sep 18, 2015

pchiusano commented Sep 18, 2015

Ericson2314 commented Sep 19, 2015

pchiusano commented Jan 28, 2016

nightscape commented Jan 30, 2016

pchiusano commented Jan 31, 2016

nightscape commented Feb 3, 2016

pchiusano commented Feb 3, 2016

nightscape commented Feb 5, 2016

nightscape commented Feb 5, 2016

nightscape commented Feb 5, 2016

nightscape commented Feb 5, 2016

pchiusano commented Feb 5, 2016

nightscape commented Feb 5, 2016

pchiusano commented Feb 5, 2016

nightscape commented Feb 5, 2016

nightscape commented Feb 5, 2016