Support reproducible builds via Nix #1285

YPares · 2015-11-04T15:34:31Z

This pull request implements the capability for stack to launch the build inside a nix-shell, in order to specify dependencies to libraries out of Hackage that exist in Nixpkgs.

Simply modify your stack.yaml so it contains for instance:

nix:
    enable: false
    packages: [glpk]
    #shell-file: shell.nix

Build this project with stack --nix build. It will open a Nix-shell that will first download the glpk dependency, and then launch the build. Of course, this requires nix to be installed on your system, and it requires your stack.yaml to depend on an LTS resolver that has been mirrored in the nix packages (see for instance in https://github.com/NixOS/nixpkgs/tree/master/pkgs/development/haskell-modules the configuration-lts-X.Y.nix files).
If you know nix, you can also specify a shell.nix that creates a derivation that will be used to set the build environment. You cannot specify packages and shell-file at the same time!
For now, this works much like the docker backend, by relauching the same stack executable inside a new environment. It requires GHC to be installed by Nix.

It's not possible to use stack ghci, because GHCi fails to find the extra libraries, but this is not specific to stack, as the command:
nix-shell --pure -p ghc extra-libs --command "stack ghci"
without enabling nix support in stack already fails.

Theoretically, activating both Nix and Docker could be possible (resulting in a nix-shell running inside a container), but that requires a Docker image providing Nix, of course.

Don't dispatch on the ExecEnv type. Using Docker and nix is no longer mutually exclusive. We run commands in a docker container if configured, and then fork a nix-shell inside if configured.

same time

and reports the use of nix-provided GHC

Using PackageNames forbid to use '.' in names, which is required when you use attributes

borsboom · 2015-11-26T22:49:18Z

src/Stack/Nix.hs

+            Just filePath -> "from file: " ++ filePath
+            Nothing -> "with nix packages: " ++ (intercalate ", " pkgsInConfig))
+     e <- try (callProcess'
+                 (if isTerm then id else \cp -> cp { delegate_ctlc = False })


I doubt the extra logic to prevent delegating Ctrl-C is necessary in this case. For Docker, it's there because of some specific weird stuff that docker run and docker start do with container terminals, but this nix-shell looks like a regular process that uses the existing std* file handles and TTY normally.

Fixed along with previous

borsboom · 2015-11-26T22:56:53Z

When I tried this with a resolver that hadn't been mirrored to nix yet, I got the following error:

error: attribute ‘lts-3_14’ missing, at (string):1:163
(use ‘--show-trace’ to show detailed location information)

After reading the docs I understood what this meant, but is it possible to make this more user-friendly? It would be nice if the error at least indicates why this would happen and perhaps suggests running nix-channel --update (if that makes sense). Perhaps it could even run the update automatically and retry, kind of like how Stack will download a snapshot automatically the first time it's encountered (not sure if Nix users would like automatic update though; I defer to your judgement about whether this is a good idea).

borsboom · 2015-11-26T23:01:17Z

Thanks for all the work on this, it looks really close to being ready to merge! Don't forget to add a changelog entry. It should go in the major changes section; this deserves lots of attention!

YPares · 2015-11-27T17:01:33Z

@borsboom I pushed a new commit addressing your comments.

Although, regarding this Nix error about the missing attribute, to provide a more informative error message I would need to ask nix about which attributes are available, which is not easy to automate as I pointed out in the manual.

The real fix would be to stop to depend on a GHC provided by Nix and use a GHC downloaded by Stack, but for now on Linux this results in failure to find libs in the nix store.

borsboom · 2015-11-27T19:15:28Z

Fair enough, I don't think the error message should hold up merging this, although it would be nice to improve at some point. I'm not surprised that trying to use a non-Nix GHC causes problems, so leaving that as-is is reasonable.

Do you think this may end up with similar problems as Docker in #911 and #1367? I'm beginning work on a fix for those two now, and most likely the same approach can be used for Nix as well.

borsboom · 2015-11-27T19:26:56Z

src/Stack/Nix.hs

+            Just filePath -> "from file: " <> (T.pack filePath)
+            Nothing -> "with nix packages: " <> (T.intercalate ", " pkgsInConfig))
+     e <- try (exec
+                 (EnvSettings {esIncludeLocals = False


You can use Stack.Config.Types.minimalEnvSettings instead.

YPares · 2015-12-01T12:00:54Z

@borsboom I corrected the escaping of the command line and re-merged the master.
Also, for coherence's sake, I made a slight adjustment: the section in the yaml is just called nix: now (no longer nix-shell:).
That's because the CL option was already --nix and because soon nix will adopt a git-like CL interface. I.e. the nix-shell tool will become just another subcommand of a big nix command.

Support reproducible builds via Nix

borsboom · 2015-12-03T00:11:41Z

LGTM, thanks for this awesome new feature!

mgsloan · 2015-12-03T03:19:38Z

Awesome :D

I think this warrants a Major changes entry in the ChangeLog

YPares · 2015-12-03T08:47:00Z

@mgsloan @borsboom Thanks guys!

YPares · 2015-12-03T11:12:29Z

I will publish a blog post about this feature shortly.

codygman · 2015-12-13T19:50:42Z

@YPares looking forward to it, will you post it back here?

borsboom · 2015-12-13T20:03:26Z

http://www.tweag.io/blog/stack-nix-portable-reproducible-builds

YPares and others added 30 commits November 4, 2015 16:00

Added config parsing for nix-shell execution environment

99f7fa8

Nix-shell: added option for initial env expr file

3dac5d9

Adding branching in main (docker or nix)

2df0c24

Stack launches a nix-shell

162d807

Some opts of nix-shell were forgotten

97b1f9b

Hack: fixing stack version in arg and nix-shell no longer pure

cee6f0b

Using ghc from nixpkgs.haskell.packages.lts-X_Y when possible

6fbce23

Cleaning up the NixShell file

a36c7f9

Remove redundant LANGUAGE pragmas.

bdb32f7

Remove trailing whitespace.

ffea96d

Rename ExecEnv to Nix.

a8e9b7f

Remove dead code.

9e8a2dc

Remove ExecEnv abstraction. Allow docker + nix simultaneously.

2809ff9

Don't dispatch on the ExecEnv type. Using Docker and nix is no longer mutually exclusive. We run commands in a docker container if configured, and then fork a nix-shell inside if configured.

Remove redundant imports and arguments.

e33a65b

Nix: delegate ctrlc if in terminal.

68480b3

twiddle: intercalate rather than concat . intersperse.

dd5ee9c

Use $logDebug.

95c24b8

Remove redundant extra-dep.

a2930f9

Nix options (--nix & --no-nix) and help added to command line

292bd3d

Added --nix-shell-options to be passed to nix-shell

086e9d3

--nix* option group showing in stack --help output

68ef716

Nix can read a premade shell.nix file

210772b

Throwing an exception when trying to use shell file and packages at the

2554209

same time

Sharing reExecArgName with Docker config

b4689a4

Fixing a comment

f25fcad

When nix enabled, stack setup downloads ghc and system deps through nix

c44aaf9

and reports the use of nix-provided GHC

Added some tests for nix-shell configuration

a2c417c

Using Strings instead of PackageName for nix packages

b0213f9

Using PackageNames forbid to use '.' in names, which is required when you use attributes

Trying to fix the LD_LIBRARY_PATH problem

5ef1584

Added default dep to glibcLocales when lauching nix-shell

c3233ec

borsboom reviewed Nov 26, 2015
View reviewed changes

YPares added 2 commits November 27, 2015 17:44

Simplifying src/Stack/Nix.hs and using the right CL parsing combinators

8acfb7a

Merge remote-tracking branch 'fpco.r/master' into nix-merged

e0f0994

YPares added 3 commits November 27, 2015 18:14

Changelog updated

ae7e0ff

Stack/Nix: shell subcommand args put between simple quotes

ae9d301

Fixed OverloadedStrings missing in tests

95051af

borsboom reviewed Nov 27, 2015
View reviewed changes

YPares added 3 commits December 1, 2015 12:31

Merge remote-tracking branch 'fpco.r/master' into nix-merged

d1279e5

Nix: Escaping quote characters before re-issuing the stack cmd

c2a1135

nix-shell: section in stack.yaml renamed to nix: for coherence with CLI

6be2541

mboes mentioned this pull request Dec 2, 2015

stack install hmatrix is ugly #1463

Closed

Testing validity of nix: config moved to Config/Nix.hs

3554309

borsboom added a commit that referenced this pull request Dec 3, 2015

Merge pull request #1285 from tweag/nix-merged

931cb02

Support reproducible builds via Nix

borsboom merged commit 931cb02 into commercialhaskell:master Dec 3, 2015

mboes deleted the nix-merged branch December 13, 2015 20:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support reproducible builds via Nix #1285

Support reproducible builds via Nix #1285

YPares commented Nov 4, 2015

borsboom Nov 26, 2015

YPares Nov 27, 2015

borsboom commented Nov 26, 2015

borsboom commented Nov 26, 2015

YPares commented Nov 27, 2015

borsboom commented Nov 27, 2015

borsboom Nov 27, 2015

YPares commented Dec 1, 2015

borsboom commented Dec 3, 2015

mgsloan commented Dec 3, 2015

YPares commented Dec 3, 2015

YPares commented Dec 3, 2015

codygman commented Dec 13, 2015

borsboom commented Dec 13, 2015

Support reproducible builds via Nix #1285

Support reproducible builds via Nix #1285

Conversation

YPares commented Nov 4, 2015

borsboom Nov 26, 2015

Choose a reason for hiding this comment

YPares Nov 27, 2015

Choose a reason for hiding this comment

borsboom commented Nov 26, 2015

borsboom commented Nov 26, 2015

YPares commented Nov 27, 2015

borsboom commented Nov 27, 2015

borsboom Nov 27, 2015

Choose a reason for hiding this comment

YPares commented Dec 1, 2015

borsboom commented Dec 3, 2015

mgsloan commented Dec 3, 2015

YPares commented Dec 3, 2015

YPares commented Dec 3, 2015

codygman commented Dec 13, 2015

borsboom commented Dec 13, 2015