Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support reproducible builds via Nix #1285

Merged
merged 60 commits into from
Dec 3, 2015
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
99f7fa8
Added config parsing for nix-shell execution environment
YPares Oct 22, 2015
3dac5d9
Nix-shell: added option for initial env expr file
YPares Oct 22, 2015
2df0c24
Adding branching in main (docker or nix)
YPares Oct 23, 2015
162d807
Stack launches a nix-shell
YPares Oct 23, 2015
97b1f9b
Some opts of nix-shell were forgotten
YPares Oct 23, 2015
cee6f0b
Hack: fixing stack version in arg and nix-shell no longer pure
YPares Oct 23, 2015
6fbce23
Using ghc from nixpkgs.haskell.packages.lts-X_Y when possible
YPares Oct 23, 2015
a36c7f9
Cleaning up the NixShell file
YPares Oct 23, 2015
bdb32f7
Remove redundant LANGUAGE pragmas.
mboes Oct 24, 2015
ffea96d
Remove trailing whitespace.
mboes Oct 24, 2015
a8e9b7f
Rename ExecEnv to Nix.
mboes Oct 25, 2015
9e8a2dc
Remove dead code.
mboes Oct 25, 2015
2809ff9
Remove ExecEnv abstraction. Allow docker + nix simultaneously.
mboes Oct 25, 2015
e33a65b
Remove redundant imports and arguments.
mboes Oct 25, 2015
68480b3
Nix: delegate ctrlc if in terminal.
mboes Oct 25, 2015
dd5ee9c
twiddle: intercalate rather than concat . intersperse.
mboes Oct 25, 2015
95c24b8
Use $logDebug.
mboes Oct 25, 2015
a2930f9
Remove redundant extra-dep.
mboes Oct 25, 2015
292bd3d
Nix options (--nix & --no-nix) and help added to command line
YPares Oct 26, 2015
086e9d3
Added --nix-shell-options to be passed to nix-shell
YPares Oct 26, 2015
68ef716
--nix* option group showing in stack --help output
YPares Oct 26, 2015
210772b
Nix can read a premade shell.nix file
YPares Oct 26, 2015
2554209
Throwing an exception when trying to use shell file and packages at the
YPares Oct 26, 2015
b4689a4
Sharing reExecArgName with Docker config
YPares Oct 26, 2015
f25fcad
Fixing a comment
YPares Oct 26, 2015
c44aaf9
When nix enabled, stack setup downloads ghc and system deps through nix
YPares Oct 28, 2015
a2c417c
Added some tests for nix-shell configuration
YPares Oct 28, 2015
b0213f9
Using Strings instead of PackageName for nix packages
YPares Oct 28, 2015
5ef1584
Trying to fix the LD_LIBRARY_PATH problem
YPares Oct 28, 2015
c3233ec
Added default dep to glibcLocales when lauching nix-shell
YPares Nov 4, 2015
8913e66
Added a comment about the LD_LIBRARY_PATH hack
YPares Nov 4, 2015
6a51e45
Following the convention RE option names
YPares Nov 4, 2015
83d8779
Added DeriveDataTypeable for Nix exception type derivation
YPares Nov 4, 2015
fa2991d
Passing libs and includes zith --extra-*-dirs to stack: beginning
YPares Nov 6, 2015
969fa35
Building with --extra-lib-dirs
YPares Nov 12, 2015
6f1ed7d
Ok on OSX BUT in a non-pure shell. Investigating...
YPares Nov 12, 2015
103f926
No exception
YPares Nov 12, 2015
8d3a004
Stack/Nix doesn't need BuildConfig anymore
YPares Nov 13, 2015
da3c61e
Removed shell string to export LD_LIBRARY_PATH
YPares Nov 13, 2015
9c84315
glibcLocales passed as buildInput only on Linux
YPares Nov 13, 2015
7c876f5
darwin.cf-private added as a dep on OSX
YPares Nov 13, 2015
3d9e6d3
Merge remote-tracking branch 'fpco.r/master' into nix-merged
YPares Nov 18, 2015
51498b7
Updated Nix test due to change of behaviour
YPares Nov 18, 2015
1b6b032
Fixing warning
YPares Nov 18, 2015
c7b7e29
Builds with --pedantic
YPares Nov 18, 2015
86d2204
Building with --pedantic with GHC 7.8.4
YPares Nov 18, 2015
44a2241
Removed warning with an import Prelude
YPares Nov 18, 2015
a6e51d0
Nix integration documentation added to manual
YPares Nov 25, 2015
1129897
Merge remote-tracking branch 'origin/master' into nix-merged
mboes Nov 25, 2015
dcd10f9
Edit Nix section of the guide.
mboes Nov 25, 2015
29135b3
Mention stack nix standard layout condition in manual.
mboes Nov 25, 2015
8acfb7a
Simplifying src/Stack/Nix.hs and using the right CL parsing combinators
YPares Nov 27, 2015
e0f0994
Merge remote-tracking branch 'fpco.r/master' into nix-merged
YPares Nov 27, 2015
ae7e0ff
Changelog updated
YPares Nov 27, 2015
ae9d301
Stack/Nix: shell subcommand args put between simple quotes
YPares Nov 27, 2015
95051af
Fixed OverloadedStrings missing in tests
YPares Nov 27, 2015
d1279e5
Merge remote-tracking branch 'fpco.r/master' into nix-merged
YPares Dec 1, 2015
c2a1135
Nix: Escaping quote characters before re-issuing the stack cmd
YPares Dec 1, 2015
6be2541
nix-shell: section in stack.yaml renamed to nix: for coherence with CLI
YPares Dec 1, 2015
3554309
Testing validity of nix: config moved to Config/Nix.hs
YPares Dec 2, 2015
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions ChangeLog.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@

Major changes:

* Support for building inside a Nix-shell providing system dependencies
[#1285](https://github.com/commercialhaskell/stack/pull/1285)

Other enhancements:

* Print latest applicable version of packages on conflicts
Expand Down
30 changes: 30 additions & 0 deletions doc/GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -1688,6 +1688,36 @@ image:
and then run `stack image container` and then `docker images` to list
the images.

### Nix

stack provides an integration with [Nix](http://nixos.org/nix),
providing you with the same two benefits as the first Docker
integration discussed above:

* more reproducible builds, since fixed versions of any system
libraries and commands required to build the project are
automatically built using Nix and managed locally per-project. These
system packages never conflict with any existing versions of these
libraries on your system. That they are managed locally to the
project means that you don't need to alter your system in any way to
build any odd project pulled from the Internet.
* implicit sharing of system packages between projects, so you don't
have more copies on-disk than you need to.

Both Docker and Nix are methods to *isolate* builds and thereby make
them more reproducible. They just differ in the means of achieving
this isolation. Nix provides slightly weaker isolation guarantees than
Docker, but is more lightweight and more portable (Linux and OS
X mainly, but also Windows). For more on Nix, its command-line
interface and its package description language, read the
[Nix manual](http://nixos.org/nix/manual). But keep in mind that the
point of stack's support is to obviate the need to write any Nix code
in the common case or even to learn how to use the Nix tools (they're
called under the hood).

For more information, see
[the Nix-integration documentation](nix_integration.md).

## Power user commands

The following commands are a little more powerful, and won't be needed by all
Expand Down
162 changes: 162 additions & 0 deletions doc/nix_integration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
# Using Nix with Stack

`stack` can build automatically inside a nix-shell (the equivalent of
a "container" in Docker parlance), provided Nix is already installed
on your system. To do so, please visit the
[Nix download page](http://nixos.org/nix/download.html).

There are two ways to create a nix-shell:

- providing a list of packages (by "attribute name") from
[Nixpkgs](http://nixos.org/nixos/packages.html), or
- providing a custom `shell.nix` file containing a Nix expression that
determines a *derivation*, i.e. a specification of what resources
are available inside the shell.

The second requires writing code in Nix's custom language. So use this
option only if you already know Nix and have special requirements,
such as using custom Nix packages that override the standard ones or
using system libraries with special requirements.

### Additions to your `stack.yaml`

Add a section to your `stack.yaml` as follows:

nix:
enable: true
packages: [glpk, pcre]

This will instruct `stack` to build inside a nix-shell that will have
the `glpk` and `pcre` libraries installed and available. Further, the
nix-shell will implicitly also include a version of GHC matching the
configured resolver. Enabling Nix support means packages will always
be built using a GHC available inside the shell, rather than your
globally installed one if any.

Note also that this also means that you cannot set your `resolver:` to
something that has not yet been mirrored in the Nixpkgs package
repository. In order to check this, the quickest way is to install and
launch a `nix-repl`:

```
$ nix-channel --update
$ nix-env -i nix-repl
$ nix-repl
```

Then, inside the `nix-repl`, do:

```
nix-repl> :l <nixpkgs>
nix-repl> haskell.packages.lts-3_13.ghc
```

Replace the resolver version with whatever version you are using. If it outputs
a path of a derivation in the Nix store, like

`«derivation /nix/store/00xx8y0p3r0dqyq2frq277yr1ldqzzg0-ghc-7.10.2.drv»`

then it means that this resolver has been mirrored and exists in your local copy of the nixpkgs. Whereas an error like

`error: attribute ‘lts-3_99’ missing, at (string):1:1`

means you should use a different resolver. You can also use
autocompletion with TAB to know which attributes `haskell.packages`
contains.

In Nixpkgs master branch, you can find the mirrored resolvers in the
Haskell modules
[here on Github](https://github.com/NixOS/nixpkgs/tree/master/pkgs/development/haskell-modules).

*Note:* currently, stack only discovers dynamic and static libraries
in the `lib/` folder of any nix package, and likewise header files in
the `include/` folder. If you're dealing with a package that doesn't
follow this standard layout, you'll have to deal with that using
a custom shell file (see below).

### Use stack as normal

With Nix enabled, `stack build` and `stack exec` will automatically
launch themselves in a nix-shell. Note that for now `stack ghci` will
not work, due to a bug in GHCi when working with external shared
libraries.

If `enable:` is set to `false`, you can still build in a nix-shell by
passing the `--nix` flag to stack, for instance `stack --nix build`.
Nix just won't be used by default.

## Command-line options

The configuration present in your `stack.yaml` can be overriden on the
command-line. See `stack --nix-help` for a list of all Nix options.


## Configuration

`stack.yaml` contains a `nix:` section with Nix settings.
Without this section, Nix will not be used.

Here is a commented configuration file, showing the default values:

nix:

# `true` by default when the nix section is present. Set
# it to `false` to disable using Nix.
enable: true

# Empty by default. The list of packages you want to be
# available in the nix-shell at build time (with `stack
# build`) and run time (with `stack exec`).
packages: []

# Unset by default. You cannot set this option if `packages:`
# is already present and not empty, this will result in an
# exception
shell-file: shell.nix

# A list of strings, empty by default. Additional options that
# will be passed verbatim to the `nix-shell` command.
nix-shell-options: []

## Using a custom shell.nix file

Nix is also a programming language, and as specified
[here](#using-nix-with-stack) if you know it you can provide to the
shell a fully customized derivation as an environment to use. Here is
the equivalent of the configuration used in
[this section](#enable-in-stackyaml), but with an explicit `shell.nix`
file:

```
with (import <nixpkgs> {});
stdenv.mkDerivation {
name = "myEnv";
buildInputs = [glpk pcre haskell.packages.lts-3_13.ghc];
STACK_IN_NIX_EXTRA_ARGS="--extra-lib-dirs=${glpk}/lib --extra-include-dirs=${glpk}/include --extra-lib-dirs=${pcre}/lib --extra-include-dirs=${pcre}/include";
}
```

Note that in this case, you _have_ to include (a version of) GHC in
your `buildInputs`! This potentially allows you to use a GHC which is
not the one of your `resolver:`. Also, you need to tell Stack where to
find the new libraries and headers. This is especially necessary on OS
X. The special variable `STACK_IN_NIX_EXTRA_ARGS` will be looked for
by the nix-shell when running the inner `stack` process.
`--extra-lib-dirs` and `--extra-include-dirs` are regular `stack
build` options. You can repeat these options for each dependency.

Defining manually a `shell.nix` file gives you the possibility to
override some Nix derivations ("packages"), for instance to change
some build options of the libraries you use.

And now for the `stack.yaml` file:

```
nix:
enable: true
shell-file: shell.nix
```

The `stack build` command will behave exactly the same as above. Note
that specifying both `packages:` and a `shell-file:` results in an
error. (Comment one out before adding the other.)
2 changes: 2 additions & 0 deletions src/Stack/Config.hs
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ import qualified Paths_stack as Meta
import Safe (headMay)
import Stack.BuildPlan
import Stack.Config.Docker
import Stack.Config.Nix
import Stack.Constants
import qualified Stack.Image as Image
import Stack.Init
Expand Down Expand Up @@ -150,6 +151,7 @@ configFromConfigMonoid configStackRoot configUserConfigPath mresolver mproject c

configDocker <-
dockerOptsFromMonoid (fmap fst mproject) configStackRoot mresolver configMonoidDockerOpts
configNix <- nixOptsFromMonoid (fmap fst mproject) configStackRoot configMonoidNixOpts

rawEnv <- liftIO getEnvironment
origEnv <- mkEnvOverride configPlatform
Expand Down
45 changes: 45 additions & 0 deletions src/Stack/Config/Nix.hs
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
{-# LANGUAGE RecordWildCards, DeriveDataTypeable #-}

-- | Nix configuration
module Stack.Config.Nix
(nixOptsFromMonoid
,StackNixException(..)
) where

import Data.Text (pack)
import Data.Maybe
import Data.Typeable
import Path
import Stack.Types
import Control.Exception.Lifted
import Control.Monad.Catch (throwM,MonadCatch)


-- | Interprets NixOptsMonoid options.
nixOptsFromMonoid :: (Monad m, MonadCatch m) => Maybe Project -> Path Abs Dir -> NixOptsMonoid -> m NixOpts
nixOptsFromMonoid mproject _stackRoot NixOptsMonoid{..} = do
let nixEnable = fromMaybe nixMonoidDefaultEnable nixMonoidEnable
nixPackages = case mproject of
Nothing -> nixMonoidPackages
Just p -> nixMonoidPackages ++ [case projectResolver p of
ResolverSnapshot (LTS x y) ->
pack ("haskell.packages.lts-" ++ show x ++ "_" ++ show y ++ ".ghc")
_ -> pack "ghc"]
nixInitFile = nixMonoidInitFile
nixShellOptions = nixMonoidShellOptions
if not (null nixMonoidPackages) && isJust nixInitFile then
throwM NixCannotUseShellFileAndPackagesException
else return ()
return NixOpts{..}

-- Exceptions thown specifically by Stack.Nix
data StackNixException
= NixCannotUseShellFileAndPackagesException
-- ^ Nix can't be given packages and a shell file at the same time
deriving (Typeable)

instance Exception StackNixException

instance Show StackNixException where
show NixCannotUseShellFileAndPackagesException =
"You cannot have packages and a shell-file filled at the same time in your nix-shell configuration."
1 change: 1 addition & 0 deletions src/Stack/Docker.hs
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ module Stack.Docker
,reexecWithOptionalContainer
,reset
,reExecArgName
,StackDockerException(..)
) where

import Control.Applicative
Expand Down
Loading