Overriding (Haskell) packages repeatedly is surprising and non-modular

[ocharles]

Issue description

The .override attribute doesn't allow you to repeatedly override the set of Haskell packages. This means it's impossible to compose multiple overlays that change the Haskell package set.

Steps to reproduce

The following Nix expression fails to evaluate:

let hs-0 = (import <nixpkgs> {}).haskellPackages;
    hs-1 = hs-0.override { overrides = self: super: { x = 42; }; };
    hs-2 = hs-1.override { overrides = self: super: { y = super.x; }; };
in hs-2.y

error: attribute ‘x’ missing, at (string):3:59

But this is surpring, because:

let hs-0 = (import <nixpkgs> {}).haskellPackages;
    hs-1 = hs-0.override { overrides = self: super: { x = 42; }; };
in hs-1.x

evaluates to 42 as expected. The problem is overrides only looks at the initial result. Further overrides does not update what overrides begins with.

This issue is probably not limited to Haskell packaging.

Importance

The new overlays system means that one would expect it to be possible to combine multiple Haskell overlays. I wanted to have an overlay that contained some build fixes & unreleased (but public) Haskell libraries, and another overlay that contained company internal projects. This isn't possible, because as shown above - these overlays don't compose.

[ljli]

You can do that with extend.

let hs-0 = (import <nixpkgs> {}).haskellPackages;
    hs-1 = hs-0.extend (self: super: { x = 42; });
    hs-2 = hs-1.extend (self: super: { y = super.x; });
in hs-2.y

It is not surprising if it is clear what override does and where overrides comes from. override overrides function arguments. But it doesn't give you the previous set of arguments as as an argument like e.g. overrideAttrs does (not that you could use that instead). So you can only keep an argument in place or replace it completely. If override would provide you with the previous set of arguments, this would work:

hs-0.override (oldArgs: { overrides =
  composeExtensions (self: super: { y = super.x })
    (oldArgs.overrides or (self: super: {});
})

Theoretically override could work behind the scenes to consume and accumulate the overrides arguments. But despite the similar names those are two independent mechanisms. The overrides argument is haskell specific, it is declared in haskell-modules/default.nix. The override attribute is the result of wrapping a function call into callPackage which in turn wraps it into makeOverrideable. That happens in all-packages.nix.
To complete the saga, the extend attribute I suggested exists because all haskell package sets are wrapped with makeExtensible also in haskell-modules/default.nix.

EDIT:
override works with functions as well as attribute sets, so what is said above, that it couldn't give the old arguments, is false. The example in question could be written like this:

let lib = (import <nixpkgs> {}).lib;
    hs-0 = (import <nixpkgs> {}).haskellPackages;
    hs-2 =
      (hs-0.override (oldArgs: { overrides =
        lib.composeExtensions
          (lib.composeExtensions
            (oldArgs.overrides or (self: super: {}))
            (self: super: { x = 42; }))
          (self: super: { y = super.x; });}));
in hs-2.y

This even seems to be the recommended way to do it, please see the discussion linked further down the thread.

[ocharles]

Thanks, I wasn't even aware of extend!

[srhb]

Is every example in the documentation rewritable with extend? Since the introduction of overlays, this has only become more confusing, since an overlay in the overrides style will entirely nuke nixpkgs packageOverrides, as far as I can tell. If this is not the case with extend, it seems a better choice. Thoughts?

[ElvishJerricco]

FWIW, this was a duplicate of #26561. See there for discussion on how to work around this with override, and how extend needs to be fixed.

[bgamari]

Is extend mentioned anywhere in the nixpkgs documentation? It seems its existence really needs to be better documented and advertised.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/help-overriding-a-derivations-runtime-haskell-dependencies-reproducible-pinned-nix-file-included/8184/7