Skip to content

Commit

Permalink
Create outputOf primop.
Browse files Browse the repository at this point in the history
In the Nix language, given a drv path, we should be able to construct
another string referencing to one of its output. We can do this today
with `(import drvPath).output`, but this only works for derivations we
already have.

With dynamic derivations, however, that doesn't work well because the
`drvPath` isn't yet built: importing it like would need to trigger IFD,
when the whole point of this feature is to do "dynamic build graph"
without IFD!

Instead, what we want to do is create a placeholder value with the right
string context to refer to the output of the as-yet unbuilt derivation.
A new primop in the language, analogous to `builtins.placeholder` can be
used to create one. This will achieve all the right properties. The
placeholder machinery also will match out the `outPath` attribute for CA
derivations works.

In 60b7121 we added that type of
placeholder, and the derived path and string holder changes necessary to
support it. Then in the previous commit we cleaned up the code
(inspiration finally hit me!) to deduplicate the code and expose exactly
what we need. Now, we can wire up the primop trivally!

Part of RFC 92: dynamic derivations (tracking issue #6316)

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
  • Loading branch information
Ericson2314 and roberth committed Aug 14, 2023
1 parent e7c39ff commit 11c9d8b
Show file tree
Hide file tree
Showing 6 changed files with 140 additions and 4 deletions.
39 changes: 39 additions & 0 deletions src/libexpr/primops.cc
Original file line number Diff line number Diff line change
Expand Up @@ -1838,6 +1838,45 @@ static RegisterPrimOp primop_readDir({
.fun = prim_readDir,
});

/* Extend single element string context with another output. */
static void prim_outputOf(EvalState & state, const PosIdx pos, Value * * args, Value & v)
{
SingleDerivedPath drvPath = state.coerceToSingleDerivedPath(pos, *args[0], "while evaluating the first argument to builtins.outputOf");

std::string_view outputName = state.forceStringNoCtx(*args[1], pos, "while evaluating the second argument to builtins.outputOf");

state.mkSingleDerivedPathString(
SingleDerivedPath::Built {
.drvPath = make_ref<SingleDerivedPath>(drvPath),
.output = std::string { outputName },
},
v);
}

static RegisterPrimOp primop_outputOf({
.name = "__outputOf",
.args = {"derivation reference", "output name"},
.doc = R"(
Return the output path of a derivation, literally or using a placeholder if needed.
If the derivation has a statically-known output path (i.e. the derivation output is input-addressed, or fixed content-addresed), the output path will just be returned.
But if the derivation is content-addressed or if the derivation is itself not-statically produced (i.e. is the output of another derivation), a placeholder will be returned instead.
*`derivation reference`* must be a string that may contain a regular store path to a derivation, or may be a placeholder reference. If the derivation is produced by a derivation, you must explicitly select `drv.outPath`.
This primop can be chained arbitrarily deeply.
For instance,
```nix
builtins.outputOf
(builtins.outputOf myDrv "out)
"out"
```
will return a placeholder for the output of the output of `myDrv`.
This primop corresponds to the `^` sigil for derivable paths, e.g. as part of installable syntax on the command line.
)",
.fun = prim_outputOf,
.experimentalFeature = Xp::DynamicDerivations,
});

/*************************************************************
* Creating files
Expand Down
9 changes: 9 additions & 0 deletions tests/dyn-drv/dep-built-drv.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
#!/usr/bin/env bash

source common.sh

out1=$(nix-build ./text-hashed-output.nix -A hello --no-out-link)

clearStore

expectStderr 1 nix-build ./text-hashed-output.nix -A wrapper --no-out-link | grepQuiet "Dependencies on the outputs of dynamic derivations are not yet supported"
80 changes: 80 additions & 0 deletions tests/dyn-drv/eval-outputOf.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
#!/usr/bin/env bash

source ./common.sh

# Without the dynamic-derivations XP feature, we don't have the builtin.
nix --experimental-features 'nix-command' eval --impure --expr \
'assert ! (builtins ? outputOf); ""'

# Test that a string is required.
#
# We currently require a string to be passed, rather than a derivation
# object that could be coerced to a string. We might liberalise this in
# the future so it does work, but there are some design questions to
# resolve first. Adding a test so we don't liberalise it by accident.
expectStderr 1 nix --experimental-features 'nix-command dynamic-derivations' eval --impure --expr \
'builtins.outputOf (import ../dependencies.nix) "out"' \
| grepQuiet "value is a set while a string was expected"

# Test that "DrvDeep" string contexts are not supported at this time
#
# Like the above, this is a restriction we could relax later.
expectStderr 1 nix --experimental-features 'nix-command dynamic-derivations' eval --impure --expr \
'builtins.outputOf (import ../dependencies.nix).drvPath "out"' \
| grepQuiet "has a context which refers to a complete source and binary closure. This is not supported at this time"

# Test using `builtins.outputOf` with static derivations
testStaticHello () {
nix eval --impure --expr \
'with (import ./text-hashed-output.nix); let
a = hello.outPath;
b = builtins.outputOf (builtins.unsafeDiscardOutputDependency hello.drvPath) "out";
in builtins.trace a
(builtins.trace b
(assert a == b; null))'
}

# Test with a regular old input-addresed derivation
#
# `builtins.outputOf` works without ca-derivations and doesn't create a
# placeholder but just returns the output path.
testStaticHello

# Test with content addressed derivation.
NIX_TESTS_CA_BY_DEFAULT=1 testStaticHello

# Test with derivation-producing derivation
#
# This is hardly difference from the preceding cases, except that we're
# only taking 1 outputOf out of 2 possible outputOfs. Note that
# `.outPath` could be defined as `outputOf drvPath`, which is what we're
# testing here. The other `outputOf` that we're not testing here is the
# use of _dynamic_ derivations.
nix eval --impure --expr \
'with (import ./text-hashed-output.nix); let
a = producingDrv.outPath;
b = builtins.outputOf (builtins.builtins.unsafeDiscardOutputDependency producingDrv.drvPath) "out";
in builtins.trace a
(builtins.trace b
(assert a == b; null))'

# Test with unbuilt output of derivation-producing derivation.
#
# These are similar to the previous test, but instead of checking the
# property on a constant derivation, we check it on a derivation that's
# from another derivation's output (outPath).
testDynamicHello () {
nix eval --impure --expr \
'with (import ./text-hashed-output.nix); let
a = builtins.outputOf producingDrv.outPath "out";
b = builtins.outputOf (builtins.outputOf (builtins.unsafeDiscardOutputDependency producingDrv.drvPath) "out") "out";
in builtins.trace a
(builtins.trace b
(assert a == b; null))'
}

# inner dynamic derivation is input-addressed
testDynamicHello

# inner dynamic derivation is content-addressed
NIX_TESTS_CA_BY_DEFAULT=1 testDynamicHello
4 changes: 3 additions & 1 deletion tests/dyn-drv/local.mk
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
dyn-drv-tests := \
$(d)/text-hashed-output.sh \
$(d)/recursive-mod-json.sh \
$(d)/build-built-drv.sh
$(d)/build-built-drv.sh \
$(d)/eval-outputOf.sh \
$(d)/dep-built-drv.sh

install-tests-groups += dyn-drv

Expand Down
2 changes: 2 additions & 0 deletions tests/dyn-drv/recursive-mod-json.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ source common.sh
# FIXME
if [[ $(uname) != Linux ]]; then skipTest "Not running Linux"; fi

export NIX_TESTS_CA_BY_DEFAULT=1

enableFeatures 'recursive-nix'
restartDaemon

Expand Down
10 changes: 7 additions & 3 deletions tests/dyn-drv/text-hashed-output.nix
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,6 @@ rec {
mkdir -p $out
echo "Hello World" > $out/hello
'';
__contentAddressed = true;
outputHashMode = "recursive";
outputHashAlgo = "sha256";
};
producingDrv = mkDerivation {
name = "hello.drv";
Expand All @@ -26,4 +23,11 @@ rec {
outputHashMode = "text";
outputHashAlgo = "sha256";
};
wrapper = mkDerivation {
name = "use-dynamic-drv-in-ca-drv";
buildCommand = ''
echo "Copying the output of the dynamic derivation"
cp -r ${builtins.outputOf producingDrv.outPath "out"} $out
'';
};
}

0 comments on commit 11c9d8b

Please sign in to comment.