diff --git a/doc/manual/src/SUMMARY.md.in b/doc/manual/src/SUMMARY.md.in index e49e77cf548b..36fb8d245dbc 100644 --- a/doc/manual/src/SUMMARY.md.in +++ b/doc/manual/src/SUMMARY.md.in @@ -31,6 +31,7 @@ - [Language Constructs](language/constructs.md) - [String interpolation](language/string-interpolation.md) - [Lookup path](language/constructs/lookup-path.md) + - [String context](language/string-context.md) - [Operators](language/operators.md) - [Derivations](language/derivations.md) - [Advanced Attributes](language/advanced-attributes.md) diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md index 611d9576f3fb..c96019a3df4b 100644 --- a/doc/manual/src/glossary.md +++ b/doc/manual/src/glossary.md @@ -216,6 +216,18 @@ [output path]: #gloss-output-path +- [derived path]{#gloss-derived-path} + + A [store]-level expression (not Nix language expression) that denotes a [store object]. + There are two forms: + + - *constant*: A constant derived path is just a [store path] + + - *output*: An output derived path is a pair of a [store path] to a [derivation] and an [output] name. + + Derived paths are necessary because in general, we do not know what the [output path] path of an [output] will buntil it is [realise]d. + We need a way to refer to unrelised outputs in order to be able to tell Nix to realise them! + - [deriver]{#gloss-deriver} The [store derivation] that produced an [output path]. diff --git a/doc/manual/src/language/string-context.md b/doc/manual/src/language/string-context.md new file mode 100644 index 000000000000..a86c5ed00a99 --- /dev/null +++ b/doc/manual/src/language/string-context.md @@ -0,0 +1,144 @@ +# String context + +> **Note** +> +> This is an advanced topic. +> The Nix language is designed to be used without the programmer knowing string contexts are. + +Strings in the Nix language are not just a sequences of characters like strings in other languages. +They are actually pairs of sequences of characters and what is known as a *string context*. +The string context is an (unordered) set of *string context elements*. + +The goal of string contexts are to collect non-string values combined with strings via +[string concatenation](./operators.md#string-concatenation), +[string interpolation](./string-interpolation.md), +and similar operations. +The idea is that a user can combine together values to create a build recipe without manually keeping track of where the "ingredients" come from, and then the Nix language does that bookkeeping implicitly to come up with the right derivation inputs. + +> In line with this goal, string contexts are *not* explicitly manipulated in idiomatic Nix code. +> Strings with non-empty context are only concatenated and eventually passed to `builtins.derivation`. +> "Regular" strings an empty context are the only ones inspected, e.g. compared with `==`. + +String context elements come in 3 forms: + +- [*constant*]{#string-context-element-constant} +- [*output*]{#string-context-element-output} +- [*derivation deep*]{#string-context-element-derivation-deep} + +*Constant* and *output* string contexts elements are just +[derived paths](@docroot@/glossary.md#gloss-derived-path); +those are just the names of the two kinds of derived path. +See the documentation on derived paths for further details. + +*derivation deep* is an advanced feature intended to be used with the +[`exportReferencesGraph`](./advanced-attributes.html#adv-attr-exportReferencesGraph) +advanced derivation feature. +A *derivation deep* string context element is a derivation path, and refers to both its outputs and the entire build closure of that derivation: +all its outputs, all the other derivations the given derivation depends on, and all their outputs too. + +## Inspecting string contexts + +Most basically, [`builtins.hasContext`] will whether a string as a non-empty context. + +When more granular information than merely whether the string context is empty is needed, [`builtins.getContext`] can be used. +It creates an [attribute set] representing the string context; the attribute set can be inspected normally. + +[`builtins.hasContext`]: ./builtins.md#builtins-hasContext +[`builtins.getContext`]: ./builtins.md#builtins-getContext +[attribute set]: ./values.md#attribute-set + +## Clearing string contexts + +[`buitins.unsafeDiscardStringContext`](./builtins.md#) will make a copy of another string, but with an empty string context. +The new string can be inspected in more ways, e.g. by operators that require that the string context is empty. +Explicitly discarding the string context and then expecting it helps ensure that string context elements are not lost by mistake. + +## Constructing string contexts + +[`builtins.appendContext`] will create copy of a string but with additional string context elements. +The context is specified explicitly by an [attribute set] in the format that [`builtins.hasContext`] produces. +We there can create strings arbitrary contexts in 3 steps: + +1. Create strings with the string context elements we want +2. Dump their contexts with [`builtins.getContext`] +3. Combine them together with a base string and repeated [`builtins.appendContext`] calls. + +The remainder of this section will focus on step 1: making strings with individual string context elements on which to apply `builtins.getContext`. + +[`builtins.appendContext`]: ./builtins.md#builtins-appendContext + +## Constant string context elements + +A constant string context element is just a constant [derived path]; +a constant derived path is just a [store path]. +We therefore want to use [`builtins.storePath`] to create a string with a single constant string context element: + +> **Example** +> +> ```nix +> builtins.getContext (builtins.storePath "/nix/store/wkhdf9jinag5750mqlax6z2zbwhqb76n-hello-2.10") +> ``` +> evaluates to +> ```nix +> { +> "/nix/store/wkhdf9jinag5750mqlax6z2zbwhqb76n-hello-2.10" = { +> path = true; +> }; +> } +> ``` + +[derived path]: @docroot@/glossary.md#gloss-derived-path +[store path]: @docroot@/glossary.md#gloss-store-path +[`builtins.storePath`]: ./builtins.md#builtins-storePath + +## Output string context elements + +The best way to illustrate this with a builtin function that is still experimental: [`builtins.ouputOf`]. +This example will *not* work the stable Nix! + +> **Example** +> +> ```nix +> builtins.getContext +> (builtins.outputOf +> (builtins.storePath "/nix/store/fvchh9cvcr7kdla6n860hshchsba305w-hello-2.12.drv") +> "out") +> ``` +> evaluates to +> ```nix +> { +> "/nix/store/fvchh9cvcr7kdla6n860hshchsba305w-hello-2.12.drv" = { +> outputs = [ "out" ]; +> }; +> } +> ``` + +[`builtins.outputOf`]: ./builtins.md#builtins-outputOf + +## "Derivation deep" string context elements + +These best way to illustrate this is with [`builtins.addDrvOutputDependencies`]. +We take a regular constant string context element pointing to a derivation, and transform it unto a "Derivation deep" string context element. + +> **Example** +> +> ```nix +> builtins.getContext +> (builtins.addDrvOutputDependencies +> (builtins.storePath "/nix/store/fvchh9cvcr7kdla6n860hshchsba305w-hello-2.12.drv")) +> ``` +> evaluates to +> ```nix +> { +> "/nix/store/fvchh9cvcr7kdla6n860hshchsba305w-hello-2.12.drv" = { +> allOutputs = true; +> }; +> } +> ``` + +[`builtins.unsafeDiscardOutputDependency`] does this the opposite of [`builtins.addDrvOutputDependencies`], but is not prefered because it "weakens" rather than "strengens" the string context. +What is meant by that is that since the "derivation deep" string context element always refers to the underlying derivation (among many more things), +replacing a constant string context element with a "derivation deep" one is a safe operation that just enlargens the string context without forgetting anything. + +[`builtins.addDrvOutputDependencies`]: ./builtins.md#builtins-addDrvOutputDependencies +[`builtins.unsafeDiscardOutputDependency`]: ./builtins.md#builtins-unsafeDiscardOutputDependency diff --git a/src/libexpr/primops/context.cc b/src/libexpr/primops/context.cc index e8542503a42b..9dbed086e318 100644 --- a/src/libexpr/primops/context.cc +++ b/src/libexpr/primops/context.cc @@ -30,20 +30,27 @@ static RegisterPrimOp primop_hasContext({ .name = "__hasContext", .args = {"s"}, .doc = R"( - Return `true` if string *s* has a non-empty context. The - context can be obtained with + Return `true` if string *s* has a non-empty context. + The context can be obtained with [`getContext`](#builtins-getContext). + + > **Example** + > + > Many operations require a string context to be empty because they are intended only to work with "regular" strings, and also to help users avoid unintentionally loosing track of string context elements. + > `builtins.hasContext` can help create better domain-specific errors in those case. + > + > ```nix + > name: meta: + > + > if builtins.hasContext name + > then throw "package name cannot contain string context" + > else { ${name} = meta; } + > ``` )", .fun = prim_hasContext }); -/* Sometimes we want to pass a derivation path (i.e. pkg.drvPath) to a - builder without causing the derivation to be built (for instance, - in the derivation that builds NARs in nix-push, when doing - source-only deployment). This primop marks the string context so - that builtins.derivation adds the path to drv.inputSrcs rather than - drv.inputDrvs. */ static void prim_unsafeDiscardOutputDependency(EvalState & state, const PosIdx pos, Value * * args, Value & v) { NixStringContext context; @@ -66,11 +73,86 @@ static void prim_unsafeDiscardOutputDependency(EvalState & state, const PosIdx p static RegisterPrimOp primop_unsafeDiscardOutputDependency({ .name = "__unsafeDiscardOutputDependency", - .arity = 1, + .args = {"s"}, + .doc = R"( + Create a copy of the given string where every + [derivation deep](@docroot@/language/string-context.md#string-context-element-derivation-deep) + string context element is turned into a + [constant](@docroot@/language/string-context.md#string-context-element-constant) + string context element. + + This is unsafe because it allows us to "forgot" store objects we would have otherwise refered to with the string context. + Safe operations "grow" but never "shrink" string contexts. + + Opposite of [`builtins.addDrvOutputDependencies`](#builtins-addDrvOutputDependencies). + )", .fun = prim_unsafeDiscardOutputDependency }); +static void prim_addDrvOutputDependencies(EvalState & state, const PosIdx pos, Value * * args, Value & v) +{ + NixStringContext context; + auto s = state.coerceToString(pos, *args[0], context, "while evaluating the argument passed to builtins.addDrvOutputDependencies"); + + auto contextSize = context.size(); + if (contextSize != 1) { + throw EvalError({ + .msg = hintfmt("context of string '%s' must have exactly one element, but has %d", *s, contextSize), + .errPos = state.positions[pos] + }); + } + NixStringContext context2 { + (NixStringContextElem { std::visit(overloaded { + [&](const NixStringContextElem::Opaque & c) -> NixStringContextElem::DrvDeep { + if (!c.path.isDerivation()) { + throw EvalError({ + .msg = hintfmt("path '%s' is not a derivation", + state.store->printStorePath(c.path)), + .errPos = state.positions[pos], + }); + } + return NixStringContextElem::DrvDeep { + .drvPath = c.path, + }; + }, + [&](const NixStringContextElem::Built & c) -> NixStringContextElem::DrvDeep { + throw EvalError({ + .msg = hintfmt("`addDrvOutputDependencies` can only act on derivations, not derivation outputs"), + .errPos = state.positions[pos], + }); + }, + [&](const NixStringContextElem::DrvDeep & c) -> NixStringContextElem::DrvDeep { + /* Reuse original item because we want this to be idempotent. */ + return std::move(c); + }, + }, context.begin()->raw) }), + }; + + v.mkString(*s, context2); +} + +static RegisterPrimOp primop_addDrvOutputDependencies({ + .name = "__addDrvOutputDependencies", + .args = {"s"}, + .doc = R"( + Create a copy of the given string where a single + [constant](@docroot@/language/string-context.md#string-context-element-constant) + string context element is turned into a + [derivation deep](@docroot@/language/string-context.md#string-context-element-derivation-deep) + string context element. + + The store path that is the constant string context element should point to a valid derivation, and end in `.drv`. + + The original string context element must not be empty or have multiple elements, and it must not have any other type of element other than a constant or derivation deep element. + The latter is supported so this function is idempotent. + + Opposite of [`builtins.unsafeDiscardOutputDependency`](#builtins-addDrvOutputDependencies). + )", + .fun = prim_addDrvOutputDependencies +}); + + /* Extract the context of a string as a structured Nix value. The context is represented as an attribute set whose keys are the diff --git a/tests/functional/lang/eval-fail-addDrvOutputDependencies-empty-context.err.exp b/tests/functional/lang/eval-fail-addDrvOutputDependencies-empty-context.err.exp new file mode 100644 index 000000000000..ad91a22aa5b3 --- /dev/null +++ b/tests/functional/lang/eval-fail-addDrvOutputDependencies-empty-context.err.exp @@ -0,0 +1,10 @@ +error: + … while calling the 'addDrvOutputDependencies' builtin + + at /pwd/lang/eval-fail-addDrvOutputDependencies-empty-context.nix:1:1: + + 1| builtins.addDrvOutputDependencies "" + | ^ + 2| + + error: context of string '' must have exactly one element, but has 0 diff --git a/tests/functional/lang/eval-fail-addDrvOutputDependencies-empty-context.nix b/tests/functional/lang/eval-fail-addDrvOutputDependencies-empty-context.nix new file mode 100644 index 000000000000..dc9ee3ba2e59 --- /dev/null +++ b/tests/functional/lang/eval-fail-addDrvOutputDependencies-empty-context.nix @@ -0,0 +1 @@ +builtins.addDrvOutputDependencies "" diff --git a/tests/functional/lang/eval-fail-addDrvOutputDependencies-multi-elem-context.err.exp b/tests/functional/lang/eval-fail-addDrvOutputDependencies-multi-elem-context.err.exp new file mode 100644 index 000000000000..8438292034ab --- /dev/null +++ b/tests/functional/lang/eval-fail-addDrvOutputDependencies-multi-elem-context.err.exp @@ -0,0 +1,17 @@ +error: + … while calling the 'addDrvOutputDependencies' builtin + + at /pwd/lang/eval-fail-addDrvOutputDependencies-multi-elem-context.nix:18:4: + + 17| + 18| in builtins.addDrvOutputDependencies combo-path + | ^ + 19| + + … while calling the 'derivationStrict' builtin + + at /builtin/derivation.nix:9:12: + … while evaluating derivation 'fail' + whose name attribute is located at /pwd/lang/eval-fail-addDrvOutputDependencies-multi-elem-context.nix:3:5 + + error: operation 'addTextToStore' is not supported by store 'dummy' diff --git a/tests/functional/lang/eval-fail-addDrvOutputDependencies-multi-elem-context.nix b/tests/functional/lang/eval-fail-addDrvOutputDependencies-multi-elem-context.nix new file mode 100644 index 000000000000..dbde264dfaeb --- /dev/null +++ b/tests/functional/lang/eval-fail-addDrvOutputDependencies-multi-elem-context.nix @@ -0,0 +1,18 @@ +let + drv0 = derivation { + name = "fail"; + builder = "/bin/false"; + system = "x86_64-linux"; + outputs = [ "out" "foo" ]; + }; + + drv1 = derivation { + name = "fail-2"; + builder = "/bin/false"; + system = "x86_64-linux"; + outputs = [ "out" "foo" ]; + }; + + combo-path = "${drv0.drvPath}${drv1.drvPath}"; + +in builtins.addDrvOutputDependencies combo-path diff --git a/tests/functional/lang/eval-fail-addDrvOutputDependencies-wrong-element-kind.err.exp b/tests/functional/lang/eval-fail-addDrvOutputDependencies-wrong-element-kind.err.exp new file mode 100644 index 000000000000..acf68bc69123 --- /dev/null +++ b/tests/functional/lang/eval-fail-addDrvOutputDependencies-wrong-element-kind.err.exp @@ -0,0 +1,20 @@ +error: + … while calling the 'addDrvOutputDependencies' builtin + + at /pwd/lang/eval-fail-addDrvOutputDependencies-wrong-element-kind.nix:9:4: + + 8| + 9| in builtins.addDrvOutputDependencies drv.outPath + | ^ + 10| + + … while calling the 'getAttr' builtin + + at /builtin/derivation.nix:19:19: + … while calling the 'derivationStrict' builtin + + at /builtin/derivation.nix:9:12: + … while evaluating derivation 'fail' + whose name attribute is located at /pwd/lang/eval-fail-addDrvOutputDependencies-wrong-element-kind.nix:3:5 + + error: operation 'addTextToStore' is not supported by store 'dummy' diff --git a/tests/functional/lang/eval-fail-addDrvOutputDependencies-wrong-element-kind.nix b/tests/functional/lang/eval-fail-addDrvOutputDependencies-wrong-element-kind.nix new file mode 100644 index 000000000000..e379e1d9598b --- /dev/null +++ b/tests/functional/lang/eval-fail-addDrvOutputDependencies-wrong-element-kind.nix @@ -0,0 +1,9 @@ +let + drv = derivation { + name = "fail"; + builder = "/bin/false"; + system = "x86_64-linux"; + outputs = [ "out" "foo" ]; + }; + +in builtins.addDrvOutputDependencies drv.outPath diff --git a/tests/functional/lang/eval-okay-context-introspection.exp b/tests/functional/lang/eval-okay-context-introspection.exp index 03b400cc8862..a136b0035e0a 100644 --- a/tests/functional/lang/eval-okay-context-introspection.exp +++ b/tests/functional/lang/eval-okay-context-introspection.exp @@ -1 +1 @@ -[ true true true true true true ] +[ true true true true true true true true true true true true true ] diff --git a/tests/functional/lang/eval-okay-context-introspection.nix b/tests/functional/lang/eval-okay-context-introspection.nix index 50a78d946e76..a55d3ec47fc2 100644 --- a/tests/functional/lang/eval-okay-context-introspection.nix +++ b/tests/functional/lang/eval-okay-context-introspection.nix @@ -31,11 +31,29 @@ let (builtins.unsafeDiscardStringContext str) (builtins.getContext str); + # Only holds true if string context contains both a `DrvDeep` and + # `Opaque` element. + almostEtaRule = str: + str == builtins.addDrvOutputDependencies + (builtins.unsafeDiscardOutputDependency str); + + addDrvOutputDependencies_idemopotent = str: + builtins.addDrvOutputDependencies str == + builtins.addDrvOutputDependencies (builtins.addDrvOutputDependencies str); + + rules = str: [ + (etaRule str) + (almostEtaRule str) + (addDrvOutputDependencies_idemopotent str) + ]; + in [ (legit-context == desired-context) (reconstructed-path == combo-path) (etaRule "foo") - (etaRule drv.drvPath) (etaRule drv.foo.outPath) - (etaRule (builtins.unsafeDiscardOutputDependency drv.drvPath)) +] ++ builtins.concatMap rules [ + drv.drvPath + (builtins.addDrvOutputDependencies drv.drvPath) + (builtins.unsafeDiscardOutputDependency drv.drvPath) ]