Tracking issue for RFC 92: Dynamic derivations #6316
Comments
|
FYI, the middle 2 PRs might be better to review together, as the second one revises the |
Ericson2314
changed the title
|
This issue has been mentioned on NixOS Discourse. There might be relevant details there: |
roberth
added
the
significant
|
This issue has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/zurich-23-05-zhf-hackathon-and-workshop-report/29093/1 |
This makes it more useful. In general, the derivation will be in one store, and the realisation info is in another. This also helps us avoid duplication. See how `resolveDerivedPath` is now simpler because it uses `queryPartialDerivationOutputMap`. In NixOS#8369 we get more flavors of derived path, and need more code to resolve them all, and this problem only gets worse. The fact that we need a new method to deal with the multiple dispatch is unfortunate, but this generally relates to the fact that `Store` is a sub-par interface, too bulky/unwieldy and conflating separate concerns. Solving that is out of scope of this PR. This is part of the RFC 92 work. See tracking issue NixOS#6316
In the Nix language, given a drv path, we should be able to construct another string referencing to one of its output. We can do this today with `(import drvPath).output`, but this only works for derivations we already have. With dynamic derivations, however, that doesn't work well because the `drvPath` isn't yet built: importing it like would need to trigger IFD, when the whole point of this feature is to do "dynamic build graph" without IFD! Instead, what we want to do is create a placeholder value with the right string context to refer to the output of the as-yet unbuilt derivation. A new primop in the language, analogous to `builtins.placeholder` can be used to create one. This will achieve all the right properties. The placeholder machinery also will match out the `outPath` attribute for CA derivations works. In 60b7121 we added that type of placeholder, and the derived path and string holder changes necessary to support it. Now, we can wire up the primop. Part of RFC 92: dynamic derivations (tracking issue NixOS#6316)
In the Nix language, given a drv path, we should be able to construct another string referencing to one of its output. We can do this today with `(import drvPath).output`, but this only works for derivations we already have. With dynamic derivations, however, that doesn't work well because the `drvPath` isn't yet built: importing it like would need to trigger IFD, when the whole point of this feature is to do "dynamic build graph" without IFD! Instead, what we want to do is create a placeholder value with the right string context to refer to the output of the as-yet unbuilt derivation. A new primop in the language, analogous to `builtins.placeholder` can be used to create one. This will achieve all the right properties. The placeholder machinery also will match out the `outPath` attribute for CA derivations works. In 60b7121 we added that type of placeholder, and the derived path and string holder changes necessary to support it. Then in the previous commit we cleaned up the code (inspiration finally hit me!) to deduplicate the code and expose exactly what we need. Now, we can wire up the primop trivally! Part of RFC 92: dynamic derivations (tracking issue NixOS#6316)
In the Nix language, given a drv path, we should be able to construct another string referencing to one of its output. We can do this today with `(import drvPath).output`, but this only works for derivations we already have. With dynamic derivations, however, that doesn't work well because the `drvPath` isn't yet built: importing it like would need to trigger IFD, when the whole point of this feature is to do "dynamic build graph" without IFD! Instead, what we want to do is create a placeholder value with the right string context to refer to the output of the as-yet unbuilt derivation. A new primop in the language, analogous to `builtins.placeholder` can be used to create one. This will achieve all the right properties. The placeholder machinery also will match out the `outPath` attribute for CA derivations works. In 60b7121 we added that type of placeholder, and the derived path and string holder changes necessary to support it. Then in the previous commit we cleaned up the code (inspiration finally hit me!) to deduplicate the code and expose exactly what we need. Now, we can wire up the primop trivally! Part of RFC 92: dynamic derivations (tracking issue NixOS#6316) Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
In the Nix language, given a drv path, we should be able to construct another string referencing to one of its output. We can do this today with `(import drvPath).output`, but this only works for derivations we already have. With dynamic derivations, however, that doesn't work well because the `drvPath` isn't yet built: importing it like would need to trigger IFD, when the whole point of this feature is to do "dynamic build graph" without IFD! Instead, what we want to do is create a placeholder value with the right string context to refer to the output of the as-yet unbuilt derivation. A new primop in the language, analogous to `builtins.placeholder` can be used to create one. This will achieve all the right properties. The placeholder machinery also will match out the `outPath` attribute for CA derivations works. In 60b7121 we added that type of placeholder, and the derived path and string holder changes necessary to support it. Then in the previous commit we cleaned up the code (inspiration finally hit me!) to deduplicate the code and expose exactly what we need. Now, we can wire up the primop trivally! Part of RFC 92: dynamic derivations (tracking issue NixOS#6316) Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
In the Nix language, given a drv path, we should be able to construct another string referencing to one of its output. We can do this today with `(import drvPath).output`, but this only works for derivations we already have. With dynamic derivations, however, that doesn't work well because the `drvPath` isn't yet built: importing it like would need to trigger IFD, when the whole point of this feature is to do "dynamic build graph" without IFD! Instead, what we want to do is create a placeholder value with the right string context to refer to the output of the as-yet unbuilt derivation. A new primop in the language, analogous to `builtins.placeholder` can be used to create one. This will achieve all the right properties. The placeholder machinery also will match out the `outPath` attribute for CA derivations works. In 60b7121 we added that type of placeholder, and the derived path and string holder changes necessary to support it. Then in the previous commit we cleaned up the code (inspiration finally hit me!) to deduplicate the code and expose exactly what we need. Now, we can wire up the primop trivally! Part of RFC 92: dynamic derivations (tracking issue NixOS#6316) Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
In the Nix language, given a drv path, we should be able to construct another string referencing to one of its output. We can do this today with `(import drvPath).output`, but this only works for derivations we already have. With dynamic derivations, however, that doesn't work well because the `drvPath` isn't yet built: importing it like would need to trigger IFD, when the whole point of this feature is to do "dynamic build graph" without IFD! Instead, what we want to do is create a placeholder value with the right string context to refer to the output of the as-yet unbuilt derivation. A new primop in the language, analogous to `builtins.placeholder` can be used to create one. This will achieve all the right properties. The placeholder machinery also will match out the `outPath` attribute for CA derivations works. In 60b7121 we added that type of placeholder, and the derived path and string holder changes necessary to support it. Then in the previous commit we cleaned up the code (inspiration finally hit me!) to deduplicate the code and expose exactly what we need. Now, we can wire up the primop trivally! Part of RFC 92: dynamic derivations (tracking issue NixOS#6316) Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
In the Nix language, given a drv path, we should be able to construct another string referencing to one of its output. We can do this today with `(import drvPath).output`, but this only works for derivations we already have. With dynamic derivations, however, that doesn't work well because the `drvPath` isn't yet built: importing it like would need to trigger IFD, when the whole point of this feature is to do "dynamic build graph" without IFD! Instead, what we want to do is create a placeholder value with the right string context to refer to the output of the as-yet unbuilt derivation. A new primop in the language, analogous to `builtins.placeholder` can be used to create one. This will achieve all the right properties. The placeholder machinery also will match out the `outPath` attribute for CA derivations works. In 60b7121 we added that type of placeholder, and the derived path and string holder changes necessary to support it. Then in the previous commit we cleaned up the code (inspiration finally hit me!) to deduplicate the code and expose exactly what we need. Now, we can wire up the primop trivally! Part of RFC 92: dynamic derivations (tracking issue NixOS#6316) Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
In the Nix language, given a drv path, we should be able to construct another string referencing to one of its output. We can do this today with `(import drvPath).output`, but this only works for derivations we already have. With dynamic derivations, however, that doesn't work well because the `drvPath` isn't yet built: importing it like would need to trigger IFD, when the whole point of this feature is to do "dynamic build graph" without IFD! Instead, what we want to do is create a placeholder value with the right string context to refer to the output of the as-yet unbuilt derivation. A new primop in the language, analogous to `builtins.placeholder` can be used to create one. This will achieve all the right properties. The placeholder machinery also will match out the `outPath` attribute for CA derivations works. In 60b7121 we added that type of placeholder, and the derived path and string holder changes necessary to support it. Then in the previous commit we cleaned up the code (inspiration finally hit me!) to deduplicate the code and expose exactly what we need. Now, we can wire up the primop trivally! Part of RFC 92: dynamic derivations (tracking issue NixOS#6316) Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
In the Nix language, given a drv path, we should be able to construct another string referencing to one of its output. We can do this today with `(import drvPath).output`, but this only works for derivations we already have. With dynamic derivations, however, that doesn't work well because the `drvPath` isn't yet built: importing it like would need to trigger IFD, when the whole point of this feature is to do "dynamic build graph" without IFD! Instead, what we want to do is create a placeholder value with the right string context to refer to the output of the as-yet unbuilt derivation. A new primop in the language, analogous to `builtins.placeholder` can be used to create one. This will achieve all the right properties. The placeholder machinery also will match out the `outPath` attribute for CA derivations works. In 60b7121 we added that type of placeholder, and the derived path and string holder changes necessary to support it. Then in the previous commit we cleaned up the code (inspiration finally hit me!) to deduplicate the code and expose exactly what we need. Now, we can wire up the primop trivally! Part of RFC 92: dynamic derivations (tracking issue NixOS#6316) Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
In the Nix language, given a drv path, we should be able to construct another string referencing to one of its output. We can do this today with `(import drvPath).output`, but this only works for derivations we already have. With dynamic derivations, however, that doesn't work well because the `drvPath` isn't yet built: importing it like would need to trigger IFD, when the whole point of this feature is to do "dynamic build graph" without IFD! Instead, what we want to do is create a placeholder value with the right string context to refer to the output of the as-yet unbuilt derivation. A new primop in the language, analogous to `builtins.placeholder` can be used to create one. This will achieve all the right properties. The placeholder machinery also will match out the `outPath` attribute for CA derivations works. In 60b7121 we added that type of placeholder, and the derived path and string holder changes necessary to support it. Then in the previous commit we cleaned up the code (inspiration finally hit me!) to deduplicate the code and expose exactly what we need. Now, we can wire up the primop trivally! Part of RFC 92: dynamic derivations (tracking issue NixOS#6316) Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
In the Nix language, given a drv path, we should be able to construct another string referencing to one of its output. We can do this today with `(import drvPath).output`, but this only works for derivations we already have. With dynamic derivations, however, that doesn't work well because the `drvPath` isn't yet built: importing it like would need to trigger IFD, when the whole point of this feature is to do "dynamic build graph" without IFD! Instead, what we want to do is create a placeholder value with the right string context to refer to the output of the as-yet unbuilt derivation. A new primop in the language, analogous to `builtins.placeholder` can be used to create one. This will achieve all the right properties. The placeholder machinery also will match out the `outPath` attribute for CA derivations works. In 60b7121 we added that type of placeholder, and the derived path and string holder changes necessary to support it. Then in the previous commit we cleaned up the code (inspiration finally hit me!) to deduplicate the code and expose exactly what we need. Now, we can wire up the primop trivally! Part of RFC 92: dynamic derivations (tracking issue NixOS#6316) Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
In the Nix language, given a drv path, we should be able to construct another string referencing to one of its output. We can do this today with `(import drvPath).output`, but this only works for derivations we already have. With dynamic derivations, however, that doesn't work well because the `drvPath` isn't yet built: importing it like would need to trigger IFD, when the whole point of this feature is to do "dynamic build graph" without IFD! Instead, what we want to do is create a placeholder value with the right string context to refer to the output of the as-yet unbuilt derivation. A new primop in the language, analogous to `builtins.placeholder` can be used to create one. This will achieve all the right properties. The placeholder machinery also will match out the `outPath` attribute for CA derivations works. In 60b7121 we added that type of placeholder, and the derived path and string holder changes necessary to support it. Then in the previous commit we cleaned up the code (inspiration finally hit me!) to deduplicate the code and expose exactly what we need. Now, we can wire up the primop trivally! Part of RFC 92: dynamic derivations (tracking issue NixOS#6316) Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
|
This issue has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/2023-08-25-nix-team-meeting-minutes-82/32283/1 |
|
This issue has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/nixcon-governance-workshop/32705/9 |
|
If a dynamic derivation A performs
So while we could consider such deriver rows to be valid, adding them consistently is probably more trouble than it's worth. We should still have a deriver row for A and B, and I believe that's enough for querying any relevant paths through the combined deriver and reference graphs. TODO
|
|
Does |
|
Consider the following derivations:
I wouldn't expect it to list So, probably not, but this should be checked and documented. |
Could a case not be made for including a reference to |
|
Actually, having thought about it further, maybe dynamic derivations should not be allowed to be fixed-output at all. Even with CA derivations, a FOD dynamically generated by a nondeterministic derivation is not guarenteed to be reproducible from source. You could for example code in several possible URLs and their respective hashes, and then have |
|
@physics-enthusiast Isn't that why FODs require a hash as one of their parameters, to verify that they were reproduced? |
|
@Dessix the issue is that if the FOD is built dynamically (by another derivation), that hash can also be altered dynamically (i.e. during build-time rather than eval-time), breaking reproducibility. Come to think of it, setting hashes via IFD seem to have this problem too. |
I suppose something could be done for
As mentioned by @Dessix, FODs don't introduce any new build impurities. Dynamic derivations do turn actual build impurities into "instantiation impurities", but this is a necessary evil if we want to have this kind of dynamism. Anything that writes derivations should be written with care, whether that's our own evaluator or a dynamic derivation deriver builder. (Do we have a term for the derivation that produces another derivation? Dynamic derivation seems to refer to the output, not the derivation writer if that's what we want to call it.)
IFD is a somewhat more benign source of instantiation impurities, because at least you get the opportunity to do some of the processing in the Nix language, which has reproducibility as a design goal, especially with pure mode.
Generally IFD is allowed, but some of the metadata commands forbid it, under the assumption that they'd be evaluated and indexed centrally, and those commands should obey the same restrictions. However, that's not what flakestry.dev or flakehub do, as they just accept the readily evaluated JSON from CI. I know of at least one flake that disables the IFD restriction before uploading to a registry. |
|
@roberth Is this different from the question of whether CA derivations (which includes derivation-producing derivations) ought to use the deriver field for their outputs? |
|
@Ericson2314 a lot has been discussed, so I can't tell what this refers to. Also not sure what is the exact use of the deriver field in that context. I can see a vague parallel with the tracking of build input hashes (transitively) for any build, and CA in particular. Also relevant is the idea of those "large step realisation" objects #8947, which would become somewhat useless in cases where the discussed Coming to think of it, |
|
@roberth The possibility of offline rebuilds without having to copy over the entire store is actually a large part of the reason I raised these queries in the first place. At least in theory it should be possible to gather a binary cache of only the FODs in the build-time closure of a package and rebuild that package with it. My concern was that one of the things instantiation-time impurities can do that build-time impurities cannot is introduce additional external dependencies. This means that even if you could get every FOD that was used in the build (which RFC 92 would also prevent us from doing without the |
Info
Steps
Here are the PRs to review:
Preparatory work
<drvPath>^<outputName>installable syntax to match existing<highLevelInstallable>^<outputNames>syntax #4543OutputSpecforDerivationGoalandDerivedPath, today'sOutputSpec->ExtendedOutputSpec#6815.drvspecial-casing for store path installable #7600StoreReferencesandContentAddressWithReferences#3746exportReferencesGraphshould build onlydrvs when used withunsafeDiscardOutputsDependency#7339queryPartialDerivationOutputMapanevalStoreparameter #8724Actual implementation
downstreamPlaceholderto a type with methods #8353outputOfprimop. #8813Quality of life / Nice to have
fetchClosureCC @tomberek
The text was updated successfully, but these errors were encountered: