Greatly expand architecture section, including splitting into abstract vs concrete model

[Ericson2314]

I want to complete specify the store layer.

---

The concrete vs abstract split I would expect to be controversial, so here my rationalization for it:

Explaining everything at once makes for a huge tsunami of information that is highly likely to overwhelm the reader, even if they are "random accessing" the reference rather than reading it in order, end to end. However, if one learns the abstract model first, then they can develop their mental model for the "skeleton" on which the concrete details is the "meat". Rather than appearing as a wave of assorted details, the details make sense as a combination of the requirements of the abstract model with only a few arbitrary choices + historical evolutions mixed. That makes for a "compessed" mental representation (the concrete model encoded against the abstract mode) which is a lot less mentally taxing.

In the language of https://documentation.divio.com/'s documentation quadrant, the abstract model is pure explanation, describing what we are trying to do with out the constraints of how Unix and software for unix work today. The concrete model is mainly reference, with just enough explanation to tie it back to the abstract model.

This order of information I think matches how @edolstra designed Nix in the first place (see patterns in functional programming that can be reused for new purpose, design system accordingly), and also matches how I think most advanced users / developer Nix think about it. I thus think it is both a proper and a "proven" strategy for how "bottom half" of the documentation quadrant can be conveyed.

[toraritte]

Does it still hold that Nix is a purely functional model?

[toraritte]

I had a hard time wrapping my head around this section, until I jumped ahead to read both abstract.md and concrete.md. Basically,

• the "abstract" section introduces Nix fundamental building blocks, and explains how the build prcoess works at a high level

• the "concrete" section takes the introduced terms (derivation, store object, etc.) and functional programming principles, and applies them in the context of file systems

If this is correct, then the phrase "concrete specifics" here means "examples of the application of functional programming principles to build system operations over a file system", right? Also, wouldn't it be clearer to call "abstract model" as "model" and "concrete model" as "application" / "implementation"?

On a related note, I think this approach could be applied to the entirety of this section or to the Nix manual - naybe even splitting into two levels of documents?

• the first level of doc(s) expound(s) the model / specification (building blocks, abstract rules, core paradigms, guiding principles, etc.)

• the second level are companion guides that drive the level one (i.e., spec) concepts home by showing how they are applied in the implementation, referring back to level one docs when needed.

(Of course, if the spec changes, both levels need to be updated.)

---

Suggestion:

To ease understanding, we will explain the Nix store in two parts:

1. Subsection X introduces the theoretical functional model<1>
2. Subsection Y shows how the functional paradigm and elements of the model are applied when it comes to building software in an actual filesystem<2>

[1]: or notion, concept, etc.

[2]: There could be other examples to show the versatility of Nix when using it to assemble stuff other than software

[toraritte]

Both reviews should have probably been submitted to #6420, right?.. (It's highly confusing to have 2 PRs for the same topic - are there more?)

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/summer-of-nix-documentation-stream/20351/4

[Ericson2314]

Both reviews should have probably been submitted to #6420, right?.. (It's highly confusing to have 2 PRs for the same topic - are there more?)

Well this one is supposed to be pick up against the last one. Even before I did the abstract vs concrete thing, I made the separate branch to have more content at the cost of less polishing.

[edolstra]

I'm not convinced that having a big architecture document is the way to go. I think it would be better to have more "literate", cross-referenced comments in the source code in combination with doxygen (or whatever). For example:

• reference-scanning.md should be a doxygen comment in references.hh.
• The description of the NAR format can go into archive.hh.
• Not sure what the best anchor would be for describing the store path hash computation, but probably we should move all makeStorePath() and friends out of store-api.cc into their own source file and then we can keep the docs there.
• The description of the build algoritm can go into libstore/build.

I also don't think we should have Haskell-style pseudo-code that reproduces C++ types, like this:

data DerivedPath
  = OpaquePath { path : StorePath }
  | BuiltPath {
      drv    : StorePath,
      output : OutputName,
    }

Given how much churn there has been in these types on the implementation side, having to constantly update an architecture document to reflect implementation changes would be painful. It is hard enough to keep comments in sync with the source, but the architecture document is likely to become outdated immediately. Again I think it would be preferable to have doxygen comments on the C++ types that describe their function.

There is also a lot of terminology here that doesn't appear in the implementation, which is likely to confuse people, e.g. "references are capabilities", FSOs, etc. Also, there are some reflections that would be more appropriate for an academic paper, like monadic vs applicative build systems and comparisons to Von Neumann/Harvard architectures.

[edolstra]

I have a hard time understanding this section and I don't really know what it's trying to say. Nix absolutely does have to scan for references, there is no option to disable that.

[Ericson2314]

This section is pretty WIP.

There is a subtlety where reference scanning isn't normative in that when one does a --check we don't attempt to rescan, but I can make that more explicit

[blaggacao]

As a pure tooling advice, there is handy: https://github.com/JoelCourtney/mdbook-kroki-preprocessor

[Ericson2314]

@edolstra the entire point of having a specification document is that it is separate from the implementation. I want people to know the the Nix store layer works without having to read any C++ or even Doxygen.

[Ericson2314]

Given how much churn there has been in these types on the implementation side, having to constantly update an architecture document to reflect implementation changes would be painful.

A lot less painful that doing it out of tree though!

Also, I think the type changes have been mainly replacing strings and more std::variant and thus are by and large converging on new stable design. I expect new changes to be not from refactoring existing features once they are cleaned up, but adding new features like "deep content-addressing".

It still benefits being in-tree in that case so we can update the spec and implementation together in the new feature case, but it is not so onerous that it is futile to have a spec at all.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/summer-of-nix-documentation-stream/20351/2

[fricklerhandwerk]

doxygen

@edolstra Do we have infrastructure to make this visible on a par with the manual? As long as we don't, I'm strongly in favor of adding this type of material to the manual. I'm agnostic to how we build this, if the result is easily discoverable and not too expensive in terms of setup and maintenance.

I agree that this kind of thing should be right next to the code, but we can still move it in there later. What's more important is that it's written down at all, and easy to find, read, and modify.

I also don't think we should have Haskell-style pseudo-code that reproduces C++ types

@edolstra Agreed. The architecture spec should show the principles, not the implementation details. Actually we should not use any kind of specific code to represent those principles, but ideally something more universal (and self-explanatory) such as UML diagrams (no specific proposal, just an example) where it's suitable. Even if Haskell or whatever is more precise, we cannot assume people to know any of this. Learning also has dependencies, and we should make them explicit and keep the closure small.

There is also a lot of terminology here that doesn't appear in the implementation

Concerning really abstract stuff that doesn't touch on what's really going on in the implementation I agree, but I suppose some things can have more structured naming. Sure, the term FSO is not used in the code, but why not change the code?

@Ericson2314 I'm not convinced by the abstract/concrete setup. There is overwhelming evidence that people don't learn well from first principles. The manual, even the architecture spec, should describe Nix specifically, and nothing else. As suggested above, we can squint a little bit or add some wishful thinking about code factoring and naming to make things appear less thorny, but we should stay with what there is.

While I believe all of what you wrote is true, it is often hard to read and understand. And because we made the mistake already in #6420, I think we should go through this with multiple PRs section by section (as outlined here), even if it is more work for authors. It's really really hard to review such a large amount of text, and we have to take reviewer's effort into consideration as well.

@edolstra If you could please revert 81e1013 we can do exactly that for #6420 and make a PR for each section topologically sorted, now that we have developed an idea on the general direction we would like to take.

[Ericson2314]

The point of the data types is that it is the interface, not just a mere implementation detail.

I am fine replacing with something that doesn't look like code, but let's be clear what this is achieving. People have a bias "programing language = implementation", so we are simply laundering the exact same information in a format that seems more "platonic".

If we understand that we're just trying to find a presentation that plays nicely with people's biases, great, we're on the same page. If we think there actually is an objective rather than rhetorical problem with data types in the spec, then we're still talking past each other on what a "specification" even means.

[fricklerhandwerk]

People have a bias "programing language = implementation", so we are simply laundering the exact same information in a format that seems more "platonic".

@Ericson2314 when we worked on #6420 we were pretty clear in that regard. We used non-code interface descriptions to be more universally approachable and simplified them to a point where the principle became visible. Sure, if we rewrote the Store in Haskell, that would be a moot point. But that's not what Nix is today.

It's not about dumbing down, it's about making it easy to understand with minimal prerequisites.

[Ericson2314]

@fricklerhandwerk but that PR did contain data definitions, like https://github.com/nix-community/nix/blob/39d32ac4c63f4aa3784d114b19c0eca83e306ca9/doc/manual/src/architecture/store/fso.md This PR adds more of them, but it uses them in exactly the same way: to specify Nix, irrespective of how it is implemented.

[roberth]

This is written in a somewhat narrative, certainly linear and learning-oriented style. An explanation, whereas the manual is a reference manual.

It starts in a way that is so abstract that I doubt that opening with it is a good idea, considering its central place in the ecosystem and the expectation most users have to learn about the tool, not the architecture. Opening this way signals to them that Nix is a highly abstract and academic piece of software that's not for solving their practical problems. Of course these thoughts are misguided, but that does not make them less real or less prevalent.

The language used is only understandable by functional programmers, which is a mere fraction of this document's audience.

I think we might be able to include documentation like this in the manual, but then I think it should be in a 'second half' after the reference documentation. It would also be wise to only link to the explanations from the places where they are introduced in the reference documentation. The architecture and explanation section of the manual _ must not_ be used as a substitute for reference documentation.