Write implementation documentation (#2042)

Also rename Dispatcher to CompilationDispatcher for clarity

CONTRIBUTING.md

@@ -10,6 +10,7 @@ Want to contribute? Great! First, read this page.
   * [Synchronizing](#synchronizing)
   * [File Headers](#file-headers)
 * [Release Process](#release-process)
+* [Package Structure](#package-structure)
 
 ## Before You Contribute
 
@@ -245,3 +246,13 @@ few things to do before pushing that tag:
 
 You *don't* need to create tags for packages in `pkg`; that will be handled
 automatically by GitHub actions.
+
+## Package Structure
+
+The structure of the Sass package is documented in README.md files in most
+directories under `lib/`. This documentation is intended to help contributors
+quickly build a basic understanding of the structure of the compiler and how its
+various pieces fit together. [`lib/src/README.md`] is a good starting point to get
+an overview of the compiler as a whole.
+
+[`lib/src/README.md`]: lib/src/README.md

lib/src/README.md

@@ -0,0 +1,230 @@
+# The Sass Compiler
+
+* [Life of a Compilation](#life-of-a-compilation)
+  * [Late Parsing](#late-parsing)
+  * [Early Serialization](#early-serialization)
+* [JS Support](#js-support)
+* [APIs](#apis)
+  * [Importers](#importers)
+  * [Custom Functions](#custom-functions)
+  * [Loggers](#loggers)
+* [Built-In Functions](#built-in-functions)
+* [`@extend`](#extend)
+
+This is the root directory of Dart Sass's private implementation libraries. This
+contains essentially all the business logic defining how Sass is actually
+compiled, as well as the APIs that users use to interact with Sass. There are
+two exceptions:
+
+* [`../../bin/sass.dart`] is the entrypoint for the Dart Sass CLI (on all
+  platforms). While most of the logic it runs exists in this directory, it does
+  contain some logic to drive the basic compilation logic and handle errors. All
+  the most complex parts of the CLI, such as option parsing and the `--watch`
+  command, are handled in the [`executable`] directory. Even Embedded Sass runs
+  through this entrypoint, although it gets immediately gets handed off to [the
+  embedded compiler].
+
+  [`../../bin/sass.dart`]: ../../bin/sass.dart
+  [`executable`]: executable
+  [the embedded compiler]: embedded/README.md
+
+* [`../sass.dart`] is the entrypoint for the public Dart API. This is what's
+  loaded when a Dart package imports Sass. It just contains the basic
+  compilation functions, and exports the rest of the public APIs from this
+  directory.
+
+  [`../sass.dart`]: ../sass.dart
+
+Everything else is contained here, and each file and some subdirectories have
+their own documentation. But before you dive into those, let's take a look at
+the general lifecycle of a Sass compilation.
+
+## Life of a Compilation
+
+Whether it's invoked through the Dart API, the JS API, the CLI, or the embedded
+host, the basic process of a Sass compilation is the same. Sass is implemented
+as an AST-walking [interpreter] that operates in roughly three passes:
+
+[interpreter]: https://en.wikipedia.org/wiki/Interpreter_(computing)
+
+1. **Parsing**. The first step of a Sass compilation is always to parse the
+   source file, whether it's SCSS, the indented syntax, or CSS. The parsing
+   logic lives in the [`parse`] directory, while the abstract syntax tree that
+   represents the parsed file lives in [`ast/sass`].
+
+   [`parse`]: parse/README.md
+   [`ast/sass`]: ast/sass/README.md
+
+2. **Evaluation**. Once a Sass file is parsed, it's evaluated by
+   [`visitor/async_evaluate.dart`]. (Why is there both an async and a sync
+   version of this file? See [Synchronizing] for details!) The evaluator handles
+   all the Sass-specific logic: it resolves variables, includes mixins, executes
+   control flow, and so on. As it goes, it builds up a new AST that represents
+   the plain CSS that is the compilation result, which is defined in
+   [`ast/css`].
+
+   [`visitor/async_evaluate.dart`]: visitor/async_evaluate.dart
+   [Synchronizing]: ../../CONTRIBUTING.md#synchronizing
+   [`ast/css`]: ast/css/README.md
+
+   Sass evaluation is almost entirely linear: it begins at the first statement
+   of the file, evaluates it (which may involve evaluating its nested children),
+   adds its result to the CSS AST, and then moves on to the second statement. On
+   it goes until it reaches the end of the file, at which point it's done. The
+   only exception is module resolution: every Sass module has its own compiled
+   CSS AST, and once the entrypoint file is done compiling the evaluator will go
+   back through these modules, resolve `@extend`s across them as necessary, and
+   stitch them together into the final stylesheet.
+
+   SassScript, the expression-level syntax, is handled by the same evaluator.
+   The main difference between SassScript and statement-level evaluation is that
+   the same SassScript values are used during evaluation _and_ as part of the
+   CSS AST. This means that it's possible to end up with a Sass-specific value,
+   such as a map or a first-class function, as the value of a CSS declaration.
+   If that happens, the Serialization phase will signal an error when it
+   encounters the invalid value.
+
+3. **Serialization**. Once we have the CSS AST that represents the compiled
+   stylesheet, we need to convert it into actual CSS text. This is done by
+   [`visitor/serialize.dart`], which walks the AST and builds up a big buffer of
+   the resulting CSS. It uses [a special string buffer] that tracks source and
+   destination locations in order to generate [source maps] as well.
+
+   [`visitor/serialize.dart`]: visitor/serialize.dart
+   [a special string buffer]: util/source_map_buffer.dart
+   [source maps]: https://web.dev/source-maps/
+
+There's actually one slight complication here: the first and second pass aren't
+as separate as they appear. When one Sass stylesheet loads another with `@use`,
+`@forward`, or `@import`, that rule is handled by the evaluator and _only at
+that point_ is the loaded file parsed. So in practice, compilation actually
+switches between parsing and evaluation, although each individual stylesheet
+naturally has to be parsed before it can be evaluated.
+
+### Late Parsing
+
+Some syntax within a stylesheet is only parsed _during_ evaluation. This allows
+authors to use `#{}` interpolation to inject Sass variables and other dynamic
+values into various locations, such as selectors, while still allowing Sass to
+parse them to support features like nesting and `@extend`. The following
+syntaxes are parsed during evaluation:
+
+* [Selectors](parse/selector.dart)
+* [`@keyframes` frames](parse/keyframe_selector.dart)
+* [Media queries](parse/media_query.dart) (for historical reasons, these are
+  parsed before evaluation and then _reparsed_ after they've been fully
+  evaluated)
+
+### Early Serialization
+
+There are also some cases where the evaluator can serialize values before the
+main serialization pass. For example, if you inject a variable into a selector
+using `#{}`, that variable's value has to be converted to a string during
+evaluation so that the evaluator can then parse and handle the newly-generated
+selector. The evaluator does this by invoking the serializer _just_ for that
+specific value. As a rule of thumb, this happens anywhere interpolation is used
+in the original stylesheet, although there are a few other circumstances as
+well.
+
+## JS Support
+
+One of the main benefits of Dart as an implementation language is that it allows
+us to distribute Dart Sass both as an extremely efficient stand-alone executable
+_and_ an easy-to-install pure-JavaScript package, using the dart2js compilation
+tool. However, properly supporting JS isn't seamless. There are two major places
+where we need to think about JS support:
+
+1. When interfacing with the filesystem. None of Dart's IO APIs are natively
+   supported on JS, so for anything that needs to work on both the Dart VM _and_
+   Node.js we define a shim in the [`io`] directory that will be implemented in
+   terms of `dart:io` if we're running on the Dart VM or the `fs` or `process`
+   modules if we're running on Node. (We don't support IO at all on the browser
+   except to print messages to the console.)
+
+   [`io`]: io/README.md
+
+2. When exposing an API. Dart's JS interop is geared towards _consuming_ JS
+   libraries from Dart, not producing a JS library written in Dart, so we have
+   to jump through some hoops to make it work. This is all handled in the [`js`]
+   directory.
+
+   [`js`]: js/README.md
+
+## APIs
+
+One of Sass's core features is its APIs, which not only compile stylesheets but
+also allow users to provide plugins that can be invoked from within Sass. In
+both the JS API, the Dart API, and the embedded compiler, Sass provides three
+types of plugins: importers, custom functions, and loggers.
+
+### Importers
+
+Importers control how Sass loads stylesheets through `@use`, `@forward`, and
+`@import`. Internally, _all_ stylesheet loads are modeled as importers. When a
+user passes a load path to an API or compiles a stylesheet through the CLI, we
+just use the built-in [`FilesystemImporter`] which implements the same interface
+that we make available to users.
+
+[`FilesystemImporter`]: importer/filesystem.dart
+
+In the Dart API, the importer root class is [`importer/async_importer.dart`].
+The JS API and the embedded compiler wrap the Dart importer API in
+[`importer/node_to_dart`] and [`embedded/importer`] respectively.
+
+[`importer/async_importer.dart`]: importer/async_importer.dart
+[`importer/node_to_dart`]: importer/node_to_dart
+[`embedded/importer`]: embedded/importer
+
+### Custom Functions
+
+Custom functions are defined by users of the Sass API but invoked by Sass
+stylesheets. To a Sass stylesheet, they look like any other built-in function:
+users pass SassScript values to them and get SassScript values back. In fact,
+all the core Sass functions are implemented using the Dart custom function API.
+
+Because custom functions take and return SassScript values, that means we need
+to make _all_ values available to the various APIs. For Dart, this is
+straightforward: we need to have objects to represent those values anyway, so we
+just expose those objects publicly (with a few `@internal` annotations here and
+there to hide APIs we don't want users relying on). These value types live in
+the [`value`] directory.
+
+[`value`]: value/README.md
+
+Exposing values is a bit more complex for other platforms. For the JS API, we do
+a bit of metaprogramming in [`js/value`] so that we can return the
+same Dart values we use internally while still having them expose a JS API that
+feels native to that language. For the embedded host, we convert them to and
+from a protocol buffer representation in [`embedded/protofier.dart`].
+
+[`js/value`]: js/value/README.md
+[`embedded/value.dart`]: embedded/value.dart
+
+### Loggers
+
+Loggers are the simplest of the plugins. They're just callbacks that are invoked
+any time Dart Sass would emit a warning (from the language or from `@warn`) or a
+debug message from `@debug`. They're defined in:
+
+* [`logger.dart`](logger.dart) for Dart
+* [`js/logger.dart`](js/logger.dart) for Node
+* [`embedded/logger.dart`](embedded/logger.dart) for the embedded compiler
+
+## Built-In Functions
+
+All of Sass's built-in functions are defined in the [`functions`] directory,
+including both global functions and functions defined in core modules like
+`sass:math`. As mentioned before, these are defined using the standard custom
+function API, although in a few cases they use additional private features like
+the ability to define multiple overloads of the same function name.
+
+[`functions`]: functions/README.md
+
+## `@extend`
+
+The logic for Sass's `@extend` rule is particularly complex, since it requires
+Sass to not only parse selectors but to understand how to combine them and when
+they can be safely optimized away. Most of the logic for this is contained
+within the [`extend`] directory.
+
+[`extend`]: extend/README.md

lib/src/ast/css/README.md

@@ -0,0 +1,52 @@
+# CSS Abstract Syntax Tree
+
+This directory contains the abstract syntax tree that represents a plain CSS
+file generated by Sass compilation. It differs from other Sass ASTs in two major
+ways:
+
+1. Instead of being created by [a parser], it's created by [the evaluator] as it
+   traverses the [Sass AST].
+
+   [a parser]: ../../parse/README.md
+   [the evaluator]: ../../visitor/async_evaluate.dart
+   [Sass AST]: ../sass/README.md
+
+2. Because of various Sass features like `@extend` and at-rule hoisting, the CSS
+   AST is mutable even though all other ASTs are immutable.
+
+**Note:** the CSS AST doesn't have its own representation of declaration values.
+Instead, declaration values are represented as [`Value`] objects. This does mean
+that a CSS AST can be in a state where some of its values aren't representable
+in plain CSS (such as maps)—in this case, [the serializer] will emit an error.
+
+[`Value`]: ../../value/README.md
+[the serializer]: ../../visitor/serialize.dart
+
+## Mutable and Immutable Views
+
+Internally, the CSS AST is mutable to allow for operations like hoisting rules
+to the root of the AST and updating existing selectors when `@extend` rules are
+encountered. However, because mutability poses a high risk for "spooky [action
+at a distance]", we limit access to mutating APIs exclusively to the evaluator.
+
+[action at a distance]: https://en.wikipedia.org/wiki/Action_at_a_distance_(computer_programming)
+
+We do this by having an _unmodifiable_ interface (written in this directory) for
+each CSS AST node which only exposes members that don't modify the node in
+question. The implementations of those interfaces, which _do_ have modifying
+methods, live in the [`modifiable`] directory. We then universally refer to the
+immutable node interfaces except specifically in the evaluator, and the type
+system automatically ensures we don't accidentally mutate anything we don't
+intend to.
+
+[`modifiable`]: modifiable
+
+(Of course, it's always possible to cast an immutable node type to a mutable
+one, but that's a very clear code smell that a reviewer can easily identify.)
+
+## CSS Source Files
+
+A lesser-known fact about Sass is that it actually supports _three_ syntaxes for
+its source files: SCSS, the indented syntax, and plain CSS. But even when it
+parses plain CSS, it uses the Sass AST rather than the CSS AST to represent it
+so that parsing logic can easily be shared with the other stylesheet parsers.

lib/src/ast/sass/README.md

@@ -0,0 +1,34 @@
+# Sass Abstract Syntax Tree
+
+This directory contains the abstract syntax tree that represents a Sass source
+file, regardless of which syntax it was written in (SCSS, the indented syntax,
+or plain CSS). The AST is constructed recursively by [a parser] from the leaf
+nodes in towards the root, which allows it to be fully immutable.
+
+[a parser]: ../../parse/README.md
+
+The Sass AST is broken up into three categories:
+
+1. The [statement AST], which represents statement-level constructs like
+   variable assignments, style rules, and at-rules.
+
+   [statement AST]: statement
+
+2. The [expression AST], which represents SassScript expressions like function
+   calls, operations, and value literals.
+
+   [expression AST]: exprssion
+
+3. Miscellaneous AST nodes that are used by both statements and expressions or
+   don't fit cleanly into either category that live directly in this directory.
+
+The Sass AST nodes are processed (usually from the root [`Stylesheet`]) by [the
+evaluator], which runs the logic they encode and builds up a [CSS AST] that
+represents the compiled stylesheet. They can also be transformed back into Sass
+source using the `toString()` method. Since this is only ever used for debugging
+and doesn't need configuration or full-featured indentation tracking, it doesn't
+use a full visitor.
+
+[`Stylesheet`]: statement/stylesheet.dart
+[the evaluator]: ../../visitor/async_evaluate.dart
+[CSS AST]: ../css/README.md

lib/src/ast/selector/README.md

@@ -0,0 +1,24 @@
+# Selector Abstract Syntax Tree
+
+This directory contains the abstract syntax tree that represents a parsed CSS
+selector. This AST is constructed recursively by [the selector parser]. It's
+fully immutable.
+
+[the selector parser]: ../../parse/selector.dart
+
+Unlike the [Sass AST], which is parsed from a raw source string before being
+evaluated, the selector AST is parsed _during evaluation_. This is necessary to
+ensure that there's a chance to resolve interpolation before fully parsing the
+selectors in question.
+
+[Sass AST]: ../sass/README.md
+
+Although this AST doesn't include any SassScript, it _does_ include a few
+Sass-specific constructs: the [parent selector] `&` and [placeholder selectors].
+Parent selectors are resolved by [the evaluator] before it hands the AST off to
+[the serializer], while placeholders are omitted in the serializer itself.
+
+[parent selector]: parent.dart
+[placeholder selectors]: placeholder.dart
+[the evaluator]: ../../visitor/async_evaluate.dart
+[the serializer]: ../../visitor/serialize.dart

lib/src/embedded/README.md

@@ -0,0 +1,28 @@
+# Embedded Sass Compiler
+
+This directory contains the Dart Sass embedded compiler. This is a special mode
+of the Dart Sass command-line executable, only supported on the Dart VM, in
+which it uses stdin and stdout to communicate with another endpoint, the
+"embedded host", using a protocol buffer-based protocol. See [the embedded
+protocol specification] for details.
+
+[the embedded protocol specification]: https://github.com/sass/sass/blob/main/spec/embedded-protocol.md
+
+The embedded compiler has two different levels of dispatchers for handling
+incoming messages from the embedded host:
+
+1. The [`IsolateDispatcher`] is the first recipient of each packet. It decodes
+   the packets _just enough_ to determine which compilation they belong to, and
+   forwards them to the appropriate compilation dispatcher. It also parses and
+   handles messages that aren't compilation specific, such as `VersionRequest`.
+
+   [`IsolateDispatcher`]: isolate_dispatcher.dart
+
+2. The [`CompilationDispatcher`] fully parses and handles messages for a single
+   compilation. Each `CompilationDispatcher` runs in a separate isolate so that
+   the embedded compiler can run multiple compilations in parallel.
+
+   [`CompilationDispatcher`]: compilation_dispatcher.dart
+
+Otherwise, most of the code in this directory just wraps Dart APIs to
+communicate with their protocol buffer equivalents.