🪓 Split Output into Outputs → Output[]

[agoose77]

This PR closes #1674 by introducing two new versions (1* and 2) of the MyST AST:

Version 2

type CodeBlock = {
  type: "block";
  kind: "notebook-code",
  children: [
    Code,
    Outputs
  ]
}

type Outputs = {
  type: "outputs";
  children: Output[];
  visibility?: ...;
  identifier?: ...;
  html_id?: ...;
  label?: ...;
}

type Output = {
  type: "output";
  children: GenericNode[];
  jupyter_data: IOutput;
  identifier?: ...;
  html_id?: ...;
  label?: ...;
}

Version 1*

type CodeBlock = {
  type: "block";
  kind: "notebook-code",
  children: [
    Code,
    Output
  ]
}
type Output = {
  type: "output";
  children: GenericNode[];
  data: IOutput[];
  _future_ast: Outputs
}

This PR does the following things with these AST types:

1. Recognise version 1* and 2 in input data (e.g. .myst.json or XRefs) and upgrade it to version 2
2. Process MyST AST in version 2
3. Export MyST AST to disk as version 1* (to avoid breaking old consumers)

The idea here is to start publishing a modified version of the current schema that is forward-compatible (version 1*) ASAP. Then, at some future moment, we will trigger an AST upgrade and start publishing version 2 in a future build of MyST.

Versions of mystmd that include #1671 will recognise this AST, and have no problems in handling it.

Warning

Upgrading from 1 to 2 is a lossy process — there is no obvious way to figure out which output.children belong to which output.data iff. the output.data.length > 1. I think that's a rare case. If we really care about it, we can look to store the original version 1 AST to support round-tripping.

This PR does not yet solve #1734 entirely -- for that, we need to pull in the myst-compat package more dynamically, so that it can be pulled in for old mystmd binaries.

To Do

• Validate behavior of reduceOutputs -- do we want this removing Outputs nodes?
• Validate change to blockMetadataTransform that leaves empty Outputs nodes
• Apply block data labelling after execution (follow up)
• Investigate consequences of parsing Markdown & images etc. before making a release

[changeset-bot]

🦋 Changeset detected

Latest commit: 90ac03d

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 13 packages

Name	Type
myst-spec-ext	Minor
mystmd	Minor
myst-directives	Patch
myst-execute	Patch
myst-compat	Patch
myst-cli	Minor
myst-common	Minor
myst-config	Minor
myst-frontmatter	Minor
myst-parser	Patch
myst-roles	Patch
myst-to-html	Patch
myst-transforms	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[rowanc1]

From meeting:

• Split out the compat repo.
• Make it simple, e.g. footnote number --> enumerated (currently)
• merge that!
• Open an outputs pr on top (only focus on breaking changes, no _futureAst)
• Make that at the same time for a theme PR. We release those at the same time.
• Them do the markdown from outputs PR.

[agoose77]

Closing this in favour of a rework as a simple package