include does not inject code into a non-executable block

[aronatkins]

Bug description

Using Quarto 0.9.607 (contained in a recent RStudio build -- 2022.10.0 Build 9), but have also tested Quarto 0.9.629. The same rendered results occur with both versions, and within and outside the RStudio IDE.

Observed on macOS Monterey - 12.4

Given index.qmd defined as:

---
title: embedding code using includes
---

## in a code block

The `cats.R` file should be injected into the R code block and correctly 
formatted without being executed.

```r
{{< include cats.R >}}
```

## outside a code block

Here is an example showing how the R code can be injected, but it does not
have the desired code-block styling.

{{< include cats.R >}}

## expected

This is the expected result when formatted:

```r
cat("cats!\n")
```

and a cats.R file defined as:

cat("cats!\n")

The rendering should present the code from cats.R as formatted R code in the first code block (containing the include).

Instead, the rendered result has an empty code block and looks like:

Checklist

• formatted your issue so it is easier for us to read?
• included a minimal, self-contained, and reproducible example?
• documented the quarto version you're running, by providing the output produced by quarto check in a terminal in your issue?
• documented the RStudio IDE version you're running (if applicable), by providing the value displayed in the "About RStudio" main menu dialog?
• documented which operating system you're running? If on Linux, please provide the specific distribution as well.
• upgraded to the latest version, including your versions of R, the RStudio IDE, and relevant R packages?

[cscheid]

Thanks! I can repro this locally.

[cscheid]

The "issue" here is that shortcodes are currently only detected as "standalone" markdown constructs. This is admittedly bad, but I'm concerned about changing this ahead of v1.0.

It's not the only problem with shortcodes, either. A related problem happens, for example, if we try including from inside a bulleted list:

* Hello.

  This is a paragraph.

  Include?

  {{< include cats.R >}}

* Another item.

The result looks like so:

This happens because our detection of shortcodes is not syntax-aware, and doesn't know to expand the file by including the bulleted list indentation.

[aronatkins]

Codes like {{< meta ... >}} and {{< var ... >}} also cannot be used within code blocks, probably for similar reasons.

[jjallaire]

Those definitely should be working in code blocks (here we specifically look for them: https://github.com/quarto-dev/quarto-cli/blob/main/src/resources/filters/quarto-pre/shortcodes.lua#L14). If they don't work its a regression we need to fix for v1.0 (as we currently use that in the quarto website and RS Workbench docs)

[jjallaire]

@aronatkins These do work in static/display code blocks (i.e. non-executable) but they don't work for executable code blocks (and they will likely not ever work inside executable blocks).

[jjallaire]

Note that knitr already has a native feature for specifying the contents of code chunk from an external file (file option): https://quarto.org/docs/reference/cells/cells-knitr.html#include

[jjallaire]

One line we don't want to cross is allowing content in code chunks/cells that is not syntactically valid for the language (which in turn prevents cell-by-cell execution in RStudio, VS Code, Jupyter Lab, etc.)

[aronatkins]

Thanks @jjallaire - I've confirmed that shortcodes like env work in non-executable code chunks. Keep this issue only against include.

In case others have similar problems:

My workaround to have code examples included in plain code blocks is to use the knitr engine embed. For example, here is how I have included an R script within the document:

---
title: embedded example
engine: knitr
---

```{embed, file = "environ.R"}
```

To have values available throughout the document, I am currently using environment variables, as they can be accessed in code (using language-specific techniques), in prose, in YAML fields, etc.

---
title: "Going {{< env HOME >}}"
engine: knitr
---

It is so nice to go {{< env HOME >}}

Using environment variables in code blocks using language-specific references:

```{bash}
echo "${HOME}"
```

```{r}
cat(Sys.getenv("HOME"))
```

Environment variables in code blocks (not executable):

```r
echo "{{< env HOME >}}"
```

[BeastyBlacksmith]

For those looking for a workaround: in julia you can use the following to include the content of source files (similar should work for other languages):

```{julia}
#| echo: false
using Markdown
content = read("filename.jl", String)
Markdown.parse(string(
"```julia\n",
content,
"\n```"
))

[aronatkins]

@cscheid - this issue needs to be reopened; include cannot be used to inject code into simple, non-executable blocks.

[jjallaire]

As a stopgap we created this extension: https://github.com/quarto-ext/include-code-files

But yes, we do need to make this work (and also create a built-in feature similar to what the extension does)

[cscheid]

The reason we haven't easily resolved for include specifically is that the include shortcode resolution happens in typescript, before Pandoc. Other shortcodes like meta, var, etc, happen in Lua.

We need to add include processing to Lua in one shape or another, but this is trickier than it looks. We need to be aware of a number of issues (making sure that paths etc are resolved identically in Lua and Typescript). It's also going to be the case that a "Lua {{< include >}}" will behave differently than a typescript one out of necessity (in Typescript, we can create includes that break syntactic boundaries; in Lua, that won't be the case).

This is going to be potentially confusing for users. I wonder if we should have different names for each.

[jjallaire]

I think we discussed having include be special and get treated as a strict pre-processor (perhaps w/ some sort of line based parsing).

[aronatkins]

I have successfully used the https://github.com/quarto-ext/include-code-files filter in a revealjs presentation. Thanks for the pointer.