Use Variables and Macros in Markup Documentation (Like Overview)

[brittianwarner]

Describe the feature

I want to be able to use variables and macros to dynamically populate the documentation markup descriptions.

Describe alternatives you've considered

I'm not aware of any alternatives.

Who will this benefit?

Anyone who wants to leverage variables and macros for generating documentation.

[jtcohen6]

@brittianwarner Thanks for opening! I noticed your title's inclusion of Overview. What's your specific use case for having a dynamic overview that includes macros/variables?

Beyond that, this feels like a neat way of recasting a familiar proposal (#3277), which ties into the larger discussion around extending/inheriting model properties in #2995 (and really kicked off a long time ago by #1158).

What if:

• The description rendering context included contextual variables (e.g. {{ model }}, {{ column_name }}), in addition to what it includes today (docs blocks, vars, env vars).
• Docs blocks were "late rendered," like python f-strings, such that they could be defined with context variables, vars, env vars. If any of those values are missing in the rendering context, dbt would return a parsing error.
• There were a clearly specified + documented context available to "special" docs blocks, such as __overview__, which would be "eagerly rendered" instead

This pulls on a similar thread as some more recent thinking about generic test descriptions (#3249) and test aliases (#3815).

I'm really unsure about whether docs blocks should support calling static macros in their rendering context, or if we should instead make it possible for docs blocks to call other docs blocks, which would accomplish the same purpose. (Docs blocks are really just a special kind of macro, with a more focused and rigid purpose. Should we make docs blocks even more like macros, and accept arguments?)

In any event, this is something I want to keep thinking about, possibly for next year. We aren't going to have the ability to do this ahead of releasing dbt v1.0 later this year.

In the meantime, even though docs blocks cannot reference each other, the description rendering context does include variables, at least. As long as you're able to write your descriptions in a non-overlapping way, you can do things like:

models:
  - name: model_a
    description: >
      {{ doc('generic_explanation') }}
      In particular, {{ var('specific_info')}} {{ env_var('CONTEXTUAL_INFO')}}

I'm inclined to close this issue as a duplicate of #3277, while recognizing there are some subtle differences, in the hopes of keeping the conversation about dynamic descriptions as centralized as possible. But first, I want to hear a bit more about what you're thinking around dynamic __overview__, since that's a piece of the puzzle we hadn't yet considered.

[brittianwarner]

Thanks for the write-up! Our scenario includes the possibility of hundreds of packages where each one would have its own documentation. In order to scale and create richer documentation with dynamic links and other features we would need to be able to use variables at a minimum. I will try to use the description section in the meantime to see if this works. Thanks again for going through in detail.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest; add a comment to notify the maintainers.