[Feature] User Defined Macros & Filters

[afranzi]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

The main idea would be allowing DBT users to define and provide Python macros & filters to be used inside the Jinja templating, similar to how Airflow works when creating a new DAG.

Describe alternatives you've considered

No response

Who will this benefit?

Everyone looking to expand their DBT jinja capabilities with custom macros & filters.

Some use cases that could come to my mind:

• Loading configuration from an API
• Loading credentials from AWS secrets
• Moving complex jinja logic from SQL files to native Python.

Are you interested in contributing this feature?

Yes

Anything else?

https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html

[jtcohen6]

Hey @afranzi! Thanks for opening the issue here, and for the accompanying PR as well. Sorry for the delay getting back to you.

We've discussed a very similar proposal twice before, in May 2022 & in June 2023:

• draft: allow users to augment their jinja context with python objects dbt-core#5274 (comment)
• [CT-2769] [Feature] Allow yaml/jinja to get rendered with additional context dbt-core#8000

Our answer now (February 2023) is going to be the same as before: We won't be creating a new entry point to inject arbitrary user code into the Jinja context.

We already have a supported interface for this: adapters, which can define Python methods and expose them as callables in the Jinja context. We've done a bunch of work over the past few months to solidify the adapter interface (dbt-labs/dbt-adapters#87), and we've documented how to create your own adapter (https://docs.getdbt.com/guides/adapter-creation). It's also possible to extend an existing adapter, as dbt-synapse extends dbt-fabric or dbt-materialize extends dbt-postgres.

I get the benefit of rewriting complex Jinja macros as static Python functions, to take advantage of more mature unit testing capabilities. (Though we've had some thoughts here!) The other two points — loading configuration from an API, or credentials from AWS secrets — are outside the immediate scope of dbt-core, which knows only:

• how to load configuration from files, environment variables, and CLI arguments
• how to authenticate & connect to the data warehouse

I'm going to close this as unplanned. Thanks again for the time you spent thinking through the problem, poking at the code, and writing up the issue.

[afranzi]

Hey @jtcohen6, thank you for the detailed explanation about the adapters and the reasons behind them.

However, I was wondering if we could have any possibility to discuss and iterate this proposal to include it as part of the immediate scope so we can impact positively the community. The code is ready to be merged, and it's not adding any breaking change.

[ismailsimsek]

@jtcohen6 it will be crazy to duplicate and create custom adapter only to inject single python function!
+1 to discuss and iterate this proposal.

[ismailsimsek]

@jtcohen6 @afranzi created discussion here dbt-labs/dbt-adapters#259

IMO Adapter factory should allow user provided customized adapter class