- Sponsor
-
Notifications
You must be signed in to change notification settings - Fork 756
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added dbt incremental macro to default_config.cfg #363
Added dbt incremental macro to default_config.cfg #363
Conversation
@azhard - I like the thinking here. There's already an implementation of That other implementation is there to allow these macros to be used in macro folders. I think we might need a refactor of some of this code more broadly. I'm curious for your views on what we should do with the |
Hm that's actually a really good point about wanting to test the logic for both Then on pass / fail your output would be something like
Of course I'm sure you'd inevitably end up with a scenario where there are multiple conditions used per file which gets more complicated but can potentially still be handled by looping through all permutations. Output for that case might be more like:
Also not familiar enough with the parser to know if something like that is even feasible but those are some initial thoughts |
I totally agree with your intent of having multiple runs of the files. On whether it's feasible that's an excellent question! I'm thinking more and more that trying to "mock" the dbt compile step is more and more of a headache to maintain. Maybe we should just use the dbt compile step from dbt itself (and then use all the dependencies and modules that people have installed)? In this case we would do two compiles, one for incremental and one for not. Thoughts? |
I like that idea a lot! My only concern would be with macros where dbt adds SQL which might not conform to the sqlfluff linting rules eg.
SQL
If you use dbt and compile this you might end up with linting errors around extra line breaks or incorrect tabbing |
@azhard - I think you're totally right. In my head we have a few options:
Which do you think would be the most sensible to start with? |
I'm inclined to pick a hybrid of 2 to start although 3 would definitely be a great long-term solution. My thinking against 1 is like you mentioned in some cases it might be impossible to have the formats match as well as the fact that if 3 is the ideal long term goal, people might spend unnecessary time cleaning up macros which becomes redundant after 3 is release. Allowing bugs to go through with 2 is a bit iffy as people are probably reliant on |
In #508 we are implementing a templater that re-uses the dbt compiler In #541 there is mapping between the source "pre-templating" and "post-templating" code, which should allow work on downgrading or silencing errors coming from templated code (#563) From #554 @alanmcruickshank you say there's still potentially value in supporting the JinjaTemplater (and Built-in macros) so this PR should still be merged! On the following:
I'm wondering if we wouldn't get into a lot of complexity. For example if there are 2 different |
Currently when using the
is_incremental
dbt macro inside a SQL file, sqlfluff gets confused such as in the following file:This change sets the is_incremental macro to always evaluate to true so that the full SQL in the file can be evaluated.