Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid caching schemas for tests when store_failures is not enabled #3715

Closed
jtcohen6 opened this issue Aug 10, 2021 · 0 comments · Fixed by #3716
Closed

Avoid caching schemas for tests when store_failures is not enabled #3715

jtcohen6 opened this issue Aug 10, 2021 · 0 comments · Fixed by #3716
Labels
enhancement New feature or request

Comments

@jtcohen6
Copy link
Contributor

In dbt v0.20.0, tests are now configured by default to use a schema named dbt_test__audit, which is useful for the "store failures" feature. On some databases, this is causing dbt to run an extra metadata query when --store-failures is not enabled.

On Snowflake, because of how we list objects to populate the relational cache, such a metadata query often appears in the query history as a failed statement (show terse objects in database.dbt_test__audit), leading to user confusion.

https://github.com/dbt-labs/dbt/blob/43642956a21571e4589d3f7b9a7ad69439c6d4cc/plugins/snowflake/dbt/adapters/snowflake/impl.py#L135-L141

I view this as a low-priority cosmetic issue, with a simple fix: We could change the logic in _get_cache_schemas to check for relational nodes (ones that map to database objects) instead of executable nodes (ones dbt knows how to compile/execute):

https://github.com/dbt-labs/dbt/blob/43642956a21571e4589d3f7b9a7ad69439c6d4cc/core/dbt/adapters/base/impl.py#L304-L316

            for node in manifest.nodes.values()
            if (
                node.is_relational and not node.is_ephemeral_model
            )

The is_relational method already includes logic around whether to consider tests relational:
https://github.com/dbt-labs/dbt/blob/43642956a21571e4589d3f7b9a7ad69439c6d4cc/core/dbt/contracts/graph/parsed.py#L127-L133

And this is the same method we now use inside runnable tasks, to determine which schemas need creating:
https://github.com/dbt-labs/dbt/blob/43642956a21571e4589d3f7b9a7ad69439c6d4cc/core/dbt/task/runnable.py#L460-L462

This would remove Analysis, Operation, Documentation, and RPCCall from the schema-caching list, but frankly it doesn't make much sense that they were there in the first place!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant