Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

catalog: use an Operation to generate the SQL result #765

Closed
cmcarthur opened this issue May 9, 2018 · 1 comment
Closed

catalog: use an Operation to generate the SQL result #765

cmcarthur opened this issue May 9, 2018 · 1 comment

Comments

@cmcarthur
Copy link
Member

right now, we have a SQL query inline in the adapter that we use to generate the catalog. we want this to be a SQL script that's included in the global project and that is executed via a dbt "Operation."

there's an Operation node type in node_types.py -- that is the thing we want to use.

there's no OperationRunner in node_runners.py. we'll need one of those.

an operation is just arbitrary sql that gets run, committed, and has the result optionally returned. operations can be selected from the dbt graph just like any other node. but note that there is no CLI syntax for selecting operations right now. you can use dbt internal code to do that.

so the task flow would be:

  • generate parsedmanifest
  • generate schemas list
  • find this special operation in the graph
    • this should probably be done inside the adapter function that you added previously so that we can add bigquery support later on. it's a good idea to add a helper function to dbt that allows us to find and call an operation by name very simply, because there could be other places in dbt that we want to create similar behavior
  • call it and fetch the result
  • process the result into the catalog
@drewbanin
Copy link
Contributor

Added in #768 - closing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants