A project template for building, testing, scheduling, and running an ELT data pipeline as a SQL workflow using:
- dbt-core
- Github Actions used as runner, scheduler, and orchestrator.
- Python virtual environment.
- Snowflake as a data warehouse option.
- BigQuery as a data warehouse option.
- sqlfluff used for linting SQL files
- sqlfmt used for formatting SQL files
Run bash setup.sh once to install project dependencies and configure the desired data warehouse and agent connection.
Run the following target commands to execute the desired SQL workflow operation:
dbt compiledbt testdbt rundbt docs generatedbt docs serve
git checkout main
The main branch is read-only.
git checkout -b <feature-name>
Raise a PR
Merge PR to main branch
- Follow Mozilla Data Documentation's (SQL Style Guide)[https://docs.telemetry.mozilla.org/concepts/sql_style#sql-style-guide]
- See dbt docs on Best Practices
By default, everything gets created as a view. You can override that at the directory level so everything in that directory will materialize to a different materialization.
- See repository notes on dbt
- Learn more about dbt in the docs
- Check out Discourse for commonly asked questions and answers
- Join the chat on Slack for live discussions and support
- Find dbt events near you
- Check out the blog for the latest news on dbt's development and best practices