Example of running dbt against steampipe.
Steampipe spins up a Postgres database, with a writeable public
schema so we can re-use the Postgres adapter.
This docker compose environment mounts local AWS and GCP credentials to access the APIs from the container.
- Build the image: change the plugins list to what you need, in the docker-compose.override.yml file
docker compose build steampipe
- Start steampipe:
docker compose up
- Install dbt dependencies:
pip install -r requirements.txt
You can query de database through psql
(password: steampipe
s):
psql -h localhost -p 9193 -U steampipe -d steampipe
Or open a steampipe query shell:
docker compose exec steampipe steampipe query
In this example I pull the number of bytes processed by GCP BigQuery jobs and join it to a simple seed to figure out how much they cost.
The model bq_billable_bytes.sql uses a steampipe table so you might incur some API call costs if you run it.
-
Update conf/config.spc with your GCP project
-
Run
cd steampipe_example dbt seed --full-refresh dbt run