dossierfacile-data

data transformation for DossierFacile using DBT framework

Install DBT

Install DBT in a virtual environment

python -m venv .venv
source .venv/bin/activate

pip install dbt-core dbt-postgres

Configure DBT

Create a profiles.yml file in the root of the project

dossierfacile:
  target: dev
  outputs:
    dev:
      type: postgres
      host: {{ host }}
      user: {{ user}}
      password: {{ password }}
      port: {{ port }}
      dbname: {{ database }}
      schema: {{ schema }}
      threads: 3

Replace the {{}} with the values in the vaultwarden secret "user SQL df-data dbt_dev"

To test the connection, run the following command:

dbt debug

Sqlfluff linter

Install Sqlfluff in a virtual environment:

source .venv/bin/activate
pip install sqlfluff sqlfluff-templater-dbt

Sqlfluff usage:

sqlfluff lint #to lint all files
sqlfluff fix #to fix the files

Pre-commit hook

Install pre-commit:

pip install pre-commit

You can add sqlfluff fix to the pre-commit hook by adding the following to the .pre-commit-config.yaml file:

repos:
  - repo: https://github.com/sqlfluff/sqlfluff
    rev: 3.3.0
    hooks:
      - id: sqlfluff-lint
      - id: sqlfluff-fix

To add the pre-commit hook, run the following command:

pre-commit install

Run DBT

Try running the following commands:

dbt run

Select a model to run:

dbt run -s model_name