Merge pull request #82 from fivetran/feature/standardized-billing-lin…

…e-item-model Feature/standardized billing line item model
fivetran · Aug 14, 2024 · a912d17 · a912d17
2 parents 2f792bb + 4cabd5d
commit a912d17
Show file tree

Hide file tree

Showing 23 changed files with 581 additions and 20 deletions.
diff --git a/.DS_Store b/.DS_Store
diff --git a/.gitignore b/.gitignore
@@ -4,4 +4,5 @@ dbt_modules/
 logs/
 env/
 dbt_packages/
-package-lock.yml
+package-lock.yml
+.DS_Store
diff --git a/.quickstart/quickstart.yml b/.quickstart/quickstart.yml
@@ -12,6 +12,7 @@ table_variables:
     - payment_method
   stripe__using_subscriptions:
     - subscription_history
+    - product
   stripe__using_credit_notes:
     - credit_note_line_item
     - credit_note

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,19 @@
+# dbt_stripe v0.14.0
+[PR #82](https://github.com/fivetran/dbt_stripe/pull/82) includes the following updates:
+
+## Feature Updates
+- Addition of the `stripe__line_item_enhanced` model. This model constructs a comprehensive, denormalized analytical table that enables reporting on key revenue, subscription, customer, and product metrics from your billing platform. It’s designed to align with the schema of the `*__line_item_enhanced` model found in Stripe, Recharge, Recurly, Shopify, and Zuora, offering standardized reporting across various billing platforms. To see the kinds of insights this model can generate, explore example visualizations in the [Fivetran Billing Model Streamlit App](https://fivetran-billing-model.streamlit.app/). Visit the app for more details.
+  - This model is currently disabled by default. You may enable it by setting the `stripe__standardized_billing_model_enabled` as `true` in your `dbt_project.yml`.
+
+## Relevant Upstream Updates ([dbt_stripe_source v0.12.0](https://github.com/fivetran/dbt_stripe_source/releases/tag/v0.12.0))
+- Addition of the following new staging models and accompanying upstream references:
+  - `stg_stripe__discount` (required for downstream `dbt_stripe` model transformations)
+  - `stg_stripe__product` (enabled by default, but can be disabled by setting the `stripe__using_subscriptions` variable to `false`)
+
+## Under the Hood
+- Added consistency test within integration_tests for the `stripe__line_item_enhanced` model.
+- Updated the `quickstart.yml` to include the `product` source table as a requirement for the `stripe__using_subscriptions` variable.
+
 # dbt_stripe v0.13.0
 [PR #78](https://github.com/fivetran/dbt_stripe/pull/78) includes the following updates:
 

diff --git a/README.md b/README.md
@@ -29,14 +29,25 @@ The following table provides a detailed list of all models materialized within t
 |--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | [stripe__balance_transactions](https://fivetran.github.io/dbt_stripe/#!/model/model.stripe.stripe__balance_transactions)    | Each record represents a change to your account balance, enriched with data about the transaction.                                                                                                                                       |
 | [stripe__invoice_details](https://fivetran.github.io/dbt_stripe/#!/model/model.stripe.stripe__invoice_details)      | Each record represents an invoice, enriched with details about the associated charge, customer, and subscription data.                                                                                         
-| [stripe__invoice_line_item_details](https://fivetran.github.io/dbt_stripe/#!/model/model.stripe.stripe__invoice_line_item_details)      | Each record represents an invoice line item, enriched with details about the associated charge, customer, subscription, and pricing data.                                                                                 
+| [stripe__invoice_line_item_details](https://fivetran.github.io/dbt_stripe/#!/model/model.stripe.stripe__invoice_line_item_details)      | Each record represents an invoice line item, enriched with details about the associated charge, customer, subscription, and pricing data.  
 | [stripe__daily_overview](https://fivetran.github.io/dbt_stripe/#!/model/model.stripe.stripe__daily_overview)      | Each record represents, per day, a summary of daily totals and rolling totals by transaction type (balances, payments, refunds, payouts, and other transactions). You may use this model to roll up into weekly, quarterly, monthly, and other time grains. You may also use this model to create a MRR report.                                                  |
 | [stripe__subscription_details](https://fivetran.github.io/dbt_stripe/#!/model/model.stripe.stripe__subscription_details)    | Each record represents a subscription, enriched with customer details and payment aggregations.                                                                                                                                          |
 | [stripe__customer_overview](https://fivetran.github.io/dbt_stripe/#!/model/model.stripe.stripe__customer_overview)       | Each record represents a customer, enriched with metrics about their associated transactions.  Transactions with no associated customer will have a customer description of "No associated customer".                                                                                                                                               |
 | [stripe__activity_itemized_2](https://fivetran.github.io/dbt_stripe/#!/model/model.stripe.stripe__activity_itemized_2)       |  This report displays balance transactions alongside associated customer, charge, refund, fee, and invoice details, useful for Interchange Plus (IC+) pricing users.                                                                                                                                             |
 | [stripe__balance_change_from_activity_itemized_3](https://fivetran.github.io/dbt_stripe/#!/model/model.stripe.stripe__balance_change_from_activity_itemized_3)       | This report functions like a bank statement for reconciling your Stripe balance, especially beneficial for treating Stripe as a bank account for accounting purposes. It offers a detailed breakdown of transactions affecting your balance.                                                                                                                                               |
 | [stripe__ending_balance_reconciliation_itemized_4](https://fivetran.github.io/dbt_stripe/#!/model/model.stripe.stripe__ending_balance_reconciliation_itemized_4)       | This reports models after Stripe's payout reconciliation reports, helping match bank account payouts with related transactions. It provides details for automatic payouts and transactions that hadn't settled as of the report's end date. This report is only available for users with automatic payouts enabled and optimized for those who prefer to reconcile the transactions included in each payout as a settlement batch.                                                                                                                                               |
-| [stripe__payout_itemized_3](https://fivetran.github.io/dbt_stripe/#!/model/model.stripe.stripe__payout_itemized_3)       | This report represents payouts with information on expected arrival dates and status, akin to a bank statement for reconciling your Stripe balance, particularly useful for accounting purposes.                                                                                                                                                      |
+| [stripe__payout_itemized_3](https://fivetran.github.io/dbt_stripe/#!/model/model.stripe.stripe__payout_itemized_3)       | This report represents payouts with information on expected arrival dates and status, akin to a bank statement for reconciling your Stripe balance, particularly useful for accounting purposes.                                                                                                                                             
+| [stripe__line_item_enhanced](https://fivetran.github.io/dbt_stripe/#!/model/model.stripe.stripe__line_item_enhanced)       | This model constructs a comprehensive, denormalized analytical table that enables reporting on key revenue, subscription, customer, and product metrics from your billing platform. It’s designed to align with the schema of the `*__line_item_enhanced` model found in Stripe, Recharge, Recurly, Shopify, and Zuora, offering standardized reporting across various billing platforms. To see the kinds of insights this model can generate, explore example visualizations in the [Fivetran Billing Model Streamlit App](https://fivetran-billing-model.streamlit.app/). Visit the app for more details.  |
+
+## Example Visualizations
+Curious what these models can do? Check out example visualizations from the [stripe__line_item_enhanced](https://fivetran.github.io/dbt_stripe/#!/model/model.stripe.stripe__line_item_enhanced) model in the [Fivetran Billing Model Streamlit App](https://fivetran-billing-model.streamlit.app/), and see how you can use these models in your own reporting. Below is a screenshot of an example report—explore the app for more.
+
+<p align="center">
+  <a href="https://fivetran-billing-model.streamlit.app/">
+    <img src="https://raw.githubusercontent.com/fivetran/dbt_stripe/main/images/streamlit_example.png" alt="Streamlit Billing Model App" width="75%">
+  </a>
+</p>
+
 <!--section-end-->
 
 # 🎯 How do I use the dbt package?
@@ -61,8 +72,7 @@ Include the following stripe package version in your `packages.yml` file:
 ```yaml
 packages:
   - package: fivetran/stripe
-    version: [">=0.13.0", "<0.14.0"]
-
+    version: [">=0.14.0", "<0.15.0"]
 ```
 Do **NOT** include the `stripe_source` package in this file. The transformation package itself has a dependency on it and will install the source package as well. 
 
@@ -89,7 +99,15 @@ vars:
     stripe__using_credit_notes:    True   #Enable if you are using the credit note tables.
 ```
 ## (Optional) Step 5: Additional configurations
-<details><summary>Expand for configurations</summary>
+<details open><summary>Expand to view configurations</summary>
+
+### Enabling Standardized Billing Model
+This package contains the `stripe__line_item_enhanced` model which constructs a comprehensive, denormalized analytical table that enables reporting on key revenue, subscription, customer, and product metrics from your billing platform. It’s designed to align with the schema of the `*__line_item_enhanced` model found in Recurly, Recharge, Stripe, Shopify, and Zuora, offering standardized reporting across various billing platforms. To see the kinds of insights this model can generate, explore example visualizations in the [Fivetran Billing Model Streamlit App](https://fivetran-billing-model.streamlit.app/). For the time being, this model is disabled by default. If you would like to enable this model you will need to adjust the `stripe__standardized_billing_model_enabled` variable to be `true` within your `dbt_project.yml`:
+
+```yml
+vars:
+  stripe__standardized_billing_model_enabled: true # false by default.
+```
 
 ### Unioning Multiple Stripe Connectors
 If you have multiple Stripe connectors you would like to use this package on simultaneously, we have added the ability to do so. Data from disparate connectors will be unioned together and be passed downstream to the end models. The `source_relation` column will specify where each record comes from. To use this functionality, you will need to either set the `stripe_union_schemas` or `stripe_union_databases` variables. Please also make sure the single-source `stripe_database` and `stripe_schema` variables are removed.
@@ -236,7 +254,7 @@ This dbt package is dependent on the following dbt packages. Please be aware tha
 ```yml
 packages:
     - package: fivetran/stripe_source
-      version: [">=0.11.0", "<0.12.0"]
+      version: [">=0.12.0", "<0.13.0"]
 
     - package: fivetran/fivetran_utils
       version: [">=0.4.0", "<0.5.0"]

diff --git a/dbt_project.yml b/dbt_project.yml
@@ -1,7 +1,7 @@
 config-version: 2
 name: 'stripe'
 
-version: '0.13.0'
+version: '0.14.0'
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 models:
   stripe:
@@ -17,6 +17,7 @@ vars:
     charge: "{{ ref('stg_stripe__charge') }}"
     customer: "{{ ref('stg_stripe__customer') }}"
     dispute: "{{ ref('stg_stripe__dispute') }}"
+    discount: "{{ ref('stg_stripe__discount') }}"
     fee: "{{ ref('stg_stripe__fee') }}"
     invoice: "{{ ref('stg_stripe__invoice') }}"
     invoice_line_item: "{{ ref('stg_stripe__invoice_line_item') }}"
@@ -25,6 +26,7 @@ vars:
     payment_method: "{{ ref('stg_stripe__payment_method') }}"
     payout: "{{ ref('stg_stripe__payout') }}"
     price_plan: "{{ ref('stg_stripe__price_plan') }}"
+    product: "{{ ref('stg_stripe__product') }}"
     refund: "{{ ref('stg_stripe__refund') }}"
     subscription: "{{ ref('stg_stripe__subscription') }}"
     transfer: "{{ ref('stg_stripe__transfer') }}"
diff --git a/docs/catalog.json b/docs/catalog.json
diff --git a/docs/manifest.json b/docs/manifest.json
diff --git a/docs/run_results.json b/docs/run_results.json
diff --git a/images/streamlit_example.png b/images/streamlit_example.png
diff --git a/integration_tests/ci/sample.profiles.yml b/integration_tests/ci/sample.profiles.yml
@@ -16,13 +16,13 @@ integration_tests:
       pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
       dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
       port: 5439
-      schema: stripe_integrations_tests_11
+      schema: stripe_integrations_tests_14
       threads: 8
     bigquery:
       type: bigquery
       method: service-account-json
       project: 'dbt-package-testing'
-      schema: stripe_integrations_tests_11
+      schema: stripe_integrations_tests_14
       threads: 8
       keyfile_json: "{{ env_var('GCLOUD_SERVICE_KEY') | as_native }}"
     snowflake:
@@ -33,7 +33,7 @@ integration_tests:
       role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
       database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
       warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
-      schema: stripe_integrations_tests_11
+      schema: stripe_integrations_tests_14
       threads: 8
     postgres:
       type: postgres
@@ -42,13 +42,13 @@ integration_tests:
       pass: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
       dbname: "{{ env_var('CI_POSTGRES_DBT_DBNAME') }}"
       port: 5432
-      schema: stripe_integrations_tests_11
+      schema: stripe_integrations_tests_14
       threads: 8
     databricks:
       catalog: "{{ env_var('CI_DATABRICKS_DBT_CATALOG') }}"
       host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
       http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
-      schema: stripe_integrations_tests_11
+      schema: stripe_integrations_tests_14
       threads: 8
       token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
       type: databricks
diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml
@@ -1,7 +1,7 @@
 config-version: 2
 
 name: 'stripe_integration_tests'
-version: '0.13.0'
+version: '0.14.0'
 
 profile: 'integration_tests'
 
@@ -10,14 +10,15 @@ models:
   +schema: "stripe_{{ var('directed_schema','dev') }}"
 
 vars:
-  stripe_schema: stripe_integrations_tests_11
+  stripe_schema: stripe_integrations_tests_14
   stripe_source:
     stripe_account_identifier:                "account_data"
     stripe_group_identifier:                  "group_data"
     stripe_balance_transaction_identifier:    "balance_transaction_data"
     stripe_card_identifier:                   "card_data"
     stripe_charge_identifier:                 "charge_data"
     stripe_customer_identifier:               "customer_data"
+    stripe_discount_identifier:               "discount_data"
     stripe_dispute_identifier:                "dispute_data"
     stripe_fee_identifier:                    "fee_data"
     stripe_invoice_identifier:                "invoice_data"
@@ -28,6 +29,7 @@ vars:
     stripe_payout_identifier:                 "payout_data"
     stripe_plan_identifier:                   "plan_data"
     stripe_price_identifier:                  "price_data"
+    stripe_product_identifier:                "product_data"
     stripe_refund_identifier:                 "refund_data"
     stripe_subscription_history_identifier:   "subscription_history_data"
     stripe_subscription_identifier:           "subscription_data"
@@ -47,6 +49,7 @@ seeds:
       voided_at: timestamp
       canceled_at: timestamp
       invoice_id: "{{ 'varchar(500)' if target.type in ('redshift','postgres') else 'string'}}"
+      end: timestamp
     balance_transaction_data:
       +column_types:
         available_on: timestamp
@@ -61,6 +64,15 @@ seeds:
       +column_types:
         destination_bank_account_id: "{{ 'varchar(500)' if target.type in ('redshift','postgres') else 'string'}}"
         destination_card_id: "{{ 'varchar(500)' if target.type in ('redshift','postgres') else 'string'}}"
+    discount_data:
+      +enabled: "{{ true if target.type not in ('snowflake', 'postgres') else false }}"
+    discount_data_snowflake:
+      +alias: discount_data
+      +enabled: "{{ true if target.type == 'snowflake' else false }}"
+    discount_data_postgres:
+      +alias: discount_data
+      +enabled: "{{ true if target.type == 'postgres' else false }}"
+
 
 dispatch:
   - macro_namespace: dbt_utils

diff --git a/integration_tests/packages.yml b/integration_tests/packages.yml
@@ -1,2 +1,2 @@
 packages:
- - local: ../
+  - local: ../
diff --git a/integration_tests/seeds/discount_data.csv b/integration_tests/seeds/discount_data.csv
@@ -0,0 +1,11 @@
+id,type,type_id,_fivetran_synced,amount,checkout_session_id,checkout_session_line_item_id,coupon_id,credit_note_line_item_id,customer_id,end,invoice_id,invoice_item_id,promotion_code,start,subscription_id
+one1,,cus_twelve,,1,,,2022-09-19 05:50:07.000000,sub_nine,cus_MSXtGSAxxtYWe1,,in_1OvvZQDZlKRVG8GnByl08x0P,,,2022-09-19 05:50:07.000000,sub_1LjcrrDZlKRVG8Gnv1Bjosun
+2wo,,cus_twelve,,2,,,2022-09-19 05:50:07.000000,sub_nine,cus_MSXtGSAxxtYWe1,,in_1OlPk3DZlKRVG8GnkWTFA0go,,,2022-09-19 05:50:07.000000,sub_1LjcrrDZlKRVG8Gnv1Bjosun
+3hree,,cus_twelve,,3,,,2022-09-19 05:50:07.000000,sub_nine,cus_MSXtGSAxxtYWe1,,in_1OaAyXDZlKRVG8GnQo19CbEo,,,2022-09-19 05:50:07.000000,sub_1LjcrrDZlKRVG8Gnv1Bjosun
+4our,,cus_twelve,,4,,,2022-09-19 05:50:07.000000,sub_nine,cus_MSXtGSAxxtYWe1,,,,,2022-09-19 05:50:07.000000,sub_1LjcrrDZlKRVG8Gnv1Bjosun
+5ive,,cus_twelve,,5,,,2022-09-19 05:50:07.000000,sub_nine,cus_MSXtGSAxxtYWe1,,in_1P7ALFDZlKRVG8GnTUCtI40b,,,2022-09-19 05:50:07.000000,sub_1LjcrrDZlKRVG8Gnv1Bjosun
+6ix,,cus_twelve,,6,,,2022-09-19 05:50:07.000000,sub_nine,cus_MSXtGSAxxtYWe1,,in_1OOwCbDZlKRVG8Gn3Tfhz2RW,,,2022-09-19 05:50:07.000000,sub_1LjcrrDZlKRVG8Gnv1Bjosun
+7even,,cus_twelve,,7,,,2022-09-19 05:50:07.000000,sub_nine,cus_MSXtGSAxxtYWe1,,in_1PI2dvDZlKRVG8Gn4I34kEqR,,,2022-09-19 05:50:07.000000,sub_1LjcrrDZlKRVG8Gnv1Bjosun
+8ight,,cus_twelve,,8,,,2022-09-19 05:50:07.000000,sub_nine,cus_MSXtGSAxxtYWe1,,,,,2022-09-19 05:50:07.000000,sub_1LjcrrDZlKRVG8Gnv1Bjosun
+9ine,,cus_thirteen,,9,,,2023-08-07 09:37:39.000000,,cus_OPEJRxhitWa90P,,,,,2023-08-07 09:37:39.000000,
+10n,,cus_OPEKG00sHdsUs8,,10,,,2023-08-07 09:38:05.000000,,cus_OPEKG00sHdsUs8,,,,,2023-08-07 09:38:05.000000,