Merge pull request #21 from fivetran/feature/disable-vc-abs

disable vendor_categories and multibook accounting tables
fivetran · Sep 21, 2022 · 799a3d9 · 799a3d9
2 parents abb33e1 + 9f76e9d
commit 799a3d9
Show file tree

Hide file tree

Showing 12 changed files with 47 additions and 11 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -30,6 +30,7 @@ jobs:
             cd integration_tests
             dbt deps
             dbt seed --target postgres --full-refresh
+            dbt run --target postgres --full-refresh --vars '{netsuite2__multibook_accounting_enabled: false, netsuite2__using_vendor_categories: false}'
             dbt run --target postgres --full-refresh
             dbt test --target postgres
       - run:
@@ -40,6 +41,7 @@ jobs:
             cd integration_tests
             dbt deps
             dbt seed --target redshift --full-refresh
+            dbt run --target redshift --full-refresh --vars '{netsuite2__multibook_accounting_enabled: false, netsuite2__using_vendor_categories: false}'
             dbt run --target redshift --full-refresh
             dbt test --target redshift
       - run:
@@ -50,6 +52,7 @@ jobs:
             cd integration_tests
             dbt deps
             dbt seed --target snowflake --full-refresh
+            dbt run --target snowflake --full-refresh --vars '{netsuite2__multibook_accounting_enabled: false, netsuite2__using_vendor_categories: false}'
             dbt run --target snowflake --full-refresh
             dbt test --target snowflake
       - run:
@@ -63,6 +66,7 @@ jobs:
             cd integration_tests
             dbt deps
             dbt seed --target bigquery --full-refresh
+            dbt run --target bigquery --full-refresh --vars '{netsuite2__multibook_accounting_enabled: false, netsuite2__using_vendor_categories: false}'
             dbt run --target bigquery --full-refresh
             dbt test --target bigquery
       - save_cache:

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,10 @@
+# dbt_netsuite_source v0.5.1
+
+## Features 🎉 (affects Netsuite2 users only)
+- Introduces the `netsuite2__multibook_accounting_enabled` and `netsuite2__using_vendor_categories` variables to disable their related source tables and downstream models ([#21](https://github.com/fivetran/dbt_netsuite_source/pull/21)).
+  - `netsuite2__multibook_accounting_enabled` is `True` by default. Set it to `False` if you do not use the [Multi-Book Accounting](https://docs.oracle.com/en/cloud/saas/netsuite/ns-online-help/book_3831565332.html) feature in Netsuite and/or do not have the `accountingbook` and `accountingbooksubsidiaries` source tables.
+  - `netsuite2__using_vendor_categories` is `True` by default. Set it to `False` if you do not categorize vendors in Netsuite and/or do not have the `vendorcategory` source table.
+
 # dbt_netsuite_source v0.5.0
 🎉 [Netsuite2](https://docs.oracle.com/en/cloud/saas/netsuite/ns-online-help/article_163465025391.html) Compatibility 🎉
 PRs [#15](https://github.com/fivetran/dbt_netsuite_source/pull/15) and [#17](https://github.com/fivetran/dbt_netsuite_source/pull/17) include the following update to the dbt_netsuite_source package:

diff --git a/README.md b/README.md
@@ -94,7 +94,20 @@ vars:
     netsuite_schema: your_schema_name 
 ```
 
-## (Optional) Step 5: Additional configurations
+## Step 5: Disable models for non-existent sources (Netsuite2 only)
+It's possible that your Netsuite connector does not sync every table that this package expects. If your syncs exclude certain tables, it is because you either don't use that feature in Netsuite or actively excluded some tables from your syncs. To disable the corresponding functionality in the package, you must add the relevant variables. By default, all variables are assumed to be true. Add variables for only the tables you would like to disable:
+```yml
+vars:
+    netsuite2__multibook_accounting_enabled: false # True by default. Disable `accountingbooksubsidiary` and `accountingbook` if you are not using the Multi-Book Accounting feature
+    netsuite2__using_vendor_categories: false # True by default. Disable `vendorcategory` if you don't categorize your vendors
+```
+> **Note**: The Netsuite dbt package currently only supports disabling transforms of [Multi-Book Accounting](https://docs.oracle.com/en/cloud/saas/netsuite/ns-online-help/book_3831565332.html) related tables (`accountingbooksubsidiary` and `accountingbook`) and the `vendorcategory` source table. Please create an issue to request additional tables and/or [features](https://docs.oracle.com/en/cloud/saas/netsuite/ns-online-help/bridgehead_N233872.html) to exclude. 
+> 
+> To determine if a table or field is activated by a feature, access the [Records Catalog](https://docs.oracle.com/en/cloud/saas/netsuite/ns-online-help/article_159367781370.html).
+
+## (Optional) Step 6: Additional configurations
+<details><summary>Expand for configurations</summary>
+
 ### Passing Through Additional Fields
 This package includes all source columns defined in the macros folder. You can add more columns using our pass-through column variables. These variables allow for the pass-through fields to be aliased (`alias`) and casted (`transform_sql`) if desired, but not required. Datatype casting is configured via a sql snippet within the `transform_sql` key. You may add the desired sql while omitting the `as field_name` at the end and your custom pass-though fields will be casted accordingly. Use the below format for declaring the respective pass-through variables:
 
@@ -150,10 +163,16 @@ vars:
 ### Override the data models variable
 This package is designed to run **either** the Netsuite.com or Netsuite2 data models. However, for documentation purposes, an additional variable `netsuite_data_model_override` was created to allow for both data model types to be run at the same time by setting the variable value to `netsuite`. This is only to ensure the [dbt docs](https://fivetran.github.io/dbt_netsuite_source/) (which is hosted on this repository) is generated for both model types. While this variable is provided, we recommend you do not adjust the variable and instead change the `netsuite_data_model` variable to fit your configuration needs.
 
-## (Optional) Step 6: Orchestrate your models with Fivetran Transformations for dbt Core™
+</details>
+
+## (Optional) Step 7: Orchestrate your models with Fivetran Transformations for dbt Core™
+<details><summary>Expand for details</summary>
+<br>
 
 Fivetran offers the ability for you to orchestrate your dbt project through [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt). Learn how to set up your project for orchestration through Fivetran in our [Transformations for dbt Core™ setup guides](https://fivetran.com/docs/transformations/dbt#setupguide).
-
+
+</details>
+
 # 🔍 Does this package have dependencies?
 This dbt package is dependent on the following dbt packages. Please be aware that these dependencies are installed by default within this package. For more information on the following packages, refer to the [dbt hub](https://hub.getdbt.com/) site.
 > IMPORTANT: If you have any of these dependent packages in your own `packages.yml` file, we highly recommend that you remove them from your root `packages.yml` to avoid package version conflicts.

diff --git a/dbt_project.yml b/dbt_project.yml
@@ -2,7 +2,7 @@ config-version: 2
 require-dbt-version: [">=1.0.0", "<2.0.0"]
 
 name: 'netsuite_source'
-version: '0.5.0'
+version: '0.5.1'
 
 models:
   netsuite_source:

diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml
@@ -1,5 +1,5 @@
 name: 'netsuite_source_integration_tests'
-version: '0.5.0'
+version: '0.5.1'
 profile: 'integration_tests'
 config-version: 2
 

diff --git a/models/netsuite2/src_netsuite2.yml b/models/netsuite2/src_netsuite2.yml
@@ -33,6 +33,8 @@ sources:
       - name: accounting_book_subsidiaries
         identifier: "{{ var('netsuite2_accounting_book_subsidiaries_identifier', 'accountingbooksubsidiaries') }}"
         description: "{{ doc('accounting_book_subsidiaries_table') }}"
+        config:
+          enabled: "{{ var('netsuite2__multibook_accounting_enabled', true) }}"
         columns:
           - name: _fivetran_id
             description: "{{ doc('_fivetran_id') }}"
@@ -48,6 +50,8 @@ sources:
       - name: accounting_book
         identifier: "{{ var('netsuite2_accounting_book_identifier', 'accountingbook') }}"
         description: "{{ doc('accounting_books_table') }}"
+        config:
+          enabled: "{{ var('netsuite2__multibook_accounting_enabled', true) }}"
         columns:
           - name: _fivetran_synced
             description: "{{ doc('_fivetran_synced') }}"
@@ -567,6 +571,8 @@ sources:
       - name: vendor_category
         identifier: "{{ var('netsuite2_vendor_category_identifier', 'vendorcategory') }}"
         description: "{{ doc('vendor_categories_table') }}"
+        config:
+          enabled: "{{ var('netsuite2__using_vendor_categories', true) }}"
         columns:
           - name: id
             description: "{{ doc('vendor_category_id') }}"

diff --git a/models/netsuite2/stg_netsuite2__accounting_book_subsidiaries.sql b/models/netsuite2/stg_netsuite2__accounting_book_subsidiaries.sql
@@ -1,4 +1,4 @@
-{{ config(enabled=var('netsuite_data_model', 'netsuite') == var('netsuite_data_model_override','netsuite2')) }}
+{{ config(enabled=(var('netsuite_data_model', 'netsuite') == var('netsuite_data_model_override','netsuite2') and var('netsuite2__multibook_accounting_enabled', true))) }}
 
 with base as (
 

diff --git a/models/netsuite2/stg_netsuite2__accounting_books.sql b/models/netsuite2/stg_netsuite2__accounting_books.sql
@@ -1,4 +1,4 @@
-{{ config(enabled=var('netsuite_data_model', 'netsuite') == var('netsuite_data_model_override','netsuite2')) }}
+{{ config(enabled=(var('netsuite_data_model', 'netsuite') == var('netsuite_data_model_override','netsuite2') and var('netsuite2__multibook_accounting_enabled', true))) }}
 
 with base as (
 

diff --git a/models/netsuite2/stg_netsuite2__vendor_categories.sql b/models/netsuite2/stg_netsuite2__vendor_categories.sql
@@ -1,4 +1,4 @@
-{{ config(enabled=var('netsuite_data_model', 'netsuite') == var('netsuite_data_model_override','netsuite2')) }}
+{{ config(enabled=(var('netsuite_data_model', 'netsuite') == var('netsuite_data_model_override','netsuite2') and var('netsuite2__using_vendor_categories', true))) }}
 
 with base as (
 

diff --git a/models/netsuite2/tmp/stg_netsuite2__accounting_book_subsidiaries_tmp.sql b/models/netsuite2/tmp/stg_netsuite2__accounting_book_subsidiaries_tmp.sql
@@ -1,4 +1,4 @@
-{{ config(enabled=var('netsuite_data_model', 'netsuite') == var('netsuite_data_model_override','netsuite2')) }}
+{{ config(enabled=(var('netsuite_data_model', 'netsuite') == var('netsuite_data_model_override','netsuite2') and var('netsuite2__multibook_accounting_enabled', true))) }}
 
 select * 
 from {{ var('netsuite2_accounting_book_subsidiaries') }}
diff --git a/models/netsuite2/tmp/stg_netsuite2__accounting_books_tmp.sql b/models/netsuite2/tmp/stg_netsuite2__accounting_books_tmp.sql
@@ -1,4 +1,4 @@
-{{ config(enabled=var('netsuite_data_model', 'netsuite') == var('netsuite_data_model_override','netsuite2')) }}
+{{ config(enabled=(var('netsuite_data_model', 'netsuite') == var('netsuite_data_model_override','netsuite2') and var('netsuite2__multibook_accounting_enabled', true))) }}
 
 select * 
 from {{ var('netsuite2_accounting_books') }}
diff --git a/models/netsuite2/tmp/stg_netsuite2__vendor_categories_tmp.sql b/models/netsuite2/tmp/stg_netsuite2__vendor_categories_tmp.sql
@@ -1,4 +1,4 @@
-{{ config(enabled=var('netsuite_data_model', 'netsuite') == var('netsuite_data_model_override','netsuite2')) }}
+{{ config(enabled=(var('netsuite_data_model', 'netsuite') == var('netsuite_data_model_override','netsuite2') and var('netsuite2__using_vendor_categories', true))) }}
 
 select * 
 from {{ var('netsuite2_vendor_categories') }}