Add passthrough columns for USER + ORG and add support for updated pa…

…ssthrough format for TICKET (#49)

* add passthrough columns for user + org and revamp ticket passthru var

* add default value

* docs

* doc twweaks

CHANGELOG.md

@@ -1,3 +1,14 @@
+# dbt_zendesk_source v0.11.2
+
+[PR #49](https://github.com/fivetran/dbt_zendesk_source/pull/49) includes the following updates:
+
+## Feature Updates
+- Adds passthrough column support for `USER` and `ORGANIZATION`. 
+  - Using the new `zendesk__user_passthrough_columns` and `zendesk__organization_passthrough_columns` variables, you can include custom columns from these source tables in their respective staging models. See [README](https://github.com/fivetran/dbt_zendesk_source?tab=readme-ov-file#add-passthrough-columns) for more details on how to configure.
+- Also updated the format of the pre-existing `TICKET` passthrough column variable, `zendesk__ticket_passthrough_columns`, to align with the newly added passthrough variables delineated above.
+  - Previously, you could only provide a list of custom fields to be included in `stg_zendesk__ticket`. Now, you have the option to provide an `alias` and `transform_sql` clause to be applied to each field (see [README](https://github.com/fivetran/dbt_zendesk_source?tab=readme-ov-file#add-passthrough-columns) for more details).
+  - **Note**: the package is and will continue to be backwards compatible with the old list-format.
+
 # dbt_zendesk_source v0.11.1
 
 [PR #48](https://github.com/fivetran/dbt_zendesk_source/pull/48) includes the following updates:

README.md

@@ -66,14 +66,33 @@ vars:
 ```
 
 ## (Optional) Step 5: Additional configurations
+<details open><summary>Collapse/Expand configurations</summary>
 
 ### Add passthrough columns
-This package includes all source columns defined in the staging models. However, the `stg_zendesk__ticket` model allows for additional columns to be added using a pass-through column variable. This is extremely useful if you'd like to include custom fields to the package.
+This package includes all source columns defined in the macros folder. You can add more columns from the `TICKET`, `USER`, and `ORGANIZATION` tables using our pass-through column variables, which will persist these custom fields to the `stg_zendesk__ticket`, `stg_zendesk__user`, and `stg_zendesk__organization` models, respectively.
+
+These variables allow for the pass-through fields to be aliased (`alias`) and casted (`transform_sql`) if desired, but not required. Datatype casting is configured via a sql snippet within the `transform_sql` key. You may add the desired sql while omitting the `as field_name` at the end and your custom pass-through fields will be casted accordingly. Use the below format for declaring the respective pass-through variables:
+
 ```yml
 vars:
-  zendesk__ticket_passthrough_columns: [account_custom_field_1, account_custom_field_2]
+  zendesk__ticket_passthrough_columns:
+    - name: "account_custom_field_1" # required
+      alias: "account_1" # optional
+      transform_sql: "cast(account_1 as string)" # optional, must reference the alias if an alias is provided (otherwise the original name)
+    - name: "account_custom_field_2"
+      transform_sql: "cast(account_custom_field_2 as string)"
+    - name: "account_custom_field_3"
+  zendesk__user_passthrough_columns:
+    - name: "internal_app_id_c"
+      alias: "app_id"
+  zendesk__organization_passthrough_columns:
+    - name: "custom_org_field_1"
 ```
 
+> Note: Earlier versions of this package employed a more rudimentary format for passthrough columns, in which the user provided a list of field names to pass in, rather than a mapping. In the above `ticket` example, this would be `[account_custom_field_1, account_custom_field_2, account_custom_field_3]`.
+>
+> This old format will still work, as our passthrough-column macros are all backwards compatible.
+
 ### Mark Former Internal Users as Agents
 If a team member leaves your organization and their internal account is deactivated, their `USER.role` will switch from `agent` or `admin` to `end-user`. This will skew historical ticket SLA metrics, as we calculate reply times and other metrics based on `agent` or `admin` activity only.
 
@@ -115,6 +134,7 @@ vars:
     zendesk_group_identifier: "Group" # as an example, must include the double-quotes and correct case!
 ```  
 
+</details>
 
 ## (Optional) Step 6: Orchestrate your models with Fivetran Transformations for dbt Core™
 <details><summary>Expand to view details</summary>

dbt_project.yml

@@ -1,6 +1,6 @@
 config-version: 2
 name: 'zendesk_source'
-version: '0.11.1'
+version: '0.11.2'
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 models:
   zendesk_source:
@@ -28,3 +28,7 @@ vars:
     user: "{{ source('zendesk', 'user') }}"
     time_zone: "{{ source('zendesk', 'time_zone') }}"
     daylight_time: "{{ source('zendesk', 'daylight_time') }}"
+
+  zendesk__ticket_passthrough_columns: []
+  zendesk__user_passthrough_columns: []
+  zendesk__organization_passthrough_columns: []

integration_tests/dbt_project.yml

@@ -1,7 +1,7 @@
 config-version: 2
 
 name: 'zendesk_source_integration_tests'
-version: '0.11.1'
+version: '0.11.2'
 
 profile: 'integration_tests'
 

macros/get_organization_columns.sql

@@ -15,6 +15,8 @@
     {"name": "url", "datatype": dbt.type_string()}
 ] %}
 
+{{ fivetran_utils.add_pass_through_columns(columns, var('zendesk__organization_passthrough_columns')) }}
+
 {{ return(columns) }}
 
 {% endmacro %}

macros/get_ticket_columns.sql

@@ -46,6 +46,8 @@
     {"name": "via_source_to_name", "datatype": dbt.type_int()}
 ] %}
 
+{{ fivetran_utils.add_pass_through_columns(columns, var('zendesk__ticket_passthrough_columns')) }}
+
 {{ return(columns) }}
 
 {% endmacro %}

macros/get_user_columns.sql

@@ -35,6 +35,8 @@
     {"name": "verified", "datatype": "boolean"}
 ] %}
 
+{{ fivetran_utils.add_pass_through_columns(columns, var('zendesk__user_passthrough_columns')) }}
+
 {{ return(columns) }}
 
 {% endmacro %}

models/stg_zendesk__organization.sql

@@ -35,6 +35,8 @@ final as (
         name,
         external_id
 
+        {{ fivetran_utils.fill_pass_through_columns('zendesk__organization_passthrough_columns') }}
+
     from fields
 )
 

models/stg_zendesk__ticket.sql

@@ -21,13 +21,6 @@ fields as (
                 staging_columns=get_ticket_columns()
             )
         }}
-
-        --The below script allows for pass through columns.
-        {% if var('zendesk__ticket_passthrough_columns',[]) != [] %}
-        ,
-        {{ var('zendesk__ticket_passthrough_columns') | join (", ") }}
-
-        {% endif %}
 
     from base
 ),
@@ -64,12 +57,7 @@ final as (
         via_source_to_address as source_to_address,
         via_source_to_name as source_to_name
 
-        --The below script allows for pass through columns.
-        {% if var('zendesk__ticket_passthrough_columns',[]) != [] %}
-        ,
-        {{ var('zendesk__ticket_passthrough_columns') | join (", ") }}
-
-        {% endif %}
+        {{ fivetran_utils.fill_pass_through_columns('zendesk__ticket_passthrough_columns') }}
 
     from fields
 )

models/stg_zendesk__user.sql

@@ -51,6 +51,9 @@ final as (
         locale,
         active as is_active,
         suspended as is_suspended
+
+        {{ fivetran_utils.fill_pass_through_columns('zendesk__user_passthrough_columns') }}
+
     from fields
 )