Release/v0.17.0 (#169)

* initial * feature/unstructured-data * add coalesce_cast * update filters * update and consolidate models * model revisions * restructure * documentation * remove extra comma * regen docs * formatting * update max token docs * Update CHANGELOG.md * bug/missing-sla-policies * update changelog and add integrity test * update test * update changelog, readme and tests * update test * bug/intercepted-period-joins * adjustmnt * update weeks * update weeks * add integrity test * update weeks * update changelog * bugfix/too-many-partitions (#165) * bugfix/too-many-partitions * docs regen * Update CHANGELOG.md Co-authored-by: fivetran-catfritz <111930712+fivetran-catfritz@users.noreply.github.com> --------- Co-authored-by: fivetran-catfritz <111930712+fivetran-catfritz@users.noreply.github.com> * update changelog * revert docs to main * Documentation Standard Updates (#166) * MagicBot/documentation-updates * Apply suggestions from code review * Update README.md Co-authored-by: fivetran-catfritz <111930712+fivetran-catfritz@users.noreply.github.com> --------- Co-authored-by: fivetran-catfritz <111930712+fivetran-catfritz@users.noreply.github.com> * update default max_tokens * update changelog * Apply suggestions from code review Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com> * update readme * regen docs * update yml * Apply suggestions from code review Co-authored-by: Renee Li <91097070+fivetran-reneeli@users.noreply.github.com> * add comments and update changelog * update changelog * Update packages.yml --------- Co-authored-by: Renee Li <renee.li@fivetran.com> Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com> Co-authored-by: Renee Li <91097070+fivetran-reneeli@users.noreply.github.com>
fivetran · Sep 4, 2024 · 0b73c19 · 0b73c19
1 parent 66d7b7c
commit 0b73c19
Show file tree

Hide file tree

Showing 40 changed files with 540 additions and 157 deletions.
diff --git a/.buildkite/pipeline.yml b/.buildkite/pipeline.yml
@@ -58,7 +58,7 @@ steps:
     commands: |
       bash .buildkite/scripts/run_models.sh redshift
 
-  - label: ":bricks: Run Tests - Databricks"
+  - label: ":databricks: Run Tests - Databricks"
     key: "run_dbt_databricks"
     plugins:
       - docker#v3.13.0:

diff --git a/.buildkite/scripts/run_models.sh b/.buildkite/scripts/run_models.sh
@@ -19,7 +19,7 @@ dbt deps
 dbt seed --target "$db" --full-refresh
 dbt run --target "$db" --full-refresh
 dbt test --target "$db"
-dbt run --vars '{using_schedules: false, using_domain_names: false, using_user_tags: false, using_ticket_form_history: false, using_organization_tags: false}' --target "$db" --full-refresh
+dbt run --vars '{zendesk__unstructured_enabled: true, using_schedules: false, using_domain_names: false, using_user_tags: false, using_ticket_form_history: false, using_organization_tags: false}' --target "$db" --full-refresh
 dbt test --target "$db"
 
 dbt run-operation fivetran_utils.drop_schemas_automation --target "$db"
diff --git a/.github/ISSUE_TEMPLATE/bug-report.yml b/.github/ISSUE_TEMPLATE/bug-report.yml
@@ -1,7 +1,7 @@
 name: 🐞 Bug
 description: Report a bug or an issue you've found within the dbt package
 title: "[Bug] <title>"
-labels: ["bug", "triage"]
+labels: ["type:bug"]
 body:
   - type: markdown
     attributes:
@@ -35,6 +35,12 @@ body:
       description: A concise description of what you expected to happen.
     validations:
       required: true
+  - type: textarea
+    attributes:
+      label: Possible solution
+      description: Were you able to investigate and/or discover a potential fix to this bug in your investigation? If so, it would be much appreciated if you could submit code samples to show us how your fix resolved this issue. 
+    validations:
+      required: false
   - type: textarea
     attributes:
       label: dbt Project configurations
@@ -61,6 +67,19 @@ body:
         - other (mention it in "Additional Context")
     validations:
       required: true
+  - type: dropdown
+    id: orchestration_type
+    attributes:
+      label: How are you running this dbt package?
+      multiple: true
+      options:
+        - Fivetran Quickstart Data Model
+        - Fivetran Transformations
+        - dbt Core™
+        - dbt Cloud™
+        - other (mention it in "Additional Context")
+    validations:
+      required: true
   - type: textarea
     attributes:
       label: dbt Version
@@ -83,6 +102,6 @@ body:
       description: Our team will assess this issue and let you know if we will add it to a future sprint. However, if you would like to expedite the solution, we encourage you to contribute to the package via a PR. Our team will then work with you to approve and merge your contributions as soon as possible.
       options:
         - label: Yes.
-        - label: Yes, but I will need assistance and will schedule time during our [office hours](https://calendly.com/fivetran-solutions-team/fivetran-solutions-team-office-hours) for guidance
+        - label: Yes, but I will need assistance.
         - label: No.
           required: false
diff --git a/.github/ISSUE_TEMPLATE/feature-request.yml b/.github/ISSUE_TEMPLATE/feature-request.yml
@@ -1,7 +1,7 @@
 name: 🎉 Feature
 description: Suggest a new feature for the Fivetran dbt package
 title: "[Feature] <title>"
-labels: ["enhancement"]
+labels: ["type:enhancement"]
 body:
   - type: markdown
     attributes:
@@ -20,6 +20,13 @@ body:
       description: A clear and concise description of what you want to happen and why you want the new feature.
     validations:
       required: true
+  - type: textarea
+    attributes:
+      label: How would you implement this feature?
+      description: |
+        How would you build out this feature with your existing data? Any code examples you can provide to help accelerate development on this issue?
+    validations:
+      required: true
   - type: textarea
     attributes:
       label: Describe alternatives you've considered
@@ -34,7 +41,7 @@ body:
       description: Our team will assess this feature and let you know if we will add it to a future sprint. However, if you would like to expedite the feature, we encourage you to contribute to the package via a PR. Our team will then work with you to approve and merge your contributions as soon as possible.
       options:
         - label: Yes.
-        - label: Yes, but I will need assistance and will schedule time during your [office hours](https://calendly.com/fivetran-solutions-team/fivetran-solutions-team-office-hours) for guidance.
+        - label: Yes, but I will need assistance.
         - label: No.
           required: false
   - type: textarea

diff --git a/.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md b/.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md
@@ -16,7 +16,6 @@ Please acknowledge that you have successfully performed the following commands l
 Before marking this PR as "ready for review" the following have been applied:
 - [ ] The appropriate issue has been linked, tagged, and properly assigned
 - [ ] All necessary documentation and version upgrades have been applied
-    <!--- Be sure to update the package version in the dbt_project.yml, integration_tests/dbt_project.yml, and README if necessary. -->
 - [ ] docs were regenerated (unless this PR does not include any code or yml updates)
 - [ ] BuildKite integration tests are passing
 - [ ] Detailed validation steps have been provided below

diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
@@ -19,7 +19,13 @@
 <!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
 - [ ] Yes
 
-**Provide an emoji that best describes your current mood**
+**Typically there are additional maintenance changes required before this will be ready for an upcoming release. Are you comfortable with the Fivetran team making a few commits directly to your branch?**
+<!--- If you select Yes this will help expedite your PR in case there are small changes required before approval. We encourage you not to use this branch in a production environment as we may make additional updates.  -->
+<!--- If you select No, we will not make any changes directly to your branch and will either communicate any planned changes via the PR thread or will merge your PR into a separate branch so we may make changes without modifying your branch.. -->
+- [ ] Yes
+- [ ] No
+
+**If you had to summarize this PR in an emoji, which would it be?**
 <!--- For a complete list of markdown compatible emojis check our this git repo (https://gist.github.com/rxaviers/7360908)  --> 
 :dancer:
 
@@ -30,4 +36,4 @@ We are so excited you decided to contribute to the Fivetran community dbt packag
 **PR Template** 
 - [Community Pull Request Template](?expand=1&template=pull_request_template.md) (default)
 
-- [Maintainer Pull Request Template](?expand=1&template=maintainer_pull_request_template.md) (to be used by maintainers)
+- [Maintainer Pull Request Template](?expand=1&template=maintainer_pull_request_template.md) (to be used by maintainers)
diff --git a/.quickstart/quickstart.yml b/.quickstart/quickstart.yml
@@ -23,3 +23,12 @@ destination_configurations:
     dispatch:
       - macro_namespace: dbt_utils
         search_order: [ 'spark_utils', 'dbt_utils' ]
+
+public_models: [
+  "zendesk__ticket_enriched",
+  "zendesk__ticket_metrics",
+  "zendesk__ticket_summary",
+  "zendesk__ticket_field_history",
+  "zendesk__sla_policies",
+  "zendesk__ticket_backlog"
+]
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,42 @@
+# dbt_zendesk v0.17.0
+
+## New model ([#161](https://github.com/fivetran/dbt_zendesk/pull/161))
+- Addition of the `zendesk__document` model, designed to structure Zendesk textual data for vectorization and integration into NLP workflows. The model outputs a table with:
+  - `document_id`: Corresponding to the `ticket_id`
+  - `chunk_index`: For text segmentation
+  - `chunk`: The text chunk itself
+  - `chunk_tokens_approximate`: Approximate token count for each segment
+- This model is currently disabled by default. You may enable it by setting the `zendesk__unstructured_enabled` variable as `true` in your `dbt_project.yml`.
+  - This model was developed with the limit of chunk sizes to approximately 5000 tokens for use with OpenAI, however you can change this limit by setting the variable `zendesk_max_tokens` in your `dbt_project.yml`.
+  - See the README section [Enabling the unstructured document model for NLP](https://github.com/fivetran/dbt_zendesk/blob/main/README.md#enabling-the-unstructured-document-model-for-nlp) for more information.
+
+## Breaking Changes (Full refresh required after upgrading)
+- Incremental models running on BigQuery have had the `partition_by` logic adjusted to include a granularity of a month. This change only impacts BigQuery warehouses and was applied to avoid the common `too many partitions` error some users have experienced when partitioning by day. Therefore, adjusting the partition to a month granularity will decrease the number of partitions created and allow for more performant querying and incremental loads. This change was applied to the following models ([#165](https://github.com/fivetran/dbt_zendesk/pull/165)):
+  - `int_zendesk__field_calendar_spine`
+  - `int_zendesk__field_history_pivot`
+  - `zendesk__ticket_field_history`
+
+- In the [dbt_zendesk_source v0.12.0 release](https://github.com/fivetran/dbt_zendesk_source/releases/tag/v0.12.0), the field `_fivetran_deleted` was added to the following models for use in `zendesk__document` model ([#161](https://github.com/fivetran/dbt_zendesk/pull/161)):
+  - `stg_zendesk__ticket`
+  - `stg_zendesk__ticket_comment`
+  - `stg_zendesk__user`
+  - If you have already added `_fivetran_deleted` as a passthrough column via the `zendesk__ticket_passthrough_columns` or `zendesk__user_passthrough_columns` variable, you will need to remove or alias this field from the variable to avoid duplicate column errors.
+
+## Bug Fixes
+- Fixed an issue in the `zendesk__sla_policies` model where tickets that were opened and solved outside of scheduled hours were not being reported, specifically for the metrics `requester_wait_time` and `agent_work_time`. 
+  - Resolved by adjusting the join logic in models `int_zendesk__agent_work_time_business_hours` and `int_zendesk__requester_wait_time_business_hours`. ([#164](https://github.com/fivetran/dbt_zendesk/pull/164), [#156](https://github.com/fivetran/dbt_zendesk/pull/156))
+- Fixed an issue in the `zendesk__ticket_metrics` model where certain tickets had miscalculated metrics. 
+  - Resolved by adjusting the join logic in models `int_zendesk__ticket_work_time_business`, `int_zendesk__ticket_first_resolution_time_business`, and `int_zendesk__ticket_full_resolution_time_business`. ([#167](https://github.com/fivetran/dbt_zendesk/pull/167))
+
+## Under the hood
+- Added integrity validations:
+  - Test to ensure `zendesk__sla_policies` and `zendesk__ticket_metrics` models produce consistent time results. ([#164](https://github.com/fivetran/dbt_zendesk/pull/164))
+  - Test to ensure `zendesk__ticket_metrics` contains all the tickets found in `stg_zendesk__ticket`. ([#167](https://github.com/fivetran/dbt_zendesk/pull/167))
+- Modified the `consistency_sla_policy_count` validation test to group by `ticket_id` for more accurate testing. ([#165](https://github.com/fivetran/dbt_zendesk/pull/165))
+- Updated casting in joins from timestamps to dates so that the whole day is considered. This produces more accurate results. ([#164](https://github.com/fivetran/dbt_zendesk/pull/164), [#156](https://github.com/fivetran/dbt_zendesk/pull/156), [#167](https://github.com/fivetran/dbt_zendesk/pull/167))
+- Reduced the weeks looking ahead from 208 to 52 to improve performance, as tracking ticket SLAs beyond one year was unnecessary. ([#156](https://github.com/fivetran/dbt_zendesk/pull/156), [#167](https://github.com/fivetran/dbt_zendesk/pull/167))
+- Updated seed files to reflect a real world ticket field history update scenario. ([#165](https://github.com/fivetran/dbt_zendesk/pull/165))
+
 # dbt_zendesk v0.16.0
 ## 🚨 Minor Upgrade 🚨
 Although this update is not a breaking change, it will likely impact the output of the `zendesk__sla_policies` and `zendesk__sla_metrics` models. [PR #154](https://github.com/fivetran/dbt_zendesk/pull/154) includes the following changes: