BUG - Freshness check fails for organization_tag

[wuservices]

Are you a current Fivetran customer?
Jon Wu, Architect, Paper Culture

Describe the bug

zendesk.organization_tag fails freshness checks. It appears that it shouldn't be checked.

Steps to reproduce

1. Install this package and sync zendesk data
2. Run dbt source snapshot-freshness
3. You'll see an error like ERROR STALE freshness of zendesk.organization_tag

Expected behavior

No freshness error should occur if data sync is working. In my case, it appears that this data just doesn't get updated very often. I noticed that freshness: null is configured for zendesk.organization. Should that just be copied to organization_tag to ignore freshness here too?

Project variables configuration

name: [project-name]
version: 1.0.0
config-version: 2

profile: default
source-paths: [models]
analysis-paths: [analysis]
test-paths: [tests]
data-paths: [data]
macro-paths: [macros]
snapshot-paths: [snapshots]
target-path: target
clean-targets:
  - target
  - dbt_modules

vars:
  zendesk_source:
    zendesk_database: [project]
  # Disable models for Zendesk features we aren't using or syncing
  using_ticket_form_history: False
  using_schedules: False

Package Version

packages:
  - package: fishtown-analytics/dbt_utils
    version: 0.6.6
  - package: fivetran/ad_reporting
    version: 0.2.2
  - package: fivetran/facebook_ads
    version: 0.2.1
  - package: fivetran/google_ads
    version: 0.2.1
  - package: fivetran/marketo
    version: 0.4.0
  - package: fivetran/microsoft_ads
    version: 0.2.1
  - package: fivetran/pinterest
    version: 0.3.2
  - package: fivetran/zendesk
    version: 0.5.1

Warehouse

• BigQuery
• Redshift
• Snowflake
• Postgres
• Databricks
• Other (provide details below)

Additional context
Freshness is ignored for organization: https://github.com/fivetran/dbt_zendesk_source/blob/master/models/src_zendesk.yml#L130

Screenshots

Please indicate the level of urgency

Testing for the first time, but hopefully this is a simple fix.

Are you interested in contributing to this package?

• Yes, I can do this and open a PR for your review.
• Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this fixed.
• No, I'd prefer if someone else fixed this. I don't have the time and/or don't know what the root cause of the problem is.

[fivetran-joemarkiewicz]

Hi @wuservices thanks for opening this Issue as well as Issue #18!

This is actually a request we have seen come through on our dbt_hubspot package and have been able to institute a relatively low impact fix for these disabled tables. You can see the fix we did here. I will be able to easily add this feature to the package. However, I will focus first on upgrading our dbt_zendesk package for dbt 20 compatibility first so there are no compatibility issues between this source package and the transform package.

[fivetran-jamie]

hey @wuservices our own data is in line with the pre-existing freshness tests, so i am thinking that the ideal solution here may be for you to override the zendesk source and set the organization_tag freshness to null

however, worth noting that dbt 0.21.0 will introduce --exclude for freshness tests 👀 (see PR and pre-release notes)

[wuservices]

hi @fivetran-jamie thanks for the tips! @fivetran-joemarkiewicz pointed out the same freshness override in office hours, and that was a huge help! We were able to get our freshness tuned up nicely with that, so we're OK.

However, we did also discuss that good defaults are always nice when possible. For the benefit of others, I'd be curious why you'd have freshness by default on organization_tag, but no freshness for organization? For many companies using Zendesk, I could imagine that the clients they work with organization may not change often (or never at all for B2C). This seems to be reflected in your default disabled freshness for organization. However, if organization_tag has freshness, does that mean you'd assume that people would be changing tags on a regular basis? Although we don't use those, I'd imagine that most people would have stable tags.

[fivetran-jamie]

that's a good point that joe and i were honestly unsure about. in our own zendesk data, organization doesn't change too often, but it does appear that we have fresh tags on them. i'll look into this more and talk to our internal customer success folks who are using zendesk (and maybe do a quick poll in dbt slack or something) to see if this manual tagging is actually happening often.

definitely aiming for the nicest defaults possible and really appreciate your input! i think it'd be good to keep this issue open and mark it as won't fix for now in case other customers are coming across the same staleness issues with this table. glad to hear the override works for you for the time being 🙂

[fivetran-joemarkiewicz]

Closing this issue as it is a wontfix.