updates and rm run results in docs

CHANGELOG.md

@@ -1,3 +1,16 @@
+# dbt_iterable_source v0.10.0
+[PR #39](https://github.com/fivetran/dbt_iterable_source/pull/39) contains the following updates:
+
+## Breaking Change (`--full-refresh` required after upgrading)  
+- Updates the materialization in `stg_iterable__user_history` from table to view in order to improve performance. A `--full-refresh` is required after upgrading to ensure there are no issues.
+
+## Bug Fixes 
+- Removes the source freshness tests in the `src_iterable.yml`. This was originally causing run errors for users where tables weren't present in their schema but listed in the `src_iterable.yml`.
+- Removes `created_at` from the uniqueness test for `stg_iterable__event` in `stg_iterable.yml`. Uniqueness is now tested solely on `unique_event_id`, a surrogate key made up of `event_id` (`_fivetran_id` in the raw table, which is a Fivetran-created unique identifier derived from hashing `campaign_id`, `created_at`, and `event_name`) and `_fivetran_user_id` (a Fivetran-created column derived from a hash of `user_id` and/or `email`).
+
+## Documentation Update
+- Updates the descriptions of timestamp-based fields. Previously they were described as milliseconds since epoch time, but they should be standard timestamps.
+
 # dbt_iterable_source v0.9.0
 [PR #31](https://github.com/fivetran/dbt_iterable_source/pull/31) contains the following updates:
 

README.md

@@ -20,6 +20,8 @@
 - Generates a comprehensive data dictionary of your source and modeled Iterable data through the [dbt docs site](https://fivetran.github.io/dbt_iterable_source/).
 - These tables are designed to work simultaneously with our [Iterable transformation package](https://github.com/fivetran/dbt_iterable/).
 
+> This package does not apply freshness tests.
+
 ## How do I use the dbt package?
 
 ### Step 1: Prerequisites
@@ -42,7 +44,7 @@ Include the following Iterable package version in your `packages.yml` file.
 ```yaml
 packages:
   - package: fivetran/iterable_source
-    version: [">=0.9.0", "<0.10.0"]
+    version: [">=0.10.0", "<0.11.0"]
 ```
 
 ### Step 3: Define database and schema variables
@@ -71,8 +73,6 @@ vars:
 
 
 ### (Optional) Step 5: Additional configurations
-<details><summary>Expand for details</summary>
-<br>
 
 #### Passing Through Additional Fields
 
@@ -124,8 +124,6 @@ vars:
     iterable_campaign_suppression_list_history_identifier: "campaign_supression_list_history"
 ```
 
-</details>
-
 ### (Optional) Step 6: Orchestrate your models with Fivetran Transformations for dbt Core™
 <details><summary>Expand for details</summary>
 <br>

dbt_project.yml

@@ -1,7 +1,7 @@
 config-version: 2
 name: 'iterable_source'
 
-version: '0.9.0'
+version: '0.10.0'
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 
 models:

integration_tests/dbt_project.yml

@@ -1,7 +1,7 @@
 config-version: 2
 name: 'iterable_source_integration_tests'
 
-version: '0.9.0'
+version: '0.10.0'
 profile: 'integration_tests'
 
 vars:

integration_tests/seeds/event_data.csv

@@ -1,3 +1,7 @@
-_fivetran_id,email,campaign_id,message_id,content_id,created_at,event_name,ip,message_bus_id,message_type_id,recipient_state,status,unsub_source,user_agent,user_agent_device,transactional_data,additional_properties,_fivetran_synced, _fivetran_user_id
+_fivetran_id,email,campaign_id,message_id,content_id,created_at,event_name,ip,message_bus_id,message_type_id,recipient_state,status,unsub_source,user_agent,user_agent_device,transactional_data,additional_properties,_fivetran_synced,_fivetran_user_id
 HFNB4mJWJeXKk7AwJAEPCCmvpkM=,nobody@gmail.co.uk,,,,2020-06-12 14:42:41.000,emailSubscribe,,,,,,,,,null,"{""channelIds"": ""[]"",""emailListIds"": ""[575561]"",""hasThisThing"": ""Yes"",""hasOtherThing"": ""Yes"",""hasOverdueThing"": ""Yes"",""didAThing"": ""Yes"",""isAFriend"": ""Yes"",""isCool"": ""Yes"",""isMean"": ""No"",""profileUpdatedAt"": ""2020-06-12 14:39:11 +00:00"",""signupSource"": ""Import""}",2020-08-04 12:04:05.997,987654
-8nxrQcLoFB96e+oOBqzBDsXnMBU=,myson@gmail.com,,,,2020-03-26 12:30:43.000,emailSubscribe,,,,,,,,,null,"{""channelIds"": ""[]"",""emailListIds"": ""[501079]"",""profileUpdatedAt"": ""2020-03-26 12:30:42 +00:00"",""signupSource"": ""Import""}",2020-08-04 14:10:39.711,12345
+6dfiQcLoFB96e+oOBqzBDsXnMXU=,myson@gmail.com,,,,2020-03-26 12:30:43.000,emailSubscribe,,,,,,,,,null,"{""channelIds"": ""[]"",""emailListIds"": ""[501079]"",""profileUpdatedAt"": ""2020-03-26 12:30:42 +00:00"",""signupSource"": ""Import""}",2020-08-04 14:10:39.711,12345
+7sdfQcLoFB96e+oOBqzBDsXnNYU=,myson@gmail.com,,,,2020-06-12 18:30:43.000,newEvent,,,,,,,,,null,"{""channelIds"": ""[]"",""emailListIds"": ""[501079,101]"",""profileUpdatedAt"": ""2020-04-21 12:30:42 +00:00"",""signupSource"": ""Import""}",2020-08-07 02:10:19.243,12345
+USA101,nobody@gmail.co.uk,,,,2020-06-12 18:30:43.000,newEvent,,,,,,,,,null,,2020-08-07 02:10:19.243,987654
+8nxrQcLoFB96e+oOBqzBDsXnMBU=,myson@gmail.com,,,,2020-07-12 18:30:43.000,newEvent,,,,,,,,,null,"{""channelIds"": ""[]"",""emailListIds"": ""[501079,101]"",""profileUpdatedAt"": ""2020-04-21 12:30:42 +00:00"",""signupSource"": ""Import""}",2020-08-07 02:10:19.243,12345
+djjfhdjkdagS3r5fd,abc@gmail.com,,,,2020-07-12 22:30:43.000,newEvent,,,,,,,,,null,"{""channelIds"": ""[]"",""emailListIds"": ""[501079,101]"",""profileUpdatedAt"": ""2020-04-21 12:30:42 +00:00"",""signupSource"": ""Import""}",2020-08-07 02:10:19.243,5678

models/src_iterable.yml

@@ -8,11 +8,6 @@ sources:
     loader: fivetran
     loaded_at_field: _fivetran_synced
 
-    freshness:
-      warn_after: {count: 72, period: hour}
-      error_after: {count: 168, period: hour}
-
-
     tables:
       - name: campaign_history
         identifier: "{{ var('iterable_campaign_history_identifier', 'campaign_history')}}"
@@ -21,7 +16,7 @@ sources:
           - name: id
             description: Unique identifier of the campaign record
           - name: updated_at
-            description: Last update's timestamp as epoch time in milliseconds
+            description: Last update's timestamp
           - name: name
             description: User defined name of the campaign.
           - name: campaign_state
@@ -31,11 +26,11 @@ sources:
           - name: send_size
             description: Size of the campaign. Number of individuals included in the campaign
           - name: start_at
-            description: Start timestamp as epoch time in milliseconds
+            description: Start timestamp
           - name: ended_at
-            description: Ended timestamp as epoch time in milliseconds
+            description: Ended timestamp
           - name: created_at
-            description: Creation timestamp as epoch time in milliseconds
+            description: Creation timestamp
           - name: recurring_campaign_id
             description: Reference to the recurring campaign, if applicable
           - name: created_by_user_id
@@ -58,7 +53,7 @@ sources:
           - name: label
             description: Name of the label
           - name: updated_at
-            description: Last update timestamp as epoch time in milliseconds
+            description: Last update timestamp
 
       - name: campaign_list_history
         identifier: "{{ var('iterable_campaign_list_history_identifier', 'campaign_list_history')}}"
@@ -69,7 +64,7 @@ sources:
           - name: list_id
             description: Reference to the list used
           - name: updated_at
-            description: Last update timestamp as epoch time in milliseconds
+            description: Last update timestamp
 
       - name: campaign_suppression_list_history  # this is the correct spelling
         identifier: "{{ var('iterable_campaign_suppression_list_history_identifier', 'campaign_suppression_list_history') }}"
@@ -81,7 +76,7 @@ sources:
           - name: suppressed_list_id
             description: Reference to the list used
           - name: updated_at
-            description: Last update timestamp as epoch time in milliseconds
+            description: Last update timestamp
 
       - name: channel
         identifier: "{{ var('iterable_channel_identifier', 'channel')}}"
@@ -107,7 +102,7 @@ sources:
           - name: content_id
             description: Reference to the content the event is associated with.
           - name: created_at
-            description: Creation timestamp as epoch time in milliseconds
+            description: Creation timestamp
           - name: email
             description: The user's email. User identifier, for email-based projects. Previously was the unique identifier for user records, for Iterable customers before August 2023.  Exists if using the history version of the table.
           - name: additional_properties
@@ -263,7 +258,7 @@ sources:
           - name: list_type
             description: The list type
           - name: created_at
-            description: Creation timestamp as epoch time in milliseconds
+            description: Creation timestamp
           - name: description
             description: Information about the list
 
@@ -299,15 +294,15 @@ sources:
           - name: template_type
             description: The type of the template
           - name: created_at
-            description: Creation timestamp as epoch time in milliseconds
+            description: Creation timestamp
           - name: client_template_id
             description: Reference to the client template
           - name: creator_user_id
             description: Reference to the user who created the template
           - name: message_type_id
             description: Reference to the message type associated with the template
           - name: updated_at
-            description: Last update timestamp as epoch time in milliseconds
+            description: Last update timestamp
 
       - name: user_history
         identifier: "{{ var('iterable_user_history_identifier', 'user_history')}}"
@@ -326,11 +321,11 @@ sources:
           - name: email_list_ids
             description: Array of references to the lists associated with the user
           - name: signup_date
-            description: Timestamp as epoch time in milliseconds of when the user signed up
+            description: Timestamp of when the user signed up
           - name: signup_source
             description: Source where the user signed up
           - name: updated_at
-            description: Last update timestamp as epoch time in milliseconds
+            description: Last update timestamp
 
       - name: user_unsubscribed_channel
         identifier: "{{ var('iterable_user_unsubscribed_channel_identifier', 'user_unsubscribed_channel')}}"
@@ -362,7 +357,7 @@ sources:
           - name: email
             description: Unique identifier of the user
           - name: updated_at
-            description: Last update timestamp as epoch time in milliseconds
+            description: Last update timestamp
 
       - name: user_unsubscribed_message_type_history
         identifier: "{{ var('iterable_user_unsubscribed_message_type_history_identifier', 'user_unsubscribed_message_type_history')}}"
@@ -375,4 +370,4 @@ sources:
           - name: email
             description: Unique identifier of the user
           - name: updated_at
-            description: Last update timestamp as epoch time in milliseconds
+            description: Last update timestamp

models/stg_iterable.yml

@@ -14,7 +14,7 @@ models:
         tests:
           - not_null
       - name: updated_at
-        description: Last update timestamp as epoch time in milliseconds
+        description: Last update timestamp
       - name: campaign_name
         description: User defined name of the campaign.
       - name: campaign_state
@@ -24,11 +24,11 @@ models:
       - name: send_size
         description: Size of the campaign. Number of individuals included in the campaign
       - name: start_at
-        description: Start timestamp as epoch time in milliseconds
+        description: Start timestamp
       - name: ended_at
-        description: Ended timestamp as epoch time in milliseconds
+        description: Ended timestamp
       - name: created_at
-        description: Creation timestamp as epoch time in milliseconds
+        description: Creation timestamp
       - name: recurring_campaign_id
         description: Reference to the recurring campaign, if applicable
       - name: created_by_user_id
@@ -54,7 +54,7 @@ models:
       - name: label
         description: Name of the label
       - name: updated_at
-        description: Last update timestamp as epoch time in milliseconds
+        description: Last update timestamp
       - name: _fivetran_synced
         description: "{{ doc('_fivetran_synced') }}"
 
@@ -68,7 +68,7 @@ models:
       - name: list_id
         description: Reference to the list used
       - name: updated_at
-        description: Last update timestamp as epoch time in milliseconds
+        description: Last update timestamp
       - name: _fivetran_synced
         description: "{{ doc('_fivetran_synced') }}"
 
@@ -82,7 +82,7 @@ models:
       - name: suppressed_list_id
         description: Reference to the list used
       - name: updated_at
-        description: Last update timestamp as epoch time in milliseconds
+        description: Last update timestamp
       - name: _fivetran_synced
         description: "{{ doc('_fivetran_synced') }}"
 
@@ -105,24 +105,21 @@ models:
 
   - name: stg_iterable__event
     description: Table containing individual events within campaigns.
-    tests:
-      - dbt_utils.unique_combination_of_columns:
-          combination_of_columns:
-            - unique_event_id
-            - created_at
     columns:
       - name: event_id
         description: A Fivetran-created unique identifier derived from hashing campaign_id, created_at, and event_name. 
         tests:
           - not_null
       - name: unique_event_id
         description: A surrogate key made up of event_id and _fivetran_user_id.
+        tests:
+          - unique
       - name: campaign_id
         description: Reference to the campaign from which the event originated
       - name: content_id
         description: Reference to the content the event is associated with
       - name: created_at
-        description: Creation timestamp as epoch time in milliseconds
+        description: Creation timestamp
       - name: created_on
         description: Creation date
       - name: email
@@ -261,7 +258,7 @@ models:
       - name: list_type
         description: The list type
       - name: created_at
-        description: Creation timestamp as epoch time in milliseconds
+        description: Creation timestamp
       - name: list_description
         description: Information about the list
       - name: _fivetran_synced
@@ -309,15 +306,15 @@ models:
       - name: template_type
         description: The type of the template
       - name: created_at
-        description: Creation timestamp as epoch time in milliseconds
+        description: Creation timestamp
       - name: client_template_id
         description: Reference to the client template
       - name: creator_user_id
         description: Reference to the user who created the template
       - name: message_type_id
         description: Reference to the message type associated with the template
       - name: updated_at
-        description: Last update timestamp as epoch time in milliseconds
+        description: Last update timestamp
       - name: _fivetran_synced
         description: "{{ doc('_fivetran_synced') }}"
 
@@ -344,11 +341,11 @@ models:
       - name: email_list_ids
         description: Array of references to the lists associated with the user
       - name: signup_date
-        description: Timestamp as epoch time in milliseconds of when the user signed up
+        description: Timestamp of when the user signed up
       - name: signup_source
         description: Source where the user signed up
       - name: updated_at
-        description: Last update timestamp as epoch time in milliseconds
+        description: Last update timestamp
       - name: phone_number 
         description: Phone number associated with the user
       - name: iterable_user_id
@@ -376,7 +373,7 @@ models:
       - name: latest_batch_index
         description: Indexes each user record based on the most recent available.
       - name: updated_at
-        description: Last update timestamp as epoch time in milliseconds
+        description: Last update timestamp
       - name: _fivetran_synced
         description: "{{ doc('_fivetran_synced') }}"
 
@@ -398,6 +395,6 @@ models:
       - name: latest_batch_index
         description: Indexes each user record based on the most recent available.
       - name: updated_at
-        description: Last update timestamp as epoch time in milliseconds
+        description: Last update timestamp
       - name: _fivetran_synced
         description: "{{ doc('_fivetran_synced') }}"

models/stg_iterable__user_history.sql

@@ -1,3 +1,4 @@
+{{ config(materialized='view') }}
 
 with base as (