PR #57 includes the following update:
- Casted specific timestamp fields across all staging models as dates where the Netsuite UI does not perform timezone conversion. Keeping these fields as type timestamp causes issues in reporting tools that perform automatic timezone conversion.
- As this will change the datatype of the underlying fields, this will require a
--full-refreshfor downstream incremental models.
- As this will change the datatype of the underlying fields, this will require a
- Existing fields that were converted from timestamp to date in the following
stg_netsuite2__*models:
| Table | Source Field Name | Field Alias |
|---|---|---|
| accounting_periods | startdate | starting_at |
| accounting_periods | enddate | ending_at |
| customers | firstorderdate | date_first_order_at |
| transactions | closedate | closed_at |
| transactions | duedate | due_date_at |
| transactions | trandate | transaction_date |
- Adds additional commonly used fields within the
stg_netsuite2__*models.
| Table | Source Field Name | Field Alias |
|---|---|---|
| accounts | accountsearchdisplaynamecopy | display_name |
| customers | altname | alt_name |
| subsidiaries | iselimination | is_elimination |
| transaction_accounting_lines | exchangerate | exchange_rate |
| transaction_lines | eliminate | is_eliminate |
| transaction_lines | netamount | netamount |
| transactions | reversal | reversal_transaction_id |
| transactions | reversaldate | reversal_date |
| transactions | reversaldefer | is_reversal_defer |
IMPORTANT: Nearly all of the affected models have pass-through functionality. If you have already been using passthrough column variables to include the above newly added fields (without aliases), you MUST remove the fields from your passthrough variable configuration in order to avoid duplicate column errors.
- Introduced the
stg_netsuite2__employeesmodel to bring in data from theemployeesource table.- Since this model is only used by a subset of customers, we've introduced the variable
netsuite2__using_employeesto allow users who don't utilize theemployeetable in Netsuite2 the ability to disable that functionality within yourdbt_project.yml. This value is set to true by default. Instructions are available in the README for how to disable this variable.
- Since this model is only used by a subset of customers, we've introduced the variable
- Created new seed data in
integration_teststo support the newstg_netsuite2__employeesmodel, as well as the new fields introduced into the new Netsuite2 staging models.
PR #51 includes the following update:
- Added the
entities_pass_through_columnsvariable to be leveraged within thestg_netsuite2__entitiesmodel. This variable will allow users to pass through fields which are not natively included to the package staging model.- Removed fields from the
get_entity_columnsmacro that are not brought into thestg_netsuite2__entitiesmodel to ensure that the passthrough feature works properly.
- Removed fields from the
PR #47 includes the following update:
- Added column
_fivetran_synced_dateto modelstg_netsuite2__transactionsfor use in downstream models. - To reduce storage, updated default materialization of staging models to views.
⚠️ Running a--full-refreshwill be required if you have previously run these staging models as tables and get the following error:Trying to create view <model path> but it currently exists as a table. Either drop <model path> manually, or run dbt with `--full-refresh` and dbt will drop it for you.
- Added integration testing pipeline for Databricks SQL Warehouse.
- Included auto-releaser GitHub Actions workflow to automate future releases.
PR #42 includes the following update:
- Addition of the
currencyfield ascurrency_idwithin thestg_netsuite2__subsidiariesmodel.- This addition likely won't be breaking for a large number of users. However, if you are leveraging the
subsidiaries_pass_through_columnsvariable and have includedcurrencyas a pass through column, you will experience a duplicate column error. We advise you remove thecurrencyfield as a pass through column to avoid runtime errors following this upgrade.
- This addition likely won't be breaking for a large number of users. However, if you are leveraging the
- Addition of the
is_eliminateandspecial_account_type_idfields within thestg_netsuite2__accountsmodel. (PR #39 and PR #40)- This addition likely won't be breaking for a large number of users. However, if you are leveraging the
accounts_pass_through_columnsvariable and have included eithereliminateorsspecacctas a pass through columns then you will experience a duplicate column error. We advise you remove theeliminateandsspecacctfields as a pass through columns to avoid runtime errors following this upgrade.
- This addition likely won't be breaking for a large number of users. However, if you are leveraging the
- Brought in
accounting_book_id(accountingbook) to thestg_netsuite2__consolidated_exchange_ratesmodel. (PR #36)- Due to logic within the dbt_netsuite v0.8.1 package version, this update will result in an ambiguous columns error. Therefore, this update is treated as a breaking change.
PR #35 applies the following changes:
- Introduces variable
netsuite2__using_exchange_rateto allow users who don't utilize exchange rates in Netsuite2 the ability to disable that functionality within yourdbt_project.yml, since these models aren't being used. - Note: If you are using the
dbt_netsuitepackage, be sure to set it globally by inserting this code into yourdbt_project.yml:
vars:
netsuite2__using_exchange_rate: false- Incorporated the new
fivetran_utils.drop_schemas_automationmacro into the end of each Buildkite integration test job. - Updated the pull request templates.
- Introduces the
netsuite2__using_jobsvariable in order to disable thestg_netsuite2__jobsmodel. Setnetsuite2__using_jobstoFalseif you do not have thejobtable in your Netsuite2 schema (PR #30).
- In v0.6.0, we introduced pass through columns for the
vendorsanditemstables. However, this update was only applied to the originalNetsuiteversion of this package, and notNetsuite2as well. These pass through columns are accessible in Netsuite2 data models (PR #27).
- Now introducing...Databricks compatibility 🧱 (PR #26)
PR #24 includes the following breaking changes:
- Dispatch update for dbt-utils to dbt-core cross-db macros migration. Specifically
{{ dbt_utils.<macro> }}have been updated to{{ dbt.<macro> }}for the below macros:any_valuebool_orcast_bool_to_textconcatdate_truncdateadddatediffescape_single_quotesexcepthashintersectlast_daylengthlistaggpositionreplacerightsafe_castsplit_partstring_literaltype_biginttype_floattype_inttype_numerictype_stringtype_timestamparray_appendarray_concatarray_construct
- For
current_timestampandcurrent_timestamp_in_utcmacros, the dispatch AND the macro names have been updated to the below, respectively:dbt.current_timestamp_backcompatdbt.current_timestamp_in_utc_backcompat
dbt_utils.surrogate_keyhas also been updated todbt_utils.generate_surrogate_key. Since the method for creating surrogate keys differ, we suggest all users do afull-refreshfor the most accurate data. For more information, please refer to dbt-utils release notes for this update.packages.ymlhas been updated to reflect new defaultfivetran/fivetran_utilsversion, previously[">=0.3.0", "<0.4.0"]now[">=0.4.0", "<0.5.0"].
- Added the
vendors_pass_through_columnsvariable to be leveraged within thestg_netsuite__vendorsmodel. This variable will allow users to pass through fields which are not natively included to the package staging model. This variable is empty by default and may be overridden within a users rootdbt_project.yml. (#23) - Added the
items_pass_through_columnsvariable to be leveraged within thestg_netsuite__itemsmodel. This variable will allow users to pass through fields which are not natively included to the package staging model. This variable is empty by default and may be overridden within a users rootdbt_project.yml. (#23)
- Introduces the
netsuite2__multibook_accounting_enabledandnetsuite2__using_vendor_categoriesvariables to disable their related source tables and downstream models (#21).netsuite2__multibook_accounting_enabledisTrueby default. Set it toFalseif you do not use the Multi-Book Accounting feature in Netsuite and/or do not have theaccountingbookandaccountingbooksubsidiariessource tables.netsuite2__using_vendor_categoriesisTrueby default. Set it toFalseif you do not categorize vendors in Netsuite and/or do not have thevendorcategorysource table.
🎉 Netsuite2 Compatibility 🎉 PRs #15 and #17 include the following update to the dbt_netsuite_source package:
- The declaration of passthrough variables within your root
dbt_project.ymlhas changed. To allow for more flexibility and better tracking of passthrough columns, you will now want to define passthrough columns in the following format:
This applies to all passthrough columns within the
dbt_netsuite_sourcepackage and not just thecustomers_pass_through_columnsexample.
vars:
customers_pass_through_columns:
- name: "my_field_to_include" # Required: Name of the field within the source.
alias: "field_alias" # Optional: If you wish to alias the field within the staging model.
transform_sql: "cast(field_alias as string)" # Optional: If you wish to define the datatype or apply a light transformation.- Addition of the
netsuite_data_modelvariable. This variable may either benetsuite(the original Netsuite.com connector endpoint) ornetsuite2(the new Netsuite2 connector endpoint).- The variable is set to
netsuiteby default. If you wish to run the data models for the Netsuite2 connector, you may simply change the variable within your root dbt_project.yml tonetsuite2.
- The variable is set to
- Postgres compatibility!
- Added identifier variables to each Netsuite.com and Netsuite2 source to enable dynamic source table-name adjustments.
- Applied schema level tests to each source table to ensure data validation.
- README updates for easier navigation and package use.
- Removed the
_fivetran_syncedcolumn definition from theget_customers_columnsmacro, as this is consistent with other macros in the project. Previously, this introduced a bug that caused an ambiguous column name error when passing the_fivetran_syncedcolumn to thecustomers_pass_through_columnsvariable since the column is not actually listed in thestg_netsuite__customersmodel field list. Now, when passing the_fivetran_syncedcolumn to the pass through variable, the ambiguous column error no longer happens. (#16)
- Added this changelog to capture iterations of the package!
- Added the
subsidiaries_pass_through_columnsvariable to be leveraged within thestg_netsuite__subsidiariesmodel. This variable will allow users to pass through fields which are not natively included to the package staging model. This variable is empty by default and may be overridden within a users rootdbt_project.yml. (#13) - Added the
consolidated_exchange_rates_pass_through_columnsvariable to be leveraged within thestg_netsuite__consolidated_exchange_ratesmodel. This variable will allow users to pass through fields which are not natively included to the package staging model. This variable is empty by default and may be overridden within a users rootdbt_project.yml. (#13)
🎉 dbt v1.0.0 Compatibility 🎉
- Adjusts the
require-dbt-versionto now be within the range [">=1.0.0", "<2.0.0"]. Additionally, the package has been updated for dbt v1.0.0 compatibility. If you are using a dbt version <1.0.0, you will need to upgrade in order to leverage the latest version of the package.- For help upgrading your package, I recommend reviewing this GitHub repo's Release Notes on what changes have been implemented since your last upgrade.
- For help upgrading your dbt project to dbt v1.0.0, I recommend reviewing dbt-labs upgrading to 1.0.0 docs for more details on what changes must be made.
- Upgrades the package dependency to refer to the latest
dbt_fivetran_utils. The latestdbt_fivetran_utilspackage also has a dependency ondbt_utils[">=0.8.0", "<0.9.0"].- Please note, if you are installing a version of
dbt_utilsin yourpackages.ymlthat is not in the range above then you will encounter a package dependency error.
- Please note, if you are installing a version of
- Added this changelog to capture iterations of the package!
- Added the
customers_pass_through_columnsvariable to be leveraged within thestg_netsuite__customersmodel. This variable will allow users to pass through fields which are not natively included to the package staging model. This variable is empty by default and may be overridden within a users rootdbt_project.yml. (#10) - Added the
locations_pass_through_columnsvariable to be leveraged within thestg_netsuite__locationsmodel. This variable will allow users to pass through fields which are not natively included to the package staging model. This variable is empty by default and may be overridden within a users rootdbt_project.yml. (#10)
Refer to the relevant release notes on the Github repository for specific details for the previous releases. Thank you!