3.0.8

What's Changed

• Added logic to fully resolve targets (with prefixes/suffixes) when updating the Data Preparation YAML. by @fernst in #1866
• Publishing Dataform security patches by @Tuseeq1 in #1867

Full Changelog: 3.0.7...3.0.8

3.0.7

What's Changed

• Creating release notes as part of publishing process by @Tuseeq1 in #1857
• Fix uniqueKey assertions for views, incremental tables by @bmagyarkuti in #1836
• Bump version to 3.5 to rollup uniquekeys fix by @Ekrekr in #1858
• Bump the patch version to 3.0.6 by @Tuseeq1 in #1859
• Change variable name in cloudbuild.yaml by @Tuseeq1 in #1860
• Bump the patch version by @Tuseeq1 in #1861

Full Changelog: 3.0.4...3.0.7

3.0.4

What's Changed

• Update WORKSPACE by @a2wd in #1512
• Refresh test credentials by @Ekrekr in #1810
• Update output for Data Preparation processing by @fernst in #1808
• Sync proto structure and message names for data preparation. by @fernst in #1811
• Temporarily disable docs script during publish by @Ekrekr in #1817
• Added support for compilation overrides in Data Preparations by @fernst in #1818
• Bump dataprep proto to latest config by @Ekrekr in #1825
• fix: vscode extension crash on windows due to *.cmd spawn by @moker-spaghetti in #1814
• Update unit test's result messages by @spider-man-tm in #1812
• Restore ability to unit test views by @bmagyarkuti in #1824
• Bump elliptic from 6.5.4 to 6.5.7 by @dependabot in #1831
• fix(1805): dry run fails when table/view/assertion already does not exists by @ashish10alex in #1806
• Revert "fix(1805): dry run fails when table/view/assertion already does not exists" by @Ekrekr in #1834
• Update Data Preparation processor to remove proto dependency. by @fernst in #1839
• MOSS Cap1 by @Tuseeq1 in #1833
• Correct permission and branch name by @Tuseeq1 in #1845
• Remove cloudbuild.yaml by @Tuseeq1 in #1847
• Make published packages include the version.bzl file by @Ekrekr in #1848
• Remove data preparation proto definition. by @fernst in #1849
• Publish a new NPM versions by @Tuseeq1 in #1852
• Branch name from master to main by @Tuseeq1 in #1853
• Local test fix with npm install on cloudbuild by @Tuseeq1 in #1850
• Fixing the name for automaticallty created PR by @Tuseeq1 in #1856
• Publish a new NPM versions by @Tuseeq1 in #1855

New Contributors

• @spider-man-tm made their first contribution in #1812

Full Changelog: 3.0.2...3.0.4

3.0.2

What's Changed

• Fix catchall errors by @Ekrekr in #1802

Full Changelog: 3.0.1...3.0.2

3.0.1

Important: Prefer 3.0.2 over this release - this release has issues.

See #1800 for more context.

What's Changed

• Enable File-Specific Formatting in dataform format by @hiracky16 in #1754
• Assertions configs as protos by @Ekrekr in #1731
• Fix vscode extension crashes on windows by @moker-spaghetti in #1751
• Update the version of the vscode extension by @GJMcGowan in #1759
• Fix empty object internal error by @Ekrekr in #1763
• Split table tests by @Ekrekr in #1761
• Fix protected default, add a unit test for SQLX config parameters by @Ekrekr in #1718
• Declaration configs as protos by @Ekrekr in #1767
• Add support for action configs table assertions and hermeticity, add action configs fields tests by @Ekrekr in #1762
• Rethrow useful errors when reading workflow_settings.yaml by @Ekrekr in #1772
• View config as protos, via a new class by @Ekrekr in #1760
• Incremental tables as protos, via a new class by @Ekrekr in #1773
• Operation config as protos by @Ekrekr in #1766
• Fix vscode ref issue #1768 by @moker-spaghetti in #1769
• Add workflow_settings.yaml validation/completion VSCode extension by @moker-spaghetti in #1770
• fix(1776): dry run of assertions failing by @ashish10alex in #1778
• feat: vscode extension, more intutive compile errors by @ashish10alex in #1775
• Update vscode version to incorporate recent changes by @GJMcGowan in #1784
• Added proto definitions for Data Preparations by @fernst in #1788
• Handling for Data Preparation actions by @fernst in #1789
• Support building on arm Mac, Linux environments by @bmagyarkuti in #1790
• fix(1781): go to definition when namePrefix is specified in workflow_settings.yaml by @ashish10alex in #1783
• Fix vscode server linting by @Ekrekr in #1797
• Make the stateless CLI package installation error clearer by @Ekrekr in #1795
• Regenerate configs reference with camelCase fields by @Ekrekr in #1796
• Fix notebook dependency targets by @Ekrekr in #1792
• Bumped version to 3.0.1 by @fernst in #1798
• Remove some outdated configs references, fix generated formatting by @Ekrekr in #1799

New Contributors

• @hiracky16 made their first contribution in #1754
• @moker-spaghetti made their first contribution in #1751
• @ashish10alex made their first contribution in #1778
• @fernst made their first contribution in #1788

Full Changelog: 3.0.0...3.0.1

3.0.0

TL;DR of What's Changed Since 2.9.0

dataform.json -> workflow_settings.yaml

workflow_settings.yaml has been introduced, which will gradually replace dataform.json in a later version; there is no immediate action to be taken, as dataform.json files are still valid in projects with Dataform Core 3.0.0.

dataform.json is being deprecated in favor of workflow_settings.yaml. This means that:

• Workflow settings are now strictly typed, in Protobuf format.
• The Dataform Core version can be specified directly in the workflow_settings.yaml file. Note: to have more than just @dataform/core as a dependency, a package.json must still be used.

Example conversion of workflow_settings.yaml:

defaultProject: dataform-demos
defaultLocation: us
defaultDataset: dataform
defaultAssertionDataset: dataform_assertions
dataformCoreVersion: 3.0.0
vars:
    environmentName: "development"

The above is equivalent to the dataform.json file:

{
  "warehouse": "bigquery",
  "defaultDatabase": "dataform-demos",
  "defaultLocation": "us",
  "defaultSchema": "dataform",
  "assertionSchema": "dataform_assertions"
  "vars": {
    "environmentName": "development"
  }
}

Notebooks Actions and actions.yaml

Notebooks as Dataform actions are on their way - but not quite yet! They're part of the compiled graph, and soon they'll be executable.

A new way of configuring action configs through actions.yaml has been implemented to support this.

An example of loading a notebook in Dataform can be seen at https://github.com/dataform-co/dataform/tree/main/examples/extreme_weather_programming.

Stateless Package Installation by @dataform/cli

Package installation by @dataform/cli is now stateless! The CLI will install NPM packages during compilation if version is defined in the workflow_settings.yaml file.

This means no node_modules folder has to be seen in the project, and Dataform users no longer need to be familiar with NPM.

Compilation Output is Now Warehouse Agnostic

Previously the output of compilation results from @dataform/core would insert warehouse specific SQL into the compiled graph. Where possible, this has been removed - transferring the responsibility of inserting warehouse specific SQL into whichever execution engine is running Dataform.

Additionally, support for non-BigQuery warehouses has been dropped. We're in discussions with Datashell for them to provide a warehouse-agnostic CLI execution engine based off of Dataform compiled graphs. In the meantime however, if you need support for a non-BigQuery warehouse, please continue using the latest version starting with 2.x.x!

dependOnDependencyAssertions

An easier ways to add assertions from dependency as dependencies has been introduced.

dependOnDependencyAssertions in config blocks can be used to add assertions from all dependencies of the action as dependencies.

config { 
    type: "view",
    dependOnDependencyAssertions: true,
    dependencies: ["some_table"]
}

select test from ${ref("some_other_table")}

Additionally, the includeDependentAssertions parameter can be used when setting individual dependencies either in config.dependencies or in ref() to add assertions for these dependencies as the dependencies for current action.

config { 
    type: "view",
    dependencies: [{name: "some_table", includeDependentAssertions: true}]
}

select test from ${ref({name: "some_other_table", includeDependentAssertions: true})}

Full Changelog from 2.9.0: 2.9.0...3.0.0

3.0.0-beta.6

What's Changed

• Fix and re-enable integration tests by @Ekrekr in #1729
• Make example tests run off of the CLI, tidy testing/ by @Ekrekr in #1730
• Bump semver from 5.6.0 to 7.6.0 by @dependabot in #1724
• Some project config improvements by @Ekrekr in #1732
• Make proto object verification give more descriptive error messages by @Ekrekr in #1733
• Add tests for locking in SQLX config options validity by @Ekrekr in #1734
• Add tests for compilation errors where they're missing by @Ekrekr in #1735
• Fix bigquery dryrun by @Ekrekr in #1742
• Make formatting in the CLI use the same SQL formatter as BigQuery by @Ekrekr in #1741

Full Changelog: 3.0.0-beta.5...3.0.0-beta.6

3.0.0-beta.5

The description of dependOnDependencyAssertions has been moved to https://github.com/dataform-co/dataform/releases/tag/3.0.0.

What's Changed

• Update to using GitHub actions for tests by @Ekrekr in #1708
• Re-add cloud build while a new GCP project is set up for GitHub actions by @Ekrekr in #1710
• Update contributing tests guide to just test core by @Ekrekr in #1711
• Bump tar from 6.1.11 to 6.2.1 by @dependabot in #1715
• Bump semver from 5.6.0 to 7.6.0 by @dependabot in #1716
• Bump protobufjs from 7.2.4 to 7.2.5 by @dependabot in #1714
• Update cloudbuild file to default name, make yaml by @Ekrekr in #1722
• Make cloud build for dataform-open-source work for pull requests by @Ekrekr in #1728
• Feature: dependent assertions as dependencies by @Tuseeq1 in #1720

New Contributors

• @Tuseeq1 made their first contribution in #1720

Full Changelog: 3.0.0-beta.4...3.0.0-beta.5

3.0.0-beta4

What's Changed

• Expose defaultLocation in dataform.projectConfig. by @BenBirt in #1705
• Publish 3.0.0-beta.4. by @BenBirt in #1706

Full Changelog: 3.0.0-beta.3...3.0.0-beta.4

3.0.0-beta.3

What's Changed

• Add notebook runtime options to workflow settings by @Ekrekr in #1686
• Add a configs reference page, and tidy up other readmes by @Ekrekr in #1688
• Bump follow-redirects from 1.15.4 to 1.15.6 by @dependabot in #1690
• Remove rules docker by @Ekrekr in #1689
• Fix credentials path resolution when projectDir is specified by @Ekrekr in #1694
• Fix CLI tags, allow comma delisted tags, add test by @Ekrekr in #1695
• Prevent confusion around JSON printing of the execution graph by @Ekrekr in #1697
• Extend dryrun to do dry runs against BigQuery by @Ekrekr in #1698
• Add bool argument to disable, default true to keep the API the same by @Ekrekr in #1700
• Throw errors when dots are in fields in targets by @Ekrekr in #1699
• Normalize paths during compilation by @Ekrekr in #1702

Full Changelog: 3.0.0-beta.1...3.0.0-beta.3