Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove some outdated configs references, fix generated formatting #1799

Merged
merged 1 commit into from
Aug 7, 2024
Merged

Remove some outdated configs references, fix generated formatting #1799

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 7 additions & 25 deletions docs/configs-reference.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,14 +168,8 @@ Action config defines the contents of `actions.yaml` configuration files.
| requirePartitionFilter | [bool](#bool) | | Declares whether the partitioned table requires a WHERE clause predicate filter that filters the partitioning column. |
| updatePartitionFilter | [string](#string) | | SQL-based filter for when incremental updates are applied. |
| clusterBy | [string](#string) | repeated | The keys by which to cluster partitions by. See https://cloud.google.com/dataform/docs/partitions-clusters. |
| labels | [ActionConfig.IncrementalTableConfig.LabelsEntry](#dataform-ActionConfig-IncrementalTableConfig-LabelsEntry) | repeated | Key-value pairs for BigQuery labels. If the label name contains special characters, e.g. hyphens, then quote its name, e.g. `labels: { "label-name": "value" }`. |
| additionalOptions | [ActionConfig.IncrementalTableConfig.AdditionalOptionsEntry](#dataform-ActionConfig-IncrementalTableConfig-AdditionalOptionsEntry) | repeated | Key-value pairs of additional options to pass to the BigQuery API.

Some options, for example, partitionExpirationDays, have dedicated type/validity checked fields. For such options, use the dedicated fields.

String values must be encapsulated in double-quotes, for example: additionalOptions: {numeric_option: "5", string_option: '"string-value"'}

If the option name contains special characters, encapsulate the name in quotes, for example: additionalOptions: { "option-name": "value" }. |
| labels | [ActionConfig.IncrementalTableConfig.LabelsEntry](#dataform-ActionConfig-IncrementalTableConfig-LabelsEntry) | repeated | Key-value pairs for BigQuery labels. |
| additionalOptions | [ActionConfig.IncrementalTableConfig.AdditionalOptionsEntry](#dataform-ActionConfig-IncrementalTableConfig-AdditionalOptionsEntry) | repeated | Key-value pairs of additional options to pass to the BigQuery API. Some options, for example, partitionExpirationDays, have dedicated type/validity checked fields. For such options, use the dedicated fields. |
| dependOnDependencyAssertions | [bool](#bool) | | When set to true, assertions dependent upon any dependency will be add as dedpendency to this action |
| assertions | [ActionConfig.TableAssertionsConfig](#dataform-ActionConfig-TableAssertionsConfig) | | Assertions to be run on the dataset. If configured, relevant assertions will automatically be created and run as a dependency of this dataset. |
| hermetic | [bool](#bool) | | If true, this indicates that the action only depends on data from explicitly-declared dependencies. Otherwise if false, it indicates that the action depends on data from a source which has not been declared as a dependency. |
Expand Down Expand Up @@ -326,14 +320,8 @@ the column(s) in the unique key(s).
| partitionExpirationDays | [int32](#int32) | | The number of days for which BigQuery stores data in each partition. The setting applies to all partitions in a table, but is calculated independently for each partition based on the partition time. |
| requirePartitionFilter | [bool](#bool) | | Declares whether the partitioned table requires a WHERE clause predicate filter that filters the partitioning column. |
| clusterBy | [string](#string) | repeated | The keys by which to cluster partitions by. See https://cloud.google.com/dataform/docs/partitions-clusters. |
| labels | [ActionConfig.TableConfig.LabelsEntry](#dataform-ActionConfig-TableConfig-LabelsEntry) | repeated | Key-value pairs for BigQuery labels. If the label name contains special characters, e.g. hyphens, then quote its name, e.g. `labels: { "label-name": "value" }`. |
| additionalOptions | [ActionConfig.TableConfig.AdditionalOptionsEntry](#dataform-ActionConfig-TableConfig-AdditionalOptionsEntry) | repeated | Key-value pairs of additional options to pass to the BigQuery API.

Some options, for example, partitionExpirationDays, have dedicated type/validity checked fields. For such options, use the dedicated fields.

String values must be encapsulated in double-quotes, for example: additionalOptions: {numeric_option: "5", string_option: '"string-value"'}

If the option name contains special characters, encapsulate the name in quotes, for example: additionalOptions: { "option-name": "value" }. |
| labels | [ActionConfig.TableConfig.LabelsEntry](#dataform-ActionConfig-TableConfig-LabelsEntry) | repeated | Key-value pairs for BigQuery labels. |
| additionalOptions | [ActionConfig.TableConfig.AdditionalOptionsEntry](#dataform-ActionConfig-TableConfig-AdditionalOptionsEntry) | repeated | Key-value pairs of additional options to pass to the BigQuery API. Some options, for example, partitionExpirationDays, have dedicated type/validity checked fields. For such options, use the dedicated fields. |
| dependOnDependencyAssertions | [bool](#bool) | | When set to true, assertions dependent upon any dependency will be add as dedpendency to this action |
| assertions | [ActionConfig.TableAssertionsConfig](#dataform-ActionConfig-TableAssertionsConfig) | | Assertions to be run on the dataset. If configured, relevant assertions will automatically be created and run as a dependency of this dataset. |
| hermetic | [bool](#bool) | | If true, this indicates that the action only depends on data from explicitly-declared dependencies. Otherwise if false, it indicates that the action depends on data from a source which has not been declared as a dependency. |
Expand Down Expand Up @@ -413,14 +401,8 @@ Target represents a unique action identifier.
| materialized | [bool](#bool) | | Applies the materialized view optimization, see https://cloud.google.com/bigquery/docs/materialized-views-intro. |
| description | [string](#string) | | Description of the view. |
| columns | [ActionConfig.ColumnDescriptor](#dataform-ActionConfig-ColumnDescriptor) | repeated | Descriptions of columns within the table. |
| labels | [ActionConfig.ViewConfig.LabelsEntry](#dataform-ActionConfig-ViewConfig-LabelsEntry) | repeated | Key-value pairs for BigQuery labels. If the label name contains special characters, e.g. hyphens, then quote its name, e.g. `labels: { "label-name": "value" }`. |
| additionalOptions | [ActionConfig.ViewConfig.AdditionalOptionsEntry](#dataform-ActionConfig-ViewConfig-AdditionalOptionsEntry) | repeated | Key-value pairs of additional options to pass to the BigQuery API.

Some options, for example, partitionExpirationDays, have dedicated type/validity checked fields. For such options, use the dedicated fields.

String values must be encapsulated in double-quotes, for example: additionalOptions: {numeric_option: "5", string_option: '"string-value"'}

If the option name contains special characters, encapsulate the name in quotes, for example: additionalOptions: { "option-name": "value" }. |
| labels | [ActionConfig.ViewConfig.LabelsEntry](#dataform-ActionConfig-ViewConfig-LabelsEntry) | repeated | Key-value pairs for BigQuery labels. |
| additionalOptions | [ActionConfig.ViewConfig.AdditionalOptionsEntry](#dataform-ActionConfig-ViewConfig-AdditionalOptionsEntry) | repeated | Key-value pairs of additional options to pass to the BigQuery API. Some options, for example, partitionExpirationDays, have dedicated type/validity checked fields. For such options, use the dedicated fields. |
| dependOnDependencyAssertions | [bool](#bool) | | When set to true, assertions dependent upon any dependency will be add as dedpendency to this action |
| hermetic | [bool](#bool) | | If true, this indicates that the action only depends on data from explicitly-declared dependencies. Otherwise if false, it indicates that the action depends on data from a source which has not been declared as a dependency. |
| assertions | [ActionConfig.TableAssertionsConfig](#dataform-ActionConfig-TableAssertionsConfig) | | Assertions to be run on the dataset. If configured, relevant assertions will automatically be created and run as a dependency of this dataset. |
Expand Down Expand Up @@ -506,7 +488,7 @@ configuration file.
| defaultDataset | [string](#string) | | Required. The default dataset (schema). |
| defaultLocation | [string](#string) | | Required. The default BigQuery location to use. For more information on BigQuery locations, see https://cloud.google.com/bigquery/docs/locations. |
| defaultAssertionDataset | [string](#string) | | Required. The default dataset (schema) for assertions. |
| vars | [WorkflowSettings.VarsEntry](#dataform-WorkflowSettings-VarsEntry) | repeated | Optional. User-defined variables that are made available to project code during compilation. An object containing a list of "key": value pairs. Example: `{ "name": "wrench", "mass": "1.3kg", "count": "3" }`. |
| vars | [WorkflowSettings.VarsEntry](#dataform-WorkflowSettings-VarsEntry) | repeated | Optional. User-defined variables that are made available to project code during compilation. An object containing a list of "key": value pairs. |
| projectSuffix | [string](#string) | | Optional. The suffix to append to all Google Cloud project references. |
| datasetSuffix | [string](#string) | | Optional. The suffix to append to all dataset references. |
| namePrefix | [string](#string) | | Optional. The prefix to append to all action names. |
Expand Down
55 changes: 12 additions & 43 deletions protos/configs.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@ message WorkflowSettings {

// Optional. User-defined variables that are made available to project code
// during compilation. An object containing a list of "key": value pairs.
// Example: `{ "name": "wrench", "mass": "1.3kg", "count": "3" }`.
map<string, string> vars = 6;

// Optional. The suffix to append to all Google Cloud project references.
Expand Down Expand Up @@ -167,22 +166,12 @@ message ActionConfig {
// https://cloud.google.com/dataform/docs/partitions-clusters.
repeated string cluster_by = 15;

// Key-value pairs for BigQuery labels. If the label name contains special
// characters, e.g. hyphens, then quote its name, e.g. `labels: {
// "label-name": "value" }`.
// Key-value pairs for BigQuery labels.
map<string, string> labels = 16;

// Key-value pairs of additional options to pass to the BigQuery API.
//
// Some options, for example, partitionExpirationDays, have dedicated
// type/validity checked fields. For such options, use the dedicated
// fields.
//
// String values must be encapsulated in double-quotes, for example:
// additionalOptions: {numeric_option: "5", string_option: '"string-value"'}
//
// If the option name contains special characters, encapsulate the name in
// quotes, for example: additionalOptions: { "option-name": "value" }.
// Key-value pairs of additional options to pass to the BigQuery API. Some
// options, for example, partitionExpirationDays, have dedicated
// type/validity checked fields. For such options, use the dedicated fields.
map<string, string> additional_options = 17;

// When set to true, assertions dependent upon any dependency will
Expand Down Expand Up @@ -242,22 +231,12 @@ message ActionConfig {
// Descriptions of columns within the table.
repeated ColumnDescriptor columns = 12;

// Key-value pairs for BigQuery labels. If the label name contains special
// characters, e.g. hyphens, then quote its name, e.g. `labels: {
// "label-name": "value" }`.
// Key-value pairs for BigQuery labels.
map<string, string> labels = 13;

// Key-value pairs of additional options to pass to the BigQuery API.
//
// Some options, for example, partitionExpirationDays, have dedicated
// type/validity checked fields. For such options, use the dedicated
// fields.
//
// String values must be encapsulated in double-quotes, for example:
// additionalOptions: {numeric_option: "5", string_option: '"string-value"'}
//
// If the option name contains special characters, encapsulate the name in
// quotes, for example: additionalOptions: { "option-name": "value" }.
// Key-value pairs of additional options to pass to the BigQuery API. Some
// options, for example, partitionExpirationDays, have dedicated
// type/validity checked fields. For such options, use the dedicated fields.
map<string, string> additional_options = 14;

// When set to true, assertions dependent upon any dependency will
Expand Down Expand Up @@ -345,22 +324,12 @@ message ActionConfig {
// https://cloud.google.com/dataform/docs/partitions-clusters.
repeated string cluster_by = 18;

// Key-value pairs for BigQuery labels. If the label name contains special
// characters, e.g. hyphens, then quote its name, e.g. `labels: {
// "label-name": "value" }`.
// Key-value pairs for BigQuery labels.
map<string, string> labels = 19;

// Key-value pairs of additional options to pass to the BigQuery API.
//
// Some options, for example, partitionExpirationDays, have dedicated
// type/validity checked fields. For such options, use the dedicated
// fields.
//
// String values must be encapsulated in double-quotes, for example:
// additionalOptions: {numeric_option: "5", string_option: '"string-value"'}
//
// If the option name contains special characters, encapsulate the name in
// quotes, for example: additionalOptions: { "option-name": "value" }.
// Key-value pairs of additional options to pass to the BigQuery API. Some
// options, for example, partitionExpirationDays, have dedicated
// type/validity checked fields. For such options, use the dedicated fields.
map<string, string> additional_options = 20;

// When set to true, assertions dependent upon any dependency will
Expand Down