Skip to content

Commit

Permalink
Add optional attribute to spec
Browse files Browse the repository at this point in the history
The current Recap null-field handling is a bit cumbersome. I discussed with
@gwukelic, @adrianisk, and @cpard. We decided to add support for an `optional`
field to the CST. This will allow us to specify that a field is optional.

See the conversation here:

gabledata/recap#335

Users can now specify an optional field as:

```yaml
type: struct
fields:
  - name: secondary_phone
    type: string32
    optional: true
```

This will get treated the same as:

```yaml
type: struct
fields:
  - name: secondary_phone
    type: "union",
    types: ["null", "string32"],
    default: null
```
  • Loading branch information
criccomini committed Jul 29, 2023
1 parent f4f390c commit 292de0f
Showing 1 changed file with 98 additions and 19 deletions.
117 changes: 98 additions & 19 deletions specs/type/0.2.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -209,16 +209,11 @@ An ordered collection of Recap types. Table schemas are typically represented as
#### `field` Attributes
{: .no_toc }

Recap types in the `fields` attribute can have two extra attributes set: `name` and `default`.
Recap types in the `fields` attribute can have an extra `name` attribute to set a field's name.

| Name | Description | Type | Required | Default |
|------|-------------|------|----------|---------|
| name | The field's name. | String | NO | |
| default | The default value for a reader if the field is not set in the struct. | Literal of any type | NO | |

An unset default is differentiated from a default with a null value. An unset default is treated as "no default", while a default that's been set to null is treated as a null default.

*NOTE: Database defaults often appear as strings like `nextval('\"public\".some_id_seq'::regclass)` or `'USD'::character varying`. Such defaults are left to the developer to interpret based on the database they're using.*

#### Examples
{: .no_toc }
Expand All @@ -235,18 +230,6 @@ fields:
bytes: 255
```

Optional fields are expressed as a union with a `null` type and a null default (similar to [Avro's fields](https://avro.apache.org/docs/1.10.2/spec.html#schema_record)).

```yaml
# A struct with an optional string field called "secondary_phone"
type: struct
fields:
- name: secondary_phone
type: union
types: ["null", "string32"]
default: null
```

### `enum`

An enumeration of string symbols.
Expand Down Expand Up @@ -314,6 +297,93 @@ types:
bits: 32
```

## Defaults

Recap types can have a `default` attribute with a literal value. The value is used when the field is not set in a struct.

| Name | Description | Type | Required | Default |
|------|-------------|------|----------|---------|
| default | The default value for a reader if the field is not set in the struct. | Literal of any type | NO | |

```yaml
# A struct with a field called "id" that defaults to 0
type: struct
fields:
- name: id
type: int
bits: 32
default: 0
```

A missing default is differentiated from a default with a null value. An unset default is treated as "no default", while a default that's been set to null is treated as a null default.

*NOTE: Database defaults often appear as strings like `nextval('\"public\".some_id_seq'::regclass)` or `'USD'::character varying`. Such defaults are left to the developer to interpret based on the database they're using.*

## Optional Types

Optional types are expressed as a union with a `null` type and a null default (similar to [Avro's fields](https://avro.apache.org/docs/1.10.2/spec.html#schema_record)).

```yaml
# A struct with an optional string field called "secondary_phone"
type: struct
fields:
- name: secondary_phone
type: union
types: ["null", "string32"]
default: null
```

Unions can be cumbersome, so Recap supports a shorthand syntax for types using `optional`:

```yaml
# A struct with an optional string field called "secondary_phone"
type: struct
fields:
- name: secondary_phone
type: string32?
optional: true
```

The above shorthand is equivalent to the union example.

Optional types also work for unions:

```yaml
# A union type of null or a 32-bit signed int
type: union
types:
- type: int32
- type: float32
optional: true
```

Optionals also work with aliases:

```yaml
# A struct with an optional string field called "secondary_phone"
type: struct
fields:
- name: phone
alias: phone_type
type: string32
- name: secondary_phone
type: phone_type
optional: true
```

Optionality is not inherited in aliases:

```yaml
type: struct
fields:
- name: phone
alias: phone_type
type: string32
optional: true
- name: secondary_phone
type: phone_type
```

## Logical Types

Logical types annotate one of the 11 Recap types listed in the *Types* section at the top of the spec. Logical types add additional context to Recap types that help converters. For example, a `decimal` logical type can be defined as:
Expand Down Expand Up @@ -665,4 +735,13 @@ Recap comes with a collection of built-in aliases:
* `time64`: Time since midnight without timezones and leap seconds in a 64-bit signed integer.
* `timestamp64`: Time elapsed (in a time unit) since a specific epoch.
* `date32`: Date since the UNIX epoch without timezones and leap seconds in a 32-bit integer.
* `date64`: Date since the UNIX epoch without timezones and leap seconds in a 64-bit integer.
* `date64`: Date since the UNIX epoch without timezones and leap seconds in a 64-bit integer.

## Changelog

* 0.2.0
* Add `optional` attribute. ([#335](https://github.com/recap-build/recap/pull/335))
* 0.1.1
* Make `string` and `bytes` byte attribute optional. ([#284](https://github.com/recap-build/recap/discussions/284) [#292](https://github.com/recap-build/recap/pull/292) [#293](https://github.com/recap-build/recap/pull/293))
* 0.1.0
* Initial release.

0 comments on commit 292de0f

Please sign in to comment.