Protocol update for column defaults

[dtenedor]

Description

Update Delta log protocol to support column default values.

This will support column default values for Delta Lake tables.
Users should be able to associate default values with Delta Lake columns at table creation time or thereafter.

Support for column defaults is a key requirement to facilitate updating the table schema over time and performing DML operations on wide tables with sparse data.

Please refer to an open design doc here.

How was this patch tested?

N/A

Does this PR introduce any user-facing changes?

No, this is just a protocol change.

[dtenedor]

cc @tdas

[dtenedor]

Thanks @felipepessoto for your review, responded to your comments, please take another look!

[felipepessoto]

Any engine that assigns this value can use its own SQL dialect of choice to represent the expression as a string, and use that same dialect to evaluate that expression later for future writes. If one engine writes the string metadata using its own SQL dialect and another engine then consumes it later when performing writes, the results are undefined.

For other features, like invariant and check constraint, we only say SQL expression, I assume it would be standard SQL. Accepting any SQL dialect seems dangerous to me as it would make the table dependent on a specific engine.

[dtenedor]

OK, I updated this to just say "SQL expression" like we do for the "Generated Columns" section above.

[felipepessoto]

@dtenedor, at #2238 you mentioned this https://issues.apache.org/jira/browse/SPARK-38334. Does it mean we are following the same behavior?

It says when you add a new column, the existing rows are updated with the default value, I think we need to describe this behavior in the protocol:

CREATE TABLE T(a INT, b INT NOT NULL);

-- The default default is NULL
INSERT INTO T VALUES (DEFAULT, 0);
INSERT INTO T(b)  VALUES (1);
SELECT * FROM T;
(NULL, 0)
(NULL, 1)

-- Adding a default to a table with rows, sets the values for the
-- existing rows (exist default) and new rows (current default).
ALTER TABLE T ADD COLUMN c INT DEFAULT 5;
INSERT INTO T VALUES (1, 2, DEFAULT);
SELECT * FROM T;
(NULL, 0, 5)
(NULL, 1, 5)
(1, 2, 5) 

[dtenedor]

It says when you add a new column, the existing rows are updated with the default value, I think we need to describe this behavior in the protocol.

@felipepessoto good point, we are not including this part for the Delta Lake data source type in Spark because it adds a lot of complexity on the reader path. We will follow the rest of the Spark specification for the writer path (e.g. for future DML commands like INSERT and UPDATE) since the implementation is much simpler.

[felipepessoto]

Got it. I think we need to explicitly mention that, as most of users will assume the same behavior as Spark.

[felipepessoto]

Got it. I think we need to explicitly mention that, as most of users will assume the same behavior as Spark.

@dtenedor. Can we do this in a follow up PR? I think this is important and can cause confusion to library authors.

[dtenedor]

@dtenedor. Can we do this in a follow up PR? I think this is important and can cause confusion to library authors.

@felipepessoto yes, sorry I missed your latest comment. I will prepare a follow-up PR to fix this.

[dtenedor]

@felipepessoto here's the requested update with respect to read operations: #2266