Automatic type widening in MERGE

[johanl-db]

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Description

This change is part of the type widening table feature.
Type widening feature request: #2622
Type Widening protocol RFC: #2624

It adds automatic type widening as part of schema evolution in MERGE INTO:

• During resolution of the DeltaMergeInto plan, when merging the target and source schema to compute the schema after evolution, we keep the wider source type when type widening is enabled on the table.
• When updating the table schema at the beginning of MERGE execution, metadata is added to the schema to record type changes.

How was this patch tested?

• A new test suite DeltaTypeWideningSchemaEvolutionSuite is added to cover type evolution in MERGE

This PR introduces the following user-facing changes

The table feature is available in testing only, there are no user-facing changes as of now.

When automatic schema evolution is enabled in MERGE and the source schema contains a type that is wider than the target schema:

With type widening disabled: the type in the target schema is not changed. the ingestion behavior follows the storeAssignmentPolicy configuration:

• LEGACY: source values that overflow the target type are stored as null
• ANSI: a runtime check is injected to fail on source values that overflow the target type.
• STRICT: the MERGE operation fails during analysis.

With type widening enabled: the type in the target schema is updated to the wider source type. The MERGE operation always succeeds:

-- target: key int, value short
-- source: key int, value int
MERGE INTO target
USING source
ON target.key = source.key
WHEN MATCHED THEN UPDATE SET *

After the MERGE operation, the target schema is key int, value int.

[sabir-akhadov]

left some comments.

[sabir-akhadov]

lgtm

[tomvanbussel]

Nice! This is going to be really powerful when combined with the already existing schema auto migration! I have two questions and a small nit, but otherwise it LGTM.

[tomvanbussel]

What will happen if type widening is disabled after this statement, but before the main transaction starts?

[johanl-db]

The two calls to SchemaMergingUtils.mergeSchemas (here and in ImplicitMetadataOperation) will return a different schema: first one with the wider type, second one with the original type.

This works currently because the second call to mergeSchemas doesn't allow implicit casts and will fail but it's quite brittle. I added a check at the start of MERGE to properly fail when type widening is enabled/disabled concurrently

[tdas]

This is awesome.