[Bug]: Cannot find a matching Calcite SqlTypeName for Beam type: LOGICAL_TYPE seen in 2.44.0 SNAPSHOT

[Abacn]

What happened?

There is a report of seeing following exception in current Beam 2.44.0 SNAPSHOT. It workes in 2.43.0. Possibly #23891 and/or #24013 break Beam SQL.

WARNING: No NameResolverProviders found via ServiceLoader, including for DNS. This is probably due to a broken build. If using ProGuard, check your configuration
Exception in thread "main" java.lang.IllegalArgumentException: Cannot find a matching Calcite SqlTypeName for Beam type: LOGICAL_TYPE
	at org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toSqlTypeName(CalciteUtils.java:198)
	at org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toRelDataType(CalciteUtils.java:305)
	at org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toRelDataType(CalciteUtils.java:312)
	at org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.lambda$toCalciteRowType$0(CalciteUtils.java:280)
	at java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:110)
	at java.util.stream.IntPipeline$Head.forEach(IntPipeline.java:557)

Issue Priority

Priority: 1

Issue Component

Component: dsl-sql

[Abacn]

A breaking change I could imagine is that we removed SqlCharType in #24013. I thought we removed it thoroughly (and replaced with FixedString) but if it is not the case in whatever reason then CalciteUtil will throw exception there.

It is also possible that when the SDK harness container does not bump version (from

beam/runners/google-cloud-dataflow-java/build.gradle

Line 57 in 166b881

'dataflow.legacy_container_version' : 'beam-master-20220816',

, the legacy SDK container version is still in August on master branch), When the SNAPSHOT version does not have synchronized SDK harness container, I observed inconsistent state on schema parse and it caused internal PROD integration test failures. Release candidates and released versions do not have this issue.

[Abacn]

Datetime portable logical type was introduced in 2.42.0; decimal introduced in 2.43.0. Reported that the job works as expected for Beam 2.43.0 so that it is not caused by these changes. Between 2.43.0 and 2.44.0 the change is portable BINARY/VARBINARY/CHAR/VARCHAR types. The change added support to these types in org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toSqlTypeName. The only logical type not supported I could imagine is the removed CharType. #24414 should have fixed it. Close the issue for now.

[Abacn]

Update: after #24414 it is found the issue is

No NameResolverProviders found via ServiceLoader, including for DNS. This is probably due to a broken build.
If using ProGuard, check your configuration Exception in thread "main" java.lang.IllegalArgumentException:
Cannot find a matching Calcite SqlTypeName for Beam logical type: beam:logical_type:micros_instant:v1 at
org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toSqlTypeName(CalciteUtils.java:206) at
org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toRelDataType(CalciteUtils.java:319) at
org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toRelDataType(CalciteUtils.java:326) at
org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.lambda$toCalciteRowType$0(CalciteUtils.java:294)

culprit is likely #23969. Previously AvroUtils.toFieldType will reduce to its base type INT64, as it is an unsupported logical type backed by LONG. Now AvroUtils support TimestampMicros explicitly and returns beam logical type SqlTypes.TIMESTAMP (micros_instant), but throughout Beam SQL codebase SqlTypes.TIMESTAMP is not supported.

Need a partial rollback of #23969

[Abacn]

Reopen for the PR of cherry-pick into release-2.44.0 branch

[kennknowles]

Merged to 2.44. branch