Improve BQ typed support #5529

RustedBones · 2024-11-27T10:06:57Z

Leverages upstream changes available for BQ:
Annotated BQ typed avro translation leverages logical-type to have symmetric read/write. This fixes integration testfailure introduced in #5523

shnapz · 2024-11-27T14:59:32Z

...google-cloud-platform/src/main/scala/com/spotify/scio/bigquery/types/ConverterProvider.scala

        case t if t =:= typeOf[LocalTime] =>
-          q"_root_.com.spotify.scio.bigquery.Time($tree)"
+          q"_root_.com.spotify.scio.bigquery.Time.micros($tree)"


so it was returning String before?

Indeed, by changing this, I created a behavior change.

When extracting from avro, we disabled the logical types -> those types are returned as STRING (mentioned here). This is very annoying because the same record can't be written back in the table.

Setting useAvroLogicalTypes on the IO enables that. This will however be breaking for all pipelines reading BQ DATE, TIME and DATETIME that expects string and will get logical type.

This is also not in sync with storage API that uses logical types for such columns.

Updated the PR to keep the same behavior.
Introduced new Format.GenericRecordWithLogicalType to enable BQ avro read with desired setup.
Change BigQueryTyped.Table to always use logical types

RustedBones · 2024-11-28T09:54:58Z

...oogle-cloud-platform/src/main/scala/com/spotify/scio/bigquery/syntax/SCollectionSyntax.scala

@@ -203,7 +203,7 @@ final class SCollectionGenericRecordOps[T <: GenericRecord](private val self: SC
    self
      .covary[GenericRecord]
      .write(
-        BigQueryTypedTable(table, Format.GenericRecord)(
+        BigQueryTypedTable(table, Format.GenericRecordWithLogicalTypes)(


On write, logical types were enabled by default

This is aligned with the BQ storage API as alluded to in another comment?

No, I'm just keeping old behavior from here

RustedBones · 2024-11-28T09:57:51Z

scio-google-cloud-platform/src/main/scala/com/spotify/scio/bigquery/types/TypeProvider.scala

@@ -334,7 +334,7 @@ private[types] object TypeProvider {
          q"override def schema: ${p(c, GModel)}.TableSchema = ${p(c, SUtil)}.parseSchema(${schema.toString})"
        }
        val defAvroSchema =
-          q"override def avroSchema: org.apache.avro.Schema = ${p(c, BigQueryUtils)}.toGenericAvroSchema(${cName.toString}, this.schema.getFields)"
+          q"override def avroSchema: org.apache.avro.Schema = ${p(c, BigQueryUtils)}.toGenericAvroSchema(this.schema, true)"


Use logical types by default for typed BQ avro

RustedBones · 2024-11-28T10:01:45Z

scio-google-cloud-platform/src/main/scala/com/spotify/scio/bigquery/BigQueryIO.scala

+      val schemaFactory = Functions.serializableFn[TableSchema, org.apache.avro.Schema] { _ =>
+        BigQueryType[T].avroSchema
+      }


use strict schema from the annotated class instead of relying on the table conversion

codecov · 2024-11-29T13:03:58Z

Codecov Report

Attention: Patch coverage is 54.28571% with 16 lines in your changes missing coverage. Please review.

Project coverage is 61.25%. Comparing base (db7b14e) to head (f64ac98).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...n/scala/com/spotify/scio/bigquery/BigQueryIO.scala	36.36%	14 Missing ⚠️
...la/com/spotify/scio/bigquery/client/TableOps.scala	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5529      +/-   ##
==========================================
- Coverage   61.31%   61.25%   -0.07%     
==========================================
  Files         314      314              
  Lines       11249    11262      +13     
  Branches      770      796      +26     
==========================================
+ Hits         6897     6898       +1     
- Misses       4352     4364      +12

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

kellen · 2025-01-23T11:53:21Z

scio-google-cloud-platform/src/main/scala/com/spotify/scio/bigquery/BigQueryTypes.scala

@@ -250,6 +257,9 @@ object DateTime {
  /** Convert BigQuery `DATETIME` string to `LocalDateTime`. */
  def parse(datetime: String): LocalDateTime =
    Parser.parseLocalDateTime(datetime)
+
+  //  For BigQueryType macros only, do not use directly
+  def format(datetime: LocalDateTime): String = apply(datetime)


Can't be private[scio] for some reason?

unfortunatelly no. This is used by code generated in macros, meaning it can be referenced from any package.

kellen · 2025-01-23T11:56:34Z

integration/src/test/scala/com/spotify/scio/bigquery/TypedBigQueryIT.scala

@@ -48,7 +48,8 @@ object TypedBigQueryIT {
    timestamp: Instant,
    date: LocalDate,
    time: LocalTime,
-    datetime: LocalDateTime,
+    // BQ DATETIME is problematic with avro: export as 'string(datetime)', load as '(long)local-timestamp-micros'


Please make this comment slightly more verbose

kellen · 2025-01-23T12:01:16Z

...oogle-cloud-platform/src/main/scala/com/spotify/scio/bigquery/syntax/SCollectionSyntax.scala

@@ -203,7 +203,7 @@ final class SCollectionGenericRecordOps[T <: GenericRecord](private val self: SC
    self
      .covary[GenericRecord]
      .write(
-        BigQueryTypedTable(table, Format.GenericRecord)(
+        BigQueryTypedTable(table, Format.GenericRecordWithLogicalTypes)(


This is aligned with the BQ storage API as alluded to in another comment?

Base automatically changed from beam-2.61 to main November 27, 2024 14:55

RustedBones marked this pull request as ready for review November 27, 2024 14:56

shnapz reviewed Nov 27, 2024

View reviewed changes

RustedBones force-pushed the fix-typed-bq-avro branch 2 times, most recently from 35a1b7e to 158c9bf Compare November 28, 2024 09:53

RustedBones commented Nov 28, 2024

View reviewed changes

RustedBones force-pushed the fix-typed-bq-avro branch from 158c9bf to 22908a8 Compare November 28, 2024 10:09

RustedBones mentioned this pull request Dec 17, 2024

BigQuery JSON column: encode as Jackson JsonNode #5523

Merged

RustedBones added 8 commits January 23, 2025 11:19

Improve BQ typed support

be1ef2f

Stable behavior

8a1bd1f

Fix toPrettyString

9e1713e

Descriptive test class names

9c148b3

Add required avro schema factory

2a1fdf8

Replace closure cleaner with Serializable function

43624b6

Add more test cases

263ee77

Remove useless wait

aadb8fb

RustedBones force-pushed the fix-typed-bq-avro branch from 6da6ac2 to aadb8fb Compare January 23, 2025 10:22

kellen approved these changes Jan 23, 2025

View reviewed changes

More detailed comment on disabled DATETIME column

f64ac98

RustedBones mentioned this pull request Jan 27, 2025

BigQuery load fails with JSON array #5542

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve BQ typed support #5529

Improve BQ typed support #5529

RustedBones commented Nov 27, 2024

shnapz Nov 27, 2024

RustedBones Nov 27, 2024

RustedBones Nov 28, 2024

RustedBones Nov 28, 2024

kellen Jan 23, 2025

RustedBones Jan 23, 2025

RustedBones Nov 28, 2024

RustedBones Nov 28, 2024

codecov bot commented Nov 29, 2024 •

edited

Loading

kellen Jan 23, 2025

RustedBones Jan 23, 2025

kellen Jan 23, 2025

kellen Jan 23, 2025

Improve BQ typed support #5529

Are you sure you want to change the base?

Improve BQ typed support #5529

Conversation

RustedBones commented Nov 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Nov 29, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Nov 29, 2024 •

edited

Loading