Time unit mismatch issue when calling Bigquery Push API

[phil-park]

Expected Behavior

When data is ingested with the push api, the same time series data must be added to the online store and the offline store (bigquery).

Current Behavior

When calling the Push API with the BigQuery Offline Store, if the timestamp field exists, the pyarrow timestamp unit in the code defaults to us (microsecond).

However, when bigquery load_table is executed, the timestamp column operates in ns (nano second), so the unit is inconsistent.

Because there is a time difference of 1000 times, bigquery actually inputs 1/1000 times the time, so if you put the current time, the old time of the 1970s is entered.

Steps to reproduce

After creating Bigquery Push Source with timestamp field, call Push API (w/ONLINE_AND_OFFLINE).

Specifications

• Version:
• Platform:
• Subsystem: BigQuery

Possible Solution

When converting to pyarrow schema, the time unit is input according to the offline store.