Unable to read timestamp fields from column statistics

[cmackenzie1]

Environment

Delta-rs version: 0.10.0

Binding: rust

Environment:
N/A

---

Bug

What happened:

With a field defined as

SchemaField::new(
  "EdgeEndTimestamp".to_string(),
  SchemaDataType::primitive("timestamp".to_string()),
  true,
  HashMap::new(),
),

I get the following error when querying using datafusion.

{
  "timestamp": "2023-05-16T18:58:40.206275Z",
  "level": "WARN",
  "fields": {
    "message": "Unexpected type when parsing min/max values for EdgeEndTimestamp. Found 2023-05-16 18:56:10 +00:00",
    "log.target": "deltalake::action::parquet_read",
    "log.module_path": "deltalake::action::parquet_read",
    "log.file": "/root/.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/deltalake-0.10.0/src/action/parquet_read/mod.rs",
    "log.line": 216
  },
  "target": "deltalake::action::parquet_read"
}

What you expected to happen:

Query to succeed, statistics read correctly.

How to reproduce it:

1. define a timestamp field
2. write data
3. query back data

More details:

It looks like the format might have changed for timestamp fields? It no longer looks like RFC3339. I have some data written using this (maybe incorrect) format. Would be great if the fix also supports this format too

[cmackenzie1]

Error is happening in this chunk of code when reading .checkpoint.parquet files with timestamps.

delta-rs/rust/src/action/parquet_read/mod.rs

Lines 213 to 223 in 8a4b2b8

	if let Ok(val) = primitive_parquet_field_to_json_value(field) {
	Some(ColumnValueStat::Value(val))
	} else {
	log::warn!(
	"Unexpected type when parsing min/max values for {}. Found {}",
	field_name,
	field
	);
	None
	}
	}

Using parquet-tools, the field is encoded in the parquet as such:

"Tag": "name=EdgeEndTimestamp, type=INT64, convertedtype=TIMESTAMP_MICROS, repetitiontype=OPTIONAL"

Looking at

delta-rs/rust/src/action/parquet_read/mod.rs

Lines 247 to 270 in 8a4b2b8

	fn primitive_parquet_field_to_json_value(field: &Field) -> Result<serde_json::Value, &'static str> {
	match field {
	Field::Bool(value) => Ok(json!(value)),
	Field::Byte(value) => Ok(json!(value)),
	Field::Short(value) => Ok(json!(value)),
	Field::Int(value) => Ok(json!(value)),
	Field::Long(value) => Ok(json!(value)),
	Field::Float(value) => Ok(json!(value)),
	Field::Double(value) => Ok(json!(value)),
	Field::Str(value) => Ok(json!(value)),
	Field::Decimal(decimal) => match BigInt::from_signed_bytes_be(decimal.data()).to_f64() {
	Some(int) => Ok(json!(
	int / (10_i64.pow((decimal.scale()).try_into().unwrap()) as f64)
	)),
	_ => Err("Invalid type for min/max values."),
	},
	Field::TimestampMillis(timestamp) => Ok(serde_json::Value::String(
	convert_timestamp_millis_to_string(*timestamp)?,
	)),
	Field::Date(date) => Ok(serde_json::Value::String(convert_date_to_string(*date)?)),
	_ => Err("Invalid type for min/max values."),
	}
	}

I see there is no match for Field::TimestampMicros - could that be the issue?