Skip to content

Commit

Permalink
refactor: use pa.table.cast in delta_arrow_schema_from_pandas (#1573)
Browse files Browse the repository at this point in the history
# Description
I noticed that `pa.Table.from_pandas` was called twice even after the
schema was constructed. Seems more efficient to just cast the existing
that data has been read into memory with the newly created schema.

Co-authored-by: Will Jones <willjones127@gmail.com>
  • Loading branch information
ion-elgreco and wjones127 authored Aug 6, 2023
1 parent 9cb5b15 commit b29766a
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions python/deltalake/schema.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,5 +43,5 @@ def delta_arrow_schema_from_pandas(
else:
schema_out.append(field)
schema = pa.schema(schema_out, metadata=schema.metadata)
data = pa.Table.from_pandas(data, schema=schema)
return data, schema
table = table.cast(target_schema=schema)
return table, schema

0 comments on commit b29766a

Please sign in to comment.