Skip to content

Commit

Permalink
refactor: use pa.table.cast in delta_arrow_schema_from_pandas (delta-…
Browse files Browse the repository at this point in the history
…io#1573)

# Description
I noticed that `pa.Table.from_pandas` was called twice even after the
schema was constructed. Seems more efficient to just cast the existing
that data has been read into memory with the newly created schema.

Co-authored-by: Will Jones <willjones127@gmail.com>
  • Loading branch information
2 people authored and polynomialherder committed Aug 15, 2023
1 parent a2b6479 commit 9f17a19
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions python/deltalake/schema.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,5 +43,5 @@ def delta_arrow_schema_from_pandas(
else:
schema_out.append(field)
schema = pa.schema(schema_out, metadata=schema.metadata)
data = pa.Table.from_pandas(data, schema=schema)
return data, schema
table = table.cast(target_schema=schema)
return table, schema

0 comments on commit 9f17a19

Please sign in to comment.