refactor: logical plans in writer

[ion-elgreco]

Description

We were still using physical plans as starting point, we now only take in logical plans and pass them around. This solves also issues with physical type coercion, since logical type coercion is more flexible.

This is a breaking change though since folks who use .with_execution_plan now how to provide a logical plan instead of a physical plan.

Related Issue(s)

• close ValueError: Invalid comparison operation: Utf8 == LargeUtf8 on write_deltalake in overwrite-mode #3139

[codecov]

Codecov Report

Attention: Patch coverage is 61.19403% with 26 lines in your changes missing coverage. Please review.

Project coverage is 71.92%. Comparing base (a73d646) to head (ee6a91b).
Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/core/src/operations/write.rs	61.19%	15 Missing and 11 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3141      +/-   ##
==========================================
+ Coverage   71.88%   71.92%   +0.03%     
==========================================
  Files         134      134              
  Lines       43479    43431      -48     
  Branches    43479    43431      -48     
==========================================
- Hits        31257    31236      -21     
+ Misses      10201    10165      -36     
- Partials     2021     2030       +9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[roeap]

Love us adopting logical plans more and more!!

Will give this a more thorough read later on, just looking at the changes alone w/o context - do we want to pass around a DataFrame or could we just use the LogicalPlan directly without the attached session?

[ion-elgreco]

@roeap the way I was thinking about it, internally we can use dataframes since we use them directly with DF API, and the input for the builder would just be a plan.

I was thinking yesterday to also make the schema evolution into a custom logical plan node. Any thoughts on this?

[hntd187]

@roeap the way I was thinking about it, internally we can use dataframes since we use them directly with DF API, and the input for the builder would just be a plan.

I was thinking yesterday to also make the schema evolution into a custom logical plan node. Any thoughts on this?

Most everything should become a logical plan node at some point. I know that's much easier said than done, but still.