fix: add support for microsoft onelake #1564
Conversation
|
ACTION NEEDED delta-rs follows the Conventional Commits specification for release automation. The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. |
vmuddassir-msft
changed the title
|
Made code change to test utils to handle abfss (abfss://@onelake.dfs.fabric.microsoft.com) Issue #1418 |
vmuddassir-msft
changed the title
vmuddassir-msft
changed the title
vmuddassir-msft
changed the title
Co-authored-by: Christopher Watford <132389385+watfordkcf@users.noreply.github.com>
Co-authored-by: Christopher Watford <132389385+watfordkcf@users.noreply.github.com>
Co-authored-by: Christopher Watford <132389385+watfordkcf@users.noreply.github.com>
Co-authored-by: Christopher Watford <132389385+watfordkcf@users.noreply.github.com>
|
@vmuddassir-msft are you getting to a point to where you're ready to put this into Review? |
Delta-rs needs to be upgraded to the latest version of object_store v0.7.0 , in order to add Support for Microsoft Fabric / OneLake . I have upgraded the versions for object_store with other dependencies which reference object_store v0.6.1 I am facing an issue after the upgrade : This is because datafusion seems to be internally using the dependency crate arrow-array v45 while the latest version is arrow-array = { version = "46.0.0" } 
|
vmuddassir-msft
requested review from
wjones127,
fvaleye,
roeap,
rtyler and
mosyp
as code owners
Yes, we need to wait for datafusion to upgrade versions of arrow before we can. Can you upgrade object store independent of arrow? |
Unable to upgrade Object store independently as this is also a dependency for parquet and the latest version of paraquet v46 has linked dependencies to the latest versions of all arrow related crates which includes arrow-array
|
|
@vmuddassir-msft understood. The next version of datafusion is out for vote, so it should be released sometime next week. Once that happens we should be able to get this PR in a merge-able state. :) |
rust/src/writer/record_batch.rs
Outdated
| @@ -372,7 +372,8 @@ pub(crate) fn divide_by_partition_values( | |||
| }) | |||
| .collect::<Result<Vec<_>, DeltaWriterError>>()?; | |||
|
|
|||
| let partition_ranges = lexicographical_partition_ranges(sorted_partition_columns.as_slice())?; | |||
There was a problem hiding this comment.
#[allow(warnings)] for now as :lexicographical_partition_ranges is Deprecated
There was a problem hiding this comment.
@wjones127 / @rtyler - Datafusion packages have been upgraded
There was a problem hiding this comment.
FYI another dedicated PR was merged to upgrade datafusion, so you'll need to address conflicts with that. Should be quick.
@rtyler - PR is ready to be reviewed The test for onelake is failing as this requires a bearer token , test can be marked as ignored 
|
|
@vmuddassir-msft is your expectation that the test will never be run? (in which case we should just delete it), or do you want to configure a bearer token in GitHub secrets so that at least CI can run? |
Head branch was pushed to by a user without write access
|
Closing this in favor of the rebased #1642 |
No description provided.