-
-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
polars 0.44.0 includes pyo3
and other extra dependencies by default
#19485
Comments
PyO3 0.21 dependency also causes troubles when using rust Polars for building python packages with PyO3 0.22 |
That shouldn't happen. We must have messed up the feature flags somewhere. I can take a look, I have worked on this recently. |
Ai, yes, this needs to be patched. Can you make this high prio @stinodego? |
It does pull in the (main) ~/playground/fadd ⚡ ❯ cargo add polars
Updating crates.io index
Adding polars v0.44.0 to dependencies
Features:
+ csv
+ docs
+ dtype-date
+ dtype-datetime
+ dtype-duration
+ dtype-slim
+ fmt
+ polars-io
+ polars-time
+ temporal
+ zip_with
139 deactivated features
Updating crates.io index
Locking 317 packages to latest compatible versions
Adding atoi_simd v0.15.6 (latest: v0.16.0)
Adding brotli v6.0.0 (latest: v7.0.0)
Adding chrono-tz v0.8.6 (latest: v0.10.0)
Adding chrono-tz-build v0.2.1 (latest: v0.4.0)
Adding core-foundation v0.9.4 (latest: v0.10.0)
Adding crossterm v0.27.0 (latest: v0.28.1)
Adding fs4 v0.9.1 (latest: v0.10.0)
Adding generic-array v0.14.7 (latest: v1.1.0)
Adding hashbrown v0.14.5 (latest: v0.15.0)
Adding heck v0.4.1 (latest: v0.5.0)
Adding hermit-abi v0.3.9 (latest: v0.4.0)
Adding idna v0.5.0 (latest: v1.0.2)
Adding linux-raw-sys v0.4.14 (latest: v0.6.5)
Adding memmap2 v0.7.1 (latest: v0.9.5)
Adding object_store v0.10.2 (latest: v0.11.1)
Adding planus v0.3.1 (latest: v1.0.0)
Adding pyo3 v0.21.2 (latest: v0.22.5)
Adding pyo3-build-config v0.21.2 (latest: v0.22.5)
Adding pyo3-ffi v0.21.2 (latest: v0.22.5)
Adding pyo3-macros v0.21.2 (latest: v0.22.5)
Adding pyo3-macros-backend v0.21.2 (latest: v0.22.5)
Adding quick-xml v0.36.2 (latest: v0.37.0)
Adding security-framework v2.11.1 (latest: v3.0.0)
Adding siphasher v0.3.11 (latest: v1.0.1)
Adding snafu v0.7.5 (latest: v0.8.5)
Adding snafu-derive v0.7.5 (latest: v0.8.5)
Adding sqlparser v0.49.0 (latest: v0.51.0)
Adding syn v1.0.109 (latest: v2.0.85)
Adding sysinfo v0.31.4 (latest: v0.32.0)
Adding unicode-width v0.1.14 (latest: v0.2.0)
Adding wasi v0.11.0+wasi-snapshot-preview1 (latest: v0.13.3+wasi-0.2.2)
Adding windows v0.57.0 (latest: v0.58.0)
Adding windows-core v0.52.0 (latest: v0.58.0)
Adding windows-core v0.57.0 (latest: v0.58.0)
Adding windows-implement v0.57.0 (latest: v0.58.0)
Adding windows-interface v0.57.0 (latest: v0.58.0)
Adding windows-registry v0.2.0 (latest: v0.3.0)
Adding windows-result v0.1.2 (latest: v0.2.0)
Adding windows-strings v0.1.0 (latest: v0.2.0)
Adding windows-sys v0.52.0 (latest: v0.59.0)
Adding zerocopy v0.7.35 (latest: v0.8.7)
Adding zerocopy-derive v0.7.35 (latest: v0.8.7) (main) ~/playground/fadd took 0.1s ❯ cargo tree --all-features -vv --depth=3
fadd v0.1.0 (~/playground/fadd)
└── polars v0.44.0
├── polars-arrow v0.44.0
│ ├── ahash v0.8.11
│ │ [build-dependencies]
│ ├── atoi_simd v0.15.6
│ ├── bytemuck v1.19.0
│ ├── chrono v0.4.38
│ ├── dyn-clone v1.0.17
│ ├── either v1.13.0
│ ├── ethnum v1.5.0
│ ├── fast-float v0.2.0
│ ├── hashbrown v0.15.0
│ ├── itoa v1.0.11
│ ├── multiversion v0.7.4
│ ├── num-traits v0.2.19
│ │ [build-dependencies]
│ ├── parking_lot v0.12.3
│ ├── polars-arrow-format v0.1.0
│ ├── polars-error v0.44.0
│ ├── polars-schema v0.44.0
│ │ [build-dependencies]
│ ├── polars-utils v0.44.0
│ │ [build-dependencies]
│ ├── ryu v1.0.18
│ ├── simdutf8 v0.1.5
│ ├── streaming-iterator v0.1.9
│ ├── strength_reduce v0.2.4
│ └── strum_macros v0.26.4 (proc-macro)
│ [build-dependencies]
│ └── version_check v0.9.5
├── polars-core v0.44.0
│ ├── ahash v0.8.11 (*)
│ ├── bitflags v2.6.0
│ ├── bytemuck v1.19.0 (*)
│ ├── chrono v0.4.38 (*)
│ ├── comfy-table v7.1.1
│ ├── either v1.13.0
│ ├── hashbrown v0.14.5
│ ├── hashbrown v0.15.0 (*)
│ ├── indexmap v2.6.0
│ ├── num-traits v0.2.19 (*)
│ ├── once_cell v1.20.2
│ ├── polars-arrow v0.44.0 (*)
│ ├── polars-compute v0.44.0
│ │ [build-dependencies]
│ ├── polars-error v0.44.0 (*)
│ ├── polars-row v0.44.0
│ ├── polars-schema v0.44.0 (*)
│ ├── polars-utils v0.44.0 (*)
│ ├── rayon v1.10.0
│ ├── regex v1.11.1
│ ├── strum_macros v0.26.4 (proc-macro) (*)
│ ├── thiserror v1.0.65
│ └── xxhash-rust v0.8.12
│ [build-dependencies]
│ └── version_check v0.9.5
├── polars-error v0.44.0 (*)
├── polars-io v0.44.0
│ ├── ahash v0.8.11 (*)
│ ├── atoi_simd v0.15.6
│ ├── bytes v1.8.0
│ ├── chrono v0.4.38 (*)
│ ├── fast-float v0.2.0
│ ├── glob v0.3.1
│ ├── hashbrown v0.15.0 (*)
│ ├── home v0.5.9
│ ├── itoa v1.0.11
│ ├── memchr v2.7.4
│ ├── memmap2 v0.7.1
│ ├── num-traits v0.2.19 (*)
│ ├── once_cell v1.20.2
│ ├── percent-encoding v2.3.1
│ ├── polars-arrow v0.44.0 (*)
│ ├── polars-core v0.44.0 (*)
│ ├── polars-error v0.44.0 (*)
│ ├── polars-schema v0.44.0 (*)
│ ├── polars-time v0.44.0
│ ├── polars-utils v0.44.0 (*)
│ ├── rayon v1.10.0 (*)
│ ├── regex v1.11.1 (*)
│ ├── ryu v1.0.18
│ └── simdutf8 v0.1.5
├── polars-ops v0.44.0
│ ├── ahash v0.8.11 (*)
│ ├── argminmax v0.6.2
│ ├── bytemuck v1.19.0 (*)
│ ├── either v1.13.0
│ ├── hashbrown v0.15.0 (*)
│ ├── indexmap v2.6.0 (*)
│ ├── memchr v2.7.4
│ ├── num-traits v0.2.19 (*)
│ ├── polars-arrow v0.44.0 (*)
│ ├── polars-compute v0.44.0 (*)
│ ├── polars-core v0.44.0 (*)
│ ├── polars-error v0.44.0 (*)
│ ├── polars-schema v0.44.0 (*)
│ ├── polars-utils v0.44.0 (*)
│ ├── rayon v1.10.0 (*)
│ ├── regex v1.11.1 (*)
│ ├── regex-syntax v0.8.5
│ └── strum_macros v0.26.4 (proc-macro) (*)
│ [build-dependencies]
│ └── version_check v0.9.5
├── polars-parquet v0.44.0
│ ├── ahash v0.8.11 (*)
│ ├── base64 v0.22.1
│ ├── bytemuck v1.19.0 (*)
│ ├── ethnum v1.5.0
│ ├── hashbrown v0.15.0 (*)
│ ├── num-traits v0.2.19 (*)
│ ├── polars-arrow v0.44.0 (*)
│ ├── polars-compute v0.44.0 (*)
│ ├── polars-error v0.44.0 (*)
│ ├── polars-parquet-format v0.1.0
│ ├── polars-utils v0.44.0 (*)
│ ├── simdutf8 v0.1.5
│ └── streaming-decompression v0.1.2
├── polars-time v0.44.0 (*)
└── polars-utils v0.44.0 (*)
[build-dependencies]
└── version_check v0.9.5 (main) ~\playground\fadd took 0.4s ❯ cargo r
Compiling proc-macro2 v1.0.89
Compiling unicode-ident v1.0.13
Compiling version_check v0.9.5
Compiling windows_x86_64_gnu v0.52.6
Compiling autocfg v1.4.0
Compiling crossbeam-utils v0.8.20
Compiling cfg-if v1.0.0
Compiling serde v1.0.214
Compiling shlex v1.3.0
Compiling rustversion v1.0.18
Compiling rayon-core v1.12.1
Compiling either v1.13.0
Compiling memchr v2.7.4
Compiling parking_lot_core v0.9.10
Compiling cc v1.1.31
Compiling bitflags v2.6.0
Compiling array-init-cursor v0.2.0
Compiling thiserror v1.0.65
Compiling ahash v0.8.11
Compiling regex-syntax v0.8.5
Compiling allocator-api2 v0.2.18
Compiling num-traits v0.2.19
Compiling aho-corasick v1.1.3
Compiling lock_api v0.4.12
Compiling libc v0.2.161
Compiling windows-targets v0.52.6
Compiling psm v0.1.23
Compiling stacker v0.1.17
Compiling planus v0.3.1
Compiling quote v1.0.37
Compiling crossbeam-epoch v0.9.18
Compiling getrandom v0.2.15
Compiling regex-automata v0.4.8
Compiling smallvec v1.13.2
Compiling once_cell v1.20.2
Compiling equivalent v1.0.1
Compiling target-features v0.1.6
Compiling syn v2.0.85
Compiling crossbeam-deque v0.8.5
Compiling syn v1.0.109
Compiling foldhash v0.1.3
Compiling zerocopy v0.7.35
Compiling scopeguard v1.2.0
Compiling castaway v0.2.3
Compiling windows-sys v0.59.0
Compiling polars-utils v0.44.0
Compiling rayon v1.10.0
Compiling itoa v1.0.11
Compiling simdutf8 v0.1.5
Compiling ryu v1.0.18
Compiling static_assertions v1.1.0
Compiling parking_lot v0.12.3
Compiling raw-cpuid v11.2.0
Compiling polars-schema v0.44.0
Compiling memmap2 v0.7.1
Compiling winapi-x86_64-pc-windows-gnu v0.4.0
Compiling heck v0.5.0
Compiling bytes v1.8.0
Compiling chrono v0.4.38
Compiling polars-arrow v0.44.0
Compiling winapi v0.3.9
Compiling fast-float v0.2.0
Compiling regex v1.11.1
Compiling ethnum v1.5.0
Compiling dyn-clone v1.0.17
Compiling streaming-iterator v0.1.9
Compiling atoi_simd v0.15.6
Compiling strength_reduce v0.2.4
Compiling polars-compute v0.44.0
Compiling polars-core v0.44.0
Compiling unicode-width v0.1.14
Compiling strum v0.26.3
Compiling polars-ops v0.44.0
Compiling xxhash-rust v0.8.12
Compiling argminmax v0.6.2
Compiling now v0.1.3
Compiling atoi v2.0.0
Compiling windows-sys v0.52.0
Compiling fallible-streaming-iterator v0.1.9
Compiling serde_derive v1.0.214
Compiling thiserror-impl v1.0.65
Compiling bytemuck_derive v1.8.0
Compiling strum_macros v0.26.4
Compiling streaming-decompression v0.1.2
Compiling polars v0.44.0
Compiling percent-encoding v2.3.1
Compiling base64 v0.22.1
Compiling polars-parquet-format v0.1.0
Compiling glob v0.3.1
Compiling home v0.5.9
Compiling bytemuck v1.19.0
Compiling crossterm_winapi v0.9.1
Compiling crossterm v0.27.0
Compiling multiversion-macros v0.7.4
Compiling comfy-table v7.1.1
Compiling multiversion v0.7.4
Compiling polars-arrow-format v0.1.0
Compiling hashbrown v0.15.0
Compiling compact_str v0.8.0
Compiling hashbrown v0.14.5
Compiling indexmap v2.6.0
Compiling polars-error v0.44.0
Compiling polars-row v0.44.0
Compiling polars-parquet v0.44.0
Compiling polars-time v0.44.0
Compiling polars-io v0.44.0
Compiling fadd v0.1.0 (C:\Users\Rahul\playground\fadd)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 49s
Running `target\debug\fadd.exe` My (main) ~\playground\fadd took 0.4s ❯ cat .\Cargo.toml
[package]
name = "fadd"
version = "0.1.0"
edition = "2021"
[dependencies]
polars = "0.44.0" I noticed that the I can confirm that this issue occurs when adding other packages too. For example, when I tried to add |
It seems to be a modification in the registry, as indicated by this. Starting with commit a31184b, |
Ok, but this is actually intended. There is a new crate So when building |
It seems that sometimes it doesn't work unless it links to Python eitsupi/neo-r-polars#25 |
Incidentally, I believe this change occurred fairly recently (between py-1.10.0 and py-1.12.0). |
It is possible that the polars-stream dependency has been broken for some time, and for some reason it no longer works without python when some feature flags are enabled. polars/crates/polars-stream/Cargo.toml Lines 30 to 33 in 00b4d61
|
In support of what @eitsupi said, the simplest repro example I can come up with is an empty Rust project that includes Polars v0.44.0, and the Interestingly, the dependencies in https://github.com/pola-rs/polars/blame/main/crates/polars-lazy/Cargo.toml |
Yes @stinodego, the dependencies are not included in the build or dependency tree, but are instead synchronized with a local index. |
I'm at fault here, the offending lines are the following in # TODO: feature gate
polars-mem-engine = { workspace = true, features = ["parquet", "csv", "json", "ipc", "cloud", "python", "dtype-categorical", "dtype-i8", "dtype-i16", "dtype-u8", "dtype-u16", "dtype-decimal", "dtype-struct", "object"] }
polars-parquet = { workspace = true }
polars-plan = { workspace = true, features = ["parquet", "csv", "json", "ipc", "cloud", "python", "serde", "dtype-categorical", "dtype-i8", "dtype-i16", "dtype-u8", "dtype-u16", "dtype-decimal", "dtype-struct", "object"] } This unconditionally turns on Python (and a bunch of other flags). EDIT: actually it appears these lines were added by @nameexhaustion . I didn't catch it in review though, so you can still blame me :) |
I can reproduce this, thanks. |
Polars-stream features in lazy should also be activated with a question mark: E.g. |
Hi there, I tried it out again and it still adds (main) ~\playground\gemmi took 0.1s ❯ cargo add --git https://github.com/pola-rs/polars polars
Updating git repository `https://github.com/pola-rs/polars`
Adding polars (git) to dependencies
Features:
+ csv
+ docs
+ dtype-date
+ dtype-datetime
+ dtype-duration
+ dtype-slim
+ fmt
+ polars-io
+ polars-time
+ temporal
+ zip_with
139 deactivated features
Updating git repository `https://github.com/pola-rs/polars`
Updating crates.io index
Locking 318 packages to latest compatible versions
Adding atoi_simd v0.15.6 (latest: v0.16.0)
Adding brotli v6.0.0 (latest: v7.0.0)
Adding chrono-tz v0.8.6 (latest: v0.10.0)
Adding chrono-tz-build v0.2.1 (latest: v0.4.0)
Adding core-foundation v0.9.4 (latest: v0.10.0)
Adding crossterm v0.27.0 (latest: v0.28.1)
Adding fs4 v0.9.1 (latest: v0.10.0)
Adding generic-array v0.14.7 (latest: v1.1.0)
Adding hashbrown v0.14.5 (latest: v0.15.0)
Adding heck v0.4.1 (latest: v0.5.0)
Adding hermit-abi v0.3.9 (latest: v0.4.0)
Adding idna v0.5.0 (latest: v1.0.2)
Adding linux-raw-sys v0.4.14 (latest: v0.6.5)
Adding memmap2 v0.7.1 (latest: v0.9.5)
Adding object_store v0.10.2 (latest: v0.11.1)
Adding planus v0.3.1 (latest: v1.0.0)
Adding pyo3 v0.21.2 (latest: v0.22.5)
Adding pyo3-build-config v0.21.2 (latest: v0.22.5)
Adding pyo3-ffi v0.21.2 (latest: v0.22.5)
Adding pyo3-macros v0.21.2 (latest: v0.22.5)
Adding pyo3-macros-backend v0.21.2 (latest: v0.22.5)
Adding quick-xml v0.36.2 (latest: v0.37.0)
Adding security-framework v2.11.1 (latest: v3.0.0)
Adding siphasher v0.3.11 (latest: v1.0.1)
Adding snafu v0.7.5 (latest: v0.8.5)
Adding snafu-derive v0.7.5 (latest: v0.8.5)
Adding sqlparser v0.49.0 (latest: v0.51.0)
Adding syn v1.0.109 (latest: v2.0.85)
Adding sysinfo v0.31.4 (latest: v0.32.0)
Adding unicode-width v0.1.14 (latest: v0.2.0)
Adding wasi v0.11.0+wasi-snapshot-preview1 (latest: v0.13.3+wasi-0.2.2)
Adding windows v0.57.0 (latest: v0.58.0)
Adding windows-core v0.52.0 (latest: v0.58.0)
Adding windows-core v0.57.0 (latest: v0.58.0)
Adding windows-implement v0.57.0 (latest: v0.58.0)
Adding windows-interface v0.57.0 (latest: v0.58.0)
Adding windows-registry v0.2.0 (latest: v0.3.0)
Adding windows-result v0.1.2 (latest: v0.2.0)
Adding windows-strings v0.1.0 (latest: v0.2.0)
Adding windows-sys v0.52.0 (latest: v0.59.0)
Adding zerocopy v0.7.35 (latest: v0.8.7)
Adding zerocopy-derive v0.7.35 (latest: v0.8.7)
(main) ~\playground\gemmi took 81.4s ❯ This is expected behaviour for cargo, as described here |
The real problem here was not that pyo3 was added to the Cargo.lock file, but that pyo3 was actually compiled. |
Oh, I didn't encounter that problem, that's why I view it from a different perspective. Anyhow it seemed to resolve the issue, thanks. |
I've confirmed that my problem is definitely fixed with this change. Thanks so much for the super-fast turnaround! |
Thanks @stinodego and @ritchie46! It still seems like the issue isn't fully resolved, but sounds like that's known (#19512 (comment)). |
Checks
Reproducible example
Enable the following features in Cargo.toml:
And try to compile any program.
Log output
When upgrading from 0.43.1,
cargo build
outputs:Issue description
From what I can tell, this is due to
polars-lazy
addingpolars-stream
to a few features, which enables thecloud
andpython
features.Expected behavior
Less dependencies like earlier versions of Polars
Installed versions
See above
The text was updated successfully, but these errors were encountered: