Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

polars 0.44.0 includes pyo3 and other extra dependencies by default #19485

Closed
2 tasks done
ankane opened this issue Oct 27, 2024 · 19 comments · Fixed by #19512
Closed
2 tasks done

polars 0.44.0 includes pyo3 and other extra dependencies by default #19485

ankane opened this issue Oct 27, 2024 · 19 comments · Fixed by #19512
Assignees
Labels
accepted Ready for implementation bug Something isn't working build Changes that affect the build system or external dependencies P-high Priority: high rust Related to Rust Polars

Comments

@ankane
Copy link
Contributor

ankane commented Oct 27, 2024

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

Enable the following features in Cargo.toml:

[dependencies]
polars = "0.44.0"

And try to compile any program.

Log output

When upgrading from 0.43.1, cargo build outputs:

      Adding addr2line v0.24.2
      Adding adler2 v2.0.0
      Adding alloc-no-stdlib v2.0.4
      Adding alloc-stdlib v0.2.2
      Adding arrayref v0.3.9
      Adding arrayvec v0.7.6
      Adding async-stream v0.3.6
      Adding async-stream-impl v0.3.6
      Adding async-trait v0.1.83
      Adding atomic-waker v1.1.2
      Adding backtrace v0.3.74
      Adding blake3 v1.5.4
      Adding block-buffer v0.10.4
      Adding brotli v6.0.0 (latest: v7.0.0)
      Adding brotli-decompressor v4.0.1
      Adding ciborium v0.2.2
      Adding ciborium-io v0.2.2
      Adding ciborium-ll v0.2.2
      Adding constant_time_eq v0.3.1
      Adding core-foundation v0.9.4 (latest: v0.10.0)
      Adding crc32fast v1.4.2
      Adding crunchy v0.2.2
      Adding crypto-common v0.1.6
      Adding digest v0.10.7
      Adding doc-comment v0.3.3
      Adding errno v0.3.9
      Adding flate2 v1.0.34
      Adding float-cmp v0.10.0
      Adding fnv v1.0.7
      Adding foldhash v0.1.3
      Adding form_urlencoded v1.2.1
      Adding fs4 v0.9.1 (latest: v0.10.0)
      Adding futures v0.3.31
      Adding futures-channel v0.3.31
      Adding futures-core v0.3.31
      Adding futures-executor v0.3.31
      Adding futures-io v0.3.31
      Adding futures-macro v0.3.31
      Adding futures-sink v0.3.31
      Adding futures-task v0.3.31
      Adding futures-util v0.3.31
      Adding generic-array v0.14.7 (latest: v1.1.0)
      Adding gimli v0.31.1
      Adding h2 v0.4.6
      Adding half v2.4.1
      Adding halfbrown v0.2.5
      Adding hashbrown v0.15.0
      Adding heck v0.4.1 (latest: v0.5.0)
      Adding hermit-abi v0.3.9 (latest: v0.4.0)
      Adding http v1.1.0
      Adding http-body v1.0.1
      Adding http-body-util v0.1.2
      Adding httparse v1.9.5
      Adding humantime v2.1.0
      Adding hyper v1.5.0
      Adding hyper-rustls v0.27.3
      Adding hyper-util v0.1.9
      Adding idna v0.5.0 (latest: v1.0.2)
      Adding indoc v2.0.5
      Adding ipnet v2.10.1
      Adding itertools v0.13.0
      Adding linux-raw-sys v0.4.14 (latest: v0.6.5)
      Adding md-5 v0.10.6
      Adding memoffset v0.9.1
      Adding mime v0.3.17
      Adding miniz_oxide v0.8.0
      Adding mio v1.0.2
      Adding object v0.36.5
      Adding object_store v0.10.2 (latest: v0.11.1)
      Adding openssl-probe v0.1.5
      Adding pin-project-lite v0.2.15
      Adding pin-utils v0.1.0
    Updating polars v0.43.1 -> v0.44.0
    Updating polars-arrow v0.43.1 -> v0.44.0
    Updating polars-compute v0.43.1 -> v0.44.0
    Updating polars-core v0.43.1 -> v0.44.0
    Updating polars-error v0.43.1 -> v0.44.0
    Updating polars-expr v0.43.1 -> v0.44.0
    Updating polars-io v0.43.1 -> v0.44.0
      Adding polars-json v0.44.0
    Updating polars-lazy v0.43.1 -> v0.44.0
    Updating polars-mem-engine v0.43.1 -> v0.44.0
    Updating polars-ops v0.43.1 -> v0.44.0
    Updating polars-parquet v0.43.1 -> v0.44.0
      Adding polars-parquet-format v0.1.0
    Updating polars-pipe v0.43.1 -> v0.44.0
    Updating polars-plan v0.43.1 -> v0.44.0
    Updating polars-row v0.43.1 -> v0.44.0
    Updating polars-schema v0.43.1 -> v0.44.0
    Updating polars-sql v0.43.1 -> v0.44.0
      Adding polars-stream v0.44.0
    Updating polars-time v0.43.1 -> v0.44.0
    Updating polars-utils v0.43.1 -> v0.44.0
      Adding portable-atomic v1.9.0
      Adding pyo3 v0.21.2 (latest: v0.22.5)
      Adding pyo3-build-config v0.21.2 (latest: v0.22.5)
      Adding pyo3-ffi v0.21.2 (latest: v0.22.5)
      Adding pyo3-macros v0.21.2 (latest: v0.22.5)
      Adding pyo3-macros-backend v0.21.2 (latest: v0.22.5)
      Adding quick-xml v0.36.2
      Adding quinn v0.11.5
      Adding quinn-proto v0.11.8
      Adding quinn-udp v0.5.5
      Adding ref-cast v1.0.23
      Adding ref-cast-impl v1.0.23
      Adding reqwest v0.12.8
      Adding ring v0.17.8
      Adding rustc-demangle v0.1.24
      Adding rustc-hash v2.0.0
      Adding rustix v0.38.37 (latest: v0.38.38)
      Adding rustls v0.23.15
      Adding rustls-native-certs v0.8.0
      Adding rustls-pemfile v2.2.0
      Adding rustls-pki-types v1.10.0
      Adding rustls-webpki v0.102.8
      Adding same-file v1.0.6
      Adding schannel v0.1.26
      Adding security-framework v2.11.1 (latest: v3.0.0)
      Adding security-framework-sys v2.12.0
      Adding serde_urlencoded v0.7.1
      Adding simd-json v0.14.2
      Adding slab v0.4.9
      Adding slotmap v1.0.7
      Adding snafu v0.7.5 (latest: v0.8.5)
      Adding snafu-derive v0.7.5 (latest: v0.8.5)
      Adding snap v1.1.1
      Adding socket2 v0.5.7
      Adding spin v0.9.8
      Adding subtle v2.6.1
      Adding sync_wrapper v1.0.1
      Adding target-lexicon v0.12.16
      Adding tinyvec v1.8.0
      Adding tinyvec_macros v0.1.1
      Adding tokio v1.41.0
      Adding tokio-macros v2.4.0
      Adding tokio-rustls v0.26.0
      Adding tokio-util v0.7.12
      Adding tower-service v0.3.3
      Adding tracing v0.1.40
      Adding tracing-attributes v0.1.27
      Adding tracing-core v0.1.32
      Adding try-lock v0.2.5
      Adding typenum v1.17.0
      Adding unicode-bidi v0.3.17
      Adding unicode-normalization v0.1.24
      Adding unindent v0.2.3
      Adding untrusted v0.9.0
      Adding url v2.5.2
      Adding value-trait v0.10.1
      Adding walkdir v2.5.0
      Adding want v0.3.1
      Adding wasm-bindgen-futures v0.4.43 (latest: v0.4.45)
      Adding wasm-streams v0.4.1 (latest: v0.4.2)
      Adding web-sys v0.3.70 (latest: v0.3.72)
      Adding winapi-util v0.1.9
      Adding windows-registry v0.2.0 (latest: v0.3.0)
      Adding windows-result v0.2.0
      Adding windows-strings v0.1.0 (latest: v0.2.0)
      Adding zeroize v1.8.1

Issue description

From what I can tell, this is due to polars-lazy adding polars-stream to a few features, which enables the cloud and python features.

Expected behavior

Less dependencies like earlier versions of Polars

Installed versions

See above

@ankane ankane added bug Something isn't working needs triage Awaiting prioritization by a maintainer rust Related to Rust Polars labels Oct 27, 2024
@chingiztob
Copy link

chingiztob commented Oct 27, 2024

PyO3 0.21 dependency also causes troubles when using rust Polars for building python packages with PyO3 0.22

@stinodego stinodego added the build Changes that affect the build system or external dependencies label Oct 28, 2024
@stinodego
Copy link
Member

That shouldn't happen. We must have messed up the feature flags somewhere. I can take a look, I have worked on this recently.

@stinodego stinodego self-assigned this Oct 28, 2024
@ritchie46
Copy link
Member

Ai, yes, this needs to be patched. Can you make this high prio @stinodego?

@rawhuul
Copy link

rawhuul commented Oct 28, 2024

It does pull in the pyo3 dependency, but it isn't included in the tree or compiled. Nevertheless, it indicates that the dependency has been added when running cargo add polars

(main) ~/playground/fadd ⚡ ❯ cargo add polars
    Updating crates.io index
      Adding polars v0.44.0 to dependencies
             Features:
             + csv
             + docs
             + dtype-date
             + dtype-datetime
             + dtype-duration
             + dtype-slim
             + fmt
             + polars-io
             + polars-time
             + temporal
             + zip_with
             139 deactivated features
    Updating crates.io index
     Locking 317 packages to latest compatible versions
      Adding atoi_simd v0.15.6 (latest: v0.16.0)
      Adding brotli v6.0.0 (latest: v7.0.0)
      Adding chrono-tz v0.8.6 (latest: v0.10.0)
      Adding chrono-tz-build v0.2.1 (latest: v0.4.0)
      Adding core-foundation v0.9.4 (latest: v0.10.0)
      Adding crossterm v0.27.0 (latest: v0.28.1)
      Adding fs4 v0.9.1 (latest: v0.10.0)
      Adding generic-array v0.14.7 (latest: v1.1.0)
      Adding hashbrown v0.14.5 (latest: v0.15.0)
      Adding heck v0.4.1 (latest: v0.5.0)
      Adding hermit-abi v0.3.9 (latest: v0.4.0)
      Adding idna v0.5.0 (latest: v1.0.2)
      Adding linux-raw-sys v0.4.14 (latest: v0.6.5)
      Adding memmap2 v0.7.1 (latest: v0.9.5)
      Adding object_store v0.10.2 (latest: v0.11.1)
      Adding planus v0.3.1 (latest: v1.0.0)
      Adding pyo3 v0.21.2 (latest: v0.22.5)
      Adding pyo3-build-config v0.21.2 (latest: v0.22.5)
      Adding pyo3-ffi v0.21.2 (latest: v0.22.5)
      Adding pyo3-macros v0.21.2 (latest: v0.22.5)
      Adding pyo3-macros-backend v0.21.2 (latest: v0.22.5)
      Adding quick-xml v0.36.2 (latest: v0.37.0)
      Adding security-framework v2.11.1 (latest: v3.0.0)
      Adding siphasher v0.3.11 (latest: v1.0.1)
      Adding snafu v0.7.5 (latest: v0.8.5)
      Adding snafu-derive v0.7.5 (latest: v0.8.5)
      Adding sqlparser v0.49.0 (latest: v0.51.0)
      Adding syn v1.0.109 (latest: v2.0.85)
      Adding sysinfo v0.31.4 (latest: v0.32.0)
      Adding unicode-width v0.1.14 (latest: v0.2.0)
      Adding wasi v0.11.0+wasi-snapshot-preview1 (latest: v0.13.3+wasi-0.2.2)
      Adding windows v0.57.0 (latest: v0.58.0)
      Adding windows-core v0.52.0 (latest: v0.58.0)
      Adding windows-core v0.57.0 (latest: v0.58.0)
      Adding windows-implement v0.57.0 (latest: v0.58.0)
      Adding windows-interface v0.57.0 (latest: v0.58.0)
      Adding windows-registry v0.2.0 (latest: v0.3.0)
      Adding windows-result v0.1.2 (latest: v0.2.0)
      Adding windows-strings v0.1.0 (latest: v0.2.0)
      Adding windows-sys v0.52.0 (latest: v0.59.0)
      Adding zerocopy v0.7.35 (latest: v0.8.7)
      Adding zerocopy-derive v0.7.35 (latest: v0.8.7)
(main) ~/playground/fadd took 0.1s ❯ cargo tree --all-features -vv --depth=3

fadd v0.1.0 (~/playground/fadd)
└── polars v0.44.0
    ├── polars-arrow v0.44.0
    │   ├── ahash v0.8.11
    │   │   [build-dependencies]
    │   ├── atoi_simd v0.15.6
    │   ├── bytemuck v1.19.0
    │   ├── chrono v0.4.38
    │   ├── dyn-clone v1.0.17
    │   ├── either v1.13.0
    │   ├── ethnum v1.5.0
    │   ├── fast-float v0.2.0
    │   ├── hashbrown v0.15.0
    │   ├── itoa v1.0.11
    │   ├── multiversion v0.7.4
    │   ├── num-traits v0.2.19
    │   │   [build-dependencies]
    │   ├── parking_lot v0.12.3
    │   ├── polars-arrow-format v0.1.0
    │   ├── polars-error v0.44.0
    │   ├── polars-schema v0.44.0
    │   │   [build-dependencies]
    │   ├── polars-utils v0.44.0
    │   │   [build-dependencies]
    │   ├── ryu v1.0.18
    │   ├── simdutf8 v0.1.5
    │   ├── streaming-iterator v0.1.9
    │   ├── strength_reduce v0.2.4
    │   └── strum_macros v0.26.4 (proc-macro)
    │   [build-dependencies]
    │   └── version_check v0.9.5
    ├── polars-core v0.44.0
    │   ├── ahash v0.8.11 (*)
    │   ├── bitflags v2.6.0
    │   ├── bytemuck v1.19.0 (*)
    │   ├── chrono v0.4.38 (*)
    │   ├── comfy-table v7.1.1
    │   ├── either v1.13.0
    │   ├── hashbrown v0.14.5
    │   ├── hashbrown v0.15.0 (*)
    │   ├── indexmap v2.6.0
    │   ├── num-traits v0.2.19 (*)
    │   ├── once_cell v1.20.2
    │   ├── polars-arrow v0.44.0 (*)
    │   ├── polars-compute v0.44.0
    │   │   [build-dependencies]
    │   ├── polars-error v0.44.0 (*)
    │   ├── polars-row v0.44.0
    │   ├── polars-schema v0.44.0 (*)
    │   ├── polars-utils v0.44.0 (*)
    │   ├── rayon v1.10.0
    │   ├── regex v1.11.1
    │   ├── strum_macros v0.26.4 (proc-macro) (*)
    │   ├── thiserror v1.0.65
    │   └── xxhash-rust v0.8.12
    │   [build-dependencies]
    │   └── version_check v0.9.5
    ├── polars-error v0.44.0 (*)
    ├── polars-io v0.44.0
    │   ├── ahash v0.8.11 (*)
    │   ├── atoi_simd v0.15.6
    │   ├── bytes v1.8.0
    │   ├── chrono v0.4.38 (*)
    │   ├── fast-float v0.2.0
    │   ├── glob v0.3.1
    │   ├── hashbrown v0.15.0 (*)
    │   ├── home v0.5.9
    │   ├── itoa v1.0.11
    │   ├── memchr v2.7.4
    │   ├── memmap2 v0.7.1
    │   ├── num-traits v0.2.19 (*)
    │   ├── once_cell v1.20.2
    │   ├── percent-encoding v2.3.1
    │   ├── polars-arrow v0.44.0 (*)
    │   ├── polars-core v0.44.0 (*)
    │   ├── polars-error v0.44.0 (*)
    │   ├── polars-schema v0.44.0 (*)
    │   ├── polars-time v0.44.0
    │   ├── polars-utils v0.44.0 (*)
    │   ├── rayon v1.10.0 (*)
    │   ├── regex v1.11.1 (*)
    │   ├── ryu v1.0.18
    │   └── simdutf8 v0.1.5
    ├── polars-ops v0.44.0
    │   ├── ahash v0.8.11 (*)
    │   ├── argminmax v0.6.2
    │   ├── bytemuck v1.19.0 (*)
    │   ├── either v1.13.0
    │   ├── hashbrown v0.15.0 (*)
    │   ├── indexmap v2.6.0 (*)
    │   ├── memchr v2.7.4
    │   ├── num-traits v0.2.19 (*)
    │   ├── polars-arrow v0.44.0 (*)
    │   ├── polars-compute v0.44.0 (*)
    │   ├── polars-core v0.44.0 (*)
    │   ├── polars-error v0.44.0 (*)
    │   ├── polars-schema v0.44.0 (*)
    │   ├── polars-utils v0.44.0 (*)
    │   ├── rayon v1.10.0 (*)
    │   ├── regex v1.11.1 (*)
    │   ├── regex-syntax v0.8.5
    │   └── strum_macros v0.26.4 (proc-macro) (*)
    │   [build-dependencies]
    │   └── version_check v0.9.5
    ├── polars-parquet v0.44.0
    │   ├── ahash v0.8.11 (*)
    │   ├── base64 v0.22.1
    │   ├── bytemuck v1.19.0 (*)
    │   ├── ethnum v1.5.0
    │   ├── hashbrown v0.15.0 (*)
    │   ├── num-traits v0.2.19 (*)
    │   ├── polars-arrow v0.44.0 (*)
    │   ├── polars-compute v0.44.0 (*)
    │   ├── polars-error v0.44.0 (*)
    │   ├── polars-parquet-format v0.1.0
    │   ├── polars-utils v0.44.0 (*)
    │   ├── simdutf8 v0.1.5
    │   └── streaming-decompression v0.1.2
    ├── polars-time v0.44.0 (*)
    └── polars-utils v0.44.0 (*)
    [build-dependencies]
    └── version_check v0.9.5
(main) ~\playground\fadd took 0.4s ❯ cargo r
   Compiling proc-macro2 v1.0.89
   Compiling unicode-ident v1.0.13
   Compiling version_check v0.9.5
   Compiling windows_x86_64_gnu v0.52.6
   Compiling autocfg v1.4.0
   Compiling crossbeam-utils v0.8.20
   Compiling cfg-if v1.0.0
   Compiling serde v1.0.214
   Compiling shlex v1.3.0
   Compiling rustversion v1.0.18
   Compiling rayon-core v1.12.1
   Compiling either v1.13.0
   Compiling memchr v2.7.4
   Compiling parking_lot_core v0.9.10
   Compiling cc v1.1.31
   Compiling bitflags v2.6.0
   Compiling array-init-cursor v0.2.0
   Compiling thiserror v1.0.65
   Compiling ahash v0.8.11
   Compiling regex-syntax v0.8.5
   Compiling allocator-api2 v0.2.18
   Compiling num-traits v0.2.19
   Compiling aho-corasick v1.1.3
   Compiling lock_api v0.4.12
   Compiling libc v0.2.161
   Compiling windows-targets v0.52.6
   Compiling psm v0.1.23
   Compiling stacker v0.1.17
   Compiling planus v0.3.1
   Compiling quote v1.0.37
   Compiling crossbeam-epoch v0.9.18
   Compiling getrandom v0.2.15
   Compiling regex-automata v0.4.8
   Compiling smallvec v1.13.2
   Compiling once_cell v1.20.2
   Compiling equivalent v1.0.1
   Compiling target-features v0.1.6
   Compiling syn v2.0.85
   Compiling crossbeam-deque v0.8.5
   Compiling syn v1.0.109
   Compiling foldhash v0.1.3
   Compiling zerocopy v0.7.35
   Compiling scopeguard v1.2.0
   Compiling castaway v0.2.3
   Compiling windows-sys v0.59.0
   Compiling polars-utils v0.44.0
   Compiling rayon v1.10.0
   Compiling itoa v1.0.11
   Compiling simdutf8 v0.1.5
   Compiling ryu v1.0.18
   Compiling static_assertions v1.1.0
   Compiling parking_lot v0.12.3
   Compiling raw-cpuid v11.2.0
   Compiling polars-schema v0.44.0
   Compiling memmap2 v0.7.1
   Compiling winapi-x86_64-pc-windows-gnu v0.4.0
   Compiling heck v0.5.0
   Compiling bytes v1.8.0
   Compiling chrono v0.4.38
   Compiling polars-arrow v0.44.0
   Compiling winapi v0.3.9
   Compiling fast-float v0.2.0
   Compiling regex v1.11.1
   Compiling ethnum v1.5.0
   Compiling dyn-clone v1.0.17
   Compiling streaming-iterator v0.1.9
   Compiling atoi_simd v0.15.6
   Compiling strength_reduce v0.2.4
   Compiling polars-compute v0.44.0
   Compiling polars-core v0.44.0
   Compiling unicode-width v0.1.14
   Compiling strum v0.26.3
   Compiling polars-ops v0.44.0
   Compiling xxhash-rust v0.8.12
   Compiling argminmax v0.6.2
   Compiling now v0.1.3
   Compiling atoi v2.0.0
   Compiling windows-sys v0.52.0
   Compiling fallible-streaming-iterator v0.1.9
   Compiling serde_derive v1.0.214
   Compiling thiserror-impl v1.0.65
   Compiling bytemuck_derive v1.8.0
   Compiling strum_macros v0.26.4
   Compiling streaming-decompression v0.1.2
   Compiling polars v0.44.0
   Compiling percent-encoding v2.3.1
   Compiling base64 v0.22.1
   Compiling polars-parquet-format v0.1.0
   Compiling glob v0.3.1
   Compiling home v0.5.9
   Compiling bytemuck v1.19.0
   Compiling crossterm_winapi v0.9.1
   Compiling crossterm v0.27.0
   Compiling multiversion-macros v0.7.4
   Compiling comfy-table v7.1.1
   Compiling multiversion v0.7.4
   Compiling polars-arrow-format v0.1.0
   Compiling hashbrown v0.15.0
   Compiling compact_str v0.8.0
   Compiling hashbrown v0.14.5
   Compiling indexmap v2.6.0
   Compiling polars-error v0.44.0
   Compiling polars-row v0.44.0
   Compiling polars-parquet v0.44.0
   Compiling polars-time v0.44.0
   Compiling polars-io v0.44.0
   Compiling fadd v0.1.0 (C:\Users\Rahul\playground\fadd)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 49s
     Running `target\debug\fadd.exe`

My Cargo.toml:

(main) ~\playground\fadd took 0.4s ❯ cat .\Cargo.toml
[package]
name = "fadd"
version = "0.1.0"
edition = "2021"

[dependencies]
polars = "0.44.0"

I noticed that the Cargo.lock file includes entries for pyo3, pyo3-build-config, pyo3-ffi, pyo3-macros, and pyo3-macros-backend.

I can confirm that this issue occurs when adding other packages too. For example, when I tried to add eframe to an empty project, it pulled in dependencies like ndk-sys and wasi, which are unnecessary and don't even get compiled.

@rawhuul
Copy link

rawhuul commented Oct 28, 2024

It seems to be a modification in the registry, as indicated by this.

Starting with commit a31184b, cargo-add now uses QueryKind::Normalized, which is defined here. This change likely explains why Cargo is retrieving all dependencies.

@stinodego
Copy link
Member

stinodego commented Oct 28, 2024

Ok, but this is actually intended. There is a new crate polars-python that includes functionality for Python implementations of Polars. It's gated behind a feature flag.

So when building polars, you won't have to compile pyo3 and friends. But they will be part of the dependency tree. Not sure what the actual problem is here?

@eitsupi
Copy link
Contributor

eitsupi commented Oct 28, 2024

It seems that sometimes it doesn't work unless it links to Python eitsupi/neo-r-polars#25
Perhaps the same thing happens with nodejs-polars?
The CI runners have Python installed, so it can be hard to notice the problem.

@eitsupi
Copy link
Contributor

eitsupi commented Oct 28, 2024

Incidentally, I believe this change occurred fairly recently (between py-1.10.0 and py-1.12.0).
Long after polars-python was created (py-1.6.0).

@eitsupi
Copy link
Contributor

eitsupi commented Oct 28, 2024

It is possible that the polars-stream dependency has been broken for some time, and for some reason it no longer works without python when some feature flags are enabled.

# TODO: feature gate
polars-mem-engine = { workspace = true, features = ["parquet", "csv", "json", "ipc", "cloud", "python", "dtype-categorical", "dtype-i8", "dtype-i16", "dtype-u8", "dtype-u16", "dtype-decimal", "dtype-struct", "object"] }
polars-parquet = { workspace = true }
polars-plan = { workspace = true, features = ["parquet", "csv", "json", "ipc", "cloud", "python", "serde", "dtype-categorical", "dtype-i8", "dtype-i16", "dtype-u8", "dtype-u16", "dtype-decimal", "dtype-struct", "object"] }

@LegoWolf
Copy link

LegoWolf commented Oct 28, 2024

In support of what @eitsupi said, the simplest repro example I can come up with is an empty Rust project that includes Polars v0.44.0, and the lazy feature as well as any one of the strings, bitwise, ordynamic_group_by features. That builds polars-stream which in turn compiles some pyo3 modules.

Interestingly, the dependencies in polars-lazy/Cargo.toml did add explicit references to polars-stream/strings, polars-stream/bitwise and polars-stream/dynamic_group_by quite recently. This might well be an exposed problem in the dependencies of polars-stream.

https://github.com/pola-rs/polars/blame/main/crates/polars-lazy/Cargo.toml

@rawhuul
Copy link

rawhuul commented Oct 29, 2024

So when building polars, you won't have to compile pyo3 and friends. But they will be part of the dependency tree. Not sure what the actual problem is here?

Yes @stinodego, the dependencies are not included in the build or dependency tree, but are instead synchronized with a local index.

@orlp
Copy link
Collaborator

orlp commented Oct 29, 2024

I'm at fault here, the offending lines are the following in polars-stream/Cargo.toml:

# TODO: feature gate
polars-mem-engine = { workspace = true, features = ["parquet", "csv", "json", "ipc", "cloud", "python", "dtype-categorical", "dtype-i8", "dtype-i16", "dtype-u8", "dtype-u16", "dtype-decimal", "dtype-struct", "object"] }
polars-parquet = { workspace = true }
polars-plan = { workspace = true, features = ["parquet", "csv", "json", "ipc", "cloud", "python", "serde", "dtype-categorical", "dtype-i8", "dtype-i16", "dtype-u8", "dtype-u16", "dtype-decimal", "dtype-struct", "object"] }

This unconditionally turns on Python (and a bunch of other flags).

EDIT: actually it appears these lines were added by @nameexhaustion . I didn't catch it in review though, so you can still blame me :)

@stinodego
Copy link
Member

In support of what @eitsupi said, the simplest repro example I can come up with is an empty Rust project that includes Polars v0.44.0, and the lazy feature as well as any one of the strings, bitwise, ordynamic_group_by features. That builds polars-stream which in turn compiles some pyo3 modules.

Interestingly, the dependencies in polars-lazy/Cargo.toml did add explicit references to polars-stream/strings, polars-stream/bitwise and polars-stream/dynamic_group_by quite recently. This might well be an exposed problem in the dependencies of polars-stream.

main/crates/polars-lazy/Cargo.toml (blame)

I can reproduce this, thanks.

@stinodego stinodego added P-high Priority: high and removed needs triage Awaiting prioritization by a maintainer labels Oct 29, 2024
@github-project-automation github-project-automation bot moved this to Ready in Backlog Oct 29, 2024
@ritchie46
Copy link
Member

Polars-stream features in lazy should also be activated with a question mark:

E.g. "polars-stream?/strings".

@rawhuul
Copy link

rawhuul commented Oct 29, 2024

Hi there, I tried it out again and it still adds pyo3 dependencies.

(main) ~\playground\gemmi took 0.1s ❯ cargo add --git https://github.com/pola-rs/polars polars
    Updating git repository `https://github.com/pola-rs/polars`
      Adding polars (git) to dependencies
             Features:
             + csv
             + docs
             + dtype-date
             + dtype-datetime
             + dtype-duration
             + dtype-slim
             + fmt
             + polars-io
             + polars-time
             + temporal
             + zip_with
             139 deactivated features
    Updating git repository `https://github.com/pola-rs/polars`
    Updating crates.io index
     Locking 318 packages to latest compatible versions
      Adding atoi_simd v0.15.6 (latest: v0.16.0)
      Adding brotli v6.0.0 (latest: v7.0.0)
      Adding chrono-tz v0.8.6 (latest: v0.10.0)
      Adding chrono-tz-build v0.2.1 (latest: v0.4.0)
      Adding core-foundation v0.9.4 (latest: v0.10.0)
      Adding crossterm v0.27.0 (latest: v0.28.1)
      Adding fs4 v0.9.1 (latest: v0.10.0)
      Adding generic-array v0.14.7 (latest: v1.1.0)
      Adding hashbrown v0.14.5 (latest: v0.15.0)
      Adding heck v0.4.1 (latest: v0.5.0)
      Adding hermit-abi v0.3.9 (latest: v0.4.0)
      Adding idna v0.5.0 (latest: v1.0.2)
      Adding linux-raw-sys v0.4.14 (latest: v0.6.5)
      Adding memmap2 v0.7.1 (latest: v0.9.5)
      Adding object_store v0.10.2 (latest: v0.11.1)
      Adding planus v0.3.1 (latest: v1.0.0)
      Adding pyo3 v0.21.2 (latest: v0.22.5)
      Adding pyo3-build-config v0.21.2 (latest: v0.22.5)
      Adding pyo3-ffi v0.21.2 (latest: v0.22.5)
      Adding pyo3-macros v0.21.2 (latest: v0.22.5)
      Adding pyo3-macros-backend v0.21.2 (latest: v0.22.5)
      Adding quick-xml v0.36.2 (latest: v0.37.0)
      Adding security-framework v2.11.1 (latest: v3.0.0)
      Adding siphasher v0.3.11 (latest: v1.0.1)
      Adding snafu v0.7.5 (latest: v0.8.5)
      Adding snafu-derive v0.7.5 (latest: v0.8.5)
      Adding sqlparser v0.49.0 (latest: v0.51.0)
      Adding syn v1.0.109 (latest: v2.0.85)
      Adding sysinfo v0.31.4 (latest: v0.32.0)
      Adding unicode-width v0.1.14 (latest: v0.2.0)
      Adding wasi v0.11.0+wasi-snapshot-preview1 (latest: v0.13.3+wasi-0.2.2)
      Adding windows v0.57.0 (latest: v0.58.0)
      Adding windows-core v0.52.0 (latest: v0.58.0)
      Adding windows-core v0.57.0 (latest: v0.58.0)
      Adding windows-implement v0.57.0 (latest: v0.58.0)
      Adding windows-interface v0.57.0 (latest: v0.58.0)
      Adding windows-registry v0.2.0 (latest: v0.3.0)
      Adding windows-result v0.1.2 (latest: v0.2.0)
      Adding windows-strings v0.1.0 (latest: v0.2.0)
      Adding windows-sys v0.52.0 (latest: v0.59.0)
      Adding zerocopy v0.7.35 (latest: v0.8.7)
      Adding zerocopy-derive v0.7.35 (latest: v0.8.7)
(main) ~\playground\gemmi took 81.4s ❯

This is expected behaviour for cargo, as described here

@eitsupi
Copy link
Contributor

eitsupi commented Oct 29, 2024

The real problem here was not that pyo3 was added to the Cargo.lock file, but that pyo3 was actually compiled.

@rawhuul
Copy link

rawhuul commented Oct 29, 2024

Oh, I didn't encounter that problem, that's why I view it from a different perspective. Anyhow it seemed to resolve the issue, thanks.

@LegoWolf
Copy link

I've confirmed that my problem is definitely fixed with this change. Thanks so much for the super-fast turnaround!

@ankane
Copy link
Contributor Author

ankane commented Oct 29, 2024

Thanks @stinodego and @ritchie46! It still seems like the issue isn't fully resolved, but sounds like that's known (#19512 (comment)).

@c-peters c-peters added the accepted Ready for implementation label Nov 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
accepted Ready for implementation bug Something isn't working build Changes that affect the build system or external dependencies P-high Priority: high rust Related to Rust Polars
Projects
Archived in project
9 participants