Skip to content

Releases: pola-rs/polars

Python Polars 0.20.17

28 Mar 16:37
d999c01
Compare
Choose a tag to compare

🏆 Highlights

⚠️ Deprecations

  • Rename parameter by to group_by in DataFrame.upsample/group_by_dynamic/rolling (#14840)
  • Rename from_repr parameter from tbl to data (#15156)

🚀 Performance improvements

  • load bits one word at a time for BitmapIter (#15333)
  • Ipc exec multiple paths (#15040)
  • add SIMD support for if-then-else kernels (#15131)

✨ Enhancements

  • Remove 'FileCacher' optimization (#15357)
  • check input type in entropy (#15351)
  • Implements arr.n_unique (#15296)
  • CSE don't scan share if predicate pushdown predicates don't match (#15328)
  • Add read_database support for SurrealDB ("ws" and "http") (#15269)
  • Only allow inputs of type Sequence in from_records (#15329)
  • In hypothesis testing strategies, enable Decimal strategy by default (#15321)
  • Remove cached nodes when finished (#15310)
  • Full plan CSE (#15264)
  • More robust handling of async database calls (#15202)
  • Add name parameter to GroupBy.len method (#15235)
  • Add IR for expressions. (#15168)
  • Improve read_database when reading from Kùzu graph database (#15218)
  • Warn if map_elements is called without return_dtype specified (#15188)
  • Add support for async SQLAlchemy connections to read_database (#15162)
  • Infer time_unit in pl.duration when nanoseconds is specified (#14987)
  • Add strict parameter to from_dict/from_records (#15158)

🐞 Bug fixes

  • Return correct dtype for s.clear() when dtype is Object (#15315)
  • ensure first datapoint is always included in group_by_dynamic (#15312)
  • Non-exhaustive patterns: arrow-schema::DataType in polars-arrow (#15250)
  • use dynamic stacks for problematic recursive functions (#15355)
  • Adding default ddof for Series.list.std and Series.list.var (#15267)
  • Raise properly for slices not supported by LazyFrame (#15331)
  • Propagate strictness in from_dicts (#15344)
  • Raise error when schema_overrides contains nonexistent column name (#15290)
  • Enforce integer dtype input for int_range and int_ranges (#15339)
  • Preserve Decimal precision when constructing empty Series (#15320)
  • Fix cache dot visualization (#15311)
  • Handle special case correctly when slicing a LazyFrame (#15297)
  • Properly propagate strict flag when constructing a Struct Series from any values (#15302)
  • Consistent expansion of nested struct data during DataFrame init from dict (#15217)
  • Raise when join projects name with suffix that doesn't exist (#15256)
  • Ensure ooc_start is set (#15255)
  • Fix bug where rolling operations were ignoring check_sorted in some cases (#15227)
  • Fix lazy schema for rle expression (#15248)
  • incorrect negative offset in multi-byte string slicing (#15140)
  • do not clamp negative offsets to start of array prematurely (#15242)
  • allow null index in list.get and array.get (#15239)
  • Avoid loading all columns in read_parquet when columns parameter is specified (#15229)
  • properly support nulls_last + descending (#15212)
  • fix nested runtime panic (#15216)
  • Block rounding/truncating to negative durations (#15175)
  • Ensure the cs.temporal() selector uses wildcard time zone matching for Datetime (#13683)
  • Consistently raise TypeError on constructor failure (#15178)
  • Properly propagate strictness in some constructor cases (#15166)
  • Fix constructing a Series from a list of Series with given dtype (#15144)

📖 Documentation

  • Fix time unit in timestamp example (#15281)
  • Fix link to renamed method (.list.lengths -> .list.len) (#15228)
  • Update Excel and database pages in user guide (#14721)
  • Add examples for Series.search_sorted (#14737)
  • Correct the definition of an expression in the user guide (#14750)
  • Add a note about the behaviour of lower/upper bounds for is_between, and add an example (#15197)

📦 Build system

🛠️ Other improvements

  • Memory usage test infrastructure, plus a test for #15098 (#15285)
  • Update CODEOWNERS (#15352)
  • remove try_apply_values (#15336)
  • always use non-legacy float_sum for mean (#15343)
  • remove legacy bitmap module (#15335)
  • Fix test not writing to temporary directory (#15318)
  • Reorganize tests for clear operation (#15304)
  • Rename Cache[count] to Cache[cache_hits] (#15300)
  • Cleanup file_caching optimization call (#15299)
  • Minor refactor of PyDataFrame.from_dicts (#15274)
  • remove 'fast-projection' node (#15253)
  • cleanup column names in optimizer (#15252)
  • remove left_most_input_name from expr ir (#15251)
  • add AlignedBitmapSlice (#15171)
  • Run non-benchmark tests in benchmark workflow (#15207)
  • Add wrapping_abs to arithmetic kernel (#15210)
  • remove raw buffers from BinViewArray (#15206)
  • Enable RUST_BACKTRACE=1 in the CI test suite (#15204)
  • Split read_database functionality into cleaner module structure (#15201)
  • Clean up some of the AnyValue conversion logic (#15190)
  • remove parts of legacy bit_util (#15169)
  • remove legacy arrow compute (#15164)
  • Split up dataframe module in PyO3 bindings (#15165)
  • Remove unused private constructors (#15160)

Thank you to all our contributors for making this release possible!
@MarcoGorelli, @alexander-beedie, @braaannigan, @c-peters, @cojmeister, @deanm0000, @dependabot, @dependabot[bot], @itamarst, @kszlim, @mbuhidar, @mcrumiller, @mickvangelderen, @orlp, @petrosbar, @reswqa, @ritchie46, @rob-sil, @sportfloh, @stinodego and @yutannihilation

Rust Polars 0.38.3

19 Mar 07:49
3e2cab8
Compare
Choose a tag to compare

🚀 Performance improvements

  • add new when-then-otherwise kernels (#15089)
  • Coerce sorted flag of unit arrays during concat (#15104)
  • Use sorted flag for (first|last)_non_null (#15050)
  • OOC sort improvements (#14994)

✨ Enhancements

  • raise if both closed and by are passed to rolling_* aggregations (#15108)
  • raise informative error for rolling_* aggs with by of invalid dtype (#15088)
  • add non_existent arg to replace_time_zone (#15062)
  • Support single nested row encodings (#15105)
  • make ooc sort configurable (#15084)
  • Make register_plugin a standalone function and include shared lib discovery (#14804)
  • Async parquet: Decode parquet on a blocking thread pool (#15083)
  • let "ambiguous" take "null" value (#14961)
  • Raise informative error message when join would introduce duplicate column name (#15042)
  • Allow cast of decimal to boolean (#15015)
  • Return error when no supertype can be determined in AnyValue constructor when strict=false (#15025)
  • Implement IpcReaderAsync (#14984)
  • Support Array statistics in parquet (#15031)
  • Support decimal groupby (#15000)
  • Add thread names to rayon thread pool (#15024)
  • Support decimal uniq (#15001)
  • expose timings in verbose state of OOC sort (#14979)

🐞 Bug fixes

  • Fix Series construction from nested list with mixed data types (#15046)
  • Support BinaryView in row decoder to prevent a panic in streaming group by (#15117)
  • Binview chunked gather; don't modify inlined view (#15124)
  • Fix chunked_id gather for binview buffers (#15123)
  • Don't cache HTTP object stores as they maintain URL state (#15121)
  • use wrapping_add in csv line snooping (#15109)
  • Output u32 when sum_horizontal provided with single boolean column (#15114)
  • Ensure eprintln! is only called within debug/verbose context (#15100)
  • Propagate error instead of panicking when calling product on an invalid type (#15093)
  • Raise error when casting Array to different width (#14995)
  • Fix file scan bugs for ipc, csv and parquet that occur with combinations of glob paths, row indices and predicates (#15065)
  • Incorrectly preserved sorted flag when concatenating sorted series containing nulls (#15082)
  • Return largest non-NaN value for max() on sorted float arrays if it exists instead of NaN (#15060)
  • return NaN for all-NaN min/max (#15066)
  • Prevent "index out of range for slice" error in parquet reader (#15021)
  • Respect nulls_last in streaming sort (#15061)
  • Fix Series construction from nested list with mixed data types (#15046)
  • Don't count nulls in streaming count agg (#15051)
  • agg_list on decimal lost scale (#15054)
  • Block predicate pushdown on equality that are use in join (#15055)
  • Enum equality based on categories (#15053)
  • Strict cast in when/then/otherwise operation (#15052)
  • Don't panic in string_addition_to_linear_concat (#15006)
  • CSV do utf8-validation after escaping fields (#15004)
  • Use primitive constructors to create a Series of lists when dtype is provided (#15002)
  • replace_time_zone with single-null-element "ambiguous" was panicking (#14971)

📖 Documentation

  • Fix typo in comment (#14997)

🛠️ Other improvements

  • Extend and speed up scan tests (#15127)
  • always assert on ChunkedArray::get (#15120)
  • Use ObjectStore instead of AsyncRead in parquet get metadata (#15069)
  • Minor refactor of Rust any value constructors (#15077)
  • Simplify streaming execution (#15039)
  • Ensure we hit the spilled source path in ooc sort test (#15010)
  • Refactor constructor code (#15009)
  • Apply clippy:assigning_clones lint (#14999)
  • fix features (#14977)

Thank you to all our contributors for making this release possible!
@JackRolfe, @MKisilyov, @MarcoGorelli, @alexander-beedie, @c-peters, @flisky, @jqnatividad, @mcrumiller, @mickvangelderen, @nameexhaustion, @orlp, @petrosbar, @ritchie46, @stinodego and @trueb2

Python Polars 0.20.16

18 Mar 18:33
ec8f5e3
Compare
Choose a tag to compare

🚀 Performance improvements

  • add new when-then-otherwise kernels (#15089)
  • Coerce sorted flag of unit arrays during concat (#15104)
  • Use sorted flag for (first|last)_non_null (#15050)
  • OOC sort improvements (#14994)

✨ Enhancements

  • improved dtype inference/refinement for read_database results (#15126)
  • raise if both closed and by are passed to rolling_* aggregations (#15108)
  • raise informative error for rolling_* aggs with by of invalid dtype (#15088)
  • add non_existent arg to replace_time_zone (#15062)
  • Support single nested row encodings (#15105)
  • make ooc sort configurable (#15084)
  • Make register_plugin a standalone function and include shared lib discovery (#14804)
  • Expose infer_schema_length parameter on read_database (#15076)
  • Async parquet: Decode parquet on a blocking thread pool (#15083)
  • let "ambiguous" take "null" value (#14961)
  • Raise informative error message when join would introduce duplicate column name (#15042)
  • Allow cast of decimal to boolean (#15015)
  • Add strict parameter to DataFrame constructor to allow non-strict construction (#15034)
  • Support Array statistics in parquet (#15031)
  • Support decimal groupby (#15000)
  • Add thread names to rayon thread pool (#15024)
  • Support decimal uniq (#15001)
  • expose timings in verbose state of OOC sort (#14979)

🐞 Bug fixes

  • Support BinaryView in row decoder to prevent a panic in streaming group by (#15117)
  • Binview chunked gather; don't modify inlined view (#15124)
  • Fix chunked_id gather for binview buffers (#15123)
  • Don't cache HTTP object stores as they maintain URL state (#15121)
  • Output u32 when sum_horizontal provided with single boolean column (#15114)
  • Propagate error instead of panicking when calling product on an invalid type (#15093)
  • Raise error when casting Array to different width (#14995)
  • Fix file scan bugs for ipc, csv and parquet that occur with combinations of glob paths, row indices and predicates (#15065)
  • Incorrectly preserved sorted flag when concatenating sorted series containing nulls (#15082)
  • Return largest non-NaN value for max() on sorted float arrays if it exists instead of NaN (#15060)
  • return NaN for all-NaN min/max (#15066)
  • Prevent "index out of range for slice" error in parquet reader (#15021)
  • Respect nulls_last in streaming sort (#15061)
  • Fix Series construction from nested list with mixed data types (#15046)
  • Don't count nulls in streaming count agg (#15051)
  • agg_list on decimal lost scale (#15054)
  • Block predicate pushdown on equality that are use in join (#15055)
  • Enum equality based on categories (#15053)
  • Don't panic in string_addition_to_linear_concat (#15006)
  • CSV do utf8-validation after escaping fields (#15004)
  • Use primitive constructors to create a Series of lists when dtype is provided (#15002)
  • replace_time_zone with single-null-element "ambiguous" was panicking (#14971)

📖 Documentation

  • Update write_database code blocks in user guide (#15106)
  • Add missing docstring examples in the Struct namespace (#15071)
  • Improve API reference landing page (#14888)
  • improve join_asof example (#14993)
  • Fix inadvertent swap of new and old parameters in replace description (#15019)

🛠️ Other improvements

  • Extend and speed up scan tests (#15127)
  • Add parameterized-scan-tests (#15057)
  • Simplify streaming execution (#15039)
  • Ensure we hit the spilled source path in ooc sort test (#15010)
  • Refactor constructor code (#15009)
  • fix features (#14977)
  • Revert pinning PyPI publish action (#14975)

Thank you to all our contributors for making this release possible!
@JackRolfe, @MKisilyov, @MarcoGorelli, @alexander-beedie, @c-peters, @flisky, @jqnatividad, @mcrumiller, @mickvangelderen, @nameexhaustion, @orlp, @petrosbar, @ritchie46, @stinodego and @trueb2

Python Polars 0.20.16-rc.1

14 Mar 14:40
d0a8392
Compare
Choose a tag to compare
Pre-release

🚀 Performance improvements

  • Use sorted flag for (first|last)_non_null (#15050)
  • OOC sort improvements (#14994)

✨ Enhancements

  • let "ambiguous" take "null" value (#14961)
  • Raise informative error message when join would introduce duplicate column name (#15042)
  • Allow cast of decimal to boolean (#15015)
  • Add strict parameter to DataFrame constructor to allow non-strict construction (#15034)
  • Support Array statistics in parquet (#15031)
  • Support decimal groupby (#15000)
  • Add thread names to rayon thread pool (#15024)
  • Support decimal uniq (#15001)
  • expose timings in verbose state of OOC sort (#14979)

🐞 Bug fixes

  • Prevent "index out of range for slice" error in parquet reader (#15021)
  • Respect nulls_last in streaming sort (#15061)
  • Fix Series construction from nested list with mixed data types (#15046)
  • Don't count nulls in streaming count agg (#15051)
  • agg_list on decimal lost scale (#15054)
  • Block predicate pushdown on equality that are use in join (#15055)
  • Enum equality based on categories (#15053)
  • Don't panic in string_addition_to_linear_concat (#15006)
  • CSV do utf8-validation after escaping fields (#15004)
  • Use primitive constructors to create a Series of lists when dtype is provided (#15002)
  • replace_time_zone with single-null-element "ambiguous" was panicking (#14971)

📖 Documentation

  • Improve API reference landing page (#14888)
  • improve join_asof example (#14993)
  • Fix inadvertent swap of new and old parameters in replace description (#15019)

🛠️ Other improvements

  • Add parameterized-scan-tests (#15057)
  • Simplify streaming execution (#15039)
  • Ensure we hit the spilled source path in ooc sort test (#15010)
  • Refactor constructor code (#15009)
  • fix features (#14977)
  • Revert pinning PyPI publish action (#14975)

Thank you to all our contributors for making this release possible!
@MKisilyov, @MarcoGorelli, @alexander-beedie, @c-peters, @flisky, @jqnatividad, @mcrumiller, @mickvangelderen, @nameexhaustion, @petrosbar, @ritchie46, @stinodego and @trueb2

Rust Polars 0.38.2

11 Mar 08:15
946fad7
Compare
Choose a tag to compare

🏆 Highlights

  • Streaming outer joins (#14828)

🚀 Performance improvements

  • Ensure parallel encoding/compression in sink_parquet (#14964)
  • hoist errors out of iterators in parquet (#14945)
  • add basic AVX-512 filters (#14892)
  • improve join-asof materialization (#14884)
  • Optimize chunked-id gather for binaryviews (#14878)
  • rework scalar filter kernels (#14865)
  • Reduce size of optional join-indexes (#14856)
  • Streaming outer joins (#14828)
  • Set sorted flag for cum_count on columns (#14849)

✨ Enhancements

  • Support writing Array type in parquet (#14943)
  • Sort decimal fields (#14649)
  • Import NamedFrom in df! macro (#14860)
  • try-improve concurrency tuner (#14827)
  • Streaming outer joins (#14828)
  • Set sorted flag for cum_count on columns (#14849)
  • Ensure binview types are rle-encoded in parquet write (#14818)
  • Implement strict/nonstrict conversion for primitive AnyValues (#14186)
  • Disable timeouts (#14809)
  • cleanup spill disks in process (#14807)

🐞 Bug fixes

  • Fix invalid partitionable query (#14966)
  • allow nonstrict cast of categorical/enum to enum (#14910)
  • count_rows multi-threaded under-counting in parser.rs (#14963)
  • raise proper error instead of panicking when result of truncation is non-existent datetime (#14958)
  • ooc-sort issues (#14959)
  • Do not raise when constructing from a list of Series with Nones (#14942)
  • Don't access out-of-bounds for null indices in bitmap gather (#14932)
  • std when ddof>=n_values returns None even in rolling context (#11750)
  • Don't rechunk categoricals when moving to physical (#14934)
  • parquet rle boolean decoder (#14931)
  • boolean filter gave overly large buffers to Bitmap::from_u8_vec (#14924)
  • Fix sliced dictionary state in parquet (#14917)
  • Fix possibly incorrect order of columns when using ipc stream with_columns (#14859)
  • Fully qualify polars_bail! in polars_ensure! (#14901)
  • Fix DataFrame.min/max for decimals (#14890)
  • Assert chunks are equal after physical cast to prevent OOB (#14873)
  • not all cpu feature flag tests were mocked (#14864)

📖 Documentation

  • Remove some repetition in comments/docstrings (#14912)
  • Update contributing link (#14882)
  • Fix some word-repetition in code comments (#14825)
  • Seperate asof from join strategy, change parameter from strategy to how in user guide (#14793)

🛠️ Other improvements

  • fix features (#14977)
  • fix chrono deprecation warnings (#14928)
  • Update Cargo.lock and remove cmake limit workaround (#14905)
  • Simplify streaming placeholder replacement. (#14915)
  • Optional deps should include fastexcel (#14907)
  • Deduplicate POLARS_FORCE_ASYNC env var parsing (#14909)
  • Make assumption about column name to index conversion having occurred explicit (#14894)
  • Make assumption about wildcards having been resolved explicit (#14899)
  • reactivate argminmax simd (#14679)
  • sort by 'idx' after outer join (#14867)
  • Simplify computation of with_columns attribute in physical csv scanner of default engine. (#14837)
  • centrally define IdxSize (#14854)
  • run and fix pext64_polyfill test (#14852)
  • introduce partitioned table (#14819)
  • add missing deprecation directive in groupby.count (#14817)
  • Extract key value construction (#14812)
  • Fix Makefile build commands (#14806)

Thank you to all our contributors for making this release possible!
@MarcoGorelli, @Sol-Hee, @alexander-beedie, @ambidextrous, @battmdpkq, @deanm0000, @dependabot, @dependabot[bot], @eitsupi, @flisky, @geekvest, @mcrumiller, @mickvangelderen, @nameexhaustion, @orlp, @petrosbar, @ritchie46 and @stinodego

Python Polars 0.20.15

10 Mar 20:33
4bc67a0
Compare
Choose a tag to compare

🚀 Performance improvements

  • Ensure parallel encoding/compression in sink_parquet (#14964)
  • hoist errors out of iterators in parquet (#14945)
  • add basic AVX-512 filters (#14892)

✨ Enhancements

  • Support writing Array type in parquet (#14943)
  • Add drop_first parameter to Series.to_dummies (#14846)
  • Add "execute_options" support for read_database_uri (#14682)

🐞 Bug fixes

  • Fix invalid paritionable query (#14966)
  • allow nonstrict cast of categorical/enum to enum (#14910)
  • count_rows multi-threaded under-counting in parser.rs (#14963)
  • raise proper error instead of panicking when result of truncation is non-existent datetime (#14958)
  • ooc-sort issues (#14959)
  • Do not raise when constructing from a list of Series with Nones (#14942)
  • Don't access out-of-bounds for null indices in bitmap gather (#14932)
  • std when ddof>=n_values returns None even in rolling context (#11750)
  • Don't rechunk categoricals when moving to physical (#14934)
  • Ensure consistent read_database behaviour with empty ODBC "iter_batches" (#14918)
  • parquet rle boolean decoder (#14931)
  • Fix frame init from single RecordBatch objects when pyarrow <= 12 (#14922)
  • boolean filter gave overly large buffers to Bitmap::from_u8_vec (#14924)
  • Fix sliced dictionary state in parquet (#14917)
  • read_database now properly handles empty result sets from arrow-odbc (#14916)
  • Fix possibly incorrect order of columns when using ipc stream with_columns (#14859)

📖 Documentation

  • Add note about include_index in from_pandas regarding "default indices" (#14920)
  • Remove some repetition in comments/docstrings (#14912)

🛠️ Other improvements

  • Update Cargo.lock and remove cmake limit workaround (#14905)
  • Simplify streaming placeholder replacement. (#14915)
  • Optional deps should include fastexcel (#14907)
  • Deduplicate POLARS_FORCE_ASYNC env var parsing (#14909)

Thank you to all our contributors for making this release possible!
@MarcoGorelli, @alexander-beedie, @ambidextrous, @battmdpkq, @mcrumiller, @mickvangelderen, @orlp, @petrosbar and @ritchie46

Python Polars 0.20.14

07 Mar 11:40
25d0a2f
Compare
Choose a tag to compare

🏆 Highlights

  • Streaming outer joins (#14828)

⚠️ Deprecations

  • Deprecate overwrite_schema parameter for DataFrame.write_delta (#14879)

🚀 Performance improvements

  • improve join-asof materialization (#14884)
  • Optimize chunked-id gather for binaryviews (#14878)
  • rework scalar filter kernels (#14865)
  • Reduce size of optional join-indexes (#14856)
  • Streaming outer joins (#14828)
  • Set sorted flag for cum_count on columns (#14849)

✨ Enhancements

  • Sort decimal fields (#14649)
  • Revert addition of __slots__ to Polars classes (#14857)
  • Add fastexcel to show_versions (#14869)
  • try-improve concurrency tuner (#14827)
  • Streaming outer joins (#14828)
  • Set sorted flag for cum_count on columns (#14849)
  • support use of KùzuDB via pl.read_database (#14822)
  • Ensure binview types are rle-encoded in parquet write (#14818)
  • Disable timeouts (#14809)
  • cleanup spill disks in process (#14807)
  • Implement compression and skipping for binview IPC (#14789)

🐞 Bug fixes

  • Fix DataFrame.min/max for decimals (#14890)
  • Assert chunks are equal after physical cast to prevent OOB (#14873)
  • not all cpu feature flag tests were mocked (#14864)
  • Remove custom __reduce__ implementation on DataType object (#14778)
  • Allow non-strict construction / initialization of Enum columns (#14728)
  • Fix streaming parquet limit (#14783)

📖 Documentation

  • Update contributing link (#14882)
  • update to use ambiguous instead of use_earliest (#14820)
  • Seperate asof from join strategy, change parameter from strategy to how in user guide (#14793)

🛠️ Other improvements

  • Pin PyPI publish action to commit (#14896)
  • reactivate argminmax simd (#14679)
  • sort by 'idx' after outer join (#14867)
  • run and fix pext64_polyfill test (#14852)
  • add missing deprecation directive in groupby.count (#14817)
  • Fix Makefile build commands (#14806)
  • Bump ruff from 0.2.0 to 0.3.0 in /py-polars (#14800)
  • Rename utils module to _utils to explicitly mark it as private (#14772)
  • Add test coverage for _cpu_check module (#14768)

Thank you to all our contributors for making this release possible!
@MarcoGorelli, @Sol-Hee, @alexander-beedie, @c-peters, @deanm0000, @dependabot, @dependabot[bot], @eitsupi, @flisky, @geekvest, @mcrumiller, @mickvangelderen, @nameexhaustion, @orlp, @petrosbar, @ritchie46 and @stinodego

Rust Polars 0.38.1

01 Mar 08:40
a199ed5
Compare
Choose a tag to compare

🚀 Performance improvements

  • Elide utf8/binary cast in Parquet reading (#14757)

✨ Enhancements

  • Implement compression and skipping for binview IPC (#14789)

🐞 Bug fixes

  • fix feature flags (#14802)
  • Allow non-strict construction / initialization of Enum columns (#14728)
  • Fix streaming parquet limit (#14783)

📦 Build system

  • bump rayon from 1.8.1 to 1.9.0 (#14797)

Thank you to all our contributors for making this release possible!
@alexander-beedie, @c-peters, @dependabot, @dependabot[bot], @ritchie46 and @stinodego

Rust Polars 0.38.0

29 Feb 05:55
b959a6c
Compare
Choose a tag to compare

🏆 Highlights

  • fast path for COUNT(*) queries (#14574)
  • Implemented tree formatting for LogicalPlan (#14221)

💥 Breaking changes

  • Infer values columns in DataFrame.pivot when values is None (#14477)
  • Mark DataFrame::new_no_checks and DataFrame::new_no_length_checks unsafe (#14443)
  • Remove DatetimeChunked::convert_time_zone (#14046)
  • Rename LiteralValue::to_anyvalue to LiteralValue::to_any_value (#14033)

🚀 Performance improvements

  • auto-tune concurrency budget (#14753)
  • Don't materialize for broadcasting fill_null value and default value of replace (#14736)
  • Improve performance of boolean filters 1-100x. (#14746)
  • fix accidental quadratic utf8 validation in parquet (#14705)
  • fast path for COUNT(*) queries (#14574)
  • Elide the total order wrapper for non-(float/option) types (#14648)
  • add utf8-validation fast paths for utf8view (#14644)
  • don't reassign chunks back to df owner (#14633)
  • If there are many small chunks in write_parquet(), convert to a single chunk (#14484) (#14487)
  • Polars thread pool was not used properly in various functions (#14583)
  • use owned arithmetic in horizontal_sum (#14525)
  • Combine small chunks in sinks for streaming pipelines (#14346)
  • reduce heap allocs in expression/logical-plan iteration (#14440)
  • simplify and speed up cum_sum and cum_prod (#14409)
  • simplify negated predicates to improve row groups skipping (#14370)
  • prune parquet row groups when is_not_null is used (#14260)
  • use is_between to skip parquet row groups (#14244)
  • Use a compression API that is designed for this use case (#11699) (#14194)
  • Use UnitVec in polars-plan traversal (#14199)
  • use UnitVec in streaming joins (#14197)
  • improve ChunkId (#14175)
  • improve iteration performance (#14126)
  • elide unneeded work in window? (#14108)
  • run window functions more in parallel (#14095)
  • improve skip row group using statistics condition (#14056)

✨ Enhancements

  • Change default for maximum number of Series items printed to 10 to match DataFrame (#14703)
  • Infer values columns in DataFrame.pivot when values is None (#14477)
  • fast path for COUNT(*) queries (#14574)
  • let rolling accept index_column of type UInt32 or UInt64 (#14669)
  • Treat float -0.0 == 0.0 and -NaN == NaN in group-by, joins and unique (#14617)
  • Properly cache object-stores (#14598)
  • Mark DataFrame::new_no_checks and DataFrame::new_no_length_checks unsafe (#14443)
  • flatten aliases (#14512)
  • Make formatting more consistent in DOT graphs (#14486)
  • add flush operator to streaming operators (#14500)
  • Increase verbosity of duplicate column error message (#11899)
  • change print to warn in reading csv from python file like object (#14469)
  • Raise if pivot would introduce duplicate column names (#14431)
  • apply negate in simplify expression pass (#14436)
  • restrict more cloud interop to semaphore budget (#14435)
  • Implement min/max for categorical dtype (#14112)
  • add boolean rle decoding for parquet (#14403)
  • Allow brackets in SQL join conditions (#14263)
  • Improve panic message for missing struct feature in DataType::from_arrow (#14392)
  • Implement the IntoLazy trait for LazyFrame (#14323)
  • Implemented tree formatting for LogicalPlan (#14221)
  • Implement mean_horizontal expression (#14369)
  • support decimal comparison (#14338)
  • Implements arr.shift (#14298)
  • Implements list.n_unique (#14306)
  • Do not panic when casting from an empty Series to pl.Decimal (#14330)
  • add u8/i8/u16/i16 parsers to CSV reader (#14241)
  • Implements list.gather_every (#14253)
  • Implements prefix/suffix_fields (#14251)
  • Polish decimal arithmetic (#14172)
  • Introduce arr.to_struct (#14202)
  • Supports map fields name of struct (#14203)
  • make IdxVec generic as UnitVec (#14196)
  • add new arithmetic kernels (#14026)
  • Supports unique and hash_rows for null column (#14111)
  • Implement arithmetic operations for Null columns (#14107)
  • Add strict/non-strict construction of Boolean/Binary series (#14073)
  • Improve Series::from_any_values logic (#14052)
  • Adapt extend_constant to function expr architecture and expressify it (#14058)
  • add integer negation (#14049)
  • list & array measures of dispersion (#13245)
  • gc binview when writing ipc (#14035)
  • When calling convert_time_zone on time-zone-naive datetime, convert as if converting from UTC (#13960)

🐞 Bug fixes

  • fix hashing specialization (#14754)
  • Sum after filter in aggregation context sometimes returned NULL (#14752)
  • Allow list.contains() for list of categoricals (#14744)
  • Fix bug where alias was ignored in COUNT(*) optimization (#14738)
  • Fix DataFrame.sum for decimals (#14732)
  • Fix parallel strategy for LazyFrame not being applied (#14696)
  • Block slice pushdown past non-literal projections or when the projection doesn't contain any columns from the input (#14684)
  • Fix number of rows printed in DataFrame/Series repr (edge cases) (#14548)
  • Fix contention panics in file gc threads (#14690)
  • Fix feature combination (#14688)
  • Only push predicates depending on the subset columns past unique() (#14668)
  • Reading RLE_DICTIONARY-encoded parquet incorrectly coalesced NULL to empty string in some cases (#14670)
  • use correct flooring division/modulo operator in literal optimizer and const_lhs <> series ops (#14671)
  • Enable is_in for string in categorical/enum (#14576)
  • Polars thread pool was not used properly in various functions (#14583)
  • Semi-join and multiple keys outer-join did not respect POLARS_MAX_THREADS (#14571)
  • Correct sorted flag of chunked gather (#14570)
  • ensure the streaming dispatcher can replace placeholders in unions (#14537)
  • Ensure series are contiguous prior to transpose (#14527)
  • write csv header if necessary when finishing sinks (#14518)
  • fix logical dtypes in take_chunked (#14517)
  • fix binary-offset row-encode (#14514)
  • race conditions in OOC writing (#14510)
  • don't gc after variadic buffers are written (#14473)
  • Increase verbosity of duplicate column error message (#11899)
  • Return appropriate data type for duration mean and median (#14376)
  • change print to warn in reading csv from python file like object (#14469)
  • regression in out-of-core group-by by new string-type (#14464)
  • DataFrame.pivot was returning incorrect results when multiple columns were passed to index and one of them was Struct (#14438)
  • remove literal Series from projection state (#14437)
  • pivot was producing incorrect results when (single) index was Struct (#14308)
  • Error on some invalid clip inputs (#14416)
  • Series.hist panicking on empty/all-null (#14407)
  • rechunk series when apply_lambda (#14406)
  • don't make column from filenames, don't ignore directories with (.) (#14317)
  • Remove duplicated content in error messages (#8107)
  • Fix set_operation if the input is sliced and be broadcast (#14303)
  • Wrap par_iter in list.to_struct by POOL.install (#14304)
  • Do not panic when casting from an empty Series to pl.Decimal (#14330)
  • Preserve name when casting to Enum (#14320)
  • list.get does not work on list of decimals (#14276)
  • relax precision when up scaling (#14270)
  • Allow format object series with registry (#14272)
  • deduplicate recursive growables (#14264)
  • Fix glimpse overload signature (#14258)
  • allow set operations on list of categoricals (#14110)
  • any/all_horizontal with single input has incorrect type (#14256)
  • load numpy array with np array values #14237 (#14238)
  • Fix join validation for String types (#14229)
  • make csv parser more robust to edge cases (#14210)
  • Fix for set_operations of binary dtype (#14152)
  • fix read_csv date/datetime inference and parsing (#14113)
  • don't see files as hive partitions (#14128)
  • allow eval on list of categoricals (#14132)
  • add missing conditional compile flag for StringFunction::Find (#14129)
  • Forbid casting from Date to Time and vice versa (#14127)
  • preserve old naming convention for multi-value pivot (this will change in 1.0 to no longer redundantly have the column name in the middle) (#14120)
  • Implements gt/lt cmp for null dtype (#14119)
  • ignore comments at beginning of csv if schema provided (#14115)
  • fix pivot when multiple columns are passed. Output is now aligned with what tidyverse / pandas.pivot_table would do (#14048)
  • some temporal conversion errors for datetimes earlier than 1970-01-01 (#14050)
  • Preserve name when casting from categorical (#14085)
  • fix cse bug when window function is nested (#14070)
  • Fix melt panic when there are no value vars (#14057)
  • json_encode should respect the logical type (#14063)
  • improve skip row group using statistics condition (#14056)
  • Raise for .dt.epoch and .dt.timestamp for Duration dtype (#13962)
  • handle SliceSink with empty data (#14025)
  • correct field type schema inference (using read_csv) (#14042)
  • Map AnyValue::Null to datatype Null (#14045)
  • Use int formatter for unsigned ints (#14043)
  • quick fix for multiple chunks binary reverse (#14024)
  • count matches on list categorical (#14021)
  • list.min/max with empty and/or None elements (#14018)

📖 Documentation

  • Link to plugins tutorial more prominently (#14727)
  • Separate "writing a plugin" from "registering an expression" in user guide, add some extra links, don't use deprecated _register_plugin (#14621)
  • Remove some outdated information in polars crate docs (#14608)
  • Fix code block path for group by example in getting started guide (#14612)
  • Add missing 'string' column in reading-writing Rust example to match Python example (#14597)
  • Fix typo of "Cartesian" product (#14585)
  • Mention in contributing guide that PR titles should start with an uppercase letter (#14584)
  • Fix markdown newline for rendering function description in VSCode (#14567)
  • Clar...
Read more

Python Polars 0.20.13

29 Feb 10:53
becfb5b
Compare
Choose a tag to compare

🚀 Performance improvements

  • Elide utf8/binary cast in Parquet reading (#14757)

🐞 Bug fixes

  • Add missing "pclmulqdq" instruction to _cpu_check ("read_cpu_flags") (#14758)

🛠️ Other improvements

  • Test release wheels on x86-64 (#14761)

Thank you to all our contributors for making this release possible!
@alexander-beedie, @ritchie46 and @stinodego