Releases: pola-rs/polars
Python Polars 0.20.17
🏆 Highlights
- Full plan CSE (#15264)
⚠️ Deprecations
- Rename parameter
by
togroup_by
inDataFrame.upsample/group_by_dynamic/rolling
(#14840) - Rename
from_repr
parameter fromtbl
todata
(#15156)
🚀 Performance improvements
- load bits one word at a time for BitmapIter (#15333)
- Ipc exec multiple paths (#15040)
- add SIMD support for if-then-else kernels (#15131)
✨ Enhancements
- Remove 'FileCacher' optimization (#15357)
- check input type in entropy (#15351)
- Implements
arr.n_unique
(#15296) - CSE don't scan share if predicate pushdown predicates don't match (#15328)
- Add
read_database
support forSurrealDB
("ws" and "http") (#15269) - Only allow inputs of type
Sequence
infrom_records
(#15329) - In hypothesis testing strategies, enable Decimal strategy by default (#15321)
- Remove cached nodes when finished (#15310)
- Full plan CSE (#15264)
- More robust handling of
async
database calls (#15202) - Add
name
parameter toGroupBy.len
method (#15235) - Add IR for expressions. (#15168)
- Improve
read_database
when reading from Kùzu graph database (#15218) - Warn if
map_elements
is called withoutreturn_dtype
specified (#15188) - Add support for
async
SQLAlchemy connections toread_database
(#15162) - Infer
time_unit
inpl.duration
when nanoseconds is specified (#14987) - Add
strict
parameter tofrom_dict/from_records
(#15158)
🐞 Bug fixes
- Return correct dtype for
s.clear()
when dtype isObject
(#15315) - ensure first datapoint is always included in group_by_dynamic (#15312)
- Non-exhaustive patterns: arrow-schema::DataType in polars-arrow (#15250)
- use dynamic stacks for problematic recursive functions (#15355)
- Adding default ddof for
Series.list.std
andSeries.list.var
(#15267) - Raise properly for slices not supported by
LazyFrame
(#15331) - Propagate strictness in
from_dicts
(#15344) - Raise error when
schema_overrides
contains nonexistent column name (#15290) - Enforce integer
dtype
input forint_range
andint_ranges
(#15339) - Preserve Decimal precision when constructing empty Series (#15320)
- Fix cache dot visualization (#15311)
- Handle special case correctly when slicing a
LazyFrame
(#15297) - Properly propagate
strict
flag when constructing a Struct Series from any values (#15302) - Consistent expansion of nested struct data during
DataFrame
init from dict (#15217) - Raise when join projects name with suffix that doesn't exist (#15256)
- Ensure ooc_start is set (#15255)
- Fix bug where rolling operations were ignoring
check_sorted
in some cases (#15227) - Fix lazy schema for
rle
expression (#15248) - incorrect negative offset in multi-byte string slicing (#15140)
- do not clamp negative offsets to start of array prematurely (#15242)
- allow null index in list.get and array.get (#15239)
- Avoid loading all columns in
read_parquet
whencolumns
parameter is specified (#15229) - properly support nulls_last + descending (#15212)
- fix nested runtime panic (#15216)
- Block rounding/truncating to negative durations (#15175)
- Ensure the
cs.temporal()
selector uses wildcard time zone matching forDatetime
(#13683) - Consistently raise
TypeError
on constructor failure (#15178) - Properly propagate strictness in some constructor cases (#15166)
- Fix constructing a Series from a list of Series with given dtype (#15144)
📖 Documentation
- Fix time unit in
timestamp
example (#15281) - Fix link to renamed method (.list.lengths -> .list.len) (#15228)
- Update Excel and database pages in user guide (#14721)
- Add examples for
Series.search_sorted
(#14737) - Correct the definition of an expression in the user guide (#14750)
- Add a note about the behaviour of lower/upper bounds for
is_between
, and add an example (#15197)
📦 Build system
- Update Cargo lock (#15370)
🛠️ Other improvements
- Memory usage test infrastructure, plus a test for #15098 (#15285)
- Update CODEOWNERS (#15352)
- remove try_apply_values (#15336)
- always use non-legacy float_sum for mean (#15343)
- remove legacy bitmap module (#15335)
- Fix test not writing to temporary directory (#15318)
- Reorganize tests for
clear
operation (#15304) - Rename
Cache[count]
toCache[cache_hits]
(#15300) - Cleanup file_caching optimization call (#15299)
- Minor refactor of
PyDataFrame.from_dicts
(#15274) - remove 'fast-projection' node (#15253)
- cleanup column names in optimizer (#15252)
- remove left_most_input_name from expr ir (#15251)
- add AlignedBitmapSlice (#15171)
- Run non-benchmark tests in benchmark workflow (#15207)
- Add
wrapping_abs
to arithmetic kernel (#15210) - remove raw buffers from BinViewArray (#15206)
- Enable
RUST_BACKTRACE=1
in the CI test suite (#15204) - Split
read_database
functionality into cleaner module structure (#15201) - Clean up some of the AnyValue conversion logic (#15190)
- remove parts of legacy bit_util (#15169)
- remove legacy arrow compute (#15164)
- Split up
dataframe
module in PyO3 bindings (#15165) - Remove unused private constructors (#15160)
Thank you to all our contributors for making this release possible!
@MarcoGorelli, @alexander-beedie, @braaannigan, @c-peters, @cojmeister, @deanm0000, @dependabot, @dependabot[bot], @itamarst, @kszlim, @mbuhidar, @mcrumiller, @mickvangelderen, @orlp, @petrosbar, @reswqa, @ritchie46, @rob-sil, @sportfloh, @stinodego and @yutannihilation
Rust Polars 0.38.3
🚀 Performance improvements
- add new when-then-otherwise kernels (#15089)
- Coerce sorted flag of unit arrays during concat (#15104)
- Use sorted flag for
(first|last)_non_null
(#15050) - OOC sort improvements (#14994)
✨ Enhancements
- raise if both
closed
andby
are passed torolling_*
aggregations (#15108) - raise informative error for rolling_* aggs with
by
of invalid dtype (#15088) - add
non_existent
arg toreplace_time_zone
(#15062) - Support single nested row encodings (#15105)
- make ooc sort configurable (#15084)
- Make
register_plugin
a standalone function and include shared lib discovery (#14804) - Async parquet: Decode parquet on a blocking thread pool (#15083)
- let "ambiguous" take "null" value (#14961)
- Raise informative error message when join would introduce duplicate column name (#15042)
- Allow cast of decimal to boolean (#15015)
- Return error when no supertype can be determined in AnyValue constructor when
strict=false
(#15025) - Implement IpcReaderAsync (#14984)
- Support Array statistics in parquet (#15031)
- Support decimal groupby (#15000)
- Add thread names to rayon thread pool (#15024)
- Support decimal uniq (#15001)
- expose timings in verbose state of OOC sort (#14979)
🐞 Bug fixes
- Fix Series construction from nested list with mixed data types (#15046)
- Support BinaryView in row decoder to prevent a panic in streaming group by (#15117)
- Binview chunked gather; don't modify inlined view (#15124)
- Fix chunked_id gather for binview buffers (#15123)
- Don't cache HTTP object stores as they maintain URL state (#15121)
- use wrapping_add in csv line snooping (#15109)
- Output
u32
whensum_horizontal
provided with single boolean column (#15114) - Ensure
eprintln!
is only called within debug/verbose context (#15100) - Propagate error instead of panicking when calling
product
on an invalid type (#15093) - Raise error when casting Array to different width (#14995)
- Fix file scan bugs for ipc, csv and parquet that occur with combinations of glob paths, row indices and predicates (#15065)
- Incorrectly preserved sorted flag when concatenating sorted series containing nulls (#15082)
- Return largest non-NaN value for
max()
on sorted float arrays if it exists instead of NaN (#15060) - return NaN for all-NaN min/max (#15066)
- Prevent "index out of range for slice" error in parquet reader (#15021)
- Respect
nulls_last
in streaming sort (#15061) - Fix Series construction from nested list with mixed data types (#15046)
- Don't count nulls in streaming
count
agg (#15051) - agg_list on decimal lost scale (#15054)
- Block predicate pushdown on equality that are use in join (#15055)
- Enum equality based on categories (#15053)
- Strict cast in when/then/otherwise operation (#15052)
- Don't panic in
string_addition_to_linear_concat
(#15006) - CSV do utf8-validation after escaping fields (#15004)
- Use primitive constructors to create a Series of lists when dtype is provided (#15002)
- replace_time_zone with single-null-element "ambiguous" was panicking (#14971)
📖 Documentation
- Fix typo in comment (#14997)
🛠️ Other improvements
- Extend and speed up scan tests (#15127)
- always assert on ChunkedArray::get (#15120)
- Use ObjectStore instead of AsyncRead in parquet get metadata (#15069)
- Minor refactor of Rust any value constructors (#15077)
- Simplify streaming execution (#15039)
- Ensure we hit the spilled source path in ooc sort test (#15010)
- Refactor constructor code (#15009)
- Apply
clippy:assigning_clones
lint (#14999) - fix features (#14977)
Thank you to all our contributors for making this release possible!
@JackRolfe, @MKisilyov, @MarcoGorelli, @alexander-beedie, @c-peters, @flisky, @jqnatividad, @mcrumiller, @mickvangelderen, @nameexhaustion, @orlp, @petrosbar, @ritchie46, @stinodego and @trueb2
Python Polars 0.20.16
🚀 Performance improvements
- add new when-then-otherwise kernels (#15089)
- Coerce sorted flag of unit arrays during concat (#15104)
- Use sorted flag for
(first|last)_non_null
(#15050) - OOC sort improvements (#14994)
✨ Enhancements
- improved dtype inference/refinement for
read_database
results (#15126) - raise if both
closed
andby
are passed torolling_*
aggregations (#15108) - raise informative error for rolling_* aggs with
by
of invalid dtype (#15088) - add
non_existent
arg toreplace_time_zone
(#15062) - Support single nested row encodings (#15105)
- make ooc sort configurable (#15084)
- Make
register_plugin
a standalone function and include shared lib discovery (#14804) - Expose
infer_schema_length
parameter onread_database
(#15076) - Async parquet: Decode parquet on a blocking thread pool (#15083)
- let "ambiguous" take "null" value (#14961)
- Raise informative error message when join would introduce duplicate column name (#15042)
- Allow cast of decimal to boolean (#15015)
- Add
strict
parameter toDataFrame
constructor to allow non-strict construction (#15034) - Support Array statistics in parquet (#15031)
- Support decimal groupby (#15000)
- Add thread names to rayon thread pool (#15024)
- Support decimal uniq (#15001)
- expose timings in verbose state of OOC sort (#14979)
🐞 Bug fixes
- Support BinaryView in row decoder to prevent a panic in streaming group by (#15117)
- Binview chunked gather; don't modify inlined view (#15124)
- Fix chunked_id gather for binview buffers (#15123)
- Don't cache HTTP object stores as they maintain URL state (#15121)
- Output
u32
whensum_horizontal
provided with single boolean column (#15114) - Propagate error instead of panicking when calling
product
on an invalid type (#15093) - Raise error when casting Array to different width (#14995)
- Fix file scan bugs for ipc, csv and parquet that occur with combinations of glob paths, row indices and predicates (#15065)
- Incorrectly preserved sorted flag when concatenating sorted series containing nulls (#15082)
- Return largest non-NaN value for
max()
on sorted float arrays if it exists instead of NaN (#15060) - return NaN for all-NaN min/max (#15066)
- Prevent "index out of range for slice" error in parquet reader (#15021)
- Respect
nulls_last
in streaming sort (#15061) - Fix Series construction from nested list with mixed data types (#15046)
- Don't count nulls in streaming
count
agg (#15051) - agg_list on decimal lost scale (#15054)
- Block predicate pushdown on equality that are use in join (#15055)
- Enum equality based on categories (#15053)
- Don't panic in
string_addition_to_linear_concat
(#15006) - CSV do utf8-validation after escaping fields (#15004)
- Use primitive constructors to create a Series of lists when dtype is provided (#15002)
- replace_time_zone with single-null-element "ambiguous" was panicking (#14971)
📖 Documentation
- Update write_database code blocks in user guide (#15106)
- Add missing docstring examples in the Struct namespace (#15071)
- Improve API reference landing page (#14888)
- improve join_asof example (#14993)
- Fix inadvertent swap of
new
andold
parameters inreplace
description (#15019)
🛠️ Other improvements
- Extend and speed up scan tests (#15127)
- Add parameterized-scan-tests (#15057)
- Simplify streaming execution (#15039)
- Ensure we hit the spilled source path in ooc sort test (#15010)
- Refactor constructor code (#15009)
- fix features (#14977)
- Revert pinning PyPI publish action (#14975)
Thank you to all our contributors for making this release possible!
@JackRolfe, @MKisilyov, @MarcoGorelli, @alexander-beedie, @c-peters, @flisky, @jqnatividad, @mcrumiller, @mickvangelderen, @nameexhaustion, @orlp, @petrosbar, @ritchie46, @stinodego and @trueb2
Python Polars 0.20.16-rc.1
🚀 Performance improvements
✨ Enhancements
- let "ambiguous" take "null" value (#14961)
- Raise informative error message when join would introduce duplicate column name (#15042)
- Allow cast of decimal to boolean (#15015)
- Add
strict
parameter toDataFrame
constructor to allow non-strict construction (#15034) - Support Array statistics in parquet (#15031)
- Support decimal groupby (#15000)
- Add thread names to rayon thread pool (#15024)
- Support decimal uniq (#15001)
- expose timings in verbose state of OOC sort (#14979)
🐞 Bug fixes
- Prevent "index out of range for slice" error in parquet reader (#15021)
- Respect
nulls_last
in streaming sort (#15061) - Fix Series construction from nested list with mixed data types (#15046)
- Don't count nulls in streaming
count
agg (#15051) - agg_list on decimal lost scale (#15054)
- Block predicate pushdown on equality that are use in join (#15055)
- Enum equality based on categories (#15053)
- Don't panic in
string_addition_to_linear_concat
(#15006) - CSV do utf8-validation after escaping fields (#15004)
- Use primitive constructors to create a Series of lists when dtype is provided (#15002)
- replace_time_zone with single-null-element "ambiguous" was panicking (#14971)
📖 Documentation
- Improve API reference landing page (#14888)
- improve join_asof example (#14993)
- Fix inadvertent swap of
new
andold
parameters inreplace
description (#15019)
🛠️ Other improvements
- Add parameterized-scan-tests (#15057)
- Simplify streaming execution (#15039)
- Ensure we hit the spilled source path in ooc sort test (#15010)
- Refactor constructor code (#15009)
- fix features (#14977)
- Revert pinning PyPI publish action (#14975)
Thank you to all our contributors for making this release possible!
@MKisilyov, @MarcoGorelli, @alexander-beedie, @c-peters, @flisky, @jqnatividad, @mcrumiller, @mickvangelderen, @nameexhaustion, @petrosbar, @ritchie46, @stinodego and @trueb2
Rust Polars 0.38.2
🏆 Highlights
- Streaming outer joins (#14828)
🚀 Performance improvements
- Ensure parallel encoding/compression in
sink_parquet
(#14964) - hoist errors out of iterators in parquet (#14945)
- add basic AVX-512 filters (#14892)
- improve join-asof materialization (#14884)
- Optimize chunked-id gather for binaryviews (#14878)
- rework scalar filter kernels (#14865)
- Reduce size of optional join-indexes (#14856)
- Streaming outer joins (#14828)
- Set sorted flag for
cum_count
on columns (#14849)
✨ Enhancements
- Support writing
Array
type in parquet (#14943) - Sort decimal fields (#14649)
- Import
NamedFrom
indf!
macro (#14860) - try-improve concurrency tuner (#14827)
- Streaming outer joins (#14828)
- Set sorted flag for
cum_count
on columns (#14849) - Ensure binview types are rle-encoded in parquet write (#14818)
- Implement strict/nonstrict conversion for primitive AnyValues (#14186)
- Disable timeouts (#14809)
- cleanup spill disks in process (#14807)
🐞 Bug fixes
- Fix invalid partitionable query (#14966)
- allow nonstrict cast of categorical/enum to enum (#14910)
count_rows
multi-threaded under-counting in parser.rs (#14963)- raise proper error instead of panicking when result of truncation is non-existent datetime (#14958)
- ooc-sort issues (#14959)
- Do not raise when constructing from a list of Series with Nones (#14942)
- Don't access out-of-bounds for null indices in bitmap gather (#14932)
- std when ddof>=n_values returns None even in rolling context (#11750)
- Don't rechunk categoricals when moving to physical (#14934)
- parquet rle boolean decoder (#14931)
- boolean filter gave overly large buffers to Bitmap::from_u8_vec (#14924)
- Fix sliced dictionary state in parquet (#14917)
- Fix possibly incorrect order of columns when using ipc stream
with_columns
(#14859) - Fully qualify
polars_bail!
inpolars_ensure!
(#14901) - Fix
DataFrame.min
/max
for decimals (#14890) - Assert chunks are equal after physical cast to prevent OOB (#14873)
- not all cpu feature flag tests were mocked (#14864)
📖 Documentation
- Remove some repetition in comments/docstrings (#14912)
- Update contributing link (#14882)
- Fix some word-repetition in code comments (#14825)
- Seperate
asof
from join strategy, change parameter fromstrategy
tohow
in user guide (#14793)
🛠️ Other improvements
- fix features (#14977)
- fix chrono deprecation warnings (#14928)
- Update Cargo.lock and remove cmake limit workaround (#14905)
- Simplify streaming placeholder replacement. (#14915)
- Optional deps should include
fastexcel
(#14907) - Deduplicate
POLARS_FORCE_ASYNC
env var parsing (#14909) - Make assumption about column name to index conversion having occurred explicit (#14894)
- Make assumption about wildcards having been resolved explicit (#14899)
- reactivate argminmax simd (#14679)
- sort by 'idx' after outer join (#14867)
- Simplify computation of
with_columns
attribute in physical csv scanner of default engine. (#14837) - centrally define IdxSize (#14854)
- run and fix pext64_polyfill test (#14852)
- introduce partitioned table (#14819)
- add missing deprecation directive in groupby.count (#14817)
- Extract key value construction (#14812)
- Fix Makefile build commands (#14806)
Thank you to all our contributors for making this release possible!
@MarcoGorelli, @Sol-Hee, @alexander-beedie, @ambidextrous, @battmdpkq, @deanm0000, @dependabot, @dependabot[bot], @eitsupi, @flisky, @geekvest, @mcrumiller, @mickvangelderen, @nameexhaustion, @orlp, @petrosbar, @ritchie46 and @stinodego
Python Polars 0.20.15
🚀 Performance improvements
- Ensure parallel encoding/compression in
sink_parquet
(#14964) - hoist errors out of iterators in parquet (#14945)
- add basic AVX-512 filters (#14892)
✨ Enhancements
- Support writing
Array
type in parquet (#14943) - Add
drop_first
parameter toSeries.to_dummies
(#14846) - Add "execute_options" support for
read_database_uri
(#14682)
🐞 Bug fixes
- Fix invalid paritionable query (#14966)
- allow nonstrict cast of categorical/enum to enum (#14910)
count_rows
multi-threaded under-counting in parser.rs (#14963)- raise proper error instead of panicking when result of truncation is non-existent datetime (#14958)
- ooc-sort issues (#14959)
- Do not raise when constructing from a list of Series with Nones (#14942)
- Don't access out-of-bounds for null indices in bitmap gather (#14932)
- std when ddof>=n_values returns None even in rolling context (#11750)
- Don't rechunk categoricals when moving to physical (#14934)
- Ensure consistent
read_database
behaviour with empty ODBC "iter_batches" (#14918) - parquet rle boolean decoder (#14931)
- Fix frame init from single
RecordBatch
objects whenpyarrow <= 12
(#14922) - boolean filter gave overly large buffers to Bitmap::from_u8_vec (#14924)
- Fix sliced dictionary state in parquet (#14917)
read_database
now properly handles empty result sets fromarrow-odbc
(#14916)- Fix possibly incorrect order of columns when using ipc stream
with_columns
(#14859)
📖 Documentation
- Add note about
include_index
infrom_pandas
regarding "default indices" (#14920) - Remove some repetition in comments/docstrings (#14912)
🛠️ Other improvements
- Update Cargo.lock and remove cmake limit workaround (#14905)
- Simplify streaming placeholder replacement. (#14915)
- Optional deps should include
fastexcel
(#14907) - Deduplicate
POLARS_FORCE_ASYNC
env var parsing (#14909)
Thank you to all our contributors for making this release possible!
@MarcoGorelli, @alexander-beedie, @ambidextrous, @battmdpkq, @mcrumiller, @mickvangelderen, @orlp, @petrosbar and @ritchie46
Python Polars 0.20.14
🏆 Highlights
- Streaming outer joins (#14828)
⚠️ Deprecations
- Deprecate
overwrite_schema
parameter forDataFrame.write_delta
(#14879)
🚀 Performance improvements
- improve join-asof materialization (#14884)
- Optimize chunked-id gather for binaryviews (#14878)
- rework scalar filter kernels (#14865)
- Reduce size of optional join-indexes (#14856)
- Streaming outer joins (#14828)
- Set sorted flag for
cum_count
on columns (#14849)
✨ Enhancements
- Sort decimal fields (#14649)
- Revert addition of
__slots__
to Polars classes (#14857) - Add
fastexcel
toshow_versions
(#14869) - try-improve concurrency tuner (#14827)
- Streaming outer joins (#14828)
- Set sorted flag for
cum_count
on columns (#14849) - support use of KùzuDB via
pl.read_database
(#14822) - Ensure binview types are rle-encoded in parquet write (#14818)
- Disable timeouts (#14809)
- cleanup spill disks in process (#14807)
- Implement compression and skipping for binview IPC (#14789)
🐞 Bug fixes
- Fix
DataFrame.min
/max
for decimals (#14890) - Assert chunks are equal after physical cast to prevent OOB (#14873)
- not all cpu feature flag tests were mocked (#14864)
- Remove custom
__reduce__
implementation onDataType
object (#14778) - Allow non-strict construction / initialization of Enum columns (#14728)
- Fix streaming parquet limit (#14783)
📖 Documentation
- Update contributing link (#14882)
- update to use
ambiguous
instead ofuse_earliest
(#14820) - Seperate
asof
from join strategy, change parameter fromstrategy
tohow
in user guide (#14793)
🛠️ Other improvements
- Pin PyPI publish action to commit (#14896)
- reactivate argminmax simd (#14679)
- sort by 'idx' after outer join (#14867)
- run and fix pext64_polyfill test (#14852)
- add missing deprecation directive in groupby.count (#14817)
- Fix Makefile build commands (#14806)
- Bump ruff from 0.2.0 to 0.3.0 in /py-polars (#14800)
- Rename
utils
module to_utils
to explicitly mark it as private (#14772) - Add test coverage for
_cpu_check
module (#14768)
Thank you to all our contributors for making this release possible!
@MarcoGorelli, @Sol-Hee, @alexander-beedie, @c-peters, @deanm0000, @dependabot, @dependabot[bot], @eitsupi, @flisky, @geekvest, @mcrumiller, @mickvangelderen, @nameexhaustion, @orlp, @petrosbar, @ritchie46 and @stinodego
Rust Polars 0.38.1
🚀 Performance improvements
- Elide utf8/binary cast in Parquet reading (#14757)
✨ Enhancements
- Implement compression and skipping for binview IPC (#14789)
🐞 Bug fixes
- fix feature flags (#14802)
- Allow non-strict construction / initialization of Enum columns (#14728)
- Fix streaming parquet limit (#14783)
📦 Build system
- bump rayon from 1.8.1 to 1.9.0 (#14797)
Thank you to all our contributors for making this release possible!
@alexander-beedie, @c-peters, @dependabot, @dependabot[bot], @ritchie46 and @stinodego
Rust Polars 0.38.0
🏆 Highlights
💥 Breaking changes
- Infer
values
columns inDataFrame.pivot
whenvalues
is None (#14477) - Mark
DataFrame::new_no_checks
andDataFrame::new_no_length_checks
unsafe (#14443) - Remove
DatetimeChunked::convert_time_zone
(#14046) - Rename
LiteralValue::to_anyvalue
toLiteralValue::to_any_value
(#14033)
🚀 Performance improvements
- auto-tune concurrency budget (#14753)
- Don't materialize for broadcasting
fill_null
value and default value ofreplace
(#14736) - Improve performance of boolean filters
1-100x
. (#14746) - fix accidental quadratic utf8 validation in parquet (#14705)
- fast path for COUNT(*) queries (#14574)
- Elide the total order wrapper for non-(float/option) types (#14648)
- add utf8-validation fast paths for utf8view (#14644)
- don't reassign chunks back to df owner (#14633)
- If there are many small chunks in write_parquet(), convert to a single chunk (#14484) (#14487)
- Polars thread pool was not used properly in various functions (#14583)
- use owned arithmetic in horizontal_sum (#14525)
- Combine small chunks in sinks for streaming pipelines (#14346)
- reduce heap allocs in expression/logical-plan iteration (#14440)
- simplify and speed up cum_sum and cum_prod (#14409)
- simplify negated predicates to improve row groups skipping (#14370)
- prune parquet row groups when
is_not_null
is used (#14260) - use is_between to skip parquet row groups (#14244)
- Use a compression API that is designed for this use case (#11699) (#14194)
- Use
UnitVec
in polars-plan traversal (#14199) - use
UnitVec
in streaming joins (#14197) - improve
ChunkId
(#14175) - improve iteration performance (#14126)
- elide unneeded work in window? (#14108)
- run window functions more in parallel (#14095)
- improve skip row group using statistics condition (#14056)
✨ Enhancements
- Change default for maximum number of Series items printed to 10 to match DataFrame (#14703)
- Infer
values
columns inDataFrame.pivot
whenvalues
is None (#14477) - fast path for COUNT(*) queries (#14574)
- let
rolling
acceptindex_column
of type UInt32 or UInt64 (#14669) - Treat float -0.0 == 0.0 and -NaN == NaN in group-by, joins and unique (#14617)
- Properly cache object-stores (#14598)
- Mark
DataFrame::new_no_checks
andDataFrame::new_no_length_checks
unsafe (#14443) - flatten aliases (#14512)
- Make formatting more consistent in DOT graphs (#14486)
- add
flush
operator to streaming operators (#14500) - Increase verbosity of duplicate column error message (#11899)
- change print to warn in reading csv from python file like object (#14469)
- Raise if
pivot
would introduce duplicate column names (#14431) - apply negate in simplify expression pass (#14436)
- restrict more cloud interop to semaphore budget (#14435)
- Implement
min
/max
for categorical dtype (#14112) - add boolean rle decoding for parquet (#14403)
- Allow brackets in SQL join conditions (#14263)
- Improve panic message for missing struct feature in
DataType::from_arrow
(#14392) - Implement the
IntoLazy
trait for LazyFrame (#14323) - Implemented tree formatting for LogicalPlan (#14221)
- Implement
mean_horizontal
expression (#14369) - support decimal comparison (#14338)
- Implements
arr.shift
(#14298) - Implements
list.n_unique
(#14306) - Do not panic when casting from an empty Series to pl.Decimal (#14330)
- add
u8
/i8
/u16
/i16
parsers to CSV reader (#14241) - Implements
list.gather_every
(#14253) - Implements
prefix/suffix_fields
(#14251) - Polish decimal arithmetic (#14172)
- Introduce
arr.to_struct
(#14202) - Supports map fields name of struct (#14203)
- make
IdxVec
generic asUnitVec
(#14196) - add new arithmetic kernels (#14026)
- Supports
unique
andhash_rows
fornull
column (#14111) - Implement arithmetic operations for
Null
columns (#14107) - Add strict/non-strict construction of Boolean/Binary series (#14073)
- Improve
Series::from_any_values
logic (#14052) - Adapt extend_constant to function expr architecture and expressify it (#14058)
- add integer negation (#14049)
list
&array
measures of dispersion (#13245)- gc binview when writing ipc (#14035)
- When calling
convert_time_zone
on time-zone-naive datetime, convert as if converting from UTC (#13960)
🐞 Bug fixes
- fix hashing specialization (#14754)
- Sum after filter in aggregation context sometimes returned NULL (#14752)
- Allow
list.contains()
for list of categoricals (#14744) - Fix bug where alias was ignored in COUNT(*) optimization (#14738)
- Fix
DataFrame.sum
for decimals (#14732) - Fix parallel strategy for LazyFrame not being applied (#14696)
- Block slice pushdown past non-literal projections or when the projection doesn't contain any columns from the input (#14684)
- Fix number of rows printed in
DataFrame/Series
repr (edge cases) (#14548) - Fix contention panics in file gc threads (#14690)
- Fix feature combination (#14688)
- Only push predicates depending on the subset columns past
unique()
(#14668) - Reading RLE_DICTIONARY-encoded parquet incorrectly coalesced NULL to empty string in some cases (#14670)
- use correct flooring division/modulo operator in literal optimizer and const_lhs <> series ops (#14671)
- Enable
is_in
for string in categorical/enum (#14576) - Polars thread pool was not used properly in various functions (#14583)
- Semi-join and multiple keys outer-join did not respect POLARS_MAX_THREADS (#14571)
- Correct sorted flag of chunked gather (#14570)
- ensure the streaming dispatcher can replace placeholders in unions (#14537)
- Ensure series are contiguous prior to
transpose
(#14527) - write csv header if necessary when finishing sinks (#14518)
- fix logical dtypes in take_chunked (#14517)
- fix binary-offset row-encode (#14514)
- race conditions in OOC writing (#14510)
- don't gc after variadic buffers are written (#14473)
- Increase verbosity of duplicate column error message (#11899)
- Return appropriate data type for duration
mean
andmedian
(#14376) - change print to warn in reading csv from python file like object (#14469)
- regression in out-of-core group-by by new string-type (#14464)
- DataFrame.pivot was returning incorrect results when multiple columns were passed to
index
and one of them was Struct (#14438) - remove literal
Series
from projection state (#14437) - pivot was producing incorrect results when (single)
index
was Struct (#14308) - Error on some invalid
clip
inputs (#14416) - Series.hist panicking on empty/all-null (#14407)
- rechunk series when apply_lambda (#14406)
- don't make column from filenames, don't ignore directories with (.) (#14317)
- Remove duplicated content in error messages (#8107)
- Fix
set_operation
if the input is sliced and be broadcast (#14303) - Wrap
par_iter
inlist.to_struct
byPOOL.install
(#14304) - Do not panic when casting from an empty Series to pl.Decimal (#14330)
- Preserve name when casting to Enum (#14320)
list.get
does not work on list of decimals (#14276)- relax precision when up scaling (#14270)
- Allow format object series with registry (#14272)
- deduplicate recursive growables (#14264)
- Fix
glimpse
overload signature (#14258) - allow set operations on list of categoricals (#14110)
any/all_horizontal
with single input has incorrect type (#14256)- load numpy array with np array values #14237 (#14238)
- Fix join validation for String types (#14229)
- make csv parser more robust to edge cases (#14210)
- Fix for
set_operations
of binary dtype (#14152) - fix read_csv date/datetime inference and parsing (#14113)
- don't see files as hive partitions (#14128)
- allow eval on list of categoricals (#14132)
- add missing conditional compile flag for
StringFunction::Find
(#14129) - Forbid casting from
Date
toTime
and vice versa (#14127) - preserve old naming convention for multi-value pivot (this will change in 1.0 to no longer redundantly have the column name in the middle) (#14120)
- Implements
gt/lt
cmp for null dtype (#14119) - ignore comments at beginning of csv if schema provided (#14115)
- fix pivot when multiple columns are passed. Output is now aligned with what tidyverse / pandas.pivot_table would do (#14048)
- some temporal conversion errors for datetimes earlier than
1970-01-01
(#14050) - Preserve name when casting from categorical (#14085)
- fix cse bug when window function is nested (#14070)
- Fix
melt
panic when there are no value vars (#14057) json_encode
should respect the logical type (#14063)- improve skip row group using statistics condition (#14056)
- Raise for .dt.epoch and .dt.timestamp for Duration dtype (#13962)
- handle
SliceSink
with empty data (#14025) - correct field type schema inference (using read_csv) (#14042)
- Map
AnyValue::Null
to datatypeNull
(#14045) - Use int formatter for unsigned ints (#14043)
- quick fix for multiple chunks binary reverse (#14024)
- count matches on list categorical (#14021)
list.min/max
with empty and/or None elements (#14018)
📖 Documentation
- Link to plugins tutorial more prominently (#14727)
- Separate "writing a plugin" from "registering an expression" in user guide, add some extra links, don't use deprecated _register_plugin (#14621)
- Remove some outdated information in
polars
crate docs (#14608) - Fix code block path for group by example in getting started guide (#14612)
- Add missing 'string' column in reading-writing Rust example to match Python example (#14597)
- Fix typo of "Cartesian" product (#14585)
- Mention in contributing guide that PR titles should start with an uppercase letter (#14584)
- Fix markdown newline for rendering function description in VSCode (#14567)
- Clar...
Python Polars 0.20.13
🚀 Performance improvements
- Elide utf8/binary cast in Parquet reading (#14757)
🐞 Bug fixes
- Add missing "pclmulqdq" instruction to
_cpu_check
("read_cpu_flags") (#14758)
🛠️ Other improvements
- Test release wheels on x86-64 (#14761)
Thank you to all our contributors for making this release possible!
@alexander-beedie, @ritchie46 and @stinodego