Skip to content

v0.8.0
Compare
Choose a tag to compare
@philss philss released this 20 Jan 17:43
· 151 commits to main since this release
v0.8.0
e0e242b
This commit was created on github.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.

Added

  • Add explode/2 to Explorer.DataFrame. This function is useful to expand the contents of a {:list, inner_dtype} series into a "inner_dtype" series.

  • Add the new series functions all?/1 and any?/1, to work with boolean series.

  • Add support for the "struct" dtype. This new dtype represents the struct dtype from Polars/Arrow.

  • Add map/2 and map_with/2 to the Explorer.Series module.
    This change enables the usage of the Explore.Query features in a series.

  • Add sort_by/2 and sort_with/2 to the Explorer.Series module.
    This change enables the usage of the lazy computations and the Explorer.Query module.

  • Add unnest/2 to Explorer.DataFrame. It works by taking the fields of a "struct" - the new dtype - and transform them into columns.

  • Add pairwise correlation - Explorer.DataFrame.correlation/2 - to calculate the correlation between numeric columns inside a data frame.

  • Add pairwise covariance - Explorer.DataFrame.covariance/2 - to calculate the covariance between numeric columns inside a data frame.

  • Add support for more integer dtypes. This change introduces new signed and unsigned integer dtypes:

    • {:s, 8}, {:s, 16}, {:s, 32}
    • {:u, 8}, {:u, 16}, {:u, 32}, {:u, 64}.

    The existing :integer dtype is now represented as {:s, 64}, and it's still the default dtype for integers. But series and data frames can now work with the new dtypes. Short names for these new dtypes can be used in functions like Explorer.Series.from_list/2. For example, {:u, 32} can be represented with the atom :u32.

    This may bring more interoperability with Nx, and with Arrow related things, like ADBC and Parquet.

  • Add ewm_standard_deviation/2 and ewm_variance/2 to Explorer.Series.
    They calculate the "exponentially weighted moving" variance and standard deviation.

  • Add support for :skip_rows_after_header option for the CSV reader functions.

  • Support {:list, numeric_dtype} for Explorer.Series.frequencies/1.

  • Support pins in cond, inside the context of Explorer.Query.

  • Introduce the :null dtype. This is a special dtype from Polars and Apache Arrow to represent "all null" series.

  • Add Explorer.DataFrame.transpose/2 to transpose a data frame.

Changed

  • Rename the functions related to sorting/arranging of the Explorer.DataFrame.
    Now arrange_with is named sort_with, and arrange is sort_by.

    The sort_by/3 is a macro and it is going to work using the Explorer.Query module. On the other side, the sort_with/2 uses a callback function.

  • Remove unnecessary casts to {:s, 64} now that we support more integer dtypes.
    It affects some functions, like the following in the Explorer.Series module:

    • argsort
    • count
    • rank
    • day_of_week, day_of_year, week_of_year, month, year, hour, minute, second
    • abs
    • clip
    • lengths
    • slice
    • n_distinct
    • frequencies

    And also some functions from the Explorer.DataFrame module:

    • mutate - mostly because of series changes
    • summarise - mostly because of series changes
    • slice

Fixed

  • Fix inspection of series and data frames between nodes.

  • Fix cast of :string series to {:datetime, any()}

  • Fix mismatched types in Explorer.Series.pow/2, making it more consistent.

  • Normalize sorting options.

  • Fix functions with dtype mismatching the result from Polars.
    This fix is affecting the following functions:

    • quantile/2 in the context of a lazy series
    • mode/1 inside a summarisation
    • strftime/2 in the context of a lazy series
    • mutate_with/2 when creating a column from a NaiveDateTime or Explorer.Duration.

Pull requests

New Contributors

Full Changelog: v0.7.2...v0.8.0
Official Changelog: https://hexdocs.pm/explorer/changelog.html

SHA256 of the artifacts

dda5a178f65d1babb4f58a6db9c9f4a860ba80cd806f2b9dbc1f982b5b4ff3cf  explorer-v0.8.0-nif-2.15-x86_64-pc-windows-gnu--legacy_cpu.dll.tar.gz
5402324ad7bce16e04250c9e84abb48f7d9880f33d984354cc221230da92860e  explorer-v0.8.0-nif-2.15-x86_64-pc-windows-gnu.dll.tar.gz
f1dffe2cca5a2f1dd38c083c837f1a42b82510e387ec0d0a6407ad17589614b7  explorer-v0.8.0-nif-2.15-x86_64-pc-windows-msvc--legacy_cpu.dll.tar.gz
c3bd886e71e42dfea88e86e709b42997ea3d10f5dbeb04fa7d6ab208fa25dcdd  explorer-v0.8.0-nif-2.15-x86_64-pc-windows-msvc.dll.tar.gz
8e75a9b8bac4cf4ef8b996ff1c6ad730496293cffa0e9ef8f00386c79801f8c4  libexplorer-v0.8.0-nif-2.15-aarch64-apple-darwin.so.tar.gz
8fdb0c7cbbce17766543381f0c0521a1861995048223710603c88b1667046ae3  libexplorer-v0.8.0-nif-2.15-aarch64-unknown-linux-gnu.so.tar.gz
86b673bb6313498e21f8d1675910519e2246f65e9e7d17cb476ce71d5370b37c  libexplorer-v0.8.0-nif-2.15-aarch64-unknown-linux-musl.so.tar.gz
4895fc9d84ba23cefe3c560b9a84c119c98936983a5dd6da79325044bc813a28  libexplorer-v0.8.0-nif-2.15-x86_64-apple-darwin.so.tar.gz
dd9b52480e6f6b66e4c5c00e035ceb17d9014f48afd2c812a6631e527db1d6e7  libexplorer-v0.8.0-nif-2.15-x86_64-unknown-freebsd--legacy_cpu.so.tar.gz
a1c6c7de1c48176329e0389e03a38acd441a1ce3ab2327f828a4a9d322c84f1b  libexplorer-v0.8.0-nif-2.15-x86_64-unknown-freebsd.so.tar.gz
bdb29fe7cdf8d9b2642cc79a76ef2397de44b6c375ed0957edce76a7f473f430  libexplorer-v0.8.0-nif-2.15-x86_64-unknown-linux-gnu--legacy_cpu.so.tar.gz
efbcb90526b19a6b4918b248852392c36115405898994de26adfaf2fe6b74558  libexplorer-v0.8.0-nif-2.15-x86_64-unknown-linux-gnu.so.tar.gz
df8be414caeb828e2f6c67dacf33bf2324e15e6b22423c4eb26cede9ff7b63c8  libexplorer-v0.8.0-nif-2.15-x86_64-unknown-linux-musl.so.tar.gz

Contributors

JonGretar, kellyfelkins, and 9 other contributors
Assets 15
Loading