Skip to content

Commit

Permalink
Update for Base.stack, Julia 1.9 (#14)
Browse files Browse the repository at this point in the history
* update for Base.stack

* don't test on 1.3

* tidy
  • Loading branch information
mcabbott authored Sep 2, 2022
1 parent 24c5c42 commit 5d7234f
Show file tree
Hide file tree
Showing 8 changed files with 740 additions and 525 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:
fail-fast: false
matrix:
version:
- '1.3'
- '1.6'
- '1' # Leave this line unchanged. '1' will automatically expand to the latest stable 1.x release of Julia.
- 'nightly'
os:
Expand Down
10 changes: 0 additions & 10 deletions .travis.yml

This file was deleted.

19 changes: 11 additions & 8 deletions Project.toml
Original file line number Diff line number Diff line change
@@ -1,23 +1,26 @@
name = "LazyStack"
uuid = "1fad7336-0346-5a1a-a56f-a06ba010965b"
authors = ["Michael Abbott"]
version = "0.0.8"
version = "0.1.0"

[deps]
ChainRulesCore = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4"
Compat = "34da2185-b29b-5c13-b0c7-acf172513d20"
LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
NamedDims = "356022a1-0364-5f58-8944-0da4b18d706f"
OffsetArrays = "6fe1bfb0-de20-5000-8ca7-80f57d26f881"
# NamedDims = "356022a1-0364-5f58-8944-0da4b18d706f"
# OffsetArrays = "6fe1bfb0-de20-5000-8ca7-80f57d26f881"

[compat]
ChainRulesCore = "0.10.9, 1"
NamedDims = "0.2.16, 0.3, 1.0"
OffsetArrays = "1"
julia = "1.3"
ChainRulesCore = "1"
Compat = "3.46, 4.2"
# NamedDims = "0.2.16, 0.3, 1.0" # try to remove
# OffsetArrays = "1" # try to remove
julia = "1.6"

[extras]
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
OffsetArrays = "6fe1bfb0-de20-5000-8ca7-80f57d26f881"
Zygote = "e88e6eb3-aa80-5325-afca-941959d7151f"

[targets]
test = ["Test", "Zygote"]
test = ["Test", "OffsetArrays", "Zygote"]
73 changes: 39 additions & 34 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,67 +1,72 @@
# LazyStack.jl

[![Travis CI](https://travis-ci.org/mcabbott/LazyStack.jl.svg?branch=master)](https://travis-ci.org/mcabbott/LazyStack.jl)
[![Github CI](https://github.com/mcabbott/LazyStack.jl/workflows/CI/badge.svg)](https://github.com/mcabbott/LazyStack.jl/actions?query=workflow%3ACI+branch%3Amaster)

This package exports one function, `stack`, for turning a list of arrays
This package exports one function, `lazystack`, for turning a list of arrays
into one `AbstractArray`. Given several arrays with the same `eltype`,
or an array of such arrays, it returns a lazy `Stacked{T,N}` view of these:

```julia
stack([zeros(2,2), ones(2,2)]) # isa Stacked{Float64, 3, <:Vector{<:Matrix}}
stack([1,2,3], 4:6) # isa Stacked{Int, 2, <:Tuple{<:Vector, <:UnitRange}}
julia> lazystack([1:2, 3:4, 5:6])
2×3 lazystack(::Vector{UnitRange{Int64}}) with eltype Int64:
1 3 5
2 4 6

julia> lazystack([pi^ℯ], [ℯ^pi])
1×2 lazystack(::Tuple{Vector{Float64}, Vector{Float64}}) with eltype Float64:
22.4592 23.1407
```

Given a generator, it instead iterates through the elements and writes into a new array.
Given a function and then some arrays, it behaves like `map(f, A, B)` but immediately writes
into a new array:
Before v0.1 this function used to be called `stack`, but that name is now exported by Base (from Julia 1.9).
Like this package, `Base.stack` makes an array with `size(result) = (size(inner)..., size(outer)...)`.
It always returns a new dense array, not a lazy container.
And instead of two vectors (in the above example) it would want a tuple `stack(([pi^ℯ], [ℯ^pi]))`.

```julia
stack([i,2i] for i in 1:5) # isa Matrix{Int} # size(ans) == (2, 5)
stack(*, eachcol(ones(2,4)), 1:4) # == Matrix(stack(map(*, eachcol(...), 1:4)))
```

The same `stack_iter` method is also used for any list of arrays of heterogeneous element type,
and for arrays of tuples. Notice that like `map(identity, Any[1, 1.0, 5im])`, this promotes using
`promote_typejoin`, to `Number` here, rather than to `Complex{Float64}`:

```julia
stack([1,2], [3.0, 4.0], [5im, 6im]) # isa Matrix{Number} # size(ans) == (2, 3)
stack([(i,2.0,3//j) for i=1:4, j=1:5])# isa Array{Real, 3} # size(ans) == (3, 4, 5)
```
Generators such as `lazystack([i,2i] for i in 1:5)` and arrays of mixed eltype like `lazystack([1,2], [3.0, 4.0], [5im, 6im])` used to be be handled here, making a dense array, but are now simply passed through to `Base.stack`.

The slices must all have the same `size`, but they (and the container)
can have any number of dimensions. `stack` always places the slice dimensions first.
There are no options.
When the individual slices aren't backed by an `Array`, as for instance with `CuArray`s on a GPU, then again `Base.stack` is called.
This should make one big `CuArray`, since scalar indexing of individual slices won't work well.

### Ragged stack

There is also a version which does not demand that slices have equal `size` (or equal `ndims`),
which always returns a new `Array`. You can control the position of slices `using OffsetArrays`:
There is also a version which does not demand that slices have equal `size` (or equal `ndims`).
For now this is not lazy:

```julia
rstack([1:n for n in 1:10]) # upper triangular Matrix{Int}
rstack(OffsetArray(fill(n,4), rand(-2:2)) for n in 1:10; fill=NaN)
julia> raggedstack([10:10+n for n in 1:3])
4×3 Matrix{Int64}:
10 10 10
11 11 11
0 12 12
0 0 13

julia> using OffsetArrays

julia> raggedstack(OffsetArray(fill(1.0n, 3), rand(-1:1)) for n in 1:10; fill=NaN)
5×10 OffsetArray(::Matrix{Float64}, 0:4, 1:10) with eltype Float64 with indices 0:4×1:10:
NaN 2.0 NaN 4.0 NaN 6.0 7.0 NaN 9.0 NaN
1.0 2.0 3.0 4.0 5.0 6.0 7.0 NaN 9.0 10.0
1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0
1.0 NaN 3.0 NaN 5.0 NaN NaN 8.0 NaN 10.0
NaN NaN NaN NaN NaN NaN NaN 8.0 NaN NaN
```

### Other packages

This one plays well with [OffsetArrays.jl](https://github.com/JuliaArrays/OffsetArrays.jl),
[NamedDims.jl](https://github.com/invenia/NamedDims.jl), and
[Zygote.jl](https://github.com/FluxML/Zygote.jl).
This one plays well with [OffsetArrays.jl](https://github.com/JuliaArrays/OffsetArrays.jl), and [ChainRules.jl](https://github.com/JuliaDiff/ChainRules.jl)-compatible AD such as [Zygote.jl](https://github.com/FluxML/Zygote.jl). It's also used internally by [TensorCast.jl](https://github.com/mcabbott/TensorCast.jl).

Besides which, there are several other ways to achieve similar things:

* For an array of arrays, you can also use [`JuliennedArrays.Align`](https://bramtayl.github.io/JuliennedArrays.jl/latest/#JuliennedArrays.Align). This requires (or enables) you to specify which dimensions of the output belong to the sub-arrays, instead of writing `PermutedDimsArray(stack(...), ...)`.
* There is also [`RecursiveArrayTools.VectorOfArray`](https://github.com/JuliaDiffEq/RecursiveArrayTools.jl#vectorofarray) which as its name hints only allows a one-dimensional container. Linear indexing retreives a slice, not an element, which is sometimes surprising.
* There is also [`RecursiveArrayTools.VectorOfArray`](https://github.com/JuliaDiffEq/RecursiveArrayTools.jl#vectorofarray) which as its name hints only allows a one-dimensional container. (And unlike the package name, nothing is recursive.) Linear indexing retreives a slice, not an element, which is sometimes surprising.
* For a tuple of arrays, [`LazyArrays.Hcat`](https://github.com/JuliaArrays/LazyArrays.jl#concatenation) is at present faster to index than `stack`, but doesn't allow arbitrary dimensions.
* For a generator of arrays, the built-in `reduce(hcat,...)` may work, but it slow compared to `stack`: see [test/speed.jl](test/speed.jl) for some examples.

And a few more:

* When writing this I missed [`SplitApplyCombine.combinedimsview`](https://github.com/JuliaData/SplitApplyCombine.jl#combinedimsviewarray), which is very similar to `stack`, but doesn't handle tuples.
* Newer than this package is [StackViews.jl](https://github.com/JuliaArrays/StackViews.jl) handles both, with `StackView(A,B,dims=4) == StackView([A,B],4)` creating a 4th dimension; the container is always one-dimensional.
* [`Flux.stack`](https://fluxml.ai/Flux.jl/stable/utilities/#Flux.stack) similarly takes a dimension, but eagerly creates an `Array`.
* Finally, [CatViews.jl](https://github.com/ahwillia/CatViews.jl) offers a lazy `vcat`. But the package is old and I think not so fast.

The lazy inverse:

Expand All @@ -71,10 +76,10 @@ The lazy inverse:

* As does [`PackedVectorsOfVectors`](https://github.com/synchronoustechnologies/PackedVectorsOfVectors.jl), although only 1+1 dimensions. Also has an eager `pack` method which turns a vector of vectors into view of a single larger matrix.

* [`Base.eachslice`](https://docs.julialang.org/en/v1/base/arrays/#Base.eachslice) also views one large array as many slices. This is a generator, but [JuliaLang#32310](https://github.com/JuliaLang/julia/pull/32310) should upgrade it to a multi-dimensional container indexable container.
* [`Base.eachslice`](https://docs.julialang.org/en/v1/base/arrays/#Base.eachslice) also views one large array as many slices. This was a generator, but [JuliaLang#32310](https://github.com/JuliaLang/julia/pull/32310) upgrades it to a multi-dimensional indexable container, in Julia 1.9.

Eager:

* After writing this I learned of [JuliaLang#31644](https://github.com/JuliaLang/julia/pull/31644) which extends `reduce(hcat,...)` to work on generators.

* Later, [JuliaLang#31644](https://github.com/JuliaLang/julia/pull/43334) proposes to add the eager `stack_iter` method of this package to Base.
* Later, [JuliaLang#43334](https://github.com/JuliaLang/julia/pull/43334) has added a better version of this package's `stack_iter` method to Base.
Loading

0 comments on commit 5d7234f

Please sign in to comment.