Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC/WIP: Support array type as input for functions returning AbstractArray instance #16740

Closed
wants to merge 12 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
173 changes: 167 additions & 6 deletions base/abstractarray.jl
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,34 @@ _length(A) = length(A)
endof(a::AbstractArray) = length(a)
first(a::AbstractArray) = a[first(eachindex(a))]

function fixate_eltype{T<:AbstractArray}(::Type{T}, default::Type)
@_pure_meta
aa_type = T
while aa_type.name.primary != AbstractArray
aa_type = supertype(aa_type)
end
if isa(aa_type.parameters[1], TypeVar)
varname = aa_type.parameters[1].name
new_params = collect(T.parameters)
for i in eachindex(new_params)
p = new_params[i]
if isa(p, TypeVar) && p.name == varname
new_params[i] = default
end
end
return T.name.primary{new_params...}
else
return T
end
end
Copy link
Contributor

@pabloferz pabloferz Nov 9, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love the idea. Maybe we can do it even more general? E.g.

function set_eltype{A<:AbstractArray}(::Type{A}, T::Type)
    Base.@_pure_meta
    AA = A
    AT = AA.name.primary
    while AT.name.primary != AbstractArray
        AA = supertype(AA)
        AT = supertype(AT)
    end
    if AA.parameters[1] !== T
        tvarname = AT.parameters[1].name
        params = A.parameters
        new_params = collect(A.name.primary.parameters)
        for i in eachindex(new_params)
            p = new_params[i]
            if isa(p, TypeVar) && p.name == tvarname
                new_params[i] = T
            else
                new_params[i] = params[i]
            end
        end
        return A.name.primary{new_params...}
    else
        return A
    end
end

so it also gives set_type(Diagonal{Int}, Float64) === Diagonal{Float64})

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I deliberately wanted fixate_eltype(Diagonal{Int}, Float64) === Diagonal{Int} to be able to specify a default element type. E.g. zeros(Diagonal, 2, 2) should give a Diagonal{Float64}, but zeros(Diagonal{Int}, 2, 2) should give a Diagonal{Int}. Maybe a third, boolean argument could be used to choose between "set always" and "set only if unset".

Copy link
Contributor

@pabloferz pabloferz Nov 10, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. I had in mind another use case, using something like this combined with something like deepcopy_internal to write a generic similar(::AbstractArray, ::Type). Handling the dimensions in a generic fashion might not be possible and we would still need specialization (as you have here).

With this in mind and what is needed here, a third boolean argument would seem like the way to cover both scenarios.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function is fundamentally invalid. T.name.primary isn't a meaningful type and its type parameters don't have any fixed relation to the type parameters of AbtractArray. If you can't express a type relationship via dispatch, that is highly indicative that the relation you want is not valid.

I think you want to call the similar function, whose definition is similar to what it appears the function above was attempting to do.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it could be something like

function set_eltype{A<:AbstractArray}(::Type{A}, T::Type, force=false)
    Base.@_pure_meta
    SA = A
    AA = A.name.primary
    while SA.name.primary != AbstractArray
        SA = supertype(SA)
        force && (AA = supertype(AA))
    end
    if (!force && !isa(SA.parameters[1], TypeVar)) ||
       (force && SA.parameters[1] === T)
        return A
    end
    varname = (force ? AA : SA).parameters[1].name
    params = A.parameters
    new_params = collect(force ? A.name.primary.parameters : params)
    for i in eachindex(new_params)
        p = new_params[i]
        if isa(p, TypeVar) && p.name == varname
            new_params[i] = T
        elseif force
            new_params[i] = params[i]
        end
    end
    return A.name.primary{new_params...}
end

This would give set_eltype(Diagonal, Float64) === Diagonal{Float64}, set_eltype(Diagonal{Int}, Float64) === Diagonal{Int}, set_eltype(Diagonal{Int}, Float64, true) === Diagonal{Float64}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've often wished that similar(::Type, ...) returned a type rather than an instance, and that AbstractArray subtypes implemented only it. Then a generic fallback would automatically return instances when passed instances. Wouldn't that be all we need here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for sharing these insights, @vtjnash, it certainly is a valuable contribution to the manual. But I still don't see why the fixate_eltype from this PR would be invalid. As said, it seems to work as I want it:

julia> fixate_eltype(Array, Float16)
Array{Float16,N}

julia> fixate_eltype(Array{Int}, Float16)
Array{Int64,N}

julia> fixate_eltype(Vector, Float16)
Array{Float16,1}

julia> fixate_eltype(Vector{Int}, Float16)
Array{Int64,1}

julia> fixate_eltype(BitArray, Float16)
BitArray{N}

julia> fixate_eltype(BitVector, Float16)
BitArray{1}

julia> type FunkyArray{NumDims, Foo, ElType, ElNiño} <: AbstractArray{ElType, NumDims}; end

julia> fixate_eltype(FunkyArray, Float16)
FunkyArray{NumDims,Foo,Float16,ElNiño}

julia> fixate_eltype(FunkyArray{42}, Float16)
FunkyArray{42,Foo,Float16,ElNiño}

julia> fixate_eltype(FunkyArray{42, :bar}, Float16)
FunkyArray{42,:bar,Float16,ElNiño}

julia> fixate_eltype(FunkyArray{42, :bar, String}, Float16)
FunkyArray{42,:bar,String,ElNiño}

julia> fixate_eltype(FunkyArray{42, :bar, String, -1}, Float16)
FunkyArray{42,:bar,String,-1}

There is one corner case I could find:

julia> type UntypedArray <: AbstractArray; end

julia> fixate_eltype(UntypedArray, Float16)
UntypedArray

I'm not sure what would be desired in this case, though. Maybe throwing instead of returning a type with still unspecified element type would be better. Anyway, this kind of type does not look exactly useful, so I'm not worried too much.

T.name.primary isn't a meaningful type and its type parameters don't have any fixed relation to the type parameters of AbtractArray.

That sounds indeed worrying for the proposed fixate_eltype. However, T.name.primary is used in a couple of places around base, so there has to be something of value in it. Also, the relationship of the type parameters between, say, Array and AbstractArray has be to recorded somewhere. Why shouldn't we be able to exploit it here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vtjnash Comments?

Copy link
Member

@andreasnoack andreasnoack Nov 28, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the conclusion is that this should be implemented with dispatch cf. the new paragraphs in the documentation and that T.name.primary should generally be avoided. The places in Base where T.name.primary are few and special.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OTOH https://github.com/JuliaLang/julia/blob/7f230adcf13d7a2bcb76e55327f211b7b4f25046/doc/devdocs/types.rst#typenames reads as though the use of T.name.primary here is perfectly valid. (Maybe the relationship of the TypeVars is a real problem?) It feels a bit unsatisfying to require a method to be implemented for every array type that should support e.g. zeros(MyArrayType, (5, 3)) when there is this generic alternative available.


function check_array_size(dims::DimsInteger)
for i in 1:length(dims)
dims[i] >= 0 || throw(ArgumentError("dimension size must be ≥ 0, got $(dims[i]) for dimension $i"))
end
end
check_array_size(dims::Integer...) = check_array_size(dims)

function first(itr)
state = start(itr)
done(itr, state) && throw(ArgumentError("collection must be non-empty"))
Expand Down Expand Up @@ -661,8 +689,132 @@ function copymutable(a::AbstractArray)
end
copymutable(itr) = collect(itr)

"""
fill(array_type, x, dims)

Create an array of type `array_type` filled with the value `x`. For example,
`fill(BitArray, true, (5,5))` returns a 5×5 `BitArray`, with each element initialized to
`true`.

```jldoctest
julia> fill(BitArray, true, (5,5))
5×5 BitArray{2}:
true true true true true
true true true true true
true true true true true
true true true true true
true true true true true
```

If `x` is an object reference, all elements will refer to the same object. `fill(Array,
Foo(), dims)` will return an array filled with the result of evaluating `Foo()` once.
"""
fill{T<:AbstractArray}(::Type{T}, v, dims::Dims) = fill!(fixate_eltype(T, typeof(v))(dims), v)

zero{T}(x::AbstractArray{T}) = fill!(similar(x), zero(T))

for (fname, felt) in ((:zeros,:zero), (:ones,:one))
@eval begin
function ($fname){T<:AbstractArray}(::Type{T}, dims::Dims)
array_type=fixate_eltype(T,Float64)
fill(array_type, ($felt)(eltype(array_type)), dims)
end
($fname){T}(A::AbstractArray{T}) = fill!(similar(A), ($felt)(T))
end
end

"""
zeros(array_type, dims)

Create an array of specified type of all zeros.
The element type defaults to `Float64` if not specified.

```jldoctest
julia> zeros(Diagonal, 3, 3)
3×3 Diagonal{Float64}:
0.0 ⋅ ⋅
⋅ 0.0 ⋅
⋅ ⋅ 0.0
```

```jldoctest
julia> zeros(Diagonal{Int8}, 3, 3)
3×3 Diagonal{Int8}:
0 ⋅ ⋅
⋅ 0 ⋅
⋅ ⋅ 0
```
"""
zeros{T<:AbstractArray}(::Type{T}, dims::Integer...) = zeros(T, convert(Dims, dims))

"""
zeros(array_type, A)

Create an array of specified type of all zeros with the shape of A.
"""
zeros{T<:AbstractArray}(::Type{T}, A::AbstractArray) = zeros(fixate_eltype(T, eltype(A)), size(A))

"""
ones(array_type, dims)

Create an array of specified type of all ones.
The element type defaults to `Float64` if not specified.

```jldoctest
julia> ones(SparseMatrixCSC, 2, 3)
2×3 sparse matrix with 6 Float64 nonzero entries:
[1, 1] = 1.0
[2, 1] = 1.0
[1, 2] = 1.0
[2, 2] = 1.0
[1, 3] = 1.0
[2, 3] = 1.0
```

```jldoctest
julia> ones(SparseMatrixCSC{Int8}, 1, 2)
1×2 sparse matrix with 2 Int8 nonzero entries:
[1, 1] = 1
[1, 2] = 1
```
"""
ones{T<:AbstractArray}(::Type{T}, dims::Integer...) = ones(T, convert(Dims, dims))

"""
ones(array_type, A)

Create an array of specified type of all ones with the shape of A.
"""
ones{T<:AbstractArray}(::Type{T}, A::AbstractArray) = ones(fixate_eltype(T, eltype(A)), size(A))

"""
eye(array_type, m::Integer, n::Integer)

`m`-by-`n` identity matrix of type `array_type`.
"""
function eye{T<:AbstractMatrix}(::Type{T}, m::Integer, n::Integer)
a = zeros(T,m,n)
for i = 1:min(m,n)
a[i,i] = one(eltype(a))
end
return a
end

"""
eye(array_type, n::Integer)

`n`-by-`n` identity matrix of type `array_type`.
"""
eye{T<:AbstractMatrix}(::Type{T}, n::Integer) = eye(T, n, n)

"""
eye(array_type, A)

Identity matrix of type `array_type` of the same dimensions as `A`.
"""
eye{T<:AbstractMatrix}(::Type{T}, x::AbstractMatrix) =
eye(fixate_eltype(T, eltype(x)), size(x, 1), size(x, 2))

## iteration support for arrays by iterating over `eachindex` in the array ##
# Allows fast iteration by default for both LinearFast and LinearSlow arrays

Expand Down Expand Up @@ -1001,7 +1153,9 @@ cat(catdim::Integer) = Array{Any,1}(0)
vcat() = Array{Any,1}(0)
hcat() = Array{Any,1}(0)
typed_vcat{T}(::Type{T}) = Array{T,1}(0)
typed_vcat{T<:AbstractArray}(::Type{T}) = T(0)
typed_hcat{T}(::Type{T}) = Array{T,1}(0)
typed_hcat{T<:AbstractArray}(::Type{T}) = T(0)

## cat: special cases
vcat{T}(X::T...) = T[ X[i] for i=1:length(X) ]
Expand All @@ -1012,17 +1166,24 @@ hcat{T<:Number}(X::T...) = T[ X[j] for i=1:1, j=1:length(X) ]
vcat(X::Number...) = hvcat_fill(Array{promote_typeof(X...)}(length(X)), X)
hcat(X::Number...) = hvcat_fill(Array{promote_typeof(X...)}(1,length(X)), X)
typed_vcat{T}(::Type{T}, X::Number...) = hvcat_fill(Array{T,1}(length(X)), X)
typed_vcat{T<:AbstractArray}(::Type{T}, X::Number...) = hvcat_fill(T(length(X)), X)
typed_hcat{T}(::Type{T}, X::Number...) = hvcat_fill(Array{T,2}(1,length(X)), X)
typed_hcat{T<:AbstractArray}(::Type{T}, X::Number...) = hvcat_fill(T(1,length(X)), X)

vcat(V::AbstractVector...) = typed_vcat(promote_eltype(V...), V...)
vcat{T}(V::AbstractVector{T}...) = typed_vcat(T, V...)

prepare_cat_result{T}(::Type{T}, dims::Dims, A1::AbstractArray, A...) = similar(A1, T, dims)
prepare_cat_result{T}(::Type{T}, dims::Dims, A...) = similar([A[1]], T, dims)
prepare_cat_result{T<:AbstractArray}(::Type{T}, dims::Dims, A1::AbstractArray, A...) = fixate_eltype(T, promote_eltype(A1, A...))(dims)
prepare_cat_result{T<:AbstractArray}(::Type{T}, dims::Dims, A...) = fixate_eltype(T, promote_eltype(A...))(dims)

function typed_vcat{T}(::Type{T}, V::AbstractVector...)
n::Int = 0
for Vk in V
n += length(Vk)
end
a = similar(V[1], T, n)
a = prepare_cat_result(T, (n,), V...)
pos = 1
for k=1:length(V)
Vk = V[k]
Expand Down Expand Up @@ -1050,7 +1211,7 @@ function typed_hcat{T}(::Type{T}, A::AbstractVecOrMat...)
nd = ndims(Aj)
ncols += (nd==2 ? size(Aj,2) : 1)
end
B = similar(A[1], T, nrows, ncols)
B = prepare_cat_result(T, (nrows, ncols), A...)
pos = 1
if dense
for k=1:nargs
Expand Down Expand Up @@ -1082,7 +1243,7 @@ function typed_vcat{T}(::Type{T}, A::AbstractMatrix...)
throw(ArgumentError("number of columns of each array must match (got $(map(x->size(x,2), A)))"))
end
end
B = similar(A[1], T, nrows, ncols)
B = prepare_cat_result(T, (nrows, ncols), A...)
pos = 1
for k=1:nargs
Ak = A[k]
Expand Down Expand Up @@ -1129,7 +1290,7 @@ function cat_t(catdims, typeC::Type, X...)
end
end

C = similar(isa(X[1],AbstractArray) ? X[1] : [X[1]], typeC, tuple(dimsC...))
C = prepare_cat_result(typeC, tuple(dimsC...), X...)
if length(catdims)>1
fill!(C,0)
end
Expand Down Expand Up @@ -1311,7 +1472,7 @@ function typed_hvcat{T}(::Type{T}, rows::Tuple{Vararg{Int}}, as::AbstractMatrix.
a += rows[i]
end

out = similar(as[1], T, nr, nc)
out = prepare_cat_result(T, (nr, nc), as...)

a = 1
r = 1
Expand Down Expand Up @@ -1363,7 +1524,7 @@ function hvcat{T<:Number}(rows::Tuple{Vararg{Int}}, xs::T...)
a
end

function hvcat_fill(a::Array, xs::Tuple)
function hvcat_fill(a::AbstractArray, xs::Tuple)
k = 1
nr, nc = size(a,1), size(a,2)
for i=1:nr
Expand Down
30 changes: 20 additions & 10 deletions base/array.jl
Original file line number Diff line number Diff line change
Expand Up @@ -202,14 +202,13 @@ julia> fill(1.0, (5,5))
If `x` is an object reference, all elements will refer to the same object. `fill(Foo(),
dims)` will return an array filled with the result of evaluating `Foo()` once.
"""
fill(v, dims::Dims) = fill!(Array{typeof(v)}(dims), v)
fill(v, dims::Integer...) = fill!(Array{typeof(v)}(dims...), v)
fill(v, dims::Dims) = fill(Array, v, dims)
fill(v, dims::Integer...) = fill(Array, v, convert(Dims, dims))

for (fname, felt) in ((:zeros,:zero), (:ones,:one))
@eval begin
($fname)(T::Type, dims...) = fill!(Array{T}(dims...), ($felt)(T))
($fname)(dims...) = fill!(Array{Float64}(dims...), ($felt)(Float64))
($fname){T}(A::AbstractArray{T}) = fill!(similar(A), ($felt)(T))
end
end

Expand All @@ -219,13 +218,7 @@ end
`m`-by-`n` identity matrix.
The default element type is `Float64`.
"""
function eye(T::Type, m::Integer, n::Integer)
a = zeros(T,m,n)
for i = 1:min(m,n)
a[i,i] = one(T)
end
return a
end
eye(T::Type, m::Integer, n::Integer) = eye(Matrix{T}, m, n)

"""
eye(m, n)
Expand Down Expand Up @@ -982,6 +975,23 @@ function vcat{T}(arrays::Vector{T}...)
return arr
end

hcat(A::Matrix...) = typed_hcat(Array{promote_eltype(A...)}, A...)
hcat{T}(A::Matrix{T}...) = typed_hcat(Array{T}, A...)

vcat(A::Matrix...) = typed_vcat(Array{promote_eltype(A...)}, A...)
vcat{T}(A::Matrix{T}...) = typed_vcat(Array{T}, A...)

hcat(A::Union{Matrix, Vector}...) = typed_hcat(Array{promote_eltype(A...)}, A...)
hcat{T}(A::Union{Matrix{T}, Vector{T}}...) = typed_hcat(Array{T}, A...)

vcat(A::Union{Matrix, Vector}...) = typed_vcat(Array{promote_eltype(A...)}, A...)
vcat{T}(A::Union{Matrix{T}, Vector{T}}...) = typed_vcat(Array{T}, A...)

hvcat(rows::Tuple{Vararg{Int}}, xs::Vector...) = typed_hvcat(Array{promote_eltype(xs...)}, rows, xs...)
hvcat{T}(rows::Tuple{Vararg{Int}}, xs::Vector{T}...) = typed_hvcat(Array{T}, rows, xs...)

hvcat(rows::Tuple{Vararg{Int}}, xs::Matrix...) = typed_hvcat(Array{promote_eltype(xs...)}, rows, xs...)
hvcat{T}(rows::Tuple{Vararg{Int}}, xs::Matrix{T}...) = typed_hvcat(Array{T}, rows, xs...)

## find ##

Expand Down
1 change: 1 addition & 0 deletions base/bitarray.jl
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ type BitArray{N} <: DenseArray{Bool, N}
N != 1 && (b.dims = dims)
return b
end
BitArray(dims::Dims{N}) = BitArray{N}(dims...)
end

# note: the docs for the two signatures are unified, but only
Expand Down
17 changes: 17 additions & 0 deletions base/linalg/bidiag.jl
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,12 @@ type Bidiagonal{T} <: AbstractMatrix{T}
end
new(dv, ev, isupper)
end
function Bidiagonal(m::Integer, n::Integer)
check_array_size(m, n)
checksquaredims(m, n, Bidiagonal{T})
new(Vector{T}(n), Vector{T}(n-1))
end
Bidiagonal(dims::Dims{2}) = Bidiagonal{T}(dims...)
end
"""
Bidiagonal(dv, ev, isupper::Bool)
Expand Down Expand Up @@ -55,6 +61,17 @@ julia> ev = [7; 8; 9]
Bidiagonal{T}(dv::AbstractVector{T}, ev::AbstractVector{T}, isupper::Bool) = Bidiagonal{T}(collect(dv), collect(ev), isupper)
Bidiagonal(dv::AbstractVector, ev::AbstractVector) = throw(ArgumentError("did you want an upper or lower Bidiagonal? Try again with an additional true (upper) or false (lower) argument."))

"""
Bidiagonal{T}(dims)

Constructs a bidiagonal matrix with uninitialized diagonal and off-diagonal elements of
type `T`. If `T` is omitted, it defaults to `Float64`. The dims may be given as two integer
arguments or as tuple of two `Int`s, both of which have to be equal as `Bidiagonal`
matrices are always square.
"""
Bidiagonal(m::Integer, n::Integer) = Bidiagonal{Float64}(m, n)
Bidiagonal(dims::Dims{2}) = Bidiagonal(dims...)

"""
Bidiagonal(dv, ev, uplo::Char)

Expand Down
18 changes: 17 additions & 1 deletion base/linalg/diagonal.jl
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,13 @@

immutable Diagonal{T} <: AbstractMatrix{T}
diag::Vector{T}
Diagonal(v::Vector{T}) = new(v)
function Diagonal(m::Integer, n::Integer)
check_array_size(m, n)
checksquaredims(m, n, Diagonal{T})
return Diagonal{T}(Vector{T}(n))
end
Diagonal(dims::Dims{2}) = Diagonal{T}(dims...)
end
"""
Diagonal(A::AbstractMatrix)
Expand Down Expand Up @@ -46,7 +53,16 @@ julia> Diagonal(V)
⋅ 2
```
"""
Diagonal(V::AbstractVector) = Diagonal(collect(V))
Diagonal{T}(V::AbstractVector{T}) = Diagonal{T}(collect(V))
"""
Diagonal{T}(dims)

Constructs a diagonal matrix with uninitialized diagonal elements of type `T`. If `T` is
omitted, it defaults to `Float64`. The `dims` may be given as two integer arguments or as
tuple of two `Int`s, both of which have to be equal as `Diagonal` matrices are always
square.
"""
Diagonal(dims...) = Diagonal{Float64}(dims...)

convert{T}(::Type{Diagonal{T}}, D::Diagonal{T}) = D
convert{T}(::Type{Diagonal{T}}, D::Diagonal) = Diagonal{T}(convert(Vector{T}, D.diag))
Expand Down
7 changes: 6 additions & 1 deletion base/linalg/linalg.jl
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import Base: USE_BLAS64, abs, big, ceil, conj, convert, copy, copy!, copy_transp
imag, inv, isapprox, kron, ndims, parent, power_by_squaring, print_matrix,
promote_rule, real, round, setindex!, show, similar, size, transpose, transpose!,
trunc, broadcast
using Base: promote_op, _length
using Base: promote_op, _length, check_array_size
# We use `_length` because of non-1 indices; releases after julia 0.5
# can go back to `length`. `_length(A)` is equivalent to `length(linearindices(A))`.

Expand Down Expand Up @@ -218,6 +218,11 @@ function checksquare(A...)
return sizes
end

function checksquaredims(m::Integer, n::Integer, mat_type::Type)
n==m || throw(DimensionMismatch("$mat_type matrix must be square"))
return n
end

function char_uplo(uplo::Symbol)
if uplo == :U
'U'
Expand Down
Loading