RFC: add Histogram types

[simonbyrne]

This adds explicit types for histograms. The main advantage is that we can define methods that operate on histograms, notably plot. This approach closely mirrors existing functionality, but there are some alternatives that may be worth thinking about:

• Use hist to create both 1d and 2d versions: this could be done using tuples (similar to the kde function in KernelDensity.jl).
• Create a general type for an N-dimensional histogram. For example

immutable Histogram{N,E,T}
   edges::NTuple{N,E}
   weights::Array{T,N}
end

• Methods for combining multiple histograms (for example, with distributed arrays): we could overload + for this?
• Move all this out of Base into StatsBase.jl (see Histograms JuliaStats/StatsBase.jl#49)

[johnmyleswhite]

+1 for a Histogram type

[StefanKarpinski]

+1 for a Histogram type and to @jiahao's suggestion.

[simonbyrne]

Okay, there's now a single Histogram type, and a single hist function, which accepts tuples as arguments for higher-order histograms.

The question is now what to do with matrix arguments: if size(X) == (m,n), should hist(X):

• create an array of Histogram{T,1} objects for each column (this is analogous to current hist behaviour), or
• create a single Histogram{T,n} object (similar to current hist2d behaviour)?

[simonbyrne]

One other thing to think about: which way should we round? (i.e. should the bins be upper- or lower-inclusive)?

At the moment, we round down, i.e. with an edge vector 0:2, an observation of 1 will go into the (0,1) bin:

julia> hist([1],0:2)
Histogram{Int64,1,(UnitRange{Int64},)}((0:2,),[1,0])

This is the same as the R default (which has the option for either), but the opposite of numpy (which doesn't allow a choice).

The historical reason for this choice was that pre-FloatRange, this reduced the number of odd things would happen if using ranges like 0:0.1:1 (since 0.1 > 1//10). This problem can be seen in numpy:

In [8]: np.histogram([0.1,0.2,0.3,0.4],10,(0.0,1.0))
Out[8]: 
(array([0, 1, 2, 0, 1, 0, 0, 0, 0, 0]),
 array([ 0. ,  0.1,  0.2,  0.3,  0.4,  0.5,  0.6,  0.7,  0.8,  0.9,  1. ]))

However now this shouldn't be a problem, either way should give reasonable answers.

If we want to allow the option of either, a neat way to do this would be to add another type parameter for RoundUp/RoundDown, and then we can dispatch on searchsortedfirst/searchsortedlast as appropriate.

[nalimilan]

I'm not sure it matters much, but it seems to me it's more common to use intervals closed on the left and open on the right, i.e. [a; b). This is interesting when you have only positive values, as it means 0 will be included without having to pass artificially a negative value. And integer age ranges (for example) are usually defined as 20-29, 30-39, etc.

[simonbyrne]

I've added an optional interval argument that is either :left or :right, indicating the closed side.

However I think it makes sense to keep the current behaviour (interval=:right) as default, since that matches nicely with 1-based indexing:

julia> hist(1:100;interval=:right)
Histogram{Int64,1,(StepRange{Int64,Int64},)}((0:20:100,),[20,20,20,20,20],:right)

julia> hist(1:100;interval=:left)
Histogram{Int64,1,(StepRange{Int64,Int64},)}((0:20:120,),[19,20,20,20,20,1],:left)

[ViralBShah]

+1 for moving this into StatsBase. Only downside is that many users may expect hist to be part of Base, coming from Matlab, where it is standard. I am assuming that R users get such functionality without jumping through package installation hoops.

[simonbyrne]

Perhaps this is an argument for having searchsortedfirst return 1 less by default? cf. #5664

[simonbyrne]

@ViralBShah That would perhaps be the easiest in terms of upgrade path, as we could deprecate the existing functionality gradually without breaking too much.

[nalimilan]

@simonbyrne As you like, though I'm not sure the comparison really has practical applications. But wouldn't it be more explicit to call the argument closed rather than interval? In R I can never remember in what direction it works.

[simonbyrne]

@nalimilan closed does make more sense.

[BobPortmann]

Not a big thing, but wouldn't counts make more sense then weights for the actual histogram field.

[simonbyrne]

@BobPortmann I had in mind that the same structure could be used for weighted observations, or could be normalised to one (which is why I didn't restrict the field to be Int).

[simonbyrne]

So I take it that no one is opposed to moving this to StatsBase.jl?

[ViralBShah]

I would think so.

[johnmyleswhite]

+1 for putting histograms in StatsBase.jl

[lindahua]

Late to this party. Agree wholeheartedly with the move.

[prcastro]

Why isn't this closed?

[pao]

Because no one got to it. @simonbyrne want to do the honors, or is there something else here keeping this open?

[ViralBShah]

Bump.

[simonbyrne]

Sorry, been away. Yep, now in StatsBase.