Add support for dim transforms to set style options

[philippjfr]

As discussed in #768 this PR adds support for scaling various style options by values in the element. It adds an op object which allows defining deferred transforms on an element dimension implementing the standard mathematical operations and common NumPy operations.

You can see what is currently supported here.

As we had discussed in the issue we will also support tuple and string syntax so you only have to define an op to define complex transforms more easily.

Here are some example:

scatter = hv.Scatter(np.arange(11))
o = (op('y')-op('y').min())/op('y').max()
print(o)
o.eval(scatter)

truediv(sub('y', amin('y')), amax('y'))
array([ 0. ,  0.1,  0.2,  0.3,  0.4,  0.5,  0.6,  0.7,  0.8,  0.9,  1. ])

%%opts Points (alpha=op('z') size=np.log1p(op('z'))*20)
hv.Points(np.random.rand(100, 3), vdims=['z'])

%%opts Curve [show_legend=False width=500] (line_width=10*op('y').mean())
hv.NdOverlay({i: hv.Curve(np.random.rand(10)) for i in range(5)})

[jlstevens]

I'll be happy to see this supported though I do have one request...

I really want to see op used in the simplest cases, matching what we already support. One goal of op is to support more advanced control than has been possible so far (which your examples above demonstrate) but another goal should be to keep simple things looking simple.

I want to make sure the simplest uses of op 1. work as a good default in many cases 2. remains succinct and readable.

Edit: I see a few examples in the notebook you linked to, though a few more examples shown in the comments of this PR wouldn't go amiss!

[philippjfr]

I'd also like this PR to address #237.

[jlstevens]

How do you think op can help with #237? It doesn't seem obvious to me yet...

[philippjfr]

I've discussed this before but I was going to suggest an apply method (or similar) for elements, which would let you transform dimensions/columns using ops or simply replace it. I'll put together a fully fleshed out proposal at some point.

[jlstevens]

I remember discussing apply with you but I didn't remember the idea that it could make use of op. It seems like a reasonable idea but I don't see why that would need to be part of this PR...

[philippjfr]

It wouldn't of course, could be a followup PR. I just wanted to establish a link between ops and that issue, so I don't forget about it.

[philippjfr]

To track how complete this PR is I'm going to make a list of elements that are fully tested:

Elements tested

Charts

• Scatter/Points
  • Matplotlib
  • Bokeh
• Curve
  • Matplotlib
  • Bokeh
• Histogram
  • Matplotlib
  • Bokeh
• Bars
  • Matplotlib (not supported, plot implementation is a mess)
  • Bokeh
• Spikes
  • Matplotlib
  • Bokeh
• Labels
  • Matplotlib
  • Bokeh
• VectorField
  • Matplotlib
  • Bokeh
• ErrorBars
  • Matplotlib
  • Bokeh

Geometries

• Path
  • Matplotlib
  • Bokeh
• Contours
  • Matplotlib
  • Bokeh
• Polygons
  • Matplotlib
  • Bokeh

Graphs

• Graph
  • Bokeh
  • Matplotlib
• Chord
  • Bokeh
  • Matplotlib
• Sankey
  • Bokeh
  • Matplotlib
• TriMesh
  • Matplotlib
  • Bokeh

Statistics

• BoxWhisker
  • Bokeh
  • Matplotlib (not supported)
• Violin
  • Bokeh
  • Matplotlib (not supported)
• HexTiles
  • Bokeh
  • Matplotlib (not supported)

Not supported (i.e. no vectorized glyphs)

• Raster
• QuadMesh
• Image
• RGB
• HSV
• HeatMap
• Area
• Spread
• Distribution
• Bivariate
• Text
• HLine
• VLine
• Spline

Note that when I something is not supported that means that this property cannot be set as a vectorized property. Therefore to support these cases we would have to plot multiple separate artists/glyphs, which won't be handled in the first iteration of this feature.

[philippjfr]

This notebook also tracks what is and isn't supported yet:

https://anaconda.org/philippjfr/op_transforms/notebook

[philippjfr]

So I think we are now in a place where this can be merged, but there are still a few outstanding items which will need to happen in later PRs:

• Matplotlib BarPlot needs rewriting from scratch, I'm not touching the current implementation with a 10 foot barge pole
• SankeyPlot does not yet support style mapping, this will require some work because the styles need to be carried through the operation
• I need to finish writing a style mapping tutorial and or integrate the material in some existing tutorial
• Eventually we want to support applying ops directly to elements to either transform existing dimension values or add new ones
• Write more unit tests for plot updates

Some other features this PR introduces:

• The normalization system now handles normalizing Graph edge start/end values along the node indices which is important for consistent coloration between nodes and edges
• The normalization system now handles computing discrete factors across plots

Another idea I've had, is that we could allow specifying a transform(s) directly on a Dimension object so when you query the dimension on an element the transform is applied. This could be very powerful as it could support a huge range of transformations:

• Unit transforms, e.g. when querying the values would be transformed to some fixed unit
• Coordinate system transforms
• Storing regularly spaced log transformed values from datashader on Image coordinates (solving Log axis when plotting holoviews object through datashader #2195)
• Jitter of axis locations
• Lots more I can't think of right now

[jbednar]

Sounds great!

[philippjfr]

Tests should pass shortly. Let's get this merged so we can get a few days of testing in before a release.

[jlstevens]

I think this is a stray instance of the tuple format...

[jlstevens]

And another tuple...

[jlstevens]

I'm happy to merge but I think I spotted a few stray instances of the tuple format above.

[jlstevens]

Tests are now green.

I'm very happy to merge this PR as I think it will greatly simplify things throughout while exposing a more powerful way of working. It will require a fair bit of testing and of course we may well have various tweaks to make. For the time being, I'll finally go ahead and merge...nearly a year since this PR was first opened!