Dynamic map tutorial

[jlstevens]

In addition to adding an important new tutorial, I would like to use this PR to achieve a number of things:

• Try out a new, incremental approach to building new tutorials.
• Offer a clear example of how we want notebook tutorials to be created in future.
• Exercise the new documentation building infrastructure.

Until now, writing a new tutorial has been a daunting task. What has happened is that a tutorial begins life on someone's hard drive where it slowly grows until it suddenly appears for everyone to see in a nearly complete form. Often I've found such tutorials languish till they get forgotten or deleted.

I think we should work in a more incremental fashion using pull requests:

• We can get input from users and other devs as the tutorial develops. Typo fixes, clarification, grammatical fixes etc are always welcome.
• Even if it is unfinished, the tutorial is visible to everyone so anyone else can help finish it off.
• We don't need to dedicate a huge amount of time when introducing a new tutorial. As soon as we feel a new tutorial is necessary, we can create the pull request and take it from there.

There is one downside of this approach: I don't want to swamp the main repository with small documentation updates/fixes as the tutorial grows. For this reason, I suggest the following:

• There is no shame in push forcing to squash commits together as long as only one person is working on the tutorial.
• If someone else wants to contribute, they just need to say that they want to make a change, at which point push forcing is not allowed. However, before the tutorial is merged, it should be squashed down to a single commit.

I hope this approach will make it less of a hurdle to add new notebooks and docs. To start this tutorial, I've added a simple dynamic map example based on the sine rings we introduce in the existing tutorials.

Edit: After a website build is triggered, this PR can be previewed on build.holoviews.org .

[jbednar]

Sounds good. How can people see the output in the tutorial before it's completed? It's hard to make any use of it when there are no images yet.

Can people safely use Github's browser-based editing, to avoid problems with squashing? That's an easy way to fix typos, and avoids having a local copy of the branch to get confused about later.

[jbednar]

Or are the images available via the Travis builds above (if so, can this page include a link to it to make it easy for people to find?)

[jlstevens]

Sounds good. How can people see the output in the tutorial before it's completed? It's hard to make any use of it when there are no images yet.

Good point. Once a build is triggered, it will be viewable on build.holoviews.org but that might change if a different documentation PR is built. I guess, in the ideal case we could have PR specific parts of the website but that would require a fair bit of effort to configure.

Anyway, any such PRs should like to the website and I've given an example by making an edit above.

Can people safely use Github's browser-based editing, to avoid problems with squashing? That's an easy way to fix typos, and avoids having a local copy of the branch to get confused about later.

Yes, for things like typo fixes that is a good idea. For bigger edits, you really need a notebook to work in so I see now way to avoid checking out a branch. Maybe someone should offer an online service to edit the JSON in notebooks without running a live server. A business opportunity perhaps? :-)

[jlstevens]

I have an idea that would be easy to implement but I'm not sure how useful it would be.

Instead of worrying about build.holoviews.org getting clobbered, we could easy have a zip of the website available to download after the build (static link). So then if someone could grab this zip immediately after the build and make it available (somehow), then at least we could have persistent snapshots of the website as a PR progresses.

What do you think?

Edit: To be clear, what I'm not sure about is how to make hosting such zip files easy.

[jlstevens]

Well that was easy: simple_test.txt.zip

I wonder if there is a size limit though....

Edit: I also wonder about persistence - surely GitHub has to clean up these files at some point.

[jlstevens]

Ok. I've found some information here. In summary:

• The maximum size for files is 25MB and the maximum size for images is 10MB.

This is not enough to hold the tutorial notebook HTML (42.2MB zipped) but would be enough for any single notebook that is being worked on. Would that be useful?

[jbednar]

The zip could be useful, but it requires a few extra steps, so probably most people would just want to click on a link at build.holoviews.org. However, when I click there, I don't see any obvious way to reach the Dynamic Map tutorial; e.g. http://build.holoviews.org/Tutorials/DynamicMap.html doesn't work. Probably it would be more useful to put in a link directly to the appropriate tutorial file.

BTW, I think the filename should be Dynamic_Map (with an underscore) to match the other tutorials. And the PR should include editing the Tutorial index to show how to reach this tutorial.

[jlstevens]

BTW, I think the filename should be Dynamic_Map (with an underscore) to match the other tutorials.

Sure. I'll make the change now.

I've had a think about snapshots and found it hard to find a system I like. Cycling a fixed number of S3 bins hardly helps as any URL would keep changing. Accumulating website builds for PRs isn't great as we have to pay for S3 storage. In general, I don't want to open up a can of worms because anything involving buildbot and S3 requires a fair bit of effort to configure.

Here is what I would like:

• An ability to snapshot that is easy but not entirely automatic. If making a snapshot is easy but not automatic, we are unlikely to accumulate website builds too quickly. That way the functionality is there if we want it whereas an automatic system is likely to be wasteful.
• Snapshots of individual notebooks to keep file sizes down. PRs that change the website all at once should be rare and for that I think build.holoviews.org is enough.
• A system that is easy and orthogonal to the current one. The current system is complicated enough!

The most interesting idea I have is to use Google Drive as it is possible to allow anyone to upload files to a google drive (holoviews@gmail.com). You can make a public web form to interface with the drive as you can see in this video.

There is a tiny bit of code involved and you can easily deploy a Google App. Here is the API and really only one core function is needed for our purposes:

DriveApp.createFile('New HTML File', '<b>Hello, world!</b>', MimeType.HTML);

This could let us grab the HTML from the build and easily turn it into a file on google drive which immediately is returned as a URL to the snapshot. Anyway, as I said there is very little code involved (less than 80 lines adapting the example in the video) but I won't work on that now (getting a minimal DynamicMap tutorial is more important).

[jlstevens]

Ok, I'm done for tonight. I triggered a website build so here is a WIP preview:

http://build.holoviews.org/Tutorials/Dynamic_Map.html

[jlstevens]

@jbednar @philippjfr The first draft is complete! It may not be polished but it does cover all the core content...

Have a look here - unfortunately, we now encounter the biggest problem with DynamicMap in that a live server is required to properly view the content. It would be good if you could download the notebook (available here) and try it out properly.

Other than general polish, here are the things I want to do:

• Add an entry to the main Tutorial page.
• Adapt some of examples to use something other than Images - I'm sick of those sine rings!
• Update the section about slicing. I hope I can implement sensible semantics for slicing (updating the dimension ranges) tonight.

@philippjfr It is interesting that a DynamicMap + HoloMap does export to a static page. Obviously, this only shows the DynamicMap cache but maybe this is still a lot better than losing all the data? Any chance we could leverage this effect elsewhere?

[jlstevens]

I'll continue to make fixes and changes and I'll rebuild the website once I've accumulated enough of them. Till then, keep an eye out to see if I've addressed any comments you have in the following commit messages.

[jbednar]

My first comment (other than, yay!), is that yes, it would be great if the static copy of the notebook would show some limited subset of the dynamic functionality, as for DynamicMap+HoloMap. Otherwise the website is very confusing!

[jlstevens]

@jbednar I've had a think about this and it is mostly not possible, unfortunately,

The issue is that the boring automaton that is the website building script has no desire to fiddle around sliders like any sensible human would. :-)

This means that unless an explicit cache filling operation is used (described in the tutorial) the cache will always be nearly empty. No displayed DynamicMap actually has an empty cache as one frame is needed to initialize the plotting system. This means, there is one frame that could be displayed and I would certainly like it to be displayed!

Sadly, @philippjfr thinks getting this working for the matplotlib backend (maybe the bokeh backend is ok?) is too much effort right now. I guess if the bokeh backend doesn't have this issue (not tried it yet) we could use that instead. My main objection then would be that this tutorial isn't about bokeh and matplotlib is always the safest backend to assume people have available.

[jbednar]

In [7]: looks pretty if you use the cursor keys; it makes Moire patterns.

[jlstevens]

Just completed a new build:

http://build.holoviews.org/Tutorials/Dynamic_Map.html

Thanks to Philipp, the first frame now always show. There has been a fair bit added but there are a few things that I haven't mentioned yet:

• Styling of DynamicMaps is the same as always (I'll use an existing example).
• Can output overlays from a DynamicMap but dmap * dmap is not supported (yet)

As for functionality, everything is now working the way I want which means I think we are ready for the 1.4.3 release unless some major bugs crop up.

[jbednar]

Note that you can index a DynamicMap with a literal key in exactly the same way as a HoloMap, so long as you use an exact key value that already exists in the cache:

But you can't index it by just specifying the value of each dimension, generating the result dynamically as if it were a function call?

In open mode, elements are naturally serialized by a linear sequence of next() calls. This doesn't mean that they can't have multiple key dimensions:

Can you explain what happens in that case? It looks like all dimensions proceed in parallel?

[jbednar]

I'll have tutorial edits shortly.

[jbednar]

Happy to have it merged; looks great!

[jlstevens]

Thanks for the edits!

I'll run a website build now to have a look. It would have been nice to see you have a go running a build yourself using the new system! As a quick reminder: Show all checks -> s3-reference-data-cache -> Details -> Update website.

But you can't index it by just specifying the value of each dimension, generating the result dynamically as if it were a function call?

Oops! I obviously got confused myself and yes, if the key exists in the cache it returns that, otherwise it computes a new element.

In open mode, elements are naturally serialized by a linear sequence of next() calls. This doesn't mean that they can't have multiple key dimensions:

Open mode is always displayed as a linear sequence that matches the sequence of next() calls. What I meant there was that you can have multiple key dimensions (as many as you want) and in each next call you can specify where you are in the multi-dimensional space by specifying a tuple key (i.e when you use the (<key>, <element>) return format).

In other words, the generator can specify a linear path through some high-dimensional space as you move along the sequence. If you cast a multi-dimensional open mode DynamicMap you should end up with a multi-dimensional HoloMap with a bunch of sliders.

I think that is worth demonstrating and there are a few other bits I would like to mention - a quick example of styling, a quick description of overlays and I'll try to move away from the sine rings for some variety.

@philippjfr There should also be at least one note about normalization. If I remember correctly, you only get axiswise normalization and the only way to normalize across DynamicMaps is to specify the ranges on the value dimensions? This makes sense as you obviously can't know what values your callback may return in future...

[jlstevens]

Reading through the edits now.

One general convention we should decide on is how to write 'Elements'. In my opinion, it shouldn't be capitalized unless we really mean the class in which case it should be Element. Using Elements is debatable as Elements looks silly.

Here this would mean 'subclasses of the real class Element' though it is tricky as we do have pluralized class names e.g Points and Paths.

[jlstevens]

I've updated the central example to something less boring than the sine rings. This new example demonstrates both styling and overlays. Right now the end of the bounded section has an intermittent bug that we hope to fix shortly.

[jlstevens]

Just to say I've kept the content the same but updated the code to show off more varied and interesting examples. As I said before, I'm sick of sine rings! A website build should complete shortly so that you can have a look at the new examples...

[philippjfr]

I've gone through the Tutorial and it seems great overall. It has a large number of examples for all the different modes which is probably most important, and goes into a good amount of detail. I'd be happy to see this merged immediately but I'll wait for you to add a small blurb about normalization, i.e. having to define dimension ranges.

[jlstevens]

I've added a small section about normalization so I think I've covered all the material I intended to. I'll run a website build now and if you are happy with it, please go ahead and merge.

In a few minutes, the page will be updated here.

[jlstevens]

The page is now updated. If you think it is ok or any changes are small ones we can do on master, please merge.

[jlstevens]

Turns out we need to make some changes to the testing script before new tutorials can get testing data (the script skips testing if it finds no reference data). I've open an issue about this #498 so I think we might as well merge now as we can't tackle that immediately.

[philippjfr]

Great work on this! Merging.