Improve documentation for fan in operators

[joost-de-vries]

The first step is for this doc page to describe what the individual operators do.

[joost-de-vries]

Btw I'm getting an exception when trying to build the docs. I don't see what's wrong.
Does this seem familiar?

When running docs/makeSite I get

error] (docs/paradox:paradoxMarkdownToHtml) com.lightbend.paradox.markdown.SnipDirective$LinkException: Unknown snippet [/Users/joost1/source/swave/docs/src/paradox/usage/show-off/$test$/MD5Spec.scala (No such file or directory)] referenced from [usage/show-off/md5.html]

[sirthias]

The problem is that the swave docs actually depend on sbt-paradox version 0.2.9 but the sbt-site plugin is not yet compatible with that release.
I've just put in a PR to fix the issue: sbt/sbt-site#93

If you want to build the docs locally right now you need to:

1. Clone my fork of sbt-site's develop branch
2. Run sbt publishLocal on that working copy
3. Bump the plugin versions in your swave working copy to 0.2.9 and 1.3.0-SNAPSHOT.

[ktoso]

FYI @eed3si9n about the incompatibility between paradox and sbt-site

[joost-de-vries]

Tx @sirthias I'll do that for now.

Wrt the documentation:

I've tried to come up with one liner descriptions that are as intuïtive as possible. (since a lot of descriptions of streaming operators read like temporal legalese)
And I'm looking for a common structure. When it comes to the common structure:

• I think an important aspect is: is the output an or of the inputs, or an and.
• and if it's an and: how are the values combined to a single result.

What do you think: are those the most important semantic differences between those operators?

[sirthias]

@joost-de-vries There are more factors that differentiate fan-in transformations. This page in the docs tries to highlight them.

The reason that "descriptions of streaming operators read like temporal legalese" is that it's truly not as simple as "and" or "or". Apart from how they treat incoming elements the transformation also differ in when they do it. And there is the question of how and when they request from their upstreams and when they complete to downstream.

One guideline could be the Akka documentation. Here is their documentation for fan-ins.

Their structure is (general description, emits, backpressures, completes), which I would improve to (general description, requests from upstream, emits to downstream, completes to downstream), so very similar but turning "backpressures" from its head on its toes.

[joost-de-vries]

Yeah, I agree that the 'temporal legalese' is necessary.

The way I understand it is that a given stream has a semantic that could be informally expressed as (element)* (error | complete)?
(of course that only communicates what's going downstream, not the demand going upstream)
More formally I guess the semantics of a stream can be expressed using temporal logic. I was referring to that with the quip 'temporal legalese'.

It's just that for a user of Swave I thought it would be nice to to have a short description per operator. Before diving into the temporal legalese of a single operator.

Currently I have this

* @ref[fanInConcat] emits all events of the first spout, then the second, etc
* @ref[fanInMerge] emits the event of either of the input spouts as soon as it comes in
* @ref[fanInRoundRobin]
* @ref[fanInSorted]
* @ref[fanInToTuple] emits all input events combined as a tuple
* @ref[fanInToHList] emits all input events combined as an HList. This can be turned into a stream of case class instances.
* @ref[fanInToCoproduct] ?how does this differ from fanInMerge?
* @ref[fanInToProduct] ?how does this differ from fanInToHList?
* @ref[fanInToSum] emits the event of either of the input spouts as soon as it comes in. The input spouts are enumerable instances of a sealed trait. The result is of the type of the sealed trait.

As you can tell I wasn't sure about fanInToCoproduct and fanInToProduct.

[sirthias]

A short description per operator definitely makes sense.

How about something like this?:

* @ref[fanInConcat] drains all its upstreams in sequential order and emits their elements to downstream unchanged
* @ref[fanInMerge] drains all its upstreams at the same time and emits their elements to downstream in the order that they arrive
* @ref[fanInRoundRobin] emits the 1st element from the 1st upstream, then the 1st element from the 2nd upstream, ... then the 2nd element from the 1st upstream, then the 2nd element from the 2nd upstream and so on...
* @ref[fanInSorted] emits a sorted stream of elements to downstream, according to some order, provided each upstream itself produces its elements according to the same order 
* @ref[fanInToTuple] iteratively buffers one element from each upstream and emits them to downstream as a tuple
* @ref[fanInToHList] iteratively buffers one element from each upstream and emits them to downstream as an HList instance
* @ref[fanInToCoproduct] like @ref[fanInMerge], but with an element type that is the shapeless `Coproduct` of all upstream types
* @ref[fanInToProduct] iteratively buffers one element from each upstream and emits them to downstream as an instance of some product type (e.g. a case class)
* @ref[fanInToSum] like @ref[fanInMerge], but with a (given) element type that is the sum type (ADT) of all upstream types

The main difference to your proposal would be "elements" instead of "events" and "upstreams" instead of "inputs".

WDYT?

[joost-de-vries]

That looks very clear.
I do wonder about fanIntoSum: I thought a sum type in Shapeless was basically a union type which values are enumerable. Ie something like a sealed trait with case objects. Not an ADT/case class?
But then I think that a Coproduct in Shapeless is a union type.... Ie similar but the values are not enumerable.

Are you purely targeting Shapeless developers? Because most Scala developers are lost when one starts talking about Coproducts and sum types...
It's perhaps not the most accessible terminology.

[sirthias]

swave doesn't only target shapeless developers. I think you don't need any knowledge of shapeless in order to be able to use it effectively.

In order to understand fanInToSum check out the implementation of mergeToEither:

final def mergeToEither[B](right: Spout[B]): Repr[Either[A, B]] =
    map(Left[A, B]).attach(right.map(Right[A, B])).fanInToSum[Either[A, B]]()

So, suppose you have this ADT:

sealed trait Animal
case class Cat(name: String) extends Animal
case class Dog(name: String) extends Animal
case object LastUnicorn extends Animal

then you can merge three upstreams with type Cat, Dog and LastUnicorn.type to a stream of Animal with fanInToSum[Animal].

[joost-de-vries]

I guess the difference between fanIntoCoproduct and fanInToSum is that a sum type requires the existence of a sealed trait that all extend. While a coproduct can be build out of any disparate types.
I don't think this is general knowledge. But I'm not sure what to do about that. My ideal would be to provide an intuitive/what-can-you-use-it-for description first. And then mention the shapeless term. Perhaps with a link to the shapeless docs.
Maybe the best way to explain the different operators would be to write concise examples in the operator doc page.

I'll make a beginning with writing those.

[joost-de-vries]

Here's my proposal for the fanOut oneliners:

* @ref[fanOutBroadcast] emits upstream elements to all downstreams
* @ref[fanOutRoundRobin] emits upstream elements alternatingly. The first element to the 1st downstream, the next to the 2nd ...
* @ref[fanOutSequential] ??
* @ref[fanOutSwitch] takes a function that determines based on an element value which of the downstreams the element should be emitted to.
* @ref[fanOutToAny] ??
* @ref[fanOutUnZip] upstream tuples are broken up into constituent values and those are each emitted to the matching downstream

[sirthias]

I guess the difference between fanIntoCoproduct and fanInToSum is that a sum type requires the existence of a sealed trait that all extend. While a coproduct can be build out of any disparate types.

Yes, that's right.

I don't think this is general knowledge. But I'm not sure what to do about that.

I agree but I don't think we have to do anything about it.
If you don't know what a shapeless Coproduct is then you don't need fanInToCoproduct. We don't need to put any special focus on that particular variant.
We can link to it from the description page of fanInToSum with the explanation as you described it. That should be fine.

Maybe the best way to explain the different operators would be to write concise examples in the operator doc page.

Absolutely. This has been my goal all along...

[joost-de-vries]

I've created a PR of the ongoing doc improvement.