Add support for multiple sinks per pipeline #2060
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jtfmumm
force-pushed
the
multi-sink
branch
2 times, most recently
from
e9af2f2
to
f42e09e
Compare
|
@jtfmumm integration tests are added and enabled for the two apps:
|
This adds to_sinks() to the Pony API and updates the way we store information about sinks in the Pipeline object to allow one or more sinks per pipeline.
Prior to this commit, Wallaroo only supported one routing target per output message from a Step. This adds support in our routers for routing messages to more than one routing target per message (duplicating the same message, i.e. fan out).
Prior to this, creation of LocalTopology objects assumed that there would be at most one sink per pipeline. This updates the init algorithm prior to distribution of LocalTopology workers across the cluster to support the new to_sinks() API call which can specify more than one sink per pipeline.
Prior to this commit, the algorithm for transforming a LocalTopology graph into a collection of initiated step actors assumed that there would only be one output per node in the graph (unless we were dealing with a partition). This removes that assumption from the algorithm and supports multiple outputs per node.
- rename old `multi_sink` test app to `multi_pipeline` - create new multi_sink test app that is one-pipeline-to-two-sinks
to_sinks() was creating multiple pipeline objects when there should only have been one.
This change allows the application developer to add a multisink to a Go application. The Pony side is not yet implemented.
aturley
reviewed
Feb 22, 2018
| @@ -51,10 +51,13 @@ func ApplicationSetup() *C.char { | |||
| application.NewPipeline("Split and Count", app.MakeTCPSourceConfig(inHost, inPort, &Decoder{})). | |||
| ToMulti(&SplitBuilder{}). | |||
| ToStatePartition(&CountWord{}, &WordTotalsBuilder{}, "word totals", &WordPartitionFunction{}, LetterPartition()). | |||
| ToSink(app.MakeTCPSinkConfig(outHost, outPort, &Encoder{})) | |||
| // ToSink(app.MakeTCPSinkConfig(outHost, outPort, &Encoder{})) | |||
This commit adds the Pony side of support for to_sinks functionality with the Go API, completing the Go API support for multiple sinks per pipeline.
This pull request was closed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #2060