You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Read your article with great interest and reaching hoping you can clarify how Akka/Actors can be used with Spark.
I am confused by articles about deprecation and removal of Akka in Spark documentation. Can Akka/Actors be used in a Spark solution still, or is it not recommended/deprecated.
Reason asking, is investigating how to solve more complex workflows/dataflows that may lend themselves to be event driven. When a step finished, fires event (sends a message), and Actor could be listening for event, and execute the next step. Current simpler implementation in PoC is using Oozie.
And investigating whether this workflow/dataflow engine could be within Spark itself, for example with Akka/Actors, or perhaps external (Askaban, Airflow, other).
Thanks, Kurt
The text was updated successfully, but these errors were encountered:
Hi Richard,
Read your article with great interest and reaching hoping you can clarify how Akka/Actors can be used with Spark.
I am confused by articles about deprecation and removal of Akka in Spark documentation. Can Akka/Actors be used in a Spark solution still, or is it not recommended/deprecated.
** (not here) https://github.com/apache/spark/tree/v2.1.0/examples/src/main/scala/org/apache/spark/examples/streaming
** (but here) https://github.com/apache/spark/tree/branch-1.6/examples/src/main/scala/org/apache/spark/examples/streaming
Or is the issue how you create Akka solution?
Don't integrate with Spark's Akka implementation
** class FeederActor extends Actor
But create your own, as you have done in your article (https://www.spantree.net/blog/2016/10/18/spark-akka-sangria.html)
Reason asking, is investigating how to solve more complex workflows/dataflows that may lend themselves to be event driven. When a step finished, fires event (sends a message), and Actor could be listening for event, and execute the next step. Current simpler implementation in PoC is using Oozie.
And investigating whether this workflow/dataflow engine could be within Spark itself, for example with Akka/Actors, or perhaps external (Askaban, Airflow, other).
Thanks, Kurt
The text was updated successfully, but these errors were encountered: