SPARK-1830 Deploy failover, Make Persistence engine and LeaderAgent Pluggable #771

ScrapCodes · 2014-05-14T12:17:19Z

No description provided.

AmplabJenkins · 2014-05-14T12:17:57Z

Merged build triggered.

AmplabJenkins · 2014-05-14T12:18:07Z

Merged build started.

AmplabJenkins · 2014-05-14T12:56:17Z

Merged build finished.

AmplabJenkins · 2014-05-14T12:56:18Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14979/

ScrapCodes · 2014-05-15T07:43:17Z

Jenkins, test this please.

AmplabJenkins · 2014-05-15T07:47:59Z

Merged build triggered.

AmplabJenkins · 2014-05-15T07:48:04Z

Merged build started.

AmplabJenkins · 2014-05-15T08:28:55Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-15T08:28:55Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15021/

ScrapCodes · 2014-05-19T10:56:23Z

@aarondav Mind taking a look ?

aarondav · 2014-05-19T17:57:06Z

@ScrapCodes, I definitely like the idea. I think we need to update the API a little before it's ready to be publicly accessible, though. First, LeaderElectionAgent and PersistenceEngine should be @DeveloperApi and not private[spark].

Second, LeaderElectionAgent should be refactored to not be an Actor (there is a TODO since the Curator refactor, I think it just needs to be done before we can finalize this API). This should be pretty straightforward to do, I would just add a trait like

trait LeaderElectable {
  def electedLeader()
  def revokedLeadership()
}

which Master implements by simply calling self ! ElectedLeader, for instance, to ensure we maintain the thread safety of the actor without leaking the actor abstraction.

aarondav · 2014-05-19T18:08:07Z

core/src/main/scala/org/apache/spark/deploy/master/Master.scala

@@ -143,6 +146,9 @@ private[spark] class Master(
    leaderElectionAgent = RECOVERY_MODE match {
        case "ZOOKEEPER" =>
          context.actorOf(Props(classOf[ZooKeeperLeaderElectionAgent], self, masterUrl, conf))
+        case "CUSTOM" =>
+          val clazz = Class.forName(conf.get("spark.deploy.recoveryMode.leaderAgentActor"))
+          context.actorOf(Props(clazz, self, masterUrl, conf))


It may be a good idea to abstract out the creation of the persistenceEngine and leaderElectionAgent into something like

abstract class StandaloneRecoveryModeFactory(conf: SparkConf) { def createPersistenceEngine(): PersistenceEngine def createLeaderElectionAgent(master: LeaderElectable, masterUrl: String): LeaderElectionAgent }

with 2 subclasses, FileSystemRecoveryModeFactory and ZooKeeperRecoveryModeFactory (maybe the name "RecoveryModeFactory" is not so good).

The advantage of this approach is that we formalize the arguments needed for constructing the 2 APIs, and centralize the creation of everything necessary for one recovery mode. Thoughts?

AmplabJenkins · 2014-06-05T11:43:00Z

Merged build triggered.

ScrapCodes · 2014-06-05T11:44:10Z

@aarondav I am not 100% sure about the idea but for a moment I felt may be we could club PersistenceEngine and LeaderElectionAgent together into one and call it FailoverAgent ? I am not supporting it but it sort of occurred to me so thought of sharing it. Take a look at this updated patch !

AmplabJenkins · 2014-06-05T12:23:07Z

Merged build started.

AmplabJenkins · 2014-06-05T12:24:42Z

Merged build finished.

AmplabJenkins · 2014-06-05T12:24:42Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15479/

aarondav · 2014-06-05T17:30:00Z

That does sound reasonable, especially since the LeaderElectionAgent relies on the behavior of the PersistenceEngine in order to actually enable recovery.

AmplabJenkins · 2014-06-06T05:02:49Z

Merged build triggered.

AmplabJenkins · 2014-06-06T05:51:16Z

Merged build started.

AmplabJenkins · 2014-06-06T05:53:03Z

Merged build finished.

AmplabJenkins · 2014-06-06T05:53:04Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15496/

AmplabJenkins · 2014-06-06T06:02:51Z

Merged build triggered.

AmplabJenkins · 2014-06-06T06:55:32Z

Merged build started.

AmplabJenkins · 2014-06-06T07:35:13Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-06T07:35:13Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15498/

ScrapCodes · 2014-09-01T06:13:44Z

Hey @aarondav, Do you think its worth having in its current condition ? I can rebase it ofcourse. I was actually unsure of changing it further.

pwendell · 2014-09-21T05:36:52Z

@aarondav could you give feedback one way or the other on this?

aarondav · 2014-09-22T16:10:15Z

core/src/main/scala/org/apache/spark/deploy/master/FileSystemPersistenceEngine.scala

    val out = new FileOutputStream(file)
-    out.write(serialized)
+    out.write(serialized.array())


Perhaps we could use serializeStream() instead to avoid relying on a ByteBuffer's array()?

I did not understand, what you meant here.

I was just suggesting that we could do

val out = serializer.serializeStream(new FileOutputStream(file)) out.writeObject(value) out.close()

It's a very minor point about ByteBuffers not always having a valid array().

aarondav · 2014-10-21T17:06:03Z

core/src/main/scala/org/apache/spark/deploy/master/PersistenceEngine.scala

+@DeveloperApi
+trait PersistenceEngine {
+
+  def persist(name: String, obj: Object)


Please add documentation to these three methods to make the requirements clear.

…luggable. Refactored Leader Election agent and added a RecoveryModeFactory. Implemented new proposal with some convenient modifications. Added a read method.

SparkQA · 2014-10-22T07:54:47Z

QA tests have started for PR 771 at commit fef35ec.

This patch merges cleanly.

SparkQA · 2014-10-22T07:57:35Z

QA tests have finished for PR 771 at commit fef35ec.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait LeaderElectionAgent
- trait LeaderElectable
- trait PersistenceEngine
- abstract class StandaloneRecoveryModeFactory(conf: SparkConf)

AmplabJenkins · 2014-10-22T07:57:36Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22028/
Test FAILed.

SparkQA · 2014-10-22T08:05:06Z

QA tests have started for PR 771 at commit 29ba440.

This patch merges cleanly.

SparkQA · 2014-10-22T08:59:37Z

QA tests have finished for PR 771 at commit 29ba440.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait LeaderElectionAgent
- trait LeaderElectable
- trait PersistenceEngine
- abstract class StandaloneRecoveryModeFactory(conf: SparkConf)

AmplabJenkins · 2014-10-22T08:59:40Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22030/
Test FAILed.

ScrapCodes · 2014-10-24T07:09:59Z

Jenkins, retest this please.

SparkQA · 2014-10-24T07:14:40Z

Test build #22127 has started for PR 771 at commit 29ba440.

This patch merges cleanly.

SparkQA · 2014-10-24T08:23:02Z

Test build #22127 has finished for PR 771 at commit 29ba440.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait LeaderElectionAgent
- trait LeaderElectable
- trait PersistenceEngine
- abstract class StandaloneRecoveryModeFactory(conf: SparkConf)

AmplabJenkins · 2014-10-24T08:23:05Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22127/
Test PASSed.

ScrapCodes · 2014-11-11T09:54:18Z

Did I miss something ?

aarondav · 2014-11-11T17:29:19Z

LGTM, merging into master. Thanks!

…luggable Author: Prashant Sharma <prashant.s@imaginea.com> Closes apache#771 from ScrapCodes/deploy-failover-pluggable and squashes the following commits: 29ba440 [Prashant Sharma] fixed a compilation error fef35ec [Prashant Sharma] Code review 57ee6f0 [Prashant Sharma] SPARK-1830 Deploy failover, Make Persistence engine and LeaderAgent Pluggable.

aarondav reviewed May 19, 2014
View reviewed changes

aarondav reviewed Sep 22, 2014
View reviewed changes

aarondav reviewed Oct 21, 2014
View reviewed changes

ScrapCodes added 2 commits October 22, 2014 12:30

SPARK-1830 Deploy failover, Make Persistence engine and LeaderAgent P…

57ee6f0

…luggable. Refactored Leader Election agent and added a RecoveryModeFactory. Implemented new proposal with some convenient modifications. Added a read method.

Code review

fef35ec

ScrapCodes force-pushed the deploy-failover-pluggable branch from d2a1c62 to fef35ec Compare October 22, 2014 07:50

fixed a compilation error

29ba440

asfgit closed this in deefd9d Nov 11, 2014

aarondav mentioned this pull request Nov 23, 2014

[SPARK-4377] Fixed serialization issue by switching to akka provided serializer. #3402

Closed

SPARK-1830 Deploy failover, Make Persistence engine and LeaderAgent Pluggable #771

SPARK-1830 Deploy failover, Make Persistence engine and LeaderAgent Pluggable #771

Conversation

ScrapCodes commented May 14, 2014

AmplabJenkins commented May 14, 2014

AmplabJenkins commented May 14, 2014

AmplabJenkins commented May 14, 2014

AmplabJenkins commented May 14, 2014

ScrapCodes commented May 15, 2014

AmplabJenkins commented May 15, 2014

AmplabJenkins commented May 15, 2014

AmplabJenkins commented May 15, 2014

AmplabJenkins commented May 15, 2014

ScrapCodes commented May 19, 2014

aarondav commented May 19, 2014

aarondav May 19, 2014

Choose a reason for hiding this comment

AmplabJenkins commented Jun 5, 2014

ScrapCodes commented Jun 5, 2014

AmplabJenkins commented Jun 5, 2014

AmplabJenkins commented Jun 5, 2014

AmplabJenkins commented Jun 5, 2014

aarondav commented Jun 5, 2014

AmplabJenkins commented Jun 6, 2014

AmplabJenkins commented Jun 6, 2014

AmplabJenkins commented Jun 6, 2014

AmplabJenkins commented Jun 6, 2014

AmplabJenkins commented Jun 6, 2014

AmplabJenkins commented Jun 6, 2014

AmplabJenkins commented Jun 6, 2014

AmplabJenkins commented Jun 6, 2014

ScrapCodes commented Sep 1, 2014

pwendell commented Sep 21, 2014

aarondav Sep 22, 2014

Choose a reason for hiding this comment

ScrapCodes Sep 30, 2014

Choose a reason for hiding this comment

aarondav Sep 30, 2014

Choose a reason for hiding this comment

aarondav Oct 21, 2014

Choose a reason for hiding this comment

SparkQA commented Oct 22, 2014

SparkQA commented Oct 22, 2014

AmplabJenkins commented Oct 22, 2014

SparkQA commented Oct 22, 2014

SparkQA commented Oct 22, 2014

AmplabJenkins commented Oct 22, 2014

ScrapCodes commented Oct 24, 2014

SparkQA commented Oct 24, 2014

SparkQA commented Oct 24, 2014

AmplabJenkins commented Oct 24, 2014

ScrapCodes commented Nov 11, 2014

aarondav commented Nov 11, 2014