Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

handle failure in spark getOrCreate #7470

Closed
scottdraves opened this issue Jun 2, 2018 · 0 comments
Closed

handle failure in spark getOrCreate #7470

scottdraves opened this issue Jun 2, 2018 · 0 comments

Comments

@scottdraves
Copy link
Contributor

scottdraves commented Jun 2, 2018

enter a bad master url, like spark://1.2.3.4:7077
after a timeout of a minute, i get a stack trace and then kernel dies.

2018-06-02 15:48:10:746 -0400 [Thread-0] ERROR SparkContext - Error initializing SparkContext.
java.lang.NullPointerException
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:567)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2516)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:918)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:910)
        at scala.Option.getOrElse(Option.scala:121)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:910)
        at com.twosigma.beakerx.widget.SparkEngineImpl.getOrCreate(SparkEngineImpl.java:86)
        at com.twosigma.beakerx.widget.SparkEngineImpl.configure(SparkEngineImpl.java:74)
        at com.twosigma.beakerx.widget.SparkUI.initSparkContext(SparkUI.java:165)
        at com.twosigma.beakerx.widget.SparkUI.lambda$createConnectButton$0(SparkUI.java:158)
        at com.twosigma.beakerx.widget.Button.onClick(Button.java:93)
        at com.twosigma.beakerx.widget.CompiledCodeRunner.runCommEvent(CompiledCodeRunner.java:54)
        at com.twosigma.beakerx.widget.Widget.lambda$handleCommEventSync$0(Widget.java:167)
        at com.twosigma.beakerx.handler.KernelHandlerWrapper.wrapBusyIdle(KernelHandlerWrapper.java:25)
        at com.twosigma.beakerx.widget.Widget.handleCommEventSync(Widget.java:167)
        at com.twosigma.beakerx.widget.Button.handleOnClick(Button.java:88)
        at com.twosigma.beakerx.kernel.comm.Comm.handleMsg(Comm.java:297)
        at com.twosigma.beakerx.kernel.handler.CommMsgHandler.handleMsg(CommMsgHandler.java:53)
        at com.twosigma.beakerx.kernel.handler.CommMsgHandler.lambda$handle$0(CommMsgHandler.java:43)
        at com.twosigma.beakerx.handler.KernelHandlerWrapper.wrapBusyIdle(KernelHandlerWrapper.java:25)
        at com.twosigma.beakerx.kernel.handler.CommMsgHandler.handle(CommMsgHandler.java:42)
        at com.twosigma.beakerx.kernel.handler.CommMsgHandler.handle(CommMsgHandler.java:33)
        at com.twosigma.beakerx.socket.KernelSocketsZMQ.handleShell(KernelSocketsZMQ.java:190)
        at com.twosigma.beakerx.socket.KernelSocketsZMQ.run(KernelSocketsZMQ.java:165)
  • there should be a spinner until it starts or errs.
  • if it errs, it should return to the form (not go to connected/status).
  • the form then shows the error in red, the the right of the "start" button.
  • if spark API reports an error string or code, show it in red too.

i found some other errors do show in red correctly, so this issue is really about handling this specific error.

adding the spinner is not high priority, that can go in a 2nd PR.

jaroslawmalekcodete added a commit that referenced this issue Jun 5, 2018
jaroslawmalekcodete added a commit that referenced this issue Jun 5, 2018
jaroslawmalekcodete added a commit that referenced this issue Jun 5, 2018
scottdraves pushed a commit that referenced this issue Jun 6, 2018
* #7470: handle errors

* #7470: SparkUI refactoring

* #7470: adjust spinner

* #7470: display error at the bottom of the form + add tooltip to spinner

* #7470: move error next to start button
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants