[SPARK-3454] Expose JSON representation of data shown in WebU #2333

sarutak · 2014-09-09T10:08:57Z

No description provided.

SparkQA · 2014-09-09T10:49:46Z

QA tests have started for PR 2333 at commit 1cbf7d5.

This patch merges cleanly.

SparkQA · 2014-09-09T10:54:20Z

QA tests have finished for PR 2333 at commit 1cbf7d5.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

…g-feature

SparkQA · 2014-09-11T06:53:31Z

QA tests have started for PR 2333 at commit f7958b0.

This patch merges cleanly.

SparkQA · 2014-09-11T06:54:35Z

QA tests have finished for PR 2333 at commit f7958b0.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-09-11T07:53:23Z

QA tests have started for PR 2333 at commit 8850706.

This patch merges cleanly.

SparkQA · 2014-09-11T08:55:37Z

QA tests have finished for PR 2333 at commit 8850706.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

…sPage#renderJson

SparkQA · 2014-09-11T09:53:20Z

QA tests have started for PR 2333 at commit c108200.

This patch merges cleanly.

…renderJson

SparkQA · 2014-09-11T10:53:50Z

QA tests have started for PR 2333 at commit a349d0e.

This patch merges cleanly.

SparkQA · 2014-09-11T10:54:45Z

QA tests have finished for PR 2333 at commit c108200.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-09-11T11:58:51Z

QA tests have finished for PR 2333 at commit a349d0e.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

andrewor14 · 2014-09-11T19:28:03Z

core/src/main/scala/org/apache/spark/ui/env/EnvironmentPage.scala

@@ -26,6 +29,23 @@ import org.apache.spark.ui.{UIUtils, WebUIPage}
 private[ui] class EnvironmentPage(parent: EnvironmentTab) extends WebUIPage("") {
  private val listener = parent.listener

+  override def renderJson(request: HttpServletRequest): JValue = {
+    val jvmInfoJson =
+      ("RUntime Informationf" -> listener.jvmInformation.foldLeft(JObject())(_ ~ _))


There are a lot of typos in this line...

Can you show what the final JSON looks like? Since this is a map it might be good to convert this faithfully, e.g.

{ "Spark Properties": { "spark.master":"local" "spark.app.name":"spark-shell" ... } ... }

This is the result of JSON for environment.

{ "Runtime Information" : { "Java Home" : "/usr/java/jdk1.7.0_65/jre", "Java Version" : "1.7.0_65 (Oracle Corporation)", "Scala Version" : "version 2.10.4" }, "Spark Properties" : { "spark.serializer" : "org.apache.spark.serializer.KryoSerializer", ... "spark.tachyonStore.folderName" : "spark-a8ac134b-187d-4d4b-b9ef-140e626ee96c" }, "System Properties" : { "java.io.tmpdir" : "/tmp", ... "java.version" : "1.7.0_65", "sun.io.unicode.encoding" : "UnicodeLittle" }, "Classpath Entries" : { "/home/sarutak/work/spark/sql/hive/target/scala-2.10/test-classes" : "System Classpath", ... "/home/sarutak/work/spark/assembly/target/scala-2.10/spark-assembly-1.1.0-SNAPSHOT-hadoop2.2.0.jar" : "System Classpath" }

}

SparkQA · 2014-09-12T03:09:16Z

QA tests have started for PR 2333 at commit 72c0644.

This patch merges cleanly.

sarutak · 2014-09-12T03:16:37Z

@andrewor14 Thank you for reviewing this PR.
I agree with removing category from the JSON blob and have done.

SparkQA · 2014-09-12T04:13:37Z

QA tests have finished for PR 2333 at commit 72c0644.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

andrewor14 · 2014-09-12T20:13:04Z

@JoshRosen

…g-feature

… json-exposing-feature

…g-feature

SparkQA · 2014-09-14T02:14:17Z

QA tests have started for PR 2333 at commit 7f51a4f.

This patch merges cleanly.

SparkQA · 2014-09-14T03:19:50Z

QA tests have finished for PR 2333 at commit 7f51a4f.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

…g-feature

sarutak · 2014-09-15T05:49:18Z

test this please.

SparkQA · 2014-09-15T05:54:32Z

QA tests have started for PR 2333 at commit d41b3ca.

This patch merges cleanly.

SparkQA · 2014-09-15T07:00:44Z

QA tests have finished for PR 2333 at commit d41b3ca.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class JavaSparkContext(val sc: SparkContext)
- throw new IllegalStateException("The main method in the given main class must be static")
- class TaskCompletionListenerException(errorMessages: Seq[String]) extends Exception
- class Dummy(object):
- class RatingDeserializer(FramedSerializer):
- class Encoder[T <: NativeType](columnType: NativeColumnType[T]) extends compression.Encoder[T]
- class Encoder[T <: NativeType](columnType: NativeColumnType[T]) extends compression.Encoder[T]
- class Encoder[T <: NativeType](columnType: NativeColumnType[T]) extends compression.Encoder[T]
- class Encoder extends compression.Encoder[IntegerType.type]
- class Decoder(buffer: ByteBuffer, columnType: NativeColumnType[IntegerType.type])
- class Encoder extends compression.Encoder[LongType.type]
- class Decoder(buffer: ByteBuffer, columnType: NativeColumnType[LongType.type])
- class JavaStreamingContext(val ssc: StreamingContext) extends Closeable

sarutak · 2014-09-16T00:13:07Z

Can anyone review this if you have time?

…g-feature

sarutak · 2014-09-17T21:58:08Z

test this please.

SparkQA · 2014-09-17T22:04:23Z

QA tests have started for PR 2333 at commit 7b4d6eb.

This patch merges cleanly.

SparkQA · 2014-09-17T23:10:33Z

QA tests have finished for PR 2333 at commit 7b4d6eb.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

andrewor14 · 2014-09-17T23:26:43Z

@sarutak, I believe @JoshRosen is working on a more general framework for extracting the info displayed on the UI as JSON, so there's a chance that we won't go with the approach here. Just a heads up.

sarutak · 2014-09-17T23:32:42Z

@andrewor14 Thank you for notification!
Actually, I need JSON representation for #2342 .
I'm planning to parse JSON to use D3.

JoshRosen · 2014-09-17T23:47:13Z

@sarutak In the long run, I'd be interested in re-writing the UI in terms of a richer REST API that exposes data as JSON, exactly for the visualization use-case that you mentioned. This change (Javascript-based rendering, etc) is blocked by a couple of other features right now. I'm swamped with a bunch of other work, but I'll try to get a public design draft posted in the next couple of days.

sarutak · 2014-09-17T23:58:02Z

Thanks @JoshRosen !
So, for now, I use self implementation for #2342 and I'll use the feature you'll try.

JoshRosen · 2014-09-22T18:42:50Z

I've opened SPARK-3644 as a forum for discussing the design of a REST API; sorry for the delay.

sarutak · 2014-09-24T14:28:16Z

Thank you for you work @JoshRosen !
I'll check it out.

squito · 2015-01-26T20:32:09Z

Hi @sarutak thanks for your work on this. Josh's other PR #2696 has been merged for a while now. I'm gonna take another crack at implementing this on top of the lastest changes and getting this integrated into the history server too. Can you please close this PR (unless you are planning on updating it in the very near future)?

Hopefully that will let your other PR #2342 to get merged in -- looks really cool!

andrewor14 · 2015-02-19T22:46:29Z

@sarutak thanks a lot for working on this. There is another patch #4435 that provides a more elaborate implementation of the functionality here. Given that this PR has mostly gone stale at this point, would you mind closing this?

sarutak · 2015-02-20T08:41:35Z

OK. I close this PR.

sarutak changed the title ~~Implemented renderJson for StagePage~~ (WIP) [SPARK-3454] Expose JSON representation of data shown in WebU Sep 9, 2014

sarutak added 2 commits September 10, 2014 18:46

Implemented renderJson for StagePage

353a423

Implemented renderJson for EnvironmentPage

b8578a7

sarutak force-pushed the json-exposing-feature branch from 3cbb461 to b8578a7 Compare September 10, 2014 09:46

sarutak added 4 commits September 10, 2014 19:07

tmp

937c8b7

Merge branch 'master' of git://git.apache.org/spark into json-exposin…

437c241

…g-feature

Implemented renderJson for ExecutorPage

5556856

Implemented renderJson for JobProgressPage

f7958b0

sarutak added 3 commits September 11, 2014 16:17

Implemented renderJson for PoolPage

9e0010a

Modified style

8850706

Implemented renderJson for StoragePage

e537be6

sarutak added 2 commits September 11, 2014 18:20

Implemented renderJson for RDDPage

c108200

Added "Scheduling Mode" field in the json data returned by JobProgres…

2f8f9f3

…sPage#renderJson

Added "Stage Summary" field to the json data returned from StagePage#…

a349d0e

…renderJson

sarutak changed the title ~~(WIP) [SPARK-3454] Expose JSON representation of data shown in WebU~~ [[SPARK-3454] Expose JSON representation of data shown in WebU Sep 11, 2014

sarutak changed the title ~~[[SPARK-3454] Expose JSON representation of data shown in WebU~~ [SPARK-3454] Expose JSON representation of data shown in WebU Sep 11, 2014

andrewor14 reviewed Sep 11, 2014
View reviewed changes

Modified variable name in StagePage.scala

72c0644

sarutak added 3 commits September 13, 2014 16:38

Merge branch 'master' of git://git.apache.org/spark into json-exposin…

a2dbe2e

…g-feature

Merge branch 'json-exposing-feature' of github.com:sarutak/spark into…

b50a383

… json-exposing-feature

Merge branch 'master' of git://git.apache.org/spark into json-exposin…

7f51a4f

…g-feature

Merge branch 'master' of git://git.apache.org/spark into json-exposin…

d41b3ca

…g-feature

Merge branch 'master' of git://git.apache.org/spark into json-exposin…

7b4d6eb

…g-feature

sarutak closed this Feb 20, 2015

sarutak deleted the json-exposing-feature branch May 18, 2015 01:52

[SPARK-3454] Expose JSON representation of data shown in WebU #2333

[SPARK-3454] Expose JSON representation of data shown in WebU #2333

Conversation

sarutak commented Sep 9, 2014

SparkQA commented Sep 9, 2014

SparkQA commented Sep 9, 2014

SparkQA commented Sep 11, 2014

SparkQA commented Sep 11, 2014

SparkQA commented Sep 11, 2014

SparkQA commented Sep 11, 2014

SparkQA commented Sep 11, 2014

SparkQA commented Sep 11, 2014

SparkQA commented Sep 11, 2014

SparkQA commented Sep 11, 2014

andrewor14 Sep 11, 2014

Choose a reason for hiding this comment

andrewor14 Sep 11, 2014

Choose a reason for hiding this comment

sarutak Sep 12, 2014

Choose a reason for hiding this comment

SparkQA commented Sep 12, 2014

sarutak commented Sep 12, 2014

SparkQA commented Sep 12, 2014

andrewor14 commented Sep 12, 2014

SparkQA commented Sep 14, 2014

SparkQA commented Sep 14, 2014

sarutak commented Sep 15, 2014

SparkQA commented Sep 15, 2014

SparkQA commented Sep 15, 2014

sarutak commented Sep 16, 2014

sarutak commented Sep 17, 2014

SparkQA commented Sep 17, 2014

SparkQA commented Sep 17, 2014

andrewor14 commented Sep 17, 2014

sarutak commented Sep 17, 2014

JoshRosen commented Sep 17, 2014

sarutak commented Sep 17, 2014

JoshRosen commented Sep 22, 2014

sarutak commented Sep 24, 2014

squito commented Jan 26, 2015

andrewor14 commented Feb 19, 2015

sarutak commented Feb 20, 2015