[SPARK-6862][Streaming][WebUI] Add BatchPage to display details of a batch #5473

zsxwing · 2015-04-11T17:34:47Z

This is an initial commit for SPARK-6862. Once SPARK-6796 is merged, I will add the links to StreamingPage so that the user can jump to BatchPage.

Screenshots:

zsxwing · 2015-04-11T17:35:13Z

My test codes:

import org.apache.spark._
import org.apache.spark.streaming._

object StreamingApp {

  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setMaster("local[2]").setAppName("NetworkWordCount").set("spark.streaming.concurrentJobs", "3")
    val ssc = new StreamingContext(conf, Seconds(10))
    val lines = ssc.socketTextStream("localhost", 9999)
    val words = lines.flatMap(_.split(" "))
    val pairs = words.map(word => {
      Thread.sleep(1000)
      (word, 1)
    })
    val wordCounts = pairs.reduceByKey((x: Int, y: Int) => x + y, 2)
    wordCounts.foreachRDD {
      rdd => rdd.foreach { v =>
        println(v)
      }
    }
    wordCounts.foreachRDD { rdd =>
      rdd.foreach { v =>
        println(v)
      }
      rdd.foreach { v =>
        println(v)
        //throw new RuntimeException("Oops!")
      }
    }
    ssc.start()
    ssc.awaitTermination()
  }
}

zsxwing · 2015-04-11T17:35:25Z

cc @tdas

SparkQA · 2015-04-11T19:06:29Z

Test build #30084 has finished for PR 5473 at commit 0c7b2eb.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class BatchPage(parent: StreamingTab) extends WebUIPage("batch")
This patch does not change any dependencies.

tdas · 2015-04-13T09:13:49Z

streaming/src/main/scala/org/apache/spark/streaming/scheduler/BatchInfo.scala

+  /**
+   * The number of recorders received by the receivers in this batch.
+   */
+  def numRecords: Long = receivedBlockInfo.map { case (_, infos) =>


+1 on exposing this.

tdas · 2015-04-13T09:20:47Z

Its looking pretty good! Here are some preliminary comments. I will do a more detailed pass tomorrow during the day.

There should be tests in the StreamingListenerSuite that tests the local property setting.
In the screenshots, is there any way to limit the max col size of the error? When there is error, it squishes all the other columns.

tdas · 2015-04-13T09:28:06Z

streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala

+      <th>Duration</th>
+      <th class="sorttable_nosort">Stages: Succeeded/Total</th>
+      <th class="sorttable_nosort">Tasks (for all stages): Succeeded/Total</th>
+      <th>Last Error</th>


Why is this Last Error and not just Error

I was thinking Last Error of tasks. But here is Stage. I agree it should be Error. Fixed.

…[streaming]

tdas · 2015-04-14T05:26:06Z

@zsxwing Any thoughts on this?

zsxwing · 2015-04-14T05:40:30Z

There should be tests in the StreamingListenerSuite that tests the local property setting.

As I explained in https://github.com/apache/spark/pull/5288#issuecomment-92579899, we may get wrong properties in SparkListenerJobStart.properties. Because the tests will be non-deterministic, I think it's better to add them after @JoshRosen fixes the issue.

In the screenshots, is there any way to limit the max col size of the error? When there is error, it squishes all the other columns.

Let me try some css style to fix it.

tdas · 2015-04-14T05:49:16Z

Why would we get wrong properties in this case? We are explicitly setting it in the thread that is launching the Spark job, so there is no question of inheriting local properties by child threads, etc. So that issue should not affect this case, isnt it? Or am I missing something?

zsxwing · 2015-04-14T06:01:55Z

Why would we get wrong properties in this case? We are explicitly setting it in the thread that is launching the Spark job, so there is no question of inheriting local properties by child threads, etc. So that issue should not affect this case, isnt it? Or am I missing something?

Without #5288, SparkListenerJobStart is created like SparkListenerJobStart(..., localProperties.get). So the properties in SparkListenerJobStart is the same object of localProperties.get, which is supposed to be limited in its thread.

And after job.run, we will clear some properties in localProperties.get, because SparkListenerJobStart.properties is the same object of localProperties.get, if job.run is very fast, we may not find the properties we need, because they have been cleared.

zsxwing · 2015-04-14T06:32:06Z

I set the max-width of the error column to 300px. Here is the new screenshot:

zsxwing · 2015-04-14T06:33:02Z

Please ignore that Processing time and Total delay are -. It's an issue about JobScheduler. I will open a JIRA for it.

zsxwing · 2015-04-14T07:05:07Z

Please ignore that Processing time and Total delay are -. It's an issue about JobScheduler. I will open a JIRA for it.

After thinking it carefully, this may not be an issue, since StreamingContext should be stopped in such case. @tdas, do you think if we still need to show Processing time and Total delay in this case?

SparkQA · 2015-04-14T07:07:17Z

Test build #30220 has finished for PR 5473 at commit 0b226f9.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class BatchPage(parent: StreamingTab) extends WebUIPage("batch")
This patch does not change any dependencies.

SparkQA · 2015-04-14T07:59:22Z

Test build #30223 has finished for PR 5473 at commit 7168807.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class BatchPage(parent: StreamingTab) extends WebUIPage("batch")
This patch does not change any dependencies.

Conflicts: streaming/src/test/scala/org/apache/spark/streaming/UISeleniumSuite.scala

SparkQA · 2015-04-15T10:01:29Z

Test build #30330 has finished for PR 5473 at commit 15bdf9b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class BatchPage(parent: StreamingTab) extends WebUIPage("batch")
This patch does not change any dependencies.

zsxwing · 2015-04-15T18:05:48Z

@tdas this is ready for code review.

Conflicts: streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobScheduler.scala

SparkQA · 2015-04-20T13:09:31Z

Test build #30585 has finished for PR 5473 at commit 35ffd80.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class BatchPage(parent: StreamingTab) extends WebUIPage("batch")
This patch does not change any dependencies.

zsxwing · 2015-04-22T07:45:36Z

In addition, because of rowspan, the table cannot be sortable.

tdas · 2015-04-28T20:27:15Z

High level points as discussed offline.

We are planning to increase the number of retainedBatches, hence we need to be careful about all the data that we retain in memory. And its not a good idea to retain BatchInfo objects in memory because it contains a lot of arbitrary ReceivedBlockInfo objects, which can have arbitrary metadata ( #5732 ). So a better design would be to train only BatchUIData objects which contains only the data necessary for rendering the UI (timings, numrecords, etc.). So waitingBatches, runningBatches, completedBatches should have only BatchUIData objects.

SparkQA · 2015-04-29T05:43:28Z

Test build #31237 has finished for PR 5473 at commit 087ba98.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

tdas · 2015-04-29T08:33:27Z

streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingJobProgressListener.scala

-  private val runningBatchInfos = new HashMap[Time, BatchInfo]
-  private val completedBatchInfos = new Queue[BatchInfo]
-  private val batchInfoLimit = ssc.conf.getInt("spark.streaming.ui.retainedBatches", 100)
+  private val waitingBatchUIDatas = new HashMap[Time, BatchUIData]


Datas sounds really weird. Just leave it as BatchUIData. Or if you want to follow other hashmaps naming styles timeToWaitingBatchUIData. Either is fine by me.

zsxwing · 2015-04-29T14:33:45Z

@tdas UISeleniumSuite is unstable because we may get wrong batchTime and outputOpId properties due to #5288.

SparkQA · 2015-04-29T16:07:25Z

Test build #31284 has finished for PR 5473 at commit b380cfb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

SparkQA · 2015-04-29T16:21:56Z

Test build #31287 has finished for PR 5473 at commit 0727d35.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

tdas · 2015-04-29T20:32:59Z

I have merged #5288. Let me take another look at this PR.

tdas · 2015-04-29T20:33:18Z

retest this please.

tdas · 2015-04-29T21:48:24Z

rest this please.

SparkQA · 2015-04-29T22:15:25Z

Test build #31323 has finished for PR 5473 at commit 0727d35.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

tdas · 2015-04-29T22:49:44Z

retest this please.

SparkQA · 2015-04-29T22:53:05Z

Test build #733 has started for PR 5473 at commit 0727d35.

zsxwing · 2015-04-29T23:13:17Z

retest this please.

SparkQA · 2015-04-30T00:56:17Z

Test build #31346 has finished for PR 5473 at commit 0727d35.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

tdas · 2015-04-30T01:21:39Z

The test has passed. Merging this! Thanks @zsxwing !

…a batch This is an initial commit for SPARK-6862. Once SPARK-6796 is merged, I will add the links to StreamingPage so that the user can jump to BatchPage. Screenshots: ![success](https://cloud.githubusercontent.com/assets/1000778/7102439/bbe75406-e0b3-11e4-84fe-3e6de629a49a.png) ![failure](https://cloud.githubusercontent.com/assets/1000778/7102440/bc124454-e0b3-11e4-921a-c8b39d6b61bc.png) Author: zsxwing <zsxwing@gmail.com> Closes apache#5473 from zsxwing/SPARK-6862 and squashes the following commits: 0727d35 [zsxwing] Change BatchUIData to a case class b380cfb [zsxwing] Add createJobStart to eliminate duplicate codes 9a3083d [zsxwing] Rename XxxDatas -> XxxData 087ba98 [zsxwing] Refactor BatchInfo to store only necessary fields cb62e4f [zsxwing] Use Seq[(OutputOpId, SparkJobId)] to store the id relations 72f8e7e [zsxwing] Add unit tests for BatchPage 1282b10 [zsxwing] Handle some corner cases and add tests for StreamingJobProgressListener 77a69ae [zsxwing] Refactor codes as per TD's comments 35ffd80 [zsxwing] Merge branch 'master' into SPARK-6862 15bdf9b [zsxwing] Add batch links and unit tests 4bf66b6 [zsxwing] Merge branch 'master' into SPARK-6862 7168807 [zsxwing] Limit the max width of the error message and fix nits in the UI 0b226f9 [zsxwing] Change 'Last Error' to 'Error' fc98a43 [zsxwing] Put clearing local properties to finally and remove redundant private[streaming] 0c7b2eb [zsxwing] Add BatchPage to display details of a batch

Add BatchPage to display details of a batch

0c7b2eb

tdas reviewed Apr 13, 2015
View reviewed changes

zsxwing mentioned this pull request Apr 14, 2015

[SPARK-6629] cancelJobGroup() may not work for jobs whose job groups are inherited from parent threads #5288

Closed

Put clearing local properties to finally and remove redundant private…

fc98a43

…[streaming]

Change 'Last Error' to 'Error'

0b226f9

Limit the max width of the error message and fix nits in the UI

7168807

zsxwing added 2 commits April 15, 2015 15:00

Merge branch 'master' into SPARK-6862

4bf66b6

Conflicts: streaming/src/test/scala/org/apache/spark/streaming/UISeleniumSuite.scala

Add batch links and unit tests

15bdf9b

zsxwing changed the title ~~[SPARK-6862][Streaming][WebUI][WIP] Add BatchPage to display details of a batch~~ [SPARK-6862][Streaming][WebUI] Add BatchPage to display details of a batch Apr 15, 2015

Merge branch 'master' into SPARK-6862

35ffd80

Conflicts: streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobScheduler.scala

Refactor BatchInfo to store only necessary fields

087ba98

tdas reviewed Apr 29, 2015
View reviewed changes

zsxwing added 2 commits April 29, 2015 07:29

Rename XxxDatas -> XxxData

9a3083d

Add createJobStart to eliminate duplicate codes

b380cfb

Change BatchUIData to a case class

0727d35

asfgit closed this in 1b7106b Apr 30, 2015

zsxwing deleted the SPARK-6862 branch April 30, 2015 03:34

[SPARK-6862][Streaming][WebUI] Add BatchPage to display details of a batch #5473

[SPARK-6862][Streaming][WebUI] Add BatchPage to display details of a batch #5473

Conversation

zsxwing commented Apr 11, 2015

zsxwing commented Apr 11, 2015

zsxwing commented Apr 11, 2015

SparkQA commented Apr 11, 2015

tdas Apr 13, 2015

Choose a reason for hiding this comment

tdas commented Apr 13, 2015

tdas Apr 13, 2015

Choose a reason for hiding this comment

zsxwing Apr 14, 2015

Choose a reason for hiding this comment

tdas commented Apr 14, 2015

zsxwing commented Apr 14, 2015

tdas commented Apr 14, 2015

zsxwing commented Apr 14, 2015

zsxwing commented Apr 14, 2015

zsxwing commented Apr 14, 2015

zsxwing commented Apr 14, 2015

SparkQA commented Apr 14, 2015

SparkQA commented Apr 14, 2015

SparkQA commented Apr 15, 2015

zsxwing commented Apr 15, 2015

SparkQA commented Apr 20, 2015

zsxwing commented Apr 22, 2015

tdas commented Apr 28, 2015

SparkQA commented Apr 29, 2015

tdas Apr 29, 2015

Choose a reason for hiding this comment

zsxwing commented Apr 29, 2015

SparkQA commented Apr 29, 2015

SparkQA commented Apr 29, 2015

tdas commented Apr 29, 2015

tdas commented Apr 29, 2015

tdas commented Apr 29, 2015

SparkQA commented Apr 29, 2015

tdas commented Apr 29, 2015

SparkQA commented Apr 29, 2015

zsxwing commented Apr 29, 2015

SparkQA commented Apr 30, 2015

tdas commented Apr 30, 2015