[SPARK-44394][CONNECT][WEBUI] Add a Spark UI page for Spark Connect #41964

jasonli-db · 2023-07-12T17:21:17Z

What changes were proposed in this pull request?

Add a new Spark UI page to display session and execution information for Spark Connect. This builds of the work in SPARK-43923 (#41443) that adds the relevant SparkListenerEvents and mirrors the ThriftServerPage in the Spark UI for JDBC/ODBC.

Why are the changes needed?

This gives users a way to access session and execution information for Spark Connect via the UI and provides the frontend interface for the related SparkListenerEvents.

Does this PR introduce any user-facing change?

Yes, it will add a new tab/page in the Spark UI

How was this patch tested?

Unit tests

connector/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala

jasonli-db · 2023-07-14T21:04:15Z

@juliuszsompolski @grundprinzip Can you help review this PR? This builds off of #41443, so the commit to look at is fcf0ca6.

Also cc @rednaxelafx @gengliangwang

gengliangwang · 2023-07-14T21:18:26Z

Since the Spark connect is more about SQL, shall we show the SQL execution links on the UI?

...connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala

juliuszsompolski

reviewed connect service parts, didn't review ui package.

.../connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectServer.scala

...connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala

gengliangwang · 2023-07-24T17:31:39Z

Since the Spark connect is more about SQL, shall we show the SQL execution links on the UI?

@jasonli-db What is your opinion of this one?

jasonli-db · 2023-07-24T17:39:13Z

Since the Spark connect is more about SQL, shall we show the SQL execution links on the UI?

@jasonli-db What is your opinion of this one?
I agree that adding the SQL execution links would be ideal and useful for the user. @juliuszsompolski Can you give your input on this and confirm whether or not this is currently possible. Do we have a link between SQL execution IDs and Connect requests?

...nnect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala

juliuszsompolski

Reviewed and LGTM on the connect part; didn't review details of the UI and state store part.

juliuszsompolski · 2023-07-26T10:12:23Z

...nnect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala

+    val executionIdOpt: Option[String] = Option(jobStart.properties)
+      .flatMap { p => Option(p.getProperty(SQLExecution.EXECUTION_ID_KEY)) }


Discussed offline with @jasonli-db and @jdesjean that for executions that don't start a Spark Job (e.g. SELECT 1), they will not be recorded. For that reason, as a followup, it would be best if jobTags could also be added to the SparkListenerSQLExecutionStart event.

juliuszsompolski · 2023-07-27T22:19:26Z

In the screenshot above, the close time and finish time column seems duplicated. And the execution time and duration also seems quite similar. Why do we have so many columns about the timings?

@gengliangwang Finish time is finish execution, close time is when all the results were transferred; execution time is the time query was executing, while duration is end to end. It is meaningful for queries with large results, or e.g. complex queries that spend a lot of time in optimizer. It's the same with Thriftserver UI.

jasonli-db · 2023-07-28T00:22:49Z

@jasonli-db Please try more tests. I did a quick test and find the SQL IDs need deduplication.

Thanks for catching this. It's been fixed

jasonli-db · 2023-07-28T00:23:27Z

Nit: we can have follow-up to improve the page when a session doesn't exist(e.g. evicted from KV store):

For example, id not existed is handled in the SQL page:

Addressed to match the behavior for the SQL page.

jasonli-db · 2023-07-28T00:29:52Z

OperationId could indeed maybe be more useful instead. If a user uses connect api like SparkSession.interruptOperation, they pass just operationId. The only time a user would want to use jobTag is if they wanted to directly call SparkContext.interruptJobsWithTag, which is a rather internal use. We could consider displaying SparkListenerConnectOperationStarted.sparkSessionTags, because these are uses settable (via connect SparkSession.addTag / removeTag / clearTags), and they are used by users to call connect SparkSession.interruptTag.

@juliuszsompolski @gengliangwang I've updated the screenshots. I added the operationID and sparkSessionTags. For now, I kept the jobTag since it seems it could still be used if only internally. I'll remove it if/when we come to a final consensus to do so. Please take a look!

gengliangwang · 2023-07-28T03:04:23Z

@jasonli-db I have to say to the current table is quite wide.

Can we at least put the operationID column behind the column "State" since it is not commonly useful?

A good way to resolve this is to have checkboxes and make operationID/sparkSessionTags/jobTag as additional metrics:

This can be done in a followup

...server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerAppStatusStore.scala

juliuszsompolski · 2023-07-28T15:57:35Z

...r/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala

+        </span>
+      </td>
+      <td>
+        {if (info.isExecutionActive) "RUNNING" else info.state}


LGTM, I think it makes more sense, thanks!

gengliangwang · 2023-07-28T20:37:59Z

@jasonli-db Thanks for the work. I will merge the PR once the tests are passed.

gengliangwang · 2023-07-29T16:19:55Z

Thanks, merging to master

gengliangwang · 2023-07-29T16:21:14Z

@jasonli-db This PR has conflict with the branch-3.5
Please cherry-pick it before the RC1 on Aug 1

## What changes were proposed in this pull request? Add a new Spark UI page to display session and execution information for Spark Connect. This builds of the work in SPARK-43923 (apache#41443) that adds the relevant SparkListenerEvents and mirrors the ThriftServerPage in the Spark UI for JDBC/ODBC. <img width="1709" alt="Screenshot 2023-07-27 at 11 29 22 PM" src="https://github.com/apache/spark/assets/65624911/934b7c69-3b44-460b-8fbb-36a9eb3f0798"> <img width="1716" alt="Screenshot 2023-07-27 at 11 29 15 PM" src="https://github.com/apache/spark/assets/65624911/33dbe6ab-44bf-49a5-ad4c-5ba4a476a1f0"> ### Why are the changes needed? This gives users a way to access session and execution information for Spark Connect via the UI and provides the frontend interface for the related SparkListenerEvents. ### Does this PR introduce _any_ user-facing change? Yes, it will add a new tab/page in the Spark UI ### How was this patch tested? Unit tests Closes apache#41964 from jasonli-db/spark-connect-ui. Authored-by: Jason Li <jason.li@databricks.com> Signed-off-by: Gengliang Wang <gengliang@apache.org> (cherry picked from commit f8786f0)

juliuszsompolski · 2023-07-29T16:39:51Z

Thanks @gengliangwang . Since the conflict was trivial (imports), I resolved and raised #42224 for branch-3.5.

## What changes were proposed in this pull request? Add a new Spark UI page to display session and execution information for Spark Connect. This builds of the work in SPARK-43923 (#41443) that adds the relevant SparkListenerEvents and mirrors the ThriftServerPage in the Spark UI for JDBC/ODBC. <img width="1709" alt="Screenshot 2023-07-27 at 11 29 22 PM" src="https://github.com/apache/spark/assets/65624911/934b7c69-3b44-460b-8fbb-36a9eb3f0798"> <img width="1716" alt="Screenshot 2023-07-27 at 11 29 15 PM" src="https://github.com/apache/spark/assets/65624911/33dbe6ab-44bf-49a5-ad4c-5ba4a476a1f0"> ### Why are the changes needed? This gives users a way to access session and execution information for Spark Connect via the UI and provides the frontend interface for the related SparkListenerEvents. ### Does this PR introduce _any_ user-facing change? Yes, it will add a new tab/page in the Spark UI ### How was this patch tested? Unit tests Closes #41964 from jasonli-db/spark-connect-ui. Authored-by: Jason Li <jason.lidatabricks.com> Signed-off-by: Gengliang Wang <gengliangapache.org> (cherry picked from commit f8786f0) Closes #42224 from juliuszsompolski/SPARK-44394-3.5. Authored-by: Jason Li <jason.li@databricks.com> Signed-off-by: Gengliang Wang <gengliang@apache.org>

## What changes were proposed in this pull request? Add a new Spark UI page to display session and execution information for Spark Connect. This builds of the work in SPARK-43923 (apache#41443) that adds the relevant SparkListenerEvents and mirrors the ThriftServerPage in the Spark UI for JDBC/ODBC. <img width="1709" alt="Screenshot 2023-07-27 at 11 29 22 PM" src="https://github.com/apache/spark/assets/65624911/934b7c69-3b44-460b-8fbb-36a9eb3f0798"> <img width="1716" alt="Screenshot 2023-07-27 at 11 29 15 PM" src="https://github.com/apache/spark/assets/65624911/33dbe6ab-44bf-49a5-ad4c-5ba4a476a1f0"> ### Why are the changes needed? This gives users a way to access session and execution information for Spark Connect via the UI and provides the frontend interface for the related SparkListenerEvents. ### Does this PR introduce _any_ user-facing change? Yes, it will add a new tab/page in the Spark UI ### How was this patch tested? Unit tests Closes apache#41964 from jasonli-db/spark-connect-ui. Authored-by: Jason Li <jason.li@databricks.com> Signed-off-by: Gengliang Wang <gengliang@apache.org>

github-actions bot added SQL WEB UI CONNECT labels Jul 12, 2023

jasonli-db force-pushed the spark-connect-ui branch 4 times, most recently from 8caf083 to b54d2b8 Compare July 14, 2023 08:38

jasonli-db changed the title ~~[WIP][SPARK-44394] Add a Spark UI page for Spark Connect~~ [SPARK-44394] Add a Spark UI page for Spark Connect Jul 14, 2023

jasonli-db changed the title ~~[SPARK-44394] Add a Spark UI page for Spark Connect~~ [SPARK-44394][CONNECT][WEBUI] Add a Spark UI page for Spark Connect Jul 14, 2023

jasonli-db marked this pull request as ready for review July 14, 2023 08:55

jasonli-db force-pushed the spark-connect-ui branch 2 times, most recently from 5c70890 to 6718961 Compare July 14, 2023 20:45

gengliangwang reviewed Jul 14, 2023

View reviewed changes

connector/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala Show resolved Hide resolved

jasonli-db force-pushed the spark-connect-ui branch from 6718961 to fcf0ca6 Compare July 14, 2023 20:56

juliuszsompolski reviewed Jul 14, 2023

View reviewed changes

...connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala Outdated Show resolved Hide resolved

jasonli-db force-pushed the spark-connect-ui branch 2 times, most recently from 5842eb0 to 7118923 Compare July 18, 2023 07:44

juliuszsompolski reviewed Jul 18, 2023

View reviewed changes

jasonli-db force-pushed the spark-connect-ui branch 2 times, most recently from 16ee63f to ff161f8 Compare July 19, 2023 17:43

jasonli-db requested a review from gengliangwang July 19, 2023 17:49

jasonli-db force-pushed the spark-connect-ui branch from ff161f8 to 78a4a0f Compare July 19, 2023 18:03

jasonli-db force-pushed the spark-connect-ui branch from 5023786 to f0c7479 Compare July 25, 2023 15:53

juliuszsompolski reviewed Jul 25, 2023

View reviewed changes

...nnect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala Outdated Show resolved Hide resolved

jasonli-db force-pushed the spark-connect-ui branch from f0c7479 to a07af7b Compare July 26, 2023 08:14

juliuszsompolski approved these changes Jul 26, 2023

View reviewed changes

jasonli-db force-pushed the spark-connect-ui branch from 3f11caf to 69f76df Compare July 27, 2023 22:33

Add a Spark UI page for Spark Connect

cf3a49e

jasonli-db force-pushed the spark-connect-ui branch 2 times, most recently from ff8f815 to 59b415c Compare July 28, 2023 00:22

Address comments and fix tests

5280131

jasonli-db force-pushed the spark-connect-ui branch from 59b415c to 5280131 Compare July 28, 2023 06:31

juliuszsompolski reviewed Jul 28, 2023

View reviewed changes

...server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerAppStatusStore.scala Show resolved Hide resolved

Show RUNNING as the state in the UI if the execution is active

8abdecd

juliuszsompolski reviewed Jul 28, 2023

View reviewed changes

jasonli-db requested a review from gengliangwang July 28, 2023 17:12

gengliangwang approved these changes Jul 28, 2023

View reviewed changes

github-actions bot added the CORE label Jul 28, 2023

jasonli-db force-pushed the spark-connect-ui branch from e4376c0 to ccab221 Compare July 28, 2023 22:58

Address issue where handler can be attached more than once

c44bc52

jasonli-db force-pushed the spark-connect-ui branch from ccab221 to c44bc52 Compare July 28, 2023 23:25

gengliangwang closed this in f8786f0 Jul 29, 2023

This was referenced Jul 29, 2023

[SPARK-44591][CONNECT][SQL] Add jobTags to SparkListenerSQLExecutionStart #42216

Closed

[SPARK-44394][CONNECT][WEBUI] Add a Spark UI page for Spark Connect #42224

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-44394][CONNECT][WEBUI] Add a Spark UI page for Spark Connect #41964

[SPARK-44394][CONNECT][WEBUI] Add a Spark UI page for Spark Connect #41964

jasonli-db commented Jul 12, 2023 •

edited

Loading

jasonli-db commented Jul 14, 2023

gengliangwang commented Jul 14, 2023

juliuszsompolski left a comment

gengliangwang commented Jul 24, 2023

jasonli-db commented Jul 24, 2023

juliuszsompolski left a comment

juliuszsompolski Jul 26, 2023

juliuszsompolski commented Jul 27, 2023

jasonli-db commented Jul 28, 2023

jasonli-db commented Jul 28, 2023

jasonli-db commented Jul 28, 2023

gengliangwang commented Jul 28, 2023

juliuszsompolski Jul 28, 2023

gengliangwang commented Jul 28, 2023

gengliangwang commented Jul 29, 2023 •

edited

Loading

gengliangwang commented Jul 29, 2023

juliuszsompolski commented Jul 29, 2023

		val executionIdOpt: Option[String] = Option(jobStart.properties)
		.flatMap { p => Option(p.getProperty(SQLExecution.EXECUTION_ID_KEY)) }

[SPARK-44394][CONNECT][WEBUI] Add a Spark UI page for Spark Connect #41964

[SPARK-44394][CONNECT][WEBUI] Add a Spark UI page for Spark Connect #41964

Conversation

jasonli-db commented Jul 12, 2023 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

jasonli-db commented Jul 14, 2023

gengliangwang commented Jul 14, 2023

juliuszsompolski left a comment

Choose a reason for hiding this comment

gengliangwang commented Jul 24, 2023

jasonli-db commented Jul 24, 2023

juliuszsompolski left a comment

Choose a reason for hiding this comment

juliuszsompolski Jul 26, 2023

Choose a reason for hiding this comment

juliuszsompolski commented Jul 27, 2023

jasonli-db commented Jul 28, 2023

jasonli-db commented Jul 28, 2023

jasonli-db commented Jul 28, 2023

gengliangwang commented Jul 28, 2023

juliuszsompolski Jul 28, 2023

Choose a reason for hiding this comment

gengliangwang commented Jul 28, 2023

gengliangwang commented Jul 29, 2023 • edited Loading

gengliangwang commented Jul 29, 2023

juliuszsompolski commented Jul 29, 2023

jasonli-db commented Jul 12, 2023 •

edited

Loading

gengliangwang commented Jul 29, 2023 •

edited

Loading