[SPARK-18364][YARN] Expose metrics for YarnShuffleService #22485

mareksimunek · 2018-09-20T08:04:14Z

What changes were proposed in this pull request?

This PR is follow-up of closed #17401 which only ended due to of inactivity, but its still nice feature to have.
Given review by @jerryshao taken in consideration and edited:

@VisibleForTesting deleted because of dependency conflicts
removed unnecessary reflection for MetricsSystemImpl
added more available types for gauge

How was this patch tested?

Manual deploy of new yarn-shuffle jar into a Node Manager and verifying that the metrics appear in the Node Manager-standard location. This is JMX with an query endpoint running on hostname:port

Resulting metrics look like this:

curl -sk -XGET hostname:port |  grep -v '#' | grep 'shuffleService'
hadoop_nodemanager_openblockrequestlatencymillis_rate15{name="shuffleService",} 0.31428910657834713
hadoop_nodemanager_blocktransferratebytes_rate15{name="shuffleService",} 566144.9983653595
hadoop_nodemanager_blocktransferratebytes_ratemean{name="shuffleService",} 2464409.9678099006
hadoop_nodemanager_openblockrequestlatencymillis_rate1{name="shuffleService",} 1.2893844732240272
hadoop_nodemanager_registeredexecutorssize{name="shuffleService",} 2.0
hadoop_nodemanager_openblockrequestlatencymillis_ratemean{name="shuffleService",} 1.255574678369966
hadoop_nodemanager_openblockrequestlatencymillis_count{name="shuffleService",} 315.0
hadoop_nodemanager_openblockrequestlatencymillis_rate5{name="shuffleService",} 0.7661929192569739
hadoop_nodemanager_registerexecutorrequestlatencymillis_ratemean{name="shuffleService",} 0.0
hadoop_nodemanager_registerexecutorrequestlatencymillis_count{name="shuffleService",} 0.0
hadoop_nodemanager_registerexecutorrequestlatencymillis_rate1{name="shuffleService",} 0.0
hadoop_nodemanager_registerexecutorrequestlatencymillis_rate5{name="shuffleService",} 0.0
hadoop_nodemanager_blocktransferratebytes_count{name="shuffleService",} 6.18271213E8
hadoop_nodemanager_registerexecutorrequestlatencymillis_rate15{name="shuffleService",} 0.0
hadoop_nodemanager_blocktransferratebytes_rate5{name="shuffleService",} 1154114.4881816586
hadoop_nodemanager_blocktransferratebytes_rate1{name="shuffleService",} 574745.0749848988

Registers the shuffle server's metrics with the Hadoop Node Manager's DefaultMetricsSystem. Test metric collector gets right converted calls camel-case shuffleService Pass scalastyle Reformat and organize imports With import order specified at http://spark.apache.org/contributing.html

tgravescs · 2018-09-20T13:00:14Z

ok to test

SparkQA · 2018-09-20T13:17:25Z

Test build #96357 has finished for PR 22485 at commit 6c96397.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class YarnShuffleServiceMetrics implements MetricsSource

ash211 · 2018-09-20T16:39:16Z

Sorry for not following through on getting this into Apache.

FWIW, it's been in the Palantir fork of Spark for over a year: https://github.com/palantir/spark/search?q=SPARK-18364&unscoped_q=SPARK-18364

tgravescs · 2018-09-20T20:36:36Z

@ash211 thanks for commenting, looks like you added palantir#236 on top of what is here. looks like it gets snapshot of the timer values to return some of the raw values and percentiles rather then just some rates, is that correct?

tgravescs · 2018-09-20T20:45:45Z

common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java

@@ -168,6 +170,15 @@ protected void serviceInit(Configuration conf) throws Exception {
      TransportConf transportConf = new TransportConf("shuffle", new HadoopConfigProvider(conf));
      blockHandler = new ExternalShuffleBlockHandler(transportConf, registeredExecutorFile);

+      // register metrics on the block handler into the Node Manager's metrics system.
+      YarnShuffleServiceMetrics serviceMetrics =
+              new YarnShuffleServiceMetrics(blockHandler.getAllMetrics());


indentation should be 2 spaces, fix here and below

tgravescs · 2018-09-21T13:56:13Z

...agers/yarn/src/test/scala/org/apache/spark/network/yarn/YarnShuffleServiceMetricsSuite.scala

+      "blockTransferRateBytes")) {
+    test(s"$testname - collector receives correct types") {
+      val builder = mock(classOf[MetricsRecordBuilder])
+      when(builder.addCounter(any(), anyLong())).thenReturn(builder)


I don't see where these when's are used?

these metrics are Timer or Meter so its called in YarnShuffleServiceMetrics.collectMetrics.
https://github.com/apache/spark/pull/22485/files#diff-04b0327ae6bce4204c03442b0e6d8718R63

tgravescs · 2018-09-21T13:56:21Z

...agers/yarn/src/test/scala/org/apache/spark/network/yarn/YarnShuffleServiceMetricsSuite.scala

+  // this metric writes only one gauge to the collector
+  test("registeredExecutorsSize - collector receives correct types") {
+    val builder = mock(classOf[MetricsRecordBuilder])
+    when(builder.addCounter(any(), anyLong())).thenReturn(builder)


Don't seem to use these?

deleted this wasnt used.

tgravescs · 2018-09-21T13:57:50Z

common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java

+              new YarnShuffleServiceMetrics(blockHandler.getAllMetrics());
+
+      MetricsSystemImpl metricsSystem = (MetricsSystemImpl) DefaultMetricsSystem.instance();
+      metricsSystem.register(


@ash211 you are calling registerSource here instead of register which required the reflection was there some reason you were doing that?

register method only did some fancy stuff on metrics name, so I didnt understand the motivation to use registerSource. Also I think Ash only follows notifications on JIRA ticket so I notified him there.

Sorry for the delay.

My explanation for using reflection was here: https://github.com/palantir/spark/pull/149/files#r107770857 -- basically registerSource isn't accessible from here.

As to why use registerSource vs register, I don't think I knew about register at the time. Looking at https://github.com/apache/hadoop/blame/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java registerSource ends up being called both ways.

If you can get the right behavior using the accessible method rather than doing reflection, I'd say we should go with that way.

cc @robert3005 @mccheah fysa for potential future merge conflict

tgravescs · 2018-09-21T14:02:59Z

...agers/yarn/src/test/scala/org/apache/spark/network/yarn/YarnShuffleServiceMetricsSuite.scala

+      when(builder.addCounter(any(), anyLong())).thenReturn(builder)
+      when(builder.addGauge(any(), anyDouble())).thenReturn(builder)
+
+      YarnShuffleServiceMetrics.collectMetric(builder, testname,


you are calling a private function here

tgravescs · 2018-09-21T14:03:08Z

...agers/yarn/src/test/scala/org/apache/spark/network/yarn/YarnShuffleServiceMetricsSuite.scala

+    when(builder.addCounter(any(), anyLong())).thenReturn(builder)
+    when(builder.addGauge(any(), anyDouble())).thenReturn(builder)
+
+    YarnShuffleServiceMetrics.collectMetric(builder, "registeredExecutorsSize",


you are calling a private function here

mareksimunek · 2018-09-24T10:25:14Z

ok to test

mareksimunek · 2018-09-24T10:28:13Z

@tgravescs thanks for review, I made corrections for your notes and probably need some permissions to retest. Could you please test this PR again?

tgravescs · 2018-09-24T13:02:15Z

ok to test

tgravescs · 2018-09-24T14:22:20Z

test this please

tgravescs · 2018-09-24T14:34:57Z

common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleServiceMetrics.java

@@ -56,7 +56,7 @@ public void getMetrics(MetricsCollector collector, boolean all) {
  /**
   * The metric types used in {@link ExternalShuffleBlockHandler.ShuffleMetrics}
   */
-  private static void collectMetric(MetricsRecordBuilder metricsRecordBuilder, String name, Metric metric) {


we should add a comment to the java "Exposed for testing."

tgravescs · 2018-09-24T14:36:59Z

common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleServiceMetrics.java

+import org.apache.hadoop.metrics2.MetricsCollector;
+import org.apache.hadoop.metrics2.MetricsInfo;
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.hadoop.metrics2.MetricsSource;


add a newline between hadoop and spark

spark import removed because of checkstyle wont allow to have import in javadoc when you have {@link className} syntax. I think checkstyle rules should be corrected.

SparkQA · 2018-09-24T15:00:16Z

Test build #96508 has finished for PR 22485 at commit 764404a.

This patch fails Java style tests.
This patch merges cleanly.
This patch adds no public classes.

tgravescs · 2018-09-24T15:53:48Z

common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleServiceMetrics.java

+import org.apache.hadoop.metrics2.MetricsSource;
+import org.apache.spark.network.shuffle.ExternalShuffleBlockHandler;
+
+import java.util.Map;


nit: java includes should go before the org.apache ones

tgravescs · 2018-09-24T19:10:34Z

common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleServiceMetrics.java

+   */
+  @Override
+  public void getMetrics(MetricsCollector collector, boolean all) {
+    MetricsRecordBuilder metricsRecordBuilder = collector.addRecord("shuffleService");


I think we should rename this to be sparkShuffleService in order to differentiate from other shuffle servers that could be on the nodemanager (MR/tez).

tgravescs · 2018-09-24T19:11:06Z

common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleServiceMetrics.java

+
+  /**
+   * The metric types used in {@link ExternalShuffleBlockHandler.ShuffleMetrics}
+   */


add a comment to javadoc that its visible for testing

tgravescs · 2018-09-24T19:13:34Z

few minor nits that I missed in my first pass, otherwise looks good.

tgravescs · 2018-09-24T19:14:43Z

test this please

SparkQA · 2018-09-24T19:34:45Z

Test build #96516 has finished for PR 22485 at commit 764404a.

This patch fails Java style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-09-25T10:50:01Z

Test build #96547 has finished for PR 22485 at commit 95856e9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tgravescs

Thanks for the updates, one last minor comment, otherwise lgtm, I'll be offline til next Monday so will commit then if no one else gets to first.

tgravescs · 2018-09-25T13:13:14Z

common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java

+
+      MetricsSystemImpl metricsSystem = (MetricsSystemImpl) DefaultMetricsSystem.instance();
+      metricsSystem.register(
+        "shuffleService", "Metrics on the Spark Shuffle Service", serviceMetrics);


Probably best to make this sparkshuffleservice as well.

SparkQA · 2018-09-25T13:53:39Z

Test build #96552 has finished for PR 22485 at commit e259b6f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tgravescs

+1

tgravescs · 2018-10-01T16:05:03Z

merged to master, thanks @mareksimunek

…gistered connections as well as the number of active connections to YARN Shuffle Service Recently, the ability to expose the metrics for YARN Shuffle Service was added as part of [SPARK-18364](apache#22485). We need to add some metrics to be able to determine the number of active connections as well as open connections to the external shuffle service to benchmark network and connection issues on large cluster environments. Added two more shuffle server metrics for Spark Yarn shuffle service: numRegisteredConnections which indicate the number of registered connections to the shuffle service and numActiveConnections which indicate the number of active connections to the shuffle service at any given point in time. If these metrics are outputted to a file, we get something like this: 1533674653489 default.shuffleService: Hostname=server1.abc.com, openBlockRequestLatencyMillis_count=729, openBlockRequestLatencyMillis_rate15=0.7110833548897356, openBlockRequestLatencyMillis_rate5=1.657808981793011, openBlockRequestLatencyMillis_rate1=2.2404486061620474, openBlockRequestLatencyMillis_rateMean=0.9242558551196706, numRegisteredConnections=35, blockTransferRateBytes_count=2635880512, blockTransferRateBytes_rate15=2578547.6094160094, blockTransferRateBytes_rate5=6048721.726302424, blockTransferRateBytes_rate1=8548922.518223226, blockTransferRateBytes_rateMean=3341878.633637769, registeredExecutorsSize=5, registerExecutorRequestLatencyMillis_count=5, registerExecutorRequestLatencyMillis_rate15=0.0027973949328659836, registerExecutorRequestLatencyMillis_rate5=0.0021278007987206426, registerExecutorRequestLatencyMillis_rate1=2.8270296777387467E-6, registerExecutorRequestLatencyMillis_rateMean=0.006339206380043053, numActiveConnections=35 Closes apache#22498 from pgandhi999/SPARK-18364. Authored-by: pgandhi <pgandhi@oath.com> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>

## What changes were proposed in this pull request? This PR is follow-up of closed apache#17401 which only ended due to of inactivity, but its still nice feature to have. Given review by jerryshao taken in consideration and edited: - VisibleForTesting deleted because of dependency conflicts - removed unnecessary reflection for `MetricsSystemImpl` - added more available types for gauge ## How was this patch tested? Manual deploy of new yarn-shuffle jar into a Node Manager and verifying that the metrics appear in the Node Manager-standard location. This is JMX with an query endpoint running on `hostname:port` Resulting metrics look like this: ``` curl -sk -XGET hostname:port | grep -v '#' | grep 'shuffleService' hadoop_nodemanager_openblockrequestlatencymillis_rate15{name="shuffleService",} 0.31428910657834713 hadoop_nodemanager_blocktransferratebytes_rate15{name="shuffleService",} 566144.9983653595 hadoop_nodemanager_blocktransferratebytes_ratemean{name="shuffleService",} 2464409.9678099006 hadoop_nodemanager_openblockrequestlatencymillis_rate1{name="shuffleService",} 1.2893844732240272 hadoop_nodemanager_registeredexecutorssize{name="shuffleService",} 2.0 hadoop_nodemanager_openblockrequestlatencymillis_ratemean{name="shuffleService",} 1.255574678369966 hadoop_nodemanager_openblockrequestlatencymillis_count{name="shuffleService",} 315.0 hadoop_nodemanager_openblockrequestlatencymillis_rate5{name="shuffleService",} 0.7661929192569739 hadoop_nodemanager_registerexecutorrequestlatencymillis_ratemean{name="shuffleService",} 0.0 hadoop_nodemanager_registerexecutorrequestlatencymillis_count{name="shuffleService",} 0.0 hadoop_nodemanager_registerexecutorrequestlatencymillis_rate1{name="shuffleService",} 0.0 hadoop_nodemanager_registerexecutorrequestlatencymillis_rate5{name="shuffleService",} 0.0 hadoop_nodemanager_blocktransferratebytes_count{name="shuffleService",} 6.18271213E8 hadoop_nodemanager_registerexecutorrequestlatencymillis_rate15{name="shuffleService",} 0.0 hadoop_nodemanager_blocktransferratebytes_rate5{name="shuffleService",} 1154114.4881816586 hadoop_nodemanager_blocktransferratebytes_rate1{name="shuffleService",} 574745.0749848988 ``` Closes apache#22485 from mareksimunek/SPARK-18364. Lead-authored-by: marek.simunek <marek.simunek@firma.seznam.cz> Co-authored-by: Andrew Ash <andrew@andrewash.com> Signed-off-by: Thomas Graves <tgraves@apache.org>

…gistered connections as well as the number of active connections to YARN Shuffle Service Recently, the ability to expose the metrics for YARN Shuffle Service was added as part of [SPARK-18364](apache#22485). We need to add some metrics to be able to determine the number of active connections as well as open connections to the external shuffle service to benchmark network and connection issues on large cluster environments. Added two more shuffle server metrics for Spark Yarn shuffle service: numRegisteredConnections which indicate the number of registered connections to the shuffle service and numActiveConnections which indicate the number of active connections to the shuffle service at any given point in time. If these metrics are outputted to a file, we get something like this: 1533674653489 default.shuffleService: Hostname=server1.abc.com, openBlockRequestLatencyMillis_count=729, openBlockRequestLatencyMillis_rate15=0.7110833548897356, openBlockRequestLatencyMillis_rate5=1.657808981793011, openBlockRequestLatencyMillis_rate1=2.2404486061620474, openBlockRequestLatencyMillis_rateMean=0.9242558551196706, numRegisteredConnections=35, blockTransferRateBytes_count=2635880512, blockTransferRateBytes_rate15=2578547.6094160094, blockTransferRateBytes_rate5=6048721.726302424, blockTransferRateBytes_rate1=8548922.518223226, blockTransferRateBytes_rateMean=3341878.633637769, registeredExecutorsSize=5, registerExecutorRequestLatencyMillis_count=5, registerExecutorRequestLatencyMillis_rate15=0.0027973949328659836, registerExecutorRequestLatencyMillis_rate5=0.0021278007987206426, registerExecutorRequestLatencyMillis_rate1=2.8270296777387467E-6, registerExecutorRequestLatencyMillis_rateMean=0.006339206380043053, numActiveConnections=35 Closes apache#22498 from pgandhi999/SPARK-18364. Authored-by: pgandhi <pgandhi@oath.com> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>

## What changes were proposed in this pull request? This PR is follow-up of closed apache#17401 which only ended due to of inactivity, but its still nice feature to have. Given review by jerryshao taken in consideration and edited: - VisibleForTesting deleted because of dependency conflicts - removed unnecessary reflection for `MetricsSystemImpl` - added more available types for gauge ## How was this patch tested? Manual deploy of new yarn-shuffle jar into a Node Manager and verifying that the metrics appear in the Node Manager-standard location. This is JMX with an query endpoint running on `hostname:port` Resulting metrics look like this: ``` curl -sk -XGET hostname:port | grep -v '#' | grep 'shuffleService' hadoop_nodemanager_openblockrequestlatencymillis_rate15{name="shuffleService",} 0.31428910657834713 hadoop_nodemanager_blocktransferratebytes_rate15{name="shuffleService",} 566144.9983653595 hadoop_nodemanager_blocktransferratebytes_ratemean{name="shuffleService",} 2464409.9678099006 hadoop_nodemanager_openblockrequestlatencymillis_rate1{name="shuffleService",} 1.2893844732240272 hadoop_nodemanager_registeredexecutorssize{name="shuffleService",} 2.0 hadoop_nodemanager_openblockrequestlatencymillis_ratemean{name="shuffleService",} 1.255574678369966 hadoop_nodemanager_openblockrequestlatencymillis_count{name="shuffleService",} 315.0 hadoop_nodemanager_openblockrequestlatencymillis_rate5{name="shuffleService",} 0.7661929192569739 hadoop_nodemanager_registerexecutorrequestlatencymillis_ratemean{name="shuffleService",} 0.0 hadoop_nodemanager_registerexecutorrequestlatencymillis_count{name="shuffleService",} 0.0 hadoop_nodemanager_registerexecutorrequestlatencymillis_rate1{name="shuffleService",} 0.0 hadoop_nodemanager_registerexecutorrequestlatencymillis_rate5{name="shuffleService",} 0.0 hadoop_nodemanager_blocktransferratebytes_count{name="shuffleService",} 6.18271213E8 hadoop_nodemanager_registerexecutorrequestlatencymillis_rate15{name="shuffleService",} 0.0 hadoop_nodemanager_blocktransferratebytes_rate5{name="shuffleService",} 1154114.4881816586 hadoop_nodemanager_blocktransferratebytes_rate1{name="shuffleService",} 574745.0749848988 ``` Closes apache#22485 from mareksimunek/SPARK-18364. Lead-authored-by: marek.simunek <marek.simunek@firma.seznam.cz> Co-authored-by: Andrew Ash <andrew@andrewash.com> Signed-off-by: Thomas Graves <tgraves@apache.org> (cherry picked from commit a802c69)

…gistered connections as well as the number of active connections to YARN Shuffle Service Recently, the ability to expose the metrics for YARN Shuffle Service was added as part of [SPARK-18364](apache#22485). We need to add some metrics to be able to determine the number of active connections as well as open connections to the external shuffle service to benchmark network and connection issues on large cluster environments. Added two more shuffle server metrics for Spark Yarn shuffle service: numRegisteredConnections which indicate the number of registered connections to the shuffle service and numActiveConnections which indicate the number of active connections to the shuffle service at any given point in time. If these metrics are outputted to a file, we get something like this: 1533674653489 default.shuffleService: Hostname=server1.abc.com, openBlockRequestLatencyMillis_count=729, openBlockRequestLatencyMillis_rate15=0.7110833548897356, openBlockRequestLatencyMillis_rate5=1.657808981793011, openBlockRequestLatencyMillis_rate1=2.2404486061620474, openBlockRequestLatencyMillis_rateMean=0.9242558551196706, numRegisteredConnections=35, blockTransferRateBytes_count=2635880512, blockTransferRateBytes_rate15=2578547.6094160094, blockTransferRateBytes_rate5=6048721.726302424, blockTransferRateBytes_rate1=8548922.518223226, blockTransferRateBytes_rateMean=3341878.633637769, registeredExecutorsSize=5, registerExecutorRequestLatencyMillis_count=5, registerExecutorRequestLatencyMillis_rate15=0.0027973949328659836, registerExecutorRequestLatencyMillis_rate5=0.0021278007987206426, registerExecutorRequestLatencyMillis_rate1=2.8270296777387467E-6, registerExecutorRequestLatencyMillis_rateMean=0.006339206380043053, numActiveConnections=35 Closes apache#22498 from pgandhi999/SPARK-18364. Authored-by: pgandhi <pgandhi@oath.com> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com> (cherry picked from commit 8dd29fe)

This PR is follow-up of closed apache#17401 which only ended due to of inactivity, but its still nice feature to have. Given review by jerryshao taken in consideration and edited: - VisibleForTesting deleted because of dependency conflicts - removed unnecessary reflection for `MetricsSystemImpl` - added more available types for gauge Manual deploy of new yarn-shuffle jar into a Node Manager and verifying that the metrics appear in the Node Manager-standard location. This is JMX with an query endpoint running on `hostname:port` Resulting metrics look like this: ``` curl -sk -XGET hostname:port | grep -v '#' | grep 'shuffleService' hadoop_nodemanager_openblockrequestlatencymillis_rate15{name="shuffleService",} 0.31428910657834713 hadoop_nodemanager_blocktransferratebytes_rate15{name="shuffleService",} 566144.9983653595 hadoop_nodemanager_blocktransferratebytes_ratemean{name="shuffleService",} 2464409.9678099006 hadoop_nodemanager_openblockrequestlatencymillis_rate1{name="shuffleService",} 1.2893844732240272 hadoop_nodemanager_registeredexecutorssize{name="shuffleService",} 2.0 hadoop_nodemanager_openblockrequestlatencymillis_ratemean{name="shuffleService",} 1.255574678369966 hadoop_nodemanager_openblockrequestlatencymillis_count{name="shuffleService",} 315.0 hadoop_nodemanager_openblockrequestlatencymillis_rate5{name="shuffleService",} 0.7661929192569739 hadoop_nodemanager_registerexecutorrequestlatencymillis_ratemean{name="shuffleService",} 0.0 hadoop_nodemanager_registerexecutorrequestlatencymillis_count{name="shuffleService",} 0.0 hadoop_nodemanager_registerexecutorrequestlatencymillis_rate1{name="shuffleService",} 0.0 hadoop_nodemanager_registerexecutorrequestlatencymillis_rate5{name="shuffleService",} 0.0 hadoop_nodemanager_blocktransferratebytes_count{name="shuffleService",} 6.18271213E8 hadoop_nodemanager_registerexecutorrequestlatencymillis_rate15{name="shuffleService",} 0.0 hadoop_nodemanager_blocktransferratebytes_rate5{name="shuffleService",} 1154114.4881816586 hadoop_nodemanager_blocktransferratebytes_rate1{name="shuffleService",} 574745.0749848988 ``` Closes apache#22485 from mareksimunek/SPARK-18364. Lead-authored-by: marek.simunek <marek.simunek@firma.seznam.cz> Co-authored-by: Andrew Ash <andrew@andrewash.com> Signed-off-by: Thomas Graves <tgraves@apache.org> (cherry picked from commit a802c69) (cherry picked from commit d1bb3ea1eed2ea037d97ca9ac95b514ecaf6d50a) Change-Id: Ia7ed71f9de822e7c968f7618eb0f1b86e5d49912 (cherry picked from commit af2e4cd6f754222888272de16ff4c3df88c2c38b)

ash211 and others added 2 commits September 19, 2018 15:54

[SPARK-18364][YARN] YarnShuffleService metrics correction

6c96397

tgravescs reviewed Sep 21, 2018

View reviewed changes

[SPARK-18364][YARN] YarnShuffleService metrics tests fix

764404a

tgravescs reviewed Sep 24, 2018

View reviewed changes

tgravescs reviewed Sep 25, 2018

View reviewed changes

[SPARK-18364][YARN] YarnShuffleService code style corrections

e259b6f

mareksimunek force-pushed the SPARK-18364 branch from 95856e9 to e259b6f Compare September 25, 2018 13:25

felixcheung approved these changes Oct 1, 2018

View reviewed changes

tgravescs approved these changes Oct 1, 2018

View reviewed changes

asfgit closed this in a802c69 Oct 1, 2018

pgandhi999 mentioned this pull request Oct 4, 2018

[SPARK-25642] : Adding two new metrics to record the number of registered connections as well as the number of active connections to YARN Shuffle Service #22498

Closed

[SPARK-18364][YARN] Expose metrics for YarnShuffleService #22485

[SPARK-18364][YARN] Expose metrics for YarnShuffleService #22485

Conversation

mareksimunek commented Sep 20, 2018

What changes were proposed in this pull request?

How was this patch tested?

tgravescs commented Sep 20, 2018

SparkQA commented Sep 20, 2018

ash211 commented Sep 20, 2018

tgravescs commented Sep 20, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mareksimunek commented Sep 24, 2018

mareksimunek commented Sep 24, 2018

tgravescs commented Sep 24, 2018

tgravescs commented Sep 24, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Sep 24, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tgravescs commented Sep 24, 2018

tgravescs commented Sep 24, 2018

SparkQA commented Sep 24, 2018

SparkQA commented Sep 25, 2018

tgravescs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Sep 25, 2018

tgravescs left a comment

Choose a reason for hiding this comment

tgravescs commented Oct 1, 2018