[branch-1.0][SPARK-4355] ColumnStatisticsAggregator doesn't merge mean correctly

[mengxr]

This backports the bug fix in #3220.

[SparkQA]

Test build #24914 has started for PR 3850 at commit ae9b94a.

• This patch merges cleanly.

[SparkQA]

Test build #24914 timed out for PR 3850 at commit ae9b94a after a configured wait of 120m.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24914/
Test FAILed.

[mengxr]

test this please

[SparkQA]

Test build #24962 has started for PR 3850 at commit ae9b94a.

• This patch merges cleanly.

[SparkQA]

Test build #24962 timed out for PR 3850 at commit ae9b94a after a configured wait of 120m.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24962/
Test FAILed.

[mengxr]

@JoshRosen It seems that the unit tests took more than 120 minutes. Could you take a look?

[JoshRosen]

I don't see any obvious culprits in the log, although Jenkins was pretty heavily loaded when that test ran. I suppose you could try testing again? Otherwise, I can help you to log into Jenkins to look at the actual unit-test.log files.

[mengxr]

test this please

[SparkQA]

Test build #24973 has started for PR 3850 at commit ae9b94a.

• This patch merges cleanly.

[SparkQA]

Test build #24973 has finished for PR 3850 at commit ae9b94a.

• This patch fails some tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24973/
Test FAILed.

[mengxr]

@JoshRosen It seems that the branch-1.0 build is broken: https://amplab.cs.berkeley.edu/jenkins/job/Spark-1.0-Maven-pre-YARN/

[JoshRosen]

Some of those are new flaky failures, while others are longstanding issues that still need to be fixed: https://issues.apache.org/jira/browse/SPARK-5039

[JoshRosen]

Actually, this is timely, since there's now #3864 for that linked JIRA

[JoshRosen]

This looks like a legitimate build break / test failure in branch-1.0, possibly related to SPARK_HOME settings in the sbt build; I'm investigating now; see #3864 for more details.

[mengxr]

test this please

[SparkQA]

Test build #25355 has started for PR 3850 at commit ae9b94a.

• This patch merges cleanly.

[SparkQA]

Test build #25355 has finished for PR 3850 at commit ae9b94a.

• This patch fails some tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25355/
Test FAILed.

[SparkQA]

Test build #26529 has started for PR 3850 at commit ae9b94a.

• This patch merges cleanly.

[SparkQA]

Test build #26529 has finished for PR 3850 at commit ae9b94a.

• This patch fails some tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26529/
Test FAILed.

[rxin]

What do we do with this?

[JoshRosen]

@rxin I think folks have been holding off on merging branch 1.0 backports because the tests usually fail there, although in many cases I think that's because our test stability improvements haven't been backported to that branch, leaving those tests much more prone to spurious failures due to timing issues port contention, etc.

Since the number of branch-1.0 backports is so small, we could just merge them in now as long as they compile and choose to fix up the tests later or run the tests in an environment with less contention (e.g. a dedicated EC2 node rather than AMPLab Jenkins). This isn't an ideal situation, but I think it might be better than just closing these PRs or leaving them to clutter up the review queue.

[JoshRosen]

Jenkins, retest this please.

[SparkQA]

Test build #27034 has started for PR 3850 at commit ae9b94a.

• This patch merges cleanly.

[SparkQA]

Test build #27034 has finished for PR 3850 at commit ae9b94a.

• This patch fails some tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27034/
Test FAILed.

[mengxr]

test this please

[SparkQA]

Test build #27683 has started for PR 3850 at commit ae9b94a.

• This patch merges cleanly.

[SparkQA]

Test build #27683 has finished for PR 3850 at commit ae9b94a.

• This patch fails some tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27683/
Test FAILed.

[SparkQA]

Test build #27818 has started for PR 3850 at commit ae9b94a.

• This patch merges cleanly.

[SparkQA]

Test build #27818 has finished for PR 3850 at commit ae9b94a.

• This patch fails some tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27818/
Test FAILed.

[srowen]

In order to close this out, is it worth just merging this into 1.0 as @JoshRosen suggests?
Same for some of these other back-ports to 0.9 or 1.0.

[mengxr]

Okay. Merge into branch-1.0:)