[SPARK-7277][SQL] Throw exception if the property mapred.reduce.tasks is set to -1 #5811

viirya · 2015-04-30T18:09:31Z

JIRA: https://issues.apache.org/jira/browse/SPARK-7277

As automatically determining the number of reducers is not supported (mapred.reduce.tasks is set to -1), we should throw exception to users.

SparkQA · 2015-04-30T19:51:29Z

Test build #31436 has finished for PR 5811 at commit 68a1c70.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- "public class " + className + extendsText + " implements java.io.Serializable
This patch does not change any dependencies.

squito · 2015-04-30T21:20:11Z

sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala

+      if (value == "-1") {
+        logWarning(
+          s"Set this property to -1 for automatically determining the number of reducers " +
+            s"is not supported, showing current ${SQLConf.SHUFFLE_PARTITIONS} instead.")


typo: "Setting this property ..."

actually can I suggest a small rewording a think is a little clearer?

Setting this property to -1 for automatically determing the number of reducers is not supported. Ignoring request, and showing ${SQLConf.SHUFFLE_PARTITIONS} instead

squito · 2015-04-30T21:26:44Z

could you also add a test that things work when you try to set it to -1? from the jira it seems that was a major goal here, to prevent incorrect results.

squito · 2015-04-30T21:28:58Z

though I wonder if instead it should fail (with a good error) if you give a bad value, rather than just logging a message and ignoring the request. any thoughts, @marmbrus?

marmbrus · 2015-04-30T21:40:52Z

Warnings seem likely to be missed. I would throw an exception.

viirya · 2015-05-01T03:00:48Z

Thanks. Updated to throw exception now. Also added a test for it.

SparkQA · 2015-05-01T03:16:06Z

Test build #31504 has finished for PR 5811 at commit 4ede705.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

viirya · 2015-05-01T03:19:27Z

should not be a related failure.

viirya · 2015-05-01T03:19:51Z

test again please.

squito · 2015-05-01T13:39:03Z

Jenkins, retest this please

SparkQA · 2015-05-01T15:25:22Z

Test build #31552 has finished for PR 5811 at commit 4ede705.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

squito · 2015-05-01T15:32:25Z

lgtm

TanninOne · 2015-05-04T08:23:31Z

Has anyone verified that a string comparison against -1 completely fully fixes the issue? Isn't it interpreted as an integer later?
What I'm getting at is: What if the user has the variable to "-01". or to "-2"? Maybe those were valid values for mapred.reduce.tasks as well? How was zero handled in hadoop?
Wouldn't it be safer and cleaner to test any variable against a valid range?
I.e. "if (Integer.parseInt("value") < 1) ..."

viirya · 2015-05-04T09:35:44Z

Setting mapred.reduce.tasks to '-1' has special meaning that asks Hive to automatically determine the number of reducers. It is the default value in hive-default.xml. You can also refer to HADOOPNUMREDUCERS.

Since it will parse the given string value, the configuration value of "-01" or "-2" should be able to be parsed to integer value too. As I can tell from the Hive sources, it checks if mapred.reduce.tasks is greater than 0 before using it in SemanticAnalyzer and SetReducerParallelism. So I think these values "-01", "-2" would have the same effect as setting it as '-1'.

Although I don't think it will be possible to have such values as setting values for mapred.reduce.tasks, it is still good to consider these cases. I will update it later.

TanninOne · 2015-05-04T11:39:16Z

Great, thanks for looking into this.

SparkQA · 2015-05-04T11:57:10Z

Test build #31754 has finished for PR 5811 at commit e518f96.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait LDAOptimizer
- class EMLDAOptimizer extends LDAOptimizer
- class OnlineLDAOptimizer extends LDAOptimizer

…s is set to -1 JIRA: https://issues.apache.org/jira/browse/SPARK-7277 As automatically determining the number of reducers is not supported (`mapred.reduce.tasks` is set to `-1`), we should throw exception to users. Author: Liang-Chi Hsieh <viirya@gmail.com> Closes #5811 from viirya/no_neg_reduce_tasks and squashes the following commits: e518f96 [Liang-Chi Hsieh] Consider other wrong setting values. fd9c817 [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into no_neg_reduce_tasks 4ede705 [Liang-Chi Hsieh] Throw exception instead of warning message. 68a1c70 [Liang-Chi Hsieh] Show warning message if mapred.reduce.tasks is set to -1. (cherry picked from commit ea3077f) Signed-off-by: Michael Armbrust <michael@databricks.com>

marmbrus · 2015-05-07T23:23:22Z

Thanks, merged to master and 1.4

…s is set to -1 JIRA: https://issues.apache.org/jira/browse/SPARK-7277 As automatically determining the number of reducers is not supported (`mapred.reduce.tasks` is set to `-1`), we should throw exception to users. Author: Liang-Chi Hsieh <viirya@gmail.com> Closes apache#5811 from viirya/no_neg_reduce_tasks and squashes the following commits: e518f96 [Liang-Chi Hsieh] Consider other wrong setting values. fd9c817 [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into no_neg_reduce_tasks 4ede705 [Liang-Chi Hsieh] Throw exception instead of warning message. 68a1c70 [Liang-Chi Hsieh] Show warning message if mapred.reduce.tasks is set to -1.

Show warning message if mapred.reduce.tasks is set to -1.

68a1c70

squito reviewed Apr 30, 2015
View reviewed changes

viirya changed the title ~~[SPARK-7277][SQL] Show warning message if the property mapred.reduce.tasks is set to -1~~ [SPARK-7277][SQL] Throw exception if the property mapred.reduce.tasks is set to -1 May 1, 2015

Throw exception instead of warning message.

4ede705

viirya added 2 commits May 4, 2015 17:37

Merge remote-tracking branch 'upstream/master' into no_neg_reduce_tasks

fd9c817

Consider other wrong setting values.

e518f96

asfgit closed this in ea3077f May 7, 2015

viirya deleted the no_neg_reduce_tasks branch December 27, 2023 18:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-7277][SQL] Throw exception if the property mapred.reduce.tasks is set to -1 #5811

[SPARK-7277][SQL] Throw exception if the property mapred.reduce.tasks is set to -1 #5811

viirya commented Apr 30, 2015

SparkQA commented Apr 30, 2015

squito Apr 30, 2015

squito Apr 30, 2015

squito commented Apr 30, 2015

squito commented Apr 30, 2015

marmbrus commented Apr 30, 2015

viirya commented May 1, 2015

SparkQA commented May 1, 2015

viirya commented May 1, 2015

viirya commented May 1, 2015

squito commented May 1, 2015

SparkQA commented May 1, 2015

squito commented May 1, 2015

TanninOne commented May 4, 2015

viirya commented May 4, 2015

TanninOne commented May 4, 2015

SparkQA commented May 4, 2015

marmbrus commented May 7, 2015

[SPARK-7277][SQL] Throw exception if the property mapred.reduce.tasks is set to -1 #5811

[SPARK-7277][SQL] Throw exception if the property mapred.reduce.tasks is set to -1 #5811

Conversation

viirya commented Apr 30, 2015

SparkQA commented Apr 30, 2015

squito Apr 30, 2015

Choose a reason for hiding this comment

squito Apr 30, 2015

Choose a reason for hiding this comment

squito commented Apr 30, 2015

squito commented Apr 30, 2015

marmbrus commented Apr 30, 2015

viirya commented May 1, 2015

SparkQA commented May 1, 2015

viirya commented May 1, 2015

viirya commented May 1, 2015

squito commented May 1, 2015

SparkQA commented May 1, 2015

squito commented May 1, 2015

TanninOne commented May 4, 2015

viirya commented May 4, 2015

TanninOne commented May 4, 2015

SparkQA commented May 4, 2015

marmbrus commented May 7, 2015