-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add a util method for changing the log level while running #2433
add a util method for changing the log level while running #2433
Conversation
QA tests have started for PR 2433 at commit
|
QA tests have finished for PR 2433 at commit
|
Is the idea here for this to be called by spark users or by spark internal components? If the former, this won't be visible because it's in a private object. It could be nice to expose this in |
One case where this could be useful is in the Spark shell:
|
Sounds good, how about I have the spark context do the conversion and call the utils method? |
yeah, that works |
QA tests have started for PR 2433 at commit
|
QA tests have finished for PR 2433 at commit
|
Test PASSed. |
@@ -1453,6 +1453,14 @@ private[spark] object Utils extends Logging { | |||
} | |||
|
|||
/** | |||
* configure a new log4j level | |||
*/ | |||
def setLogLevel(l: org.apache.log4j.Level) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we call this setLoggingLevel
so it's consistent with the other one?
Minor comments, otherwise looks good! |
…ark context for changing the log level and add a note about overriding user-defined log settings.
QA tests have started for PR 2433 at commit
|
QA tests have finished for PR 2433 at commit
|
Test PASSed. |
Hey @holdenk forgot to mention - can you add this in Java and Python also? Guessing this will be really useful for people working on the Spark shell. |
Sure I will do that tomorrow :) |
QA tests have started for PR 2433 at commit
|
QA tests have finished for PR 2433 at commit
|
Test FAILed. |
QA tests have started for PR 2433 at commit
|
QA tests have finished for PR 2433 at commit
|
Test PASSed. |
Great! But how does this interact in situations where the downstream applications have explicitly removed Log4j from their path because they use other logging libraries (they do)? Will this throw an error? |
@tdas If we switch to a different logging library it would just be a matter of calling the new log libraries change log level function (and possibly translating some string names). |
@tdas if people are doing major dependency re-writes of Spark to replace it's logging subsystem, IMO, they should be willing to tolerate a runtime error if they try to call this. I've only seen this in the case of embedded Spark applications where the user controls all the code anyways. Holden what about enumerating the logging levels per @davies suggestion? This seems like a good idea. Modulo that suggestion, this looks good to me. |
Let's close this issue pending an update from @holdenk. |
…ontext. Based on an earlier PR, apache#2433 includes CR feedback from @pwendel & @davies
Add support for changing the log level at run time through the SparkContext. Based on an earlier PR, #2433 includes CR feedback from pwendel & davies Author: Holden Karau <holden@pigscanfly.ca> Closes #5791 from holdenk/SPARK-3444-provide-an-easy-way-to-change-log-level-r2 and squashes the following commits: 3bf3be9 [Holden Karau] fix exception 42ba873 [Holden Karau] fix exception 9117244 [Holden Karau] Only allow valid log levels, throw exception if invalid log level. 338d7bf [Holden Karau] rename setLoggingLevel to setLogLevel fac14a0 [Holden Karau] Fix style errors d9d03f3 [Holden Karau] Add support for changing the log level at run time through the SparkContext. Based on an earlier PR, #2433 includes CR feedback from @pwendel & @davies
Add support for changing the log level at run time through the SparkContext. Based on an earlier PR, apache#2433 includes CR feedback from pwendel & davies Author: Holden Karau <holden@pigscanfly.ca> Closes apache#5791 from holdenk/SPARK-3444-provide-an-easy-way-to-change-log-level-r2 and squashes the following commits: 3bf3be9 [Holden Karau] fix exception 42ba873 [Holden Karau] fix exception 9117244 [Holden Karau] Only allow valid log levels, throw exception if invalid log level. 338d7bf [Holden Karau] rename setLoggingLevel to setLogLevel fac14a0 [Holden Karau] Fix style errors d9d03f3 [Holden Karau] Add support for changing the log level at run time through the SparkContext. Based on an earlier PR, apache#2433 includes CR feedback from @pwendel & @davies
Add support for changing the log level at run time through the SparkContext. Based on an earlier PR, apache#2433 includes CR feedback from pwendel & davies Author: Holden Karau <holden@pigscanfly.ca> Closes apache#5791 from holdenk/SPARK-3444-provide-an-easy-way-to-change-log-level-r2 and squashes the following commits: 3bf3be9 [Holden Karau] fix exception 42ba873 [Holden Karau] fix exception 9117244 [Holden Karau] Only allow valid log levels, throw exception if invalid log level. 338d7bf [Holden Karau] rename setLoggingLevel to setLogLevel fac14a0 [Holden Karau] Fix style errors d9d03f3 [Holden Karau] Add support for changing the log level at run time through the SparkContext. Based on an earlier PR, apache#2433 includes CR feedback from @pwendel & @davies
Add support for changing the log level at run time through the SparkContext. Based on an earlier PR, apache#2433 includes CR feedback from pwendel & davies Author: Holden Karau <holden@pigscanfly.ca> Closes apache#5791 from holdenk/SPARK-3444-provide-an-easy-way-to-change-log-level-r2 and squashes the following commits: 3bf3be9 [Holden Karau] fix exception 42ba873 [Holden Karau] fix exception 9117244 [Holden Karau] Only allow valid log levels, throw exception if invalid log level. 338d7bf [Holden Karau] rename setLoggingLevel to setLogLevel fac14a0 [Holden Karau] Fix style errors d9d03f3 [Holden Karau] Add support for changing the log level at run time through the SparkContext. Based on an earlier PR, apache#2433 includes CR feedback from @pwendel & @davies
No description provided.