-
Notifications
You must be signed in to change notification settings - Fork 160
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make log handlers configurable, shorten entries #378
Merged
Merged
Changes from 15 commits
Commits
Show all changes
17 commits
Select commit
Hold shift + click to select a range
402379f
Fix get_logger() reuse and module name trimming
borzunov b3cde88
Fix logging in run_trainer.py
borzunov 6b2e46e
Make transformers logging use same style
borzunov ca9c618
Implement logging modes
borzunov f559bc1
Make interface more friendly
borzunov f50bbbb
Blackify
borzunov 61945b4
Fix @mryab's comments
borzunov 1c490bd
Improve names, add docstrings
borzunov d051bc9
Move HandlerMode definition
borzunov 3e186dd
Remove special case stripping "hivemind."
borzunov 825a45b
Rename fetch_collaboration_state() -> _fetch_state()
borzunov d57a75d
Move global variable definitions, remove PACKAGE_NAME
borzunov 63e0e2d
Log prefix in CollaborativeOptimizer
borzunov 6bf9073
Move enum definition
borzunov cbe402e
Fix @mryab's comments
borzunov ee1733b
Add "towards step N" to the log message
borzunov 3c6c551
Add "#" before step numbers in log messages
borzunov File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,6 +1,5 @@ | ||
#!/usr/bin/env python | ||
|
||
import logging | ||
import os | ||
import pickle | ||
from dataclasses import asdict | ||
|
@@ -18,32 +17,22 @@ | |
from transformers.trainer_utils import is_main_process | ||
|
||
import hivemind | ||
from hivemind.utils.logging import get_logger, use_hivemind_log_handler | ||
|
||
import utils | ||
from arguments import AlbertTrainingArguments, AveragerArguments, CollaborationArguments, DatasetArguments | ||
|
||
logger = logging.getLogger(__name__) | ||
LRSchedulerBase = getattr(torch.optim.lr_scheduler, "_LRScheduler", None) | ||
use_hivemind_log_handler("in_root_logger") | ||
logger = get_logger() | ||
|
||
LRSchedulerBase = getattr(torch.optim.lr_scheduler, "_LRScheduler", None) | ||
|
||
def setup_logging(training_args): | ||
logging.basicConfig( | ||
format="%(asctime)s - %(levelname)s - %(name)s - %(message)s", | ||
datefmt="%m/%d/%Y %H:%M:%S", | ||
level=logging.INFO if is_main_process(training_args.local_rank) else logging.WARN, | ||
) | ||
|
||
# Log on each process the small summary: | ||
logger.warning( | ||
f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}" | ||
+ f"distributed training: {bool(training_args.local_rank != -1)}, 16-bits training: {training_args.fp16}" | ||
) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This message is removed since |
||
# Set the verbosity to info of the Transformers logger (on main process only): | ||
if is_main_process(training_args.local_rank): | ||
def setup_transformers_logging(process_rank: int): | ||
if is_main_process(process_rank): | ||
transformers.utils.logging.set_verbosity_info() | ||
transformers.utils.logging.enable_default_handler() | ||
transformers.utils.logging.enable_explicit_format() | ||
logger.info("Training/evaluation parameters %s", training_args) | ||
transformers.utils.logging.disable_default_handler() | ||
transformers.utils.logging.enable_propagation() | ||
|
||
|
||
def get_model(training_args, config, tokenizer): | ||
|
@@ -220,7 +209,8 @@ def main(): | |
if len(collaboration_args.initial_peers) == 0: | ||
raise ValueError("Please specify at least one network endpoint in initial peers.") | ||
|
||
setup_logging(training_args) | ||
setup_transformers_logging(training_args.local_rank) | ||
logger.info(f"Training/evaluation parameters:\n{training_args}") | ||
|
||
# Set seed before initializing model. | ||
set_seed(training_args.seed) | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This
basicConfig()
setting had no effect:> This function does nothing if the root logger already has handlers configured, unless the keyword argument force is set to True.