Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor add straighten algo interface #8435

Merged
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/source/graph.rst
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ Base class for running neural networks in Static Graph Mode.
set_zero_redundancy_optimizer_mode,
set_zero_redundancy_optimizer_min_size_after_split,
enable_cudnn_conv_heuristic_search_algo,
enable_random_straighten_nodes,
:member-order: bysource


Expand Down
4 changes: 2 additions & 2 deletions oneflow/core/graph/task_graph.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -420,7 +420,7 @@ void ForEachOpGraphNecessaryCtrlEdge(

} // namespace

TaskGraph::TaskGraph() {
TaskGraph::TaskGraph(bool random_straighten_nodes) {
OpGraph* op_graph = Global<OpGraph>::Get();
sub_tsk_gph_builder_ctx_.reset(new SubTskGphBuilderCtx(this));
boxing_logger_ = CreateBoxingLogger();
Expand Down Expand Up @@ -451,7 +451,7 @@ TaskGraph::TaskGraph() {
}
});

if (ParseBooleanFromEnv("ONEFLOW_RANDOM_STRAIGHTEN_NODES", false)) {
if (random_straighten_nodes) {
SetOrderInGraphForEachNode();
} else {
StraightenNodes(this, &ordered_task_nodes_);
Expand Down
2 changes: 1 addition & 1 deletion oneflow/core/graph/task_graph.h
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ class TaskGraph final : public Graph<TaskNode, TaskEdge> {
OF_DISALLOW_COPY_AND_MOVE(TaskGraph);
~TaskGraph() override;

explicit TaskGraph();
explicit TaskGraph(bool random_straighten_nodes);

const char* TypeName() const override { return "TaskGraph"; }
void RemoveEmptyRegsts();
Expand Down
3 changes: 2 additions & 1 deletion oneflow/core/job/compiler.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,8 @@ void Compiler::Compile(Job* job, Plan* plan, bool need_job_complete) const {

// Step3: build task_gph.
// TODO(levi): we can rewrite this part of code in visitor pattern.
auto task_gph = std::make_unique<TaskGraph>();
auto task_gph =
std::make_unique<TaskGraph>(job->job_conf().random_straighten_nodes_in_task_graph());
using std::placeholders::_1;
task_gph->ForEachNode(std::bind(&TaskNode::ProduceAllRegstsAndBindEdges, _1));
task_gph->ForEachNode(std::bind(&TaskNode::ConsumeAllRegsts, _1));
Expand Down
2 changes: 2 additions & 0 deletions oneflow/core/job/job_conf.proto
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,8 @@ message JobConfigProto {
optional bool cudnn_conv_enable_pseudo_half = 600 [default = true];
optional bool enable_auto_mixed_precision = 602 [default = false];
optional bool enable_quantization_aware_training = 603 [default = false];

optional bool random_straighten_nodes_in_task_graph = 700 [default = false];

optional int64 concurrency_width = 1000 [default = 128];

Expand Down
10 changes: 10 additions & 0 deletions python/oneflow/nn/graph/graph_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -269,6 +269,16 @@ def build(self, x):
"""
self.proto.cudnn_conv_heuristic_search_algo = mode

def enable_random_straighten_nodes(self, mode: bool = False):
r""" Whether turn off the straighten algorithm.

If using nccl compute stream, turning it on will not speed up the training.
If not using nccl compute stream, turning it on will slow down data parallelism by 0.6% and slow down model parallelism by 6%.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

slow down ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个选项打开表示用旧的算法,也就是随机拓扑排序,所以会慢 🤣

这个文档得再描述清一些,要不用户看见还会迷惑 @Yipeng1994

wyg1997 marked this conversation as resolved.
Show resolved Hide resolved

The switch is off by default (i.e. use the straighten algorithm by default).
"""
self.proto.random_straighten_nodes_in_task_graph = mode

def _generate_optimizer_and_variable_configs(
self, opt_dict: OptDict = None, variables_conf: OrderedDict = None,
):
Expand Down