Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Profiler: which steps are contained in others and which are "leaf" operations? #6405

Closed
tdvginz opened this issue Mar 8, 2021 · 0 comments · Fixed by #6618
Closed

Profiler: which steps are contained in others and which are "leaf" operations? #6405

tdvginz opened this issue Mar 8, 2021 · 0 comments · Fixed by #6618
Labels
feature Is an improvement or enhancement help wanted Open to be worked on

Comments

@tdvginz
Copy link

tdvginz commented Mar 8, 2021

Looking at the given profiler status:

Profiler Report                                                                                                                                              
                                                                                                                                                             
Action                          |  Mean duration (s)    |Num calls              |  Total time (s)       |  Percentage %         |                            
-----------------------------------------------------------------------------------------------------------------------------                                
Total                           |  -                    |_                      |  5.5595e+04           |  100 %                |                            
-----------------------------------------------------------------------------------------------------------------------------                                
run_training_epoch              |  1.0899e+04           |5                      |  5.4494e+04           |  98.02                |                            
run_training_batch              |  1.2468               |40031                  |  4.9912e+04           |  89.777               |                            
training_step_and_backward      |  1.2069               |40031                  |  4.8313e+04           |  86.902               |                            
model_forward                   |  0.67271              |40031                  |  2.6929e+04           |  48.438               |                            
optimizer_step_and_closure_0    |  1.3005               |20015                  |  2.603e+04            |  46.82                |                            
model_backward                  |  0.53271              |40031                  |  2.1325e+04           |  38.357               |                            
evaluation_step_and_end         |  0.29524              |10626                  |  3137.2               |  5.6429               |                            
on_validation_epoch_end         |  328.91               |6                      |  1973.4               |  3.5496               |                            
on_train_start                  |  1006.1               |1                      |  1006.1               |  1.8098               |
on_train_batch_end              |  0.0038002            |40030                  |  152.12               |  0.27362              |
get_train_batch                 |  0.0032787            |40031                  |  131.25               |  0.23608              |
on_validation_end               |  12.656               |6                      |  75.933               |  0.13658              |

We see that the Percentage column doesn't sum to 100, and indeed, model_forward in contained inside each run_training_batch thus its time is partial to run_training_batch time.
But what about model_backward? is it contained inside model_forward? I assume no, but it there is the ability to see the profiler in a "graph" like structure, where you can understand what operations are "leaf" nodes, or part-of some other operation, it will really ease up the analysis.

Example (taken from pytorch docs):
image

@tdvginz tdvginz added feature Is an improvement or enhancement help wanted Open to be worked on labels Mar 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Is an improvement or enhancement help wanted Open to be worked on
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant