[backends] Add functionality to TRT backend #1753
Conversation
gs-olive
force-pushed
the
trt_backend_improvements
branch
3 times, most recently
from
23724b7 to
0465851
Compare
|
Thank you for the review. I think we would be interested in setting up a CI to maintain it. @narendasan - what are your thoughts on this? |
|
@xuzhao9 - what would you need from us to set up a CI to maintain the TRT backend? |
- Add argument parsing for backend arguments to pass to TRT - Add capability to specify IR via command line CLI - Add functionality to compilation path and clean up code
gs-olive
force-pushed
the
trt_backend_improvements
branch
from
0465851 to
695842c
Compare
|
@gs-olive Here is the guide on how to develop a CI: https://github.com/pytorch/benchmark/blob/main/userbenchmark/ADDING_USERBENCHMARKS.md |
- Add functionality to perform benchmarking using the Torch-TRT backend, along with output metrics
|
Hi @xuzhao9 - thanks for the reference. I've added functionality to this PR in a new commit which adds the # Invokes a singular model
python run_benchmark.py torch_trt --model resnet18 --precision fp32 --bs 4 --ir ts
# Invokes all models in the directory
python run_benchmark.py torch_trt --precision fp32 --bs 4 |
torchbenchmark/util/backends/trt.py
Outdated
|
|
||
| trt_input = [ | ||
| torch_tensorrt.Input(shape=input_.shape, dtype=input_.dtype) | ||
| for input_ in example_inputs |
There was a problem hiding this comment.
Is it necessary to add check that the type of example_inputs is List[tensor]? Actually, many models have different types of inputs.
There was a problem hiding this comment.
One suggestion is to use pytree to traverse the input and cast them to torch_tensorrt.Input. Similar to this:
benchmark/torchbenchmark/util/env_check.py
Line 186 in 9d84f9e
There was a problem hiding this comment.
Since the different ir choices for Torch-TRT process inputs differently, but all can handle Torch Tensor inputs, I passed the example inputs directly to the compiler instead, so the selected ir can handle the inputs/casting as necessary.
| all_metrics = {} | ||
|
|
||
| for Model in list_models(): | ||
| metrics = run_single_model( |
There was a problem hiding this comment.
This code will work for running a single model. However, it won't work to run a batch of models.
This is because there is no isolation between running the models. For example, model 1 might set some global torch configuration that will model 2 to be very slow or even crash (for example, torch.cudnn.benchmark). Some models have benign "memory leak" that won't cause problem in model training, but it will cause problem in benchmarking multiple models in the same process.
We suggest using the ModelTask() approach used by the torch-nightly userbenchmark: https://github.com/pytorch/benchmark/blob/main/userbenchmark/torch-nightly/run.py#L163
It will run each model in an isolated process, and doesn't have the limits mentioned above.
|
I suggest we can first remove the |
|
@xuzhao9 - Thank you for the detailed comments and review. This sounds like a good idea to me; I have removed the |
|
I have addressed the comment regarding example inputs here: #1753 (comment), with this commit: 8ccf8e6 |
|
@xuzhao9 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
cc: @narendasan