[benchmarks] Add no-skip argument.
#6484
Conversation
ysiraichi
requested review from
vanbasten23,
frgossen,
golechwierowicz and
zpcore
| # 2. the model is not compatible with the experiment configuration | ||
| # | ||
| # Otherwise, we should go ahead and execute it. | ||
| if (not self._args.no_skip and not self.model_loader.is_compatible( |
There was a problem hiding this comment.
don't we need to check for no_skip in line 99? hoping to save some time not loading a model we don't intend to run.
There was a problem hiding this comment.
I don't think we can do that. We still have to "load the model" for updating the environment variables of the child process (which will actually execute the benchmark). That said, this loading won't take many cycles, since we are calling with dummy=True: "instantiate the wrapper BenchmarkModel, but don't instantiate the actual torchbench model."
|
Thanks, by the way, I think we can remove |
|
@zpcore Sure. I will open a new PR doing that. |
This PR adds and argument for not skipping models.
cc @miladm