Run the verifier on Torchbench in the Nigthly CI.

[ysiraichi]

As far as I understand it, we have a nightly CI that runs Torchbench daily. I believe it would be beneficial to also run the verifier (as documented here), so as to catch, as early as, possible regressions.

cc @miladm @zpcore @JackCaoG

[zpcore]

Currently the CI script runs

python experiment_runner.py --suite-name=torchbench --accelerator=cuda --progress-bar --xla=PJRT --xla=None --dynamo=None --dynamo=openxla --dynamo=inductor

Hi @ysiraichi , correct me if I am wrong. Looks like we need to run the same command again with --verify flag to get the verification code for each configufation. Apart from that, we also need to add the field verification_code and verification_mean_rel_error etc in result_analyzer.py.

We also need to update our looker studio UI to display the verification result. I can take those action items later.

[ysiraichi]

That's about it. I don't think there's a verification_mean_rel_error anymore, though.