Use `xmlrunner.XMLTestRunner` accordingly in `tests/L0/run_test.py` #1451

crcrpar · 2022-08-10T01:25:22Z

Goal

Export junit xml style report only when ~~xmlrunner is available and the environment variable of CI=1~~ --xml-report is passed to python tests/L0/run_test.py.

TODO:

Remove subTest because it's not compatible with the current way
of running L0 tests. See the example trace below

======================================================================
Traceback (most recent call last):
  File "/home/mkozuki/ghq/github.com/crcrpar/apex/tests/L0/run_test.py", line 91, in <module>
    main(args)
  File "/home/mkozuki/ghq/github.com/crcrpar/apex/tests/L0/run_test.py", line 78, in main
    result = runner.run(suite)
  File "/home/mkozuki/miniconda3/envs/torchdev/lib/python3.9/site-packages/xmlrunner/xmlrunner.py", line 421, in run
    result.printErrors()
  File "/home/mkozuki/miniconda3/envs/torchdev/lib/python3.9/unittest/runner.py", line 117, in printErrors
    self.printErrorList('FAIL', self.failures)
  File "/home/mkozuki/miniconda3/envs/torchdev/lib/python3.9/site-packages/xmlrunner/xmlrunner.py", line 217, in printErrorList
    '%s [%.3fs]: %s' % (flavour, test_info.elapsed_time,
AttributeError: '_SubTest' object has no attribute 'elapsed_time'

Follow-up

Use torch.testing._internal.common_utils.parametrize and torch.testing._internal.common_device_type.instantiate_device_type_tests(TestFooBar, globals(), only_for=("cuda",)) for the readability of log

cc @xwang233

TODO: - [ ] Remove `subTest` because it's not compatible with the current way of running L0 tests Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com>

Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com>

tests/L0/run_test.py

tests/L0/run_transformer/test_layers.py

tests/L0/run_transformer/test_random.py

Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com>

* setUpClass to define `test_` methods Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> * manually define Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com>

Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com>

…VIDIA#1451) * Use xmlrunner.XMLTestRunner accordingly TODO: - [x] Remove `subTest` because it's not compatible with the current way of running L0 tests Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> * use `torch.testing` more to enable xmlrunner Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> * Remove `subTest` for xmlrunner Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> * removing subTest Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> * not depend on an env var Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> * fix syntax errors * open with `"wb"` * xml file per dir Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> * remove comment-out Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> * Refactor `TestTransformer`: define member methods (#5) * setUpClass to define `test_` methods Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> * manually define Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> * add a missing test Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> * remove print Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> * remove ext Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com>

…VIDIA#1451) * Use xmlrunner.XMLTestRunner accordingly TODO: - [x] Remove `subTest` because it's not compatible with the current way of running L0 tests Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> * use `torch.testing` more to enable xmlrunner Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> * Remove `subTest` for xmlrunner Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> * removing subTest Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> * not depend on an env var Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> * fix syntax errors * open with `"wb"` * xml file per dir Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> * remove comment-out Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> * Refactor `TestTransformer`: define member methods (NVIDIA#5) * setUpClass to define `test_` methods Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> * manually define Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> * add a missing test Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> * remove print Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> * remove ext Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com>

crcrpar added 3 commits August 9, 2022 18:22

Use xmlrunner.XMLTestRunner accordingly

b98962c

TODO: - [ ] Remove `subTest` because it's not compatible with the current way of running L0 tests Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com>

use torch.testing more to enable xmlrunner

e6eb6d8

Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com>

Remove subTest for xmlrunner

fba2a7b

Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com>

crcrpar changed the title ~~Use xmlrunner.XMLTestRunner accordingly~~ Use xmlrunner.XMLTestRunner accordingly in tests/L0/run_test.py Aug 11, 2022

crcrpar added 2 commits August 11, 2022 12:22

removing subTest

a592743

Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com>

not depend on an env var

475ee27

Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com>

crcrpar commented Aug 11, 2022

View reviewed changes

tests/L0/run_test.py Outdated Show resolved Hide resolved

crcrpar commented Aug 11, 2022

View reviewed changes

tests/L0/run_transformer/test_layers.py Outdated Show resolved Hide resolved

tests/L0/run_transformer/test_random.py Outdated Show resolved Hide resolved

crcrpar added 4 commits August 11, 2022 14:19

fix syntax errors

cdd0881

open with "wb"

b7e0f18

xml file per dir

81bb029

Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com>

remove comment-out

b732149

Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com>

crcrpar marked this pull request as ready for review August 12, 2022 03:57

crcrpar added 4 commits August 13, 2022 20:50

Refactor TestTransformer: define member methods (#5)

6d657fe

* setUpClass to define `test_` methods Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> * manually define Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com> Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com>

add a missing test

3f4c33a

Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com>

remove print

961843d

Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com>

remove ext

b5a9035

Signed-off-by: Masaki Kozuki <mkozuki@nvidia.com>

crcrpar merged commit e75cb25 into NVIDIA:master Aug 15, 2022

crcrpar deleted the xmlrunner-support branch August 15, 2022 19:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use `xmlrunner.XMLTestRunner` accordingly in `tests/L0/run_test.py` #1451

Use `xmlrunner.XMLTestRunner` accordingly in `tests/L0/run_test.py` #1451

crcrpar commented Aug 10, 2022 •

edited

Loading

Use xmlrunner.XMLTestRunner accordingly in tests/L0/run_test.py #1451

Use xmlrunner.XMLTestRunner accordingly in tests/L0/run_test.py #1451

Conversation

crcrpar commented Aug 10, 2022 • edited Loading

Goal

Use `xmlrunner.XMLTestRunner` accordingly in `tests/L0/run_test.py` #1451

Use `xmlrunner.XMLTestRunner` accordingly in `tests/L0/run_test.py` #1451

crcrpar commented Aug 10, 2022 •

edited

Loading