ND behavior in test_matmul.py::test_sd_matmul

[TT-billteng]

pytest tests/ttnn/unit_tests/operations/test_matmul.py::test_sd_matmul

This appears only on N150, but I haven't been able to repro locally yet though

This is not failing on a specific VM or BM.

Some failing runs:

https://github.com/tenstorrent-metal/tt-metal/actions/runs/8558424886/job/23453740581
https://github.com/tenstorrent-metal/tt-metal/actions/runs/8558853752/job/23454745853
https://github.com/tenstorrent-metal/tt-metal/actions/runs/8545275089/job/23414372279
https://github.com/tenstorrent-metal/tt-metal/actions/runs/8542369244/job/23403999132
https://github.com/tenstorrent-metal/tt-metal/actions/runs/8559093722/job/23455622019

[TT-billteng]

not test_sd_matmul specifically, but this is a deterministic failure and may be related:

pip install pytest-repeat
pytest --count=2 tests/ttnn/unit_tests/operations/test_matmul.py -xv

PASSED tests/ttnn/unit_tests/operations/test_matmul.py::test_matmul_with_matched_width_height_from_1D[k_size=4-n_size=4-2-2]
PASSED tests/ttnn/unit_tests/operations/test_matmul.py::test_matmul_with_matched_width_height_4D[n_size=1-c=1-h=2-w=4-1-2]
SKIPPED [4] tests/ttnn/unit_tests/operations/test_matmul.py:70: ttnn.reshape doesn't support reshaping the input tensors used in this test
FAILED tests/ttnn/unit_tests/operations/test_matmul.py::test_matmul_with_matched_width_height_4D[n_size=1-c=1-h=2-w=4-2-2] - AssertionError: 0.9996211303338057

[cfjchu]

not test_sd_matmul specifically, but this is a deterministic failure and may be related:

pip install pytest-repeat pytest --count=2 tests/ttnn/unit_tests/operations/test_matmul.py -xv

PASSED tests/ttnn/unit_tests/operations/test_matmul.py::test_matmul_with_matched_width_height_from_1D[k_size=4-n_size=4-2-2]
PASSED tests/ttnn/unit_tests/operations/test_matmul.py::test_matmul_with_matched_width_height_4D[n_size=1-c=1-h=2-w=4-1-2]
SKIPPED [4] tests/ttnn/unit_tests/operations/test_matmul.py:70: ttnn.reshape doesn't support reshaping the input tensors used in this test
FAILED tests/ttnn/unit_tests/operations/test_matmul.py::test_matmul_with_matched_width_height_4D[n_size=1-c=1-h=2-w=4-2-2] - AssertionError: 0.9996211303338057

@TT-billteng that just looks like a PCC thresholding assertion maybe due to different seeds across runs?

[TT-billteng]

not test_sd_matmul specifically, but this is a deterministic failure and may be related:
pip install pytest-repeat pytest --count=2 tests/ttnn/unit_tests/operations/test_matmul.py -xv

PASSED tests/ttnn/unit_tests/operations/test_matmul.py::test_matmul_with_matched_width_height_from_1D[k_size=4-n_size=4-2-2]
PASSED tests/ttnn/unit_tests/operations/test_matmul.py::test_matmul_with_matched_width_height_4D[n_size=1-c=1-h=2-w=4-1-2]
SKIPPED [4] tests/ttnn/unit_tests/operations/test_matmul.py:70: ttnn.reshape doesn't support reshaping the input tensors used in this test
FAILED tests/ttnn/unit_tests/operations/test_matmul.py::test_matmul_with_matched_width_height_4D[n_size=1-c=1-h=2-w=4-2-2] - AssertionError: 0.9996211303338057

@TT-billteng that just looks like a PCC thresholding assertion maybe due to different seeds across runs?

Ah yes that's true, running again with lower threshold

I feel like we should use a "standard" globally-agreed upon PCC threshold to use for testing, or is this too much of an ask?

[cfjchu]

I don't think that's feasible because there could be variability based on:

• operation type
• distribution of floating point values based on seed
• stacking operations
• data formats relative to torch.float32/torch.bfloat16

[TT-billteng]

can we disable? I still see this failing on main

[eyonland]

I would rather not disable this test. @TT-BrianLiu , do we need to lower the PCC for this matmul test when running on WH or is this something bigger?

[TT-BrianLiu]

Was it not failing before? Otherwise, 0.999 is pretty reasonable pcc for a matmul

[bbradelTT]

@TT-billteng

1. Re: I feel like we should use a "standard" globally-agreed upon PCC threshold to use for testing, or is this too much of an ask?
   Too big of an ask. Model owners use PCC thresholds as close as possible to existing values to notice any changes.

Having said that, the PCCs in test_matmul.py should be updated so that anything above .999 or with too many digits would be changed.

2. test_matmul.py has many tests with skip_for_wormhole_b0. If we enable the tests again, I'm worried about All commit runtime increasing. When updating the PCCs, would you prefer to

• just enable test_sd_matmul
• enable all the tests
• enable some subset of tests, in which case, what would be the criteria?

[prajaramanTT]

@TT-billteng @bbradelTT Can we close this issue ?

[bbradelTT]

@prajaramanTT We can't.

[prajaramanTT]

@bbradelTT Do we have any updates on this ?

[bbradelTT]

I just looked into this.

On WH N150 and BH the tests pass. They are skipped on N300 since the grid is too small.

I'll create a PR to re-enable the tests.

[bbradelTT]

All post commit passed in main after the merge. Checked subsequent runs as well, and there are o failures in all post commit related to GS.

Closing.